2023-10-16T11:57:58.795 INFO:root:teuthology version: 0.0.1.dev197+g8cdab07 2023-10-16T11:57:58.796 DEBUG:teuthology.run:Teuthology command: teuthology --owner mchangir --description fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} --verbose --name mchangir-2023-10-16_11:33:41-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi --archive /home/teuthworker/archive/mchangir-2023-10-16_11:33:41-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi/7428984 -- /home/teuthworker/archive/mchangir-2023-10-16_11:33:41-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi/7428984/orig.config.yaml 2023-10-16T11:57:58.843 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-16T11:57:58.922 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/mchangir-2023-10-16_11:33:41-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi/7428984 branch: wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} email: mchangir@redhat.com first_in_suite: false job_id: '7428984' kernel: client: sha1: distro kdb: true sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: mchangir-2023-10-16_11:33:41-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '8.6' overrides: admin_socket: branch: wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug finisher: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding log-whitelist: - OSD full dropping all updates - OSD near full - pausewr flag - failsafe engaged, dropping updates - failsafe disengaged, no longer dropping - is full \(reached quota - POOL_FULL - POOL_BACKFILLFULL sha1: c7471068426370a3754c2250e7729280233d552b ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: c7471068426370a3754c2250e7729280233d552b kclient: syntax: v1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: mgr-snap_schedule-support-subvol-and-group-arguments sha1: 9359c5b40091bccbce966736dc1a3c5bcc76ab0d owner: mchangir priority: 25 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 7945 sha1: c7471068426370a3754c2250e7729280233d552b sleep_before_teardown: 0 suite: fs suite_branch: mgr-snap_schedule-support-subvol-and-group-arguments suite_path: /home/teuthworker/src/github.com_mchangir_ceph_9359c5b40091bccbce966736dc1a3c5bcc76ab0d/qa suite_relpath: qa suite_repo: https://github.com/mchangir/ceph.git suite_sha1: 9359c5b40091bccbce966736dc1a3c5bcc76ab0d syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi016.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs= smithi049.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - kclient: null - cephfs_test_runner: modules: - tasks.cephfs.test_snap_schedules teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 8cdab074dcca9a68965bc5a50e9c30b691949723 timestamp: 2023-10-16_11:33:41 tube: smithi user: mchangir verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-10-16T11:57:58.922 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_mchangir_ceph_9359c5b40091bccbce966736dc1a3c5bcc76ab0d/qa; will attempt to use it 2023-10-16T11:57:58.923 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_mchangir_ceph_9359c5b40091bccbce966736dc1a3c5bcc76ab0d/qa/tasks 2023-10-16T11:57:58.923 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-10-16T11:57:58.925 INFO:teuthology.task.internal:Checking packages... 2023-10-16T11:57:58.948 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash 'c7471068426370a3754c2250e7729280233d552b' 2023-10-16T11:57:58.948 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-10-16T11:57:58.948 INFO:teuthology.packaging:ref: None 2023-10-16T11:57:58.949 INFO:teuthology.packaging:tag: None 2023-10-16T11:57:58.949 INFO:teuthology.packaging:branch: wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments 2023-10-16T11:57:58.949 INFO:teuthology.packaging:sha1: c7471068426370a3754c2250e7729280233d552b 2023-10-16T11:57:58.949 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments 2023-10-16T11:57:59.212 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-6675.gc7471068 2023-10-16T11:57:59.212 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-10-16T11:57:59.219 INFO:teuthology.task.internal:no buildpackages task found 2023-10-16T11:57:59.220 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-10-16T11:57:59.233 INFO:teuthology.task.internal:Saving configuration 2023-10-16T11:57:59.249 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-10-16T11:57:59.255 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-16T11:57:59.281 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi016.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2023-10-16_11:33:41-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi/7428984', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-16 11:51:14.494926', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-10-16T11:57:59.301 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi049.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2023-10-16_11:33:41-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi/7428984', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-16 11:51:14.496837', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-10-16T11:57:59.301 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-10-16T11:57:59.310 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2023-10-16T11:57:59.310 INFO:teuthology.task.internal:roles: ubuntu@smithi049.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2023-10-16T11:57:59.310 INFO:teuthology.run_tasks:Running task console_log... 2023-10-16T11:57:59.378 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7feb5cd109d0>, signals=[15]) 2023-10-16T11:57:59.379 INFO:teuthology.run_tasks:Running task internal.connect... 2023-10-16T11:57:59.384 INFO:teuthology.task.internal:Opening connections... 2023-10-16T11:57:59.385 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi016.front.sepia.ceph.com 2023-10-16T11:57:59.387 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-16T11:57:59.455 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi049.front.sepia.ceph.com 2023-10-16T11:57:59.456 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-16T11:57:59.523 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-10-16T11:57:59.529 DEBUG:teuthology.orchestra.run.smithi016:> uname -m 2023-10-16T11:57:59.551 INFO:teuthology.orchestra.run.smithi016.stdout:x86_64 2023-10-16T11:57:59.552 DEBUG:teuthology.orchestra.run.smithi016:> cat /etc/os-release 2023-10-16T11:57:59.610 INFO:teuthology.orchestra.run.smithi016.stdout:NAME="Red Hat Enterprise Linux" 2023-10-16T11:57:59.611 INFO:teuthology.orchestra.run.smithi016.stdout:VERSION="8.6 (Ootpa)" 2023-10-16T11:57:59.611 INFO:teuthology.orchestra.run.smithi016.stdout:ID="rhel" 2023-10-16T11:57:59.611 INFO:teuthology.orchestra.run.smithi016.stdout:ID_LIKE="fedora" 2023-10-16T11:57:59.611 INFO:teuthology.orchestra.run.smithi016.stdout:VERSION_ID="8.6" 2023-10-16T11:57:59.612 INFO:teuthology.orchestra.run.smithi016.stdout:PLATFORM_ID="platform:el8" 2023-10-16T11:57:59.612 INFO:teuthology.orchestra.run.smithi016.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-10-16T11:57:59.612 INFO:teuthology.orchestra.run.smithi016.stdout:ANSI_COLOR="0;31" 2023-10-16T11:57:59.612 INFO:teuthology.orchestra.run.smithi016.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-10-16T11:57:59.612 INFO:teuthology.orchestra.run.smithi016.stdout:HOME_URL="https://www.redhat.com/" 2023-10-16T11:57:59.613 INFO:teuthology.orchestra.run.smithi016.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-10-16T11:57:59.613 INFO:teuthology.orchestra.run.smithi016.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-16T11:57:59.613 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T11:57:59.613 INFO:teuthology.orchestra.run.smithi016.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-10-16T11:57:59.614 INFO:teuthology.orchestra.run.smithi016.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-10-16T11:57:59.614 INFO:teuthology.orchestra.run.smithi016.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-10-16T11:57:59.614 INFO:teuthology.orchestra.run.smithi016.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-10-16T11:57:59.615 INFO:teuthology.lock.ops:Updating smithi016.front.sepia.ceph.com on lock server 2023-10-16T11:57:59.638 DEBUG:teuthology.orchestra.run.smithi049:> uname -m 2023-10-16T11:57:59.659 INFO:teuthology.orchestra.run.smithi049.stdout:x86_64 2023-10-16T11:57:59.659 DEBUG:teuthology.orchestra.run.smithi049:> cat /etc/os-release 2023-10-16T11:57:59.719 INFO:teuthology.orchestra.run.smithi049.stdout:NAME="Red Hat Enterprise Linux" 2023-10-16T11:57:59.719 INFO:teuthology.orchestra.run.smithi049.stdout:VERSION="8.6 (Ootpa)" 2023-10-16T11:57:59.719 INFO:teuthology.orchestra.run.smithi049.stdout:ID="rhel" 2023-10-16T11:57:59.720 INFO:teuthology.orchestra.run.smithi049.stdout:ID_LIKE="fedora" 2023-10-16T11:57:59.720 INFO:teuthology.orchestra.run.smithi049.stdout:VERSION_ID="8.6" 2023-10-16T11:57:59.720 INFO:teuthology.orchestra.run.smithi049.stdout:PLATFORM_ID="platform:el8" 2023-10-16T11:57:59.720 INFO:teuthology.orchestra.run.smithi049.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-10-16T11:57:59.720 INFO:teuthology.orchestra.run.smithi049.stdout:ANSI_COLOR="0;31" 2023-10-16T11:57:59.720 INFO:teuthology.orchestra.run.smithi049.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-10-16T11:57:59.721 INFO:teuthology.orchestra.run.smithi049.stdout:HOME_URL="https://www.redhat.com/" 2023-10-16T11:57:59.721 INFO:teuthology.orchestra.run.smithi049.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-10-16T11:57:59.721 INFO:teuthology.orchestra.run.smithi049.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-16T11:57:59.721 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-16T11:57:59.721 INFO:teuthology.orchestra.run.smithi049.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-10-16T11:57:59.722 INFO:teuthology.orchestra.run.smithi049.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-10-16T11:57:59.722 INFO:teuthology.orchestra.run.smithi049.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-10-16T11:57:59.722 INFO:teuthology.orchestra.run.smithi049.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-10-16T11:57:59.723 INFO:teuthology.lock.ops:Updating smithi049.front.sepia.ceph.com on lock server 2023-10-16T11:57:59.746 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-10-16T11:57:59.756 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-10-16T11:57:59.761 INFO:teuthology.task.internal:Checking for old test directory... 2023-10-16T11:57:59.761 DEBUG:teuthology.orchestra.run.smithi016:> test '!' -e /home/ubuntu/cephtest 2023-10-16T11:57:59.764 DEBUG:teuthology.orchestra.run.smithi049:> test '!' -e /home/ubuntu/cephtest 2023-10-16T11:57:59.783 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-10-16T11:57:59.788 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-10-16T11:57:59.788 DEBUG:teuthology.orchestra.run.smithi016:> test -z $(ls -A /var/lib/ceph) 2023-10-16T11:57:59.824 DEBUG:teuthology.orchestra.run.smithi049:> test -z $(ls -A /var/lib/ceph) 2023-10-16T11:57:59.845 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-10-16T11:57:59.903 INFO:teuthology.run_tasks:Running task kernel... 2023-10-16T11:57:59.917 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'} 2023-10-16T11:57:59.918 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}, 'client.2': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}, 'client.3': {'sha1': 'distro'}} 2023-10-16T11:57:59.918 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}}, timeout 300 2023-10-16T11:57:59.919 DEBUG:teuthology.orchestra.run.smithi016:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-16T11:57:59.919 DEBUG:teuthology.orchestra.run.smithi049:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-16T11:57:59.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-16T11:57:59.939 DEBUG:teuthology.orchestra.run.smithi016:> uname -r 2023-10-16T11:57:59.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-16T11:57:59.940 DEBUG:teuthology.orchestra.run.smithi049:> uname -r 2023-10-16T11:57:59.999 INFO:teuthology.orchestra.run.smithi016.stdout:4.18.0-372.9.1.el8.x86_64 2023-10-16T11:57:59.999 INFO:teuthology.task.kernel:Running kernel on smithi016: 4.18.0-372.9.1.el8.x86_64 2023-10-16T11:58:00.000 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum install -y kernel 2023-10-16T11:58:00.001 INFO:teuthology.orchestra.run.smithi049.stdout:4.18.0-372.9.1.el8.x86_64 2023-10-16T11:58:00.001 INFO:teuthology.task.kernel:Running kernel on smithi049: 4.18.0-372.9.1.el8.x86_64 2023-10-16T11:58:00.001 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum install -y kernel 2023-10-16T11:58:01.130 INFO:teuthology.orchestra.run.smithi049.stdout:Updating Subscription Management repositories. 2023-10-16T11:58:01.130 INFO:teuthology.orchestra.run.smithi049.stdout:Unable to read consumer identity 2023-10-16T11:58:01.275 INFO:teuthology.orchestra.run.smithi016.stdout:Updating Subscription Management repositories. 2023-10-16T11:58:01.275 INFO:teuthology.orchestra.run.smithi016.stdout:Unable to read consumer identity 2023-10-16T11:58:02.550 INFO:teuthology.orchestra.run.smithi049.stdout:Copr repo for python3-asyncssh owned by ceph 20 kB/s | 3.5 kB 00:00 2023-10-16T11:58:02.739 INFO:teuthology.orchestra.run.smithi016.stdout:Copr repo for python3-asyncssh owned by ceph 18 kB/s | 3.5 kB 00:00 2023-10-16T11:58:04.167 INFO:teuthology.orchestra.run.smithi049.stdout:Extra Packages for Enterprise Linux 10 MB/s | 16 MB 00:01 2023-10-16T11:58:04.464 INFO:teuthology.orchestra.run.smithi016.stdout:Extra Packages for Enterprise Linux 9.3 MB/s | 16 MB 00:01 2023-10-16T11:58:07.992 INFO:teuthology.orchestra.run.smithi049.stdout:lab-extras 348 kB/s | 24 kB 00:00 2023-10-16T11:58:08.156 INFO:teuthology.orchestra.run.smithi016.stdout:lab-extras 434 kB/s | 24 kB 00:00 2023-10-16T11:58:08.744 INFO:teuthology.orchestra.run.smithi049.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2023-10-16T11:58:08.801 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-10-16T11:58:08.802 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2023-10-16T11:58:08.802 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-10-16T11:58:08.884 INFO:teuthology.orchestra.run.smithi016.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2023-10-16T11:58:08.925 DEBUG:teuthology.orchestra.run.smithi049:> echo no | sudo yum reinstall kernel || true 2023-10-16T11:58:08.938 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2023-10-16T11:58:08.939 INFO:teuthology.orchestra.run.smithi016.stdout:Nothing to do. 2023-10-16T11:58:08.940 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2023-10-16T11:58:09.054 DEBUG:teuthology.orchestra.run.smithi016:> echo no | sudo yum reinstall kernel || true 2023-10-16T11:58:09.277 INFO:teuthology.orchestra.run.smithi049.stdout:Updating Subscription Management repositories. 2023-10-16T11:58:09.278 INFO:teuthology.orchestra.run.smithi049.stdout:Unable to read consumer identity 2023-10-16T11:58:09.403 INFO:teuthology.orchestra.run.smithi016.stdout:Updating Subscription Management repositories. 2023-10-16T11:58:09.403 INFO:teuthology.orchestra.run.smithi016.stdout:Unable to read consumer identity 2023-10-16T11:58:09.550 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:00:02 ago on Mon 16 Oct 2023 11:58:07 AM UTC. 2023-10-16T11:58:09.595 INFO:teuthology.orchestra.run.smithi049.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-10-16T11:58:09.636 INFO:teuthology.orchestra.run.smithi049.stderr:Error: No packages marked for reinstall. 2023-10-16T11:58:09.680 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:00:01 ago on Mon 16 Oct 2023 11:58:08 AM UTC. 2023-10-16T11:58:09.683 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum reinstall -y kernel || true 2023-10-16T11:58:09.736 INFO:teuthology.orchestra.run.smithi016.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-10-16T11:58:09.774 INFO:teuthology.orchestra.run.smithi016.stderr:Error: No packages marked for reinstall. 2023-10-16T11:58:09.821 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum reinstall -y kernel || true 2023-10-16T11:58:10.031 INFO:teuthology.orchestra.run.smithi049.stdout:Updating Subscription Management repositories. 2023-10-16T11:58:10.032 INFO:teuthology.orchestra.run.smithi049.stdout:Unable to read consumer identity 2023-10-16T11:58:10.170 INFO:teuthology.orchestra.run.smithi016.stdout:Updating Subscription Management repositories. 2023-10-16T11:58:10.171 INFO:teuthology.orchestra.run.smithi016.stdout:Unable to read consumer identity 2023-10-16T11:58:10.302 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:00:03 ago on Mon 16 Oct 2023 11:58:07 AM UTC. 2023-10-16T11:58:10.344 INFO:teuthology.orchestra.run.smithi049.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-10-16T11:58:10.368 INFO:teuthology.orchestra.run.smithi049.stderr:Error: No packages marked for reinstall. 2023-10-16T11:58:10.414 DEBUG:teuthology.orchestra.run.smithi049:> rpm -q kernel | sort -rV | head -n 1 2023-10-16T11:58:10.439 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:00:02 ago on Mon 16 Oct 2023 11:58:08 AM UTC. 2023-10-16T11:58:10.483 INFO:teuthology.orchestra.run.smithi016.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-10-16T11:58:10.496 INFO:teuthology.orchestra.run.smithi049.stdout:kernel-4.18.0-372.9.1.el8.x86_64 2023-10-16T11:58:10.497 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-372.9.1.el8.x86_64 2023-10-16T11:58:10.497 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2023-10-16T11:58:10.498 DEBUG:teuthology.parallel:result is None 2023-10-16T11:58:10.512 INFO:teuthology.orchestra.run.smithi016.stderr:Error: No packages marked for reinstall. 2023-10-16T11:58:10.559 DEBUG:teuthology.orchestra.run.smithi016:> rpm -q kernel | sort -rV | head -n 1 2023-10-16T11:58:10.641 INFO:teuthology.orchestra.run.smithi016.stdout:kernel-4.18.0-372.9.1.el8.x86_64 2023-10-16T11:58:10.642 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-372.9.1.el8.x86_64 2023-10-16T11:58:10.642 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2023-10-16T11:58:10.642 DEBUG:teuthology.parallel:result is None 2023-10-16T11:58:10.643 INFO:teuthology.run_tasks:Running task internal.base... 2023-10-16T11:58:10.650 INFO:teuthology.task.internal:Creating test directory... 2023-10-16T11:58:10.651 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-16T11:58:10.653 DEBUG:teuthology.orchestra.run.smithi049:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-16T11:58:10.671 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-10-16T11:58:10.678 INFO:teuthology.run_tasks:Running task internal.archive... 2023-10-16T11:58:10.685 INFO:teuthology.task.internal:Creating archive directory... 2023-10-16T11:58:10.686 DEBUG:teuthology.orchestra.run.smithi016:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-16T11:58:10.711 DEBUG:teuthology.orchestra.run.smithi049:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-16T11:58:10.733 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-10-16T11:58:10.740 INFO:teuthology.task.internal:Enabling coredump saving... 2023-10-16T11:58:10.740 DEBUG:teuthology.orchestra.run.smithi016:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-16T11:58:10.768 DEBUG:teuthology.orchestra.run.smithi049:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-16T11:58:10.804 INFO:teuthology.orchestra.run.smithi016.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-16T11:58:10.810 INFO:teuthology.orchestra.run.smithi049.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-16T11:58:10.821 INFO:teuthology.orchestra.run.smithi016.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-16T11:58:10.822 INFO:teuthology.orchestra.run.smithi049.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-16T11:58:10.823 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-10-16T11:58:10.830 INFO:teuthology.task.internal:Configuring sudo... 2023-10-16T11:58:10.831 DEBUG:teuthology.orchestra.run.smithi016:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-16T11:58:10.867 DEBUG:teuthology.orchestra.run.smithi049:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-16T11:58:10.895 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-10-16T11:58:10.904 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-10-16T11:58:10.904 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-16T11:58:10.935 DEBUG:teuthology.orchestra.run.smithi049:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-16T11:58:10.952 DEBUG:teuthology.orchestra.run.smithi016:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-16T11:58:11.031 DEBUG:teuthology.orchestra.run.smithi016:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-16T11:58:11.089 DEBUG:teuthology.orchestra.run.smithi016:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-16T11:58:11.169 DEBUG:teuthology.orchestra.run.smithi016:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-16T11:58:11.215 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T11:58:11.216 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-16T11:58:11.282 DEBUG:teuthology.orchestra.run.smithi049:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-16T11:58:11.328 DEBUG:teuthology.orchestra.run.smithi049:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-16T11:58:11.379 DEBUG:teuthology.orchestra.run.smithi049:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-16T11:58:11.463 DEBUG:teuthology.orchestra.run.smithi049:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-16T11:58:11.505 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T11:58:11.505 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-16T11:58:11.573 DEBUG:teuthology.orchestra.run.smithi016:> sudo service rsyslog restart 2023-10-16T11:58:11.576 DEBUG:teuthology.orchestra.run.smithi049:> sudo service rsyslog restart 2023-10-16T11:58:11.645 INFO:teuthology.orchestra.run.smithi016.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-16T11:58:11.678 INFO:teuthology.orchestra.run.smithi049.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-16T11:58:12.057 INFO:teuthology.run_tasks:Running task internal.timer... 2023-10-16T11:58:12.066 INFO:teuthology.task.internal:Starting timer... 2023-10-16T11:58:12.066 INFO:teuthology.run_tasks:Running task pcp... 2023-10-16T11:58:12.075 INFO:teuthology.run_tasks:Running task selinux... 2023-10-16T11:58:12.084 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-10-16T11:58:12.084 DEBUG:teuthology.orchestra.run.smithi016:> sudo service auditd rotate 2023-10-16T11:58:12.146 INFO:teuthology.orchestra.run.smithi016.stdout:Rotating logs: 2023-10-16T11:58:12.148 DEBUG:teuthology.orchestra.run.smithi049:> sudo service auditd rotate 2023-10-16T11:58:12.203 INFO:teuthology.orchestra.run.smithi049.stdout:Rotating logs: 2023-10-16T11:58:12.205 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-10-16T11:58:12.205 DEBUG:teuthology.orchestra.run.smithi016:> /usr/sbin/getenforce 2023-10-16T11:58:12.230 INFO:teuthology.orchestra.run.smithi016.stdout:Permissive 2023-10-16T11:58:12.230 DEBUG:teuthology.orchestra.run.smithi049:> /usr/sbin/getenforce 2023-10-16T11:58:12.258 INFO:teuthology.orchestra.run.smithi049.stdout:Permissive 2023-10-16T11:58:12.258 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi016.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi049.front.sepia.ceph.com': 'permissive'} 2023-10-16T11:58:12.259 DEBUG:teuthology.orchestra.run.smithi016:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-16T11:58:12.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-16T11:58:12.299 DEBUG:teuthology.orchestra.run.smithi049:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-16T11:58:12.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-16T11:58:12.330 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-10-16T11:58:12.330 DEBUG:teuthology.orchestra.run.smithi016:> sudo /usr/sbin/setenforce permissive 2023-10-16T11:58:12.366 DEBUG:teuthology.orchestra.run.smithi049:> sudo /usr/sbin/setenforce permissive 2023-10-16T11:58:12.398 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-10-16T11:58:12.407 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-10-16T11:58:12.412 INFO:teuthology.repo_utils:Fetching main from origin 2023-10-16T11:58:12.527 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-10-16T11:58:12.542 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-10-16T11:58:12.543 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi016.front.sepia.ceph.com,smithi049.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-10-16T12:07:21.625 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi016.front.sepia.ceph.com'), Remote(name='ubuntu@smithi049.front.sepia.ceph.com')] 2023-10-16T12:07:21.627 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-10-16T12:07:21.627 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-16T12:07:21.697 DEBUG:teuthology.orchestra.run.smithi016:> true 2023-10-16T12:07:21.764 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-10-16T12:07:21.764 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi049.front.sepia.ceph.com' 2023-10-16T12:07:21.765 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-16T12:07:21.833 DEBUG:teuthology.orchestra.run.smithi049:> true 2023-10-16T12:07:21.899 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi049.front.sepia.ceph.com' 2023-10-16T12:07:21.900 INFO:teuthology.run_tasks:Running task clock... 2023-10-16T12:07:21.914 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-10-16T12:07:21.915 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-16T12:07:21.915 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-16T12:07:21.918 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-16T12:07:21.918 DEBUG:teuthology.orchestra.run.smithi049:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-16T12:07:21.951 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-10-16T12:07:21.967 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-10-16T12:07:21.972 INFO:teuthology.orchestra.run.smithi049.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-10-16T12:07:21.987 INFO:teuthology.orchestra.run.smithi049.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-10-16T12:07:22.027 INFO:teuthology.orchestra.run.smithi016.stderr:sudo: ntpd: command not found 2023-10-16T12:07:22.040 INFO:teuthology.orchestra.run.smithi016.stdout:506 Cannot talk to daemon 2023-10-16T12:07:22.055 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-10-16T12:07:22.064 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: ntpd: command not found 2023-10-16T12:07:22.070 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-10-16T12:07:22.076 INFO:teuthology.orchestra.run.smithi049.stdout:506 Cannot talk to daemon 2023-10-16T12:07:22.091 INFO:teuthology.orchestra.run.smithi049.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-10-16T12:07:22.105 INFO:teuthology.orchestra.run.smithi049.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-10-16T12:07:22.143 INFO:teuthology.orchestra.run.smithi016.stderr:bash: ntpq: command not found 2023-10-16T12:07:22.146 INFO:teuthology.orchestra.run.smithi016.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-16T12:07:22.147 INFO:teuthology.orchestra.run.smithi016.stdout:=============================================================================== 2023-10-16T12:07:22.147 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-16T12:07:22.147 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-16T12:07:22.147 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-16T12:07:22.147 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-16T12:07:22.175 INFO:teuthology.orchestra.run.smithi049.stderr:bash: ntpq: command not found 2023-10-16T12:07:22.179 INFO:teuthology.orchestra.run.smithi049.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-16T12:07:22.179 INFO:teuthology.orchestra.run.smithi049.stdout:=============================================================================== 2023-10-16T12:07:22.179 INFO:teuthology.orchestra.run.smithi049.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-16T12:07:22.179 INFO:teuthology.orchestra.run.smithi049.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-16T12:07:22.180 INFO:teuthology.orchestra.run.smithi049.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-16T12:07:22.180 INFO:teuthology.orchestra.run.smithi049.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-16T12:07:22.181 INFO:teuthology.run_tasks:Running task install... 2023-10-16T12:07:22.189 DEBUG:teuthology.task.install:project ceph 2023-10-16T12:07:22.189 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'c7471068426370a3754c2250e7729280233d552b'}} 2023-10-16T12:07:22.189 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'c7471068426370a3754c2250e7729280233d552b'} 2023-10-16T12:07:22.190 INFO:teuthology.task.install:Using flavor: default 2023-10-16T12:07:22.195 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-10-16T12:07:22.195 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-10-16T12:07:22.195 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'c7471068426370a3754c2250e7729280233d552b', 'tag': None, 'wait_for_package': False} 2023-10-16T12:07:22.196 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=c7471068426370a3754c2250e7729280233d552b 2023-10-16T12:07:22.198 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'c7471068426370a3754c2250e7729280233d552b', 'tag': None, 'wait_for_package': False} 2023-10-16T12:07:22.198 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=c7471068426370a3754c2250e7729280233d552b 2023-10-16T12:07:22.365 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments/c7471068426370a3754c2250e7729280233d552b/centos/8/flavors/default/ 2023-10-16T12:07:22.365 INFO:teuthology.task.install.rpm:Package version is 18.0.0-6675.gc7471068 2023-10-16T12:07:22.440 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments/c7471068426370a3754c2250e7729280233d552b/centos/8/flavors/default/ 2023-10-16T12:07:22.441 INFO:teuthology.task.install.rpm:Package version is 18.0.0-6675.gc7471068 2023-10-16T12:07:22.550 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments/c7471068426370a3754c2250e7729280233d552b/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments/c7471068426370a3754c2250e7729280233d552b/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments/c7471068426370a3754c2250e7729280233d552b/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-10-16T12:07:22.551 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T12:07:22.551 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-10-16T12:07:22.553 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments/c7471068426370a3754c2250e7729280233d552b/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments/c7471068426370a3754c2250e7729280233d552b/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments/c7471068426370a3754c2250e7729280233d552b/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-10-16T12:07:22.554 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:07:22.554 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-10-16T12:07:22.583 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-10-16T12:07:22.583 DEBUG:teuthology.orchestra.run.smithi049:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/c7471068426370a3754c2250e7729280233d552b/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-10-16T12:07:22.584 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-10-16T12:07:22.584 DEBUG:teuthology.orchestra.run.smithi016:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/c7471068426370a3754c2250e7729280233d552b/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-10-16T12:07:22.651 DEBUG:teuthology.orchestra.run.smithi049:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-10-16T12:07:22.654 DEBUG:teuthology.orchestra.run.smithi016:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-10-16T12:07:22.729 DEBUG:teuthology.orchestra.run.smithi049:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-10-16T12:07:22.733 DEBUG:teuthology.orchestra.run.smithi016:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-10-16T12:07:22.795 INFO:teuthology.orchestra.run.smithi049.stdout:check_obsoletes = 1 2023-10-16T12:07:22.796 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum clean all 2023-10-16T12:07:22.800 INFO:teuthology.orchestra.run.smithi016.stdout:check_obsoletes = 1 2023-10-16T12:07:22.802 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum clean all 2023-10-16T12:07:23.185 INFO:teuthology.orchestra.run.smithi049.stdout:Updating Subscription Management repositories. 2023-10-16T12:07:23.192 INFO:teuthology.orchestra.run.smithi016.stdout:Updating Subscription Management repositories. 2023-10-16T12:07:23.565 INFO:teuthology.orchestra.run.smithi016.stdout:44 files removed 2023-10-16T12:07:23.580 INFO:teuthology.orchestra.run.smithi049.stdout:44 files removed 2023-10-16T12:07:23.613 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-10-16T12:07:23.628 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-10-16T12:07:23.962 INFO:teuthology.orchestra.run.smithi016.stdout:Updating Subscription Management repositories. 2023-10-16T12:07:23.976 INFO:teuthology.orchestra.run.smithi049.stdout:Updating Subscription Management repositories. 2023-10-16T12:07:24.659 INFO:teuthology.orchestra.run.smithi016.stdout:ceph packages for x86_64 256 kB/s | 81 kB 00:00 2023-10-16T12:07:24.739 INFO:teuthology.orchestra.run.smithi049.stdout:ceph packages for x86_64 209 kB/s | 81 kB 00:00 2023-10-16T12:07:24.985 INFO:teuthology.orchestra.run.smithi016.stdout:ceph noarch packages 32 kB/s | 9.4 kB 00:00 2023-10-16T12:07:25.115 INFO:teuthology.orchestra.run.smithi049.stdout:ceph noarch packages 28 kB/s | 9.4 kB 00:00 2023-10-16T12:07:25.369 INFO:teuthology.orchestra.run.smithi016.stdout:ceph source packages 4.2 kB/s | 1.5 kB 00:00 2023-10-16T12:07:25.528 INFO:teuthology.orchestra.run.smithi016.stdout:Copr repo for python3-asyncssh owned by ceph 25 kB/s | 3.5 kB 00:00 2023-10-16T12:07:25.532 INFO:teuthology.orchestra.run.smithi049.stdout:ceph source packages 3.8 kB/s | 1.5 kB 00:00 2023-10-16T12:07:25.683 INFO:teuthology.orchestra.run.smithi049.stdout:Copr repo for python3-asyncssh owned by ceph 27 kB/s | 3.5 kB 00:00 2023-10-16T12:07:26.733 INFO:teuthology.orchestra.run.smithi049.stdout:Extra Packages for Enterprise Linux 15 MB/s | 16 MB 00:01 2023-10-16T12:07:26.931 INFO:teuthology.orchestra.run.smithi016.stdout:Extra Packages for Enterprise Linux 12 MB/s | 16 MB 00:01 2023-10-16T12:07:30.759 INFO:teuthology.orchestra.run.smithi049.stdout:lab-extras 64 kB/s | 24 kB 00:00 2023-10-16T12:07:30.762 INFO:teuthology.orchestra.run.smithi016.stdout:lab-extras 180 kB/s | 24 kB 00:00 2023-10-16T12:07:31.644 INFO:teuthology.orchestra.run.smithi049.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 59 MB/s | 47 MB 00:00 2023-10-16T12:07:31.714 INFO:teuthology.orchestra.run.smithi016.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 61 MB/s | 53 MB 00:00 2023-10-16T12:07:39.945 INFO:teuthology.orchestra.run.smithi049.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 62 MB/s | 53 MB 00:00 2023-10-16T12:07:40.382 INFO:teuthology.orchestra.run.smithi016.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 58 MB/s | 47 MB 00:00 2023-10-16T12:07:51.844 INFO:teuthology.orchestra.run.smithi049.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-10-16T12:07:51.845 INFO:teuthology.orchestra.run.smithi049.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-10-16T12:07:51.861 INFO:teuthology.orchestra.run.smithi049.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-10-16T12:07:51.862 INFO:teuthology.orchestra.run.smithi049.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-10-16T12:07:51.863 INFO:teuthology.orchestra.run.smithi049.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-10-16T12:07:51.864 INFO:teuthology.orchestra.run.smithi049.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-10-16T12:07:51.867 INFO:teuthology.orchestra.run.smithi049.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-10-16T12:07:51.870 INFO:teuthology.orchestra.run.smithi049.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-10-16T12:07:51.872 INFO:teuthology.orchestra.run.smithi049.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-10-16T12:07:52.016 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-10-16T12:07:52.029 INFO:teuthology.orchestra.run.smithi049.stdout:=========================================================================================================================== 2023-10-16T12:07:52.029 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2023-10-16T12:07:52.030 INFO:teuthology.orchestra.run.smithi049.stdout:=========================================================================================================================== 2023-10-16T12:07:52.030 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2023-10-16T12:07:52.030 INFO:teuthology.orchestra.run.smithi049.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-10-16T12:07:52.030 INFO:teuthology.orchestra.run.smithi049.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-10-16T12:07:52.031 INFO:teuthology.orchestra.run.smithi049.stdout: ceph x86_64 2:18.0.0-6675.gc7471068.el8 ceph 6.4 k 2023-10-16T12:07:52.031 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-base x86_64 2:18.0.0-6675.gc7471068.el8 ceph 5.4 M 2023-10-16T12:07:52.031 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-fuse x86_64 2:18.0.0-6675.gc7471068.el8 ceph 919 k 2023-10-16T12:07:52.031 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-6675.gc7471068.el8 ceph 167 k 2023-10-16T12:07:52.032 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr x86_64 2:18.0.0-6675.gc7471068.el8 ceph 1.6 M 2023-10-16T12:07:52.032 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-cephadm noarch 2:18.0.0-6675.gc7471068.el8 ceph-noarch 132 k 2023-10-16T12:07:52.032 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-dashboard noarch 2:18.0.0-6675.gc7471068.el8 ceph-noarch 1.8 M 2023-10-16T12:07:52.032 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-6675.gc7471068.el8 ceph-noarch 7.4 M 2023-10-16T12:07:52.032 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-rook noarch 2:18.0.0-6675.gc7471068.el8 ceph-noarch 47 k 2023-10-16T12:07:52.033 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-radosgw x86_64 2:18.0.0-6675.gc7471068.el8 ceph 14 M 2023-10-16T12:07:52.033 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-test x86_64 2:18.0.0-6675.gc7471068.el8 ceph 49 M 2023-10-16T12:07:52.033 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-volume noarch 2:18.0.0-6675.gc7471068.el8 ceph-noarch 261 k 2023-10-16T12:07:52.033 INFO:teuthology.orchestra.run.smithi049.stdout: cephadm noarch 2:18.0.0-6675.gc7471068.el8 ceph-noarch 192 k 2023-10-16T12:07:52.034 INFO:teuthology.orchestra.run.smithi049.stdout: cephfs-mirror x86_64 2:18.0.0-6675.gc7471068.el8 ceph 261 k 2023-10-16T12:07:52.034 INFO:teuthology.orchestra.run.smithi049.stdout: cephfs-top noarch 2:18.0.0-6675.gc7471068.el8 ceph-noarch 18 k 2023-10-16T12:07:52.034 INFO:teuthology.orchestra.run.smithi049.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-10-16T12:07:52.034 INFO:teuthology.orchestra.run.smithi049.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-10-16T12:07:52.034 INFO:teuthology.orchestra.run.smithi049.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-10-16T12:07:52.035 INFO:teuthology.orchestra.run.smithi049.stdout: libcephfs-devel x86_64 2:18.0.0-6675.gc7471068.el8 ceph 31 k 2023-10-16T12:07:52.035 INFO:teuthology.orchestra.run.smithi049.stdout: libcephfs2 x86_64 2:18.0.0-6675.gc7471068.el8 ceph 839 k 2023-10-16T12:07:52.035 INFO:teuthology.orchestra.run.smithi049.stdout: librados-devel x86_64 2:18.0.0-6675.gc7471068.el8 ceph 139 k 2023-10-16T12:07:52.035 INFO:teuthology.orchestra.run.smithi049.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-10-16T12:07:52.036 INFO:teuthology.orchestra.run.smithi049.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-10-16T12:07:52.036 INFO:teuthology.orchestra.run.smithi049.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-10-16T12:07:52.036 INFO:teuthology.orchestra.run.smithi049.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-10-16T12:07:52.036 INFO:teuthology.orchestra.run.smithi049.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-10-16T12:07:52.036 INFO:teuthology.orchestra.run.smithi049.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-10-16T12:07:52.037 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cephfs x86_64 2:18.0.0-6675.gc7471068.el8 ceph 197 k 2023-10-16T12:07:52.037 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rados x86_64 2:18.0.0-6675.gc7471068.el8 ceph 371 k 2023-10-16T12:07:52.037 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rbd x86_64 2:18.0.0-6675.gc7471068.el8 ceph 364 k 2023-10-16T12:07:52.037 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rgw x86_64 2:18.0.0-6675.gc7471068.el8 ceph 107 k 2023-10-16T12:07:52.037 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-fuse x86_64 2:18.0.0-6675.gc7471068.el8 ceph 94 k 2023-10-16T12:07:52.038 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-mirror x86_64 2:18.0.0-6675.gc7471068.el8 ceph 3.6 M 2023-10-16T12:07:52.038 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-nbd x86_64 2:18.0.0-6675.gc7471068.el8 ceph 180 k 2023-10-16T12:07:52.038 INFO:teuthology.orchestra.run.smithi049.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-10-16T12:07:52.038 INFO:teuthology.orchestra.run.smithi049.stdout:Upgrading: 2023-10-16T12:07:52.038 INFO:teuthology.orchestra.run.smithi049.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-10-16T12:07:52.039 INFO:teuthology.orchestra.run.smithi049.stdout: librados2 x86_64 2:18.0.0-6675.gc7471068.el8 ceph 3.7 M 2023-10-16T12:07:52.039 INFO:teuthology.orchestra.run.smithi049.stdout: librbd1 x86_64 2:18.0.0-6675.gc7471068.el8 ceph 3.7 M 2023-10-16T12:07:52.039 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2023-10-16T12:07:52.039 INFO:teuthology.orchestra.run.smithi049.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-10-16T12:07:52.039 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-common x86_64 2:18.0.0-6675.gc7471068.el8 ceph 23 M 2023-10-16T12:07:52.040 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-grafana-dashboards noarch 2:18.0.0-6675.gc7471068.el8 ceph-noarch 23 k 2023-10-16T12:07:52.040 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mds x86_64 2:18.0.0-6675.gc7471068.el8 ceph 2.3 M 2023-10-16T12:07:52.040 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-modules-core noarch 2:18.0.0-6675.gc7471068.el8 ceph-noarch 243 k 2023-10-16T12:07:52.040 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mon x86_64 2:18.0.0-6675.gc7471068.el8 ceph 4.6 M 2023-10-16T12:07:52.041 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-osd x86_64 2:18.0.0-6675.gc7471068.el8 ceph 18 M 2023-10-16T12:07:52.041 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-prometheus-alerts noarch 2:18.0.0-6675.gc7471068.el8 ceph-noarch 15 k 2023-10-16T12:07:52.041 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-selinux x86_64 2:18.0.0-6675.gc7471068.el8 ceph 24 k 2023-10-16T12:07:52.041 INFO:teuthology.orchestra.run.smithi049.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-10-16T12:07:52.041 INFO:teuthology.orchestra.run.smithi049.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-10-16T12:07:52.041 INFO:teuthology.orchestra.run.smithi049.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-10-16T12:07:52.042 INFO:teuthology.orchestra.run.smithi049.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-10-16T12:07:52.042 INFO:teuthology.orchestra.run.smithi049.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-10-16T12:07:52.042 INFO:teuthology.orchestra.run.smithi049.stdout: libcephsqlite x86_64 2:18.0.0-6675.gc7471068.el8 ceph 184 k 2023-10-16T12:07:52.042 INFO:teuthology.orchestra.run.smithi049.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-10-16T12:07:52.043 INFO:teuthology.orchestra.run.smithi049.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-10-16T12:07:52.043 INFO:teuthology.orchestra.run.smithi049.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-10-16T12:07:52.043 INFO:teuthology.orchestra.run.smithi049.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-10-16T12:07:52.043 INFO:teuthology.orchestra.run.smithi049.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-10-16T12:07:52.043 INFO:teuthology.orchestra.run.smithi049.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-10-16T12:07:52.044 INFO:teuthology.orchestra.run.smithi049.stdout: libradosstriper1 x86_64 2:18.0.0-6675.gc7471068.el8 ceph 518 k 2023-10-16T12:07:52.044 INFO:teuthology.orchestra.run.smithi049.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-10-16T12:07:52.044 INFO:teuthology.orchestra.run.smithi049.stdout: librgw2 x86_64 2:18.0.0-6675.gc7471068.el8 ceph 6.9 M 2023-10-16T12:07:52.044 INFO:teuthology.orchestra.run.smithi049.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-10-16T12:07:52.044 INFO:teuthology.orchestra.run.smithi049.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-10-16T12:07:52.045 INFO:teuthology.orchestra.run.smithi049.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-10-16T12:07:52.045 INFO:teuthology.orchestra.run.smithi049.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-10-16T12:07:52.045 INFO:teuthology.orchestra.run.smithi049.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-10-16T12:07:52.045 INFO:teuthology.orchestra.run.smithi049.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-10-16T12:07:52.046 INFO:teuthology.orchestra.run.smithi049.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-10-16T12:07:52.046 INFO:teuthology.orchestra.run.smithi049.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-10-16T12:07:52.046 INFO:teuthology.orchestra.run.smithi049.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-10-16T12:07:52.046 INFO:teuthology.orchestra.run.smithi049.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-10-16T12:07:52.046 INFO:teuthology.orchestra.run.smithi049.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-10-16T12:07:52.047 INFO:teuthology.orchestra.run.smithi049.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-10-16T12:07:52.047 INFO:teuthology.orchestra.run.smithi049.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-10-16T12:07:52.047 INFO:teuthology.orchestra.run.smithi049.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-10-16T12:07:52.047 INFO:teuthology.orchestra.run.smithi049.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-10-16T12:07:52.047 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-10-16T12:07:52.048 INFO:teuthology.orchestra.run.smithi049.stdout: python3-ceph-argparse x86_64 2:18.0.0-6675.gc7471068.el8 ceph 45 k 2023-10-16T12:07:52.048 INFO:teuthology.orchestra.run.smithi049.stdout: python3-ceph-common x86_64 2:18.0.0-6675.gc7471068.el8 ceph 126 k 2023-10-16T12:07:52.048 INFO:teuthology.orchestra.run.smithi049.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-10-16T12:07:52.048 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-10-16T12:07:52.049 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-10-16T12:07:52.049 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-10-16T12:07:52.049 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-10-16T12:07:52.049 INFO:teuthology.orchestra.run.smithi049.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-10-16T12:07:52.050 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-10-16T12:07:52.050 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-10-16T12:07:52.050 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-10-16T12:07:52.050 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-10-16T12:07:52.050 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-10-16T12:07:52.051 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-10-16T12:07:52.051 INFO:teuthology.orchestra.run.smithi049.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-10-16T12:07:52.051 INFO:teuthology.orchestra.run.smithi049.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-10-16T12:07:52.051 INFO:teuthology.orchestra.run.smithi049.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-10-16T12:07:52.051 INFO:teuthology.orchestra.run.smithi049.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-10-16T12:07:52.052 INFO:teuthology.orchestra.run.smithi049.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-10-16T12:07:52.052 INFO:teuthology.orchestra.run.smithi049.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-10-16T12:07:52.052 INFO:teuthology.orchestra.run.smithi049.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-10-16T12:07:52.052 INFO:teuthology.orchestra.run.smithi049.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-10-16T12:07:52.053 INFO:teuthology.orchestra.run.smithi049.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-10-16T12:07:52.053 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-10-16T12:07:52.053 INFO:teuthology.orchestra.run.smithi049.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-10-16T12:07:52.053 INFO:teuthology.orchestra.run.smithi049.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-10-16T12:07:52.053 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-10-16T12:07:52.054 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-10-16T12:07:52.054 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-10-16T12:07:52.054 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-10-16T12:07:52.054 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-10-16T12:07:52.055 INFO:teuthology.orchestra.run.smithi049.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-10-16T12:07:52.055 INFO:teuthology.orchestra.run.smithi049.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-10-16T12:07:52.055 INFO:teuthology.orchestra.run.smithi049.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-10-16T12:07:52.055 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-10-16T12:07:52.055 INFO:teuthology.orchestra.run.smithi049.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-10-16T12:07:52.056 INFO:teuthology.orchestra.run.smithi049.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-10-16T12:07:52.056 INFO:teuthology.orchestra.run.smithi049.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-10-16T12:07:52.056 INFO:teuthology.orchestra.run.smithi049.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-10-16T12:07:52.056 INFO:teuthology.orchestra.run.smithi049.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-10-16T12:07:52.057 INFO:teuthology.orchestra.run.smithi049.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-10-16T12:07:52.057 INFO:teuthology.orchestra.run.smithi049.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-10-16T12:07:52.057 INFO:teuthology.orchestra.run.smithi049.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-10-16T12:07:52.057 INFO:teuthology.orchestra.run.smithi049.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-10-16T12:07:52.057 INFO:teuthology.orchestra.run.smithi049.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-10-16T12:07:52.058 INFO:teuthology.orchestra.run.smithi049.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-10-16T12:07:52.058 INFO:teuthology.orchestra.run.smithi049.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-10-16T12:07:52.058 INFO:teuthology.orchestra.run.smithi049.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-10-16T12:07:52.058 INFO:teuthology.orchestra.run.smithi049.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-10-16T12:07:52.059 INFO:teuthology.orchestra.run.smithi049.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-10-16T12:07:52.059 INFO:teuthology.orchestra.run.smithi049.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-10-16T12:07:52.059 INFO:teuthology.orchestra.run.smithi049.stdout:Enabling module streams: 2023-10-16T12:07:52.059 INFO:teuthology.orchestra.run.smithi049.stdout: postgresql 10 2023-10-16T12:07:52.059 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-16T12:07:52.060 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2023-10-16T12:07:52.060 INFO:teuthology.orchestra.run.smithi049.stdout:=========================================================================================================================== 2023-10-16T12:07:52.060 INFO:teuthology.orchestra.run.smithi049.stdout:Install 128 Packages 2023-10-16T12:07:52.060 INFO:teuthology.orchestra.run.smithi049.stdout:Upgrade 3 Packages 2023-10-16T12:07:52.061 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-16T12:07:52.061 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 209 M 2023-10-16T12:07:52.062 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2023-10-16T12:07:52.108 INFO:teuthology.orchestra.run.smithi016.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-10-16T12:07:52.109 INFO:teuthology.orchestra.run.smithi016.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-10-16T12:07:52.125 INFO:teuthology.orchestra.run.smithi016.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-10-16T12:07:52.126 INFO:teuthology.orchestra.run.smithi016.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-10-16T12:07:52.128 INFO:teuthology.orchestra.run.smithi016.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-10-16T12:07:52.129 INFO:teuthology.orchestra.run.smithi016.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-10-16T12:07:52.131 INFO:teuthology.orchestra.run.smithi016.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-10-16T12:07:52.135 INFO:teuthology.orchestra.run.smithi016.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-10-16T12:07:52.136 INFO:teuthology.orchestra.run.smithi016.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-10-16T12:07:52.284 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2023-10-16T12:07:52.298 INFO:teuthology.orchestra.run.smithi016.stdout:=========================================================================================================================== 2023-10-16T12:07:52.298 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2023-10-16T12:07:52.298 INFO:teuthology.orchestra.run.smithi016.stdout:=========================================================================================================================== 2023-10-16T12:07:52.298 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2023-10-16T12:07:52.299 INFO:teuthology.orchestra.run.smithi016.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-10-16T12:07:52.299 INFO:teuthology.orchestra.run.smithi016.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-10-16T12:07:52.299 INFO:teuthology.orchestra.run.smithi016.stdout: ceph x86_64 2:18.0.0-6675.gc7471068.el8 ceph 6.4 k 2023-10-16T12:07:52.299 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-base x86_64 2:18.0.0-6675.gc7471068.el8 ceph 5.4 M 2023-10-16T12:07:52.299 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-fuse x86_64 2:18.0.0-6675.gc7471068.el8 ceph 919 k 2023-10-16T12:07:52.300 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-6675.gc7471068.el8 ceph 167 k 2023-10-16T12:07:52.300 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr x86_64 2:18.0.0-6675.gc7471068.el8 ceph 1.6 M 2023-10-16T12:07:52.300 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-cephadm noarch 2:18.0.0-6675.gc7471068.el8 ceph-noarch 132 k 2023-10-16T12:07:52.300 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-dashboard noarch 2:18.0.0-6675.gc7471068.el8 ceph-noarch 1.8 M 2023-10-16T12:07:52.300 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-6675.gc7471068.el8 ceph-noarch 7.4 M 2023-10-16T12:07:52.301 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-rook noarch 2:18.0.0-6675.gc7471068.el8 ceph-noarch 47 k 2023-10-16T12:07:52.301 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-radosgw x86_64 2:18.0.0-6675.gc7471068.el8 ceph 14 M 2023-10-16T12:07:52.301 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-test x86_64 2:18.0.0-6675.gc7471068.el8 ceph 49 M 2023-10-16T12:07:52.301 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-volume noarch 2:18.0.0-6675.gc7471068.el8 ceph-noarch 261 k 2023-10-16T12:07:52.301 INFO:teuthology.orchestra.run.smithi016.stdout: cephadm noarch 2:18.0.0-6675.gc7471068.el8 ceph-noarch 192 k 2023-10-16T12:07:52.302 INFO:teuthology.orchestra.run.smithi016.stdout: cephfs-mirror x86_64 2:18.0.0-6675.gc7471068.el8 ceph 261 k 2023-10-16T12:07:52.302 INFO:teuthology.orchestra.run.smithi016.stdout: cephfs-top noarch 2:18.0.0-6675.gc7471068.el8 ceph-noarch 18 k 2023-10-16T12:07:52.302 INFO:teuthology.orchestra.run.smithi016.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-10-16T12:07:52.302 INFO:teuthology.orchestra.run.smithi016.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-10-16T12:07:52.302 INFO:teuthology.orchestra.run.smithi016.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-10-16T12:07:52.303 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs-devel x86_64 2:18.0.0-6675.gc7471068.el8 ceph 31 k 2023-10-16T12:07:52.303 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs2 x86_64 2:18.0.0-6675.gc7471068.el8 ceph 839 k 2023-10-16T12:07:52.303 INFO:teuthology.orchestra.run.smithi016.stdout: librados-devel x86_64 2:18.0.0-6675.gc7471068.el8 ceph 139 k 2023-10-16T12:07:52.303 INFO:teuthology.orchestra.run.smithi016.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-10-16T12:07:52.303 INFO:teuthology.orchestra.run.smithi016.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-10-16T12:07:52.303 INFO:teuthology.orchestra.run.smithi016.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-10-16T12:07:52.304 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-10-16T12:07:52.304 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-10-16T12:07:52.304 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-10-16T12:07:52.304 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cephfs x86_64 2:18.0.0-6675.gc7471068.el8 ceph 197 k 2023-10-16T12:07:52.304 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rados x86_64 2:18.0.0-6675.gc7471068.el8 ceph 371 k 2023-10-16T12:07:52.305 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rbd x86_64 2:18.0.0-6675.gc7471068.el8 ceph 364 k 2023-10-16T12:07:52.305 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rgw x86_64 2:18.0.0-6675.gc7471068.el8 ceph 107 k 2023-10-16T12:07:52.305 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-fuse x86_64 2:18.0.0-6675.gc7471068.el8 ceph 94 k 2023-10-16T12:07:52.305 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-mirror x86_64 2:18.0.0-6675.gc7471068.el8 ceph 3.6 M 2023-10-16T12:07:52.305 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-nbd x86_64 2:18.0.0-6675.gc7471068.el8 ceph 180 k 2023-10-16T12:07:52.306 INFO:teuthology.orchestra.run.smithi016.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-10-16T12:07:52.306 INFO:teuthology.orchestra.run.smithi016.stdout:Upgrading: 2023-10-16T12:07:52.306 INFO:teuthology.orchestra.run.smithi016.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-10-16T12:07:52.306 INFO:teuthology.orchestra.run.smithi016.stdout: librados2 x86_64 2:18.0.0-6675.gc7471068.el8 ceph 3.7 M 2023-10-16T12:07:52.306 INFO:teuthology.orchestra.run.smithi016.stdout: librbd1 x86_64 2:18.0.0-6675.gc7471068.el8 ceph 3.7 M 2023-10-16T12:07:52.307 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2023-10-16T12:07:52.307 INFO:teuthology.orchestra.run.smithi016.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-10-16T12:07:52.307 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-common x86_64 2:18.0.0-6675.gc7471068.el8 ceph 23 M 2023-10-16T12:07:52.307 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-grafana-dashboards noarch 2:18.0.0-6675.gc7471068.el8 ceph-noarch 23 k 2023-10-16T12:07:52.307 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mds x86_64 2:18.0.0-6675.gc7471068.el8 ceph 2.3 M 2023-10-16T12:07:52.308 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-modules-core noarch 2:18.0.0-6675.gc7471068.el8 ceph-noarch 243 k 2023-10-16T12:07:52.308 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mon x86_64 2:18.0.0-6675.gc7471068.el8 ceph 4.6 M 2023-10-16T12:07:52.308 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-osd x86_64 2:18.0.0-6675.gc7471068.el8 ceph 18 M 2023-10-16T12:07:52.308 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-prometheus-alerts noarch 2:18.0.0-6675.gc7471068.el8 ceph-noarch 15 k 2023-10-16T12:07:52.308 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-selinux x86_64 2:18.0.0-6675.gc7471068.el8 ceph 24 k 2023-10-16T12:07:52.309 INFO:teuthology.orchestra.run.smithi016.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-10-16T12:07:52.309 INFO:teuthology.orchestra.run.smithi016.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-10-16T12:07:52.309 INFO:teuthology.orchestra.run.smithi016.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-10-16T12:07:52.309 INFO:teuthology.orchestra.run.smithi016.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-10-16T12:07:52.309 INFO:teuthology.orchestra.run.smithi016.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-10-16T12:07:52.310 INFO:teuthology.orchestra.run.smithi016.stdout: libcephsqlite x86_64 2:18.0.0-6675.gc7471068.el8 ceph 184 k 2023-10-16T12:07:52.310 INFO:teuthology.orchestra.run.smithi016.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-10-16T12:07:52.310 INFO:teuthology.orchestra.run.smithi016.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-10-16T12:07:52.310 INFO:teuthology.orchestra.run.smithi016.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-10-16T12:07:52.310 INFO:teuthology.orchestra.run.smithi016.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-10-16T12:07:52.311 INFO:teuthology.orchestra.run.smithi016.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-10-16T12:07:52.311 INFO:teuthology.orchestra.run.smithi016.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-10-16T12:07:52.311 INFO:teuthology.orchestra.run.smithi016.stdout: libradosstriper1 x86_64 2:18.0.0-6675.gc7471068.el8 ceph 518 k 2023-10-16T12:07:52.311 INFO:teuthology.orchestra.run.smithi016.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-10-16T12:07:52.311 INFO:teuthology.orchestra.run.smithi016.stdout: librgw2 x86_64 2:18.0.0-6675.gc7471068.el8 ceph 6.9 M 2023-10-16T12:07:52.312 INFO:teuthology.orchestra.run.smithi016.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-10-16T12:07:52.312 INFO:teuthology.orchestra.run.smithi016.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-10-16T12:07:52.312 INFO:teuthology.orchestra.run.smithi016.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-10-16T12:07:52.312 INFO:teuthology.orchestra.run.smithi016.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-10-16T12:07:52.312 INFO:teuthology.orchestra.run.smithi016.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-10-16T12:07:52.313 INFO:teuthology.orchestra.run.smithi016.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-10-16T12:07:52.313 INFO:teuthology.orchestra.run.smithi016.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-10-16T12:07:52.313 INFO:teuthology.orchestra.run.smithi016.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-10-16T12:07:52.313 INFO:teuthology.orchestra.run.smithi016.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-10-16T12:07:52.313 INFO:teuthology.orchestra.run.smithi016.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-10-16T12:07:52.314 INFO:teuthology.orchestra.run.smithi016.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-10-16T12:07:52.314 INFO:teuthology.orchestra.run.smithi016.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-10-16T12:07:52.314 INFO:teuthology.orchestra.run.smithi016.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-10-16T12:07:52.314 INFO:teuthology.orchestra.run.smithi016.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-10-16T12:07:52.314 INFO:teuthology.orchestra.run.smithi016.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-10-16T12:07:52.315 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-10-16T12:07:52.315 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-argparse x86_64 2:18.0.0-6675.gc7471068.el8 ceph 45 k 2023-10-16T12:07:52.315 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-common x86_64 2:18.0.0-6675.gc7471068.el8 ceph 126 k 2023-10-16T12:07:52.315 INFO:teuthology.orchestra.run.smithi016.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-10-16T12:07:52.315 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-10-16T12:07:52.316 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-10-16T12:07:52.316 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-10-16T12:07:52.316 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-10-16T12:07:52.316 INFO:teuthology.orchestra.run.smithi016.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-10-16T12:07:52.316 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-10-16T12:07:52.317 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-10-16T12:07:52.317 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-10-16T12:07:52.317 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-10-16T12:07:52.317 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-10-16T12:07:52.317 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-10-16T12:07:52.317 INFO:teuthology.orchestra.run.smithi016.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-10-16T12:07:52.318 INFO:teuthology.orchestra.run.smithi016.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-10-16T12:07:52.318 INFO:teuthology.orchestra.run.smithi016.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-10-16T12:07:52.318 INFO:teuthology.orchestra.run.smithi016.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-10-16T12:07:52.318 INFO:teuthology.orchestra.run.smithi016.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-10-16T12:07:52.319 INFO:teuthology.orchestra.run.smithi016.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-10-16T12:07:52.319 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-10-16T12:07:52.319 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-10-16T12:07:52.319 INFO:teuthology.orchestra.run.smithi016.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-10-16T12:07:52.319 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-10-16T12:07:52.319 INFO:teuthology.orchestra.run.smithi016.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-10-16T12:07:52.320 INFO:teuthology.orchestra.run.smithi016.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-10-16T12:07:52.320 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-10-16T12:07:52.320 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-10-16T12:07:52.320 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-10-16T12:07:52.320 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-10-16T12:07:52.321 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-10-16T12:07:52.321 INFO:teuthology.orchestra.run.smithi016.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-10-16T12:07:52.321 INFO:teuthology.orchestra.run.smithi016.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-10-16T12:07:52.321 INFO:teuthology.orchestra.run.smithi016.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-10-16T12:07:52.321 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-10-16T12:07:52.322 INFO:teuthology.orchestra.run.smithi016.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-10-16T12:07:52.322 INFO:teuthology.orchestra.run.smithi016.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-10-16T12:07:52.322 INFO:teuthology.orchestra.run.smithi016.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-10-16T12:07:52.322 INFO:teuthology.orchestra.run.smithi016.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-10-16T12:07:52.323 INFO:teuthology.orchestra.run.smithi016.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-10-16T12:07:52.323 INFO:teuthology.orchestra.run.smithi016.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-10-16T12:07:52.323 INFO:teuthology.orchestra.run.smithi016.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-10-16T12:07:52.323 INFO:teuthology.orchestra.run.smithi016.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-10-16T12:07:52.323 INFO:teuthology.orchestra.run.smithi016.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-10-16T12:07:52.323 INFO:teuthology.orchestra.run.smithi016.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-10-16T12:07:52.324 INFO:teuthology.orchestra.run.smithi016.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-10-16T12:07:52.324 INFO:teuthology.orchestra.run.smithi016.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-10-16T12:07:52.324 INFO:teuthology.orchestra.run.smithi016.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-10-16T12:07:52.324 INFO:teuthology.orchestra.run.smithi016.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-10-16T12:07:52.324 INFO:teuthology.orchestra.run.smithi016.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-10-16T12:07:52.325 INFO:teuthology.orchestra.run.smithi016.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-10-16T12:07:52.325 INFO:teuthology.orchestra.run.smithi016.stdout:Enabling module streams: 2023-10-16T12:07:52.325 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql 10 2023-10-16T12:07:52.325 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:07:52.325 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2023-10-16T12:07:52.326 INFO:teuthology.orchestra.run.smithi016.stdout:=========================================================================================================================== 2023-10-16T12:07:52.326 INFO:teuthology.orchestra.run.smithi016.stdout:Install 128 Packages 2023-10-16T12:07:52.326 INFO:teuthology.orchestra.run.smithi016.stdout:Upgrade 3 Packages 2023-10-16T12:07:52.326 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:07:52.327 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 209 M 2023-10-16T12:07:52.327 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2023-10-16T12:07:52.337 INFO:teuthology.orchestra.run.smithi049.stdout:(1/131): ceph-18.0.0-6675.gc7471068.el8.x86_64. 25 kB/s | 6.4 kB 00:00 2023-10-16T12:07:52.712 INFO:teuthology.orchestra.run.smithi049.stdout:(2/131): ceph-fuse-18.0.0-6675.gc7471068.el8.x8 2.4 MB/s | 919 kB 00:00 2023-10-16T12:07:52.799 INFO:teuthology.orchestra.run.smithi016.stdout:(1/131): ceph-18.0.0-6675.gc7471068.el8.x86_64. 48 kB/s | 6.4 kB 00:00 2023-10-16T12:07:52.812 INFO:teuthology.orchestra.run.smithi049.stdout:(3/131): ceph-immutable-object-cache-18.0.0-667 1.6 MB/s | 167 kB 00:00 2023-10-16T12:07:53.124 INFO:teuthology.orchestra.run.smithi016.stdout:(2/131): ceph-fuse-18.0.0-6675.gc7471068.el8.x8 2.8 MB/s | 919 kB 00:00 2023-10-16T12:07:53.241 INFO:teuthology.orchestra.run.smithi016.stdout:(3/131): ceph-immutable-object-cache-18.0.0-667 1.4 MB/s | 167 kB 00:00 2023-10-16T12:07:53.487 INFO:teuthology.orchestra.run.smithi049.stdout:(4/131): ceph-mds-18.0.0-6675.gc7471068.el8.x86 3.4 MB/s | 2.3 MB 00:00 2023-10-16T12:07:54.042 INFO:teuthology.orchestra.run.smithi016.stdout:(4/131): ceph-mds-18.0.0-6675.gc7471068.el8.x86 2.9 MB/s | 2.3 MB 00:00 2023-10-16T12:07:54.096 INFO:teuthology.orchestra.run.smithi049.stdout:(5/131): ceph-mgr-18.0.0-6675.gc7471068.el8.x86 2.7 MB/s | 1.6 MB 00:00 2023-10-16T12:07:54.117 INFO:teuthology.orchestra.run.smithi016.stdout:(5/131): ceph-base-18.0.0-6675.gc7471068.el8.x8 3.7 MB/s | 5.4 MB 00:01 2023-10-16T12:07:54.338 INFO:teuthology.orchestra.run.smithi049.stdout:(6/131): ceph-common-18.0.0-6675.gc7471068.el8. 10 MB/s | 23 MB 00:02 2023-10-16T12:07:54.350 INFO:teuthology.orchestra.run.smithi016.stdout:(6/131): ceph-mgr-18.0.0-6675.gc7471068.el8.x86 5.3 MB/s | 1.6 MB 00:00 2023-10-16T12:07:54.413 INFO:teuthology.orchestra.run.smithi049.stdout:(7/131): ceph-base-18.0.0-6675.gc7471068.el8.x8 2.3 MB/s | 5.4 MB 00:02 2023-10-16T12:07:54.584 INFO:teuthology.orchestra.run.smithi016.stdout:(7/131): ceph-mon-18.0.0-6675.gc7471068.el8.x86 9.9 MB/s | 4.6 MB 00:00 2023-10-16T12:07:56.572 INFO:teuthology.orchestra.run.smithi049.stdout:(8/131): ceph-mon-18.0.0-6675.gc7471068.el8.x86 1.9 MB/s | 4.6 MB 00:02 2023-10-16T12:07:56.701 INFO:teuthology.orchestra.run.smithi016.stdout:(8/131): ceph-radosgw-18.0.0-6675.gc7471068.el8 6.6 MB/s | 14 MB 00:02 2023-10-16T12:07:56.751 INFO:teuthology.orchestra.run.smithi016.stdout:(9/131): ceph-selinux-18.0.0-6675.gc7471068.el8 489 kB/s | 24 kB 00:00 2023-10-16T12:07:56.755 INFO:teuthology.orchestra.run.smithi049.stdout:(9/131): ceph-osd-18.0.0-6675.gc7471068.el8.x86 7.4 MB/s | 18 MB 00:02 2023-10-16T12:07:56.780 INFO:teuthology.orchestra.run.smithi049.stdout:(10/131): ceph-selinux-18.0.0-6675.gc7471068.el 117 kB/s | 24 kB 00:00 2023-10-16T12:07:57.014 INFO:teuthology.orchestra.run.smithi049.stdout:(11/131): cephfs-mirror-18.0.0-6675.gc7471068.e 1.1 MB/s | 261 kB 00:00 2023-10-16T12:07:57.139 INFO:teuthology.orchestra.run.smithi049.stdout:(12/131): libcephfs-devel-18.0.0-6675.gc7471068 248 kB/s | 31 kB 00:00 2023-10-16T12:08:04.297 INFO:teuthology.orchestra.run.smithi049.stdout:(13/131): libcephfs2-18.0.0-6675.gc7471068.el8. 117 kB/s | 839 kB 00:07 2023-10-16T12:08:04.448 INFO:teuthology.orchestra.run.smithi049.stdout:(14/131): libcephsqlite-18.0.0-6675.gc7471068.e 1.2 MB/s | 184 kB 00:00 2023-10-16T12:08:04.598 INFO:teuthology.orchestra.run.smithi049.stdout:(15/131): librados-devel-18.0.0-6675.gc7471068. 927 kB/s | 139 kB 00:00 2023-10-16T12:08:04.823 INFO:teuthology.orchestra.run.smithi049.stdout:(16/131): libradosstriper1-18.0.0-6675.gc747106 2.3 MB/s | 518 kB 00:00 2023-10-16T12:08:05.167 INFO:teuthology.orchestra.run.smithi016.stdout:(10/131): ceph-common-18.0.0-6675.gc7471068.el8 1.8 MB/s | 23 MB 00:12 2023-10-16T12:08:05.343 INFO:teuthology.orchestra.run.smithi016.stdout:(11/131): cephfs-mirror-18.0.0-6675.gc7471068.e 1.5 MB/s | 261 kB 00:00 2023-10-16T12:08:05.443 INFO:teuthology.orchestra.run.smithi016.stdout:(12/131): libcephfs-devel-18.0.0-6675.gc7471068 310 kB/s | 31 kB 00:00 2023-10-16T12:08:05.735 INFO:teuthology.orchestra.run.smithi016.stdout:(13/131): libcephfs2-18.0.0-6675.gc7471068.el8. 2.8 MB/s | 839 kB 00:00 2023-10-16T12:08:05.961 INFO:teuthology.orchestra.run.smithi016.stdout:(14/131): ceph-osd-18.0.0-6675.gc7471068.el8.x8 1.5 MB/s | 18 MB 00:11 2023-10-16T12:08:05.986 INFO:teuthology.orchestra.run.smithi016.stdout:(15/131): libcephsqlite-18.0.0-6675.gc7471068.e 734 kB/s | 184 kB 00:00 2023-10-16T12:08:06.044 INFO:teuthology.orchestra.run.smithi016.stdout:(16/131): librados-devel-18.0.0-6675.gc7471068. 1.6 MB/s | 139 kB 00:00 2023-10-16T12:08:06.178 INFO:teuthology.orchestra.run.smithi016.stdout:(17/131): libradosstriper1-18.0.0-6675.gc747106 2.6 MB/s | 518 kB 00:00 2023-10-16T12:08:06.286 INFO:teuthology.orchestra.run.smithi016.stdout:(18/131): python3-ceph-argparse-18.0.0-6675.gc7 413 kB/s | 45 kB 00:00 2023-10-16T12:08:06.412 INFO:teuthology.orchestra.run.smithi016.stdout:(19/131): python3-ceph-common-18.0.0-6675.gc747 1.0 MB/s | 126 kB 00:00 2023-10-16T12:08:06.520 INFO:teuthology.orchestra.run.smithi016.stdout:(20/131): python3-cephfs-18.0.0-6675.gc7471068. 1.8 MB/s | 197 kB 00:00 2023-10-16T12:08:06.665 INFO:teuthology.orchestra.run.smithi049.stdout:(17/131): librgw2-18.0.0-6675.gc7471068.el8.x86 3.8 MB/s | 6.9 MB 00:01 2023-10-16T12:08:06.696 INFO:teuthology.orchestra.run.smithi016.stdout:(21/131): python3-rados-18.0.0-6675.gc7471068.e 2.1 MB/s | 371 kB 00:00 2023-10-16T12:08:06.765 INFO:teuthology.orchestra.run.smithi049.stdout:(18/131): python3-ceph-argparse-18.0.0-6675.gc7 448 kB/s | 45 kB 00:00 2023-10-16T12:08:06.888 INFO:teuthology.orchestra.run.smithi016.stdout:(22/131): python3-rbd-18.0.0-6675.gc7471068.el8 1.9 MB/s | 364 kB 00:00 2023-10-16T12:08:06.932 INFO:teuthology.orchestra.run.smithi049.stdout:(19/131): python3-ceph-common-18.0.0-6675.gc747 753 kB/s | 126 kB 00:00 2023-10-16T12:08:06.988 INFO:teuthology.orchestra.run.smithi016.stdout:(23/131): python3-rgw-18.0.0-6675.gc7471068.el8 1.0 MB/s | 107 kB 00:00 2023-10-16T12:08:07.233 INFO:teuthology.orchestra.run.smithi049.stdout:(20/131): ceph-radosgw-18.0.0-6675.gc7471068.el 1.1 MB/s | 14 MB 00:12 2023-10-16T12:08:07.650 INFO:teuthology.orchestra.run.smithi049.stdout:(21/131): python3-cephfs-18.0.0-6675.gc7471068. 274 kB/s | 197 kB 00:00 2023-10-16T12:08:07.696 INFO:teuthology.orchestra.run.smithi016.stdout:(24/131): rbd-fuse-18.0.0-6675.gc7471068.el8.x8 132 kB/s | 94 kB 00:00 2023-10-16T12:08:07.825 INFO:teuthology.orchestra.run.smithi049.stdout:(22/131): python3-rbd-18.0.0-6675.gc7471068.el8 2.0 MB/s | 364 kB 00:00 2023-10-16T12:08:07.850 INFO:teuthology.orchestra.run.smithi049.stdout:(23/131): python3-rados-18.0.0-6675.gc7471068.e 601 kB/s | 371 kB 00:00 2023-10-16T12:08:07.917 INFO:teuthology.orchestra.run.smithi049.stdout:(24/131): python3-rgw-18.0.0-6675.gc7471068.el8 1.1 MB/s | 107 kB 00:00 2023-10-16T12:08:07.950 INFO:teuthology.orchestra.run.smithi049.stdout:(25/131): rbd-fuse-18.0.0-6675.gc7471068.el8.x8 933 kB/s | 94 kB 00:00 2023-10-16T12:08:08.109 INFO:teuthology.orchestra.run.smithi049.stdout:(26/131): rbd-nbd-18.0.0-6675.gc7471068.el8.x86 1.1 MB/s | 180 kB 00:00 2023-10-16T12:08:08.172 INFO:teuthology.orchestra.run.smithi016.stdout:(25/131): librgw2-18.0.0-6675.gc7471068.el8.x86 3.3 MB/s | 6.9 MB 00:02 2023-10-16T12:08:08.193 INFO:teuthology.orchestra.run.smithi049.stdout:(27/131): ceph-grafana-dashboards-18.0.0-6675.g 277 kB/s | 23 kB 00:00 2023-10-16T12:08:08.305 INFO:teuthology.orchestra.run.smithi016.stdout:(26/131): rbd-nbd-18.0.0-6675.gc7471068.el8.x86 1.3 MB/s | 180 kB 00:00 2023-10-16T12:08:08.343 INFO:teuthology.orchestra.run.smithi049.stdout:(28/131): ceph-mgr-cephadm-18.0.0-6675.gc747106 880 kB/s | 132 kB 00:00 2023-10-16T12:08:08.406 INFO:teuthology.orchestra.run.smithi016.stdout:(27/131): ceph-grafana-dashboards-18.0.0-6675.g 230 kB/s | 23 kB 00:00 2023-10-16T12:08:08.506 INFO:teuthology.orchestra.run.smithi016.stdout:(28/131): ceph-mgr-cephadm-18.0.0-6675.gc747106 1.3 MB/s | 132 kB 00:00 2023-10-16T12:08:08.835 INFO:teuthology.orchestra.run.smithi049.stdout:(29/131): rbd-mirror-18.0.0-6675.gc7471068.el8. 3.9 MB/s | 3.6 MB 00:00 2023-10-16T12:08:08.910 INFO:teuthology.orchestra.run.smithi049.stdout:(30/131): ceph-mgr-dashboard-18.0.0-6675.gc7471 3.1 MB/s | 1.8 MB 00:00 2023-10-16T12:08:08.973 INFO:teuthology.orchestra.run.smithi016.stdout:(29/131): ceph-test-18.0.0-6675.gc7471068.el8.x 4.0 MB/s | 49 MB 00:12 2023-10-16T12:08:09.040 INFO:teuthology.orchestra.run.smithi016.stdout:(30/131): rbd-mirror-18.0.0-6675.gc7471068.el8. 2.7 MB/s | 3.6 MB 00:01 2023-10-16T12:08:09.044 INFO:teuthology.orchestra.run.smithi049.stdout:(31/131): ceph-mgr-modules-core-18.0.0-6675.gc7 1.8 MB/s | 243 kB 00:00 2023-10-16T12:08:09.090 INFO:teuthology.orchestra.run.smithi016.stdout:(31/131): ceph-mgr-dashboard-18.0.0-6675.gc7471 3.0 MB/s | 1.8 MB 00:00 2023-10-16T12:08:09.132 INFO:teuthology.orchestra.run.smithi016.stdout:(32/131): ceph-mgr-rook-18.0.0-6675.gc7471068.e 1.1 MB/s | 47 kB 00:00 2023-10-16T12:08:09.157 INFO:teuthology.orchestra.run.smithi016.stdout:(33/131): ceph-mgr-modules-core-18.0.0-6675.gc7 2.0 MB/s | 243 kB 00:00 2023-10-16T12:08:09.174 INFO:teuthology.orchestra.run.smithi016.stdout:(34/131): ceph-prometheus-alerts-18.0.0-6675.gc 348 kB/s | 15 kB 00:00 2023-10-16T12:08:09.224 INFO:teuthology.orchestra.run.smithi016.stdout:(35/131): ceph-volume-18.0.0-6675.gc7471068.el8 3.8 MB/s | 261 kB 00:00 2023-10-16T12:08:09.266 INFO:teuthology.orchestra.run.smithi016.stdout:(36/131): cephadm-18.0.0-6675.gc7471068.el8.noa 2.0 MB/s | 192 kB 00:00 2023-10-16T12:08:09.509 INFO:teuthology.orchestra.run.smithi016.stdout:(37/131): ceph-mgr-diskprediction-local-18.0.0- 14 MB/s | 7.4 MB 00:00 2023-10-16T12:08:09.527 INFO:teuthology.orchestra.run.smithi049.stdout:(32/131): ceph-test-18.0.0-6675.gc7471068.el8.x 3.8 MB/s | 49 MB 00:12 2023-10-16T12:08:09.534 INFO:teuthology.orchestra.run.smithi016.stdout:(38/131): cephfs-top-18.0.0-6675.gc7471068.el8. 59 kB/s | 18 kB 00:00 2023-10-16T12:08:09.553 INFO:teuthology.orchestra.run.smithi049.stdout:(33/131): ceph-mgr-rook-18.0.0-6675.gc7471068.e 93 kB/s | 47 kB 00:00 2023-10-16T12:08:09.578 INFO:teuthology.orchestra.run.smithi049.stdout:(34/131): ceph-prometheus-alerts-18.0.0-6675.gc 290 kB/s | 15 kB 00:00 2023-10-16T12:08:09.629 INFO:teuthology.orchestra.run.smithi049.stdout:(35/131): cephadm-18.0.0-6675.gc7471068.el8.noa 3.8 MB/s | 192 kB 00:00 2023-10-16T12:08:09.654 INFO:teuthology.orchestra.run.smithi049.stdout:(36/131): ceph-volume-18.0.0-6675.gc7471068.el8 2.5 MB/s | 261 kB 00:00 2023-10-16T12:08:09.681 INFO:teuthology.orchestra.run.smithi049.stdout:(37/131): cephfs-top-18.0.0-6675.gc7471068.el8. 362 kB/s | 18 kB 00:00 2023-10-16T12:08:09.917 INFO:teuthology.orchestra.run.smithi016.stdout:(39/131): gperftools-libs-2.7-9.el8.x86_64.rpm 471 kB/s | 306 kB 00:00 2023-10-16T12:08:09.943 INFO:teuthology.orchestra.run.smithi016.stdout:(40/131): liboath-2.6.2-3.el8.x86_64.rpm 136 kB/s | 59 kB 00:00 2023-10-16T12:08:09.968 INFO:teuthology.orchestra.run.smithi016.stdout:(41/131): libunwind-1.3.1-3.el8.x86_64.rpm 173 kB/s | 75 kB 00:00 2023-10-16T12:08:09.987 INFO:teuthology.orchestra.run.smithi049.stdout:(38/131): liboath-2.6.2-3.el8.x86_64.rpm 191 kB/s | 59 kB 00:00 2023-10-16T12:08:10.052 INFO:teuthology.orchestra.run.smithi016.stdout:(42/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 402 kB/s | 44 kB 00:00 2023-10-16T12:08:10.071 INFO:teuthology.orchestra.run.smithi049.stdout:(39/131): ceph-mgr-diskprediction-local-18.0.0- 6.0 MB/s | 7.4 MB 00:01 2023-10-16T12:08:10.102 INFO:teuthology.orchestra.run.smithi016.stdout:(43/131): python3-asyncssh-2.7.0-2.el8.noarch.r 2.5 MB/s | 476 kB 00:00 2023-10-16T12:08:10.146 INFO:teuthology.orchestra.run.smithi049.stdout:(40/131): libunwind-1.3.1-3.el8.x86_64.rpm 473 kB/s | 75 kB 00:00 2023-10-16T12:08:10.152 INFO:teuthology.orchestra.run.smithi016.stdout:(44/131): python3-cachetools-3.1.1-4.el8.noarch 329 kB/s | 33 kB 00:00 2023-10-16T12:08:10.172 INFO:teuthology.orchestra.run.smithi049.stdout:(41/131): gperftools-libs-2.7-9.el8.x86_64.rpm 592 kB/s | 306 kB 00:00 2023-10-16T12:08:10.178 INFO:teuthology.orchestra.run.smithi016.stdout:(45/131): python3-beautifulsoup4-4.6.3-2.el8.1. 886 kB/s | 185 kB 00:00 2023-10-16T12:08:10.203 INFO:teuthology.orchestra.run.smithi016.stdout:(46/131): python3-certifi-2018.10.15-7.el8.noar 162 kB/s | 16 kB 00:00 2023-10-16T12:08:10.247 INFO:teuthology.orchestra.run.smithi049.stdout:(42/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 434 kB/s | 44 kB 00:00 2023-10-16T12:08:10.272 INFO:teuthology.orchestra.run.smithi049.stdout:(43/131): python3-beautifulsoup4-4.6.3-2.el8.1. 1.8 MB/s | 185 kB 00:00 2023-10-16T12:08:10.303 INFO:teuthology.orchestra.run.smithi016.stdout:(47/131): python3-google-auth-1.1.1-10.el8.noar 813 kB/s | 81 kB 00:00 2023-10-16T12:08:10.328 INFO:teuthology.orchestra.run.smithi016.stdout:(48/131): python3-cheroot-8.5.2-1.el8.noarch.rp 982 kB/s | 173 kB 00:00 2023-10-16T12:08:10.340 INFO:teuthology.orchestra.run.smithi049.stdout:(44/131): python3-cachetools-3.1.1-4.el8.noarch 359 kB/s | 33 kB 00:00 2023-10-16T12:08:10.370 INFO:teuthology.orchestra.run.smithi016.stdout:(49/131): python3-cherrypy-18.4.0-1.el8.noarch. 2.0 MB/s | 384 kB 00:00 2023-10-16T12:08:10.373 INFO:teuthology.orchestra.run.smithi049.stdout:(45/131): python3-certifi-2018.10.15-7.el8.noar 162 kB/s | 16 kB 00:00 2023-10-16T12:08:10.404 INFO:teuthology.orchestra.run.smithi016.stdout:(50/131): python3-jaraco-6.2-6.el8.noarch.rpm 110 kB/s | 11 kB 00:00 2023-10-16T12:08:10.429 INFO:teuthology.orchestra.run.smithi016.stdout:(51/131): python3-jaraco-functools-2.0-4.el8.no 175 kB/s | 18 kB 00:00 2023-10-16T12:08:10.505 INFO:teuthology.orchestra.run.smithi016.stdout:(52/131): python3-logutils-0.3.5-11.el8.noarch. 485 kB/s | 49 kB 00:00 2023-10-16T12:08:10.507 INFO:teuthology.orchestra.run.smithi049.stdout:(46/131): python3-cheroot-8.5.2-1.el8.noarch.rp 1.0 MB/s | 173 kB 00:00 2023-10-16T12:08:10.530 INFO:teuthology.orchestra.run.smithi016.stdout:(53/131): python3-more-itertools-7.2.0-3.el8.no 586 kB/s | 59 kB 00:00 2023-10-16T12:08:10.540 INFO:teuthology.orchestra.run.smithi049.stdout:(47/131): python3-cherrypy-18.4.0-1.el8.noarch. 2.3 MB/s | 384 kB 00:00 2023-10-16T12:08:10.572 INFO:teuthology.orchestra.run.smithi016.stdout:(54/131): python3-kubernetes-11.0.0-6.el8.noarc 5.8 MB/s | 1.2 MB 00:00 2023-10-16T12:08:10.599 INFO:teuthology.orchestra.run.smithi049.stdout:(48/131): python3-google-auth-1.1.1-10.el8.noar 884 kB/s | 81 kB 00:00 2023-10-16T12:08:10.630 INFO:teuthology.orchestra.run.smithi016.stdout:(55/131): python3-natsort-7.1.1-2.el8.noarch.rp 480 kB/s | 60 kB 00:00 2023-10-16T12:08:10.632 INFO:teuthology.orchestra.run.smithi049.stdout:(49/131): python3-asyncssh-2.7.0-2.el8.noarch.r 848 kB/s | 476 kB 00:00 2023-10-16T12:08:10.658 INFO:teuthology.orchestra.run.smithi049.stdout:(50/131): python3-jaraco-6.2-6.el8.noarch.rpm 95 kB/s | 11 kB 00:00 2023-10-16T12:08:10.680 INFO:teuthology.orchestra.run.smithi016.stdout:(56/131): python3-portend-2.6-1.el8.noarch.rpm 143 kB/s | 16 kB 00:00 2023-10-16T12:08:10.692 INFO:teuthology.orchestra.run.smithi049.stdout:(51/131): python3-jaraco-functools-2.0-4.el8.no 190 kB/s | 18 kB 00:00 2023-10-16T12:08:10.706 INFO:teuthology.orchestra.run.smithi016.stdout:(57/131): python3-pecan-1.3.2-9.el8.noarch.rpm 1.6 MB/s | 283 kB 00:00 2023-10-16T12:08:10.731 INFO:teuthology.orchestra.run.smithi016.stdout:(58/131): python3-repoze-lru-0.7-6.el8.noarch.r 335 kB/s | 34 kB 00:00 2023-10-16T12:08:10.758 INFO:teuthology.orchestra.run.smithi049.stdout:(52/131): python3-logutils-0.3.5-11.el8.noarch. 486 kB/s | 49 kB 00:00 2023-10-16T12:08:10.781 INFO:teuthology.orchestra.run.smithi016.stdout:(59/131): python3-routes-2.4.1-12.el8.noarch.rp 1.9 MB/s | 196 kB 00:00 2023-10-16T12:08:10.784 INFO:teuthology.orchestra.run.smithi049.stdout:(53/131): python3-more-itertools-7.2.0-3.el8.no 638 kB/s | 59 kB 00:00 2023-10-16T12:08:10.815 INFO:teuthology.orchestra.run.smithi016.stdout:(60/131): python3-rsa-4.9-2.el8.noarch.rpm 590 kB/s | 64 kB 00:00 2023-10-16T12:08:10.817 INFO:teuthology.orchestra.run.smithi049.stdout:(54/131): python3-kubernetes-11.0.0-6.el8.noarc 6.3 MB/s | 1.2 MB 00:00 2023-10-16T12:08:10.840 INFO:teuthology.orchestra.run.smithi016.stdout:(61/131): python3-simplegeneric-0.8.1-17.el8.no 172 kB/s | 19 kB 00:00 2023-10-16T12:08:10.859 INFO:teuthology.orchestra.run.smithi049.stdout:(55/131): python3-natsort-7.1.1-2.el8.noarch.rp 599 kB/s | 60 kB 00:00 2023-10-16T12:08:10.882 INFO:teuthology.orchestra.run.smithi016.stdout:(62/131): python3-singledispatch-3.4.0.3-18.el8 237 kB/s | 24 kB 00:00 2023-10-16T12:08:10.884 INFO:teuthology.orchestra.run.smithi049.stdout:(56/131): python3-pecan-1.3.2-9.el8.noarch.rpm 2.7 MB/s | 283 kB 00:00 2023-10-16T12:08:10.916 INFO:teuthology.orchestra.run.smithi016.stdout:(63/131): python3-tempora-1.14.1-5.el8.noarch.r 291 kB/s | 29 kB 00:00 2023-10-16T12:08:10.918 INFO:teuthology.orchestra.run.smithi049.stdout:(57/131): python3-portend-2.6-1.el8.noarch.rpm 155 kB/s | 16 kB 00:00 2023-10-16T12:08:10.941 INFO:teuthology.orchestra.run.smithi016.stdout:(64/131): python3-trustme-0.6.0-4.el8.noarch.rp 273 kB/s | 27 kB 00:00 2023-10-16T12:08:10.960 INFO:teuthology.orchestra.run.smithi049.stdout:(58/131): python3-repoze-lru-0.7-6.el8.noarch.r 334 kB/s | 34 kB 00:00 2023-10-16T12:08:10.985 INFO:teuthology.orchestra.run.smithi049.stdout:(59/131): python3-routes-2.4.1-12.el8.noarch.rp 1.9 MB/s | 196 kB 00:00 2023-10-16T12:08:10.991 INFO:teuthology.orchestra.run.smithi016.stdout:(65/131): python3-waitress-1.4.3-1.el8.noarch.r 2.3 MB/s | 255 kB 00:00 2023-10-16T12:08:11.019 INFO:teuthology.orchestra.run.smithi049.stdout:(60/131): python3-rsa-4.9-2.el8.noarch.rpm 639 kB/s | 64 kB 00:00 2023-10-16T12:08:11.025 INFO:teuthology.orchestra.run.smithi016.stdout:(66/131): python3-webob-1.8.5-1.el8.1.noarch.rp 2.3 MB/s | 251 kB 00:00 2023-10-16T12:08:11.050 INFO:teuthology.orchestra.run.smithi016.stdout:(67/131): python3-websocket-client-0.56.0-5.el8 564 kB/s | 61 kB 00:00 2023-10-16T12:08:11.069 INFO:teuthology.orchestra.run.smithi049.stdout:(61/131): python3-simplegeneric-0.8.1-17.el8.no 172 kB/s | 19 kB 00:00 2023-10-16T12:08:11.086 INFO:teuthology.orchestra.run.smithi049.stdout:(62/131): python3-singledispatch-3.4.0.3-18.el8 238 kB/s | 24 kB 00:00 2023-10-16T12:08:11.092 INFO:teuthology.orchestra.run.smithi016.stdout:(68/131): python3-webtest-2.0.33-1.el8.noarch.r 849 kB/s | 85 kB 00:00 2023-10-16T12:08:11.120 INFO:teuthology.orchestra.run.smithi049.stdout:(63/131): python3-tempora-1.14.1-5.el8.noarch.r 290 kB/s | 29 kB 00:00 2023-10-16T12:08:11.126 INFO:teuthology.orchestra.run.smithi016.stdout:(69/131): python3-zc-lockfile-2.0-2.el8.noarch. 228 kB/s | 23 kB 00:00 2023-10-16T12:08:11.162 INFO:teuthology.orchestra.run.smithi049.stdout:(64/131): python3-trustme-0.6.0-4.el8.noarch.rp 298 kB/s | 27 kB 00:00 2023-10-16T12:08:11.168 INFO:teuthology.orchestra.run.smithi016.stdout:(70/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 922 kB/s | 70 kB 00:00 2023-10-16T12:08:11.187 INFO:teuthology.orchestra.run.smithi049.stdout:(65/131): python3-waitress-1.4.3-1.el8.noarch.r 2.5 MB/s | 255 kB 00:00 2023-10-16T12:08:11.193 INFO:teuthology.orchestra.run.smithi016.stdout:(71/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 697 kB/s | 47 kB 00:00 2023-10-16T12:08:11.221 INFO:teuthology.orchestra.run.smithi049.stdout:(66/131): python3-webob-1.8.5-1.el8.1.noarch.rp 2.4 MB/s | 251 kB 00:00 2023-10-16T12:08:11.252 INFO:teuthology.orchestra.run.smithi016.stdout:(72/131): thrift-0.13.0-2.el8.x86_64.rpm 8.6 MB/s | 1.7 MB 00:00 2023-10-16T12:08:11.269 INFO:teuthology.orchestra.run.smithi016.stdout:(73/131): python3-jwt-1.6.1-2.el8.noarch.rpm 639 kB/s | 43 kB 00:00 2023-10-16T12:08:11.279 INFO:teuthology.orchestra.run.smithi049.stdout:(67/131): python3-websocket-client-0.56.0-5.el8 523 kB/s | 61 kB 00:00 2023-10-16T12:08:11.302 INFO:teuthology.orchestra.run.smithi016.stdout:(74/131): libselinux-devel-2.9-5.el8.x86_64.rpm 1.5 MB/s | 200 kB 00:00 2023-10-16T12:08:11.304 INFO:teuthology.orchestra.run.smithi049.stdout:(68/131): python3-webtest-2.0.33-1.el8.noarch.r 728 kB/s | 85 kB 00:00 2023-10-16T12:08:11.321 INFO:teuthology.orchestra.run.smithi049.stdout:(69/131): python3-zc-lockfile-2.0-2.el8.noarch. 229 kB/s | 23 kB 00:00 2023-10-16T12:08:11.328 INFO:teuthology.orchestra.run.smithi016.stdout:(75/131): keyutils-libs-devel-1.5.10-9.el8.x86_ 961 kB/s | 48 kB 00:00 2023-10-16T12:08:11.353 INFO:teuthology.orchestra.run.smithi016.stdout:(76/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 2.1 MB/s | 220 kB 00:00 2023-10-16T12:08:11.355 INFO:teuthology.orchestra.run.smithi049.stdout:(70/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.4 MB/s | 70 kB 00:00 2023-10-16T12:08:11.370 INFO:teuthology.orchestra.run.smithi016.stdout:(77/131): python3-requests-oauthlib-1.0.0-1.el8 1.0 MB/s | 43 kB 00:00 2023-10-16T12:08:11.395 INFO:teuthology.orchestra.run.smithi016.stdout:(78/131): libsepol-devel-2.9-3.el8.x86_64.rpm 940 kB/s | 87 kB 00:00 2023-10-16T12:08:11.420 INFO:teuthology.orchestra.run.smithi016.stdout:(79/131): python3-pycparser-2.14-14.el8.noarch. 1.6 MB/s | 109 kB 00:00 2023-10-16T12:08:11.439 INFO:teuthology.orchestra.run.smithi049.stdout:(71/131): python3-pyasn1-modules-0.3.7-6.el8.no 1.4 MB/s | 110 kB 00:00 2023-10-16T12:08:11.471 INFO:teuthology.orchestra.run.smithi016.stdout:(80/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 2.2 MB/s | 229 kB 00:00 2023-10-16T12:08:11.481 INFO:teuthology.orchestra.run.smithi049.stdout:(72/131): python3-pytz-2017.2-9.el8.noarch.rpm 1.3 MB/s | 54 kB 00:00 2023-10-16T12:08:11.496 INFO:teuthology.orchestra.run.smithi016.stdout:(81/131): libaio-devel-0.3.112-1.el8.x86_64.rpm 248 kB/s | 19 kB 00:00 2023-10-16T12:08:11.522 INFO:teuthology.orchestra.run.smithi016.stdout:(82/131): python3-oauthlib-2.1.0-1.el8.noarch.r 3.0 MB/s | 155 kB 00:00 2023-10-16T12:08:11.523 INFO:teuthology.orchestra.run.smithi049.stdout:(73/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 6.1 MB/s | 259 kB 00:00 2023-10-16T12:08:11.555 INFO:teuthology.orchestra.run.smithi016.stdout:(83/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 3.4 MB/s | 560 kB 00:00 2023-10-16T12:08:11.565 INFO:teuthology.orchestra.run.smithi049.stdout:(74/131): thrift-0.13.0-2.el8.x86_64.rpm 6.1 MB/s | 1.7 MB 00:00 2023-10-16T12:08:11.580 INFO:teuthology.orchestra.run.smithi016.stdout:(84/131): libverto-devel-0.3.0-5.el8.x86_64.rpm 306 kB/s | 18 kB 00:00 2023-10-16T12:08:11.598 INFO:teuthology.orchestra.run.smithi049.stdout:(75/131): socat-1.7.4.1-1.el8.x86_64.rpm 4.2 MB/s | 323 kB 00:00 2023-10-16T12:08:11.614 INFO:teuthology.orchestra.run.smithi016.stdout:(85/131): python3-cryptography-3.2.1-5.el8.x86_ 4.7 MB/s | 559 kB 00:00 2023-10-16T12:08:11.647 INFO:teuthology.orchestra.run.smithi016.stdout:(86/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 3.5 MB/s | 238 kB 00:00 2023-10-16T12:08:11.649 INFO:teuthology.orchestra.run.smithi049.stdout:(76/131): python3-markupsafe-0.23-19.el8.x86_64 464 kB/s | 39 kB 00:00 2023-10-16T12:08:11.681 INFO:teuthology.orchestra.run.smithi016.stdout:(87/131): pcre2-devel-10.32-2.el8.x86_64.rpm 4.7 MB/s | 605 kB 00:00 2023-10-16T12:08:11.682 INFO:teuthology.orchestra.run.smithi049.stdout:(77/131): bison-3.0.4-10.el8.x86_64.rpm 8.9 MB/s | 688 kB 00:00 2023-10-16T12:08:11.706 INFO:teuthology.orchestra.run.smithi016.stdout:(88/131): xfsprogs-devel-5.0.0-10.el8.x86_64.rp 1.2 MB/s | 72 kB 00:00 2023-10-16T12:08:11.716 INFO:teuthology.orchestra.run.smithi049.stdout:(78/131): python3-jinja2-2.10.1-3.el8.noarch.rp 7.9 MB/s | 538 kB 00:00 2023-10-16T12:08:11.732 INFO:teuthology.orchestra.run.smithi016.stdout:(89/131): fuse-2.9.7-15.el8.x86_64.rpm 1.6 MB/s | 83 kB 00:00 2023-10-16T12:08:11.741 INFO:teuthology.orchestra.run.smithi049.stdout:(79/131): python3-mako-1.0.6-13.el8.noarch.rpm 2.6 MB/s | 157 kB 00:00 2023-10-16T12:08:11.757 INFO:teuthology.orchestra.run.smithi016.stdout:(90/131): elfutils-libelf-devel-0.186-1.el8.x86 1.2 MB/s | 60 kB 00:00 2023-10-16T12:08:11.766 INFO:teuthology.orchestra.run.smithi049.stdout:(80/131): oniguruma-6.8.2-2.el8.x86_64.rpm 3.6 MB/s | 187 kB 00:00 2023-10-16T12:08:11.782 INFO:teuthology.orchestra.run.smithi016.stdout:(91/131): libuuid-devel-2.32.1-35.el8.x86_64.rp 1.9 MB/s | 98 kB 00:00 2023-10-16T12:08:11.800 INFO:teuthology.orchestra.run.smithi049.stdout:(81/131): automake-1.16.1-7.el8.noarch.rpm 12 MB/s | 713 kB 00:00 2023-10-16T12:08:11.807 INFO:teuthology.orchestra.run.smithi016.stdout:(92/131): zlib-devel-1.2.11-18.el8_5.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2023-10-16T12:08:11.833 INFO:teuthology.orchestra.run.smithi016.stdout:(93/131): libcom_err-devel-1.45.6-4.el8.x86_64. 769 kB/s | 39 kB 00:00 2023-10-16T12:08:11.842 INFO:teuthology.orchestra.run.smithi049.stdout:(82/131): python3-jsonpatch-1.21-2.el8.noarch.r 635 kB/s | 27 kB 00:00 2023-10-16T12:08:11.875 INFO:teuthology.orchestra.run.smithi016.stdout:(94/131): libgfortran-8.5.0-10.1.el8_6.x86_64.r 9.4 MB/s | 644 kB 00:00 2023-10-16T12:08:11.892 INFO:teuthology.orchestra.run.smithi049.stdout:(83/131): flex-2.6.1-9.el8.x86_64.rpm 6.3 MB/s | 320 kB 00:00 2023-10-16T12:08:11.900 INFO:teuthology.orchestra.run.smithi016.stdout:(95/131): libquadmath-8.5.0-10.1.el8_6.x86_64.r 2.5 MB/s | 171 kB 00:00 2023-10-16T12:08:11.925 INFO:teuthology.orchestra.run.smithi016.stdout:(96/131): NetworkManager-initscripts-updown-1.3 2.7 MB/s | 138 kB 00:00 2023-10-16T12:08:11.934 INFO:teuthology.orchestra.run.smithi049.stdout:(84/131): python3-prettytable-0.7.2-14.el8.noar 1.0 MB/s | 44 kB 00:00 2023-10-16T12:08:11.984 INFO:teuthology.orchestra.run.smithi016.stdout:(97/131): openssl-devel-1.1.1k-7.el8_6.x86_64.r 6.3 MB/s | 2.3 MB 00:00 2023-10-16T12:08:12.009 INFO:teuthology.orchestra.run.smithi016.stdout:(98/131): python3-pyasn1-modules-0.3.7-6.el8.no 1.3 MB/s | 110 kB 00:00 2023-10-16T12:08:12.035 INFO:teuthology.orchestra.run.smithi016.stdout:(99/131): python3-pytz-2017.2-9.el8.noarch.rpm 1.0 MB/s | 54 kB 00:00 2023-10-16T12:08:12.068 INFO:teuthology.orchestra.run.smithi016.stdout:(100/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 4.3 MB/s | 259 kB 00:00 2023-10-16T12:08:12.102 INFO:teuthology.orchestra.run.smithi016.stdout:(101/131): socat-1.7.4.1-1.el8.x86_64.rpm 4.7 MB/s | 323 kB 00:00 2023-10-16T12:08:12.127 INFO:teuthology.orchestra.run.smithi016.stdout:(102/131): python3-markupsafe-0.23-19.el8.x86_6 663 kB/s | 39 kB 00:00 2023-10-16T12:08:12.151 INFO:teuthology.orchestra.run.smithi049.stdout:(85/131): python3-scipy-1.0.0-21.module+el8.5.0 17 MB/s | 14 MB 00:00 2023-10-16T12:08:12.169 INFO:teuthology.orchestra.run.smithi016.stdout:(103/131): bison-3.0.4-10.el8.x86_64.rpm 10 MB/s | 688 kB 00:00 2023-10-16T12:08:12.278 INFO:teuthology.orchestra.run.smithi016.stdout:(104/131): python3-jinja2-2.10.1-3.el8.noarch.r 3.5 MB/s | 538 kB 00:00 2023-10-16T12:08:12.418 INFO:teuthology.orchestra.run.smithi049.stdout:(86/131): python3-babel-2.5.1-7.el8.noarch.rpm 7.3 MB/s | 4.8 MB 00:00 2023-10-16T12:08:12.452 INFO:teuthology.orchestra.run.smithi049.stdout:(87/131): python3-werkzeug-0.12.2-4.el8.noarch. 1.5 MB/s | 457 kB 00:00 2023-10-16T12:08:12.477 INFO:teuthology.orchestra.run.smithi049.stdout:(88/131): python3-numpy-f2py-1.14.3-10.el8.x86_ 3.8 MB/s | 225 kB 00:00 2023-10-16T12:08:12.502 INFO:teuthology.orchestra.run.smithi049.stdout:(89/131): uuid-1.6.2-43.el8.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2023-10-16T12:08:12.528 INFO:teuthology.orchestra.run.smithi049.stdout:(90/131): python3-pyasn1-0.3.7-6.el8.noarch.rpm 2.5 MB/s | 126 kB 00:00 2023-10-16T12:08:12.528 INFO:teuthology.orchestra.run.smithi016.stdout:(105/131): python3-mako-1.0.6-13.el8.noarch.rpm 437 kB/s | 157 kB 00:00 2023-10-16T12:08:12.553 INFO:teuthology.orchestra.run.smithi049.stdout:(91/131): python3-jsonpointer-1.10-11.el8.noarc 398 kB/s | 20 kB 00:00 2023-10-16T12:08:12.554 INFO:teuthology.orchestra.run.smithi016.stdout:(106/131): oniguruma-6.8.2-2.el8.x86_64.rpm 681 kB/s | 187 kB 00:00 2023-10-16T12:08:12.570 INFO:teuthology.orchestra.run.smithi049.stdout:(92/131): python3-pyOpenSSL-19.0.0-1.el8.noarch 2.4 MB/s | 103 kB 00:00 2023-10-16T12:08:12.595 INFO:teuthology.orchestra.run.smithi016.stdout:(107/131): automake-1.16.1-7.el8.noarch.rpm 10 MB/s | 713 kB 00:00 2023-10-16T12:08:12.637 INFO:teuthology.orchestra.run.smithi049.stdout:(93/131): libtool-2.4.6-25.el8.x86_64.rpm 8.3 MB/s | 709 kB 00:00 2023-10-16T12:08:12.638 INFO:teuthology.orchestra.run.smithi016.stdout:(108/131): python3-jsonpatch-1.21-2.el8.noarch. 642 kB/s | 27 kB 00:00 2023-10-16T12:08:12.679 INFO:teuthology.orchestra.run.smithi016.stdout:(109/131): flex-2.6.1-9.el8.x86_64.rpm 7.5 MB/s | 320 kB 00:00 2023-10-16T12:08:12.704 INFO:teuthology.orchestra.run.smithi049.stdout:(94/131): python3-numpy-1.14.3-10.el8.x86_64.rp 4.9 MB/s | 3.7 MB 00:00 2023-10-16T12:08:12.713 INFO:teuthology.orchestra.run.smithi016.stdout:(110/131): python3-prettytable-0.7.2-14.el8.noa 1.3 MB/s | 44 kB 00:00 2023-10-16T12:08:12.729 INFO:teuthology.orchestra.run.smithi049.stdout:(95/131): libpq-13.5-1.el8.x86_64.rpm 2.1 MB/s | 198 kB 00:00 2023-10-16T12:08:12.771 INFO:teuthology.orchestra.run.smithi049.stdout:(96/131): jq-1.6-3.el8.x86_64.rpm 4.8 MB/s | 202 kB 00:00 2023-10-16T12:08:12.813 INFO:teuthology.orchestra.run.smithi049.stdout:(97/131): librdkafka-0.11.4-3.el8.x86_64.rpm 8.3 MB/s | 354 kB 00:00 2023-10-16T12:08:12.821 INFO:teuthology.orchestra.run.smithi016.stdout:(111/131): python3-babel-2.5.1-7.el8.noarch.rpm 18 MB/s | 4.8 MB 00:00 2023-10-16T12:08:12.872 INFO:teuthology.orchestra.run.smithi016.stdout:(112/131): python3-werkzeug-0.12.2-4.el8.noarch 9.0 MB/s | 457 kB 00:00 2023-10-16T12:08:12.896 INFO:teuthology.orchestra.run.smithi049.stdout:(98/131): openblas-threads-0.3.15-3.el8.x86_64. 15 MB/s | 4.9 MB 00:00 2023-10-16T12:08:12.914 INFO:teuthology.orchestra.run.smithi016.stdout:(113/131): python3-numpy-f2py-1.14.3-10.el8.x86 5.3 MB/s | 225 kB 00:00 2023-10-16T12:08:12.964 INFO:teuthology.orchestra.run.smithi016.stdout:(114/131): python3-numpy-1.14.3-10.el8.x86_64.r 15 MB/s | 3.7 MB 00:00 2023-10-16T12:08:12.989 INFO:teuthology.orchestra.run.smithi016.stdout:(115/131): uuid-1.6.2-43.el8.x86_64.rpm 852 kB/s | 64 kB 00:00 2023-10-16T12:08:12.997 INFO:teuthology.orchestra.run.smithi049.stdout:(99/131): openblas-0.3.15-3.el8.x86_64.rpm 16 MB/s | 4.7 MB 00:00 2023-10-16T12:08:13.014 INFO:teuthology.orchestra.run.smithi016.stdout:(116/131): python3-pyasn1-0.3.7-6.el8.noarch.rp 2.5 MB/s | 126 kB 00:00 2023-10-16T12:08:13.031 INFO:teuthology.orchestra.run.smithi016.stdout:(117/131): python3-jsonpointer-1.10-11.el8.noar 476 kB/s | 20 kB 00:00 2023-10-16T12:08:13.047 INFO:teuthology.orchestra.run.smithi049.stdout:(100/131): postgresql-contrib-10.21-2.module+el 5.2 MB/s | 808 kB 00:00 2023-10-16T12:08:13.057 INFO:teuthology.orchestra.run.smithi016.stdout:(118/131): python3-pyOpenSSL-19.0.0-1.el8.noarc 2.4 MB/s | 103 kB 00:00 2023-10-16T12:08:13.073 INFO:teuthology.orchestra.run.smithi049.stdout:(101/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 1.8 MB/s | 47 kB 00:00 2023-10-16T12:08:13.115 INFO:teuthology.orchestra.run.smithi049.stdout:(102/131): postgresql-10.21-2.module+el8.6.0+15 13 MB/s | 1.5 MB 00:00 2023-10-16T12:08:13.132 INFO:teuthology.orchestra.run.smithi049.stdout:(103/131): libselinux-devel-2.9-5.el8.x86_64.rp 3.3 MB/s | 200 kB 00:00 2023-10-16T12:08:13.132 INFO:teuthology.orchestra.run.smithi016.stdout:(119/131): libtool-2.4.6-25.el8.x86_64.rpm 6.9 MB/s | 709 kB 00:00 2023-10-16T12:08:13.149 INFO:teuthology.orchestra.run.smithi049.stdout:(104/131): python3-jwt-1.6.1-2.el8.noarch.rpm 1.2 MB/s | 43 kB 00:00 2023-10-16T12:08:13.174 INFO:teuthology.orchestra.run.smithi016.stdout:(120/131): libpq-13.5-1.el8.x86_64.rpm 4.6 MB/s | 198 kB 00:00 2023-10-16T12:08:13.175 INFO:teuthology.orchestra.run.smithi049.stdout:(105/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 5.1 MB/s | 220 kB 00:00 2023-10-16T12:08:13.199 INFO:teuthology.orchestra.run.smithi049.stdout:(106/131): keyutils-libs-devel-1.5.10-9.el8.x86 960 kB/s | 48 kB 00:00 2023-10-16T12:08:13.225 INFO:teuthology.orchestra.run.smithi049.stdout:(107/131): libsepol-devel-2.9-3.el8.x86_64.rpm 1.7 MB/s | 87 kB 00:00 2023-10-16T12:08:13.250 INFO:teuthology.orchestra.run.smithi049.stdout:(108/131): python3-requests-oauthlib-1.0.0-1.el 849 kB/s | 43 kB 00:00 2023-10-16T12:08:13.275 INFO:teuthology.orchestra.run.smithi049.stdout:(109/131): python3-pycparser-2.14-14.el8.noarch 2.1 MB/s | 109 kB 00:00 2023-10-16T12:08:13.300 INFO:teuthology.orchestra.run.smithi049.stdout:(110/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 4.4 MB/s | 229 kB 00:00 2023-10-16T12:08:13.333 INFO:teuthology.orchestra.run.smithi016.stdout:(121/131): python3-scipy-1.0.0-21.module+el8.5. 9.6 MB/s | 14 MB 00:01 2023-10-16T12:08:13.374 INFO:teuthology.orchestra.run.smithi016.stdout:(122/131): jq-1.6-3.el8.x86_64.rpm 4.8 MB/s | 202 kB 00:00 2023-10-16T12:08:13.376 INFO:teuthology.orchestra.run.smithi049.stdout:(111/131): postgresql-server-10.21-2.module+el8 9.0 MB/s | 5.1 MB 00:00 2023-10-16T12:08:13.401 INFO:teuthology.orchestra.run.smithi049.stdout:(112/131): libaio-devel-0.3.112-1.el8.x86_64.rp 186 kB/s | 19 kB 00:00 2023-10-16T12:08:13.425 INFO:teuthology.orchestra.run.smithi016.stdout:(123/131): librdkafka-0.11.4-3.el8.x86_64.rpm 6.9 MB/s | 354 kB 00:00 2023-10-16T12:08:13.426 INFO:teuthology.orchestra.run.smithi049.stdout:(113/131): python3-oauthlib-2.1.0-1.el8.noarch. 3.0 MB/s | 155 kB 00:00 2023-10-16T12:08:13.452 INFO:teuthology.orchestra.run.smithi049.stdout:(114/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 3.1 MB/s | 560 kB 00:00 2023-10-16T12:08:13.477 INFO:teuthology.orchestra.run.smithi049.stdout:(115/131): libverto-devel-0.3.0-5.el8.x86_64.rp 357 kB/s | 18 kB 00:00 2023-10-16T12:08:13.508 INFO:teuthology.orchestra.run.smithi016.stdout:(124/131): openblas-threads-0.3.15-3.el8.x86_64 11 MB/s | 4.9 MB 00:00 2023-10-16T12:08:13.510 INFO:teuthology.orchestra.run.smithi049.stdout:(116/131): python3-cryptography-3.2.1-5.el8.x86 5.0 MB/s | 559 kB 00:00 2023-10-16T12:08:13.544 INFO:teuthology.orchestra.run.smithi049.stdout:(117/131): pcre2-devel-10.32-2.el8.x86_64.rpm 6.4 MB/s | 605 kB 00:00 2023-10-16T12:08:13.567 INFO:teuthology.orchestra.run.smithi016.stdout:(125/131): postgresql-contrib-10.21-2.module+el 14 MB/s | 808 kB 00:00 2023-10-16T12:08:13.569 INFO:teuthology.orchestra.run.smithi049.stdout:(118/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 2.5 MB/s | 238 kB 00:00 2023-10-16T12:08:13.586 INFO:teuthology.orchestra.run.smithi049.stdout:(119/131): xfsprogs-devel-5.0.0-10.el8.x86_64.r 1.7 MB/s | 72 kB 00:00 2023-10-16T12:08:13.603 INFO:teuthology.orchestra.run.smithi049.stdout:(120/131): fuse-2.9.7-15.el8.x86_64.rpm 2.4 MB/s | 83 kB 00:00 2023-10-16T12:08:13.628 INFO:teuthology.orchestra.run.smithi049.stdout:(121/131): elfutils-libelf-devel-0.186-1.el8.x8 1.4 MB/s | 60 kB 00:00 2023-10-16T12:08:13.642 INFO:teuthology.orchestra.run.smithi016.stdout:(126/131): openblas-0.3.15-3.el8.x86_64.rpm 10 MB/s | 4.7 MB 00:00 2023-10-16T12:08:13.645 INFO:teuthology.orchestra.run.smithi049.stdout:(122/131): libuuid-devel-2.32.1-35.el8.x86_64.r 2.3 MB/s | 98 kB 00:00 2023-10-16T12:08:13.671 INFO:teuthology.orchestra.run.smithi049.stdout:(123/131): zlib-devel-1.2.11-18.el8_5.x86_64.rp 1.3 MB/s | 58 kB 00:00 2023-10-16T12:08:13.696 INFO:teuthology.orchestra.run.smithi049.stdout:(124/131): libcom_err-devel-1.45.6-4.el8.x86_64 770 kB/s | 39 kB 00:00 2023-10-16T12:08:13.701 INFO:teuthology.orchestra.run.smithi016.stdout:(127/131): postgresql-10.21-2.module+el8.6.0+15 11 MB/s | 1.5 MB 00:00 2023-10-16T12:08:13.746 INFO:teuthology.orchestra.run.smithi049.stdout:(125/131): libgfortran-8.5.0-10.1.el8_6.x86_64. 8.4 MB/s | 644 kB 00:00 2023-10-16T12:08:13.771 INFO:teuthology.orchestra.run.smithi049.stdout:(126/131): libquadmath-8.5.0-10.1.el8_6.x86_64. 2.2 MB/s | 171 kB 00:00 2023-10-16T12:08:13.785 INFO:teuthology.orchestra.run.smithi016.stdout:(128/131): postgresql-server-10.21-2.module+el8 14 MB/s | 5.1 MB 00:00 2023-10-16T12:08:13.797 INFO:teuthology.orchestra.run.smithi049.stdout:(127/131): NetworkManager-initscripts-updown-1. 2.7 MB/s | 138 kB 00:00 2023-10-16T12:08:13.851 INFO:teuthology.orchestra.run.smithi016.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 12 MB/s | 798 kB 00:00 2023-10-16T12:08:13.856 INFO:teuthology.orchestra.run.smithi049.stdout:(128/131): openssl-devel-1.1.1k-7.el8_6.x86_64. 6.7 MB/s | 2.3 MB 00:00 2023-10-16T12:08:13.906 INFO:teuthology.orchestra.run.smithi049.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 16 MB/s | 798 kB 00:00 2023-10-16T12:08:14.927 INFO:teuthology.orchestra.run.smithi016.stdout:(130/131): librbd1-18.0.0-6675.gc7471068.el8.x8 3.0 MB/s | 3.7 MB 00:01 2023-10-16T12:08:15.069 INFO:teuthology.orchestra.run.smithi016.stdout:(131/131): librados2-18.0.0-6675.gc7471068.el8. 2.6 MB/s | 3.7 MB 00:01 2023-10-16T12:08:15.073 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2023-10-16T12:08:15.074 INFO:teuthology.orchestra.run.smithi016.stdout:Total 9.2 MB/s | 209 MB 00:22 2023-10-16T12:08:15.356 INFO:teuthology.orchestra.run.smithi049.stdout:(130/131): librados2-18.0.0-6675.gc7471068.el8. 2.4 MB/s | 3.7 MB 00:01 2023-10-16T12:08:15.581 INFO:teuthology.orchestra.run.smithi049.stdout:(131/131): librbd1-18.0.0-6675.gc7471068.el8.x8 2.1 MB/s | 3.7 MB 00:01 2023-10-16T12:08:15.584 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2023-10-16T12:08:15.584 INFO:teuthology.orchestra.run.smithi049.stdout:Total 8.9 MB/s | 209 MB 00:23 2023-10-16T12:08:15.945 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2023-10-16T12:08:16.077 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2023-10-16T12:08:16.077 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2023-10-16T12:08:16.439 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2023-10-16T12:08:16.571 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2023-10-16T12:08:16.571 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2023-10-16T12:08:17.266 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2023-10-16T12:08:17.270 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2023-10-16T12:08:17.960 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2023-10-16T12:08:17.965 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2023-10-16T12:08:22.415 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2023-10-16T12:08:22.591 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-10-16T12:08:22.776 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-10-16T12:08:22.916 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2023-10-16T12:08:22.934 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 3/134 2023-10-16T12:08:23.051 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 3/134 2023-10-16T12:08:23.106 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-10-16T12:08:23.277 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-10-16T12:08:23.301 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-10-16T12:08:23.534 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 3/134 2023-10-16T12:08:23.700 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 4/134 2023-10-16T12:08:23.722 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-10-16T12:08:23.835 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 4/134 2023-10-16T12:08:23.967 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : librados2-2:18.0.0-6675.gc7471068.el8.x86_64 5/134 2023-10-16T12:08:24.033 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-10-16T12:08:24.389 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librados2-2:18.0.0-6675.gc7471068.el8.x86_64 5/134 2023-10-16T12:08:24.469 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-10-16T12:08:24.568 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : librbd1-2:18.0.0-6675.gc7471068.el8.x86_64 6/134 2023-10-16T12:08:24.683 INFO:teuthology.orchestra.run.smithi049.stdout: Upgrading : librados2-2:18.0.0-6675.gc7471068.el8.x86_64 6/134 2023-10-16T12:08:24.703 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librbd1-2:18.0.0-6675.gc7471068.el8.x86_64 6/134 2023-10-16T12:08:24.934 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libcephfs2-2:18.0.0-6675.gc7471068.el8.x86_64 7/134 2023-10-16T12:08:25.034 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libcephfs2-2:18.0.0-6675.gc7471068.el8.x86_64 7/134 2023-10-16T12:08:25.149 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librados2-2:18.0.0-6675.gc7471068.el8.x86_64 6/134 2023-10-16T12:08:25.197 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rados-2:18.0.0-6675.gc7471068.el8.x86_64 8/134 2023-10-16T12:08:25.342 INFO:teuthology.orchestra.run.smithi049.stdout: Upgrading : librbd1-2:18.0.0-6675.gc7471068.el8.x86_64 7/134 2023-10-16T12:08:25.394 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 9/134 2023-10-16T12:08:25.474 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librbd1-2:18.0.0-6675.gc7471068.el8.x86_64 7/134 2023-10-16T12:08:25.666 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libcephfs2-2:18.0.0-6675.gc7471068.el8.x86_64 8/134 2023-10-16T12:08:25.706 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-10-16T12:08:25.763 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: libcephfs2-2:18.0.0-6675.gc7471068.el8.x86_64 8/134 2023-10-16T12:08:25.931 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-10-16T12:08:25.951 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-rados-2:18.0.0-6675.gc7471068.el8.x86_64 9/134 2023-10-16T12:08:26.231 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-10-16T12:08:26.366 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-10-16T12:08:26.411 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-10-16T12:08:26.642 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-10-16T12:08:26.811 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-10-16T12:08:26.817 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libcephsqlite-2:18.0.0-6675.gc7471068.el8.x86_64 14/134 2023-10-16T12:08:26.937 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libcephsqlite-2:18.0.0-6675.gc7471068.el8.x86_64 14/134 2023-10-16T12:08:27.057 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-10-16T12:08:27.109 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libradosstriper1-2:18.0.0-6675.gc7471068.el8.x86 15/134 2023-10-16T12:08:27.224 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libcephsqlite-2:18.0.0-6675.gc7471068.el8.x86_64 14/134 2023-10-16T12:08:27.263 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libradosstriper1-2:18.0.0-6675.gc7471068.el8.x86 15/134 2023-10-16T12:08:27.333 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: libcephsqlite-2:18.0.0-6675.gc7471068.el8.x86_64 14/134 2023-10-16T12:08:27.447 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 16/134 2023-10-16T12:08:27.541 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libradosstriper1-2:18.0.0-6675.gc7471068.el8.x86 15/134 2023-10-16T12:08:27.606 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: libradosstriper1-2:18.0.0-6675.gc7471068.el8.x86 15/134 2023-10-16T12:08:27.625 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 17/134 2023-10-16T12:08:27.724 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-10-16T12:08:27.805 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 18/134 2023-10-16T12:08:27.867 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-10-16T12:08:27.950 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-10-16T12:08:28.007 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-10-16T12:08:28.100 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-10-16T12:08:28.234 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-10-16T12:08:28.653 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-10-16T12:08:28.824 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-10-16T12:08:28.896 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-10-16T12:08:29.075 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-10-16T12:08:29.444 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-10-16T12:08:29.615 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : openblas-0.3.15-3.el8.x86_64 19/134 2023-10-16T12:08:29.694 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-10-16T12:08:29.892 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : openblas-0.3.15-3.el8.x86_64 22/134 2023-10-16T12:08:30.366 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 19/134 2023-10-16T12:08:30.631 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 20/134 2023-10-16T12:08:30.648 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 22/134 2023-10-16T12:08:30.855 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 21/134 2023-10-16T12:08:30.898 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 23/134 2023-10-16T12:08:31.061 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 22/134 2023-10-16T12:08:31.085 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 24/134 2023-10-16T12:08:31.222 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 23/134 2023-10-16T12:08:31.313 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-10-16T12:08:31.416 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 24/134 2023-10-16T12:08:31.515 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-ceph-common-2:18.0.0-6675.gc7471068.el8. 26/134 2023-10-16T12:08:31.654 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-ceph-argparse-2:18.0.0-6675.gc7471068.el 27/134 2023-10-16T12:08:31.787 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-10-16T12:08:31.893 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cephfs-2:18.0.0-6675.gc7471068.el8.x86_6 28/134 2023-10-16T12:08:31.972 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-ceph-common-2:18.0.0-6675.gc7471068.el8. 26/134 2023-10-16T12:08:32.100 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-10-16T12:08:32.111 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-ceph-argparse-2:18.0.0-6675.gc7471068.el 27/134 2023-10-16T12:08:33.048 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cephfs-2:18.0.0-6675.gc7471068.el8.x86_6 28/134 2023-10-16T12:08:33.379 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 29/134 2023-10-16T12:08:33.629 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 30/134 2023-10-16T12:08:33.868 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 31/134 2023-10-16T12:08:33.975 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-10-16T12:08:34.057 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-10-16T12:08:35.050 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 30/134 2023-10-16T12:08:35.365 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 31/134 2023-10-16T12:08:35.657 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 32/134 2023-10-16T12:08:35.857 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 33/134 2023-10-16T12:08:35.927 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-10-16T12:08:36.081 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rsa-4.9-2.el8.noarch 34/134 2023-10-16T12:08:36.243 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 33/134 2023-10-16T12:08:36.279 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-10-16T12:08:36.445 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-rsa-4.9-2.el8.noarch 34/134 2023-10-16T12:08:36.498 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rbd-2:18.0.0-6675.gc7471068.el8.x86_64 36/134 2023-10-16T12:08:36.627 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-10-16T12:08:36.641 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librados-devel-2:18.0.0-6675.gc7471068.el8.x86_6 37/134 2023-10-16T12:08:36.796 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-rbd-2:18.0.0-6675.gc7471068.el8.x86_64 36/134 2023-10-16T12:08:36.800 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 38/134 2023-10-16T12:08:36.953 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 39/134 2023-10-16T12:08:36.985 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : librados-devel-2:18.0.0-6675.gc7471068.el8.x86_6 37/134 2023-10-16T12:08:37.124 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : uuid-1.6.2-43.el8.x86_64 40/134 2023-10-16T12:08:37.144 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 38/134 2023-10-16T12:08:37.200 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 40/134 2023-10-16T12:08:37.333 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 39/134 2023-10-16T12:08:37.468 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 41/134 2023-10-16T12:08:37.508 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : fuse-2.9.7-15.el8.x86_64 40/134 2023-10-16T12:08:37.696 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : automake-1.16.1-7.el8.noarch 42/134 2023-10-16T12:08:37.778 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 41/134 2023-10-16T12:08:37.841 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 43/134 2023-10-16T12:08:37.920 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 43/134 2023-10-16T12:08:37.992 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 42/134 2023-10-16T12:08:38.176 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : jq-1.6-3.el8.x86_64 44/134 2023-10-16T12:08:38.199 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 43/134 2023-10-16T12:08:38.362 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 45/134 2023-10-16T12:08:38.551 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 46/134 2023-10-16T12:08:38.583 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 44/134 2023-10-16T12:08:38.761 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 47/134 2023-10-16T12:08:38.869 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 45/134 2023-10-16T12:08:38.985 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : fuse-2.9.7-15.el8.x86_64 48/134 2023-10-16T12:08:39.059 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 46/134 2023-10-16T12:08:39.138 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 49/134 2023-10-16T12:08:39.282 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 47/134 2023-10-16T12:08:39.360 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 50/134 2023-10-16T12:08:39.584 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 51/134 2023-10-16T12:08:39.630 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 48/134 2023-10-16T12:08:39.926 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 49/134 2023-10-16T12:08:39.997 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 52/134 2023-10-16T12:08:40.164 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 50/134 2023-10-16T12:08:40.244 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 53/134 2023-10-16T12:08:40.374 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 51/134 2023-10-16T12:08:40.470 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 54/134 2023-10-16T12:08:40.597 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 52/134 2023-10-16T12:08:40.743 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 55/134 2023-10-16T12:08:40.762 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 53/134 2023-10-16T12:08:40.952 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 54/134 2023-10-16T12:08:40.972 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 56/134 2023-10-16T12:08:41.283 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 55/134 2023-10-16T12:08:41.309 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 57/134 2023-10-16T12:08:41.549 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 58/134 2023-10-16T12:08:41.559 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 56/134 2023-10-16T12:08:41.746 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 59/134 2023-10-16T12:08:41.788 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 57/134 2023-10-16T12:08:41.948 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 60/134 2023-10-16T12:08:42.012 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 58/134 2023-10-16T12:08:42.113 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 61/134 2023-10-16T12:08:42.179 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 59/134 2023-10-16T12:08:42.303 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 62/134 2023-10-16T12:08:42.357 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 60/134 2023-10-16T12:08:42.546 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : uuid-1.6.2-43.el8.x86_64 61/134 2023-10-16T12:08:42.617 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 61/134 2023-10-16T12:08:42.618 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 63/134 2023-10-16T12:08:42.872 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 62/134 2023-10-16T12:08:42.877 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 64/134 2023-10-16T12:08:43.101 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : automake-1.16.1-7.el8.noarch 63/134 2023-10-16T12:08:43.263 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 64/134 2023-10-16T12:08:43.267 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 65/134 2023-10-16T12:08:43.329 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 64/134 2023-10-16T12:08:43.492 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 66/134 2023-10-16T12:08:43.530 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : jq-1.6-3.el8.x86_64 65/134 2023-10-16T12:08:43.657 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-10-16T12:08:43.715 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 66/134 2023-10-16T12:08:43.820 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-10-16T12:08:43.881 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-10-16T12:08:44.078 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-10-16T12:08:44.100 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-10-16T12:08:44.306 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-10-16T12:08:44.359 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-10-16T12:08:44.453 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-10-16T12:08:44.588 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-10-16T12:08:44.610 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-10-16T12:08:44.752 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-10-16T12:08:44.833 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-10-16T12:08:44.951 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-10-16T12:08:45.046 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-10-16T12:08:45.151 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-10-16T12:08:45.229 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-10-16T12:08:45.361 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-10-16T12:08:45.376 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-10-16T12:08:45.522 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-10-16T12:08:45.552 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-10-16T12:08:45.708 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-10-16T12:08:45.775 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-10-16T12:08:45.853 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-10-16T12:08:45.961 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-10-16T12:08:46.056 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-10-16T12:08:46.121 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-10-16T12:08:46.291 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-10-16T12:08:46.360 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-10-16T12:08:46.486 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-10-16T12:08:46.590 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-10-16T12:08:46.750 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-10-16T12:08:46.750 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-10-16T12:08:46.959 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-10-16T12:08:46.996 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-10-16T12:08:47.189 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-10-16T12:08:47.423 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-10-16T12:08:48.104 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-10-16T12:08:48.488 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-10-16T12:08:48.606 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-10-16T12:08:48.716 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-10-16T12:08:48.885 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-10-16T12:08:49.000 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-10-16T12:08:49.285 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-10-16T12:08:49.474 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-10-16T12:08:49.483 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-10-16T12:08:49.784 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-10-16T12:08:50.057 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-10-16T12:08:50.253 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-10-16T12:08:50.341 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-10-16T12:08:50.531 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librgw2-2:18.0.0-6675.gc7471068.el8.x86_64 92/134 2023-10-16T12:08:50.631 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librgw2-2:18.0.0-6675.gc7471068.el8.x86_64 92/134 2023-10-16T12:08:50.864 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rgw-2:18.0.0-6675.gc7471068.el8.x86_64 93/134 2023-10-16T12:08:51.129 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-10-16T12:08:51.312 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : librgw2-2:18.0.0-6675.gc7471068.el8.x86_64 92/134 2023-10-16T12:08:51.388 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librgw2-2:18.0.0-6675.gc7471068.el8.x86_64 92/134 2023-10-16T12:08:51.546 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-rgw-2:18.0.0-6675.gc7471068.el8.x86_64 93/134 2023-10-16T12:08:54.833 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-common-2:18.0.0-6675.gc7471068.el8.x86_64 94/134 2023-10-16T12:08:55.031 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-common-2:18.0.0-6675.gc7471068.el8.x86_64 94/134 2023-10-16T12:08:55.654 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-common-2:18.0.0-6675.gc7471068.el8.x86_64 94/134 2023-10-16T12:08:55.655 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-common-2:18.0.0-6675.gc7471068.el8.x86_64 94/134 2023-10-16T12:08:55.878 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-common-2:18.0.0-6675.gc7471068.el8.x86_64 94/134 2023-10-16T12:08:55.918 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-base-2:18.0.0-6675.gc7471068.el8.x86_64 95/134 2023-10-16T12:08:56.005 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-base-2:18.0.0-6675.gc7471068.el8.x86_64 95/134 2023-10-16T12:08:56.122 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-selinux-2:18.0.0-6675.gc7471068.el8.x86_64 96/134 2023-10-16T12:08:56.513 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-common-2:18.0.0-6675.gc7471068.el8.x86_64 94/134 2023-10-16T12:08:56.790 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-base-2:18.0.0-6675.gc7471068.el8.x86_64 95/134 2023-10-16T12:08:56.874 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-base-2:18.0.0-6675.gc7471068.el8.x86_64 95/134 2023-10-16T12:08:57.020 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-selinux-2:18.0.0-6675.gc7471068.el8.x86_64 96/134 2023-10-16T12:09:15.829 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-selinux-2:18.0.0-6675.gc7471068.el8.x86_64 96/134 2023-10-16T12:09:15.829 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /sys 2023-10-16T12:09:15.830 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /proc 2023-10-16T12:09:15.830 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /dev 2023-10-16T12:09:15.830 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /run 2023-10-16T12:09:15.830 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /mnt 2023-10-16T12:09:15.830 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /var/tmp 2023-10-16T12:09:15.830 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /home 2023-10-16T12:09:15.831 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /tmp 2023-10-16T12:09:15.831 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /dev 2023-10-16T12:09:15.831 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:09:16.153 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-volume-2:18.0.0-6675.gc7471068.el8.noarch 97/134 2023-10-16T12:09:16.439 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-selinux-2:18.0.0-6675.gc7471068.el8.x86_64 96/134 2023-10-16T12:09:16.439 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /sys 2023-10-16T12:09:16.440 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /proc 2023-10-16T12:09:16.440 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /dev 2023-10-16T12:09:16.440 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /run 2023-10-16T12:09:16.440 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /mnt 2023-10-16T12:09:16.440 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /var/tmp 2023-10-16T12:09:16.441 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /home 2023-10-16T12:09:16.441 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /tmp 2023-10-16T12:09:16.441 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /dev 2023-10-16T12:09:16.441 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-16T12:09:16.783 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-volume-2:18.0.0-6675.gc7471068.el8.noarch 97/134 2023-10-16T12:09:17.845 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-volume-2:18.0.0-6675.gc7471068.el8.noarch 97/134 2023-10-16T12:09:18.015 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-osd-2:18.0.0-6675.gc7471068.el8.x86_64 98/134 2023-10-16T12:09:18.248 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-osd-2:18.0.0-6675.gc7471068.el8.x86_64 98/134 2023-10-16T12:09:18.386 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mds-2:18.0.0-6675.gc7471068.el8.x86_64 99/134 2023-10-16T12:09:18.472 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-volume-2:18.0.0-6675.gc7471068.el8.noarch 97/134 2023-10-16T12:09:18.653 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-osd-2:18.0.0-6675.gc7471068.el8.x86_64 98/134 2023-10-16T12:09:18.871 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mds-2:18.0.0-6675.gc7471068.el8.x86_64 99/134 2023-10-16T12:09:18.885 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-osd-2:18.0.0-6675.gc7471068.el8.x86_64 98/134 2023-10-16T12:09:19.033 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mds-2:18.0.0-6675.gc7471068.el8.x86_64 99/134 2023-10-16T12:09:19.077 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mon-2:18.0.0-6675.gc7471068.el8.x86_64 100/134 2023-10-16T12:09:19.095 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mon-2:18.0.0-6675.gc7471068.el8.x86_64 100/134 2023-10-16T12:09:19.516 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mds-2:18.0.0-6675.gc7471068.el8.x86_64 99/134 2023-10-16T12:09:19.691 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mon-2:18.0.0-6675.gc7471068.el8.x86_64 100/134 2023-10-16T12:09:19.709 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mon-2:18.0.0-6675.gc7471068.el8.x86_64 100/134 2023-10-16T12:09:20.858 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: cephadm-2:18.0.0-6675.gc7471068.el8.noarch 101/134 2023-10-16T12:09:21.059 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : cephadm-2:18.0.0-6675.gc7471068.el8.noarch 101/134 2023-10-16T12:09:21.192 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-6675.gc7471068.e 102/134 2023-10-16T12:09:21.387 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-6675.gc7471068. 103/134 2023-10-16T12:09:21.516 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: cephadm-2:18.0.0-6675.gc7471068.el8.noarch 101/134 2023-10-16T12:09:21.601 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-6675.gc7471068.el8.noa 104/134 2023-10-16T12:09:21.705 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : cephadm-2:18.0.0-6675.gc7471068.el8.noarch 101/134 2023-10-16T12:09:21.813 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-6675.gc7471068.e 102/134 2023-10-16T12:09:21.830 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-6675.gc7471068.el8.noa 104/134 2023-10-16T12:09:21.984 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-6675.gc7471068. 103/134 2023-10-16T12:09:22.157 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-6675.gc7471068.el8.noa 104/134 2023-10-16T12:09:22.256 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-6675.gc7471068.el8.n 105/134 2023-10-16T12:09:22.384 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-6675.gc7471068.el8.noa 104/134 2023-10-16T12:09:22.653 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-6675.gc7471068.el8.n 105/134 2023-10-16T12:09:23.284 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-6675.gc7471068.el8.n 105/134 2023-10-16T12:09:23.435 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-6675.gc74 106/134 2023-10-16T12:09:23.531 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-6675.gc74 106/134 2023-10-16T12:09:23.675 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-6675.gc7471068.el8.n 105/134 2023-10-16T12:09:23.848 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-6675.gc74 106/134 2023-10-16T12:09:23.908 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-6675.gc7471068.el 107/134 2023-10-16T12:09:23.944 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-6675.gc74 106/134 2023-10-16T12:09:24.122 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-2:18.0.0-6675.gc7471068.el8.x86_64 108/134 2023-10-16T12:09:24.160 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-2:18.0.0-6675.gc7471068.el8.x86_64 108/134 2023-10-16T12:09:24.326 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-rook-2:18.0.0-6675.gc7471068.el8.noarch 109/134 2023-10-16T12:09:24.338 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-6675.gc7471068.el 107/134 2023-10-16T12:09:24.338 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-6675.gc7471068.el8.noarch 109/134 2023-10-16T12:09:24.511 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mgr-2:18.0.0-6675.gc7471068.el8.x86_64 108/134 2023-10-16T12:09:24.548 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mgr-2:18.0.0-6675.gc7471068.el8.x86_64 108/134 2023-10-16T12:09:24.740 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mgr-rook-2:18.0.0-6675.gc7471068.el8.noarch 109/134 2023-10-16T12:09:24.751 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-6675.gc7471068.el8.noarch 109/134 2023-10-16T12:09:25.767 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-2:18.0.0-6675.gc7471068.el8.x86_64 110/134 2023-10-16T12:09:25.956 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-radosgw-2:18.0.0-6675.gc7471068.el8.x86_64 111/134 2023-10-16T12:09:25.990 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-6675.gc7471068.el8.x86_64 111/134 2023-10-16T12:09:26.143 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-6675.gc7471 112/134 2023-10-16T12:09:26.175 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-2:18.0.0-6675.gc7471068.el8.x86_64 110/134 2023-10-16T12:09:26.187 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-6675.gc7471 112/134 2023-10-16T12:09:26.343 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : cephfs-mirror-2:18.0.0-6675.gc7471068.el8.x86_64 113/134 2023-10-16T12:09:26.410 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-radosgw-2:18.0.0-6675.gc7471068.el8.x86_64 111/134 2023-10-16T12:09:26.445 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-6675.gc7471068.el8.x86_64 111/134 2023-10-16T12:09:26.582 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-6675.gc7471 112/134 2023-10-16T12:09:26.625 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-6675.gc7471 112/134 2023-10-16T12:09:26.729 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-6675.gc7471068.el8.x86_64 113/134 2023-10-16T12:09:26.789 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : cephfs-mirror-2:18.0.0-6675.gc7471068.el8.x86_64 113/134 2023-10-16T12:09:26.885 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : rbd-mirror-2:18.0.0-6675.gc7471068.el8.x86_64 114/134 2023-10-16T12:09:27.171 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-6675.gc7471068.el8.x86_64 113/134 2023-10-16T12:09:27.332 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : rbd-mirror-2:18.0.0-6675.gc7471068.el8.x86_64 114/134 2023-10-16T12:09:31.860 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: rbd-mirror-2:18.0.0-6675.gc7471068.el8.x86_64 114/134 2023-10-16T12:09:32.309 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: rbd-mirror-2:18.0.0-6675.gc7471068.el8.x86_64 114/134 2023-10-16T12:09:32.587 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-test-2:18.0.0-6675.gc7471068.el8.x86_64 115/134 2023-10-16T12:09:32.746 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libtool-2.4.6-25.el8.x86_64 116/134 2023-10-16T12:09:32.766 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-test-2:18.0.0-6675.gc7471068.el8.x86_64 115/134 2023-10-16T12:09:32.949 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 116/134 2023-10-16T12:09:33.076 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 116/134 2023-10-16T12:09:33.300 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-fuse-2:18.0.0-6675.gc7471068.el8.x86_64 117/134 2023-10-16T12:09:33.540 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 118/134 2023-10-16T12:09:33.692 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libtool-2.4.6-25.el8.x86_64 119/134 2023-10-16T12:09:33.883 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 117/134 2023-10-16T12:09:33.898 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 119/134 2023-10-16T12:09:34.138 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 120/134 2023-10-16T12:09:34.219 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 118/134 2023-10-16T12:09:34.376 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-fuse-2:18.0.0-6675.gc7471068.el8.x86_64 119/134 2023-10-16T12:09:34.446 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libcephfs-devel-2:18.0.0-6675.gc7471068.el8.x86_ 121/134 2023-10-16T12:09:34.551 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-10-16T12:09:34.593 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-10-16T12:09:34.726 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libcephfs-devel-2:18.0.0-6675.gc7471068.el8.x86_ 121/134 2023-10-16T12:09:34.898 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-10-16T12:09:36.931 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-10-16T12:09:37.175 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-10-16T12:09:37.193 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-10-16T12:09:37.275 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-10-16T12:09:37.384 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : cephfs-top-2:18.0.0-6675.gc7471068.el8.noarch 124/134 2023-10-16T12:09:37.496 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-10-16T12:09:37.513 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-10-16T12:09:37.697 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : cephfs-top-2:18.0.0-6675.gc7471068.el8.noarch 124/134 2023-10-16T12:09:37.783 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : rbd-fuse-2:18.0.0-6675.gc7471068.el8.x86_64 125/134 2023-10-16T12:09:37.971 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : rbd-fuse-2:18.0.0-6675.gc7471068.el8.x86_64 125/134 2023-10-16T12:09:38.068 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : rbd-nbd-2:18.0.0-6675.gc7471068.el8.x86_64 126/134 2023-10-16T12:09:38.255 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : rbd-nbd-2:18.0.0-6675.gc7471068.el8.x86_64 126/134 2023-10-16T12:09:38.454 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-10-16T12:09:38.540 INFO:teuthology.orchestra.run.smithi049.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-10-16T12:09:38.616 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : flex-2.6.1-9.el8.x86_64 128/134 2023-10-16T12:09:38.698 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 128/134 2023-10-16T12:09:38.704 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-10-16T12:09:38.714 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-10-16T12:09:38.866 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : bison-3.0.4-10.el8.x86_64 129/134 2023-10-16T12:09:38.879 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 129/134 2023-10-16T12:09:38.944 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 129/134 2023-10-16T12:09:39.074 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-10-16T12:09:39.083 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-10-16T12:09:39.137 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : flex-2.6.1-9.el8.x86_64 130/134 2023-10-16T12:09:39.218 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 131/134 2023-10-16T12:09:39.219 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-10-16T12:09:39.224 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 130/134 2023-10-16T12:09:39.403 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-10-16T12:09:39.403 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-10-16T12:09:39.437 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : bison-3.0.4-10.el8.x86_64 131/134 2023-10-16T12:09:39.451 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 131/134 2023-10-16T12:09:39.451 INFO:teuthology.orchestra.run.smithi049.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-10-16T12:09:39.614 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-10-16T12:09:39.649 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-10-16T12:09:39.649 INFO:teuthology.orchestra.run.smithi049.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-10-16T12:09:39.782 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-10-16T12:09:39.869 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-10-16T12:09:40.045 INFO:teuthology.orchestra.run.smithi049.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-10-16T12:09:40.362 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-10-16T12:09:40.363 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-2:18.0.0-6675.gc7471068.el8.x86_64 1/134 2023-10-16T12:09:40.363 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-base-2:18.0.0-6675.gc7471068.el8.x86_64 2/134 2023-10-16T12:09:40.363 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-common-2:18.0.0-6675.gc7471068.el8.x86_64 3/134 2023-10-16T12:09:40.363 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-fuse-2:18.0.0-6675.gc7471068.el8.x86_64 4/134 2023-10-16T12:09:40.363 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-6675.gc7471 5/134 2023-10-16T12:09:40.364 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mds-2:18.0.0-6675.gc7471068.el8.x86_64 6/134 2023-10-16T12:09:40.364 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-2:18.0.0-6675.gc7471068.el8.x86_64 7/134 2023-10-16T12:09:40.364 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mon-2:18.0.0-6675.gc7471068.el8.x86_64 8/134 2023-10-16T12:09:40.364 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-osd-2:18.0.0-6675.gc7471068.el8.x86_64 9/134 2023-10-16T12:09:40.364 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-radosgw-2:18.0.0-6675.gc7471068.el8.x86_64 10/134 2023-10-16T12:09:40.365 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-selinux-2:18.0.0-6675.gc7471068.el8.x86_64 11/134 2023-10-16T12:09:40.365 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-test-2:18.0.0-6675.gc7471068.el8.x86_64 12/134 2023-10-16T12:09:40.365 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : cephfs-mirror-2:18.0.0-6675.gc7471068.el8.x86_64 13/134 2023-10-16T12:09:40.365 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libcephfs-devel-2:18.0.0-6675.gc7471068.el8.x86_ 14/134 2023-10-16T12:09:40.365 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libcephfs2-2:18.0.0-6675.gc7471068.el8.x86_64 15/134 2023-10-16T12:09:40.366 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libcephsqlite-2:18.0.0-6675.gc7471068.el8.x86_64 16/134 2023-10-16T12:09:40.366 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librados-devel-2:18.0.0-6675.gc7471068.el8.x86_6 17/134 2023-10-16T12:09:40.366 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libradosstriper1-2:18.0.0-6675.gc7471068.el8.x86 18/134 2023-10-16T12:09:40.366 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librgw2-2:18.0.0-6675.gc7471068.el8.x86_64 19/134 2023-10-16T12:09:40.367 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-ceph-argparse-2:18.0.0-6675.gc7471068.el 20/134 2023-10-16T12:09:40.367 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-ceph-common-2:18.0.0-6675.gc7471068.el8. 21/134 2023-10-16T12:09:40.367 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cephfs-2:18.0.0-6675.gc7471068.el8.x86_6 22/134 2023-10-16T12:09:40.367 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rados-2:18.0.0-6675.gc7471068.el8.x86_64 23/134 2023-10-16T12:09:40.367 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rbd-2:18.0.0-6675.gc7471068.el8.x86_64 24/134 2023-10-16T12:09:40.368 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rgw-2:18.0.0-6675.gc7471068.el8.x86_64 25/134 2023-10-16T12:09:40.368 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : rbd-fuse-2:18.0.0-6675.gc7471068.el8.x86_64 26/134 2023-10-16T12:09:40.368 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : rbd-mirror-2:18.0.0-6675.gc7471068.el8.x86_64 27/134 2023-10-16T12:09:40.368 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : rbd-nbd-2:18.0.0-6675.gc7471068.el8.x86_64 28/134 2023-10-16T12:09:40.368 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-6675.gc7471068. 29/134 2023-10-16T12:09:40.369 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-6675.gc7471068.el8.noa 30/134 2023-10-16T12:09:40.369 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-6675.gc7471068.el8.n 31/134 2023-10-16T12:09:40.369 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-6675.gc74 32/134 2023-10-16T12:09:40.369 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-6675.gc7471068.el 33/134 2023-10-16T12:09:40.369 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-rook-2:18.0.0-6675.gc7471068.el8.noarch 34/134 2023-10-16T12:09:40.370 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-6675.gc7471068.e 35/134 2023-10-16T12:09:40.370 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-volume-2:18.0.0-6675.gc7471068.el8.noarch 36/134 2023-10-16T12:09:40.370 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : cephadm-2:18.0.0-6675.gc7471068.el8.noarch 37/134 2023-10-16T12:09:40.370 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : cephfs-top-2:18.0.0-6675.gc7471068.el8.noarch 38/134 2023-10-16T12:09:40.370 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-10-16T12:09:40.371 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-10-16T12:09:40.371 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-10-16T12:09:40.371 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-10-16T12:09:40.371 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-10-16T12:09:40.371 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-10-16T12:09:40.371 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-10-16T12:09:40.372 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-10-16T12:09:40.372 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-10-16T12:09:40.372 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-10-16T12:09:40.372 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-10-16T12:09:40.372 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-10-16T12:09:40.372 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-10-16T12:09:40.373 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-10-16T12:09:40.373 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-10-16T12:09:40.373 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-10-16T12:09:40.373 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-10-16T12:09:40.373 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-10-16T12:09:40.374 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-10-16T12:09:40.374 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-10-16T12:09:40.374 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-10-16T12:09:40.374 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 60/134 2023-10-16T12:09:40.374 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-10-16T12:09:40.375 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-10-16T12:09:40.375 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-10-16T12:09:40.375 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-10-16T12:09:40.375 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-10-16T12:09:40.375 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-10-16T12:09:40.376 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-10-16T12:09:40.376 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-10-16T12:09:40.376 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-10-16T12:09:40.376 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-10-16T12:09:40.376 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-10-16T12:09:40.376 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 72/134 2023-10-16T12:09:40.377 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 73/134 2023-10-16T12:09:40.377 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 74/134 2023-10-16T12:09:40.377 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 75/134 2023-10-16T12:09:40.377 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 76/134 2023-10-16T12:09:40.377 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 77/134 2023-10-16T12:09:40.378 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 78/134 2023-10-16T12:09:40.378 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 79/134 2023-10-16T12:09:40.378 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 80/134 2023-10-16T12:09:40.378 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 81/134 2023-10-16T12:09:40.378 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 82/134 2023-10-16T12:09:40.378 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 83/134 2023-10-16T12:09:40.379 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 84/134 2023-10-16T12:09:40.379 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 85/134 2023-10-16T12:09:40.379 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 86/134 2023-10-16T12:09:40.379 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 87/134 2023-10-16T12:09:40.379 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 88/134 2023-10-16T12:09:40.380 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 89/134 2023-10-16T12:09:40.380 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 90/134 2023-10-16T12:09:40.380 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 91/134 2023-10-16T12:09:40.380 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 92/134 2023-10-16T12:09:40.380 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 93/134 2023-10-16T12:09:40.381 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 94/134 2023-10-16T12:09:40.381 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 95/134 2023-10-16T12:09:40.381 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 96/134 2023-10-16T12:09:40.381 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 97/134 2023-10-16T12:09:40.381 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 98/134 2023-10-16T12:09:40.382 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 99/134 2023-10-16T12:09:40.382 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 100/134 2023-10-16T12:09:40.382 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 101/134 2023-10-16T12:09:40.382 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 102/134 2023-10-16T12:09:40.382 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 103/134 2023-10-16T12:09:40.383 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : bison-3.0.4-10.el8.x86_64 104/134 2023-10-16T12:09:40.383 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 105/134 2023-10-16T12:09:40.383 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 106/134 2023-10-16T12:09:40.383 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 107/134 2023-10-16T12:09:40.383 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : automake-1.16.1-7.el8.noarch 108/134 2023-10-16T12:09:40.384 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 109/134 2023-10-16T12:09:40.384 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 110/134 2023-10-16T12:09:40.384 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : flex-2.6.1-9.el8.x86_64 111/134 2023-10-16T12:09:40.384 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 112/134 2023-10-16T12:09:40.384 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 113/134 2023-10-16T12:09:40.385 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 114/134 2023-10-16T12:09:40.385 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 115/134 2023-10-16T12:09:40.385 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 116/134 2023-10-16T12:09:40.385 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 117/134 2023-10-16T12:09:40.385 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 118/134 2023-10-16T12:09:40.386 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 119/134 2023-10-16T12:09:40.386 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 120/134 2023-10-16T12:09:40.386 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 121/134 2023-10-16T12:09:40.386 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libpq-13.5-1.el8.x86_64 122/134 2023-10-16T12:09:40.386 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 123/134 2023-10-16T12:09:40.387 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : jq-1.6-3.el8.x86_64 124/134 2023-10-16T12:09:40.387 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 125/134 2023-10-16T12:09:40.387 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 126/134 2023-10-16T12:09:40.387 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 127/134 2023-10-16T12:09:40.387 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 128/134 2023-10-16T12:09:40.387 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librados2-2:18.0.0-6675.gc7471068.el8.x86_64 129/134 2023-10-16T12:09:40.388 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-10-16T12:09:40.388 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librbd1-2:18.0.0-6675.gc7471068.el8.x86_64 131/134 2023-10-16T12:09:40.388 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-10-16T12:09:40.388 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-10-16T12:09:40.620 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-10-16T12:09:40.620 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-2:18.0.0-6675.gc7471068.el8.x86_64 1/134 2023-10-16T12:09:40.620 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-base-2:18.0.0-6675.gc7471068.el8.x86_64 2/134 2023-10-16T12:09:40.620 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-common-2:18.0.0-6675.gc7471068.el8.x86_64 3/134 2023-10-16T12:09:40.621 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-fuse-2:18.0.0-6675.gc7471068.el8.x86_64 4/134 2023-10-16T12:09:40.621 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-6675.gc7471 5/134 2023-10-16T12:09:40.621 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mds-2:18.0.0-6675.gc7471068.el8.x86_64 6/134 2023-10-16T12:09:40.621 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mgr-2:18.0.0-6675.gc7471068.el8.x86_64 7/134 2023-10-16T12:09:40.621 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mon-2:18.0.0-6675.gc7471068.el8.x86_64 8/134 2023-10-16T12:09:40.621 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-osd-2:18.0.0-6675.gc7471068.el8.x86_64 9/134 2023-10-16T12:09:40.622 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-radosgw-2:18.0.0-6675.gc7471068.el8.x86_64 10/134 2023-10-16T12:09:40.622 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-selinux-2:18.0.0-6675.gc7471068.el8.x86_64 11/134 2023-10-16T12:09:40.622 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-test-2:18.0.0-6675.gc7471068.el8.x86_64 12/134 2023-10-16T12:09:40.622 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : cephfs-mirror-2:18.0.0-6675.gc7471068.el8.x86_64 13/134 2023-10-16T12:09:40.622 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libcephfs-devel-2:18.0.0-6675.gc7471068.el8.x86_ 14/134 2023-10-16T12:09:40.623 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libcephfs2-2:18.0.0-6675.gc7471068.el8.x86_64 15/134 2023-10-16T12:09:40.623 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libcephsqlite-2:18.0.0-6675.gc7471068.el8.x86_64 16/134 2023-10-16T12:09:40.623 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librados-devel-2:18.0.0-6675.gc7471068.el8.x86_6 17/134 2023-10-16T12:09:40.623 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libradosstriper1-2:18.0.0-6675.gc7471068.el8.x86 18/134 2023-10-16T12:09:40.623 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librgw2-2:18.0.0-6675.gc7471068.el8.x86_64 19/134 2023-10-16T12:09:40.624 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-ceph-argparse-2:18.0.0-6675.gc7471068.el 20/134 2023-10-16T12:09:40.624 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-ceph-common-2:18.0.0-6675.gc7471068.el8. 21/134 2023-10-16T12:09:40.624 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cephfs-2:18.0.0-6675.gc7471068.el8.x86_6 22/134 2023-10-16T12:09:40.624 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-rados-2:18.0.0-6675.gc7471068.el8.x86_64 23/134 2023-10-16T12:09:40.624 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-rbd-2:18.0.0-6675.gc7471068.el8.x86_64 24/134 2023-10-16T12:09:40.625 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-rgw-2:18.0.0-6675.gc7471068.el8.x86_64 25/134 2023-10-16T12:09:40.625 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : rbd-fuse-2:18.0.0-6675.gc7471068.el8.x86_64 26/134 2023-10-16T12:09:40.625 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : rbd-mirror-2:18.0.0-6675.gc7471068.el8.x86_64 27/134 2023-10-16T12:09:40.625 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : rbd-nbd-2:18.0.0-6675.gc7471068.el8.x86_64 28/134 2023-10-16T12:09:40.625 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-6675.gc7471068. 29/134 2023-10-16T12:09:40.626 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-6675.gc7471068.el8.noa 30/134 2023-10-16T12:09:40.626 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-6675.gc7471068.el8.n 31/134 2023-10-16T12:09:40.626 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-6675.gc74 32/134 2023-10-16T12:09:40.626 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-6675.gc7471068.el 33/134 2023-10-16T12:09:40.626 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mgr-rook-2:18.0.0-6675.gc7471068.el8.noarch 34/134 2023-10-16T12:09:40.627 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-6675.gc7471068.e 35/134 2023-10-16T12:09:40.627 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-volume-2:18.0.0-6675.gc7471068.el8.noarch 36/134 2023-10-16T12:09:40.627 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : cephadm-2:18.0.0-6675.gc7471068.el8.noarch 37/134 2023-10-16T12:09:40.627 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : cephfs-top-2:18.0.0-6675.gc7471068.el8.noarch 38/134 2023-10-16T12:09:40.627 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-10-16T12:09:40.627 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-10-16T12:09:40.628 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-10-16T12:09:40.628 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-10-16T12:09:40.628 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-10-16T12:09:40.628 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-10-16T12:09:40.628 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-10-16T12:09:40.628 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-10-16T12:09:40.629 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-10-16T12:09:40.629 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-10-16T12:09:40.629 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-10-16T12:09:40.629 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-10-16T12:09:40.629 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-10-16T12:09:40.630 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-10-16T12:09:40.630 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-10-16T12:09:40.630 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-10-16T12:09:40.630 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-10-16T12:09:40.630 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-10-16T12:09:40.631 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-10-16T12:09:40.631 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-10-16T12:09:40.631 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-10-16T12:09:40.631 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 60/134 2023-10-16T12:09:40.631 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-10-16T12:09:40.632 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-10-16T12:09:40.632 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-10-16T12:09:40.632 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-10-16T12:09:40.632 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-10-16T12:09:40.632 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-10-16T12:09:40.632 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-10-16T12:09:40.633 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-10-16T12:09:40.633 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-10-16T12:09:40.633 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-10-16T12:09:40.633 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-10-16T12:09:40.633 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 72/134 2023-10-16T12:09:40.634 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 73/134 2023-10-16T12:09:40.634 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 74/134 2023-10-16T12:09:40.634 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 75/134 2023-10-16T12:09:40.634 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 76/134 2023-10-16T12:09:40.634 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 77/134 2023-10-16T12:09:40.635 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : bison-3.0.4-10.el8.x86_64 78/134 2023-10-16T12:09:40.635 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 79/134 2023-10-16T12:09:40.635 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 80/134 2023-10-16T12:09:40.635 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 81/134 2023-10-16T12:09:40.635 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : automake-1.16.1-7.el8.noarch 82/134 2023-10-16T12:09:40.635 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 83/134 2023-10-16T12:09:40.636 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 84/134 2023-10-16T12:09:40.636 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : flex-2.6.1-9.el8.x86_64 85/134 2023-10-16T12:09:40.636 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 86/134 2023-10-16T12:09:40.636 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 87/134 2023-10-16T12:09:40.636 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 88/134 2023-10-16T12:09:40.636 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 89/134 2023-10-16T12:09:40.637 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 90/134 2023-10-16T12:09:40.637 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 91/134 2023-10-16T12:09:40.637 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 92/134 2023-10-16T12:09:40.637 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 93/134 2023-10-16T12:09:40.637 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 94/134 2023-10-16T12:09:40.638 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 95/134 2023-10-16T12:09:40.638 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libpq-13.5-1.el8.x86_64 96/134 2023-10-16T12:09:40.638 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 97/134 2023-10-16T12:09:40.638 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : jq-1.6-3.el8.x86_64 98/134 2023-10-16T12:09:40.638 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 99/134 2023-10-16T12:09:40.638 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 100/134 2023-10-16T12:09:40.639 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 101/134 2023-10-16T12:09:40.639 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 102/134 2023-10-16T12:09:40.639 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 103/134 2023-10-16T12:09:40.639 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 104/134 2023-10-16T12:09:40.639 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 105/134 2023-10-16T12:09:40.640 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 106/134 2023-10-16T12:09:40.640 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 107/134 2023-10-16T12:09:40.640 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 108/134 2023-10-16T12:09:40.640 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 109/134 2023-10-16T12:09:40.640 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 110/134 2023-10-16T12:09:40.641 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 111/134 2023-10-16T12:09:40.641 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 112/134 2023-10-16T12:09:40.641 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 113/134 2023-10-16T12:09:40.641 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 114/134 2023-10-16T12:09:40.641 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 115/134 2023-10-16T12:09:40.641 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 116/134 2023-10-16T12:09:40.642 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 117/134 2023-10-16T12:09:40.642 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 118/134 2023-10-16T12:09:40.642 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 119/134 2023-10-16T12:09:40.642 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-10-16T12:09:40.642 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 121/134 2023-10-16T12:09:40.643 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-10-16T12:09:40.643 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 123/134 2023-10-16T12:09:40.643 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 124/134 2023-10-16T12:09:40.643 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 125/134 2023-10-16T12:09:40.643 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 126/134 2023-10-16T12:09:40.643 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 127/134 2023-10-16T12:09:40.644 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-10-16T12:09:40.644 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librados2-2:18.0.0-6675.gc7471068.el8.x86_64 129/134 2023-10-16T12:09:40.644 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-10-16T12:09:40.644 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librbd1-2:18.0.0-6675.gc7471068.el8.x86_64 131/134 2023-10-16T12:09:40.644 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-10-16T12:09:40.644 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-10-16T12:09:43.052 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-10-16T12:09:43.053 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:02:04 ago on Mon 16 Oct 2023 12:07:39 PM UTC. 2023-10-16T12:09:43.282 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-10-16T12:09:43.282 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:04 ago on Mon 16 Oct 2023 12:07:39 PM UTC. 2023-10-16T12:09:45.873 INFO:teuthology.orchestra.run.smithi016.stdout:Installed products updated. 2023-10-16T12:09:46.024 INFO:teuthology.orchestra.run.smithi049.stdout:Installed products updated. 2023-10-16T12:09:46.107 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:09:46.108 INFO:teuthology.orchestra.run.smithi016.stdout:Upgraded: 2023-10-16T12:09:46.108 INFO:teuthology.orchestra.run.smithi016.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-10-16T12:09:46.108 INFO:teuthology.orchestra.run.smithi016.stdout: librados2-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T12:09:46.108 INFO:teuthology.orchestra.run.smithi016.stdout: librbd1-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T12:09:46.108 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2023-10-16T12:09:46.109 INFO:teuthology.orchestra.run.smithi016.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-10-16T12:09:46.109 INFO:teuthology.orchestra.run.smithi016.stdout: automake-1.16.1-7.el8.noarch 2023-10-16T12:09:46.109 INFO:teuthology.orchestra.run.smithi016.stdout: bison-3.0.4-10.el8.x86_64 2023-10-16T12:09:46.109 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T12:09:46.109 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-base-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T12:09:46.110 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-common-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T12:09:46.110 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-fuse-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T12:09:46.110 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-grafana-dashboards-2:18.0.0-6675.gc7471068.el8.noarch 2023-10-16T12:09:46.110 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-immutable-object-cache-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T12:09:46.110 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mds-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T12:09:46.111 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T12:09:46.111 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-cephadm-2:18.0.0-6675.gc7471068.el8.noarch 2023-10-16T12:09:46.111 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-dashboard-2:18.0.0-6675.gc7471068.el8.noarch 2023-10-16T12:09:46.111 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-diskprediction-local-2:18.0.0-6675.gc7471068.el8.noarch 2023-10-16T12:09:46.111 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-modules-core-2:18.0.0-6675.gc7471068.el8.noarch 2023-10-16T12:09:46.111 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-rook-2:18.0.0-6675.gc7471068.el8.noarch 2023-10-16T12:09:46.112 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mon-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T12:09:46.112 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-osd-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T12:09:46.112 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-prometheus-alerts-2:18.0.0-6675.gc7471068.el8.noarch 2023-10-16T12:09:46.112 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-radosgw-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T12:09:46.112 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-selinux-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T12:09:46.113 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-test-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T12:09:46.113 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-volume-2:18.0.0-6675.gc7471068.el8.noarch 2023-10-16T12:09:46.113 INFO:teuthology.orchestra.run.smithi016.stdout: cephadm-2:18.0.0-6675.gc7471068.el8.noarch 2023-10-16T12:09:46.113 INFO:teuthology.orchestra.run.smithi016.stdout: cephfs-mirror-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T12:09:46.113 INFO:teuthology.orchestra.run.smithi016.stdout: cephfs-top-2:18.0.0-6675.gc7471068.el8.noarch 2023-10-16T12:09:46.113 INFO:teuthology.orchestra.run.smithi016.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-10-16T12:09:46.114 INFO:teuthology.orchestra.run.smithi016.stdout: flex-2.6.1-9.el8.x86_64 2023-10-16T12:09:46.114 INFO:teuthology.orchestra.run.smithi016.stdout: fuse-2.9.7-15.el8.x86_64 2023-10-16T12:09:46.114 INFO:teuthology.orchestra.run.smithi016.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-10-16T12:09:46.114 INFO:teuthology.orchestra.run.smithi016.stdout: jq-1.6-3.el8.x86_64 2023-10-16T12:09:46.114 INFO:teuthology.orchestra.run.smithi016.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-10-16T12:09:46.114 INFO:teuthology.orchestra.run.smithi016.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-10-16T12:09:46.115 INFO:teuthology.orchestra.run.smithi016.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-10-16T12:09:46.115 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs-devel-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T12:09:46.115 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs2-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T12:09:46.115 INFO:teuthology.orchestra.run.smithi016.stdout: libcephsqlite-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T12:09:46.115 INFO:teuthology.orchestra.run.smithi016.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-10-16T12:09:46.116 INFO:teuthology.orchestra.run.smithi016.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-10-16T12:09:46.116 INFO:teuthology.orchestra.run.smithi016.stdout: liboath-2.6.2-3.el8.x86_64 2023-10-16T12:09:46.116 INFO:teuthology.orchestra.run.smithi016.stdout: libpq-13.5-1.el8.x86_64 2023-10-16T12:09:46.116 INFO:teuthology.orchestra.run.smithi016.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-10-16T12:09:46.116 INFO:teuthology.orchestra.run.smithi016.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-10-16T12:09:46.117 INFO:teuthology.orchestra.run.smithi016.stdout: librados-devel-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T12:09:46.117 INFO:teuthology.orchestra.run.smithi016.stdout: libradosstriper1-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T12:09:46.117 INFO:teuthology.orchestra.run.smithi016.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-10-16T12:09:46.117 INFO:teuthology.orchestra.run.smithi016.stdout: librgw2-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T12:09:46.117 INFO:teuthology.orchestra.run.smithi016.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-10-16T12:09:46.117 INFO:teuthology.orchestra.run.smithi016.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-10-16T12:09:46.118 INFO:teuthology.orchestra.run.smithi016.stdout: libtool-2.4.6-25.el8.x86_64 2023-10-16T12:09:46.118 INFO:teuthology.orchestra.run.smithi016.stdout: libunwind-1.3.1-3.el8.x86_64 2023-10-16T12:09:46.118 INFO:teuthology.orchestra.run.smithi016.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-10-16T12:09:46.118 INFO:teuthology.orchestra.run.smithi016.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-10-16T12:09:46.118 INFO:teuthology.orchestra.run.smithi016.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-10-16T12:09:46.119 INFO:teuthology.orchestra.run.smithi016.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-10-16T12:09:46.119 INFO:teuthology.orchestra.run.smithi016.stdout: openblas-0.3.15-3.el8.x86_64 2023-10-16T12:09:46.119 INFO:teuthology.orchestra.run.smithi016.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-10-16T12:09:46.119 INFO:teuthology.orchestra.run.smithi016.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-10-16T12:09:46.119 INFO:teuthology.orchestra.run.smithi016.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-10-16T12:09:46.119 INFO:teuthology.orchestra.run.smithi016.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-10-16T12:09:46.120 INFO:teuthology.orchestra.run.smithi016.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-10-16T12:09:46.120 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-10-16T12:09:46.120 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-10-16T12:09:46.120 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-10-16T12:09:46.120 INFO:teuthology.orchestra.run.smithi016.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-10-16T12:09:46.121 INFO:teuthology.orchestra.run.smithi016.stdout: python3-babel-2.5.1-7.el8.noarch 2023-10-16T12:09:46.121 INFO:teuthology.orchestra.run.smithi016.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-10-16T12:09:46.121 INFO:teuthology.orchestra.run.smithi016.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-10-16T12:09:46.121 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-10-16T12:09:46.121 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-argparse-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T12:09:46.121 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-common-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T12:09:46.122 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cephfs-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T12:09:46.122 INFO:teuthology.orchestra.run.smithi016.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-10-16T12:09:46.122 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-10-16T12:09:46.122 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-10-16T12:09:46.122 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-10-16T12:09:46.123 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-10-16T12:09:46.123 INFO:teuthology.orchestra.run.smithi016.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-10-16T12:09:46.123 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-6.2-6.el8.noarch 2023-10-16T12:09:46.123 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-10-16T12:09:46.123 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-10-16T12:09:46.124 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-10-16T12:09:46.124 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-10-16T12:09:46.124 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-10-16T12:09:46.124 INFO:teuthology.orchestra.run.smithi016.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-10-16T12:09:46.124 INFO:teuthology.orchestra.run.smithi016.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-10-16T12:09:46.125 INFO:teuthology.orchestra.run.smithi016.stdout: python3-mako-1.0.6-13.el8.noarch 2023-10-16T12:09:46.125 INFO:teuthology.orchestra.run.smithi016.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-10-16T12:09:46.125 INFO:teuthology.orchestra.run.smithi016.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-10-16T12:09:46.125 INFO:teuthology.orchestra.run.smithi016.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-10-16T12:09:46.125 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-10-16T12:09:46.126 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-10-16T12:09:46.126 INFO:teuthology.orchestra.run.smithi016.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-10-16T12:09:46.126 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-10-16T12:09:46.126 INFO:teuthology.orchestra.run.smithi016.stdout: python3-portend-2.6-1.el8.noarch 2023-10-16T12:09:46.126 INFO:teuthology.orchestra.run.smithi016.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-10-16T12:09:46.127 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-10-16T12:09:46.127 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-10-16T12:09:46.127 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-10-16T12:09:46.127 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pycparser-2.14-14.el8.noarch 2023-10-16T12:09:46.127 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pytz-2017.2-9.el8.noarch 2023-10-16T12:09:46.128 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rados-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T12:09:46.128 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rbd-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T12:09:46.128 INFO:teuthology.orchestra.run.smithi016.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-10-16T12:09:46.128 INFO:teuthology.orchestra.run.smithi016.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-10-16T12:09:46.128 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rgw-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T12:09:46.128 INFO:teuthology.orchestra.run.smithi016.stdout: python3-routes-2.4.1-12.el8.noarch 2023-10-16T12:09:46.129 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rsa-4.9-2.el8.noarch 2023-10-16T12:09:46.129 INFO:teuthology.orchestra.run.smithi016.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-10-16T12:09:46.129 INFO:teuthology.orchestra.run.smithi016.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-10-16T12:09:46.129 INFO:teuthology.orchestra.run.smithi016.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-10-16T12:09:46.129 INFO:teuthology.orchestra.run.smithi016.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-10-16T12:09:46.130 INFO:teuthology.orchestra.run.smithi016.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-10-16T12:09:46.130 INFO:teuthology.orchestra.run.smithi016.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-10-16T12:09:46.130 INFO:teuthology.orchestra.run.smithi016.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-10-16T12:09:46.130 INFO:teuthology.orchestra.run.smithi016.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-10-16T12:09:46.130 INFO:teuthology.orchestra.run.smithi016.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-10-16T12:09:46.131 INFO:teuthology.orchestra.run.smithi016.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-10-16T12:09:46.131 INFO:teuthology.orchestra.run.smithi016.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-10-16T12:09:46.131 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-fuse-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T12:09:46.131 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-mirror-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T12:09:46.131 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-nbd-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T12:09:46.132 INFO:teuthology.orchestra.run.smithi016.stdout: socat-1.7.4.1-1.el8.x86_64 2023-10-16T12:09:46.132 INFO:teuthology.orchestra.run.smithi016.stdout: thrift-0.13.0-2.el8.x86_64 2023-10-16T12:09:46.132 INFO:teuthology.orchestra.run.smithi016.stdout: uuid-1.6.2-43.el8.x86_64 2023-10-16T12:09:46.132 INFO:teuthology.orchestra.run.smithi016.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-10-16T12:09:46.132 INFO:teuthology.orchestra.run.smithi016.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-10-16T12:09:46.133 INFO:teuthology.orchestra.run.smithi016.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-10-16T12:09:46.133 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:09:46.133 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2023-10-16T12:09:46.266 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-16T12:09:46.267 INFO:teuthology.orchestra.run.smithi049.stdout:Upgraded: 2023-10-16T12:09:46.267 INFO:teuthology.orchestra.run.smithi049.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-10-16T12:09:46.267 INFO:teuthology.orchestra.run.smithi049.stdout: librados2-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T12:09:46.267 INFO:teuthology.orchestra.run.smithi049.stdout: librbd1-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T12:09:46.268 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2023-10-16T12:09:46.268 INFO:teuthology.orchestra.run.smithi049.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-10-16T12:09:46.268 INFO:teuthology.orchestra.run.smithi049.stdout: automake-1.16.1-7.el8.noarch 2023-10-16T12:09:46.268 INFO:teuthology.orchestra.run.smithi049.stdout: bison-3.0.4-10.el8.x86_64 2023-10-16T12:09:46.268 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T12:09:46.269 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-base-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T12:09:46.269 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-common-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T12:09:46.269 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-fuse-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T12:09:46.269 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-grafana-dashboards-2:18.0.0-6675.gc7471068.el8.noarch 2023-10-16T12:09:46.269 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-immutable-object-cache-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T12:09:46.270 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mds-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T12:09:46.270 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T12:09:46.270 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-cephadm-2:18.0.0-6675.gc7471068.el8.noarch 2023-10-16T12:09:46.270 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-dashboard-2:18.0.0-6675.gc7471068.el8.noarch 2023-10-16T12:09:46.270 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-diskprediction-local-2:18.0.0-6675.gc7471068.el8.noarch 2023-10-16T12:09:46.271 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-modules-core-2:18.0.0-6675.gc7471068.el8.noarch 2023-10-16T12:09:46.271 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-rook-2:18.0.0-6675.gc7471068.el8.noarch 2023-10-16T12:09:46.271 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mon-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T12:09:46.271 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-osd-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T12:09:46.271 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-prometheus-alerts-2:18.0.0-6675.gc7471068.el8.noarch 2023-10-16T12:09:46.272 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-radosgw-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T12:09:46.272 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-selinux-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T12:09:46.272 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-test-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T12:09:46.272 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-volume-2:18.0.0-6675.gc7471068.el8.noarch 2023-10-16T12:09:46.272 INFO:teuthology.orchestra.run.smithi049.stdout: cephadm-2:18.0.0-6675.gc7471068.el8.noarch 2023-10-16T12:09:46.272 INFO:teuthology.orchestra.run.smithi049.stdout: cephfs-mirror-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T12:09:46.273 INFO:teuthology.orchestra.run.smithi049.stdout: cephfs-top-2:18.0.0-6675.gc7471068.el8.noarch 2023-10-16T12:09:46.273 INFO:teuthology.orchestra.run.smithi049.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-10-16T12:09:46.273 INFO:teuthology.orchestra.run.smithi049.stdout: flex-2.6.1-9.el8.x86_64 2023-10-16T12:09:46.273 INFO:teuthology.orchestra.run.smithi049.stdout: fuse-2.9.7-15.el8.x86_64 2023-10-16T12:09:46.273 INFO:teuthology.orchestra.run.smithi049.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-10-16T12:09:46.273 INFO:teuthology.orchestra.run.smithi049.stdout: jq-1.6-3.el8.x86_64 2023-10-16T12:09:46.274 INFO:teuthology.orchestra.run.smithi049.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-10-16T12:09:46.274 INFO:teuthology.orchestra.run.smithi049.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-10-16T12:09:46.274 INFO:teuthology.orchestra.run.smithi049.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-10-16T12:09:46.274 INFO:teuthology.orchestra.run.smithi049.stdout: libcephfs-devel-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T12:09:46.274 INFO:teuthology.orchestra.run.smithi049.stdout: libcephfs2-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T12:09:46.275 INFO:teuthology.orchestra.run.smithi049.stdout: libcephsqlite-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T12:09:46.275 INFO:teuthology.orchestra.run.smithi049.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-10-16T12:09:46.275 INFO:teuthology.orchestra.run.smithi049.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-10-16T12:09:46.275 INFO:teuthology.orchestra.run.smithi049.stdout: liboath-2.6.2-3.el8.x86_64 2023-10-16T12:09:46.275 INFO:teuthology.orchestra.run.smithi049.stdout: libpq-13.5-1.el8.x86_64 2023-10-16T12:09:46.276 INFO:teuthology.orchestra.run.smithi049.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-10-16T12:09:46.276 INFO:teuthology.orchestra.run.smithi049.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-10-16T12:09:46.276 INFO:teuthology.orchestra.run.smithi049.stdout: librados-devel-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T12:09:46.276 INFO:teuthology.orchestra.run.smithi049.stdout: libradosstriper1-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T12:09:46.276 INFO:teuthology.orchestra.run.smithi049.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-10-16T12:09:46.277 INFO:teuthology.orchestra.run.smithi049.stdout: librgw2-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T12:09:46.277 INFO:teuthology.orchestra.run.smithi049.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-10-16T12:09:46.277 INFO:teuthology.orchestra.run.smithi049.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-10-16T12:09:46.277 INFO:teuthology.orchestra.run.smithi049.stdout: libtool-2.4.6-25.el8.x86_64 2023-10-16T12:09:46.277 INFO:teuthology.orchestra.run.smithi049.stdout: libunwind-1.3.1-3.el8.x86_64 2023-10-16T12:09:46.277 INFO:teuthology.orchestra.run.smithi049.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-10-16T12:09:46.278 INFO:teuthology.orchestra.run.smithi049.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-10-16T12:09:46.278 INFO:teuthology.orchestra.run.smithi049.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-10-16T12:09:46.278 INFO:teuthology.orchestra.run.smithi049.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-10-16T12:09:46.278 INFO:teuthology.orchestra.run.smithi049.stdout: openblas-0.3.15-3.el8.x86_64 2023-10-16T12:09:46.278 INFO:teuthology.orchestra.run.smithi049.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-10-16T12:09:46.279 INFO:teuthology.orchestra.run.smithi049.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-10-16T12:09:46.279 INFO:teuthology.orchestra.run.smithi049.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-10-16T12:09:46.279 INFO:teuthology.orchestra.run.smithi049.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-10-16T12:09:46.279 INFO:teuthology.orchestra.run.smithi049.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-10-16T12:09:46.279 INFO:teuthology.orchestra.run.smithi049.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-10-16T12:09:46.279 INFO:teuthology.orchestra.run.smithi049.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-10-16T12:09:46.280 INFO:teuthology.orchestra.run.smithi049.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-10-16T12:09:46.280 INFO:teuthology.orchestra.run.smithi049.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-10-16T12:09:46.280 INFO:teuthology.orchestra.run.smithi049.stdout: python3-babel-2.5.1-7.el8.noarch 2023-10-16T12:09:46.280 INFO:teuthology.orchestra.run.smithi049.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-10-16T12:09:46.280 INFO:teuthology.orchestra.run.smithi049.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-10-16T12:09:46.281 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-10-16T12:09:46.281 INFO:teuthology.orchestra.run.smithi049.stdout: python3-ceph-argparse-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T12:09:46.281 INFO:teuthology.orchestra.run.smithi049.stdout: python3-ceph-common-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T12:09:46.281 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cephfs-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T12:09:46.281 INFO:teuthology.orchestra.run.smithi049.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-10-16T12:09:46.282 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-10-16T12:09:46.282 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-10-16T12:09:46.282 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-10-16T12:09:46.282 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-10-16T12:09:46.282 INFO:teuthology.orchestra.run.smithi049.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-10-16T12:09:46.282 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-6.2-6.el8.noarch 2023-10-16T12:09:46.283 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-10-16T12:09:46.283 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-10-16T12:09:46.283 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-10-16T12:09:46.283 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-10-16T12:09:46.283 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-10-16T12:09:46.284 INFO:teuthology.orchestra.run.smithi049.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-10-16T12:09:46.284 INFO:teuthology.orchestra.run.smithi049.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-10-16T12:09:46.284 INFO:teuthology.orchestra.run.smithi049.stdout: python3-mako-1.0.6-13.el8.noarch 2023-10-16T12:09:46.284 INFO:teuthology.orchestra.run.smithi049.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-10-16T12:09:46.284 INFO:teuthology.orchestra.run.smithi049.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-10-16T12:09:46.284 INFO:teuthology.orchestra.run.smithi049.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-10-16T12:09:46.285 INFO:teuthology.orchestra.run.smithi049.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-10-16T12:09:46.285 INFO:teuthology.orchestra.run.smithi049.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-10-16T12:09:46.285 INFO:teuthology.orchestra.run.smithi049.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-10-16T12:09:46.285 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-10-16T12:09:46.285 INFO:teuthology.orchestra.run.smithi049.stdout: python3-portend-2.6-1.el8.noarch 2023-10-16T12:09:46.286 INFO:teuthology.orchestra.run.smithi049.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-10-16T12:09:46.286 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-10-16T12:09:46.286 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-10-16T12:09:46.286 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-10-16T12:09:46.286 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pycparser-2.14-14.el8.noarch 2023-10-16T12:09:46.286 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pytz-2017.2-9.el8.noarch 2023-10-16T12:09:46.287 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rados-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T12:09:46.287 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rbd-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T12:09:46.287 INFO:teuthology.orchestra.run.smithi049.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-10-16T12:09:46.287 INFO:teuthology.orchestra.run.smithi049.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-10-16T12:09:46.287 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rgw-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T12:09:46.288 INFO:teuthology.orchestra.run.smithi049.stdout: python3-routes-2.4.1-12.el8.noarch 2023-10-16T12:09:46.288 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rsa-4.9-2.el8.noarch 2023-10-16T12:09:46.288 INFO:teuthology.orchestra.run.smithi049.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-10-16T12:09:46.288 INFO:teuthology.orchestra.run.smithi049.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-10-16T12:09:46.288 INFO:teuthology.orchestra.run.smithi049.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-10-16T12:09:46.289 INFO:teuthology.orchestra.run.smithi049.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-10-16T12:09:46.289 INFO:teuthology.orchestra.run.smithi049.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-10-16T12:09:46.289 INFO:teuthology.orchestra.run.smithi049.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-10-16T12:09:46.289 INFO:teuthology.orchestra.run.smithi049.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-10-16T12:09:46.289 INFO:teuthology.orchestra.run.smithi049.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-10-16T12:09:46.290 INFO:teuthology.orchestra.run.smithi049.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-10-16T12:09:46.290 INFO:teuthology.orchestra.run.smithi049.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-10-16T12:09:46.290 INFO:teuthology.orchestra.run.smithi049.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-10-16T12:09:46.290 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-fuse-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T12:09:46.290 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-mirror-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T12:09:46.291 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-nbd-2:18.0.0-6675.gc7471068.el8.x86_64 2023-10-16T12:09:46.291 INFO:teuthology.orchestra.run.smithi049.stdout: socat-1.7.4.1-1.el8.x86_64 2023-10-16T12:09:46.291 INFO:teuthology.orchestra.run.smithi049.stdout: thrift-0.13.0-2.el8.x86_64 2023-10-16T12:09:46.291 INFO:teuthology.orchestra.run.smithi049.stdout: uuid-1.6.2-43.el8.x86_64 2023-10-16T12:09:46.291 INFO:teuthology.orchestra.run.smithi049.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-10-16T12:09:46.291 INFO:teuthology.orchestra.run.smithi049.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-10-16T12:09:46.292 INFO:teuthology.orchestra.run.smithi049.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-10-16T12:09:46.292 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-16T12:09:46.292 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-10-16T12:09:46.316 DEBUG:teuthology.parallel:result is None 2023-10-16T12:09:46.607 DEBUG:teuthology.parallel:result is None 2023-10-16T12:09:46.608 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=c7471068426370a3754c2250e7729280233d552b 2023-10-16T12:09:46.751 DEBUG:teuthology.orchestra.run.smithi016:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-10-16T12:09:46.777 INFO:teuthology.orchestra.run.smithi016.stdout:18.0.0-6675.gc7471068.el8 2023-10-16T12:09:46.777 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6675.gc7471068.el8 2023-10-16T12:09:46.778 INFO:teuthology.task.install:The correct ceph version 18.0.0-6675.gc7471068 is installed. 2023-10-16T12:09:46.778 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=c7471068426370a3754c2250e7729280233d552b 2023-10-16T12:09:46.920 DEBUG:teuthology.orchestra.run.smithi049:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-10-16T12:09:46.945 INFO:teuthology.orchestra.run.smithi049.stdout:18.0.0-6675.gc7471068.el8 2023-10-16T12:09:46.946 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6675.gc7471068.el8 2023-10-16T12:09:46.946 INFO:teuthology.task.install:The correct ceph version 18.0.0-6675.gc7471068 is installed. 2023-10-16T12:09:46.946 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-10-16T12:09:46.947 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:09:46.947 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-16T12:09:46.993 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T12:09:46.994 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-16T12:09:47.036 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-10-16T12:09:47.037 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:09:47.037 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/usr/bin/daemon-helper 2023-10-16T12:09:47.067 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-16T12:09:47.136 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T12:09:47.136 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/usr/bin/daemon-helper 2023-10-16T12:09:47.164 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-16T12:09:47.231 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-10-16T12:09:47.231 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:09:47.231 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-16T12:09:47.262 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-16T12:09:47.332 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T12:09:47.332 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-16T12:09:47.359 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-16T12:09:47.426 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-10-16T12:09:47.427 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:09:47.427 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/usr/bin/stdin-killer 2023-10-16T12:09:47.457 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-16T12:09:47.528 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T12:09:47.528 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/usr/bin/stdin-killer 2023-10-16T12:09:47.555 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-16T12:09:47.622 INFO:teuthology.run_tasks:Running task ceph... 2023-10-16T12:09:47.681 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-10-16T12:09:47.681 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 777 /var/log/ceph 2023-10-16T12:09:47.684 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod 777 /var/log/ceph 2023-10-16T12:09:47.713 INFO:tasks.ceph:Disabling ceph logrotate... 2023-10-16T12:09:47.713 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-16T12:09:47.756 DEBUG:teuthology.orchestra.run.smithi049:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-16T12:09:47.784 INFO:tasks.ceph:Creating extra log directories... 2023-10-16T12:09:47.785 DEBUG:teuthology.orchestra.run.smithi016:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-16T12:09:47.827 DEBUG:teuthology.orchestra.run.smithi049:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-16T12:09:47.861 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2023-10-16T12:09:47.861 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2023-10-16T12:09:47.862 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2023-10-16T12:09:47.862 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:09:47.862 DEBUG:teuthology.orchestra.run.smithi016:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-10-16T12:09:47.914 DEBUG:teuthology.orchestra.run.smithi016:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-10-16T12:09:48.076 DEBUG:teuthology.orchestra.run.smithi016:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-10-16T12:09:48.106 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T12:09:48.106 DEBUG:teuthology.orchestra.run.smithi049:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-10-16T12:09:48.124 DEBUG:teuthology.orchestra.run.smithi049:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-10-16T12:09:48.255 DEBUG:teuthology.orchestra.run.smithi049:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-10-16T12:09:48.288 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-10-16T12:09:48.288 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-10-16T12:09:48.288 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/mchangir-2023-10-16_11:33:41-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi/7428984', 'branch': 'wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule}', 'email': 'mchangir@redhat.com', 'first_in_suite': False, 'job_id': '7428984', 'kernel': {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'mchangir-2023-10-16_11:33:41-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'rhel', 'os_version': '8.6', 'overrides': {'admin_socket': {'branch': 'wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments'}, 'ceph': {'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'log-whitelist': ['OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': 'c7471068426370a3754c2250e7729280233d552b'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'c7471068426370a3754c2250e7729280233d552b'}}, 'kclient': {'syntax': 'v1'}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'mgr-snap_schedule-support-subvol-and-group-arguments', 'sha1': '9359c5b40091bccbce966736dc1a3c5bcc76ab0d'}}, 'owner': 'mchangir', 'priority': 25, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 7945, 'sha1': 'c7471068426370a3754c2250e7729280233d552b', 'sleep_before_teardown': 0, 'suite': 'fs', 'suite_branch': 'mgr-snap_schedule-support-subvol-and-group-arguments', 'suite_path': '/home/teuthworker/src/github.com_mchangir_ceph_9359c5b40091bccbce966736dc1a3c5bcc76ab0d/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/mchangir/ceph.git', 'suite_sha1': '9359c5b40091bccbce966736dc1a3c5bcc76ab0d', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi016.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs=', 'smithi049.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'c7471068426370a3754c2250e7729280233d552b'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'log-whitelist': ['OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': 'c7471068426370a3754c2250e7729280233d552b', 'cluster': 'ceph'}}, {'kclient': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_snap_schedules']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': '8cdab074dcca9a68965bc5a50e9c30b691949723', 'timestamp': '2023-10-16_11:33:41', 'tube': 'smithi', 'user': 'mchangir', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-10-16T12:09:48.289 DEBUG:teuthology.orchestra.run.smithi016:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-16T12:09:48.292 DEBUG:teuthology.orchestra.run.smithi049:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-16T12:09:48.340 DEBUG:teuthology.orchestra.run.smithi016:> sudo install -d -m0777 -- /var/run/ceph 2023-10-16T12:09:48.351 DEBUG:teuthology.orchestra.run.smithi049:> sudo install -d -m0777 -- /var/run/ceph 2023-10-16T12:09:48.407 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:09:48.407 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/scratch_devs of=/dev/stdout 2023-10-16T12:09:48.437 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-16T12:09:48.438 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_1 2023-10-16T12:09:48.496 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-16T12:09:48.496 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-16T12:09:48.496 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 6h/6d Inode: 391908 Links: 1 2023-10-16T12:09:48.496 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-16T12:09:48.496 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2023-10-16T12:09:48.497 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2023-10-16 12:09:40.037895721 +0000 2023-10-16T12:09:48.497 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2023-10-16 12:06:51.526426832 +0000 2023-10-16T12:09:48.497 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2023-10-16 12:06:51.526426832 +0000 2023-10-16T12:09:48.497 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: - 2023-10-16T12:09:48.498 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-16T12:09:48.563 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2023-10-16T12:09:48.563 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2023-10-16T12:09:48.563 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.00012829 s, 4.0 MB/s 2023-10-16T12:09:48.564 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-16T12:09:48.623 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_2 2023-10-16T12:09:48.681 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-16T12:09:48.682 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-16T12:09:48.682 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 6h/6d Inode: 388083 Links: 1 2023-10-16T12:09:48.682 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-16T12:09:48.682 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2023-10-16T12:09:48.683 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2023-10-16 12:09:40.037895721 +0000 2023-10-16T12:09:48.683 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2023-10-16 12:06:52.035418721 +0000 2023-10-16T12:09:48.683 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2023-10-16 12:06:52.035418721 +0000 2023-10-16T12:09:48.683 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: - 2023-10-16T12:09:48.683 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-16T12:09:48.748 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2023-10-16T12:09:48.748 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2023-10-16T12:09:48.749 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.000109811 s, 4.7 MB/s 2023-10-16T12:09:48.750 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-16T12:09:48.808 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_3 2023-10-16T12:09:48.866 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-16T12:09:48.866 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-16T12:09:48.867 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 6h/6d Inode: 395436 Links: 1 2023-10-16T12:09:48.867 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-16T12:09:48.867 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2023-10-16T12:09:48.867 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2023-10-16 12:09:40.037895721 +0000 2023-10-16T12:09:48.867 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2023-10-16 12:06:52.549410530 +0000 2023-10-16T12:09:48.868 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2023-10-16 12:06:52.549410530 +0000 2023-10-16T12:09:48.868 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: - 2023-10-16T12:09:48.868 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-16T12:09:48.934 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2023-10-16T12:09:48.935 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2023-10-16T12:09:48.935 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.000109756 s, 4.7 MB/s 2023-10-16T12:09:48.935 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-16T12:09:48.992 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_4 2023-10-16T12:09:49.050 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-16T12:09:49.050 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-16T12:09:49.050 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 6h/6d Inode: 381833 Links: 1 2023-10-16T12:09:49.051 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-16T12:09:49.051 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2023-10-16T12:09:49.051 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2023-10-16 12:09:40.037895721 +0000 2023-10-16T12:09:49.051 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2023-10-16 12:06:53.038402737 +0000 2023-10-16T12:09:49.051 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2023-10-16 12:06:53.038402737 +0000 2023-10-16T12:09:49.052 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: - 2023-10-16T12:09:49.052 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-16T12:09:49.116 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2023-10-16T12:09:49.117 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2023-10-16T12:09:49.117 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.000117732 s, 4.3 MB/s 2023-10-16T12:09:49.118 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-16T12:09:49.175 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-16T12:09:49.176 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T12:09:49.176 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/scratch_devs of=/dev/stdout 2023-10-16T12:09:49.197 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-16T12:09:49.197 DEBUG:teuthology.orchestra.run.smithi049:> stat /dev/vg_nvme/lv_1 2023-10-16T12:09:49.258 INFO:teuthology.orchestra.run.smithi049.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-16T12:09:49.258 INFO:teuthology.orchestra.run.smithi049.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-16T12:09:49.258 INFO:teuthology.orchestra.run.smithi049.stdout:Device: 6h/6d Inode: 393454 Links: 1 2023-10-16T12:09:49.258 INFO:teuthology.orchestra.run.smithi049.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-16T12:09:49.259 INFO:teuthology.orchestra.run.smithi049.stdout:Context: system_u:object_r:device_t:s0 2023-10-16T12:09:49.259 INFO:teuthology.orchestra.run.smithi049.stdout:Access: 2023-10-16 12:09:40.296362535 +0000 2023-10-16T12:09:49.259 INFO:teuthology.orchestra.run.smithi049.stdout:Modify: 2023-10-16 12:06:51.519762061 +0000 2023-10-16T12:09:49.259 INFO:teuthology.orchestra.run.smithi049.stdout:Change: 2023-10-16 12:06:51.519762061 +0000 2023-10-16T12:09:49.259 INFO:teuthology.orchestra.run.smithi049.stdout: Birth: - 2023-10-16T12:09:49.260 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-16T12:09:49.328 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records in 2023-10-16T12:09:49.329 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records out 2023-10-16T12:09:49.329 INFO:teuthology.orchestra.run.smithi049.stderr:512 bytes copied, 0.00014938 s, 3.4 MB/s 2023-10-16T12:09:49.330 DEBUG:teuthology.orchestra.run.smithi049:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-16T12:09:49.391 DEBUG:teuthology.orchestra.run.smithi049:> stat /dev/vg_nvme/lv_2 2023-10-16T12:09:49.450 INFO:teuthology.orchestra.run.smithi049.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-16T12:09:49.450 INFO:teuthology.orchestra.run.smithi049.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-16T12:09:49.450 INFO:teuthology.orchestra.run.smithi049.stdout:Device: 6h/6d Inode: 390830 Links: 1 2023-10-16T12:09:49.451 INFO:teuthology.orchestra.run.smithi049.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-16T12:09:49.451 INFO:teuthology.orchestra.run.smithi049.stdout:Context: system_u:object_r:device_t:s0 2023-10-16T12:09:49.451 INFO:teuthology.orchestra.run.smithi049.stdout:Access: 2023-10-16 12:09:40.296362535 +0000 2023-10-16T12:09:49.451 INFO:teuthology.orchestra.run.smithi049.stdout:Modify: 2023-10-16 12:06:52.029747334 +0000 2023-10-16T12:09:49.451 INFO:teuthology.orchestra.run.smithi049.stdout:Change: 2023-10-16 12:06:52.029747334 +0000 2023-10-16T12:09:49.451 INFO:teuthology.orchestra.run.smithi049.stdout: Birth: - 2023-10-16T12:09:49.452 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-16T12:09:49.517 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records in 2023-10-16T12:09:49.518 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records out 2023-10-16T12:09:49.518 INFO:teuthology.orchestra.run.smithi049.stderr:512 bytes copied, 0.000119986 s, 4.3 MB/s 2023-10-16T12:09:49.519 DEBUG:teuthology.orchestra.run.smithi049:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-16T12:09:49.578 DEBUG:teuthology.orchestra.run.smithi049:> stat /dev/vg_nvme/lv_3 2023-10-16T12:09:49.636 INFO:teuthology.orchestra.run.smithi049.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-16T12:09:49.636 INFO:teuthology.orchestra.run.smithi049.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-16T12:09:49.636 INFO:teuthology.orchestra.run.smithi049.stdout:Device: 6h/6d Inode: 387561 Links: 1 2023-10-16T12:09:49.636 INFO:teuthology.orchestra.run.smithi049.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-16T12:09:49.636 INFO:teuthology.orchestra.run.smithi049.stdout:Context: system_u:object_r:device_t:s0 2023-10-16T12:09:49.637 INFO:teuthology.orchestra.run.smithi049.stdout:Access: 2023-10-16 12:09:40.296362535 +0000 2023-10-16T12:09:49.637 INFO:teuthology.orchestra.run.smithi049.stdout:Modify: 2023-10-16 12:06:52.522733097 +0000 2023-10-16T12:09:49.637 INFO:teuthology.orchestra.run.smithi049.stdout:Change: 2023-10-16 12:06:52.522733097 +0000 2023-10-16T12:09:49.637 INFO:teuthology.orchestra.run.smithi049.stdout: Birth: - 2023-10-16T12:09:49.637 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-16T12:09:49.703 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records in 2023-10-16T12:09:49.704 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records out 2023-10-16T12:09:49.704 INFO:teuthology.orchestra.run.smithi049.stderr:512 bytes copied, 0.000138795 s, 3.7 MB/s 2023-10-16T12:09:49.705 DEBUG:teuthology.orchestra.run.smithi049:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-16T12:09:49.763 DEBUG:teuthology.orchestra.run.smithi049:> stat /dev/vg_nvme/lv_4 2023-10-16T12:09:49.821 INFO:teuthology.orchestra.run.smithi049.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-16T12:09:49.822 INFO:teuthology.orchestra.run.smithi049.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-16T12:09:49.822 INFO:teuthology.orchestra.run.smithi049.stdout:Device: 6h/6d Inode: 395660 Links: 1 2023-10-16T12:09:49.822 INFO:teuthology.orchestra.run.smithi049.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-16T12:09:49.822 INFO:teuthology.orchestra.run.smithi049.stdout:Context: system_u:object_r:device_t:s0 2023-10-16T12:09:49.823 INFO:teuthology.orchestra.run.smithi049.stdout:Access: 2023-10-16 12:09:40.296362535 +0000 2023-10-16T12:09:49.823 INFO:teuthology.orchestra.run.smithi049.stdout:Modify: 2023-10-16 12:06:53.016718831 +0000 2023-10-16T12:09:49.823 INFO:teuthology.orchestra.run.smithi049.stdout:Change: 2023-10-16 12:06:53.016718831 +0000 2023-10-16T12:09:49.823 INFO:teuthology.orchestra.run.smithi049.stdout: Birth: - 2023-10-16T12:09:49.823 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-16T12:09:49.888 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records in 2023-10-16T12:09:49.888 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records out 2023-10-16T12:09:49.888 INFO:teuthology.orchestra.run.smithi049.stderr:512 bytes copied, 0.000114439 s, 4.5 MB/s 2023-10-16T12:09:49.889 DEBUG:teuthology.orchestra.run.smithi049:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-16T12:09:49.948 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-16T12:09:49.948 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi016.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi049.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-10-16T12:09:49.948 INFO:tasks.ceph:Generating config... 2023-10-16T12:09:49.949 INFO:tasks.ceph:[client] client mount timeout = 600 2023-10-16T12:09:49.950 INFO:tasks.ceph:[client] debug client = 20 2023-10-16T12:09:49.950 INFO:tasks.ceph:[client] debug ms = 1 2023-10-16T12:09:49.950 INFO:tasks.ceph:[client] pid file = 2023-10-16T12:09:49.950 INFO:tasks.ceph:[client] rados mon op timeout = 900 2023-10-16T12:09:49.951 INFO:tasks.ceph:[client] rados osd op timeout = 900 2023-10-16T12:09:49.951 INFO:tasks.ceph:[global] ms die on skipped message = False 2023-10-16T12:09:49.951 INFO:tasks.ceph:[mds] debug mds = 20 2023-10-16T12:09:49.951 INFO:tasks.ceph:[mds] debug mds balancer = 20 2023-10-16T12:09:49.952 INFO:tasks.ceph:[mds] debug ms = 1 2023-10-16T12:09:49.952 INFO:tasks.ceph:[mds] mds debug frag = True 2023-10-16T12:09:49.952 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2023-10-16T12:09:49.952 INFO:tasks.ceph:[mds] mds op complaint time = 180 2023-10-16T12:09:49.952 INFO:tasks.ceph:[mds] mds verify scatter = True 2023-10-16T12:09:49.953 INFO:tasks.ceph:[mds] osd op complaint time = 180 2023-10-16T12:09:49.953 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2023-10-16T12:09:49.953 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2023-10-16T12:09:49.953 INFO:tasks.ceph:[mgr] debug client = 20 2023-10-16T12:09:49.954 INFO:tasks.ceph:[mgr] debug finisher = 20 2023-10-16T12:09:49.954 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-10-16T12:09:49.954 INFO:tasks.ceph:[mgr] debug ms = 1 2023-10-16T12:09:49.954 INFO:tasks.ceph:[mon] debug mon = 20 2023-10-16T12:09:49.955 INFO:tasks.ceph:[mon] debug ms = 1 2023-10-16T12:09:49.955 INFO:tasks.ceph:[mon] debug paxos = 20 2023-10-16T12:09:49.955 INFO:tasks.ceph:[mon] mon op complaint time = 120 2023-10-16T12:09:49.955 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-10-16T12:09:49.955 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-10-16T12:09:49.956 INFO:tasks.ceph:[osd] debug bluefs = 20 2023-10-16T12:09:49.956 INFO:tasks.ceph:[osd] debug bluestore = 20 2023-10-16T12:09:49.956 INFO:tasks.ceph:[osd] debug ms = 1 2023-10-16T12:09:49.956 INFO:tasks.ceph:[osd] debug osd = 20 2023-10-16T12:09:49.957 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-10-16T12:09:49.957 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-10-16T12:09:49.957 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-10-16T12:09:49.957 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-10-16T12:09:49.958 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-10-16T12:09:49.958 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-10-16T12:09:49.958 INFO:tasks.ceph:[osd] osd op complaint time = 180 2023-10-16T12:09:49.958 INFO:tasks.ceph:Setting up mon.a... 2023-10-16T12:09:49.959 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-10-16T12:09:50.023 INFO:teuthology.orchestra.run.smithi016.stdout:creating /etc/ceph/ceph.keyring 2023-10-16T12:09:50.026 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-10-16T12:09:50.076 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-16T12:09:50.104 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.16'), ('mon.b', '172.21.15.49'), ('mon.c', '[v2:172.21.15.49:3301,v1:172.21.15.49:6790]')] 2023-10-16T12:09:50.104 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.16,172.21.15.49,[v2:172.21.15.49:3301,v1:172.21.15.49:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20, 'debug finisher': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2023-10-16T12:09:50.105 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:09:50.105 DEBUG:teuthology.orchestra.run.smithi016:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-10-16T12:09:50.161 DEBUG:teuthology.orchestra.run.smithi016:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.16 --add b 172.21.15.49 --addv c '[v2:172.21.15.49:3301,v1:172.21.15.49:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2023-10-16T12:09:50.252 INFO:teuthology.orchestra.run.smithi016.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-10-16T12:09:50.252 INFO:teuthology.orchestra.run.smithi016.stdout:monmaptool: generated fsid 5b28dfa1-b21f-474a-9b2e-9fb77af58254 2023-10-16T12:09:50.253 INFO:teuthology.orchestra.run.smithi016.stdout:setting min_mon_release = pacific 2023-10-16T12:09:50.253 INFO:teuthology.orchestra.run.smithi016.stdout:epoch 0 2023-10-16T12:09:50.253 INFO:teuthology.orchestra.run.smithi016.stdout:fsid 5b28dfa1-b21f-474a-9b2e-9fb77af58254 2023-10-16T12:09:50.253 INFO:teuthology.orchestra.run.smithi016.stdout:last_changed 2023-10-16T12:09:50.251667+0000 2023-10-16T12:09:50.253 INFO:teuthology.orchestra.run.smithi016.stdout:created 2023-10-16T12:09:50.251667+0000 2023-10-16T12:09:50.254 INFO:teuthology.orchestra.run.smithi016.stdout:min_mon_release 16 (pacific) 2023-10-16T12:09:50.254 INFO:teuthology.orchestra.run.smithi016.stdout:election_strategy: 1 2023-10-16T12:09:50.254 INFO:teuthology.orchestra.run.smithi016.stdout:0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2023-10-16T12:09:50.254 INFO:teuthology.orchestra.run.smithi016.stdout:1: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.b 2023-10-16T12:09:50.254 INFO:teuthology.orchestra.run.smithi016.stdout:2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2023-10-16T12:09:50.255 INFO:teuthology.orchestra.run.smithi016.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-10-16T12:09:50.256 DEBUG:teuthology.orchestra.run.smithi016:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-10-16T12:09:50.272 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 5b28dfa1-b21f-474a-9b2e-9fb77af58254... 2023-10-16T12:09:50.273 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-16T12:09:50.315 DEBUG:teuthology.orchestra.run.smithi049:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-16T12:09:50.359 INFO:teuthology.orchestra.run.smithi016.stdout:[global] 2023-10-16T12:09:50.360 INFO:teuthology.orchestra.run.smithi016.stdout: chdir = "" 2023-10-16T12:09:50.360 INFO:teuthology.orchestra.run.smithi016.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-16T12:09:50.360 INFO:teuthology.orchestra.run.smithi016.stdout: auth supported = cephx 2023-10-16T12:09:50.360 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:09:50.360 INFO:teuthology.orchestra.run.smithi016.stdout: filestore xattr use omap = true 2023-10-16T12:09:50.361 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:09:50.361 INFO:teuthology.orchestra.run.smithi016.stdout: mon clock drift allowed = 1.000 2023-10-16T12:09:50.361 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:09:50.361 INFO:teuthology.orchestra.run.smithi016.stdout: osd crush chooseleaf type = 0 2023-10-16T12:09:50.361 INFO:teuthology.orchestra.run.smithi016.stdout: auth debug = true 2023-10-16T12:09:50.361 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:09:50.362 INFO:teuthology.orchestra.run.smithi016.stdout: ms die on old message = true 2023-10-16T12:09:50.362 INFO:teuthology.orchestra.run.smithi016.stdout: ms die on bug = true 2023-10-16T12:09:50.362 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:09:50.362 INFO:teuthology.orchestra.run.smithi016.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-16T12:09:50.362 INFO:teuthology.orchestra.run.smithi016.stdout: mon pg warn max object skew = 0 2023-10-16T12:09:50.363 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:09:50.363 INFO:teuthology.orchestra.run.smithi016.stdout: # disable pg_autoscaler by default for new pools 2023-10-16T12:09:50.363 INFO:teuthology.orchestra.run.smithi016.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-16T12:09:50.363 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:09:50.363 INFO:teuthology.orchestra.run.smithi016.stdout: osd pool default size = 2 2023-10-16T12:09:50.364 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:09:50.364 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd allow primary affinity = true 2023-10-16T12:09:50.364 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd allow pg remap = true 2023-10-16T12:09:50.364 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on legacy crush tunables = false 2023-10-16T12:09:50.364 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on crush straw calc version zero = false 2023-10-16T12:09:50.365 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on no sortbitwise = false 2023-10-16T12:09:50.365 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on osd down out interval zero = false 2023-10-16T12:09:50.365 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on too few osds = false 2023-10-16T12:09:50.365 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-16T12:09:50.365 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-16T12:09:50.365 INFO:teuthology.orchestra.run.smithi016.stdout: mon_allow_pool_size_one = true 2023-10-16T12:09:50.366 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:09:50.366 INFO:teuthology.orchestra.run.smithi016.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-16T12:09:50.366 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:09:50.366 INFO:teuthology.orchestra.run.smithi016.stdout: osd default data pool replay window = 5 2023-10-16T12:09:50.366 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:09:50.367 INFO:teuthology.orchestra.run.smithi016.stdout: mon allow pool delete = true 2023-10-16T12:09:50.367 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:09:50.367 INFO:teuthology.orchestra.run.smithi016.stdout: mon cluster log file level = debug 2023-10-16T12:09:50.367 INFO:teuthology.orchestra.run.smithi016.stdout: debug asserts on shutdown = true 2023-10-16T12:09:50.367 INFO:teuthology.orchestra.run.smithi016.stdout: mon health detail to clog = false 2023-10-16T12:09:50.367 INFO:teuthology.orchestra.run.smithi016.stdout: mon host = "172.21.15.16,172.21.15.49,[v2:172.21.15.49:3301,v1:172.21.15.49:6790]" 2023-10-16T12:09:50.368 INFO:teuthology.orchestra.run.smithi016.stdout: ms die on skipped message = False 2023-10-16T12:09:50.368 INFO:teuthology.orchestra.run.smithi016.stdout: fsid = 5b28dfa1-b21f-474a-9b2e-9fb77af58254 2023-10-16T12:09:50.368 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:09:50.368 INFO:teuthology.orchestra.run.smithi016.stdout:[osd] 2023-10-16T12:09:50.368 INFO:teuthology.orchestra.run.smithi016.stdout: osd journal size = 100 2023-10-16T12:09:50.369 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:09:50.369 INFO:teuthology.orchestra.run.smithi016.stdout: osd scrub load threshold = 5.0 2023-10-16T12:09:50.369 INFO:teuthology.orchestra.run.smithi016.stdout: osd scrub max interval = 600 2023-10-16T12:09:50.369 INFO:teuthology.orchestra.run.smithi016.stdout: osd mclock profile = high_recovery_ops 2023-10-16T12:09:50.369 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:09:50.369 INFO:teuthology.orchestra.run.smithi016.stdout: osd recover clone overlap = true 2023-10-16T12:09:50.370 INFO:teuthology.orchestra.run.smithi016.stdout: osd recovery max chunk = 1048576 2023-10-16T12:09:50.370 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:09:50.370 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug shutdown = true 2023-10-16T12:09:50.370 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug op order = true 2023-10-16T12:09:50.370 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug verify stray on activate = true 2023-10-16T12:09:50.371 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:09:50.371 INFO:teuthology.orchestra.run.smithi016.stdout: osd open classes on start = true 2023-10-16T12:09:50.371 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug pg log writeout = true 2023-10-16T12:09:50.371 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:09:50.371 INFO:teuthology.orchestra.run.smithi016.stdout: osd deep scrub update digest min age = 30 2023-10-16T12:09:50.371 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:09:50.372 INFO:teuthology.orchestra.run.smithi016.stdout: osd map max advance = 10 2023-10-16T12:09:50.372 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:09:50.372 INFO:teuthology.orchestra.run.smithi016.stdout: journal zero on create = true 2023-10-16T12:09:50.372 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:09:50.372 INFO:teuthology.orchestra.run.smithi016.stdout: filestore ondisk finisher threads = 3 2023-10-16T12:09:50.372 INFO:teuthology.orchestra.run.smithi016.stdout: filestore apply finisher threads = 3 2023-10-16T12:09:50.373 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:09:50.373 INFO:teuthology.orchestra.run.smithi016.stdout: bdev debug aio = true 2023-10-16T12:09:50.373 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug misdirected ops = true 2023-10-16T12:09:50.373 INFO:teuthology.orchestra.run.smithi016.stdout: bluestore block size = 96636764160 2023-10-16T12:09:50.373 INFO:teuthology.orchestra.run.smithi016.stdout: bluestore fsck on mount = True 2023-10-16T12:09:50.374 INFO:teuthology.orchestra.run.smithi016.stdout: debug bluefs = 20 2023-10-16T12:09:50.374 INFO:teuthology.orchestra.run.smithi016.stdout: debug bluestore = 20 2023-10-16T12:09:50.374 INFO:teuthology.orchestra.run.smithi016.stdout: debug ms = 1 2023-10-16T12:09:50.374 INFO:teuthology.orchestra.run.smithi016.stdout: debug osd = 20 2023-10-16T12:09:50.374 INFO:teuthology.orchestra.run.smithi016.stdout: debug rocksdb = 10 2023-10-16T12:09:50.374 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd backfillfull_ratio = 0.85 2023-10-16T12:09:50.375 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd full ratio = 0.9 2023-10-16T12:09:50.375 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd nearfull ratio = 0.8 2023-10-16T12:09:50.375 INFO:teuthology.orchestra.run.smithi016.stdout: osd failsafe full ratio = 0.95 2023-10-16T12:09:50.375 INFO:teuthology.orchestra.run.smithi016.stdout: osd objectstore = bluestore 2023-10-16T12:09:50.375 INFO:teuthology.orchestra.run.smithi016.stdout: osd op complaint time = 180 2023-10-16T12:09:50.376 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:09:50.376 INFO:teuthology.orchestra.run.smithi016.stdout:[mgr] 2023-10-16T12:09:50.376 INFO:teuthology.orchestra.run.smithi016.stdout: debug ms = 1 2023-10-16T12:09:50.376 INFO:teuthology.orchestra.run.smithi016.stdout: debug mgr = 20 2023-10-16T12:09:50.376 INFO:teuthology.orchestra.run.smithi016.stdout: debug mon = 20 2023-10-16T12:09:50.376 INFO:teuthology.orchestra.run.smithi016.stdout: debug auth = 20 2023-10-16T12:09:50.377 INFO:teuthology.orchestra.run.smithi016.stdout: mon reweight min pgs per osd = 4 2023-10-16T12:09:50.377 INFO:teuthology.orchestra.run.smithi016.stdout: mon reweight min bytes per osd = 10 2023-10-16T12:09:50.377 INFO:teuthology.orchestra.run.smithi016.stdout: mgr/telemetry/nag = false 2023-10-16T12:09:50.377 INFO:teuthology.orchestra.run.smithi016.stdout: debug client = 20 2023-10-16T12:09:50.377 INFO:teuthology.orchestra.run.smithi016.stdout: debug finisher = 20 2023-10-16T12:09:50.377 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:09:50.378 INFO:teuthology.orchestra.run.smithi016.stdout:[mon] 2023-10-16T12:09:50.378 INFO:teuthology.orchestra.run.smithi016.stdout: debug ms = 1 2023-10-16T12:09:50.378 INFO:teuthology.orchestra.run.smithi016.stdout: debug mon = 20 2023-10-16T12:09:50.378 INFO:teuthology.orchestra.run.smithi016.stdout: debug paxos = 20 2023-10-16T12:09:50.378 INFO:teuthology.orchestra.run.smithi016.stdout: debug auth = 20 2023-10-16T12:09:50.379 INFO:teuthology.orchestra.run.smithi016.stdout: mon data avail warn = 5 2023-10-16T12:09:50.379 INFO:teuthology.orchestra.run.smithi016.stdout: mon mgr mkfs grace = 240 2023-10-16T12:09:50.379 INFO:teuthology.orchestra.run.smithi016.stdout: mon reweight min pgs per osd = 4 2023-10-16T12:09:50.379 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd reporter subtree level = osd 2023-10-16T12:09:50.379 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd prime pg temp = true 2023-10-16T12:09:50.379 INFO:teuthology.orchestra.run.smithi016.stdout: mon reweight min bytes per osd = 10 2023-10-16T12:09:50.380 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:09:50.380 INFO:teuthology.orchestra.run.smithi016.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-16T12:09:50.380 INFO:teuthology.orchestra.run.smithi016.stdout: auth mon ticket ttl = 660 # 11m 2023-10-16T12:09:50.380 INFO:teuthology.orchestra.run.smithi016.stdout: auth service ticket ttl = 240 # 4m 2023-10-16T12:09:50.380 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:09:50.380 INFO:teuthology.orchestra.run.smithi016.stdout: # don't complain about insecure global_id in the test suite 2023-10-16T12:09:50.381 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-16T12:09:50.381 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-16T12:09:50.381 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:09:50.381 INFO:teuthology.orchestra.run.smithi016.stdout: # 1m isn't quite enough 2023-10-16T12:09:50.381 INFO:teuthology.orchestra.run.smithi016.stdout: mon_down_mkfs_grace = 2m 2023-10-16T12:09:50.382 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:09:50.382 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_filestore_osds = false 2023-10-16T12:09:50.382 INFO:teuthology.orchestra.run.smithi016.stdout: mon op complaint time = 120 2023-10-16T12:09:50.382 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:09:50.382 INFO:teuthology.orchestra.run.smithi016.stdout:[client] 2023-10-16T12:09:50.382 INFO:teuthology.orchestra.run.smithi016.stdout: rgw cache enabled = true 2023-10-16T12:09:50.383 INFO:teuthology.orchestra.run.smithi016.stdout: rgw enable ops log = true 2023-10-16T12:09:50.383 INFO:teuthology.orchestra.run.smithi016.stdout: rgw enable usage log = true 2023-10-16T12:09:50.383 INFO:teuthology.orchestra.run.smithi016.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-16T12:09:50.383 INFO:teuthology.orchestra.run.smithi016.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-16T12:09:50.383 INFO:teuthology.orchestra.run.smithi016.stdout: client mount timeout = 600 2023-10-16T12:09:50.383 INFO:teuthology.orchestra.run.smithi016.stdout: debug client = 20 2023-10-16T12:09:50.384 INFO:teuthology.orchestra.run.smithi016.stdout: debug ms = 1 2023-10-16T12:09:50.384 INFO:teuthology.orchestra.run.smithi016.stdout: pid file = "" 2023-10-16T12:09:50.384 INFO:teuthology.orchestra.run.smithi016.stdout: rados mon op timeout = 900 2023-10-16T12:09:50.384 INFO:teuthology.orchestra.run.smithi016.stdout: rados osd op timeout = 900 2023-10-16T12:09:50.384 INFO:teuthology.orchestra.run.smithi016.stdout:[mon.a] 2023-10-16T12:09:50.384 INFO:teuthology.orchestra.run.smithi016.stdout:[mon.b] 2023-10-16T12:09:50.385 INFO:teuthology.orchestra.run.smithi016.stdout:[mon.c] 2023-10-16T12:09:50.385 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.a] 2023-10-16T12:09:50.385 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.c] 2023-10-16T12:09:50.385 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.b] 2023-10-16T12:09:50.385 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.d] 2023-10-16T12:09:50.385 INFO:teuthology.orchestra.run.smithi016.stdout:[mds] 2023-10-16T12:09:50.386 INFO:teuthology.orchestra.run.smithi016.stdout: debug mds = 20 2023-10-16T12:09:50.386 INFO:teuthology.orchestra.run.smithi016.stdout: debug mds balancer = 20 2023-10-16T12:09:50.386 INFO:teuthology.orchestra.run.smithi016.stdout: debug ms = 1 2023-10-16T12:09:50.386 INFO:teuthology.orchestra.run.smithi016.stdout: mds debug frag = True 2023-10-16T12:09:50.386 INFO:teuthology.orchestra.run.smithi016.stdout: mds debug scatterstat = True 2023-10-16T12:09:50.387 INFO:teuthology.orchestra.run.smithi016.stdout: mds op complaint time = 180 2023-10-16T12:09:50.387 INFO:teuthology.orchestra.run.smithi016.stdout: mds verify scatter = True 2023-10-16T12:09:50.387 INFO:teuthology.orchestra.run.smithi016.stdout: osd op complaint time = 180 2023-10-16T12:09:50.387 INFO:teuthology.orchestra.run.smithi016.stdout: rados mon op timeout = 900 2023-10-16T12:09:50.387 INFO:teuthology.orchestra.run.smithi016.stdout: rados osd op timeout = 900 2023-10-16T12:09:50.389 INFO:teuthology.orchestra.run.smithi049.stdout:[global] 2023-10-16T12:09:50.389 INFO:teuthology.orchestra.run.smithi049.stdout: chdir = "" 2023-10-16T12:09:50.389 INFO:teuthology.orchestra.run.smithi049.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-16T12:09:50.389 INFO:teuthology.orchestra.run.smithi049.stdout: auth supported = cephx 2023-10-16T12:09:50.389 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-16T12:09:50.390 INFO:teuthology.orchestra.run.smithi049.stdout: filestore xattr use omap = true 2023-10-16T12:09:50.390 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-16T12:09:50.390 INFO:teuthology.orchestra.run.smithi049.stdout: mon clock drift allowed = 1.000 2023-10-16T12:09:50.390 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-16T12:09:50.390 INFO:teuthology.orchestra.run.smithi049.stdout: osd crush chooseleaf type = 0 2023-10-16T12:09:50.390 INFO:teuthology.orchestra.run.smithi049.stdout: auth debug = true 2023-10-16T12:09:50.391 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-16T12:09:50.391 INFO:teuthology.orchestra.run.smithi049.stdout: ms die on old message = true 2023-10-16T12:09:50.391 INFO:teuthology.orchestra.run.smithi049.stdout: ms die on bug = true 2023-10-16T12:09:50.391 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-16T12:09:50.391 INFO:teuthology.orchestra.run.smithi049.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-16T12:09:50.392 INFO:teuthology.orchestra.run.smithi049.stdout: mon pg warn max object skew = 0 2023-10-16T12:09:50.392 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-16T12:09:50.392 INFO:teuthology.orchestra.run.smithi049.stdout: # disable pg_autoscaler by default for new pools 2023-10-16T12:09:50.392 INFO:teuthology.orchestra.run.smithi049.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-16T12:09:50.392 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-16T12:09:50.392 INFO:teuthology.orchestra.run.smithi049.stdout: osd pool default size = 2 2023-10-16T12:09:50.393 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-16T12:09:50.393 INFO:teuthology.orchestra.run.smithi049.stdout: mon osd allow primary affinity = true 2023-10-16T12:09:50.393 INFO:teuthology.orchestra.run.smithi049.stdout: mon osd allow pg remap = true 2023-10-16T12:09:50.393 INFO:teuthology.orchestra.run.smithi049.stdout: mon warn on legacy crush tunables = false 2023-10-16T12:09:50.393 INFO:teuthology.orchestra.run.smithi049.stdout: mon warn on crush straw calc version zero = false 2023-10-16T12:09:50.393 INFO:teuthology.orchestra.run.smithi049.stdout: mon warn on no sortbitwise = false 2023-10-16T12:09:50.394 INFO:teuthology.orchestra.run.smithi049.stdout: mon warn on osd down out interval zero = false 2023-10-16T12:09:50.394 INFO:teuthology.orchestra.run.smithi049.stdout: mon warn on too few osds = false 2023-10-16T12:09:50.394 INFO:teuthology.orchestra.run.smithi049.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-16T12:09:50.394 INFO:teuthology.orchestra.run.smithi049.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-16T12:09:50.394 INFO:teuthology.orchestra.run.smithi049.stdout: mon_allow_pool_size_one = true 2023-10-16T12:09:50.395 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-16T12:09:50.395 INFO:teuthology.orchestra.run.smithi049.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-16T12:09:50.395 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-16T12:09:50.395 INFO:teuthology.orchestra.run.smithi049.stdout: osd default data pool replay window = 5 2023-10-16T12:09:50.395 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-16T12:09:50.395 INFO:teuthology.orchestra.run.smithi049.stdout: mon allow pool delete = true 2023-10-16T12:09:50.396 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-16T12:09:50.396 INFO:teuthology.orchestra.run.smithi049.stdout: mon cluster log file level = debug 2023-10-16T12:09:50.396 INFO:teuthology.orchestra.run.smithi049.stdout: debug asserts on shutdown = true 2023-10-16T12:09:50.396 INFO:teuthology.orchestra.run.smithi049.stdout: mon health detail to clog = false 2023-10-16T12:09:50.396 INFO:teuthology.orchestra.run.smithi049.stdout: mon host = "172.21.15.16,172.21.15.49,[v2:172.21.15.49:3301,v1:172.21.15.49:6790]" 2023-10-16T12:09:50.396 INFO:teuthology.orchestra.run.smithi049.stdout: ms die on skipped message = False 2023-10-16T12:09:50.397 INFO:teuthology.orchestra.run.smithi049.stdout: fsid = 5b28dfa1-b21f-474a-9b2e-9fb77af58254 2023-10-16T12:09:50.397 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-16T12:09:50.397 INFO:teuthology.orchestra.run.smithi049.stdout:[osd] 2023-10-16T12:09:50.397 INFO:teuthology.orchestra.run.smithi049.stdout: osd journal size = 100 2023-10-16T12:09:50.397 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-16T12:09:50.397 INFO:teuthology.orchestra.run.smithi049.stdout: osd scrub load threshold = 5.0 2023-10-16T12:09:50.398 INFO:teuthology.orchestra.run.smithi049.stdout: osd scrub max interval = 600 2023-10-16T12:09:50.398 INFO:teuthology.orchestra.run.smithi049.stdout: osd mclock profile = high_recovery_ops 2023-10-16T12:09:50.398 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-16T12:09:50.398 INFO:teuthology.orchestra.run.smithi049.stdout: osd recover clone overlap = true 2023-10-16T12:09:50.398 INFO:teuthology.orchestra.run.smithi049.stdout: osd recovery max chunk = 1048576 2023-10-16T12:09:50.398 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-16T12:09:50.399 INFO:teuthology.orchestra.run.smithi049.stdout: osd debug shutdown = true 2023-10-16T12:09:50.399 INFO:teuthology.orchestra.run.smithi049.stdout: osd debug op order = true 2023-10-16T12:09:50.399 INFO:teuthology.orchestra.run.smithi049.stdout: osd debug verify stray on activate = true 2023-10-16T12:09:50.399 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-16T12:09:50.399 INFO:teuthology.orchestra.run.smithi049.stdout: osd open classes on start = true 2023-10-16T12:09:50.400 INFO:teuthology.orchestra.run.smithi049.stdout: osd debug pg log writeout = true 2023-10-16T12:09:50.400 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-16T12:09:50.400 INFO:teuthology.orchestra.run.smithi049.stdout: osd deep scrub update digest min age = 30 2023-10-16T12:09:50.400 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-16T12:09:50.400 INFO:teuthology.orchestra.run.smithi049.stdout: osd map max advance = 10 2023-10-16T12:09:50.400 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-16T12:09:50.401 INFO:teuthology.orchestra.run.smithi049.stdout: journal zero on create = true 2023-10-16T12:09:50.401 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-16T12:09:50.401 INFO:teuthology.orchestra.run.smithi049.stdout: filestore ondisk finisher threads = 3 2023-10-16T12:09:50.401 INFO:teuthology.orchestra.run.smithi049.stdout: filestore apply finisher threads = 3 2023-10-16T12:09:50.401 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-16T12:09:50.401 INFO:teuthology.orchestra.run.smithi049.stdout: bdev debug aio = true 2023-10-16T12:09:50.402 INFO:teuthology.orchestra.run.smithi049.stdout: osd debug misdirected ops = true 2023-10-16T12:09:50.402 INFO:teuthology.orchestra.run.smithi049.stdout: bluestore block size = 96636764160 2023-10-16T12:09:50.402 INFO:teuthology.orchestra.run.smithi049.stdout: bluestore fsck on mount = True 2023-10-16T12:09:50.402 INFO:teuthology.orchestra.run.smithi049.stdout: debug bluefs = 20 2023-10-16T12:09:50.402 INFO:teuthology.orchestra.run.smithi049.stdout: debug bluestore = 20 2023-10-16T12:09:50.402 INFO:teuthology.orchestra.run.smithi049.stdout: debug ms = 1 2023-10-16T12:09:50.403 INFO:teuthology.orchestra.run.smithi049.stdout: debug osd = 20 2023-10-16T12:09:50.403 INFO:teuthology.orchestra.run.smithi049.stdout: debug rocksdb = 10 2023-10-16T12:09:50.403 INFO:teuthology.orchestra.run.smithi049.stdout: mon osd backfillfull_ratio = 0.85 2023-10-16T12:09:50.403 INFO:teuthology.orchestra.run.smithi049.stdout: mon osd full ratio = 0.9 2023-10-16T12:09:50.403 INFO:teuthology.orchestra.run.smithi049.stdout: mon osd nearfull ratio = 0.8 2023-10-16T12:09:50.404 INFO:teuthology.orchestra.run.smithi049.stdout: osd failsafe full ratio = 0.95 2023-10-16T12:09:50.404 INFO:teuthology.orchestra.run.smithi049.stdout: osd objectstore = bluestore 2023-10-16T12:09:50.404 INFO:teuthology.orchestra.run.smithi049.stdout: osd op complaint time = 180 2023-10-16T12:09:50.404 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-16T12:09:50.404 INFO:teuthology.orchestra.run.smithi049.stdout:[mgr] 2023-10-16T12:09:50.404 INFO:teuthology.orchestra.run.smithi049.stdout: debug ms = 1 2023-10-16T12:09:50.405 INFO:teuthology.orchestra.run.smithi049.stdout: debug mgr = 20 2023-10-16T12:09:50.405 INFO:teuthology.orchestra.run.smithi049.stdout: debug mon = 20 2023-10-16T12:09:50.405 INFO:teuthology.orchestra.run.smithi049.stdout: debug auth = 20 2023-10-16T12:09:50.405 INFO:teuthology.orchestra.run.smithi049.stdout: mon reweight min pgs per osd = 4 2023-10-16T12:09:50.405 INFO:teuthology.orchestra.run.smithi049.stdout: mon reweight min bytes per osd = 10 2023-10-16T12:09:50.405 INFO:teuthology.orchestra.run.smithi049.stdout: mgr/telemetry/nag = false 2023-10-16T12:09:50.406 INFO:teuthology.orchestra.run.smithi049.stdout: debug client = 20 2023-10-16T12:09:50.406 INFO:teuthology.orchestra.run.smithi049.stdout: debug finisher = 20 2023-10-16T12:09:50.406 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-16T12:09:50.406 INFO:teuthology.orchestra.run.smithi049.stdout:[mon] 2023-10-16T12:09:50.406 INFO:teuthology.orchestra.run.smithi049.stdout: debug ms = 1 2023-10-16T12:09:50.407 INFO:teuthology.orchestra.run.smithi049.stdout: debug mon = 20 2023-10-16T12:09:50.407 INFO:teuthology.orchestra.run.smithi049.stdout: debug paxos = 20 2023-10-16T12:09:50.407 INFO:teuthology.orchestra.run.smithi049.stdout: debug auth = 20 2023-10-16T12:09:50.407 INFO:teuthology.orchestra.run.smithi049.stdout: mon data avail warn = 5 2023-10-16T12:09:50.407 INFO:teuthology.orchestra.run.smithi049.stdout: mon mgr mkfs grace = 240 2023-10-16T12:09:50.407 INFO:teuthology.orchestra.run.smithi049.stdout: mon reweight min pgs per osd = 4 2023-10-16T12:09:50.408 INFO:teuthology.orchestra.run.smithi049.stdout: mon osd reporter subtree level = osd 2023-10-16T12:09:50.408 INFO:teuthology.orchestra.run.smithi049.stdout: mon osd prime pg temp = true 2023-10-16T12:09:50.408 INFO:teuthology.orchestra.run.smithi049.stdout: mon reweight min bytes per osd = 10 2023-10-16T12:09:50.408 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-16T12:09:50.408 INFO:teuthology.orchestra.run.smithi049.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-16T12:09:50.408 INFO:teuthology.orchestra.run.smithi049.stdout: auth mon ticket ttl = 660 # 11m 2023-10-16T12:09:50.409 INFO:teuthology.orchestra.run.smithi049.stdout: auth service ticket ttl = 240 # 4m 2023-10-16T12:09:50.409 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-16T12:09:50.409 INFO:teuthology.orchestra.run.smithi049.stdout: # don't complain about insecure global_id in the test suite 2023-10-16T12:09:50.409 INFO:teuthology.orchestra.run.smithi049.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-16T12:09:50.409 INFO:teuthology.orchestra.run.smithi049.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-16T12:09:50.409 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-16T12:09:50.410 INFO:teuthology.orchestra.run.smithi049.stdout: # 1m isn't quite enough 2023-10-16T12:09:50.410 INFO:teuthology.orchestra.run.smithi049.stdout: mon_down_mkfs_grace = 2m 2023-10-16T12:09:50.410 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-16T12:09:50.410 INFO:teuthology.orchestra.run.smithi049.stdout: mon_warn_on_filestore_osds = false 2023-10-16T12:09:50.410 INFO:teuthology.orchestra.run.smithi049.stdout: mon op complaint time = 120 2023-10-16T12:09:50.410 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-10-16T12:09:50.411 INFO:teuthology.orchestra.run.smithi049.stdout:[client] 2023-10-16T12:09:50.411 INFO:teuthology.orchestra.run.smithi049.stdout: rgw cache enabled = true 2023-10-16T12:09:50.411 INFO:teuthology.orchestra.run.smithi049.stdout: rgw enable ops log = true 2023-10-16T12:09:50.411 INFO:teuthology.orchestra.run.smithi049.stdout: rgw enable usage log = true 2023-10-16T12:09:50.411 INFO:teuthology.orchestra.run.smithi049.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-16T12:09:50.411 INFO:teuthology.orchestra.run.smithi049.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-16T12:09:50.412 INFO:teuthology.orchestra.run.smithi049.stdout: client mount timeout = 600 2023-10-16T12:09:50.412 INFO:teuthology.orchestra.run.smithi049.stdout: debug client = 20 2023-10-16T12:09:50.412 INFO:teuthology.orchestra.run.smithi049.stdout: debug ms = 1 2023-10-16T12:09:50.412 INFO:teuthology.orchestra.run.smithi049.stdout: pid file = "" 2023-10-16T12:09:50.412 INFO:teuthology.orchestra.run.smithi049.stdout: rados mon op timeout = 900 2023-10-16T12:09:50.412 INFO:teuthology.orchestra.run.smithi049.stdout: rados osd op timeout = 900 2023-10-16T12:09:50.413 INFO:teuthology.orchestra.run.smithi049.stdout:[mon.a] 2023-10-16T12:09:50.413 INFO:teuthology.orchestra.run.smithi049.stdout:[mon.b] 2023-10-16T12:09:50.413 INFO:teuthology.orchestra.run.smithi049.stdout:[mon.c] 2023-10-16T12:09:50.413 INFO:teuthology.orchestra.run.smithi049.stdout:[mds.a] 2023-10-16T12:09:50.413 INFO:teuthology.orchestra.run.smithi049.stdout:[mds.c] 2023-10-16T12:09:50.413 INFO:teuthology.orchestra.run.smithi049.stdout:[mds.b] 2023-10-16T12:09:50.414 INFO:teuthology.orchestra.run.smithi049.stdout:[mds.d] 2023-10-16T12:09:50.414 INFO:teuthology.orchestra.run.smithi049.stdout:[mds] 2023-10-16T12:09:50.414 INFO:teuthology.orchestra.run.smithi049.stdout: debug mds = 20 2023-10-16T12:09:50.414 INFO:teuthology.orchestra.run.smithi049.stdout: debug mds balancer = 20 2023-10-16T12:09:50.414 INFO:teuthology.orchestra.run.smithi049.stdout: debug ms = 1 2023-10-16T12:09:50.414 INFO:teuthology.orchestra.run.smithi049.stdout: mds debug frag = True 2023-10-16T12:09:50.415 INFO:teuthology.orchestra.run.smithi049.stdout: mds debug scatterstat = True 2023-10-16T12:09:50.415 INFO:teuthology.orchestra.run.smithi049.stdout: mds op complaint time = 180 2023-10-16T12:09:50.415 INFO:teuthology.orchestra.run.smithi049.stdout: mds verify scatter = True 2023-10-16T12:09:50.415 INFO:teuthology.orchestra.run.smithi049.stdout: osd op complaint time = 180 2023-10-16T12:09:50.415 INFO:teuthology.orchestra.run.smithi049.stdout: rados mon op timeout = 900 2023-10-16T12:09:50.415 INFO:teuthology.orchestra.run.smithi049.stdout: rados osd op timeout = 900 2023-10-16T12:09:50.416 INFO:tasks.ceph:Creating admin key on mon.a... 2023-10-16T12:09:50.416 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-10-16T12:09:50.487 INFO:tasks.ceph:Copying monmap to all nodes... 2023-10-16T12:09:50.487 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:09:50.487 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-10-16T12:09:50.504 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:09:50.504 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-10-16T12:09:50.561 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi016.front.sepia.ceph.com 2023-10-16T12:09:50.561 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:09:50.561 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-16T12:09:50.561 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-16T12:09:50.637 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:09:50.638 DEBUG:teuthology.orchestra.run.smithi016:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-16T12:09:50.694 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi049.front.sepia.ceph.com 2023-10-16T12:09:50.695 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T12:09:50.695 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-16T12:09:50.695 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-16T12:09:50.737 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T12:09:50.737 DEBUG:teuthology.orchestra.run.smithi049:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-16T12:09:50.795 INFO:tasks.ceph:Setting up mon nodes... 2023-10-16T12:09:50.795 INFO:tasks.ceph:Setting up mgr nodes... 2023-10-16T12:09:50.795 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-10-16T12:09:50.850 INFO:teuthology.orchestra.run.smithi016.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-10-16T12:09:50.853 DEBUG:teuthology.orchestra.run.smithi049:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-10-16T12:09:50.922 INFO:teuthology.orchestra.run.smithi049.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-10-16T12:09:50.925 INFO:tasks.ceph:Setting up mds nodes... 2023-10-16T12:09:50.925 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2023-10-16T12:09:50.981 INFO:teuthology.orchestra.run.smithi016.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2023-10-16T12:09:50.983 DEBUG:teuthology.orchestra.run.smithi016:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2023-10-16T12:09:51.010 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2023-10-16T12:09:51.104 INFO:teuthology.orchestra.run.smithi016.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2023-10-16T12:09:51.107 DEBUG:teuthology.orchestra.run.smithi016:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2023-10-16T12:09:51.134 DEBUG:teuthology.orchestra.run.smithi049:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2023-10-16T12:09:51.192 INFO:teuthology.orchestra.run.smithi049.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2023-10-16T12:09:51.194 DEBUG:teuthology.orchestra.run.smithi049:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2023-10-16T12:09:51.222 DEBUG:teuthology.orchestra.run.smithi049:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2023-10-16T12:09:51.317 INFO:teuthology.orchestra.run.smithi049.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2023-10-16T12:09:51.320 DEBUG:teuthology.orchestra.run.smithi049:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2023-10-16T12:09:51.346 INFO:tasks.ceph_client:Setting up client nodes... 2023-10-16T12:09:51.347 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-10-16T12:09:51.395 INFO:teuthology.orchestra.run.smithi016.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-10-16T12:09:51.410 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2023-10-16T12:09:51.456 INFO:teuthology.orchestra.run.smithi016.stdout:creating /etc/ceph/ceph.client.2.keyring 2023-10-16T12:09:51.553 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-10-16T12:09:51.601 INFO:teuthology.orchestra.run.smithi049.stdout:creating /etc/ceph/ceph.client.1.keyring 2023-10-16T12:09:51.614 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2023-10-16T12:09:51.659 INFO:teuthology.orchestra.run.smithi049.stdout:creating /etc/ceph/ceph.client.3.keyring 2023-10-16T12:09:51.672 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-10-16T12:09:51.672 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi016.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi049.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-10-16T12:09:51.672 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-10-16T12:09:51.700 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-16T12:09:51.700 INFO:tasks.ceph:role: osd.0 2023-10-16T12:09:51.701 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi016.front.sepia.ceph.com 2023-10-16T12:09:51.701 DEBUG:teuthology.orchestra.run.smithi016:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-16T12:09:51.767 INFO:teuthology.orchestra.run.smithi016.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-16T12:09:51.768 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-16T12:09:51.768 INFO:teuthology.orchestra.run.smithi016.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-16T12:09:51.768 INFO:teuthology.orchestra.run.smithi016.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-16T12:09:51.768 INFO:teuthology.orchestra.run.smithi016.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-16T12:09:51.768 INFO:teuthology.orchestra.run.smithi016.stdout: = sunit=0 swidth=0 blks 2023-10-16T12:09:51.768 INFO:teuthology.orchestra.run.smithi016.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-16T12:09:51.769 INFO:teuthology.orchestra.run.smithi016.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-16T12:09:51.769 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-16T12:09:51.769 INFO:teuthology.orchestra.run.smithi016.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-16T12:09:51.791 INFO:teuthology.orchestra.run.smithi016.stdout:Discarding blocks...Done. 2023-10-16T12:09:51.792 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi016.front.sepia.ceph.com -o noatime 2023-10-16T12:09:51.792 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-10-16T12:09:51.874 DEBUG:teuthology.orchestra.run.smithi016:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-10-16T12:09:51.946 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-10-16T12:09:52.013 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-16T12:09:52.013 INFO:tasks.ceph:role: osd.1 2023-10-16T12:09:52.014 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi016.front.sepia.ceph.com 2023-10-16T12:09:52.014 DEBUG:teuthology.orchestra.run.smithi016:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-16T12:09:52.080 INFO:teuthology.orchestra.run.smithi016.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-16T12:09:52.080 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-16T12:09:52.081 INFO:teuthology.orchestra.run.smithi016.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-16T12:09:52.081 INFO:teuthology.orchestra.run.smithi016.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-16T12:09:52.081 INFO:teuthology.orchestra.run.smithi016.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-16T12:09:52.081 INFO:teuthology.orchestra.run.smithi016.stdout: = sunit=0 swidth=0 blks 2023-10-16T12:09:52.081 INFO:teuthology.orchestra.run.smithi016.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-16T12:09:52.081 INFO:teuthology.orchestra.run.smithi016.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-16T12:09:52.082 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-16T12:09:52.082 INFO:teuthology.orchestra.run.smithi016.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-16T12:09:52.103 INFO:teuthology.orchestra.run.smithi016.stdout:Discarding blocks...Done. 2023-10-16T12:09:52.105 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi016.front.sepia.ceph.com -o noatime 2023-10-16T12:09:52.105 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-10-16T12:09:52.185 DEBUG:teuthology.orchestra.run.smithi016:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-10-16T12:09:52.256 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-10-16T12:09:52.323 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-16T12:09:52.323 INFO:tasks.ceph:role: osd.2 2023-10-16T12:09:52.323 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi016.front.sepia.ceph.com 2023-10-16T12:09:52.324 DEBUG:teuthology.orchestra.run.smithi016:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-16T12:09:52.390 INFO:teuthology.orchestra.run.smithi016.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-16T12:09:52.391 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-16T12:09:52.391 INFO:teuthology.orchestra.run.smithi016.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-16T12:09:52.391 INFO:teuthology.orchestra.run.smithi016.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-16T12:09:52.391 INFO:teuthology.orchestra.run.smithi016.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-16T12:09:52.391 INFO:teuthology.orchestra.run.smithi016.stdout: = sunit=0 swidth=0 blks 2023-10-16T12:09:52.392 INFO:teuthology.orchestra.run.smithi016.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-16T12:09:52.392 INFO:teuthology.orchestra.run.smithi016.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-16T12:09:52.392 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-16T12:09:52.392 INFO:teuthology.orchestra.run.smithi016.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-16T12:09:52.413 INFO:teuthology.orchestra.run.smithi016.stdout:Discarding blocks...Done. 2023-10-16T12:09:52.415 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi016.front.sepia.ceph.com -o noatime 2023-10-16T12:09:52.415 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-10-16T12:09:52.495 DEBUG:teuthology.orchestra.run.smithi016:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-10-16T12:09:52.566 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-10-16T12:09:52.634 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-16T12:09:52.634 INFO:tasks.ceph:role: osd.3 2023-10-16T12:09:52.634 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi016.front.sepia.ceph.com 2023-10-16T12:09:52.635 DEBUG:teuthology.orchestra.run.smithi016:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-10-16T12:09:52.701 INFO:teuthology.orchestra.run.smithi016.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-10-16T12:09:52.701 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-16T12:09:52.702 INFO:teuthology.orchestra.run.smithi016.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-16T12:09:52.702 INFO:teuthology.orchestra.run.smithi016.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-16T12:09:52.702 INFO:teuthology.orchestra.run.smithi016.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-16T12:09:52.702 INFO:teuthology.orchestra.run.smithi016.stdout: = sunit=0 swidth=0 blks 2023-10-16T12:09:52.702 INFO:teuthology.orchestra.run.smithi016.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-16T12:09:52.702 INFO:teuthology.orchestra.run.smithi016.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-16T12:09:52.703 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-16T12:09:52.703 INFO:teuthology.orchestra.run.smithi016.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-16T12:09:52.723 INFO:teuthology.orchestra.run.smithi016.stdout:Discarding blocks...Done. 2023-10-16T12:09:52.725 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi016.front.sepia.ceph.com -o noatime 2023-10-16T12:09:52.726 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-10-16T12:09:52.805 DEBUG:teuthology.orchestra.run.smithi016:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-10-16T12:09:52.876 DEBUG:teuthology.orchestra.run.smithi016:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-16T12:09:52.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:09:52.978+0000 7f40aa6fe640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-10-16T12:09:52.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:09:52.979+0000 7f40aa6fe640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-10-16T12:09:52.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:09:52.979+0000 7f40aa6fe640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-16T12:09:52.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:09:52.979+0000 7f40aa6fe640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-16T12:09:52.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:09:52.979+0000 7f40aa6fe640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-16T12:09:52.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:09:52.979+0000 7f40aa6fe640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-10-16T12:09:55.857 DEBUG:teuthology.orchestra.run.smithi016:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-10-16T12:09:55.889 DEBUG:teuthology.orchestra.run.smithi016:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-16T12:09:55.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:09:55.981+0000 7fbe5e342640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-10-16T12:09:55.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:09:55.981+0000 7fbe5e342640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-10-16T12:09:55.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:09:55.981+0000 7fbe5e342640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-16T12:09:55.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:09:55.981+0000 7fbe5e342640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-16T12:09:55.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:09:55.981+0000 7fbe5e342640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-16T12:09:55.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:09:55.981+0000 7fbe5e342640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-10-16T12:09:58.860 DEBUG:teuthology.orchestra.run.smithi016:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-10-16T12:09:58.892 DEBUG:teuthology.orchestra.run.smithi016:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-16T12:09:59.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:09:59.023+0000 7fb193b3f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-10-16T12:09:59.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:09:59.023+0000 7fb193b3f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-10-16T12:09:59.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:09:59.023+0000 7fb193b3f640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-16T12:09:59.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:09:59.023+0000 7fb193b3f640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-16T12:09:59.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:09:59.023+0000 7fb193b3f640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-16T12:09:59.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:09:59.023+0000 7fb193b3f640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-10-16T12:10:01.904 DEBUG:teuthology.orchestra.run.smithi016:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-10-16T12:10:01.937 DEBUG:teuthology.orchestra.run.smithi016:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-16T12:10:02.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:02.030+0000 7fc02423c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-10-16T12:10:02.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:02.031+0000 7fc02423c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-10-16T12:10:02.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:02.031+0000 7fc02423c640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-16T12:10:02.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:02.031+0000 7fc02423c640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-16T12:10:02.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:02.031+0000 7fc02423c640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-16T12:10:02.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:02.031+0000 7fc02423c640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-10-16T12:10:04.892 DEBUG:teuthology.orchestra.run.smithi016:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-10-16T12:10:04.925 DEBUG:teuthology.orchestra.run.smithi049:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-10-16T12:10:04.960 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-16T12:10:04.960 INFO:tasks.ceph:role: osd.4 2023-10-16T12:10:04.961 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi049.front.sepia.ceph.com 2023-10-16T12:10:04.961 DEBUG:teuthology.orchestra.run.smithi049:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-16T12:10:05.031 INFO:teuthology.orchestra.run.smithi049.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-16T12:10:05.031 INFO:teuthology.orchestra.run.smithi049.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-16T12:10:05.032 INFO:teuthology.orchestra.run.smithi049.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-16T12:10:05.032 INFO:teuthology.orchestra.run.smithi049.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-16T12:10:05.032 INFO:teuthology.orchestra.run.smithi049.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-16T12:10:05.032 INFO:teuthology.orchestra.run.smithi049.stdout: = sunit=0 swidth=0 blks 2023-10-16T12:10:05.032 INFO:teuthology.orchestra.run.smithi049.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-16T12:10:05.033 INFO:teuthology.orchestra.run.smithi049.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-16T12:10:05.033 INFO:teuthology.orchestra.run.smithi049.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-16T12:10:05.033 INFO:teuthology.orchestra.run.smithi049.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-16T12:10:05.062 INFO:teuthology.orchestra.run.smithi049.stdout:Discarding blocks...Done. 2023-10-16T12:10:05.120 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi049.front.sepia.ceph.com -o noatime 2023-10-16T12:10:05.120 DEBUG:teuthology.orchestra.run.smithi049:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-10-16T12:10:05.207 DEBUG:teuthology.orchestra.run.smithi049:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-10-16T12:10:05.239 DEBUG:teuthology.orchestra.run.smithi049:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-10-16T12:10:05.308 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-16T12:10:05.308 INFO:tasks.ceph:role: osd.5 2023-10-16T12:10:05.309 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi049.front.sepia.ceph.com 2023-10-16T12:10:05.309 DEBUG:teuthology.orchestra.run.smithi049:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-16T12:10:05.377 INFO:teuthology.orchestra.run.smithi049.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-16T12:10:05.377 INFO:teuthology.orchestra.run.smithi049.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-16T12:10:05.377 INFO:teuthology.orchestra.run.smithi049.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-16T12:10:05.377 INFO:teuthology.orchestra.run.smithi049.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-16T12:10:05.378 INFO:teuthology.orchestra.run.smithi049.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-16T12:10:05.378 INFO:teuthology.orchestra.run.smithi049.stdout: = sunit=0 swidth=0 blks 2023-10-16T12:10:05.378 INFO:teuthology.orchestra.run.smithi049.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-16T12:10:05.378 INFO:teuthology.orchestra.run.smithi049.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-16T12:10:05.378 INFO:teuthology.orchestra.run.smithi049.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-16T12:10:05.379 INFO:teuthology.orchestra.run.smithi049.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-16T12:10:05.407 INFO:teuthology.orchestra.run.smithi049.stdout:Discarding blocks...Done. 2023-10-16T12:10:05.409 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi049.front.sepia.ceph.com -o noatime 2023-10-16T12:10:05.409 DEBUG:teuthology.orchestra.run.smithi049:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-10-16T12:10:05.489 DEBUG:teuthology.orchestra.run.smithi049:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-10-16T12:10:05.560 DEBUG:teuthology.orchestra.run.smithi049:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-10-16T12:10:05.627 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-16T12:10:05.627 INFO:tasks.ceph:role: osd.6 2023-10-16T12:10:05.627 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi049.front.sepia.ceph.com 2023-10-16T12:10:05.628 DEBUG:teuthology.orchestra.run.smithi049:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-16T12:10:05.693 INFO:teuthology.orchestra.run.smithi049.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-16T12:10:05.694 INFO:teuthology.orchestra.run.smithi049.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-16T12:10:05.694 INFO:teuthology.orchestra.run.smithi049.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-16T12:10:05.694 INFO:teuthology.orchestra.run.smithi049.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-16T12:10:05.694 INFO:teuthology.orchestra.run.smithi049.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-16T12:10:05.694 INFO:teuthology.orchestra.run.smithi049.stdout: = sunit=0 swidth=0 blks 2023-10-16T12:10:05.694 INFO:teuthology.orchestra.run.smithi049.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-16T12:10:05.695 INFO:teuthology.orchestra.run.smithi049.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-16T12:10:05.695 INFO:teuthology.orchestra.run.smithi049.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-16T12:10:05.695 INFO:teuthology.orchestra.run.smithi049.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-16T12:10:05.723 INFO:teuthology.orchestra.run.smithi049.stdout:Discarding blocks...Done. 2023-10-16T12:10:05.725 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi049.front.sepia.ceph.com -o noatime 2023-10-16T12:10:05.725 DEBUG:teuthology.orchestra.run.smithi049:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-10-16T12:10:05.805 DEBUG:teuthology.orchestra.run.smithi049:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-10-16T12:10:05.875 DEBUG:teuthology.orchestra.run.smithi049:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-10-16T12:10:05.943 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-16T12:10:05.943 INFO:tasks.ceph:role: osd.7 2023-10-16T12:10:05.943 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi049.front.sepia.ceph.com 2023-10-16T12:10:05.943 DEBUG:teuthology.orchestra.run.smithi049:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-10-16T12:10:06.009 INFO:teuthology.orchestra.run.smithi049.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-10-16T12:10:06.009 INFO:teuthology.orchestra.run.smithi049.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-16T12:10:06.009 INFO:teuthology.orchestra.run.smithi049.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-16T12:10:06.010 INFO:teuthology.orchestra.run.smithi049.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-16T12:10:06.010 INFO:teuthology.orchestra.run.smithi049.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-16T12:10:06.010 INFO:teuthology.orchestra.run.smithi049.stdout: = sunit=0 swidth=0 blks 2023-10-16T12:10:06.010 INFO:teuthology.orchestra.run.smithi049.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-16T12:10:06.010 INFO:teuthology.orchestra.run.smithi049.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-16T12:10:06.010 INFO:teuthology.orchestra.run.smithi049.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-16T12:10:06.011 INFO:teuthology.orchestra.run.smithi049.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-16T12:10:06.038 INFO:teuthology.orchestra.run.smithi049.stdout:Discarding blocks...Done. 2023-10-16T12:10:06.040 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi049.front.sepia.ceph.com -o noatime 2023-10-16T12:10:06.040 DEBUG:teuthology.orchestra.run.smithi049:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-10-16T12:10:06.121 DEBUG:teuthology.orchestra.run.smithi049:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-10-16T12:10:06.193 DEBUG:teuthology.orchestra.run.smithi049:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-16T12:10:06.301 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:06.298+0000 7f1fbeeb7640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-10-16T12:10:06.301 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:06.299+0000 7f1fbeeb7640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-10-16T12:10:06.301 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:06.299+0000 7f1fbeeb7640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-16T12:10:06.302 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:06.299+0000 7f1fbeeb7640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-16T12:10:06.302 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:06.299+0000 7f1fbeeb7640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-16T12:10:06.302 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:06.299+0000 7f1fbeeb7640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-10-16T12:10:09.157 DEBUG:teuthology.orchestra.run.smithi049:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-10-16T12:10:09.190 DEBUG:teuthology.orchestra.run.smithi049:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-16T12:10:09.285 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:09.282+0000 7f0023954640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-10-16T12:10:09.285 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:09.283+0000 7f0023954640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-10-16T12:10:09.285 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:09.283+0000 7f0023954640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-16T12:10:09.285 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:09.283+0000 7f0023954640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-16T12:10:09.286 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:09.283+0000 7f0023954640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-16T12:10:09.286 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:09.283+0000 7f0023954640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-10-16T12:10:12.145 DEBUG:teuthology.orchestra.run.smithi049:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-10-16T12:10:12.178 DEBUG:teuthology.orchestra.run.smithi049:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-16T12:10:12.272 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:12.269+0000 7f9a9cfa2640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-10-16T12:10:12.272 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:12.269+0000 7f9a9cfa2640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-10-16T12:10:12.272 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:12.269+0000 7f9a9cfa2640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-16T12:10:12.272 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:12.269+0000 7f9a9cfa2640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-16T12:10:12.273 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:12.269+0000 7f9a9cfa2640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-16T12:10:12.273 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:12.270+0000 7f9a9cfa2640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-10-16T12:10:15.139 DEBUG:teuthology.orchestra.run.smithi049:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-10-16T12:10:15.171 DEBUG:teuthology.orchestra.run.smithi049:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-16T12:10:15.264 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:15.262+0000 7f210a4bf640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-10-16T12:10:15.265 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:15.262+0000 7f210a4bf640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-10-16T12:10:15.265 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:15.262+0000 7f210a4bf640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-16T12:10:15.265 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:15.262+0000 7f210a4bf640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-16T12:10:15.265 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:15.262+0000 7f210a4bf640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-16T12:10:15.265 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:15.263+0000 7f210a4bf640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-10-16T12:10:18.128 DEBUG:teuthology.orchestra.run.smithi049:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-10-16T12:10:18.161 INFO:tasks.ceph:Reading keys from all nodes... 2023-10-16T12:10:18.161 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:10:18.162 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-10-16T12:10:18.196 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:10:18.197 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2023-10-16T12:10:18.270 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:10:18.270 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2023-10-16T12:10:18.291 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:10:18.313 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:10:18.341 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:10:18.341 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-10-16T12:10:18.409 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:10:18.409 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-10-16T12:10:18.477 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:10:18.477 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-10-16T12:10:18.544 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:10:18.545 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-10-16T12:10:18.611 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T12:10:18.612 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-10-16T12:10:18.645 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T12:10:18.645 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2023-10-16T12:10:18.716 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T12:10:18.716 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2023-10-16T12:10:18.785 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T12:10:18.786 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-10-16T12:10:18.855 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T12:10:18.855 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-10-16T12:10:18.923 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T12:10:18.924 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-10-16T12:10:18.992 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T12:10:18.992 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-10-16T12:10:19.060 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:10:19.061 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-10-16T12:10:19.080 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:10:19.081 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2023-10-16T12:10:19.140 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T12:10:19.140 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2023-10-16T12:10:19.158 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T12:10:19.159 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2023-10-16T12:10:19.216 INFO:tasks.ceph:Adding keys to all mons... 2023-10-16T12:10:19.216 DEBUG:teuthology.orchestra.run.smithi016:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-16T12:10:19.219 DEBUG:teuthology.orchestra.run.smithi049:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-16T12:10:19.259 INFO:teuthology.orchestra.run.smithi016.stdout:[mgr.y] 2023-10-16T12:10:19.259 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAOKC1lLz2nMhAA8NqFYZ5nDKkzPBMrkwsVuQ== 2023-10-16T12:10:19.259 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.a] 2023-10-16T12:10:19.259 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAOKC1l/rBtOhAA/ob5WUvpz8Glp4E+xPZOYA== 2023-10-16T12:10:19.259 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.c] 2023-10-16T12:10:19.260 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAPKC1lvBowBhAA3waJNYhdgs8PGCYqYNYUog== 2023-10-16T12:10:19.260 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.0] 2023-10-16T12:10:19.260 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAQKC1lbxllOhAAZeLdp2GMl6HqQPaQjKlNqw== 2023-10-16T12:10:19.260 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.1] 2023-10-16T12:10:19.260 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQATKC1lnU6KOhAAHxvlbmzBptf1keGgL+Rbkg== 2023-10-16T12:10:19.261 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.2] 2023-10-16T12:10:19.261 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAXKC1l9jFyARAAr/NAwtWhBMut9sAAoDEGWg== 2023-10-16T12:10:19.261 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.3] 2023-10-16T12:10:19.261 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAaKC1lbw/lARAAGJ/9DNb5/7kLFSq7vrLtXg== 2023-10-16T12:10:19.261 INFO:teuthology.orchestra.run.smithi016.stdout:[mgr.x] 2023-10-16T12:10:19.262 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAOKC1l6b3vNhAAmj5xbfid2sZM2KfHmAf1Wg== 2023-10-16T12:10:19.262 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.b] 2023-10-16T12:10:19.262 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAPKC1lKAVsCxAAvhXuLY6i1+Iz1yq5BVPMNQ== 2023-10-16T12:10:19.262 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.d] 2023-10-16T12:10:19.262 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAPKC1lS/fnEhAAfNYgQ5BW86JMjYm/bBfnaQ== 2023-10-16T12:10:19.263 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.4] 2023-10-16T12:10:19.263 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAeKC1lZb7jERAAYpvb1xcSli9CTq7L8ChLOA== 2023-10-16T12:10:19.263 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.5] 2023-10-16T12:10:19.263 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAhKC1l/4HuEBAAqfnRqjknij9KES/fqiyIBQ== 2023-10-16T12:10:19.263 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.6] 2023-10-16T12:10:19.263 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAkKC1lzoUkEBAA+Q82OiBMNag8+R316WQtUw== 2023-10-16T12:10:19.264 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.7] 2023-10-16T12:10:19.264 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAnKC1ljUu4DxAAc+G4anJANC2GhLJ8wb6gYg== 2023-10-16T12:10:19.264 INFO:teuthology.orchestra.run.smithi016.stdout:[client.0] 2023-10-16T12:10:19.264 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAPKC1l2l6PFxAA9QyXtBAHdGAXpwH0JfknuQ== 2023-10-16T12:10:19.264 INFO:teuthology.orchestra.run.smithi016.stdout:[client.2] 2023-10-16T12:10:19.265 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAPKC1lA6wnGxAAdtaqYhtIJu8qENMrD8xvVA== 2023-10-16T12:10:19.265 INFO:teuthology.orchestra.run.smithi016.stdout:[client.1] 2023-10-16T12:10:19.265 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAPKC1lAcHFIxAAcNt2bmv2R6j5L3DYkdw7eQ== 2023-10-16T12:10:19.265 INFO:teuthology.orchestra.run.smithi016.stdout:[client.3] 2023-10-16T12:10:19.265 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAPKC1l/ddAJxAA3fNKKn9XkXas1BlDN+ggjw== 2023-10-16T12:10:19.282 INFO:teuthology.orchestra.run.smithi049.stdout:[mgr.y] 2023-10-16T12:10:19.282 INFO:teuthology.orchestra.run.smithi049.stdout: key = AQAOKC1lLz2nMhAA8NqFYZ5nDKkzPBMrkwsVuQ== 2023-10-16T12:10:19.282 INFO:teuthology.orchestra.run.smithi049.stdout:[mds.a] 2023-10-16T12:10:19.282 INFO:teuthology.orchestra.run.smithi049.stdout: key = AQAOKC1l/rBtOhAA/ob5WUvpz8Glp4E+xPZOYA== 2023-10-16T12:10:19.282 INFO:teuthology.orchestra.run.smithi049.stdout:[mds.c] 2023-10-16T12:10:19.283 INFO:teuthology.orchestra.run.smithi049.stdout: key = AQAPKC1lvBowBhAA3waJNYhdgs8PGCYqYNYUog== 2023-10-16T12:10:19.283 INFO:teuthology.orchestra.run.smithi049.stdout:[osd.0] 2023-10-16T12:10:19.283 INFO:teuthology.orchestra.run.smithi049.stdout: key = AQAQKC1lbxllOhAAZeLdp2GMl6HqQPaQjKlNqw== 2023-10-16T12:10:19.283 INFO:teuthology.orchestra.run.smithi049.stdout:[osd.1] 2023-10-16T12:10:19.283 INFO:teuthology.orchestra.run.smithi049.stdout: key = AQATKC1lnU6KOhAAHxvlbmzBptf1keGgL+Rbkg== 2023-10-16T12:10:19.284 INFO:teuthology.orchestra.run.smithi049.stdout:[osd.2] 2023-10-16T12:10:19.284 INFO:teuthology.orchestra.run.smithi049.stdout: key = AQAXKC1l9jFyARAAr/NAwtWhBMut9sAAoDEGWg== 2023-10-16T12:10:19.284 INFO:teuthology.orchestra.run.smithi049.stdout:[osd.3] 2023-10-16T12:10:19.284 INFO:teuthology.orchestra.run.smithi049.stdout: key = AQAaKC1lbw/lARAAGJ/9DNb5/7kLFSq7vrLtXg== 2023-10-16T12:10:19.285 INFO:teuthology.orchestra.run.smithi049.stdout:[mgr.x] 2023-10-16T12:10:19.285 INFO:teuthology.orchestra.run.smithi049.stdout: key = AQAOKC1l6b3vNhAAmj5xbfid2sZM2KfHmAf1Wg== 2023-10-16T12:10:19.285 INFO:teuthology.orchestra.run.smithi049.stdout:[mds.b] 2023-10-16T12:10:19.285 INFO:teuthology.orchestra.run.smithi049.stdout: key = AQAPKC1lKAVsCxAAvhXuLY6i1+Iz1yq5BVPMNQ== 2023-10-16T12:10:19.285 INFO:teuthology.orchestra.run.smithi049.stdout:[mds.d] 2023-10-16T12:10:19.286 INFO:teuthology.orchestra.run.smithi049.stdout: key = AQAPKC1lS/fnEhAAfNYgQ5BW86JMjYm/bBfnaQ== 2023-10-16T12:10:19.286 INFO:teuthology.orchestra.run.smithi049.stdout:[osd.4] 2023-10-16T12:10:19.286 INFO:teuthology.orchestra.run.smithi049.stdout: key = AQAeKC1lZb7jERAAYpvb1xcSli9CTq7L8ChLOA== 2023-10-16T12:10:19.286 INFO:teuthology.orchestra.run.smithi049.stdout:[osd.5] 2023-10-16T12:10:19.286 INFO:teuthology.orchestra.run.smithi049.stdout: key = AQAhKC1l/4HuEBAAqfnRqjknij9KES/fqiyIBQ== 2023-10-16T12:10:19.287 INFO:teuthology.orchestra.run.smithi049.stdout:[osd.6] 2023-10-16T12:10:19.287 INFO:teuthology.orchestra.run.smithi049.stdout: key = AQAkKC1lzoUkEBAA+Q82OiBMNag8+R316WQtUw== 2023-10-16T12:10:19.287 INFO:teuthology.orchestra.run.smithi049.stdout:[osd.7] 2023-10-16T12:10:19.287 INFO:teuthology.orchestra.run.smithi049.stdout: key = AQAnKC1ljUu4DxAAc+G4anJANC2GhLJ8wb6gYg== 2023-10-16T12:10:19.287 INFO:teuthology.orchestra.run.smithi049.stdout:[client.0] 2023-10-16T12:10:19.288 INFO:teuthology.orchestra.run.smithi049.stdout: key = AQAPKC1l2l6PFxAA9QyXtBAHdGAXpwH0JfknuQ== 2023-10-16T12:10:19.288 INFO:teuthology.orchestra.run.smithi049.stdout:[client.2] 2023-10-16T12:10:19.288 INFO:teuthology.orchestra.run.smithi049.stdout: key = AQAPKC1lA6wnGxAAdtaqYhtIJu8qENMrD8xvVA== 2023-10-16T12:10:19.288 INFO:teuthology.orchestra.run.smithi049.stdout:[client.1] 2023-10-16T12:10:19.289 INFO:teuthology.orchestra.run.smithi049.stdout: key = AQAPKC1lAcHFIxAAcNt2bmv2R6j5L3DYkdw7eQ== 2023-10-16T12:10:19.289 INFO:teuthology.orchestra.run.smithi049.stdout:[client.3] 2023-10-16T12:10:19.289 INFO:teuthology.orchestra.run.smithi049.stdout: key = AQAPKC1l/ddAJxAA3fNKKn9XkXas1BlDN+ggjw== 2023-10-16T12:10:19.290 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-16T12:10:19.309 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-16T12:10:19.384 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-16T12:10:19.387 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-16T12:10:19.445 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-16T12:10:19.447 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-16T12:10:19.505 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T12:10:19.507 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T12:10:19.561 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T12:10:19.564 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T12:10:19.618 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T12:10:19.620 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T12:10:19.674 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T12:10:19.676 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T12:10:19.730 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-16T12:10:19.732 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-16T12:10:19.787 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-16T12:10:19.789 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-16T12:10:19.847 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-16T12:10:19.849 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-16T12:10:19.906 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T12:10:19.909 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T12:10:19.963 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T12:10:19.966 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T12:10:20.020 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T12:10:20.023 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T12:10:20.077 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T12:10:20.079 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T12:10:20.134 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-16T12:10:20.136 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-16T12:10:20.194 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-16T12:10:20.196 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-16T12:10:20.253 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-16T12:10:20.256 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-16T12:10:20.314 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-16T12:10:20.316 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-16T12:10:20.374 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-10-16T12:10:20.374 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-10-16T12:10:20.401 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-16T12:10:20.501 DEBUG:teuthology.orchestra.run.smithi016:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-10-16T12:10:20.528 DEBUG:teuthology.orchestra.run.smithi049:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-10-16T12:10:20.556 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-16T12:10:20.655 DEBUG:teuthology.orchestra.run.smithi049:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-10-16T12:10:20.682 DEBUG:teuthology.orchestra.run.smithi049:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-10-16T12:10:20.749 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-16T12:10:20.849 DEBUG:teuthology.orchestra.run.smithi049:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-10-16T12:10:20.877 DEBUG:teuthology.orchestra.run.smithi016:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-16T12:10:20.879 DEBUG:teuthology.orchestra.run.smithi049:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-16T12:10:20.933 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-10-16T12:10:20.933 INFO:tasks.ceph.mon.a:Restarting daemon 2023-10-16T12:10:20.934 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-10-16T12:10:20.936 INFO:tasks.ceph.mon.a:Started 2023-10-16T12:10:20.937 INFO:tasks.ceph.mon.b:Restarting daemon 2023-10-16T12:10:20.937 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-10-16T12:10:20.976 INFO:tasks.ceph.mon.b:Started 2023-10-16T12:10:20.976 INFO:tasks.ceph.mon.c:Restarting daemon 2023-10-16T12:10:20.976 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-10-16T12:10:20.978 INFO:tasks.ceph.mon.c:Started 2023-10-16T12:10:20.979 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-10-16T12:10:20.979 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-10-16T12:10:20.979 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2023-10-16T12:10:20.981 INFO:tasks.ceph.mgr.y:Started 2023-10-16T12:10:20.982 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-10-16T12:10:20.982 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-10-16T12:10:20.984 INFO:tasks.ceph.mgr.x:Started 2023-10-16T12:10:20.984 INFO:tasks.ceph:Setting crush tunables to default 2023-10-16T12:10:20.985 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --cluster ceph osd crush tunables default 2023-10-16T12:10:21.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.110+0000 7f8efffdf700 1 Processor -- start 2023-10-16T12:10:21.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.110+0000 7f8efffdf700 1 -- start start 2023-10-16T12:10:21.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.111+0000 7f8efffdf700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8ef815b070 0x7f8ef815b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:21.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.111+0000 7f8efffdf700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f8ef806e170 0x7f8ef80cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:21.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.111+0000 7f8efffdf700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ef80cef70 con 0x7f8ef80cb880 2023-10-16T12:10:21.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.111+0000 7f8efffdf700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ef80ce360 con 0x7f8ef806e170 2023-10-16T12:10:21.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.111+0000 7f8efffdf700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef80ce640 con 0x7f8ef815b070 2023-10-16T12:10:21.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.111+0000 7f8efdd7b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8ef815b070 0x7f8ef815b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:21.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.111+0000 7f8efe57c700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f8ef806e170 0x7f8ef80cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:21.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.111+0000 7f8efdd7b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8ef815b070 0x7f8ef815b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:48884/0 (socket says 172.21.15.16:48884) 2023-10-16T12:10:21.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.111+0000 7f8efdd7b700 1 -- 172.21.15.16:0/2146524654 learned_addr learned my addr 172.21.15.16:0/2146524654 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:10:21.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.111+0000 7f8efdd7b700 1 -- 172.21.15.16:0/2146524654 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8ef815b070 msgr2=0x7f8ef815b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-16T12:10:21.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.111+0000 7f8efdd7b700 1 -- 172.21.15.16:0/2146524654 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8ef815b070 msgr2=0x7f8ef815b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-16T12:10:21.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.111+0000 7f8efdd7b700 1 --2- 172.21.15.16:0/2146524654 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8ef815b070 0x7f8ef815b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-16T12:10:21.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.111+0000 7f8efe57c700 1 -- 172.21.15.16:0/2146524654 >> v2:172.21.15.49:3300/0 conn(0x7f8ef806e170 msgr2=0x7f8ef80cbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-16T12:10:21.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.111+0000 7f8efe57c700 1 -- 172.21.15.16:0/2146524654 >> v2:172.21.15.49:3300/0 conn(0x7f8ef806e170 msgr2=0x7f8ef80cbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-16T12:10:21.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.111+0000 7f8efe57c700 1 --2- 172.21.15.16:0/2146524654 >> v2:172.21.15.49:3300/0 conn(0x7f8ef806e170 0x7f8ef80cbd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-16T12:10:21.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.111+0000 7f8efdd7b700 1 --2- 172.21.15.16:0/2146524654 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8ef815b070 0x7f8ef815b450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:10:21.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.111+0000 7f8efe57c700 1 --2- 172.21.15.16:0/2146524654 >> v2:172.21.15.49:3300/0 conn(0x7f8ef806e170 0x7f8ef80cbd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:10:21.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.282+0000 7f8efcd79700 1 -- 172.21.15.16:0/2146524654 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1229278894 0 0) 0x7f8ef80cef70 con 0x7f8ef80cb880 2023-10-16T12:10:21.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.283+0000 7f8efcd79700 1 -- 172.21.15.16:0/2146524654 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ee40035b0 con 0x7f8ef80cb880 2023-10-16T12:10:21.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.283+0000 7f8efcd79700 1 -- 172.21.15.16:0/2146524654 <== mon.1 v1:172.21.15.49:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3966839666 0 0) 0x7f8ee40035b0 con 0x7f8ef80cb880 2023-10-16T12:10:21.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.283+0000 7f8efcd79700 1 -- 172.21.15.16:0/2146524654 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8ef815b070 msgr2=0x7f8ef815b450 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-16T12:10:21.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.283+0000 7f8efcd79700 1 --2- 172.21.15.16:0/2146524654 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8ef815b070 0x7f8ef815b450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:21.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.283+0000 7f8efcd79700 1 -- 172.21.15.16:0/2146524654 >> v2:172.21.15.49:3300/0 conn(0x7f8ef806e170 msgr2=0x7f8ef80cbd00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-16T12:10:21.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.283+0000 7f8efcd79700 1 --2- 172.21.15.16:0/2146524654 >> v2:172.21.15.49:3300/0 conn(0x7f8ef806e170 0x7f8ef80cbd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:21.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.283+0000 7f8efcd79700 1 -- 172.21.15.16:0/2146524654 --> v1:172.21.15.49:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ef80ce920 con 0x7f8ef80cb880 2023-10-16T12:10:21.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.283+0000 7f8efcd79700 1 -- 172.21.15.16:0/2146524654 <== mon.1 v1:172.21.15.49:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f8eec001670 con 0x7f8ef80cb880 2023-10-16T12:10:21.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.283+0000 7f8efcd79700 1 -- 172.21.15.16:0/2146524654 >> v1:172.21.15.49:6789/0 conn(0x7f8ef80cb880 legacy=0x7f8ef806da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:21.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.283+0000 7f8efcd79700 1 --2- 172.21.15.16:0/2146524654 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8ef815b070 0x7f8ee4003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:21.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.283+0000 7f8efcd79700 1 --2- 172.21.15.16:0/2146524654 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ee4004030 0x7f8ee4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:21.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.283+0000 7f8efcd79700 1 --2- 172.21.15.16:0/2146524654 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8ee4004d90 0x7f8ee4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:21.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.283+0000 7f8efcd79700 1 -- 172.21.15.16:0/2146524654 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef80ce360 con 0x7f8ee4004030 2023-10-16T12:10:21.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.283+0000 7f8efcd79700 1 -- 172.21.15.16:0/2146524654 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef80ce640 con 0x7f8ee4004d90 2023-10-16T12:10:21.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.283+0000 7f8efcd79700 1 -- 172.21.15.16:0/2146524654 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ee400bbd0 con 0x7f8ef815b070 2023-10-16T12:10:21.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.283+0000 7f8efdd7b700 1 --2- 172.21.15.16:0/2146524654 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8ef815b070 0x7f8ee4003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:21.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.283+0000 7f8efd57a700 1 --2- 172.21.15.16:0/2146524654 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ee4004030 0x7f8ee4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:21.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.283+0000 7f8efe57c700 1 --2- 172.21.15.16:0/2146524654 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8ee4004d90 0x7f8ee4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:21.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.284+0000 7f8efd57a700 1 -- 172.21.15.16:0/2146524654 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8ef815b070 msgr2=0x7f8ee4003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:10:21.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.284+0000 7f8efd57a700 1 --2- 172.21.15.16:0/2146524654 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8ef815b070 0x7f8ee4003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:21.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.284+0000 7f8efd57a700 1 -- 172.21.15.16:0/2146524654 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8ee4004d90 msgr2=0x7f8ee4005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:10:21.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.284+0000 7f8efd57a700 1 --2- 172.21.15.16:0/2146524654 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8ee4004d90 0x7f8ee4005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:21.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.284+0000 7f8efd57a700 1 -- 172.21.15.16:0/2146524654 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ee40035b0 con 0x7f8ee4004030 2023-10-16T12:10:21.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.284+0000 7f8efd57a700 1 --2- 172.21.15.16:0/2146524654 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ee4004030 0x7f8ee4004850 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f8eec00c190 tx=0x7f8ee4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36ffb641402eef26 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:21.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.285+0000 7f8efcd79700 1 -- 172.21.15.16:0/2146524654 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eec007b60 con 0x7f8ee4004030 2023-10-16T12:10:21.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.285+0000 7f8efcd79700 1 -- 172.21.15.16:0/2146524654 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8eec00cc90 con 0x7f8ee4004030 2023-10-16T12:10:21.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.285+0000 7f8efcd79700 1 -- 172.21.15.16:0/2146524654 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eec012420 con 0x7f8ee4004030 2023-10-16T12:10:21.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.286+0000 7f8efffdf700 1 -- 172.21.15.16:0/2146524654 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ee4004030 msgr2=0x7f8ee4004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:21.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.286+0000 7f8efffdf700 1 --2- 172.21.15.16:0/2146524654 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ee4004030 0x7f8ee4004850 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f8eec00c190 tx=0x7f8ee4005880 comp rx=0 tx=0).stop 2023-10-16T12:10:21.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.286+0000 7f8efe57c700 1 -- 172.21.15.16:0/2146524654 reap_dead start 2023-10-16T12:10:21.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.286+0000 7f8efffdf700 1 -- 172.21.15.16:0/2146524654 shutdown_connections 2023-10-16T12:10:21.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.286+0000 7f8efffdf700 1 -- 172.21.15.16:0/2146524654 >> 172.21.15.16:0/2146524654 conn(0x7f8ef80b1f40 msgr2=0x7f8ef80b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:21.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.286+0000 7f8efffdf700 1 -- 172.21.15.16:0/2146524654 shutdown_connections 2023-10-16T12:10:21.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.286+0000 7f8efffdf700 1 -- 172.21.15.16:0/2146524654 wait complete. 2023-10-16T12:10:21.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.288+0000 7f8efffdf700 1 Processor -- start 2023-10-16T12:10:21.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.288+0000 7f8efffdf700 1 -- start start 2023-10-16T12:10:21.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.289+0000 7f8efffdf700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8ee4004030 0x7f8ef8151d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:21.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.289+0000 7f8efffdf700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ef80b3a40 0x7f8ef80b3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:21.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.289+0000 7f8efffdf700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8ef80b0850 0x7f8ef80b0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:21.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.289+0000 7f8efffdf700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef80b1210 con 0x7f8ef80b3a40 2023-10-16T12:10:21.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.289+0000 7f8efffdf700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef80b44e0 con 0x7f8ef80b0850 2023-10-16T12:10:21.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.289+0000 7f8efffdf700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef80b47c0 con 0x7f8ee4004030 2023-10-16T12:10:21.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.289+0000 7f8efd57a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ef80b3a40 0x7f8ef80b3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:21.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.289+0000 7f8efd57a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ef80b3a40 0x7f8ef80b3ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60660/0 (socket says 172.21.15.16:60660) 2023-10-16T12:10:21.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.289+0000 7f8efd57a700 1 -- 172.21.15.16:0/1065184032 learned_addr learned my addr 172.21.15.16:0/1065184032 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:10:21.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.289+0000 7f8efe57c700 1 --2- 172.21.15.16:0/1065184032 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8ef80b0850 0x7f8ef80b0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:21.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.289+0000 7f8efd57a700 1 -- 172.21.15.16:0/1065184032 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8ee4004030 msgr2=0x7f8ef8151d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:21.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.289+0000 7f8efd57a700 1 --2- 172.21.15.16:0/1065184032 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8ee4004030 0x7f8ef8151d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:21.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.289+0000 7f8efd57a700 1 -- 172.21.15.16:0/1065184032 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8ef80b0850 msgr2=0x7f8ef80b0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:21.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.289+0000 7f8efd57a700 1 --2- 172.21.15.16:0/1065184032 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8ef80b0850 0x7f8ef80b0cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:21.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.289+0000 7f8efd57a700 1 -- 172.21.15.16:0/1065184032 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ef80b4aa0 con 0x7f8ef80b3a40 2023-10-16T12:10:21.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.289+0000 7f8efd57a700 1 --2- 172.21.15.16:0/1065184032 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ef80b3a40 0x7f8ef80b3ec0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f8eec000c00 tx=0x7f8eec000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:21.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.291+0000 7f8eea7fc700 1 -- 172.21.15.16:0/1065184032 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eec0052e0 con 0x7f8ef80b3a40 2023-10-16T12:10:21.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.291+0000 7f8eea7fc700 1 -- 172.21.15.16:0/1065184032 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8eec021070 con 0x7f8ef80b3a40 2023-10-16T12:10:21.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.291+0000 7f8efffdf700 1 -- 172.21.15.16:0/1065184032 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ef80b4d80 con 0x7f8ef80b3a40 2023-10-16T12:10:21.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.291+0000 7f8eea7fc700 1 -- 172.21.15.16:0/1065184032 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eec006860 con 0x7f8ef80b3a40 2023-10-16T12:10:21.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.291+0000 7f8efffdf700 1 -- 172.21.15.16:0/1065184032 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ef8159cc0 con 0x7f8ef80b3a40 2023-10-16T12:10:21.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.292+0000 7f8efffdf700 1 -- 172.21.15.16:0/1065184032 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ef80b52a0 con 0x7f8ef80b3a40 2023-10-16T12:10:21.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.294+0000 7f8eea7fc700 1 -- 172.21.15.16:0/1065184032 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f8eec006cc0 con 0x7f8ef80b3a40 2023-10-16T12:10:21.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.294+0000 7f8eea7fc700 1 -- 172.21.15.16:0/1065184032 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f8eec003680 con 0x7f8ef80b3a40 2023-10-16T12:10:21.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.296+0000 7f8eea7fc700 1 -- 172.21.15.16:0/1065184032 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f8ef80b52a0 con 0x7f8ef80b3a40 2023-10-16T12:10:21.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.372+0000 7f8efffdf700 1 -- 172.21.15.16:0/1065184032 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f8ef8155910 con 0x7f8ef80b3a40 2023-10-16T12:10:21.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.374+0000 7f8eea7fc700 1 -- 172.21.15.16:0/1065184032 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f8eec02b060 con 0x7f8ef80b3a40 2023-10-16T12:10:21.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.375+0000 7f8efffdf700 1 -- 172.21.15.16:0/1065184032 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ef80b3a40 msgr2=0x7f8ef80b3ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:21.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.375+0000 7f8efffdf700 1 --2- 172.21.15.16:0/1065184032 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ef80b3a40 0x7f8ef80b3ec0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f8eec000c00 tx=0x7f8eec000fe0 comp rx=0 tx=0).stop 2023-10-16T12:10:21.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.375+0000 7f8efffdf700 1 -- 172.21.15.16:0/1065184032 shutdown_connections 2023-10-16T12:10:21.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.375+0000 7f8efffdf700 1 --2- 172.21.15.16:0/1065184032 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8ef80b0850 0x7f8ef80b0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:21.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.375+0000 7f8efffdf700 1 --2- 172.21.15.16:0/1065184032 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8ee4004030 0x7f8ef8151d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:21.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.375+0000 7f8efffdf700 1 --2- 172.21.15.16:0/1065184032 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ef80b3a40 0x7f8ef80b3ec0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:21.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.375+0000 7f8efffdf700 1 -- 172.21.15.16:0/1065184032 >> 172.21.15.16:0/1065184032 conn(0x7f8ef80b1f40 msgr2=0x7f8ef81544a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:21.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.375+0000 7f8efffdf700 1 -- 172.21.15.16:0/1065184032 shutdown_connections 2023-10-16T12:10:21.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:21.375+0000 7f8efffdf700 1 -- 172.21.15.16:0/1065184032 wait complete. 2023-10-16T12:10:21.379 INFO:teuthology.orchestra.run.smithi016.stderr:adjusted tunables profile to default 2023-10-16T12:10:21.389 INFO:tasks.ceph:check_enable_crimson: False 2023-10-16T12:10:21.389 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-10-16T12:10:21.389 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:10:21.390 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-10-16T12:10:21.405 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:10:21.403+0000 7f77e25c4200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T12:10:21.407 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:10:21.406+0000 7ff76d9e4200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T12:10:21.417 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:10:21.417 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-10-16T12:10:21.473 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:10:21.471+0000 7f77e25c4200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T12:10:21.478 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:10:21.476+0000 7ff76d9e4200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T12:10:21.483 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:10:21.483 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-10-16T12:10:21.550 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:10:21.551 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-10-16T12:10:21.554 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:10:21.552+0000 7f77e25c4200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T12:10:21.561 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:10:21.560+0000 7ff76d9e4200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T12:10:21.588 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T12:10:21.588 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-10-16T12:10:21.616 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T12:10:21.616 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-10-16T12:10:21.638 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:10:21.636+0000 7f77e25c4200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T12:10:21.647 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:10:21.645+0000 7ff76d9e4200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T12:10:21.664 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T12:10:21.664 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-10-16T12:10:21.732 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T12:10:21.732 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-10-16T12:10:21.790 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:10:21.788+0000 7f77e25c4200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T12:10:21.795 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:10:21.794+0000 7ff76d9e4200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T12:10:21.799 DEBUG:teuthology.orchestra.run.smithi049:> sudo ceph --cluster ceph osd new 3d176986-e18f-4c8c-b318-ce9319b69906 0 2023-10-16T12:10:21.964 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:21.962+0000 7f3f75d17700 1 Processor -- start 2023-10-16T12:10:21.964 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:21.962+0000 7f3f75d17700 1 -- start start 2023-10-16T12:10:21.966 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:21.962+0000 7f3f75d17700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f3f7015b070 0x7f3f7015b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:21.967 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:21.963+0000 7f3f75d17700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3f700cb880 0x7f3f700cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:21.967 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:21.963+0000 7f3f75d17700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f700cef70 con 0x7f3f7006da60 2023-10-16T12:10:21.967 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:21.963+0000 7f3f75d17700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f700ce360 con 0x7f3f7015b070 2023-10-16T12:10:21.967 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:21.963+0000 7f3f75d17700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f700ce640 con 0x7f3f700cb880 2023-10-16T12:10:21.968 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:21.963+0000 7f3f6effd700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3f700cb880 0x7f3f700cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:21.968 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:21.963+0000 7f3f6effd700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3f700cb880 0x7f3f700cbd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.49:42552/0 (socket says 172.21.15.49:42552) 2023-10-16T12:10:21.968 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:21.963+0000 7f3f6effd700 1 -- 172.21.15.49:0/3394434283 learned_addr learned my addr 172.21.15.49:0/3394434283 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-10-16T12:10:21.968 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:21.963+0000 7f3f6f7fe700 1 --2- 172.21.15.49:0/3394434283 >> v2:172.21.15.49:3300/0 conn(0x7f3f7015b070 0x7f3f7015b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:21.968 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:21.963+0000 7f3f6effd700 1 -- 172.21.15.49:0/3394434283 >> v2:172.21.15.49:3300/0 conn(0x7f3f7015b070 msgr2=0x7f3f7015b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:10:21.969 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:21.963+0000 7f3f6effd700 1 --2- 172.21.15.49:0/3394434283 >> v2:172.21.15.49:3300/0 conn(0x7f3f7015b070 0x7f3f7015b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:21.969 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:21.963+0000 7f3f6effd700 1 -- 172.21.15.49:0/3394434283 >> v1:172.21.15.49:6789/0 conn(0x7f3f7006da60 legacy=0x7f3f700cf400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:21.969 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:21.963+0000 7f3f6effd700 1 -- 172.21.15.49:0/3394434283 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f700ce920 con 0x7f3f700cb880 2023-10-16T12:10:21.969 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:21.963+0000 7f3f6f7fe700 1 --2- 172.21.15.49:0/3394434283 >> v2:172.21.15.49:3300/0 conn(0x7f3f7015b070 0x7f3f7015b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:10:21.969 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:21.963+0000 7f3f6effd700 1 --2- 172.21.15.49:0/3394434283 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3f700cb880 0x7f3f700cbd00 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3f6400bb90 tx=0x7f3f6400bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=61627d030c725801 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:21.969 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:21.963+0000 7f3f6e7fc700 1 -- 172.21.15.49:0/3394434283 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f6400dd30 con 0x7f3f700cb880 2023-10-16T12:10:21.970 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:21.963+0000 7f3f6e7fc700 1 -- 172.21.15.49:0/3394434283 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f6400de90 con 0x7f3f700cb880 2023-10-16T12:10:21.970 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:21.963+0000 7f3f6e7fc700 1 -- 172.21.15.49:0/3394434283 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f64010690 con 0x7f3f700cb880 2023-10-16T12:10:21.970 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:21.964+0000 7f3f75d17700 1 -- 172.21.15.49:0/3394434283 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3f700cb880 msgr2=0x7f3f700cbd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:21.970 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:21.964+0000 7f3f75d17700 1 --2- 172.21.15.49:0/3394434283 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3f700cb880 0x7f3f700cbd00 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3f6400bb90 tx=0x7f3f6400bea0 comp rx=0 tx=0).stop 2023-10-16T12:10:21.970 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:21.964+0000 7f3f75d17700 1 -- 172.21.15.49:0/3394434283 shutdown_connections 2023-10-16T12:10:21.971 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:21.964+0000 7f3f75d17700 1 --2- 172.21.15.49:0/3394434283 >> v2:172.21.15.49:3300/0 conn(0x7f3f7015b070 0x7f3f7015b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:21.971 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:21.964+0000 7f3f75d17700 1 --2- 172.21.15.49:0/3394434283 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3f700cb880 0x7f3f700cbd00 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:21.971 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:21.964+0000 7f3f75d17700 1 -- 172.21.15.49:0/3394434283 >> 172.21.15.49:0/3394434283 conn(0x7f3f700b1f40 msgr2=0x7f3f700b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:21.971 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:21.964+0000 7f3f75d17700 1 -- 172.21.15.49:0/3394434283 shutdown_connections 2023-10-16T12:10:21.971 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:21.964+0000 7f3f75d17700 1 -- 172.21.15.49:0/3394434283 wait complete. 2023-10-16T12:10:21.973 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:21.965+0000 7f3f75d17700 1 Processor -- start 2023-10-16T12:10:21.974 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:21.965+0000 7f3f75d17700 1 -- start start 2023-10-16T12:10:21.974 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:21.966+0000 7f3f75d17700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f7006da60 0x7f3f700c9f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:21.974 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:21.966+0000 7f3f75d17700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3f700cb880 0x7f3f700ad1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:21.974 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:21.966+0000 7f3f75d17700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3f7015b070 0x7f3f700ad700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:21.975 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:21.966+0000 7f3f75d17700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f70151400 con 0x7f3f7006da60 2023-10-16T12:10:21.975 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:21.966+0000 7f3f75d17700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f7015c830 con 0x7f3f7015b070 2023-10-16T12:10:21.975 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:21.966+0000 7f3f75d17700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f7015d690 con 0x7f3f700cb880 2023-10-16T12:10:21.975 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:21.966+0000 7f3f6effd700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3f700cb880 0x7f3f700ad1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:21.975 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:21.966+0000 7f3f6ffff700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3f7015b070 0x7f3f700ad700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:21.975 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:21.966+0000 7f3f6effd700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3f700cb880 0x7f3f700ad1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.49:42558/0 (socket says 172.21.15.49:42558) 2023-10-16T12:10:21.976 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:21.966+0000 7f3f6effd700 1 -- 172.21.15.49:0/2003099084 learned_addr learned my addr 172.21.15.49:0/2003099084 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-10-16T12:10:21.976 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:21.966+0000 7f3f6f7fe700 1 --2- 172.21.15.49:0/2003099084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f7006da60 0x7f3f700c9f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:21.976 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:21.967+0000 7f3f6ffff700 1 -- 172.21.15.49:0/2003099084 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3f700cb880 msgr2=0x7f3f700ad1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:21.976 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:21.967+0000 7f3f6ffff700 1 --2- 172.21.15.49:0/2003099084 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3f700cb880 0x7f3f700ad1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:21.976 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:21.967+0000 7f3f6ffff700 1 -- 172.21.15.49:0/2003099084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f7006da60 msgr2=0x7f3f700c9f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:21.977 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:21.967+0000 7f3f6ffff700 1 --2- 172.21.15.49:0/2003099084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f7006da60 0x7f3f700c9f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:21.977 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:21.967+0000 7f3f6ffff700 1 -- 172.21.15.49:0/2003099084 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f600094a0 con 0x7f3f7015b070 2023-10-16T12:10:21.977 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:21.967+0000 7f3f6effd700 1 --2- 172.21.15.49:0/2003099084 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3f700cb880 0x7f3f700ad1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:10:21.977 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:21.967+0000 7f3f6ffff700 1 --2- 172.21.15.49:0/2003099084 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3f7015b070 0x7f3f700ad700 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3f6800cbc0 tx=0x7f3f6800ced0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:21.977 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:21.967+0000 7f3f57fff700 1 -- 172.21.15.49:0/2003099084 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f68003c00 con 0x7f3f7015b070 2023-10-16T12:10:21.978 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:21.967+0000 7f3f57fff700 1 -- 172.21.15.49:0/2003099084 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f68003d60 con 0x7f3f7015b070 2023-10-16T12:10:21.978 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:21.967+0000 7f3f75d17700 1 -- 172.21.15.49:0/2003099084 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f6400b7f0 con 0x7f3f7015b070 2023-10-16T12:10:21.978 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:21.967+0000 7f3f57fff700 1 -- 172.21.15.49:0/2003099084 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f6800aa40 con 0x7f3f7015b070 2023-10-16T12:10:21.978 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:21.967+0000 7f3f75d17700 1 -- 172.21.15.49:0/2003099084 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f7015cd60 con 0x7f3f7015b070 2023-10-16T12:10:21.978 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:21.967+0000 7f3f57fff700 1 -- 172.21.15.49:0/2003099084 <== mon.1 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f3f680050d0 con 0x7f3f7015b070 2023-10-16T12:10:21.978 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:21.967+0000 7f3f57fff700 1 -- 172.21.15.49:0/2003099084 <== mon.1 v2:172.21.15.49:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7f3f68012850 con 0x7f3f7015b070 2023-10-16T12:10:21.979 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:21.968+0000 7f3f75d17700 1 -- 172.21.15.49:0/2003099084 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f7015da20 con 0x7f3f7015b070 2023-10-16T12:10:21.979 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:21.971+0000 7f3f57fff700 1 -- 172.21.15.49:0/2003099084 <== mon.1 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f3f7015da20 con 0x7f3f7015b070 2023-10-16T12:10:22.058 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.056+0000 7f3f75d17700 1 -- 172.21.15.49:0/2003099084 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd new", "uuid": "3d176986-e18f-4c8c-b318-ce9319b69906", "id": 0} v 0) v1 -- 0x7f3f7004ebb0 con 0x7f3f7015b070 2023-10-16T12:10:22.061 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.059+0000 7f3f57fff700 1 -- 172.21.15.49:0/2003099084 <== mon.1 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3d176986-e18f-4c8c-b318-ce9319b69906", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f3f68005510 con 0x7f3f7015b070 2023-10-16T12:10:22.061 INFO:teuthology.orchestra.run.smithi049.stdout:0 2023-10-16T12:10:22.063 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.060+0000 7f3f75d17700 1 -- 172.21.15.49:0/2003099084 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3f7015b070 msgr2=0x7f3f700ad700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:22.063 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.060+0000 7f3f75d17700 1 --2- 172.21.15.49:0/2003099084 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3f7015b070 0x7f3f700ad700 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3f6800cbc0 tx=0x7f3f6800ced0 comp rx=0 tx=0).stop 2023-10-16T12:10:22.063 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.060+0000 7f3f75d17700 1 -- 172.21.15.49:0/2003099084 shutdown_connections 2023-10-16T12:10:22.063 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.060+0000 7f3f75d17700 1 --2- 172.21.15.49:0/2003099084 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3f7015b070 0x7f3f700ad700 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:22.063 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.060+0000 7f3f75d17700 1 --2- 172.21.15.49:0/2003099084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f7006da60 0x7f3f700c9f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:22.063 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.060+0000 7f3f75d17700 1 --2- 172.21.15.49:0/2003099084 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3f700cb880 0x7f3f700ad1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:22.064 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.060+0000 7f3f75d17700 1 -- 172.21.15.49:0/2003099084 >> 172.21.15.49:0/2003099084 conn(0x7f3f700b1f40 msgr2=0x7f3f70154650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:22.064 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.060+0000 7f3f75d17700 1 -- 172.21.15.49:0/2003099084 shutdown_connections 2023-10-16T12:10:22.064 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.060+0000 7f3f75d17700 1 -- 172.21.15.49:0/2003099084 wait complete. 2023-10-16T12:10:22.074 DEBUG:teuthology.orchestra.run.smithi049:> sudo ceph --cluster ceph osd new 93e01cb9-fe53-4210-a128-3cdf71d0420e 1 2023-10-16T12:10:22.198 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.196+0000 7f203ac8d700 1 Processor -- start 2023-10-16T12:10:22.198 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.196+0000 7f203ac8d700 1 -- start start 2023-10-16T12:10:22.201 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.198+0000 7f203ac8d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f203415b070 0x7f203415b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:22.201 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.198+0000 7f203ac8d700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f203406e170 0x7f20340cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:22.201 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.198+0000 7f203ac8d700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20340cef70 con 0x7f20340cb880 2023-10-16T12:10:22.201 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.198+0000 7f203ac8d700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20340ce360 con 0x7f203415b070 2023-10-16T12:10:22.202 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.198+0000 7f203ac8d700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20340ce640 con 0x7f203406e170 2023-10-16T12:10:22.202 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.198+0000 7f203922a700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f203406e170 0x7f20340cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:22.202 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.198+0000 7f203922a700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f203406e170 0x7f20340cbd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:52560/0 (socket says 172.21.15.49:52560) 2023-10-16T12:10:22.202 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.198+0000 7f203922a700 1 -- 172.21.15.49:0/751849439 learned_addr learned my addr 172.21.15.49:0/751849439 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-10-16T12:10:22.203 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.198+0000 7f2033fff700 1 --1- 172.21.15.49:0/751849439 >> v1:172.21.15.49:6789/0 conn(0x7f20340cb880 0x7f203406da60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.49:34926/0 (socket says 172.21.15.49:34926) 2023-10-16T12:10:22.203 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.198+0000 7f2038a29700 1 --2- 172.21.15.49:0/751849439 >> v2:172.21.15.16:3300/0 conn(0x7f203415b070 0x7f203415b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:22.203 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.198+0000 7f203922a700 1 -- 172.21.15.49:0/751849439 >> v2:172.21.15.16:3300/0 conn(0x7f203415b070 msgr2=0x7f203415b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:10:22.203 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.198+0000 7f203922a700 1 --2- 172.21.15.49:0/751849439 >> v2:172.21.15.16:3300/0 conn(0x7f203415b070 0x7f203415b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:22.203 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.198+0000 7f203922a700 1 -- 172.21.15.49:0/751849439 >> v1:172.21.15.49:6789/0 conn(0x7f20340cb880 legacy=0x7f203406da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:22.203 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.198+0000 7f20337fe700 1 -- 172.21.15.49:0/751849439 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4007661239 0 0) 0x7f20340cef70 con 0x7f20340cb880 2023-10-16T12:10:22.204 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.198+0000 7f203922a700 1 -- 172.21.15.49:0/751849439 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20340ce920 con 0x7f203406e170 2023-10-16T12:10:22.204 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.198+0000 7f203922a700 1 --2- 172.21.15.49:0/751849439 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f203406e170 0x7f20340cbd00 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f203415aad0 tx=0x7f202c00d430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=479b6831394111c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:22.204 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.198+0000 7f20337fe700 1 -- 172.21.15.49:0/751849439 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f202c010070 con 0x7f203406e170 2023-10-16T12:10:22.204 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.198+0000 7f20337fe700 1 -- 172.21.15.49:0/751849439 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f202c00dce0 con 0x7f203406e170 2023-10-16T12:10:22.204 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.198+0000 7f20337fe700 1 -- 172.21.15.49:0/751849439 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f202c00b310 con 0x7f203406e170 2023-10-16T12:10:22.205 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:10:22.200+0000 7ff76d9e4200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T12:10:22.208 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.199+0000 7f203ac8d700 1 -- 172.21.15.49:0/751849439 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f203406e170 msgr2=0x7f20340cbd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:22.208 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.199+0000 7f203ac8d700 1 --2- 172.21.15.49:0/751849439 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f203406e170 0x7f20340cbd00 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f203415aad0 tx=0x7f202c00d430 comp rx=0 tx=0).stop 2023-10-16T12:10:22.208 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.199+0000 7f203ac8d700 1 -- 172.21.15.49:0/751849439 shutdown_connections 2023-10-16T12:10:22.209 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.199+0000 7f203ac8d700 1 --2- 172.21.15.49:0/751849439 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f203406e170 0x7f20340cbd00 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:22.209 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.199+0000 7f203ac8d700 1 --2- 172.21.15.49:0/751849439 >> v2:172.21.15.16:3300/0 conn(0x7f203415b070 0x7f203415b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:22.209 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.199+0000 7f203ac8d700 1 -- 172.21.15.49:0/751849439 >> 172.21.15.49:0/751849439 conn(0x7f20340b1f40 msgr2=0x7f20340b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:22.209 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.199+0000 7f203ac8d700 1 -- 172.21.15.49:0/751849439 shutdown_connections 2023-10-16T12:10:22.209 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.199+0000 7f203ac8d700 1 -- 172.21.15.49:0/751849439 wait complete. 2023-10-16T12:10:22.209 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.201+0000 7f203ac8d700 1 Processor -- start 2023-10-16T12:10:22.210 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.201+0000 7f203ac8d700 1 -- start start 2023-10-16T12:10:22.210 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.201+0000 7f203ac8d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f203406e170 0x7f203415a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:22.210 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.201+0000 7f203ac8d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f20340cb880 0x7f20341586c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:22.210 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.201+0000 7f203ac8d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f203415b070 0x7f20340c6000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:22.210 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.201+0000 7f203ac8d700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2034156ef0 con 0x7f203415b070 2023-10-16T12:10:22.210 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.202+0000 7f203ac8d700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20341595d0 con 0x7f20340cb880 2023-10-16T12:10:22.211 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.202+0000 7f203ac8d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2034158d30 con 0x7f203406e170 2023-10-16T12:10:22.211 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.202+0000 7f2033fff700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f20340cb880 0x7f20341586c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:22.211 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.202+0000 7f2038a29700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f203406e170 0x7f203415a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:22.211 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.202+0000 7f2033fff700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f20340cb880 0x7f20341586c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:52564/0 (socket says 172.21.15.49:52564) 2023-10-16T12:10:22.211 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.202+0000 7f2033fff700 1 -- 172.21.15.49:0/2333325470 learned_addr learned my addr 172.21.15.49:0/2333325470 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-10-16T12:10:22.212 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.202+0000 7f203922a700 1 --2- 172.21.15.49:0/2333325470 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f203415b070 0x7f20340c6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:22.212 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.202+0000 7f2033fff700 1 -- 172.21.15.49:0/2333325470 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f203406e170 msgr2=0x7f203415a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:22.212 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.202+0000 7f2033fff700 1 --2- 172.21.15.49:0/2333325470 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f203406e170 0x7f203415a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:22.212 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.202+0000 7f2033fff700 1 -- 172.21.15.49:0/2333325470 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f203415b070 msgr2=0x7f20340c6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:22.212 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.202+0000 7f2033fff700 1 --2- 172.21.15.49:0/2333325470 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f203415b070 0x7f20340c6000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:22.212 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.202+0000 7f2033fff700 1 -- 172.21.15.49:0/2333325470 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f202c00bd40 con 0x7f20340cb880 2023-10-16T12:10:22.213 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.202+0000 7f2033fff700 1 --2- 172.21.15.49:0/2333325470 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f20340cb880 0x7f20341586c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f202000c750 tx=0x7f202000ca60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:22.213 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.202+0000 7f20317fa700 1 -- 172.21.15.49:0/2333325470 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2020004c80 con 0x7f20340cb880 2023-10-16T12:10:22.213 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.202+0000 7f20317fa700 1 -- 172.21.15.49:0/2333325470 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2020004e00 con 0x7f20340cb880 2023-10-16T12:10:22.213 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.202+0000 7f203ac8d700 1 -- 172.21.15.49:0/2333325470 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2034159010 con 0x7f20340cb880 2023-10-16T12:10:22.213 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.202+0000 7f20317fa700 1 -- 172.21.15.49:0/2333325470 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2020007310 con 0x7f20340cb880 2023-10-16T12:10:22.213 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.202+0000 7f203ac8d700 1 -- 172.21.15.49:0/2333325470 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2034155230 con 0x7f20340cb880 2023-10-16T12:10:22.214 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.202+0000 7f20317fa700 1 -- 172.21.15.49:0/2333325470 <== mon.1 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f202001d3f0 con 0x7f20340cb880 2023-10-16T12:10:22.214 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.203+0000 7f20317fa700 1 -- 172.21.15.49:0/2333325470 <== mon.1 v2:172.21.15.49:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f2020013770 con 0x7f20340cb880 2023-10-16T12:10:22.214 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.203+0000 7f203ac8d700 1 -- 172.21.15.49:0/2333325470 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2034155450 con 0x7f20340cb880 2023-10-16T12:10:22.214 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.205+0000 7f20317fa700 1 -- 172.21.15.49:0/2333325470 <== mon.1 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f2020013a30 con 0x7f20340cb880 2023-10-16T12:10:22.217 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:10:22.215+0000 7f77e25c4200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T12:10:22.293 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.291+0000 7f203ac8d700 1 -- 172.21.15.49:0/2333325470 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd new", "uuid": "93e01cb9-fe53-4210-a128-3cdf71d0420e", "id": 1} v 0) v1 -- 0x7f203404ebb0 con 0x7f20340cb880 2023-10-16T12:10:22.296 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.294+0000 7f20317fa700 1 -- 172.21.15.49:0/2333325470 <== mon.1 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "93e01cb9-fe53-4210-a128-3cdf71d0420e", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f2020013ca0 con 0x7f20340cb880 2023-10-16T12:10:22.297 INFO:teuthology.orchestra.run.smithi049.stdout:1 2023-10-16T12:10:22.298 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.295+0000 7f203ac8d700 1 -- 172.21.15.49:0/2333325470 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f20340cb880 msgr2=0x7f20341586c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:22.298 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.295+0000 7f203ac8d700 1 --2- 172.21.15.49:0/2333325470 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f20340cb880 0x7f20341586c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f202000c750 tx=0x7f202000ca60 comp rx=0 tx=0).stop 2023-10-16T12:10:22.298 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.295+0000 7f203ac8d700 1 -- 172.21.15.49:0/2333325470 shutdown_connections 2023-10-16T12:10:22.298 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.295+0000 7f203ac8d700 1 --2- 172.21.15.49:0/2333325470 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f203415b070 0x7f20340c6000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:22.299 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.295+0000 7f203ac8d700 1 --2- 172.21.15.49:0/2333325470 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f203406e170 0x7f203415a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:22.299 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.295+0000 7f203ac8d700 1 --2- 172.21.15.49:0/2333325470 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f20340cb880 0x7f20341586c0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:22.299 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.295+0000 7f203ac8d700 1 -- 172.21.15.49:0/2333325470 >> 172.21.15.49:0/2333325470 conn(0x7f20340b1f40 msgr2=0x7f203415e050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:22.299 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.295+0000 7f203ac8d700 1 -- 172.21.15.49:0/2333325470 shutdown_connections 2023-10-16T12:10:22.299 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.295+0000 7f203ac8d700 1 -- 172.21.15.49:0/2333325470 wait complete. 2023-10-16T12:10:22.310 DEBUG:teuthology.orchestra.run.smithi049:> sudo ceph --cluster ceph osd new ff4dc68c-c44f-480d-bc8c-2264bf034517 2 2023-10-16T12:10:22.324 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:10:22.322+0000 7ff76d9e4200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T12:10:22.342 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:10:22.340+0000 7f77e25c4200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T12:10:22.417 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:10:22.415+0000 7ff76d9e4200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T12:10:22.434 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:10:22.432+0000 7f77e25c4200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T12:10:22.435 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.433+0000 7fd36df72700 1 Processor -- start 2023-10-16T12:10:22.435 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.433+0000 7fd36df72700 1 -- start start 2023-10-16T12:10:22.438 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.434+0000 7fd36df72700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fd36815b2c0 0x7fd368161340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:22.438 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.434+0000 7fd36df72700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd36806e570 0x7fd36815e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:22.438 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.434+0000 7fd36df72700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd36806a7e0 0x7fd36806ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:22.439 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.434+0000 7fd36df72700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd36806b1b0 con 0x7fd36806a7e0 2023-10-16T12:10:22.439 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.434+0000 7fd36df72700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd368161880 con 0x7fd36815b2c0 2023-10-16T12:10:22.439 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.434+0000 7fd36df72700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd36815b6a0 con 0x7fd36806e570 2023-10-16T12:10:22.439 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.434+0000 7fd366ffd700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd36806e570 0x7fd36815e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:22.439 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.434+0000 7fd3677fe700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fd36815b2c0 0x7fd368161340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:22.440 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.434+0000 7fd3677fe700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fd36815b2c0 0x7fd368161340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:52568/0 (socket says 172.21.15.49:52568) 2023-10-16T12:10:22.440 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.434+0000 7fd366ffd700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd36806e570 0x7fd36815e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.49:42574/0 (socket says 172.21.15.49:42574) 2023-10-16T12:10:22.440 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.434+0000 7fd3677fe700 1 -- 172.21.15.49:0/346563284 learned_addr learned my addr 172.21.15.49:0/346563284 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-10-16T12:10:22.440 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.434+0000 7fd367fff700 1 --2- 172.21.15.49:0/346563284 >> v2:172.21.15.16:3300/0 conn(0x7fd36806a7e0 0x7fd36806ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:22.440 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.435+0000 7fd366ffd700 1 -- 172.21.15.49:0/346563284 >> v2:172.21.15.49:3300/0 conn(0x7fd36815b2c0 msgr2=0x7fd368161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:10:22.440 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.435+0000 7fd366ffd700 1 --2- 172.21.15.49:0/346563284 >> v2:172.21.15.49:3300/0 conn(0x7fd36815b2c0 0x7fd368161340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:22.441 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.435+0000 7fd366ffd700 1 -- 172.21.15.49:0/346563284 >> v2:172.21.15.16:3300/0 conn(0x7fd36806a7e0 msgr2=0x7fd36806ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:10:22.441 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.435+0000 7fd366ffd700 1 --2- 172.21.15.49:0/346563284 >> v2:172.21.15.16:3300/0 conn(0x7fd36806a7e0 0x7fd36806ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:22.441 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.435+0000 7fd366ffd700 1 -- 172.21.15.49:0/346563284 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd368161b20 con 0x7fd36806e570 2023-10-16T12:10:22.441 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.435+0000 7fd3677fe700 1 --2- 172.21.15.49:0/346563284 >> v2:172.21.15.49:3300/0 conn(0x7fd36815b2c0 0x7fd368161340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:10:22.441 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.435+0000 7fd366ffd700 1 --2- 172.21.15.49:0/346563284 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd36806e570 0x7fd36815e5a0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fd36806b4a0 tx=0x7fd350009b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4245f7b575fc83e5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:22.442 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.435+0000 7fd3667fc700 1 -- 172.21.15.49:0/346563284 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd350011830 con 0x7fd36806e570 2023-10-16T12:10:22.442 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.435+0000 7fd3667fc700 1 -- 172.21.15.49:0/346563284 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd350011990 con 0x7fd36806e570 2023-10-16T12:10:22.442 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.435+0000 7fd3667fc700 1 -- 172.21.15.49:0/346563284 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd350011d20 con 0x7fd36806e570 2023-10-16T12:10:22.442 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.436+0000 7fd36df72700 1 -- 172.21.15.49:0/346563284 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd36806e570 msgr2=0x7fd36815e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:22.442 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.436+0000 7fd36df72700 1 --2- 172.21.15.49:0/346563284 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd36806e570 0x7fd36815e5a0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fd36806b4a0 tx=0x7fd350009b20 comp rx=0 tx=0).stop 2023-10-16T12:10:22.442 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.436+0000 7fd36df72700 1 -- 172.21.15.49:0/346563284 shutdown_connections 2023-10-16T12:10:22.443 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.436+0000 7fd36df72700 1 --2- 172.21.15.49:0/346563284 >> v2:172.21.15.16:3300/0 conn(0x7fd36806a7e0 0x7fd36806ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:22.443 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.436+0000 7fd36df72700 1 --2- 172.21.15.49:0/346563284 >> v2:172.21.15.49:3300/0 conn(0x7fd36815b2c0 0x7fd368161340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:22.443 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.436+0000 7fd36df72700 1 --2- 172.21.15.49:0/346563284 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd36806e570 0x7fd36815e5a0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:22.443 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.436+0000 7fd36df72700 1 -- 172.21.15.49:0/346563284 >> 172.21.15.49:0/346563284 conn(0x7fd3680b1f40 msgr2=0x7fd3680b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:22.443 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.436+0000 7fd36df72700 1 -- 172.21.15.49:0/346563284 shutdown_connections 2023-10-16T12:10:22.444 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.436+0000 7fd36df72700 1 -- 172.21.15.49:0/346563284 wait complete. 2023-10-16T12:10:22.446 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.438+0000 7fd36df72700 1 Processor -- start 2023-10-16T12:10:22.446 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.438+0000 7fd36df72700 1 -- start start 2023-10-16T12:10:22.446 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.439+0000 7fd36df72700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd36806a7e0 0x7fd3681f29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:22.447 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.439+0000 7fd36df72700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd36806e570 0x7fd3680b4510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:22.447 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.439+0000 7fd36df72700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd36815b2c0 0x7fd3680d0030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:22.447 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.439+0000 7fd36df72700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3680b3fe0 con 0x7fd36806e570 2023-10-16T12:10:22.447 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.439+0000 7fd36df72700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3680b3b50 con 0x7fd36815b2c0 2023-10-16T12:10:22.447 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.439+0000 7fd36df72700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd36814b0e0 con 0x7fd36806a7e0 2023-10-16T12:10:22.448 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.439+0000 7fd3677fe700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd36806a7e0 0x7fd3681f29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:22.448 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.439+0000 7fd367fff700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd36815b2c0 0x7fd3680d0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:22.448 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.439+0000 7fd3677fe700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd36806a7e0 0x7fd3681f29e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.49:42580/0 (socket says 172.21.15.49:42580) 2023-10-16T12:10:22.448 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.439+0000 7fd3677fe700 1 -- 172.21.15.49:0/3438062998 learned_addr learned my addr 172.21.15.49:0/3438062998 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-10-16T12:10:22.448 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.439+0000 7fd366ffd700 1 --2- 172.21.15.49:0/3438062998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd36806e570 0x7fd3680b4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:22.449 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.440+0000 7fd366ffd700 1 -- 172.21.15.49:0/3438062998 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd36806a7e0 msgr2=0x7fd3681f29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:22.449 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.440+0000 7fd366ffd700 1 --2- 172.21.15.49:0/3438062998 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd36806a7e0 0x7fd3681f29e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:22.449 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.440+0000 7fd366ffd700 1 -- 172.21.15.49:0/3438062998 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd36815b2c0 msgr2=0x7fd3680d0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:22.449 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.440+0000 7fd366ffd700 1 --2- 172.21.15.49:0/3438062998 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd36815b2c0 0x7fd3680d0030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:22.449 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.440+0000 7fd366ffd700 1 -- 172.21.15.49:0/3438062998 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd358009b30 con 0x7fd36806e570 2023-10-16T12:10:22.449 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.440+0000 7fd3677fe700 1 --2- 172.21.15.49:0/3438062998 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd36806a7e0 0x7fd3681f29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:10:22.450 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.440+0000 7fd367fff700 1 --2- 172.21.15.49:0/3438062998 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd36815b2c0 0x7fd3680d0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:10:22.450 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.440+0000 7fd366ffd700 1 --2- 172.21.15.49:0/3438062998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd36806e570 0x7fd3680b4510 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd3680ca970 tx=0x7fd350004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:22.450 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.440+0000 7fd35ffff700 1 -- 172.21.15.49:0/3438062998 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd350004450 con 0x7fd36806e570 2023-10-16T12:10:22.450 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.440+0000 7fd35ffff700 1 -- 172.21.15.49:0/3438062998 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd35001e070 con 0x7fd36806e570 2023-10-16T12:10:22.450 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.440+0000 7fd36df72700 1 -- 172.21.15.49:0/3438062998 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3500094a0 con 0x7fd36806e570 2023-10-16T12:10:22.450 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.440+0000 7fd35ffff700 1 -- 172.21.15.49:0/3438062998 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3500038e0 con 0x7fd36806e570 2023-10-16T12:10:22.451 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.440+0000 7fd36df72700 1 -- 172.21.15.49:0/3438062998 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3680b2fb0 con 0x7fd36806e570 2023-10-16T12:10:22.451 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.440+0000 7fd35ffff700 1 -- 172.21.15.49:0/3438062998 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fd350003d40 con 0x7fd36806e570 2023-10-16T12:10:22.451 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.440+0000 7fd35ffff700 1 -- 172.21.15.49:0/3438062998 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7fd350024400 con 0x7fd36806e570 2023-10-16T12:10:22.451 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.441+0000 7fd36df72700 1 -- 172.21.15.49:0/3438062998 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3680b3ce0 con 0x7fd36806e570 2023-10-16T12:10:22.451 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.443+0000 7fd35ffff700 1 -- 172.21.15.49:0/3438062998 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7fd350032060 con 0x7fd36806e570 2023-10-16T12:10:22.502 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:10:22.501+0000 7ff76d9e4200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T12:10:22.522 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:10:22.520+0000 7f77e25c4200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T12:10:22.531 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.529+0000 7fd36df72700 1 -- 172.21.15.49:0/3438062998 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd new", "uuid": "ff4dc68c-c44f-480d-bc8c-2264bf034517", "id": 2} v 0) v1 -- 0x7fd3681620b0 con 0x7fd36806e570 2023-10-16T12:10:22.533 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.531+0000 7fd35ffff700 1 -- 172.21.15.49:0/3438062998 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ff4dc68c-c44f-480d-bc8c-2264bf034517", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7fd350020070 con 0x7fd36806e570 2023-10-16T12:10:22.533 INFO:teuthology.orchestra.run.smithi049.stdout:2 2023-10-16T12:10:22.534 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.532+0000 7fd36df72700 1 -- 172.21.15.49:0/3438062998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd36806e570 msgr2=0x7fd3680b4510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:22.535 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.532+0000 7fd36df72700 1 --2- 172.21.15.49:0/3438062998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd36806e570 0x7fd3680b4510 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd3680ca970 tx=0x7fd350004750 comp rx=0 tx=0).stop 2023-10-16T12:10:22.535 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.532+0000 7fd36df72700 1 -- 172.21.15.49:0/3438062998 shutdown_connections 2023-10-16T12:10:22.535 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.532+0000 7fd36df72700 1 --2- 172.21.15.49:0/3438062998 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd36815b2c0 0x7fd3680d0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:22.535 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.532+0000 7fd36df72700 1 --2- 172.21.15.49:0/3438062998 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd36806a7e0 0x7fd3681f29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:22.535 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.532+0000 7fd36df72700 1 --2- 172.21.15.49:0/3438062998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd36806e570 0x7fd3680b4510 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:22.535 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.532+0000 7fd36df72700 1 -- 172.21.15.49:0/3438062998 >> 172.21.15.49:0/3438062998 conn(0x7fd3680b1f40 msgr2=0x7fd3680cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:22.536 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.532+0000 7fd36df72700 1 -- 172.21.15.49:0/3438062998 shutdown_connections 2023-10-16T12:10:22.536 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.532+0000 7fd36df72700 1 -- 172.21.15.49:0/3438062998 wait complete. 2023-10-16T12:10:22.546 DEBUG:teuthology.orchestra.run.smithi049:> sudo ceph --cluster ceph osd new 33c469f5-2ffc-466e-966f-a411ca1e0e29 3 2023-10-16T12:10:22.713 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.711+0000 7f3dcc0f6700 1 Processor -- start 2023-10-16T12:10:22.713 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.711+0000 7f3dcc0f6700 1 -- start start 2023-10-16T12:10:22.715 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.712+0000 7f3dcc0f6700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f3dc415b2c0 0x7f3dc4161340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:22.716 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.712+0000 7f3dcc0f6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3dc406a7e0 0x7f3dc406ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:22.716 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.712+0000 7f3dcc0f6700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dc415ecb0 con 0x7f3dc406e570 2023-10-16T12:10:22.716 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.712+0000 7f3dcc0f6700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3dc406b190 con 0x7f3dc406a7e0 2023-10-16T12:10:22.716 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.712+0000 7f3dcc0f6700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3dc4161880 con 0x7f3dc415b2c0 2023-10-16T12:10:22.716 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.712+0000 7f3dc9e92700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f3dc415b2c0 0x7f3dc4161340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:22.716 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.712+0000 7f3dc9e92700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f3dc415b2c0 0x7f3dc4161340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:52580/0 (socket says 172.21.15.49:52580) 2023-10-16T12:10:22.717 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.713+0000 7f3dc9e92700 1 -- 172.21.15.49:0/620136231 learned_addr learned my addr 172.21.15.49:0/620136231 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-10-16T12:10:22.717 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.713+0000 7f3dca693700 1 --2- 172.21.15.49:0/620136231 >> v2:172.21.15.16:3300/0 conn(0x7f3dc406a7e0 0x7f3dc406ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:22.717 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.713+0000 7f3dc9e92700 1 -- 172.21.15.49:0/620136231 >> v2:172.21.15.16:3300/0 conn(0x7f3dc406a7e0 msgr2=0x7f3dc406ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:10:22.717 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.713+0000 7f3dc9e92700 1 --2- 172.21.15.49:0/620136231 >> v2:172.21.15.16:3300/0 conn(0x7f3dc406a7e0 0x7f3dc406ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:22.717 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.713+0000 7f3dc9e92700 1 -- 172.21.15.49:0/620136231 >> v1:172.21.15.49:6789/0 conn(0x7f3dc406e570 legacy=0x7f3dc415e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:22.718 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.713+0000 7f3dc9e92700 1 -- 172.21.15.49:0/620136231 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dc415b6a0 con 0x7f3dc415b2c0 2023-10-16T12:10:22.718 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.713+0000 7f3dc9e92700 1 --2- 172.21.15.49:0/620136231 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3dc415b2c0 0x7f3dc4161340 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3dc406b4a0 tx=0x7f3dc0009bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de74b79607e616 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:22.718 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.713+0000 7f3dc8e90700 1 -- 172.21.15.49:0/620136231 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dc0011830 con 0x7f3dc415b2c0 2023-10-16T12:10:22.718 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.713+0000 7f3dc8e90700 1 -- 172.21.15.49:0/620136231 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dc0011990 con 0x7f3dc415b2c0 2023-10-16T12:10:22.718 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.713+0000 7f3dc8e90700 1 -- 172.21.15.49:0/620136231 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dc0011d20 con 0x7f3dc415b2c0 2023-10-16T12:10:22.721 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.714+0000 7f3dcc0f6700 1 -- 172.21.15.49:0/620136231 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3dc415b2c0 msgr2=0x7f3dc4161340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:22.721 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.714+0000 7f3dcc0f6700 1 --2- 172.21.15.49:0/620136231 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3dc415b2c0 0x7f3dc4161340 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3dc406b4a0 tx=0x7f3dc0009bd0 comp rx=0 tx=0).stop 2023-10-16T12:10:22.721 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.714+0000 7f3dcc0f6700 1 -- 172.21.15.49:0/620136231 shutdown_connections 2023-10-16T12:10:22.722 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.714+0000 7f3dcc0f6700 1 --2- 172.21.15.49:0/620136231 >> v2:172.21.15.16:3300/0 conn(0x7f3dc406a7e0 0x7f3dc406ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:22.722 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.714+0000 7f3dcc0f6700 1 --2- 172.21.15.49:0/620136231 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3dc415b2c0 0x7f3dc4161340 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:22.722 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.714+0000 7f3dcc0f6700 1 -- 172.21.15.49:0/620136231 >> 172.21.15.49:0/620136231 conn(0x7f3dc40b1f40 msgr2=0x7f3dc40b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:22.722 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.714+0000 7f3dcc0f6700 1 -- 172.21.15.49:0/620136231 shutdown_connections 2023-10-16T12:10:22.722 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.714+0000 7f3dcc0f6700 1 -- 172.21.15.49:0/620136231 wait complete. 2023-10-16T12:10:22.723 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.715+0000 7f3dcc0f6700 1 Processor -- start 2023-10-16T12:10:22.723 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.715+0000 7f3dcc0f6700 1 -- start start 2023-10-16T12:10:22.723 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.716+0000 7f3dcc0f6700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3dc406a7e0 0x7f3dc41541a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:22.723 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.716+0000 7f3dcc0f6700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3dc406e570 0x7f3dc40c7e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:22.723 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.716+0000 7f3dcc0f6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3dc415b2c0 0x7f3dc415fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:22.723 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.716+0000 7f3dcc0f6700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc40c7b20 con 0x7f3dc415b2c0 2023-10-16T12:10:22.724 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.716+0000 7f3dcc0f6700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc40c7560 con 0x7f3dc406a7e0 2023-10-16T12:10:22.724 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.716+0000 7f3dcc0f6700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc40c7840 con 0x7f3dc406e570 2023-10-16T12:10:22.724 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.716+0000 7f3dc9691700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3dc406e570 0x7f3dc40c7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:22.724 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.716+0000 7f3dc9e92700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3dc406a7e0 0x7f3dc41541a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:22.724 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.716+0000 7f3dc9691700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3dc406e570 0x7f3dc40c7e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.49:42594/0 (socket says 172.21.15.49:42594) 2023-10-16T12:10:22.725 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.716+0000 7f3dc9691700 1 -- 172.21.15.49:0/3976156053 learned_addr learned my addr 172.21.15.49:0/3976156053 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-10-16T12:10:22.725 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.716+0000 7f3dca693700 1 --2- 172.21.15.49:0/3976156053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3dc415b2c0 0x7f3dc415fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:22.725 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.716+0000 7f3dc9e92700 1 -- 172.21.15.49:0/3976156053 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3dc406e570 msgr2=0x7f3dc40c7e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:22.725 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.716+0000 7f3dc9e92700 1 --2- 172.21.15.49:0/3976156053 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3dc406e570 0x7f3dc40c7e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:22.725 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.716+0000 7f3dc9e92700 1 -- 172.21.15.49:0/3976156053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3dc415b2c0 msgr2=0x7f3dc415fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:22.725 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.716+0000 7f3dc9e92700 1 --2- 172.21.15.49:0/3976156053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3dc415b2c0 0x7f3dc415fe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:22.726 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.716+0000 7f3dc9e92700 1 -- 172.21.15.49:0/3976156053 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dc0009550 con 0x7f3dc406a7e0 2023-10-16T12:10:22.726 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.716+0000 7f3dca693700 1 --2- 172.21.15.49:0/3976156053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3dc415b2c0 0x7f3dc415fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:10:22.726 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.716+0000 7f3dc9691700 1 --2- 172.21.15.49:0/3976156053 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3dc406e570 0x7f3dc40c7e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:10:22.726 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.717+0000 7f3dc9e92700 1 --2- 172.21.15.49:0/3976156053 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3dc406a7e0 0x7f3dc41541a0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f3dc0016000 tx=0x7f3dc0004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:22.726 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.717+0000 7f3dba7fc700 1 -- 172.21.15.49:0/3976156053 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dc0004510 con 0x7f3dc406a7e0 2023-10-16T12:10:22.727 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.717+0000 7f3dba7fc700 1 -- 172.21.15.49:0/3976156053 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dc001e070 con 0x7f3dc406a7e0 2023-10-16T12:10:22.727 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.717+0000 7f3dba7fc700 1 -- 172.21.15.49:0/3976156053 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dc00038c0 con 0x7f3dc406a7e0 2023-10-16T12:10:22.727 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.717+0000 7f3dcc0f6700 1 -- 172.21.15.49:0/3976156053 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dc40b2fb0 con 0x7f3dc406a7e0 2023-10-16T12:10:22.727 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.717+0000 7f3dcc0f6700 1 -- 172.21.15.49:0/3976156053 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dc41546e0 con 0x7f3dc406a7e0 2023-10-16T12:10:22.727 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.719+0000 7f3dba7fc700 1 -- 172.21.15.49:0/3976156053 <== mon.1 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f3dc0003d20 con 0x7f3dc406a7e0 2023-10-16T12:10:22.727 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.719+0000 7f3dcc0f6700 1 -- 172.21.15.49:0/3976156053 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3dc40b34d0 con 0x7f3dc406a7e0 2023-10-16T12:10:22.728 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.719+0000 7f3dba7fc700 1 -- 172.21.15.49:0/3976156053 <== mon.1 v2:172.21.15.49:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f3dc0020070 con 0x7f3dc406a7e0 2023-10-16T12:10:22.728 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.721+0000 7f3dba7fc700 1 -- 172.21.15.49:0/3976156053 <== mon.1 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f3dc40b34d0 con 0x7f3dc406a7e0 2023-10-16T12:10:22.808 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.807+0000 7f3dcc0f6700 1 -- 172.21.15.49:0/3976156053 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd new", "uuid": "33c469f5-2ffc-466e-966f-a411ca1e0e29", "id": 3} v 0) v1 -- 0x7f3dc404ebb0 con 0x7f3dc406a7e0 2023-10-16T12:10:22.811 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.810+0000 7f3dba7fc700 1 -- 172.21.15.49:0/3976156053 <== mon.1 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "33c469f5-2ffc-466e-966f-a411ca1e0e29", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f3dc002a020 con 0x7f3dc406a7e0 2023-10-16T12:10:22.811 INFO:teuthology.orchestra.run.smithi049.stdout:3 2023-10-16T12:10:22.812 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.811+0000 7f3dcc0f6700 1 -- 172.21.15.49:0/3976156053 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3dc406a7e0 msgr2=0x7f3dc41541a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:22.812 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.811+0000 7f3dcc0f6700 1 --2- 172.21.15.49:0/3976156053 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3dc406a7e0 0x7f3dc41541a0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f3dc0016000 tx=0x7f3dc0004750 comp rx=0 tx=0).stop 2023-10-16T12:10:22.813 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.811+0000 7f3dcc0f6700 1 -- 172.21.15.49:0/3976156053 shutdown_connections 2023-10-16T12:10:22.813 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.811+0000 7f3dcc0f6700 1 --2- 172.21.15.49:0/3976156053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3dc415b2c0 0x7f3dc415fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:22.813 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.811+0000 7f3dcc0f6700 1 --2- 172.21.15.49:0/3976156053 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3dc406a7e0 0x7f3dc41541a0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:22.813 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.811+0000 7f3dcc0f6700 1 --2- 172.21.15.49:0/3976156053 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3dc406e570 0x7f3dc40c7e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:22.813 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.811+0000 7f3dcc0f6700 1 -- 172.21.15.49:0/3976156053 >> 172.21.15.49:0/3976156053 conn(0x7f3dc40b1f40 msgr2=0x7f3dc40cc4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:22.814 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.811+0000 7f3dcc0f6700 1 -- 172.21.15.49:0/3976156053 shutdown_connections 2023-10-16T12:10:22.814 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.811+0000 7f3dcc0f6700 1 -- 172.21.15.49:0/3976156053 wait complete. 2023-10-16T12:10:22.824 DEBUG:teuthology.orchestra.run.smithi049:> sudo ceph --cluster ceph osd new 500e63ec-cf34-409a-84c1-bbcf7d17606e 4 2023-10-16T12:10:22.947 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.946+0000 7f577712d700 1 Processor -- start 2023-10-16T12:10:22.948 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.946+0000 7f577712d700 1 -- start start 2023-10-16T12:10:22.948 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:10:22.946+0000 7ff76d9e4200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T12:10:22.951 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.947+0000 7f577712d700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f57700cb880 0x7f57700cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:22.951 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.947+0000 7f577712d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f577006da60 0x7f577006def0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:22.951 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.947+0000 7f577712d700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57700cef70 con 0x7f577015b070 2023-10-16T12:10:22.952 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.947+0000 7f577712d700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f577006e430 con 0x7f57700cb880 2023-10-16T12:10:22.952 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.947+0000 7f577712d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57700ce360 con 0x7f577006da60 2023-10-16T12:10:22.952 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.947+0000 7f57756ca700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f577006da60 0x7f577006def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:22.952 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.947+0000 7f576ffff700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f57700cb880 0x7f57700cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:22.952 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.947+0000 7f57756ca700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f577006da60 0x7f577006def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.49:42602/0 (socket says 172.21.15.49:42602) 2023-10-16T12:10:22.953 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.947+0000 7f57756ca700 1 -- 172.21.15.49:0/2783179365 learned_addr learned my addr 172.21.15.49:0/2783179365 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-10-16T12:10:22.953 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.948+0000 7f57756ca700 1 -- 172.21.15.49:0/2783179365 >> v2:172.21.15.49:3300/0 conn(0x7f57700cb880 msgr2=0x7f57700cbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:10:22.953 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.948+0000 7f57756ca700 1 --2- 172.21.15.49:0/2783179365 >> v2:172.21.15.49:3300/0 conn(0x7f57700cb880 0x7f57700cbd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:22.953 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.948+0000 7f57756ca700 1 -- 172.21.15.49:0/2783179365 >> v1:172.21.15.49:6789/0 conn(0x7f577015b070 legacy=0x7f577015b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:22.953 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.948+0000 7f57756ca700 1 -- 172.21.15.49:0/2783179365 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57700ce640 con 0x7f577006da60 2023-10-16T12:10:22.954 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.948+0000 7f576ffff700 1 --2- 172.21.15.49:0/2783179365 >> v2:172.21.15.49:3300/0 conn(0x7f57700cb880 0x7f57700cbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:10:22.954 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.948+0000 7f57756ca700 1 --2- 172.21.15.49:0/2783179365 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f577006da60 0x7f577006def0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5768009e50 tx=0x7f576800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e254ee76e29f80d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:22.954 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.948+0000 7f576f7fe700 1 -- 172.21.15.49:0/2783179365 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f576800e070 con 0x7f577006da60 2023-10-16T12:10:22.954 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.948+0000 7f576f7fe700 1 -- 172.21.15.49:0/2783179365 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f576800b910 con 0x7f577006da60 2023-10-16T12:10:22.954 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.948+0000 7f576f7fe700 1 -- 172.21.15.49:0/2783179365 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f576800bcc0 con 0x7f577006da60 2023-10-16T12:10:22.955 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.949+0000 7f577712d700 1 -- 172.21.15.49:0/2783179365 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f577006da60 msgr2=0x7f577006def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:22.955 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.949+0000 7f577712d700 1 --2- 172.21.15.49:0/2783179365 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f577006da60 0x7f577006def0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5768009e50 tx=0x7f576800b330 comp rx=0 tx=0).stop 2023-10-16T12:10:22.955 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.949+0000 7f577712d700 1 -- 172.21.15.49:0/2783179365 shutdown_connections 2023-10-16T12:10:22.955 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.949+0000 7f577712d700 1 --2- 172.21.15.49:0/2783179365 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f577006da60 0x7f577006def0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:22.955 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.949+0000 7f577712d700 1 --2- 172.21.15.49:0/2783179365 >> v2:172.21.15.49:3300/0 conn(0x7f57700cb880 0x7f57700cbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:22.955 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.949+0000 7f577712d700 1 -- 172.21.15.49:0/2783179365 >> 172.21.15.49:0/2783179365 conn(0x7f57700b1f40 msgr2=0x7f57700b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:22.956 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.949+0000 7f577712d700 1 -- 172.21.15.49:0/2783179365 shutdown_connections 2023-10-16T12:10:22.956 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.949+0000 7f577712d700 1 -- 172.21.15.49:0/2783179365 wait complete. 2023-10-16T12:10:22.956 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.950+0000 7f577712d700 1 Processor -- start 2023-10-16T12:10:22.956 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.950+0000 7f577712d700 1 -- start start 2023-10-16T12:10:22.959 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.951+0000 7f577712d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f577006da60 0x7f5770157940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:22.959 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.951+0000 7f577712d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f57700cb880 0x7f5770152280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:22.959 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.951+0000 7f577712d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f577015b070 0x7f57701527c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:22.959 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.951+0000 7f577712d700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57700c4b60 con 0x7f577006da60 2023-10-16T12:10:22.959 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.951+0000 7f577712d700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57700c3ee0 con 0x7f57700cb880 2023-10-16T12:10:22.960 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.951+0000 7f577712d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57700c41c0 con 0x7f577015b070 2023-10-16T12:10:22.960 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.951+0000 7f57756ca700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f577015b070 0x7f57701527c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:22.960 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.951+0000 7f57756ca700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f577015b070 0x7f57701527c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.49:42606/0 (socket says 172.21.15.49:42606) 2023-10-16T12:10:22.960 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.951+0000 7f576ffff700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f57700cb880 0x7f5770152280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:22.960 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.951+0000 7f57756ca700 1 -- 172.21.15.49:0/1086056440 learned_addr learned my addr 172.21.15.49:0/1086056440 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-10-16T12:10:22.961 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.951+0000 7f5774ec9700 1 --2- 172.21.15.49:0/1086056440 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f577006da60 0x7f5770157940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:22.961 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.952+0000 7f57756ca700 1 -- 172.21.15.49:0/1086056440 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f57700cb880 msgr2=0x7f5770152280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:22.961 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.952+0000 7f57756ca700 1 --2- 172.21.15.49:0/1086056440 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f57700cb880 0x7f5770152280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:22.961 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.952+0000 7f57756ca700 1 -- 172.21.15.49:0/1086056440 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f577006da60 msgr2=0x7f5770157940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:22.961 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.952+0000 7f57756ca700 1 --2- 172.21.15.49:0/1086056440 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f577006da60 0x7f5770157940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:22.962 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.952+0000 7f57756ca700 1 -- 172.21.15.49:0/1086056440 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f575c0094a0 con 0x7f577015b070 2023-10-16T12:10:22.962 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.952+0000 7f57756ca700 1 --2- 172.21.15.49:0/1086056440 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f577015b070 0x7f57701527c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f57700ce4a0 tx=0x7f5768004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:22.962 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.952+0000 7f576d7fa700 1 -- 172.21.15.49:0/1086056440 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5768004800 con 0x7f577015b070 2023-10-16T12:10:22.962 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.952+0000 7f576d7fa700 1 -- 172.21.15.49:0/1086056440 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5768004960 con 0x7f577015b070 2023-10-16T12:10:22.962 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.952+0000 7f577712d700 1 -- 172.21.15.49:0/1086056440 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5768009b30 con 0x7f577015b070 2023-10-16T12:10:22.963 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.952+0000 7f576d7fa700 1 -- 172.21.15.49:0/1086056440 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f576801b670 con 0x7f577015b070 2023-10-16T12:10:22.963 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.952+0000 7f577712d700 1 -- 172.21.15.49:0/1086056440 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57700c4780 con 0x7f577015b070 2023-10-16T12:10:22.963 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.952+0000 7f576d7fa700 1 -- 172.21.15.49:0/1086056440 <== mon.2 v2:172.21.15.49:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f576801bad0 con 0x7f577015b070 2023-10-16T12:10:22.963 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.952+0000 7f576d7fa700 1 -- 172.21.15.49:0/1086056440 <== mon.2 v2:172.21.15.49:3301/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f576801bd80 con 0x7f577015b070 2023-10-16T12:10:22.963 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.953+0000 7f577712d700 1 -- 172.21.15.49:0/1086056440 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f577004ebb0 con 0x7f577015b070 2023-10-16T12:10:22.964 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:22.955+0000 7f576d7fa700 1 -- 172.21.15.49:0/1086056440 <== mon.2 v2:172.21.15.49:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f5768043b40 con 0x7f577015b070 2023-10-16T12:10:22.982 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:10:22.981+0000 7f77e25c4200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T12:10:23.043 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.042+0000 7f577712d700 1 -- 172.21.15.49:0/1086056440 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command({"prefix": "osd new", "uuid": "500e63ec-cf34-409a-84c1-bbcf7d17606e", "id": 4} v 0) v1 -- 0x7f57701583d0 con 0x7f577015b070 2023-10-16T12:10:23.046 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.045+0000 7f576d7fa700 1 -- 172.21.15.49:0/1086056440 <== mon.2 v2:172.21.15.49:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "500e63ec-cf34-409a-84c1-bbcf7d17606e", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f5768032020 con 0x7f577015b070 2023-10-16T12:10:23.047 INFO:teuthology.orchestra.run.smithi049.stdout:4 2023-10-16T12:10:23.048 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.046+0000 7f577712d700 1 -- 172.21.15.49:0/1086056440 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f577015b070 msgr2=0x7f57701527c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:23.048 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.046+0000 7f577712d700 1 --2- 172.21.15.49:0/1086056440 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f577015b070 0x7f57701527c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f57700ce4a0 tx=0x7f5768004ad0 comp rx=0 tx=0).stop 2023-10-16T12:10:23.048 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.046+0000 7f577712d700 1 -- 172.21.15.49:0/1086056440 shutdown_connections 2023-10-16T12:10:23.048 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.046+0000 7f577712d700 1 --2- 172.21.15.49:0/1086056440 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f577015b070 0x7f57701527c0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:23.048 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.046+0000 7f577712d700 1 --2- 172.21.15.49:0/1086056440 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f577006da60 0x7f5770157940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:23.049 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.046+0000 7f577712d700 1 --2- 172.21.15.49:0/1086056440 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f57700cb880 0x7f5770152280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:23.049 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.046+0000 7f577712d700 1 -- 172.21.15.49:0/1086056440 >> 172.21.15.49:0/1086056440 conn(0x7f57700b1f40 msgr2=0x7f57700b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:23.049 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.046+0000 7f577712d700 1 -- 172.21.15.49:0/1086056440 shutdown_connections 2023-10-16T12:10:23.049 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.046+0000 7f577712d700 1 -- 172.21.15.49:0/1086056440 wait complete. 2023-10-16T12:10:23.059 DEBUG:teuthology.orchestra.run.smithi049:> sudo ceph --cluster ceph osd new 841bc611-c518-4610-b086-049bad607070 5 2023-10-16T12:10:23.183 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.182+0000 7f77a1e43700 1 Processor -- start 2023-10-16T12:10:23.184 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.182+0000 7f77a1e43700 1 -- start start 2023-10-16T12:10:23.185 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.183+0000 7f77a1e43700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f779c0cb880 0x7f779c0cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:23.185 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.183+0000 7f77a1e43700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f779c0cef70 con 0x7f779c15b070 2023-10-16T12:10:23.186 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.183+0000 7f77a1e43700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f779c0ce360 con 0x7f779c06da60 2023-10-16T12:10:23.186 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.183+0000 7f77a1e43700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f779c0ce640 con 0x7f779c0cb880 2023-10-16T12:10:23.186 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.183+0000 7f779affd700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f779c0cb880 0x7f779c0cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:23.186 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.183+0000 7f779bfff700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7f779c06da60 0x7f779c0cf400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.49:34978/0 (socket says 172.21.15.49:34978) 2023-10-16T12:10:23.186 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.183+0000 7f779bfff700 1 -- 172.21.15.49:0/4138180286 learned_addr learned my addr 172.21.15.49:0/4138180286 (peer_addr_for_me v1:172.21.15.49:0/0) 2023-10-16T12:10:23.186 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.183+0000 7f779affd700 1 -- 172.21.15.49:0/4138180286 >> v1:172.21.15.49:6789/0 conn(0x7f779c06da60 legacy=0x7f779c0cf400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:23.187 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.184+0000 7f779affd700 1 -- 172.21.15.49:0/4138180286 >> v1:172.21.15.16:6789/0 conn(0x7f779c15b070 legacy=0x7f779c15b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:23.187 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.184+0000 7f779affd700 1 -- 172.21.15.49:0/4138180286 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f779c0ce920 con 0x7f779c0cb880 2023-10-16T12:10:23.187 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.184+0000 7f779affd700 1 --2- 172.21.15.49:0/4138180286 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f779c0cb880 0x7f779c0cbd00 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7784009870 tx=0x7f7784009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a7b1a095de87a4ea server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:23.187 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.184+0000 7f779a7fc700 1 -- 172.21.15.49:0/4138180286 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7784011830 con 0x7f779c0cb880 2023-10-16T12:10:23.187 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.184+0000 7f779a7fc700 1 -- 172.21.15.49:0/4138180286 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7784011990 con 0x7f779c0cb880 2023-10-16T12:10:23.188 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.184+0000 7f779a7fc700 1 -- 172.21.15.49:0/4138180286 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7784011d20 con 0x7f779c0cb880 2023-10-16T12:10:23.190 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.185+0000 7f77a1e43700 1 -- 172.21.15.49:0/4138180286 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f779c0cb880 msgr2=0x7f779c0cbd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:23.190 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.185+0000 7f77a1e43700 1 --2- 172.21.15.49:0/4138180286 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f779c0cb880 0x7f779c0cbd00 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7784009870 tx=0x7f7784009b80 comp rx=0 tx=0).stop 2023-10-16T12:10:23.191 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.185+0000 7f77a1e43700 1 -- 172.21.15.49:0/4138180286 shutdown_connections 2023-10-16T12:10:23.191 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.185+0000 7f77a1e43700 1 --2- 172.21.15.49:0/4138180286 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f779c0cb880 0x7f779c0cbd00 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:23.191 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.185+0000 7f77a1e43700 1 -- 172.21.15.49:0/4138180286 >> 172.21.15.49:0/4138180286 conn(0x7f779c0b1f40 msgr2=0x7f779c0b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:23.191 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.185+0000 7f77a1e43700 1 -- 172.21.15.49:0/4138180286 shutdown_connections 2023-10-16T12:10:23.191 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.185+0000 7f77a1e43700 1 -- 172.21.15.49:0/4138180286 wait complete. 2023-10-16T12:10:23.192 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.186+0000 7f77a1e43700 1 Processor -- start 2023-10-16T12:10:23.192 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.186+0000 7f77a1e43700 1 -- start start 2023-10-16T12:10:23.192 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.187+0000 7f77a1e43700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f779c06da60 0x7f779c15d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:23.192 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.187+0000 7f77a1e43700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f779c0cb880 0x7f779c1556c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:23.192 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.187+0000 7f77a1e43700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f779c15b070 0x7f779c15c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:23.193 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.187+0000 7f77a1e43700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f779c157f20 con 0x7f779c06da60 2023-10-16T12:10:23.193 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.187+0000 7f77a1e43700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f779c157580 con 0x7f779c0cb880 2023-10-16T12:10:23.193 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.187+0000 7f77a1e43700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f779c157860 con 0x7f779c15b070 2023-10-16T12:10:23.193 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.187+0000 7f779bfff700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f779c15b070 0x7f779c15c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:23.193 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.187+0000 7f779affd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f779c0cb880 0x7f779c1556c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:23.193 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.187+0000 7f779affd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f779c0cb880 0x7f779c1556c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:52612/0 (socket says 172.21.15.49:52612) 2023-10-16T12:10:23.194 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.187+0000 7f779affd700 1 -- 172.21.15.49:0/2179540195 learned_addr learned my addr 172.21.15.49:0/2179540195 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-10-16T12:10:23.194 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.187+0000 7f779bfff700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f779c15b070 0x7f779c15c5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.49:42620/0 (socket says 172.21.15.49:42620) 2023-10-16T12:10:23.194 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.187+0000 7f779b7fe700 1 --2- 172.21.15.49:0/2179540195 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f779c06da60 0x7f779c15d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:23.194 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.187+0000 7f779affd700 1 -- 172.21.15.49:0/2179540195 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f779c15b070 msgr2=0x7f779c15c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:23.194 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.187+0000 7f779affd700 1 --2- 172.21.15.49:0/2179540195 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f779c15b070 0x7f779c15c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:23.195 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.187+0000 7f779affd700 1 -- 172.21.15.49:0/2179540195 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f779c06da60 msgr2=0x7f779c15d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:23.195 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.187+0000 7f779affd700 1 --2- 172.21.15.49:0/2179540195 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f779c06da60 0x7f779c15d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:23.195 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.187+0000 7f779affd700 1 -- 172.21.15.49:0/2179540195 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77840094a0 con 0x7f779c0cb880 2023-10-16T12:10:23.195 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.187+0000 7f779affd700 1 --2- 172.21.15.49:0/2179540195 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f779c0cb880 0x7f779c1556c0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f7784000c00 tx=0x7f7784004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:23.195 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.188+0000 7f7793fff700 1 -- 172.21.15.49:0/2179540195 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77840036a0 con 0x7f779c0cb880 2023-10-16T12:10:23.195 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.188+0000 7f7793fff700 1 -- 172.21.15.49:0/2179540195 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f778401f070 con 0x7f779c0cb880 2023-10-16T12:10:23.196 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.188+0000 7f77a1e43700 1 -- 172.21.15.49:0/2179540195 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f779c157b40 con 0x7f779c0cb880 2023-10-16T12:10:23.196 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.188+0000 7f7793fff700 1 -- 172.21.15.49:0/2179540195 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7784003a40 con 0x7f779c0cb880 2023-10-16T12:10:23.196 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.188+0000 7f77a1e43700 1 -- 172.21.15.49:0/2179540195 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f779c157230 con 0x7f779c0cb880 2023-10-16T12:10:23.196 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.188+0000 7f7793fff700 1 -- 172.21.15.49:0/2179540195 <== mon.1 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f7784026480 con 0x7f779c0cb880 2023-10-16T12:10:23.196 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.188+0000 7f7793fff700 1 -- 172.21.15.49:0/2179540195 <== mon.1 v2:172.21.15.49:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f7784021070 con 0x7f779c0cb880 2023-10-16T12:10:23.197 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.189+0000 7f77a1e43700 1 -- 172.21.15.49:0/2179540195 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f779c158060 con 0x7f779c0cb880 2023-10-16T12:10:23.197 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.191+0000 7f7793fff700 1 -- 172.21.15.49:0/2179540195 <== mon.1 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f779c158060 con 0x7f779c0cb880 2023-10-16T12:10:23.277 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.276+0000 7f77a1e43700 1 -- 172.21.15.49:0/2179540195 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd new", "uuid": "841bc611-c518-4610-b086-049bad607070", "id": 5} v 0) v1 -- 0x7f779c04ebb0 con 0x7f779c0cb880 2023-10-16T12:10:23.280 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.279+0000 7f7793fff700 1 -- 172.21.15.49:0/2179540195 <== mon.1 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "841bc611-c518-4610-b086-049bad607070", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f7784035020 con 0x7f779c0cb880 2023-10-16T12:10:23.281 INFO:teuthology.orchestra.run.smithi049.stdout:5 2023-10-16T12:10:23.282 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.280+0000 7f77a1e43700 1 -- 172.21.15.49:0/2179540195 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f779c0cb880 msgr2=0x7f779c1556c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:23.282 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.280+0000 7f77a1e43700 1 --2- 172.21.15.49:0/2179540195 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f779c0cb880 0x7f779c1556c0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f7784000c00 tx=0x7f7784004750 comp rx=0 tx=0).stop 2023-10-16T12:10:23.282 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.280+0000 7f77a1e43700 1 -- 172.21.15.49:0/2179540195 shutdown_connections 2023-10-16T12:10:23.282 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.280+0000 7f77a1e43700 1 --2- 172.21.15.49:0/2179540195 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f779c15b070 0x7f779c15c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:23.283 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.280+0000 7f77a1e43700 1 --2- 172.21.15.49:0/2179540195 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f779c06da60 0x7f779c15d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:23.283 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.280+0000 7f77a1e43700 1 --2- 172.21.15.49:0/2179540195 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f779c0cb880 0x7f779c1556c0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:23.283 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.280+0000 7f77a1e43700 1 -- 172.21.15.49:0/2179540195 >> 172.21.15.49:0/2179540195 conn(0x7f779c0b1f40 msgr2=0x7f779c0b0a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:23.283 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.280+0000 7f77a1e43700 1 -- 172.21.15.49:0/2179540195 shutdown_connections 2023-10-16T12:10:23.283 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.280+0000 7f77a1e43700 1 -- 172.21.15.49:0/2179540195 wait complete. 2023-10-16T12:10:23.294 DEBUG:teuthology.orchestra.run.smithi049:> sudo ceph --cluster ceph osd new 0dc6a215-f320-45c5-bb08-176400b05e59 6 2023-10-16T12:10:23.459 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.458+0000 7f0d13174700 1 Processor -- start 2023-10-16T12:10:23.459 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.458+0000 7f0d13174700 1 -- start start 2023-10-16T12:10:23.461 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.459+0000 7f0d13174700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f0d0c06e570 0x7f0d0c15e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:23.461 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.459+0000 7f0d13174700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d0c15b6a0 con 0x7f0d0c06a7e0 2023-10-16T12:10:23.461 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.459+0000 7f0d13174700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d0c161b20 con 0x7f0d0c15b2c0 2023-10-16T12:10:23.461 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.459+0000 7f0d13174700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d0c161e00 con 0x7f0d0c06e570 2023-10-16T12:10:23.462 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.459+0000 7f0d0bfff700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f0d0c06e570 0x7f0d0c15e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:23.462 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.459+0000 7f0d0bfff700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f0d0c06e570 0x7f0d0c15e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:52618/0 (socket says 172.21.15.49:52618) 2023-10-16T12:10:23.462 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.459+0000 7f0d10f10700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7f0d0c15b2c0 0x7f0d0c161340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.49:34986/0 (socket says 172.21.15.49:34986) 2023-10-16T12:10:23.462 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.459+0000 7f0d0bfff700 1 -- 172.21.15.49:0/2684857087 learned_addr learned my addr 172.21.15.49:0/2684857087 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-10-16T12:10:23.462 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.459+0000 7f0d0bfff700 1 -- 172.21.15.49:0/2684857087 >> v1:172.21.15.49:6789/0 conn(0x7f0d0c15b2c0 legacy=0x7f0d0c161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:23.463 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.459+0000 7f0d0bfff700 1 -- 172.21.15.49:0/2684857087 >> v1:172.21.15.16:6789/0 conn(0x7f0d0c06a7e0 legacy=0x7f0d0c071a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:23.463 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.459+0000 7f0d0bfff700 1 -- 172.21.15.49:0/2684857087 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d0c1620e0 con 0x7f0d0c06e570 2023-10-16T12:10:23.463 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.459+0000 7f0d0bfff700 1 --2- 172.21.15.49:0/2684857087 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0d0c06e570 0x7f0d0c15e5a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f0cf80098f0 tx=0x7f0cf8009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e71ba760f4a8c43c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:23.463 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.460+0000 7f0d0b7fe700 1 -- 172.21.15.49:0/2684857087 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cf800b8a0 con 0x7f0d0c06e570 2023-10-16T12:10:23.463 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.460+0000 7f0d0b7fe700 1 -- 172.21.15.49:0/2684857087 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cf800ba00 con 0x7f0d0c06e570 2023-10-16T12:10:23.464 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.460+0000 7f0d0b7fe700 1 -- 172.21.15.49:0/2684857087 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cf800bd90 con 0x7f0d0c06e570 2023-10-16T12:10:23.467 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.460+0000 7f0d13174700 1 -- 172.21.15.49:0/2684857087 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0d0c06e570 msgr2=0x7f0d0c15e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:23.467 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.460+0000 7f0d13174700 1 --2- 172.21.15.49:0/2684857087 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0d0c06e570 0x7f0d0c15e5a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f0cf80098f0 tx=0x7f0cf8009c00 comp rx=0 tx=0).stop 2023-10-16T12:10:23.467 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.460+0000 7f0d13174700 1 -- 172.21.15.49:0/2684857087 shutdown_connections 2023-10-16T12:10:23.467 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.460+0000 7f0d13174700 1 --2- 172.21.15.49:0/2684857087 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0d0c06e570 0x7f0d0c15e5a0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:23.467 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.460+0000 7f0d13174700 1 -- 172.21.15.49:0/2684857087 >> 172.21.15.49:0/2684857087 conn(0x7f0d0c0b1f40 msgr2=0x7f0d0c0b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:23.468 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.460+0000 7f0d13174700 1 -- 172.21.15.49:0/2684857087 shutdown_connections 2023-10-16T12:10:23.468 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.460+0000 7f0d13174700 1 -- 172.21.15.49:0/2684857087 wait complete. 2023-10-16T12:10:23.468 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.462+0000 7f0d13174700 1 Processor -- start 2023-10-16T12:10:23.468 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.462+0000 7f0d13174700 1 -- start start 2023-10-16T12:10:23.468 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.464+0000 7f0d13174700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0d0c06a7e0 0x7f0d0c159e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:23.469 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.464+0000 7f0d13174700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d0c06e570 0x7f0d0c1f2d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:23.469 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.464+0000 7f0d13174700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0d0c15b2c0 0x7f0d0c0c9570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:23.469 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.464+0000 7f0d13174700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d0c0c7c70 con 0x7f0d0c06e570 2023-10-16T12:10:23.469 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.464+0000 7f0d13174700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d0c0c7990 con 0x7f0d0c06a7e0 2023-10-16T12:10:23.469 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.464+0000 7f0d13174700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d0c0b2fb0 con 0x7f0d0c15b2c0 2023-10-16T12:10:23.470 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.464+0000 7f0d10f10700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0d0c06a7e0 0x7f0d0c159e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:23.470 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.464+0000 7f0d10f10700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0d0c06a7e0 0x7f0d0c159e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:52622/0 (socket says 172.21.15.49:52622) 2023-10-16T12:10:23.470 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.464+0000 7f0d10f10700 1 -- 172.21.15.49:0/1533733124 learned_addr learned my addr 172.21.15.49:0/1533733124 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-10-16T12:10:23.470 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.464+0000 7f0d11711700 1 --2- 172.21.15.49:0/1533733124 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0d0c15b2c0 0x7f0d0c0c9570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:23.470 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.464+0000 7f0d0bfff700 1 --2- 172.21.15.49:0/1533733124 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d0c06e570 0x7f0d0c1f2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:23.471 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.464+0000 7f0d10f10700 1 -- 172.21.15.49:0/1533733124 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0d0c15b2c0 msgr2=0x7f0d0c0c9570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:23.471 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.464+0000 7f0d10f10700 1 --2- 172.21.15.49:0/1533733124 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0d0c15b2c0 0x7f0d0c0c9570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:23.471 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.464+0000 7f0d10f10700 1 -- 172.21.15.49:0/1533733124 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d0c06e570 msgr2=0x7f0d0c1f2d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:23.471 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.464+0000 7f0d10f10700 1 --2- 172.21.15.49:0/1533733124 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d0c06e570 0x7f0d0c1f2d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:23.471 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.464+0000 7f0d10f10700 1 -- 172.21.15.49:0/1533733124 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cf8009550 con 0x7f0d0c06a7e0 2023-10-16T12:10:23.472 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.464+0000 7f0d11711700 1 --2- 172.21.15.49:0/1533733124 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0d0c15b2c0 0x7f0d0c0c9570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:10:23.472 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.464+0000 7f0d10f10700 1 --2- 172.21.15.49:0/1533733124 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0d0c06a7e0 0x7f0d0c159e70 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f0d0000c500 tx=0x7f0d0000c810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:23.472 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.465+0000 7f0d097fa700 1 -- 172.21.15.49:0/1533733124 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d00005cb0 con 0x7f0d0c06a7e0 2023-10-16T12:10:23.472 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.465+0000 7f0d097fa700 1 -- 172.21.15.49:0/1533733124 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d00005e10 con 0x7f0d0c06a7e0 2023-10-16T12:10:23.472 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.465+0000 7f0d13174700 1 -- 172.21.15.49:0/1533733124 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d0c0b3290 con 0x7f0d0c06a7e0 2023-10-16T12:10:23.473 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.465+0000 7f0d097fa700 1 -- 172.21.15.49:0/1533733124 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d000072d0 con 0x7f0d0c06a7e0 2023-10-16T12:10:23.473 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.465+0000 7f0d13174700 1 -- 172.21.15.49:0/1533733124 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d0c0b3850 con 0x7f0d0c06a7e0 2023-10-16T12:10:23.473 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.465+0000 7f0d097fa700 1 -- 172.21.15.49:0/1533733124 <== mon.1 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f0d0001c440 con 0x7f0d0c06a7e0 2023-10-16T12:10:23.473 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.465+0000 7f0d097fa700 1 -- 172.21.15.49:0/1533733124 <== mon.1 v2:172.21.15.49:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7f0d000129f0 con 0x7f0d0c06a7e0 2023-10-16T12:10:23.473 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.466+0000 7f0d13174700 1 -- 172.21.15.49:0/1533733124 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d0c04ebb0 con 0x7f0d0c06a7e0 2023-10-16T12:10:23.474 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.468+0000 7f0d097fa700 1 -- 172.21.15.49:0/1533733124 <== mon.1 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f0d00021060 con 0x7f0d0c06a7e0 2023-10-16T12:10:23.554 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.553+0000 7f0d13174700 1 -- 172.21.15.49:0/1533733124 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd new", "uuid": "0dc6a215-f320-45c5-bb08-176400b05e59", "id": 6} v 0) v1 -- 0x7f0d0c0b6370 con 0x7f0d0c06a7e0 2023-10-16T12:10:23.557 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.556+0000 7f0d097fa700 1 -- 172.21.15.49:0/1533733124 <== mon.1 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "0dc6a215-f320-45c5-bb08-176400b05e59", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f0d00017070 con 0x7f0d0c06a7e0 2023-10-16T12:10:23.557 INFO:teuthology.orchestra.run.smithi049.stdout:6 2023-10-16T12:10:23.558 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.557+0000 7f0d13174700 1 -- 172.21.15.49:0/1533733124 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0d0c06a7e0 msgr2=0x7f0d0c159e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:23.558 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.557+0000 7f0d13174700 1 --2- 172.21.15.49:0/1533733124 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0d0c06a7e0 0x7f0d0c159e70 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f0d0000c500 tx=0x7f0d0000c810 comp rx=0 tx=0).stop 2023-10-16T12:10:23.559 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.557+0000 7f0d13174700 1 -- 172.21.15.49:0/1533733124 shutdown_connections 2023-10-16T12:10:23.559 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.557+0000 7f0d13174700 1 --2- 172.21.15.49:0/1533733124 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0d0c15b2c0 0x7f0d0c0c9570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:23.559 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.557+0000 7f0d13174700 1 --2- 172.21.15.49:0/1533733124 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0d0c06a7e0 0x7f0d0c159e70 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:23.559 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.557+0000 7f0d13174700 1 --2- 172.21.15.49:0/1533733124 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d0c06e570 0x7f0d0c1f2d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:23.560 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.557+0000 7f0d13174700 1 -- 172.21.15.49:0/1533733124 >> 172.21.15.49:0/1533733124 conn(0x7f0d0c0b1f40 msgr2=0x7f0d0c15b830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:23.560 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.557+0000 7f0d13174700 1 -- 172.21.15.49:0/1533733124 shutdown_connections 2023-10-16T12:10:23.560 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.557+0000 7f0d13174700 1 -- 172.21.15.49:0/1533733124 wait complete. 2023-10-16T12:10:23.570 DEBUG:teuthology.orchestra.run.smithi049:> sudo ceph --cluster ceph osd new 69af17a2-6ebd-486c-9f55-00a38839e155 7 2023-10-16T12:10:23.616 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:10:23.615+0000 7ff76d9e4200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T12:10:23.675 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:10:23.674+0000 7f77e25c4200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T12:10:23.688 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:10:23.686+0000 7ff76d9e4200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T12:10:23.697 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.696+0000 7fa46e76b700 1 Processor -- start 2023-10-16T12:10:23.697 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.696+0000 7fa46e76b700 1 -- start start 2023-10-16T12:10:23.700 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.697+0000 7fa46e76b700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fa46815b070 0x7fa46815b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:23.700 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.697+0000 7fa46e76b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa46806e170 0x7fa4680cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:23.700 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.697+0000 7fa46e76b700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4680cef70 con 0x7fa4680cb880 2023-10-16T12:10:23.700 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.697+0000 7fa46e76b700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4680ce360 con 0x7fa46815b070 2023-10-16T12:10:23.700 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.697+0000 7fa46e76b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4680ce640 con 0x7fa46806e170 2023-10-16T12:10:23.701 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.697+0000 7fa467fff700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fa46815b070 0x7fa46815b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:23.701 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.697+0000 7fa46cd08700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa46806e170 0x7fa4680cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:23.701 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.697+0000 7fa467fff700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fa46815b070 0x7fa46815b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:52628/0 (socket says 172.21.15.49:52628) 2023-10-16T12:10:23.701 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.697+0000 7fa467fff700 1 -- 172.21.15.49:0/719620256 learned_addr learned my addr 172.21.15.49:0/719620256 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-10-16T12:10:23.701 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.697+0000 7fa467fff700 1 -- 172.21.15.49:0/719620256 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa46806e170 msgr2=0x7fa4680cbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:10:23.702 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.697+0000 7fa467fff700 1 --2- 172.21.15.49:0/719620256 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa46806e170 0x7fa4680cbd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:23.702 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.697+0000 7fa467fff700 1 -- 172.21.15.49:0/719620256 >> v1:172.21.15.16:6789/0 conn(0x7fa4680cb880 legacy=0x7fa46806da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:23.702 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.697+0000 7fa467fff700 1 -- 172.21.15.49:0/719620256 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4680ce920 con 0x7fa46815b070 2023-10-16T12:10:23.702 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.697+0000 7fa46cd08700 1 --2- 172.21.15.49:0/719620256 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa46806e170 0x7fa4680cbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:10:23.702 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.698+0000 7fa467fff700 1 --2- 172.21.15.49:0/719620256 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa46815b070 0x7fa46815b450 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa4580098f0 tx=0x7fa458009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=172ecc46260f7950 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:23.703 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.698+0000 7fa466ffd700 1 -- 172.21.15.49:0/719620256 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa458011830 con 0x7fa46815b070 2023-10-16T12:10:23.703 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.698+0000 7fa466ffd700 1 -- 172.21.15.49:0/719620256 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa458011990 con 0x7fa46815b070 2023-10-16T12:10:23.703 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.698+0000 7fa466ffd700 1 -- 172.21.15.49:0/719620256 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa458011d20 con 0x7fa46815b070 2023-10-16T12:10:23.703 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.698+0000 7fa46e76b700 1 -- 172.21.15.49:0/719620256 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa46815b070 msgr2=0x7fa46815b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:23.703 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.698+0000 7fa46e76b700 1 --2- 172.21.15.49:0/719620256 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa46815b070 0x7fa46815b450 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa4580098f0 tx=0x7fa458009c00 comp rx=0 tx=0).stop 2023-10-16T12:10:23.704 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.698+0000 7fa46e76b700 1 -- 172.21.15.49:0/719620256 shutdown_connections 2023-10-16T12:10:23.704 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.698+0000 7fa46e76b700 1 --2- 172.21.15.49:0/719620256 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa46806e170 0x7fa4680cbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:23.704 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.698+0000 7fa46e76b700 1 --2- 172.21.15.49:0/719620256 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa46815b070 0x7fa46815b450 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:23.704 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.698+0000 7fa46e76b700 1 -- 172.21.15.49:0/719620256 >> 172.21.15.49:0/719620256 conn(0x7fa4680b1f40 msgr2=0x7fa4680b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:23.704 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.698+0000 7fa46e76b700 1 -- 172.21.15.49:0/719620256 shutdown_connections 2023-10-16T12:10:23.705 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.698+0000 7fa46e76b700 1 -- 172.21.15.49:0/719620256 wait complete. 2023-10-16T12:10:23.707 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.700+0000 7fa46e76b700 1 Processor -- start 2023-10-16T12:10:23.707 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.700+0000 7fa46e76b700 1 -- start start 2023-10-16T12:10:23.707 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.701+0000 7fa46e76b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa46806e170 0x7fa468166e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:23.708 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.701+0000 7fa46e76b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa4680cb880 0x7fa4681678a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:23.708 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.701+0000 7fa46e76b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa46815b070 0x7fa468155270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:23.708 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.701+0000 7fa46e76b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa468168260 con 0x7fa46815b070 2023-10-16T12:10:23.708 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.701+0000 7fa46e76b700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4681557b0 con 0x7fa46806e170 2023-10-16T12:10:23.708 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.701+0000 7fa46e76b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa468167f80 con 0x7fa4680cb880 2023-10-16T12:10:23.709 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.701+0000 7fa467fff700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa46806e170 0x7fa468166e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:23.709 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.701+0000 7fa467fff700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa46806e170 0x7fa468166e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:52634/0 (socket says 172.21.15.49:52634) 2023-10-16T12:10:23.709 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.701+0000 7fa467fff700 1 -- 172.21.15.49:0/3108022211 learned_addr learned my addr 172.21.15.49:0/3108022211 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-10-16T12:10:23.709 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.701+0000 7fa46cd08700 1 --2- 172.21.15.49:0/3108022211 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa46815b070 0x7fa468155270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:23.709 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.701+0000 7fa4677fe700 1 --2- 172.21.15.49:0/3108022211 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa4680cb880 0x7fa4681678a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:23.710 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.701+0000 7fa467fff700 1 -- 172.21.15.49:0/3108022211 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa4680cb880 msgr2=0x7fa4681678a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:23.710 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.701+0000 7fa467fff700 1 --2- 172.21.15.49:0/3108022211 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa4680cb880 0x7fa4681678a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:23.710 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.701+0000 7fa467fff700 1 -- 172.21.15.49:0/3108022211 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa46815b070 msgr2=0x7fa468155270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:23.710 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.701+0000 7fa467fff700 1 --2- 172.21.15.49:0/3108022211 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa46815b070 0x7fa468155270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:23.710 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.701+0000 7fa467fff700 1 -- 172.21.15.49:0/3108022211 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa460009a70 con 0x7fa46806e170 2023-10-16T12:10:23.711 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.701+0000 7fa467fff700 1 --2- 172.21.15.49:0/3108022211 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa46806e170 0x7fa468166e60 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa458009c40 tx=0x7fa458004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:23.711 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.701+0000 7fa464ff9700 1 -- 172.21.15.49:0/3108022211 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4580044d0 con 0x7fa46806e170 2023-10-16T12:10:23.711 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.701+0000 7fa464ff9700 1 -- 172.21.15.49:0/3108022211 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa45801e070 con 0x7fa46806e170 2023-10-16T12:10:23.711 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.702+0000 7fa46e76b700 1 -- 172.21.15.49:0/3108022211 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa458009550 con 0x7fa46806e170 2023-10-16T12:10:23.712 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.702+0000 7fa464ff9700 1 -- 172.21.15.49:0/3108022211 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4580038e0 con 0x7fa46806e170 2023-10-16T12:10:23.712 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.702+0000 7fa46e76b700 1 -- 172.21.15.49:0/3108022211 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa468166660 con 0x7fa46806e170 2023-10-16T12:10:23.712 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.702+0000 7fa464ff9700 1 -- 172.21.15.49:0/3108022211 <== mon.1 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fa458003d40 con 0x7fa46806e170 2023-10-16T12:10:23.712 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.702+0000 7fa464ff9700 1 -- 172.21.15.49:0/3108022211 <== mon.1 v2:172.21.15.49:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7fa45801aba0 con 0x7fa46806e170 2023-10-16T12:10:23.712 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.702+0000 7fa46e76b700 1 -- 172.21.15.49:0/3108022211 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4680b6330 con 0x7fa46806e170 2023-10-16T12:10:23.712 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.705+0000 7fa464ff9700 1 -- 172.21.15.49:0/3108022211 <== mon.1 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7fa458032060 con 0x7fa46806e170 2023-10-16T12:10:23.749 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:10:23.748+0000 7f77e25c4200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T12:10:23.791 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.790+0000 7fa46e76b700 1 -- 172.21.15.49:0/3108022211 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd new", "uuid": "69af17a2-6ebd-486c-9f55-00a38839e155", "id": 7} v 0) v1 -- 0x7fa46804ebb0 con 0x7fa46806e170 2023-10-16T12:10:23.794 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.793+0000 7fa464ff9700 1 -- 172.21.15.49:0/3108022211 <== mon.1 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "69af17a2-6ebd-486c-9f55-00a38839e155", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7fa458020070 con 0x7fa46806e170 2023-10-16T12:10:23.795 INFO:teuthology.orchestra.run.smithi049.stdout:7 2023-10-16T12:10:23.796 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.794+0000 7fa46e76b700 1 -- 172.21.15.49:0/3108022211 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa46806e170 msgr2=0x7fa468166e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:23.796 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.794+0000 7fa46e76b700 1 --2- 172.21.15.49:0/3108022211 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa46806e170 0x7fa468166e60 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa458009c40 tx=0x7fa458004750 comp rx=0 tx=0).stop 2023-10-16T12:10:23.796 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.794+0000 7fa46e76b700 1 -- 172.21.15.49:0/3108022211 shutdown_connections 2023-10-16T12:10:23.796 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.794+0000 7fa46e76b700 1 --2- 172.21.15.49:0/3108022211 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa46815b070 0x7fa468155270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:23.796 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.794+0000 7fa46e76b700 1 --2- 172.21.15.49:0/3108022211 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa46806e170 0x7fa468166e60 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:23.797 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.794+0000 7fa46e76b700 1 --2- 172.21.15.49:0/3108022211 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa4680cb880 0x7fa4681678a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:23.797 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.794+0000 7fa46e76b700 1 -- 172.21.15.49:0/3108022211 >> 172.21.15.49:0/3108022211 conn(0x7fa4680b1f40 msgr2=0x7fa4680c6a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:23.797 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.795+0000 7fa46e76b700 1 -- 172.21.15.49:0/3108022211 shutdown_connections 2023-10-16T12:10:23.797 INFO:teuthology.orchestra.run.smithi049.stderr:2023-10-16T12:10:23.795+0000 7fa46e76b700 1 -- 172.21.15.49:0/3108022211 wait complete. 2023-10-16T12:10:23.808 INFO:tasks.ceph.osd.0:Restarting daemon 2023-10-16T12:10:23.808 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-10-16T12:10:23.810 INFO:tasks.ceph.osd.0:Started 2023-10-16T12:10:23.811 INFO:tasks.ceph.osd.1:Restarting daemon 2023-10-16T12:10:23.811 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-10-16T12:10:23.813 INFO:tasks.ceph.osd.1:Started 2023-10-16T12:10:23.813 INFO:tasks.ceph.osd.2:Restarting daemon 2023-10-16T12:10:23.814 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-10-16T12:10:23.816 INFO:tasks.ceph.osd.2:Started 2023-10-16T12:10:23.816 INFO:tasks.ceph.osd.3:Restarting daemon 2023-10-16T12:10:23.816 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-10-16T12:10:23.818 INFO:tasks.ceph.osd.3:Started 2023-10-16T12:10:23.819 INFO:tasks.ceph.osd.4:Restarting daemon 2023-10-16T12:10:23.819 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-10-16T12:10:23.821 INFO:tasks.ceph.osd.4:Started 2023-10-16T12:10:23.822 INFO:tasks.ceph.osd.5:Restarting daemon 2023-10-16T12:10:23.822 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-10-16T12:10:23.824 INFO:tasks.ceph.osd.5:Started 2023-10-16T12:10:23.824 INFO:tasks.ceph.osd.6:Restarting daemon 2023-10-16T12:10:23.825 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2023-10-16T12:10:23.827 INFO:tasks.ceph.osd.6:Started 2023-10-16T12:10:23.827 INFO:tasks.ceph.osd.7:Restarting daemon 2023-10-16T12:10:23.827 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2023-10-16T12:10:23.829 INFO:tasks.ceph.osd.7:Started 2023-10-16T12:10:23.830 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:10:23.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:23.991+0000 7fa8338ae700 1 Processor -- start 2023-10-16T12:10:23.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:23.991+0000 7fa8338ae700 1 -- start start 2023-10-16T12:10:23.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:23.993+0000 7fa8338ae700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa82c15b630 0x7fa82c0cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:23.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:23.993+0000 7fa8338ae700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fa82c0ce4f0 0x7fa82c0cfca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:23.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:23.993+0000 7fa8338ae700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa82c0cc2a0 con 0x7fa82c15ba10 2023-10-16T12:10:23.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:23.993+0000 7fa8338ae700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa82c0cdb20 con 0x7fa82c0ce4f0 2023-10-16T12:10:23.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:23.993+0000 7fa8338ae700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa82c0cc540 con 0x7fa82c15b630 2023-10-16T12:10:23.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:23.993+0000 7fa831e4b700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fa82c0ce4f0 0x7fa82c0cfca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:23.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:23.993+0000 7fa83164a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa82c15b630 0x7fa82c0cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:23.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:23.993+0000 7fa831e4b700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fa82c0ce4f0 0x7fa82c0cfca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:59754/0 (socket says 172.21.15.16:59754) 2023-10-16T12:10:23.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:23.993+0000 7fa831e4b700 1 -- 172.21.15.16:0/3185099442 learned_addr learned my addr 172.21.15.16:0/3185099442 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:10:23.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:23.993+0000 7fa831e4b700 1 -- 172.21.15.16:0/3185099442 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa82c15b630 msgr2=0x7fa82c0cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:10:23.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:23.993+0000 7fa831e4b700 1 --2- 172.21.15.16:0/3185099442 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa82c15b630 0x7fa82c0cd5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:23.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:23.993+0000 7fa831e4b700 1 -- 172.21.15.16:0/3185099442 >> v1:172.21.15.49:6789/0 conn(0x7fa82c15ba10 legacy=0x7fa82c0cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:23.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:23.993+0000 7fa831e4b700 1 -- 172.21.15.16:0/3185099442 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa82c0cc820 con 0x7fa82c0ce4f0 2023-10-16T12:10:23.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:23.993+0000 7fa83164a700 1 --2- 172.21.15.16:0/3185099442 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa82c15b630 0x7fa82c0cd5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:10:23.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:23.993+0000 7fa831e4b700 1 --2- 172.21.15.16:0/3185099442 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa82c0ce4f0 0x7fa82c0cfca0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa824009e50 tx=0x7fa82400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eb99653e1929c1b9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:23.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:23.994+0000 7fa823fff700 1 -- 172.21.15.16:0/3185099442 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa82400e070 con 0x7fa82c0ce4f0 2023-10-16T12:10:23.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:23.994+0000 7fa823fff700 1 -- 172.21.15.16:0/3185099442 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa82400b910 con 0x7fa82c0ce4f0 2023-10-16T12:10:23.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:23.994+0000 7fa823fff700 1 -- 172.21.15.16:0/3185099442 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa82400bcc0 con 0x7fa82c0ce4f0 2023-10-16T12:10:24.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:23.994+0000 7fa8338ae700 1 -- 172.21.15.16:0/3185099442 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa82c0ce4f0 msgr2=0x7fa82c0cfca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:24.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:23.994+0000 7fa8338ae700 1 --2- 172.21.15.16:0/3185099442 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa82c0ce4f0 0x7fa82c0cfca0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa824009e50 tx=0x7fa82400b330 comp rx=0 tx=0).stop 2023-10-16T12:10:24.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:23.995+0000 7fa8338ae700 1 -- 172.21.15.16:0/3185099442 shutdown_connections 2023-10-16T12:10:24.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:23.995+0000 7fa8338ae700 1 --2- 172.21.15.16:0/3185099442 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa82c0ce4f0 0x7fa82c0cfca0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:24.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:23.995+0000 7fa8338ae700 1 --2- 172.21.15.16:0/3185099442 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa82c15b630 0x7fa82c0cd5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:24.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:23.995+0000 7fa8338ae700 1 -- 172.21.15.16:0/3185099442 >> 172.21.15.16:0/3185099442 conn(0x7fa82c0d14b0 msgr2=0x7fa82c0d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:24.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:23.995+0000 7fa8338ae700 1 -- 172.21.15.16:0/3185099442 shutdown_connections 2023-10-16T12:10:24.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:23.995+0000 7fa8338ae700 1 -- 172.21.15.16:0/3185099442 wait complete. 2023-10-16T12:10:24.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:23.996+0000 7fa8338ae700 1 Processor -- start 2023-10-16T12:10:24.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:23.996+0000 7fa8338ae700 1 -- start start 2023-10-16T12:10:24.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:23.997+0000 7fa8338ae700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa82c0ce4f0 0x7fa82c158d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:24.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:23.997+0000 7fa8338ae700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa82c15b630 0x7fa82c0ca490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:24.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:23.997+0000 7fa8338ae700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa82c15ba10 0x7fa82c0ca9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:24.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:23.997+0000 7fa8338ae700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa82c0c5a60 con 0x7fa82c15b630 2023-10-16T12:10:24.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:23.997+0000 7fa8338ae700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa82c0c5d40 con 0x7fa82c0ce4f0 2023-10-16T12:10:24.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:23.997+0000 7fa8338ae700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa82c0c6020 con 0x7fa82c15ba10 2023-10-16T12:10:24.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:23.997+0000 7fa830e49700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa82c15b630 0x7fa82c0ca490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:24.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:23.997+0000 7fa830e49700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa82c15b630 0x7fa82c0ca490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60696/0 (socket says 172.21.15.16:60696) 2023-10-16T12:10:24.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:23.997+0000 7fa83164a700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa82c0ce4f0 0x7fa82c158d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:24.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:23.997+0000 7fa830e49700 1 -- 172.21.15.16:0/1683567067 learned_addr learned my addr 172.21.15.16:0/1683567067 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:10:24.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:23.997+0000 7fa831e4b700 1 --2- 172.21.15.16:0/1683567067 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa82c15ba10 0x7fa82c0ca9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:24.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:23.998+0000 7fa830e49700 1 -- 172.21.15.16:0/1683567067 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa82c15ba10 msgr2=0x7fa82c0ca9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:24.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:23.998+0000 7fa830e49700 1 --2- 172.21.15.16:0/1683567067 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa82c15ba10 0x7fa82c0ca9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:24.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:23.998+0000 7fa830e49700 1 -- 172.21.15.16:0/1683567067 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa82c0ce4f0 msgr2=0x7fa82c158d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:24.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:23.998+0000 7fa830e49700 1 --2- 172.21.15.16:0/1683567067 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa82c0ce4f0 0x7fa82c158d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:24.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:23.998+0000 7fa830e49700 1 -- 172.21.15.16:0/1683567067 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa828009430 con 0x7fa82c15b630 2023-10-16T12:10:24.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:23.998+0000 7fa83164a700 1 --2- 172.21.15.16:0/1683567067 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa82c0ce4f0 0x7fa82c158d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:10:24.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:23.998+0000 7fa830e49700 1 --2- 172.21.15.16:0/1683567067 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa82c15b630 0x7fa82c0ca490 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fa81c00c4e0 tx=0x7fa81c00c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:24.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:23.998+0000 7fa821ffb700 1 -- 172.21.15.16:0/1683567067 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa81c0052d0 con 0x7fa82c15b630 2023-10-16T12:10:24.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:23.998+0000 7fa821ffb700 1 -- 172.21.15.16:0/1683567067 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa81c005430 con 0x7fa82c15b630 2023-10-16T12:10:24.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:23.998+0000 7fa8338ae700 1 -- 172.21.15.16:0/1683567067 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa824009b30 con 0x7fa82c15b630 2023-10-16T12:10:24.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:23.998+0000 7fa821ffb700 1 -- 172.21.15.16:0/1683567067 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa81c0038c0 con 0x7fa82c15b630 2023-10-16T12:10:24.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:23.998+0000 7fa8338ae700 1 -- 172.21.15.16:0/1683567067 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa82c0c6820 con 0x7fa82c15b630 2023-10-16T12:10:24.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:23.998+0000 7fa821ffb700 1 -- 172.21.15.16:0/1683567067 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fa81c003d20 con 0x7fa82c15b630 2023-10-16T12:10:24.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:23.998+0000 7fa821ffb700 1 -- 172.21.15.16:0/1683567067 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fa81c011d60 con 0x7fa82c15b630 2023-10-16T12:10:24.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:23.999+0000 7fa8338ae700 1 -- 172.21.15.16:0/1683567067 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa82c0c6540 con 0x7fa82c15b630 2023-10-16T12:10:24.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:24.001+0000 7fa821ffb700 1 -- 172.21.15.16:0/1683567067 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7fa82c0c6540 con 0x7fa82c15b630 2023-10-16T12:10:24.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:24.071+0000 7fa8338ae700 1 -- 172.21.15.16:0/1683567067 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa82c0cd070 con 0x7fa82c15b630 2023-10-16T12:10:24.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:24.072+0000 7fa821ffb700 1 -- 172.21.15.16:0/1683567067 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fa81c017070 con 0x7fa82c15b630 2023-10-16T12:10:24.074 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:10:24.074 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":10,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:10:23.791850+0000","last_up_change":"0.000000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:10:24.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:24.073+0000 7fa8338ae700 1 -- 172.21.15.16:0/1683567067 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa82c15b630 msgr2=0x7fa82c0ca490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:24.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:24.073+0000 7fa8338ae700 1 --2- 172.21.15.16:0/1683567067 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa82c15b630 0x7fa82c0ca490 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fa81c00c4e0 tx=0x7fa81c00c7f0 comp rx=0 tx=0).stop 2023-10-16T12:10:24.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:24.073+0000 7fa8338ae700 1 -- 172.21.15.16:0/1683567067 shutdown_connections 2023-10-16T12:10:24.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:24.073+0000 7fa8338ae700 1 --2- 172.21.15.16:0/1683567067 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa82c15ba10 0x7fa82c0ca9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:24.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:24.073+0000 7fa8338ae700 1 --2- 172.21.15.16:0/1683567067 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa82c0ce4f0 0x7fa82c158d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:24.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:24.073+0000 7fa8338ae700 1 --2- 172.21.15.16:0/1683567067 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa82c15b630 0x7fa82c0ca490 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:24.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:24.073+0000 7fa8338ae700 1 -- 172.21.15.16:0/1683567067 >> 172.21.15.16:0/1683567067 conn(0x7fa82c0d14b0 msgr2=0x7fa82c0b51b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:24.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:24.073+0000 7fa8338ae700 1 -- 172.21.15.16:0/1683567067 shutdown_connections 2023-10-16T12:10:24.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:24.073+0000 7fa8338ae700 1 -- 172.21.15.16:0/1683567067 wait complete. 2023-10-16T12:10:24.087 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-10-16T12:10:24.087 INFO:tasks.ceph:Waiting for OSDs to come up 2023-10-16T12:10:24.204 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-10-16T12:10:24.202+0000 7fb39b3bc640 -1 Falling back to public interface 2023-10-16T12:10:24.430 INFO:tasks.ceph.osd.4.smithi049.stderr:2023-10-16T12:10:24.429+0000 7f9fe7c17640 -1 Falling back to public interface 2023-10-16T12:10:24.434 INFO:tasks.ceph.osd.6.smithi049.stderr:2023-10-16T12:10:24.433+0000 7f69009ae640 -1 Falling back to public interface 2023-10-16T12:10:24.444 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:10:24.442+0000 7ff76d9e4200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:10:24.448 INFO:tasks.ceph.osd.5.smithi049.stderr:2023-10-16T12:10:24.447+0000 7f21392fb640 -1 Falling back to public interface 2023-10-16T12:10:24.451 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-10-16T12:10:24.449+0000 7f0a70186640 -1 Falling back to public interface 2023-10-16T12:10:24.454 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-10-16T12:10:24.452+0000 7fd666f65640 -1 Falling back to public interface 2023-10-16T12:10:24.454 INFO:tasks.ceph.osd.3.smithi016.stderr:2023-10-16T12:10:24.452+0000 7f5210e9d640 -1 Falling back to public interface 2023-10-16T12:10:24.462 INFO:tasks.ceph.osd.7.smithi049.stderr:2023-10-16T12:10:24.461+0000 7fdfe778b640 -1 Falling back to public interface 2023-10-16T12:10:24.501 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:10:24.500+0000 7f77e25c4200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:10:24.528 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:10:24.526+0000 7ff76d9e4200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T12:10:24.582 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:10:24.581+0000 7f77e25c4200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T12:10:24.675 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:10:24.673+0000 7ff76d9e4200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:10:24.730 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:10:24.729+0000 7f77e25c4200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:10:24.893 DEBUG:teuthology.orchestra.run.smithi016:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-16T12:10:24.920 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:10:24.919+0000 7ff76d9e4200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T12:10:24.977 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:10:24.976+0000 7f77e25c4200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T12:10:25.009 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:25.007+0000 7f0ece1e5700 1 Processor -- start 2023-10-16T12:10:25.009 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:25.007+0000 7f0ece1e5700 1 -- start start 2023-10-16T12:10:25.010 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:25.008+0000 7f0ece1e5700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f0ec80cf3f0 0x7f0ec815a070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:25.010 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:25.008+0000 7f0ece1e5700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ec806e4c0 con 0x7f0ec8159c90 2023-10-16T12:10:25.010 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:25.008+0000 7f0ece1e5700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ec806e700 con 0x7f0ec815a5b0 2023-10-16T12:10:25.011 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:25.008+0000 7f0ece1e5700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ec806e9e0 con 0x7f0ec80cf3f0 2023-10-16T12:10:25.011 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:25.008+0000 7f0ec77fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0ec8159c90 0x7f0ec806ddb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49974/0 (socket says 172.21.15.16:49974) 2023-10-16T12:10:25.011 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:25.008+0000 7f0ec77fe700 1 -- 172.21.15.16:0/591824429 learned_addr learned my addr 172.21.15.16:0/591824429 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:10:25.011 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:25.008+0000 7f0ec6ffd700 1 --2- 172.21.15.16:0/591824429 >> v2:172.21.15.49:3300/0 conn(0x7f0ec80cf3f0 0x7f0ec815a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:25.011 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:25.008+0000 7f0ec67fc700 1 -- 172.21.15.16:0/591824429 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3258873292 0 0) 0x7f0eb8001490 con 0x7f0ec8159c90 2023-10-16T12:10:25.011 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:25.008+0000 7f0ec67fc700 1 -- 172.21.15.16:0/591824429 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0eb40035b0 con 0x7f0ec8159c90 2023-10-16T12:10:25.012 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:25.008+0000 7f0ec6ffd700 1 -- 172.21.15.16:0/591824429 >> v1:172.21.15.49:6789/0 conn(0x7f0ec815a5b0 legacy=0x7f0ec80ca640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:25.012 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:25.008+0000 7f0ec6ffd700 1 -- 172.21.15.16:0/591824429 >> v1:172.21.15.16:6789/0 conn(0x7f0ec8159c90 legacy=0x7f0ec806ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:25.012 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:25.008+0000 7f0ec6ffd700 1 -- 172.21.15.16:0/591824429 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ec806ecc0 con 0x7f0ec80cf3f0 2023-10-16T12:10:25.015 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:25.009+0000 7f0ec6ffd700 1 --2- 172.21.15.16:0/591824429 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0ec80cf3f0 0x7f0ec815a070 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f0eb000a9e0 tx=0x7f0eb000acf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=af1bc62f5474590a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:25.015 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:25.009+0000 7f0ec67fc700 1 -- 172.21.15.16:0/591824429 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eb000cb70 con 0x7f0ec80cf3f0 2023-10-16T12:10:25.016 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:25.009+0000 7f0ec67fc700 1 -- 172.21.15.16:0/591824429 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0eb000ccd0 con 0x7f0ec80cf3f0 2023-10-16T12:10:25.016 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:25.009+0000 7f0ec67fc700 1 -- 172.21.15.16:0/591824429 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eb0018620 con 0x7f0ec80cf3f0 2023-10-16T12:10:25.016 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:25.009+0000 7f0ece1e5700 1 -- 172.21.15.16:0/591824429 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0ec80cf3f0 msgr2=0x7f0ec815a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:25.016 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:25.009+0000 7f0ece1e5700 1 --2- 172.21.15.16:0/591824429 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0ec80cf3f0 0x7f0ec815a070 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f0eb000a9e0 tx=0x7f0eb000acf0 comp rx=0 tx=0).stop 2023-10-16T12:10:25.016 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:25.009+0000 7f0ece1e5700 1 -- 172.21.15.16:0/591824429 shutdown_connections 2023-10-16T12:10:25.017 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:25.009+0000 7f0ece1e5700 1 --2- 172.21.15.16:0/591824429 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0ec80cf3f0 0x7f0ec815a070 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:25.017 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:25.010+0000 7f0ece1e5700 1 -- 172.21.15.16:0/591824429 >> 172.21.15.16:0/591824429 conn(0x7f0ec80cc500 msgr2=0x7f0ec80cc910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:25.017 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:25.010+0000 7f0ece1e5700 1 -- 172.21.15.16:0/591824429 shutdown_connections 2023-10-16T12:10:25.017 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:25.010+0000 7f0ece1e5700 1 -- 172.21.15.16:0/591824429 wait complete. 2023-10-16T12:10:25.017 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:25.011+0000 7f0ece1e5700 1 Processor -- start 2023-10-16T12:10:25.017 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:25.011+0000 7f0ece1e5700 1 -- start start 2023-10-16T12:10:25.018 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:25.012+0000 7f0ece1e5700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0ec80cf3f0 0x7f0ec81576b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:25.018 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:25.012+0000 7f0ece1e5700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0ec8159c90 0x7f0ec81f6310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:25.018 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:25.012+0000 7f0ece1e5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ec815a5b0 0x7f0ec81f6850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:25.018 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:25.012+0000 7f0ece1e5700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec815cd60 con 0x7f0ec815a5b0 2023-10-16T12:10:25.018 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:25.012+0000 7f0ece1e5700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec8157bf0 con 0x7f0ec80cf3f0 2023-10-16T12:10:25.019 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:25.012+0000 7f0ece1e5700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec815c8d0 con 0x7f0ec8159c90 2023-10-16T12:10:25.019 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:25.012+0000 7f0ec7fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ec815a5b0 0x7f0ec81f6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:25.019 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:25.012+0000 7f0ec77fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0ec80cf3f0 0x7f0ec81576b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:25.019 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:25.012+0000 7f0ec77fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0ec80cf3f0 0x7f0ec81576b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:59768/0 (socket says 172.21.15.16:59768) 2023-10-16T12:10:25.019 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:25.012+0000 7f0ec7fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ec815a5b0 0x7f0ec81f6850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60710/0 (socket says 172.21.15.16:60710) 2023-10-16T12:10:25.020 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:25.012+0000 7f0ec77fe700 1 -- 172.21.15.16:0/1196124060 learned_addr learned my addr 172.21.15.16:0/1196124060 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:10:25.020 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:25.012+0000 7f0ec6ffd700 1 --2- 172.21.15.16:0/1196124060 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0ec8159c90 0x7f0ec81f6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:25.020 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:25.012+0000 7f0ec7fff700 1 -- 172.21.15.16:0/1196124060 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0ec8159c90 msgr2=0x7f0ec81f6310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:25.020 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:25.012+0000 7f0ec7fff700 1 --2- 172.21.15.16:0/1196124060 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0ec8159c90 0x7f0ec81f6310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:25.020 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:25.012+0000 7f0ec7fff700 1 -- 172.21.15.16:0/1196124060 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0ec80cf3f0 msgr2=0x7f0ec81576b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:25.020 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:25.012+0000 7f0ec7fff700 1 --2- 172.21.15.16:0/1196124060 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0ec80cf3f0 0x7f0ec81576b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:25.021 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:25.012+0000 7f0ec7fff700 1 -- 172.21.15.16:0/1196124060 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eb000a640 con 0x7f0ec815a5b0 2023-10-16T12:10:25.021 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:25.012+0000 7f0ec7fff700 1 --2- 172.21.15.16:0/1196124060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ec815a5b0 0x7f0ec81f6850 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0ec000c3c0 tx=0x7f0ec000c6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:25.021 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:25.013+0000 7f0ebffff700 1 -- 172.21.15.16:0/1196124060 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ec000cea0 con 0x7f0ec815a5b0 2023-10-16T12:10:25.021 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:25.013+0000 7f0ebffff700 1 -- 172.21.15.16:0/1196124060 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ec00052d0 con 0x7f0ec815a5b0 2023-10-16T12:10:25.021 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:25.013+0000 7f0ece1e5700 1 -- 172.21.15.16:0/1196124060 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ec8158630 con 0x7f0ec815a5b0 2023-10-16T12:10:25.021 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:25.013+0000 7f0ebffff700 1 -- 172.21.15.16:0/1196124060 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ec0003680 con 0x7f0ec815a5b0 2023-10-16T12:10:25.022 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:25.013+0000 7f0ece1e5700 1 -- 172.21.15.16:0/1196124060 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ec815bfa0 con 0x7f0ec815a5b0 2023-10-16T12:10:25.022 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:25.013+0000 7f0ebffff700 1 -- 172.21.15.16:0/1196124060 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f0ec0003ae0 con 0x7f0ec815a5b0 2023-10-16T12:10:25.022 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:25.013+0000 7f0ebffff700 1 -- 172.21.15.16:0/1196124060 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f0ec0012af0 con 0x7f0ec815a5b0 2023-10-16T12:10:25.022 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:25.014+0000 7f0ece1e5700 1 -- 172.21.15.16:0/1196124060 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ec8073820 con 0x7f0ec815a5b0 2023-10-16T12:10:25.023 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:25.018+0000 7f0ebffff700 1 -- 172.21.15.16:0/1196124060 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f0ec0017070 con 0x7f0ec815a5b0 2023-10-16T12:10:25.091 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:25.089+0000 7f0ece1e5700 1 -- 172.21.15.16:0/1196124060 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0ec80cdff0 con 0x7f0ec815a5b0 2023-10-16T12:10:25.092 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:25.090+0000 7f0ebffff700 1 -- 172.21.15.16:0/1196124060 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f0ec001c4f0 con 0x7f0ec815a5b0 2023-10-16T12:10:25.092 INFO:teuthology.misc.health.smithi016.stdout: 2023-10-16T12:10:25.092 INFO:teuthology.misc.health.smithi016.stdout:{"epoch":10,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:10:23.791850+0000","last_up_change":"0.000000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:10:25.093 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:25.091+0000 7f0ece1e5700 1 -- 172.21.15.16:0/1196124060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ec815a5b0 msgr2=0x7f0ec81f6850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:25.093 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:25.091+0000 7f0ece1e5700 1 --2- 172.21.15.16:0/1196124060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ec815a5b0 0x7f0ec81f6850 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0ec000c3c0 tx=0x7f0ec000c6d0 comp rx=0 tx=0).stop 2023-10-16T12:10:25.093 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:25.091+0000 7f0ece1e5700 1 -- 172.21.15.16:0/1196124060 shutdown_connections 2023-10-16T12:10:25.094 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:25.091+0000 7f0ece1e5700 1 --2- 172.21.15.16:0/1196124060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ec815a5b0 0x7f0ec81f6850 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:25.094 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:25.091+0000 7f0ece1e5700 1 --2- 172.21.15.16:0/1196124060 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0ec80cf3f0 0x7f0ec81576b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:25.094 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:25.091+0000 7f0ece1e5700 1 --2- 172.21.15.16:0/1196124060 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0ec8159c90 0x7f0ec81f6310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:25.094 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:25.091+0000 7f0ece1e5700 1 -- 172.21.15.16:0/1196124060 >> 172.21.15.16:0/1196124060 conn(0x7f0ec80cc500 msgr2=0x7f0ec8159060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:25.094 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:25.091+0000 7f0ece1e5700 1 -- 172.21.15.16:0/1196124060 shutdown_connections 2023-10-16T12:10:25.095 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:25.091+0000 7f0ece1e5700 1 -- 172.21.15.16:0/1196124060 wait complete. 2023-10-16T12:10:25.104 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-10-16T12:10:25.146 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:10:25.144+0000 7ff76d9e4200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T12:10:25.199 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:10:25.198+0000 7f77e25c4200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T12:10:25.285 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:10:25.283+0000 7ff76d9e4200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T12:10:25.338 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:10:25.337+0000 7f77e25c4200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T12:10:25.514 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:10:25.512+0000 7ff76d9e4200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T12:10:25.568 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:10:25.567+0000 7f77e25c4200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T12:10:25.675 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:10:25.674+0000 7ff76d9e4200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T12:10:25.730 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:10:25.729+0000 7f77e25c4200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T12:10:25.795 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:10:25.794+0000 7ff76d9e4200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T12:10:25.851 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:10:25.850+0000 7f77e25c4200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T12:10:26.130 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:10:26.128+0000 7ff76d9e4200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T12:10:26.207 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:10:26.206+0000 7f77e25c4200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T12:10:26.234 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:10:26.232+0000 7ff76d9e4200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T12:10:26.311 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:10:26.310+0000 7f77e25c4200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T12:10:26.428 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:10:26.427+0000 7ff76d9e4200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T12:10:26.506 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:10:26.505+0000 7f77e25c4200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T12:10:27.075 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:10:27.074+0000 7ff76d9e4200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T12:10:27.142 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:10:27.142+0000 7f77e25c4200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T12:10:27.727 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-10-16T12:10:27.725+0000 7fb39b3bc640 -1 osd.0 0 log_to_monitors true 2023-10-16T12:10:27.932 INFO:tasks.ceph.osd.4.smithi049.stderr:2023-10-16T12:10:27.931+0000 7f9fe7c17640 -1 osd.4 0 log_to_monitors true 2023-10-16T12:10:27.969 INFO:tasks.ceph.osd.6.smithi049.stderr:2023-10-16T12:10:27.968+0000 7f69009ae640 -1 osd.6 0 log_to_monitors true 2023-10-16T12:10:27.978 INFO:tasks.ceph.osd.3.smithi016.stderr:2023-10-16T12:10:27.976+0000 7f5210e9d640 -1 osd.3 0 log_to_monitors true 2023-10-16T12:10:27.986 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-10-16T12:10:27.985+0000 7fd666f65640 -1 osd.1 0 log_to_monitors true 2023-10-16T12:10:27.990 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-10-16T12:10:27.988+0000 7f0a70186640 -1 osd.2 0 log_to_monitors true 2023-10-16T12:10:27.990 INFO:tasks.ceph.osd.5.smithi049.stderr:2023-10-16T12:10:27.990+0000 7f21392fb640 -1 osd.5 0 log_to_monitors true 2023-10-16T12:10:28.009 INFO:tasks.ceph.osd.7.smithi049.stderr:2023-10-16T12:10:28.008+0000 7fdfe778b640 -1 osd.7 0 log_to_monitors true 2023-10-16T12:10:28.099 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:10:28.097+0000 7ff725948700 -1 mgr.server handle_report got status from non-daemon mon.b 2023-10-16T12:10:28.099 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:10:28.098+0000 7ff725948700 -1 mgr.server handle_report got status from non-daemon mon.c 2023-10-16T12:10:30.539 INFO:tasks.ceph.osd.3.smithi016.stderr:2023-10-16T12:10:30.537+0000 7f5205255700 -1 osd.3 0 waiting for initial osdmap 2023-10-16T12:10:30.560 INFO:tasks.ceph.osd.3.smithi016.stderr:2023-10-16T12:10:30.558+0000 7f51fce2f700 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-16T12:10:30.562 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-10-16T12:10:30.560+0000 7fb38f774700 -1 osd.0 0 waiting for initial osdmap 2023-10-16T12:10:30.563 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-10-16T12:10:30.562+0000 7fd65b31d700 -1 osd.1 0 waiting for initial osdmap 2023-10-16T12:10:30.566 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-10-16T12:10:30.564+0000 7f0a6453e700 -1 osd.2 0 waiting for initial osdmap 2023-10-16T12:10:30.574 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-10-16T12:10:30.573+0000 7fd652ef7700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-16T12:10:30.581 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-10-16T12:10:30.580+0000 7fb38734e700 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-16T12:10:30.582 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-10-16T12:10:30.580+0000 7f0a5c118700 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-16T12:10:30.735 INFO:tasks.ceph.osd.4.smithi049.stderr:2023-10-16T12:10:30.731+0000 7f9fdbfcf700 -1 osd.4 0 waiting for initial osdmap 2023-10-16T12:10:30.737 INFO:tasks.ceph.osd.5.smithi049.stderr:2023-10-16T12:10:30.735+0000 7f212d6b3700 -1 osd.5 0 waiting for initial osdmap 2023-10-16T12:10:30.748 INFO:tasks.ceph.osd.4.smithi049.stderr:2023-10-16T12:10:30.746+0000 7f9fd3ba9700 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-16T12:10:30.750 INFO:tasks.ceph.osd.6.smithi049.stderr:2023-10-16T12:10:30.748+0000 7f68f4d66700 -1 osd.6 0 waiting for initial osdmap 2023-10-16T12:10:30.754 INFO:tasks.ceph.osd.5.smithi049.stderr:2023-10-16T12:10:30.752+0000 7f212528d700 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-16T12:10:30.768 INFO:tasks.ceph.osd.6.smithi049.stderr:2023-10-16T12:10:30.766+0000 7f68ec940700 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-16T12:10:30.777 INFO:tasks.ceph.osd.7.smithi049.stderr:2023-10-16T12:10:30.776+0000 7fdfdbb43700 -1 osd.7 0 waiting for initial osdmap 2023-10-16T12:10:30.788 INFO:tasks.ceph.osd.7.smithi049.stderr:2023-10-16T12:10:30.786+0000 7fdfd371d700 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-16T12:10:31.909 DEBUG:teuthology.orchestra.run.smithi016:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-16T12:10:32.039 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:32.038+0000 7f345900c700 1 Processor -- start 2023-10-16T12:10:32.040 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:32.038+0000 7f345900c700 1 -- start start 2023-10-16T12:10:32.042 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:32.038+0000 7f345900c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f34540d08f0 0x7f345406e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:32.042 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:32.038+0000 7f345900c700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f34540728e0 0x7f34540718b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:32.042 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:32.038+0000 7f345900c700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3454071df0 con 0x7f34540d0cd0 2023-10-16T12:10:32.043 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:32.038+0000 7f345900c700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3454162180 con 0x7f34540d08f0 2023-10-16T12:10:32.043 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:32.039+0000 7f345900c700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3454161600 con 0x7f34540728e0 2023-10-16T12:10:32.043 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:32.039+0000 7f3452d9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f34540d08f0 0x7f345406e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:32.043 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:32.039+0000 7f345259c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f34540d0cd0 0x7f34540721d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50228/0 (socket says 172.21.15.16:50228) 2023-10-16T12:10:32.043 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:32.039+0000 7f345259c700 1 -- 172.21.15.16:0/2010718920 learned_addr learned my addr 172.21.15.16:0/2010718920 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:10:32.044 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:32.039+0000 7f345359e700 1 --2- 172.21.15.16:0/2010718920 >> v2:172.21.15.49:3300/0 conn(0x7f34540728e0 0x7f34540718b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:32.044 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:32.039+0000 7f3452d9d700 1 -- 172.21.15.16:0/2010718920 >> v2:172.21.15.49:3300/0 conn(0x7f34540728e0 msgr2=0x7f34540718b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:10:32.044 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:32.039+0000 7f3452d9d700 1 --2- 172.21.15.16:0/2010718920 >> v2:172.21.15.49:3300/0 conn(0x7f34540728e0 0x7f34540718b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:32.044 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:32.039+0000 7f3452d9d700 1 -- 172.21.15.16:0/2010718920 >> v1:172.21.15.16:6789/0 conn(0x7f34540d0cd0 legacy=0x7f34540721d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:32.044 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:32.039+0000 7f3452d9d700 1 -- 172.21.15.16:0/2010718920 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34541618e0 con 0x7f34540d08f0 2023-10-16T12:10:32.044 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:32.039+0000 7f3452d9d700 1 --2- 172.21.15.16:0/2010718920 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34540d08f0 0x7f345406e430 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f34540cf6b0 tx=0x7f3444009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c18773c4416dbdf server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:32.045 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:32.039+0000 7f3451d9b700 1 -- 172.21.15.16:0/2010718920 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3444011830 con 0x7f34540d08f0 2023-10-16T12:10:32.045 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:32.039+0000 7f3451d9b700 1 -- 172.21.15.16:0/2010718920 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3444011990 con 0x7f34540d08f0 2023-10-16T12:10:32.045 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:32.039+0000 7f3451d9b700 1 -- 172.21.15.16:0/2010718920 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3444011d20 con 0x7f34540d08f0 2023-10-16T12:10:32.045 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:32.040+0000 7f345900c700 1 -- 172.21.15.16:0/2010718920 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34540d08f0 msgr2=0x7f345406e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:32.045 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:32.040+0000 7f345900c700 1 --2- 172.21.15.16:0/2010718920 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34540d08f0 0x7f345406e430 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f34540cf6b0 tx=0x7f3444009b20 comp rx=0 tx=0).stop 2023-10-16T12:10:32.046 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:32.040+0000 7f345900c700 1 -- 172.21.15.16:0/2010718920 shutdown_connections 2023-10-16T12:10:32.046 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:32.040+0000 7f345900c700 1 --2- 172.21.15.16:0/2010718920 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34540d08f0 0x7f345406e430 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:32.046 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:32.040+0000 7f345900c700 1 --2- 172.21.15.16:0/2010718920 >> v2:172.21.15.49:3300/0 conn(0x7f34540728e0 0x7f34540718b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:32.046 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:32.040+0000 7f345900c700 1 -- 172.21.15.16:0/2010718920 >> 172.21.15.16:0/2010718920 conn(0x7f345406bca0 msgr2=0x7f345406c0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:32.046 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:32.040+0000 7f345900c700 1 -- 172.21.15.16:0/2010718920 shutdown_connections 2023-10-16T12:10:32.047 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:32.040+0000 7f345900c700 1 -- 172.21.15.16:0/2010718920 wait complete. 2023-10-16T12:10:32.049 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:32.041+0000 7f345900c700 1 Processor -- start 2023-10-16T12:10:32.049 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:32.042+0000 7f345900c700 1 -- start start 2023-10-16T12:10:32.050 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:32.042+0000 7f345900c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f34540728e0 0x7f34541537d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:32.050 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:32.042+0000 7f345900c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34540d08f0 0x7f34540cb750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:32.050 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:32.042+0000 7f345900c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f34540d0cd0 0x7f3454155320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:32.050 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:32.042+0000 7f345900c700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3454153d10 con 0x7f34540d08f0 2023-10-16T12:10:32.050 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:32.042+0000 7f345900c700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34540c7d30 con 0x7f34540728e0 2023-10-16T12:10:32.051 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:32.042+0000 7f345900c700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34540cc1a0 con 0x7f34540d0cd0 2023-10-16T12:10:32.051 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:32.042+0000 7f345259c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34540d08f0 0x7f34540cb750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:32.051 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:32.042+0000 7f345259c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34540d08f0 0x7f34540cb750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60960/0 (socket says 172.21.15.16:60960) 2023-10-16T12:10:32.051 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:32.042+0000 7f345259c700 1 -- 172.21.15.16:0/3987055316 learned_addr learned my addr 172.21.15.16:0/3987055316 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:10:32.051 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:32.043+0000 7f3452d9d700 1 --2- 172.21.15.16:0/3987055316 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f34540728e0 0x7f34541537d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:32.052 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:32.043+0000 7f345359e700 1 --2- 172.21.15.16:0/3987055316 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f34540d0cd0 0x7f3454155320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:32.052 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:32.043+0000 7f345259c700 1 -- 172.21.15.16:0/3987055316 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f34540d0cd0 msgr2=0x7f3454155320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:32.052 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:32.043+0000 7f345259c700 1 --2- 172.21.15.16:0/3987055316 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f34540d0cd0 0x7f3454155320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:32.052 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:32.043+0000 7f345259c700 1 -- 172.21.15.16:0/3987055316 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f34540728e0 msgr2=0x7f34541537d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:32.052 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:32.043+0000 7f345259c700 1 --2- 172.21.15.16:0/3987055316 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f34540728e0 0x7f34541537d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:32.053 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:32.043+0000 7f345259c700 1 -- 172.21.15.16:0/3987055316 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34440094a0 con 0x7f34540d08f0 2023-10-16T12:10:32.053 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:32.043+0000 7f345259c700 1 --2- 172.21.15.16:0/3987055316 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34540d08f0 0x7f34540cb750 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f344800c450 tx=0x7f344800c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:32.053 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:32.043+0000 7f343b7fe700 1 -- 172.21.15.16:0/3987055316 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3448005d70 con 0x7f34540d08f0 2023-10-16T12:10:32.053 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:32.043+0000 7f343b7fe700 1 -- 172.21.15.16:0/3987055316 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34480070d0 con 0x7f34540d08f0 2023-10-16T12:10:32.053 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:32.043+0000 7f345900c700 1 -- 172.21.15.16:0/3987055316 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34540c7400 con 0x7f34540d08f0 2023-10-16T12:10:32.054 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:32.043+0000 7f343b7fe700 1 -- 172.21.15.16:0/3987055316 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3448007440 con 0x7f34540d08f0 2023-10-16T12:10:32.054 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:32.043+0000 7f345900c700 1 -- 172.21.15.16:0/3987055316 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34540c79c0 con 0x7f34540d08f0 2023-10-16T12:10:32.054 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:32.044+0000 7f345900c700 1 -- 172.21.15.16:0/3987055316 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3454073820 con 0x7f34540d08f0 2023-10-16T12:10:32.054 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:32.045+0000 7f343b7fe700 1 -- 172.21.15.16:0/3987055316 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 4) v1 ==== 89173+0+0 (secure 0 0 0) 0x7f34480075a0 con 0x7f34540d08f0 2023-10-16T12:10:32.054 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:32.045+0000 7f343b7fe700 1 --2- 172.21.15.16:0/3987055316 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f344006ae70 0x7f344006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:32.055 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:32.045+0000 7f343b7fe700 1 -- 172.21.15.16:0/3987055316 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 4471+0+0 (secure 0 0 0) 0x7f3448088310 con 0x7f34540d08f0 2023-10-16T12:10:32.055 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:32.045+0000 7f3452d9d700 1 --2- 172.21.15.16:0/3987055316 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f344006ae70 0x7f344006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:32.055 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:32.046+0000 7f3452d9d700 1 --2- 172.21.15.16:0/3987055316 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f344006ae70 0x7f344006d330 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f34440090b0 tx=0x7f3444008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:32.055 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:32.050+0000 7f343b7fe700 1 -- 172.21.15.16:0/3987055316 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3448053a20 con 0x7f34540d08f0 2023-10-16T12:10:32.261 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:32.259+0000 7f345900c700 1 -- 172.21.15.16:0/3987055316 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f34540c7e70 con 0x7f34540d08f0 2023-10-16T12:10:32.261 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:32.260+0000 7f343b7fe700 1 -- 172.21.15.16:0/3987055316 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) v1 ==== 74+0+10800 (secure 0 0 0) 0x7f3448057040 con 0x7f34540d08f0 2023-10-16T12:10:32.262 INFO:teuthology.misc.health.smithi016.stdout: 2023-10-16T12:10:32.262 INFO:teuthology.misc.health.smithi016.stdout:{"epoch":13,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:10:31.570642+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:10:32.264 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:32.262+0000 7f345900c700 1 -- 172.21.15.16:0/3987055316 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f344006ae70 msgr2=0x7f344006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:32.264 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:32.262+0000 7f345900c700 1 --2- 172.21.15.16:0/3987055316 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f344006ae70 0x7f344006d330 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f34440090b0 tx=0x7f3444008030 comp rx=0 tx=0).stop 2023-10-16T12:10:32.264 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:32.262+0000 7f345900c700 1 -- 172.21.15.16:0/3987055316 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34540d08f0 msgr2=0x7f34540cb750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:32.265 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:32.262+0000 7f345900c700 1 --2- 172.21.15.16:0/3987055316 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34540d08f0 0x7f34540cb750 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f344800c450 tx=0x7f344800c810 comp rx=0 tx=0).stop 2023-10-16T12:10:32.265 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:32.262+0000 7f345900c700 1 -- 172.21.15.16:0/3987055316 shutdown_connections 2023-10-16T12:10:32.265 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:32.262+0000 7f345900c700 1 --2- 172.21.15.16:0/3987055316 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f344006ae70 0x7f344006d330 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:32.265 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:32.262+0000 7f345900c700 1 --2- 172.21.15.16:0/3987055316 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f34540d0cd0 0x7f3454155320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:32.265 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:32.262+0000 7f345900c700 1 --2- 172.21.15.16:0/3987055316 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f34540728e0 0x7f34541537d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:32.266 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:32.262+0000 7f345900c700 1 --2- 172.21.15.16:0/3987055316 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34540d08f0 0x7f34540cb750 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:32.266 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:32.262+0000 7f345900c700 1 -- 172.21.15.16:0/3987055316 >> 172.21.15.16:0/3987055316 conn(0x7f345406bca0 msgr2=0x7f34540ae9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:32.266 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:32.262+0000 7f345900c700 1 -- 172.21.15.16:0/3987055316 shutdown_connections 2023-10-16T12:10:32.266 INFO:teuthology.misc.health.smithi016.stderr:2023-10-16T12:10:32.262+0000 7f345900c700 1 -- 172.21.15.16:0/3987055316 wait complete. 2023-10-16T12:10:32.275 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-10-16T12:10:32.275 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-10-16T12:10:32.276 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-16T12:10:32.276 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-16T12:10:32.279 INFO:tasks.ceph.mds.a:Started 2023-10-16T12:10:32.279 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-16T12:10:32.279 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-16T12:10:32.281 INFO:tasks.ceph.mds.c:Started 2023-10-16T12:10:32.282 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-16T12:10:32.282 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-16T12:10:32.285 INFO:tasks.ceph.mds.b:Started 2023-10-16T12:10:32.285 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-16T12:10:32.285 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-16T12:10:32.288 INFO:tasks.ceph.mds.d:Started 2023-10-16T12:10:32.288 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2023-10-16T12:10:32.288 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:10:32.362 INFO:tasks.ceph.mds.a.smithi016.stdout:starting mds.a at 2023-10-16T12:10:32.363 INFO:tasks.ceph.mds.c.smithi016.stdout:starting mds.c at 2023-10-16T12:10:32.378 INFO:tasks.ceph.mds.b.smithi049.stdout:starting mds.b at 2023-10-16T12:10:32.379 INFO:tasks.ceph.mds.d.smithi049.stdout:starting mds.d at 2023-10-16T12:10:32.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.422+0000 7fda2cd66700 1 Processor -- start 2023-10-16T12:10:32.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.422+0000 7fda2cd66700 1 -- start start 2023-10-16T12:10:32.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.422+0000 7fda2cd66700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fda2815b630 0x7fda280cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:32.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.423+0000 7fda2cd66700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fda2815ba10 0x7fda280cdde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:32.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.423+0000 7fda2cd66700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda280ce770 con 0x7fda280ce320 2023-10-16T12:10:32.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.423+0000 7fda2cd66700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda280cdb20 con 0x7fda2815ba10 2023-10-16T12:10:32.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.423+0000 7fda2cd66700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda280cc2a0 con 0x7fda2815b630 2023-10-16T12:10:32.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.423+0000 7fda26d9d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fda280ce320 0x7fda280cfca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50290/0 (socket says 172.21.15.16:50290) 2023-10-16T12:10:32.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.423+0000 7fda26d9d700 1 -- 172.21.15.16:0/3309565008 learned_addr learned my addr 172.21.15.16:0/3309565008 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:10:32.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.423+0000 7fda2659c700 1 --2- 172.21.15.16:0/3309565008 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fda2815b630 0x7fda280cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:32.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.423+0000 7fda25d9b700 1 --2- 172.21.15.16:0/3309565008 >> v2:172.21.15.49:3300/0 conn(0x7fda2815ba10 0x7fda280cdde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:32.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.423+0000 7fda2559a700 1 -- 172.21.15.16:0/3309565008 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1739140876 0 0) 0x7fda18001110 con 0x7fda280ce320 2023-10-16T12:10:32.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.423+0000 7fda2559a700 1 -- 172.21.15.16:0/3309565008 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda100035b0 con 0x7fda280ce320 2023-10-16T12:10:32.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.423+0000 7fda2559a700 1 -- 172.21.15.16:0/3309565008 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1855517259 0 0) 0x7fda100035b0 con 0x7fda280ce320 2023-10-16T12:10:32.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.423+0000 7fda2659c700 1 -- 172.21.15.16:0/3309565008 >> v2:172.21.15.49:3300/0 conn(0x7fda2815ba10 msgr2=0x7fda280cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:10:32.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.423+0000 7fda2659c700 1 --2- 172.21.15.16:0/3309565008 >> v2:172.21.15.49:3300/0 conn(0x7fda2815ba10 0x7fda280cdde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:32.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.423+0000 7fda2659c700 1 -- 172.21.15.16:0/3309565008 >> v1:172.21.15.16:6789/0 conn(0x7fda280ce320 legacy=0x7fda280cfca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:32.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.423+0000 7fda2659c700 1 -- 172.21.15.16:0/3309565008 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda280cc540 con 0x7fda2815b630 2023-10-16T12:10:32.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.423+0000 7fda2659c700 1 --2- 172.21.15.16:0/3309565008 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fda2815b630 0x7fda280cd5e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fda20009de0 tx=0x7fda2000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c182aac5cd4f69c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:32.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.424+0000 7fda2559a700 1 -- 172.21.15.16:0/3309565008 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda2000e070 con 0x7fda2815b630 2023-10-16T12:10:32.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.424+0000 7fda2559a700 1 -- 172.21.15.16:0/3309565008 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda20003a20 con 0x7fda2815b630 2023-10-16T12:10:32.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.424+0000 7fda2559a700 1 -- 172.21.15.16:0/3309565008 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda18001800 con 0x7fda2815b630 2023-10-16T12:10:32.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.424+0000 7fda2cd66700 1 -- 172.21.15.16:0/3309565008 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fda2815b630 msgr2=0x7fda280cd5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:32.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.424+0000 7fda2cd66700 1 --2- 172.21.15.16:0/3309565008 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fda2815b630 0x7fda280cd5e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fda20009de0 tx=0x7fda2000b330 comp rx=0 tx=0).stop 2023-10-16T12:10:32.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.424+0000 7fda2cd66700 1 -- 172.21.15.16:0/3309565008 shutdown_connections 2023-10-16T12:10:32.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.424+0000 7fda2cd66700 1 --2- 172.21.15.16:0/3309565008 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fda2815b630 0x7fda280cd5e0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:32.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.424+0000 7fda2cd66700 1 --2- 172.21.15.16:0/3309565008 >> v2:172.21.15.49:3300/0 conn(0x7fda2815ba10 0x7fda280cdde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:32.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.424+0000 7fda2cd66700 1 -- 172.21.15.16:0/3309565008 >> 172.21.15.16:0/3309565008 conn(0x7fda280d14b0 msgr2=0x7fda280d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:32.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.424+0000 7fda2cd66700 1 -- 172.21.15.16:0/3309565008 shutdown_connections 2023-10-16T12:10:32.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.424+0000 7fda2cd66700 1 -- 172.21.15.16:0/3309565008 wait complete. 2023-10-16T12:10:32.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.426+0000 7fda2cd66700 1 Processor -- start 2023-10-16T12:10:32.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.426+0000 7fda2cd66700 1 -- start start 2023-10-16T12:10:32.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.426+0000 7fda2cd66700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fda280ce320 0x7fda28157a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:32.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.426+0000 7fda2cd66700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda2815b630 0x7fda2806c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:32.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.426+0000 7fda2cd66700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fda2815ba10 0x7fda2815cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:32.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.426+0000 7fda2cd66700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda280ae020 con 0x7fda2815b630 2023-10-16T12:10:32.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.426+0000 7fda2cd66700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda28157fd0 con 0x7fda280ce320 2023-10-16T12:10:32.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.426+0000 7fda2cd66700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda2806cdc0 con 0x7fda2815ba10 2023-10-16T12:10:32.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.426+0000 7fda25d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda2815b630 0x7fda2806c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:32.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.427+0000 7fda25d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda2815b630 0x7fda2806c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:32788/0 (socket says 172.21.15.16:32788) 2023-10-16T12:10:32.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.427+0000 7fda25d9b700 1 -- 172.21.15.16:0/3616442088 learned_addr learned my addr 172.21.15.16:0/3616442088 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:10:32.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.427+0000 7fda2659c700 1 --2- 172.21.15.16:0/3616442088 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fda280ce320 0x7fda28157a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:32.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.427+0000 7fda26d9d700 1 --2- 172.21.15.16:0/3616442088 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fda2815ba10 0x7fda2815cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:32.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.427+0000 7fda25d9b700 1 -- 172.21.15.16:0/3616442088 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fda2815ba10 msgr2=0x7fda2815cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:32.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.427+0000 7fda25d9b700 1 --2- 172.21.15.16:0/3616442088 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fda2815ba10 0x7fda2815cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:32.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.427+0000 7fda25d9b700 1 -- 172.21.15.16:0/3616442088 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fda280ce320 msgr2=0x7fda28157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:32.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.427+0000 7fda25d9b700 1 --2- 172.21.15.16:0/3616442088 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fda280ce320 0x7fda28157a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:32.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.427+0000 7fda25d9b700 1 -- 172.21.15.16:0/3616442088 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda20009a70 con 0x7fda2815b630 2023-10-16T12:10:32.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.427+0000 7fda25d9b700 1 --2- 172.21.15.16:0/3616442088 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda2815b630 0x7fda2806c300 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fda1c00e870 tx=0x7fda1c00eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:32.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.427+0000 7fda16ffd700 1 -- 172.21.15.16:0/3616442088 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda1c009590 con 0x7fda2815b630 2023-10-16T12:10:32.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.427+0000 7fda16ffd700 1 -- 172.21.15.16:0/3616442088 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda1c013680 con 0x7fda2815b630 2023-10-16T12:10:32.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.427+0000 7fda2cd66700 1 -- 172.21.15.16:0/3616442088 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda280c8580 con 0x7fda2815b630 2023-10-16T12:10:32.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.427+0000 7fda16ffd700 1 -- 172.21.15.16:0/3616442088 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda1c0139f0 con 0x7fda2815b630 2023-10-16T12:10:32.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.427+0000 7fda2cd66700 1 -- 172.21.15.16:0/3616442088 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda280c8b40 con 0x7fda2815b630 2023-10-16T12:10:32.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.428+0000 7fda2cd66700 1 -- 172.21.15.16:0/3616442088 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda2804ebb0 con 0x7fda2815b630 2023-10-16T12:10:32.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.429+0000 7fda16ffd700 1 -- 172.21.15.16:0/3616442088 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 4) v1 ==== 89173+0+0 (secure 0 0 0) 0x7fda1c013b50 con 0x7fda2815b630 2023-10-16T12:10:32.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.429+0000 7fda16ffd700 1 --2- 172.21.15.16:0/3616442088 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fda0806ae70 0x7fda0806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:32.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.429+0000 7fda16ffd700 1 -- 172.21.15.16:0/3616442088 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 4471+0+0 (secure 0 0 0) 0x7fda1c088e50 con 0x7fda2815b630 2023-10-16T12:10:32.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.429+0000 7fda2659c700 1 --2- 172.21.15.16:0/3616442088 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fda0806ae70 0x7fda0806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:32.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.430+0000 7fda2659c700 1 --2- 172.21.15.16:0/3616442088 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fda0806ae70 0x7fda0806d330 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fda20000f80 tx=0x7fda20011000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:32.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.434+0000 7fda16ffd700 1 -- 172.21.15.16:0/3616442088 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fda1c0544b0 con 0x7fda2815b630 2023-10-16T12:10:32.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.641+0000 7fda2cd66700 1 -- 172.21.15.16:0/3616442088 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fda280cd0b0 con 0x7fda2815b630 2023-10-16T12:10:32.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.642+0000 7fda16ffd700 1 -- 172.21.15.16:0/3616442088 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) v1 ==== 74+0+10984 (secure 0 0 0) 0x7fda1c057ad0 con 0x7fda2815b630 2023-10-16T12:10:32.644 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:10:32.644 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":14,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:10:32.572294+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:10:32.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.644+0000 7fda2cd66700 1 -- 172.21.15.16:0/3616442088 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fda0806ae70 msgr2=0x7fda0806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:32.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.644+0000 7fda2cd66700 1 --2- 172.21.15.16:0/3616442088 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fda0806ae70 0x7fda0806d330 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fda20000f80 tx=0x7fda20011000 comp rx=0 tx=0).stop 2023-10-16T12:10:32.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.644+0000 7fda2cd66700 1 -- 172.21.15.16:0/3616442088 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda2815b630 msgr2=0x7fda2806c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:32.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.644+0000 7fda2cd66700 1 --2- 172.21.15.16:0/3616442088 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda2815b630 0x7fda2806c300 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fda1c00e870 tx=0x7fda1c00eb80 comp rx=0 tx=0).stop 2023-10-16T12:10:32.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.644+0000 7fda2cd66700 1 -- 172.21.15.16:0/3616442088 shutdown_connections 2023-10-16T12:10:32.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.644+0000 7fda2cd66700 1 --2- 172.21.15.16:0/3616442088 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fda0806ae70 0x7fda0806d330 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:32.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.644+0000 7fda2cd66700 1 --2- 172.21.15.16:0/3616442088 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fda2815ba10 0x7fda2815cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:32.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.644+0000 7fda2cd66700 1 --2- 172.21.15.16:0/3616442088 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fda280ce320 0x7fda28157a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:32.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.644+0000 7fda2cd66700 1 --2- 172.21.15.16:0/3616442088 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda2815b630 0x7fda2806c300 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:32.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.644+0000 7fda2cd66700 1 -- 172.21.15.16:0/3616442088 >> 172.21.15.16:0/3616442088 conn(0x7fda280d14b0 msgr2=0x7fda2806f410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:32.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.644+0000 7fda2cd66700 1 -- 172.21.15.16:0/3616442088 shutdown_connections 2023-10-16T12:10:32.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.644+0000 7fda2cd66700 1 -- 172.21.15.16:0/3616442088 wait complete. 2023-10-16T12:10:32.658 INFO:tasks.cephfs.filesystem.ceph_manager:[] 2023-10-16T12:10:32.658 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:10:32.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.781+0000 7fa6c3363700 1 Processor -- start 2023-10-16T12:10:32.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.781+0000 7fa6c3363700 1 -- start start 2023-10-16T12:10:32.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.783+0000 7fa6c3363700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fa6bc0d03b0 0x7fa6bc0cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:32.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.783+0000 7fa6c3363700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6bc0cc2a0 con 0x7fa6bc15b630 2023-10-16T12:10:32.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.783+0000 7fa6c3363700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6bc0cc540 con 0x7fa6bc0ce4f0 2023-10-16T12:10:32.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.783+0000 7fa6c3363700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6bc0cdb20 con 0x7fa6bc0d03b0 2023-10-16T12:10:32.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.783+0000 7fa6c10ff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa6bc15b630 0x7fa6bc0cdde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50308/0 (socket says 172.21.15.16:50308) 2023-10-16T12:10:32.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.783+0000 7fa6c10ff700 1 -- 172.21.15.16:0/2454611698 learned_addr learned my addr 172.21.15.16:0/2454611698 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:10:32.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.783+0000 7fa6c1900700 1 --2- 172.21.15.16:0/2454611698 >> v2:172.21.15.49:3300/0 conn(0x7fa6bc0d03b0 0x7fa6bc0cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:32.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.783+0000 7fa6b3fff700 1 -- 172.21.15.16:0/2454611698 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3397509470 0 0) 0x7fa6bc0cc2a0 con 0x7fa6bc15b630 2023-10-16T12:10:32.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.783+0000 7fa6b3fff700 1 -- 172.21.15.16:0/2454611698 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6a80035b0 con 0x7fa6bc15b630 2023-10-16T12:10:32.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.783+0000 7fa6b3fff700 1 -- 172.21.15.16:0/2454611698 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 665401963 0 0) 0x7fa6bc0cc540 con 0x7fa6bc0ce4f0 2023-10-16T12:10:32.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.783+0000 7fa6b3fff700 1 -- 172.21.15.16:0/2454611698 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6bc0cc2a0 con 0x7fa6bc0ce4f0 2023-10-16T12:10:32.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.783+0000 7fa6b3fff700 1 -- 172.21.15.16:0/2454611698 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4036102257 0 0) 0x7fa6a80035b0 con 0x7fa6bc15b630 2023-10-16T12:10:32.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.783+0000 7fa6b3fff700 1 -- 172.21.15.16:0/2454611698 >> v2:172.21.15.49:3300/0 conn(0x7fa6bc0d03b0 msgr2=0x7fa6bc0cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:10:32.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.783+0000 7fa6b3fff700 1 --2- 172.21.15.16:0/2454611698 >> v2:172.21.15.49:3300/0 conn(0x7fa6bc0d03b0 0x7fa6bc0cd5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:32.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.783+0000 7fa6b3fff700 1 -- 172.21.15.16:0/2454611698 >> v1:172.21.15.49:6789/0 conn(0x7fa6bc0ce4f0 legacy=0x7fa6bc0cfca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:32.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.783+0000 7fa6b3fff700 1 -- 172.21.15.16:0/2454611698 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6bc0cc820 con 0x7fa6bc15b630 2023-10-16T12:10:32.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.783+0000 7fa6b3fff700 1 -- 172.21.15.16:0/2454611698 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7fa6ac0017f0 con 0x7fa6bc15b630 2023-10-16T12:10:32.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.783+0000 7fa6b3fff700 1 -- 172.21.15.16:0/2454611698 >> v1:172.21.15.16:6789/0 conn(0x7fa6bc15b630 legacy=0x7fa6bc0cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:32.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.783+0000 7fa6b3fff700 1 --2- 172.21.15.16:0/2454611698 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa6a8003b90 0x7fa6a8003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:32.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.783+0000 7fa6b3fff700 1 --2- 172.21.15.16:0/2454611698 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6a8004850 0x7fa6a8004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:32.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.783+0000 7fa6b3fff700 1 --2- 172.21.15.16:0/2454611698 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa6a8005210 0x7fa6a800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:32.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.783+0000 7fa6b3fff700 1 -- 172.21.15.16:0/2454611698 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a800c110 con 0x7fa6a8004850 2023-10-16T12:10:32.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.783+0000 7fa6b3fff700 1 -- 172.21.15.16:0/2454611698 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a800c280 con 0x7fa6a8003b90 2023-10-16T12:10:32.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.783+0000 7fa6b3fff700 1 -- 172.21.15.16:0/2454611698 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a800c440 con 0x7fa6a8005210 2023-10-16T12:10:32.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.783+0000 7fa6c08fe700 1 --2- 172.21.15.16:0/2454611698 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6a8004850 0x7fa6a8004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:32.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.784+0000 7fa6c10ff700 1 --2- 172.21.15.16:0/2454611698 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa6a8003b90 0x7fa6a8003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:32.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.784+0000 7fa6c1900700 1 --2- 172.21.15.16:0/2454611698 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa6a8005210 0x7fa6a800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:32.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.784+0000 7fa6c08fe700 1 -- 172.21.15.16:0/2454611698 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa6a8005210 msgr2=0x7fa6a800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:10:32.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.784+0000 7fa6c08fe700 1 --2- 172.21.15.16:0/2454611698 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa6a8005210 0x7fa6a800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:32.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.784+0000 7fa6c08fe700 1 -- 172.21.15.16:0/2454611698 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa6a8003b90 msgr2=0x7fa6a8003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:10:32.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.784+0000 7fa6c08fe700 1 --2- 172.21.15.16:0/2454611698 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa6a8003b90 0x7fa6a8003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:32.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.784+0000 7fa6c08fe700 1 -- 172.21.15.16:0/2454611698 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6a80035b0 con 0x7fa6a8004850 2023-10-16T12:10:32.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.784+0000 7fa6c1900700 1 -- 172.21.15.16:0/2454611698 reap_dead start 2023-10-16T12:10:32.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.784+0000 7fa6c08fe700 1 --2- 172.21.15.16:0/2454611698 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6a8004850 0x7fa6a8004cd0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa6b8001940 tx=0x7fa6a8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eef87e193c5f057c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:32.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.784+0000 7fa6b3fff700 1 -- 172.21.15.16:0/2454611698 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b8001430 con 0x7fa6a8004850 2023-10-16T12:10:32.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.784+0000 7fa6b3fff700 1 -- 172.21.15.16:0/2454611698 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6b80052d0 con 0x7fa6a8004850 2023-10-16T12:10:32.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.784+0000 7fa6b3fff700 1 -- 172.21.15.16:0/2454611698 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b8001d10 con 0x7fa6a8004850 2023-10-16T12:10:32.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.785+0000 7fa6c3363700 1 -- 172.21.15.16:0/2454611698 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6a8004850 msgr2=0x7fa6a8004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:32.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.785+0000 7fa6c3363700 1 --2- 172.21.15.16:0/2454611698 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6a8004850 0x7fa6a8004cd0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa6b8001940 tx=0x7fa6a8005880 comp rx=0 tx=0).stop 2023-10-16T12:10:32.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.785+0000 7fa6c3363700 1 -- 172.21.15.16:0/2454611698 shutdown_connections 2023-10-16T12:10:32.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.785+0000 7fa6c3363700 1 --2- 172.21.15.16:0/2454611698 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6a8004850 0x7fa6a8004cd0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:32.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.785+0000 7fa6c3363700 1 -- 172.21.15.16:0/2454611698 >> 172.21.15.16:0/2454611698 conn(0x7fa6bc0d14b0 msgr2=0x7fa6bc0d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:32.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.785+0000 7fa6c3363700 1 -- 172.21.15.16:0/2454611698 shutdown_connections 2023-10-16T12:10:32.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.785+0000 7fa6c3363700 1 -- 172.21.15.16:0/2454611698 wait complete. 2023-10-16T12:10:32.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.786+0000 7fa6c3363700 1 Processor -- start 2023-10-16T12:10:32.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.786+0000 7fa6c3363700 1 -- start start 2023-10-16T12:10:32.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.787+0000 7fa6c3363700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6a8004850 0x7fa6bc0b1a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:32.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.787+0000 7fa6c3363700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa6bc0c7f40 0x7fa6bc158910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:32.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.787+0000 7fa6c3363700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa6bc15e210 0x7fa6bc15e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:32.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.787+0000 7fa6c3363700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6bc0c83c0 con 0x7fa6a8004850 2023-10-16T12:10:32.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.787+0000 7fa6c3363700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6bc15ebe0 con 0x7fa6bc15e210 2023-10-16T12:10:32.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.787+0000 7fa6c3363700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6bc1f17c0 con 0x7fa6bc0c7f40 2023-10-16T12:10:32.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.787+0000 7fa6c10ff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6a8004850 0x7fa6bc0b1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:32.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.787+0000 7fa6c10ff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6a8004850 0x7fa6bc0b1a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:32814/0 (socket says 172.21.15.16:32814) 2023-10-16T12:10:32.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.787+0000 7fa6c10ff700 1 -- 172.21.15.16:0/805165441 learned_addr learned my addr 172.21.15.16:0/805165441 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:10:32.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.787+0000 7fa6c08fe700 1 --2- 172.21.15.16:0/805165441 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa6bc0c7f40 0x7fa6bc158910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:32.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.787+0000 7fa6c1900700 1 --2- 172.21.15.16:0/805165441 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa6bc15e210 0x7fa6bc15e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:32.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.787+0000 7fa6c10ff700 1 -- 172.21.15.16:0/805165441 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa6bc0c7f40 msgr2=0x7fa6bc158910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:32.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.787+0000 7fa6c10ff700 1 --2- 172.21.15.16:0/805165441 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa6bc0c7f40 0x7fa6bc158910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:32.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.787+0000 7fa6c10ff700 1 -- 172.21.15.16:0/805165441 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa6bc15e210 msgr2=0x7fa6bc15e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:32.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.787+0000 7fa6c10ff700 1 --2- 172.21.15.16:0/805165441 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa6bc15e210 0x7fa6bc15e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:32.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.787+0000 7fa6c10ff700 1 -- 172.21.15.16:0/805165441 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6bc1f0900 con 0x7fa6a8004850 2023-10-16T12:10:32.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.787+0000 7fa6c10ff700 1 --2- 172.21.15.16:0/805165441 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6a8004850 0x7fa6bc0b1a90 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa6ac00f8b0 tx=0x7fa6ac00fbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:32.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.787+0000 7fa6b1ffb700 1 -- 172.21.15.16:0/805165441 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6ac0072e0 con 0x7fa6a8004850 2023-10-16T12:10:32.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.787+0000 7fa6b1ffb700 1 -- 172.21.15.16:0/805165441 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6ac007460 con 0x7fa6a8004850 2023-10-16T12:10:32.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.787+0000 7fa6c3363700 1 -- 172.21.15.16:0/805165441 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6bc1f0bb0 con 0x7fa6a8004850 2023-10-16T12:10:32.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.787+0000 7fa6b1ffb700 1 -- 172.21.15.16:0/805165441 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6ac005830 con 0x7fa6a8004850 2023-10-16T12:10:32.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.788+0000 7fa6c3363700 1 -- 172.21.15.16:0/805165441 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6bc1f1170 con 0x7fa6a8004850 2023-10-16T12:10:32.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.788+0000 7fa6c3363700 1 -- 172.21.15.16:0/805165441 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6bc04ebb0 con 0x7fa6a8004850 2023-10-16T12:10:32.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.789+0000 7fa6b1ffb700 1 -- 172.21.15.16:0/805165441 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 4) v1 ==== 89173+0+0 (secure 0 0 0) 0x7fa6ac005990 con 0x7fa6a8004850 2023-10-16T12:10:32.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.789+0000 7fa6b1ffb700 1 --2- 172.21.15.16:0/805165441 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fa6a006ae70 0x7fa6a006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:32.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.789+0000 7fa6b1ffb700 1 -- 172.21.15.16:0/805165441 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 4471+0+0 (secure 0 0 0) 0x7fa6ac088e50 con 0x7fa6a8004850 2023-10-16T12:10:32.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.789+0000 7fa6c08fe700 1 --2- 172.21.15.16:0/805165441 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fa6a006ae70 0x7fa6a006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:32.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.790+0000 7fa6c08fe700 1 --2- 172.21.15.16:0/805165441 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fa6a006ae70 0x7fa6a006d330 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa6b8005040 tx=0x7fa6b800a000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:32.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:32.794+0000 7fa6b1ffb700 1 -- 172.21.15.16:0/805165441 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa6ac0544c0 con 0x7fa6a8004850 2023-10-16T12:10:33.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:33.044+0000 7fa6c3363700 1 -- 172.21.15.16:0/805165441 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa6bc06b5d0 con 0x7fa6a8004850 2023-10-16T12:10:33.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:33.044+0000 7fa6b1ffb700 1 -- 172.21.15.16:0/805165441 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1324 (secure 0 0 0) 0x7fa6ac057ae0 con 0x7fa6a8004850 2023-10-16T12:10:33.046 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:10:33.046 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4254,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3168304671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":3168304671},{"type":"v1","addr":"172.21.15.16:6837","nonce":3168304671}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2023-10-16T12:10:33.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:33.046+0000 7fa6c3363700 1 -- 172.21.15.16:0/805165441 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fa6a006ae70 msgr2=0x7fa6a006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:33.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:33.046+0000 7fa6c3363700 1 --2- 172.21.15.16:0/805165441 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fa6a006ae70 0x7fa6a006d330 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa6b8005040 tx=0x7fa6b800a000 comp rx=0 tx=0).stop 2023-10-16T12:10:33.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:33.046+0000 7fa6c3363700 1 -- 172.21.15.16:0/805165441 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6a8004850 msgr2=0x7fa6bc0b1a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:33.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:33.046+0000 7fa6c3363700 1 --2- 172.21.15.16:0/805165441 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6a8004850 0x7fa6bc0b1a90 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa6ac00f8b0 tx=0x7fa6ac00fbc0 comp rx=0 tx=0).stop 2023-10-16T12:10:33.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:33.046+0000 7fa6c3363700 1 -- 172.21.15.16:0/805165441 shutdown_connections 2023-10-16T12:10:33.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:33.046+0000 7fa6c3363700 1 --2- 172.21.15.16:0/805165441 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fa6a006ae70 0x7fa6a006d330 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:33.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:33.046+0000 7fa6c3363700 1 --2- 172.21.15.16:0/805165441 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa6bc15e210 0x7fa6bc15e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:33.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:33.046+0000 7fa6c3363700 1 --2- 172.21.15.16:0/805165441 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6a8004850 0x7fa6bc0b1a90 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:33.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:33.046+0000 7fa6c3363700 1 --2- 172.21.15.16:0/805165441 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa6bc0c7f40 0x7fa6bc158910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:33.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:33.046+0000 7fa6c3363700 1 -- 172.21.15.16:0/805165441 >> 172.21.15.16:0/805165441 conn(0x7fa6bc0d14b0 msgr2=0x7fa6bc0b5270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:33.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:33.046+0000 7fa6c3363700 1 -- 172.21.15.16:0/805165441 shutdown_connections 2023-10-16T12:10:33.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:33.046+0000 7fa6c3363700 1 -- 172.21.15.16:0/805165441 wait complete. 2023-10-16T12:10:33.051 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 2 2023-10-16T12:10:35.061 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:10:35.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.188+0000 7f4ff91f5700 1 Processor -- start 2023-10-16T12:10:35.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.188+0000 7f4ff91f5700 1 -- start start 2023-10-16T12:10:35.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.189+0000 7f4ff91f5700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4ff415db40 0x7f4ff40cb980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:35.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.189+0000 7f4ff91f5700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f4ff40ce870 0x7f4ff406ac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:35.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.189+0000 7f4ff91f5700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ff40cedb0 con 0x7f4ff406a7e0 2023-10-16T12:10:35.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.189+0000 7f4ff91f5700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ff406b1a0 con 0x7f4ff415db40 2023-10-16T12:10:35.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.189+0000 7f4ff91f5700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ff415df20 con 0x7f4ff40ce870 2023-10-16T12:10:35.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.189+0000 7f4ff2d9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4ff415db40 0x7f4ff40cb980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:35.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.189+0000 7f4ff259c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f4ff406a7e0 0x7f4ff40ce160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50362/0 (socket says 172.21.15.16:50362) 2023-10-16T12:10:35.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.189+0000 7f4ff259c700 1 -- 172.21.15.16:0/2541483943 learned_addr learned my addr 172.21.15.16:0/2541483943 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:10:35.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.189+0000 7f4ff359e700 1 --2- 172.21.15.16:0/2541483943 >> v2:172.21.15.49:3300/0 conn(0x7f4ff40ce870 0x7f4ff406ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:35.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.189+0000 7f4ff2d9d700 1 -- 172.21.15.16:0/2541483943 >> v2:172.21.15.49:3300/0 conn(0x7f4ff40ce870 msgr2=0x7f4ff406ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:10:35.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.189+0000 7f4ff1d9b700 1 -- 172.21.15.16:0/2541483943 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1809821744 0 0) 0x7f4ff40cedb0 con 0x7f4ff406a7e0 2023-10-16T12:10:35.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.189+0000 7f4ff2d9d700 1 --2- 172.21.15.16:0/2541483943 >> v2:172.21.15.49:3300/0 conn(0x7f4ff40ce870 0x7f4ff406ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:35.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.189+0000 7f4ff2d9d700 1 -- 172.21.15.16:0/2541483943 >> v1:172.21.15.16:6789/0 conn(0x7f4ff406a7e0 legacy=0x7f4ff40ce160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:35.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.189+0000 7f4ff2d9d700 1 -- 172.21.15.16:0/2541483943 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ff40d12c0 con 0x7f4ff415db40 2023-10-16T12:10:35.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.189+0000 7f4ff359e700 1 --2- 172.21.15.16:0/2541483943 >> v2:172.21.15.49:3300/0 conn(0x7f4ff40ce870 0x7f4ff406ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:10:35.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.189+0000 7f4ff2d9d700 1 --2- 172.21.15.16:0/2541483943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ff415db40 0x7f4ff40cb980 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4ff40afee0 tx=0x7f4fe4009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73133d4c1bf01f4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:35.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.189+0000 7f4ff1d9b700 1 -- 172.21.15.16:0/2541483943 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe4011830 con 0x7f4ff415db40 2023-10-16T12:10:35.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.189+0000 7f4ff1d9b700 1 -- 172.21.15.16:0/2541483943 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fe4011990 con 0x7f4ff415db40 2023-10-16T12:10:35.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.189+0000 7f4ff1d9b700 1 -- 172.21.15.16:0/2541483943 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe4011d20 con 0x7f4ff415db40 2023-10-16T12:10:35.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.190+0000 7f4ff91f5700 1 -- 172.21.15.16:0/2541483943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ff415db40 msgr2=0x7f4ff40cb980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:35.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.190+0000 7f4ff91f5700 1 --2- 172.21.15.16:0/2541483943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ff415db40 0x7f4ff40cb980 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4ff40afee0 tx=0x7f4fe4009b20 comp rx=0 tx=0).stop 2023-10-16T12:10:35.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.190+0000 7f4ff91f5700 1 -- 172.21.15.16:0/2541483943 shutdown_connections 2023-10-16T12:10:35.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.190+0000 7f4ff91f5700 1 --2- 172.21.15.16:0/2541483943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ff415db40 0x7f4ff40cb980 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:35.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.190+0000 7f4ff91f5700 1 --2- 172.21.15.16:0/2541483943 >> v2:172.21.15.49:3300/0 conn(0x7f4ff40ce870 0x7f4ff406ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:35.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.190+0000 7f4ff91f5700 1 -- 172.21.15.16:0/2541483943 >> 172.21.15.16:0/2541483943 conn(0x7f4ff40b1fd0 msgr2=0x7f4ff40b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:35.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.190+0000 7f4ff91f5700 1 -- 172.21.15.16:0/2541483943 shutdown_connections 2023-10-16T12:10:35.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.190+0000 7f4ff91f5700 1 -- 172.21.15.16:0/2541483943 wait complete. 2023-10-16T12:10:35.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.191+0000 7f4ff91f5700 1 Processor -- start 2023-10-16T12:10:35.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.191+0000 7f4ff91f5700 1 -- start start 2023-10-16T12:10:35.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.192+0000 7f4ff91f5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ff406a7e0 0x7f4ff40c9c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:35.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.192+0000 7f4ff91f5700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4ff40ce870 0x7f4ff40c5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:35.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.192+0000 7f4ff91f5700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4ff415db40 0x7f4ff41f3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:35.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.192+0000 7f4ff91f5700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff40ca1b0 con 0x7f4ff406a7e0 2023-10-16T12:10:35.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.192+0000 7f4ff91f5700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff40c9840 con 0x7f4ff415db40 2023-10-16T12:10:35.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.192+0000 7f4ff91f5700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff40c44e0 con 0x7f4ff40ce870 2023-10-16T12:10:35.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.192+0000 7f4ff259c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4ff40ce870 0x7f4ff40c5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:35.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.192+0000 7f4ff259c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4ff40ce870 0x7f4ff40c5ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:49352/0 (socket says 172.21.15.16:49352) 2023-10-16T12:10:35.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.192+0000 7f4ff359e700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4ff415db40 0x7f4ff41f3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:35.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.192+0000 7f4ff259c700 1 -- 172.21.15.16:0/675987502 learned_addr learned my addr 172.21.15.16:0/675987502 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:10:35.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.193+0000 7f4ff2d9d700 1 --2- 172.21.15.16:0/675987502 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ff406a7e0 0x7f4ff40c9c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:35.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.193+0000 7f4ff359e700 1 -- 172.21.15.16:0/675987502 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4ff40ce870 msgr2=0x7f4ff40c5ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:35.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.193+0000 7f4ff359e700 1 --2- 172.21.15.16:0/675987502 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4ff40ce870 0x7f4ff40c5ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:35.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.193+0000 7f4ff359e700 1 -- 172.21.15.16:0/675987502 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ff406a7e0 msgr2=0x7f4ff40c9c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:35.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.193+0000 7f4ff359e700 1 --2- 172.21.15.16:0/675987502 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ff406a7e0 0x7f4ff40c9c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:35.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.193+0000 7f4ff359e700 1 -- 172.21.15.16:0/675987502 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fe40094a0 con 0x7f4ff415db40 2023-10-16T12:10:35.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.193+0000 7f4ff259c700 1 --2- 172.21.15.16:0/675987502 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4ff40ce870 0x7f4ff40c5ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:10:35.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.193+0000 7f4ff359e700 1 --2- 172.21.15.16:0/675987502 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4ff415db40 0x7f4ff41f3ed0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4fec00cd10 tx=0x7f4fec00a560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:35.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.193+0000 7f4fdb7fe700 1 -- 172.21.15.16:0/675987502 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fec00acc0 con 0x7f4ff415db40 2023-10-16T12:10:35.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.193+0000 7f4fdb7fe700 1 -- 172.21.15.16:0/675987502 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fec00ae40 con 0x7f4ff415db40 2023-10-16T12:10:35.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.193+0000 7f4ff91f5700 1 -- 172.21.15.16:0/675987502 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ff40c47c0 con 0x7f4ff415db40 2023-10-16T12:10:35.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.193+0000 7f4fdb7fe700 1 -- 172.21.15.16:0/675987502 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fec00f6c0 con 0x7f4ff415db40 2023-10-16T12:10:35.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.193+0000 7f4ff91f5700 1 -- 172.21.15.16:0/675987502 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ff40c4d80 con 0x7f4ff415db40 2023-10-16T12:10:35.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.195+0000 7f4fdb7fe700 1 -- 172.21.15.16:0/675987502 <== mon.1 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 4) v1 ==== 89173+0+0 (secure 0 0 0) 0x7f4fec00b040 con 0x7f4ff415db40 2023-10-16T12:10:35.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.195+0000 7f4ff91f5700 1 -- 172.21.15.16:0/675987502 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ff404ebb0 con 0x7f4ff415db40 2023-10-16T12:10:35.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.195+0000 7f4fdb7fe700 1 --2- 172.21.15.16:0/675987502 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f4fe006ae70 0x7f4fe006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:35.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.195+0000 7f4fdb7fe700 1 -- 172.21.15.16:0/675987502 <== mon.1 v2:172.21.15.49:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f4fec0897a0 con 0x7f4ff415db40 2023-10-16T12:10:35.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.195+0000 7f4ff2d9d700 1 --2- 172.21.15.16:0/675987502 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f4fe006ae70 0x7f4fe006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:35.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.196+0000 7f4ff2d9d700 1 --2- 172.21.15.16:0/675987502 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f4fe006ae70 0x7f4fe006d330 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4fe40090b0 tx=0x7f4fe4008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:35.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.200+0000 7f4fdb7fe700 1 -- 172.21.15.16:0/675987502 <== mon.1 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4fec054c80 con 0x7f4ff415db40 2023-10-16T12:10:35.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.465+0000 7f4ff91f5700 1 -- 172.21.15.16:0/675987502 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4ff415cd00 con 0x7f4ff415db40 2023-10-16T12:10:35.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.466+0000 7f4fdb7fe700 1 -- 172.21.15.16:0/675987502 <== mon.1 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3694 (secure 0 0 0) 0x7f4fec0582a0 con 0x7f4ff415db40 2023-10-16T12:10:35.468 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:10:35.468 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/2979266800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":2979266800},{"type":"v1","addr":"172.21.15.49:6834","nonce":2979266800}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4251,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/1398336801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1398336801},{"type":"v1","addr":"172.21.15.16:6836","nonce":1398336801}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4254,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3168304671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":3168304671},{"type":"v1","addr":"172.21.15.16:6837","nonce":3168304671}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/2892536404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":2892536404},{"type":"v1","addr":"172.21.15.49:6835","nonce":2892536404}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2023-10-16T12:10:35.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.468+0000 7f4ff91f5700 1 -- 172.21.15.16:0/675987502 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f4fe006ae70 msgr2=0x7f4fe006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:35.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.468+0000 7f4ff91f5700 1 --2- 172.21.15.16:0/675987502 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f4fe006ae70 0x7f4fe006d330 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4fe40090b0 tx=0x7f4fe4008030 comp rx=0 tx=0).stop 2023-10-16T12:10:35.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.468+0000 7f4ff91f5700 1 -- 172.21.15.16:0/675987502 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4ff415db40 msgr2=0x7f4ff41f3ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:35.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.468+0000 7f4ff91f5700 1 --2- 172.21.15.16:0/675987502 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4ff415db40 0x7f4ff41f3ed0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4fec00cd10 tx=0x7f4fec00a560 comp rx=0 tx=0).stop 2023-10-16T12:10:35.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.468+0000 7f4ff91f5700 1 -- 172.21.15.16:0/675987502 shutdown_connections 2023-10-16T12:10:35.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.468+0000 7f4ff91f5700 1 --2- 172.21.15.16:0/675987502 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f4fe006ae70 0x7f4fe006d330 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:35.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.468+0000 7f4ff91f5700 1 --2- 172.21.15.16:0/675987502 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4ff415db40 0x7f4ff41f3ed0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:35.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.468+0000 7f4ff91f5700 1 --2- 172.21.15.16:0/675987502 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ff406a7e0 0x7f4ff40c9c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:35.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.468+0000 7f4ff91f5700 1 --2- 172.21.15.16:0/675987502 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4ff40ce870 0x7f4ff40c5ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:35.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.468+0000 7f4ff91f5700 1 -- 172.21.15.16:0/675987502 >> 172.21.15.16:0/675987502 conn(0x7f4ff40b1fd0 msgr2=0x7f4ff40d0ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:35.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.468+0000 7f4ff91f5700 1 -- 172.21.15.16:0/675987502 shutdown_connections 2023-10-16T12:10:35.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.468+0000 7f4ff91f5700 1 -- 172.21.15.16:0/675987502 wait complete. 2023-10-16T12:10:35.473 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 3 2023-10-16T12:10:35.482 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:10:35.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.608+0000 7f9d40637700 1 Processor -- start 2023-10-16T12:10:35.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.608+0000 7f9d40637700 1 -- start start 2023-10-16T12:10:35.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.608+0000 7f9d40637700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9d38157760 0x7f9d380cf470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:35.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.609+0000 7f9d40637700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f9d38157b40 0x7f9d3806ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:35.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.609+0000 7f9d40637700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d380cbf40 con 0x7f9d3815da80 2023-10-16T12:10:35.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.609+0000 7f9d40637700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d3806ac60 con 0x7f9d38157760 2023-10-16T12:10:35.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.609+0000 7f9d40637700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d3806f950 con 0x7f9d38157b40 2023-10-16T12:10:35.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.609+0000 7f9d3e3d3700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9d38157760 0x7f9d380cf470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:35.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.609+0000 7f9d3e3d3700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9d38157760 0x7f9d380cf470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:32874/0 (socket says 172.21.15.16:32874) 2023-10-16T12:10:35.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.609+0000 7f9d3e3d3700 1 -- 172.21.15.16:0/1808657993 learned_addr learned my addr 172.21.15.16:0/1808657993 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:10:35.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.609+0000 7f9d3dbd2700 1 --2- 172.21.15.16:0/1808657993 >> v2:172.21.15.49:3300/0 conn(0x7f9d38157b40 0x7f9d3806ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:35.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.609+0000 7f9d3e3d3700 1 -- 172.21.15.16:0/1808657993 >> v2:172.21.15.49:3300/0 conn(0x7f9d38157b40 msgr2=0x7f9d3806ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:10:35.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.609+0000 7f9d3e3d3700 1 --2- 172.21.15.16:0/1808657993 >> v2:172.21.15.49:3300/0 conn(0x7f9d38157b40 0x7f9d3806ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:35.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.609+0000 7f9d3e3d3700 1 -- 172.21.15.16:0/1808657993 >> v1:172.21.15.49:6789/0 conn(0x7f9d3815da80 legacy=0x7f9d380ae520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:35.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.609+0000 7f9d3e3d3700 1 -- 172.21.15.16:0/1808657993 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d3806f110 con 0x7f9d38157760 2023-10-16T12:10:35.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.609+0000 7f9d3dbd2700 1 --2- 172.21.15.16:0/1808657993 >> v2:172.21.15.49:3300/0 conn(0x7f9d38157b40 0x7f9d3806ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:10:35.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.609+0000 7f9d3e3d3700 1 --2- 172.21.15.16:0/1808657993 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d38157760 0x7f9d380cf470 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9d380c6b90 tx=0x7f9d3400bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af9013477985ea71 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:35.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.609+0000 7f9d3d3d1700 1 -- 172.21.15.16:0/1808657993 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d3400dd10 con 0x7f9d38157760 2023-10-16T12:10:35.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.609+0000 7f9d3d3d1700 1 -- 172.21.15.16:0/1808657993 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d3400de70 con 0x7f9d38157760 2023-10-16T12:10:35.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.609+0000 7f9d3d3d1700 1 -- 172.21.15.16:0/1808657993 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d34010670 con 0x7f9d38157760 2023-10-16T12:10:35.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.610+0000 7f9d40637700 1 -- 172.21.15.16:0/1808657993 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d38157760 msgr2=0x7f9d380cf470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:35.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.610+0000 7f9d40637700 1 --2- 172.21.15.16:0/1808657993 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d38157760 0x7f9d380cf470 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9d380c6b90 tx=0x7f9d3400bea0 comp rx=0 tx=0).stop 2023-10-16T12:10:35.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.610+0000 7f9d40637700 1 -- 172.21.15.16:0/1808657993 shutdown_connections 2023-10-16T12:10:35.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.610+0000 7f9d40637700 1 --2- 172.21.15.16:0/1808657993 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d38157760 0x7f9d380cf470 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:35.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.610+0000 7f9d40637700 1 --2- 172.21.15.16:0/1808657993 >> v2:172.21.15.49:3300/0 conn(0x7f9d38157b40 0x7f9d3806ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:35.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.610+0000 7f9d40637700 1 -- 172.21.15.16:0/1808657993 >> 172.21.15.16:0/1808657993 conn(0x7f9d380cc5b0 msgr2=0x7f9d380cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:35.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.610+0000 7f9d40637700 1 -- 172.21.15.16:0/1808657993 shutdown_connections 2023-10-16T12:10:35.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.610+0000 7f9d40637700 1 -- 172.21.15.16:0/1808657993 wait complete. 2023-10-16T12:10:35.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.611+0000 7f9d40637700 1 Processor -- start 2023-10-16T12:10:35.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.611+0000 7f9d40637700 1 -- start start 2023-10-16T12:10:35.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.612+0000 7f9d40637700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d38157760 0x7f9d380b1050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:35.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.612+0000 7f9d40637700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9d38157b40 0x7f9d38155000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:35.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.612+0000 7f9d40637700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9d3815da80 0x7f9d3815c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:35.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.612+0000 7f9d40637700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3815c070 con 0x7f9d38157760 2023-10-16T12:10:35.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.612+0000 7f9d40637700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d380c7870 con 0x7f9d38157b40 2023-10-16T12:10:35.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.612+0000 7f9d40637700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d380c7b50 con 0x7f9d3815da80 2023-10-16T12:10:35.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.612+0000 7f9d3e3d3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d38157760 0x7f9d380b1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:35.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.612+0000 7f9d3e3d3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d38157760 0x7f9d380b1050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:32882/0 (socket says 172.21.15.16:32882) 2023-10-16T12:10:35.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.612+0000 7f9d3e3d3700 1 -- 172.21.15.16:0/2067839160 learned_addr learned my addr 172.21.15.16:0/2067839160 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:10:35.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.612+0000 7f9d3ebd4700 1 --2- 172.21.15.16:0/2067839160 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9d3815da80 0x7f9d3815c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:35.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.612+0000 7f9d3dbd2700 1 --2- 172.21.15.16:0/2067839160 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9d38157b40 0x7f9d38155000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:35.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.612+0000 7f9d3e3d3700 1 -- 172.21.15.16:0/2067839160 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9d3815da80 msgr2=0x7f9d3815c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:35.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.612+0000 7f9d3e3d3700 1 --2- 172.21.15.16:0/2067839160 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9d3815da80 0x7f9d3815c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:35.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.612+0000 7f9d3e3d3700 1 -- 172.21.15.16:0/2067839160 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9d38157b40 msgr2=0x7f9d38155000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:35.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.612+0000 7f9d3e3d3700 1 --2- 172.21.15.16:0/2067839160 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9d38157b40 0x7f9d38155000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:35.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.612+0000 7f9d3e3d3700 1 -- 172.21.15.16:0/2067839160 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d3400b7f0 con 0x7f9d38157760 2023-10-16T12:10:35.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.612+0000 7f9d3dbd2700 1 --2- 172.21.15.16:0/2067839160 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9d38157b40 0x7f9d38155000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:10:35.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.612+0000 7f9d3e3d3700 1 --2- 172.21.15.16:0/2067839160 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d38157760 0x7f9d380b1050 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9d34000c00 tx=0x7f9d34004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:35.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.613+0000 7f9d2effd700 1 -- 172.21.15.16:0/2067839160 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d3401f070 con 0x7f9d38157760 2023-10-16T12:10:35.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.613+0000 7f9d2effd700 1 -- 172.21.15.16:0/2067839160 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d340039f0 con 0x7f9d38157760 2023-10-16T12:10:35.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.613+0000 7f9d40637700 1 -- 172.21.15.16:0/2067839160 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d3815c310 con 0x7f9d38157760 2023-10-16T12:10:35.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.613+0000 7f9d2effd700 1 -- 172.21.15.16:0/2067839160 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d34003d80 con 0x7f9d38157760 2023-10-16T12:10:35.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.613+0000 7f9d40637700 1 -- 172.21.15.16:0/2067839160 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d380c7e30 con 0x7f9d38157760 2023-10-16T12:10:35.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.613+0000 7f9d40637700 1 -- 172.21.15.16:0/2067839160 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d3804ebb0 con 0x7f9d38157760 2023-10-16T12:10:35.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.614+0000 7f9d2effd700 1 -- 172.21.15.16:0/2067839160 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 4) v1 ==== 89173+0+0 (secure 0 0 0) 0x7f9d34019400 con 0x7f9d38157760 2023-10-16T12:10:35.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.615+0000 7f9d2effd700 1 --2- 172.21.15.16:0/2067839160 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f9d1c06ae70 0x7f9d1c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:35.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.615+0000 7f9d2effd700 1 -- 172.21.15.16:0/2067839160 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f9d3408c380 con 0x7f9d38157760 2023-10-16T12:10:35.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.615+0000 7f9d3dbd2700 1 --2- 172.21.15.16:0/2067839160 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f9d1c06ae70 0x7f9d1c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:35.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.615+0000 7f9d3dbd2700 1 --2- 172.21.15.16:0/2067839160 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f9d1c06ae70 0x7f9d1c06d330 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9d381f2280 tx=0x7f9d28008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:35.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.620+0000 7f9d2effd700 1 -- 172.21.15.16:0/2067839160 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9d340578f0 con 0x7f9d38157760 2023-10-16T12:10:35.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.827+0000 7f9d40637700 1 -- 172.21.15.16:0/2067839160 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9d380ce410 con 0x7f9d38157760 2023-10-16T12:10:35.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.828+0000 7f9d2effd700 1 -- 172.21.15.16:0/2067839160 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+12742 (secure 0 0 0) 0x7f9d3405af10 con 0x7f9d38157760 2023-10-16T12:10:35.830 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:10:35.830 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":17,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:10:35.586476+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:10:35.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.830+0000 7f9d40637700 1 -- 172.21.15.16:0/2067839160 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f9d1c06ae70 msgr2=0x7f9d1c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:35.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.830+0000 7f9d40637700 1 --2- 172.21.15.16:0/2067839160 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f9d1c06ae70 0x7f9d1c06d330 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9d381f2280 tx=0x7f9d28008040 comp rx=0 tx=0).stop 2023-10-16T12:10:35.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.830+0000 7f9d40637700 1 -- 172.21.15.16:0/2067839160 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d38157760 msgr2=0x7f9d380b1050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:35.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.830+0000 7f9d40637700 1 --2- 172.21.15.16:0/2067839160 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d38157760 0x7f9d380b1050 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9d34000c00 tx=0x7f9d34004750 comp rx=0 tx=0).stop 2023-10-16T12:10:35.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.830+0000 7f9d40637700 1 -- 172.21.15.16:0/2067839160 shutdown_connections 2023-10-16T12:10:35.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.830+0000 7f9d40637700 1 --2- 172.21.15.16:0/2067839160 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f9d1c06ae70 0x7f9d1c06d330 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:35.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.830+0000 7f9d40637700 1 --2- 172.21.15.16:0/2067839160 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9d3815da80 0x7f9d3815c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:35.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.830+0000 7f9d40637700 1 --2- 172.21.15.16:0/2067839160 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d38157760 0x7f9d380b1050 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:35.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.830+0000 7f9d40637700 1 --2- 172.21.15.16:0/2067839160 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9d38157b40 0x7f9d38155000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:35.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.830+0000 7f9d40637700 1 -- 172.21.15.16:0/2067839160 >> 172.21.15.16:0/2067839160 conn(0x7f9d380cc5b0 msgr2=0x7f9d3815def0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:35.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.830+0000 7f9d40637700 1 -- 172.21.15.16:0/2067839160 shutdown_connections 2023-10-16T12:10:35.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.830+0000 7f9d40637700 1 -- 172.21.15.16:0/2067839160 wait complete. 2023-10-16T12:10:35.844 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T12:10:35.845 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:10:35.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.970+0000 7f136c4e0700 1 Processor -- start 2023-10-16T12:10:35.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.970+0000 7f136c4e0700 1 -- start start 2023-10-16T12:10:35.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.971+0000 7f136c4e0700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f136415b070 0x7f136415b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:35.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.971+0000 7f136c4e0700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f13640cb880 0x7f13640cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:35.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.971+0000 7f136c4e0700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13640cef70 con 0x7f136406da60 2023-10-16T12:10:35.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.971+0000 7f136c4e0700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13640ce360 con 0x7f136415b070 2023-10-16T12:10:35.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.971+0000 7f136c4e0700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13640ce640 con 0x7f13640cb880 2023-10-16T12:10:35.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.971+0000 7f136aa7d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f136406da60 0x7f13640cf400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50406/0 (socket says 172.21.15.16:50406) 2023-10-16T12:10:35.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.971+0000 7f136aa7d700 1 -- 172.21.15.16:0/2724534595 learned_addr learned my addr 172.21.15.16:0/2724534595 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:10:35.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.971+0000 7f136a27c700 1 --2- 172.21.15.16:0/2724534595 >> v2:172.21.15.49:3300/0 conn(0x7f136415b070 0x7f136415b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:35.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.972+0000 7f1369a7b700 1 --2- 172.21.15.16:0/2724534595 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f13640cb880 0x7f13640cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:35.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.972+0000 7f136927a700 1 -- 172.21.15.16:0/2724534595 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1015324012 0 0) 0x7f13640cef70 con 0x7f136406da60 2023-10-16T12:10:35.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.972+0000 7f136927a700 1 -- 172.21.15.16:0/2724534595 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13500035b0 con 0x7f136406da60 2023-10-16T12:10:35.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.972+0000 7f136a27c700 1 -- 172.21.15.16:0/2724534595 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f13640cb880 msgr2=0x7f13640cbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:10:35.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.972+0000 7f136a27c700 1 --2- 172.21.15.16:0/2724534595 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f13640cb880 0x7f13640cbd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:35.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.972+0000 7f136a27c700 1 -- 172.21.15.16:0/2724534595 >> v1:172.21.15.16:6789/0 conn(0x7f136406da60 legacy=0x7f13640cf400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:35.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.972+0000 7f136a27c700 1 -- 172.21.15.16:0/2724534595 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13640ce920 con 0x7f136415b070 2023-10-16T12:10:35.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.972+0000 7f136a27c700 1 --2- 172.21.15.16:0/2724534595 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f136415b070 0x7f136415b450 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1354009840 tx=0x7f1354009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=30c048a0fcc016d8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:35.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.972+0000 7f136927a700 1 -- 172.21.15.16:0/2724534595 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1354011830 con 0x7f136415b070 2023-10-16T12:10:35.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.972+0000 7f136927a700 1 -- 172.21.15.16:0/2724534595 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1354011990 con 0x7f136415b070 2023-10-16T12:10:35.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.972+0000 7f136927a700 1 -- 172.21.15.16:0/2724534595 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1354011d20 con 0x7f136415b070 2023-10-16T12:10:35.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.973+0000 7f136c4e0700 1 -- 172.21.15.16:0/2724534595 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f136415b070 msgr2=0x7f136415b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:35.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.973+0000 7f136c4e0700 1 --2- 172.21.15.16:0/2724534595 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f136415b070 0x7f136415b450 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1354009840 tx=0x7f1354009b50 comp rx=0 tx=0).stop 2023-10-16T12:10:35.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.973+0000 7f136c4e0700 1 -- 172.21.15.16:0/2724534595 shutdown_connections 2023-10-16T12:10:35.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.973+0000 7f136c4e0700 1 --2- 172.21.15.16:0/2724534595 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f136415b070 0x7f136415b450 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:35.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.973+0000 7f136c4e0700 1 --2- 172.21.15.16:0/2724534595 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f13640cb880 0x7f13640cbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:35.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.973+0000 7f136c4e0700 1 -- 172.21.15.16:0/2724534595 >> 172.21.15.16:0/2724534595 conn(0x7f13640b1f40 msgr2=0x7f13640b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:35.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.973+0000 7f136c4e0700 1 -- 172.21.15.16:0/2724534595 shutdown_connections 2023-10-16T12:10:35.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.973+0000 7f136c4e0700 1 -- 172.21.15.16:0/2724534595 wait complete. 2023-10-16T12:10:35.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.974+0000 7f136c4e0700 1 Processor -- start 2023-10-16T12:10:35.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.974+0000 7f136c4e0700 1 -- start start 2023-10-16T12:10:35.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.975+0000 7f136c4e0700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f136406da60 0x7f1364151450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:35.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.975+0000 7f136c4e0700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f13640cb880 0x7f13640cc1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:35.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.975+0000 7f136c4e0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f136415b070 0x7f13640cc710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:35.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.975+0000 7f136c4e0700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13641f6ec0 con 0x7f136415b070 2023-10-16T12:10:35.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.975+0000 7f136c4e0700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13640b4150 con 0x7f13640cb880 2023-10-16T12:10:35.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.975+0000 7f136c4e0700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1364163990 con 0x7f136406da60 2023-10-16T12:10:35.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.975+0000 7f136aa7d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f136415b070 0x7f13640cc710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:35.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.975+0000 7f1369a7b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f13640cb880 0x7f13640cc1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:35.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.975+0000 7f1369a7b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f13640cb880 0x7f13640cc1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:60198/0 (socket says 172.21.15.16:60198) 2023-10-16T12:10:35.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.975+0000 7f136aa7d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f136415b070 0x7f13640cc710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:32906/0 (socket says 172.21.15.16:32906) 2023-10-16T12:10:35.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.975+0000 7f136aa7d700 1 -- 172.21.15.16:0/471745036 learned_addr learned my addr 172.21.15.16:0/471745036 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:10:35.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.975+0000 7f136a27c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f136406da60 0x7f1364151450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:35.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.975+0000 7f136aa7d700 1 -- 172.21.15.16:0/471745036 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f136406da60 msgr2=0x7f1364151450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:35.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.975+0000 7f136aa7d700 1 --2- 172.21.15.16:0/471745036 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f136406da60 0x7f1364151450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:35.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.975+0000 7f136aa7d700 1 -- 172.21.15.16:0/471745036 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f13640cb880 msgr2=0x7f13640cc1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:35.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.975+0000 7f136aa7d700 1 --2- 172.21.15.16:0/471745036 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f13640cb880 0x7f13640cc1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:35.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.975+0000 7f136aa7d700 1 -- 172.21.15.16:0/471745036 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13540094a0 con 0x7f136415b070 2023-10-16T12:10:35.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.976+0000 7f136aa7d700 1 --2- 172.21.15.16:0/471745036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f136415b070 0x7f13640cc710 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f135c00cb70 tx=0x7f135c00ce80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:35.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.976+0000 7f135affd700 1 -- 172.21.15.16:0/471745036 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f135c00aa60 con 0x7f136415b070 2023-10-16T12:10:35.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.976+0000 7f135affd700 1 -- 172.21.15.16:0/471745036 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f135c00abc0 con 0x7f136415b070 2023-10-16T12:10:35.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.976+0000 7f136c4e0700 1 -- 172.21.15.16:0/471745036 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1364167530 con 0x7f136415b070 2023-10-16T12:10:35.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.976+0000 7f135affd700 1 -- 172.21.15.16:0/471745036 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f135c012400 con 0x7f136415b070 2023-10-16T12:10:35.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.976+0000 7f136c4e0700 1 -- 172.21.15.16:0/471745036 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1364167af0 con 0x7f136415b070 2023-10-16T12:10:35.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.977+0000 7f136c4e0700 1 -- 172.21.15.16:0/471745036 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f136404ebb0 con 0x7f136415b070 2023-10-16T12:10:35.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.977+0000 7f135affd700 1 -- 172.21.15.16:0/471745036 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 4) v1 ==== 89173+0+0 (secure 0 0 0) 0x7f135c01b3b0 con 0x7f136415b070 2023-10-16T12:10:35.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.978+0000 7f135affd700 1 --2- 172.21.15.16:0/471745036 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f134806ae70 0x7f134806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:35.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.978+0000 7f135affd700 1 -- 172.21.15.16:0/471745036 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f135c088e20 con 0x7f136415b070 2023-10-16T12:10:35.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.978+0000 7f136a27c700 1 --2- 172.21.15.16:0/471745036 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f134806ae70 0x7f134806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:35.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.978+0000 7f136a27c700 1 --2- 172.21.15.16:0/471745036 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f134806ae70 0x7f134806d330 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1354000c00 tx=0x7f135401c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:35.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:35.983+0000 7f135affd700 1 -- 172.21.15.16:0/471745036 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f135c054300 con 0x7f136415b070 2023-10-16T12:10:36.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.009+0000 7f135affd700 1 -- 172.21.15.16:0/471745036 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f135c0546d0 con 0x7f136415b070 2023-10-16T12:10:36.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.190+0000 7f136c4e0700 1 -- 172.21.15.16:0/471745036 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f13640b4290 con 0x7f136415b070 2023-10-16T12:10:36.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.190+0000 7f135affd700 1 -- 172.21.15.16:0/471745036 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) v1 ==== 93+0+10 (secure 0 0 0) 0x7f135c073f70 con 0x7f136415b070 2023-10-16T12:10:36.193 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T12:10:36.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.192+0000 7f136c4e0700 1 -- 172.21.15.16:0/471745036 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f134806ae70 msgr2=0x7f134806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:36.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.192+0000 7f136c4e0700 1 --2- 172.21.15.16:0/471745036 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f134806ae70 0x7f134806d330 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1354000c00 tx=0x7f135401c040 comp rx=0 tx=0).stop 2023-10-16T12:10:36.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.192+0000 7f136c4e0700 1 -- 172.21.15.16:0/471745036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f136415b070 msgr2=0x7f13640cc710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:36.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.192+0000 7f136c4e0700 1 --2- 172.21.15.16:0/471745036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f136415b070 0x7f13640cc710 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f135c00cb70 tx=0x7f135c00ce80 comp rx=0 tx=0).stop 2023-10-16T12:10:36.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.192+0000 7f136c4e0700 1 -- 172.21.15.16:0/471745036 shutdown_connections 2023-10-16T12:10:36.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.192+0000 7f136c4e0700 1 --2- 172.21.15.16:0/471745036 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f134806ae70 0x7f134806d330 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:36.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.192+0000 7f136c4e0700 1 --2- 172.21.15.16:0/471745036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f136415b070 0x7f13640cc710 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:36.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.192+0000 7f136c4e0700 1 --2- 172.21.15.16:0/471745036 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f136406da60 0x7f1364151450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:36.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.192+0000 7f136c4e0700 1 --2- 172.21.15.16:0/471745036 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f13640cb880 0x7f13640cc1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:36.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.192+0000 7f136c4e0700 1 -- 172.21.15.16:0/471745036 >> 172.21.15.16:0/471745036 conn(0x7f13640b1f40 msgr2=0x7f13640d0380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:36.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.193+0000 7f136c4e0700 1 -- 172.21.15.16:0/471745036 shutdown_connections 2023-10-16T12:10:36.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.193+0000 7f136c4e0700 1 -- 172.21.15.16:0/471745036 wait complete. 2023-10-16T12:10:36.206 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-16T12:10:36.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.329+0000 7f8a110e6700 1 Processor -- start 2023-10-16T12:10:36.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.329+0000 7f8a110e6700 1 -- start start 2023-10-16T12:10:36.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.330+0000 7f8a110e6700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f8a0c15b280 0x7f8a0c1613e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:36.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.330+0000 7f8a110e6700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a0c161c30 con 0x7f8a0c06a7e0 2023-10-16T12:10:36.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.330+0000 7f8a110e6700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a0c161f10 con 0x7f8a0c0cc8a0 2023-10-16T12:10:36.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.330+0000 7f8a110e6700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a0c1621f0 con 0x7f8a0c15b280 2023-10-16T12:10:36.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.330+0000 7f8a0b59e700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f8a0c06a7e0 0x7f8a0c0d02b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50420/0 (socket says 172.21.15.16:50420) 2023-10-16T12:10:36.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.330+0000 7f8a0b59e700 1 -- 172.21.15.16:0/1300955005 learned_addr learned my addr 172.21.15.16:0/1300955005 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:10:36.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.330+0000 7f8a0ad9d700 1 --2- 172.21.15.16:0/1300955005 >> v2:172.21.15.49:3300/0 conn(0x7f8a0c15b280 0x7f8a0c1613e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:36.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.331+0000 7f8a09d9b700 1 -- 172.21.15.16:0/1300955005 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1283557541 0 0) 0x7f8a0c161c30 con 0x7f8a0c06a7e0 2023-10-16T12:10:36.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.331+0000 7f8a09d9b700 1 -- 172.21.15.16:0/1300955005 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89f40035b0 con 0x7f8a0c06a7e0 2023-10-16T12:10:36.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.331+0000 7f8a09d9b700 1 -- 172.21.15.16:0/1300955005 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 388985588 0 0) 0x7f8a0c161f10 con 0x7f8a0c0cc8a0 2023-10-16T12:10:36.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.331+0000 7f8a09d9b700 1 -- 172.21.15.16:0/1300955005 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a0c161c30 con 0x7f8a0c0cc8a0 2023-10-16T12:10:36.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.331+0000 7f8a0ad9d700 1 -- 172.21.15.16:0/1300955005 >> v1:172.21.15.49:6789/0 conn(0x7f8a0c0cc8a0 legacy=0x7f8a0c15e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:36.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.331+0000 7f8a0ad9d700 1 -- 172.21.15.16:0/1300955005 >> v1:172.21.15.16:6789/0 conn(0x7f8a0c06a7e0 legacy=0x7f8a0c0d02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:36.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.331+0000 7f8a0ad9d700 1 -- 172.21.15.16:0/1300955005 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a0c15df00 con 0x7f8a0c15b280 2023-10-16T12:10:36.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.331+0000 7f8a0ad9d700 1 --2- 172.21.15.16:0/1300955005 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8a0c15b280 0x7f8a0c1613e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f8a00009840 tx=0x7f8a00009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ddcd44d627ad040 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:36.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.331+0000 7f8a09d9b700 1 -- 172.21.15.16:0/1300955005 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a00011830 con 0x7f8a0c15b280 2023-10-16T12:10:36.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.331+0000 7f8a09d9b700 1 -- 172.21.15.16:0/1300955005 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a00011990 con 0x7f8a0c15b280 2023-10-16T12:10:36.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.331+0000 7f8a09d9b700 1 -- 172.21.15.16:0/1300955005 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a00011d20 con 0x7f8a0c15b280 2023-10-16T12:10:36.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.332+0000 7f8a110e6700 1 -- 172.21.15.16:0/1300955005 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8a0c15b280 msgr2=0x7f8a0c1613e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:36.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.332+0000 7f8a110e6700 1 --2- 172.21.15.16:0/1300955005 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8a0c15b280 0x7f8a0c1613e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f8a00009840 tx=0x7f8a00009b50 comp rx=0 tx=0).stop 2023-10-16T12:10:36.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.332+0000 7f8a110e6700 1 -- 172.21.15.16:0/1300955005 shutdown_connections 2023-10-16T12:10:36.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.332+0000 7f8a110e6700 1 --2- 172.21.15.16:0/1300955005 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8a0c15b280 0x7f8a0c1613e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:36.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.332+0000 7f8a110e6700 1 -- 172.21.15.16:0/1300955005 >> 172.21.15.16:0/1300955005 conn(0x7f8a0c0b1fd0 msgr2=0x7f8a0c0b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:36.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.332+0000 7f8a110e6700 1 -- 172.21.15.16:0/1300955005 shutdown_connections 2023-10-16T12:10:36.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.332+0000 7f8a110e6700 1 -- 172.21.15.16:0/1300955005 wait complete. 2023-10-16T12:10:36.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.333+0000 7f8a110e6700 1 Processor -- start 2023-10-16T12:10:36.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.333+0000 7f8a110e6700 1 -- start start 2023-10-16T12:10:36.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.334+0000 7f8a110e6700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8a0c06a7e0 0x7f8a0c14a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:36.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.334+0000 7f8a110e6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a0c0cc8a0 0x7f8a0c14d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:36.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.334+0000 7f8a110e6700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8a0c15b280 0x7f8a0c15b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:36.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.334+0000 7f8a110e6700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a0c14daa0 con 0x7f8a0c0cc8a0 2023-10-16T12:10:36.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.334+0000 7f8a110e6700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a0c15cd30 con 0x7f8a0c15b280 2023-10-16T12:10:36.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.334+0000 7f8a110e6700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a0c1478e0 con 0x7f8a0c06a7e0 2023-10-16T12:10:36.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.334+0000 7f8a0a59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a0c0cc8a0 0x7f8a0c14d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:36.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.334+0000 7f8a0a59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a0c0cc8a0 0x7f8a0c14d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:32918/0 (socket says 172.21.15.16:32918) 2023-10-16T12:10:36.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.334+0000 7f8a0a59c700 1 -- 172.21.15.16:0/864593078 learned_addr learned my addr 172.21.15.16:0/864593078 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:10:36.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.334+0000 7f8a0ad9d700 1 --2- 172.21.15.16:0/864593078 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8a0c06a7e0 0x7f8a0c14a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:36.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.334+0000 7f8a0b59e700 1 --2- 172.21.15.16:0/864593078 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8a0c15b280 0x7f8a0c15b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:36.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.334+0000 7f8a0a59c700 1 -- 172.21.15.16:0/864593078 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8a0c06a7e0 msgr2=0x7f8a0c14a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:36.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.334+0000 7f8a0a59c700 1 --2- 172.21.15.16:0/864593078 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8a0c06a7e0 0x7f8a0c14a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:36.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.334+0000 7f8a0a59c700 1 -- 172.21.15.16:0/864593078 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8a0c15b280 msgr2=0x7f8a0c15b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:36.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.334+0000 7f8a0a59c700 1 --2- 172.21.15.16:0/864593078 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8a0c15b280 0x7f8a0c15b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:36.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.334+0000 7f8a0a59c700 1 -- 172.21.15.16:0/864593078 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a000094a0 con 0x7f8a0c0cc8a0 2023-10-16T12:10:36.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.334+0000 7f8a0a59c700 1 --2- 172.21.15.16:0/864593078 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a0c0cc8a0 0x7f8a0c14d560 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f89fc00c520 tx=0x7f89fc00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:36.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.335+0000 7f89f37fe700 1 -- 172.21.15.16:0/864593078 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89fc0052d0 con 0x7f8a0c0cc8a0 2023-10-16T12:10:36.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.335+0000 7f89f37fe700 1 -- 172.21.15.16:0/864593078 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89fc005430 con 0x7f8a0c0cc8a0 2023-10-16T12:10:36.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.335+0000 7f8a110e6700 1 -- 172.21.15.16:0/864593078 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a0c15be70 con 0x7f8a0c0cc8a0 2023-10-16T12:10:36.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.335+0000 7f89f37fe700 1 -- 172.21.15.16:0/864593078 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89fc003880 con 0x7f8a0c0cc8a0 2023-10-16T12:10:36.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.335+0000 7f8a110e6700 1 -- 172.21.15.16:0/864593078 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a0c15c400 con 0x7f8a0c0cc8a0 2023-10-16T12:10:36.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.336+0000 7f8a110e6700 1 -- 172.21.15.16:0/864593078 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a0c15ce70 con 0x7f8a0c0cc8a0 2023-10-16T12:10:36.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.336+0000 7f89f37fe700 1 -- 172.21.15.16:0/864593078 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f89fc0039e0 con 0x7f8a0c0cc8a0 2023-10-16T12:10:36.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.336+0000 7f89f37fe700 1 --2- 172.21.15.16:0/864593078 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f89f806af00 0x7f89f806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:36.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.337+0000 7f89f37fe700 1 -- 172.21.15.16:0/864593078 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f89fc0886a0 con 0x7f8a0c0cc8a0 2023-10-16T12:10:36.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.337+0000 7f8a0ad9d700 1 --2- 172.21.15.16:0/864593078 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f89f806af00 0x7f89f806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:36.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.337+0000 7f8a0ad9d700 1 --2- 172.21.15.16:0/864593078 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f89f806af00 0x7f89f806d3c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8a00009fd0 tx=0x7f8a0001c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:36.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.341+0000 7f89f37fe700 1 -- 172.21.15.16:0/864593078 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8a0c15ce70 con 0x7f8a0c0cc8a0 2023-10-16T12:10:36.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.546+0000 7f8a110e6700 1 -- 172.21.15.16:0/864593078 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f8a0c15ce70 con 0x7f8a0c0cc8a0 2023-10-16T12:10:36.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.546+0000 7f89f37fe700 1 -- 172.21.15.16:0/864593078 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v17) v1 ==== 84+0+65 (secure 0 0 0) 0x7f89fc017060 con 0x7f8a0c0cc8a0 2023-10-16T12:10:36.550 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:10:36.550 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2023-10-16T12:10:36.550 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:10:36.550 INFO:teuthology.orchestra.run.smithi016.stdout: "poolnum": 1, 2023-10-16T12:10:36.551 INFO:teuthology.orchestra.run.smithi016.stdout: "poolname": ".mgr" 2023-10-16T12:10:36.551 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:10:36.551 INFO:teuthology.orchestra.run.smithi016.stdout:] 2023-10-16T12:10:36.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.549+0000 7f8a110e6700 1 -- 172.21.15.16:0/864593078 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f89f806af00 msgr2=0x7f89f806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:36.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.549+0000 7f8a110e6700 1 --2- 172.21.15.16:0/864593078 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f89f806af00 0x7f89f806d3c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8a00009fd0 tx=0x7f8a0001c040 comp rx=0 tx=0).stop 2023-10-16T12:10:36.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.549+0000 7f8a110e6700 1 -- 172.21.15.16:0/864593078 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a0c0cc8a0 msgr2=0x7f8a0c14d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:36.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.549+0000 7f8a110e6700 1 --2- 172.21.15.16:0/864593078 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a0c0cc8a0 0x7f8a0c14d560 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f89fc00c520 tx=0x7f89fc00c8e0 comp rx=0 tx=0).stop 2023-10-16T12:10:36.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.549+0000 7f8a110e6700 1 -- 172.21.15.16:0/864593078 shutdown_connections 2023-10-16T12:10:36.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.549+0000 7f8a110e6700 1 --2- 172.21.15.16:0/864593078 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f89f806af00 0x7f89f806d3c0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:36.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.549+0000 7f8a110e6700 1 --2- 172.21.15.16:0/864593078 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8a0c15b280 0x7f8a0c15b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:36.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.549+0000 7f8a110e6700 1 --2- 172.21.15.16:0/864593078 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8a0c06a7e0 0x7f8a0c14a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:36.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.549+0000 7f8a110e6700 1 --2- 172.21.15.16:0/864593078 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a0c0cc8a0 0x7f8a0c14d560 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:36.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.549+0000 7f8a110e6700 1 -- 172.21.15.16:0/864593078 >> 172.21.15.16:0/864593078 conn(0x7f8a0c0b1fd0 msgr2=0x7f8a0c0719c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:36.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.549+0000 7f8a110e6700 1 -- 172.21.15.16:0/864593078 shutdown_connections 2023-10-16T12:10:36.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.549+0000 7f8a110e6700 1 -- 172.21.15.16:0/864593078 wait complete. 2023-10-16T12:10:36.564 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-16T12:10:36.564 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-16T12:10:36.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.691+0000 7f89f2f16700 1 Processor -- start 2023-10-16T12:10:36.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.691+0000 7f89f2f16700 1 -- start start 2023-10-16T12:10:36.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.692+0000 7f89f2f16700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f89ec160e60 0x7f89ec15e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:36.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.692+0000 7f89f2f16700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f89ec06a7e0 0x7f89ec06ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:36.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.692+0000 7f89f2f16700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89ec0cdeb0 con 0x7f89ec06d980 2023-10-16T12:10:36.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.692+0000 7f89f2f16700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89ec15f6c0 con 0x7f89ec160e60 2023-10-16T12:10:36.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.692+0000 7f89f2f16700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89ec06b190 con 0x7f89ec06a7e0 2023-10-16T12:10:36.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.692+0000 7f89f0cb2700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f89ec160e60 0x7f89ec15e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:36.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.692+0000 7f89ebfff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f89ec06d980 0x7f89ec15efb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50432/0 (socket says 172.21.15.16:50432) 2023-10-16T12:10:36.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.692+0000 7f89ebfff700 1 -- 172.21.15.16:0/3830684422 learned_addr learned my addr 172.21.15.16:0/3830684422 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:10:36.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.692+0000 7f89f14b3700 1 --2- 172.21.15.16:0/3830684422 >> v2:172.21.15.49:3300/0 conn(0x7f89ec06a7e0 0x7f89ec06ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:36.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.692+0000 7f89f0cb2700 1 -- 172.21.15.16:0/3830684422 >> v2:172.21.15.49:3300/0 conn(0x7f89ec06a7e0 msgr2=0x7f89ec06ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:10:36.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.692+0000 7f89f0cb2700 1 --2- 172.21.15.16:0/3830684422 >> v2:172.21.15.49:3300/0 conn(0x7f89ec06a7e0 0x7f89ec06ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:36.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.692+0000 7f89f0cb2700 1 -- 172.21.15.16:0/3830684422 >> v1:172.21.15.16:6789/0 conn(0x7f89ec06d980 legacy=0x7f89ec15efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:36.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.692+0000 7f89f0cb2700 1 -- 172.21.15.16:0/3830684422 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89ec1613e0 con 0x7f89ec160e60 2023-10-16T12:10:36.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.692+0000 7f89f14b3700 1 --2- 172.21.15.16:0/3830684422 >> v2:172.21.15.49:3300/0 conn(0x7f89ec06a7e0 0x7f89ec06ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:10:36.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.692+0000 7f89eb7fe700 1 -- 172.21.15.16:0/3830684422 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3229447155 0 0) 0x7f89ec0cdeb0 con 0x7f89ec06d980 2023-10-16T12:10:36.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.693+0000 7f89f0cb2700 1 --2- 172.21.15.16:0/3830684422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89ec160e60 0x7f89ec15e870 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f89ec0cf5a0 tx=0x7f89e0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a1ea7b65cb0935a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:36.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.693+0000 7f89eb7fe700 1 -- 172.21.15.16:0/3830684422 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89e0011830 con 0x7f89ec160e60 2023-10-16T12:10:36.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.693+0000 7f89eb7fe700 1 -- 172.21.15.16:0/3830684422 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89e0011990 con 0x7f89ec160e60 2023-10-16T12:10:36.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.693+0000 7f89eb7fe700 1 -- 172.21.15.16:0/3830684422 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89e0011d20 con 0x7f89ec160e60 2023-10-16T12:10:36.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.693+0000 7f89f2f16700 1 -- 172.21.15.16:0/3830684422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89ec160e60 msgr2=0x7f89ec15e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:36.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.693+0000 7f89f2f16700 1 --2- 172.21.15.16:0/3830684422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89ec160e60 0x7f89ec15e870 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f89ec0cf5a0 tx=0x7f89e0009b50 comp rx=0 tx=0).stop 2023-10-16T12:10:36.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.693+0000 7f89f2f16700 1 -- 172.21.15.16:0/3830684422 shutdown_connections 2023-10-16T12:10:36.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.693+0000 7f89f2f16700 1 --2- 172.21.15.16:0/3830684422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89ec160e60 0x7f89ec15e870 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:36.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.693+0000 7f89f2f16700 1 --2- 172.21.15.16:0/3830684422 >> v2:172.21.15.49:3300/0 conn(0x7f89ec06a7e0 0x7f89ec06ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:36.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.693+0000 7f89f2f16700 1 -- 172.21.15.16:0/3830684422 >> 172.21.15.16:0/3830684422 conn(0x7f89ec06f430 msgr2=0x7f89ec06f810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:36.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.693+0000 7f89f2f16700 1 -- 172.21.15.16:0/3830684422 shutdown_connections 2023-10-16T12:10:36.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.693+0000 7f89f2f16700 1 -- 172.21.15.16:0/3830684422 wait complete. 2023-10-16T12:10:36.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.694+0000 7f89f2f16700 1 Processor -- start 2023-10-16T12:10:36.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.695+0000 7f89f2f16700 1 -- start start 2023-10-16T12:10:36.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.695+0000 7f89f2f16700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89ec06a7e0 0x7f89ec146650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:36.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.695+0000 7f89f2f16700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f89ec06d980 0x7f89ec0ae0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:36.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.695+0000 7f89f2f16700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f89ec160e60 0x7f89ec0ae5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:36.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.695+0000 7f89f2f16700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89ec0af860 con 0x7f89ec06a7e0 2023-10-16T12:10:36.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.695+0000 7f89f2f16700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89ec0aef30 con 0x7f89ec06d980 2023-10-16T12:10:36.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.695+0000 7f89f2f16700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89ec0af210 con 0x7f89ec160e60 2023-10-16T12:10:36.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.695+0000 7f89f0cb2700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89ec06a7e0 0x7f89ec146650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:36.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.696+0000 7f89f0cb2700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89ec06a7e0 0x7f89ec146650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:32930/0 (socket says 172.21.15.16:32930) 2023-10-16T12:10:36.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.696+0000 7f89f0cb2700 1 -- 172.21.15.16:0/1290951397 learned_addr learned my addr 172.21.15.16:0/1290951397 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:10:36.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.696+0000 7f89ebfff700 1 --2- 172.21.15.16:0/1290951397 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f89ec06d980 0x7f89ec0ae0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:36.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.696+0000 7f89f0cb2700 1 -- 172.21.15.16:0/1290951397 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f89ec160e60 msgr2=0x7f89ec0ae5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:36.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.696+0000 7f89f14b3700 1 --2- 172.21.15.16:0/1290951397 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f89ec160e60 0x7f89ec0ae5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:36.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.696+0000 7f89f0cb2700 1 --2- 172.21.15.16:0/1290951397 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f89ec160e60 0x7f89ec0ae5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:36.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.696+0000 7f89f0cb2700 1 -- 172.21.15.16:0/1290951397 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f89ec06d980 msgr2=0x7f89ec0ae0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:36.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.696+0000 7f89f0cb2700 1 --2- 172.21.15.16:0/1290951397 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f89ec06d980 0x7f89ec0ae0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:36.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.696+0000 7f89f0cb2700 1 -- 172.21.15.16:0/1290951397 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89e00094a0 con 0x7f89ec06a7e0 2023-10-16T12:10:36.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.696+0000 7f89ebfff700 1 --2- 172.21.15.16:0/1290951397 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f89ec06d980 0x7f89ec0ae0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:10:36.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.696+0000 7f89f0cb2700 1 --2- 172.21.15.16:0/1290951397 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89ec06a7e0 0x7f89ec146650 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f89e0014040 tx=0x7f89e0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:36.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.696+0000 7f89e97fa700 1 -- 172.21.15.16:0/1290951397 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89e00038a0 con 0x7f89ec06a7e0 2023-10-16T12:10:36.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.696+0000 7f89e97fa700 1 -- 172.21.15.16:0/1290951397 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89e001e070 con 0x7f89ec06a7e0 2023-10-16T12:10:36.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.696+0000 7f89f2f16700 1 -- 172.21.15.16:0/1290951397 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89ec0af4f0 con 0x7f89ec06a7e0 2023-10-16T12:10:36.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.696+0000 7f89e97fa700 1 -- 172.21.15.16:0/1290951397 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89e0003c40 con 0x7f89ec06a7e0 2023-10-16T12:10:36.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.696+0000 7f89f2f16700 1 -- 172.21.15.16:0/1290951397 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89ec155ca0 con 0x7f89ec06a7e0 2023-10-16T12:10:36.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.697+0000 7f89f2f16700 1 -- 172.21.15.16:0/1290951397 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89ec04ec20 con 0x7f89ec06a7e0 2023-10-16T12:10:36.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.698+0000 7f89e97fa700 1 -- 172.21.15.16:0/1290951397 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f89e002d450 con 0x7f89ec06a7e0 2023-10-16T12:10:36.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.698+0000 7f89e97fa700 1 --2- 172.21.15.16:0/1290951397 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f89d006af00 0x7f89d006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:36.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.698+0000 7f89ebfff700 1 --2- 172.21.15.16:0/1290951397 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f89d006af00 0x7f89d006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:36.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.698+0000 7f89e97fa700 1 -- 172.21.15.16:0/1290951397 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f89e008a360 con 0x7f89ec06a7e0 2023-10-16T12:10:36.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.698+0000 7f89ebfff700 1 --2- 172.21.15.16:0/1290951397 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f89d006af00 0x7f89d006d3c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f89ec071f30 tx=0x7f89d8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:36.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.702+0000 7f89e97fa700 1 -- 172.21.15.16:0/1290951397 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f89e00557f0 con 0x7f89ec06a7e0 2023-10-16T12:10:36.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:36.904+0000 7f89f2f16700 1 -- 172.21.15.16:0/1290951397 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f89ec15ea20 con 0x7f89ec06a7e0 2023-10-16T12:10:37.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:37.595+0000 7f89e97fa700 1 -- 172.21.15.16:0/1290951397 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v18) v1 ==== 138+0+0 (secure 0 0 0) 0x7f89e0058e10 con 0x7f89ec06a7e0 2023-10-16T12:10:37.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:37.599+0000 7f89f2f16700 1 -- 172.21.15.16:0/1290951397 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f89d006af00 msgr2=0x7f89d006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:37.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:37.599+0000 7f89f2f16700 1 --2- 172.21.15.16:0/1290951397 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f89d006af00 0x7f89d006d3c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f89ec071f30 tx=0x7f89d8009040 comp rx=0 tx=0).stop 2023-10-16T12:10:37.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:37.599+0000 7f89f2f16700 1 -- 172.21.15.16:0/1290951397 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89ec06a7e0 msgr2=0x7f89ec146650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:37.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:37.599+0000 7f89f2f16700 1 --2- 172.21.15.16:0/1290951397 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89ec06a7e0 0x7f89ec146650 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f89e0014040 tx=0x7f89e0004750 comp rx=0 tx=0).stop 2023-10-16T12:10:37.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:37.599+0000 7f89f2f16700 1 -- 172.21.15.16:0/1290951397 shutdown_connections 2023-10-16T12:10:37.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:37.599+0000 7f89f2f16700 1 --2- 172.21.15.16:0/1290951397 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f89d006af00 0x7f89d006d3c0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:37.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:37.599+0000 7f89f2f16700 1 --2- 172.21.15.16:0/1290951397 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f89ec160e60 0x7f89ec0ae5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:37.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:37.599+0000 7f89f2f16700 1 --2- 172.21.15.16:0/1290951397 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89ec06a7e0 0x7f89ec146650 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:37.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:37.599+0000 7f89f2f16700 1 --2- 172.21.15.16:0/1290951397 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f89ec06d980 0x7f89ec0ae0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:37.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:37.599+0000 7f89f2f16700 1 -- 172.21.15.16:0/1290951397 >> 172.21.15.16:0/1290951397 conn(0x7f89ec06f430 msgr2=0x7f89ec15e1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:37.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:37.602+0000 7f89f2f16700 1 -- 172.21.15.16:0/1290951397 shutdown_connections 2023-10-16T12:10:37.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:37.602+0000 7f89f2f16700 1 -- 172.21.15.16:0/1290951397 wait complete. 2023-10-16T12:10:37.606 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' created 2023-10-16T12:10:37.621 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2023-10-16T12:10:37.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:37.756+0000 7feb31529700 1 Processor -- start 2023-10-16T12:10:37.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:37.756+0000 7feb31529700 1 -- start start 2023-10-16T12:10:37.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:37.757+0000 7feb31529700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7feb2c15ec80 0x7feb2c0cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:37.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:37.757+0000 7feb31529700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb2c0b2440 con 0x7feb2c0d2110 2023-10-16T12:10:37.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:37.757+0000 7feb31529700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb2c0d00e0 con 0x7feb2c15f060 2023-10-16T12:10:37.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:37.757+0000 7feb31529700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb2c0b2c50 con 0x7feb2c15ec80 2023-10-16T12:10:37.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:37.757+0000 7feb2b7fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7feb2c0d2110 0x7feb2c0b4b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50446/0 (socket says 172.21.15.16:50446) 2023-10-16T12:10:37.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:37.757+0000 7feb2b7fe700 1 -- 172.21.15.16:0/3089084928 learned_addr learned my addr 172.21.15.16:0/3089084928 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:10:37.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:37.757+0000 7feb2affd700 1 --2- 172.21.15.16:0/3089084928 >> v2:172.21.15.49:3300/0 conn(0x7feb2c15ec80 0x7feb2c0cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:37.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:37.757+0000 7feb29ffb700 1 -- 172.21.15.16:0/3089084928 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3935860065 0 0) 0x7feb2c0b2440 con 0x7feb2c0d2110 2023-10-16T12:10:37.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:37.757+0000 7feb29ffb700 1 -- 172.21.15.16:0/3089084928 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb140035b0 con 0x7feb2c0d2110 2023-10-16T12:10:37.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:37.758+0000 7feb2affd700 1 -- 172.21.15.16:0/3089084928 >> v1:172.21.15.49:6789/0 conn(0x7feb2c15f060 legacy=0x7feb2c0b4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:37.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:37.758+0000 7feb2affd700 1 -- 172.21.15.16:0/3089084928 >> v1:172.21.15.16:6789/0 conn(0x7feb2c0d2110 legacy=0x7feb2c0b4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:37.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:37.758+0000 7feb2affd700 1 -- 172.21.15.16:0/3089084928 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb2c0cef60 con 0x7feb2c15ec80 2023-10-16T12:10:37.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:37.758+0000 7feb2affd700 1 --2- 172.21.15.16:0/3089084928 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feb2c15ec80 0x7feb2c0cdec0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7feb20009840 tx=0x7feb20009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d7db5f81a74c8f7c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:37.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:37.758+0000 7feb29ffb700 1 -- 172.21.15.16:0/3089084928 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb20011830 con 0x7feb2c15ec80 2023-10-16T12:10:37.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:37.758+0000 7feb29ffb700 1 -- 172.21.15.16:0/3089084928 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb20011990 con 0x7feb2c15ec80 2023-10-16T12:10:37.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:37.758+0000 7feb29ffb700 1 -- 172.21.15.16:0/3089084928 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb20011d20 con 0x7feb2c15ec80 2023-10-16T12:10:37.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:37.758+0000 7feb31529700 1 -- 172.21.15.16:0/3089084928 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feb2c15ec80 msgr2=0x7feb2c0cdec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:37.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:37.758+0000 7feb31529700 1 --2- 172.21.15.16:0/3089084928 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feb2c15ec80 0x7feb2c0cdec0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7feb20009840 tx=0x7feb20009b50 comp rx=0 tx=0).stop 2023-10-16T12:10:37.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:37.758+0000 7feb31529700 1 -- 172.21.15.16:0/3089084928 shutdown_connections 2023-10-16T12:10:37.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:37.758+0000 7feb31529700 1 --2- 172.21.15.16:0/3089084928 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feb2c15ec80 0x7feb2c0cdec0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:37.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:37.758+0000 7feb31529700 1 -- 172.21.15.16:0/3089084928 >> 172.21.15.16:0/3089084928 conn(0x7feb2c0d15f0 msgr2=0x7feb2c0d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:37.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:37.759+0000 7feb31529700 1 -- 172.21.15.16:0/3089084928 shutdown_connections 2023-10-16T12:10:37.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:37.759+0000 7feb31529700 1 -- 172.21.15.16:0/3089084928 wait complete. 2023-10-16T12:10:37.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:37.760+0000 7feb31529700 1 Processor -- start 2023-10-16T12:10:37.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:37.760+0000 7feb31529700 1 -- start start 2023-10-16T12:10:37.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:37.761+0000 7feb31529700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7feb2c0d2110 0x7feb2c0cafa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:37.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:37.761+0000 7feb31529700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb2c15ec80 0x7feb2c1e4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:37.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:37.761+0000 7feb31529700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feb2c15f060 0x7feb2c0b5a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:37.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:37.761+0000 7feb31529700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb2c0cb4e0 con 0x7feb2c15ec80 2023-10-16T12:10:37.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:37.761+0000 7feb31529700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb2c0c86c0 con 0x7feb2c15f060 2023-10-16T12:10:37.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:37.761+0000 7feb31529700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb2c1e5400 con 0x7feb2c0d2110 2023-10-16T12:10:37.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:37.761+0000 7feb2a7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb2c15ec80 0x7feb2c1e4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:37.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:37.761+0000 7feb2a7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb2c15ec80 0x7feb2c1e4bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:32944/0 (socket says 172.21.15.16:32944) 2023-10-16T12:10:37.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:37.761+0000 7feb2a7fc700 1 -- 172.21.15.16:0/2619019237 learned_addr learned my addr 172.21.15.16:0/2619019237 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:10:37.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:37.761+0000 7feb2b7fe700 1 --2- 172.21.15.16:0/2619019237 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feb2c15f060 0x7feb2c0b5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:37.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:37.761+0000 7feb2affd700 1 --2- 172.21.15.16:0/2619019237 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7feb2c0d2110 0x7feb2c0cafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:37.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:37.761+0000 7feb2a7fc700 1 -- 172.21.15.16:0/2619019237 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7feb2c0d2110 msgr2=0x7feb2c0cafa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:37.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:37.761+0000 7feb2a7fc700 1 --2- 172.21.15.16:0/2619019237 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7feb2c0d2110 0x7feb2c0cafa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:37.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:37.761+0000 7feb2a7fc700 1 -- 172.21.15.16:0/2619019237 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feb2c15f060 msgr2=0x7feb2c0b5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:37.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:37.761+0000 7feb2a7fc700 1 --2- 172.21.15.16:0/2619019237 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feb2c15f060 0x7feb2c0b5a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:37.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:37.761+0000 7feb2a7fc700 1 -- 172.21.15.16:0/2619019237 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb200094a0 con 0x7feb2c15ec80 2023-10-16T12:10:37.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:37.761+0000 7feb2b7fe700 1 --2- 172.21.15.16:0/2619019237 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feb2c15f060 0x7feb2c0b5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:10:37.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:37.761+0000 7feb2a7fc700 1 --2- 172.21.15.16:0/2619019237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb2c15ec80 0x7feb2c1e4bb0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7feb1c00c420 tx=0x7feb1c00c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:37.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:37.761+0000 7feb137fe700 1 -- 172.21.15.16:0/2619019237 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb1c0052d0 con 0x7feb2c15ec80 2023-10-16T12:10:37.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:37.761+0000 7feb137fe700 1 -- 172.21.15.16:0/2619019237 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb1c005430 con 0x7feb2c15ec80 2023-10-16T12:10:37.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:37.761+0000 7feb31529700 1 -- 172.21.15.16:0/2619019237 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb2c06b8f0 con 0x7feb2c15ec80 2023-10-16T12:10:37.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:37.761+0000 7feb137fe700 1 -- 172.21.15.16:0/2619019237 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb1c0038c0 con 0x7feb2c15ec80 2023-10-16T12:10:37.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:37.761+0000 7feb31529700 1 -- 172.21.15.16:0/2619019237 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb2c06beb0 con 0x7feb2c15ec80 2023-10-16T12:10:37.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:37.762+0000 7feb31529700 1 -- 172.21.15.16:0/2619019237 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb2c04ebb0 con 0x7feb2c15ec80 2023-10-16T12:10:37.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:37.763+0000 7feb137fe700 1 -- 172.21.15.16:0/2619019237 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7feb1c003a20 con 0x7feb2c15ec80 2023-10-16T12:10:37.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:37.763+0000 7feb137fe700 1 --2- 172.21.15.16:0/2619019237 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7feb1806af00 0x7feb1806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:37.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:37.763+0000 7feb137fe700 1 -- 172.21.15.16:0/2619019237 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5227+0+0 (secure 0 0 0) 0x7feb1c088820 con 0x7feb2c15ec80 2023-10-16T12:10:37.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:37.763+0000 7feb2affd700 1 --2- 172.21.15.16:0/2619019237 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7feb1806af00 0x7feb1806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:37.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:37.764+0000 7feb2affd700 1 --2- 172.21.15.16:0/2619019237 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7feb1806af00 0x7feb1806d3c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7feb200090b0 tx=0x7feb20008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:37.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:37.768+0000 7feb137fe700 1 -- 172.21.15.16:0/2619019237 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7feb1c053ab0 con 0x7feb2c15ec80 2023-10-16T12:10:37.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:37.973+0000 7feb31529700 1 -- 172.21.15.16:0/2619019237 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) v1 -- 0x7feb2c0cbab0 con 0x7feb2c15ec80 2023-10-16T12:10:38.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:38.598+0000 7feb137fe700 1 -- 172.21.15.16:0/2619019237 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v19) v1 ==== 171+0+0 (secure 0 0 0) 0x7feb1c0170d0 con 0x7feb2c15ec80 2023-10-16T12:10:38.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:38.601+0000 7feb31529700 1 -- 172.21.15.16:0/2619019237 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7feb1806af00 msgr2=0x7feb1806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:38.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:38.601+0000 7feb31529700 1 --2- 172.21.15.16:0/2619019237 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7feb1806af00 0x7feb1806d3c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7feb200090b0 tx=0x7feb20008030 comp rx=0 tx=0).stop 2023-10-16T12:10:38.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:38.601+0000 7feb31529700 1 -- 172.21.15.16:0/2619019237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb2c15ec80 msgr2=0x7feb2c1e4bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:38.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:38.601+0000 7feb31529700 1 --2- 172.21.15.16:0/2619019237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb2c15ec80 0x7feb2c1e4bb0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7feb1c00c420 tx=0x7feb1c00c7e0 comp rx=0 tx=0).stop 2023-10-16T12:10:38.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:38.601+0000 7feb31529700 1 -- 172.21.15.16:0/2619019237 shutdown_connections 2023-10-16T12:10:38.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:38.601+0000 7feb31529700 1 --2- 172.21.15.16:0/2619019237 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7feb1806af00 0x7feb1806d3c0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:38.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:38.601+0000 7feb31529700 1 --2- 172.21.15.16:0/2619019237 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feb2c15f060 0x7feb2c0b5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:38.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:38.601+0000 7feb31529700 1 --2- 172.21.15.16:0/2619019237 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7feb2c0d2110 0x7feb2c0cafa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:38.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:38.602+0000 7feb31529700 1 --2- 172.21.15.16:0/2619019237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb2c15ec80 0x7feb2c1e4bb0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:38.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:38.602+0000 7feb31529700 1 -- 172.21.15.16:0/2619019237 >> 172.21.15.16:0/2619019237 conn(0x7feb2c0d15f0 msgr2=0x7feb2c1538c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:38.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:38.602+0000 7feb31529700 1 -- 172.21.15.16:0/2619019237 shutdown_connections 2023-10-16T12:10:38.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:38.603+0000 7feb31529700 1 -- 172.21.15.16:0/2619019237 wait complete. 2023-10-16T12:10:38.613 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' created 2023-10-16T12:10:38.633 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-16T12:10:38.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:38.763+0000 7f2e3bd13700 1 Processor -- start 2023-10-16T12:10:38.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:38.763+0000 7f2e3bd13700 1 -- start start 2023-10-16T12:10:38.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:38.764+0000 7f2e3bd13700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2e3406e570 0x7f2e3415e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:38.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:38.764+0000 7f2e3bd13700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2e3406a7e0 0x7f2e3406ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:38.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:38.764+0000 7f2e3bd13700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e3415b6a0 con 0x7f2e3415b2c0 2023-10-16T12:10:38.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:38.764+0000 7f2e3bd13700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e3406b1b0 con 0x7f2e3406e570 2023-10-16T12:10:38.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:38.764+0000 7f2e3bd13700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e34161b20 con 0x7f2e3406a7e0 2023-10-16T12:10:38.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:38.764+0000 7f2e392ae700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2e3406e570 0x7f2e3415e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:38.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:38.764+0000 7f2e392ae700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2e3406e570 0x7f2e3415e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:32952/0 (socket says 172.21.15.16:32952) 2023-10-16T12:10:38.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:38.764+0000 7f2e392ae700 1 -- 172.21.15.16:0/2375109292 learned_addr learned my addr 172.21.15.16:0/2375109292 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:10:38.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:38.764+0000 7f2e3a2b0700 1 --2- 172.21.15.16:0/2375109292 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2e3406a7e0 0x7f2e3406ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:38.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:38.764+0000 7f2e392ae700 1 -- 172.21.15.16:0/2375109292 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2e3406a7e0 msgr2=0x7f2e3406ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:10:38.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:38.764+0000 7f2e392ae700 1 --2- 172.21.15.16:0/2375109292 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2e3406a7e0 0x7f2e3406ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:38.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:38.764+0000 7f2e392ae700 1 -- 172.21.15.16:0/2375109292 >> v1:172.21.15.49:6789/0 conn(0x7f2e3415b2c0 legacy=0x7f2e34161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:38.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:38.764+0000 7f2e392ae700 1 -- 172.21.15.16:0/2375109292 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e34161e00 con 0x7f2e3406e570 2023-10-16T12:10:38.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:38.765+0000 7f2e392ae700 1 --2- 172.21.15.16:0/2375109292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e3406e570 0x7f2e3415e5a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2e300098f0 tx=0x7f2e30009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cd209288f686922 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:38.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:38.765+0000 7f2e38aad700 1 -- 172.21.15.16:0/2375109292 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e3000b8a0 con 0x7f2e3406e570 2023-10-16T12:10:38.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:38.765+0000 7f2e38aad700 1 -- 172.21.15.16:0/2375109292 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e3000ba00 con 0x7f2e3406e570 2023-10-16T12:10:38.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:38.765+0000 7f2e38aad700 1 -- 172.21.15.16:0/2375109292 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e3000bd90 con 0x7f2e3406e570 2023-10-16T12:10:38.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:38.765+0000 7f2e3bd13700 1 -- 172.21.15.16:0/2375109292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e3406e570 msgr2=0x7f2e3415e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:38.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:38.765+0000 7f2e3bd13700 1 --2- 172.21.15.16:0/2375109292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e3406e570 0x7f2e3415e5a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2e300098f0 tx=0x7f2e30009c00 comp rx=0 tx=0).stop 2023-10-16T12:10:38.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:38.765+0000 7f2e3bd13700 1 -- 172.21.15.16:0/2375109292 shutdown_connections 2023-10-16T12:10:38.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:38.765+0000 7f2e3bd13700 1 --2- 172.21.15.16:0/2375109292 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2e3406a7e0 0x7f2e3406ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:38.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:38.765+0000 7f2e3bd13700 1 --2- 172.21.15.16:0/2375109292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e3406e570 0x7f2e3415e5a0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:38.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:38.765+0000 7f2e3bd13700 1 -- 172.21.15.16:0/2375109292 >> 172.21.15.16:0/2375109292 conn(0x7f2e340b1f40 msgr2=0x7f2e340b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:38.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:38.766+0000 7f2e3bd13700 1 -- 172.21.15.16:0/2375109292 shutdown_connections 2023-10-16T12:10:38.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:38.766+0000 7f2e3bd13700 1 -- 172.21.15.16:0/2375109292 wait complete. 2023-10-16T12:10:38.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:38.767+0000 7f2e3bd13700 1 Processor -- start 2023-10-16T12:10:38.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:38.767+0000 7f2e3bd13700 1 -- start start 2023-10-16T12:10:38.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:38.767+0000 7f2e3bd13700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2e3406a7e0 0x7f2e34158880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:38.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:38.767+0000 7f2e3bd13700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e3406e570 0x7f2e3415b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:38.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:38.767+0000 7f2e3bd13700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2e3415b2c0 0x7f2e3415be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:38.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:38.767+0000 7f2e3bd13700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e34158dc0 con 0x7f2e3406e570 2023-10-16T12:10:38.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:38.767+0000 7f2e3bd13700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e340b2fb0 con 0x7f2e3415b2c0 2023-10-16T12:10:38.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:38.767+0000 7f2e3bd13700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e340b3290 con 0x7f2e3406a7e0 2023-10-16T12:10:38.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:38.768+0000 7f2e392ae700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e3406e570 0x7f2e3415b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:38.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:38.768+0000 7f2e392ae700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e3406e570 0x7f2e3415b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:32958/0 (socket says 172.21.15.16:32958) 2023-10-16T12:10:38.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:38.768+0000 7f2e392ae700 1 -- 172.21.15.16:0/3260779357 learned_addr learned my addr 172.21.15.16:0/3260779357 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:10:38.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:38.768+0000 7f2e39aaf700 1 --2- 172.21.15.16:0/3260779357 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2e3406a7e0 0x7f2e34158880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:38.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:38.768+0000 7f2e392ae700 1 -- 172.21.15.16:0/3260779357 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2e3406a7e0 msgr2=0x7f2e34158880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:38.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:38.768+0000 7f2e392ae700 1 --2- 172.21.15.16:0/3260779357 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2e3406a7e0 0x7f2e34158880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:38.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:38.768+0000 7f2e392ae700 1 -- 172.21.15.16:0/3260779357 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2e3415b2c0 msgr2=0x7f2e3415be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:38.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:38.768+0000 7f2e3a2b0700 1 --2- 172.21.15.16:0/3260779357 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2e3415b2c0 0x7f2e3415be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:38.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:38.768+0000 7f2e392ae700 1 --2- 172.21.15.16:0/3260779357 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2e3415b2c0 0x7f2e3415be00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:38.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:38.768+0000 7f2e392ae700 1 -- 172.21.15.16:0/3260779357 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e30009550 con 0x7f2e3406e570 2023-10-16T12:10:38.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:38.768+0000 7f2e39aaf700 1 --2- 172.21.15.16:0/3260779357 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2e3406a7e0 0x7f2e34158880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:10:38.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:38.768+0000 7f2e392ae700 1 --2- 172.21.15.16:0/3260779357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e3406e570 0x7f2e3415b8c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2e30000c00 tx=0x7f2e30004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:38.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:38.768+0000 7f2e2a7fc700 1 -- 172.21.15.16:0/3260779357 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e30004480 con 0x7f2e3406e570 2023-10-16T12:10:38.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:38.768+0000 7f2e2a7fc700 1 -- 172.21.15.16:0/3260779357 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e3001f070 con 0x7f2e3406e570 2023-10-16T12:10:38.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:38.768+0000 7f2e3bd13700 1 -- 172.21.15.16:0/3260779357 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e340b3570 con 0x7f2e3406e570 2023-10-16T12:10:38.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:38.768+0000 7f2e2a7fc700 1 -- 172.21.15.16:0/3260779357 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e300038c0 con 0x7f2e3406e570 2023-10-16T12:10:38.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:38.768+0000 7f2e3bd13700 1 -- 172.21.15.16:0/3260779357 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e34160450 con 0x7f2e3406e570 2023-10-16T12:10:38.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:38.769+0000 7f2e3bd13700 1 -- 172.21.15.16:0/3260779357 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e340b3a90 con 0x7f2e3406e570 2023-10-16T12:10:38.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:38.770+0000 7f2e2a7fc700 1 -- 172.21.15.16:0/3260779357 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f2e30021030 con 0x7f2e3406e570 2023-10-16T12:10:38.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:38.770+0000 7f2e2a7fc700 1 --2- 172.21.15.16:0/3260779357 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f2e1806af00 0x7f2e1806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:38.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:38.770+0000 7f2e2a7fc700 1 -- 172.21.15.16:0/3260779357 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f2e3008ac00 con 0x7f2e3406e570 2023-10-16T12:10:38.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:38.770+0000 7f2e39aaf700 1 --2- 172.21.15.16:0/3260779357 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f2e1806af00 0x7f2e1806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:38.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:38.771+0000 7f2e39aaf700 1 --2- 172.21.15.16:0/3260779357 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f2e1806af00 0x7f2e1806d3c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2e24000f60 tx=0x7f2e24009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:38.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:38.774+0000 7f2e2a7fc700 1 -- 172.21.15.16:0/3260779357 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2e30055fa0 con 0x7f2e3406e570 2023-10-16T12:10:39.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.027+0000 7f2e3bd13700 1 -- 172.21.15.16:0/3260779357 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f2e340b3a90 con 0x7f2e3406e570 2023-10-16T12:10:39.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.627+0000 7f2e2a7fc700 1 -- 172.21.15.16:0/3260779357 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f2e340b3a90 con 0x7f2e3406e570 2023-10-16T12:10:39.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.634+0000 7f2e3bd13700 1 -- 172.21.15.16:0/3260779357 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f2e1806af00 msgr2=0x7f2e1806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:39.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.634+0000 7f2e3bd13700 1 --2- 172.21.15.16:0/3260779357 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f2e1806af00 0x7f2e1806d3c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2e24000f60 tx=0x7f2e24009040 comp rx=0 tx=0).stop 2023-10-16T12:10:39.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.634+0000 7f2e3bd13700 1 -- 172.21.15.16:0/3260779357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e3406e570 msgr2=0x7f2e3415b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:39.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.634+0000 7f2e3bd13700 1 --2- 172.21.15.16:0/3260779357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e3406e570 0x7f2e3415b8c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2e30000c00 tx=0x7f2e30004750 comp rx=0 tx=0).stop 2023-10-16T12:10:39.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.634+0000 7f2e3bd13700 1 -- 172.21.15.16:0/3260779357 shutdown_connections 2023-10-16T12:10:39.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.634+0000 7f2e3bd13700 1 --2- 172.21.15.16:0/3260779357 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f2e1806af00 0x7f2e1806d3c0 secure :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2e24000f60 tx=0x7f2e24009040 comp rx=0 tx=0).stop 2023-10-16T12:10:39.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.634+0000 7f2e3bd13700 1 --2- 172.21.15.16:0/3260779357 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2e3415b2c0 0x7f2e3415be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:39.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.634+0000 7f2e3bd13700 1 --2- 172.21.15.16:0/3260779357 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2e3406a7e0 0x7f2e34158880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:39.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.634+0000 7f2e3bd13700 1 --2- 172.21.15.16:0/3260779357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e3406e570 0x7f2e3415b8c0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:39.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.634+0000 7f2e3bd13700 1 -- 172.21.15.16:0/3260779357 >> 172.21.15.16:0/3260779357 conn(0x7f2e340b1f40 msgr2=0x7f2e34159a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:39.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.639+0000 7f2e3bd13700 1 -- 172.21.15.16:0/3260779357 shutdown_connections 2023-10-16T12:10:39.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.639+0000 7f2e3bd13700 1 -- 172.21.15.16:0/3260779357 wait complete. 2023-10-16T12:10:39.645 INFO:teuthology.orchestra.run.smithi016.stderr:new fs with metadata pool 2 and data pool 3 2023-10-16T12:10:39.665 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2023-10-16T12:10:39.666 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2023-10-16T12:10:39.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.797+0000 7fe613fdd700 1 Processor -- start 2023-10-16T12:10:39.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.798+0000 7fe613fdd700 1 -- start start 2023-10-16T12:10:39.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.798+0000 7fe613fdd700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe60c155e40 0x7fe60c15bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:39.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.798+0000 7fe613fdd700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fe60c15c190 0x7fe60c0cba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:39.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.798+0000 7fe613fdd700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe60c15c780 con 0x7fe60c155a60 2023-10-16T12:10:39.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.798+0000 7fe613fdd700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe60c15f240 con 0x7fe60c15c190 2023-10-16T12:10:39.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.798+0000 7fe613fdd700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe60c15ca60 con 0x7fe60c155e40 2023-10-16T12:10:39.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.799+0000 7fe611d79700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe60c155a60 0x7fe60c15fc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50470/0 (socket says 172.21.15.16:50470) 2023-10-16T12:10:39.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.799+0000 7fe611d79700 1 -- 172.21.15.16:0/2897152771 learned_addr learned my addr 172.21.15.16:0/2897152771 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:10:39.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.799+0000 7fe611578700 1 --2- 172.21.15.16:0/2897152771 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe60c155e40 0x7fe60c15bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:39.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.799+0000 7fe61257a700 1 --2- 172.21.15.16:0/2897152771 >> v2:172.21.15.49:3300/0 conn(0x7fe60c15c190 0x7fe60c0cba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:39.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.799+0000 7fe610d77700 1 -- 172.21.15.16:0/2897152771 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3213622784 0 0) 0x7fe60c15c780 con 0x7fe60c155a60 2023-10-16T12:10:39.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.799+0000 7fe610d77700 1 -- 172.21.15.16:0/2897152771 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5f80035b0 con 0x7fe60c155a60 2023-10-16T12:10:39.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.799+0000 7fe610d77700 1 -- 172.21.15.16:0/2897152771 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2873395675 0 0) 0x7fe5f80035b0 con 0x7fe60c155a60 2023-10-16T12:10:39.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.799+0000 7fe610d77700 1 -- 172.21.15.16:0/2897152771 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe60c155e40 msgr2=0x7fe60c15bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:10:39.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.799+0000 7fe610d77700 1 --2- 172.21.15.16:0/2897152771 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe60c155e40 0x7fe60c15bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:39.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.799+0000 7fe610d77700 1 -- 172.21.15.16:0/2897152771 >> v2:172.21.15.49:3300/0 conn(0x7fe60c15c190 msgr2=0x7fe60c0cba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:10:39.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.799+0000 7fe610d77700 1 --2- 172.21.15.16:0/2897152771 >> v2:172.21.15.49:3300/0 conn(0x7fe60c15c190 0x7fe60c0cba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:39.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.799+0000 7fe610d77700 1 -- 172.21.15.16:0/2897152771 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe60c15f450 con 0x7fe60c155a60 2023-10-16T12:10:39.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.799+0000 7fe610d77700 1 -- 172.21.15.16:0/2897152771 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7fe608001670 con 0x7fe60c155a60 2023-10-16T12:10:39.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.799+0000 7fe610d77700 1 -- 172.21.15.16:0/2897152771 >> v1:172.21.15.16:6789/0 conn(0x7fe60c155a60 legacy=0x7fe60c15fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:39.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.799+0000 7fe610d77700 1 --2- 172.21.15.16:0/2897152771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5f8003a00 0x7fe5f8003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:39.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.799+0000 7fe610d77700 1 --2- 172.21.15.16:0/2897152771 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe60c155e40 0x7fe5f8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:39.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.799+0000 7fe610d77700 1 --2- 172.21.15.16:0/2897152771 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe5f8004d90 0x7fe5f8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:39.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.799+0000 7fe610d77700 1 -- 172.21.15.16:0/2897152771 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe60c15f240 con 0x7fe5f8003a00 2023-10-16T12:10:39.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.799+0000 7fe610d77700 1 -- 172.21.15.16:0/2897152771 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe60c15ca60 con 0x7fe5f8004d90 2023-10-16T12:10:39.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.799+0000 7fe610d77700 1 -- 172.21.15.16:0/2897152771 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f800ba10 con 0x7fe60c155e40 2023-10-16T12:10:39.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.799+0000 7fe611d79700 1 --2- 172.21.15.16:0/2897152771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5f8003a00 0x7fe5f8003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:39.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.799+0000 7fe611578700 1 --2- 172.21.15.16:0/2897152771 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe60c155e40 0x7fe5f8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:39.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.799+0000 7fe61257a700 1 --2- 172.21.15.16:0/2897152771 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe5f8004d90 0x7fe5f8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:39.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.800+0000 7fe611d79700 1 -- 172.21.15.16:0/2897152771 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe60c155e40 msgr2=0x7fe5f8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:10:39.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.800+0000 7fe611d79700 1 --2- 172.21.15.16:0/2897152771 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe60c155e40 0x7fe5f8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:39.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.800+0000 7fe611d79700 1 -- 172.21.15.16:0/2897152771 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe5f8004d90 msgr2=0x7fe5f8005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:10:39.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.800+0000 7fe611d79700 1 --2- 172.21.15.16:0/2897152771 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe5f8004d90 0x7fe5f8005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:39.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.800+0000 7fe611d79700 1 -- 172.21.15.16:0/2897152771 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5f80035b0 con 0x7fe5f8003a00 2023-10-16T12:10:39.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.800+0000 7fe611d79700 1 --2- 172.21.15.16:0/2897152771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5f8003a00 0x7fe5f8003e40 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe60800c2f0 tx=0x7fe5f8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ec429c9e6182c41 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:39.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.800+0000 7fe610d77700 1 -- 172.21.15.16:0/2897152771 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe60800ce30 con 0x7fe5f8003a00 2023-10-16T12:10:39.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.800+0000 7fe610d77700 1 -- 172.21.15.16:0/2897152771 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe608005960 con 0x7fe5f8003a00 2023-10-16T12:10:39.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.800+0000 7fe610d77700 1 -- 172.21.15.16:0/2897152771 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe608005d10 con 0x7fe5f8003a00 2023-10-16T12:10:39.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.800+0000 7fe613fdd700 1 -- 172.21.15.16:0/2897152771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5f8003a00 msgr2=0x7fe5f8003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:39.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.800+0000 7fe613fdd700 1 --2- 172.21.15.16:0/2897152771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5f8003a00 0x7fe5f8003e40 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe60800c2f0 tx=0x7fe5f8005880 comp rx=0 tx=0).stop 2023-10-16T12:10:39.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.801+0000 7fe61257a700 1 -- 172.21.15.16:0/2897152771 reap_dead start 2023-10-16T12:10:39.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.801+0000 7fe613fdd700 1 -- 172.21.15.16:0/2897152771 shutdown_connections 2023-10-16T12:10:39.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.801+0000 7fe613fdd700 1 -- 172.21.15.16:0/2897152771 >> 172.21.15.16:0/2897152771 conn(0x7fe60c06a7e0 msgr2=0x7fe60c06abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:39.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.801+0000 7fe613fdd700 1 -- 172.21.15.16:0/2897152771 shutdown_connections 2023-10-16T12:10:39.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.801+0000 7fe613fdd700 1 -- 172.21.15.16:0/2897152771 wait complete. 2023-10-16T12:10:39.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.802+0000 7fe613fdd700 1 Processor -- start 2023-10-16T12:10:39.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.802+0000 7fe613fdd700 1 -- start start 2023-10-16T12:10:39.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.803+0000 7fe613fdd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe5f8003a00 0x7fe60c156be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:39.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.803+0000 7fe613fdd700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe60c0cf6f0 0x7fe60c1efcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:39.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.803+0000 7fe613fdd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe60c0cfb70 0x7fe60c1f01f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:39.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.803+0000 7fe613fdd700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe60c1f0730 con 0x7fe60c0cfb70 2023-10-16T12:10:39.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.803+0000 7fe613fdd700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe60c15b990 con 0x7fe5f8003a00 2023-10-16T12:10:39.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.803+0000 7fe613fdd700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe60c15ad80 con 0x7fe60c0cf6f0 2023-10-16T12:10:39.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.803+0000 7fe61257a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe60c0cfb70 0x7fe60c1f01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:39.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.803+0000 7fe61257a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe60c0cfb70 0x7fe60c1f01f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:32980/0 (socket says 172.21.15.16:32980) 2023-10-16T12:10:39.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.803+0000 7fe61257a700 1 -- 172.21.15.16:0/1434583021 learned_addr learned my addr 172.21.15.16:0/1434583021 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:10:39.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.804+0000 7fe611578700 1 --2- 172.21.15.16:0/1434583021 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe60c0cf6f0 0x7fe60c1efcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:39.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.804+0000 7fe611d79700 1 --2- 172.21.15.16:0/1434583021 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe5f8003a00 0x7fe60c156be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:39.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.804+0000 7fe61257a700 1 -- 172.21.15.16:0/1434583021 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe60c0cf6f0 msgr2=0x7fe60c1efcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:39.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.804+0000 7fe61257a700 1 --2- 172.21.15.16:0/1434583021 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe60c0cf6f0 0x7fe60c1efcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:39.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.804+0000 7fe61257a700 1 -- 172.21.15.16:0/1434583021 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe5f8003a00 msgr2=0x7fe60c156be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:39.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.804+0000 7fe61257a700 1 --2- 172.21.15.16:0/1434583021 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe5f8003a00 0x7fe60c156be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:39.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.804+0000 7fe61257a700 1 -- 172.21.15.16:0/1434583021 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe60c15b060 con 0x7fe60c0cfb70 2023-10-16T12:10:39.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.804+0000 7fe611d79700 1 --2- 172.21.15.16:0/1434583021 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe5f8003a00 0x7fe60c156be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:10:39.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.804+0000 7fe61257a700 1 --2- 172.21.15.16:0/1434583021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe60c0cfb70 0x7fe60c1f01f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe60c0d13b0 tx=0x7fe604011f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:39.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.804+0000 7fe5fe7fc700 1 -- 172.21.15.16:0/1434583021 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe60400e040 con 0x7fe60c0cfb70 2023-10-16T12:10:39.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.804+0000 7fe5fe7fc700 1 -- 172.21.15.16:0/1434583021 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe60400f040 con 0x7fe60c0cfb70 2023-10-16T12:10:39.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.804+0000 7fe613fdd700 1 -- 172.21.15.16:0/1434583021 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe60c15b340 con 0x7fe60c0cfb70 2023-10-16T12:10:39.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.804+0000 7fe5fe7fc700 1 -- 172.21.15.16:0/1434583021 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe60400fdb0 con 0x7fe60c0cfb70 2023-10-16T12:10:39.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.804+0000 7fe613fdd700 1 -- 172.21.15.16:0/1434583021 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe60c15aa30 con 0x7fe60c0cfb70 2023-10-16T12:10:39.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.805+0000 7fe613fdd700 1 -- 172.21.15.16:0/1434583021 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe60c04ec20 con 0x7fe60c0cfb70 2023-10-16T12:10:39.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.806+0000 7fe5fe7fc700 1 -- 172.21.15.16:0/1434583021 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fe604022460 con 0x7fe60c0cfb70 2023-10-16T12:10:39.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.806+0000 7fe5fe7fc700 1 --2- 172.21.15.16:0/1434583021 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fe5f006af00 0x7fe5f006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:39.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.806+0000 7fe5fe7fc700 1 -- 172.21.15.16:0/1434583021 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe604089b00 con 0x7fe60c0cfb70 2023-10-16T12:10:39.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.806+0000 7fe611d79700 1 --2- 172.21.15.16:0/1434583021 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fe5f006af00 0x7fe5f006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:39.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.806+0000 7fe611d79700 1 --2- 172.21.15.16:0/1434583021 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fe5f006af00 0x7fe5f006d3c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe608001930 tx=0x7fe60800a000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:39.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:39.811+0000 7fe5fe7fc700 1 -- 172.21.15.16:0/1434583021 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe604054bb0 con 0x7fe60c0cfb70 2023-10-16T12:10:40.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.024+0000 7fe613fdd700 1 -- 172.21.15.16:0/1434583021 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) v1 -- 0x7fe60c15d7c0 con 0x7fe60c0cfb70 2023-10-16T12:10:40.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.615+0000 7fe5fe7fc700 1 -- 172.21.15.16:0/1434583021 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v21) v1 ==== 157+0+0 (secure 0 0 0) 0x7fe6040581d0 con 0x7fe60c0cfb70 2023-10-16T12:10:40.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.619+0000 7fe613fdd700 1 -- 172.21.15.16:0/1434583021 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fe5f006af00 msgr2=0x7fe5f006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:40.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.619+0000 7fe613fdd700 1 --2- 172.21.15.16:0/1434583021 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fe5f006af00 0x7fe5f006d3c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe608001930 tx=0x7fe60800a000 comp rx=0 tx=0).stop 2023-10-16T12:10:40.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.619+0000 7fe613fdd700 1 -- 172.21.15.16:0/1434583021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe60c0cfb70 msgr2=0x7fe60c1f01f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:40.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.619+0000 7fe613fdd700 1 --2- 172.21.15.16:0/1434583021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe60c0cfb70 0x7fe60c1f01f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe60c0d13b0 tx=0x7fe604011f80 comp rx=0 tx=0).stop 2023-10-16T12:10:40.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.619+0000 7fe613fdd700 1 -- 172.21.15.16:0/1434583021 shutdown_connections 2023-10-16T12:10:40.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.619+0000 7fe613fdd700 1 --2- 172.21.15.16:0/1434583021 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fe5f006af00 0x7fe5f006d3c0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:40.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.619+0000 7fe613fdd700 1 --2- 172.21.15.16:0/1434583021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe60c0cfb70 0x7fe60c1f01f0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:40.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.619+0000 7fe613fdd700 1 --2- 172.21.15.16:0/1434583021 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe5f8003a00 0x7fe60c156be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:40.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.619+0000 7fe613fdd700 1 --2- 172.21.15.16:0/1434583021 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe60c0cf6f0 0x7fe60c1efcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:40.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.619+0000 7fe613fdd700 1 -- 172.21.15.16:0/1434583021 >> 172.21.15.16:0/1434583021 conn(0x7fe60c06a7e0 msgr2=0x7fe60c15e710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:40.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.619+0000 7fe613fdd700 1 -- 172.21.15.16:0/1434583021 shutdown_connections 2023-10-16T12:10:40.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.619+0000 7fe613fdd700 1 -- 172.21.15.16:0/1434583021 wait complete. 2023-10-16T12:10:40.638 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2023-10-16T12:10:40.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.775+0000 7fa811006700 1 Processor -- start 2023-10-16T12:10:40.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.775+0000 7fa811006700 1 -- start start 2023-10-16T12:10:40.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.776+0000 7fa811006700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa80c0cbb50 0x7fa80c0cbf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:40.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.776+0000 7fa811006700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fa80c0b4330 0x7fa80c15f110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:40.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.776+0000 7fa811006700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa80c0cc470 con 0x7fa80c15f650 2023-10-16T12:10:40.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.776+0000 7fa811006700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa80c160100 con 0x7fa80c0b4330 2023-10-16T12:10:40.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.776+0000 7fa811006700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa80c15fb40 con 0x7fa80c0cbb50 2023-10-16T12:10:40.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.776+0000 7fa80b59e700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa80c15f650 0x7fa80c0cfa70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50494/0 (socket says 172.21.15.16:50494) 2023-10-16T12:10:40.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.776+0000 7fa80b59e700 1 -- 172.21.15.16:0/3682478272 learned_addr learned my addr 172.21.15.16:0/3682478272 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:10:40.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.776+0000 7fa80a59c700 1 --2- 172.21.15.16:0/3682478272 >> v2:172.21.15.49:3300/0 conn(0x7fa80c0b4330 0x7fa80c15f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:40.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.776+0000 7fa80ad9d700 1 --2- 172.21.15.16:0/3682478272 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa80c0cbb50 0x7fa80c0cbf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:40.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.776+0000 7fa809d9b700 1 -- 172.21.15.16:0/3682478272 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1008852096 0 0) 0x7fa80c0cc470 con 0x7fa80c15f650 2023-10-16T12:10:40.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.776+0000 7fa809d9b700 1 -- 172.21.15.16:0/3682478272 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7f40035b0 con 0x7fa80c15f650 2023-10-16T12:10:40.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.776+0000 7fa809d9b700 1 -- 172.21.15.16:0/3682478272 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 533491040 0 0) 0x7fa7f40035b0 con 0x7fa80c15f650 2023-10-16T12:10:40.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.776+0000 7fa809d9b700 1 -- 172.21.15.16:0/3682478272 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa80c0cbb50 msgr2=0x7fa80c0cbf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:10:40.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.776+0000 7fa809d9b700 1 --2- 172.21.15.16:0/3682478272 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa80c0cbb50 0x7fa80c0cbf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:40.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.776+0000 7fa809d9b700 1 -- 172.21.15.16:0/3682478272 >> v2:172.21.15.49:3300/0 conn(0x7fa80c0b4330 msgr2=0x7fa80c15f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:10:40.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.776+0000 7fa809d9b700 1 --2- 172.21.15.16:0/3682478272 >> v2:172.21.15.49:3300/0 conn(0x7fa80c0b4330 0x7fa80c15f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:40.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.776+0000 7fa809d9b700 1 -- 172.21.15.16:0/3682478272 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa80c15fe20 con 0x7fa80c15f650 2023-10-16T12:10:40.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.776+0000 7fa809d9b700 1 -- 172.21.15.16:0/3682478272 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7fa8040019f0 con 0x7fa80c15f650 2023-10-16T12:10:40.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.776+0000 7fa809d9b700 1 -- 172.21.15.16:0/3682478272 >> v1:172.21.15.16:6789/0 conn(0x7fa80c15f650 legacy=0x7fa80c0cfa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:40.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.776+0000 7fa809d9b700 1 --2- 172.21.15.16:0/3682478272 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa80c0cbb50 0x7fa7f4003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:40.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.776+0000 7fa809d9b700 1 --2- 172.21.15.16:0/3682478272 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa7f4003f80 0x7fa7f4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:40.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.776+0000 7fa809d9b700 1 --2- 172.21.15.16:0/3682478272 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7f4004d90 0x7fa7f4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:40.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.776+0000 7fa809d9b700 1 -- 172.21.15.16:0/3682478272 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa80c160100 con 0x7fa7f4004d90 2023-10-16T12:10:40.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.776+0000 7fa809d9b700 1 -- 172.21.15.16:0/3682478272 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa80c15fb40 con 0x7fa7f4003f80 2023-10-16T12:10:40.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.776+0000 7fa809d9b700 1 -- 172.21.15.16:0/3682478272 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7f400ba60 con 0x7fa80c0cbb50 2023-10-16T12:10:40.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.777+0000 7fa80b59e700 1 --2- 172.21.15.16:0/3682478272 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7f4004d90 0x7fa7f4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:40.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.777+0000 7fa80a59c700 1 --2- 172.21.15.16:0/3682478272 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa7f4003f80 0x7fa7f4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:40.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.777+0000 7fa80ad9d700 1 --2- 172.21.15.16:0/3682478272 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa80c0cbb50 0x7fa7f4003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:40.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.777+0000 7fa80b59e700 1 -- 172.21.15.16:0/3682478272 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa80c0cbb50 msgr2=0x7fa7f4003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:10:40.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.777+0000 7fa80b59e700 1 --2- 172.21.15.16:0/3682478272 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa80c0cbb50 0x7fa7f4003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:40.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.777+0000 7fa80b59e700 1 -- 172.21.15.16:0/3682478272 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa7f4003f80 msgr2=0x7fa7f4004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:10:40.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.777+0000 7fa80b59e700 1 --2- 172.21.15.16:0/3682478272 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa7f4003f80 0x7fa7f4004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:40.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.777+0000 7fa80b59e700 1 -- 172.21.15.16:0/3682478272 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7f40035b0 con 0x7fa7f4004d90 2023-10-16T12:10:40.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.777+0000 7fa80b59e700 1 --2- 172.21.15.16:0/3682478272 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7f4004d90 0x7fa7f4005210 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa80400a900 tx=0x7fa7f4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90d8b715c4578e7b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:40.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.777+0000 7fa809d9b700 1 -- 172.21.15.16:0/3682478272 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa80400a340 con 0x7fa7f4004d90 2023-10-16T12:10:40.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.777+0000 7fa809d9b700 1 -- 172.21.15.16:0/3682478272 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa804012620 con 0x7fa7f4004d90 2023-10-16T12:10:40.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.777+0000 7fa809d9b700 1 -- 172.21.15.16:0/3682478272 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8040129d0 con 0x7fa7f4004d90 2023-10-16T12:10:40.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.777+0000 7fa811006700 1 -- 172.21.15.16:0/3682478272 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7f4004d90 msgr2=0x7fa7f4005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:40.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.777+0000 7fa811006700 1 --2- 172.21.15.16:0/3682478272 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7f4004d90 0x7fa7f4005210 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa80400a900 tx=0x7fa7f4005880 comp rx=0 tx=0).stop 2023-10-16T12:10:40.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.778+0000 7fa80b59e700 1 -- 172.21.15.16:0/3682478272 reap_dead start 2023-10-16T12:10:40.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.778+0000 7fa811006700 1 -- 172.21.15.16:0/3682478272 shutdown_connections 2023-10-16T12:10:40.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.778+0000 7fa811006700 1 -- 172.21.15.16:0/3682478272 >> 172.21.15.16:0/3682478272 conn(0x7fa80c06a7e0 msgr2=0x7fa80c06abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:40.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.778+0000 7fa811006700 1 -- 172.21.15.16:0/3682478272 shutdown_connections 2023-10-16T12:10:40.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.778+0000 7fa811006700 1 -- 172.21.15.16:0/3682478272 wait complete. 2023-10-16T12:10:40.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.779+0000 7fa811006700 1 Processor -- start 2023-10-16T12:10:40.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.779+0000 7fa811006700 1 -- start start 2023-10-16T12:10:40.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.780+0000 7fa811006700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7f4004d90 0x7fa80c06c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:40.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.780+0000 7fa811006700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa80c14faa0 0x7fa80c0afda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:40.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.780+0000 7fa811006700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa80c0c6330 0x7fa80c0c67b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:40.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.780+0000 7fa811006700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa80c06c8e0 con 0x7fa7f4004d90 2023-10-16T12:10:40.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.780+0000 7fa811006700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa80c151060 con 0x7fa80c14faa0 2023-10-16T12:10:40.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.780+0000 7fa811006700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa80c14ff20 con 0x7fa80c0c6330 2023-10-16T12:10:40.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.780+0000 7fa80ad9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7f4004d90 0x7fa80c06c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:40.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.780+0000 7fa80ad9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7f4004d90 0x7fa80c06c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:32996/0 (socket says 172.21.15.16:32996) 2023-10-16T12:10:40.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.780+0000 7fa80ad9d700 1 -- 172.21.15.16:0/1984174474 learned_addr learned my addr 172.21.15.16:0/1984174474 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:10:40.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.780+0000 7fa80a59c700 1 --2- 172.21.15.16:0/1984174474 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa80c14faa0 0x7fa80c0afda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:40.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.780+0000 7fa80b59e700 1 --2- 172.21.15.16:0/1984174474 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa80c0c6330 0x7fa80c0c67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:40.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.780+0000 7fa80ad9d700 1 -- 172.21.15.16:0/1984174474 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa80c0c6330 msgr2=0x7fa80c0c67b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:40.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.780+0000 7fa80ad9d700 1 --2- 172.21.15.16:0/1984174474 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa80c0c6330 0x7fa80c0c67b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:40.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.780+0000 7fa80ad9d700 1 -- 172.21.15.16:0/1984174474 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa80c14faa0 msgr2=0x7fa80c0afda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:40.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.780+0000 7fa80ad9d700 1 --2- 172.21.15.16:0/1984174474 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa80c14faa0 0x7fa80c0afda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:40.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.780+0000 7fa80ad9d700 1 -- 172.21.15.16:0/1984174474 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa80c0b02e0 con 0x7fa7f4004d90 2023-10-16T12:10:40.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.780+0000 7fa80ad9d700 1 --2- 172.21.15.16:0/1984174474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7f4004d90 0x7fa80c06c2d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc011ad0 tx=0x7fa7fc011de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:40.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.781+0000 7fa7f37fe700 1 -- 172.21.15.16:0/1984174474 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7fc00f8e0 con 0x7fa7f4004d90 2023-10-16T12:10:40.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.781+0000 7fa7f37fe700 1 -- 172.21.15.16:0/1984174474 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7fc00fa40 con 0x7fa7f4004d90 2023-10-16T12:10:40.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.781+0000 7fa811006700 1 -- 172.21.15.16:0/1984174474 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa80c150730 con 0x7fa7f4004d90 2023-10-16T12:10:40.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.781+0000 7fa7f37fe700 1 -- 172.21.15.16:0/1984174474 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7fc00fdf0 con 0x7fa7f4004d90 2023-10-16T12:10:40.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.781+0000 7fa811006700 1 -- 172.21.15.16:0/1984174474 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa80c150cf0 con 0x7fa7f4004d90 2023-10-16T12:10:40.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.782+0000 7fa811006700 1 -- 172.21.15.16:0/1984174474 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa80c151340 con 0x7fa7f4004d90 2023-10-16T12:10:40.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.782+0000 7fa7f37fe700 1 -- 172.21.15.16:0/1984174474 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fa7fc014440 con 0x7fa7f4004d90 2023-10-16T12:10:40.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.782+0000 7fa7f37fe700 1 --2- 172.21.15.16:0/1984174474 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fa7f806af00 0x7fa7f806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:40.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.783+0000 7fa7f37fe700 1 -- 172.21.15.16:0/1984174474 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5935+0+0 (secure 0 0 0) 0x7fa7fc089c50 con 0x7fa7f4004d90 2023-10-16T12:10:40.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.783+0000 7fa80a59c700 1 --2- 172.21.15.16:0/1984174474 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fa7f806af00 0x7fa7f806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:40.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.783+0000 7fa80a59c700 1 --2- 172.21.15.16:0/1984174474 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fa7f806af00 0x7fa7f806d3c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa80c0cabd0 tx=0x7fa800009210 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:40.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.787+0000 7fa7f37fe700 1 -- 172.21.15.16:0/1984174474 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa7fc08f020 con 0x7fa7f4004d90 2023-10-16T12:10:40.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:40.989+0000 7fa811006700 1 -- 172.21.15.16:0/1984174474 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fa80c074b00 con 0x7fa7f4004d90 2023-10-16T12:10:42.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.636+0000 7fa7f37fe700 1 -- 172.21.15.16:0/1984174474 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v23) v1 ==== 228+0+0 (secure 0 0 0) 0x7fa7fc054e20 con 0x7fa7f4004d90 2023-10-16T12:10:42.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.641+0000 7fa811006700 1 -- 172.21.15.16:0/1984174474 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fa7f806af00 msgr2=0x7fa7f806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:42.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.641+0000 7fa811006700 1 --2- 172.21.15.16:0/1984174474 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fa7f806af00 0x7fa7f806d3c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa80c0cabd0 tx=0x7fa800009210 comp rx=0 tx=0).stop 2023-10-16T12:10:42.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.641+0000 7fa811006700 1 -- 172.21.15.16:0/1984174474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7f4004d90 msgr2=0x7fa80c06c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:42.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.641+0000 7fa811006700 1 --2- 172.21.15.16:0/1984174474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7f4004d90 0x7fa80c06c2d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc011ad0 tx=0x7fa7fc011de0 comp rx=0 tx=0).stop 2023-10-16T12:10:42.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.642+0000 7fa811006700 1 -- 172.21.15.16:0/1984174474 shutdown_connections 2023-10-16T12:10:42.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.642+0000 7fa811006700 1 --2- 172.21.15.16:0/1984174474 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fa7f806af00 0x7fa7f806d3c0 secure :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa80c0cabd0 tx=0x7fa800009210 comp rx=0 tx=0).stop 2023-10-16T12:10:42.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.642+0000 7fa811006700 1 --2- 172.21.15.16:0/1984174474 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa80c0c6330 0x7fa80c0c67b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:42.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.642+0000 7fa811006700 1 --2- 172.21.15.16:0/1984174474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7f4004d90 0x7fa80c06c2d0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:42.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.642+0000 7fa811006700 1 --2- 172.21.15.16:0/1984174474 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa80c14faa0 0x7fa80c0afda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:42.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.642+0000 7fa811006700 1 -- 172.21.15.16:0/1984174474 >> 172.21.15.16:0/1984174474 conn(0x7fa80c06a7e0 msgr2=0x7fa80c0d04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:42.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.644+0000 7fa811006700 1 -- 172.21.15.16:0/1984174474 shutdown_connections 2023-10-16T12:10:42.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.644+0000 7fa811006700 1 -- 172.21.15.16:0/1984174474 wait complete. 2023-10-16T12:10:42.653 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data_ec' created 2023-10-16T12:10:42.667 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2023-10-16T12:10:42.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.799+0000 7f56d198a700 1 Processor -- start 2023-10-16T12:10:42.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.800+0000 7f56d198a700 1 -- start start 2023-10-16T12:10:42.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.801+0000 7f56d198a700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f56cc15b500 0x7f56cc15fab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:42.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.801+0000 7f56d198a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f56cc06e590 0x7f56cc0ca930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:42.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.801+0000 7f56d198a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f56cc0cae70 0x7f56cc071970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:42.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.801+0000 7f56d198a700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56cc160090 con 0x7f56cc06e590 2023-10-16T12:10:42.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.801+0000 7f56d198a700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56cc0cb320 con 0x7f56cc15b500 2023-10-16T12:10:42.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.801+0000 7f56d198a700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56cc0cc800 con 0x7f56cc0cae70 2023-10-16T12:10:42.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.801+0000 7f56ca7fc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f56cc06e590 0x7f56cc0ca930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:42.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.801+0000 7f56ca7fc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f56cc06e590 0x7f56cc0ca930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33004/0 (socket says 172.21.15.16:33004) 2023-10-16T12:10:42.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.801+0000 7f56ca7fc700 1 -- 172.21.15.16:0/3278974978 learned_addr learned my addr 172.21.15.16:0/3278974978 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:10:42.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.801+0000 7f56cb7fe700 1 --2- 172.21.15.16:0/3278974978 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f56cc0cae70 0x7f56cc071970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:42.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.801+0000 7f56caffd700 1 --2- 172.21.15.16:0/3278974978 >> v2:172.21.15.49:3300/0 conn(0x7f56cc15b500 0x7f56cc15fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:42.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.801+0000 7f56ca7fc700 1 -- 172.21.15.16:0/3278974978 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f56cc0cae70 msgr2=0x7f56cc071970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:10:42.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.801+0000 7f56ca7fc700 1 --2- 172.21.15.16:0/3278974978 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f56cc0cae70 0x7f56cc071970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:42.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.801+0000 7f56ca7fc700 1 -- 172.21.15.16:0/3278974978 >> v2:172.21.15.49:3300/0 conn(0x7f56cc15b500 msgr2=0x7f56cc15fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:10:42.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.801+0000 7f56ca7fc700 1 --2- 172.21.15.16:0/3278974978 >> v2:172.21.15.49:3300/0 conn(0x7f56cc15b500 0x7f56cc15fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:42.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.801+0000 7f56ca7fc700 1 -- 172.21.15.16:0/3278974978 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56cc06b220 con 0x7f56cc06e590 2023-10-16T12:10:42.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.801+0000 7f56caffd700 1 --2- 172.21.15.16:0/3278974978 >> v2:172.21.15.49:3300/0 conn(0x7f56cc15b500 0x7f56cc15fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:10:42.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.801+0000 7f56ca7fc700 1 --2- 172.21.15.16:0/3278974978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56cc06e590 0x7f56cc0ca930 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f56cc073080 tx=0x7f56bc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc3101a2fa03b058 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:42.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.801+0000 7f56c9ffb700 1 -- 172.21.15.16:0/3278974978 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56bc011830 con 0x7f56cc06e590 2023-10-16T12:10:42.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.802+0000 7f56c9ffb700 1 -- 172.21.15.16:0/3278974978 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56bc011990 con 0x7f56cc06e590 2023-10-16T12:10:42.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.802+0000 7f56c9ffb700 1 -- 172.21.15.16:0/3278974978 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56bc011d20 con 0x7f56cc06e590 2023-10-16T12:10:42.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.802+0000 7f56d198a700 1 -- 172.21.15.16:0/3278974978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56cc06e590 msgr2=0x7f56cc0ca930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:42.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.802+0000 7f56d198a700 1 --2- 172.21.15.16:0/3278974978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56cc06e590 0x7f56cc0ca930 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f56cc073080 tx=0x7f56bc009bd0 comp rx=0 tx=0).stop 2023-10-16T12:10:42.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.802+0000 7f56d198a700 1 -- 172.21.15.16:0/3278974978 shutdown_connections 2023-10-16T12:10:42.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.802+0000 7f56d198a700 1 --2- 172.21.15.16:0/3278974978 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f56cc0cae70 0x7f56cc071970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:42.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.802+0000 7f56d198a700 1 --2- 172.21.15.16:0/3278974978 >> v2:172.21.15.49:3300/0 conn(0x7f56cc15b500 0x7f56cc15fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:42.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.802+0000 7f56d198a700 1 --2- 172.21.15.16:0/3278974978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56cc06e590 0x7f56cc0ca930 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:42.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.802+0000 7f56d198a700 1 -- 172.21.15.16:0/3278974978 >> 172.21.15.16:0/3278974978 conn(0x7f56cc06f430 msgr2=0x7f56cc06f810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:42.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.802+0000 7f56d198a700 1 -- 172.21.15.16:0/3278974978 shutdown_connections 2023-10-16T12:10:42.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.802+0000 7f56d198a700 1 -- 172.21.15.16:0/3278974978 wait complete. 2023-10-16T12:10:42.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.803+0000 7f56d198a700 1 Processor -- start 2023-10-16T12:10:42.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.804+0000 7f56d198a700 1 -- start start 2023-10-16T12:10:42.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.804+0000 7f56d198a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56cc06e590 0x7f56cc14c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:42.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.804+0000 7f56d198a700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f56cc0cae70 0x7f56cc0b2eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:42.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.804+0000 7f56d198a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f56cc15b500 0x7f56cc15dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:42.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.804+0000 7f56d198a700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56cc15ba10 con 0x7f56cc06e590 2023-10-16T12:10:42.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.804+0000 7f56d198a700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56cc0b3820 con 0x7f56cc0cae70 2023-10-16T12:10:42.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.804+0000 7f56d198a700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56cc0b3b00 con 0x7f56cc15b500 2023-10-16T12:10:42.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.804+0000 7f56caffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56cc06e590 0x7f56cc14c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:42.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.804+0000 7f56caffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56cc06e590 0x7f56cc14c610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33008/0 (socket says 172.21.15.16:33008) 2023-10-16T12:10:42.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.804+0000 7f56caffd700 1 -- 172.21.15.16:0/831209659 learned_addr learned my addr 172.21.15.16:0/831209659 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:10:42.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.804+0000 7f56ca7fc700 1 --2- 172.21.15.16:0/831209659 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f56cc0cae70 0x7f56cc0b2eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:42.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.804+0000 7f56cb7fe700 1 --2- 172.21.15.16:0/831209659 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f56cc15b500 0x7f56cc15dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:42.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.805+0000 7f56caffd700 1 -- 172.21.15.16:0/831209659 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f56cc15b500 msgr2=0x7f56cc15dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:42.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.805+0000 7f56caffd700 1 --2- 172.21.15.16:0/831209659 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f56cc15b500 0x7f56cc15dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:42.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.805+0000 7f56caffd700 1 -- 172.21.15.16:0/831209659 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f56cc0cae70 msgr2=0x7f56cc0b2eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:42.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.805+0000 7f56caffd700 1 --2- 172.21.15.16:0/831209659 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f56cc0cae70 0x7f56cc0b2eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:42.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.805+0000 7f56caffd700 1 -- 172.21.15.16:0/831209659 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56bc009550 con 0x7f56cc06e590 2023-10-16T12:10:42.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.805+0000 7f56caffd700 1 --2- 172.21.15.16:0/831209659 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56cc06e590 0x7f56cc14c610 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f56c000e810 tx=0x7f56c000eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:42.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.805+0000 7f56b3fff700 1 -- 172.21.15.16:0/831209659 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56c0013610 con 0x7f56cc06e590 2023-10-16T12:10:42.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.805+0000 7f56b3fff700 1 -- 172.21.15.16:0/831209659 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56c0013770 con 0x7f56cc06e590 2023-10-16T12:10:42.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.805+0000 7f56d198a700 1 -- 172.21.15.16:0/831209659 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56cc0b3de0 con 0x7f56cc06e590 2023-10-16T12:10:42.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.805+0000 7f56b3fff700 1 -- 172.21.15.16:0/831209659 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56c0013b20 con 0x7f56cc06e590 2023-10-16T12:10:42.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.805+0000 7f56d198a700 1 -- 172.21.15.16:0/831209659 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56cc0b40c0 con 0x7f56cc06e590 2023-10-16T12:10:42.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.806+0000 7f56d198a700 1 -- 172.21.15.16:0/831209659 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56cc15bcf0 con 0x7f56cc06e590 2023-10-16T12:10:42.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.807+0000 7f56b3fff700 1 -- 172.21.15.16:0/831209659 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f56c0013c80 con 0x7f56cc06e590 2023-10-16T12:10:42.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.807+0000 7f56b3fff700 1 --2- 172.21.15.16:0/831209659 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f56b806af00 0x7f56b806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:42.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.807+0000 7f56b3fff700 1 -- 172.21.15.16:0/831209659 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6428+0+0 (secure 0 0 0) 0x7f56c0089450 con 0x7f56cc06e590 2023-10-16T12:10:42.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.807+0000 7f56ca7fc700 1 --2- 172.21.15.16:0/831209659 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f56b806af00 0x7f56b806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:42.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.807+0000 7f56ca7fc700 1 --2- 172.21.15.16:0/831209659 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f56b806af00 0x7f56b806d3c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f56cc06e9d0 tx=0x7f56bc008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:42.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:42.812+0000 7f56b3fff700 1 -- 172.21.15.16:0/831209659 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f56c0054230 con 0x7f56cc06e590 2023-10-16T12:10:43.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:43.013+0000 7f56d198a700 1 -- 172.21.15.16:0/831209659 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) v1 -- 0x7f56cc04ec20 con 0x7f56cc06e590 2023-10-16T12:10:43.389 INFO:tasks.ceph.mds.d.smithi049.stderr:2023-10-16T12:10:43.387+0000 7fd982013700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T12:10:43.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:43.638+0000 7f56b3fff700 1 -- 172.21.15.16:0/831209659 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v24) v1 ==== 169+0+0 (secure 0 0 0) 0x7f56c0057850 con 0x7f56cc06e590 2023-10-16T12:10:43.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:43.642+0000 7f56d198a700 1 -- 172.21.15.16:0/831209659 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f56b806af00 msgr2=0x7f56b806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:43.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:43.642+0000 7f56d198a700 1 --2- 172.21.15.16:0/831209659 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f56b806af00 0x7f56b806d3c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f56cc06e9d0 tx=0x7f56bc008030 comp rx=0 tx=0).stop 2023-10-16T12:10:43.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:43.642+0000 7f56d198a700 1 -- 172.21.15.16:0/831209659 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56cc06e590 msgr2=0x7f56cc14c610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:43.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:43.642+0000 7f56d198a700 1 --2- 172.21.15.16:0/831209659 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56cc06e590 0x7f56cc14c610 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f56c000e810 tx=0x7f56c000eb20 comp rx=0 tx=0).stop 2023-10-16T12:10:43.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:43.644+0000 7f56d198a700 1 -- 172.21.15.16:0/831209659 shutdown_connections 2023-10-16T12:10:43.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:43.644+0000 7f56d198a700 1 --2- 172.21.15.16:0/831209659 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f56b806af00 0x7f56b806d3c0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:43.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:43.644+0000 7f56d198a700 1 --2- 172.21.15.16:0/831209659 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f56cc15b500 0x7f56cc15dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:43.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:43.644+0000 7f56d198a700 1 --2- 172.21.15.16:0/831209659 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56cc06e590 0x7f56cc14c610 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:43.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:43.644+0000 7f56d198a700 1 --2- 172.21.15.16:0/831209659 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f56cc0cae70 0x7f56cc0b2eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:43.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:43.644+0000 7f56d198a700 1 -- 172.21.15.16:0/831209659 >> 172.21.15.16:0/831209659 conn(0x7f56cc06f430 msgr2=0x7f56cc159ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:43.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:43.645+0000 7f56d198a700 1 -- 172.21.15.16:0/831209659 shutdown_connections 2023-10-16T12:10:43.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:43.645+0000 7f56d198a700 1 -- 172.21.15.16:0/831209659 wait complete. 2023-10-16T12:10:43.655 INFO:teuthology.orchestra.run.smithi016.stderr:set pool 4 allow_ec_overwrites to true 2023-10-16T12:10:43.676 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2023-10-16T12:10:43.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:43.808+0000 7f4b089c5700 1 Processor -- start 2023-10-16T12:10:43.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:43.808+0000 7f4b089c5700 1 -- start start 2023-10-16T12:10:43.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:43.809+0000 7f4b089c5700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4b0415d8d0 0x7f4b040b2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:43.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:43.809+0000 7f4b089c5700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b040b2f30 con 0x7f4b040ce350 2023-10-16T12:10:43.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:43.809+0000 7f4b089c5700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b040d1c40 con 0x7f4b040b8fa0 2023-10-16T12:10:43.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:43.809+0000 7f4b089c5700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b040d0bb0 con 0x7f4b0415d8d0 2023-10-16T12:10:43.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:43.809+0000 7f4b02d9d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f4b040ce350 0x7f4b040d29c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50526/0 (socket says 172.21.15.16:50526) 2023-10-16T12:10:43.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:43.809+0000 7f4b02d9d700 1 -- 172.21.15.16:0/639017283 learned_addr learned my addr 172.21.15.16:0/639017283 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:10:43.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:43.809+0000 7f4b0259c700 1 --2- 172.21.15.16:0/639017283 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4b0415d8d0 0x7f4b040b2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:43.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:43.809+0000 7f4b0159a700 1 -- 172.21.15.16:0/639017283 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 454344190 0 0) 0x7f4b040b2f30 con 0x7f4b040ce350 2023-10-16T12:10:43.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:43.810+0000 7f4b0159a700 1 -- 172.21.15.16:0/639017283 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4aec0035b0 con 0x7f4b040ce350 2023-10-16T12:10:43.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:43.810+0000 7f4b0159a700 1 -- 172.21.15.16:0/639017283 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2307274706 0 0) 0x7f4b040d1c40 con 0x7f4b040b8fa0 2023-10-16T12:10:43.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:43.810+0000 7f4b0159a700 1 -- 172.21.15.16:0/639017283 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b040b2f30 con 0x7f4b040b8fa0 2023-10-16T12:10:43.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:43.810+0000 7f4b0259c700 1 -- 172.21.15.16:0/639017283 >> v1:172.21.15.49:6789/0 conn(0x7f4b040b8fa0 legacy=0x7f4b040b6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:43.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:43.810+0000 7f4b0259c700 1 -- 172.21.15.16:0/639017283 >> v1:172.21.15.16:6789/0 conn(0x7f4b040ce350 legacy=0x7f4b040d29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:43.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:43.810+0000 7f4b0259c700 1 -- 172.21.15.16:0/639017283 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b040d1ee0 con 0x7f4b0415d8d0 2023-10-16T12:10:43.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:43.810+0000 7f4b0259c700 1 --2- 172.21.15.16:0/639017283 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4b0415d8d0 0x7f4b040b2830 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4af4009870 tx=0x7f4af4009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d9c28cdc8054fd18 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:43.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:43.810+0000 7f4b0159a700 1 -- 172.21.15.16:0/639017283 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4af4011830 con 0x7f4b0415d8d0 2023-10-16T12:10:43.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:43.810+0000 7f4b0159a700 1 -- 172.21.15.16:0/639017283 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4af4011990 con 0x7f4b0415d8d0 2023-10-16T12:10:43.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:43.810+0000 7f4b0159a700 1 -- 172.21.15.16:0/639017283 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4af4011d20 con 0x7f4b0415d8d0 2023-10-16T12:10:43.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:43.811+0000 7f4b089c5700 1 -- 172.21.15.16:0/639017283 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4b0415d8d0 msgr2=0x7f4b040b2830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:43.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:43.811+0000 7f4b089c5700 1 --2- 172.21.15.16:0/639017283 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4b0415d8d0 0x7f4b040b2830 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4af4009870 tx=0x7f4af4009b80 comp rx=0 tx=0).stop 2023-10-16T12:10:43.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:43.811+0000 7f4b089c5700 1 -- 172.21.15.16:0/639017283 shutdown_connections 2023-10-16T12:10:43.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:43.811+0000 7f4b089c5700 1 --2- 172.21.15.16:0/639017283 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4b0415d8d0 0x7f4b040b2830 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:43.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:43.811+0000 7f4b089c5700 1 -- 172.21.15.16:0/639017283 >> 172.21.15.16:0/639017283 conn(0x7f4b04072330 msgr2=0x7f4b04072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:43.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:43.811+0000 7f4b089c5700 1 -- 172.21.15.16:0/639017283 shutdown_connections 2023-10-16T12:10:43.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:43.811+0000 7f4b089c5700 1 -- 172.21.15.16:0/639017283 wait complete. 2023-10-16T12:10:43.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:43.812+0000 7f4b089c5700 1 Processor -- start 2023-10-16T12:10:43.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:43.812+0000 7f4b089c5700 1 -- start start 2023-10-16T12:10:43.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:43.813+0000 7f4b089c5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b040b8fa0 0x7f4b040d0170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:43.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:43.813+0000 7f4b089c5700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4b040ce350 0x7f4b04160370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:43.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:43.813+0000 7f4b089c5700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4b0415d8d0 0x7f4b041608b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:43.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:43.813+0000 7f4b089c5700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b0415e9b0 con 0x7f4b040b8fa0 2023-10-16T12:10:43.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:43.813+0000 7f4b089c5700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b041f2d50 con 0x7f4b040ce350 2023-10-16T12:10:43.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:43.813+0000 7f4b089c5700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b0415dde0 con 0x7f4b0415d8d0 2023-10-16T12:10:43.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:43.813+0000 7f4b0259c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b040b8fa0 0x7f4b040d0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:43.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:43.813+0000 7f4b0259c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b040b8fa0 0x7f4b040d0170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33022/0 (socket says 172.21.15.16:33022) 2023-10-16T12:10:43.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:43.813+0000 7f4b0259c700 1 -- 172.21.15.16:0/448742112 learned_addr learned my addr 172.21.15.16:0/448742112 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:10:43.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:43.813+0000 7f4b01d9b700 1 --2- 172.21.15.16:0/448742112 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4b040ce350 0x7f4b04160370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:43.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:43.813+0000 7f4b02d9d700 1 --2- 172.21.15.16:0/448742112 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4b0415d8d0 0x7f4b041608b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:43.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:43.813+0000 7f4b0259c700 1 -- 172.21.15.16:0/448742112 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4b0415d8d0 msgr2=0x7f4b041608b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:43.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:43.813+0000 7f4b0259c700 1 --2- 172.21.15.16:0/448742112 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4b0415d8d0 0x7f4b041608b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:43.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:43.813+0000 7f4b0259c700 1 -- 172.21.15.16:0/448742112 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4b040ce350 msgr2=0x7f4b04160370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:43.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:43.813+0000 7f4b0259c700 1 --2- 172.21.15.16:0/448742112 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4b040ce350 0x7f4b04160370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:43.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:43.813+0000 7f4b0259c700 1 -- 172.21.15.16:0/448742112 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4af40094a0 con 0x7f4b040b8fa0 2023-10-16T12:10:43.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:43.813+0000 7f4b02d9d700 1 --2- 172.21.15.16:0/448742112 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4b0415d8d0 0x7f4b041608b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:10:43.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:43.813+0000 7f4b0259c700 1 --2- 172.21.15.16:0/448742112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b040b8fa0 0x7f4b040d0170 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4af40097c0 tx=0x7f4af4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:43.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:43.813+0000 7f4af2ffd700 1 -- 172.21.15.16:0/448742112 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4af401b870 con 0x7f4b040b8fa0 2023-10-16T12:10:43.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:43.813+0000 7f4af2ffd700 1 -- 172.21.15.16:0/448742112 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4af401b9d0 con 0x7f4b040b8fa0 2023-10-16T12:10:43.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:43.813+0000 7f4b089c5700 1 -- 172.21.15.16:0/448742112 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b0415e080 con 0x7f4b040b8fa0 2023-10-16T12:10:43.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:43.814+0000 7f4af2ffd700 1 -- 172.21.15.16:0/448742112 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4af401bd80 con 0x7f4b040b8fa0 2023-10-16T12:10:43.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:43.814+0000 7f4b089c5700 1 -- 172.21.15.16:0/448742112 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b0415e640 con 0x7f4b040b8fa0 2023-10-16T12:10:43.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:43.814+0000 7f4b089c5700 1 -- 172.21.15.16:0/448742112 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b0404ebb0 con 0x7f4b040b8fa0 2023-10-16T12:10:43.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:43.815+0000 7f4af2ffd700 1 -- 172.21.15.16:0/448742112 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f4af4023070 con 0x7f4b040b8fa0 2023-10-16T12:10:43.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:43.815+0000 7f4af2ffd700 1 --2- 172.21.15.16:0/448742112 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f4ae406af00 0x7f4ae406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:43.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:43.815+0000 7f4af2ffd700 1 -- 172.21.15.16:0/448742112 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6428+0+0 (secure 0 0 0) 0x7f4af408a860 con 0x7f4b040b8fa0 2023-10-16T12:10:43.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:43.815+0000 7f4b01d9b700 1 --2- 172.21.15.16:0/448742112 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f4ae406af00 0x7f4ae406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:43.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:43.816+0000 7f4b01d9b700 1 --2- 172.21.15.16:0/448742112 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f4ae406af00 0x7f4ae406d3c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4af8007fd0 tx=0x7f4af8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:43.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:43.820+0000 7f4af2ffd700 1 -- 172.21.15.16:0/448742112 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4af40557f0 con 0x7f4b040b8fa0 2023-10-16T12:10:44.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:44.077+0000 7f4b089c5700 1 -- 172.21.15.16:0/448742112 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) v1 -- 0x7f4b040d2470 con 0x7f4b040b8fa0 2023-10-16T12:10:44.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:44.683+0000 7f4af2ffd700 1 -- 172.21.15.16:0/448742112 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v7) v1 ==== 137+0+0 (secure 0 0 0) 0x7f4af4058e10 con 0x7f4b040b8fa0 2023-10-16T12:10:44.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:44.686+0000 7f4b089c5700 1 -- 172.21.15.16:0/448742112 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f4ae406af00 msgr2=0x7f4ae406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:44.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:44.686+0000 7f4b089c5700 1 --2- 172.21.15.16:0/448742112 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f4ae406af00 0x7f4ae406d3c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4af8007fd0 tx=0x7f4af8009040 comp rx=0 tx=0).stop 2023-10-16T12:10:44.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:44.686+0000 7f4b089c5700 1 -- 172.21.15.16:0/448742112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b040b8fa0 msgr2=0x7f4b040d0170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:44.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:44.686+0000 7f4b089c5700 1 --2- 172.21.15.16:0/448742112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b040b8fa0 0x7f4b040d0170 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4af40097c0 tx=0x7f4af4004750 comp rx=0 tx=0).stop 2023-10-16T12:10:44.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:44.686+0000 7f4b089c5700 1 -- 172.21.15.16:0/448742112 shutdown_connections 2023-10-16T12:10:44.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:44.686+0000 7f4b089c5700 1 --2- 172.21.15.16:0/448742112 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f4ae406af00 0x7f4ae406d3c0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:44.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:44.686+0000 7f4b089c5700 1 --2- 172.21.15.16:0/448742112 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4b0415d8d0 0x7f4b041608b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:44.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:44.686+0000 7f4b089c5700 1 --2- 172.21.15.16:0/448742112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b040b8fa0 0x7f4b040d0170 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:44.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:44.686+0000 7f4b089c5700 1 --2- 172.21.15.16:0/448742112 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4b040ce350 0x7f4b04160370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:44.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:44.686+0000 7f4b089c5700 1 -- 172.21.15.16:0/448742112 >> 172.21.15.16:0/448742112 conn(0x7f4b04072330 msgr2=0x7f4b0415cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:44.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:44.686+0000 7f4b089c5700 1 -- 172.21.15.16:0/448742112 shutdown_connections 2023-10-16T12:10:44.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:44.686+0000 7f4b089c5700 1 -- 172.21.15.16:0/448742112 wait complete. 2023-10-16T12:10:44.692 INFO:teuthology.orchestra.run.smithi016.stderr:added data pool 4 to fsmap 2023-10-16T12:10:44.704 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:10:44.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:44.832+0000 7ff8827fd700 1 Processor -- start 2023-10-16T12:10:44.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:44.832+0000 7ff8827fd700 1 -- start start 2023-10-16T12:10:44.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:44.833+0000 7ff8827fd700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff87c0cc8a0 0x7ff87c15e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:44.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:44.833+0000 7ff8827fd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff87c06a7e0 0x7ff87c06ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:44.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:44.833+0000 7ff8827fd700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff87c06b170 con 0x7ff87c15b280 2023-10-16T12:10:44.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:44.833+0000 7ff8827fd700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff87c161c30 con 0x7ff87c06a7e0 2023-10-16T12:10:44.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:44.833+0000 7ff8827fd700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff87c161f10 con 0x7ff87c0cc8a0 2023-10-16T12:10:44.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:44.833+0000 7ff880d9a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff87c06a7e0 0x7ff87c06ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:44.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:44.833+0000 7ff87bfff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff87c15b280 0x7ff87c1613e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50536/0 (socket says 172.21.15.16:50536) 2023-10-16T12:10:44.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:44.833+0000 7ff87bfff700 1 -- 172.21.15.16:0/862890926 learned_addr learned my addr 172.21.15.16:0/862890926 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:10:44.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:44.833+0000 7ff87b7fe700 1 --2- 172.21.15.16:0/862890926 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff87c0cc8a0 0x7ff87c15e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:44.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:44.834+0000 7ff880d9a700 1 -- 172.21.15.16:0/862890926 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff87c0cc8a0 msgr2=0x7ff87c15e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:10:44.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:44.834+0000 7ff880d9a700 1 --2- 172.21.15.16:0/862890926 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff87c0cc8a0 0x7ff87c15e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:44.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:44.834+0000 7ff880d9a700 1 -- 172.21.15.16:0/862890926 >> v1:172.21.15.16:6789/0 conn(0x7ff87c15b280 legacy=0x7ff87c1613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:44.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:44.834+0000 7ff880d9a700 1 -- 172.21.15.16:0/862890926 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff87c1621f0 con 0x7ff87c06a7e0 2023-10-16T12:10:44.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:44.834+0000 7ff87affd700 1 -- 172.21.15.16:0/862890926 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1753449949 0 0) 0x7ff86c001110 con 0x7ff87c15b280 2023-10-16T12:10:44.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:44.834+0000 7ff880d9a700 1 --2- 172.21.15.16:0/862890926 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff87c06a7e0 0x7ff87c06ac30 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff874009d90 tx=0x7ff87400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98963bdc7c118b4c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:44.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:44.834+0000 7ff87affd700 1 -- 172.21.15.16:0/862890926 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff87400e070 con 0x7ff87c06a7e0 2023-10-16T12:10:44.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:44.834+0000 7ff87affd700 1 -- 172.21.15.16:0/862890926 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff87400bb20 con 0x7ff87c06a7e0 2023-10-16T12:10:44.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:44.834+0000 7ff87affd700 1 -- 172.21.15.16:0/862890926 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff874012430 con 0x7ff87c06a7e0 2023-10-16T12:10:44.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:44.834+0000 7ff8827fd700 1 -- 172.21.15.16:0/862890926 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff87c06a7e0 msgr2=0x7ff87c06ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:44.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:44.834+0000 7ff8827fd700 1 --2- 172.21.15.16:0/862890926 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff87c06a7e0 0x7ff87c06ac30 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff874009d90 tx=0x7ff87400b330 comp rx=0 tx=0).stop 2023-10-16T12:10:44.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:44.834+0000 7ff8827fd700 1 -- 172.21.15.16:0/862890926 shutdown_connections 2023-10-16T12:10:44.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:44.834+0000 7ff8827fd700 1 --2- 172.21.15.16:0/862890926 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff87c06a7e0 0x7ff87c06ac30 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:44.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:44.834+0000 7ff8827fd700 1 --2- 172.21.15.16:0/862890926 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff87c0cc8a0 0x7ff87c15e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:44.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:44.834+0000 7ff8827fd700 1 -- 172.21.15.16:0/862890926 >> 172.21.15.16:0/862890926 conn(0x7ff87c0b1fd0 msgr2=0x7ff87c0b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:44.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:44.834+0000 7ff8827fd700 1 -- 172.21.15.16:0/862890926 shutdown_connections 2023-10-16T12:10:44.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:44.835+0000 7ff8827fd700 1 -- 172.21.15.16:0/862890926 wait complete. 2023-10-16T12:10:44.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:44.836+0000 7ff8827fd700 1 Processor -- start 2023-10-16T12:10:44.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:44.836+0000 7ff8827fd700 1 -- start start 2023-10-16T12:10:44.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:44.837+0000 7ff8827fd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff87c06a7e0 0x7ff87c156ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:44.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:44.837+0000 7ff8827fd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff87c0cc8a0 0x7ff87c0c4c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:44.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:44.837+0000 7ff8827fd700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff87c15b280 0x7ff87c15d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:44.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:44.837+0000 7ff8827fd700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff87c15c1b0 con 0x7ff87c06a7e0 2023-10-16T12:10:44.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:44.837+0000 7ff8827fd700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff87c15c490 con 0x7ff87c0cc8a0 2023-10-16T12:10:44.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:44.837+0000 7ff8827fd700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff87c15c770 con 0x7ff87c15b280 2023-10-16T12:10:44.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:44.837+0000 7ff87bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff87c06a7e0 0x7ff87c156ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:44.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:44.837+0000 7ff87bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff87c06a7e0 0x7ff87c156ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33036/0 (socket says 172.21.15.16:33036) 2023-10-16T12:10:44.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:44.837+0000 7ff87bfff700 1 -- 172.21.15.16:0/3443162227 learned_addr learned my addr 172.21.15.16:0/3443162227 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:10:44.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:44.837+0000 7ff87b7fe700 1 --2- 172.21.15.16:0/3443162227 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff87c0cc8a0 0x7ff87c0c4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:44.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:44.837+0000 7ff880d9a700 1 --2- 172.21.15.16:0/3443162227 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff87c15b280 0x7ff87c15d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:44.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:44.837+0000 7ff87bfff700 1 -- 172.21.15.16:0/3443162227 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff87c15b280 msgr2=0x7ff87c15d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:44.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:44.837+0000 7ff87bfff700 1 --2- 172.21.15.16:0/3443162227 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff87c15b280 0x7ff87c15d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:44.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:44.837+0000 7ff87bfff700 1 -- 172.21.15.16:0/3443162227 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff87c0cc8a0 msgr2=0x7ff87c0c4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:44.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:44.837+0000 7ff87bfff700 1 --2- 172.21.15.16:0/3443162227 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff87c0cc8a0 0x7ff87c0c4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:44.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:44.837+0000 7ff87bfff700 1 -- 172.21.15.16:0/3443162227 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff874009a70 con 0x7ff87c06a7e0 2023-10-16T12:10:44.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:44.837+0000 7ff87bfff700 1 --2- 172.21.15.16:0/3443162227 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff87c06a7e0 0x7ff87c156ad0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff86c00c550 tx=0x7ff86c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:44.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:44.837+0000 7ff878ff9700 1 -- 172.21.15.16:0/3443162227 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff86c004c80 con 0x7ff87c06a7e0 2023-10-16T12:10:44.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:44.837+0000 7ff878ff9700 1 -- 172.21.15.16:0/3443162227 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff86c004e00 con 0x7ff87c06a7e0 2023-10-16T12:10:44.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:44.837+0000 7ff8827fd700 1 -- 172.21.15.16:0/3443162227 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff87c15ca50 con 0x7ff87c06a7e0 2023-10-16T12:10:44.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:44.837+0000 7ff878ff9700 1 -- 172.21.15.16:0/3443162227 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff86c007310 con 0x7ff87c06a7e0 2023-10-16T12:10:44.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:44.837+0000 7ff8827fd700 1 -- 172.21.15.16:0/3443162227 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff87c0c5140 con 0x7ff87c06a7e0 2023-10-16T12:10:44.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:44.838+0000 7ff8827fd700 1 -- 172.21.15.16:0/3443162227 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff87c15cf70 con 0x7ff87c06a7e0 2023-10-16T12:10:44.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:44.839+0000 7ff878ff9700 1 -- 172.21.15.16:0/3443162227 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7ff86c007470 con 0x7ff87c06a7e0 2023-10-16T12:10:44.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:44.839+0000 7ff878ff9700 1 --2- 172.21.15.16:0/3443162227 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7ff85c06af00 0x7ff85c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:44.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:44.839+0000 7ff878ff9700 1 -- 172.21.15.16:0/3443162227 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7ff86c089450 con 0x7ff87c06a7e0 2023-10-16T12:10:44.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:44.839+0000 7ff87b7fe700 1 --2- 172.21.15.16:0/3443162227 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7ff85c06af00 0x7ff85c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:44.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:44.840+0000 7ff87b7fe700 1 --2- 172.21.15.16:0/3443162227 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7ff85c06af00 0x7ff85c06d3c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff864009b70 tx=0x7ff864008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:44.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:44.844+0000 7ff878ff9700 1 -- 172.21.15.16:0/3443162227 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff86c054210 con 0x7ff87c06a7e0 2023-10-16T12:10:45.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.092+0000 7ff8827fd700 1 -- 172.21.15.16:0/3443162227 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff87c06b170 con 0x7ff87c06a7e0 2023-10-16T12:10:45.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.093+0000 7ff878ff9700 1 -- 172.21.15.16:0/3443162227 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4909 (secure 0 0 0) 0x7ff87c06b170 con 0x7ff87c06a7e0 2023-10-16T12:10:45.096 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:10:45.096 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/2979266800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":2979266800},{"type":"v1","addr":"172.21.15.49:6834","nonce":2979266800}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4251,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/1398336801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1398336801},{"type":"v1","addr":"172.21.15.16:6836","nonce":1398336801}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4254,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3168304671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":3168304671},{"type":"v1","addr":"172.21.15.16:6837","nonce":3168304671}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:10:39.030243+0000","modified":"2023-10-16T12:10:44.080297+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4269":{"gid":4269,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.49:6835/2892536404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":2892536404},{"type":"v1","addr":"172.21.15.49:6835","nonce":2892536404}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2023-10-16T12:10:45.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.095+0000 7ff8827fd700 1 -- 172.21.15.16:0/3443162227 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7ff85c06af00 msgr2=0x7ff85c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:45.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.095+0000 7ff8827fd700 1 --2- 172.21.15.16:0/3443162227 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7ff85c06af00 0x7ff85c06d3c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff864009b70 tx=0x7ff864008040 comp rx=0 tx=0).stop 2023-10-16T12:10:45.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.095+0000 7ff8827fd700 1 -- 172.21.15.16:0/3443162227 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff87c06a7e0 msgr2=0x7ff87c156ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:45.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.095+0000 7ff8827fd700 1 --2- 172.21.15.16:0/3443162227 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff87c06a7e0 0x7ff87c156ad0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff86c00c550 tx=0x7ff86c00c910 comp rx=0 tx=0).stop 2023-10-16T12:10:45.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.095+0000 7ff8827fd700 1 -- 172.21.15.16:0/3443162227 shutdown_connections 2023-10-16T12:10:45.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.095+0000 7ff8827fd700 1 --2- 172.21.15.16:0/3443162227 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7ff85c06af00 0x7ff85c06d3c0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:45.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.095+0000 7ff8827fd700 1 --2- 172.21.15.16:0/3443162227 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff87c15b280 0x7ff87c15d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:45.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.095+0000 7ff8827fd700 1 --2- 172.21.15.16:0/3443162227 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff87c06a7e0 0x7ff87c156ad0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:45.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.095+0000 7ff8827fd700 1 --2- 172.21.15.16:0/3443162227 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff87c0cc8a0 0x7ff87c0c4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:45.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.095+0000 7ff8827fd700 1 -- 172.21.15.16:0/3443162227 >> 172.21.15.16:0/3443162227 conn(0x7ff87c0b1fd0 msgr2=0x7ff87c0d0270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:45.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.095+0000 7ff8827fd700 1 -- 172.21.15.16:0/3443162227 shutdown_connections 2023-10-16T12:10:45.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.095+0000 7ff8827fd700 1 -- 172.21.15.16:0/3443162227 wait complete. 2023-10-16T12:10:45.100 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 7 2023-10-16T12:10:45.110 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:10:45.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.232+0000 7f5b65219700 1 Processor -- start 2023-10-16T12:10:45.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.232+0000 7f5b65219700 1 -- start start 2023-10-16T12:10:45.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.233+0000 7f5b65219700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5b6015d230 0x7f5b600b5a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:45.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.233+0000 7f5b65219700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b6015bc80 con 0x7f5b600cf880 2023-10-16T12:10:45.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.233+0000 7f5b65219700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b6015b070 con 0x7f5b600cf4a0 2023-10-16T12:10:45.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.233+0000 7f5b65219700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b6015b350 con 0x7f5b6015d230 2023-10-16T12:10:45.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.233+0000 7f5b5f59e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5b6015d230 0x7f5b600b5a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:45.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.233+0000 7f5b5e59c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f5b600cf880 0x7f5b600b52f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50552/0 (socket says 172.21.15.16:50552) 2023-10-16T12:10:45.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.233+0000 7f5b5e59c700 1 -- 172.21.15.16:0/1930642028 learned_addr learned my addr 172.21.15.16:0/1930642028 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:10:45.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.233+0000 7f5b5f59e700 1 -- 172.21.15.16:0/1930642028 >> v1:172.21.15.49:6789/0 conn(0x7f5b600cf4a0 legacy=0x7f5b600726e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:10:45.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.233+0000 7f5b5f59e700 1 -- 172.21.15.16:0/1930642028 >> v1:172.21.15.16:6789/0 conn(0x7f5b600cf880 legacy=0x7f5b600b52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:45.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.233+0000 7f5b5f59e700 1 -- 172.21.15.16:0/1930642028 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b6015b630 con 0x7f5b6015d230 2023-10-16T12:10:45.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.233+0000 7f5b5f59e700 1 --2- 172.21.15.16:0/1930642028 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b6015d230 0x7f5b600b5a00 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5b58009d90 tx=0x7f5b5800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=905d5bd200fb6054 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:45.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.234+0000 7f5b5dd9b700 1 -- 172.21.15.16:0/1930642028 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b5800e070 con 0x7f5b6015d230 2023-10-16T12:10:45.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.234+0000 7f5b5dd9b700 1 -- 172.21.15.16:0/1930642028 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b5800bb90 con 0x7f5b6015d230 2023-10-16T12:10:45.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.234+0000 7f5b5dd9b700 1 -- 172.21.15.16:0/1930642028 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b58003a20 con 0x7f5b6015d230 2023-10-16T12:10:45.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.234+0000 7f5b65219700 1 -- 172.21.15.16:0/1930642028 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b6015d230 msgr2=0x7f5b600b5a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:45.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.234+0000 7f5b65219700 1 --2- 172.21.15.16:0/1930642028 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b6015d230 0x7f5b600b5a00 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5b58009d90 tx=0x7f5b5800b330 comp rx=0 tx=0).stop 2023-10-16T12:10:45.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.234+0000 7f5b65219700 1 -- 172.21.15.16:0/1930642028 shutdown_connections 2023-10-16T12:10:45.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.234+0000 7f5b65219700 1 --2- 172.21.15.16:0/1930642028 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b6015d230 0x7f5b600b5a00 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:45.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.234+0000 7f5b65219700 1 -- 172.21.15.16:0/1930642028 >> 172.21.15.16:0/1930642028 conn(0x7f5b600b1fd0 msgr2=0x7f5b600b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:45.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.234+0000 7f5b65219700 1 -- 172.21.15.16:0/1930642028 shutdown_connections 2023-10-16T12:10:45.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.234+0000 7f5b65219700 1 -- 172.21.15.16:0/1930642028 wait complete. 2023-10-16T12:10:45.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.236+0000 7f5b65219700 1 Processor -- start 2023-10-16T12:10:45.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.236+0000 7f5b65219700 1 -- start start 2023-10-16T12:10:45.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.236+0000 7f5b65219700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5b600cf4a0 0x7f5b600c6970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:45.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.236+0000 7f5b65219700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5b600cf880 0x7f5b600c9d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:45.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.236+0000 7f5b65219700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b6015d230 0x7f5b6006aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:45.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.236+0000 7f5b65219700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b600c6eb0 con 0x7f5b6015d230 2023-10-16T12:10:45.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.236+0000 7f5b65219700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b600c4a60 con 0x7f5b600cf880 2023-10-16T12:10:45.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.236+0000 7f5b65219700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b600c44a0 con 0x7f5b600cf4a0 2023-10-16T12:10:45.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.237+0000 7f5b5f59e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b6015d230 0x7f5b6006aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:45.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.237+0000 7f5b5f59e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b6015d230 0x7f5b6006aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33054/0 (socket says 172.21.15.16:33054) 2023-10-16T12:10:45.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.237+0000 7f5b5f59e700 1 -- 172.21.15.16:0/2740473743 learned_addr learned my addr 172.21.15.16:0/2740473743 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:10:45.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.237+0000 7f5b5e59c700 1 --2- 172.21.15.16:0/2740473743 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5b600cf880 0x7f5b600c9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:45.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.237+0000 7f5b5ed9d700 1 --2- 172.21.15.16:0/2740473743 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5b600cf4a0 0x7f5b600c6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:45.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.237+0000 7f5b5f59e700 1 -- 172.21.15.16:0/2740473743 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5b600cf4a0 msgr2=0x7f5b600c6970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:45.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.237+0000 7f5b5f59e700 1 --2- 172.21.15.16:0/2740473743 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5b600cf4a0 0x7f5b600c6970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:45.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.237+0000 7f5b5f59e700 1 -- 172.21.15.16:0/2740473743 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5b600cf880 msgr2=0x7f5b600c9d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:45.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.237+0000 7f5b5f59e700 1 --2- 172.21.15.16:0/2740473743 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5b600cf880 0x7f5b600c9d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:45.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.237+0000 7f5b5f59e700 1 -- 172.21.15.16:0/2740473743 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b58009a70 con 0x7f5b6015d230 2023-10-16T12:10:45.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.237+0000 7f5b5e59c700 1 --2- 172.21.15.16:0/2740473743 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5b600cf880 0x7f5b600c9d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:10:45.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.237+0000 7f5b5f59e700 1 --2- 172.21.15.16:0/2740473743 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b6015d230 0x7f5b6006aa50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5b600cdfc0 tx=0x7f5b58004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:45.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.237+0000 7f5b477fe700 1 -- 172.21.15.16:0/2740473743 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b58012630 con 0x7f5b6015d230 2023-10-16T12:10:45.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.237+0000 7f5b477fe700 1 -- 172.21.15.16:0/2740473743 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b580127b0 con 0x7f5b6015d230 2023-10-16T12:10:45.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.237+0000 7f5b65219700 1 -- 172.21.15.16:0/2740473743 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b600c4780 con 0x7f5b6015d230 2023-10-16T12:10:45.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.237+0000 7f5b477fe700 1 -- 172.21.15.16:0/2740473743 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b58012b60 con 0x7f5b6015d230 2023-10-16T12:10:45.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.237+0000 7f5b65219700 1 -- 172.21.15.16:0/2740473743 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b600c6450 con 0x7f5b6015d230 2023-10-16T12:10:45.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.238+0000 7f5b65219700 1 -- 172.21.15.16:0/2740473743 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b600c5fd0 con 0x7f5b6015d230 2023-10-16T12:10:45.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.239+0000 7f5b477fe700 1 -- 172.21.15.16:0/2740473743 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f5b58012cc0 con 0x7f5b6015d230 2023-10-16T12:10:45.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.239+0000 7f5b477fe700 1 --2- 172.21.15.16:0/2740473743 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f5b4c06af00 0x7f5b4c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:45.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.239+0000 7f5b477fe700 1 -- 172.21.15.16:0/2740473743 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f5b5808bad0 con 0x7f5b6015d230 2023-10-16T12:10:45.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.239+0000 7f5b5ed9d700 1 --2- 172.21.15.16:0/2740473743 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f5b4c06af00 0x7f5b4c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:45.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.240+0000 7f5b5ed9d700 1 --2- 172.21.15.16:0/2740473743 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f5b4c06af00 0x7f5b4c06d3c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5b54003e60 tx=0x7f5b54009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:45.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.244+0000 7f5b477fe700 1 -- 172.21.15.16:0/2740473743 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5b600c5fd0 con 0x7f5b6015d230 2023-10-16T12:10:45.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.451+0000 7f5b65219700 1 -- 172.21.15.16:0/2740473743 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5b600c5fd0 con 0x7f5b6015d230 2023-10-16T12:10:45.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.452+0000 7f5b477fe700 1 -- 172.21.15.16:0/2740473743 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18238 (secure 0 0 0) 0x7f5b600c5fd0 con 0x7f5b6015d230 2023-10-16T12:10:45.455 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:10:45.456 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":25,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:10:44.080141+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:10:36.935773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T12:10:38.004870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-16T12:10:41.672025+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:10:45.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.454+0000 7f5b65219700 1 -- 172.21.15.16:0/2740473743 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f5b4c06af00 msgr2=0x7f5b4c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:45.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.454+0000 7f5b65219700 1 --2- 172.21.15.16:0/2740473743 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f5b4c06af00 0x7f5b4c06d3c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5b54003e60 tx=0x7f5b54009040 comp rx=0 tx=0).stop 2023-10-16T12:10:45.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.454+0000 7f5b65219700 1 -- 172.21.15.16:0/2740473743 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b6015d230 msgr2=0x7f5b6006aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:45.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.454+0000 7f5b65219700 1 --2- 172.21.15.16:0/2740473743 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b6015d230 0x7f5b6006aa50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5b600cdfc0 tx=0x7f5b58004ad0 comp rx=0 tx=0).stop 2023-10-16T12:10:45.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.454+0000 7f5b65219700 1 -- 172.21.15.16:0/2740473743 shutdown_connections 2023-10-16T12:10:45.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.454+0000 7f5b65219700 1 --2- 172.21.15.16:0/2740473743 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f5b4c06af00 0x7f5b4c06d3c0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:45.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.454+0000 7f5b65219700 1 --2- 172.21.15.16:0/2740473743 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b6015d230 0x7f5b6006aa50 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:45.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.454+0000 7f5b65219700 1 --2- 172.21.15.16:0/2740473743 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5b600cf4a0 0x7f5b600c6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:45.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.454+0000 7f5b65219700 1 --2- 172.21.15.16:0/2740473743 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5b600cf880 0x7f5b600c9d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:45.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.454+0000 7f5b65219700 1 -- 172.21.15.16:0/2740473743 >> 172.21.15.16:0/2740473743 conn(0x7f5b600b1fd0 msgr2=0x7f5b600cdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:45.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.454+0000 7f5b65219700 1 -- 172.21.15.16:0/2740473743 shutdown_connections 2023-10-16T12:10:45.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.455+0000 7f5b65219700 1 -- 172.21.15.16:0/2740473743 wait complete. 2023-10-16T12:10:45.470 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:10:45.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.592+0000 7facbd2c1700 1 Processor -- start 2023-10-16T12:10:45.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.592+0000 7facbd2c1700 1 -- start start 2023-10-16T12:10:45.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.592+0000 7facbd2c1700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7facb815d8d0 0x7facb80d1c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:45.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.592+0000 7facbd2c1700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7facb80d0750 0x7facb806c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:45.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.592+0000 7facbd2c1700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facb806ba00 con 0x7facb80d41c0 2023-10-16T12:10:45.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.592+0000 7facbd2c1700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facb80d2150 con 0x7facb815d8d0 2023-10-16T12:10:45.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.592+0000 7facbd2c1700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facb80d26d0 con 0x7facb80d0750 2023-10-16T12:10:45.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.593+0000 7facb6ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7facb815d8d0 0x7facb80d1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:45.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.593+0000 7facb6ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7facb815d8d0 0x7facb80d1c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33058/0 (socket says 172.21.15.16:33058) 2023-10-16T12:10:45.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.593+0000 7facb6ffd700 1 -- 172.21.15.16:0/68652756 learned_addr learned my addr 172.21.15.16:0/68652756 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:10:45.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.593+0000 7facb77fe700 1 --2- 172.21.15.16:0/68652756 >> v2:172.21.15.49:3300/0 conn(0x7facb80d0750 0x7facb806c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:45.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.593+0000 7facb6ffd700 1 -- 172.21.15.16:0/68652756 >> v2:172.21.15.49:3300/0 conn(0x7facb80d0750 msgr2=0x7facb806c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:10:45.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.593+0000 7facb6ffd700 1 --2- 172.21.15.16:0/68652756 >> v2:172.21.15.49:3300/0 conn(0x7facb80d0750 0x7facb806c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:45.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.593+0000 7facb6ffd700 1 -- 172.21.15.16:0/68652756 >> v1:172.21.15.49:6789/0 conn(0x7facb80d41c0 legacy=0x7facb80d0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:10:45.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.593+0000 7facb6ffd700 1 -- 172.21.15.16:0/68652756 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facb80d23f0 con 0x7facb815d8d0 2023-10-16T12:10:45.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.593+0000 7facb6ffd700 1 --2- 172.21.15.16:0/68652756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facb815d8d0 0x7facb80d1c10 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7facb80d4640 tx=0x7facac009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62ca833e2fb45780 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:45.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.593+0000 7facb5ffb700 1 -- 172.21.15.16:0/68652756 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facac004230 con 0x7facb815d8d0 2023-10-16T12:10:45.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.593+0000 7facb5ffb700 1 -- 172.21.15.16:0/68652756 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facac004390 con 0x7facb815d8d0 2023-10-16T12:10:45.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.593+0000 7facb5ffb700 1 -- 172.21.15.16:0/68652756 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facac0036a0 con 0x7facb815d8d0 2023-10-16T12:10:45.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.594+0000 7facbd2c1700 1 -- 172.21.15.16:0/68652756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facb815d8d0 msgr2=0x7facb80d1c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:45.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.594+0000 7facbd2c1700 1 --2- 172.21.15.16:0/68652756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facb815d8d0 0x7facb80d1c10 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7facb80d4640 tx=0x7facac009bd0 comp rx=0 tx=0).stop 2023-10-16T12:10:45.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.594+0000 7facbd2c1700 1 -- 172.21.15.16:0/68652756 shutdown_connections 2023-10-16T12:10:45.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.594+0000 7facbd2c1700 1 --2- 172.21.15.16:0/68652756 >> v2:172.21.15.49:3300/0 conn(0x7facb80d0750 0x7facb806c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:45.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.594+0000 7facbd2c1700 1 --2- 172.21.15.16:0/68652756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facb815d8d0 0x7facb80d1c10 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:45.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.594+0000 7facbd2c1700 1 -- 172.21.15.16:0/68652756 >> 172.21.15.16:0/68652756 conn(0x7facb80d3630 msgr2=0x7facb80d3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:45.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.594+0000 7facbd2c1700 1 -- 172.21.15.16:0/68652756 shutdown_connections 2023-10-16T12:10:45.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.594+0000 7facbd2c1700 1 -- 172.21.15.16:0/68652756 wait complete. 2023-10-16T12:10:45.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.595+0000 7facbd2c1700 1 Processor -- start 2023-10-16T12:10:45.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.595+0000 7facbd2c1700 1 -- start start 2023-10-16T12:10:45.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.596+0000 7facbd2c1700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7facb80d0750 0x7facb8152a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:45.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.596+0000 7facbd2c1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facb80d41c0 0x7facb809e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:45.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.596+0000 7facbd2c1700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7facb815d8d0 0x7facb80a06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:45.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.596+0000 7facbd2c1700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facb809e290 con 0x7facb80d41c0 2023-10-16T12:10:45.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.596+0000 7facbd2c1700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facb809d680 con 0x7facb80d0750 2023-10-16T12:10:45.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.596+0000 7facbd2c1700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facb809d960 con 0x7facb815d8d0 2023-10-16T12:10:45.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.596+0000 7facb67fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facb80d41c0 0x7facb809e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:45.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.596+0000 7facb67fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facb80d41c0 0x7facb809e720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33064/0 (socket says 172.21.15.16:33064) 2023-10-16T12:10:45.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.596+0000 7facb67fc700 1 -- 172.21.15.16:0/560286600 learned_addr learned my addr 172.21.15.16:0/560286600 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:10:45.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.596+0000 7facb6ffd700 1 --2- 172.21.15.16:0/560286600 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7facb80d0750 0x7facb8152a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:45.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.596+0000 7facb77fe700 1 --2- 172.21.15.16:0/560286600 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7facb815d8d0 0x7facb80a06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:45.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.596+0000 7facb67fc700 1 -- 172.21.15.16:0/560286600 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7facb815d8d0 msgr2=0x7facb80a06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:45.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.596+0000 7facb67fc700 1 --2- 172.21.15.16:0/560286600 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7facb815d8d0 0x7facb80a06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:45.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.596+0000 7facb67fc700 1 -- 172.21.15.16:0/560286600 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7facb80d0750 msgr2=0x7facb8152a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:45.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.596+0000 7facb67fc700 1 --2- 172.21.15.16:0/560286600 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7facb80d0750 0x7facb8152a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:45.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.596+0000 7facb67fc700 1 -- 172.21.15.16:0/560286600 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facac009550 con 0x7facb80d41c0 2023-10-16T12:10:45.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.596+0000 7facb67fc700 1 --2- 172.21.15.16:0/560286600 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facb80d41c0 0x7facb809e720 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7faca800c4d0 tx=0x7faca800c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:45.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.597+0000 7fac9f7fe700 1 -- 172.21.15.16:0/560286600 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faca8011790 con 0x7facb80d41c0 2023-10-16T12:10:45.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.597+0000 7fac9f7fe700 1 -- 172.21.15.16:0/560286600 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faca80118f0 con 0x7facb80d41c0 2023-10-16T12:10:45.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.597+0000 7facbd2c1700 1 -- 172.21.15.16:0/560286600 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facb809dc40 con 0x7facb80d41c0 2023-10-16T12:10:45.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.597+0000 7fac9f7fe700 1 -- 172.21.15.16:0/560286600 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faca8011ca0 con 0x7facb80d41c0 2023-10-16T12:10:45.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.597+0000 7facbd2c1700 1 -- 172.21.15.16:0/560286600 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facb8152f90 con 0x7facb80d41c0 2023-10-16T12:10:45.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.598+0000 7facbd2c1700 1 -- 172.21.15.16:0/560286600 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facb809e440 con 0x7facb80d41c0 2023-10-16T12:10:45.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.598+0000 7fac9f7fe700 1 -- 172.21.15.16:0/560286600 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7faca801c460 con 0x7facb80d41c0 2023-10-16T12:10:45.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.599+0000 7fac9f7fe700 1 --2- 172.21.15.16:0/560286600 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7faca406af00 0x7faca406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:45.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.599+0000 7fac9f7fe700 1 -- 172.21.15.16:0/560286600 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7faca80897a0 con 0x7facb80d41c0 2023-10-16T12:10:45.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.599+0000 7facb6ffd700 1 --2- 172.21.15.16:0/560286600 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7faca406af00 0x7faca406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:45.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.599+0000 7facb6ffd700 1 --2- 172.21.15.16:0/560286600 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7faca406af00 0x7faca406d3c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7facac009130 tx=0x7facac008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:45.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.603+0000 7fac9f7fe700 1 -- 172.21.15.16:0/560286600 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7facb809e440 con 0x7facb80d41c0 2023-10-16T12:10:45.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.808+0000 7facbd2c1700 1 -- 172.21.15.16:0/560286600 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7facb806bf50 con 0x7facb80d41c0 2023-10-16T12:10:45.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.809+0000 7fac9f7fe700 1 -- 172.21.15.16:0/560286600 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18238 (secure 0 0 0) 0x7faca8054430 con 0x7facb80d41c0 2023-10-16T12:10:45.812 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:10:45.813 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":25,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:10:44.080141+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:10:36.935773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T12:10:38.004870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-16T12:10:41.672025+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:10:45.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.811+0000 7facbd2c1700 1 -- 172.21.15.16:0/560286600 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7faca406af00 msgr2=0x7faca406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:45.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.811+0000 7facbd2c1700 1 --2- 172.21.15.16:0/560286600 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7faca406af00 0x7faca406d3c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7facac009130 tx=0x7facac008030 comp rx=0 tx=0).stop 2023-10-16T12:10:45.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.811+0000 7facbd2c1700 1 -- 172.21.15.16:0/560286600 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facb80d41c0 msgr2=0x7facb809e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:45.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.811+0000 7facbd2c1700 1 --2- 172.21.15.16:0/560286600 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facb80d41c0 0x7facb809e720 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7faca800c4d0 tx=0x7faca800c7e0 comp rx=0 tx=0).stop 2023-10-16T12:10:45.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.811+0000 7facbd2c1700 1 -- 172.21.15.16:0/560286600 shutdown_connections 2023-10-16T12:10:45.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.811+0000 7facbd2c1700 1 --2- 172.21.15.16:0/560286600 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7faca406af00 0x7faca406d3c0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:45.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.811+0000 7facbd2c1700 1 --2- 172.21.15.16:0/560286600 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7facb815d8d0 0x7facb80a06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:45.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.811+0000 7facbd2c1700 1 --2- 172.21.15.16:0/560286600 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7facb80d0750 0x7facb8152a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:45.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.811+0000 7facbd2c1700 1 --2- 172.21.15.16:0/560286600 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facb80d41c0 0x7facb809e720 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:45.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.811+0000 7facbd2c1700 1 -- 172.21.15.16:0/560286600 >> 172.21.15.16:0/560286600 conn(0x7facb80d3630 msgr2=0x7facb80b1c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:45.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.811+0000 7facbd2c1700 1 -- 172.21.15.16:0/560286600 shutdown_connections 2023-10-16T12:10:45.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.811+0000 7facbd2c1700 1 -- 172.21.15.16:0/560286600 wait complete. 2023-10-16T12:10:45.841 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:10:45.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.963+0000 7f9e9924b700 1 Processor -- start 2023-10-16T12:10:45.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.963+0000 7f9e9924b700 1 -- start start 2023-10-16T12:10:45.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.964+0000 7f9e9924b700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f9e940cf4a0 0x7f9e940726e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:45.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.964+0000 7f9e9924b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9e940b5a00 0x7f9e9415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:45.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.964+0000 7f9e9924b700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e9415bc80 con 0x7f9e940cf880 2023-10-16T12:10:45.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.964+0000 7f9e9924b700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e940b5e40 con 0x7f9e940cf4a0 2023-10-16T12:10:45.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.964+0000 7f9e9924b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e94072c20 con 0x7f9e940b5a00 2023-10-16T12:10:45.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.964+0000 7f9e9359e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9e940b5a00 0x7f9e9415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:45.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.964+0000 7f9e92d9d700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f9e940cf4a0 0x7f9e940726e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:45.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.964+0000 7f9e9259c700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7f9e940cf880 0x7f9e940b52f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:57758/0 (socket says 172.21.15.16:57758) 2023-10-16T12:10:45.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.964+0000 7f9e92d9d700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f9e940cf4a0 0x7f9e940726e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:60362/0 (socket says 172.21.15.16:60362) 2023-10-16T12:10:45.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.964+0000 7f9e9259c700 1 -- 172.21.15.16:0/468703615 learned_addr learned my addr 172.21.15.16:0/468703615 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:10:45.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.964+0000 7f9e9359e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9e940b5a00 0x7f9e9415a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:49562/0 (socket says 172.21.15.16:49562) 2023-10-16T12:10:45.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.965+0000 7f9e92d9d700 1 -- 172.21.15.16:0/468703615 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9e940b5a00 msgr2=0x7f9e9415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:10:45.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.965+0000 7f9e92d9d700 1 --2- 172.21.15.16:0/468703615 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9e940b5a00 0x7f9e9415a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:45.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.965+0000 7f9e92d9d700 1 -- 172.21.15.16:0/468703615 >> v1:172.21.15.49:6789/0 conn(0x7f9e940cf880 legacy=0x7f9e940b52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:45.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.965+0000 7f9e91d9b700 1 -- 172.21.15.16:0/468703615 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1249716318 0 0) 0x7f9e9415bc80 con 0x7f9e940cf880 2023-10-16T12:10:45.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.965+0000 7f9e92d9d700 1 -- 172.21.15.16:0/468703615 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e9415b070 con 0x7f9e940cf4a0 2023-10-16T12:10:45.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.965+0000 7f9e9359e700 1 --2- 172.21.15.16:0/468703615 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9e940b5a00 0x7f9e9415a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:10:45.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.965+0000 7f9e92d9d700 1 --2- 172.21.15.16:0/468703615 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9e940cf4a0 0x7f9e940726e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f9e88009840 tx=0x7f9e88009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5732e2603ec2402 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:45.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.965+0000 7f9e91d9b700 1 -- 172.21.15.16:0/468703615 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e88011830 con 0x7f9e940cf4a0 2023-10-16T12:10:45.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.965+0000 7f9e91d9b700 1 -- 172.21.15.16:0/468703615 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e88011990 con 0x7f9e940cf4a0 2023-10-16T12:10:45.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.965+0000 7f9e91d9b700 1 -- 172.21.15.16:0/468703615 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e88011d20 con 0x7f9e940cf4a0 2023-10-16T12:10:45.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.966+0000 7f9e9924b700 1 -- 172.21.15.16:0/468703615 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9e940cf4a0 msgr2=0x7f9e940726e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:45.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.966+0000 7f9e9924b700 1 --2- 172.21.15.16:0/468703615 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9e940cf4a0 0x7f9e940726e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f9e88009840 tx=0x7f9e88009b50 comp rx=0 tx=0).stop 2023-10-16T12:10:45.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.966+0000 7f9e9924b700 1 -- 172.21.15.16:0/468703615 shutdown_connections 2023-10-16T12:10:45.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.966+0000 7f9e9924b700 1 --2- 172.21.15.16:0/468703615 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9e940b5a00 0x7f9e9415a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:45.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.966+0000 7f9e9924b700 1 --2- 172.21.15.16:0/468703615 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9e940cf4a0 0x7f9e940726e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:45.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.966+0000 7f9e9924b700 1 -- 172.21.15.16:0/468703615 >> 172.21.15.16:0/468703615 conn(0x7f9e940b1fd0 msgr2=0x7f9e940b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:45.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.966+0000 7f9e9924b700 1 -- 172.21.15.16:0/468703615 shutdown_connections 2023-10-16T12:10:45.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.966+0000 7f9e9924b700 1 -- 172.21.15.16:0/468703615 wait complete. 2023-10-16T12:10:45.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.967+0000 7f9e9924b700 1 Processor -- start 2023-10-16T12:10:45.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.967+0000 7f9e9924b700 1 -- start start 2023-10-16T12:10:45.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.968+0000 7f9e9924b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9e940b5a00 0x7f9e941519f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:45.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.968+0000 7f9e9924b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9e940cf4a0 0x7f9e940c8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:45.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.968+0000 7f9e9924b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e940cf880 0x7f9e940c9040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:45.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.968+0000 7f9e9924b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e94158d20 con 0x7f9e940cf880 2023-10-16T12:10:45.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.968+0000 7f9e9924b700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e94158f70 con 0x7f9e940b5a00 2023-10-16T12:10:45.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.968+0000 7f9e9924b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e94159250 con 0x7f9e940cf4a0 2023-10-16T12:10:45.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.968+0000 7f9e9359e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e940cf880 0x7f9e940c9040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:45.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.968+0000 7f9e9359e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e940cf880 0x7f9e940c9040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33078/0 (socket says 172.21.15.16:33078) 2023-10-16T12:10:45.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.968+0000 7f9e9359e700 1 -- 172.21.15.16:0/1534147863 learned_addr learned my addr 172.21.15.16:0/1534147863 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:10:45.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.968+0000 7f9e92d9d700 1 --2- 172.21.15.16:0/1534147863 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9e940b5a00 0x7f9e941519f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:45.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.968+0000 7f9e9259c700 1 --2- 172.21.15.16:0/1534147863 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9e940cf4a0 0x7f9e940c8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:45.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.968+0000 7f9e9359e700 1 -- 172.21.15.16:0/1534147863 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9e940cf4a0 msgr2=0x7f9e940c8b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:45.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.968+0000 7f9e9359e700 1 --2- 172.21.15.16:0/1534147863 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9e940cf4a0 0x7f9e940c8b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:45.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.968+0000 7f9e9359e700 1 -- 172.21.15.16:0/1534147863 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9e940b5a00 msgr2=0x7f9e941519f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:45.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.968+0000 7f9e9359e700 1 --2- 172.21.15.16:0/1534147863 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9e940b5a00 0x7f9e941519f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:45.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.968+0000 7f9e9359e700 1 -- 172.21.15.16:0/1534147863 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e8c009a70 con 0x7f9e940cf880 2023-10-16T12:10:45.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.969+0000 7f9e9359e700 1 --2- 172.21.15.16:0/1534147863 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e940cf880 0x7f9e940c9040 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c00efc0 tx=0x7f9e8c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:45.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.969+0000 7f9e7b7fe700 1 -- 172.21.15.16:0/1534147863 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e8c00ca90 con 0x7f9e940cf880 2023-10-16T12:10:45.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.969+0000 7f9e7b7fe700 1 -- 172.21.15.16:0/1534147863 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e8c00f040 con 0x7f9e940cf880 2023-10-16T12:10:45.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.969+0000 7f9e7b7fe700 1 -- 172.21.15.16:0/1534147863 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e8c00ce10 con 0x7f9e940cf880 2023-10-16T12:10:45.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.969+0000 7f9e9924b700 1 -- 172.21.15.16:0/1534147863 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e880094a0 con 0x7f9e940cf880 2023-10-16T12:10:45.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.969+0000 7f9e9924b700 1 -- 172.21.15.16:0/1534147863 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e94159810 con 0x7f9e940cf880 2023-10-16T12:10:45.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.970+0000 7f9e9924b700 1 -- 172.21.15.16:0/1534147863 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e94159a30 con 0x7f9e940cf880 2023-10-16T12:10:45.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.970+0000 7f9e7b7fe700 1 -- 172.21.15.16:0/1534147863 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f9e8c014480 con 0x7f9e940cf880 2023-10-16T12:10:45.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.971+0000 7f9e7b7fe700 1 --2- 172.21.15.16:0/1534147863 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f9e8006af00 0x7f9e8006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:45.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.971+0000 7f9e7b7fe700 1 -- 172.21.15.16:0/1534147863 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f9e8c089f20 con 0x7f9e940cf880 2023-10-16T12:10:45.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.971+0000 7f9e92d9d700 1 --2- 172.21.15.16:0/1534147863 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f9e8006af00 0x7f9e8006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:45.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.971+0000 7f9e92d9d700 1 --2- 172.21.15.16:0/1534147863 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f9e8006af00 0x7f9e8006d3c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9e88000c00 tx=0x7f9e88008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:45.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:45.976+0000 7f9e7b7fe700 1 -- 172.21.15.16:0/1534147863 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9e8c054d70 con 0x7f9e940cf880 2023-10-16T12:10:46.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.226+0000 7f9e9924b700 1 -- 172.21.15.16:0/1534147863 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e9404ebb0 con 0x7f9e940cf880 2023-10-16T12:10:46.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.226+0000 7f9e7b7fe700 1 -- 172.21.15.16:0/1534147863 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4909 (secure 0 0 0) 0x7f9e8c009d70 con 0x7f9e940cf880 2023-10-16T12:10:46.230 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:10:46.230 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/2979266800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":2979266800},{"type":"v1","addr":"172.21.15.49:6834","nonce":2979266800}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4251,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/1398336801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1398336801},{"type":"v1","addr":"172.21.15.16:6836","nonce":1398336801}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4254,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3168304671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":3168304671},{"type":"v1","addr":"172.21.15.16:6837","nonce":3168304671}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:10:39.030243+0000","modified":"2023-10-16T12:10:44.080297+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4269":{"gid":4269,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.49:6835/2892536404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":2892536404},{"type":"v1","addr":"172.21.15.49:6835","nonce":2892536404}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2023-10-16T12:10:46.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.229+0000 7f9e9924b700 1 -- 172.21.15.16:0/1534147863 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f9e8006af00 msgr2=0x7f9e8006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:46.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.229+0000 7f9e9924b700 1 --2- 172.21.15.16:0/1534147863 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f9e8006af00 0x7f9e8006d3c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9e88000c00 tx=0x7f9e88008030 comp rx=0 tx=0).stop 2023-10-16T12:10:46.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.229+0000 7f9e9924b700 1 -- 172.21.15.16:0/1534147863 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e940cf880 msgr2=0x7f9e940c9040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:46.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.229+0000 7f9e9924b700 1 --2- 172.21.15.16:0/1534147863 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e940cf880 0x7f9e940c9040 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c00efc0 tx=0x7f9e8c00c3a0 comp rx=0 tx=0).stop 2023-10-16T12:10:46.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.229+0000 7f9e9924b700 1 -- 172.21.15.16:0/1534147863 shutdown_connections 2023-10-16T12:10:46.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.229+0000 7f9e9924b700 1 --2- 172.21.15.16:0/1534147863 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f9e8006af00 0x7f9e8006d3c0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:46.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.229+0000 7f9e9924b700 1 --2- 172.21.15.16:0/1534147863 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e940cf880 0x7f9e940c9040 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:46.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.229+0000 7f9e9924b700 1 --2- 172.21.15.16:0/1534147863 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9e940b5a00 0x7f9e941519f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:46.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.229+0000 7f9e9924b700 1 --2- 172.21.15.16:0/1534147863 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9e940cf4a0 0x7f9e940c8b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:46.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.229+0000 7f9e9924b700 1 -- 172.21.15.16:0/1534147863 >> 172.21.15.16:0/1534147863 conn(0x7f9e940b1fd0 msgr2=0x7f9e9415d2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:46.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.229+0000 7f9e9924b700 1 -- 172.21.15.16:0/1534147863 shutdown_connections 2023-10-16T12:10:46.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.229+0000 7f9e9924b700 1 -- 172.21.15.16:0/1534147863 wait complete. 2023-10-16T12:10:46.235 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 7 2023-10-16T12:10:46.244 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:10:46.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.367+0000 7f8064b2d700 1 Processor -- start 2023-10-16T12:10:46.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.367+0000 7f8064b2d700 1 -- start start 2023-10-16T12:10:46.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.367+0000 7f8064b2d700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f806015b630 0x7f80600cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:46.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.367+0000 7f8064b2d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f806015ba10 0x7f80600cdde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:46.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.368+0000 7f8064b2d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f80600ce320 0x7f80600cfca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:46.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.368+0000 7f8064b2d700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80600cdb20 con 0x7f80600ce320 2023-10-16T12:10:46.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.368+0000 7f8064b2d700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80600ce770 con 0x7f806015b630 2023-10-16T12:10:46.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.368+0000 7f8064b2d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80600cc2a0 con 0x7f806015ba10 2023-10-16T12:10:46.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.368+0000 7f805ed9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f80600ce320 0x7f80600cfca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:46.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.368+0000 7f805ed9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f80600ce320 0x7f80600cfca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33086/0 (socket says 172.21.15.16:33086) 2023-10-16T12:10:46.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.368+0000 7f805ed9d700 1 -- 172.21.15.16:0/419426973 learned_addr learned my addr 172.21.15.16:0/419426973 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:10:46.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.368+0000 7f805dd9b700 1 --2- 172.21.15.16:0/419426973 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f806015ba10 0x7f80600cdde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:46.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.368+0000 7f805e59c700 1 --2- 172.21.15.16:0/419426973 >> v2:172.21.15.49:3300/0 conn(0x7f806015b630 0x7f80600cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:46.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.368+0000 7f805ed9d700 1 -- 172.21.15.16:0/419426973 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f806015ba10 msgr2=0x7f80600cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:10:46.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.368+0000 7f805ed9d700 1 --2- 172.21.15.16:0/419426973 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f806015ba10 0x7f80600cdde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:46.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.368+0000 7f805ed9d700 1 -- 172.21.15.16:0/419426973 >> v2:172.21.15.49:3300/0 conn(0x7f806015b630 msgr2=0x7f80600cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:10:46.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.368+0000 7f805ed9d700 1 --2- 172.21.15.16:0/419426973 >> v2:172.21.15.49:3300/0 conn(0x7f806015b630 0x7f80600cd5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:46.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.368+0000 7f805ed9d700 1 -- 172.21.15.16:0/419426973 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80600cc540 con 0x7f80600ce320 2023-10-16T12:10:46.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.368+0000 7f805e59c700 1 --2- 172.21.15.16:0/419426973 >> v2:172.21.15.49:3300/0 conn(0x7f806015b630 0x7f80600cd5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:10:46.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.368+0000 7f805ed9d700 1 --2- 172.21.15.16:0/419426973 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80600ce320 0x7f80600cfca0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f80600cefb0 tx=0x7f805800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a63c703efaeb66eb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:46.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.368+0000 7f805d59a700 1 -- 172.21.15.16:0/419426973 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f805800e070 con 0x7f80600ce320 2023-10-16T12:10:46.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.368+0000 7f805d59a700 1 -- 172.21.15.16:0/419426973 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f805800b910 con 0x7f80600ce320 2023-10-16T12:10:46.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.368+0000 7f805d59a700 1 -- 172.21.15.16:0/419426973 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f805800bcc0 con 0x7f80600ce320 2023-10-16T12:10:46.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.369+0000 7f8064b2d700 1 -- 172.21.15.16:0/419426973 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80600ce320 msgr2=0x7f80600cfca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:46.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.369+0000 7f8064b2d700 1 --2- 172.21.15.16:0/419426973 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80600ce320 0x7f80600cfca0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f80600cefb0 tx=0x7f805800b330 comp rx=0 tx=0).stop 2023-10-16T12:10:46.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.369+0000 7f8064b2d700 1 -- 172.21.15.16:0/419426973 shutdown_connections 2023-10-16T12:10:46.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.369+0000 7f8064b2d700 1 --2- 172.21.15.16:0/419426973 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80600ce320 0x7f80600cfca0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:46.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.369+0000 7f8064b2d700 1 --2- 172.21.15.16:0/419426973 >> v2:172.21.15.49:3300/0 conn(0x7f806015b630 0x7f80600cd5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:46.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.369+0000 7f8064b2d700 1 --2- 172.21.15.16:0/419426973 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f806015ba10 0x7f80600cdde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:46.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.369+0000 7f8064b2d700 1 -- 172.21.15.16:0/419426973 >> 172.21.15.16:0/419426973 conn(0x7f80600d14b0 msgr2=0x7f80600d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:46.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.369+0000 7f8064b2d700 1 -- 172.21.15.16:0/419426973 shutdown_connections 2023-10-16T12:10:46.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.369+0000 7f8064b2d700 1 -- 172.21.15.16:0/419426973 wait complete. 2023-10-16T12:10:46.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.370+0000 7f8064b2d700 1 Processor -- start 2023-10-16T12:10:46.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.370+0000 7f8064b2d700 1 -- start start 2023-10-16T12:10:46.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.371+0000 7f8064b2d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f80600ce320 0x7f80600c7240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:46.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.371+0000 7f8064b2d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f806015b630 0x7f806006ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:46.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.371+0000 7f8064b2d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f806015ba10 0x7f80600cabb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:46.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.371+0000 7f8064b2d700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f806006c5e0 con 0x7f806015b630 2023-10-16T12:10:46.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.371+0000 7f8064b2d700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f806006c2b0 con 0x7f806015ba10 2023-10-16T12:10:46.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.371+0000 7f8064b2d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80600c9990 con 0x7f80600ce320 2023-10-16T12:10:46.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.372+0000 7f805dd9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f806015b630 0x7f806006ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:46.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.372+0000 7f805dd9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f806015b630 0x7f806006ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33090/0 (socket says 172.21.15.16:33090) 2023-10-16T12:10:46.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.372+0000 7f805dd9b700 1 -- 172.21.15.16:0/629053574 learned_addr learned my addr 172.21.15.16:0/629053574 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:10:46.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.372+0000 7f805e59c700 1 --2- 172.21.15.16:0/629053574 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f80600ce320 0x7f80600c7240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:46.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.372+0000 7f805ed9d700 1 --2- 172.21.15.16:0/629053574 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f806015ba10 0x7f80600cabb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:46.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.372+0000 7f805dd9b700 1 -- 172.21.15.16:0/629053574 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f80600ce320 msgr2=0x7f80600c7240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:46.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.372+0000 7f805dd9b700 1 --2- 172.21.15.16:0/629053574 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f80600ce320 0x7f80600c7240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:46.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.372+0000 7f805dd9b700 1 -- 172.21.15.16:0/629053574 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f806015ba10 msgr2=0x7f80600cabb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:46.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.372+0000 7f805dd9b700 1 --2- 172.21.15.16:0/629053574 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f806015ba10 0x7f80600cabb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:46.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.372+0000 7f805dd9b700 1 -- 172.21.15.16:0/629053574 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8058009b30 con 0x7f806015b630 2023-10-16T12:10:46.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.372+0000 7f805dd9b700 1 --2- 172.21.15.16:0/629053574 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f806015b630 0x7f806006ca70 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f805400e9b0 tx=0x7f805400ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:46.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.372+0000 7f804effd700 1 -- 172.21.15.16:0/629053574 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8054009710 con 0x7f806015b630 2023-10-16T12:10:46.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.372+0000 7f804effd700 1 -- 172.21.15.16:0/629053574 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f805400c850 con 0x7f806015b630 2023-10-16T12:10:46.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.372+0000 7f8064b2d700 1 -- 172.21.15.16:0/629053574 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80600c9c70 con 0x7f806015b630 2023-10-16T12:10:46.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.372+0000 7f804effd700 1 -- 172.21.15.16:0/629053574 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f805400cbc0 con 0x7f806015b630 2023-10-16T12:10:46.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.372+0000 7f8064b2d700 1 -- 172.21.15.16:0/629053574 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f806006cfb0 con 0x7f806015b630 2023-10-16T12:10:46.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.373+0000 7f8064b2d700 1 -- 172.21.15.16:0/629053574 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f806006d1d0 con 0x7f806015b630 2023-10-16T12:10:46.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.374+0000 7f804effd700 1 -- 172.21.15.16:0/629053574 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f805400cd20 con 0x7f806015b630 2023-10-16T12:10:46.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.374+0000 7f804effd700 1 --2- 172.21.15.16:0/629053574 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f804006af00 0x7f804006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:46.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.374+0000 7f804effd700 1 -- 172.21.15.16:0/629053574 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f80540899d0 con 0x7f806015b630 2023-10-16T12:10:46.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.374+0000 7f805e59c700 1 --2- 172.21.15.16:0/629053574 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f804006af00 0x7f804006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:46.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.375+0000 7f805e59c700 1 --2- 172.21.15.16:0/629053574 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f804006af00 0x7f804006d3c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f805000c8a0 tx=0x7f8050008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:46.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.380+0000 7f804effd700 1 -- 172.21.15.16:0/629053574 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8054054780 con 0x7f806015b630 2023-10-16T12:10:46.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.585+0000 7f8064b2d700 1 -- 172.21.15.16:0/629053574 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f806004ebb0 con 0x7f806015b630 2023-10-16T12:10:46.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.586+0000 7f804effd700 1 -- 172.21.15.16:0/629053574 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18238 (secure 0 0 0) 0x7f8054057da0 con 0x7f806015b630 2023-10-16T12:10:46.590 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:10:46.590 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":25,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:10:44.080141+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:10:36.935773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T12:10:38.004870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-16T12:10:41.672025+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:10:46.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.588+0000 7f8064b2d700 1 -- 172.21.15.16:0/629053574 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f804006af00 msgr2=0x7f804006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:46.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.588+0000 7f8064b2d700 1 --2- 172.21.15.16:0/629053574 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f804006af00 0x7f804006d3c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f805000c8a0 tx=0x7f8050008040 comp rx=0 tx=0).stop 2023-10-16T12:10:46.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.588+0000 7f8064b2d700 1 -- 172.21.15.16:0/629053574 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f806015b630 msgr2=0x7f806006ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:46.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.588+0000 7f8064b2d700 1 --2- 172.21.15.16:0/629053574 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f806015b630 0x7f806006ca70 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f805400e9b0 tx=0x7f805400ed70 comp rx=0 tx=0).stop 2023-10-16T12:10:46.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.588+0000 7f8064b2d700 1 -- 172.21.15.16:0/629053574 shutdown_connections 2023-10-16T12:10:46.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.588+0000 7f8064b2d700 1 --2- 172.21.15.16:0/629053574 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f804006af00 0x7f804006d3c0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:46.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.588+0000 7f8064b2d700 1 --2- 172.21.15.16:0/629053574 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f806015ba10 0x7f80600cabb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:46.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.588+0000 7f8064b2d700 1 --2- 172.21.15.16:0/629053574 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f80600ce320 0x7f80600c7240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:46.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.588+0000 7f8064b2d700 1 --2- 172.21.15.16:0/629053574 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f806015b630 0x7f806006ca70 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:46.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.588+0000 7f8064b2d700 1 -- 172.21.15.16:0/629053574 >> 172.21.15.16:0/629053574 conn(0x7f80600d14b0 msgr2=0x7f80600b5d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:46.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.588+0000 7f8064b2d700 1 -- 172.21.15.16:0/629053574 shutdown_connections 2023-10-16T12:10:46.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:46.588+0000 7f8064b2d700 1 -- 172.21.15.16:0/629053574 wait complete. 2023-10-16T12:10:47.006 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 7, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T12:10:39.030243+0000', 'modified': '2023-10-16T12:10:44.080297+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4269}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4269': {'gid': 4269, 'name': 'd', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.49:6835/2892536404', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6833', 'nonce': 2892536404}, {'type': 'v1', 'addr': '172.21.15.49:6835', 'nonce': 2892536404}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1} 2023-10-16T12:10:47.006 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-16T12:10:47.007 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2023-10-16T12:10:47.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.135+0000 7f6bd94f6700 1 Processor -- start 2023-10-16T12:10:47.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.135+0000 7f6bd94f6700 1 -- start start 2023-10-16T12:10:47.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.135+0000 7f6bd94f6700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f6bd415d8d0 0x7f6bd40d1c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:47.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.135+0000 7f6bd94f6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6bd40d41c0 0x7f6bd40d0040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:47.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.135+0000 7f6bd94f6700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6bd40d0580 0x7f6bd40ce240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:47.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.135+0000 7f6bd94f6700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bd40d0a10 con 0x7f6bd40d41c0 2023-10-16T12:10:47.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.135+0000 7f6bd94f6700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bd406ba00 con 0x7f6bd415d8d0 2023-10-16T12:10:47.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.135+0000 7f6bd94f6700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd40d2150 con 0x7f6bd40d0580 2023-10-16T12:10:47.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.135+0000 7f6bd27fc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6bd40d41c0 0x7f6bd40d0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:47.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.136+0000 7f6bd27fc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6bd40d41c0 0x7f6bd40d0040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33096/0 (socket says 172.21.15.16:33096) 2023-10-16T12:10:47.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.136+0000 7f6bd27fc700 1 -- 172.21.15.16:0/2037045779 learned_addr learned my addr 172.21.15.16:0/2037045779 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:10:47.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.136+0000 7f6bd2ffd700 1 --2- 172.21.15.16:0/2037045779 >> v2:172.21.15.49:3300/0 conn(0x7f6bd415d8d0 0x7f6bd40d1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:47.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.136+0000 7f6bd37fe700 1 --2- 172.21.15.16:0/2037045779 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6bd40d0580 0x7f6bd40ce240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:47.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.136+0000 7f6bd27fc700 1 -- 172.21.15.16:0/2037045779 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6bd40d0580 msgr2=0x7f6bd40ce240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:10:47.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.136+0000 7f6bd27fc700 1 --2- 172.21.15.16:0/2037045779 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6bd40d0580 0x7f6bd40ce240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:47.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.136+0000 7f6bd27fc700 1 -- 172.21.15.16:0/2037045779 >> v2:172.21.15.49:3300/0 conn(0x7f6bd415d8d0 msgr2=0x7f6bd40d1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:10:47.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.136+0000 7f6bd27fc700 1 --2- 172.21.15.16:0/2037045779 >> v2:172.21.15.49:3300/0 conn(0x7f6bd415d8d0 0x7f6bd40d1c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:47.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.136+0000 7f6bd27fc700 1 -- 172.21.15.16:0/2037045779 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bd40d26d0 con 0x7f6bd40d41c0 2023-10-16T12:10:47.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.136+0000 7f6bd27fc700 1 --2- 172.21.15.16:0/2037045779 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bd40d41c0 0x7f6bd40d0040 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f6bd408b460 tx=0x7f6bc8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec6201bf7b3d4476 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:47.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.136+0000 7f6bd1ffb700 1 -- 172.21.15.16:0/2037045779 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bc8011830 con 0x7f6bd40d41c0 2023-10-16T12:10:47.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.136+0000 7f6bd1ffb700 1 -- 172.21.15.16:0/2037045779 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bc8011990 con 0x7f6bd40d41c0 2023-10-16T12:10:47.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.136+0000 7f6bd1ffb700 1 -- 172.21.15.16:0/2037045779 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bc8011d20 con 0x7f6bd40d41c0 2023-10-16T12:10:47.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.137+0000 7f6bd94f6700 1 -- 172.21.15.16:0/2037045779 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bd40d41c0 msgr2=0x7f6bd40d0040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:47.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.137+0000 7f6bd94f6700 1 --2- 172.21.15.16:0/2037045779 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bd40d41c0 0x7f6bd40d0040 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f6bd408b460 tx=0x7f6bc8009bd0 comp rx=0 tx=0).stop 2023-10-16T12:10:47.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.137+0000 7f6bd94f6700 1 -- 172.21.15.16:0/2037045779 shutdown_connections 2023-10-16T12:10:47.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.137+0000 7f6bd94f6700 1 --2- 172.21.15.16:0/2037045779 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6bd40d0580 0x7f6bd40ce240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:47.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.137+0000 7f6bd94f6700 1 --2- 172.21.15.16:0/2037045779 >> v2:172.21.15.49:3300/0 conn(0x7f6bd415d8d0 0x7f6bd40d1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:47.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.137+0000 7f6bd94f6700 1 --2- 172.21.15.16:0/2037045779 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bd40d41c0 0x7f6bd40d0040 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:47.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.137+0000 7f6bd94f6700 1 -- 172.21.15.16:0/2037045779 >> 172.21.15.16:0/2037045779 conn(0x7f6bd40d3630 msgr2=0x7f6bd40d3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:47.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.137+0000 7f6bd94f6700 1 -- 172.21.15.16:0/2037045779 shutdown_connections 2023-10-16T12:10:47.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.137+0000 7f6bd94f6700 1 -- 172.21.15.16:0/2037045779 wait complete. 2023-10-16T12:10:47.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.138+0000 7f6bd94f6700 1 Processor -- start 2023-10-16T12:10:47.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.138+0000 7f6bd94f6700 1 -- start start 2023-10-16T12:10:47.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.139+0000 7f6bd94f6700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6bd40d0580 0x7f6bd40c9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:47.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.139+0000 7f6bd94f6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bd40d41c0 0x7f6bd415bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:47.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.139+0000 7f6bd94f6700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6bd415d8d0 0x7f6bd41e72c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:47.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.139+0000 7f6bd94f6700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd41e7070 con 0x7f6bd40d41c0 2023-10-16T12:10:47.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.139+0000 7f6bd94f6700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd409de50 con 0x7f6bd415d8d0 2023-10-16T12:10:47.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.139+0000 7f6bd94f6700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd40c9fe0 con 0x7f6bd40d0580 2023-10-16T12:10:47.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.139+0000 7f6bd27fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bd40d41c0 0x7f6bd415bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:47.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.139+0000 7f6bd27fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bd40d41c0 0x7f6bd415bf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33104/0 (socket says 172.21.15.16:33104) 2023-10-16T12:10:47.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.139+0000 7f6bd27fc700 1 -- 172.21.15.16:0/3936869294 learned_addr learned my addr 172.21.15.16:0/3936869294 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:10:47.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.139+0000 7f6bd2ffd700 1 --2- 172.21.15.16:0/3936869294 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6bd40d0580 0x7f6bd40c9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:47.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.139+0000 7f6bd37fe700 1 --2- 172.21.15.16:0/3936869294 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6bd415d8d0 0x7f6bd41e72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:47.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.139+0000 7f6bd27fc700 1 -- 172.21.15.16:0/3936869294 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6bd40d0580 msgr2=0x7f6bd40c9aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:47.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.139+0000 7f6bd27fc700 1 --2- 172.21.15.16:0/3936869294 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6bd40d0580 0x7f6bd40c9aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:47.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.139+0000 7f6bd27fc700 1 -- 172.21.15.16:0/3936869294 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6bd415d8d0 msgr2=0x7f6bd41e72c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:47.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.139+0000 7f6bd27fc700 1 --2- 172.21.15.16:0/3936869294 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6bd415d8d0 0x7f6bd41e72c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:47.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.139+0000 7f6bd27fc700 1 -- 172.21.15.16:0/3936869294 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bc8009550 con 0x7f6bd40d41c0 2023-10-16T12:10:47.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.139+0000 7f6bd27fc700 1 --2- 172.21.15.16:0/3936869294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bd40d41c0 0x7f6bd415bf00 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6bc8000c00 tx=0x7f6bc8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:47.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.140+0000 7f6bbb7fe700 1 -- 172.21.15.16:0/3936869294 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bc8004300 con 0x7f6bd40d41c0 2023-10-16T12:10:47.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.140+0000 7f6bbb7fe700 1 -- 172.21.15.16:0/3936869294 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bc801f070 con 0x7f6bd40d41c0 2023-10-16T12:10:47.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.140+0000 7f6bd94f6700 1 -- 172.21.15.16:0/3936869294 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bd40c7cf0 con 0x7f6bd40d41c0 2023-10-16T12:10:47.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.140+0000 7f6bbb7fe700 1 -- 172.21.15.16:0/3936869294 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bc80036a0 con 0x7f6bd40d41c0 2023-10-16T12:10:47.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.140+0000 7f6bd94f6700 1 -- 172.21.15.16:0/3936869294 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bd40c82b0 con 0x7f6bd40d41c0 2023-10-16T12:10:47.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.141+0000 7f6bbb7fe700 1 -- 172.21.15.16:0/3936869294 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f6bc8021030 con 0x7f6bd40d41c0 2023-10-16T12:10:47.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.141+0000 7f6bbb7fe700 1 --2- 172.21.15.16:0/3936869294 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f6bc006af00 0x7f6bc006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:47.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.142+0000 7f6bbb7fe700 1 -- 172.21.15.16:0/3936869294 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f6bc808aa60 con 0x7f6bd40d41c0 2023-10-16T12:10:47.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.142+0000 7f6bd2ffd700 1 --2- 172.21.15.16:0/3936869294 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f6bc006af00 0x7f6bc006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:47.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.142+0000 7f6bd2ffd700 1 --2- 172.21.15.16:0/3936869294 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f6bc006af00 0x7f6bc006d3c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f6bc4009b40 tx=0x7f6bc4009210 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:47.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.144+0000 7f6be56e0b80 1 Processor -- start 2023-10-16T12:10:47.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.144+0000 7f6be56e0b80 1 -- start start 2023-10-16T12:10:47.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.145+0000 7f6be56e0b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5595022e6220 0x5595022eda70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:47.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.145+0000 7f6be56e0b80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5595022eb630 0x5595022edfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:47.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.145+0000 7f6be56e0b80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x5595022c1010 0x5595022c16c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:47.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.145+0000 7f6be56e0b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x559502237c20 con 0x5595022e6220 2023-10-16T12:10:47.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.145+0000 7f6be56e0b80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x55950221cc10 con 0x5595022eb630 2023-10-16T12:10:47.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.145+0000 7f6be56e0b80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x55950221a530 con 0x5595022c1010 2023-10-16T12:10:47.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.145+0000 7f6bd2ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5595022e6220 0x5595022eda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:47.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.145+0000 7f6bd2ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5595022e6220 0x5595022eda70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33110/0 (socket says 172.21.15.16:33110) 2023-10-16T12:10:47.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.145+0000 7f6bd2ffd700 1 -- 172.21.15.16:0/2586354990 learned_addr learned my addr 172.21.15.16:0/2586354990 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:10:47.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.145+0000 7f6bd27fc700 1 --2- 172.21.15.16:0/2586354990 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5595022eb630 0x5595022edfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:47.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.145+0000 7f6bd37fe700 1 --2- 172.21.15.16:0/2586354990 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x5595022c1010 0x5595022c16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:47.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.145+0000 7f6bd2ffd700 1 -- 172.21.15.16:0/2586354990 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x5595022c1010 msgr2=0x5595022c16c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:10:47.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.145+0000 7f6bd2ffd700 1 --2- 172.21.15.16:0/2586354990 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x5595022c1010 0x5595022c16c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:47.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.145+0000 7f6bd2ffd700 1 -- 172.21.15.16:0/2586354990 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5595022eb630 msgr2=0x5595022edfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:10:47.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.145+0000 7f6bd2ffd700 1 --2- 172.21.15.16:0/2586354990 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5595022eb630 0x5595022edfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:47.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.145+0000 7f6bd2ffd700 1 -- 172.21.15.16:0/2586354990 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5595021d33a0 con 0x5595022e6220 2023-10-16T12:10:47.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.145+0000 7f6bd2ffd700 1 --2- 172.21.15.16:0/2586354990 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5595022e6220 0x5595022eda70 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f6bc40044f0 tx=0x7f6bc4014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73938cd2e9b86cca server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:47.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.145+0000 7f6bb8ff9700 1 -- 172.21.15.16:0/2586354990 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bc401c040 con 0x5595022e6220 2023-10-16T12:10:47.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.145+0000 7f6bb8ff9700 1 -- 172.21.15.16:0/2586354990 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bc4004520 con 0x5595022e6220 2023-10-16T12:10:47.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.145+0000 7f6bb8ff9700 1 -- 172.21.15.16:0/2586354990 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bc4003ab0 con 0x5595022e6220 2023-10-16T12:10:47.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.146+0000 7f6be56e0b80 1 -- 172.21.15.16:0/2586354990 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5595022e6220 msgr2=0x5595022eda70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:47.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.146+0000 7f6be56e0b80 1 --2- 172.21.15.16:0/2586354990 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5595022e6220 0x5595022eda70 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f6bc40044f0 tx=0x7f6bc4014000 comp rx=0 tx=0).stop 2023-10-16T12:10:47.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.146+0000 7f6be56e0b80 1 -- 172.21.15.16:0/2586354990 shutdown_connections 2023-10-16T12:10:47.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.146+0000 7f6be56e0b80 1 --2- 172.21.15.16:0/2586354990 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x5595022c1010 0x5595022c16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:47.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.146+0000 7f6be56e0b80 1 --2- 172.21.15.16:0/2586354990 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5595022e6220 0x5595022eda70 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:47.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.146+0000 7f6be56e0b80 1 --2- 172.21.15.16:0/2586354990 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5595022eb630 0x5595022edfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:47.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.146+0000 7f6be56e0b80 1 -- 172.21.15.16:0/2586354990 >> 172.21.15.16:0/2586354990 conn(0x5595021f0d80 msgr2=0x55950230bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:47.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.146+0000 7f6be56e0b80 1 -- 172.21.15.16:0/2586354990 shutdown_connections 2023-10-16T12:10:47.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.146+0000 7f6be56e0b80 1 -- 172.21.15.16:0/2586354990 wait complete. 2023-10-16T12:10:47.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.147+0000 7f6be56e0b80 1 -- start start 2023-10-16T12:10:47.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.147+0000 7f6be56e0b80 1 Processor -- start 2023-10-16T12:10:47.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.148+0000 7f6be56e0b80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x5595022c1010 0x5595022e5350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:47.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.148+0000 7f6be56e0b80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5595022e6220 0x5595022c3500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:47.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.148+0000 7f6be56e0b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5595022eb630 0x5595022c3a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:47.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.148+0000 7f6be56e0b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x559502215e20 con 0x5595022eb630 2023-10-16T12:10:47.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.148+0000 7f6be56e0b80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595021b26c0 con 0x5595022e6220 2023-10-16T12:10:47.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.148+0000 7f6be56e0b80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x5595021b2360 con 0x5595022c1010 2023-10-16T12:10:47.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.148+0000 7f6be56e0b80 10 client.0 _ll_register_callbacks cb 0x55950200f7b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T12:10:47.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.148+0000 7f6bd37fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5595022eb630 0x5595022c3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:47.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.148+0000 7f6bd37fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5595022eb630 0x5595022c3a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33120/0 (socket says 172.21.15.16:33120) 2023-10-16T12:10:47.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.148+0000 7f6bd37fe700 1 -- 172.21.15.16:0/2663834690 learned_addr learned my addr 172.21.15.16:0/2663834690 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:10:47.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.148+0000 7f6bd2ffd700 1 --2- 172.21.15.16:0/2663834690 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x5595022c1010 0x5595022e5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:47.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.148+0000 7f6bd27fc700 1 --2- 172.21.15.16:0/2663834690 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5595022e6220 0x5595022c3500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:47.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.148+0000 7f6bd37fe700 1 -- 172.21.15.16:0/2663834690 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x5595022c1010 msgr2=0x5595022e5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:10:47.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.148+0000 7f6bd37fe700 1 --2- 172.21.15.16:0/2663834690 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x5595022c1010 0x5595022e5350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:47.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.148+0000 7f6bd37fe700 1 -- 172.21.15.16:0/2663834690 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5595022e6220 msgr2=0x5595022c3500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:10:47.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.148+0000 7f6bd37fe700 1 --2- 172.21.15.16:0/2663834690 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5595022e6220 0x5595022c3500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:47.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.148+0000 7f6bd37fe700 1 -- 172.21.15.16:0/2663834690 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bc40041d0 con 0x5595022eb630 2023-10-16T12:10:47.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.148+0000 7f6bd37fe700 1 --2- 172.21.15.16:0/2663834690 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5595022eb630 0x5595022c3a40 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc007d00 tx=0x7f6bcc011490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49c134712fb04dea server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:47.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.148+0000 7f6bb1ffb700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2023-10-16T12:10:47.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.148+0000 7f6bb1ffb700 1 -- 172.21.15.16:0/2663834690 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bcc010040 con 0x5595022eb630 2023-10-16T12:10:47.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.148+0000 7f6bb1ffb700 1 -- 172.21.15.16:0/2663834690 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f6bcc011c20 con 0x5595022eb630 2023-10-16T12:10:47.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.148+0000 7f6be56e0b80 1 -- 172.21.15.16:0/2663834690 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5595022b65b0 con 0x5595022eb630 2023-10-16T12:10:47.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.148+0000 7f6bb1ffb700 1 -- 172.21.15.16:0/2663834690 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6bcc011de0 con 0x5595022eb630 2023-10-16T12:10:47.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.148+0000 7f6be56e0b80 10 client.4407 fetch_fsmap learned FSMap version 7 2023-10-16T12:10:47.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.148+0000 7f6be56e0b80 1 -- 172.21.15.16:0/2663834690 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) v3 -- 0x559502040d90 con 0x5595022eb630 2023-10-16T12:10:47.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.149+0000 7f6bb1ffb700 1 -- 172.21.15.16:0/2663834690 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5595022b65b0 con 0x5595022eb630 2023-10-16T12:10:47.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.149+0000 7f6bb1ffb700 1 -- 172.21.15.16:0/2663834690 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 7) v1 ==== 3302+0+0 (secure 0 0 0) 0x7f6bcc020360 con 0x5595022eb630 2023-10-16T12:10:47.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.149+0000 7f6be56e0b80 10 client.4407 fetch_fsmap finished waiting for FSMap version 7 2023-10-16T12:10:47.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.149+0000 7f6be56e0b80 10 client.4407 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-16T12:10:47.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.149+0000 7f6be56e0b80 20 client.4407 populate_metadata read hostname 'smithi016' 2023-10-16T12:10:47.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.149+0000 7f6be56e0b80 1 --2- 172.21.15.16:0/2663834690 >> [v2:172.21.15.49:6833/2892536404,v1:172.21.15.49:6835/2892536404] conn(0x5595022e9400 0x5595022eefb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:47.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.149+0000 7f6be56e0b80 4 client.4407 mds_command: new command op to 4269 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T12:10:47.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.149+0000 7f6be56e0b80 1 -- 172.21.15.16:0/2663834690 --> [v2:172.21.15.49:6833/2892536404,v1:172.21.15.49:6835/2892536404] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55950220ee00 con 0x5595022e9400 2023-10-16T12:10:47.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.149+0000 7f6bd2ffd700 1 --2- 172.21.15.16:0/2663834690 >> [v2:172.21.15.49:6833/2892536404,v1:172.21.15.49:6835/2892536404] conn(0x5595022e9400 0x5595022eefb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:47.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.149+0000 7f6bb1ffb700 10 client.4407 ms_handle_connect on v2:172.21.15.49:6833/2892536404 2023-10-16T12:10:47.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.149+0000 7f6bd2ffd700 1 --2- 172.21.15.16:0/2663834690 >> [v2:172.21.15.49:6833/2892536404,v1:172.21.15.49:6835/2892536404] conn(0x5595022e9400 0x5595022eefb0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aea4d59a54f68e7e server_cookie=f93f2d2360600890 in_seq=0 out_seq=0 2023-10-16T12:10:47.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.150+0000 7f6bb1ffb700 1 -- 172.21.15.16:0/2663834690 <== mds.0 v2:172.21.15.49:6833/2892536404 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55950220ee00 con 0x5595022e9400 2023-10-16T12:10:47.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.150+0000 7f6bb1ffb700 10 client.4407 handle_command_reply: tid=0 2023-10-16T12:10:47.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.150+0000 7f6be56e0b80 1 client.4407 shutdown 2023-10-16T12:10:47.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.150+0000 7f6be56e0b80 1 -- 172.21.15.16:0/2663834690 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5595022eb630 msgr2=0x5595022c3a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:47.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.150+0000 7f6be56e0b80 1 --2- 172.21.15.16:0/2663834690 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5595022eb630 0x5595022c3a40 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc007d00 tx=0x7f6bcc011490 comp rx=0 tx=0).stop 2023-10-16T12:10:47.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.150+0000 7f6be56e0b80 1 -- 172.21.15.16:0/2663834690 shutdown_connections 2023-10-16T12:10:47.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.150+0000 7f6be56e0b80 1 --2- 172.21.15.16:0/2663834690 >> [v2:172.21.15.49:6833/2892536404,v1:172.21.15.49:6835/2892536404] conn(0x5595022e9400 0x5595022eefb0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:47.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.150+0000 7f6be56e0b80 1 --2- 172.21.15.16:0/2663834690 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5595022eb630 0x5595022c3a40 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:47.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.150+0000 7f6be56e0b80 1 --2- 172.21.15.16:0/2663834690 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x5595022c1010 0x5595022e5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:47.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.150+0000 7f6bb1ffb700 0 client.4407 ms_handle_reset on v2:172.21.15.49:6833/2892536404 2023-10-16T12:10:47.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.150+0000 7f6be56e0b80 1 --2- 172.21.15.16:0/2663834690 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5595022e6220 0x5595022c3500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:47.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.150+0000 7f6be56e0b80 1 -- 172.21.15.16:0/2663834690 >> 172.21.15.16:0/2663834690 conn(0x5595021f0d80 msgr2=0x5595022c29f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:47.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.150+0000 7f6be56e0b80 1 -- 172.21.15.16:0/2663834690 shutdown_connections 2023-10-16T12:10:47.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.150+0000 7f6be56e0b80 1 -- 172.21.15.16:0/2663834690 wait complete. 2023-10-16T12:10:47.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.151+0000 7f6be56e0b80 20 client.4407 trim_cache size 0 max 16384 2023-10-16T12:10:47.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.169+0000 7f6be56e0b80 1 Processor -- start 2023-10-16T12:10:47.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.169+0000 7f6be56e0b80 1 -- start start 2023-10-16T12:10:47.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.170+0000 7f6be56e0b80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x5595022c1010 0x5595022e5160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:47.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.170+0000 7f6be56e0b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5595022e6220 0x5595022e56a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:47.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.170+0000 7f6be56e0b80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5595022eb630 0x5595022f10e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:47.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.170+0000 7f6be56e0b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595020968d0 con 0x5595022e6220 2023-10-16T12:10:47.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.170+0000 7f6be56e0b80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x559502307dd0 con 0x5595022eb630 2023-10-16T12:10:47.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.170+0000 7f6be56e0b80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x5595022e9da0 con 0x5595022c1010 2023-10-16T12:10:47.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.170+0000 7f6bd27fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5595022e6220 0x5595022e56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:47.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.170+0000 7f6bd27fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5595022e6220 0x5595022e56a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33124/0 (socket says 172.21.15.16:33124) 2023-10-16T12:10:47.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.170+0000 7f6bd27fc700 1 -- 172.21.15.16:0/1359962636 learned_addr learned my addr 172.21.15.16:0/1359962636 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:10:47.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.170+0000 7f6bd2ffd700 1 --2- 172.21.15.16:0/1359962636 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x5595022c1010 0x5595022e5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:47.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.170+0000 7f6bd37fe700 1 --2- 172.21.15.16:0/1359962636 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5595022eb630 0x5595022f10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:47.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.170+0000 7f6bd27fc700 1 -- 172.21.15.16:0/1359962636 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x5595022c1010 msgr2=0x5595022e5160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:10:47.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.170+0000 7f6bd27fc700 1 --2- 172.21.15.16:0/1359962636 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x5595022c1010 0x5595022e5160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:47.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.170+0000 7f6bd27fc700 1 -- 172.21.15.16:0/1359962636 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5595022eb630 msgr2=0x5595022f10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:10:47.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.170+0000 7f6bd27fc700 1 --2- 172.21.15.16:0/1359962636 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5595022eb630 0x5595022f10e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:47.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.170+0000 7f6bd27fc700 1 -- 172.21.15.16:0/1359962636 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559502029c10 con 0x5595022e6220 2023-10-16T12:10:47.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.170+0000 7f6bd37fe700 1 --2- 172.21.15.16:0/1359962636 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5595022eb630 0x5595022f10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:10:47.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.170+0000 7f6bd27fc700 1 --2- 172.21.15.16:0/1359962636 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5595022e6220 0x5595022e56a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6bc8055b90 tx=0x7f6bc8006df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=988b94130520ee82 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:47.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.170+0000 7f6bb97fa700 1 -- 172.21.15.16:0/1359962636 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bc8058ff0 con 0x5595022e6220 2023-10-16T12:10:47.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.170+0000 7f6bb97fa700 1 -- 172.21.15.16:0/1359962636 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bc8059d90 con 0x5595022e6220 2023-10-16T12:10:47.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.170+0000 7f6bb97fa700 1 -- 172.21.15.16:0/1359962636 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bc8059690 con 0x5595022e6220 2023-10-16T12:10:47.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.171+0000 7f6be56e0b80 1 -- 172.21.15.16:0/1359962636 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5595022e6220 msgr2=0x5595022e56a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:47.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.171+0000 7f6be56e0b80 1 --2- 172.21.15.16:0/1359962636 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5595022e6220 0x5595022e56a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6bc8055b90 tx=0x7f6bc8006df0 comp rx=0 tx=0).stop 2023-10-16T12:10:47.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.171+0000 7f6be56e0b80 1 -- 172.21.15.16:0/1359962636 shutdown_connections 2023-10-16T12:10:47.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.171+0000 7f6be56e0b80 1 --2- 172.21.15.16:0/1359962636 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5595022eb630 0x5595022f10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:47.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.171+0000 7f6be56e0b80 1 --2- 172.21.15.16:0/1359962636 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x5595022c1010 0x5595022e5160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:47.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.171+0000 7f6be56e0b80 1 --2- 172.21.15.16:0/1359962636 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5595022e6220 0x5595022e56a0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:47.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.171+0000 7f6be56e0b80 1 -- 172.21.15.16:0/1359962636 >> 172.21.15.16:0/1359962636 conn(0x5595021f0d80 msgr2=0x55950232a6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:47.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.171+0000 7f6be56e0b80 1 -- 172.21.15.16:0/1359962636 shutdown_connections 2023-10-16T12:10:47.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.171+0000 7f6be56e0b80 1 -- 172.21.15.16:0/1359962636 wait complete. 2023-10-16T12:10:47.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.172+0000 7f6be56e0b80 1 -- start start 2023-10-16T12:10:47.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.172+0000 7f6be56e0b80 1 Processor -- start 2023-10-16T12:10:47.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.172+0000 7f6be56e0b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5595022c1010 0x559502372930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:47.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.172+0000 7f6be56e0b80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5595022e6220 0x5595022f1860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:47.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.172+0000 7f6be56e0b80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x5595022eb630 0x5595022e5560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:47.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.172+0000 7f6be56e0b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x559502373180 con 0x5595022c1010 2023-10-16T12:10:47.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.172+0000 7f6be56e0b80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595022b8720 con 0x5595022e6220 2023-10-16T12:10:47.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.172+0000 7f6be56e0b80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x5595022e7140 con 0x5595022eb630 2023-10-16T12:10:47.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.173+0000 7f6be56e0b80 10 client.0 _ll_register_callbacks cb 0x7f6bcc0076d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T12:10:47.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.173+0000 7f6bd2ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5595022c1010 0x559502372930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:47.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.173+0000 7f6bd2ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5595022c1010 0x559502372930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33130/0 (socket says 172.21.15.16:33130) 2023-10-16T12:10:47.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.173+0000 7f6bd2ffd700 1 -- 172.21.15.16:0/2511508298 learned_addr learned my addr 172.21.15.16:0/2511508298 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:10:47.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.173+0000 7f6bd27fc700 1 --2- 172.21.15.16:0/2511508298 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5595022e6220 0x5595022f1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:47.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.173+0000 7f6bd37fe700 1 --2- 172.21.15.16:0/2511508298 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x5595022eb630 0x5595022e5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:47.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.173+0000 7f6bd2ffd700 1 -- 172.21.15.16:0/2511508298 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x5595022eb630 msgr2=0x5595022e5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:10:47.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.173+0000 7f6bd2ffd700 1 --2- 172.21.15.16:0/2511508298 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x5595022eb630 0x5595022e5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:47.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.173+0000 7f6bd2ffd700 1 -- 172.21.15.16:0/2511508298 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5595022e6220 msgr2=0x5595022f1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:10:47.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.173+0000 7f6bd2ffd700 1 --2- 172.21.15.16:0/2511508298 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5595022e6220 0x5595022f1860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:47.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.173+0000 7f6bd2ffd700 1 -- 172.21.15.16:0/2511508298 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bd40c82b0 con 0x5595022c1010 2023-10-16T12:10:47.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.173+0000 7f6bd2ffd700 1 --2- 172.21.15.16:0/2511508298 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5595022c1010 0x559502372930 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x5595022c4fa0 tx=0x7f6bc40047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3720ee6bc9b4908 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:47.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.173+0000 7f6bb1ffb700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2023-10-16T12:10:47.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.173+0000 7f6bb1ffb700 1 -- 172.21.15.16:0/2511508298 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bc4032070 con 0x5595022c1010 2023-10-16T12:10:47.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.173+0000 7f6bb1ffb700 1 -- 172.21.15.16:0/2511508298 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f6bc4003e60 con 0x5595022c1010 2023-10-16T12:10:47.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.173+0000 7f6be56e0b80 1 -- 172.21.15.16:0/2511508298 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6bc4004520 con 0x5595022c1010 2023-10-16T12:10:47.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.173+0000 7f6bb1ffb700 1 -- 172.21.15.16:0/2511508298 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6bc40354c0 con 0x5595022c1010 2023-10-16T12:10:47.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.173+0000 7f6be56e0b80 10 client.4413 fetch_fsmap learned FSMap version 7 2023-10-16T12:10:47.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.173+0000 7f6be56e0b80 1 -- 172.21.15.16:0/2511508298 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) v3 -- 0x559502067200 con 0x5595022c1010 2023-10-16T12:10:47.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.173+0000 7f6bb1ffb700 1 -- 172.21.15.16:0/2511508298 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bc4004520 con 0x5595022c1010 2023-10-16T12:10:47.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.173+0000 7f6bb1ffb700 1 -- 172.21.15.16:0/2511508298 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 7) v1 ==== 3302+0+0 (secure 0 0 0) 0x7f6bc40357a0 con 0x5595022c1010 2023-10-16T12:10:47.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.174+0000 7f6be56e0b80 10 client.4413 fetch_fsmap finished waiting for FSMap version 7 2023-10-16T12:10:47.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.174+0000 7f6be56e0b80 10 client.4413 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-16T12:10:47.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.174+0000 7f6be56e0b80 20 client.4413 populate_metadata read hostname 'smithi016' 2023-10-16T12:10:47.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.174+0000 7f6be56e0b80 1 --2- 172.21.15.16:0/2511508298 >> [v2:172.21.15.49:6833/2892536404,v1:172.21.15.49:6835/2892536404] conn(0x559502354c50 0x559502355030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:47.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.174+0000 7f6be56e0b80 4 client.4413 mds_command: new command op to 4269 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T12:10:47.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.174+0000 7f6be56e0b80 1 -- 172.21.15.16:0/2511508298 --> [v2:172.21.15.49:6833/2892536404,v1:172.21.15.49:6835/2892536404] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55950220eb20 con 0x559502354c50 2023-10-16T12:10:47.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.174+0000 7f6bd27fc700 1 --2- 172.21.15.16:0/2511508298 >> [v2:172.21.15.49:6833/2892536404,v1:172.21.15.49:6835/2892536404] conn(0x559502354c50 0x559502355030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:47.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.174+0000 7f6bb1ffb700 10 client.4413 ms_handle_connect on v2:172.21.15.49:6833/2892536404 2023-10-16T12:10:47.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.174+0000 7f6bd27fc700 1 --2- 172.21.15.16:0/2511508298 >> [v2:172.21.15.49:6833/2892536404,v1:172.21.15.49:6835/2892536404] conn(0x559502354c50 0x559502355030 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=99148e3b44bbf844 server_cookie=9d88de82136f865c in_seq=0 out_seq=0 2023-10-16T12:10:47.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.174+0000 7f6bb1ffb700 1 -- 172.21.15.16:0/2511508298 <== mds.0 v2:172.21.15.49:6833/2892536404 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+330 (crc 0 0 0) 0x55950220eb20 con 0x559502354c50 2023-10-16T12:10:47.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.174+0000 7f6bb1ffb700 10 client.4413 handle_command_reply: tid=0 2023-10-16T12:10:47.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.174+0000 7f6be56e0b80 1 client.4413 shutdown 2023-10-16T12:10:47.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.174+0000 7f6be56e0b80 1 -- 172.21.15.16:0/2511508298 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5595022c1010 msgr2=0x559502372930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:47.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.174+0000 7f6be56e0b80 1 --2- 172.21.15.16:0/2511508298 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5595022c1010 0x559502372930 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x5595022c4fa0 tx=0x7f6bc40047c0 comp rx=0 tx=0).stop 2023-10-16T12:10:47.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.174+0000 7f6be56e0b80 1 -- 172.21.15.16:0/2511508298 shutdown_connections 2023-10-16T12:10:47.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.174+0000 7f6be56e0b80 1 --2- 172.21.15.16:0/2511508298 >> [v2:172.21.15.49:6833/2892536404,v1:172.21.15.49:6835/2892536404] conn(0x559502354c50 0x559502355030 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:47.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.174+0000 7f6be56e0b80 1 --2- 172.21.15.16:0/2511508298 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x5595022eb630 0x5595022e5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:47.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.175+0000 7f6be56e0b80 1 --2- 172.21.15.16:0/2511508298 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5595022c1010 0x559502372930 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:47.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.175+0000 7f6bb1ffb700 0 client.4413 ms_handle_reset on v2:172.21.15.49:6833/2892536404 2023-10-16T12:10:47.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.175+0000 7f6be56e0b80 1 --2- 172.21.15.16:0/2511508298 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5595022e6220 0x5595022f1860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:47.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.175+0000 7f6be56e0b80 1 -- 172.21.15.16:0/2511508298 >> 172.21.15.16:0/2511508298 conn(0x5595021f0d80 msgr2=0x5595022bda90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:47.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.175+0000 7f6be56e0b80 1 -- 172.21.15.16:0/2511508298 shutdown_connections 2023-10-16T12:10:47.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.175+0000 7f6be56e0b80 1 -- 172.21.15.16:0/2511508298 wait complete. 2023-10-16T12:10:47.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.175+0000 7f6be56e0b80 20 client.4413 trim_cache size 0 max 16384 2023-10-16T12:10:47.196 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-10-16T12:10:47.196 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "5b28dfa1-b21f-474a-9b2e-9fb77af58254", 2023-10-16T12:10:47.197 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 0, 2023-10-16T12:10:47.197 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 4269, 2023-10-16T12:10:47.197 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2023-10-16T12:10:47.197 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2023-10-16T12:10:47.197 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2023-10-16T12:10:47.198 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 7.5450196610000004, 2023-10-16T12:10:47.198 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 7, 2023-10-16T12:10:47.198 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 25, 2023-10-16T12:10:47.198 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 25, 2023-10-16T12:10:47.198 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 14.795660935000001 2023-10-16T12:10:47.199 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-10-16T12:10:47.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.175+0000 7f6bd94f6700 1 -- 172.21.15.16:0/3936869294 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f6bc006af00 msgr2=0x7f6bc006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:47.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.175+0000 7f6bd94f6700 1 --2- 172.21.15.16:0/3936869294 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f6bc006af00 0x7f6bc006d3c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f6bc4009b40 tx=0x7f6bc4009210 comp rx=0 tx=0).stop 2023-10-16T12:10:47.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.175+0000 7f6bd94f6700 1 -- 172.21.15.16:0/3936869294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bd40d41c0 msgr2=0x7f6bd415bf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:47.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.175+0000 7f6bd94f6700 1 --2- 172.21.15.16:0/3936869294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bd40d41c0 0x7f6bd415bf00 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6bc8000c00 tx=0x7f6bc8004750 comp rx=0 tx=0).stop 2023-10-16T12:10:47.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.175+0000 7f6bd94f6700 1 -- 172.21.15.16:0/3936869294 shutdown_connections 2023-10-16T12:10:47.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.175+0000 7f6bd94f6700 1 --2- 172.21.15.16:0/3936869294 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f6bc006af00 0x7f6bc006d3c0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:47.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.175+0000 7f6bd94f6700 1 --2- 172.21.15.16:0/3936869294 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6bd415d8d0 0x7f6bd41e72c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:47.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.175+0000 7f6bd94f6700 1 --2- 172.21.15.16:0/3936869294 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6bd40d0580 0x7f6bd40c9aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:47.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.175+0000 7f6bd94f6700 1 --2- 172.21.15.16:0/3936869294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bd40d41c0 0x7f6bd415bf00 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:47.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.175+0000 7f6bd94f6700 1 -- 172.21.15.16:0/3936869294 >> 172.21.15.16:0/3936869294 conn(0x7f6bd40d3630 msgr2=0x7f6bd40b1e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:47.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.175+0000 7f6bd94f6700 1 -- 172.21.15.16:0/3936869294 shutdown_connections 2023-10-16T12:10:47.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.175+0000 7f6bd94f6700 1 -- 172.21.15.16:0/3936869294 wait complete. 2023-10-16T12:10:47.203 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-10-16T12:10:47.204 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2023-10-16T12:10:47.204 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T12:10:47.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.327+0000 7fd2ffaf9700 1 Processor -- start 2023-10-16T12:10:47.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.328+0000 7fd2ffaf9700 1 -- start start 2023-10-16T12:10:47.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.328+0000 7fd2ffaf9700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd2f80d03b0 0x7fd2f80cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:47.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.328+0000 7fd2ffaf9700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2f80cc2a0 con 0x7fd2f815b630 2023-10-16T12:10:47.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.328+0000 7fd2ffaf9700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2f80cc540 con 0x7fd2f80ce4f0 2023-10-16T12:10:47.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.328+0000 7fd2ffaf9700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f80cdb20 con 0x7fd2f80d03b0 2023-10-16T12:10:47.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.329+0000 7fd2fd895700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd2f815b630 0x7fd2f80cdde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50644/0 (socket says 172.21.15.16:50644) 2023-10-16T12:10:47.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.329+0000 7fd2fd895700 1 -- 172.21.15.16:0/1967094438 learned_addr learned my addr 172.21.15.16:0/1967094438 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:10:47.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.329+0000 7fd2fe096700 1 --2- 172.21.15.16:0/1967094438 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd2f80d03b0 0x7fd2f80cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:47.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.329+0000 7fd2fc893700 1 -- 172.21.15.16:0/1967094438 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1527676569 0 0) 0x7fd2f80cc2a0 con 0x7fd2f815b630 2023-10-16T12:10:47.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.329+0000 7fd2fc893700 1 -- 172.21.15.16:0/1967094438 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2e40035b0 con 0x7fd2f815b630 2023-10-16T12:10:47.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.329+0000 7fd2fc893700 1 -- 172.21.15.16:0/1967094438 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 199588973 0 0) 0x7fd2f80cc540 con 0x7fd2f80ce4f0 2023-10-16T12:10:47.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.329+0000 7fd2fc893700 1 -- 172.21.15.16:0/1967094438 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2f80cc2a0 con 0x7fd2f80ce4f0 2023-10-16T12:10:47.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.329+0000 7fd2fc893700 1 -- 172.21.15.16:0/1967094438 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2259524562 0 0) 0x7fd2e40035b0 con 0x7fd2f815b630 2023-10-16T12:10:47.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.329+0000 7fd2fe096700 1 -- 172.21.15.16:0/1967094438 >> v1:172.21.15.49:6789/0 conn(0x7fd2f80ce4f0 legacy=0x7fd2f80cfca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:47.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.329+0000 7fd2fe096700 1 -- 172.21.15.16:0/1967094438 >> v1:172.21.15.16:6789/0 conn(0x7fd2f815b630 legacy=0x7fd2f80cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:47.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.329+0000 7fd2fe096700 1 -- 172.21.15.16:0/1967094438 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2f80cc820 con 0x7fd2f80d03b0 2023-10-16T12:10:47.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.329+0000 7fd2fe096700 1 --2- 172.21.15.16:0/1967094438 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd2f80d03b0 0x7fd2f80cd5e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd2f0009dc0 tx=0x7fd2f000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c60bfaa7e0bd868d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:47.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.329+0000 7fd2fc893700 1 -- 172.21.15.16:0/1967094438 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f000e070 con 0x7fd2f80d03b0 2023-10-16T12:10:47.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.330+0000 7fd2fc893700 1 -- 172.21.15.16:0/1967094438 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2f0003a20 con 0x7fd2f80d03b0 2023-10-16T12:10:47.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.330+0000 7fd2fc893700 1 -- 172.21.15.16:0/1967094438 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f40017f0 con 0x7fd2f80d03b0 2023-10-16T12:10:47.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.330+0000 7fd2ffaf9700 1 -- 172.21.15.16:0/1967094438 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd2f80d03b0 msgr2=0x7fd2f80cd5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:47.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.330+0000 7fd2ffaf9700 1 --2- 172.21.15.16:0/1967094438 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd2f80d03b0 0x7fd2f80cd5e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd2f0009dc0 tx=0x7fd2f000b330 comp rx=0 tx=0).stop 2023-10-16T12:10:47.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.330+0000 7fd2ffaf9700 1 -- 172.21.15.16:0/1967094438 shutdown_connections 2023-10-16T12:10:47.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.330+0000 7fd2ffaf9700 1 --2- 172.21.15.16:0/1967094438 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd2f80d03b0 0x7fd2f80cd5e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:47.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.330+0000 7fd2ffaf9700 1 -- 172.21.15.16:0/1967094438 >> 172.21.15.16:0/1967094438 conn(0x7fd2f80d14b0 msgr2=0x7fd2f80d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:47.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.330+0000 7fd2ffaf9700 1 -- 172.21.15.16:0/1967094438 shutdown_connections 2023-10-16T12:10:47.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.330+0000 7fd2ffaf9700 1 -- 172.21.15.16:0/1967094438 wait complete. 2023-10-16T12:10:47.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.331+0000 7fd2ffaf9700 1 Processor -- start 2023-10-16T12:10:47.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.331+0000 7fd2ffaf9700 1 -- start start 2023-10-16T12:10:47.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.332+0000 7fd2ffaf9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2f80ce4f0 0x7fd2f8152140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:47.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.332+0000 7fd2ffaf9700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd2f80d03b0 0x7fd2f809b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:47.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.332+0000 7fd2ffaf9700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd2f815b630 0x7fd2f809b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:47.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.332+0000 7fd2ffaf9700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f814c1c0 con 0x7fd2f80ce4f0 2023-10-16T12:10:47.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.332+0000 7fd2ffaf9700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f8152680 con 0x7fd2f815b630 2023-10-16T12:10:47.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.332+0000 7fd2ffaf9700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f80caef0 con 0x7fd2f80d03b0 2023-10-16T12:10:47.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.332+0000 7fd2fd895700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2f80ce4f0 0x7fd2f8152140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:47.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.332+0000 7fd2fd895700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2f80ce4f0 0x7fd2f8152140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33144/0 (socket says 172.21.15.16:33144) 2023-10-16T12:10:47.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.332+0000 7fd2fd895700 1 -- 172.21.15.16:0/2794712179 learned_addr learned my addr 172.21.15.16:0/2794712179 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:10:47.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.332+0000 7fd2fd094700 1 --2- 172.21.15.16:0/2794712179 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd2f80d03b0 0x7fd2f809b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:47.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.332+0000 7fd2fe096700 1 --2- 172.21.15.16:0/2794712179 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd2f815b630 0x7fd2f809b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:47.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.332+0000 7fd2fd895700 1 -- 172.21.15.16:0/2794712179 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd2f80d03b0 msgr2=0x7fd2f809b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:47.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.332+0000 7fd2fd895700 1 --2- 172.21.15.16:0/2794712179 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd2f80d03b0 0x7fd2f809b130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:47.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.332+0000 7fd2fd895700 1 -- 172.21.15.16:0/2794712179 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd2f815b630 msgr2=0x7fd2f809b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:47.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.332+0000 7fd2fd895700 1 --2- 172.21.15.16:0/2794712179 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd2f815b630 0x7fd2f809b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:47.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.332+0000 7fd2fd895700 1 -- 172.21.15.16:0/2794712179 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2f0009a70 con 0x7fd2f80ce4f0 2023-10-16T12:10:47.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.332+0000 7fd2fe096700 1 --2- 172.21.15.16:0/2794712179 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd2f815b630 0x7fd2f809b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:10:47.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.333+0000 7fd2fd895700 1 --2- 172.21.15.16:0/2794712179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2f80ce4f0 0x7fd2f8152140 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd2f400c4b0 tx=0x7fd2f400c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:47.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.333+0000 7fd2ea7fc700 1 -- 172.21.15.16:0/2794712179 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f40074e0 con 0x7fd2f80ce4f0 2023-10-16T12:10:47.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.333+0000 7fd2ea7fc700 1 -- 172.21.15.16:0/2794712179 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2f4007640 con 0x7fd2f80ce4f0 2023-10-16T12:10:47.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.333+0000 7fd2ffaf9700 1 -- 172.21.15.16:0/2794712179 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2f80ca370 con 0x7fd2f80ce4f0 2023-10-16T12:10:47.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.333+0000 7fd2ea7fc700 1 -- 172.21.15.16:0/2794712179 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f4005a50 con 0x7fd2f80ce4f0 2023-10-16T12:10:47.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.333+0000 7fd2ffaf9700 1 -- 172.21.15.16:0/2794712179 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2f80ca930 con 0x7fd2f80ce4f0 2023-10-16T12:10:47.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.334+0000 7fd2ffaf9700 1 -- 172.21.15.16:0/2794712179 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2f804ebb0 con 0x7fd2f80ce4f0 2023-10-16T12:10:47.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.334+0000 7fd2ea7fc700 1 -- 172.21.15.16:0/2794712179 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fd2f4005bb0 con 0x7fd2f80ce4f0 2023-10-16T12:10:47.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.335+0000 7fd2ea7fc700 1 --2- 172.21.15.16:0/2794712179 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fd2dc06af00 0x7fd2dc06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:47.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.335+0000 7fd2ea7fc700 1 -- 172.21.15.16:0/2794712179 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fd2f4088e20 con 0x7fd2f80ce4f0 2023-10-16T12:10:47.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.335+0000 7fd2fd094700 1 --2- 172.21.15.16:0/2794712179 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fd2dc06af00 0x7fd2dc06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:47.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.335+0000 7fd2fd094700 1 --2- 172.21.15.16:0/2794712179 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fd2dc06af00 0x7fd2dc06d3c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd2f815bf00 tx=0x7fd2ec009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:47.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.339+0000 7fd2ea7fc700 1 -- 172.21.15.16:0/2794712179 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd2f4053be0 con 0x7fd2f80ce4f0 2023-10-16T12:10:47.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.591+0000 7fd2ffaf9700 1 -- 172.21.15.16:0/2794712179 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fd2f80ccdb0 con 0x7fd2f80ce4f0 2023-10-16T12:10:47.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.591+0000 7fd2ea7fc700 1 -- 172.21.15.16:0/2794712179 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) v1 ==== 53+0+88 (secure 0 0 0) 0x7fd2f4017020 con 0x7fd2f80ce4f0 2023-10-16T12:10:47.593 INFO:teuthology.orchestra.run.smithi016.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-10-16T12:10:47.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.593+0000 7fd2ffaf9700 1 -- 172.21.15.16:0/2794712179 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fd2dc06af00 msgr2=0x7fd2dc06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:47.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.593+0000 7fd2ffaf9700 1 --2- 172.21.15.16:0/2794712179 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fd2dc06af00 0x7fd2dc06d3c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd2f815bf00 tx=0x7fd2ec009040 comp rx=0 tx=0).stop 2023-10-16T12:10:47.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.593+0000 7fd2ffaf9700 1 -- 172.21.15.16:0/2794712179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2f80ce4f0 msgr2=0x7fd2f8152140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:47.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.593+0000 7fd2ffaf9700 1 --2- 172.21.15.16:0/2794712179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2f80ce4f0 0x7fd2f8152140 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd2f400c4b0 tx=0x7fd2f400c7c0 comp rx=0 tx=0).stop 2023-10-16T12:10:47.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.594+0000 7fd2ffaf9700 1 -- 172.21.15.16:0/2794712179 shutdown_connections 2023-10-16T12:10:47.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.594+0000 7fd2ffaf9700 1 --2- 172.21.15.16:0/2794712179 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fd2dc06af00 0x7fd2dc06d3c0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:47.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.594+0000 7fd2ffaf9700 1 --2- 172.21.15.16:0/2794712179 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd2f815b630 0x7fd2f809b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:47.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.594+0000 7fd2ffaf9700 1 --2- 172.21.15.16:0/2794712179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2f80ce4f0 0x7fd2f8152140 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:47.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.594+0000 7fd2ffaf9700 1 --2- 172.21.15.16:0/2794712179 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd2f80d03b0 0x7fd2f809b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:47.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.594+0000 7fd2ffaf9700 1 -- 172.21.15.16:0/2794712179 >> 172.21.15.16:0/2794712179 conn(0x7fd2f80d14b0 msgr2=0x7fd2f80b6310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:47.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.594+0000 7fd2ffaf9700 1 -- 172.21.15.16:0/2794712179 shutdown_connections 2023-10-16T12:10:47.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:47.594+0000 7fd2ffaf9700 1 -- 172.21.15.16:0/2794712179 wait complete. 2023-10-16T12:10:47.607 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-16T12:10:47.608 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T12:10:47.608 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi016.front.sepia.ceph.com 2023-10-16T12:10:47.608 INFO:tasks.cephfs.mount:self.client.name = client.admin 2023-10-16T12:10:47.609 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2023-10-16T12:10:47.609 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-16T12:10:47.609 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-16T12:10:47.610 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T12:10:47.610 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2023-10-16T12:10:47.610 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:10:47.611 DEBUG:teuthology.orchestra.run.smithi016:> ip addr 2023-10-16T12:10:47.628 INFO:teuthology.orchestra.run.smithi016.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T12:10:47.628 INFO:teuthology.orchestra.run.smithi016.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T12:10:47.628 INFO:teuthology.orchestra.run.smithi016.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T12:10:47.629 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:10:47.629 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 ::1/128 scope host 2023-10-16T12:10:47.629 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:10:47.629 INFO:teuthology.orchestra.run.smithi016.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T12:10:47.629 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e0 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:10:47.630 INFO:teuthology.orchestra.run.smithi016.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T12:10:47.630 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e1 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:10:47.630 INFO:teuthology.orchestra.run.smithi016.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T12:10:47.630 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:64 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:10:47.630 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.21.15.16/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-16T12:10:47.631 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft 42288sec preferred_lft 42288sec 2023-10-16T12:10:47.631 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::ec4:7aff:febd:1564/64 scope link 2023-10-16T12:10:47.631 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:10:47.631 INFO:teuthology.orchestra.run.smithi016.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T12:10:47.631 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:65 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:10:47.632 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-10-16T12:10:47.632 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:10:47.633 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec bash -c ' 2023-10-16T12:10:47.633 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2023-10-16T12:10:47.633 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link add name ceph-brx type bridge 2023-10-16T12:10:47.634 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip addr flush dev ceph-brx 2023-10-16T12:10:47.634 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set ceph-brx up 2023-10-16T12:10:47.634 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-10-16T12:10:47.635 DEBUG:teuthology.orchestra.run.smithi016:> ') 2023-10-16T12:10:47.799 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:10:47.800 DEBUG:teuthology.orchestra.run.smithi016:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T12:10:47.828 INFO:teuthology.orchestra.run.smithi016.stdout:1 2023-10-16T12:10:47.829 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:10:47.830 DEBUG:teuthology.orchestra.run.smithi016:> route 2023-10-16T12:10:47.898 INFO:teuthology.orchestra.run.smithi016.stdout:Kernel IP routing table 2023-10-16T12:10:47.898 INFO:teuthology.orchestra.run.smithi016.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T12:10:47.898 INFO:teuthology.orchestra.run.smithi016.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-16T12:10:47.898 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-16T12:10:47.899 INFO:teuthology.orchestra.run.smithi016.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T12:10:47.899 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:10:47.899 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec bash -c ' 2023-10-16T12:10:47.900 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2023-10-16T12:10:47.900 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-16T12:10:47.900 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-16T12:10:47.901 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-16T12:10:47.901 DEBUG:teuthology.orchestra.run.smithi016:> ') 2023-10-16T12:10:48.153 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:10:48.153 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2023-10-16T12:10:48.171 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:10:48.172 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list-id 2023-10-16T12:10:48.228 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:10:48.229 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec bash -c ' 2023-10-16T12:10:48.229 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2023-10-16T12:10:48.229 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-10-16T12:10:48.230 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2023-10-16T12:10:48.230 DEBUG:teuthology.orchestra.run.smithi016:> ') 2023-10-16T12:10:48.324 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2023-10-16T12:10:48.324 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:10:48.324 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec bash -c ' 2023-10-16T12:10:48.325 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2023-10-16T12:10:48.325 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2023-10-16T12:10:48.325 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-10-16T12:10:48.326 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2023-10-16T12:10:48.326 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2023-10-16T12:10:48.326 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2023-10-16T12:10:48.327 DEBUG:teuthology.orchestra.run.smithi016:> ') 2023-10-16T12:10:48.385 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:10:48.388 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:10:48.538 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:10:48.539 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec bash -c ' 2023-10-16T12:10:48.539 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2023-10-16T12:10:48.539 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set brx.0 up 2023-10-16T12:10:48.540 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set dev brx.0 master ceph-brx 2023-10-16T12:10:48.540 DEBUG:teuthology.orchestra.run.smithi016:> ') 2023-10-16T12:10:48.647 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2023-10-16T12:10:48.648 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T12:10:48.648 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2023-10-16T12:10:48.668 INFO:teuthology.orchestra.run.smithi016.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2023-10-16T12:10:48.668 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T12:10:48.668 DEBUG:teuthology.orchestra.run.smithi016:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2023-10-16T12:10:48.724 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2023-10-16T12:10:48.792 DEBUG:teuthology.orchestra.run.smithi016:> cat /proc/self/mounts | awk '{print $2}' 2023-10-16T12:10:48.850 INFO:teuthology.orchestra.run.smithi016.stdout:/sys 2023-10-16T12:10:48.850 INFO:teuthology.orchestra.run.smithi016.stdout:/proc 2023-10-16T12:10:48.850 INFO:teuthology.orchestra.run.smithi016.stdout:/dev 2023-10-16T12:10:48.850 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/security 2023-10-16T12:10:48.851 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/shm 2023-10-16T12:10:48.851 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/pts 2023-10-16T12:10:48.851 INFO:teuthology.orchestra.run.smithi016.stdout:/run 2023-10-16T12:10:48.851 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup 2023-10-16T12:10:48.851 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup/systemd 2023-10-16T12:10:48.852 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/pstore 2023-10-16T12:10:48.852 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/bpf 2023-10-16T12:10:48.852 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup/cpuset 2023-10-16T12:10:48.852 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup/cpu,cpuacct 2023-10-16T12:10:48.852 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup/blkio 2023-10-16T12:10:48.852 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup/perf_event 2023-10-16T12:10:48.853 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup/rdma 2023-10-16T12:10:48.853 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup/freezer 2023-10-16T12:10:48.853 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup/memory 2023-10-16T12:10:48.853 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup/devices 2023-10-16T12:10:48.853 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup/hugetlb 2023-10-16T12:10:48.853 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup/net_cls,net_prio 2023-10-16T12:10:48.854 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup/pids 2023-10-16T12:10:48.854 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/tracing 2023-10-16T12:10:48.854 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/config 2023-10-16T12:10:48.854 INFO:teuthology.orchestra.run.smithi016.stdout:/ 2023-10-16T12:10:48.854 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/selinux 2023-10-16T12:10:48.855 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/mqueue 2023-10-16T12:10:48.855 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/hugepages 2023-10-16T12:10:48.855 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/debug 2023-10-16T12:10:48.855 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2023-10-16T12:10:48.855 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/fuse/connections 2023-10-16T12:10:48.855 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/nfs/rpc_pipefs 2023-10-16T12:10:48.856 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/1000 2023-10-16T12:10:48.856 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph 2023-10-16T12:10:48.856 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-0 2023-10-16T12:10:48.856 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-1 2023-10-16T12:10:48.856 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-2 2023-10-16T12:10:48.857 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-3 2023-10-16T12:10:48.857 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns 2023-10-16T12:10:48.857 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-10-16T12:10:48.857 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-10-16T12:10:48.857 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:10:48.858 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2023-10-16T12:10:48.907 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-10-16T12:10:48.908 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2023-10-16T12:10:48.950 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2023-10-16T12:10:48.977 DEBUG:teuthology.orchestra.run.smithi016:> cat /proc/self/mounts | awk '{print $2}' 2023-10-16T12:10:49.036 INFO:teuthology.orchestra.run.smithi016.stdout:/sys 2023-10-16T12:10:49.037 INFO:teuthology.orchestra.run.smithi016.stdout:/proc 2023-10-16T12:10:49.037 INFO:teuthology.orchestra.run.smithi016.stdout:/dev 2023-10-16T12:10:49.037 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/security 2023-10-16T12:10:49.037 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/shm 2023-10-16T12:10:49.037 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/pts 2023-10-16T12:10:49.038 INFO:teuthology.orchestra.run.smithi016.stdout:/run 2023-10-16T12:10:49.038 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup 2023-10-16T12:10:49.038 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup/systemd 2023-10-16T12:10:49.038 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/pstore 2023-10-16T12:10:49.038 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/bpf 2023-10-16T12:10:49.039 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup/cpuset 2023-10-16T12:10:49.039 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup/cpu,cpuacct 2023-10-16T12:10:49.039 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup/blkio 2023-10-16T12:10:49.039 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup/perf_event 2023-10-16T12:10:49.040 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup/rdma 2023-10-16T12:10:49.040 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup/freezer 2023-10-16T12:10:49.040 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup/memory 2023-10-16T12:10:49.040 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup/devices 2023-10-16T12:10:49.040 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup/hugetlb 2023-10-16T12:10:49.040 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup/net_cls,net_prio 2023-10-16T12:10:49.041 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup/pids 2023-10-16T12:10:49.041 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/tracing 2023-10-16T12:10:49.041 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/config 2023-10-16T12:10:49.041 INFO:teuthology.orchestra.run.smithi016.stdout:/ 2023-10-16T12:10:49.041 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/selinux 2023-10-16T12:10:49.042 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/mqueue 2023-10-16T12:10:49.042 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/hugepages 2023-10-16T12:10:49.042 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/debug 2023-10-16T12:10:49.042 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2023-10-16T12:10:49.042 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/fuse/connections 2023-10-16T12:10:49.042 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/nfs/rpc_pipefs 2023-10-16T12:10:49.043 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/1000 2023-10-16T12:10:49.043 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph 2023-10-16T12:10:49.043 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-0 2023-10-16T12:10:49.043 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-1 2023-10-16T12:10:49.043 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-2 2023-10-16T12:10:49.044 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-3 2023-10-16T12:10:49.044 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns 2023-10-16T12:10:49.044 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-10-16T12:10:49.044 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-10-16T12:10:49.044 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:10:49.045 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2023-10-16T12:10:49.070 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi016.stderr:2023-10-16T12:10:49.068+0000 7f883e3ed5c0 -1 init, newargv = 0x555bb277e740 newargc=15 2023-10-16T12:10:49.070 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi016.stderr:ceph-fuse[111111]: starting ceph client 2023-10-16T12:10:49.076 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi016.stderr:ceph-fuse[111111]: starting fuse 2023-10-16T12:10:49.087 INFO:teuthology.orchestra.run.smithi016.stdout:47 2023-10-16T12:10:49.087 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47] 2023-10-16T12:10:49.088 DEBUG:teuthology.orchestra.run.smithi016:> sudo stdin-killer -- python3 -c ' 2023-10-16T12:10:49.088 DEBUG:teuthology.orchestra.run.smithi016:> import glob 2023-10-16T12:10:49.088 DEBUG:teuthology.orchestra.run.smithi016:> import re 2023-10-16T12:10:49.089 DEBUG:teuthology.orchestra.run.smithi016:> import os 2023-10-16T12:10:49.089 DEBUG:teuthology.orchestra.run.smithi016:> import subprocess 2023-10-16T12:10:49.090 DEBUG:teuthology.orchestra.run.smithi016:> 2023-10-16T12:10:49.090 DEBUG:teuthology.orchestra.run.smithi016:> def _find_admin_socket(client_name): 2023-10-16T12:10:49.090 DEBUG:teuthology.orchestra.run.smithi016:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2023-10-16T12:10:49.091 DEBUG:teuthology.orchestra.run.smithi016:> files = glob.glob(asok_path) 2023-10-16T12:10:49.091 DEBUG:teuthology.orchestra.run.smithi016:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2023-10-16T12:10:49.091 DEBUG:teuthology.orchestra.run.smithi016:> 2023-10-16T12:10:49.092 DEBUG:teuthology.orchestra.run.smithi016:> # Given a non-glob path, it better be there 2023-10-16T12:10:49.092 DEBUG:teuthology.orchestra.run.smithi016:> if "*" not in asok_path: 2023-10-16T12:10:49.092 DEBUG:teuthology.orchestra.run.smithi016:> assert(len(files) == 1) 2023-10-16T12:10:49.093 DEBUG:teuthology.orchestra.run.smithi016:> return files[0] 2023-10-16T12:10:49.093 DEBUG:teuthology.orchestra.run.smithi016:> 2023-10-16T12:10:49.093 DEBUG:teuthology.orchestra.run.smithi016:> for f in files: 2023-10-16T12:10:49.094 DEBUG:teuthology.orchestra.run.smithi016:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-10-16T12:10:49.094 DEBUG:teuthology.orchestra.run.smithi016:> if os.path.exists("/proc/{0}".format(pid)): 2023-10-16T12:10:49.094 DEBUG:teuthology.orchestra.run.smithi016:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-10-16T12:10:49.095 DEBUG:teuthology.orchestra.run.smithi016:> contents = proc_f.read() 2023-10-16T12:10:49.095 DEBUG:teuthology.orchestra.run.smithi016:> if mountpoint in contents: 2023-10-16T12:10:49.095 DEBUG:teuthology.orchestra.run.smithi016:> return f 2023-10-16T12:10:49.096 DEBUG:teuthology.orchestra.run.smithi016:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-10-16T12:10:49.096 DEBUG:teuthology.orchestra.run.smithi016:> 2023-10-16T12:10:49.096 DEBUG:teuthology.orchestra.run.smithi016:> print(_find_admin_socket("client.admin")) 2023-10-16T12:10:49.097 DEBUG:teuthology.orchestra.run.smithi016:> ' 2023-10-16T12:10:49.213 INFO:teuthology.orchestra.run.smithi016.stdout:/var/run/ceph/ceph-client.admin.111111.asok 2023-10-16T12:10:49.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2023-10-16T12:10:49.223 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.111111.asok 2023-10-16T12:10:49.224 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:10:49.224 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.111111.asok status 2023-10-16T12:10:49.383 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-10-16T12:10:49.384 INFO:teuthology.orchestra.run.smithi016.stdout: "metadata": { 2023-10-16T12:10:49.384 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_sha1": "c7471068426370a3754c2250e7729280233d552b", 2023-10-16T12:10:49.384 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_version": "ceph version 18.0.0-6675-gc7471068 (c7471068426370a3754c2250e7729280233d552b) reef (dev)", 2023-10-16T12:10:49.384 INFO:teuthology.orchestra.run.smithi016.stdout: "entity_id": "admin", 2023-10-16T12:10:49.385 INFO:teuthology.orchestra.run.smithi016.stdout: "hostname": "smithi016", 2023-10-16T12:10:49.385 INFO:teuthology.orchestra.run.smithi016.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2023-10-16T12:10:49.385 INFO:teuthology.orchestra.run.smithi016.stdout: "pid": "111111", 2023-10-16T12:10:49.385 INFO:teuthology.orchestra.run.smithi016.stdout: "root": "/" 2023-10-16T12:10:49.385 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:10:49.386 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_count": 0, 2023-10-16T12:10:49.386 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_pinned_count": 0, 2023-10-16T12:10:49.386 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 4425, 2023-10-16T12:10:49.386 INFO:teuthology.orchestra.run.smithi016.stdout: "inst": { 2023-10-16T12:10:49.386 INFO:teuthology.orchestra.run.smithi016.stdout: "name": { 2023-10-16T12:10:49.387 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "client", 2023-10-16T12:10:49.387 INFO:teuthology.orchestra.run.smithi016.stdout: "num": 4425 2023-10-16T12:10:49.387 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:10:49.387 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2023-10-16T12:10:49.387 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-10-16T12:10:49.388 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.1:0", 2023-10-16T12:10:49.388 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1696070147 2023-10-16T12:10:49.388 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:10:49.388 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:10:49.388 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2023-10-16T12:10:49.389 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-10-16T12:10:49.389 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.1:0", 2023-10-16T12:10:49.389 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1696070147 2023-10-16T12:10:49.389 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:10:49.389 INFO:teuthology.orchestra.run.smithi016.stdout: "inst_str": "client.4425 192.168.0.1:0/1696070147", 2023-10-16T12:10:49.390 INFO:teuthology.orchestra.run.smithi016.stdout: "addr_str": "192.168.0.1:0/1696070147", 2023-10-16T12:10:49.390 INFO:teuthology.orchestra.run.smithi016.stdout: "inode_count": 1, 2023-10-16T12:10:49.390 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_epoch": 7, 2023-10-16T12:10:49.390 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch": 25, 2023-10-16T12:10:49.390 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch_barrier": 0, 2023-10-16T12:10:49.391 INFO:teuthology.orchestra.run.smithi016.stdout: "blocklisted": false, 2023-10-16T12:10:49.391 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs" 2023-10-16T12:10:49.391 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-10-16T12:10:49.394 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:10:49.394 DEBUG:teuthology.orchestra.run.smithi016:> stat --file-system '--printf=%T 2023-10-16T12:10:49.394 DEBUG:teuthology.orchestra.run.smithi016:> ' -- /home/ubuntu/cephtest/mnt.admin 2023-10-16T12:10:49.412 INFO:teuthology.orchestra.run.smithi016.stdout:fuseblk 2023-10-16T12:10:49.412 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2023-10-16T12:10:49.413 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:10:49.413 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2023-10-16T12:10:49.481 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:10:49.482 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.admin && exec bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2023-10-16T12:10:49.582 INFO:teuthology.orchestra.run.smithi016.stdout:# file: . 2023-10-16T12:10:49.582 INFO:teuthology.orchestra.run.smithi016.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2023-10-16T12:10:49.582 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:10:49.583 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:10:49.583 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:10:49.601 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:10:49.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.763+0000 7f435654b700 1 Processor -- start 2023-10-16T12:10:49.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.763+0000 7f435654b700 1 -- start start 2023-10-16T12:10:49.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.764+0000 7f435654b700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f4350160f10 0x7f43500ccec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:49.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.764+0000 7f435654b700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4350161a00 con 0x7f435006ff50 2023-10-16T12:10:49.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.764+0000 7f435654b700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43501625a0 con 0x7f43500ce3d0 2023-10-16T12:10:49.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.764+0000 7f435654b700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4350161700 con 0x7f4350160f10 2023-10-16T12:10:49.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.764+0000 7f434f7fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f435006ff50 0x7f4350161e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50676/0 (socket says 172.21.15.16:50676) 2023-10-16T12:10:49.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.764+0000 7f434f7fe700 1 -- 172.21.15.16:0/2274360427 learned_addr learned my addr 172.21.15.16:0/2274360427 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:10:49.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.765+0000 7f434ffff700 1 --2- 172.21.15.16:0/2274360427 >> v2:172.21.15.49:3300/0 conn(0x7f4350160f10 0x7f43500ccec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:49.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.765+0000 7f434effd700 1 -- 172.21.15.16:0/2274360427 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3365051430 0 0) 0x7f4350161a00 con 0x7f435006ff50 2023-10-16T12:10:49.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.765+0000 7f434effd700 1 -- 172.21.15.16:0/2274360427 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43380035b0 con 0x7f435006ff50 2023-10-16T12:10:49.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.765+0000 7f434effd700 1 -- 172.21.15.16:0/2274360427 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4143221985 0 0) 0x7f43380035b0 con 0x7f435006ff50 2023-10-16T12:10:49.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.765+0000 7f434effd700 1 -- 172.21.15.16:0/2274360427 >> v2:172.21.15.49:3300/0 conn(0x7f4350160f10 msgr2=0x7f43500ccec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:10:49.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.765+0000 7f434effd700 1 --2- 172.21.15.16:0/2274360427 >> v2:172.21.15.49:3300/0 conn(0x7f4350160f10 0x7f43500ccec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:49.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.765+0000 7f434effd700 1 -- 172.21.15.16:0/2274360427 >> v1:172.21.15.49:6789/0 conn(0x7f43500ce3d0 legacy=0x7f43500cab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:49.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.765+0000 7f434effd700 1 -- 172.21.15.16:0/2274360427 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43500cd6f0 con 0x7f435006ff50 2023-10-16T12:10:49.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.765+0000 7f434effd700 1 -- 172.21.15.16:0/2274360427 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f4344001670 con 0x7f435006ff50 2023-10-16T12:10:49.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.765+0000 7f434ffff700 1 --2- 172.21.15.16:0/2274360427 >> v2:172.21.15.49:3300/0 conn(0x7f4350160f10 0x7f43500ccec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:10:49.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.765+0000 7f434effd700 1 -- 172.21.15.16:0/2274360427 >> v1:172.21.15.16:6789/0 conn(0x7f435006ff50 legacy=0x7f4350161e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:49.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.765+0000 7f434effd700 1 --2- 172.21.15.16:0/2274360427 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4338003a00 0x7f4338003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:49.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.765+0000 7f434effd700 1 --2- 172.21.15.16:0/2274360427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4338004400 0x7f43380048d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:49.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.765+0000 7f434effd700 1 --2- 172.21.15.16:0/2274360427 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4338005170 0x7f433800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:49.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.765+0000 7f434effd700 1 -- 172.21.15.16:0/2274360427 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4350161700 con 0x7f4338004400 2023-10-16T12:10:49.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.765+0000 7f434effd700 1 -- 172.21.15.16:0/2274360427 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f433800c0b0 con 0x7f4338005170 2023-10-16T12:10:49.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.765+0000 7f434effd700 1 -- 172.21.15.16:0/2274360427 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f433800c270 con 0x7f4338003a00 2023-10-16T12:10:49.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.765+0000 7f434f7fe700 1 --2- 172.21.15.16:0/2274360427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4338004400 0x7f43380048d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:49.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.765+0000 7f434ffff700 1 --2- 172.21.15.16:0/2274360427 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4338003a00 0x7f4338003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:49.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.765+0000 7f4354ae8700 1 --2- 172.21.15.16:0/2274360427 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4338005170 0x7f433800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:49.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.766+0000 7f434f7fe700 1 -- 172.21.15.16:0/2274360427 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4338003a00 msgr2=0x7f4338003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:10:49.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.766+0000 7f434f7fe700 1 --2- 172.21.15.16:0/2274360427 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4338003a00 0x7f4338003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:49.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.766+0000 7f434f7fe700 1 -- 172.21.15.16:0/2274360427 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4338005170 msgr2=0x7f433800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:10:49.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.766+0000 7f434f7fe700 1 --2- 172.21.15.16:0/2274360427 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4338005170 0x7f433800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:49.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.766+0000 7f434f7fe700 1 -- 172.21.15.16:0/2274360427 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43380035b0 con 0x7f4338004400 2023-10-16T12:10:49.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.766+0000 7f4354ae8700 1 -- 172.21.15.16:0/2274360427 reap_dead start 2023-10-16T12:10:49.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.766+0000 7f434f7fe700 1 --2- 172.21.15.16:0/2274360427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4338004400 0x7f43380048d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4350070400 tx=0x7f4338005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80e139bd01a8b760 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:49.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.766+0000 7f434effd700 1 -- 172.21.15.16:0/2274360427 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f434400cd40 con 0x7f4338004400 2023-10-16T12:10:49.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.766+0000 7f434effd700 1 -- 172.21.15.16:0/2274360427 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4344005960 con 0x7f4338004400 2023-10-16T12:10:49.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.766+0000 7f434effd700 1 -- 172.21.15.16:0/2274360427 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4344005d10 con 0x7f4338004400 2023-10-16T12:10:49.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.766+0000 7f435654b700 1 -- 172.21.15.16:0/2274360427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4338004400 msgr2=0x7f43380048d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:49.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.766+0000 7f435654b700 1 --2- 172.21.15.16:0/2274360427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4338004400 0x7f43380048d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4350070400 tx=0x7f4338005880 comp rx=0 tx=0).stop 2023-10-16T12:10:49.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.766+0000 7f435654b700 1 -- 172.21.15.16:0/2274360427 shutdown_connections 2023-10-16T12:10:49.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.766+0000 7f435654b700 1 --2- 172.21.15.16:0/2274360427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4338004400 0x7f43380048d0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:49.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.766+0000 7f435654b700 1 -- 172.21.15.16:0/2274360427 >> 172.21.15.16:0/2274360427 conn(0x7f43500b2040 msgr2=0x7f43500b2420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:49.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.767+0000 7f435654b700 1 -- 172.21.15.16:0/2274360427 shutdown_connections 2023-10-16T12:10:49.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.767+0000 7f435654b700 1 -- 172.21.15.16:0/2274360427 wait complete. 2023-10-16T12:10:49.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.768+0000 7f435654b700 1 Processor -- start 2023-10-16T12:10:49.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.768+0000 7f435654b700 1 -- start start 2023-10-16T12:10:49.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.769+0000 7f435654b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4338004400 0x7f43500c2df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:49.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.769+0000 7f435654b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43500ca1b0 0x7f4350146440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:49.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.769+0000 7f435654b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f43500cc740 0x7f4350146980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:49.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.769+0000 7f435654b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43500c3480 con 0x7f43500ca1b0 2023-10-16T12:10:49.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.769+0000 7f435654b700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43500b3670 con 0x7f43500cc740 2023-10-16T12:10:49.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.769+0000 7f435654b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43500b30d0 con 0x7f4338004400 2023-10-16T12:10:49.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.769+0000 7f434f7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43500ca1b0 0x7f4350146440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:49.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.769+0000 7f434f7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43500ca1b0 0x7f4350146440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33184/0 (socket says 172.21.15.16:33184) 2023-10-16T12:10:49.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.769+0000 7f434f7fe700 1 -- 172.21.15.16:0/1810287712 learned_addr learned my addr 172.21.15.16:0/1810287712 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:10:49.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.769+0000 7f434ffff700 1 --2- 172.21.15.16:0/1810287712 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4338004400 0x7f43500c2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:49.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.769+0000 7f4354ae8700 1 --2- 172.21.15.16:0/1810287712 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f43500cc740 0x7f4350146980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:49.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.769+0000 7f434f7fe700 1 -- 172.21.15.16:0/1810287712 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4338004400 msgr2=0x7f43500c2df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:49.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.769+0000 7f434f7fe700 1 --2- 172.21.15.16:0/1810287712 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4338004400 0x7f43500c2df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:49.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.769+0000 7f434f7fe700 1 -- 172.21.15.16:0/1810287712 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f43500cc740 msgr2=0x7f4350146980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:49.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.769+0000 7f434f7fe700 1 --2- 172.21.15.16:0/1810287712 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f43500cc740 0x7f4350146980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:49.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.769+0000 7f434f7fe700 1 -- 172.21.15.16:0/1810287712 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43500c3690 con 0x7f43500ca1b0 2023-10-16T12:10:49.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.769+0000 7f434f7fe700 1 --2- 172.21.15.16:0/1810287712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43500ca1b0 0x7f4350146440 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f434400c8b0 tx=0x7f4344001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:49.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.769+0000 7f434cff9700 1 -- 172.21.15.16:0/1810287712 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4344001040 con 0x7f43500ca1b0 2023-10-16T12:10:49.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.769+0000 7f434cff9700 1 -- 172.21.15.16:0/1810287712 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4344004a80 con 0x7f43500ca1b0 2023-10-16T12:10:49.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.769+0000 7f435654b700 1 -- 172.21.15.16:0/1810287712 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43500b3390 con 0x7f43500ca1b0 2023-10-16T12:10:49.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.769+0000 7f434cff9700 1 -- 172.21.15.16:0/1810287712 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4344004df0 con 0x7f43500ca1b0 2023-10-16T12:10:49.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.770+0000 7f435654b700 1 -- 172.21.15.16:0/1810287712 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4350147020 con 0x7f43500ca1b0 2023-10-16T12:10:49.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.771+0000 7f435654b700 1 -- 172.21.15.16:0/1810287712 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43500b5fb0 con 0x7f43500ca1b0 2023-10-16T12:10:49.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.771+0000 7f434cff9700 1 -- 172.21.15.16:0/1810287712 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f4344007070 con 0x7f43500ca1b0 2023-10-16T12:10:49.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.771+0000 7f434cff9700 1 --2- 172.21.15.16:0/1810287712 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f433c06af00 0x7f433c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:49.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.771+0000 7f434cff9700 1 -- 172.21.15.16:0/1810287712 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f434408a460 con 0x7f43500ca1b0 2023-10-16T12:10:49.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.772+0000 7f434ffff700 1 --2- 172.21.15.16:0/1810287712 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f433c06af00 0x7f433c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:49.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.772+0000 7f434ffff700 1 --2- 172.21.15.16:0/1810287712 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f433c06af00 0x7f433c06d3c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4340009470 tx=0x7f4340009210 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:49.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.776+0000 7f434cff9700 1 -- 172.21.15.16:0/1810287712 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f43500b5fb0 con 0x7f43500ca1b0 2023-10-16T12:10:49.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.980+0000 7f435654b700 1 -- 172.21.15.16:0/1810287712 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f43500b5fb0 con 0x7f43500ca1b0 2023-10-16T12:10:49.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.981+0000 7f434cff9700 1 -- 172.21.15.16:0/1810287712 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18238 (secure 0 0 0) 0x7f43500b5fb0 con 0x7f43500ca1b0 2023-10-16T12:10:49.983 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:10:49.984 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":25,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:10:44.080141+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:10:36.935773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T12:10:38.004870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-16T12:10:41.672025+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:10:49.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.983+0000 7f435654b700 1 -- 172.21.15.16:0/1810287712 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f433c06af00 msgr2=0x7f433c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:49.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.983+0000 7f435654b700 1 --2- 172.21.15.16:0/1810287712 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f433c06af00 0x7f433c06d3c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4340009470 tx=0x7f4340009210 comp rx=0 tx=0).stop 2023-10-16T12:10:49.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.983+0000 7f435654b700 1 -- 172.21.15.16:0/1810287712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43500ca1b0 msgr2=0x7f4350146440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:49.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.983+0000 7f435654b700 1 --2- 172.21.15.16:0/1810287712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43500ca1b0 0x7f4350146440 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f434400c8b0 tx=0x7f4344001a00 comp rx=0 tx=0).stop 2023-10-16T12:10:49.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.983+0000 7f435654b700 1 -- 172.21.15.16:0/1810287712 shutdown_connections 2023-10-16T12:10:49.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.983+0000 7f435654b700 1 --2- 172.21.15.16:0/1810287712 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f433c06af00 0x7f433c06d3c0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:49.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.984+0000 7f435654b700 1 --2- 172.21.15.16:0/1810287712 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f43500cc740 0x7f4350146980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:49.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.984+0000 7f435654b700 1 --2- 172.21.15.16:0/1810287712 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4338004400 0x7f43500c2df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:49.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.984+0000 7f435654b700 1 --2- 172.21.15.16:0/1810287712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43500ca1b0 0x7f4350146440 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:49.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.984+0000 7f435654b700 1 -- 172.21.15.16:0/1810287712 >> 172.21.15.16:0/1810287712 conn(0x7f43500b2040 msgr2=0x7f43500d0900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:49.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.984+0000 7f435654b700 1 -- 172.21.15.16:0/1810287712 shutdown_connections 2023-10-16T12:10:49.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:49.984+0000 7f435654b700 1 -- 172.21.15.16:0/1810287712 wait complete. 2023-10-16T12:10:49.997 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T12:10:36.935773+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T12:10:38.004870+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-16T12:10:41.672025+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-16T12:10:49.998 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:10:50.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.125+0000 7f95d0205700 1 Processor -- start 2023-10-16T12:10:50.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.125+0000 7f95d0205700 1 -- start start 2023-10-16T12:10:50.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.126+0000 7f95d0205700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f95c81577a0 0x7f95c80cf1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:50.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.126+0000 7f95d0205700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f95c80cf700 0x7f95c80710a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:50.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.126+0000 7f95d0205700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f95c80715e0 0x7f95c815d890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:50.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.126+0000 7f95d0205700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95c80cfb90 con 0x7f95c80cf700 2023-10-16T12:10:50.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.126+0000 7f95d0205700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95c8072350 con 0x7f95c81577a0 2023-10-16T12:10:50.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.126+0000 7f95d0205700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95c8071d90 con 0x7f95c80715e0 2023-10-16T12:10:50.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.126+0000 7f95cd7a0700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f95c80cf700 0x7f95c80710a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:50.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.126+0000 7f95cd7a0700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f95c80cf700 0x7f95c80710a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33192/0 (socket says 172.21.15.16:33192) 2023-10-16T12:10:50.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.126+0000 7f95cd7a0700 1 -- 172.21.15.16:0/463244374 learned_addr learned my addr 172.21.15.16:0/463244374 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:10:50.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.126+0000 7f95ce7a2700 1 --2- 172.21.15.16:0/463244374 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f95c80715e0 0x7f95c815d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:50.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.126+0000 7f95cdfa1700 1 --2- 172.21.15.16:0/463244374 >> v2:172.21.15.49:3300/0 conn(0x7f95c81577a0 0x7f95c80cf1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:50.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.126+0000 7f95cd7a0700 1 -- 172.21.15.16:0/463244374 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f95c80715e0 msgr2=0x7f95c815d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:10:50.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.126+0000 7f95cd7a0700 1 --2- 172.21.15.16:0/463244374 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f95c80715e0 0x7f95c815d890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:50.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.126+0000 7f95cd7a0700 1 -- 172.21.15.16:0/463244374 >> v2:172.21.15.49:3300/0 conn(0x7f95c81577a0 msgr2=0x7f95c80cf1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:10:50.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.126+0000 7f95cd7a0700 1 --2- 172.21.15.16:0/463244374 >> v2:172.21.15.49:3300/0 conn(0x7f95c81577a0 0x7f95c80cf1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:50.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.126+0000 7f95cd7a0700 1 -- 172.21.15.16:0/463244374 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95c8072070 con 0x7f95c80cf700 2023-10-16T12:10:50.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.126+0000 7f95cdfa1700 1 --2- 172.21.15.16:0/463244374 >> v2:172.21.15.49:3300/0 conn(0x7f95c81577a0 0x7f95c80cf1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:10:50.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.126+0000 7f95cd7a0700 1 --2- 172.21.15.16:0/463244374 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95c80cf700 0x7f95c80710a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f95c80705a0 tx=0x7f95b800ad70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80291df61639da1d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:50.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.126+0000 7f95ccf9f700 1 -- 172.21.15.16:0/463244374 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95b800cb70 con 0x7f95c80cf700 2023-10-16T12:10:50.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.127+0000 7f95ccf9f700 1 -- 172.21.15.16:0/463244374 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95b800ccd0 con 0x7f95c80cf700 2023-10-16T12:10:50.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.127+0000 7f95ccf9f700 1 -- 172.21.15.16:0/463244374 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95b8018620 con 0x7f95c80cf700 2023-10-16T12:10:50.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.127+0000 7f95d0205700 1 -- 172.21.15.16:0/463244374 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95c80cf700 msgr2=0x7f95c80710a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:50.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.127+0000 7f95d0205700 1 --2- 172.21.15.16:0/463244374 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95c80cf700 0x7f95c80710a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f95c80705a0 tx=0x7f95b800ad70 comp rx=0 tx=0).stop 2023-10-16T12:10:50.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.127+0000 7f95d0205700 1 -- 172.21.15.16:0/463244374 shutdown_connections 2023-10-16T12:10:50.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.127+0000 7f95d0205700 1 --2- 172.21.15.16:0/463244374 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f95c80715e0 0x7f95c815d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:50.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.127+0000 7f95d0205700 1 --2- 172.21.15.16:0/463244374 >> v2:172.21.15.49:3300/0 conn(0x7f95c81577a0 0x7f95c80cf1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:50.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.127+0000 7f95d0205700 1 --2- 172.21.15.16:0/463244374 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95c80cf700 0x7f95c80710a0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:50.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.127+0000 7f95d0205700 1 -- 172.21.15.16:0/463244374 >> 172.21.15.16:0/463244374 conn(0x7f95c806a7e0 msgr2=0x7f95c806abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:50.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.127+0000 7f95d0205700 1 -- 172.21.15.16:0/463244374 shutdown_connections 2023-10-16T12:10:50.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.127+0000 7f95d0205700 1 -- 172.21.15.16:0/463244374 wait complete. 2023-10-16T12:10:50.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.128+0000 7f95d0205700 1 Processor -- start 2023-10-16T12:10:50.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.128+0000 7f95d0205700 1 -- start start 2023-10-16T12:10:50.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.129+0000 7f95d0205700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95c80715e0 0x7f95c81f3090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:50.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.129+0000 7f95d0205700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f95c80cf700 0x7f95c8147bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:50.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.129+0000 7f95d0205700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f95c81577a0 0x7f95c80c8710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:50.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.129+0000 7f95d0205700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95c80c92d0 con 0x7f95c80715e0 2023-10-16T12:10:50.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.129+0000 7f95d0205700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95c80c95b0 con 0x7f95c81577a0 2023-10-16T12:10:50.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.129+0000 7f95d0205700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95c80c9890 con 0x7f95c80cf700 2023-10-16T12:10:50.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.129+0000 7f95cdfa1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95c80715e0 0x7f95c81f3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:50.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.129+0000 7f95cdfa1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95c80715e0 0x7f95c81f3090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33196/0 (socket says 172.21.15.16:33196) 2023-10-16T12:10:50.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.129+0000 7f95cdfa1700 1 -- 172.21.15.16:0/3539992128 learned_addr learned my addr 172.21.15.16:0/3539992128 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:10:50.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.129+0000 7f95cd7a0700 1 --2- 172.21.15.16:0/3539992128 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f95c80cf700 0x7f95c8147bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:50.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.129+0000 7f95ce7a2700 1 --2- 172.21.15.16:0/3539992128 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f95c81577a0 0x7f95c80c8710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:50.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.130+0000 7f95cdfa1700 1 -- 172.21.15.16:0/3539992128 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f95c80cf700 msgr2=0x7f95c8147bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:50.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.130+0000 7f95cdfa1700 1 --2- 172.21.15.16:0/3539992128 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f95c80cf700 0x7f95c8147bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:50.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.130+0000 7f95cdfa1700 1 -- 172.21.15.16:0/3539992128 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f95c81577a0 msgr2=0x7f95c80c8710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:50.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.130+0000 7f95cdfa1700 1 --2- 172.21.15.16:0/3539992128 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f95c81577a0 0x7f95c80c8710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:50.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.130+0000 7f95cdfa1700 1 -- 172.21.15.16:0/3539992128 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95b800a6f0 con 0x7f95c80715e0 2023-10-16T12:10:50.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.130+0000 7f95cd7a0700 1 --2- 172.21.15.16:0/3539992128 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f95c80cf700 0x7f95c8147bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:10:50.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.130+0000 7f95cdfa1700 1 --2- 172.21.15.16:0/3539992128 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95c80715e0 0x7f95c81f3090 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f95c400c7f0 tx=0x7f95c400cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:50.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.130+0000 7f95be7fc700 1 -- 172.21.15.16:0/3539992128 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95c4011800 con 0x7f95c80715e0 2023-10-16T12:10:50.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.130+0000 7f95be7fc700 1 -- 172.21.15.16:0/3539992128 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95c4011960 con 0x7f95c80715e0 2023-10-16T12:10:50.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.130+0000 7f95d0205700 1 -- 172.21.15.16:0/3539992128 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95c80c9b70 con 0x7f95c80715e0 2023-10-16T12:10:50.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.130+0000 7f95be7fc700 1 -- 172.21.15.16:0/3539992128 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95c4011d10 con 0x7f95c80715e0 2023-10-16T12:10:50.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.130+0000 7f95d0205700 1 -- 172.21.15.16:0/3539992128 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95c81e3940 con 0x7f95c80715e0 2023-10-16T12:10:50.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.131+0000 7f95d0205700 1 -- 172.21.15.16:0/3539992128 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95c804ebb0 con 0x7f95c80715e0 2023-10-16T12:10:50.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.131+0000 7f95be7fc700 1 -- 172.21.15.16:0/3539992128 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f95c4018360 con 0x7f95c80715e0 2023-10-16T12:10:50.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.132+0000 7f95be7fc700 1 --2- 172.21.15.16:0/3539992128 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f95ac06af30 0x7f95ac06d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:50.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.132+0000 7f95be7fc700 1 -- 172.21.15.16:0/3539992128 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f95c4089820 con 0x7f95c80715e0 2023-10-16T12:10:50.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.132+0000 7f95cd7a0700 1 --2- 172.21.15.16:0/3539992128 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f95ac06af30 0x7f95ac06d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:50.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.132+0000 7f95cd7a0700 1 --2- 172.21.15.16:0/3539992128 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f95ac06af30 0x7f95ac06d3f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f95c8154910 tx=0x7f95b8012000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:50.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.136+0000 7f95be7fc700 1 -- 172.21.15.16:0/3539992128 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f95c40545e0 con 0x7f95c80715e0 2023-10-16T12:10:50.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.338+0000 7f95d0205700 1 -- 172.21.15.16:0/3539992128 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f95c8071100 con 0x7f95c80715e0 2023-10-16T12:10:50.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.338+0000 7f95be7fc700 1 -- 172.21.15.16:0/3539992128 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) v1 ==== 93+0+10 (secure 0 0 0) 0x7f95c4057c00 con 0x7f95c80715e0 2023-10-16T12:10:50.340 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T12:10:50.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.340+0000 7f95d0205700 1 -- 172.21.15.16:0/3539992128 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f95ac06af30 msgr2=0x7f95ac06d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:50.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.340+0000 7f95d0205700 1 --2- 172.21.15.16:0/3539992128 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f95ac06af30 0x7f95ac06d3f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f95c8154910 tx=0x7f95b8012000 comp rx=0 tx=0).stop 2023-10-16T12:10:50.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.340+0000 7f95d0205700 1 -- 172.21.15.16:0/3539992128 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95c80715e0 msgr2=0x7f95c81f3090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:50.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.340+0000 7f95d0205700 1 --2- 172.21.15.16:0/3539992128 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95c80715e0 0x7f95c81f3090 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f95c400c7f0 tx=0x7f95c400cbb0 comp rx=0 tx=0).stop 2023-10-16T12:10:50.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.340+0000 7f95d0205700 1 -- 172.21.15.16:0/3539992128 shutdown_connections 2023-10-16T12:10:50.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.340+0000 7f95d0205700 1 --2- 172.21.15.16:0/3539992128 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f95ac06af30 0x7f95ac06d3f0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:50.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.340+0000 7f95d0205700 1 --2- 172.21.15.16:0/3539992128 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f95c81577a0 0x7f95c80c8710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:50.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.340+0000 7f95d0205700 1 --2- 172.21.15.16:0/3539992128 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95c80715e0 0x7f95c81f3090 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:50.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.340+0000 7f95d0205700 1 --2- 172.21.15.16:0/3539992128 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f95c80cf700 0x7f95c8147bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:50.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.340+0000 7f95d0205700 1 -- 172.21.15.16:0/3539992128 >> 172.21.15.16:0/3539992128 conn(0x7f95c806a7e0 msgr2=0x7f95c8159690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:50.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.341+0000 7f95d0205700 1 -- 172.21.15.16:0/3539992128 shutdown_connections 2023-10-16T12:10:50.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.341+0000 7f95d0205700 1 -- 172.21.15.16:0/3539992128 wait complete. 2023-10-16T12:10:50.354 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T12:10:50.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.480+0000 7f4480a50700 1 Processor -- start 2023-10-16T12:10:50.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.480+0000 7f4480a50700 1 -- start start 2023-10-16T12:10:50.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.481+0000 7f4480a50700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f447c06e570 0x7f447c15e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:50.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.481+0000 7f4480a50700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f447c06a7e0 0x7f447c06ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:50.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.481+0000 7f4480a50700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f447c15b6a0 con 0x7f447c15b2c0 2023-10-16T12:10:50.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.481+0000 7f4480a50700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f447c06b1b0 con 0x7f447c06a7e0 2023-10-16T12:10:50.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.481+0000 7f4480a50700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f447c161b20 con 0x7f447c06e570 2023-10-16T12:10:50.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.481+0000 7f447ad9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f447c06a7e0 0x7f447c06ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:50.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.481+0000 7f447ad9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f447c06a7e0 0x7f447c06ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33208/0 (socket says 172.21.15.16:33208) 2023-10-16T12:10:50.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.481+0000 7f447ad9d700 1 -- 172.21.15.16:0/1550911620 learned_addr learned my addr 172.21.15.16:0/1550911620 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:10:50.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.481+0000 7f4479d9b700 1 --2- 172.21.15.16:0/1550911620 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f447c06e570 0x7f447c15e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:50.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.481+0000 7f447ad9d700 1 -- 172.21.15.16:0/1550911620 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f447c06e570 msgr2=0x7f447c15e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:10:50.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.481+0000 7f447ad9d700 1 --2- 172.21.15.16:0/1550911620 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f447c06e570 0x7f447c15e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:50.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.481+0000 7f447ad9d700 1 -- 172.21.15.16:0/1550911620 >> v1:172.21.15.49:6789/0 conn(0x7f447c15b2c0 legacy=0x7f447c161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:50.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.481+0000 7f447ad9d700 1 -- 172.21.15.16:0/1550911620 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f447c161e00 con 0x7f447c06a7e0 2023-10-16T12:10:50.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.482+0000 7f447ad9d700 1 --2- 172.21.15.16:0/1550911620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f447c06a7e0 0x7f447c06ac70 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4474009e50 tx=0x7f447400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79e967452d522fe3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:50.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.482+0000 7f447959a700 1 -- 172.21.15.16:0/1550911620 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f447400e070 con 0x7f447c06a7e0 2023-10-16T12:10:50.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.482+0000 7f447959a700 1 -- 172.21.15.16:0/1550911620 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f447400b9c0 con 0x7f447c06a7e0 2023-10-16T12:10:50.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.482+0000 7f447959a700 1 -- 172.21.15.16:0/1550911620 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f447400bd70 con 0x7f447c06a7e0 2023-10-16T12:10:50.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.482+0000 7f4480a50700 1 -- 172.21.15.16:0/1550911620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f447c06a7e0 msgr2=0x7f447c06ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:50.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.482+0000 7f4480a50700 1 --2- 172.21.15.16:0/1550911620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f447c06a7e0 0x7f447c06ac70 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4474009e50 tx=0x7f447400b330 comp rx=0 tx=0).stop 2023-10-16T12:10:50.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.482+0000 7f4480a50700 1 -- 172.21.15.16:0/1550911620 shutdown_connections 2023-10-16T12:10:50.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.482+0000 7f4480a50700 1 --2- 172.21.15.16:0/1550911620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f447c06a7e0 0x7f447c06ac70 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:50.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.482+0000 7f4480a50700 1 --2- 172.21.15.16:0/1550911620 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f447c06e570 0x7f447c15e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:50.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.482+0000 7f4480a50700 1 -- 172.21.15.16:0/1550911620 >> 172.21.15.16:0/1550911620 conn(0x7f447c0b1f40 msgr2=0x7f447c0b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:50.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.482+0000 7f4480a50700 1 -- 172.21.15.16:0/1550911620 shutdown_connections 2023-10-16T12:10:50.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.482+0000 7f4480a50700 1 -- 172.21.15.16:0/1550911620 wait complete. 2023-10-16T12:10:50.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.484+0000 7f4480a50700 1 Processor -- start 2023-10-16T12:10:50.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.484+0000 7f4480a50700 1 -- start start 2023-10-16T12:10:50.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.484+0000 7f4480a50700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f447c06a7e0 0x7f447c15bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:50.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.484+0000 7f4480a50700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f447c06e570 0x7f447c0c99a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:50.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.484+0000 7f4480a50700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f447c15b2c0 0x7f447c1589a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:50.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.484+0000 7f4480a50700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f447c0c9ee0 con 0x7f447c15b2c0 2023-10-16T12:10:50.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.484+0000 7f4480a50700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f447c0b2fb0 con 0x7f447c06a7e0 2023-10-16T12:10:50.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.484+0000 7f4480a50700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f447c0ca160 con 0x7f447c06e570 2023-10-16T12:10:50.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.485+0000 7f447ad9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f447c15b2c0 0x7f447c1589a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:50.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.485+0000 7f447ad9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f447c15b2c0 0x7f447c1589a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33214/0 (socket says 172.21.15.16:33214) 2023-10-16T12:10:50.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.485+0000 7f447ad9d700 1 -- 172.21.15.16:0/3753042139 learned_addr learned my addr 172.21.15.16:0/3753042139 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:10:50.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.485+0000 7f4479d9b700 1 --2- 172.21.15.16:0/3753042139 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f447c06e570 0x7f447c0c99a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:50.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.485+0000 7f447a59c700 1 --2- 172.21.15.16:0/3753042139 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f447c06a7e0 0x7f447c15bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:50.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.485+0000 7f447ad9d700 1 -- 172.21.15.16:0/3753042139 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f447c06e570 msgr2=0x7f447c0c99a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:50.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.485+0000 7f447ad9d700 1 --2- 172.21.15.16:0/3753042139 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f447c06e570 0x7f447c0c99a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:50.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.485+0000 7f447ad9d700 1 -- 172.21.15.16:0/3753042139 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f447c06a7e0 msgr2=0x7f447c15bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:50.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.485+0000 7f447ad9d700 1 --2- 172.21.15.16:0/3753042139 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f447c06a7e0 0x7f447c15bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:50.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.485+0000 7f447ad9d700 1 -- 172.21.15.16:0/3753042139 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4474009b30 con 0x7f447c15b2c0 2023-10-16T12:10:50.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.485+0000 7f447ad9d700 1 --2- 172.21.15.16:0/3753042139 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f447c15b2c0 0x7f447c1589a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f447c070620 tx=0x7f4474004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:50.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.485+0000 7f4472ffd700 1 -- 172.21.15.16:0/3753042139 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4474004800 con 0x7f447c15b2c0 2023-10-16T12:10:50.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.485+0000 7f4472ffd700 1 -- 172.21.15.16:0/3753042139 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4474004960 con 0x7f447c15b2c0 2023-10-16T12:10:50.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.485+0000 7f4480a50700 1 -- 172.21.15.16:0/3753042139 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f447c0ca440 con 0x7f447c15b2c0 2023-10-16T12:10:50.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.485+0000 7f4472ffd700 1 -- 172.21.15.16:0/3753042139 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f447401b670 con 0x7f447c15b2c0 2023-10-16T12:10:50.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.485+0000 7f4480a50700 1 -- 172.21.15.16:0/3753042139 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f447c158ee0 con 0x7f447c15b2c0 2023-10-16T12:10:50.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.486+0000 7f4480a50700 1 -- 172.21.15.16:0/3753042139 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f447c04ebb0 con 0x7f447c15b2c0 2023-10-16T12:10:50.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.487+0000 7f4472ffd700 1 -- 172.21.15.16:0/3753042139 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f447401b7d0 con 0x7f447c15b2c0 2023-10-16T12:10:50.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.487+0000 7f4472ffd700 1 --2- 172.21.15.16:0/3753042139 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f445c06af00 0x7f445c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:50.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.487+0000 7f4472ffd700 1 -- 172.21.15.16:0/3753042139 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f447408b0b0 con 0x7f447c15b2c0 2023-10-16T12:10:50.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.487+0000 7f447a59c700 1 --2- 172.21.15.16:0/3753042139 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f445c06af00 0x7f445c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:50.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.488+0000 7f447a59c700 1 --2- 172.21.15.16:0/3753042139 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f445c06af00 0x7f445c06d3c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f4464007fd0 tx=0x7f4464009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:50.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.492+0000 7f4472ffd700 1 -- 172.21.15.16:0/3753042139 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4474055f00 con 0x7f447c15b2c0 2023-10-16T12:10:50.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.693+0000 7f4480a50700 1 -- 172.21.15.16:0/3753042139 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f447c0b6370 con 0x7f447c15b2c0 2023-10-16T12:10:50.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.693+0000 7f4472ffd700 1 -- 172.21.15.16:0/3753042139 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4474059520 con 0x7f447c15b2c0 2023-10-16T12:10:50.695 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-10-16T12:10:50.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.695+0000 7f4480a50700 1 -- 172.21.15.16:0/3753042139 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f445c06af00 msgr2=0x7f445c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:50.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.695+0000 7f4480a50700 1 --2- 172.21.15.16:0/3753042139 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f445c06af00 0x7f445c06d3c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f4464007fd0 tx=0x7f4464009040 comp rx=0 tx=0).stop 2023-10-16T12:10:50.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.695+0000 7f4480a50700 1 -- 172.21.15.16:0/3753042139 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f447c15b2c0 msgr2=0x7f447c1589a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:50.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.695+0000 7f4480a50700 1 --2- 172.21.15.16:0/3753042139 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f447c15b2c0 0x7f447c1589a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f447c070620 tx=0x7f4474004ad0 comp rx=0 tx=0).stop 2023-10-16T12:10:50.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.695+0000 7f4480a50700 1 -- 172.21.15.16:0/3753042139 shutdown_connections 2023-10-16T12:10:50.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.695+0000 7f4480a50700 1 --2- 172.21.15.16:0/3753042139 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f445c06af00 0x7f445c06d3c0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:50.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.695+0000 7f4480a50700 1 --2- 172.21.15.16:0/3753042139 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f447c15b2c0 0x7f447c1589a0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:50.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.695+0000 7f4480a50700 1 --2- 172.21.15.16:0/3753042139 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f447c06a7e0 0x7f447c15bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:50.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.695+0000 7f4480a50700 1 --2- 172.21.15.16:0/3753042139 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f447c06e570 0x7f447c0c99a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:50.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.695+0000 7f4480a50700 1 -- 172.21.15.16:0/3753042139 >> 172.21.15.16:0/3753042139 conn(0x7f447c0b1f40 msgr2=0x7f447c0cce00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:50.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.695+0000 7f4480a50700 1 -- 172.21.15.16:0/3753042139 shutdown_connections 2023-10-16T12:10:50.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.696+0000 7f4480a50700 1 -- 172.21.15.16:0/3753042139 wait complete. 2023-10-16T12:10:50.709 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T12:10:50.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.835+0000 7f31fda0d700 1 Processor -- start 2023-10-16T12:10:50.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.835+0000 7f31fda0d700 1 -- start start 2023-10-16T12:10:50.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.836+0000 7f31fda0d700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f31f815d8d0 0x7f31f80b2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:50.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.836+0000 7f31fda0d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f31f80b8fa0 0x7f31f80b6720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:50.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.836+0000 7f31fda0d700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31f80b2f30 con 0x7f31f80b6c60 2023-10-16T12:10:50.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.836+0000 7f31fda0d700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31f80d0bb0 con 0x7f31f815d8d0 2023-10-16T12:10:50.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.836+0000 7f31fda0d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31f815dcb0 con 0x7f31f80b8fa0 2023-10-16T12:10:50.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.836+0000 7f31f77fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f31f80b6c60 0x7f31f80ce350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50728/0 (socket says 172.21.15.16:50728) 2023-10-16T12:10:50.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.836+0000 7f31f77fe700 1 -- 172.21.15.16:0/4085369633 learned_addr learned my addr 172.21.15.16:0/4085369633 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:10:50.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.836+0000 7f31f67fc700 1 --2- 172.21.15.16:0/4085369633 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f31f80b8fa0 0x7f31f80b6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:50.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.836+0000 7f31f6ffd700 1 --2- 172.21.15.16:0/4085369633 >> v2:172.21.15.49:3300/0 conn(0x7f31f815d8d0 0x7f31f80b2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:50.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.836+0000 7f31f5ffb700 1 -- 172.21.15.16:0/4085369633 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1223420342 0 0) 0x7f31f80b2f30 con 0x7f31f80b6c60 2023-10-16T12:10:50.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.836+0000 7f31f5ffb700 1 -- 172.21.15.16:0/4085369633 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31e00035b0 con 0x7f31f80b6c60 2023-10-16T12:10:50.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.837+0000 7f31f5ffb700 1 -- 172.21.15.16:0/4085369633 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2825111041 0 0) 0x7f31e00035b0 con 0x7f31f80b6c60 2023-10-16T12:10:50.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.837+0000 7f31f5ffb700 1 -- 172.21.15.16:0/4085369633 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f31f80b8fa0 msgr2=0x7f31f80b6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:10:50.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.837+0000 7f31f5ffb700 1 --2- 172.21.15.16:0/4085369633 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f31f80b8fa0 0x7f31f80b6720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:50.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.837+0000 7f31f5ffb700 1 -- 172.21.15.16:0/4085369633 >> v2:172.21.15.49:3300/0 conn(0x7f31f815d8d0 msgr2=0x7f31f80b2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:10:50.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.837+0000 7f31f5ffb700 1 --2- 172.21.15.16:0/4085369633 >> v2:172.21.15.49:3300/0 conn(0x7f31f815d8d0 0x7f31f80b2830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:50.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.837+0000 7f31f5ffb700 1 -- 172.21.15.16:0/4085369633 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31f80d1c40 con 0x7f31f80b6c60 2023-10-16T12:10:50.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.837+0000 7f31f5ffb700 1 -- 172.21.15.16:0/4085369633 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f31f00019f0 con 0x7f31f80b6c60 2023-10-16T12:10:50.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.837+0000 7f31f5ffb700 1 -- 172.21.15.16:0/4085369633 >> v1:172.21.15.16:6789/0 conn(0x7f31f80b6c60 legacy=0x7f31f80ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:50.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.837+0000 7f31f5ffb700 1 --2- 172.21.15.16:0/4085369633 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f31e0003a00 0x7f31e0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:50.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.837+0000 7f31f5ffb700 1 --2- 172.21.15.16:0/4085369633 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31e0004850 0x7f31e0004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:50.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.837+0000 7f31f5ffb700 1 --2- 172.21.15.16:0/4085369633 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f31f80b8fa0 0x7f31e0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:50.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.837+0000 7f31f5ffb700 1 -- 172.21.15.16:0/4085369633 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31f80d0bb0 con 0x7f31e0004850 2023-10-16T12:10:50.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.837+0000 7f31f5ffb700 1 -- 172.21.15.16:0/4085369633 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31f815dcb0 con 0x7f31e0003a00 2023-10-16T12:10:50.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.837+0000 7f31f5ffb700 1 -- 172.21.15.16:0/4085369633 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31e000bac0 con 0x7f31f80b8fa0 2023-10-16T12:10:50.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.837+0000 7f31f67fc700 1 --2- 172.21.15.16:0/4085369633 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31e0004850 0x7f31e0004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:50.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.837+0000 7f31f6ffd700 1 --2- 172.21.15.16:0/4085369633 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f31e0003a00 0x7f31e0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:50.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.837+0000 7f31f77fe700 1 --2- 172.21.15.16:0/4085369633 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f31f80b8fa0 0x7f31e0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:50.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.837+0000 7f31f67fc700 1 -- 172.21.15.16:0/4085369633 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f31f80b8fa0 msgr2=0x7f31e0005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:10:50.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.837+0000 7f31f67fc700 1 --2- 172.21.15.16:0/4085369633 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f31f80b8fa0 0x7f31e0005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:50.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.837+0000 7f31f67fc700 1 -- 172.21.15.16:0/4085369633 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f31e0003a00 msgr2=0x7f31e0003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:10:50.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.837+0000 7f31f67fc700 1 --2- 172.21.15.16:0/4085369633 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f31e0003a00 0x7f31e0003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:50.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.837+0000 7f31f67fc700 1 -- 172.21.15.16:0/4085369633 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31e00035b0 con 0x7f31e0004850 2023-10-16T12:10:50.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.837+0000 7f31f67fc700 1 --2- 172.21.15.16:0/4085369633 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31e0004850 0x7f31e0004cd0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f31e8009b60 tx=0x7f31e0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38c3f9f73190a592 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:50.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.837+0000 7f31f5ffb700 1 -- 172.21.15.16:0/4085369633 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31e800e6c0 con 0x7f31e0004850 2023-10-16T12:10:50.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.837+0000 7f31f5ffb700 1 -- 172.21.15.16:0/4085369633 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31e800e820 con 0x7f31e0004850 2023-10-16T12:10:50.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.837+0000 7f31f5ffb700 1 -- 172.21.15.16:0/4085369633 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31e800eb90 con 0x7f31e0004850 2023-10-16T12:10:50.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.838+0000 7f31fda0d700 1 -- 172.21.15.16:0/4085369633 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31e0004850 msgr2=0x7f31e0004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:50.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.838+0000 7f31fda0d700 1 --2- 172.21.15.16:0/4085369633 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31e0004850 0x7f31e0004cd0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f31e8009b60 tx=0x7f31e0005880 comp rx=0 tx=0).stop 2023-10-16T12:10:50.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.838+0000 7f31f77fe700 1 -- 172.21.15.16:0/4085369633 reap_dead start 2023-10-16T12:10:50.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.838+0000 7f31fda0d700 1 -- 172.21.15.16:0/4085369633 shutdown_connections 2023-10-16T12:10:50.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.838+0000 7f31fda0d700 1 -- 172.21.15.16:0/4085369633 >> 172.21.15.16:0/4085369633 conn(0x7f31f8072330 msgr2=0x7f31f8072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:50.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.838+0000 7f31fda0d700 1 -- 172.21.15.16:0/4085369633 shutdown_connections 2023-10-16T12:10:50.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.838+0000 7f31fda0d700 1 -- 172.21.15.16:0/4085369633 wait complete. 2023-10-16T12:10:50.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.840+0000 7f31fda0d700 1 Processor -- start 2023-10-16T12:10:50.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.840+0000 7f31fda0d700 1 -- start start 2023-10-16T12:10:50.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.841+0000 7f31fda0d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f31e0004850 0x7f31f81f6640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:50.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.841+0000 7f31fda0d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f31f80b5b20 0x7f31f80cb6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:50.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.841+0000 7f31fda0d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31f80ca700 0x7f31f815de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:50.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.841+0000 7f31fda0d700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31f80b5fa0 con 0x7f31f80ca700 2023-10-16T12:10:50.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.841+0000 7f31fda0d700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31f81f3180 con 0x7f31f80b5b20 2023-10-16T12:10:50.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.841+0000 7f31fda0d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31f81f6c20 con 0x7f31e0004850 2023-10-16T12:10:50.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.841+0000 7f31f77fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31f80ca700 0x7f31f815de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:50.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.841+0000 7f31f6ffd700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f31e0004850 0x7f31f81f6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:50.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.841+0000 7f31f77fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31f80ca700 0x7f31f815de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33234/0 (socket says 172.21.15.16:33234) 2023-10-16T12:10:50.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.841+0000 7f31f77fe700 1 -- 172.21.15.16:0/1062338302 learned_addr learned my addr 172.21.15.16:0/1062338302 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:10:50.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.841+0000 7f31f67fc700 1 --2- 172.21.15.16:0/1062338302 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f31f80b5b20 0x7f31f80cb6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:50.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.841+0000 7f31f77fe700 1 -- 172.21.15.16:0/1062338302 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f31e0004850 msgr2=0x7f31f81f6640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:50.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.841+0000 7f31f77fe700 1 --2- 172.21.15.16:0/1062338302 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f31e0004850 0x7f31f81f6640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:50.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.841+0000 7f31f77fe700 1 -- 172.21.15.16:0/1062338302 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f31f80b5b20 msgr2=0x7f31f80cb6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:50.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.841+0000 7f31f77fe700 1 --2- 172.21.15.16:0/1062338302 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f31f80b5b20 0x7f31f80cb6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:50.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.841+0000 7f31f77fe700 1 -- 172.21.15.16:0/1062338302 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31f81f6f00 con 0x7f31f80ca700 2023-10-16T12:10:50.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.841+0000 7f31f6ffd700 1 --2- 172.21.15.16:0/1062338302 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f31e0004850 0x7f31f81f6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:10:50.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.841+0000 7f31f77fe700 1 --2- 172.21.15.16:0/1062338302 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31f80ca700 0x7f31f815de80 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f31f80ca6b0 tx=0x7f31f000a640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:50.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.842+0000 7f31dffff700 1 -- 172.21.15.16:0/1062338302 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31f000ad50 con 0x7f31f80ca700 2023-10-16T12:10:50.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.842+0000 7f31dffff700 1 -- 172.21.15.16:0/1062338302 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31f00133f0 con 0x7f31f80ca700 2023-10-16T12:10:50.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.842+0000 7f31fda0d700 1 -- 172.21.15.16:0/1062338302 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31f81f71e0 con 0x7f31f80ca700 2023-10-16T12:10:50.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.842+0000 7f31dffff700 1 -- 172.21.15.16:0/1062338302 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31f00137a0 con 0x7f31f80ca700 2023-10-16T12:10:50.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.842+0000 7f31fda0d700 1 -- 172.21.15.16:0/1062338302 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31f80cc250 con 0x7f31f80ca700 2023-10-16T12:10:50.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.843+0000 7f31fda0d700 1 -- 172.21.15.16:0/1062338302 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31f81f3330 con 0x7f31f80ca700 2023-10-16T12:10:50.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.843+0000 7f31dffff700 1 -- 172.21.15.16:0/1062338302 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f31f0013900 con 0x7f31f80ca700 2023-10-16T12:10:50.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.844+0000 7f31dffff700 1 --2- 172.21.15.16:0/1062338302 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f31e406af00 0x7f31e406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:50.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.844+0000 7f31dffff700 1 -- 172.21.15.16:0/1062338302 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f31f008a2d0 con 0x7f31f80ca700 2023-10-16T12:10:50.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.844+0000 7f31f6ffd700 1 --2- 172.21.15.16:0/1062338302 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f31e406af00 0x7f31e406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:50.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.844+0000 7f31f6ffd700 1 --2- 172.21.15.16:0/1062338302 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f31e406af00 0x7f31e406d3c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f31ec014400 tx=0x7f31ec009210 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:50.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:50.848+0000 7f31dffff700 1 -- 172.21.15.16:0/1062338302 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f31f81f3330 con 0x7f31f80ca700 2023-10-16T12:10:51.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.048+0000 7f31fda0d700 1 -- 172.21.15.16:0/1062338302 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f31f81f32f0 con 0x7f31f80ca700 2023-10-16T12:10:51.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.048+0000 7f31dffff700 1 -- 172.21.15.16:0/1062338302 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) v1 ==== 100+0+11 (secure 0 0 0) 0x7f31f0055080 con 0x7f31f80ca700 2023-10-16T12:10:51.050 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-10-16T12:10:51.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.050+0000 7f31fda0d700 1 -- 172.21.15.16:0/1062338302 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f31e406af00 msgr2=0x7f31e406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:51.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.050+0000 7f31fda0d700 1 --2- 172.21.15.16:0/1062338302 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f31e406af00 0x7f31e406d3c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f31ec014400 tx=0x7f31ec009210 comp rx=0 tx=0).stop 2023-10-16T12:10:51.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.050+0000 7f31fda0d700 1 -- 172.21.15.16:0/1062338302 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31f80ca700 msgr2=0x7f31f815de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:51.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.050+0000 7f31fda0d700 1 --2- 172.21.15.16:0/1062338302 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31f80ca700 0x7f31f815de80 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f31f80ca6b0 tx=0x7f31f000a640 comp rx=0 tx=0).stop 2023-10-16T12:10:51.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.050+0000 7f31fda0d700 1 -- 172.21.15.16:0/1062338302 shutdown_connections 2023-10-16T12:10:51.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.050+0000 7f31fda0d700 1 --2- 172.21.15.16:0/1062338302 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f31e406af00 0x7f31e406d3c0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:51.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.050+0000 7f31fda0d700 1 --2- 172.21.15.16:0/1062338302 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31f80ca700 0x7f31f815de80 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:51.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.050+0000 7f31fda0d700 1 --2- 172.21.15.16:0/1062338302 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f31e0004850 0x7f31f81f6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:51.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.050+0000 7f31fda0d700 1 --2- 172.21.15.16:0/1062338302 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f31f80b5b20 0x7f31f80cb6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:51.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.050+0000 7f31fda0d700 1 -- 172.21.15.16:0/1062338302 >> 172.21.15.16:0/1062338302 conn(0x7f31f8072330 msgr2=0x7f31f815cae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:51.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.050+0000 7f31fda0d700 1 -- 172.21.15.16:0/1062338302 shutdown_connections 2023-10-16T12:10:51.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.050+0000 7f31fda0d700 1 -- 172.21.15.16:0/1062338302 wait complete. 2023-10-16T12:10:51.064 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-16T12:10:51.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.190+0000 7fe756ec4700 1 Processor -- start 2023-10-16T12:10:51.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.190+0000 7fe756ec4700 1 -- start start 2023-10-16T12:10:51.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.191+0000 7fe756ec4700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fe75015ec80 0x7fe7500cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:51.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.191+0000 7fe756ec4700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe75015f060 0x7fe7500ce400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:51.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.191+0000 7fe756ec4700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7500b2440 con 0x7fe7500d2110 2023-10-16T12:10:51.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.191+0000 7fe756ec4700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7500b2c50 con 0x7fe75015ec80 2023-10-16T12:10:51.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.191+0000 7fe756ec4700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7500d00e0 con 0x7fe75015f060 2023-10-16T12:10:51.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.191+0000 7fe755461700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe7500d2110 0x7fe7500b4440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50748/0 (socket says 172.21.15.16:50748) 2023-10-16T12:10:51.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.191+0000 7fe755461700 1 -- 172.21.15.16:0/1827313293 learned_addr learned my addr 172.21.15.16:0/1827313293 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:10:51.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.191+0000 7fe754c60700 1 --2- 172.21.15.16:0/1827313293 >> v2:172.21.15.49:3300/0 conn(0x7fe75015ec80 0x7fe7500cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:51.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.191+0000 7fe74ffff700 1 --2- 172.21.15.16:0/1827313293 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe75015f060 0x7fe7500ce400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:51.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.192+0000 7fe74f7fe700 1 -- 172.21.15.16:0/1827313293 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2828320000 0 0) 0x7fe7500b2440 con 0x7fe7500d2110 2023-10-16T12:10:51.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.192+0000 7fe74f7fe700 1 -- 172.21.15.16:0/1827313293 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7400035b0 con 0x7fe7500d2110 2023-10-16T12:10:51.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.192+0000 7fe754c60700 1 -- 172.21.15.16:0/1827313293 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe75015f060 msgr2=0x7fe7500ce400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:10:51.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.192+0000 7fe754c60700 1 --2- 172.21.15.16:0/1827313293 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe75015f060 0x7fe7500ce400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:51.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.192+0000 7fe754c60700 1 -- 172.21.15.16:0/1827313293 >> v1:172.21.15.16:6789/0 conn(0x7fe7500d2110 legacy=0x7fe7500b4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:51.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.192+0000 7fe754c60700 1 -- 172.21.15.16:0/1827313293 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7500cef60 con 0x7fe75015ec80 2023-10-16T12:10:51.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.192+0000 7fe754c60700 1 --2- 172.21.15.16:0/1827313293 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe75015ec80 0x7fe7500cdec0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe744009840 tx=0x7fe744009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=68fd20ba60ec9959 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:51.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.192+0000 7fe74f7fe700 1 -- 172.21.15.16:0/1827313293 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe744011830 con 0x7fe75015ec80 2023-10-16T12:10:51.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.192+0000 7fe74f7fe700 1 -- 172.21.15.16:0/1827313293 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe744011990 con 0x7fe75015ec80 2023-10-16T12:10:51.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.192+0000 7fe74f7fe700 1 -- 172.21.15.16:0/1827313293 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe744011d20 con 0x7fe75015ec80 2023-10-16T12:10:51.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.193+0000 7fe756ec4700 1 -- 172.21.15.16:0/1827313293 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe75015ec80 msgr2=0x7fe7500cdec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:51.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.193+0000 7fe756ec4700 1 --2- 172.21.15.16:0/1827313293 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe75015ec80 0x7fe7500cdec0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe744009840 tx=0x7fe744009b50 comp rx=0 tx=0).stop 2023-10-16T12:10:51.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.193+0000 7fe756ec4700 1 -- 172.21.15.16:0/1827313293 shutdown_connections 2023-10-16T12:10:51.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.193+0000 7fe756ec4700 1 --2- 172.21.15.16:0/1827313293 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe75015ec80 0x7fe7500cdec0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:51.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.193+0000 7fe756ec4700 1 --2- 172.21.15.16:0/1827313293 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe75015f060 0x7fe7500ce400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:51.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.193+0000 7fe756ec4700 1 -- 172.21.15.16:0/1827313293 >> 172.21.15.16:0/1827313293 conn(0x7fe7500d15f0 msgr2=0x7fe7500d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:51.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.193+0000 7fe756ec4700 1 -- 172.21.15.16:0/1827313293 shutdown_connections 2023-10-16T12:10:51.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.193+0000 7fe756ec4700 1 -- 172.21.15.16:0/1827313293 wait complete. 2023-10-16T12:10:51.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.194+0000 7fe756ec4700 1 Processor -- start 2023-10-16T12:10:51.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.194+0000 7fe756ec4700 1 -- start start 2023-10-16T12:10:51.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.195+0000 7fe756ec4700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7500d2110 0x7fe7501f2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:51.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.195+0000 7fe756ec4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe75015ec80 0x7fe7500caef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:51.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.195+0000 7fe756ec4700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe75015f060 0x7fe7500b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:51.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.195+0000 7fe756ec4700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe75015be20 con 0x7fe75015ec80 2023-10-16T12:10:51.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.195+0000 7fe756ec4700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7501550f0 con 0x7fe75015f060 2023-10-16T12:10:51.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.195+0000 7fe756ec4700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7501553d0 con 0x7fe7500d2110 2023-10-16T12:10:51.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.195+0000 7fe74ffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe75015ec80 0x7fe7500caef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:51.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.195+0000 7fe74ffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe75015ec80 0x7fe7500caef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33244/0 (socket says 172.21.15.16:33244) 2023-10-16T12:10:51.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.195+0000 7fe74ffff700 1 -- 172.21.15.16:0/4241065933 learned_addr learned my addr 172.21.15.16:0/4241065933 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:10:51.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.195+0000 7fe754c60700 1 --2- 172.21.15.16:0/4241065933 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7500d2110 0x7fe7501f2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:51.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.195+0000 7fe755461700 1 --2- 172.21.15.16:0/4241065933 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe75015f060 0x7fe7500b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:51.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.195+0000 7fe74ffff700 1 -- 172.21.15.16:0/4241065933 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7500d2110 msgr2=0x7fe7501f2ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:51.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.195+0000 7fe74ffff700 1 --2- 172.21.15.16:0/4241065933 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7500d2110 0x7fe7501f2ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:51.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.195+0000 7fe74ffff700 1 -- 172.21.15.16:0/4241065933 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe75015f060 msgr2=0x7fe7500b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:51.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.195+0000 7fe74ffff700 1 --2- 172.21.15.16:0/4241065933 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe75015f060 0x7fe7500b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:51.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.195+0000 7fe74ffff700 1 -- 172.21.15.16:0/4241065933 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7440094a0 con 0x7fe75015ec80 2023-10-16T12:10:51.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.196+0000 7fe74ffff700 1 --2- 172.21.15.16:0/4241065933 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe75015ec80 0x7fe7500caef0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fe73c00e840 tx=0x7fe73c00eb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:51.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.196+0000 7fe74d7fa700 1 -- 172.21.15.16:0/4241065933 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe73c009590 con 0x7fe75015ec80 2023-10-16T12:10:51.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.196+0000 7fe74d7fa700 1 -- 172.21.15.16:0/4241065933 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe73c013680 con 0x7fe75015ec80 2023-10-16T12:10:51.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.196+0000 7fe756ec4700 1 -- 172.21.15.16:0/4241065933 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe75015b7d0 con 0x7fe75015ec80 2023-10-16T12:10:51.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.196+0000 7fe74d7fa700 1 -- 172.21.15.16:0/4241065933 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe73c013a30 con 0x7fe75015ec80 2023-10-16T12:10:51.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.196+0000 7fe756ec4700 1 -- 172.21.15.16:0/4241065933 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe75015c150 con 0x7fe75015ec80 2023-10-16T12:10:51.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.197+0000 7fe756ec4700 1 -- 172.21.15.16:0/4241065933 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe750154eb0 con 0x7fe75015ec80 2023-10-16T12:10:51.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.197+0000 7fe74d7fa700 1 -- 172.21.15.16:0/4241065933 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fe73c013b90 con 0x7fe75015ec80 2023-10-16T12:10:51.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.198+0000 7fe74d7fa700 1 --2- 172.21.15.16:0/4241065933 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fe73406af00 0x7fe73406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:51.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.198+0000 7fe74d7fa700 1 -- 172.21.15.16:0/4241065933 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fe73c0896f0 con 0x7fe75015ec80 2023-10-16T12:10:51.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.198+0000 7fe754c60700 1 --2- 172.21.15.16:0/4241065933 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fe73406af00 0x7fe73406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:51.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.198+0000 7fe754c60700 1 --2- 172.21.15.16:0/4241065933 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fe73406af00 0x7fe73406d3c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe744000c00 tx=0x7fe744008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:51.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.202+0000 7fe74d7fa700 1 -- 172.21.15.16:0/4241065933 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe750154eb0 con 0x7fe75015ec80 2023-10-16T12:10:51.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.405+0000 7fe756ec4700 1 -- 172.21.15.16:0/4241065933 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fe750154eb0 con 0x7fe75015ec80 2023-10-16T12:10:51.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.405+0000 7fe74d7fa700 1 -- 172.21.15.16:0/4241065933 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) v1 ==== 103+0+11 (secure 0 0 0) 0x7fe750154eb0 con 0x7fe75015ec80 2023-10-16T12:10:51.408 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-10-16T12:10:51.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.407+0000 7fe756ec4700 1 -- 172.21.15.16:0/4241065933 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fe73406af00 msgr2=0x7fe73406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:51.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.407+0000 7fe756ec4700 1 --2- 172.21.15.16:0/4241065933 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fe73406af00 0x7fe73406d3c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe744000c00 tx=0x7fe744008030 comp rx=0 tx=0).stop 2023-10-16T12:10:51.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.407+0000 7fe756ec4700 1 -- 172.21.15.16:0/4241065933 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe75015ec80 msgr2=0x7fe7500caef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:51.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.407+0000 7fe756ec4700 1 --2- 172.21.15.16:0/4241065933 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe75015ec80 0x7fe7500caef0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fe73c00e840 tx=0x7fe73c00eb50 comp rx=0 tx=0).stop 2023-10-16T12:10:51.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.407+0000 7fe756ec4700 1 -- 172.21.15.16:0/4241065933 shutdown_connections 2023-10-16T12:10:51.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.407+0000 7fe756ec4700 1 --2- 172.21.15.16:0/4241065933 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fe73406af00 0x7fe73406d3c0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:51.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.407+0000 7fe756ec4700 1 --2- 172.21.15.16:0/4241065933 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe75015f060 0x7fe7500b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:51.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.407+0000 7fe756ec4700 1 --2- 172.21.15.16:0/4241065933 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7500d2110 0x7fe7501f2ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:51.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.407+0000 7fe756ec4700 1 --2- 172.21.15.16:0/4241065933 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe75015ec80 0x7fe7500caef0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:51.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.407+0000 7fe756ec4700 1 -- 172.21.15.16:0/4241065933 >> 172.21.15.16:0/4241065933 conn(0x7fe7500d15f0 msgr2=0x7fe750156d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:51.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.407+0000 7fe756ec4700 1 -- 172.21.15.16:0/4241065933 shutdown_connections 2023-10-16T12:10:51.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.407+0000 7fe756ec4700 1 -- 172.21.15.16:0/4241065933 wait complete. 2023-10-16T12:10:51.421 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T12:10:51.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.545+0000 7faed5c90700 1 Processor -- start 2023-10-16T12:10:51.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.545+0000 7faed5c90700 1 -- start start 2023-10-16T12:10:51.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.546+0000 7faed5c90700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7faed015b630 0x7faed00cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:51.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.546+0000 7faed5c90700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7faed015ba10 0x7faed00cdde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:51.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.546+0000 7faed5c90700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faed00ce320 0x7faed00cfca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:51.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.546+0000 7faed5c90700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faed00cdb20 con 0x7faed015ba10 2023-10-16T12:10:51.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.546+0000 7faed5c90700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faed00ce750 con 0x7faed015b630 2023-10-16T12:10:51.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.546+0000 7faed5c90700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faed00cc2a0 con 0x7faed00ce320 2023-10-16T12:10:51.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.546+0000 7faeceffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7faed015ba10 0x7faed00cdde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:51.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.546+0000 7faeceffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7faed015ba10 0x7faed00cdde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33252/0 (socket says 172.21.15.16:33252) 2023-10-16T12:10:51.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.546+0000 7faeceffd700 1 -- 172.21.15.16:0/1747653952 learned_addr learned my addr 172.21.15.16:0/1747653952 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:10:51.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.546+0000 7faecf7fe700 1 --2- 172.21.15.16:0/1747653952 >> v2:172.21.15.49:3300/0 conn(0x7faed015b630 0x7faed00cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:51.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.546+0000 7faecffff700 1 --2- 172.21.15.16:0/1747653952 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faed00ce320 0x7faed00cfca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:51.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.547+0000 7faeceffd700 1 -- 172.21.15.16:0/1747653952 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faed00ce320 msgr2=0x7faed00cfca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:10:51.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.547+0000 7faeceffd700 1 --2- 172.21.15.16:0/1747653952 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faed00ce320 0x7faed00cfca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:51.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.547+0000 7faeceffd700 1 -- 172.21.15.16:0/1747653952 >> v2:172.21.15.49:3300/0 conn(0x7faed015b630 msgr2=0x7faed00cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:10:51.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.547+0000 7faeceffd700 1 --2- 172.21.15.16:0/1747653952 >> v2:172.21.15.49:3300/0 conn(0x7faed015b630 0x7faed00cd5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:51.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.547+0000 7faeceffd700 1 -- 172.21.15.16:0/1747653952 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faed00cc540 con 0x7faed015ba10 2023-10-16T12:10:51.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.547+0000 7faeceffd700 1 --2- 172.21.15.16:0/1747653952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faed015ba10 0x7faed00cdde0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7faed00cefb0 tx=0x7faeb8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fd0805903f45faa server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:51.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.547+0000 7faece7fc700 1 -- 172.21.15.16:0/1747653952 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeb8011830 con 0x7faed015ba10 2023-10-16T12:10:51.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.547+0000 7faece7fc700 1 -- 172.21.15.16:0/1747653952 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faeb8011990 con 0x7faed015ba10 2023-10-16T12:10:51.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.547+0000 7faece7fc700 1 -- 172.21.15.16:0/1747653952 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeb8011d20 con 0x7faed015ba10 2023-10-16T12:10:51.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.547+0000 7faed5c90700 1 -- 172.21.15.16:0/1747653952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faed015ba10 msgr2=0x7faed00cdde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:51.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.547+0000 7faed5c90700 1 --2- 172.21.15.16:0/1747653952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faed015ba10 0x7faed00cdde0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7faed00cefb0 tx=0x7faeb8009bd0 comp rx=0 tx=0).stop 2023-10-16T12:10:51.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.548+0000 7faed5c90700 1 -- 172.21.15.16:0/1747653952 shutdown_connections 2023-10-16T12:10:51.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.548+0000 7faed5c90700 1 --2- 172.21.15.16:0/1747653952 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faed00ce320 0x7faed00cfca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:51.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.548+0000 7faed5c90700 1 --2- 172.21.15.16:0/1747653952 >> v2:172.21.15.49:3300/0 conn(0x7faed015b630 0x7faed00cd5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:51.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.548+0000 7faed5c90700 1 --2- 172.21.15.16:0/1747653952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faed015ba10 0x7faed00cdde0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:51.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.548+0000 7faed5c90700 1 -- 172.21.15.16:0/1747653952 >> 172.21.15.16:0/1747653952 conn(0x7faed00d14b0 msgr2=0x7faed00d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:51.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.548+0000 7faed5c90700 1 -- 172.21.15.16:0/1747653952 shutdown_connections 2023-10-16T12:10:51.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.548+0000 7faed5c90700 1 -- 172.21.15.16:0/1747653952 wait complete. 2023-10-16T12:10:51.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.549+0000 7faed5c90700 1 Processor -- start 2023-10-16T12:10:51.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.549+0000 7faed5c90700 1 -- start start 2023-10-16T12:10:51.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.550+0000 7faed5c90700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faed00ce320 0x7faed0149cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:51.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.550+0000 7faed5c90700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faed015b630 0x7faed014c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:51.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.550+0000 7faed5c90700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faed015ba10 0x7faed014caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:51.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.550+0000 7faed5c90700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faed014a4d0 con 0x7faed015ba10 2023-10-16T12:10:51.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.550+0000 7faed5c90700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faed014a230 con 0x7faed00ce320 2023-10-16T12:10:51.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.550+0000 7faed5c90700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faed00ca4e0 con 0x7faed015b630 2023-10-16T12:10:51.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.550+0000 7faecffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faed015ba10 0x7faed014caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:51.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.550+0000 7faecffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faed015ba10 0x7faed014caa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33260/0 (socket says 172.21.15.16:33260) 2023-10-16T12:10:51.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.550+0000 7faecffff700 1 -- 172.21.15.16:0/471516912 learned_addr learned my addr 172.21.15.16:0/471516912 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:10:51.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.550+0000 7faecf7fe700 1 --2- 172.21.15.16:0/471516912 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faed00ce320 0x7faed0149cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:51.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.550+0000 7faeceffd700 1 --2- 172.21.15.16:0/471516912 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faed015b630 0x7faed014c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:51.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.550+0000 7faecffff700 1 -- 172.21.15.16:0/471516912 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faed015b630 msgr2=0x7faed014c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:51.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.550+0000 7faecffff700 1 --2- 172.21.15.16:0/471516912 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faed015b630 0x7faed014c560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:51.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.550+0000 7faecffff700 1 -- 172.21.15.16:0/471516912 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faed00ce320 msgr2=0x7faed0149cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:51.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.550+0000 7faecffff700 1 --2- 172.21.15.16:0/471516912 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faed00ce320 0x7faed0149cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:51.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.550+0000 7faecffff700 1 -- 172.21.15.16:0/471516912 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeb8009550 con 0x7faed015ba10 2023-10-16T12:10:51.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.550+0000 7faecffff700 1 --2- 172.21.15.16:0/471516912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faed015ba10 0x7faed014caa0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7faec800eed0 tx=0x7faec800ef00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:51.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.550+0000 7faec7fff700 1 -- 172.21.15.16:0/471516912 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec8010040 con 0x7faed015ba10 2023-10-16T12:10:51.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.550+0000 7faec7fff700 1 -- 172.21.15.16:0/471516912 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faec800b990 con 0x7faed015ba10 2023-10-16T12:10:51.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.550+0000 7faed5c90700 1 -- 172.21.15.16:0/471516912 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faed00ca7c0 con 0x7faed015ba10 2023-10-16T12:10:51.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.550+0000 7faec7fff700 1 -- 172.21.15.16:0/471516912 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec800bd40 con 0x7faed015ba10 2023-10-16T12:10:51.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.551+0000 7faed5c90700 1 -- 172.21.15.16:0/471516912 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faed00cad80 con 0x7faed015ba10 2023-10-16T12:10:51.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.552+0000 7faed5c90700 1 -- 172.21.15.16:0/471516912 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faed00caf70 con 0x7faed015ba10 2023-10-16T12:10:51.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.552+0000 7faec7fff700 1 -- 172.21.15.16:0/471516912 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7faec8014400 con 0x7faed015ba10 2023-10-16T12:10:51.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.552+0000 7faec7fff700 1 --2- 172.21.15.16:0/471516912 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7faeb006af00 0x7faeb006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:51.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.552+0000 7faec7fff700 1 -- 172.21.15.16:0/471516912 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7faec8089bf0 con 0x7faed015ba10 2023-10-16T12:10:51.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.552+0000 7faecf7fe700 1 --2- 172.21.15.16:0/471516912 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7faeb006af00 0x7faeb006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:51.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.553+0000 7faecf7fe700 1 --2- 172.21.15.16:0/471516912 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7faeb006af00 0x7faeb006d3c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7faed0150b70 tx=0x7faec00091c0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:51.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.557+0000 7faec7fff700 1 -- 172.21.15.16:0/471516912 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7faed00caf70 con 0x7faed015ba10 2023-10-16T12:10:51.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.778+0000 7faed5c90700 1 -- 172.21.15.16:0/471516912 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7faed014a640 con 0x7faed015ba10 2023-10-16T12:10:51.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.778+0000 7faec7fff700 1 -- 172.21.15.16:0/471516912 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v25) v1 ==== 80+0+0 (secure 0 0 0) 0x7faed014a640 con 0x7faed015ba10 2023-10-16T12:10:51.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.780+0000 7faed5c90700 1 -- 172.21.15.16:0/471516912 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7faeb006af00 msgr2=0x7faeb006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:51.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.780+0000 7faed5c90700 1 --2- 172.21.15.16:0/471516912 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7faeb006af00 0x7faeb006d3c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7faed0150b70 tx=0x7faec00091c0 comp rx=0 tx=0).stop 2023-10-16T12:10:51.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.780+0000 7faed5c90700 1 -- 172.21.15.16:0/471516912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faed015ba10 msgr2=0x7faed014caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:51.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.780+0000 7faed5c90700 1 --2- 172.21.15.16:0/471516912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faed015ba10 0x7faed014caa0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7faec800eed0 tx=0x7faec800ef00 comp rx=0 tx=0).stop 2023-10-16T12:10:51.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.780+0000 7faed5c90700 1 -- 172.21.15.16:0/471516912 shutdown_connections 2023-10-16T12:10:51.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.780+0000 7faed5c90700 1 --2- 172.21.15.16:0/471516912 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7faeb006af00 0x7faeb006d3c0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:51.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.780+0000 7faed5c90700 1 --2- 172.21.15.16:0/471516912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faed015ba10 0x7faed014caa0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:51.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.780+0000 7faed5c90700 1 --2- 172.21.15.16:0/471516912 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faed00ce320 0x7faed0149cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:51.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.780+0000 7faed5c90700 1 --2- 172.21.15.16:0/471516912 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faed015b630 0x7faed014c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:51.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.780+0000 7faed5c90700 1 -- 172.21.15.16:0/471516912 >> 172.21.15.16:0/471516912 conn(0x7faed00d14b0 msgr2=0x7faed00b4300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:51.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.780+0000 7faed5c90700 1 -- 172.21.15.16:0/471516912 shutdown_connections 2023-10-16T12:10:51.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.780+0000 7faed5c90700 1 -- 172.21.15.16:0/471516912 wait complete. 2023-10-16T12:10:51.785 INFO:teuthology.orchestra.run.smithi016.stderr:listed 0 entries 2023-10-16T12:10:51.794 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:10:51.794 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:10:51.812 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:10:51.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.976+0000 7f8b7a492700 1 Processor -- start 2023-10-16T12:10:51.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.976+0000 7f8b7a492700 1 -- start start 2023-10-16T12:10:51.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.977+0000 7f8b7a492700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8b7415b630 0x7f8b740cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:51.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.977+0000 7f8b7a492700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8b7415ba10 0x7f8b740cdde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:51.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.977+0000 7f8b7a492700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b740ce770 con 0x7f8b740ce320 2023-10-16T12:10:51.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.977+0000 7f8b7a492700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b740cdb20 con 0x7f8b7415ba10 2023-10-16T12:10:51.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.977+0000 7f8b7a492700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b740cc2a0 con 0x7f8b7415b630 2023-10-16T12:10:51.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.977+0000 7f8b737fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8b7415ba10 0x7f8b740cdde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:51.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.977+0000 7f8b737fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8b7415ba10 0x7f8b740cdde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33272/0 (socket says 172.21.15.16:33272) 2023-10-16T12:10:51.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.977+0000 7f8b78a2f700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f8b740ce320 0x7f8b740cfca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50780/0 (socket says 172.21.15.16:50780) 2023-10-16T12:10:51.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.977+0000 7f8b737fe700 1 -- 172.21.15.16:0/3494719522 learned_addr learned my addr 172.21.15.16:0/3494719522 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:10:51.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.977+0000 7f8b73fff700 1 --2- 172.21.15.16:0/3494719522 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8b7415b630 0x7f8b740cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:51.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.977+0000 7f8b737fe700 1 -- 172.21.15.16:0/3494719522 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8b7415b630 msgr2=0x7f8b740cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:10:51.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.977+0000 7f8b737fe700 1 --2- 172.21.15.16:0/3494719522 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8b7415b630 0x7f8b740cd5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:51.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.977+0000 7f8b737fe700 1 -- 172.21.15.16:0/3494719522 >> v1:172.21.15.16:6789/0 conn(0x7f8b740ce320 legacy=0x7f8b740cfca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:51.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.977+0000 7f8b737fe700 1 -- 172.21.15.16:0/3494719522 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b740cc540 con 0x7f8b7415ba10 2023-10-16T12:10:51.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.977+0000 7f8b72ffd700 1 -- 172.21.15.16:0/3494719522 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2611741756 0 0) 0x7f8b6c001490 con 0x7f8b740ce320 2023-10-16T12:10:51.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.977+0000 7f8b737fe700 1 --2- 172.21.15.16:0/3494719522 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b7415ba10 0x7f8b740cdde0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f8b680098f0 tx=0x7f8b68009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cbda5d2daaf5450 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:51.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.977+0000 7f8b72ffd700 1 -- 172.21.15.16:0/3494719522 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b68011830 con 0x7f8b7415ba10 2023-10-16T12:10:51.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.977+0000 7f8b72ffd700 1 -- 172.21.15.16:0/3494719522 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b68011990 con 0x7f8b7415ba10 2023-10-16T12:10:51.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.978+0000 7f8b72ffd700 1 -- 172.21.15.16:0/3494719522 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b68011d20 con 0x7f8b7415ba10 2023-10-16T12:10:51.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.978+0000 7f8b7a492700 1 -- 172.21.15.16:0/3494719522 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b7415ba10 msgr2=0x7f8b740cdde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:51.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.978+0000 7f8b7a492700 1 --2- 172.21.15.16:0/3494719522 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b7415ba10 0x7f8b740cdde0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f8b680098f0 tx=0x7f8b68009c00 comp rx=0 tx=0).stop 2023-10-16T12:10:51.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.978+0000 7f8b7a492700 1 -- 172.21.15.16:0/3494719522 shutdown_connections 2023-10-16T12:10:51.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.978+0000 7f8b7a492700 1 --2- 172.21.15.16:0/3494719522 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8b7415b630 0x7f8b740cd5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:51.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.978+0000 7f8b7a492700 1 --2- 172.21.15.16:0/3494719522 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b7415ba10 0x7f8b740cdde0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:51.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.978+0000 7f8b7a492700 1 -- 172.21.15.16:0/3494719522 >> 172.21.15.16:0/3494719522 conn(0x7f8b740d14b0 msgr2=0x7f8b740d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:51.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.978+0000 7f8b7a492700 1 -- 172.21.15.16:0/3494719522 shutdown_connections 2023-10-16T12:10:51.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.978+0000 7f8b7a492700 1 -- 172.21.15.16:0/3494719522 wait complete. 2023-10-16T12:10:51.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.980+0000 7f8b7a492700 1 Processor -- start 2023-10-16T12:10:51.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.980+0000 7f8b7a492700 1 -- start start 2023-10-16T12:10:51.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.981+0000 7f8b7a492700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8b740ce320 0x7f8b74151790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:51.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.981+0000 7f8b7a492700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b7415b630 0x7f8b740cac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:51.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.981+0000 7f8b7a492700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8b7415ba10 0x7f8b740cb170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:51.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.981+0000 7f8b7a492700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b741f33e0 con 0x7f8b7415b630 2023-10-16T12:10:51.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.981+0000 7f8b7a492700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b740c9200 con 0x7f8b7415ba10 2023-10-16T12:10:51.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.981+0000 7f8b7a492700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b740c94e0 con 0x7f8b740ce320 2023-10-16T12:10:51.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.981+0000 7f8b737fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b7415b630 0x7f8b740cac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:51.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.981+0000 7f8b737fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b7415b630 0x7f8b740cac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33278/0 (socket says 172.21.15.16:33278) 2023-10-16T12:10:51.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.981+0000 7f8b737fe700 1 -- 172.21.15.16:0/146034652 learned_addr learned my addr 172.21.15.16:0/146034652 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:10:51.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.981+0000 7f8b73fff700 1 --2- 172.21.15.16:0/146034652 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8b740ce320 0x7f8b74151790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:51.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.981+0000 7f8b78a2f700 1 --2- 172.21.15.16:0/146034652 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8b7415ba10 0x7f8b740cb170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:51.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.981+0000 7f8b737fe700 1 -- 172.21.15.16:0/146034652 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8b740ce320 msgr2=0x7f8b74151790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:51.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.981+0000 7f8b737fe700 1 --2- 172.21.15.16:0/146034652 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8b740ce320 0x7f8b74151790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:51.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.981+0000 7f8b737fe700 1 -- 172.21.15.16:0/146034652 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8b7415ba10 msgr2=0x7f8b740cb170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:51.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.981+0000 7f8b737fe700 1 --2- 172.21.15.16:0/146034652 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8b7415ba10 0x7f8b740cb170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:51.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.981+0000 7f8b737fe700 1 -- 172.21.15.16:0/146034652 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b68009550 con 0x7f8b7415b630 2023-10-16T12:10:51.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.981+0000 7f8b73fff700 1 --2- 172.21.15.16:0/146034652 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8b740ce320 0x7f8b74151790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:10:51.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.981+0000 7f8b737fe700 1 --2- 172.21.15.16:0/146034652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b7415b630 0x7f8b740cac30 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f8b74072680 tx=0x7f8b68004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:51.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.982+0000 7f8b70ff9700 1 -- 172.21.15.16:0/146034652 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b68004590 con 0x7f8b7415b630 2023-10-16T12:10:51.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.982+0000 7f8b70ff9700 1 -- 172.21.15.16:0/146034652 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b680036a0 con 0x7f8b7415b630 2023-10-16T12:10:51.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.982+0000 7f8b7a492700 1 -- 172.21.15.16:0/146034652 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b740c97c0 con 0x7f8b7415b630 2023-10-16T12:10:51.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.982+0000 7f8b70ff9700 1 -- 172.21.15.16:0/146034652 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b68003a30 con 0x7f8b7415b630 2023-10-16T12:10:51.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.982+0000 7f8b7a492700 1 -- 172.21.15.16:0/146034652 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b740c9d80 con 0x7f8b7415b630 2023-10-16T12:10:51.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.982+0000 7f8b7a492700 1 -- 172.21.15.16:0/146034652 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b740ccdc0 con 0x7f8b7415b630 2023-10-16T12:10:51.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.983+0000 7f8b70ff9700 1 -- 172.21.15.16:0/146034652 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f8b68003b90 con 0x7f8b7415b630 2023-10-16T12:10:51.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.984+0000 7f8b70ff9700 1 --2- 172.21.15.16:0/146034652 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f8b6006af00 0x7f8b6006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:51.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.984+0000 7f8b73fff700 1 --2- 172.21.15.16:0/146034652 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f8b6006af00 0x7f8b6006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:51.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.984+0000 7f8b70ff9700 1 -- 172.21.15.16:0/146034652 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f8b6808aa80 con 0x7f8b7415b630 2023-10-16T12:10:51.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.984+0000 7f8b73fff700 1 --2- 172.21.15.16:0/146034652 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f8b6006af00 0x7f8b6006d3c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8b64009c10 tx=0x7f8b64008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:51.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:51.988+0000 7f8b70ff9700 1 -- 172.21.15.16:0/146034652 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8b740ccdc0 con 0x7f8b7415b630 2023-10-16T12:10:52.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.195+0000 7f8b7a492700 1 -- 172.21.15.16:0/146034652 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8b740ccd90 con 0x7f8b7415b630 2023-10-16T12:10:52.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.196+0000 7f8b70ff9700 1 -- 172.21.15.16:0/146034652 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18238 (secure 0 0 0) 0x7f8b68015090 con 0x7f8b7415b630 2023-10-16T12:10:52.199 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:10:52.199 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":25,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:10:44.080141+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:10:36.935773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T12:10:38.004870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-16T12:10:41.672025+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:10:52.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.199+0000 7f8b7a492700 1 -- 172.21.15.16:0/146034652 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f8b6006af00 msgr2=0x7f8b6006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:52.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.199+0000 7f8b7a492700 1 --2- 172.21.15.16:0/146034652 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f8b6006af00 0x7f8b6006d3c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8b64009c10 tx=0x7f8b64008040 comp rx=0 tx=0).stop 2023-10-16T12:10:52.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.199+0000 7f8b7a492700 1 -- 172.21.15.16:0/146034652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b7415b630 msgr2=0x7f8b740cac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:52.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.199+0000 7f8b7a492700 1 --2- 172.21.15.16:0/146034652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b7415b630 0x7f8b740cac30 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f8b74072680 tx=0x7f8b68004750 comp rx=0 tx=0).stop 2023-10-16T12:10:52.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.199+0000 7f8b7a492700 1 -- 172.21.15.16:0/146034652 shutdown_connections 2023-10-16T12:10:52.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.199+0000 7f8b7a492700 1 --2- 172.21.15.16:0/146034652 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f8b6006af00 0x7f8b6006d3c0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:52.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.199+0000 7f8b7a492700 1 --2- 172.21.15.16:0/146034652 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8b7415ba10 0x7f8b740cb170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:52.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.199+0000 7f8b7a492700 1 --2- 172.21.15.16:0/146034652 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8b740ce320 0x7f8b74151790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:52.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.199+0000 7f8b7a492700 1 --2- 172.21.15.16:0/146034652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b7415b630 0x7f8b740cac30 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:52.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.199+0000 7f8b7a492700 1 -- 172.21.15.16:0/146034652 >> 172.21.15.16:0/146034652 conn(0x7f8b740d14b0 msgr2=0x7f8b740b65a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:52.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.199+0000 7f8b7a492700 1 -- 172.21.15.16:0/146034652 shutdown_connections 2023-10-16T12:10:52.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.199+0000 7f8b7a492700 1 -- 172.21.15.16:0/146034652 wait complete. 2023-10-16T12:10:52.213 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T12:10:36.935773+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T12:10:38.004870+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-16T12:10:41.672025+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-16T12:10:52.214 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:10:52.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.340+0000 7f3939c61700 1 Processor -- start 2023-10-16T12:10:52.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.340+0000 7f3939c61700 1 -- start start 2023-10-16T12:10:52.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.341+0000 7f3939c61700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f39341614c0 0x7f39340ca9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:52.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.341+0000 7f3939c61700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3934071a70 con 0x7f393415d9f0 2023-10-16T12:10:52.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.341+0000 7f3939c61700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f393415ddd0 con 0x7f393406aef0 2023-10-16T12:10:52.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.341+0000 7f3939c61700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f393406eb20 con 0x7f39341614c0 2023-10-16T12:10:52.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.341+0000 7f39337fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f393415d9f0 0x7f393406a7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50790/0 (socket says 172.21.15.16:50790) 2023-10-16T12:10:52.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.341+0000 7f39337fe700 1 -- 172.21.15.16:0/3308366489 learned_addr learned my addr 172.21.15.16:0/3308366489 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:10:52.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.341+0000 7f3933fff700 1 --2- 172.21.15.16:0/3308366489 >> v2:172.21.15.49:3300/0 conn(0x7f39341614c0 0x7f39340ca9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:52.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.341+0000 7f39327fc700 1 -- 172.21.15.16:0/3308366489 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1630718382 0 0) 0x7f3934071a70 con 0x7f393415d9f0 2023-10-16T12:10:52.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.341+0000 7f39327fc700 1 -- 172.21.15.16:0/3308366489 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f391c0035b0 con 0x7f393415d9f0 2023-10-16T12:10:52.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.342+0000 7f39327fc700 1 -- 172.21.15.16:0/3308366489 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1785203985 0 0) 0x7f391c0035b0 con 0x7f393415d9f0 2023-10-16T12:10:52.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.342+0000 7f39327fc700 1 -- 172.21.15.16:0/3308366489 >> v2:172.21.15.49:3300/0 conn(0x7f39341614c0 msgr2=0x7f39340ca9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:10:52.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.342+0000 7f39327fc700 1 --2- 172.21.15.16:0/3308366489 >> v2:172.21.15.49:3300/0 conn(0x7f39341614c0 0x7f39340ca9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:52.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.342+0000 7f39327fc700 1 -- 172.21.15.16:0/3308366489 >> v1:172.21.15.49:6789/0 conn(0x7f393406aef0 legacy=0x7f3934071f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:52.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.342+0000 7f39327fc700 1 -- 172.21.15.16:0/3308366489 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3934161a60 con 0x7f393415d9f0 2023-10-16T12:10:52.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.342+0000 7f39327fc700 1 -- 172.21.15.16:0/3308366489 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f3924001670 con 0x7f393415d9f0 2023-10-16T12:10:52.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.342+0000 7f39327fc700 1 -- 172.21.15.16:0/3308366489 >> v1:172.21.15.16:6789/0 conn(0x7f393415d9f0 legacy=0x7f393406a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:52.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.342+0000 7f39327fc700 1 --2- 172.21.15.16:0/3308366489 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f391c003a00 0x7f391c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:52.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.342+0000 7f39327fc700 1 --2- 172.21.15.16:0/3308366489 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f391c004850 0x7f391c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:52.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.342+0000 7f39327fc700 1 --2- 172.21.15.16:0/3308366489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f391c005210 0x7f391c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:52.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.342+0000 7f39327fc700 1 -- 172.21.15.16:0/3308366489 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f393406eb20 con 0x7f391c005210 2023-10-16T12:10:52.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.342+0000 7f39327fc700 1 -- 172.21.15.16:0/3308366489 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f391c00c0b0 con 0x7f391c003a00 2023-10-16T12:10:52.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.342+0000 7f39327fc700 1 -- 172.21.15.16:0/3308366489 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f391c00c270 con 0x7f391c004850 2023-10-16T12:10:52.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.342+0000 7f3933fff700 1 --2- 172.21.15.16:0/3308366489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f391c005210 0x7f391c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:52.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.342+0000 7f39337fe700 1 --2- 172.21.15.16:0/3308366489 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f391c003a00 0x7f391c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:52.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.342+0000 7f3932ffd700 1 --2- 172.21.15.16:0/3308366489 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f391c004850 0x7f391c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:52.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.342+0000 7f3933fff700 1 -- 172.21.15.16:0/3308366489 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f391c004850 msgr2=0x7f391c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:10:52.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.342+0000 7f3933fff700 1 --2- 172.21.15.16:0/3308366489 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f391c004850 0x7f391c004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:52.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.342+0000 7f3933fff700 1 -- 172.21.15.16:0/3308366489 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f391c003a00 msgr2=0x7f391c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:10:52.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.342+0000 7f3933fff700 1 --2- 172.21.15.16:0/3308366489 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f391c003a00 0x7f391c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:52.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.342+0000 7f3933fff700 1 -- 172.21.15.16:0/3308366489 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f391c0035b0 con 0x7f391c005210 2023-10-16T12:10:52.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.342+0000 7f3933fff700 1 -- 172.21.15.16:0/3308366489 reap_dead start 2023-10-16T12:10:52.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.342+0000 7f3933fff700 1 --2- 172.21.15.16:0/3308366489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f391c005210 0x7f391c00ba10 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f392c009fc0 tx=0x7f391c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d06885a53a9d76f5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:52.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.343+0000 7f39327fc700 1 -- 172.21.15.16:0/3308366489 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f392c003a20 con 0x7f391c005210 2023-10-16T12:10:52.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.343+0000 7f39327fc700 1 -- 172.21.15.16:0/3308366489 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f392c00f040 con 0x7f391c005210 2023-10-16T12:10:52.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.343+0000 7f39327fc700 1 -- 172.21.15.16:0/3308366489 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f392c003da0 con 0x7f391c005210 2023-10-16T12:10:52.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.343+0000 7f3939c61700 1 -- 172.21.15.16:0/3308366489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f391c005210 msgr2=0x7f391c00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:52.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.343+0000 7f3939c61700 1 --2- 172.21.15.16:0/3308366489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f391c005210 0x7f391c00ba10 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f392c009fc0 tx=0x7f391c005880 comp rx=0 tx=0).stop 2023-10-16T12:10:52.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.343+0000 7f3939c61700 1 -- 172.21.15.16:0/3308366489 shutdown_connections 2023-10-16T12:10:52.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.343+0000 7f3939c61700 1 --2- 172.21.15.16:0/3308366489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f391c005210 0x7f391c00ba10 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:52.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.343+0000 7f3939c61700 1 -- 172.21.15.16:0/3308366489 >> 172.21.15.16:0/3308366489 conn(0x7f39340ccde0 msgr2=0x7f39340cd1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:52.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.343+0000 7f3939c61700 1 -- 172.21.15.16:0/3308366489 shutdown_connections 2023-10-16T12:10:52.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.343+0000 7f3939c61700 1 -- 172.21.15.16:0/3308366489 wait complete. 2023-10-16T12:10:52.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.344+0000 7f3939c61700 1 Processor -- start 2023-10-16T12:10:52.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.345+0000 7f3939c61700 1 -- start start 2023-10-16T12:10:52.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.345+0000 7f3939c61700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f391c005210 0x7f39341f0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:52.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.345+0000 7f3939c61700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3934150de0 0x7f39341f4020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:52.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.345+0000 7f3939c61700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3934159ed0 0x7f393415a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:52.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.345+0000 7f3939c61700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f393415a8c0 con 0x7f3934159ed0 2023-10-16T12:10:52.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.345+0000 7f3939c61700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f393414f1b0 con 0x7f391c005210 2023-10-16T12:10:52.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.345+0000 7f3939c61700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f393414f490 con 0x7f3934150de0 2023-10-16T12:10:52.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.345+0000 7f3933fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3934159ed0 0x7f393415a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:52.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.346+0000 7f3933fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3934159ed0 0x7f393415a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33300/0 (socket says 172.21.15.16:33300) 2023-10-16T12:10:52.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.346+0000 7f3933fff700 1 -- 172.21.15.16:0/1344310282 learned_addr learned my addr 172.21.15.16:0/1344310282 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:10:52.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.346+0000 7f39337fe700 1 --2- 172.21.15.16:0/1344310282 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f391c005210 0x7f39341f0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:52.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.346+0000 7f3932ffd700 1 --2- 172.21.15.16:0/1344310282 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3934150de0 0x7f39341f4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:52.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.346+0000 7f3933fff700 1 -- 172.21.15.16:0/1344310282 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3934150de0 msgr2=0x7f39341f4020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:52.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.346+0000 7f3933fff700 1 --2- 172.21.15.16:0/1344310282 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3934150de0 0x7f39341f4020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:52.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.346+0000 7f3933fff700 1 -- 172.21.15.16:0/1344310282 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f391c005210 msgr2=0x7f39341f0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:52.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.346+0000 7f3933fff700 1 --2- 172.21.15.16:0/1344310282 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f391c005210 0x7f39341f0ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:52.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.346+0000 7f3933fff700 1 -- 172.21.15.16:0/1344310282 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f393414f770 con 0x7f3934159ed0 2023-10-16T12:10:52.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.346+0000 7f3933fff700 1 --2- 172.21.15.16:0/1344310282 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3934159ed0 0x7f393415a380 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f39340c7360 tx=0x7f392c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:52.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.346+0000 7f391bfff700 1 -- 172.21.15.16:0/1344310282 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f392c003e10 con 0x7f3934159ed0 2023-10-16T12:10:52.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.346+0000 7f391bfff700 1 -- 172.21.15.16:0/1344310282 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f392c004230 con 0x7f3934159ed0 2023-10-16T12:10:52.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.346+0000 7f3939c61700 1 -- 172.21.15.16:0/1344310282 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f393414fa50 con 0x7f3934159ed0 2023-10-16T12:10:52.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.346+0000 7f391bfff700 1 -- 172.21.15.16:0/1344310282 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f392c0045e0 con 0x7f3934159ed0 2023-10-16T12:10:52.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.346+0000 7f3939c61700 1 -- 172.21.15.16:0/1344310282 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3934150010 con 0x7f3934159ed0 2023-10-16T12:10:52.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.347+0000 7f3939c61700 1 -- 172.21.15.16:0/1344310282 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39341501a0 con 0x7f3934159ed0 2023-10-16T12:10:52.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.348+0000 7f391bfff700 1 -- 172.21.15.16:0/1344310282 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f392c024070 con 0x7f3934159ed0 2023-10-16T12:10:52.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.348+0000 7f391bfff700 1 --2- 172.21.15.16:0/1344310282 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f392006af00 0x7f392006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:52.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.348+0000 7f391bfff700 1 -- 172.21.15.16:0/1344310282 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f392c08b430 con 0x7f3934159ed0 2023-10-16T12:10:52.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.348+0000 7f39337fe700 1 --2- 172.21.15.16:0/1344310282 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f392006af00 0x7f392006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:52.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.348+0000 7f39337fe700 1 --2- 172.21.15.16:0/1344310282 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f392006af00 0x7f392006d3c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3924001050 tx=0x7f3924001c30 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:52.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.353+0000 7f391bfff700 1 -- 172.21.15.16:0/1344310282 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f392c056200 con 0x7f3934159ed0 2023-10-16T12:10:52.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.558+0000 7f3939c61700 1 -- 172.21.15.16:0/1344310282 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f393404ebb0 con 0x7f3934159ed0 2023-10-16T12:10:52.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.558+0000 7f391bfff700 1 -- 172.21.15.16:0/1344310282 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) v1 ==== 93+0+10 (secure 0 0 0) 0x7f392c059820 con 0x7f3934159ed0 2023-10-16T12:10:52.560 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T12:10:52.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.560+0000 7f3939c61700 1 -- 172.21.15.16:0/1344310282 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f392006af00 msgr2=0x7f392006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:52.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.560+0000 7f3939c61700 1 --2- 172.21.15.16:0/1344310282 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f392006af00 0x7f392006d3c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3924001050 tx=0x7f3924001c30 comp rx=0 tx=0).stop 2023-10-16T12:10:52.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.560+0000 7f3939c61700 1 -- 172.21.15.16:0/1344310282 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3934159ed0 msgr2=0x7f393415a380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:52.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.560+0000 7f3939c61700 1 --2- 172.21.15.16:0/1344310282 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3934159ed0 0x7f393415a380 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f39340c7360 tx=0x7f392c009750 comp rx=0 tx=0).stop 2023-10-16T12:10:52.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.561+0000 7f3939c61700 1 -- 172.21.15.16:0/1344310282 shutdown_connections 2023-10-16T12:10:52.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.561+0000 7f3939c61700 1 --2- 172.21.15.16:0/1344310282 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f392006af00 0x7f392006d3c0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:52.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.561+0000 7f3939c61700 1 --2- 172.21.15.16:0/1344310282 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3934159ed0 0x7f393415a380 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:52.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.561+0000 7f3939c61700 1 --2- 172.21.15.16:0/1344310282 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f391c005210 0x7f39341f0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:52.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.561+0000 7f3939c61700 1 --2- 172.21.15.16:0/1344310282 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3934150de0 0x7f39341f4020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:52.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.561+0000 7f3939c61700 1 -- 172.21.15.16:0/1344310282 >> 172.21.15.16:0/1344310282 conn(0x7f39340ccde0 msgr2=0x7f393406a7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:52.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.561+0000 7f3939c61700 1 -- 172.21.15.16:0/1344310282 shutdown_connections 2023-10-16T12:10:52.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.561+0000 7f3939c61700 1 -- 172.21.15.16:0/1344310282 wait complete. 2023-10-16T12:10:52.575 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T12:10:52.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.701+0000 7fb07a428700 1 Processor -- start 2023-10-16T12:10:52.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.701+0000 7fb07a428700 1 -- start start 2023-10-16T12:10:52.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.702+0000 7fb07a428700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb0740ce6a0 0x7fb07406c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:52.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.702+0000 7fb07a428700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fb0740d2110 0x7fb07406c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:52.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.702+0000 7fb07a428700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0740b20f0 con 0x7fb0740b2620 2023-10-16T12:10:52.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.702+0000 7fb07a428700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0740af1f0 con 0x7fb0740ce6a0 2023-10-16T12:10:52.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.702+0000 7fb07a428700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0740d03d0 con 0x7fb0740d2110 2023-10-16T12:10:52.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.702+0000 7fb073fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb0740ce6a0 0x7fb07406c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:52.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.702+0000 7fb073fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb0740ce6a0 0x7fb07406c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33304/0 (socket says 172.21.15.16:33304) 2023-10-16T12:10:52.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.702+0000 7fb073fff700 1 -- 172.21.15.16:0/2186251964 learned_addr learned my addr 172.21.15.16:0/2186251964 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:10:52.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.702+0000 7fb0737fe700 1 --2- 172.21.15.16:0/2186251964 >> v2:172.21.15.49:3300/0 conn(0x7fb0740d2110 0x7fb07406c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:52.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.702+0000 7fb073fff700 1 -- 172.21.15.16:0/2186251964 >> v2:172.21.15.49:3300/0 conn(0x7fb0740d2110 msgr2=0x7fb07406c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:10:52.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.702+0000 7fb073fff700 1 --2- 172.21.15.16:0/2186251964 >> v2:172.21.15.49:3300/0 conn(0x7fb0740d2110 0x7fb07406c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:52.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.702+0000 7fb073fff700 1 -- 172.21.15.16:0/2186251964 >> v1:172.21.15.49:6789/0 conn(0x7fb0740b2620 legacy=0x7fb0740b0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:52.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.702+0000 7fb073fff700 1 -- 172.21.15.16:0/2186251964 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0740b2340 con 0x7fb0740ce6a0 2023-10-16T12:10:52.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.702+0000 7fb073fff700 1 --2- 172.21.15.16:0/2186251964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0740ce6a0 0x7fb07406c300 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb0740cf0f0 tx=0x7fb064009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f32f606c0a0ceecd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:52.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.702+0000 7fb072ffd700 1 -- 172.21.15.16:0/2186251964 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb064011830 con 0x7fb0740ce6a0 2023-10-16T12:10:52.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.703+0000 7fb072ffd700 1 -- 172.21.15.16:0/2186251964 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb064011990 con 0x7fb0740ce6a0 2023-10-16T12:10:52.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.703+0000 7fb072ffd700 1 -- 172.21.15.16:0/2186251964 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb064011d20 con 0x7fb0740ce6a0 2023-10-16T12:10:52.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.703+0000 7fb07a428700 1 -- 172.21.15.16:0/2186251964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0740ce6a0 msgr2=0x7fb07406c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:52.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.703+0000 7fb07a428700 1 --2- 172.21.15.16:0/2186251964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0740ce6a0 0x7fb07406c300 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb0740cf0f0 tx=0x7fb064009bd0 comp rx=0 tx=0).stop 2023-10-16T12:10:52.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.703+0000 7fb07a428700 1 -- 172.21.15.16:0/2186251964 shutdown_connections 2023-10-16T12:10:52.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.703+0000 7fb07a428700 1 --2- 172.21.15.16:0/2186251964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0740ce6a0 0x7fb07406c300 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:52.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.703+0000 7fb07a428700 1 --2- 172.21.15.16:0/2186251964 >> v2:172.21.15.49:3300/0 conn(0x7fb0740d2110 0x7fb07406c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:52.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.703+0000 7fb07a428700 1 -- 172.21.15.16:0/2186251964 >> 172.21.15.16:0/2186251964 conn(0x7fb0740d15f0 msgr2=0x7fb0740d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:52.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.703+0000 7fb07a428700 1 -- 172.21.15.16:0/2186251964 shutdown_connections 2023-10-16T12:10:52.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.703+0000 7fb07a428700 1 -- 172.21.15.16:0/2186251964 wait complete. 2023-10-16T12:10:52.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.704+0000 7fb07a428700 1 Processor -- start 2023-10-16T12:10:52.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.705+0000 7fb07a428700 1 -- start start 2023-10-16T12:10:52.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.705+0000 7fb07a428700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb0740b2620 0x7fb0740ca0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:52.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.705+0000 7fb07a428700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb0740ce6a0 0x7fb0740ca600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:52.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.705+0000 7fb07a428700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0740d2110 0x7fb0741f6d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:52.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.705+0000 7fb07a428700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0740c5120 con 0x7fb0740d2110 2023-10-16T12:10:52.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.705+0000 7fb07a428700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0741f3930 con 0x7fb0740b2620 2023-10-16T12:10:52.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.705+0000 7fb07a428700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0741f3c10 con 0x7fb0740ce6a0 2023-10-16T12:10:52.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.705+0000 7fb0789c5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0740d2110 0x7fb0741f6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:52.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.706+0000 7fb0789c5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0740d2110 0x7fb0741f6d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33312/0 (socket says 172.21.15.16:33312) 2023-10-16T12:10:52.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.706+0000 7fb0789c5700 1 -- 172.21.15.16:0/2890696906 learned_addr learned my addr 172.21.15.16:0/2890696906 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:10:52.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.706+0000 7fb0737fe700 1 --2- 172.21.15.16:0/2890696906 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb0740ce6a0 0x7fb0740ca600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:52.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.706+0000 7fb073fff700 1 --2- 172.21.15.16:0/2890696906 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb0740b2620 0x7fb0740ca0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:52.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.706+0000 7fb0789c5700 1 -- 172.21.15.16:0/2890696906 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb0740ce6a0 msgr2=0x7fb0740ca600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:52.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.706+0000 7fb0789c5700 1 --2- 172.21.15.16:0/2890696906 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb0740ce6a0 0x7fb0740ca600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:52.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.706+0000 7fb0789c5700 1 -- 172.21.15.16:0/2890696906 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb0740b2620 msgr2=0x7fb0740ca0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:52.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.706+0000 7fb0789c5700 1 --2- 172.21.15.16:0/2890696906 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb0740b2620 0x7fb0740ca0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:52.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.706+0000 7fb0789c5700 1 -- 172.21.15.16:0/2890696906 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb064009550 con 0x7fb0740d2110 2023-10-16T12:10:52.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.706+0000 7fb0789c5700 1 --2- 172.21.15.16:0/2890696906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0740d2110 0x7fb0741f6d60 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb06c00cc50 tx=0x7fb06c00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:52.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.706+0000 7fb070ff9700 1 -- 172.21.15.16:0/2890696906 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb06c003c00 con 0x7fb0740d2110 2023-10-16T12:10:52.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.706+0000 7fb070ff9700 1 -- 172.21.15.16:0/2890696906 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb06c003d60 con 0x7fb0740d2110 2023-10-16T12:10:52.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.706+0000 7fb07a428700 1 -- 172.21.15.16:0/2890696906 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0741f3ef0 con 0x7fb0740d2110 2023-10-16T12:10:52.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.706+0000 7fb070ff9700 1 -- 172.21.15.16:0/2890696906 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb06c00aab0 con 0x7fb0740d2110 2023-10-16T12:10:52.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.706+0000 7fb07a428700 1 -- 172.21.15.16:0/2890696906 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0740b00b0 con 0x7fb0740d2110 2023-10-16T12:10:52.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.707+0000 7fb07a428700 1 -- 172.21.15.16:0/2890696906 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0740b7790 con 0x7fb0740d2110 2023-10-16T12:10:52.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.708+0000 7fb070ff9700 1 -- 172.21.15.16:0/2890696906 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fb06c00ac10 con 0x7fb0740d2110 2023-10-16T12:10:52.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.708+0000 7fb070ff9700 1 --2- 172.21.15.16:0/2890696906 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fb06006af00 0x7fb06006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:52.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.708+0000 7fb070ff9700 1 -- 172.21.15.16:0/2890696906 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fb06c0898d0 con 0x7fb0740d2110 2023-10-16T12:10:52.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.708+0000 7fb073fff700 1 --2- 172.21.15.16:0/2890696906 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fb06006af00 0x7fb06006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:52.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.709+0000 7fb073fff700 1 --2- 172.21.15.16:0/2890696906 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fb06006af00 0x7fb06006d3c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb064009c10 tx=0x7fb064008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:52.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.713+0000 7fb070ff9700 1 -- 172.21.15.16:0/2890696906 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb06c0545f0 con 0x7fb0740d2110 2023-10-16T12:10:52.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.915+0000 7fb07a428700 1 -- 172.21.15.16:0/2890696906 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb0740b0240 con 0x7fb0740d2110 2023-10-16T12:10:52.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.916+0000 7fb070ff9700 1 -- 172.21.15.16:0/2890696906 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb06c057c10 con 0x7fb0740d2110 2023-10-16T12:10:52.918 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-10-16T12:10:52.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.918+0000 7fb07a428700 1 -- 172.21.15.16:0/2890696906 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fb06006af00 msgr2=0x7fb06006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:52.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.918+0000 7fb07a428700 1 --2- 172.21.15.16:0/2890696906 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fb06006af00 0x7fb06006d3c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb064009c10 tx=0x7fb064008030 comp rx=0 tx=0).stop 2023-10-16T12:10:52.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.918+0000 7fb07a428700 1 -- 172.21.15.16:0/2890696906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0740d2110 msgr2=0x7fb0741f6d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:52.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.918+0000 7fb07a428700 1 --2- 172.21.15.16:0/2890696906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0740d2110 0x7fb0741f6d60 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb06c00cc50 tx=0x7fb06c00cf60 comp rx=0 tx=0).stop 2023-10-16T12:10:52.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.918+0000 7fb07a428700 1 -- 172.21.15.16:0/2890696906 shutdown_connections 2023-10-16T12:10:52.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.918+0000 7fb07a428700 1 --2- 172.21.15.16:0/2890696906 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fb06006af00 0x7fb06006d3c0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:52.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.918+0000 7fb07a428700 1 --2- 172.21.15.16:0/2890696906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0740d2110 0x7fb0741f6d60 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:52.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.918+0000 7fb07a428700 1 --2- 172.21.15.16:0/2890696906 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb0740b2620 0x7fb0740ca0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:52.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.918+0000 7fb07a428700 1 --2- 172.21.15.16:0/2890696906 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb0740ce6a0 0x7fb0740ca600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:52.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.918+0000 7fb07a428700 1 -- 172.21.15.16:0/2890696906 >> 172.21.15.16:0/2890696906 conn(0x7fb0740d15f0 msgr2=0x7fb0740cd0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:52.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.918+0000 7fb07a428700 1 -- 172.21.15.16:0/2890696906 shutdown_connections 2023-10-16T12:10:52.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:52.918+0000 7fb07a428700 1 -- 172.21.15.16:0/2890696906 wait complete. 2023-10-16T12:10:52.932 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T12:10:53.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.057+0000 7f2ed0640700 1 Processor -- start 2023-10-16T12:10:53.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.057+0000 7f2ed0640700 1 -- start start 2023-10-16T12:10:53.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.058+0000 7f2ed0640700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2ec815f060 0x7f2ec80cc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:53.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.058+0000 7f2ed0640700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ec80b2440 con 0x7f2ec80ce5d0 2023-10-16T12:10:53.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.058+0000 7f2ed0640700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ec80d00e0 con 0x7f2ec815ec80 2023-10-16T12:10:53.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.058+0000 7f2ed0640700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ec80b2c50 con 0x7f2ec815f060 2023-10-16T12:10:53.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.059+0000 7f2ecebdd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2ec815f060 0x7f2ec80cc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:53.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.059+0000 7f2ecdbdb700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2ec80ce5d0 0x7f2ec80cc190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50822/0 (socket says 172.21.15.16:50822) 2023-10-16T12:10:53.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.059+0000 7f2ecdbdb700 1 -- 172.21.15.16:0/307416110 learned_addr learned my addr 172.21.15.16:0/307416110 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:10:53.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.059+0000 7f2ecebdd700 1 -- 172.21.15.16:0/307416110 >> v1:172.21.15.49:6789/0 conn(0x7f2ec815ec80 legacy=0x7f2ec80cdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:53.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.059+0000 7f2ecebdd700 1 -- 172.21.15.16:0/307416110 >> v1:172.21.15.16:6789/0 conn(0x7f2ec80ce5d0 legacy=0x7f2ec80cc190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:53.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.059+0000 7f2ecebdd700 1 -- 172.21.15.16:0/307416110 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ec80cef60 con 0x7f2ec815f060 2023-10-16T12:10:53.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.059+0000 7f2ecd3da700 1 -- 172.21.15.16:0/307416110 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2114658765 0 0) 0x7f2ec80b2440 con 0x7f2ec80ce5d0 2023-10-16T12:10:53.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.059+0000 7f2ecebdd700 1 --2- 172.21.15.16:0/307416110 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ec815f060 0x7f2ec80cc8a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f2ec0009dc0 tx=0x7f2ec000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7876ecd42d92f6bd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:53.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.059+0000 7f2ecd3da700 1 -- 172.21.15.16:0/307416110 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec000e070 con 0x7f2ec815f060 2023-10-16T12:10:53.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.059+0000 7f2ecd3da700 1 -- 172.21.15.16:0/307416110 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ec000bb90 con 0x7f2ec815f060 2023-10-16T12:10:53.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.059+0000 7f2ecd3da700 1 -- 172.21.15.16:0/307416110 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec0012430 con 0x7f2ec815f060 2023-10-16T12:10:53.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.060+0000 7f2ed0640700 1 -- 172.21.15.16:0/307416110 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ec815f060 msgr2=0x7f2ec80cc8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:53.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.060+0000 7f2ed0640700 1 --2- 172.21.15.16:0/307416110 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ec815f060 0x7f2ec80cc8a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f2ec0009dc0 tx=0x7f2ec000b330 comp rx=0 tx=0).stop 2023-10-16T12:10:53.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.060+0000 7f2ed0640700 1 -- 172.21.15.16:0/307416110 shutdown_connections 2023-10-16T12:10:53.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.060+0000 7f2ed0640700 1 --2- 172.21.15.16:0/307416110 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ec815f060 0x7f2ec80cc8a0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:53.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.060+0000 7f2ed0640700 1 -- 172.21.15.16:0/307416110 >> 172.21.15.16:0/307416110 conn(0x7f2ec80d15f0 msgr2=0x7f2ec80d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:53.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.060+0000 7f2ed0640700 1 -- 172.21.15.16:0/307416110 shutdown_connections 2023-10-16T12:10:53.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.060+0000 7f2ed0640700 1 -- 172.21.15.16:0/307416110 wait complete. 2023-10-16T12:10:53.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.061+0000 7f2ed0640700 1 Processor -- start 2023-10-16T12:10:53.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.061+0000 7f2ed0640700 1 -- start start 2023-10-16T12:10:53.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.062+0000 7f2ed0640700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ec80ce5d0 0x7f2ec81f3a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:53.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.062+0000 7f2ed0640700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2ec815ec80 0x7f2ec806cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:53.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.062+0000 7f2ed0640700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2ec815f060 0x7f2ec80cad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:53.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.062+0000 7f2ed0640700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ec80ca130 con 0x7f2ec80ce5d0 2023-10-16T12:10:53.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.062+0000 7f2ed0640700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ec806ba20 con 0x7f2ec815f060 2023-10-16T12:10:53.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.062+0000 7f2ed0640700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ec806bd00 con 0x7f2ec815ec80 2023-10-16T12:10:53.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.062+0000 7f2ece3dc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ec80ce5d0 0x7f2ec81f3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:53.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.062+0000 7f2ece3dc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ec80ce5d0 0x7f2ec81f3a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33322/0 (socket says 172.21.15.16:33322) 2023-10-16T12:10:53.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.062+0000 7f2ece3dc700 1 -- 172.21.15.16:0/2766905026 learned_addr learned my addr 172.21.15.16:0/2766905026 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:10:53.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.062+0000 7f2ecdbdb700 1 --2- 172.21.15.16:0/2766905026 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2ec815ec80 0x7f2ec806cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:53.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.062+0000 7f2ecebdd700 1 --2- 172.21.15.16:0/2766905026 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2ec815f060 0x7f2ec80cad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:53.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.062+0000 7f2ece3dc700 1 -- 172.21.15.16:0/2766905026 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2ec815ec80 msgr2=0x7f2ec806cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:53.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.062+0000 7f2ece3dc700 1 --2- 172.21.15.16:0/2766905026 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2ec815ec80 0x7f2ec806cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:53.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.062+0000 7f2ece3dc700 1 -- 172.21.15.16:0/2766905026 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2ec815f060 msgr2=0x7f2ec80cad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:53.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.062+0000 7f2ece3dc700 1 --2- 172.21.15.16:0/2766905026 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2ec815f060 0x7f2ec80cad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:53.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.062+0000 7f2ece3dc700 1 -- 172.21.15.16:0/2766905026 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ec0009a70 con 0x7f2ec80ce5d0 2023-10-16T12:10:53.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.063+0000 7f2ece3dc700 1 --2- 172.21.15.16:0/2766905026 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ec80ce5d0 0x7f2ec81f3a40 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f2eb800c4e0 tx=0x7f2eb800c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:53.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.063+0000 7f2ebeffd700 1 -- 172.21.15.16:0/2766905026 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eb80052d0 con 0x7f2ec80ce5d0 2023-10-16T12:10:53.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.063+0000 7f2ebeffd700 1 -- 172.21.15.16:0/2766905026 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2eb8005430 con 0x7f2ec80ce5d0 2023-10-16T12:10:53.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.063+0000 7f2ed0640700 1 -- 172.21.15.16:0/2766905026 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ec806bfe0 con 0x7f2ec80ce5d0 2023-10-16T12:10:53.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.063+0000 7f2ebeffd700 1 -- 172.21.15.16:0/2766905026 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eb80038c0 con 0x7f2ec80ce5d0 2023-10-16T12:10:53.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.063+0000 7f2ed0640700 1 -- 172.21.15.16:0/2766905026 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ec806d000 con 0x7f2ec80ce5d0 2023-10-16T12:10:53.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.064+0000 7f2ed0640700 1 -- 172.21.15.16:0/2766905026 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ec806c910 con 0x7f2ec80ce5d0 2023-10-16T12:10:53.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.064+0000 7f2ebeffd700 1 -- 172.21.15.16:0/2766905026 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f2eb8003a20 con 0x7f2ec80ce5d0 2023-10-16T12:10:53.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.065+0000 7f2ebeffd700 1 --2- 172.21.15.16:0/2766905026 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f2eac06af30 0x7f2eac06d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:53.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.065+0000 7f2ebeffd700 1 -- 172.21.15.16:0/2766905026 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f2eb8088c60 con 0x7f2ec80ce5d0 2023-10-16T12:10:53.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.065+0000 7f2ecdbdb700 1 --2- 172.21.15.16:0/2766905026 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f2eac06af30 0x7f2eac06d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:53.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.065+0000 7f2ecdbdb700 1 --2- 172.21.15.16:0/2766905026 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f2eac06af30 0x7f2eac06d3f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2ec4007b10 tx=0x7f2ec4007a60 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:53.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.069+0000 7f2ebeffd700 1 -- 172.21.15.16:0/2766905026 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2eb8053a30 con 0x7f2ec80ce5d0 2023-10-16T12:10:53.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.271+0000 7f2ed0640700 1 -- 172.21.15.16:0/2766905026 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f2ec804ebb0 con 0x7f2ec80ce5d0 2023-10-16T12:10:53.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.271+0000 7f2ebeffd700 1 -- 172.21.15.16:0/2766905026 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2eb8057050 con 0x7f2ec80ce5d0 2023-10-16T12:10:53.274 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-10-16T12:10:53.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.273+0000 7f2ed0640700 1 -- 172.21.15.16:0/2766905026 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f2eac06af30 msgr2=0x7f2eac06d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:53.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.273+0000 7f2ed0640700 1 --2- 172.21.15.16:0/2766905026 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f2eac06af30 0x7f2eac06d3f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2ec4007b10 tx=0x7f2ec4007a60 comp rx=0 tx=0).stop 2023-10-16T12:10:53.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.273+0000 7f2ed0640700 1 -- 172.21.15.16:0/2766905026 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ec80ce5d0 msgr2=0x7f2ec81f3a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:53.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.273+0000 7f2ed0640700 1 --2- 172.21.15.16:0/2766905026 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ec80ce5d0 0x7f2ec81f3a40 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f2eb800c4e0 tx=0x7f2eb800c7f0 comp rx=0 tx=0).stop 2023-10-16T12:10:53.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.273+0000 7f2ed0640700 1 -- 172.21.15.16:0/2766905026 shutdown_connections 2023-10-16T12:10:53.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.273+0000 7f2ed0640700 1 --2- 172.21.15.16:0/2766905026 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f2eac06af30 0x7f2eac06d3f0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:53.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.273+0000 7f2ed0640700 1 --2- 172.21.15.16:0/2766905026 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2ec815f060 0x7f2ec80cad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:53.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.273+0000 7f2ed0640700 1 --2- 172.21.15.16:0/2766905026 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ec80ce5d0 0x7f2ec81f3a40 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:53.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.273+0000 7f2ed0640700 1 --2- 172.21.15.16:0/2766905026 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2ec815ec80 0x7f2ec806cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:53.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.273+0000 7f2ed0640700 1 -- 172.21.15.16:0/2766905026 >> 172.21.15.16:0/2766905026 conn(0x7f2ec80d15f0 msgr2=0x7f2ec8153c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:53.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.274+0000 7f2ed0640700 1 -- 172.21.15.16:0/2766905026 shutdown_connections 2023-10-16T12:10:53.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.274+0000 7f2ed0640700 1 -- 172.21.15.16:0/2766905026 wait complete. 2023-10-16T12:10:53.287 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-16T12:10:53.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.413+0000 7f842e2b6700 1 Processor -- start 2023-10-16T12:10:53.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.413+0000 7f842e2b6700 1 -- start start 2023-10-16T12:10:53.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.414+0000 7f842e2b6700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f842815b2c0 0x7f8428161340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:53.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.414+0000 7f842e2b6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f842806a7e0 0x7f842806ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:53.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.415+0000 7f842e2b6700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f842815ecb0 con 0x7f842806e570 2023-10-16T12:10:53.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.415+0000 7f842e2b6700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f842806b190 con 0x7f842806a7e0 2023-10-16T12:10:53.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.415+0000 7f842e2b6700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8428161880 con 0x7f842815b2c0 2023-10-16T12:10:53.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.415+0000 7f842c853700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f842806a7e0 0x7f842806ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:53.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.415+0000 7f84277fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f842806e570 0x7f842815e5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50836/0 (socket says 172.21.15.16:50836) 2023-10-16T12:10:53.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.415+0000 7f84277fe700 1 -- 172.21.15.16:0/4011805475 learned_addr learned my addr 172.21.15.16:0/4011805475 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:10:53.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.415+0000 7f8427fff700 1 --2- 172.21.15.16:0/4011805475 >> v2:172.21.15.49:3300/0 conn(0x7f842815b2c0 0x7f8428161340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:53.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.415+0000 7f842c853700 1 -- 172.21.15.16:0/4011805475 >> v2:172.21.15.49:3300/0 conn(0x7f842815b2c0 msgr2=0x7f8428161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:10:53.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.415+0000 7f842c853700 1 --2- 172.21.15.16:0/4011805475 >> v2:172.21.15.49:3300/0 conn(0x7f842815b2c0 0x7f8428161340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:53.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.415+0000 7f842c853700 1 -- 172.21.15.16:0/4011805475 >> v1:172.21.15.16:6789/0 conn(0x7f842806e570 legacy=0x7f842815e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:53.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.415+0000 7f842c853700 1 -- 172.21.15.16:0/4011805475 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f842815b6a0 con 0x7f842806a7e0 2023-10-16T12:10:53.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.415+0000 7f842c853700 1 --2- 172.21.15.16:0/4011805475 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f842806a7e0 0x7f842806ac50 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f842806b4a0 tx=0x7f842000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a807e59b9cc9aaff server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:53.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.415+0000 7f8426ffd700 1 -- 172.21.15.16:0/4011805475 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f842000e070 con 0x7f842806a7e0 2023-10-16T12:10:53.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.415+0000 7f8426ffd700 1 -- 172.21.15.16:0/4011805475 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8420003a20 con 0x7f842806a7e0 2023-10-16T12:10:53.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.415+0000 7f8426ffd700 1 -- 172.21.15.16:0/4011805475 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8420003dd0 con 0x7f842806a7e0 2023-10-16T12:10:53.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.416+0000 7f842e2b6700 1 -- 172.21.15.16:0/4011805475 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f842806a7e0 msgr2=0x7f842806ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:53.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.416+0000 7f842e2b6700 1 --2- 172.21.15.16:0/4011805475 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f842806a7e0 0x7f842806ac50 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f842806b4a0 tx=0x7f842000b330 comp rx=0 tx=0).stop 2023-10-16T12:10:53.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.416+0000 7f842e2b6700 1 -- 172.21.15.16:0/4011805475 shutdown_connections 2023-10-16T12:10:53.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.416+0000 7f842e2b6700 1 --2- 172.21.15.16:0/4011805475 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f842806a7e0 0x7f842806ac50 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:53.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.416+0000 7f842e2b6700 1 --2- 172.21.15.16:0/4011805475 >> v2:172.21.15.49:3300/0 conn(0x7f842815b2c0 0x7f8428161340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:53.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.416+0000 7f842e2b6700 1 -- 172.21.15.16:0/4011805475 >> 172.21.15.16:0/4011805475 conn(0x7f84280b1f40 msgr2=0x7f84280b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:53.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.416+0000 7f842e2b6700 1 -- 172.21.15.16:0/4011805475 shutdown_connections 2023-10-16T12:10:53.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.416+0000 7f842e2b6700 1 -- 172.21.15.16:0/4011805475 wait complete. 2023-10-16T12:10:53.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.418+0000 7f842e2b6700 1 Processor -- start 2023-10-16T12:10:53.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.418+0000 7f842e2b6700 1 -- start start 2023-10-16T12:10:53.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.419+0000 7f842e2b6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f842806a7e0 0x7f842815c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:53.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.419+0000 7f842e2b6700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f842806e570 0x7f842815cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:53.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.419+0000 7f842e2b6700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f842815b2c0 0x7f84281f6690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:53.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.419+0000 7f842e2b6700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84281f7630 con 0x7f842806a7e0 2023-10-16T12:10:53.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.419+0000 7f842e2b6700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84281f7350 con 0x7f842806e570 2023-10-16T12:10:53.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.419+0000 7f842e2b6700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84280b4840 con 0x7f842815b2c0 2023-10-16T12:10:53.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.419+0000 7f8427fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f842806a7e0 0x7f842815c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:53.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.419+0000 7f8427fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f842806a7e0 0x7f842815c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33336/0 (socket says 172.21.15.16:33336) 2023-10-16T12:10:53.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.419+0000 7f8427fff700 1 -- 172.21.15.16:0/2549263862 learned_addr learned my addr 172.21.15.16:0/2549263862 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:10:53.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.419+0000 7f84277fe700 1 --2- 172.21.15.16:0/2549263862 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f842806e570 0x7f842815cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:53.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.419+0000 7f842c853700 1 --2- 172.21.15.16:0/2549263862 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f842815b2c0 0x7f84281f6690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:53.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.419+0000 7f8427fff700 1 -- 172.21.15.16:0/2549263862 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f842815b2c0 msgr2=0x7f84281f6690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:53.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.419+0000 7f8427fff700 1 --2- 172.21.15.16:0/2549263862 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f842815b2c0 0x7f84281f6690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:53.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.419+0000 7f8427fff700 1 -- 172.21.15.16:0/2549263862 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f842806e570 msgr2=0x7f842815cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:53.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.419+0000 7f8427fff700 1 --2- 172.21.15.16:0/2549263862 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f842806e570 0x7f842815cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:53.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.419+0000 7f8427fff700 1 -- 172.21.15.16:0/2549263862 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8420009a70 con 0x7f842806a7e0 2023-10-16T12:10:53.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.419+0000 7f842c853700 1 --2- 172.21.15.16:0/2549263862 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f842815b2c0 0x7f84281f6690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:10:53.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.419+0000 7f8427fff700 1 --2- 172.21.15.16:0/2549263862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f842806a7e0 0x7f842815c9b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f841800e850 tx=0x7f841800ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:53.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.419+0000 7f8424ff9700 1 -- 172.21.15.16:0/2549263862 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84180095b0 con 0x7f842806a7e0 2023-10-16T12:10:53.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.420+0000 7f8424ff9700 1 -- 172.21.15.16:0/2549263862 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f841800ee50 con 0x7f842806a7e0 2023-10-16T12:10:53.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.420+0000 7f8424ff9700 1 -- 172.21.15.16:0/2549263862 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f841800ca70 con 0x7f842806a7e0 2023-10-16T12:10:53.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.420+0000 7f842e2b6700 1 -- 172.21.15.16:0/2549263862 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84280b4b20 con 0x7f842806a7e0 2023-10-16T12:10:53.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.420+0000 7f842e2b6700 1 -- 172.21.15.16:0/2549263862 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84281f6bd0 con 0x7f842806a7e0 2023-10-16T12:10:53.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.421+0000 7f842e2b6700 1 -- 172.21.15.16:0/2549263862 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84280b46a0 con 0x7f842806a7e0 2023-10-16T12:10:53.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.421+0000 7f8424ff9700 1 -- 172.21.15.16:0/2549263862 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f841800cbd0 con 0x7f842806a7e0 2023-10-16T12:10:53.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.421+0000 7f8424ff9700 1 --2- 172.21.15.16:0/2549263862 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f841406af00 0x7f841406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:53.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.422+0000 7f8424ff9700 1 -- 172.21.15.16:0/2549263862 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f8418089860 con 0x7f842806a7e0 2023-10-16T12:10:53.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.422+0000 7f84277fe700 1 --2- 172.21.15.16:0/2549263862 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f841406af00 0x7f841406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:53.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.422+0000 7f84277fe700 1 --2- 172.21.15.16:0/2549263862 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f841406af00 0x7f841406d3c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f841c007c20 tx=0x7f841c007d70 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:53.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.426+0000 7f8424ff9700 1 -- 172.21.15.16:0/2549263862 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f84280b46a0 con 0x7f842806a7e0 2023-10-16T12:10:53.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.628+0000 7f842e2b6700 1 -- 172.21.15.16:0/2549263862 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f84280b46a0 con 0x7f842806a7e0 2023-10-16T12:10:53.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.629+0000 7f8424ff9700 1 -- 172.21.15.16:0/2549263862 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) v1 ==== 103+0+11 (secure 0 0 0) 0x7f84280b46a0 con 0x7f842806a7e0 2023-10-16T12:10:53.631 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-10-16T12:10:53.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.630+0000 7f842e2b6700 1 -- 172.21.15.16:0/2549263862 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f841406af00 msgr2=0x7f841406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:53.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.630+0000 7f842e2b6700 1 --2- 172.21.15.16:0/2549263862 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f841406af00 0x7f841406d3c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f841c007c20 tx=0x7f841c007d70 comp rx=0 tx=0).stop 2023-10-16T12:10:53.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.630+0000 7f842e2b6700 1 -- 172.21.15.16:0/2549263862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f842806a7e0 msgr2=0x7f842815c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:53.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.630+0000 7f842e2b6700 1 --2- 172.21.15.16:0/2549263862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f842806a7e0 0x7f842815c9b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f841800e850 tx=0x7f841800ec10 comp rx=0 tx=0).stop 2023-10-16T12:10:53.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.631+0000 7f842e2b6700 1 -- 172.21.15.16:0/2549263862 shutdown_connections 2023-10-16T12:10:53.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.631+0000 7f842e2b6700 1 --2- 172.21.15.16:0/2549263862 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f841406af00 0x7f841406d3c0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:53.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.631+0000 7f842e2b6700 1 --2- 172.21.15.16:0/2549263862 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f842815b2c0 0x7f84281f6690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:53.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.631+0000 7f842e2b6700 1 --2- 172.21.15.16:0/2549263862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f842806a7e0 0x7f842815c9b0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:53.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.631+0000 7f842e2b6700 1 --2- 172.21.15.16:0/2549263862 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f842806e570 0x7f842815cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:53.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.631+0000 7f842e2b6700 1 -- 172.21.15.16:0/2549263862 >> 172.21.15.16:0/2549263862 conn(0x7f84280b1f40 msgr2=0x7f842815c260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:53.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.631+0000 7f842e2b6700 1 -- 172.21.15.16:0/2549263862 shutdown_connections 2023-10-16T12:10:53.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.631+0000 7f842e2b6700 1 -- 172.21.15.16:0/2549263862 wait complete. 2023-10-16T12:10:53.645 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T12:10:53.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.768+0000 7f27f7e47700 1 Processor -- start 2023-10-16T12:10:53.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.769+0000 7f27f7e47700 1 -- start start 2023-10-16T12:10:53.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.769+0000 7f27f7e47700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f27f00cc8a0 0x7f27f015e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:53.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.769+0000 7f27f7e47700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f27f006a7e0 0x7f27f006ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:53.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.769+0000 7f27f7e47700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27f006b150 con 0x7f27f015b280 2023-10-16T12:10:53.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.769+0000 7f27f7e47700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27f0161c30 con 0x7f27f006a7e0 2023-10-16T12:10:53.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.769+0000 7f27f7e47700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27f0161f10 con 0x7f27f00cc8a0 2023-10-16T12:10:53.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.770+0000 7f27f63e4700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f27f006a7e0 0x7f27f006ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:53.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.770+0000 7f27f5be3700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f27f015b280 0x7f27f01613e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50850/0 (socket says 172.21.15.16:50850) 2023-10-16T12:10:53.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.770+0000 7f27f63e4700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f27f006a7e0 0x7f27f006ac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33348/0 (socket says 172.21.15.16:33348) 2023-10-16T12:10:53.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.770+0000 7f27f5be3700 1 -- 172.21.15.16:0/839987585 learned_addr learned my addr 172.21.15.16:0/839987585 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:10:53.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.770+0000 7f27f53e2700 1 --2- 172.21.15.16:0/839987585 >> v2:172.21.15.49:3300/0 conn(0x7f27f00cc8a0 0x7f27f015e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:53.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.770+0000 7f27f63e4700 1 -- 172.21.15.16:0/839987585 >> v2:172.21.15.49:3300/0 conn(0x7f27f00cc8a0 msgr2=0x7f27f015e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:10:53.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.770+0000 7f27f63e4700 1 --2- 172.21.15.16:0/839987585 >> v2:172.21.15.49:3300/0 conn(0x7f27f00cc8a0 0x7f27f015e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:53.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.770+0000 7f27f63e4700 1 -- 172.21.15.16:0/839987585 >> v1:172.21.15.16:6789/0 conn(0x7f27f015b280 legacy=0x7f27f01613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:53.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.770+0000 7f27f63e4700 1 -- 172.21.15.16:0/839987585 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27f01621f0 con 0x7f27f006a7e0 2023-10-16T12:10:53.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.770+0000 7f27f4be1700 1 -- 172.21.15.16:0/839987585 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 324542831 0 0) 0x7f27f006b150 con 0x7f27f015b280 2023-10-16T12:10:53.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.770+0000 7f27f63e4700 1 --2- 172.21.15.16:0/839987585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27f006a7e0 0x7f27f006ac10 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f27f0071970 tx=0x7f27e800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58971a5f6e2ddf93 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:53.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.770+0000 7f27f4be1700 1 -- 172.21.15.16:0/839987585 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27e800e070 con 0x7f27f006a7e0 2023-10-16T12:10:53.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.770+0000 7f27f4be1700 1 -- 172.21.15.16:0/839987585 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27e800bbd0 con 0x7f27f006a7e0 2023-10-16T12:10:53.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.770+0000 7f27f4be1700 1 -- 172.21.15.16:0/839987585 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27e8012430 con 0x7f27f006a7e0 2023-10-16T12:10:53.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.771+0000 7f27f7e47700 1 -- 172.21.15.16:0/839987585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27f006a7e0 msgr2=0x7f27f006ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:53.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.771+0000 7f27f7e47700 1 --2- 172.21.15.16:0/839987585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27f006a7e0 0x7f27f006ac10 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f27f0071970 tx=0x7f27e800b330 comp rx=0 tx=0).stop 2023-10-16T12:10:53.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.771+0000 7f27f7e47700 1 -- 172.21.15.16:0/839987585 shutdown_connections 2023-10-16T12:10:53.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.771+0000 7f27f7e47700 1 --2- 172.21.15.16:0/839987585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27f006a7e0 0x7f27f006ac10 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:53.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.771+0000 7f27f7e47700 1 --2- 172.21.15.16:0/839987585 >> v2:172.21.15.49:3300/0 conn(0x7f27f00cc8a0 0x7f27f015e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:53.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.771+0000 7f27f7e47700 1 -- 172.21.15.16:0/839987585 >> 172.21.15.16:0/839987585 conn(0x7f27f00b1fd0 msgr2=0x7f27f00b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:53.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.771+0000 7f27f7e47700 1 -- 172.21.15.16:0/839987585 shutdown_connections 2023-10-16T12:10:53.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.771+0000 7f27f7e47700 1 -- 172.21.15.16:0/839987585 wait complete. 2023-10-16T12:10:53.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.772+0000 7f27f7e47700 1 Processor -- start 2023-10-16T12:10:53.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.772+0000 7f27f7e47700 1 -- start start 2023-10-16T12:10:53.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.773+0000 7f27f7e47700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f27f006a7e0 0x7f27f0152a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:53.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.773+0000 7f27f7e47700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27f00cc8a0 0x7f27f00c4180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:53.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.773+0000 7f27f7e47700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f27f015b280 0x7f27f015ff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:53.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.773+0000 7f27f7e47700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27f01f6960 con 0x7f27f00cc8a0 2023-10-16T12:10:53.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.773+0000 7f27f7e47700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27f00c4f20 con 0x7f27f006a7e0 2023-10-16T12:10:53.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.773+0000 7f27f7e47700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27f00c4c40 con 0x7f27f015b280 2023-10-16T12:10:53.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.773+0000 7f27f53e2700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27f00cc8a0 0x7f27f00c4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:53.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.773+0000 7f27f53e2700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27f00cc8a0 0x7f27f00c4180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33352/0 (socket says 172.21.15.16:33352) 2023-10-16T12:10:53.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.773+0000 7f27f53e2700 1 -- 172.21.15.16:0/3569608206 learned_addr learned my addr 172.21.15.16:0/3569608206 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:10:53.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.773+0000 7f27f5be3700 1 --2- 172.21.15.16:0/3569608206 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f27f006a7e0 0x7f27f0152a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:53.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.773+0000 7f27f63e4700 1 --2- 172.21.15.16:0/3569608206 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f27f015b280 0x7f27f015ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:53.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.773+0000 7f27f53e2700 1 -- 172.21.15.16:0/3569608206 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f27f015b280 msgr2=0x7f27f015ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:53.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.773+0000 7f27f53e2700 1 --2- 172.21.15.16:0/3569608206 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f27f015b280 0x7f27f015ff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:53.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.773+0000 7f27f53e2700 1 -- 172.21.15.16:0/3569608206 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f27f006a7e0 msgr2=0x7f27f0152a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:53.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.773+0000 7f27f53e2700 1 --2- 172.21.15.16:0/3569608206 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f27f006a7e0 0x7f27f0152a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:53.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.773+0000 7f27f53e2700 1 -- 172.21.15.16:0/3569608206 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27e8009a70 con 0x7f27f00cc8a0 2023-10-16T12:10:53.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.774+0000 7f27f53e2700 1 --2- 172.21.15.16:0/3569608206 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27f00cc8a0 0x7f27f00c4180 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f27ec00e7e0 tx=0x7f27ec00eaf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:53.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.774+0000 7f27e67fc700 1 -- 172.21.15.16:0/3569608206 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27ec00c070 con 0x7f27f00cc8a0 2023-10-16T12:10:53.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.774+0000 7f27e67fc700 1 -- 172.21.15.16:0/3569608206 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27ec013610 con 0x7f27f00cc8a0 2023-10-16T12:10:53.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.774+0000 7f27e67fc700 1 -- 172.21.15.16:0/3569608206 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27ec013980 con 0x7f27f00cc8a0 2023-10-16T12:10:53.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.774+0000 7f27f7e47700 1 -- 172.21.15.16:0/3569608206 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27f01f6310 con 0x7f27f00cc8a0 2023-10-16T12:10:53.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.774+0000 7f27f7e47700 1 -- 172.21.15.16:0/3569608206 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27f0156ac0 con 0x7f27f00cc8a0 2023-10-16T12:10:53.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.775+0000 7f27f7e47700 1 -- 172.21.15.16:0/3569608206 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27f01f6170 con 0x7f27f00cc8a0 2023-10-16T12:10:53.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.775+0000 7f27e67fc700 1 -- 172.21.15.16:0/3569608206 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f27ec013ae0 con 0x7f27f00cc8a0 2023-10-16T12:10:53.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.776+0000 7f27e67fc700 1 --2- 172.21.15.16:0/3569608206 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f27d406af00 0x7f27d406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:10:53.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.776+0000 7f27e67fc700 1 -- 172.21.15.16:0/3569608206 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f27ec0896a0 con 0x7f27f00cc8a0 2023-10-16T12:10:53.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.776+0000 7f27f5be3700 1 --2- 172.21.15.16:0/3569608206 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f27d406af00 0x7f27d406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:10:53.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.776+0000 7f27f5be3700 1 --2- 172.21.15.16:0/3569608206 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f27d406af00 0x7f27d406d3c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f27e0000f60 tx=0x7f27e0007e70 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:10:53.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.780+0000 7f27e67fc700 1 -- 172.21.15.16:0/3569608206 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f27f01f6170 con 0x7f27f00cc8a0 2023-10-16T12:10:54.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.998+0000 7f27f7e47700 1 -- 172.21.15.16:0/3569608206 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f27f01f6170 con 0x7f27f00cc8a0 2023-10-16T12:10:54.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:53.998+0000 7f27e67fc700 1 -- 172.21.15.16:0/3569608206 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v25) v1 ==== 80+0+0 (secure 0 0 0) 0x7f27f01f6170 con 0x7f27f00cc8a0 2023-10-16T12:10:54.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:54.000+0000 7f27f7e47700 1 -- 172.21.15.16:0/3569608206 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f27d406af00 msgr2=0x7f27d406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:54.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:54.000+0000 7f27f7e47700 1 --2- 172.21.15.16:0/3569608206 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f27d406af00 0x7f27d406d3c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f27e0000f60 tx=0x7f27e0007e70 comp rx=0 tx=0).stop 2023-10-16T12:10:54.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:54.000+0000 7f27f7e47700 1 -- 172.21.15.16:0/3569608206 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27f00cc8a0 msgr2=0x7f27f00c4180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:10:54.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:54.000+0000 7f27f7e47700 1 --2- 172.21.15.16:0/3569608206 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27f00cc8a0 0x7f27f00c4180 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f27ec00e7e0 tx=0x7f27ec00eaf0 comp rx=0 tx=0).stop 2023-10-16T12:10:54.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:54.000+0000 7f27f7e47700 1 -- 172.21.15.16:0/3569608206 shutdown_connections 2023-10-16T12:10:54.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:54.000+0000 7f27f7e47700 1 --2- 172.21.15.16:0/3569608206 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f27d406af00 0x7f27d406d3c0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:54.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:54.000+0000 7f27f7e47700 1 --2- 172.21.15.16:0/3569608206 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f27f015b280 0x7f27f015ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:54.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:54.000+0000 7f27f7e47700 1 --2- 172.21.15.16:0/3569608206 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f27f006a7e0 0x7f27f0152a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:54.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:54.000+0000 7f27f7e47700 1 --2- 172.21.15.16:0/3569608206 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27f00cc8a0 0x7f27f00c4180 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:10:54.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:54.001+0000 7f27f7e47700 1 -- 172.21.15.16:0/3569608206 >> 172.21.15.16:0/3569608206 conn(0x7f27f00b1fd0 msgr2=0x7f27f0159c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:10:54.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:54.001+0000 7f27f7e47700 1 -- 172.21.15.16:0/3569608206 shutdown_connections 2023-10-16T12:10:54.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:10:54.001+0000 7f27f7e47700 1 -- 172.21.15.16:0/3569608206 wait complete. 2023-10-16T12:10:54.005 INFO:teuthology.orchestra.run.smithi016.stderr:listed 0 entries 2023-10-16T12:10:54.014 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi016.front.sepia.ceph.com... 2023-10-16T12:10:54.015 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:10:54.015 DEBUG:teuthology.orchestra.run.smithi016:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2023-10-16T12:10:54.053 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi016.stderr:ceph-fuse[111111]: fuse finished with error 0 and tester_r 0 2023-10-16T12:10:54.055 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T12:11:00.157 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2023-10-16T12:11:00.157 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:00.157 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2023-10-16T12:11:00.190 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:11:00.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.366+0000 7f8b66d58700 1 Processor -- start 2023-10-16T12:11:00.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.366+0000 7f8b66d58700 1 -- start start 2023-10-16T12:11:00.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.368+0000 7f8b66d58700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8b600d03b0 0x7f8b600cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:00.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.368+0000 7f8b66d58700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b600cc2a0 con 0x7f8b600ce4f0 2023-10-16T12:11:00.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.368+0000 7f8b66d58700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b600cc540 con 0x7f8b6015b630 2023-10-16T12:11:00.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.368+0000 7f8b66d58700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b600cdb20 con 0x7f8b600d03b0 2023-10-16T12:11:00.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.368+0000 7f8b652f5700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8b600d03b0 0x7f8b600cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:00.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.368+0000 7f8b5ffff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f8b600ce4f0 0x7f8b600cfca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50866/0 (socket says 172.21.15.16:50866) 2023-10-16T12:11:00.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.368+0000 7f8b5ffff700 1 -- 172.21.15.16:0/2370246627 learned_addr learned my addr 172.21.15.16:0/2370246627 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:00.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.368+0000 7f8b652f5700 1 -- 172.21.15.16:0/2370246627 >> v1:172.21.15.49:6789/0 conn(0x7f8b6015b630 legacy=0x7f8b600cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:00.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.368+0000 7f8b652f5700 1 -- 172.21.15.16:0/2370246627 >> v1:172.21.15.16:6789/0 conn(0x7f8b600ce4f0 legacy=0x7f8b600cfca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:00.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.368+0000 7f8b652f5700 1 -- 172.21.15.16:0/2370246627 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b600cc820 con 0x7f8b600d03b0 2023-10-16T12:11:00.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.368+0000 7f8b5f7fe700 1 -- 172.21.15.16:0/2370246627 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1263780126 0 0) 0x7f8b600cc2a0 con 0x7f8b600ce4f0 2023-10-16T12:11:00.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.368+0000 7f8b652f5700 1 --2- 172.21.15.16:0/2370246627 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b600d03b0 0x7f8b600cd5e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f8b58009d90 tx=0x7f8b5800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=224dc1aeb496ee6d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:00.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.368+0000 7f8b5f7fe700 1 -- 172.21.15.16:0/2370246627 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b5800e070 con 0x7f8b600d03b0 2023-10-16T12:11:00.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.368+0000 7f8b5f7fe700 1 -- 172.21.15.16:0/2370246627 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b5800bbd0 con 0x7f8b600d03b0 2023-10-16T12:11:00.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.368+0000 7f8b5f7fe700 1 -- 172.21.15.16:0/2370246627 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b58012430 con 0x7f8b600d03b0 2023-10-16T12:11:00.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.369+0000 7f8b66d58700 1 -- 172.21.15.16:0/2370246627 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b600d03b0 msgr2=0x7f8b600cd5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:00.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.369+0000 7f8b66d58700 1 --2- 172.21.15.16:0/2370246627 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b600d03b0 0x7f8b600cd5e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f8b58009d90 tx=0x7f8b5800b330 comp rx=0 tx=0).stop 2023-10-16T12:11:00.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.369+0000 7f8b66d58700 1 -- 172.21.15.16:0/2370246627 shutdown_connections 2023-10-16T12:11:00.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.369+0000 7f8b66d58700 1 --2- 172.21.15.16:0/2370246627 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b600d03b0 0x7f8b600cd5e0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:00.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.369+0000 7f8b66d58700 1 -- 172.21.15.16:0/2370246627 >> 172.21.15.16:0/2370246627 conn(0x7f8b600d14b0 msgr2=0x7f8b600d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:00.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.369+0000 7f8b66d58700 1 -- 172.21.15.16:0/2370246627 shutdown_connections 2023-10-16T12:11:00.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.369+0000 7f8b66d58700 1 -- 172.21.15.16:0/2370246627 wait complete. 2023-10-16T12:11:00.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.370+0000 7f8b66d58700 1 Processor -- start 2023-10-16T12:11:00.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.370+0000 7f8b66d58700 1 -- start start 2023-10-16T12:11:00.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.371+0000 7f8b66d58700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b600ce4f0 0x7f8b600c58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:00.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.371+0000 7f8b66d58700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8b600d03b0 0x7f8b601f0d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:00.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.371+0000 7f8b66d58700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8b6015b630 0x7f8b6006fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:00.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.371+0000 7f8b66d58700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b601f1ca0 con 0x7f8b600ce4f0 2023-10-16T12:11:00.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.371+0000 7f8b66d58700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b601f16e0 con 0x7f8b6015b630 2023-10-16T12:11:00.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.371+0000 7f8b66d58700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b601f19c0 con 0x7f8b600d03b0 2023-10-16T12:11:00.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.371+0000 7f8b64af4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b600ce4f0 0x7f8b600c58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:00.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.371+0000 7f8b64af4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b600ce4f0 0x7f8b600c58f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33364/0 (socket says 172.21.15.16:33364) 2023-10-16T12:11:00.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.371+0000 7f8b64af4700 1 -- 172.21.15.16:0/1546129220 learned_addr learned my addr 172.21.15.16:0/1546129220 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:00.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.372+0000 7f8b5ffff700 1 --2- 172.21.15.16:0/1546129220 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8b600d03b0 0x7f8b601f0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:00.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.372+0000 7f8b652f5700 1 --2- 172.21.15.16:0/1546129220 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8b6015b630 0x7f8b6006fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:00.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.372+0000 7f8b64af4700 1 -- 172.21.15.16:0/1546129220 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8b600d03b0 msgr2=0x7f8b601f0d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:00.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.372+0000 7f8b64af4700 1 --2- 172.21.15.16:0/1546129220 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8b600d03b0 0x7f8b601f0d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:00.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.372+0000 7f8b64af4700 1 -- 172.21.15.16:0/1546129220 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8b6015b630 msgr2=0x7f8b6006fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:00.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.372+0000 7f8b64af4700 1 --2- 172.21.15.16:0/1546129220 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8b6015b630 0x7f8b6006fbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:00.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.372+0000 7f8b64af4700 1 -- 172.21.15.16:0/1546129220 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b58009a70 con 0x7f8b600ce4f0 2023-10-16T12:11:00.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.372+0000 7f8b5ffff700 1 --2- 172.21.15.16:0/1546129220 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8b600d03b0 0x7f8b601f0d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:11:00.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.372+0000 7f8b64af4700 1 --2- 172.21.15.16:0/1546129220 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b600ce4f0 0x7f8b600c58f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f8b5400c4d0 tx=0x7f8b5400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:00.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.372+0000 7f8b5d7fa700 1 -- 172.21.15.16:0/1546129220 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b540052d0 con 0x7f8b600ce4f0 2023-10-16T12:11:00.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.372+0000 7f8b5d7fa700 1 -- 172.21.15.16:0/1546129220 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b54005430 con 0x7f8b600ce4f0 2023-10-16T12:11:00.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.372+0000 7f8b66d58700 1 -- 172.21.15.16:0/1546129220 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b600c50f0 con 0x7f8b600ce4f0 2023-10-16T12:11:00.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.372+0000 7f8b5d7fa700 1 -- 172.21.15.16:0/1546129220 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b540038f0 con 0x7f8b600ce4f0 2023-10-16T12:11:00.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.372+0000 7f8b66d58700 1 -- 172.21.15.16:0/1546129220 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b600c5e30 con 0x7f8b600ce4f0 2023-10-16T12:11:00.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.373+0000 7f8b66d58700 1 -- 172.21.15.16:0/1546129220 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b600c5610 con 0x7f8b600ce4f0 2023-10-16T12:11:00.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.374+0000 7f8b5d7fa700 1 -- 172.21.15.16:0/1546129220 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f8b54003a50 con 0x7f8b600ce4f0 2023-10-16T12:11:00.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.374+0000 7f8b5d7fa700 1 --2- 172.21.15.16:0/1546129220 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f8b4406af00 0x7f8b4406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:00.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.374+0000 7f8b5d7fa700 1 -- 172.21.15.16:0/1546129220 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f8b54088c80 con 0x7f8b600ce4f0 2023-10-16T12:11:00.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.374+0000 7f8b5ffff700 1 --2- 172.21.15.16:0/1546129220 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f8b4406af00 0x7f8b4406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:00.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.374+0000 7f8b5ffff700 1 --2- 172.21.15.16:0/1546129220 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f8b4406af00 0x7f8b4406d3c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8b600b5830 tx=0x7f8b4c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:00.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.379+0000 7f8b5d7fa700 1 -- 172.21.15.16:0/1546129220 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8b600c5610 con 0x7f8b600ce4f0 2023-10-16T12:11:00.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.582+0000 7f8b66d58700 1 -- 172.21.15.16:0/1546129220 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8b6004ebb0 con 0x7f8b600ce4f0 2023-10-16T12:11:00.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.583+0000 7f8b5d7fa700 1 -- 172.21.15.16:0/1546129220 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18238 (secure 0 0 0) 0x7f8b54017090 con 0x7f8b600ce4f0 2023-10-16T12:11:00.585 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:11:00.586 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":25,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:10:44.080141+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:10:36.935773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T12:10:38.004870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-16T12:10:41.672025+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:11:00.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.585+0000 7f8b66d58700 1 -- 172.21.15.16:0/1546129220 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f8b4406af00 msgr2=0x7f8b4406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:00.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.585+0000 7f8b66d58700 1 --2- 172.21.15.16:0/1546129220 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f8b4406af00 0x7f8b4406d3c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8b600b5830 tx=0x7f8b4c009040 comp rx=0 tx=0).stop 2023-10-16T12:11:00.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.585+0000 7f8b66d58700 1 -- 172.21.15.16:0/1546129220 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b600ce4f0 msgr2=0x7f8b600c58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:00.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.585+0000 7f8b66d58700 1 --2- 172.21.15.16:0/1546129220 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b600ce4f0 0x7f8b600c58f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f8b5400c4d0 tx=0x7f8b5400c890 comp rx=0 tx=0).stop 2023-10-16T12:11:00.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.585+0000 7f8b66d58700 1 -- 172.21.15.16:0/1546129220 shutdown_connections 2023-10-16T12:11:00.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.585+0000 7f8b66d58700 1 --2- 172.21.15.16:0/1546129220 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f8b4406af00 0x7f8b4406d3c0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:00.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.585+0000 7f8b66d58700 1 --2- 172.21.15.16:0/1546129220 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8b6015b630 0x7f8b6006fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:00.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.585+0000 7f8b66d58700 1 --2- 172.21.15.16:0/1546129220 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b600ce4f0 0x7f8b600c58f0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:00.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.585+0000 7f8b66d58700 1 --2- 172.21.15.16:0/1546129220 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8b600d03b0 0x7f8b601f0d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:00.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.585+0000 7f8b66d58700 1 -- 172.21.15.16:0/1546129220 >> 172.21.15.16:0/1546129220 conn(0x7f8b600d14b0 msgr2=0x7f8b6015a990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:00.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.585+0000 7f8b66d58700 1 -- 172.21.15.16:0/1546129220 shutdown_connections 2023-10-16T12:11:00.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.585+0000 7f8b66d58700 1 -- 172.21.15.16:0/1546129220 wait complete. 2023-10-16T12:11:00.599 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:11:00.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.722+0000 7f48a4fc7700 1 Processor -- start 2023-10-16T12:11:00.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.723+0000 7f48a4fc7700 1 -- start start 2023-10-16T12:11:00.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.723+0000 7f48a4fc7700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f48a0159c90 0x7f48a015d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:00.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.723+0000 7f48a4fc7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f48a015a070 0x7f48a015dee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:00.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.724+0000 7f48a4fc7700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48a006b1f0 con 0x7f48a00cf580 2023-10-16T12:11:00.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.724+0000 7f48a4fc7700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48a006b4b0 con 0x7f48a015a070 2023-10-16T12:11:00.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.724+0000 7f48a4fc7700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48a006b790 con 0x7f48a0159c90 2023-10-16T12:11:00.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.724+0000 7f489dd9b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f48a015a070 0x7f48a015dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:00.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.724+0000 7f489dd9b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f48a015a070 0x7f48a015dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33372/0 (socket says 172.21.15.16:33372) 2023-10-16T12:11:00.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.724+0000 7f489dd9b700 1 -- 172.21.15.16:0/457192523 learned_addr learned my addr 172.21.15.16:0/457192523 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:00.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.724+0000 7f489e59c700 1 --2- 172.21.15.16:0/457192523 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f48a0159c90 0x7f48a015d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:00.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.724+0000 7f489dd9b700 1 -- 172.21.15.16:0/457192523 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f48a0159c90 msgr2=0x7f48a015d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:00.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.724+0000 7f489dd9b700 1 --2- 172.21.15.16:0/457192523 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f48a0159c90 0x7f48a015d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:00.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.724+0000 7f489dd9b700 1 -- 172.21.15.16:0/457192523 >> v1:172.21.15.49:6789/0 conn(0x7f48a00cf580 legacy=0x7f48a00ae520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:00.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.724+0000 7f489dd9b700 1 -- 172.21.15.16:0/457192523 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48a006ba70 con 0x7f48a015a070 2023-10-16T12:11:00.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.724+0000 7f489dd9b700 1 --2- 172.21.15.16:0/457192523 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48a015a070 0x7f48a015dee0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f488800aa90 tx=0x7f488800ada0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b9cdb925fb5267a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:00.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.724+0000 7f489d59a700 1 -- 172.21.15.16:0/457192523 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f488800cb70 con 0x7f48a015a070 2023-10-16T12:11:00.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.724+0000 7f489d59a700 1 -- 172.21.15.16:0/457192523 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f488800ccd0 con 0x7f48a015a070 2023-10-16T12:11:00.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.724+0000 7f489d59a700 1 -- 172.21.15.16:0/457192523 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4888018620 con 0x7f48a015a070 2023-10-16T12:11:00.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.725+0000 7f48a4fc7700 1 -- 172.21.15.16:0/457192523 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48a015a070 msgr2=0x7f48a015dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:00.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.725+0000 7f48a4fc7700 1 --2- 172.21.15.16:0/457192523 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48a015a070 0x7f48a015dee0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f488800aa90 tx=0x7f488800ada0 comp rx=0 tx=0).stop 2023-10-16T12:11:00.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.725+0000 7f48a4fc7700 1 -- 172.21.15.16:0/457192523 shutdown_connections 2023-10-16T12:11:00.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.725+0000 7f48a4fc7700 1 --2- 172.21.15.16:0/457192523 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f48a0159c90 0x7f48a015d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:00.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.725+0000 7f48a4fc7700 1 --2- 172.21.15.16:0/457192523 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48a015a070 0x7f48a015dee0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:00.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.725+0000 7f48a4fc7700 1 -- 172.21.15.16:0/457192523 >> 172.21.15.16:0/457192523 conn(0x7f48a00cc5b0 msgr2=0x7f48a00cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:00.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.725+0000 7f48a4fc7700 1 -- 172.21.15.16:0/457192523 shutdown_connections 2023-10-16T12:11:00.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.725+0000 7f48a4fc7700 1 -- 172.21.15.16:0/457192523 wait complete. 2023-10-16T12:11:00.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.727+0000 7f48a4fc7700 1 Processor -- start 2023-10-16T12:11:00.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.727+0000 7f48a4fc7700 1 -- start start 2023-10-16T12:11:00.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.728+0000 7f48a4fc7700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f48a00cf580 0x7f48a01562e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:00.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.728+0000 7f48a4fc7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48a0159c90 0x7f48a00c4540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:00.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.728+0000 7f48a4fc7700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f48a015a070 0x7f48a00b39f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:00.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.728+0000 7f48a4fc7700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48a0156820 con 0x7f48a0159c90 2023-10-16T12:11:00.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.728+0000 7f48a4fc7700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48a00b2950 con 0x7f48a015a070 2023-10-16T12:11:00.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.728+0000 7f48a4fc7700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48a00b2c30 con 0x7f48a00cf580 2023-10-16T12:11:00.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.728+0000 7f489dd9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48a0159c90 0x7f48a00c4540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:00.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.728+0000 7f489dd9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48a0159c90 0x7f48a00c4540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33380/0 (socket says 172.21.15.16:33380) 2023-10-16T12:11:00.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.728+0000 7f489dd9b700 1 -- 172.21.15.16:0/4012677412 learned_addr learned my addr 172.21.15.16:0/4012677412 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:00.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.728+0000 7f489e59c700 1 --2- 172.21.15.16:0/4012677412 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f48a00cf580 0x7f48a01562e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:00.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.728+0000 7f489ed9d700 1 --2- 172.21.15.16:0/4012677412 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f48a015a070 0x7f48a00b39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:00.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.728+0000 7f489dd9b700 1 -- 172.21.15.16:0/4012677412 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f48a00cf580 msgr2=0x7f48a01562e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:00.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.728+0000 7f489dd9b700 1 --2- 172.21.15.16:0/4012677412 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f48a00cf580 0x7f48a01562e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:00.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.728+0000 7f489dd9b700 1 -- 172.21.15.16:0/4012677412 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f48a015a070 msgr2=0x7f48a00b39f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:00.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.728+0000 7f489dd9b700 1 --2- 172.21.15.16:0/4012677412 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f48a015a070 0x7f48a00b39f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:00.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.728+0000 7f489dd9b700 1 -- 172.21.15.16:0/4012677412 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f488800a6f0 con 0x7f48a0159c90 2023-10-16T12:11:00.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.728+0000 7f489ed9d700 1 --2- 172.21.15.16:0/4012677412 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f48a015a070 0x7f48a00b39f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:11:00.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.728+0000 7f489dd9b700 1 --2- 172.21.15.16:0/4012677412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48a0159c90 0x7f48a00c4540 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f488800ade0 tx=0x7f4888004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:00.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.729+0000 7f4896ffd700 1 -- 172.21.15.16:0/4012677412 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4888003ab0 con 0x7f48a0159c90 2023-10-16T12:11:00.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.729+0000 7f4896ffd700 1 -- 172.21.15.16:0/4012677412 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4888003c30 con 0x7f48a0159c90 2023-10-16T12:11:00.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.729+0000 7f48a4fc7700 1 -- 172.21.15.16:0/4012677412 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48a00b2f10 con 0x7f48a0159c90 2023-10-16T12:11:00.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.729+0000 7f4896ffd700 1 -- 172.21.15.16:0/4012677412 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4888020410 con 0x7f48a0159c90 2023-10-16T12:11:00.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.729+0000 7f48a4fc7700 1 -- 172.21.15.16:0/4012677412 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48a00b34d0 con 0x7f48a0159c90 2023-10-16T12:11:00.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.730+0000 7f48a4fc7700 1 -- 172.21.15.16:0/4012677412 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48a004ebb0 con 0x7f48a0159c90 2023-10-16T12:11:00.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.730+0000 7f4896ffd700 1 -- 172.21.15.16:0/4012677412 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f4888020590 con 0x7f48a0159c90 2023-10-16T12:11:00.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.731+0000 7f4896ffd700 1 --2- 172.21.15.16:0/4012677412 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f488006af00 0x7f488006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:00.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.731+0000 7f4896ffd700 1 -- 172.21.15.16:0/4012677412 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f488808d0b0 con 0x7f48a0159c90 2023-10-16T12:11:00.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.731+0000 7f489e59c700 1 --2- 172.21.15.16:0/4012677412 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f488006af00 0x7f488006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:00.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.731+0000 7f489e59c700 1 --2- 172.21.15.16:0/4012677412 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f488006af00 0x7f488006d3c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4890009be0 tx=0x7f4890009210 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:00.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.735+0000 7f4896ffd700 1 -- 172.21.15.16:0/4012677412 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4888057f00 con 0x7f48a0159c90 2023-10-16T12:11:00.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.943+0000 7f48a4fc7700 1 -- 172.21.15.16:0/4012677412 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f48a00c3910 con 0x7f48a0159c90 2023-10-16T12:11:00.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.944+0000 7f4896ffd700 1 -- 172.21.15.16:0/4012677412 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18238 (secure 0 0 0) 0x7f488805b520 con 0x7f48a0159c90 2023-10-16T12:11:00.947 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:11:00.947 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":25,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:10:44.080141+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:10:36.935773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T12:10:38.004870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-16T12:10:41.672025+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:11:00.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.947+0000 7f48a4fc7700 1 -- 172.21.15.16:0/4012677412 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f488006af00 msgr2=0x7f488006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:00.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.947+0000 7f48a4fc7700 1 --2- 172.21.15.16:0/4012677412 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f488006af00 0x7f488006d3c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4890009be0 tx=0x7f4890009210 comp rx=0 tx=0).stop 2023-10-16T12:11:00.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.947+0000 7f48a4fc7700 1 -- 172.21.15.16:0/4012677412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48a0159c90 msgr2=0x7f48a00c4540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:00.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.947+0000 7f48a4fc7700 1 --2- 172.21.15.16:0/4012677412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48a0159c90 0x7f48a00c4540 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f488800ade0 tx=0x7f4888004750 comp rx=0 tx=0).stop 2023-10-16T12:11:00.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.947+0000 7f48a4fc7700 1 -- 172.21.15.16:0/4012677412 shutdown_connections 2023-10-16T12:11:00.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.947+0000 7f48a4fc7700 1 --2- 172.21.15.16:0/4012677412 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f488006af00 0x7f488006d3c0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:00.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.947+0000 7f48a4fc7700 1 --2- 172.21.15.16:0/4012677412 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f48a015a070 0x7f48a00b39f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:00.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.947+0000 7f48a4fc7700 1 --2- 172.21.15.16:0/4012677412 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f48a00cf580 0x7f48a01562e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:00.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.947+0000 7f48a4fc7700 1 --2- 172.21.15.16:0/4012677412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48a0159c90 0x7f48a00c4540 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:00.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.947+0000 7f48a4fc7700 1 -- 172.21.15.16:0/4012677412 >> 172.21.15.16:0/4012677412 conn(0x7f48a00cc5b0 msgr2=0x7f48a0157060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:00.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.947+0000 7f48a4fc7700 1 -- 172.21.15.16:0/4012677412 shutdown_connections 2023-10-16T12:11:00.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:00.947+0000 7f48a4fc7700 1 -- 172.21.15.16:0/4012677412 wait complete. 2023-10-16T12:11:00.961 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-16T12:11:01.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.088+0000 7f1769bbe700 1 Processor -- start 2023-10-16T12:11:01.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.088+0000 7f1769bbe700 1 -- start start 2023-10-16T12:11:01.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.088+0000 7f1769bbe700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f176406beb0 0x7f17640ce890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:01.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.088+0000 7f1769bbe700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f176406aa10 con 0x7f1764160ee0 2023-10-16T12:11:01.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.088+0000 7f1769bbe700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17640ceed0 con 0x7f17640af210 2023-10-16T12:11:01.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.088+0000 7f1769bbe700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17640cf770 con 0x7f176406beb0 2023-10-16T12:11:01.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.089+0000 7f17637fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f1764160ee0 0x7f17640aeb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50892/0 (socket says 172.21.15.16:50892) 2023-10-16T12:11:01.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.089+0000 7f17637fe700 1 -- 172.21.15.16:0/2873112704 learned_addr learned my addr 172.21.15.16:0/2873112704 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:01.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.089+0000 7f1763fff700 1 --2- 172.21.15.16:0/2873112704 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f176406beb0 0x7f17640ce890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:01.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.089+0000 7f17627fc700 1 -- 172.21.15.16:0/2873112704 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 803027205 0 0) 0x7f176406aa10 con 0x7f1764160ee0 2023-10-16T12:11:01.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.089+0000 7f17627fc700 1 -- 172.21.15.16:0/2873112704 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f174c0035b0 con 0x7f1764160ee0 2023-10-16T12:11:01.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.089+0000 7f17627fc700 1 -- 172.21.15.16:0/2873112704 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2803503790 0 0) 0x7f17640ceed0 con 0x7f17640af210 2023-10-16T12:11:01.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.089+0000 7f17627fc700 1 -- 172.21.15.16:0/2873112704 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f176406aa10 con 0x7f17640af210 2023-10-16T12:11:01.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.089+0000 7f17627fc700 1 -- 172.21.15.16:0/2873112704 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3168094205 0 0) 0x7f174c0035b0 con 0x7f1764160ee0 2023-10-16T12:11:01.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.089+0000 7f17627fc700 1 -- 172.21.15.16:0/2873112704 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f176406beb0 msgr2=0x7f17640ce890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:01.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.089+0000 7f17627fc700 1 --2- 172.21.15.16:0/2873112704 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f176406beb0 0x7f17640ce890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:01.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.089+0000 7f17627fc700 1 -- 172.21.15.16:0/2873112704 >> v1:172.21.15.49:6789/0 conn(0x7f17640af210 legacy=0x7f17640ce180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:01.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.089+0000 7f17627fc700 1 -- 172.21.15.16:0/2873112704 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17640cf1b0 con 0x7f1764160ee0 2023-10-16T12:11:01.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.089+0000 7f17627fc700 1 -- 172.21.15.16:0/2873112704 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f1754001670 con 0x7f1764160ee0 2023-10-16T12:11:01.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.089+0000 7f17627fc700 1 -- 172.21.15.16:0/2873112704 >> v1:172.21.15.16:6789/0 conn(0x7f1764160ee0 legacy=0x7f17640aeb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:01.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.089+0000 7f17627fc700 1 --2- 172.21.15.16:0/2873112704 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f174c003bc0 0x7f174c003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:01.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.089+0000 7f17627fc700 1 --2- 172.21.15.16:0/2873112704 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f176406beb0 0x7f174c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:01.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.089+0000 7f17627fc700 1 --2- 172.21.15.16:0/2873112704 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f174c004d90 0x7f174c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:01.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.089+0000 7f17627fc700 1 -- 172.21.15.16:0/2873112704 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17640cf770 con 0x7f174c003bc0 2023-10-16T12:11:01.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.089+0000 7f17627fc700 1 -- 172.21.15.16:0/2873112704 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f174c00baf0 con 0x7f174c004d90 2023-10-16T12:11:01.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.089+0000 7f17627fc700 1 -- 172.21.15.16:0/2873112704 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f174c00bcb0 con 0x7f176406beb0 2023-10-16T12:11:01.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.089+0000 7f17637fe700 1 --2- 172.21.15.16:0/2873112704 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f174c003bc0 0x7f174c003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:01.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.090+0000 7f1763fff700 1 --2- 172.21.15.16:0/2873112704 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f174c004d90 0x7f174c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:01.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.090+0000 7f1762ffd700 1 --2- 172.21.15.16:0/2873112704 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f176406beb0 0x7f174c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:01.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.090+0000 7f17637fe700 1 -- 172.21.15.16:0/2873112704 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f176406beb0 msgr2=0x7f174c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:01.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.090+0000 7f17637fe700 1 --2- 172.21.15.16:0/2873112704 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f176406beb0 0x7f174c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:01.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.090+0000 7f17637fe700 1 -- 172.21.15.16:0/2873112704 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f174c004d90 msgr2=0x7f174c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:01.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.090+0000 7f17637fe700 1 --2- 172.21.15.16:0/2873112704 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f174c004d90 0x7f174c005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:01.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.090+0000 7f17637fe700 1 -- 172.21.15.16:0/2873112704 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f174c0035b0 con 0x7f174c003bc0 2023-10-16T12:11:01.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.090+0000 7f17637fe700 1 --2- 172.21.15.16:0/2873112704 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f174c003bc0 0x7f174c003fc0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f175400c110 tx=0x7f174c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51d009449ae215d1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:01.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.090+0000 7f17627fc700 1 -- 172.21.15.16:0/2873112704 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f175400ccb0 con 0x7f174c003bc0 2023-10-16T12:11:01.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.090+0000 7f17627fc700 1 -- 172.21.15.16:0/2873112704 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f175400ce10 con 0x7f174c003bc0 2023-10-16T12:11:01.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.090+0000 7f17627fc700 1 -- 172.21.15.16:0/2873112704 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1754005ba0 con 0x7f174c003bc0 2023-10-16T12:11:01.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.091+0000 7f1769bbe700 1 -- 172.21.15.16:0/2873112704 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f174c003bc0 msgr2=0x7f174c003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:01.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.091+0000 7f1769bbe700 1 --2- 172.21.15.16:0/2873112704 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f174c003bc0 0x7f174c003fc0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f175400c110 tx=0x7f174c005880 comp rx=0 tx=0).stop 2023-10-16T12:11:01.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.091+0000 7f1763fff700 1 -- 172.21.15.16:0/2873112704 reap_dead start 2023-10-16T12:11:01.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.091+0000 7f1769bbe700 1 -- 172.21.15.16:0/2873112704 shutdown_connections 2023-10-16T12:11:01.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.091+0000 7f1769bbe700 1 -- 172.21.15.16:0/2873112704 >> 172.21.15.16:0/2873112704 conn(0x7f17640cc5e0 msgr2=0x7f17640cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:01.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.091+0000 7f1769bbe700 1 -- 172.21.15.16:0/2873112704 shutdown_connections 2023-10-16T12:11:01.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.091+0000 7f1769bbe700 1 -- 172.21.15.16:0/2873112704 wait complete. 2023-10-16T12:11:01.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.092+0000 7f1769bbe700 1 Processor -- start 2023-10-16T12:11:01.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.092+0000 7f1769bbe700 1 -- start start 2023-10-16T12:11:01.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.093+0000 7f1769bbe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f174c003bc0 0x7f17640c3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:01.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.093+0000 7f1769bbe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1764158c40 0x7f1764156060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:01.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.093+0000 7f1769bbe700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f17640cb020 0x7f17640cb4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:01.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.093+0000 7f1769bbe700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17640cb9f0 con 0x7f1764158c40 2023-10-16T12:11:01.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.093+0000 7f1769bbe700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17640c46f0 con 0x7f174c003bc0 2023-10-16T12:11:01.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.093+0000 7f1769bbe700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17640c3bd0 con 0x7f17640cb020 2023-10-16T12:11:01.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.093+0000 7f1762ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1764158c40 0x7f1764156060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:01.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.093+0000 7f1762ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1764158c40 0x7f1764156060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33402/0 (socket says 172.21.15.16:33402) 2023-10-16T12:11:01.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.093+0000 7f1762ffd700 1 -- 172.21.15.16:0/3190271128 learned_addr learned my addr 172.21.15.16:0/3190271128 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:01.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.093+0000 7f17637fe700 1 --2- 172.21.15.16:0/3190271128 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f174c003bc0 0x7f17640c3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:01.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.093+0000 7f1763fff700 1 --2- 172.21.15.16:0/3190271128 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f17640cb020 0x7f17640cb4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:01.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.093+0000 7f1762ffd700 1 -- 172.21.15.16:0/3190271128 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f17640cb020 msgr2=0x7f17640cb4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:01.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.093+0000 7f1762ffd700 1 --2- 172.21.15.16:0/3190271128 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f17640cb020 0x7f17640cb4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:01.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.093+0000 7f1762ffd700 1 -- 172.21.15.16:0/3190271128 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f174c003bc0 msgr2=0x7f17640c3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:01.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.093+0000 7f1762ffd700 1 --2- 172.21.15.16:0/3190271128 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f174c003bc0 0x7f17640c3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:01.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.093+0000 7f1762ffd700 1 -- 172.21.15.16:0/3190271128 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17640c3e50 con 0x7f1764158c40 2023-10-16T12:11:01.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.093+0000 7f17637fe700 1 --2- 172.21.15.16:0/3190271128 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f174c003bc0 0x7f17640c3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:11:01.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.093+0000 7f1762ffd700 1 --2- 172.21.15.16:0/3190271128 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1764158c40 0x7f1764156060 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f175800f840 tx=0x7f175800fc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:01.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.093+0000 7f174bfff700 1 -- 172.21.15.16:0/3190271128 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1758005320 con 0x7f1764158c40 2023-10-16T12:11:01.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.094+0000 7f174bfff700 1 -- 172.21.15.16:0/3190271128 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1758005480 con 0x7f1764158c40 2023-10-16T12:11:01.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.094+0000 7f1769bbe700 1 -- 172.21.15.16:0/3190271128 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17640c4130 con 0x7f1764158c40 2023-10-16T12:11:01.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.094+0000 7f174bfff700 1 -- 172.21.15.16:0/3190271128 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1758006aa0 con 0x7f1764158c40 2023-10-16T12:11:01.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.094+0000 7f1769bbe700 1 -- 172.21.15.16:0/3190271128 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17640c74d0 con 0x7f1764158c40 2023-10-16T12:11:01.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.094+0000 7f1769bbe700 1 -- 172.21.15.16:0/3190271128 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17640c76f0 con 0x7f1764158c40 2023-10-16T12:11:01.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.095+0000 7f174bfff700 1 -- 172.21.15.16:0/3190271128 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f1758006c00 con 0x7f1764158c40 2023-10-16T12:11:01.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.095+0000 7f174bfff700 1 --2- 172.21.15.16:0/3190271128 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f175006af00 0x7f175006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:01.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.095+0000 7f174bfff700 1 -- 172.21.15.16:0/3190271128 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f17580892f0 con 0x7f1764158c40 2023-10-16T12:11:01.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.096+0000 7f17637fe700 1 --2- 172.21.15.16:0/3190271128 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f175006af00 0x7f175006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:01.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.096+0000 7f17637fe700 1 --2- 172.21.15.16:0/3190271128 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f175006af00 0x7f175006d3c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f175400c110 tx=0x7f1754007000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:01.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.100+0000 7f174bfff700 1 -- 172.21.15.16:0/3190271128 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f17580540b0 con 0x7f1764158c40 2023-10-16T12:11:01.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:01.355+0000 7f1769bbe700 1 -- 172.21.15.16:0/3190271128 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f176404ebb0 con 0x7f1764158c40 2023-10-16T12:11:02.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:02.171+0000 7f174bfff700 1 -- 172.21.15.16:0/3190271128 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v8) v1 ==== 118+0+0 (secure 0 0 0) 0x7f17580576d0 con 0x7f1764158c40 2023-10-16T12:11:02.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:02.174+0000 7f1769bbe700 1 -- 172.21.15.16:0/3190271128 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f175006af00 msgr2=0x7f175006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:02.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:02.174+0000 7f1769bbe700 1 --2- 172.21.15.16:0/3190271128 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f175006af00 0x7f175006d3c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f175400c110 tx=0x7f1754007000 comp rx=0 tx=0).stop 2023-10-16T12:11:02.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:02.174+0000 7f1769bbe700 1 -- 172.21.15.16:0/3190271128 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1764158c40 msgr2=0x7f1764156060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:02.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:02.174+0000 7f1769bbe700 1 --2- 172.21.15.16:0/3190271128 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1764158c40 0x7f1764156060 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f175800f840 tx=0x7f175800fc00 comp rx=0 tx=0).stop 2023-10-16T12:11:02.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:02.174+0000 7f1769bbe700 1 -- 172.21.15.16:0/3190271128 shutdown_connections 2023-10-16T12:11:02.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:02.174+0000 7f1769bbe700 1 --2- 172.21.15.16:0/3190271128 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f175006af00 0x7f175006d3c0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:02.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:02.174+0000 7f1769bbe700 1 --2- 172.21.15.16:0/3190271128 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f17640cb020 0x7f17640cb4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:02.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:02.174+0000 7f1769bbe700 1 --2- 172.21.15.16:0/3190271128 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f174c003bc0 0x7f17640c3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:02.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:02.174+0000 7f1769bbe700 1 --2- 172.21.15.16:0/3190271128 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1764158c40 0x7f1764156060 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:02.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:02.174+0000 7f1769bbe700 1 -- 172.21.15.16:0/3190271128 >> 172.21.15.16:0/3190271128 conn(0x7f17640cc5e0 msgr2=0x7f17640b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:02.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:02.174+0000 7f1769bbe700 1 -- 172.21.15.16:0/3190271128 shutdown_connections 2023-10-16T12:11:02.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:02.174+0000 7f1769bbe700 1 -- 172.21.15.16:0/3190271128 wait complete. 2023-10-16T12:11:02.191 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']} 2023-10-16T12:11:02.191 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-16T12:11:02.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:02.325+0000 7ff648678700 1 Processor -- start 2023-10-16T12:11:02.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:02.325+0000 7ff648678700 1 -- start start 2023-10-16T12:11:02.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:02.326+0000 7ff648678700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff6400d2110 0x7ff6400ae1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:02.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:02.326+0000 7ff648678700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7ff64006ca10 0x7ff6400ae720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:02.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:02.326+0000 7ff648678700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6400b20f0 con 0x7ff6400ce6a0 2023-10-16T12:11:02.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:02.326+0000 7ff648678700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6400d03d0 con 0x7ff6400d2110 2023-10-16T12:11:02.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:02.326+0000 7ff648678700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6400b2340 con 0x7ff64006ca10 2023-10-16T12:11:02.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:02.326+0000 7ff645c13700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff6400d2110 0x7ff6400ae1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:02.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:02.326+0000 7ff645c13700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff6400d2110 0x7ff6400ae1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33408/0 (socket says 172.21.15.16:33408) 2023-10-16T12:11:02.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:02.326+0000 7ff645c13700 1 -- 172.21.15.16:0/812452345 learned_addr learned my addr 172.21.15.16:0/812452345 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:02.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:02.326+0000 7ff646c15700 1 --2- 172.21.15.16:0/812452345 >> v2:172.21.15.49:3300/0 conn(0x7ff64006ca10 0x7ff6400ae720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:02.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:02.326+0000 7ff645c13700 1 -- 172.21.15.16:0/812452345 >> v2:172.21.15.49:3300/0 conn(0x7ff64006ca10 msgr2=0x7ff6400ae720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:02.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:02.326+0000 7ff645c13700 1 --2- 172.21.15.16:0/812452345 >> v2:172.21.15.49:3300/0 conn(0x7ff64006ca10 0x7ff6400ae720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:02.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:02.326+0000 7ff645c13700 1 -- 172.21.15.16:0/812452345 >> v1:172.21.15.49:6789/0 conn(0x7ff6400ce6a0 legacy=0x7ff64006c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:02.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:02.326+0000 7ff645c13700 1 -- 172.21.15.16:0/812452345 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff64006bb00 con 0x7ff6400d2110 2023-10-16T12:11:02.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:02.326+0000 7ff645c13700 1 --2- 172.21.15.16:0/812452345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6400d2110 0x7ff6400ae1e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff64006b7c0 tx=0x7ff630009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f73571bcceaa6758 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:02.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:02.327+0000 7ff645412700 1 -- 172.21.15.16:0/812452345 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff630004230 con 0x7ff6400d2110 2023-10-16T12:11:02.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:02.327+0000 7ff645412700 1 -- 172.21.15.16:0/812452345 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff630004390 con 0x7ff6400d2110 2023-10-16T12:11:02.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:02.327+0000 7ff645412700 1 -- 172.21.15.16:0/812452345 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6300036a0 con 0x7ff6400d2110 2023-10-16T12:11:02.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:02.327+0000 7ff648678700 1 -- 172.21.15.16:0/812452345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6400d2110 msgr2=0x7ff6400ae1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:02.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:02.327+0000 7ff648678700 1 --2- 172.21.15.16:0/812452345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6400d2110 0x7ff6400ae1e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff64006b7c0 tx=0x7ff630009bd0 comp rx=0 tx=0).stop 2023-10-16T12:11:02.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:02.327+0000 7ff648678700 1 -- 172.21.15.16:0/812452345 shutdown_connections 2023-10-16T12:11:02.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:02.327+0000 7ff648678700 1 --2- 172.21.15.16:0/812452345 >> v2:172.21.15.49:3300/0 conn(0x7ff64006ca10 0x7ff6400ae720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:02.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:02.327+0000 7ff648678700 1 --2- 172.21.15.16:0/812452345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6400d2110 0x7ff6400ae1e0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:02.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:02.327+0000 7ff648678700 1 -- 172.21.15.16:0/812452345 >> 172.21.15.16:0/812452345 conn(0x7ff6400d15f0 msgr2=0x7ff6400d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:02.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:02.327+0000 7ff648678700 1 -- 172.21.15.16:0/812452345 shutdown_connections 2023-10-16T12:11:02.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:02.327+0000 7ff648678700 1 -- 172.21.15.16:0/812452345 wait complete. 2023-10-16T12:11:02.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:02.328+0000 7ff648678700 1 Processor -- start 2023-10-16T12:11:02.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:02.329+0000 7ff648678700 1 -- start start 2023-10-16T12:11:02.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:02.329+0000 7ff648678700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff64006ca10 0x7ff6400c5340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:02.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:02.329+0000 7ff648678700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6400ce6a0 0x7ff6401525e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:02.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:02.329+0000 7ff648678700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff6400d2110 0x7ff640152b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:02.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:02.329+0000 7ff648678700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6400c7d80 con 0x7ff6400ce6a0 2023-10-16T12:11:02.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:02.329+0000 7ff648678700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6400c7170 con 0x7ff64006ca10 2023-10-16T12:11:02.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:02.329+0000 7ff648678700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6400c7450 con 0x7ff6400d2110 2023-10-16T12:11:02.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:02.329+0000 7ff645c13700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6400ce6a0 0x7ff6401525e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:02.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:02.329+0000 7ff645c13700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6400ce6a0 0x7ff6401525e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33414/0 (socket says 172.21.15.16:33414) 2023-10-16T12:11:02.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:02.329+0000 7ff645c13700 1 -- 172.21.15.16:0/2911519828 learned_addr learned my addr 172.21.15.16:0/2911519828 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:02.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:02.330+0000 7ff646414700 1 --2- 172.21.15.16:0/2911519828 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff64006ca10 0x7ff6400c5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:02.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:02.330+0000 7ff646c15700 1 --2- 172.21.15.16:0/2911519828 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff6400d2110 0x7ff640152b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:02.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:02.330+0000 7ff645c13700 1 -- 172.21.15.16:0/2911519828 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff6400d2110 msgr2=0x7ff640152b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:02.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:02.330+0000 7ff645c13700 1 --2- 172.21.15.16:0/2911519828 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff6400d2110 0x7ff640152b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:02.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:02.330+0000 7ff645c13700 1 -- 172.21.15.16:0/2911519828 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff64006ca10 msgr2=0x7ff6400c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:02.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:02.330+0000 7ff645c13700 1 --2- 172.21.15.16:0/2911519828 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff64006ca10 0x7ff6400c5340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:02.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:02.330+0000 7ff645c13700 1 -- 172.21.15.16:0/2911519828 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff630009550 con 0x7ff6400ce6a0 2023-10-16T12:11:02.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:02.330+0000 7ff645c13700 1 --2- 172.21.15.16:0/2911519828 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6400ce6a0 0x7ff6401525e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ff630009130 tx=0x7ff630004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:02.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:02.330+0000 7ff636ffd700 1 -- 172.21.15.16:0/2911519828 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff630011630 con 0x7ff6400ce6a0 2023-10-16T12:11:02.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:02.330+0000 7ff636ffd700 1 -- 172.21.15.16:0/2911519828 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff630011790 con 0x7ff6400ce6a0 2023-10-16T12:11:02.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:02.330+0000 7ff648678700 1 -- 172.21.15.16:0/2911519828 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6400c7730 con 0x7ff6400ce6a0 2023-10-16T12:11:02.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:02.330+0000 7ff636ffd700 1 -- 172.21.15.16:0/2911519828 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff630011b40 con 0x7ff6400ce6a0 2023-10-16T12:11:02.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:02.330+0000 7ff648678700 1 -- 172.21.15.16:0/2911519828 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6401501e0 con 0x7ff6400ce6a0 2023-10-16T12:11:02.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:02.331+0000 7ff648678700 1 -- 172.21.15.16:0/2911519828 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6400d0670 con 0x7ff6400ce6a0 2023-10-16T12:11:02.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:02.332+0000 7ff636ffd700 1 -- 172.21.15.16:0/2911519828 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7ff630011ca0 con 0x7ff6400ce6a0 2023-10-16T12:11:02.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:02.332+0000 7ff636ffd700 1 --2- 172.21.15.16:0/2911519828 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7ff62406af00 0x7ff62406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:02.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:02.332+0000 7ff646414700 1 --2- 172.21.15.16:0/2911519828 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7ff62406af00 0x7ff62406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:02.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:02.332+0000 7ff636ffd700 1 -- 172.21.15.16:0/2911519828 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7ff63008ae50 con 0x7ff6400ce6a0 2023-10-16T12:11:02.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:02.332+0000 7ff646414700 1 --2- 172.21.15.16:0/2911519828 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7ff62406af00 0x7ff62406d3c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff63c004f30 tx=0x7ff63c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:02.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:02.337+0000 7ff636ffd700 1 -- 172.21.15.16:0/2911519828 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff6400d0670 con 0x7ff6400ce6a0 2023-10-16T12:11:02.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:02.592+0000 7ff648678700 1 -- 172.21.15.16:0/2911519828 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7ff6400d0670 con 0x7ff6400ce6a0 2023-10-16T12:11:03.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.176+0000 7ff636ffd700 1 -- 172.21.15.16:0/2911519828 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v9) v1 ==== 122+0+0 (secure 0 0 0) 0x7ff6400d0670 con 0x7ff6400ce6a0 2023-10-16T12:11:03.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.179+0000 7ff648678700 1 -- 172.21.15.16:0/2911519828 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7ff62406af00 msgr2=0x7ff62406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:03.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.179+0000 7ff648678700 1 --2- 172.21.15.16:0/2911519828 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7ff62406af00 0x7ff62406d3c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff63c004f30 tx=0x7ff63c009040 comp rx=0 tx=0).stop 2023-10-16T12:11:03.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.179+0000 7ff648678700 1 -- 172.21.15.16:0/2911519828 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6400ce6a0 msgr2=0x7ff6401525e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:03.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.179+0000 7ff648678700 1 --2- 172.21.15.16:0/2911519828 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6400ce6a0 0x7ff6401525e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ff630009130 tx=0x7ff630004750 comp rx=0 tx=0).stop 2023-10-16T12:11:03.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.179+0000 7ff648678700 1 -- 172.21.15.16:0/2911519828 shutdown_connections 2023-10-16T12:11:03.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.179+0000 7ff648678700 1 --2- 172.21.15.16:0/2911519828 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7ff62406af00 0x7ff62406d3c0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:03.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.179+0000 7ff648678700 1 --2- 172.21.15.16:0/2911519828 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff6400d2110 0x7ff640152b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:03.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.179+0000 7ff648678700 1 --2- 172.21.15.16:0/2911519828 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff64006ca10 0x7ff6400c5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:03.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.179+0000 7ff648678700 1 --2- 172.21.15.16:0/2911519828 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6400ce6a0 0x7ff6401525e0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:03.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.179+0000 7ff648678700 1 -- 172.21.15.16:0/2911519828 >> 172.21.15.16:0/2911519828 conn(0x7ff6400d15f0 msgr2=0x7ff6400cbc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:03.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.179+0000 7ff648678700 1 -- 172.21.15.16:0/2911519828 shutdown_connections 2023-10-16T12:11:03.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.179+0000 7ff648678700 1 -- 172.21.15.16:0/2911519828 wait complete. 2023-10-16T12:11:03.195 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-16T12:11:03.196 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:11:03.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.327+0000 7f49622ec700 1 Processor -- start 2023-10-16T12:11:03.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.327+0000 7f49622ec700 1 -- start start 2023-10-16T12:11:03.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.328+0000 7f49622ec700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f495c159c10 0x7f495c06c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:03.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.328+0000 7f49622ec700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f495c06cd40 0x7f495c0b3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:03.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.328+0000 7f49622ec700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f495c06d1a0 con 0x7f495c0b3b80 2023-10-16T12:11:03.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.328+0000 7f49622ec700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f495c0b2b20 con 0x7f495c06cd40 2023-10-16T12:11:03.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.328+0000 7f49622ec700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f495c0b21c0 con 0x7f495c159c10 2023-10-16T12:11:03.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.328+0000 7f495bfff700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f495c159c10 0x7f495c06c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:03.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.328+0000 7f495b7fe700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f495c06cd40 0x7f495c0b3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:03.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.328+0000 7f495bfff700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f495c159c10 0x7f495c06c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:49910/0 (socket says 172.21.15.16:49910) 2023-10-16T12:11:03.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.328+0000 7f4960889700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7f495c0b3b80 0x7f495c070e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:58112/0 (socket says 172.21.15.16:58112) 2023-10-16T12:11:03.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.328+0000 7f495bfff700 1 -- 172.21.15.16:0/1853651896 learned_addr learned my addr 172.21.15.16:0/1853651896 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:03.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.329+0000 7f495bfff700 1 -- 172.21.15.16:0/1853651896 >> v2:172.21.15.49:3300/0 conn(0x7f495c06cd40 msgr2=0x7f495c0b3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:03.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.329+0000 7f495bfff700 1 --2- 172.21.15.16:0/1853651896 >> v2:172.21.15.49:3300/0 conn(0x7f495c06cd40 0x7f495c0b3640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:03.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.329+0000 7f495bfff700 1 -- 172.21.15.16:0/1853651896 >> v1:172.21.15.49:6789/0 conn(0x7f495c0b3b80 legacy=0x7f495c070e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:03.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.329+0000 7f495bfff700 1 -- 172.21.15.16:0/1853651896 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f495c0b2460 con 0x7f495c159c10 2023-10-16T12:11:03.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.329+0000 7f495b7fe700 1 --2- 172.21.15.16:0/1853651896 >> v2:172.21.15.49:3300/0 conn(0x7f495c06cd40 0x7f495c0b3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:11:03.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.329+0000 7f495bfff700 1 --2- 172.21.15.16:0/1853651896 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f495c159c10 0x7f495c06c800 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f494c00aa90 tx=0x7f494c00ada0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3061518dc8ddf1f9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:03.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.329+0000 7f495affd700 1 -- 172.21.15.16:0/1853651896 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f494c00cb70 con 0x7f495c159c10 2023-10-16T12:11:03.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.329+0000 7f495affd700 1 -- 172.21.15.16:0/1853651896 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f494c00ccd0 con 0x7f495c159c10 2023-10-16T12:11:03.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.329+0000 7f495affd700 1 -- 172.21.15.16:0/1853651896 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f494c018620 con 0x7f495c159c10 2023-10-16T12:11:03.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.330+0000 7f49622ec700 1 -- 172.21.15.16:0/1853651896 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f495c159c10 msgr2=0x7f495c06c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:03.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.330+0000 7f49622ec700 1 --2- 172.21.15.16:0/1853651896 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f495c159c10 0x7f495c06c800 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f494c00aa90 tx=0x7f494c00ada0 comp rx=0 tx=0).stop 2023-10-16T12:11:03.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.330+0000 7f49622ec700 1 -- 172.21.15.16:0/1853651896 shutdown_connections 2023-10-16T12:11:03.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.330+0000 7f49622ec700 1 --2- 172.21.15.16:0/1853651896 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f495c159c10 0x7f495c06c800 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:03.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.330+0000 7f49622ec700 1 --2- 172.21.15.16:0/1853651896 >> v2:172.21.15.49:3300/0 conn(0x7f495c06cd40 0x7f495c0b3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:03.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.330+0000 7f49622ec700 1 -- 172.21.15.16:0/1853651896 >> 172.21.15.16:0/1853651896 conn(0x7f495c0b4cd0 msgr2=0x7f495c06a7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:03.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.330+0000 7f49622ec700 1 -- 172.21.15.16:0/1853651896 shutdown_connections 2023-10-16T12:11:03.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.330+0000 7f49622ec700 1 -- 172.21.15.16:0/1853651896 wait complete. 2023-10-16T12:11:03.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.331+0000 7f49622ec700 1 Processor -- start 2023-10-16T12:11:03.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.331+0000 7f49622ec700 1 -- start start 2023-10-16T12:11:03.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.332+0000 7f49622ec700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f495c06cd40 0x7f495c15b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:03.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.332+0000 7f49622ec700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f495c0b3b80 0x7f495c1e5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:03.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.332+0000 7f49622ec700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f495c159c10 0x7f495c148da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:03.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.332+0000 7f49622ec700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f495c0ae8b0 con 0x7f495c0b3b80 2023-10-16T12:11:03.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.332+0000 7f49622ec700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f495c0add30 con 0x7f495c06cd40 2023-10-16T12:11:03.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.332+0000 7f49622ec700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f495c1e5750 con 0x7f495c159c10 2023-10-16T12:11:03.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.332+0000 7f495b7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f495c0b3b80 0x7f495c1e5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:03.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.332+0000 7f495bfff700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f495c06cd40 0x7f495c15b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:03.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.332+0000 7f495b7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f495c0b3b80 0x7f495c1e5210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33428/0 (socket says 172.21.15.16:33428) 2023-10-16T12:11:03.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.332+0000 7f495b7fe700 1 -- 172.21.15.16:0/3871603097 learned_addr learned my addr 172.21.15.16:0/3871603097 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:03.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.332+0000 7f4960889700 1 --2- 172.21.15.16:0/3871603097 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f495c159c10 0x7f495c148da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:03.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.333+0000 7f495bfff700 1 -- 172.21.15.16:0/3871603097 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f495c159c10 msgr2=0x7f495c148da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:03.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.333+0000 7f495bfff700 1 --2- 172.21.15.16:0/3871603097 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f495c159c10 0x7f495c148da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:03.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.333+0000 7f495bfff700 1 -- 172.21.15.16:0/3871603097 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f495c0b3b80 msgr2=0x7f495c1e5210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:03.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.333+0000 7f495bfff700 1 --2- 172.21.15.16:0/3871603097 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f495c0b3b80 0x7f495c1e5210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:03.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.333+0000 7f495bfff700 1 -- 172.21.15.16:0/3871603097 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49440094a0 con 0x7f495c06cd40 2023-10-16T12:11:03.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.333+0000 7f495b7fe700 1 --2- 172.21.15.16:0/3871603097 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f495c0b3b80 0x7f495c1e5210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:11:03.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.333+0000 7f495bfff700 1 --2- 172.21.15.16:0/3871603097 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f495c06cd40 0x7f495c15b6c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f494c017000 tx=0x7f494c0047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:03.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.333+0000 7f4958ff9700 1 -- 172.21.15.16:0/3871603097 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f494c003ab0 con 0x7f495c06cd40 2023-10-16T12:11:03.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.333+0000 7f4958ff9700 1 -- 172.21.15.16:0/3871603097 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f494c003c10 con 0x7f495c06cd40 2023-10-16T12:11:03.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.333+0000 7f49622ec700 1 -- 172.21.15.16:0/3871603097 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f494c00a6f0 con 0x7f495c06cd40 2023-10-16T12:11:03.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.333+0000 7f4958ff9700 1 -- 172.21.15.16:0/3871603097 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f494c020400 con 0x7f495c06cd40 2023-10-16T12:11:03.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.333+0000 7f49622ec700 1 -- 172.21.15.16:0/3871603097 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f495c0adf80 con 0x7f495c06cd40 2023-10-16T12:11:03.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.334+0000 7f49622ec700 1 -- 172.21.15.16:0/3871603097 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f495c04ebb0 con 0x7f495c06cd40 2023-10-16T12:11:03.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.334+0000 7f4958ff9700 1 -- 172.21.15.16:0/3871603097 <== mon.1 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f494c018400 con 0x7f495c06cd40 2023-10-16T12:11:03.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.335+0000 7f4958ff9700 1 --2- 172.21.15.16:0/3871603097 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f493c06af00 0x7f493c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:03.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.335+0000 7f4958ff9700 1 -- 172.21.15.16:0/3871603097 <== mon.1 v2:172.21.15.49:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f494c08aa80 con 0x7f495c06cd40 2023-10-16T12:11:03.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.335+0000 7f495b7fe700 1 --2- 172.21.15.16:0/3871603097 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f493c06af00 0x7f493c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:03.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.335+0000 7f495b7fe700 1 --2- 172.21.15.16:0/3871603097 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f493c06af00 0x7f493c06d3c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f495c071450 tx=0x7f4944009210 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:03.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.341+0000 7f4958ff9700 1 -- 172.21.15.16:0/3871603097 <== mon.1 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f494c0558d0 con 0x7f495c06cd40 2023-10-16T12:11:03.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.588+0000 7f49622ec700 1 -- 172.21.15.16:0/3871603097 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f495c0b01c0 con 0x7f495c06cd40 2023-10-16T12:11:03.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.589+0000 7f4958ff9700 1 -- 172.21.15.16:0/3871603097 <== mon.1 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4909 (secure 0 0 0) 0x7f494c058ef0 con 0x7f495c06cd40 2023-10-16T12:11:03.591 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:11:03.591 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/2979266800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":2979266800},{"type":"v1","addr":"172.21.15.49:6834","nonce":2979266800}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4251,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/1398336801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1398336801},{"type":"v1","addr":"172.21.15.16:6836","nonce":1398336801}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4254,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3168304671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":3168304671},{"type":"v1","addr":"172.21.15.16:6837","nonce":3168304671}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:10:39.030243+0000","modified":"2023-10-16T12:11:02.593358+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4269":{"gid":4269,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.49:6835/2892536404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":2892536404},{"type":"v1","addr":"172.21.15.49:6835","nonce":2892536404}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T12:11:03.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.591+0000 7f49622ec700 1 -- 172.21.15.16:0/3871603097 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f493c06af00 msgr2=0x7f493c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:03.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.591+0000 7f49622ec700 1 --2- 172.21.15.16:0/3871603097 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f493c06af00 0x7f493c06d3c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f495c071450 tx=0x7f4944009210 comp rx=0 tx=0).stop 2023-10-16T12:11:03.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.591+0000 7f49622ec700 1 -- 172.21.15.16:0/3871603097 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f495c06cd40 msgr2=0x7f495c15b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:03.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.591+0000 7f49622ec700 1 --2- 172.21.15.16:0/3871603097 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f495c06cd40 0x7f495c15b6c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f494c017000 tx=0x7f494c0047c0 comp rx=0 tx=0).stop 2023-10-16T12:11:03.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.591+0000 7f49622ec700 1 -- 172.21.15.16:0/3871603097 shutdown_connections 2023-10-16T12:11:03.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.591+0000 7f49622ec700 1 --2- 172.21.15.16:0/3871603097 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f493c06af00 0x7f493c06d3c0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:03.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.591+0000 7f49622ec700 1 --2- 172.21.15.16:0/3871603097 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f495c159c10 0x7f495c148da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:03.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.591+0000 7f49622ec700 1 --2- 172.21.15.16:0/3871603097 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f495c06cd40 0x7f495c15b6c0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:03.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.591+0000 7f49622ec700 1 --2- 172.21.15.16:0/3871603097 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f495c0b3b80 0x7f495c1e5210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:03.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.591+0000 7f49622ec700 1 -- 172.21.15.16:0/3871603097 >> 172.21.15.16:0/3871603097 conn(0x7f495c0b4cd0 msgr2=0x7f495c06a7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:03.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.591+0000 7f49622ec700 1 -- 172.21.15.16:0/3871603097 shutdown_connections 2023-10-16T12:11:03.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.591+0000 7f49622ec700 1 -- 172.21.15.16:0/3871603097 wait complete. 2023-10-16T12:11:03.596 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 9 2023-10-16T12:11:03.605 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:11:03.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.730+0000 7f63caa2d700 1 Processor -- start 2023-10-16T12:11:03.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.730+0000 7f63caa2d700 1 -- start start 2023-10-16T12:11:03.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.731+0000 7f63caa2d700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f63c40cc8a0 0x7f63c415e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:03.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.731+0000 7f63caa2d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f63c406a7e0 0x7f63c406ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:03.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.731+0000 7f63caa2d700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63c406b150 con 0x7f63c415b280 2023-10-16T12:11:03.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.731+0000 7f63caa2d700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63c4161c30 con 0x7f63c406a7e0 2023-10-16T12:11:03.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.731+0000 7f63caa2d700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63c4161f10 con 0x7f63c40cc8a0 2023-10-16T12:11:03.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.731+0000 7f63c8fca700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f63c406a7e0 0x7f63c406ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:03.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.731+0000 7f63c3fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f63c415b280 0x7f63c41613e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50940/0 (socket says 172.21.15.16:50940) 2023-10-16T12:11:03.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.731+0000 7f63c3fff700 1 -- 172.21.15.16:0/258904487 learned_addr learned my addr 172.21.15.16:0/258904487 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:03.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.731+0000 7f63c37fe700 1 --2- 172.21.15.16:0/258904487 >> v2:172.21.15.49:3300/0 conn(0x7f63c40cc8a0 0x7f63c415e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:03.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.731+0000 7f63c8fca700 1 -- 172.21.15.16:0/258904487 >> v2:172.21.15.49:3300/0 conn(0x7f63c40cc8a0 msgr2=0x7f63c415e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:03.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.731+0000 7f63c8fca700 1 --2- 172.21.15.16:0/258904487 >> v2:172.21.15.49:3300/0 conn(0x7f63c40cc8a0 0x7f63c415e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:03.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.731+0000 7f63c8fca700 1 -- 172.21.15.16:0/258904487 >> v1:172.21.15.16:6789/0 conn(0x7f63c415b280 legacy=0x7f63c41613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:03.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.731+0000 7f63c8fca700 1 -- 172.21.15.16:0/258904487 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63c41621f0 con 0x7f63c406a7e0 2023-10-16T12:11:03.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.731+0000 7f63c2ffd700 1 -- 172.21.15.16:0/258904487 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 812364821 0 0) 0x7f63c406b150 con 0x7f63c415b280 2023-10-16T12:11:03.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.732+0000 7f63c8fca700 1 --2- 172.21.15.16:0/258904487 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f63c406a7e0 0x7f63c406ac10 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f63c4071970 tx=0x7f63bc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec09bfea3809fbe4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:03.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.732+0000 7f63c2ffd700 1 -- 172.21.15.16:0/258904487 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63bc00e070 con 0x7f63c406a7e0 2023-10-16T12:11:03.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.732+0000 7f63c2ffd700 1 -- 172.21.15.16:0/258904487 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63bc00bbd0 con 0x7f63c406a7e0 2023-10-16T12:11:03.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.732+0000 7f63c2ffd700 1 -- 172.21.15.16:0/258904487 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63bc012430 con 0x7f63c406a7e0 2023-10-16T12:11:03.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.732+0000 7f63caa2d700 1 -- 172.21.15.16:0/258904487 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f63c406a7e0 msgr2=0x7f63c406ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:03.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.732+0000 7f63caa2d700 1 --2- 172.21.15.16:0/258904487 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f63c406a7e0 0x7f63c406ac10 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f63c4071970 tx=0x7f63bc00b330 comp rx=0 tx=0).stop 2023-10-16T12:11:03.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.732+0000 7f63caa2d700 1 -- 172.21.15.16:0/258904487 shutdown_connections 2023-10-16T12:11:03.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.732+0000 7f63caa2d700 1 --2- 172.21.15.16:0/258904487 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f63c406a7e0 0x7f63c406ac10 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:03.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.732+0000 7f63caa2d700 1 --2- 172.21.15.16:0/258904487 >> v2:172.21.15.49:3300/0 conn(0x7f63c40cc8a0 0x7f63c415e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:03.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.732+0000 7f63caa2d700 1 -- 172.21.15.16:0/258904487 >> 172.21.15.16:0/258904487 conn(0x7f63c40b1fd0 msgr2=0x7f63c40b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:03.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.732+0000 7f63caa2d700 1 -- 172.21.15.16:0/258904487 shutdown_connections 2023-10-16T12:11:03.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.732+0000 7f63caa2d700 1 -- 172.21.15.16:0/258904487 wait complete. 2023-10-16T12:11:03.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.734+0000 7f63caa2d700 1 Processor -- start 2023-10-16T12:11:03.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.734+0000 7f63caa2d700 1 -- start start 2023-10-16T12:11:03.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.734+0000 7f63caa2d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f63c406a7e0 0x7f63c41562c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:03.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.734+0000 7f63caa2d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f63c40cc8a0 0x7f63c40c3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:03.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.734+0000 7f63caa2d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f63c415b280 0x7f63c40c44b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:03.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.735+0000 7f63caa2d700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c4157a70 con 0x7f63c415b280 2023-10-16T12:11:03.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.735+0000 7f63caa2d700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c415cfe0 con 0x7f63c406a7e0 2023-10-16T12:11:03.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.735+0000 7f63caa2d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63c415cd00 con 0x7f63c40cc8a0 2023-10-16T12:11:03.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.735+0000 7f63c8fca700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f63c415b280 0x7f63c40c44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:03.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.735+0000 7f63c8fca700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f63c415b280 0x7f63c40c44b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33444/0 (socket says 172.21.15.16:33444) 2023-10-16T12:11:03.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.735+0000 7f63c8fca700 1 -- 172.21.15.16:0/3782409278 learned_addr learned my addr 172.21.15.16:0/3782409278 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:03.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.735+0000 7f63c3fff700 1 --2- 172.21.15.16:0/3782409278 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f63c406a7e0 0x7f63c41562c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:03.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.735+0000 7f63c37fe700 1 --2- 172.21.15.16:0/3782409278 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f63c40cc8a0 0x7f63c40c3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:03.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.735+0000 7f63c8fca700 1 -- 172.21.15.16:0/3782409278 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f63c40cc8a0 msgr2=0x7f63c40c3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:03.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.735+0000 7f63c8fca700 1 --2- 172.21.15.16:0/3782409278 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f63c40cc8a0 0x7f63c40c3f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:03.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.735+0000 7f63c8fca700 1 -- 172.21.15.16:0/3782409278 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f63c406a7e0 msgr2=0x7f63c41562c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:03.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.735+0000 7f63c8fca700 1 --2- 172.21.15.16:0/3782409278 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f63c406a7e0 0x7f63c41562c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:03.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.735+0000 7f63c8fca700 1 -- 172.21.15.16:0/3782409278 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63bc009a70 con 0x7f63c415b280 2023-10-16T12:11:03.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.735+0000 7f63c37fe700 1 --2- 172.21.15.16:0/3782409278 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f63c40cc8a0 0x7f63c40c3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:11:03.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.735+0000 7f63c8fca700 1 --2- 172.21.15.16:0/3782409278 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f63c415b280 0x7f63c40c44b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f63c406b2e0 tx=0x7f63bc004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:03.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.735+0000 7f63c0ff9700 1 -- 172.21.15.16:0/3782409278 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63bc012430 con 0x7f63c415b280 2023-10-16T12:11:03.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.735+0000 7f63c0ff9700 1 -- 172.21.15.16:0/3782409278 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63bc020070 con 0x7f63c415b280 2023-10-16T12:11:03.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.735+0000 7f63caa2d700 1 -- 172.21.15.16:0/3782409278 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63c4157420 con 0x7f63c415b280 2023-10-16T12:11:03.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.735+0000 7f63c0ff9700 1 -- 172.21.15.16:0/3782409278 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63bc004280 con 0x7f63c415b280 2023-10-16T12:11:03.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.735+0000 7f63caa2d700 1 -- 172.21.15.16:0/3782409278 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63c4156800 con 0x7f63c415b280 2023-10-16T12:11:03.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.736+0000 7f63caa2d700 1 -- 172.21.15.16:0/3782409278 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63c4157c20 con 0x7f63c415b280 2023-10-16T12:11:03.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.737+0000 7f63c0ff9700 1 -- 172.21.15.16:0/3782409278 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f63bc0043e0 con 0x7f63c415b280 2023-10-16T12:11:03.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.737+0000 7f63c0ff9700 1 --2- 172.21.15.16:0/3782409278 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f63a406af00 0x7f63a406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:03.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.737+0000 7f63c3fff700 1 --2- 172.21.15.16:0/3782409278 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f63a406af00 0x7f63a406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:03.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.737+0000 7f63c0ff9700 1 -- 172.21.15.16:0/3782409278 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f63bc022070 con 0x7f63c415b280 2023-10-16T12:11:03.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.738+0000 7f63c3fff700 1 --2- 172.21.15.16:0/3782409278 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f63a406af00 0x7f63a406d3c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f63b4000f60 tx=0x7f63b4007c90 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:03.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.742+0000 7f63c0ff9700 1 -- 172.21.15.16:0/3782409278 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f63c4157c20 con 0x7f63c415b280 2023-10-16T12:11:03.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.947+0000 7f63caa2d700 1 -- 172.21.15.16:0/3782409278 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f63c4156990 con 0x7f63c415b280 2023-10-16T12:11:03.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.948+0000 7f63c0ff9700 1 -- 172.21.15.16:0/3782409278 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18238 (secure 0 0 0) 0x7f63bc0568e0 con 0x7f63c415b280 2023-10-16T12:11:03.951 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:11:03.951 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":25,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:10:44.080141+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:10:36.935773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T12:10:38.004870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-16T12:10:41.672025+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:11:03.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.950+0000 7f63caa2d700 1 -- 172.21.15.16:0/3782409278 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f63a406af00 msgr2=0x7f63a406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:03.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.950+0000 7f63caa2d700 1 --2- 172.21.15.16:0/3782409278 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f63a406af00 0x7f63a406d3c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f63b4000f60 tx=0x7f63b4007c90 comp rx=0 tx=0).stop 2023-10-16T12:11:03.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.950+0000 7f63caa2d700 1 -- 172.21.15.16:0/3782409278 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f63c415b280 msgr2=0x7f63c40c44b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:03.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.950+0000 7f63caa2d700 1 --2- 172.21.15.16:0/3782409278 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f63c415b280 0x7f63c40c44b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f63c406b2e0 tx=0x7f63bc004ad0 comp rx=0 tx=0).stop 2023-10-16T12:11:03.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.950+0000 7f63caa2d700 1 -- 172.21.15.16:0/3782409278 shutdown_connections 2023-10-16T12:11:03.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.950+0000 7f63caa2d700 1 --2- 172.21.15.16:0/3782409278 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f63a406af00 0x7f63a406d3c0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:03.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.950+0000 7f63caa2d700 1 --2- 172.21.15.16:0/3782409278 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f63c415b280 0x7f63c40c44b0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:03.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.950+0000 7f63caa2d700 1 --2- 172.21.15.16:0/3782409278 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f63c406a7e0 0x7f63c41562c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:03.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.950+0000 7f63caa2d700 1 --2- 172.21.15.16:0/3782409278 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f63c40cc8a0 0x7f63c40c3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:03.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.950+0000 7f63caa2d700 1 -- 172.21.15.16:0/3782409278 >> 172.21.15.16:0/3782409278 conn(0x7f63c40b1fd0 msgr2=0x7f63c415a4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:03.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.951+0000 7f63caa2d700 1 -- 172.21.15.16:0/3782409278 shutdown_connections 2023-10-16T12:11:03.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:03.951+0000 7f63caa2d700 1 -- 172.21.15.16:0/3782409278 wait complete. 2023-10-16T12:11:03.965 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-16T12:11:03.965 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T12:11:04.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.088+0000 7fdb19365700 1 Processor -- start 2023-10-16T12:11:04.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.088+0000 7fdb19365700 1 -- start start 2023-10-16T12:11:04.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.089+0000 7fdb19365700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fdb140ce4f0 0x7fdb140d0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:04.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.089+0000 7fdb19365700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb140b0970 con 0x7fdb1415b630 2023-10-16T12:11:04.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.089+0000 7fdb19365700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb140cd690 con 0x7fdb1415ba10 2023-10-16T12:11:04.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.089+0000 7fdb19365700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb140cd970 con 0x7fdb140ce4f0 2023-10-16T12:11:04.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.089+0000 7fdb12ffd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fdb1415b630 0x7fdb140cdde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50954/0 (socket says 172.21.15.16:50954) 2023-10-16T12:11:04.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.089+0000 7fdb12ffd700 1 -- 172.21.15.16:0/1081364397 learned_addr learned my addr 172.21.15.16:0/1081364397 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:04.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.089+0000 7fdb127fc700 1 --2- 172.21.15.16:0/1081364397 >> v2:172.21.15.49:3300/0 conn(0x7fdb140ce4f0 0x7fdb140d0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:04.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.089+0000 7fdb11ffb700 1 -- 172.21.15.16:0/1081364397 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1285176336 0 0) 0x7fdb140b0970 con 0x7fdb1415b630 2023-10-16T12:11:04.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.089+0000 7fdb11ffb700 1 -- 172.21.15.16:0/1081364397 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdafc0035b0 con 0x7fdb1415b630 2023-10-16T12:11:04.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.089+0000 7fdb11ffb700 1 -- 172.21.15.16:0/1081364397 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1927657030 0 0) 0x7fdafc0035b0 con 0x7fdb1415b630 2023-10-16T12:11:04.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.089+0000 7fdb11ffb700 1 -- 172.21.15.16:0/1081364397 >> v2:172.21.15.49:3300/0 conn(0x7fdb140ce4f0 msgr2=0x7fdb140d0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:04.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.089+0000 7fdb11ffb700 1 --2- 172.21.15.16:0/1081364397 >> v2:172.21.15.49:3300/0 conn(0x7fdb140ce4f0 0x7fdb140d0380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:04.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.089+0000 7fdb11ffb700 1 -- 172.21.15.16:0/1081364397 >> v1:172.21.15.49:6789/0 conn(0x7fdb1415ba10 legacy=0x7fdb140b1430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:04.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.089+0000 7fdb11ffb700 1 -- 172.21.15.16:0/1081364397 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb140b0c30 con 0x7fdb1415b630 2023-10-16T12:11:04.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.090+0000 7fdb11ffb700 1 -- 172.21.15.16:0/1081364397 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7fdb08001670 con 0x7fdb1415b630 2023-10-16T12:11:04.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.090+0000 7fdb11ffb700 1 -- 172.21.15.16:0/1081364397 >> v1:172.21.15.16:6789/0 conn(0x7fdb1415b630 legacy=0x7fdb140cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:04.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.090+0000 7fdb11ffb700 1 --2- 172.21.15.16:0/1081364397 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdafc003a00 0x7fdafc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:04.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.090+0000 7fdb11ffb700 1 --2- 172.21.15.16:0/1081364397 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdafc004850 0x7fdafc004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:04.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.090+0000 7fdb11ffb700 1 --2- 172.21.15.16:0/1081364397 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdafc005210 0x7fdafc00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:04.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.090+0000 7fdb11ffb700 1 -- 172.21.15.16:0/1081364397 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb140cd970 con 0x7fdafc004850 2023-10-16T12:11:04.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.090+0000 7fdb11ffb700 1 -- 172.21.15.16:0/1081364397 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdafc00c0b0 con 0x7fdafc005210 2023-10-16T12:11:04.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.090+0000 7fdb11ffb700 1 -- 172.21.15.16:0/1081364397 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdafc00c270 con 0x7fdafc003a00 2023-10-16T12:11:04.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.090+0000 7fdb127fc700 1 --2- 172.21.15.16:0/1081364397 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdafc004850 0x7fdafc004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:04.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.090+0000 7fdb12ffd700 1 --2- 172.21.15.16:0/1081364397 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdafc003a00 0x7fdafc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:04.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.090+0000 7fdb137fe700 1 --2- 172.21.15.16:0/1081364397 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdafc005210 0x7fdafc00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:04.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.090+0000 7fdb127fc700 1 -- 172.21.15.16:0/1081364397 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdafc003a00 msgr2=0x7fdafc003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:04.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.090+0000 7fdb127fc700 1 --2- 172.21.15.16:0/1081364397 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdafc003a00 0x7fdafc003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:04.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.090+0000 7fdb127fc700 1 -- 172.21.15.16:0/1081364397 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdafc005210 msgr2=0x7fdafc00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:04.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.090+0000 7fdb127fc700 1 --2- 172.21.15.16:0/1081364397 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdafc005210 0x7fdafc00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:04.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.090+0000 7fdb127fc700 1 -- 172.21.15.16:0/1081364397 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdafc0035b0 con 0x7fdafc004850 2023-10-16T12:11:04.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.090+0000 7fdb137fe700 1 --2- 172.21.15.16:0/1081364397 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdafc005210 0x7fdafc00ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:11:04.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.090+0000 7fdb137fe700 1 -- 172.21.15.16:0/1081364397 reap_dead start 2023-10-16T12:11:04.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.090+0000 7fdb127fc700 1 --2- 172.21.15.16:0/1081364397 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdafc004850 0x7fdafc004cd0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fdb04009b60 tx=0x7fdafc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=443b00c9ea76a354 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:04.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.090+0000 7fdb11ffb700 1 -- 172.21.15.16:0/1081364397 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb0400e6c0 con 0x7fdafc004850 2023-10-16T12:11:04.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.090+0000 7fdb11ffb700 1 -- 172.21.15.16:0/1081364397 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb0400e820 con 0x7fdafc004850 2023-10-16T12:11:04.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.090+0000 7fdb11ffb700 1 -- 172.21.15.16:0/1081364397 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb0400eb90 con 0x7fdafc004850 2023-10-16T12:11:04.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.091+0000 7fdb19365700 1 -- 172.21.15.16:0/1081364397 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdafc004850 msgr2=0x7fdafc004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:04.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.091+0000 7fdb19365700 1 --2- 172.21.15.16:0/1081364397 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdafc004850 0x7fdafc004cd0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fdb04009b60 tx=0x7fdafc005880 comp rx=0 tx=0).stop 2023-10-16T12:11:04.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.091+0000 7fdb19365700 1 -- 172.21.15.16:0/1081364397 shutdown_connections 2023-10-16T12:11:04.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.091+0000 7fdb19365700 1 --2- 172.21.15.16:0/1081364397 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdafc004850 0x7fdafc004cd0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:04.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.091+0000 7fdb19365700 1 -- 172.21.15.16:0/1081364397 >> 172.21.15.16:0/1081364397 conn(0x7fdb140d14b0 msgr2=0x7fdb140d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:04.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.091+0000 7fdb19365700 1 -- 172.21.15.16:0/1081364397 shutdown_connections 2023-10-16T12:11:04.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.091+0000 7fdb19365700 1 -- 172.21.15.16:0/1081364397 wait complete. 2023-10-16T12:11:04.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.092+0000 7fdb19365700 1 Processor -- start 2023-10-16T12:11:04.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.092+0000 7fdb19365700 1 -- start start 2023-10-16T12:11:04.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.093+0000 7fdb19365700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdafc004850 0x7fdb1415c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:04.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.093+0000 7fdb19365700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdb1415e4e0 0x7fdb1415e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:04.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.093+0000 7fdb19365700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdb140ca020 0x7fdb140ca4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:04.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.093+0000 7fdb19365700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb140ca9e0 con 0x7fdafc004850 2023-10-16T12:11:04.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.093+0000 7fdb19365700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb1406c650 con 0x7fdb140ca020 2023-10-16T12:11:04.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.093+0000 7fdb19365700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb1406c930 con 0x7fdb1415e4e0 2023-10-16T12:11:04.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.093+0000 7fdb12ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdafc004850 0x7fdb1415c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:04.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.093+0000 7fdb12ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdafc004850 0x7fdb1415c140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33460/0 (socket says 172.21.15.16:33460) 2023-10-16T12:11:04.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.093+0000 7fdb12ffd700 1 -- 172.21.15.16:0/1680521138 learned_addr learned my addr 172.21.15.16:0/1680521138 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:04.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.093+0000 7fdb127fc700 1 --2- 172.21.15.16:0/1680521138 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdb1415e4e0 0x7fdb1415e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:04.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.093+0000 7fdb137fe700 1 --2- 172.21.15.16:0/1680521138 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdb140ca020 0x7fdb140ca4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:04.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.093+0000 7fdb12ffd700 1 -- 172.21.15.16:0/1680521138 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdb1415e4e0 msgr2=0x7fdb1415e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:04.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.093+0000 7fdb12ffd700 1 --2- 172.21.15.16:0/1680521138 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdb1415e4e0 0x7fdb1415e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:04.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.093+0000 7fdb12ffd700 1 -- 172.21.15.16:0/1680521138 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdb140ca020 msgr2=0x7fdb140ca4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:04.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.093+0000 7fdb12ffd700 1 --2- 172.21.15.16:0/1680521138 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdb140ca020 0x7fdb140ca4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:04.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.093+0000 7fdb12ffd700 1 -- 172.21.15.16:0/1680521138 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb1406cc10 con 0x7fdafc004850 2023-10-16T12:11:04.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.093+0000 7fdb137fe700 1 --2- 172.21.15.16:0/1680521138 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdb140ca020 0x7fdb140ca4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:11:04.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.094+0000 7fdb12ffd700 1 --2- 172.21.15.16:0/1680521138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdafc004850 0x7fdb1415c140 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fdb080072c0 tx=0x7fdb080075d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:04.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.094+0000 7fdafb7fe700 1 -- 172.21.15.16:0/1680521138 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb080077b0 con 0x7fdafc004850 2023-10-16T12:11:04.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.094+0000 7fdafb7fe700 1 -- 172.21.15.16:0/1680521138 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb0800f8d0 con 0x7fdafc004850 2023-10-16T12:11:04.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.094+0000 7fdb19365700 1 -- 172.21.15.16:0/1680521138 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb1406cef0 con 0x7fdafc004850 2023-10-16T12:11:04.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.094+0000 7fdafb7fe700 1 -- 172.21.15.16:0/1680521138 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb0800fc80 con 0x7fdafc004850 2023-10-16T12:11:04.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.094+0000 7fdb19365700 1 -- 172.21.15.16:0/1680521138 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb1406c300 con 0x7fdafc004850 2023-10-16T12:11:04.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.095+0000 7fdb19365700 1 -- 172.21.15.16:0/1680521138 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb1404ebb0 con 0x7fdafc004850 2023-10-16T12:11:04.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.095+0000 7fdafb7fe700 1 -- 172.21.15.16:0/1680521138 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fdb0801d460 con 0x7fdafc004850 2023-10-16T12:11:04.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.096+0000 7fdafb7fe700 1 --2- 172.21.15.16:0/1680521138 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fdb0006af00 0x7fdb0006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:04.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.096+0000 7fdafb7fe700 1 -- 172.21.15.16:0/1680521138 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fdb080899e0 con 0x7fdafc004850 2023-10-16T12:11:04.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.096+0000 7fdb127fc700 1 --2- 172.21.15.16:0/1680521138 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fdb0006af00 0x7fdb0006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:04.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.096+0000 7fdb127fc700 1 --2- 172.21.15.16:0/1680521138 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fdb0006af00 0x7fdb0006d3c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fdb040090b0 tx=0x7fdb0401e040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:04.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.100+0000 7fdafb7fe700 1 -- 172.21.15.16:0/1680521138 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdb08054700 con 0x7fdafc004850 2023-10-16T12:11:04.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.300+0000 7fdb19365700 1 -- 172.21.15.16:0/1680521138 --> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fdb140b0f10 con 0x7fdb0006af00 2023-10-16T12:11:04.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.310+0000 7fdafb7fe700 1 -- 172.21.15.16:0/1680521138 <== mgr.4104 v2:172.21.15.16:6832/365921436 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+345260 (secure 0 0 0) 0x7fdb0400ea90 con 0x7fdb0006af00 2023-10-16T12:11:04.316 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:11:04.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.314+0000 7fdb19365700 1 -- 172.21.15.16:0/1680521138 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fdb0006af00 msgr2=0x7fdb0006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:04.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.314+0000 7fdb19365700 1 --2- 172.21.15.16:0/1680521138 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fdb0006af00 0x7fdb0006d3c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fdb040090b0 tx=0x7fdb0401e040 comp rx=0 tx=0).stop 2023-10-16T12:11:04.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.314+0000 7fdb19365700 1 -- 172.21.15.16:0/1680521138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdafc004850 msgr2=0x7fdb1415c140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:04.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.314+0000 7fdb19365700 1 --2- 172.21.15.16:0/1680521138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdafc004850 0x7fdb1415c140 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fdb080072c0 tx=0x7fdb080075d0 comp rx=0 tx=0).stop 2023-10-16T12:11:04.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.314+0000 7fdb19365700 1 -- 172.21.15.16:0/1680521138 shutdown_connections 2023-10-16T12:11:04.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.314+0000 7fdb19365700 1 --2- 172.21.15.16:0/1680521138 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fdb0006af00 0x7fdb0006d3c0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:04.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.314+0000 7fdb19365700 1 --2- 172.21.15.16:0/1680521138 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdb140ca020 0x7fdb140ca4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:04.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.314+0000 7fdb19365700 1 --2- 172.21.15.16:0/1680521138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdafc004850 0x7fdb1415c140 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:04.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.314+0000 7fdb19365700 1 --2- 172.21.15.16:0/1680521138 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdb1415e4e0 0x7fdb1415e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:04.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.314+0000 7fdb19365700 1 -- 172.21.15.16:0/1680521138 >> 172.21.15.16:0/1680521138 conn(0x7fdb140d14b0 msgr2=0x7fdb140b1330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:04.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.314+0000 7fdb19365700 1 -- 172.21.15.16:0/1680521138 shutdown_connections 2023-10-16T12:11:04.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.314+0000 7fdb19365700 1 -- 172.21.15.16:0/1680521138 wait complete. 2023-10-16T12:11:04.322 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-10-16T12:11:04.328 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2023-10-16T12:11:03.100385+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219472,"kb_used_data":3304,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755248,"statfs":{"total":773094113280,"available":772869373952,"internally_reserved":0,"allocated":3383296,"data_stored":2025624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12740,"internal_metadata":221236796},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2230,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002987"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.797013+0000","last_change":"2023-10-16T12:10:39.641670+0000","last_active":"2023-10-16T12:10:43.797013+0000","last_peered":"2023-10-16T12:10:43.797013+0000","last_clean":"2023-10-16T12:10:43.797013+0000","last_became_active":"2023-10-16T12:10:39.639911+0000","last_became_peered":"2023-10-16T12:10:39.639911+0000","last_unstale":"2023-10-16T12:10:43.797013+0000","last_undegraded":"2023-10-16T12:10:43.797013+0000","last_fullsized":"2023-10-16T12:10:43.797013+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:29:43.494490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.656566+0000","last_change":"2023-10-16T12:10:39.637937+0000","last_active":"2023-10-16T12:10:43.656566+0000","last_peered":"2023-10-16T12:10:43.656566+0000","last_clean":"2023-10-16T12:10:43.656566+0000","last_became_active":"2023-10-16T12:10:39.637595+0000","last_became_peered":"2023-10-16T12:10:39.637595+0000","last_unstale":"2023-10-16T12:10:43.656566+0000","last_undegraded":"2023-10-16T12:10:43.656566+0000","last_fullsized":"2023-10-16T12:10:43.656566+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:03:12.325906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646168+0000","last_change":"2023-10-16T12:10:39.630003+0000","last_active":"2023-10-16T12:10:43.646168+0000","last_peered":"2023-10-16T12:10:43.646168+0000","last_clean":"2023-10-16T12:10:43.646168+0000","last_became_active":"2023-10-16T12:10:39.627471+0000","last_became_peered":"2023-10-16T12:10:39.627471+0000","last_unstale":"2023-10-16T12:10:43.646168+0000","last_undegraded":"2023-10-16T12:10:43.646168+0000","last_fullsized":"2023-10-16T12:10:43.646168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:33:20.807621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646890+0000","last_change":"2023-10-16T12:10:39.645834+0000","last_active":"2023-10-16T12:10:43.646890+0000","last_peered":"2023-10-16T12:10:43.646890+0000","last_clean":"2023-10-16T12:10:43.646890+0000","last_became_active":"2023-10-16T12:10:39.645631+0000","last_became_peered":"2023-10-16T12:10:39.645631+0000","last_unstale":"2023-10-16T12:10:43.646890+0000","last_undegraded":"2023-10-16T12:10:43.646890+0000","last_fullsized":"2023-10-16T12:10:43.646890+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:14:00.725551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.794209+0000","last_change":"2023-10-16T12:10:39.642967+0000","last_active":"2023-10-16T12:10:43.794209+0000","last_peered":"2023-10-16T12:10:43.794209+0000","last_clean":"2023-10-16T12:10:43.794209+0000","last_became_active":"2023-10-16T12:10:39.642015+0000","last_became_peered":"2023-10-16T12:10:39.642015+0000","last_unstale":"2023-10-16T12:10:43.794209+0000","last_undegraded":"2023-10-16T12:10:43.794209+0000","last_fullsized":"2023-10-16T12:10:43.794209+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:10:19.840696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646377+0000","last_change":"2023-10-16T12:10:39.648342+0000","last_active":"2023-10-16T12:10:43.646377+0000","last_peered":"2023-10-16T12:10:43.646377+0000","last_clean":"2023-10-16T12:10:43.646377+0000","last_became_active":"2023-10-16T12:10:39.648046+0000","last_became_peered":"2023-10-16T12:10:39.648046+0000","last_unstale":"2023-10-16T12:10:43.646377+0000","last_undegraded":"2023-10-16T12:10:43.646377+0000","last_fullsized":"2023-10-16T12:10:43.646377+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:27:44.016376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.793686+0000","last_change":"2023-10-16T12:10:39.640757+0000","last_active":"2023-10-16T12:10:43.793686+0000","last_peered":"2023-10-16T12:10:43.793686+0000","last_clean":"2023-10-16T12:10:43.793686+0000","last_became_active":"2023-10-16T12:10:39.639250+0000","last_became_peered":"2023-10-16T12:10:39.639250+0000","last_unstale":"2023-10-16T12:10:43.793686+0000","last_undegraded":"2023-10-16T12:10:43.793686+0000","last_fullsized":"2023-10-16T12:10:43.793686+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:59:08.633021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.647118+0000","last_change":"2023-10-16T12:10:39.643535+0000","last_active":"2023-10-16T12:10:43.647118+0000","last_peered":"2023-10-16T12:10:43.647118+0000","last_clean":"2023-10-16T12:10:43.647118+0000","last_became_active":"2023-10-16T12:10:39.643242+0000","last_became_peered":"2023-10-16T12:10:39.643242+0000","last_unstale":"2023-10-16T12:10:43.647118+0000","last_undegraded":"2023-10-16T12:10:43.647118+0000","last_fullsized":"2023-10-16T12:10:43.647118+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:49:21.678699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.796912+0000","last_change":"2023-10-16T12:10:39.645759+0000","last_active":"2023-10-16T12:10:43.796912+0000","last_peered":"2023-10-16T12:10:43.796912+0000","last_clean":"2023-10-16T12:10:43.796912+0000","last_became_active":"2023-10-16T12:10:39.642863+0000","last_became_peered":"2023-10-16T12:10:39.642863+0000","last_unstale":"2023-10-16T12:10:43.796912+0000","last_undegraded":"2023-10-16T12:10:43.796912+0000","last_fullsized":"2023-10-16T12:10:43.796912+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:40:23.467426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.650642+0000","last_change":"2023-10-16T12:10:39.641392+0000","last_active":"2023-10-16T12:10:43.650642+0000","last_peered":"2023-10-16T12:10:43.650642+0000","last_clean":"2023-10-16T12:10:43.650642+0000","last_became_active":"2023-10-16T12:10:39.641111+0000","last_became_peered":"2023-10-16T12:10:39.641111+0000","last_unstale":"2023-10-16T12:10:43.650642+0000","last_undegraded":"2023-10-16T12:10:43.650642+0000","last_fullsized":"2023-10-16T12:10:43.650642+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:38:01.595985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.650275+0000","last_change":"2023-10-16T12:10:39.642376+0000","last_active":"2023-10-16T12:10:43.650275+0000","last_peered":"2023-10-16T12:10:43.650275+0000","last_clean":"2023-10-16T12:10:43.650275+0000","last_became_active":"2023-10-16T12:10:39.642086+0000","last_became_peered":"2023-10-16T12:10:39.642086+0000","last_unstale":"2023-10-16T12:10:43.650275+0000","last_undegraded":"2023-10-16T12:10:43.650275+0000","last_fullsized":"2023-10-16T12:10:43.650275+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:05:05.699350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.648829+0000","last_change":"2023-10-16T12:10:39.637561+0000","last_active":"2023-10-16T12:10:43.648829+0000","last_peered":"2023-10-16T12:10:43.648829+0000","last_clean":"2023-10-16T12:10:43.648829+0000","last_became_active":"2023-10-16T12:10:39.637109+0000","last_became_peered":"2023-10-16T12:10:39.637109+0000","last_unstale":"2023-10-16T12:10:43.648829+0000","last_undegraded":"2023-10-16T12:10:43.648829+0000","last_fullsized":"2023-10-16T12:10:43.648829+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:59:48.851909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.645820+0000","last_change":"2023-10-16T12:10:39.631791+0000","last_active":"2023-10-16T12:10:43.645820+0000","last_peered":"2023-10-16T12:10:43.645820+0000","last_clean":"2023-10-16T12:10:43.645820+0000","last_became_active":"2023-10-16T12:10:39.631245+0000","last_became_peered":"2023-10-16T12:10:39.631245+0000","last_unstale":"2023-10-16T12:10:43.645820+0000","last_undegraded":"2023-10-16T12:10:43.645820+0000","last_fullsized":"2023-10-16T12:10:43.645820+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:05:05.699350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646175+0000","last_change":"2023-10-16T12:10:39.631112+0000","last_active":"2023-10-16T12:10:43.646175+0000","last_peered":"2023-10-16T12:10:43.646175+0000","last_clean":"2023-10-16T12:10:43.646175+0000","last_became_active":"2023-10-16T12:10:39.630312+0000","last_became_peered":"2023-10-16T12:10:39.630312+0000","last_unstale":"2023-10-16T12:10:43.646175+0000","last_undegraded":"2023-10-16T12:10:43.646175+0000","last_fullsized":"2023-10-16T12:10:43.646175+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:04:48.768318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.653241+0000","last_change":"2023-10-16T12:10:39.642725+0000","last_active":"2023-10-16T12:10:43.653241+0000","last_peered":"2023-10-16T12:10:43.653241+0000","last_clean":"2023-10-16T12:10:43.653241+0000","last_became_active":"2023-10-16T12:10:39.642437+0000","last_became_peered":"2023-10-16T12:10:39.642437+0000","last_unstale":"2023-10-16T12:10:43.653241+0000","last_undegraded":"2023-10-16T12:10:43.653241+0000","last_fullsized":"2023-10-16T12:10:43.653241+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:46:02.593591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.652749+0000","last_change":"2023-10-16T12:10:39.641780+0000","last_active":"2023-10-16T12:10:43.652749+0000","last_peered":"2023-10-16T12:10:43.652749+0000","last_clean":"2023-10-16T12:10:43.652749+0000","last_became_active":"2023-10-16T12:10:39.641275+0000","last_became_peered":"2023-10-16T12:10:39.641275+0000","last_unstale":"2023-10-16T12:10:43.652749+0000","last_undegraded":"2023-10-16T12:10:43.652749+0000","last_fullsized":"2023-10-16T12:10:43.652749+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:38:01.595985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.647146+0000","last_change":"2023-10-16T12:10:39.641557+0000","last_active":"2023-10-16T12:10:43.647146+0000","last_peered":"2023-10-16T12:10:43.647146+0000","last_clean":"2023-10-16T12:10:43.647146+0000","last_became_active":"2023-10-16T12:10:39.641242+0000","last_became_peered":"2023-10-16T12:10:39.641242+0000","last_unstale":"2023-10-16T12:10:43.647146+0000","last_undegraded":"2023-10-16T12:10:43.647146+0000","last_fullsized":"2023-10-16T12:10:43.647146+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:10:19.840696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.642631+0000","last_change":"2023-10-16T12:10:39.638264+0000","last_active":"2023-10-16T12:10:43.642631+0000","last_peered":"2023-10-16T12:10:43.642631+0000","last_clean":"2023-10-16T12:10:43.642631+0000","last_became_active":"2023-10-16T12:10:39.637959+0000","last_became_peered":"2023-10-16T12:10:39.637959+0000","last_unstale":"2023-10-16T12:10:43.642631+0000","last_undegraded":"2023-10-16T12:10:43.642631+0000","last_fullsized":"2023-10-16T12:10:43.642631+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:27:06.186345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.796665+0000","last_change":"2023-10-16T12:10:39.646694+0000","last_active":"2023-10-16T12:10:43.796665+0000","last_peered":"2023-10-16T12:10:43.796665+0000","last_clean":"2023-10-16T12:10:43.796665+0000","last_became_active":"2023-10-16T12:10:39.643245+0000","last_became_peered":"2023-10-16T12:10:39.643245+0000","last_unstale":"2023-10-16T12:10:43.796665+0000","last_undegraded":"2023-10-16T12:10:43.796665+0000","last_fullsized":"2023-10-16T12:10:43.796665+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:13:03.744288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:44.684038+0000","last_change":"2023-10-16T12:10:43.800090+0000","last_active":"2023-10-16T12:10:44.684038+0000","last_peered":"2023-10-16T12:10:44.684038+0000","last_clean":"2023-10-16T12:10:44.684038+0000","last_became_active":"2023-10-16T12:10:43.799746+0000","last_became_peered":"2023-10-16T12:10:43.799746+0000","last_unstale":"2023-10-16T12:10:44.684038+0000","last_undegraded":"2023-10-16T12:10:44.684038+0000","last_fullsized":"2023-10-16T12:10:44.684038+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:46:18.137970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646491+0000","last_change":"2023-10-16T12:10:38.620314+0000","last_active":"2023-10-16T12:10:43.646491+0000","last_peered":"2023-10-16T12:10:43.646491+0000","last_clean":"2023-10-16T12:10:43.646491+0000","last_became_active":"2023-10-16T12:10:38.620036+0000","last_became_peered":"2023-10-16T12:10:38.620036+0000","last_unstale":"2023-10-16T12:10:43.646491+0000","last_undegraded":"2023-10-16T12:10:43.646491+0000","last_fullsized":"2023-10-16T12:10:43.646491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:27:30.532335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.645988+0000","last_change":"2023-10-16T12:10:39.638746+0000","last_active":"2023-10-16T12:10:43.645988+0000","last_peered":"2023-10-16T12:10:43.645988+0000","last_clean":"2023-10-16T12:10:43.645988+0000","last_became_active":"2023-10-16T12:10:39.637152+0000","last_became_peered":"2023-10-16T12:10:39.637152+0000","last_unstale":"2023-10-16T12:10:43.645988+0000","last_undegraded":"2023-10-16T12:10:43.645988+0000","last_fullsized":"2023-10-16T12:10:43.645988+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:30:10.006613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:45.375335+0000","last_change":"2023-10-16T12:10:43.662213+0000","last_active":"2023-10-16T12:10:45.375335+0000","last_peered":"2023-10-16T12:10:45.375335+0000","last_clean":"2023-10-16T12:10:45.375335+0000","last_became_active":"2023-10-16T12:10:43.661755+0000","last_became_peered":"2023-10-16T12:10:43.661755+0000","last_unstale":"2023-10-16T12:10:45.375335+0000","last_undegraded":"2023-10-16T12:10:45.375335+0000","last_fullsized":"2023-10-16T12:10:45.375335+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:30:14.045561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646110+0000","last_change":"2023-10-16T12:10:38.625571+0000","last_active":"2023-10-16T12:10:43.646110+0000","last_peered":"2023-10-16T12:10:43.646110+0000","last_clean":"2023-10-16T12:10:43.646110+0000","last_became_active":"2023-10-16T12:10:38.625353+0000","last_became_peered":"2023-10-16T12:10:38.625353+0000","last_unstale":"2023-10-16T12:10:43.646110+0000","last_undegraded":"2023-10-16T12:10:43.646110+0000","last_fullsized":"2023-10-16T12:10:43.646110+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:39:25.396697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646379+0000","last_change":"2023-10-16T12:10:39.641482+0000","last_active":"2023-10-16T12:10:43.646379+0000","last_peered":"2023-10-16T12:10:43.646379+0000","last_clean":"2023-10-16T12:10:43.646379+0000","last_became_active":"2023-10-16T12:10:39.641168+0000","last_became_peered":"2023-10-16T12:10:39.641168+0000","last_unstale":"2023-10-16T12:10:43.646379+0000","last_undegraded":"2023-10-16T12:10:43.646379+0000","last_fullsized":"2023-10-16T12:10:43.646379+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:47:52.332290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:45.376007+0000","last_change":"2023-10-16T12:10:43.668484+0000","last_active":"2023-10-16T12:10:45.376007+0000","last_peered":"2023-10-16T12:10:45.376007+0000","last_clean":"2023-10-16T12:10:45.376007+0000","last_became_active":"2023-10-16T12:10:43.667862+0000","last_became_peered":"2023-10-16T12:10:43.667862+0000","last_unstale":"2023-10-16T12:10:45.376007+0000","last_undegraded":"2023-10-16T12:10:45.376007+0000","last_fullsized":"2023-10-16T12:10:45.376007+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:34:54.098418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.796065+0000","last_change":"2023-10-16T12:10:38.623974+0000","last_active":"2023-10-16T12:10:43.796065+0000","last_peered":"2023-10-16T12:10:43.796065+0000","last_clean":"2023-10-16T12:10:43.796065+0000","last_became_active":"2023-10-16T12:10:38.623532+0000","last_became_peered":"2023-10-16T12:10:38.623532+0000","last_unstale":"2023-10-16T12:10:43.796065+0000","last_undegraded":"2023-10-16T12:10:43.796065+0000","last_fullsized":"2023-10-16T12:10:43.796065+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:03:11.321777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646640+0000","last_change":"2023-10-16T12:10:39.651203+0000","last_active":"2023-10-16T12:10:43.646640+0000","last_peered":"2023-10-16T12:10:43.646640+0000","last_clean":"2023-10-16T12:10:43.646640+0000","last_became_active":"2023-10-16T12:10:39.651001+0000","last_became_peered":"2023-10-16T12:10:39.651001+0000","last_unstale":"2023-10-16T12:10:43.646640+0000","last_undegraded":"2023-10-16T12:10:43.646640+0000","last_fullsized":"2023-10-16T12:10:43.646640+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:34:50.059469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:45.375823+0000","last_change":"2023-10-16T12:10:43.659820+0000","last_active":"2023-10-16T12:10:45.375823+0000","last_peered":"2023-10-16T12:10:45.375823+0000","last_clean":"2023-10-16T12:10:45.375823+0000","last_became_active":"2023-10-16T12:10:43.659301+0000","last_became_peered":"2023-10-16T12:10:43.659301+0000","last_unstale":"2023-10-16T12:10:45.375823+0000","last_undegraded":"2023-10-16T12:10:45.375823+0000","last_fullsized":"2023-10-16T12:10:45.375823+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:59:12.671969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.795902+0000","last_change":"2023-10-16T12:10:38.622949+0000","last_active":"2023-10-16T12:10:43.795902+0000","last_peered":"2023-10-16T12:10:43.795902+0000","last_clean":"2023-10-16T12:10:43.795902+0000","last_became_active":"2023-10-16T12:10:38.622666+0000","last_became_peered":"2023-10-16T12:10:38.622666+0000","last_unstale":"2023-10-16T12:10:43.795902+0000","last_undegraded":"2023-10-16T12:10:43.795902+0000","last_fullsized":"2023-10-16T12:10:43.795902+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:27:30.532335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.653416+0000","last_change":"2023-10-16T12:10:39.650483+0000","last_active":"2023-10-16T12:10:43.653416+0000","last_peered":"2023-10-16T12:10:43.653416+0000","last_clean":"2023-10-16T12:10:43.653416+0000","last_became_active":"2023-10-16T12:10:39.650230+0000","last_became_peered":"2023-10-16T12:10:39.650230+0000","last_unstale":"2023-10-16T12:10:43.653416+0000","last_undegraded":"2023-10-16T12:10:43.653416+0000","last_fullsized":"2023-10-16T12:10:43.653416+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:38:53.624950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:44.684035+0000","last_change":"2023-10-16T12:10:43.667121+0000","last_active":"2023-10-16T12:10:44.684035+0000","last_peered":"2023-10-16T12:10:44.684035+0000","last_clean":"2023-10-16T12:10:44.684035+0000","last_became_active":"2023-10-16T12:10:43.666708+0000","last_became_peered":"2023-10-16T12:10:43.666708+0000","last_unstale":"2023-10-16T12:10:44.684035+0000","last_undegraded":"2023-10-16T12:10:44.684035+0000","last_fullsized":"2023-10-16T12:10:44.684035+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:19:32.391115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.645944+0000","last_change":"2023-10-16T12:10:38.623302+0000","last_active":"2023-10-16T12:10:43.645944+0000","last_peered":"2023-10-16T12:10:43.645944+0000","last_clean":"2023-10-16T12:10:43.645944+0000","last_became_active":"2023-10-16T12:10:38.621867+0000","last_became_peered":"2023-10-16T12:10:38.621867+0000","last_unstale":"2023-10-16T12:10:43.645944+0000","last_undegraded":"2023-10-16T12:10:43.645944+0000","last_fullsized":"2023-10-16T12:10:43.645944+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:59:30.615293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.649101+0000","last_change":"2023-10-16T12:10:39.639254+0000","last_active":"2023-10-16T12:10:43.649101+0000","last_peered":"2023-10-16T12:10:43.649101+0000","last_clean":"2023-10-16T12:10:43.649101+0000","last_became_active":"2023-10-16T12:10:39.638956+0000","last_became_peered":"2023-10-16T12:10:39.638956+0000","last_unstale":"2023-10-16T12:10:43.649101+0000","last_undegraded":"2023-10-16T12:10:43.649101+0000","last_fullsized":"2023-10-16T12:10:43.649101+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:27:31.536464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:45.374724+0000","last_change":"2023-10-16T12:10:43.667499+0000","last_active":"2023-10-16T12:10:45.374724+0000","last_peered":"2023-10-16T12:10:45.374724+0000","last_clean":"2023-10-16T12:10:45.374724+0000","last_became_active":"2023-10-16T12:10:43.667116+0000","last_became_peered":"2023-10-16T12:10:43.667116+0000","last_unstale":"2023-10-16T12:10:45.374724+0000","last_undegraded":"2023-10-16T12:10:45.374724+0000","last_fullsized":"2023-10-16T12:10:45.374724+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:59:35.658370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.651387+0000","last_change":"2023-10-16T12:10:38.619514+0000","last_active":"2023-10-16T12:10:43.651387+0000","last_peered":"2023-10-16T12:10:43.651387+0000","last_clean":"2023-10-16T12:10:43.651387+0000","last_became_active":"2023-10-16T12:10:38.619191+0000","last_became_peered":"2023-10-16T12:10:38.619191+0000","last_unstale":"2023-10-16T12:10:43.651387+0000","last_undegraded":"2023-10-16T12:10:43.651387+0000","last_fullsized":"2023-10-16T12:10:43.651387+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:39:25.396697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.649794+0000","last_change":"2023-10-16T12:10:39.642796+0000","last_active":"2023-10-16T12:10:43.649794+0000","last_peered":"2023-10-16T12:10:43.649794+0000","last_clean":"2023-10-16T12:10:43.649794+0000","last_became_active":"2023-10-16T12:10:39.642536+0000","last_became_peered":"2023-10-16T12:10:39.642536+0000","last_unstale":"2023-10-16T12:10:43.649794+0000","last_undegraded":"2023-10-16T12:10:43.649794+0000","last_fullsized":"2023-10-16T12:10:43.649794+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:27:44.016376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:44.874851+0000","last_change":"2023-10-16T12:10:43.662296+0000","last_active":"2023-10-16T12:10:44.874851+0000","last_peered":"2023-10-16T12:10:44.874851+0000","last_clean":"2023-10-16T12:10:44.874851+0000","last_became_active":"2023-10-16T12:10:43.661893+0000","last_became_peered":"2023-10-16T12:10:43.661893+0000","last_unstale":"2023-10-16T12:10:44.874851+0000","last_undegraded":"2023-10-16T12:10:44.874851+0000","last_fullsized":"2023-10-16T12:10:44.874851+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:14:04.764499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646516+0000","last_change":"2023-10-16T12:10:38.623322+0000","last_active":"2023-10-16T12:10:43.646516+0000","last_peered":"2023-10-16T12:10:43.646516+0000","last_clean":"2023-10-16T12:10:43.646516+0000","last_became_active":"2023-10-16T12:10:38.622826+0000","last_became_peered":"2023-10-16T12:10:38.622826+0000","last_unstale":"2023-10-16T12:10:43.646516+0000","last_undegraded":"2023-10-16T12:10:43.646516+0000","last_fullsized":"2023-10-16T12:10:43.646516+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:30:56.479201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646312+0000","last_change":"2023-10-16T12:10:39.624731+0000","last_active":"2023-10-16T12:10:43.646312+0000","last_peered":"2023-10-16T12:10:43.646312+0000","last_clean":"2023-10-16T12:10:43.646312+0000","last_became_active":"2023-10-16T12:10:39.624399+0000","last_became_peered":"2023-10-16T12:10:39.624399+0000","last_unstale":"2023-10-16T12:10:43.646312+0000","last_undegraded":"2023-10-16T12:10:43.646312+0000","last_fullsized":"2023-10-16T12:10:43.646312+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:47:52.332290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:45.375461+0000","last_change":"2023-10-16T12:10:43.798638+0000","last_active":"2023-10-16T12:10:45.375461+0000","last_peered":"2023-10-16T12:10:45.375461+0000","last_clean":"2023-10-16T12:10:45.375461+0000","last_became_active":"2023-10-16T12:10:43.797973+0000","last_became_peered":"2023-10-16T12:10:43.797973+0000","last_unstale":"2023-10-16T12:10:45.375461+0000","last_undegraded":"2023-10-16T12:10:45.375461+0000","last_fullsized":"2023-10-16T12:10:45.375461+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:29:47.533439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.652773+0000","last_change":"2023-10-16T12:10:38.627818+0000","last_active":"2023-10-16T12:10:43.652773+0000","last_peered":"2023-10-16T12:10:43.652773+0000","last_clean":"2023-10-16T12:10:43.652773+0000","last_became_active":"2023-10-16T12:10:38.627492+0000","last_became_peered":"2023-10-16T12:10:38.627492+0000","last_unstale":"2023-10-16T12:10:43.652773+0000","last_undegraded":"2023-10-16T12:10:43.652773+0000","last_fullsized":"2023-10-16T12:10:43.652773+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:05:04.695222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.653456+0000","last_change":"2023-10-16T12:10:39.633123+0000","last_active":"2023-10-16T12:10:43.653456+0000","last_peered":"2023-10-16T12:10:43.653456+0000","last_clean":"2023-10-16T12:10:43.653456+0000","last_became_active":"2023-10-16T12:10:39.632868+0000","last_became_peered":"2023-10-16T12:10:39.632868+0000","last_unstale":"2023-10-16T12:10:43.653456+0000","last_undegraded":"2023-10-16T12:10:43.653456+0000","last_fullsized":"2023-10-16T12:10:43.653456+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:33:20.807621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:44.683898+0000","last_change":"2023-10-16T12:10:43.794748+0000","last_active":"2023-10-16T12:10:44.683898+0000","last_peered":"2023-10-16T12:10:44.683898+0000","last_clean":"2023-10-16T12:10:44.683898+0000","last_became_active":"2023-10-16T12:10:43.794362+0000","last_became_peered":"2023-10-16T12:10:43.794362+0000","last_unstale":"2023-10-16T12:10:44.683898+0000","last_undegraded":"2023-10-16T12:10:44.683898+0000","last_fullsized":"2023-10-16T12:10:44.683898+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:57:20.445348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.652120+0000","last_change":"2023-10-16T12:10:38.619590+0000","last_active":"2023-10-16T12:10:43.652120+0000","last_peered":"2023-10-16T12:10:43.652120+0000","last_clean":"2023-10-16T12:10:43.652120+0000","last_became_active":"2023-10-16T12:10:38.618984+0000","last_became_peered":"2023-10-16T12:10:38.618984+0000","last_unstale":"2023-10-16T12:10:43.652120+0000","last_undegraded":"2023-10-16T12:10:43.652120+0000","last_fullsized":"2023-10-16T12:10:43.652120+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:27:05.182216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646297+0000","last_change":"2023-10-16T12:10:39.644469+0000","last_active":"2023-10-16T12:10:43.646297+0000","last_peered":"2023-10-16T12:10:43.646297+0000","last_clean":"2023-10-16T12:10:43.646297+0000","last_became_active":"2023-10-16T12:10:39.642359+0000","last_became_peered":"2023-10-16T12:10:39.642359+0000","last_unstale":"2023-10-16T12:10:43.646297+0000","last_undegraded":"2023-10-16T12:10:43.646297+0000","last_fullsized":"2023-10-16T12:10:43.646297+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:30:10.006613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:44.684242+0000","last_change":"2023-10-16T12:10:43.667401+0000","last_active":"2023-10-16T12:10:44.684242+0000","last_peered":"2023-10-16T12:10:44.684242+0000","last_clean":"2023-10-16T12:10:44.684242+0000","last_became_active":"2023-10-16T12:10:43.666829+0000","last_became_peered":"2023-10-16T12:10:43.666829+0000","last_unstale":"2023-10-16T12:10:44.684242+0000","last_undegraded":"2023-10-16T12:10:44.684242+0000","last_fullsized":"2023-10-16T12:10:44.684242+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:13:48.034204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646585+0000","last_change":"2023-10-16T12:10:38.627372+0000","last_active":"2023-10-16T12:10:43.646585+0000","last_peered":"2023-10-16T12:10:43.646585+0000","last_clean":"2023-10-16T12:10:43.646585+0000","last_became_active":"2023-10-16T12:10:38.622500+0000","last_became_peered":"2023-10-16T12:10:38.622500+0000","last_unstale":"2023-10-16T12:10:43.646585+0000","last_undegraded":"2023-10-16T12:10:43.646585+0000","last_fullsized":"2023-10-16T12:10:43.646585+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:57:56.978587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646591+0000","last_change":"2023-10-16T12:10:39.641571+0000","last_active":"2023-10-16T12:10:43.646591+0000","last_peered":"2023-10-16T12:10:43.646591+0000","last_clean":"2023-10-16T12:10:43.646591+0000","last_became_active":"2023-10-16T12:10:39.641228+0000","last_became_peered":"2023-10-16T12:10:39.641228+0000","last_unstale":"2023-10-16T12:10:43.646591+0000","last_undegraded":"2023-10-16T12:10:43.646591+0000","last_fullsized":"2023-10-16T12:10:43.646591+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:38:01.595985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:45.376570+0000","last_change":"2023-10-16T12:10:43.671066+0000","last_active":"2023-10-16T12:10:45.376570+0000","last_peered":"2023-10-16T12:10:45.376570+0000","last_clean":"2023-10-16T12:10:45.376570+0000","last_became_active":"2023-10-16T12:10:43.670693+0000","last_became_peered":"2023-10-16T12:10:43.670693+0000","last_unstale":"2023-10-16T12:10:45.376570+0000","last_undegraded":"2023-10-16T12:10:45.376570+0000","last_fullsized":"2023-10-16T12:10:45.376570+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:24:19.577814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.654541+0000","last_change":"2023-10-16T12:10:38.623191+0000","last_active":"2023-10-16T12:10:43.654541+0000","last_peered":"2023-10-16T12:10:43.654541+0000","last_clean":"2023-10-16T12:10:43.654541+0000","last_became_active":"2023-10-16T12:10:38.622876+0000","last_became_peered":"2023-10-16T12:10:38.622876+0000","last_unstale":"2023-10-16T12:10:43.654541+0000","last_undegraded":"2023-10-16T12:10:43.654541+0000","last_fullsized":"2023-10-16T12:10:43.654541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:59:30.615293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.648686+0000","last_change":"2023-10-16T12:10:39.643461+0000","last_active":"2023-10-16T12:10:43.648686+0000","last_peered":"2023-10-16T12:10:43.648686+0000","last_clean":"2023-10-16T12:10:43.648686+0000","last_became_active":"2023-10-16T12:10:39.643148+0000","last_became_peered":"2023-10-16T12:10:39.643148+0000","last_unstale":"2023-10-16T12:10:43.648686+0000","last_undegraded":"2023-10-16T12:10:43.648686+0000","last_fullsized":"2023-10-16T12:10:43.648686+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:27:06.186345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:44.682345+0000","last_change":"2023-10-16T12:10:43.662674+0000","last_active":"2023-10-16T12:10:44.682345+0000","last_peered":"2023-10-16T12:10:44.682345+0000","last_clean":"2023-10-16T12:10:44.682345+0000","last_became_active":"2023-10-16T12:10:43.662249+0000","last_became_peered":"2023-10-16T12:10:43.662249+0000","last_unstale":"2023-10-16T12:10:44.682345+0000","last_undegraded":"2023-10-16T12:10:44.682345+0000","last_fullsized":"2023-10-16T12:10:44.682345+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:14:04.764499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"25'9","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:49.578853+0000","last_change":"2023-10-16T12:10:38.621924+0000","last_active":"2023-10-16T12:10:49.578853+0000","last_peered":"2023-10-16T12:10:49.578853+0000","last_clean":"2023-10-16T12:10:49.578853+0000","last_became_active":"2023-10-16T12:10:38.621621+0000","last_became_peered":"2023-10-16T12:10:38.621621+0000","last_unstale":"2023-10-16T12:10:49.578853+0000","last_undegraded":"2023-10-16T12:10:49.578853+0000","last_fullsized":"2023-10-16T12:10:49.578853+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:27:30.532335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.796277+0000","last_change":"2023-10-16T12:10:39.641165+0000","last_active":"2023-10-16T12:10:43.796277+0000","last_peered":"2023-10-16T12:10:43.796277+0000","last_clean":"2023-10-16T12:10:43.796277+0000","last_became_active":"2023-10-16T12:10:39.640241+0000","last_became_peered":"2023-10-16T12:10:39.640241+0000","last_unstale":"2023-10-16T12:10:43.796277+0000","last_undegraded":"2023-10-16T12:10:43.796277+0000","last_fullsized":"2023-10-16T12:10:43.796277+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:38:53.624950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:45.377704+0000","last_change":"2023-10-16T12:10:43.796949+0000","last_active":"2023-10-16T12:10:45.377704+0000","last_peered":"2023-10-16T12:10:45.377704+0000","last_clean":"2023-10-16T12:10:45.377704+0000","last_became_active":"2023-10-16T12:10:43.796518+0000","last_became_peered":"2023-10-16T12:10:43.796518+0000","last_unstale":"2023-10-16T12:10:45.377704+0000","last_undegraded":"2023-10-16T12:10:45.377704+0000","last_fullsized":"2023-10-16T12:10:45.377704+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:58:35.336755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.650051+0000","last_change":"2023-10-16T12:10:38.612552+0000","last_active":"2023-10-16T12:10:43.650051+0000","last_peered":"2023-10-16T12:10:43.650051+0000","last_clean":"2023-10-16T12:10:43.650051+0000","last_became_active":"2023-10-16T12:10:38.612221+0000","last_became_peered":"2023-10-16T12:10:38.612221+0000","last_unstale":"2023-10-16T12:10:43.650051+0000","last_undegraded":"2023-10-16T12:10:43.650051+0000","last_fullsized":"2023-10-16T12:10:43.650051+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:27:05.182216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646287+0000","last_change":"2023-10-16T12:10:39.638781+0000","last_active":"2023-10-16T12:10:43.646287+0000","last_peered":"2023-10-16T12:10:43.646287+0000","last_clean":"2023-10-16T12:10:43.646287+0000","last_became_active":"2023-10-16T12:10:39.638523+0000","last_became_peered":"2023-10-16T12:10:39.638523+0000","last_unstale":"2023-10-16T12:10:43.646287+0000","last_undegraded":"2023-10-16T12:10:43.646287+0000","last_fullsized":"2023-10-16T12:10:43.646287+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:05:05.699350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:44.683241+0000","last_change":"2023-10-16T12:10:43.794196+0000","last_active":"2023-10-16T12:10:44.683241+0000","last_peered":"2023-10-16T12:10:44.683241+0000","last_clean":"2023-10-16T12:10:44.683241+0000","last_became_active":"2023-10-16T12:10:43.793712+0000","last_became_peered":"2023-10-16T12:10:43.793712+0000","last_unstale":"2023-10-16T12:10:44.683241+0000","last_undegraded":"2023-10-16T12:10:44.683241+0000","last_fullsized":"2023-10-16T12:10:44.683241+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:40:27.506374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646125+0000","last_change":"2023-10-16T12:10:38.627403+0000","last_active":"2023-10-16T12:10:43.646125+0000","last_peered":"2023-10-16T12:10:43.646125+0000","last_clean":"2023-10-16T12:10:43.646125+0000","last_became_active":"2023-10-16T12:10:38.623013+0000","last_became_peered":"2023-10-16T12:10:38.623013+0000","last_unstale":"2023-10-16T12:10:43.646125+0000","last_undegraded":"2023-10-16T12:10:43.646125+0000","last_fullsized":"2023-10-16T12:10:43.646125+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:47:51.328162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.794292+0000","last_change":"2023-10-16T12:10:39.640179+0000","last_active":"2023-10-16T12:10:43.794292+0000","last_peered":"2023-10-16T12:10:43.794292+0000","last_clean":"2023-10-16T12:10:43.794292+0000","last_became_active":"2023-10-16T12:10:39.639843+0000","last_became_peered":"2023-10-16T12:10:39.639843+0000","last_unstale":"2023-10-16T12:10:43.794292+0000","last_undegraded":"2023-10-16T12:10:43.794292+0000","last_fullsized":"2023-10-16T12:10:43.794292+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:24:15.538866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:44.683431+0000","last_change":"2023-10-16T12:10:43.796968+0000","last_active":"2023-10-16T12:10:44.683431+0000","last_peered":"2023-10-16T12:10:44.683431+0000","last_clean":"2023-10-16T12:10:44.683431+0000","last_became_active":"2023-10-16T12:10:43.796562+0000","last_became_peered":"2023-10-16T12:10:43.796562+0000","last_unstale":"2023-10-16T12:10:44.683431+0000","last_undegraded":"2023-10-16T12:10:44.683431+0000","last_fullsized":"2023-10-16T12:10:44.683431+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:44:47.973077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.650016+0000","last_change":"2023-10-16T12:10:38.622066+0000","last_active":"2023-10-16T12:10:43.650016+0000","last_peered":"2023-10-16T12:10:43.650016+0000","last_clean":"2023-10-16T12:10:43.650016+0000","last_became_active":"2023-10-16T12:10:38.621453+0000","last_became_peered":"2023-10-16T12:10:38.621453+0000","last_unstale":"2023-10-16T12:10:43.650016+0000","last_undegraded":"2023-10-16T12:10:43.650016+0000","last_fullsized":"2023-10-16T12:10:43.650016+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:45:22.011420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.645479+0000","last_change":"2023-10-16T12:10:39.636432+0000","last_active":"2023-10-16T12:10:43.645479+0000","last_peered":"2023-10-16T12:10:43.645479+0000","last_clean":"2023-10-16T12:10:43.645479+0000","last_became_active":"2023-10-16T12:10:39.636136+0000","last_became_peered":"2023-10-16T12:10:39.636136+0000","last_unstale":"2023-10-16T12:10:43.645479+0000","last_undegraded":"2023-10-16T12:10:43.645479+0000","last_fullsized":"2023-10-16T12:10:43.645479+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:46:02.593591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:45.375145+0000","last_change":"2023-10-16T12:10:43.661450+0000","last_active":"2023-10-16T12:10:45.375145+0000","last_peered":"2023-10-16T12:10:45.375145+0000","last_clean":"2023-10-16T12:10:45.375145+0000","last_became_active":"2023-10-16T12:10:43.660931+0000","last_became_peered":"2023-10-16T12:10:43.660931+0000","last_unstale":"2023-10-16T12:10:45.375145+0000","last_undegraded":"2023-10-16T12:10:45.375145+0000","last_fullsized":"2023-10-16T12:10:45.375145+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:53:08.121149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.642296+0000","last_change":"2023-10-16T12:10:38.626933+0000","last_active":"2023-10-16T12:10:43.642296+0000","last_peered":"2023-10-16T12:10:43.642296+0000","last_clean":"2023-10-16T12:10:43.642296+0000","last_became_active":"2023-10-16T12:10:38.626643+0000","last_became_peered":"2023-10-16T12:10:38.626643+0000","last_unstale":"2023-10-16T12:10:43.642296+0000","last_undegraded":"2023-10-16T12:10:43.642296+0000","last_fullsized":"2023-10-16T12:10:43.642296+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:01:27.462901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.794002+0000","last_change":"2023-10-16T12:10:39.648098+0000","last_active":"2023-10-16T12:10:43.794002+0000","last_peered":"2023-10-16T12:10:43.794002+0000","last_clean":"2023-10-16T12:10:43.794002+0000","last_became_active":"2023-10-16T12:10:39.643195+0000","last_became_peered":"2023-10-16T12:10:39.643195+0000","last_unstale":"2023-10-16T12:10:43.794002+0000","last_undegraded":"2023-10-16T12:10:43.794002+0000","last_fullsized":"2023-10-16T12:10:43.794002+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:34:50.059469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:45.374887+0000","last_change":"2023-10-16T12:10:43.662584+0000","last_active":"2023-10-16T12:10:45.374887+0000","last_peered":"2023-10-16T12:10:45.374887+0000","last_clean":"2023-10-16T12:10:45.374887+0000","last_became_active":"2023-10-16T12:10:43.662164+0000","last_became_peered":"2023-10-16T12:10:43.662164+0000","last_unstale":"2023-10-16T12:10:45.374887+0000","last_undegraded":"2023-10-16T12:10:45.374887+0000","last_fullsized":"2023-10-16T12:10:45.374887+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:03:16.364854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.648873+0000","last_change":"2023-10-16T12:10:38.613102+0000","last_active":"2023-10-16T12:10:43.648873+0000","last_peered":"2023-10-16T12:10:43.648873+0000","last_clean":"2023-10-16T12:10:43.648873+0000","last_became_active":"2023-10-16T12:10:38.612596+0000","last_became_peered":"2023-10-16T12:10:38.612596+0000","last_unstale":"2023-10-16T12:10:43.648873+0000","last_undegraded":"2023-10-16T12:10:43.648873+0000","last_fullsized":"2023-10-16T12:10:43.648873+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:59:50.957247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.796815+0000","last_change":"2023-10-16T12:10:39.640141+0000","last_active":"2023-10-16T12:10:43.796815+0000","last_peered":"2023-10-16T12:10:43.796815+0000","last_clean":"2023-10-16T12:10:43.796815+0000","last_became_active":"2023-10-16T12:10:39.639770+0000","last_became_peered":"2023-10-16T12:10:39.639770+0000","last_unstale":"2023-10-16T12:10:43.796815+0000","last_undegraded":"2023-10-16T12:10:43.796815+0000","last_fullsized":"2023-10-16T12:10:43.796815+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:14:00.725551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:44.683486+0000","last_change":"2023-10-16T12:10:43.666858+0000","last_active":"2023-10-16T12:10:44.683486+0000","last_peered":"2023-10-16T12:10:44.683486+0000","last_clean":"2023-10-16T12:10:44.683486+0000","last_became_active":"2023-10-16T12:10:43.666461+0000","last_became_peered":"2023-10-16T12:10:43.666461+0000","last_unstale":"2023-10-16T12:10:44.683486+0000","last_undegraded":"2023-10-16T12:10:44.683486+0000","last_fullsized":"2023-10-16T12:10:44.683486+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:53:08.121149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.663314+0000","last_change":"2023-10-16T12:10:38.625538+0000","last_active":"2023-10-16T12:10:43.663314+0000","last_peered":"2023-10-16T12:10:43.663314+0000","last_clean":"2023-10-16T12:10:43.663314+0000","last_became_active":"2023-10-16T12:10:38.625227+0000","last_became_peered":"2023-10-16T12:10:38.625227+0000","last_unstale":"2023-10-16T12:10:43.663314+0000","last_undegraded":"2023-10-16T12:10:43.663314+0000","last_fullsized":"2023-10-16T12:10:43.663314+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:39:25.396697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.653596+0000","last_change":"2023-10-16T12:10:39.645373+0000","last_active":"2023-10-16T12:10:43.653596+0000","last_peered":"2023-10-16T12:10:43.653596+0000","last_clean":"2023-10-16T12:10:43.653596+0000","last_became_active":"2023-10-16T12:10:39.645083+0000","last_became_peered":"2023-10-16T12:10:39.645083+0000","last_unstale":"2023-10-16T12:10:43.653596+0000","last_undegraded":"2023-10-16T12:10:43.653596+0000","last_fullsized":"2023-10-16T12:10:43.653596+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:53:04.082200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:44.683426+0000","last_change":"2023-10-16T12:10:43.673415+0000","last_active":"2023-10-16T12:10:44.683426+0000","last_peered":"2023-10-16T12:10:44.683426+0000","last_clean":"2023-10-16T12:10:44.683426+0000","last_became_active":"2023-10-16T12:10:43.672874+0000","last_became_peered":"2023-10-16T12:10:43.672874+0000","last_unstale":"2023-10-16T12:10:44.683426+0000","last_undegraded":"2023-10-16T12:10:44.683426+0000","last_fullsized":"2023-10-16T12:10:44.683426+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:38:57.663899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646014+0000","last_change":"2023-10-16T12:10:38.628007+0000","last_active":"2023-10-16T12:10:43.646014+0000","last_peered":"2023-10-16T12:10:43.646014+0000","last_clean":"2023-10-16T12:10:43.646014+0000","last_became_active":"2023-10-16T12:10:38.627760+0000","last_became_peered":"2023-10-16T12:10:38.627760+0000","last_unstale":"2023-10-16T12:10:43.646014+0000","last_undegraded":"2023-10-16T12:10:43.646014+0000","last_fullsized":"2023-10-16T12:10:43.646014+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:46:01.589462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.652978+0000","last_change":"2023-10-16T12:10:39.632645+0000","last_active":"2023-10-16T12:10:43.652978+0000","last_peered":"2023-10-16T12:10:43.652978+0000","last_clean":"2023-10-16T12:10:43.652978+0000","last_became_active":"2023-10-16T12:10:39.632345+0000","last_became_peered":"2023-10-16T12:10:39.632345+0000","last_unstale":"2023-10-16T12:10:43.652978+0000","last_undegraded":"2023-10-16T12:10:43.652978+0000","last_fullsized":"2023-10-16T12:10:43.652978+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:58:21.398705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:44.683787+0000","last_change":"2023-10-16T12:10:43.667036+0000","last_active":"2023-10-16T12:10:44.683787+0000","last_peered":"2023-10-16T12:10:44.683787+0000","last_clean":"2023-10-16T12:10:44.683787+0000","last_became_active":"2023-10-16T12:10:43.666639+0000","last_became_peered":"2023-10-16T12:10:43.666639+0000","last_unstale":"2023-10-16T12:10:44.683787+0000","last_undegraded":"2023-10-16T12:10:44.683787+0000","last_fullsized":"2023-10-16T12:10:44.683787+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:29:47.533439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.657522+0000","last_change":"2023-10-16T12:10:39.633035+0000","last_active":"2023-10-16T12:10:43.657522+0000","last_peered":"2023-10-16T12:10:43.657522+0000","last_clean":"2023-10-16T12:10:43.657522+0000","last_became_active":"2023-10-16T12:10:39.631630+0000","last_became_peered":"2023-10-16T12:10:39.631630+0000","last_unstale":"2023-10-16T12:10:43.657522+0000","last_undegraded":"2023-10-16T12:10:43.657522+0000","last_fullsized":"2023-10-16T12:10:43.657522+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:47:52.332290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.642429+0000","last_change":"2023-10-16T12:10:38.622647+0000","last_active":"2023-10-16T12:10:43.642429+0000","last_peered":"2023-10-16T12:10:43.642429+0000","last_clean":"2023-10-16T12:10:43.642429+0000","last_became_active":"2023-10-16T12:10:38.622323+0000","last_became_peered":"2023-10-16T12:10:38.622323+0000","last_unstale":"2023-10-16T12:10:43.642429+0000","last_undegraded":"2023-10-16T12:10:43.642429+0000","last_fullsized":"2023-10-16T12:10:43.642429+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:27:30.532335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:44.874638+0000","last_change":"2023-10-16T12:10:43.797789+0000","last_active":"2023-10-16T12:10:44.874638+0000","last_peered":"2023-10-16T12:10:44.874638+0000","last_clean":"2023-10-16T12:10:44.874638+0000","last_became_active":"2023-10-16T12:10:43.797367+0000","last_became_peered":"2023-10-16T12:10:43.797367+0000","last_unstale":"2023-10-16T12:10:44.874638+0000","last_undegraded":"2023-10-16T12:10:44.874638+0000","last_fullsized":"2023-10-16T12:10:44.874638+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:24:19.577814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646387+0000","last_change":"2023-10-16T12:10:39.650651+0000","last_active":"2023-10-16T12:10:43.646387+0000","last_peered":"2023-10-16T12:10:43.646387+0000","last_clean":"2023-10-16T12:10:43.646387+0000","last_became_active":"2023-10-16T12:10:39.650279+0000","last_became_peered":"2023-10-16T12:10:39.650279+0000","last_unstale":"2023-10-16T12:10:43.646387+0000","last_undegraded":"2023-10-16T12:10:43.646387+0000","last_fullsized":"2023-10-16T12:10:43.646387+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:03:12.325906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646002+0000","last_change":"2023-10-16T12:10:38.622760+0000","last_active":"2023-10-16T12:10:43.646002+0000","last_peered":"2023-10-16T12:10:43.646002+0000","last_clean":"2023-10-16T12:10:43.646002+0000","last_became_active":"2023-10-16T12:10:38.622408+0000","last_became_peered":"2023-10-16T12:10:38.622408+0000","last_unstale":"2023-10-16T12:10:43.646002+0000","last_undegraded":"2023-10-16T12:10:43.646002+0000","last_fullsized":"2023-10-16T12:10:43.646002+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:52:07.438053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":169,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.793810+0000","last_change":"2023-10-16T12:10:34.746820+0000","last_active":"2023-10-16T12:10:43.793810+0000","last_peered":"2023-10-16T12:10:43.793810+0000","last_clean":"2023-10-16T12:10:43.793810+0000","last_became_active":"2023-10-16T12:10:34.745574+0000","last_became_peered":"2023-10-16T12:10:34.745574+0000","last_unstale":"2023-10-16T12:10:43.793810+0000","last_undegraded":"2023-10-16T12:10:43.793810+0000","last_fullsized":"2023-10-16T12:10:43.793810+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_clean_scrub_stamp":"2023-10-16T12:10:33.577111+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:14.461986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:45.375419+0000","last_change":"2023-10-16T12:10:43.798352+0000","last_active":"2023-10-16T12:10:45.375419+0000","last_peered":"2023-10-16T12:10:45.375419+0000","last_clean":"2023-10-16T12:10:45.375419+0000","last_became_active":"2023-10-16T12:10:43.797410+0000","last_became_peered":"2023-10-16T12:10:43.797410+0000","last_unstale":"2023-10-16T12:10:45.375419+0000","last_undegraded":"2023-10-16T12:10:45.375419+0000","last_fullsized":"2023-10-16T12:10:45.375419+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:03:18.128385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646524+0000","last_change":"2023-10-16T12:10:39.635017+0000","last_active":"2023-10-16T12:10:43.646524+0000","last_peered":"2023-10-16T12:10:43.646524+0000","last_clean":"2023-10-16T12:10:43.646524+0000","last_became_active":"2023-10-16T12:10:39.634564+0000","last_became_peered":"2023-10-16T12:10:39.634564+0000","last_unstale":"2023-10-16T12:10:43.646524+0000","last_undegraded":"2023-10-16T12:10:43.646524+0000","last_fullsized":"2023-10-16T12:10:43.646524+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:30:57.483330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.793813+0000","last_change":"2023-10-16T12:10:38.626904+0000","last_active":"2023-10-16T12:10:43.793813+0000","last_peered":"2023-10-16T12:10:43.793813+0000","last_clean":"2023-10-16T12:10:43.793813+0000","last_became_active":"2023-10-16T12:10:38.626683+0000","last_became_peered":"2023-10-16T12:10:38.626683+0000","last_unstale":"2023-10-16T12:10:43.793813+0000","last_undegraded":"2023-10-16T12:10:43.793813+0000","last_fullsized":"2023-10-16T12:10:43.793813+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:58:20.394576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:45.375040+0000","last_change":"2023-10-16T12:10:43.794928+0000","last_active":"2023-10-16T12:10:45.375040+0000","last_peered":"2023-10-16T12:10:45.375040+0000","last_clean":"2023-10-16T12:10:45.375040+0000","last_became_active":"2023-10-16T12:10:43.794276+0000","last_became_peered":"2023-10-16T12:10:43.794276+0000","last_unstale":"2023-10-16T12:10:45.375040+0000","last_undegraded":"2023-10-16T12:10:45.375040+0000","last_fullsized":"2023-10-16T12:10:45.375040+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:38:57.663899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.795963+0000","last_change":"2023-10-16T12:10:38.620637+0000","last_active":"2023-10-16T12:10:43.795963+0000","last_peered":"2023-10-16T12:10:43.795963+0000","last_clean":"2023-10-16T12:10:43.795963+0000","last_became_active":"2023-10-16T12:10:38.620275+0000","last_became_peered":"2023-10-16T12:10:38.620275+0000","last_unstale":"2023-10-16T12:10:43.795963+0000","last_undegraded":"2023-10-16T12:10:43.795963+0000","last_fullsized":"2023-10-16T12:10:43.795963+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:38:00.591856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.663386+0000","last_change":"2023-10-16T12:10:39.632571+0000","last_active":"2023-10-16T12:10:43.663386+0000","last_peered":"2023-10-16T12:10:43.663386+0000","last_clean":"2023-10-16T12:10:43.663386+0000","last_became_active":"2023-10-16T12:10:39.631893+0000","last_became_peered":"2023-10-16T12:10:39.631893+0000","last_unstale":"2023-10-16T12:10:43.663386+0000","last_undegraded":"2023-10-16T12:10:43.663386+0000","last_fullsized":"2023-10-16T12:10:43.663386+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:05:05.699350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:44.683543+0000","last_change":"2023-10-16T12:10:43.797399+0000","last_active":"2023-10-16T12:10:44.683543+0000","last_peered":"2023-10-16T12:10:44.683543+0000","last_clean":"2023-10-16T12:10:44.683543+0000","last_became_active":"2023-10-16T12:10:43.796570+0000","last_became_peered":"2023-10-16T12:10:43.796570+0000","last_unstale":"2023-10-16T12:10:44.683543+0000","last_undegraded":"2023-10-16T12:10:44.683543+0000","last_fullsized":"2023-10-16T12:10:44.683543+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:58:35.336755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.656426+0000","last_change":"2023-10-16T12:10:38.620866+0000","last_active":"2023-10-16T12:10:43.656426+0000","last_peered":"2023-10-16T12:10:43.656426+0000","last_clean":"2023-10-16T12:10:43.656426+0000","last_became_active":"2023-10-16T12:10:38.620517+0000","last_became_peered":"2023-10-16T12:10:38.620517+0000","last_unstale":"2023-10-16T12:10:43.656426+0000","last_undegraded":"2023-10-16T12:10:43.656426+0000","last_fullsized":"2023-10-16T12:10:43.656426+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:58:20.394576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646193+0000","last_change":"2023-10-16T12:10:39.639395+0000","last_active":"2023-10-16T12:10:43.646193+0000","last_peered":"2023-10-16T12:10:43.646193+0000","last_clean":"2023-10-16T12:10:43.646193+0000","last_became_active":"2023-10-16T12:10:39.639030+0000","last_became_peered":"2023-10-16T12:10:39.639030+0000","last_unstale":"2023-10-16T12:10:43.646193+0000","last_undegraded":"2023-10-16T12:10:43.646193+0000","last_fullsized":"2023-10-16T12:10:43.646193+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:34:50.059469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:45.375076+0000","last_change":"2023-10-16T12:10:43.800001+0000","last_active":"2023-10-16T12:10:45.375076+0000","last_peered":"2023-10-16T12:10:45.375076+0000","last_clean":"2023-10-16T12:10:45.375076+0000","last_became_active":"2023-10-16T12:10:43.799598+0000","last_became_peered":"2023-10-16T12:10:43.799598+0000","last_unstale":"2023-10-16T12:10:45.375076+0000","last_undegraded":"2023-10-16T12:10:45.375076+0000","last_fullsized":"2023-10-16T12:10:45.375076+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:27:48.055325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.652023+0000","last_change":"2023-10-16T12:10:38.619506+0000","last_active":"2023-10-16T12:10:43.652023+0000","last_peered":"2023-10-16T12:10:43.652023+0000","last_clean":"2023-10-16T12:10:43.652023+0000","last_became_active":"2023-10-16T12:10:38.619203+0000","last_became_peered":"2023-10-16T12:10:38.619203+0000","last_unstale":"2023-10-16T12:10:43.652023+0000","last_undegraded":"2023-10-16T12:10:43.652023+0000","last_fullsized":"2023-10-16T12:10:43.652023+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:58:20.394576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.648502+0000","last_change":"2023-10-16T12:10:39.630721+0000","last_active":"2023-10-16T12:10:43.648502+0000","last_peered":"2023-10-16T12:10:43.648502+0000","last_clean":"2023-10-16T12:10:43.648502+0000","last_became_active":"2023-10-16T12:10:39.630448+0000","last_became_peered":"2023-10-16T12:10:39.630448+0000","last_unstale":"2023-10-16T12:10:43.648502+0000","last_undegraded":"2023-10-16T12:10:43.648502+0000","last_fullsized":"2023-10-16T12:10:43.648502+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:03:12.325906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:45.376403+0000","last_change":"2023-10-16T12:10:43.662711+0000","last_active":"2023-10-16T12:10:45.376403+0000","last_peered":"2023-10-16T12:10:45.376403+0000","last_clean":"2023-10-16T12:10:45.376403+0000","last_became_active":"2023-10-16T12:10:43.661726+0000","last_became_peered":"2023-10-16T12:10:43.661726+0000","last_unstale":"2023-10-16T12:10:45.376403+0000","last_undegraded":"2023-10-16T12:10:45.376403+0000","last_fullsized":"2023-10-16T12:10:45.376403+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:13:07.783237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.793165+0000","last_change":"2023-10-16T12:10:38.620553+0000","last_active":"2023-10-16T12:10:43.793165+0000","last_peered":"2023-10-16T12:10:43.793165+0000","last_clean":"2023-10-16T12:10:43.793165+0000","last_became_active":"2023-10-16T12:10:38.620248+0000","last_became_peered":"2023-10-16T12:10:38.620248+0000","last_unstale":"2023-10-16T12:10:43.793165+0000","last_undegraded":"2023-10-16T12:10:43.793165+0000","last_fullsized":"2023-10-16T12:10:43.793165+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:45:22.011420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.647607+0000","last_change":"2023-10-16T12:10:39.631773+0000","last_active":"2023-10-16T12:10:43.647607+0000","last_peered":"2023-10-16T12:10:43.647607+0000","last_clean":"2023-10-16T12:10:43.647607+0000","last_became_active":"2023-10-16T12:10:39.631418+0000","last_became_peered":"2023-10-16T12:10:39.631418+0000","last_unstale":"2023-10-16T12:10:43.647607+0000","last_undegraded":"2023-10-16T12:10:43.647607+0000","last_fullsized":"2023-10-16T12:10:43.647607+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:45:23.015548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:44.683656+0000","last_change":"2023-10-16T12:10:43.794614+0000","last_active":"2023-10-16T12:10:44.683656+0000","last_peered":"2023-10-16T12:10:44.683656+0000","last_clean":"2023-10-16T12:10:44.683656+0000","last_became_active":"2023-10-16T12:10:43.794206+0000","last_became_peered":"2023-10-16T12:10:43.794206+0000","last_unstale":"2023-10-16T12:10:44.683656+0000","last_undegraded":"2023-10-16T12:10:44.683656+0000","last_fullsized":"2023-10-16T12:10:44.683656+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:03:18.128385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.653349+0000","last_change":"2023-10-16T12:10:38.621473+0000","last_active":"2023-10-16T12:10:43.653349+0000","last_peered":"2023-10-16T12:10:43.653349+0000","last_clean":"2023-10-16T12:10:43.653349+0000","last_became_active":"2023-10-16T12:10:38.619100+0000","last_became_peered":"2023-10-16T12:10:38.619100+0000","last_unstale":"2023-10-16T12:10:43.653349+0000","last_undegraded":"2023-10-16T12:10:43.653349+0000","last_fullsized":"2023-10-16T12:10:43.653349+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:04:47.764190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.647008+0000","last_change":"2023-10-16T12:10:39.630605+0000","last_active":"2023-10-16T12:10:43.647008+0000","last_peered":"2023-10-16T12:10:43.647008+0000","last_clean":"2023-10-16T12:10:43.647008+0000","last_became_active":"2023-10-16T12:10:39.630234+0000","last_became_peered":"2023-10-16T12:10:39.630234+0000","last_unstale":"2023-10-16T12:10:43.647008+0000","last_undegraded":"2023-10-16T12:10:43.647008+0000","last_fullsized":"2023-10-16T12:10:43.647008+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:33:20.807621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:45.376060+0000","last_change":"2023-10-16T12:10:43.665913+0000","last_active":"2023-10-16T12:10:45.376060+0000","last_peered":"2023-10-16T12:10:45.376060+0000","last_clean":"2023-10-16T12:10:45.376060+0000","last_became_active":"2023-10-16T12:10:43.665498+0000","last_became_peered":"2023-10-16T12:10:43.665498+0000","last_unstale":"2023-10-16T12:10:45.376060+0000","last_undegraded":"2023-10-16T12:10:45.376060+0000","last_fullsized":"2023-10-16T12:10:45.376060+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:10:23.879645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.652500+0000","last_change":"2023-10-16T12:10:38.623243+0000","last_active":"2023-10-16T12:10:43.652500+0000","last_peered":"2023-10-16T12:10:43.652500+0000","last_clean":"2023-10-16T12:10:43.652500+0000","last_became_active":"2023-10-16T12:10:38.622935+0000","last_became_peered":"2023-10-16T12:10:38.622935+0000","last_unstale":"2023-10-16T12:10:43.652500+0000","last_undegraded":"2023-10-16T12:10:43.652500+0000","last_fullsized":"2023-10-16T12:10:43.652500+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:27:30.532335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.645994+0000","last_change":"2023-10-16T12:10:39.624625+0000","last_active":"2023-10-16T12:10:43.645994+0000","last_peered":"2023-10-16T12:10:43.645994+0000","last_clean":"2023-10-16T12:10:43.645994+0000","last_became_active":"2023-10-16T12:10:39.624281+0000","last_became_peered":"2023-10-16T12:10:39.624281+0000","last_unstale":"2023-10-16T12:10:43.645994+0000","last_undegraded":"2023-10-16T12:10:43.645994+0000","last_fullsized":"2023-10-16T12:10:43.645994+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:27:44.016376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:45.377425+0000","last_change":"2023-10-16T12:10:43.662337+0000","last_active":"2023-10-16T12:10:45.377425+0000","last_peered":"2023-10-16T12:10:45.377425+0000","last_clean":"2023-10-16T12:10:45.377425+0000","last_became_active":"2023-10-16T12:10:43.661865+0000","last_became_peered":"2023-10-16T12:10:43.661865+0000","last_unstale":"2023-10-16T12:10:45.377425+0000","last_undegraded":"2023-10-16T12:10:45.377425+0000","last_fullsized":"2023-10-16T12:10:45.377425+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:14:04.764499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.794403+0000","last_change":"2023-10-16T12:10:38.624192+0000","last_active":"2023-10-16T12:10:43.794403+0000","last_peered":"2023-10-16T12:10:43.794403+0000","last_clean":"2023-10-16T12:10:43.794403+0000","last_became_active":"2023-10-16T12:10:38.623767+0000","last_became_peered":"2023-10-16T12:10:38.623767+0000","last_unstale":"2023-10-16T12:10:43.794403+0000","last_undegraded":"2023-10-16T12:10:43.794403+0000","last_fullsized":"2023-10-16T12:10:43.794403+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:59:30.615293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.651482+0000","last_change":"2023-10-16T12:10:39.645023+0000","last_active":"2023-10-16T12:10:43.651482+0000","last_peered":"2023-10-16T12:10:43.651482+0000","last_clean":"2023-10-16T12:10:43.651482+0000","last_became_active":"2023-10-16T12:10:39.644719+0000","last_became_peered":"2023-10-16T12:10:39.644719+0000","last_unstale":"2023-10-16T12:10:43.651482+0000","last_undegraded":"2023-10-16T12:10:43.651482+0000","last_fullsized":"2023-10-16T12:10:43.651482+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:47:52.332290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:44.874338+0000","last_change":"2023-10-16T12:10:43.794795+0000","last_active":"2023-10-16T12:10:44.874338+0000","last_peered":"2023-10-16T12:10:44.874338+0000","last_clean":"2023-10-16T12:10:44.874338+0000","last_became_active":"2023-10-16T12:10:43.794315+0000","last_became_peered":"2023-10-16T12:10:43.794315+0000","last_unstale":"2023-10-16T12:10:44.874338+0000","last_undegraded":"2023-10-16T12:10:44.874338+0000","last_fullsized":"2023-10-16T12:10:44.874338+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:13:07.783237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.642895+0000","last_change":"2023-10-16T12:10:38.622408+0000","last_active":"2023-10-16T12:10:43.642895+0000","last_peered":"2023-10-16T12:10:43.642895+0000","last_clean":"2023-10-16T12:10:43.642895+0000","last_became_active":"2023-10-16T12:10:38.622020+0000","last_became_peered":"2023-10-16T12:10:38.622020+0000","last_unstale":"2023-10-16T12:10:43.642895+0000","last_undegraded":"2023-10-16T12:10:43.642895+0000","last_fullsized":"2023-10-16T12:10:43.642895+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:57:56.978587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.796175+0000","last_change":"2023-10-16T12:10:39.645109+0000","last_active":"2023-10-16T12:10:43.796175+0000","last_peered":"2023-10-16T12:10:43.796175+0000","last_clean":"2023-10-16T12:10:43.796175+0000","last_became_active":"2023-10-16T12:10:39.641743+0000","last_became_peered":"2023-10-16T12:10:39.641743+0000","last_unstale":"2023-10-16T12:10:43.796175+0000","last_undegraded":"2023-10-16T12:10:43.796175+0000","last_fullsized":"2023-10-16T12:10:43.796175+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:53:04.082200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:45.374790+0000","last_change":"2023-10-16T12:10:43.666962+0000","last_active":"2023-10-16T12:10:45.374790+0000","last_peered":"2023-10-16T12:10:45.374790+0000","last_clean":"2023-10-16T12:10:45.374790+0000","last_became_active":"2023-10-16T12:10:43.666469+0000","last_became_peered":"2023-10-16T12:10:43.666469+0000","last_unstale":"2023-10-16T12:10:45.374790+0000","last_undegraded":"2023-10-16T12:10:45.374790+0000","last_fullsized":"2023-10-16T12:10:45.374790+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:14:04.764499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.654298+0000","last_change":"2023-10-16T12:10:38.611610+0000","last_active":"2023-10-16T12:10:43.654298+0000","last_peered":"2023-10-16T12:10:43.654298+0000","last_clean":"2023-10-16T12:10:43.654298+0000","last_became_active":"2023-10-16T12:10:38.611193+0000","last_became_peered":"2023-10-16T12:10:38.611193+0000","last_unstale":"2023-10-16T12:10:43.654298+0000","last_undegraded":"2023-10-16T12:10:43.654298+0000","last_fullsized":"2023-10-16T12:10:43.654298+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:06:08.125512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646733+0000","last_change":"2023-10-16T12:10:39.649078+0000","last_active":"2023-10-16T12:10:43.646733+0000","last_peered":"2023-10-16T12:10:43.646733+0000","last_clean":"2023-10-16T12:10:43.646733+0000","last_became_active":"2023-10-16T12:10:39.648797+0000","last_became_peered":"2023-10-16T12:10:39.648797+0000","last_unstale":"2023-10-16T12:10:43.646733+0000","last_undegraded":"2023-10-16T12:10:43.646733+0000","last_fullsized":"2023-10-16T12:10:43.646733+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:13:03.744288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:45.374995+0000","last_change":"2023-10-16T12:10:43.794916+0000","last_active":"2023-10-16T12:10:45.374995+0000","last_peered":"2023-10-16T12:10:45.374995+0000","last_clean":"2023-10-16T12:10:45.374995+0000","last_became_active":"2023-10-16T12:10:43.794495+0000","last_became_peered":"2023-10-16T12:10:43.794495+0000","last_unstale":"2023-10-16T12:10:45.374995+0000","last_undegraded":"2023-10-16T12:10:45.374995+0000","last_fullsized":"2023-10-16T12:10:45.374995+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:44:47.973077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.642563+0000","last_change":"2023-10-16T12:10:38.624223+0000","last_active":"2023-10-16T12:10:43.642563+0000","last_peered":"2023-10-16T12:10:43.642563+0000","last_clean":"2023-10-16T12:10:43.642563+0000","last_became_active":"2023-10-16T12:10:38.623924+0000","last_became_peered":"2023-10-16T12:10:38.623924+0000","last_unstale":"2023-10-16T12:10:43.642563+0000","last_undegraded":"2023-10-16T12:10:43.642563+0000","last_fullsized":"2023-10-16T12:10:43.642563+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:45:22.011420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.796741+0000","last_change":"2023-10-16T12:10:39.645947+0000","last_active":"2023-10-16T12:10:43.796741+0000","last_peered":"2023-10-16T12:10:43.796741+0000","last_clean":"2023-10-16T12:10:43.796741+0000","last_became_active":"2023-10-16T12:10:39.645261+0000","last_became_peered":"2023-10-16T12:10:39.645261+0000","last_unstale":"2023-10-16T12:10:43.796741+0000","last_undegraded":"2023-10-16T12:10:43.796741+0000","last_fullsized":"2023-10-16T12:10:43.796741+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:03:14.089436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.648802+0000","last_change":"2023-10-16T12:10:39.631025+0000","last_active":"2023-10-16T12:10:43.648802+0000","last_peered":"2023-10-16T12:10:43.648802+0000","last_clean":"2023-10-16T12:10:43.648802+0000","last_became_active":"2023-10-16T12:10:39.630701+0000","last_became_peered":"2023-10-16T12:10:39.630701+0000","last_unstale":"2023-10-16T12:10:43.648802+0000","last_undegraded":"2023-10-16T12:10:43.648802+0000","last_fullsized":"2023-10-16T12:10:43.648802+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:49:21.678699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.642384+0000","last_change":"2023-10-16T12:10:39.640966+0000","last_active":"2023-10-16T12:10:43.642384+0000","last_peered":"2023-10-16T12:10:43.642384+0000","last_clean":"2023-10-16T12:10:43.642384+0000","last_became_active":"2023-10-16T12:10:39.640624+0000","last_became_peered":"2023-10-16T12:10:39.640624+0000","last_unstale":"2023-10-16T12:10:43.642384+0000","last_undegraded":"2023-10-16T12:10:43.642384+0000","last_fullsized":"2023-10-16T12:10:43.642384+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:58:21.398705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646243+0000","last_change":"2023-10-16T12:10:39.647316+0000","last_active":"2023-10-16T12:10:43.646243+0000","last_peered":"2023-10-16T12:10:43.646243+0000","last_clean":"2023-10-16T12:10:43.646243+0000","last_became_active":"2023-10-16T12:10:39.646403+0000","last_became_peered":"2023-10-16T12:10:39.646403+0000","last_unstale":"2023-10-16T12:10:43.646243+0000","last_undegraded":"2023-10-16T12:10:43.646243+0000","last_fullsized":"2023-10-16T12:10:43.646243+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:24:15.538866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.649898+0000","last_change":"2023-10-16T12:10:39.625142+0000","last_active":"2023-10-16T12:10:43.649898+0000","last_peered":"2023-10-16T12:10:43.649898+0000","last_clean":"2023-10-16T12:10:43.649898+0000","last_became_active":"2023-10-16T12:10:39.624714+0000","last_became_peered":"2023-10-16T12:10:39.624714+0000","last_unstale":"2023-10-16T12:10:43.649898+0000","last_undegraded":"2023-10-16T12:10:43.649898+0000","last_fullsized":"2023-10-16T12:10:43.649898+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:46:02.593591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.648251+0000","last_change":"2023-10-16T12:10:39.642728+0000","last_active":"2023-10-16T12:10:43.648251+0000","last_peered":"2023-10-16T12:10:43.648251+0000","last_clean":"2023-10-16T12:10:43.648251+0000","last_became_active":"2023-10-16T12:10:39.642396+0000","last_became_peered":"2023-10-16T12:10:39.642396+0000","last_unstale":"2023-10-16T12:10:43.648251+0000","last_undegraded":"2023-10-16T12:10:43.648251+0000","last_fullsized":"2023-10-16T12:10:43.648251+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:38:01.595985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.654215+0000","last_change":"2023-10-16T12:10:39.635187+0000","last_active":"2023-10-16T12:10:43.654215+0000","last_peered":"2023-10-16T12:10:43.654215+0000","last_clean":"2023-10-16T12:10:43.654215+0000","last_became_active":"2023-10-16T12:10:39.634499+0000","last_became_peered":"2023-10-16T12:10:39.634499+0000","last_unstale":"2023-10-16T12:10:43.654215+0000","last_undegraded":"2023-10-16T12:10:43.654215+0000","last_fullsized":"2023-10-16T12:10:43.654215+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:38:01.595985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.650893+0000","last_change":"2023-10-16T12:10:39.643166+0000","last_active":"2023-10-16T12:10:43.650893+0000","last_peered":"2023-10-16T12:10:43.650893+0000","last_clean":"2023-10-16T12:10:43.650893+0000","last_became_active":"2023-10-16T12:10:39.642893+0000","last_became_peered":"2023-10-16T12:10:39.642893+0000","last_unstale":"2023-10-16T12:10:43.650893+0000","last_undegraded":"2023-10-16T12:10:43.650893+0000","last_fullsized":"2023-10-16T12:10:43.650893+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:33:20.807621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646227+0000","last_change":"2023-10-16T12:10:39.633538+0000","last_active":"2023-10-16T12:10:43.646227+0000","last_peered":"2023-10-16T12:10:43.646227+0000","last_clean":"2023-10-16T12:10:43.646227+0000","last_became_active":"2023-10-16T12:10:39.633019+0000","last_became_peered":"2023-10-16T12:10:39.633019+0000","last_unstale":"2023-10-16T12:10:43.646227+0000","last_undegraded":"2023-10-16T12:10:43.646227+0000","last_fullsized":"2023-10-16T12:10:43.646227+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:59:08.633021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.652820+0000","last_change":"2023-10-16T12:10:39.641581+0000","last_active":"2023-10-16T12:10:43.652820+0000","last_peered":"2023-10-16T12:10:43.652820+0000","last_clean":"2023-10-16T12:10:43.652820+0000","last_became_active":"2023-10-16T12:10:39.640847+0000","last_became_peered":"2023-10-16T12:10:39.640847+0000","last_unstale":"2023-10-16T12:10:43.652820+0000","last_undegraded":"2023-10-16T12:10:43.652820+0000","last_fullsized":"2023-10-16T12:10:43.652820+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:03:12.325906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646580+0000","last_change":"2023-10-16T12:10:39.642631+0000","last_active":"2023-10-16T12:10:43.646580+0000","last_peered":"2023-10-16T12:10:43.646580+0000","last_clean":"2023-10-16T12:10:43.646580+0000","last_became_active":"2023-10-16T12:10:39.642334+0000","last_became_peered":"2023-10-16T12:10:39.642334+0000","last_unstale":"2023-10-16T12:10:43.646580+0000","last_undegraded":"2023-10-16T12:10:43.646580+0000","last_fullsized":"2023-10-16T12:10:43.646580+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:46:02.593591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646326+0000","last_change":"2023-10-16T12:10:39.647489+0000","last_active":"2023-10-16T12:10:43.646326+0000","last_peered":"2023-10-16T12:10:43.646326+0000","last_clean":"2023-10-16T12:10:43.646326+0000","last_became_active":"2023-10-16T12:10:39.646439+0000","last_became_peered":"2023-10-16T12:10:39.646439+0000","last_unstale":"2023-10-16T12:10:43.646326+0000","last_undegraded":"2023-10-16T12:10:43.646326+0000","last_fullsized":"2023-10-16T12:10:43.646326+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:59:08.633021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.649955+0000","last_change":"2023-10-16T12:10:39.638887+0000","last_active":"2023-10-16T12:10:43.649955+0000","last_peered":"2023-10-16T12:10:43.649955+0000","last_clean":"2023-10-16T12:10:43.649955+0000","last_became_active":"2023-10-16T12:10:39.638548+0000","last_became_peered":"2023-10-16T12:10:39.638548+0000","last_unstale":"2023-10-16T12:10:43.649955+0000","last_undegraded":"2023-10-16T12:10:43.649955+0000","last_fullsized":"2023-10-16T12:10:43.649955+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:49:21.678699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.650677+0000","last_change":"2023-10-16T12:10:39.644900+0000","last_active":"2023-10-16T12:10:43.650677+0000","last_peered":"2023-10-16T12:10:43.650677+0000","last_clean":"2023-10-16T12:10:43.650677+0000","last_became_active":"2023-10-16T12:10:39.644516+0000","last_became_peered":"2023-10-16T12:10:39.644516+0000","last_unstale":"2023-10-16T12:10:43.650677+0000","last_undegraded":"2023-10-16T12:10:43.650677+0000","last_fullsized":"2023-10-16T12:10:43.650677+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:03:12.325906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574858,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27864,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343976,"statfs":{"total":96636764160,"available":96608231424,"internally_reserved":0,"allocated":864256,"data_stored":695401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72599999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68100000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71399999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82499999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52600000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":108293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.624}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54100000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45200000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51100000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69499999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76100000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":274432,"data_stored":104611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.374}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66900000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80400000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82399999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574857,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27876,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343964,"statfs":{"total":96636764160,"available":96608219136,"internally_reserved":0,"allocated":872448,"data_stored":695457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.312}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71399999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65000000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56299999999999994}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67500000000000004}]}]},{"osd":2,"up_from":13,"seq":55834574857,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":266240,"data_stored":104451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.31}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29299999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39200000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45500000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41099999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42699999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]}]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":108293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.252}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32700000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69499999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.627}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80800000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67100000000000004}]}]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":278528,"data_stored":104577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54300000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69299999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61399999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64100000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60099999999999998}]}]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":270336,"data_stored":104541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.497}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46700000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52500000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55300000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67600000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T12:11:04.335 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T12:11:04.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.491+0000 7f3a48966700 1 Processor -- start 2023-10-16T12:11:04.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.491+0000 7f3a48966700 1 -- start start 2023-10-16T12:11:04.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.492+0000 7f3a48966700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3a440cc8a0 0x7f3a4415e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:04.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.492+0000 7f3a48966700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f3a4406a7e0 0x7f3a4406ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:04.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.492+0000 7f3a48966700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a4406b150 con 0x7f3a4415b280 2023-10-16T12:11:04.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.492+0000 7f3a48966700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a44161c30 con 0x7f3a440cc8a0 2023-10-16T12:11:04.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.492+0000 7f3a48966700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a44161f10 con 0x7f3a4406a7e0 2023-10-16T12:11:04.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.492+0000 7f3a41d9b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3a440cc8a0 0x7f3a4415e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:04.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.493+0000 7f3a41d9b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3a440cc8a0 0x7f3a4415e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33470/0 (socket says 172.21.15.16:33470) 2023-10-16T12:11:04.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.493+0000 7f3a41d9b700 1 -- 172.21.15.16:0/1819450144 learned_addr learned my addr 172.21.15.16:0/1819450144 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:04.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.493+0000 7f3a42d9d700 1 --2- 172.21.15.16:0/1819450144 >> v2:172.21.15.49:3300/0 conn(0x7f3a4406a7e0 0x7f3a4406ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:04.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.493+0000 7f3a41d9b700 1 -- 172.21.15.16:0/1819450144 >> v2:172.21.15.49:3300/0 conn(0x7f3a4406a7e0 msgr2=0x7f3a4406ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:04.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.493+0000 7f3a41d9b700 1 --2- 172.21.15.16:0/1819450144 >> v2:172.21.15.49:3300/0 conn(0x7f3a4406a7e0 0x7f3a4406ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:04.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.493+0000 7f3a41d9b700 1 -- 172.21.15.16:0/1819450144 >> v1:172.21.15.49:6789/0 conn(0x7f3a4415b280 legacy=0x7f3a441613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:04.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.493+0000 7f3a41d9b700 1 -- 172.21.15.16:0/1819450144 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a441621f0 con 0x7f3a440cc8a0 2023-10-16T12:11:04.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.493+0000 7f3a41d9b700 1 --2- 172.21.15.16:0/1819450144 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a440cc8a0 0x7f3a4415e700 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3a440cbed0 tx=0x7f3a34009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbc751c7c67ad036 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:04.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.493+0000 7f3a4159a700 1 -- 172.21.15.16:0/1819450144 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a3400b870 con 0x7f3a440cc8a0 2023-10-16T12:11:04.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.493+0000 7f3a4159a700 1 -- 172.21.15.16:0/1819450144 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a3400b9d0 con 0x7f3a440cc8a0 2023-10-16T12:11:04.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.493+0000 7f3a4159a700 1 -- 172.21.15.16:0/1819450144 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a3400bd60 con 0x7f3a440cc8a0 2023-10-16T12:11:04.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.494+0000 7f3a48966700 1 -- 172.21.15.16:0/1819450144 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a440cc8a0 msgr2=0x7f3a4415e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:04.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.494+0000 7f3a48966700 1 --2- 172.21.15.16:0/1819450144 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a440cc8a0 0x7f3a4415e700 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3a440cbed0 tx=0x7f3a34009bd0 comp rx=0 tx=0).stop 2023-10-16T12:11:04.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.494+0000 7f3a48966700 1 -- 172.21.15.16:0/1819450144 shutdown_connections 2023-10-16T12:11:04.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.494+0000 7f3a48966700 1 --2- 172.21.15.16:0/1819450144 >> v2:172.21.15.49:3300/0 conn(0x7f3a4406a7e0 0x7f3a4406ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:04.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.494+0000 7f3a48966700 1 --2- 172.21.15.16:0/1819450144 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a440cc8a0 0x7f3a4415e700 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:04.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.494+0000 7f3a48966700 1 -- 172.21.15.16:0/1819450144 >> 172.21.15.16:0/1819450144 conn(0x7f3a440b1fd0 msgr2=0x7f3a440b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:04.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.494+0000 7f3a48966700 1 -- 172.21.15.16:0/1819450144 shutdown_connections 2023-10-16T12:11:04.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.494+0000 7f3a48966700 1 -- 172.21.15.16:0/1819450144 wait complete. 2023-10-16T12:11:04.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.495+0000 7f3a48966700 1 Processor -- start 2023-10-16T12:11:04.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.495+0000 7f3a48966700 1 -- start start 2023-10-16T12:11:04.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.496+0000 7f3a48966700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3a4406a7e0 0x7f3a440c59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:04.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.496+0000 7f3a48966700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3a440cc8a0 0x7f3a44071c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:04.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.496+0000 7f3a48966700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a4415b280 0x7f3a441f5800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:04.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.496+0000 7f3a48966700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a440c6860 con 0x7f3a4415b280 2023-10-16T12:11:04.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.496+0000 7f3a48966700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a440ae140 con 0x7f3a4406a7e0 2023-10-16T12:11:04.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.496+0000 7f3a48966700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a440ae420 con 0x7f3a440cc8a0 2023-10-16T12:11:04.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.496+0000 7f3a42d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a4415b280 0x7f3a441f5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:04.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.496+0000 7f3a42d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a4415b280 0x7f3a441f5800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33478/0 (socket says 172.21.15.16:33478) 2023-10-16T12:11:04.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.496+0000 7f3a42d9d700 1 -- 172.21.15.16:0/3643088106 learned_addr learned my addr 172.21.15.16:0/3643088106 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:04.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.496+0000 7f3a41d9b700 1 --2- 172.21.15.16:0/3643088106 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3a440cc8a0 0x7f3a44071c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:04.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.496+0000 7f3a4259c700 1 --2- 172.21.15.16:0/3643088106 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3a4406a7e0 0x7f3a440c59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:04.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.496+0000 7f3a42d9d700 1 -- 172.21.15.16:0/3643088106 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3a440cc8a0 msgr2=0x7f3a44071c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:04.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.496+0000 7f3a42d9d700 1 --2- 172.21.15.16:0/3643088106 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3a440cc8a0 0x7f3a44071c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:04.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.496+0000 7f3a42d9d700 1 -- 172.21.15.16:0/3643088106 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3a4406a7e0 msgr2=0x7f3a440c59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:04.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.496+0000 7f3a42d9d700 1 --2- 172.21.15.16:0/3643088106 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3a4406a7e0 0x7f3a440c59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:04.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.496+0000 7f3a42d9d700 1 -- 172.21.15.16:0/3643088106 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a34009550 con 0x7f3a4415b280 2023-10-16T12:11:04.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.497+0000 7f3a42d9d700 1 --2- 172.21.15.16:0/3643088106 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a4415b280 0x7f3a441f5800 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c007e40 tx=0x7f3a3c00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:04.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.497+0000 7f3a3affd700 1 -- 172.21.15.16:0/3643088106 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a3c00ed30 con 0x7f3a4415b280 2023-10-16T12:11:04.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.497+0000 7f3a3affd700 1 -- 172.21.15.16:0/3643088106 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a3c014420 con 0x7f3a4415b280 2023-10-16T12:11:04.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.497+0000 7f3a48966700 1 -- 172.21.15.16:0/3643088106 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a440ae700 con 0x7f3a4415b280 2023-10-16T12:11:04.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.497+0000 7f3a3affd700 1 -- 172.21.15.16:0/3643088106 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a3c0147d0 con 0x7f3a4415b280 2023-10-16T12:11:04.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.497+0000 7f3a48966700 1 -- 172.21.15.16:0/3643088106 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a440aecc0 con 0x7f3a4415b280 2023-10-16T12:11:04.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.498+0000 7f3a48966700 1 -- 172.21.15.16:0/3643088106 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a4404ebb0 con 0x7f3a4415b280 2023-10-16T12:11:04.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.498+0000 7f3a3affd700 1 -- 172.21.15.16:0/3643088106 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f3a3c014930 con 0x7f3a4415b280 2023-10-16T12:11:04.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.499+0000 7f3a3affd700 1 --2- 172.21.15.16:0/3643088106 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f3a2406af00 0x7f3a2406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:04.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.499+0000 7f3a3affd700 1 -- 172.21.15.16:0/3643088106 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f3a3c089f80 con 0x7f3a4415b280 2023-10-16T12:11:04.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.499+0000 7f3a4259c700 1 --2- 172.21.15.16:0/3643088106 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f3a2406af00 0x7f3a2406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:04.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.499+0000 7f3a4259c700 1 --2- 172.21.15.16:0/3643088106 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f3a2406af00 0x7f3a2406d3c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3a2c007fd0 tx=0x7f3a2c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:04.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.503+0000 7f3a3affd700 1 -- 172.21.15.16:0/3643088106 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3a3c0bcfe0 con 0x7f3a4415b280 2023-10-16T12:11:04.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.702+0000 7f3a48966700 1 -- 172.21.15.16:0/3643088106 --> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3a4415e1e0 con 0x7f3a2406af00 2023-10-16T12:11:04.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.711+0000 7f3a3affd700 1 -- 172.21.15.16:0/3643088106 <== mgr.4104 v2:172.21.15.16:6832/365921436 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+345260 (secure 0 0 0) 0x7f3a2c001cc0 con 0x7f3a2406af00 2023-10-16T12:11:04.717 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:11:04.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.715+0000 7f3a48966700 1 -- 172.21.15.16:0/3643088106 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f3a2406af00 msgr2=0x7f3a2406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:04.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.715+0000 7f3a48966700 1 --2- 172.21.15.16:0/3643088106 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f3a2406af00 0x7f3a2406d3c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3a2c007fd0 tx=0x7f3a2c009040 comp rx=0 tx=0).stop 2023-10-16T12:11:04.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.715+0000 7f3a48966700 1 -- 172.21.15.16:0/3643088106 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a4415b280 msgr2=0x7f3a441f5800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:04.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.715+0000 7f3a48966700 1 --2- 172.21.15.16:0/3643088106 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a4415b280 0x7f3a441f5800 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c007e40 tx=0x7f3a3c00e3c0 comp rx=0 tx=0).stop 2023-10-16T12:11:04.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.715+0000 7f3a48966700 1 -- 172.21.15.16:0/3643088106 shutdown_connections 2023-10-16T12:11:04.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.715+0000 7f3a48966700 1 --2- 172.21.15.16:0/3643088106 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f3a2406af00 0x7f3a2406d3c0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:04.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.715+0000 7f3a48966700 1 --2- 172.21.15.16:0/3643088106 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a4415b280 0x7f3a441f5800 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:04.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.715+0000 7f3a48966700 1 --2- 172.21.15.16:0/3643088106 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3a4406a7e0 0x7f3a440c59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:04.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.715+0000 7f3a48966700 1 --2- 172.21.15.16:0/3643088106 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3a440cc8a0 0x7f3a44071c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:04.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.715+0000 7f3a48966700 1 -- 172.21.15.16:0/3643088106 >> 172.21.15.16:0/3643088106 conn(0x7f3a440b1fd0 msgr2=0x7f3a440726e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:04.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.715+0000 7f3a48966700 1 -- 172.21.15.16:0/3643088106 shutdown_connections 2023-10-16T12:11:04.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.715+0000 7f3a48966700 1 -- 172.21.15.16:0/3643088106 wait complete. 2023-10-16T12:11:04.722 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-10-16T12:11:04.729 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2023-10-16T12:11:03.100385+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219472,"kb_used_data":3304,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755248,"statfs":{"total":773094113280,"available":772869373952,"internally_reserved":0,"allocated":3383296,"data_stored":2025624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12740,"internal_metadata":221236796},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2230,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002987"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.797013+0000","last_change":"2023-10-16T12:10:39.641670+0000","last_active":"2023-10-16T12:10:43.797013+0000","last_peered":"2023-10-16T12:10:43.797013+0000","last_clean":"2023-10-16T12:10:43.797013+0000","last_became_active":"2023-10-16T12:10:39.639911+0000","last_became_peered":"2023-10-16T12:10:39.639911+0000","last_unstale":"2023-10-16T12:10:43.797013+0000","last_undegraded":"2023-10-16T12:10:43.797013+0000","last_fullsized":"2023-10-16T12:10:43.797013+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:29:43.494490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.656566+0000","last_change":"2023-10-16T12:10:39.637937+0000","last_active":"2023-10-16T12:10:43.656566+0000","last_peered":"2023-10-16T12:10:43.656566+0000","last_clean":"2023-10-16T12:10:43.656566+0000","last_became_active":"2023-10-16T12:10:39.637595+0000","last_became_peered":"2023-10-16T12:10:39.637595+0000","last_unstale":"2023-10-16T12:10:43.656566+0000","last_undegraded":"2023-10-16T12:10:43.656566+0000","last_fullsized":"2023-10-16T12:10:43.656566+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:03:12.325906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646168+0000","last_change":"2023-10-16T12:10:39.630003+0000","last_active":"2023-10-16T12:10:43.646168+0000","last_peered":"2023-10-16T12:10:43.646168+0000","last_clean":"2023-10-16T12:10:43.646168+0000","last_became_active":"2023-10-16T12:10:39.627471+0000","last_became_peered":"2023-10-16T12:10:39.627471+0000","last_unstale":"2023-10-16T12:10:43.646168+0000","last_undegraded":"2023-10-16T12:10:43.646168+0000","last_fullsized":"2023-10-16T12:10:43.646168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:33:20.807621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646890+0000","last_change":"2023-10-16T12:10:39.645834+0000","last_active":"2023-10-16T12:10:43.646890+0000","last_peered":"2023-10-16T12:10:43.646890+0000","last_clean":"2023-10-16T12:10:43.646890+0000","last_became_active":"2023-10-16T12:10:39.645631+0000","last_became_peered":"2023-10-16T12:10:39.645631+0000","last_unstale":"2023-10-16T12:10:43.646890+0000","last_undegraded":"2023-10-16T12:10:43.646890+0000","last_fullsized":"2023-10-16T12:10:43.646890+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:14:00.725551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.794209+0000","last_change":"2023-10-16T12:10:39.642967+0000","last_active":"2023-10-16T12:10:43.794209+0000","last_peered":"2023-10-16T12:10:43.794209+0000","last_clean":"2023-10-16T12:10:43.794209+0000","last_became_active":"2023-10-16T12:10:39.642015+0000","last_became_peered":"2023-10-16T12:10:39.642015+0000","last_unstale":"2023-10-16T12:10:43.794209+0000","last_undegraded":"2023-10-16T12:10:43.794209+0000","last_fullsized":"2023-10-16T12:10:43.794209+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:10:19.840696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646377+0000","last_change":"2023-10-16T12:10:39.648342+0000","last_active":"2023-10-16T12:10:43.646377+0000","last_peered":"2023-10-16T12:10:43.646377+0000","last_clean":"2023-10-16T12:10:43.646377+0000","last_became_active":"2023-10-16T12:10:39.648046+0000","last_became_peered":"2023-10-16T12:10:39.648046+0000","last_unstale":"2023-10-16T12:10:43.646377+0000","last_undegraded":"2023-10-16T12:10:43.646377+0000","last_fullsized":"2023-10-16T12:10:43.646377+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:27:44.016376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.793686+0000","last_change":"2023-10-16T12:10:39.640757+0000","last_active":"2023-10-16T12:10:43.793686+0000","last_peered":"2023-10-16T12:10:43.793686+0000","last_clean":"2023-10-16T12:10:43.793686+0000","last_became_active":"2023-10-16T12:10:39.639250+0000","last_became_peered":"2023-10-16T12:10:39.639250+0000","last_unstale":"2023-10-16T12:10:43.793686+0000","last_undegraded":"2023-10-16T12:10:43.793686+0000","last_fullsized":"2023-10-16T12:10:43.793686+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:59:08.633021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.647118+0000","last_change":"2023-10-16T12:10:39.643535+0000","last_active":"2023-10-16T12:10:43.647118+0000","last_peered":"2023-10-16T12:10:43.647118+0000","last_clean":"2023-10-16T12:10:43.647118+0000","last_became_active":"2023-10-16T12:10:39.643242+0000","last_became_peered":"2023-10-16T12:10:39.643242+0000","last_unstale":"2023-10-16T12:10:43.647118+0000","last_undegraded":"2023-10-16T12:10:43.647118+0000","last_fullsized":"2023-10-16T12:10:43.647118+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:49:21.678699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.796912+0000","last_change":"2023-10-16T12:10:39.645759+0000","last_active":"2023-10-16T12:10:43.796912+0000","last_peered":"2023-10-16T12:10:43.796912+0000","last_clean":"2023-10-16T12:10:43.796912+0000","last_became_active":"2023-10-16T12:10:39.642863+0000","last_became_peered":"2023-10-16T12:10:39.642863+0000","last_unstale":"2023-10-16T12:10:43.796912+0000","last_undegraded":"2023-10-16T12:10:43.796912+0000","last_fullsized":"2023-10-16T12:10:43.796912+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:40:23.467426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.650642+0000","last_change":"2023-10-16T12:10:39.641392+0000","last_active":"2023-10-16T12:10:43.650642+0000","last_peered":"2023-10-16T12:10:43.650642+0000","last_clean":"2023-10-16T12:10:43.650642+0000","last_became_active":"2023-10-16T12:10:39.641111+0000","last_became_peered":"2023-10-16T12:10:39.641111+0000","last_unstale":"2023-10-16T12:10:43.650642+0000","last_undegraded":"2023-10-16T12:10:43.650642+0000","last_fullsized":"2023-10-16T12:10:43.650642+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:38:01.595985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.650275+0000","last_change":"2023-10-16T12:10:39.642376+0000","last_active":"2023-10-16T12:10:43.650275+0000","last_peered":"2023-10-16T12:10:43.650275+0000","last_clean":"2023-10-16T12:10:43.650275+0000","last_became_active":"2023-10-16T12:10:39.642086+0000","last_became_peered":"2023-10-16T12:10:39.642086+0000","last_unstale":"2023-10-16T12:10:43.650275+0000","last_undegraded":"2023-10-16T12:10:43.650275+0000","last_fullsized":"2023-10-16T12:10:43.650275+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:05:05.699350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.648829+0000","last_change":"2023-10-16T12:10:39.637561+0000","last_active":"2023-10-16T12:10:43.648829+0000","last_peered":"2023-10-16T12:10:43.648829+0000","last_clean":"2023-10-16T12:10:43.648829+0000","last_became_active":"2023-10-16T12:10:39.637109+0000","last_became_peered":"2023-10-16T12:10:39.637109+0000","last_unstale":"2023-10-16T12:10:43.648829+0000","last_undegraded":"2023-10-16T12:10:43.648829+0000","last_fullsized":"2023-10-16T12:10:43.648829+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:59:48.851909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.645820+0000","last_change":"2023-10-16T12:10:39.631791+0000","last_active":"2023-10-16T12:10:43.645820+0000","last_peered":"2023-10-16T12:10:43.645820+0000","last_clean":"2023-10-16T12:10:43.645820+0000","last_became_active":"2023-10-16T12:10:39.631245+0000","last_became_peered":"2023-10-16T12:10:39.631245+0000","last_unstale":"2023-10-16T12:10:43.645820+0000","last_undegraded":"2023-10-16T12:10:43.645820+0000","last_fullsized":"2023-10-16T12:10:43.645820+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:05:05.699350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646175+0000","last_change":"2023-10-16T12:10:39.631112+0000","last_active":"2023-10-16T12:10:43.646175+0000","last_peered":"2023-10-16T12:10:43.646175+0000","last_clean":"2023-10-16T12:10:43.646175+0000","last_became_active":"2023-10-16T12:10:39.630312+0000","last_became_peered":"2023-10-16T12:10:39.630312+0000","last_unstale":"2023-10-16T12:10:43.646175+0000","last_undegraded":"2023-10-16T12:10:43.646175+0000","last_fullsized":"2023-10-16T12:10:43.646175+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:04:48.768318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.653241+0000","last_change":"2023-10-16T12:10:39.642725+0000","last_active":"2023-10-16T12:10:43.653241+0000","last_peered":"2023-10-16T12:10:43.653241+0000","last_clean":"2023-10-16T12:10:43.653241+0000","last_became_active":"2023-10-16T12:10:39.642437+0000","last_became_peered":"2023-10-16T12:10:39.642437+0000","last_unstale":"2023-10-16T12:10:43.653241+0000","last_undegraded":"2023-10-16T12:10:43.653241+0000","last_fullsized":"2023-10-16T12:10:43.653241+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:46:02.593591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.652749+0000","last_change":"2023-10-16T12:10:39.641780+0000","last_active":"2023-10-16T12:10:43.652749+0000","last_peered":"2023-10-16T12:10:43.652749+0000","last_clean":"2023-10-16T12:10:43.652749+0000","last_became_active":"2023-10-16T12:10:39.641275+0000","last_became_peered":"2023-10-16T12:10:39.641275+0000","last_unstale":"2023-10-16T12:10:43.652749+0000","last_undegraded":"2023-10-16T12:10:43.652749+0000","last_fullsized":"2023-10-16T12:10:43.652749+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:38:01.595985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.647146+0000","last_change":"2023-10-16T12:10:39.641557+0000","last_active":"2023-10-16T12:10:43.647146+0000","last_peered":"2023-10-16T12:10:43.647146+0000","last_clean":"2023-10-16T12:10:43.647146+0000","last_became_active":"2023-10-16T12:10:39.641242+0000","last_became_peered":"2023-10-16T12:10:39.641242+0000","last_unstale":"2023-10-16T12:10:43.647146+0000","last_undegraded":"2023-10-16T12:10:43.647146+0000","last_fullsized":"2023-10-16T12:10:43.647146+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:10:19.840696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.642631+0000","last_change":"2023-10-16T12:10:39.638264+0000","last_active":"2023-10-16T12:10:43.642631+0000","last_peered":"2023-10-16T12:10:43.642631+0000","last_clean":"2023-10-16T12:10:43.642631+0000","last_became_active":"2023-10-16T12:10:39.637959+0000","last_became_peered":"2023-10-16T12:10:39.637959+0000","last_unstale":"2023-10-16T12:10:43.642631+0000","last_undegraded":"2023-10-16T12:10:43.642631+0000","last_fullsized":"2023-10-16T12:10:43.642631+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:27:06.186345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.796665+0000","last_change":"2023-10-16T12:10:39.646694+0000","last_active":"2023-10-16T12:10:43.796665+0000","last_peered":"2023-10-16T12:10:43.796665+0000","last_clean":"2023-10-16T12:10:43.796665+0000","last_became_active":"2023-10-16T12:10:39.643245+0000","last_became_peered":"2023-10-16T12:10:39.643245+0000","last_unstale":"2023-10-16T12:10:43.796665+0000","last_undegraded":"2023-10-16T12:10:43.796665+0000","last_fullsized":"2023-10-16T12:10:43.796665+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:13:03.744288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:44.684038+0000","last_change":"2023-10-16T12:10:43.800090+0000","last_active":"2023-10-16T12:10:44.684038+0000","last_peered":"2023-10-16T12:10:44.684038+0000","last_clean":"2023-10-16T12:10:44.684038+0000","last_became_active":"2023-10-16T12:10:43.799746+0000","last_became_peered":"2023-10-16T12:10:43.799746+0000","last_unstale":"2023-10-16T12:10:44.684038+0000","last_undegraded":"2023-10-16T12:10:44.684038+0000","last_fullsized":"2023-10-16T12:10:44.684038+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:46:18.137970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646491+0000","last_change":"2023-10-16T12:10:38.620314+0000","last_active":"2023-10-16T12:10:43.646491+0000","last_peered":"2023-10-16T12:10:43.646491+0000","last_clean":"2023-10-16T12:10:43.646491+0000","last_became_active":"2023-10-16T12:10:38.620036+0000","last_became_peered":"2023-10-16T12:10:38.620036+0000","last_unstale":"2023-10-16T12:10:43.646491+0000","last_undegraded":"2023-10-16T12:10:43.646491+0000","last_fullsized":"2023-10-16T12:10:43.646491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:27:30.532335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.645988+0000","last_change":"2023-10-16T12:10:39.638746+0000","last_active":"2023-10-16T12:10:43.645988+0000","last_peered":"2023-10-16T12:10:43.645988+0000","last_clean":"2023-10-16T12:10:43.645988+0000","last_became_active":"2023-10-16T12:10:39.637152+0000","last_became_peered":"2023-10-16T12:10:39.637152+0000","last_unstale":"2023-10-16T12:10:43.645988+0000","last_undegraded":"2023-10-16T12:10:43.645988+0000","last_fullsized":"2023-10-16T12:10:43.645988+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:30:10.006613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:45.375335+0000","last_change":"2023-10-16T12:10:43.662213+0000","last_active":"2023-10-16T12:10:45.375335+0000","last_peered":"2023-10-16T12:10:45.375335+0000","last_clean":"2023-10-16T12:10:45.375335+0000","last_became_active":"2023-10-16T12:10:43.661755+0000","last_became_peered":"2023-10-16T12:10:43.661755+0000","last_unstale":"2023-10-16T12:10:45.375335+0000","last_undegraded":"2023-10-16T12:10:45.375335+0000","last_fullsized":"2023-10-16T12:10:45.375335+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:30:14.045561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646110+0000","last_change":"2023-10-16T12:10:38.625571+0000","last_active":"2023-10-16T12:10:43.646110+0000","last_peered":"2023-10-16T12:10:43.646110+0000","last_clean":"2023-10-16T12:10:43.646110+0000","last_became_active":"2023-10-16T12:10:38.625353+0000","last_became_peered":"2023-10-16T12:10:38.625353+0000","last_unstale":"2023-10-16T12:10:43.646110+0000","last_undegraded":"2023-10-16T12:10:43.646110+0000","last_fullsized":"2023-10-16T12:10:43.646110+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:39:25.396697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646379+0000","last_change":"2023-10-16T12:10:39.641482+0000","last_active":"2023-10-16T12:10:43.646379+0000","last_peered":"2023-10-16T12:10:43.646379+0000","last_clean":"2023-10-16T12:10:43.646379+0000","last_became_active":"2023-10-16T12:10:39.641168+0000","last_became_peered":"2023-10-16T12:10:39.641168+0000","last_unstale":"2023-10-16T12:10:43.646379+0000","last_undegraded":"2023-10-16T12:10:43.646379+0000","last_fullsized":"2023-10-16T12:10:43.646379+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:47:52.332290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:45.376007+0000","last_change":"2023-10-16T12:10:43.668484+0000","last_active":"2023-10-16T12:10:45.376007+0000","last_peered":"2023-10-16T12:10:45.376007+0000","last_clean":"2023-10-16T12:10:45.376007+0000","last_became_active":"2023-10-16T12:10:43.667862+0000","last_became_peered":"2023-10-16T12:10:43.667862+0000","last_unstale":"2023-10-16T12:10:45.376007+0000","last_undegraded":"2023-10-16T12:10:45.376007+0000","last_fullsized":"2023-10-16T12:10:45.376007+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:34:54.098418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.796065+0000","last_change":"2023-10-16T12:10:38.623974+0000","last_active":"2023-10-16T12:10:43.796065+0000","last_peered":"2023-10-16T12:10:43.796065+0000","last_clean":"2023-10-16T12:10:43.796065+0000","last_became_active":"2023-10-16T12:10:38.623532+0000","last_became_peered":"2023-10-16T12:10:38.623532+0000","last_unstale":"2023-10-16T12:10:43.796065+0000","last_undegraded":"2023-10-16T12:10:43.796065+0000","last_fullsized":"2023-10-16T12:10:43.796065+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:03:11.321777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646640+0000","last_change":"2023-10-16T12:10:39.651203+0000","last_active":"2023-10-16T12:10:43.646640+0000","last_peered":"2023-10-16T12:10:43.646640+0000","last_clean":"2023-10-16T12:10:43.646640+0000","last_became_active":"2023-10-16T12:10:39.651001+0000","last_became_peered":"2023-10-16T12:10:39.651001+0000","last_unstale":"2023-10-16T12:10:43.646640+0000","last_undegraded":"2023-10-16T12:10:43.646640+0000","last_fullsized":"2023-10-16T12:10:43.646640+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:34:50.059469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:45.375823+0000","last_change":"2023-10-16T12:10:43.659820+0000","last_active":"2023-10-16T12:10:45.375823+0000","last_peered":"2023-10-16T12:10:45.375823+0000","last_clean":"2023-10-16T12:10:45.375823+0000","last_became_active":"2023-10-16T12:10:43.659301+0000","last_became_peered":"2023-10-16T12:10:43.659301+0000","last_unstale":"2023-10-16T12:10:45.375823+0000","last_undegraded":"2023-10-16T12:10:45.375823+0000","last_fullsized":"2023-10-16T12:10:45.375823+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:59:12.671969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.795902+0000","last_change":"2023-10-16T12:10:38.622949+0000","last_active":"2023-10-16T12:10:43.795902+0000","last_peered":"2023-10-16T12:10:43.795902+0000","last_clean":"2023-10-16T12:10:43.795902+0000","last_became_active":"2023-10-16T12:10:38.622666+0000","last_became_peered":"2023-10-16T12:10:38.622666+0000","last_unstale":"2023-10-16T12:10:43.795902+0000","last_undegraded":"2023-10-16T12:10:43.795902+0000","last_fullsized":"2023-10-16T12:10:43.795902+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:27:30.532335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.653416+0000","last_change":"2023-10-16T12:10:39.650483+0000","last_active":"2023-10-16T12:10:43.653416+0000","last_peered":"2023-10-16T12:10:43.653416+0000","last_clean":"2023-10-16T12:10:43.653416+0000","last_became_active":"2023-10-16T12:10:39.650230+0000","last_became_peered":"2023-10-16T12:10:39.650230+0000","last_unstale":"2023-10-16T12:10:43.653416+0000","last_undegraded":"2023-10-16T12:10:43.653416+0000","last_fullsized":"2023-10-16T12:10:43.653416+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:38:53.624950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:44.684035+0000","last_change":"2023-10-16T12:10:43.667121+0000","last_active":"2023-10-16T12:10:44.684035+0000","last_peered":"2023-10-16T12:10:44.684035+0000","last_clean":"2023-10-16T12:10:44.684035+0000","last_became_active":"2023-10-16T12:10:43.666708+0000","last_became_peered":"2023-10-16T12:10:43.666708+0000","last_unstale":"2023-10-16T12:10:44.684035+0000","last_undegraded":"2023-10-16T12:10:44.684035+0000","last_fullsized":"2023-10-16T12:10:44.684035+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:19:32.391115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.645944+0000","last_change":"2023-10-16T12:10:38.623302+0000","last_active":"2023-10-16T12:10:43.645944+0000","last_peered":"2023-10-16T12:10:43.645944+0000","last_clean":"2023-10-16T12:10:43.645944+0000","last_became_active":"2023-10-16T12:10:38.621867+0000","last_became_peered":"2023-10-16T12:10:38.621867+0000","last_unstale":"2023-10-16T12:10:43.645944+0000","last_undegraded":"2023-10-16T12:10:43.645944+0000","last_fullsized":"2023-10-16T12:10:43.645944+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:59:30.615293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.649101+0000","last_change":"2023-10-16T12:10:39.639254+0000","last_active":"2023-10-16T12:10:43.649101+0000","last_peered":"2023-10-16T12:10:43.649101+0000","last_clean":"2023-10-16T12:10:43.649101+0000","last_became_active":"2023-10-16T12:10:39.638956+0000","last_became_peered":"2023-10-16T12:10:39.638956+0000","last_unstale":"2023-10-16T12:10:43.649101+0000","last_undegraded":"2023-10-16T12:10:43.649101+0000","last_fullsized":"2023-10-16T12:10:43.649101+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:27:31.536464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:45.374724+0000","last_change":"2023-10-16T12:10:43.667499+0000","last_active":"2023-10-16T12:10:45.374724+0000","last_peered":"2023-10-16T12:10:45.374724+0000","last_clean":"2023-10-16T12:10:45.374724+0000","last_became_active":"2023-10-16T12:10:43.667116+0000","last_became_peered":"2023-10-16T12:10:43.667116+0000","last_unstale":"2023-10-16T12:10:45.374724+0000","last_undegraded":"2023-10-16T12:10:45.374724+0000","last_fullsized":"2023-10-16T12:10:45.374724+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:59:35.658370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.651387+0000","last_change":"2023-10-16T12:10:38.619514+0000","last_active":"2023-10-16T12:10:43.651387+0000","last_peered":"2023-10-16T12:10:43.651387+0000","last_clean":"2023-10-16T12:10:43.651387+0000","last_became_active":"2023-10-16T12:10:38.619191+0000","last_became_peered":"2023-10-16T12:10:38.619191+0000","last_unstale":"2023-10-16T12:10:43.651387+0000","last_undegraded":"2023-10-16T12:10:43.651387+0000","last_fullsized":"2023-10-16T12:10:43.651387+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:39:25.396697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.649794+0000","last_change":"2023-10-16T12:10:39.642796+0000","last_active":"2023-10-16T12:10:43.649794+0000","last_peered":"2023-10-16T12:10:43.649794+0000","last_clean":"2023-10-16T12:10:43.649794+0000","last_became_active":"2023-10-16T12:10:39.642536+0000","last_became_peered":"2023-10-16T12:10:39.642536+0000","last_unstale":"2023-10-16T12:10:43.649794+0000","last_undegraded":"2023-10-16T12:10:43.649794+0000","last_fullsized":"2023-10-16T12:10:43.649794+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:27:44.016376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:44.874851+0000","last_change":"2023-10-16T12:10:43.662296+0000","last_active":"2023-10-16T12:10:44.874851+0000","last_peered":"2023-10-16T12:10:44.874851+0000","last_clean":"2023-10-16T12:10:44.874851+0000","last_became_active":"2023-10-16T12:10:43.661893+0000","last_became_peered":"2023-10-16T12:10:43.661893+0000","last_unstale":"2023-10-16T12:10:44.874851+0000","last_undegraded":"2023-10-16T12:10:44.874851+0000","last_fullsized":"2023-10-16T12:10:44.874851+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:14:04.764499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646516+0000","last_change":"2023-10-16T12:10:38.623322+0000","last_active":"2023-10-16T12:10:43.646516+0000","last_peered":"2023-10-16T12:10:43.646516+0000","last_clean":"2023-10-16T12:10:43.646516+0000","last_became_active":"2023-10-16T12:10:38.622826+0000","last_became_peered":"2023-10-16T12:10:38.622826+0000","last_unstale":"2023-10-16T12:10:43.646516+0000","last_undegraded":"2023-10-16T12:10:43.646516+0000","last_fullsized":"2023-10-16T12:10:43.646516+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:30:56.479201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646312+0000","last_change":"2023-10-16T12:10:39.624731+0000","last_active":"2023-10-16T12:10:43.646312+0000","last_peered":"2023-10-16T12:10:43.646312+0000","last_clean":"2023-10-16T12:10:43.646312+0000","last_became_active":"2023-10-16T12:10:39.624399+0000","last_became_peered":"2023-10-16T12:10:39.624399+0000","last_unstale":"2023-10-16T12:10:43.646312+0000","last_undegraded":"2023-10-16T12:10:43.646312+0000","last_fullsized":"2023-10-16T12:10:43.646312+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:47:52.332290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:45.375461+0000","last_change":"2023-10-16T12:10:43.798638+0000","last_active":"2023-10-16T12:10:45.375461+0000","last_peered":"2023-10-16T12:10:45.375461+0000","last_clean":"2023-10-16T12:10:45.375461+0000","last_became_active":"2023-10-16T12:10:43.797973+0000","last_became_peered":"2023-10-16T12:10:43.797973+0000","last_unstale":"2023-10-16T12:10:45.375461+0000","last_undegraded":"2023-10-16T12:10:45.375461+0000","last_fullsized":"2023-10-16T12:10:45.375461+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:29:47.533439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.652773+0000","last_change":"2023-10-16T12:10:38.627818+0000","last_active":"2023-10-16T12:10:43.652773+0000","last_peered":"2023-10-16T12:10:43.652773+0000","last_clean":"2023-10-16T12:10:43.652773+0000","last_became_active":"2023-10-16T12:10:38.627492+0000","last_became_peered":"2023-10-16T12:10:38.627492+0000","last_unstale":"2023-10-16T12:10:43.652773+0000","last_undegraded":"2023-10-16T12:10:43.652773+0000","last_fullsized":"2023-10-16T12:10:43.652773+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:05:04.695222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.653456+0000","last_change":"2023-10-16T12:10:39.633123+0000","last_active":"2023-10-16T12:10:43.653456+0000","last_peered":"2023-10-16T12:10:43.653456+0000","last_clean":"2023-10-16T12:10:43.653456+0000","last_became_active":"2023-10-16T12:10:39.632868+0000","last_became_peered":"2023-10-16T12:10:39.632868+0000","last_unstale":"2023-10-16T12:10:43.653456+0000","last_undegraded":"2023-10-16T12:10:43.653456+0000","last_fullsized":"2023-10-16T12:10:43.653456+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:33:20.807621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:44.683898+0000","last_change":"2023-10-16T12:10:43.794748+0000","last_active":"2023-10-16T12:10:44.683898+0000","last_peered":"2023-10-16T12:10:44.683898+0000","last_clean":"2023-10-16T12:10:44.683898+0000","last_became_active":"2023-10-16T12:10:43.794362+0000","last_became_peered":"2023-10-16T12:10:43.794362+0000","last_unstale":"2023-10-16T12:10:44.683898+0000","last_undegraded":"2023-10-16T12:10:44.683898+0000","last_fullsized":"2023-10-16T12:10:44.683898+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:57:20.445348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.652120+0000","last_change":"2023-10-16T12:10:38.619590+0000","last_active":"2023-10-16T12:10:43.652120+0000","last_peered":"2023-10-16T12:10:43.652120+0000","last_clean":"2023-10-16T12:10:43.652120+0000","last_became_active":"2023-10-16T12:10:38.618984+0000","last_became_peered":"2023-10-16T12:10:38.618984+0000","last_unstale":"2023-10-16T12:10:43.652120+0000","last_undegraded":"2023-10-16T12:10:43.652120+0000","last_fullsized":"2023-10-16T12:10:43.652120+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:27:05.182216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646297+0000","last_change":"2023-10-16T12:10:39.644469+0000","last_active":"2023-10-16T12:10:43.646297+0000","last_peered":"2023-10-16T12:10:43.646297+0000","last_clean":"2023-10-16T12:10:43.646297+0000","last_became_active":"2023-10-16T12:10:39.642359+0000","last_became_peered":"2023-10-16T12:10:39.642359+0000","last_unstale":"2023-10-16T12:10:43.646297+0000","last_undegraded":"2023-10-16T12:10:43.646297+0000","last_fullsized":"2023-10-16T12:10:43.646297+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:30:10.006613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:44.684242+0000","last_change":"2023-10-16T12:10:43.667401+0000","last_active":"2023-10-16T12:10:44.684242+0000","last_peered":"2023-10-16T12:10:44.684242+0000","last_clean":"2023-10-16T12:10:44.684242+0000","last_became_active":"2023-10-16T12:10:43.666829+0000","last_became_peered":"2023-10-16T12:10:43.666829+0000","last_unstale":"2023-10-16T12:10:44.684242+0000","last_undegraded":"2023-10-16T12:10:44.684242+0000","last_fullsized":"2023-10-16T12:10:44.684242+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:13:48.034204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646585+0000","last_change":"2023-10-16T12:10:38.627372+0000","last_active":"2023-10-16T12:10:43.646585+0000","last_peered":"2023-10-16T12:10:43.646585+0000","last_clean":"2023-10-16T12:10:43.646585+0000","last_became_active":"2023-10-16T12:10:38.622500+0000","last_became_peered":"2023-10-16T12:10:38.622500+0000","last_unstale":"2023-10-16T12:10:43.646585+0000","last_undegraded":"2023-10-16T12:10:43.646585+0000","last_fullsized":"2023-10-16T12:10:43.646585+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:57:56.978587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646591+0000","last_change":"2023-10-16T12:10:39.641571+0000","last_active":"2023-10-16T12:10:43.646591+0000","last_peered":"2023-10-16T12:10:43.646591+0000","last_clean":"2023-10-16T12:10:43.646591+0000","last_became_active":"2023-10-16T12:10:39.641228+0000","last_became_peered":"2023-10-16T12:10:39.641228+0000","last_unstale":"2023-10-16T12:10:43.646591+0000","last_undegraded":"2023-10-16T12:10:43.646591+0000","last_fullsized":"2023-10-16T12:10:43.646591+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:38:01.595985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:45.376570+0000","last_change":"2023-10-16T12:10:43.671066+0000","last_active":"2023-10-16T12:10:45.376570+0000","last_peered":"2023-10-16T12:10:45.376570+0000","last_clean":"2023-10-16T12:10:45.376570+0000","last_became_active":"2023-10-16T12:10:43.670693+0000","last_became_peered":"2023-10-16T12:10:43.670693+0000","last_unstale":"2023-10-16T12:10:45.376570+0000","last_undegraded":"2023-10-16T12:10:45.376570+0000","last_fullsized":"2023-10-16T12:10:45.376570+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:24:19.577814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.654541+0000","last_change":"2023-10-16T12:10:38.623191+0000","last_active":"2023-10-16T12:10:43.654541+0000","last_peered":"2023-10-16T12:10:43.654541+0000","last_clean":"2023-10-16T12:10:43.654541+0000","last_became_active":"2023-10-16T12:10:38.622876+0000","last_became_peered":"2023-10-16T12:10:38.622876+0000","last_unstale":"2023-10-16T12:10:43.654541+0000","last_undegraded":"2023-10-16T12:10:43.654541+0000","last_fullsized":"2023-10-16T12:10:43.654541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:59:30.615293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.648686+0000","last_change":"2023-10-16T12:10:39.643461+0000","last_active":"2023-10-16T12:10:43.648686+0000","last_peered":"2023-10-16T12:10:43.648686+0000","last_clean":"2023-10-16T12:10:43.648686+0000","last_became_active":"2023-10-16T12:10:39.643148+0000","last_became_peered":"2023-10-16T12:10:39.643148+0000","last_unstale":"2023-10-16T12:10:43.648686+0000","last_undegraded":"2023-10-16T12:10:43.648686+0000","last_fullsized":"2023-10-16T12:10:43.648686+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:27:06.186345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:44.682345+0000","last_change":"2023-10-16T12:10:43.662674+0000","last_active":"2023-10-16T12:10:44.682345+0000","last_peered":"2023-10-16T12:10:44.682345+0000","last_clean":"2023-10-16T12:10:44.682345+0000","last_became_active":"2023-10-16T12:10:43.662249+0000","last_became_peered":"2023-10-16T12:10:43.662249+0000","last_unstale":"2023-10-16T12:10:44.682345+0000","last_undegraded":"2023-10-16T12:10:44.682345+0000","last_fullsized":"2023-10-16T12:10:44.682345+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:14:04.764499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"25'9","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:49.578853+0000","last_change":"2023-10-16T12:10:38.621924+0000","last_active":"2023-10-16T12:10:49.578853+0000","last_peered":"2023-10-16T12:10:49.578853+0000","last_clean":"2023-10-16T12:10:49.578853+0000","last_became_active":"2023-10-16T12:10:38.621621+0000","last_became_peered":"2023-10-16T12:10:38.621621+0000","last_unstale":"2023-10-16T12:10:49.578853+0000","last_undegraded":"2023-10-16T12:10:49.578853+0000","last_fullsized":"2023-10-16T12:10:49.578853+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:27:30.532335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.796277+0000","last_change":"2023-10-16T12:10:39.641165+0000","last_active":"2023-10-16T12:10:43.796277+0000","last_peered":"2023-10-16T12:10:43.796277+0000","last_clean":"2023-10-16T12:10:43.796277+0000","last_became_active":"2023-10-16T12:10:39.640241+0000","last_became_peered":"2023-10-16T12:10:39.640241+0000","last_unstale":"2023-10-16T12:10:43.796277+0000","last_undegraded":"2023-10-16T12:10:43.796277+0000","last_fullsized":"2023-10-16T12:10:43.796277+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:38:53.624950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:45.377704+0000","last_change":"2023-10-16T12:10:43.796949+0000","last_active":"2023-10-16T12:10:45.377704+0000","last_peered":"2023-10-16T12:10:45.377704+0000","last_clean":"2023-10-16T12:10:45.377704+0000","last_became_active":"2023-10-16T12:10:43.796518+0000","last_became_peered":"2023-10-16T12:10:43.796518+0000","last_unstale":"2023-10-16T12:10:45.377704+0000","last_undegraded":"2023-10-16T12:10:45.377704+0000","last_fullsized":"2023-10-16T12:10:45.377704+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:58:35.336755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.650051+0000","last_change":"2023-10-16T12:10:38.612552+0000","last_active":"2023-10-16T12:10:43.650051+0000","last_peered":"2023-10-16T12:10:43.650051+0000","last_clean":"2023-10-16T12:10:43.650051+0000","last_became_active":"2023-10-16T12:10:38.612221+0000","last_became_peered":"2023-10-16T12:10:38.612221+0000","last_unstale":"2023-10-16T12:10:43.650051+0000","last_undegraded":"2023-10-16T12:10:43.650051+0000","last_fullsized":"2023-10-16T12:10:43.650051+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:27:05.182216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646287+0000","last_change":"2023-10-16T12:10:39.638781+0000","last_active":"2023-10-16T12:10:43.646287+0000","last_peered":"2023-10-16T12:10:43.646287+0000","last_clean":"2023-10-16T12:10:43.646287+0000","last_became_active":"2023-10-16T12:10:39.638523+0000","last_became_peered":"2023-10-16T12:10:39.638523+0000","last_unstale":"2023-10-16T12:10:43.646287+0000","last_undegraded":"2023-10-16T12:10:43.646287+0000","last_fullsized":"2023-10-16T12:10:43.646287+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:05:05.699350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:44.683241+0000","last_change":"2023-10-16T12:10:43.794196+0000","last_active":"2023-10-16T12:10:44.683241+0000","last_peered":"2023-10-16T12:10:44.683241+0000","last_clean":"2023-10-16T12:10:44.683241+0000","last_became_active":"2023-10-16T12:10:43.793712+0000","last_became_peered":"2023-10-16T12:10:43.793712+0000","last_unstale":"2023-10-16T12:10:44.683241+0000","last_undegraded":"2023-10-16T12:10:44.683241+0000","last_fullsized":"2023-10-16T12:10:44.683241+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:40:27.506374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646125+0000","last_change":"2023-10-16T12:10:38.627403+0000","last_active":"2023-10-16T12:10:43.646125+0000","last_peered":"2023-10-16T12:10:43.646125+0000","last_clean":"2023-10-16T12:10:43.646125+0000","last_became_active":"2023-10-16T12:10:38.623013+0000","last_became_peered":"2023-10-16T12:10:38.623013+0000","last_unstale":"2023-10-16T12:10:43.646125+0000","last_undegraded":"2023-10-16T12:10:43.646125+0000","last_fullsized":"2023-10-16T12:10:43.646125+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:47:51.328162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.794292+0000","last_change":"2023-10-16T12:10:39.640179+0000","last_active":"2023-10-16T12:10:43.794292+0000","last_peered":"2023-10-16T12:10:43.794292+0000","last_clean":"2023-10-16T12:10:43.794292+0000","last_became_active":"2023-10-16T12:10:39.639843+0000","last_became_peered":"2023-10-16T12:10:39.639843+0000","last_unstale":"2023-10-16T12:10:43.794292+0000","last_undegraded":"2023-10-16T12:10:43.794292+0000","last_fullsized":"2023-10-16T12:10:43.794292+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:24:15.538866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:44.683431+0000","last_change":"2023-10-16T12:10:43.796968+0000","last_active":"2023-10-16T12:10:44.683431+0000","last_peered":"2023-10-16T12:10:44.683431+0000","last_clean":"2023-10-16T12:10:44.683431+0000","last_became_active":"2023-10-16T12:10:43.796562+0000","last_became_peered":"2023-10-16T12:10:43.796562+0000","last_unstale":"2023-10-16T12:10:44.683431+0000","last_undegraded":"2023-10-16T12:10:44.683431+0000","last_fullsized":"2023-10-16T12:10:44.683431+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:44:47.973077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.650016+0000","last_change":"2023-10-16T12:10:38.622066+0000","last_active":"2023-10-16T12:10:43.650016+0000","last_peered":"2023-10-16T12:10:43.650016+0000","last_clean":"2023-10-16T12:10:43.650016+0000","last_became_active":"2023-10-16T12:10:38.621453+0000","last_became_peered":"2023-10-16T12:10:38.621453+0000","last_unstale":"2023-10-16T12:10:43.650016+0000","last_undegraded":"2023-10-16T12:10:43.650016+0000","last_fullsized":"2023-10-16T12:10:43.650016+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:45:22.011420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.645479+0000","last_change":"2023-10-16T12:10:39.636432+0000","last_active":"2023-10-16T12:10:43.645479+0000","last_peered":"2023-10-16T12:10:43.645479+0000","last_clean":"2023-10-16T12:10:43.645479+0000","last_became_active":"2023-10-16T12:10:39.636136+0000","last_became_peered":"2023-10-16T12:10:39.636136+0000","last_unstale":"2023-10-16T12:10:43.645479+0000","last_undegraded":"2023-10-16T12:10:43.645479+0000","last_fullsized":"2023-10-16T12:10:43.645479+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:46:02.593591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:45.375145+0000","last_change":"2023-10-16T12:10:43.661450+0000","last_active":"2023-10-16T12:10:45.375145+0000","last_peered":"2023-10-16T12:10:45.375145+0000","last_clean":"2023-10-16T12:10:45.375145+0000","last_became_active":"2023-10-16T12:10:43.660931+0000","last_became_peered":"2023-10-16T12:10:43.660931+0000","last_unstale":"2023-10-16T12:10:45.375145+0000","last_undegraded":"2023-10-16T12:10:45.375145+0000","last_fullsized":"2023-10-16T12:10:45.375145+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:53:08.121149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.642296+0000","last_change":"2023-10-16T12:10:38.626933+0000","last_active":"2023-10-16T12:10:43.642296+0000","last_peered":"2023-10-16T12:10:43.642296+0000","last_clean":"2023-10-16T12:10:43.642296+0000","last_became_active":"2023-10-16T12:10:38.626643+0000","last_became_peered":"2023-10-16T12:10:38.626643+0000","last_unstale":"2023-10-16T12:10:43.642296+0000","last_undegraded":"2023-10-16T12:10:43.642296+0000","last_fullsized":"2023-10-16T12:10:43.642296+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:01:27.462901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.794002+0000","last_change":"2023-10-16T12:10:39.648098+0000","last_active":"2023-10-16T12:10:43.794002+0000","last_peered":"2023-10-16T12:10:43.794002+0000","last_clean":"2023-10-16T12:10:43.794002+0000","last_became_active":"2023-10-16T12:10:39.643195+0000","last_became_peered":"2023-10-16T12:10:39.643195+0000","last_unstale":"2023-10-16T12:10:43.794002+0000","last_undegraded":"2023-10-16T12:10:43.794002+0000","last_fullsized":"2023-10-16T12:10:43.794002+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:34:50.059469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:45.374887+0000","last_change":"2023-10-16T12:10:43.662584+0000","last_active":"2023-10-16T12:10:45.374887+0000","last_peered":"2023-10-16T12:10:45.374887+0000","last_clean":"2023-10-16T12:10:45.374887+0000","last_became_active":"2023-10-16T12:10:43.662164+0000","last_became_peered":"2023-10-16T12:10:43.662164+0000","last_unstale":"2023-10-16T12:10:45.374887+0000","last_undegraded":"2023-10-16T12:10:45.374887+0000","last_fullsized":"2023-10-16T12:10:45.374887+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:03:16.364854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.648873+0000","last_change":"2023-10-16T12:10:38.613102+0000","last_active":"2023-10-16T12:10:43.648873+0000","last_peered":"2023-10-16T12:10:43.648873+0000","last_clean":"2023-10-16T12:10:43.648873+0000","last_became_active":"2023-10-16T12:10:38.612596+0000","last_became_peered":"2023-10-16T12:10:38.612596+0000","last_unstale":"2023-10-16T12:10:43.648873+0000","last_undegraded":"2023-10-16T12:10:43.648873+0000","last_fullsized":"2023-10-16T12:10:43.648873+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:59:50.957247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.796815+0000","last_change":"2023-10-16T12:10:39.640141+0000","last_active":"2023-10-16T12:10:43.796815+0000","last_peered":"2023-10-16T12:10:43.796815+0000","last_clean":"2023-10-16T12:10:43.796815+0000","last_became_active":"2023-10-16T12:10:39.639770+0000","last_became_peered":"2023-10-16T12:10:39.639770+0000","last_unstale":"2023-10-16T12:10:43.796815+0000","last_undegraded":"2023-10-16T12:10:43.796815+0000","last_fullsized":"2023-10-16T12:10:43.796815+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:14:00.725551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:44.683486+0000","last_change":"2023-10-16T12:10:43.666858+0000","last_active":"2023-10-16T12:10:44.683486+0000","last_peered":"2023-10-16T12:10:44.683486+0000","last_clean":"2023-10-16T12:10:44.683486+0000","last_became_active":"2023-10-16T12:10:43.666461+0000","last_became_peered":"2023-10-16T12:10:43.666461+0000","last_unstale":"2023-10-16T12:10:44.683486+0000","last_undegraded":"2023-10-16T12:10:44.683486+0000","last_fullsized":"2023-10-16T12:10:44.683486+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:53:08.121149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.663314+0000","last_change":"2023-10-16T12:10:38.625538+0000","last_active":"2023-10-16T12:10:43.663314+0000","last_peered":"2023-10-16T12:10:43.663314+0000","last_clean":"2023-10-16T12:10:43.663314+0000","last_became_active":"2023-10-16T12:10:38.625227+0000","last_became_peered":"2023-10-16T12:10:38.625227+0000","last_unstale":"2023-10-16T12:10:43.663314+0000","last_undegraded":"2023-10-16T12:10:43.663314+0000","last_fullsized":"2023-10-16T12:10:43.663314+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:39:25.396697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.653596+0000","last_change":"2023-10-16T12:10:39.645373+0000","last_active":"2023-10-16T12:10:43.653596+0000","last_peered":"2023-10-16T12:10:43.653596+0000","last_clean":"2023-10-16T12:10:43.653596+0000","last_became_active":"2023-10-16T12:10:39.645083+0000","last_became_peered":"2023-10-16T12:10:39.645083+0000","last_unstale":"2023-10-16T12:10:43.653596+0000","last_undegraded":"2023-10-16T12:10:43.653596+0000","last_fullsized":"2023-10-16T12:10:43.653596+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:53:04.082200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:44.683426+0000","last_change":"2023-10-16T12:10:43.673415+0000","last_active":"2023-10-16T12:10:44.683426+0000","last_peered":"2023-10-16T12:10:44.683426+0000","last_clean":"2023-10-16T12:10:44.683426+0000","last_became_active":"2023-10-16T12:10:43.672874+0000","last_became_peered":"2023-10-16T12:10:43.672874+0000","last_unstale":"2023-10-16T12:10:44.683426+0000","last_undegraded":"2023-10-16T12:10:44.683426+0000","last_fullsized":"2023-10-16T12:10:44.683426+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:38:57.663899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646014+0000","last_change":"2023-10-16T12:10:38.628007+0000","last_active":"2023-10-16T12:10:43.646014+0000","last_peered":"2023-10-16T12:10:43.646014+0000","last_clean":"2023-10-16T12:10:43.646014+0000","last_became_active":"2023-10-16T12:10:38.627760+0000","last_became_peered":"2023-10-16T12:10:38.627760+0000","last_unstale":"2023-10-16T12:10:43.646014+0000","last_undegraded":"2023-10-16T12:10:43.646014+0000","last_fullsized":"2023-10-16T12:10:43.646014+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:46:01.589462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.652978+0000","last_change":"2023-10-16T12:10:39.632645+0000","last_active":"2023-10-16T12:10:43.652978+0000","last_peered":"2023-10-16T12:10:43.652978+0000","last_clean":"2023-10-16T12:10:43.652978+0000","last_became_active":"2023-10-16T12:10:39.632345+0000","last_became_peered":"2023-10-16T12:10:39.632345+0000","last_unstale":"2023-10-16T12:10:43.652978+0000","last_undegraded":"2023-10-16T12:10:43.652978+0000","last_fullsized":"2023-10-16T12:10:43.652978+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:58:21.398705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:44.683787+0000","last_change":"2023-10-16T12:10:43.667036+0000","last_active":"2023-10-16T12:10:44.683787+0000","last_peered":"2023-10-16T12:10:44.683787+0000","last_clean":"2023-10-16T12:10:44.683787+0000","last_became_active":"2023-10-16T12:10:43.666639+0000","last_became_peered":"2023-10-16T12:10:43.666639+0000","last_unstale":"2023-10-16T12:10:44.683787+0000","last_undegraded":"2023-10-16T12:10:44.683787+0000","last_fullsized":"2023-10-16T12:10:44.683787+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:29:47.533439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.657522+0000","last_change":"2023-10-16T12:10:39.633035+0000","last_active":"2023-10-16T12:10:43.657522+0000","last_peered":"2023-10-16T12:10:43.657522+0000","last_clean":"2023-10-16T12:10:43.657522+0000","last_became_active":"2023-10-16T12:10:39.631630+0000","last_became_peered":"2023-10-16T12:10:39.631630+0000","last_unstale":"2023-10-16T12:10:43.657522+0000","last_undegraded":"2023-10-16T12:10:43.657522+0000","last_fullsized":"2023-10-16T12:10:43.657522+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:47:52.332290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.642429+0000","last_change":"2023-10-16T12:10:38.622647+0000","last_active":"2023-10-16T12:10:43.642429+0000","last_peered":"2023-10-16T12:10:43.642429+0000","last_clean":"2023-10-16T12:10:43.642429+0000","last_became_active":"2023-10-16T12:10:38.622323+0000","last_became_peered":"2023-10-16T12:10:38.622323+0000","last_unstale":"2023-10-16T12:10:43.642429+0000","last_undegraded":"2023-10-16T12:10:43.642429+0000","last_fullsized":"2023-10-16T12:10:43.642429+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:27:30.532335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:44.874638+0000","last_change":"2023-10-16T12:10:43.797789+0000","last_active":"2023-10-16T12:10:44.874638+0000","last_peered":"2023-10-16T12:10:44.874638+0000","last_clean":"2023-10-16T12:10:44.874638+0000","last_became_active":"2023-10-16T12:10:43.797367+0000","last_became_peered":"2023-10-16T12:10:43.797367+0000","last_unstale":"2023-10-16T12:10:44.874638+0000","last_undegraded":"2023-10-16T12:10:44.874638+0000","last_fullsized":"2023-10-16T12:10:44.874638+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:24:19.577814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646387+0000","last_change":"2023-10-16T12:10:39.650651+0000","last_active":"2023-10-16T12:10:43.646387+0000","last_peered":"2023-10-16T12:10:43.646387+0000","last_clean":"2023-10-16T12:10:43.646387+0000","last_became_active":"2023-10-16T12:10:39.650279+0000","last_became_peered":"2023-10-16T12:10:39.650279+0000","last_unstale":"2023-10-16T12:10:43.646387+0000","last_undegraded":"2023-10-16T12:10:43.646387+0000","last_fullsized":"2023-10-16T12:10:43.646387+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:03:12.325906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646002+0000","last_change":"2023-10-16T12:10:38.622760+0000","last_active":"2023-10-16T12:10:43.646002+0000","last_peered":"2023-10-16T12:10:43.646002+0000","last_clean":"2023-10-16T12:10:43.646002+0000","last_became_active":"2023-10-16T12:10:38.622408+0000","last_became_peered":"2023-10-16T12:10:38.622408+0000","last_unstale":"2023-10-16T12:10:43.646002+0000","last_undegraded":"2023-10-16T12:10:43.646002+0000","last_fullsized":"2023-10-16T12:10:43.646002+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:52:07.438053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":169,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.793810+0000","last_change":"2023-10-16T12:10:34.746820+0000","last_active":"2023-10-16T12:10:43.793810+0000","last_peered":"2023-10-16T12:10:43.793810+0000","last_clean":"2023-10-16T12:10:43.793810+0000","last_became_active":"2023-10-16T12:10:34.745574+0000","last_became_peered":"2023-10-16T12:10:34.745574+0000","last_unstale":"2023-10-16T12:10:43.793810+0000","last_undegraded":"2023-10-16T12:10:43.793810+0000","last_fullsized":"2023-10-16T12:10:43.793810+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_clean_scrub_stamp":"2023-10-16T12:10:33.577111+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:14.461986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:45.375419+0000","last_change":"2023-10-16T12:10:43.798352+0000","last_active":"2023-10-16T12:10:45.375419+0000","last_peered":"2023-10-16T12:10:45.375419+0000","last_clean":"2023-10-16T12:10:45.375419+0000","last_became_active":"2023-10-16T12:10:43.797410+0000","last_became_peered":"2023-10-16T12:10:43.797410+0000","last_unstale":"2023-10-16T12:10:45.375419+0000","last_undegraded":"2023-10-16T12:10:45.375419+0000","last_fullsized":"2023-10-16T12:10:45.375419+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:03:18.128385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646524+0000","last_change":"2023-10-16T12:10:39.635017+0000","last_active":"2023-10-16T12:10:43.646524+0000","last_peered":"2023-10-16T12:10:43.646524+0000","last_clean":"2023-10-16T12:10:43.646524+0000","last_became_active":"2023-10-16T12:10:39.634564+0000","last_became_peered":"2023-10-16T12:10:39.634564+0000","last_unstale":"2023-10-16T12:10:43.646524+0000","last_undegraded":"2023-10-16T12:10:43.646524+0000","last_fullsized":"2023-10-16T12:10:43.646524+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:30:57.483330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.793813+0000","last_change":"2023-10-16T12:10:38.626904+0000","last_active":"2023-10-16T12:10:43.793813+0000","last_peered":"2023-10-16T12:10:43.793813+0000","last_clean":"2023-10-16T12:10:43.793813+0000","last_became_active":"2023-10-16T12:10:38.626683+0000","last_became_peered":"2023-10-16T12:10:38.626683+0000","last_unstale":"2023-10-16T12:10:43.793813+0000","last_undegraded":"2023-10-16T12:10:43.793813+0000","last_fullsized":"2023-10-16T12:10:43.793813+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:58:20.394576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:45.375040+0000","last_change":"2023-10-16T12:10:43.794928+0000","last_active":"2023-10-16T12:10:45.375040+0000","last_peered":"2023-10-16T12:10:45.375040+0000","last_clean":"2023-10-16T12:10:45.375040+0000","last_became_active":"2023-10-16T12:10:43.794276+0000","last_became_peered":"2023-10-16T12:10:43.794276+0000","last_unstale":"2023-10-16T12:10:45.375040+0000","last_undegraded":"2023-10-16T12:10:45.375040+0000","last_fullsized":"2023-10-16T12:10:45.375040+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:38:57.663899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.795963+0000","last_change":"2023-10-16T12:10:38.620637+0000","last_active":"2023-10-16T12:10:43.795963+0000","last_peered":"2023-10-16T12:10:43.795963+0000","last_clean":"2023-10-16T12:10:43.795963+0000","last_became_active":"2023-10-16T12:10:38.620275+0000","last_became_peered":"2023-10-16T12:10:38.620275+0000","last_unstale":"2023-10-16T12:10:43.795963+0000","last_undegraded":"2023-10-16T12:10:43.795963+0000","last_fullsized":"2023-10-16T12:10:43.795963+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:38:00.591856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.663386+0000","last_change":"2023-10-16T12:10:39.632571+0000","last_active":"2023-10-16T12:10:43.663386+0000","last_peered":"2023-10-16T12:10:43.663386+0000","last_clean":"2023-10-16T12:10:43.663386+0000","last_became_active":"2023-10-16T12:10:39.631893+0000","last_became_peered":"2023-10-16T12:10:39.631893+0000","last_unstale":"2023-10-16T12:10:43.663386+0000","last_undegraded":"2023-10-16T12:10:43.663386+0000","last_fullsized":"2023-10-16T12:10:43.663386+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:05:05.699350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:44.683543+0000","last_change":"2023-10-16T12:10:43.797399+0000","last_active":"2023-10-16T12:10:44.683543+0000","last_peered":"2023-10-16T12:10:44.683543+0000","last_clean":"2023-10-16T12:10:44.683543+0000","last_became_active":"2023-10-16T12:10:43.796570+0000","last_became_peered":"2023-10-16T12:10:43.796570+0000","last_unstale":"2023-10-16T12:10:44.683543+0000","last_undegraded":"2023-10-16T12:10:44.683543+0000","last_fullsized":"2023-10-16T12:10:44.683543+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:58:35.336755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.656426+0000","last_change":"2023-10-16T12:10:38.620866+0000","last_active":"2023-10-16T12:10:43.656426+0000","last_peered":"2023-10-16T12:10:43.656426+0000","last_clean":"2023-10-16T12:10:43.656426+0000","last_became_active":"2023-10-16T12:10:38.620517+0000","last_became_peered":"2023-10-16T12:10:38.620517+0000","last_unstale":"2023-10-16T12:10:43.656426+0000","last_undegraded":"2023-10-16T12:10:43.656426+0000","last_fullsized":"2023-10-16T12:10:43.656426+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:58:20.394576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646193+0000","last_change":"2023-10-16T12:10:39.639395+0000","last_active":"2023-10-16T12:10:43.646193+0000","last_peered":"2023-10-16T12:10:43.646193+0000","last_clean":"2023-10-16T12:10:43.646193+0000","last_became_active":"2023-10-16T12:10:39.639030+0000","last_became_peered":"2023-10-16T12:10:39.639030+0000","last_unstale":"2023-10-16T12:10:43.646193+0000","last_undegraded":"2023-10-16T12:10:43.646193+0000","last_fullsized":"2023-10-16T12:10:43.646193+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:34:50.059469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:45.375076+0000","last_change":"2023-10-16T12:10:43.800001+0000","last_active":"2023-10-16T12:10:45.375076+0000","last_peered":"2023-10-16T12:10:45.375076+0000","last_clean":"2023-10-16T12:10:45.375076+0000","last_became_active":"2023-10-16T12:10:43.799598+0000","last_became_peered":"2023-10-16T12:10:43.799598+0000","last_unstale":"2023-10-16T12:10:45.375076+0000","last_undegraded":"2023-10-16T12:10:45.375076+0000","last_fullsized":"2023-10-16T12:10:45.375076+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:27:48.055325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.652023+0000","last_change":"2023-10-16T12:10:38.619506+0000","last_active":"2023-10-16T12:10:43.652023+0000","last_peered":"2023-10-16T12:10:43.652023+0000","last_clean":"2023-10-16T12:10:43.652023+0000","last_became_active":"2023-10-16T12:10:38.619203+0000","last_became_peered":"2023-10-16T12:10:38.619203+0000","last_unstale":"2023-10-16T12:10:43.652023+0000","last_undegraded":"2023-10-16T12:10:43.652023+0000","last_fullsized":"2023-10-16T12:10:43.652023+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:58:20.394576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.648502+0000","last_change":"2023-10-16T12:10:39.630721+0000","last_active":"2023-10-16T12:10:43.648502+0000","last_peered":"2023-10-16T12:10:43.648502+0000","last_clean":"2023-10-16T12:10:43.648502+0000","last_became_active":"2023-10-16T12:10:39.630448+0000","last_became_peered":"2023-10-16T12:10:39.630448+0000","last_unstale":"2023-10-16T12:10:43.648502+0000","last_undegraded":"2023-10-16T12:10:43.648502+0000","last_fullsized":"2023-10-16T12:10:43.648502+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:03:12.325906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:45.376403+0000","last_change":"2023-10-16T12:10:43.662711+0000","last_active":"2023-10-16T12:10:45.376403+0000","last_peered":"2023-10-16T12:10:45.376403+0000","last_clean":"2023-10-16T12:10:45.376403+0000","last_became_active":"2023-10-16T12:10:43.661726+0000","last_became_peered":"2023-10-16T12:10:43.661726+0000","last_unstale":"2023-10-16T12:10:45.376403+0000","last_undegraded":"2023-10-16T12:10:45.376403+0000","last_fullsized":"2023-10-16T12:10:45.376403+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:13:07.783237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.793165+0000","last_change":"2023-10-16T12:10:38.620553+0000","last_active":"2023-10-16T12:10:43.793165+0000","last_peered":"2023-10-16T12:10:43.793165+0000","last_clean":"2023-10-16T12:10:43.793165+0000","last_became_active":"2023-10-16T12:10:38.620248+0000","last_became_peered":"2023-10-16T12:10:38.620248+0000","last_unstale":"2023-10-16T12:10:43.793165+0000","last_undegraded":"2023-10-16T12:10:43.793165+0000","last_fullsized":"2023-10-16T12:10:43.793165+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:45:22.011420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.647607+0000","last_change":"2023-10-16T12:10:39.631773+0000","last_active":"2023-10-16T12:10:43.647607+0000","last_peered":"2023-10-16T12:10:43.647607+0000","last_clean":"2023-10-16T12:10:43.647607+0000","last_became_active":"2023-10-16T12:10:39.631418+0000","last_became_peered":"2023-10-16T12:10:39.631418+0000","last_unstale":"2023-10-16T12:10:43.647607+0000","last_undegraded":"2023-10-16T12:10:43.647607+0000","last_fullsized":"2023-10-16T12:10:43.647607+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:45:23.015548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:44.683656+0000","last_change":"2023-10-16T12:10:43.794614+0000","last_active":"2023-10-16T12:10:44.683656+0000","last_peered":"2023-10-16T12:10:44.683656+0000","last_clean":"2023-10-16T12:10:44.683656+0000","last_became_active":"2023-10-16T12:10:43.794206+0000","last_became_peered":"2023-10-16T12:10:43.794206+0000","last_unstale":"2023-10-16T12:10:44.683656+0000","last_undegraded":"2023-10-16T12:10:44.683656+0000","last_fullsized":"2023-10-16T12:10:44.683656+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:03:18.128385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.653349+0000","last_change":"2023-10-16T12:10:38.621473+0000","last_active":"2023-10-16T12:10:43.653349+0000","last_peered":"2023-10-16T12:10:43.653349+0000","last_clean":"2023-10-16T12:10:43.653349+0000","last_became_active":"2023-10-16T12:10:38.619100+0000","last_became_peered":"2023-10-16T12:10:38.619100+0000","last_unstale":"2023-10-16T12:10:43.653349+0000","last_undegraded":"2023-10-16T12:10:43.653349+0000","last_fullsized":"2023-10-16T12:10:43.653349+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:04:47.764190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.647008+0000","last_change":"2023-10-16T12:10:39.630605+0000","last_active":"2023-10-16T12:10:43.647008+0000","last_peered":"2023-10-16T12:10:43.647008+0000","last_clean":"2023-10-16T12:10:43.647008+0000","last_became_active":"2023-10-16T12:10:39.630234+0000","last_became_peered":"2023-10-16T12:10:39.630234+0000","last_unstale":"2023-10-16T12:10:43.647008+0000","last_undegraded":"2023-10-16T12:10:43.647008+0000","last_fullsized":"2023-10-16T12:10:43.647008+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:33:20.807621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:45.376060+0000","last_change":"2023-10-16T12:10:43.665913+0000","last_active":"2023-10-16T12:10:45.376060+0000","last_peered":"2023-10-16T12:10:45.376060+0000","last_clean":"2023-10-16T12:10:45.376060+0000","last_became_active":"2023-10-16T12:10:43.665498+0000","last_became_peered":"2023-10-16T12:10:43.665498+0000","last_unstale":"2023-10-16T12:10:45.376060+0000","last_undegraded":"2023-10-16T12:10:45.376060+0000","last_fullsized":"2023-10-16T12:10:45.376060+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:10:23.879645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.652500+0000","last_change":"2023-10-16T12:10:38.623243+0000","last_active":"2023-10-16T12:10:43.652500+0000","last_peered":"2023-10-16T12:10:43.652500+0000","last_clean":"2023-10-16T12:10:43.652500+0000","last_became_active":"2023-10-16T12:10:38.622935+0000","last_became_peered":"2023-10-16T12:10:38.622935+0000","last_unstale":"2023-10-16T12:10:43.652500+0000","last_undegraded":"2023-10-16T12:10:43.652500+0000","last_fullsized":"2023-10-16T12:10:43.652500+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:27:30.532335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.645994+0000","last_change":"2023-10-16T12:10:39.624625+0000","last_active":"2023-10-16T12:10:43.645994+0000","last_peered":"2023-10-16T12:10:43.645994+0000","last_clean":"2023-10-16T12:10:43.645994+0000","last_became_active":"2023-10-16T12:10:39.624281+0000","last_became_peered":"2023-10-16T12:10:39.624281+0000","last_unstale":"2023-10-16T12:10:43.645994+0000","last_undegraded":"2023-10-16T12:10:43.645994+0000","last_fullsized":"2023-10-16T12:10:43.645994+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:27:44.016376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:45.377425+0000","last_change":"2023-10-16T12:10:43.662337+0000","last_active":"2023-10-16T12:10:45.377425+0000","last_peered":"2023-10-16T12:10:45.377425+0000","last_clean":"2023-10-16T12:10:45.377425+0000","last_became_active":"2023-10-16T12:10:43.661865+0000","last_became_peered":"2023-10-16T12:10:43.661865+0000","last_unstale":"2023-10-16T12:10:45.377425+0000","last_undegraded":"2023-10-16T12:10:45.377425+0000","last_fullsized":"2023-10-16T12:10:45.377425+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:14:04.764499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.794403+0000","last_change":"2023-10-16T12:10:38.624192+0000","last_active":"2023-10-16T12:10:43.794403+0000","last_peered":"2023-10-16T12:10:43.794403+0000","last_clean":"2023-10-16T12:10:43.794403+0000","last_became_active":"2023-10-16T12:10:38.623767+0000","last_became_peered":"2023-10-16T12:10:38.623767+0000","last_unstale":"2023-10-16T12:10:43.794403+0000","last_undegraded":"2023-10-16T12:10:43.794403+0000","last_fullsized":"2023-10-16T12:10:43.794403+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:59:30.615293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.651482+0000","last_change":"2023-10-16T12:10:39.645023+0000","last_active":"2023-10-16T12:10:43.651482+0000","last_peered":"2023-10-16T12:10:43.651482+0000","last_clean":"2023-10-16T12:10:43.651482+0000","last_became_active":"2023-10-16T12:10:39.644719+0000","last_became_peered":"2023-10-16T12:10:39.644719+0000","last_unstale":"2023-10-16T12:10:43.651482+0000","last_undegraded":"2023-10-16T12:10:43.651482+0000","last_fullsized":"2023-10-16T12:10:43.651482+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:47:52.332290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:44.874338+0000","last_change":"2023-10-16T12:10:43.794795+0000","last_active":"2023-10-16T12:10:44.874338+0000","last_peered":"2023-10-16T12:10:44.874338+0000","last_clean":"2023-10-16T12:10:44.874338+0000","last_became_active":"2023-10-16T12:10:43.794315+0000","last_became_peered":"2023-10-16T12:10:43.794315+0000","last_unstale":"2023-10-16T12:10:44.874338+0000","last_undegraded":"2023-10-16T12:10:44.874338+0000","last_fullsized":"2023-10-16T12:10:44.874338+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:13:07.783237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.642895+0000","last_change":"2023-10-16T12:10:38.622408+0000","last_active":"2023-10-16T12:10:43.642895+0000","last_peered":"2023-10-16T12:10:43.642895+0000","last_clean":"2023-10-16T12:10:43.642895+0000","last_became_active":"2023-10-16T12:10:38.622020+0000","last_became_peered":"2023-10-16T12:10:38.622020+0000","last_unstale":"2023-10-16T12:10:43.642895+0000","last_undegraded":"2023-10-16T12:10:43.642895+0000","last_fullsized":"2023-10-16T12:10:43.642895+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:57:56.978587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.796175+0000","last_change":"2023-10-16T12:10:39.645109+0000","last_active":"2023-10-16T12:10:43.796175+0000","last_peered":"2023-10-16T12:10:43.796175+0000","last_clean":"2023-10-16T12:10:43.796175+0000","last_became_active":"2023-10-16T12:10:39.641743+0000","last_became_peered":"2023-10-16T12:10:39.641743+0000","last_unstale":"2023-10-16T12:10:43.796175+0000","last_undegraded":"2023-10-16T12:10:43.796175+0000","last_fullsized":"2023-10-16T12:10:43.796175+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:53:04.082200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:45.374790+0000","last_change":"2023-10-16T12:10:43.666962+0000","last_active":"2023-10-16T12:10:45.374790+0000","last_peered":"2023-10-16T12:10:45.374790+0000","last_clean":"2023-10-16T12:10:45.374790+0000","last_became_active":"2023-10-16T12:10:43.666469+0000","last_became_peered":"2023-10-16T12:10:43.666469+0000","last_unstale":"2023-10-16T12:10:45.374790+0000","last_undegraded":"2023-10-16T12:10:45.374790+0000","last_fullsized":"2023-10-16T12:10:45.374790+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:14:04.764499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.654298+0000","last_change":"2023-10-16T12:10:38.611610+0000","last_active":"2023-10-16T12:10:43.654298+0000","last_peered":"2023-10-16T12:10:43.654298+0000","last_clean":"2023-10-16T12:10:43.654298+0000","last_became_active":"2023-10-16T12:10:38.611193+0000","last_became_peered":"2023-10-16T12:10:38.611193+0000","last_unstale":"2023-10-16T12:10:43.654298+0000","last_undegraded":"2023-10-16T12:10:43.654298+0000","last_fullsized":"2023-10-16T12:10:43.654298+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:06:08.125512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646733+0000","last_change":"2023-10-16T12:10:39.649078+0000","last_active":"2023-10-16T12:10:43.646733+0000","last_peered":"2023-10-16T12:10:43.646733+0000","last_clean":"2023-10-16T12:10:43.646733+0000","last_became_active":"2023-10-16T12:10:39.648797+0000","last_became_peered":"2023-10-16T12:10:39.648797+0000","last_unstale":"2023-10-16T12:10:43.646733+0000","last_undegraded":"2023-10-16T12:10:43.646733+0000","last_fullsized":"2023-10-16T12:10:43.646733+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:13:03.744288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:45.374995+0000","last_change":"2023-10-16T12:10:43.794916+0000","last_active":"2023-10-16T12:10:45.374995+0000","last_peered":"2023-10-16T12:10:45.374995+0000","last_clean":"2023-10-16T12:10:45.374995+0000","last_became_active":"2023-10-16T12:10:43.794495+0000","last_became_peered":"2023-10-16T12:10:43.794495+0000","last_unstale":"2023-10-16T12:10:45.374995+0000","last_undegraded":"2023-10-16T12:10:45.374995+0000","last_fullsized":"2023-10-16T12:10:45.374995+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:44:47.973077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.642563+0000","last_change":"2023-10-16T12:10:38.624223+0000","last_active":"2023-10-16T12:10:43.642563+0000","last_peered":"2023-10-16T12:10:43.642563+0000","last_clean":"2023-10-16T12:10:43.642563+0000","last_became_active":"2023-10-16T12:10:38.623924+0000","last_became_peered":"2023-10-16T12:10:38.623924+0000","last_unstale":"2023-10-16T12:10:43.642563+0000","last_undegraded":"2023-10-16T12:10:43.642563+0000","last_fullsized":"2023-10-16T12:10:43.642563+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:45:22.011420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.796741+0000","last_change":"2023-10-16T12:10:39.645947+0000","last_active":"2023-10-16T12:10:43.796741+0000","last_peered":"2023-10-16T12:10:43.796741+0000","last_clean":"2023-10-16T12:10:43.796741+0000","last_became_active":"2023-10-16T12:10:39.645261+0000","last_became_peered":"2023-10-16T12:10:39.645261+0000","last_unstale":"2023-10-16T12:10:43.796741+0000","last_undegraded":"2023-10-16T12:10:43.796741+0000","last_fullsized":"2023-10-16T12:10:43.796741+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:03:14.089436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.648802+0000","last_change":"2023-10-16T12:10:39.631025+0000","last_active":"2023-10-16T12:10:43.648802+0000","last_peered":"2023-10-16T12:10:43.648802+0000","last_clean":"2023-10-16T12:10:43.648802+0000","last_became_active":"2023-10-16T12:10:39.630701+0000","last_became_peered":"2023-10-16T12:10:39.630701+0000","last_unstale":"2023-10-16T12:10:43.648802+0000","last_undegraded":"2023-10-16T12:10:43.648802+0000","last_fullsized":"2023-10-16T12:10:43.648802+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:49:21.678699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.642384+0000","last_change":"2023-10-16T12:10:39.640966+0000","last_active":"2023-10-16T12:10:43.642384+0000","last_peered":"2023-10-16T12:10:43.642384+0000","last_clean":"2023-10-16T12:10:43.642384+0000","last_became_active":"2023-10-16T12:10:39.640624+0000","last_became_peered":"2023-10-16T12:10:39.640624+0000","last_unstale":"2023-10-16T12:10:43.642384+0000","last_undegraded":"2023-10-16T12:10:43.642384+0000","last_fullsized":"2023-10-16T12:10:43.642384+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:58:21.398705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646243+0000","last_change":"2023-10-16T12:10:39.647316+0000","last_active":"2023-10-16T12:10:43.646243+0000","last_peered":"2023-10-16T12:10:43.646243+0000","last_clean":"2023-10-16T12:10:43.646243+0000","last_became_active":"2023-10-16T12:10:39.646403+0000","last_became_peered":"2023-10-16T12:10:39.646403+0000","last_unstale":"2023-10-16T12:10:43.646243+0000","last_undegraded":"2023-10-16T12:10:43.646243+0000","last_fullsized":"2023-10-16T12:10:43.646243+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:24:15.538866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.649898+0000","last_change":"2023-10-16T12:10:39.625142+0000","last_active":"2023-10-16T12:10:43.649898+0000","last_peered":"2023-10-16T12:10:43.649898+0000","last_clean":"2023-10-16T12:10:43.649898+0000","last_became_active":"2023-10-16T12:10:39.624714+0000","last_became_peered":"2023-10-16T12:10:39.624714+0000","last_unstale":"2023-10-16T12:10:43.649898+0000","last_undegraded":"2023-10-16T12:10:43.649898+0000","last_fullsized":"2023-10-16T12:10:43.649898+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:46:02.593591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.648251+0000","last_change":"2023-10-16T12:10:39.642728+0000","last_active":"2023-10-16T12:10:43.648251+0000","last_peered":"2023-10-16T12:10:43.648251+0000","last_clean":"2023-10-16T12:10:43.648251+0000","last_became_active":"2023-10-16T12:10:39.642396+0000","last_became_peered":"2023-10-16T12:10:39.642396+0000","last_unstale":"2023-10-16T12:10:43.648251+0000","last_undegraded":"2023-10-16T12:10:43.648251+0000","last_fullsized":"2023-10-16T12:10:43.648251+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:38:01.595985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.654215+0000","last_change":"2023-10-16T12:10:39.635187+0000","last_active":"2023-10-16T12:10:43.654215+0000","last_peered":"2023-10-16T12:10:43.654215+0000","last_clean":"2023-10-16T12:10:43.654215+0000","last_became_active":"2023-10-16T12:10:39.634499+0000","last_became_peered":"2023-10-16T12:10:39.634499+0000","last_unstale":"2023-10-16T12:10:43.654215+0000","last_undegraded":"2023-10-16T12:10:43.654215+0000","last_fullsized":"2023-10-16T12:10:43.654215+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:38:01.595985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.650893+0000","last_change":"2023-10-16T12:10:39.643166+0000","last_active":"2023-10-16T12:10:43.650893+0000","last_peered":"2023-10-16T12:10:43.650893+0000","last_clean":"2023-10-16T12:10:43.650893+0000","last_became_active":"2023-10-16T12:10:39.642893+0000","last_became_peered":"2023-10-16T12:10:39.642893+0000","last_unstale":"2023-10-16T12:10:43.650893+0000","last_undegraded":"2023-10-16T12:10:43.650893+0000","last_fullsized":"2023-10-16T12:10:43.650893+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:33:20.807621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646227+0000","last_change":"2023-10-16T12:10:39.633538+0000","last_active":"2023-10-16T12:10:43.646227+0000","last_peered":"2023-10-16T12:10:43.646227+0000","last_clean":"2023-10-16T12:10:43.646227+0000","last_became_active":"2023-10-16T12:10:39.633019+0000","last_became_peered":"2023-10-16T12:10:39.633019+0000","last_unstale":"2023-10-16T12:10:43.646227+0000","last_undegraded":"2023-10-16T12:10:43.646227+0000","last_fullsized":"2023-10-16T12:10:43.646227+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:59:08.633021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.652820+0000","last_change":"2023-10-16T12:10:39.641581+0000","last_active":"2023-10-16T12:10:43.652820+0000","last_peered":"2023-10-16T12:10:43.652820+0000","last_clean":"2023-10-16T12:10:43.652820+0000","last_became_active":"2023-10-16T12:10:39.640847+0000","last_became_peered":"2023-10-16T12:10:39.640847+0000","last_unstale":"2023-10-16T12:10:43.652820+0000","last_undegraded":"2023-10-16T12:10:43.652820+0000","last_fullsized":"2023-10-16T12:10:43.652820+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:03:12.325906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646580+0000","last_change":"2023-10-16T12:10:39.642631+0000","last_active":"2023-10-16T12:10:43.646580+0000","last_peered":"2023-10-16T12:10:43.646580+0000","last_clean":"2023-10-16T12:10:43.646580+0000","last_became_active":"2023-10-16T12:10:39.642334+0000","last_became_peered":"2023-10-16T12:10:39.642334+0000","last_unstale":"2023-10-16T12:10:43.646580+0000","last_undegraded":"2023-10-16T12:10:43.646580+0000","last_fullsized":"2023-10-16T12:10:43.646580+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:46:02.593591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646326+0000","last_change":"2023-10-16T12:10:39.647489+0000","last_active":"2023-10-16T12:10:43.646326+0000","last_peered":"2023-10-16T12:10:43.646326+0000","last_clean":"2023-10-16T12:10:43.646326+0000","last_became_active":"2023-10-16T12:10:39.646439+0000","last_became_peered":"2023-10-16T12:10:39.646439+0000","last_unstale":"2023-10-16T12:10:43.646326+0000","last_undegraded":"2023-10-16T12:10:43.646326+0000","last_fullsized":"2023-10-16T12:10:43.646326+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:59:08.633021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.649955+0000","last_change":"2023-10-16T12:10:39.638887+0000","last_active":"2023-10-16T12:10:43.649955+0000","last_peered":"2023-10-16T12:10:43.649955+0000","last_clean":"2023-10-16T12:10:43.649955+0000","last_became_active":"2023-10-16T12:10:39.638548+0000","last_became_peered":"2023-10-16T12:10:39.638548+0000","last_unstale":"2023-10-16T12:10:43.649955+0000","last_undegraded":"2023-10-16T12:10:43.649955+0000","last_fullsized":"2023-10-16T12:10:43.649955+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:49:21.678699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.650677+0000","last_change":"2023-10-16T12:10:39.644900+0000","last_active":"2023-10-16T12:10:43.650677+0000","last_peered":"2023-10-16T12:10:43.650677+0000","last_clean":"2023-10-16T12:10:43.650677+0000","last_became_active":"2023-10-16T12:10:39.644516+0000","last_became_peered":"2023-10-16T12:10:39.644516+0000","last_unstale":"2023-10-16T12:10:43.650677+0000","last_undegraded":"2023-10-16T12:10:43.650677+0000","last_fullsized":"2023-10-16T12:10:43.650677+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:03:12.325906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574858,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27864,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343976,"statfs":{"total":96636764160,"available":96608231424,"internally_reserved":0,"allocated":864256,"data_stored":695401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72599999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68100000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71399999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82499999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52600000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":108293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.624}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54100000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45200000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51100000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69499999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76100000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":274432,"data_stored":104611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.374}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66900000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80400000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82399999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574857,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27876,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343964,"statfs":{"total":96636764160,"available":96608219136,"internally_reserved":0,"allocated":872448,"data_stored":695457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.312}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71399999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65000000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56299999999999994}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67500000000000004}]}]},{"osd":2,"up_from":13,"seq":55834574857,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":266240,"data_stored":104451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.31}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29299999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39200000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45500000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41099999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42699999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]}]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":108293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.252}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32700000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69499999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.627}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80800000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67100000000000004}]}]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":278528,"data_stored":104577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54300000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69299999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61399999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64100000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60099999999999998}]}]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":270336,"data_stored":104541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.497}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46700000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52500000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55300000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67600000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T12:11:04.736 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-16T12:11:04.736 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-10-16T12:11:04.736 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-10-16T12:11:04.737 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-10-16T12:11:04.737 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-10-16T12:11:04.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.892+0000 7fe4db29e700 1 Processor -- start 2023-10-16T12:11:04.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.893+0000 7fe4db29e700 1 -- start start 2023-10-16T12:11:04.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.893+0000 7fe4db29e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe4d40ce4f0 0x7fe4d40cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:04.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.893+0000 7fe4db29e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe4d415ba10 0x7fe4d40cfca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:04.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.893+0000 7fe4db29e700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4d40cc2a0 con 0x7fe4d415b630 2023-10-16T12:11:04.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.893+0000 7fe4db29e700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4d40cdb20 con 0x7fe4d40ce4f0 2023-10-16T12:11:04.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.893+0000 7fe4db29e700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d40cc540 con 0x7fe4d415ba10 2023-10-16T12:11:04.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.894+0000 7fe4d8839700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe4d40ce4f0 0x7fe4d40cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:04.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.894+0000 7fe4d8839700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe4d40ce4f0 0x7fe4d40cd5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33484/0 (socket says 172.21.15.16:33484) 2023-10-16T12:11:04.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.894+0000 7fe4d8839700 1 -- 172.21.15.16:0/1910769721 learned_addr learned my addr 172.21.15.16:0/1910769721 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:04.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.894+0000 7fe4d983b700 1 --2- 172.21.15.16:0/1910769721 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe4d415ba10 0x7fe4d40cfca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:04.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.894+0000 7fe4d8839700 1 -- 172.21.15.16:0/1910769721 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe4d415ba10 msgr2=0x7fe4d40cfca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:04.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.894+0000 7fe4d8839700 1 --2- 172.21.15.16:0/1910769721 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe4d415ba10 0x7fe4d40cfca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:04.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.894+0000 7fe4d8839700 1 -- 172.21.15.16:0/1910769721 >> v1:172.21.15.49:6789/0 conn(0x7fe4d415b630 legacy=0x7fe4d40cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:04.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.894+0000 7fe4d8839700 1 -- 172.21.15.16:0/1910769721 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4d40cc820 con 0x7fe4d40ce4f0 2023-10-16T12:11:04.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.894+0000 7fe4d8839700 1 --2- 172.21.15.16:0/1910769721 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4d40ce4f0 0x7fe4d40cd5e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe4c80098f0 tx=0x7fe4c8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae5fbb28e474d4e8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:04.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.894+0000 7fe4c7fff700 1 -- 172.21.15.16:0/1910769721 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4c8011830 con 0x7fe4d40ce4f0 2023-10-16T12:11:04.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.894+0000 7fe4c7fff700 1 -- 172.21.15.16:0/1910769721 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4c8011990 con 0x7fe4d40ce4f0 2023-10-16T12:11:04.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.894+0000 7fe4c7fff700 1 -- 172.21.15.16:0/1910769721 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4c8011d20 con 0x7fe4d40ce4f0 2023-10-16T12:11:04.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.895+0000 7fe4db29e700 1 -- 172.21.15.16:0/1910769721 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4d40ce4f0 msgr2=0x7fe4d40cd5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:04.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.895+0000 7fe4db29e700 1 --2- 172.21.15.16:0/1910769721 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4d40ce4f0 0x7fe4d40cd5e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe4c80098f0 tx=0x7fe4c8009c00 comp rx=0 tx=0).stop 2023-10-16T12:11:04.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.895+0000 7fe4db29e700 1 -- 172.21.15.16:0/1910769721 shutdown_connections 2023-10-16T12:11:04.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.895+0000 7fe4db29e700 1 --2- 172.21.15.16:0/1910769721 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe4d415ba10 0x7fe4d40cfca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:04.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.895+0000 7fe4db29e700 1 --2- 172.21.15.16:0/1910769721 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4d40ce4f0 0x7fe4d40cd5e0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:04.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.895+0000 7fe4db29e700 1 -- 172.21.15.16:0/1910769721 >> 172.21.15.16:0/1910769721 conn(0x7fe4d40d14b0 msgr2=0x7fe4d40d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:04.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.895+0000 7fe4db29e700 1 -- 172.21.15.16:0/1910769721 shutdown_connections 2023-10-16T12:11:04.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.895+0000 7fe4db29e700 1 -- 172.21.15.16:0/1910769721 wait complete. 2023-10-16T12:11:04.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.896+0000 7fe4db29e700 1 Processor -- start 2023-10-16T12:11:04.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.896+0000 7fe4db29e700 1 -- start start 2023-10-16T12:11:04.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.897+0000 7fe4db29e700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe4d40ce4f0 0x7fe4d41f4190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:04.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.897+0000 7fe4db29e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4d415b630 0x7fe4d41f15e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:04.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.897+0000 7fe4db29e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe4d415ba10 0x7fe4d41499d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:04.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.897+0000 7fe4db29e700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d414a200 con 0x7fe4d415b630 2023-10-16T12:11:04.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.897+0000 7fe4db29e700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d41f1b20 con 0x7fe4d40ce4f0 2023-10-16T12:11:04.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.897+0000 7fe4db29e700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d4149f10 con 0x7fe4d415ba10 2023-10-16T12:11:04.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.897+0000 7fe4d8839700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4d415b630 0x7fe4d41f15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:04.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.897+0000 7fe4d8839700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4d415b630 0x7fe4d41f15e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33488/0 (socket says 172.21.15.16:33488) 2023-10-16T12:11:04.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.897+0000 7fe4d8839700 1 -- 172.21.15.16:0/355425958 learned_addr learned my addr 172.21.15.16:0/355425958 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:04.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.897+0000 7fe4d903a700 1 --2- 172.21.15.16:0/355425958 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe4d40ce4f0 0x7fe4d41f4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:04.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.897+0000 7fe4d983b700 1 --2- 172.21.15.16:0/355425958 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe4d415ba10 0x7fe4d41499d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:04.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.897+0000 7fe4d8839700 1 -- 172.21.15.16:0/355425958 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe4d415ba10 msgr2=0x7fe4d41499d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:04.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.897+0000 7fe4d8839700 1 --2- 172.21.15.16:0/355425958 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe4d415ba10 0x7fe4d41499d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:04.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.897+0000 7fe4d8839700 1 -- 172.21.15.16:0/355425958 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe4d40ce4f0 msgr2=0x7fe4d41f4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:04.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.897+0000 7fe4d8839700 1 --2- 172.21.15.16:0/355425958 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe4d40ce4f0 0x7fe4d41f4190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:04.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.897+0000 7fe4d8839700 1 -- 172.21.15.16:0/355425958 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4c8009550 con 0x7fe4d415b630 2023-10-16T12:11:04.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.897+0000 7fe4d8839700 1 --2- 172.21.15.16:0/355425958 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4d415b630 0x7fe4d41f15e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe4c8014040 tx=0x7fe4c8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:04.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.898+0000 7fe4c5ffb700 1 -- 172.21.15.16:0/355425958 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4c8004510 con 0x7fe4d415b630 2023-10-16T12:11:04.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.898+0000 7fe4c5ffb700 1 -- 172.21.15.16:0/355425958 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4c801e070 con 0x7fe4d415b630 2023-10-16T12:11:04.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.898+0000 7fe4c5ffb700 1 -- 172.21.15.16:0/355425958 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4c80038e0 con 0x7fe4d415b630 2023-10-16T12:11:04.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.898+0000 7fe4db29e700 1 -- 172.21.15.16:0/355425958 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4d41f5060 con 0x7fe4d415b630 2023-10-16T12:11:04.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.898+0000 7fe4db29e700 1 -- 172.21.15.16:0/355425958 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4d41f4aa0 con 0x7fe4d415b630 2023-10-16T12:11:04.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.899+0000 7fe4db29e700 1 -- 172.21.15.16:0/355425958 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4d414a390 con 0x7fe4d415b630 2023-10-16T12:11:04.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.899+0000 7fe4c5ffb700 1 -- 172.21.15.16:0/355425958 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fe4c8003a40 con 0x7fe4d415b630 2023-10-16T12:11:04.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.900+0000 7fe4c5ffb700 1 --2- 172.21.15.16:0/355425958 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fe4b806af00 0x7fe4b806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:04.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.900+0000 7fe4c5ffb700 1 -- 172.21.15.16:0/355425958 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fe4c808a740 con 0x7fe4d415b630 2023-10-16T12:11:04.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.900+0000 7fe4d903a700 1 --2- 172.21.15.16:0/355425958 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fe4b806af00 0x7fe4b806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:04.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.900+0000 7fe4d903a700 1 --2- 172.21.15.16:0/355425958 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fe4b806af00 0x7fe4b806d3c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fe4d40cba80 tx=0x7fe4d0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:04.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:04.904+0000 7fe4c5ffb700 1 -- 172.21.15.16:0/355425958 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe4c80554f0 con 0x7fe4d415b630 2023-10-16T12:11:05.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.168+0000 7fe4db29e700 1 -- 172.21.15.16:0/355425958 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fe4d404ebb0 con 0x7fe4d415b630 2023-10-16T12:11:05.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.172+0000 7fe4c5ffb700 1 -- 172.21.15.16:0/355425958 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+171026 (secure 0 0 0) 0x7fe4c8058b10 con 0x7fe4d415b630 2023-10-16T12:11:05.175 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:11:05.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.175+0000 7fe4db29e700 1 -- 172.21.15.16:0/355425958 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fe4b806af00 msgr2=0x7fe4b806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:05.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.175+0000 7fe4db29e700 1 --2- 172.21.15.16:0/355425958 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fe4b806af00 0x7fe4b806d3c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fe4d40cba80 tx=0x7fe4d0009040 comp rx=0 tx=0).stop 2023-10-16T12:11:05.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.175+0000 7fe4db29e700 1 -- 172.21.15.16:0/355425958 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4d415b630 msgr2=0x7fe4d41f15e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:05.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.175+0000 7fe4db29e700 1 --2- 172.21.15.16:0/355425958 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4d415b630 0x7fe4d41f15e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe4c8014040 tx=0x7fe4c8004750 comp rx=0 tx=0).stop 2023-10-16T12:11:05.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.175+0000 7fe4db29e700 1 -- 172.21.15.16:0/355425958 shutdown_connections 2023-10-16T12:11:05.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.175+0000 7fe4db29e700 1 --2- 172.21.15.16:0/355425958 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fe4b806af00 0x7fe4b806d3c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:05.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.175+0000 7fe4db29e700 1 --2- 172.21.15.16:0/355425958 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe4d415ba10 0x7fe4d41499d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:05.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.175+0000 7fe4db29e700 1 --2- 172.21.15.16:0/355425958 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe4d40ce4f0 0x7fe4d41f4190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:05.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.175+0000 7fe4db29e700 1 --2- 172.21.15.16:0/355425958 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4d415b630 0x7fe4d41f15e0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:05.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.175+0000 7fe4db29e700 1 -- 172.21.15.16:0/355425958 >> 172.21.15.16:0/355425958 conn(0x7fe4d40d14b0 msgr2=0x7fe4d4070430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:05.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.175+0000 7fe4db29e700 1 -- 172.21.15.16:0/355425958 shutdown_connections 2023-10-16T12:11:05.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.175+0000 7fe4db29e700 1 -- 172.21.15.16:0/355425958 wait complete. 2023-10-16T12:11:05.189 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":5,"active_gid":4104,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":365921436},{"type":"v1","addr":"172.21.15.16:6833","nonce":365921436}]},"active_addr":"172.21.15.16:6833/365921436","active_change":"2023-10-16T12:10:27.087791+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4100,"name":"x","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":1893346008}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":3547353312}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":675356848}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":1300148187}]}]} 2023-10-16T12:11:05.193 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-10-16T12:11:05.193 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-10-16T12:11:05.193 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:11:05.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.316+0000 7fe801e8d700 1 Processor -- start 2023-10-16T12:11:05.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.317+0000 7fe801e8d700 1 -- start start 2023-10-16T12:11:05.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.317+0000 7fe801e8d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7fc15a5c0 0x7fe7fc0cef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:05.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.317+0000 7fe801e8d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe7fc0d0ae0 0x7fe7fc0cc1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:05.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.317+0000 7fe801e8d700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7fc072a50 con 0x7fe7fc0ce010 2023-10-16T12:11:05.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.317+0000 7fe801e8d700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7fc0cf440 con 0x7fe7fc0d0ae0 2023-10-16T12:11:05.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.317+0000 7fe801e8d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7fc0cba30 con 0x7fe7fc15a5c0 2023-10-16T12:11:05.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.317+0000 7fe7fbfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe7fc0d0ae0 0x7fe7fc0cc1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:05.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.318+0000 7fe7fbfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe7fc0d0ae0 0x7fe7fc0cc1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33500/0 (socket says 172.21.15.16:33500) 2023-10-16T12:11:05.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.318+0000 7fe7fbfff700 1 -- 172.21.15.16:0/2547728101 learned_addr learned my addr 172.21.15.16:0/2547728101 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:05.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.318+0000 7fe7fb7fe700 1 --2- 172.21.15.16:0/2547728101 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7fc15a5c0 0x7fe7fc0cef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:05.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.318+0000 7fe7fbfff700 1 -- 172.21.15.16:0/2547728101 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7fc15a5c0 msgr2=0x7fe7fc0cef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:05.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.318+0000 7fe7fbfff700 1 --2- 172.21.15.16:0/2547728101 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7fc15a5c0 0x7fe7fc0cef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:05.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.318+0000 7fe7fbfff700 1 -- 172.21.15.16:0/2547728101 >> v1:172.21.15.49:6789/0 conn(0x7fe7fc0ce010 legacy=0x7fe7fc0d03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:05.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.318+0000 7fe7fbfff700 1 -- 172.21.15.16:0/2547728101 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7fc0cf630 con 0x7fe7fc0d0ae0 2023-10-16T12:11:05.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.318+0000 7fe7fbfff700 1 --2- 172.21.15.16:0/2547728101 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7fc0d0ae0 0x7fe7fc0cc1f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe7f400d150 tx=0x7fe7f400d460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d03952fd060cda35 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:05.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.318+0000 7fe7fa7fc700 1 -- 172.21.15.16:0/2547728101 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f4010070 con 0x7fe7fc0d0ae0 2023-10-16T12:11:05.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.318+0000 7fe7fa7fc700 1 -- 172.21.15.16:0/2547728101 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7f400dd10 con 0x7fe7fc0d0ae0 2023-10-16T12:11:05.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.318+0000 7fe7fa7fc700 1 -- 172.21.15.16:0/2547728101 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f400b310 con 0x7fe7fc0d0ae0 2023-10-16T12:11:05.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.319+0000 7fe801e8d700 1 -- 172.21.15.16:0/2547728101 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7fc0d0ae0 msgr2=0x7fe7fc0cc1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:05.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.319+0000 7fe801e8d700 1 --2- 172.21.15.16:0/2547728101 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7fc0d0ae0 0x7fe7fc0cc1f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe7f400d150 tx=0x7fe7f400d460 comp rx=0 tx=0).stop 2023-10-16T12:11:05.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.319+0000 7fe801e8d700 1 -- 172.21.15.16:0/2547728101 shutdown_connections 2023-10-16T12:11:05.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.319+0000 7fe801e8d700 1 --2- 172.21.15.16:0/2547728101 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7fc0d0ae0 0x7fe7fc0cc1f0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:05.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.319+0000 7fe801e8d700 1 --2- 172.21.15.16:0/2547728101 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7fc15a5c0 0x7fe7fc0cef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:05.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.319+0000 7fe801e8d700 1 -- 172.21.15.16:0/2547728101 >> 172.21.15.16:0/2547728101 conn(0x7fe7fc0b1fd0 msgr2=0x7fe7fc0b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:05.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.319+0000 7fe801e8d700 1 -- 172.21.15.16:0/2547728101 shutdown_connections 2023-10-16T12:11:05.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.319+0000 7fe801e8d700 1 -- 172.21.15.16:0/2547728101 wait complete. 2023-10-16T12:11:05.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.320+0000 7fe801e8d700 1 Processor -- start 2023-10-16T12:11:05.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.320+0000 7fe801e8d700 1 -- start start 2023-10-16T12:11:05.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.321+0000 7fe801e8d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7fc0ce010 0x7fe7fc0c6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:05.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.321+0000 7fe801e8d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7fc0d0ae0 0x7fe7fc154660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:05.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.321+0000 7fe801e8d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7fc15a5c0 0x7fe7fc0c3c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:05.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.321+0000 7fe801e8d700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7fc0b3b20 con 0x7fe7fc15a5c0 2023-10-16T12:11:05.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.321+0000 7fe801e8d700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7fc0c68d0 con 0x7fe7fc0ce010 2023-10-16T12:11:05.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.321+0000 7fe801e8d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7fc154ba0 con 0x7fe7fc0d0ae0 2023-10-16T12:11:05.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.321+0000 7fe7fbfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7fc15a5c0 0x7fe7fc0c3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:05.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.321+0000 7fe7fbfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7fc15a5c0 0x7fe7fc0c3c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33506/0 (socket says 172.21.15.16:33506) 2023-10-16T12:11:05.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.321+0000 7fe7fb7fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7fc0ce010 0x7fe7fc0c6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:05.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.321+0000 7fe7fbfff700 1 -- 172.21.15.16:0/761560398 learned_addr learned my addr 172.21.15.16:0/761560398 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:05.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.321+0000 7fe7faffd700 1 --2- 172.21.15.16:0/761560398 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7fc0d0ae0 0x7fe7fc154660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:05.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.321+0000 7fe7fbfff700 1 -- 172.21.15.16:0/761560398 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7fc0d0ae0 msgr2=0x7fe7fc154660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:05.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.321+0000 7fe7fbfff700 1 --2- 172.21.15.16:0/761560398 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7fc0d0ae0 0x7fe7fc154660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:05.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.321+0000 7fe7fbfff700 1 -- 172.21.15.16:0/761560398 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7fc0ce010 msgr2=0x7fe7fc0c6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:05.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.321+0000 7fe7fbfff700 1 --2- 172.21.15.16:0/761560398 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7fc0ce010 0x7fe7fc0c6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:05.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.321+0000 7fe7fbfff700 1 -- 172.21.15.16:0/761560398 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7f400bd40 con 0x7fe7fc15a5c0 2023-10-16T12:11:05.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.321+0000 7fe7fbfff700 1 --2- 172.21.15.16:0/761560398 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7fc15a5c0 0x7fe7fc0c3c80 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fe7f4014000 tx=0x7fe7f4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:05.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.322+0000 7fe7f3fff700 1 -- 172.21.15.16:0/761560398 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f400bbc0 con 0x7fe7fc15a5c0 2023-10-16T12:11:05.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.322+0000 7fe7f3fff700 1 -- 172.21.15.16:0/761560398 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7f400b4c0 con 0x7fe7fc15a5c0 2023-10-16T12:11:05.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.322+0000 7fe7f3fff700 1 -- 172.21.15.16:0/761560398 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f400ae70 con 0x7fe7fc15a5c0 2023-10-16T12:11:05.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.322+0000 7fe801e8d700 1 -- 172.21.15.16:0/761560398 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7fc0c6b30 con 0x7fe7fc15a5c0 2023-10-16T12:11:05.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.322+0000 7fe801e8d700 1 -- 172.21.15.16:0/761560398 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7fc0b34d0 con 0x7fe7fc15a5c0 2023-10-16T12:11:05.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.323+0000 7fe801e8d700 1 -- 172.21.15.16:0/761560398 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7fc04ebb0 con 0x7fe7fc15a5c0 2023-10-16T12:11:05.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.323+0000 7fe7f3fff700 1 -- 172.21.15.16:0/761560398 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fe7f4004040 con 0x7fe7fc15a5c0 2023-10-16T12:11:05.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.323+0000 7fe7f3fff700 1 --2- 172.21.15.16:0/761560398 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fe7dc06af00 0x7fe7dc06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:05.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.324+0000 7fe7f3fff700 1 -- 172.21.15.16:0/761560398 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fe7f408d1d0 con 0x7fe7fc15a5c0 2023-10-16T12:11:05.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.324+0000 7fe7fb7fe700 1 --2- 172.21.15.16:0/761560398 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fe7dc06af00 0x7fe7dc06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:05.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.324+0000 7fe7fb7fe700 1 --2- 172.21.15.16:0/761560398 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fe7dc06af00 0x7fe7dc06d3c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fe7e4009be0 tx=0x7fe7e4009210 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:05.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.328+0000 7fe7f3fff700 1 -- 172.21.15.16:0/761560398 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe7f4058020 con 0x7fe7fc15a5c0 2023-10-16T12:11:05.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.535+0000 7fe801e8d700 1 -- 172.21.15.16:0/761560398 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe7fc0cfbc0 con 0x7fe7fc15a5c0 2023-10-16T12:11:05.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.536+0000 7fe7f3fff700 1 -- 172.21.15.16:0/761560398 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18238 (secure 0 0 0) 0x7fe7f405b640 con 0x7fe7fc15a5c0 2023-10-16T12:11:05.538 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:11:05.539 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":25,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:10:44.080141+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:10:36.935773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T12:10:38.004870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-16T12:10:41.672025+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:11:05.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.538+0000 7fe801e8d700 1 -- 172.21.15.16:0/761560398 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fe7dc06af00 msgr2=0x7fe7dc06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:05.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.538+0000 7fe801e8d700 1 --2- 172.21.15.16:0/761560398 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fe7dc06af00 0x7fe7dc06d3c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fe7e4009be0 tx=0x7fe7e4009210 comp rx=0 tx=0).stop 2023-10-16T12:11:05.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.538+0000 7fe801e8d700 1 -- 172.21.15.16:0/761560398 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7fc15a5c0 msgr2=0x7fe7fc0c3c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:05.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.538+0000 7fe801e8d700 1 --2- 172.21.15.16:0/761560398 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7fc15a5c0 0x7fe7fc0c3c80 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fe7f4014000 tx=0x7fe7f4004ad0 comp rx=0 tx=0).stop 2023-10-16T12:11:05.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.538+0000 7fe801e8d700 1 -- 172.21.15.16:0/761560398 shutdown_connections 2023-10-16T12:11:05.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.538+0000 7fe801e8d700 1 --2- 172.21.15.16:0/761560398 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fe7dc06af00 0x7fe7dc06d3c0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:05.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.538+0000 7fe801e8d700 1 --2- 172.21.15.16:0/761560398 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7fc15a5c0 0x7fe7fc0c3c80 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:05.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.538+0000 7fe801e8d700 1 --2- 172.21.15.16:0/761560398 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7fc0ce010 0x7fe7fc0c6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:05.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.538+0000 7fe801e8d700 1 --2- 172.21.15.16:0/761560398 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7fc0d0ae0 0x7fe7fc154660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:05.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.538+0000 7fe801e8d700 1 -- 172.21.15.16:0/761560398 >> 172.21.15.16:0/761560398 conn(0x7fe7fc0b1fd0 msgr2=0x7fe7fc0d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:05.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.539+0000 7fe801e8d700 1 -- 172.21.15.16:0/761560398 shutdown_connections 2023-10-16T12:11:05.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.539+0000 7fe801e8d700 1 -- 172.21.15.16:0/761560398 wait complete. 2023-10-16T12:11:05.553 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-10-16T12:11:05.553 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:11:05.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.676+0000 7fe073870700 1 Processor -- start 2023-10-16T12:11:05.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.676+0000 7fe073870700 1 -- start start 2023-10-16T12:11:05.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.677+0000 7fe073870700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fe06c0cf4a0 0x7fe06c0726e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:05.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.677+0000 7fe073870700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe06c0cf880 0x7fe06c0b52f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:05.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.677+0000 7fe073870700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe06c0b5830 0x7fe06c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:05.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.677+0000 7fe073870700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe06c15bc80 con 0x7fe06c0b5830 2023-10-16T12:11:05.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.677+0000 7fe073870700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe06c072c20 con 0x7fe06c0cf4a0 2023-10-16T12:11:05.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.677+0000 7fe073870700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe06c15b070 con 0x7fe06c0cf880 2023-10-16T12:11:05.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.677+0000 7fe071e0d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe06c0b5830 0x7fe06c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:05.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.677+0000 7fe071e0d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe06c0b5830 0x7fe06c15a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33514/0 (socket says 172.21.15.16:33514) 2023-10-16T12:11:05.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.677+0000 7fe071e0d700 1 -- 172.21.15.16:0/1895392199 learned_addr learned my addr 172.21.15.16:0/1895392199 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:05.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.677+0000 7fe07160c700 1 --2- 172.21.15.16:0/1895392199 >> v2:172.21.15.49:3300/0 conn(0x7fe06c0cf4a0 0x7fe06c0726e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:05.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.677+0000 7fe070e0b700 1 --2- 172.21.15.16:0/1895392199 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe06c0cf880 0x7fe06c0b52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:05.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.678+0000 7fe071e0d700 1 -- 172.21.15.16:0/1895392199 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe06c0cf880 msgr2=0x7fe06c0b52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:05.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.678+0000 7fe071e0d700 1 --2- 172.21.15.16:0/1895392199 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe06c0cf880 0x7fe06c0b52f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:05.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.678+0000 7fe071e0d700 1 -- 172.21.15.16:0/1895392199 >> v2:172.21.15.49:3300/0 conn(0x7fe06c0cf4a0 msgr2=0x7fe06c0726e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:05.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.678+0000 7fe071e0d700 1 --2- 172.21.15.16:0/1895392199 >> v2:172.21.15.49:3300/0 conn(0x7fe06c0cf4a0 0x7fe06c0726e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:05.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.678+0000 7fe071e0d700 1 -- 172.21.15.16:0/1895392199 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe06c15b350 con 0x7fe06c0b5830 2023-10-16T12:11:05.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.678+0000 7fe07160c700 1 --2- 172.21.15.16:0/1895392199 >> v2:172.21.15.49:3300/0 conn(0x7fe06c0cf4a0 0x7fe06c0726e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:11:05.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.678+0000 7fe071e0d700 1 --2- 172.21.15.16:0/1895392199 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe06c0b5830 0x7fe06c15a090 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe06c0b2d20 tx=0x7fe06400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d41c8a24eaf447c2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:05.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.678+0000 7fe063fff700 1 -- 172.21.15.16:0/1895392199 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe06400e070 con 0x7fe06c0b5830 2023-10-16T12:11:05.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.678+0000 7fe063fff700 1 -- 172.21.15.16:0/1895392199 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe06400b910 con 0x7fe06c0b5830 2023-10-16T12:11:05.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.678+0000 7fe063fff700 1 -- 172.21.15.16:0/1895392199 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe06400bcc0 con 0x7fe06c0b5830 2023-10-16T12:11:05.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.678+0000 7fe073870700 1 -- 172.21.15.16:0/1895392199 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe06c0b5830 msgr2=0x7fe06c15a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:05.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.678+0000 7fe073870700 1 --2- 172.21.15.16:0/1895392199 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe06c0b5830 0x7fe06c15a090 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe06c0b2d20 tx=0x7fe06400b330 comp rx=0 tx=0).stop 2023-10-16T12:11:05.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.679+0000 7fe073870700 1 -- 172.21.15.16:0/1895392199 shutdown_connections 2023-10-16T12:11:05.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.679+0000 7fe073870700 1 --2- 172.21.15.16:0/1895392199 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe06c0b5830 0x7fe06c15a090 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:05.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.679+0000 7fe073870700 1 --2- 172.21.15.16:0/1895392199 >> v2:172.21.15.49:3300/0 conn(0x7fe06c0cf4a0 0x7fe06c0726e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:05.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.679+0000 7fe073870700 1 --2- 172.21.15.16:0/1895392199 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe06c0cf880 0x7fe06c0b52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:05.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.679+0000 7fe073870700 1 -- 172.21.15.16:0/1895392199 >> 172.21.15.16:0/1895392199 conn(0x7fe06c0b1fd0 msgr2=0x7fe06c0b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:05.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.679+0000 7fe073870700 1 -- 172.21.15.16:0/1895392199 shutdown_connections 2023-10-16T12:11:05.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.679+0000 7fe073870700 1 -- 172.21.15.16:0/1895392199 wait complete. 2023-10-16T12:11:05.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.680+0000 7fe073870700 1 Processor -- start 2023-10-16T12:11:05.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.680+0000 7fe073870700 1 -- start start 2023-10-16T12:11:05.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.681+0000 7fe073870700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe06c0b5830 0x7fe06c158730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:05.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.681+0000 7fe073870700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe06c0cf4a0 0x7fe06c0b0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:05.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.681+0000 7fe073870700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe06c0cf880 0x7fe06c15a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:05.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.681+0000 7fe073870700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe06c158c70 con 0x7fe06c0cf880 2023-10-16T12:11:05.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.681+0000 7fe073870700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe06c0c9350 con 0x7fe06c0b5830 2023-10-16T12:11:05.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.681+0000 7fe073870700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe06c0c9630 con 0x7fe06c0cf4a0 2023-10-16T12:11:05.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.681+0000 7fe071e0d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe06c0cf880 0x7fe06c15a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:05.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.681+0000 7fe071e0d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe06c0cf880 0x7fe06c15a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33520/0 (socket says 172.21.15.16:33520) 2023-10-16T12:11:05.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.681+0000 7fe071e0d700 1 -- 172.21.15.16:0/1503532529 learned_addr learned my addr 172.21.15.16:0/1503532529 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:05.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.681+0000 7fe07160c700 1 --2- 172.21.15.16:0/1503532529 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe06c0b5830 0x7fe06c158730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:05.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.682+0000 7fe070e0b700 1 --2- 172.21.15.16:0/1503532529 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe06c0cf4a0 0x7fe06c0b0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:05.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.682+0000 7fe071e0d700 1 -- 172.21.15.16:0/1503532529 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe06c0cf4a0 msgr2=0x7fe06c0b0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:05.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.682+0000 7fe071e0d700 1 --2- 172.21.15.16:0/1503532529 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe06c0cf4a0 0x7fe06c0b0eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:05.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.682+0000 7fe071e0d700 1 -- 172.21.15.16:0/1503532529 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe06c0b5830 msgr2=0x7fe06c158730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:05.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.682+0000 7fe071e0d700 1 --2- 172.21.15.16:0/1503532529 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe06c0b5830 0x7fe06c158730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:05.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.682+0000 7fe071e0d700 1 -- 172.21.15.16:0/1503532529 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe05c0094a0 con 0x7fe06c0cf880 2023-10-16T12:11:05.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.682+0000 7fe071e0d700 1 --2- 172.21.15.16:0/1503532529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe06c0cf880 0x7fe06c15a320 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe06c0cdfc0 tx=0x7fe064004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:05.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.682+0000 7fe061ffb700 1 -- 172.21.15.16:0/1503532529 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0640099b0 con 0x7fe06c0cf880 2023-10-16T12:11:05.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.682+0000 7fe061ffb700 1 -- 172.21.15.16:0/1503532529 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe06400ce80 con 0x7fe06c0cf880 2023-10-16T12:11:05.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.682+0000 7fe061ffb700 1 -- 172.21.15.16:0/1503532529 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe064004490 con 0x7fe06c0cf880 2023-10-16T12:11:05.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.682+0000 7fe073870700 1 -- 172.21.15.16:0/1503532529 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe064009b30 con 0x7fe06c0cf880 2023-10-16T12:11:05.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.682+0000 7fe073870700 1 -- 172.21.15.16:0/1503532529 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe06c0c9bf0 con 0x7fe06c0cf880 2023-10-16T12:11:05.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.683+0000 7fe073870700 1 -- 172.21.15.16:0/1503532529 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe06c04ebb0 con 0x7fe06c0cf880 2023-10-16T12:11:05.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.684+0000 7fe061ffb700 1 -- 172.21.15.16:0/1503532529 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fe0640045f0 con 0x7fe06c0cf880 2023-10-16T12:11:05.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.684+0000 7fe061ffb700 1 --2- 172.21.15.16:0/1503532529 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fe05006af00 0x7fe05006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:05.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.684+0000 7fe061ffb700 1 -- 172.21.15.16:0/1503532529 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fe06408ab80 con 0x7fe06c0cf880 2023-10-16T12:11:05.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.684+0000 7fe07160c700 1 --2- 172.21.15.16:0/1503532529 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fe05006af00 0x7fe05006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:05.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.685+0000 7fe07160c700 1 --2- 172.21.15.16:0/1503532529 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fe05006af00 0x7fe05006d3c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe06c0c6680 tx=0x7fe05c009210 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:05.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.689+0000 7fe061ffb700 1 -- 172.21.15.16:0/1503532529 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe0640559d0 con 0x7fe06c0cf880 2023-10-16T12:11:05.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.893+0000 7fe073870700 1 -- 172.21.15.16:0/1503532529 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe06c15d920 con 0x7fe06c0cf880 2023-10-16T12:11:05.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.894+0000 7fe061ffb700 1 -- 172.21.15.16:0/1503532529 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18238 (secure 0 0 0) 0x7fe064020020 con 0x7fe06c0cf880 2023-10-16T12:11:05.896 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:11:05.897 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":25,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:10:44.080141+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:10:36.935773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T12:10:38.004870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-16T12:10:41.672025+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:11:05.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.896+0000 7fe073870700 1 -- 172.21.15.16:0/1503532529 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fe05006af00 msgr2=0x7fe05006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:05.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.896+0000 7fe073870700 1 --2- 172.21.15.16:0/1503532529 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fe05006af00 0x7fe05006d3c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe06c0c6680 tx=0x7fe05c009210 comp rx=0 tx=0).stop 2023-10-16T12:11:05.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.896+0000 7fe073870700 1 -- 172.21.15.16:0/1503532529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe06c0cf880 msgr2=0x7fe06c15a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:05.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.896+0000 7fe073870700 1 --2- 172.21.15.16:0/1503532529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe06c0cf880 0x7fe06c15a320 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe06c0cdfc0 tx=0x7fe064004ad0 comp rx=0 tx=0).stop 2023-10-16T12:11:05.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.896+0000 7fe073870700 1 -- 172.21.15.16:0/1503532529 shutdown_connections 2023-10-16T12:11:05.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.896+0000 7fe073870700 1 --2- 172.21.15.16:0/1503532529 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fe05006af00 0x7fe05006d3c0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:05.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.896+0000 7fe073870700 1 --2- 172.21.15.16:0/1503532529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe06c0cf880 0x7fe06c15a320 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:05.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.896+0000 7fe073870700 1 --2- 172.21.15.16:0/1503532529 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe06c0b5830 0x7fe06c158730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:05.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.896+0000 7fe073870700 1 --2- 172.21.15.16:0/1503532529 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe06c0cf4a0 0x7fe06c0b0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:05.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.896+0000 7fe073870700 1 -- 172.21.15.16:0/1503532529 >> 172.21.15.16:0/1503532529 conn(0x7fe06c0b1fd0 msgr2=0x7fe06c1599b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:05.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.897+0000 7fe073870700 1 -- 172.21.15.16:0/1503532529 shutdown_connections 2023-10-16T12:11:05.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:05.897+0000 7fe073870700 1 -- 172.21.15.16:0/1503532529 wait complete. 2023-10-16T12:11:05.911 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-10-16T12:11:05.912 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-10-16T12:11:05.912 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-10-16T12:11:05.913 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-10-16T12:11:05.913 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-10-16T12:11:05.914 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-10-16T12:11:05.914 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-10-16T12:11:05.915 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-10-16T12:11:06.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.118+0000 7f1a424f7700 1 Processor -- start 2023-10-16T12:11:06.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.118+0000 7f1a424f7700 1 -- start start 2023-10-16T12:11:06.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.119+0000 7f1a424f7700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1a3c1599f0 0x7f1a3c15ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.119+0000 7f1a424f7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1a3c071250 0x7f1a3c0716d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.119+0000 7f1a424f7700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a3c15e300 con 0x7f1a3c06c800 2023-10-16T12:11:06.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.119+0000 7f1a424f7700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a3c15d240 con 0x7f1a3c071250 2023-10-16T12:11:06.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.119+0000 7f1a424f7700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a3c071c10 con 0x7f1a3c1599f0 2023-10-16T12:11:06.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.120+0000 7f1a3b7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1a3c071250 0x7f1a3c0716d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.120+0000 7f1a3b7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1a3c071250 0x7f1a3c0716d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33528/0 (socket says 172.21.15.16:33528) 2023-10-16T12:11:06.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.120+0000 7f1a3b7fe700 1 -- 172.21.15.16:0/60699101 learned_addr learned my addr 172.21.15.16:0/60699101 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:06.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.120+0000 7f1a3bfff700 1 --2- 172.21.15.16:0/60699101 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1a3c1599f0 0x7f1a3c15ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.120+0000 7f1a3b7fe700 1 -- 172.21.15.16:0/60699101 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1a3c1599f0 msgr2=0x7f1a3c15ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:06.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.120+0000 7f1a3b7fe700 1 --2- 172.21.15.16:0/60699101 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1a3c1599f0 0x7f1a3c15ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.120+0000 7f1a3b7fe700 1 -- 172.21.15.16:0/60699101 >> v1:172.21.15.49:6789/0 conn(0x7f1a3c06c800 legacy=0x7f1a3c06cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.120+0000 7f1a3b7fe700 1 -- 172.21.15.16:0/60699101 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a3c0b2fd0 con 0x7f1a3c071250 2023-10-16T12:11:06.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.120+0000 7f1a3b7fe700 1 --2- 172.21.15.16:0/60699101 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a3c071250 0x7f1a3c0716d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f1a3000a990 tx=0x7f1a30010730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbb2eb80103e04d8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:06.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.121+0000 7f1a3affd700 1 -- 172.21.15.16:0/60699101 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a3000ace0 con 0x7f1a3c071250 2023-10-16T12:11:06.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.121+0000 7f1a3affd700 1 -- 172.21.15.16:0/60699101 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a3000ae40 con 0x7f1a3c071250 2023-10-16T12:11:06.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.121+0000 7f1a3affd700 1 -- 172.21.15.16:0/60699101 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a3000b270 con 0x7f1a3c071250 2023-10-16T12:11:06.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.127+0000 7f1a424f7700 1 -- 172.21.15.16:0/60699101 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a3c071250 msgr2=0x7f1a3c0716d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.127+0000 7f1a424f7700 1 --2- 172.21.15.16:0/60699101 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a3c071250 0x7f1a3c0716d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f1a3000a990 tx=0x7f1a30010730 comp rx=0 tx=0).stop 2023-10-16T12:11:06.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.127+0000 7f1a424f7700 1 -- 172.21.15.16:0/60699101 shutdown_connections 2023-10-16T12:11:06.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.127+0000 7f1a424f7700 1 --2- 172.21.15.16:0/60699101 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1a3c1599f0 0x7f1a3c15ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.127+0000 7f1a424f7700 1 --2- 172.21.15.16:0/60699101 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a3c071250 0x7f1a3c0716d0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.127+0000 7f1a424f7700 1 -- 172.21.15.16:0/60699101 >> 172.21.15.16:0/60699101 conn(0x7f1a3c06a7e0 msgr2=0x7f1a3c06abf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:06.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.128+0000 7f1a424f7700 1 -- 172.21.15.16:0/60699101 shutdown_connections 2023-10-16T12:11:06.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.128+0000 7f1a424f7700 1 -- 172.21.15.16:0/60699101 wait complete. 2023-10-16T12:11:06.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.130+0000 7f1a424f7700 1 Processor -- start 2023-10-16T12:11:06.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.132+0000 7fb61c4f5700 1 Processor -- start 2023-10-16T12:11:06.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.131+0000 7f1a424f7700 1 -- start start 2023-10-16T12:11:06.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.132+0000 7f1a424f7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a3c06c800 0x7f1a3c156430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.132+0000 7f1a424f7700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1a3c071250 0x7f1a3c0ad690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.132+0000 7f1a424f7700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1a3c1599f0 0x7f1a3c0ab630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.132+0000 7f1a424f7700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a3c156970 con 0x7f1a3c06c800 2023-10-16T12:11:06.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.132+0000 7f1a424f7700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a3c0ae370 con 0x7f1a3c1599f0 2023-10-16T12:11:06.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.132+0000 7f1a424f7700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a3c0a94e0 con 0x7f1a3c071250 2023-10-16T12:11:06.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.132+0000 7f1a3bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a3c06c800 0x7f1a3c156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.132+0000 7f1a3bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a3c06c800 0x7f1a3c156430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33530/0 (socket says 172.21.15.16:33530) 2023-10-16T12:11:06.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.132+0000 7f1a3bfff700 1 -- 172.21.15.16:0/2602751565 learned_addr learned my addr 172.21.15.16:0/2602751565 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:06.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.133+0000 7f1a3bfff700 1 -- 172.21.15.16:0/2602751565 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1a3c071250 msgr2=0x7f1a3c0ad690 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-16T12:11:06.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.133+0000 7f1a3bfff700 1 --2- 172.21.15.16:0/2602751565 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1a3c071250 0x7f1a3c0ad690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.133+0000 7f1a3bfff700 1 -- 172.21.15.16:0/2602751565 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1a3c1599f0 msgr2=0x7f1a3c0ab630 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-16T12:11:06.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.133+0000 7f1a3bfff700 1 --2- 172.21.15.16:0/2602751565 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1a3c1599f0 0x7f1a3c0ab630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.133+0000 7f1a3bfff700 1 -- 172.21.15.16:0/2602751565 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a3000a5f0 con 0x7f1a3c06c800 2023-10-16T12:11:06.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.133+0000 7f1a3bfff700 1 --2- 172.21.15.16:0/2602751565 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a3c06c800 0x7f1a3c156430 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f1a34009d10 tx=0x7f1a3400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:06.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.134+0000 7f1a38ff9700 1 -- 172.21.15.16:0/2602751565 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a3400cd60 con 0x7f1a3c06c800 2023-10-16T12:11:06.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.134+0000 7f1a38ff9700 1 -- 172.21.15.16:0/2602751565 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a3400f040 con 0x7f1a3c06c800 2023-10-16T12:11:06.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.134+0000 7f1a38ff9700 1 -- 172.21.15.16:0/2602751565 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a340146a0 con 0x7f1a3c06c800 2023-10-16T12:11:06.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.134+0000 7f1a424f7700 1 -- 172.21.15.16:0/2602751565 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a3c1ffa90 con 0x7f1a3c06c800 2023-10-16T12:11:06.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.134+0000 7f1a424f7700 1 -- 172.21.15.16:0/2602751565 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a3c0a8e90 con 0x7f1a3c06c800 2023-10-16T12:11:06.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.132+0000 7fb61c4f5700 1 -- start start 2023-10-16T12:11:06.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.134+0000 7fb61c4f5700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb6140ceff0 0x7fb6140cf470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.134+0000 7fb61c4f5700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fb6140cc170 0x7fb6140cc5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.134+0000 7fb61c4f5700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb614070f70 con 0x7fb614161c90 2023-10-16T12:11:06.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.134+0000 7fb61c4f5700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb614071530 con 0x7fb6140cc170 2023-10-16T12:11:06.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.134+0000 7fb61c4f5700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6140cf9b0 con 0x7fb6140ceff0 2023-10-16T12:11:06.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.134+0000 7fb619a90700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb6140ceff0 0x7fb6140cf470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.134+0000 7fb619a90700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb6140ceff0 0x7fb6140cf470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:50026/0 (socket says 172.21.15.16:50026) 2023-10-16T12:11:06.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.134+0000 7fb619a90700 1 -- 172.21.15.16:0/1186294078 learned_addr learned my addr 172.21.15.16:0/1186294078 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:06.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.134+0000 7fb61aa92700 1 --2- 172.21.15.16:0/1186294078 >> v2:172.21.15.49:3300/0 conn(0x7fb6140cc170 0x7fb6140cc5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.135+0000 7fb619a90700 1 -- 172.21.15.16:0/1186294078 >> v2:172.21.15.49:3300/0 conn(0x7fb6140cc170 msgr2=0x7fb6140cc5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:06.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.135+0000 7fb619a90700 1 --2- 172.21.15.16:0/1186294078 >> v2:172.21.15.49:3300/0 conn(0x7fb6140cc170 0x7fb6140cc5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.135+0000 7fb619a90700 1 -- 172.21.15.16:0/1186294078 >> v1:172.21.15.49:6789/0 conn(0x7fb614161c90 legacy=0x7fb614070860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:06.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.135+0000 7fb619a90700 1 -- 172.21.15.16:0/1186294078 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6140b38b0 con 0x7fb6140ceff0 2023-10-16T12:11:06.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.135+0000 7fb619a90700 1 --2- 172.21.15.16:0/1186294078 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb6140ceff0 0x7fb6140cf470 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb6080098f0 tx=0x7fb608009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db34eada717d2400 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:06.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.136+0000 7fb61928f700 1 -- 172.21.15.16:0/1186294078 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb608004230 con 0x7fb6140ceff0 2023-10-16T12:11:06.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.136+0000 7fb61928f700 1 -- 172.21.15.16:0/1186294078 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb608004390 con 0x7fb6140ceff0 2023-10-16T12:11:06.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.136+0000 7fb61928f700 1 -- 172.21.15.16:0/1186294078 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6080036a0 con 0x7fb6140ceff0 2023-10-16T12:11:06.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.136+0000 7f1a424f7700 1 -- 172.21.15.16:0/2602751565 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1a1c000fc0 con 0x7f1a3c06c800 2023-10-16T12:11:06.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.136+0000 7f56249bf700 1 Processor -- start 2023-10-16T12:11:06.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.136+0000 7f56249bf700 1 -- start start 2023-10-16T12:11:06.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.137+0000 7fb61c4f5700 1 -- 172.21.15.16:0/1186294078 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb6140ceff0 msgr2=0x7fb6140cf470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.137+0000 7fb61c4f5700 1 --2- 172.21.15.16:0/1186294078 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb6140ceff0 0x7fb6140cf470 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb6080098f0 tx=0x7fb608009c00 comp rx=0 tx=0).stop 2023-10-16T12:11:06.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.137+0000 7f56249bf700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f56200726e0 0x7f56200cf4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.137+0000 7f56249bf700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f56200b52f0 0x7f56200b5750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.137+0000 7f56249bf700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f562015a090 0x7f562015a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.137+0000 7f56249bf700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f562015bc80 con 0x7f56200b52f0 2023-10-16T12:11:06.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.137+0000 7f56249bf700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5620162350 con 0x7f562015a090 2023-10-16T12:11:06.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.137+0000 7f56249bf700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f562015b070 con 0x7f56200726e0 2023-10-16T12:11:06.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.137+0000 7f1a38ff9700 1 -- 172.21.15.16:0/2602751565 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f1a34014800 con 0x7f1a3c06c800 2023-10-16T12:11:06.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.138+0000 7f2f7f2bb700 1 Processor -- start 2023-10-16T12:11:06.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.138+0000 7f561ed9d700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f562015a090 0x7f562015a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.138+0000 7f561ed9d700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f562015a090 0x7f562015a4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:60840/0 (socket says 172.21.15.16:60840) 2023-10-16T12:11:06.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.138+0000 7f561ed9d700 1 -- 172.21.15.16:0/2729322346 learned_addr learned my addr 172.21.15.16:0/2729322346 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:06.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.138+0000 7f2f7f2bb700 1 -- start start 2023-10-16T12:11:06.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.138+0000 7f561ed9d700 1 -- 172.21.15.16:0/2729322346 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f56200726e0 msgr2=0x7f56200cf4a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-16T12:11:06.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.138+0000 7f561ed9d700 1 --2- 172.21.15.16:0/2729322346 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f56200726e0 0x7f56200cf4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.138+0000 7f561ed9d700 1 -- 172.21.15.16:0/2729322346 >> v2:172.21.15.16:3300/0 conn(0x7f56200b52f0 msgr2=0x7f56200b5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:06.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.138+0000 7f561ed9d700 1 --2- 172.21.15.16:0/2729322346 >> v2:172.21.15.16:3300/0 conn(0x7f56200b52f0 0x7f56200b5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.138+0000 7f561ed9d700 1 -- 172.21.15.16:0/2729322346 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f562015b350 con 0x7f562015a090 2023-10-16T12:11:06.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.138+0000 7f1a38ff9700 1 --2- 172.21.15.16:0/2602751565 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f1a2806af00 0x7f1a2806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.138+0000 7f1a38ff9700 1 -- 172.21.15.16:0/2602751565 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f1a3408a940 con 0x7f1a3c06c800 2023-10-16T12:11:06.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.138+0000 7f561ed9d700 1 --2- 172.21.15.16:0/2729322346 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f562015a090 0x7f562015a4f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5620071da0 tx=0x7f5610009bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5cf44039db79a4b6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:06.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.138+0000 7f1a38ff9700 1 --2- 172.21.15.16:0/2602751565 >> [v2:172.21.15.16:6817/375531758,v1:172.21.15.16:6819/375531758] conn(0x7f1a280744c0 0x7f1a28076900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.138+0000 7f1a38ff9700 1 -- 172.21.15.16:0/2602751565 --> [v2:172.21.15.16:6817/375531758,v1:172.21.15.16:6819/375531758] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1a28076fd0 con 0x7f1a280744c0 2023-10-16T12:11:06.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.138+0000 7f1a38ff9700 1 -- 172.21.15.16:0/2602751565 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1a3408ad90 con 0x7f1a3c06c800 2023-10-16T12:11:06.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.138+0000 7f561d59a700 1 -- 172.21.15.16:0/2729322346 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f561000b870 con 0x7f562015a090 2023-10-16T12:11:06.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.139+0000 7f561d59a700 1 -- 172.21.15.16:0/2729322346 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f561000b9d0 con 0x7f562015a090 2023-10-16T12:11:06.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.139+0000 7f561d59a700 1 -- 172.21.15.16:0/2729322346 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f561000bd60 con 0x7f562015a090 2023-10-16T12:11:06.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.139+0000 7f1a3b7fe700 1 --2- 172.21.15.16:0/2602751565 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f1a2806af00 0x7f1a2806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.139+0000 7f2f7f2bb700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2f78071250 0x7f2f780716b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.139+0000 7f2f7f2bb700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f2f7815a030 0x7f2f7815a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.139+0000 7f2f7f2bb700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f78071bf0 con 0x7f2f781599f0 2023-10-16T12:11:06.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.139+0000 7f1a3b7fe700 1 --2- 172.21.15.16:0/2602751565 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f1a2806af00 0x7f1a2806d3c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1a3c1590a0 tx=0x7f1a3000d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:06.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.139+0000 7f2f7f2bb700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f7815d240 con 0x7f2f7815a030 2023-10-16T12:11:06.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.139+0000 7f2f7f2bb700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f780b2fd0 con 0x7f2f78071250 2023-10-16T12:11:06.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.139+0000 7f2f7d057700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2f781599f0 0x7f2f7815ddc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51058/0 (socket says 172.21.15.16:51058) 2023-10-16T12:11:06.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.139+0000 7f2f7d858700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f2f7815a030 0x7f2f7815a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.139+0000 7f2f7c856700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2f78071250 0x7f2f780716b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.139+0000 7f2f7d057700 1 -- 172.21.15.16:0/1917470803 learned_addr learned my addr 172.21.15.16:0/1917470803 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:06.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.139+0000 7f5da98e4700 1 Processor -- start 2023-10-16T12:11:06.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.139+0000 7f5da98e4700 1 -- start start 2023-10-16T12:11:06.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.140+0000 7f56249bf700 1 -- 172.21.15.16:0/2729322346 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f562015a090 msgr2=0x7f562015a4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.140+0000 7f56249bf700 1 --2- 172.21.15.16:0/2729322346 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f562015a090 0x7f562015a4f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5620071da0 tx=0x7f5610009bd0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.140+0000 7f2f7d858700 1 -- 172.21.15.16:0/1917470803 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2f78071250 msgr2=0x7f2f780716b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:06.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.140+0000 7f2f7d858700 1 --2- 172.21.15.16:0/1917470803 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2f78071250 0x7f2f780716b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.140+0000 7f2f7d858700 1 -- 172.21.15.16:0/1917470803 >> v1:172.21.15.16:6789/0 conn(0x7f2f781599f0 legacy=0x7f2f7815ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.140+0000 7f2f7d858700 1 -- 172.21.15.16:0/1917470803 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f780b32b0 con 0x7f2f7815a030 2023-10-16T12:11:06.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.140+0000 7f56249bf700 1 -- 172.21.15.16:0/2729322346 shutdown_connections 2023-10-16T12:11:06.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.140+0000 7f56249bf700 1 --2- 172.21.15.16:0/2729322346 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f562015a090 0x7f562015a4f0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.140+0000 7f56249bf700 1 --2- 172.21.15.16:0/2729322346 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f56200726e0 0x7f56200cf4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.140+0000 7f56249bf700 1 --2- 172.21.15.16:0/2729322346 >> v2:172.21.15.16:3300/0 conn(0x7f56200b52f0 0x7f56200b5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.140+0000 7f56249bf700 1 -- 172.21.15.16:0/2729322346 >> 172.21.15.16:0/2729322346 conn(0x7f56200b1fd0 msgr2=0x7f56200b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:06.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.140+0000 7f56249bf700 1 -- 172.21.15.16:0/2729322346 shutdown_connections 2023-10-16T12:11:06.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.140+0000 7f56249bf700 1 -- 172.21.15.16:0/2729322346 wait complete. 2023-10-16T12:11:06.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.140+0000 7f2f7d858700 1 --2- 172.21.15.16:0/1917470803 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2f7815a030 0x7f2f7815a4c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2f70009d90 tx=0x7f2f7000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9c3653f374ac4aaa server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:06.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.140+0000 7f2f6ffff700 1 -- 172.21.15.16:0/1917470803 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f7000bdd0 con 0x7f2f7815a030 2023-10-16T12:11:06.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.140+0000 7f2f6ffff700 1 -- 172.21.15.16:0/1917470803 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f70011460 con 0x7f2f7815a030 2023-10-16T12:11:06.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.140+0000 7f2f6ffff700 1 -- 172.21.15.16:0/1917470803 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f70011810 con 0x7f2f7815a030 2023-10-16T12:11:06.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.141+0000 7f5da98e4700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5da41652b0 0x7f5da40ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.141+0000 7f5da98e4700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f5da40cab20 0x7f5da40cafb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.141+0000 7f5da98e4700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5da40ca600 con 0x7f5da40ca180 2023-10-16T12:11:06.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.141+0000 7f5da98e4700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5da40cb4f0 con 0x7f5da40cab20 2023-10-16T12:11:06.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.141+0000 7f5da98e4700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5da406c1a0 con 0x7f5da41652b0 2023-10-16T12:11:06.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.141+0000 7f5da2ffd700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5da41652b0 0x7f5da40ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.141+0000 7f5da27fc700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f5da40ca180 0x7f5da406ba90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51066/0 (socket says 172.21.15.16:51066) 2023-10-16T12:11:06.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.141+0000 7f5da27fc700 1 -- 172.21.15.16:0/580149113 learned_addr learned my addr 172.21.15.16:0/580149113 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:06.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.141+0000 7f2f7f2bb700 1 -- 172.21.15.16:0/1917470803 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2f7815a030 msgr2=0x7f2f7815a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.141+0000 7f2f7f2bb700 1 --2- 172.21.15.16:0/1917470803 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2f7815a030 0x7f2f7815a4c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2f70009d90 tx=0x7f2f7000b330 comp rx=0 tx=0).stop 2023-10-16T12:11:06.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.141+0000 7fb61c4f5700 1 -- 172.21.15.16:0/1186294078 shutdown_connections 2023-10-16T12:11:06.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.141+0000 7fb61c4f5700 1 --2- 172.21.15.16:0/1186294078 >> v2:172.21.15.49:3300/0 conn(0x7fb6140cc170 0x7fb6140cc5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.141+0000 7fb61c4f5700 1 --2- 172.21.15.16:0/1186294078 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb6140ceff0 0x7fb6140cf470 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.141+0000 7fb61c4f5700 1 -- 172.21.15.16:0/1186294078 >> 172.21.15.16:0/1186294078 conn(0x7fb6140b4cd0 msgr2=0x7fb61406a7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:06.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.141+0000 7f2f7f2bb700 1 -- 172.21.15.16:0/1917470803 shutdown_connections 2023-10-16T12:11:06.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.141+0000 7f2f7f2bb700 1 --2- 172.21.15.16:0/1917470803 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2f7815a030 0x7f2f7815a4c0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.141+0000 7f2f7f2bb700 1 --2- 172.21.15.16:0/1917470803 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2f78071250 0x7f2f780716b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.141+0000 7f2f7f2bb700 1 -- 172.21.15.16:0/1917470803 >> 172.21.15.16:0/1917470803 conn(0x7f2f7806a7e0 msgr2=0x7f2f7806abf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:06.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.142+0000 7fb61c4f5700 1 -- 172.21.15.16:0/1186294078 shutdown_connections 2023-10-16T12:11:06.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.141+0000 7f5da2ffd700 1 -- 172.21.15.16:0/580149113 >> v2:172.21.15.49:3300/0 conn(0x7f5da40cab20 msgr2=0x7f5da40cafb0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-16T12:11:06.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.141+0000 7f5da2ffd700 1 --2- 172.21.15.16:0/580149113 >> v2:172.21.15.49:3300/0 conn(0x7f5da40cab20 0x7f5da40cafb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.142+0000 7fb61c4f5700 1 -- 172.21.15.16:0/1186294078 wait complete. 2023-10-16T12:11:06.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.141+0000 7f5da2ffd700 1 -- 172.21.15.16:0/580149113 >> v1:172.21.15.16:6789/0 conn(0x7f5da40ca180 legacy=0x7f5da406ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.141+0000 7f5da2ffd700 1 -- 172.21.15.16:0/580149113 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5da40c2ec0 con 0x7f5da41652b0 2023-10-16T12:11:06.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.142+0000 7f2f7f2bb700 1 -- 172.21.15.16:0/1917470803 shutdown_connections 2023-10-16T12:11:06.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.142+0000 7f5da2ffd700 1 --2- 172.21.15.16:0/580149113 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5da41652b0 0x7f5da40ae8e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5d9c009d90 tx=0x7f5d9c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e20c03b5fd62ee9d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:06.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.142+0000 7f56249bf700 1 Processor -- start 2023-10-16T12:11:06.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.142+0000 7f56249bf700 1 -- start start 2023-10-16T12:11:06.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.143+0000 7f56249bf700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f56200726e0 0x7f56200b3620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.143+0000 7fb61c4f5700 1 Processor -- start 2023-10-16T12:11:06.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.142+0000 7f2f7f2bb700 1 -- 172.21.15.16:0/1917470803 wait complete. 2023-10-16T12:11:06.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.144+0000 7f2f7f2bb700 1 Processor -- start 2023-10-16T12:11:06.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.144+0000 7f2f7f2bb700 1 -- start start 2023-10-16T12:11:06.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.143+0000 7f56249bf700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56200b52f0 0x7f56200b3b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.143+0000 7f56249bf700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f562015a090 0x7f56200b40a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.143+0000 7f56249bf700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56200cea60 con 0x7f56200b52f0 2023-10-16T12:11:06.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.143+0000 7f56249bf700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56201452f0 con 0x7f562015a090 2023-10-16T12:11:06.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.143+0000 7f56249bf700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5620144830 con 0x7f56200726e0 2023-10-16T12:11:06.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.144+0000 7f561dd9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56200b52f0 0x7f56200b3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.144+0000 7f561dd9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56200b52f0 0x7f56200b3b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33566/0 (socket says 172.21.15.16:33566) 2023-10-16T12:11:06.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.144+0000 7f561dd9b700 1 -- 172.21.15.16:0/2388481391 learned_addr learned my addr 172.21.15.16:0/2388481391 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:06.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.144+0000 7f561e59c700 1 --2- 172.21.15.16:0/2388481391 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f56200726e0 0x7f56200b3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.144+0000 7f561ed9d700 1 --2- 172.21.15.16:0/2388481391 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f562015a090 0x7f56200b40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.144+0000 7f561dd9b700 1 -- 172.21.15.16:0/2388481391 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f56200726e0 msgr2=0x7f56200b3620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.144+0000 7f561dd9b700 1 --2- 172.21.15.16:0/2388481391 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f56200726e0 0x7f56200b3620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.144+0000 7f561dd9b700 1 -- 172.21.15.16:0/2388481391 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f562015a090 msgr2=0x7f56200b40a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.144+0000 7f561dd9b700 1 --2- 172.21.15.16:0/2388481391 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f562015a090 0x7f56200b40a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.144+0000 7f561dd9b700 1 -- 172.21.15.16:0/2388481391 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5610009550 con 0x7f56200b52f0 2023-10-16T12:11:06.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.144+0000 7f561ed9d700 1 --2- 172.21.15.16:0/2388481391 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f562015a090 0x7f56200b40a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:11:06.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.145+0000 7f561dd9b700 1 --2- 172.21.15.16:0/2388481391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56200b52f0 0x7f56200b3b60 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f561800ba80 tx=0x7f561800bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:06.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.144+0000 7fb61c4f5700 1 -- start start 2023-10-16T12:11:06.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.145+0000 7fb61c4f5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6140cc170 0x7fb614138a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.145+0000 7fb61c4f5700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb6140ceff0 0x7fb6140ccc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.145+0000 7fb61c4f5700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb614161c90 0x7fb61413ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.145+0000 7fb61c4f5700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb614131e50 con 0x7fb6140cc170 2023-10-16T12:11:06.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.145+0000 7fb61c4f5700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6140cd740 con 0x7fb6140ceff0 2023-10-16T12:11:06.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.145+0000 7fb61c4f5700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6140cda20 con 0x7fb614161c90 2023-10-16T12:11:06.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.145+0000 7fb61aa92700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb614161c90 0x7fb61413ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.145+0000 7f2f7f2bb700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f78071250 0x7f2f78152f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.145+0000 7f2f7f2bb700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2f781599f0 0x7f2f781ffb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.145+0000 7f2f7f2bb700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2f7815a030 0x7f2f781e7410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.145+0000 7f2f7f2bb700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f781534a0 con 0x7f2f78071250 2023-10-16T12:11:06.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.145+0000 7f2f7f2bb700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f780b0c80 con 0x7f2f781599f0 2023-10-16T12:11:06.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.145+0000 7f2f7f2bb700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f78200af0 con 0x7f2f7815a030 2023-10-16T12:11:06.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.145+0000 7f5da1ffb700 1 -- 172.21.15.16:0/580149113 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d9c00e070 con 0x7f5da41652b0 2023-10-16T12:11:06.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.145+0000 7f5da1ffb700 1 -- 172.21.15.16:0/580149113 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d9c00bad0 con 0x7f5da41652b0 2023-10-16T12:11:06.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.145+0000 7f5da1ffb700 1 -- 172.21.15.16:0/580149113 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d9c00be80 con 0x7f5da41652b0 2023-10-16T12:11:06.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.145+0000 7fb61aa92700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb614161c90 0x7fb61413ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:50060/0 (socket says 172.21.15.16:50060) 2023-10-16T12:11:06.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.145+0000 7fb61aa92700 1 -- 172.21.15.16:0/579057281 learned_addr learned my addr 172.21.15.16:0/579057281 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:06.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.145+0000 7f2f7d057700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f78071250 0x7f2f78152f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.145+0000 7fb619a90700 1 --2- 172.21.15.16:0/579057281 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb6140ceff0 0x7fb6140ccc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.145+0000 7fb61a291700 1 --2- 172.21.15.16:0/579057281 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6140cc170 0x7fb614138a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.145+0000 7f2f7d057700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f78071250 0x7f2f78152f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33574/0 (socket says 172.21.15.16:33574) 2023-10-16T12:11:06.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.145+0000 7f2f7c856700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2f781599f0 0x7f2f781ffb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.145+0000 7f2f7d057700 1 -- 172.21.15.16:0/835743600 learned_addr learned my addr 172.21.15.16:0/835743600 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:06.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.145+0000 7f2f7d858700 1 --2- 172.21.15.16:0/835743600 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2f7815a030 0x7f2f781e7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.145+0000 7f2f7d057700 1 -- 172.21.15.16:0/835743600 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2f7815a030 msgr2=0x7f2f781e7410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.145+0000 7f2f7d057700 1 --2- 172.21.15.16:0/835743600 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2f7815a030 0x7f2f781e7410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.145+0000 7f2f7d057700 1 -- 172.21.15.16:0/835743600 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2f781599f0 msgr2=0x7f2f781ffb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.145+0000 7f2f7d057700 1 --2- 172.21.15.16:0/835743600 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2f781599f0 0x7f2f781ffb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.145+0000 7f2f7d057700 1 -- 172.21.15.16:0/835743600 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f70009a70 con 0x7f2f78071250 2023-10-16T12:11:06.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.146+0000 7fb619a90700 1 -- 172.21.15.16:0/579057281 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb614161c90 msgr2=0x7fb61413ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.146+0000 7fb619a90700 1 --2- 172.21.15.16:0/579057281 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb614161c90 0x7fb61413ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.146+0000 7fb619a90700 1 -- 172.21.15.16:0/579057281 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6140cc170 msgr2=0x7fb614138a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.146+0000 7fb619a90700 1 --2- 172.21.15.16:0/579057281 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6140cc170 0x7fb614138a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.146+0000 7fb619a90700 1 -- 172.21.15.16:0/579057281 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb608009550 con 0x7fb6140ceff0 2023-10-16T12:11:06.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.146+0000 7fb619a90700 1 --2- 172.21.15.16:0/579057281 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb6140ceff0 0x7fb6140ccc80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb608015000 tx=0x7fb608004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:06.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.146+0000 7f2f7d057700 1 --2- 172.21.15.16:0/835743600 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f78071250 0x7f2f78152f60 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f2f6800c520 tx=0x7f2f6800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:06.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.145+0000 7f560effd700 1 -- 172.21.15.16:0/2388481391 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f561800c9b0 con 0x7f56200b52f0 2023-10-16T12:11:06.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.145+0000 7f560effd700 1 -- 172.21.15.16:0/2388481391 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f561800cb30 con 0x7f56200b52f0 2023-10-16T12:11:06.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.145+0000 7f560effd700 1 -- 172.21.15.16:0/2388481391 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5618014440 con 0x7f56200b52f0 2023-10-16T12:11:06.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.146+0000 7f5da98e4700 1 -- 172.21.15.16:0/580149113 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5da41652b0 msgr2=0x7f5da40ae8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.146+0000 7f5da98e4700 1 --2- 172.21.15.16:0/580149113 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5da41652b0 0x7f5da40ae8e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5d9c009d90 tx=0x7f5d9c00b330 comp rx=0 tx=0).stop 2023-10-16T12:11:06.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.146+0000 7f5da98e4700 1 -- 172.21.15.16:0/580149113 shutdown_connections 2023-10-16T12:11:06.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.146+0000 7f5da98e4700 1 --2- 172.21.15.16:0/580149113 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5da41652b0 0x7f5da40ae8e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.146+0000 7f5da98e4700 1 --2- 172.21.15.16:0/580149113 >> v2:172.21.15.49:3300/0 conn(0x7f5da40cab20 0x7f5da40cafb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.146+0000 7f5da98e4700 1 -- 172.21.15.16:0/580149113 >> 172.21.15.16:0/580149113 conn(0x7f5da40cc5b0 msgr2=0x7f5da40cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:06.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.146+0000 7fb612ffd700 1 -- 172.21.15.16:0/579057281 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb608003c10 con 0x7fb6140ceff0 2023-10-16T12:11:06.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.146+0000 7fb612ffd700 1 -- 172.21.15.16:0/579057281 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb60801f070 con 0x7fb6140ceff0 2023-10-16T12:11:06.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.146+0000 7fb612ffd700 1 -- 172.21.15.16:0/579057281 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb60800b660 con 0x7fb6140ceff0 2023-10-16T12:11:06.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.146+0000 7f56249bf700 1 -- 172.21.15.16:0/2388481391 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56200b1dc0 con 0x7f56200b52f0 2023-10-16T12:11:06.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.146+0000 7f56249bf700 1 -- 172.21.15.16:0/2388481391 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5620145010 con 0x7f56200b52f0 2023-10-16T12:11:06.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.147+0000 7f56249bf700 1 -- 172.21.15.16:0/2388481391 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f562015b910 con 0x7f56200b52f0 2023-10-16T12:11:06.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.147+0000 7f2f6dffb700 1 -- 172.21.15.16:0/835743600 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f68005d70 con 0x7f2f78071250 2023-10-16T12:11:06.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.147+0000 7f2f6dffb700 1 -- 172.21.15.16:0/835743600 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f680070d0 con 0x7f2f78071250 2023-10-16T12:11:06.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.147+0000 7f2f7f2bb700 1 -- 172.21.15.16:0/835743600 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f78200530 con 0x7f2f78071250 2023-10-16T12:11:06.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.147+0000 7fb61c4f5700 1 -- 172.21.15.16:0/579057281 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6140cdd00 con 0x7fb6140ceff0 2023-10-16T12:11:06.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.147+0000 7f2f7f2bb700 1 -- 172.21.15.16:0/835743600 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f780b0910 con 0x7f2f78071250 2023-10-16T12:11:06.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.146+0000 7f26856bc700 1 Processor -- start 2023-10-16T12:11:06.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.146+0000 7f26856bc700 1 -- start start 2023-10-16T12:11:06.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.147+0000 7f26856bc700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f268015c330 0x7f2680163480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.147+0000 7f26856bc700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2680071470 0x7f2680162d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.147+0000 7f26856bc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f26800b2a50 0x7f26800b2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.148+0000 7f26856bc700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26800b3420 con 0x7f26800b2a50 2023-10-16T12:11:06.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.148+0000 7f26856bc700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2680163280 con 0x7f268015c330 2023-10-16T12:11:06.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.148+0000 7f26856bc700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2680070860 con 0x7f2680071470 2023-10-16T12:11:06.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.147+0000 7f1a40a94700 1 --2- 172.21.15.16:0/2602751565 >> [v2:172.21.15.16:6817/375531758,v1:172.21.15.16:6819/375531758] conn(0x7f1a280744c0 0x7f1a28076900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.148+0000 7f267f7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f26800b2a50 0x7f26800b2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.148+0000 7f267f7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f26800b2a50 0x7f26800b2ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33582/0 (socket says 172.21.15.16:33582) 2023-10-16T12:11:06.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.148+0000 7f267f7fe700 1 -- 172.21.15.16:0/1417599710 learned_addr learned my addr 172.21.15.16:0/1417599710 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:06.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.148+0000 7f2f6dffb700 1 -- 172.21.15.16:0/835743600 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f68007440 con 0x7f2f78071250 2023-10-16T12:11:06.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.147+0000 7fb61c4f5700 1 -- 172.21.15.16:0/579057281 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb61414c1f0 con 0x7fb6140ceff0 2023-10-16T12:11:06.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.148+0000 7f1a40a94700 1 --2- 172.21.15.16:0/2602751565 >> [v2:172.21.15.16:6817/375531758,v1:172.21.15.16:6819/375531758] conn(0x7f1a280744c0 0x7f1a28076900 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:06.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.148+0000 7fb610ff9700 1 -- 172.21.15.16:0/579057281 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb5f8000fc0 con 0x7fb6140ceff0 2023-10-16T12:11:06.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.148+0000 7f267f7fe700 1 -- 172.21.15.16:0/1417599710 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2680071470 msgr2=0x7f2680162d40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-16T12:11:06.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.148+0000 7f267f7fe700 1 --2- 172.21.15.16:0/1417599710 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2680071470 0x7f2680162d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.148+0000 7f267f7fe700 1 -- 172.21.15.16:0/1417599710 >> v2:172.21.15.49:3300/0 conn(0x7f268015c330 msgr2=0x7f2680163480 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-16T12:11:06.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.148+0000 7f267f7fe700 1 --2- 172.21.15.16:0/1417599710 >> v2:172.21.15.49:3300/0 conn(0x7f268015c330 0x7f2680163480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.148+0000 7f267f7fe700 1 -- 172.21.15.16:0/1417599710 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2680070b00 con 0x7f26800b2a50 2023-10-16T12:11:06.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.149+0000 7f267f7fe700 1 --2- 172.21.15.16:0/1417599710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26800b2a50 0x7f26800b2ee0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f268006e370 tx=0x7f267800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f71c2b7a58681085 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:06.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.149+0000 7f5da98e4700 1 -- 172.21.15.16:0/580149113 shutdown_connections 2023-10-16T12:11:06.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.149+0000 7f560effd700 1 -- 172.21.15.16:0/2388481391 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f561801d3b0 con 0x7f56200b52f0 2023-10-16T12:11:06.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.149+0000 7f560effd700 1 --2- 172.21.15.16:0/2388481391 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f560006afd0 0x7f560006d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.149+0000 7f560effd700 1 -- 172.21.15.16:0/2388481391 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f5618006e40 con 0x7f56200b52f0 2023-10-16T12:11:06.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.150+0000 7f561e59c700 1 --2- 172.21.15.16:0/2388481391 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f560006afd0 0x7f560006d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.150+0000 7f560effd700 1 --2- 172.21.15.16:0/2388481391 >> [v2:172.21.15.49:6800/682437317,v1:172.21.15.49:6801/682437317] conn(0x7f5600074610 0x7f5600076a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.150+0000 7f560effd700 1 -- 172.21.15.16:0/2388481391 --> [v2:172.21.15.49:6800/682437317,v1:172.21.15.49:6801/682437317] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5600077120 con 0x7f5600074610 2023-10-16T12:11:06.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.150+0000 7f560effd700 1 -- 172.21.15.16:0/2388481391 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5618089f60 con 0x7f56200b52f0 2023-10-16T12:11:06.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.150+0000 7f5da98e4700 1 -- 172.21.15.16:0/580149113 wait complete. 2023-10-16T12:11:06.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.150+0000 7fb612ffd700 1 -- 172.21.15.16:0/579057281 <== mon.1 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fb608021030 con 0x7fb6140ceff0 2023-10-16T12:11:06.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.150+0000 7f2f7f2bb700 1 -- 172.21.15.16:0/835743600 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2f58000fc0 con 0x7f2f78071250 2023-10-16T12:11:06.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.150+0000 7f1a38ff9700 1 -- 172.21.15.16:0/2602751565 <== osd.3 v2:172.21.15.16:6817/375531758 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7f1a28076fd0 con 0x7f1a280744c0 2023-10-16T12:11:06.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.150+0000 7f561ed9d700 1 --2- 172.21.15.16:0/2388481391 >> [v2:172.21.15.49:6800/682437317,v1:172.21.15.49:6801/682437317] conn(0x7f5600074610 0x7f5600076a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.150+0000 7f561e59c700 1 --2- 172.21.15.16:0/2388481391 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f560006afd0 0x7f560006d490 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f562015a060 tx=0x7f56140090e0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:06.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.151+0000 7f561ed9d700 1 --2- 172.21.15.16:0/2388481391 >> [v2:172.21.15.49:6800/682437317,v1:172.21.15.49:6801/682437317] conn(0x7f5600074610 0x7f5600076a50 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:06.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.150+0000 7f6720e90700 1 Processor -- start 2023-10-16T12:11:06.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.150+0000 7f6720e90700 1 -- start start 2023-10-16T12:11:06.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.150+0000 7f2f6dffb700 1 -- 172.21.15.16:0/835743600 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f2f680075a0 con 0x7f2f78071250 2023-10-16T12:11:06.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.151+0000 7f2f6dffb700 1 --2- 172.21.15.16:0/835743600 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f2f5c06afd0 0x7f2f5c06d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.151+0000 7f2f6dffb700 1 -- 172.21.15.16:0/835743600 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f2f68088b20 con 0x7f2f78071250 2023-10-16T12:11:06.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.151+0000 7f2f6dffb700 1 --2- 172.21.15.16:0/835743600 >> [v2:172.21.15.16:6816/78500784,v1:172.21.15.16:6818/78500784] conn(0x7f2f5c074590 0x7f2f5c0769d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.151+0000 7f2f6dffb700 1 -- 172.21.15.16:0/835743600 --> [v2:172.21.15.16:6816/78500784,v1:172.21.15.16:6818/78500784] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2f5c0770a0 con 0x7f2f5c074590 2023-10-16T12:11:06.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.152+0000 7f2f7d858700 1 --2- 172.21.15.16:0/835743600 >> [v2:172.21.15.16:6816/78500784,v1:172.21.15.16:6818/78500784] conn(0x7f2f5c074590 0x7f2f5c0769d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.152+0000 7f560effd700 1 -- 172.21.15.16:0/2388481391 <== osd.4 v2:172.21.15.49:6800/682437317 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7f5600077120 con 0x7f5600074610 2023-10-16T12:11:06.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.152+0000 7f2f7c856700 1 --2- 172.21.15.16:0/835743600 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f2f5c06afd0 0x7f2f5c06d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.150+0000 7fb612ffd700 1 --2- 172.21.15.16:0/579057281 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fb60006afa0 0x7fb60006d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.151+0000 7fb612ffd700 1 -- 172.21.15.16:0/579057281 <== mon.1 v2:172.21.15.49:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fb60808ad50 con 0x7fb6140ceff0 2023-10-16T12:11:06.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.151+0000 7fb612ffd700 1 --2- 172.21.15.16:0/579057281 >> [v2:172.21.15.16:6800/2366702185,v1:172.21.15.16:6801/2366702185] conn(0x7fb6000745e0 0x7fb600076a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.151+0000 7fb612ffd700 1 -- 172.21.15.16:0/579057281 --> [v2:172.21.15.16:6800/2366702185,v1:172.21.15.16:6801/2366702185] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb6000770f0 con 0x7fb6000745e0 2023-10-16T12:11:06.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.151+0000 7fb612ffd700 1 -- 172.21.15.16:0/579057281 <== mon.1 v2:172.21.15.49:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb60808d370 con 0x7fb6140ceff0 2023-10-16T12:11:06.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.151+0000 7fb61a291700 1 --2- 172.21.15.16:0/579057281 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fb60006afa0 0x7fb60006d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.151+0000 7fb61aa92700 1 --2- 172.21.15.16:0/579057281 >> [v2:172.21.15.16:6800/2366702185,v1:172.21.15.16:6801/2366702185] conn(0x7fb6000745e0 0x7fb600076a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.152+0000 7fb61a291700 1 --2- 172.21.15.16:0/579057281 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fb60006afa0 0x7fb60006d460 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb61414bdd0 tx=0x7fb60c0090e0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:06.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.152+0000 7fb61aa92700 1 --2- 172.21.15.16:0/579057281 >> [v2:172.21.15.16:6800/2366702185,v1:172.21.15.16:6801/2366702185] conn(0x7fb6000745e0 0x7fb600076a20 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:06.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.153+0000 7f2f7c856700 1 --2- 172.21.15.16:0/835743600 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f2f5c06afd0 0x7f2f5c06d490 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2f780b0530 tx=0x7f2f7400c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:06.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.153+0000 7f5da98e4700 1 Processor -- start 2023-10-16T12:11:06.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.153+0000 7f2f6dffb700 1 -- 172.21.15.16:0/835743600 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2f68001280 con 0x7f2f78071250 2023-10-16T12:11:06.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.151+0000 7f267dffb700 1 -- 172.21.15.16:0/1417599710 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f267800e070 con 0x7f26800b2a50 2023-10-16T12:11:06.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.151+0000 7f267dffb700 1 -- 172.21.15.16:0/1417599710 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f267800b9c0 con 0x7f26800b2a50 2023-10-16T12:11:06.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.151+0000 7f267dffb700 1 -- 172.21.15.16:0/1417599710 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f267800bd70 con 0x7f26800b2a50 2023-10-16T12:11:06.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.153+0000 7f26856bc700 1 -- 172.21.15.16:0/1417599710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26800b2a50 msgr2=0x7f26800b2ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.153+0000 7f26856bc700 1 --2- 172.21.15.16:0/1417599710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26800b2a50 0x7f26800b2ee0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f268006e370 tx=0x7f267800b330 comp rx=0 tx=0).stop 2023-10-16T12:11:06.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.153+0000 7fb612ffd700 1 -- 172.21.15.16:0/579057281 <== osd.0 v2:172.21.15.16:6800/2366702185 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7fb6000770f0 con 0x7fb6000745e0 2023-10-16T12:11:06.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.153+0000 7f5da98e4700 1 -- start start 2023-10-16T12:11:06.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.153+0000 7f2f7d858700 1 --2- 172.21.15.16:0/835743600 >> [v2:172.21.15.16:6816/78500784,v1:172.21.15.16:6818/78500784] conn(0x7f2f5c074590 0x7f2f5c0769d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:06.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.152+0000 7f6720e90700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f671c0cf1e0 0x7f671c16ec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.152+0000 7f6720e90700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f671c06d210 0x7f671c0cbda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.152+0000 7f6720e90700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f671c16e800 con 0x7f671c071350 2023-10-16T12:11:06.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.152+0000 7f6720e90700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f671c06d6a0 con 0x7f671c0cf1e0 2023-10-16T12:11:06.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.152+0000 7f6720e90700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f671c070860 con 0x7f671c06d210 2023-10-16T12:11:06.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.153+0000 7f6719d9b700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f671c071350 0x7f671c0cf5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51102/0 (socket says 172.21.15.16:51102) 2023-10-16T12:11:06.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.153+0000 7f6719d9b700 1 -- 172.21.15.16:0/1091188563 learned_addr learned my addr 172.21.15.16:0/1091188563 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:06.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.153+0000 7f671a59c700 1 --2- 172.21.15.16:0/1091188563 >> v2:172.21.15.49:3300/0 conn(0x7f671c0cf1e0 0x7f671c16ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.153+0000 7f671ad9d700 1 --2- 172.21.15.16:0/1091188563 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f671c06d210 0x7f671c0cbda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.154+0000 7f671ad9d700 1 -- 172.21.15.16:0/1091188563 >> v2:172.21.15.49:3300/0 conn(0x7f671c0cf1e0 msgr2=0x7f671c16ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:06.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.154+0000 7f671959a700 1 -- 172.21.15.16:0/1091188563 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3081131995 0 0) 0x7f671c16e800 con 0x7f671c071350 2023-10-16T12:11:06.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.154+0000 7f671ad9d700 1 --2- 172.21.15.16:0/1091188563 >> v2:172.21.15.49:3300/0 conn(0x7f671c0cf1e0 0x7f671c16ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.154+0000 7f671ad9d700 1 -- 172.21.15.16:0/1091188563 >> v1:172.21.15.16:6789/0 conn(0x7f671c071350 legacy=0x7f671c0cf5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.154+0000 7f671ad9d700 1 -- 172.21.15.16:0/1091188563 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f671c070b40 con 0x7f671c06d210 2023-10-16T12:11:06.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.154+0000 7f5da98e4700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5da40ca180 0x7f5da40b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.154+0000 7f5da98e4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5da40cab20 0x7f5da40a8e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.154+0000 7f5da98e4700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5da41652b0 0x7f5da4163ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.154+0000 7f5da98e4700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5da40b53e0 con 0x7f5da40cab20 2023-10-16T12:11:06.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.154+0000 7f5da98e4700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5da40a84c0 con 0x7f5da40ca180 2023-10-16T12:11:06.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.154+0000 7f5da98e4700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5da40b5090 con 0x7f5da41652b0 2023-10-16T12:11:06.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.154+0000 7f5da2ffd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5da40ca180 0x7f5da40b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.154+0000 7f671ad9d700 1 --2- 172.21.15.16:0/1091188563 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f671c06d210 0x7f671c0cbda0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6714009d90 tx=0x7f671400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8abdb785afc2b6f2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:06.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.154+0000 7f5da2ffd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5da40ca180 0x7f5da40b4af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:60896/0 (socket says 172.21.15.16:60896) 2023-10-16T12:11:06.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.154+0000 7f5da2ffd700 1 -- 172.21.15.16:0/247922506 learned_addr learned my addr 172.21.15.16:0/247922506 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:06.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.154+0000 7f5da37fe700 1 --2- 172.21.15.16:0/247922506 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5da41652b0 0x7f5da4163ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.155+0000 7f5da2ffd700 1 -- 172.21.15.16:0/247922506 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5da41652b0 msgr2=0x7f5da4163ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.155+0000 7f5da2ffd700 1 --2- 172.21.15.16:0/247922506 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5da41652b0 0x7f5da4163ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.155+0000 7f5da2ffd700 1 -- 172.21.15.16:0/247922506 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5da40cab20 msgr2=0x7f5da40a8e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.155+0000 7f5da27fc700 1 --2- 172.21.15.16:0/247922506 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5da40cab20 0x7f5da40a8e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.155+0000 7f5da2ffd700 1 --2- 172.21.15.16:0/247922506 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5da40cab20 0x7f5da40a8e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.155+0000 7f5da2ffd700 1 -- 172.21.15.16:0/247922506 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d9c009a70 con 0x7f5da40ca180 2023-10-16T12:11:06.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.155+0000 7f5da27fc700 1 --2- 172.21.15.16:0/247922506 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5da40cab20 0x7f5da40a8e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:11:06.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.155+0000 7f5da2ffd700 1 --2- 172.21.15.16:0/247922506 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5da40ca180 0x7f5da40b4af0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5d9c0094c0 tx=0x7f5d9c004ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:06.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.155+0000 7f2f6dffb700 1 -- 172.21.15.16:0/835743600 <== osd.1 v2:172.21.15.16:6816/78500784 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7f2f5c0770a0 con 0x7f2f5c074590 2023-10-16T12:11:06.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.154+0000 7f671959a700 1 -- 172.21.15.16:0/1091188563 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f671400bd20 con 0x7f671c06d210 2023-10-16T12:11:06.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.154+0000 7f671959a700 1 -- 172.21.15.16:0/1091188563 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f671400be80 con 0x7f671c06d210 2023-10-16T12:11:06.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.154+0000 7f671959a700 1 -- 172.21.15.16:0/1091188563 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67140116a0 con 0x7f671c06d210 2023-10-16T12:11:06.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.155+0000 7f6720e90700 1 -- 172.21.15.16:0/1091188563 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f671c06d210 msgr2=0x7f671c0cbda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.155+0000 7f6720e90700 1 --2- 172.21.15.16:0/1091188563 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f671c06d210 0x7f671c0cbda0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6714009d90 tx=0x7f671400b330 comp rx=0 tx=0).stop 2023-10-16T12:11:06.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.156+0000 7f6720e90700 1 -- 172.21.15.16:0/1091188563 shutdown_connections 2023-10-16T12:11:06.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.156+0000 7f6720e90700 1 --2- 172.21.15.16:0/1091188563 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f671c06d210 0x7f671c0cbda0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.156+0000 7f6720e90700 1 --2- 172.21.15.16:0/1091188563 >> v2:172.21.15.49:3300/0 conn(0x7f671c0cf1e0 0x7f671c16ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.156+0000 7f6720e90700 1 -- 172.21.15.16:0/1091188563 >> 172.21.15.16:0/1091188563 conn(0x7f671c06a7e0 msgr2=0x7f671c06abf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:06.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.156+0000 7f6720e90700 1 -- 172.21.15.16:0/1091188563 shutdown_connections 2023-10-16T12:11:06.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.156+0000 7f6720e90700 1 -- 172.21.15.16:0/1091188563 wait complete. 2023-10-16T12:11:06.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.158+0000 7f6720e90700 1 Processor -- start 2023-10-16T12:11:06.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.158+0000 7f6720e90700 1 -- start start 2023-10-16T12:11:06.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.159+0000 7f6720e90700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f671c06d210 0x7f671c169220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.160+0000 7f6720e90700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f671c071350 0x7f671c167520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.160+0000 7f6720e90700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f671c0cf1e0 0x7f671c15ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.160+0000 7f6720e90700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f671c168d90 con 0x7f671c0cf1e0 2023-10-16T12:11:06.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.160+0000 7f6720e90700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f671c167ea0 con 0x7f671c071350 2023-10-16T12:11:06.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.160+0000 7f6720e90700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f671c168180 con 0x7f671c06d210 2023-10-16T12:11:06.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.160+0000 7f671ad9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f671c0cf1e0 0x7f671c15ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.160+0000 7f671ad9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f671c0cf1e0 0x7f671c15ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33612/0 (socket says 172.21.15.16:33612) 2023-10-16T12:11:06.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.160+0000 7f671ad9d700 1 -- 172.21.15.16:0/2550176316 learned_addr learned my addr 172.21.15.16:0/2550176316 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:06.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.161+0000 7f671ad9d700 1 -- 172.21.15.16:0/2550176316 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f671c06d210 msgr2=0x7f671c169220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.161+0000 7f671ad9d700 1 --2- 172.21.15.16:0/2550176316 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f671c06d210 0x7f671c169220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.161+0000 7f671ad9d700 1 -- 172.21.15.16:0/2550176316 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f671c071350 msgr2=0x7f671c167520 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-16T12:11:06.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.161+0000 7f671ad9d700 1 --2- 172.21.15.16:0/2550176316 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f671c071350 0x7f671c167520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.161+0000 7f671ad9d700 1 -- 172.21.15.16:0/2550176316 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6714009a70 con 0x7f671c0cf1e0 2023-10-16T12:11:06.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.161+0000 7f671ad9d700 1 --2- 172.21.15.16:0/2550176316 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f671c0cf1e0 0x7f671c15ec10 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f671c0cb9e0 tx=0x7f6714004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:06.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.161+0000 7f670affd700 1 -- 172.21.15.16:0/2550176316 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6714009410 con 0x7f671c0cf1e0 2023-10-16T12:11:06.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.161+0000 7f670affd700 1 -- 172.21.15.16:0/2550176316 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6714011460 con 0x7f671c0cf1e0 2023-10-16T12:11:06.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.161+0000 7f6720e90700 1 -- 172.21.15.16:0/2550176316 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f671c168460 con 0x7f671c0cf1e0 2023-10-16T12:11:06.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.161+0000 7f6720e90700 1 -- 172.21.15.16:0/2550176316 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f671c168a20 con 0x7f671c0cf1e0 2023-10-16T12:11:06.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.161+0000 7f670affd700 1 -- 172.21.15.16:0/2550176316 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67140117f0 con 0x7f671c0cf1e0 2023-10-16T12:11:06.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.160+0000 7f5d8bfff700 1 -- 172.21.15.16:0/247922506 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d9c0047d0 con 0x7f5da40ca180 2023-10-16T12:11:06.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.160+0000 7f5d8bfff700 1 -- 172.21.15.16:0/247922506 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d9c004950 con 0x7f5da40ca180 2023-10-16T12:11:06.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.160+0000 7f5d8bfff700 1 -- 172.21.15.16:0/247922506 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d9c01e670 con 0x7f5da40ca180 2023-10-16T12:11:06.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.160+0000 7f5da98e4700 1 -- 172.21.15.16:0/247922506 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5da40a9390 con 0x7f5da40ca180 2023-10-16T12:11:06.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.160+0000 7f5da98e4700 1 -- 172.21.15.16:0/247922506 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5da40a8710 con 0x7f5da40ca180 2023-10-16T12:11:06.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.163+0000 7f6720e90700 1 -- 172.21.15.16:0/2550176316 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6700000fc0 con 0x7f671c0cf1e0 2023-10-16T12:11:06.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.164+0000 7f5da98e4700 1 -- 172.21.15.16:0/247922506 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5d84000fc0 con 0x7f5da40ca180 2023-10-16T12:11:06.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.164+0000 7f26856bc700 1 -- 172.21.15.16:0/1417599710 shutdown_connections 2023-10-16T12:11:06.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.164+0000 7f26856bc700 1 --2- 172.21.15.16:0/1417599710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26800b2a50 0x7f26800b2ee0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.164+0000 7f26856bc700 1 --2- 172.21.15.16:0/1417599710 >> v2:172.21.15.49:3300/0 conn(0x7f268015c330 0x7f2680163480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.164+0000 7f26856bc700 1 --2- 172.21.15.16:0/1417599710 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2680071470 0x7f2680162d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.164+0000 7f26856bc700 1 -- 172.21.15.16:0/1417599710 >> 172.21.15.16:0/1417599710 conn(0x7f268006a7e0 msgr2=0x7f268006abf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:06.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.164+0000 7f26856bc700 1 -- 172.21.15.16:0/1417599710 shutdown_connections 2023-10-16T12:11:06.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.164+0000 7f26856bc700 1 -- 172.21.15.16:0/1417599710 wait complete. 2023-10-16T12:11:06.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.164+0000 7f670affd700 1 -- 172.21.15.16:0/2550176316 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f6714011950 con 0x7f671c0cf1e0 2023-10-16T12:11:06.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.165+0000 7f670affd700 1 --2- 172.21.15.16:0/2550176316 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f66fc06aed0 0x7f66fc06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.165+0000 7f670affd700 1 -- 172.21.15.16:0/2550176316 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f671408b930 con 0x7f671c0cf1e0 2023-10-16T12:11:06.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.165+0000 7f670affd700 1 --2- 172.21.15.16:0/2550176316 >> [v2:172.21.15.49:6824/2356409928,v1:172.21.15.49:6825/2356409928] conn(0x7f66fc074490 0x7f66fc0768d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.165+0000 7f670affd700 1 -- 172.21.15.16:0/2550176316 --> [v2:172.21.15.49:6824/2356409928,v1:172.21.15.49:6825/2356409928] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f66fc076fa0 con 0x7f66fc074490 2023-10-16T12:11:06.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.165+0000 7f670affd700 1 -- 172.21.15.16:0/2550176316 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f67140902a0 con 0x7f671c0cf1e0 2023-10-16T12:11:06.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.166+0000 7f671a59c700 1 --2- 172.21.15.16:0/2550176316 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f66fc06aed0 0x7f66fc06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.166+0000 7f671a59c700 1 --2- 172.21.15.16:0/2550176316 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f66fc06aed0 0x7f66fc06d390 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f670c009350 tx=0x7f670c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:06.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.166+0000 7f26856bc700 1 Processor -- start 2023-10-16T12:11:06.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.167+0000 7f26856bc700 1 -- start start 2023-10-16T12:11:06.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.167+0000 7f5d8bfff700 1 -- 172.21.15.16:0/247922506 <== mon.1 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f5d9c01e7d0 con 0x7f5da40ca180 2023-10-16T12:11:06.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.167+0000 7f6719d9b700 1 --2- 172.21.15.16:0/2550176316 >> [v2:172.21.15.49:6824/2356409928,v1:172.21.15.49:6825/2356409928] conn(0x7f66fc074490 0x7f66fc0768d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.167+0000 7f6719d9b700 1 --2- 172.21.15.16:0/2550176316 >> [v2:172.21.15.49:6824/2356409928,v1:172.21.15.49:6825/2356409928] conn(0x7f66fc074490 0x7f66fc0768d0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:06.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.168+0000 7f26856bc700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2680071470 0x7f26800cbcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.168+0000 7f26856bc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26800b2a50 0x7f268013a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.168+0000 7f26856bc700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f268015c330 0x7f268014c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.168+0000 7f26856bc700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26800cfce0 con 0x7f26800b2a50 2023-10-16T12:11:06.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.168+0000 7f26856bc700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f268014b7c0 con 0x7f268015c330 2023-10-16T12:11:06.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.168+0000 7f26856bc700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f268014baa0 con 0x7f2680071470 2023-10-16T12:11:06.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.168+0000 7f267f7fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f268015c330 0x7f268014c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.168+0000 7f267f7fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f268015c330 0x7f268014c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:60914/0 (socket says 172.21.15.16:60914) 2023-10-16T12:11:06.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.168+0000 7f267f7fe700 1 -- 172.21.15.16:0/2232655345 learned_addr learned my addr 172.21.15.16:0/2232655345 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:06.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.168+0000 7f267e7fc700 1 --2- 172.21.15.16:0/2232655345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26800b2a50 0x7f268013a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.168+0000 7f267effd700 1 --2- 172.21.15.16:0/2232655345 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2680071470 0x7f26800cbcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.169+0000 7f267f7fe700 1 -- 172.21.15.16:0/2232655345 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2680071470 msgr2=0x7f26800cbcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.169+0000 7f267f7fe700 1 --2- 172.21.15.16:0/2232655345 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2680071470 0x7f26800cbcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.169+0000 7f267f7fe700 1 -- 172.21.15.16:0/2232655345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26800b2a50 msgr2=0x7f268013a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.169+0000 7f267f7fe700 1 --2- 172.21.15.16:0/2232655345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26800b2a50 0x7f268013a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.169+0000 7f267f7fe700 1 -- 172.21.15.16:0/2232655345 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2678009b30 con 0x7f268015c330 2023-10-16T12:11:06.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.169+0000 7f267f7fe700 1 --2- 172.21.15.16:0/2232655345 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f268015c330 0x7f268014c860 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2680067580 tx=0x7f2678004ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:06.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.169+0000 7f670affd700 1 -- 172.21.15.16:0/2550176316 <== osd.7 v2:172.21.15.49:6824/2356409928 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7f66fc076fa0 con 0x7f66fc074490 2023-10-16T12:11:06.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.171+0000 7f26677fe700 1 -- 172.21.15.16:0/2232655345 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2678004450 con 0x7f268015c330 2023-10-16T12:11:06.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.171+0000 7f26677fe700 1 -- 172.21.15.16:0/2232655345 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26780045b0 con 0x7f268015c330 2023-10-16T12:11:06.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.171+0000 7f26677fe700 1 -- 172.21.15.16:0/2232655345 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2678004940 con 0x7f268015c330 2023-10-16T12:11:06.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.171+0000 7f26856bc700 1 -- 172.21.15.16:0/2232655345 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f268014bd80 con 0x7f268015c330 2023-10-16T12:11:06.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.171+0000 7f26856bc700 1 -- 172.21.15.16:0/2232655345 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f268014c340 con 0x7f268015c330 2023-10-16T12:11:06.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.167+0000 7f5d8bfff700 1 --2- 172.21.15.16:0/247922506 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f5d9006af50 0x7f5d9006d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.168+0000 7f5d8bfff700 1 -- 172.21.15.16:0/247922506 <== mon.1 v2:172.21.15.49:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f5d9c08beb0 con 0x7f5da40ca180 2023-10-16T12:11:06.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.168+0000 7f5da27fc700 1 --2- 172.21.15.16:0/247922506 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f5d9006af50 0x7f5d9006d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.168+0000 7f5d8bfff700 1 --2- 172.21.15.16:0/247922506 >> [v2:172.21.15.49:6816/3830572508,v1:172.21.15.49:6817/3830572508] conn(0x7f5d90074510 0x7f5d90076950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.168+0000 7f5d8bfff700 1 -- 172.21.15.16:0/247922506 --> [v2:172.21.15.49:6816/3830572508,v1:172.21.15.49:6817/3830572508] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5d90077020 con 0x7f5d90074510 2023-10-16T12:11:06.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.169+0000 7f5da27fc700 1 --2- 172.21.15.16:0/247922506 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f5d9006af50 0x7f5d9006d410 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f5da40c2070 tx=0x7f5d94009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:06.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.173+0000 7f5da37fe700 1 --2- 172.21.15.16:0/247922506 >> [v2:172.21.15.49:6816/3830572508,v1:172.21.15.49:6817/3830572508] conn(0x7f5d90074510 0x7f5d90076950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.173+0000 7f26856bc700 1 -- 172.21.15.16:0/2232655345 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2660000fc0 con 0x7f268015c330 2023-10-16T12:11:06.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.173+0000 7f5da37fe700 1 --2- 172.21.15.16:0/247922506 >> [v2:172.21.15.49:6816/3830572508,v1:172.21.15.49:6817/3830572508] conn(0x7f5d90074510 0x7f5d90076950 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:06.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.174+0000 7f26677fe700 1 -- 172.21.15.16:0/2232655345 <== mon.1 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f2678025450 con 0x7f268015c330 2023-10-16T12:11:06.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.174+0000 7f26677fe700 1 --2- 172.21.15.16:0/2232655345 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f266c06afd0 0x7f266c06d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.174+0000 7f26677fe700 1 -- 172.21.15.16:0/2232655345 <== mon.1 v2:172.21.15.49:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f267808b140 con 0x7f268015c330 2023-10-16T12:11:06.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.175+0000 7f26677fe700 1 --2- 172.21.15.16:0/2232655345 >> [v2:172.21.15.16:6808/47933305,v1:172.21.15.16:6809/47933305] conn(0x7f266c074590 0x7f266c0769d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.175+0000 7f26677fe700 1 -- 172.21.15.16:0/2232655345 --> [v2:172.21.15.16:6808/47933305,v1:172.21.15.16:6809/47933305] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f266c0770a0 con 0x7f266c074590 2023-10-16T12:11:06.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.175+0000 7f267effd700 1 --2- 172.21.15.16:0/2232655345 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f266c06afd0 0x7f266c06d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.175+0000 7f26677fe700 1 -- 172.21.15.16:0/2232655345 <== mon.1 v2:172.21.15.49:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2678055f20 con 0x7f268015c330 2023-10-16T12:11:06.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.175+0000 7f267effd700 1 --2- 172.21.15.16:0/2232655345 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f266c06afd0 0x7f266c06d490 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f2670009870 tx=0x7f2670008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:06.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.176+0000 7f267e7fc700 1 --2- 172.21.15.16:0/2232655345 >> [v2:172.21.15.16:6808/47933305,v1:172.21.15.16:6809/47933305] conn(0x7f266c074590 0x7f266c0769d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.176+0000 7f267e7fc700 1 --2- 172.21.15.16:0/2232655345 >> [v2:172.21.15.16:6808/47933305,v1:172.21.15.16:6809/47933305] conn(0x7f266c074590 0x7f266c0769d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:06.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.177+0000 7f5d8bfff700 1 -- 172.21.15.16:0/247922506 <== mon.1 v2:172.21.15.49:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5d9c056ce0 con 0x7f5da40ca180 2023-10-16T12:11:06.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.177+0000 7f5d8bfff700 1 -- 172.21.15.16:0/247922506 <== osd.5 v2:172.21.15.49:6816/3830572508 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7f5d90077020 con 0x7f5d90074510 2023-10-16T12:11:06.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.177+0000 7f26677fe700 1 -- 172.21.15.16:0/2232655345 <== osd.2 v2:172.21.15.16:6808/47933305 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7f266c0770a0 con 0x7f266c074590 2023-10-16T12:11:06.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.179+0000 7f1a424f7700 1 -- 172.21.15.16:0/2602751565 --> [v2:172.21.15.16:6817/375531758,v1:172.21.15.16:6819/375531758] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1a1c002d40 con 0x7f1a280744c0 2023-10-16T12:11:06.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.187+0000 7fb610ff9700 1 -- 172.21.15.16:0/579057281 --> [v2:172.21.15.16:6800/2366702185,v1:172.21.15.16:6801/2366702185] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb5f8002d40 con 0x7fb6000745e0 2023-10-16T12:11:06.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.188+0000 7f56249bf700 1 -- 172.21.15.16:0/2388481391 --> [v2:172.21.15.49:6800/682437317,v1:172.21.15.49:6801/682437317] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f562015b630 con 0x7f5600074610 2023-10-16T12:11:06.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.188+0000 7fb612ffd700 1 -- 172.21.15.16:0/579057281 <== osd.0 v2:172.21.15.16:6800/2366702185 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fb5f8002d40 con 0x7fb6000745e0 2023-10-16T12:11:06.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.188+0000 7f560effd700 1 -- 172.21.15.16:0/2388481391 <== osd.4 v2:172.21.15.49:6800/682437317 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f5610006d40 con 0x7f5600074610 2023-10-16T12:11:06.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.188+0000 7fb610ff9700 1 -- 172.21.15.16:0/579057281 >> [v2:172.21.15.16:6800/2366702185,v1:172.21.15.16:6801/2366702185] conn(0x7fb6000745e0 msgr2=0x7fb600076a20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.188+0000 7fb610ff9700 1 --2- 172.21.15.16:0/579057281 >> [v2:172.21.15.16:6800/2366702185,v1:172.21.15.16:6801/2366702185] conn(0x7fb6000745e0 0x7fb600076a20 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.189+0000 7fb610ff9700 1 -- 172.21.15.16:0/579057281 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fb60006afa0 msgr2=0x7fb60006d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.189+0000 7fb610ff9700 1 --2- 172.21.15.16:0/579057281 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fb60006afa0 0x7fb60006d460 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb61414bdd0 tx=0x7fb60c0090e0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.189+0000 7fb610ff9700 1 -- 172.21.15.16:0/579057281 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb6140ceff0 msgr2=0x7fb6140ccc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.189+0000 7fb610ff9700 1 --2- 172.21.15.16:0/579057281 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb6140ceff0 0x7fb6140ccc80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb608015000 tx=0x7fb608004750 comp rx=0 tx=0).stop 2023-10-16T12:11:06.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.189+0000 7fb61aa92700 1 -- 172.21.15.16:0/579057281 reap_dead start 2023-10-16T12:11:06.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.189+0000 7fb610ff9700 1 -- 172.21.15.16:0/579057281 shutdown_connections 2023-10-16T12:11:06.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.189+0000 7fb610ff9700 1 -- 172.21.15.16:0/579057281 >> 172.21.15.16:0/579057281 conn(0x7fb6140b4cd0 msgr2=0x7fb6140aef40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:06.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.189+0000 7fb610ff9700 1 -- 172.21.15.16:0/579057281 shutdown_connections 2023-10-16T12:11:06.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.189+0000 7fb610ff9700 1 -- 172.21.15.16:0/579057281 wait complete. 2023-10-16T12:11:06.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.190+0000 7f1a38ff9700 1 -- 172.21.15.16:0/2602751565 <== osd.3 v2:172.21.15.16:6817/375531758 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f1a1c002d40 con 0x7f1a280744c0 2023-10-16T12:11:06.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.191+0000 7f2f7f2bb700 1 -- 172.21.15.16:0/835743600 --> [v2:172.21.15.16:6816/78500784,v1:172.21.15.16:6818/78500784] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2f58002d40 con 0x7f2f5c074590 2023-10-16T12:11:06.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.192+0000 7f2f6dffb700 1 -- 172.21.15.16:0/835743600 <== osd.1 v2:172.21.15.16:6816/78500784 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f2f58002d40 con 0x7f2f5c074590 2023-10-16T12:11:06.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.192+0000 7f2f7f2bb700 1 -- 172.21.15.16:0/835743600 >> [v2:172.21.15.16:6816/78500784,v1:172.21.15.16:6818/78500784] conn(0x7f2f5c074590 msgr2=0x7f2f5c0769d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.192+0000 7f2f7f2bb700 1 --2- 172.21.15.16:0/835743600 >> [v2:172.21.15.16:6816/78500784,v1:172.21.15.16:6818/78500784] conn(0x7f2f5c074590 0x7f2f5c0769d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.193+0000 7f56249bf700 1 -- 172.21.15.16:0/2388481391 >> [v2:172.21.15.49:6800/682437317,v1:172.21.15.49:6801/682437317] conn(0x7f5600074610 msgr2=0x7f5600076a50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.193+0000 7f56249bf700 1 --2- 172.21.15.16:0/2388481391 >> [v2:172.21.15.49:6800/682437317,v1:172.21.15.49:6801/682437317] conn(0x7f5600074610 0x7f5600076a50 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.194+0000 7f56249bf700 1 -- 172.21.15.16:0/2388481391 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f560006afd0 msgr2=0x7f560006d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.194+0000 7f56249bf700 1 --2- 172.21.15.16:0/2388481391 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f560006afd0 0x7f560006d490 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f562015a060 tx=0x7f56140090e0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.194+0000 7f56249bf700 1 -- 172.21.15.16:0/2388481391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56200b52f0 msgr2=0x7f56200b3b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.194+0000 7f56249bf700 1 --2- 172.21.15.16:0/2388481391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56200b52f0 0x7f56200b3b60 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f561800ba80 tx=0x7f561800bd90 comp rx=0 tx=0).stop 2023-10-16T12:11:06.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.194+0000 7f561ed9d700 1 -- 172.21.15.16:0/2388481391 reap_dead start 2023-10-16T12:11:06.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.194+0000 7f56249bf700 1 -- 172.21.15.16:0/2388481391 shutdown_connections 2023-10-16T12:11:06.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.194+0000 7f56249bf700 1 -- 172.21.15.16:0/2388481391 >> 172.21.15.16:0/2388481391 conn(0x7f56200b1fd0 msgr2=0x7f56200b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:06.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.194+0000 7f2f7f2bb700 1 -- 172.21.15.16:0/835743600 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f2f5c06afd0 msgr2=0x7f2f5c06d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.194+0000 7f2f7f2bb700 1 --2- 172.21.15.16:0/835743600 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f2f5c06afd0 0x7f2f5c06d490 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2f780b0530 tx=0x7f2f7400c040 comp rx=0 tx=0).stop 2023-10-16T12:11:06.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.194+0000 7f2f7f2bb700 1 -- 172.21.15.16:0/835743600 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f78071250 msgr2=0x7f2f78152f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.194+0000 7f2f7f2bb700 1 --2- 172.21.15.16:0/835743600 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f78071250 0x7f2f78152f60 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f2f6800c520 tx=0x7f2f6800c830 comp rx=0 tx=0).stop 2023-10-16T12:11:06.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.194+0000 7f2f7d858700 1 -- 172.21.15.16:0/835743600 reap_dead start 2023-10-16T12:11:06.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.195+0000 7f2f7f2bb700 1 -- 172.21.15.16:0/835743600 shutdown_connections 2023-10-16T12:11:06.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.195+0000 7f2f7f2bb700 1 -- 172.21.15.16:0/835743600 >> 172.21.15.16:0/835743600 conn(0x7f2f7806a7e0 msgr2=0x7f2f781e4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:06.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.195+0000 7f1a424f7700 1 -- 172.21.15.16:0/2602751565 >> [v2:172.21.15.16:6817/375531758,v1:172.21.15.16:6819/375531758] conn(0x7f1a280744c0 msgr2=0x7f1a28076900 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.195+0000 7f1a424f7700 1 --2- 172.21.15.16:0/2602751565 >> [v2:172.21.15.16:6817/375531758,v1:172.21.15.16:6819/375531758] conn(0x7f1a280744c0 0x7f1a28076900 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.195+0000 7f1a424f7700 1 -- 172.21.15.16:0/2602751565 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f1a2806af00 msgr2=0x7f1a2806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.195+0000 7f1a424f7700 1 --2- 172.21.15.16:0/2602751565 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f1a2806af00 0x7f1a2806d3c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1a3c1590a0 tx=0x7f1a3000d040 comp rx=0 tx=0).stop 2023-10-16T12:11:06.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.195+0000 7f1a424f7700 1 -- 172.21.15.16:0/2602751565 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a3c06c800 msgr2=0x7f1a3c156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.195+0000 7f1a424f7700 1 --2- 172.21.15.16:0/2602751565 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a3c06c800 0x7f1a3c156430 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f1a34009d10 tx=0x7f1a3400c3a0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.195+0000 7f1a40a94700 1 -- 172.21.15.16:0/2602751565 reap_dead start 2023-10-16T12:11:06.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.195+0000 7f2f7f2bb700 1 -- 172.21.15.16:0/835743600 shutdown_connections 2023-10-16T12:11:06.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.195+0000 7f2f7f2bb700 1 -- 172.21.15.16:0/835743600 wait complete. 2023-10-16T12:11:06.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.195+0000 7f1a424f7700 1 -- 172.21.15.16:0/2602751565 shutdown_connections 2023-10-16T12:11:06.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.195+0000 7f1a424f7700 1 -- 172.21.15.16:0/2602751565 >> 172.21.15.16:0/2602751565 conn(0x7f1a3c06a7e0 msgr2=0x7f1a3c06cbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:06.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.195+0000 7f1a424f7700 1 -- 172.21.15.16:0/2602751565 shutdown_connections 2023-10-16T12:11:06.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.195+0000 7f1a424f7700 1 -- 172.21.15.16:0/2602751565 wait complete. 2023-10-16T12:11:06.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.200+0000 7f26856bc700 1 -- 172.21.15.16:0/2232655345 --> [v2:172.21.15.16:6808/47933305,v1:172.21.15.16:6809/47933305] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2660002d40 con 0x7f266c074590 2023-10-16T12:11:06.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.200+0000 7f26677fe700 1 -- 172.21.15.16:0/2232655345 <== osd.2 v2:172.21.15.16:6808/47933305 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f2660002d40 con 0x7f266c074590 2023-10-16T12:11:06.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.201+0000 7f26856bc700 1 -- 172.21.15.16:0/2232655345 >> [v2:172.21.15.16:6808/47933305,v1:172.21.15.16:6809/47933305] conn(0x7f266c074590 msgr2=0x7f266c0769d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.201+0000 7f26856bc700 1 --2- 172.21.15.16:0/2232655345 >> [v2:172.21.15.16:6808/47933305,v1:172.21.15.16:6809/47933305] conn(0x7f266c074590 0x7f266c0769d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.201+0000 7f56249bf700 1 -- 172.21.15.16:0/2388481391 shutdown_connections 2023-10-16T12:11:06.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.201+0000 7f56249bf700 1 -- 172.21.15.16:0/2388481391 wait complete. 2023-10-16T12:11:06.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.202+0000 7f6720e90700 1 -- 172.21.15.16:0/2550176316 --> [v2:172.21.15.49:6824/2356409928,v1:172.21.15.49:6825/2356409928] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6700002d40 con 0x7f66fc074490 2023-10-16T12:11:06.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.203+0000 7f26856bc700 1 -- 172.21.15.16:0/2232655345 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f266c06afd0 msgr2=0x7f266c06d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.203+0000 7f26856bc700 1 --2- 172.21.15.16:0/2232655345 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f266c06afd0 0x7f266c06d490 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f2670009870 tx=0x7f2670008040 comp rx=0 tx=0).stop 2023-10-16T12:11:06.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.203+0000 7f26856bc700 1 -- 172.21.15.16:0/2232655345 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f268015c330 msgr2=0x7f268014c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.203+0000 7f26856bc700 1 --2- 172.21.15.16:0/2232655345 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f268015c330 0x7f268014c860 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2680067580 tx=0x7f2678004ad0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.203+0000 7f267f7fe700 1 -- 172.21.15.16:0/2232655345 reap_dead start 2023-10-16T12:11:06.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.203+0000 7f670affd700 1 -- 172.21.15.16:0/2550176316 <== osd.7 v2:172.21.15.49:6824/2356409928 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f6700002d40 con 0x7f66fc074490 2023-10-16T12:11:06.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.203+0000 7f26856bc700 1 -- 172.21.15.16:0/2232655345 shutdown_connections 2023-10-16T12:11:06.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.203+0000 7f26856bc700 1 -- 172.21.15.16:0/2232655345 >> 172.21.15.16:0/2232655345 conn(0x7f268006a7e0 msgr2=0x7f2680161590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:06.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.203+0000 7f26856bc700 1 -- 172.21.15.16:0/2232655345 shutdown_connections 2023-10-16T12:11:06.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.203+0000 7f26856bc700 1 -- 172.21.15.16:0/2232655345 wait complete. 2023-10-16T12:11:06.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.203+0000 7f6720e90700 1 -- 172.21.15.16:0/2550176316 >> [v2:172.21.15.49:6824/2356409928,v1:172.21.15.49:6825/2356409928] conn(0x7f66fc074490 msgr2=0x7f66fc0768d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.203+0000 7f6720e90700 1 --2- 172.21.15.16:0/2550176316 >> [v2:172.21.15.49:6824/2356409928,v1:172.21.15.49:6825/2356409928] conn(0x7f66fc074490 0x7f66fc0768d0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.203+0000 7f6720e90700 1 -- 172.21.15.16:0/2550176316 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f66fc06aed0 msgr2=0x7f66fc06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.203+0000 7f6720e90700 1 --2- 172.21.15.16:0/2550176316 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f66fc06aed0 0x7f66fc06d390 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f670c009350 tx=0x7f670c008040 comp rx=0 tx=0).stop 2023-10-16T12:11:06.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.203+0000 7f6720e90700 1 -- 172.21.15.16:0/2550176316 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f671c0cf1e0 msgr2=0x7f671c15ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.203+0000 7f6720e90700 1 --2- 172.21.15.16:0/2550176316 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f671c0cf1e0 0x7f671c15ec10 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f671c0cb9e0 tx=0x7f6714004b40 comp rx=0 tx=0).stop 2023-10-16T12:11:06.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.203+0000 7f671ad9d700 1 -- 172.21.15.16:0/2550176316 reap_dead start 2023-10-16T12:11:06.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.204+0000 7f6720e90700 1 -- 172.21.15.16:0/2550176316 shutdown_connections 2023-10-16T12:11:06.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.204+0000 7f6720e90700 1 -- 172.21.15.16:0/2550176316 >> 172.21.15.16:0/2550176316 conn(0x7f671c06a7e0 msgr2=0x7f671c162e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:06.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.204+0000 7f6720e90700 1 -- 172.21.15.16:0/2550176316 shutdown_connections 2023-10-16T12:11:06.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.204+0000 7f6720e90700 1 -- 172.21.15.16:0/2550176316 wait complete. 2023-10-16T12:11:06.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.208+0000 7f11fbc84700 1 Processor -- start 2023-10-16T12:11:06.274 INFO:teuthology.orchestra.run.smithi016.stdout:55834574858 2023-10-16T12:11:06.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.208+0000 7f11fbc84700 1 -- start start 2023-10-16T12:11:06.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.209+0000 7f11fbc84700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f11f4071250 0x7f11f40716b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.209+0000 7f11fbc84700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f11f415a030 0x7f11f415a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.209+0000 7f11fbc84700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11f4071bf0 con 0x7f11f41599f0 2023-10-16T12:11:06.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.209+0000 7f11fbc84700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11f415d240 con 0x7f11f415a030 2023-10-16T12:11:06.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.209+0000 7f11fbc84700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11f40b2fd0 con 0x7f11f4071250 2023-10-16T12:11:06.276 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-16T12:11:06.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.210+0000 7f11f9a20700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7f11f41599f0 0x7f11f415ddc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:58320/0 (socket says 172.21.15.16:58320) 2023-10-16T12:11:06.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.210+0000 7f11f9a20700 1 -- 172.21.15.16:0/3579986041 learned_addr learned my addr 172.21.15.16:0/3579986041 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:06.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.210+0000 7f11fa221700 1 --2- 172.21.15.16:0/3579986041 >> v2:172.21.15.49:3300/0 conn(0x7f11f415a030 0x7f11f415a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.210+0000 7f11f921f700 1 --2- 172.21.15.16:0/3579986041 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f11f4071250 0x7f11f40716b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.210+0000 7f11fa221700 1 -- 172.21.15.16:0/3579986041 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f11f4071250 msgr2=0x7f11f40716b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:06.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.210+0000 7f11fa221700 1 --2- 172.21.15.16:0/3579986041 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f11f4071250 0x7f11f40716b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.210+0000 7f11fa221700 1 -- 172.21.15.16:0/3579986041 >> v1:172.21.15.49:6789/0 conn(0x7f11f41599f0 legacy=0x7f11f415ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.210+0000 7f11fa221700 1 -- 172.21.15.16:0/3579986041 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11f40b32b0 con 0x7f11f415a030 2023-10-16T12:11:06.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.210+0000 7f11fa221700 1 --2- 172.21.15.16:0/3579986041 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f11f415a030 0x7f11f415a4c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f11ec009d90 tx=0x7f11ec00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2cb1911e4896fdd0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:06.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.211+0000 7f11f8a1e700 1 -- 172.21.15.16:0/3579986041 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11ec00e070 con 0x7f11f415a030 2023-10-16T12:11:06.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.211+0000 7f11f8a1e700 1 -- 172.21.15.16:0/3579986041 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11ec003a20 con 0x7f11f415a030 2023-10-16T12:11:06.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.211+0000 7f11f8a1e700 1 -- 172.21.15.16:0/3579986041 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11ec003dd0 con 0x7f11f415a030 2023-10-16T12:11:06.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.211+0000 7f11fbc84700 1 -- 172.21.15.16:0/3579986041 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f11f415a030 msgr2=0x7f11f415a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.211+0000 7f11fbc84700 1 --2- 172.21.15.16:0/3579986041 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f11f415a030 0x7f11f415a4c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f11ec009d90 tx=0x7f11ec00b330 comp rx=0 tx=0).stop 2023-10-16T12:11:06.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.212+0000 7f11fbc84700 1 -- 172.21.15.16:0/3579986041 shutdown_connections 2023-10-16T12:11:06.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.212+0000 7f11fbc84700 1 --2- 172.21.15.16:0/3579986041 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f11f415a030 0x7f11f415a4c0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.212+0000 7f11fbc84700 1 --2- 172.21.15.16:0/3579986041 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f11f4071250 0x7f11f40716b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.212+0000 7f11fbc84700 1 -- 172.21.15.16:0/3579986041 >> 172.21.15.16:0/3579986041 conn(0x7f11f406a7e0 msgr2=0x7f11f406abf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:06.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.212+0000 7f11fbc84700 1 -- 172.21.15.16:0/3579986041 shutdown_connections 2023-10-16T12:11:06.282 INFO:teuthology.orchestra.run.smithi016.stdout:55834574857 2023-10-16T12:11:06.283 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-10-16T12:11:06.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.213+0000 7f11fbc84700 1 -- 172.21.15.16:0/3579986041 wait complete. 2023-10-16T12:11:06.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.214+0000 7f11fbc84700 1 Processor -- start 2023-10-16T12:11:06.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.214+0000 7f11fbc84700 1 -- start start 2023-10-16T12:11:06.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.215+0000 7f11fbc84700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f11f4071250 0x7f11f414e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.215+0000 7f11fbc84700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f11f41599f0 0x7f11f40ade30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.215+0000 7f11fbc84700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11f415a030 0x7f11f40b1080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.215+0000 7f11fbc84700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11f414f950 con 0x7f11f415a030 2023-10-16T12:11:06.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.215+0000 7f11fbc84700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11f414fc30 con 0x7f11f4071250 2023-10-16T12:11:06.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.215+0000 7f11fbc84700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11f414ff10 con 0x7f11f41599f0 2023-10-16T12:11:06.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.215+0000 7f11fa221700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11f415a030 0x7f11f40b1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.215+0000 7f11f921f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f11f41599f0 0x7f11f40ade30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.215+0000 7f11f9a20700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f11f4071250 0x7f11f414e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.215+0000 7f11f921f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f11f41599f0 0x7f11f40ade30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:50130/0 (socket says 172.21.15.16:50130) 2023-10-16T12:11:06.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.215+0000 7f11fa221700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11f415a030 0x7f11f40b1080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33642/0 (socket says 172.21.15.16:33642) 2023-10-16T12:11:06.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.215+0000 7f11f9a20700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f11f4071250 0x7f11f414e1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:60932/0 (socket says 172.21.15.16:60932) 2023-10-16T12:11:06.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.215+0000 7f11f921f700 1 -- 172.21.15.16:0/4168491569 learned_addr learned my addr 172.21.15.16:0/4168491569 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:06.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.215+0000 7f11f921f700 1 -- 172.21.15.16:0/4168491569 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f11f4071250 msgr2=0x7f11f414e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.215+0000 7f11f921f700 1 --2- 172.21.15.16:0/4168491569 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f11f4071250 0x7f11f414e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.215+0000 7f11f921f700 1 -- 172.21.15.16:0/4168491569 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11f415a030 msgr2=0x7f11f40b1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.215+0000 7f11f921f700 1 --2- 172.21.15.16:0/4168491569 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11f415a030 0x7f11f40b1080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.215+0000 7f11f921f700 1 -- 172.21.15.16:0/4168491569 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11ec009a70 con 0x7f11f41599f0 2023-10-16T12:11:06.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.216+0000 7f11f9a20700 1 --2- 172.21.15.16:0/4168491569 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f11f4071250 0x7f11f414e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:11:06.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.216+0000 7f11fa221700 1 --2- 172.21.15.16:0/4168491569 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11f415a030 0x7f11f40b1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:11:06.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.216+0000 7f11f921f700 1 --2- 172.21.15.16:0/4168491569 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f11f41599f0 0x7f11f40ade30 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f11f000b790 tx=0x7f11f000bb50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:06.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.216+0000 7f11ea7fc700 1 -- 172.21.15.16:0/4168491569 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f0010040 con 0x7f11f41599f0 2023-10-16T12:11:06.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.216+0000 7f11ea7fc700 1 -- 172.21.15.16:0/4168491569 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11f000dbc0 con 0x7f11f41599f0 2023-10-16T12:11:06.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.216+0000 7f11ea7fc700 1 -- 172.21.15.16:0/4168491569 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f0009760 con 0x7f11f41599f0 2023-10-16T12:11:06.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.217+0000 7f11fbc84700 1 -- 172.21.15.16:0/4168491569 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11f41501f0 con 0x7f11f41599f0 2023-10-16T12:11:06.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.217+0000 7f11fbc84700 1 -- 172.21.15.16:0/4168491569 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11f414e6e0 con 0x7f11f41599f0 2023-10-16T12:11:06.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.217+0000 7f11fbc84700 1 -- 172.21.15.16:0/4168491569 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f11f4153420 con 0x7f11f41599f0 2023-10-16T12:11:06.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.219+0000 7f11ea7fc700 1 -- 172.21.15.16:0/4168491569 <== mon.2 v2:172.21.15.49:3301/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f11f0005a20 con 0x7f11f41599f0 2023-10-16T12:11:06.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.219+0000 7f11ea7fc700 1 --2- 172.21.15.16:0/4168491569 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f11d806af00 0x7f11d806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.219+0000 7f11ea7fc700 1 -- 172.21.15.16:0/4168491569 <== mon.2 v2:172.21.15.49:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f11f0089d60 con 0x7f11f41599f0 2023-10-16T12:11:06.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.219+0000 7f11f9a20700 1 --2- 172.21.15.16:0/4168491569 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f11d806af00 0x7f11d806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.219+0000 7f11ea7fc700 1 --2- 172.21.15.16:0/4168491569 >> [v2:172.21.15.49:6808/3582923927,v1:172.21.15.49:6809/3582923927] conn(0x7f11d80744c0 0x7f11d8076900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.219+0000 7f11ea7fc700 1 -- 172.21.15.16:0/4168491569 --> [v2:172.21.15.49:6808/3582923927,v1:172.21.15.49:6809/3582923927] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f11d8076fd0 con 0x7f11d80744c0 2023-10-16T12:11:06.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.219+0000 7f11ea7fc700 1 -- 172.21.15.16:0/4168491569 <== mon.2 v2:172.21.15.49:3301/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f11f008a1b0 con 0x7f11f41599f0 2023-10-16T12:11:06.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.219+0000 7f11fa221700 1 --2- 172.21.15.16:0/4168491569 >> [v2:172.21.15.49:6808/3582923927,v1:172.21.15.49:6809/3582923927] conn(0x7f11d80744c0 0x7f11d8076900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.220+0000 7f11f9a20700 1 --2- 172.21.15.16:0/4168491569 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f11d806af00 0x7f11d806d3c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f11e4007050 tx=0x7f11e4007e10 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:06.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.220+0000 7f11fa221700 1 --2- 172.21.15.16:0/4168491569 >> [v2:172.21.15.49:6808/3582923927,v1:172.21.15.49:6809/3582923927] conn(0x7f11d80744c0 0x7f11d8076900 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:06.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.221+0000 7f11ea7fc700 1 -- 172.21.15.16:0/4168491569 <== osd.6 v2:172.21.15.49:6808/3582923927 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7f11d8076fd0 con 0x7f11d80744c0 2023-10-16T12:11:06.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.225+0000 7f5da98e4700 1 -- 172.21.15.16:0/247922506 --> [v2:172.21.15.49:6816/3830572508,v1:172.21.15.49:6817/3830572508] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5d84002d40 con 0x7f5d90074510 2023-10-16T12:11:06.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.225+0000 7f5d8bfff700 1 -- 172.21.15.16:0/247922506 <== osd.5 v2:172.21.15.49:6816/3830572508 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f5d84002d40 con 0x7f5d90074510 2023-10-16T12:11:06.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.226+0000 7f5da98e4700 1 -- 172.21.15.16:0/247922506 >> [v2:172.21.15.49:6816/3830572508,v1:172.21.15.49:6817/3830572508] conn(0x7f5d90074510 msgr2=0x7f5d90076950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.226+0000 7f5da98e4700 1 --2- 172.21.15.16:0/247922506 >> [v2:172.21.15.49:6816/3830572508,v1:172.21.15.49:6817/3830572508] conn(0x7f5d90074510 0x7f5d90076950 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.295 INFO:teuthology.orchestra.run.smithi016.stdout:55834574858 2023-10-16T12:11:06.295 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-16T12:11:06.296 INFO:teuthology.orchestra.run.smithi016.stdout:55834574857 2023-10-16T12:11:06.296 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-16T12:11:06.297 INFO:teuthology.orchestra.run.smithi016.stdout:55834574859 2023-10-16T12:11:06.298 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-10-16T12:11:06.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.227+0000 7f5da98e4700 1 -- 172.21.15.16:0/247922506 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f5d9006af50 msgr2=0x7f5d9006d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.227+0000 7f5da98e4700 1 --2- 172.21.15.16:0/247922506 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f5d9006af50 0x7f5d9006d410 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f5da40c2070 tx=0x7f5d94009040 comp rx=0 tx=0).stop 2023-10-16T12:11:06.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.227+0000 7f5da98e4700 1 -- 172.21.15.16:0/247922506 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5da40ca180 msgr2=0x7f5da40b4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.227+0000 7f5da98e4700 1 --2- 172.21.15.16:0/247922506 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5da40ca180 0x7f5da40b4af0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5d9c0094c0 tx=0x7f5d9c004ad0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.227+0000 7f5da37fe700 1 -- 172.21.15.16:0/247922506 reap_dead start 2023-10-16T12:11:06.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.228+0000 7f5da98e4700 1 -- 172.21.15.16:0/247922506 shutdown_connections 2023-10-16T12:11:06.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.228+0000 7f5da98e4700 1 -- 172.21.15.16:0/247922506 >> 172.21.15.16:0/247922506 conn(0x7f5da40cc5b0 msgr2=0x7f5da40af610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:06.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.228+0000 7f5da98e4700 1 -- 172.21.15.16:0/247922506 shutdown_connections 2023-10-16T12:11:06.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.228+0000 7f5da98e4700 1 -- 172.21.15.16:0/247922506 wait complete. 2023-10-16T12:11:06.301 INFO:teuthology.orchestra.run.smithi016.stdout:55834574856 2023-10-16T12:11:06.301 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-10-16T12:11:06.302 INFO:teuthology.orchestra.run.smithi016.stdout:55834574857 2023-10-16T12:11:06.303 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-10-16T12:11:06.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.245+0000 7f11fbc84700 1 -- 172.21.15.16:0/4168491569 --> [v2:172.21.15.49:6808/3582923927,v1:172.21.15.49:6809/3582923927] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f11f40b3870 con 0x7f11d80744c0 2023-10-16T12:11:06.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.246+0000 7f11ea7fc700 1 -- 172.21.15.16:0/4168491569 <== osd.6 v2:172.21.15.49:6808/3582923927 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f11ec003d90 con 0x7f11d80744c0 2023-10-16T12:11:06.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.246+0000 7f11fbc84700 1 -- 172.21.15.16:0/4168491569 >> [v2:172.21.15.49:6808/3582923927,v1:172.21.15.49:6809/3582923927] conn(0x7f11d80744c0 msgr2=0x7f11d8076900 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.246+0000 7f11fbc84700 1 --2- 172.21.15.16:0/4168491569 >> [v2:172.21.15.49:6808/3582923927,v1:172.21.15.49:6809/3582923927] conn(0x7f11d80744c0 0x7f11d8076900 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.246+0000 7f11fbc84700 1 -- 172.21.15.16:0/4168491569 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f11d806af00 msgr2=0x7f11d806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.246+0000 7f11fbc84700 1 --2- 172.21.15.16:0/4168491569 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f11d806af00 0x7f11d806d3c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f11e4007050 tx=0x7f11e4007e10 comp rx=0 tx=0).stop 2023-10-16T12:11:06.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.246+0000 7f11fbc84700 1 -- 172.21.15.16:0/4168491569 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f11f41599f0 msgr2=0x7f11f40ade30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.246+0000 7f11fbc84700 1 --2- 172.21.15.16:0/4168491569 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f11f41599f0 0x7f11f40ade30 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f11f000b790 tx=0x7f11f000bb50 comp rx=0 tx=0).stop 2023-10-16T12:11:06.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.246+0000 7f11fa221700 1 -- 172.21.15.16:0/4168491569 reap_dead start 2023-10-16T12:11:06.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.247+0000 7f11fbc84700 1 -- 172.21.15.16:0/4168491569 shutdown_connections 2023-10-16T12:11:06.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.247+0000 7f11fbc84700 1 -- 172.21.15.16:0/4168491569 >> 172.21.15.16:0/4168491569 conn(0x7f11f406a7e0 msgr2=0x7f11f415c450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:06.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.247+0000 7f11fbc84700 1 -- 172.21.15.16:0/4168491569 shutdown_connections 2023-10-16T12:11:06.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.247+0000 7f11fbc84700 1 -- 172.21.15.16:0/4168491569 wait complete. 2023-10-16T12:11:06.307 INFO:teuthology.orchestra.run.smithi016.stdout:55834574857 2023-10-16T12:11:06.308 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-10-16T12:11:06.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.542+0000 7f226a208700 1 Processor -- start 2023-10-16T12:11:06.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.542+0000 7f226a208700 1 -- start start 2023-10-16T12:11:06.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.543+0000 7f226a208700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f22640ae570 0x7f226415e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.543+0000 7f226a208700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2264161440 0x7f22640cef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.543+0000 7f226a208700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f226415ee10 con 0x7f22640ae950 2023-10-16T12:11:06.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.543+0000 7f226a208700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22640b2220 con 0x7f2264161440 2023-10-16T12:11:06.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.543+0000 7f226a208700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2264161a00 con 0x7f22640ae570 2023-10-16T12:11:06.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.543+0000 7f2263fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2264161440 0x7f22640cef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.543+0000 7f2263fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2264161440 0x7f22640cef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33648/0 (socket says 172.21.15.16:33648) 2023-10-16T12:11:06.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.543+0000 7f2263fff700 1 -- 172.21.15.16:0/620006422 learned_addr learned my addr 172.21.15.16:0/620006422 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:06.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.544+0000 7f2263fff700 1 -- 172.21.15.16:0/620006422 >> v2:172.21.15.49:3300/0 conn(0x7f22640ae570 msgr2=0x7f226415e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:06.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.544+0000 7f2263fff700 1 --2- 172.21.15.16:0/620006422 >> v2:172.21.15.49:3300/0 conn(0x7f22640ae570 0x7f226415e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.544+0000 7f2263fff700 1 -- 172.21.15.16:0/620006422 >> v1:172.21.15.16:6789/0 conn(0x7f22640ae950 legacy=0x7f2264070ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.544+0000 7f2263fff700 1 -- 172.21.15.16:0/620006422 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2264161ce0 con 0x7f2264161440 2023-10-16T12:11:06.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.544+0000 7f2263fff700 1 --2- 172.21.15.16:0/620006422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2264161440 0x7f22640cef30 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f226406e370 tx=0x7f2258009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec4fd5f12992b43 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:06.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.545+0000 7fd4a22b3700 1 Processor -- start 2023-10-16T12:11:06.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.546+0000 7f2e8f8b2700 1 Processor -- start 2023-10-16T12:11:06.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.545+0000 7fd4a22b3700 1 -- start start 2023-10-16T12:11:06.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.546+0000 7fd4a22b3700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd49c161c90 0x7fd49c070860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.546+0000 7fd4a22b3700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd49c0b38b0 con 0x7fd49c0ceff0 2023-10-16T12:11:06.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.546+0000 7fd4a22b3700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd49c0b32d0 con 0x7fd49c0cd180 2023-10-16T12:11:06.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.546+0000 7fd4a22b3700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd49c071530 con 0x7fd49c161c90 2023-10-16T12:11:06.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.547+0000 7fd49b7fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd49c0ceff0 0x7fd49c06c800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51162/0 (socket says 172.21.15.16:51162) 2023-10-16T12:11:06.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.547+0000 7fd49bfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd49c161c90 0x7fd49c070860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.547+0000 7fd49b7fe700 1 -- 172.21.15.16:0/1797438605 learned_addr learned my addr 172.21.15.16:0/1797438605 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:06.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.547+0000 7fd49bfff700 1 -- 172.21.15.16:0/1797438605 >> v1:172.21.15.49:6789/0 conn(0x7fd49c0cd180 legacy=0x7fd49c06cf10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-16T12:11:06.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.547+0000 7fd49bfff700 1 -- 172.21.15.16:0/1797438605 >> v1:172.21.15.16:6789/0 conn(0x7fd49c0ceff0 legacy=0x7fd49c06c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.547+0000 7fd49bfff700 1 -- 172.21.15.16:0/1797438605 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd49c06d620 con 0x7fd49c161c90 2023-10-16T12:11:06.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.547+0000 7f22627fc700 1 -- 172.21.15.16:0/620006422 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2258004230 con 0x7f2264161440 2023-10-16T12:11:06.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.547+0000 7f22627fc700 1 -- 172.21.15.16:0/620006422 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2258004390 con 0x7f2264161440 2023-10-16T12:11:06.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.547+0000 7fd49bfff700 1 --2- 172.21.15.16:0/1797438605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd49c161c90 0x7fd49c070860 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd494009d90 tx=0x7fd49400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c001615ab53ac919 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:06.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.546+0000 7f2e8f8b2700 1 -- start start 2023-10-16T12:11:06.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.547+0000 7f2e8f8b2700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2e88161c80 0x7f2e880d0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.547+0000 7f2e8f8b2700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e88165d70 con 0x7f2e88166200 2023-10-16T12:11:06.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.547+0000 7f2e8f8b2700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e880ce920 con 0x7f2e8806a7e0 2023-10-16T12:11:06.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.547+0000 7f2e8f8b2700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e880ceee0 con 0x7f2e88161c80 2023-10-16T12:11:06.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.547+0000 7f2e8d64e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2e88161c80 0x7f2e880d0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.547+0000 7f2e8d64e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2e88161c80 0x7f2e880d0ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33660/0 (socket says 172.21.15.16:33660) 2023-10-16T12:11:06.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.547+0000 7f2e8d64e700 1 -- 172.21.15.16:0/4191459500 learned_addr learned my addr 172.21.15.16:0/4191459500 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:06.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.548+0000 7f22627fc700 1 -- 172.21.15.16:0/620006422 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22580036a0 con 0x7f2264161440 2023-10-16T12:11:06.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.548+0000 7fd49affd700 1 -- 172.21.15.16:0/1797438605 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd49400e070 con 0x7fd49c161c90 2023-10-16T12:11:06.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.548+0000 7fd49affd700 1 -- 172.21.15.16:0/1797438605 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd49400bb90 con 0x7fd49c161c90 2023-10-16T12:11:06.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.548+0000 7fd49affd700 1 -- 172.21.15.16:0/1797438605 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd494003a20 con 0x7fd49c161c90 2023-10-16T12:11:06.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.548+0000 7f099a6bb700 1 Processor -- start 2023-10-16T12:11:06.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.548+0000 7f226a208700 1 -- 172.21.15.16:0/620006422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2264161440 msgr2=0x7f22640cef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.548+0000 7f226a208700 1 --2- 172.21.15.16:0/620006422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2264161440 0x7f22640cef30 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f226406e370 tx=0x7f2258009bd0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.548+0000 7f099a6bb700 1 -- start start 2023-10-16T12:11:06.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.548+0000 7f2e8d64e700 1 -- 172.21.15.16:0/4191459500 >> v1:172.21.15.49:6789/0 conn(0x7f2e8806a7e0 legacy=0x7f2e880cce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:06.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.548+0000 7f2e8d64e700 1 -- 172.21.15.16:0/4191459500 >> v1:172.21.15.16:6789/0 conn(0x7f2e88166200 legacy=0x7f2e880ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.548+0000 7f2e8d64e700 1 -- 172.21.15.16:0/4191459500 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e880cec00 con 0x7f2e88161c80 2023-10-16T12:11:06.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.548+0000 7f2e8d64e700 1 --2- 172.21.15.16:0/4191459500 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e88161c80 0x7f2e880d0ec0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f2e840098f0 tx=0x7f2e84009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e468802e7fd0955e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:06.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.548+0000 7f2e7ffff700 1 -- 172.21.15.16:0/4191459500 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e84004230 con 0x7f2e88161c80 2023-10-16T12:11:06.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.548+0000 7f2e7ffff700 1 -- 172.21.15.16:0/4191459500 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e84004390 con 0x7f2e88161c80 2023-10-16T12:11:06.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.548+0000 7f2e7ffff700 1 -- 172.21.15.16:0/4191459500 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e840036a0 con 0x7f2e88161c80 2023-10-16T12:11:06.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.549+0000 7f3244e6e700 1 Processor -- start 2023-10-16T12:11:06.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.549+0000 7f2e8f8b2700 1 -- 172.21.15.16:0/4191459500 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e88161c80 msgr2=0x7f2e880d0ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.549+0000 7f2e8f8b2700 1 --2- 172.21.15.16:0/4191459500 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e88161c80 0x7f2e880d0ec0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f2e840098f0 tx=0x7f2e84009c00 comp rx=0 tx=0).stop 2023-10-16T12:11:06.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.549+0000 7fd4a22b3700 1 -- 172.21.15.16:0/1797438605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd49c161c90 msgr2=0x7fd49c070860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.549+0000 7fd4a22b3700 1 --2- 172.21.15.16:0/1797438605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd49c161c90 0x7fd49c070860 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd494009d90 tx=0x7fd49400b330 comp rx=0 tx=0).stop 2023-10-16T12:11:06.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.550+0000 7f226a208700 1 -- 172.21.15.16:0/620006422 shutdown_connections 2023-10-16T12:11:06.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.550+0000 7f226a208700 1 --2- 172.21.15.16:0/620006422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2264161440 0x7f22640cef30 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.550+0000 7f226a208700 1 --2- 172.21.15.16:0/620006422 >> v2:172.21.15.49:3300/0 conn(0x7f22640ae570 0x7f226415e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.550+0000 7f226a208700 1 -- 172.21.15.16:0/620006422 >> 172.21.15.16:0/620006422 conn(0x7f226406a7e0 msgr2=0x7f226406abf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:06.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.549+0000 7f099a6bb700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f09940ae8e0 0x7f09940ca180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.549+0000 7f099a6bb700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f09940cab20 0x7f09940cafb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.550+0000 7f099a6bb700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09940c2ec0 con 0x7f09941652b0 2023-10-16T12:11:06.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.550+0000 7f099a6bb700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09940cb4f0 con 0x7f09940cab20 2023-10-16T12:11:06.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.550+0000 7f099a6bb700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f099406c1a0 con 0x7f09940ae8e0 2023-10-16T12:11:06.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.550+0000 7f2e8f8b2700 1 -- 172.21.15.16:0/4191459500 shutdown_connections 2023-10-16T12:11:06.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.550+0000 7f2e8f8b2700 1 --2- 172.21.15.16:0/4191459500 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e88161c80 0x7f2e880d0ec0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.550+0000 7f2e8f8b2700 1 -- 172.21.15.16:0/4191459500 >> 172.21.15.16:0/4191459500 conn(0x7f2e880b1fd0 msgr2=0x7f2e880b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:06.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.550+0000 7f09937fe700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f09940ae8e0 0x7f09940ca180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.550+0000 7fd4a22b3700 1 -- 172.21.15.16:0/1797438605 shutdown_connections 2023-10-16T12:11:06.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.550+0000 7fd4a22b3700 1 --2- 172.21.15.16:0/1797438605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd49c161c90 0x7fd49c070860 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.550+0000 7fd4a22b3700 1 -- 172.21.15.16:0/1797438605 >> 172.21.15.16:0/1797438605 conn(0x7fd49c0b4cd0 msgr2=0x7fd49c06a7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:06.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.550+0000 7f09937fe700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f09940ae8e0 0x7f09940ca180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:50156/0 (socket says 172.21.15.16:50156) 2023-10-16T12:11:06.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.550+0000 7f0998c58700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f09940cab20 0x7f09940cafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.550+0000 7f09937fe700 1 -- 172.21.15.16:0/2663611833 learned_addr learned my addr 172.21.15.16:0/2663611833 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:06.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.549+0000 7f3244e6e700 1 -- start start 2023-10-16T12:11:06.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.550+0000 7f226a208700 1 -- 172.21.15.16:0/620006422 shutdown_connections 2023-10-16T12:11:06.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.550+0000 7fd4a22b3700 1 -- 172.21.15.16:0/1797438605 shutdown_connections 2023-10-16T12:11:06.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.550+0000 7f09937fe700 1 -- 172.21.15.16:0/2663611833 >> v2:172.21.15.16:3300/0 conn(0x7f09940cab20 msgr2=0x7f09940cafb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:06.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.550+0000 7f09937fe700 1 --2- 172.21.15.16:0/2663611833 >> v2:172.21.15.16:3300/0 conn(0x7f09940cab20 0x7f09940cafb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.550+0000 7f09937fe700 1 -- 172.21.15.16:0/2663611833 >> v1:172.21.15.49:6789/0 conn(0x7f09941652b0 legacy=0x7f099406ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.550+0000 7f09937fe700 1 -- 172.21.15.16:0/2663611833 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09940c3440 con 0x7f09940ae8e0 2023-10-16T12:11:06.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.550+0000 7f09937fe700 1 --2- 172.21.15.16:0/2663611833 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f09940ae8e0 0x7f09940ca180 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f098c009e50 tx=0x7f098c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f19d659ae28179c7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:06.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.550+0000 7f3244e6e700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f32400d3dd0 0x7f32400b64f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.550+0000 7f3244e6e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f32400d41b0 0x7f3240165880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.550+0000 7f3244e6e700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32400cfe90 con 0x7f3240165dc0 2023-10-16T12:11:06.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.551+0000 7f3244e6e700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32400d18d0 con 0x7f32400d3dd0 2023-10-16T12:11:06.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.551+0000 7f3244e6e700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3240166250 con 0x7f32400d41b0 2023-10-16T12:11:06.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.550+0000 7f04520ee700 1 Processor -- start 2023-10-16T12:11:06.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.551+0000 7f2e8f8b2700 1 -- 172.21.15.16:0/4191459500 shutdown_connections 2023-10-16T12:11:06.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.551+0000 7f323e59c700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f32400d3dd0 0x7f32400b64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.551+0000 7f323e59c700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f32400d3dd0 0x7f32400b64f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:60966/0 (socket says 172.21.15.16:60966) 2023-10-16T12:11:06.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.551+0000 7f323e59c700 1 -- 172.21.15.16:0/3649497660 learned_addr learned my addr 172.21.15.16:0/3649497660 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:06.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.550+0000 7f226a208700 1 -- 172.21.15.16:0/620006422 wait complete. 2023-10-16T12:11:06.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.551+0000 7f323dd9b700 1 --2- 172.21.15.16:0/3649497660 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f32400d41b0 0x7f3240165880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.551+0000 7f323e59c700 1 -- 172.21.15.16:0/3649497660 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f32400d41b0 msgr2=0x7f3240165880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:06.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.551+0000 7f323e59c700 1 --2- 172.21.15.16:0/3649497660 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f32400d41b0 0x7f3240165880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.551+0000 7f323e59c700 1 -- 172.21.15.16:0/3649497660 >> v1:172.21.15.16:6789/0 conn(0x7f3240165dc0 legacy=0x7f32400d1cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:06.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.551+0000 7f323e59c700 1 -- 172.21.15.16:0/3649497660 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32400d1280 con 0x7f32400d3dd0 2023-10-16T12:11:06.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.551+0000 7f0992ffd700 1 -- 172.21.15.16:0/2663611833 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f098c00e070 con 0x7f09940ae8e0 2023-10-16T12:11:06.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.552+0000 7f226a208700 1 Processor -- start 2023-10-16T12:11:06.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.551+0000 7f0992ffd700 1 -- 172.21.15.16:0/2663611833 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f098c00b910 con 0x7f09940ae8e0 2023-10-16T12:11:06.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.551+0000 7f0992ffd700 1 -- 172.21.15.16:0/2663611833 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f098c00bcc0 con 0x7f09940ae8e0 2023-10-16T12:11:06.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.552+0000 7f099a6bb700 1 -- 172.21.15.16:0/2663611833 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f09940ae8e0 msgr2=0x7f09940ca180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.552+0000 7f099a6bb700 1 --2- 172.21.15.16:0/2663611833 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f09940ae8e0 0x7f09940ca180 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f098c009e50 tx=0x7f098c00b330 comp rx=0 tx=0).stop 2023-10-16T12:11:06.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.553+0000 7f099a6bb700 1 -- 172.21.15.16:0/2663611833 shutdown_connections 2023-10-16T12:11:06.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.553+0000 7f099a6bb700 1 --2- 172.21.15.16:0/2663611833 >> v2:172.21.15.16:3300/0 conn(0x7f09940cab20 0x7f09940cafb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.553+0000 7f099a6bb700 1 --2- 172.21.15.16:0/2663611833 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f09940ae8e0 0x7f09940ca180 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.553+0000 7f099a6bb700 1 -- 172.21.15.16:0/2663611833 >> 172.21.15.16:0/2663611833 conn(0x7f09940cc5b0 msgr2=0x7f09940cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:06.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.553+0000 7f2e8f8b2700 1 -- 172.21.15.16:0/4191459500 wait complete. 2023-10-16T12:11:06.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.550+0000 7fd4a22b3700 1 -- 172.21.15.16:0/1797438605 wait complete. 2023-10-16T12:11:06.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.552+0000 7fd4a22b3700 1 Processor -- start 2023-10-16T12:11:06.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.552+0000 7fd4a22b3700 1 -- start start 2023-10-16T12:11:06.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.552+0000 7f04520ee700 1 -- start start 2023-10-16T12:11:06.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.553+0000 7f04520ee700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f044c1599f0 0x7f044c15ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.553+0000 7f04520ee700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f044c15e300 con 0x7f044c15b040 2023-10-16T12:11:06.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.553+0000 7f04520ee700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f044c06d620 con 0x7f044c071250 2023-10-16T12:11:06.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.553+0000 7f04520ee700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f044c15d240 con 0x7f044c1599f0 2023-10-16T12:11:06.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.553+0000 7f323e59c700 1 --2- 172.21.15.16:0/3649497660 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f32400d3dd0 0x7f32400b64f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f32300098f0 tx=0x7f3230009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c62dde36c2145af server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:06.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.553+0000 7f044bfff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f044c15b040 0x7f044c06cf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51188/0 (socket says 172.21.15.16:51188) 2023-10-16T12:11:06.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.553+0000 7f044bfff700 1 -- 172.21.15.16:0/1393781044 learned_addr learned my addr 172.21.15.16:0/1393781044 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:06.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.552+0000 7f226a208700 1 -- start start 2023-10-16T12:11:06.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.553+0000 7f226a208700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22640ae570 0x7f22641533c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.553+0000 7f226a208700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f22640ae950 0x7f22640ce380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.553+0000 7f226a208700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2264161440 0x7f22641562f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.553+0000 7f226a208700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f226415f470 con 0x7f22640ae570 2023-10-16T12:11:06.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.553+0000 7f226a208700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2264159170 con 0x7f22640ae950 2023-10-16T12:11:06.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.553+0000 7f226a208700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2264158bb0 con 0x7f2264161440 2023-10-16T12:11:06.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.553+0000 7f22637fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22640ae570 0x7f22641533c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.553+0000 7f22637fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22640ae570 0x7f22641533c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33680/0 (socket says 172.21.15.16:33680) 2023-10-16T12:11:06.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.553+0000 7f22637fe700 1 -- 172.21.15.16:0/1604361620 learned_addr learned my addr 172.21.15.16:0/1604361620 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:06.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.553+0000 7fd4a22b3700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd49c0cd180 0x7fd49c152850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.553+0000 7fd4a22b3700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd49c0ceff0 0x7fd49c1639a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.553+0000 7fd4a22b3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd49c161c90 0x7fd49c0cdb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.553+0000 7fd4a22b3700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd49c158e90 con 0x7fd49c161c90 2023-10-16T12:11:06.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.553+0000 7fd4a22b3700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd49c164d00 con 0x7fd49c0ceff0 2023-10-16T12:11:06.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.553+0000 7fd4a22b3700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd49c164460 con 0x7fd49c0cd180 2023-10-16T12:11:06.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.554+0000 7fd4a0850700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd49c161c90 0x7fd49c0cdb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.554+0000 7fd4a0850700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd49c161c90 0x7fd49c0cdb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33686/0 (socket says 172.21.15.16:33686) 2023-10-16T12:11:06.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.554+0000 7fd4a0850700 1 -- 172.21.15.16:0/1553026101 learned_addr learned my addr 172.21.15.16:0/1553026101 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:06.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.553+0000 7f323d59a700 1 -- 172.21.15.16:0/3649497660 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3230004230 con 0x7f32400d3dd0 2023-10-16T12:11:06.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.554+0000 7f323d59a700 1 -- 172.21.15.16:0/3649497660 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3230004390 con 0x7f32400d3dd0 2023-10-16T12:11:06.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.554+0000 7f323d59a700 1 -- 172.21.15.16:0/3649497660 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32300036a0 con 0x7f32400d3dd0 2023-10-16T12:11:06.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.554+0000 7f2262ffd700 1 --2- 172.21.15.16:0/1604361620 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f22640ae950 0x7f22640ce380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.554+0000 7fd49b7fe700 1 --2- 172.21.15.16:0/1553026101 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd49c0ceff0 0x7fd49c1639a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.554+0000 7f22637fe700 1 -- 172.21.15.16:0/1604361620 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2264161440 msgr2=0x7f22641562f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-16T12:11:06.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.554+0000 7f22637fe700 1 --2- 172.21.15.16:0/1604361620 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2264161440 0x7f22641562f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.554+0000 7f22637fe700 1 -- 172.21.15.16:0/1604361620 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f22640ae950 msgr2=0x7f22640ce380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.554+0000 7f22637fe700 1 --2- 172.21.15.16:0/1604361620 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f22640ae950 0x7f22640ce380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.554+0000 7f22637fe700 1 -- 172.21.15.16:0/1604361620 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2258009550 con 0x7f22640ae570 2023-10-16T12:11:06.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.554+0000 7fd4a0850700 1 -- 172.21.15.16:0/1553026101 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd49c0cd180 msgr2=0x7fd49c152850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.554+0000 7fd4a0850700 1 --2- 172.21.15.16:0/1553026101 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd49c0cd180 0x7fd49c152850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.554+0000 7fd4a0850700 1 -- 172.21.15.16:0/1553026101 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd49c0ceff0 msgr2=0x7fd49c1639a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.554+0000 7fd4a0850700 1 --2- 172.21.15.16:0/1553026101 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd49c0ceff0 0x7fd49c1639a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.554+0000 7fd4a0850700 1 -- 172.21.15.16:0/1553026101 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd494009a70 con 0x7fd49c161c90 2023-10-16T12:11:06.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.554+0000 7f22637fe700 1 --2- 172.21.15.16:0/1604361620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22640ae570 0x7f22641533c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f225c00ba50 tx=0x7f225c00bd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:06.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.555+0000 7f3244e6e700 1 -- 172.21.15.16:0/3649497660 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f32400d3dd0 msgr2=0x7f32400b64f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.555+0000 7f2e8f8b2700 1 Processor -- start 2023-10-16T12:11:06.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.555+0000 7f3244e6e700 1 --2- 172.21.15.16:0/3649497660 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f32400d3dd0 0x7f32400b64f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f32300098f0 tx=0x7f3230009c00 comp rx=0 tx=0).stop 2023-10-16T12:11:06.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.555+0000 7f044b7fe700 1 --2- 172.21.15.16:0/1393781044 >> v2:172.21.15.49:3300/0 conn(0x7f044c1599f0 0x7f044c15ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.555+0000 7f2e8f8b2700 1 -- start start 2023-10-16T12:11:06.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.555+0000 7f044a7fc700 1 -- 172.21.15.16:0/1393781044 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1859685680 0 0) 0x7f044c15e300 con 0x7f044c15b040 2023-10-16T12:11:06.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.555+0000 7f044a7fc700 1 -- 172.21.15.16:0/1393781044 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04340035b0 con 0x7f044c15b040 2023-10-16T12:11:06.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.555+0000 7f044a7fc700 1 -- 172.21.15.16:0/1393781044 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3481404813 0 0) 0x7f044c06d620 con 0x7f044c071250 2023-10-16T12:11:06.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.555+0000 7f044a7fc700 1 -- 172.21.15.16:0/1393781044 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f044c15e300 con 0x7f044c071250 2023-10-16T12:11:06.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.555+0000 7f224bfff700 1 -- 172.21.15.16:0/1604361620 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f225c00c9b0 con 0x7f22640ae570 2023-10-16T12:11:06.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.555+0000 7f224bfff700 1 -- 172.21.15.16:0/1604361620 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f225c00cb30 con 0x7f22640ae570 2023-10-16T12:11:06.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.555+0000 7f224bfff700 1 -- 172.21.15.16:0/1604361620 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f225c014440 con 0x7f22640ae570 2023-10-16T12:11:06.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.555+0000 7f226a208700 1 -- 172.21.15.16:0/1604361620 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2264158e90 con 0x7f22640ae570 2023-10-16T12:11:06.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.555+0000 7f044a7fc700 1 -- 172.21.15.16:0/1393781044 <== mon.1 v1:172.21.15.49:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3389384981 0 0) 0x7f044c15e300 con 0x7f044c071250 2023-10-16T12:11:06.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.555+0000 7f044a7fc700 1 -- 172.21.15.16:0/1393781044 >> v2:172.21.15.49:3300/0 conn(0x7f044c1599f0 msgr2=0x7f044c15ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:06.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.555+0000 7f044a7fc700 1 --2- 172.21.15.16:0/1393781044 >> v2:172.21.15.49:3300/0 conn(0x7f044c1599f0 0x7f044c15ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.555+0000 7f044a7fc700 1 -- 172.21.15.16:0/1393781044 >> v1:172.21.15.16:6789/0 conn(0x7f044c15b040 legacy=0x7f044c06cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.555+0000 7f044a7fc700 1 -- 172.21.15.16:0/1393781044 --> v1:172.21.15.49:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f044c0b2fd0 con 0x7f044c071250 2023-10-16T12:11:06.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.555+0000 7f044a7fc700 1 -- 172.21.15.16:0/1393781044 <== mon.1 v1:172.21.15.49:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f0440001520 con 0x7f044c071250 2023-10-16T12:11:06.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.555+0000 7f044a7fc700 1 -- 172.21.15.16:0/1393781044 >> v1:172.21.15.49:6789/0 conn(0x7f044c071250 legacy=0x7f044c06c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.555+0000 7f044a7fc700 1 --2- 172.21.15.16:0/1393781044 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0434004c40 0x7f0434005040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.555+0000 7f044a7fc700 1 --2- 172.21.15.16:0/1393781044 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f04340037f0 0x7f0434003c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.555+0000 7f044a7fc700 1 --2- 172.21.15.16:0/1393781044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0434004200 0x7f043400ba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.555+0000 7f044a7fc700 1 -- 172.21.15.16:0/1393781044 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f043400c1e0 con 0x7f0434004200 2023-10-16T12:11:06.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.555+0000 7f044a7fc700 1 -- 172.21.15.16:0/1393781044 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f043400c350 con 0x7f0434004c40 2023-10-16T12:11:06.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.555+0000 7f044a7fc700 1 -- 172.21.15.16:0/1393781044 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f043400c510 con 0x7f04340037f0 2023-10-16T12:11:06.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.556+0000 7f2e8f8b2700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e8806a7e0 0x7f2e88162c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.556+0000 7f2e8f8b2700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2e88161c80 0x7f2e881f7390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.555+0000 7f099a6bb700 1 -- 172.21.15.16:0/2663611833 shutdown_connections 2023-10-16T12:11:06.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.555+0000 7f099a6bb700 1 -- 172.21.15.16:0/2663611833 wait complete. 2023-10-16T12:11:06.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.556+0000 7f2e8f8b2700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2e88166200 0x7f2e8815a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.556+0000 7f2e8f8b2700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e88163700 con 0x7f2e8806a7e0 2023-10-16T12:11:06.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.556+0000 7f2e8f8b2700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e881f78d0 con 0x7f2e88166200 2023-10-16T12:11:06.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.556+0000 7f2e8f8b2700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e8815a7c0 con 0x7f2e88161c80 2023-10-16T12:11:06.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.556+0000 7f2e8d64e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e8806a7e0 0x7f2e88162c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.556+0000 7f2e8d64e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e8806a7e0 0x7f2e88162c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33698/0 (socket says 172.21.15.16:33698) 2023-10-16T12:11:06.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.556+0000 7f2e8d64e700 1 -- 172.21.15.16:0/4027063823 learned_addr learned my addr 172.21.15.16:0/4027063823 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:06.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.556+0000 7f2e8de4f700 1 --2- 172.21.15.16:0/4027063823 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2e88166200 0x7f2e8815a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.557+0000 7f2e8ce4d700 1 --2- 172.21.15.16:0/4027063823 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2e88161c80 0x7f2e881f7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.557+0000 7f044bfff700 1 --2- 172.21.15.16:0/1393781044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0434004200 0x7f043400ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.557+0000 7f044affd700 1 --2- 172.21.15.16:0/1393781044 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f04340037f0 0x7f0434003c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.557+0000 7f2e8de4f700 1 -- 172.21.15.16:0/4027063823 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2e88161c80 msgr2=0x7f2e881f7390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.557+0000 7f2e8de4f700 1 --2- 172.21.15.16:0/4027063823 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2e88161c80 0x7f2e881f7390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.557+0000 7f2e8de4f700 1 -- 172.21.15.16:0/4027063823 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e8806a7e0 msgr2=0x7f2e88162c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.557+0000 7f2e8de4f700 1 --2- 172.21.15.16:0/4027063823 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e8806a7e0 0x7f2e88162c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.557+0000 7f2e8de4f700 1 -- 172.21.15.16:0/4027063823 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e84009550 con 0x7f2e88166200 2023-10-16T12:11:06.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.557+0000 7f044bfff700 1 -- 172.21.15.16:0/1393781044 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f04340037f0 msgr2=0x7f0434003c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:06.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.557+0000 7f044bfff700 1 --2- 172.21.15.16:0/1393781044 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f04340037f0 0x7f0434003c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.557+0000 7f044bfff700 1 -- 172.21.15.16:0/1393781044 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0434004c40 msgr2=0x7f0434005040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:06.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.557+0000 7f044bfff700 1 --2- 172.21.15.16:0/1393781044 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0434004c40 0x7f0434005040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.557+0000 7f044bfff700 1 -- 172.21.15.16:0/1393781044 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f044c0b2fd0 con 0x7f0434004200 2023-10-16T12:11:06.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.557+0000 7f044bfff700 1 -- 172.21.15.16:0/1393781044 reap_dead start 2023-10-16T12:11:06.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.557+0000 7f099a6bb700 1 Processor -- start 2023-10-16T12:11:06.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.557+0000 7f2e8de4f700 1 --2- 172.21.15.16:0/4027063823 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2e88166200 0x7f2e8815a280 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2e8000be20 tx=0x7f2e80009ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:06.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.554+0000 7fd4a0850700 1 --2- 172.21.15.16:0/1553026101 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd49c161c90 0x7fd49c0cdb00 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd49000b3f0 tx=0x7fd49000b700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:06.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.555+0000 7fd498ff9700 1 -- 172.21.15.16:0/1553026101 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd490011830 con 0x7fd49c161c90 2023-10-16T12:11:06.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.555+0000 7fd498ff9700 1 -- 172.21.15.16:0/1553026101 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4900119b0 con 0x7fd49c161c90 2023-10-16T12:11:06.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.555+0000 7fd4a22b3700 1 -- 172.21.15.16:0/1553026101 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd49c164740 con 0x7fd49c161c90 2023-10-16T12:11:06.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.555+0000 7fd498ff9700 1 -- 172.21.15.16:0/1553026101 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd490011d60 con 0x7fd49c161c90 2023-10-16T12:11:06.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.556+0000 7fd4a22b3700 1 -- 172.21.15.16:0/1553026101 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd49c1622a0 con 0x7fd49c161c90 2023-10-16T12:11:06.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.557+0000 7fd4a22b3700 1 -- 172.21.15.16:0/1553026101 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd49c04ebb0 con 0x7fd49c161c90 2023-10-16T12:11:06.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.558+0000 7f226a208700 1 -- 172.21.15.16:0/1604361620 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22640ce8c0 con 0x7f22640ae570 2023-10-16T12:11:06.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.558+0000 7f3244e6e700 1 -- 172.21.15.16:0/3649497660 shutdown_connections 2023-10-16T12:11:06.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.558+0000 7f3244e6e700 1 --2- 172.21.15.16:0/3649497660 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f32400d3dd0 0x7f32400b64f0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.558+0000 7f3244e6e700 1 --2- 172.21.15.16:0/3649497660 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f32400d41b0 0x7f3240165880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.558+0000 7f3244e6e700 1 -- 172.21.15.16:0/3649497660 >> 172.21.15.16:0/3649497660 conn(0x7f3240072330 msgr2=0x7f3240072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:06.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.558+0000 7f044bfff700 1 --2- 172.21.15.16:0/1393781044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0434004200 0x7f043400ba90 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f04440054e0 tx=0x7f0434005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19512c0f5b03069a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:06.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.558+0000 7f3244e6e700 1 -- 172.21.15.16:0/3649497660 shutdown_connections 2023-10-16T12:11:06.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.558+0000 7f3244e6e700 1 -- 172.21.15.16:0/3649497660 wait complete. 2023-10-16T12:11:06.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.559+0000 7f226a208700 1 -- 172.21.15.16:0/1604361620 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f226404ebb0 con 0x7f22640ae570 2023-10-16T12:11:06.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.558+0000 7f2e7dffb700 1 -- 172.21.15.16:0/4027063823 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e8000cbb0 con 0x7f2e88166200 2023-10-16T12:11:06.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.558+0000 7f2e7dffb700 1 -- 172.21.15.16:0/4027063823 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e8000cd50 con 0x7f2e88166200 2023-10-16T12:11:06.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.558+0000 7f2e7dffb700 1 -- 172.21.15.16:0/4027063823 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e80014680 con 0x7f2e88166200 2023-10-16T12:11:06.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.559+0000 7f2e8f8b2700 1 -- 172.21.15.16:0/4027063823 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e881f7b30 con 0x7f2e88166200 2023-10-16T12:11:06.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.559+0000 7f2e8f8b2700 1 -- 172.21.15.16:0/4027063823 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e881f80f0 con 0x7f2e88166200 2023-10-16T12:11:06.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.558+0000 7f099a6bb700 1 -- start start 2023-10-16T12:11:06.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.559+0000 7f099a6bb700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f09940ae8e0 0x7f099415e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.559+0000 7f099a6bb700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09940cab20 0x7f09941f6f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.559+0000 7f099a6bb700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f09941652b0 0x7f09940c4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.559+0000 7f099a6bb700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f099415f770 con 0x7f09940cab20 2023-10-16T12:11:06.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.559+0000 7f099a6bb700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09941f7450 con 0x7f09940ae8e0 2023-10-16T12:11:06.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.559+0000 7f099a6bb700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f099415fb90 con 0x7f09941652b0 2023-10-16T12:11:06.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.559+0000 7f0993fff700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f09940ae8e0 0x7f099415e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.559+0000 7f0993fff700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f09940ae8e0 0x7f099415e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:32768/0 (socket says 172.21.15.16:32768) 2023-10-16T12:11:06.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.559+0000 7f0993fff700 1 -- 172.21.15.16:0/2168211327 learned_addr learned my addr 172.21.15.16:0/2168211327 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:06.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.559+0000 7efe5fed3700 1 Processor -- start 2023-10-16T12:11:06.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.561+0000 7f4a3ed99700 1 Processor -- start 2023-10-16T12:11:06.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.561+0000 7f4a3ed99700 1 -- start start 2023-10-16T12:11:06.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.560+0000 7f0993fff700 1 -- 172.21.15.16:0/2168211327 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f09941652b0 msgr2=0x7f09940c4fd0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-16T12:11:06.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.560+0000 7f0993fff700 1 --2- 172.21.15.16:0/2168211327 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f09941652b0 0x7f09940c4fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.560+0000 7f0993fff700 1 -- 172.21.15.16:0/2168211327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09940cab20 msgr2=0x7f09941f6f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.560+0000 7f0993fff700 1 --2- 172.21.15.16:0/2168211327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09940cab20 0x7f09941f6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.560+0000 7f0993fff700 1 -- 172.21.15.16:0/2168211327 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f098c009b30 con 0x7f09940ae8e0 2023-10-16T12:11:06.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.560+0000 7f2e8f8b2700 1 -- 172.21.15.16:0/4027063823 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e881638b0 con 0x7f2e88166200 2023-10-16T12:11:06.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.560+0000 7f3244e6e700 1 Processor -- start 2023-10-16T12:11:06.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.560+0000 7f3244e6e700 1 -- start start 2023-10-16T12:11:06.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.560+0000 7efe5fed3700 1 -- start start 2023-10-16T12:11:06.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.561+0000 7efe5fed3700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efe581599f0 0x7efe5815ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.561+0000 7efe5fed3700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7efe5815b040 0x7efe580716d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.561+0000 7efe5fed3700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe5815e300 con 0x7efe58071250 2023-10-16T12:11:06.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.561+0000 7efe5fed3700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe5815d240 con 0x7efe5815b040 2023-10-16T12:11:06.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.561+0000 7efe5fed3700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe58071c10 con 0x7efe581599f0 2023-10-16T12:11:06.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.561+0000 7efe5dc6f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efe581599f0 0x7efe5815ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.561+0000 7efe5dc6f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efe581599f0 0x7efe5815ddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:50202/0 (socket says 172.21.15.16:50202) 2023-10-16T12:11:06.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.561+0000 7efe5dc6f700 1 -- 172.21.15.16:0/2922803405 learned_addr learned my addr 172.21.15.16:0/2922803405 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:06.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.561+0000 7f3244e6e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32400d3dd0 0x7f3240156a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.561+0000 7f3244e6e700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f32400d41b0 0x7f3240155cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.561+0000 7f3244e6e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3240165dc0 0x7f32401f8470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.561+0000 7f3244e6e700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32401f89b0 con 0x7f32400d3dd0 2023-10-16T12:11:06.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.561+0000 7f3244e6e700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3240156fe0 con 0x7f32400d41b0 2023-10-16T12:11:06.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.561+0000 7f3244e6e700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32401572c0 con 0x7f3240165dc0 2023-10-16T12:11:06.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.562+0000 7efe5e470700 1 --2- 172.21.15.16:0/2922803405 >> v2:172.21.15.49:3300/0 conn(0x7efe5815b040 0x7efe580716d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.562+0000 7f323ed9d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3240165dc0 0x7f32401f8470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.562+0000 7f323ed9d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3240165dc0 0x7f32401f8470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:50212/0 (socket says 172.21.15.16:50212) 2023-10-16T12:11:06.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.562+0000 7f323ed9d700 1 -- 172.21.15.16:0/1421793764 learned_addr learned my addr 172.21.15.16:0/1421793764 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:06.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.561+0000 7f2e7dffb700 1 -- 172.21.15.16:0/4027063823 <== mon.1 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f2e80006d50 con 0x7f2e88166200 2023-10-16T12:11:06.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.562+0000 7f2e7dffb700 1 --2- 172.21.15.16:0/4027063823 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f2e6c06af50 0x7f2e6c06d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.562+0000 7f2e7dffb700 1 -- 172.21.15.16:0/4027063823 <== mon.1 v2:172.21.15.49:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f2e80089220 con 0x7f2e88166200 2023-10-16T12:11:06.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.562+0000 7f2e8d64e700 1 --2- 172.21.15.16:0/4027063823 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f2e6c06af50 0x7f2e6c06d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.562+0000 7efe5dc6f700 1 -- 172.21.15.16:0/2922803405 >> v2:172.21.15.49:3300/0 conn(0x7efe5815b040 msgr2=0x7efe580716d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:06.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.562+0000 7efe5dc6f700 1 --2- 172.21.15.16:0/2922803405 >> v2:172.21.15.49:3300/0 conn(0x7efe5815b040 0x7efe580716d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.562+0000 7efe5dc6f700 1 -- 172.21.15.16:0/2922803405 >> v1:172.21.15.16:6789/0 conn(0x7efe58071250 legacy=0x7efe5806c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:06.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.562+0000 7efe5dc6f700 1 -- 172.21.15.16:0/2922803405 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe580b2fd0 con 0x7efe581599f0 2023-10-16T12:11:06.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.562+0000 7f4a3ed99700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4a380d2f70 0x7f4a380d2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.562+0000 7f4a3ed99700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4a380ce9f0 0x7f4a380cdc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.562+0000 7f4a3ed99700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a380ce1a0 con 0x7f4a3815d900 2023-10-16T12:11:06.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.562+0000 7f4a3ed99700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a380cee80 con 0x7f4a380d2f70 2023-10-16T12:11:06.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.562+0000 7f4a3ed99700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a380ce400 con 0x7f4a380ce9f0 2023-10-16T12:11:06.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.562+0000 7efe5dc6f700 1 --2- 172.21.15.16:0/2922803405 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efe581599f0 0x7efe5815ddc0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7efe540092e0 tx=0x7efe540095f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8e64c47cb9eb642 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:06.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.562+0000 7f323ed9d700 1 -- 172.21.15.16:0/1421793764 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f32400d41b0 msgr2=0x7f3240155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.562+0000 7f323ed9d700 1 --2- 172.21.15.16:0/1421793764 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f32400d41b0 0x7f3240155cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.562+0000 7f323ed9d700 1 -- 172.21.15.16:0/1421793764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32400d3dd0 msgr2=0x7f3240156a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.562+0000 7f323ed9d700 1 --2- 172.21.15.16:0/1421793764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32400d3dd0 0x7f3240156a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.562+0000 7f323ed9d700 1 -- 172.21.15.16:0/1421793764 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3230009550 con 0x7f3240165dc0 2023-10-16T12:11:06.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.563+0000 7fd498ff9700 1 -- 172.21.15.16:0/1553026101 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fd49001a400 con 0x7fd49c161c90 2023-10-16T12:11:06.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.562+0000 7f4a3cb35700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7f4a3815d900 0x7f4a380d2860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:58414/0 (socket says 172.21.15.16:58414) 2023-10-16T12:11:06.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.562+0000 7f4a3cb35700 1 -- 172.21.15.16:0/3034343447 learned_addr learned my addr 172.21.15.16:0/3034343447 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:06.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.563+0000 7f323ed9d700 1 --2- 172.21.15.16:0/1421793764 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3240165dc0 0x7f32401f8470 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f323800cb60 tx=0x7f323800cf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:06.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.563+0000 7efe5cc6d700 1 -- 172.21.15.16:0/2922803405 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe54004230 con 0x7efe581599f0 2023-10-16T12:11:06.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.563+0000 7f2e8d64e700 1 --2- 172.21.15.16:0/4027063823 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f2e6c06af50 0x7f2e6c06d410 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f2e84004720 tx=0x7f2e84008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:06.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.563+0000 7f4a3d336700 1 --2- 172.21.15.16:0/3034343447 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4a380ce9f0 0x7f4a380cdc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.563+0000 7efe5cc6d700 1 -- 172.21.15.16:0/2922803405 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe54004390 con 0x7efe581599f0 2023-10-16T12:11:06.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.563+0000 7efe5cc6d700 1 -- 172.21.15.16:0/2922803405 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe540036a0 con 0x7efe581599f0 2023-10-16T12:11:06.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.563+0000 7f322effd700 1 -- 172.21.15.16:0/1421793764 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f323800ab10 con 0x7f3240165dc0 2023-10-16T12:11:06.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.563+0000 7fd498ff9700 1 --2- 172.21.15.16:0/1553026101 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fd48806afd0 0x7fd48806d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.563+0000 7fd498ff9700 1 -- 172.21.15.16:0/1553026101 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fd490089d20 con 0x7fd49c161c90 2023-10-16T12:11:06.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.563+0000 7f4a3d336700 1 -- 172.21.15.16:0/3034343447 >> v2:172.21.15.16:3300/0 conn(0x7f4a380d2f70 msgr2=0x7f4a380d2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:06.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.563+0000 7f4a3d336700 1 --2- 172.21.15.16:0/3034343447 >> v2:172.21.15.16:3300/0 conn(0x7f4a380d2f70 0x7f4a380d2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.563+0000 7f4a3d336700 1 -- 172.21.15.16:0/3034343447 >> v1:172.21.15.49:6789/0 conn(0x7f4a3815d900 legacy=0x7f4a380d2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.563+0000 7f4a3d336700 1 -- 172.21.15.16:0/3034343447 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a3815dce0 con 0x7f4a380ce9f0 2023-10-16T12:11:06.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.563+0000 7f4a3d336700 1 --2- 172.21.15.16:0/3034343447 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4a380ce9f0 0x7f4a380cdc60 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4a30009d90 tx=0x7f4a3000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e542160d1666c40e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:06.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.563+0000 7fd49bfff700 1 --2- 172.21.15.16:0/1553026101 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fd48806afd0 0x7fd48806d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.564+0000 7f044a7fc700 1 -- 172.21.15.16:0/1393781044 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04440082e0 con 0x7f0434004200 2023-10-16T12:11:06.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.564+0000 7efe5fed3700 1 -- 172.21.15.16:0/2922803405 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efe581599f0 msgr2=0x7efe5815ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.564+0000 7efe5fed3700 1 --2- 172.21.15.16:0/2922803405 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efe581599f0 0x7efe5815ddc0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7efe540092e0 tx=0x7efe540095f0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.564+0000 7efe5fed3700 1 -- 172.21.15.16:0/2922803405 shutdown_connections 2023-10-16T12:11:06.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.564+0000 7efe5fed3700 1 --2- 172.21.15.16:0/2922803405 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efe581599f0 0x7efe5815ddc0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.564+0000 7efe5fed3700 1 --2- 172.21.15.16:0/2922803405 >> v2:172.21.15.49:3300/0 conn(0x7efe5815b040 0x7efe580716d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.564+0000 7efe5fed3700 1 -- 172.21.15.16:0/2922803405 >> 172.21.15.16:0/2922803405 conn(0x7efe5806a7e0 msgr2=0x7efe5806abf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:06.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.564+0000 7fd49bfff700 1 --2- 172.21.15.16:0/1553026101 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fd48806afd0 0x7fd48806d490 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd49400b370 tx=0x7fd494010000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:06.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.563+0000 7f322effd700 1 -- 172.21.15.16:0/1421793764 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f323800ac70 con 0x7f3240165dc0 2023-10-16T12:11:06.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.563+0000 7f3244e6e700 1 -- 172.21.15.16:0/1421793764 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32401575a0 con 0x7f3240165dc0 2023-10-16T12:11:06.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.563+0000 7f322effd700 1 -- 172.21.15.16:0/1421793764 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3238012420 con 0x7f3240165dc0 2023-10-16T12:11:06.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.563+0000 7f3244e6e700 1 -- 172.21.15.16:0/1421793764 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32401589e0 con 0x7f3240165dc0 2023-10-16T12:11:06.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.564+0000 7f3244e6e700 1 -- 172.21.15.16:0/1421793764 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f324004ebb0 con 0x7f3240165dc0 2023-10-16T12:11:06.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.565+0000 7f4a377fe700 1 -- 172.21.15.16:0/3034343447 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a3000e070 con 0x7f4a380ce9f0 2023-10-16T12:11:06.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.565+0000 7f4a377fe700 1 -- 172.21.15.16:0/3034343447 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a3000bbd0 con 0x7f4a380ce9f0 2023-10-16T12:11:06.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.565+0000 7f4a377fe700 1 -- 172.21.15.16:0/3034343447 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a30012430 con 0x7f4a380ce9f0 2023-10-16T12:11:06.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.565+0000 7efe5fed3700 1 -- 172.21.15.16:0/2922803405 shutdown_connections 2023-10-16T12:11:06.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.565+0000 7efe5fed3700 1 -- 172.21.15.16:0/2922803405 wait complete. 2023-10-16T12:11:06.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.565+0000 7f0993fff700 1 --2- 172.21.15.16:0/2168211327 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f09940ae8e0 0x7f099415e460 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f098400c450 tx=0x7f098400c810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:06.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.565+0000 7f322effd700 1 -- 172.21.15.16:0/1421793764 <== mon.2 v2:172.21.15.49:3301/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f3238012580 con 0x7f3240165dc0 2023-10-16T12:11:06.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.566+0000 7f224bfff700 1 -- 172.21.15.16:0/1604361620 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f225c01d3b0 con 0x7f22640ae570 2023-10-16T12:11:06.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.566+0000 7f322effd700 1 --2- 172.21.15.16:0/1421793764 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f322006af00 0x7f322006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.566+0000 7f4a3ed99700 1 -- 172.21.15.16:0/3034343447 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4a380ce9f0 msgr2=0x7f4a380cdc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.566+0000 7f4a3ed99700 1 --2- 172.21.15.16:0/3034343447 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4a380ce9f0 0x7f4a380cdc60 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4a30009d90 tx=0x7f4a3000b330 comp rx=0 tx=0).stop 2023-10-16T12:11:06.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.566+0000 7f322effd700 1 -- 172.21.15.16:0/1421793764 <== mon.2 v2:172.21.15.49:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f3238089ba0 con 0x7f3240165dc0 2023-10-16T12:11:06.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.566+0000 7f323e59c700 1 --2- 172.21.15.16:0/1421793764 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f322006af00 0x7f322006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.566+0000 7f224bfff700 1 --2- 172.21.15.16:0/1604361620 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f225006af50 0x7f225006d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.566+0000 7f2262ffd700 1 --2- 172.21.15.16:0/1604361620 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f225006af50 0x7f225006d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.566+0000 7f323e59c700 1 --2- 172.21.15.16:0/1421793764 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f322006af00 0x7f322006d3c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f32400d3120 tx=0x7f3230010000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:06.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.567+0000 7f2262ffd700 1 --2- 172.21.15.16:0/1604361620 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f225006af50 0x7f225006d410 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f22641fa6d0 tx=0x7f2254009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:06.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.567+0000 7efe5fed3700 1 Processor -- start 2023-10-16T12:11:06.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.567+0000 7f4a3ed99700 1 -- 172.21.15.16:0/3034343447 shutdown_connections 2023-10-16T12:11:06.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.567+0000 7f4a3ed99700 1 --2- 172.21.15.16:0/3034343447 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4a380ce9f0 0x7f4a380cdc60 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.567+0000 7f4a3ed99700 1 --2- 172.21.15.16:0/3034343447 >> v2:172.21.15.16:3300/0 conn(0x7f4a380d2f70 0x7f4a380d2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.567+0000 7f4a3ed99700 1 -- 172.21.15.16:0/3034343447 >> 172.21.15.16:0/3034343447 conn(0x7f4a38072330 msgr2=0x7f4a38072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:06.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.567+0000 7efe5fed3700 1 -- start start 2023-10-16T12:11:06.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.567+0000 7f0990ff9700 1 -- 172.21.15.16:0/2168211327 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09840052d0 con 0x7f09940ae8e0 2023-10-16T12:11:06.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.567+0000 7f4a3ed99700 1 -- 172.21.15.16:0/3034343447 shutdown_connections 2023-10-16T12:11:06.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.567+0000 7f4a3ed99700 1 -- 172.21.15.16:0/3034343447 wait complete. 2023-10-16T12:11:06.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.567+0000 7f0990ff9700 1 -- 172.21.15.16:0/2168211327 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0984005430 con 0x7f09940ae8e0 2023-10-16T12:11:06.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.567+0000 7f099a6bb700 1 -- 172.21.15.16:0/2168211327 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f099415eaf0 con 0x7f09940ae8e0 2023-10-16T12:11:06.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.567+0000 7f0990ff9700 1 -- 172.21.15.16:0/2168211327 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09840038c0 con 0x7f09940ae8e0 2023-10-16T12:11:06.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.567+0000 7f2e7dffb700 1 -- 172.21.15.16:0/4027063823 <== mon.1 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2e881638b0 con 0x7f2e88166200 2023-10-16T12:11:06.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.567+0000 7f099a6bb700 1 -- 172.21.15.16:0/2168211327 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f099415f0b0 con 0x7f09940ae8e0 2023-10-16T12:11:06.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.568+0000 7efe5fed3700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efe58071250 0x7efe581527c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.568+0000 7efe5fed3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe581599f0 0x7efe581554a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.568+0000 7efe5fed3700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efe5815b040 0x7efe581f2e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.568+0000 7efe5fed3700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe58152d00 con 0x7efe581599f0 2023-10-16T12:11:06.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.568+0000 7efe5fed3700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe58150530 con 0x7efe58071250 2023-10-16T12:11:06.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.568+0000 7efe5fed3700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe58150250 con 0x7efe5815b040 2023-10-16T12:11:06.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.568+0000 7f099a6bb700 1 -- 172.21.15.16:0/2168211327 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f099404ebb0 con 0x7f09940ae8e0 2023-10-16T12:11:06.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.569+0000 7f4a3ed99700 1 Processor -- start 2023-10-16T12:11:06.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.569+0000 7f4a3ed99700 1 -- start start 2023-10-16T12:11:06.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.569+0000 7f224bfff700 1 -- 172.21.15.16:0/1604361620 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f225c006e40 con 0x7f22640ae570 2023-10-16T12:11:06.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.569+0000 7fd498ff9700 1 -- 172.21.15.16:0/1553026101 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd490054b50 con 0x7fd49c161c90 2023-10-16T12:11:06.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.569+0000 7f044a7fc700 1 -- 172.21.15.16:0/1393781044 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0444001bc0 con 0x7f0434004200 2023-10-16T12:11:06.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.569+0000 7f044a7fc700 1 -- 172.21.15.16:0/1393781044 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0444001f50 con 0x7f0434004200 2023-10-16T12:11:06.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.569+0000 7efe5e470700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efe5815b040 0x7efe581f2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.570+0000 7efe5e470700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efe5815b040 0x7efe581f2e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:50228/0 (socket says 172.21.15.16:50228) 2023-10-16T12:11:06.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.570+0000 7efe5e470700 1 -- 172.21.15.16:0/121237887 learned_addr learned my addr 172.21.15.16:0/121237887 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:06.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.569+0000 7f0990ff9700 1 -- 172.21.15.16:0/2168211327 <== mon.1 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f0984003a20 con 0x7f09940ae8e0 2023-10-16T12:11:06.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.570+0000 7f0990ff9700 1 --2- 172.21.15.16:0/2168211327 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f098006af00 0x7f098006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.570+0000 7f0990ff9700 1 -- 172.21.15.16:0/2168211327 <== mon.1 v2:172.21.15.49:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f0984088d70 con 0x7f09940ae8e0 2023-10-16T12:11:06.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.570+0000 7f09937fe700 1 --2- 172.21.15.16:0/2168211327 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f098006af00 0x7f098006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.570+0000 7f4a3ed99700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4a380ce9f0 0x7f4a380b4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.570+0000 7f4a3ed99700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a380d2f70 0x7f4a381607d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.570+0000 7f4a3ed99700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4a3815d900 0x7f4a3814b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.570+0000 7f4a3ed99700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a38160d10 con 0x7f4a380d2f70 2023-10-16T12:11:06.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.570+0000 7f4a3ed99700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a3815f730 con 0x7f4a380ce9f0 2023-10-16T12:11:06.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.570+0000 7f4a3ed99700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a3815fa10 con 0x7f4a3815d900 2023-10-16T12:11:06.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.570+0000 7f04520ee700 1 -- 172.21.15.16:0/1393781044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0434004200 msgr2=0x7f043400ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.570+0000 7f04520ee700 1 --2- 172.21.15.16:0/1393781044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0434004200 0x7f043400ba90 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f04440054e0 tx=0x7f0434005880 comp rx=0 tx=0).stop 2023-10-16T12:11:06.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.570+0000 7f09937fe700 1 --2- 172.21.15.16:0/2168211327 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f098006af00 0x7f098006d3c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f09941faa60 tx=0x7f098c010000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:06.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.570+0000 7efe5e470700 1 -- 172.21.15.16:0/121237887 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efe58071250 msgr2=0x7efe581527c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.570+0000 7efe5e470700 1 --2- 172.21.15.16:0/121237887 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efe58071250 0x7efe581527c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.570+0000 7efe5e470700 1 -- 172.21.15.16:0/121237887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe581599f0 msgr2=0x7efe581554a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.570+0000 7efe5e470700 1 --2- 172.21.15.16:0/121237887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe581599f0 0x7efe581554a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.571+0000 7efe5e470700 1 -- 172.21.15.16:0/121237887 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe54009de0 con 0x7efe5815b040 2023-10-16T12:11:06.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.571+0000 7f04520ee700 1 -- 172.21.15.16:0/1393781044 shutdown_connections 2023-10-16T12:11:06.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.571+0000 7f04520ee700 1 --2- 172.21.15.16:0/1393781044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0434004200 0x7f043400ba90 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.571+0000 7f04520ee700 1 -- 172.21.15.16:0/1393781044 >> 172.21.15.16:0/1393781044 conn(0x7f044c06a7e0 msgr2=0x7f044c06abf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:06.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.571+0000 7efe5e470700 1 --2- 172.21.15.16:0/121237887 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efe5815b040 0x7efe581f2e10 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7efe50009f10 tx=0x7efe5000e3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:06.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.571+0000 7efe4e7fc700 1 -- 172.21.15.16:0/121237887 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe50010040 con 0x7efe5815b040 2023-10-16T12:11:06.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.571+0000 7efe4e7fc700 1 -- 172.21.15.16:0/121237887 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe5000ee80 con 0x7efe5815b040 2023-10-16T12:11:06.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.571+0000 7efe4e7fc700 1 -- 172.21.15.16:0/121237887 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe50014660 con 0x7efe5815b040 2023-10-16T12:11:06.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.571+0000 7efe5fed3700 1 -- 172.21.15.16:0/121237887 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe58155de0 con 0x7efe5815b040 2023-10-16T12:11:06.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.571+0000 7efe5fed3700 1 -- 172.21.15.16:0/121237887 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe581563a0 con 0x7efe5815b040 2023-10-16T12:11:06.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.572+0000 7f4a3cb35700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4a380ce9f0 0x7f4a380b4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.572+0000 7f4a3cb35700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4a380ce9f0 0x7f4a380b4580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:32812/0 (socket says 172.21.15.16:32812) 2023-10-16T12:11:06.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.572+0000 7f4a3cb35700 1 -- 172.21.15.16:0/201075592 learned_addr learned my addr 172.21.15.16:0/201075592 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:06.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.572+0000 7f4a3cb35700 1 -- 172.21.15.16:0/201075592 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4a3815d900 msgr2=0x7f4a3814b920 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-16T12:11:06.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.572+0000 7f4a3cb35700 1 --2- 172.21.15.16:0/201075592 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4a3815d900 0x7f4a3814b920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.572+0000 7f4a3cb35700 1 -- 172.21.15.16:0/201075592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a380d2f70 msgr2=0x7f4a381607d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.572+0000 7f4a3cb35700 1 --2- 172.21.15.16:0/201075592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a380d2f70 0x7f4a381607d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.572+0000 7f4a3cb35700 1 -- 172.21.15.16:0/201075592 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a30009a70 con 0x7f4a380ce9f0 2023-10-16T12:11:06.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.572+0000 7f4a3cb35700 1 --2- 172.21.15.16:0/201075592 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4a380ce9f0 0x7f4a380b4580 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c00c480 tx=0x7f4a2c00c840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:06.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.573+0000 7efe4e7fc700 1 -- 172.21.15.16:0/121237887 <== mon.2 v2:172.21.15.49:3301/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7efe500147c0 con 0x7efe5815b040 2023-10-16T12:11:06.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.573+0000 7efe5fed3700 1 -- 172.21.15.16:0/121237887 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe5804ebb0 con 0x7efe5815b040 2023-10-16T12:11:06.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.574+0000 7efe4e7fc700 1 --2- 172.21.15.16:0/121237887 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7efe3c06af00 0x7efe3c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.574+0000 7efe5dc6f700 1 --2- 172.21.15.16:0/121237887 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7efe3c06af00 0x7efe3c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.574+0000 7efe4e7fc700 1 -- 172.21.15.16:0/121237887 <== mon.2 v2:172.21.15.49:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7efe5008a530 con 0x7efe5815b040 2023-10-16T12:11:06.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.574+0000 7f322effd700 1 -- 172.21.15.16:0/1421793764 <== mon.2 v2:172.21.15.49:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f32380549d0 con 0x7f3240165dc0 2023-10-16T12:11:06.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.574+0000 7f4a357fa700 1 -- 172.21.15.16:0/201075592 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a2c005e30 con 0x7f4a380ce9f0 2023-10-16T12:11:06.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.574+0000 7efe5dc6f700 1 --2- 172.21.15.16:0/121237887 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7efe3c06af00 0x7efe3c06d3c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7efe54000c00 tx=0x7efe54010000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:06.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.575+0000 7f0990ff9700 1 -- 172.21.15.16:0/2168211327 <== mon.1 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f098408d070 con 0x7f09940ae8e0 2023-10-16T12:11:06.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.574+0000 7f4a357fa700 1 -- 172.21.15.16:0/201075592 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a2c0070d0 con 0x7f4a380ce9f0 2023-10-16T12:11:06.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.574+0000 7f4a357fa700 1 -- 172.21.15.16:0/201075592 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a2c007440 con 0x7f4a380ce9f0 2023-10-16T12:11:06.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.574+0000 7f4a3ed99700 1 -- 172.21.15.16:0/201075592 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a3815fcf0 con 0x7f4a380ce9f0 2023-10-16T12:11:06.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.574+0000 7f4a3ed99700 1 -- 172.21.15.16:0/201075592 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a381602b0 con 0x7f4a380ce9f0 2023-10-16T12:11:06.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.575+0000 7f224bfff700 1 -- 172.21.15.16:0/1604361620 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f225c0547e0 con 0x7f22640ae570 2023-10-16T12:11:06.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.576+0000 7f4a3ed99700 1 -- 172.21.15.16:0/201075592 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a3804ebb0 con 0x7f4a380ce9f0 2023-10-16T12:11:06.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.576+0000 7f4a357fa700 1 -- 172.21.15.16:0/201075592 <== mon.1 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f4a2c0075a0 con 0x7f4a380ce9f0 2023-10-16T12:11:06.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.577+0000 7f4a357fa700 1 --2- 172.21.15.16:0/201075592 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f4a1c06af00 0x7f4a1c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.577+0000 7f4a357fa700 1 -- 172.21.15.16:0/201075592 <== mon.1 v2:172.21.15.49:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f4a2c088cd0 con 0x7f4a380ce9f0 2023-10-16T12:11:06.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.577+0000 7f4a37fff700 1 --2- 172.21.15.16:0/201075592 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f4a1c06af00 0x7f4a1c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.578+0000 7f4a37fff700 1 --2- 172.21.15.16:0/201075592 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f4a1c06af00 0x7f4a1c06d3c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4a38071ee0 tx=0x7f4a24006d20 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:06.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.579+0000 7f04520ee700 1 -- 172.21.15.16:0/1393781044 shutdown_connections 2023-10-16T12:11:06.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.579+0000 7f04520ee700 1 -- 172.21.15.16:0/1393781044 wait complete. 2023-10-16T12:11:06.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.579+0000 7efe4e7fc700 1 -- 172.21.15.16:0/121237887 <== mon.2 v2:172.21.15.49:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7efe50055380 con 0x7efe5815b040 2023-10-16T12:11:06.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.581+0000 7f04520ee700 1 Processor -- start 2023-10-16T12:11:06.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.582+0000 7f4a357fa700 1 -- 172.21.15.16:0/201075592 <== mon.1 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4a2c053b40 con 0x7f4a380ce9f0 2023-10-16T12:11:06.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.581+0000 7f04520ee700 1 -- start start 2023-10-16T12:11:06.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.582+0000 7f04520ee700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0434004200 0x7f044c14abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.582+0000 7f04520ee700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f044c0cd5d0 0x7f044c159ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.582+0000 7f04520ee700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f044c15a410 0x7f044c0a85b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.582+0000 7f04520ee700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f044c0cd280 con 0x7f044c15a410 2023-10-16T12:11:06.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.582+0000 7f04520ee700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f044c15a8a0 con 0x7f044c0cd5d0 2023-10-16T12:11:06.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.582+0000 7f04520ee700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f044c0a9230 con 0x7f0434004200 2023-10-16T12:11:06.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.583+0000 7f044bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f044c15a410 0x7f044c0a85b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.583+0000 7f044bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f044c15a410 0x7f044c0a85b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33760/0 (socket says 172.21.15.16:33760) 2023-10-16T12:11:06.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.583+0000 7f044bfff700 1 -- 172.21.15.16:0/2716344695 learned_addr learned my addr 172.21.15.16:0/2716344695 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:06.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.583+0000 7f044bfff700 1 -- 172.21.15.16:0/2716344695 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0434004200 msgr2=0x7f044c14abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.583+0000 7f044bfff700 1 --2- 172.21.15.16:0/2716344695 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0434004200 0x7f044c14abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.583+0000 7f044bfff700 1 -- 172.21.15.16:0/2716344695 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f044c0cd5d0 msgr2=0x7f044c159ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:06.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.583+0000 7f044bfff700 1 --2- 172.21.15.16:0/2716344695 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f044c0cd5d0 0x7f044c159ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:06.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.584+0000 7f044bfff700 1 -- 172.21.15.16:0/2716344695 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f044c14b500 con 0x7f044c15a410 2023-10-16T12:11:06.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.584+0000 7f044bfff700 1 --2- 172.21.15.16:0/2716344695 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f044c15a410 0x7f044c0a85b0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f044c072d10 tx=0x7f0444004d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:06.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.584+0000 7f0433fff700 1 -- 172.21.15.16:0/2716344695 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0444008680 con 0x7f044c15a410 2023-10-16T12:11:06.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.585+0000 7f0433fff700 1 -- 172.21.15.16:0/2716344695 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04440056f0 con 0x7f044c15a410 2023-10-16T12:11:06.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.585+0000 7f0433fff700 1 -- 172.21.15.16:0/2716344695 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0444005a60 con 0x7f044c15a410 2023-10-16T12:11:06.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.586+0000 7f04520ee700 1 -- 172.21.15.16:0/2716344695 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f044c14b750 con 0x7f044c15a410 2023-10-16T12:11:06.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.586+0000 7f04520ee700 1 -- 172.21.15.16:0/2716344695 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f044c14bd10 con 0x7f044c15a410 2023-10-16T12:11:06.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.589+0000 7f0433fff700 1 -- 172.21.15.16:0/2716344695 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f0444005bc0 con 0x7f044c15a410 2023-10-16T12:11:06.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.590+0000 7f0433fff700 1 --2- 172.21.15.16:0/2716344695 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f043806afd0 0x7f043806d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:06.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.590+0000 7f0433fff700 1 -- 172.21.15.16:0/2716344695 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f044408b320 con 0x7f044c15a410 2023-10-16T12:11:06.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.591+0000 7f04520ee700 1 -- 172.21.15.16:0/2716344695 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f044c04ebb0 con 0x7f044c15a410 2023-10-16T12:11:06.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.591+0000 7f044b7fe700 1 --2- 172.21.15.16:0/2716344695 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f043806afd0 0x7f043806d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:06.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.591+0000 7f044b7fe700 1 --2- 172.21.15.16:0/2716344695 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f043806afd0 0x7f043806d490 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f044c04fec0 tx=0x7f043c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:06.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.601+0000 7f0433fff700 1 -- 172.21.15.16:0/2716344695 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0444056270 con 0x7f044c15a410 2023-10-16T12:11:06.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.991+0000 7f2e8f8b2700 1 -- 172.21.15.16:0/4027063823 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f2e88163870 con 0x7f2e88166200 2023-10-16T12:11:06.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.991+0000 7fd4a22b3700 1 -- 172.21.15.16:0/1553026101 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fd49c158fd0 con 0x7fd49c161c90 2023-10-16T12:11:06.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.992+0000 7fd498ff9700 1 -- 172.21.15.16:0/1553026101 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd49c158fd0 con 0x7fd49c161c90 2023-10-16T12:11:06.995 INFO:teuthology.orchestra.run.smithi016.stdout:55834574855 2023-10-16T12:11:06.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.993+0000 7f2e7dffb700 1 -- 172.21.15.16:0/4027063823 <== mon.1 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2e80054200 con 0x7f2e88166200 2023-10-16T12:11:06.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.995+0000 7f226a208700 1 -- 172.21.15.16:0/1604361620 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f22640ac6a0 con 0x7f22640ae570 2023-10-16T12:11:06.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.995+0000 7f099a6bb700 1 -- 172.21.15.16:0/2168211327 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f09940c2ba0 con 0x7f09940ae8e0 2023-10-16T12:11:06.999 INFO:teuthology.orchestra.run.smithi016.stdout:55834574856 2023-10-16T12:11:06.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.996+0000 7f224bfff700 1 -- 172.21.15.16:0/1604361620 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f225c019020 con 0x7f22640ae570 2023-10-16T12:11:07.000 INFO:teuthology.orchestra.run.smithi016.stdout:55834574856 2023-10-16T12:11:07.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.996+0000 7f0990ff9700 1 -- 172.21.15.16:0/2168211327 <== mon.1 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0984017020 con 0x7f09940ae8e0 2023-10-16T12:11:07.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.996+0000 7f2e8f8b2700 1 -- 172.21.15.16:0/4027063823 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f2e6c06af50 msgr2=0x7f2e6c06d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:07.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.996+0000 7f2e8f8b2700 1 --2- 172.21.15.16:0/4027063823 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f2e6c06af50 0x7f2e6c06d410 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f2e84004720 tx=0x7f2e84008030 comp rx=0 tx=0).stop 2023-10-16T12:11:07.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.996+0000 7f2e8f8b2700 1 -- 172.21.15.16:0/4027063823 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2e88166200 msgr2=0x7f2e8815a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:07.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.996+0000 7f2e8f8b2700 1 --2- 172.21.15.16:0/4027063823 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2e88166200 0x7f2e8815a280 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2e8000be20 tx=0x7f2e80009ef0 comp rx=0 tx=0).stop 2023-10-16T12:11:07.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.997+0000 7f2e8f8b2700 1 -- 172.21.15.16:0/4027063823 shutdown_connections 2023-10-16T12:11:07.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.997+0000 7f2e8f8b2700 1 --2- 172.21.15.16:0/4027063823 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f2e6c06af50 0x7f2e6c06d410 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:07.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.997+0000 7f2e8f8b2700 1 --2- 172.21.15.16:0/4027063823 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2e88166200 0x7f2e8815a280 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:07.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.997+0000 7f2e8f8b2700 1 --2- 172.21.15.16:0/4027063823 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e8806a7e0 0x7f2e88162c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:07.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.997+0000 7f2e8f8b2700 1 --2- 172.21.15.16:0/4027063823 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2e88161c80 0x7f2e881f7390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:07.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.997+0000 7f2e8f8b2700 1 -- 172.21.15.16:0/4027063823 >> 172.21.15.16:0/4027063823 conn(0x7f2e880b1fd0 msgr2=0x7f2e880687c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:07.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.997+0000 7f2e8f8b2700 1 -- 172.21.15.16:0/4027063823 shutdown_connections 2023-10-16T12:11:07.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:06.997+0000 7f2e8f8b2700 1 -- 172.21.15.16:0/4027063823 wait complete. 2023-10-16T12:11:07.004 INFO:teuthology.orchestra.run.smithi016.stdout:55834574857 2023-10-16T12:11:07.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.001+0000 7f4a3ed99700 1 -- 172.21.15.16:0/201075592 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f4a38154fa0 con 0x7f4a380ce9f0 2023-10-16T12:11:07.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.001+0000 7f4a357fa700 1 -- 172.21.15.16:0/201075592 <== mon.1 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4a2c017090 con 0x7f4a380ce9f0 2023-10-16T12:11:07.005 INFO:teuthology.orchestra.run.smithi016.stdout:55834574857 2023-10-16T12:11:07.006 INFO:teuthology.orchestra.run.smithi016.stdout:55834574858 2023-10-16T12:11:07.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.002+0000 7efe5fed3700 1 -- 172.21.15.16:0/121237887 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7efe580b27a0 con 0x7efe5815b040 2023-10-16T12:11:07.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.002+0000 7efe4e7fc700 1 -- 172.21.15.16:0/121237887 <== mon.2 v2:172.21.15.49:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7efe500589a0 con 0x7efe5815b040 2023-10-16T12:11:07.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.003+0000 7f3244e6e700 1 -- 172.21.15.16:0/1421793764 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f32401567e0 con 0x7f3240165dc0 2023-10-16T12:11:07.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.004+0000 7f322effd700 1 -- 172.21.15.16:0/1421793764 <== mon.2 v2:172.21.15.49:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3238057ff0 con 0x7f3240165dc0 2023-10-16T12:11:07.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.004+0000 7f4a3ed99700 1 -- 172.21.15.16:0/201075592 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f4a1c06af00 msgr2=0x7f4a1c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:07.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.004+0000 7f4a3ed99700 1 --2- 172.21.15.16:0/201075592 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f4a1c06af00 0x7f4a1c06d3c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4a38071ee0 tx=0x7f4a24006d20 comp rx=0 tx=0).stop 2023-10-16T12:11:07.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.004+0000 7f4a3ed99700 1 -- 172.21.15.16:0/201075592 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4a380ce9f0 msgr2=0x7f4a380b4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:07.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.004+0000 7f4a3ed99700 1 --2- 172.21.15.16:0/201075592 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4a380ce9f0 0x7f4a380b4580 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c00c480 tx=0x7f4a2c00c840 comp rx=0 tx=0).stop 2023-10-16T12:11:07.009 INFO:teuthology.orchestra.run.smithi016.stdout:55834574856 2023-10-16T12:11:07.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.005+0000 7f226a208700 1 -- 172.21.15.16:0/1604361620 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f225006af50 msgr2=0x7f225006d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:07.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.005+0000 7f226a208700 1 --2- 172.21.15.16:0/1604361620 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f225006af50 0x7f225006d410 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f22641fa6d0 tx=0x7f2254009040 comp rx=0 tx=0).stop 2023-10-16T12:11:07.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.005+0000 7f226a208700 1 -- 172.21.15.16:0/1604361620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22640ae570 msgr2=0x7f22641533c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:07.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.005+0000 7f226a208700 1 --2- 172.21.15.16:0/1604361620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22640ae570 0x7f22641533c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f225c00ba50 tx=0x7f225c00bd60 comp rx=0 tx=0).stop 2023-10-16T12:11:07.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.005+0000 7f099a6bb700 1 -- 172.21.15.16:0/2168211327 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f098006af00 msgr2=0x7f098006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:07.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.005+0000 7f099a6bb700 1 --2- 172.21.15.16:0/2168211327 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f098006af00 0x7f098006d3c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f09941faa60 tx=0x7f098c010000 comp rx=0 tx=0).stop 2023-10-16T12:11:07.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.005+0000 7f099a6bb700 1 -- 172.21.15.16:0/2168211327 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f09940ae8e0 msgr2=0x7f099415e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:07.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.005+0000 7f099a6bb700 1 --2- 172.21.15.16:0/2168211327 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f09940ae8e0 0x7f099415e460 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f098400c450 tx=0x7f098400c810 comp rx=0 tx=0).stop 2023-10-16T12:11:07.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.005+0000 7f226a208700 1 -- 172.21.15.16:0/1604361620 shutdown_connections 2023-10-16T12:11:07.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.005+0000 7f226a208700 1 --2- 172.21.15.16:0/1604361620 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f225006af50 0x7f225006d410 secure :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f22641fa6d0 tx=0x7f2254009040 comp rx=0 tx=0).stop 2023-10-16T12:11:07.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.005+0000 7f226a208700 1 --2- 172.21.15.16:0/1604361620 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2264161440 0x7f22641562f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:07.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.005+0000 7f226a208700 1 --2- 172.21.15.16:0/1604361620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22640ae570 0x7f22641533c0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:07.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.005+0000 7f226a208700 1 --2- 172.21.15.16:0/1604361620 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f22640ae950 0x7f22640ce380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:07.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.005+0000 7f226a208700 1 -- 172.21.15.16:0/1604361620 >> 172.21.15.16:0/1604361620 conn(0x7f226406a7e0 msgr2=0x7f226415c800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:07.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.005+0000 7f099a6bb700 1 -- 172.21.15.16:0/2168211327 shutdown_connections 2023-10-16T12:11:07.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.005+0000 7f099a6bb700 1 --2- 172.21.15.16:0/2168211327 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f098006af00 0x7f098006d3c0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:07.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.005+0000 7f099a6bb700 1 --2- 172.21.15.16:0/2168211327 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f09941652b0 0x7f09940c4fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:07.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.005+0000 7f099a6bb700 1 --2- 172.21.15.16:0/2168211327 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f09940ae8e0 0x7f099415e460 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:07.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.005+0000 7f099a6bb700 1 --2- 172.21.15.16:0/2168211327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09940cab20 0x7f09941f6f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:07.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.005+0000 7f099a6bb700 1 -- 172.21.15.16:0/2168211327 >> 172.21.15.16:0/2168211327 conn(0x7f09940cc5b0 msgr2=0x7f09941644e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:07.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.002+0000 7fd4a22b3700 1 -- 172.21.15.16:0/1553026101 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fd48806afd0 msgr2=0x7fd48806d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:07.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.002+0000 7fd4a22b3700 1 --2- 172.21.15.16:0/1553026101 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fd48806afd0 0x7fd48806d490 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd49400b370 tx=0x7fd494010000 comp rx=0 tx=0).stop 2023-10-16T12:11:07.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.002+0000 7fd4a22b3700 1 -- 172.21.15.16:0/1553026101 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd49c161c90 msgr2=0x7fd49c0cdb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:07.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.002+0000 7fd4a22b3700 1 --2- 172.21.15.16:0/1553026101 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd49c161c90 0x7fd49c0cdb00 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd49000b3f0 tx=0x7fd49000b700 comp rx=0 tx=0).stop 2023-10-16T12:11:07.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.006+0000 7f4a3ed99700 1 -- 172.21.15.16:0/201075592 shutdown_connections 2023-10-16T12:11:07.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.006+0000 7f4a3ed99700 1 --2- 172.21.15.16:0/201075592 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f4a1c06af00 0x7f4a1c06d3c0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:07.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.006+0000 7f4a3ed99700 1 --2- 172.21.15.16:0/201075592 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4a3815d900 0x7f4a3814b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:07.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.006+0000 7f4a3ed99700 1 --2- 172.21.15.16:0/201075592 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4a380ce9f0 0x7f4a380b4580 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:07.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.003+0000 7fd4a22b3700 1 -- 172.21.15.16:0/1553026101 shutdown_connections 2023-10-16T12:11:07.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.003+0000 7fd4a22b3700 1 --2- 172.21.15.16:0/1553026101 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fd48806afd0 0x7fd48806d490 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:07.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.003+0000 7fd4a22b3700 1 --2- 172.21.15.16:0/1553026101 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd49c161c90 0x7fd49c0cdb00 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:07.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.006+0000 7f4a3ed99700 1 --2- 172.21.15.16:0/201075592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a380d2f70 0x7f4a381607d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:07.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.006+0000 7f4a3ed99700 1 -- 172.21.15.16:0/201075592 >> 172.21.15.16:0/201075592 conn(0x7f4a38072330 msgr2=0x7f4a38072710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:07.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.003+0000 7fd4a22b3700 1 --2- 172.21.15.16:0/1553026101 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd49c0cd180 0x7fd49c152850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:07.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.003+0000 7fd4a22b3700 1 --2- 172.21.15.16:0/1553026101 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd49c0ceff0 0x7fd49c1639a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:07.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.003+0000 7fd4a22b3700 1 -- 172.21.15.16:0/1553026101 >> 172.21.15.16:0/1553026101 conn(0x7fd49c0b4cd0 msgr2=0x7fd49c06a7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:07.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.004+0000 7fd4a22b3700 1 -- 172.21.15.16:0/1553026101 shutdown_connections 2023-10-16T12:11:07.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.004+0000 7fd4a22b3700 1 -- 172.21.15.16:0/1553026101 wait complete. 2023-10-16T12:11:07.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.006+0000 7f4a3ed99700 1 -- 172.21.15.16:0/201075592 shutdown_connections 2023-10-16T12:11:07.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.006+0000 7f4a3ed99700 1 -- 172.21.15.16:0/201075592 wait complete. 2023-10-16T12:11:07.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.006+0000 7efe5fed3700 1 -- 172.21.15.16:0/121237887 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7efe3c06af00 msgr2=0x7efe3c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:07.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.006+0000 7efe5fed3700 1 --2- 172.21.15.16:0/121237887 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7efe3c06af00 0x7efe3c06d3c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7efe54000c00 tx=0x7efe54010000 comp rx=0 tx=0).stop 2023-10-16T12:11:07.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.006+0000 7efe5fed3700 1 -- 172.21.15.16:0/121237887 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efe5815b040 msgr2=0x7efe581f2e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:07.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.006+0000 7efe5fed3700 1 --2- 172.21.15.16:0/121237887 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efe5815b040 0x7efe581f2e10 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7efe50009f10 tx=0x7efe5000e3a0 comp rx=0 tx=0).stop 2023-10-16T12:11:07.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.006+0000 7efe5fed3700 1 -- 172.21.15.16:0/121237887 shutdown_connections 2023-10-16T12:11:07.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.006+0000 7efe5fed3700 1 --2- 172.21.15.16:0/121237887 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7efe3c06af00 0x7efe3c06d3c0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:07.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.006+0000 7efe5fed3700 1 --2- 172.21.15.16:0/121237887 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efe5815b040 0x7efe581f2e10 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:07.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.006+0000 7efe5fed3700 1 --2- 172.21.15.16:0/121237887 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efe58071250 0x7efe581527c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:07.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.006+0000 7efe5fed3700 1 --2- 172.21.15.16:0/121237887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe581599f0 0x7efe581554a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:07.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.006+0000 7efe5fed3700 1 -- 172.21.15.16:0/121237887 >> 172.21.15.16:0/121237887 conn(0x7efe5806a7e0 msgr2=0x7efe580acba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:07.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.006+0000 7efe5fed3700 1 -- 172.21.15.16:0/121237887 shutdown_connections 2023-10-16T12:11:07.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.006+0000 7efe5fed3700 1 -- 172.21.15.16:0/121237887 wait complete. 2023-10-16T12:11:07.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.007+0000 7f3244e6e700 1 -- 172.21.15.16:0/1421793764 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f322006af00 msgr2=0x7f322006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:07.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.007+0000 7f3244e6e700 1 --2- 172.21.15.16:0/1421793764 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f322006af00 0x7f322006d3c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f32400d3120 tx=0x7f3230010000 comp rx=0 tx=0).stop 2023-10-16T12:11:07.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.007+0000 7f3244e6e700 1 -- 172.21.15.16:0/1421793764 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3240165dc0 msgr2=0x7f32401f8470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:07.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.007+0000 7f3244e6e700 1 --2- 172.21.15.16:0/1421793764 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3240165dc0 0x7f32401f8470 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f323800cb60 tx=0x7f323800cf20 comp rx=0 tx=0).stop 2023-10-16T12:11:07.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.007+0000 7f3244e6e700 1 -- 172.21.15.16:0/1421793764 shutdown_connections 2023-10-16T12:11:07.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.007+0000 7f3244e6e700 1 --2- 172.21.15.16:0/1421793764 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f322006af00 0x7f322006d3c0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:07.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.007+0000 7f3244e6e700 1 --2- 172.21.15.16:0/1421793764 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3240165dc0 0x7f32401f8470 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:07.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.007+0000 7f3244e6e700 1 --2- 172.21.15.16:0/1421793764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32400d3dd0 0x7f3240156a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:07.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.007+0000 7f3244e6e700 1 --2- 172.21.15.16:0/1421793764 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f32400d41b0 0x7f3240155cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:07.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.007+0000 7f3244e6e700 1 -- 172.21.15.16:0/1421793764 >> 172.21.15.16:0/1421793764 conn(0x7f3240072330 msgr2=0x7f32400c9e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:07.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.007+0000 7f3244e6e700 1 -- 172.21.15.16:0/1421793764 shutdown_connections 2023-10-16T12:11:07.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.007+0000 7f3244e6e700 1 -- 172.21.15.16:0/1421793764 wait complete. 2023-10-16T12:11:07.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.009+0000 7f099a6bb700 1 -- 172.21.15.16:0/2168211327 shutdown_connections 2023-10-16T12:11:07.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.009+0000 7f099a6bb700 1 -- 172.21.15.16:0/2168211327 wait complete. 2023-10-16T12:11:07.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.018+0000 7f226a208700 1 -- 172.21.15.16:0/1604361620 shutdown_connections 2023-10-16T12:11:07.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.018+0000 7f226a208700 1 -- 172.21.15.16:0/1604361620 wait complete. 2023-10-16T12:11:07.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.023+0000 7f04520ee700 1 -- 172.21.15.16:0/2716344695 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f044c15bcd0 con 0x7f044c15a410 2023-10-16T12:11:07.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.024+0000 7f0433fff700 1 -- 172.21.15.16:0/2716344695 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0444059890 con 0x7f044c15a410 2023-10-16T12:11:07.030 INFO:teuthology.orchestra.run.smithi016.stdout:55834574856 2023-10-16T12:11:07.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.028+0000 7f04520ee700 1 -- 172.21.15.16:0/2716344695 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f043806afd0 msgr2=0x7f043806d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:07.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.028+0000 7f04520ee700 1 --2- 172.21.15.16:0/2716344695 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f043806afd0 0x7f043806d490 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f044c04fec0 tx=0x7f043c008040 comp rx=0 tx=0).stop 2023-10-16T12:11:07.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.028+0000 7f04520ee700 1 -- 172.21.15.16:0/2716344695 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f044c15a410 msgr2=0x7f044c0a85b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:07.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.028+0000 7f04520ee700 1 --2- 172.21.15.16:0/2716344695 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f044c15a410 0x7f044c0a85b0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f044c072d10 tx=0x7f0444004d40 comp rx=0 tx=0).stop 2023-10-16T12:11:07.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.029+0000 7f04520ee700 1 -- 172.21.15.16:0/2716344695 shutdown_connections 2023-10-16T12:11:07.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.029+0000 7f04520ee700 1 --2- 172.21.15.16:0/2716344695 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f043806afd0 0x7f043806d490 secure :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f044c04fec0 tx=0x7f043c008040 comp rx=0 tx=0).stop 2023-10-16T12:11:07.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.029+0000 7f04520ee700 1 --2- 172.21.15.16:0/2716344695 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f044c15a410 0x7f044c0a85b0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:07.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.029+0000 7f04520ee700 1 --2- 172.21.15.16:0/2716344695 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0434004200 0x7f044c14abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:07.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.029+0000 7f04520ee700 1 --2- 172.21.15.16:0/2716344695 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f044c0cd5d0 0x7f044c159ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:07.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.029+0000 7f04520ee700 1 -- 172.21.15.16:0/2716344695 >> 172.21.15.16:0/2716344695 conn(0x7f044c06a7e0 msgr2=0x7f044c0b1340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:07.034 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574858 for osd.7 2023-10-16T12:11:07.035 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.1 2023-10-16T12:11:07.036 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574857 for osd.2 2023-10-16T12:11:07.036 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574857 for osd.0 2023-10-16T12:11:07.037 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.5 2023-10-16T12:11:07.038 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.4 2023-10-16T12:11:07.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.035+0000 7f04520ee700 1 -- 172.21.15.16:0/2716344695 shutdown_connections 2023-10-16T12:11:07.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:07.035+0000 7f04520ee700 1 -- 172.21.15.16:0/2716344695 wait complete. 2023-10-16T12:11:07.042 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.3 2023-10-16T12:11:07.049 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.6 2023-10-16T12:11:08.035 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-10-16T12:11:08.036 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-16T12:11:08.037 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-16T12:11:08.038 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-16T12:11:08.038 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-10-16T12:11:08.039 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-10-16T12:11:08.043 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-10-16T12:11:08.050 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-10-16T12:11:08.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.228+0000 7f6cdbcec700 1 Processor -- start 2023-10-16T12:11:08.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.228+0000 7f6cdbcec700 1 -- start start 2023-10-16T12:11:08.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.229+0000 7f6cdbcec700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f6cd4071470 0x7f6cd4163480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:08.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.229+0000 7f6cdbcec700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6cd40b2a50 0x7f6cd40b2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:08.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.229+0000 7f6cdbcec700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cd4070860 con 0x7f6cd415c330 2023-10-16T12:11:08.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.229+0000 7f6cdbcec700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cd40b3420 con 0x7f6cd4071470 2023-10-16T12:11:08.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.229+0000 7f6cdbcec700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cd4070b00 con 0x7f6cd40b2a50 2023-10-16T12:11:08.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.231+0000 7f6cd9287700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f6cd4071470 0x7f6cd4163480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:08.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.231+0000 7f6cda289700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6cd40b2a50 0x7f6cd40b2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:08.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.231+0000 7f6cd9a88700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7f6cd415c330 0x7f6cd4162d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:58456/0 (socket says 172.21.15.16:58456) 2023-10-16T12:11:08.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.231+0000 7f6cd9a88700 1 -- 172.21.15.16:0/1175551727 learned_addr learned my addr 172.21.15.16:0/1175551727 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:08.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.233+0000 7f6cda289700 1 -- 172.21.15.16:0/1175551727 >> v2:172.21.15.49:3300/0 conn(0x7f6cd4071470 msgr2=0x7f6cd4163480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:08.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.233+0000 7f6cda289700 1 --2- 172.21.15.16:0/1175551727 >> v2:172.21.15.49:3300/0 conn(0x7f6cd4071470 0x7f6cd4163480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:08.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.233+0000 7f6cda289700 1 -- 172.21.15.16:0/1175551727 >> v1:172.21.15.49:6789/0 conn(0x7f6cd415c330 legacy=0x7f6cd4162d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:08.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.233+0000 7f6cda289700 1 -- 172.21.15.16:0/1175551727 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cd4070de0 con 0x7f6cd40b2a50 2023-10-16T12:11:08.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.234+0000 7f6cda289700 1 --2- 172.21.15.16:0/1175551727 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6cd40b2a50 0x7f6cd40b2ee0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6cd0009870 tx=0x7f6cd0009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f35a437908a237bb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:08.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.234+0000 7f6cd8a86700 1 -- 172.21.15.16:0/1175551727 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd0011830 con 0x7f6cd40b2a50 2023-10-16T12:11:08.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.235+0000 7f6cd8a86700 1 -- 172.21.15.16:0/1175551727 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cd0011990 con 0x7f6cd40b2a50 2023-10-16T12:11:08.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.235+0000 7f6cd8a86700 1 -- 172.21.15.16:0/1175551727 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd0011d00 con 0x7f6cd40b2a50 2023-10-16T12:11:08.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.236+0000 7f6cdbcec700 1 -- 172.21.15.16:0/1175551727 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6cd40b2a50 msgr2=0x7f6cd40b2ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:08.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.236+0000 7f6cdbcec700 1 --2- 172.21.15.16:0/1175551727 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6cd40b2a50 0x7f6cd40b2ee0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6cd0009870 tx=0x7f6cd0009b80 comp rx=0 tx=0).stop 2023-10-16T12:11:08.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.237+0000 7f6cdbcec700 1 -- 172.21.15.16:0/1175551727 shutdown_connections 2023-10-16T12:11:08.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.237+0000 7f6cdbcec700 1 --2- 172.21.15.16:0/1175551727 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6cd40b2a50 0x7f6cd40b2ee0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:08.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.237+0000 7f6cdbcec700 1 --2- 172.21.15.16:0/1175551727 >> v2:172.21.15.49:3300/0 conn(0x7f6cd4071470 0x7f6cd4163480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:08.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.237+0000 7f6cdbcec700 1 -- 172.21.15.16:0/1175551727 >> 172.21.15.16:0/1175551727 conn(0x7f6cd406a7e0 msgr2=0x7f6cd406abf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:08.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.237+0000 7f6cdbcec700 1 -- 172.21.15.16:0/1175551727 shutdown_connections 2023-10-16T12:11:08.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.237+0000 7f6cdbcec700 1 -- 172.21.15.16:0/1175551727 wait complete. 2023-10-16T12:11:08.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.239+0000 7f6cdbcec700 1 Processor -- start 2023-10-16T12:11:08.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.239+0000 7f6cdbcec700 1 -- start start 2023-10-16T12:11:08.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.240+0000 7f6cdbcec700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6cd4071470 0x7f6cd415d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:08.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.240+0000 7f6cdbcec700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cd40b2a50 0x7f6cd415ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:08.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.240+0000 7f6cdbcec700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6cd415c330 0x7f6cd41640e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:08.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.240+0000 7f6cdbcec700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cd415ceb0 con 0x7f6cd40b2a50 2023-10-16T12:11:08.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.240+0000 7f6cdbcec700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cd413acc0 con 0x7f6cd4071470 2023-10-16T12:11:08.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.240+0000 7f6cdbcec700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cd413d6f0 con 0x7f6cd415c330 2023-10-16T12:11:08.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.241+0000 7f6cd9a88700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6cd4071470 0x7f6cd415d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:08.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.241+0000 7f6cd9a88700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6cd4071470 0x7f6cd415d500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:32832/0 (socket says 172.21.15.16:32832) 2023-10-16T12:11:08.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.241+0000 7f6cd9a88700 1 -- 172.21.15.16:0/3610258077 learned_addr learned my addr 172.21.15.16:0/3610258077 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:08.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.241+0000 7f6cd9287700 1 --2- 172.21.15.16:0/3610258077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cd40b2a50 0x7f6cd415ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:08.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.241+0000 7f6cd9a88700 1 -- 172.21.15.16:0/3610258077 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6cd415c330 msgr2=0x7f6cd41640e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:11:08.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.241+0000 7f6cd9a88700 1 --2- 172.21.15.16:0/3610258077 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6cd415c330 0x7f6cd41640e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:08.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.241+0000 7f6cd9a88700 1 -- 172.21.15.16:0/3610258077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cd40b2a50 msgr2=0x7f6cd415ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:08.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.241+0000 7f6cd9a88700 1 --2- 172.21.15.16:0/3610258077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cd40b2a50 0x7f6cd415ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:08.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.241+0000 7f6cd9a88700 1 -- 172.21.15.16:0/3610258077 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cd00094a0 con 0x7f6cd4071470 2023-10-16T12:11:08.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.241+0000 7f6cd9a88700 1 --2- 172.21.15.16:0/3610258077 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6cd4071470 0x7f6cd415d500 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f6cc400c480 tx=0x7f6cc400c790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:08.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.242+0000 7f6cca7fc700 1 -- 172.21.15.16:0/3610258077 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cc4005d70 con 0x7f6cd4071470 2023-10-16T12:11:08.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.242+0000 7f6cca7fc700 1 -- 172.21.15.16:0/3610258077 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cc40070d0 con 0x7f6cd4071470 2023-10-16T12:11:08.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.242+0000 7f6cca7fc700 1 -- 172.21.15.16:0/3610258077 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cc4007440 con 0x7f6cd4071470 2023-10-16T12:11:08.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.242+0000 7f6cdbcec700 1 -- 172.21.15.16:0/3610258077 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cd415fa50 con 0x7f6cd4071470 2023-10-16T12:11:08.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.242+0000 7f6cdbcec700 1 -- 172.21.15.16:0/3610258077 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cd413b1f0 con 0x7f6cd4071470 2023-10-16T12:11:08.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.244+0000 7faa4ea4a700 1 Processor -- start 2023-10-16T12:11:08.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.244+0000 7f6cca7fc700 1 -- 172.21.15.16:0/3610258077 <== mon.1 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f6cc40075a0 con 0x7f6cd4071470 2023-10-16T12:11:08.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.244+0000 7faa4ea4a700 1 -- start start 2023-10-16T12:11:08.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.245+0000 7f6cdbcec700 1 -- 172.21.15.16:0/3610258077 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cbc0052f0 con 0x7f6cd4071470 2023-10-16T12:11:08.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.245+0000 7f6cca7fc700 1 --2- 172.21.15.16:0/3610258077 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f6cb806afd0 0x7f6cb806d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:08.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.246+0000 7f6cca7fc700 1 -- 172.21.15.16:0/3610258077 <== mon.1 v2:172.21.15.49:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f6cc4088ba0 con 0x7f6cd4071470 2023-10-16T12:11:08.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.246+0000 7faa4ea4a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faa481599f0 0x7faa4815ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:08.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.246+0000 7faa4ea4a700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7faa4815b040 0x7faa480716d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:08.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.246+0000 7faa4ea4a700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa4815e300 con 0x7faa48071250 2023-10-16T12:11:08.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.246+0000 7faa4ea4a700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa4815d240 con 0x7faa4815b040 2023-10-16T12:11:08.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.246+0000 7faa4ea4a700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa48071c10 con 0x7faa481599f0 2023-10-16T12:11:08.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.246+0000 7faa47fff700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faa481599f0 0x7faa4815ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:08.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.246+0000 7faa477fe700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7faa48071250 0x7faa4806c800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:58464/0 (socket says 172.21.15.16:58464) 2023-10-16T12:11:08.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.246+0000 7faa477fe700 1 -- 172.21.15.16:0/3001743661 learned_addr learned my addr 172.21.15.16:0/3001743661 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:08.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.246+0000 7faa47fff700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faa481599f0 0x7faa4815ddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:50264/0 (socket says 172.21.15.16:50264) 2023-10-16T12:11:08.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.246+0000 7faa4cfe7700 1 --2- 172.21.15.16:0/3001743661 >> v2:172.21.15.49:3300/0 conn(0x7faa4815b040 0x7faa480716d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:08.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.246+0000 7faa4cfe7700 1 -- 172.21.15.16:0/3001743661 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faa481599f0 msgr2=0x7faa4815ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:08.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.246+0000 7faa4cfe7700 1 --2- 172.21.15.16:0/3001743661 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faa481599f0 0x7faa4815ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:08.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.246+0000 7faa4cfe7700 1 -- 172.21.15.16:0/3001743661 >> v1:172.21.15.49:6789/0 conn(0x7faa48071250 legacy=0x7faa4806c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:08.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.246+0000 7faa4cfe7700 1 -- 172.21.15.16:0/3001743661 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa480b2fd0 con 0x7faa4815b040 2023-10-16T12:11:08.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.247+0000 7faa47fff700 1 --2- 172.21.15.16:0/3001743661 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faa481599f0 0x7faa4815ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:11:08.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.247+0000 7faa4cfe7700 1 --2- 172.21.15.16:0/3001743661 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faa4815b040 0x7faa480716d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7faa40009b10 tx=0x7faa4000b050 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ebf1db6977342d27 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:08.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.248+0000 7f6cd9287700 1 --2- 172.21.15.16:0/3610258077 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f6cb806afd0 0x7f6cb806d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:08.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.248+0000 7fe2b684c700 1 Processor -- start 2023-10-16T12:11:08.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.249+0000 7f6cd9287700 1 --2- 172.21.15.16:0/3610258077 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f6cb806afd0 0x7f6cb806d490 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6cd414be00 tx=0x7f6ccc00b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:08.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.248+0000 7fe2b684c700 1 -- start start 2023-10-16T12:11:08.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.250+0000 7fe2b684c700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fe2b01599f0 0x7fe2b015ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:08.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.250+0000 7fe2b684c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe2b0071250 0x7fe2b0071700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:08.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.250+0000 7fe2b684c700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2b015e300 con 0x7fe2b006c800 2023-10-16T12:11:08.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.250+0000 7fe2b684c700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2b0071c40 con 0x7fe2b0071250 2023-10-16T12:11:08.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.250+0000 7fe2b684c700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2b015d240 con 0x7fe2b01599f0 2023-10-16T12:11:08.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.250+0000 7fe2b4de9700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe2b006c800 0x7fe2b006cc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51290/0 (socket says 172.21.15.16:51290) 2023-10-16T12:11:08.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.250+0000 7fe2b4de9700 1 -- 172.21.15.16:0/4133123246 learned_addr learned my addr 172.21.15.16:0/4133123246 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:08.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.250+0000 7f6cca7fc700 1 -- 172.21.15.16:0/3610258077 <== mon.1 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6cc4053ab0 con 0x7f6cd4071470 2023-10-16T12:11:08.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.250+0000 7faa46ffd700 1 -- 172.21.15.16:0/3001743661 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa4000b8f0 con 0x7faa4815b040 2023-10-16T12:11:08.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.250+0000 7faa46ffd700 1 -- 172.21.15.16:0/3001743661 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa4000ba50 con 0x7faa4815b040 2023-10-16T12:11:08.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.252+0000 7faa46ffd700 1 -- 172.21.15.16:0/3001743661 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa4000be00 con 0x7faa4815b040 2023-10-16T12:11:08.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.251+0000 7fe2af7fe700 1 --2- 172.21.15.16:0/4133123246 >> v2:172.21.15.16:3300/0 conn(0x7fe2b0071250 0x7fe2b0071700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:08.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.251+0000 7fe2af7fe700 1 -- 172.21.15.16:0/4133123246 >> v2:172.21.15.49:3300/0 conn(0x7fe2b01599f0 msgr2=0x7fe2b015ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:08.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.251+0000 7fe2af7fe700 1 --2- 172.21.15.16:0/4133123246 >> v2:172.21.15.49:3300/0 conn(0x7fe2b01599f0 0x7fe2b015ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:08.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.251+0000 7fe2af7fe700 1 -- 172.21.15.16:0/4133123246 >> v1:172.21.15.16:6789/0 conn(0x7fe2b006c800 legacy=0x7fe2b006cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:08.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.251+0000 7fe2af7fe700 1 -- 172.21.15.16:0/4133123246 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2b00b2fd0 con 0x7fe2b0071250 2023-10-16T12:11:08.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.252+0000 7fe2af7fe700 1 --2- 172.21.15.16:0/4133123246 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2b0071250 0x7fe2b0071700 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe2b00cfe50 tx=0x7fe2a4007320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f6179c1337eccf server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:08.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.253+0000 7fe2aeffd700 1 -- 172.21.15.16:0/4133123246 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2a4007ae0 con 0x7fe2b0071250 2023-10-16T12:11:08.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.253+0000 7fe2aeffd700 1 -- 172.21.15.16:0/4133123246 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2a4007c40 con 0x7fe2b0071250 2023-10-16T12:11:08.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.253+0000 7fe2aeffd700 1 -- 172.21.15.16:0/4133123246 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2a40090d0 con 0x7fe2b0071250 2023-10-16T12:11:08.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.254+0000 7fe2b684c700 1 -- 172.21.15.16:0/4133123246 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2b0071250 msgr2=0x7fe2b0071700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:08.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.254+0000 7fe2b684c700 1 --2- 172.21.15.16:0/4133123246 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2b0071250 0x7fe2b0071700 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe2b00cfe50 tx=0x7fe2a4007320 comp rx=0 tx=0).stop 2023-10-16T12:11:08.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.253+0000 7faa4ea4a700 1 -- 172.21.15.16:0/3001743661 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faa4815b040 msgr2=0x7faa480716d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:08.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.253+0000 7faa4ea4a700 1 --2- 172.21.15.16:0/3001743661 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faa4815b040 0x7faa480716d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7faa40009b10 tx=0x7faa4000b050 comp rx=0 tx=0).stop 2023-10-16T12:11:08.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.255+0000 7faa4ea4a700 1 -- 172.21.15.16:0/3001743661 shutdown_connections 2023-10-16T12:11:08.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.255+0000 7faa4ea4a700 1 --2- 172.21.15.16:0/3001743661 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faa4815b040 0x7faa480716d0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:08.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.255+0000 7faa4ea4a700 1 --2- 172.21.15.16:0/3001743661 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faa481599f0 0x7faa4815ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:08.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.255+0000 7faa4ea4a700 1 -- 172.21.15.16:0/3001743661 >> 172.21.15.16:0/3001743661 conn(0x7faa4806a7e0 msgr2=0x7faa4806abf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:08.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.255+0000 7faa4ea4a700 1 -- 172.21.15.16:0/3001743661 shutdown_connections 2023-10-16T12:11:08.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.255+0000 7faa4ea4a700 1 -- 172.21.15.16:0/3001743661 wait complete. 2023-10-16T12:11:08.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.257+0000 7faa4ea4a700 1 Processor -- start 2023-10-16T12:11:08.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.257+0000 7faa4ea4a700 1 -- start start 2023-10-16T12:11:08.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.259+0000 7faa4ea4a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faa48071250 0x7faa48150510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:08.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.259+0000 7faa4ea4a700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faa481599f0 0x7faa480acd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:08.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.259+0000 7faa4ea4a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa4815b040 0x7faa481577e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:08.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.259+0000 7faa4ea4a700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa481548d0 con 0x7faa4815b040 2023-10-16T12:11:08.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.259+0000 7faa4ea4a700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa48150a50 con 0x7faa481599f0 2023-10-16T12:11:08.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.259+0000 7faa4ea4a700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa481562a0 con 0x7faa48071250 2023-10-16T12:11:08.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.259+0000 7faa47fff700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faa48071250 0x7faa48150510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:08.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.259+0000 7faa47fff700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faa48071250 0x7faa48150510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:50278/0 (socket says 172.21.15.16:50278) 2023-10-16T12:11:08.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.259+0000 7faa47fff700 1 -- 172.21.15.16:0/3972818196 learned_addr learned my addr 172.21.15.16:0/3972818196 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:08.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.260+0000 7faa4cfe7700 1 --2- 172.21.15.16:0/3972818196 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa4815b040 0x7faa481577e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:08.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.261+0000 7faa4cfe7700 1 -- 172.21.15.16:0/3972818196 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faa48071250 msgr2=0x7faa48150510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:08.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.261+0000 7faa4cfe7700 1 --2- 172.21.15.16:0/3972818196 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faa48071250 0x7faa48150510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:08.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.261+0000 7faa4cfe7700 1 -- 172.21.15.16:0/3972818196 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faa481599f0 msgr2=0x7faa480acd60 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-16T12:11:08.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.261+0000 7faa4cfe7700 1 --2- 172.21.15.16:0/3972818196 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faa481599f0 0x7faa480acd60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:08.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.261+0000 7faa4cfe7700 1 -- 172.21.15.16:0/3972818196 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa3c0094a0 con 0x7faa4815b040 2023-10-16T12:11:08.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.261+0000 7faa4cfe7700 1 --2- 172.21.15.16:0/3972818196 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa4815b040 0x7faa481577e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7faa40017000 tx=0x7faa40004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:08.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.262+0000 7faa44ff9700 1 -- 172.21.15.16:0/3972818196 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa40004450 con 0x7faa4815b040 2023-10-16T12:11:08.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.262+0000 7faa44ff9700 1 -- 172.21.15.16:0/3972818196 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa4001d030 con 0x7faa4815b040 2023-10-16T12:11:08.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.262+0000 7faa4ea4a700 1 -- 172.21.15.16:0/3972818196 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa40009770 con 0x7faa4815b040 2023-10-16T12:11:08.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.262+0000 7faa4ea4a700 1 -- 172.21.15.16:0/3972818196 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa48158630 con 0x7faa4815b040 2023-10-16T12:11:08.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.263+0000 7faa44ff9700 1 -- 172.21.15.16:0/3972818196 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa400047b0 con 0x7faa4815b040 2023-10-16T12:11:08.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.261+0000 7fe2b684c700 1 -- 172.21.15.16:0/4133123246 shutdown_connections 2023-10-16T12:11:08.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.261+0000 7fe2b684c700 1 --2- 172.21.15.16:0/4133123246 >> v2:172.21.15.49:3300/0 conn(0x7fe2b01599f0 0x7fe2b015ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:08.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.261+0000 7fe2b684c700 1 --2- 172.21.15.16:0/4133123246 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2b0071250 0x7fe2b0071700 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:08.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.261+0000 7fe2b684c700 1 -- 172.21.15.16:0/4133123246 >> 172.21.15.16:0/4133123246 conn(0x7fe2b006a7e0 msgr2=0x7fe2b006abf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:08.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.265+0000 7faa44ff9700 1 -- 172.21.15.16:0/3972818196 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7faa40023470 con 0x7faa4815b040 2023-10-16T12:11:08.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.265+0000 7faa44ff9700 1 --2- 172.21.15.16:0/3972818196 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7faa3406af50 0x7faa3406d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:08.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.265+0000 7faa44ff9700 1 -- 172.21.15.16:0/3972818196 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7faa4008ae90 con 0x7faa4815b040 2023-10-16T12:11:08.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.270+0000 7fe2b684c700 1 -- 172.21.15.16:0/4133123246 shutdown_connections 2023-10-16T12:11:08.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.270+0000 7fe2b684c700 1 -- 172.21.15.16:0/4133123246 wait complete. 2023-10-16T12:11:08.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.271+0000 7fe2b684c700 1 Processor -- start 2023-10-16T12:11:08.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.273+0000 7faa47fff700 1 --2- 172.21.15.16:0/3972818196 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7faa3406af50 0x7faa3406d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:08.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.272+0000 7fe2b684c700 1 -- start start 2023-10-16T12:11:08.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.273+0000 7fe2b684c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe2b006c800 0x7fe2b0152e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:08.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.273+0000 7fe2b684c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2b0071250 0x7fe2b01390b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:08.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.273+0000 7fe2b684c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe2b01599f0 0x7fe2b0152130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:08.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.273+0000 7fe2b684c700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b01f7a30 con 0x7fe2b0071250 2023-10-16T12:11:08.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.273+0000 7fe2b684c700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b0151090 con 0x7fe2b006c800 2023-10-16T12:11:08.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.273+0000 7fe2b684c700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b0151370 con 0x7fe2b01599f0 2023-10-16T12:11:08.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.273+0000 7fe2af7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2b0071250 0x7fe2b01390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:08.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.273+0000 7fe2af7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2b0071250 0x7fe2b01390b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33796/0 (socket says 172.21.15.16:33796) 2023-10-16T12:11:08.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.273+0000 7fe2af7fe700 1 -- 172.21.15.16:0/281667587 learned_addr learned my addr 172.21.15.16:0/281667587 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:08.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.273+0000 7fe2b4de9700 1 --2- 172.21.15.16:0/281667587 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe2b01599f0 0x7fe2b0152130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:08.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.277+0000 7faa4ea4a700 1 -- 172.21.15.16:0/3972818196 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa280052f0 con 0x7faa4815b040 2023-10-16T12:11:08.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.274+0000 7fe2af7fe700 1 -- 172.21.15.16:0/281667587 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe2b01599f0 msgr2=0x7fe2b0152130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:08.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.274+0000 7fe2af7fe700 1 --2- 172.21.15.16:0/281667587 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe2b01599f0 0x7fe2b0152130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:08.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.274+0000 7fe2af7fe700 1 -- 172.21.15.16:0/281667587 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe2b006c800 msgr2=0x7fe2b0152e00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:11:08.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.274+0000 7fe2af7fe700 1 --2- 172.21.15.16:0/281667587 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe2b006c800 0x7fe2b0152e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:08.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.274+0000 7fe2af7fe700 1 -- 172.21.15.16:0/281667587 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2a4009dc0 con 0x7fe2b0071250 2023-10-16T12:11:08.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.274+0000 7fe2af7fe700 1 --2- 172.21.15.16:0/281667587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2b0071250 0x7fe2b01390b0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fe2a4000c00 tx=0x7fe2a4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:08.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.275+0000 7fe2acff9700 1 -- 172.21.15.16:0/281667587 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2a4009490 con 0x7fe2b0071250 2023-10-16T12:11:08.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.275+0000 7fe2b684c700 1 -- 172.21.15.16:0/281667587 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2b0151650 con 0x7fe2b0071250 2023-10-16T12:11:08.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.275+0000 7fe2b684c700 1 -- 172.21.15.16:0/281667587 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2b0151c10 con 0x7fe2b0071250 2023-10-16T12:11:08.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.275+0000 7fe2acff9700 1 -- 172.21.15.16:0/281667587 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2a401f070 con 0x7fe2b0071250 2023-10-16T12:11:08.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.275+0000 7fe2acff9700 1 -- 172.21.15.16:0/281667587 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2a40036a0 con 0x7fe2b0071250 2023-10-16T12:11:08.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.277+0000 7fe2acff9700 1 -- 172.21.15.16:0/281667587 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fe2a4021030 con 0x7fe2b0071250 2023-10-16T12:11:08.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.278+0000 7fe2acff9700 1 --2- 172.21.15.16:0/281667587 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fe29c06af50 0x7fe29c06d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:08.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.278+0000 7fe2acff9700 1 -- 172.21.15.16:0/281667587 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fe2a408aae0 con 0x7fe2b0071250 2023-10-16T12:11:08.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.278+0000 7fe2affff700 1 --2- 172.21.15.16:0/281667587 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fe29c06af50 0x7fe29c06d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:08.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.279+0000 7fe2affff700 1 --2- 172.21.15.16:0/281667587 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fe29c06af50 0x7fe29c06d410 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fe2a0009690 tx=0x7fe2a0006c10 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:08.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.280+0000 7fe2b684c700 1 -- 172.21.15.16:0/281667587 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2900052f0 con 0x7fe2b0071250 2023-10-16T12:11:08.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.279+0000 7faa47fff700 1 --2- 172.21.15.16:0/3972818196 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7faa3406af50 0x7faa3406d410 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7faa3c009c20 tx=0x7faa3c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:08.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.286+0000 7faa44ff9700 1 -- 172.21.15.16:0/3972818196 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7faa40055eb0 con 0x7faa4815b040 2023-10-16T12:11:08.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.288+0000 7fe2acff9700 1 -- 172.21.15.16:0/281667587 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe2a4055a10 con 0x7fe2b0071250 2023-10-16T12:11:08.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.293+0000 7f53ac55b700 1 Processor -- start 2023-10-16T12:11:08.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.293+0000 7f53ac55b700 1 -- start start 2023-10-16T12:11:08.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.294+0000 7f53ac55b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f53a415a230 0x7f53a406e540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:08.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.294+0000 7f53ac55b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f53a415dd90 0x7f53a40cb600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:08.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.294+0000 7f53ac55b700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53a406faa0 con 0x7f53a40cbb40 2023-10-16T12:11:08.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.294+0000 7f53ac55b700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53a40cbfd0 con 0x7f53a415dd90 2023-10-16T12:11:08.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.294+0000 7f53ac55b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53a415a610 con 0x7f53a415a230 2023-10-16T12:11:08.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.295+0000 7f53aa2f7700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f53a415a230 0x7f53a406e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:08.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.295+0000 7f53aa2f7700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f53a415a230 0x7f53a406e540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:50292/0 (socket says 172.21.15.16:50292) 2023-10-16T12:11:08.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.295+0000 7f53aa2f7700 1 -- 172.21.15.16:0/3870378791 learned_addr learned my addr 172.21.15.16:0/3870378791 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:08.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.296+0000 7f53aa2f7700 1 -- 172.21.15.16:0/3870378791 >> v2:172.21.15.16:3300/0 conn(0x7f53a415dd90 msgr2=0x7f53a40cb600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:08.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.296+0000 7f53aa2f7700 1 --2- 172.21.15.16:0/3870378791 >> v2:172.21.15.16:3300/0 conn(0x7f53a415dd90 0x7f53a40cb600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:08.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.296+0000 7f53aa2f7700 1 -- 172.21.15.16:0/3870378791 >> v1:172.21.15.16:6789/0 conn(0x7f53a40cbb40 legacy=0x7f53a40ca4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:08.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.296+0000 7f53aa2f7700 1 -- 172.21.15.16:0/3870378791 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53a406ee90 con 0x7f53a415a230 2023-10-16T12:11:08.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.295+0000 7eff1e24d700 1 Processor -- start 2023-10-16T12:11:08.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.295+0000 7eff1e24d700 1 -- start start 2023-10-16T12:11:08.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.296+0000 7eff1e24d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7eff1808aa20 0x7eff18070860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:08.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.296+0000 7eff1e24d700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff1816aa80 con 0x7eff1808a640 2023-10-16T12:11:08.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.296+0000 7eff1e24d700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff1816e1b0 con 0x7eff1806d1d0 2023-10-16T12:11:08.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.296+0000 7eff1e24d700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff18070eb0 con 0x7eff1808aa20 2023-10-16T12:11:08.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.297+0000 7eff17fff700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7eff1806d1d0 0x7eff180cf1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:58500/0 (socket says 172.21.15.16:58500) 2023-10-16T12:11:08.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.297+0000 7eff17fff700 1 -- 172.21.15.16:0/505420321 learned_addr learned my addr 172.21.15.16:0/505420321 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:08.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.297+0000 7eff16ffd700 1 --2- 172.21.15.16:0/505420321 >> v2:172.21.15.16:3300/0 conn(0x7eff1808aa20 0x7eff18070860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:08.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.297+0000 7eff167fc700 1 -- 172.21.15.16:0/505420321 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3205601764 0 0) 0x7eff1816e1b0 con 0x7eff1806d1d0 2023-10-16T12:11:08.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.297+0000 7eff16ffd700 1 -- 172.21.15.16:0/505420321 >> v1:172.21.15.49:6789/0 conn(0x7eff1806d1d0 legacy=0x7eff180cf1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:08.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.297+0000 7eff16ffd700 1 -- 172.21.15.16:0/505420321 >> v1:172.21.15.16:6789/0 conn(0x7eff1808a640 legacy=0x7eff180cbda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:08.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.297+0000 7eff16ffd700 1 -- 172.21.15.16:0/505420321 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff180b3d50 con 0x7eff1808aa20 2023-10-16T12:11:08.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.298+0000 7eff16ffd700 1 --2- 172.21.15.16:0/505420321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff1808aa20 0x7eff18070860 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7eff0c00a950 tx=0x7eff0c00ac60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95ba754631eb5f02 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:08.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.298+0000 7eff167fc700 1 -- 172.21.15.16:0/505420321 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff0c00ca60 con 0x7eff1808aa20 2023-10-16T12:11:08.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.298+0000 7eff167fc700 1 -- 172.21.15.16:0/505420321 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff0c00cbc0 con 0x7eff1808aa20 2023-10-16T12:11:08.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.298+0000 7eff167fc700 1 -- 172.21.15.16:0/505420321 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff0c012420 con 0x7eff1808aa20 2023-10-16T12:11:08.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.299+0000 7eff1e24d700 1 -- 172.21.15.16:0/505420321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff1808aa20 msgr2=0x7eff18070860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:08.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.299+0000 7eff1e24d700 1 --2- 172.21.15.16:0/505420321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff1808aa20 0x7eff18070860 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7eff0c00a950 tx=0x7eff0c00ac60 comp rx=0 tx=0).stop 2023-10-16T12:11:08.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.299+0000 7eff1e24d700 1 -- 172.21.15.16:0/505420321 shutdown_connections 2023-10-16T12:11:08.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.299+0000 7eff1e24d700 1 --2- 172.21.15.16:0/505420321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff1808aa20 0x7eff18070860 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:08.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.299+0000 7eff1e24d700 1 -- 172.21.15.16:0/505420321 >> 172.21.15.16:0/505420321 conn(0x7eff1806a7e0 msgr2=0x7eff1806abf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:08.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.299+0000 7eff1e24d700 1 -- 172.21.15.16:0/505420321 shutdown_connections 2023-10-16T12:11:08.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.299+0000 7eff1e24d700 1 -- 172.21.15.16:0/505420321 wait complete. 2023-10-16T12:11:08.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.301+0000 7eff1e24d700 1 Processor -- start 2023-10-16T12:11:08.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.301+0000 7eff1e24d700 1 -- start start 2023-10-16T12:11:08.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.303+0000 7eff1e24d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7eff1806d1d0 0x7eff18203750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:08.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.303+0000 7eff1e24d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7eff1808a640 0x7eff180cf120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:08.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.303+0000 7eff1e24d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff1808aa20 0x7eff181662d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:08.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.303+0000 7eff1e24d700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff18161550 con 0x7eff1808aa20 2023-10-16T12:11:08.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.303+0000 7eff1e24d700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff180cf660 con 0x7eff1808a640 2023-10-16T12:11:08.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.303+0000 7eff1e24d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff18200270 con 0x7eff1806d1d0 2023-10-16T12:11:08.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.303+0000 7eff17fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff1808aa20 0x7eff181662d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:08.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.303+0000 7eff177fe700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7eff1806d1d0 0x7eff18203750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:08.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.303+0000 7eff17fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff1808aa20 0x7eff181662d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33816/0 (socket says 172.21.15.16:33816) 2023-10-16T12:11:08.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.303+0000 7eff17fff700 1 -- 172.21.15.16:0/342809733 learned_addr learned my addr 172.21.15.16:0/342809733 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:08.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.303+0000 7eff177fe700 1 -- 172.21.15.16:0/342809733 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7eff1808a640 msgr2=0x7eff180cf120 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-16T12:11:08.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.303+0000 7eff177fe700 1 --2- 172.21.15.16:0/342809733 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7eff1808a640 0x7eff180cf120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:08.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.303+0000 7eff177fe700 1 -- 172.21.15.16:0/342809733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff1808aa20 msgr2=0x7eff181662d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:08.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.303+0000 7eff177fe700 1 --2- 172.21.15.16:0/342809733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff1808aa20 0x7eff181662d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:08.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.303+0000 7eff177fe700 1 -- 172.21.15.16:0/342809733 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff0c00a5b0 con 0x7eff1806d1d0 2023-10-16T12:11:08.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.304+0000 7eff177fe700 1 --2- 172.21.15.16:0/342809733 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7eff1806d1d0 0x7eff18203750 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7eff08000f60 tx=0x7eff0800e9e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:08.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.304+0000 7efefffff700 1 -- 172.21.15.16:0/342809733 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff080052d0 con 0x7eff1806d1d0 2023-10-16T12:11:08.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.304+0000 7efefffff700 1 -- 172.21.15.16:0/342809733 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff08005430 con 0x7eff1806d1d0 2023-10-16T12:11:08.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.304+0000 7efefffff700 1 -- 172.21.15.16:0/342809733 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff080108e0 con 0x7eff1806d1d0 2023-10-16T12:11:08.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.305+0000 7eff1e24d700 1 -- 172.21.15.16:0/342809733 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff18203d30 con 0x7eff1806d1d0 2023-10-16T12:11:08.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.305+0000 7eff1e24d700 1 -- 172.21.15.16:0/342809733 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff182042f0 con 0x7eff1806d1d0 2023-10-16T12:11:08.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.307+0000 7eff1e24d700 1 -- 172.21.15.16:0/342809733 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff180cbb10 con 0x7eff1806d1d0 2023-10-16T12:11:08.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.307+0000 7f53aa2f7700 1 --2- 172.21.15.16:0/3870378791 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f53a415a230 0x7f53a406e540 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f539800bb90 tx=0x7f539800bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d79b6191d5db57da server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:08.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.307+0000 7efefffff700 1 -- 172.21.15.16:0/342809733 <== mon.2 v2:172.21.15.49:3301/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7eff08010a40 con 0x7eff1806d1d0 2023-10-16T12:11:08.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.308+0000 7f53a92f5700 1 -- 172.21.15.16:0/3870378791 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f539800dc00 con 0x7f53a415a230 2023-10-16T12:11:08.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.308+0000 7efefffff700 1 --2- 172.21.15.16:0/342809733 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7eff0406af00 0x7eff0406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:08.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.308+0000 7f53a92f5700 1 -- 172.21.15.16:0/3870378791 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f539800dd60 con 0x7f53a415a230 2023-10-16T12:11:08.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.308+0000 7f53a92f5700 1 -- 172.21.15.16:0/3870378791 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5398004270 con 0x7f53a415a230 2023-10-16T12:11:08.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.308+0000 7efefffff700 1 -- 172.21.15.16:0/342809733 <== mon.2 v2:172.21.15.49:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7eff0808b7b0 con 0x7eff1806d1d0 2023-10-16T12:11:08.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.308+0000 7f53ac55b700 1 -- 172.21.15.16:0/3870378791 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f53a415a230 msgr2=0x7f53a406e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:08.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.308+0000 7f53ac55b700 1 --2- 172.21.15.16:0/3870378791 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f53a415a230 0x7f53a406e540 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f539800bb90 tx=0x7f539800bea0 comp rx=0 tx=0).stop 2023-10-16T12:11:08.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.309+0000 7f53ac55b700 1 -- 172.21.15.16:0/3870378791 shutdown_connections 2023-10-16T12:11:08.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.309+0000 7f53ac55b700 1 --2- 172.21.15.16:0/3870378791 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f53a415a230 0x7f53a406e540 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:08.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.309+0000 7f53ac55b700 1 --2- 172.21.15.16:0/3870378791 >> v2:172.21.15.16:3300/0 conn(0x7f53a415dd90 0x7f53a40cb600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:08.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.309+0000 7f53ac55b700 1 -- 172.21.15.16:0/3870378791 >> 172.21.15.16:0/3870378791 conn(0x7f53a40cc5e0 msgr2=0x7f53a40cc9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:08.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.310+0000 7eff16ffd700 1 --2- 172.21.15.16:0/342809733 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7eff0406af00 0x7eff0406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:08.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.310+0000 7eff16ffd700 1 --2- 172.21.15.16:0/342809733 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7eff0406af00 0x7eff0406d3c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7eff0c00a950 tx=0x7eff0c009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:08.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.312+0000 7efefffff700 1 -- 172.21.15.16:0/342809733 <== mon.2 v2:172.21.15.49:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7eff08056590 con 0x7eff1806d1d0 2023-10-16T12:11:08.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.313+0000 7f53ac55b700 1 -- 172.21.15.16:0/3870378791 shutdown_connections 2023-10-16T12:11:08.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.313+0000 7f53ac55b700 1 -- 172.21.15.16:0/3870378791 wait complete. 2023-10-16T12:11:08.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.315+0000 7f53ac55b700 1 Processor -- start 2023-10-16T12:11:08.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.315+0000 7f53ac55b700 1 -- start start 2023-10-16T12:11:08.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.316+0000 7f53ac55b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f53a40cbb40 0x7f53a41514b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:08.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.316+0000 7f53ac55b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f53a415a230 0x7f53a40c6c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:08.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.316+0000 7f53ac55b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53a415dd90 0x7f53a40c8430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:08.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.316+0000 7f53ac55b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53a40c7cd0 con 0x7f53a415dd90 2023-10-16T12:11:08.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.316+0000 7f53ac55b700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53a40c3ba0 con 0x7f53a415a230 2023-10-16T12:11:08.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.316+0000 7f53ac55b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53a41f8a10 con 0x7f53a40cbb40 2023-10-16T12:11:08.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.317+0000 7f53aaaf8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53a415dd90 0x7f53a40c8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:08.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.317+0000 7f53aaaf8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53a415dd90 0x7f53a40c8430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33826/0 (socket says 172.21.15.16:33826) 2023-10-16T12:11:08.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.317+0000 7f53aaaf8700 1 -- 172.21.15.16:0/2841366340 learned_addr learned my addr 172.21.15.16:0/2841366340 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:08.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.317+0000 7f53aa2f7700 1 --2- 172.21.15.16:0/2841366340 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f53a40cbb40 0x7f53a41514b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:08.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.317+0000 7f53aaaf8700 1 -- 172.21.15.16:0/2841366340 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f53a40cbb40 msgr2=0x7f53a41514b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:08.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.317+0000 7f53aaaf8700 1 --2- 172.21.15.16:0/2841366340 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f53a40cbb40 0x7f53a41514b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:08.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.317+0000 7f53aaaf8700 1 -- 172.21.15.16:0/2841366340 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f53a415a230 msgr2=0x7f53a40c6c30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-16T12:11:08.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.317+0000 7f53aaaf8700 1 --2- 172.21.15.16:0/2841366340 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f53a415a230 0x7f53a40c6c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:08.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.317+0000 7f53aaaf8700 1 -- 172.21.15.16:0/2841366340 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f539800b7f0 con 0x7f53a415dd90 2023-10-16T12:11:08.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.317+0000 7f53aaaf8700 1 --2- 172.21.15.16:0/2841366340 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53a415dd90 0x7f53a40c8430 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f539c00c7d0 tx=0x7f539c00cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:08.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.319+0000 7f5396ffd700 1 -- 172.21.15.16:0/2841366340 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f539c003e10 con 0x7f53a415dd90 2023-10-16T12:11:08.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.319+0000 7f5396ffd700 1 -- 172.21.15.16:0/2841366340 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f539c005070 con 0x7f53a415dd90 2023-10-16T12:11:08.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.319+0000 7f5396ffd700 1 -- 172.21.15.16:0/2841366340 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f539c005420 con 0x7f53a415dd90 2023-10-16T12:11:08.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.319+0000 7f53ac55b700 1 -- 172.21.15.16:0/2841366340 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53a40c7710 con 0x7f53a415dd90 2023-10-16T12:11:08.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.319+0000 7f53ac55b700 1 -- 172.21.15.16:0/2841366340 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53a40c3830 con 0x7f53a415dd90 2023-10-16T12:11:08.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.322+0000 7f5396ffd700 1 -- 172.21.15.16:0/2841366340 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f539c005580 con 0x7f53a415dd90 2023-10-16T12:11:08.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.323+0000 7f5396ffd700 1 --2- 172.21.15.16:0/2841366340 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f539006b0b0 0x7f539006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:08.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.323+0000 7f5396ffd700 1 -- 172.21.15.16:0/2841366340 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f539c088ce0 con 0x7f53a415dd90 2023-10-16T12:11:08.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.323+0000 7f53aa2f7700 1 --2- 172.21.15.16:0/2841366340 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f539006b0b0 0x7f539006d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:08.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.324+0000 7f53aa2f7700 1 --2- 172.21.15.16:0/2841366340 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f539006b0b0 0x7f539006d570 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f5398019040 tx=0x7f5398008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:08.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.330+0000 7f53ac55b700 1 -- 172.21.15.16:0/2841366340 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f538c0052f0 con 0x7f53a415dd90 2023-10-16T12:11:08.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.331+0000 7f070b1bb700 1 Processor -- start 2023-10-16T12:11:08.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.331+0000 7f070b1bb700 1 -- start start 2023-10-16T12:11:08.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.332+0000 7f070b1bb700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f070415d900 0x7f07040d2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:08.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.332+0000 7f070b1bb700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07040cee70 con 0x7f07040ce9f0 2023-10-16T12:11:08.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.332+0000 7f070b1bb700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07040ce400 con 0x7f07040d2f70 2023-10-16T12:11:08.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.332+0000 7f070b1bb700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f070415dce0 con 0x7f070415d900 2023-10-16T12:11:08.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.333+0000 7f0703fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f07040ce9f0 0x7f07040d2860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51338/0 (socket says 172.21.15.16:51338) 2023-10-16T12:11:08.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.333+0000 7f0703fff700 1 -- 172.21.15.16:0/3216889303 learned_addr learned my addr 172.21.15.16:0/3216889303 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:08.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.336+0000 7f0708f57700 1 --2- 172.21.15.16:0/3216889303 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f070415d900 0x7f07040d2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:08.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.339+0000 7f0708f57700 1 -- 172.21.15.16:0/3216889303 >> v1:172.21.15.49:6789/0 conn(0x7f07040d2f70 legacy=0x7f07040b6790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:08.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.339+0000 7f0708f57700 1 -- 172.21.15.16:0/3216889303 >> v1:172.21.15.16:6789/0 conn(0x7f07040ce9f0 legacy=0x7f07040d2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:08.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.339+0000 7f0708f57700 1 -- 172.21.15.16:0/3216889303 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07040d1280 con 0x7f070415d900 2023-10-16T12:11:08.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.340+0000 7f5396ffd700 1 -- 172.21.15.16:0/2841366340 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f539c053cc0 con 0x7f53a415dd90 2023-10-16T12:11:08.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.340+0000 7f0708f57700 1 --2- 172.21.15.16:0/3216889303 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f070415d900 0x7f07040d2040 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f06f0009870 tx=0x7f06f0009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d452b38fb1ce01e8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:08.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.340+0000 7f07037fe700 1 -- 172.21.15.16:0/3216889303 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06f0004230 con 0x7f070415d900 2023-10-16T12:11:08.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.341+0000 7f07037fe700 1 -- 172.21.15.16:0/3216889303 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06f0004390 con 0x7f070415d900 2023-10-16T12:11:08.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.341+0000 7f07037fe700 1 -- 172.21.15.16:0/3216889303 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06f00036a0 con 0x7f070415d900 2023-10-16T12:11:08.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.342+0000 7f070b1bb700 1 -- 172.21.15.16:0/3216889303 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f070415d900 msgr2=0x7f07040d2040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:08.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.342+0000 7f070b1bb700 1 --2- 172.21.15.16:0/3216889303 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f070415d900 0x7f07040d2040 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f06f0009870 tx=0x7f06f0009b80 comp rx=0 tx=0).stop 2023-10-16T12:11:08.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.342+0000 7f070b1bb700 1 -- 172.21.15.16:0/3216889303 shutdown_connections 2023-10-16T12:11:08.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.342+0000 7f070b1bb700 1 --2- 172.21.15.16:0/3216889303 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f070415d900 0x7f07040d2040 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:08.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.342+0000 7f070b1bb700 1 -- 172.21.15.16:0/3216889303 >> 172.21.15.16:0/3216889303 conn(0x7f0704072330 msgr2=0x7f0704072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:08.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.349+0000 7f070b1bb700 1 -- 172.21.15.16:0/3216889303 shutdown_connections 2023-10-16T12:11:08.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.349+0000 7f070b1bb700 1 -- 172.21.15.16:0/3216889303 wait complete. 2023-10-16T12:11:08.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.351+0000 7f070b1bb700 1 Processor -- start 2023-10-16T12:11:08.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.352+0000 7f070b1bb700 1 -- start start 2023-10-16T12:11:08.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.353+0000 7f070b1bb700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f07040d2f70 0x7f07040c7970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:08.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.353+0000 7f070b1bb700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f070415d900 0x7f070415f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:08.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.353+0000 7f070b1bb700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0704160410 0x7f07041f3990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:08.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.353+0000 7f070b1bb700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07040c8050 con 0x7f070415d900 2023-10-16T12:11:08.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.353+0000 7f070b1bb700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07040c8330 con 0x7f07040d2f70 2023-10-16T12:11:08.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.353+0000 7f070b1bb700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07040c8610 con 0x7f0704160410 2023-10-16T12:11:08.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.354+0000 7f0708f57700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f07040d2f70 0x7f07040c7970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:08.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.354+0000 7f0708f57700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f07040d2f70 0x7f07040c7970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:32902/0 (socket says 172.21.15.16:32902) 2023-10-16T12:11:08.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.354+0000 7f0708f57700 1 -- 172.21.15.16:0/2655684751 learned_addr learned my addr 172.21.15.16:0/2655684751 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:08.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.354+0000 7f0709758700 1 --2- 172.21.15.16:0/2655684751 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0704160410 0x7f07041f3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:08.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.355+0000 7f0708f57700 1 -- 172.21.15.16:0/2655684751 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0704160410 msgr2=0x7f07041f3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:08.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.355+0000 7f0708f57700 1 --2- 172.21.15.16:0/2655684751 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0704160410 0x7f07041f3990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:08.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.355+0000 7f0708f57700 1 -- 172.21.15.16:0/2655684751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f070415d900 msgr2=0x7f070415f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:08.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.355+0000 7f0708f57700 1 --2- 172.21.15.16:0/2655684751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f070415d900 0x7f070415f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:08.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.355+0000 7f0708f57700 1 -- 172.21.15.16:0/2655684751 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06f00094a0 con 0x7f07040d2f70 2023-10-16T12:11:08.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.355+0000 7f0708f57700 1 --2- 172.21.15.16:0/2655684751 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f07040d2f70 0x7f07040c7970 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f06f0009f80 tx=0x7f06f0004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:08.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.356+0000 7f07017fa700 1 -- 172.21.15.16:0/2655684751 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06f0003e70 con 0x7f07040d2f70 2023-10-16T12:11:08.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.356+0000 7f07017fa700 1 -- 172.21.15.16:0/2655684751 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06f0010590 con 0x7f07040d2f70 2023-10-16T12:11:08.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.356+0000 7f07017fa700 1 -- 172.21.15.16:0/2655684751 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06f0010940 con 0x7f07040d2f70 2023-10-16T12:11:08.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.357+0000 7f070b1bb700 1 -- 172.21.15.16:0/2655684751 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07040c88f0 con 0x7f07040d2f70 2023-10-16T12:11:08.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.357+0000 7f070b1bb700 1 -- 172.21.15.16:0/2655684751 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f070415f800 con 0x7f07040d2f70 2023-10-16T12:11:08.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.360+0000 7f07017fa700 1 -- 172.21.15.16:0/2655684751 <== mon.1 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f06f0010aa0 con 0x7f07040d2f70 2023-10-16T12:11:08.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.360+0000 7f07017fa700 1 --2- 172.21.15.16:0/2655684751 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f06e806b000 0x7f06e806d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:08.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.360+0000 7f07017fa700 1 -- 172.21.15.16:0/2655684751 <== mon.1 v2:172.21.15.49:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f06f008ac20 con 0x7f07040d2f70 2023-10-16T12:11:08.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.361+0000 7f070b1bb700 1 -- 172.21.15.16:0/2655684751 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07040c8e10 con 0x7f07040d2f70 2023-10-16T12:11:08.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.361+0000 7f0703fff700 1 --2- 172.21.15.16:0/2655684751 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f06e806b000 0x7f06e806d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:08.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.362+0000 7f0703fff700 1 --2- 172.21.15.16:0/2655684751 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f06e806b000 0x7f06e806d4c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f06f8000c00 tx=0x7f06f8007a50 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:08.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.369+0000 7f07017fa700 1 -- 172.21.15.16:0/2655684751 <== mon.1 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f07040c8e10 con 0x7f07040d2f70 2023-10-16T12:11:08.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.372+0000 7facd035f700 1 Processor -- start 2023-10-16T12:11:08.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.372+0000 7facd035f700 1 -- start start 2023-10-16T12:11:08.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.373+0000 7facd035f700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7facc815c330 0x7facc8163480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:08.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.373+0000 7facd035f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7facc8071470 0x7facc8162d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:08.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.374+0000 7facd035f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7facc80b2a50 0x7facc80b2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:08.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.374+0000 7facd035f700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facc80b3420 con 0x7facc8071470 2023-10-16T12:11:08.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.374+0000 7facd035f700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facc8163280 con 0x7facc815c330 2023-10-16T12:11:08.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.374+0000 7facd035f700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facc8070860 con 0x7facc80b2a50 2023-10-16T12:11:08.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.374+0000 7faccd8fa700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7facc8071470 0x7facc8162d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:08.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.374+0000 7faccd8fa700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7facc8071470 0x7facc8162d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33844/0 (socket says 172.21.15.16:33844) 2023-10-16T12:11:08.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.374+0000 7faccd8fa700 1 -- 172.21.15.16:0/1049939813 learned_addr learned my addr 172.21.15.16:0/1049939813 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:08.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.375+0000 7faccd8fa700 1 -- 172.21.15.16:0/1049939813 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7facc80b2a50 msgr2=0x7facc80b2ee0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-16T12:11:08.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.375+0000 7faccd8fa700 1 --2- 172.21.15.16:0/1049939813 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7facc80b2a50 0x7facc80b2ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:08.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.375+0000 7faccd8fa700 1 -- 172.21.15.16:0/1049939813 >> v2:172.21.15.49:3300/0 conn(0x7facc815c330 msgr2=0x7facc8163480 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-16T12:11:08.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.375+0000 7faccd8fa700 1 --2- 172.21.15.16:0/1049939813 >> v2:172.21.15.49:3300/0 conn(0x7facc815c330 0x7facc8163480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:08.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.375+0000 7faccd8fa700 1 -- 172.21.15.16:0/1049939813 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facc8070b00 con 0x7facc8071470 2023-10-16T12:11:08.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.375+0000 7faccd8fa700 1 --2- 172.21.15.16:0/1049939813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facc8071470 0x7facc8162d40 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7facc40098f0 tx=0x7facc4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15b4c8e5490fdac2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:08.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.376+0000 7faccd0f9700 1 -- 172.21.15.16:0/1049939813 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc400b860 con 0x7facc8071470 2023-10-16T12:11:08.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.376+0000 7faccd0f9700 1 -- 172.21.15.16:0/1049939813 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facc400b9c0 con 0x7facc8071470 2023-10-16T12:11:08.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.376+0000 7faccd0f9700 1 -- 172.21.15.16:0/1049939813 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc400bd50 con 0x7facc8071470 2023-10-16T12:11:08.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.377+0000 7facd035f700 1 -- 172.21.15.16:0/1049939813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facc8071470 msgr2=0x7facc8162d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:08.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.377+0000 7facd035f700 1 --2- 172.21.15.16:0/1049939813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facc8071470 0x7facc8162d40 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7facc40098f0 tx=0x7facc4009c00 comp rx=0 tx=0).stop 2023-10-16T12:11:08.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.384+0000 7facd035f700 1 -- 172.21.15.16:0/1049939813 shutdown_connections 2023-10-16T12:11:08.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.384+0000 7facd035f700 1 --2- 172.21.15.16:0/1049939813 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7facc80b2a50 0x7facc80b2ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:08.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.384+0000 7facd035f700 1 --2- 172.21.15.16:0/1049939813 >> v2:172.21.15.49:3300/0 conn(0x7facc815c330 0x7facc8163480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:08.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.384+0000 7facd035f700 1 --2- 172.21.15.16:0/1049939813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facc8071470 0x7facc8162d40 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:08.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.384+0000 7facd035f700 1 -- 172.21.15.16:0/1049939813 >> 172.21.15.16:0/1049939813 conn(0x7facc806a7e0 msgr2=0x7facc806abf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:08.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.384+0000 7facd035f700 1 -- 172.21.15.16:0/1049939813 shutdown_connections 2023-10-16T12:11:08.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.384+0000 7facd035f700 1 -- 172.21.15.16:0/1049939813 wait complete. 2023-10-16T12:11:08.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.386+0000 7facd035f700 1 Processor -- start 2023-10-16T12:11:08.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.386+0000 7facd035f700 1 -- start start 2023-10-16T12:11:08.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.387+0000 7facd035f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facc8071470 0x7facc8159550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:08.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.388+0000 7facd035f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7facc80b2a50 0x7facc80a84b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:08.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.388+0000 7facd035f700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7facc815c330 0x7facc80a89f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:08.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.388+0000 7facd035f700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facc82007a0 con 0x7facc8071470 2023-10-16T12:11:08.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.388+0000 7facd035f700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facc8200990 con 0x7facc815c330 2023-10-16T12:11:08.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.388+0000 7facd035f700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facc8200c70 con 0x7facc80b2a50 2023-10-16T12:11:08.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.388+0000 7facce0fb700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facc8071470 0x7facc8159550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:08.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.388+0000 7facce0fb700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facc8071470 0x7facc8159550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33852/0 (socket says 172.21.15.16:33852) 2023-10-16T12:11:08.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.388+0000 7facce0fb700 1 -- 172.21.15.16:0/281759993 learned_addr learned my addr 172.21.15.16:0/281759993 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:08.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.388+0000 7facce0fb700 1 -- 172.21.15.16:0/281759993 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7facc80b2a50 msgr2=0x7facc80a84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:08.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.388+0000 7facce0fb700 1 --2- 172.21.15.16:0/281759993 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7facc80b2a50 0x7facc80a84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:08.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.388+0000 7facce0fb700 1 -- 172.21.15.16:0/281759993 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7facc815c330 msgr2=0x7facc80a89f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-16T12:11:08.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.388+0000 7facce0fb700 1 --2- 172.21.15.16:0/281759993 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7facc815c330 0x7facc80a89f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:08.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.388+0000 7facce0fb700 1 -- 172.21.15.16:0/281759993 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facc4009550 con 0x7facc8071470 2023-10-16T12:11:08.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.389+0000 7facce0fb700 1 --2- 172.21.15.16:0/281759993 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facc8071470 0x7facc8159550 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7facb800b550 tx=0x7facb800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:08.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.390+0000 7facbeffd700 1 -- 172.21.15.16:0/281759993 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facb8011830 con 0x7facc8071470 2023-10-16T12:11:08.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.390+0000 7facbeffd700 1 -- 172.21.15.16:0/281759993 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facb80119b0 con 0x7facc8071470 2023-10-16T12:11:08.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.390+0000 7facbeffd700 1 -- 172.21.15.16:0/281759993 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facb8011d60 con 0x7facc8071470 2023-10-16T12:11:08.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.390+0000 7facd035f700 1 -- 172.21.15.16:0/281759993 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facc8200f50 con 0x7facc8071470 2023-10-16T12:11:08.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.390+0000 7facd035f700 1 -- 172.21.15.16:0/281759993 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facc8201510 con 0x7facc8071470 2023-10-16T12:11:08.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.391+0000 7facd035f700 1 -- 172.21.15.16:0/281759993 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facc8201730 con 0x7facc8071470 2023-10-16T12:11:08.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.394+0000 7facbeffd700 1 -- 172.21.15.16:0/281759993 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7facb801a400 con 0x7facc8071470 2023-10-16T12:11:08.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.394+0000 7facbeffd700 1 --2- 172.21.15.16:0/281759993 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7facb406b130 0x7facb406d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:08.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.394+0000 7facbeffd700 1 -- 172.21.15.16:0/281759993 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7facb8089fa0 con 0x7facc8071470 2023-10-16T12:11:08.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.395+0000 7f82c5a7b700 1 Processor -- start 2023-10-16T12:11:08.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.395+0000 7f82c5a7b700 1 -- start start 2023-10-16T12:11:08.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.396+0000 7f82c5a7b700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f82c00716e0 0x7f82c015a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:08.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.396+0000 7f82c5a7b700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82c0071b50 con 0x7f82c01599f0 2023-10-16T12:11:08.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.396+0000 7f82c5a7b700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82c00b2fd0 con 0x7f82c0071250 2023-10-16T12:11:08.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.396+0000 7f82c5a7b700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82c015d240 con 0x7f82c00716e0 2023-10-16T12:11:08.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.396+0000 7f82bf7fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f82c01599f0 0x7f82c015ddc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51362/0 (socket says 172.21.15.16:51362) 2023-10-16T12:11:08.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.396+0000 7f82bf7fe700 1 -- 172.21.15.16:0/1937649654 learned_addr learned my addr 172.21.15.16:0/1937649654 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:08.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.398+0000 7faccd8fa700 1 --2- 172.21.15.16:0/281759993 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7facb406b130 0x7facb406d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:08.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.399+0000 7faccd8fa700 1 --2- 172.21.15.16:0/281759993 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7facb406b130 0x7facb406d5f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7facc8071900 tx=0x7facc4010000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:08.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.401+0000 7f82bffff700 1 --2- 172.21.15.16:0/1937649654 >> v2:172.21.15.49:3300/0 conn(0x7f82c00716e0 0x7f82c015a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:08.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.401+0000 7facbeffd700 1 -- 172.21.15.16:0/281759993 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7facb8054d40 con 0x7facc8071470 2023-10-16T12:11:08.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.402+0000 7f82bffff700 1 -- 172.21.15.16:0/1937649654 >> v1:172.21.15.49:6789/0 conn(0x7f82c0071250 legacy=0x7f82c015a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:08.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.402+0000 7f82bffff700 1 -- 172.21.15.16:0/1937649654 >> v1:172.21.15.16:6789/0 conn(0x7f82c01599f0 legacy=0x7f82c015ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:08.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.402+0000 7f82bffff700 1 -- 172.21.15.16:0/1937649654 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82c00b32b0 con 0x7f82c00716e0 2023-10-16T12:11:08.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.402+0000 7f82bffff700 1 --2- 172.21.15.16:0/1937649654 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f82c00716e0 0x7f82c015a740 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f82b0009840 tx=0x7f82b0009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e979f3eae515848e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:08.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.408+0000 7f82be7fc700 1 -- 172.21.15.16:0/1937649654 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82b0004230 con 0x7f82c00716e0 2023-10-16T12:11:08.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.408+0000 7f82be7fc700 1 -- 172.21.15.16:0/1937649654 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82b0004390 con 0x7f82c00716e0 2023-10-16T12:11:08.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.408+0000 7f82be7fc700 1 -- 172.21.15.16:0/1937649654 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82b00036a0 con 0x7f82c00716e0 2023-10-16T12:11:08.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.409+0000 7f82c5a7b700 1 -- 172.21.15.16:0/1937649654 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f82c00716e0 msgr2=0x7f82c015a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:08.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.409+0000 7f82c5a7b700 1 --2- 172.21.15.16:0/1937649654 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f82c00716e0 0x7f82c015a740 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f82b0009840 tx=0x7f82b0009b50 comp rx=0 tx=0).stop 2023-10-16T12:11:08.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.416+0000 7f82c5a7b700 1 -- 172.21.15.16:0/1937649654 shutdown_connections 2023-10-16T12:11:08.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.416+0000 7f82c5a7b700 1 --2- 172.21.15.16:0/1937649654 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f82c00716e0 0x7f82c015a740 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:08.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.416+0000 7f82c5a7b700 1 -- 172.21.15.16:0/1937649654 >> 172.21.15.16:0/1937649654 conn(0x7f82c006a7e0 msgr2=0x7f82c006abf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:08.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.426+0000 7f82c5a7b700 1 -- 172.21.15.16:0/1937649654 shutdown_connections 2023-10-16T12:11:08.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.426+0000 7f82c5a7b700 1 -- 172.21.15.16:0/1937649654 wait complete. 2023-10-16T12:11:08.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.428+0000 7f82c5a7b700 1 Processor -- start 2023-10-16T12:11:08.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.429+0000 7f82c5a7b700 1 -- start start 2023-10-16T12:11:08.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.430+0000 7f82c5a7b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82c0071250 0x7f82c0151c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:08.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.430+0000 7f82c5a7b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f82c00716e0 0x7f82c00cc200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:08.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.430+0000 7f82c5a7b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f82c01599f0 0x7f82c0154180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:08.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.430+0000 7f82c5a7b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82c00cdbd0 con 0x7f82c0071250 2023-10-16T12:11:08.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.430+0000 7f82c5a7b700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82c00ccb30 con 0x7f82c00716e0 2023-10-16T12:11:08.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.430+0000 7f82c5a7b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82c00cce10 con 0x7f82c01599f0 2023-10-16T12:11:08.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.431+0000 7f82bffff700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f82c01599f0 0x7f82c0154180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:08.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.431+0000 7f82bffff700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f82c01599f0 0x7f82c0154180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:50356/0 (socket says 172.21.15.16:50356) 2023-10-16T12:11:08.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.431+0000 7f82beffd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f82c00716e0 0x7f82c00cc200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:08.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.431+0000 7f82bffff700 1 -- 172.21.15.16:0/1665339686 learned_addr learned my addr 172.21.15.16:0/1665339686 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:08.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.431+0000 7f82bf7fe700 1 --2- 172.21.15.16:0/1665339686 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82c0071250 0x7f82c0151c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:08.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.432+0000 7f82bf7fe700 1 -- 172.21.15.16:0/1665339686 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f82c01599f0 msgr2=0x7f82c0154180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:08.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.432+0000 7f82bf7fe700 1 --2- 172.21.15.16:0/1665339686 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f82c01599f0 0x7f82c0154180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:08.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.432+0000 7f82bf7fe700 1 -- 172.21.15.16:0/1665339686 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f82c00716e0 msgr2=0x7f82c00cc200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:08.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.432+0000 7f82bf7fe700 1 --2- 172.21.15.16:0/1665339686 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f82c00716e0 0x7f82c00cc200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:08.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.432+0000 7f82bf7fe700 1 -- 172.21.15.16:0/1665339686 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82b00094a0 con 0x7f82c0071250 2023-10-16T12:11:08.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.432+0000 7f82bf7fe700 1 --2- 172.21.15.16:0/1665339686 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82c0071250 0x7f82c0151c10 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f82b800cc20 tx=0x7f82b800cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:08.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.434+0000 7f82a7fff700 1 -- 172.21.15.16:0/1665339686 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82b800aaa0 con 0x7f82c0071250 2023-10-16T12:11:08.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.434+0000 7f82a7fff700 1 -- 172.21.15.16:0/1665339686 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82b800ac00 con 0x7f82c0071250 2023-10-16T12:11:08.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.434+0000 7f82a7fff700 1 -- 172.21.15.16:0/1665339686 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82b80070d0 con 0x7f82c0071250 2023-10-16T12:11:08.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.434+0000 7f82c5a7b700 1 -- 172.21.15.16:0/1665339686 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82c00cd0f0 con 0x7f82c0071250 2023-10-16T12:11:08.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.434+0000 7f82c5a7b700 1 -- 172.21.15.16:0/1665339686 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82c00cd6b0 con 0x7f82c0071250 2023-10-16T12:11:08.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.438+0000 7f82a7fff700 1 -- 172.21.15.16:0/1665339686 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f82b8007230 con 0x7f82c0071250 2023-10-16T12:11:08.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.438+0000 7f82a7fff700 1 --2- 172.21.15.16:0/1665339686 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f82a806b130 0x7f82a806d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:08.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.439+0000 7f82a7fff700 1 -- 172.21.15.16:0/1665339686 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f82b80895c0 con 0x7f82c0071250 2023-10-16T12:11:08.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.440+0000 7f82beffd700 1 --2- 172.21.15.16:0/1665339686 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f82a806b130 0x7f82a806d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:08.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.441+0000 7f82beffd700 1 --2- 172.21.15.16:0/1665339686 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f82a806b130 0x7f82a806d5f0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f82c00cf200 tx=0x7f82b4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:08.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.442+0000 7f82c5a7b700 1 -- 172.21.15.16:0/1665339686 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82a00052f0 con 0x7f82c0071250 2023-10-16T12:11:08.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.451+0000 7f82a7fff700 1 -- 172.21.15.16:0/1665339686 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f82b8057ea0 con 0x7f82c0071250 2023-10-16T12:11:08.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.654+0000 7f6cdbcec700 1 -- 172.21.15.16:0/3610258077 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f6cbc0055a0 con 0x7f6cd4071470 2023-10-16T12:11:08.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.663+0000 7f6cca7fc700 1 -- 172.21.15.16:0/3610258077 <== mon.1 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6cc4017090 con 0x7f6cd4071470 2023-10-16T12:11:08.667 INFO:teuthology.orchestra.run.smithi016.stdout:55834574859 2023-10-16T12:11:08.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.666+0000 7f6cdbcec700 1 -- 172.21.15.16:0/3610258077 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f6cb806afd0 msgr2=0x7f6cb806d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:08.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.666+0000 7f6cdbcec700 1 --2- 172.21.15.16:0/3610258077 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f6cb806afd0 0x7f6cb806d490 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6cd414be00 tx=0x7f6ccc00b040 comp rx=0 tx=0).stop 2023-10-16T12:11:08.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.666+0000 7f6cdbcec700 1 -- 172.21.15.16:0/3610258077 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6cd4071470 msgr2=0x7f6cd415d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:08.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.666+0000 7f6cdbcec700 1 --2- 172.21.15.16:0/3610258077 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6cd4071470 0x7f6cd415d500 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f6cc400c480 tx=0x7f6cc400c790 comp rx=0 tx=0).stop 2023-10-16T12:11:08.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.667+0000 7f6cdbcec700 1 -- 172.21.15.16:0/3610258077 shutdown_connections 2023-10-16T12:11:08.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.667+0000 7f6cdbcec700 1 --2- 172.21.15.16:0/3610258077 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f6cb806afd0 0x7f6cb806d490 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:08.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.667+0000 7f6cdbcec700 1 --2- 172.21.15.16:0/3610258077 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6cd415c330 0x7f6cd41640e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:08.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.667+0000 7f6cdbcec700 1 --2- 172.21.15.16:0/3610258077 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6cd4071470 0x7f6cd415d500 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:08.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.667+0000 7f6cdbcec700 1 --2- 172.21.15.16:0/3610258077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cd40b2a50 0x7f6cd415ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:08.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.667+0000 7f6cdbcec700 1 -- 172.21.15.16:0/3610258077 >> 172.21.15.16:0/3610258077 conn(0x7f6cd406a7e0 msgr2=0x7f6cd414c8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:08.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.667+0000 7f6cdbcec700 1 -- 172.21.15.16:0/3610258077 shutdown_connections 2023-10-16T12:11:08.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.667+0000 7f6cdbcec700 1 -- 172.21.15.16:0/3610258077 wait complete. 2023-10-16T12:11:08.687 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574859 for osd.7 2023-10-16T12:11:08.687 DEBUG:teuthology.parallel:result is None 2023-10-16T12:11:08.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.689+0000 7fe2b684c700 1 -- 172.21.15.16:0/281667587 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fe290005870 con 0x7fe2b0071250 2023-10-16T12:11:08.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.690+0000 7fe2acff9700 1 -- 172.21.15.16:0/281667587 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe2a4015020 con 0x7fe2b0071250 2023-10-16T12:11:08.692 INFO:teuthology.orchestra.run.smithi016.stdout:55834574858 2023-10-16T12:11:08.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.694+0000 7fe2b684c700 1 -- 172.21.15.16:0/281667587 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fe29c06af50 msgr2=0x7fe29c06d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:08.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.694+0000 7fe2b684c700 1 --2- 172.21.15.16:0/281667587 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fe29c06af50 0x7fe29c06d410 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fe2a0009690 tx=0x7fe2a0006c10 comp rx=0 tx=0).stop 2023-10-16T12:11:08.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.694+0000 7fe2b684c700 1 -- 172.21.15.16:0/281667587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2b0071250 msgr2=0x7fe2b01390b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:08.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.694+0000 7fe2b684c700 1 --2- 172.21.15.16:0/281667587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2b0071250 0x7fe2b01390b0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fe2a4000c00 tx=0x7fe2a4004750 comp rx=0 tx=0).stop 2023-10-16T12:11:08.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.698+0000 7fe2b684c700 1 -- 172.21.15.16:0/281667587 shutdown_connections 2023-10-16T12:11:08.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.698+0000 7fe2b684c700 1 --2- 172.21.15.16:0/281667587 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fe29c06af50 0x7fe29c06d410 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:08.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.698+0000 7fe2b684c700 1 --2- 172.21.15.16:0/281667587 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe2b01599f0 0x7fe2b0152130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:08.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.698+0000 7fe2b684c700 1 --2- 172.21.15.16:0/281667587 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe2b006c800 0x7fe2b0152e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:08.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.698+0000 7fe2b684c700 1 --2- 172.21.15.16:0/281667587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2b0071250 0x7fe2b01390b0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:08.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.698+0000 7fe2b684c700 1 -- 172.21.15.16:0/281667587 >> 172.21.15.16:0/281667587 conn(0x7fe2b006a7e0 msgr2=0x7fe2b006cc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:08.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.698+0000 7fe2b684c700 1 -- 172.21.15.16:0/281667587 shutdown_connections 2023-10-16T12:11:08.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.698+0000 7fe2b684c700 1 -- 172.21.15.16:0/281667587 wait complete. 2023-10-16T12:11:08.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.713+0000 7faa4ea4a700 1 -- 172.21.15.16:0/3972818196 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7faa28005870 con 0x7faa4815b040 2023-10-16T12:11:08.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.713+0000 7faa44ff9700 1 -- 172.21.15.16:0/3972818196 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7faa40028070 con 0x7faa4815b040 2023-10-16T12:11:08.715 INFO:teuthology.orchestra.run.smithi016.stdout:55834574857 2023-10-16T12:11:08.717 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574858 for osd.2 2023-10-16T12:11:08.718 DEBUG:teuthology.parallel:result is None 2023-10-16T12:11:08.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.717+0000 7faa4ea4a700 1 -- 172.21.15.16:0/3972818196 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7faa3406af50 msgr2=0x7faa3406d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:08.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.717+0000 7faa4ea4a700 1 --2- 172.21.15.16:0/3972818196 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7faa3406af50 0x7faa3406d410 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7faa3c009c20 tx=0x7faa3c008040 comp rx=0 tx=0).stop 2023-10-16T12:11:08.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.717+0000 7faa4ea4a700 1 -- 172.21.15.16:0/3972818196 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa4815b040 msgr2=0x7faa481577e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:08.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.717+0000 7faa4ea4a700 1 --2- 172.21.15.16:0/3972818196 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa4815b040 0x7faa481577e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7faa40017000 tx=0x7faa40004ad0 comp rx=0 tx=0).stop 2023-10-16T12:11:08.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.718+0000 7faa4ea4a700 1 -- 172.21.15.16:0/3972818196 shutdown_connections 2023-10-16T12:11:08.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.718+0000 7faa4ea4a700 1 --2- 172.21.15.16:0/3972818196 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7faa3406af50 0x7faa3406d410 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:08.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.718+0000 7faa4ea4a700 1 --2- 172.21.15.16:0/3972818196 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa4815b040 0x7faa481577e0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:08.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.718+0000 7faa4ea4a700 1 --2- 172.21.15.16:0/3972818196 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faa48071250 0x7faa48150510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:08.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.718+0000 7faa4ea4a700 1 --2- 172.21.15.16:0/3972818196 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faa481599f0 0x7faa480acd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:08.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.718+0000 7faa4ea4a700 1 -- 172.21.15.16:0/3972818196 >> 172.21.15.16:0/3972818196 conn(0x7faa4806a7e0 msgr2=0x7faa480af3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:08.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.718+0000 7faa4ea4a700 1 -- 172.21.15.16:0/3972818196 shutdown_connections 2023-10-16T12:11:08.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.718+0000 7faa4ea4a700 1 -- 172.21.15.16:0/3972818196 wait complete. 2023-10-16T12:11:08.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.727+0000 7eff1e24d700 1 -- 172.21.15.16:0/342809733 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7eff18200440 con 0x7eff1806d1d0 2023-10-16T12:11:08.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.727+0000 7efefffff700 1 -- 172.21.15.16:0/342809733 <== mon.2 v2:172.21.15.49:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7eff0801f020 con 0x7eff1806d1d0 2023-10-16T12:11:08.729 INFO:teuthology.orchestra.run.smithi016.stdout:55834574857 2023-10-16T12:11:08.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.730+0000 7f53ac55b700 1 -- 172.21.15.16:0/2841366340 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f538c005870 con 0x7f53a415dd90 2023-10-16T12:11:08.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.730+0000 7f5396ffd700 1 -- 172.21.15.16:0/2841366340 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f539c017070 con 0x7f53a415dd90 2023-10-16T12:11:08.733 INFO:teuthology.orchestra.run.smithi016.stdout:55834574858 2023-10-16T12:11:08.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.731+0000 7eff1e24d700 1 -- 172.21.15.16:0/342809733 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7eff0406af00 msgr2=0x7eff0406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:08.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.731+0000 7eff1e24d700 1 --2- 172.21.15.16:0/342809733 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7eff0406af00 0x7eff0406d3c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7eff0c00a950 tx=0x7eff0c009030 comp rx=0 tx=0).stop 2023-10-16T12:11:08.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.731+0000 7eff1e24d700 1 -- 172.21.15.16:0/342809733 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7eff1806d1d0 msgr2=0x7eff18203750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:08.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.731+0000 7eff1e24d700 1 --2- 172.21.15.16:0/342809733 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7eff1806d1d0 0x7eff18203750 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7eff08000f60 tx=0x7eff0800e9e0 comp rx=0 tx=0).stop 2023-10-16T12:11:08.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.732+0000 7eff1e24d700 1 -- 172.21.15.16:0/342809733 shutdown_connections 2023-10-16T12:11:08.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.732+0000 7eff1e24d700 1 --2- 172.21.15.16:0/342809733 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7eff0406af00 0x7eff0406d3c0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:08.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.732+0000 7eff1e24d700 1 --2- 172.21.15.16:0/342809733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff1808aa20 0x7eff181662d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:08.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.732+0000 7eff1e24d700 1 --2- 172.21.15.16:0/342809733 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7eff1806d1d0 0x7eff18203750 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:08.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.732+0000 7eff1e24d700 1 --2- 172.21.15.16:0/342809733 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7eff1808a640 0x7eff180cf120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:08.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.733+0000 7eff1e24d700 1 -- 172.21.15.16:0/342809733 >> 172.21.15.16:0/342809733 conn(0x7eff1806a7e0 msgr2=0x7eff18162de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:08.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.733+0000 7f53ac55b700 1 -- 172.21.15.16:0/2841366340 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f539006b0b0 msgr2=0x7f539006d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:08.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.733+0000 7f53ac55b700 1 --2- 172.21.15.16:0/2841366340 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f539006b0b0 0x7f539006d570 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f5398019040 tx=0x7f5398008040 comp rx=0 tx=0).stop 2023-10-16T12:11:08.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.733+0000 7f53ac55b700 1 -- 172.21.15.16:0/2841366340 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53a415dd90 msgr2=0x7f53a40c8430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:08.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.733+0000 7f53ac55b700 1 --2- 172.21.15.16:0/2841366340 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53a415dd90 0x7f53a40c8430 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f539c00c7d0 tx=0x7f539c00cae0 comp rx=0 tx=0).stop 2023-10-16T12:11:08.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.733+0000 7eff1e24d700 1 -- 172.21.15.16:0/342809733 shutdown_connections 2023-10-16T12:11:08.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.733+0000 7eff1e24d700 1 -- 172.21.15.16:0/342809733 wait complete. 2023-10-16T12:11:08.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.734+0000 7f53ac55b700 1 -- 172.21.15.16:0/2841366340 shutdown_connections 2023-10-16T12:11:08.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.734+0000 7f53ac55b700 1 --2- 172.21.15.16:0/2841366340 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f539006b0b0 0x7f539006d570 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:08.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.734+0000 7f53ac55b700 1 --2- 172.21.15.16:0/2841366340 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53a415dd90 0x7f53a40c8430 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:08.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.734+0000 7f53ac55b700 1 --2- 172.21.15.16:0/2841366340 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f53a40cbb40 0x7f53a41514b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:08.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.734+0000 7f53ac55b700 1 --2- 172.21.15.16:0/2841366340 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f53a415a230 0x7f53a40c6c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:08.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.734+0000 7f53ac55b700 1 -- 172.21.15.16:0/2841366340 >> 172.21.15.16:0/2841366340 conn(0x7f53a40cc5e0 msgr2=0x7f53a40b2e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:08.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.735+0000 7f53ac55b700 1 -- 172.21.15.16:0/2841366340 shutdown_connections 2023-10-16T12:11:08.740 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.1 2023-10-16T12:11:08.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.739+0000 7f53ac55b700 1 -- 172.21.15.16:0/2841366340 wait complete. 2023-10-16T12:11:08.741 DEBUG:teuthology.parallel:result is None 2023-10-16T12:11:08.753 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.5 2023-10-16T12:11:08.754 DEBUG:teuthology.parallel:result is None 2023-10-16T12:11:08.754 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574858 for osd.0 2023-10-16T12:11:08.755 DEBUG:teuthology.parallel:result is None 2023-10-16T12:11:08.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.755+0000 7f070b1bb700 1 -- 172.21.15.16:0/2655684751 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f07040c8e10 con 0x7f07040d2f70 2023-10-16T12:11:08.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.755+0000 7f07017fa700 1 -- 172.21.15.16:0/2655684751 <== mon.1 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f07040c8e10 con 0x7f07040d2f70 2023-10-16T12:11:08.757 INFO:teuthology.orchestra.run.smithi016.stdout:55834574856 2023-10-16T12:11:08.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.757+0000 7f070b1bb700 1 -- 172.21.15.16:0/2655684751 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f06e806b000 msgr2=0x7f06e806d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:08.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.757+0000 7f070b1bb700 1 --2- 172.21.15.16:0/2655684751 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f06e806b000 0x7f06e806d4c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f06f8000c00 tx=0x7f06f8007a50 comp rx=0 tx=0).stop 2023-10-16T12:11:08.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.757+0000 7f070b1bb700 1 -- 172.21.15.16:0/2655684751 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f07040d2f70 msgr2=0x7f07040c7970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:08.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.757+0000 7f070b1bb700 1 --2- 172.21.15.16:0/2655684751 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f07040d2f70 0x7f07040c7970 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f06f0009f80 tx=0x7f06f0004750 comp rx=0 tx=0).stop 2023-10-16T12:11:08.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.757+0000 7f070b1bb700 1 -- 172.21.15.16:0/2655684751 shutdown_connections 2023-10-16T12:11:08.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.757+0000 7f070b1bb700 1 --2- 172.21.15.16:0/2655684751 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f06e806b000 0x7f06e806d4c0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:08.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.757+0000 7f070b1bb700 1 --2- 172.21.15.16:0/2655684751 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0704160410 0x7f07041f3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:08.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.757+0000 7f070b1bb700 1 --2- 172.21.15.16:0/2655684751 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f07040d2f70 0x7f07040c7970 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:08.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.757+0000 7f070b1bb700 1 --2- 172.21.15.16:0/2655684751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f070415d900 0x7f070415f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:08.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.757+0000 7f070b1bb700 1 -- 172.21.15.16:0/2655684751 >> 172.21.15.16:0/2655684751 conn(0x7f0704072330 msgr2=0x7f070406bc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:08.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.758+0000 7f070b1bb700 1 -- 172.21.15.16:0/2655684751 shutdown_connections 2023-10-16T12:11:08.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.758+0000 7f070b1bb700 1 -- 172.21.15.16:0/2655684751 wait complete. 2023-10-16T12:11:08.771 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.4 2023-10-16T12:11:08.772 DEBUG:teuthology.parallel:result is None 2023-10-16T12:11:08.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.774+0000 7facd035f700 1 -- 172.21.15.16:0/281759993 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7facb00051a0 con 0x7facc8071470 2023-10-16T12:11:08.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.774+0000 7facbeffd700 1 -- 172.21.15.16:0/281759993 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7facb8015090 con 0x7facc8071470 2023-10-16T12:11:08.777 INFO:teuthology.orchestra.run.smithi016.stdout:55834574857 2023-10-16T12:11:08.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.777+0000 7facd035f700 1 -- 172.21.15.16:0/281759993 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7facb406b130 msgr2=0x7facb406d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:08.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.777+0000 7facd035f700 1 --2- 172.21.15.16:0/281759993 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7facb406b130 0x7facb406d5f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7facc8071900 tx=0x7facc4010000 comp rx=0 tx=0).stop 2023-10-16T12:11:08.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.777+0000 7facd035f700 1 -- 172.21.15.16:0/281759993 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facc8071470 msgr2=0x7facc8159550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:08.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.777+0000 7facd035f700 1 --2- 172.21.15.16:0/281759993 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facc8071470 0x7facc8159550 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7facb800b550 tx=0x7facb800b910 comp rx=0 tx=0).stop 2023-10-16T12:11:08.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.777+0000 7facd035f700 1 -- 172.21.15.16:0/281759993 shutdown_connections 2023-10-16T12:11:08.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.777+0000 7facd035f700 1 --2- 172.21.15.16:0/281759993 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7facb406b130 0x7facb406d5f0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:08.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.777+0000 7facd035f700 1 --2- 172.21.15.16:0/281759993 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7facc815c330 0x7facc80a89f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:08.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.777+0000 7facd035f700 1 --2- 172.21.15.16:0/281759993 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facc8071470 0x7facc8159550 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:08.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.777+0000 7facd035f700 1 --2- 172.21.15.16:0/281759993 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7facc80b2a50 0x7facc80a84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:08.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.777+0000 7facd035f700 1 -- 172.21.15.16:0/281759993 >> 172.21.15.16:0/281759993 conn(0x7facc806a7e0 msgr2=0x7facc81646d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:08.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.777+0000 7facd035f700 1 -- 172.21.15.16:0/281759993 shutdown_connections 2023-10-16T12:11:08.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.777+0000 7facd035f700 1 -- 172.21.15.16:0/281759993 wait complete. 2023-10-16T12:11:08.791 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.6 2023-10-16T12:11:08.791 DEBUG:teuthology.parallel:result is None 2023-10-16T12:11:08.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.807+0000 7f82c5a7b700 1 -- 172.21.15.16:0/1665339686 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f82a00055a0 con 0x7f82c0071250 2023-10-16T12:11:08.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.808+0000 7f82a7fff700 1 -- 172.21.15.16:0/1665339686 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f82b8018030 con 0x7f82c0071250 2023-10-16T12:11:08.810 INFO:teuthology.orchestra.run.smithi016.stdout:55834574857 2023-10-16T12:11:08.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.810+0000 7f82c5a7b700 1 -- 172.21.15.16:0/1665339686 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f82a806b130 msgr2=0x7f82a806d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:08.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.810+0000 7f82c5a7b700 1 --2- 172.21.15.16:0/1665339686 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f82a806b130 0x7f82a806d5f0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f82c00cf200 tx=0x7f82b4009040 comp rx=0 tx=0).stop 2023-10-16T12:11:08.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.810+0000 7f82c5a7b700 1 -- 172.21.15.16:0/1665339686 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82c0071250 msgr2=0x7f82c0151c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:08.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.810+0000 7f82c5a7b700 1 --2- 172.21.15.16:0/1665339686 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82c0071250 0x7f82c0151c10 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f82b800cc20 tx=0x7f82b800cf30 comp rx=0 tx=0).stop 2023-10-16T12:11:08.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.810+0000 7f82c5a7b700 1 -- 172.21.15.16:0/1665339686 shutdown_connections 2023-10-16T12:11:08.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.810+0000 7f82c5a7b700 1 --2- 172.21.15.16:0/1665339686 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f82a806b130 0x7f82a806d5f0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:08.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.810+0000 7f82c5a7b700 1 --2- 172.21.15.16:0/1665339686 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f82c01599f0 0x7f82c0154180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:08.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.810+0000 7f82c5a7b700 1 --2- 172.21.15.16:0/1665339686 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82c0071250 0x7f82c0151c10 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:08.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.810+0000 7f82c5a7b700 1 --2- 172.21.15.16:0/1665339686 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f82c00716e0 0x7f82c00cc200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:08.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.810+0000 7f82c5a7b700 1 -- 172.21.15.16:0/1665339686 >> 172.21.15.16:0/1665339686 conn(0x7f82c006a7e0 msgr2=0x7f82c015b450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:08.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.810+0000 7f82c5a7b700 1 -- 172.21.15.16:0/1665339686 shutdown_connections 2023-10-16T12:11:08.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.810+0000 7f82c5a7b700 1 -- 172.21.15.16:0/1665339686 wait complete. 2023-10-16T12:11:08.824 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.3 2023-10-16T12:11:08.825 DEBUG:teuthology.parallel:result is None 2023-10-16T12:11:08.825 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-16T12:11:08.825 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T12:11:08.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.987+0000 7f6b36489700 1 Processor -- start 2023-10-16T12:11:08.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.988+0000 7f6b36489700 1 -- start start 2023-10-16T12:11:08.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.988+0000 7f6b36489700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f6b3015db30 0x7f6b300cf920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:08.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.988+0000 7f6b36489700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6b30161490 0x7f6b301618c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:08.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.988+0000 7f6b36489700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b30161e00 con 0x7f6b3015d750 2023-10-16T12:11:08.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.988+0000 7f6b36489700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b300cfe60 con 0x7f6b3015db30 2023-10-16T12:11:08.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.988+0000 7f6b36489700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b300d0070 con 0x7f6b30161490 2023-10-16T12:11:08.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.989+0000 7f6b2ffff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f6b3015d750 0x7f6b3006c960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51378/0 (socket says 172.21.15.16:51378) 2023-10-16T12:11:08.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.989+0000 7f6b2ffff700 1 -- 172.21.15.16:0/2206136730 learned_addr learned my addr 172.21.15.16:0/2206136730 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:08.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.989+0000 7f6b34a26700 1 --2- 172.21.15.16:0/2206136730 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6b30161490 0x7f6b301618c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:08.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.989+0000 7f6b2f7fe700 1 --2- 172.21.15.16:0/2206136730 >> v2:172.21.15.49:3300/0 conn(0x7f6b3015db30 0x7f6b300cf920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:08.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.989+0000 7f6b2effd700 1 -- 172.21.15.16:0/2206136730 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 56368393 0 0) 0x7f6b30161e00 con 0x7f6b3015d750 2023-10-16T12:11:08.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.989+0000 7f6b2effd700 1 -- 172.21.15.16:0/2206136730 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b180035b0 con 0x7f6b3015d750 2023-10-16T12:11:08.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.989+0000 7f6b34a26700 1 -- 172.21.15.16:0/2206136730 >> v2:172.21.15.49:3300/0 conn(0x7f6b3015db30 msgr2=0x7f6b300cf920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:09.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.989+0000 7f6b34a26700 1 --2- 172.21.15.16:0/2206136730 >> v2:172.21.15.49:3300/0 conn(0x7f6b3015db30 0x7f6b300cf920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:09.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.989+0000 7f6b34a26700 1 -- 172.21.15.16:0/2206136730 >> v1:172.21.15.16:6789/0 conn(0x7f6b3015d750 legacy=0x7f6b3006c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:09.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.989+0000 7f6b2effd700 1 -- 172.21.15.16:0/2206136730 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1417874658 0 0) 0x7f6b180035b0 con 0x7f6b3015d750 2023-10-16T12:11:09.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.989+0000 7f6b34a26700 1 -- 172.21.15.16:0/2206136730 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b300d0350 con 0x7f6b30161490 2023-10-16T12:11:09.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.989+0000 7f6b34a26700 1 --2- 172.21.15.16:0/2206136730 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6b30161490 0x7f6b301618c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6b28009d90 tx=0x7f6b2800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=759704e9452ca8dc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:09.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.989+0000 7f6b2effd700 1 -- 172.21.15.16:0/2206136730 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b2800e070 con 0x7f6b30161490 2023-10-16T12:11:09.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.989+0000 7f6b2effd700 1 -- 172.21.15.16:0/2206136730 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b28003a20 con 0x7f6b30161490 2023-10-16T12:11:09.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.989+0000 7f6b2effd700 1 -- 172.21.15.16:0/2206136730 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b200017f0 con 0x7f6b30161490 2023-10-16T12:11:09.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.990+0000 7f6b36489700 1 -- 172.21.15.16:0/2206136730 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6b30161490 msgr2=0x7f6b301618c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:09.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.990+0000 7f6b36489700 1 --2- 172.21.15.16:0/2206136730 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6b30161490 0x7f6b301618c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6b28009d90 tx=0x7f6b2800b330 comp rx=0 tx=0).stop 2023-10-16T12:11:09.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.990+0000 7f6b36489700 1 -- 172.21.15.16:0/2206136730 shutdown_connections 2023-10-16T12:11:09.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.990+0000 7f6b36489700 1 --2- 172.21.15.16:0/2206136730 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6b30161490 0x7f6b301618c0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:09.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.990+0000 7f6b36489700 1 --2- 172.21.15.16:0/2206136730 >> v2:172.21.15.49:3300/0 conn(0x7f6b3015db30 0x7f6b300cf920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:09.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.990+0000 7f6b36489700 1 -- 172.21.15.16:0/2206136730 >> 172.21.15.16:0/2206136730 conn(0x7f6b300d14b0 msgr2=0x7f6b300d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:09.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.990+0000 7f6b36489700 1 -- 172.21.15.16:0/2206136730 shutdown_connections 2023-10-16T12:11:09.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.990+0000 7f6b36489700 1 -- 172.21.15.16:0/2206136730 wait complete. 2023-10-16T12:11:09.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.991+0000 7f6b36489700 1 Processor -- start 2023-10-16T12:11:09.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.992+0000 7f6b36489700 1 -- start start 2023-10-16T12:11:09.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.992+0000 7f6b36489700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6b3015d750 0x7f6b300b6f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:09.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.992+0000 7f6b36489700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b3015db30 0x7f6b300c6c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:09.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.992+0000 7f6b36489700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6b30161490 0x7f6b300c7140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:09.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.992+0000 7f6b36489700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b300b4530 con 0x7f6b3015db30 2023-10-16T12:11:09.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.992+0000 7f6b36489700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b300c64e0 con 0x7f6b30161490 2023-10-16T12:11:09.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.992+0000 7f6b36489700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b300c5970 con 0x7f6b3015d750 2023-10-16T12:11:09.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.992+0000 7f6b2f7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b3015db30 0x7f6b300c6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:09.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.992+0000 7f6b2f7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b3015db30 0x7f6b300c6c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33880/0 (socket says 172.21.15.16:33880) 2023-10-16T12:11:09.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.992+0000 7f6b2f7fe700 1 -- 172.21.15.16:0/2409248609 learned_addr learned my addr 172.21.15.16:0/2409248609 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:09.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.993+0000 7f6b2ffff700 1 --2- 172.21.15.16:0/2409248609 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6b3015d750 0x7f6b300b6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:09.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.993+0000 7f6b34a26700 1 --2- 172.21.15.16:0/2409248609 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6b30161490 0x7f6b300c7140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:09.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.993+0000 7f6b2f7fe700 1 -- 172.21.15.16:0/2409248609 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6b3015d750 msgr2=0x7f6b300b6f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:09.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.993+0000 7f6b2f7fe700 1 --2- 172.21.15.16:0/2409248609 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6b3015d750 0x7f6b300b6f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:09.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.993+0000 7f6b2f7fe700 1 -- 172.21.15.16:0/2409248609 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6b30161490 msgr2=0x7f6b300c7140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:09.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.993+0000 7f6b2f7fe700 1 --2- 172.21.15.16:0/2409248609 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6b30161490 0x7f6b300c7140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:09.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.993+0000 7f6b2f7fe700 1 -- 172.21.15.16:0/2409248609 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b28009a70 con 0x7f6b3015db30 2023-10-16T12:11:09.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.993+0000 7f6b2ffff700 1 --2- 172.21.15.16:0/2409248609 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6b3015d750 0x7f6b300b6f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:11:09.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.993+0000 7f6b2f7fe700 1 --2- 172.21.15.16:0/2409248609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b3015db30 0x7f6b300c6c00 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6b2400e9b0 tx=0x7f6b2400ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:09.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.993+0000 7f6b2cff9700 1 -- 172.21.15.16:0/2409248609 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b240096d0 con 0x7f6b3015db30 2023-10-16T12:11:09.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.993+0000 7f6b2cff9700 1 -- 172.21.15.16:0/2409248609 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b2400c850 con 0x7f6b3015db30 2023-10-16T12:11:09.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.993+0000 7f6b36489700 1 -- 172.21.15.16:0/2409248609 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b300c5c40 con 0x7f6b3015db30 2023-10-16T12:11:09.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.993+0000 7f6b2cff9700 1 -- 172.21.15.16:0/2409248609 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b2400cbe0 con 0x7f6b3015db30 2023-10-16T12:11:09.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.993+0000 7f6b36489700 1 -- 172.21.15.16:0/2409248609 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b300c6200 con 0x7f6b3015db30 2023-10-16T12:11:09.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.994+0000 7f6b36489700 1 -- 172.21.15.16:0/2409248609 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b300b4670 con 0x7f6b3015db30 2023-10-16T12:11:09.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.994+0000 7f6b2cff9700 1 -- 172.21.15.16:0/2409248609 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f6b2400cd40 con 0x7f6b3015db30 2023-10-16T12:11:09.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.995+0000 7f6b2cff9700 1 --2- 172.21.15.16:0/2409248609 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f6b1c06af00 0x7f6b1c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:09.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.995+0000 7f6b2cff9700 1 -- 172.21.15.16:0/2409248609 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f6b24089d00 con 0x7f6b3015db30 2023-10-16T12:11:09.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.995+0000 7f6b2ffff700 1 --2- 172.21.15.16:0/2409248609 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f6b1c06af00 0x7f6b1c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:09.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.995+0000 7f6b2ffff700 1 --2- 172.21.15.16:0/2409248609 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f6b1c06af00 0x7f6b1c06d3c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f6b20000ef0 tx=0x7f6b20009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:09.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:08.999+0000 7f6b2cff9700 1 -- 172.21.15.16:0/2409248609 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6b300b4670 con 0x7f6b3015db30 2023-10-16T12:11:09.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.198+0000 7f6b36489700 1 -- 172.21.15.16:0/2409248609 --> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6b300d0910 con 0x7f6b1c06af00 2023-10-16T12:11:09.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.208+0000 7f6b2cff9700 1 -- 172.21.15.16:0/2409248609 <== mgr.4104 v2:172.21.15.16:6832/365921436 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+345271 (secure 0 0 0) 0x7f6b20001710 con 0x7f6b1c06af00 2023-10-16T12:11:09.214 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:11:09.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.211+0000 7f6b36489700 1 -- 172.21.15.16:0/2409248609 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f6b1c06af00 msgr2=0x7f6b1c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:09.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.211+0000 7f6b36489700 1 --2- 172.21.15.16:0/2409248609 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f6b1c06af00 0x7f6b1c06d3c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f6b20000ef0 tx=0x7f6b20009040 comp rx=0 tx=0).stop 2023-10-16T12:11:09.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.211+0000 7f6b36489700 1 -- 172.21.15.16:0/2409248609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b3015db30 msgr2=0x7f6b300c6c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:09.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.211+0000 7f6b36489700 1 --2- 172.21.15.16:0/2409248609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b3015db30 0x7f6b300c6c00 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6b2400e9b0 tx=0x7f6b2400ed70 comp rx=0 tx=0).stop 2023-10-16T12:11:09.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.211+0000 7f6b36489700 1 -- 172.21.15.16:0/2409248609 shutdown_connections 2023-10-16T12:11:09.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.211+0000 7f6b36489700 1 --2- 172.21.15.16:0/2409248609 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f6b1c06af00 0x7f6b1c06d3c0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:09.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.211+0000 7f6b36489700 1 --2- 172.21.15.16:0/2409248609 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6b30161490 0x7f6b300c7140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:09.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.211+0000 7f6b36489700 1 --2- 172.21.15.16:0/2409248609 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6b3015d750 0x7f6b300b6f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:09.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.211+0000 7f6b36489700 1 --2- 172.21.15.16:0/2409248609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b3015db30 0x7f6b300c6c00 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:09.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.212+0000 7f6b36489700 1 -- 172.21.15.16:0/2409248609 >> 172.21.15.16:0/2409248609 conn(0x7f6b300d14b0 msgr2=0x7f6b3015fb50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:09.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.212+0000 7f6b36489700 1 -- 172.21.15.16:0/2409248609 shutdown_connections 2023-10-16T12:11:09.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.212+0000 7f6b36489700 1 -- 172.21.15.16:0/2409248609 wait complete. 2023-10-16T12:11:09.219 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-10-16T12:11:09.226 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2023-10-16T12:11:09.102112+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219472,"kb_used_data":3304,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755248,"statfs":{"total":773094113280,"available":772869373952,"internally_reserved":0,"allocated":3383296,"data_stored":2025624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12740,"internal_metadata":221236796},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":141,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003280"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.797013+0000","last_change":"2023-10-16T12:10:39.641670+0000","last_active":"2023-10-16T12:10:43.797013+0000","last_peered":"2023-10-16T12:10:43.797013+0000","last_clean":"2023-10-16T12:10:43.797013+0000","last_became_active":"2023-10-16T12:10:39.639911+0000","last_became_peered":"2023-10-16T12:10:39.639911+0000","last_unstale":"2023-10-16T12:10:43.797013+0000","last_undegraded":"2023-10-16T12:10:43.797013+0000","last_fullsized":"2023-10-16T12:10:43.797013+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:29:43.494490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.656566+0000","last_change":"2023-10-16T12:10:39.637937+0000","last_active":"2023-10-16T12:10:43.656566+0000","last_peered":"2023-10-16T12:10:43.656566+0000","last_clean":"2023-10-16T12:10:43.656566+0000","last_became_active":"2023-10-16T12:10:39.637595+0000","last_became_peered":"2023-10-16T12:10:39.637595+0000","last_unstale":"2023-10-16T12:10:43.656566+0000","last_undegraded":"2023-10-16T12:10:43.656566+0000","last_fullsized":"2023-10-16T12:10:43.656566+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:03:12.325906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646168+0000","last_change":"2023-10-16T12:10:39.630003+0000","last_active":"2023-10-16T12:10:43.646168+0000","last_peered":"2023-10-16T12:10:43.646168+0000","last_clean":"2023-10-16T12:10:43.646168+0000","last_became_active":"2023-10-16T12:10:39.627471+0000","last_became_peered":"2023-10-16T12:10:39.627471+0000","last_unstale":"2023-10-16T12:10:43.646168+0000","last_undegraded":"2023-10-16T12:10:43.646168+0000","last_fullsized":"2023-10-16T12:10:43.646168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:33:20.807621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646890+0000","last_change":"2023-10-16T12:10:39.645834+0000","last_active":"2023-10-16T12:10:43.646890+0000","last_peered":"2023-10-16T12:10:43.646890+0000","last_clean":"2023-10-16T12:10:43.646890+0000","last_became_active":"2023-10-16T12:10:39.645631+0000","last_became_peered":"2023-10-16T12:10:39.645631+0000","last_unstale":"2023-10-16T12:10:43.646890+0000","last_undegraded":"2023-10-16T12:10:43.646890+0000","last_fullsized":"2023-10-16T12:10:43.646890+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:14:00.725551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.794209+0000","last_change":"2023-10-16T12:10:39.642967+0000","last_active":"2023-10-16T12:10:43.794209+0000","last_peered":"2023-10-16T12:10:43.794209+0000","last_clean":"2023-10-16T12:10:43.794209+0000","last_became_active":"2023-10-16T12:10:39.642015+0000","last_became_peered":"2023-10-16T12:10:39.642015+0000","last_unstale":"2023-10-16T12:10:43.794209+0000","last_undegraded":"2023-10-16T12:10:43.794209+0000","last_fullsized":"2023-10-16T12:10:43.794209+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:10:19.840696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646377+0000","last_change":"2023-10-16T12:10:39.648342+0000","last_active":"2023-10-16T12:10:43.646377+0000","last_peered":"2023-10-16T12:10:43.646377+0000","last_clean":"2023-10-16T12:10:43.646377+0000","last_became_active":"2023-10-16T12:10:39.648046+0000","last_became_peered":"2023-10-16T12:10:39.648046+0000","last_unstale":"2023-10-16T12:10:43.646377+0000","last_undegraded":"2023-10-16T12:10:43.646377+0000","last_fullsized":"2023-10-16T12:10:43.646377+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:27:44.016376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.793686+0000","last_change":"2023-10-16T12:10:39.640757+0000","last_active":"2023-10-16T12:10:43.793686+0000","last_peered":"2023-10-16T12:10:43.793686+0000","last_clean":"2023-10-16T12:10:43.793686+0000","last_became_active":"2023-10-16T12:10:39.639250+0000","last_became_peered":"2023-10-16T12:10:39.639250+0000","last_unstale":"2023-10-16T12:10:43.793686+0000","last_undegraded":"2023-10-16T12:10:43.793686+0000","last_fullsized":"2023-10-16T12:10:43.793686+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:59:08.633021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.647118+0000","last_change":"2023-10-16T12:10:39.643535+0000","last_active":"2023-10-16T12:10:43.647118+0000","last_peered":"2023-10-16T12:10:43.647118+0000","last_clean":"2023-10-16T12:10:43.647118+0000","last_became_active":"2023-10-16T12:10:39.643242+0000","last_became_peered":"2023-10-16T12:10:39.643242+0000","last_unstale":"2023-10-16T12:10:43.647118+0000","last_undegraded":"2023-10-16T12:10:43.647118+0000","last_fullsized":"2023-10-16T12:10:43.647118+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:49:21.678699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.796912+0000","last_change":"2023-10-16T12:10:39.645759+0000","last_active":"2023-10-16T12:10:43.796912+0000","last_peered":"2023-10-16T12:10:43.796912+0000","last_clean":"2023-10-16T12:10:43.796912+0000","last_became_active":"2023-10-16T12:10:39.642863+0000","last_became_peered":"2023-10-16T12:10:39.642863+0000","last_unstale":"2023-10-16T12:10:43.796912+0000","last_undegraded":"2023-10-16T12:10:43.796912+0000","last_fullsized":"2023-10-16T12:10:43.796912+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:40:23.467426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.650642+0000","last_change":"2023-10-16T12:10:39.641392+0000","last_active":"2023-10-16T12:10:43.650642+0000","last_peered":"2023-10-16T12:10:43.650642+0000","last_clean":"2023-10-16T12:10:43.650642+0000","last_became_active":"2023-10-16T12:10:39.641111+0000","last_became_peered":"2023-10-16T12:10:39.641111+0000","last_unstale":"2023-10-16T12:10:43.650642+0000","last_undegraded":"2023-10-16T12:10:43.650642+0000","last_fullsized":"2023-10-16T12:10:43.650642+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:38:01.595985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.650275+0000","last_change":"2023-10-16T12:10:39.642376+0000","last_active":"2023-10-16T12:10:43.650275+0000","last_peered":"2023-10-16T12:10:43.650275+0000","last_clean":"2023-10-16T12:10:43.650275+0000","last_became_active":"2023-10-16T12:10:39.642086+0000","last_became_peered":"2023-10-16T12:10:39.642086+0000","last_unstale":"2023-10-16T12:10:43.650275+0000","last_undegraded":"2023-10-16T12:10:43.650275+0000","last_fullsized":"2023-10-16T12:10:43.650275+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:05:05.699350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.648829+0000","last_change":"2023-10-16T12:10:39.637561+0000","last_active":"2023-10-16T12:10:43.648829+0000","last_peered":"2023-10-16T12:10:43.648829+0000","last_clean":"2023-10-16T12:10:43.648829+0000","last_became_active":"2023-10-16T12:10:39.637109+0000","last_became_peered":"2023-10-16T12:10:39.637109+0000","last_unstale":"2023-10-16T12:10:43.648829+0000","last_undegraded":"2023-10-16T12:10:43.648829+0000","last_fullsized":"2023-10-16T12:10:43.648829+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:59:48.851909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.645820+0000","last_change":"2023-10-16T12:10:39.631791+0000","last_active":"2023-10-16T12:10:43.645820+0000","last_peered":"2023-10-16T12:10:43.645820+0000","last_clean":"2023-10-16T12:10:43.645820+0000","last_became_active":"2023-10-16T12:10:39.631245+0000","last_became_peered":"2023-10-16T12:10:39.631245+0000","last_unstale":"2023-10-16T12:10:43.645820+0000","last_undegraded":"2023-10-16T12:10:43.645820+0000","last_fullsized":"2023-10-16T12:10:43.645820+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:05:05.699350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646175+0000","last_change":"2023-10-16T12:10:39.631112+0000","last_active":"2023-10-16T12:10:43.646175+0000","last_peered":"2023-10-16T12:10:43.646175+0000","last_clean":"2023-10-16T12:10:43.646175+0000","last_became_active":"2023-10-16T12:10:39.630312+0000","last_became_peered":"2023-10-16T12:10:39.630312+0000","last_unstale":"2023-10-16T12:10:43.646175+0000","last_undegraded":"2023-10-16T12:10:43.646175+0000","last_fullsized":"2023-10-16T12:10:43.646175+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:04:48.768318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.653241+0000","last_change":"2023-10-16T12:10:39.642725+0000","last_active":"2023-10-16T12:10:43.653241+0000","last_peered":"2023-10-16T12:10:43.653241+0000","last_clean":"2023-10-16T12:10:43.653241+0000","last_became_active":"2023-10-16T12:10:39.642437+0000","last_became_peered":"2023-10-16T12:10:39.642437+0000","last_unstale":"2023-10-16T12:10:43.653241+0000","last_undegraded":"2023-10-16T12:10:43.653241+0000","last_fullsized":"2023-10-16T12:10:43.653241+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:46:02.593591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.652749+0000","last_change":"2023-10-16T12:10:39.641780+0000","last_active":"2023-10-16T12:10:43.652749+0000","last_peered":"2023-10-16T12:10:43.652749+0000","last_clean":"2023-10-16T12:10:43.652749+0000","last_became_active":"2023-10-16T12:10:39.641275+0000","last_became_peered":"2023-10-16T12:10:39.641275+0000","last_unstale":"2023-10-16T12:10:43.652749+0000","last_undegraded":"2023-10-16T12:10:43.652749+0000","last_fullsized":"2023-10-16T12:10:43.652749+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:38:01.595985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.647146+0000","last_change":"2023-10-16T12:10:39.641557+0000","last_active":"2023-10-16T12:10:43.647146+0000","last_peered":"2023-10-16T12:10:43.647146+0000","last_clean":"2023-10-16T12:10:43.647146+0000","last_became_active":"2023-10-16T12:10:39.641242+0000","last_became_peered":"2023-10-16T12:10:39.641242+0000","last_unstale":"2023-10-16T12:10:43.647146+0000","last_undegraded":"2023-10-16T12:10:43.647146+0000","last_fullsized":"2023-10-16T12:10:43.647146+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:10:19.840696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.642631+0000","last_change":"2023-10-16T12:10:39.638264+0000","last_active":"2023-10-16T12:10:43.642631+0000","last_peered":"2023-10-16T12:10:43.642631+0000","last_clean":"2023-10-16T12:10:43.642631+0000","last_became_active":"2023-10-16T12:10:39.637959+0000","last_became_peered":"2023-10-16T12:10:39.637959+0000","last_unstale":"2023-10-16T12:10:43.642631+0000","last_undegraded":"2023-10-16T12:10:43.642631+0000","last_fullsized":"2023-10-16T12:10:43.642631+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:27:06.186345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.796665+0000","last_change":"2023-10-16T12:10:39.646694+0000","last_active":"2023-10-16T12:10:43.796665+0000","last_peered":"2023-10-16T12:10:43.796665+0000","last_clean":"2023-10-16T12:10:43.796665+0000","last_became_active":"2023-10-16T12:10:39.643245+0000","last_became_peered":"2023-10-16T12:10:39.643245+0000","last_unstale":"2023-10-16T12:10:43.796665+0000","last_undegraded":"2023-10-16T12:10:43.796665+0000","last_fullsized":"2023-10-16T12:10:43.796665+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:13:03.744288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:44.684038+0000","last_change":"2023-10-16T12:10:43.800090+0000","last_active":"2023-10-16T12:10:44.684038+0000","last_peered":"2023-10-16T12:10:44.684038+0000","last_clean":"2023-10-16T12:10:44.684038+0000","last_became_active":"2023-10-16T12:10:43.799746+0000","last_became_peered":"2023-10-16T12:10:43.799746+0000","last_unstale":"2023-10-16T12:10:44.684038+0000","last_undegraded":"2023-10-16T12:10:44.684038+0000","last_fullsized":"2023-10-16T12:10:44.684038+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:46:18.137970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646491+0000","last_change":"2023-10-16T12:10:38.620314+0000","last_active":"2023-10-16T12:10:43.646491+0000","last_peered":"2023-10-16T12:10:43.646491+0000","last_clean":"2023-10-16T12:10:43.646491+0000","last_became_active":"2023-10-16T12:10:38.620036+0000","last_became_peered":"2023-10-16T12:10:38.620036+0000","last_unstale":"2023-10-16T12:10:43.646491+0000","last_undegraded":"2023-10-16T12:10:43.646491+0000","last_fullsized":"2023-10-16T12:10:43.646491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:27:30.532335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.645988+0000","last_change":"2023-10-16T12:10:39.638746+0000","last_active":"2023-10-16T12:10:43.645988+0000","last_peered":"2023-10-16T12:10:43.645988+0000","last_clean":"2023-10-16T12:10:43.645988+0000","last_became_active":"2023-10-16T12:10:39.637152+0000","last_became_peered":"2023-10-16T12:10:39.637152+0000","last_unstale":"2023-10-16T12:10:43.645988+0000","last_undegraded":"2023-10-16T12:10:43.645988+0000","last_fullsized":"2023-10-16T12:10:43.645988+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:30:10.006613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:45.375335+0000","last_change":"2023-10-16T12:10:43.662213+0000","last_active":"2023-10-16T12:10:45.375335+0000","last_peered":"2023-10-16T12:10:45.375335+0000","last_clean":"2023-10-16T12:10:45.375335+0000","last_became_active":"2023-10-16T12:10:43.661755+0000","last_became_peered":"2023-10-16T12:10:43.661755+0000","last_unstale":"2023-10-16T12:10:45.375335+0000","last_undegraded":"2023-10-16T12:10:45.375335+0000","last_fullsized":"2023-10-16T12:10:45.375335+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:30:14.045561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646110+0000","last_change":"2023-10-16T12:10:38.625571+0000","last_active":"2023-10-16T12:10:43.646110+0000","last_peered":"2023-10-16T12:10:43.646110+0000","last_clean":"2023-10-16T12:10:43.646110+0000","last_became_active":"2023-10-16T12:10:38.625353+0000","last_became_peered":"2023-10-16T12:10:38.625353+0000","last_unstale":"2023-10-16T12:10:43.646110+0000","last_undegraded":"2023-10-16T12:10:43.646110+0000","last_fullsized":"2023-10-16T12:10:43.646110+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:39:25.396697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646379+0000","last_change":"2023-10-16T12:10:39.641482+0000","last_active":"2023-10-16T12:10:43.646379+0000","last_peered":"2023-10-16T12:10:43.646379+0000","last_clean":"2023-10-16T12:10:43.646379+0000","last_became_active":"2023-10-16T12:10:39.641168+0000","last_became_peered":"2023-10-16T12:10:39.641168+0000","last_unstale":"2023-10-16T12:10:43.646379+0000","last_undegraded":"2023-10-16T12:10:43.646379+0000","last_fullsized":"2023-10-16T12:10:43.646379+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:47:52.332290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:45.376007+0000","last_change":"2023-10-16T12:10:43.668484+0000","last_active":"2023-10-16T12:10:45.376007+0000","last_peered":"2023-10-16T12:10:45.376007+0000","last_clean":"2023-10-16T12:10:45.376007+0000","last_became_active":"2023-10-16T12:10:43.667862+0000","last_became_peered":"2023-10-16T12:10:43.667862+0000","last_unstale":"2023-10-16T12:10:45.376007+0000","last_undegraded":"2023-10-16T12:10:45.376007+0000","last_fullsized":"2023-10-16T12:10:45.376007+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:34:54.098418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.796065+0000","last_change":"2023-10-16T12:10:38.623974+0000","last_active":"2023-10-16T12:10:43.796065+0000","last_peered":"2023-10-16T12:10:43.796065+0000","last_clean":"2023-10-16T12:10:43.796065+0000","last_became_active":"2023-10-16T12:10:38.623532+0000","last_became_peered":"2023-10-16T12:10:38.623532+0000","last_unstale":"2023-10-16T12:10:43.796065+0000","last_undegraded":"2023-10-16T12:10:43.796065+0000","last_fullsized":"2023-10-16T12:10:43.796065+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:03:11.321777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646640+0000","last_change":"2023-10-16T12:10:39.651203+0000","last_active":"2023-10-16T12:10:43.646640+0000","last_peered":"2023-10-16T12:10:43.646640+0000","last_clean":"2023-10-16T12:10:43.646640+0000","last_became_active":"2023-10-16T12:10:39.651001+0000","last_became_peered":"2023-10-16T12:10:39.651001+0000","last_unstale":"2023-10-16T12:10:43.646640+0000","last_undegraded":"2023-10-16T12:10:43.646640+0000","last_fullsized":"2023-10-16T12:10:43.646640+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:34:50.059469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:45.375823+0000","last_change":"2023-10-16T12:10:43.659820+0000","last_active":"2023-10-16T12:10:45.375823+0000","last_peered":"2023-10-16T12:10:45.375823+0000","last_clean":"2023-10-16T12:10:45.375823+0000","last_became_active":"2023-10-16T12:10:43.659301+0000","last_became_peered":"2023-10-16T12:10:43.659301+0000","last_unstale":"2023-10-16T12:10:45.375823+0000","last_undegraded":"2023-10-16T12:10:45.375823+0000","last_fullsized":"2023-10-16T12:10:45.375823+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:59:12.671969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.795902+0000","last_change":"2023-10-16T12:10:38.622949+0000","last_active":"2023-10-16T12:10:43.795902+0000","last_peered":"2023-10-16T12:10:43.795902+0000","last_clean":"2023-10-16T12:10:43.795902+0000","last_became_active":"2023-10-16T12:10:38.622666+0000","last_became_peered":"2023-10-16T12:10:38.622666+0000","last_unstale":"2023-10-16T12:10:43.795902+0000","last_undegraded":"2023-10-16T12:10:43.795902+0000","last_fullsized":"2023-10-16T12:10:43.795902+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:27:30.532335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.653416+0000","last_change":"2023-10-16T12:10:39.650483+0000","last_active":"2023-10-16T12:10:43.653416+0000","last_peered":"2023-10-16T12:10:43.653416+0000","last_clean":"2023-10-16T12:10:43.653416+0000","last_became_active":"2023-10-16T12:10:39.650230+0000","last_became_peered":"2023-10-16T12:10:39.650230+0000","last_unstale":"2023-10-16T12:10:43.653416+0000","last_undegraded":"2023-10-16T12:10:43.653416+0000","last_fullsized":"2023-10-16T12:10:43.653416+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:38:53.624950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:44.684035+0000","last_change":"2023-10-16T12:10:43.667121+0000","last_active":"2023-10-16T12:10:44.684035+0000","last_peered":"2023-10-16T12:10:44.684035+0000","last_clean":"2023-10-16T12:10:44.684035+0000","last_became_active":"2023-10-16T12:10:43.666708+0000","last_became_peered":"2023-10-16T12:10:43.666708+0000","last_unstale":"2023-10-16T12:10:44.684035+0000","last_undegraded":"2023-10-16T12:10:44.684035+0000","last_fullsized":"2023-10-16T12:10:44.684035+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:19:32.391115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.645944+0000","last_change":"2023-10-16T12:10:38.623302+0000","last_active":"2023-10-16T12:10:43.645944+0000","last_peered":"2023-10-16T12:10:43.645944+0000","last_clean":"2023-10-16T12:10:43.645944+0000","last_became_active":"2023-10-16T12:10:38.621867+0000","last_became_peered":"2023-10-16T12:10:38.621867+0000","last_unstale":"2023-10-16T12:10:43.645944+0000","last_undegraded":"2023-10-16T12:10:43.645944+0000","last_fullsized":"2023-10-16T12:10:43.645944+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:59:30.615293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.649101+0000","last_change":"2023-10-16T12:10:39.639254+0000","last_active":"2023-10-16T12:10:43.649101+0000","last_peered":"2023-10-16T12:10:43.649101+0000","last_clean":"2023-10-16T12:10:43.649101+0000","last_became_active":"2023-10-16T12:10:39.638956+0000","last_became_peered":"2023-10-16T12:10:39.638956+0000","last_unstale":"2023-10-16T12:10:43.649101+0000","last_undegraded":"2023-10-16T12:10:43.649101+0000","last_fullsized":"2023-10-16T12:10:43.649101+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:27:31.536464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:45.374724+0000","last_change":"2023-10-16T12:10:43.667499+0000","last_active":"2023-10-16T12:10:45.374724+0000","last_peered":"2023-10-16T12:10:45.374724+0000","last_clean":"2023-10-16T12:10:45.374724+0000","last_became_active":"2023-10-16T12:10:43.667116+0000","last_became_peered":"2023-10-16T12:10:43.667116+0000","last_unstale":"2023-10-16T12:10:45.374724+0000","last_undegraded":"2023-10-16T12:10:45.374724+0000","last_fullsized":"2023-10-16T12:10:45.374724+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:59:35.658370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.651387+0000","last_change":"2023-10-16T12:10:38.619514+0000","last_active":"2023-10-16T12:10:43.651387+0000","last_peered":"2023-10-16T12:10:43.651387+0000","last_clean":"2023-10-16T12:10:43.651387+0000","last_became_active":"2023-10-16T12:10:38.619191+0000","last_became_peered":"2023-10-16T12:10:38.619191+0000","last_unstale":"2023-10-16T12:10:43.651387+0000","last_undegraded":"2023-10-16T12:10:43.651387+0000","last_fullsized":"2023-10-16T12:10:43.651387+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:39:25.396697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.649794+0000","last_change":"2023-10-16T12:10:39.642796+0000","last_active":"2023-10-16T12:10:43.649794+0000","last_peered":"2023-10-16T12:10:43.649794+0000","last_clean":"2023-10-16T12:10:43.649794+0000","last_became_active":"2023-10-16T12:10:39.642536+0000","last_became_peered":"2023-10-16T12:10:39.642536+0000","last_unstale":"2023-10-16T12:10:43.649794+0000","last_undegraded":"2023-10-16T12:10:43.649794+0000","last_fullsized":"2023-10-16T12:10:43.649794+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:27:44.016376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:44.874851+0000","last_change":"2023-10-16T12:10:43.662296+0000","last_active":"2023-10-16T12:10:44.874851+0000","last_peered":"2023-10-16T12:10:44.874851+0000","last_clean":"2023-10-16T12:10:44.874851+0000","last_became_active":"2023-10-16T12:10:43.661893+0000","last_became_peered":"2023-10-16T12:10:43.661893+0000","last_unstale":"2023-10-16T12:10:44.874851+0000","last_undegraded":"2023-10-16T12:10:44.874851+0000","last_fullsized":"2023-10-16T12:10:44.874851+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:14:04.764499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646516+0000","last_change":"2023-10-16T12:10:38.623322+0000","last_active":"2023-10-16T12:10:43.646516+0000","last_peered":"2023-10-16T12:10:43.646516+0000","last_clean":"2023-10-16T12:10:43.646516+0000","last_became_active":"2023-10-16T12:10:38.622826+0000","last_became_peered":"2023-10-16T12:10:38.622826+0000","last_unstale":"2023-10-16T12:10:43.646516+0000","last_undegraded":"2023-10-16T12:10:43.646516+0000","last_fullsized":"2023-10-16T12:10:43.646516+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:30:56.479201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646312+0000","last_change":"2023-10-16T12:10:39.624731+0000","last_active":"2023-10-16T12:10:43.646312+0000","last_peered":"2023-10-16T12:10:43.646312+0000","last_clean":"2023-10-16T12:10:43.646312+0000","last_became_active":"2023-10-16T12:10:39.624399+0000","last_became_peered":"2023-10-16T12:10:39.624399+0000","last_unstale":"2023-10-16T12:10:43.646312+0000","last_undegraded":"2023-10-16T12:10:43.646312+0000","last_fullsized":"2023-10-16T12:10:43.646312+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:47:52.332290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:45.375461+0000","last_change":"2023-10-16T12:10:43.798638+0000","last_active":"2023-10-16T12:10:45.375461+0000","last_peered":"2023-10-16T12:10:45.375461+0000","last_clean":"2023-10-16T12:10:45.375461+0000","last_became_active":"2023-10-16T12:10:43.797973+0000","last_became_peered":"2023-10-16T12:10:43.797973+0000","last_unstale":"2023-10-16T12:10:45.375461+0000","last_undegraded":"2023-10-16T12:10:45.375461+0000","last_fullsized":"2023-10-16T12:10:45.375461+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:29:47.533439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.652773+0000","last_change":"2023-10-16T12:10:38.627818+0000","last_active":"2023-10-16T12:10:43.652773+0000","last_peered":"2023-10-16T12:10:43.652773+0000","last_clean":"2023-10-16T12:10:43.652773+0000","last_became_active":"2023-10-16T12:10:38.627492+0000","last_became_peered":"2023-10-16T12:10:38.627492+0000","last_unstale":"2023-10-16T12:10:43.652773+0000","last_undegraded":"2023-10-16T12:10:43.652773+0000","last_fullsized":"2023-10-16T12:10:43.652773+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:05:04.695222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.653456+0000","last_change":"2023-10-16T12:10:39.633123+0000","last_active":"2023-10-16T12:10:43.653456+0000","last_peered":"2023-10-16T12:10:43.653456+0000","last_clean":"2023-10-16T12:10:43.653456+0000","last_became_active":"2023-10-16T12:10:39.632868+0000","last_became_peered":"2023-10-16T12:10:39.632868+0000","last_unstale":"2023-10-16T12:10:43.653456+0000","last_undegraded":"2023-10-16T12:10:43.653456+0000","last_fullsized":"2023-10-16T12:10:43.653456+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:33:20.807621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:44.683898+0000","last_change":"2023-10-16T12:10:43.794748+0000","last_active":"2023-10-16T12:10:44.683898+0000","last_peered":"2023-10-16T12:10:44.683898+0000","last_clean":"2023-10-16T12:10:44.683898+0000","last_became_active":"2023-10-16T12:10:43.794362+0000","last_became_peered":"2023-10-16T12:10:43.794362+0000","last_unstale":"2023-10-16T12:10:44.683898+0000","last_undegraded":"2023-10-16T12:10:44.683898+0000","last_fullsized":"2023-10-16T12:10:44.683898+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:57:20.445348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.652120+0000","last_change":"2023-10-16T12:10:38.619590+0000","last_active":"2023-10-16T12:10:43.652120+0000","last_peered":"2023-10-16T12:10:43.652120+0000","last_clean":"2023-10-16T12:10:43.652120+0000","last_became_active":"2023-10-16T12:10:38.618984+0000","last_became_peered":"2023-10-16T12:10:38.618984+0000","last_unstale":"2023-10-16T12:10:43.652120+0000","last_undegraded":"2023-10-16T12:10:43.652120+0000","last_fullsized":"2023-10-16T12:10:43.652120+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:27:05.182216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646297+0000","last_change":"2023-10-16T12:10:39.644469+0000","last_active":"2023-10-16T12:10:43.646297+0000","last_peered":"2023-10-16T12:10:43.646297+0000","last_clean":"2023-10-16T12:10:43.646297+0000","last_became_active":"2023-10-16T12:10:39.642359+0000","last_became_peered":"2023-10-16T12:10:39.642359+0000","last_unstale":"2023-10-16T12:10:43.646297+0000","last_undegraded":"2023-10-16T12:10:43.646297+0000","last_fullsized":"2023-10-16T12:10:43.646297+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:30:10.006613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:44.684242+0000","last_change":"2023-10-16T12:10:43.667401+0000","last_active":"2023-10-16T12:10:44.684242+0000","last_peered":"2023-10-16T12:10:44.684242+0000","last_clean":"2023-10-16T12:10:44.684242+0000","last_became_active":"2023-10-16T12:10:43.666829+0000","last_became_peered":"2023-10-16T12:10:43.666829+0000","last_unstale":"2023-10-16T12:10:44.684242+0000","last_undegraded":"2023-10-16T12:10:44.684242+0000","last_fullsized":"2023-10-16T12:10:44.684242+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:13:48.034204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646585+0000","last_change":"2023-10-16T12:10:38.627372+0000","last_active":"2023-10-16T12:10:43.646585+0000","last_peered":"2023-10-16T12:10:43.646585+0000","last_clean":"2023-10-16T12:10:43.646585+0000","last_became_active":"2023-10-16T12:10:38.622500+0000","last_became_peered":"2023-10-16T12:10:38.622500+0000","last_unstale":"2023-10-16T12:10:43.646585+0000","last_undegraded":"2023-10-16T12:10:43.646585+0000","last_fullsized":"2023-10-16T12:10:43.646585+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:57:56.978587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646591+0000","last_change":"2023-10-16T12:10:39.641571+0000","last_active":"2023-10-16T12:10:43.646591+0000","last_peered":"2023-10-16T12:10:43.646591+0000","last_clean":"2023-10-16T12:10:43.646591+0000","last_became_active":"2023-10-16T12:10:39.641228+0000","last_became_peered":"2023-10-16T12:10:39.641228+0000","last_unstale":"2023-10-16T12:10:43.646591+0000","last_undegraded":"2023-10-16T12:10:43.646591+0000","last_fullsized":"2023-10-16T12:10:43.646591+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:38:01.595985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:45.376570+0000","last_change":"2023-10-16T12:10:43.671066+0000","last_active":"2023-10-16T12:10:45.376570+0000","last_peered":"2023-10-16T12:10:45.376570+0000","last_clean":"2023-10-16T12:10:45.376570+0000","last_became_active":"2023-10-16T12:10:43.670693+0000","last_became_peered":"2023-10-16T12:10:43.670693+0000","last_unstale":"2023-10-16T12:10:45.376570+0000","last_undegraded":"2023-10-16T12:10:45.376570+0000","last_fullsized":"2023-10-16T12:10:45.376570+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:24:19.577814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.654541+0000","last_change":"2023-10-16T12:10:38.623191+0000","last_active":"2023-10-16T12:10:43.654541+0000","last_peered":"2023-10-16T12:10:43.654541+0000","last_clean":"2023-10-16T12:10:43.654541+0000","last_became_active":"2023-10-16T12:10:38.622876+0000","last_became_peered":"2023-10-16T12:10:38.622876+0000","last_unstale":"2023-10-16T12:10:43.654541+0000","last_undegraded":"2023-10-16T12:10:43.654541+0000","last_fullsized":"2023-10-16T12:10:43.654541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:59:30.615293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.648686+0000","last_change":"2023-10-16T12:10:39.643461+0000","last_active":"2023-10-16T12:10:43.648686+0000","last_peered":"2023-10-16T12:10:43.648686+0000","last_clean":"2023-10-16T12:10:43.648686+0000","last_became_active":"2023-10-16T12:10:39.643148+0000","last_became_peered":"2023-10-16T12:10:39.643148+0000","last_unstale":"2023-10-16T12:10:43.648686+0000","last_undegraded":"2023-10-16T12:10:43.648686+0000","last_fullsized":"2023-10-16T12:10:43.648686+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:27:06.186345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:44.682345+0000","last_change":"2023-10-16T12:10:43.662674+0000","last_active":"2023-10-16T12:10:44.682345+0000","last_peered":"2023-10-16T12:10:44.682345+0000","last_clean":"2023-10-16T12:10:44.682345+0000","last_became_active":"2023-10-16T12:10:43.662249+0000","last_became_peered":"2023-10-16T12:10:43.662249+0000","last_unstale":"2023-10-16T12:10:44.682345+0000","last_undegraded":"2023-10-16T12:10:44.682345+0000","last_fullsized":"2023-10-16T12:10:44.682345+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:14:04.764499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"25'9","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:49.578853+0000","last_change":"2023-10-16T12:10:38.621924+0000","last_active":"2023-10-16T12:10:49.578853+0000","last_peered":"2023-10-16T12:10:49.578853+0000","last_clean":"2023-10-16T12:10:49.578853+0000","last_became_active":"2023-10-16T12:10:38.621621+0000","last_became_peered":"2023-10-16T12:10:38.621621+0000","last_unstale":"2023-10-16T12:10:49.578853+0000","last_undegraded":"2023-10-16T12:10:49.578853+0000","last_fullsized":"2023-10-16T12:10:49.578853+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:27:30.532335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.796277+0000","last_change":"2023-10-16T12:10:39.641165+0000","last_active":"2023-10-16T12:10:43.796277+0000","last_peered":"2023-10-16T12:10:43.796277+0000","last_clean":"2023-10-16T12:10:43.796277+0000","last_became_active":"2023-10-16T12:10:39.640241+0000","last_became_peered":"2023-10-16T12:10:39.640241+0000","last_unstale":"2023-10-16T12:10:43.796277+0000","last_undegraded":"2023-10-16T12:10:43.796277+0000","last_fullsized":"2023-10-16T12:10:43.796277+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:38:53.624950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:45.377704+0000","last_change":"2023-10-16T12:10:43.796949+0000","last_active":"2023-10-16T12:10:45.377704+0000","last_peered":"2023-10-16T12:10:45.377704+0000","last_clean":"2023-10-16T12:10:45.377704+0000","last_became_active":"2023-10-16T12:10:43.796518+0000","last_became_peered":"2023-10-16T12:10:43.796518+0000","last_unstale":"2023-10-16T12:10:45.377704+0000","last_undegraded":"2023-10-16T12:10:45.377704+0000","last_fullsized":"2023-10-16T12:10:45.377704+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:58:35.336755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.650051+0000","last_change":"2023-10-16T12:10:38.612552+0000","last_active":"2023-10-16T12:10:43.650051+0000","last_peered":"2023-10-16T12:10:43.650051+0000","last_clean":"2023-10-16T12:10:43.650051+0000","last_became_active":"2023-10-16T12:10:38.612221+0000","last_became_peered":"2023-10-16T12:10:38.612221+0000","last_unstale":"2023-10-16T12:10:43.650051+0000","last_undegraded":"2023-10-16T12:10:43.650051+0000","last_fullsized":"2023-10-16T12:10:43.650051+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:27:05.182216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646287+0000","last_change":"2023-10-16T12:10:39.638781+0000","last_active":"2023-10-16T12:10:43.646287+0000","last_peered":"2023-10-16T12:10:43.646287+0000","last_clean":"2023-10-16T12:10:43.646287+0000","last_became_active":"2023-10-16T12:10:39.638523+0000","last_became_peered":"2023-10-16T12:10:39.638523+0000","last_unstale":"2023-10-16T12:10:43.646287+0000","last_undegraded":"2023-10-16T12:10:43.646287+0000","last_fullsized":"2023-10-16T12:10:43.646287+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:05:05.699350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:44.683241+0000","last_change":"2023-10-16T12:10:43.794196+0000","last_active":"2023-10-16T12:10:44.683241+0000","last_peered":"2023-10-16T12:10:44.683241+0000","last_clean":"2023-10-16T12:10:44.683241+0000","last_became_active":"2023-10-16T12:10:43.793712+0000","last_became_peered":"2023-10-16T12:10:43.793712+0000","last_unstale":"2023-10-16T12:10:44.683241+0000","last_undegraded":"2023-10-16T12:10:44.683241+0000","last_fullsized":"2023-10-16T12:10:44.683241+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:40:27.506374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646125+0000","last_change":"2023-10-16T12:10:38.627403+0000","last_active":"2023-10-16T12:10:43.646125+0000","last_peered":"2023-10-16T12:10:43.646125+0000","last_clean":"2023-10-16T12:10:43.646125+0000","last_became_active":"2023-10-16T12:10:38.623013+0000","last_became_peered":"2023-10-16T12:10:38.623013+0000","last_unstale":"2023-10-16T12:10:43.646125+0000","last_undegraded":"2023-10-16T12:10:43.646125+0000","last_fullsized":"2023-10-16T12:10:43.646125+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:47:51.328162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.794292+0000","last_change":"2023-10-16T12:10:39.640179+0000","last_active":"2023-10-16T12:10:43.794292+0000","last_peered":"2023-10-16T12:10:43.794292+0000","last_clean":"2023-10-16T12:10:43.794292+0000","last_became_active":"2023-10-16T12:10:39.639843+0000","last_became_peered":"2023-10-16T12:10:39.639843+0000","last_unstale":"2023-10-16T12:10:43.794292+0000","last_undegraded":"2023-10-16T12:10:43.794292+0000","last_fullsized":"2023-10-16T12:10:43.794292+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:24:15.538866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:44.683431+0000","last_change":"2023-10-16T12:10:43.796968+0000","last_active":"2023-10-16T12:10:44.683431+0000","last_peered":"2023-10-16T12:10:44.683431+0000","last_clean":"2023-10-16T12:10:44.683431+0000","last_became_active":"2023-10-16T12:10:43.796562+0000","last_became_peered":"2023-10-16T12:10:43.796562+0000","last_unstale":"2023-10-16T12:10:44.683431+0000","last_undegraded":"2023-10-16T12:10:44.683431+0000","last_fullsized":"2023-10-16T12:10:44.683431+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:44:47.973077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.650016+0000","last_change":"2023-10-16T12:10:38.622066+0000","last_active":"2023-10-16T12:10:43.650016+0000","last_peered":"2023-10-16T12:10:43.650016+0000","last_clean":"2023-10-16T12:10:43.650016+0000","last_became_active":"2023-10-16T12:10:38.621453+0000","last_became_peered":"2023-10-16T12:10:38.621453+0000","last_unstale":"2023-10-16T12:10:43.650016+0000","last_undegraded":"2023-10-16T12:10:43.650016+0000","last_fullsized":"2023-10-16T12:10:43.650016+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:45:22.011420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.645479+0000","last_change":"2023-10-16T12:10:39.636432+0000","last_active":"2023-10-16T12:10:43.645479+0000","last_peered":"2023-10-16T12:10:43.645479+0000","last_clean":"2023-10-16T12:10:43.645479+0000","last_became_active":"2023-10-16T12:10:39.636136+0000","last_became_peered":"2023-10-16T12:10:39.636136+0000","last_unstale":"2023-10-16T12:10:43.645479+0000","last_undegraded":"2023-10-16T12:10:43.645479+0000","last_fullsized":"2023-10-16T12:10:43.645479+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:46:02.593591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:45.375145+0000","last_change":"2023-10-16T12:10:43.661450+0000","last_active":"2023-10-16T12:10:45.375145+0000","last_peered":"2023-10-16T12:10:45.375145+0000","last_clean":"2023-10-16T12:10:45.375145+0000","last_became_active":"2023-10-16T12:10:43.660931+0000","last_became_peered":"2023-10-16T12:10:43.660931+0000","last_unstale":"2023-10-16T12:10:45.375145+0000","last_undegraded":"2023-10-16T12:10:45.375145+0000","last_fullsized":"2023-10-16T12:10:45.375145+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:53:08.121149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.642296+0000","last_change":"2023-10-16T12:10:38.626933+0000","last_active":"2023-10-16T12:10:43.642296+0000","last_peered":"2023-10-16T12:10:43.642296+0000","last_clean":"2023-10-16T12:10:43.642296+0000","last_became_active":"2023-10-16T12:10:38.626643+0000","last_became_peered":"2023-10-16T12:10:38.626643+0000","last_unstale":"2023-10-16T12:10:43.642296+0000","last_undegraded":"2023-10-16T12:10:43.642296+0000","last_fullsized":"2023-10-16T12:10:43.642296+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:01:27.462901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.794002+0000","last_change":"2023-10-16T12:10:39.648098+0000","last_active":"2023-10-16T12:10:43.794002+0000","last_peered":"2023-10-16T12:10:43.794002+0000","last_clean":"2023-10-16T12:10:43.794002+0000","last_became_active":"2023-10-16T12:10:39.643195+0000","last_became_peered":"2023-10-16T12:10:39.643195+0000","last_unstale":"2023-10-16T12:10:43.794002+0000","last_undegraded":"2023-10-16T12:10:43.794002+0000","last_fullsized":"2023-10-16T12:10:43.794002+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:34:50.059469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:45.374887+0000","last_change":"2023-10-16T12:10:43.662584+0000","last_active":"2023-10-16T12:10:45.374887+0000","last_peered":"2023-10-16T12:10:45.374887+0000","last_clean":"2023-10-16T12:10:45.374887+0000","last_became_active":"2023-10-16T12:10:43.662164+0000","last_became_peered":"2023-10-16T12:10:43.662164+0000","last_unstale":"2023-10-16T12:10:45.374887+0000","last_undegraded":"2023-10-16T12:10:45.374887+0000","last_fullsized":"2023-10-16T12:10:45.374887+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:03:16.364854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.648873+0000","last_change":"2023-10-16T12:10:38.613102+0000","last_active":"2023-10-16T12:10:43.648873+0000","last_peered":"2023-10-16T12:10:43.648873+0000","last_clean":"2023-10-16T12:10:43.648873+0000","last_became_active":"2023-10-16T12:10:38.612596+0000","last_became_peered":"2023-10-16T12:10:38.612596+0000","last_unstale":"2023-10-16T12:10:43.648873+0000","last_undegraded":"2023-10-16T12:10:43.648873+0000","last_fullsized":"2023-10-16T12:10:43.648873+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:59:50.957247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.796815+0000","last_change":"2023-10-16T12:10:39.640141+0000","last_active":"2023-10-16T12:10:43.796815+0000","last_peered":"2023-10-16T12:10:43.796815+0000","last_clean":"2023-10-16T12:10:43.796815+0000","last_became_active":"2023-10-16T12:10:39.639770+0000","last_became_peered":"2023-10-16T12:10:39.639770+0000","last_unstale":"2023-10-16T12:10:43.796815+0000","last_undegraded":"2023-10-16T12:10:43.796815+0000","last_fullsized":"2023-10-16T12:10:43.796815+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:14:00.725551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:44.683486+0000","last_change":"2023-10-16T12:10:43.666858+0000","last_active":"2023-10-16T12:10:44.683486+0000","last_peered":"2023-10-16T12:10:44.683486+0000","last_clean":"2023-10-16T12:10:44.683486+0000","last_became_active":"2023-10-16T12:10:43.666461+0000","last_became_peered":"2023-10-16T12:10:43.666461+0000","last_unstale":"2023-10-16T12:10:44.683486+0000","last_undegraded":"2023-10-16T12:10:44.683486+0000","last_fullsized":"2023-10-16T12:10:44.683486+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:53:08.121149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.663314+0000","last_change":"2023-10-16T12:10:38.625538+0000","last_active":"2023-10-16T12:10:43.663314+0000","last_peered":"2023-10-16T12:10:43.663314+0000","last_clean":"2023-10-16T12:10:43.663314+0000","last_became_active":"2023-10-16T12:10:38.625227+0000","last_became_peered":"2023-10-16T12:10:38.625227+0000","last_unstale":"2023-10-16T12:10:43.663314+0000","last_undegraded":"2023-10-16T12:10:43.663314+0000","last_fullsized":"2023-10-16T12:10:43.663314+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:39:25.396697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.653596+0000","last_change":"2023-10-16T12:10:39.645373+0000","last_active":"2023-10-16T12:10:43.653596+0000","last_peered":"2023-10-16T12:10:43.653596+0000","last_clean":"2023-10-16T12:10:43.653596+0000","last_became_active":"2023-10-16T12:10:39.645083+0000","last_became_peered":"2023-10-16T12:10:39.645083+0000","last_unstale":"2023-10-16T12:10:43.653596+0000","last_undegraded":"2023-10-16T12:10:43.653596+0000","last_fullsized":"2023-10-16T12:10:43.653596+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:53:04.082200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:44.683426+0000","last_change":"2023-10-16T12:10:43.673415+0000","last_active":"2023-10-16T12:10:44.683426+0000","last_peered":"2023-10-16T12:10:44.683426+0000","last_clean":"2023-10-16T12:10:44.683426+0000","last_became_active":"2023-10-16T12:10:43.672874+0000","last_became_peered":"2023-10-16T12:10:43.672874+0000","last_unstale":"2023-10-16T12:10:44.683426+0000","last_undegraded":"2023-10-16T12:10:44.683426+0000","last_fullsized":"2023-10-16T12:10:44.683426+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:38:57.663899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646014+0000","last_change":"2023-10-16T12:10:38.628007+0000","last_active":"2023-10-16T12:10:43.646014+0000","last_peered":"2023-10-16T12:10:43.646014+0000","last_clean":"2023-10-16T12:10:43.646014+0000","last_became_active":"2023-10-16T12:10:38.627760+0000","last_became_peered":"2023-10-16T12:10:38.627760+0000","last_unstale":"2023-10-16T12:10:43.646014+0000","last_undegraded":"2023-10-16T12:10:43.646014+0000","last_fullsized":"2023-10-16T12:10:43.646014+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:46:01.589462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.652978+0000","last_change":"2023-10-16T12:10:39.632645+0000","last_active":"2023-10-16T12:10:43.652978+0000","last_peered":"2023-10-16T12:10:43.652978+0000","last_clean":"2023-10-16T12:10:43.652978+0000","last_became_active":"2023-10-16T12:10:39.632345+0000","last_became_peered":"2023-10-16T12:10:39.632345+0000","last_unstale":"2023-10-16T12:10:43.652978+0000","last_undegraded":"2023-10-16T12:10:43.652978+0000","last_fullsized":"2023-10-16T12:10:43.652978+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:58:21.398705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:44.683787+0000","last_change":"2023-10-16T12:10:43.667036+0000","last_active":"2023-10-16T12:10:44.683787+0000","last_peered":"2023-10-16T12:10:44.683787+0000","last_clean":"2023-10-16T12:10:44.683787+0000","last_became_active":"2023-10-16T12:10:43.666639+0000","last_became_peered":"2023-10-16T12:10:43.666639+0000","last_unstale":"2023-10-16T12:10:44.683787+0000","last_undegraded":"2023-10-16T12:10:44.683787+0000","last_fullsized":"2023-10-16T12:10:44.683787+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:29:47.533439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.657522+0000","last_change":"2023-10-16T12:10:39.633035+0000","last_active":"2023-10-16T12:10:43.657522+0000","last_peered":"2023-10-16T12:10:43.657522+0000","last_clean":"2023-10-16T12:10:43.657522+0000","last_became_active":"2023-10-16T12:10:39.631630+0000","last_became_peered":"2023-10-16T12:10:39.631630+0000","last_unstale":"2023-10-16T12:10:43.657522+0000","last_undegraded":"2023-10-16T12:10:43.657522+0000","last_fullsized":"2023-10-16T12:10:43.657522+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:47:52.332290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.642429+0000","last_change":"2023-10-16T12:10:38.622647+0000","last_active":"2023-10-16T12:10:43.642429+0000","last_peered":"2023-10-16T12:10:43.642429+0000","last_clean":"2023-10-16T12:10:43.642429+0000","last_became_active":"2023-10-16T12:10:38.622323+0000","last_became_peered":"2023-10-16T12:10:38.622323+0000","last_unstale":"2023-10-16T12:10:43.642429+0000","last_undegraded":"2023-10-16T12:10:43.642429+0000","last_fullsized":"2023-10-16T12:10:43.642429+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:27:30.532335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:44.874638+0000","last_change":"2023-10-16T12:10:43.797789+0000","last_active":"2023-10-16T12:10:44.874638+0000","last_peered":"2023-10-16T12:10:44.874638+0000","last_clean":"2023-10-16T12:10:44.874638+0000","last_became_active":"2023-10-16T12:10:43.797367+0000","last_became_peered":"2023-10-16T12:10:43.797367+0000","last_unstale":"2023-10-16T12:10:44.874638+0000","last_undegraded":"2023-10-16T12:10:44.874638+0000","last_fullsized":"2023-10-16T12:10:44.874638+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:24:19.577814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646387+0000","last_change":"2023-10-16T12:10:39.650651+0000","last_active":"2023-10-16T12:10:43.646387+0000","last_peered":"2023-10-16T12:10:43.646387+0000","last_clean":"2023-10-16T12:10:43.646387+0000","last_became_active":"2023-10-16T12:10:39.650279+0000","last_became_peered":"2023-10-16T12:10:39.650279+0000","last_unstale":"2023-10-16T12:10:43.646387+0000","last_undegraded":"2023-10-16T12:10:43.646387+0000","last_fullsized":"2023-10-16T12:10:43.646387+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:03:12.325906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646002+0000","last_change":"2023-10-16T12:10:38.622760+0000","last_active":"2023-10-16T12:10:43.646002+0000","last_peered":"2023-10-16T12:10:43.646002+0000","last_clean":"2023-10-16T12:10:43.646002+0000","last_became_active":"2023-10-16T12:10:38.622408+0000","last_became_peered":"2023-10-16T12:10:38.622408+0000","last_unstale":"2023-10-16T12:10:43.646002+0000","last_undegraded":"2023-10-16T12:10:43.646002+0000","last_fullsized":"2023-10-16T12:10:43.646002+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:52:07.438053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":169,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.793810+0000","last_change":"2023-10-16T12:10:34.746820+0000","last_active":"2023-10-16T12:10:43.793810+0000","last_peered":"2023-10-16T12:10:43.793810+0000","last_clean":"2023-10-16T12:10:43.793810+0000","last_became_active":"2023-10-16T12:10:34.745574+0000","last_became_peered":"2023-10-16T12:10:34.745574+0000","last_unstale":"2023-10-16T12:10:43.793810+0000","last_undegraded":"2023-10-16T12:10:43.793810+0000","last_fullsized":"2023-10-16T12:10:43.793810+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_clean_scrub_stamp":"2023-10-16T12:10:33.577111+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:14.461986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:45.375419+0000","last_change":"2023-10-16T12:10:43.798352+0000","last_active":"2023-10-16T12:10:45.375419+0000","last_peered":"2023-10-16T12:10:45.375419+0000","last_clean":"2023-10-16T12:10:45.375419+0000","last_became_active":"2023-10-16T12:10:43.797410+0000","last_became_peered":"2023-10-16T12:10:43.797410+0000","last_unstale":"2023-10-16T12:10:45.375419+0000","last_undegraded":"2023-10-16T12:10:45.375419+0000","last_fullsized":"2023-10-16T12:10:45.375419+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:03:18.128385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646524+0000","last_change":"2023-10-16T12:10:39.635017+0000","last_active":"2023-10-16T12:10:43.646524+0000","last_peered":"2023-10-16T12:10:43.646524+0000","last_clean":"2023-10-16T12:10:43.646524+0000","last_became_active":"2023-10-16T12:10:39.634564+0000","last_became_peered":"2023-10-16T12:10:39.634564+0000","last_unstale":"2023-10-16T12:10:43.646524+0000","last_undegraded":"2023-10-16T12:10:43.646524+0000","last_fullsized":"2023-10-16T12:10:43.646524+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:30:57.483330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.793813+0000","last_change":"2023-10-16T12:10:38.626904+0000","last_active":"2023-10-16T12:10:43.793813+0000","last_peered":"2023-10-16T12:10:43.793813+0000","last_clean":"2023-10-16T12:10:43.793813+0000","last_became_active":"2023-10-16T12:10:38.626683+0000","last_became_peered":"2023-10-16T12:10:38.626683+0000","last_unstale":"2023-10-16T12:10:43.793813+0000","last_undegraded":"2023-10-16T12:10:43.793813+0000","last_fullsized":"2023-10-16T12:10:43.793813+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:58:20.394576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:45.375040+0000","last_change":"2023-10-16T12:10:43.794928+0000","last_active":"2023-10-16T12:10:45.375040+0000","last_peered":"2023-10-16T12:10:45.375040+0000","last_clean":"2023-10-16T12:10:45.375040+0000","last_became_active":"2023-10-16T12:10:43.794276+0000","last_became_peered":"2023-10-16T12:10:43.794276+0000","last_unstale":"2023-10-16T12:10:45.375040+0000","last_undegraded":"2023-10-16T12:10:45.375040+0000","last_fullsized":"2023-10-16T12:10:45.375040+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:38:57.663899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.795963+0000","last_change":"2023-10-16T12:10:38.620637+0000","last_active":"2023-10-16T12:10:43.795963+0000","last_peered":"2023-10-16T12:10:43.795963+0000","last_clean":"2023-10-16T12:10:43.795963+0000","last_became_active":"2023-10-16T12:10:38.620275+0000","last_became_peered":"2023-10-16T12:10:38.620275+0000","last_unstale":"2023-10-16T12:10:43.795963+0000","last_undegraded":"2023-10-16T12:10:43.795963+0000","last_fullsized":"2023-10-16T12:10:43.795963+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:38:00.591856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.663386+0000","last_change":"2023-10-16T12:10:39.632571+0000","last_active":"2023-10-16T12:10:43.663386+0000","last_peered":"2023-10-16T12:10:43.663386+0000","last_clean":"2023-10-16T12:10:43.663386+0000","last_became_active":"2023-10-16T12:10:39.631893+0000","last_became_peered":"2023-10-16T12:10:39.631893+0000","last_unstale":"2023-10-16T12:10:43.663386+0000","last_undegraded":"2023-10-16T12:10:43.663386+0000","last_fullsized":"2023-10-16T12:10:43.663386+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:05:05.699350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:44.683543+0000","last_change":"2023-10-16T12:10:43.797399+0000","last_active":"2023-10-16T12:10:44.683543+0000","last_peered":"2023-10-16T12:10:44.683543+0000","last_clean":"2023-10-16T12:10:44.683543+0000","last_became_active":"2023-10-16T12:10:43.796570+0000","last_became_peered":"2023-10-16T12:10:43.796570+0000","last_unstale":"2023-10-16T12:10:44.683543+0000","last_undegraded":"2023-10-16T12:10:44.683543+0000","last_fullsized":"2023-10-16T12:10:44.683543+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:58:35.336755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.656426+0000","last_change":"2023-10-16T12:10:38.620866+0000","last_active":"2023-10-16T12:10:43.656426+0000","last_peered":"2023-10-16T12:10:43.656426+0000","last_clean":"2023-10-16T12:10:43.656426+0000","last_became_active":"2023-10-16T12:10:38.620517+0000","last_became_peered":"2023-10-16T12:10:38.620517+0000","last_unstale":"2023-10-16T12:10:43.656426+0000","last_undegraded":"2023-10-16T12:10:43.656426+0000","last_fullsized":"2023-10-16T12:10:43.656426+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:58:20.394576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646193+0000","last_change":"2023-10-16T12:10:39.639395+0000","last_active":"2023-10-16T12:10:43.646193+0000","last_peered":"2023-10-16T12:10:43.646193+0000","last_clean":"2023-10-16T12:10:43.646193+0000","last_became_active":"2023-10-16T12:10:39.639030+0000","last_became_peered":"2023-10-16T12:10:39.639030+0000","last_unstale":"2023-10-16T12:10:43.646193+0000","last_undegraded":"2023-10-16T12:10:43.646193+0000","last_fullsized":"2023-10-16T12:10:43.646193+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:34:50.059469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:45.375076+0000","last_change":"2023-10-16T12:10:43.800001+0000","last_active":"2023-10-16T12:10:45.375076+0000","last_peered":"2023-10-16T12:10:45.375076+0000","last_clean":"2023-10-16T12:10:45.375076+0000","last_became_active":"2023-10-16T12:10:43.799598+0000","last_became_peered":"2023-10-16T12:10:43.799598+0000","last_unstale":"2023-10-16T12:10:45.375076+0000","last_undegraded":"2023-10-16T12:10:45.375076+0000","last_fullsized":"2023-10-16T12:10:45.375076+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:27:48.055325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.652023+0000","last_change":"2023-10-16T12:10:38.619506+0000","last_active":"2023-10-16T12:10:43.652023+0000","last_peered":"2023-10-16T12:10:43.652023+0000","last_clean":"2023-10-16T12:10:43.652023+0000","last_became_active":"2023-10-16T12:10:38.619203+0000","last_became_peered":"2023-10-16T12:10:38.619203+0000","last_unstale":"2023-10-16T12:10:43.652023+0000","last_undegraded":"2023-10-16T12:10:43.652023+0000","last_fullsized":"2023-10-16T12:10:43.652023+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:58:20.394576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.648502+0000","last_change":"2023-10-16T12:10:39.630721+0000","last_active":"2023-10-16T12:10:43.648502+0000","last_peered":"2023-10-16T12:10:43.648502+0000","last_clean":"2023-10-16T12:10:43.648502+0000","last_became_active":"2023-10-16T12:10:39.630448+0000","last_became_peered":"2023-10-16T12:10:39.630448+0000","last_unstale":"2023-10-16T12:10:43.648502+0000","last_undegraded":"2023-10-16T12:10:43.648502+0000","last_fullsized":"2023-10-16T12:10:43.648502+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:03:12.325906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:45.376403+0000","last_change":"2023-10-16T12:10:43.662711+0000","last_active":"2023-10-16T12:10:45.376403+0000","last_peered":"2023-10-16T12:10:45.376403+0000","last_clean":"2023-10-16T12:10:45.376403+0000","last_became_active":"2023-10-16T12:10:43.661726+0000","last_became_peered":"2023-10-16T12:10:43.661726+0000","last_unstale":"2023-10-16T12:10:45.376403+0000","last_undegraded":"2023-10-16T12:10:45.376403+0000","last_fullsized":"2023-10-16T12:10:45.376403+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:13:07.783237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.793165+0000","last_change":"2023-10-16T12:10:38.620553+0000","last_active":"2023-10-16T12:10:43.793165+0000","last_peered":"2023-10-16T12:10:43.793165+0000","last_clean":"2023-10-16T12:10:43.793165+0000","last_became_active":"2023-10-16T12:10:38.620248+0000","last_became_peered":"2023-10-16T12:10:38.620248+0000","last_unstale":"2023-10-16T12:10:43.793165+0000","last_undegraded":"2023-10-16T12:10:43.793165+0000","last_fullsized":"2023-10-16T12:10:43.793165+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:45:22.011420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.647607+0000","last_change":"2023-10-16T12:10:39.631773+0000","last_active":"2023-10-16T12:10:43.647607+0000","last_peered":"2023-10-16T12:10:43.647607+0000","last_clean":"2023-10-16T12:10:43.647607+0000","last_became_active":"2023-10-16T12:10:39.631418+0000","last_became_peered":"2023-10-16T12:10:39.631418+0000","last_unstale":"2023-10-16T12:10:43.647607+0000","last_undegraded":"2023-10-16T12:10:43.647607+0000","last_fullsized":"2023-10-16T12:10:43.647607+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:45:23.015548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:44.683656+0000","last_change":"2023-10-16T12:10:43.794614+0000","last_active":"2023-10-16T12:10:44.683656+0000","last_peered":"2023-10-16T12:10:44.683656+0000","last_clean":"2023-10-16T12:10:44.683656+0000","last_became_active":"2023-10-16T12:10:43.794206+0000","last_became_peered":"2023-10-16T12:10:43.794206+0000","last_unstale":"2023-10-16T12:10:44.683656+0000","last_undegraded":"2023-10-16T12:10:44.683656+0000","last_fullsized":"2023-10-16T12:10:44.683656+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:03:18.128385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.653349+0000","last_change":"2023-10-16T12:10:38.621473+0000","last_active":"2023-10-16T12:10:43.653349+0000","last_peered":"2023-10-16T12:10:43.653349+0000","last_clean":"2023-10-16T12:10:43.653349+0000","last_became_active":"2023-10-16T12:10:38.619100+0000","last_became_peered":"2023-10-16T12:10:38.619100+0000","last_unstale":"2023-10-16T12:10:43.653349+0000","last_undegraded":"2023-10-16T12:10:43.653349+0000","last_fullsized":"2023-10-16T12:10:43.653349+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:04:47.764190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.647008+0000","last_change":"2023-10-16T12:10:39.630605+0000","last_active":"2023-10-16T12:10:43.647008+0000","last_peered":"2023-10-16T12:10:43.647008+0000","last_clean":"2023-10-16T12:10:43.647008+0000","last_became_active":"2023-10-16T12:10:39.630234+0000","last_became_peered":"2023-10-16T12:10:39.630234+0000","last_unstale":"2023-10-16T12:10:43.647008+0000","last_undegraded":"2023-10-16T12:10:43.647008+0000","last_fullsized":"2023-10-16T12:10:43.647008+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:33:20.807621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:45.376060+0000","last_change":"2023-10-16T12:10:43.665913+0000","last_active":"2023-10-16T12:10:45.376060+0000","last_peered":"2023-10-16T12:10:45.376060+0000","last_clean":"2023-10-16T12:10:45.376060+0000","last_became_active":"2023-10-16T12:10:43.665498+0000","last_became_peered":"2023-10-16T12:10:43.665498+0000","last_unstale":"2023-10-16T12:10:45.376060+0000","last_undegraded":"2023-10-16T12:10:45.376060+0000","last_fullsized":"2023-10-16T12:10:45.376060+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:10:23.879645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.652500+0000","last_change":"2023-10-16T12:10:38.623243+0000","last_active":"2023-10-16T12:10:43.652500+0000","last_peered":"2023-10-16T12:10:43.652500+0000","last_clean":"2023-10-16T12:10:43.652500+0000","last_became_active":"2023-10-16T12:10:38.622935+0000","last_became_peered":"2023-10-16T12:10:38.622935+0000","last_unstale":"2023-10-16T12:10:43.652500+0000","last_undegraded":"2023-10-16T12:10:43.652500+0000","last_fullsized":"2023-10-16T12:10:43.652500+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:27:30.532335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.645994+0000","last_change":"2023-10-16T12:10:39.624625+0000","last_active":"2023-10-16T12:10:43.645994+0000","last_peered":"2023-10-16T12:10:43.645994+0000","last_clean":"2023-10-16T12:10:43.645994+0000","last_became_active":"2023-10-16T12:10:39.624281+0000","last_became_peered":"2023-10-16T12:10:39.624281+0000","last_unstale":"2023-10-16T12:10:43.645994+0000","last_undegraded":"2023-10-16T12:10:43.645994+0000","last_fullsized":"2023-10-16T12:10:43.645994+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:27:44.016376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:45.377425+0000","last_change":"2023-10-16T12:10:43.662337+0000","last_active":"2023-10-16T12:10:45.377425+0000","last_peered":"2023-10-16T12:10:45.377425+0000","last_clean":"2023-10-16T12:10:45.377425+0000","last_became_active":"2023-10-16T12:10:43.661865+0000","last_became_peered":"2023-10-16T12:10:43.661865+0000","last_unstale":"2023-10-16T12:10:45.377425+0000","last_undegraded":"2023-10-16T12:10:45.377425+0000","last_fullsized":"2023-10-16T12:10:45.377425+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:14:04.764499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.794403+0000","last_change":"2023-10-16T12:10:38.624192+0000","last_active":"2023-10-16T12:10:43.794403+0000","last_peered":"2023-10-16T12:10:43.794403+0000","last_clean":"2023-10-16T12:10:43.794403+0000","last_became_active":"2023-10-16T12:10:38.623767+0000","last_became_peered":"2023-10-16T12:10:38.623767+0000","last_unstale":"2023-10-16T12:10:43.794403+0000","last_undegraded":"2023-10-16T12:10:43.794403+0000","last_fullsized":"2023-10-16T12:10:43.794403+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:59:30.615293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.651482+0000","last_change":"2023-10-16T12:10:39.645023+0000","last_active":"2023-10-16T12:10:43.651482+0000","last_peered":"2023-10-16T12:10:43.651482+0000","last_clean":"2023-10-16T12:10:43.651482+0000","last_became_active":"2023-10-16T12:10:39.644719+0000","last_became_peered":"2023-10-16T12:10:39.644719+0000","last_unstale":"2023-10-16T12:10:43.651482+0000","last_undegraded":"2023-10-16T12:10:43.651482+0000","last_fullsized":"2023-10-16T12:10:43.651482+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:47:52.332290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:44.874338+0000","last_change":"2023-10-16T12:10:43.794795+0000","last_active":"2023-10-16T12:10:44.874338+0000","last_peered":"2023-10-16T12:10:44.874338+0000","last_clean":"2023-10-16T12:10:44.874338+0000","last_became_active":"2023-10-16T12:10:43.794315+0000","last_became_peered":"2023-10-16T12:10:43.794315+0000","last_unstale":"2023-10-16T12:10:44.874338+0000","last_undegraded":"2023-10-16T12:10:44.874338+0000","last_fullsized":"2023-10-16T12:10:44.874338+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:13:07.783237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.642895+0000","last_change":"2023-10-16T12:10:38.622408+0000","last_active":"2023-10-16T12:10:43.642895+0000","last_peered":"2023-10-16T12:10:43.642895+0000","last_clean":"2023-10-16T12:10:43.642895+0000","last_became_active":"2023-10-16T12:10:38.622020+0000","last_became_peered":"2023-10-16T12:10:38.622020+0000","last_unstale":"2023-10-16T12:10:43.642895+0000","last_undegraded":"2023-10-16T12:10:43.642895+0000","last_fullsized":"2023-10-16T12:10:43.642895+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:57:56.978587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.796175+0000","last_change":"2023-10-16T12:10:39.645109+0000","last_active":"2023-10-16T12:10:43.796175+0000","last_peered":"2023-10-16T12:10:43.796175+0000","last_clean":"2023-10-16T12:10:43.796175+0000","last_became_active":"2023-10-16T12:10:39.641743+0000","last_became_peered":"2023-10-16T12:10:39.641743+0000","last_unstale":"2023-10-16T12:10:43.796175+0000","last_undegraded":"2023-10-16T12:10:43.796175+0000","last_fullsized":"2023-10-16T12:10:43.796175+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:53:04.082200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:45.374790+0000","last_change":"2023-10-16T12:10:43.666962+0000","last_active":"2023-10-16T12:10:45.374790+0000","last_peered":"2023-10-16T12:10:45.374790+0000","last_clean":"2023-10-16T12:10:45.374790+0000","last_became_active":"2023-10-16T12:10:43.666469+0000","last_became_peered":"2023-10-16T12:10:43.666469+0000","last_unstale":"2023-10-16T12:10:45.374790+0000","last_undegraded":"2023-10-16T12:10:45.374790+0000","last_fullsized":"2023-10-16T12:10:45.374790+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:14:04.764499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.654298+0000","last_change":"2023-10-16T12:10:38.611610+0000","last_active":"2023-10-16T12:10:43.654298+0000","last_peered":"2023-10-16T12:10:43.654298+0000","last_clean":"2023-10-16T12:10:43.654298+0000","last_became_active":"2023-10-16T12:10:38.611193+0000","last_became_peered":"2023-10-16T12:10:38.611193+0000","last_unstale":"2023-10-16T12:10:43.654298+0000","last_undegraded":"2023-10-16T12:10:43.654298+0000","last_fullsized":"2023-10-16T12:10:43.654298+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:06:08.125512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646733+0000","last_change":"2023-10-16T12:10:39.649078+0000","last_active":"2023-10-16T12:10:43.646733+0000","last_peered":"2023-10-16T12:10:43.646733+0000","last_clean":"2023-10-16T12:10:43.646733+0000","last_became_active":"2023-10-16T12:10:39.648797+0000","last_became_peered":"2023-10-16T12:10:39.648797+0000","last_unstale":"2023-10-16T12:10:43.646733+0000","last_undegraded":"2023-10-16T12:10:43.646733+0000","last_fullsized":"2023-10-16T12:10:43.646733+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:13:03.744288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:45.374995+0000","last_change":"2023-10-16T12:10:43.794916+0000","last_active":"2023-10-16T12:10:45.374995+0000","last_peered":"2023-10-16T12:10:45.374995+0000","last_clean":"2023-10-16T12:10:45.374995+0000","last_became_active":"2023-10-16T12:10:43.794495+0000","last_became_peered":"2023-10-16T12:10:43.794495+0000","last_unstale":"2023-10-16T12:10:45.374995+0000","last_undegraded":"2023-10-16T12:10:45.374995+0000","last_fullsized":"2023-10-16T12:10:45.374995+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:44:47.973077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.642563+0000","last_change":"2023-10-16T12:10:38.624223+0000","last_active":"2023-10-16T12:10:43.642563+0000","last_peered":"2023-10-16T12:10:43.642563+0000","last_clean":"2023-10-16T12:10:43.642563+0000","last_became_active":"2023-10-16T12:10:38.623924+0000","last_became_peered":"2023-10-16T12:10:38.623924+0000","last_unstale":"2023-10-16T12:10:43.642563+0000","last_undegraded":"2023-10-16T12:10:43.642563+0000","last_fullsized":"2023-10-16T12:10:43.642563+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:45:22.011420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.796741+0000","last_change":"2023-10-16T12:10:39.645947+0000","last_active":"2023-10-16T12:10:43.796741+0000","last_peered":"2023-10-16T12:10:43.796741+0000","last_clean":"2023-10-16T12:10:43.796741+0000","last_became_active":"2023-10-16T12:10:39.645261+0000","last_became_peered":"2023-10-16T12:10:39.645261+0000","last_unstale":"2023-10-16T12:10:43.796741+0000","last_undegraded":"2023-10-16T12:10:43.796741+0000","last_fullsized":"2023-10-16T12:10:43.796741+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:03:14.089436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.648802+0000","last_change":"2023-10-16T12:10:39.631025+0000","last_active":"2023-10-16T12:10:43.648802+0000","last_peered":"2023-10-16T12:10:43.648802+0000","last_clean":"2023-10-16T12:10:43.648802+0000","last_became_active":"2023-10-16T12:10:39.630701+0000","last_became_peered":"2023-10-16T12:10:39.630701+0000","last_unstale":"2023-10-16T12:10:43.648802+0000","last_undegraded":"2023-10-16T12:10:43.648802+0000","last_fullsized":"2023-10-16T12:10:43.648802+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:49:21.678699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.642384+0000","last_change":"2023-10-16T12:10:39.640966+0000","last_active":"2023-10-16T12:10:43.642384+0000","last_peered":"2023-10-16T12:10:43.642384+0000","last_clean":"2023-10-16T12:10:43.642384+0000","last_became_active":"2023-10-16T12:10:39.640624+0000","last_became_peered":"2023-10-16T12:10:39.640624+0000","last_unstale":"2023-10-16T12:10:43.642384+0000","last_undegraded":"2023-10-16T12:10:43.642384+0000","last_fullsized":"2023-10-16T12:10:43.642384+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:58:21.398705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646243+0000","last_change":"2023-10-16T12:10:39.647316+0000","last_active":"2023-10-16T12:10:43.646243+0000","last_peered":"2023-10-16T12:10:43.646243+0000","last_clean":"2023-10-16T12:10:43.646243+0000","last_became_active":"2023-10-16T12:10:39.646403+0000","last_became_peered":"2023-10-16T12:10:39.646403+0000","last_unstale":"2023-10-16T12:10:43.646243+0000","last_undegraded":"2023-10-16T12:10:43.646243+0000","last_fullsized":"2023-10-16T12:10:43.646243+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:24:15.538866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.649898+0000","last_change":"2023-10-16T12:10:39.625142+0000","last_active":"2023-10-16T12:10:43.649898+0000","last_peered":"2023-10-16T12:10:43.649898+0000","last_clean":"2023-10-16T12:10:43.649898+0000","last_became_active":"2023-10-16T12:10:39.624714+0000","last_became_peered":"2023-10-16T12:10:39.624714+0000","last_unstale":"2023-10-16T12:10:43.649898+0000","last_undegraded":"2023-10-16T12:10:43.649898+0000","last_fullsized":"2023-10-16T12:10:43.649898+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:46:02.593591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.648251+0000","last_change":"2023-10-16T12:10:39.642728+0000","last_active":"2023-10-16T12:10:43.648251+0000","last_peered":"2023-10-16T12:10:43.648251+0000","last_clean":"2023-10-16T12:10:43.648251+0000","last_became_active":"2023-10-16T12:10:39.642396+0000","last_became_peered":"2023-10-16T12:10:39.642396+0000","last_unstale":"2023-10-16T12:10:43.648251+0000","last_undegraded":"2023-10-16T12:10:43.648251+0000","last_fullsized":"2023-10-16T12:10:43.648251+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:38:01.595985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.654215+0000","last_change":"2023-10-16T12:10:39.635187+0000","last_active":"2023-10-16T12:10:43.654215+0000","last_peered":"2023-10-16T12:10:43.654215+0000","last_clean":"2023-10-16T12:10:43.654215+0000","last_became_active":"2023-10-16T12:10:39.634499+0000","last_became_peered":"2023-10-16T12:10:39.634499+0000","last_unstale":"2023-10-16T12:10:43.654215+0000","last_undegraded":"2023-10-16T12:10:43.654215+0000","last_fullsized":"2023-10-16T12:10:43.654215+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:38:01.595985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.650893+0000","last_change":"2023-10-16T12:10:39.643166+0000","last_active":"2023-10-16T12:10:43.650893+0000","last_peered":"2023-10-16T12:10:43.650893+0000","last_clean":"2023-10-16T12:10:43.650893+0000","last_became_active":"2023-10-16T12:10:39.642893+0000","last_became_peered":"2023-10-16T12:10:39.642893+0000","last_unstale":"2023-10-16T12:10:43.650893+0000","last_undegraded":"2023-10-16T12:10:43.650893+0000","last_fullsized":"2023-10-16T12:10:43.650893+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:33:20.807621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646227+0000","last_change":"2023-10-16T12:10:39.633538+0000","last_active":"2023-10-16T12:10:43.646227+0000","last_peered":"2023-10-16T12:10:43.646227+0000","last_clean":"2023-10-16T12:10:43.646227+0000","last_became_active":"2023-10-16T12:10:39.633019+0000","last_became_peered":"2023-10-16T12:10:39.633019+0000","last_unstale":"2023-10-16T12:10:43.646227+0000","last_undegraded":"2023-10-16T12:10:43.646227+0000","last_fullsized":"2023-10-16T12:10:43.646227+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:59:08.633021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.652820+0000","last_change":"2023-10-16T12:10:39.641581+0000","last_active":"2023-10-16T12:10:43.652820+0000","last_peered":"2023-10-16T12:10:43.652820+0000","last_clean":"2023-10-16T12:10:43.652820+0000","last_became_active":"2023-10-16T12:10:39.640847+0000","last_became_peered":"2023-10-16T12:10:39.640847+0000","last_unstale":"2023-10-16T12:10:43.652820+0000","last_undegraded":"2023-10-16T12:10:43.652820+0000","last_fullsized":"2023-10-16T12:10:43.652820+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:03:12.325906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646580+0000","last_change":"2023-10-16T12:10:39.642631+0000","last_active":"2023-10-16T12:10:43.646580+0000","last_peered":"2023-10-16T12:10:43.646580+0000","last_clean":"2023-10-16T12:10:43.646580+0000","last_became_active":"2023-10-16T12:10:39.642334+0000","last_became_peered":"2023-10-16T12:10:39.642334+0000","last_unstale":"2023-10-16T12:10:43.646580+0000","last_undegraded":"2023-10-16T12:10:43.646580+0000","last_fullsized":"2023-10-16T12:10:43.646580+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:46:02.593591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646326+0000","last_change":"2023-10-16T12:10:39.647489+0000","last_active":"2023-10-16T12:10:43.646326+0000","last_peered":"2023-10-16T12:10:43.646326+0000","last_clean":"2023-10-16T12:10:43.646326+0000","last_became_active":"2023-10-16T12:10:39.646439+0000","last_became_peered":"2023-10-16T12:10:39.646439+0000","last_unstale":"2023-10-16T12:10:43.646326+0000","last_undegraded":"2023-10-16T12:10:43.646326+0000","last_fullsized":"2023-10-16T12:10:43.646326+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:59:08.633021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.649955+0000","last_change":"2023-10-16T12:10:39.638887+0000","last_active":"2023-10-16T12:10:43.649955+0000","last_peered":"2023-10-16T12:10:43.649955+0000","last_clean":"2023-10-16T12:10:43.649955+0000","last_became_active":"2023-10-16T12:10:39.638548+0000","last_became_peered":"2023-10-16T12:10:39.638548+0000","last_unstale":"2023-10-16T12:10:43.649955+0000","last_undegraded":"2023-10-16T12:10:43.649955+0000","last_fullsized":"2023-10-16T12:10:43.649955+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:49:21.678699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.650677+0000","last_change":"2023-10-16T12:10:39.644900+0000","last_active":"2023-10-16T12:10:43.650677+0000","last_peered":"2023-10-16T12:10:43.650677+0000","last_clean":"2023-10-16T12:10:43.650677+0000","last_became_active":"2023-10-16T12:10:39.644516+0000","last_became_peered":"2023-10-16T12:10:39.644516+0000","last_unstale":"2023-10-16T12:10:43.650677+0000","last_undegraded":"2023-10-16T12:10:43.650677+0000","last_fullsized":"2023-10-16T12:10:43.650677+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:03:12.325906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574860,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27864,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343976,"statfs":{"total":96636764160,"available":96608231424,"internally_reserved":0,"allocated":864256,"data_stored":695401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33900000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48299999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50600000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40100000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72799999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52700000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574858,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":108293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.9349999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.87}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.0489999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.083}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574858,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":274432,"data_stored":104611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46400000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35399999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.499}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60999999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574859,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27876,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343964,"statfs":{"total":96636764160,"available":96608219136,"internally_reserved":0,"allocated":872448,"data_stored":695457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.249}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27000000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55200000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52400000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]}]},{"osd":2,"up_from":13,"seq":55834574859,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":266240,"data_stored":104451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36099999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34100000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.317}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74399999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57299999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75600000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77500000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574858,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":108293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42999999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40699999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52500000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60199999999999998}]}]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":278528,"data_stored":104577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46500000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40100000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63600000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76300000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73299999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80200000000000005}]}]},{"osd":5,"up_from":13,"seq":55834574858,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":270336,"data_stored":104541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48699999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76000000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77600000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79100000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T12:11:09.232 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T12:11:09.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.389+0000 7fa595377700 1 Processor -- start 2023-10-16T12:11:09.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.389+0000 7fa595377700 1 -- start start 2023-10-16T12:11:09.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.390+0000 7fa595377700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa5900cf4a0 0x7fa5900726e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:09.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.390+0000 7fa595377700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa5900cf880 0x7fa5900b52f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:09.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.390+0000 7fa595377700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa59015bc80 con 0x7fa5900b5830 2023-10-16T12:11:09.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.390+0000 7fa595377700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa590072c20 con 0x7fa5900cf880 2023-10-16T12:11:09.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.390+0000 7fa595377700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa59015b070 con 0x7fa5900cf4a0 2023-10-16T12:11:09.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.390+0000 7fa58e7fc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa5900cf880 0x7fa5900b52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:09.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.390+0000 7fa58e7fc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa5900cf880 0x7fa5900b52f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33888/0 (socket says 172.21.15.16:33888) 2023-10-16T12:11:09.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.390+0000 7fa58e7fc700 1 -- 172.21.15.16:0/192474210 learned_addr learned my addr 172.21.15.16:0/192474210 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:09.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.390+0000 7fa58effd700 1 --2- 172.21.15.16:0/192474210 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa5900cf4a0 0x7fa5900726e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:09.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.391+0000 7fa58e7fc700 1 -- 172.21.15.16:0/192474210 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa5900cf4a0 msgr2=0x7fa5900726e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:09.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.391+0000 7fa58e7fc700 1 --2- 172.21.15.16:0/192474210 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa5900cf4a0 0x7fa5900726e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:09.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.391+0000 7fa58e7fc700 1 -- 172.21.15.16:0/192474210 >> v1:172.21.15.49:6789/0 conn(0x7fa5900b5830 legacy=0x7fa59015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:09.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.391+0000 7fa58e7fc700 1 -- 172.21.15.16:0/192474210 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa59015b350 con 0x7fa5900cf880 2023-10-16T12:11:09.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.391+0000 7fa58e7fc700 1 --2- 172.21.15.16:0/192474210 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5900cf880 0x7fa5900b52f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa5780098f0 tx=0x7fa578009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41e79acc112a6d2b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:09.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.391+0000 7fa58dffb700 1 -- 172.21.15.16:0/192474210 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa578004230 con 0x7fa5900cf880 2023-10-16T12:11:09.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.391+0000 7fa58dffb700 1 -- 172.21.15.16:0/192474210 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa578004390 con 0x7fa5900cf880 2023-10-16T12:11:09.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.391+0000 7fa58dffb700 1 -- 172.21.15.16:0/192474210 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5780036a0 con 0x7fa5900cf880 2023-10-16T12:11:09.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.391+0000 7fa595377700 1 -- 172.21.15.16:0/192474210 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5900cf880 msgr2=0x7fa5900b52f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:09.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.391+0000 7fa595377700 1 --2- 172.21.15.16:0/192474210 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5900cf880 0x7fa5900b52f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa5780098f0 tx=0x7fa578009c00 comp rx=0 tx=0).stop 2023-10-16T12:11:09.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.391+0000 7fa595377700 1 -- 172.21.15.16:0/192474210 shutdown_connections 2023-10-16T12:11:09.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.391+0000 7fa595377700 1 --2- 172.21.15.16:0/192474210 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa5900cf4a0 0x7fa5900726e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:09.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.391+0000 7fa595377700 1 --2- 172.21.15.16:0/192474210 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5900cf880 0x7fa5900b52f0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:09.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.391+0000 7fa595377700 1 -- 172.21.15.16:0/192474210 >> 172.21.15.16:0/192474210 conn(0x7fa5900b1fd0 msgr2=0x7fa5900b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:09.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.392+0000 7fa595377700 1 -- 172.21.15.16:0/192474210 shutdown_connections 2023-10-16T12:11:09.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.392+0000 7fa595377700 1 -- 172.21.15.16:0/192474210 wait complete. 2023-10-16T12:11:09.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.393+0000 7fa595377700 1 Processor -- start 2023-10-16T12:11:09.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.393+0000 7fa595377700 1 -- start start 2023-10-16T12:11:09.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.394+0000 7fa595377700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5900b5830 0x7fa5900c7f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:09.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.394+0000 7fa595377700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa5900cf4a0 0x7fa5900c63c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:09.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.394+0000 7fa595377700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa5900cf880 0x7fa590157590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:09.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.394+0000 7fa595377700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5900c9b50 con 0x7fa5900b5830 2023-10-16T12:11:09.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.394+0000 7fa595377700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5900c8ad0 con 0x7fa5900cf4a0 2023-10-16T12:11:09.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.394+0000 7fa595377700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5900c8d90 con 0x7fa5900cf880 2023-10-16T12:11:09.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.394+0000 7fa58effd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5900b5830 0x7fa5900c7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:09.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.394+0000 7fa58effd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5900b5830 0x7fa5900c7f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33892/0 (socket says 172.21.15.16:33892) 2023-10-16T12:11:09.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.394+0000 7fa58effd700 1 -- 172.21.15.16:0/1895819988 learned_addr learned my addr 172.21.15.16:0/1895819988 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:09.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.395+0000 7fa58f7fe700 1 --2- 172.21.15.16:0/1895819988 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa5900cf880 0x7fa590157590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:09.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.395+0000 7fa58e7fc700 1 --2- 172.21.15.16:0/1895819988 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa5900cf4a0 0x7fa5900c63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:09.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.395+0000 7fa58effd700 1 -- 172.21.15.16:0/1895819988 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa5900cf880 msgr2=0x7fa590157590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:09.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.395+0000 7fa58effd700 1 --2- 172.21.15.16:0/1895819988 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa5900cf880 0x7fa590157590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:09.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.395+0000 7fa58effd700 1 -- 172.21.15.16:0/1895819988 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa5900cf4a0 msgr2=0x7fa5900c63c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:09.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.395+0000 7fa58effd700 1 --2- 172.21.15.16:0/1895819988 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa5900cf4a0 0x7fa5900c63c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:09.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.395+0000 7fa58effd700 1 -- 172.21.15.16:0/1895819988 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa578009550 con 0x7fa5900b5830 2023-10-16T12:11:09.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.395+0000 7fa58effd700 1 --2- 172.21.15.16:0/1895819988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5900b5830 0x7fa5900c7f20 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa58000e810 tx=0x7fa58000ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:09.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.395+0000 7fa5877fe700 1 -- 172.21.15.16:0/1895819988 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa580013610 con 0x7fa5900b5830 2023-10-16T12:11:09.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.395+0000 7fa5877fe700 1 -- 172.21.15.16:0/1895819988 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa580013770 con 0x7fa5900b5830 2023-10-16T12:11:09.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.395+0000 7fa595377700 1 -- 172.21.15.16:0/1895819988 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5900c9070 con 0x7fa5900b5830 2023-10-16T12:11:09.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.395+0000 7fa5877fe700 1 -- 172.21.15.16:0/1895819988 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa580013b00 con 0x7fa5900b5830 2023-10-16T12:11:09.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.395+0000 7fa595377700 1 -- 172.21.15.16:0/1895819988 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5900c9630 con 0x7fa5900b5830 2023-10-16T12:11:09.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.396+0000 7fa595377700 1 -- 172.21.15.16:0/1895819988 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5900c99a0 con 0x7fa5900b5830 2023-10-16T12:11:09.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.396+0000 7fa5877fe700 1 -- 172.21.15.16:0/1895819988 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fa580013c60 con 0x7fa5900b5830 2023-10-16T12:11:09.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.397+0000 7fa5877fe700 1 --2- 172.21.15.16:0/1895819988 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fa57006af00 0x7fa57006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:09.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.397+0000 7fa5877fe700 1 -- 172.21.15.16:0/1895819988 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fa580089520 con 0x7fa5900b5830 2023-10-16T12:11:09.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.397+0000 7fa58e7fc700 1 --2- 172.21.15.16:0/1895819988 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fa57006af00 0x7fa57006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:09.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.397+0000 7fa58e7fc700 1 --2- 172.21.15.16:0/1895819988 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fa57006af00 0x7fa57006d3c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa5901594d0 tx=0x7fa578008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:09.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.402+0000 7fa5877fe700 1 -- 172.21.15.16:0/1895819988 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa5800542f0 con 0x7fa5900b5830 2023-10-16T12:11:09.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.601+0000 7fa595377700 1 -- 172.21.15.16:0/1895819988 --> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa59015b910 con 0x7fa57006af00 2023-10-16T12:11:09.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.610+0000 7fa5877fe700 1 -- 172.21.15.16:0/1895819988 <== mgr.4104 v2:172.21.15.16:6832/365921436 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+345271 (secure 0 0 0) 0x7fa57801f070 con 0x7fa57006af00 2023-10-16T12:11:09.617 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:11:09.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.614+0000 7fa595377700 1 -- 172.21.15.16:0/1895819988 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fa57006af00 msgr2=0x7fa57006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:09.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.614+0000 7fa595377700 1 --2- 172.21.15.16:0/1895819988 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fa57006af00 0x7fa57006d3c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa5901594d0 tx=0x7fa578008030 comp rx=0 tx=0).stop 2023-10-16T12:11:09.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.614+0000 7fa595377700 1 -- 172.21.15.16:0/1895819988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5900b5830 msgr2=0x7fa5900c7f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:09.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.614+0000 7fa595377700 1 --2- 172.21.15.16:0/1895819988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5900b5830 0x7fa5900c7f20 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa58000e810 tx=0x7fa58000ebd0 comp rx=0 tx=0).stop 2023-10-16T12:11:09.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.614+0000 7fa595377700 1 -- 172.21.15.16:0/1895819988 shutdown_connections 2023-10-16T12:11:09.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.614+0000 7fa595377700 1 --2- 172.21.15.16:0/1895819988 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fa57006af00 0x7fa57006d3c0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:09.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.614+0000 7fa595377700 1 --2- 172.21.15.16:0/1895819988 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa5900cf880 0x7fa590157590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:09.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.614+0000 7fa595377700 1 --2- 172.21.15.16:0/1895819988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5900b5830 0x7fa5900c7f20 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:09.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.614+0000 7fa595377700 1 --2- 172.21.15.16:0/1895819988 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa5900cf4a0 0x7fa5900c63c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:09.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.614+0000 7fa595377700 1 -- 172.21.15.16:0/1895819988 >> 172.21.15.16:0/1895819988 conn(0x7fa5900b1fd0 msgr2=0x7fa59015cd30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:09.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.615+0000 7fa595377700 1 -- 172.21.15.16:0/1895819988 shutdown_connections 2023-10-16T12:11:09.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.615+0000 7fa595377700 1 -- 172.21.15.16:0/1895819988 wait complete. 2023-10-16T12:11:09.622 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-10-16T12:11:09.629 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2023-10-16T12:11:09.102112+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219472,"kb_used_data":3304,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755248,"statfs":{"total":773094113280,"available":772869373952,"internally_reserved":0,"allocated":3383296,"data_stored":2025624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12740,"internal_metadata":221236796},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":141,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003280"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.797013+0000","last_change":"2023-10-16T12:10:39.641670+0000","last_active":"2023-10-16T12:10:43.797013+0000","last_peered":"2023-10-16T12:10:43.797013+0000","last_clean":"2023-10-16T12:10:43.797013+0000","last_became_active":"2023-10-16T12:10:39.639911+0000","last_became_peered":"2023-10-16T12:10:39.639911+0000","last_unstale":"2023-10-16T12:10:43.797013+0000","last_undegraded":"2023-10-16T12:10:43.797013+0000","last_fullsized":"2023-10-16T12:10:43.797013+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:29:43.494490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.656566+0000","last_change":"2023-10-16T12:10:39.637937+0000","last_active":"2023-10-16T12:10:43.656566+0000","last_peered":"2023-10-16T12:10:43.656566+0000","last_clean":"2023-10-16T12:10:43.656566+0000","last_became_active":"2023-10-16T12:10:39.637595+0000","last_became_peered":"2023-10-16T12:10:39.637595+0000","last_unstale":"2023-10-16T12:10:43.656566+0000","last_undegraded":"2023-10-16T12:10:43.656566+0000","last_fullsized":"2023-10-16T12:10:43.656566+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:03:12.325906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646168+0000","last_change":"2023-10-16T12:10:39.630003+0000","last_active":"2023-10-16T12:10:43.646168+0000","last_peered":"2023-10-16T12:10:43.646168+0000","last_clean":"2023-10-16T12:10:43.646168+0000","last_became_active":"2023-10-16T12:10:39.627471+0000","last_became_peered":"2023-10-16T12:10:39.627471+0000","last_unstale":"2023-10-16T12:10:43.646168+0000","last_undegraded":"2023-10-16T12:10:43.646168+0000","last_fullsized":"2023-10-16T12:10:43.646168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:33:20.807621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646890+0000","last_change":"2023-10-16T12:10:39.645834+0000","last_active":"2023-10-16T12:10:43.646890+0000","last_peered":"2023-10-16T12:10:43.646890+0000","last_clean":"2023-10-16T12:10:43.646890+0000","last_became_active":"2023-10-16T12:10:39.645631+0000","last_became_peered":"2023-10-16T12:10:39.645631+0000","last_unstale":"2023-10-16T12:10:43.646890+0000","last_undegraded":"2023-10-16T12:10:43.646890+0000","last_fullsized":"2023-10-16T12:10:43.646890+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:14:00.725551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.794209+0000","last_change":"2023-10-16T12:10:39.642967+0000","last_active":"2023-10-16T12:10:43.794209+0000","last_peered":"2023-10-16T12:10:43.794209+0000","last_clean":"2023-10-16T12:10:43.794209+0000","last_became_active":"2023-10-16T12:10:39.642015+0000","last_became_peered":"2023-10-16T12:10:39.642015+0000","last_unstale":"2023-10-16T12:10:43.794209+0000","last_undegraded":"2023-10-16T12:10:43.794209+0000","last_fullsized":"2023-10-16T12:10:43.794209+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:10:19.840696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646377+0000","last_change":"2023-10-16T12:10:39.648342+0000","last_active":"2023-10-16T12:10:43.646377+0000","last_peered":"2023-10-16T12:10:43.646377+0000","last_clean":"2023-10-16T12:10:43.646377+0000","last_became_active":"2023-10-16T12:10:39.648046+0000","last_became_peered":"2023-10-16T12:10:39.648046+0000","last_unstale":"2023-10-16T12:10:43.646377+0000","last_undegraded":"2023-10-16T12:10:43.646377+0000","last_fullsized":"2023-10-16T12:10:43.646377+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:27:44.016376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.793686+0000","last_change":"2023-10-16T12:10:39.640757+0000","last_active":"2023-10-16T12:10:43.793686+0000","last_peered":"2023-10-16T12:10:43.793686+0000","last_clean":"2023-10-16T12:10:43.793686+0000","last_became_active":"2023-10-16T12:10:39.639250+0000","last_became_peered":"2023-10-16T12:10:39.639250+0000","last_unstale":"2023-10-16T12:10:43.793686+0000","last_undegraded":"2023-10-16T12:10:43.793686+0000","last_fullsized":"2023-10-16T12:10:43.793686+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:59:08.633021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.647118+0000","last_change":"2023-10-16T12:10:39.643535+0000","last_active":"2023-10-16T12:10:43.647118+0000","last_peered":"2023-10-16T12:10:43.647118+0000","last_clean":"2023-10-16T12:10:43.647118+0000","last_became_active":"2023-10-16T12:10:39.643242+0000","last_became_peered":"2023-10-16T12:10:39.643242+0000","last_unstale":"2023-10-16T12:10:43.647118+0000","last_undegraded":"2023-10-16T12:10:43.647118+0000","last_fullsized":"2023-10-16T12:10:43.647118+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:49:21.678699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.796912+0000","last_change":"2023-10-16T12:10:39.645759+0000","last_active":"2023-10-16T12:10:43.796912+0000","last_peered":"2023-10-16T12:10:43.796912+0000","last_clean":"2023-10-16T12:10:43.796912+0000","last_became_active":"2023-10-16T12:10:39.642863+0000","last_became_peered":"2023-10-16T12:10:39.642863+0000","last_unstale":"2023-10-16T12:10:43.796912+0000","last_undegraded":"2023-10-16T12:10:43.796912+0000","last_fullsized":"2023-10-16T12:10:43.796912+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:40:23.467426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.650642+0000","last_change":"2023-10-16T12:10:39.641392+0000","last_active":"2023-10-16T12:10:43.650642+0000","last_peered":"2023-10-16T12:10:43.650642+0000","last_clean":"2023-10-16T12:10:43.650642+0000","last_became_active":"2023-10-16T12:10:39.641111+0000","last_became_peered":"2023-10-16T12:10:39.641111+0000","last_unstale":"2023-10-16T12:10:43.650642+0000","last_undegraded":"2023-10-16T12:10:43.650642+0000","last_fullsized":"2023-10-16T12:10:43.650642+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:38:01.595985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.650275+0000","last_change":"2023-10-16T12:10:39.642376+0000","last_active":"2023-10-16T12:10:43.650275+0000","last_peered":"2023-10-16T12:10:43.650275+0000","last_clean":"2023-10-16T12:10:43.650275+0000","last_became_active":"2023-10-16T12:10:39.642086+0000","last_became_peered":"2023-10-16T12:10:39.642086+0000","last_unstale":"2023-10-16T12:10:43.650275+0000","last_undegraded":"2023-10-16T12:10:43.650275+0000","last_fullsized":"2023-10-16T12:10:43.650275+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:05:05.699350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.648829+0000","last_change":"2023-10-16T12:10:39.637561+0000","last_active":"2023-10-16T12:10:43.648829+0000","last_peered":"2023-10-16T12:10:43.648829+0000","last_clean":"2023-10-16T12:10:43.648829+0000","last_became_active":"2023-10-16T12:10:39.637109+0000","last_became_peered":"2023-10-16T12:10:39.637109+0000","last_unstale":"2023-10-16T12:10:43.648829+0000","last_undegraded":"2023-10-16T12:10:43.648829+0000","last_fullsized":"2023-10-16T12:10:43.648829+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:59:48.851909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.645820+0000","last_change":"2023-10-16T12:10:39.631791+0000","last_active":"2023-10-16T12:10:43.645820+0000","last_peered":"2023-10-16T12:10:43.645820+0000","last_clean":"2023-10-16T12:10:43.645820+0000","last_became_active":"2023-10-16T12:10:39.631245+0000","last_became_peered":"2023-10-16T12:10:39.631245+0000","last_unstale":"2023-10-16T12:10:43.645820+0000","last_undegraded":"2023-10-16T12:10:43.645820+0000","last_fullsized":"2023-10-16T12:10:43.645820+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:05:05.699350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646175+0000","last_change":"2023-10-16T12:10:39.631112+0000","last_active":"2023-10-16T12:10:43.646175+0000","last_peered":"2023-10-16T12:10:43.646175+0000","last_clean":"2023-10-16T12:10:43.646175+0000","last_became_active":"2023-10-16T12:10:39.630312+0000","last_became_peered":"2023-10-16T12:10:39.630312+0000","last_unstale":"2023-10-16T12:10:43.646175+0000","last_undegraded":"2023-10-16T12:10:43.646175+0000","last_fullsized":"2023-10-16T12:10:43.646175+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:04:48.768318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.653241+0000","last_change":"2023-10-16T12:10:39.642725+0000","last_active":"2023-10-16T12:10:43.653241+0000","last_peered":"2023-10-16T12:10:43.653241+0000","last_clean":"2023-10-16T12:10:43.653241+0000","last_became_active":"2023-10-16T12:10:39.642437+0000","last_became_peered":"2023-10-16T12:10:39.642437+0000","last_unstale":"2023-10-16T12:10:43.653241+0000","last_undegraded":"2023-10-16T12:10:43.653241+0000","last_fullsized":"2023-10-16T12:10:43.653241+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:46:02.593591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.652749+0000","last_change":"2023-10-16T12:10:39.641780+0000","last_active":"2023-10-16T12:10:43.652749+0000","last_peered":"2023-10-16T12:10:43.652749+0000","last_clean":"2023-10-16T12:10:43.652749+0000","last_became_active":"2023-10-16T12:10:39.641275+0000","last_became_peered":"2023-10-16T12:10:39.641275+0000","last_unstale":"2023-10-16T12:10:43.652749+0000","last_undegraded":"2023-10-16T12:10:43.652749+0000","last_fullsized":"2023-10-16T12:10:43.652749+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:38:01.595985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.647146+0000","last_change":"2023-10-16T12:10:39.641557+0000","last_active":"2023-10-16T12:10:43.647146+0000","last_peered":"2023-10-16T12:10:43.647146+0000","last_clean":"2023-10-16T12:10:43.647146+0000","last_became_active":"2023-10-16T12:10:39.641242+0000","last_became_peered":"2023-10-16T12:10:39.641242+0000","last_unstale":"2023-10-16T12:10:43.647146+0000","last_undegraded":"2023-10-16T12:10:43.647146+0000","last_fullsized":"2023-10-16T12:10:43.647146+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:10:19.840696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.642631+0000","last_change":"2023-10-16T12:10:39.638264+0000","last_active":"2023-10-16T12:10:43.642631+0000","last_peered":"2023-10-16T12:10:43.642631+0000","last_clean":"2023-10-16T12:10:43.642631+0000","last_became_active":"2023-10-16T12:10:39.637959+0000","last_became_peered":"2023-10-16T12:10:39.637959+0000","last_unstale":"2023-10-16T12:10:43.642631+0000","last_undegraded":"2023-10-16T12:10:43.642631+0000","last_fullsized":"2023-10-16T12:10:43.642631+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:27:06.186345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.796665+0000","last_change":"2023-10-16T12:10:39.646694+0000","last_active":"2023-10-16T12:10:43.796665+0000","last_peered":"2023-10-16T12:10:43.796665+0000","last_clean":"2023-10-16T12:10:43.796665+0000","last_became_active":"2023-10-16T12:10:39.643245+0000","last_became_peered":"2023-10-16T12:10:39.643245+0000","last_unstale":"2023-10-16T12:10:43.796665+0000","last_undegraded":"2023-10-16T12:10:43.796665+0000","last_fullsized":"2023-10-16T12:10:43.796665+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:13:03.744288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:44.684038+0000","last_change":"2023-10-16T12:10:43.800090+0000","last_active":"2023-10-16T12:10:44.684038+0000","last_peered":"2023-10-16T12:10:44.684038+0000","last_clean":"2023-10-16T12:10:44.684038+0000","last_became_active":"2023-10-16T12:10:43.799746+0000","last_became_peered":"2023-10-16T12:10:43.799746+0000","last_unstale":"2023-10-16T12:10:44.684038+0000","last_undegraded":"2023-10-16T12:10:44.684038+0000","last_fullsized":"2023-10-16T12:10:44.684038+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:46:18.137970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646491+0000","last_change":"2023-10-16T12:10:38.620314+0000","last_active":"2023-10-16T12:10:43.646491+0000","last_peered":"2023-10-16T12:10:43.646491+0000","last_clean":"2023-10-16T12:10:43.646491+0000","last_became_active":"2023-10-16T12:10:38.620036+0000","last_became_peered":"2023-10-16T12:10:38.620036+0000","last_unstale":"2023-10-16T12:10:43.646491+0000","last_undegraded":"2023-10-16T12:10:43.646491+0000","last_fullsized":"2023-10-16T12:10:43.646491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:27:30.532335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.645988+0000","last_change":"2023-10-16T12:10:39.638746+0000","last_active":"2023-10-16T12:10:43.645988+0000","last_peered":"2023-10-16T12:10:43.645988+0000","last_clean":"2023-10-16T12:10:43.645988+0000","last_became_active":"2023-10-16T12:10:39.637152+0000","last_became_peered":"2023-10-16T12:10:39.637152+0000","last_unstale":"2023-10-16T12:10:43.645988+0000","last_undegraded":"2023-10-16T12:10:43.645988+0000","last_fullsized":"2023-10-16T12:10:43.645988+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:30:10.006613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:45.375335+0000","last_change":"2023-10-16T12:10:43.662213+0000","last_active":"2023-10-16T12:10:45.375335+0000","last_peered":"2023-10-16T12:10:45.375335+0000","last_clean":"2023-10-16T12:10:45.375335+0000","last_became_active":"2023-10-16T12:10:43.661755+0000","last_became_peered":"2023-10-16T12:10:43.661755+0000","last_unstale":"2023-10-16T12:10:45.375335+0000","last_undegraded":"2023-10-16T12:10:45.375335+0000","last_fullsized":"2023-10-16T12:10:45.375335+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:30:14.045561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646110+0000","last_change":"2023-10-16T12:10:38.625571+0000","last_active":"2023-10-16T12:10:43.646110+0000","last_peered":"2023-10-16T12:10:43.646110+0000","last_clean":"2023-10-16T12:10:43.646110+0000","last_became_active":"2023-10-16T12:10:38.625353+0000","last_became_peered":"2023-10-16T12:10:38.625353+0000","last_unstale":"2023-10-16T12:10:43.646110+0000","last_undegraded":"2023-10-16T12:10:43.646110+0000","last_fullsized":"2023-10-16T12:10:43.646110+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:39:25.396697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646379+0000","last_change":"2023-10-16T12:10:39.641482+0000","last_active":"2023-10-16T12:10:43.646379+0000","last_peered":"2023-10-16T12:10:43.646379+0000","last_clean":"2023-10-16T12:10:43.646379+0000","last_became_active":"2023-10-16T12:10:39.641168+0000","last_became_peered":"2023-10-16T12:10:39.641168+0000","last_unstale":"2023-10-16T12:10:43.646379+0000","last_undegraded":"2023-10-16T12:10:43.646379+0000","last_fullsized":"2023-10-16T12:10:43.646379+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:47:52.332290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:45.376007+0000","last_change":"2023-10-16T12:10:43.668484+0000","last_active":"2023-10-16T12:10:45.376007+0000","last_peered":"2023-10-16T12:10:45.376007+0000","last_clean":"2023-10-16T12:10:45.376007+0000","last_became_active":"2023-10-16T12:10:43.667862+0000","last_became_peered":"2023-10-16T12:10:43.667862+0000","last_unstale":"2023-10-16T12:10:45.376007+0000","last_undegraded":"2023-10-16T12:10:45.376007+0000","last_fullsized":"2023-10-16T12:10:45.376007+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:34:54.098418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.796065+0000","last_change":"2023-10-16T12:10:38.623974+0000","last_active":"2023-10-16T12:10:43.796065+0000","last_peered":"2023-10-16T12:10:43.796065+0000","last_clean":"2023-10-16T12:10:43.796065+0000","last_became_active":"2023-10-16T12:10:38.623532+0000","last_became_peered":"2023-10-16T12:10:38.623532+0000","last_unstale":"2023-10-16T12:10:43.796065+0000","last_undegraded":"2023-10-16T12:10:43.796065+0000","last_fullsized":"2023-10-16T12:10:43.796065+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:03:11.321777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646640+0000","last_change":"2023-10-16T12:10:39.651203+0000","last_active":"2023-10-16T12:10:43.646640+0000","last_peered":"2023-10-16T12:10:43.646640+0000","last_clean":"2023-10-16T12:10:43.646640+0000","last_became_active":"2023-10-16T12:10:39.651001+0000","last_became_peered":"2023-10-16T12:10:39.651001+0000","last_unstale":"2023-10-16T12:10:43.646640+0000","last_undegraded":"2023-10-16T12:10:43.646640+0000","last_fullsized":"2023-10-16T12:10:43.646640+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:34:50.059469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:45.375823+0000","last_change":"2023-10-16T12:10:43.659820+0000","last_active":"2023-10-16T12:10:45.375823+0000","last_peered":"2023-10-16T12:10:45.375823+0000","last_clean":"2023-10-16T12:10:45.375823+0000","last_became_active":"2023-10-16T12:10:43.659301+0000","last_became_peered":"2023-10-16T12:10:43.659301+0000","last_unstale":"2023-10-16T12:10:45.375823+0000","last_undegraded":"2023-10-16T12:10:45.375823+0000","last_fullsized":"2023-10-16T12:10:45.375823+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:59:12.671969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.795902+0000","last_change":"2023-10-16T12:10:38.622949+0000","last_active":"2023-10-16T12:10:43.795902+0000","last_peered":"2023-10-16T12:10:43.795902+0000","last_clean":"2023-10-16T12:10:43.795902+0000","last_became_active":"2023-10-16T12:10:38.622666+0000","last_became_peered":"2023-10-16T12:10:38.622666+0000","last_unstale":"2023-10-16T12:10:43.795902+0000","last_undegraded":"2023-10-16T12:10:43.795902+0000","last_fullsized":"2023-10-16T12:10:43.795902+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:27:30.532335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.653416+0000","last_change":"2023-10-16T12:10:39.650483+0000","last_active":"2023-10-16T12:10:43.653416+0000","last_peered":"2023-10-16T12:10:43.653416+0000","last_clean":"2023-10-16T12:10:43.653416+0000","last_became_active":"2023-10-16T12:10:39.650230+0000","last_became_peered":"2023-10-16T12:10:39.650230+0000","last_unstale":"2023-10-16T12:10:43.653416+0000","last_undegraded":"2023-10-16T12:10:43.653416+0000","last_fullsized":"2023-10-16T12:10:43.653416+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:38:53.624950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:44.684035+0000","last_change":"2023-10-16T12:10:43.667121+0000","last_active":"2023-10-16T12:10:44.684035+0000","last_peered":"2023-10-16T12:10:44.684035+0000","last_clean":"2023-10-16T12:10:44.684035+0000","last_became_active":"2023-10-16T12:10:43.666708+0000","last_became_peered":"2023-10-16T12:10:43.666708+0000","last_unstale":"2023-10-16T12:10:44.684035+0000","last_undegraded":"2023-10-16T12:10:44.684035+0000","last_fullsized":"2023-10-16T12:10:44.684035+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:19:32.391115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.645944+0000","last_change":"2023-10-16T12:10:38.623302+0000","last_active":"2023-10-16T12:10:43.645944+0000","last_peered":"2023-10-16T12:10:43.645944+0000","last_clean":"2023-10-16T12:10:43.645944+0000","last_became_active":"2023-10-16T12:10:38.621867+0000","last_became_peered":"2023-10-16T12:10:38.621867+0000","last_unstale":"2023-10-16T12:10:43.645944+0000","last_undegraded":"2023-10-16T12:10:43.645944+0000","last_fullsized":"2023-10-16T12:10:43.645944+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:59:30.615293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.649101+0000","last_change":"2023-10-16T12:10:39.639254+0000","last_active":"2023-10-16T12:10:43.649101+0000","last_peered":"2023-10-16T12:10:43.649101+0000","last_clean":"2023-10-16T12:10:43.649101+0000","last_became_active":"2023-10-16T12:10:39.638956+0000","last_became_peered":"2023-10-16T12:10:39.638956+0000","last_unstale":"2023-10-16T12:10:43.649101+0000","last_undegraded":"2023-10-16T12:10:43.649101+0000","last_fullsized":"2023-10-16T12:10:43.649101+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:27:31.536464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:45.374724+0000","last_change":"2023-10-16T12:10:43.667499+0000","last_active":"2023-10-16T12:10:45.374724+0000","last_peered":"2023-10-16T12:10:45.374724+0000","last_clean":"2023-10-16T12:10:45.374724+0000","last_became_active":"2023-10-16T12:10:43.667116+0000","last_became_peered":"2023-10-16T12:10:43.667116+0000","last_unstale":"2023-10-16T12:10:45.374724+0000","last_undegraded":"2023-10-16T12:10:45.374724+0000","last_fullsized":"2023-10-16T12:10:45.374724+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:59:35.658370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.651387+0000","last_change":"2023-10-16T12:10:38.619514+0000","last_active":"2023-10-16T12:10:43.651387+0000","last_peered":"2023-10-16T12:10:43.651387+0000","last_clean":"2023-10-16T12:10:43.651387+0000","last_became_active":"2023-10-16T12:10:38.619191+0000","last_became_peered":"2023-10-16T12:10:38.619191+0000","last_unstale":"2023-10-16T12:10:43.651387+0000","last_undegraded":"2023-10-16T12:10:43.651387+0000","last_fullsized":"2023-10-16T12:10:43.651387+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:39:25.396697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.649794+0000","last_change":"2023-10-16T12:10:39.642796+0000","last_active":"2023-10-16T12:10:43.649794+0000","last_peered":"2023-10-16T12:10:43.649794+0000","last_clean":"2023-10-16T12:10:43.649794+0000","last_became_active":"2023-10-16T12:10:39.642536+0000","last_became_peered":"2023-10-16T12:10:39.642536+0000","last_unstale":"2023-10-16T12:10:43.649794+0000","last_undegraded":"2023-10-16T12:10:43.649794+0000","last_fullsized":"2023-10-16T12:10:43.649794+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:27:44.016376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:44.874851+0000","last_change":"2023-10-16T12:10:43.662296+0000","last_active":"2023-10-16T12:10:44.874851+0000","last_peered":"2023-10-16T12:10:44.874851+0000","last_clean":"2023-10-16T12:10:44.874851+0000","last_became_active":"2023-10-16T12:10:43.661893+0000","last_became_peered":"2023-10-16T12:10:43.661893+0000","last_unstale":"2023-10-16T12:10:44.874851+0000","last_undegraded":"2023-10-16T12:10:44.874851+0000","last_fullsized":"2023-10-16T12:10:44.874851+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:14:04.764499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646516+0000","last_change":"2023-10-16T12:10:38.623322+0000","last_active":"2023-10-16T12:10:43.646516+0000","last_peered":"2023-10-16T12:10:43.646516+0000","last_clean":"2023-10-16T12:10:43.646516+0000","last_became_active":"2023-10-16T12:10:38.622826+0000","last_became_peered":"2023-10-16T12:10:38.622826+0000","last_unstale":"2023-10-16T12:10:43.646516+0000","last_undegraded":"2023-10-16T12:10:43.646516+0000","last_fullsized":"2023-10-16T12:10:43.646516+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:30:56.479201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646312+0000","last_change":"2023-10-16T12:10:39.624731+0000","last_active":"2023-10-16T12:10:43.646312+0000","last_peered":"2023-10-16T12:10:43.646312+0000","last_clean":"2023-10-16T12:10:43.646312+0000","last_became_active":"2023-10-16T12:10:39.624399+0000","last_became_peered":"2023-10-16T12:10:39.624399+0000","last_unstale":"2023-10-16T12:10:43.646312+0000","last_undegraded":"2023-10-16T12:10:43.646312+0000","last_fullsized":"2023-10-16T12:10:43.646312+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:47:52.332290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:45.375461+0000","last_change":"2023-10-16T12:10:43.798638+0000","last_active":"2023-10-16T12:10:45.375461+0000","last_peered":"2023-10-16T12:10:45.375461+0000","last_clean":"2023-10-16T12:10:45.375461+0000","last_became_active":"2023-10-16T12:10:43.797973+0000","last_became_peered":"2023-10-16T12:10:43.797973+0000","last_unstale":"2023-10-16T12:10:45.375461+0000","last_undegraded":"2023-10-16T12:10:45.375461+0000","last_fullsized":"2023-10-16T12:10:45.375461+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:29:47.533439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.652773+0000","last_change":"2023-10-16T12:10:38.627818+0000","last_active":"2023-10-16T12:10:43.652773+0000","last_peered":"2023-10-16T12:10:43.652773+0000","last_clean":"2023-10-16T12:10:43.652773+0000","last_became_active":"2023-10-16T12:10:38.627492+0000","last_became_peered":"2023-10-16T12:10:38.627492+0000","last_unstale":"2023-10-16T12:10:43.652773+0000","last_undegraded":"2023-10-16T12:10:43.652773+0000","last_fullsized":"2023-10-16T12:10:43.652773+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:05:04.695222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.653456+0000","last_change":"2023-10-16T12:10:39.633123+0000","last_active":"2023-10-16T12:10:43.653456+0000","last_peered":"2023-10-16T12:10:43.653456+0000","last_clean":"2023-10-16T12:10:43.653456+0000","last_became_active":"2023-10-16T12:10:39.632868+0000","last_became_peered":"2023-10-16T12:10:39.632868+0000","last_unstale":"2023-10-16T12:10:43.653456+0000","last_undegraded":"2023-10-16T12:10:43.653456+0000","last_fullsized":"2023-10-16T12:10:43.653456+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:33:20.807621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:44.683898+0000","last_change":"2023-10-16T12:10:43.794748+0000","last_active":"2023-10-16T12:10:44.683898+0000","last_peered":"2023-10-16T12:10:44.683898+0000","last_clean":"2023-10-16T12:10:44.683898+0000","last_became_active":"2023-10-16T12:10:43.794362+0000","last_became_peered":"2023-10-16T12:10:43.794362+0000","last_unstale":"2023-10-16T12:10:44.683898+0000","last_undegraded":"2023-10-16T12:10:44.683898+0000","last_fullsized":"2023-10-16T12:10:44.683898+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:57:20.445348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.652120+0000","last_change":"2023-10-16T12:10:38.619590+0000","last_active":"2023-10-16T12:10:43.652120+0000","last_peered":"2023-10-16T12:10:43.652120+0000","last_clean":"2023-10-16T12:10:43.652120+0000","last_became_active":"2023-10-16T12:10:38.618984+0000","last_became_peered":"2023-10-16T12:10:38.618984+0000","last_unstale":"2023-10-16T12:10:43.652120+0000","last_undegraded":"2023-10-16T12:10:43.652120+0000","last_fullsized":"2023-10-16T12:10:43.652120+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:27:05.182216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646297+0000","last_change":"2023-10-16T12:10:39.644469+0000","last_active":"2023-10-16T12:10:43.646297+0000","last_peered":"2023-10-16T12:10:43.646297+0000","last_clean":"2023-10-16T12:10:43.646297+0000","last_became_active":"2023-10-16T12:10:39.642359+0000","last_became_peered":"2023-10-16T12:10:39.642359+0000","last_unstale":"2023-10-16T12:10:43.646297+0000","last_undegraded":"2023-10-16T12:10:43.646297+0000","last_fullsized":"2023-10-16T12:10:43.646297+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:30:10.006613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:44.684242+0000","last_change":"2023-10-16T12:10:43.667401+0000","last_active":"2023-10-16T12:10:44.684242+0000","last_peered":"2023-10-16T12:10:44.684242+0000","last_clean":"2023-10-16T12:10:44.684242+0000","last_became_active":"2023-10-16T12:10:43.666829+0000","last_became_peered":"2023-10-16T12:10:43.666829+0000","last_unstale":"2023-10-16T12:10:44.684242+0000","last_undegraded":"2023-10-16T12:10:44.684242+0000","last_fullsized":"2023-10-16T12:10:44.684242+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:13:48.034204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646585+0000","last_change":"2023-10-16T12:10:38.627372+0000","last_active":"2023-10-16T12:10:43.646585+0000","last_peered":"2023-10-16T12:10:43.646585+0000","last_clean":"2023-10-16T12:10:43.646585+0000","last_became_active":"2023-10-16T12:10:38.622500+0000","last_became_peered":"2023-10-16T12:10:38.622500+0000","last_unstale":"2023-10-16T12:10:43.646585+0000","last_undegraded":"2023-10-16T12:10:43.646585+0000","last_fullsized":"2023-10-16T12:10:43.646585+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:57:56.978587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646591+0000","last_change":"2023-10-16T12:10:39.641571+0000","last_active":"2023-10-16T12:10:43.646591+0000","last_peered":"2023-10-16T12:10:43.646591+0000","last_clean":"2023-10-16T12:10:43.646591+0000","last_became_active":"2023-10-16T12:10:39.641228+0000","last_became_peered":"2023-10-16T12:10:39.641228+0000","last_unstale":"2023-10-16T12:10:43.646591+0000","last_undegraded":"2023-10-16T12:10:43.646591+0000","last_fullsized":"2023-10-16T12:10:43.646591+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:38:01.595985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:45.376570+0000","last_change":"2023-10-16T12:10:43.671066+0000","last_active":"2023-10-16T12:10:45.376570+0000","last_peered":"2023-10-16T12:10:45.376570+0000","last_clean":"2023-10-16T12:10:45.376570+0000","last_became_active":"2023-10-16T12:10:43.670693+0000","last_became_peered":"2023-10-16T12:10:43.670693+0000","last_unstale":"2023-10-16T12:10:45.376570+0000","last_undegraded":"2023-10-16T12:10:45.376570+0000","last_fullsized":"2023-10-16T12:10:45.376570+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:24:19.577814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.654541+0000","last_change":"2023-10-16T12:10:38.623191+0000","last_active":"2023-10-16T12:10:43.654541+0000","last_peered":"2023-10-16T12:10:43.654541+0000","last_clean":"2023-10-16T12:10:43.654541+0000","last_became_active":"2023-10-16T12:10:38.622876+0000","last_became_peered":"2023-10-16T12:10:38.622876+0000","last_unstale":"2023-10-16T12:10:43.654541+0000","last_undegraded":"2023-10-16T12:10:43.654541+0000","last_fullsized":"2023-10-16T12:10:43.654541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:59:30.615293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.648686+0000","last_change":"2023-10-16T12:10:39.643461+0000","last_active":"2023-10-16T12:10:43.648686+0000","last_peered":"2023-10-16T12:10:43.648686+0000","last_clean":"2023-10-16T12:10:43.648686+0000","last_became_active":"2023-10-16T12:10:39.643148+0000","last_became_peered":"2023-10-16T12:10:39.643148+0000","last_unstale":"2023-10-16T12:10:43.648686+0000","last_undegraded":"2023-10-16T12:10:43.648686+0000","last_fullsized":"2023-10-16T12:10:43.648686+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:27:06.186345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:44.682345+0000","last_change":"2023-10-16T12:10:43.662674+0000","last_active":"2023-10-16T12:10:44.682345+0000","last_peered":"2023-10-16T12:10:44.682345+0000","last_clean":"2023-10-16T12:10:44.682345+0000","last_became_active":"2023-10-16T12:10:43.662249+0000","last_became_peered":"2023-10-16T12:10:43.662249+0000","last_unstale":"2023-10-16T12:10:44.682345+0000","last_undegraded":"2023-10-16T12:10:44.682345+0000","last_fullsized":"2023-10-16T12:10:44.682345+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:14:04.764499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"25'9","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:49.578853+0000","last_change":"2023-10-16T12:10:38.621924+0000","last_active":"2023-10-16T12:10:49.578853+0000","last_peered":"2023-10-16T12:10:49.578853+0000","last_clean":"2023-10-16T12:10:49.578853+0000","last_became_active":"2023-10-16T12:10:38.621621+0000","last_became_peered":"2023-10-16T12:10:38.621621+0000","last_unstale":"2023-10-16T12:10:49.578853+0000","last_undegraded":"2023-10-16T12:10:49.578853+0000","last_fullsized":"2023-10-16T12:10:49.578853+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:27:30.532335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.796277+0000","last_change":"2023-10-16T12:10:39.641165+0000","last_active":"2023-10-16T12:10:43.796277+0000","last_peered":"2023-10-16T12:10:43.796277+0000","last_clean":"2023-10-16T12:10:43.796277+0000","last_became_active":"2023-10-16T12:10:39.640241+0000","last_became_peered":"2023-10-16T12:10:39.640241+0000","last_unstale":"2023-10-16T12:10:43.796277+0000","last_undegraded":"2023-10-16T12:10:43.796277+0000","last_fullsized":"2023-10-16T12:10:43.796277+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:38:53.624950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:45.377704+0000","last_change":"2023-10-16T12:10:43.796949+0000","last_active":"2023-10-16T12:10:45.377704+0000","last_peered":"2023-10-16T12:10:45.377704+0000","last_clean":"2023-10-16T12:10:45.377704+0000","last_became_active":"2023-10-16T12:10:43.796518+0000","last_became_peered":"2023-10-16T12:10:43.796518+0000","last_unstale":"2023-10-16T12:10:45.377704+0000","last_undegraded":"2023-10-16T12:10:45.377704+0000","last_fullsized":"2023-10-16T12:10:45.377704+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:58:35.336755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.650051+0000","last_change":"2023-10-16T12:10:38.612552+0000","last_active":"2023-10-16T12:10:43.650051+0000","last_peered":"2023-10-16T12:10:43.650051+0000","last_clean":"2023-10-16T12:10:43.650051+0000","last_became_active":"2023-10-16T12:10:38.612221+0000","last_became_peered":"2023-10-16T12:10:38.612221+0000","last_unstale":"2023-10-16T12:10:43.650051+0000","last_undegraded":"2023-10-16T12:10:43.650051+0000","last_fullsized":"2023-10-16T12:10:43.650051+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:27:05.182216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646287+0000","last_change":"2023-10-16T12:10:39.638781+0000","last_active":"2023-10-16T12:10:43.646287+0000","last_peered":"2023-10-16T12:10:43.646287+0000","last_clean":"2023-10-16T12:10:43.646287+0000","last_became_active":"2023-10-16T12:10:39.638523+0000","last_became_peered":"2023-10-16T12:10:39.638523+0000","last_unstale":"2023-10-16T12:10:43.646287+0000","last_undegraded":"2023-10-16T12:10:43.646287+0000","last_fullsized":"2023-10-16T12:10:43.646287+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:05:05.699350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:44.683241+0000","last_change":"2023-10-16T12:10:43.794196+0000","last_active":"2023-10-16T12:10:44.683241+0000","last_peered":"2023-10-16T12:10:44.683241+0000","last_clean":"2023-10-16T12:10:44.683241+0000","last_became_active":"2023-10-16T12:10:43.793712+0000","last_became_peered":"2023-10-16T12:10:43.793712+0000","last_unstale":"2023-10-16T12:10:44.683241+0000","last_undegraded":"2023-10-16T12:10:44.683241+0000","last_fullsized":"2023-10-16T12:10:44.683241+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:40:27.506374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646125+0000","last_change":"2023-10-16T12:10:38.627403+0000","last_active":"2023-10-16T12:10:43.646125+0000","last_peered":"2023-10-16T12:10:43.646125+0000","last_clean":"2023-10-16T12:10:43.646125+0000","last_became_active":"2023-10-16T12:10:38.623013+0000","last_became_peered":"2023-10-16T12:10:38.623013+0000","last_unstale":"2023-10-16T12:10:43.646125+0000","last_undegraded":"2023-10-16T12:10:43.646125+0000","last_fullsized":"2023-10-16T12:10:43.646125+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:47:51.328162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.794292+0000","last_change":"2023-10-16T12:10:39.640179+0000","last_active":"2023-10-16T12:10:43.794292+0000","last_peered":"2023-10-16T12:10:43.794292+0000","last_clean":"2023-10-16T12:10:43.794292+0000","last_became_active":"2023-10-16T12:10:39.639843+0000","last_became_peered":"2023-10-16T12:10:39.639843+0000","last_unstale":"2023-10-16T12:10:43.794292+0000","last_undegraded":"2023-10-16T12:10:43.794292+0000","last_fullsized":"2023-10-16T12:10:43.794292+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:24:15.538866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:44.683431+0000","last_change":"2023-10-16T12:10:43.796968+0000","last_active":"2023-10-16T12:10:44.683431+0000","last_peered":"2023-10-16T12:10:44.683431+0000","last_clean":"2023-10-16T12:10:44.683431+0000","last_became_active":"2023-10-16T12:10:43.796562+0000","last_became_peered":"2023-10-16T12:10:43.796562+0000","last_unstale":"2023-10-16T12:10:44.683431+0000","last_undegraded":"2023-10-16T12:10:44.683431+0000","last_fullsized":"2023-10-16T12:10:44.683431+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:44:47.973077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.650016+0000","last_change":"2023-10-16T12:10:38.622066+0000","last_active":"2023-10-16T12:10:43.650016+0000","last_peered":"2023-10-16T12:10:43.650016+0000","last_clean":"2023-10-16T12:10:43.650016+0000","last_became_active":"2023-10-16T12:10:38.621453+0000","last_became_peered":"2023-10-16T12:10:38.621453+0000","last_unstale":"2023-10-16T12:10:43.650016+0000","last_undegraded":"2023-10-16T12:10:43.650016+0000","last_fullsized":"2023-10-16T12:10:43.650016+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:45:22.011420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.645479+0000","last_change":"2023-10-16T12:10:39.636432+0000","last_active":"2023-10-16T12:10:43.645479+0000","last_peered":"2023-10-16T12:10:43.645479+0000","last_clean":"2023-10-16T12:10:43.645479+0000","last_became_active":"2023-10-16T12:10:39.636136+0000","last_became_peered":"2023-10-16T12:10:39.636136+0000","last_unstale":"2023-10-16T12:10:43.645479+0000","last_undegraded":"2023-10-16T12:10:43.645479+0000","last_fullsized":"2023-10-16T12:10:43.645479+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:46:02.593591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:45.375145+0000","last_change":"2023-10-16T12:10:43.661450+0000","last_active":"2023-10-16T12:10:45.375145+0000","last_peered":"2023-10-16T12:10:45.375145+0000","last_clean":"2023-10-16T12:10:45.375145+0000","last_became_active":"2023-10-16T12:10:43.660931+0000","last_became_peered":"2023-10-16T12:10:43.660931+0000","last_unstale":"2023-10-16T12:10:45.375145+0000","last_undegraded":"2023-10-16T12:10:45.375145+0000","last_fullsized":"2023-10-16T12:10:45.375145+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:53:08.121149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.642296+0000","last_change":"2023-10-16T12:10:38.626933+0000","last_active":"2023-10-16T12:10:43.642296+0000","last_peered":"2023-10-16T12:10:43.642296+0000","last_clean":"2023-10-16T12:10:43.642296+0000","last_became_active":"2023-10-16T12:10:38.626643+0000","last_became_peered":"2023-10-16T12:10:38.626643+0000","last_unstale":"2023-10-16T12:10:43.642296+0000","last_undegraded":"2023-10-16T12:10:43.642296+0000","last_fullsized":"2023-10-16T12:10:43.642296+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:01:27.462901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.794002+0000","last_change":"2023-10-16T12:10:39.648098+0000","last_active":"2023-10-16T12:10:43.794002+0000","last_peered":"2023-10-16T12:10:43.794002+0000","last_clean":"2023-10-16T12:10:43.794002+0000","last_became_active":"2023-10-16T12:10:39.643195+0000","last_became_peered":"2023-10-16T12:10:39.643195+0000","last_unstale":"2023-10-16T12:10:43.794002+0000","last_undegraded":"2023-10-16T12:10:43.794002+0000","last_fullsized":"2023-10-16T12:10:43.794002+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:34:50.059469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:45.374887+0000","last_change":"2023-10-16T12:10:43.662584+0000","last_active":"2023-10-16T12:10:45.374887+0000","last_peered":"2023-10-16T12:10:45.374887+0000","last_clean":"2023-10-16T12:10:45.374887+0000","last_became_active":"2023-10-16T12:10:43.662164+0000","last_became_peered":"2023-10-16T12:10:43.662164+0000","last_unstale":"2023-10-16T12:10:45.374887+0000","last_undegraded":"2023-10-16T12:10:45.374887+0000","last_fullsized":"2023-10-16T12:10:45.374887+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:03:16.364854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.648873+0000","last_change":"2023-10-16T12:10:38.613102+0000","last_active":"2023-10-16T12:10:43.648873+0000","last_peered":"2023-10-16T12:10:43.648873+0000","last_clean":"2023-10-16T12:10:43.648873+0000","last_became_active":"2023-10-16T12:10:38.612596+0000","last_became_peered":"2023-10-16T12:10:38.612596+0000","last_unstale":"2023-10-16T12:10:43.648873+0000","last_undegraded":"2023-10-16T12:10:43.648873+0000","last_fullsized":"2023-10-16T12:10:43.648873+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:59:50.957247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.796815+0000","last_change":"2023-10-16T12:10:39.640141+0000","last_active":"2023-10-16T12:10:43.796815+0000","last_peered":"2023-10-16T12:10:43.796815+0000","last_clean":"2023-10-16T12:10:43.796815+0000","last_became_active":"2023-10-16T12:10:39.639770+0000","last_became_peered":"2023-10-16T12:10:39.639770+0000","last_unstale":"2023-10-16T12:10:43.796815+0000","last_undegraded":"2023-10-16T12:10:43.796815+0000","last_fullsized":"2023-10-16T12:10:43.796815+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:14:00.725551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:44.683486+0000","last_change":"2023-10-16T12:10:43.666858+0000","last_active":"2023-10-16T12:10:44.683486+0000","last_peered":"2023-10-16T12:10:44.683486+0000","last_clean":"2023-10-16T12:10:44.683486+0000","last_became_active":"2023-10-16T12:10:43.666461+0000","last_became_peered":"2023-10-16T12:10:43.666461+0000","last_unstale":"2023-10-16T12:10:44.683486+0000","last_undegraded":"2023-10-16T12:10:44.683486+0000","last_fullsized":"2023-10-16T12:10:44.683486+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:53:08.121149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.663314+0000","last_change":"2023-10-16T12:10:38.625538+0000","last_active":"2023-10-16T12:10:43.663314+0000","last_peered":"2023-10-16T12:10:43.663314+0000","last_clean":"2023-10-16T12:10:43.663314+0000","last_became_active":"2023-10-16T12:10:38.625227+0000","last_became_peered":"2023-10-16T12:10:38.625227+0000","last_unstale":"2023-10-16T12:10:43.663314+0000","last_undegraded":"2023-10-16T12:10:43.663314+0000","last_fullsized":"2023-10-16T12:10:43.663314+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:39:25.396697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.653596+0000","last_change":"2023-10-16T12:10:39.645373+0000","last_active":"2023-10-16T12:10:43.653596+0000","last_peered":"2023-10-16T12:10:43.653596+0000","last_clean":"2023-10-16T12:10:43.653596+0000","last_became_active":"2023-10-16T12:10:39.645083+0000","last_became_peered":"2023-10-16T12:10:39.645083+0000","last_unstale":"2023-10-16T12:10:43.653596+0000","last_undegraded":"2023-10-16T12:10:43.653596+0000","last_fullsized":"2023-10-16T12:10:43.653596+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:53:04.082200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:44.683426+0000","last_change":"2023-10-16T12:10:43.673415+0000","last_active":"2023-10-16T12:10:44.683426+0000","last_peered":"2023-10-16T12:10:44.683426+0000","last_clean":"2023-10-16T12:10:44.683426+0000","last_became_active":"2023-10-16T12:10:43.672874+0000","last_became_peered":"2023-10-16T12:10:43.672874+0000","last_unstale":"2023-10-16T12:10:44.683426+0000","last_undegraded":"2023-10-16T12:10:44.683426+0000","last_fullsized":"2023-10-16T12:10:44.683426+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:38:57.663899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646014+0000","last_change":"2023-10-16T12:10:38.628007+0000","last_active":"2023-10-16T12:10:43.646014+0000","last_peered":"2023-10-16T12:10:43.646014+0000","last_clean":"2023-10-16T12:10:43.646014+0000","last_became_active":"2023-10-16T12:10:38.627760+0000","last_became_peered":"2023-10-16T12:10:38.627760+0000","last_unstale":"2023-10-16T12:10:43.646014+0000","last_undegraded":"2023-10-16T12:10:43.646014+0000","last_fullsized":"2023-10-16T12:10:43.646014+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:46:01.589462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.652978+0000","last_change":"2023-10-16T12:10:39.632645+0000","last_active":"2023-10-16T12:10:43.652978+0000","last_peered":"2023-10-16T12:10:43.652978+0000","last_clean":"2023-10-16T12:10:43.652978+0000","last_became_active":"2023-10-16T12:10:39.632345+0000","last_became_peered":"2023-10-16T12:10:39.632345+0000","last_unstale":"2023-10-16T12:10:43.652978+0000","last_undegraded":"2023-10-16T12:10:43.652978+0000","last_fullsized":"2023-10-16T12:10:43.652978+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:58:21.398705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:44.683787+0000","last_change":"2023-10-16T12:10:43.667036+0000","last_active":"2023-10-16T12:10:44.683787+0000","last_peered":"2023-10-16T12:10:44.683787+0000","last_clean":"2023-10-16T12:10:44.683787+0000","last_became_active":"2023-10-16T12:10:43.666639+0000","last_became_peered":"2023-10-16T12:10:43.666639+0000","last_unstale":"2023-10-16T12:10:44.683787+0000","last_undegraded":"2023-10-16T12:10:44.683787+0000","last_fullsized":"2023-10-16T12:10:44.683787+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:29:47.533439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.657522+0000","last_change":"2023-10-16T12:10:39.633035+0000","last_active":"2023-10-16T12:10:43.657522+0000","last_peered":"2023-10-16T12:10:43.657522+0000","last_clean":"2023-10-16T12:10:43.657522+0000","last_became_active":"2023-10-16T12:10:39.631630+0000","last_became_peered":"2023-10-16T12:10:39.631630+0000","last_unstale":"2023-10-16T12:10:43.657522+0000","last_undegraded":"2023-10-16T12:10:43.657522+0000","last_fullsized":"2023-10-16T12:10:43.657522+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:47:52.332290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.642429+0000","last_change":"2023-10-16T12:10:38.622647+0000","last_active":"2023-10-16T12:10:43.642429+0000","last_peered":"2023-10-16T12:10:43.642429+0000","last_clean":"2023-10-16T12:10:43.642429+0000","last_became_active":"2023-10-16T12:10:38.622323+0000","last_became_peered":"2023-10-16T12:10:38.622323+0000","last_unstale":"2023-10-16T12:10:43.642429+0000","last_undegraded":"2023-10-16T12:10:43.642429+0000","last_fullsized":"2023-10-16T12:10:43.642429+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:27:30.532335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:44.874638+0000","last_change":"2023-10-16T12:10:43.797789+0000","last_active":"2023-10-16T12:10:44.874638+0000","last_peered":"2023-10-16T12:10:44.874638+0000","last_clean":"2023-10-16T12:10:44.874638+0000","last_became_active":"2023-10-16T12:10:43.797367+0000","last_became_peered":"2023-10-16T12:10:43.797367+0000","last_unstale":"2023-10-16T12:10:44.874638+0000","last_undegraded":"2023-10-16T12:10:44.874638+0000","last_fullsized":"2023-10-16T12:10:44.874638+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:24:19.577814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646387+0000","last_change":"2023-10-16T12:10:39.650651+0000","last_active":"2023-10-16T12:10:43.646387+0000","last_peered":"2023-10-16T12:10:43.646387+0000","last_clean":"2023-10-16T12:10:43.646387+0000","last_became_active":"2023-10-16T12:10:39.650279+0000","last_became_peered":"2023-10-16T12:10:39.650279+0000","last_unstale":"2023-10-16T12:10:43.646387+0000","last_undegraded":"2023-10-16T12:10:43.646387+0000","last_fullsized":"2023-10-16T12:10:43.646387+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:03:12.325906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646002+0000","last_change":"2023-10-16T12:10:38.622760+0000","last_active":"2023-10-16T12:10:43.646002+0000","last_peered":"2023-10-16T12:10:43.646002+0000","last_clean":"2023-10-16T12:10:43.646002+0000","last_became_active":"2023-10-16T12:10:38.622408+0000","last_became_peered":"2023-10-16T12:10:38.622408+0000","last_unstale":"2023-10-16T12:10:43.646002+0000","last_undegraded":"2023-10-16T12:10:43.646002+0000","last_fullsized":"2023-10-16T12:10:43.646002+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:52:07.438053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":169,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.793810+0000","last_change":"2023-10-16T12:10:34.746820+0000","last_active":"2023-10-16T12:10:43.793810+0000","last_peered":"2023-10-16T12:10:43.793810+0000","last_clean":"2023-10-16T12:10:43.793810+0000","last_became_active":"2023-10-16T12:10:34.745574+0000","last_became_peered":"2023-10-16T12:10:34.745574+0000","last_unstale":"2023-10-16T12:10:43.793810+0000","last_undegraded":"2023-10-16T12:10:43.793810+0000","last_fullsized":"2023-10-16T12:10:43.793810+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_clean_scrub_stamp":"2023-10-16T12:10:33.577111+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:14.461986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:45.375419+0000","last_change":"2023-10-16T12:10:43.798352+0000","last_active":"2023-10-16T12:10:45.375419+0000","last_peered":"2023-10-16T12:10:45.375419+0000","last_clean":"2023-10-16T12:10:45.375419+0000","last_became_active":"2023-10-16T12:10:43.797410+0000","last_became_peered":"2023-10-16T12:10:43.797410+0000","last_unstale":"2023-10-16T12:10:45.375419+0000","last_undegraded":"2023-10-16T12:10:45.375419+0000","last_fullsized":"2023-10-16T12:10:45.375419+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:03:18.128385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646524+0000","last_change":"2023-10-16T12:10:39.635017+0000","last_active":"2023-10-16T12:10:43.646524+0000","last_peered":"2023-10-16T12:10:43.646524+0000","last_clean":"2023-10-16T12:10:43.646524+0000","last_became_active":"2023-10-16T12:10:39.634564+0000","last_became_peered":"2023-10-16T12:10:39.634564+0000","last_unstale":"2023-10-16T12:10:43.646524+0000","last_undegraded":"2023-10-16T12:10:43.646524+0000","last_fullsized":"2023-10-16T12:10:43.646524+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:30:57.483330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.793813+0000","last_change":"2023-10-16T12:10:38.626904+0000","last_active":"2023-10-16T12:10:43.793813+0000","last_peered":"2023-10-16T12:10:43.793813+0000","last_clean":"2023-10-16T12:10:43.793813+0000","last_became_active":"2023-10-16T12:10:38.626683+0000","last_became_peered":"2023-10-16T12:10:38.626683+0000","last_unstale":"2023-10-16T12:10:43.793813+0000","last_undegraded":"2023-10-16T12:10:43.793813+0000","last_fullsized":"2023-10-16T12:10:43.793813+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:58:20.394576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:45.375040+0000","last_change":"2023-10-16T12:10:43.794928+0000","last_active":"2023-10-16T12:10:45.375040+0000","last_peered":"2023-10-16T12:10:45.375040+0000","last_clean":"2023-10-16T12:10:45.375040+0000","last_became_active":"2023-10-16T12:10:43.794276+0000","last_became_peered":"2023-10-16T12:10:43.794276+0000","last_unstale":"2023-10-16T12:10:45.375040+0000","last_undegraded":"2023-10-16T12:10:45.375040+0000","last_fullsized":"2023-10-16T12:10:45.375040+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:38:57.663899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.795963+0000","last_change":"2023-10-16T12:10:38.620637+0000","last_active":"2023-10-16T12:10:43.795963+0000","last_peered":"2023-10-16T12:10:43.795963+0000","last_clean":"2023-10-16T12:10:43.795963+0000","last_became_active":"2023-10-16T12:10:38.620275+0000","last_became_peered":"2023-10-16T12:10:38.620275+0000","last_unstale":"2023-10-16T12:10:43.795963+0000","last_undegraded":"2023-10-16T12:10:43.795963+0000","last_fullsized":"2023-10-16T12:10:43.795963+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:38:00.591856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.663386+0000","last_change":"2023-10-16T12:10:39.632571+0000","last_active":"2023-10-16T12:10:43.663386+0000","last_peered":"2023-10-16T12:10:43.663386+0000","last_clean":"2023-10-16T12:10:43.663386+0000","last_became_active":"2023-10-16T12:10:39.631893+0000","last_became_peered":"2023-10-16T12:10:39.631893+0000","last_unstale":"2023-10-16T12:10:43.663386+0000","last_undegraded":"2023-10-16T12:10:43.663386+0000","last_fullsized":"2023-10-16T12:10:43.663386+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:05:05.699350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:44.683543+0000","last_change":"2023-10-16T12:10:43.797399+0000","last_active":"2023-10-16T12:10:44.683543+0000","last_peered":"2023-10-16T12:10:44.683543+0000","last_clean":"2023-10-16T12:10:44.683543+0000","last_became_active":"2023-10-16T12:10:43.796570+0000","last_became_peered":"2023-10-16T12:10:43.796570+0000","last_unstale":"2023-10-16T12:10:44.683543+0000","last_undegraded":"2023-10-16T12:10:44.683543+0000","last_fullsized":"2023-10-16T12:10:44.683543+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:58:35.336755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.656426+0000","last_change":"2023-10-16T12:10:38.620866+0000","last_active":"2023-10-16T12:10:43.656426+0000","last_peered":"2023-10-16T12:10:43.656426+0000","last_clean":"2023-10-16T12:10:43.656426+0000","last_became_active":"2023-10-16T12:10:38.620517+0000","last_became_peered":"2023-10-16T12:10:38.620517+0000","last_unstale":"2023-10-16T12:10:43.656426+0000","last_undegraded":"2023-10-16T12:10:43.656426+0000","last_fullsized":"2023-10-16T12:10:43.656426+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:58:20.394576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646193+0000","last_change":"2023-10-16T12:10:39.639395+0000","last_active":"2023-10-16T12:10:43.646193+0000","last_peered":"2023-10-16T12:10:43.646193+0000","last_clean":"2023-10-16T12:10:43.646193+0000","last_became_active":"2023-10-16T12:10:39.639030+0000","last_became_peered":"2023-10-16T12:10:39.639030+0000","last_unstale":"2023-10-16T12:10:43.646193+0000","last_undegraded":"2023-10-16T12:10:43.646193+0000","last_fullsized":"2023-10-16T12:10:43.646193+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:34:50.059469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:45.375076+0000","last_change":"2023-10-16T12:10:43.800001+0000","last_active":"2023-10-16T12:10:45.375076+0000","last_peered":"2023-10-16T12:10:45.375076+0000","last_clean":"2023-10-16T12:10:45.375076+0000","last_became_active":"2023-10-16T12:10:43.799598+0000","last_became_peered":"2023-10-16T12:10:43.799598+0000","last_unstale":"2023-10-16T12:10:45.375076+0000","last_undegraded":"2023-10-16T12:10:45.375076+0000","last_fullsized":"2023-10-16T12:10:45.375076+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:27:48.055325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.652023+0000","last_change":"2023-10-16T12:10:38.619506+0000","last_active":"2023-10-16T12:10:43.652023+0000","last_peered":"2023-10-16T12:10:43.652023+0000","last_clean":"2023-10-16T12:10:43.652023+0000","last_became_active":"2023-10-16T12:10:38.619203+0000","last_became_peered":"2023-10-16T12:10:38.619203+0000","last_unstale":"2023-10-16T12:10:43.652023+0000","last_undegraded":"2023-10-16T12:10:43.652023+0000","last_fullsized":"2023-10-16T12:10:43.652023+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:58:20.394576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.648502+0000","last_change":"2023-10-16T12:10:39.630721+0000","last_active":"2023-10-16T12:10:43.648502+0000","last_peered":"2023-10-16T12:10:43.648502+0000","last_clean":"2023-10-16T12:10:43.648502+0000","last_became_active":"2023-10-16T12:10:39.630448+0000","last_became_peered":"2023-10-16T12:10:39.630448+0000","last_unstale":"2023-10-16T12:10:43.648502+0000","last_undegraded":"2023-10-16T12:10:43.648502+0000","last_fullsized":"2023-10-16T12:10:43.648502+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:03:12.325906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:45.376403+0000","last_change":"2023-10-16T12:10:43.662711+0000","last_active":"2023-10-16T12:10:45.376403+0000","last_peered":"2023-10-16T12:10:45.376403+0000","last_clean":"2023-10-16T12:10:45.376403+0000","last_became_active":"2023-10-16T12:10:43.661726+0000","last_became_peered":"2023-10-16T12:10:43.661726+0000","last_unstale":"2023-10-16T12:10:45.376403+0000","last_undegraded":"2023-10-16T12:10:45.376403+0000","last_fullsized":"2023-10-16T12:10:45.376403+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:13:07.783237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.793165+0000","last_change":"2023-10-16T12:10:38.620553+0000","last_active":"2023-10-16T12:10:43.793165+0000","last_peered":"2023-10-16T12:10:43.793165+0000","last_clean":"2023-10-16T12:10:43.793165+0000","last_became_active":"2023-10-16T12:10:38.620248+0000","last_became_peered":"2023-10-16T12:10:38.620248+0000","last_unstale":"2023-10-16T12:10:43.793165+0000","last_undegraded":"2023-10-16T12:10:43.793165+0000","last_fullsized":"2023-10-16T12:10:43.793165+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:45:22.011420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.647607+0000","last_change":"2023-10-16T12:10:39.631773+0000","last_active":"2023-10-16T12:10:43.647607+0000","last_peered":"2023-10-16T12:10:43.647607+0000","last_clean":"2023-10-16T12:10:43.647607+0000","last_became_active":"2023-10-16T12:10:39.631418+0000","last_became_peered":"2023-10-16T12:10:39.631418+0000","last_unstale":"2023-10-16T12:10:43.647607+0000","last_undegraded":"2023-10-16T12:10:43.647607+0000","last_fullsized":"2023-10-16T12:10:43.647607+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:45:23.015548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:44.683656+0000","last_change":"2023-10-16T12:10:43.794614+0000","last_active":"2023-10-16T12:10:44.683656+0000","last_peered":"2023-10-16T12:10:44.683656+0000","last_clean":"2023-10-16T12:10:44.683656+0000","last_became_active":"2023-10-16T12:10:43.794206+0000","last_became_peered":"2023-10-16T12:10:43.794206+0000","last_unstale":"2023-10-16T12:10:44.683656+0000","last_undegraded":"2023-10-16T12:10:44.683656+0000","last_fullsized":"2023-10-16T12:10:44.683656+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:03:18.128385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.653349+0000","last_change":"2023-10-16T12:10:38.621473+0000","last_active":"2023-10-16T12:10:43.653349+0000","last_peered":"2023-10-16T12:10:43.653349+0000","last_clean":"2023-10-16T12:10:43.653349+0000","last_became_active":"2023-10-16T12:10:38.619100+0000","last_became_peered":"2023-10-16T12:10:38.619100+0000","last_unstale":"2023-10-16T12:10:43.653349+0000","last_undegraded":"2023-10-16T12:10:43.653349+0000","last_fullsized":"2023-10-16T12:10:43.653349+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:04:47.764190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.647008+0000","last_change":"2023-10-16T12:10:39.630605+0000","last_active":"2023-10-16T12:10:43.647008+0000","last_peered":"2023-10-16T12:10:43.647008+0000","last_clean":"2023-10-16T12:10:43.647008+0000","last_became_active":"2023-10-16T12:10:39.630234+0000","last_became_peered":"2023-10-16T12:10:39.630234+0000","last_unstale":"2023-10-16T12:10:43.647008+0000","last_undegraded":"2023-10-16T12:10:43.647008+0000","last_fullsized":"2023-10-16T12:10:43.647008+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:33:20.807621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:45.376060+0000","last_change":"2023-10-16T12:10:43.665913+0000","last_active":"2023-10-16T12:10:45.376060+0000","last_peered":"2023-10-16T12:10:45.376060+0000","last_clean":"2023-10-16T12:10:45.376060+0000","last_became_active":"2023-10-16T12:10:43.665498+0000","last_became_peered":"2023-10-16T12:10:43.665498+0000","last_unstale":"2023-10-16T12:10:45.376060+0000","last_undegraded":"2023-10-16T12:10:45.376060+0000","last_fullsized":"2023-10-16T12:10:45.376060+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:10:23.879645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.652500+0000","last_change":"2023-10-16T12:10:38.623243+0000","last_active":"2023-10-16T12:10:43.652500+0000","last_peered":"2023-10-16T12:10:43.652500+0000","last_clean":"2023-10-16T12:10:43.652500+0000","last_became_active":"2023-10-16T12:10:38.622935+0000","last_became_peered":"2023-10-16T12:10:38.622935+0000","last_unstale":"2023-10-16T12:10:43.652500+0000","last_undegraded":"2023-10-16T12:10:43.652500+0000","last_fullsized":"2023-10-16T12:10:43.652500+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:27:30.532335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.645994+0000","last_change":"2023-10-16T12:10:39.624625+0000","last_active":"2023-10-16T12:10:43.645994+0000","last_peered":"2023-10-16T12:10:43.645994+0000","last_clean":"2023-10-16T12:10:43.645994+0000","last_became_active":"2023-10-16T12:10:39.624281+0000","last_became_peered":"2023-10-16T12:10:39.624281+0000","last_unstale":"2023-10-16T12:10:43.645994+0000","last_undegraded":"2023-10-16T12:10:43.645994+0000","last_fullsized":"2023-10-16T12:10:43.645994+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:27:44.016376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:45.377425+0000","last_change":"2023-10-16T12:10:43.662337+0000","last_active":"2023-10-16T12:10:45.377425+0000","last_peered":"2023-10-16T12:10:45.377425+0000","last_clean":"2023-10-16T12:10:45.377425+0000","last_became_active":"2023-10-16T12:10:43.661865+0000","last_became_peered":"2023-10-16T12:10:43.661865+0000","last_unstale":"2023-10-16T12:10:45.377425+0000","last_undegraded":"2023-10-16T12:10:45.377425+0000","last_fullsized":"2023-10-16T12:10:45.377425+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:14:04.764499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.794403+0000","last_change":"2023-10-16T12:10:38.624192+0000","last_active":"2023-10-16T12:10:43.794403+0000","last_peered":"2023-10-16T12:10:43.794403+0000","last_clean":"2023-10-16T12:10:43.794403+0000","last_became_active":"2023-10-16T12:10:38.623767+0000","last_became_peered":"2023-10-16T12:10:38.623767+0000","last_unstale":"2023-10-16T12:10:43.794403+0000","last_undegraded":"2023-10-16T12:10:43.794403+0000","last_fullsized":"2023-10-16T12:10:43.794403+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:59:30.615293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.651482+0000","last_change":"2023-10-16T12:10:39.645023+0000","last_active":"2023-10-16T12:10:43.651482+0000","last_peered":"2023-10-16T12:10:43.651482+0000","last_clean":"2023-10-16T12:10:43.651482+0000","last_became_active":"2023-10-16T12:10:39.644719+0000","last_became_peered":"2023-10-16T12:10:39.644719+0000","last_unstale":"2023-10-16T12:10:43.651482+0000","last_undegraded":"2023-10-16T12:10:43.651482+0000","last_fullsized":"2023-10-16T12:10:43.651482+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:47:52.332290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:44.874338+0000","last_change":"2023-10-16T12:10:43.794795+0000","last_active":"2023-10-16T12:10:44.874338+0000","last_peered":"2023-10-16T12:10:44.874338+0000","last_clean":"2023-10-16T12:10:44.874338+0000","last_became_active":"2023-10-16T12:10:43.794315+0000","last_became_peered":"2023-10-16T12:10:43.794315+0000","last_unstale":"2023-10-16T12:10:44.874338+0000","last_undegraded":"2023-10-16T12:10:44.874338+0000","last_fullsized":"2023-10-16T12:10:44.874338+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:13:07.783237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.642895+0000","last_change":"2023-10-16T12:10:38.622408+0000","last_active":"2023-10-16T12:10:43.642895+0000","last_peered":"2023-10-16T12:10:43.642895+0000","last_clean":"2023-10-16T12:10:43.642895+0000","last_became_active":"2023-10-16T12:10:38.622020+0000","last_became_peered":"2023-10-16T12:10:38.622020+0000","last_unstale":"2023-10-16T12:10:43.642895+0000","last_undegraded":"2023-10-16T12:10:43.642895+0000","last_fullsized":"2023-10-16T12:10:43.642895+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:57:56.978587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.796175+0000","last_change":"2023-10-16T12:10:39.645109+0000","last_active":"2023-10-16T12:10:43.796175+0000","last_peered":"2023-10-16T12:10:43.796175+0000","last_clean":"2023-10-16T12:10:43.796175+0000","last_became_active":"2023-10-16T12:10:39.641743+0000","last_became_peered":"2023-10-16T12:10:39.641743+0000","last_unstale":"2023-10-16T12:10:43.796175+0000","last_undegraded":"2023-10-16T12:10:43.796175+0000","last_fullsized":"2023-10-16T12:10:43.796175+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:53:04.082200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:45.374790+0000","last_change":"2023-10-16T12:10:43.666962+0000","last_active":"2023-10-16T12:10:45.374790+0000","last_peered":"2023-10-16T12:10:45.374790+0000","last_clean":"2023-10-16T12:10:45.374790+0000","last_became_active":"2023-10-16T12:10:43.666469+0000","last_became_peered":"2023-10-16T12:10:43.666469+0000","last_unstale":"2023-10-16T12:10:45.374790+0000","last_undegraded":"2023-10-16T12:10:45.374790+0000","last_fullsized":"2023-10-16T12:10:45.374790+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:14:04.764499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.654298+0000","last_change":"2023-10-16T12:10:38.611610+0000","last_active":"2023-10-16T12:10:43.654298+0000","last_peered":"2023-10-16T12:10:43.654298+0000","last_clean":"2023-10-16T12:10:43.654298+0000","last_became_active":"2023-10-16T12:10:38.611193+0000","last_became_peered":"2023-10-16T12:10:38.611193+0000","last_unstale":"2023-10-16T12:10:43.654298+0000","last_undegraded":"2023-10-16T12:10:43.654298+0000","last_fullsized":"2023-10-16T12:10:43.654298+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:06:08.125512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646733+0000","last_change":"2023-10-16T12:10:39.649078+0000","last_active":"2023-10-16T12:10:43.646733+0000","last_peered":"2023-10-16T12:10:43.646733+0000","last_clean":"2023-10-16T12:10:43.646733+0000","last_became_active":"2023-10-16T12:10:39.648797+0000","last_became_peered":"2023-10-16T12:10:39.648797+0000","last_unstale":"2023-10-16T12:10:43.646733+0000","last_undegraded":"2023-10-16T12:10:43.646733+0000","last_fullsized":"2023-10-16T12:10:43.646733+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:13:03.744288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-16T12:10:45.374995+0000","last_change":"2023-10-16T12:10:43.794916+0000","last_active":"2023-10-16T12:10:45.374995+0000","last_peered":"2023-10-16T12:10:45.374995+0000","last_clean":"2023-10-16T12:10:45.374995+0000","last_became_active":"2023-10-16T12:10:43.794495+0000","last_became_peered":"2023-10-16T12:10:43.794495+0000","last_unstale":"2023-10-16T12:10:45.374995+0000","last_undegraded":"2023-10-16T12:10:45.374995+0000","last_fullsized":"2023-10-16T12:10:45.374995+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:42.636438+0000","last_clean_scrub_stamp":"2023-10-16T12:10:42.636438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:44:47.973077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.642563+0000","last_change":"2023-10-16T12:10:38.624223+0000","last_active":"2023-10-16T12:10:43.642563+0000","last_peered":"2023-10-16T12:10:43.642563+0000","last_clean":"2023-10-16T12:10:43.642563+0000","last_became_active":"2023-10-16T12:10:38.623924+0000","last_became_peered":"2023-10-16T12:10:38.623924+0000","last_unstale":"2023-10-16T12:10:43.642563+0000","last_undegraded":"2023-10-16T12:10:43.642563+0000","last_fullsized":"2023-10-16T12:10:43.642563+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:37.593361+0000","last_clean_scrub_stamp":"2023-10-16T12:10:37.593361+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:45:22.011420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.796741+0000","last_change":"2023-10-16T12:10:39.645947+0000","last_active":"2023-10-16T12:10:43.796741+0000","last_peered":"2023-10-16T12:10:43.796741+0000","last_clean":"2023-10-16T12:10:43.796741+0000","last_became_active":"2023-10-16T12:10:39.645261+0000","last_became_peered":"2023-10-16T12:10:39.645261+0000","last_unstale":"2023-10-16T12:10:43.796741+0000","last_undegraded":"2023-10-16T12:10:43.796741+0000","last_fullsized":"2023-10-16T12:10:43.796741+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:03:14.089436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.648802+0000","last_change":"2023-10-16T12:10:39.631025+0000","last_active":"2023-10-16T12:10:43.648802+0000","last_peered":"2023-10-16T12:10:43.648802+0000","last_clean":"2023-10-16T12:10:43.648802+0000","last_became_active":"2023-10-16T12:10:39.630701+0000","last_became_peered":"2023-10-16T12:10:39.630701+0000","last_unstale":"2023-10-16T12:10:43.648802+0000","last_undegraded":"2023-10-16T12:10:43.648802+0000","last_fullsized":"2023-10-16T12:10:43.648802+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:49:21.678699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.642384+0000","last_change":"2023-10-16T12:10:39.640966+0000","last_active":"2023-10-16T12:10:43.642384+0000","last_peered":"2023-10-16T12:10:43.642384+0000","last_clean":"2023-10-16T12:10:43.642384+0000","last_became_active":"2023-10-16T12:10:39.640624+0000","last_became_peered":"2023-10-16T12:10:39.640624+0000","last_unstale":"2023-10-16T12:10:43.642384+0000","last_undegraded":"2023-10-16T12:10:43.642384+0000","last_fullsized":"2023-10-16T12:10:43.642384+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:58:21.398705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646243+0000","last_change":"2023-10-16T12:10:39.647316+0000","last_active":"2023-10-16T12:10:43.646243+0000","last_peered":"2023-10-16T12:10:43.646243+0000","last_clean":"2023-10-16T12:10:43.646243+0000","last_became_active":"2023-10-16T12:10:39.646403+0000","last_became_peered":"2023-10-16T12:10:39.646403+0000","last_unstale":"2023-10-16T12:10:43.646243+0000","last_undegraded":"2023-10-16T12:10:43.646243+0000","last_fullsized":"2023-10-16T12:10:43.646243+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:24:15.538866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.649898+0000","last_change":"2023-10-16T12:10:39.625142+0000","last_active":"2023-10-16T12:10:43.649898+0000","last_peered":"2023-10-16T12:10:43.649898+0000","last_clean":"2023-10-16T12:10:43.649898+0000","last_became_active":"2023-10-16T12:10:39.624714+0000","last_became_peered":"2023-10-16T12:10:39.624714+0000","last_unstale":"2023-10-16T12:10:43.649898+0000","last_undegraded":"2023-10-16T12:10:43.649898+0000","last_fullsized":"2023-10-16T12:10:43.649898+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:46:02.593591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.648251+0000","last_change":"2023-10-16T12:10:39.642728+0000","last_active":"2023-10-16T12:10:43.648251+0000","last_peered":"2023-10-16T12:10:43.648251+0000","last_clean":"2023-10-16T12:10:43.648251+0000","last_became_active":"2023-10-16T12:10:39.642396+0000","last_became_peered":"2023-10-16T12:10:39.642396+0000","last_unstale":"2023-10-16T12:10:43.648251+0000","last_undegraded":"2023-10-16T12:10:43.648251+0000","last_fullsized":"2023-10-16T12:10:43.648251+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:38:01.595985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.654215+0000","last_change":"2023-10-16T12:10:39.635187+0000","last_active":"2023-10-16T12:10:43.654215+0000","last_peered":"2023-10-16T12:10:43.654215+0000","last_clean":"2023-10-16T12:10:43.654215+0000","last_became_active":"2023-10-16T12:10:39.634499+0000","last_became_peered":"2023-10-16T12:10:39.634499+0000","last_unstale":"2023-10-16T12:10:43.654215+0000","last_undegraded":"2023-10-16T12:10:43.654215+0000","last_fullsized":"2023-10-16T12:10:43.654215+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:38:01.595985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.650893+0000","last_change":"2023-10-16T12:10:39.643166+0000","last_active":"2023-10-16T12:10:43.650893+0000","last_peered":"2023-10-16T12:10:43.650893+0000","last_clean":"2023-10-16T12:10:43.650893+0000","last_became_active":"2023-10-16T12:10:39.642893+0000","last_became_peered":"2023-10-16T12:10:39.642893+0000","last_unstale":"2023-10-16T12:10:43.650893+0000","last_undegraded":"2023-10-16T12:10:43.650893+0000","last_fullsized":"2023-10-16T12:10:43.650893+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:33:20.807621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646227+0000","last_change":"2023-10-16T12:10:39.633538+0000","last_active":"2023-10-16T12:10:43.646227+0000","last_peered":"2023-10-16T12:10:43.646227+0000","last_clean":"2023-10-16T12:10:43.646227+0000","last_became_active":"2023-10-16T12:10:39.633019+0000","last_became_peered":"2023-10-16T12:10:39.633019+0000","last_unstale":"2023-10-16T12:10:43.646227+0000","last_undegraded":"2023-10-16T12:10:43.646227+0000","last_fullsized":"2023-10-16T12:10:43.646227+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:59:08.633021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.652820+0000","last_change":"2023-10-16T12:10:39.641581+0000","last_active":"2023-10-16T12:10:43.652820+0000","last_peered":"2023-10-16T12:10:43.652820+0000","last_clean":"2023-10-16T12:10:43.652820+0000","last_became_active":"2023-10-16T12:10:39.640847+0000","last_became_peered":"2023-10-16T12:10:39.640847+0000","last_unstale":"2023-10-16T12:10:43.652820+0000","last_undegraded":"2023-10-16T12:10:43.652820+0000","last_fullsized":"2023-10-16T12:10:43.652820+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:03:12.325906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646580+0000","last_change":"2023-10-16T12:10:39.642631+0000","last_active":"2023-10-16T12:10:43.646580+0000","last_peered":"2023-10-16T12:10:43.646580+0000","last_clean":"2023-10-16T12:10:43.646580+0000","last_became_active":"2023-10-16T12:10:39.642334+0000","last_became_peered":"2023-10-16T12:10:39.642334+0000","last_unstale":"2023-10-16T12:10:43.646580+0000","last_undegraded":"2023-10-16T12:10:43.646580+0000","last_fullsized":"2023-10-16T12:10:43.646580+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:46:02.593591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.646326+0000","last_change":"2023-10-16T12:10:39.647489+0000","last_active":"2023-10-16T12:10:43.646326+0000","last_peered":"2023-10-16T12:10:43.646326+0000","last_clean":"2023-10-16T12:10:43.646326+0000","last_became_active":"2023-10-16T12:10:39.646439+0000","last_became_peered":"2023-10-16T12:10:39.646439+0000","last_unstale":"2023-10-16T12:10:43.646326+0000","last_undegraded":"2023-10-16T12:10:43.646326+0000","last_fullsized":"2023-10-16T12:10:43.646326+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:59:08.633021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.649955+0000","last_change":"2023-10-16T12:10:39.638887+0000","last_active":"2023-10-16T12:10:43.649955+0000","last_peered":"2023-10-16T12:10:43.649955+0000","last_clean":"2023-10-16T12:10:43.649955+0000","last_became_active":"2023-10-16T12:10:39.638548+0000","last_became_peered":"2023-10-16T12:10:39.638548+0000","last_unstale":"2023-10-16T12:10:43.649955+0000","last_undegraded":"2023-10-16T12:10:43.649955+0000","last_fullsized":"2023-10-16T12:10:43.649955+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:49:21.678699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-16T12:10:43.650677+0000","last_change":"2023-10-16T12:10:39.644900+0000","last_active":"2023-10-16T12:10:43.650677+0000","last_peered":"2023-10-16T12:10:43.650677+0000","last_clean":"2023-10-16T12:10:43.650677+0000","last_became_active":"2023-10-16T12:10:39.644516+0000","last_became_peered":"2023-10-16T12:10:39.644516+0000","last_unstale":"2023-10-16T12:10:43.650677+0000","last_undegraded":"2023-10-16T12:10:43.650677+0000","last_fullsized":"2023-10-16T12:10:43.650677+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:38.597489+0000","last_clean_scrub_stamp":"2023-10-16T12:10:38.597489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:03:12.325906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574860,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27864,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343976,"statfs":{"total":96636764160,"available":96608231424,"internally_reserved":0,"allocated":864256,"data_stored":695401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33900000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48299999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50600000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40100000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72799999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52700000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574858,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":108293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.9349999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.87}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.0489999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.083}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574858,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":274432,"data_stored":104611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46400000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35399999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.499}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60999999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574859,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27876,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343964,"statfs":{"total":96636764160,"available":96608219136,"internally_reserved":0,"allocated":872448,"data_stored":695457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.249}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27000000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55200000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52400000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]}]},{"osd":2,"up_from":13,"seq":55834574859,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":266240,"data_stored":104451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36099999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34100000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.317}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74399999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57299999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75600000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77500000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574858,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":108293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42999999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40699999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52500000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60199999999999998}]}]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":278528,"data_stored":104577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46500000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40100000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63600000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76300000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73299999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80200000000000005}]}]},{"osd":5,"up_from":13,"seq":55834574858,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":270336,"data_stored":104541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48699999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76000000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77600000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79100000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T12:11:09.636 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-16T12:11:09.636 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-10-16T12:11:09.636 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-10-16T12:11:09.637 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-16T12:11:09.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.790+0000 7f4999a78700 1 Processor -- start 2023-10-16T12:11:09.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.790+0000 7f4999a78700 1 -- start start 2023-10-16T12:11:09.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.791+0000 7f4999a78700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f499406de80 0x7f4994161640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:09.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.791+0000 7f4999a78700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f499406a7e0 0x7f499406ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:09.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.791+0000 7f4999a78700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4994071a50 con 0x7f4994160f10 2023-10-16T12:11:09.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.791+0000 7f4999a78700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f499406b1b0 con 0x7f499406a7e0 2023-10-16T12:11:09.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.791+0000 7f4999a78700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4994161e90 con 0x7f499406de80 2023-10-16T12:11:09.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.791+0000 7f49937fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f4994160f10 0x7f49940caa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51406/0 (socket says 172.21.15.16:51406) 2023-10-16T12:11:09.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.791+0000 7f49937fe700 1 -- 172.21.15.16:0/255536865 learned_addr learned my addr 172.21.15.16:0/255536865 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:09.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.791+0000 7f4992ffd700 1 --2- 172.21.15.16:0/255536865 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f499406de80 0x7f4994161640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:09.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.791+0000 7f4993fff700 1 --2- 172.21.15.16:0/255536865 >> v2:172.21.15.49:3300/0 conn(0x7f499406a7e0 0x7f499406ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:09.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.792+0000 7f49927fc700 1 -- 172.21.15.16:0/255536865 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 133064134 0 0) 0x7f4994071a50 con 0x7f4994160f10 2023-10-16T12:11:09.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.792+0000 7f49927fc700 1 -- 172.21.15.16:0/255536865 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49800035b0 con 0x7f4994160f10 2023-10-16T12:11:09.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.792+0000 7f4993fff700 1 -- 172.21.15.16:0/255536865 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f499406de80 msgr2=0x7f4994161640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:09.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.792+0000 7f4993fff700 1 --2- 172.21.15.16:0/255536865 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f499406de80 0x7f4994161640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:09.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.792+0000 7f4993fff700 1 -- 172.21.15.16:0/255536865 >> v1:172.21.15.16:6789/0 conn(0x7f4994160f10 legacy=0x7f49940caa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:09.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.792+0000 7f4993fff700 1 -- 172.21.15.16:0/255536865 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4994162170 con 0x7f499406a7e0 2023-10-16T12:11:09.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.792+0000 7f4992ffd700 1 --2- 172.21.15.16:0/255536865 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f499406de80 0x7f4994161640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:11:09.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.792+0000 7f4993fff700 1 --2- 172.21.15.16:0/255536865 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f499406a7e0 0x7f499406ac70 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f498c009d90 tx=0x7f498c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc7d2881c7ebae3c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:09.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.792+0000 7f49927fc700 1 -- 172.21.15.16:0/255536865 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f498c00e070 con 0x7f499406a7e0 2023-10-16T12:11:09.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.792+0000 7f49927fc700 1 -- 172.21.15.16:0/255536865 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f498c00bb20 con 0x7f499406a7e0 2023-10-16T12:11:09.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.792+0000 7f49927fc700 1 -- 172.21.15.16:0/255536865 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f498c012430 con 0x7f499406a7e0 2023-10-16T12:11:09.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.793+0000 7f4999a78700 1 -- 172.21.15.16:0/255536865 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f499406a7e0 msgr2=0x7f499406ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:09.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.793+0000 7f4999a78700 1 --2- 172.21.15.16:0/255536865 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f499406a7e0 0x7f499406ac70 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f498c009d90 tx=0x7f498c00b330 comp rx=0 tx=0).stop 2023-10-16T12:11:09.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.793+0000 7f4999a78700 1 -- 172.21.15.16:0/255536865 shutdown_connections 2023-10-16T12:11:09.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.793+0000 7f4999a78700 1 --2- 172.21.15.16:0/255536865 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f499406a7e0 0x7f499406ac70 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:09.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.793+0000 7f4999a78700 1 --2- 172.21.15.16:0/255536865 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f499406de80 0x7f4994161640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:09.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.793+0000 7f4999a78700 1 -- 172.21.15.16:0/255536865 >> 172.21.15.16:0/255536865 conn(0x7f499406bce0 msgr2=0x7f499406c0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:09.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.793+0000 7f4999a78700 1 -- 172.21.15.16:0/255536865 shutdown_connections 2023-10-16T12:11:09.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.793+0000 7f4999a78700 1 -- 172.21.15.16:0/255536865 wait complete. 2023-10-16T12:11:09.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.794+0000 7f4999a78700 1 Processor -- start 2023-10-16T12:11:09.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.794+0000 7f4999a78700 1 -- start start 2023-10-16T12:11:09.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.795+0000 7f4999a78700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f499406a7e0 0x7f49940ae2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:09.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.795+0000 7f4999a78700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f499406de80 0x7f49940d0320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:09.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.795+0000 7f4999a78700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4994160f10 0x7f49940d0860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:09.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.795+0000 7f4999a78700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49941f3fa0 con 0x7f4994160f10 2023-10-16T12:11:09.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.795+0000 7f4999a78700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49941f3110 con 0x7f499406a7e0 2023-10-16T12:11:09.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.795+0000 7f4999a78700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49940ae830 con 0x7f499406de80 2023-10-16T12:11:09.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.795+0000 7f4993fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4994160f10 0x7f49940d0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:09.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.795+0000 7f4993fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4994160f10 0x7f49940d0860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33908/0 (socket says 172.21.15.16:33908) 2023-10-16T12:11:09.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.795+0000 7f4993fff700 1 -- 172.21.15.16:0/2200558114 learned_addr learned my addr 172.21.15.16:0/2200558114 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:09.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.795+0000 7f49937fe700 1 --2- 172.21.15.16:0/2200558114 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f499406a7e0 0x7f49940ae2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:09.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.795+0000 7f4992ffd700 1 --2- 172.21.15.16:0/2200558114 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f499406de80 0x7f49940d0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:09.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.795+0000 7f4993fff700 1 -- 172.21.15.16:0/2200558114 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f499406de80 msgr2=0x7f49940d0320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:09.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.795+0000 7f4993fff700 1 --2- 172.21.15.16:0/2200558114 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f499406de80 0x7f49940d0320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:09.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.795+0000 7f4993fff700 1 -- 172.21.15.16:0/2200558114 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f499406a7e0 msgr2=0x7f49940ae2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:09.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.795+0000 7f4993fff700 1 --2- 172.21.15.16:0/2200558114 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f499406a7e0 0x7f49940ae2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:09.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.795+0000 7f4993fff700 1 -- 172.21.15.16:0/2200558114 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f497c0094a0 con 0x7f4994160f10 2023-10-16T12:11:09.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.795+0000 7f4993fff700 1 --2- 172.21.15.16:0/2200558114 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4994160f10 0x7f49940d0860 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f498c00b370 tx=0x7f498c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:09.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.796+0000 7f498bfff700 1 -- 172.21.15.16:0/2200558114 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f498c012430 con 0x7f4994160f10 2023-10-16T12:11:09.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.796+0000 7f498bfff700 1 -- 172.21.15.16:0/2200558114 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f498c020070 con 0x7f4994160f10 2023-10-16T12:11:09.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.796+0000 7f4999a78700 1 -- 172.21.15.16:0/2200558114 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f498c009a70 con 0x7f4994160f10 2023-10-16T12:11:09.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.796+0000 7f498bfff700 1 -- 172.21.15.16:0/2200558114 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f498c0098f0 con 0x7f4994160f10 2023-10-16T12:11:09.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.796+0000 7f4999a78700 1 -- 172.21.15.16:0/2200558114 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49941f3600 con 0x7f4994160f10 2023-10-16T12:11:09.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.797+0000 7f4999a78700 1 -- 172.21.15.16:0/2200558114 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f499404ec20 con 0x7f4994160f10 2023-10-16T12:11:09.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.797+0000 7f498bfff700 1 -- 172.21.15.16:0/2200558114 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f498c004040 con 0x7f4994160f10 2023-10-16T12:11:09.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.797+0000 7f498bfff700 1 --2- 172.21.15.16:0/2200558114 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f497406af00 0x7f497406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:09.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.797+0000 7f498bfff700 1 -- 172.21.15.16:0/2200558114 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f498c022070 con 0x7f4994160f10 2023-10-16T12:11:09.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.798+0000 7f49937fe700 1 --2- 172.21.15.16:0/2200558114 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f497406af00 0x7f497406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:09.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.798+0000 7f49937fe700 1 --2- 172.21.15.16:0/2200558114 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f497406af00 0x7f497406d3c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f49840010e0 tx=0x7f4984007c70 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:09.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:09.802+0000 7f498bfff700 1 -- 172.21.15.16:0/2200558114 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f498c059c90 con 0x7f4994160f10 2023-10-16T12:11:10.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.048+0000 7f4999a78700 1 -- 172.21.15.16:0/2200558114 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f49941f40e0 con 0x7f4994160f10 2023-10-16T12:11:10.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.049+0000 7f498bfff700 1 -- 172.21.15.16:0/2200558114 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f498c059820 con 0x7f4994160f10 2023-10-16T12:11:10.051 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:11:10.051 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-10-16T12:11:10.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.051+0000 7f4999a78700 1 -- 172.21.15.16:0/2200558114 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f497406af00 msgr2=0x7f497406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:10.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.051+0000 7f4999a78700 1 --2- 172.21.15.16:0/2200558114 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f497406af00 0x7f497406d3c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f49840010e0 tx=0x7f4984007c70 comp rx=0 tx=0).stop 2023-10-16T12:11:10.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.051+0000 7f4999a78700 1 -- 172.21.15.16:0/2200558114 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4994160f10 msgr2=0x7f49940d0860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:10.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.051+0000 7f4999a78700 1 --2- 172.21.15.16:0/2200558114 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4994160f10 0x7f49940d0860 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f498c00b370 tx=0x7f498c004ad0 comp rx=0 tx=0).stop 2023-10-16T12:11:10.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.051+0000 7f4999a78700 1 -- 172.21.15.16:0/2200558114 shutdown_connections 2023-10-16T12:11:10.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.051+0000 7f4999a78700 1 --2- 172.21.15.16:0/2200558114 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f497406af00 0x7f497406d3c0 secure :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f49840010e0 tx=0x7f4984007c70 comp rx=0 tx=0).stop 2023-10-16T12:11:10.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.051+0000 7f4999a78700 1 --2- 172.21.15.16:0/2200558114 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4994160f10 0x7f49940d0860 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:10.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.051+0000 7f4999a78700 1 --2- 172.21.15.16:0/2200558114 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f499406a7e0 0x7f49940ae2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:10.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.051+0000 7f4999a78700 1 --2- 172.21.15.16:0/2200558114 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f499406de80 0x7f49940d0320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:10.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.051+0000 7f4999a78700 1 -- 172.21.15.16:0/2200558114 >> 172.21.15.16:0/2200558114 conn(0x7f499406bce0 msgr2=0x7f499415a850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:10.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.051+0000 7f4999a78700 1 -- 172.21.15.16:0/2200558114 shutdown_connections 2023-10-16T12:11:10.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.051+0000 7f4999a78700 1 -- 172.21.15.16:0/2200558114 wait complete. 2023-10-16T12:11:10.065 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-10-16T12:11:10.065 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:11:10.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.188+0000 7fa1adfd0700 1 Processor -- start 2023-10-16T12:11:10.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.188+0000 7fa1adfd0700 1 -- start start 2023-10-16T12:11:10.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.189+0000 7fa1adfd0700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa1a80cbed0 0x7fa1a806eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:10.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.189+0000 7fa1adfd0700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1a806f0a0 con 0x7fa1a806d350 2023-10-16T12:11:10.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.189+0000 7fa1adfd0700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1a806f380 con 0x7fa1a815a870 2023-10-16T12:11:10.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.189+0000 7fa1adfd0700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1a815ac50 con 0x7fa1a80cbed0 2023-10-16T12:11:10.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.189+0000 7fa1a7fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa1a80cbed0 0x7fa1a806eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:10.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.189+0000 7fa1a7fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa1a80cbed0 0x7fa1a806eaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33918/0 (socket says 172.21.15.16:33918) 2023-10-16T12:11:10.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.189+0000 7fa1a6ffd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa1a806d350 0x7fa1a80cde10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51422/0 (socket says 172.21.15.16:51422) 2023-10-16T12:11:10.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.189+0000 7fa1a7fff700 1 -- 172.21.15.16:0/2215759747 learned_addr learned my addr 172.21.15.16:0/2215759747 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:10.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.190+0000 7fa1a7fff700 1 -- 172.21.15.16:0/2215759747 >> v1:172.21.15.49:6789/0 conn(0x7fa1a815a870 legacy=0x7fa1a80cb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:10.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.190+0000 7fa1a7fff700 1 -- 172.21.15.16:0/2215759747 >> v1:172.21.15.16:6789/0 conn(0x7fa1a806d350 legacy=0x7fa1a80cde10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:10.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.190+0000 7fa1a67fc700 1 -- 172.21.15.16:0/2215759747 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4256562713 0 0) 0x7fa1a806f380 con 0x7fa1a815a870 2023-10-16T12:11:10.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.190+0000 7fa1a7fff700 1 -- 172.21.15.16:0/2215759747 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1a806f660 con 0x7fa1a80cbed0 2023-10-16T12:11:10.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.190+0000 7fa1a7fff700 1 --2- 172.21.15.16:0/2215759747 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1a80cbed0 0x7fa1a806eaa0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa1a000d150 tx=0x7fa1a000d460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96e46ebe82b5e4e3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:10.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.190+0000 7fa1a67fc700 1 -- 172.21.15.16:0/2215759747 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1a0010070 con 0x7fa1a80cbed0 2023-10-16T12:11:10.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.190+0000 7fa1a67fc700 1 -- 172.21.15.16:0/2215759747 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1a000b2e0 con 0x7fa1a80cbed0 2023-10-16T12:11:10.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.190+0000 7fa1a67fc700 1 -- 172.21.15.16:0/2215759747 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1a000dcf0 con 0x7fa1a80cbed0 2023-10-16T12:11:10.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.190+0000 7fa1adfd0700 1 -- 172.21.15.16:0/2215759747 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1a80cbed0 msgr2=0x7fa1a806eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:10.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.190+0000 7fa1adfd0700 1 --2- 172.21.15.16:0/2215759747 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1a80cbed0 0x7fa1a806eaa0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa1a000d150 tx=0x7fa1a000d460 comp rx=0 tx=0).stop 2023-10-16T12:11:10.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.191+0000 7fa1adfd0700 1 -- 172.21.15.16:0/2215759747 shutdown_connections 2023-10-16T12:11:10.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.191+0000 7fa1adfd0700 1 --2- 172.21.15.16:0/2215759747 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1a80cbed0 0x7fa1a806eaa0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:10.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.191+0000 7fa1adfd0700 1 -- 172.21.15.16:0/2215759747 >> 172.21.15.16:0/2215759747 conn(0x7fa1a80cc5b0 msgr2=0x7fa1a80cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:10.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.191+0000 7fa1adfd0700 1 -- 172.21.15.16:0/2215759747 shutdown_connections 2023-10-16T12:11:10.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.191+0000 7fa1adfd0700 1 -- 172.21.15.16:0/2215759747 wait complete. 2023-10-16T12:11:10.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.192+0000 7fa1adfd0700 1 Processor -- start 2023-10-16T12:11:10.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.192+0000 7fa1adfd0700 1 -- start start 2023-10-16T12:11:10.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.193+0000 7fa1adfd0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1a806d350 0x7fa1a81ee870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:10.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.193+0000 7fa1adfd0700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa1a80cbed0 0x7fa1a815add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:10.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.193+0000 7fa1adfd0700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa1a815a870 0x7fa1a80c7940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:10.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.193+0000 7fa1adfd0700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a8157f00 con 0x7fa1a806d350 2023-10-16T12:11:10.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.193+0000 7fa1adfd0700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a81581e0 con 0x7fa1a815a870 2023-10-16T12:11:10.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.193+0000 7fa1adfd0700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a81584c0 con 0x7fa1a80cbed0 2023-10-16T12:11:10.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.193+0000 7fa1a77fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1a806d350 0x7fa1a81ee870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:10.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.193+0000 7fa1a77fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1a806d350 0x7fa1a81ee870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33920/0 (socket says 172.21.15.16:33920) 2023-10-16T12:11:10.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.193+0000 7fa1a77fe700 1 -- 172.21.15.16:0/487611427 learned_addr learned my addr 172.21.15.16:0/487611427 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:10.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.193+0000 7fa1a77fe700 1 -- 172.21.15.16:0/487611427 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa1a80cbed0 msgr2=0x7fa1a815add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:10.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.193+0000 7fa1a77fe700 1 --2- 172.21.15.16:0/487611427 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa1a80cbed0 0x7fa1a815add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:10.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.193+0000 7fa1a7fff700 1 --2- 172.21.15.16:0/487611427 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa1a815a870 0x7fa1a80c7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:10.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.193+0000 7fa1a77fe700 1 -- 172.21.15.16:0/487611427 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa1a815a870 msgr2=0x7fa1a80c7940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:10.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.193+0000 7fa1a77fe700 1 --2- 172.21.15.16:0/487611427 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa1a815a870 0x7fa1a80c7940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:10.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.193+0000 7fa1a77fe700 1 -- 172.21.15.16:0/487611427 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1a000bd40 con 0x7fa1a806d350 2023-10-16T12:11:10.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.193+0000 7fa1a7fff700 1 --2- 172.21.15.16:0/487611427 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa1a815a870 0x7fa1a80c7940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:11:10.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.193+0000 7fa1a77fe700 1 --2- 172.21.15.16:0/487611427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1a806d350 0x7fa1a81ee870 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fa19800c450 tx=0x7fa19800c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:10.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.193+0000 7fa18ffff700 1 -- 172.21.15.16:0/487611427 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1980054e0 con 0x7fa1a806d350 2023-10-16T12:11:10.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.193+0000 7fa18ffff700 1 -- 172.21.15.16:0/487611427 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa198003680 con 0x7fa1a806d350 2023-10-16T12:11:10.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.193+0000 7fa1adfd0700 1 -- 172.21.15.16:0/487611427 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1a81587a0 con 0x7fa1a806d350 2023-10-16T12:11:10.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.193+0000 7fa18ffff700 1 -- 172.21.15.16:0/487611427 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa198003a30 con 0x7fa1a806d350 2023-10-16T12:11:10.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.194+0000 7fa1adfd0700 1 -- 172.21.15.16:0/487611427 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1a80c7e80 con 0x7fa1a806d350 2023-10-16T12:11:10.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.194+0000 7fa1adfd0700 1 -- 172.21.15.16:0/487611427 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1a8158cc0 con 0x7fa1a806d350 2023-10-16T12:11:10.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.195+0000 7fa18ffff700 1 -- 172.21.15.16:0/487611427 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fa198003b90 con 0x7fa1a806d350 2023-10-16T12:11:10.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.196+0000 7fa18ffff700 1 --2- 172.21.15.16:0/487611427 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fa19406af00 0x7fa19406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:10.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.196+0000 7fa18ffff700 1 -- 172.21.15.16:0/487611427 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fa198088ee0 con 0x7fa1a806d350 2023-10-16T12:11:10.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.196+0000 7fa1a6ffd700 1 --2- 172.21.15.16:0/487611427 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fa19406af00 0x7fa19406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:10.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.196+0000 7fa1a6ffd700 1 --2- 172.21.15.16:0/487611427 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fa19406af00 0x7fa19406d3c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa1a8154450 tx=0x7fa19c007a30 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:10.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.200+0000 7fa18ffff700 1 -- 172.21.15.16:0/487611427 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa198053e10 con 0x7fa1a806d350 2023-10-16T12:11:10.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.404+0000 7fa1adfd0700 1 -- 172.21.15.16:0/487611427 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa1a8158cc0 con 0x7fa1a806d350 2023-10-16T12:11:10.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.405+0000 7fa18ffff700 1 -- 172.21.15.16:0/487611427 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18238 (secure 0 0 0) 0x7fa1a8158cc0 con 0x7fa1a806d350 2023-10-16T12:11:10.407 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:11:10.407 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":25,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:10:44.080141+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:10:36.935773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T12:10:38.004870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-16T12:10:41.672025+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:11:10.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.407+0000 7fa1adfd0700 1 -- 172.21.15.16:0/487611427 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fa19406af00 msgr2=0x7fa19406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:10.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.407+0000 7fa1adfd0700 1 --2- 172.21.15.16:0/487611427 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fa19406af00 0x7fa19406d3c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa1a8154450 tx=0x7fa19c007a30 comp rx=0 tx=0).stop 2023-10-16T12:11:10.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.407+0000 7fa1adfd0700 1 -- 172.21.15.16:0/487611427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1a806d350 msgr2=0x7fa1a81ee870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:10.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.407+0000 7fa1adfd0700 1 --2- 172.21.15.16:0/487611427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1a806d350 0x7fa1a81ee870 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fa19800c450 tx=0x7fa19800c810 comp rx=0 tx=0).stop 2023-10-16T12:11:10.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.407+0000 7fa1adfd0700 1 -- 172.21.15.16:0/487611427 shutdown_connections 2023-10-16T12:11:10.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.407+0000 7fa1adfd0700 1 --2- 172.21.15.16:0/487611427 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fa19406af00 0x7fa19406d3c0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:10.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.407+0000 7fa1adfd0700 1 --2- 172.21.15.16:0/487611427 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa1a815a870 0x7fa1a80c7940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:10.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.407+0000 7fa1adfd0700 1 --2- 172.21.15.16:0/487611427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1a806d350 0x7fa1a81ee870 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:10.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.407+0000 7fa1adfd0700 1 --2- 172.21.15.16:0/487611427 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa1a80cbed0 0x7fa1a815add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:10.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.407+0000 7fa1adfd0700 1 -- 172.21.15.16:0/487611427 >> 172.21.15.16:0/487611427 conn(0x7fa1a80cc5b0 msgr2=0x7fa1a8155140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:10.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.407+0000 7fa1adfd0700 1 -- 172.21.15.16:0/487611427 shutdown_connections 2023-10-16T12:11:10.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.407+0000 7fa1adfd0700 1 -- 172.21.15.16:0/487611427 wait complete. 2023-10-16T12:11:10.421 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T12:10:36.935773+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T12:10:38.004870+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-16T12:10:41.672025+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-16T12:11:10.422 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:11:10.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.548+0000 7f7af7e6f700 1 Processor -- start 2023-10-16T12:11:10.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.548+0000 7f7af7e6f700 1 -- start start 2023-10-16T12:11:10.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.549+0000 7f7af7e6f700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f7af00d2110 0x7f7af00ae1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:10.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.549+0000 7f7af7e6f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7af006ca10 0x7f7af00ae720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:10.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.549+0000 7f7af7e6f700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7af00b20f0 con 0x7f7af00ce6a0 2023-10-16T12:11:10.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.549+0000 7f7af7e6f700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7af00d03d0 con 0x7f7af00d2110 2023-10-16T12:11:10.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.549+0000 7f7af7e6f700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7af00b2340 con 0x7f7af006ca10 2023-10-16T12:11:10.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.550+0000 7f7af640c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7af006ca10 0x7f7af00ae720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:10.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.550+0000 7f7af540a700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f7af00d2110 0x7f7af00ae1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:10.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.550+0000 7f7af640c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7af006ca10 0x7f7af00ae720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:50422/0 (socket says 172.21.15.16:50422) 2023-10-16T12:11:10.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.550+0000 7f7af5c0b700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7f7af00ce6a0 0x7f7af006c300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:58616/0 (socket says 172.21.15.16:58616) 2023-10-16T12:11:10.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.550+0000 7f7af640c700 1 -- 172.21.15.16:0/179041180 learned_addr learned my addr 172.21.15.16:0/179041180 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:10.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.550+0000 7f7af540a700 1 -- 172.21.15.16:0/179041180 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7af006ca10 msgr2=0x7f7af00ae720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:10.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.550+0000 7f7af540a700 1 --2- 172.21.15.16:0/179041180 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7af006ca10 0x7f7af00ae720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:10.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.550+0000 7f7af540a700 1 -- 172.21.15.16:0/179041180 >> v1:172.21.15.49:6789/0 conn(0x7f7af00ce6a0 legacy=0x7f7af006c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:10.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.550+0000 7f7af540a700 1 -- 172.21.15.16:0/179041180 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7af006bb00 con 0x7f7af00d2110 2023-10-16T12:11:10.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.550+0000 7f7af640c700 1 --2- 172.21.15.16:0/179041180 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7af006ca10 0x7f7af00ae720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:11:10.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.550+0000 7f7af540a700 1 --2- 172.21.15.16:0/179041180 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7af00d2110 0x7f7af00ae1e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7aec009840 tx=0x7f7aec009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c92ac60489b4e9a3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:10.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.550+0000 7f7af4c09700 1 -- 172.21.15.16:0/179041180 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aec011830 con 0x7f7af00d2110 2023-10-16T12:11:10.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.550+0000 7f7af4c09700 1 -- 172.21.15.16:0/179041180 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7aec011990 con 0x7f7af00d2110 2023-10-16T12:11:10.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.550+0000 7f7af4c09700 1 -- 172.21.15.16:0/179041180 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aec011d20 con 0x7f7af00d2110 2023-10-16T12:11:10.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.551+0000 7f7af7e6f700 1 -- 172.21.15.16:0/179041180 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7af00d2110 msgr2=0x7f7af00ae1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:10.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.551+0000 7f7af7e6f700 1 --2- 172.21.15.16:0/179041180 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7af00d2110 0x7f7af00ae1e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7aec009840 tx=0x7f7aec009b50 comp rx=0 tx=0).stop 2023-10-16T12:11:10.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.551+0000 7f7af7e6f700 1 -- 172.21.15.16:0/179041180 shutdown_connections 2023-10-16T12:11:10.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.551+0000 7f7af7e6f700 1 --2- 172.21.15.16:0/179041180 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7af006ca10 0x7f7af00ae720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:10.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.551+0000 7f7af7e6f700 1 --2- 172.21.15.16:0/179041180 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7af00d2110 0x7f7af00ae1e0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:10.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.551+0000 7f7af7e6f700 1 -- 172.21.15.16:0/179041180 >> 172.21.15.16:0/179041180 conn(0x7f7af00d15f0 msgr2=0x7f7af00d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:10.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.551+0000 7f7af7e6f700 1 -- 172.21.15.16:0/179041180 shutdown_connections 2023-10-16T12:11:10.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.551+0000 7f7af7e6f700 1 -- 172.21.15.16:0/179041180 wait complete. 2023-10-16T12:11:10.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.552+0000 7f7af7e6f700 1 Processor -- start 2023-10-16T12:11:10.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.552+0000 7f7af7e6f700 1 -- start start 2023-10-16T12:11:10.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.553+0000 7f7af7e6f700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7af006ca10 0x7f7af00b4170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:10.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.553+0000 7f7af7e6f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7af00ce6a0 0x7f7af0156b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:10.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.553+0000 7f7af7e6f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7af00d2110 0x7f7af00af360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:10.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.553+0000 7f7af7e6f700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7af00ca4f0 con 0x7f7af00d2110 2023-10-16T12:11:10.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.553+0000 7f7af7e6f700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7af00b46b0 con 0x7f7af006ca10 2023-10-16T12:11:10.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.553+0000 7f7af7e6f700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7af01570b0 con 0x7f7af00ce6a0 2023-10-16T12:11:10.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.553+0000 7f7af640c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7af00d2110 0x7f7af00af360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:10.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.553+0000 7f7af640c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7af00d2110 0x7f7af00af360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33938/0 (socket says 172.21.15.16:33938) 2023-10-16T12:11:10.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.553+0000 7f7af640c700 1 -- 172.21.15.16:0/3649929870 learned_addr learned my addr 172.21.15.16:0/3649929870 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:10.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.553+0000 7f7af5c0b700 1 --2- 172.21.15.16:0/3649929870 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7af006ca10 0x7f7af00b4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:10.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.553+0000 7f7af540a700 1 --2- 172.21.15.16:0/3649929870 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7af00ce6a0 0x7f7af0156b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:10.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.553+0000 7f7af640c700 1 -- 172.21.15.16:0/3649929870 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7af00ce6a0 msgr2=0x7f7af0156b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:10.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.553+0000 7f7af640c700 1 --2- 172.21.15.16:0/3649929870 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7af00ce6a0 0x7f7af0156b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:10.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.553+0000 7f7af640c700 1 -- 172.21.15.16:0/3649929870 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7af006ca10 msgr2=0x7f7af00b4170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:10.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.553+0000 7f7af640c700 1 --2- 172.21.15.16:0/3649929870 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7af006ca10 0x7f7af00b4170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:10.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.553+0000 7f7af640c700 1 -- 172.21.15.16:0/3649929870 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ae8009b30 con 0x7f7af00d2110 2023-10-16T12:11:10.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.554+0000 7f7af640c700 1 --2- 172.21.15.16:0/3649929870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7af00d2110 0x7f7af00af360 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f7ae800efc0 tx=0x7f7ae800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:10.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.554+0000 7f7ae67fc700 1 -- 172.21.15.16:0/3649929870 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ae8018070 con 0x7f7af00d2110 2023-10-16T12:11:10.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.554+0000 7f7ae67fc700 1 -- 172.21.15.16:0/3649929870 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ae800f040 con 0x7f7af00d2110 2023-10-16T12:11:10.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.554+0000 7f7af7e6f700 1 -- 172.21.15.16:0/3649929870 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7aec0094a0 con 0x7f7af00d2110 2023-10-16T12:11:10.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.554+0000 7f7ae67fc700 1 -- 172.21.15.16:0/3649929870 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ae8003c20 con 0x7f7af00d2110 2023-10-16T12:11:10.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.554+0000 7f7af7e6f700 1 -- 172.21.15.16:0/3649929870 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7af01f72b0 con 0x7f7af00d2110 2023-10-16T12:11:10.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.555+0000 7f7af7e6f700 1 -- 172.21.15.16:0/3649929870 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7af00ca630 con 0x7f7af00d2110 2023-10-16T12:11:10.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.555+0000 7f7ae67fc700 1 -- 172.21.15.16:0/3649929870 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f7ae800c880 con 0x7f7af00d2110 2023-10-16T12:11:10.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.556+0000 7f7ae67fc700 1 --2- 172.21.15.16:0/3649929870 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f7ad406af00 0x7f7ad406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:10.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.556+0000 7f7ae67fc700 1 -- 172.21.15.16:0/3649929870 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f7ae80898e0 con 0x7f7af00d2110 2023-10-16T12:11:10.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.556+0000 7f7af5c0b700 1 --2- 172.21.15.16:0/3649929870 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f7ad406af00 0x7f7ad406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:10.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.556+0000 7f7af5c0b700 1 --2- 172.21.15.16:0/3649929870 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f7ad406af00 0x7f7ad406d3c0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7ae0000f60 tx=0x7f7ae0007ed0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:10.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.560+0000 7f7ae67fc700 1 -- 172.21.15.16:0/3649929870 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7af00ca630 con 0x7f7af00d2110 2023-10-16T12:11:10.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.762+0000 7f7af7e6f700 1 -- 172.21.15.16:0/3649929870 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7af00ca630 con 0x7f7af00d2110 2023-10-16T12:11:10.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.762+0000 7f7ae67fc700 1 -- 172.21.15.16:0/3649929870 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7af00ca630 con 0x7f7af00d2110 2023-10-16T12:11:10.764 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T12:11:10.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.764+0000 7f7af7e6f700 1 -- 172.21.15.16:0/3649929870 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f7ad406af00 msgr2=0x7f7ad406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:10.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.764+0000 7f7af7e6f700 1 --2- 172.21.15.16:0/3649929870 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f7ad406af00 0x7f7ad406d3c0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7ae0000f60 tx=0x7f7ae0007ed0 comp rx=0 tx=0).stop 2023-10-16T12:11:10.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.764+0000 7f7af7e6f700 1 -- 172.21.15.16:0/3649929870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7af00d2110 msgr2=0x7f7af00af360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:10.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.764+0000 7f7af7e6f700 1 --2- 172.21.15.16:0/3649929870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7af00d2110 0x7f7af00af360 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f7ae800efc0 tx=0x7f7ae800c3a0 comp rx=0 tx=0).stop 2023-10-16T12:11:10.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.764+0000 7f7af7e6f700 1 -- 172.21.15.16:0/3649929870 shutdown_connections 2023-10-16T12:11:10.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.764+0000 7f7af7e6f700 1 --2- 172.21.15.16:0/3649929870 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f7ad406af00 0x7f7ad406d3c0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:10.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.764+0000 7f7af7e6f700 1 --2- 172.21.15.16:0/3649929870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7af00d2110 0x7f7af00af360 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:10.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.764+0000 7f7af7e6f700 1 --2- 172.21.15.16:0/3649929870 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7af006ca10 0x7f7af00b4170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:10.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.764+0000 7f7af7e6f700 1 --2- 172.21.15.16:0/3649929870 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7af00ce6a0 0x7f7af0156b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:10.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.764+0000 7f7af7e6f700 1 -- 172.21.15.16:0/3649929870 >> 172.21.15.16:0/3649929870 conn(0x7f7af00d15f0 msgr2=0x7f7af00d19d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:10.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.765+0000 7f7af7e6f700 1 -- 172.21.15.16:0/3649929870 shutdown_connections 2023-10-16T12:11:10.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.765+0000 7f7af7e6f700 1 -- 172.21.15.16:0/3649929870 wait complete. 2023-10-16T12:11:10.778 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T12:11:10.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.905+0000 7fbf7caee700 1 Processor -- start 2023-10-16T12:11:10.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.905+0000 7fbf7caee700 1 -- start start 2023-10-16T12:11:10.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.906+0000 7fbf7caee700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fbf7815b2c0 0x7fbf78161340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:10.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.906+0000 7fbf7caee700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbf7806a7e0 0x7fbf7806ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:10.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.906+0000 7fbf7caee700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf7815ecb0 con 0x7fbf7806e570 2023-10-16T12:11:10.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.906+0000 7fbf7caee700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf7806b1b0 con 0x7fbf7815b2c0 2023-10-16T12:11:10.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.906+0000 7fbf7caee700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf78161880 con 0x7fbf7806a7e0 2023-10-16T12:11:10.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.906+0000 7fbf76d9d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbf7806a7e0 0x7fbf7806ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:10.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.906+0000 7fbf7659c700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fbf7815b2c0 0x7fbf78161340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:10.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.906+0000 7fbf76d9d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbf7806a7e0 0x7fbf7806ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:50436/0 (socket says 172.21.15.16:50436) 2023-10-16T12:11:10.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.906+0000 7fbf76d9d700 1 -- 172.21.15.16:0/981544739 learned_addr learned my addr 172.21.15.16:0/981544739 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:10.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.906+0000 7fbf7659c700 1 -- 172.21.15.16:0/981544739 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbf7806a7e0 msgr2=0x7fbf7806ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:10.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.906+0000 7fbf7659c700 1 --2- 172.21.15.16:0/981544739 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbf7806a7e0 0x7fbf7806ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:10.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.906+0000 7fbf7659c700 1 -- 172.21.15.16:0/981544739 >> v1:172.21.15.49:6789/0 conn(0x7fbf7806e570 legacy=0x7fbf7815e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:10.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.906+0000 7fbf7659c700 1 -- 172.21.15.16:0/981544739 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf7815b6a0 con 0x7fbf7815b2c0 2023-10-16T12:11:10.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.906+0000 7fbf76d9d700 1 --2- 172.21.15.16:0/981544739 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbf7806a7e0 0x7fbf7806ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:11:10.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.906+0000 7fbf7659c700 1 --2- 172.21.15.16:0/981544739 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbf7815b2c0 0x7fbf78161340 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fbf68009840 tx=0x7fbf68009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6b7898aac6fd1744 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:10.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.907+0000 7fbf7559a700 1 -- 172.21.15.16:0/981544739 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf68011830 con 0x7fbf7815b2c0 2023-10-16T12:11:10.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.907+0000 7fbf7559a700 1 -- 172.21.15.16:0/981544739 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf68011990 con 0x7fbf7815b2c0 2023-10-16T12:11:10.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.907+0000 7fbf7559a700 1 -- 172.21.15.16:0/981544739 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf68011d20 con 0x7fbf7815b2c0 2023-10-16T12:11:10.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.907+0000 7fbf7caee700 1 -- 172.21.15.16:0/981544739 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbf7815b2c0 msgr2=0x7fbf78161340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:10.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.907+0000 7fbf7caee700 1 --2- 172.21.15.16:0/981544739 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbf7815b2c0 0x7fbf78161340 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fbf68009840 tx=0x7fbf68009b50 comp rx=0 tx=0).stop 2023-10-16T12:11:10.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.907+0000 7fbf7caee700 1 -- 172.21.15.16:0/981544739 shutdown_connections 2023-10-16T12:11:10.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.907+0000 7fbf7caee700 1 --2- 172.21.15.16:0/981544739 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbf7806a7e0 0x7fbf7806ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:10.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.907+0000 7fbf7caee700 1 --2- 172.21.15.16:0/981544739 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbf7815b2c0 0x7fbf78161340 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:10.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.907+0000 7fbf7caee700 1 -- 172.21.15.16:0/981544739 >> 172.21.15.16:0/981544739 conn(0x7fbf780b1f40 msgr2=0x7fbf780b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:10.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.907+0000 7fbf7caee700 1 -- 172.21.15.16:0/981544739 shutdown_connections 2023-10-16T12:11:10.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.907+0000 7fbf7caee700 1 -- 172.21.15.16:0/981544739 wait complete. 2023-10-16T12:11:10.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.909+0000 7fbf7caee700 1 Processor -- start 2023-10-16T12:11:10.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.909+0000 7fbf7caee700 1 -- start start 2023-10-16T12:11:10.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.909+0000 7fbf7caee700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbf7806a7e0 0x7fbf780cbc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:10.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.909+0000 7fbf7caee700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf7806e570 0x7fbf780cc180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:10.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.909+0000 7fbf7caee700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbf7815b2c0 0x7fbf780c6290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:10.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.910+0000 7fbf7caee700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf781f3f60 con 0x7fbf7806e570 2023-10-16T12:11:10.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.910+0000 7fbf7caee700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf781f3c80 con 0x7fbf7815b2c0 2023-10-16T12:11:10.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.910+0000 7fbf7caee700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf7814b9e0 con 0x7fbf7806a7e0 2023-10-16T12:11:10.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.910+0000 7fbf75d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf7806e570 0x7fbf780cc180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:10.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.910+0000 7fbf75d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf7806e570 0x7fbf780cc180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33950/0 (socket says 172.21.15.16:33950) 2023-10-16T12:11:10.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.910+0000 7fbf75d9b700 1 -- 172.21.15.16:0/3160446409 learned_addr learned my addr 172.21.15.16:0/3160446409 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:10.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.910+0000 7fbf76d9d700 1 --2- 172.21.15.16:0/3160446409 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbf7815b2c0 0x7fbf780c6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:10.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.910+0000 7fbf7659c700 1 --2- 172.21.15.16:0/3160446409 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbf7806a7e0 0x7fbf780cbc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:10.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.910+0000 7fbf75d9b700 1 -- 172.21.15.16:0/3160446409 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbf7806a7e0 msgr2=0x7fbf780cbc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:10.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.910+0000 7fbf75d9b700 1 --2- 172.21.15.16:0/3160446409 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbf7806a7e0 0x7fbf780cbc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:10.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.910+0000 7fbf75d9b700 1 -- 172.21.15.16:0/3160446409 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbf7815b2c0 msgr2=0x7fbf780c6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:10.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.910+0000 7fbf75d9b700 1 --2- 172.21.15.16:0/3160446409 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbf7815b2c0 0x7fbf780c6290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:10.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.910+0000 7fbf75d9b700 1 -- 172.21.15.16:0/3160446409 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf70009b30 con 0x7fbf7806e570 2023-10-16T12:11:10.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.910+0000 7fbf7659c700 1 --2- 172.21.15.16:0/3160446409 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbf7806a7e0 0x7fbf780cbc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:11:10.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.910+0000 7fbf75d9b700 1 --2- 172.21.15.16:0/3160446409 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf7806e570 0x7fbf780cc180 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c00c450 tx=0x7fbf6c00c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:10.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.910+0000 7fbf66ffd700 1 -- 172.21.15.16:0/3160446409 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf6c0052d0 con 0x7fbf7806e570 2023-10-16T12:11:10.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.910+0000 7fbf7caee700 1 -- 172.21.15.16:0/3160446409 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf680094a0 con 0x7fbf7806e570 2023-10-16T12:11:10.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.910+0000 7fbf66ffd700 1 -- 172.21.15.16:0/3160446409 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf6c0011a0 con 0x7fbf7806e570 2023-10-16T12:11:10.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.910+0000 7fbf7caee700 1 -- 172.21.15.16:0/3160446409 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf7814bfa0 con 0x7fbf7806e570 2023-10-16T12:11:10.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.910+0000 7fbf66ffd700 1 -- 172.21.15.16:0/3160446409 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf6c003680 con 0x7fbf7806e570 2023-10-16T12:11:10.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.911+0000 7fbf7caee700 1 -- 172.21.15.16:0/3160446409 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf7814b840 con 0x7fbf7806e570 2023-10-16T12:11:10.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.912+0000 7fbf66ffd700 1 -- 172.21.15.16:0/3160446409 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fbf6c0037e0 con 0x7fbf7806e570 2023-10-16T12:11:10.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.912+0000 7fbf66ffd700 1 --2- 172.21.15.16:0/3160446409 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fbf5806af00 0x7fbf5806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:10.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.912+0000 7fbf66ffd700 1 -- 172.21.15.16:0/3160446409 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fbf6c088c60 con 0x7fbf7806e570 2023-10-16T12:11:10.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.912+0000 7fbf7659c700 1 --2- 172.21.15.16:0/3160446409 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fbf5806af00 0x7fbf5806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:10.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.913+0000 7fbf7659c700 1 --2- 172.21.15.16:0/3160446409 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fbf5806af00 0x7fbf5806d3c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fbf68000c00 tx=0x7fbf68008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:10.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:10.917+0000 7fbf66ffd700 1 -- 172.21.15.16:0/3160446409 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbf7814b840 con 0x7fbf7806e570 2023-10-16T12:11:11.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.118+0000 7fbf7caee700 1 -- 172.21.15.16:0/3160446409 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fbf7814b840 con 0x7fbf7806e570 2023-10-16T12:11:11.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.118+0000 7fbf66ffd700 1 -- 172.21.15.16:0/3160446409 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) v1 ==== 104+0+11 (secure 0 0 0) 0x7fbf7814b840 con 0x7fbf7806e570 2023-10-16T12:11:11.121 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-10-16T12:11:11.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.120+0000 7fbf7caee700 1 -- 172.21.15.16:0/3160446409 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fbf5806af00 msgr2=0x7fbf5806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:11.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.120+0000 7fbf7caee700 1 --2- 172.21.15.16:0/3160446409 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fbf5806af00 0x7fbf5806d3c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fbf68000c00 tx=0x7fbf68008030 comp rx=0 tx=0).stop 2023-10-16T12:11:11.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.120+0000 7fbf7caee700 1 -- 172.21.15.16:0/3160446409 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf7806e570 msgr2=0x7fbf780cc180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:11.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.120+0000 7fbf7caee700 1 --2- 172.21.15.16:0/3160446409 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf7806e570 0x7fbf780cc180 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c00c450 tx=0x7fbf6c00c810 comp rx=0 tx=0).stop 2023-10-16T12:11:11.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.120+0000 7fbf7caee700 1 -- 172.21.15.16:0/3160446409 shutdown_connections 2023-10-16T12:11:11.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.120+0000 7fbf7caee700 1 --2- 172.21.15.16:0/3160446409 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fbf5806af00 0x7fbf5806d3c0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:11.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.120+0000 7fbf7caee700 1 --2- 172.21.15.16:0/3160446409 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbf7815b2c0 0x7fbf780c6290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:11.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.120+0000 7fbf7caee700 1 --2- 172.21.15.16:0/3160446409 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbf7806a7e0 0x7fbf780cbc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:11.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.120+0000 7fbf7caee700 1 --2- 172.21.15.16:0/3160446409 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf7806e570 0x7fbf780cc180 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:11.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.120+0000 7fbf7caee700 1 -- 172.21.15.16:0/3160446409 >> 172.21.15.16:0/3160446409 conn(0x7fbf780b1f40 msgr2=0x7fbf7815baf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:11.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.120+0000 7fbf7caee700 1 -- 172.21.15.16:0/3160446409 shutdown_connections 2023-10-16T12:11:11.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.120+0000 7fbf7caee700 1 -- 172.21.15.16:0/3160446409 wait complete. 2023-10-16T12:11:11.134 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T12:11:11.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.260+0000 7f89056ce700 1 Processor -- start 2023-10-16T12:11:11.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.261+0000 7f89056ce700 1 -- start start 2023-10-16T12:11:11.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.261+0000 7f89056ce700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f890015d8d0 0x7f89000b2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:11.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.261+0000 7f89056ce700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f89000ce350 0x7f89000ce780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:11.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.261+0000 7f89056ce700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89000cecc0 con 0x7f89000b8fa0 2023-10-16T12:11:11.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.261+0000 7f89056ce700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89000d0bb0 con 0x7f890015d8d0 2023-10-16T12:11:11.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.261+0000 7f89056ce700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f890015dcb0 con 0x7f89000ce350 2023-10-16T12:11:11.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.262+0000 7f88fe7fc700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f89000b8fa0 0x7f89000b6720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51462/0 (socket says 172.21.15.16:51462) 2023-10-16T12:11:11.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.262+0000 7f88feffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f890015d8d0 0x7f89000b2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:11.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.262+0000 7f88fe7fc700 1 -- 172.21.15.16:0/3614861559 learned_addr learned my addr 172.21.15.16:0/3614861559 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:11.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.262+0000 7f88ff7fe700 1 --2- 172.21.15.16:0/3614861559 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f89000ce350 0x7f89000ce780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:11.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.262+0000 7f88feffd700 1 -- 172.21.15.16:0/3614861559 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f89000ce350 msgr2=0x7f89000ce780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:11.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.262+0000 7f88feffd700 1 --2- 172.21.15.16:0/3614861559 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f89000ce350 0x7f89000ce780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:11.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.262+0000 7f88feffd700 1 -- 172.21.15.16:0/3614861559 >> v1:172.21.15.16:6789/0 conn(0x7f89000b8fa0 legacy=0x7f89000b6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:11.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.262+0000 7f88feffd700 1 -- 172.21.15.16:0/3614861559 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89000b2f30 con 0x7f890015d8d0 2023-10-16T12:11:11.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.262+0000 7f88fdffb700 1 -- 172.21.15.16:0/3614861559 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3596217257 0 0) 0x7f89000cecc0 con 0x7f89000b8fa0 2023-10-16T12:11:11.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.262+0000 7f88feffd700 1 --2- 172.21.15.16:0/3614861559 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f890015d8d0 0x7f89000b2830 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f88f4009840 tx=0x7f88f4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87b9627d18fec5e5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:11.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.262+0000 7f88fdffb700 1 -- 172.21.15.16:0/3614861559 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f4011830 con 0x7f890015d8d0 2023-10-16T12:11:11.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.262+0000 7f88fdffb700 1 -- 172.21.15.16:0/3614861559 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88f4011990 con 0x7f890015d8d0 2023-10-16T12:11:11.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.262+0000 7f88fdffb700 1 -- 172.21.15.16:0/3614861559 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f4011d20 con 0x7f890015d8d0 2023-10-16T12:11:11.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.263+0000 7f89056ce700 1 -- 172.21.15.16:0/3614861559 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f890015d8d0 msgr2=0x7f89000b2830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:11.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.263+0000 7f89056ce700 1 --2- 172.21.15.16:0/3614861559 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f890015d8d0 0x7f89000b2830 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f88f4009840 tx=0x7f88f4009b50 comp rx=0 tx=0).stop 2023-10-16T12:11:11.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.263+0000 7f89056ce700 1 -- 172.21.15.16:0/3614861559 shutdown_connections 2023-10-16T12:11:11.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.263+0000 7f89056ce700 1 --2- 172.21.15.16:0/3614861559 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f89000ce350 0x7f89000ce780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:11.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.263+0000 7f89056ce700 1 --2- 172.21.15.16:0/3614861559 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f890015d8d0 0x7f89000b2830 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:11.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.263+0000 7f89056ce700 1 -- 172.21.15.16:0/3614861559 >> 172.21.15.16:0/3614861559 conn(0x7f8900072330 msgr2=0x7f8900072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:11.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.263+0000 7f89056ce700 1 -- 172.21.15.16:0/3614861559 shutdown_connections 2023-10-16T12:11:11.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.263+0000 7f89056ce700 1 -- 172.21.15.16:0/3614861559 wait complete. 2023-10-16T12:11:11.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.264+0000 7f89056ce700 1 Processor -- start 2023-10-16T12:11:11.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.264+0000 7f89056ce700 1 -- start start 2023-10-16T12:11:11.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.265+0000 7f89056ce700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f89000b8fa0 0x7f89000ccf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:11.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.265+0000 7f89056ce700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89000ce350 0x7f89001f5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:11.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.265+0000 7f89056ce700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f890015d8d0 0x7f89001f5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:11.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.265+0000 7f89056ce700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89000c8940 con 0x7f89000ce350 2023-10-16T12:11:11.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.265+0000 7f89056ce700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89001f6530 con 0x7f89000b8fa0 2023-10-16T12:11:11.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.265+0000 7f89056ce700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89000c7cc0 con 0x7f890015d8d0 2023-10-16T12:11:11.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.265+0000 7f88fe7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89000ce350 0x7f89001f5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:11.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.265+0000 7f88fe7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89000ce350 0x7f89001f5ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33964/0 (socket says 172.21.15.16:33964) 2023-10-16T12:11:11.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.265+0000 7f88fe7fc700 1 -- 172.21.15.16:0/1480470628 learned_addr learned my addr 172.21.15.16:0/1480470628 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:11.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.265+0000 7f88ff7fe700 1 --2- 172.21.15.16:0/1480470628 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f890015d8d0 0x7f89001f5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:11.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.265+0000 7f88feffd700 1 --2- 172.21.15.16:0/1480470628 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f89000b8fa0 0x7f89000ccf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:11.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.265+0000 7f88fe7fc700 1 -- 172.21.15.16:0/1480470628 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f890015d8d0 msgr2=0x7f89001f5ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:11.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.265+0000 7f88fe7fc700 1 --2- 172.21.15.16:0/1480470628 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f890015d8d0 0x7f89001f5ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:11.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.265+0000 7f88fe7fc700 1 -- 172.21.15.16:0/1480470628 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f89000b8fa0 msgr2=0x7f89000ccf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:11.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.265+0000 7f88fe7fc700 1 --2- 172.21.15.16:0/1480470628 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f89000b8fa0 0x7f89000ccf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:11.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.265+0000 7f88fe7fc700 1 -- 172.21.15.16:0/1480470628 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88f40094a0 con 0x7f89000ce350 2023-10-16T12:11:11.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.265+0000 7f88fe7fc700 1 --2- 172.21.15.16:0/1480470628 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89000ce350 0x7f89001f5ab0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f88f000c550 tx=0x7f88f000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:11.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.266+0000 7f88e77fe700 1 -- 172.21.15.16:0/1480470628 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f0004c80 con 0x7f89000ce350 2023-10-16T12:11:11.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.266+0000 7f88e77fe700 1 -- 172.21.15.16:0/1480470628 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88f0004e00 con 0x7f89000ce350 2023-10-16T12:11:11.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.266+0000 7f89056ce700 1 -- 172.21.15.16:0/1480470628 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89000c7fa0 con 0x7f89000ce350 2023-10-16T12:11:11.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.266+0000 7f88e77fe700 1 -- 172.21.15.16:0/1480470628 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f0007310 con 0x7f89000ce350 2023-10-16T12:11:11.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.266+0000 7f89056ce700 1 -- 172.21.15.16:0/1480470628 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89000c8560 con 0x7f89000ce350 2023-10-16T12:11:11.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.267+0000 7f89056ce700 1 -- 172.21.15.16:0/1480470628 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f890004ebb0 con 0x7f89000ce350 2023-10-16T12:11:11.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.267+0000 7f88e77fe700 1 -- 172.21.15.16:0/1480470628 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f88f0007470 con 0x7f89000ce350 2023-10-16T12:11:11.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.267+0000 7f88e77fe700 1 --2- 172.21.15.16:0/1480470628 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f88ec06aed0 0x7f88ec06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:11.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.268+0000 7f88e77fe700 1 -- 172.21.15.16:0/1480470628 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f88f00893e0 con 0x7f89000ce350 2023-10-16T12:11:11.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.268+0000 7f88feffd700 1 --2- 172.21.15.16:0/1480470628 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f88ec06aed0 0x7f88ec06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:11.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.268+0000 7f88feffd700 1 --2- 172.21.15.16:0/1480470628 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f88ec06aed0 0x7f88ec06d390 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f88f40090b0 tx=0x7f88f401c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:11.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.272+0000 7f88e77fe700 1 -- 172.21.15.16:0/1480470628 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f88f0054230 con 0x7f89000ce350 2023-10-16T12:11:11.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.477+0000 7f89056ce700 1 -- 172.21.15.16:0/1480470628 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f89000d3da0 con 0x7f89000ce350 2023-10-16T12:11:11.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.477+0000 7f88e77fe700 1 -- 172.21.15.16:0/1480470628 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) v1 ==== 100+0+11 (secure 0 0 0) 0x7f88f0057850 con 0x7f89000ce350 2023-10-16T12:11:11.479 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-10-16T12:11:11.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.479+0000 7f89056ce700 1 -- 172.21.15.16:0/1480470628 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f88ec06aed0 msgr2=0x7f88ec06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:11.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.479+0000 7f89056ce700 1 --2- 172.21.15.16:0/1480470628 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f88ec06aed0 0x7f88ec06d390 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f88f40090b0 tx=0x7f88f401c040 comp rx=0 tx=0).stop 2023-10-16T12:11:11.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.479+0000 7f89056ce700 1 -- 172.21.15.16:0/1480470628 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89000ce350 msgr2=0x7f89001f5ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:11.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.479+0000 7f89056ce700 1 --2- 172.21.15.16:0/1480470628 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89000ce350 0x7f89001f5ab0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f88f000c550 tx=0x7f88f000c860 comp rx=0 tx=0).stop 2023-10-16T12:11:11.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.479+0000 7f89056ce700 1 -- 172.21.15.16:0/1480470628 shutdown_connections 2023-10-16T12:11:11.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.479+0000 7f89056ce700 1 --2- 172.21.15.16:0/1480470628 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f88ec06aed0 0x7f88ec06d390 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:11.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.479+0000 7f89056ce700 1 --2- 172.21.15.16:0/1480470628 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f890015d8d0 0x7f89001f5ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:11.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.479+0000 7f89056ce700 1 --2- 172.21.15.16:0/1480470628 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f89000b8fa0 0x7f89000ccf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:11.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.479+0000 7f89056ce700 1 --2- 172.21.15.16:0/1480470628 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89000ce350 0x7f89001f5ab0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:11.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.479+0000 7f89056ce700 1 -- 172.21.15.16:0/1480470628 >> 172.21.15.16:0/1480470628 conn(0x7f8900072330 msgr2=0x7f89000b7b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:11.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.479+0000 7f89056ce700 1 -- 172.21.15.16:0/1480470628 shutdown_connections 2023-10-16T12:11:11.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.479+0000 7f89056ce700 1 -- 172.21.15.16:0/1480470628 wait complete. 2023-10-16T12:11:11.493 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-16T12:11:11.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.619+0000 7fb283f55700 1 Processor -- start 2023-10-16T12:11:11.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.619+0000 7fb283f55700 1 -- start start 2023-10-16T12:11:11.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.620+0000 7fb283f55700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fb27c06e170 0x7fb27c0cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:11.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.620+0000 7fb283f55700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb27c0cef70 con 0x7fb27c15b070 2023-10-16T12:11:11.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.620+0000 7fb283f55700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb27c0ce360 con 0x7fb27c0cb880 2023-10-16T12:11:11.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.620+0000 7fb283f55700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb27c0ce640 con 0x7fb27c06e170 2023-10-16T12:11:11.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.620+0000 7fb281cf1700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb27c15b070 0x7fb27c15b450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51476/0 (socket says 172.21.15.16:51476) 2023-10-16T12:11:11.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.620+0000 7fb281cf1700 1 -- 172.21.15.16:0/4119344084 learned_addr learned my addr 172.21.15.16:0/4119344084 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:11.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.621+0000 7fb2824f2700 1 --2- 172.21.15.16:0/4119344084 >> v2:172.21.15.49:3300/0 conn(0x7fb27c06e170 0x7fb27c0cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:11.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.621+0000 7fb280cef700 1 -- 172.21.15.16:0/4119344084 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1511419301 0 0) 0x7fb27c0cef70 con 0x7fb27c15b070 2023-10-16T12:11:11.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.621+0000 7fb280cef700 1 -- 172.21.15.16:0/4119344084 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2680035b0 con 0x7fb27c15b070 2023-10-16T12:11:11.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.621+0000 7fb2824f2700 1 -- 172.21.15.16:0/4119344084 >> v1:172.21.15.49:6789/0 conn(0x7fb27c0cb880 legacy=0x7fb27c06da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:11.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.621+0000 7fb2824f2700 1 -- 172.21.15.16:0/4119344084 >> v1:172.21.15.16:6789/0 conn(0x7fb27c15b070 legacy=0x7fb27c15b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:11.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.621+0000 7fb2824f2700 1 -- 172.21.15.16:0/4119344084 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb27c0ce920 con 0x7fb27c06e170 2023-10-16T12:11:11.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.621+0000 7fb2824f2700 1 --2- 172.21.15.16:0/4119344084 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb27c06e170 0x7fb27c0cbd00 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb274009d90 tx=0x7fb27400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=839e2eecfdbf2a0f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:11.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.621+0000 7fb280cef700 1 -- 172.21.15.16:0/4119344084 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb27400e070 con 0x7fb27c06e170 2023-10-16T12:11:11.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.621+0000 7fb280cef700 1 -- 172.21.15.16:0/4119344084 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb27400bb20 con 0x7fb27c06e170 2023-10-16T12:11:11.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.621+0000 7fb280cef700 1 -- 172.21.15.16:0/4119344084 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb274012430 con 0x7fb27c06e170 2023-10-16T12:11:11.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.622+0000 7fb283f55700 1 -- 172.21.15.16:0/4119344084 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb27c06e170 msgr2=0x7fb27c0cbd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:11.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.622+0000 7fb283f55700 1 --2- 172.21.15.16:0/4119344084 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb27c06e170 0x7fb27c0cbd00 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb274009d90 tx=0x7fb27400b330 comp rx=0 tx=0).stop 2023-10-16T12:11:11.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.622+0000 7fb283f55700 1 -- 172.21.15.16:0/4119344084 shutdown_connections 2023-10-16T12:11:11.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.622+0000 7fb283f55700 1 --2- 172.21.15.16:0/4119344084 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb27c06e170 0x7fb27c0cbd00 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:11.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.622+0000 7fb283f55700 1 -- 172.21.15.16:0/4119344084 >> 172.21.15.16:0/4119344084 conn(0x7fb27c0b1f40 msgr2=0x7fb27c0b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:11.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.622+0000 7fb283f55700 1 -- 172.21.15.16:0/4119344084 shutdown_connections 2023-10-16T12:11:11.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.622+0000 7fb283f55700 1 -- 172.21.15.16:0/4119344084 wait complete. 2023-10-16T12:11:11.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.624+0000 7fb283f55700 1 Processor -- start 2023-10-16T12:11:11.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.624+0000 7fb283f55700 1 -- start start 2023-10-16T12:11:11.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.625+0000 7fb283f55700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb27c06e170 0x7fb27c15ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:11.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.625+0000 7fb283f55700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb27c0cb880 0x7fb27c1f25a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:11.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.625+0000 7fb283f55700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb27c15b070 0x7fb27c157930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:11.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.625+0000 7fb283f55700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb27c15d3f0 con 0x7fb27c15b070 2023-10-16T12:11:11.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.625+0000 7fb283f55700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb27c1f2ae0 con 0x7fb27c0cb880 2023-10-16T12:11:11.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.625+0000 7fb283f55700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb27c155190 con 0x7fb27c06e170 2023-10-16T12:11:11.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.625+0000 7fb2824f2700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb27c15b070 0x7fb27c157930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:11.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.625+0000 7fb2824f2700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb27c15b070 0x7fb27c157930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33980/0 (socket says 172.21.15.16:33980) 2023-10-16T12:11:11.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.625+0000 7fb2824f2700 1 -- 172.21.15.16:0/1049999645 learned_addr learned my addr 172.21.15.16:0/1049999645 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:11.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.625+0000 7fb281cf1700 1 --2- 172.21.15.16:0/1049999645 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb27c06e170 0x7fb27c15ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:11.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.625+0000 7fb2814f0700 1 --2- 172.21.15.16:0/1049999645 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb27c0cb880 0x7fb27c1f25a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:11.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.625+0000 7fb2824f2700 1 -- 172.21.15.16:0/1049999645 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb27c06e170 msgr2=0x7fb27c15ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:11.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.625+0000 7fb2824f2700 1 --2- 172.21.15.16:0/1049999645 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb27c06e170 0x7fb27c15ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:11.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.625+0000 7fb2824f2700 1 -- 172.21.15.16:0/1049999645 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb27c0cb880 msgr2=0x7fb27c1f25a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:11.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.625+0000 7fb2824f2700 1 --2- 172.21.15.16:0/1049999645 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb27c0cb880 0x7fb27c1f25a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:11.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.625+0000 7fb2824f2700 1 -- 172.21.15.16:0/1049999645 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb274009a70 con 0x7fb27c15b070 2023-10-16T12:11:11.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.625+0000 7fb2824f2700 1 --2- 172.21.15.16:0/1049999645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb27c15b070 0x7fb27c157930 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb274013000 tx=0x7fb274004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:11.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.625+0000 7fb2727fc700 1 -- 172.21.15.16:0/1049999645 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb274012430 con 0x7fb27c15b070 2023-10-16T12:11:11.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.625+0000 7fb2727fc700 1 -- 172.21.15.16:0/1049999645 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb274020070 con 0x7fb27c15b070 2023-10-16T12:11:11.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.625+0000 7fb283f55700 1 -- 172.21.15.16:0/1049999645 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb27c155470 con 0x7fb27c15b070 2023-10-16T12:11:11.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.625+0000 7fb2727fc700 1 -- 172.21.15.16:0/1049999645 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb274004280 con 0x7fb27c15b070 2023-10-16T12:11:11.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.625+0000 7fb283f55700 1 -- 172.21.15.16:0/1049999645 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb27c157130 con 0x7fb27c15b070 2023-10-16T12:11:11.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.626+0000 7fb283f55700 1 -- 172.21.15.16:0/1049999645 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb27c04ebb0 con 0x7fb27c15b070 2023-10-16T12:11:11.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.627+0000 7fb2727fc700 1 -- 172.21.15.16:0/1049999645 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fb2740043e0 con 0x7fb27c15b070 2023-10-16T12:11:11.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.628+0000 7fb2727fc700 1 --2- 172.21.15.16:0/1049999645 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fb26006aed0 0x7fb26006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:11.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.628+0000 7fb2727fc700 1 -- 172.21.15.16:0/1049999645 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fb274022070 con 0x7fb27c15b070 2023-10-16T12:11:11.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.628+0000 7fb281cf1700 1 --2- 172.21.15.16:0/1049999645 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fb26006aed0 0x7fb26006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:11.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.628+0000 7fb281cf1700 1 --2- 172.21.15.16:0/1049999645 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fb26006aed0 0x7fb26006d390 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fb26c0011e0 tx=0x7fb26c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:11.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.632+0000 7fb2727fc700 1 -- 172.21.15.16:0/1049999645 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb274056ad0 con 0x7fb27c15b070 2023-10-16T12:11:11.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.834+0000 7fb283f55700 1 -- 172.21.15.16:0/1049999645 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fb27c0cc9f0 con 0x7fb27c15b070 2023-10-16T12:11:11.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.835+0000 7fb2727fc700 1 -- 172.21.15.16:0/1049999645 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) v1 ==== 103+0+11 (secure 0 0 0) 0x7fb27402c020 con 0x7fb27c15b070 2023-10-16T12:11:11.837 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-10-16T12:11:11.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.837+0000 7fb283f55700 1 -- 172.21.15.16:0/1049999645 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fb26006aed0 msgr2=0x7fb26006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:11.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.837+0000 7fb283f55700 1 --2- 172.21.15.16:0/1049999645 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fb26006aed0 0x7fb26006d390 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fb26c0011e0 tx=0x7fb26c009040 comp rx=0 tx=0).stop 2023-10-16T12:11:11.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.837+0000 7fb283f55700 1 -- 172.21.15.16:0/1049999645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb27c15b070 msgr2=0x7fb27c157930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:11.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.837+0000 7fb283f55700 1 --2- 172.21.15.16:0/1049999645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb27c15b070 0x7fb27c157930 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb274013000 tx=0x7fb274004ad0 comp rx=0 tx=0).stop 2023-10-16T12:11:11.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.837+0000 7fb283f55700 1 -- 172.21.15.16:0/1049999645 shutdown_connections 2023-10-16T12:11:11.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.837+0000 7fb283f55700 1 --2- 172.21.15.16:0/1049999645 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fb26006aed0 0x7fb26006d390 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:11.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.837+0000 7fb283f55700 1 --2- 172.21.15.16:0/1049999645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb27c15b070 0x7fb27c157930 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:11.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.837+0000 7fb283f55700 1 --2- 172.21.15.16:0/1049999645 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb27c06e170 0x7fb27c15ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:11.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.837+0000 7fb283f55700 1 --2- 172.21.15.16:0/1049999645 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb27c0cb880 0x7fb27c1f25a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:11.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.837+0000 7fb283f55700 1 -- 172.21.15.16:0/1049999645 >> 172.21.15.16:0/1049999645 conn(0x7fb27c0b1f40 msgr2=0x7fb27c0cf300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:11.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.837+0000 7fb283f55700 1 -- 172.21.15.16:0/1049999645 shutdown_connections 2023-10-16T12:11:11.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.837+0000 7fb283f55700 1 -- 172.21.15.16:0/1049999645 wait complete. 2023-10-16T12:11:11.851 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:11:11.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.974+0000 7f21457ba700 1 Processor -- start 2023-10-16T12:11:11.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.974+0000 7f21457ba700 1 -- start start 2023-10-16T12:11:11.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.975+0000 7f21457ba700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f214015b630 0x7f21400d0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:11.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.975+0000 7f21457ba700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f214015ba10 0x7f21400cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:11.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.975+0000 7f21457ba700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21400b0970 con 0x7f21400cdde0 2023-10-16T12:11:11.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.975+0000 7f21457ba700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21400cd690 con 0x7f214015ba10 2023-10-16T12:11:11.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.975+0000 7f21457ba700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21400cd970 con 0x7f214015b630 2023-10-16T12:11:11.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.975+0000 7f213e7fc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f214015ba10 0x7f21400cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:11.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.975+0000 7f213e7fc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f214015ba10 0x7f21400cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33984/0 (socket says 172.21.15.16:33984) 2023-10-16T12:11:11.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.975+0000 7f213e7fc700 1 -- 172.21.15.16:0/455130594 learned_addr learned my addr 172.21.15.16:0/455130594 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:11.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.975+0000 7f213e7fc700 1 -- 172.21.15.16:0/455130594 >> v2:172.21.15.49:3300/0 conn(0x7f214015b630 msgr2=0x7f21400d0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:11.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.976+0000 7f213effd700 1 --2- 172.21.15.16:0/455130594 >> v2:172.21.15.49:3300/0 conn(0x7f214015b630 0x7f21400d0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:11.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.976+0000 7f213e7fc700 1 --2- 172.21.15.16:0/455130594 >> v2:172.21.15.49:3300/0 conn(0x7f214015b630 0x7f21400d0380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:11.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.976+0000 7f213e7fc700 1 -- 172.21.15.16:0/455130594 >> v1:172.21.15.49:6789/0 conn(0x7f21400cdde0 legacy=0x7f21400ce1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:11.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.976+0000 7f213e7fc700 1 -- 172.21.15.16:0/455130594 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21400b0c30 con 0x7f214015ba10 2023-10-16T12:11:11.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.976+0000 7f213effd700 1 --2- 172.21.15.16:0/455130594 >> v2:172.21.15.49:3300/0 conn(0x7f214015b630 0x7f21400d0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:11:11.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.976+0000 7f213e7fc700 1 --2- 172.21.15.16:0/455130594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f214015ba10 0x7f21400cf290 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f2140067350 tx=0x7f2128009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51ac4a56c5d95c86 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:11.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.976+0000 7f213dffb700 1 -- 172.21.15.16:0/455130594 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2128004230 con 0x7f214015ba10 2023-10-16T12:11:11.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.976+0000 7f213dffb700 1 -- 172.21.15.16:0/455130594 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2128004390 con 0x7f214015ba10 2023-10-16T12:11:11.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.976+0000 7f213dffb700 1 -- 172.21.15.16:0/455130594 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21280036a0 con 0x7f214015ba10 2023-10-16T12:11:11.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.976+0000 7f21457ba700 1 -- 172.21.15.16:0/455130594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f214015ba10 msgr2=0x7f21400cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:11.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.976+0000 7f21457ba700 1 --2- 172.21.15.16:0/455130594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f214015ba10 0x7f21400cf290 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f2140067350 tx=0x7f2128009bd0 comp rx=0 tx=0).stop 2023-10-16T12:11:11.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.976+0000 7f21457ba700 1 -- 172.21.15.16:0/455130594 shutdown_connections 2023-10-16T12:11:11.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.976+0000 7f21457ba700 1 --2- 172.21.15.16:0/455130594 >> v2:172.21.15.49:3300/0 conn(0x7f214015b630 0x7f21400d0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:11.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.976+0000 7f21457ba700 1 --2- 172.21.15.16:0/455130594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f214015ba10 0x7f21400cf290 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:11.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.976+0000 7f21457ba700 1 -- 172.21.15.16:0/455130594 >> 172.21.15.16:0/455130594 conn(0x7f21400d14b0 msgr2=0x7f21400d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:11.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.977+0000 7f21457ba700 1 -- 172.21.15.16:0/455130594 shutdown_connections 2023-10-16T12:11:11.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.977+0000 7f21457ba700 1 -- 172.21.15.16:0/455130594 wait complete. 2023-10-16T12:11:11.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.978+0000 7f21457ba700 1 Processor -- start 2023-10-16T12:11:11.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.978+0000 7f21457ba700 1 -- start start 2023-10-16T12:11:11.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.979+0000 7f21457ba700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f21400cdde0 0x7f21401f2790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:11.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.979+0000 7f21457ba700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f214015b630 0x7f214015e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:11.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.979+0000 7f21457ba700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f214015ba10 0x7f214015e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:11.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.979+0000 7f21457ba700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21401f2cd0 con 0x7f214015b630 2023-10-16T12:11:11.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.979+0000 7f21457ba700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f214006c300 con 0x7f21400cdde0 2023-10-16T12:11:11.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.979+0000 7f21457ba700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f214006c5e0 con 0x7f214015ba10 2023-10-16T12:11:11.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.979+0000 7f213e7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f214015b630 0x7f214015e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:11.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.979+0000 7f213e7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f214015b630 0x7f214015e230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33990/0 (socket says 172.21.15.16:33990) 2023-10-16T12:11:11.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.979+0000 7f213e7fc700 1 -- 172.21.15.16:0/2494528013 learned_addr learned my addr 172.21.15.16:0/2494528013 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:11.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.979+0000 7f213effd700 1 --2- 172.21.15.16:0/2494528013 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f21400cdde0 0x7f21401f2790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:11.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.979+0000 7f213f7fe700 1 --2- 172.21.15.16:0/2494528013 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f214015ba10 0x7f214015e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:11.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.979+0000 7f213e7fc700 1 -- 172.21.15.16:0/2494528013 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f214015ba10 msgr2=0x7f214015e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:11.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.979+0000 7f213e7fc700 1 --2- 172.21.15.16:0/2494528013 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f214015ba10 0x7f214015e770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:11.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.979+0000 7f213e7fc700 1 -- 172.21.15.16:0/2494528013 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f21400cdde0 msgr2=0x7f21401f2790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:11.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.979+0000 7f213e7fc700 1 --2- 172.21.15.16:0/2494528013 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f21400cdde0 0x7f21401f2790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:11.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.979+0000 7f213e7fc700 1 -- 172.21.15.16:0/2494528013 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2128009550 con 0x7f214015b630 2023-10-16T12:11:11.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.979+0000 7f213e7fc700 1 --2- 172.21.15.16:0/2494528013 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f214015b630 0x7f214015e230 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f2140072680 tx=0x7f2128004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:11.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.979+0000 7f21377fe700 1 -- 172.21.15.16:0/2494528013 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2128003ea0 con 0x7f214015b630 2023-10-16T12:11:11.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.979+0000 7f21377fe700 1 -- 172.21.15.16:0/2494528013 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2128011630 con 0x7f214015b630 2023-10-16T12:11:11.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.980+0000 7f21377fe700 1 -- 172.21.15.16:0/2494528013 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21280119e0 con 0x7f214015b630 2023-10-16T12:11:11.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.980+0000 7f21457ba700 1 -- 172.21.15.16:0/2494528013 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f214006c8c0 con 0x7f214015b630 2023-10-16T12:11:11.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.980+0000 7f21457ba700 1 -- 172.21.15.16:0/2494528013 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f214006ce80 con 0x7f214015b630 2023-10-16T12:11:11.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.980+0000 7f21457ba700 1 -- 172.21.15.16:0/2494528013 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f214004ebb0 con 0x7f214015b630 2023-10-16T12:11:11.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.981+0000 7f21377fe700 1 -- 172.21.15.16:0/2494528013 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f2128011b40 con 0x7f214015b630 2023-10-16T12:11:11.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.981+0000 7f21377fe700 1 --2- 172.21.15.16:0/2494528013 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f212006af00 0x7f212006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:11.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.981+0000 7f21377fe700 1 -- 172.21.15.16:0/2494528013 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f212808ad60 con 0x7f214015b630 2023-10-16T12:11:11.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.982+0000 7f213effd700 1 --2- 172.21.15.16:0/2494528013 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f212006af00 0x7f212006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:11.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.982+0000 7f213effd700 1 --2- 172.21.15.16:0/2494528013 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f212006af00 0x7f212006d3c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f2130006fd0 tx=0x7f2130008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:11.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:11.986+0000 7f21377fe700 1 -- 172.21.15.16:0/2494528013 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2128055cd0 con 0x7f214015b630 2023-10-16T12:11:12.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.240+0000 7f21457ba700 1 -- 172.21.15.16:0/2494528013 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f214006b5d0 con 0x7f214015b630 2023-10-16T12:11:12.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.240+0000 7f21377fe700 1 -- 172.21.15.16:0/2494528013 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4909 (secure 0 0 0) 0x7f21280592f0 con 0x7f214015b630 2023-10-16T12:11:12.243 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:11:12.243 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/2979266800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":2979266800},{"type":"v1","addr":"172.21.15.49:6834","nonce":2979266800}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4251,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/1398336801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1398336801},{"type":"v1","addr":"172.21.15.16:6836","nonce":1398336801}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4254,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3168304671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":3168304671},{"type":"v1","addr":"172.21.15.16:6837","nonce":3168304671}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:10:39.030243+0000","modified":"2023-10-16T12:11:02.593358+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4269":{"gid":4269,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.49:6835/2892536404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":2892536404},{"type":"v1","addr":"172.21.15.49:6835","nonce":2892536404}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T12:11:12.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.243+0000 7f21457ba700 1 -- 172.21.15.16:0/2494528013 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f212006af00 msgr2=0x7f212006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:12.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.243+0000 7f21457ba700 1 --2- 172.21.15.16:0/2494528013 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f212006af00 0x7f212006d3c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f2130006fd0 tx=0x7f2130008040 comp rx=0 tx=0).stop 2023-10-16T12:11:12.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.243+0000 7f21457ba700 1 -- 172.21.15.16:0/2494528013 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f214015b630 msgr2=0x7f214015e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:12.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.243+0000 7f21457ba700 1 --2- 172.21.15.16:0/2494528013 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f214015b630 0x7f214015e230 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f2140072680 tx=0x7f2128004750 comp rx=0 tx=0).stop 2023-10-16T12:11:12.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.243+0000 7f21457ba700 1 -- 172.21.15.16:0/2494528013 shutdown_connections 2023-10-16T12:11:12.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.243+0000 7f21457ba700 1 --2- 172.21.15.16:0/2494528013 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f212006af00 0x7f212006d3c0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:12.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.243+0000 7f21457ba700 1 --2- 172.21.15.16:0/2494528013 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f214015ba10 0x7f214015e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:12.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.243+0000 7f21457ba700 1 --2- 172.21.15.16:0/2494528013 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f21400cdde0 0x7f21401f2790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:12.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.243+0000 7f21457ba700 1 --2- 172.21.15.16:0/2494528013 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f214015b630 0x7f214015e230 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:12.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.243+0000 7f21457ba700 1 -- 172.21.15.16:0/2494528013 >> 172.21.15.16:0/2494528013 conn(0x7f21400d14b0 msgr2=0x7f21400d1890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:12.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.243+0000 7f21457ba700 1 -- 172.21.15.16:0/2494528013 shutdown_connections 2023-10-16T12:11:12.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.243+0000 7f21457ba700 1 -- 172.21.15.16:0/2494528013 wait complete. 2023-10-16T12:11:12.248 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 9 2023-10-16T12:11:12.257 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:11:12.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.380+0000 7f97e3f8c700 1 Processor -- start 2023-10-16T12:11:12.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.381+0000 7f97e3f8c700 1 -- start start 2023-10-16T12:11:12.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.381+0000 7f97e3f8c700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f97dc0cf4a0 0x7f97dc0726e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:12.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.381+0000 7f97e3f8c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f97dc0cf880 0x7f97dc0b52f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:12.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.381+0000 7f97e3f8c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f97dc0b5830 0x7f97dc15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:12.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.381+0000 7f97e3f8c700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97dc15bc80 con 0x7f97dc0b5830 2023-10-16T12:11:12.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.381+0000 7f97e3f8c700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97dc072c20 con 0x7f97dc0cf4a0 2023-10-16T12:11:12.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.381+0000 7f97e3f8c700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97dc15b070 con 0x7f97dc0cf880 2023-10-16T12:11:12.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.382+0000 7f97e2529700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f97dc0b5830 0x7f97dc15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:12.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.382+0000 7f97e2529700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f97dc0b5830 0x7f97dc15a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34000/0 (socket says 172.21.15.16:34000) 2023-10-16T12:11:12.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.382+0000 7f97e2529700 1 -- 172.21.15.16:0/1697675134 learned_addr learned my addr 172.21.15.16:0/1697675134 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:12.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.382+0000 7f97e1527700 1 --2- 172.21.15.16:0/1697675134 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f97dc0cf880 0x7f97dc0b52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:12.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.382+0000 7f97e1d28700 1 --2- 172.21.15.16:0/1697675134 >> v2:172.21.15.49:3300/0 conn(0x7f97dc0cf4a0 0x7f97dc0726e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:12.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.382+0000 7f97e2529700 1 -- 172.21.15.16:0/1697675134 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f97dc0cf880 msgr2=0x7f97dc0b52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:12.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.382+0000 7f97e2529700 1 --2- 172.21.15.16:0/1697675134 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f97dc0cf880 0x7f97dc0b52f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:12.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.382+0000 7f97e2529700 1 -- 172.21.15.16:0/1697675134 >> v2:172.21.15.49:3300/0 conn(0x7f97dc0cf4a0 msgr2=0x7f97dc0726e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:12.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.382+0000 7f97e2529700 1 --2- 172.21.15.16:0/1697675134 >> v2:172.21.15.49:3300/0 conn(0x7f97dc0cf4a0 0x7f97dc0726e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:12.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.382+0000 7f97e2529700 1 -- 172.21.15.16:0/1697675134 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97dc15b350 con 0x7f97dc0b5830 2023-10-16T12:11:12.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.382+0000 7f97e1d28700 1 --2- 172.21.15.16:0/1697675134 >> v2:172.21.15.49:3300/0 conn(0x7f97dc0cf4a0 0x7f97dc0726e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:11:12.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.382+0000 7f97e2529700 1 --2- 172.21.15.16:0/1697675134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97dc0b5830 0x7f97dc15a090 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f97dc0b2d20 tx=0x7f97d400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75ea3fe5f46aebae server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:12.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.382+0000 7f97e0d26700 1 -- 172.21.15.16:0/1697675134 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d400e070 con 0x7f97dc0b5830 2023-10-16T12:11:12.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.382+0000 7f97e0d26700 1 -- 172.21.15.16:0/1697675134 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97d400b9c0 con 0x7f97dc0b5830 2023-10-16T12:11:12.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.382+0000 7f97e0d26700 1 -- 172.21.15.16:0/1697675134 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d400bd70 con 0x7f97dc0b5830 2023-10-16T12:11:12.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.383+0000 7f97e3f8c700 1 -- 172.21.15.16:0/1697675134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97dc0b5830 msgr2=0x7f97dc15a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:12.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.383+0000 7f97e3f8c700 1 --2- 172.21.15.16:0/1697675134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97dc0b5830 0x7f97dc15a090 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f97dc0b2d20 tx=0x7f97d400b330 comp rx=0 tx=0).stop 2023-10-16T12:11:12.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.383+0000 7f97e3f8c700 1 -- 172.21.15.16:0/1697675134 shutdown_connections 2023-10-16T12:11:12.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.383+0000 7f97e3f8c700 1 --2- 172.21.15.16:0/1697675134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97dc0b5830 0x7f97dc15a090 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:12.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.383+0000 7f97e3f8c700 1 --2- 172.21.15.16:0/1697675134 >> v2:172.21.15.49:3300/0 conn(0x7f97dc0cf4a0 0x7f97dc0726e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:12.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.383+0000 7f97e3f8c700 1 --2- 172.21.15.16:0/1697675134 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f97dc0cf880 0x7f97dc0b52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:12.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.383+0000 7f97e3f8c700 1 -- 172.21.15.16:0/1697675134 >> 172.21.15.16:0/1697675134 conn(0x7f97dc0b1fd0 msgr2=0x7f97dc0b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:12.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.383+0000 7f97e3f8c700 1 -- 172.21.15.16:0/1697675134 shutdown_connections 2023-10-16T12:11:12.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.383+0000 7f97e3f8c700 1 -- 172.21.15.16:0/1697675134 wait complete. 2023-10-16T12:11:12.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.384+0000 7f97e3f8c700 1 Processor -- start 2023-10-16T12:11:12.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.384+0000 7f97e3f8c700 1 -- start start 2023-10-16T12:11:12.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.385+0000 7f97e3f8c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f97dc0b5830 0x7f97dc0c6b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:12.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.385+0000 7f97e3f8c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f97dc0cf4a0 0x7f97dc158510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:12.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.385+0000 7f97e3f8c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97dc0cf880 0x7f97dc0b08c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:12.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.385+0000 7f97e3f8c700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97dc0afb30 con 0x7f97dc0cf880 2023-10-16T12:11:12.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.385+0000 7f97e3f8c700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97dc1f3c50 con 0x7f97dc0b5830 2023-10-16T12:11:12.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.385+0000 7f97e3f8c700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97dc0b0e00 con 0x7f97dc0cf4a0 2023-10-16T12:11:12.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.385+0000 7f97e2529700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97dc0cf880 0x7f97dc0b08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:12.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.385+0000 7f97e1d28700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f97dc0b5830 0x7f97dc0c6b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:12.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.385+0000 7f97e2529700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97dc0cf880 0x7f97dc0b08c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34006/0 (socket says 172.21.15.16:34006) 2023-10-16T12:11:12.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.385+0000 7f97e2529700 1 -- 172.21.15.16:0/112770780 learned_addr learned my addr 172.21.15.16:0/112770780 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:12.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.385+0000 7f97e1527700 1 --2- 172.21.15.16:0/112770780 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f97dc0cf4a0 0x7f97dc158510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:12.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.385+0000 7f97e2529700 1 -- 172.21.15.16:0/112770780 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f97dc0cf4a0 msgr2=0x7f97dc158510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:12.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.385+0000 7f97e2529700 1 --2- 172.21.15.16:0/112770780 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f97dc0cf4a0 0x7f97dc158510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:12.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.385+0000 7f97e2529700 1 -- 172.21.15.16:0/112770780 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f97dc0b5830 msgr2=0x7f97dc0c6b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:12.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.385+0000 7f97e2529700 1 --2- 172.21.15.16:0/112770780 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f97dc0b5830 0x7f97dc0c6b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:12.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.385+0000 7f97e2529700 1 -- 172.21.15.16:0/112770780 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97d4009b30 con 0x7f97dc0cf880 2023-10-16T12:11:12.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.385+0000 7f97e2529700 1 --2- 172.21.15.16:0/112770780 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97dc0cf880 0x7f97dc0b08c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f97dc0cdfc0 tx=0x7f97d4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:12.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.386+0000 7f97ce7fc700 1 -- 172.21.15.16:0/112770780 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d4004250 con 0x7f97dc0cf880 2023-10-16T12:11:12.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.386+0000 7f97ce7fc700 1 -- 172.21.15.16:0/112770780 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97d40043f0 con 0x7f97dc0cf880 2023-10-16T12:11:12.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.386+0000 7f97e3f8c700 1 -- 172.21.15.16:0/112770780 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97dc1f3ea0 con 0x7f97dc0cf880 2023-10-16T12:11:12.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.386+0000 7f97ce7fc700 1 -- 172.21.15.16:0/112770780 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d40047a0 con 0x7f97dc0cf880 2023-10-16T12:11:12.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.386+0000 7f97e3f8c700 1 -- 172.21.15.16:0/112770780 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97dc1f4460 con 0x7f97dc0cf880 2023-10-16T12:11:12.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.387+0000 7f97e3f8c700 1 -- 172.21.15.16:0/112770780 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97dc04ebb0 con 0x7f97dc0cf880 2023-10-16T12:11:12.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.387+0000 7f97ce7fc700 1 -- 172.21.15.16:0/112770780 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f97d401c440 con 0x7f97dc0cf880 2023-10-16T12:11:12.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.387+0000 7f97ce7fc700 1 --2- 172.21.15.16:0/112770780 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f97c006af00 0x7f97c006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:12.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.388+0000 7f97ce7fc700 1 -- 172.21.15.16:0/112770780 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f97d408ad60 con 0x7f97dc0cf880 2023-10-16T12:11:12.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.388+0000 7f97e1d28700 1 --2- 172.21.15.16:0/112770780 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f97c006af00 0x7f97c006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:12.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.388+0000 7f97e1d28700 1 --2- 172.21.15.16:0/112770780 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f97c006af00 0x7f97c006d3c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f97d0006fd0 tx=0x7f97d0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:12.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.392+0000 7f97ce7fc700 1 -- 172.21.15.16:0/112770780 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f97d4055c60 con 0x7f97dc0cf880 2023-10-16T12:11:12.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.601+0000 7f97e3f8c700 1 -- 172.21.15.16:0/112770780 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f97dc15d360 con 0x7f97dc0cf880 2023-10-16T12:11:12.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.602+0000 7f97ce7fc700 1 -- 172.21.15.16:0/112770780 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18238 (secure 0 0 0) 0x7f97d4020020 con 0x7f97dc0cf880 2023-10-16T12:11:12.604 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:11:12.605 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":25,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:10:44.080141+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:10:36.935773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T12:10:38.004870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-16T12:10:41.672025+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:11:12.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.604+0000 7f97e3f8c700 1 -- 172.21.15.16:0/112770780 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f97c006af00 msgr2=0x7f97c006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:12.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.604+0000 7f97e3f8c700 1 --2- 172.21.15.16:0/112770780 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f97c006af00 0x7f97c006d3c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f97d0006fd0 tx=0x7f97d0008040 comp rx=0 tx=0).stop 2023-10-16T12:11:12.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.604+0000 7f97e3f8c700 1 -- 172.21.15.16:0/112770780 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97dc0cf880 msgr2=0x7f97dc0b08c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:12.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.604+0000 7f97e3f8c700 1 --2- 172.21.15.16:0/112770780 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97dc0cf880 0x7f97dc0b08c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f97dc0cdfc0 tx=0x7f97d4004ad0 comp rx=0 tx=0).stop 2023-10-16T12:11:12.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.604+0000 7f97e3f8c700 1 -- 172.21.15.16:0/112770780 shutdown_connections 2023-10-16T12:11:12.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.604+0000 7f97e3f8c700 1 --2- 172.21.15.16:0/112770780 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f97c006af00 0x7f97c006d3c0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:12.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.604+0000 7f97e3f8c700 1 --2- 172.21.15.16:0/112770780 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97dc0cf880 0x7f97dc0b08c0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:12.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.604+0000 7f97e3f8c700 1 --2- 172.21.15.16:0/112770780 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f97dc0b5830 0x7f97dc0c6b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:12.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.604+0000 7f97e3f8c700 1 --2- 172.21.15.16:0/112770780 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f97dc0cf4a0 0x7f97dc158510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:12.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.604+0000 7f97e3f8c700 1 -- 172.21.15.16:0/112770780 >> 172.21.15.16:0/112770780 conn(0x7f97dc0b1fd0 msgr2=0x7f97dc1598f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:12.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.604+0000 7f97e3f8c700 1 -- 172.21.15.16:0/112770780 shutdown_connections 2023-10-16T12:11:12.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.604+0000 7f97e3f8c700 1 -- 172.21.15.16:0/112770780 wait complete. 2023-10-16T12:11:12.618 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T12:10:36.935773+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T12:10:38.004870+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-16T12:10:41.672025+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-16T12:11:12.619 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:11:12.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.744+0000 7f0bc0fa9700 1 Processor -- start 2023-10-16T12:11:12.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.745+0000 7f0bc0fa9700 1 -- start start 2023-10-16T12:11:12.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.745+0000 7f0bc0fa9700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0bbc15b070 0x7f0bbc15b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:12.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.746+0000 7f0bc0fa9700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bbc0cef70 con 0x7f0bbc0cb880 2023-10-16T12:11:12.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.746+0000 7f0bc0fa9700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bbc0ce360 con 0x7f0bbc06e170 2023-10-16T12:11:12.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.746+0000 7f0bc0fa9700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bbc0ce640 con 0x7f0bbc15b070 2023-10-16T12:11:12.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.746+0000 7f0bb9d9b700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0bbc0cb880 0x7f0bbc06da60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51516/0 (socket says 172.21.15.16:51516) 2023-10-16T12:11:12.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.746+0000 7f0bb9d9b700 1 -- 172.21.15.16:0/2143976184 learned_addr learned my addr 172.21.15.16:0/2143976184 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:12.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.746+0000 7f0bba59c700 1 --2- 172.21.15.16:0/2143976184 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0bbc15b070 0x7f0bbc15b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:12.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.746+0000 7f0bb959a700 1 -- 172.21.15.16:0/2143976184 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4004941035 0 0) 0x7f0bbc0cef70 con 0x7f0bbc0cb880 2023-10-16T12:11:12.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.746+0000 7f0bb959a700 1 -- 172.21.15.16:0/2143976184 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ba40035b0 con 0x7f0bbc0cb880 2023-10-16T12:11:12.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.746+0000 7f0bb959a700 1 -- 172.21.15.16:0/2143976184 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 113844426 0 0) 0x7f0bbc0ce360 con 0x7f0bbc06e170 2023-10-16T12:11:12.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.746+0000 7f0bba59c700 1 -- 172.21.15.16:0/2143976184 >> v1:172.21.15.49:6789/0 conn(0x7f0bbc06e170 legacy=0x7f0bbc0cf400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:12.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.746+0000 7f0bba59c700 1 -- 172.21.15.16:0/2143976184 >> v1:172.21.15.16:6789/0 conn(0x7f0bbc0cb880 legacy=0x7f0bbc06da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:12.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.746+0000 7f0bba59c700 1 -- 172.21.15.16:0/2143976184 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bbc0ce920 con 0x7f0bbc15b070 2023-10-16T12:11:12.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.746+0000 7f0bba59c700 1 --2- 172.21.15.16:0/2143976184 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0bbc15b070 0x7f0bbc15b450 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0bb4009dc0 tx=0x7f0bb400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b0484353ef605be server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:12.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.747+0000 7f0bb959a700 1 -- 172.21.15.16:0/2143976184 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb400e070 con 0x7f0bbc15b070 2023-10-16T12:11:12.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.747+0000 7f0bb959a700 1 -- 172.21.15.16:0/2143976184 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bb400bb20 con 0x7f0bbc15b070 2023-10-16T12:11:12.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.747+0000 7f0bb959a700 1 -- 172.21.15.16:0/2143976184 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb400bed0 con 0x7f0bbc15b070 2023-10-16T12:11:12.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.747+0000 7f0bc0fa9700 1 -- 172.21.15.16:0/2143976184 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0bbc15b070 msgr2=0x7f0bbc15b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:12.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.747+0000 7f0bc0fa9700 1 --2- 172.21.15.16:0/2143976184 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0bbc15b070 0x7f0bbc15b450 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0bb4009dc0 tx=0x7f0bb400b330 comp rx=0 tx=0).stop 2023-10-16T12:11:12.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.747+0000 7f0bc0fa9700 1 -- 172.21.15.16:0/2143976184 shutdown_connections 2023-10-16T12:11:12.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.747+0000 7f0bc0fa9700 1 --2- 172.21.15.16:0/2143976184 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0bbc15b070 0x7f0bbc15b450 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:12.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.747+0000 7f0bc0fa9700 1 -- 172.21.15.16:0/2143976184 >> 172.21.15.16:0/2143976184 conn(0x7f0bbc0b1f40 msgr2=0x7f0bbc0b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:12.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.747+0000 7f0bc0fa9700 1 -- 172.21.15.16:0/2143976184 shutdown_connections 2023-10-16T12:11:12.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.747+0000 7f0bc0fa9700 1 -- 172.21.15.16:0/2143976184 wait complete. 2023-10-16T12:11:12.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.749+0000 7f0bc0fa9700 1 Processor -- start 2023-10-16T12:11:12.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.749+0000 7f0bc0fa9700 1 -- start start 2023-10-16T12:11:12.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.749+0000 7f0bc0fa9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bbc06e170 0x7f0bbc15d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:12.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.749+0000 7f0bc0fa9700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0bbc0cb880 0x7f0bbc148fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:12.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.749+0000 7f0bc0fa9700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0bbc15b070 0x7f0bbc149510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:12.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.749+0000 7f0bc0fa9700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bbc149a50 con 0x7f0bbc06e170 2023-10-16T12:11:12.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.749+0000 7f0bc0fa9700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bbc14a060 con 0x7f0bbc15b070 2023-10-16T12:11:12.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.749+0000 7f0bc0fa9700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bbc15db20 con 0x7f0bbc0cb880 2023-10-16T12:11:12.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.749+0000 7f0bba59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bbc06e170 0x7f0bbc15d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:12.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.750+0000 7f0bba59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bbc06e170 0x7f0bbc15d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34016/0 (socket says 172.21.15.16:34016) 2023-10-16T12:11:12.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.750+0000 7f0bba59c700 1 -- 172.21.15.16:0/3819486925 learned_addr learned my addr 172.21.15.16:0/3819486925 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:12.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.750+0000 7f0bb9d9b700 1 --2- 172.21.15.16:0/3819486925 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0bbc0cb880 0x7f0bbc148fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:12.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.750+0000 7f0bbad9d700 1 --2- 172.21.15.16:0/3819486925 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0bbc15b070 0x7f0bbc149510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:12.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.750+0000 7f0bba59c700 1 -- 172.21.15.16:0/3819486925 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0bbc0cb880 msgr2=0x7f0bbc148fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:12.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.750+0000 7f0bba59c700 1 --2- 172.21.15.16:0/3819486925 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0bbc0cb880 0x7f0bbc148fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:12.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.750+0000 7f0bba59c700 1 -- 172.21.15.16:0/3819486925 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0bbc15b070 msgr2=0x7f0bbc149510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:12.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.750+0000 7f0bba59c700 1 --2- 172.21.15.16:0/3819486925 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0bbc15b070 0x7f0bbc149510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:12.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.750+0000 7f0bba59c700 1 -- 172.21.15.16:0/3819486925 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bb4009a70 con 0x7f0bbc06e170 2023-10-16T12:11:12.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.750+0000 7f0bba59c700 1 --2- 172.21.15.16:0/3819486925 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bbc06e170 0x7f0bbc15d2d0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f0bb4000f80 tx=0x7f0bb4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:12.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.750+0000 7f0baaffd700 1 -- 172.21.15.16:0/3819486925 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb401b810 con 0x7f0bbc06e170 2023-10-16T12:11:12.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.750+0000 7f0baaffd700 1 -- 172.21.15.16:0/3819486925 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bb4020070 con 0x7f0bbc06e170 2023-10-16T12:11:12.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.750+0000 7f0bc0fa9700 1 -- 172.21.15.16:0/3819486925 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bbc158790 con 0x7f0bbc06e170 2023-10-16T12:11:12.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.750+0000 7f0baaffd700 1 -- 172.21.15.16:0/3819486925 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb401bb90 con 0x7f0bbc06e170 2023-10-16T12:11:12.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.750+0000 7f0bc0fa9700 1 -- 172.21.15.16:0/3819486925 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bbc158d50 con 0x7f0bbc06e170 2023-10-16T12:11:12.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.751+0000 7f0bc0fa9700 1 -- 172.21.15.16:0/3819486925 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bbc15b730 con 0x7f0bbc06e170 2023-10-16T12:11:12.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.752+0000 7f0baaffd700 1 -- 172.21.15.16:0/3819486925 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f0bb401bcf0 con 0x7f0bbc06e170 2023-10-16T12:11:12.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.752+0000 7f0baaffd700 1 --2- 172.21.15.16:0/3819486925 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f0b9c06af00 0x7f0b9c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:12.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.752+0000 7f0baaffd700 1 -- 172.21.15.16:0/3819486925 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f0bb4022070 con 0x7f0bbc06e170 2023-10-16T12:11:12.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.752+0000 7f0bb9d9b700 1 --2- 172.21.15.16:0/3819486925 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f0b9c06af00 0x7f0b9c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:12.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.753+0000 7f0bb9d9b700 1 --2- 172.21.15.16:0/3819486925 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f0b9c06af00 0x7f0b9c06d3c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f0bbc15d160 tx=0x7f0bb0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:12.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.757+0000 7f0baaffd700 1 -- 172.21.15.16:0/3819486925 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0bb4056b10 con 0x7f0bbc06e170 2023-10-16T12:11:12.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.957+0000 7f0bc0fa9700 1 -- 172.21.15.16:0/3819486925 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0bbc04ebb0 con 0x7f0bbc06e170 2023-10-16T12:11:12.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.958+0000 7f0baaffd700 1 -- 172.21.15.16:0/3819486925 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0bb405a130 con 0x7f0bbc06e170 2023-10-16T12:11:12.960 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T12:11:12.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.960+0000 7f0bc0fa9700 1 -- 172.21.15.16:0/3819486925 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f0b9c06af00 msgr2=0x7f0b9c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:12.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.960+0000 7f0bc0fa9700 1 --2- 172.21.15.16:0/3819486925 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f0b9c06af00 0x7f0b9c06d3c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f0bbc15d160 tx=0x7f0bb0009040 comp rx=0 tx=0).stop 2023-10-16T12:11:12.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.960+0000 7f0bc0fa9700 1 -- 172.21.15.16:0/3819486925 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bbc06e170 msgr2=0x7f0bbc15d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:12.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.960+0000 7f0bc0fa9700 1 --2- 172.21.15.16:0/3819486925 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bbc06e170 0x7f0bbc15d2d0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f0bb4000f80 tx=0x7f0bb4004ad0 comp rx=0 tx=0).stop 2023-10-16T12:11:12.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.960+0000 7f0bc0fa9700 1 -- 172.21.15.16:0/3819486925 shutdown_connections 2023-10-16T12:11:12.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.960+0000 7f0bc0fa9700 1 --2- 172.21.15.16:0/3819486925 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f0b9c06af00 0x7f0b9c06d3c0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:12.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.960+0000 7f0bc0fa9700 1 --2- 172.21.15.16:0/3819486925 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0bbc15b070 0x7f0bbc149510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:12.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.960+0000 7f0bc0fa9700 1 --2- 172.21.15.16:0/3819486925 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bbc06e170 0x7f0bbc15d2d0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:12.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.960+0000 7f0bc0fa9700 1 --2- 172.21.15.16:0/3819486925 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0bbc0cb880 0x7f0bbc148fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:12.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.960+0000 7f0bc0fa9700 1 -- 172.21.15.16:0/3819486925 >> 172.21.15.16:0/3819486925 conn(0x7f0bbc0b1f40 msgr2=0x7f0bbc15ea60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:12.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.960+0000 7f0bc0fa9700 1 -- 172.21.15.16:0/3819486925 shutdown_connections 2023-10-16T12:11:12.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:12.960+0000 7f0bc0fa9700 1 -- 172.21.15.16:0/3819486925 wait complete. 2023-10-16T12:11:12.974 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T12:11:13.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.099+0000 7f4bd35f5700 1 Processor -- start 2023-10-16T12:11:13.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.100+0000 7f4bd35f5700 1 -- start start 2023-10-16T12:11:13.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.100+0000 7f4bd35f5700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f4bcc06e6b0 0x7f4bcc0cad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:13.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.100+0000 7f4bd35f5700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4bcc15cd10 0x7f4bcc0cd0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:13.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.100+0000 7f4bd35f5700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bcc0cc920 con 0x7f4bcc0ca9b0 2023-10-16T12:11:13.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.100+0000 7f4bd35f5700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bcc0cb2d0 con 0x7f4bcc06e6b0 2023-10-16T12:11:13.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.100+0000 7f4bd35f5700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bcc15c330 con 0x7f4bcc15cd10 2023-10-16T12:11:13.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.101+0000 7f4bd1391700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f4bcc0ca9b0 0x7f4bcc15bb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51530/0 (socket says 172.21.15.16:51530) 2023-10-16T12:11:13.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.101+0000 7f4bd1391700 1 -- 172.21.15.16:0/1625306614 learned_addr learned my addr 172.21.15.16:0/1625306614 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:13.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.101+0000 7f4bd0b90700 1 --2- 172.21.15.16:0/1625306614 >> v2:172.21.15.49:3300/0 conn(0x7f4bcc06e6b0 0x7f4bcc0cad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:13.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.101+0000 7f4bd1b92700 1 --2- 172.21.15.16:0/1625306614 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4bcc15cd10 0x7f4bcc0cd0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:13.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.101+0000 7f4bbffff700 1 -- 172.21.15.16:0/1625306614 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3172208773 0 0) 0x7f4bcc0cc920 con 0x7f4bcc0ca9b0 2023-10-16T12:11:13.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.101+0000 7f4bbffff700 1 -- 172.21.15.16:0/1625306614 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bb80035b0 con 0x7f4bcc0ca9b0 2023-10-16T12:11:13.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.101+0000 7f4bd0b90700 1 -- 172.21.15.16:0/1625306614 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4bcc15cd10 msgr2=0x7f4bcc0cd0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:13.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.101+0000 7f4bd0b90700 1 --2- 172.21.15.16:0/1625306614 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4bcc15cd10 0x7f4bcc0cd0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:13.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.101+0000 7f4bd0b90700 1 -- 172.21.15.16:0/1625306614 >> v1:172.21.15.16:6789/0 conn(0x7f4bcc0ca9b0 legacy=0x7f4bcc15bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:13.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.101+0000 7f4bd0b90700 1 -- 172.21.15.16:0/1625306614 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bcc15c610 con 0x7f4bcc06e6b0 2023-10-16T12:11:13.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.101+0000 7f4bd1b92700 1 --2- 172.21.15.16:0/1625306614 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4bcc15cd10 0x7f4bcc0cd0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:11:13.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.101+0000 7f4bd0b90700 1 --2- 172.21.15.16:0/1625306614 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4bcc06e6b0 0x7f4bcc0cad90 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f4bc0009840 tx=0x7f4bc0009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=49ab79060a805fe6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:13.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.101+0000 7f4bbffff700 1 -- 172.21.15.16:0/1625306614 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bc0011830 con 0x7f4bcc06e6b0 2023-10-16T12:11:13.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.101+0000 7f4bbffff700 1 -- 172.21.15.16:0/1625306614 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bc0011990 con 0x7f4bcc06e6b0 2023-10-16T12:11:13.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.101+0000 7f4bbffff700 1 -- 172.21.15.16:0/1625306614 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bc0011d20 con 0x7f4bcc06e6b0 2023-10-16T12:11:13.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.102+0000 7f4bd35f5700 1 -- 172.21.15.16:0/1625306614 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4bcc06e6b0 msgr2=0x7f4bcc0cad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:13.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.102+0000 7f4bd35f5700 1 --2- 172.21.15.16:0/1625306614 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4bcc06e6b0 0x7f4bcc0cad90 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f4bc0009840 tx=0x7f4bc0009b50 comp rx=0 tx=0).stop 2023-10-16T12:11:13.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.102+0000 7f4bd35f5700 1 -- 172.21.15.16:0/1625306614 shutdown_connections 2023-10-16T12:11:13.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.102+0000 7f4bd35f5700 1 --2- 172.21.15.16:0/1625306614 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4bcc15cd10 0x7f4bcc0cd0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:13.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.102+0000 7f4bd35f5700 1 --2- 172.21.15.16:0/1625306614 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4bcc06e6b0 0x7f4bcc0cad90 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:13.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.102+0000 7f4bd35f5700 1 -- 172.21.15.16:0/1625306614 >> 172.21.15.16:0/1625306614 conn(0x7f4bcc0b1ed0 msgr2=0x7f4bcc0b22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:13.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.102+0000 7f4bd35f5700 1 -- 172.21.15.16:0/1625306614 shutdown_connections 2023-10-16T12:11:13.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.102+0000 7f4bd35f5700 1 -- 172.21.15.16:0/1625306614 wait complete. 2023-10-16T12:11:13.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.103+0000 7f4bd35f5700 1 Processor -- start 2023-10-16T12:11:13.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.103+0000 7f4bd35f5700 1 -- start start 2023-10-16T12:11:13.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.104+0000 7f4bd35f5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bcc06e6b0 0x7f4bcc153d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:13.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.104+0000 7f4bd35f5700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4bcc0ca9b0 0x7f4bcc1f3a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:13.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.104+0000 7f4bd35f5700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4bcc15cd10 0x7f4bcc158360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:13.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.104+0000 7f4bd35f5700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bcc0ad360 con 0x7f4bcc06e6b0 2023-10-16T12:11:13.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.104+0000 7f4bd35f5700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bcc15d940 con 0x7f4bcc0ca9b0 2023-10-16T12:11:13.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.104+0000 7f4bd35f5700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bcc1f3f40 con 0x7f4bcc15cd10 2023-10-16T12:11:13.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.104+0000 7f4bd1391700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bcc06e6b0 0x7f4bcc153d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:13.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.104+0000 7f4bd1391700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bcc06e6b0 0x7f4bcc153d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34030/0 (socket says 172.21.15.16:34030) 2023-10-16T12:11:13.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.104+0000 7f4bd1391700 1 -- 172.21.15.16:0/193714133 learned_addr learned my addr 172.21.15.16:0/193714133 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:13.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.104+0000 7f4bd0b90700 1 --2- 172.21.15.16:0/193714133 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4bcc0ca9b0 0x7f4bcc1f3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:13.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.104+0000 7f4bd1b92700 1 --2- 172.21.15.16:0/193714133 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4bcc15cd10 0x7f4bcc158360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:13.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.104+0000 7f4bd1391700 1 -- 172.21.15.16:0/193714133 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4bcc15cd10 msgr2=0x7f4bcc158360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:13.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.104+0000 7f4bd1391700 1 --2- 172.21.15.16:0/193714133 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4bcc15cd10 0x7f4bcc158360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:13.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.104+0000 7f4bd1391700 1 -- 172.21.15.16:0/193714133 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4bcc0ca9b0 msgr2=0x7f4bcc1f3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:13.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.104+0000 7f4bd1391700 1 --2- 172.21.15.16:0/193714133 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4bcc0ca9b0 0x7f4bcc1f3a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:13.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.104+0000 7f4bd1391700 1 -- 172.21.15.16:0/193714133 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bc4009a70 con 0x7f4bcc06e6b0 2023-10-16T12:11:13.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.105+0000 7f4bd1391700 1 --2- 172.21.15.16:0/193714133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bcc06e6b0 0x7f4bcc153d60 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f4bc8001260 tx=0x7f4bc800c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:13.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.105+0000 7f4bbdffb700 1 -- 172.21.15.16:0/193714133 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bc80074e0 con 0x7f4bcc06e6b0 2023-10-16T12:11:13.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.105+0000 7f4bbdffb700 1 -- 172.21.15.16:0/193714133 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bc8007640 con 0x7f4bcc06e6b0 2023-10-16T12:11:13.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.105+0000 7f4bd35f5700 1 -- 172.21.15.16:0/193714133 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bc00094a0 con 0x7f4bcc06e6b0 2023-10-16T12:11:13.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.105+0000 7f4bbdffb700 1 -- 172.21.15.16:0/193714133 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bc8005a50 con 0x7f4bcc06e6b0 2023-10-16T12:11:13.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.105+0000 7f4bd35f5700 1 -- 172.21.15.16:0/193714133 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bcc0ad8b0 con 0x7f4bcc06e6b0 2023-10-16T12:11:13.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.106+0000 7f4bd35f5700 1 -- 172.21.15.16:0/193714133 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bcc04ebb0 con 0x7f4bcc06e6b0 2023-10-16T12:11:13.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.106+0000 7f4bbdffb700 1 -- 172.21.15.16:0/193714133 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f4bc8005bb0 con 0x7f4bcc06e6b0 2023-10-16T12:11:13.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.107+0000 7f4bbdffb700 1 --2- 172.21.15.16:0/193714133 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f4bb006af00 0x7f4bb006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:13.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.107+0000 7f4bbdffb700 1 -- 172.21.15.16:0/193714133 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f4bc8088cb0 con 0x7f4bcc06e6b0 2023-10-16T12:11:13.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.107+0000 7f4bd0b90700 1 --2- 172.21.15.16:0/193714133 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f4bb006af00 0x7f4bb006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:13.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.107+0000 7f4bd0b90700 1 --2- 172.21.15.16:0/193714133 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f4bb006af00 0x7f4bb006d3c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f4bc0000c00 tx=0x7f4bc001c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:13.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.111+0000 7f4bbdffb700 1 -- 172.21.15.16:0/193714133 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4bc8057590 con 0x7f4bcc06e6b0 2023-10-16T12:11:13.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.319+0000 7f4bd35f5700 1 -- 172.21.15.16:0/193714133 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f4bcc06b240 con 0x7f4bcc06e6b0 2023-10-16T12:11:13.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.319+0000 7f4bbdffb700 1 -- 172.21.15.16:0/193714133 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4bc80170a0 con 0x7f4bcc06e6b0 2023-10-16T12:11:13.322 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-10-16T12:11:13.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.322+0000 7f4bd35f5700 1 -- 172.21.15.16:0/193714133 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f4bb006af00 msgr2=0x7f4bb006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:13.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.322+0000 7f4bd35f5700 1 --2- 172.21.15.16:0/193714133 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f4bb006af00 0x7f4bb006d3c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f4bc0000c00 tx=0x7f4bc001c040 comp rx=0 tx=0).stop 2023-10-16T12:11:13.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.322+0000 7f4bd35f5700 1 -- 172.21.15.16:0/193714133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bcc06e6b0 msgr2=0x7f4bcc153d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:13.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.322+0000 7f4bd35f5700 1 --2- 172.21.15.16:0/193714133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bcc06e6b0 0x7f4bcc153d60 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f4bc8001260 tx=0x7f4bc800c9e0 comp rx=0 tx=0).stop 2023-10-16T12:11:13.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.322+0000 7f4bd35f5700 1 -- 172.21.15.16:0/193714133 shutdown_connections 2023-10-16T12:11:13.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.322+0000 7f4bd35f5700 1 --2- 172.21.15.16:0/193714133 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f4bb006af00 0x7f4bb006d3c0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:13.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.322+0000 7f4bd35f5700 1 --2- 172.21.15.16:0/193714133 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4bcc15cd10 0x7f4bcc158360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:13.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.322+0000 7f4bd35f5700 1 --2- 172.21.15.16:0/193714133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bcc06e6b0 0x7f4bcc153d60 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:13.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.322+0000 7f4bd35f5700 1 --2- 172.21.15.16:0/193714133 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4bcc0ca9b0 0x7f4bcc1f3a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:13.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.322+0000 7f4bd35f5700 1 -- 172.21.15.16:0/193714133 >> 172.21.15.16:0/193714133 conn(0x7f4bcc0b1ed0 msgr2=0x7f4bcc0ccfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:13.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.322+0000 7f4bd35f5700 1 -- 172.21.15.16:0/193714133 shutdown_connections 2023-10-16T12:11:13.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.322+0000 7f4bd35f5700 1 -- 172.21.15.16:0/193714133 wait complete. 2023-10-16T12:11:13.336 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T12:11:13.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.462+0000 7fd74365d700 1 Processor -- start 2023-10-16T12:11:13.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.462+0000 7fd74365d700 1 -- start start 2023-10-16T12:11:13.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.463+0000 7fd74365d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd73c15da10 0x7fd73c0d1520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:13.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.463+0000 7fd74365d700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fd73c0d4060 0x7fd73c0cf2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:13.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.463+0000 7fd74365d700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd73c0d4540 con 0x7fd73c0d1d20 2023-10-16T12:11:13.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.463+0000 7fd74365d700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd73c0ce7e0 con 0x7fd73c0d4060 2023-10-16T12:11:13.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.463+0000 7fd74365d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd73c0ceac0 con 0x7fd73c15da10 2023-10-16T12:11:13.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.463+0000 7fd7413f9700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd73c15da10 0x7fd73c0d1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:13.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.463+0000 7fd740bf8700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fd73c0d4060 0x7fd73c0cf2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:13.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.463+0000 7fd741bfa700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7fd73c0d1d20 0x7fd73c0d2130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:58730/0 (socket says 172.21.15.16:58730) 2023-10-16T12:11:13.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.463+0000 7fd7413f9700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd73c15da10 0x7fd73c0d1520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:50528/0 (socket says 172.21.15.16:50528) 2023-10-16T12:11:13.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.463+0000 7fd741bfa700 1 -- 172.21.15.16:0/1614627034 learned_addr learned my addr 172.21.15.16:0/1614627034 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:13.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.463+0000 7fd740bf8700 1 -- 172.21.15.16:0/1614627034 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd73c15da10 msgr2=0x7fd73c0d1520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:13.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.463+0000 7fd740bf8700 1 --2- 172.21.15.16:0/1614627034 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd73c15da10 0x7fd73c0d1520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:13.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.463+0000 7fd740bf8700 1 -- 172.21.15.16:0/1614627034 >> v1:172.21.15.49:6789/0 conn(0x7fd73c0d1d20 legacy=0x7fd73c0d2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:13.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.464+0000 7fd740bf8700 1 -- 172.21.15.16:0/1614627034 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd73c0ceda0 con 0x7fd73c0d4060 2023-10-16T12:11:13.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.464+0000 7fd7413f9700 1 --2- 172.21.15.16:0/1614627034 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd73c15da10 0x7fd73c0d1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:11:13.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.464+0000 7fd740bf8700 1 --2- 172.21.15.16:0/1614627034 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd73c0d4060 0x7fd73c0cf2a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fd72c009840 tx=0x7fd72c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c3986c778995a5c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:13.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.464+0000 7fd733fff700 1 -- 172.21.15.16:0/1614627034 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd72c011830 con 0x7fd73c0d4060 2023-10-16T12:11:13.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.464+0000 7fd733fff700 1 -- 172.21.15.16:0/1614627034 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd72c011990 con 0x7fd73c0d4060 2023-10-16T12:11:13.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.464+0000 7fd733fff700 1 -- 172.21.15.16:0/1614627034 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd72c011d20 con 0x7fd73c0d4060 2023-10-16T12:11:13.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.464+0000 7fd74365d700 1 -- 172.21.15.16:0/1614627034 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd73c0d4060 msgr2=0x7fd73c0cf2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:13.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.464+0000 7fd74365d700 1 --2- 172.21.15.16:0/1614627034 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd73c0d4060 0x7fd73c0cf2a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fd72c009840 tx=0x7fd72c009b50 comp rx=0 tx=0).stop 2023-10-16T12:11:13.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.465+0000 7fd74365d700 1 -- 172.21.15.16:0/1614627034 shutdown_connections 2023-10-16T12:11:13.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.465+0000 7fd74365d700 1 --2- 172.21.15.16:0/1614627034 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd73c15da10 0x7fd73c0d1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:13.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.465+0000 7fd74365d700 1 --2- 172.21.15.16:0/1614627034 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd73c0d4060 0x7fd73c0cf2a0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:13.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.465+0000 7fd74365d700 1 -- 172.21.15.16:0/1614627034 >> 172.21.15.16:0/1614627034 conn(0x7fd73c09c9f0 msgr2=0x7fd73c0d3740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:13.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.465+0000 7fd74365d700 1 -- 172.21.15.16:0/1614627034 shutdown_connections 2023-10-16T12:11:13.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.465+0000 7fd74365d700 1 -- 172.21.15.16:0/1614627034 wait complete. 2023-10-16T12:11:13.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.466+0000 7fd74365d700 1 Processor -- start 2023-10-16T12:11:13.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.466+0000 7fd74365d700 1 -- start start 2023-10-16T12:11:13.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.467+0000 7fd74365d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd73c0d1d20 0x7fd73c0b8c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:13.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.467+0000 7fd74365d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd73c0d4060 0x7fd73c0b52b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:13.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.467+0000 7fd74365d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd73c15da10 0x7fd73c0b3460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:13.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.467+0000 7fd74365d700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd73c0b72e0 con 0x7fd73c15da10 2023-10-16T12:11:13.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.467+0000 7fd74365d700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd73c0b75c0 con 0x7fd73c0d4060 2023-10-16T12:11:13.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.467+0000 7fd74365d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd73c0b78a0 con 0x7fd73c0d1d20 2023-10-16T12:11:13.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.467+0000 7fd741bfa700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd73c15da10 0x7fd73c0b3460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:13.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.467+0000 7fd741bfa700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd73c15da10 0x7fd73c0b3460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34048/0 (socket says 172.21.15.16:34048) 2023-10-16T12:11:13.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.467+0000 7fd741bfa700 1 -- 172.21.15.16:0/2064040207 learned_addr learned my addr 172.21.15.16:0/2064040207 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:13.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.467+0000 7fd7413f9700 1 --2- 172.21.15.16:0/2064040207 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd73c0d1d20 0x7fd73c0b8c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:13.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.467+0000 7fd740bf8700 1 --2- 172.21.15.16:0/2064040207 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd73c0d4060 0x7fd73c0b52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:13.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.467+0000 7fd741bfa700 1 -- 172.21.15.16:0/2064040207 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd73c0d1d20 msgr2=0x7fd73c0b8c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:13.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.467+0000 7fd741bfa700 1 --2- 172.21.15.16:0/2064040207 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd73c0d1d20 0x7fd73c0b8c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:13.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.467+0000 7fd741bfa700 1 -- 172.21.15.16:0/2064040207 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd73c0d4060 msgr2=0x7fd73c0b52b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:13.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.467+0000 7fd741bfa700 1 --2- 172.21.15.16:0/2064040207 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd73c0d4060 0x7fd73c0b52b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:13.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.467+0000 7fd741bfa700 1 -- 172.21.15.16:0/2064040207 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd738009430 con 0x7fd73c15da10 2023-10-16T12:11:13.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.467+0000 7fd740bf8700 1 --2- 172.21.15.16:0/2064040207 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd73c0d4060 0x7fd73c0b52b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:11:13.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.467+0000 7fd741bfa700 1 --2- 172.21.15.16:0/2064040207 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd73c15da10 0x7fd73c0b3460 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd73400cbc0 tx=0x7fd73400ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:13.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.467+0000 7fd731ffb700 1 -- 172.21.15.16:0/2064040207 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd73400aaa0 con 0x7fd73c15da10 2023-10-16T12:11:13.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.467+0000 7fd731ffb700 1 -- 172.21.15.16:0/2064040207 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd73400ac00 con 0x7fd73c15da10 2023-10-16T12:11:13.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.467+0000 7fd74365d700 1 -- 172.21.15.16:0/2064040207 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd72c0094a0 con 0x7fd73c15da10 2023-10-16T12:11:13.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.467+0000 7fd731ffb700 1 -- 172.21.15.16:0/2064040207 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd734012400 con 0x7fd73c15da10 2023-10-16T12:11:13.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.468+0000 7fd74365d700 1 -- 172.21.15.16:0/2064040207 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd73c0b2f40 con 0x7fd73c15da10 2023-10-16T12:11:13.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.468+0000 7fd74365d700 1 -- 172.21.15.16:0/2064040207 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd73c0b3130 con 0x7fd73c15da10 2023-10-16T12:11:13.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.469+0000 7fd731ffb700 1 -- 172.21.15.16:0/2064040207 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fd73401b3b0 con 0x7fd73c15da10 2023-10-16T12:11:13.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.469+0000 7fd731ffb700 1 --2- 172.21.15.16:0/2064040207 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fd72006af00 0x7fd72006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:13.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.469+0000 7fd731ffb700 1 -- 172.21.15.16:0/2064040207 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fd7340893f0 con 0x7fd73c15da10 2023-10-16T12:11:13.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.469+0000 7fd7413f9700 1 --2- 172.21.15.16:0/2064040207 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fd72006af00 0x7fd72006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:13.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.470+0000 7fd7413f9700 1 --2- 172.21.15.16:0/2064040207 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fd72006af00 0x7fd72006d3c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd738009cb0 tx=0x7fd7380091c0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:13.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.474+0000 7fd731ffb700 1 -- 172.21.15.16:0/2064040207 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd7340542f0 con 0x7fd73c15da10 2023-10-16T12:11:13.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.680+0000 7fd74365d700 1 -- 172.21.15.16:0/2064040207 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd73c0b7420 con 0x7fd73c15da10 2023-10-16T12:11:13.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.680+0000 7fd731ffb700 1 -- 172.21.15.16:0/2064040207 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd73c0b7420 con 0x7fd73c15da10 2023-10-16T12:11:13.682 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-10-16T12:11:13.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.682+0000 7fd74365d700 1 -- 172.21.15.16:0/2064040207 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fd72006af00 msgr2=0x7fd72006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:13.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.682+0000 7fd74365d700 1 --2- 172.21.15.16:0/2064040207 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fd72006af00 0x7fd72006d3c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd738009cb0 tx=0x7fd7380091c0 comp rx=0 tx=0).stop 2023-10-16T12:11:13.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.682+0000 7fd74365d700 1 -- 172.21.15.16:0/2064040207 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd73c15da10 msgr2=0x7fd73c0b3460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:13.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.682+0000 7fd74365d700 1 --2- 172.21.15.16:0/2064040207 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd73c15da10 0x7fd73c0b3460 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd73400cbc0 tx=0x7fd73400ced0 comp rx=0 tx=0).stop 2023-10-16T12:11:13.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.682+0000 7fd74365d700 1 -- 172.21.15.16:0/2064040207 shutdown_connections 2023-10-16T12:11:13.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.682+0000 7fd74365d700 1 --2- 172.21.15.16:0/2064040207 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fd72006af00 0x7fd72006d3c0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:13.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.682+0000 7fd74365d700 1 --2- 172.21.15.16:0/2064040207 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd73c15da10 0x7fd73c0b3460 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:13.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.682+0000 7fd74365d700 1 --2- 172.21.15.16:0/2064040207 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd73c0d1d20 0x7fd73c0b8c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:13.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.682+0000 7fd74365d700 1 --2- 172.21.15.16:0/2064040207 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd73c0d4060 0x7fd73c0b52b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:13.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.682+0000 7fd74365d700 1 -- 172.21.15.16:0/2064040207 >> 172.21.15.16:0/2064040207 conn(0x7fd73c09c9f0 msgr2=0x7fd73c15df70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:13.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.683+0000 7fd74365d700 1 -- 172.21.15.16:0/2064040207 shutdown_connections 2023-10-16T12:11:13.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.683+0000 7fd74365d700 1 -- 172.21.15.16:0/2064040207 wait complete. 2023-10-16T12:11:13.697 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-16T12:11:13.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.823+0000 7f1dbf607700 1 Processor -- start 2023-10-16T12:11:13.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.823+0000 7f1dbf607700 1 -- start start 2023-10-16T12:11:13.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.824+0000 7f1dbf607700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1db806e570 0x7f1db815e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:13.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.824+0000 7f1dbf607700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1db806a7e0 0x7f1db806ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:13.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.824+0000 7f1dbf607700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1db815b6a0 con 0x7f1db815b2c0 2023-10-16T12:11:13.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.824+0000 7f1dbf607700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1db806b1b0 con 0x7f1db806a7e0 2023-10-16T12:11:13.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.824+0000 7f1dbf607700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1db8161b20 con 0x7f1db806e570 2023-10-16T12:11:13.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.824+0000 7f1dbdba4700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1db806a7e0 0x7f1db806ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:13.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.824+0000 7f1dbdba4700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1db806a7e0 0x7f1db806ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34056/0 (socket says 172.21.15.16:34056) 2023-10-16T12:11:13.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.824+0000 7f1dbdba4700 1 -- 172.21.15.16:0/1641408913 learned_addr learned my addr 172.21.15.16:0/1641408913 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:13.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.824+0000 7f1dbcba2700 1 --2- 172.21.15.16:0/1641408913 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1db806e570 0x7f1db815e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:13.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.824+0000 7f1dbdba4700 1 -- 172.21.15.16:0/1641408913 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1db806e570 msgr2=0x7f1db815e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:13.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.824+0000 7f1dbdba4700 1 --2- 172.21.15.16:0/1641408913 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1db806e570 0x7f1db815e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:13.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.824+0000 7f1dbdba4700 1 -- 172.21.15.16:0/1641408913 >> v1:172.21.15.49:6789/0 conn(0x7f1db815b2c0 legacy=0x7f1db8161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:13.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.824+0000 7f1dbdba4700 1 -- 172.21.15.16:0/1641408913 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1db8161e00 con 0x7f1db806a7e0 2023-10-16T12:11:13.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.824+0000 7f1dbdba4700 1 --2- 172.21.15.16:0/1641408913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1db806a7e0 0x7f1db806ac70 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f1db0009e50 tx=0x7f1db000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c672f83032423a7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:13.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.824+0000 7f1daffff700 1 -- 172.21.15.16:0/1641408913 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1db000e070 con 0x7f1db806a7e0 2023-10-16T12:11:13.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.825+0000 7f1daffff700 1 -- 172.21.15.16:0/1641408913 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1db000b910 con 0x7f1db806a7e0 2023-10-16T12:11:13.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.825+0000 7f1daffff700 1 -- 172.21.15.16:0/1641408913 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1db000bcc0 con 0x7f1db806a7e0 2023-10-16T12:11:13.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.825+0000 7f1dbf607700 1 -- 172.21.15.16:0/1641408913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1db806a7e0 msgr2=0x7f1db806ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:13.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.825+0000 7f1dbf607700 1 --2- 172.21.15.16:0/1641408913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1db806a7e0 0x7f1db806ac70 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f1db0009e50 tx=0x7f1db000b330 comp rx=0 tx=0).stop 2023-10-16T12:11:13.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.825+0000 7f1dbf607700 1 -- 172.21.15.16:0/1641408913 shutdown_connections 2023-10-16T12:11:13.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.825+0000 7f1dbf607700 1 --2- 172.21.15.16:0/1641408913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1db806a7e0 0x7f1db806ac70 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:13.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.825+0000 7f1dbf607700 1 --2- 172.21.15.16:0/1641408913 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1db806e570 0x7f1db815e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:13.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.825+0000 7f1dbf607700 1 -- 172.21.15.16:0/1641408913 >> 172.21.15.16:0/1641408913 conn(0x7f1db80b1f40 msgr2=0x7f1db80b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:13.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.825+0000 7f1dbf607700 1 -- 172.21.15.16:0/1641408913 shutdown_connections 2023-10-16T12:11:13.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.825+0000 7f1dbf607700 1 -- 172.21.15.16:0/1641408913 wait complete. 2023-10-16T12:11:13.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.826+0000 7f1dbf607700 1 Processor -- start 2023-10-16T12:11:13.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.826+0000 7f1dbf607700 1 -- start start 2023-10-16T12:11:13.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.827+0000 7f1dbf607700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1db806a7e0 0x7f1db80aeed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:13.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.827+0000 7f1dbf607700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1db806e570 0x7f1db8158a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:13.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.827+0000 7f1dbf607700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1db815b2c0 0x7f1db80ca170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:13.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.827+0000 7f1dbf607700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1db8158f50 con 0x7f1db806a7e0 2023-10-16T12:11:13.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.827+0000 7f1dbf607700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1db80c7de0 con 0x7f1db806e570 2023-10-16T12:11:13.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.827+0000 7f1dbf607700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1db80ca6b0 con 0x7f1db815b2c0 2023-10-16T12:11:13.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.827+0000 7f1dbd3a3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1db806a7e0 0x7f1db80aeed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:13.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.827+0000 7f1dbd3a3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1db806a7e0 0x7f1db80aeed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34058/0 (socket says 172.21.15.16:34058) 2023-10-16T12:11:13.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.827+0000 7f1dbd3a3700 1 -- 172.21.15.16:0/2779591804 learned_addr learned my addr 172.21.15.16:0/2779591804 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:13.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.827+0000 7f1dbdba4700 1 --2- 172.21.15.16:0/2779591804 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1db815b2c0 0x7f1db80ca170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:13.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.827+0000 7f1dbcba2700 1 --2- 172.21.15.16:0/2779591804 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1db806e570 0x7f1db8158a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:13.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.828+0000 7f1dbd3a3700 1 -- 172.21.15.16:0/2779591804 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1db815b2c0 msgr2=0x7f1db80ca170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:13.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.828+0000 7f1dbd3a3700 1 --2- 172.21.15.16:0/2779591804 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1db815b2c0 0x7f1db80ca170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:13.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.828+0000 7f1dbd3a3700 1 -- 172.21.15.16:0/2779591804 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1db806e570 msgr2=0x7f1db8158a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:13.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.828+0000 7f1dbd3a3700 1 --2- 172.21.15.16:0/2779591804 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1db806e570 0x7f1db8158a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:13.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.828+0000 7f1dbd3a3700 1 -- 172.21.15.16:0/2779591804 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1db0009b30 con 0x7f1db806a7e0 2023-10-16T12:11:13.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.828+0000 7f1dbd3a3700 1 --2- 172.21.15.16:0/2779591804 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1db806a7e0 0x7f1db80aeed0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f1da800c510 tx=0x7f1da800c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:13.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.828+0000 7f1dadffb700 1 -- 172.21.15.16:0/2779591804 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da80052d0 con 0x7f1db806a7e0 2023-10-16T12:11:13.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.828+0000 7f1dadffb700 1 -- 172.21.15.16:0/2779591804 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1da8005430 con 0x7f1db806a7e0 2023-10-16T12:11:13.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.828+0000 7f1dbf607700 1 -- 172.21.15.16:0/2779591804 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1db80af520 con 0x7f1db806a7e0 2023-10-16T12:11:13.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.828+0000 7f1dadffb700 1 -- 172.21.15.16:0/2779591804 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da80038c0 con 0x7f1db806a7e0 2023-10-16T12:11:13.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.828+0000 7f1dbf607700 1 -- 172.21.15.16:0/2779591804 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1db80c7a70 con 0x7f1db806a7e0 2023-10-16T12:11:13.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.829+0000 7f1dbf607700 1 -- 172.21.15.16:0/2779591804 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1db804ebb0 con 0x7f1db806a7e0 2023-10-16T12:11:13.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.830+0000 7f1dadffb700 1 -- 172.21.15.16:0/2779591804 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f1da8003a20 con 0x7f1db806a7e0 2023-10-16T12:11:13.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.830+0000 7f1dadffb700 1 --2- 172.21.15.16:0/2779591804 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f1d9c06af00 0x7f1d9c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:13.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.830+0000 7f1dbcba2700 1 --2- 172.21.15.16:0/2779591804 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f1d9c06af00 0x7f1d9c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:13.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.830+0000 7f1dadffb700 1 -- 172.21.15.16:0/2779591804 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f1da8088b30 con 0x7f1db806a7e0 2023-10-16T12:11:13.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.830+0000 7f1dbcba2700 1 --2- 172.21.15.16:0/2779591804 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f1d9c06af00 0x7f1d9c06d3c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1db80c9700 tx=0x7f1db4009210 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:13.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:13.835+0000 7f1dadffb700 1 -- 172.21.15.16:0/2779591804 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1da80538f0 con 0x7f1db806a7e0 2023-10-16T12:11:14.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.036+0000 7f1dbf607700 1 -- 172.21.15.16:0/2779591804 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f1db80b6370 con 0x7f1db806a7e0 2023-10-16T12:11:14.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.037+0000 7f1dadffb700 1 -- 172.21.15.16:0/2779591804 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) v1 ==== 103+0+11 (secure 0 0 0) 0x7f1da8056f10 con 0x7f1db806a7e0 2023-10-16T12:11:14.039 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-10-16T12:11:14.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.039+0000 7f1dbf607700 1 -- 172.21.15.16:0/2779591804 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f1d9c06af00 msgr2=0x7f1d9c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:14.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.039+0000 7f1dbf607700 1 --2- 172.21.15.16:0/2779591804 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f1d9c06af00 0x7f1d9c06d3c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1db80c9700 tx=0x7f1db4009210 comp rx=0 tx=0).stop 2023-10-16T12:11:14.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.039+0000 7f1dbf607700 1 -- 172.21.15.16:0/2779591804 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1db806a7e0 msgr2=0x7f1db80aeed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:14.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.039+0000 7f1dbf607700 1 --2- 172.21.15.16:0/2779591804 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1db806a7e0 0x7f1db80aeed0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f1da800c510 tx=0x7f1da800c820 comp rx=0 tx=0).stop 2023-10-16T12:11:14.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.039+0000 7f1dbf607700 1 -- 172.21.15.16:0/2779591804 shutdown_connections 2023-10-16T12:11:14.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.039+0000 7f1dbf607700 1 --2- 172.21.15.16:0/2779591804 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f1d9c06af00 0x7f1d9c06d3c0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:14.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.039+0000 7f1dbf607700 1 --2- 172.21.15.16:0/2779591804 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1db815b2c0 0x7f1db80ca170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:14.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.039+0000 7f1dbf607700 1 --2- 172.21.15.16:0/2779591804 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1db806a7e0 0x7f1db80aeed0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:14.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.039+0000 7f1dbf607700 1 --2- 172.21.15.16:0/2779591804 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1db806e570 0x7f1db8158a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:14.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.039+0000 7f1dbf607700 1 -- 172.21.15.16:0/2779591804 >> 172.21.15.16:0/2779591804 conn(0x7f1db80b1f40 msgr2=0x7f1db80b10f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:14.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.039+0000 7f1dbf607700 1 -- 172.21.15.16:0/2779591804 shutdown_connections 2023-10-16T12:11:14.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.039+0000 7f1dbf607700 1 -- 172.21.15.16:0/2779591804 wait complete. 2023-10-16T12:11:14.053 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:11:14.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.177+0000 7fd33de68700 1 Processor -- start 2023-10-16T12:11:14.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.177+0000 7fd33de68700 1 -- start start 2023-10-16T12:11:14.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.177+0000 7fd33de68700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd33815b280 0x7fd3381613e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:14.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.177+0000 7fd33de68700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd338161c30 con 0x7fd33806a7e0 2023-10-16T12:11:14.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.177+0000 7fd33de68700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd338161f10 con 0x7fd3380cc8a0 2023-10-16T12:11:14.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.177+0000 7fd33de68700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3381621f0 con 0x7fd33815b280 2023-10-16T12:11:14.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.178+0000 7fd337fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd33806a7e0 0x7fd3380d02b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51576/0 (socket says 172.21.15.16:51576) 2023-10-16T12:11:14.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.178+0000 7fd337fff700 1 -- 172.21.15.16:0/1508400545 learned_addr learned my addr 172.21.15.16:0/1508400545 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:14.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.178+0000 7fd3377fe700 1 --2- 172.21.15.16:0/1508400545 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd33815b280 0x7fd3381613e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:14.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.178+0000 7fd3367fc700 1 -- 172.21.15.16:0/1508400545 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2225318107 0 0) 0x7fd338161c30 con 0x7fd33806a7e0 2023-10-16T12:11:14.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.178+0000 7fd3367fc700 1 -- 172.21.15.16:0/1508400545 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3200035b0 con 0x7fd33806a7e0 2023-10-16T12:11:14.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.178+0000 7fd3367fc700 1 -- 172.21.15.16:0/1508400545 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 943848535 0 0) 0x7fd3200035b0 con 0x7fd33806a7e0 2023-10-16T12:11:14.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.178+0000 7fd3367fc700 1 -- 172.21.15.16:0/1508400545 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd33815b280 msgr2=0x7fd3381613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:14.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.178+0000 7fd3367fc700 1 --2- 172.21.15.16:0/1508400545 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd33815b280 0x7fd3381613e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:14.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.178+0000 7fd3367fc700 1 -- 172.21.15.16:0/1508400545 >> v1:172.21.15.49:6789/0 conn(0x7fd3380cc8a0 legacy=0x7fd33815e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:14.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.178+0000 7fd3367fc700 1 -- 172.21.15.16:0/1508400545 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd33815df00 con 0x7fd33806a7e0 2023-10-16T12:11:14.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.178+0000 7fd3367fc700 1 -- 172.21.15.16:0/1508400545 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7fd3300019f0 con 0x7fd33806a7e0 2023-10-16T12:11:14.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.178+0000 7fd3367fc700 1 -- 172.21.15.16:0/1508400545 >> v1:172.21.15.16:6789/0 conn(0x7fd33806a7e0 legacy=0x7fd3380d02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:14.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.178+0000 7fd3367fc700 1 --2- 172.21.15.16:0/1508400545 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd33815b280 0x7fd320003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:14.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.178+0000 7fd3367fc700 1 --2- 172.21.15.16:0/1508400545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd320003f70 0x7fd320004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:14.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.178+0000 7fd3367fc700 1 --2- 172.21.15.16:0/1508400545 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd320004d90 0x7fd320005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:14.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.178+0000 7fd3367fc700 1 -- 172.21.15.16:0/1508400545 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3381621f0 con 0x7fd320003f70 2023-10-16T12:11:14.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.178+0000 7fd3367fc700 1 -- 172.21.15.16:0/1508400545 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd32000ba70 con 0x7fd320004d90 2023-10-16T12:11:14.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.178+0000 7fd3367fc700 1 -- 172.21.15.16:0/1508400545 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd32000bc30 con 0x7fd33815b280 2023-10-16T12:11:14.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.178+0000 7fd336ffd700 1 --2- 172.21.15.16:0/1508400545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd320003f70 0x7fd320004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:14.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.178+0000 7fd3377fe700 1 --2- 172.21.15.16:0/1508400545 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd33815b280 0x7fd320003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:14.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.178+0000 7fd337fff700 1 --2- 172.21.15.16:0/1508400545 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd320004d90 0x7fd320005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:14.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.179+0000 7fd336ffd700 1 -- 172.21.15.16:0/1508400545 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd33815b280 msgr2=0x7fd320003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:14.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.179+0000 7fd336ffd700 1 --2- 172.21.15.16:0/1508400545 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd33815b280 0x7fd320003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:14.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.179+0000 7fd336ffd700 1 -- 172.21.15.16:0/1508400545 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd320004d90 msgr2=0x7fd320005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:14.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.179+0000 7fd336ffd700 1 --2- 172.21.15.16:0/1508400545 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd320004d90 0x7fd320005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:14.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.179+0000 7fd336ffd700 1 -- 172.21.15.16:0/1508400545 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3200035b0 con 0x7fd320003f70 2023-10-16T12:11:14.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.179+0000 7fd337fff700 1 --2- 172.21.15.16:0/1508400545 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd320004d90 0x7fd320005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:11:14.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.179+0000 7fd336ffd700 1 --2- 172.21.15.16:0/1508400545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd320003f70 0x7fd320004850 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fd32c0018d0 tx=0x7fd320005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5df15c55aa37566 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:14.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.179+0000 7fd3367fc700 1 -- 172.21.15.16:0/1508400545 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd32c012620 con 0x7fd320003f70 2023-10-16T12:11:14.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.179+0000 7fd3367fc700 1 -- 172.21.15.16:0/1508400545 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd32c012780 con 0x7fd320003f70 2023-10-16T12:11:14.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.179+0000 7fd3367fc700 1 -- 172.21.15.16:0/1508400545 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd32c012b30 con 0x7fd320003f70 2023-10-16T12:11:14.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.179+0000 7fd33de68700 1 -- 172.21.15.16:0/1508400545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd320003f70 msgr2=0x7fd320004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:14.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.179+0000 7fd33de68700 1 --2- 172.21.15.16:0/1508400545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd320003f70 0x7fd320004850 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fd32c0018d0 tx=0x7fd320005880 comp rx=0 tx=0).stop 2023-10-16T12:11:14.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.179+0000 7fd337fff700 1 -- 172.21.15.16:0/1508400545 reap_dead start 2023-10-16T12:11:14.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.179+0000 7fd33de68700 1 -- 172.21.15.16:0/1508400545 shutdown_connections 2023-10-16T12:11:14.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.179+0000 7fd33de68700 1 -- 172.21.15.16:0/1508400545 >> 172.21.15.16:0/1508400545 conn(0x7fd3380b1fd0 msgr2=0x7fd3380b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:14.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.179+0000 7fd33de68700 1 -- 172.21.15.16:0/1508400545 shutdown_connections 2023-10-16T12:11:14.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.180+0000 7fd33de68700 1 -- 172.21.15.16:0/1508400545 wait complete. 2023-10-16T12:11:14.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.181+0000 7fd33de68700 1 Processor -- start 2023-10-16T12:11:14.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.181+0000 7fd33de68700 1 -- start start 2023-10-16T12:11:14.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.182+0000 7fd33de68700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd320003f70 0x7fd3381605b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:14.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.182+0000 7fd33de68700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd338160af0 0x7fd3381f3f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:14.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.182+0000 7fd33de68700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd3381f44d0 0x7fd3380c6200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:14.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.182+0000 7fd33de68700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd33815f5a0 con 0x7fd338160af0 2023-10-16T12:11:14.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.182+0000 7fd33de68700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3381f4980 con 0x7fd3381f44d0 2023-10-16T12:11:14.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.182+0000 7fd33de68700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd33815f7f0 con 0x7fd320003f70 2023-10-16T12:11:14.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.182+0000 7fd336ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd338160af0 0x7fd3381f3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:14.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.182+0000 7fd336ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd338160af0 0x7fd3381f3f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34080/0 (socket says 172.21.15.16:34080) 2023-10-16T12:11:14.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.182+0000 7fd336ffd700 1 -- 172.21.15.16:0/628521379 learned_addr learned my addr 172.21.15.16:0/628521379 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:14.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.182+0000 7fd337fff700 1 --2- 172.21.15.16:0/628521379 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd3381f44d0 0x7fd3380c6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:14.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.182+0000 7fd3377fe700 1 --2- 172.21.15.16:0/628521379 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd320003f70 0x7fd3381605b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:14.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.182+0000 7fd336ffd700 1 -- 172.21.15.16:0/628521379 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd320003f70 msgr2=0x7fd3381605b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:14.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.182+0000 7fd336ffd700 1 --2- 172.21.15.16:0/628521379 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd320003f70 0x7fd3381605b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:14.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.182+0000 7fd336ffd700 1 -- 172.21.15.16:0/628521379 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd3381f44d0 msgr2=0x7fd3380c6200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:14.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.182+0000 7fd336ffd700 1 --2- 172.21.15.16:0/628521379 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd3381f44d0 0x7fd3380c6200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:14.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.182+0000 7fd336ffd700 1 -- 172.21.15.16:0/628521379 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd33815fad0 con 0x7fd338160af0 2023-10-16T12:11:14.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.182+0000 7fd337fff700 1 --2- 172.21.15.16:0/628521379 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd3381f44d0 0x7fd3380c6200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:11:14.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.182+0000 7fd3377fe700 1 --2- 172.21.15.16:0/628521379 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd320003f70 0x7fd3381605b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:11:14.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.182+0000 7fd336ffd700 1 --2- 172.21.15.16:0/628521379 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd338160af0 0x7fd3381f3f90 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fd32c00c0e0 tx=0x7fd32c001230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:14.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.182+0000 7fd31ffff700 1 -- 172.21.15.16:0/628521379 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd32c012af0 con 0x7fd338160af0 2023-10-16T12:11:14.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.182+0000 7fd31ffff700 1 -- 172.21.15.16:0/628521379 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd32c021070 con 0x7fd338160af0 2023-10-16T12:11:14.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.182+0000 7fd33de68700 1 -- 172.21.15.16:0/628521379 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd33815fdb0 con 0x7fd338160af0 2023-10-16T12:11:14.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.182+0000 7fd31ffff700 1 -- 172.21.15.16:0/628521379 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd32c012e90 con 0x7fd338160af0 2023-10-16T12:11:14.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.183+0000 7fd33de68700 1 -- 172.21.15.16:0/628521379 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3380c3450 con 0x7fd338160af0 2023-10-16T12:11:14.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.183+0000 7fd33de68700 1 -- 172.21.15.16:0/628521379 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd338073580 con 0x7fd338160af0 2023-10-16T12:11:14.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.184+0000 7fd31ffff700 1 -- 172.21.15.16:0/628521379 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fd32c023030 con 0x7fd338160af0 2023-10-16T12:11:14.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.184+0000 7fd31ffff700 1 --2- 172.21.15.16:0/628521379 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fd32406af00 0x7fd32406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:14.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.184+0000 7fd31ffff700 1 -- 172.21.15.16:0/628521379 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fd32c08a910 con 0x7fd338160af0 2023-10-16T12:11:14.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.184+0000 7fd3377fe700 1 --2- 172.21.15.16:0/628521379 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fd32406af00 0x7fd32406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:14.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.185+0000 7fd3377fe700 1 --2- 172.21.15.16:0/628521379 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fd32406af00 0x7fd32406d3c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fd328009b50 tx=0x7fd328008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:14.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.189+0000 7fd31ffff700 1 -- 172.21.15.16:0/628521379 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd32c055760 con 0x7fd338160af0 2023-10-16T12:11:14.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.439+0000 7fd33de68700 1 -- 172.21.15.16:0/628521379 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd3380c3610 con 0x7fd338160af0 2023-10-16T12:11:14.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.439+0000 7fd31ffff700 1 -- 172.21.15.16:0/628521379 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4909 (secure 0 0 0) 0x7fd32c058d80 con 0x7fd338160af0 2023-10-16T12:11:14.442 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:11:14.442 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/2979266800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":2979266800},{"type":"v1","addr":"172.21.15.49:6834","nonce":2979266800}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4251,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/1398336801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1398336801},{"type":"v1","addr":"172.21.15.16:6836","nonce":1398336801}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4254,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3168304671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":3168304671},{"type":"v1","addr":"172.21.15.16:6837","nonce":3168304671}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:10:39.030243+0000","modified":"2023-10-16T12:11:02.593358+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4269":{"gid":4269,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.49:6835/2892536404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":2892536404},{"type":"v1","addr":"172.21.15.49:6835","nonce":2892536404}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T12:11:14.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.442+0000 7fd33de68700 1 -- 172.21.15.16:0/628521379 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fd32406af00 msgr2=0x7fd32406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:14.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.442+0000 7fd33de68700 1 --2- 172.21.15.16:0/628521379 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fd32406af00 0x7fd32406d3c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fd328009b50 tx=0x7fd328008040 comp rx=0 tx=0).stop 2023-10-16T12:11:14.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.442+0000 7fd33de68700 1 -- 172.21.15.16:0/628521379 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd338160af0 msgr2=0x7fd3381f3f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:14.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.442+0000 7fd33de68700 1 --2- 172.21.15.16:0/628521379 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd338160af0 0x7fd3381f3f90 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fd32c00c0e0 tx=0x7fd32c001230 comp rx=0 tx=0).stop 2023-10-16T12:11:14.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.442+0000 7fd33de68700 1 -- 172.21.15.16:0/628521379 shutdown_connections 2023-10-16T12:11:14.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.442+0000 7fd33de68700 1 --2- 172.21.15.16:0/628521379 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fd32406af00 0x7fd32406d3c0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:14.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.442+0000 7fd33de68700 1 --2- 172.21.15.16:0/628521379 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd3381f44d0 0x7fd3380c6200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:14.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.442+0000 7fd33de68700 1 --2- 172.21.15.16:0/628521379 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd320003f70 0x7fd3381605b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:14.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.442+0000 7fd33de68700 1 --2- 172.21.15.16:0/628521379 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd338160af0 0x7fd3381f3f90 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:14.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.442+0000 7fd33de68700 1 -- 172.21.15.16:0/628521379 >> 172.21.15.16:0/628521379 conn(0x7fd3380b1fd0 msgr2=0x7fd338072210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:14.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.442+0000 7fd33de68700 1 -- 172.21.15.16:0/628521379 shutdown_connections 2023-10-16T12:11:14.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.442+0000 7fd33de68700 1 -- 172.21.15.16:0/628521379 wait complete. 2023-10-16T12:11:14.447 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 9 2023-10-16T12:11:14.456 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:11:14.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.579+0000 7f589b6b1700 1 Processor -- start 2023-10-16T12:11:14.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.579+0000 7f589b6b1700 1 -- start start 2023-10-16T12:11:14.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.580+0000 7f589b6b1700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f589415d750 0x7f58940cf920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:14.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.580+0000 7f589b6b1700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f589415db30 0x7f589406c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:14.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.580+0000 7f589b6b1700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58940d0070 con 0x7f5894161490 2023-10-16T12:11:14.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.580+0000 7f589b6b1700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58940cfe60 con 0x7f589415db30 2023-10-16T12:11:14.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.580+0000 7f589b6b1700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58940d0350 con 0x7f589415d750 2023-10-16T12:11:14.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.580+0000 7f5898c4c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f589415db30 0x7f589406c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:14.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.580+0000 7f5898c4c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f589415db30 0x7f589406c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34086/0 (socket says 172.21.15.16:34086) 2023-10-16T12:11:14.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.580+0000 7f5898c4c700 1 -- 172.21.15.16:0/1945914174 learned_addr learned my addr 172.21.15.16:0/1945914174 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:14.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.580+0000 7f589944d700 1 --2- 172.21.15.16:0/1945914174 >> v2:172.21.15.49:3300/0 conn(0x7f589415d750 0x7f58940cf920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:14.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.580+0000 7f5898c4c700 1 -- 172.21.15.16:0/1945914174 >> v2:172.21.15.49:3300/0 conn(0x7f589415d750 msgr2=0x7f58940cf920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:14.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.580+0000 7f5898c4c700 1 --2- 172.21.15.16:0/1945914174 >> v2:172.21.15.49:3300/0 conn(0x7f589415d750 0x7f58940cf920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:14.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.580+0000 7f5898c4c700 1 -- 172.21.15.16:0/1945914174 >> v1:172.21.15.49:6789/0 conn(0x7f5894161490 legacy=0x7f5894161870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:14.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.580+0000 7f5898c4c700 1 -- 172.21.15.16:0/1945914174 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58940d0630 con 0x7f589415db30 2023-10-16T12:11:14.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.580+0000 7f5898c4c700 1 --2- 172.21.15.16:0/1945914174 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f589415db30 0x7f589406c960 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f589406e9b0 tx=0x7f5884009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdea7b6482ef7c42 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:14.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.581+0000 7f588bfff700 1 -- 172.21.15.16:0/1945914174 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f588400b870 con 0x7f589415db30 2023-10-16T12:11:14.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.581+0000 7f588bfff700 1 -- 172.21.15.16:0/1945914174 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f588400b9d0 con 0x7f589415db30 2023-10-16T12:11:14.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.581+0000 7f588bfff700 1 -- 172.21.15.16:0/1945914174 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f588400bd60 con 0x7f589415db30 2023-10-16T12:11:14.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.581+0000 7f589b6b1700 1 -- 172.21.15.16:0/1945914174 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f589415db30 msgr2=0x7f589406c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:14.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.581+0000 7f589b6b1700 1 --2- 172.21.15.16:0/1945914174 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f589415db30 0x7f589406c960 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f589406e9b0 tx=0x7f5884009bd0 comp rx=0 tx=0).stop 2023-10-16T12:11:14.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.581+0000 7f589b6b1700 1 -- 172.21.15.16:0/1945914174 shutdown_connections 2023-10-16T12:11:14.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.581+0000 7f589b6b1700 1 --2- 172.21.15.16:0/1945914174 >> v2:172.21.15.49:3300/0 conn(0x7f589415d750 0x7f58940cf920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:14.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.581+0000 7f589b6b1700 1 --2- 172.21.15.16:0/1945914174 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f589415db30 0x7f589406c960 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:14.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.581+0000 7f589b6b1700 1 -- 172.21.15.16:0/1945914174 >> 172.21.15.16:0/1945914174 conn(0x7f58940d14b0 msgr2=0x7f58940d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:14.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.581+0000 7f589b6b1700 1 -- 172.21.15.16:0/1945914174 shutdown_connections 2023-10-16T12:11:14.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.581+0000 7f589b6b1700 1 -- 172.21.15.16:0/1945914174 wait complete. 2023-10-16T12:11:14.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.582+0000 7f589b6b1700 1 Processor -- start 2023-10-16T12:11:14.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.583+0000 7f589b6b1700 1 -- start start 2023-10-16T12:11:14.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.583+0000 7f589b6b1700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f589415d750 0x7f5894152fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:14.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.583+0000 7f589b6b1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f589415db30 0x7f58940c7570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:14.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.583+0000 7f589b6b1700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5894161490 0x7f58940b6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:14.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.583+0000 7f589b6b1700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58940b5540 con 0x7f589415db30 2023-10-16T12:11:14.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.583+0000 7f589b6b1700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58940b44a0 con 0x7f5894161490 2023-10-16T12:11:14.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.583+0000 7f589b6b1700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58940b4780 con 0x7f589415d750 2023-10-16T12:11:14.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.583+0000 7f5898c4c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f589415db30 0x7f58940c7570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:14.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.583+0000 7f5898c4c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f589415db30 0x7f58940c7570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34092/0 (socket says 172.21.15.16:34092) 2023-10-16T12:11:14.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.583+0000 7f5898c4c700 1 -- 172.21.15.16:0/2308699792 learned_addr learned my addr 172.21.15.16:0/2308699792 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:14.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.584+0000 7f589944d700 1 --2- 172.21.15.16:0/2308699792 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f589415d750 0x7f5894152fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:14.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.584+0000 7f5899c4e700 1 --2- 172.21.15.16:0/2308699792 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5894161490 0x7f58940b6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:14.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.584+0000 7f5898c4c700 1 -- 172.21.15.16:0/2308699792 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f589415d750 msgr2=0x7f5894152fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:14.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.584+0000 7f5898c4c700 1 --2- 172.21.15.16:0/2308699792 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f589415d750 0x7f5894152fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:14.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.584+0000 7f5898c4c700 1 -- 172.21.15.16:0/2308699792 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5894161490 msgr2=0x7f58940b6100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:14.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.584+0000 7f5898c4c700 1 --2- 172.21.15.16:0/2308699792 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5894161490 0x7f58940b6100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:14.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.584+0000 7f5898c4c700 1 -- 172.21.15.16:0/2308699792 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5884009550 con 0x7f589415db30 2023-10-16T12:11:14.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.584+0000 7f589944d700 1 --2- 172.21.15.16:0/2308699792 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f589415d750 0x7f5894152fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:11:14.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.584+0000 7f5898c4c700 1 --2- 172.21.15.16:0/2308699792 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f589415db30 0x7f58940c7570 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f5884000c00 tx=0x7f5884004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:14.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.584+0000 7f5889ffb700 1 -- 172.21.15.16:0/2308699792 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5884004540 con 0x7f589415db30 2023-10-16T12:11:14.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.584+0000 7f5889ffb700 1 -- 172.21.15.16:0/2308699792 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58840036a0 con 0x7f589415db30 2023-10-16T12:11:14.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.584+0000 7f589b6b1700 1 -- 172.21.15.16:0/2308699792 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58940b4a60 con 0x7f589415db30 2023-10-16T12:11:14.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.584+0000 7f5889ffb700 1 -- 172.21.15.16:0/2308699792 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5884003a30 con 0x7f589415db30 2023-10-16T12:11:14.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.584+0000 7f589b6b1700 1 -- 172.21.15.16:0/2308699792 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58940b5020 con 0x7f589415db30 2023-10-16T12:11:14.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.585+0000 7f589b6b1700 1 -- 172.21.15.16:0/2308699792 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f589404ebb0 con 0x7f589415db30 2023-10-16T12:11:14.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.586+0000 7f5889ffb700 1 -- 172.21.15.16:0/2308699792 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f5884021070 con 0x7f589415db30 2023-10-16T12:11:14.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.586+0000 7f5889ffb700 1 --2- 172.21.15.16:0/2308699792 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f587806af00 0x7f587806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:14.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.586+0000 7f5889ffb700 1 -- 172.21.15.16:0/2308699792 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f588408af50 con 0x7f589415db30 2023-10-16T12:11:14.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.586+0000 7f589944d700 1 --2- 172.21.15.16:0/2308699792 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f587806af00 0x7f587806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:14.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.586+0000 7f589944d700 1 --2- 172.21.15.16:0/2308699792 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f587806af00 0x7f587806d3c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f5894160970 tx=0x7f5890008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:14.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.590+0000 7f5889ffb700 1 -- 172.21.15.16:0/2308699792 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5884055ec0 con 0x7f589415db30 2023-10-16T12:11:14.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.793+0000 7f589b6b1700 1 -- 172.21.15.16:0/2308699792 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f58940703a0 con 0x7f589415db30 2023-10-16T12:11:14.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.794+0000 7f5889ffb700 1 -- 172.21.15.16:0/2308699792 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18238 (secure 0 0 0) 0x7f5884015020 con 0x7f589415db30 2023-10-16T12:11:14.796 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:11:14.796 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":25,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:10:44.080141+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:10:36.935773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T12:10:38.004870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-16T12:10:41.672025+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:11:14.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.796+0000 7f589b6b1700 1 -- 172.21.15.16:0/2308699792 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f587806af00 msgr2=0x7f587806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:14.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.796+0000 7f589b6b1700 1 --2- 172.21.15.16:0/2308699792 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f587806af00 0x7f587806d3c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f5894160970 tx=0x7f5890008040 comp rx=0 tx=0).stop 2023-10-16T12:11:14.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.796+0000 7f589b6b1700 1 -- 172.21.15.16:0/2308699792 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f589415db30 msgr2=0x7f58940c7570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:14.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.796+0000 7f589b6b1700 1 --2- 172.21.15.16:0/2308699792 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f589415db30 0x7f58940c7570 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f5884000c00 tx=0x7f5884004750 comp rx=0 tx=0).stop 2023-10-16T12:11:14.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.796+0000 7f589b6b1700 1 -- 172.21.15.16:0/2308699792 shutdown_connections 2023-10-16T12:11:14.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.796+0000 7f589b6b1700 1 --2- 172.21.15.16:0/2308699792 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f587806af00 0x7f587806d3c0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:14.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.796+0000 7f589b6b1700 1 --2- 172.21.15.16:0/2308699792 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5894161490 0x7f58940b6100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:14.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.796+0000 7f589b6b1700 1 --2- 172.21.15.16:0/2308699792 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f589415d750 0x7f5894152fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:14.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.796+0000 7f589b6b1700 1 --2- 172.21.15.16:0/2308699792 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f589415db30 0x7f58940c7570 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:14.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.796+0000 7f589b6b1700 1 -- 172.21.15.16:0/2308699792 >> 172.21.15.16:0/2308699792 conn(0x7f58940d14b0 msgr2=0x7f5894161870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:14.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.796+0000 7f589b6b1700 1 -- 172.21.15.16:0/2308699792 shutdown_connections 2023-10-16T12:11:14.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.796+0000 7f589b6b1700 1 -- 172.21.15.16:0/2308699792 wait complete. 2023-10-16T12:11:14.810 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:11:14.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.933+0000 7fa605b7d700 1 Processor -- start 2023-10-16T12:11:14.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.933+0000 7fa605b7d700 1 -- start start 2023-10-16T12:11:14.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.934+0000 7fa605b7d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa60015b9f0 0x7fa6000b29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:14.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.934+0000 7fa605b7d700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fa6000cf280 0x7fa60006ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:14.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.934+0000 7fa605b7d700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6000c3490 con 0x7fa60006d940 2023-10-16T12:11:14.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.934+0000 7fa605b7d700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa60006e300 con 0x7fa6000cf280 2023-10-16T12:11:14.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.934+0000 7fa605b7d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6000cdad0 con 0x7fa60015b9f0 2023-10-16T12:11:14.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.934+0000 7fa5fffff700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fa6000cf280 0x7fa60006ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:14.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.934+0000 7fa5ff7fe700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa60015b9f0 0x7fa6000b29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:14.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.934+0000 7fa5feffd700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7fa60006d940 0x7fa6000ceb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:58790/0 (socket says 172.21.15.16:58790) 2023-10-16T12:11:14.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.934+0000 7fa5feffd700 1 -- 172.21.15.16:0/2820429538 learned_addr learned my addr 172.21.15.16:0/2820429538 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:14.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.934+0000 7fa5fffff700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fa6000cf280 0x7fa60006ddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:33160/0 (socket says 172.21.15.16:33160) 2023-10-16T12:11:14.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.934+0000 7fa5ff7fe700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa60015b9f0 0x7fa6000b29a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:50588/0 (socket says 172.21.15.16:50588) 2023-10-16T12:11:14.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.935+0000 7fa5fffff700 1 -- 172.21.15.16:0/2820429538 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa60015b9f0 msgr2=0x7fa6000b29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:14.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.935+0000 7fa5fffff700 1 --2- 172.21.15.16:0/2820429538 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa60015b9f0 0x7fa6000b29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:14.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.935+0000 7fa5fffff700 1 -- 172.21.15.16:0/2820429538 >> v1:172.21.15.49:6789/0 conn(0x7fa60006d940 legacy=0x7fa6000ceb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:14.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.935+0000 7fa5fffff700 1 -- 172.21.15.16:0/2820429538 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6000cddb0 con 0x7fa6000cf280 2023-10-16T12:11:14.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.935+0000 7fa5ff7fe700 1 --2- 172.21.15.16:0/2820429538 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa60015b9f0 0x7fa6000b29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:11:14.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.935+0000 7fa5fffff700 1 --2- 172.21.15.16:0/2820429538 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa6000cf280 0x7fa60006ddc0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa5f8009d90 tx=0x7fa5f800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b47568723e8205e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:14.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.935+0000 7fa5fe7fc700 1 -- 172.21.15.16:0/2820429538 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5f800e070 con 0x7fa6000cf280 2023-10-16T12:11:14.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.935+0000 7fa5fe7fc700 1 -- 172.21.15.16:0/2820429538 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5f800bb20 con 0x7fa6000cf280 2023-10-16T12:11:14.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.935+0000 7fa5fe7fc700 1 -- 172.21.15.16:0/2820429538 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5f8012430 con 0x7fa6000cf280 2023-10-16T12:11:14.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.936+0000 7fa605b7d700 1 -- 172.21.15.16:0/2820429538 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa6000cf280 msgr2=0x7fa60006ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:14.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.936+0000 7fa605b7d700 1 --2- 172.21.15.16:0/2820429538 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa6000cf280 0x7fa60006ddc0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa5f8009d90 tx=0x7fa5f800b330 comp rx=0 tx=0).stop 2023-10-16T12:11:14.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.936+0000 7fa605b7d700 1 -- 172.21.15.16:0/2820429538 shutdown_connections 2023-10-16T12:11:14.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.936+0000 7fa605b7d700 1 --2- 172.21.15.16:0/2820429538 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa6000cf280 0x7fa60006ddc0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:14.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.936+0000 7fa605b7d700 1 --2- 172.21.15.16:0/2820429538 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa60015b9f0 0x7fa6000b29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:14.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.936+0000 7fa605b7d700 1 -- 172.21.15.16:0/2820429538 >> 172.21.15.16:0/2820429538 conn(0x7fa6000cc5b0 msgr2=0x7fa6000cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:14.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.936+0000 7fa605b7d700 1 -- 172.21.15.16:0/2820429538 shutdown_connections 2023-10-16T12:11:14.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.936+0000 7fa605b7d700 1 -- 172.21.15.16:0/2820429538 wait complete. 2023-10-16T12:11:14.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.937+0000 7fa605b7d700 1 Processor -- start 2023-10-16T12:11:14.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.937+0000 7fa605b7d700 1 -- start start 2023-10-16T12:11:14.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.938+0000 7fa605b7d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa60006d940 0x7fa60014b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:14.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.938+0000 7fa605b7d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6000cf280 0x7fa6000c5960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:14.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.938+0000 7fa605b7d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa60015c5b0 0x7fa6000c6d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:14.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.938+0000 7fa605b7d700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6000c7980 con 0x7fa6000cf280 2023-10-16T12:11:14.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.938+0000 7fa605b7d700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa60014b550 con 0x7fa60006d940 2023-10-16T12:11:14.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.938+0000 7fa605b7d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6000c4670 con 0x7fa60015c5b0 2023-10-16T12:11:14.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.938+0000 7fa5feffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6000cf280 0x7fa6000c5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:14.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.938+0000 7fa5feffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6000cf280 0x7fa6000c5960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34106/0 (socket says 172.21.15.16:34106) 2023-10-16T12:11:14.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.938+0000 7fa5feffd700 1 -- 172.21.15.16:0/3124156815 learned_addr learned my addr 172.21.15.16:0/3124156815 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:14.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.938+0000 7fa5ff7fe700 1 --2- 172.21.15.16:0/3124156815 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa60006d940 0x7fa60014b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:14.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.938+0000 7fa5fffff700 1 --2- 172.21.15.16:0/3124156815 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa60015c5b0 0x7fa6000c6d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:14.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.938+0000 7fa5feffd700 1 -- 172.21.15.16:0/3124156815 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa60015c5b0 msgr2=0x7fa6000c6d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:14.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.938+0000 7fa5feffd700 1 --2- 172.21.15.16:0/3124156815 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa60015c5b0 0x7fa6000c6d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:14.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.938+0000 7fa5feffd700 1 -- 172.21.15.16:0/3124156815 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa60006d940 msgr2=0x7fa60014b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:14.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.938+0000 7fa5feffd700 1 --2- 172.21.15.16:0/3124156815 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa60006d940 0x7fa60014b010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:14.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.938+0000 7fa5feffd700 1 -- 172.21.15.16:0/3124156815 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5f00094a0 con 0x7fa6000cf280 2023-10-16T12:11:14.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.938+0000 7fa5ff7fe700 1 --2- 172.21.15.16:0/3124156815 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa60006d940 0x7fa60014b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:11:14.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.938+0000 7fa5feffd700 1 --2- 172.21.15.16:0/3124156815 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6000cf280 0x7fa6000c5960 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fa5e800c480 tx=0x7fa5e800c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:14.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.939+0000 7fa5f7fff700 1 -- 172.21.15.16:0/3124156815 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e8005d70 con 0x7fa6000cf280 2023-10-16T12:11:14.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.939+0000 7fa5f7fff700 1 -- 172.21.15.16:0/3124156815 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5e800ce90 con 0x7fa6000cf280 2023-10-16T12:11:14.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.939+0000 7fa605b7d700 1 -- 172.21.15.16:0/3124156815 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5f8009a70 con 0x7fa6000cf280 2023-10-16T12:11:14.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.939+0000 7fa5f7fff700 1 -- 172.21.15.16:0/3124156815 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e80072d0 con 0x7fa6000cf280 2023-10-16T12:11:14.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.939+0000 7fa605b7d700 1 -- 172.21.15.16:0/3124156815 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6000c4ba0 con 0x7fa6000cf280 2023-10-16T12:11:14.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.940+0000 7fa605b7d700 1 -- 172.21.15.16:0/3124156815 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa60004ebb0 con 0x7fa6000cf280 2023-10-16T12:11:14.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.940+0000 7fa5f7fff700 1 -- 172.21.15.16:0/3124156815 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fa5e8007430 con 0x7fa6000cf280 2023-10-16T12:11:14.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.940+0000 7fa5f7fff700 1 --2- 172.21.15.16:0/3124156815 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fa5e006af00 0x7fa5e006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:14.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.941+0000 7fa5f7fff700 1 -- 172.21.15.16:0/3124156815 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fa5e8088ad0 con 0x7fa6000cf280 2023-10-16T12:11:14.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.941+0000 7fa5ff7fe700 1 --2- 172.21.15.16:0/3124156815 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fa5e006af00 0x7fa5e006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:14.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.941+0000 7fa5ff7fe700 1 --2- 172.21.15.16:0/3124156815 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fa5e006af00 0x7fa5e006d3c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa5f00114b0 tx=0x7fa5f0009210 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:14.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:14.945+0000 7fa5f7fff700 1 -- 172.21.15.16:0/3124156815 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa5e8053940 con 0x7fa6000cf280 2023-10-16T12:11:15.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.196+0000 7fa605b7d700 1 -- 172.21.15.16:0/3124156815 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa6000ce340 con 0x7fa6000cf280 2023-10-16T12:11:15.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.197+0000 7fa5f7fff700 1 -- 172.21.15.16:0/3124156815 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4909 (secure 0 0 0) 0x7fa5e8017090 con 0x7fa6000cf280 2023-10-16T12:11:15.199 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:11:15.199 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/2979266800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":2979266800},{"type":"v1","addr":"172.21.15.49:6834","nonce":2979266800}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4251,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/1398336801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1398336801},{"type":"v1","addr":"172.21.15.16:6836","nonce":1398336801}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4254,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3168304671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":3168304671},{"type":"v1","addr":"172.21.15.16:6837","nonce":3168304671}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:10:39.030243+0000","modified":"2023-10-16T12:11:02.593358+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4269":{"gid":4269,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.49:6835/2892536404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":2892536404},{"type":"v1","addr":"172.21.15.49:6835","nonce":2892536404}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T12:11:15.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.199+0000 7fa605b7d700 1 -- 172.21.15.16:0/3124156815 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fa5e006af00 msgr2=0x7fa5e006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:15.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.199+0000 7fa605b7d700 1 --2- 172.21.15.16:0/3124156815 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fa5e006af00 0x7fa5e006d3c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa5f00114b0 tx=0x7fa5f0009210 comp rx=0 tx=0).stop 2023-10-16T12:11:15.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.199+0000 7fa605b7d700 1 -- 172.21.15.16:0/3124156815 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6000cf280 msgr2=0x7fa6000c5960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:15.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.199+0000 7fa605b7d700 1 --2- 172.21.15.16:0/3124156815 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6000cf280 0x7fa6000c5960 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fa5e800c480 tx=0x7fa5e800c790 comp rx=0 tx=0).stop 2023-10-16T12:11:15.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.199+0000 7fa605b7d700 1 -- 172.21.15.16:0/3124156815 shutdown_connections 2023-10-16T12:11:15.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.199+0000 7fa605b7d700 1 --2- 172.21.15.16:0/3124156815 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fa5e006af00 0x7fa5e006d3c0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:15.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.199+0000 7fa605b7d700 1 --2- 172.21.15.16:0/3124156815 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa60015c5b0 0x7fa6000c6d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:15.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.199+0000 7fa605b7d700 1 --2- 172.21.15.16:0/3124156815 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa60006d940 0x7fa60014b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:15.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.199+0000 7fa605b7d700 1 --2- 172.21.15.16:0/3124156815 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6000cf280 0x7fa6000c5960 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:15.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.199+0000 7fa605b7d700 1 -- 172.21.15.16:0/3124156815 >> 172.21.15.16:0/3124156815 conn(0x7fa6000cc5b0 msgr2=0x7fa60006bb50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:15.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.199+0000 7fa605b7d700 1 -- 172.21.15.16:0/3124156815 shutdown_connections 2023-10-16T12:11:15.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.199+0000 7fa605b7d700 1 -- 172.21.15.16:0/3124156815 wait complete. 2023-10-16T12:11:15.204 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 9 2023-10-16T12:11:15.615 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 9, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T12:10:39.030243+0000', 'modified': '2023-10-16T12:11:02.593358+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4269}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4269': {'gid': 4269, 'name': 'd', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.49:6835/2892536404', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6833', 'nonce': 2892536404}, {'type': 'v1', 'addr': '172.21.15.49:6835', 'nonce': 2892536404}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-16T12:11:15.616 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-16T12:11:15.616 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2023-10-16T12:11:15.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.750+0000 7fb0be338700 1 Processor -- start 2023-10-16T12:11:15.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.751+0000 7fb0be338700 1 -- start start 2023-10-16T12:11:15.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.751+0000 7fb0be338700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb0b815b630 0x7fb0b80cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:15.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.751+0000 7fb0be338700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fb0b815ba10 0x7fb0b80cdde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:15.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.751+0000 7fb0be338700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0b80ce770 con 0x7fb0b80ce320 2023-10-16T12:11:15.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.751+0000 7fb0be338700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0b80cdb20 con 0x7fb0b815ba10 2023-10-16T12:11:15.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.751+0000 7fb0be338700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b80cc2a0 con 0x7fb0b815b630 2023-10-16T12:11:15.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.752+0000 7fb0b7fff700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb0b815b630 0x7fb0b80cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:15.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.752+0000 7fb0b77fe700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fb0b815ba10 0x7fb0b80cdde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:15.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.752+0000 7fb0bc8d5700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7fb0b80ce320 0x7fb0b80cfca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:58802/0 (socket says 172.21.15.16:58802) 2023-10-16T12:11:15.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.752+0000 7fb0b77fe700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fb0b815ba10 0x7fb0b80cdde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:33174/0 (socket says 172.21.15.16:33174) 2023-10-16T12:11:15.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.752+0000 7fb0bc8d5700 1 -- 172.21.15.16:0/1755637691 learned_addr learned my addr 172.21.15.16:0/1755637691 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:15.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.752+0000 7fb0b7fff700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb0b815b630 0x7fb0b80cd5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:50602/0 (socket says 172.21.15.16:50602) 2023-10-16T12:11:15.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.752+0000 7fb0b77fe700 1 -- 172.21.15.16:0/1755637691 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb0b815b630 msgr2=0x7fb0b80cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:15.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.752+0000 7fb0b77fe700 1 --2- 172.21.15.16:0/1755637691 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb0b815b630 0x7fb0b80cd5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:15.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.752+0000 7fb0b77fe700 1 -- 172.21.15.16:0/1755637691 >> v1:172.21.15.49:6789/0 conn(0x7fb0b80ce320 legacy=0x7fb0b80cfca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:15.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.752+0000 7fb0b77fe700 1 -- 172.21.15.16:0/1755637691 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0b80cc540 con 0x7fb0b815ba10 2023-10-16T12:11:15.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.752+0000 7fb0b7fff700 1 --2- 172.21.15.16:0/1755637691 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb0b815b630 0x7fb0b80cd5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:11:15.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.752+0000 7fb0b77fe700 1 --2- 172.21.15.16:0/1755637691 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb0b815ba10 0x7fb0b80cdde0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb0a8009840 tx=0x7fb0a8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ea18fe3e7559491 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:15.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.753+0000 7fb0b6ffd700 1 -- 172.21.15.16:0/1755637691 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a8011830 con 0x7fb0b815ba10 2023-10-16T12:11:15.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.753+0000 7fb0b6ffd700 1 -- 172.21.15.16:0/1755637691 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0a8011990 con 0x7fb0b815ba10 2023-10-16T12:11:15.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.753+0000 7fb0b6ffd700 1 -- 172.21.15.16:0/1755637691 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a8011d20 con 0x7fb0b815ba10 2023-10-16T12:11:15.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.753+0000 7fb0be338700 1 -- 172.21.15.16:0/1755637691 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb0b815ba10 msgr2=0x7fb0b80cdde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:15.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.753+0000 7fb0be338700 1 --2- 172.21.15.16:0/1755637691 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb0b815ba10 0x7fb0b80cdde0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb0a8009840 tx=0x7fb0a8009b50 comp rx=0 tx=0).stop 2023-10-16T12:11:15.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.753+0000 7fb0be338700 1 -- 172.21.15.16:0/1755637691 shutdown_connections 2023-10-16T12:11:15.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.753+0000 7fb0be338700 1 --2- 172.21.15.16:0/1755637691 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb0b815b630 0x7fb0b80cd5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:15.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.753+0000 7fb0be338700 1 --2- 172.21.15.16:0/1755637691 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb0b815ba10 0x7fb0b80cdde0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:15.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.753+0000 7fb0be338700 1 -- 172.21.15.16:0/1755637691 >> 172.21.15.16:0/1755637691 conn(0x7fb0b80d14b0 msgr2=0x7fb0b80d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:15.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.753+0000 7fb0be338700 1 -- 172.21.15.16:0/1755637691 shutdown_connections 2023-10-16T12:11:15.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.753+0000 7fb0be338700 1 -- 172.21.15.16:0/1755637691 wait complete. 2023-10-16T12:11:15.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.754+0000 7fb0be338700 1 Processor -- start 2023-10-16T12:11:15.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.755+0000 7fb0be338700 1 -- start start 2023-10-16T12:11:15.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.755+0000 7fb0be338700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb0b80ce320 0x7fb0b81589c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:15.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.755+0000 7fb0be338700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0b815b630 0x7fb0b81f17e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:15.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.755+0000 7fb0be338700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb0b815ba10 0x7fb0b809ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:15.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.755+0000 7fb0be338700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b81f1d20 con 0x7fb0b815b630 2023-10-16T12:11:15.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.755+0000 7fb0be338700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b8159340 con 0x7fb0b815ba10 2023-10-16T12:11:15.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.755+0000 7fb0be338700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b81f0c60 con 0x7fb0b80ce320 2023-10-16T12:11:15.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.755+0000 7fb0b77fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0b815b630 0x7fb0b81f17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:15.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.755+0000 7fb0b77fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0b815b630 0x7fb0b81f17e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34118/0 (socket says 172.21.15.16:34118) 2023-10-16T12:11:15.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.755+0000 7fb0b77fe700 1 -- 172.21.15.16:0/999041856 learned_addr learned my addr 172.21.15.16:0/999041856 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:15.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.756+0000 7fb0bc8d5700 1 --2- 172.21.15.16:0/999041856 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb0b815ba10 0x7fb0b809ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:15.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.756+0000 7fb0b7fff700 1 --2- 172.21.15.16:0/999041856 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb0b80ce320 0x7fb0b81589c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:15.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.756+0000 7fb0b77fe700 1 -- 172.21.15.16:0/999041856 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb0b80ce320 msgr2=0x7fb0b81589c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:15.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.756+0000 7fb0b77fe700 1 --2- 172.21.15.16:0/999041856 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb0b80ce320 0x7fb0b81589c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:15.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.756+0000 7fb0b77fe700 1 -- 172.21.15.16:0/999041856 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb0b815ba10 msgr2=0x7fb0b809ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:15.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.756+0000 7fb0b77fe700 1 --2- 172.21.15.16:0/999041856 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb0b815ba10 0x7fb0b809ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:15.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.756+0000 7fb0b77fe700 1 -- 172.21.15.16:0/999041856 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0ac009430 con 0x7fb0b815b630 2023-10-16T12:11:15.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.756+0000 7fb0bc8d5700 1 --2- 172.21.15.16:0/999041856 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb0b815ba10 0x7fb0b809ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:11:15.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.756+0000 7fb0b7fff700 1 --2- 172.21.15.16:0/999041856 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb0b80ce320 0x7fb0b81589c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:11:15.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.756+0000 7fb0b77fe700 1 --2- 172.21.15.16:0/999041856 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0b815b630 0x7fb0b81f17e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fb0a8009b90 tx=0x7fb0a8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:15.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.756+0000 7fb0b4ff9700 1 -- 172.21.15.16:0/999041856 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a80038a0 con 0x7fb0b815b630 2023-10-16T12:11:15.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.756+0000 7fb0b4ff9700 1 -- 172.21.15.16:0/999041856 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0a8003a00 con 0x7fb0b815b630 2023-10-16T12:11:15.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.756+0000 7fb0be338700 1 -- 172.21.15.16:0/999041856 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0a80094a0 con 0x7fb0b815b630 2023-10-16T12:11:15.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.756+0000 7fb0b4ff9700 1 -- 172.21.15.16:0/999041856 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a8003d90 con 0x7fb0b815b630 2023-10-16T12:11:15.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.756+0000 7fb0be338700 1 -- 172.21.15.16:0/999041856 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0b81f1220 con 0x7fb0b815b630 2023-10-16T12:11:15.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.758+0000 7fb0b4ff9700 1 -- 172.21.15.16:0/999041856 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fb0a802e450 con 0x7fb0b815b630 2023-10-16T12:11:15.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.758+0000 7fb0b4ff9700 1 --2- 172.21.15.16:0/999041856 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fb0a406af00 0x7fb0a406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:15.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.758+0000 7fb0b4ff9700 1 -- 172.21.15.16:0/999041856 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fb0a808acd0 con 0x7fb0b815b630 2023-10-16T12:11:15.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.758+0000 7fb0b7fff700 1 --2- 172.21.15.16:0/999041856 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fb0a406af00 0x7fb0a406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:15.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.759+0000 7fb0b7fff700 1 --2- 172.21.15.16:0/999041856 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fb0a406af00 0x7fb0a406d3c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb0ac009cd0 tx=0x7fb0ac008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:15.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.760+0000 7fb0ca522b80 1 Processor -- start 2023-10-16T12:11:15.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.760+0000 7fb0ca522b80 1 -- start start 2023-10-16T12:11:15.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.761+0000 7fb0ca522b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d97586220 0x556d9758da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:15.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.761+0000 7fb0ca522b80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x556d9758b630 0x556d9758dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:15.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.761+0000 7fb0ca522b80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556d97561010 0x556d975616c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:15.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.761+0000 7fb0ca522b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d974d7c20 con 0x556d97586220 2023-10-16T12:11:15.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.761+0000 7fb0ca522b80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d974bcc10 con 0x556d97561010 2023-10-16T12:11:15.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.761+0000 7fb0ca522b80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x556d974ba530 con 0x556d9758b630 2023-10-16T12:11:15.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.761+0000 7fb0b7fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d97586220 0x556d9758da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:15.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.761+0000 7fb0b7fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d97586220 0x556d9758da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34124/0 (socket says 172.21.15.16:34124) 2023-10-16T12:11:15.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.761+0000 7fb0b7fff700 1 -- 172.21.15.16:0/1939351769 learned_addr learned my addr 172.21.15.16:0/1939351769 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:15.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.761+0000 7fb0b77fe700 1 --2- 172.21.15.16:0/1939351769 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x556d9758b630 0x556d9758dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:15.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.761+0000 7fb0bc8d5700 1 --2- 172.21.15.16:0/1939351769 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556d97561010 0x556d975616c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:15.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.761+0000 7fb0b7fff700 1 -- 172.21.15.16:0/1939351769 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x556d9758b630 msgr2=0x556d9758dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:15.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.761+0000 7fb0b7fff700 1 --2- 172.21.15.16:0/1939351769 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x556d9758b630 0x556d9758dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:15.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.761+0000 7fb0b7fff700 1 -- 172.21.15.16:0/1939351769 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556d97561010 msgr2=0x556d975616c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:15.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.761+0000 7fb0b7fff700 1 --2- 172.21.15.16:0/1939351769 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556d97561010 0x556d975616c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:15.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.761+0000 7fb0b7fff700 1 -- 172.21.15.16:0/1939351769 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556d974733a0 con 0x556d97586220 2023-10-16T12:11:15.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.762+0000 7fb0b7fff700 1 --2- 172.21.15.16:0/1939351769 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d97586220 0x556d9758da70 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb0ac010c00 tx=0x7fb0ac013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef7bfad7ab369094 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:15.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.762+0000 7fb09dffb700 1 -- 172.21.15.16:0/1939351769 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0ac015d10 con 0x556d97586220 2023-10-16T12:11:15.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.762+0000 7fb09dffb700 1 -- 172.21.15.16:0/1939351769 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0ac01a070 con 0x556d97586220 2023-10-16T12:11:15.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.762+0000 7fb09dffb700 1 -- 172.21.15.16:0/1939351769 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0ac021660 con 0x556d97586220 2023-10-16T12:11:15.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.762+0000 7fb0ca522b80 1 -- 172.21.15.16:0/1939351769 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d97586220 msgr2=0x556d9758da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:15.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.762+0000 7fb0ca522b80 1 --2- 172.21.15.16:0/1939351769 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d97586220 0x556d9758da70 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb0ac010c00 tx=0x7fb0ac013000 comp rx=0 tx=0).stop 2023-10-16T12:11:15.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.762+0000 7fb0ca522b80 1 -- 172.21.15.16:0/1939351769 shutdown_connections 2023-10-16T12:11:15.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.762+0000 7fb0ca522b80 1 --2- 172.21.15.16:0/1939351769 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556d97561010 0x556d975616c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:15.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.762+0000 7fb0ca522b80 1 --2- 172.21.15.16:0/1939351769 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d97586220 0x556d9758da70 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:15.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.762+0000 7fb0ca522b80 1 --2- 172.21.15.16:0/1939351769 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x556d9758b630 0x556d9758dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:15.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.762+0000 7fb0ca522b80 1 -- 172.21.15.16:0/1939351769 >> 172.21.15.16:0/1939351769 conn(0x556d97490d80 msgr2=0x556d975abff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:15.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.762+0000 7fb0ca522b80 1 -- 172.21.15.16:0/1939351769 shutdown_connections 2023-10-16T12:11:15.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.763+0000 7fb0ca522b80 1 -- 172.21.15.16:0/1939351769 wait complete. 2023-10-16T12:11:15.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.763+0000 7fb0ca522b80 1 -- start start 2023-10-16T12:11:15.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.763+0000 7fb0ca522b80 1 Processor -- start 2023-10-16T12:11:15.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.764+0000 7fb0ca522b80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556d97561010 0x556d97585240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:15.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.764+0000 7fb0ca522b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d97586220 0x556d975873e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:15.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.764+0000 7fb0ca522b80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x556d9758b630 0x556d97561480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:15.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.764+0000 7fb0ca522b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d974b5e20 con 0x556d97586220 2023-10-16T12:11:15.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.764+0000 7fb0ca522b80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d974526c0 con 0x556d97561010 2023-10-16T12:11:15.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.764+0000 7fb0ca522b80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x556d97452360 con 0x556d9758b630 2023-10-16T12:11:15.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.764+0000 7fb0ca522b80 10 client.0 _ll_register_callbacks cb 0x556d972af7b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T12:11:15.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.764+0000 7fb0b77fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d97586220 0x556d975873e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:15.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.764+0000 7fb0b77fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d97586220 0x556d975873e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34132/0 (socket says 172.21.15.16:34132) 2023-10-16T12:11:15.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.764+0000 7fb0b77fe700 1 -- 172.21.15.16:0/3717553439 learned_addr learned my addr 172.21.15.16:0/3717553439 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:15.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.764+0000 7fb0b7fff700 1 --2- 172.21.15.16:0/3717553439 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556d97561010 0x556d97585240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:15.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.764+0000 7fb0bc8d5700 1 --2- 172.21.15.16:0/3717553439 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x556d9758b630 0x556d97561480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:15.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.765+0000 7fb0b77fe700 1 -- 172.21.15.16:0/3717553439 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x556d9758b630 msgr2=0x556d97561480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:15.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.765+0000 7fb0b77fe700 1 --2- 172.21.15.16:0/3717553439 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x556d9758b630 0x556d97561480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:15.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.765+0000 7fb0b77fe700 1 -- 172.21.15.16:0/3717553439 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556d97561010 msgr2=0x556d97585240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:15.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.765+0000 7fb0b77fe700 1 --2- 172.21.15.16:0/3717553439 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556d97561010 0x556d97585240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:15.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.765+0000 7fb0b77fe700 1 -- 172.21.15.16:0/3717553439 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0ac00e050 con 0x556d97586220 2023-10-16T12:11:15.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.765+0000 7fb0bc8d5700 1 --2- 172.21.15.16:0/3717553439 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x556d9758b630 0x556d97561480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:11:15.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.765+0000 7fb0b77fe700 1 --2- 172.21.15.16:0/3717553439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d97586220 0x556d975873e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fb0a80594e0 tx=0x7fb0a8059230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27a0ff2ad71d2dbb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:15.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.765+0000 7fb096ffd700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2023-10-16T12:11:15.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.765+0000 7fb096ffd700 1 -- 172.21.15.16:0/3717553439 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a8058ed0 con 0x556d97586220 2023-10-16T12:11:15.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.765+0000 7fb096ffd700 1 -- 172.21.15.16:0/3717553439 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fb0a8071030 con 0x556d97586220 2023-10-16T12:11:15.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.765+0000 7fb0ca522b80 1 -- 172.21.15.16:0/3717553439 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556d975565b0 con 0x556d97586220 2023-10-16T12:11:15.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.765+0000 7fb096ffd700 1 -- 172.21.15.16:0/3717553439 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=9) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb0a8059ee0 con 0x556d97586220 2023-10-16T12:11:15.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.765+0000 7fb0ca522b80 10 client.4791 fetch_fsmap learned FSMap version 9 2023-10-16T12:11:15.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.765+0000 7fb0ca522b80 1 -- 172.21.15.16:0/3717553439 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=9,monmap=2+}) v3 -- 0x556d972e0d90 con 0x556d97586220 2023-10-16T12:11:15.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.765+0000 7fb096ffd700 1 -- 172.21.15.16:0/3717553439 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x556d975565b0 con 0x556d97586220 2023-10-16T12:11:15.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.765+0000 7fb096ffd700 1 -- 172.21.15.16:0/3717553439 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 9) v1 ==== 3302+0+0 (secure 0 0 0) 0x7fb0a8061650 con 0x556d97586220 2023-10-16T12:11:15.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.765+0000 7fb0ca522b80 10 client.4791 fetch_fsmap finished waiting for FSMap version 9 2023-10-16T12:11:15.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.765+0000 7fb0ca522b80 10 client.4791 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-16T12:11:15.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.765+0000 7fb0ca522b80 20 client.4791 populate_metadata read hostname 'smithi016' 2023-10-16T12:11:15.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.765+0000 7fb0ca522b80 1 --2- 172.21.15.16:0/3717553439 >> [v2:172.21.15.49:6833/2892536404,v1:172.21.15.49:6835/2892536404] conn(0x556d97589810 0x556d97589bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:15.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.765+0000 7fb0ca522b80 4 client.4791 mds_command: new command op to 4269 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T12:11:15.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.765+0000 7fb0ca522b80 1 -- 172.21.15.16:0/3717553439 --> [v2:172.21.15.49:6833/2892536404,v1:172.21.15.49:6835/2892536404] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556d974aee00 con 0x556d97589810 2023-10-16T12:11:15.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.766+0000 7fb0b7fff700 1 --2- 172.21.15.16:0/3717553439 >> [v2:172.21.15.49:6833/2892536404,v1:172.21.15.49:6835/2892536404] conn(0x556d97589810 0x556d97589bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:15.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.766+0000 7fb096ffd700 10 client.4791 ms_handle_connect on v2:172.21.15.49:6833/2892536404 2023-10-16T12:11:15.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.766+0000 7fb0b7fff700 1 --2- 172.21.15.16:0/3717553439 >> [v2:172.21.15.49:6833/2892536404,v1:172.21.15.49:6835/2892536404] conn(0x556d97589810 0x556d97589bf0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c8a20a92ea43a5c8 server_cookie=a20e55062da95927 in_seq=0 out_seq=0 2023-10-16T12:11:15.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.767+0000 7fb096ffd700 1 -- 172.21.15.16:0/3717553439 <== mds.0 v2:172.21.15.49:6833/2892536404 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x556d974aee00 con 0x556d97589810 2023-10-16T12:11:15.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.767+0000 7fb096ffd700 10 client.4791 handle_command_reply: tid=0 2023-10-16T12:11:15.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.767+0000 7fb0ca522b80 1 client.4791 shutdown 2023-10-16T12:11:15.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.767+0000 7fb0ca522b80 1 -- 172.21.15.16:0/3717553439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d97586220 msgr2=0x556d975873e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:15.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.767+0000 7fb0ca522b80 1 --2- 172.21.15.16:0/3717553439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d97586220 0x556d975873e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fb0a80594e0 tx=0x7fb0a8059230 comp rx=0 tx=0).stop 2023-10-16T12:11:15.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.767+0000 7fb0ca522b80 1 -- 172.21.15.16:0/3717553439 shutdown_connections 2023-10-16T12:11:15.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.767+0000 7fb0ca522b80 1 --2- 172.21.15.16:0/3717553439 >> [v2:172.21.15.49:6833/2892536404,v1:172.21.15.49:6835/2892536404] conn(0x556d97589810 0x556d97589bf0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:15.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.767+0000 7fb0ca522b80 1 --2- 172.21.15.16:0/3717553439 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x556d9758b630 0x556d97561480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:15.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.767+0000 7fb0ca522b80 1 --2- 172.21.15.16:0/3717553439 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556d97561010 0x556d97585240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:15.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.767+0000 7fb0ca522b80 1 --2- 172.21.15.16:0/3717553439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d97586220 0x556d975873e0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:15.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.767+0000 7fb096ffd700 0 client.4791 ms_handle_reset on v2:172.21.15.49:6833/2892536404 2023-10-16T12:11:15.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.767+0000 7fb0ca522b80 1 -- 172.21.15.16:0/3717553439 >> 172.21.15.16:0/3717553439 conn(0x556d97490d80 msgr2=0x556d9758da40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:15.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.767+0000 7fb0ca522b80 1 -- 172.21.15.16:0/3717553439 shutdown_connections 2023-10-16T12:11:15.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.767+0000 7fb0ca522b80 1 -- 172.21.15.16:0/3717553439 wait complete. 2023-10-16T12:11:15.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.767+0000 7fb0ca522b80 20 client.4791 trim_cache size 0 max 16384 2023-10-16T12:11:15.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.785+0000 7fb0ca522b80 1 Processor -- start 2023-10-16T12:11:15.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.785+0000 7fb0ca522b80 1 -- start start 2023-10-16T12:11:15.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.786+0000 7fb0ca522b80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556d97561010 0x556d97561480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:15.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.786+0000 7fb0ca522b80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x556d97586220 0x556d9758de40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:15.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.786+0000 7fb0ca522b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d9758b630 0x556d97586f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:15.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.786+0000 7fb0ca522b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d973368d0 con 0x556d9758b630 2023-10-16T12:11:15.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.786+0000 7fb0ca522b80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d975b7560 con 0x556d97561010 2023-10-16T12:11:15.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.786+0000 7fb0ca522b80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x556d97540490 con 0x556d97586220 2023-10-16T12:11:15.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.786+0000 7fb0bc8d5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d9758b630 0x556d97586f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:15.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.786+0000 7fb0bc8d5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d9758b630 0x556d97586f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34142/0 (socket says 172.21.15.16:34142) 2023-10-16T12:11:15.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.786+0000 7fb0bc8d5700 1 -- 172.21.15.16:0/65342589 learned_addr learned my addr 172.21.15.16:0/65342589 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:15.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.786+0000 7fb0b77fe700 1 --2- 172.21.15.16:0/65342589 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x556d97586220 0x556d9758de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:15.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.786+0000 7fb0b7fff700 1 --2- 172.21.15.16:0/65342589 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556d97561010 0x556d97561480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:15.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.786+0000 7fb0bc8d5700 1 -- 172.21.15.16:0/65342589 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x556d97586220 msgr2=0x556d9758de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:15.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.786+0000 7fb0bc8d5700 1 --2- 172.21.15.16:0/65342589 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x556d97586220 0x556d9758de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:15.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.786+0000 7fb0bc8d5700 1 -- 172.21.15.16:0/65342589 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556d97561010 msgr2=0x556d97561480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:15.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.786+0000 7fb0bc8d5700 1 --2- 172.21.15.16:0/65342589 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556d97561010 0x556d97561480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:15.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.786+0000 7fb0bc8d5700 1 -- 172.21.15.16:0/65342589 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556d975a8a50 con 0x556d9758b630 2023-10-16T12:11:15.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.786+0000 7fb0bc8d5700 1 --2- 172.21.15.16:0/65342589 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d9758b630 0x556d97586f10 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fb0b00020d0 tx=0x7fb0b0007330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b013df1d10ec93f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:15.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.787+0000 7fb09e7fc700 1 -- 172.21.15.16:0/65342589 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b0015440 con 0x556d9758b630 2023-10-16T12:11:15.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.787+0000 7fb09e7fc700 1 -- 172.21.15.16:0/65342589 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0b000d030 con 0x556d9758b630 2023-10-16T12:11:15.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.787+0000 7fb09e7fc700 1 -- 172.21.15.16:0/65342589 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b00157e0 con 0x556d9758b630 2023-10-16T12:11:15.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.787+0000 7fb0ca522b80 1 -- 172.21.15.16:0/65342589 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d9758b630 msgr2=0x556d97586f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:15.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.787+0000 7fb0ca522b80 1 --2- 172.21.15.16:0/65342589 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d9758b630 0x556d97586f10 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fb0b00020d0 tx=0x7fb0b0007330 comp rx=0 tx=0).stop 2023-10-16T12:11:15.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.787+0000 7fb0ca522b80 1 -- 172.21.15.16:0/65342589 shutdown_connections 2023-10-16T12:11:15.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.787+0000 7fb0ca522b80 1 --2- 172.21.15.16:0/65342589 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d9758b630 0x556d97586f10 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:15.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.787+0000 7fb0ca522b80 1 --2- 172.21.15.16:0/65342589 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556d97561010 0x556d97561480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:15.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.787+0000 7fb0ca522b80 1 --2- 172.21.15.16:0/65342589 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x556d97586220 0x556d9758de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:15.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.787+0000 7fb0ca522b80 1 -- 172.21.15.16:0/65342589 >> 172.21.15.16:0/65342589 conn(0x556d97490d80 msgr2=0x556d97557f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:15.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.787+0000 7fb0ca522b80 1 -- 172.21.15.16:0/65342589 shutdown_connections 2023-10-16T12:11:15.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.787+0000 7fb0ca522b80 1 -- 172.21.15.16:0/65342589 wait complete. 2023-10-16T12:11:15.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.788+0000 7fb0ca522b80 1 -- start start 2023-10-16T12:11:15.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.788+0000 7fb0ca522b80 1 Processor -- start 2023-10-16T12:11:15.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.789+0000 7fb0ca522b80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x556d97561010 0x556d97613820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:15.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.789+0000 7fb0ca522b80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556d97586220 0x556d97589bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:15.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.789+0000 7fb0ca522b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d9758b630 0x556d9758a130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:15.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.789+0000 7fb0ca522b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d97589060 con 0x556d9758b630 2023-10-16T12:11:15.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.789+0000 7fb0ca522b80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d975a49b0 con 0x556d97586220 2023-10-16T12:11:15.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.789+0000 7fb0ca522b80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x556d975a7bd0 con 0x556d97561010 2023-10-16T12:11:15.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.789+0000 7fb0ca522b80 10 client.0 _ll_register_callbacks cb 0x7fb0a8059650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T12:11:15.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.789+0000 7fb0bc8d5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d9758b630 0x556d9758a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:15.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.790+0000 7fb0bc8d5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d9758b630 0x556d9758a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34148/0 (socket says 172.21.15.16:34148) 2023-10-16T12:11:15.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.790+0000 7fb0bc8d5700 1 -- 172.21.15.16:0/2703246648 learned_addr learned my addr 172.21.15.16:0/2703246648 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:15.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.790+0000 7fb0b7fff700 1 --2- 172.21.15.16:0/2703246648 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x556d97561010 0x556d97613820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:15.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.790+0000 7fb0b77fe700 1 --2- 172.21.15.16:0/2703246648 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556d97586220 0x556d97589bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:15.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.790+0000 7fb0bc8d5700 1 -- 172.21.15.16:0/2703246648 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x556d97561010 msgr2=0x556d97613820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:15.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.790+0000 7fb0bc8d5700 1 --2- 172.21.15.16:0/2703246648 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x556d97561010 0x556d97613820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:15.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.790+0000 7fb0bc8d5700 1 -- 172.21.15.16:0/2703246648 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556d97586220 msgr2=0x556d97589bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:15.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.790+0000 7fb0bc8d5700 1 --2- 172.21.15.16:0/2703246648 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556d97586220 0x556d97589bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:15.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.790+0000 7fb0bc8d5700 1 -- 172.21.15.16:0/2703246648 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0b0001e00 con 0x556d9758b630 2023-10-16T12:11:15.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.790+0000 7fb0b7fff700 1 --2- 172.21.15.16:0/2703246648 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x556d97561010 0x556d97613820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:11:15.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.790+0000 7fb0bc8d5700 1 --2- 172.21.15.16:0/2703246648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d9758b630 0x556d9758a130 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb0b0018040 tx=0x7fb0b0001250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95cd55766018bcf5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:15.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.790+0000 7fb096ffd700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2023-10-16T12:11:15.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.790+0000 7fb096ffd700 1 -- 172.21.15.16:0/2703246648 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b0024070 con 0x556d9758b630 2023-10-16T12:11:15.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.790+0000 7fb0ca522b80 1 -- 172.21.15.16:0/2703246648 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb0ac01a070 con 0x556d9758b630 2023-10-16T12:11:15.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.790+0000 7fb096ffd700 1 -- 172.21.15.16:0/2703246648 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fb0b0015c00 con 0x556d9758b630 2023-10-16T12:11:15.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.790+0000 7fb096ffd700 1 -- 172.21.15.16:0/2703246648 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=9) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb0b0015dc0 con 0x556d9758b630 2023-10-16T12:11:15.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.790+0000 7fb0ca522b80 10 client.4797 fetch_fsmap learned FSMap version 9 2023-10-16T12:11:15.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.790+0000 7fb0ca522b80 1 -- 172.21.15.16:0/2703246648 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=9,monmap=2+}) v3 -- 0x556d972c9c10 con 0x556d9758b630 2023-10-16T12:11:15.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.790+0000 7fb096ffd700 1 -- 172.21.15.16:0/2703246648 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0ac01a070 con 0x556d9758b630 2023-10-16T12:11:15.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.791+0000 7fb096ffd700 1 -- 172.21.15.16:0/2703246648 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 9) v1 ==== 3302+0+0 (secure 0 0 0) 0x7fb0b0012050 con 0x556d9758b630 2023-10-16T12:11:15.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.791+0000 7fb0ca522b80 10 client.4797 fetch_fsmap finished waiting for FSMap version 9 2023-10-16T12:11:15.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.791+0000 7fb0ca522b80 10 client.4797 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-16T12:11:15.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.791+0000 7fb0ca522b80 20 client.4797 populate_metadata read hostname 'smithi016' 2023-10-16T12:11:15.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.791+0000 7fb0ca522b80 1 --2- 172.21.15.16:0/2703246648 >> [v2:172.21.15.49:6833/2892536404,v1:172.21.15.49:6835/2892536404] conn(0x556d975ac020 0x556d975ac400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:15.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.791+0000 7fb0ca522b80 4 client.4797 mds_command: new command op to 4269 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T12:11:15.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.791+0000 7fb0ca522b80 1 -- 172.21.15.16:0/2703246648 --> [v2:172.21.15.49:6833/2892536404,v1:172.21.15.49:6835/2892536404] -- command(tid 0: {"prefix": "status"}) v1 -- 0x556d974aeb20 con 0x556d975ac020 2023-10-16T12:11:15.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.791+0000 7fb0b7fff700 1 --2- 172.21.15.16:0/2703246648 >> [v2:172.21.15.49:6833/2892536404,v1:172.21.15.49:6835/2892536404] conn(0x556d975ac020 0x556d975ac400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:15.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.791+0000 7fb096ffd700 10 client.4797 ms_handle_connect on v2:172.21.15.49:6833/2892536404 2023-10-16T12:11:15.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.791+0000 7fb0b7fff700 1 --2- 172.21.15.16:0/2703246648 >> [v2:172.21.15.49:6833/2892536404,v1:172.21.15.49:6835/2892536404] conn(0x556d975ac020 0x556d975ac400 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=71be79755937f97 server_cookie=83e21891d1efe161 in_seq=0 out_seq=0 2023-10-16T12:11:15.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.791+0000 7fb096ffd700 1 -- 172.21.15.16:0/2703246648 <== mds.0 v2:172.21.15.49:6833/2892536404 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+330 (crc 0 0 0) 0x556d974aeb20 con 0x556d975ac020 2023-10-16T12:11:15.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.791+0000 7fb096ffd700 10 client.4797 handle_command_reply: tid=0 2023-10-16T12:11:15.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.791+0000 7fb0ca522b80 1 client.4797 shutdown 2023-10-16T12:11:15.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.792+0000 7fb0ca522b80 1 -- 172.21.15.16:0/2703246648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d9758b630 msgr2=0x556d9758a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:15.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.792+0000 7fb0ca522b80 1 --2- 172.21.15.16:0/2703246648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d9758b630 0x556d9758a130 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb0b0018040 tx=0x7fb0b0001250 comp rx=0 tx=0).stop 2023-10-16T12:11:15.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.792+0000 7fb0ca522b80 1 -- 172.21.15.16:0/2703246648 shutdown_connections 2023-10-16T12:11:15.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.792+0000 7fb0ca522b80 1 --2- 172.21.15.16:0/2703246648 >> [v2:172.21.15.49:6833/2892536404,v1:172.21.15.49:6835/2892536404] conn(0x556d975ac020 0x556d975ac400 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:15.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.792+0000 7fb0ca522b80 1 --2- 172.21.15.16:0/2703246648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556d9758b630 0x556d9758a130 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:15.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.792+0000 7fb0ca522b80 1 --2- 172.21.15.16:0/2703246648 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x556d97561010 0x556d97613820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:15.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.792+0000 7fb096ffd700 0 client.4797 ms_handle_reset on v2:172.21.15.49:6833/2892536404 2023-10-16T12:11:15.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.792+0000 7fb0ca522b80 1 --2- 172.21.15.16:0/2703246648 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556d97586220 0x556d97589bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:15.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.792+0000 7fb0ca522b80 1 -- 172.21.15.16:0/2703246648 >> 172.21.15.16:0/2703246648 conn(0x556d97490d80 msgr2=0x556d9758a9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:15.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.792+0000 7fb0ca522b80 1 -- 172.21.15.16:0/2703246648 shutdown_connections 2023-10-16T12:11:15.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.792+0000 7fb0ca522b80 1 -- 172.21.15.16:0/2703246648 wait complete. 2023-10-16T12:11:15.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.792+0000 7fb0ca522b80 20 client.4797 trim_cache size 0 max 16384 2023-10-16T12:11:15.816 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-10-16T12:11:15.817 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "5b28dfa1-b21f-474a-9b2e-9fb77af58254", 2023-10-16T12:11:15.817 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 0, 2023-10-16T12:11:15.817 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 4269, 2023-10-16T12:11:15.817 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2023-10-16T12:11:15.817 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2023-10-16T12:11:15.818 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2023-10-16T12:11:15.818 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 36.162122512000003, 2023-10-16T12:11:15.818 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 9, 2023-10-16T12:11:15.818 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 25, 2023-10-16T12:11:15.818 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 25, 2023-10-16T12:11:15.819 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 43.412764629999998 2023-10-16T12:11:15.819 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-10-16T12:11:15.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.792+0000 7fb0be338700 1 -- 172.21.15.16:0/999041856 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fb0a406af00 msgr2=0x7fb0a406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:15.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.792+0000 7fb0be338700 1 --2- 172.21.15.16:0/999041856 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fb0a406af00 0x7fb0a406d3c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb0ac009cd0 tx=0x7fb0ac008040 comp rx=0 tx=0).stop 2023-10-16T12:11:15.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.792+0000 7fb0be338700 1 -- 172.21.15.16:0/999041856 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0b815b630 msgr2=0x7fb0b81f17e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:15.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.792+0000 7fb0be338700 1 --2- 172.21.15.16:0/999041856 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0b815b630 0x7fb0b81f17e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fb0a8009b90 tx=0x7fb0a8004750 comp rx=0 tx=0).stop 2023-10-16T12:11:15.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.792+0000 7fb0be338700 1 -- 172.21.15.16:0/999041856 shutdown_connections 2023-10-16T12:11:15.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.792+0000 7fb0be338700 1 --2- 172.21.15.16:0/999041856 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fb0a406af00 0x7fb0a406d3c0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:15.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.792+0000 7fb0be338700 1 --2- 172.21.15.16:0/999041856 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb0b815ba10 0x7fb0b809ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:15.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.792+0000 7fb0be338700 1 --2- 172.21.15.16:0/999041856 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb0b80ce320 0x7fb0b81589c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:15.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.792+0000 7fb0be338700 1 --2- 172.21.15.16:0/999041856 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0b815b630 0x7fb0b81f17e0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:15.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.792+0000 7fb0be338700 1 -- 172.21.15.16:0/999041856 >> 172.21.15.16:0/999041856 conn(0x7fb0b80d14b0 msgr2=0x7fb0b80b4eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:15.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.792+0000 7fb0be338700 1 -- 172.21.15.16:0/999041856 shutdown_connections 2023-10-16T12:11:15.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:15.793+0000 7fb0be338700 1 -- 172.21.15.16:0/999041856 wait complete. 2023-10-16T12:11:15.823 INFO:teuthology.run_tasks:Running task kclient... 2023-10-16T12:11:15.837 INFO:tasks.kclient:Mounting kernel clients... 2023-10-16T12:11:15.837 INFO:tasks.kclient:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2023-10-16T12:11:15.837 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:15.838 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2023-10-16T12:11:15.855 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2023-10-16T12:11:15.856 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:15.856 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-10-16T12:11:15.930 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:15.930 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link delete ceph-brx 2023-10-16T12:11:16.047 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:16.047 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2023-10-16T12:11:16.065 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:16.066 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link delete ceph-brx 2023-10-16T12:11:16.135 INFO:teuthology.orchestra.run.smithi016.stderr:Cannot find device "ceph-brx" 2023-10-16T12:11:16.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-16T12:11:16.137 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:16.137 DEBUG:teuthology.orchestra.run.smithi049:> ip netns list 2023-10-16T12:11:16.159 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:16.159 DEBUG:teuthology.orchestra.run.smithi049:> sudo ip link delete ceph-brx 2023-10-16T12:11:16.235 INFO:teuthology.orchestra.run.smithi049.stderr:Cannot find device "ceph-brx" 2023-10-16T12:11:16.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-16T12:11:16.237 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:16.237 DEBUG:teuthology.orchestra.run.smithi049:> ip netns list 2023-10-16T12:11:16.297 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:16.297 DEBUG:teuthology.orchestra.run.smithi049:> sudo ip link delete ceph-brx 2023-10-16T12:11:16.370 INFO:teuthology.orchestra.run.smithi049.stderr:Cannot find device "ceph-brx" 2023-10-16T12:11:16.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-16T12:11:16.372 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2023-10-16T12:11:16.372 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-10-16T12:11:16.373 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-16T12:11:16.373 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T12:11:16.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:16.498+0000 7fcbf722a700 1 Processor -- start 2023-10-16T12:11:16.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:16.499+0000 7fcbf722a700 1 -- start start 2023-10-16T12:11:16.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:16.499+0000 7fcbf722a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcbf015db40 0x7fcbf00cb980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:16.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:16.499+0000 7fcbf722a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcbf006a7e0 0x7fcbf006ac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:16.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:16.499+0000 7fcbf722a700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbf00cedb0 con 0x7fcbf00ce160 2023-10-16T12:11:16.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:16.499+0000 7fcbf722a700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbf006b1a0 con 0x7fcbf015db40 2023-10-16T12:11:16.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:16.499+0000 7fcbf722a700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf015df20 con 0x7fcbf006a7e0 2023-10-16T12:11:16.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:16.499+0000 7fcbf4fc6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcbf015db40 0x7fcbf00cb980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:16.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:16.499+0000 7fcbf57c7700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fcbf00ce160 0x7fcbf00ce5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51658/0 (socket says 172.21.15.16:51658) 2023-10-16T12:11:16.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:16.499+0000 7fcbf57c7700 1 -- 172.21.15.16:0/3514922113 learned_addr learned my addr 172.21.15.16:0/3514922113 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:16.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:16.500+0000 7fcbeffff700 1 --2- 172.21.15.16:0/3514922113 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcbf006a7e0 0x7fcbf006ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:16.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:16.500+0000 7fcbf4fc6700 1 -- 172.21.15.16:0/3514922113 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcbf006a7e0 msgr2=0x7fcbf006ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:16.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:16.500+0000 7fcbf4fc6700 1 --2- 172.21.15.16:0/3514922113 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcbf006a7e0 0x7fcbf006ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:16.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:16.500+0000 7fcbf4fc6700 1 -- 172.21.15.16:0/3514922113 >> v1:172.21.15.16:6789/0 conn(0x7fcbf00ce160 legacy=0x7fcbf00ce5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:16.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:16.500+0000 7fcbf4fc6700 1 -- 172.21.15.16:0/3514922113 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbf00d12c0 con 0x7fcbf015db40 2023-10-16T12:11:16.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:16.500+0000 7fcbef7fe700 1 -- 172.21.15.16:0/3514922113 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4136629565 0 0) 0x7fcbf00cedb0 con 0x7fcbf00ce160 2023-10-16T12:11:16.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:16.500+0000 7fcbf4fc6700 1 --2- 172.21.15.16:0/3514922113 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbf015db40 0x7fcbf00cb980 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fcbe4009840 tx=0x7fcbe4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10210be012a160f2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:16.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:16.500+0000 7fcbef7fe700 1 -- 172.21.15.16:0/3514922113 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbe4011830 con 0x7fcbf015db40 2023-10-16T12:11:16.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:16.500+0000 7fcbef7fe700 1 -- 172.21.15.16:0/3514922113 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbe4011990 con 0x7fcbf015db40 2023-10-16T12:11:16.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:16.500+0000 7fcbef7fe700 1 -- 172.21.15.16:0/3514922113 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbe4011d20 con 0x7fcbf015db40 2023-10-16T12:11:16.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:16.501+0000 7fcbf722a700 1 -- 172.21.15.16:0/3514922113 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbf015db40 msgr2=0x7fcbf00cb980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:16.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:16.501+0000 7fcbf722a700 1 --2- 172.21.15.16:0/3514922113 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbf015db40 0x7fcbf00cb980 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fcbe4009840 tx=0x7fcbe4009b50 comp rx=0 tx=0).stop 2023-10-16T12:11:16.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:16.501+0000 7fcbf722a700 1 -- 172.21.15.16:0/3514922113 shutdown_connections 2023-10-16T12:11:16.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:16.501+0000 7fcbf722a700 1 --2- 172.21.15.16:0/3514922113 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbf015db40 0x7fcbf00cb980 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:16.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:16.501+0000 7fcbf722a700 1 --2- 172.21.15.16:0/3514922113 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcbf006a7e0 0x7fcbf006ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:16.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:16.501+0000 7fcbf722a700 1 -- 172.21.15.16:0/3514922113 >> 172.21.15.16:0/3514922113 conn(0x7fcbf00b1fd0 msgr2=0x7fcbf00b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:16.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:16.501+0000 7fcbf722a700 1 -- 172.21.15.16:0/3514922113 shutdown_connections 2023-10-16T12:11:16.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:16.501+0000 7fcbf722a700 1 -- 172.21.15.16:0/3514922113 wait complete. 2023-10-16T12:11:16.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:16.502+0000 7fcbf722a700 1 Processor -- start 2023-10-16T12:11:16.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:16.502+0000 7fcbf722a700 1 -- start start 2023-10-16T12:11:16.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:16.503+0000 7fcbf722a700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcbf006a7e0 0x7fcbf0153430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:16.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:16.503+0000 7fcbf722a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbf00ce160 0x7fcbf00c8890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:16.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:16.503+0000 7fcbf722a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcbf015db40 0x7fcbf00b3a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:16.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:16.503+0000 7fcbf722a700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf0153970 con 0x7fcbf00ce160 2023-10-16T12:11:16.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:16.503+0000 7fcbf722a700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf00c7510 con 0x7fcbf006a7e0 2023-10-16T12:11:16.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:16.503+0000 7fcbf722a700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf00c77f0 con 0x7fcbf015db40 2023-10-16T12:11:16.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:16.503+0000 7fcbeffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbf00ce160 0x7fcbf00c8890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:16.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:16.503+0000 7fcbeffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbf00ce160 0x7fcbf00c8890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34160/0 (socket says 172.21.15.16:34160) 2023-10-16T12:11:16.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:16.503+0000 7fcbeffff700 1 -- 172.21.15.16:0/2898527748 learned_addr learned my addr 172.21.15.16:0/2898527748 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:16.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:16.503+0000 7fcbf4fc6700 1 --2- 172.21.15.16:0/2898527748 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcbf006a7e0 0x7fcbf0153430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:16.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:16.503+0000 7fcbf57c7700 1 --2- 172.21.15.16:0/2898527748 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcbf015db40 0x7fcbf00b3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:16.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:16.503+0000 7fcbeffff700 1 -- 172.21.15.16:0/2898527748 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcbf015db40 msgr2=0x7fcbf00b3a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:16.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:16.503+0000 7fcbeffff700 1 --2- 172.21.15.16:0/2898527748 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcbf015db40 0x7fcbf00b3a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:16.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:16.503+0000 7fcbeffff700 1 -- 172.21.15.16:0/2898527748 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcbf006a7e0 msgr2=0x7fcbf0153430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:16.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:16.503+0000 7fcbeffff700 1 --2- 172.21.15.16:0/2898527748 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcbf006a7e0 0x7fcbf0153430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:16.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:16.503+0000 7fcbeffff700 1 -- 172.21.15.16:0/2898527748 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbe40094a0 con 0x7fcbf00ce160 2023-10-16T12:11:16.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:16.504+0000 7fcbeffff700 1 --2- 172.21.15.16:0/2898527748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbf00ce160 0x7fcbf00c8890 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fcbdc00e870 tx=0x7fcbdc00ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:16.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:16.504+0000 7fcbed7fa700 1 -- 172.21.15.16:0/2898527748 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbdc009590 con 0x7fcbf00ce160 2023-10-16T12:11:16.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:16.504+0000 7fcbed7fa700 1 -- 172.21.15.16:0/2898527748 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbdc00ee70 con 0x7fcbf00ce160 2023-10-16T12:11:16.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:16.504+0000 7fcbf722a700 1 -- 172.21.15.16:0/2898527748 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbf00c7ad0 con 0x7fcbf00ce160 2023-10-16T12:11:16.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:16.504+0000 7fcbed7fa700 1 -- 172.21.15.16:0/2898527748 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbdc00ca50 con 0x7fcbf00ce160 2023-10-16T12:11:16.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:16.504+0000 7fcbf722a700 1 -- 172.21.15.16:0/2898527748 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbf00c8090 con 0x7fcbf00ce160 2023-10-16T12:11:16.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:16.505+0000 7fcbf722a700 1 -- 172.21.15.16:0/2898527748 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbf004ebb0 con 0x7fcbf00ce160 2023-10-16T12:11:16.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:16.505+0000 7fcbed7fa700 1 -- 172.21.15.16:0/2898527748 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fcbdc00cbb0 con 0x7fcbf00ce160 2023-10-16T12:11:16.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:16.506+0000 7fcbed7fa700 1 --2- 172.21.15.16:0/2898527748 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fcbd406af00 0x7fcbd406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:16.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:16.506+0000 7fcbed7fa700 1 -- 172.21.15.16:0/2898527748 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fcbdc089840 con 0x7fcbf00ce160 2023-10-16T12:11:16.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:16.506+0000 7fcbf4fc6700 1 --2- 172.21.15.16:0/2898527748 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fcbd406af00 0x7fcbd406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:16.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:16.506+0000 7fcbf4fc6700 1 --2- 172.21.15.16:0/2898527748 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fcbd406af00 0x7fcbd406d3c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fcbe40090b0 tx=0x7fcbe4008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:16.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:16.510+0000 7fcbed7fa700 1 -- 172.21.15.16:0/2898527748 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcbdc054600 con 0x7fcbf00ce160 2023-10-16T12:11:16.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:16.760+0000 7fcbf722a700 1 -- 172.21.15.16:0/2898527748 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fcbf00b1db0 con 0x7fcbf00ce160 2023-10-16T12:11:16.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:16.760+0000 7fcbed7fa700 1 -- 172.21.15.16:0/2898527748 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) v1 ==== 53+0+88 (secure 0 0 0) 0x7fcbdc021070 con 0x7fcbf00ce160 2023-10-16T12:11:16.762 INFO:teuthology.orchestra.run.smithi016.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-10-16T12:11:16.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:16.762+0000 7fcbf722a700 1 -- 172.21.15.16:0/2898527748 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fcbd406af00 msgr2=0x7fcbd406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:16.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:16.762+0000 7fcbf722a700 1 --2- 172.21.15.16:0/2898527748 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fcbd406af00 0x7fcbd406d3c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fcbe40090b0 tx=0x7fcbe4008030 comp rx=0 tx=0).stop 2023-10-16T12:11:16.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:16.762+0000 7fcbf722a700 1 -- 172.21.15.16:0/2898527748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbf00ce160 msgr2=0x7fcbf00c8890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:16.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:16.762+0000 7fcbf722a700 1 --2- 172.21.15.16:0/2898527748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbf00ce160 0x7fcbf00c8890 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fcbdc00e870 tx=0x7fcbdc00ec30 comp rx=0 tx=0).stop 2023-10-16T12:11:16.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:16.762+0000 7fcbf722a700 1 -- 172.21.15.16:0/2898527748 shutdown_connections 2023-10-16T12:11:16.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:16.762+0000 7fcbf722a700 1 --2- 172.21.15.16:0/2898527748 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fcbd406af00 0x7fcbd406d3c0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:16.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:16.762+0000 7fcbf722a700 1 --2- 172.21.15.16:0/2898527748 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcbf015db40 0x7fcbf00b3a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:16.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:16.762+0000 7fcbf722a700 1 --2- 172.21.15.16:0/2898527748 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcbf006a7e0 0x7fcbf0153430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:16.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:16.762+0000 7fcbf722a700 1 --2- 172.21.15.16:0/2898527748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbf00ce160 0x7fcbf00c8890 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:16.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:16.762+0000 7fcbf722a700 1 -- 172.21.15.16:0/2898527748 >> 172.21.15.16:0/2898527748 conn(0x7fcbf00b1fd0 msgr2=0x7fcbf00d09d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:16.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:16.762+0000 7fcbf722a700 1 -- 172.21.15.16:0/2898527748 shutdown_connections 2023-10-16T12:11:16.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:16.762+0000 7fcbf722a700 1 -- 172.21.15.16:0/2898527748 wait complete. 2023-10-16T12:11:16.776 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-10-16T12:11:16.776 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T12:11:16.777 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi016.front.sepia.ceph.com 2023-10-16T12:11:16.777 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-16T12:11:16.777 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-16T12:11:16.778 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-10-16T12:11:16.778 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-16T12:11:16.778 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T12:11:16.778 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T12:11:16.779 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:16.779 DEBUG:teuthology.orchestra.run.smithi016:> ip addr 2023-10-16T12:11:16.796 INFO:teuthology.orchestra.run.smithi016.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T12:11:16.797 INFO:teuthology.orchestra.run.smithi016.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T12:11:16.797 INFO:teuthology.orchestra.run.smithi016.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T12:11:16.797 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:11:16.797 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 ::1/128 scope host 2023-10-16T12:11:16.797 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:11:16.798 INFO:teuthology.orchestra.run.smithi016.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T12:11:16.798 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e0 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:11:16.798 INFO:teuthology.orchestra.run.smithi016.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T12:11:16.798 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e1 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:11:16.798 INFO:teuthology.orchestra.run.smithi016.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T12:11:16.799 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:64 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:11:16.799 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.21.15.16/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-16T12:11:16.799 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft 42259sec preferred_lft 42259sec 2023-10-16T12:11:16.799 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::ec4:7aff:febd:1564/64 scope link 2023-10-16T12:11:16.799 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:11:16.800 INFO:teuthology.orchestra.run.smithi016.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T12:11:16.800 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:65 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:11:16.800 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-10-16T12:11:16.800 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:16.801 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec bash -c ' 2023-10-16T12:11:16.801 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2023-10-16T12:11:16.801 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link add name ceph-brx type bridge 2023-10-16T12:11:16.802 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip addr flush dev ceph-brx 2023-10-16T12:11:16.802 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set ceph-brx up 2023-10-16T12:11:16.802 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-10-16T12:11:16.802 DEBUG:teuthology.orchestra.run.smithi016:> ') 2023-10-16T12:11:16.915 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:16.915 DEBUG:teuthology.orchestra.run.smithi016:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T12:11:16.942 INFO:teuthology.orchestra.run.smithi016.stdout:1 2023-10-16T12:11:16.943 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:16.943 DEBUG:teuthology.orchestra.run.smithi016:> route 2023-10-16T12:11:17.002 INFO:teuthology.orchestra.run.smithi016.stdout:Kernel IP routing table 2023-10-16T12:11:17.003 INFO:teuthology.orchestra.run.smithi016.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T12:11:17.003 INFO:teuthology.orchestra.run.smithi016.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-16T12:11:17.003 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-16T12:11:17.003 INFO:teuthology.orchestra.run.smithi016.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T12:11:17.004 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:17.004 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec bash -c ' 2023-10-16T12:11:17.004 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2023-10-16T12:11:17.004 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-16T12:11:17.005 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-16T12:11:17.005 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-16T12:11:17.005 DEBUG:teuthology.orchestra.run.smithi016:> ') 2023-10-16T12:11:17.109 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:17.110 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2023-10-16T12:11:17.127 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:17.127 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list-id 2023-10-16T12:11:17.184 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:17.184 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec bash -c ' 2023-10-16T12:11:17.184 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2023-10-16T12:11:17.184 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T12:11:17.185 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2023-10-16T12:11:17.185 DEBUG:teuthology.orchestra.run.smithi016:> ') 2023-10-16T12:11:17.279 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2023-10-16T12:11:17.279 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:17.279 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec bash -c ' 2023-10-16T12:11:17.280 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2023-10-16T12:11:17.280 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2023-10-16T12:11:17.280 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-10-16T12:11:17.280 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2023-10-16T12:11:17.281 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2023-10-16T12:11:17.281 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2023-10-16T12:11:17.281 DEBUG:teuthology.orchestra.run.smithi016:> ') 2023-10-16T12:11:17.492 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:17.492 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec bash -c ' 2023-10-16T12:11:17.493 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2023-10-16T12:11:17.493 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set brx.0 up 2023-10-16T12:11:17.493 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set dev brx.0 master ceph-brx 2023-10-16T12:11:17.494 DEBUG:teuthology.orchestra.run.smithi016:> ') 2023-10-16T12:11:17.608 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T12:11:17.608 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-16T12:11:17.631 INFO:teuthology.orchestra.run.smithi016.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-16T12:11:17.632 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T12:11:17.632 DEBUG:teuthology.orchestra.run.smithi016:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-16T12:11:17.688 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-16T12:11:17.688 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:17.689 DEBUG:teuthology.orchestra.run.smithi016:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T12:11:17.775 INFO:teuthology.orchestra.run.smithi016.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T12:11:17.775 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T12:11:17.775 INFO:teuthology.orchestra.run.smithi016.stdout:invalid new device string format 2023-10-16T12:11:17.895 INFO:teuthology.orchestra.run.smithi016.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T12:11:17.895 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T12:11:17.895 INFO:teuthology.orchestra.run.smithi016.stdout:invalid new device string format 2023-10-16T12:11:17.896 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: resolved to: "172.21.15.16:3300,172.21.15.49:3300,172.21.15.49:3301" 2023-10-16T12:11:17.896 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T12:11:17.969 INFO:teuthology.orchestra.run.smithi016.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T12:11:17.970 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T12:11:17.970 INFO:teuthology.orchestra.run.smithi016.stdout:invalid new device string format 2023-10-16T12:11:17.970 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: resolved to: "172.21.15.16:3300,172.21.15.49:3300,172.21.15.49:3301" 2023-10-16T12:11:17.970 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T12:11:17.970 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: resolved to: "172.21.15.16:6789,172.21.15.49:6789,172.21.15.49:6790" 2023-10-16T12:11:17.971 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.16:6789,172.21.15.49:6789,172.21.15.49:6790:/ 2023-10-16T12:11:17.971 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=5b28dfa1-b21f-474a-9b2e-9fb77af58254" will pass to kernel 2023-10-16T12:11:17.972 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-16T12:11:17.973 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:17.973 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-16T12:11:18.000 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:18.001 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T12:11:18.073 INFO:teuthology.orchestra.run.smithi016.stdout:client4756 2023-10-16T12:11:18.074 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:11:18.074 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:11:18.131 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:18.132 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T12:11:18.204 INFO:teuthology.orchestra.run.smithi016.stdout:client4756 2023-10-16T12:11:18.204 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:18.205 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T12:11:18.277 INFO:teuthology.orchestra.run.smithi016.stdout:client4756 2023-10-16T12:11:18.278 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:18.278 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/5b28dfa1-b21f-474a-9b2e-9fb77af58254.client4756/status') 2023-10-16T12:11:18.359 INFO:teuthology.orchestra.run.smithi016.stdout:instance: client.4756 (3)172.21.15.16:0/2030211237 2023-10-16T12:11:18.359 INFO:teuthology.orchestra.run.smithi016.stdout:blocklisted: false 2023-10-16T12:11:18.360 INFO:teuthology.orchestra.run.smithi016.stderr:0+1 records in 2023-10-16T12:11:18.360 INFO:teuthology.orchestra.run.smithi016.stderr:0+1 records out 2023-10-16T12:11:18.361 INFO:teuthology.orchestra.run.smithi016.stderr:70 bytes copied, 2.3769e-05 s, 2.9 MB/s 2023-10-16T12:11:18.361 INFO:tasks.kclient:client.2 config is {'syntax': 'v1'} 2023-10-16T12:11:18.361 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-10-16T12:11:18.362 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2023-10-16T12:11:18.362 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T12:11:18.474 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:11:18.477 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:11:18.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:18.524+0000 7f3f911f2700 1 Processor -- start 2023-10-16T12:11:18.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:18.524+0000 7f3f911f2700 1 -- start start 2023-10-16T12:11:18.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:18.525+0000 7f3f911f2700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3f8c15b280 0x7f3f8c1613e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:18.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:18.525+0000 7f3f911f2700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3f8c0cc8a0 0x7f3f8c15e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:18.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:18.525+0000 7f3f911f2700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f8c161c30 con 0x7f3f8c06a7e0 2023-10-16T12:11:18.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:18.525+0000 7f3f911f2700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f8c161f10 con 0x7f3f8c0cc8a0 2023-10-16T12:11:18.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:18.525+0000 7f3f911f2700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f8c1621f0 con 0x7f3f8c15b280 2023-10-16T12:11:18.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:18.525+0000 7f3f8a59c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3f8c0cc8a0 0x7f3f8c15e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:18.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:18.525+0000 7f3f8a59c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3f8c0cc8a0 0x7f3f8c15e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34170/0 (socket says 172.21.15.16:34170) 2023-10-16T12:11:18.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:18.525+0000 7f3f8a59c700 1 -- 172.21.15.16:0/4167172738 learned_addr learned my addr 172.21.15.16:0/4167172738 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:18.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:18.525+0000 7f3f8ad9d700 1 --2- 172.21.15.16:0/4167172738 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3f8c15b280 0x7f3f8c1613e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:18.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:18.526+0000 7f3f8a59c700 1 -- 172.21.15.16:0/4167172738 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3f8c15b280 msgr2=0x7f3f8c1613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:18.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:18.526+0000 7f3f8a59c700 1 --2- 172.21.15.16:0/4167172738 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3f8c15b280 0x7f3f8c1613e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:18.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:18.526+0000 7f3f8a59c700 1 -- 172.21.15.16:0/4167172738 >> v1:172.21.15.49:6789/0 conn(0x7f3f8c06a7e0 legacy=0x7f3f8c0d02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:18.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:18.526+0000 7f3f8a59c700 1 -- 172.21.15.16:0/4167172738 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f8c15df00 con 0x7f3f8c0cc8a0 2023-10-16T12:11:18.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:18.526+0000 7f3f8a59c700 1 --2- 172.21.15.16:0/4167172738 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f8c0cc8a0 0x7f3f8c15e700 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f3f7c0098f0 tx=0x7f3f7c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c70700f71d88090a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:18.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:18.526+0000 7f3f89d9b700 1 -- 172.21.15.16:0/4167172738 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f7c00b8a0 con 0x7f3f8c0cc8a0 2023-10-16T12:11:18.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:18.526+0000 7f3f89d9b700 1 -- 172.21.15.16:0/4167172738 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f7c00ba00 con 0x7f3f8c0cc8a0 2023-10-16T12:11:18.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:18.526+0000 7f3f89d9b700 1 -- 172.21.15.16:0/4167172738 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f7c00bd90 con 0x7f3f8c0cc8a0 2023-10-16T12:11:18.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:18.526+0000 7f3f911f2700 1 -- 172.21.15.16:0/4167172738 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f8c0cc8a0 msgr2=0x7f3f8c15e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:18.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:18.526+0000 7f3f911f2700 1 --2- 172.21.15.16:0/4167172738 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f8c0cc8a0 0x7f3f8c15e700 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f3f7c0098f0 tx=0x7f3f7c009c00 comp rx=0 tx=0).stop 2023-10-16T12:11:18.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:18.526+0000 7f3f911f2700 1 -- 172.21.15.16:0/4167172738 shutdown_connections 2023-10-16T12:11:18.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:18.526+0000 7f3f911f2700 1 --2- 172.21.15.16:0/4167172738 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3f8c15b280 0x7f3f8c1613e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:18.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:18.526+0000 7f3f911f2700 1 --2- 172.21.15.16:0/4167172738 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f8c0cc8a0 0x7f3f8c15e700 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:18.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:18.526+0000 7f3f911f2700 1 -- 172.21.15.16:0/4167172738 >> 172.21.15.16:0/4167172738 conn(0x7f3f8c0b1fd0 msgr2=0x7f3f8c0b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:18.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:18.527+0000 7f3f911f2700 1 -- 172.21.15.16:0/4167172738 shutdown_connections 2023-10-16T12:11:18.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:18.527+0000 7f3f911f2700 1 -- 172.21.15.16:0/4167172738 wait complete. 2023-10-16T12:11:18.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:18.528+0000 7f3f911f2700 1 Processor -- start 2023-10-16T12:11:18.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:18.528+0000 7f3f911f2700 1 -- start start 2023-10-16T12:11:18.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:18.529+0000 7f3f911f2700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3f8c06a7e0 0x7f3f8c0c4380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:18.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:18.529+0000 7f3f911f2700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3f8c0cc8a0 0x7f3f8c0c6960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:18.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:18.529+0000 7f3f911f2700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f8c15b280 0x7f3f8c15b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:18.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:18.529+0000 7f3f911f2700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f8c145120 con 0x7f3f8c15b280 2023-10-16T12:11:18.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:18.529+0000 7f3f911f2700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f8c0c7db0 con 0x7f3f8c0cc8a0 2023-10-16T12:11:18.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:18.529+0000 7f3f911f2700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f8c0c7510 con 0x7f3f8c06a7e0 2023-10-16T12:11:18.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:18.529+0000 7f3f8ad9d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3f8c06a7e0 0x7f3f8c0c4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:18.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:18.529+0000 7f3f8b59e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f8c15b280 0x7f3f8c15b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:18.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:18.529+0000 7f3f8a59c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3f8c0cc8a0 0x7f3f8c0c6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:18.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:18.529+0000 7f3f8ad9d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3f8c06a7e0 0x7f3f8c0c4380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:50666/0 (socket says 172.21.15.16:50666) 2023-10-16T12:11:18.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:18.529+0000 7f3f8ad9d700 1 -- 172.21.15.16:0/2302172949 learned_addr learned my addr 172.21.15.16:0/2302172949 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:18.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:18.529+0000 7f3f8b59e700 1 -- 172.21.15.16:0/2302172949 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3f8c06a7e0 msgr2=0x7f3f8c0c4380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:18.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:18.529+0000 7f3f8b59e700 1 --2- 172.21.15.16:0/2302172949 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3f8c06a7e0 0x7f3f8c0c4380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:18.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:18.529+0000 7f3f8b59e700 1 -- 172.21.15.16:0/2302172949 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3f8c0cc8a0 msgr2=0x7f3f8c0c6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:18.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:18.529+0000 7f3f8b59e700 1 --2- 172.21.15.16:0/2302172949 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3f8c0cc8a0 0x7f3f8c0c6960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:18.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:18.529+0000 7f3f8b59e700 1 -- 172.21.15.16:0/2302172949 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f7c009550 con 0x7f3f8c15b280 2023-10-16T12:11:18.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:18.529+0000 7f3f8ad9d700 1 --2- 172.21.15.16:0/2302172949 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3f8c06a7e0 0x7f3f8c0c4380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:11:18.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:18.529+0000 7f3f8b59e700 1 --2- 172.21.15.16:0/2302172949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f8c15b280 0x7f3f8c15b660 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f3f8400cb90 tx=0x7f3f8400cea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:18.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:18.530+0000 7f3f737fe700 1 -- 172.21.15.16:0/2302172949 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f84003c00 con 0x7f3f8c15b280 2023-10-16T12:11:18.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:18.530+0000 7f3f737fe700 1 -- 172.21.15.16:0/2302172949 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f84003d60 con 0x7f3f8c15b280 2023-10-16T12:11:18.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:18.530+0000 7f3f911f2700 1 -- 172.21.15.16:0/2302172949 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f8c0c77f0 con 0x7f3f8c15b280 2023-10-16T12:11:18.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:18.530+0000 7f3f737fe700 1 -- 172.21.15.16:0/2302172949 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f8400aa80 con 0x7f3f8c15b280 2023-10-16T12:11:18.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:18.530+0000 7f3f911f2700 1 -- 172.21.15.16:0/2302172949 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f8c0c4ad0 con 0x7f3f8c15b280 2023-10-16T12:11:18.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:18.531+0000 7f3f911f2700 1 -- 172.21.15.16:0/2302172949 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f8c0c4cc0 con 0x7f3f8c15b280 2023-10-16T12:11:18.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:18.531+0000 7f3f737fe700 1 -- 172.21.15.16:0/2302172949 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f3f8400abe0 con 0x7f3f8c15b280 2023-10-16T12:11:18.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:18.531+0000 7f3f737fe700 1 --2- 172.21.15.16:0/2302172949 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f3f7806af00 0x7f3f7806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:18.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:18.532+0000 7f3f737fe700 1 -- 172.21.15.16:0/2302172949 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f3f840897f0 con 0x7f3f8c15b280 2023-10-16T12:11:18.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:18.532+0000 7f3f8ad9d700 1 --2- 172.21.15.16:0/2302172949 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f3f7806af00 0x7f3f7806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:18.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:18.532+0000 7f3f8ad9d700 1 --2- 172.21.15.16:0/2302172949 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f3f7806af00 0x7f3f7806d3c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f3f80007ee0 tx=0x7f3f80008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:18.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:18.536+0000 7f3f737fe700 1 -- 172.21.15.16:0/2302172949 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3f840545b0 con 0x7f3f8c15b280 2023-10-16T12:11:18.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:18.791+0000 7f3f911f2700 1 -- 172.21.15.16:0/2302172949 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f3f8c04ebb0 con 0x7f3f8c15b280 2023-10-16T12:11:18.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:18.792+0000 7f3f737fe700 1 -- 172.21.15.16:0/2302172949 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) v1 ==== 53+0+88 (secure 0 0 0) 0x7f3f84057bd0 con 0x7f3f8c15b280 2023-10-16T12:11:18.794 INFO:teuthology.orchestra.run.smithi016.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-10-16T12:11:18.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:18.794+0000 7f3f911f2700 1 -- 172.21.15.16:0/2302172949 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f3f7806af00 msgr2=0x7f3f7806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:18.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:18.794+0000 7f3f911f2700 1 --2- 172.21.15.16:0/2302172949 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f3f7806af00 0x7f3f7806d3c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f3f80007ee0 tx=0x7f3f80008040 comp rx=0 tx=0).stop 2023-10-16T12:11:18.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:18.794+0000 7f3f911f2700 1 -- 172.21.15.16:0/2302172949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f8c15b280 msgr2=0x7f3f8c15b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:18.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:18.794+0000 7f3f911f2700 1 --2- 172.21.15.16:0/2302172949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f8c15b280 0x7f3f8c15b660 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f3f8400cb90 tx=0x7f3f8400cea0 comp rx=0 tx=0).stop 2023-10-16T12:11:18.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:18.794+0000 7f3f911f2700 1 -- 172.21.15.16:0/2302172949 shutdown_connections 2023-10-16T12:11:18.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:18.794+0000 7f3f911f2700 1 --2- 172.21.15.16:0/2302172949 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f3f7806af00 0x7f3f7806d3c0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:18.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:18.794+0000 7f3f911f2700 1 --2- 172.21.15.16:0/2302172949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f8c15b280 0x7f3f8c15b660 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:18.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:18.794+0000 7f3f911f2700 1 --2- 172.21.15.16:0/2302172949 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3f8c06a7e0 0x7f3f8c0c4380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:18.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:18.794+0000 7f3f911f2700 1 --2- 172.21.15.16:0/2302172949 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3f8c0cc8a0 0x7f3f8c0c6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:18.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:18.794+0000 7f3f911f2700 1 -- 172.21.15.16:0/2302172949 >> 172.21.15.16:0/2302172949 conn(0x7f3f8c0b1fd0 msgr2=0x7f3f8c0724e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:18.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:18.794+0000 7f3f911f2700 1 -- 172.21.15.16:0/2302172949 shutdown_connections 2023-10-16T12:11:18.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:18.794+0000 7f3f911f2700 1 -- 172.21.15.16:0/2302172949 wait complete. 2023-10-16T12:11:18.808 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-10-16T12:11:18.808 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T12:11:18.808 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi016.front.sepia.ceph.com 2023-10-16T12:11:18.809 INFO:tasks.cephfs.mount:self.client.name = client.2 2023-10-16T12:11:18.809 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2023-10-16T12:11:18.809 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-10-16T12:11:18.809 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-16T12:11:18.810 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T12:11:18.810 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2023-10-16T12:11:18.810 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:18.811 DEBUG:teuthology.orchestra.run.smithi016:> ip addr 2023-10-16T12:11:18.828 INFO:teuthology.orchestra.run.smithi016.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T12:11:18.828 INFO:teuthology.orchestra.run.smithi016.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T12:11:18.828 INFO:teuthology.orchestra.run.smithi016.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T12:11:18.829 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:11:18.829 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 ::1/128 scope host 2023-10-16T12:11:18.829 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:11:18.829 INFO:teuthology.orchestra.run.smithi016.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T12:11:18.829 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e0 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:11:18.829 INFO:teuthology.orchestra.run.smithi016.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T12:11:18.830 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e1 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:11:18.830 INFO:teuthology.orchestra.run.smithi016.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T12:11:18.830 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:64 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:11:18.830 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.21.15.16/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-16T12:11:18.830 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft 42257sec preferred_lft 42257sec 2023-10-16T12:11:18.831 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::ec4:7aff:febd:1564/64 scope link 2023-10-16T12:11:18.831 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:11:18.831 INFO:teuthology.orchestra.run.smithi016.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T12:11:18.831 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:65 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:11:18.832 INFO:teuthology.orchestra.run.smithi016.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T12:11:18.832 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 6e:62:9d:a4:43:23 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:11:18.832 INFO:teuthology.orchestra.run.smithi016.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-16T12:11:18.832 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:11:18.832 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::cfc:c5ff:fe52:a786/64 scope link 2023-10-16T12:11:18.833 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:11:18.833 INFO:teuthology.orchestra.run.smithi016.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T12:11:18.833 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 6e:62:9d:a4:43:23 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T12:11:18.833 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::6c62:9dff:fea4:4323/64 scope link tentative 2023-10-16T12:11:18.833 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:11:18.834 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:18.834 DEBUG:teuthology.orchestra.run.smithi016:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T12:11:18.894 INFO:teuthology.orchestra.run.smithi016.stdout:1 2023-10-16T12:11:18.896 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:18.896 DEBUG:teuthology.orchestra.run.smithi016:> route 2023-10-16T12:11:18.954 INFO:teuthology.orchestra.run.smithi016.stdout:Kernel IP routing table 2023-10-16T12:11:18.955 INFO:teuthology.orchestra.run.smithi016.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T12:11:18.955 INFO:teuthology.orchestra.run.smithi016.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-16T12:11:18.955 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-16T12:11:18.955 INFO:teuthology.orchestra.run.smithi016.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T12:11:18.956 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:18.956 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec bash -c ' 2023-10-16T12:11:18.956 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2023-10-16T12:11:18.957 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-16T12:11:18.957 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-16T12:11:18.957 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-16T12:11:18.957 DEBUG:teuthology.orchestra.run.smithi016:> ') 2023-10-16T12:11:19.060 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:19.061 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2023-10-16T12:11:19.078 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-16T12:11:19.079 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:19.079 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list-id 2023-10-16T12:11:19.135 INFO:teuthology.orchestra.run.smithi016.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2023-10-16T12:11:19.136 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:19.136 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec bash -c ' 2023-10-16T12:11:19.137 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2023-10-16T12:11:19.137 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-16T12:11:19.137 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2023-10-16T12:11:19.137 DEBUG:teuthology.orchestra.run.smithi016:> ') 2023-10-16T12:11:19.231 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:19.232 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2023-10-16T12:11:19.269 INFO:teuthology.orchestra.run.smithi016.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T12:11:19.270 INFO:teuthology.orchestra.run.smithi016.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T12:11:19.270 INFO:teuthology.orchestra.run.smithi016.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T12:11:19.270 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:11:19.270 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 ::1/128 scope host 2023-10-16T12:11:19.270 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:11:19.271 INFO:teuthology.orchestra.run.smithi016.stdout:2: veth0@if9: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T12:11:19.271 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 16:c5:54:61:a6:9d brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-10-16T12:11:19.271 INFO:teuthology.orchestra.run.smithi016.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-10-16T12:11:19.271 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:11:19.271 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::14c5:54ff:fe61:a69d/64 scope link 2023-10-16T12:11:19.271 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:11:19.281 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:19.281 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2023-10-16T12:11:19.364 INFO:teuthology.orchestra.run.smithi016.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T12:11:19.365 INFO:teuthology.orchestra.run.smithi016.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T12:11:19.365 INFO:teuthology.orchestra.run.smithi016.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T12:11:19.365 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:11:19.365 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 ::1/128 scope host 2023-10-16T12:11:19.365 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:11:19.366 INFO:teuthology.orchestra.run.smithi016.stdout:2: veth0@if9: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T12:11:19.366 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 16:c5:54:61:a6:9d brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-10-16T12:11:19.366 INFO:teuthology.orchestra.run.smithi016.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-10-16T12:11:19.366 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:11:19.366 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::14c5:54ff:fe61:a69d/64 scope link 2023-10-16T12:11:19.367 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:11:19.380 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2023-10-16T12:11:19.380 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:19.380 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec bash -c ' 2023-10-16T12:11:19.381 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2023-10-16T12:11:19.381 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2023-10-16T12:11:19.381 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2023-10-16T12:11:19.382 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2023-10-16T12:11:19.382 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2023-10-16T12:11:19.382 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2023-10-16T12:11:19.382 DEBUG:teuthology.orchestra.run.smithi016:> ') 2023-10-16T12:11:19.606 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:19.606 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec bash -c ' 2023-10-16T12:11:19.606 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2023-10-16T12:11:19.607 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set brx.1 up 2023-10-16T12:11:19.607 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set dev brx.1 master ceph-brx 2023-10-16T12:11:19.607 DEBUG:teuthology.orchestra.run.smithi016:> ') 2023-10-16T12:11:19.740 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T12:11:19.741 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2023-10-16T12:11:19.766 INFO:teuthology.orchestra.run.smithi016.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2023-10-16T12:11:19.767 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T12:11:19.767 DEBUG:teuthology.orchestra.run.smithi016:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2023-10-16T12:11:19.823 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-16T12:11:19.823 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:19.824 DEBUG:teuthology.orchestra.run.smithi016:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.2 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2023-10-16T12:11:19.910 INFO:teuthology.orchestra.run.smithi016.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2023-10-16T12:11:19.910 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs". 2023-10-16T12:11:19.910 INFO:teuthology.orchestra.run.smithi016.stdout:invalid new device string format 2023-10-16T12:11:19.918 INFO:teuthology.orchestra.run.smithi016.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2023-10-16T12:11:19.919 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs". 2023-10-16T12:11:19.919 INFO:teuthology.orchestra.run.smithi016.stdout:invalid new device string format 2023-10-16T12:11:19.919 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: resolved to: "172.21.15.16:3300,172.21.15.49:3300,172.21.15.49:3301" 2023-10-16T12:11:19.919 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T12:11:19.993 INFO:teuthology.orchestra.run.smithi016.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2023-10-16T12:11:19.993 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs". 2023-10-16T12:11:19.993 INFO:teuthology.orchestra.run.smithi016.stdout:invalid new device string format 2023-10-16T12:11:19.993 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: resolved to: "172.21.15.16:3300,172.21.15.49:3300,172.21.15.49:3301" 2023-10-16T12:11:19.994 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T12:11:19.994 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: resolved to: "172.21.15.16:6789,172.21.15.49:6789,172.21.15.49:6790" 2023-10-16T12:11:19.994 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.16:6789,172.21.15.49:6789,172.21.15.49:6790:/ 2023-10-16T12:11:19.994 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs,key=2,fsid=5b28dfa1-b21f-474a-9b2e-9fb77af58254" will pass to kernel 2023-10-16T12:11:19.996 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-16T12:11:19.996 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:19.996 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2023-10-16T12:11:20.023 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:20.024 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T12:11:20.096 INFO:teuthology.orchestra.run.smithi016.stdout:client4619 2023-10-16T12:11:20.096 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:11:20.097 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:11:20.154 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:20.154 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T12:11:20.227 INFO:teuthology.orchestra.run.smithi016.stdout:client4619 2023-10-16T12:11:20.228 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:20.228 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T12:11:20.300 INFO:teuthology.orchestra.run.smithi016.stdout:client4619 2023-10-16T12:11:20.301 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:20.301 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/5b28dfa1-b21f-474a-9b2e-9fb77af58254.client4619/status') 2023-10-16T12:11:20.381 INFO:teuthology.orchestra.run.smithi016.stdout:instance: client.4619 (3)172.21.15.16:0/3765036122 2023-10-16T12:11:20.382 INFO:teuthology.orchestra.run.smithi016.stdout:blocklisted: false 2023-10-16T12:11:20.382 INFO:teuthology.orchestra.run.smithi016.stderr:0+1 records in 2023-10-16T12:11:20.382 INFO:teuthology.orchestra.run.smithi016.stderr:0+1 records out 2023-10-16T12:11:20.383 INFO:teuthology.orchestra.run.smithi016.stderr:70 bytes copied, 2.3612e-05 s, 3.0 MB/s 2023-10-16T12:11:20.383 INFO:tasks.kclient:client.1 config is {'syntax': 'v1'} 2023-10-16T12:11:20.384 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-10-16T12:11:20.384 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-10-16T12:11:20.384 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T12:11:20.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.547+0000 7f558f2ed700 1 Processor -- start 2023-10-16T12:11:20.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.547+0000 7f558f2ed700 1 -- start start 2023-10-16T12:11:20.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.548+0000 7f558f2ed700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f558806aef0 0x7f55880b3480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:20.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.548+0000 7f558f2ed700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f55880b2be0 0x7f55880b39c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:20.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.548+0000 7f558f2ed700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55880d11c0 con 0x7f558815b860 2023-10-16T12:11:20.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.548+0000 7f558f2ed700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55880d03d0 con 0x7f558806aef0 2023-10-16T12:11:20.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.548+0000 7f558f2ed700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55880cd360 con 0x7f55880b2be0 2023-10-16T12:11:20.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.548+0000 7f558d089700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f558815b860 0x7f558806a7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51694/0 (socket says 172.21.15.16:51694) 2023-10-16T12:11:20.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.548+0000 7f558d089700 1 -- 172.21.15.16:0/1797795819 learned_addr learned my addr 172.21.15.16:0/1797795819 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:20.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.548+0000 7f558d88a700 1 --2- 172.21.15.16:0/1797795819 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f55880b2be0 0x7f55880b39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:20.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.548+0000 7f557ffff700 1 -- 172.21.15.16:0/1797795819 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3152461482 0 0) 0x7f55880d11c0 con 0x7f558815b860 2023-10-16T12:11:20.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.548+0000 7f557ffff700 1 -- 172.21.15.16:0/1797795819 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55740035b0 con 0x7f558815b860 2023-10-16T12:11:20.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.548+0000 7f558c888700 1 --2- 172.21.15.16:0/1797795819 >> v2:172.21.15.49:3300/0 conn(0x7f558806aef0 0x7f55880b3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:20.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.549+0000 7f557ffff700 1 -- 172.21.15.16:0/1797795819 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 372323279 0 0) 0x7f55740035b0 con 0x7f558815b860 2023-10-16T12:11:20.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.549+0000 7f557ffff700 1 -- 172.21.15.16:0/1797795819 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f55880b2be0 msgr2=0x7f55880b39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:20.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.549+0000 7f557ffff700 1 --2- 172.21.15.16:0/1797795819 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f55880b2be0 0x7f55880b39c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:20.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.549+0000 7f557ffff700 1 -- 172.21.15.16:0/1797795819 >> v2:172.21.15.49:3300/0 conn(0x7f558806aef0 msgr2=0x7f55880b3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:20.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.549+0000 7f557ffff700 1 --2- 172.21.15.16:0/1797795819 >> v2:172.21.15.49:3300/0 conn(0x7f558806aef0 0x7f55880b3480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:20.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.549+0000 7f557ffff700 1 -- 172.21.15.16:0/1797795819 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55880d05b0 con 0x7f558815b860 2023-10-16T12:11:20.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.549+0000 7f557ffff700 1 -- 172.21.15.16:0/1797795819 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f5584001670 con 0x7f558815b860 2023-10-16T12:11:20.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.549+0000 7f557ffff700 1 -- 172.21.15.16:0/1797795819 >> v1:172.21.15.16:6789/0 conn(0x7f558815b860 legacy=0x7f558806a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:20.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.549+0000 7f557ffff700 1 --2- 172.21.15.16:0/1797795819 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5574003a00 0x7f5574003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:20.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.549+0000 7f557ffff700 1 --2- 172.21.15.16:0/1797795819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5574004850 0x7f5574004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:20.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.549+0000 7f557ffff700 1 --2- 172.21.15.16:0/1797795819 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f55880b2be0 0x7f5574005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:20.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.549+0000 7f557ffff700 1 -- 172.21.15.16:0/1797795819 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55880d03d0 con 0x7f5574004850 2023-10-16T12:11:20.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.549+0000 7f557ffff700 1 -- 172.21.15.16:0/1797795819 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55880cd360 con 0x7f5574003a00 2023-10-16T12:11:20.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.549+0000 7f557ffff700 1 -- 172.21.15.16:0/1797795819 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f557400ba10 con 0x7f55880b2be0 2023-10-16T12:11:20.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.549+0000 7f558c888700 1 --2- 172.21.15.16:0/1797795819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5574004850 0x7f5574004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:20.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.549+0000 7f558d089700 1 --2- 172.21.15.16:0/1797795819 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5574003a00 0x7f5574003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:20.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.549+0000 7f558d88a700 1 --2- 172.21.15.16:0/1797795819 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f55880b2be0 0x7f5574005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:20.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.549+0000 7f558c888700 1 -- 172.21.15.16:0/1797795819 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f55880b2be0 msgr2=0x7f5574005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:20.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.549+0000 7f558c888700 1 --2- 172.21.15.16:0/1797795819 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f55880b2be0 0x7f5574005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:20.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.549+0000 7f558c888700 1 -- 172.21.15.16:0/1797795819 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5574003a00 msgr2=0x7f5574003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:20.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.549+0000 7f558c888700 1 --2- 172.21.15.16:0/1797795819 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5574003a00 0x7f5574003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:20.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.549+0000 7f558c888700 1 -- 172.21.15.16:0/1797795819 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55740035b0 con 0x7f5574004850 2023-10-16T12:11:20.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.549+0000 7f558c888700 1 --2- 172.21.15.16:0/1797795819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5574004850 0x7f5574004cd0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f5578009b60 tx=0x7f5574005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f85833beebfd8316 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:20.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.549+0000 7f557ffff700 1 -- 172.21.15.16:0/1797795819 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f557800e6c0 con 0x7f5574004850 2023-10-16T12:11:20.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.549+0000 7f557ffff700 1 -- 172.21.15.16:0/1797795819 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f557800e820 con 0x7f5574004850 2023-10-16T12:11:20.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.549+0000 7f557ffff700 1 -- 172.21.15.16:0/1797795819 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f557800eb90 con 0x7f5574004850 2023-10-16T12:11:20.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.550+0000 7f558f2ed700 1 -- 172.21.15.16:0/1797795819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5574004850 msgr2=0x7f5574004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:20.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.550+0000 7f558f2ed700 1 --2- 172.21.15.16:0/1797795819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5574004850 0x7f5574004cd0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f5578009b60 tx=0x7f5574005880 comp rx=0 tx=0).stop 2023-10-16T12:11:20.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.550+0000 7f558d88a700 1 -- 172.21.15.16:0/1797795819 reap_dead start 2023-10-16T12:11:20.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.550+0000 7f558f2ed700 1 -- 172.21.15.16:0/1797795819 shutdown_connections 2023-10-16T12:11:20.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.550+0000 7f558f2ed700 1 -- 172.21.15.16:0/1797795819 >> 172.21.15.16:0/1797795819 conn(0x7f55880b1fd0 msgr2=0x7f55880b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:20.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.550+0000 7f558f2ed700 1 -- 172.21.15.16:0/1797795819 shutdown_connections 2023-10-16T12:11:20.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.550+0000 7f558f2ed700 1 -- 172.21.15.16:0/1797795819 wait complete. 2023-10-16T12:11:20.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.551+0000 7f558f2ed700 1 Processor -- start 2023-10-16T12:11:20.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.551+0000 7f558f2ed700 1 -- start start 2023-10-16T12:11:20.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.552+0000 7f558f2ed700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5574004850 0x7f55880b0f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:20.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.552+0000 7f558f2ed700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f55880ca240 0x7f558814a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:20.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.552+0000 7f558f2ed700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f55880b5590 0x7f55880b5a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:20.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.552+0000 7f558f2ed700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55880b14d0 con 0x7f5574004850 2023-10-16T12:11:20.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.552+0000 7f558f2ed700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55880b5f50 con 0x7f55880ca240 2023-10-16T12:11:20.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.552+0000 7f558f2ed700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55880c7f20 con 0x7f55880b5590 2023-10-16T12:11:20.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.552+0000 7f558d089700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5574004850 0x7f55880b0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:20.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.552+0000 7f558d089700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5574004850 0x7f55880b0f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34200/0 (socket says 172.21.15.16:34200) 2023-10-16T12:11:20.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.552+0000 7f558d089700 1 -- 172.21.15.16:0/1170739354 learned_addr learned my addr 172.21.15.16:0/1170739354 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:20.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.552+0000 7f558d88a700 1 --2- 172.21.15.16:0/1170739354 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f55880b5590 0x7f55880b5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:20.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.552+0000 7f558c888700 1 --2- 172.21.15.16:0/1170739354 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f55880ca240 0x7f558814a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:20.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.552+0000 7f558d089700 1 -- 172.21.15.16:0/1170739354 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f55880b5590 msgr2=0x7f55880b5a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:20.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.552+0000 7f558d089700 1 --2- 172.21.15.16:0/1170739354 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f55880b5590 0x7f55880b5a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:20.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.552+0000 7f558d089700 1 -- 172.21.15.16:0/1170739354 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f55880ca240 msgr2=0x7f558814a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:20.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.552+0000 7f558d089700 1 --2- 172.21.15.16:0/1170739354 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f55880ca240 0x7f558814a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:20.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.552+0000 7f558d089700 1 -- 172.21.15.16:0/1170739354 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f558815ed00 con 0x7f5574004850 2023-10-16T12:11:20.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.552+0000 7f558d88a700 1 --2- 172.21.15.16:0/1170739354 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f55880b5590 0x7f55880b5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:11:20.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.553+0000 7f558d089700 1 --2- 172.21.15.16:0/1170739354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5574004850 0x7f55880b0f90 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f5584000fb0 tx=0x7f55840075f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:20.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.553+0000 7f557dffb700 1 -- 172.21.15.16:0/1170739354 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55840077d0 con 0x7f5574004850 2023-10-16T12:11:20.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.553+0000 7f557dffb700 1 -- 172.21.15.16:0/1170739354 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f558400f870 con 0x7f5574004850 2023-10-16T12:11:20.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.553+0000 7f557dffb700 1 -- 172.21.15.16:0/1170739354 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f558400fc20 con 0x7f5574004850 2023-10-16T12:11:20.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.553+0000 7f558f2ed700 1 -- 172.21.15.16:0/1170739354 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f558814adc0 con 0x7f5574004850 2023-10-16T12:11:20.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.553+0000 7f558f2ed700 1 -- 172.21.15.16:0/1170739354 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f558815bd50 con 0x7f5574004850 2023-10-16T12:11:20.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.554+0000 7f558f2ed700 1 -- 172.21.15.16:0/1170739354 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f558804ebb0 con 0x7f5574004850 2023-10-16T12:11:20.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.554+0000 7f557dffb700 1 -- 172.21.15.16:0/1170739354 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f558400fd80 con 0x7f5574004850 2023-10-16T12:11:20.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.555+0000 7f557dffb700 1 --2- 172.21.15.16:0/1170739354 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f556c06af00 0x7f556c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:20.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.555+0000 7f557dffb700 1 -- 172.21.15.16:0/1170739354 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f5584089a00 con 0x7f5574004850 2023-10-16T12:11:20.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.555+0000 7f558c888700 1 --2- 172.21.15.16:0/1170739354 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f556c06af00 0x7f556c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:20.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.555+0000 7f558c888700 1 --2- 172.21.15.16:0/1170739354 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f556c06af00 0x7f556c06d3c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f55780090b0 tx=0x7f5578008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:20.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.559+0000 7f557dffb700 1 -- 172.21.15.16:0/1170739354 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f55840547c0 con 0x7f5574004850 2023-10-16T12:11:20.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.813+0000 7f558f2ed700 1 -- 172.21.15.16:0/1170739354 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f55880cb380 con 0x7f5574004850 2023-10-16T12:11:20.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.813+0000 7f557dffb700 1 -- 172.21.15.16:0/1170739354 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) v1 ==== 53+0+88 (secure 0 0 0) 0x7f5584057de0 con 0x7f5574004850 2023-10-16T12:11:20.816 INFO:teuthology.orchestra.run.smithi016.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-10-16T12:11:20.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.815+0000 7f558f2ed700 1 -- 172.21.15.16:0/1170739354 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f556c06af00 msgr2=0x7f556c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:20.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.815+0000 7f558f2ed700 1 --2- 172.21.15.16:0/1170739354 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f556c06af00 0x7f556c06d3c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f55780090b0 tx=0x7f5578008030 comp rx=0 tx=0).stop 2023-10-16T12:11:20.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.815+0000 7f558f2ed700 1 -- 172.21.15.16:0/1170739354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5574004850 msgr2=0x7f55880b0f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:20.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.815+0000 7f558f2ed700 1 --2- 172.21.15.16:0/1170739354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5574004850 0x7f55880b0f90 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f5584000fb0 tx=0x7f55840075f0 comp rx=0 tx=0).stop 2023-10-16T12:11:20.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.815+0000 7f558f2ed700 1 -- 172.21.15.16:0/1170739354 shutdown_connections 2023-10-16T12:11:20.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.815+0000 7f558f2ed700 1 --2- 172.21.15.16:0/1170739354 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f556c06af00 0x7f556c06d3c0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:20.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.815+0000 7f558f2ed700 1 --2- 172.21.15.16:0/1170739354 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f55880b5590 0x7f55880b5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:20.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.815+0000 7f558f2ed700 1 --2- 172.21.15.16:0/1170739354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5574004850 0x7f55880b0f90 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:20.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.815+0000 7f558f2ed700 1 --2- 172.21.15.16:0/1170739354 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f55880ca240 0x7f558814a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:20.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.815+0000 7f558f2ed700 1 -- 172.21.15.16:0/1170739354 >> 172.21.15.16:0/1170739354 conn(0x7f55880b1fd0 msgr2=0x7f55880b48f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:20.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.816+0000 7f558f2ed700 1 -- 172.21.15.16:0/1170739354 shutdown_connections 2023-10-16T12:11:20.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:20.816+0000 7f558f2ed700 1 -- 172.21.15.16:0/1170739354 wait complete. 2023-10-16T12:11:20.829 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-10-16T12:11:20.830 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T12:11:20.830 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi049.front.sepia.ceph.com 2023-10-16T12:11:20.830 INFO:tasks.cephfs.mount:self.client.name = client.1 2023-10-16T12:11:20.830 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-10-16T12:11:20.831 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-10-16T12:11:20.831 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-16T12:11:20.831 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T12:11:20.831 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2023-10-16T12:11:20.832 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:20.832 DEBUG:teuthology.orchestra.run.smithi049:> ip addr 2023-10-16T12:11:20.853 INFO:teuthology.orchestra.run.smithi049.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T12:11:20.853 INFO:teuthology.orchestra.run.smithi049.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T12:11:20.854 INFO:teuthology.orchestra.run.smithi049.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T12:11:20.854 INFO:teuthology.orchestra.run.smithi049.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:11:20.854 INFO:teuthology.orchestra.run.smithi049.stdout: inet6 ::1/128 scope host 2023-10-16T12:11:20.854 INFO:teuthology.orchestra.run.smithi049.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:11:20.855 INFO:teuthology.orchestra.run.smithi049.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T12:11:20.855 INFO:teuthology.orchestra.run.smithi049.stdout: link/ether 0c:c4:7a:6b:ff:84 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:11:20.855 INFO:teuthology.orchestra.run.smithi049.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T12:11:20.855 INFO:teuthology.orchestra.run.smithi049.stdout: link/ether 0c:c4:7a:6b:ff:85 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:11:20.855 INFO:teuthology.orchestra.run.smithi049.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T12:11:20.855 INFO:teuthology.orchestra.run.smithi049.stdout: link/ether 0c:c4:7a:bd:15:66 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:11:20.856 INFO:teuthology.orchestra.run.smithi049.stdout: inet 172.21.15.49/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-16T12:11:20.856 INFO:teuthology.orchestra.run.smithi049.stdout: valid_lft 42367sec preferred_lft 42367sec 2023-10-16T12:11:20.856 INFO:teuthology.orchestra.run.smithi049.stdout: inet6 fe80::ec4:7aff:febd:1566/64 scope link 2023-10-16T12:11:20.856 INFO:teuthology.orchestra.run.smithi049.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:11:20.856 INFO:teuthology.orchestra.run.smithi049.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T12:11:20.857 INFO:teuthology.orchestra.run.smithi049.stdout: link/ether 0c:c4:7a:bd:15:67 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:11:20.857 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-10-16T12:11:20.857 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:20.858 DEBUG:teuthology.orchestra.run.smithi049:> (cd / && exec bash -c ' 2023-10-16T12:11:20.858 DEBUG:teuthology.orchestra.run.smithi049:> set -e 2023-10-16T12:11:20.858 DEBUG:teuthology.orchestra.run.smithi049:> sudo ip link add name ceph-brx type bridge 2023-10-16T12:11:20.858 DEBUG:teuthology.orchestra.run.smithi049:> sudo ip addr flush dev ceph-brx 2023-10-16T12:11:20.859 DEBUG:teuthology.orchestra.run.smithi049:> sudo ip link set ceph-brx up 2023-10-16T12:11:20.859 DEBUG:teuthology.orchestra.run.smithi049:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-10-16T12:11:20.859 DEBUG:teuthology.orchestra.run.smithi049:> ') 2023-10-16T12:11:21.039 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:21.040 DEBUG:teuthology.orchestra.run.smithi049:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T12:11:21.070 INFO:teuthology.orchestra.run.smithi049.stdout:1 2023-10-16T12:11:21.072 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:21.072 DEBUG:teuthology.orchestra.run.smithi049:> route 2023-10-16T12:11:21.146 INFO:teuthology.orchestra.run.smithi049.stdout:Kernel IP routing table 2023-10-16T12:11:21.146 INFO:teuthology.orchestra.run.smithi049.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T12:11:21.147 INFO:teuthology.orchestra.run.smithi049.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-16T12:11:21.147 INFO:teuthology.orchestra.run.smithi049.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-16T12:11:21.147 INFO:teuthology.orchestra.run.smithi049.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T12:11:21.148 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:21.148 DEBUG:teuthology.orchestra.run.smithi049:> (cd / && exec bash -c ' 2023-10-16T12:11:21.149 DEBUG:teuthology.orchestra.run.smithi049:> set -e 2023-10-16T12:11:21.149 DEBUG:teuthology.orchestra.run.smithi049:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-16T12:11:21.149 DEBUG:teuthology.orchestra.run.smithi049:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-16T12:11:21.149 DEBUG:teuthology.orchestra.run.smithi049:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-16T12:11:21.150 DEBUG:teuthology.orchestra.run.smithi049:> ') 2023-10-16T12:11:21.385 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:21.385 DEBUG:teuthology.orchestra.run.smithi049:> ip netns list 2023-10-16T12:11:21.402 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:21.403 DEBUG:teuthology.orchestra.run.smithi049:> ip netns list-id 2023-10-16T12:11:21.460 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:21.460 DEBUG:teuthology.orchestra.run.smithi049:> (cd / && exec bash -c ' 2023-10-16T12:11:21.460 DEBUG:teuthology.orchestra.run.smithi049:> set -e 2023-10-16T12:11:21.461 DEBUG:teuthology.orchestra.run.smithi049:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2023-10-16T12:11:21.461 DEBUG:teuthology.orchestra.run.smithi049:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2023-10-16T12:11:21.461 DEBUG:teuthology.orchestra.run.smithi049:> ') 2023-10-16T12:11:21.555 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2023-10-16T12:11:21.555 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:21.556 DEBUG:teuthology.orchestra.run.smithi049:> (cd / && exec bash -c ' 2023-10-16T12:11:21.556 DEBUG:teuthology.orchestra.run.smithi049:> set -e 2023-10-16T12:11:21.556 DEBUG:teuthology.orchestra.run.smithi049:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2023-10-16T12:11:21.556 DEBUG:teuthology.orchestra.run.smithi049:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-10-16T12:11:21.557 DEBUG:teuthology.orchestra.run.smithi049:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2023-10-16T12:11:21.557 DEBUG:teuthology.orchestra.run.smithi049:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2023-10-16T12:11:21.557 DEBUG:teuthology.orchestra.run.smithi049:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2023-10-16T12:11:21.557 DEBUG:teuthology.orchestra.run.smithi049:> ') 2023-10-16T12:11:21.783 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:21.784 DEBUG:teuthology.orchestra.run.smithi049:> (cd / && exec bash -c ' 2023-10-16T12:11:21.784 DEBUG:teuthology.orchestra.run.smithi049:> set -e 2023-10-16T12:11:21.784 DEBUG:teuthology.orchestra.run.smithi049:> sudo ip link set brx.0 up 2023-10-16T12:11:21.784 DEBUG:teuthology.orchestra.run.smithi049:> sudo ip link set dev brx.0 master ceph-brx 2023-10-16T12:11:21.785 DEBUG:teuthology.orchestra.run.smithi049:> ') 2023-10-16T12:11:21.888 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T12:11:21.888 DEBUG:teuthology.orchestra.run.smithi049:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2023-10-16T12:11:21.908 INFO:teuthology.orchestra.run.smithi049.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2023-10-16T12:11:21.908 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T12:11:21.909 DEBUG:teuthology.orchestra.run.smithi049:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2023-10-16T12:11:21.965 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-16T12:11:21.966 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:21.966 DEBUG:teuthology.orchestra.run.smithi049:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2023-10-16T12:11:22.060 INFO:teuthology.orchestra.run.smithi049.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2023-10-16T12:11:22.061 INFO:teuthology.orchestra.run.smithi049.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2023-10-16T12:11:22.061 INFO:teuthology.orchestra.run.smithi049.stdout:invalid new device string format 2023-10-16T12:11:22.153 INFO:teuthology.orchestra.run.smithi049.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2023-10-16T12:11:22.153 INFO:teuthology.orchestra.run.smithi049.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2023-10-16T12:11:22.153 INFO:teuthology.orchestra.run.smithi049.stdout:invalid new device string format 2023-10-16T12:11:22.154 INFO:teuthology.orchestra.run.smithi049.stdout:mount.ceph: resolved to: "172.21.15.16:3300,172.21.15.49:3300,172.21.15.49:3301" 2023-10-16T12:11:22.154 INFO:teuthology.orchestra.run.smithi049.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T12:11:22.182 INFO:teuthology.orchestra.run.smithi049.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2023-10-16T12:11:22.182 INFO:teuthology.orchestra.run.smithi049.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2023-10-16T12:11:22.182 INFO:teuthology.orchestra.run.smithi049.stdout:invalid new device string format 2023-10-16T12:11:22.182 INFO:teuthology.orchestra.run.smithi049.stdout:mount.ceph: resolved to: "172.21.15.16:3300,172.21.15.49:3300,172.21.15.49:3301" 2023-10-16T12:11:22.183 INFO:teuthology.orchestra.run.smithi049.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T12:11:22.183 INFO:teuthology.orchestra.run.smithi049.stdout:mount.ceph: resolved to: "172.21.15.16:6789,172.21.15.49:6789,172.21.15.49:6790" 2023-10-16T12:11:22.183 INFO:teuthology.orchestra.run.smithi049.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.16:6789,172.21.15.49:6789,172.21.15.49:6790:/ 2023-10-16T12:11:22.183 INFO:teuthology.orchestra.run.smithi049.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=5b28dfa1-b21f-474a-9b2e-9fb77af58254" will pass to kernel 2023-10-16T12:11:22.199 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-16T12:11:22.199 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:22.200 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2023-10-16T12:11:22.230 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:22.231 DEBUG:teuthology.orchestra.run.smithi049:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T12:11:22.303 INFO:teuthology.orchestra.run.smithi049.stdout:client4771 2023-10-16T12:11:22.304 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T12:11:22.304 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:11:22.361 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:22.361 DEBUG:teuthology.orchestra.run.smithi049:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T12:11:22.433 INFO:teuthology.orchestra.run.smithi049.stdout:client4771 2023-10-16T12:11:22.433 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:22.434 DEBUG:teuthology.orchestra.run.smithi049:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T12:11:22.506 INFO:teuthology.orchestra.run.smithi049.stdout:client4771 2023-10-16T12:11:22.507 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:22.507 DEBUG:teuthology.orchestra.run.smithi049:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/5b28dfa1-b21f-474a-9b2e-9fb77af58254.client4771/status') 2023-10-16T12:11:22.588 INFO:teuthology.orchestra.run.smithi049.stdout:instance: client.4771 (3)192.168.0.1:0/3882871295 2023-10-16T12:11:22.588 INFO:teuthology.orchestra.run.smithi049.stdout:blocklisted: false 2023-10-16T12:11:22.589 INFO:teuthology.orchestra.run.smithi049.stderr:0+1 records in 2023-10-16T12:11:22.589 INFO:teuthology.orchestra.run.smithi049.stderr:0+1 records out 2023-10-16T12:11:22.589 INFO:teuthology.orchestra.run.smithi049.stderr:69 bytes copied, 2.3288e-05 s, 3.0 MB/s 2023-10-16T12:11:22.590 INFO:tasks.kclient:client.3 config is {'syntax': 'v1'} 2023-10-16T12:11:22.590 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-10-16T12:11:22.590 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2023-10-16T12:11:22.591 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T12:11:22.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.728+0000 7f060dfab700 1 Processor -- start 2023-10-16T12:11:22.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.728+0000 7f060dfab700 1 -- start start 2023-10-16T12:11:22.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.729+0000 7f060dfab700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f060806a7e0 0x7f060806ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:22.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.729+0000 7f060dfab700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0608161c30 con 0x7f060815b280 2023-10-16T12:11:22.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.729+0000 7f060dfab700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0608161f10 con 0x7f06080cc8a0 2023-10-16T12:11:22.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.729+0000 7f060dfab700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f060806b190 con 0x7f060806a7e0 2023-10-16T12:11:22.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.729+0000 7f06077fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f060815b280 0x7f06081613e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51714/0 (socket says 172.21.15.16:51714) 2023-10-16T12:11:22.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.729+0000 7f06077fe700 1 -- 172.21.15.16:0/2968075630 learned_addr learned my addr 172.21.15.16:0/2968075630 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:22.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.729+0000 7f0607fff700 1 --2- 172.21.15.16:0/2968075630 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f060806a7e0 0x7f060806ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:22.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.729+0000 7f06067fc700 1 -- 172.21.15.16:0/2968075630 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 470574234 0 0) 0x7f0608161c30 con 0x7f060815b280 2023-10-16T12:11:22.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.729+0000 7f06067fc700 1 -- 172.21.15.16:0/2968075630 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05f00035b0 con 0x7f060815b280 2023-10-16T12:11:22.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.730+0000 7f06067fc700 1 -- 172.21.15.16:0/2968075630 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2830755964 0 0) 0x7f05f00035b0 con 0x7f060815b280 2023-10-16T12:11:22.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.730+0000 7f06067fc700 1 -- 172.21.15.16:0/2968075630 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f060806a7e0 msgr2=0x7f060806ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:22.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.730+0000 7f06067fc700 1 --2- 172.21.15.16:0/2968075630 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f060806a7e0 0x7f060806ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:22.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.730+0000 7f06067fc700 1 -- 172.21.15.16:0/2968075630 >> v1:172.21.15.49:6789/0 conn(0x7f06080cc8a0 legacy=0x7f060815e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:22.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.730+0000 7f06067fc700 1 -- 172.21.15.16:0/2968075630 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06081621f0 con 0x7f060815b280 2023-10-16T12:11:22.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.730+0000 7f06067fc700 1 -- 172.21.15.16:0/2968075630 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f05fc001670 con 0x7f060815b280 2023-10-16T12:11:22.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.730+0000 7f06067fc700 1 -- 172.21.15.16:0/2968075630 >> v1:172.21.15.16:6789/0 conn(0x7f060815b280 legacy=0x7f06081613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:22.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.730+0000 7f06067fc700 1 --2- 172.21.15.16:0/2968075630 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f060806a7e0 0x7f05f0003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:22.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.730+0000 7f06067fc700 1 --2- 172.21.15.16:0/2968075630 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f05f0003f70 0x7f05f0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:22.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.730+0000 7f06067fc700 1 --2- 172.21.15.16:0/2968075630 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05f0004d90 0x7f05f0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:22.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.730+0000 7f06067fc700 1 -- 172.21.15.16:0/2968075630 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f060806b190 con 0x7f05f0004d90 2023-10-16T12:11:22.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.730+0000 7f06067fc700 1 -- 172.21.15.16:0/2968075630 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05f000ba70 con 0x7f05f0003f70 2023-10-16T12:11:22.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.730+0000 7f06067fc700 1 -- 172.21.15.16:0/2968075630 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05f000bc30 con 0x7f060806a7e0 2023-10-16T12:11:22.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.730+0000 7f0607fff700 1 --2- 172.21.15.16:0/2968075630 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05f0004d90 0x7f05f0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:22.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.730+0000 7f06077fe700 1 --2- 172.21.15.16:0/2968075630 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f060806a7e0 0x7f05f0003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:22.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.730+0000 7f0606ffd700 1 --2- 172.21.15.16:0/2968075630 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f05f0003f70 0x7f05f0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:22.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.730+0000 7f0607fff700 1 -- 172.21.15.16:0/2968075630 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f060806a7e0 msgr2=0x7f05f0003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:22.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.730+0000 7f0607fff700 1 --2- 172.21.15.16:0/2968075630 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f060806a7e0 0x7f05f0003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:22.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.730+0000 7f0607fff700 1 -- 172.21.15.16:0/2968075630 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f05f0003f70 msgr2=0x7f05f0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:22.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.730+0000 7f0607fff700 1 --2- 172.21.15.16:0/2968075630 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f05f0003f70 0x7f05f0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:22.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.730+0000 7f0607fff700 1 -- 172.21.15.16:0/2968075630 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05f00035b0 con 0x7f05f0004d90 2023-10-16T12:11:22.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.730+0000 7f06077fe700 1 --2- 172.21.15.16:0/2968075630 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f060806a7e0 0x7f05f0003a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:11:22.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.730+0000 7f0607fff700 1 --2- 172.21.15.16:0/2968075630 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05f0004d90 0x7f05f0005210 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f0600009fc0 tx=0x7f05f0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7fe1a8eb70eaada server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:22.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.731+0000 7f06067fc700 1 -- 172.21.15.16:0/2968075630 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f060000ec30 con 0x7f05f0004d90 2023-10-16T12:11:22.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.731+0000 7f06067fc700 1 -- 172.21.15.16:0/2968075630 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f060000f040 con 0x7f05f0004d90 2023-10-16T12:11:22.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.731+0000 7f06067fc700 1 -- 172.21.15.16:0/2968075630 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0600014480 con 0x7f05f0004d90 2023-10-16T12:11:22.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.731+0000 7f060dfab700 1 -- 172.21.15.16:0/2968075630 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05f0004d90 msgr2=0x7f05f0005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:22.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.731+0000 7f060dfab700 1 --2- 172.21.15.16:0/2968075630 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05f0004d90 0x7f05f0005210 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f0600009fc0 tx=0x7f05f0005880 comp rx=0 tx=0).stop 2023-10-16T12:11:22.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.731+0000 7f0607fff700 1 -- 172.21.15.16:0/2968075630 reap_dead start 2023-10-16T12:11:22.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.731+0000 7f060dfab700 1 -- 172.21.15.16:0/2968075630 shutdown_connections 2023-10-16T12:11:22.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.731+0000 7f060dfab700 1 -- 172.21.15.16:0/2968075630 >> 172.21.15.16:0/2968075630 conn(0x7f06080b1fd0 msgr2=0x7f06080b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:22.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.731+0000 7f060dfab700 1 -- 172.21.15.16:0/2968075630 shutdown_connections 2023-10-16T12:11:22.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.731+0000 7f060dfab700 1 -- 172.21.15.16:0/2968075630 wait complete. 2023-10-16T12:11:22.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.733+0000 7f060dfab700 1 Processor -- start 2023-10-16T12:11:22.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.733+0000 7f060dfab700 1 -- start start 2023-10-16T12:11:22.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.734+0000 7f060dfab700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0608159b90 0x7f06080ca060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:22.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.734+0000 7f060dfab700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f06080c8910 0x7f060815c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:22.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.734+0000 7f060dfab700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06080c47c0 0x7f060815ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:22.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.734+0000 7f060dfab700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06080c84f0 con 0x7f06080c47c0 2023-10-16T12:11:22.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.734+0000 7f060dfab700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06080c4d80 con 0x7f06080c8910 2023-10-16T12:11:22.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.734+0000 7f060dfab700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0608159f70 con 0x7f0608159b90 2023-10-16T12:11:22.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.734+0000 7f0607fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06080c47c0 0x7f060815ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:22.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.734+0000 7f0607fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06080c47c0 0x7f060815ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34222/0 (socket says 172.21.15.16:34222) 2023-10-16T12:11:22.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.734+0000 7f0607fff700 1 -- 172.21.15.16:0/797352940 learned_addr learned my addr 172.21.15.16:0/797352940 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:22.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.734+0000 7f0606ffd700 1 --2- 172.21.15.16:0/797352940 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f06080c8910 0x7f060815c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:22.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.734+0000 7f06077fe700 1 --2- 172.21.15.16:0/797352940 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0608159b90 0x7f06080ca060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:22.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.734+0000 7f0607fff700 1 -- 172.21.15.16:0/797352940 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0608159b90 msgr2=0x7f06080ca060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:22.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.734+0000 7f0607fff700 1 --2- 172.21.15.16:0/797352940 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0608159b90 0x7f06080ca060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:22.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.734+0000 7f0607fff700 1 -- 172.21.15.16:0/797352940 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f06080c8910 msgr2=0x7f060815c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:22.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.734+0000 7f0607fff700 1 --2- 172.21.15.16:0/797352940 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f06080c8910 0x7f060815c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:22.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.735+0000 7f0607fff700 1 -- 172.21.15.16:0/797352940 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06080c7b50 con 0x7f06080c47c0 2023-10-16T12:11:22.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.735+0000 7f0607fff700 1 --2- 172.21.15.16:0/797352940 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06080c47c0 0x7f060815ce60 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f0600016000 tx=0x7f0600009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:22.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.735+0000 7f05effff700 1 -- 172.21.15.16:0/797352940 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0600007000 con 0x7f06080c47c0 2023-10-16T12:11:22.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.735+0000 7f05effff700 1 -- 172.21.15.16:0/797352940 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0600022070 con 0x7f06080c47c0 2023-10-16T12:11:22.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.735+0000 7f05effff700 1 -- 172.21.15.16:0/797352940 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0600029480 con 0x7f06080c47c0 2023-10-16T12:11:22.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.735+0000 7f060dfab700 1 -- 172.21.15.16:0/797352940 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06080c7e30 con 0x7f06080c47c0 2023-10-16T12:11:22.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.735+0000 7f060dfab700 1 -- 172.21.15.16:0/797352940 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06080c5200 con 0x7f06080c47c0 2023-10-16T12:11:22.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.736+0000 7f060dfab700 1 -- 172.21.15.16:0/797352940 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06080c8630 con 0x7f06080c47c0 2023-10-16T12:11:22.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.736+0000 7f05effff700 1 -- 172.21.15.16:0/797352940 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f060001e430 con 0x7f06080c47c0 2023-10-16T12:11:22.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.737+0000 7f05effff700 1 --2- 172.21.15.16:0/797352940 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f05f406af00 0x7f05f406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:22.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.737+0000 7f05effff700 1 -- 172.21.15.16:0/797352940 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f0600024070 con 0x7f06080c47c0 2023-10-16T12:11:22.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.737+0000 7f06077fe700 1 --2- 172.21.15.16:0/797352940 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f05f406af00 0x7f05f406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:22.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.737+0000 7f06077fe700 1 --2- 172.21.15.16:0/797352940 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f05f406af00 0x7f05f406d3c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f05fc007d70 tx=0x7f05fc007c80 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:22.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.741+0000 7f05effff700 1 -- 172.21.15.16:0/797352940 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f06080c8630 con 0x7f06080c47c0 2023-10-16T12:11:22.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.994+0000 7f060dfab700 1 -- 172.21.15.16:0/797352940 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f06080c8630 con 0x7f06080c47c0 2023-10-16T12:11:22.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.994+0000 7f05effff700 1 -- 172.21.15.16:0/797352940 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) v1 ==== 53+0+88 (secure 0 0 0) 0x7f0600018070 con 0x7f06080c47c0 2023-10-16T12:11:22.996 INFO:teuthology.orchestra.run.smithi016.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-10-16T12:11:22.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.996+0000 7f060dfab700 1 -- 172.21.15.16:0/797352940 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f05f406af00 msgr2=0x7f05f406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:22.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.996+0000 7f060dfab700 1 --2- 172.21.15.16:0/797352940 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f05f406af00 0x7f05f406d3c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f05fc007d70 tx=0x7f05fc007c80 comp rx=0 tx=0).stop 2023-10-16T12:11:22.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.996+0000 7f060dfab700 1 -- 172.21.15.16:0/797352940 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06080c47c0 msgr2=0x7f060815ce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:22.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.996+0000 7f060dfab700 1 --2- 172.21.15.16:0/797352940 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06080c47c0 0x7f060815ce60 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f0600016000 tx=0x7f0600009750 comp rx=0 tx=0).stop 2023-10-16T12:11:23.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.996+0000 7f060dfab700 1 -- 172.21.15.16:0/797352940 shutdown_connections 2023-10-16T12:11:23.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.996+0000 7f060dfab700 1 --2- 172.21.15.16:0/797352940 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f05f406af00 0x7f05f406d3c0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:23.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.996+0000 7f060dfab700 1 --2- 172.21.15.16:0/797352940 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06080c47c0 0x7f060815ce60 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:23.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.996+0000 7f060dfab700 1 --2- 172.21.15.16:0/797352940 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0608159b90 0x7f06080ca060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:23.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.996+0000 7f060dfab700 1 --2- 172.21.15.16:0/797352940 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f06080c8910 0x7f060815c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:23.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.996+0000 7f060dfab700 1 -- 172.21.15.16:0/797352940 >> 172.21.15.16:0/797352940 conn(0x7f06080b1fd0 msgr2=0x7f060815a890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:23.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.997+0000 7f060dfab700 1 -- 172.21.15.16:0/797352940 shutdown_connections 2023-10-16T12:11:23.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:22.997+0000 7f060dfab700 1 -- 172.21.15.16:0/797352940 wait complete. 2023-10-16T12:11:23.010 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-10-16T12:11:23.011 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T12:11:23.011 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi049.front.sepia.ceph.com 2023-10-16T12:11:23.011 INFO:tasks.cephfs.mount:self.client.name = client.3 2023-10-16T12:11:23.012 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2023-10-16T12:11:23.012 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-10-16T12:11:23.012 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-16T12:11:23.012 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T12:11:23.012 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2023-10-16T12:11:23.013 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:23.013 DEBUG:teuthology.orchestra.run.smithi049:> ip addr 2023-10-16T12:11:23.032 INFO:teuthology.orchestra.run.smithi049.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T12:11:23.032 INFO:teuthology.orchestra.run.smithi049.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T12:11:23.033 INFO:teuthology.orchestra.run.smithi049.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T12:11:23.033 INFO:teuthology.orchestra.run.smithi049.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:11:23.033 INFO:teuthology.orchestra.run.smithi049.stdout: inet6 ::1/128 scope host 2023-10-16T12:11:23.033 INFO:teuthology.orchestra.run.smithi049.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:11:23.033 INFO:teuthology.orchestra.run.smithi049.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T12:11:23.034 INFO:teuthology.orchestra.run.smithi049.stdout: link/ether 0c:c4:7a:6b:ff:84 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:11:23.034 INFO:teuthology.orchestra.run.smithi049.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T12:11:23.034 INFO:teuthology.orchestra.run.smithi049.stdout: link/ether 0c:c4:7a:6b:ff:85 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:11:23.034 INFO:teuthology.orchestra.run.smithi049.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T12:11:23.034 INFO:teuthology.orchestra.run.smithi049.stdout: link/ether 0c:c4:7a:bd:15:66 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:11:23.035 INFO:teuthology.orchestra.run.smithi049.stdout: inet 172.21.15.49/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-16T12:11:23.035 INFO:teuthology.orchestra.run.smithi049.stdout: valid_lft 42365sec preferred_lft 42365sec 2023-10-16T12:11:23.035 INFO:teuthology.orchestra.run.smithi049.stdout: inet6 fe80::ec4:7aff:febd:1566/64 scope link 2023-10-16T12:11:23.035 INFO:teuthology.orchestra.run.smithi049.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:11:23.035 INFO:teuthology.orchestra.run.smithi049.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T12:11:23.036 INFO:teuthology.orchestra.run.smithi049.stdout: link/ether 0c:c4:7a:bd:15:67 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:11:23.036 INFO:teuthology.orchestra.run.smithi049.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T12:11:23.036 INFO:teuthology.orchestra.run.smithi049.stdout: link/ether d2:3c:44:55:1d:2b brd ff:ff:ff:ff:ff:ff 2023-10-16T12:11:23.036 INFO:teuthology.orchestra.run.smithi049.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-16T12:11:23.036 INFO:teuthology.orchestra.run.smithi049.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:11:23.037 INFO:teuthology.orchestra.run.smithi049.stdout: inet6 fe80::b823:36ff:fe3c:ffe9/64 scope link 2023-10-16T12:11:23.037 INFO:teuthology.orchestra.run.smithi049.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:11:23.037 INFO:teuthology.orchestra.run.smithi049.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T12:11:23.037 INFO:teuthology.orchestra.run.smithi049.stdout: link/ether d2:3c:44:55:1d:2b brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2023-10-16T12:11:23.037 INFO:teuthology.orchestra.run.smithi049.stdout: inet6 fe80::d03c:44ff:fe55:1d2b/64 scope link tentative 2023-10-16T12:11:23.037 INFO:teuthology.orchestra.run.smithi049.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:11:23.038 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:23.038 DEBUG:teuthology.orchestra.run.smithi049:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T12:11:23.100 INFO:teuthology.orchestra.run.smithi049.stdout:1 2023-10-16T12:11:23.101 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:23.101 DEBUG:teuthology.orchestra.run.smithi049:> route 2023-10-16T12:11:23.161 INFO:teuthology.orchestra.run.smithi049.stdout:Kernel IP routing table 2023-10-16T12:11:23.161 INFO:teuthology.orchestra.run.smithi049.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T12:11:23.161 INFO:teuthology.orchestra.run.smithi049.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-16T12:11:23.162 INFO:teuthology.orchestra.run.smithi049.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-16T12:11:23.162 INFO:teuthology.orchestra.run.smithi049.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T12:11:23.162 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:23.162 DEBUG:teuthology.orchestra.run.smithi049:> (cd / && exec bash -c ' 2023-10-16T12:11:23.163 DEBUG:teuthology.orchestra.run.smithi049:> set -e 2023-10-16T12:11:23.163 DEBUG:teuthology.orchestra.run.smithi049:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-16T12:11:23.164 DEBUG:teuthology.orchestra.run.smithi049:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-16T12:11:23.164 DEBUG:teuthology.orchestra.run.smithi049:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-16T12:11:23.164 DEBUG:teuthology.orchestra.run.smithi049:> ') 2023-10-16T12:11:23.271 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:23.271 DEBUG:teuthology.orchestra.run.smithi049:> ip netns list 2023-10-16T12:11:23.288 INFO:teuthology.orchestra.run.smithi049.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-10-16T12:11:23.289 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:23.289 DEBUG:teuthology.orchestra.run.smithi049:> ip netns list-id 2023-10-16T12:11:23.346 INFO:teuthology.orchestra.run.smithi049.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2023-10-16T12:11:23.346 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:23.346 DEBUG:teuthology.orchestra.run.smithi049:> (cd / && exec bash -c ' 2023-10-16T12:11:23.347 DEBUG:teuthology.orchestra.run.smithi049:> set -e 2023-10-16T12:11:23.347 DEBUG:teuthology.orchestra.run.smithi049:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2023-10-16T12:11:23.347 DEBUG:teuthology.orchestra.run.smithi049:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2023-10-16T12:11:23.347 DEBUG:teuthology.orchestra.run.smithi049:> ') 2023-10-16T12:11:23.441 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:23.441 DEBUG:teuthology.orchestra.run.smithi049:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2023-10-16T12:11:23.483 INFO:teuthology.orchestra.run.smithi049.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T12:11:23.483 INFO:teuthology.orchestra.run.smithi049.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T12:11:23.484 INFO:teuthology.orchestra.run.smithi049.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T12:11:23.484 INFO:teuthology.orchestra.run.smithi049.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:11:23.484 INFO:teuthology.orchestra.run.smithi049.stdout: inet6 ::1/128 scope host 2023-10-16T12:11:23.484 INFO:teuthology.orchestra.run.smithi049.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:11:23.484 INFO:teuthology.orchestra.run.smithi049.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T12:11:23.485 INFO:teuthology.orchestra.run.smithi049.stdout: link/ether 82:92:b2:39:da:21 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-10-16T12:11:23.485 INFO:teuthology.orchestra.run.smithi049.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-10-16T12:11:23.485 INFO:teuthology.orchestra.run.smithi049.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:11:23.485 INFO:teuthology.orchestra.run.smithi049.stdout: inet6 fe80::8092:b2ff:fe39:da21/64 scope link tentative 2023-10-16T12:11:23.485 INFO:teuthology.orchestra.run.smithi049.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:11:23.499 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:23.499 DEBUG:teuthology.orchestra.run.smithi049:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2023-10-16T12:11:23.574 INFO:teuthology.orchestra.run.smithi049.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T12:11:23.575 INFO:teuthology.orchestra.run.smithi049.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T12:11:23.575 INFO:teuthology.orchestra.run.smithi049.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T12:11:23.575 INFO:teuthology.orchestra.run.smithi049.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:11:23.575 INFO:teuthology.orchestra.run.smithi049.stdout: inet6 ::1/128 scope host 2023-10-16T12:11:23.576 INFO:teuthology.orchestra.run.smithi049.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:11:23.576 INFO:teuthology.orchestra.run.smithi049.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T12:11:23.576 INFO:teuthology.orchestra.run.smithi049.stdout: link/ether 82:92:b2:39:da:21 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-10-16T12:11:23.576 INFO:teuthology.orchestra.run.smithi049.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-10-16T12:11:23.576 INFO:teuthology.orchestra.run.smithi049.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:11:23.577 INFO:teuthology.orchestra.run.smithi049.stdout: inet6 fe80::8092:b2ff:fe39:da21/64 scope link tentative 2023-10-16T12:11:23.577 INFO:teuthology.orchestra.run.smithi049.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:11:23.591 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2023-10-16T12:11:23.591 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:23.591 DEBUG:teuthology.orchestra.run.smithi049:> (cd / && exec bash -c ' 2023-10-16T12:11:23.592 DEBUG:teuthology.orchestra.run.smithi049:> set -e 2023-10-16T12:11:23.592 DEBUG:teuthology.orchestra.run.smithi049:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2023-10-16T12:11:23.592 DEBUG:teuthology.orchestra.run.smithi049:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2023-10-16T12:11:23.593 DEBUG:teuthology.orchestra.run.smithi049:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2023-10-16T12:11:23.593 DEBUG:teuthology.orchestra.run.smithi049:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2023-10-16T12:11:23.593 DEBUG:teuthology.orchestra.run.smithi049:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2023-10-16T12:11:23.594 DEBUG:teuthology.orchestra.run.smithi049:> ') 2023-10-16T12:11:23.833 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:23.834 DEBUG:teuthology.orchestra.run.smithi049:> (cd / && exec bash -c ' 2023-10-16T12:11:23.834 DEBUG:teuthology.orchestra.run.smithi049:> set -e 2023-10-16T12:11:23.834 DEBUG:teuthology.orchestra.run.smithi049:> sudo ip link set brx.1 up 2023-10-16T12:11:23.834 DEBUG:teuthology.orchestra.run.smithi049:> sudo ip link set dev brx.1 master ceph-brx 2023-10-16T12:11:23.835 DEBUG:teuthology.orchestra.run.smithi049:> ') 2023-10-16T12:11:23.954 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T12:11:23.955 DEBUG:teuthology.orchestra.run.smithi049:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2023-10-16T12:11:23.973 INFO:teuthology.orchestra.run.smithi049.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2023-10-16T12:11:23.974 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T12:11:23.974 DEBUG:teuthology.orchestra.run.smithi049:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2023-10-16T12:11:24.030 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-16T12:11:24.030 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:24.031 DEBUG:teuthology.orchestra.run.smithi049:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.3 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2023-10-16T12:11:24.116 INFO:teuthology.orchestra.run.smithi049.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2023-10-16T12:11:24.116 INFO:teuthology.orchestra.run.smithi049.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs". 2023-10-16T12:11:24.116 INFO:teuthology.orchestra.run.smithi049.stdout:invalid new device string format 2023-10-16T12:11:24.125 INFO:teuthology.orchestra.run.smithi049.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2023-10-16T12:11:24.125 INFO:teuthology.orchestra.run.smithi049.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs". 2023-10-16T12:11:24.125 INFO:teuthology.orchestra.run.smithi049.stdout:invalid new device string format 2023-10-16T12:11:24.126 INFO:teuthology.orchestra.run.smithi049.stdout:mount.ceph: resolved to: "172.21.15.16:3300,172.21.15.49:3300,172.21.15.49:3301" 2023-10-16T12:11:24.126 INFO:teuthology.orchestra.run.smithi049.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T12:11:24.150 INFO:teuthology.orchestra.run.smithi049.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2023-10-16T12:11:24.151 INFO:teuthology.orchestra.run.smithi049.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs". 2023-10-16T12:11:24.151 INFO:teuthology.orchestra.run.smithi049.stdout:invalid new device string format 2023-10-16T12:11:24.151 INFO:teuthology.orchestra.run.smithi049.stdout:mount.ceph: resolved to: "172.21.15.16:3300,172.21.15.49:3300,172.21.15.49:3301" 2023-10-16T12:11:24.151 INFO:teuthology.orchestra.run.smithi049.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T12:11:24.151 INFO:teuthology.orchestra.run.smithi049.stdout:mount.ceph: resolved to: "172.21.15.16:6789,172.21.15.49:6789,172.21.15.49:6790" 2023-10-16T12:11:24.152 INFO:teuthology.orchestra.run.smithi049.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.16:6789,172.21.15.49:6789,172.21.15.49:6790:/ 2023-10-16T12:11:24.152 INFO:teuthology.orchestra.run.smithi049.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs,key=3,fsid=5b28dfa1-b21f-474a-9b2e-9fb77af58254" will pass to kernel 2023-10-16T12:11:24.155 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-16T12:11:24.156 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:24.156 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2023-10-16T12:11:24.191 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:24.191 DEBUG:teuthology.orchestra.run.smithi049:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T12:11:24.263 INFO:teuthology.orchestra.run.smithi049.stdout:client4780 2023-10-16T12:11:24.264 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T12:11:24.264 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:11:24.321 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:24.322 DEBUG:teuthology.orchestra.run.smithi049:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T12:11:24.394 INFO:teuthology.orchestra.run.smithi049.stdout:client4780 2023-10-16T12:11:24.394 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:24.395 DEBUG:teuthology.orchestra.run.smithi049:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T12:11:24.466 INFO:teuthology.orchestra.run.smithi049.stdout:client4780 2023-10-16T12:11:24.466 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:24.467 DEBUG:teuthology.orchestra.run.smithi049:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/5b28dfa1-b21f-474a-9b2e-9fb77af58254.client4780/status') 2023-10-16T12:11:24.546 INFO:teuthology.orchestra.run.smithi049.stdout:instance: client.4780 (3)192.168.0.2:0/2115351174 2023-10-16T12:11:24.547 INFO:teuthology.orchestra.run.smithi049.stdout:blocklisted: false 2023-10-16T12:11:24.548 INFO:teuthology.orchestra.run.smithi049.stderr:0+1 records in 2023-10-16T12:11:24.548 INFO:teuthology.orchestra.run.smithi049.stderr:0+1 records out 2023-10-16T12:11:24.548 INFO:teuthology.orchestra.run.smithi049.stderr:69 bytes copied, 3.8936e-05 s, 1.8 MB/s 2023-10-16T12:11:24.549 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2023-10-16T12:11:24.564 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:11:24.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:24.701+0000 7f2369207700 1 Processor -- start 2023-10-16T12:11:24.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:24.701+0000 7f2369207700 1 -- start start 2023-10-16T12:11:24.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:24.702+0000 7f2369207700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f236415d230 0x7f23640b5a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:24.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:24.702+0000 7f2369207700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f236415bc80 con 0x7f23640cf880 2023-10-16T12:11:24.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:24.702+0000 7f2369207700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f236415b070 con 0x7f23640cf4a0 2023-10-16T12:11:24.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:24.702+0000 7f2369207700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f236415b350 con 0x7f236415d230 2023-10-16T12:11:24.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:24.702+0000 7f236359e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f236415d230 0x7f23640b5a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:24.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:24.702+0000 7f236259c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f23640cf880 0x7f23640b52f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51734/0 (socket says 172.21.15.16:51734) 2023-10-16T12:11:24.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:24.702+0000 7f236259c700 1 -- 172.21.15.16:0/4201990439 learned_addr learned my addr 172.21.15.16:0/4201990439 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:24.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:24.702+0000 7f2361d9b700 1 -- 172.21.15.16:0/4201990439 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2315647460 0 0) 0x7f236415bc80 con 0x7f23640cf880 2023-10-16T12:11:24.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:24.702+0000 7f2361d9b700 1 -- 172.21.15.16:0/4201990439 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f234c0035b0 con 0x7f23640cf880 2023-10-16T12:11:24.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:24.702+0000 7f236359e700 1 -- 172.21.15.16:0/4201990439 >> v1:172.21.15.49:6789/0 conn(0x7f23640cf4a0 legacy=0x7f23640726e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:24.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:24.702+0000 7f236359e700 1 -- 172.21.15.16:0/4201990439 >> v1:172.21.15.16:6789/0 conn(0x7f23640cf880 legacy=0x7f23640b52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:24.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:24.703+0000 7f236359e700 1 -- 172.21.15.16:0/4201990439 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f236415b630 con 0x7f236415d230 2023-10-16T12:11:24.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:24.703+0000 7f236359e700 1 --2- 172.21.15.16:0/4201990439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f236415d230 0x7f23640b5a00 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f235c009d90 tx=0x7f235c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84704c99bfab0443 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:24.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:24.703+0000 7f2361d9b700 1 -- 172.21.15.16:0/4201990439 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f235c00e070 con 0x7f236415d230 2023-10-16T12:11:24.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:24.703+0000 7f2361d9b700 1 -- 172.21.15.16:0/4201990439 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f235c00bb20 con 0x7f236415d230 2023-10-16T12:11:24.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:24.703+0000 7f2361d9b700 1 -- 172.21.15.16:0/4201990439 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f235c00be90 con 0x7f236415d230 2023-10-16T12:11:24.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:24.703+0000 7f2369207700 1 -- 172.21.15.16:0/4201990439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f236415d230 msgr2=0x7f23640b5a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:24.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:24.703+0000 7f2369207700 1 --2- 172.21.15.16:0/4201990439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f236415d230 0x7f23640b5a00 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f235c009d90 tx=0x7f235c00b330 comp rx=0 tx=0).stop 2023-10-16T12:11:24.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:24.703+0000 7f2369207700 1 -- 172.21.15.16:0/4201990439 shutdown_connections 2023-10-16T12:11:24.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:24.703+0000 7f2369207700 1 --2- 172.21.15.16:0/4201990439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f236415d230 0x7f23640b5a00 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:24.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:24.703+0000 7f2369207700 1 -- 172.21.15.16:0/4201990439 >> 172.21.15.16:0/4201990439 conn(0x7f23640b1fd0 msgr2=0x7f23640b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:24.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:24.703+0000 7f2369207700 1 -- 172.21.15.16:0/4201990439 shutdown_connections 2023-10-16T12:11:24.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:24.704+0000 7f2369207700 1 -- 172.21.15.16:0/4201990439 wait complete. 2023-10-16T12:11:24.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:24.705+0000 7f2369207700 1 Processor -- start 2023-10-16T12:11:24.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:24.705+0000 7f2369207700 1 -- start start 2023-10-16T12:11:24.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:24.706+0000 7f2369207700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23640cf4a0 0x7f2364159490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:24.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:24.706+0000 7f2369207700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f23640cf880 0x7f23640a9a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:24.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:24.706+0000 7f2369207700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f236415d230 0x7f23640a9f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:24.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:24.706+0000 7f2369207700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23640a95b0 con 0x7f23640cf4a0 2023-10-16T12:11:24.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:24.706+0000 7f2369207700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23640a89a0 con 0x7f236415d230 2023-10-16T12:11:24.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:24.706+0000 7f2369207700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23640a8c80 con 0x7f23640cf880 2023-10-16T12:11:24.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:24.706+0000 7f2362d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23640cf4a0 0x7f2364159490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:24.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:24.706+0000 7f2362d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23640cf4a0 0x7f2364159490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34232/0 (socket says 172.21.15.16:34232) 2023-10-16T12:11:24.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:24.706+0000 7f2362d9d700 1 -- 172.21.15.16:0/4241623117 learned_addr learned my addr 172.21.15.16:0/4241623117 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:24.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:24.706+0000 7f236359e700 1 --2- 172.21.15.16:0/4241623117 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f236415d230 0x7f23640a9f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:24.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:24.706+0000 7f236259c700 1 --2- 172.21.15.16:0/4241623117 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f23640cf880 0x7f23640a9a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:24.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:24.706+0000 7f2362d9d700 1 -- 172.21.15.16:0/4241623117 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f23640cf880 msgr2=0x7f23640a9a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:24.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:24.706+0000 7f2362d9d700 1 --2- 172.21.15.16:0/4241623117 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f23640cf880 0x7f23640a9a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:24.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:24.706+0000 7f2362d9d700 1 -- 172.21.15.16:0/4241623117 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f236415d230 msgr2=0x7f23640a9f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:24.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:24.706+0000 7f2362d9d700 1 --2- 172.21.15.16:0/4241623117 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f236415d230 0x7f23640a9f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:24.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:24.706+0000 7f2362d9d700 1 -- 172.21.15.16:0/4241623117 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f235c009a70 con 0x7f23640cf4a0 2023-10-16T12:11:24.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:24.706+0000 7f2362d9d700 1 --2- 172.21.15.16:0/4241623117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23640cf4a0 0x7f2364159490 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f235800c490 tx=0x7f235800c7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:24.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:24.707+0000 7f234b7fe700 1 -- 172.21.15.16:0/4241623117 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f235800cea0 con 0x7f23640cf4a0 2023-10-16T12:11:24.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:24.707+0000 7f234b7fe700 1 -- 172.21.15.16:0/4241623117 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23580052f0 con 0x7f23640cf4a0 2023-10-16T12:11:24.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:24.707+0000 7f2369207700 1 -- 172.21.15.16:0/4241623117 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23640a8f60 con 0x7f23640cf4a0 2023-10-16T12:11:24.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:24.707+0000 7f234b7fe700 1 -- 172.21.15.16:0/4241623117 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2358006860 con 0x7f23640cf4a0 2023-10-16T12:11:24.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:24.707+0000 7f2369207700 1 -- 172.21.15.16:0/4241623117 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23640a84c0 con 0x7f23640cf4a0 2023-10-16T12:11:24.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:24.708+0000 7f2369207700 1 -- 172.21.15.16:0/4241623117 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23640a9890 con 0x7f23640cf4a0 2023-10-16T12:11:24.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:24.708+0000 7f234b7fe700 1 -- 172.21.15.16:0/4241623117 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f2358003680 con 0x7f23640cf4a0 2023-10-16T12:11:24.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:24.708+0000 7f234b7fe700 1 --2- 172.21.15.16:0/4241623117 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f235006af00 0x7f235006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:24.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:24.709+0000 7f234b7fe700 1 -- 172.21.15.16:0/4241623117 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f2358088a50 con 0x7f23640cf4a0 2023-10-16T12:11:24.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:24.709+0000 7f236259c700 1 --2- 172.21.15.16:0/4241623117 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f235006af00 0x7f235006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:24.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:24.709+0000 7f236259c700 1 --2- 172.21.15.16:0/4241623117 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f235006af00 0x7f235006d3c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f23641599d0 tx=0x7f2354007dc0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:24.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:24.713+0000 7f234b7fe700 1 -- 172.21.15.16:0/4241623117 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2358053810 con 0x7f23640cf4a0 2023-10-16T12:11:24.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:24.916+0000 7f2369207700 1 -- 172.21.15.16:0/4241623117 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f236404ebb0 con 0x7f23640cf4a0 2023-10-16T12:11:24.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:24.917+0000 7f234b7fe700 1 -- 172.21.15.16:0/4241623117 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18238 (secure 0 0 0) 0x7f2358056e30 con 0x7f23640cf4a0 2023-10-16T12:11:24.919 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:11:24.920 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":25,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:10:44.080141+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:10:36.935773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T12:10:38.004870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-16T12:10:41.672025+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:11:24.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:24.919+0000 7f2369207700 1 -- 172.21.15.16:0/4241623117 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f235006af00 msgr2=0x7f235006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:24.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:24.919+0000 7f2369207700 1 --2- 172.21.15.16:0/4241623117 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f235006af00 0x7f235006d3c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f23641599d0 tx=0x7f2354007dc0 comp rx=0 tx=0).stop 2023-10-16T12:11:24.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:24.919+0000 7f2369207700 1 -- 172.21.15.16:0/4241623117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23640cf4a0 msgr2=0x7f2364159490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:24.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:24.919+0000 7f2369207700 1 --2- 172.21.15.16:0/4241623117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23640cf4a0 0x7f2364159490 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f235800c490 tx=0x7f235800c7a0 comp rx=0 tx=0).stop 2023-10-16T12:11:24.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:24.919+0000 7f2369207700 1 -- 172.21.15.16:0/4241623117 shutdown_connections 2023-10-16T12:11:24.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:24.919+0000 7f2369207700 1 --2- 172.21.15.16:0/4241623117 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f235006af00 0x7f235006d3c0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:24.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:24.919+0000 7f2369207700 1 --2- 172.21.15.16:0/4241623117 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f236415d230 0x7f23640a9f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:24.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:24.919+0000 7f2369207700 1 --2- 172.21.15.16:0/4241623117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23640cf4a0 0x7f2364159490 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:24.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:24.919+0000 7f2369207700 1 --2- 172.21.15.16:0/4241623117 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f23640cf880 0x7f23640a9a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:24.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:24.919+0000 7f2369207700 1 -- 172.21.15.16:0/4241623117 >> 172.21.15.16:0/4241623117 conn(0x7f23640b1fd0 msgr2=0x7f23640ce0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:24.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:24.920+0000 7f2369207700 1 -- 172.21.15.16:0/4241623117 shutdown_connections 2023-10-16T12:11:24.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:24.920+0000 7f2369207700 1 -- 172.21.15.16:0/4241623117 wait complete. 2023-10-16T12:11:24.934 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T12:10:36.935773+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T12:10:38.004870+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-16T12:10:41.672025+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-16T12:11:24.935 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:11:25.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.060+0000 7f30f0769700 1 Processor -- start 2023-10-16T12:11:25.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.060+0000 7f30f0769700 1 -- start start 2023-10-16T12:11:25.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.061+0000 7f30f0769700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f30e815ec80 0x7f30e80cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:25.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.061+0000 7f30f0769700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f30e815f060 0x7f30e80ce400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:25.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.061+0000 7f30f0769700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30e80b2440 con 0x7f30e80d2110 2023-10-16T12:11:25.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.061+0000 7f30f0769700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30e80b2c50 con 0x7f30e815f060 2023-10-16T12:11:25.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.061+0000 7f30f0769700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30e80d00e0 con 0x7f30e815ec80 2023-10-16T12:11:25.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.061+0000 7f30edd04700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f30e815f060 0x7f30e80ce400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:25.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.062+0000 7f30eed06700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f30e80d2110 0x7f30e80b4440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51748/0 (socket says 172.21.15.16:51748) 2023-10-16T12:11:25.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.062+0000 7f30eed06700 1 -- 172.21.15.16:0/3002682598 learned_addr learned my addr 172.21.15.16:0/3002682598 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:25.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.062+0000 7f30ee505700 1 --2- 172.21.15.16:0/3002682598 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f30e815ec80 0x7f30e80cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:25.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.062+0000 7f30edd04700 1 -- 172.21.15.16:0/3002682598 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f30e815ec80 msgr2=0x7f30e80cdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:25.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.062+0000 7f30edd04700 1 --2- 172.21.15.16:0/3002682598 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f30e815ec80 0x7f30e80cdec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:25.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.062+0000 7f30edd04700 1 -- 172.21.15.16:0/3002682598 >> v1:172.21.15.16:6789/0 conn(0x7f30e80d2110 legacy=0x7f30e80b4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:25.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.062+0000 7f30edd04700 1 -- 172.21.15.16:0/3002682598 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30e80cef60 con 0x7f30e815f060 2023-10-16T12:11:25.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.062+0000 7f30ed503700 1 -- 172.21.15.16:0/3002682598 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2276160079 0 0) 0x7f30e80b2440 con 0x7f30e80d2110 2023-10-16T12:11:25.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.062+0000 7f30edd04700 1 --2- 172.21.15.16:0/3002682598 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30e815f060 0x7f30e80ce400 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f30e4009840 tx=0x7f30e4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c57ee48a05a408f6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:25.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.062+0000 7f30ed503700 1 -- 172.21.15.16:0/3002682598 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30e4011830 con 0x7f30e815f060 2023-10-16T12:11:25.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.062+0000 7f30ed503700 1 -- 172.21.15.16:0/3002682598 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30e4011990 con 0x7f30e815f060 2023-10-16T12:11:25.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.062+0000 7f30ed503700 1 -- 172.21.15.16:0/3002682598 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30e4011d20 con 0x7f30e815f060 2023-10-16T12:11:25.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.063+0000 7f30f0769700 1 -- 172.21.15.16:0/3002682598 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30e815f060 msgr2=0x7f30e80ce400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:25.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.063+0000 7f30f0769700 1 --2- 172.21.15.16:0/3002682598 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30e815f060 0x7f30e80ce400 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f30e4009840 tx=0x7f30e4009b50 comp rx=0 tx=0).stop 2023-10-16T12:11:25.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.063+0000 7f30f0769700 1 -- 172.21.15.16:0/3002682598 shutdown_connections 2023-10-16T12:11:25.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.063+0000 7f30f0769700 1 --2- 172.21.15.16:0/3002682598 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f30e815ec80 0x7f30e80cdec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:25.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.063+0000 7f30f0769700 1 --2- 172.21.15.16:0/3002682598 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30e815f060 0x7f30e80ce400 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:25.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.063+0000 7f30f0769700 1 -- 172.21.15.16:0/3002682598 >> 172.21.15.16:0/3002682598 conn(0x7f30e80d15f0 msgr2=0x7f30e80d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:25.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.063+0000 7f30f0769700 1 -- 172.21.15.16:0/3002682598 shutdown_connections 2023-10-16T12:11:25.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.063+0000 7f30f0769700 1 -- 172.21.15.16:0/3002682598 wait complete. 2023-10-16T12:11:25.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.064+0000 7f30f0769700 1 Processor -- start 2023-10-16T12:11:25.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.064+0000 7f30f0769700 1 -- start start 2023-10-16T12:11:25.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.065+0000 7f30f0769700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30e80d2110 0x7f30e81526e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:25.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.065+0000 7f30f0769700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f30e815ec80 0x7f30e80c71c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:25.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.065+0000 7f30f0769700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f30e815f060 0x7f30e80c7700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:25.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.065+0000 7f30f0769700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30e8152c20 con 0x7f30e80d2110 2023-10-16T12:11:25.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.065+0000 7f30f0769700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30e80c5db0 con 0x7f30e815ec80 2023-10-16T12:11:25.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.065+0000 7f30f0769700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30e80c6090 con 0x7f30e815f060 2023-10-16T12:11:25.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.065+0000 7f30ee505700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30e80d2110 0x7f30e81526e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:25.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.065+0000 7f30ee505700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30e80d2110 0x7f30e81526e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34244/0 (socket says 172.21.15.16:34244) 2023-10-16T12:11:25.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.065+0000 7f30ee505700 1 -- 172.21.15.16:0/1335675091 learned_addr learned my addr 172.21.15.16:0/1335675091 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:25.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.065+0000 7f30edd04700 1 --2- 172.21.15.16:0/1335675091 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f30e815ec80 0x7f30e80c71c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:25.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.065+0000 7f30eed06700 1 --2- 172.21.15.16:0/1335675091 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f30e815f060 0x7f30e80c7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:25.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.065+0000 7f30ee505700 1 -- 172.21.15.16:0/1335675091 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f30e815f060 msgr2=0x7f30e80c7700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:25.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.065+0000 7f30ee505700 1 --2- 172.21.15.16:0/1335675091 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f30e815f060 0x7f30e80c7700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:25.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.065+0000 7f30ee505700 1 -- 172.21.15.16:0/1335675091 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f30e815ec80 msgr2=0x7f30e80c71c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:25.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.065+0000 7f30ee505700 1 --2- 172.21.15.16:0/1335675091 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f30e815ec80 0x7f30e80c71c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:25.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.065+0000 7f30ee505700 1 -- 172.21.15.16:0/1335675091 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30e40094a0 con 0x7f30e80d2110 2023-10-16T12:11:25.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.065+0000 7f30ee505700 1 --2- 172.21.15.16:0/1335675091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30e80d2110 0x7f30e81526e0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f30dc00e810 tx=0x7f30dc00eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:25.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.066+0000 7f30daffd700 1 -- 172.21.15.16:0/1335675091 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30dc00c070 con 0x7f30e80d2110 2023-10-16T12:11:25.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.066+0000 7f30daffd700 1 -- 172.21.15.16:0/1335675091 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30dc013610 con 0x7f30e80d2110 2023-10-16T12:11:25.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.066+0000 7f30daffd700 1 -- 172.21.15.16:0/1335675091 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30dc0139c0 con 0x7f30e80d2110 2023-10-16T12:11:25.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.066+0000 7f30f0769700 1 -- 172.21.15.16:0/1335675091 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30e80c6370 con 0x7f30e80d2110 2023-10-16T12:11:25.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.066+0000 7f30f0769700 1 -- 172.21.15.16:0/1335675091 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30e80c8a00 con 0x7f30e80d2110 2023-10-16T12:11:25.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.066+0000 7f30f0769700 1 -- 172.21.15.16:0/1335675091 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30e8159ee0 con 0x7f30e80d2110 2023-10-16T12:11:25.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.067+0000 7f30daffd700 1 -- 172.21.15.16:0/1335675091 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f30dc013b20 con 0x7f30e80d2110 2023-10-16T12:11:25.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.067+0000 7f30daffd700 1 --2- 172.21.15.16:0/1335675091 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f30cc06af00 0x7f30cc06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:25.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.068+0000 7f30daffd700 1 -- 172.21.15.16:0/1335675091 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f30dc0896b0 con 0x7f30e80d2110 2023-10-16T12:11:25.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.068+0000 7f30edd04700 1 --2- 172.21.15.16:0/1335675091 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f30cc06af00 0x7f30cc06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:25.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.068+0000 7f30edd04700 1 --2- 172.21.15.16:0/1335675091 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f30cc06af00 0x7f30cc06d3c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f30e815f460 tx=0x7f30e401c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:25.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.072+0000 7f30daffd700 1 -- 172.21.15.16:0/1335675091 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f30dc054480 con 0x7f30e80d2110 2023-10-16T12:11:25.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.274+0000 7f30f0769700 1 -- 172.21.15.16:0/1335675091 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f30e80c69c0 con 0x7f30e80d2110 2023-10-16T12:11:25.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.275+0000 7f30daffd700 1 -- 172.21.15.16:0/1335675091 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) v1 ==== 93+0+10 (secure 0 0 0) 0x7f30dc057aa0 con 0x7f30e80d2110 2023-10-16T12:11:25.277 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T12:11:25.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.277+0000 7f30f0769700 1 -- 172.21.15.16:0/1335675091 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f30cc06af00 msgr2=0x7f30cc06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:25.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.277+0000 7f30f0769700 1 --2- 172.21.15.16:0/1335675091 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f30cc06af00 0x7f30cc06d3c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f30e815f460 tx=0x7f30e401c040 comp rx=0 tx=0).stop 2023-10-16T12:11:25.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.277+0000 7f30f0769700 1 -- 172.21.15.16:0/1335675091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30e80d2110 msgr2=0x7f30e81526e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:25.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.277+0000 7f30f0769700 1 --2- 172.21.15.16:0/1335675091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30e80d2110 0x7f30e81526e0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f30dc00e810 tx=0x7f30dc00eb20 comp rx=0 tx=0).stop 2023-10-16T12:11:25.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.277+0000 7f30f0769700 1 -- 172.21.15.16:0/1335675091 shutdown_connections 2023-10-16T12:11:25.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.277+0000 7f30f0769700 1 --2- 172.21.15.16:0/1335675091 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f30cc06af00 0x7f30cc06d3c0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:25.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.277+0000 7f30f0769700 1 --2- 172.21.15.16:0/1335675091 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f30e815f060 0x7f30e80c7700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:25.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.277+0000 7f30f0769700 1 --2- 172.21.15.16:0/1335675091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30e80d2110 0x7f30e81526e0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:25.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.277+0000 7f30f0769700 1 --2- 172.21.15.16:0/1335675091 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f30e815ec80 0x7f30e80c71c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:25.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.277+0000 7f30f0769700 1 -- 172.21.15.16:0/1335675091 >> 172.21.15.16:0/1335675091 conn(0x7f30e80d15f0 msgr2=0x7f30e80b4510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:25.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.277+0000 7f30f0769700 1 -- 172.21.15.16:0/1335675091 shutdown_connections 2023-10-16T12:11:25.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.277+0000 7f30f0769700 1 -- 172.21.15.16:0/1335675091 wait complete. 2023-10-16T12:11:25.291 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T12:11:25.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.418+0000 7f3f24c15700 1 Processor -- start 2023-10-16T12:11:25.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.418+0000 7f3f24c15700 1 -- start start 2023-10-16T12:11:25.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.419+0000 7f3f24c15700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3f2015b2c0 0x7f3f20161340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:25.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.419+0000 7f3f24c15700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f2015ecb0 con 0x7f3f2006e570 2023-10-16T12:11:25.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.419+0000 7f3f24c15700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f20161880 con 0x7f3f2006a7e0 2023-10-16T12:11:25.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.419+0000 7f3f24c15700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f2015b6a0 con 0x7f3f2015b2c0 2023-10-16T12:11:25.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.419+0000 7f3f1e59c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3f2015b2c0 0x7f3f20161340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:25.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.419+0000 7f3f1dd9b700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f3f2006e570 0x7f3f2015e5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51760/0 (socket says 172.21.15.16:51760) 2023-10-16T12:11:25.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.419+0000 7f3f1dd9b700 1 -- 172.21.15.16:0/3382480807 learned_addr learned my addr 172.21.15.16:0/3382480807 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:25.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.419+0000 7f3f1e59c700 1 -- 172.21.15.16:0/3382480807 >> v1:172.21.15.49:6789/0 conn(0x7f3f2006a7e0 legacy=0x7f3f20071a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:25.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.419+0000 7f3f1e59c700 1 -- 172.21.15.16:0/3382480807 >> v1:172.21.15.16:6789/0 conn(0x7f3f2006e570 legacy=0x7f3f2015e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:25.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.419+0000 7f3f1e59c700 1 -- 172.21.15.16:0/3382480807 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f20161b20 con 0x7f3f2015b2c0 2023-10-16T12:11:25.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.419+0000 7f3f1e59c700 1 --2- 172.21.15.16:0/3382480807 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f2015b2c0 0x7f3f20161340 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f3f10009840 tx=0x7f3f10009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3ea9de1709ebd03 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:25.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.420+0000 7f3f1d59a700 1 -- 172.21.15.16:0/3382480807 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f10004230 con 0x7f3f2015b2c0 2023-10-16T12:11:25.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.420+0000 7f3f1d59a700 1 -- 172.21.15.16:0/3382480807 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f10004390 con 0x7f3f2015b2c0 2023-10-16T12:11:25.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.420+0000 7f3f1d59a700 1 -- 172.21.15.16:0/3382480807 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f100036a0 con 0x7f3f2015b2c0 2023-10-16T12:11:25.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.420+0000 7f3f24c15700 1 -- 172.21.15.16:0/3382480807 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f2015b2c0 msgr2=0x7f3f20161340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:25.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.420+0000 7f3f24c15700 1 --2- 172.21.15.16:0/3382480807 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f2015b2c0 0x7f3f20161340 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f3f10009840 tx=0x7f3f10009b50 comp rx=0 tx=0).stop 2023-10-16T12:11:25.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.420+0000 7f3f24c15700 1 -- 172.21.15.16:0/3382480807 shutdown_connections 2023-10-16T12:11:25.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.420+0000 7f3f24c15700 1 --2- 172.21.15.16:0/3382480807 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f2015b2c0 0x7f3f20161340 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:25.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.420+0000 7f3f24c15700 1 -- 172.21.15.16:0/3382480807 >> 172.21.15.16:0/3382480807 conn(0x7f3f200b1f40 msgr2=0x7f3f200b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:25.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.420+0000 7f3f24c15700 1 -- 172.21.15.16:0/3382480807 shutdown_connections 2023-10-16T12:11:25.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.420+0000 7f3f24c15700 1 -- 172.21.15.16:0/3382480807 wait complete. 2023-10-16T12:11:25.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.422+0000 7f3f24c15700 1 Processor -- start 2023-10-16T12:11:25.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.422+0000 7f3f24c15700 1 -- start start 2023-10-16T12:11:25.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.422+0000 7f3f24c15700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3f2006a7e0 0x7f3f200c6050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:25.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.422+0000 7f3f24c15700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3f2006e570 0x7f3f200c74c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:25.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.422+0000 7f3f24c15700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f2015b2c0 0x7f3f200c7a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:25.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.422+0000 7f3f24c15700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f20159bd0 con 0x7f3f2015b2c0 2023-10-16T12:11:25.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.422+0000 7f3f24c15700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f200c6590 con 0x7f3f2006e570 2023-10-16T12:11:25.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.422+0000 7f3f24c15700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f201598f0 con 0x7f3f2006a7e0 2023-10-16T12:11:25.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.423+0000 7f3f1ed9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f2015b2c0 0x7f3f200c7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:25.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.423+0000 7f3f1ed9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f2015b2c0 0x7f3f200c7a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34262/0 (socket says 172.21.15.16:34262) 2023-10-16T12:11:25.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.423+0000 7f3f1ed9d700 1 -- 172.21.15.16:0/2033470362 learned_addr learned my addr 172.21.15.16:0/2033470362 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:25.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.423+0000 7f3f1dd9b700 1 --2- 172.21.15.16:0/2033470362 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3f2006e570 0x7f3f200c74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:25.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.423+0000 7f3f1e59c700 1 --2- 172.21.15.16:0/2033470362 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3f2006a7e0 0x7f3f200c6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:25.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.423+0000 7f3f1ed9d700 1 -- 172.21.15.16:0/2033470362 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3f2006a7e0 msgr2=0x7f3f200c6050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:25.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.423+0000 7f3f1ed9d700 1 --2- 172.21.15.16:0/2033470362 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3f2006a7e0 0x7f3f200c6050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:25.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.423+0000 7f3f1ed9d700 1 -- 172.21.15.16:0/2033470362 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3f2006e570 msgr2=0x7f3f200c74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:25.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.423+0000 7f3f1ed9d700 1 --2- 172.21.15.16:0/2033470362 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3f2006e570 0x7f3f200c74c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:25.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.423+0000 7f3f1ed9d700 1 -- 172.21.15.16:0/2033470362 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f100094a0 con 0x7f3f2015b2c0 2023-10-16T12:11:25.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.423+0000 7f3f1ed9d700 1 --2- 172.21.15.16:0/2033470362 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f2015b2c0 0x7f3f200c7a00 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f3f1800b970 tx=0x7f3f1800bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:25.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.423+0000 7f3f0effd700 1 -- 172.21.15.16:0/2033470362 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f1800c9b0 con 0x7f3f2015b2c0 2023-10-16T12:11:25.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.423+0000 7f3f0effd700 1 -- 172.21.15.16:0/2033470362 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f1800cb10 con 0x7f3f2015b2c0 2023-10-16T12:11:25.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.423+0000 7f3f24c15700 1 -- 172.21.15.16:0/2033470362 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f200b36f0 con 0x7f3f2015b2c0 2023-10-16T12:11:25.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.423+0000 7f3f0effd700 1 -- 172.21.15.16:0/2033470362 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f18014440 con 0x7f3f2015b2c0 2023-10-16T12:11:25.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.423+0000 7f3f24c15700 1 -- 172.21.15.16:0/2033470362 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f20158bc0 con 0x7f3f2015b2c0 2023-10-16T12:11:25.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.424+0000 7f3f24c15700 1 -- 172.21.15.16:0/2033470362 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f200b3550 con 0x7f3f2015b2c0 2023-10-16T12:11:25.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.425+0000 7f3f0effd700 1 -- 172.21.15.16:0/2033470362 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f3f1801d3b0 con 0x7f3f2015b2c0 2023-10-16T12:11:25.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.425+0000 7f3f0effd700 1 --2- 172.21.15.16:0/2033470362 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f3f0006af00 0x7f3f0006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:25.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.425+0000 7f3f0effd700 1 -- 172.21.15.16:0/2033470362 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f3f180894b0 con 0x7f3f2015b2c0 2023-10-16T12:11:25.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.425+0000 7f3f1e59c700 1 --2- 172.21.15.16:0/2033470362 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f3f0006af00 0x7f3f0006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:25.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.426+0000 7f3f1e59c700 1 --2- 172.21.15.16:0/2033470362 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f3f0006af00 0x7f3f0006d3c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f3f200b5f10 tx=0x7f3f10008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:25.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.430+0000 7f3f0effd700 1 -- 172.21.15.16:0/2033470362 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3f200b3550 con 0x7f3f2015b2c0 2023-10-16T12:11:25.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.637+0000 7f3f24c15700 1 -- 172.21.15.16:0/2033470362 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3f200b3ef0 con 0x7f3f2015b2c0 2023-10-16T12:11:25.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.637+0000 7f3f0effd700 1 -- 172.21.15.16:0/2033470362 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3f200b3ef0 con 0x7f3f2015b2c0 2023-10-16T12:11:25.640 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-10-16T12:11:25.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.640+0000 7f3f24c15700 1 -- 172.21.15.16:0/2033470362 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f3f0006af00 msgr2=0x7f3f0006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:25.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.640+0000 7f3f24c15700 1 --2- 172.21.15.16:0/2033470362 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f3f0006af00 0x7f3f0006d3c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f3f200b5f10 tx=0x7f3f10008030 comp rx=0 tx=0).stop 2023-10-16T12:11:25.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.640+0000 7f3f24c15700 1 -- 172.21.15.16:0/2033470362 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f2015b2c0 msgr2=0x7f3f200c7a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:25.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.640+0000 7f3f24c15700 1 --2- 172.21.15.16:0/2033470362 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f2015b2c0 0x7f3f200c7a00 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f3f1800b970 tx=0x7f3f1800bd30 comp rx=0 tx=0).stop 2023-10-16T12:11:25.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.640+0000 7f3f24c15700 1 -- 172.21.15.16:0/2033470362 shutdown_connections 2023-10-16T12:11:25.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.640+0000 7f3f24c15700 1 --2- 172.21.15.16:0/2033470362 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f3f0006af00 0x7f3f0006d3c0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:25.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.640+0000 7f3f24c15700 1 --2- 172.21.15.16:0/2033470362 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f2015b2c0 0x7f3f200c7a00 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:25.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.640+0000 7f3f24c15700 1 --2- 172.21.15.16:0/2033470362 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3f2006a7e0 0x7f3f200c6050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:25.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.640+0000 7f3f24c15700 1 --2- 172.21.15.16:0/2033470362 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3f2006e570 0x7f3f200c74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:25.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.640+0000 7f3f24c15700 1 -- 172.21.15.16:0/2033470362 >> 172.21.15.16:0/2033470362 conn(0x7f3f200b1f40 msgr2=0x7f3f2015b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:25.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.640+0000 7f3f24c15700 1 -- 172.21.15.16:0/2033470362 shutdown_connections 2023-10-16T12:11:25.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.640+0000 7f3f24c15700 1 -- 172.21.15.16:0/2033470362 wait complete. 2023-10-16T12:11:25.654 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T12:11:25.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.780+0000 7fb4becb2700 1 Processor -- start 2023-10-16T12:11:25.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.780+0000 7fb4becb2700 1 -- start start 2023-10-16T12:11:25.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.781+0000 7fb4becb2700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fb4b806e570 0x7fb4b815e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:25.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.781+0000 7fb4becb2700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb4b806a7e0 0x7fb4b806ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:25.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.781+0000 7fb4becb2700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4b815b6a0 con 0x7fb4b815b2c0 2023-10-16T12:11:25.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.781+0000 7fb4becb2700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4b806b1b0 con 0x7fb4b806e570 2023-10-16T12:11:25.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.781+0000 7fb4becb2700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b8161b20 con 0x7fb4b806a7e0 2023-10-16T12:11:25.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.781+0000 7fb4bca4e700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb4b815b2c0 0x7fb4b8161340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51772/0 (socket says 172.21.15.16:51772) 2023-10-16T12:11:25.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.781+0000 7fb4bca4e700 1 -- 172.21.15.16:0/1370052476 learned_addr learned my addr 172.21.15.16:0/1370052476 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:25.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.781+0000 7fb4b7fff700 1 --2- 172.21.15.16:0/1370052476 >> v2:172.21.15.49:3300/0 conn(0x7fb4b806e570 0x7fb4b815e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:25.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.781+0000 7fb4bd24f700 1 --2- 172.21.15.16:0/1370052476 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb4b806a7e0 0x7fb4b806ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:25.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.781+0000 7fb4b77fe700 1 -- 172.21.15.16:0/1370052476 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1661521438 0 0) 0x7fb4b815b6a0 con 0x7fb4b815b2c0 2023-10-16T12:11:25.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.781+0000 7fb4b77fe700 1 -- 172.21.15.16:0/1370052476 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4a80035b0 con 0x7fb4b815b2c0 2023-10-16T12:11:25.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.781+0000 7fb4b77fe700 1 -- 172.21.15.16:0/1370052476 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1774279201 0 0) 0x7fb4a80035b0 con 0x7fb4b815b2c0 2023-10-16T12:11:25.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.781+0000 7fb4b77fe700 1 -- 172.21.15.16:0/1370052476 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb4b806a7e0 msgr2=0x7fb4b806ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:25.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.781+0000 7fb4b77fe700 1 --2- 172.21.15.16:0/1370052476 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb4b806a7e0 0x7fb4b806ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:25.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.781+0000 7fb4b77fe700 1 -- 172.21.15.16:0/1370052476 >> v2:172.21.15.49:3300/0 conn(0x7fb4b806e570 msgr2=0x7fb4b815e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:25.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.781+0000 7fb4b77fe700 1 --2- 172.21.15.16:0/1370052476 >> v2:172.21.15.49:3300/0 conn(0x7fb4b806e570 0x7fb4b815e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:25.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.781+0000 7fb4b77fe700 1 -- 172.21.15.16:0/1370052476 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4b8161e00 con 0x7fb4b815b2c0 2023-10-16T12:11:25.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.781+0000 7fb4b77fe700 1 -- 172.21.15.16:0/1370052476 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7fb4ac001670 con 0x7fb4b815b2c0 2023-10-16T12:11:25.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.781+0000 7fb4b77fe700 1 -- 172.21.15.16:0/1370052476 >> v1:172.21.15.16:6789/0 conn(0x7fb4b815b2c0 legacy=0x7fb4b8161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:25.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.782+0000 7fb4b77fe700 1 --2- 172.21.15.16:0/1370052476 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb4a8003a00 0x7fb4a8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:25.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.782+0000 7fb4b77fe700 1 --2- 172.21.15.16:0/1370052476 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb4b806a7e0 0x7fb4a8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:25.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.782+0000 7fb4b77fe700 1 --2- 172.21.15.16:0/1370052476 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4a8004d90 0x7fb4a8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:25.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.782+0000 7fb4b77fe700 1 -- 172.21.15.16:0/1370052476 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b806b1b0 con 0x7fb4a8004d90 2023-10-16T12:11:25.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.782+0000 7fb4b77fe700 1 -- 172.21.15.16:0/1370052476 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b8161b20 con 0x7fb4a8003a00 2023-10-16T12:11:25.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.782+0000 7fb4b77fe700 1 -- 172.21.15.16:0/1370052476 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a800ba10 con 0x7fb4b806a7e0 2023-10-16T12:11:25.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.782+0000 7fb4bd24f700 1 --2- 172.21.15.16:0/1370052476 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4a8004d90 0x7fb4a8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:25.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.782+0000 7fb4bca4e700 1 --2- 172.21.15.16:0/1370052476 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb4a8003a00 0x7fb4a8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:25.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.782+0000 7fb4b7fff700 1 --2- 172.21.15.16:0/1370052476 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb4b806a7e0 0x7fb4a8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:25.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.782+0000 7fb4bd24f700 1 -- 172.21.15.16:0/1370052476 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb4b806a7e0 msgr2=0x7fb4a8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:25.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.782+0000 7fb4bd24f700 1 --2- 172.21.15.16:0/1370052476 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb4b806a7e0 0x7fb4a8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:25.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.782+0000 7fb4bd24f700 1 -- 172.21.15.16:0/1370052476 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb4a8003a00 msgr2=0x7fb4a8003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:25.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.782+0000 7fb4bd24f700 1 --2- 172.21.15.16:0/1370052476 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb4a8003a00 0x7fb4a8003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:25.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.782+0000 7fb4bd24f700 1 -- 172.21.15.16:0/1370052476 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4a80035b0 con 0x7fb4a8004d90 2023-10-16T12:11:25.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.782+0000 7fb4bd24f700 1 --2- 172.21.15.16:0/1370052476 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4a8004d90 0x7fb4a8005210 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fb4b0009fc0 tx=0x7fb4a8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff0abda7382b60aa server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:25.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.782+0000 7fb4b77fe700 1 -- 172.21.15.16:0/1370052476 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4b000ec30 con 0x7fb4a8004d90 2023-10-16T12:11:25.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.782+0000 7fb4b77fe700 1 -- 172.21.15.16:0/1370052476 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4b000f040 con 0x7fb4a8004d90 2023-10-16T12:11:25.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.782+0000 7fb4b77fe700 1 -- 172.21.15.16:0/1370052476 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4b0014480 con 0x7fb4a8004d90 2023-10-16T12:11:25.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.783+0000 7fb4becb2700 1 -- 172.21.15.16:0/1370052476 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4a8004d90 msgr2=0x7fb4a8005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:25.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.783+0000 7fb4becb2700 1 --2- 172.21.15.16:0/1370052476 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4a8004d90 0x7fb4a8005210 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fb4b0009fc0 tx=0x7fb4a8005880 comp rx=0 tx=0).stop 2023-10-16T12:11:25.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.783+0000 7fb4bd24f700 1 -- 172.21.15.16:0/1370052476 reap_dead start 2023-10-16T12:11:25.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.783+0000 7fb4becb2700 1 -- 172.21.15.16:0/1370052476 shutdown_connections 2023-10-16T12:11:25.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.783+0000 7fb4becb2700 1 -- 172.21.15.16:0/1370052476 >> 172.21.15.16:0/1370052476 conn(0x7fb4b80b1f40 msgr2=0x7fb4b80b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:25.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.783+0000 7fb4becb2700 1 -- 172.21.15.16:0/1370052476 shutdown_connections 2023-10-16T12:11:25.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.783+0000 7fb4becb2700 1 -- 172.21.15.16:0/1370052476 wait complete. 2023-10-16T12:11:25.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.784+0000 7fb4becb2700 1 Processor -- start 2023-10-16T12:11:25.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.784+0000 7fb4becb2700 1 -- start start 2023-10-16T12:11:25.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.785+0000 7fb4becb2700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb4b80b4a40 0x7fb4b80c9930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:25.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.785+0000 7fb4becb2700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb4b80b56a0 0x7fb4b80b5b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:25.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.785+0000 7fb4becb2700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4b80b35e0 0x7fb4b80b3a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:25.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.785+0000 7fb4becb2700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b81509f0 con 0x7fb4b80b35e0 2023-10-16T12:11:25.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.785+0000 7fb4becb2700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b80c8f20 con 0x7fb4b80b4a40 2023-10-16T12:11:25.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.785+0000 7fb4becb2700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b80b6060 con 0x7fb4b80b56a0 2023-10-16T12:11:25.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.785+0000 7fb4bd24f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4b80b35e0 0x7fb4b80b3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:25.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.785+0000 7fb4bd24f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4b80b35e0 0x7fb4b80b3a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34282/0 (socket says 172.21.15.16:34282) 2023-10-16T12:11:25.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.785+0000 7fb4bd24f700 1 -- 172.21.15.16:0/612606350 learned_addr learned my addr 172.21.15.16:0/612606350 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:25.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.785+0000 7fb4bca4e700 1 --2- 172.21.15.16:0/612606350 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb4b80b4a40 0x7fb4b80c9930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:25.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.785+0000 7fb4b7fff700 1 --2- 172.21.15.16:0/612606350 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb4b80b56a0 0x7fb4b80b5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:25.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.785+0000 7fb4bd24f700 1 -- 172.21.15.16:0/612606350 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb4b80b56a0 msgr2=0x7fb4b80b5b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:25.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.786+0000 7fb4bd24f700 1 --2- 172.21.15.16:0/612606350 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb4b80b56a0 0x7fb4b80b5b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:25.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.786+0000 7fb4bd24f700 1 -- 172.21.15.16:0/612606350 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb4b80b4a40 msgr2=0x7fb4b80c9930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:25.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.786+0000 7fb4bd24f700 1 --2- 172.21.15.16:0/612606350 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb4b80b4a40 0x7fb4b80c9930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:25.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.786+0000 7fb4bd24f700 1 -- 172.21.15.16:0/612606350 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4b8150150 con 0x7fb4b80b35e0 2023-10-16T12:11:25.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.786+0000 7fb4bd24f700 1 --2- 172.21.15.16:0/612606350 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4b80b35e0 0x7fb4b80b3a60 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fb4b0016000 tx=0x7fb4b0009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:25.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.786+0000 7fb4b57fa700 1 -- 172.21.15.16:0/612606350 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4b0007000 con 0x7fb4b80b35e0 2023-10-16T12:11:25.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.786+0000 7fb4b57fa700 1 -- 172.21.15.16:0/612606350 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4b0022070 con 0x7fb4b80b35e0 2023-10-16T12:11:25.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.786+0000 7fb4becb2700 1 -- 172.21.15.16:0/612606350 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4b8150430 con 0x7fb4b80b35e0 2023-10-16T12:11:25.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.786+0000 7fb4b57fa700 1 -- 172.21.15.16:0/612606350 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4b0029480 con 0x7fb4b80b35e0 2023-10-16T12:11:25.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.786+0000 7fb4becb2700 1 -- 172.21.15.16:0/612606350 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4b80c9130 con 0x7fb4b80b35e0 2023-10-16T12:11:25.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.787+0000 7fb4becb2700 1 -- 172.21.15.16:0/612606350 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4b804ebb0 con 0x7fb4b80b35e0 2023-10-16T12:11:25.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.788+0000 7fb4b57fa700 1 -- 172.21.15.16:0/612606350 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fb4b001e430 con 0x7fb4b80b35e0 2023-10-16T12:11:25.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.788+0000 7fb4b57fa700 1 --2- 172.21.15.16:0/612606350 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fb49c06af00 0x7fb49c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:25.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.788+0000 7fb4b57fa700 1 -- 172.21.15.16:0/612606350 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fb4b0024070 con 0x7fb4b80b35e0 2023-10-16T12:11:25.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.788+0000 7fb4bca4e700 1 --2- 172.21.15.16:0/612606350 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fb49c06af00 0x7fb49c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:25.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.788+0000 7fb4bca4e700 1 --2- 172.21.15.16:0/612606350 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fb49c06af00 0x7fb49c06d3c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac0048d0 tx=0x7fb4ac009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:25.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.793+0000 7fb4b57fa700 1 -- 172.21.15.16:0/612606350 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb4b00563b0 con 0x7fb4b80b35e0 2023-10-16T12:11:25.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.993+0000 7fb4becb2700 1 -- 172.21.15.16:0/612606350 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb4b80b6370 con 0x7fb4b80b35e0 2023-10-16T12:11:25.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.994+0000 7fb4b57fa700 1 -- 172.21.15.16:0/612606350 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb4b00599d0 con 0x7fb4b80b35e0 2023-10-16T12:11:25.996 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-10-16T12:11:25.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.996+0000 7fb4becb2700 1 -- 172.21.15.16:0/612606350 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fb49c06af00 msgr2=0x7fb49c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:25.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.996+0000 7fb4becb2700 1 --2- 172.21.15.16:0/612606350 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fb49c06af00 0x7fb49c06d3c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac0048d0 tx=0x7fb4ac009040 comp rx=0 tx=0).stop 2023-10-16T12:11:25.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.996+0000 7fb4becb2700 1 -- 172.21.15.16:0/612606350 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4b80b35e0 msgr2=0x7fb4b80b3a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:25.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.996+0000 7fb4becb2700 1 --2- 172.21.15.16:0/612606350 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4b80b35e0 0x7fb4b80b3a60 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fb4b0016000 tx=0x7fb4b0009750 comp rx=0 tx=0).stop 2023-10-16T12:11:25.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.996+0000 7fb4becb2700 1 -- 172.21.15.16:0/612606350 shutdown_connections 2023-10-16T12:11:25.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.996+0000 7fb4becb2700 1 --2- 172.21.15.16:0/612606350 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fb49c06af00 0x7fb49c06d3c0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:25.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.996+0000 7fb4becb2700 1 --2- 172.21.15.16:0/612606350 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4b80b35e0 0x7fb4b80b3a60 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:25.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.996+0000 7fb4becb2700 1 --2- 172.21.15.16:0/612606350 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb4b80b4a40 0x7fb4b80c9930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:25.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.996+0000 7fb4becb2700 1 --2- 172.21.15.16:0/612606350 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb4b80b56a0 0x7fb4b80b5b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:26.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.996+0000 7fb4becb2700 1 -- 172.21.15.16:0/612606350 >> 172.21.15.16:0/612606350 conn(0x7fb4b80b1f40 msgr2=0x7fb4b80cc240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:26.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.996+0000 7fb4becb2700 1 -- 172.21.15.16:0/612606350 shutdown_connections 2023-10-16T12:11:26.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:25.996+0000 7fb4becb2700 1 -- 172.21.15.16:0/612606350 wait complete. 2023-10-16T12:11:26.010 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-16T12:11:26.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.136+0000 7f11ef84e700 1 Processor -- start 2023-10-16T12:11:26.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.136+0000 7f11ef84e700 1 -- start start 2023-10-16T12:11:26.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.137+0000 7f11ef84e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f11e80cb8d0 0x7f11e8164dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:26.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.137+0000 7f11ef84e700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11e815dda0 con 0x7f11e8159de0 2023-10-16T12:11:26.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.137+0000 7f11ef84e700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11e815e080 con 0x7f11e80cf4c0 2023-10-16T12:11:26.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.137+0000 7f11ef84e700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11e815ea20 con 0x7f11e80cb8d0 2023-10-16T12:11:26.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.137+0000 7f11ed5ea700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f11e8159de0 0x7f11e815a1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51792/0 (socket says 172.21.15.16:51792) 2023-10-16T12:11:26.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.137+0000 7f11ed5ea700 1 -- 172.21.15.16:0/4052721673 learned_addr learned my addr 172.21.15.16:0/4052721673 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:26.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.137+0000 7f11dffff700 1 -- 172.21.15.16:0/4052721673 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3880594950 0 0) 0x7f11e815dda0 con 0x7f11e8159de0 2023-10-16T12:11:26.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.137+0000 7f11eddeb700 1 --2- 172.21.15.16:0/4052721673 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f11e80cb8d0 0x7f11e8164dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:26.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.137+0000 7f11dffff700 1 -- 172.21.15.16:0/4052721673 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11d40035b0 con 0x7f11e8159de0 2023-10-16T12:11:26.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.137+0000 7f11dffff700 1 -- 172.21.15.16:0/4052721673 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2251162146 0 0) 0x7f11d40035b0 con 0x7f11e8159de0 2023-10-16T12:11:26.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.137+0000 7f11dffff700 1 -- 172.21.15.16:0/4052721673 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f11e80cb8d0 msgr2=0x7f11e8164dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:26.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.137+0000 7f11dffff700 1 --2- 172.21.15.16:0/4052721673 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f11e80cb8d0 0x7f11e8164dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:26.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.137+0000 7f11dffff700 1 -- 172.21.15.16:0/4052721673 >> v1:172.21.15.49:6789/0 conn(0x7f11e80cf4c0 legacy=0x7f11e80ca1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:26.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.137+0000 7f11dffff700 1 -- 172.21.15.16:0/4052721673 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11e815e360 con 0x7f11e8159de0 2023-10-16T12:11:26.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.137+0000 7f11dffff700 1 -- 172.21.15.16:0/4052721673 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f11d8001670 con 0x7f11e8159de0 2023-10-16T12:11:26.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.137+0000 7f11dffff700 1 -- 172.21.15.16:0/4052721673 >> v1:172.21.15.16:6789/0 conn(0x7f11e8159de0 legacy=0x7f11e815a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:26.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.137+0000 7f11dffff700 1 --2- 172.21.15.16:0/4052721673 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f11d4003a30 0x7f11d4003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:26.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.137+0000 7f11dffff700 1 --2- 172.21.15.16:0/4052721673 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11d4004850 0x7f11d4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:26.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.137+0000 7f11dffff700 1 --2- 172.21.15.16:0/4052721673 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f11e80cb8d0 0x7f11d4005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:26.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.138+0000 7f11dffff700 1 -- 172.21.15.16:0/4052721673 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11e815ea20 con 0x7f11d4004850 2023-10-16T12:11:26.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.138+0000 7f11dffff700 1 -- 172.21.15.16:0/4052721673 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11d400bb40 con 0x7f11d4003a30 2023-10-16T12:11:26.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.138+0000 7f11dffff700 1 -- 172.21.15.16:0/4052721673 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11d400bd00 con 0x7f11e80cb8d0 2023-10-16T12:11:26.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.138+0000 7f11ecde9700 1 --2- 172.21.15.16:0/4052721673 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11d4004850 0x7f11d4004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:26.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.138+0000 7f11ed5ea700 1 --2- 172.21.15.16:0/4052721673 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f11d4003a30 0x7f11d4003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:26.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.138+0000 7f11eddeb700 1 --2- 172.21.15.16:0/4052721673 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f11e80cb8d0 0x7f11d4005250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:26.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.138+0000 7f11ecde9700 1 -- 172.21.15.16:0/4052721673 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f11e80cb8d0 msgr2=0x7f11d4005250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:26.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.138+0000 7f11ecde9700 1 --2- 172.21.15.16:0/4052721673 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f11e80cb8d0 0x7f11d4005250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:26.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.138+0000 7f11ecde9700 1 -- 172.21.15.16:0/4052721673 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f11d4003a30 msgr2=0x7f11d4003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:26.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.138+0000 7f11ecde9700 1 --2- 172.21.15.16:0/4052721673 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f11d4003a30 0x7f11d4003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:26.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.138+0000 7f11ecde9700 1 -- 172.21.15.16:0/4052721673 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11d40035b0 con 0x7f11d4004850 2023-10-16T12:11:26.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.138+0000 7f11ecde9700 1 --2- 172.21.15.16:0/4052721673 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11d4004850 0x7f11d4004cd0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f11e80cf930 tx=0x7f11d4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3075fd81e2d58296 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:26.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.138+0000 7f11dffff700 1 -- 172.21.15.16:0/4052721673 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11e40012f0 con 0x7f11d4004850 2023-10-16T12:11:26.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.138+0000 7f11dffff700 1 -- 172.21.15.16:0/4052721673 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11e40052f0 con 0x7f11d4004850 2023-10-16T12:11:26.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.138+0000 7f11dffff700 1 -- 172.21.15.16:0/4052721673 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11e4006860 con 0x7f11d4004850 2023-10-16T12:11:26.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.139+0000 7f11ef84e700 1 -- 172.21.15.16:0/4052721673 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11d4004850 msgr2=0x7f11d4004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:26.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.139+0000 7f11ef84e700 1 --2- 172.21.15.16:0/4052721673 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11d4004850 0x7f11d4004cd0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f11e80cf930 tx=0x7f11d4005880 comp rx=0 tx=0).stop 2023-10-16T12:11:26.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.139+0000 7f11eddeb700 1 -- 172.21.15.16:0/4052721673 reap_dead start 2023-10-16T12:11:26.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.139+0000 7f11ef84e700 1 -- 172.21.15.16:0/4052721673 shutdown_connections 2023-10-16T12:11:26.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.139+0000 7f11ef84e700 1 -- 172.21.15.16:0/4052721673 >> 172.21.15.16:0/4052721673 conn(0x7f11e80cc5e0 msgr2=0x7f11e80cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:26.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.139+0000 7f11ef84e700 1 -- 172.21.15.16:0/4052721673 shutdown_connections 2023-10-16T12:11:26.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.139+0000 7f11ef84e700 1 -- 172.21.15.16:0/4052721673 wait complete. 2023-10-16T12:11:26.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.140+0000 7f11ef84e700 1 Processor -- start 2023-10-16T12:11:26.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.140+0000 7f11ef84e700 1 -- start start 2023-10-16T12:11:26.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.141+0000 7f11ef84e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11d4004850 0x7f11e8158750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:26.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.141+0000 7f11ef84e700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f11e80c6c30 0x7f11e8156c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:26.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.141+0000 7f11ef84e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f11e8151ef0 0x7f11e815c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:26.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.141+0000 7f11ef84e700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11e81593f0 con 0x7f11d4004850 2023-10-16T12:11:26.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.141+0000 7f11ef84e700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11e8153fd0 con 0x7f11e80c6c30 2023-10-16T12:11:26.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.141+0000 7f11ef84e700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11e80c70b0 con 0x7f11e8151ef0 2023-10-16T12:11:26.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.141+0000 7f11ed5ea700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11d4004850 0x7f11e8158750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:26.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.141+0000 7f11ed5ea700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11d4004850 0x7f11e8158750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34298/0 (socket says 172.21.15.16:34298) 2023-10-16T12:11:26.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.141+0000 7f11ed5ea700 1 -- 172.21.15.16:0/2577750984 learned_addr learned my addr 172.21.15.16:0/2577750984 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:26.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.141+0000 7f11ecde9700 1 --2- 172.21.15.16:0/2577750984 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f11e80c6c30 0x7f11e8156c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:26.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.141+0000 7f11eddeb700 1 --2- 172.21.15.16:0/2577750984 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f11e8151ef0 0x7f11e815c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:26.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.141+0000 7f11ed5ea700 1 -- 172.21.15.16:0/2577750984 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f11e8151ef0 msgr2=0x7f11e815c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:26.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.141+0000 7f11ed5ea700 1 --2- 172.21.15.16:0/2577750984 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f11e8151ef0 0x7f11e815c390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:26.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.141+0000 7f11ed5ea700 1 -- 172.21.15.16:0/2577750984 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f11e80c6c30 msgr2=0x7f11e8156c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:26.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.141+0000 7f11ed5ea700 1 --2- 172.21.15.16:0/2577750984 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f11e80c6c30 0x7f11e8156c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:26.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.141+0000 7f11ed5ea700 1 -- 172.21.15.16:0/2577750984 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11e8158e30 con 0x7f11d4004850 2023-10-16T12:11:26.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.141+0000 7f11eddeb700 1 --2- 172.21.15.16:0/2577750984 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f11e8151ef0 0x7f11e815c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:11:26.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.142+0000 7f11ed5ea700 1 --2- 172.21.15.16:0/2577750984 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11d4004850 0x7f11e8158750 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f11d80072c0 tx=0x7f11d80075d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:26.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.142+0000 7f11ddffb700 1 -- 172.21.15.16:0/2577750984 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11d80077b0 con 0x7f11d4004850 2023-10-16T12:11:26.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.142+0000 7f11ddffb700 1 -- 172.21.15.16:0/2577750984 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11d800f850 con 0x7f11d4004850 2023-10-16T12:11:26.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.142+0000 7f11ef84e700 1 -- 172.21.15.16:0/2577750984 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11e8159110 con 0x7f11d4004850 2023-10-16T12:11:26.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.142+0000 7f11ddffb700 1 -- 172.21.15.16:0/2577750984 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11d800fc00 con 0x7f11d4004850 2023-10-16T12:11:26.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.142+0000 7f11ef84e700 1 -- 172.21.15.16:0/2577750984 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11e8157180 con 0x7f11d4004850 2023-10-16T12:11:26.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.143+0000 7f11ef84e700 1 -- 172.21.15.16:0/2577750984 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11e8157340 con 0x7f11d4004850 2023-10-16T12:11:26.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.143+0000 7f11ddffb700 1 -- 172.21.15.16:0/2577750984 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f11d800fd60 con 0x7f11d4004850 2023-10-16T12:11:26.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.144+0000 7f11ddffb700 1 --2- 172.21.15.16:0/2577750984 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f11cc06af00 0x7f11cc06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:26.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.144+0000 7f11ddffb700 1 -- 172.21.15.16:0/2577750984 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f11d8089870 con 0x7f11d4004850 2023-10-16T12:11:26.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.144+0000 7f11ecde9700 1 --2- 172.21.15.16:0/2577750984 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f11cc06af00 0x7f11cc06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:26.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.144+0000 7f11ecde9700 1 --2- 172.21.15.16:0/2577750984 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f11cc06af00 0x7f11cc06d3c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f11e80ce1f0 tx=0x7f11e4011000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:26.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.148+0000 7f11ddffb700 1 -- 172.21.15.16:0/2577750984 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f11d8054630 con 0x7f11d4004850 2023-10-16T12:11:26.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.352+0000 7f11ef84e700 1 -- 172.21.15.16:0/2577750984 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f11e804ebb0 con 0x7f11d4004850 2023-10-16T12:11:26.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.352+0000 7f11ddffb700 1 -- 172.21.15.16:0/2577750984 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) v1 ==== 103+0+11 (secure 0 0 0) 0x7f11d8057c50 con 0x7f11d4004850 2023-10-16T12:11:26.355 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-10-16T12:11:26.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.354+0000 7f11ef84e700 1 -- 172.21.15.16:0/2577750984 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f11cc06af00 msgr2=0x7f11cc06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:26.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.354+0000 7f11ef84e700 1 --2- 172.21.15.16:0/2577750984 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f11cc06af00 0x7f11cc06d3c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f11e80ce1f0 tx=0x7f11e4011000 comp rx=0 tx=0).stop 2023-10-16T12:11:26.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.354+0000 7f11ef84e700 1 -- 172.21.15.16:0/2577750984 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11d4004850 msgr2=0x7f11e8158750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:26.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.354+0000 7f11ef84e700 1 --2- 172.21.15.16:0/2577750984 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11d4004850 0x7f11e8158750 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f11d80072c0 tx=0x7f11d80075d0 comp rx=0 tx=0).stop 2023-10-16T12:11:26.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.354+0000 7f11ef84e700 1 -- 172.21.15.16:0/2577750984 shutdown_connections 2023-10-16T12:11:26.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.354+0000 7f11ef84e700 1 --2- 172.21.15.16:0/2577750984 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f11cc06af00 0x7f11cc06d3c0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:26.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.354+0000 7f11ef84e700 1 --2- 172.21.15.16:0/2577750984 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f11e8151ef0 0x7f11e815c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:26.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.354+0000 7f11ef84e700 1 --2- 172.21.15.16:0/2577750984 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11d4004850 0x7f11e8158750 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:26.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.354+0000 7f11ef84e700 1 --2- 172.21.15.16:0/2577750984 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f11e80c6c30 0x7f11e8156c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:26.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.354+0000 7f11ef84e700 1 -- 172.21.15.16:0/2577750984 >> 172.21.15.16:0/2577750984 conn(0x7f11e80cc5e0 msgr2=0x7f11e80b4810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:26.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.355+0000 7f11ef84e700 1 -- 172.21.15.16:0/2577750984 shutdown_connections 2023-10-16T12:11:26.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.355+0000 7f11ef84e700 1 -- 172.21.15.16:0/2577750984 wait complete. 2023-10-16T12:11:26.369 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:11:26.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.491+0000 7f3c6468b700 1 Processor -- start 2023-10-16T12:11:26.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.491+0000 7f3c6468b700 1 -- start start 2023-10-16T12:11:26.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.492+0000 7f3c6468b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3c5c0cf880 0x7f3c5c0b52f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:26.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.492+0000 7f3c6468b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3c5c0b5830 0x7f3c5c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:26.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.492+0000 7f3c6468b700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c5c15bc80 con 0x7f3c5c0cf4a0 2023-10-16T12:11:26.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.492+0000 7f3c6468b700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c5c15b070 con 0x7f3c5c0b5830 2023-10-16T12:11:26.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.492+0000 7f3c6468b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c5c15b350 con 0x7f3c5c0cf880 2023-10-16T12:11:26.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.492+0000 7f3c62c28700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3c5c0b5830 0x7f3c5c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:26.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.493+0000 7f3c62c28700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3c5c0b5830 0x7f3c5c15a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34310/0 (socket says 172.21.15.16:34310) 2023-10-16T12:11:26.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.493+0000 7f3c62c28700 1 -- 172.21.15.16:0/1696922555 learned_addr learned my addr 172.21.15.16:0/1696922555 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:26.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.493+0000 7f3c61c26700 1 --2- 172.21.15.16:0/1696922555 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3c5c0cf880 0x7f3c5c0b52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:26.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.493+0000 7f3c62c28700 1 -- 172.21.15.16:0/1696922555 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3c5c0cf880 msgr2=0x7f3c5c0b52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:26.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.493+0000 7f3c62c28700 1 --2- 172.21.15.16:0/1696922555 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3c5c0cf880 0x7f3c5c0b52f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:26.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.493+0000 7f3c62c28700 1 -- 172.21.15.16:0/1696922555 >> v1:172.21.15.49:6789/0 conn(0x7f3c5c0cf4a0 legacy=0x7f3c5c0726e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:26.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.493+0000 7f3c62c28700 1 -- 172.21.15.16:0/1696922555 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c5c15b630 con 0x7f3c5c0b5830 2023-10-16T12:11:26.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.493+0000 7f3c62c28700 1 --2- 172.21.15.16:0/1696922555 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c5c0b5830 0x7f3c5c15a090 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f3c5400d150 tx=0x7f3c5400d460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43763b6a8095f815 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:26.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.493+0000 7f3c61425700 1 -- 172.21.15.16:0/1696922555 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c54010070 con 0x7f3c5c0b5830 2023-10-16T12:11:26.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.493+0000 7f3c61425700 1 -- 172.21.15.16:0/1696922555 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c5400ddc0 con 0x7f3c5c0b5830 2023-10-16T12:11:26.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.493+0000 7f3c61425700 1 -- 172.21.15.16:0/1696922555 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c5400b310 con 0x7f3c5c0b5830 2023-10-16T12:11:26.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.494+0000 7f3c6468b700 1 -- 172.21.15.16:0/1696922555 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c5c0b5830 msgr2=0x7f3c5c15a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:26.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.494+0000 7f3c6468b700 1 --2- 172.21.15.16:0/1696922555 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c5c0b5830 0x7f3c5c15a090 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f3c5400d150 tx=0x7f3c5400d460 comp rx=0 tx=0).stop 2023-10-16T12:11:26.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.494+0000 7f3c6468b700 1 -- 172.21.15.16:0/1696922555 shutdown_connections 2023-10-16T12:11:26.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.494+0000 7f3c6468b700 1 --2- 172.21.15.16:0/1696922555 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c5c0b5830 0x7f3c5c15a090 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:26.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.494+0000 7f3c6468b700 1 --2- 172.21.15.16:0/1696922555 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3c5c0cf880 0x7f3c5c0b52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:26.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.494+0000 7f3c6468b700 1 -- 172.21.15.16:0/1696922555 >> 172.21.15.16:0/1696922555 conn(0x7f3c5c0b1fd0 msgr2=0x7f3c5c0b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:26.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.494+0000 7f3c6468b700 1 -- 172.21.15.16:0/1696922555 shutdown_connections 2023-10-16T12:11:26.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.494+0000 7f3c6468b700 1 -- 172.21.15.16:0/1696922555 wait complete. 2023-10-16T12:11:26.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.495+0000 7f3c6468b700 1 Processor -- start 2023-10-16T12:11:26.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.495+0000 7f3c6468b700 1 -- start start 2023-10-16T12:11:26.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.496+0000 7f3c6468b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3c5c0b5830 0x7f3c5c0b4630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:26.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.496+0000 7f3c6468b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3c5c0cf4a0 0x7f3c5c0c7f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:26.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.496+0000 7f3c6468b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c5c0cf880 0x7f3c5c1585b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:26.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.496+0000 7f3c6468b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c5c0c8460 con 0x7f3c5c0cf880 2023-10-16T12:11:26.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.496+0000 7f3c6468b700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c5c0c8700 con 0x7f3c5c0cf4a0 2023-10-16T12:11:26.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.496+0000 7f3c6468b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c5c0c89e0 con 0x7f3c5c0b5830 2023-10-16T12:11:26.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.496+0000 7f3c62c28700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c5c0cf880 0x7f3c5c1585b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:26.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.496+0000 7f3c62c28700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c5c0cf880 0x7f3c5c1585b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34316/0 (socket says 172.21.15.16:34316) 2023-10-16T12:11:26.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.496+0000 7f3c62c28700 1 -- 172.21.15.16:0/2384450091 learned_addr learned my addr 172.21.15.16:0/2384450091 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:26.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.496+0000 7f3c62427700 1 --2- 172.21.15.16:0/2384450091 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3c5c0b5830 0x7f3c5c0b4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:26.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.496+0000 7f3c61c26700 1 --2- 172.21.15.16:0/2384450091 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3c5c0cf4a0 0x7f3c5c0c7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:26.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.496+0000 7f3c62c28700 1 -- 172.21.15.16:0/2384450091 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3c5c0b5830 msgr2=0x7f3c5c0b4630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:26.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.496+0000 7f3c62c28700 1 --2- 172.21.15.16:0/2384450091 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3c5c0b5830 0x7f3c5c0b4630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:26.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.496+0000 7f3c62c28700 1 -- 172.21.15.16:0/2384450091 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3c5c0cf4a0 msgr2=0x7f3c5c0c7f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:26.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.496+0000 7f3c62c28700 1 --2- 172.21.15.16:0/2384450091 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3c5c0cf4a0 0x7f3c5c0c7f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:26.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.496+0000 7f3c62c28700 1 -- 172.21.15.16:0/2384450091 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c5400bd40 con 0x7f3c5c0cf880 2023-10-16T12:11:26.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.496+0000 7f3c61c26700 1 --2- 172.21.15.16:0/2384450091 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3c5c0cf4a0 0x7f3c5c0c7f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:11:26.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.496+0000 7f3c62c28700 1 --2- 172.21.15.16:0/2384450091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c5c0cf880 0x7f3c5c1585b0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c0cdfc0 tx=0x7f3c54004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:26.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.497+0000 7f3c4effd700 1 -- 172.21.15.16:0/2384450091 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c5400bbc0 con 0x7f3c5c0cf880 2023-10-16T12:11:26.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.497+0000 7f3c4effd700 1 -- 172.21.15.16:0/2384450091 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c5400b4c0 con 0x7f3c5c0cf880 2023-10-16T12:11:26.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.497+0000 7f3c6468b700 1 -- 172.21.15.16:0/2384450091 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c5c0c8cc0 con 0x7f3c5c0cf880 2023-10-16T12:11:26.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.497+0000 7f3c4effd700 1 -- 172.21.15.16:0/2384450091 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c54004040 con 0x7f3c5c0cf880 2023-10-16T12:11:26.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.497+0000 7f3c6468b700 1 -- 172.21.15.16:0/2384450091 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c5c0c9280 con 0x7f3c5c0cf880 2023-10-16T12:11:26.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.498+0000 7f3c6468b700 1 -- 172.21.15.16:0/2384450091 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c5c0c4290 con 0x7f3c5c0cf880 2023-10-16T12:11:26.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.498+0000 7f3c4effd700 1 -- 172.21.15.16:0/2384450091 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f3c540041c0 con 0x7f3c5c0cf880 2023-10-16T12:11:26.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.498+0000 7f3c4effd700 1 --2- 172.21.15.16:0/2384450091 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f3c4006af00 0x7f3c4006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:26.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.499+0000 7f3c62427700 1 --2- 172.21.15.16:0/2384450091 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f3c4006af00 0x7f3c4006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:26.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.499+0000 7f3c4effd700 1 -- 172.21.15.16:0/2384450091 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f3c5408d300 con 0x7f3c5c0cf880 2023-10-16T12:11:26.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.499+0000 7f3c62427700 1 --2- 172.21.15.16:0/2384450091 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f3c4006af00 0x7f3c4006d3c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f3c58000f60 tx=0x7f3c58009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:26.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.503+0000 7f3c4effd700 1 -- 172.21.15.16:0/2384450091 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3c540580c0 con 0x7f3c5c0cf880 2023-10-16T12:11:26.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.708+0000 7f3c6468b700 1 -- 172.21.15.16:0/2384450091 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3c5c04ebb0 con 0x7f3c5c0cf880 2023-10-16T12:11:26.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.709+0000 7f3c4effd700 1 -- 172.21.15.16:0/2384450091 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18238 (secure 0 0 0) 0x7f3c5405b6e0 con 0x7f3c5c0cf880 2023-10-16T12:11:26.711 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:11:26.712 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":25,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:10:44.080141+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:10:36.935773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T12:10:38.004870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-16T12:10:41.672025+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:11:26.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.711+0000 7f3c6468b700 1 -- 172.21.15.16:0/2384450091 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f3c4006af00 msgr2=0x7f3c4006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:26.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.711+0000 7f3c6468b700 1 --2- 172.21.15.16:0/2384450091 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f3c4006af00 0x7f3c4006d3c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f3c58000f60 tx=0x7f3c58009040 comp rx=0 tx=0).stop 2023-10-16T12:11:26.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.711+0000 7f3c6468b700 1 -- 172.21.15.16:0/2384450091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c5c0cf880 msgr2=0x7f3c5c1585b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:26.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.711+0000 7f3c6468b700 1 --2- 172.21.15.16:0/2384450091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c5c0cf880 0x7f3c5c1585b0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c0cdfc0 tx=0x7f3c54004ad0 comp rx=0 tx=0).stop 2023-10-16T12:11:26.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.712+0000 7f3c6468b700 1 -- 172.21.15.16:0/2384450091 shutdown_connections 2023-10-16T12:11:26.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.712+0000 7f3c6468b700 1 --2- 172.21.15.16:0/2384450091 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f3c4006af00 0x7f3c4006d3c0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:26.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.712+0000 7f3c6468b700 1 --2- 172.21.15.16:0/2384450091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c5c0cf880 0x7f3c5c1585b0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:26.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.712+0000 7f3c6468b700 1 --2- 172.21.15.16:0/2384450091 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3c5c0b5830 0x7f3c5c0b4630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:26.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.712+0000 7f3c6468b700 1 --2- 172.21.15.16:0/2384450091 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3c5c0cf4a0 0x7f3c5c0c7f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:26.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.712+0000 7f3c6468b700 1 -- 172.21.15.16:0/2384450091 >> 172.21.15.16:0/2384450091 conn(0x7f3c5c0b1fd0 msgr2=0x7f3c5c0afe40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:26.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.712+0000 7f3c6468b700 1 -- 172.21.15.16:0/2384450091 shutdown_connections 2023-10-16T12:11:26.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.712+0000 7f3c6468b700 1 -- 172.21.15.16:0/2384450091 wait complete. 2023-10-16T12:11:26.726 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T12:10:36.935773+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T12:10:38.004870+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-16T12:10:41.672025+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-16T12:11:26.727 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:11:26.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.853+0000 7f2d70137700 1 Processor -- start 2023-10-16T12:11:26.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.853+0000 7f2d70137700 1 -- start start 2023-10-16T12:11:26.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.854+0000 7f2d70137700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f2d680af0c0 0x7f2d680ce030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:26.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.854+0000 7f2d70137700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2d680cb170 0x7f2d6806ada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:26.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.854+0000 7f2d70137700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d6806a910 con 0x7f2d680ce570 2023-10-16T12:11:26.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.854+0000 7f2d70137700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d680af4a0 con 0x7f2d680cb170 2023-10-16T12:11:26.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.854+0000 7f2d70137700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d680cf440 con 0x7f2d680af0c0 2023-10-16T12:11:26.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.854+0000 7f2d6e6d4700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2d680cb170 0x7f2d6806ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:26.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.854+0000 7f2d6d6d2700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2d680ce570 0x7f2d680caa60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51828/0 (socket says 172.21.15.16:51828) 2023-10-16T12:11:26.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.854+0000 7f2d6d6d2700 1 -- 172.21.15.16:0/1154895072 learned_addr learned my addr 172.21.15.16:0/1154895072 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:26.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.854+0000 7f2d6ded3700 1 --2- 172.21.15.16:0/1154895072 >> v2:172.21.15.49:3300/0 conn(0x7f2d680af0c0 0x7f2d680ce030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:26.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.854+0000 7f2d6e6d4700 1 -- 172.21.15.16:0/1154895072 >> v2:172.21.15.49:3300/0 conn(0x7f2d680af0c0 msgr2=0x7f2d680ce030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:26.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.854+0000 7f2d6e6d4700 1 --2- 172.21.15.16:0/1154895072 >> v2:172.21.15.49:3300/0 conn(0x7f2d680af0c0 0x7f2d680ce030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:26.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.854+0000 7f2d6e6d4700 1 -- 172.21.15.16:0/1154895072 >> v1:172.21.15.16:6789/0 conn(0x7f2d680ce570 legacy=0x7f2d680caa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:26.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.854+0000 7f2d6e6d4700 1 -- 172.21.15.16:0/1154895072 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d680ceba0 con 0x7f2d680cb170 2023-10-16T12:11:26.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.854+0000 7f2d6ced1700 1 -- 172.21.15.16:0/1154895072 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1990028559 0 0) 0x7f2d6806a910 con 0x7f2d680ce570 2023-10-16T12:11:26.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.854+0000 7f2d6e6d4700 1 --2- 172.21.15.16:0/1154895072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d680cb170 0x7f2d6806ada0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f2d68066b60 tx=0x7f2d6000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3f7413f6bdfd52e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:26.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.855+0000 7f2d6ced1700 1 -- 172.21.15.16:0/1154895072 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d6000e070 con 0x7f2d680cb170 2023-10-16T12:11:26.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.855+0000 7f2d6ced1700 1 -- 172.21.15.16:0/1154895072 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d6000bbd0 con 0x7f2d680cb170 2023-10-16T12:11:26.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.855+0000 7f2d6ced1700 1 -- 172.21.15.16:0/1154895072 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d60012430 con 0x7f2d680cb170 2023-10-16T12:11:26.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.855+0000 7f2d70137700 1 -- 172.21.15.16:0/1154895072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d680cb170 msgr2=0x7f2d6806ada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:26.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.855+0000 7f2d70137700 1 --2- 172.21.15.16:0/1154895072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d680cb170 0x7f2d6806ada0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f2d68066b60 tx=0x7f2d6000b330 comp rx=0 tx=0).stop 2023-10-16T12:11:26.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.855+0000 7f2d70137700 1 -- 172.21.15.16:0/1154895072 shutdown_connections 2023-10-16T12:11:26.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.855+0000 7f2d70137700 1 --2- 172.21.15.16:0/1154895072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d680cb170 0x7f2d6806ada0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:26.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.855+0000 7f2d70137700 1 --2- 172.21.15.16:0/1154895072 >> v2:172.21.15.49:3300/0 conn(0x7f2d680af0c0 0x7f2d680ce030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:26.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.855+0000 7f2d70137700 1 -- 172.21.15.16:0/1154895072 >> 172.21.15.16:0/1154895072 conn(0x7f2d680cc5c0 msgr2=0x7f2d680cc9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:26.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.855+0000 7f2d70137700 1 -- 172.21.15.16:0/1154895072 shutdown_connections 2023-10-16T12:11:26.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.855+0000 7f2d70137700 1 -- 172.21.15.16:0/1154895072 wait complete. 2023-10-16T12:11:26.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.856+0000 7f2d70137700 1 Processor -- start 2023-10-16T12:11:26.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.857+0000 7f2d70137700 1 -- start start 2023-10-16T12:11:26.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.857+0000 7f2d70137700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2d680af0c0 0x7f2d6814f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:26.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.857+0000 7f2d70137700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2d680cb170 0x7f2d6814bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:26.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.857+0000 7f2d70137700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d680ce570 0x7f2d680c76b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:26.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.857+0000 7f2d70137700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d6814c4c0 con 0x7f2d680ce570 2023-10-16T12:11:26.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.857+0000 7f2d70137700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d680c6ff0 con 0x7f2d680af0c0 2023-10-16T12:11:26.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.857+0000 7f2d70137700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d6806ddb0 con 0x7f2d680cb170 2023-10-16T12:11:26.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.857+0000 7f2d6e6d4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d680ce570 0x7f2d680c76b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:26.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.857+0000 7f2d6e6d4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d680ce570 0x7f2d680c76b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34330/0 (socket says 172.21.15.16:34330) 2023-10-16T12:11:26.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.857+0000 7f2d6e6d4700 1 -- 172.21.15.16:0/1483436269 learned_addr learned my addr 172.21.15.16:0/1483436269 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:26.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.858+0000 7f2d6ded3700 1 --2- 172.21.15.16:0/1483436269 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2d680af0c0 0x7f2d6814f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:26.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.858+0000 7f2d6d6d2700 1 --2- 172.21.15.16:0/1483436269 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2d680cb170 0x7f2d6814bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:26.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.858+0000 7f2d6e6d4700 1 -- 172.21.15.16:0/1483436269 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2d680cb170 msgr2=0x7f2d6814bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:26.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.858+0000 7f2d6e6d4700 1 --2- 172.21.15.16:0/1483436269 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2d680cb170 0x7f2d6814bf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:26.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.858+0000 7f2d6e6d4700 1 -- 172.21.15.16:0/1483436269 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2d680af0c0 msgr2=0x7f2d6814f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:26.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.858+0000 7f2d6e6d4700 1 --2- 172.21.15.16:0/1483436269 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2d680af0c0 0x7f2d6814f4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:26.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.858+0000 7f2d6e6d4700 1 -- 172.21.15.16:0/1483436269 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d60009a70 con 0x7f2d680ce570 2023-10-16T12:11:26.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.858+0000 7f2d6d6d2700 1 --2- 172.21.15.16:0/1483436269 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2d680cb170 0x7f2d6814bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:11:26.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.858+0000 7f2d6e6d4700 1 --2- 172.21.15.16:0/1483436269 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d680ce570 0x7f2d680c76b0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f2d60013000 tx=0x7f2d60004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:26.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.858+0000 7f2d5e7fc700 1 -- 172.21.15.16:0/1483436269 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d60012430 con 0x7f2d680ce570 2023-10-16T12:11:26.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.858+0000 7f2d5e7fc700 1 -- 172.21.15.16:0/1483436269 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d60020070 con 0x7f2d680ce570 2023-10-16T12:11:26.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.858+0000 7f2d70137700 1 -- 172.21.15.16:0/1483436269 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d6806e090 con 0x7f2d680ce570 2023-10-16T12:11:26.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.858+0000 7f2d5e7fc700 1 -- 172.21.15.16:0/1483436269 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d60004280 con 0x7f2d680ce570 2023-10-16T12:11:26.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.858+0000 7f2d70137700 1 -- 172.21.15.16:0/1483436269 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d6806e650 con 0x7f2d680ce570 2023-10-16T12:11:26.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.859+0000 7f2d4bfff700 1 -- 172.21.15.16:0/1483436269 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d440052f0 con 0x7f2d680ce570 2023-10-16T12:11:26.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.860+0000 7f2d5e7fc700 1 -- 172.21.15.16:0/1483436269 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f2d600043e0 con 0x7f2d680ce570 2023-10-16T12:11:26.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.860+0000 7f2d5e7fc700 1 --2- 172.21.15.16:0/1483436269 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f2d4c06af00 0x7f2d4c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:26.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.860+0000 7f2d5e7fc700 1 -- 172.21.15.16:0/1483436269 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f2d60022070 con 0x7f2d680ce570 2023-10-16T12:11:26.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.860+0000 7f2d6ded3700 1 --2- 172.21.15.16:0/1483436269 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f2d4c06af00 0x7f2d4c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:26.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.861+0000 7f2d6ded3700 1 --2- 172.21.15.16:0/1483436269 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f2d4c06af00 0x7f2d4c06d3c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f2d64009930 tx=0x7f2d64008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:26.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:26.865+0000 7f2d5e7fc700 1 -- 172.21.15.16:0/1483436269 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2d60056870 con 0x7f2d680ce570 2023-10-16T12:11:27.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.068+0000 7f2d4bfff700 1 -- 172.21.15.16:0/1483436269 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2d44005b20 con 0x7f2d680ce570 2023-10-16T12:11:27.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.069+0000 7f2d5e7fc700 1 -- 172.21.15.16:0/1483436269 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2d60059e90 con 0x7f2d680ce570 2023-10-16T12:11:27.071 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T12:11:27.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.071+0000 7f2d4bfff700 1 -- 172.21.15.16:0/1483436269 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f2d4c06af00 msgr2=0x7f2d4c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:27.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.071+0000 7f2d4bfff700 1 --2- 172.21.15.16:0/1483436269 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f2d4c06af00 0x7f2d4c06d3c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f2d64009930 tx=0x7f2d64008040 comp rx=0 tx=0).stop 2023-10-16T12:11:27.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.071+0000 7f2d4bfff700 1 -- 172.21.15.16:0/1483436269 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d680ce570 msgr2=0x7f2d680c76b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:27.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.071+0000 7f2d4bfff700 1 --2- 172.21.15.16:0/1483436269 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d680ce570 0x7f2d680c76b0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f2d60013000 tx=0x7f2d60004ad0 comp rx=0 tx=0).stop 2023-10-16T12:11:27.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.071+0000 7f2d4bfff700 1 -- 172.21.15.16:0/1483436269 shutdown_connections 2023-10-16T12:11:27.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.071+0000 7f2d4bfff700 1 --2- 172.21.15.16:0/1483436269 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f2d4c06af00 0x7f2d4c06d3c0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:27.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.071+0000 7f2d4bfff700 1 --2- 172.21.15.16:0/1483436269 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d680ce570 0x7f2d680c76b0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:27.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.071+0000 7f2d4bfff700 1 --2- 172.21.15.16:0/1483436269 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2d680af0c0 0x7f2d6814f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:27.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.071+0000 7f2d4bfff700 1 --2- 172.21.15.16:0/1483436269 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2d680cb170 0x7f2d6814bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:27.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.071+0000 7f2d4bfff700 1 -- 172.21.15.16:0/1483436269 >> 172.21.15.16:0/1483436269 conn(0x7f2d680cc5c0 msgr2=0x7f2d680afe90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:27.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.071+0000 7f2d4bfff700 1 -- 172.21.15.16:0/1483436269 shutdown_connections 2023-10-16T12:11:27.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.071+0000 7f2d4bfff700 1 -- 172.21.15.16:0/1483436269 wait complete. 2023-10-16T12:11:27.085 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T12:11:27.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.213+0000 7fe774ec3700 1 Processor -- start 2023-10-16T12:11:27.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.214+0000 7fe774ec3700 1 -- start start 2023-10-16T12:11:27.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.214+0000 7fe774ec3700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7701577a0 0x7fe7700cf1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:27.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.215+0000 7fe774ec3700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe7700717b0 0x7fe7700b2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:27.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.215+0000 7fe774ec3700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7700cfb80 con 0x7fe7700cf700 2023-10-16T12:11:27.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.215+0000 7fe774ec3700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe770072350 con 0x7fe7700717b0 2023-10-16T12:11:27.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.215+0000 7fe774ec3700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe770071d90 con 0x7fe7701577a0 2023-10-16T12:11:27.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.215+0000 7fe76ed9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe7700717b0 0x7fe7700b2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:27.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.215+0000 7fe76ed9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe7700717b0 0x7fe7700b2ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34338/0 (socket says 172.21.15.16:34338) 2023-10-16T12:11:27.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.215+0000 7fe76ed9d700 1 -- 172.21.15.16:0/1196639648 learned_addr learned my addr 172.21.15.16:0/1196639648 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:27.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.215+0000 7fe76ed9d700 1 -- 172.21.15.16:0/1196639648 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7701577a0 msgr2=0x7fe7700cf1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:27.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.215+0000 7fe76e59c700 1 --2- 172.21.15.16:0/1196639648 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7701577a0 0x7fe7700cf1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:27.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.215+0000 7fe76ed9d700 1 --2- 172.21.15.16:0/1196639648 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7701577a0 0x7fe7700cf1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:27.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.215+0000 7fe76ed9d700 1 -- 172.21.15.16:0/1196639648 >> v1:172.21.15.49:6789/0 conn(0x7fe7700cf700 legacy=0x7fe7700710a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:27.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.215+0000 7fe76ed9d700 1 -- 172.21.15.16:0/1196639648 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe770072070 con 0x7fe7700717b0 2023-10-16T12:11:27.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.215+0000 7fe76ed9d700 1 --2- 172.21.15.16:0/1196639648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7700717b0 0x7fe7700b2ac0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fe768009e50 tx=0x7fe76800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8606b8a32e6f1d23 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:27.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.215+0000 7fe76d59a700 1 -- 172.21.15.16:0/1196639648 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe76800e070 con 0x7fe7700717b0 2023-10-16T12:11:27.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.215+0000 7fe76d59a700 1 -- 172.21.15.16:0/1196639648 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe768003a20 con 0x7fe7700717b0 2023-10-16T12:11:27.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.215+0000 7fe76d59a700 1 -- 172.21.15.16:0/1196639648 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe768003dd0 con 0x7fe7700717b0 2023-10-16T12:11:27.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.216+0000 7fe774ec3700 1 -- 172.21.15.16:0/1196639648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7700717b0 msgr2=0x7fe7700b2ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:27.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.216+0000 7fe774ec3700 1 --2- 172.21.15.16:0/1196639648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7700717b0 0x7fe7700b2ac0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fe768009e50 tx=0x7fe76800b330 comp rx=0 tx=0).stop 2023-10-16T12:11:27.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.216+0000 7fe774ec3700 1 -- 172.21.15.16:0/1196639648 shutdown_connections 2023-10-16T12:11:27.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.216+0000 7fe774ec3700 1 --2- 172.21.15.16:0/1196639648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7700717b0 0x7fe7700b2ac0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:27.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.216+0000 7fe774ec3700 1 --2- 172.21.15.16:0/1196639648 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7701577a0 0x7fe7700cf1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:27.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.216+0000 7fe774ec3700 1 -- 172.21.15.16:0/1196639648 >> 172.21.15.16:0/1196639648 conn(0x7fe77006a7e0 msgr2=0x7fe77006abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:27.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.216+0000 7fe774ec3700 1 -- 172.21.15.16:0/1196639648 shutdown_connections 2023-10-16T12:11:27.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.216+0000 7fe774ec3700 1 -- 172.21.15.16:0/1196639648 wait complete. 2023-10-16T12:11:27.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.218+0000 7fe774ec3700 1 Processor -- start 2023-10-16T12:11:27.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.218+0000 7fe774ec3700 1 -- start start 2023-10-16T12:11:27.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.219+0000 7fe774ec3700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7700717b0 0x7fe7700c9770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:27.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.219+0000 7fe774ec3700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7700cf700 0x7fe770153440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:27.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.219+0000 7fe774ec3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7701577a0 0x7fe7700cdef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:27.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.219+0000 7fe774ec3700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7700caa40 con 0x7fe7701577a0 2023-10-16T12:11:27.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.219+0000 7fe774ec3700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe770153980 con 0x7fe7700717b0 2023-10-16T12:11:27.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.219+0000 7fe774ec3700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7700c9dc0 con 0x7fe7700cf700 2023-10-16T12:11:27.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.219+0000 7fe76ed9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7701577a0 0x7fe7700cdef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:27.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.219+0000 7fe76ed9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7701577a0 0x7fe7700cdef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34344/0 (socket says 172.21.15.16:34344) 2023-10-16T12:11:27.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.219+0000 7fe76ed9d700 1 -- 172.21.15.16:0/4247600792 learned_addr learned my addr 172.21.15.16:0/4247600792 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:27.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.219+0000 7fe76e59c700 1 --2- 172.21.15.16:0/4247600792 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7700717b0 0x7fe7700c9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:27.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.219+0000 7fe76dd9b700 1 --2- 172.21.15.16:0/4247600792 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7700cf700 0x7fe770153440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:27.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.219+0000 7fe76ed9d700 1 -- 172.21.15.16:0/4247600792 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7700cf700 msgr2=0x7fe770153440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:27.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.219+0000 7fe76ed9d700 1 --2- 172.21.15.16:0/4247600792 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7700cf700 0x7fe770153440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:27.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.219+0000 7fe76ed9d700 1 -- 172.21.15.16:0/4247600792 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7700717b0 msgr2=0x7fe7700c9770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:27.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.219+0000 7fe76ed9d700 1 --2- 172.21.15.16:0/4247600792 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7700717b0 0x7fe7700c9770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:27.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.219+0000 7fe76ed9d700 1 -- 172.21.15.16:0/4247600792 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe768009b30 con 0x7fe7701577a0 2023-10-16T12:11:27.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.219+0000 7fe76e59c700 1 --2- 172.21.15.16:0/4247600792 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7700717b0 0x7fe7700c9770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:11:27.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.219+0000 7fe76ed9d700 1 --2- 172.21.15.16:0/4247600792 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7701577a0 0x7fe7700cdef0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fe77015a7f0 tx=0x7fe768004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:27.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.219+0000 7fe766ffd700 1 -- 172.21.15.16:0/4247600792 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe768004440 con 0x7fe7701577a0 2023-10-16T12:11:27.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.219+0000 7fe766ffd700 1 -- 172.21.15.16:0/4247600792 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7680045a0 con 0x7fe7701577a0 2023-10-16T12:11:27.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.219+0000 7fe774ec3700 1 -- 172.21.15.16:0/4247600792 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7700ca0a0 con 0x7fe7701577a0 2023-10-16T12:11:27.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.220+0000 7fe766ffd700 1 -- 172.21.15.16:0/4247600792 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe768004950 con 0x7fe7701577a0 2023-10-16T12:11:27.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.220+0000 7fe774ec3700 1 -- 172.21.15.16:0/4247600792 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7700ca660 con 0x7fe7701577a0 2023-10-16T12:11:27.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.220+0000 7fe774ec3700 1 -- 172.21.15.16:0/4247600792 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7700cab80 con 0x7fe7701577a0 2023-10-16T12:11:27.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.221+0000 7fe766ffd700 1 -- 172.21.15.16:0/4247600792 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fe768012430 con 0x7fe7701577a0 2023-10-16T12:11:27.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.221+0000 7fe766ffd700 1 --2- 172.21.15.16:0/4247600792 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fe75006aed0 0x7fe75006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:27.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.221+0000 7fe766ffd700 1 -- 172.21.15.16:0/4247600792 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fe76808ae10 con 0x7fe7701577a0 2023-10-16T12:11:27.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.221+0000 7fe76e59c700 1 --2- 172.21.15.16:0/4247600792 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fe75006aed0 0x7fe75006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:27.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.222+0000 7fe76e59c700 1 --2- 172.21.15.16:0/4247600792 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fe75006aed0 0x7fe75006d390 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe758006fd0 tx=0x7fe758008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:27.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.226+0000 7fe766ffd700 1 -- 172.21.15.16:0/4247600792 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe768055be0 con 0x7fe7701577a0 2023-10-16T12:11:27.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.434+0000 7fe774ec3700 1 -- 172.21.15.16:0/4247600792 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe77004ebb0 con 0x7fe7701577a0 2023-10-16T12:11:27.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.434+0000 7fe766ffd700 1 -- 172.21.15.16:0/4247600792 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe768059200 con 0x7fe7701577a0 2023-10-16T12:11:27.436 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-10-16T12:11:27.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.436+0000 7fe774ec3700 1 -- 172.21.15.16:0/4247600792 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fe75006aed0 msgr2=0x7fe75006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:27.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.436+0000 7fe774ec3700 1 --2- 172.21.15.16:0/4247600792 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fe75006aed0 0x7fe75006d390 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe758006fd0 tx=0x7fe758008040 comp rx=0 tx=0).stop 2023-10-16T12:11:27.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.436+0000 7fe774ec3700 1 -- 172.21.15.16:0/4247600792 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7701577a0 msgr2=0x7fe7700cdef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:27.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.436+0000 7fe774ec3700 1 --2- 172.21.15.16:0/4247600792 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7701577a0 0x7fe7700cdef0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fe77015a7f0 tx=0x7fe768004ad0 comp rx=0 tx=0).stop 2023-10-16T12:11:27.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.436+0000 7fe774ec3700 1 -- 172.21.15.16:0/4247600792 shutdown_connections 2023-10-16T12:11:27.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.436+0000 7fe774ec3700 1 --2- 172.21.15.16:0/4247600792 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fe75006aed0 0x7fe75006d390 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:27.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.436+0000 7fe774ec3700 1 --2- 172.21.15.16:0/4247600792 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7701577a0 0x7fe7700cdef0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:27.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.436+0000 7fe774ec3700 1 --2- 172.21.15.16:0/4247600792 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7700717b0 0x7fe7700c9770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:27.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.436+0000 7fe774ec3700 1 --2- 172.21.15.16:0/4247600792 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7700cf700 0x7fe770153440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:27.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.436+0000 7fe774ec3700 1 -- 172.21.15.16:0/4247600792 >> 172.21.15.16:0/4247600792 conn(0x7fe77006a7e0 msgr2=0x7fe77015cb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:27.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.436+0000 7fe774ec3700 1 -- 172.21.15.16:0/4247600792 shutdown_connections 2023-10-16T12:11:27.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.436+0000 7fe774ec3700 1 -- 172.21.15.16:0/4247600792 wait complete. 2023-10-16T12:11:27.450 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T12:11:27.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.577+0000 7f4153022700 1 Processor -- start 2023-10-16T12:11:27.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.578+0000 7f4153022700 1 -- start start 2023-10-16T12:11:27.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.578+0000 7f4153022700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f414c15b790 0x7f414c0cda60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:27.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.578+0000 7f4153022700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f414c15bb70 0x7f414c06d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:27.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.578+0000 7f4153022700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f414c0ce0e0 con 0x7f414c06dea0 2023-10-16T12:11:27.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.578+0000 7f4153022700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f414c06e310 con 0x7f414c15bb70 2023-10-16T12:11:27.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.578+0000 7f4153022700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f414c06a7e0 con 0x7f414c15b790 2023-10-16T12:11:27.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.579+0000 7f41515bf700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f414c06dea0 0x7f414c15f1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51858/0 (socket says 172.21.15.16:51858) 2023-10-16T12:11:27.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.579+0000 7f41515bf700 1 -- 172.21.15.16:0/2703785803 learned_addr learned my addr 172.21.15.16:0/2703785803 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:27.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.579+0000 7f414bfff700 1 --2- 172.21.15.16:0/2703785803 >> v2:172.21.15.49:3300/0 conn(0x7f414c15bb70 0x7f414c06d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:27.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.579+0000 7f4150dbe700 1 --2- 172.21.15.16:0/2703785803 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f414c15b790 0x7f414c0cda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:27.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.579+0000 7f414b7fe700 1 -- 172.21.15.16:0/2703785803 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2473650477 0 0) 0x7f414c0ce0e0 con 0x7f414c06dea0 2023-10-16T12:11:27.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.579+0000 7f414b7fe700 1 -- 172.21.15.16:0/2703785803 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f413c0035b0 con 0x7f414c06dea0 2023-10-16T12:11:27.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.579+0000 7f414b7fe700 1 -- 172.21.15.16:0/2703785803 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1266865349 0 0) 0x7f413c0035b0 con 0x7f414c06dea0 2023-10-16T12:11:27.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.579+0000 7f414b7fe700 1 -- 172.21.15.16:0/2703785803 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f414c15b790 msgr2=0x7f414c0cda60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:27.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.579+0000 7f414b7fe700 1 --2- 172.21.15.16:0/2703785803 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f414c15b790 0x7f414c0cda60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:27.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.579+0000 7f414b7fe700 1 -- 172.21.15.16:0/2703785803 >> v2:172.21.15.49:3300/0 conn(0x7f414c15bb70 msgr2=0x7f414c06d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:27.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.579+0000 7f414b7fe700 1 --2- 172.21.15.16:0/2703785803 >> v2:172.21.15.49:3300/0 conn(0x7f414c15bb70 0x7f414c06d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:27.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.579+0000 7f414b7fe700 1 -- 172.21.15.16:0/2703785803 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f414c15c410 con 0x7f414c06dea0 2023-10-16T12:11:27.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.579+0000 7f414b7fe700 1 -- 172.21.15.16:0/2703785803 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f41440019f0 con 0x7f414c06dea0 2023-10-16T12:11:27.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.579+0000 7f414b7fe700 1 -- 172.21.15.16:0/2703785803 >> v1:172.21.15.16:6789/0 conn(0x7f414c06dea0 legacy=0x7f414c15f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:27.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.579+0000 7f414b7fe700 1 --2- 172.21.15.16:0/2703785803 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f413c003a00 0x7f413c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:27.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.579+0000 7f414b7fe700 1 --2- 172.21.15.16:0/2703785803 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f413c004850 0x7f413c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:27.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.579+0000 7f414b7fe700 1 --2- 172.21.15.16:0/2703785803 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f414c15b790 0x7f413c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:27.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.579+0000 7f414b7fe700 1 -- 172.21.15.16:0/2703785803 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f414c06e310 con 0x7f413c003a00 2023-10-16T12:11:27.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.579+0000 7f414b7fe700 1 -- 172.21.15.16:0/2703785803 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f414c06a7e0 con 0x7f413c004850 2023-10-16T12:11:27.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.579+0000 7f414b7fe700 1 -- 172.21.15.16:0/2703785803 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f413c00ba10 con 0x7f414c15b790 2023-10-16T12:11:27.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.579+0000 7f4150dbe700 1 --2- 172.21.15.16:0/2703785803 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f413c003a00 0x7f413c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:27.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.579+0000 7f414bfff700 1 --2- 172.21.15.16:0/2703785803 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f413c004850 0x7f413c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:27.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.579+0000 7f41515bf700 1 --2- 172.21.15.16:0/2703785803 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f414c15b790 0x7f413c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:27.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.580+0000 7f4150dbe700 1 -- 172.21.15.16:0/2703785803 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f414c15b790 msgr2=0x7f413c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:27.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.580+0000 7f4150dbe700 1 --2- 172.21.15.16:0/2703785803 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f414c15b790 0x7f413c005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:27.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.580+0000 7f4150dbe700 1 -- 172.21.15.16:0/2703785803 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f413c004850 msgr2=0x7f413c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:27.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.580+0000 7f4150dbe700 1 --2- 172.21.15.16:0/2703785803 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f413c004850 0x7f413c004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:27.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.580+0000 7f4150dbe700 1 -- 172.21.15.16:0/2703785803 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f413c0035b0 con 0x7f413c003a00 2023-10-16T12:11:27.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.580+0000 7f4150dbe700 1 --2- 172.21.15.16:0/2703785803 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f413c003a00 0x7f413c003e20 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f4140009b60 tx=0x7f413c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2da92288bf908717 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:27.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.580+0000 7f414b7fe700 1 -- 172.21.15.16:0/2703785803 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f414000e6c0 con 0x7f413c003a00 2023-10-16T12:11:27.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.580+0000 7f414b7fe700 1 -- 172.21.15.16:0/2703785803 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f414000e820 con 0x7f413c003a00 2023-10-16T12:11:27.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.580+0000 7f414b7fe700 1 -- 172.21.15.16:0/2703785803 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f414000eb90 con 0x7f413c003a00 2023-10-16T12:11:27.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.580+0000 7f4153022700 1 -- 172.21.15.16:0/2703785803 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f413c003a00 msgr2=0x7f413c003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:27.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.580+0000 7f4153022700 1 --2- 172.21.15.16:0/2703785803 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f413c003a00 0x7f413c003e20 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f4140009b60 tx=0x7f413c005880 comp rx=0 tx=0).stop 2023-10-16T12:11:27.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.580+0000 7f41515bf700 1 -- 172.21.15.16:0/2703785803 reap_dead start 2023-10-16T12:11:27.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.581+0000 7f4153022700 1 -- 172.21.15.16:0/2703785803 shutdown_connections 2023-10-16T12:11:27.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.581+0000 7f4153022700 1 -- 172.21.15.16:0/2703785803 >> 172.21.15.16:0/2703785803 conn(0x7f414c0cc5e0 msgr2=0x7f414c0cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:27.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.581+0000 7f4153022700 1 -- 172.21.15.16:0/2703785803 shutdown_connections 2023-10-16T12:11:27.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.581+0000 7f4153022700 1 -- 172.21.15.16:0/2703785803 wait complete. 2023-10-16T12:11:27.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.582+0000 7f4153022700 1 Processor -- start 2023-10-16T12:11:27.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.582+0000 7f4153022700 1 -- start start 2023-10-16T12:11:27.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.583+0000 7f4153022700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f413c003a00 0x7f414c14be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:27.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.583+0000 7f4153022700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f414c0b06b0 0x7f414c0b0b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:27.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.583+0000 7f4153022700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f414c154a40 0x7f414c154ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:27.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.583+0000 7f4153022700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f414c0b1070 con 0x7f414c154a40 2023-10-16T12:11:27.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.583+0000 7f4153022700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f414c0c3f70 con 0x7f413c003a00 2023-10-16T12:11:27.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.583+0000 7f4153022700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f414c0c70f0 con 0x7f414c0b06b0 2023-10-16T12:11:27.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.583+0000 7f41515bf700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f414c154a40 0x7f414c154ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:27.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.583+0000 7f41515bf700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f414c154a40 0x7f414c154ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34364/0 (socket says 172.21.15.16:34364) 2023-10-16T12:11:27.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.583+0000 7f41515bf700 1 -- 172.21.15.16:0/772690976 learned_addr learned my addr 172.21.15.16:0/772690976 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:27.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.583+0000 7f4150dbe700 1 --2- 172.21.15.16:0/772690976 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f413c003a00 0x7f414c14be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:27.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.583+0000 7f414bfff700 1 --2- 172.21.15.16:0/772690976 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f414c0b06b0 0x7f414c0b0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:27.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.583+0000 7f41515bf700 1 -- 172.21.15.16:0/772690976 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f414c0b06b0 msgr2=0x7f414c0b0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:27.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.583+0000 7f41515bf700 1 --2- 172.21.15.16:0/772690976 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f414c0b06b0 0x7f414c0b0b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:27.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.583+0000 7f41515bf700 1 -- 172.21.15.16:0/772690976 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f413c003a00 msgr2=0x7f414c14be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:27.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.583+0000 7f41515bf700 1 --2- 172.21.15.16:0/772690976 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f413c003a00 0x7f414c14be00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:27.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.583+0000 7f41515bf700 1 -- 172.21.15.16:0/772690976 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f414c0c3640 con 0x7f414c154a40 2023-10-16T12:11:27.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.583+0000 7f4150dbe700 1 --2- 172.21.15.16:0/772690976 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f413c003a00 0x7f414c14be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:11:27.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.583+0000 7f41515bf700 1 --2- 172.21.15.16:0/772690976 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f414c154a40 0x7f414c154ec0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f4144001330 tx=0x7f414400a640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:27.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.584+0000 7f41497fa700 1 -- 172.21.15.16:0/772690976 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41440133f0 con 0x7f414c154a40 2023-10-16T12:11:27.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.584+0000 7f41497fa700 1 -- 172.21.15.16:0/772690976 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4144013590 con 0x7f414c154a40 2023-10-16T12:11:27.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.584+0000 7f41497fa700 1 -- 172.21.15.16:0/772690976 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4144013940 con 0x7f414c154a40 2023-10-16T12:11:27.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.584+0000 7f4153022700 1 -- 172.21.15.16:0/772690976 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f414c0c3920 con 0x7f414c154a40 2023-10-16T12:11:27.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.584+0000 7f4153022700 1 -- 172.21.15.16:0/772690976 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f414c14c340 con 0x7f414c154a40 2023-10-16T12:11:27.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.585+0000 7f4153022700 1 -- 172.21.15.16:0/772690976 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f414c0c40e0 con 0x7f414c154a40 2023-10-16T12:11:27.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.585+0000 7f41497fa700 1 -- 172.21.15.16:0/772690976 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f4144013aa0 con 0x7f414c154a40 2023-10-16T12:11:27.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.585+0000 7f41497fa700 1 --2- 172.21.15.16:0/772690976 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f413006af00 0x7f413006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:27.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.585+0000 7f41497fa700 1 -- 172.21.15.16:0/772690976 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f414408a350 con 0x7f414c154a40 2023-10-16T12:11:27.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.585+0000 7f4150dbe700 1 --2- 172.21.15.16:0/772690976 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f413006af00 0x7f413006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:27.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.586+0000 7f4150dbe700 1 --2- 172.21.15.16:0/772690976 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f413006af00 0x7f413006d3c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f41400090b0 tx=0x7f4140008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:27.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.590+0000 7f41497fa700 1 -- 172.21.15.16:0/772690976 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4144054fc0 con 0x7f414c154a40 2023-10-16T12:11:27.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.797+0000 7f4153022700 1 -- 172.21.15.16:0/772690976 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f414c04ebb0 con 0x7f414c154a40 2023-10-16T12:11:27.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.797+0000 7f41497fa700 1 -- 172.21.15.16:0/772690976 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) v1 ==== 100+0+11 (secure 0 0 0) 0x7f41440585e0 con 0x7f414c154a40 2023-10-16T12:11:27.800 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-10-16T12:11:27.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.800+0000 7f4153022700 1 -- 172.21.15.16:0/772690976 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f413006af00 msgr2=0x7f413006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:27.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.800+0000 7f4153022700 1 --2- 172.21.15.16:0/772690976 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f413006af00 0x7f413006d3c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f41400090b0 tx=0x7f4140008030 comp rx=0 tx=0).stop 2023-10-16T12:11:27.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.800+0000 7f4153022700 1 -- 172.21.15.16:0/772690976 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f414c154a40 msgr2=0x7f414c154ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:27.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.800+0000 7f4153022700 1 --2- 172.21.15.16:0/772690976 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f414c154a40 0x7f414c154ec0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f4144001330 tx=0x7f414400a640 comp rx=0 tx=0).stop 2023-10-16T12:11:27.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.800+0000 7f4153022700 1 -- 172.21.15.16:0/772690976 shutdown_connections 2023-10-16T12:11:27.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.800+0000 7f4153022700 1 --2- 172.21.15.16:0/772690976 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f413006af00 0x7f413006d3c0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:27.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.800+0000 7f4153022700 1 --2- 172.21.15.16:0/772690976 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f414c154a40 0x7f414c154ec0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:27.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.800+0000 7f4153022700 1 --2- 172.21.15.16:0/772690976 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f413c003a00 0x7f414c14be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:27.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.800+0000 7f4153022700 1 --2- 172.21.15.16:0/772690976 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f414c0b06b0 0x7f414c0b0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:27.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.800+0000 7f4153022700 1 -- 172.21.15.16:0/772690976 >> 172.21.15.16:0/772690976 conn(0x7f414c0cc5e0 msgr2=0x7f414c06a9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:27.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.800+0000 7f4153022700 1 -- 172.21.15.16:0/772690976 shutdown_connections 2023-10-16T12:11:27.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.800+0000 7f4153022700 1 -- 172.21.15.16:0/772690976 wait complete. 2023-10-16T12:11:27.814 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-16T12:11:27.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.940+0000 7f8308d6e700 1 Processor -- start 2023-10-16T12:11:27.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.940+0000 7f8308d6e700 1 -- start start 2023-10-16T12:11:27.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.941+0000 7f8308d6e700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f830406e070 0x7f8304166350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:27.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.941+0000 7f8308d6e700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83040ce0e0 con 0x7f830415b790 2023-10-16T12:11:27.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.941+0000 7f8308d6e700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f830406a7e0 con 0x7f83040cda60 2023-10-16T12:11:27.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.941+0000 7f8308d6e700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f830415c410 con 0x7f830406e070 2023-10-16T12:11:27.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.941+0000 7f830259c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f830415b790 0x7f830415bb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51874/0 (socket says 172.21.15.16:51874) 2023-10-16T12:11:27.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.941+0000 7f830259c700 1 -- 172.21.15.16:0/2646086046 learned_addr learned my addr 172.21.15.16:0/2646086046 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:27.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.941+0000 7f8302d9d700 1 --2- 172.21.15.16:0/2646086046 >> v2:172.21.15.49:3300/0 conn(0x7f830406e070 0x7f8304166350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:27.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.941+0000 7f830159a700 1 -- 172.21.15.16:0/2646086046 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3006544764 0 0) 0x7f83040ce0e0 con 0x7f830415b790 2023-10-16T12:11:27.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.941+0000 7f830159a700 1 -- 172.21.15.16:0/2646086046 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82ec0035b0 con 0x7f830415b790 2023-10-16T12:11:27.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.941+0000 7f830159a700 1 -- 172.21.15.16:0/2646086046 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 691964266 0 0) 0x7f830406a7e0 con 0x7f83040cda60 2023-10-16T12:11:27.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.941+0000 7f830159a700 1 -- 172.21.15.16:0/2646086046 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83040ce0e0 con 0x7f83040cda60 2023-10-16T12:11:27.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.941+0000 7f830159a700 1 -- 172.21.15.16:0/2646086046 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3985336115 0 0) 0x7f82ec0035b0 con 0x7f830415b790 2023-10-16T12:11:27.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.941+0000 7f830159a700 1 -- 172.21.15.16:0/2646086046 >> v2:172.21.15.49:3300/0 conn(0x7f830406e070 msgr2=0x7f8304166350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:27.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.941+0000 7f830159a700 1 --2- 172.21.15.16:0/2646086046 >> v2:172.21.15.49:3300/0 conn(0x7f830406e070 0x7f8304166350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:27.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.941+0000 7f830159a700 1 -- 172.21.15.16:0/2646086046 >> v1:172.21.15.49:6789/0 conn(0x7f83040cda60 legacy=0x7f830406d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:27.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.941+0000 7f830159a700 1 -- 172.21.15.16:0/2646086046 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f830415c6f0 con 0x7f830415b790 2023-10-16T12:11:27.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.942+0000 7f830159a700 1 -- 172.21.15.16:0/2646086046 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f82f4001670 con 0x7f830415b790 2023-10-16T12:11:27.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.942+0000 7f830159a700 1 -- 172.21.15.16:0/2646086046 >> v1:172.21.15.16:6789/0 conn(0x7f830415b790 legacy=0x7f830415bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:27.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.942+0000 7f830159a700 1 --2- 172.21.15.16:0/2646086046 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f82ec003b90 0x7f82ec003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:27.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.942+0000 7f830159a700 1 --2- 172.21.15.16:0/2646086046 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f82ec004850 0x7f82ec004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:27.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.942+0000 7f830159a700 1 --2- 172.21.15.16:0/2646086046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82ec005210 0x7f82ec00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:27.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.942+0000 7f830159a700 1 -- 172.21.15.16:0/2646086046 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f830415c410 con 0x7f82ec005210 2023-10-16T12:11:27.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.942+0000 7f830159a700 1 -- 172.21.15.16:0/2646086046 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82ec00c090 con 0x7f82ec004850 2023-10-16T12:11:27.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.942+0000 7f830159a700 1 -- 172.21.15.16:0/2646086046 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82ec00c250 con 0x7f82ec003b90 2023-10-16T12:11:27.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.942+0000 7f8302d9d700 1 --2- 172.21.15.16:0/2646086046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82ec005210 0x7f82ec00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:27.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.942+0000 7f830259c700 1 --2- 172.21.15.16:0/2646086046 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f82ec003b90 0x7f82ec003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:27.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.942+0000 7f8301d9b700 1 --2- 172.21.15.16:0/2646086046 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f82ec004850 0x7f82ec004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:27.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.942+0000 7f8302d9d700 1 -- 172.21.15.16:0/2646086046 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f82ec003b90 msgr2=0x7f82ec003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:27.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.942+0000 7f8302d9d700 1 --2- 172.21.15.16:0/2646086046 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f82ec003b90 0x7f82ec003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:27.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.942+0000 7f8302d9d700 1 -- 172.21.15.16:0/2646086046 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f82ec004850 msgr2=0x7f82ec004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:27.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.942+0000 7f8302d9d700 1 --2- 172.21.15.16:0/2646086046 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f82ec004850 0x7f82ec004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:27.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.942+0000 7f8302d9d700 1 -- 172.21.15.16:0/2646086046 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82ec0035b0 con 0x7f82ec005210 2023-10-16T12:11:27.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.942+0000 7f8302d9d700 1 -- 172.21.15.16:0/2646086046 reap_dead start 2023-10-16T12:11:27.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.942+0000 7f8302d9d700 1 --2- 172.21.15.16:0/2646086046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82ec005210 0x7f82ec00ba10 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f82fc009fc0 tx=0x7f82ec005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de0d701f90463d83 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:27.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.942+0000 7f830159a700 1 -- 172.21.15.16:0/2646086046 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82fc003a20 con 0x7f82ec005210 2023-10-16T12:11:27.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.942+0000 7f830159a700 1 -- 172.21.15.16:0/2646086046 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82fc00f040 con 0x7f82ec005210 2023-10-16T12:11:27.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.942+0000 7f830159a700 1 -- 172.21.15.16:0/2646086046 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82fc003de0 con 0x7f82ec005210 2023-10-16T12:11:27.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.943+0000 7f8308d6e700 1 -- 172.21.15.16:0/2646086046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82ec005210 msgr2=0x7f82ec00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:27.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.943+0000 7f8308d6e700 1 --2- 172.21.15.16:0/2646086046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82ec005210 0x7f82ec00ba10 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f82fc009fc0 tx=0x7f82ec005880 comp rx=0 tx=0).stop 2023-10-16T12:11:27.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.943+0000 7f8308d6e700 1 -- 172.21.15.16:0/2646086046 shutdown_connections 2023-10-16T12:11:27.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.943+0000 7f8308d6e700 1 --2- 172.21.15.16:0/2646086046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82ec005210 0x7f82ec00ba10 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:27.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.943+0000 7f8308d6e700 1 -- 172.21.15.16:0/2646086046 >> 172.21.15.16:0/2646086046 conn(0x7f83040cc5e0 msgr2=0x7f83040cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:27.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.943+0000 7f8308d6e700 1 -- 172.21.15.16:0/2646086046 shutdown_connections 2023-10-16T12:11:27.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.943+0000 7f8308d6e700 1 -- 172.21.15.16:0/2646086046 wait complete. 2023-10-16T12:11:27.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.944+0000 7f8308d6e700 1 Processor -- start 2023-10-16T12:11:27.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.944+0000 7f8308d6e700 1 -- start start 2023-10-16T12:11:27.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.945+0000 7f8308d6e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f82ec005210 0x7f83040c97c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:27.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.945+0000 7f8308d6e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83040b0d40 0x7f8304071360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:27.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.945+0000 7f8308d6e700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8304154540 0x7f83041549c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:27.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.945+0000 7f8308d6e700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83040ca1b0 con 0x7f83040b0d40 2023-10-16T12:11:27.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.945+0000 7f8308d6e700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83040b11c0 con 0x7f8304154540 2023-10-16T12:11:27.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.945+0000 7f8308d6e700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8304154f00 con 0x7f82ec005210 2023-10-16T12:11:27.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.945+0000 7f8301d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83040b0d40 0x7f8304071360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:27.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.945+0000 7f8301d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83040b0d40 0x7f8304071360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34382/0 (socket says 172.21.15.16:34382) 2023-10-16T12:11:27.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.945+0000 7f8301d9b700 1 -- 172.21.15.16:0/164917186 learned_addr learned my addr 172.21.15.16:0/164917186 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:27.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.945+0000 7f8302d9d700 1 --2- 172.21.15.16:0/164917186 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8304154540 0x7f83041549c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:27.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.945+0000 7f830259c700 1 --2- 172.21.15.16:0/164917186 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f82ec005210 0x7f83040c97c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:27.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.946+0000 7f8301d9b700 1 -- 172.21.15.16:0/164917186 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f82ec005210 msgr2=0x7f83040c97c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:27.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.946+0000 7f8301d9b700 1 --2- 172.21.15.16:0/164917186 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f82ec005210 0x7f83040c97c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:27.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.946+0000 7f8301d9b700 1 -- 172.21.15.16:0/164917186 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8304154540 msgr2=0x7f83041549c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:27.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.946+0000 7f8301d9b700 1 --2- 172.21.15.16:0/164917186 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8304154540 0x7f83041549c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:27.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.946+0000 7f8301d9b700 1 -- 172.21.15.16:0/164917186 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83040b1440 con 0x7f83040b0d40 2023-10-16T12:11:27.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.946+0000 7f830259c700 1 --2- 172.21.15.16:0/164917186 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f82ec005210 0x7f83040c97c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:11:27.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.946+0000 7f8302d9d700 1 --2- 172.21.15.16:0/164917186 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8304154540 0x7f83041549c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:11:27.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.946+0000 7f8301d9b700 1 --2- 172.21.15.16:0/164917186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83040b0d40 0x7f8304071360 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f82f800f7a0 tx=0x7f82f800fb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:27.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.946+0000 7f82f2ffd700 1 -- 172.21.15.16:0/164917186 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82f8005320 con 0x7f83040b0d40 2023-10-16T12:11:27.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.946+0000 7f82f2ffd700 1 -- 172.21.15.16:0/164917186 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82f80054a0 con 0x7f83040b0d40 2023-10-16T12:11:27.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.946+0000 7f8308d6e700 1 -- 172.21.15.16:0/164917186 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83041f76b0 con 0x7f83040b0d40 2023-10-16T12:11:27.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.946+0000 7f82f2ffd700 1 -- 172.21.15.16:0/164917186 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82f8006aa0 con 0x7f83040b0d40 2023-10-16T12:11:27.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.946+0000 7f8308d6e700 1 -- 172.21.15.16:0/164917186 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83041f6bd0 con 0x7f83040b0d40 2023-10-16T12:11:27.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.947+0000 7f8308d6e700 1 -- 172.21.15.16:0/164917186 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f830404ebb0 con 0x7f83040b0d40 2023-10-16T12:11:27.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.947+0000 7f82f2ffd700 1 -- 172.21.15.16:0/164917186 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f82f8006c00 con 0x7f83040b0d40 2023-10-16T12:11:27.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.948+0000 7f82f2ffd700 1 --2- 172.21.15.16:0/164917186 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f82e406af00 0x7f82e406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:27.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.948+0000 7f82f2ffd700 1 -- 172.21.15.16:0/164917186 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f82f8088f90 con 0x7f83040b0d40 2023-10-16T12:11:27.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.948+0000 7f830259c700 1 --2- 172.21.15.16:0/164917186 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f82e406af00 0x7f82e406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:27.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.948+0000 7f830259c700 1 --2- 172.21.15.16:0/164917186 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f82e406af00 0x7f82e406d3c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f82f4001eb0 tx=0x7f82f4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:27.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:27.952+0000 7f82f2ffd700 1 -- 172.21.15.16:0/164917186 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f82f80bbfe0 con 0x7f83040b0d40 2023-10-16T12:11:28.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.154+0000 7f8308d6e700 1 -- 172.21.15.16:0/164917186 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f830406f660 con 0x7f83040b0d40 2023-10-16T12:11:28.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.154+0000 7f82f2ffd700 1 -- 172.21.15.16:0/164917186 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) v1 ==== 103+0+11 (secure 0 0 0) 0x7f82f8053c00 con 0x7f83040b0d40 2023-10-16T12:11:28.156 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-10-16T12:11:28.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.156+0000 7f8308d6e700 1 -- 172.21.15.16:0/164917186 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f82e406af00 msgr2=0x7f82e406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:28.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.156+0000 7f8308d6e700 1 --2- 172.21.15.16:0/164917186 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f82e406af00 0x7f82e406d3c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f82f4001eb0 tx=0x7f82f4009040 comp rx=0 tx=0).stop 2023-10-16T12:11:28.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.156+0000 7f8308d6e700 1 -- 172.21.15.16:0/164917186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83040b0d40 msgr2=0x7f8304071360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:28.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.156+0000 7f8308d6e700 1 --2- 172.21.15.16:0/164917186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83040b0d40 0x7f8304071360 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f82f800f7a0 tx=0x7f82f800fb60 comp rx=0 tx=0).stop 2023-10-16T12:11:28.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.156+0000 7f8308d6e700 1 -- 172.21.15.16:0/164917186 shutdown_connections 2023-10-16T12:11:28.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.156+0000 7f8308d6e700 1 --2- 172.21.15.16:0/164917186 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f82e406af00 0x7f82e406d3c0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:28.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.156+0000 7f8308d6e700 1 --2- 172.21.15.16:0/164917186 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8304154540 0x7f83041549c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:28.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.156+0000 7f8308d6e700 1 --2- 172.21.15.16:0/164917186 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f82ec005210 0x7f83040c97c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:28.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.156+0000 7f8308d6e700 1 --2- 172.21.15.16:0/164917186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83040b0d40 0x7f8304071360 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:28.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.156+0000 7f8308d6e700 1 -- 172.21.15.16:0/164917186 >> 172.21.15.16:0/164917186 conn(0x7f83040cc5e0 msgr2=0x7f830406bb70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:28.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.156+0000 7f8308d6e700 1 -- 172.21.15.16:0/164917186 shutdown_connections 2023-10-16T12:11:28.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.156+0000 7f8308d6e700 1 -- 172.21.15.16:0/164917186 wait complete. 2023-10-16T12:11:28.170 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:11:28.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.300+0000 7f50dd412700 1 Processor -- start 2023-10-16T12:11:28.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.300+0000 7f50dd412700 1 -- start start 2023-10-16T12:11:28.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.301+0000 7f50dd412700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f50d80b3440 0x7f50d8071df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:28.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.301+0000 7f50dd412700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50d815e7d0 con 0x7f50d806c800 2023-10-16T12:11:28.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.301+0000 7f50dd412700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50d815d900 con 0x7f50d8159bf0 2023-10-16T12:11:28.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.301+0000 7f50dd412700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50d8070e80 con 0x7f50d80b3440 2023-10-16T12:11:28.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.301+0000 7f50d77fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f50d806c800 0x7f50d80cc640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51898/0 (socket says 172.21.15.16:51898) 2023-10-16T12:11:28.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.301+0000 7f50d77fe700 1 -- 172.21.15.16:0/593877773 learned_addr learned my addr 172.21.15.16:0/593877773 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:28.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.302+0000 7f50d5ffb700 1 -- 172.21.15.16:0/593877773 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3330750694 0 0) 0x7f50d815e7d0 con 0x7f50d806c800 2023-10-16T12:11:28.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.302+0000 7f50d5ffb700 1 -- 172.21.15.16:0/593877773 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50c40034d0 con 0x7f50d806c800 2023-10-16T12:11:28.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.302+0000 7f50d67fc700 1 --2- 172.21.15.16:0/593877773 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f50d80b3440 0x7f50d8071df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:28.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.302+0000 7f50d5ffb700 1 -- 172.21.15.16:0/593877773 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 57985436 0 0) 0x7f50c40034d0 con 0x7f50d806c800 2023-10-16T12:11:28.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.302+0000 7f50d5ffb700 1 -- 172.21.15.16:0/593877773 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f50d80b3440 msgr2=0x7f50d8071df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:28.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.302+0000 7f50d5ffb700 1 --2- 172.21.15.16:0/593877773 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f50d80b3440 0x7f50d8071df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:28.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.302+0000 7f50d5ffb700 1 -- 172.21.15.16:0/593877773 >> v1:172.21.15.49:6789/0 conn(0x7f50d8159bf0 legacy=0x7f50d815e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:28.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.302+0000 7f50d5ffb700 1 -- 172.21.15.16:0/593877773 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50d80b4310 con 0x7f50d806c800 2023-10-16T12:11:28.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.302+0000 7f50d5ffb700 1 -- 172.21.15.16:0/593877773 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f50d00019f0 con 0x7f50d806c800 2023-10-16T12:11:28.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.302+0000 7f50d5ffb700 1 -- 172.21.15.16:0/593877773 >> v1:172.21.15.16:6789/0 conn(0x7f50d806c800 legacy=0x7f50d80cc640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:28.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.302+0000 7f50d5ffb700 1 --2- 172.21.15.16:0/593877773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50c4003b20 0x7f50c4003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:28.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.302+0000 7f50d5ffb700 1 --2- 172.21.15.16:0/593877773 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f50d80b3440 0x7f50c400a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:28.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.302+0000 7f50d5ffb700 1 --2- 172.21.15.16:0/593877773 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f50c400b030 0x7f50c400d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:28.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.302+0000 7f50d5ffb700 1 -- 172.21.15.16:0/593877773 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50d8070e80 con 0x7f50c4003b20 2023-10-16T12:11:28.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.302+0000 7f50d5ffb700 1 -- 172.21.15.16:0/593877773 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50c400dcd0 con 0x7f50c400b030 2023-10-16T12:11:28.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.302+0000 7f50d5ffb700 1 -- 172.21.15.16:0/593877773 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50c400de90 con 0x7f50d80b3440 2023-10-16T12:11:28.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.302+0000 7f50d6ffd700 1 --2- 172.21.15.16:0/593877773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50c4003b20 0x7f50c4003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:28.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.302+0000 7f50d67fc700 1 --2- 172.21.15.16:0/593877773 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f50d80b3440 0x7f50c400a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:28.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.302+0000 7f50d77fe700 1 --2- 172.21.15.16:0/593877773 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f50c400b030 0x7f50c400d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:28.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.302+0000 7f50d6ffd700 1 -- 172.21.15.16:0/593877773 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f50d80b3440 msgr2=0x7f50c400a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:28.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.302+0000 7f50d6ffd700 1 --2- 172.21.15.16:0/593877773 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f50d80b3440 0x7f50c400a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:28.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.302+0000 7f50d6ffd700 1 -- 172.21.15.16:0/593877773 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f50c400b030 msgr2=0x7f50c400d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:28.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.302+0000 7f50d6ffd700 1 --2- 172.21.15.16:0/593877773 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f50c400b030 0x7f50c400d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:28.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.303+0000 7f50d6ffd700 1 -- 172.21.15.16:0/593877773 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50c40034d0 con 0x7f50c4003b20 2023-10-16T12:11:28.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.303+0000 7f50d77fe700 1 --2- 172.21.15.16:0/593877773 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f50c400b030 0x7f50c400d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:11:28.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.303+0000 7f50d6ffd700 1 --2- 172.21.15.16:0/593877773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50c4003b20 0x7f50c4003f40 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f50c80017c0 tx=0x7f50c4003780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c18fc8a88749011 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:28.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.303+0000 7f50d5ffb700 1 -- 172.21.15.16:0/593877773 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c80052d0 con 0x7f50c4003b20 2023-10-16T12:11:28.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.303+0000 7f50d5ffb700 1 -- 172.21.15.16:0/593877773 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50c8005430 con 0x7f50c4003b20 2023-10-16T12:11:28.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.303+0000 7f50d5ffb700 1 -- 172.21.15.16:0/593877773 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c80038c0 con 0x7f50c4003b20 2023-10-16T12:11:28.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.303+0000 7f50dd412700 1 -- 172.21.15.16:0/593877773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50c4003b20 msgr2=0x7f50c4003f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:28.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.303+0000 7f50dd412700 1 --2- 172.21.15.16:0/593877773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50c4003b20 0x7f50c4003f40 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f50c80017c0 tx=0x7f50c4003780 comp rx=0 tx=0).stop 2023-10-16T12:11:28.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.303+0000 7f50d77fe700 1 -- 172.21.15.16:0/593877773 reap_dead start 2023-10-16T12:11:28.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.303+0000 7f50dd412700 1 -- 172.21.15.16:0/593877773 shutdown_connections 2023-10-16T12:11:28.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.303+0000 7f50dd412700 1 -- 172.21.15.16:0/593877773 >> 172.21.15.16:0/593877773 conn(0x7f50d80b4cd0 msgr2=0x7f50d806a7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:28.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.303+0000 7f50dd412700 1 -- 172.21.15.16:0/593877773 shutdown_connections 2023-10-16T12:11:28.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.303+0000 7f50dd412700 1 -- 172.21.15.16:0/593877773 wait complete. 2023-10-16T12:11:28.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.305+0000 7f50dd412700 1 Processor -- start 2023-10-16T12:11:28.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.305+0000 7f50dd412700 1 -- start start 2023-10-16T12:11:28.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.306+0000 7f50dd412700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f50c4003b20 0x7f50d81430d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:28.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.306+0000 7f50dd412700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f50d81466a0 0x7f50d80c50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:28.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.306+0000 7f50dd412700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50d815aa40 0x7f50d815aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:28.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.306+0000 7f50dd412700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50d8143d70 con 0x7f50d815aa40 2023-10-16T12:11:28.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.306+0000 7f50dd412700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50d815b430 con 0x7f50c4003b20 2023-10-16T12:11:28.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.306+0000 7f50dd412700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50d8143610 con 0x7f50d81466a0 2023-10-16T12:11:28.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.306+0000 7f50d77fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50d815aa40 0x7f50d815aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:28.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.306+0000 7f50d77fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50d815aa40 0x7f50d815aef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34404/0 (socket says 172.21.15.16:34404) 2023-10-16T12:11:28.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.306+0000 7f50d77fe700 1 -- 172.21.15.16:0/1456109817 learned_addr learned my addr 172.21.15.16:0/1456109817 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:28.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.306+0000 7f50d6ffd700 1 --2- 172.21.15.16:0/1456109817 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f50c4003b20 0x7f50d81430d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:28.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.306+0000 7f50d67fc700 1 --2- 172.21.15.16:0/1456109817 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f50d81466a0 0x7f50d80c50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:28.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.306+0000 7f50d77fe700 1 -- 172.21.15.16:0/1456109817 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f50d81466a0 msgr2=0x7f50d80c50e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:28.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.306+0000 7f50d77fe700 1 --2- 172.21.15.16:0/1456109817 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f50d81466a0 0x7f50d80c50e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:28.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.306+0000 7f50d77fe700 1 -- 172.21.15.16:0/1456109817 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f50c4003b20 msgr2=0x7f50d81430d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:28.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.306+0000 7f50d77fe700 1 --2- 172.21.15.16:0/1456109817 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f50c4003b20 0x7f50d81430d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:28.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.306+0000 7f50d77fe700 1 -- 172.21.15.16:0/1456109817 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50d80c5a50 con 0x7f50d815aa40 2023-10-16T12:11:28.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.306+0000 7f50d77fe700 1 --2- 172.21.15.16:0/1456109817 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50d815aa40 0x7f50d815aef0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f50d806d670 tx=0x7f50d000e440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:28.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.307+0000 7f50cf7fe700 1 -- 172.21.15.16:0/1456109817 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50d000ec10 con 0x7f50d815aa40 2023-10-16T12:11:28.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.307+0000 7f50cf7fe700 1 -- 172.21.15.16:0/1456109817 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50d000ed70 con 0x7f50d815aa40 2023-10-16T12:11:28.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.307+0000 7f50cf7fe700 1 -- 172.21.15.16:0/1456109817 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50d000eee0 con 0x7f50d815aa40 2023-10-16T12:11:28.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.307+0000 7f50dd412700 1 -- 172.21.15.16:0/1456109817 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50d80c5d30 con 0x7f50d815aa40 2023-10-16T12:11:28.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.307+0000 7f50dd412700 1 -- 172.21.15.16:0/1456109817 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50d8143990 con 0x7f50d815aa40 2023-10-16T12:11:28.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.308+0000 7f50dd412700 1 -- 172.21.15.16:0/1456109817 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50d8143eb0 con 0x7f50d815aa40 2023-10-16T12:11:28.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.308+0000 7f50cf7fe700 1 -- 172.21.15.16:0/1456109817 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f50d0013660 con 0x7f50d815aa40 2023-10-16T12:11:28.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.309+0000 7f50cf7fe700 1 --2- 172.21.15.16:0/1456109817 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f50b806af00 0x7f50b806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:28.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.309+0000 7f50d6ffd700 1 --2- 172.21.15.16:0/1456109817 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f50b806af00 0x7f50b806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:28.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.309+0000 7f50cf7fe700 1 -- 172.21.15.16:0/1456109817 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f50d0089ea0 con 0x7f50d815aa40 2023-10-16T12:11:28.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.309+0000 7f50d6ffd700 1 --2- 172.21.15.16:0/1456109817 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f50b806af00 0x7f50b806d3c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f50c8000c00 tx=0x7f50c8009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:28.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.313+0000 7f50cf7fe700 1 -- 172.21.15.16:0/1456109817 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f50d8143eb0 con 0x7f50d815aa40 2023-10-16T12:11:28.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.520+0000 7f50dd412700 1 -- 172.21.15.16:0/1456109817 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f50d8143eb0 con 0x7f50d815aa40 2023-10-16T12:11:28.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.521+0000 7f50cf7fe700 1 -- 172.21.15.16:0/1456109817 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18629 (secure 0 0 0) 0x7f50d8143eb0 con 0x7f50d815aa40 2023-10-16T12:11:28.524 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:11:28.524 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:11:28.241865+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:10:36.935773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T12:10:38.004870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-16T12:10:41.672025+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"3.2b","mappings":[{"from":0,"to":2}]},{"pgid":"3.38","mappings":[{"from":3,"to":1}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]},{"pgid":"4.1b","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:11:28.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.523+0000 7f50dd412700 1 -- 172.21.15.16:0/1456109817 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f50b806af00 msgr2=0x7f50b806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:28.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.523+0000 7f50dd412700 1 --2- 172.21.15.16:0/1456109817 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f50b806af00 0x7f50b806d3c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f50c8000c00 tx=0x7f50c8009030 comp rx=0 tx=0).stop 2023-10-16T12:11:28.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.523+0000 7f50dd412700 1 -- 172.21.15.16:0/1456109817 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50d815aa40 msgr2=0x7f50d815aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:28.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.523+0000 7f50dd412700 1 --2- 172.21.15.16:0/1456109817 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50d815aa40 0x7f50d815aef0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f50d806d670 tx=0x7f50d000e440 comp rx=0 tx=0).stop 2023-10-16T12:11:28.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.524+0000 7f50dd412700 1 -- 172.21.15.16:0/1456109817 shutdown_connections 2023-10-16T12:11:28.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.524+0000 7f50dd412700 1 --2- 172.21.15.16:0/1456109817 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f50b806af00 0x7f50b806d3c0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:28.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.524+0000 7f50dd412700 1 --2- 172.21.15.16:0/1456109817 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50d815aa40 0x7f50d815aef0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:28.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.524+0000 7f50dd412700 1 --2- 172.21.15.16:0/1456109817 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f50c4003b20 0x7f50d81430d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:28.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.524+0000 7f50dd412700 1 --2- 172.21.15.16:0/1456109817 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f50d81466a0 0x7f50d80c50e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:28.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.524+0000 7f50dd412700 1 -- 172.21.15.16:0/1456109817 >> 172.21.15.16:0/1456109817 conn(0x7f50d80b4cd0 msgr2=0x7f50d80ce610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:28.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.524+0000 7f50dd412700 1 -- 172.21.15.16:0/1456109817 shutdown_connections 2023-10-16T12:11:28.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.524+0000 7f50dd412700 1 -- 172.21.15.16:0/1456109817 wait complete. 2023-10-16T12:11:28.539 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T12:10:36.935773+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T12:10:38.004870+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-16T12:10:41.672025+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-16T12:11:28.539 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:11:28.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.666+0000 7f2272e63700 1 Processor -- start 2023-10-16T12:11:28.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.667+0000 7f2272e63700 1 -- start start 2023-10-16T12:11:28.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.668+0000 7f2272e63700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f226c0cb880 0x7f226c0cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:28.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.668+0000 7f2272e63700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f226c06da60 0x7f226c06def0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:28.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.668+0000 7f2272e63700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f226c0cef70 con 0x7f226c15b070 2023-10-16T12:11:28.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.668+0000 7f2272e63700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f226c06e430 con 0x7f226c06da60 2023-10-16T12:11:28.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.668+0000 7f2272e63700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f226c0ce360 con 0x7f226c0cb880 2023-10-16T12:11:28.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.668+0000 7f2270bff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f226c15b070 0x7f226c15b450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51914/0 (socket says 172.21.15.16:51914) 2023-10-16T12:11:28.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.668+0000 7f2270bff700 1 -- 172.21.15.16:0/1541359691 learned_addr learned my addr 172.21.15.16:0/1541359691 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:28.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.668+0000 7f226b7fe700 1 -- 172.21.15.16:0/1541359691 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3692226230 0 0) 0x7f226c0cef70 con 0x7f226c15b070 2023-10-16T12:11:28.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.668+0000 7f226bfff700 1 --2- 172.21.15.16:0/1541359691 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f226c0cb880 0x7f226c0cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:28.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.668+0000 7f2271400700 1 --2- 172.21.15.16:0/1541359691 >> v2:172.21.15.49:3300/0 conn(0x7f226c06da60 0x7f226c06def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:28.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.668+0000 7f226b7fe700 1 -- 172.21.15.16:0/1541359691 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f225c0035b0 con 0x7f226c15b070 2023-10-16T12:11:28.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.668+0000 7f226b7fe700 1 -- 172.21.15.16:0/1541359691 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 44934756 0 0) 0x7f225c0035b0 con 0x7f226c15b070 2023-10-16T12:11:28.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.668+0000 7f226b7fe700 1 -- 172.21.15.16:0/1541359691 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f226c0cb880 msgr2=0x7f226c0cbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:28.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.668+0000 7f226b7fe700 1 --2- 172.21.15.16:0/1541359691 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f226c0cb880 0x7f226c0cbd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:28.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.668+0000 7f226b7fe700 1 -- 172.21.15.16:0/1541359691 >> v2:172.21.15.49:3300/0 conn(0x7f226c06da60 msgr2=0x7f226c06def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:28.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.668+0000 7f226b7fe700 1 --2- 172.21.15.16:0/1541359691 >> v2:172.21.15.49:3300/0 conn(0x7f226c06da60 0x7f226c06def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:28.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.668+0000 7f226b7fe700 1 -- 172.21.15.16:0/1541359691 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f226c0ce640 con 0x7f226c15b070 2023-10-16T12:11:28.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.668+0000 7f226b7fe700 1 -- 172.21.15.16:0/1541359691 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f2260001670 con 0x7f226c15b070 2023-10-16T12:11:28.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.668+0000 7f226b7fe700 1 -- 172.21.15.16:0/1541359691 >> v1:172.21.15.16:6789/0 conn(0x7f226c15b070 legacy=0x7f226c15b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:28.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.668+0000 7f226b7fe700 1 --2- 172.21.15.16:0/1541359691 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f225c003a00 0x7f225c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:28.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.668+0000 7f226b7fe700 1 --2- 172.21.15.16:0/1541359691 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f226c0cb880 0x7f225c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:28.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.668+0000 7f226b7fe700 1 --2- 172.21.15.16:0/1541359691 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f225c004d90 0x7f225c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:28.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.668+0000 7f226b7fe700 1 -- 172.21.15.16:0/1541359691 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f226c06e430 con 0x7f225c003a00 2023-10-16T12:11:28.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.668+0000 7f226b7fe700 1 -- 172.21.15.16:0/1541359691 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f226c0ce360 con 0x7f225c004d90 2023-10-16T12:11:28.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.668+0000 7f226b7fe700 1 -- 172.21.15.16:0/1541359691 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f225c00ba10 con 0x7f226c0cb880 2023-10-16T12:11:28.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.669+0000 7f2270bff700 1 --2- 172.21.15.16:0/1541359691 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f225c003a00 0x7f225c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:28.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.669+0000 7f226bfff700 1 --2- 172.21.15.16:0/1541359691 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f226c0cb880 0x7f225c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:28.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.669+0000 7f2271400700 1 --2- 172.21.15.16:0/1541359691 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f225c004d90 0x7f225c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:28.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.669+0000 7f2270bff700 1 -- 172.21.15.16:0/1541359691 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f226c0cb880 msgr2=0x7f225c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:28.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.669+0000 7f2270bff700 1 --2- 172.21.15.16:0/1541359691 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f226c0cb880 0x7f225c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:28.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.669+0000 7f2270bff700 1 -- 172.21.15.16:0/1541359691 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f225c004d90 msgr2=0x7f225c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:28.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.669+0000 7f2270bff700 1 --2- 172.21.15.16:0/1541359691 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f225c004d90 0x7f225c005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:28.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.669+0000 7f2270bff700 1 -- 172.21.15.16:0/1541359691 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f225c0035b0 con 0x7f225c003a00 2023-10-16T12:11:28.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.669+0000 7f2270bff700 1 --2- 172.21.15.16:0/1541359691 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f225c003a00 0x7f225c003e20 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f226000c370 tx=0x7f225c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=718e9952cbaf87b8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:28.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.669+0000 7f226b7fe700 1 -- 172.21.15.16:0/1541359691 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2260007c60 con 0x7f225c003a00 2023-10-16T12:11:28.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.669+0000 7f226b7fe700 1 -- 172.21.15.16:0/1541359691 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2260005960 con 0x7f225c003a00 2023-10-16T12:11:28.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.669+0000 7f226b7fe700 1 -- 172.21.15.16:0/1541359691 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2260005d10 con 0x7f225c003a00 2023-10-16T12:11:28.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.670+0000 7f2272e63700 1 -- 172.21.15.16:0/1541359691 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f225c003a00 msgr2=0x7f225c003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:28.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.670+0000 7f2272e63700 1 --2- 172.21.15.16:0/1541359691 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f225c003a00 0x7f225c003e20 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f226000c370 tx=0x7f225c005880 comp rx=0 tx=0).stop 2023-10-16T12:11:28.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.670+0000 7f2271400700 1 -- 172.21.15.16:0/1541359691 reap_dead start 2023-10-16T12:11:28.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.670+0000 7f2272e63700 1 -- 172.21.15.16:0/1541359691 shutdown_connections 2023-10-16T12:11:28.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.670+0000 7f2272e63700 1 -- 172.21.15.16:0/1541359691 >> 172.21.15.16:0/1541359691 conn(0x7f226c0b1f40 msgr2=0x7f226c0b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:28.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.670+0000 7f2272e63700 1 -- 172.21.15.16:0/1541359691 shutdown_connections 2023-10-16T12:11:28.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.670+0000 7f2272e63700 1 -- 172.21.15.16:0/1541359691 wait complete. 2023-10-16T12:11:28.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.671+0000 7f2272e63700 1 Processor -- start 2023-10-16T12:11:28.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.671+0000 7f2272e63700 1 -- start start 2023-10-16T12:11:28.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.672+0000 7f2272e63700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f225c003a00 0x7f226c159200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:28.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.672+0000 7f2272e63700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f226c151dd0 0x7f226c15be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:28.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.672+0000 7f2272e63700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f226c14a420 0x7f226c14a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:28.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.672+0000 7f2272e63700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f226c14ae10 con 0x7f226c151dd0 2023-10-16T12:11:28.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.672+0000 7f2272e63700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f226c15cab0 con 0x7f225c003a00 2023-10-16T12:11:28.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.672+0000 7f2272e63700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f226c15c7d0 con 0x7f226c14a420 2023-10-16T12:11:28.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.672+0000 7f226bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f226c151dd0 0x7f226c15be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:28.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.672+0000 7f226bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f226c151dd0 0x7f226c15be60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34422/0 (socket says 172.21.15.16:34422) 2023-10-16T12:11:28.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.672+0000 7f226bfff700 1 -- 172.21.15.16:0/1389968682 learned_addr learned my addr 172.21.15.16:0/1389968682 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:28.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.672+0000 7f2270bff700 1 --2- 172.21.15.16:0/1389968682 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f225c003a00 0x7f226c159200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:28.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.672+0000 7f2271400700 1 --2- 172.21.15.16:0/1389968682 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f226c14a420 0x7f226c14a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:28.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.673+0000 7f226bfff700 1 -- 172.21.15.16:0/1389968682 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f226c14a420 msgr2=0x7f226c14a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:28.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.673+0000 7f226bfff700 1 --2- 172.21.15.16:0/1389968682 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f226c14a420 0x7f226c14a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:28.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.673+0000 7f226bfff700 1 -- 172.21.15.16:0/1389968682 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f225c003a00 msgr2=0x7f226c159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:28.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.673+0000 7f226bfff700 1 --2- 172.21.15.16:0/1389968682 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f225c003a00 0x7f226c159200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:28.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.673+0000 7f226bfff700 1 -- 172.21.15.16:0/1389968682 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f226c0aca00 con 0x7f226c151dd0 2023-10-16T12:11:28.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.673+0000 7f226bfff700 1 --2- 172.21.15.16:0/1389968682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f226c151dd0 0x7f226c15be60 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f2258011aa0 tx=0x7f2258011db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:28.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.673+0000 7f22697fa700 1 -- 172.21.15.16:0/1389968682 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2258011ed0 con 0x7f226c151dd0 2023-10-16T12:11:28.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.673+0000 7f22697fa700 1 -- 172.21.15.16:0/1389968682 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f225800f940 con 0x7f226c151dd0 2023-10-16T12:11:28.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.673+0000 7f2272e63700 1 -- 172.21.15.16:0/1389968682 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f226c0acce0 con 0x7f226c151dd0 2023-10-16T12:11:28.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.673+0000 7f22697fa700 1 -- 172.21.15.16:0/1389968682 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f225800fcf0 con 0x7f226c151dd0 2023-10-16T12:11:28.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.673+0000 7f2272e63700 1 -- 172.21.15.16:0/1389968682 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f226c159740 con 0x7f226c151dd0 2023-10-16T12:11:28.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.674+0000 7f2272e63700 1 -- 172.21.15.16:0/1389968682 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f226c0ce900 con 0x7f226c151dd0 2023-10-16T12:11:28.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.674+0000 7f22697fa700 1 -- 172.21.15.16:0/1389968682 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f2258014440 con 0x7f226c151dd0 2023-10-16T12:11:28.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.675+0000 7f22697fa700 1 --2- 172.21.15.16:0/1389968682 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f225006af00 0x7f225006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:28.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.675+0000 7f22697fa700 1 -- 172.21.15.16:0/1389968682 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f2258089f50 con 0x7f226c151dd0 2023-10-16T12:11:28.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.675+0000 7f2270bff700 1 --2- 172.21.15.16:0/1389968682 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f225006af00 0x7f225006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:28.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.675+0000 7f2270bff700 1 --2- 172.21.15.16:0/1389968682 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f225006af00 0x7f225006d3c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2260015040 tx=0x7f226000a000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:28.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.679+0000 7f22697fa700 1 -- 172.21.15.16:0/1389968682 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f226c0ce900 con 0x7f226c151dd0 2023-10-16T12:11:28.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.883+0000 7f2272e63700 1 -- 172.21.15.16:0/1389968682 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f226c0ce8d0 con 0x7f226c151dd0 2023-10-16T12:11:28.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.884+0000 7f22697fa700 1 -- 172.21.15.16:0/1389968682 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2258058820 con 0x7f226c151dd0 2023-10-16T12:11:28.886 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T12:11:28.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.886+0000 7f2272e63700 1 -- 172.21.15.16:0/1389968682 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f225006af00 msgr2=0x7f225006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:28.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.886+0000 7f2272e63700 1 --2- 172.21.15.16:0/1389968682 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f225006af00 0x7f225006d3c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2260015040 tx=0x7f226000a000 comp rx=0 tx=0).stop 2023-10-16T12:11:28.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.886+0000 7f2272e63700 1 -- 172.21.15.16:0/1389968682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f226c151dd0 msgr2=0x7f226c15be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:28.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.886+0000 7f2272e63700 1 --2- 172.21.15.16:0/1389968682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f226c151dd0 0x7f226c15be60 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f2258011aa0 tx=0x7f2258011db0 comp rx=0 tx=0).stop 2023-10-16T12:11:28.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.886+0000 7f2272e63700 1 -- 172.21.15.16:0/1389968682 shutdown_connections 2023-10-16T12:11:28.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.886+0000 7f2272e63700 1 --2- 172.21.15.16:0/1389968682 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f225006af00 0x7f225006d3c0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:28.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.886+0000 7f2272e63700 1 --2- 172.21.15.16:0/1389968682 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f226c14a420 0x7f226c14a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:28.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.886+0000 7f2272e63700 1 --2- 172.21.15.16:0/1389968682 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f225c003a00 0x7f226c159200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:28.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.886+0000 7f2272e63700 1 --2- 172.21.15.16:0/1389968682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f226c151dd0 0x7f226c15be60 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:28.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.886+0000 7f2272e63700 1 -- 172.21.15.16:0/1389968682 >> 172.21.15.16:0/1389968682 conn(0x7f226c0b1f40 msgr2=0x7f226c0ccf40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:28.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.886+0000 7f2272e63700 1 -- 172.21.15.16:0/1389968682 shutdown_connections 2023-10-16T12:11:28.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:28.886+0000 7f2272e63700 1 -- 172.21.15.16:0/1389968682 wait complete. 2023-10-16T12:11:28.900 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T12:11:29.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.026+0000 7f7a304be700 1 Processor -- start 2023-10-16T12:11:29.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.026+0000 7f7a304be700 1 -- start start 2023-10-16T12:11:29.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.027+0000 7f7a304be700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7a280cbe20 0x7f7a280cda60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:29.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.027+0000 7f7a304be700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a2815aa20 con 0x7f7a2815a640 2023-10-16T12:11:29.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.027+0000 7f7a304be700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a2806eb40 con 0x7f7a280c6770 2023-10-16T12:11:29.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.027+0000 7f7a304be700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2806ee20 con 0x7f7a280cbe20 2023-10-16T12:11:29.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.027+0000 7f7a2e25a700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f7a2815a640 0x7f7a280cb710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51934/0 (socket says 172.21.15.16:51934) 2023-10-16T12:11:29.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.027+0000 7f7a2e25a700 1 -- 172.21.15.16:0/1671833416 learned_addr learned my addr 172.21.15.16:0/1671833416 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:29.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.027+0000 7f7a2da59700 1 --2- 172.21.15.16:0/1671833416 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7a280cbe20 0x7f7a280cda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:29.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.027+0000 7f7a2d258700 1 -- 172.21.15.16:0/1671833416 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2469608739 0 0) 0x7f7a18001110 con 0x7f7a2815a640 2023-10-16T12:11:29.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.027+0000 7f7a2d258700 1 -- 172.21.15.16:0/1671833416 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a140035b0 con 0x7f7a2815a640 2023-10-16T12:11:29.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.028+0000 7f7a2d258700 1 -- 172.21.15.16:0/1671833416 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1933110070 0 0) 0x7f7a2806eb40 con 0x7f7a280c6770 2023-10-16T12:11:29.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.028+0000 7f7a2d258700 1 -- 172.21.15.16:0/1671833416 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a18001110 con 0x7f7a280c6770 2023-10-16T12:11:29.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.028+0000 7f7a2da59700 1 -- 172.21.15.16:0/1671833416 >> v1:172.21.15.49:6789/0 conn(0x7f7a280c6770 legacy=0x7f7a280cdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:29.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.028+0000 7f7a2da59700 1 -- 172.21.15.16:0/1671833416 >> v1:172.21.15.16:6789/0 conn(0x7f7a2815a640 legacy=0x7f7a280cb710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:29.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.028+0000 7f7a2da59700 1 -- 172.21.15.16:0/1671833416 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a2806f100 con 0x7f7a280cbe20 2023-10-16T12:11:29.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.028+0000 7f7a2da59700 1 --2- 172.21.15.16:0/1671833416 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7a280cbe20 0x7f7a280cda60 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7a24009870 tx=0x7f7a24009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=80d17134e97ae65a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:29.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.028+0000 7f7a2d258700 1 -- 172.21.15.16:0/1671833416 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a24011830 con 0x7f7a280cbe20 2023-10-16T12:11:29.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.028+0000 7f7a2d258700 1 -- 172.21.15.16:0/1671833416 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a24011990 con 0x7f7a280cbe20 2023-10-16T12:11:29.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.028+0000 7f7a2d258700 1 -- 172.21.15.16:0/1671833416 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a24011d00 con 0x7f7a280cbe20 2023-10-16T12:11:29.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.029+0000 7f7a304be700 1 -- 172.21.15.16:0/1671833416 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7a280cbe20 msgr2=0x7f7a280cda60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:29.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.029+0000 7f7a304be700 1 --2- 172.21.15.16:0/1671833416 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7a280cbe20 0x7f7a280cda60 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7a24009870 tx=0x7f7a24009b80 comp rx=0 tx=0).stop 2023-10-16T12:11:29.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.029+0000 7f7a304be700 1 -- 172.21.15.16:0/1671833416 shutdown_connections 2023-10-16T12:11:29.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.029+0000 7f7a304be700 1 --2- 172.21.15.16:0/1671833416 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7a280cbe20 0x7f7a280cda60 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:29.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.029+0000 7f7a304be700 1 -- 172.21.15.16:0/1671833416 >> 172.21.15.16:0/1671833416 conn(0x7f7a280cc5e0 msgr2=0x7f7a280cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:29.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.029+0000 7f7a304be700 1 -- 172.21.15.16:0/1671833416 shutdown_connections 2023-10-16T12:11:29.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.029+0000 7f7a304be700 1 -- 172.21.15.16:0/1671833416 wait complete. 2023-10-16T12:11:29.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.030+0000 7f7a304be700 1 Processor -- start 2023-10-16T12:11:29.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.030+0000 7f7a304be700 1 -- start start 2023-10-16T12:11:29.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.031+0000 7f7a304be700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7a280c6770 0x7f7a28156cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:29.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.031+0000 7f7a304be700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7a280cbe20 0x7f7a28146b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:29.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.031+0000 7f7a304be700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a2815a640 0x7f7a280c8e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:29.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.031+0000 7f7a304be700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2815c0f0 con 0x7f7a2815a640 2023-10-16T12:11:29.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.031+0000 7f7a304be700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a28154db0 con 0x7f7a280cbe20 2023-10-16T12:11:29.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.031+0000 7f7a304be700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2815b7e0 con 0x7f7a280c6770 2023-10-16T12:11:29.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.031+0000 7f7a2ea5b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a2815a640 0x7f7a280c8e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:29.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.031+0000 7f7a2ea5b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a2815a640 0x7f7a280c8e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34438/0 (socket says 172.21.15.16:34438) 2023-10-16T12:11:29.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.031+0000 7f7a2ea5b700 1 -- 172.21.15.16:0/804508597 learned_addr learned my addr 172.21.15.16:0/804508597 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:29.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.031+0000 7f7a2da59700 1 --2- 172.21.15.16:0/804508597 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7a280cbe20 0x7f7a28146b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:29.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.031+0000 7f7a2e25a700 1 --2- 172.21.15.16:0/804508597 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7a280c6770 0x7f7a28156cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:29.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.031+0000 7f7a2ea5b700 1 -- 172.21.15.16:0/804508597 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7a280c6770 msgr2=0x7f7a28156cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:29.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.031+0000 7f7a2ea5b700 1 --2- 172.21.15.16:0/804508597 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7a280c6770 0x7f7a28156cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:29.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.031+0000 7f7a2ea5b700 1 -- 172.21.15.16:0/804508597 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7a280cbe20 msgr2=0x7f7a28146b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:29.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.031+0000 7f7a2ea5b700 1 --2- 172.21.15.16:0/804508597 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7a280cbe20 0x7f7a28146b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:29.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.031+0000 7f7a2ea5b700 1 -- 172.21.15.16:0/804508597 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a240094a0 con 0x7f7a2815a640 2023-10-16T12:11:29.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.031+0000 7f7a2ea5b700 1 --2- 172.21.15.16:0/804508597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a2815a640 0x7f7a280c8e40 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f7a2000cbf0 tx=0x7f7a2000cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:29.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.031+0000 7f7a1effd700 1 -- 172.21.15.16:0/804508597 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a20003c00 con 0x7f7a2815a640 2023-10-16T12:11:29.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.031+0000 7f7a1effd700 1 -- 172.21.15.16:0/804508597 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a20003d60 con 0x7f7a2815a640 2023-10-16T12:11:29.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.031+0000 7f7a304be700 1 -- 172.21.15.16:0/804508597 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a281470d0 con 0x7f7a2815a640 2023-10-16T12:11:29.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.031+0000 7f7a1effd700 1 -- 172.21.15.16:0/804508597 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a2000aa60 con 0x7f7a2815a640 2023-10-16T12:11:29.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.031+0000 7f7a304be700 1 -- 172.21.15.16:0/804508597 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a2815ba30 con 0x7f7a2815a640 2023-10-16T12:11:29.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.032+0000 7f7a304be700 1 -- 172.21.15.16:0/804508597 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a2815c230 con 0x7f7a2815a640 2023-10-16T12:11:29.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.033+0000 7f7a1effd700 1 -- 172.21.15.16:0/804508597 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f7a2000abc0 con 0x7f7a2815a640 2023-10-16T12:11:29.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.033+0000 7f7a1effd700 1 --2- 172.21.15.16:0/804508597 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f7a0c06af00 0x7f7a0c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:29.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.033+0000 7f7a1effd700 1 -- 172.21.15.16:0/804508597 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f7a20089960 con 0x7f7a2815a640 2023-10-16T12:11:29.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.033+0000 7f7a2e25a700 1 --2- 172.21.15.16:0/804508597 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f7a0c06af00 0x7f7a0c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:29.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.034+0000 7f7a2e25a700 1 --2- 172.21.15.16:0/804508597 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f7a0c06af00 0x7f7a0c06d3c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7a18007fd0 tx=0x7f7a18007dc0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:29.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.038+0000 7f7a1effd700 1 -- 172.21.15.16:0/804508597 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7a2815c230 con 0x7f7a2815a640 2023-10-16T12:11:29.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.242+0000 7f7a304be700 1 -- 172.21.15.16:0/804508597 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7a2815c230 con 0x7f7a2815a640 2023-10-16T12:11:29.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.243+0000 7f7a1effd700 1 -- 172.21.15.16:0/804508597 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7a2815c230 con 0x7f7a2815a640 2023-10-16T12:11:29.245 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-10-16T12:11:29.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.245+0000 7f7a304be700 1 -- 172.21.15.16:0/804508597 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f7a0c06af00 msgr2=0x7f7a0c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:29.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.245+0000 7f7a304be700 1 --2- 172.21.15.16:0/804508597 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f7a0c06af00 0x7f7a0c06d3c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7a18007fd0 tx=0x7f7a18007dc0 comp rx=0 tx=0).stop 2023-10-16T12:11:29.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.245+0000 7f7a304be700 1 -- 172.21.15.16:0/804508597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a2815a640 msgr2=0x7f7a280c8e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:29.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.245+0000 7f7a304be700 1 --2- 172.21.15.16:0/804508597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a2815a640 0x7f7a280c8e40 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f7a2000cbf0 tx=0x7f7a2000cf00 comp rx=0 tx=0).stop 2023-10-16T12:11:29.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.245+0000 7f7a304be700 1 -- 172.21.15.16:0/804508597 shutdown_connections 2023-10-16T12:11:29.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.245+0000 7f7a304be700 1 --2- 172.21.15.16:0/804508597 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f7a0c06af00 0x7f7a0c06d3c0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:29.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.245+0000 7f7a304be700 1 --2- 172.21.15.16:0/804508597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a2815a640 0x7f7a280c8e40 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:29.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.245+0000 7f7a304be700 1 --2- 172.21.15.16:0/804508597 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7a280c6770 0x7f7a28156cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:29.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.245+0000 7f7a304be700 1 --2- 172.21.15.16:0/804508597 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7a280cbe20 0x7f7a28146b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:29.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.245+0000 7f7a304be700 1 -- 172.21.15.16:0/804508597 >> 172.21.15.16:0/804508597 conn(0x7f7a280cc5e0 msgr2=0x7f7a280ce8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:29.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.245+0000 7f7a304be700 1 -- 172.21.15.16:0/804508597 shutdown_connections 2023-10-16T12:11:29.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.245+0000 7f7a304be700 1 -- 172.21.15.16:0/804508597 wait complete. 2023-10-16T12:11:29.260 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T12:11:29.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.387+0000 7f57fa3a0700 1 Processor -- start 2023-10-16T12:11:29.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.387+0000 7f57fa3a0700 1 -- start start 2023-10-16T12:11:29.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.388+0000 7f57fa3a0700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f57f40cda60 0x7f57f406d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:29.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.388+0000 7f57fa3a0700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f57f406dea0 0x7f57f40ce850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:29.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.388+0000 7f57fa3a0700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57f40ce0e0 con 0x7f57f415b790 2023-10-16T12:11:29.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.388+0000 7f57fa3a0700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57f406e2d0 con 0x7f57f40cda60 2023-10-16T12:11:29.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.388+0000 7f57fa3a0700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57f406a7e0 con 0x7f57f406dea0 2023-10-16T12:11:29.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.388+0000 7f57f37fe700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f57f40cda60 0x7f57f406d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:29.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.388+0000 7f57f893d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f57f406dea0 0x7f57f40ce850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:29.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.388+0000 7f57f3fff700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7f57f415b790 0x7f57f415bb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:59130/0 (socket says 172.21.15.16:59130) 2023-10-16T12:11:29.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.388+0000 7f57f3fff700 1 -- 172.21.15.16:0/308475846 learned_addr learned my addr 172.21.15.16:0/308475846 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:29.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.388+0000 7f57f37fe700 1 --2- 172.21.15.16:0/308475846 >> v2:172.21.15.49:3300/0 conn(0x7f57f40cda60 0x7f57f406d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:33506/0 (socket says 172.21.15.16:33506) 2023-10-16T12:11:29.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.388+0000 7f57f893d700 1 -- 172.21.15.16:0/308475846 >> v2:172.21.15.49:3300/0 conn(0x7f57f40cda60 msgr2=0x7f57f406d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:29.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.388+0000 7f57f893d700 1 --2- 172.21.15.16:0/308475846 >> v2:172.21.15.49:3300/0 conn(0x7f57f40cda60 0x7f57f406d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:29.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.388+0000 7f57f893d700 1 -- 172.21.15.16:0/308475846 >> v1:172.21.15.49:6789/0 conn(0x7f57f415b790 legacy=0x7f57f415bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:29.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.388+0000 7f57f893d700 1 -- 172.21.15.16:0/308475846 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57f415c410 con 0x7f57f406dea0 2023-10-16T12:11:29.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.388+0000 7f57f37fe700 1 --2- 172.21.15.16:0/308475846 >> v2:172.21.15.49:3300/0 conn(0x7f57f40cda60 0x7f57f406d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:11:29.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.389+0000 7f57f893d700 1 --2- 172.21.15.16:0/308475846 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f57f406dea0 0x7f57f40ce850 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f57ec009d90 tx=0x7f57ec00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4053e19104adaf7f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:29.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.389+0000 7f57f2ffd700 1 -- 172.21.15.16:0/308475846 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57ec00e070 con 0x7f57f406dea0 2023-10-16T12:11:29.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.389+0000 7f57f2ffd700 1 -- 172.21.15.16:0/308475846 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57ec00bb20 con 0x7f57f406dea0 2023-10-16T12:11:29.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.389+0000 7f57f2ffd700 1 -- 172.21.15.16:0/308475846 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57ec012430 con 0x7f57f406dea0 2023-10-16T12:11:29.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.389+0000 7f57fa3a0700 1 -- 172.21.15.16:0/308475846 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f57f406dea0 msgr2=0x7f57f40ce850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:29.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.389+0000 7f57fa3a0700 1 --2- 172.21.15.16:0/308475846 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f57f406dea0 0x7f57f40ce850 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f57ec009d90 tx=0x7f57ec00b330 comp rx=0 tx=0).stop 2023-10-16T12:11:29.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.389+0000 7f57fa3a0700 1 -- 172.21.15.16:0/308475846 shutdown_connections 2023-10-16T12:11:29.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.389+0000 7f57fa3a0700 1 --2- 172.21.15.16:0/308475846 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f57f406dea0 0x7f57f40ce850 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:29.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.389+0000 7f57fa3a0700 1 --2- 172.21.15.16:0/308475846 >> v2:172.21.15.49:3300/0 conn(0x7f57f40cda60 0x7f57f406d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:29.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.389+0000 7f57fa3a0700 1 -- 172.21.15.16:0/308475846 >> 172.21.15.16:0/308475846 conn(0x7f57f40cc5e0 msgr2=0x7f57f40cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:29.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.389+0000 7f57fa3a0700 1 -- 172.21.15.16:0/308475846 shutdown_connections 2023-10-16T12:11:29.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.389+0000 7f57fa3a0700 1 -- 172.21.15.16:0/308475846 wait complete. 2023-10-16T12:11:29.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.390+0000 7f57fa3a0700 1 Processor -- start 2023-10-16T12:11:29.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.391+0000 7f57fa3a0700 1 -- start start 2023-10-16T12:11:29.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.391+0000 7f57fa3a0700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f57f406dea0 0x7f57f415d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:29.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.391+0000 7f57fa3a0700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f57f40cda60 0x7f57f415f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:29.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.391+0000 7f57fa3a0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57f415b790 0x7f57f415f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:29.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.391+0000 7f57fa3a0700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57f4144b70 con 0x7f57f415b790 2023-10-16T12:11:29.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.391+0000 7f57fa3a0700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57f415da70 con 0x7f57f40cda60 2023-10-16T12:11:29.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.391+0000 7f57fa3a0700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57f4143ca0 con 0x7f57f406dea0 2023-10-16T12:11:29.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.392+0000 7f57f893d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57f415b790 0x7f57f415f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:29.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.392+0000 7f57f893d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57f415b790 0x7f57f415f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34452/0 (socket says 172.21.15.16:34452) 2023-10-16T12:11:29.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.392+0000 7f57f893d700 1 -- 172.21.15.16:0/2589366115 learned_addr learned my addr 172.21.15.16:0/2589366115 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:29.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.392+0000 7f57f3fff700 1 --2- 172.21.15.16:0/2589366115 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f57f406dea0 0x7f57f415d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:29.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.392+0000 7f57f37fe700 1 --2- 172.21.15.16:0/2589366115 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f57f40cda60 0x7f57f415f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:29.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.392+0000 7f57f893d700 1 -- 172.21.15.16:0/2589366115 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f57f406dea0 msgr2=0x7f57f415d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:29.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.392+0000 7f57f893d700 1 --2- 172.21.15.16:0/2589366115 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f57f406dea0 0x7f57f415d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:29.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.392+0000 7f57f893d700 1 -- 172.21.15.16:0/2589366115 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f57f40cda60 msgr2=0x7f57f415f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:29.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.392+0000 7f57f893d700 1 --2- 172.21.15.16:0/2589366115 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f57f40cda60 0x7f57f415f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:29.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.392+0000 7f57f893d700 1 -- 172.21.15.16:0/2589366115 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57e80094a0 con 0x7f57f415b790 2023-10-16T12:11:29.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.392+0000 7f57f893d700 1 --2- 172.21.15.16:0/2589366115 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57f415b790 0x7f57f415f670 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f57ec014040 tx=0x7f57ec004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:29.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.392+0000 7f57f0ff9700 1 -- 172.21.15.16:0/2589366115 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57ec012430 con 0x7f57f415b790 2023-10-16T12:11:29.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.392+0000 7f57f0ff9700 1 -- 172.21.15.16:0/2589366115 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57ec020070 con 0x7f57f415b790 2023-10-16T12:11:29.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.392+0000 7f57fa3a0700 1 -- 172.21.15.16:0/2589366115 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57ec009a70 con 0x7f57f415b790 2023-10-16T12:11:29.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.392+0000 7f57f0ff9700 1 -- 172.21.15.16:0/2589366115 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57ec004280 con 0x7f57f415b790 2023-10-16T12:11:29.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.392+0000 7f57fa3a0700 1 -- 172.21.15.16:0/2589366115 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57f41441d0 con 0x7f57f415b790 2023-10-16T12:11:29.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.393+0000 7f57fa3a0700 1 -- 172.21.15.16:0/2589366115 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57f404ebb0 con 0x7f57f415b790 2023-10-16T12:11:29.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.394+0000 7f57f0ff9700 1 -- 172.21.15.16:0/2589366115 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f57ec0043e0 con 0x7f57f415b790 2023-10-16T12:11:29.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.394+0000 7f57f0ff9700 1 --2- 172.21.15.16:0/2589366115 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f57e006af00 0x7f57e006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:29.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.394+0000 7f57f0ff9700 1 -- 172.21.15.16:0/2589366115 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f57ec022070 con 0x7f57f415b790 2023-10-16T12:11:29.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.394+0000 7f57f3fff700 1 --2- 172.21.15.16:0/2589366115 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f57e006af00 0x7f57e006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:29.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.395+0000 7f57f3fff700 1 --2- 172.21.15.16:0/2589366115 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f57e006af00 0x7f57e006d3c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f57e4000f60 tx=0x7f57e4007f10 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:29.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.399+0000 7f57f0ff9700 1 -- 172.21.15.16:0/2589366115 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f57ec0564d0 con 0x7f57f415b790 2023-10-16T12:11:29.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.601+0000 7f57fa3a0700 1 -- 172.21.15.16:0/2589366115 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f57f4156df0 con 0x7f57f415b790 2023-10-16T12:11:29.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.601+0000 7f57f0ff9700 1 -- 172.21.15.16:0/2589366115 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7f57ec059af0 con 0x7f57f415b790 2023-10-16T12:11:29.603 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-10-16T12:11:29.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.603+0000 7f57fa3a0700 1 -- 172.21.15.16:0/2589366115 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f57e006af00 msgr2=0x7f57e006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:29.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.603+0000 7f57fa3a0700 1 --2- 172.21.15.16:0/2589366115 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f57e006af00 0x7f57e006d3c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f57e4000f60 tx=0x7f57e4007f10 comp rx=0 tx=0).stop 2023-10-16T12:11:29.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.603+0000 7f57fa3a0700 1 -- 172.21.15.16:0/2589366115 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57f415b790 msgr2=0x7f57f415f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:29.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.603+0000 7f57fa3a0700 1 --2- 172.21.15.16:0/2589366115 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57f415b790 0x7f57f415f670 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f57ec014040 tx=0x7f57ec004ad0 comp rx=0 tx=0).stop 2023-10-16T12:11:29.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.603+0000 7f57fa3a0700 1 -- 172.21.15.16:0/2589366115 shutdown_connections 2023-10-16T12:11:29.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.603+0000 7f57fa3a0700 1 --2- 172.21.15.16:0/2589366115 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f57e006af00 0x7f57e006d3c0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:29.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.603+0000 7f57fa3a0700 1 --2- 172.21.15.16:0/2589366115 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57f415b790 0x7f57f415f670 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:29.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.603+0000 7f57fa3a0700 1 --2- 172.21.15.16:0/2589366115 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f57f406dea0 0x7f57f415d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:29.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.603+0000 7f57fa3a0700 1 --2- 172.21.15.16:0/2589366115 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f57f40cda60 0x7f57f415f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:29.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.603+0000 7f57fa3a0700 1 -- 172.21.15.16:0/2589366115 >> 172.21.15.16:0/2589366115 conn(0x7f57f40cc5e0 msgr2=0x7f57f40cefe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:29.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.603+0000 7f57fa3a0700 1 -- 172.21.15.16:0/2589366115 shutdown_connections 2023-10-16T12:11:29.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.603+0000 7f57fa3a0700 1 -- 172.21.15.16:0/2589366115 wait complete. 2023-10-16T12:11:29.617 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-16T12:11:29.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.743+0000 7f08c853c700 1 Processor -- start 2023-10-16T12:11:29.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.743+0000 7f08c853c700 1 -- start start 2023-10-16T12:11:29.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.744+0000 7f08c853c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f08c015b2c0 0x7f08c0161340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:29.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.744+0000 7f08c853c700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f08c006e570 0x7f08c015e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:29.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.744+0000 7f08c853c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f08c006a7e0 0x7f08c006ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:29.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.744+0000 7f08c853c700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08c006b1b0 con 0x7f08c015b2c0 2023-10-16T12:11:29.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.744+0000 7f08c853c700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08c0161880 con 0x7f08c006e570 2023-10-16T12:11:29.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.744+0000 7f08c853c700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08c015b6a0 con 0x7f08c006a7e0 2023-10-16T12:11:29.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.745+0000 7f08c62d8700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f08c015b2c0 0x7f08c0161340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:29.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.745+0000 7f08c62d8700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f08c015b2c0 0x7f08c0161340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34456/0 (socket says 172.21.15.16:34456) 2023-10-16T12:11:29.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.745+0000 7f08c62d8700 1 -- 172.21.15.16:0/2181192149 learned_addr learned my addr 172.21.15.16:0/2181192149 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:29.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.745+0000 7f08c5ad7700 1 --2- 172.21.15.16:0/2181192149 >> v2:172.21.15.49:3300/0 conn(0x7f08c006e570 0x7f08c015e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:29.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.745+0000 7f08c6ad9700 1 --2- 172.21.15.16:0/2181192149 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f08c006a7e0 0x7f08c006ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:29.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.745+0000 7f08c62d8700 1 -- 172.21.15.16:0/2181192149 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f08c006a7e0 msgr2=0x7f08c006ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:29.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.745+0000 7f08c62d8700 1 --2- 172.21.15.16:0/2181192149 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f08c006a7e0 0x7f08c006ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:29.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.745+0000 7f08c62d8700 1 -- 172.21.15.16:0/2181192149 >> v2:172.21.15.49:3300/0 conn(0x7f08c006e570 msgr2=0x7f08c015e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:29.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.745+0000 7f08c62d8700 1 --2- 172.21.15.16:0/2181192149 >> v2:172.21.15.49:3300/0 conn(0x7f08c006e570 0x7f08c015e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:29.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.745+0000 7f08c62d8700 1 -- 172.21.15.16:0/2181192149 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08c0161b20 con 0x7f08c015b2c0 2023-10-16T12:11:29.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.745+0000 7f08c62d8700 1 --2- 172.21.15.16:0/2181192149 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08c015b2c0 0x7f08c0161340 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f08c0070500 tx=0x7f08b0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fafdb42a60b19b6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:29.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.745+0000 7f08c52d6700 1 -- 172.21.15.16:0/2181192149 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08b000b870 con 0x7f08c015b2c0 2023-10-16T12:11:29.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.745+0000 7f08c52d6700 1 -- 172.21.15.16:0/2181192149 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08b000b9d0 con 0x7f08c015b2c0 2023-10-16T12:11:29.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.745+0000 7f08c52d6700 1 -- 172.21.15.16:0/2181192149 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08b000bd60 con 0x7f08c015b2c0 2023-10-16T12:11:29.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.746+0000 7f08c853c700 1 -- 172.21.15.16:0/2181192149 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08c015b2c0 msgr2=0x7f08c0161340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:29.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.746+0000 7f08c853c700 1 --2- 172.21.15.16:0/2181192149 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08c015b2c0 0x7f08c0161340 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f08c0070500 tx=0x7f08b0009bd0 comp rx=0 tx=0).stop 2023-10-16T12:11:29.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.746+0000 7f08c853c700 1 -- 172.21.15.16:0/2181192149 shutdown_connections 2023-10-16T12:11:29.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.746+0000 7f08c853c700 1 --2- 172.21.15.16:0/2181192149 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f08c006a7e0 0x7f08c006ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:29.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.746+0000 7f08c853c700 1 --2- 172.21.15.16:0/2181192149 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08c015b2c0 0x7f08c0161340 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:29.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.746+0000 7f08c853c700 1 --2- 172.21.15.16:0/2181192149 >> v2:172.21.15.49:3300/0 conn(0x7f08c006e570 0x7f08c015e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:29.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.746+0000 7f08c853c700 1 -- 172.21.15.16:0/2181192149 >> 172.21.15.16:0/2181192149 conn(0x7f08c00b1f40 msgr2=0x7f08c00b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:29.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.746+0000 7f08c853c700 1 -- 172.21.15.16:0/2181192149 shutdown_connections 2023-10-16T12:11:29.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.746+0000 7f08c853c700 1 -- 172.21.15.16:0/2181192149 wait complete. 2023-10-16T12:11:29.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.747+0000 7f08c853c700 1 Processor -- start 2023-10-16T12:11:29.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.747+0000 7f08c853c700 1 -- start start 2023-10-16T12:11:29.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.748+0000 7f08c853c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f08c006a7e0 0x7f08c00b5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:29.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.748+0000 7f08c853c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08c006e570 0x7f08c01f3380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:29.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.748+0000 7f08c853c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f08c015b2c0 0x7f08c00b4620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:29.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.748+0000 7f08c853c700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08c01f4390 con 0x7f08c006e570 2023-10-16T12:11:29.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.748+0000 7f08c853c700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08c00b5ec0 con 0x7f08c006a7e0 2023-10-16T12:11:29.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.748+0000 7f08c853c700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08c01f38c0 con 0x7f08c015b2c0 2023-10-16T12:11:29.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.748+0000 7f08c5ad7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08c006e570 0x7f08c01f3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:29.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.748+0000 7f08c5ad7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08c006e570 0x7f08c01f3380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34464/0 (socket says 172.21.15.16:34464) 2023-10-16T12:11:29.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.748+0000 7f08c5ad7700 1 -- 172.21.15.16:0/2058081126 learned_addr learned my addr 172.21.15.16:0/2058081126 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:29.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.748+0000 7f08c62d8700 1 --2- 172.21.15.16:0/2058081126 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f08c006a7e0 0x7f08c00b5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:29.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.748+0000 7f08c6ad9700 1 --2- 172.21.15.16:0/2058081126 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f08c015b2c0 0x7f08c00b4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:29.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.748+0000 7f08c5ad7700 1 -- 172.21.15.16:0/2058081126 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f08c015b2c0 msgr2=0x7f08c00b4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:29.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.748+0000 7f08c5ad7700 1 --2- 172.21.15.16:0/2058081126 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f08c015b2c0 0x7f08c00b4620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:29.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.748+0000 7f08c5ad7700 1 -- 172.21.15.16:0/2058081126 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f08c006a7e0 msgr2=0x7f08c00b5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:29.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.748+0000 7f08c5ad7700 1 --2- 172.21.15.16:0/2058081126 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f08c006a7e0 0x7f08c00b5980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:29.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.748+0000 7f08c5ad7700 1 -- 172.21.15.16:0/2058081126 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08b0009550 con 0x7f08c006e570 2023-10-16T12:11:29.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.748+0000 7f08c5ad7700 1 --2- 172.21.15.16:0/2058081126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08c006e570 0x7f08c01f3380 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f08bc00d800 tx=0x7f08bc00db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:29.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.748+0000 7f08b6ffd700 1 -- 172.21.15.16:0/2058081126 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08bc0125f0 con 0x7f08c006e570 2023-10-16T12:11:29.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.749+0000 7f08b6ffd700 1 -- 172.21.15.16:0/2058081126 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08bc012750 con 0x7f08c006e570 2023-10-16T12:11:29.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.749+0000 7f08b6ffd700 1 -- 172.21.15.16:0/2058081126 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08bc012b00 con 0x7f08c006e570 2023-10-16T12:11:29.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.749+0000 7f08c853c700 1 -- 172.21.15.16:0/2058081126 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08c00b4190 con 0x7f08c006e570 2023-10-16T12:11:29.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.749+0000 7f08c853c700 1 -- 172.21.15.16:0/2058081126 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08c01f3dd0 con 0x7f08c006e570 2023-10-16T12:11:29.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.749+0000 7f08c853c700 1 -- 172.21.15.16:0/2058081126 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08c00b3f60 con 0x7f08c006e570 2023-10-16T12:11:29.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.750+0000 7f08b6ffd700 1 -- 172.21.15.16:0/2058081126 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f08bc012c60 con 0x7f08c006e570 2023-10-16T12:11:29.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.750+0000 7f08b6ffd700 1 --2- 172.21.15.16:0/2058081126 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f08a406af00 0x7f08a406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:29.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.750+0000 7f08b6ffd700 1 -- 172.21.15.16:0/2058081126 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f08bc089880 con 0x7f08c006e570 2023-10-16T12:11:29.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.750+0000 7f08c62d8700 1 --2- 172.21.15.16:0/2058081126 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f08a406af00 0x7f08a406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:29.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.751+0000 7f08c62d8700 1 --2- 172.21.15.16:0/2058081126 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f08a406af00 0x7f08a406d3c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f08b0000c00 tx=0x7f08b0008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:29.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.755+0000 7f08b6ffd700 1 -- 172.21.15.16:0/2058081126 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f08c00b3f60 con 0x7f08c006e570 2023-10-16T12:11:29.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.959+0000 7f08c853c700 1 -- 172.21.15.16:0/2058081126 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f08c00b3f60 con 0x7f08c006e570 2023-10-16T12:11:29.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.960+0000 7f08b6ffd700 1 -- 172.21.15.16:0/2058081126 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f08c00b3f60 con 0x7f08c006e570 2023-10-16T12:11:29.962 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-10-16T12:11:29.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.961+0000 7f08c853c700 1 -- 172.21.15.16:0/2058081126 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f08a406af00 msgr2=0x7f08a406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:29.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.961+0000 7f08c853c700 1 --2- 172.21.15.16:0/2058081126 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f08a406af00 0x7f08a406d3c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f08b0000c00 tx=0x7f08b0008030 comp rx=0 tx=0).stop 2023-10-16T12:11:29.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.962+0000 7f08c853c700 1 -- 172.21.15.16:0/2058081126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08c006e570 msgr2=0x7f08c01f3380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:29.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.962+0000 7f08c853c700 1 --2- 172.21.15.16:0/2058081126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08c006e570 0x7f08c01f3380 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f08bc00d800 tx=0x7f08bc00db10 comp rx=0 tx=0).stop 2023-10-16T12:11:29.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.962+0000 7f08c853c700 1 -- 172.21.15.16:0/2058081126 shutdown_connections 2023-10-16T12:11:29.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.962+0000 7f08c853c700 1 --2- 172.21.15.16:0/2058081126 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f08a406af00 0x7f08a406d3c0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:29.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.962+0000 7f08c853c700 1 --2- 172.21.15.16:0/2058081126 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f08c015b2c0 0x7f08c00b4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:29.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.962+0000 7f08c853c700 1 --2- 172.21.15.16:0/2058081126 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f08c006a7e0 0x7f08c00b5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:29.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.962+0000 7f08c853c700 1 --2- 172.21.15.16:0/2058081126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08c006e570 0x7f08c01f3380 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:29.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.962+0000 7f08c853c700 1 -- 172.21.15.16:0/2058081126 >> 172.21.15.16:0/2058081126 conn(0x7f08c00b1f40 msgr2=0x7f08c015d660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:29.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.962+0000 7f08c853c700 1 -- 172.21.15.16:0/2058081126 shutdown_connections 2023-10-16T12:11:29.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:29.962+0000 7f08c853c700 1 -- 172.21.15.16:0/2058081126 wait complete. 2023-10-16T12:11:29.976 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:11:30.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.099+0000 7f13afae5700 1 Processor -- start 2023-10-16T12:11:30.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.099+0000 7f13afae5700 1 -- start start 2023-10-16T12:11:30.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.100+0000 7f13afae5700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f13a815db30 0x7f13a80cf920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:30.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.100+0000 7f13afae5700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13a80d0070 con 0x7f13a8161490 2023-10-16T12:11:30.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.100+0000 7f13afae5700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13a80d0350 con 0x7f13a815d750 2023-10-16T12:11:30.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.100+0000 7f13afae5700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13a80cfe60 con 0x7f13a815db30 2023-10-16T12:11:30.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.100+0000 7f13ae082700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f13a8161490 0x7f13a80b2060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51980/0 (socket says 172.21.15.16:51980) 2023-10-16T12:11:30.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.100+0000 7f13ae082700 1 -- 172.21.15.16:0/2947892080 learned_addr learned my addr 172.21.15.16:0/2947892080 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:30.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.101+0000 7f13ac87f700 1 -- 172.21.15.16:0/2947892080 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3395915425 0 0) 0x7f13a80d0070 con 0x7f13a8161490 2023-10-16T12:11:30.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.101+0000 7f13ac87f700 1 -- 172.21.15.16:0/2947892080 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13940035b0 con 0x7f13a8161490 2023-10-16T12:11:30.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.101+0000 7f13ad080700 1 --2- 172.21.15.16:0/2947892080 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f13a815db30 0x7f13a80cf920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:30.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.101+0000 7f13ac87f700 1 -- 172.21.15.16:0/2947892080 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 579375526 0 0) 0x7f13940035b0 con 0x7f13a8161490 2023-10-16T12:11:30.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.101+0000 7f13ac87f700 1 -- 172.21.15.16:0/2947892080 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f13a815db30 msgr2=0x7f13a80cf920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:30.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.101+0000 7f13ac87f700 1 --2- 172.21.15.16:0/2947892080 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f13a815db30 0x7f13a80cf920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:30.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.101+0000 7f13ac87f700 1 -- 172.21.15.16:0/2947892080 >> v1:172.21.15.49:6789/0 conn(0x7f13a815d750 legacy=0x7f13a806c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:30.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.101+0000 7f13ac87f700 1 -- 172.21.15.16:0/2947892080 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13a80d0630 con 0x7f13a8161490 2023-10-16T12:11:30.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.101+0000 7f13ac87f700 1 -- 172.21.15.16:0/2947892080 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f13a00019f0 con 0x7f13a8161490 2023-10-16T12:11:30.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.101+0000 7f13ac87f700 1 -- 172.21.15.16:0/2947892080 >> v1:172.21.15.16:6789/0 conn(0x7f13a8161490 legacy=0x7f13a80b2060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:30.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.101+0000 7f13ac87f700 1 --2- 172.21.15.16:0/2947892080 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1394003a30 0x7f1394003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:30.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.101+0000 7f13ac87f700 1 --2- 172.21.15.16:0/2947892080 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1394004850 0x7f1394004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:30.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.101+0000 7f13ac87f700 1 --2- 172.21.15.16:0/2947892080 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f13a815db30 0x7f1394005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:30.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.101+0000 7f13ac87f700 1 -- 172.21.15.16:0/2947892080 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13a80cfe60 con 0x7f1394004850 2023-10-16T12:11:30.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.101+0000 7f13ac87f700 1 -- 172.21.15.16:0/2947892080 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f139400bac0 con 0x7f1394003a30 2023-10-16T12:11:30.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.101+0000 7f13ac87f700 1 -- 172.21.15.16:0/2947892080 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f139400bc50 con 0x7f13a815db30 2023-10-16T12:11:30.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.101+0000 7f13ad080700 1 --2- 172.21.15.16:0/2947892080 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1394004850 0x7f1394004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:30.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.101+0000 7f13ad881700 1 --2- 172.21.15.16:0/2947892080 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1394003a30 0x7f1394003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:30.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.101+0000 7f13ae082700 1 --2- 172.21.15.16:0/2947892080 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f13a815db30 0x7f1394005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:30.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.101+0000 7f13ad080700 1 -- 172.21.15.16:0/2947892080 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f13a815db30 msgr2=0x7f1394005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:30.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.101+0000 7f13ad080700 1 --2- 172.21.15.16:0/2947892080 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f13a815db30 0x7f1394005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:30.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.101+0000 7f13ad080700 1 -- 172.21.15.16:0/2947892080 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1394003a30 msgr2=0x7f1394003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:30.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.101+0000 7f13ad080700 1 --2- 172.21.15.16:0/2947892080 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1394003a30 0x7f1394003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:30.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.101+0000 7f13ad080700 1 -- 172.21.15.16:0/2947892080 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13940035b0 con 0x7f1394004850 2023-10-16T12:11:30.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.102+0000 7f13ad080700 1 --2- 172.21.15.16:0/2947892080 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1394004850 0x7f1394004cd0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f13a815df60 tx=0x7f1394005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57fc45df06aeb04f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:30.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.102+0000 7f13ac87f700 1 -- 172.21.15.16:0/2947892080 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a400e6c0 con 0x7f1394004850 2023-10-16T12:11:30.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.102+0000 7f13ac87f700 1 -- 172.21.15.16:0/2947892080 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13a400e820 con 0x7f1394004850 2023-10-16T12:11:30.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.102+0000 7f13ac87f700 1 -- 172.21.15.16:0/2947892080 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a400ebb0 con 0x7f1394004850 2023-10-16T12:11:30.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.102+0000 7f13afae5700 1 -- 172.21.15.16:0/2947892080 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1394004850 msgr2=0x7f1394004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:30.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.102+0000 7f13afae5700 1 --2- 172.21.15.16:0/2947892080 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1394004850 0x7f1394004cd0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f13a815df60 tx=0x7f1394005880 comp rx=0 tx=0).stop 2023-10-16T12:11:30.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.102+0000 7f13ae082700 1 -- 172.21.15.16:0/2947892080 reap_dead start 2023-10-16T12:11:30.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.102+0000 7f13afae5700 1 -- 172.21.15.16:0/2947892080 shutdown_connections 2023-10-16T12:11:30.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.102+0000 7f13afae5700 1 -- 172.21.15.16:0/2947892080 >> 172.21.15.16:0/2947892080 conn(0x7f13a80d14b0 msgr2=0x7f13a80d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:30.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.102+0000 7f13afae5700 1 -- 172.21.15.16:0/2947892080 shutdown_connections 2023-10-16T12:11:30.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.102+0000 7f13afae5700 1 -- 172.21.15.16:0/2947892080 wait complete. 2023-10-16T12:11:30.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.104+0000 7f13afae5700 1 Processor -- start 2023-10-16T12:11:30.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.104+0000 7f13afae5700 1 -- start start 2023-10-16T12:11:30.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.104+0000 7f13afae5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1394004850 0x7f13a80cb970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:30.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.104+0000 7f13afae5700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f13a815ba40 0x7f13a80b63f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:30.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.104+0000 7f13afae5700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f13a815eb10 0x7f13a815efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:30.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.104+0000 7f13afae5700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13a815f4e0 con 0x7f1394004850 2023-10-16T12:11:30.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.104+0000 7f13afae5700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13a80cc510 con 0x7f13a815eb10 2023-10-16T12:11:30.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.104+0000 7f13afae5700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13a80cbf50 con 0x7f13a815ba40 2023-10-16T12:11:30.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.104+0000 7f13ad881700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1394004850 0x7f13a80cb970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:30.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.105+0000 7f13ad881700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1394004850 0x7f13a80cb970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34482/0 (socket says 172.21.15.16:34482) 2023-10-16T12:11:30.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.105+0000 7f13ad881700 1 -- 172.21.15.16:0/1369602152 learned_addr learned my addr 172.21.15.16:0/1369602152 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:30.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.105+0000 7f13ae082700 1 --2- 172.21.15.16:0/1369602152 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f13a815eb10 0x7f13a815efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:30.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.105+0000 7f13ad080700 1 --2- 172.21.15.16:0/1369602152 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f13a815ba40 0x7f13a80b63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:30.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.105+0000 7f13ad881700 1 -- 172.21.15.16:0/1369602152 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f13a815ba40 msgr2=0x7f13a80b63f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:30.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.105+0000 7f13ad881700 1 --2- 172.21.15.16:0/1369602152 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f13a815ba40 0x7f13a80b63f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:30.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.105+0000 7f13ad881700 1 -- 172.21.15.16:0/1369602152 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f13a815eb10 msgr2=0x7f13a815efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:30.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.105+0000 7f13ad881700 1 --2- 172.21.15.16:0/1369602152 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f13a815eb10 0x7f13a815efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:30.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.105+0000 7f13ad881700 1 -- 172.21.15.16:0/1369602152 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13a80cc230 con 0x7f1394004850 2023-10-16T12:11:30.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.105+0000 7f13ad881700 1 --2- 172.21.15.16:0/1369602152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1394004850 0x7f13a80cb970 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f139800f820 tx=0x7f139800fb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:30.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.105+0000 7f139e7fc700 1 -- 172.21.15.16:0/1369602152 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13980052e0 con 0x7f1394004850 2023-10-16T12:11:30.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.105+0000 7f139e7fc700 1 -- 172.21.15.16:0/1369602152 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1398005460 con 0x7f1394004850 2023-10-16T12:11:30.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.105+0000 7f13afae5700 1 -- 172.21.15.16:0/1369602152 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13a80c8690 con 0x7f1394004850 2023-10-16T12:11:30.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.105+0000 7f139e7fc700 1 -- 172.21.15.16:0/1369602152 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1398003910 con 0x7f1394004850 2023-10-16T12:11:30.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.105+0000 7f13afae5700 1 -- 172.21.15.16:0/1369602152 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13a815bec0 con 0x7f1394004850 2023-10-16T12:11:30.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.106+0000 7f13afae5700 1 -- 172.21.15.16:0/1369602152 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13a80c8bb0 con 0x7f1394004850 2023-10-16T12:11:30.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.107+0000 7f139e7fc700 1 -- 172.21.15.16:0/1369602152 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f1398003a70 con 0x7f1394004850 2023-10-16T12:11:30.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.107+0000 7f139e7fc700 1 --2- 172.21.15.16:0/1369602152 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f138c06af00 0x7f138c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:30.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.107+0000 7f139e7fc700 1 -- 172.21.15.16:0/1369602152 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f1398089790 con 0x7f1394004850 2023-10-16T12:11:30.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.107+0000 7f13ad080700 1 --2- 172.21.15.16:0/1369602152 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f138c06af00 0x7f138c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:30.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.108+0000 7f13ad080700 1 --2- 172.21.15.16:0/1369602152 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f138c06af00 0x7f138c06d3c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f13a40090b0 tx=0x7f13a401f040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:30.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.112+0000 7f139e7fc700 1 -- 172.21.15.16:0/1369602152 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f13a80c8bb0 con 0x7f1394004850 2023-10-16T12:11:30.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.314+0000 7f13afae5700 1 -- 172.21.15.16:0/1369602152 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f13a80c8bb0 con 0x7f1394004850 2023-10-16T12:11:30.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.315+0000 7f139e7fc700 1 -- 172.21.15.16:0/1369602152 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18629 (secure 0 0 0) 0x7f13a80c8bb0 con 0x7f1394004850 2023-10-16T12:11:30.318 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:11:30.318 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:11:28.241865+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:10:36.935773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T12:10:38.004870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-16T12:10:41.672025+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"3.2b","mappings":[{"from":0,"to":2}]},{"pgid":"3.38","mappings":[{"from":3,"to":1}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]},{"pgid":"4.1b","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:11:30.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.318+0000 7f13afae5700 1 -- 172.21.15.16:0/1369602152 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f138c06af00 msgr2=0x7f138c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:30.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.318+0000 7f13afae5700 1 --2- 172.21.15.16:0/1369602152 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f138c06af00 0x7f138c06d3c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f13a40090b0 tx=0x7f13a401f040 comp rx=0 tx=0).stop 2023-10-16T12:11:30.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.318+0000 7f13afae5700 1 -- 172.21.15.16:0/1369602152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1394004850 msgr2=0x7f13a80cb970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:30.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.318+0000 7f13afae5700 1 --2- 172.21.15.16:0/1369602152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1394004850 0x7f13a80cb970 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f139800f820 tx=0x7f139800fb30 comp rx=0 tx=0).stop 2023-10-16T12:11:30.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.318+0000 7f13afae5700 1 -- 172.21.15.16:0/1369602152 shutdown_connections 2023-10-16T12:11:30.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.318+0000 7f13afae5700 1 --2- 172.21.15.16:0/1369602152 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f138c06af00 0x7f138c06d3c0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:30.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.318+0000 7f13afae5700 1 --2- 172.21.15.16:0/1369602152 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f13a815eb10 0x7f13a815efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:30.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.318+0000 7f13afae5700 1 --2- 172.21.15.16:0/1369602152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1394004850 0x7f13a80cb970 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:30.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.318+0000 7f13afae5700 1 --2- 172.21.15.16:0/1369602152 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f13a815ba40 0x7f13a80b63f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:30.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.318+0000 7f13afae5700 1 -- 172.21.15.16:0/1369602152 >> 172.21.15.16:0/1369602152 conn(0x7f13a80d14b0 msgr2=0x7f13a81618e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:30.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.318+0000 7f13afae5700 1 -- 172.21.15.16:0/1369602152 shutdown_connections 2023-10-16T12:11:30.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.318+0000 7f13afae5700 1 -- 172.21.15.16:0/1369602152 wait complete. 2023-10-16T12:11:30.332 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T12:10:36.935773+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T12:10:38.004870+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-16T12:10:41.672025+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-16T12:11:30.333 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:11:30.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.458+0000 7fb90b80f700 1 Processor -- start 2023-10-16T12:11:30.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.459+0000 7fb90b80f700 1 -- start start 2023-10-16T12:11:30.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.459+0000 7fb90b80f700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fb9040cbe20 0x7fb9040cda60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:30.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.459+0000 7fb90b80f700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb90415aa20 con 0x7fb9040c6770 2023-10-16T12:11:30.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.459+0000 7fb90b80f700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb90406eb40 con 0x7fb90415a640 2023-10-16T12:11:30.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.459+0000 7fb90b80f700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb90406ee20 con 0x7fb9040cbe20 2023-10-16T12:11:30.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.459+0000 7fb909dac700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb9040c6770 0x7fb9040cdfa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:52000/0 (socket says 172.21.15.16:52000) 2023-10-16T12:11:30.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.459+0000 7fb909dac700 1 -- 172.21.15.16:0/1675799370 learned_addr learned my addr 172.21.15.16:0/1675799370 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:30.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.460+0000 7fb908daa700 1 --2- 172.21.15.16:0/1675799370 >> v2:172.21.15.49:3300/0 conn(0x7fb9040cbe20 0x7fb9040cda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:30.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.460+0000 7fb8fbfff700 1 -- 172.21.15.16:0/1675799370 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2421757473 0 0) 0x7fb8fc001490 con 0x7fb9040c6770 2023-10-16T12:11:30.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.460+0000 7fb8fbfff700 1 -- 172.21.15.16:0/1675799370 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8f00035b0 con 0x7fb9040c6770 2023-10-16T12:11:30.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.460+0000 7fb8fbfff700 1 -- 172.21.15.16:0/1675799370 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 924254191 0 0) 0x7fb90406eb40 con 0x7fb90415a640 2023-10-16T12:11:30.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.460+0000 7fb8fbfff700 1 -- 172.21.15.16:0/1675799370 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8fc001490 con 0x7fb90415a640 2023-10-16T12:11:30.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.460+0000 7fb8fbfff700 1 -- 172.21.15.16:0/1675799370 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1329762334 0 0) 0x7fb8f00035b0 con 0x7fb9040c6770 2023-10-16T12:11:30.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.460+0000 7fb8fbfff700 1 -- 172.21.15.16:0/1675799370 >> v2:172.21.15.49:3300/0 conn(0x7fb9040cbe20 msgr2=0x7fb9040cda60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:30.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.460+0000 7fb8fbfff700 1 --2- 172.21.15.16:0/1675799370 >> v2:172.21.15.49:3300/0 conn(0x7fb9040cbe20 0x7fb9040cda60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:30.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.460+0000 7fb8fbfff700 1 -- 172.21.15.16:0/1675799370 >> v1:172.21.15.49:6789/0 conn(0x7fb90415a640 legacy=0x7fb9040cb710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:30.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.460+0000 7fb8fbfff700 1 -- 172.21.15.16:0/1675799370 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb90406f100 con 0x7fb9040c6770 2023-10-16T12:11:30.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.460+0000 7fb8fbfff700 1 -- 172.21.15.16:0/1675799370 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7fb8fc001b80 con 0x7fb9040c6770 2023-10-16T12:11:30.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.460+0000 7fb8fbfff700 1 -- 172.21.15.16:0/1675799370 >> v1:172.21.15.16:6789/0 conn(0x7fb9040c6770 legacy=0x7fb9040cdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:30.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.460+0000 7fb8fbfff700 1 --2- 172.21.15.16:0/1675799370 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb8f0003b90 0x7fb8f0003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:30.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.460+0000 7fb8fbfff700 1 --2- 172.21.15.16:0/1675799370 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb8f0004850 0x7fb8f0004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:30.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.460+0000 7fb8fbfff700 1 --2- 172.21.15.16:0/1675799370 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8f0005210 0x7fb8f000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:30.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.460+0000 7fb8fbfff700 1 -- 172.21.15.16:0/1675799370 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb90406ee20 con 0x7fb8f0005210 2023-10-16T12:11:30.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.460+0000 7fb8fbfff700 1 -- 172.21.15.16:0/1675799370 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f000c090 con 0x7fb8f0004850 2023-10-16T12:11:30.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.460+0000 7fb8fbfff700 1 -- 172.21.15.16:0/1675799370 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f000c250 con 0x7fb8f0003b90 2023-10-16T12:11:30.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.460+0000 7fb908daa700 1 --2- 172.21.15.16:0/1675799370 >> v2:172.21.15.49:3300/0 conn(0x7fb9040cbe20 0x7fb9040cda60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:11:30.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.460+0000 7fb909dac700 1 --2- 172.21.15.16:0/1675799370 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8f0005210 0x7fb8f000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:30.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.460+0000 7fb9095ab700 1 --2- 172.21.15.16:0/1675799370 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb8f0003b90 0x7fb8f0003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:30.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.460+0000 7fb908daa700 1 --2- 172.21.15.16:0/1675799370 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb8f0004850 0x7fb8f0004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:30.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.460+0000 7fb909dac700 1 -- 172.21.15.16:0/1675799370 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb8f0003b90 msgr2=0x7fb8f0003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:30.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.460+0000 7fb909dac700 1 --2- 172.21.15.16:0/1675799370 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb8f0003b90 0x7fb8f0003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:30.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.460+0000 7fb909dac700 1 -- 172.21.15.16:0/1675799370 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb8f0004850 msgr2=0x7fb8f0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:30.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.460+0000 7fb909dac700 1 --2- 172.21.15.16:0/1675799370 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb8f0004850 0x7fb8f0004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:30.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.461+0000 7fb909dac700 1 -- 172.21.15.16:0/1675799370 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8f00035b0 con 0x7fb8f0005210 2023-10-16T12:11:30.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.461+0000 7fb909dac700 1 -- 172.21.15.16:0/1675799370 reap_dead start 2023-10-16T12:11:30.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.461+0000 7fb909dac700 1 --2- 172.21.15.16:0/1675799370 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8f0005210 0x7fb8f000ba10 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fb8fc00a7b0 tx=0x7fb8f0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a38c0b90f068fe37 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:30.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.461+0000 7fb8fbfff700 1 -- 172.21.15.16:0/1675799370 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8fc012600 con 0x7fb8f0005210 2023-10-16T12:11:30.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.461+0000 7fb8fbfff700 1 -- 172.21.15.16:0/1675799370 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8fc012780 con 0x7fb8f0005210 2023-10-16T12:11:30.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.461+0000 7fb8fbfff700 1 -- 172.21.15.16:0/1675799370 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8fc012b30 con 0x7fb8f0005210 2023-10-16T12:11:30.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.461+0000 7fb90b80f700 1 -- 172.21.15.16:0/1675799370 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8f0005210 msgr2=0x7fb8f000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:30.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.461+0000 7fb90b80f700 1 --2- 172.21.15.16:0/1675799370 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8f0005210 0x7fb8f000ba10 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fb8fc00a7b0 tx=0x7fb8f0005880 comp rx=0 tx=0).stop 2023-10-16T12:11:30.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.461+0000 7fb90b80f700 1 -- 172.21.15.16:0/1675799370 shutdown_connections 2023-10-16T12:11:30.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.461+0000 7fb90b80f700 1 --2- 172.21.15.16:0/1675799370 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8f0005210 0x7fb8f000ba10 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:30.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.461+0000 7fb90b80f700 1 -- 172.21.15.16:0/1675799370 >> 172.21.15.16:0/1675799370 conn(0x7fb9040cc5e0 msgr2=0x7fb9040cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:30.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.461+0000 7fb90b80f700 1 -- 172.21.15.16:0/1675799370 shutdown_connections 2023-10-16T12:11:30.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.461+0000 7fb90b80f700 1 -- 172.21.15.16:0/1675799370 wait complete. 2023-10-16T12:11:30.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.463+0000 7fb90b80f700 1 Processor -- start 2023-10-16T12:11:30.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.463+0000 7fb90b80f700 1 -- start start 2023-10-16T12:11:30.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.463+0000 7fb90b80f700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb8f0005210 0x7fb9041e53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:30.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.463+0000 7fb90b80f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb9041f3fd0 0x7fb9040afe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:30.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.463+0000 7fb90b80f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb90414dc40 0x7fb90414e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:30.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.463+0000 7fb90b80f700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb904157b50 con 0x7fb90414dc40 2023-10-16T12:11:30.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.463+0000 7fb90b80f700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9041e5920 con 0x7fb8f0005210 2023-10-16T12:11:30.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.463+0000 7fb90b80f700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb90414e630 con 0x7fb9041f3fd0 2023-10-16T12:11:30.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.464+0000 7fb909dac700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb90414dc40 0x7fb90414e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:30.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.464+0000 7fb909dac700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb90414dc40 0x7fb90414e0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34504/0 (socket says 172.21.15.16:34504) 2023-10-16T12:11:30.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.464+0000 7fb909dac700 1 -- 172.21.15.16:0/3465708858 learned_addr learned my addr 172.21.15.16:0/3465708858 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:30.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.464+0000 7fb9095ab700 1 --2- 172.21.15.16:0/3465708858 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb8f0005210 0x7fb9041e53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:30.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.464+0000 7fb908daa700 1 --2- 172.21.15.16:0/3465708858 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb9041f3fd0 0x7fb9040afe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:30.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.464+0000 7fb909dac700 1 -- 172.21.15.16:0/3465708858 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb9041f3fd0 msgr2=0x7fb9040afe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:30.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.464+0000 7fb909dac700 1 --2- 172.21.15.16:0/3465708858 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb9041f3fd0 0x7fb9040afe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:30.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.464+0000 7fb909dac700 1 -- 172.21.15.16:0/3465708858 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb8f0005210 msgr2=0x7fb9041e53e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:30.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.464+0000 7fb909dac700 1 --2- 172.21.15.16:0/3465708858 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb8f0005210 0x7fb9041e53e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:30.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.464+0000 7fb909dac700 1 -- 172.21.15.16:0/3465708858 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9041f4460 con 0x7fb90414dc40 2023-10-16T12:11:30.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.464+0000 7fb908daa700 1 --2- 172.21.15.16:0/3465708858 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb9041f3fd0 0x7fb9040afe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:11:30.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.464+0000 7fb909dac700 1 --2- 172.21.15.16:0/3465708858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb90414dc40 0x7fb90414e0f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fb8fc000f80 tx=0x7fb8fc001e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:30.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.464+0000 7fb8f9ffb700 1 -- 172.21.15.16:0/3465708858 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8fc022070 con 0x7fb90414dc40 2023-10-16T12:11:30.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.464+0000 7fb8f9ffb700 1 -- 172.21.15.16:0/3465708858 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8fc002000 con 0x7fb90414dc40 2023-10-16T12:11:30.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.464+0000 7fb8f9ffb700 1 -- 172.21.15.16:0/3465708858 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8fc01c610 con 0x7fb90414dc40 2023-10-16T12:11:30.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.464+0000 7fb90b80f700 1 -- 172.21.15.16:0/3465708858 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9041f4740 con 0x7fb90414dc40 2023-10-16T12:11:30.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.464+0000 7fb90b80f700 1 -- 172.21.15.16:0/3465708858 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9041f4d00 con 0x7fb90414dc40 2023-10-16T12:11:30.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.465+0000 7fb90b80f700 1 -- 172.21.15.16:0/3465708858 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb904157c90 con 0x7fb90414dc40 2023-10-16T12:11:30.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.466+0000 7fb8f9ffb700 1 -- 172.21.15.16:0/3465708858 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fb8fc019070 con 0x7fb90414dc40 2023-10-16T12:11:30.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.466+0000 7fb8f9ffb700 1 --2- 172.21.15.16:0/3465708858 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fb8e806af00 0x7fb8e806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:30.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.466+0000 7fb8f9ffb700 1 -- 172.21.15.16:0/3465708858 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fb8fc08aa60 con 0x7fb90414dc40 2023-10-16T12:11:30.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.466+0000 7fb9095ab700 1 --2- 172.21.15.16:0/3465708858 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fb8e806af00 0x7fb8e806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:30.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.467+0000 7fb9095ab700 1 --2- 172.21.15.16:0/3465708858 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fb8e806af00 0x7fb8e806d3c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb8f4000c00 tx=0x7fb8f4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:30.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.471+0000 7fb8f9ffb700 1 -- 172.21.15.16:0/3465708858 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb8fc055740 con 0x7fb90414dc40 2023-10-16T12:11:30.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.671+0000 7fb90b80f700 1 -- 172.21.15.16:0/3465708858 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb904157c90 con 0x7fb90414dc40 2023-10-16T12:11:30.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.671+0000 7fb8f9ffb700 1 -- 172.21.15.16:0/3465708858 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb8fc017020 con 0x7fb90414dc40 2023-10-16T12:11:30.674 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T12:11:30.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.674+0000 7fb90b80f700 1 -- 172.21.15.16:0/3465708858 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fb8e806af00 msgr2=0x7fb8e806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:30.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.674+0000 7fb90b80f700 1 --2- 172.21.15.16:0/3465708858 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fb8e806af00 0x7fb8e806d3c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb8f4000c00 tx=0x7fb8f4009040 comp rx=0 tx=0).stop 2023-10-16T12:11:30.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.674+0000 7fb90b80f700 1 -- 172.21.15.16:0/3465708858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb90414dc40 msgr2=0x7fb90414e0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:30.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.674+0000 7fb90b80f700 1 --2- 172.21.15.16:0/3465708858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb90414dc40 0x7fb90414e0f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fb8fc000f80 tx=0x7fb8fc001e90 comp rx=0 tx=0).stop 2023-10-16T12:11:30.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.674+0000 7fb90b80f700 1 -- 172.21.15.16:0/3465708858 shutdown_connections 2023-10-16T12:11:30.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.674+0000 7fb90b80f700 1 --2- 172.21.15.16:0/3465708858 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fb8e806af00 0x7fb8e806d3c0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:30.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.674+0000 7fb90b80f700 1 --2- 172.21.15.16:0/3465708858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb90414dc40 0x7fb90414e0f0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:30.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.674+0000 7fb90b80f700 1 --2- 172.21.15.16:0/3465708858 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb8f0005210 0x7fb9041e53e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:30.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.674+0000 7fb90b80f700 1 --2- 172.21.15.16:0/3465708858 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb9041f3fd0 0x7fb9040afe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:30.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.674+0000 7fb90b80f700 1 -- 172.21.15.16:0/3465708858 >> 172.21.15.16:0/3465708858 conn(0x7fb9040cc5e0 msgr2=0x7fb9040cb430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:30.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.674+0000 7fb90b80f700 1 -- 172.21.15.16:0/3465708858 shutdown_connections 2023-10-16T12:11:30.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.674+0000 7fb90b80f700 1 -- 172.21.15.16:0/3465708858 wait complete. 2023-10-16T12:11:30.688 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T12:11:30.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.814+0000 7eff0aef0700 1 Processor -- start 2023-10-16T12:11:30.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.814+0000 7eff0aef0700 1 -- start start 2023-10-16T12:11:30.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.815+0000 7eff0aef0700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7eff04072b30 0x7eff04071a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:30.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.815+0000 7eff0aef0700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7eff0415bb40 0x7eff0406e310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:30.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.815+0000 7eff0aef0700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7eff040caa40 0x7eff040cae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:30.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.815+0000 7eff0aef0700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff040cb3d0 con 0x7eff040caa40 2023-10-16T12:11:30.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.815+0000 7eff0aef0700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff0415c040 con 0x7eff0415bb40 2023-10-16T12:11:30.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.815+0000 7eff0aef0700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff0415c320 con 0x7eff04072b30 2023-10-16T12:11:30.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.815+0000 7eff0948d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7eff040caa40 0x7eff040cae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:30.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.815+0000 7eff0948d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7eff040caa40 0x7eff040cae90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34508/0 (socket says 172.21.15.16:34508) 2023-10-16T12:11:30.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.815+0000 7eff0948d700 1 -- 172.21.15.16:0/2536962419 learned_addr learned my addr 172.21.15.16:0/2536962419 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:30.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.815+0000 7eff03fff700 1 --2- 172.21.15.16:0/2536962419 >> v2:172.21.15.49:3300/0 conn(0x7eff0415bb40 0x7eff0406e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:30.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.815+0000 7eff08c8c700 1 --2- 172.21.15.16:0/2536962419 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7eff04072b30 0x7eff04071a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:30.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.815+0000 7eff0948d700 1 -- 172.21.15.16:0/2536962419 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7eff04072b30 msgr2=0x7eff04071a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:30.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.815+0000 7eff0948d700 1 --2- 172.21.15.16:0/2536962419 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7eff04072b30 0x7eff04071a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:30.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.815+0000 7eff0948d700 1 -- 172.21.15.16:0/2536962419 >> v2:172.21.15.49:3300/0 conn(0x7eff0415bb40 msgr2=0x7eff0406e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:30.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.815+0000 7eff0948d700 1 --2- 172.21.15.16:0/2536962419 >> v2:172.21.15.49:3300/0 conn(0x7eff0415bb40 0x7eff0406e310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:30.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.815+0000 7eff0948d700 1 -- 172.21.15.16:0/2536962419 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff0415c600 con 0x7eff040caa40 2023-10-16T12:11:30.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.816+0000 7eff0948d700 1 --2- 172.21.15.16:0/2536962419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff040caa40 0x7eff040cae90 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7eff040d0200 tx=0x7efefc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79327b5d5dba4617 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:30.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.816+0000 7eff037fe700 1 -- 172.21.15.16:0/2536962419 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efefc00e070 con 0x7eff040caa40 2023-10-16T12:11:30.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.816+0000 7eff037fe700 1 -- 172.21.15.16:0/2536962419 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efefc00b9c0 con 0x7eff040caa40 2023-10-16T12:11:30.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.816+0000 7eff037fe700 1 -- 172.21.15.16:0/2536962419 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efefc00bd70 con 0x7eff040caa40 2023-10-16T12:11:30.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.816+0000 7eff0aef0700 1 -- 172.21.15.16:0/2536962419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff040caa40 msgr2=0x7eff040cae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:30.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.816+0000 7eff0aef0700 1 --2- 172.21.15.16:0/2536962419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff040caa40 0x7eff040cae90 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7eff040d0200 tx=0x7efefc00b330 comp rx=0 tx=0).stop 2023-10-16T12:11:30.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.816+0000 7eff0aef0700 1 -- 172.21.15.16:0/2536962419 shutdown_connections 2023-10-16T12:11:30.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.816+0000 7eff0aef0700 1 --2- 172.21.15.16:0/2536962419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff040caa40 0x7eff040cae90 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:30.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.816+0000 7eff0aef0700 1 --2- 172.21.15.16:0/2536962419 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7eff04072b30 0x7eff04071a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:30.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.816+0000 7eff0aef0700 1 --2- 172.21.15.16:0/2536962419 >> v2:172.21.15.49:3300/0 conn(0x7eff0415bb40 0x7eff0406e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:30.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.816+0000 7eff0aef0700 1 -- 172.21.15.16:0/2536962419 >> 172.21.15.16:0/2536962419 conn(0x7eff040b1f40 msgr2=0x7eff040b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:30.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.816+0000 7eff0aef0700 1 -- 172.21.15.16:0/2536962419 shutdown_connections 2023-10-16T12:11:30.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.816+0000 7eff0aef0700 1 -- 172.21.15.16:0/2536962419 wait complete. 2023-10-16T12:11:30.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.818+0000 7eff0aef0700 1 Processor -- start 2023-10-16T12:11:30.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.818+0000 7eff0aef0700 1 -- start start 2023-10-16T12:11:30.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.819+0000 7eff0aef0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff04072b30 0x7eff040d05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:30.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.819+0000 7eff0aef0700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7eff040caa40 0x7eff041f3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:30.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.819+0000 7eff0aef0700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7eff0415bb40 0x7eff041f3980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:30.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.819+0000 7eff0aef0700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff0415db70 con 0x7eff04072b30 2023-10-16T12:11:30.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.819+0000 7eff0aef0700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff0415eb80 con 0x7eff040caa40 2023-10-16T12:11:30.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.819+0000 7eff0aef0700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff0415ddc0 con 0x7eff0415bb40 2023-10-16T12:11:30.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.819+0000 7eff08c8c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff04072b30 0x7eff040d05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:30.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.819+0000 7eff08c8c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff04072b30 0x7eff040d05f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34512/0 (socket says 172.21.15.16:34512) 2023-10-16T12:11:30.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.819+0000 7eff08c8c700 1 -- 172.21.15.16:0/1060803766 learned_addr learned my addr 172.21.15.16:0/1060803766 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:30.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.819+0000 7eff03fff700 1 --2- 172.21.15.16:0/1060803766 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7eff040caa40 0x7eff041f3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:30.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.819+0000 7eff0948d700 1 --2- 172.21.15.16:0/1060803766 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7eff0415bb40 0x7eff041f3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:30.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.820+0000 7eff08c8c700 1 -- 172.21.15.16:0/1060803766 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7eff0415bb40 msgr2=0x7eff041f3980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:30.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.820+0000 7eff08c8c700 1 --2- 172.21.15.16:0/1060803766 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7eff0415bb40 0x7eff041f3980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:30.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.820+0000 7eff08c8c700 1 -- 172.21.15.16:0/1060803766 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7eff040caa40 msgr2=0x7eff041f3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:30.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.820+0000 7eff08c8c700 1 --2- 172.21.15.16:0/1060803766 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7eff040caa40 0x7eff041f3440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:30.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.820+0000 7eff08c8c700 1 -- 172.21.15.16:0/1060803766 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efefc009b30 con 0x7eff04072b30 2023-10-16T12:11:30.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.820+0000 7eff08c8c700 1 --2- 172.21.15.16:0/1060803766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff04072b30 0x7eff040d05f0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7efef800c7f0 tx=0x7efef800cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:30.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.820+0000 7eff017fa700 1 -- 172.21.15.16:0/1060803766 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef8011800 con 0x7eff04072b30 2023-10-16T12:11:30.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.820+0000 7eff017fa700 1 -- 172.21.15.16:0/1060803766 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efef8011960 con 0x7eff04072b30 2023-10-16T12:11:30.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.820+0000 7eff0aef0700 1 -- 172.21.15.16:0/1060803766 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff0415e0a0 con 0x7eff04072b30 2023-10-16T12:11:30.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.820+0000 7eff017fa700 1 -- 172.21.15.16:0/1060803766 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef8011cd0 con 0x7eff04072b30 2023-10-16T12:11:30.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.820+0000 7eff0aef0700 1 -- 172.21.15.16:0/1060803766 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff0415e660 con 0x7eff04072b30 2023-10-16T12:11:30.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.821+0000 7eff0aef0700 1 -- 172.21.15.16:0/1060803766 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff0404ebb0 con 0x7eff04072b30 2023-10-16T12:11:30.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.821+0000 7eff017fa700 1 -- 172.21.15.16:0/1060803766 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7efef8004020 con 0x7eff04072b30 2023-10-16T12:11:30.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.822+0000 7eff017fa700 1 --2- 172.21.15.16:0/1060803766 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7efee806af00 0x7efee806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:30.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.822+0000 7eff017fa700 1 -- 172.21.15.16:0/1060803766 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7efef8089400 con 0x7eff04072b30 2023-10-16T12:11:30.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.822+0000 7eff03fff700 1 --2- 172.21.15.16:0/1060803766 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7efee806af00 0x7efee806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:30.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.822+0000 7eff03fff700 1 --2- 172.21.15.16:0/1060803766 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7efee806af00 0x7efee806d3c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7eff040c6ab0 tx=0x7efef0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:30.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:30.826+0000 7eff017fa700 1 -- 172.21.15.16:0/1060803766 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7efef80540f0 con 0x7eff04072b30 2023-10-16T12:11:31.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.032+0000 7eff0aef0700 1 -- 172.21.15.16:0/1060803766 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7eff040b2d40 con 0x7eff04072b30 2023-10-16T12:11:31.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.032+0000 7eff017fa700 1 -- 172.21.15.16:0/1060803766 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7efef8057710 con 0x7eff04072b30 2023-10-16T12:11:31.035 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-10-16T12:11:31.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.035+0000 7eff0aef0700 1 -- 172.21.15.16:0/1060803766 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7efee806af00 msgr2=0x7efee806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:31.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.035+0000 7eff0aef0700 1 --2- 172.21.15.16:0/1060803766 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7efee806af00 0x7efee806d3c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7eff040c6ab0 tx=0x7efef0008040 comp rx=0 tx=0).stop 2023-10-16T12:11:31.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.035+0000 7eff0aef0700 1 -- 172.21.15.16:0/1060803766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff04072b30 msgr2=0x7eff040d05f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:31.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.035+0000 7eff0aef0700 1 --2- 172.21.15.16:0/1060803766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff04072b30 0x7eff040d05f0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7efef800c7f0 tx=0x7efef800cb00 comp rx=0 tx=0).stop 2023-10-16T12:11:31.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.035+0000 7eff0aef0700 1 -- 172.21.15.16:0/1060803766 shutdown_connections 2023-10-16T12:11:31.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.035+0000 7eff0aef0700 1 --2- 172.21.15.16:0/1060803766 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7efee806af00 0x7efee806d3c0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:31.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.035+0000 7eff0aef0700 1 --2- 172.21.15.16:0/1060803766 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7eff0415bb40 0x7eff041f3980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:31.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.035+0000 7eff0aef0700 1 --2- 172.21.15.16:0/1060803766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff04072b30 0x7eff040d05f0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:31.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.035+0000 7eff0aef0700 1 --2- 172.21.15.16:0/1060803766 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7eff040caa40 0x7eff041f3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:31.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.035+0000 7eff0aef0700 1 -- 172.21.15.16:0/1060803766 >> 172.21.15.16:0/1060803766 conn(0x7eff040b1f40 msgr2=0x7eff040b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:31.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.035+0000 7eff0aef0700 1 -- 172.21.15.16:0/1060803766 shutdown_connections 2023-10-16T12:11:31.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.035+0000 7eff0aef0700 1 -- 172.21.15.16:0/1060803766 wait complete. 2023-10-16T12:11:31.049 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T12:11:31.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.175+0000 7fd7c3c1c700 1 Processor -- start 2023-10-16T12:11:31.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.175+0000 7fd7c3c1c700 1 -- start start 2023-10-16T12:11:31.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.176+0000 7fd7c3c1c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd7bc15b070 0x7fd7bc15b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:31.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.176+0000 7fd7c3c1c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd7bc06e170 0x7fd7bc0cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:31.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.176+0000 7fd7c3c1c700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7bc0cef70 con 0x7fd7bc0cb880 2023-10-16T12:11:31.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.176+0000 7fd7c3c1c700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7bc0ce360 con 0x7fd7bc06e170 2023-10-16T12:11:31.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.176+0000 7fd7c3c1c700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7bc0ce640 con 0x7fd7bc15b070 2023-10-16T12:11:31.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.176+0000 7fd7c11b7700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd7bc0cb880 0x7fd7bc06da60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:52026/0 (socket says 172.21.15.16:52026) 2023-10-16T12:11:31.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.176+0000 7fd7c11b7700 1 -- 172.21.15.16:0/2279602495 learned_addr learned my addr 172.21.15.16:0/2279602495 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:31.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.176+0000 7fd7c21b9700 1 --2- 172.21.15.16:0/2279602495 >> v2:172.21.15.16:3300/0 conn(0x7fd7bc06e170 0x7fd7bc0cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:31.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.176+0000 7fd7c09b6700 1 -- 172.21.15.16:0/2279602495 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3013987894 0 0) 0x7fd7bc0cef70 con 0x7fd7bc0cb880 2023-10-16T12:11:31.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.176+0000 7fd7c09b6700 1 -- 172.21.15.16:0/2279602495 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7a80035b0 con 0x7fd7bc0cb880 2023-10-16T12:11:31.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.176+0000 7fd7c19b8700 1 --2- 172.21.15.16:0/2279602495 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd7bc15b070 0x7fd7bc15b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:31.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.176+0000 7fd7c21b9700 1 -- 172.21.15.16:0/2279602495 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd7bc15b070 msgr2=0x7fd7bc15b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:31.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.176+0000 7fd7c21b9700 1 --2- 172.21.15.16:0/2279602495 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd7bc15b070 0x7fd7bc15b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:31.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.176+0000 7fd7c21b9700 1 -- 172.21.15.16:0/2279602495 >> v1:172.21.15.16:6789/0 conn(0x7fd7bc0cb880 legacy=0x7fd7bc06da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:31.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.176+0000 7fd7c21b9700 1 -- 172.21.15.16:0/2279602495 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7bc0ce920 con 0x7fd7bc06e170 2023-10-16T12:11:31.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.176+0000 7fd7c19b8700 1 --2- 172.21.15.16:0/2279602495 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd7bc15b070 0x7fd7bc15b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:11:31.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.176+0000 7fd7c21b9700 1 --2- 172.21.15.16:0/2279602495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7bc06e170 0x7fd7bc0cbd00 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fd7b4009d90 tx=0x7fd7b400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45982a0ffb83a69c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:31.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.176+0000 7fd7c09b6700 1 -- 172.21.15.16:0/2279602495 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b400e070 con 0x7fd7bc06e170 2023-10-16T12:11:31.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.176+0000 7fd7c09b6700 1 -- 172.21.15.16:0/2279602495 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7b400bb20 con 0x7fd7bc06e170 2023-10-16T12:11:31.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.176+0000 7fd7c09b6700 1 -- 172.21.15.16:0/2279602495 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b400be90 con 0x7fd7bc06e170 2023-10-16T12:11:31.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.177+0000 7fd7c3c1c700 1 -- 172.21.15.16:0/2279602495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7bc06e170 msgr2=0x7fd7bc0cbd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:31.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.177+0000 7fd7c3c1c700 1 --2- 172.21.15.16:0/2279602495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7bc06e170 0x7fd7bc0cbd00 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fd7b4009d90 tx=0x7fd7b400b330 comp rx=0 tx=0).stop 2023-10-16T12:11:31.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.177+0000 7fd7c3c1c700 1 -- 172.21.15.16:0/2279602495 shutdown_connections 2023-10-16T12:11:31.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.177+0000 7fd7c3c1c700 1 --2- 172.21.15.16:0/2279602495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7bc06e170 0x7fd7bc0cbd00 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:31.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.177+0000 7fd7c3c1c700 1 --2- 172.21.15.16:0/2279602495 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd7bc15b070 0x7fd7bc15b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:31.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.177+0000 7fd7c3c1c700 1 -- 172.21.15.16:0/2279602495 >> 172.21.15.16:0/2279602495 conn(0x7fd7bc0b1f40 msgr2=0x7fd7bc0b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:31.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.177+0000 7fd7c3c1c700 1 -- 172.21.15.16:0/2279602495 shutdown_connections 2023-10-16T12:11:31.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.177+0000 7fd7c3c1c700 1 -- 172.21.15.16:0/2279602495 wait complete. 2023-10-16T12:11:31.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.178+0000 7fd7c3c1c700 1 Processor -- start 2023-10-16T12:11:31.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.178+0000 7fd7c3c1c700 1 -- start start 2023-10-16T12:11:31.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.179+0000 7fd7c3c1c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd7bc06e170 0x7fd7bc0b11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:31.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.179+0000 7fd7c3c1c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd7bc0cb880 0x7fd7bc0b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:31.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.179+0000 7fd7c3c1c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7bc15b070 0x7fd7bc1573a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:31.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.179+0000 7fd7c3c1c700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7bc0b4810 con 0x7fd7bc15b070 2023-10-16T12:11:31.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.179+0000 7fd7c3c1c700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7bc157df0 con 0x7fd7bc06e170 2023-10-16T12:11:31.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.179+0000 7fd7c3c1c700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7bc0b4a80 con 0x7fd7bc0cb880 2023-10-16T12:11:31.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.179+0000 7fd7c21b9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7bc15b070 0x7fd7bc1573a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:31.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.179+0000 7fd7c21b9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7bc15b070 0x7fd7bc1573a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34530/0 (socket says 172.21.15.16:34530) 2023-10-16T12:11:31.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.179+0000 7fd7c21b9700 1 -- 172.21.15.16:0/3101526150 learned_addr learned my addr 172.21.15.16:0/3101526150 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:31.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.179+0000 7fd7c19b8700 1 --2- 172.21.15.16:0/3101526150 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd7bc06e170 0x7fd7bc0b11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:31.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.179+0000 7fd7c11b7700 1 --2- 172.21.15.16:0/3101526150 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd7bc0cb880 0x7fd7bc0b5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:31.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.179+0000 7fd7c21b9700 1 -- 172.21.15.16:0/3101526150 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd7bc0cb880 msgr2=0x7fd7bc0b5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:31.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.179+0000 7fd7c21b9700 1 --2- 172.21.15.16:0/3101526150 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd7bc0cb880 0x7fd7bc0b5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:31.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.179+0000 7fd7c21b9700 1 -- 172.21.15.16:0/3101526150 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd7bc06e170 msgr2=0x7fd7bc0b11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:31.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.179+0000 7fd7c21b9700 1 --2- 172.21.15.16:0/3101526150 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd7bc06e170 0x7fd7bc0b11b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:31.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.179+0000 7fd7c21b9700 1 -- 172.21.15.16:0/3101526150 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7b4009a70 con 0x7fd7bc15b070 2023-10-16T12:11:31.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.179+0000 7fd7c19b8700 1 --2- 172.21.15.16:0/3101526150 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd7bc06e170 0x7fd7bc0b11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:11:31.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.180+0000 7fd7c21b9700 1 --2- 172.21.15.16:0/3101526150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7bc15b070 0x7fd7bc1573a0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fd7b400b370 tx=0x7fd7b4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:31.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.180+0000 7fd7b27fc700 1 -- 172.21.15.16:0/3101526150 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b40098f0 con 0x7fd7bc15b070 2023-10-16T12:11:31.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.180+0000 7fd7b27fc700 1 -- 172.21.15.16:0/3101526150 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7b4020070 con 0x7fd7bc15b070 2023-10-16T12:11:31.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.180+0000 7fd7c3c1c700 1 -- 172.21.15.16:0/3101526150 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7bc0b4d60 con 0x7fd7bc15b070 2023-10-16T12:11:31.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.180+0000 7fd7b27fc700 1 -- 172.21.15.16:0/3101526150 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b40276a0 con 0x7fd7bc15b070 2023-10-16T12:11:31.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.180+0000 7fd7c3c1c700 1 -- 172.21.15.16:0/3101526150 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7bc1f3160 con 0x7fd7bc15b070 2023-10-16T12:11:31.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.181+0000 7fd7c3c1c700 1 -- 172.21.15.16:0/3101526150 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7bc1f32f0 con 0x7fd7bc15b070 2023-10-16T12:11:31.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.181+0000 7fd7b27fc700 1 -- 172.21.15.16:0/3101526150 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fd7b4012430 con 0x7fd7bc15b070 2023-10-16T12:11:31.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.182+0000 7fd7b27fc700 1 --2- 172.21.15.16:0/3101526150 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fd7a006af00 0x7fd7a006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:31.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.182+0000 7fd7b27fc700 1 -- 172.21.15.16:0/3101526150 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fd7b4022070 con 0x7fd7bc15b070 2023-10-16T12:11:31.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.182+0000 7fd7c19b8700 1 --2- 172.21.15.16:0/3101526150 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fd7a006af00 0x7fd7a006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:31.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.182+0000 7fd7c19b8700 1 --2- 172.21.15.16:0/3101526150 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fd7a006af00 0x7fd7a006d3c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fd7ac006fd0 tx=0x7fd7ac008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:31.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.187+0000 7fd7b27fc700 1 -- 172.21.15.16:0/3101526150 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd7b40567f0 con 0x7fd7bc15b070 2023-10-16T12:11:31.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.391+0000 7fd7c3c1c700 1 -- 172.21.15.16:0/3101526150 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd7bc04ebb0 con 0x7fd7bc15b070 2023-10-16T12:11:31.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.392+0000 7fd7b27fc700 1 -- 172.21.15.16:0/3101526150 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd7b4059e10 con 0x7fd7bc15b070 2023-10-16T12:11:31.394 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-10-16T12:11:31.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.393+0000 7fd7c3c1c700 1 -- 172.21.15.16:0/3101526150 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fd7a006af00 msgr2=0x7fd7a006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:31.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.393+0000 7fd7c3c1c700 1 --2- 172.21.15.16:0/3101526150 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fd7a006af00 0x7fd7a006d3c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fd7ac006fd0 tx=0x7fd7ac008040 comp rx=0 tx=0).stop 2023-10-16T12:11:31.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.393+0000 7fd7c3c1c700 1 -- 172.21.15.16:0/3101526150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7bc15b070 msgr2=0x7fd7bc1573a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:31.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.393+0000 7fd7c3c1c700 1 --2- 172.21.15.16:0/3101526150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7bc15b070 0x7fd7bc1573a0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fd7b400b370 tx=0x7fd7b4004ad0 comp rx=0 tx=0).stop 2023-10-16T12:11:31.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.394+0000 7fd7c3c1c700 1 -- 172.21.15.16:0/3101526150 shutdown_connections 2023-10-16T12:11:31.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.394+0000 7fd7c3c1c700 1 --2- 172.21.15.16:0/3101526150 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fd7a006af00 0x7fd7a006d3c0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:31.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.394+0000 7fd7c3c1c700 1 --2- 172.21.15.16:0/3101526150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7bc15b070 0x7fd7bc1573a0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:31.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.394+0000 7fd7c3c1c700 1 --2- 172.21.15.16:0/3101526150 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd7bc06e170 0x7fd7bc0b11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:31.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.394+0000 7fd7c3c1c700 1 --2- 172.21.15.16:0/3101526150 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd7bc0cb880 0x7fd7bc0b5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:31.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.394+0000 7fd7c3c1c700 1 -- 172.21.15.16:0/3101526150 >> 172.21.15.16:0/3101526150 conn(0x7fd7bc0b1f40 msgr2=0x7fd7bc0d0380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:31.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.394+0000 7fd7c3c1c700 1 -- 172.21.15.16:0/3101526150 shutdown_connections 2023-10-16T12:11:31.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.394+0000 7fd7c3c1c700 1 -- 172.21.15.16:0/3101526150 wait complete. 2023-10-16T12:11:31.408 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-16T12:11:31.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.533+0000 7feed612d700 1 Processor -- start 2023-10-16T12:11:31.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.534+0000 7feed612d700 1 -- start start 2023-10-16T12:11:31.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.534+0000 7feed612d700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7feed015cd10 0x7feed00cd0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:31.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.534+0000 7feed612d700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feed00cc920 con 0x7feed006e6b0 2023-10-16T12:11:31.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.534+0000 7feed612d700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feed015c330 con 0x7feed00ca9b0 2023-10-16T12:11:31.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.534+0000 7feed612d700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feed015c610 con 0x7feed015cd10 2023-10-16T12:11:31.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.534+0000 7feeceffd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7feed006e6b0 0x7feed00cad90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:52042/0 (socket says 172.21.15.16:52042) 2023-10-16T12:11:31.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.534+0000 7feeceffd700 1 -- 172.21.15.16:0/2284232996 learned_addr learned my addr 172.21.15.16:0/2284232996 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:31.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.535+0000 7feecffff700 1 --2- 172.21.15.16:0/2284232996 >> v2:172.21.15.49:3300/0 conn(0x7feed015cd10 0x7feed00cd0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:31.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.535+0000 7feece7fc700 1 -- 172.21.15.16:0/2284232996 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1377732914 0 0) 0x7feed00cc920 con 0x7feed006e6b0 2023-10-16T12:11:31.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.535+0000 7feece7fc700 1 -- 172.21.15.16:0/2284232996 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feebc0035b0 con 0x7feed006e6b0 2023-10-16T12:11:31.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.535+0000 7feece7fc700 1 -- 172.21.15.16:0/2284232996 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 137328610 0 0) 0x7feebc0035b0 con 0x7feed006e6b0 2023-10-16T12:11:31.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.535+0000 7feece7fc700 1 -- 172.21.15.16:0/2284232996 >> v2:172.21.15.49:3300/0 conn(0x7feed015cd10 msgr2=0x7feed00cd0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:31.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.535+0000 7feece7fc700 1 --2- 172.21.15.16:0/2284232996 >> v2:172.21.15.49:3300/0 conn(0x7feed015cd10 0x7feed00cd0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:31.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.535+0000 7feece7fc700 1 -- 172.21.15.16:0/2284232996 >> v1:172.21.15.49:6789/0 conn(0x7feed00ca9b0 legacy=0x7feed015bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:31.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.535+0000 7feece7fc700 1 -- 172.21.15.16:0/2284232996 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feed015c8f0 con 0x7feed006e6b0 2023-10-16T12:11:31.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.535+0000 7feece7fc700 1 -- 172.21.15.16:0/2284232996 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7feec0001670 con 0x7feed006e6b0 2023-10-16T12:11:31.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.535+0000 7feece7fc700 1 -- 172.21.15.16:0/2284232996 >> v1:172.21.15.16:6789/0 conn(0x7feed006e6b0 legacy=0x7feed00cad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:31.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.535+0000 7feece7fc700 1 --2- 172.21.15.16:0/2284232996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feebc003a00 0x7feebc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:31.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.535+0000 7feece7fc700 1 --2- 172.21.15.16:0/2284232996 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feebc004850 0x7feebc004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:31.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.535+0000 7feece7fc700 1 --2- 172.21.15.16:0/2284232996 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7feebc005210 0x7feebc00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:31.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.535+0000 7feece7fc700 1 -- 172.21.15.16:0/2284232996 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feed015c610 con 0x7feebc003a00 2023-10-16T12:11:31.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.535+0000 7feece7fc700 1 -- 172.21.15.16:0/2284232996 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feebc00c0b0 con 0x7feebc004850 2023-10-16T12:11:31.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.535+0000 7feece7fc700 1 -- 172.21.15.16:0/2284232996 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feebc00c270 con 0x7feebc005210 2023-10-16T12:11:31.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.535+0000 7feecf7fe700 1 --2- 172.21.15.16:0/2284232996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feebc003a00 0x7feebc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:31.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.535+0000 7feeceffd700 1 --2- 172.21.15.16:0/2284232996 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feebc004850 0x7feebc004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:31.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.535+0000 7feecffff700 1 --2- 172.21.15.16:0/2284232996 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7feebc005210 0x7feebc00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:31.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.536+0000 7feecf7fe700 1 -- 172.21.15.16:0/2284232996 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7feebc005210 msgr2=0x7feebc00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:31.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.536+0000 7feecf7fe700 1 --2- 172.21.15.16:0/2284232996 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7feebc005210 0x7feebc00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:31.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.536+0000 7feecf7fe700 1 -- 172.21.15.16:0/2284232996 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feebc004850 msgr2=0x7feebc004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:31.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.536+0000 7feecf7fe700 1 --2- 172.21.15.16:0/2284232996 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feebc004850 0x7feebc004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:31.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.536+0000 7feecf7fe700 1 -- 172.21.15.16:0/2284232996 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feebc0035b0 con 0x7feebc003a00 2023-10-16T12:11:31.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.536+0000 7feecffff700 1 -- 172.21.15.16:0/2284232996 reap_dead start 2023-10-16T12:11:31.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.536+0000 7feecf7fe700 1 --2- 172.21.15.16:0/2284232996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feebc003a00 0x7feebc003e20 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7feeb8001820 tx=0x7feebc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb3e2a696faf0c40 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:31.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.536+0000 7feece7fc700 1 -- 172.21.15.16:0/2284232996 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeb80012e0 con 0x7feebc003a00 2023-10-16T12:11:31.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.536+0000 7feece7fc700 1 -- 172.21.15.16:0/2284232996 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feeb80052d0 con 0x7feebc003a00 2023-10-16T12:11:31.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.536+0000 7feece7fc700 1 -- 172.21.15.16:0/2284232996 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeb80036d0 con 0x7feebc003a00 2023-10-16T12:11:31.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.536+0000 7feed612d700 1 -- 172.21.15.16:0/2284232996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feebc003a00 msgr2=0x7feebc003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:31.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.536+0000 7feed612d700 1 --2- 172.21.15.16:0/2284232996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feebc003a00 0x7feebc003e20 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7feeb8001820 tx=0x7feebc005880 comp rx=0 tx=0).stop 2023-10-16T12:11:31.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.536+0000 7feed612d700 1 -- 172.21.15.16:0/2284232996 shutdown_connections 2023-10-16T12:11:31.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.536+0000 7feed612d700 1 --2- 172.21.15.16:0/2284232996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feebc003a00 0x7feebc003e20 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:31.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.536+0000 7feed612d700 1 -- 172.21.15.16:0/2284232996 >> 172.21.15.16:0/2284232996 conn(0x7feed00b1ed0 msgr2=0x7feed00b22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:31.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.536+0000 7feed612d700 1 -- 172.21.15.16:0/2284232996 shutdown_connections 2023-10-16T12:11:31.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.536+0000 7feed612d700 1 -- 172.21.15.16:0/2284232996 wait complete. 2023-10-16T12:11:31.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.538+0000 7feed612d700 1 Processor -- start 2023-10-16T12:11:31.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.538+0000 7feed612d700 1 -- start start 2023-10-16T12:11:31.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.538+0000 7feed612d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feebc003a00 0x7feed01f0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:31.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.538+0000 7feed612d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7feed015e130 0x7feed0157ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:31.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.539+0000 7feed612d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feed00c59c0 0x7feed00c5e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:31.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.539+0000 7feed612d700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feed0159030 con 0x7feed00c59c0 2023-10-16T12:11:31.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.539+0000 7feed612d700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feed00c6390 con 0x7feebc003a00 2023-10-16T12:11:31.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.539+0000 7feed612d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feed00c80f0 con 0x7feed015e130 2023-10-16T12:11:31.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.539+0000 7feecffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feed00c59c0 0x7feed00c5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:31.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.539+0000 7feecffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feed00c59c0 0x7feed00c5e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34550/0 (socket says 172.21.15.16:34550) 2023-10-16T12:11:31.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.539+0000 7feecffff700 1 -- 172.21.15.16:0/3434787358 learned_addr learned my addr 172.21.15.16:0/3434787358 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:31.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.539+0000 7feecf7fe700 1 --2- 172.21.15.16:0/3434787358 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feebc003a00 0x7feed01f0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:31.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.539+0000 7feeceffd700 1 --2- 172.21.15.16:0/3434787358 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7feed015e130 0x7feed0157ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:31.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.539+0000 7feecffff700 1 -- 172.21.15.16:0/3434787358 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7feed015e130 msgr2=0x7feed0157ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:31.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.539+0000 7feecffff700 1 --2- 172.21.15.16:0/3434787358 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7feed015e130 0x7feed0157ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:31.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.539+0000 7feecffff700 1 -- 172.21.15.16:0/3434787358 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feebc003a00 msgr2=0x7feed01f0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:31.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.539+0000 7feecffff700 1 --2- 172.21.15.16:0/3434787358 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feebc003a00 0x7feed01f0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:31.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.539+0000 7feecffff700 1 -- 172.21.15.16:0/3434787358 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feed01584b0 con 0x7feed00c59c0 2023-10-16T12:11:31.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.539+0000 7feecf7fe700 1 --2- 172.21.15.16:0/3434787358 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feebc003a00 0x7feed01f0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:11:31.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.539+0000 7feeceffd700 1 --2- 172.21.15.16:0/3434787358 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7feed015e130 0x7feed0157ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:11:31.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.539+0000 7feecffff700 1 --2- 172.21.15.16:0/3434787358 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feed00c59c0 0x7feed00c5e50 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7feed00c73c0 tx=0x7feec8011e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:31.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.539+0000 7feec7fff700 1 -- 172.21.15.16:0/3434787358 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feec800e040 con 0x7feed00c59c0 2023-10-16T12:11:31.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.539+0000 7feec7fff700 1 -- 172.21.15.16:0/3434787358 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feec800f040 con 0x7feed00c59c0 2023-10-16T12:11:31.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.539+0000 7feed612d700 1 -- 172.21.15.16:0/3434787358 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feed0158790 con 0x7feed00c59c0 2023-10-16T12:11:31.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.539+0000 7feec7fff700 1 -- 172.21.15.16:0/3434787358 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feec800fc40 con 0x7feed00c59c0 2023-10-16T12:11:31.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.539+0000 7feed612d700 1 -- 172.21.15.16:0/3434787358 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feed0158d50 con 0x7feed00c59c0 2023-10-16T12:11:31.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.540+0000 7feed612d700 1 -- 172.21.15.16:0/3434787358 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feed00c82b0 con 0x7feed00c59c0 2023-10-16T12:11:31.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.541+0000 7feec7fff700 1 -- 172.21.15.16:0/3434787358 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7feec8015430 con 0x7feed00c59c0 2023-10-16T12:11:31.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.541+0000 7feec7fff700 1 --2- 172.21.15.16:0/3434787358 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7feeb006af00 0x7feeb006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:31.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.541+0000 7feec7fff700 1 -- 172.21.15.16:0/3434787358 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7feec8089d10 con 0x7feed00c59c0 2023-10-16T12:11:31.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.541+0000 7feecf7fe700 1 --2- 172.21.15.16:0/3434787358 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7feeb006af00 0x7feeb006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:31.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.542+0000 7feecf7fe700 1 --2- 172.21.15.16:0/3434787358 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7feeb006af00 0x7feeb006d3c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7feeb8015040 tx=0x7feeb8011000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:31.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.546+0000 7feec7fff700 1 -- 172.21.15.16:0/3434787358 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7feec8054a00 con 0x7feed00c59c0 2023-10-16T12:11:31.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.748+0000 7feed612d700 1 -- 172.21.15.16:0/3434787358 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7feed004ebb0 con 0x7feed00c59c0 2023-10-16T12:11:31.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.748+0000 7feec7fff700 1 -- 172.21.15.16:0/3434787358 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7feec8058020 con 0x7feed00c59c0 2023-10-16T12:11:31.751 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-10-16T12:11:31.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.750+0000 7feed612d700 1 -- 172.21.15.16:0/3434787358 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7feeb006af00 msgr2=0x7feeb006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:31.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.750+0000 7feed612d700 1 --2- 172.21.15.16:0/3434787358 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7feeb006af00 0x7feeb006d3c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7feeb8015040 tx=0x7feeb8011000 comp rx=0 tx=0).stop 2023-10-16T12:11:31.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.751+0000 7feed612d700 1 -- 172.21.15.16:0/3434787358 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feed00c59c0 msgr2=0x7feed00c5e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:31.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.751+0000 7feed612d700 1 --2- 172.21.15.16:0/3434787358 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feed00c59c0 0x7feed00c5e50 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7feed00c73c0 tx=0x7feec8011e20 comp rx=0 tx=0).stop 2023-10-16T12:11:31.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.751+0000 7feed612d700 1 -- 172.21.15.16:0/3434787358 shutdown_connections 2023-10-16T12:11:31.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.751+0000 7feed612d700 1 --2- 172.21.15.16:0/3434787358 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7feeb006af00 0x7feeb006d3c0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:31.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.751+0000 7feed612d700 1 --2- 172.21.15.16:0/3434787358 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feed00c59c0 0x7feed00c5e50 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:31.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.751+0000 7feed612d700 1 --2- 172.21.15.16:0/3434787358 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feebc003a00 0x7feed01f0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:31.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.751+0000 7feed612d700 1 --2- 172.21.15.16:0/3434787358 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7feed015e130 0x7feed0157ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:31.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.751+0000 7feed612d700 1 -- 172.21.15.16:0/3434787358 >> 172.21.15.16:0/3434787358 conn(0x7feed00b1ed0 msgr2=0x7feed0159850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:31.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.751+0000 7feed612d700 1 -- 172.21.15.16:0/3434787358 shutdown_connections 2023-10-16T12:11:31.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.751+0000 7feed612d700 1 -- 172.21.15.16:0/3434787358 wait complete. 2023-10-16T12:11:31.775 INFO:tasks.cephfs_test_runner:Starting test: test_concurrent_snap_creates (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test concurrent snap creates in same file-system without db issues 2023-10-16T12:11:31.776 INFO:tasks.cephfs_test_runner:test_concurrent_snap_creates (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-16T12:11:31.776 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:11:31.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.900+0000 7f7fb9dae700 1 Processor -- start 2023-10-16T12:11:31.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.900+0000 7f7fb9dae700 1 -- start start 2023-10-16T12:11:31.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.901+0000 7f7fb9dae700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7fb415b280 0x7f7fb41613e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:31.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.901+0000 7f7fb9dae700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f7fb406a7e0 0x7f7fb406ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:31.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.901+0000 7f7fb9dae700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fb406b170 con 0x7f7fb40cc8a0 2023-10-16T12:11:31.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.901+0000 7f7fb9dae700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fb4161c30 con 0x7f7fb406a7e0 2023-10-16T12:11:31.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.901+0000 7f7fb9dae700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb4161f10 con 0x7f7fb415b280 2023-10-16T12:11:31.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.901+0000 7f7fb3fff700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f7fb406a7e0 0x7f7fb406ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:31.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.901+0000 7f7fb37fe700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7fb415b280 0x7f7fb41613e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:31.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.901+0000 7f7fb2ffd700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7f7fb40cc8a0 0x7f7fb415e700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:59242/0 (socket says 172.21.15.16:59242) 2023-10-16T12:11:31.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.901+0000 7f7fb2ffd700 1 -- 172.21.15.16:0/2571178700 learned_addr learned my addr 172.21.15.16:0/2571178700 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:31.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.901+0000 7f7fb3fff700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f7fb406a7e0 0x7f7fb406ac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:33620/0 (socket says 172.21.15.16:33620) 2023-10-16T12:11:31.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.901+0000 7f7fb3fff700 1 -- 172.21.15.16:0/2571178700 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7fb415b280 msgr2=0x7f7fb41613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:31.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.901+0000 7f7fb3fff700 1 --2- 172.21.15.16:0/2571178700 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7fb415b280 0x7f7fb41613e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:31.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.901+0000 7f7fb3fff700 1 -- 172.21.15.16:0/2571178700 >> v1:172.21.15.49:6789/0 conn(0x7f7fb40cc8a0 legacy=0x7f7fb415e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:31.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.901+0000 7f7fb3fff700 1 -- 172.21.15.16:0/2571178700 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fb41621f0 con 0x7f7fb406a7e0 2023-10-16T12:11:31.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.901+0000 7f7fb37fe700 1 --2- 172.21.15.16:0/2571178700 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7fb415b280 0x7f7fb41613e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:11:31.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.902+0000 7f7fb3fff700 1 --2- 172.21.15.16:0/2571178700 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7fb406a7e0 0x7f7fb406ac30 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f7fac009d90 tx=0x7f7fac00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d4f082f7208dab4f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:31.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.902+0000 7f7fb27fc700 1 -- 172.21.15.16:0/2571178700 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fac00e070 con 0x7f7fb406a7e0 2023-10-16T12:11:31.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.902+0000 7f7fb27fc700 1 -- 172.21.15.16:0/2571178700 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fac00bb20 con 0x7f7fb406a7e0 2023-10-16T12:11:31.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.902+0000 7f7fb27fc700 1 -- 172.21.15.16:0/2571178700 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fac012430 con 0x7f7fb406a7e0 2023-10-16T12:11:31.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.902+0000 7f7fb9dae700 1 -- 172.21.15.16:0/2571178700 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7fb406a7e0 msgr2=0x7f7fb406ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:31.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.902+0000 7f7fb9dae700 1 --2- 172.21.15.16:0/2571178700 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7fb406a7e0 0x7f7fb406ac30 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f7fac009d90 tx=0x7f7fac00b330 comp rx=0 tx=0).stop 2023-10-16T12:11:31.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.902+0000 7f7fb9dae700 1 -- 172.21.15.16:0/2571178700 shutdown_connections 2023-10-16T12:11:31.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.902+0000 7f7fb9dae700 1 --2- 172.21.15.16:0/2571178700 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7fb406a7e0 0x7f7fb406ac30 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:31.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.902+0000 7f7fb9dae700 1 --2- 172.21.15.16:0/2571178700 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7fb415b280 0x7f7fb41613e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:31.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.902+0000 7f7fb9dae700 1 -- 172.21.15.16:0/2571178700 >> 172.21.15.16:0/2571178700 conn(0x7f7fb40b1fd0 msgr2=0x7f7fb40b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:31.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.902+0000 7f7fb9dae700 1 -- 172.21.15.16:0/2571178700 shutdown_connections 2023-10-16T12:11:31.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.903+0000 7f7fb9dae700 1 -- 172.21.15.16:0/2571178700 wait complete. 2023-10-16T12:11:31.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.904+0000 7f7fb9dae700 1 Processor -- start 2023-10-16T12:11:31.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.904+0000 7f7fb9dae700 1 -- start start 2023-10-16T12:11:31.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.905+0000 7f7fb9dae700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7fb406a7e0 0x7f7fb40c9e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:31.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.905+0000 7f7fb9dae700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7fb40cc8a0 0x7f7fb4156d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:31.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.905+0000 7f7fb9dae700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7fb415b280 0x7f7fb41f6460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:31.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.905+0000 7f7fb9dae700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb415d030 con 0x7f7fb415b280 2023-10-16T12:11:31.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.905+0000 7f7fb9dae700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb40ca360 con 0x7f7fb40cc8a0 2023-10-16T12:11:31.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.905+0000 7f7fb9dae700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb41572a0 con 0x7f7fb406a7e0 2023-10-16T12:11:31.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.905+0000 7f7fb3fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7fb415b280 0x7f7fb41f6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:31.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.905+0000 7f7fb3fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7fb415b280 0x7f7fb41f6460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34564/0 (socket says 172.21.15.16:34564) 2023-10-16T12:11:31.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.905+0000 7f7fb3fff700 1 -- 172.21.15.16:0/1123769692 learned_addr learned my addr 172.21.15.16:0/1123769692 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:31.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.905+0000 7f7fb37fe700 1 --2- 172.21.15.16:0/1123769692 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7fb406a7e0 0x7f7fb40c9e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:31.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.905+0000 7f7fb2ffd700 1 --2- 172.21.15.16:0/1123769692 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7fb40cc8a0 0x7f7fb4156d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:31.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.905+0000 7f7fb3fff700 1 -- 172.21.15.16:0/1123769692 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7fb406a7e0 msgr2=0x7f7fb40c9e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:31.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.905+0000 7f7fb3fff700 1 --2- 172.21.15.16:0/1123769692 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7fb406a7e0 0x7f7fb40c9e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:31.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.905+0000 7f7fb3fff700 1 -- 172.21.15.16:0/1123769692 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7fb40cc8a0 msgr2=0x7f7fb4156d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:31.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.905+0000 7f7fb3fff700 1 --2- 172.21.15.16:0/1123769692 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7fb40cc8a0 0x7f7fb4156d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:31.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.905+0000 7f7fb3fff700 1 -- 172.21.15.16:0/1123769692 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f9c0094a0 con 0x7f7fb415b280 2023-10-16T12:11:31.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.905+0000 7f7fb3fff700 1 --2- 172.21.15.16:0/1123769692 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7fb415b280 0x7f7fb41f6460 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f7fac014040 tx=0x7f7fac004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:31.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.905+0000 7f7fabfff700 1 -- 172.21.15.16:0/1123769692 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fac012430 con 0x7f7fb415b280 2023-10-16T12:11:31.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.905+0000 7f7fabfff700 1 -- 172.21.15.16:0/1123769692 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fac020070 con 0x7f7fb415b280 2023-10-16T12:11:31.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.905+0000 7f7fb9dae700 1 -- 172.21.15.16:0/1123769692 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fac009a70 con 0x7f7fb415b280 2023-10-16T12:11:31.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.905+0000 7f7fabfff700 1 -- 172.21.15.16:0/1123769692 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fac004280 con 0x7f7fb415b280 2023-10-16T12:11:31.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.906+0000 7f7fb9dae700 1 -- 172.21.15.16:0/1123769692 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fb41577e0 con 0x7f7fb415b280 2023-10-16T12:11:31.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.907+0000 7f7fb9dae700 1 -- 172.21.15.16:0/1123769692 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fb4157640 con 0x7f7fb415b280 2023-10-16T12:11:31.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.907+0000 7f7fabfff700 1 -- 172.21.15.16:0/1123769692 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f7fac0043e0 con 0x7f7fb415b280 2023-10-16T12:11:31.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.907+0000 7f7fabfff700 1 --2- 172.21.15.16:0/1123769692 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f7f9406af00 0x7f7f9406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:31.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.907+0000 7f7fabfff700 1 -- 172.21.15.16:0/1123769692 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f7fac022070 con 0x7f7fb415b280 2023-10-16T12:11:31.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.907+0000 7f7fb37fe700 1 --2- 172.21.15.16:0/1123769692 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f7f9406af00 0x7f7f9406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:31.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.908+0000 7f7fb37fe700 1 --2- 172.21.15.16:0/1123769692 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f7f9406af00 0x7f7f9406d3c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7f9c009cd0 tx=0x7f7f9c009210 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:31.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:31.912+0000 7f7fabfff700 1 -- 172.21.15.16:0/1123769692 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7fb4157640 con 0x7f7fb415b280 2023-10-16T12:11:32.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.118+0000 7f7fb9dae700 1 -- 172.21.15.16:0/1123769692 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7fb4157640 con 0x7f7fb415b280 2023-10-16T12:11:32.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.119+0000 7f7fabfff700 1 -- 172.21.15.16:0/1123769692 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18629 (secure 0 0 0) 0x7f7fb4157640 con 0x7f7fb415b280 2023-10-16T12:11:32.121 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:11:32.121 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:11:28.241865+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:10:36.935773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T12:10:38.004870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-16T12:10:41.672025+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"3.2b","mappings":[{"from":0,"to":2}]},{"pgid":"3.38","mappings":[{"from":3,"to":1}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]},{"pgid":"4.1b","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:11:32.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.121+0000 7f7fb9dae700 1 -- 172.21.15.16:0/1123769692 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f7f9406af00 msgr2=0x7f7f9406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:32.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.121+0000 7f7fb9dae700 1 --2- 172.21.15.16:0/1123769692 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f7f9406af00 0x7f7f9406d3c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7f9c009cd0 tx=0x7f7f9c009210 comp rx=0 tx=0).stop 2023-10-16T12:11:32.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.121+0000 7f7fb9dae700 1 -- 172.21.15.16:0/1123769692 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7fb415b280 msgr2=0x7f7fb41f6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:32.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.121+0000 7f7fb9dae700 1 --2- 172.21.15.16:0/1123769692 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7fb415b280 0x7f7fb41f6460 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f7fac014040 tx=0x7f7fac004ad0 comp rx=0 tx=0).stop 2023-10-16T12:11:32.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.121+0000 7f7fb9dae700 1 -- 172.21.15.16:0/1123769692 shutdown_connections 2023-10-16T12:11:32.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.121+0000 7f7fb9dae700 1 --2- 172.21.15.16:0/1123769692 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f7f9406af00 0x7f7f9406d3c0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:32.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.121+0000 7f7fb9dae700 1 --2- 172.21.15.16:0/1123769692 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7fb415b280 0x7f7fb41f6460 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:32.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.121+0000 7f7fb9dae700 1 --2- 172.21.15.16:0/1123769692 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7fb406a7e0 0x7f7fb40c9e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:32.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.121+0000 7f7fb9dae700 1 --2- 172.21.15.16:0/1123769692 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7fb40cc8a0 0x7f7fb4156d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:32.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.121+0000 7f7fb9dae700 1 -- 172.21.15.16:0/1123769692 >> 172.21.15.16:0/1123769692 conn(0x7f7fb40b1fd0 msgr2=0x7f7fb415a570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:32.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.121+0000 7f7fb9dae700 1 -- 172.21.15.16:0/1123769692 shutdown_connections 2023-10-16T12:11:32.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.121+0000 7f7fb9dae700 1 -- 172.21.15.16:0/1123769692 wait complete. 2023-10-16T12:11:32.135 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T12:10:36.935773+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T12:10:38.004870+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-16T12:10:41.672025+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-16T12:11:32.136 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:11:32.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.263+0000 7fd20e784700 1 Processor -- start 2023-10-16T12:11:32.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.263+0000 7fd20e784700 1 -- start start 2023-10-16T12:11:32.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.264+0000 7fd20e784700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd20815b070 0x7fd20815b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:32.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.265+0000 7fd20e784700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd2080cb880 0x7fd2080cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:32.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.265+0000 7fd20e784700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2080cef70 con 0x7fd20806da60 2023-10-16T12:11:32.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.265+0000 7fd20e784700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2080ce360 con 0x7fd20815b070 2023-10-16T12:11:32.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.265+0000 7fd20e784700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2080ce640 con 0x7fd2080cb880 2023-10-16T12:11:32.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.265+0000 7fd207fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd20815b070 0x7fd20815b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:32.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.265+0000 7fd207fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd20815b070 0x7fd20815b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34568/0 (socket says 172.21.15.16:34568) 2023-10-16T12:11:32.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.265+0000 7fd207fff700 1 -- 172.21.15.16:0/385555488 learned_addr learned my addr 172.21.15.16:0/385555488 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:32.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.265+0000 7fd2077fe700 1 --2- 172.21.15.16:0/385555488 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd2080cb880 0x7fd2080cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:32.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.265+0000 7fd207fff700 1 -- 172.21.15.16:0/385555488 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd2080cb880 msgr2=0x7fd2080cbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:32.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.265+0000 7fd207fff700 1 --2- 172.21.15.16:0/385555488 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd2080cb880 0x7fd2080cbd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:32.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.265+0000 7fd207fff700 1 -- 172.21.15.16:0/385555488 >> v1:172.21.15.49:6789/0 conn(0x7fd20806da60 legacy=0x7fd2080cf400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:32.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.265+0000 7fd207fff700 1 -- 172.21.15.16:0/385555488 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2080ce920 con 0x7fd20815b070 2023-10-16T12:11:32.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.265+0000 7fd2077fe700 1 --2- 172.21.15.16:0/385555488 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd2080cb880 0x7fd2080cbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:11:32.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.265+0000 7fd207fff700 1 --2- 172.21.15.16:0/385555488 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd20815b070 0x7fd20815b450 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fd1f80098f0 tx=0x7fd1f8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42e884fb53442cc8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:32.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.265+0000 7fd206ffd700 1 -- 172.21.15.16:0/385555488 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1f800b8a0 con 0x7fd20815b070 2023-10-16T12:11:32.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.265+0000 7fd206ffd700 1 -- 172.21.15.16:0/385555488 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1f800ba00 con 0x7fd20815b070 2023-10-16T12:11:32.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.265+0000 7fd206ffd700 1 -- 172.21.15.16:0/385555488 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1f800bd90 con 0x7fd20815b070 2023-10-16T12:11:32.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.266+0000 7fd20e784700 1 -- 172.21.15.16:0/385555488 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd20815b070 msgr2=0x7fd20815b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:32.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.266+0000 7fd20e784700 1 --2- 172.21.15.16:0/385555488 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd20815b070 0x7fd20815b450 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fd1f80098f0 tx=0x7fd1f8009c00 comp rx=0 tx=0).stop 2023-10-16T12:11:32.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.266+0000 7fd20e784700 1 -- 172.21.15.16:0/385555488 shutdown_connections 2023-10-16T12:11:32.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.266+0000 7fd20e784700 1 --2- 172.21.15.16:0/385555488 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd20815b070 0x7fd20815b450 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:32.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.266+0000 7fd20e784700 1 --2- 172.21.15.16:0/385555488 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd2080cb880 0x7fd2080cbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:32.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.266+0000 7fd20e784700 1 -- 172.21.15.16:0/385555488 >> 172.21.15.16:0/385555488 conn(0x7fd2080b1f40 msgr2=0x7fd2080b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:32.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.266+0000 7fd20e784700 1 -- 172.21.15.16:0/385555488 shutdown_connections 2023-10-16T12:11:32.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.266+0000 7fd20e784700 1 -- 172.21.15.16:0/385555488 wait complete. 2023-10-16T12:11:32.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.268+0000 7fd20e784700 1 Processor -- start 2023-10-16T12:11:32.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.268+0000 7fd20e784700 1 -- start start 2023-10-16T12:11:32.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.269+0000 7fd20e784700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd20806da60 0x7fd208147530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:32.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.269+0000 7fd20e784700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd2080cb880 0x7fd2080c5f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:32.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.269+0000 7fd20e784700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd20815b070 0x7fd2081533c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:32.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.269+0000 7fd20e784700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd208152fa0 con 0x7fd20806da60 2023-10-16T12:11:32.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.269+0000 7fd20e784700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd208152c00 con 0x7fd20815b070 2023-10-16T12:11:32.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.269+0000 7fd20e784700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2080c9f40 con 0x7fd2080cb880 2023-10-16T12:11:32.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.269+0000 7fd207fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd20806da60 0x7fd208147530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:32.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.269+0000 7fd207fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd20806da60 0x7fd208147530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34574/0 (socket says 172.21.15.16:34574) 2023-10-16T12:11:32.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.269+0000 7fd207fff700 1 -- 172.21.15.16:0/2482123368 learned_addr learned my addr 172.21.15.16:0/2482123368 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:32.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.269+0000 7fd20cd21700 1 --2- 172.21.15.16:0/2482123368 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd20815b070 0x7fd2081533c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:32.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.269+0000 7fd2077fe700 1 --2- 172.21.15.16:0/2482123368 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd2080cb880 0x7fd2080c5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:32.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.269+0000 7fd207fff700 1 -- 172.21.15.16:0/2482123368 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd2080cb880 msgr2=0x7fd2080c5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:32.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.269+0000 7fd207fff700 1 --2- 172.21.15.16:0/2482123368 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd2080cb880 0x7fd2080c5f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:32.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.269+0000 7fd207fff700 1 -- 172.21.15.16:0/2482123368 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd20815b070 msgr2=0x7fd2081533c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:32.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.269+0000 7fd207fff700 1 --2- 172.21.15.16:0/2482123368 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd20815b070 0x7fd2081533c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:32.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.269+0000 7fd207fff700 1 -- 172.21.15.16:0/2482123368 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1f8009550 con 0x7fd20806da60 2023-10-16T12:11:32.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.269+0000 7fd20cd21700 1 --2- 172.21.15.16:0/2482123368 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd20815b070 0x7fd2081533c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:11:32.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.269+0000 7fd207fff700 1 --2- 172.21.15.16:0/2482123368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd20806da60 0x7fd208147530 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd1f8009c40 tx=0x7fd1f8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:32.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.270+0000 7fd204ff9700 1 -- 172.21.15.16:0/2482123368 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1f80036a0 con 0x7fd20806da60 2023-10-16T12:11:32.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.270+0000 7fd204ff9700 1 -- 172.21.15.16:0/2482123368 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1f8003800 con 0x7fd20806da60 2023-10-16T12:11:32.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.270+0000 7fd20e784700 1 -- 172.21.15.16:0/2482123368 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2080ca220 con 0x7fd20806da60 2023-10-16T12:11:32.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.270+0000 7fd204ff9700 1 -- 172.21.15.16:0/2482123368 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1f8003bb0 con 0x7fd20806da60 2023-10-16T12:11:32.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.270+0000 7fd20e784700 1 -- 172.21.15.16:0/2482123368 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2080ca500 con 0x7fd20806da60 2023-10-16T12:11:32.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.271+0000 7fd20e784700 1 -- 172.21.15.16:0/2482123368 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd208153210 con 0x7fd20806da60 2023-10-16T12:11:32.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.271+0000 7fd204ff9700 1 -- 172.21.15.16:0/2482123368 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fd1f8003d10 con 0x7fd20806da60 2023-10-16T12:11:32.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.272+0000 7fd204ff9700 1 --2- 172.21.15.16:0/2482123368 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fd1f406af00 0x7fd1f406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:32.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.272+0000 7fd2077fe700 1 --2- 172.21.15.16:0/2482123368 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fd1f406af00 0x7fd1f406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:32.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.272+0000 7fd204ff9700 1 -- 172.21.15.16:0/2482123368 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fd1f808ce00 con 0x7fd20806da60 2023-10-16T12:11:32.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.272+0000 7fd2077fe700 1 --2- 172.21.15.16:0/2482123368 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fd1f406af00 0x7fd1f406d3c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fd2081f3570 tx=0x7fd1fc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:32.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.276+0000 7fd204ff9700 1 -- 172.21.15.16:0/2482123368 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd1f8057ca0 con 0x7fd20806da60 2023-10-16T12:11:32.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.476+0000 7fd20e784700 1 -- 172.21.15.16:0/2482123368 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd20804ebb0 con 0x7fd20806da60 2023-10-16T12:11:32.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.477+0000 7fd204ff9700 1 -- 172.21.15.16:0/2482123368 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd1f80150a0 con 0x7fd20806da60 2023-10-16T12:11:32.479 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T12:11:32.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.479+0000 7fd20e784700 1 -- 172.21.15.16:0/2482123368 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fd1f406af00 msgr2=0x7fd1f406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:32.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.479+0000 7fd20e784700 1 --2- 172.21.15.16:0/2482123368 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fd1f406af00 0x7fd1f406d3c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fd2081f3570 tx=0x7fd1fc008040 comp rx=0 tx=0).stop 2023-10-16T12:11:32.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.479+0000 7fd20e784700 1 -- 172.21.15.16:0/2482123368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd20806da60 msgr2=0x7fd208147530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:32.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.479+0000 7fd20e784700 1 --2- 172.21.15.16:0/2482123368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd20806da60 0x7fd208147530 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd1f8009c40 tx=0x7fd1f8004750 comp rx=0 tx=0).stop 2023-10-16T12:11:32.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.479+0000 7fd20e784700 1 -- 172.21.15.16:0/2482123368 shutdown_connections 2023-10-16T12:11:32.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.479+0000 7fd20e784700 1 --2- 172.21.15.16:0/2482123368 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fd1f406af00 0x7fd1f406d3c0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:32.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.479+0000 7fd20e784700 1 --2- 172.21.15.16:0/2482123368 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd20815b070 0x7fd2081533c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:32.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.479+0000 7fd20e784700 1 --2- 172.21.15.16:0/2482123368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd20806da60 0x7fd208147530 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:32.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.479+0000 7fd20e784700 1 --2- 172.21.15.16:0/2482123368 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd2080cb880 0x7fd2080c5f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:32.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.479+0000 7fd20e784700 1 -- 172.21.15.16:0/2482123368 >> 172.21.15.16:0/2482123368 conn(0x7fd2080b1f40 msgr2=0x7fd208158790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:32.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.479+0000 7fd20e784700 1 -- 172.21.15.16:0/2482123368 shutdown_connections 2023-10-16T12:11:32.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.479+0000 7fd20e784700 1 -- 172.21.15.16:0/2482123368 wait complete. 2023-10-16T12:11:32.493 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T12:11:32.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.618+0000 7ff2a41e9700 1 Processor -- start 2023-10-16T12:11:32.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.618+0000 7ff2a41e9700 1 -- start start 2023-10-16T12:11:32.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.619+0000 7ff2a41e9700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7ff29c06a7e0 0x7ff29c06ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:32.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.619+0000 7ff2a41e9700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff29c15ecb0 con 0x7ff29c15b2c0 2023-10-16T12:11:32.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.619+0000 7ff2a41e9700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff29c15b6a0 con 0x7ff29c06e570 2023-10-16T12:11:32.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.619+0000 7ff2a41e9700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff29c06b190 con 0x7ff29c06a7e0 2023-10-16T12:11:32.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.619+0000 7ff2a1f85700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff29c15b2c0 0x7ff29c161340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:52088/0 (socket says 172.21.15.16:52088) 2023-10-16T12:11:32.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.619+0000 7ff2a1f85700 1 -- 172.21.15.16:0/2783142511 learned_addr learned my addr 172.21.15.16:0/2783142511 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:32.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.620+0000 7ff2a2786700 1 --2- 172.21.15.16:0/2783142511 >> v2:172.21.15.49:3300/0 conn(0x7ff29c06a7e0 0x7ff29c06ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:32.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.620+0000 7ff2a0f83700 1 -- 172.21.15.16:0/2783142511 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2711050640 0 0) 0x7ff29c15ecb0 con 0x7ff29c15b2c0 2023-10-16T12:11:32.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.620+0000 7ff2a0f83700 1 -- 172.21.15.16:0/2783142511 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2880035b0 con 0x7ff29c15b2c0 2023-10-16T12:11:32.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.620+0000 7ff2a0f83700 1 -- 172.21.15.16:0/2783142511 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1009603783 0 0) 0x7ff2880035b0 con 0x7ff29c15b2c0 2023-10-16T12:11:32.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.620+0000 7ff2a0f83700 1 -- 172.21.15.16:0/2783142511 >> v2:172.21.15.49:3300/0 conn(0x7ff29c06a7e0 msgr2=0x7ff29c06ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:32.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.620+0000 7ff2a0f83700 1 --2- 172.21.15.16:0/2783142511 >> v2:172.21.15.49:3300/0 conn(0x7ff29c06a7e0 0x7ff29c06ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:32.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.620+0000 7ff2a0f83700 1 -- 172.21.15.16:0/2783142511 >> v1:172.21.15.49:6789/0 conn(0x7ff29c06e570 legacy=0x7ff29c15e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:32.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.620+0000 7ff2a0f83700 1 -- 172.21.15.16:0/2783142511 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff29c161b20 con 0x7ff29c15b2c0 2023-10-16T12:11:32.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.620+0000 7ff2a0f83700 1 -- 172.21.15.16:0/2783142511 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7ff28c001670 con 0x7ff29c15b2c0 2023-10-16T12:11:32.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.620+0000 7ff2a0f83700 1 -- 172.21.15.16:0/2783142511 >> v1:172.21.15.16:6789/0 conn(0x7ff29c15b2c0 legacy=0x7ff29c161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:32.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.620+0000 7ff2a0f83700 1 --2- 172.21.15.16:0/2783142511 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff288003a00 0x7ff288003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:32.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.620+0000 7ff2a0f83700 1 --2- 172.21.15.16:0/2783142511 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff288004850 0x7ff288004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:32.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.620+0000 7ff2a0f83700 1 --2- 172.21.15.16:0/2783142511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff288005210 0x7ff28800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:32.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.620+0000 7ff2a0f83700 1 -- 172.21.15.16:0/2783142511 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff29c06b190 con 0x7ff288005210 2023-10-16T12:11:32.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.620+0000 7ff2a0f83700 1 -- 172.21.15.16:0/2783142511 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff28800c1b0 con 0x7ff288003a00 2023-10-16T12:11:32.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.620+0000 7ff2a0f83700 1 -- 172.21.15.16:0/2783142511 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff28800c370 con 0x7ff288004850 2023-10-16T12:11:32.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.620+0000 7ff2a2786700 1 --2- 172.21.15.16:0/2783142511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff288005210 0x7ff28800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:32.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.620+0000 7ff2a1f85700 1 --2- 172.21.15.16:0/2783142511 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff288003a00 0x7ff288003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:32.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.620+0000 7ff2a1784700 1 --2- 172.21.15.16:0/2783142511 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff288004850 0x7ff288004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:32.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.620+0000 7ff2a2786700 1 -- 172.21.15.16:0/2783142511 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff288004850 msgr2=0x7ff288004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:32.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.620+0000 7ff2a2786700 1 --2- 172.21.15.16:0/2783142511 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff288004850 0x7ff288004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:32.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.620+0000 7ff2a2786700 1 -- 172.21.15.16:0/2783142511 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff288003a00 msgr2=0x7ff288003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:32.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.620+0000 7ff2a2786700 1 --2- 172.21.15.16:0/2783142511 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff288003a00 0x7ff288003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:32.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.620+0000 7ff2a2786700 1 -- 172.21.15.16:0/2783142511 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2880035b0 con 0x7ff288005210 2023-10-16T12:11:32.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.621+0000 7ff2a2786700 1 -- 172.21.15.16:0/2783142511 reap_dead start 2023-10-16T12:11:32.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.621+0000 7ff2a2786700 1 --2- 172.21.15.16:0/2783142511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff288005210 0x7ff28800ba10 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7ff294009fc0 tx=0x7ff288005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72795c81b16cbf99 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:32.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.621+0000 7ff2a0f83700 1 -- 172.21.15.16:0/2783142511 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff294003a20 con 0x7ff288005210 2023-10-16T12:11:32.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.621+0000 7ff2a0f83700 1 -- 172.21.15.16:0/2783142511 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff29400f040 con 0x7ff288005210 2023-10-16T12:11:32.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.621+0000 7ff2a0f83700 1 -- 172.21.15.16:0/2783142511 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff294003de0 con 0x7ff288005210 2023-10-16T12:11:32.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.621+0000 7ff2a41e9700 1 -- 172.21.15.16:0/2783142511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff288005210 msgr2=0x7ff28800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:32.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.621+0000 7ff2a41e9700 1 --2- 172.21.15.16:0/2783142511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff288005210 0x7ff28800ba10 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7ff294009fc0 tx=0x7ff288005880 comp rx=0 tx=0).stop 2023-10-16T12:11:32.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.621+0000 7ff2a41e9700 1 -- 172.21.15.16:0/2783142511 shutdown_connections 2023-10-16T12:11:32.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.621+0000 7ff2a41e9700 1 --2- 172.21.15.16:0/2783142511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff288005210 0x7ff28800ba10 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:32.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.621+0000 7ff2a41e9700 1 -- 172.21.15.16:0/2783142511 >> 172.21.15.16:0/2783142511 conn(0x7ff29c0b1f40 msgr2=0x7ff29c0b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:32.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.621+0000 7ff2a41e9700 1 -- 172.21.15.16:0/2783142511 shutdown_connections 2023-10-16T12:11:32.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.622+0000 7ff2a41e9700 1 -- 172.21.15.16:0/2783142511 wait complete. 2023-10-16T12:11:32.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.623+0000 7ff2a41e9700 1 Processor -- start 2023-10-16T12:11:32.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.623+0000 7ff2a41e9700 1 -- start start 2023-10-16T12:11:32.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.623+0000 7ff2a41e9700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff288005210 0x7ff29c0ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:32.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.623+0000 7ff2a41e9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff29c1574b0 0x7ff29c0adf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:32.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.623+0000 7ff2a41e9700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff29c159b90 0x7ff29c15a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:32.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.623+0000 7ff2a41e9700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff29c160a90 con 0x7ff29c1574b0 2023-10-16T12:11:32.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.623+0000 7ff2a41e9700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff29c15a560 con 0x7ff288005210 2023-10-16T12:11:32.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.623+0000 7ff2a41e9700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff29c0ae490 con 0x7ff29c159b90 2023-10-16T12:11:32.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.624+0000 7ff2a1784700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff29c1574b0 0x7ff29c0adf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:32.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.624+0000 7ff2a1784700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff29c1574b0 0x7ff29c0adf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34598/0 (socket says 172.21.15.16:34598) 2023-10-16T12:11:32.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.624+0000 7ff2a1784700 1 -- 172.21.15.16:0/3016718416 learned_addr learned my addr 172.21.15.16:0/3016718416 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:32.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.624+0000 7ff2a1f85700 1 --2- 172.21.15.16:0/3016718416 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff288005210 0x7ff29c0ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:32.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.624+0000 7ff2a2786700 1 --2- 172.21.15.16:0/3016718416 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff29c159b90 0x7ff29c15a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:32.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.624+0000 7ff2a1784700 1 -- 172.21.15.16:0/3016718416 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff29c159b90 msgr2=0x7ff29c15a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:32.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.624+0000 7ff2a1784700 1 --2- 172.21.15.16:0/3016718416 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff29c159b90 0x7ff29c15a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:32.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.624+0000 7ff2a1784700 1 -- 172.21.15.16:0/3016718416 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff288005210 msgr2=0x7ff29c0ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:32.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.624+0000 7ff2a1784700 1 --2- 172.21.15.16:0/3016718416 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff288005210 0x7ff29c0ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:32.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.624+0000 7ff2a1784700 1 -- 172.21.15.16:0/3016718416 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff29c0aceb0 con 0x7ff29c1574b0 2023-10-16T12:11:32.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.624+0000 7ff2a1784700 1 --2- 172.21.15.16:0/3016718416 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff29c1574b0 0x7ff29c0adf50 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7ff29800f820 tx=0x7ff29800fb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:32.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.624+0000 7ff2927fc700 1 -- 172.21.15.16:0/3016718416 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2980052e0 con 0x7ff29c1574b0 2023-10-16T12:11:32.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.624+0000 7ff2927fc700 1 -- 172.21.15.16:0/3016718416 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff298005440 con 0x7ff29c1574b0 2023-10-16T12:11:32.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.624+0000 7ff2a41e9700 1 -- 172.21.15.16:0/3016718416 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff29c0ad190 con 0x7ff29c1574b0 2023-10-16T12:11:32.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.624+0000 7ff2927fc700 1 -- 172.21.15.16:0/3016718416 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2980038a0 con 0x7ff29c1574b0 2023-10-16T12:11:32.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.624+0000 7ff2a41e9700 1 -- 172.21.15.16:0/3016718416 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff29c0ad750 con 0x7ff29c1574b0 2023-10-16T12:11:32.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.625+0000 7ff2a41e9700 1 -- 172.21.15.16:0/3016718416 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff29c04ebb0 con 0x7ff29c1574b0 2023-10-16T12:11:32.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.626+0000 7ff2927fc700 1 -- 172.21.15.16:0/3016718416 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7ff298003a00 con 0x7ff29c1574b0 2023-10-16T12:11:32.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.626+0000 7ff2927fc700 1 --2- 172.21.15.16:0/3016718416 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7ff28006af00 0x7ff28006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:32.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.626+0000 7ff2927fc700 1 -- 172.21.15.16:0/3016718416 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7ff298089520 con 0x7ff29c1574b0 2023-10-16T12:11:32.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.626+0000 7ff2a1f85700 1 --2- 172.21.15.16:0/3016718416 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7ff28006af00 0x7ff28006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:32.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.627+0000 7ff2a1f85700 1 --2- 172.21.15.16:0/3016718416 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7ff28006af00 0x7ff28006d3c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ff29c161990 tx=0x7ff28c000ed0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:32.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.631+0000 7ff2927fc700 1 -- 172.21.15.16:0/3016718416 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff2980542a0 con 0x7ff29c1574b0 2023-10-16T12:11:32.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.832+0000 7ff2a41e9700 1 -- 172.21.15.16:0/3016718416 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff29c0b6370 con 0x7ff29c1574b0 2023-10-16T12:11:32.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.832+0000 7ff2927fc700 1 -- 172.21.15.16:0/3016718416 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff298003cb0 con 0x7ff29c1574b0 2023-10-16T12:11:32.834 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-10-16T12:11:32.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.835+0000 7ff2a41e9700 1 -- 172.21.15.16:0/3016718416 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7ff28006af00 msgr2=0x7ff28006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:32.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.835+0000 7ff2a41e9700 1 --2- 172.21.15.16:0/3016718416 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7ff28006af00 0x7ff28006d3c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ff29c161990 tx=0x7ff28c000ed0 comp rx=0 tx=0).stop 2023-10-16T12:11:32.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.835+0000 7ff2a41e9700 1 -- 172.21.15.16:0/3016718416 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff29c1574b0 msgr2=0x7ff29c0adf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:32.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.835+0000 7ff2a41e9700 1 --2- 172.21.15.16:0/3016718416 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff29c1574b0 0x7ff29c0adf50 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7ff29800f820 tx=0x7ff29800fb30 comp rx=0 tx=0).stop 2023-10-16T12:11:32.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.835+0000 7ff2a41e9700 1 -- 172.21.15.16:0/3016718416 shutdown_connections 2023-10-16T12:11:32.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.835+0000 7ff2a41e9700 1 --2- 172.21.15.16:0/3016718416 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7ff28006af00 0x7ff28006d3c0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:32.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.835+0000 7ff2a41e9700 1 --2- 172.21.15.16:0/3016718416 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff29c159b90 0x7ff29c15a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:32.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.835+0000 7ff2a41e9700 1 --2- 172.21.15.16:0/3016718416 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff288005210 0x7ff29c0ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:32.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.835+0000 7ff2a41e9700 1 --2- 172.21.15.16:0/3016718416 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff29c1574b0 0x7ff29c0adf50 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:32.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.835+0000 7ff2a41e9700 1 -- 172.21.15.16:0/3016718416 >> 172.21.15.16:0/3016718416 conn(0x7ff29c0b1f40 msgr2=0x7ff29c160fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:32.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.835+0000 7ff2a41e9700 1 -- 172.21.15.16:0/3016718416 shutdown_connections 2023-10-16T12:11:32.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.835+0000 7ff2a41e9700 1 -- 172.21.15.16:0/3016718416 wait complete. 2023-10-16T12:11:32.849 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T12:11:32.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.974+0000 7f22b1b0b700 1 Processor -- start 2023-10-16T12:11:32.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.974+0000 7f22b1b0b700 1 -- start start 2023-10-16T12:11:32.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.975+0000 7f22b1b0b700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f22ac06e570 0x7f22ac15e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:32.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.975+0000 7f22b1b0b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f22ac06a7e0 0x7f22ac06ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:32.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.975+0000 7f22b1b0b700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22ac15b6a0 con 0x7f22ac15b2c0 2023-10-16T12:11:32.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.975+0000 7f22b1b0b700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22ac06b1b0 con 0x7f22ac06e570 2023-10-16T12:11:32.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.975+0000 7f22b1b0b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22ac161b20 con 0x7f22ac06a7e0 2023-10-16T12:11:32.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.975+0000 7f22ab7fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f22ac15b2c0 0x7f22ac161340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:52108/0 (socket says 172.21.15.16:52108) 2023-10-16T12:11:32.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.975+0000 7f22ab7fe700 1 -- 172.21.15.16:0/3662076732 learned_addr learned my addr 172.21.15.16:0/3662076732 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:32.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.975+0000 7f22aaffd700 1 --2- 172.21.15.16:0/3662076732 >> v2:172.21.15.49:3300/0 conn(0x7f22ac06e570 0x7f22ac15e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:32.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.975+0000 7f22abfff700 1 --2- 172.21.15.16:0/3662076732 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f22ac06a7e0 0x7f22ac06ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:32.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.975+0000 7f22aa7fc700 1 -- 172.21.15.16:0/3662076732 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2015625414 0 0) 0x7f22ac15b6a0 con 0x7f22ac15b2c0 2023-10-16T12:11:32.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.975+0000 7f22aa7fc700 1 -- 172.21.15.16:0/3662076732 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22940035b0 con 0x7f22ac15b2c0 2023-10-16T12:11:32.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.976+0000 7f22aa7fc700 1 -- 172.21.15.16:0/3662076732 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2170073713 0 0) 0x7f22940035b0 con 0x7f22ac15b2c0 2023-10-16T12:11:32.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.976+0000 7f22aa7fc700 1 -- 172.21.15.16:0/3662076732 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f22ac06a7e0 msgr2=0x7f22ac06ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:32.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.976+0000 7f22aa7fc700 1 --2- 172.21.15.16:0/3662076732 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f22ac06a7e0 0x7f22ac06ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:32.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.976+0000 7f22aa7fc700 1 -- 172.21.15.16:0/3662076732 >> v2:172.21.15.49:3300/0 conn(0x7f22ac06e570 msgr2=0x7f22ac15e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:32.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.976+0000 7f22aa7fc700 1 --2- 172.21.15.16:0/3662076732 >> v2:172.21.15.49:3300/0 conn(0x7f22ac06e570 0x7f22ac15e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:32.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.976+0000 7f22aa7fc700 1 -- 172.21.15.16:0/3662076732 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22ac161e00 con 0x7f22ac15b2c0 2023-10-16T12:11:32.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.976+0000 7f22aa7fc700 1 -- 172.21.15.16:0/3662076732 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f22a0001670 con 0x7f22ac15b2c0 2023-10-16T12:11:32.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.976+0000 7f22abfff700 1 --2- 172.21.15.16:0/3662076732 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f22ac06a7e0 0x7f22ac06ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:11:32.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.976+0000 7f22aa7fc700 1 -- 172.21.15.16:0/3662076732 >> v1:172.21.15.16:6789/0 conn(0x7f22ac15b2c0 legacy=0x7f22ac161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:32.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.976+0000 7f22aa7fc700 1 --2- 172.21.15.16:0/3662076732 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f22ac06a7e0 0x7f2294003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:32.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.976+0000 7f22aa7fc700 1 --2- 172.21.15.16:0/3662076732 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2294003f40 0x7f2294004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:32.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.976+0000 7f22aa7fc700 1 --2- 172.21.15.16:0/3662076732 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2294004d90 0x7f2294005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:32.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.976+0000 7f22aa7fc700 1 -- 172.21.15.16:0/3662076732 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22ac06b1b0 con 0x7f2294003f40 2023-10-16T12:11:32.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.976+0000 7f22aa7fc700 1 -- 172.21.15.16:0/3662076732 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22ac161b20 con 0x7f2294004d90 2023-10-16T12:11:32.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.976+0000 7f22aa7fc700 1 -- 172.21.15.16:0/3662076732 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f229400ba10 con 0x7f22ac06a7e0 2023-10-16T12:11:32.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.976+0000 7f22aaffd700 1 --2- 172.21.15.16:0/3662076732 >> v2:172.21.15.49:3300/0 conn(0x7f22ac06e570 0x7f22ac15e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:11:32.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.976+0000 7f22aaffd700 1 --2- 172.21.15.16:0/3662076732 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2294003f40 0x7f2294004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:32.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.976+0000 7f22ab7fe700 1 --2- 172.21.15.16:0/3662076732 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f22ac06a7e0 0x7f2294003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:32.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.976+0000 7f22abfff700 1 --2- 172.21.15.16:0/3662076732 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2294004d90 0x7f2294005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:32.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.976+0000 7f22aaffd700 1 -- 172.21.15.16:0/3662076732 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f22ac06a7e0 msgr2=0x7f2294003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:32.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.976+0000 7f22aaffd700 1 --2- 172.21.15.16:0/3662076732 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f22ac06a7e0 0x7f2294003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:32.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.976+0000 7f22aaffd700 1 -- 172.21.15.16:0/3662076732 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2294004d90 msgr2=0x7f2294005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:32.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.976+0000 7f22aaffd700 1 --2- 172.21.15.16:0/3662076732 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2294004d90 0x7f2294005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:32.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.976+0000 7f22aaffd700 1 -- 172.21.15.16:0/3662076732 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22a4009a70 con 0x7f2294003f40 2023-10-16T12:11:32.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.976+0000 7f22aaffd700 1 --2- 172.21.15.16:0/3662076732 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2294003f40 0x7f2294004850 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f229c009d10 tx=0x7f2294005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c5107fb6682c25e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:32.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.976+0000 7f22aa7fc700 1 -- 172.21.15.16:0/3662076732 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f229c009e60 con 0x7f2294003f40 2023-10-16T12:11:32.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.976+0000 7f22aa7fc700 1 -- 172.21.15.16:0/3662076732 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f229c00e9a0 con 0x7f2294003f40 2023-10-16T12:11:32.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.976+0000 7f22aa7fc700 1 -- 172.21.15.16:0/3662076732 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f229c00ed10 con 0x7f2294003f40 2023-10-16T12:11:32.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.977+0000 7f22b1b0b700 1 -- 172.21.15.16:0/3662076732 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2294003f40 msgr2=0x7f2294004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:32.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.977+0000 7f22b1b0b700 1 --2- 172.21.15.16:0/3662076732 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2294003f40 0x7f2294004850 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f229c009d10 tx=0x7f2294005880 comp rx=0 tx=0).stop 2023-10-16T12:11:32.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.977+0000 7f22abfff700 1 -- 172.21.15.16:0/3662076732 reap_dead start 2023-10-16T12:11:32.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.977+0000 7f22b1b0b700 1 -- 172.21.15.16:0/3662076732 shutdown_connections 2023-10-16T12:11:32.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.977+0000 7f22b1b0b700 1 -- 172.21.15.16:0/3662076732 >> 172.21.15.16:0/3662076732 conn(0x7f22ac0b1f40 msgr2=0x7f22ac0b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:32.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.977+0000 7f22b1b0b700 1 -- 172.21.15.16:0/3662076732 shutdown_connections 2023-10-16T12:11:32.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.977+0000 7f22b1b0b700 1 -- 172.21.15.16:0/3662076732 wait complete. 2023-10-16T12:11:32.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.979+0000 7f22b1b0b700 1 Processor -- start 2023-10-16T12:11:32.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.979+0000 7f22b1b0b700 1 -- start start 2023-10-16T12:11:32.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.980+0000 7f22b1b0b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2294003f40 0x7f22ac1517b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:32.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.980+0000 7f22b1b0b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f22ac152f90 0x7f22ac0b3a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:32.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.980+0000 7f22b1b0b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f22ac1f4130 0x7f22ac0b3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:32.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.980+0000 7f22b1b0b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22ac0b3670 con 0x7f2294003f40 2023-10-16T12:11:32.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.980+0000 7f22b1b0b700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22ac152b50 con 0x7f22ac152f90 2023-10-16T12:11:32.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.980+0000 7f22b1b0b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22ac152870 con 0x7f22ac1f4130 2023-10-16T12:11:32.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.980+0000 7f22ab7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2294003f40 0x7f22ac1517b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:32.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.980+0000 7f22ab7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2294003f40 0x7f22ac1517b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34614/0 (socket says 172.21.15.16:34614) 2023-10-16T12:11:32.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.980+0000 7f22ab7fe700 1 -- 172.21.15.16:0/3595179274 learned_addr learned my addr 172.21.15.16:0/3595179274 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:32.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.980+0000 7f22abfff700 1 --2- 172.21.15.16:0/3595179274 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f22ac1f4130 0x7f22ac0b3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:32.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.980+0000 7f22aaffd700 1 --2- 172.21.15.16:0/3595179274 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f22ac152f90 0x7f22ac0b3a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:32.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.980+0000 7f22ab7fe700 1 -- 172.21.15.16:0/3595179274 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f22ac1f4130 msgr2=0x7f22ac0b3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:32.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.980+0000 7f22ab7fe700 1 --2- 172.21.15.16:0/3595179274 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f22ac1f4130 0x7f22ac0b3fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:32.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.980+0000 7f22ab7fe700 1 -- 172.21.15.16:0/3595179274 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f22ac152f90 msgr2=0x7f22ac0b3a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:32.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.980+0000 7f22ab7fe700 1 --2- 172.21.15.16:0/3595179274 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f22ac152f90 0x7f22ac0b3a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:32.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.980+0000 7f22ab7fe700 1 -- 172.21.15.16:0/3595179274 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22ac0b2fb0 con 0x7f2294003f40 2023-10-16T12:11:32.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.980+0000 7f22ab7fe700 1 --2- 172.21.15.16:0/3595179274 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2294003f40 0x7f22ac1517b0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f22a0007310 tx=0x7f22a0007620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:32.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.981+0000 7f2293fff700 1 -- 172.21.15.16:0/3595179274 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22a000f6c0 con 0x7f2294003f40 2023-10-16T12:11:32.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.981+0000 7f2293fff700 1 -- 172.21.15.16:0/3595179274 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22a000f860 con 0x7f2294003f40 2023-10-16T12:11:32.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.981+0000 7f22b1b0b700 1 -- 172.21.15.16:0/3595179274 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22ac0b3290 con 0x7f2294003f40 2023-10-16T12:11:32.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.981+0000 7f2293fff700 1 -- 172.21.15.16:0/3595179274 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22a000fc10 con 0x7f2294003f40 2023-10-16T12:11:32.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.981+0000 7f22b1b0b700 1 -- 172.21.15.16:0/3595179274 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22ac0b5cb0 con 0x7f2294003f40 2023-10-16T12:11:32.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.982+0000 7f22b1b0b700 1 -- 172.21.15.16:0/3595179274 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22ac0b37b0 con 0x7f2294003f40 2023-10-16T12:11:32.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.982+0000 7f2293fff700 1 -- 172.21.15.16:0/3595179274 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f22a000fd70 con 0x7f2294003f40 2023-10-16T12:11:32.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.983+0000 7f2293fff700 1 --2- 172.21.15.16:0/3595179274 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f229806af00 0x7f229806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:32.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.983+0000 7f22aaffd700 1 --2- 172.21.15.16:0/3595179274 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f229806af00 0x7f229806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:32.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.983+0000 7f2293fff700 1 -- 172.21.15.16:0/3595179274 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f22a00899e0 con 0x7f2294003f40 2023-10-16T12:11:32.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.983+0000 7f22aaffd700 1 --2- 172.21.15.16:0/3595179274 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f229806af00 0x7f229806d3c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f229c0090b0 tx=0x7f229c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:32.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:32.987+0000 7f2293fff700 1 -- 172.21.15.16:0/3595179274 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f22ac0b37b0 con 0x7f2294003f40 2023-10-16T12:11:33.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.191+0000 7f22b1b0b700 1 -- 172.21.15.16:0/3595179274 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f22ac0b37b0 con 0x7f2294003f40 2023-10-16T12:11:33.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.191+0000 7f2293fff700 1 -- 172.21.15.16:0/3595179274 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7f22ac0b37b0 con 0x7f2294003f40 2023-10-16T12:11:33.194 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-10-16T12:11:33.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.193+0000 7f22b1b0b700 1 -- 172.21.15.16:0/3595179274 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f229806af00 msgr2=0x7f229806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:33.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.193+0000 7f22b1b0b700 1 --2- 172.21.15.16:0/3595179274 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f229806af00 0x7f229806d3c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f229c0090b0 tx=0x7f229c008030 comp rx=0 tx=0).stop 2023-10-16T12:11:33.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.193+0000 7f22b1b0b700 1 -- 172.21.15.16:0/3595179274 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2294003f40 msgr2=0x7f22ac1517b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:33.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.193+0000 7f22b1b0b700 1 --2- 172.21.15.16:0/3595179274 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2294003f40 0x7f22ac1517b0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f22a0007310 tx=0x7f22a0007620 comp rx=0 tx=0).stop 2023-10-16T12:11:33.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.193+0000 7f22b1b0b700 1 -- 172.21.15.16:0/3595179274 shutdown_connections 2023-10-16T12:11:33.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.193+0000 7f22b1b0b700 1 --2- 172.21.15.16:0/3595179274 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f229806af00 0x7f229806d3c0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:33.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.193+0000 7f22b1b0b700 1 --2- 172.21.15.16:0/3595179274 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f22ac1f4130 0x7f22ac0b3fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:33.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.193+0000 7f22b1b0b700 1 --2- 172.21.15.16:0/3595179274 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2294003f40 0x7f22ac1517b0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:33.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.193+0000 7f22b1b0b700 1 --2- 172.21.15.16:0/3595179274 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f22ac152f90 0x7f22ac0b3a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:33.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.193+0000 7f22b1b0b700 1 -- 172.21.15.16:0/3595179274 >> 172.21.15.16:0/3595179274 conn(0x7f22ac0b1f40 msgr2=0x7f22ac15b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:33.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.194+0000 7f22b1b0b700 1 -- 172.21.15.16:0/3595179274 shutdown_connections 2023-10-16T12:11:33.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.194+0000 7f22b1b0b700 1 -- 172.21.15.16:0/3595179274 wait complete. 2023-10-16T12:11:33.207 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-16T12:11:33.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.336+0000 7fc441d28700 1 Processor -- start 2023-10-16T12:11:33.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.336+0000 7fc441d28700 1 -- start start 2023-10-16T12:11:33.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.337+0000 7fc441d28700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc43c15a640 0x7fc43c0cb710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:33.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.337+0000 7fc441d28700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fc43c15e140 0x7fc43c0cbc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:33.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.337+0000 7fc441d28700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc43c15aa20 con 0x7fc43c0c6770 2023-10-16T12:11:33.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.337+0000 7fc441d28700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc43c06eb40 con 0x7fc43c15a640 2023-10-16T12:11:33.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.337+0000 7fc441d28700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc43c06ee20 con 0x7fc43c15e140 2023-10-16T12:11:33.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.337+0000 7fc43b7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc43c15a640 0x7fc43c0cb710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:33.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.337+0000 7fc43b7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc43c15a640 0x7fc43c0cb710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34622/0 (socket says 172.21.15.16:34622) 2023-10-16T12:11:33.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.337+0000 7fc43b7fe700 1 -- 172.21.15.16:0/1608964408 learned_addr learned my addr 172.21.15.16:0/1608964408 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:33.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.338+0000 7fc43bfff700 1 --2- 172.21.15.16:0/1608964408 >> v2:172.21.15.49:3300/0 conn(0x7fc43c15e140 0x7fc43c0cbc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:33.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.338+0000 7fc43b7fe700 1 -- 172.21.15.16:0/1608964408 >> v2:172.21.15.49:3300/0 conn(0x7fc43c15e140 msgr2=0x7fc43c0cbc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:33.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.338+0000 7fc43b7fe700 1 --2- 172.21.15.16:0/1608964408 >> v2:172.21.15.49:3300/0 conn(0x7fc43c15e140 0x7fc43c0cbc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:33.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.338+0000 7fc43b7fe700 1 -- 172.21.15.16:0/1608964408 >> v1:172.21.15.49:6789/0 conn(0x7fc43c0c6770 legacy=0x7fc43c0ca1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:33.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.338+0000 7fc43b7fe700 1 -- 172.21.15.16:0/1608964408 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc43c06f100 con 0x7fc43c15a640 2023-10-16T12:11:33.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.338+0000 7fc43bfff700 1 --2- 172.21.15.16:0/1608964408 >> v2:172.21.15.49:3300/0 conn(0x7fc43c15e140 0x7fc43c0cbc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:11:33.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.338+0000 7fc43b7fe700 1 --2- 172.21.15.16:0/1608964408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc43c15a640 0x7fc43c0cb710 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fc43c071180 tx=0x7fc42c00be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45350115a0e19e4d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:33.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.338+0000 7fc43a7fc700 1 -- 172.21.15.16:0/1608964408 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc42c00dcf0 con 0x7fc43c15a640 2023-10-16T12:11:33.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.338+0000 7fc43a7fc700 1 -- 172.21.15.16:0/1608964408 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc42c00de50 con 0x7fc43c15a640 2023-10-16T12:11:33.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.338+0000 7fc43a7fc700 1 -- 172.21.15.16:0/1608964408 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc42c004250 con 0x7fc43c15a640 2023-10-16T12:11:33.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.339+0000 7fc441d28700 1 -- 172.21.15.16:0/1608964408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc43c15a640 msgr2=0x7fc43c0cb710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:33.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.339+0000 7fc441d28700 1 --2- 172.21.15.16:0/1608964408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc43c15a640 0x7fc43c0cb710 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fc43c071180 tx=0x7fc42c00be70 comp rx=0 tx=0).stop 2023-10-16T12:11:33.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.339+0000 7fc441d28700 1 -- 172.21.15.16:0/1608964408 shutdown_connections 2023-10-16T12:11:33.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.339+0000 7fc441d28700 1 --2- 172.21.15.16:0/1608964408 >> v2:172.21.15.49:3300/0 conn(0x7fc43c15e140 0x7fc43c0cbc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:33.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.339+0000 7fc441d28700 1 --2- 172.21.15.16:0/1608964408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc43c15a640 0x7fc43c0cb710 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:33.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.339+0000 7fc441d28700 1 -- 172.21.15.16:0/1608964408 >> 172.21.15.16:0/1608964408 conn(0x7fc43c0cc5e0 msgr2=0x7fc43c0cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:33.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.339+0000 7fc441d28700 1 -- 172.21.15.16:0/1608964408 shutdown_connections 2023-10-16T12:11:33.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.339+0000 7fc441d28700 1 -- 172.21.15.16:0/1608964408 wait complete. 2023-10-16T12:11:33.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.340+0000 7fc441d28700 1 Processor -- start 2023-10-16T12:11:33.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.341+0000 7fc441d28700 1 -- start start 2023-10-16T12:11:33.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.341+0000 7fc441d28700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc43c0c6770 0x7fc43c0c7c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:33.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.341+0000 7fc441d28700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc43c15a640 0x7fc43c0b3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:33.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.341+0000 7fc441d28700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc43c15e140 0x7fc43c0b4380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:33.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.341+0000 7fc441d28700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc43c15c220 con 0x7fc43c0c6770 2023-10-16T12:11:33.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.341+0000 7fc441d28700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc43c15c500 con 0x7fc43c15e140 2023-10-16T12:11:33.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.341+0000 7fc441d28700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc43c15c7e0 con 0x7fc43c15a640 2023-10-16T12:11:33.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.342+0000 7fc43b7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc43c0c6770 0x7fc43c0c7c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:33.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.342+0000 7fc43b7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc43c0c6770 0x7fc43c0c7c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34628/0 (socket says 172.21.15.16:34628) 2023-10-16T12:11:33.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.342+0000 7fc43b7fe700 1 -- 172.21.15.16:0/3378208818 learned_addr learned my addr 172.21.15.16:0/3378208818 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:33.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.342+0000 7fc43bfff700 1 --2- 172.21.15.16:0/3378208818 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc43c15e140 0x7fc43c0b4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:33.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.342+0000 7fc43affd700 1 --2- 172.21.15.16:0/3378208818 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc43c15a640 0x7fc43c0b3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:33.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.342+0000 7fc43b7fe700 1 -- 172.21.15.16:0/3378208818 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc43c15a640 msgr2=0x7fc43c0b3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:33.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.342+0000 7fc43b7fe700 1 --2- 172.21.15.16:0/3378208818 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc43c15a640 0x7fc43c0b3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:33.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.342+0000 7fc43b7fe700 1 -- 172.21.15.16:0/3378208818 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc43c15e140 msgr2=0x7fc43c0b4380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:33.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.342+0000 7fc43b7fe700 1 --2- 172.21.15.16:0/3378208818 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc43c15e140 0x7fc43c0b4380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:33.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.342+0000 7fc43b7fe700 1 -- 172.21.15.16:0/3378208818 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc42c00b7f0 con 0x7fc43c0c6770 2023-10-16T12:11:33.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.342+0000 7fc43b7fe700 1 --2- 172.21.15.16:0/3378208818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc43c0c6770 0x7fc43c0c7c80 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fc42c000c00 tx=0x7fc42c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:33.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.342+0000 7fc433fff700 1 -- 172.21.15.16:0/3378208818 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc42c020070 con 0x7fc43c0c6770 2023-10-16T12:11:33.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.342+0000 7fc433fff700 1 -- 172.21.15.16:0/3378208818 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc42c004240 con 0x7fc43c0c6770 2023-10-16T12:11:33.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.342+0000 7fc441d28700 1 -- 172.21.15.16:0/3378208818 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc43c15cac0 con 0x7fc43c0c6770 2023-10-16T12:11:33.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.342+0000 7fc433fff700 1 -- 172.21.15.16:0/3378208818 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc42c010690 con 0x7fc43c0c6770 2023-10-16T12:11:33.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.342+0000 7fc441d28700 1 -- 172.21.15.16:0/3378208818 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc43c0c81c0 con 0x7fc43c0c6770 2023-10-16T12:11:33.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.343+0000 7fc441d28700 1 -- 172.21.15.16:0/3378208818 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc43c066820 con 0x7fc43c0c6770 2023-10-16T12:11:33.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.344+0000 7fc433fff700 1 -- 172.21.15.16:0/3378208818 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fc42c0193f0 con 0x7fc43c0c6770 2023-10-16T12:11:33.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.344+0000 7fc433fff700 1 --2- 172.21.15.16:0/3378208818 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fc41c06af00 0x7fc41c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:33.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.344+0000 7fc433fff700 1 -- 172.21.15.16:0/3378208818 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fc42c08ce90 con 0x7fc43c0c6770 2023-10-16T12:11:33.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.344+0000 7fc43affd700 1 --2- 172.21.15.16:0/3378208818 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fc41c06af00 0x7fc41c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:33.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.345+0000 7fc43affd700 1 --2- 172.21.15.16:0/3378208818 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fc41c06af00 0x7fc41c06d3c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fc43c06dc50 tx=0x7fc424009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:33.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.349+0000 7fc433fff700 1 -- 172.21.15.16:0/3378208818 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc42c057c10 con 0x7fc43c0c6770 2023-10-16T12:11:33.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.552+0000 7fc441d28700 1 -- 172.21.15.16:0/3378208818 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fc43c04ebb0 con 0x7fc43c0c6770 2023-10-16T12:11:33.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.552+0000 7fc433fff700 1 -- 172.21.15.16:0/3378208818 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7fc42c00dcb0 con 0x7fc43c0c6770 2023-10-16T12:11:33.555 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-10-16T12:11:33.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.554+0000 7fc441d28700 1 -- 172.21.15.16:0/3378208818 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fc41c06af00 msgr2=0x7fc41c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:33.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.554+0000 7fc441d28700 1 --2- 172.21.15.16:0/3378208818 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fc41c06af00 0x7fc41c06d3c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fc43c06dc50 tx=0x7fc424009040 comp rx=0 tx=0).stop 2023-10-16T12:11:33.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.554+0000 7fc441d28700 1 -- 172.21.15.16:0/3378208818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc43c0c6770 msgr2=0x7fc43c0c7c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:33.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.554+0000 7fc441d28700 1 --2- 172.21.15.16:0/3378208818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc43c0c6770 0x7fc43c0c7c80 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fc42c000c00 tx=0x7fc42c004750 comp rx=0 tx=0).stop 2023-10-16T12:11:33.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.554+0000 7fc441d28700 1 -- 172.21.15.16:0/3378208818 shutdown_connections 2023-10-16T12:11:33.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.554+0000 7fc441d28700 1 --2- 172.21.15.16:0/3378208818 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fc41c06af00 0x7fc41c06d3c0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:33.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.554+0000 7fc441d28700 1 --2- 172.21.15.16:0/3378208818 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc43c15e140 0x7fc43c0b4380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:33.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.554+0000 7fc441d28700 1 --2- 172.21.15.16:0/3378208818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc43c0c6770 0x7fc43c0c7c80 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:33.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.554+0000 7fc441d28700 1 --2- 172.21.15.16:0/3378208818 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc43c15a640 0x7fc43c0b3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:33.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.554+0000 7fc441d28700 1 -- 172.21.15.16:0/3378208818 >> 172.21.15.16:0/3378208818 conn(0x7fc43c0cc5e0 msgr2=0x7fc43c0c56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:33.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.555+0000 7fc441d28700 1 -- 172.21.15.16:0/3378208818 shutdown_connections 2023-10-16T12:11:33.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.555+0000 7fc441d28700 1 -- 172.21.15.16:0/3378208818 wait complete. 2023-10-16T12:11:33.568 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates' 2023-10-16T12:11:33.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.692+0000 7ff87ae7c700 1 Processor -- start 2023-10-16T12:11:33.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.692+0000 7ff87ae7c700 1 -- start start 2023-10-16T12:11:33.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.693+0000 7ff87ae7c700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7ff8740cc8a0 0x7ff87415e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:33.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.693+0000 7ff87ae7c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff87406a7e0 0x7ff87406ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:33.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.693+0000 7ff87ae7c700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff874161c30 con 0x7ff87415b280 2023-10-16T12:11:33.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.693+0000 7ff87ae7c700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff87406b190 con 0x7ff8740cc8a0 2023-10-16T12:11:33.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.693+0000 7ff87ae7c700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff874161f10 con 0x7ff87406a7e0 2023-10-16T12:11:33.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.693+0000 7ff878c18700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff87415b280 0x7ff8741613e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:52142/0 (socket says 172.21.15.16:52142) 2023-10-16T12:11:33.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.693+0000 7ff878c18700 1 -- 172.21.15.16:0/3318079736 learned_addr learned my addr 172.21.15.16:0/3318079736 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:33.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.693+0000 7ff873fff700 1 --2- 172.21.15.16:0/3318079736 >> v2:172.21.15.49:3300/0 conn(0x7ff8740cc8a0 0x7ff87415e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:33.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.693+0000 7ff879419700 1 --2- 172.21.15.16:0/3318079736 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff87406a7e0 0x7ff87406ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:33.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.693+0000 7ff8737fe700 1 -- 172.21.15.16:0/3318079736 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3856069500 0 0) 0x7ff874161c30 con 0x7ff87415b280 2023-10-16T12:11:33.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.693+0000 7ff8737fe700 1 -- 172.21.15.16:0/3318079736 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8640035b0 con 0x7ff87415b280 2023-10-16T12:11:33.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.694+0000 7ff8737fe700 1 -- 172.21.15.16:0/3318079736 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3525083089 0 0) 0x7ff8640035b0 con 0x7ff87415b280 2023-10-16T12:11:33.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.694+0000 7ff8737fe700 1 -- 172.21.15.16:0/3318079736 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff87406a7e0 msgr2=0x7ff87406ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:33.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.694+0000 7ff8737fe700 1 --2- 172.21.15.16:0/3318079736 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff87406a7e0 0x7ff87406ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:33.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.694+0000 7ff8737fe700 1 -- 172.21.15.16:0/3318079736 >> v2:172.21.15.49:3300/0 conn(0x7ff8740cc8a0 msgr2=0x7ff87415e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:33.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.694+0000 7ff8737fe700 1 --2- 172.21.15.16:0/3318079736 >> v2:172.21.15.49:3300/0 conn(0x7ff8740cc8a0 0x7ff87415e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:33.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.694+0000 7ff8737fe700 1 -- 172.21.15.16:0/3318079736 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8741621f0 con 0x7ff87415b280 2023-10-16T12:11:33.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.694+0000 7ff8737fe700 1 -- 172.21.15.16:0/3318079736 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7ff868001670 con 0x7ff87415b280 2023-10-16T12:11:33.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.694+0000 7ff8737fe700 1 -- 172.21.15.16:0/3318079736 >> v1:172.21.15.16:6789/0 conn(0x7ff87415b280 legacy=0x7ff8741613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:33.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.694+0000 7ff8737fe700 1 --2- 172.21.15.16:0/3318079736 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff864003a00 0x7ff864003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:33.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.694+0000 7ff8737fe700 1 --2- 172.21.15.16:0/3318079736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff864004850 0x7ff864004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:33.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.694+0000 7ff8737fe700 1 --2- 172.21.15.16:0/3318079736 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff87406a7e0 0x7ff864005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:33.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.694+0000 7ff8737fe700 1 -- 172.21.15.16:0/3318079736 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff87406b190 con 0x7ff864004850 2023-10-16T12:11:33.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.694+0000 7ff8737fe700 1 -- 172.21.15.16:0/3318079736 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff874161f10 con 0x7ff864003a00 2023-10-16T12:11:33.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.694+0000 7ff8737fe700 1 -- 172.21.15.16:0/3318079736 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff86400ba10 con 0x7ff87406a7e0 2023-10-16T12:11:33.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.694+0000 7ff873fff700 1 --2- 172.21.15.16:0/3318079736 >> v2:172.21.15.49:3300/0 conn(0x7ff8740cc8a0 0x7ff87415e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:11:33.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.694+0000 7ff873fff700 1 --2- 172.21.15.16:0/3318079736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff864004850 0x7ff864004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:33.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.694+0000 7ff879419700 1 --2- 172.21.15.16:0/3318079736 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff87406a7e0 0x7ff864005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:33.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.694+0000 7ff878c18700 1 --2- 172.21.15.16:0/3318079736 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff864003a00 0x7ff864003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:33.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.694+0000 7ff873fff700 1 -- 172.21.15.16:0/3318079736 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff87406a7e0 msgr2=0x7ff864005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:33.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.694+0000 7ff873fff700 1 --2- 172.21.15.16:0/3318079736 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff87406a7e0 0x7ff864005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:33.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.694+0000 7ff873fff700 1 -- 172.21.15.16:0/3318079736 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff864003a00 msgr2=0x7ff864003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:33.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.694+0000 7ff873fff700 1 --2- 172.21.15.16:0/3318079736 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff864003a00 0x7ff864003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:33.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.694+0000 7ff873fff700 1 -- 172.21.15.16:0/3318079736 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8640035b0 con 0x7ff864004850 2023-10-16T12:11:33.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.694+0000 7ff873fff700 1 --2- 172.21.15.16:0/3318079736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff864004850 0x7ff864004cd0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7ff860009d00 tx=0x7ff864005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=460c8b1a2acfd1f7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:33.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.694+0000 7ff8737fe700 1 -- 172.21.15.16:0/3318079736 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff860009e50 con 0x7ff864004850 2023-10-16T12:11:33.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.695+0000 7ff8737fe700 1 -- 172.21.15.16:0/3318079736 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff86000e9a0 con 0x7ff864004850 2023-10-16T12:11:33.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.695+0000 7ff8737fe700 1 -- 172.21.15.16:0/3318079736 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff86000ed30 con 0x7ff864004850 2023-10-16T12:11:33.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.695+0000 7ff87ae7c700 1 -- 172.21.15.16:0/3318079736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff864004850 msgr2=0x7ff864004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:33.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.695+0000 7ff87ae7c700 1 --2- 172.21.15.16:0/3318079736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff864004850 0x7ff864004cd0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7ff860009d00 tx=0x7ff864005880 comp rx=0 tx=0).stop 2023-10-16T12:11:33.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.695+0000 7ff879419700 1 -- 172.21.15.16:0/3318079736 reap_dead start 2023-10-16T12:11:33.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.695+0000 7ff87ae7c700 1 -- 172.21.15.16:0/3318079736 shutdown_connections 2023-10-16T12:11:33.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.695+0000 7ff87ae7c700 1 -- 172.21.15.16:0/3318079736 >> 172.21.15.16:0/3318079736 conn(0x7ff8740b1fd0 msgr2=0x7ff8740b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:33.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.695+0000 7ff87ae7c700 1 -- 172.21.15.16:0/3318079736 shutdown_connections 2023-10-16T12:11:33.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.695+0000 7ff87ae7c700 1 -- 172.21.15.16:0/3318079736 wait complete. 2023-10-16T12:11:33.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.696+0000 7ff87ae7c700 1 Processor -- start 2023-10-16T12:11:33.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.696+0000 7ff87ae7c700 1 -- start start 2023-10-16T12:11:33.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.697+0000 7ff87ae7c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff864004850 0x7ff8740c3a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:33.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.697+0000 7ff87ae7c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff874071d60 0x7ff8741602d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:33.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.697+0000 7ff87ae7c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff87414c370 0x7ff8740c52f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:33.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.697+0000 7ff87ae7c700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8740c4ed0 con 0x7ff874071d60 2023-10-16T12:11:33.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.697+0000 7ff87ae7c700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8740c3fb0 con 0x7ff864004850 2023-10-16T12:11:33.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.697+0000 7ff87ae7c700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8740c4250 con 0x7ff87414c370 2023-10-16T12:11:33.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.697+0000 7ff873fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff874071d60 0x7ff8741602d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:33.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.697+0000 7ff873fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff874071d60 0x7ff8741602d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34650/0 (socket says 172.21.15.16:34650) 2023-10-16T12:11:33.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.697+0000 7ff873fff700 1 -- 172.21.15.16:0/2858195130 learned_addr learned my addr 172.21.15.16:0/2858195130 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:33.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.697+0000 7ff878c18700 1 --2- 172.21.15.16:0/2858195130 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff864004850 0x7ff8740c3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:33.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.697+0000 7ff879419700 1 --2- 172.21.15.16:0/2858195130 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff87414c370 0x7ff8740c52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:33.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.698+0000 7ff873fff700 1 -- 172.21.15.16:0/2858195130 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff87414c370 msgr2=0x7ff8740c52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:33.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.698+0000 7ff873fff700 1 --2- 172.21.15.16:0/2858195130 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff87414c370 0x7ff8740c52f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:33.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.698+0000 7ff873fff700 1 -- 172.21.15.16:0/2858195130 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff864004850 msgr2=0x7ff8740c3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:33.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.698+0000 7ff873fff700 1 --2- 172.21.15.16:0/2858195130 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff864004850 0x7ff8740c3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:33.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.698+0000 7ff873fff700 1 -- 172.21.15.16:0/2858195130 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8740c4530 con 0x7ff874071d60 2023-10-16T12:11:33.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.698+0000 7ff873fff700 1 --2- 172.21.15.16:0/2858195130 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff874071d60 0x7ff8741602d0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7ff8640046a0 tx=0x7ff8600047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:33.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.698+0000 7ff8717fa700 1 -- 172.21.15.16:0/2858195130 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff860021070 con 0x7ff874071d60 2023-10-16T12:11:33.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.698+0000 7ff8717fa700 1 -- 172.21.15.16:0/2858195130 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff860009e50 con 0x7ff874071d60 2023-10-16T12:11:33.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.698+0000 7ff87ae7c700 1 -- 172.21.15.16:0/2858195130 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8740c4810 con 0x7ff874071d60 2023-10-16T12:11:33.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.698+0000 7ff8717fa700 1 -- 172.21.15.16:0/2858195130 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff860006e60 con 0x7ff874071d60 2023-10-16T12:11:33.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.698+0000 7ff87ae7c700 1 -- 172.21.15.16:0/2858195130 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff874160810 con 0x7ff874071d60 2023-10-16T12:11:33.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.699+0000 7ff87ae7c700 1 -- 172.21.15.16:0/2858195130 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff87404ebb0 con 0x7ff874071d60 2023-10-16T12:11:33.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.699+0000 7ff8717fa700 1 -- 172.21.15.16:0/2858195130 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7ff860003af0 con 0x7ff874071d60 2023-10-16T12:11:33.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.700+0000 7ff8717fa700 1 --2- 172.21.15.16:0/2858195130 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7ff85806af00 0x7ff85806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:33.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.700+0000 7ff8717fa700 1 -- 172.21.15.16:0/2858195130 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7ff86008a990 con 0x7ff874071d60 2023-10-16T12:11:33.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.700+0000 7ff878c18700 1 --2- 172.21.15.16:0/2858195130 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7ff85806af00 0x7ff85806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:33.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.700+0000 7ff878c18700 1 --2- 172.21.15.16:0/2858195130 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7ff85806af00 0x7ff85806d3c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff868000fb0 tx=0x7ff868007040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:33.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.704+0000 7ff8717fa700 1 -- 172.21.15.16:0/2858195130 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff860055830 con 0x7ff874071d60 2023-10-16T12:11:33.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:33.987+0000 7ff87ae7c700 1 -- 172.21.15.16:0/2858195130 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]} v 0) v1 -- 0x7ff8740d04c0 con 0x7ff874071d60 2023-10-16T12:11:34.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.260+0000 7ff8717fa700 1 -- 172.21.15.16:0/2858195130 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]}]=0 v69) v1 ==== 161+0+0 (secure 0 0 0) 0x7ff860058e50 con 0x7ff874071d60 2023-10-16T12:11:34.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.262+0000 7ff87ae7c700 1 -- 172.21.15.16:0/2858195130 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7ff85806af00 msgr2=0x7ff85806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:34.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.262+0000 7ff87ae7c700 1 --2- 172.21.15.16:0/2858195130 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7ff85806af00 0x7ff85806d3c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff868000fb0 tx=0x7ff868007040 comp rx=0 tx=0).stop 2023-10-16T12:11:34.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.262+0000 7ff87ae7c700 1 -- 172.21.15.16:0/2858195130 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff874071d60 msgr2=0x7ff8741602d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:34.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.262+0000 7ff87ae7c700 1 --2- 172.21.15.16:0/2858195130 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff874071d60 0x7ff8741602d0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7ff8640046a0 tx=0x7ff8600047c0 comp rx=0 tx=0).stop 2023-10-16T12:11:34.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.263+0000 7ff87ae7c700 1 -- 172.21.15.16:0/2858195130 shutdown_connections 2023-10-16T12:11:34.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.263+0000 7ff87ae7c700 1 --2- 172.21.15.16:0/2858195130 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7ff85806af00 0x7ff85806d3c0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:34.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.263+0000 7ff87ae7c700 1 --2- 172.21.15.16:0/2858195130 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff87414c370 0x7ff8740c52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:34.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.263+0000 7ff87ae7c700 1 --2- 172.21.15.16:0/2858195130 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff864004850 0x7ff8740c3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:34.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.263+0000 7ff87ae7c700 1 --2- 172.21.15.16:0/2858195130 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff874071d60 0x7ff8741602d0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:34.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.263+0000 7ff87ae7c700 1 -- 172.21.15.16:0/2858195130 >> 172.21.15.16:0/2858195130 conn(0x7ff8740b1fd0 msgr2=0x7ff8740d0cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:34.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.263+0000 7ff87ae7c700 1 -- 172.21.15.16:0/2858195130 shutdown_connections 2023-10-16T12:11:34.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.263+0000 7ff87ae7c700 1 -- 172.21.15.16:0/2858195130 wait complete. 2023-10-16T12:11:34.278 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-16T12:11:34.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.406+0000 7f565b782700 1 Processor -- start 2023-10-16T12:11:34.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.407+0000 7f565b782700 1 -- start start 2023-10-16T12:11:34.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.407+0000 7f565b782700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f565406e170 0x7f56540cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:34.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.407+0000 7f565b782700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56540cef70 con 0x7f565415b070 2023-10-16T12:11:34.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.407+0000 7f565b782700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56540ce360 con 0x7f56540cb880 2023-10-16T12:11:34.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.407+0000 7f565b782700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56540ce640 con 0x7f565406e170 2023-10-16T12:11:34.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.407+0000 7f565951e700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f565415b070 0x7f565415b450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:52162/0 (socket says 172.21.15.16:52162) 2023-10-16T12:11:34.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.408+0000 7f565951e700 1 -- 172.21.15.16:0/3661249983 learned_addr learned my addr 172.21.15.16:0/3661249983 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:34.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.408+0000 7f5659d1f700 1 --2- 172.21.15.16:0/3661249983 >> v2:172.21.15.49:3300/0 conn(0x7f565406e170 0x7f56540cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:34.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.408+0000 7f5647fff700 1 -- 172.21.15.16:0/3661249983 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3060103791 0 0) 0x7f56540cef70 con 0x7f565415b070 2023-10-16T12:11:34.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.408+0000 7f5647fff700 1 -- 172.21.15.16:0/3661249983 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56400035b0 con 0x7f565415b070 2023-10-16T12:11:34.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.408+0000 7f5647fff700 1 -- 172.21.15.16:0/3661249983 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3280090138 0 0) 0x7f56540ce360 con 0x7f56540cb880 2023-10-16T12:11:34.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.408+0000 7f5647fff700 1 -- 172.21.15.16:0/3661249983 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56540cef70 con 0x7f56540cb880 2023-10-16T12:11:34.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.408+0000 7f5647fff700 1 -- 172.21.15.16:0/3661249983 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3454796058 0 0) 0x7f56400035b0 con 0x7f565415b070 2023-10-16T12:11:34.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.408+0000 7f5647fff700 1 -- 172.21.15.16:0/3661249983 >> v2:172.21.15.49:3300/0 conn(0x7f565406e170 msgr2=0x7f56540cbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:34.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.408+0000 7f5647fff700 1 --2- 172.21.15.16:0/3661249983 >> v2:172.21.15.49:3300/0 conn(0x7f565406e170 0x7f56540cbd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:34.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.408+0000 7f5647fff700 1 -- 172.21.15.16:0/3661249983 >> v1:172.21.15.49:6789/0 conn(0x7f56540cb880 legacy=0x7f565406da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:34.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.408+0000 7f5647fff700 1 -- 172.21.15.16:0/3661249983 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56540ce920 con 0x7f565415b070 2023-10-16T12:11:34.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.408+0000 7f5647fff700 1 -- 172.21.15.16:0/3661249983 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f5650001670 con 0x7f565415b070 2023-10-16T12:11:34.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.408+0000 7f5647fff700 1 -- 172.21.15.16:0/3661249983 >> v1:172.21.15.16:6789/0 conn(0x7f565415b070 legacy=0x7f565415b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:34.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.408+0000 7f5647fff700 1 --2- 172.21.15.16:0/3661249983 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5640003b90 0x7f5640003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:34.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.408+0000 7f5647fff700 1 --2- 172.21.15.16:0/3661249983 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5640004850 0x7f5640004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:34.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.408+0000 7f5647fff700 1 --2- 172.21.15.16:0/3661249983 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5640005210 0x7f564000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:34.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.408+0000 7f5647fff700 1 -- 172.21.15.16:0/3661249983 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56540ce640 con 0x7f5640004850 2023-10-16T12:11:34.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.408+0000 7f5647fff700 1 -- 172.21.15.16:0/3661249983 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f564000c140 con 0x7f5640003b90 2023-10-16T12:11:34.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.408+0000 7f5647fff700 1 -- 172.21.15.16:0/3661249983 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f564000c300 con 0x7f5640005210 2023-10-16T12:11:34.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.408+0000 7f5658d1d700 1 --2- 172.21.15.16:0/3661249983 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5640004850 0x7f5640004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:34.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.408+0000 7f565951e700 1 --2- 172.21.15.16:0/3661249983 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5640003b90 0x7f5640003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:34.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.408+0000 7f5659d1f700 1 --2- 172.21.15.16:0/3661249983 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5640005210 0x7f564000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:34.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.409+0000 7f5658d1d700 1 -- 172.21.15.16:0/3661249983 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5640005210 msgr2=0x7f564000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:34.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.409+0000 7f5658d1d700 1 --2- 172.21.15.16:0/3661249983 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5640005210 0x7f564000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:34.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.409+0000 7f5658d1d700 1 -- 172.21.15.16:0/3661249983 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5640003b90 msgr2=0x7f5640003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:34.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.409+0000 7f5658d1d700 1 --2- 172.21.15.16:0/3661249983 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5640003b90 0x7f5640003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:34.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.409+0000 7f5658d1d700 1 -- 172.21.15.16:0/3661249983 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56400035b0 con 0x7f5640004850 2023-10-16T12:11:34.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.409+0000 7f5659d1f700 1 --2- 172.21.15.16:0/3661249983 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5640005210 0x7f564000ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:11:34.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.409+0000 7f5659d1f700 1 -- 172.21.15.16:0/3661249983 reap_dead start 2023-10-16T12:11:34.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.409+0000 7f5658d1d700 1 --2- 172.21.15.16:0/3661249983 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5640004850 0x7f5640004cd0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f5648001940 tx=0x7f5640005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf7c351b8917d1c4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:34.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.409+0000 7f5647fff700 1 -- 172.21.15.16:0/3661249983 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5648001430 con 0x7f5640004850 2023-10-16T12:11:34.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.409+0000 7f5647fff700 1 -- 172.21.15.16:0/3661249983 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5648001d70 con 0x7f5640004850 2023-10-16T12:11:34.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.409+0000 7f5647fff700 1 -- 172.21.15.16:0/3661249983 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5648005510 con 0x7f5640004850 2023-10-16T12:11:34.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.409+0000 7f565b782700 1 -- 172.21.15.16:0/3661249983 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5640004850 msgr2=0x7f5640004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:34.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.409+0000 7f565b782700 1 --2- 172.21.15.16:0/3661249983 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5640004850 0x7f5640004cd0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f5648001940 tx=0x7f5640005880 comp rx=0 tx=0).stop 2023-10-16T12:11:34.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.410+0000 7f565b782700 1 -- 172.21.15.16:0/3661249983 shutdown_connections 2023-10-16T12:11:34.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.410+0000 7f565b782700 1 --2- 172.21.15.16:0/3661249983 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5640004850 0x7f5640004cd0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:34.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.410+0000 7f565b782700 1 -- 172.21.15.16:0/3661249983 >> 172.21.15.16:0/3661249983 conn(0x7f56540b1f40 msgr2=0x7f56540b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:34.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.410+0000 7f565b782700 1 -- 172.21.15.16:0/3661249983 shutdown_connections 2023-10-16T12:11:34.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.410+0000 7f565b782700 1 -- 172.21.15.16:0/3661249983 wait complete. 2023-10-16T12:11:34.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.411+0000 7f565b782700 1 Processor -- start 2023-10-16T12:11:34.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.411+0000 7f565b782700 1 -- start start 2023-10-16T12:11:34.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.412+0000 7f565b782700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5640004850 0x7f56540c9f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:34.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.412+0000 7f565b782700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56540c7610 0x7f565414b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:34.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.412+0000 7f565b782700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f56541f48c0 0x7f56541573a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:34.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.412+0000 7f565b782700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56541f4430 con 0x7f56540c7610 2023-10-16T12:11:34.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.412+0000 7f565b782700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56540ca630 con 0x7f56541f48c0 2023-10-16T12:11:34.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.412+0000 7f565b782700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56540ca910 con 0x7f5640004850 2023-10-16T12:11:34.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.412+0000 7f5658d1d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56540c7610 0x7f565414b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:34.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.412+0000 7f5658d1d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56540c7610 0x7f565414b7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34670/0 (socket says 172.21.15.16:34670) 2023-10-16T12:11:34.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.412+0000 7f5658d1d700 1 -- 172.21.15.16:0/1614988609 learned_addr learned my addr 172.21.15.16:0/1614988609 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:34.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.412+0000 7f5658d1d700 1 -- 172.21.15.16:0/1614988609 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5640004850 msgr2=0x7f56540c9f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:34.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.412+0000 7f5658d1d700 1 --2- 172.21.15.16:0/1614988609 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5640004850 0x7f56540c9f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:34.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.412+0000 7f5658d1d700 1 -- 172.21.15.16:0/1614988609 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f56541f48c0 msgr2=0x7f56541573a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:34.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.412+0000 7f5658d1d700 1 --2- 172.21.15.16:0/1614988609 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f56541f48c0 0x7f56541573a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:34.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.412+0000 7f5658d1d700 1 -- 172.21.15.16:0/1614988609 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56540cabf0 con 0x7f56540c7610 2023-10-16T12:11:34.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.412+0000 7f5658d1d700 1 --2- 172.21.15.16:0/1614988609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56540c7610 0x7f565414b7d0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f5648000c00 tx=0x7f56480012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:34.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.412+0000 7f5645ffb700 1 -- 172.21.15.16:0/1614988609 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5648006a70 con 0x7f56540c7610 2023-10-16T12:11:34.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.412+0000 7f5645ffb700 1 -- 172.21.15.16:0/1614988609 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5648006bd0 con 0x7f56540c7610 2023-10-16T12:11:34.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.412+0000 7f565b782700 1 -- 172.21.15.16:0/1614988609 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56541f40c0 con 0x7f56540c7610 2023-10-16T12:11:34.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.412+0000 7f5645ffb700 1 -- 172.21.15.16:0/1614988609 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5648007040 con 0x7f56540c7610 2023-10-16T12:11:34.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.413+0000 7f565b782700 1 -- 172.21.15.16:0/1614988609 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56540c7a90 con 0x7f56540c7610 2023-10-16T12:11:34.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.413+0000 7f565b782700 1 -- 172.21.15.16:0/1614988609 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f565415db40 con 0x7f56540c7610 2023-10-16T12:11:34.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.414+0000 7f5645ffb700 1 -- 172.21.15.16:0/1614988609 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f56480071a0 con 0x7f56540c7610 2023-10-16T12:11:34.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.414+0000 7f5645ffb700 1 --2- 172.21.15.16:0/1614988609 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f563806af00 0x7f563806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:34.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.414+0000 7f5645ffb700 1 -- 172.21.15.16:0/1614988609 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f564808a540 con 0x7f56540c7610 2023-10-16T12:11:34.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.415+0000 7f565951e700 1 --2- 172.21.15.16:0/1614988609 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f563806af00 0x7f563806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:34.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.415+0000 7f565951e700 1 --2- 172.21.15.16:0/1614988609 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f563806af00 0x7f563806d3c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f5650000f60 tx=0x7f5650000ed0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:34.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.419+0000 7f5645ffb700 1 -- 172.21.15.16:0/1614988609 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f56480553e0 con 0x7f56540c7610 2023-10-16T12:11:34.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.625+0000 7f565b782700 1 -- 172.21.15.16:0/1614988609 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f56541f4570 con 0x7f56540c7610 2023-10-16T12:11:34.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.628+0000 7f5645ffb700 1 -- 172.21.15.16:0/1614988609 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7f56541f4570 con 0x7f56540c7610 2023-10-16T12:11:34.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.630+0000 7f565b782700 1 -- 172.21.15.16:0/1614988609 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f563806af00 msgr2=0x7f563806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:34.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.630+0000 7f565b782700 1 --2- 172.21.15.16:0/1614988609 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f563806af00 0x7f563806d3c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f5650000f60 tx=0x7f5650000ed0 comp rx=0 tx=0).stop 2023-10-16T12:11:34.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.630+0000 7f565b782700 1 -- 172.21.15.16:0/1614988609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56540c7610 msgr2=0x7f565414b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:34.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.630+0000 7f565b782700 1 --2- 172.21.15.16:0/1614988609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56540c7610 0x7f565414b7d0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f5648000c00 tx=0x7f56480012d0 comp rx=0 tx=0).stop 2023-10-16T12:11:34.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.630+0000 7f565b782700 1 -- 172.21.15.16:0/1614988609 shutdown_connections 2023-10-16T12:11:34.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.630+0000 7f565b782700 1 --2- 172.21.15.16:0/1614988609 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f563806af00 0x7f563806d3c0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:34.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.630+0000 7f565b782700 1 --2- 172.21.15.16:0/1614988609 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f56541f48c0 0x7f56541573a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:34.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.630+0000 7f565b782700 1 --2- 172.21.15.16:0/1614988609 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5640004850 0x7f56540c9f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:34.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.630+0000 7f565b782700 1 --2- 172.21.15.16:0/1614988609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56540c7610 0x7f565414b7d0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:34.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.630+0000 7f565b782700 1 -- 172.21.15.16:0/1614988609 >> 172.21.15.16:0/1614988609 conn(0x7f56540b1f40 msgr2=0x7f56541591d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:34.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.630+0000 7f565b782700 1 -- 172.21.15.16:0/1614988609 shutdown_connections 2023-10-16T12:11:34.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:34.630+0000 7f565b782700 1 -- 172.21.15.16:0/1614988609 wait complete. 2023-10-16T12:11:34.644 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T12:11:34.645 DEBUG:teuthology.orchestra.run.smithi016:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T12:11:34.682 DEBUG:teuthology.orchestra.run.smithi049:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T12:11:34.732 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T12:11:34.733 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:11:34.733 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:11:34.751 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:11:34.752 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:11:34.809 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:11:34.810 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:11:34.867 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:11:35.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.029+0000 7fb1bc7a1700 1 Processor -- start 2023-10-16T12:11:35.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.029+0000 7fb1bc7a1700 1 -- start start 2023-10-16T12:11:35.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.030+0000 7fb1bc7a1700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb1b40cf4a0 0x7fb1b40726e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:35.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.030+0000 7fb1bc7a1700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fb1b40b5a00 0x7fb1b415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:35.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.030+0000 7fb1bc7a1700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1b415bc80 con 0x7fb1b40cf880 2023-10-16T12:11:35.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.030+0000 7fb1bc7a1700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1b40b5e60 con 0x7fb1b40b5a00 2023-10-16T12:11:35.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.030+0000 7fb1bc7a1700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1b4072c20 con 0x7fb1b40cf4a0 2023-10-16T12:11:35.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.030+0000 7fb1ba53d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb1b40cf4a0 0x7fb1b40726e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:35.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.030+0000 7fb1bad3e700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fb1b40b5a00 0x7fb1b415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:35.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.030+0000 7fb1b9d3c700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7fb1b40cf880 0x7fb1b40b52f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:59364/0 (socket says 172.21.15.16:59364) 2023-10-16T12:11:35.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.030+0000 7fb1b9d3c700 1 -- 172.21.15.16:0/2834663421 learned_addr learned my addr 172.21.15.16:0/2834663421 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:35.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.031+0000 7fb1ba53d700 1 -- 172.21.15.16:0/2834663421 >> v2:172.21.15.49:3300/0 conn(0x7fb1b40b5a00 msgr2=0x7fb1b415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:35.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.031+0000 7fb1ba53d700 1 --2- 172.21.15.16:0/2834663421 >> v2:172.21.15.49:3300/0 conn(0x7fb1b40b5a00 0x7fb1b415a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:35.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.031+0000 7fb1ba53d700 1 -- 172.21.15.16:0/2834663421 >> v1:172.21.15.49:6789/0 conn(0x7fb1b40cf880 legacy=0x7fb1b40b52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:35.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.031+0000 7fb1ba53d700 1 -- 172.21.15.16:0/2834663421 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1b415b070 con 0x7fb1b40cf4a0 2023-10-16T12:11:35.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.031+0000 7fb1bad3e700 1 --2- 172.21.15.16:0/2834663421 >> v2:172.21.15.49:3300/0 conn(0x7fb1b40b5a00 0x7fb1b415a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:11:35.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.031+0000 7fb1ba53d700 1 --2- 172.21.15.16:0/2834663421 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb1b40cf4a0 0x7fb1b40726e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb1a4009840 tx=0x7fb1a4009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ac22404148cd241 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:35.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.031+0000 7fb1b953b700 1 -- 172.21.15.16:0/2834663421 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1a4011830 con 0x7fb1b40cf4a0 2023-10-16T12:11:35.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.031+0000 7fb1b953b700 1 -- 172.21.15.16:0/2834663421 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1a4011990 con 0x7fb1b40cf4a0 2023-10-16T12:11:35.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.031+0000 7fb1b953b700 1 -- 172.21.15.16:0/2834663421 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1a4011d20 con 0x7fb1b40cf4a0 2023-10-16T12:11:35.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.032+0000 7fb1bc7a1700 1 -- 172.21.15.16:0/2834663421 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb1b40cf4a0 msgr2=0x7fb1b40726e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:35.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.032+0000 7fb1bc7a1700 1 --2- 172.21.15.16:0/2834663421 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb1b40cf4a0 0x7fb1b40726e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb1a4009840 tx=0x7fb1a4009b50 comp rx=0 tx=0).stop 2023-10-16T12:11:35.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.032+0000 7fb1bc7a1700 1 -- 172.21.15.16:0/2834663421 shutdown_connections 2023-10-16T12:11:35.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.032+0000 7fb1bc7a1700 1 --2- 172.21.15.16:0/2834663421 >> v2:172.21.15.49:3300/0 conn(0x7fb1b40b5a00 0x7fb1b415a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:35.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.032+0000 7fb1bc7a1700 1 --2- 172.21.15.16:0/2834663421 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb1b40cf4a0 0x7fb1b40726e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:35.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.032+0000 7fb1bc7a1700 1 -- 172.21.15.16:0/2834663421 >> 172.21.15.16:0/2834663421 conn(0x7fb1b40b1fd0 msgr2=0x7fb1b40b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:35.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.032+0000 7fb1bc7a1700 1 -- 172.21.15.16:0/2834663421 shutdown_connections 2023-10-16T12:11:35.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.032+0000 7fb1bc7a1700 1 -- 172.21.15.16:0/2834663421 wait complete. 2023-10-16T12:11:35.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.033+0000 7fb1bc7a1700 1 Processor -- start 2023-10-16T12:11:35.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.033+0000 7fb1bc7a1700 1 -- start start 2023-10-16T12:11:35.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.034+0000 7fb1bc7a1700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb1b40b5a00 0x7fb1b41f0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:35.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.034+0000 7fb1bc7a1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1b40cf4a0 0x7fb1b40a84c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:35.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.034+0000 7fb1bc7a1700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb1b40cf880 0x7fb1b40a8a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:35.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.034+0000 7fb1bc7a1700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1b414b0a0 con 0x7fb1b40cf4a0 2023-10-16T12:11:35.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.034+0000 7fb1bc7a1700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1b414e590 con 0x7fb1b40cf880 2023-10-16T12:11:35.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.034+0000 7fb1bc7a1700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1b40c9d10 con 0x7fb1b40b5a00 2023-10-16T12:11:35.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.034+0000 7fb1b9d3c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1b40cf4a0 0x7fb1b40a84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:35.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.034+0000 7fb1b9d3c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1b40cf4a0 0x7fb1b40a84c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34684/0 (socket says 172.21.15.16:34684) 2023-10-16T12:11:35.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.034+0000 7fb1b9d3c700 1 -- 172.21.15.16:0/965283090 learned_addr learned my addr 172.21.15.16:0/965283090 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:35.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.034+0000 7fb1bad3e700 1 --2- 172.21.15.16:0/965283090 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb1b40cf880 0x7fb1b40a8a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:35.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.034+0000 7fb1ba53d700 1 --2- 172.21.15.16:0/965283090 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb1b40b5a00 0x7fb1b41f0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:35.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.034+0000 7fb1b9d3c700 1 -- 172.21.15.16:0/965283090 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb1b40b5a00 msgr2=0x7fb1b41f0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:35.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.034+0000 7fb1b9d3c700 1 --2- 172.21.15.16:0/965283090 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb1b40b5a00 0x7fb1b41f0330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:35.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.034+0000 7fb1b9d3c700 1 -- 172.21.15.16:0/965283090 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb1b40cf880 msgr2=0x7fb1b40a8a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:35.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.034+0000 7fb1b9d3c700 1 --2- 172.21.15.16:0/965283090 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb1b40cf880 0x7fb1b40a8a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:35.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.034+0000 7fb1b9d3c700 1 -- 172.21.15.16:0/965283090 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1ac009a70 con 0x7fb1b40cf4a0 2023-10-16T12:11:35.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.034+0000 7fb1b9d3c700 1 --2- 172.21.15.16:0/965283090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1b40cf4a0 0x7fb1b40a84c0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fb1b000c480 tx=0x7fb1b000c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:35.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.035+0000 7fb1aaffd700 1 -- 172.21.15.16:0/965283090 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1b0005d70 con 0x7fb1b40cf4a0 2023-10-16T12:11:35.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.035+0000 7fb1aaffd700 1 -- 172.21.15.16:0/965283090 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1b000ce80 con 0x7fb1b40cf4a0 2023-10-16T12:11:35.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.035+0000 7fb1aaffd700 1 -- 172.21.15.16:0/965283090 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1b00072d0 con 0x7fb1b40cf4a0 2023-10-16T12:11:35.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.035+0000 7fb1bc7a1700 1 -- 172.21.15.16:0/965283090 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1a40094a0 con 0x7fb1b40cf4a0 2023-10-16T12:11:35.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.035+0000 7fb1bc7a1700 1 -- 172.21.15.16:0/965283090 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1b414b590 con 0x7fb1b40cf4a0 2023-10-16T12:11:35.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.036+0000 7fb1bc7a1700 1 -- 172.21.15.16:0/965283090 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1b404ebb0 con 0x7fb1b40cf4a0 2023-10-16T12:11:35.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.036+0000 7fb1aaffd700 1 -- 172.21.15.16:0/965283090 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fb1b0007430 con 0x7fb1b40cf4a0 2023-10-16T12:11:35.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.037+0000 7fb1aaffd700 1 --2- 172.21.15.16:0/965283090 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fb19806af00 0x7fb19806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:35.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.037+0000 7fb1aaffd700 1 -- 172.21.15.16:0/965283090 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fb1b0088c80 con 0x7fb1b40cf4a0 2023-10-16T12:11:35.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.037+0000 7fb1ba53d700 1 --2- 172.21.15.16:0/965283090 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fb19806af00 0x7fb19806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:35.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.037+0000 7fb1ba53d700 1 --2- 172.21.15.16:0/965283090 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fb19806af00 0x7fb19806d3c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fb1a4000c00 tx=0x7fb1a4008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:35.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.041+0000 7fb1aaffd700 1 -- 172.21.15.16:0/965283090 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb1b0053900 con 0x7fb1b40cf4a0 2023-10-16T12:11:35.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.247+0000 7fb1bc7a1700 1 -- 172.21.15.16:0/965283090 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb1b41f0870 con 0x7fb1b40cf4a0 2023-10-16T12:11:35.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.248+0000 7fb1aaffd700 1 -- 172.21.15.16:0/965283090 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18629 (secure 0 0 0) 0x7fb1b0088090 con 0x7fb1b40cf4a0 2023-10-16T12:11:35.251 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:11:35.251 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:11:28.241865+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:10:36.935773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T12:10:38.004870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-16T12:10:41.672025+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"3.2b","mappings":[{"from":0,"to":2}]},{"pgid":"3.38","mappings":[{"from":3,"to":1}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]},{"pgid":"4.1b","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:11:35.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.251+0000 7fb1bc7a1700 1 -- 172.21.15.16:0/965283090 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fb19806af00 msgr2=0x7fb19806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:35.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.251+0000 7fb1bc7a1700 1 --2- 172.21.15.16:0/965283090 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fb19806af00 0x7fb19806d3c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fb1a4000c00 tx=0x7fb1a4008030 comp rx=0 tx=0).stop 2023-10-16T12:11:35.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.251+0000 7fb1bc7a1700 1 -- 172.21.15.16:0/965283090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1b40cf4a0 msgr2=0x7fb1b40a84c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:35.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.251+0000 7fb1bc7a1700 1 --2- 172.21.15.16:0/965283090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1b40cf4a0 0x7fb1b40a84c0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fb1b000c480 tx=0x7fb1b000c790 comp rx=0 tx=0).stop 2023-10-16T12:11:35.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.251+0000 7fb1bc7a1700 1 -- 172.21.15.16:0/965283090 shutdown_connections 2023-10-16T12:11:35.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.251+0000 7fb1bc7a1700 1 --2- 172.21.15.16:0/965283090 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fb19806af00 0x7fb19806d3c0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:35.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.251+0000 7fb1bc7a1700 1 --2- 172.21.15.16:0/965283090 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb1b40cf880 0x7fb1b40a8a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:35.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.251+0000 7fb1bc7a1700 1 --2- 172.21.15.16:0/965283090 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb1b40b5a00 0x7fb1b41f0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:35.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.251+0000 7fb1bc7a1700 1 --2- 172.21.15.16:0/965283090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1b40cf4a0 0x7fb1b40a84c0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:35.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.251+0000 7fb1bc7a1700 1 -- 172.21.15.16:0/965283090 >> 172.21.15.16:0/965283090 conn(0x7fb1b40b1fd0 msgr2=0x7fb1b40ce8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:35.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.251+0000 7fb1bc7a1700 1 -- 172.21.15.16:0/965283090 shutdown_connections 2023-10-16T12:11:35.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.251+0000 7fb1bc7a1700 1 -- 172.21.15.16:0/965283090 wait complete. 2023-10-16T12:11:35.265 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T12:10:36.935773+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T12:10:38.004870+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-16T12:10:41.672025+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-16T12:11:35.266 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:11:35.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.392+0000 7fa1d88db700 1 Processor -- start 2023-10-16T12:11:35.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.392+0000 7fa1d88db700 1 -- start start 2023-10-16T12:11:35.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.393+0000 7fa1d88db700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa1d4160ee0 0x7fa1d40aeb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:35.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.393+0000 7fa1d88db700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fa1d40ce890 0x7fa1d406aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:35.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.393+0000 7fa1d88db700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1d40af4a0 con 0x7fa1d40af040 2023-10-16T12:11:35.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.393+0000 7fa1d88db700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1d40cf770 con 0x7fa1d4160ee0 2023-10-16T12:11:35.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.393+0000 7fa1d88db700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1d406aa10 con 0x7fa1d40ce890 2023-10-16T12:11:35.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.393+0000 7fa1d259c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa1d4160ee0 0x7fa1d40aeb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:35.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.393+0000 7fa1d1d9b700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa1d40af040 0x7fa1d40ce180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:52196/0 (socket says 172.21.15.16:52196) 2023-10-16T12:11:35.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.393+0000 7fa1d1d9b700 1 -- 172.21.15.16:0/1125169383 learned_addr learned my addr 172.21.15.16:0/1125169383 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:35.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.394+0000 7fa1d2d9d700 1 --2- 172.21.15.16:0/1125169383 >> v2:172.21.15.49:3300/0 conn(0x7fa1d40ce890 0x7fa1d406aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:35.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.394+0000 7fa1d259c700 1 -- 172.21.15.16:0/1125169383 >> v2:172.21.15.49:3300/0 conn(0x7fa1d40ce890 msgr2=0x7fa1d406aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:35.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.394+0000 7fa1d259c700 1 --2- 172.21.15.16:0/1125169383 >> v2:172.21.15.49:3300/0 conn(0x7fa1d40ce890 0x7fa1d406aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:35.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.394+0000 7fa1d259c700 1 -- 172.21.15.16:0/1125169383 >> v1:172.21.15.16:6789/0 conn(0x7fa1d40af040 legacy=0x7fa1d40ce180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:35.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.394+0000 7fa1d259c700 1 -- 172.21.15.16:0/1125169383 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1d40ceed0 con 0x7fa1d4160ee0 2023-10-16T12:11:35.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.394+0000 7fa1d159a700 1 -- 172.21.15.16:0/1125169383 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2501928250 0 0) 0x7fa1d40af4a0 con 0x7fa1d40af040 2023-10-16T12:11:35.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.394+0000 7fa1d259c700 1 --2- 172.21.15.16:0/1125169383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1d4160ee0 0x7fa1d40aeb00 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fa1d40ca040 tx=0x7fa1c8009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7284844eea3f231 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:35.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.394+0000 7fa1d159a700 1 -- 172.21.15.16:0/1125169383 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1c8011830 con 0x7fa1d4160ee0 2023-10-16T12:11:35.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.394+0000 7fa1d159a700 1 -- 172.21.15.16:0/1125169383 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1c8011990 con 0x7fa1d4160ee0 2023-10-16T12:11:35.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.394+0000 7fa1d159a700 1 -- 172.21.15.16:0/1125169383 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1c8011d20 con 0x7fa1d4160ee0 2023-10-16T12:11:35.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.395+0000 7fa1d88db700 1 -- 172.21.15.16:0/1125169383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1d4160ee0 msgr2=0x7fa1d40aeb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:35.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.395+0000 7fa1d88db700 1 --2- 172.21.15.16:0/1125169383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1d4160ee0 0x7fa1d40aeb00 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fa1d40ca040 tx=0x7fa1c8009b20 comp rx=0 tx=0).stop 2023-10-16T12:11:35.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.395+0000 7fa1d88db700 1 -- 172.21.15.16:0/1125169383 shutdown_connections 2023-10-16T12:11:35.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.395+0000 7fa1d88db700 1 --2- 172.21.15.16:0/1125169383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1d4160ee0 0x7fa1d40aeb00 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:35.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.395+0000 7fa1d88db700 1 --2- 172.21.15.16:0/1125169383 >> v2:172.21.15.49:3300/0 conn(0x7fa1d40ce890 0x7fa1d406aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:35.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.395+0000 7fa1d88db700 1 -- 172.21.15.16:0/1125169383 >> 172.21.15.16:0/1125169383 conn(0x7fa1d40cc5e0 msgr2=0x7fa1d40cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:35.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.395+0000 7fa1d88db700 1 -- 172.21.15.16:0/1125169383 shutdown_connections 2023-10-16T12:11:35.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.395+0000 7fa1d88db700 1 -- 172.21.15.16:0/1125169383 wait complete. 2023-10-16T12:11:35.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.396+0000 7fa1d88db700 1 Processor -- start 2023-10-16T12:11:35.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.396+0000 7fa1d88db700 1 -- start start 2023-10-16T12:11:35.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.397+0000 7fa1d88db700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1d40af040 0x7fa1d406b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:35.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.397+0000 7fa1d88db700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa1d40ce890 0x7fa1d414c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:35.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.397+0000 7fa1d88db700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa1d4160ee0 0x7fa1d414b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:35.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.397+0000 7fa1d88db700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1d415e4d0 con 0x7fa1d40af040 2023-10-16T12:11:35.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.397+0000 7fa1d88db700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1d40c7620 con 0x7fa1d4160ee0 2023-10-16T12:11:35.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.397+0000 7fa1d88db700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1d414c700 con 0x7fa1d40ce890 2023-10-16T12:11:35.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.397+0000 7fa1d259c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1d40af040 0x7fa1d406b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:35.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.397+0000 7fa1d259c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1d40af040 0x7fa1d406b870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34694/0 (socket says 172.21.15.16:34694) 2023-10-16T12:11:35.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.397+0000 7fa1d259c700 1 -- 172.21.15.16:0/1261060883 learned_addr learned my addr 172.21.15.16:0/1261060883 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:35.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.397+0000 7fa1d2d9d700 1 --2- 172.21.15.16:0/1261060883 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa1d4160ee0 0x7fa1d414b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:35.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.397+0000 7fa1d1d9b700 1 --2- 172.21.15.16:0/1261060883 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa1d40ce890 0x7fa1d414c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:35.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.397+0000 7fa1d259c700 1 -- 172.21.15.16:0/1261060883 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa1d40ce890 msgr2=0x7fa1d414c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:35.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.397+0000 7fa1d259c700 1 --2- 172.21.15.16:0/1261060883 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa1d40ce890 0x7fa1d414c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:35.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.397+0000 7fa1d259c700 1 -- 172.21.15.16:0/1261060883 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa1d4160ee0 msgr2=0x7fa1d414b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:35.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.397+0000 7fa1d259c700 1 --2- 172.21.15.16:0/1261060883 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa1d4160ee0 0x7fa1d414b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:35.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.397+0000 7fa1d259c700 1 -- 172.21.15.16:0/1261060883 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1c80094a0 con 0x7fa1d40af040 2023-10-16T12:11:35.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.397+0000 7fa1d259c700 1 --2- 172.21.15.16:0/1261060883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1d40af040 0x7fa1d406b870 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fa1c80090b0 tx=0x7fa1c8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:35.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.398+0000 7fa1c2ffd700 1 -- 172.21.15.16:0/1261060883 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1c80036a0 con 0x7fa1d40af040 2023-10-16T12:11:35.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.398+0000 7fa1c2ffd700 1 -- 172.21.15.16:0/1261060883 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1c801e070 con 0x7fa1d40af040 2023-10-16T12:11:35.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.398+0000 7fa1d88db700 1 -- 172.21.15.16:0/1261060883 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1d414c9e0 con 0x7fa1d40af040 2023-10-16T12:11:35.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.398+0000 7fa1c2ffd700 1 -- 172.21.15.16:0/1261060883 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1c8003a00 con 0x7fa1d40af040 2023-10-16T12:11:35.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.398+0000 7fa1d88db700 1 -- 172.21.15.16:0/1261060883 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1d414cfa0 con 0x7fa1d40af040 2023-10-16T12:11:35.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.399+0000 7fa1d88db700 1 -- 172.21.15.16:0/1261060883 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1d404ebb0 con 0x7fa1d40af040 2023-10-16T12:11:35.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.399+0000 7fa1c2ffd700 1 -- 172.21.15.16:0/1261060883 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fa1c8003b60 con 0x7fa1d40af040 2023-10-16T12:11:35.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.399+0000 7fa1c2ffd700 1 --2- 172.21.15.16:0/1261060883 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fa1b406af00 0x7fa1b406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:35.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.400+0000 7fa1c2ffd700 1 -- 172.21.15.16:0/1261060883 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fa1c808aa10 con 0x7fa1d40af040 2023-10-16T12:11:35.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.400+0000 7fa1d1d9b700 1 --2- 172.21.15.16:0/1261060883 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fa1b406af00 0x7fa1b406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:35.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.400+0000 7fa1d1d9b700 1 --2- 172.21.15.16:0/1261060883 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fa1b406af00 0x7fa1b406d3c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa1d415e920 tx=0x7fa1c4007be0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:35.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.404+0000 7fa1c2ffd700 1 -- 172.21.15.16:0/1261060883 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa1c8055790 con 0x7fa1d40af040 2023-10-16T12:11:35.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.606+0000 7fa1d88db700 1 -- 172.21.15.16:0/1261060883 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa1d40cdba0 con 0x7fa1d40af040 2023-10-16T12:11:35.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.607+0000 7fa1c2ffd700 1 -- 172.21.15.16:0/1261060883 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa1c80200a0 con 0x7fa1d40af040 2023-10-16T12:11:35.609 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T12:11:35.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.609+0000 7fa1d88db700 1 -- 172.21.15.16:0/1261060883 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fa1b406af00 msgr2=0x7fa1b406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:35.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.609+0000 7fa1d88db700 1 --2- 172.21.15.16:0/1261060883 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fa1b406af00 0x7fa1b406d3c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa1d415e920 tx=0x7fa1c4007be0 comp rx=0 tx=0).stop 2023-10-16T12:11:35.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.609+0000 7fa1d88db700 1 -- 172.21.15.16:0/1261060883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1d40af040 msgr2=0x7fa1d406b870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:35.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.609+0000 7fa1d88db700 1 --2- 172.21.15.16:0/1261060883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1d40af040 0x7fa1d406b870 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fa1c80090b0 tx=0x7fa1c8004750 comp rx=0 tx=0).stop 2023-10-16T12:11:35.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.609+0000 7fa1d88db700 1 -- 172.21.15.16:0/1261060883 shutdown_connections 2023-10-16T12:11:35.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.609+0000 7fa1d88db700 1 --2- 172.21.15.16:0/1261060883 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fa1b406af00 0x7fa1b406d3c0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:35.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.609+0000 7fa1d88db700 1 --2- 172.21.15.16:0/1261060883 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa1d4160ee0 0x7fa1d414b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:35.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.609+0000 7fa1d88db700 1 --2- 172.21.15.16:0/1261060883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1d40af040 0x7fa1d406b870 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:35.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.609+0000 7fa1d88db700 1 --2- 172.21.15.16:0/1261060883 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa1d40ce890 0x7fa1d414c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:35.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.609+0000 7fa1d88db700 1 -- 172.21.15.16:0/1261060883 >> 172.21.15.16:0/1261060883 conn(0x7fa1d40cc5e0 msgr2=0x7fa1d415fff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:35.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.609+0000 7fa1d88db700 1 -- 172.21.15.16:0/1261060883 shutdown_connections 2023-10-16T12:11:35.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.609+0000 7fa1d88db700 1 -- 172.21.15.16:0/1261060883 wait complete. 2023-10-16T12:11:35.623 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T12:11:35.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.751+0000 7f37a0564700 1 Processor -- start 2023-10-16T12:11:35.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.751+0000 7f37a0564700 1 -- start start 2023-10-16T12:11:35.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.751+0000 7f37a0564700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3798072b30 0x7f3798071a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:35.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.752+0000 7f37a0564700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f379815c040 con 0x7f379815bb40 2023-10-16T12:11:35.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.752+0000 7f37a0564700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f379815c320 con 0x7f37980caa40 2023-10-16T12:11:35.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.752+0000 7f37a0564700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f379806ea20 con 0x7f3798072b30 2023-10-16T12:11:35.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.752+0000 7f379daff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f379815bb40 0x7f379806e310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:52208/0 (socket says 172.21.15.16:52208) 2023-10-16T12:11:35.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.752+0000 7f379daff700 1 -- 172.21.15.16:0/2954403098 learned_addr learned my addr 172.21.15.16:0/2954403098 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:35.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.752+0000 7f379e300700 1 --2- 172.21.15.16:0/2954403098 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3798072b30 0x7f3798071a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:35.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.752+0000 7f379d2fe700 1 -- 172.21.15.16:0/2954403098 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3391941868 0 0) 0x7f379815c040 con 0x7f379815bb40 2023-10-16T12:11:35.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.752+0000 7f379d2fe700 1 -- 172.21.15.16:0/2954403098 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37840035b0 con 0x7f379815bb40 2023-10-16T12:11:35.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.752+0000 7f379d2fe700 1 -- 172.21.15.16:0/2954403098 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4029030410 0 0) 0x7f379815c320 con 0x7f37980caa40 2023-10-16T12:11:35.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.752+0000 7f379d2fe700 1 -- 172.21.15.16:0/2954403098 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f379815c040 con 0x7f37980caa40 2023-10-16T12:11:35.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.752+0000 7f379d2fe700 1 -- 172.21.15.16:0/2954403098 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1157581104 0 0) 0x7f37840035b0 con 0x7f379815bb40 2023-10-16T12:11:35.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.752+0000 7f379d2fe700 1 -- 172.21.15.16:0/2954403098 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3798072b30 msgr2=0x7f3798071a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:35.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.752+0000 7f379d2fe700 1 --2- 172.21.15.16:0/2954403098 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3798072b30 0x7f3798071a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:35.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.752+0000 7f379d2fe700 1 -- 172.21.15.16:0/2954403098 >> v1:172.21.15.49:6789/0 conn(0x7f37980caa40 legacy=0x7f37980ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:35.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.752+0000 7f379d2fe700 1 -- 172.21.15.16:0/2954403098 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f379815c600 con 0x7f379815bb40 2023-10-16T12:11:35.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.752+0000 7f379d2fe700 1 -- 172.21.15.16:0/2954403098 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f3794001670 con 0x7f379815bb40 2023-10-16T12:11:35.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.752+0000 7f379d2fe700 1 -- 172.21.15.16:0/2954403098 >> v1:172.21.15.16:6789/0 conn(0x7f379815bb40 legacy=0x7f379806e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:35.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.752+0000 7f379d2fe700 1 --2- 172.21.15.16:0/2954403098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3784003bc0 0x7f3784003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:35.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.752+0000 7f379d2fe700 1 --2- 172.21.15.16:0/2954403098 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3784004850 0x7f3784004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:35.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.753+0000 7f379d2fe700 1 --2- 172.21.15.16:0/2954403098 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3798072b30 0x7f3784005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:35.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.753+0000 7f379d2fe700 1 -- 172.21.15.16:0/2954403098 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f379806ea20 con 0x7f3784003bc0 2023-10-16T12:11:35.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.753+0000 7f379d2fe700 1 -- 172.21.15.16:0/2954403098 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f378400bac0 con 0x7f3784004850 2023-10-16T12:11:35.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.753+0000 7f379d2fe700 1 -- 172.21.15.16:0/2954403098 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f378400bc80 con 0x7f3798072b30 2023-10-16T12:11:35.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.753+0000 7f379e300700 1 --2- 172.21.15.16:0/2954403098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3784003bc0 0x7f3784003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:35.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.753+0000 7f379daff700 1 --2- 172.21.15.16:0/2954403098 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3784004850 0x7f3784004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:35.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.753+0000 7f379eb01700 1 --2- 172.21.15.16:0/2954403098 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3798072b30 0x7f3784005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:35.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.753+0000 7f379e300700 1 -- 172.21.15.16:0/2954403098 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3798072b30 msgr2=0x7f3784005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:35.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.753+0000 7f379e300700 1 --2- 172.21.15.16:0/2954403098 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3798072b30 0x7f3784005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:35.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.753+0000 7f379e300700 1 -- 172.21.15.16:0/2954403098 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3784004850 msgr2=0x7f3784004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:35.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.753+0000 7f379e300700 1 --2- 172.21.15.16:0/2954403098 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3784004850 0x7f3784004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:35.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.753+0000 7f379e300700 1 -- 172.21.15.16:0/2954403098 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37840035b0 con 0x7f3784003bc0 2023-10-16T12:11:35.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.753+0000 7f379daff700 1 --2- 172.21.15.16:0/2954403098 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3784004850 0x7f3784004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:11:35.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.753+0000 7f379e300700 1 --2- 172.21.15.16:0/2954403098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3784003bc0 0x7f3784003fc0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f378c009b60 tx=0x7f3784005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b85020ecc80595c2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:35.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.753+0000 7f379d2fe700 1 -- 172.21.15.16:0/2954403098 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f378c00e6c0 con 0x7f3784003bc0 2023-10-16T12:11:35.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.753+0000 7f379d2fe700 1 -- 172.21.15.16:0/2954403098 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f378c00e820 con 0x7f3784003bc0 2023-10-16T12:11:35.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.753+0000 7f379d2fe700 1 -- 172.21.15.16:0/2954403098 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f378c00eb90 con 0x7f3784003bc0 2023-10-16T12:11:35.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.754+0000 7f37a0564700 1 -- 172.21.15.16:0/2954403098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3784003bc0 msgr2=0x7f3784003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:35.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.754+0000 7f37a0564700 1 --2- 172.21.15.16:0/2954403098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3784003bc0 0x7f3784003fc0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f378c009b60 tx=0x7f3784005880 comp rx=0 tx=0).stop 2023-10-16T12:11:35.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.754+0000 7f379eb01700 1 -- 172.21.15.16:0/2954403098 reap_dead start 2023-10-16T12:11:35.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.754+0000 7f37a0564700 1 -- 172.21.15.16:0/2954403098 shutdown_connections 2023-10-16T12:11:35.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.754+0000 7f37a0564700 1 -- 172.21.15.16:0/2954403098 >> 172.21.15.16:0/2954403098 conn(0x7f37980b1f40 msgr2=0x7f37980b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:35.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.754+0000 7f37a0564700 1 -- 172.21.15.16:0/2954403098 shutdown_connections 2023-10-16T12:11:35.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.754+0000 7f37a0564700 1 -- 172.21.15.16:0/2954403098 wait complete. 2023-10-16T12:11:35.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.755+0000 7f37a0564700 1 Processor -- start 2023-10-16T12:11:35.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.755+0000 7f37a0564700 1 -- start start 2023-10-16T12:11:35.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.756+0000 7f37a0564700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3784003bc0 0x7f37981f2730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:35.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.756+0000 7f37a0564700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f37980cdfc0 0x7f37980c5180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:35.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.756+0000 7f37a0564700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37980ce440 0x7f37980c56c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:35.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.756+0000 7f37a0564700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37980cef60 con 0x7f37980ce440 2023-10-16T12:11:35.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.756+0000 7f37a0564700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37980ce9a0 con 0x7f3784003bc0 2023-10-16T12:11:35.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.756+0000 7f37a0564700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37980cec80 con 0x7f37980cdfc0 2023-10-16T12:11:35.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.756+0000 7f379eb01700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37980ce440 0x7f37980c56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:35.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.756+0000 7f379eb01700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37980ce440 0x7f37980c56c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34714/0 (socket says 172.21.15.16:34714) 2023-10-16T12:11:35.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.756+0000 7f379eb01700 1 -- 172.21.15.16:0/427646578 learned_addr learned my addr 172.21.15.16:0/427646578 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:35.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.756+0000 7f379e300700 1 --2- 172.21.15.16:0/427646578 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3784003bc0 0x7f37981f2730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:35.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.756+0000 7f379daff700 1 --2- 172.21.15.16:0/427646578 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f37980cdfc0 0x7f37980c5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:35.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.756+0000 7f379eb01700 1 -- 172.21.15.16:0/427646578 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f37980cdfc0 msgr2=0x7f37980c5180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:35.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.756+0000 7f379eb01700 1 --2- 172.21.15.16:0/427646578 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f37980cdfc0 0x7f37980c5180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:35.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.756+0000 7f379eb01700 1 -- 172.21.15.16:0/427646578 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3784003bc0 msgr2=0x7f37981f2730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:35.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.756+0000 7f379eb01700 1 --2- 172.21.15.16:0/427646578 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3784003bc0 0x7f37981f2730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:35.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.756+0000 7f379eb01700 1 -- 172.21.15.16:0/427646578 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3798152af0 con 0x7f37980ce440 2023-10-16T12:11:35.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.756+0000 7f379daff700 1 --2- 172.21.15.16:0/427646578 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f37980cdfc0 0x7f37980c5180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:11:35.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.756+0000 7f379eb01700 1 --2- 172.21.15.16:0/427646578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37980ce440 0x7f37980c56c0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f3798070400 tx=0x7f379000ef00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:35.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.756+0000 7f378affd700 1 -- 172.21.15.16:0/427646578 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3790003c10 con 0x7f37980ce440 2023-10-16T12:11:35.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.757+0000 7f378affd700 1 -- 172.21.15.16:0/427646578 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3790003d70 con 0x7f37980ce440 2023-10-16T12:11:35.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.757+0000 7f37a0564700 1 -- 172.21.15.16:0/427646578 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3798152dd0 con 0x7f37980ce440 2023-10-16T12:11:35.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.757+0000 7f378affd700 1 -- 172.21.15.16:0/427646578 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3790005310 con 0x7f37980ce440 2023-10-16T12:11:35.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.757+0000 7f37a0564700 1 -- 172.21.15.16:0/427646578 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37981f2e50 con 0x7f37980ce440 2023-10-16T12:11:35.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.758+0000 7f37a0564700 1 -- 172.21.15.16:0/427646578 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37980735f0 con 0x7f37980ce440 2023-10-16T12:11:35.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.758+0000 7f378affd700 1 -- 172.21.15.16:0/427646578 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f3790005470 con 0x7f37980ce440 2023-10-16T12:11:35.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.758+0000 7f378affd700 1 --2- 172.21.15.16:0/427646578 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f377c06af00 0x7f377c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:35.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.759+0000 7f379e300700 1 --2- 172.21.15.16:0/427646578 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f377c06af00 0x7f377c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:35.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.759+0000 7f378affd700 1 -- 172.21.15.16:0/427646578 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f3790089910 con 0x7f37980ce440 2023-10-16T12:11:35.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.759+0000 7f379e300700 1 --2- 172.21.15.16:0/427646578 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f377c06af00 0x7f377c06d3c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f378c0090b0 tx=0x7f378c01f040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:35.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.763+0000 7f378affd700 1 -- 172.21.15.16:0/427646578 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3790054690 con 0x7f37980ce440 2023-10-16T12:11:35.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.966+0000 7f37a0564700 1 -- 172.21.15.16:0/427646578 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f379804ebb0 con 0x7f37980ce440 2023-10-16T12:11:35.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.966+0000 7f378affd700 1 -- 172.21.15.16:0/427646578 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3790057cb0 con 0x7f37980ce440 2023-10-16T12:11:35.969 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-10-16T12:11:35.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.968+0000 7f37a0564700 1 -- 172.21.15.16:0/427646578 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f377c06af00 msgr2=0x7f377c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:35.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.968+0000 7f37a0564700 1 --2- 172.21.15.16:0/427646578 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f377c06af00 0x7f377c06d3c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f378c0090b0 tx=0x7f378c01f040 comp rx=0 tx=0).stop 2023-10-16T12:11:35.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.968+0000 7f37a0564700 1 -- 172.21.15.16:0/427646578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37980ce440 msgr2=0x7f37980c56c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:35.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.968+0000 7f37a0564700 1 --2- 172.21.15.16:0/427646578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37980ce440 0x7f37980c56c0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f3798070400 tx=0x7f379000ef00 comp rx=0 tx=0).stop 2023-10-16T12:11:35.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.968+0000 7f37a0564700 1 -- 172.21.15.16:0/427646578 shutdown_connections 2023-10-16T12:11:35.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.968+0000 7f37a0564700 1 --2- 172.21.15.16:0/427646578 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f377c06af00 0x7f377c06d3c0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:35.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.968+0000 7f37a0564700 1 --2- 172.21.15.16:0/427646578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37980ce440 0x7f37980c56c0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:35.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.968+0000 7f37a0564700 1 --2- 172.21.15.16:0/427646578 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3784003bc0 0x7f37981f2730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:35.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.968+0000 7f37a0564700 1 --2- 172.21.15.16:0/427646578 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f37980cdfc0 0x7f37980c5180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:35.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.969+0000 7f37a0564700 1 -- 172.21.15.16:0/427646578 >> 172.21.15.16:0/427646578 conn(0x7f37980b1f40 msgr2=0x7f379806dea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:35.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.969+0000 7f37a0564700 1 -- 172.21.15.16:0/427646578 shutdown_connections 2023-10-16T12:11:35.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:35.969+0000 7f37a0564700 1 -- 172.21.15.16:0/427646578 wait complete. 2023-10-16T12:11:35.983 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T12:11:36.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.109+0000 7f127cfd8700 1 Processor -- start 2023-10-16T12:11:36.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.109+0000 7f127cfd8700 1 -- start start 2023-10-16T12:11:36.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.110+0000 7f127cfd8700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f127806e570 0x7f127815e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:36.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.110+0000 7f127cfd8700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f127806a7e0 0x7f127806ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:36.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.110+0000 7f127cfd8700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f127815b6a0 con 0x7f127815b2c0 2023-10-16T12:11:36.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.110+0000 7f127cfd8700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f127806b1b0 con 0x7f127806a7e0 2023-10-16T12:11:36.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.110+0000 7f127cfd8700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1278161b20 con 0x7f127806e570 2023-10-16T12:11:36.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.110+0000 7f1276d9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f127806a7e0 0x7f127806ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:36.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.110+0000 7f127659c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f127815b2c0 0x7f1278161340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:52224/0 (socket says 172.21.15.16:52224) 2023-10-16T12:11:36.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.110+0000 7f127659c700 1 -- 172.21.15.16:0/3770626318 learned_addr learned my addr 172.21.15.16:0/3770626318 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:36.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.111+0000 7f1275d9b700 1 --2- 172.21.15.16:0/3770626318 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f127806e570 0x7f127815e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:36.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.111+0000 7f1276d9d700 1 -- 172.21.15.16:0/3770626318 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f127806e570 msgr2=0x7f127815e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:36.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.111+0000 7f1276d9d700 1 --2- 172.21.15.16:0/3770626318 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f127806e570 0x7f127815e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:36.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.111+0000 7f1276d9d700 1 -- 172.21.15.16:0/3770626318 >> v1:172.21.15.16:6789/0 conn(0x7f127815b2c0 legacy=0x7f1278161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:36.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.111+0000 7f1276d9d700 1 -- 172.21.15.16:0/3770626318 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1278161e00 con 0x7f127806a7e0 2023-10-16T12:11:36.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.111+0000 7f127559a700 1 -- 172.21.15.16:0/3770626318 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3590119277 0 0) 0x7f127815b6a0 con 0x7f127815b2c0 2023-10-16T12:11:36.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.111+0000 7f1276d9d700 1 --2- 172.21.15.16:0/3770626318 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f127806a7e0 0x7f127806ac70 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f1270009d90 tx=0x7f127000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd98b52df2debf8f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:36.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.111+0000 7f127559a700 1 -- 172.21.15.16:0/3770626318 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f127000e070 con 0x7f127806a7e0 2023-10-16T12:11:36.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.111+0000 7f127559a700 1 -- 172.21.15.16:0/3770626318 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f127000bbd0 con 0x7f127806a7e0 2023-10-16T12:11:36.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.111+0000 7f127559a700 1 -- 172.21.15.16:0/3770626318 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1270012430 con 0x7f127806a7e0 2023-10-16T12:11:36.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.112+0000 7f127cfd8700 1 -- 172.21.15.16:0/3770626318 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f127806a7e0 msgr2=0x7f127806ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:36.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.112+0000 7f127cfd8700 1 --2- 172.21.15.16:0/3770626318 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f127806a7e0 0x7f127806ac70 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f1270009d90 tx=0x7f127000b330 comp rx=0 tx=0).stop 2023-10-16T12:11:36.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.112+0000 7f127cfd8700 1 -- 172.21.15.16:0/3770626318 shutdown_connections 2023-10-16T12:11:36.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.112+0000 7f127cfd8700 1 --2- 172.21.15.16:0/3770626318 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f127806a7e0 0x7f127806ac70 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:36.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.112+0000 7f127cfd8700 1 --2- 172.21.15.16:0/3770626318 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f127806e570 0x7f127815e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:36.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.112+0000 7f127cfd8700 1 -- 172.21.15.16:0/3770626318 >> 172.21.15.16:0/3770626318 conn(0x7f12780b1f40 msgr2=0x7f12780b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:36.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.112+0000 7f127cfd8700 1 -- 172.21.15.16:0/3770626318 shutdown_connections 2023-10-16T12:11:36.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.112+0000 7f127cfd8700 1 -- 172.21.15.16:0/3770626318 wait complete. 2023-10-16T12:11:36.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.113+0000 7f127cfd8700 1 Processor -- start 2023-10-16T12:11:36.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.113+0000 7f127cfd8700 1 -- start start 2023-10-16T12:11:36.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.114+0000 7f127cfd8700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f127806a7e0 0x7f127815ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:36.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.114+0000 7f127cfd8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f127806e570 0x7f127814f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:36.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.114+0000 7f127cfd8700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f127815b2c0 0x7f1278159230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:36.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.114+0000 7f127cfd8700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f127815e4a0 con 0x7f127806e570 2023-10-16T12:11:36.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.114+0000 7f127cfd8700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1278160840 con 0x7f127815b2c0 2023-10-16T12:11:36.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.114+0000 7f127cfd8700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1278160b20 con 0x7f127806a7e0 2023-10-16T12:11:36.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.114+0000 7f1275d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f127806e570 0x7f127814f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:36.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.114+0000 7f1275d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f127806e570 0x7f127814f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34726/0 (socket says 172.21.15.16:34726) 2023-10-16T12:11:36.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.114+0000 7f1275d9b700 1 -- 172.21.15.16:0/3913140305 learned_addr learned my addr 172.21.15.16:0/3913140305 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:36.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.114+0000 7f127659c700 1 --2- 172.21.15.16:0/3913140305 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f127806a7e0 0x7f127815ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:36.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.114+0000 7f1276d9d700 1 --2- 172.21.15.16:0/3913140305 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f127815b2c0 0x7f1278159230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:36.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.114+0000 7f1275d9b700 1 -- 172.21.15.16:0/3913140305 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f127806a7e0 msgr2=0x7f127815ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:36.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.114+0000 7f1275d9b700 1 --2- 172.21.15.16:0/3913140305 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f127806a7e0 0x7f127815ffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:36.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.114+0000 7f1275d9b700 1 -- 172.21.15.16:0/3913140305 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f127815b2c0 msgr2=0x7f1278159230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:36.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.114+0000 7f1275d9b700 1 --2- 172.21.15.16:0/3913140305 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f127815b2c0 0x7f1278159230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:36.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.114+0000 7f1275d9b700 1 -- 172.21.15.16:0/3913140305 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1270009a70 con 0x7f127806e570 2023-10-16T12:11:36.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.114+0000 7f1276d9d700 1 --2- 172.21.15.16:0/3913140305 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f127815b2c0 0x7f1278159230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:11:36.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.114+0000 7f1275d9b700 1 --2- 172.21.15.16:0/3913140305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f127806e570 0x7f127814f880 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f126800c7f0 tx=0x7f126800cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:36.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.115+0000 7f1266ffd700 1 -- 172.21.15.16:0/3913140305 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1268004220 con 0x7f127806e570 2023-10-16T12:11:36.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.115+0000 7f1266ffd700 1 -- 172.21.15.16:0/3913140305 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1268004380 con 0x7f127806e570 2023-10-16T12:11:36.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.115+0000 7f1266ffd700 1 -- 172.21.15.16:0/3913140305 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12680036a0 con 0x7f127806e570 2023-10-16T12:11:36.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.115+0000 7f127cfd8700 1 -- 172.21.15.16:0/3913140305 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1278160e00 con 0x7f127806e570 2023-10-16T12:11:36.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.115+0000 7f127cfd8700 1 -- 172.21.15.16:0/3913140305 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12781604f0 con 0x7f127806e570 2023-10-16T12:11:36.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.116+0000 7f127cfd8700 1 -- 172.21.15.16:0/3913140305 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f127804ebb0 con 0x7f127806e570 2023-10-16T12:11:36.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.116+0000 7f1266ffd700 1 -- 172.21.15.16:0/3913140305 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f1268003800 con 0x7f127806e570 2023-10-16T12:11:36.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.117+0000 7f1266ffd700 1 --2- 172.21.15.16:0/3913140305 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f125806af00 0x7f125806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:36.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.117+0000 7f127659c700 1 --2- 172.21.15.16:0/3913140305 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f125806af00 0x7f125806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:36.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.117+0000 7f1266ffd700 1 -- 172.21.15.16:0/3913140305 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f1268089500 con 0x7f127806e570 2023-10-16T12:11:36.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.117+0000 7f127659c700 1 --2- 172.21.15.16:0/3913140305 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f125806af00 0x7f125806d3c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f126c007fd0 tx=0x7f126c007de0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:36.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.121+0000 7f1266ffd700 1 -- 172.21.15.16:0/3913140305 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1268054200 con 0x7f127806e570 2023-10-16T12:11:36.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.322+0000 7f127cfd8700 1 -- 172.21.15.16:0/3913140305 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f12780b6370 con 0x7f127806e570 2023-10-16T12:11:36.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.322+0000 7f1266ffd700 1 -- 172.21.15.16:0/3913140305 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1268057820 con 0x7f127806e570 2023-10-16T12:11:36.325 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-10-16T12:11:36.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.324+0000 7f127cfd8700 1 -- 172.21.15.16:0/3913140305 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f125806af00 msgr2=0x7f125806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:36.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.324+0000 7f127cfd8700 1 --2- 172.21.15.16:0/3913140305 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f125806af00 0x7f125806d3c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f126c007fd0 tx=0x7f126c007de0 comp rx=0 tx=0).stop 2023-10-16T12:11:36.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.324+0000 7f127cfd8700 1 -- 172.21.15.16:0/3913140305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f127806e570 msgr2=0x7f127814f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:36.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.324+0000 7f127cfd8700 1 --2- 172.21.15.16:0/3913140305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f127806e570 0x7f127814f880 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f126800c7f0 tx=0x7f126800cb00 comp rx=0 tx=0).stop 2023-10-16T12:11:36.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.324+0000 7f127cfd8700 1 -- 172.21.15.16:0/3913140305 shutdown_connections 2023-10-16T12:11:36.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.324+0000 7f127cfd8700 1 --2- 172.21.15.16:0/3913140305 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f125806af00 0x7f125806d3c0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:36.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.324+0000 7f127cfd8700 1 --2- 172.21.15.16:0/3913140305 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f127815b2c0 0x7f1278159230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:36.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.324+0000 7f127cfd8700 1 --2- 172.21.15.16:0/3913140305 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f127806a7e0 0x7f127815ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:36.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.324+0000 7f127cfd8700 1 --2- 172.21.15.16:0/3913140305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f127806e570 0x7f127814f880 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:36.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.325+0000 7f127cfd8700 1 -- 172.21.15.16:0/3913140305 >> 172.21.15.16:0/3913140305 conn(0x7f12780b1f40 msgr2=0x7f127815b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:36.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.325+0000 7f127cfd8700 1 -- 172.21.15.16:0/3913140305 shutdown_connections 2023-10-16T12:11:36.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.325+0000 7f127cfd8700 1 -- 172.21.15.16:0/3913140305 wait complete. 2023-10-16T12:11:36.338 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-16T12:11:36.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.465+0000 7f4f53d04700 1 Processor -- start 2023-10-16T12:11:36.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.465+0000 7f4f53d04700 1 -- start start 2023-10-16T12:11:36.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.466+0000 7f4f53d04700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f4f4c15b070 0x7f4f4c15b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:36.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.466+0000 7f4f53d04700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4f4c06e170 0x7f4f4c0cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:36.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.466+0000 7f4f53d04700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f4c0cef70 con 0x7f4f4c0cb880 2023-10-16T12:11:36.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.466+0000 7f4f53d04700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f4c0ce360 con 0x7f4f4c06e170 2023-10-16T12:11:36.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.466+0000 7f4f53d04700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f4c0ce640 con 0x7f4f4c15b070 2023-10-16T12:11:36.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.467+0000 7f4f522a1700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4f4c06e170 0x7f4f4c0cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:36.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.467+0000 7f4f5129f700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f4f4c0cb880 0x7f4f4c06da60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:52238/0 (socket says 172.21.15.16:52238) 2023-10-16T12:11:36.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.467+0000 7f4f5129f700 1 -- 172.21.15.16:0/1130160737 learned_addr learned my addr 172.21.15.16:0/1130160737 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:36.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.467+0000 7f4f51aa0700 1 --2- 172.21.15.16:0/1130160737 >> v2:172.21.15.49:3300/0 conn(0x7f4f4c15b070 0x7f4f4c15b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:36.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.467+0000 7f4f522a1700 1 -- 172.21.15.16:0/1130160737 >> v2:172.21.15.49:3300/0 conn(0x7f4f4c15b070 msgr2=0x7f4f4c15b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:36.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.467+0000 7f4f522a1700 1 --2- 172.21.15.16:0/1130160737 >> v2:172.21.15.49:3300/0 conn(0x7f4f4c15b070 0x7f4f4c15b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:36.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.467+0000 7f4f522a1700 1 -- 172.21.15.16:0/1130160737 >> v1:172.21.15.16:6789/0 conn(0x7f4f4c0cb880 legacy=0x7f4f4c06da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:36.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.467+0000 7f4f50a9e700 1 -- 172.21.15.16:0/1130160737 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2218004656 0 0) 0x7f4f4c0cef70 con 0x7f4f4c0cb880 2023-10-16T12:11:36.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.467+0000 7f4f522a1700 1 -- 172.21.15.16:0/1130160737 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f4c0ce920 con 0x7f4f4c06e170 2023-10-16T12:11:36.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.467+0000 7f4f522a1700 1 --2- 172.21.15.16:0/1130160737 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f4c06e170 0x7f4f4c0cbd00 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f4f4c0705a0 tx=0x7f4f4400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90b633d111314d65 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:36.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.467+0000 7f4f50a9e700 1 -- 172.21.15.16:0/1130160737 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f4400e070 con 0x7f4f4c06e170 2023-10-16T12:11:36.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.467+0000 7f4f50a9e700 1 -- 172.21.15.16:0/1130160737 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f4400bbd0 con 0x7f4f4c06e170 2023-10-16T12:11:36.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.467+0000 7f4f50a9e700 1 -- 172.21.15.16:0/1130160737 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f44012430 con 0x7f4f4c06e170 2023-10-16T12:11:36.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.468+0000 7f4f53d04700 1 -- 172.21.15.16:0/1130160737 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f4c06e170 msgr2=0x7f4f4c0cbd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:36.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.468+0000 7f4f53d04700 1 --2- 172.21.15.16:0/1130160737 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f4c06e170 0x7f4f4c0cbd00 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f4f4c0705a0 tx=0x7f4f4400b330 comp rx=0 tx=0).stop 2023-10-16T12:11:36.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.468+0000 7f4f53d04700 1 -- 172.21.15.16:0/1130160737 shutdown_connections 2023-10-16T12:11:36.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.468+0000 7f4f53d04700 1 --2- 172.21.15.16:0/1130160737 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f4c06e170 0x7f4f4c0cbd00 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:36.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.468+0000 7f4f53d04700 1 --2- 172.21.15.16:0/1130160737 >> v2:172.21.15.49:3300/0 conn(0x7f4f4c15b070 0x7f4f4c15b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:36.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.468+0000 7f4f53d04700 1 -- 172.21.15.16:0/1130160737 >> 172.21.15.16:0/1130160737 conn(0x7f4f4c0b1f40 msgr2=0x7f4f4c0b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:36.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.468+0000 7f4f53d04700 1 -- 172.21.15.16:0/1130160737 shutdown_connections 2023-10-16T12:11:36.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.468+0000 7f4f53d04700 1 -- 172.21.15.16:0/1130160737 wait complete. 2023-10-16T12:11:36.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.469+0000 7f4f53d04700 1 Processor -- start 2023-10-16T12:11:36.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.469+0000 7f4f53d04700 1 -- start start 2023-10-16T12:11:36.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.470+0000 7f4f53d04700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f4c06e170 0x7f4f4c15ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:36.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.470+0000 7f4f53d04700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4f4c0cb880 0x7f4f4c15a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:36.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.470+0000 7f4f53d04700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4f4c15b070 0x7f4f4c152230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:36.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.470+0000 7f4f53d04700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f4c0ad730 con 0x7f4f4c06e170 2023-10-16T12:11:36.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.470+0000 7f4f53d04700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f4c1591e0 con 0x7f4f4c0cb880 2023-10-16T12:11:36.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.470+0000 7f4f53d04700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f4c1594c0 con 0x7f4f4c15b070 2023-10-16T12:11:36.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.470+0000 7f4f51aa0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f4c06e170 0x7f4f4c15ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:36.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.470+0000 7f4f51aa0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f4c06e170 0x7f4f4c15ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34738/0 (socket says 172.21.15.16:34738) 2023-10-16T12:11:36.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.470+0000 7f4f51aa0700 1 -- 172.21.15.16:0/763470749 learned_addr learned my addr 172.21.15.16:0/763470749 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:36.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.470+0000 7f4f5129f700 1 --2- 172.21.15.16:0/763470749 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4f4c0cb880 0x7f4f4c15a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:36.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.470+0000 7f4f522a1700 1 --2- 172.21.15.16:0/763470749 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4f4c15b070 0x7f4f4c152230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:36.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.470+0000 7f4f51aa0700 1 -- 172.21.15.16:0/763470749 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4f4c15b070 msgr2=0x7f4f4c152230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:36.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.470+0000 7f4f51aa0700 1 --2- 172.21.15.16:0/763470749 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4f4c15b070 0x7f4f4c152230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:36.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.470+0000 7f4f51aa0700 1 -- 172.21.15.16:0/763470749 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4f4c0cb880 msgr2=0x7f4f4c15a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:36.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.470+0000 7f4f51aa0700 1 --2- 172.21.15.16:0/763470749 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4f4c0cb880 0x7f4f4c15a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:36.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.470+0000 7f4f51aa0700 1 -- 172.21.15.16:0/763470749 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f44009a70 con 0x7f4f4c06e170 2023-10-16T12:11:36.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.470+0000 7f4f522a1700 1 --2- 172.21.15.16:0/763470749 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4f4c15b070 0x7f4f4c152230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:11:36.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.470+0000 7f4f51aa0700 1 --2- 172.21.15.16:0/763470749 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f4c06e170 0x7f4f4c15ebc0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f4f3c00d800 tx=0x7f4f3c00db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:36.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.471+0000 7f4f427fc700 1 -- 172.21.15.16:0/763470749 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f3c0125f0 con 0x7f4f4c06e170 2023-10-16T12:11:36.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.471+0000 7f4f427fc700 1 -- 172.21.15.16:0/763470749 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f3c012750 con 0x7f4f4c06e170 2023-10-16T12:11:36.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.471+0000 7f4f53d04700 1 -- 172.21.15.16:0/763470749 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f4c1597a0 con 0x7f4f4c06e170 2023-10-16T12:11:36.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.471+0000 7f4f427fc700 1 -- 172.21.15.16:0/763470749 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f3c012b00 con 0x7f4f4c06e170 2023-10-16T12:11:36.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.471+0000 7f4f53d04700 1 -- 172.21.15.16:0/763470749 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f4c0b3030 con 0x7f4f4c06e170 2023-10-16T12:11:36.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.472+0000 7f4f53d04700 1 -- 172.21.15.16:0/763470749 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f4c15b450 con 0x7f4f4c06e170 2023-10-16T12:11:36.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.472+0000 7f4f427fc700 1 -- 172.21.15.16:0/763470749 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f4f3c012c60 con 0x7f4f4c06e170 2023-10-16T12:11:36.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.472+0000 7f4f427fc700 1 --2- 172.21.15.16:0/763470749 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f4f3006af30 0x7f4f3006d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:36.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.473+0000 7f4f427fc700 1 -- 172.21.15.16:0/763470749 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f4f3c089c50 con 0x7f4f4c06e170 2023-10-16T12:11:36.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.473+0000 7f4f5129f700 1 --2- 172.21.15.16:0/763470749 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f4f3006af30 0x7f4f3006d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:36.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.473+0000 7f4f5129f700 1 --2- 172.21.15.16:0/763470749 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f4f3006af30 0x7f4f3006d3f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4f4c1f72c0 tx=0x7f4f48007de0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:36.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.477+0000 7f4f427fc700 1 -- 172.21.15.16:0/763470749 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4f3c08f020 con 0x7f4f4c06e170 2023-10-16T12:11:36.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.676+0000 7f4f53d04700 1 -- 172.21.15.16:0/763470749 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f4f4c159990 con 0x7f4f4c06e170 2023-10-16T12:11:36.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.677+0000 7f4f427fc700 1 -- 172.21.15.16:0/763470749 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f4f3c054940 con 0x7f4f4c06e170 2023-10-16T12:11:36.679 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-10-16T12:11:36.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.679+0000 7f4f53d04700 1 -- 172.21.15.16:0/763470749 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f4f3006af30 msgr2=0x7f4f3006d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:36.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.679+0000 7f4f53d04700 1 --2- 172.21.15.16:0/763470749 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f4f3006af30 0x7f4f3006d3f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4f4c1f72c0 tx=0x7f4f48007de0 comp rx=0 tx=0).stop 2023-10-16T12:11:36.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.679+0000 7f4f53d04700 1 -- 172.21.15.16:0/763470749 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f4c06e170 msgr2=0x7f4f4c15ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:36.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.679+0000 7f4f53d04700 1 --2- 172.21.15.16:0/763470749 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f4c06e170 0x7f4f4c15ebc0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f4f3c00d800 tx=0x7f4f3c00db10 comp rx=0 tx=0).stop 2023-10-16T12:11:36.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.679+0000 7f4f53d04700 1 -- 172.21.15.16:0/763470749 shutdown_connections 2023-10-16T12:11:36.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.679+0000 7f4f53d04700 1 --2- 172.21.15.16:0/763470749 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f4f3006af30 0x7f4f3006d3f0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:36.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.679+0000 7f4f53d04700 1 --2- 172.21.15.16:0/763470749 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4f4c15b070 0x7f4f4c152230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:36.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.679+0000 7f4f53d04700 1 --2- 172.21.15.16:0/763470749 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f4c06e170 0x7f4f4c15ebc0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:36.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.679+0000 7f4f53d04700 1 --2- 172.21.15.16:0/763470749 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4f4c0cb880 0x7f4f4c15a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:36.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.679+0000 7f4f53d04700 1 -- 172.21.15.16:0/763470749 >> 172.21.15.16:0/763470749 conn(0x7f4f4c0b1f40 msgr2=0x7f4f4c0b14e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:36.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.679+0000 7f4f53d04700 1 -- 172.21.15.16:0/763470749 shutdown_connections 2023-10-16T12:11:36.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.679+0000 7f4f53d04700 1 -- 172.21.15.16:0/763470749 wait complete. 2023-10-16T12:11:36.693 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T12:11:36.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.817+0000 7f5753094700 1 Processor -- start 2023-10-16T12:11:36.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.817+0000 7f5753094700 1 -- start start 2023-10-16T12:11:36.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.818+0000 7f5753094700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f574c0cf4a0 0x7f574c0726e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:36.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.818+0000 7f5753094700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f574c15bc80 con 0x7f574c0cf880 2023-10-16T12:11:36.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.818+0000 7f5753094700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f574c0b5e70 con 0x7f574c0b5a00 2023-10-16T12:11:36.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.818+0000 7f5753094700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f574c072c20 con 0x7f574c0cf4a0 2023-10-16T12:11:36.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.818+0000 7f5750e30700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f574c0cf4a0 0x7f574c0726e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:36.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.818+0000 7f574bfff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f574c0cf880 0x7f574c0b52f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:52254/0 (socket says 172.21.15.16:52254) 2023-10-16T12:11:36.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.818+0000 7f574bfff700 1 -- 172.21.15.16:0/1396579594 learned_addr learned my addr 172.21.15.16:0/1396579594 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:36.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.819+0000 7f5750e30700 1 -- 172.21.15.16:0/1396579594 >> v1:172.21.15.49:6789/0 conn(0x7f574c0b5a00 legacy=0x7f574c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:36.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.819+0000 7f5750e30700 1 -- 172.21.15.16:0/1396579594 >> v1:172.21.15.16:6789/0 conn(0x7f574c0cf880 legacy=0x7f574c0b52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:36.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.819+0000 7f574b7fe700 1 -- 172.21.15.16:0/1396579594 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3584430682 0 0) 0x7f574c15bc80 con 0x7f574c0cf880 2023-10-16T12:11:36.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.819+0000 7f5750e30700 1 -- 172.21.15.16:0/1396579594 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f574c15b070 con 0x7f574c0cf4a0 2023-10-16T12:11:36.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.819+0000 7f5750e30700 1 --2- 172.21.15.16:0/1396579594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f574c0cf4a0 0x7f574c0726e0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f5740009840 tx=0x7f5740009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ca0effee156a4cb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:36.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.819+0000 7f574b7fe700 1 -- 172.21.15.16:0/1396579594 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5740004230 con 0x7f574c0cf4a0 2023-10-16T12:11:36.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.819+0000 7f574b7fe700 1 -- 172.21.15.16:0/1396579594 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5740004390 con 0x7f574c0cf4a0 2023-10-16T12:11:36.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.819+0000 7f574b7fe700 1 -- 172.21.15.16:0/1396579594 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57400036a0 con 0x7f574c0cf4a0 2023-10-16T12:11:36.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.820+0000 7f5753094700 1 -- 172.21.15.16:0/1396579594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f574c0cf4a0 msgr2=0x7f574c0726e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:36.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.820+0000 7f5753094700 1 --2- 172.21.15.16:0/1396579594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f574c0cf4a0 0x7f574c0726e0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f5740009840 tx=0x7f5740009b50 comp rx=0 tx=0).stop 2023-10-16T12:11:36.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.820+0000 7f5753094700 1 -- 172.21.15.16:0/1396579594 shutdown_connections 2023-10-16T12:11:36.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.820+0000 7f5753094700 1 --2- 172.21.15.16:0/1396579594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f574c0cf4a0 0x7f574c0726e0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:36.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.820+0000 7f5753094700 1 -- 172.21.15.16:0/1396579594 >> 172.21.15.16:0/1396579594 conn(0x7f574c0b1fd0 msgr2=0x7f574c0b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:36.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.820+0000 7f5753094700 1 -- 172.21.15.16:0/1396579594 shutdown_connections 2023-10-16T12:11:36.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.820+0000 7f5753094700 1 -- 172.21.15.16:0/1396579594 wait complete. 2023-10-16T12:11:36.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.821+0000 7f5753094700 1 Processor -- start 2023-10-16T12:11:36.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.821+0000 7f5753094700 1 -- start start 2023-10-16T12:11:36.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.822+0000 7f5753094700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f574c0b5a00 0x7f574c0cba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:36.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.822+0000 7f5753094700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f574c0cf4a0 0x7f574c14e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:36.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.822+0000 7f5753094700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f574c0cf880 0x7f574c0cd2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:36.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.822+0000 7f5753094700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f574c0cbf80 con 0x7f574c0cf880 2023-10-16T12:11:36.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.822+0000 7f5753094700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f574c0cc220 con 0x7f574c0b5a00 2023-10-16T12:11:36.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.822+0000 7f5753094700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f574c0cc500 con 0x7f574c0cf4a0 2023-10-16T12:11:36.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.822+0000 7f5751631700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f574c0cf880 0x7f574c0cd2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:36.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.822+0000 7f5751631700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f574c0cf880 0x7f574c0cd2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34756/0 (socket says 172.21.15.16:34756) 2023-10-16T12:11:36.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.822+0000 7f5751631700 1 -- 172.21.15.16:0/2877141408 learned_addr learned my addr 172.21.15.16:0/2877141408 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:36.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.822+0000 7f5750e30700 1 --2- 172.21.15.16:0/2877141408 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f574c0b5a00 0x7f574c0cba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:36.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.822+0000 7f574bfff700 1 --2- 172.21.15.16:0/2877141408 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f574c0cf4a0 0x7f574c14e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:36.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.822+0000 7f5751631700 1 -- 172.21.15.16:0/2877141408 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f574c0cf4a0 msgr2=0x7f574c14e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:36.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.822+0000 7f5751631700 1 --2- 172.21.15.16:0/2877141408 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f574c0cf4a0 0x7f574c14e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:36.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.822+0000 7f5751631700 1 -- 172.21.15.16:0/2877141408 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f574c0b5a00 msgr2=0x7f574c0cba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:36.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.822+0000 7f5751631700 1 --2- 172.21.15.16:0/2877141408 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f574c0b5a00 0x7f574c0cba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:36.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.822+0000 7f5751631700 1 -- 172.21.15.16:0/2877141408 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57400094a0 con 0x7f574c0cf880 2023-10-16T12:11:36.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.822+0000 7f5751631700 1 --2- 172.21.15.16:0/2877141408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f574c0cf880 0x7f574c0cd2c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f574400cc10 tx=0x7f574400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:36.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.823+0000 7f57497fa700 1 -- 172.21.15.16:0/2877141408 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f574400abb0 con 0x7f574c0cf880 2023-10-16T12:11:36.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.823+0000 7f57497fa700 1 -- 172.21.15.16:0/2877141408 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f574400ad10 con 0x7f574c0cf880 2023-10-16T12:11:36.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.823+0000 7f57497fa700 1 -- 172.21.15.16:0/2877141408 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5744012660 con 0x7f574c0cf880 2023-10-16T12:11:36.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.823+0000 7f5753094700 1 -- 172.21.15.16:0/2877141408 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f574c0cc7e0 con 0x7f574c0cf880 2023-10-16T12:11:36.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.823+0000 7f5753094700 1 -- 172.21.15.16:0/2877141408 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f574c0ccda0 con 0x7f574c0cf880 2023-10-16T12:11:36.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.824+0000 7f5753094700 1 -- 172.21.15.16:0/2877141408 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f574c0ce730 con 0x7f574c0cf880 2023-10-16T12:11:36.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.824+0000 7f57497fa700 1 -- 172.21.15.16:0/2877141408 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f57440127c0 con 0x7f574c0cf880 2023-10-16T12:11:36.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.824+0000 7f57497fa700 1 --2- 172.21.15.16:0/2877141408 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f573006af00 0x7f573006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:36.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.825+0000 7f57497fa700 1 -- 172.21.15.16:0/2877141408 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f574408a220 con 0x7f574c0cf880 2023-10-16T12:11:36.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.825+0000 7f5750e30700 1 --2- 172.21.15.16:0/2877141408 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f573006af00 0x7f573006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:36.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.825+0000 7f5750e30700 1 --2- 172.21.15.16:0/2877141408 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f573006af00 0x7f573006d3c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f5740000c00 tx=0x7f574001c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:36.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:36.829+0000 7f57497fa700 1 -- 172.21.15.16:0/2877141408 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5744054e80 con 0x7f574c0cf880 2023-10-16T12:11:37.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.047+0000 7f5753094700 1 -- 172.21.15.16:0/2877141408 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f574c04ebb0 con 0x7f574c0cf880 2023-10-16T12:11:37.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.048+0000 7f57497fa700 1 -- 172.21.15.16:0/2877141408 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) v1 ==== 80+0+0 (secure 0 0 0) 0x7f5744016020 con 0x7f574c0cf880 2023-10-16T12:11:37.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.050+0000 7f5753094700 1 -- 172.21.15.16:0/2877141408 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f573006af00 msgr2=0x7f573006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:37.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.050+0000 7f5753094700 1 --2- 172.21.15.16:0/2877141408 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f573006af00 0x7f573006d3c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f5740000c00 tx=0x7f574001c040 comp rx=0 tx=0).stop 2023-10-16T12:11:37.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.050+0000 7f5753094700 1 -- 172.21.15.16:0/2877141408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f574c0cf880 msgr2=0x7f574c0cd2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:37.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.050+0000 7f5753094700 1 --2- 172.21.15.16:0/2877141408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f574c0cf880 0x7f574c0cd2c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f574400cc10 tx=0x7f574400a060 comp rx=0 tx=0).stop 2023-10-16T12:11:37.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.050+0000 7f5753094700 1 -- 172.21.15.16:0/2877141408 shutdown_connections 2023-10-16T12:11:37.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.050+0000 7f5753094700 1 --2- 172.21.15.16:0/2877141408 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f573006af00 0x7f573006d3c0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:37.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.050+0000 7f5753094700 1 --2- 172.21.15.16:0/2877141408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f574c0cf880 0x7f574c0cd2c0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:37.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.050+0000 7f5753094700 1 --2- 172.21.15.16:0/2877141408 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f574c0b5a00 0x7f574c0cba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:37.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.050+0000 7f5753094700 1 --2- 172.21.15.16:0/2877141408 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f574c0cf4a0 0x7f574c14e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:37.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.050+0000 7f5753094700 1 -- 172.21.15.16:0/2877141408 >> 172.21.15.16:0/2877141408 conn(0x7f574c0b1fd0 msgr2=0x7f574c0b4fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:37.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.050+0000 7f5753094700 1 -- 172.21.15.16:0/2877141408 shutdown_connections 2023-10-16T12:11:37.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.050+0000 7f5753094700 1 -- 172.21.15.16:0/2877141408 wait complete. 2023-10-16T12:11:37.054 INFO:teuthology.orchestra.run.smithi016.stderr:listed 0 entries 2023-10-16T12:11:37.064 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-16T12:11:37.064 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:37.065 DEBUG:teuthology.orchestra.run.smithi016:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2023-10-16T12:11:37.144 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2023-10-16T12:11:37.145 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:37.145 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-16T12:11:37.162 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T12:11:37.162 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:11:37.184 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T12:11:37.185 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:11:37.245 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T12:11:37.246 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:11:37.306 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:11:37.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.430+0000 7f72d6673700 1 Processor -- start 2023-10-16T12:11:37.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.430+0000 7f72d6673700 1 -- start start 2023-10-16T12:11:37.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.431+0000 7f72d6673700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f72d015b280 0x7f72d01613e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:37.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.431+0000 7f72d6673700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f72d00cc8a0 0x7f72d015e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:37.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.431+0000 7f72d6673700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f72d006a7e0 0x7f72d006ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:37.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.431+0000 7f72d6673700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72d006b170 con 0x7f72d006a7e0 2023-10-16T12:11:37.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.431+0000 7f72d6673700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72d0161c30 con 0x7f72d00cc8a0 2023-10-16T12:11:37.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.431+0000 7f72d6673700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72d0161f10 con 0x7f72d015b280 2023-10-16T12:11:37.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.431+0000 7f72d4c10700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f72d006a7e0 0x7f72d006ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:37.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.431+0000 7f72d4c10700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f72d006a7e0 0x7f72d006ac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34764/0 (socket says 172.21.15.16:34764) 2023-10-16T12:11:37.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.431+0000 7f72d4c10700 1 -- 172.21.15.16:0/1709334944 learned_addr learned my addr 172.21.15.16:0/1709334944 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:37.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.432+0000 7f72cf7fe700 1 --2- 172.21.15.16:0/1709334944 >> v2:172.21.15.49:3300/0 conn(0x7f72d00cc8a0 0x7f72d015e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:37.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.432+0000 7f72cffff700 1 --2- 172.21.15.16:0/1709334944 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f72d015b280 0x7f72d01613e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:37.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.432+0000 7f72d4c10700 1 -- 172.21.15.16:0/1709334944 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f72d015b280 msgr2=0x7f72d01613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:37.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.432+0000 7f72d4c10700 1 --2- 172.21.15.16:0/1709334944 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f72d015b280 0x7f72d01613e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:37.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.432+0000 7f72d4c10700 1 -- 172.21.15.16:0/1709334944 >> v2:172.21.15.49:3300/0 conn(0x7f72d00cc8a0 msgr2=0x7f72d015e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:37.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.432+0000 7f72d4c10700 1 --2- 172.21.15.16:0/1709334944 >> v2:172.21.15.49:3300/0 conn(0x7f72d00cc8a0 0x7f72d015e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:37.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.432+0000 7f72d4c10700 1 -- 172.21.15.16:0/1709334944 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72d01621f0 con 0x7f72d006a7e0 2023-10-16T12:11:37.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.432+0000 7f72cf7fe700 1 --2- 172.21.15.16:0/1709334944 >> v2:172.21.15.49:3300/0 conn(0x7f72d00cc8a0 0x7f72d015e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:11:37.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.432+0000 7f72d4c10700 1 --2- 172.21.15.16:0/1709334944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72d006a7e0 0x7f72d006ac30 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f72d006ead0 tx=0x7f72c800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2be5eb6ad7ee7e42 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:37.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.432+0000 7f72ceffd700 1 -- 172.21.15.16:0/1709334944 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72c800e070 con 0x7f72d006a7e0 2023-10-16T12:11:37.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.432+0000 7f72ceffd700 1 -- 172.21.15.16:0/1709334944 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72c800b9c0 con 0x7f72d006a7e0 2023-10-16T12:11:37.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.432+0000 7f72ceffd700 1 -- 172.21.15.16:0/1709334944 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72c800bd70 con 0x7f72d006a7e0 2023-10-16T12:11:37.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.432+0000 7f72d6673700 1 -- 172.21.15.16:0/1709334944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72d006a7e0 msgr2=0x7f72d006ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:37.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.432+0000 7f72d6673700 1 --2- 172.21.15.16:0/1709334944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72d006a7e0 0x7f72d006ac30 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f72d006ead0 tx=0x7f72c800b330 comp rx=0 tx=0).stop 2023-10-16T12:11:37.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.433+0000 7f72d6673700 1 -- 172.21.15.16:0/1709334944 shutdown_connections 2023-10-16T12:11:37.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.433+0000 7f72d6673700 1 --2- 172.21.15.16:0/1709334944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72d006a7e0 0x7f72d006ac30 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:37.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.433+0000 7f72d6673700 1 --2- 172.21.15.16:0/1709334944 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f72d015b280 0x7f72d01613e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:37.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.433+0000 7f72d6673700 1 --2- 172.21.15.16:0/1709334944 >> v2:172.21.15.49:3300/0 conn(0x7f72d00cc8a0 0x7f72d015e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:37.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.433+0000 7f72d6673700 1 -- 172.21.15.16:0/1709334944 >> 172.21.15.16:0/1709334944 conn(0x7f72d00b1fd0 msgr2=0x7f72d00b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:37.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.433+0000 7f72d6673700 1 -- 172.21.15.16:0/1709334944 shutdown_connections 2023-10-16T12:11:37.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.433+0000 7f72d6673700 1 -- 172.21.15.16:0/1709334944 wait complete. 2023-10-16T12:11:37.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.434+0000 7f72d6673700 1 Processor -- start 2023-10-16T12:11:37.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.434+0000 7f72d6673700 1 -- start start 2023-10-16T12:11:37.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.435+0000 7f72d6673700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f72d006a7e0 0x7f72d00af440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:37.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.435+0000 7f72d6673700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72d00cc8a0 0x7f72d01588f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:37.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.435+0000 7f72d6673700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f72d015b280 0x7f72d0071ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:37.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.435+0000 7f72d6673700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72d00ae870 con 0x7f72d00cc8a0 2023-10-16T12:11:37.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.435+0000 7f72d6673700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72d00af980 con 0x7f72d015b280 2023-10-16T12:11:37.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.435+0000 7f72d6673700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72d00aec90 con 0x7f72d006a7e0 2023-10-16T12:11:37.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.435+0000 7f72cf7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72d00cc8a0 0x7f72d01588f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:37.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.435+0000 7f72cf7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72d00cc8a0 0x7f72d01588f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34768/0 (socket says 172.21.15.16:34768) 2023-10-16T12:11:37.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.435+0000 7f72cf7fe700 1 -- 172.21.15.16:0/1789998742 learned_addr learned my addr 172.21.15.16:0/1789998742 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:37.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.435+0000 7f72cffff700 1 --2- 172.21.15.16:0/1789998742 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f72d006a7e0 0x7f72d00af440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:37.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.435+0000 7f72d4c10700 1 --2- 172.21.15.16:0/1789998742 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f72d015b280 0x7f72d0071ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:37.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.435+0000 7f72cf7fe700 1 -- 172.21.15.16:0/1789998742 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f72d006a7e0 msgr2=0x7f72d00af440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:37.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.435+0000 7f72cf7fe700 1 --2- 172.21.15.16:0/1789998742 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f72d006a7e0 0x7f72d00af440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:37.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.435+0000 7f72cf7fe700 1 -- 172.21.15.16:0/1789998742 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f72d015b280 msgr2=0x7f72d0071ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:37.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.435+0000 7f72cf7fe700 1 --2- 172.21.15.16:0/1789998742 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f72d015b280 0x7f72d0071ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:37.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.435+0000 7f72cf7fe700 1 -- 172.21.15.16:0/1789998742 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72c8009b30 con 0x7f72d00cc8a0 2023-10-16T12:11:37.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.435+0000 7f72cf7fe700 1 --2- 172.21.15.16:0/1789998742 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72d00cc8a0 0x7f72d01588f0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f72c000c770 tx=0x7f72c000cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:37.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.436+0000 7f72ccff9700 1 -- 172.21.15.16:0/1789998742 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72c0011800 con 0x7f72d00cc8a0 2023-10-16T12:11:37.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.436+0000 7f72ccff9700 1 -- 172.21.15.16:0/1789998742 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72c0011960 con 0x7f72d00cc8a0 2023-10-16T12:11:37.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.436+0000 7f72d6673700 1 -- 172.21.15.16:0/1789998742 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72d0158e30 con 0x7f72d00cc8a0 2023-10-16T12:11:37.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.436+0000 7f72ccff9700 1 -- 172.21.15.16:0/1789998742 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72c0011cf0 con 0x7f72d00cc8a0 2023-10-16T12:11:37.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.436+0000 7f72d6673700 1 -- 172.21.15.16:0/1789998742 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72d00aded0 con 0x7f72d00cc8a0 2023-10-16T12:11:37.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.437+0000 7f72d6673700 1 -- 172.21.15.16:0/1789998742 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72d00ae9b0 con 0x7f72d00cc8a0 2023-10-16T12:11:37.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.437+0000 7f72ccff9700 1 -- 172.21.15.16:0/1789998742 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f72c0021350 con 0x7f72d00cc8a0 2023-10-16T12:11:37.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.438+0000 7f72ccff9700 1 --2- 172.21.15.16:0/1789998742 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f72b006af00 0x7f72b006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:37.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.438+0000 7f72ccff9700 1 -- 172.21.15.16:0/1789998742 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f72c00897f0 con 0x7f72d00cc8a0 2023-10-16T12:11:37.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.438+0000 7f72cffff700 1 --2- 172.21.15.16:0/1789998742 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f72b006af00 0x7f72b006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:37.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.438+0000 7f72cffff700 1 --2- 172.21.15.16:0/1789998742 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f72b006af00 0x7f72b006d3c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f72d0159ec0 tx=0x7f72b8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:37.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.442+0000 7f72ccff9700 1 -- 172.21.15.16:0/1789998742 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f72d00ae9b0 con 0x7f72d00cc8a0 2023-10-16T12:11:37.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.647+0000 7f72d6673700 1 -- 172.21.15.16:0/1789998742 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f72d00ae9b0 con 0x7f72d00cc8a0 2023-10-16T12:11:37.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.648+0000 7f72ccff9700 1 -- 172.21.15.16:0/1789998742 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18629 (secure 0 0 0) 0x7f72d00ae9b0 con 0x7f72d00cc8a0 2023-10-16T12:11:37.650 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:11:37.650 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:11:28.241865+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:10:36.935773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T12:10:38.004870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-16T12:10:41.672025+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"3.2b","mappings":[{"from":0,"to":2}]},{"pgid":"3.38","mappings":[{"from":3,"to":1}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]},{"pgid":"4.1b","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:11:37.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.650+0000 7f72d6673700 1 -- 172.21.15.16:0/1789998742 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f72b006af00 msgr2=0x7f72b006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:37.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.650+0000 7f72d6673700 1 --2- 172.21.15.16:0/1789998742 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f72b006af00 0x7f72b006d3c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f72d0159ec0 tx=0x7f72b8008040 comp rx=0 tx=0).stop 2023-10-16T12:11:37.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.650+0000 7f72d6673700 1 -- 172.21.15.16:0/1789998742 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72d00cc8a0 msgr2=0x7f72d01588f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:37.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.650+0000 7f72d6673700 1 --2- 172.21.15.16:0/1789998742 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72d00cc8a0 0x7f72d01588f0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f72c000c770 tx=0x7f72c000cb30 comp rx=0 tx=0).stop 2023-10-16T12:11:37.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.650+0000 7f72d6673700 1 -- 172.21.15.16:0/1789998742 shutdown_connections 2023-10-16T12:11:37.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.650+0000 7f72d6673700 1 --2- 172.21.15.16:0/1789998742 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f72b006af00 0x7f72b006d3c0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:37.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.650+0000 7f72d6673700 1 --2- 172.21.15.16:0/1789998742 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f72d015b280 0x7f72d0071ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:37.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.650+0000 7f72d6673700 1 --2- 172.21.15.16:0/1789998742 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f72d006a7e0 0x7f72d00af440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:37.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.650+0000 7f72d6673700 1 --2- 172.21.15.16:0/1789998742 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72d00cc8a0 0x7f72d01588f0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:37.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.650+0000 7f72d6673700 1 -- 172.21.15.16:0/1789998742 >> 172.21.15.16:0/1789998742 conn(0x7f72d00b1fd0 msgr2=0x7f72d00d0270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:37.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.650+0000 7f72d6673700 1 -- 172.21.15.16:0/1789998742 shutdown_connections 2023-10-16T12:11:37.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.650+0000 7f72d6673700 1 -- 172.21.15.16:0/1789998742 wait complete. 2023-10-16T12:11:37.664 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T12:10:36.935773+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T12:10:38.004870+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-16T12:10:41.672025+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-16T12:11:37.665 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:11:37.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.791+0000 7f0ec8925700 1 Processor -- start 2023-10-16T12:11:37.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.791+0000 7f0ec8925700 1 -- start start 2023-10-16T12:11:37.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.792+0000 7f0ec8925700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f0ec4159de0 0x7f0ec415a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:37.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.792+0000 7f0ec8925700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0ec40cf4c0 0x7f0ec406aac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:37.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.792+0000 7f0ec8925700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ec415dda0 con 0x7f0ec406b000 2023-10-16T12:11:37.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.792+0000 7f0ec8925700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ec415ea20 con 0x7f0ec4159de0 2023-10-16T12:11:37.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.792+0000 7f0ec8925700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec415e080 con 0x7f0ec40cf4c0 2023-10-16T12:11:37.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.793+0000 7f0ec1d9b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0ec40cf4c0 0x7f0ec406aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:37.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.793+0000 7f0ec259c700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f0ec4159de0 0x7f0ec415a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:37.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.793+0000 7f0ec1d9b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0ec40cf4c0 0x7f0ec406aac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:51266/0 (socket says 172.21.15.16:51266) 2023-10-16T12:11:37.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.793+0000 7f0ec259c700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f0ec4159de0 0x7f0ec415a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:33836/0 (socket says 172.21.15.16:33836) 2023-10-16T12:11:37.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.793+0000 7f0ec1d9b700 1 -- 172.21.15.16:0/1978379792 learned_addr learned my addr 172.21.15.16:0/1978379792 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:37.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.793+0000 7f0ec259c700 1 -- 172.21.15.16:0/1978379792 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0ec40cf4c0 msgr2=0x7f0ec406aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:37.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.793+0000 7f0ec259c700 1 --2- 172.21.15.16:0/1978379792 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0ec40cf4c0 0x7f0ec406aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:37.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.793+0000 7f0ec259c700 1 -- 172.21.15.16:0/1978379792 >> v1:172.21.15.49:6789/0 conn(0x7f0ec406b000 legacy=0x7f0ec406b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:37.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.793+0000 7f0ec159a700 1 -- 172.21.15.16:0/1978379792 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1737611721 0 0) 0x7f0ec415dda0 con 0x7f0ec406b000 2023-10-16T12:11:37.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.793+0000 7f0ec259c700 1 -- 172.21.15.16:0/1978379792 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ec415e360 con 0x7f0ec4159de0 2023-10-16T12:11:37.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.793+0000 7f0ec1d9b700 1 --2- 172.21.15.16:0/1978379792 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0ec40cf4c0 0x7f0ec406aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:11:37.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.793+0000 7f0ec259c700 1 --2- 172.21.15.16:0/1978379792 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0ec4159de0 0x7f0ec415a1c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f0eb800a9e0 tx=0x7f0eb800acf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=704ec2c0c46eca3c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:37.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.794+0000 7f0ec159a700 1 -- 172.21.15.16:0/1978379792 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eb800cb70 con 0x7f0ec4159de0 2023-10-16T12:11:37.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.794+0000 7f0ec159a700 1 -- 172.21.15.16:0/1978379792 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0eb800ccd0 con 0x7f0ec4159de0 2023-10-16T12:11:37.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.794+0000 7f0ec159a700 1 -- 172.21.15.16:0/1978379792 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eb8018620 con 0x7f0ec4159de0 2023-10-16T12:11:37.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.794+0000 7f0ec8925700 1 -- 172.21.15.16:0/1978379792 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0ec4159de0 msgr2=0x7f0ec415a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:37.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.794+0000 7f0ec8925700 1 --2- 172.21.15.16:0/1978379792 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0ec4159de0 0x7f0ec415a1c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f0eb800a9e0 tx=0x7f0eb800acf0 comp rx=0 tx=0).stop 2023-10-16T12:11:37.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.794+0000 7f0ec8925700 1 -- 172.21.15.16:0/1978379792 shutdown_connections 2023-10-16T12:11:37.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.794+0000 7f0ec8925700 1 --2- 172.21.15.16:0/1978379792 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0ec4159de0 0x7f0ec415a1c0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:37.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.794+0000 7f0ec8925700 1 --2- 172.21.15.16:0/1978379792 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0ec40cf4c0 0x7f0ec406aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:37.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.794+0000 7f0ec8925700 1 -- 172.21.15.16:0/1978379792 >> 172.21.15.16:0/1978379792 conn(0x7f0ec40cc5e0 msgr2=0x7f0ec40cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:37.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.794+0000 7f0ec8925700 1 -- 172.21.15.16:0/1978379792 shutdown_connections 2023-10-16T12:11:37.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.794+0000 7f0ec8925700 1 -- 172.21.15.16:0/1978379792 wait complete. 2023-10-16T12:11:37.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.796+0000 7f0ec8925700 1 Processor -- start 2023-10-16T12:11:37.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.796+0000 7f0ec8925700 1 -- start start 2023-10-16T12:11:37.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.796+0000 7f0ec8925700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0ec406b000 0x7f0ec41ecfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:37.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.796+0000 7f0ec8925700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0ec40cf4c0 0x7f0ec406e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:37.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.796+0000 7f0ec8925700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ec4159de0 0x7f0ec406e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:37.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.796+0000 7f0ec8925700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec414d8a0 con 0x7f0ec4159de0 2023-10-16T12:11:37.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.796+0000 7f0ec8925700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec4153110 con 0x7f0ec406b000 2023-10-16T12:11:37.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.796+0000 7f0ec8925700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec41f64f0 con 0x7f0ec40cf4c0 2023-10-16T12:11:37.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.797+0000 7f0ec2d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ec4159de0 0x7f0ec406e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:37.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.797+0000 7f0ec2d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ec4159de0 0x7f0ec406e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34784/0 (socket says 172.21.15.16:34784) 2023-10-16T12:11:37.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.797+0000 7f0ec2d9d700 1 -- 172.21.15.16:0/1389619867 learned_addr learned my addr 172.21.15.16:0/1389619867 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:37.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.797+0000 7f0ec1d9b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0ec40cf4c0 0x7f0ec406e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:37.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.797+0000 7f0ec259c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0ec406b000 0x7f0ec41ecfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:37.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.797+0000 7f0ec2d9d700 1 -- 172.21.15.16:0/1389619867 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0ec40cf4c0 msgr2=0x7f0ec406e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:37.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.797+0000 7f0ec2d9d700 1 --2- 172.21.15.16:0/1389619867 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0ec40cf4c0 0x7f0ec406e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:37.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.797+0000 7f0ec2d9d700 1 -- 172.21.15.16:0/1389619867 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0ec406b000 msgr2=0x7f0ec41ecfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:37.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.797+0000 7f0ec2d9d700 1 --2- 172.21.15.16:0/1389619867 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0ec406b000 0x7f0ec41ecfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:37.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.797+0000 7f0ec2d9d700 1 -- 172.21.15.16:0/1389619867 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eb4009550 con 0x7f0ec4159de0 2023-10-16T12:11:37.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.797+0000 7f0ec259c700 1 --2- 172.21.15.16:0/1389619867 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0ec406b000 0x7f0ec41ecfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:11:37.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.797+0000 7f0ec2d9d700 1 --2- 172.21.15.16:0/1389619867 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ec4159de0 0x7f0ec406e590 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc00c790 tx=0x7f0ebc00caa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:37.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.797+0000 7f0eb2ffd700 1 -- 172.21.15.16:0/1389619867 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ebc003e30 con 0x7f0ec4159de0 2023-10-16T12:11:37.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.797+0000 7f0ec8925700 1 -- 172.21.15.16:0/1389619867 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0eb800a640 con 0x7f0ec4159de0 2023-10-16T12:11:37.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.797+0000 7f0eb2ffd700 1 -- 172.21.15.16:0/1389619867 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ebc006bc0 con 0x7f0ec4159de0 2023-10-16T12:11:37.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.797+0000 7f0eb2ffd700 1 -- 172.21.15.16:0/1389619867 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ebc005070 con 0x7f0ec4159de0 2023-10-16T12:11:37.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.797+0000 7f0ec8925700 1 -- 172.21.15.16:0/1389619867 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ec41f6ab0 con 0x7f0ec4159de0 2023-10-16T12:11:37.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.798+0000 7f0ec8925700 1 -- 172.21.15.16:0/1389619867 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ec404ebb0 con 0x7f0ec4159de0 2023-10-16T12:11:37.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.799+0000 7f0eb2ffd700 1 -- 172.21.15.16:0/1389619867 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f0ebc0051d0 con 0x7f0ec4159de0 2023-10-16T12:11:37.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.799+0000 7f0eb2ffd700 1 --2- 172.21.15.16:0/1389619867 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f0ea406af00 0x7f0ea406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:37.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.799+0000 7f0eb2ffd700 1 -- 172.21.15.16:0/1389619867 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f0ebc088c70 con 0x7f0ec4159de0 2023-10-16T12:11:37.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.799+0000 7f0ec259c700 1 --2- 172.21.15.16:0/1389619867 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f0ea406af00 0x7f0ea406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:37.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.800+0000 7f0ec259c700 1 --2- 172.21.15.16:0/1389619867 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f0ea406af00 0x7f0ea406d3c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f0eb800ad30 tx=0x7f0eb8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:37.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:37.804+0000 7f0eb2ffd700 1 -- 172.21.15.16:0/1389619867 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0ebc053840 con 0x7f0ec4159de0 2023-10-16T12:11:38.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.007+0000 7f0ec8925700 1 -- 172.21.15.16:0/1389619867 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0ec40ce3d0 con 0x7f0ec4159de0 2023-10-16T12:11:38.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.007+0000 7f0eb2ffd700 1 -- 172.21.15.16:0/1389619867 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0ebc0180a0 con 0x7f0ec4159de0 2023-10-16T12:11:38.009 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T12:11:38.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.009+0000 7f0ec8925700 1 -- 172.21.15.16:0/1389619867 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f0ea406af00 msgr2=0x7f0ea406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:38.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.009+0000 7f0ec8925700 1 --2- 172.21.15.16:0/1389619867 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f0ea406af00 0x7f0ea406d3c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f0eb800ad30 tx=0x7f0eb8008040 comp rx=0 tx=0).stop 2023-10-16T12:11:38.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.009+0000 7f0ec8925700 1 -- 172.21.15.16:0/1389619867 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ec4159de0 msgr2=0x7f0ec406e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:38.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.009+0000 7f0ec8925700 1 --2- 172.21.15.16:0/1389619867 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ec4159de0 0x7f0ec406e590 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc00c790 tx=0x7f0ebc00caa0 comp rx=0 tx=0).stop 2023-10-16T12:11:38.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.009+0000 7f0ec8925700 1 -- 172.21.15.16:0/1389619867 shutdown_connections 2023-10-16T12:11:38.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.009+0000 7f0ec8925700 1 --2- 172.21.15.16:0/1389619867 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f0ea406af00 0x7f0ea406d3c0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:38.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.009+0000 7f0ec8925700 1 --2- 172.21.15.16:0/1389619867 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ec4159de0 0x7f0ec406e590 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:38.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.009+0000 7f0ec8925700 1 --2- 172.21.15.16:0/1389619867 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0ec406b000 0x7f0ec41ecfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:38.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.009+0000 7f0ec8925700 1 --2- 172.21.15.16:0/1389619867 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0ec40cf4c0 0x7f0ec406e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:38.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.009+0000 7f0ec8925700 1 -- 172.21.15.16:0/1389619867 >> 172.21.15.16:0/1389619867 conn(0x7f0ec40cc5e0 msgr2=0x7f0ec4156e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:38.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.009+0000 7f0ec8925700 1 -- 172.21.15.16:0/1389619867 shutdown_connections 2023-10-16T12:11:38.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.010+0000 7f0ec8925700 1 -- 172.21.15.16:0/1389619867 wait complete. 2023-10-16T12:11:38.024 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T12:11:38.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.150+0000 7fb95f2d4700 1 Processor -- start 2023-10-16T12:11:38.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.150+0000 7fb95f2d4700 1 -- start start 2023-10-16T12:11:38.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.151+0000 7fb95f2d4700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fb95815b2c0 0x7fb958161340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:38.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.151+0000 7fb95f2d4700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb95806e570 0x7fb95815e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:38.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.151+0000 7fb95f2d4700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb958161880 con 0x7fb95806a7e0 2023-10-16T12:11:38.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.151+0000 7fb95f2d4700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb95815b6a0 con 0x7fb95815b2c0 2023-10-16T12:11:38.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.151+0000 7fb95f2d4700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb958161b20 con 0x7fb95806e570 2023-10-16T12:11:38.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.151+0000 7fb95d871700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb95806a7e0 0x7fb958071a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:52298/0 (socket says 172.21.15.16:52298) 2023-10-16T12:11:38.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.151+0000 7fb95d871700 1 -- 172.21.15.16:0/2908808667 learned_addr learned my addr 172.21.15.16:0/2908808667 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:38.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.152+0000 7fb94ffff700 1 -- 172.21.15.16:0/2908808667 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4261554551 0 0) 0x7fb958161880 con 0x7fb95806a7e0 2023-10-16T12:11:38.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.152+0000 7fb95d070700 1 --2- 172.21.15.16:0/2908808667 >> v2:172.21.15.49:3300/0 conn(0x7fb95815b2c0 0x7fb958161340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:38.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.152+0000 7fb94ffff700 1 -- 172.21.15.16:0/2908808667 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9440035b0 con 0x7fb95806a7e0 2023-10-16T12:11:38.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.152+0000 7fb95c86f700 1 --2- 172.21.15.16:0/2908808667 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb95806e570 0x7fb95815e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:38.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.152+0000 7fb94ffff700 1 -- 172.21.15.16:0/2908808667 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2806503865 0 0) 0x7fb9440035b0 con 0x7fb95806a7e0 2023-10-16T12:11:38.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.152+0000 7fb94ffff700 1 -- 172.21.15.16:0/2908808667 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb95806e570 msgr2=0x7fb95815e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:38.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.152+0000 7fb94ffff700 1 --2- 172.21.15.16:0/2908808667 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb95806e570 0x7fb95815e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:38.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.152+0000 7fb94ffff700 1 -- 172.21.15.16:0/2908808667 >> v2:172.21.15.49:3300/0 conn(0x7fb95815b2c0 msgr2=0x7fb958161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:38.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.152+0000 7fb94ffff700 1 --2- 172.21.15.16:0/2908808667 >> v2:172.21.15.49:3300/0 conn(0x7fb95815b2c0 0x7fb958161340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:38.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.152+0000 7fb94ffff700 1 -- 172.21.15.16:0/2908808667 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb958161e00 con 0x7fb95806a7e0 2023-10-16T12:11:38.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.152+0000 7fb94ffff700 1 -- 172.21.15.16:0/2908808667 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7fb9500019f0 con 0x7fb95806a7e0 2023-10-16T12:11:38.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.152+0000 7fb94ffff700 1 -- 172.21.15.16:0/2908808667 >> v1:172.21.15.16:6789/0 conn(0x7fb95806a7e0 legacy=0x7fb958071a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:38.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.152+0000 7fb94ffff700 1 --2- 172.21.15.16:0/2908808667 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb95806e570 0x7fb944003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:38.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.152+0000 7fb94ffff700 1 --2- 172.21.15.16:0/2908808667 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb944003f80 0x7fb944004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:38.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.152+0000 7fb94ffff700 1 --2- 172.21.15.16:0/2908808667 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb944004d90 0x7fb944005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:38.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.152+0000 7fb94ffff700 1 -- 172.21.15.16:0/2908808667 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb95815b6a0 con 0x7fb944004d90 2023-10-16T12:11:38.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.152+0000 7fb94ffff700 1 -- 172.21.15.16:0/2908808667 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb958161b20 con 0x7fb944003f80 2023-10-16T12:11:38.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.152+0000 7fb94ffff700 1 -- 172.21.15.16:0/2908808667 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb94400ba60 con 0x7fb95806e570 2023-10-16T12:11:38.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.152+0000 7fb95d871700 1 --2- 172.21.15.16:0/2908808667 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb944004d90 0x7fb944005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:38.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.152+0000 7fb95c86f700 1 --2- 172.21.15.16:0/2908808667 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb944003f80 0x7fb944004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:38.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.152+0000 7fb95d070700 1 --2- 172.21.15.16:0/2908808667 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb95806e570 0x7fb944003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:38.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.152+0000 7fb95d871700 1 -- 172.21.15.16:0/2908808667 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb95806e570 msgr2=0x7fb944003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:38.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.152+0000 7fb95d871700 1 --2- 172.21.15.16:0/2908808667 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb95806e570 0x7fb944003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:38.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.152+0000 7fb95d871700 1 -- 172.21.15.16:0/2908808667 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb944003f80 msgr2=0x7fb944004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:38.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.152+0000 7fb95d871700 1 --2- 172.21.15.16:0/2908808667 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb944003f80 0x7fb944004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:38.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.152+0000 7fb95d871700 1 -- 172.21.15.16:0/2908808667 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9440035b0 con 0x7fb944004d90 2023-10-16T12:11:38.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.153+0000 7fb95d871700 1 --2- 172.21.15.16:0/2908808667 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb944004d90 0x7fb944005210 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fb95000a890 tx=0x7fb944005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84e4b36071e88f41 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:38.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.153+0000 7fb94ffff700 1 -- 172.21.15.16:0/2908808667 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb950012600 con 0x7fb944004d90 2023-10-16T12:11:38.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.153+0000 7fb94ffff700 1 -- 172.21.15.16:0/2908808667 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb950012780 con 0x7fb944004d90 2023-10-16T12:11:38.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.153+0000 7fb94ffff700 1 -- 172.21.15.16:0/2908808667 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb950012b30 con 0x7fb944004d90 2023-10-16T12:11:38.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.153+0000 7fb95f2d4700 1 -- 172.21.15.16:0/2908808667 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb944004d90 msgr2=0x7fb944005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:38.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.153+0000 7fb95f2d4700 1 --2- 172.21.15.16:0/2908808667 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb944004d90 0x7fb944005210 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fb95000a890 tx=0x7fb944005880 comp rx=0 tx=0).stop 2023-10-16T12:11:38.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.153+0000 7fb95d871700 1 -- 172.21.15.16:0/2908808667 reap_dead start 2023-10-16T12:11:38.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.153+0000 7fb95f2d4700 1 -- 172.21.15.16:0/2908808667 shutdown_connections 2023-10-16T12:11:38.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.153+0000 7fb95f2d4700 1 -- 172.21.15.16:0/2908808667 >> 172.21.15.16:0/2908808667 conn(0x7fb9580b1f40 msgr2=0x7fb9580b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:38.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.153+0000 7fb95f2d4700 1 -- 172.21.15.16:0/2908808667 shutdown_connections 2023-10-16T12:11:38.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.153+0000 7fb95f2d4700 1 -- 172.21.15.16:0/2908808667 wait complete. 2023-10-16T12:11:38.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.155+0000 7fb95f2d4700 1 Processor -- start 2023-10-16T12:11:38.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.155+0000 7fb95f2d4700 1 -- start start 2023-10-16T12:11:38.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.155+0000 7fb95f2d4700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb944004d90 0x7fb95814c170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:38.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.155+0000 7fb95f2d4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb958160fa0 0x7fb95815e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:38.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.155+0000 7fb95f2d4700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb958161420 0x7fb9580c9880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:38.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.155+0000 7fb95f2d4700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb95815fbe0 con 0x7fb958160fa0 2023-10-16T12:11:38.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.155+0000 7fb95f2d4700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb95815ef60 con 0x7fb944004d90 2023-10-16T12:11:38.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.155+0000 7fb95f2d4700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb95815f240 con 0x7fb958161420 2023-10-16T12:11:38.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.156+0000 7fb95c86f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb958160fa0 0x7fb95815e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:38.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.156+0000 7fb95c86f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb958160fa0 0x7fb95815e4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34802/0 (socket says 172.21.15.16:34802) 2023-10-16T12:11:38.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.156+0000 7fb95c86f700 1 -- 172.21.15.16:0/3903534445 learned_addr learned my addr 172.21.15.16:0/3903534445 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:38.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.156+0000 7fb95d070700 1 --2- 172.21.15.16:0/3903534445 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb944004d90 0x7fb95814c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:38.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.156+0000 7fb95d871700 1 --2- 172.21.15.16:0/3903534445 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb958161420 0x7fb9580c9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:38.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.156+0000 7fb95c86f700 1 -- 172.21.15.16:0/3903534445 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb958161420 msgr2=0x7fb9580c9880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:38.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.156+0000 7fb95c86f700 1 --2- 172.21.15.16:0/3903534445 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb958161420 0x7fb9580c9880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:38.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.156+0000 7fb95c86f700 1 -- 172.21.15.16:0/3903534445 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb944004d90 msgr2=0x7fb95814c170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:38.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.156+0000 7fb95c86f700 1 --2- 172.21.15.16:0/3903534445 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb944004d90 0x7fb95814c170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:38.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.156+0000 7fb95c86f700 1 -- 172.21.15.16:0/3903534445 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb95815f520 con 0x7fb958160fa0 2023-10-16T12:11:38.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.156+0000 7fb95c86f700 1 --2- 172.21.15.16:0/3903534445 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb958160fa0 0x7fb95815e4a0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fb9540119a0 tx=0x7fb954011cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:38.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.156+0000 7fb94dffb700 1 -- 172.21.15.16:0/3903534445 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb95400f730 con 0x7fb958160fa0 2023-10-16T12:11:38.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.156+0000 7fb94dffb700 1 -- 172.21.15.16:0/3903534445 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb95400f890 con 0x7fb958160fa0 2023-10-16T12:11:38.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.156+0000 7fb95f2d4700 1 -- 172.21.15.16:0/3903534445 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb95815f800 con 0x7fb958160fa0 2023-10-16T12:11:38.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.156+0000 7fb94dffb700 1 -- 172.21.15.16:0/3903534445 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb95400fc20 con 0x7fb958160fa0 2023-10-16T12:11:38.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.157+0000 7fb95f2d4700 1 -- 172.21.15.16:0/3903534445 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb95814c6b0 con 0x7fb958160fa0 2023-10-16T12:11:38.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.157+0000 7fb95f2d4700 1 -- 172.21.15.16:0/3903534445 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb95804ebb0 con 0x7fb958160fa0 2023-10-16T12:11:38.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.158+0000 7fb94dffb700 1 -- 172.21.15.16:0/3903534445 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fb95400fd80 con 0x7fb958160fa0 2023-10-16T12:11:38.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.158+0000 7fb94dffb700 1 --2- 172.21.15.16:0/3903534445 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fb93c06af00 0x7fb93c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:38.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.158+0000 7fb94dffb700 1 -- 172.21.15.16:0/3903534445 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fb95408a030 con 0x7fb958160fa0 2023-10-16T12:11:38.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.158+0000 7fb95d070700 1 --2- 172.21.15.16:0/3903534445 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fb93c06af00 0x7fb93c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:38.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.159+0000 7fb95d070700 1 --2- 172.21.15.16:0/3903534445 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fb93c06af00 0x7fb93c06d3c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb948009ce0 tx=0x7fb948009210 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:38.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.163+0000 7fb94dffb700 1 -- 172.21.15.16:0/3903534445 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb954054c90 con 0x7fb958160fa0 2023-10-16T12:11:38.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.368+0000 7fb95f2d4700 1 -- 172.21.15.16:0/3903534445 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb95806b240 con 0x7fb958160fa0 2023-10-16T12:11:38.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.369+0000 7fb94dffb700 1 -- 172.21.15.16:0/3903534445 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb9540582b0 con 0x7fb958160fa0 2023-10-16T12:11:38.371 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-10-16T12:11:38.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.371+0000 7fb95f2d4700 1 -- 172.21.15.16:0/3903534445 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fb93c06af00 msgr2=0x7fb93c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:38.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.371+0000 7fb95f2d4700 1 --2- 172.21.15.16:0/3903534445 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fb93c06af00 0x7fb93c06d3c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb948009ce0 tx=0x7fb948009210 comp rx=0 tx=0).stop 2023-10-16T12:11:38.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.371+0000 7fb95f2d4700 1 -- 172.21.15.16:0/3903534445 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb958160fa0 msgr2=0x7fb95815e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:38.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.371+0000 7fb95f2d4700 1 --2- 172.21.15.16:0/3903534445 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb958160fa0 0x7fb95815e4a0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fb9540119a0 tx=0x7fb954011cb0 comp rx=0 tx=0).stop 2023-10-16T12:11:38.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.371+0000 7fb95f2d4700 1 -- 172.21.15.16:0/3903534445 shutdown_connections 2023-10-16T12:11:38.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.371+0000 7fb95f2d4700 1 --2- 172.21.15.16:0/3903534445 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fb93c06af00 0x7fb93c06d3c0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:38.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.371+0000 7fb95f2d4700 1 --2- 172.21.15.16:0/3903534445 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb958161420 0x7fb9580c9880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:38.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.371+0000 7fb95f2d4700 1 --2- 172.21.15.16:0/3903534445 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb944004d90 0x7fb95814c170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:38.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.371+0000 7fb95f2d4700 1 --2- 172.21.15.16:0/3903534445 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb958160fa0 0x7fb95815e4a0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:38.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.371+0000 7fb95f2d4700 1 -- 172.21.15.16:0/3903534445 >> 172.21.15.16:0/3903534445 conn(0x7fb9580b1f40 msgr2=0x7fb95815b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:38.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.371+0000 7fb95f2d4700 1 -- 172.21.15.16:0/3903534445 shutdown_connections 2023-10-16T12:11:38.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.371+0000 7fb95f2d4700 1 -- 172.21.15.16:0/3903534445 wait complete. 2023-10-16T12:11:38.385 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T12:11:38.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.512+0000 7fdfd0559700 1 Processor -- start 2023-10-16T12:11:38.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.512+0000 7fdfd0559700 1 -- start start 2023-10-16T12:11:38.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.513+0000 7fdfd0559700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdfc80cb880 0x7fdfc80cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:38.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.513+0000 7fdfd0559700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fdfc806da60 0x7fdfc806def0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:38.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.513+0000 7fdfd0559700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfc80cef70 con 0x7fdfc815b070 2023-10-16T12:11:38.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.513+0000 7fdfd0559700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfc806e430 con 0x7fdfc80cb880 2023-10-16T12:11:38.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.513+0000 7fdfd0559700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfc80ce360 con 0x7fdfc806da60 2023-10-16T12:11:38.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.513+0000 7fdfcdaf4700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdfc80cb880 0x7fdfc80cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:38.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.513+0000 7fdfce2f5700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fdfc815b070 0x7fdfc815b450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:52314/0 (socket says 172.21.15.16:52314) 2023-10-16T12:11:38.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.513+0000 7fdfce2f5700 1 -- 172.21.15.16:0/554007812 learned_addr learned my addr 172.21.15.16:0/554007812 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:38.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.513+0000 7fdfceaf6700 1 --2- 172.21.15.16:0/554007812 >> v2:172.21.15.49:3300/0 conn(0x7fdfc806da60 0x7fdfc806def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:38.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.513+0000 7fdfcdaf4700 1 -- 172.21.15.16:0/554007812 >> v2:172.21.15.49:3300/0 conn(0x7fdfc806da60 msgr2=0x7fdfc806def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:38.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.513+0000 7fdfcdaf4700 1 --2- 172.21.15.16:0/554007812 >> v2:172.21.15.49:3300/0 conn(0x7fdfc806da60 0x7fdfc806def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:38.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.513+0000 7fdfcdaf4700 1 -- 172.21.15.16:0/554007812 >> v1:172.21.15.16:6789/0 conn(0x7fdfc815b070 legacy=0x7fdfc815b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:38.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.513+0000 7fdfcdaf4700 1 -- 172.21.15.16:0/554007812 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfc80ce640 con 0x7fdfc80cb880 2023-10-16T12:11:38.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.513+0000 7fdfcd2f3700 1 -- 172.21.15.16:0/554007812 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2735139050 0 0) 0x7fdfc80cef70 con 0x7fdfc815b070 2023-10-16T12:11:38.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.513+0000 7fdfceaf6700 1 --2- 172.21.15.16:0/554007812 >> v2:172.21.15.49:3300/0 conn(0x7fdfc806da60 0x7fdfc806def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:11:38.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.513+0000 7fdfcdaf4700 1 --2- 172.21.15.16:0/554007812 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfc80cb880 0x7fdfc80cbd00 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fdfc806edc0 tx=0x7fdfb8009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d3bd778d28ff616 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:38.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.514+0000 7fdfcd2f3700 1 -- 172.21.15.16:0/554007812 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfb8011830 con 0x7fdfc80cb880 2023-10-16T12:11:38.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.514+0000 7fdfcd2f3700 1 -- 172.21.15.16:0/554007812 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfb8011990 con 0x7fdfc80cb880 2023-10-16T12:11:38.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.514+0000 7fdfcd2f3700 1 -- 172.21.15.16:0/554007812 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfb8011d20 con 0x7fdfc80cb880 2023-10-16T12:11:38.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.514+0000 7fdfd0559700 1 -- 172.21.15.16:0/554007812 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfc80cb880 msgr2=0x7fdfc80cbd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:38.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.514+0000 7fdfd0559700 1 --2- 172.21.15.16:0/554007812 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfc80cb880 0x7fdfc80cbd00 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fdfc806edc0 tx=0x7fdfb8009b20 comp rx=0 tx=0).stop 2023-10-16T12:11:38.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.514+0000 7fdfd0559700 1 -- 172.21.15.16:0/554007812 shutdown_connections 2023-10-16T12:11:38.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.514+0000 7fdfd0559700 1 --2- 172.21.15.16:0/554007812 >> v2:172.21.15.49:3300/0 conn(0x7fdfc806da60 0x7fdfc806def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:38.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.514+0000 7fdfd0559700 1 --2- 172.21.15.16:0/554007812 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfc80cb880 0x7fdfc80cbd00 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:38.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.514+0000 7fdfd0559700 1 -- 172.21.15.16:0/554007812 >> 172.21.15.16:0/554007812 conn(0x7fdfc80b1f40 msgr2=0x7fdfc80b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:38.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.514+0000 7fdfd0559700 1 -- 172.21.15.16:0/554007812 shutdown_connections 2023-10-16T12:11:38.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.514+0000 7fdfd0559700 1 -- 172.21.15.16:0/554007812 wait complete. 2023-10-16T12:11:38.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.515+0000 7fdfd0559700 1 Processor -- start 2023-10-16T12:11:38.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.515+0000 7fdfd0559700 1 -- start start 2023-10-16T12:11:38.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.516+0000 7fdfd0559700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdfc806da60 0x7fdfc814ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:38.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.516+0000 7fdfd0559700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfc80cb880 0x7fdfc815b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:38.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.516+0000 7fdfd0559700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdfc815b070 0x7fdfc8152120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:38.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.516+0000 7fdfd0559700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfc815bed0 con 0x7fdfc80cb880 2023-10-16T12:11:38.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.516+0000 7fdfd0559700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfc815c1b0 con 0x7fdfc806da60 2023-10-16T12:11:38.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.516+0000 7fdfd0559700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfc815c490 con 0x7fdfc815b070 2023-10-16T12:11:38.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.516+0000 7fdfcdaf4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfc80cb880 0x7fdfc815b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:38.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.516+0000 7fdfcdaf4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfc80cb880 0x7fdfc815b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34816/0 (socket says 172.21.15.16:34816) 2023-10-16T12:11:38.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.516+0000 7fdfcdaf4700 1 -- 172.21.15.16:0/38674956 learned_addr learned my addr 172.21.15.16:0/38674956 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:38.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.516+0000 7fdfce2f5700 1 --2- 172.21.15.16:0/38674956 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdfc806da60 0x7fdfc814ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:38.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.516+0000 7fdfceaf6700 1 --2- 172.21.15.16:0/38674956 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdfc815b070 0x7fdfc8152120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:38.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.517+0000 7fdfcdaf4700 1 -- 172.21.15.16:0/38674956 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdfc815b070 msgr2=0x7fdfc8152120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:38.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.517+0000 7fdfcdaf4700 1 --2- 172.21.15.16:0/38674956 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdfc815b070 0x7fdfc8152120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:38.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.517+0000 7fdfcdaf4700 1 -- 172.21.15.16:0/38674956 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdfc806da60 msgr2=0x7fdfc814ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:38.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.517+0000 7fdfcdaf4700 1 --2- 172.21.15.16:0/38674956 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdfc806da60 0x7fdfc814ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:38.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.517+0000 7fdfcdaf4700 1 -- 172.21.15.16:0/38674956 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfb80094a0 con 0x7fdfc80cb880 2023-10-16T12:11:38.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.517+0000 7fdfcdaf4700 1 --2- 172.21.15.16:0/38674956 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfc80cb880 0x7fdfc815b450 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fdfb8000c00 tx=0x7fdfb8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:38.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.517+0000 7fdfbeffd700 1 -- 172.21.15.16:0/38674956 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfb80036a0 con 0x7fdfc80cb880 2023-10-16T12:11:38.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.517+0000 7fdfbeffd700 1 -- 172.21.15.16:0/38674956 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfb8003820 con 0x7fdfc80cb880 2023-10-16T12:11:38.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.517+0000 7fdfbeffd700 1 -- 172.21.15.16:0/38674956 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfb8003bd0 con 0x7fdfc80cb880 2023-10-16T12:11:38.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.517+0000 7fdfd0559700 1 -- 172.21.15.16:0/38674956 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfc815c770 con 0x7fdfc80cb880 2023-10-16T12:11:38.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.517+0000 7fdfd0559700 1 -- 172.21.15.16:0/38674956 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfc8150ad0 con 0x7fdfc80cb880 2023-10-16T12:11:38.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.518+0000 7fdfd0559700 1 -- 172.21.15.16:0/38674956 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfc804ebb0 con 0x7fdfc80cb880 2023-10-16T12:11:38.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.518+0000 7fdfbeffd700 1 -- 172.21.15.16:0/38674956 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fdfb8003d30 con 0x7fdfc80cb880 2023-10-16T12:11:38.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.519+0000 7fdfbeffd700 1 --2- 172.21.15.16:0/38674956 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fdfac06af00 0x7fdfac06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:38.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.519+0000 7fdfbeffd700 1 -- 172.21.15.16:0/38674956 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fdfb808ccd0 con 0x7fdfc80cb880 2023-10-16T12:11:38.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.519+0000 7fdfce2f5700 1 --2- 172.21.15.16:0/38674956 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fdfac06af00 0x7fdfac06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:38.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.519+0000 7fdfce2f5700 1 --2- 172.21.15.16:0/38674956 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fdfac06af00 0x7fdfac06d3c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fdfc4007fd0 tx=0x7fdfc4007de0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:38.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.523+0000 7fdfbeffd700 1 -- 172.21.15.16:0/38674956 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdfb8057a50 con 0x7fdfc80cb880 2023-10-16T12:11:38.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.726+0000 7fdfd0559700 1 -- 172.21.15.16:0/38674956 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fdfc806def0 con 0x7fdfc80cb880 2023-10-16T12:11:38.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.727+0000 7fdfbeffd700 1 -- 172.21.15.16:0/38674956 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7fdfb8015020 con 0x7fdfc80cb880 2023-10-16T12:11:38.729 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-10-16T12:11:38.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.729+0000 7fdfd0559700 1 -- 172.21.15.16:0/38674956 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fdfac06af00 msgr2=0x7fdfac06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:38.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.729+0000 7fdfd0559700 1 --2- 172.21.15.16:0/38674956 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fdfac06af00 0x7fdfac06d3c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fdfc4007fd0 tx=0x7fdfc4007de0 comp rx=0 tx=0).stop 2023-10-16T12:11:38.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.729+0000 7fdfd0559700 1 -- 172.21.15.16:0/38674956 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfc80cb880 msgr2=0x7fdfc815b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:38.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.729+0000 7fdfd0559700 1 --2- 172.21.15.16:0/38674956 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfc80cb880 0x7fdfc815b450 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fdfb8000c00 tx=0x7fdfb8004750 comp rx=0 tx=0).stop 2023-10-16T12:11:38.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.729+0000 7fdfd0559700 1 -- 172.21.15.16:0/38674956 shutdown_connections 2023-10-16T12:11:38.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.729+0000 7fdfd0559700 1 --2- 172.21.15.16:0/38674956 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fdfac06af00 0x7fdfac06d3c0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:38.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.729+0000 7fdfd0559700 1 --2- 172.21.15.16:0/38674956 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdfc815b070 0x7fdfc8152120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:38.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.729+0000 7fdfd0559700 1 --2- 172.21.15.16:0/38674956 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdfc806da60 0x7fdfc814ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:38.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.729+0000 7fdfd0559700 1 --2- 172.21.15.16:0/38674956 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfc80cb880 0x7fdfc815b450 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:38.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.729+0000 7fdfd0559700 1 -- 172.21.15.16:0/38674956 >> 172.21.15.16:0/38674956 conn(0x7fdfc80b1f40 msgr2=0x7fdfc8153b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:38.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.729+0000 7fdfd0559700 1 -- 172.21.15.16:0/38674956 shutdown_connections 2023-10-16T12:11:38.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.729+0000 7fdfd0559700 1 -- 172.21.15.16:0/38674956 wait complete. 2023-10-16T12:11:38.743 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-16T12:11:38.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.868+0000 7f8558d15700 1 Processor -- start 2023-10-16T12:11:38.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.869+0000 7f8558d15700 1 -- start start 2023-10-16T12:11:38.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.869+0000 7f8558d15700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f85540d2110 0x7f85540ae1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:38.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.870+0000 7f8558d15700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85540b20f0 con 0x7f85540ce6a0 2023-10-16T12:11:38.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.870+0000 7f8558d15700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85540d03d0 con 0x7f855406ca10 2023-10-16T12:11:38.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.870+0000 7f8558d15700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85540b2340 con 0x7f85540d2110 2023-10-16T12:11:38.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.870+0000 7f855259c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f85540ce6a0 0x7f855406c300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:52328/0 (socket says 172.21.15.16:52328) 2023-10-16T12:11:38.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.870+0000 7f855259c700 1 -- 172.21.15.16:0/698884432 learned_addr learned my addr 172.21.15.16:0/698884432 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:38.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.870+0000 7f8551d9b700 1 --2- 172.21.15.16:0/698884432 >> v2:172.21.15.49:3300/0 conn(0x7f85540d2110 0x7f85540ae1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:38.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.870+0000 7f855159a700 1 -- 172.21.15.16:0/698884432 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2926975175 0 0) 0x7f85540b20f0 con 0x7f85540ce6a0 2023-10-16T12:11:38.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.870+0000 7f855159a700 1 -- 172.21.15.16:0/698884432 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85400035b0 con 0x7f85540ce6a0 2023-10-16T12:11:38.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.870+0000 7f8551d9b700 1 -- 172.21.15.16:0/698884432 >> v1:172.21.15.49:6789/0 conn(0x7f855406ca10 legacy=0x7f85540ae720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:38.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.870+0000 7f8551d9b700 1 -- 172.21.15.16:0/698884432 >> v1:172.21.15.16:6789/0 conn(0x7f85540ce6a0 legacy=0x7f855406c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:38.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.870+0000 7f8551d9b700 1 -- 172.21.15.16:0/698884432 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f855406bb00 con 0x7f85540d2110 2023-10-16T12:11:38.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.870+0000 7f8551d9b700 1 --2- 172.21.15.16:0/698884432 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f85540d2110 0x7f85540ae1e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f853c009840 tx=0x7f853c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b35bc6abcf5509f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:38.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.870+0000 7f855159a700 1 -- 172.21.15.16:0/698884432 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f853c011830 con 0x7f85540d2110 2023-10-16T12:11:38.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.871+0000 7f855159a700 1 -- 172.21.15.16:0/698884432 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f853c011990 con 0x7f85540d2110 2023-10-16T12:11:38.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.871+0000 7f855159a700 1 -- 172.21.15.16:0/698884432 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f853c011d20 con 0x7f85540d2110 2023-10-16T12:11:38.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.871+0000 7f8558d15700 1 -- 172.21.15.16:0/698884432 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f85540d2110 msgr2=0x7f85540ae1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:38.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.871+0000 7f8558d15700 1 --2- 172.21.15.16:0/698884432 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f85540d2110 0x7f85540ae1e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f853c009840 tx=0x7f853c009b50 comp rx=0 tx=0).stop 2023-10-16T12:11:38.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.871+0000 7f8558d15700 1 -- 172.21.15.16:0/698884432 shutdown_connections 2023-10-16T12:11:38.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.871+0000 7f8558d15700 1 --2- 172.21.15.16:0/698884432 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f85540d2110 0x7f85540ae1e0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:38.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.871+0000 7f8558d15700 1 -- 172.21.15.16:0/698884432 >> 172.21.15.16:0/698884432 conn(0x7f85540d15f0 msgr2=0x7f85540d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:38.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.871+0000 7f8558d15700 1 -- 172.21.15.16:0/698884432 shutdown_connections 2023-10-16T12:11:38.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.871+0000 7f8558d15700 1 -- 172.21.15.16:0/698884432 wait complete. 2023-10-16T12:11:38.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.872+0000 7f8558d15700 1 Processor -- start 2023-10-16T12:11:38.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.872+0000 7f8558d15700 1 -- start start 2023-10-16T12:11:38.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.873+0000 7f8558d15700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f855406ca10 0x7f8554154aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:38.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.873+0000 7f8558d15700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f85540ce6a0 0x7f85540cbc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:38.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.873+0000 7f8558d15700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85540d2110 0x7f85540cc160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:38.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.873+0000 7f8558d15700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85540c67b0 con 0x7f85540d2110 2023-10-16T12:11:38.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.873+0000 7f8558d15700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85540c6a90 con 0x7f85540ce6a0 2023-10-16T12:11:38.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.873+0000 7f8558d15700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85540c6d70 con 0x7f855406ca10 2023-10-16T12:11:38.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.873+0000 7f8552d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85540d2110 0x7f85540cc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:38.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.873+0000 7f8552d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85540d2110 0x7f85540cc160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34832/0 (socket says 172.21.15.16:34832) 2023-10-16T12:11:38.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.873+0000 7f8552d9d700 1 -- 172.21.15.16:0/3380667119 learned_addr learned my addr 172.21.15.16:0/3380667119 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:38.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.873+0000 7f8551d9b700 1 --2- 172.21.15.16:0/3380667119 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f85540ce6a0 0x7f85540cbc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:38.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.873+0000 7f855259c700 1 --2- 172.21.15.16:0/3380667119 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f855406ca10 0x7f8554154aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:38.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.874+0000 7f8552d9d700 1 -- 172.21.15.16:0/3380667119 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f855406ca10 msgr2=0x7f8554154aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:38.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.874+0000 7f8552d9d700 1 --2- 172.21.15.16:0/3380667119 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f855406ca10 0x7f8554154aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:38.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.874+0000 7f8552d9d700 1 -- 172.21.15.16:0/3380667119 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f85540ce6a0 msgr2=0x7f85540cbc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:38.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.874+0000 7f8552d9d700 1 --2- 172.21.15.16:0/3380667119 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f85540ce6a0 0x7f85540cbc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:38.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.874+0000 7f8552d9d700 1 -- 172.21.15.16:0/3380667119 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f853c0094a0 con 0x7f85540d2110 2023-10-16T12:11:38.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.874+0000 7f8552d9d700 1 --2- 172.21.15.16:0/3380667119 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85540d2110 0x7f85540cc160 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f854c00cc50 tx=0x7f854c00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:38.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.874+0000 7f854affd700 1 -- 172.21.15.16:0/3380667119 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f854c003c00 con 0x7f85540d2110 2023-10-16T12:11:38.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.874+0000 7f854affd700 1 -- 172.21.15.16:0/3380667119 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f854c003d60 con 0x7f85540d2110 2023-10-16T12:11:38.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.874+0000 7f8558d15700 1 -- 172.21.15.16:0/3380667119 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85540c7050 con 0x7f85540d2110 2023-10-16T12:11:38.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.874+0000 7f854affd700 1 -- 172.21.15.16:0/3380667119 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f854c00aab0 con 0x7f85540d2110 2023-10-16T12:11:38.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.874+0000 7f8558d15700 1 -- 172.21.15.16:0/3380667119 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85540c5b30 con 0x7f85540d2110 2023-10-16T12:11:38.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.875+0000 7f8558d15700 1 -- 172.21.15.16:0/3380667119 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85540b48a0 con 0x7f85540d2110 2023-10-16T12:11:38.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.875+0000 7f854affd700 1 -- 172.21.15.16:0/3380667119 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f854c00ac10 con 0x7f85540d2110 2023-10-16T12:11:38.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.876+0000 7f854affd700 1 --2- 172.21.15.16:0/3380667119 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f853406af00 0x7f853406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:38.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.876+0000 7f854affd700 1 -- 172.21.15.16:0/3380667119 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f854c089970 con 0x7f85540d2110 2023-10-16T12:11:38.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.876+0000 7f855259c700 1 --2- 172.21.15.16:0/3380667119 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f853406af00 0x7f853406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:38.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.876+0000 7f855259c700 1 --2- 172.21.15.16:0/3380667119 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f853406af00 0x7f853406d3c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8544007fd0 tx=0x7f8544007dc0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:38.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:38.880+0000 7f854affd700 1 -- 172.21.15.16:0/3380667119 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f85540b48a0 con 0x7f85540d2110 2023-10-16T12:11:39.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.083+0000 7f8558d15700 1 -- 172.21.15.16:0/3380667119 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f85540b4880 con 0x7f85540d2110 2023-10-16T12:11:39.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.084+0000 7f854affd700 1 -- 172.21.15.16:0/3380667119 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f854c054660 con 0x7f85540d2110 2023-10-16T12:11:39.086 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-10-16T12:11:39.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.086+0000 7f8558d15700 1 -- 172.21.15.16:0/3380667119 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f853406af00 msgr2=0x7f853406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:39.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.086+0000 7f8558d15700 1 --2- 172.21.15.16:0/3380667119 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f853406af00 0x7f853406d3c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8544007fd0 tx=0x7f8544007dc0 comp rx=0 tx=0).stop 2023-10-16T12:11:39.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.086+0000 7f8558d15700 1 -- 172.21.15.16:0/3380667119 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85540d2110 msgr2=0x7f85540cc160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:39.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.086+0000 7f8558d15700 1 --2- 172.21.15.16:0/3380667119 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85540d2110 0x7f85540cc160 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f854c00cc50 tx=0x7f854c00cf60 comp rx=0 tx=0).stop 2023-10-16T12:11:39.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.086+0000 7f8558d15700 1 -- 172.21.15.16:0/3380667119 shutdown_connections 2023-10-16T12:11:39.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.086+0000 7f8558d15700 1 --2- 172.21.15.16:0/3380667119 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f853406af00 0x7f853406d3c0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:39.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.086+0000 7f8558d15700 1 --2- 172.21.15.16:0/3380667119 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85540d2110 0x7f85540cc160 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:39.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.086+0000 7f8558d15700 1 --2- 172.21.15.16:0/3380667119 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f855406ca10 0x7f8554154aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:39.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.086+0000 7f8558d15700 1 --2- 172.21.15.16:0/3380667119 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f85540ce6a0 0x7f85540cbc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:39.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.086+0000 7f8558d15700 1 -- 172.21.15.16:0/3380667119 >> 172.21.15.16:0/3380667119 conn(0x7f85540d15f0 msgr2=0x7f855406ec00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:39.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.086+0000 7f8558d15700 1 -- 172.21.15.16:0/3380667119 shutdown_connections 2023-10-16T12:11:39.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.086+0000 7f8558d15700 1 -- 172.21.15.16:0/3380667119 wait complete. 2023-10-16T12:11:39.100 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T12:11:39.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.224+0000 7f83ea2e8700 1 Processor -- start 2023-10-16T12:11:39.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.224+0000 7f83ea2e8700 1 -- start start 2023-10-16T12:11:39.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.225+0000 7f83ea2e8700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f83e40cf4a0 0x7f83e40726e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:39.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.225+0000 7f83ea2e8700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f83e40b5a00 0x7f83e415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:39.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.225+0000 7f83ea2e8700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83e415bc80 con 0x7f83e40cf880 2023-10-16T12:11:39.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.225+0000 7f83ea2e8700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83e40b5e40 con 0x7f83e40cf4a0 2023-10-16T12:11:39.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.225+0000 7f83ea2e8700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83e4072c20 con 0x7f83e40b5a00 2023-10-16T12:11:39.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.225+0000 7f83e3fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f83e40cf4a0 0x7f83e40726e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:39.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.225+0000 7f83e3fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f83e40cf4a0 0x7f83e40726e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34836/0 (socket says 172.21.15.16:34836) 2023-10-16T12:11:39.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.225+0000 7f83e3fff700 1 -- 172.21.15.16:0/4253132807 learned_addr learned my addr 172.21.15.16:0/4253132807 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:39.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.225+0000 7f83e8885700 1 --2- 172.21.15.16:0/4253132807 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f83e40b5a00 0x7f83e415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:39.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.225+0000 7f83e3fff700 1 -- 172.21.15.16:0/4253132807 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f83e40b5a00 msgr2=0x7f83e415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:39.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.225+0000 7f83e3fff700 1 --2- 172.21.15.16:0/4253132807 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f83e40b5a00 0x7f83e415a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:39.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.225+0000 7f83e3fff700 1 -- 172.21.15.16:0/4253132807 >> v1:172.21.15.49:6789/0 conn(0x7f83e40cf880 legacy=0x7f83e40b52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:39.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.225+0000 7f83e3fff700 1 -- 172.21.15.16:0/4253132807 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83e415b070 con 0x7f83e40cf4a0 2023-10-16T12:11:39.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.225+0000 7f83e3fff700 1 --2- 172.21.15.16:0/4253132807 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83e40cf4a0 0x7f83e40726e0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f83d400bb90 tx=0x7f83d400bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9705d3a0a2fcb459 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:39.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.225+0000 7f83e2ffd700 1 -- 172.21.15.16:0/4253132807 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d400dd50 con 0x7f83e40cf4a0 2023-10-16T12:11:39.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.226+0000 7f83e2ffd700 1 -- 172.21.15.16:0/4253132807 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83d4010470 con 0x7f83e40cf4a0 2023-10-16T12:11:39.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.226+0000 7f83e2ffd700 1 -- 172.21.15.16:0/4253132807 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d4010800 con 0x7f83e40cf4a0 2023-10-16T12:11:39.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.226+0000 7f83ea2e8700 1 -- 172.21.15.16:0/4253132807 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83e40cf4a0 msgr2=0x7f83e40726e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:39.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.226+0000 7f83ea2e8700 1 --2- 172.21.15.16:0/4253132807 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83e40cf4a0 0x7f83e40726e0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f83d400bb90 tx=0x7f83d400bea0 comp rx=0 tx=0).stop 2023-10-16T12:11:39.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.226+0000 7f83ea2e8700 1 -- 172.21.15.16:0/4253132807 shutdown_connections 2023-10-16T12:11:39.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.226+0000 7f83ea2e8700 1 --2- 172.21.15.16:0/4253132807 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f83e40b5a00 0x7f83e415a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:39.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.226+0000 7f83ea2e8700 1 --2- 172.21.15.16:0/4253132807 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83e40cf4a0 0x7f83e40726e0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:39.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.226+0000 7f83ea2e8700 1 -- 172.21.15.16:0/4253132807 >> 172.21.15.16:0/4253132807 conn(0x7f83e40b1fd0 msgr2=0x7f83e40b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:39.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.226+0000 7f83ea2e8700 1 -- 172.21.15.16:0/4253132807 shutdown_connections 2023-10-16T12:11:39.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.226+0000 7f83ea2e8700 1 -- 172.21.15.16:0/4253132807 wait complete. 2023-10-16T12:11:39.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.228+0000 7f83ea2e8700 1 Processor -- start 2023-10-16T12:11:39.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.228+0000 7f83ea2e8700 1 -- start start 2023-10-16T12:11:39.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.229+0000 7f83ea2e8700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f83e40b5a00 0x7f83e40c67f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:39.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.229+0000 7f83ea2e8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83e40cf4a0 0x7f83e40c4a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:39.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.229+0000 7f83ea2e8700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f83e40cf880 0x7f83e40c7510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:39.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.229+0000 7f83ea2e8700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83e41e3dc0 con 0x7f83e40cf4a0 2023-10-16T12:11:39.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.229+0000 7f83ea2e8700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83e41e40a0 con 0x7f83e40cf880 2023-10-16T12:11:39.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.229+0000 7f83ea2e8700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83e41e4380 con 0x7f83e40b5a00 2023-10-16T12:11:39.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.229+0000 7f83e37fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83e40cf4a0 0x7f83e40c4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:39.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.229+0000 7f83e37fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83e40cf4a0 0x7f83e40c4a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34842/0 (socket says 172.21.15.16:34842) 2023-10-16T12:11:39.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.229+0000 7f83e37fe700 1 -- 172.21.15.16:0/4185616963 learned_addr learned my addr 172.21.15.16:0/4185616963 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:39.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.229+0000 7f83e3fff700 1 --2- 172.21.15.16:0/4185616963 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f83e40b5a00 0x7f83e40c67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:39.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.229+0000 7f83e8885700 1 --2- 172.21.15.16:0/4185616963 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f83e40cf880 0x7f83e40c7510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:39.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.229+0000 7f83e37fe700 1 -- 172.21.15.16:0/4185616963 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f83e40b5a00 msgr2=0x7f83e40c67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:39.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.229+0000 7f83e37fe700 1 --2- 172.21.15.16:0/4185616963 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f83e40b5a00 0x7f83e40c67f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:39.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.229+0000 7f83e37fe700 1 -- 172.21.15.16:0/4185616963 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f83e40cf880 msgr2=0x7f83e40c7510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:39.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.229+0000 7f83e37fe700 1 --2- 172.21.15.16:0/4185616963 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f83e40cf880 0x7f83e40c7510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:39.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.229+0000 7f83e37fe700 1 -- 172.21.15.16:0/4185616963 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83d400b7f0 con 0x7f83e40cf4a0 2023-10-16T12:11:39.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.229+0000 7f83e37fe700 1 --2- 172.21.15.16:0/4185616963 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83e40cf4a0 0x7f83e40c4a40 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f83d800c450 tx=0x7f83d800c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:39.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.230+0000 7f83e0ff9700 1 -- 172.21.15.16:0/4185616963 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d80052d0 con 0x7f83e40cf4a0 2023-10-16T12:11:39.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.230+0000 7f83e0ff9700 1 -- 172.21.15.16:0/4185616963 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83d8005430 con 0x7f83e40cf4a0 2023-10-16T12:11:39.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.230+0000 7f83ea2e8700 1 -- 172.21.15.16:0/4185616963 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83e41e4660 con 0x7f83e40cf4a0 2023-10-16T12:11:39.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.230+0000 7f83e0ff9700 1 -- 172.21.15.16:0/4185616963 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d80038c0 con 0x7f83e40cf4a0 2023-10-16T12:11:39.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.230+0000 7f83ea2e8700 1 -- 172.21.15.16:0/4185616963 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83e40c6d30 con 0x7f83e40cf4a0 2023-10-16T12:11:39.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.231+0000 7f83ea2e8700 1 -- 172.21.15.16:0/4185616963 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83e41e4b80 con 0x7f83e40cf4a0 2023-10-16T12:11:39.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.231+0000 7f83e0ff9700 1 -- 172.21.15.16:0/4185616963 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f83d8003a20 con 0x7f83e40cf4a0 2023-10-16T12:11:39.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.231+0000 7f83e0ff9700 1 --2- 172.21.15.16:0/4185616963 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f83d006af00 0x7f83d006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:39.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.232+0000 7f83e0ff9700 1 -- 172.21.15.16:0/4185616963 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f83d8088cb0 con 0x7f83e40cf4a0 2023-10-16T12:11:39.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.232+0000 7f83e3fff700 1 --2- 172.21.15.16:0/4185616963 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f83d006af00 0x7f83d006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:39.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.232+0000 7f83e3fff700 1 --2- 172.21.15.16:0/4185616963 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f83d006af00 0x7f83d006d3c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f83d400bee0 tx=0x7f83d4013000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:39.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.236+0000 7f83e0ff9700 1 -- 172.21.15.16:0/4185616963 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f83e41e4b80 con 0x7f83e40cf4a0 2023-10-16T12:11:39.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.457+0000 7f83ea2e8700 1 -- 172.21.15.16:0/4185616963 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f83e41e4b80 con 0x7f83e40cf4a0 2023-10-16T12:11:39.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.457+0000 7f83e0ff9700 1 -- 172.21.15.16:0/4185616963 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) v1 ==== 80+0+0 (secure 0 0 0) 0x7f83e41e4b80 con 0x7f83e40cf4a0 2023-10-16T12:11:39.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.459+0000 7f83ea2e8700 1 -- 172.21.15.16:0/4185616963 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f83d006af00 msgr2=0x7f83d006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:39.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.459+0000 7f83ea2e8700 1 --2- 172.21.15.16:0/4185616963 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f83d006af00 0x7f83d006d3c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f83d400bee0 tx=0x7f83d4013000 comp rx=0 tx=0).stop 2023-10-16T12:11:39.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.459+0000 7f83ea2e8700 1 -- 172.21.15.16:0/4185616963 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83e40cf4a0 msgr2=0x7f83e40c4a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:39.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.459+0000 7f83ea2e8700 1 --2- 172.21.15.16:0/4185616963 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83e40cf4a0 0x7f83e40c4a40 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f83d800c450 tx=0x7f83d800c760 comp rx=0 tx=0).stop 2023-10-16T12:11:39.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.459+0000 7f83ea2e8700 1 -- 172.21.15.16:0/4185616963 shutdown_connections 2023-10-16T12:11:39.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.459+0000 7f83ea2e8700 1 --2- 172.21.15.16:0/4185616963 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f83d006af00 0x7f83d006d3c0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:39.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.459+0000 7f83ea2e8700 1 --2- 172.21.15.16:0/4185616963 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f83e40cf880 0x7f83e40c7510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:39.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.459+0000 7f83ea2e8700 1 --2- 172.21.15.16:0/4185616963 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f83e40b5a00 0x7f83e40c67f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:39.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.459+0000 7f83ea2e8700 1 --2- 172.21.15.16:0/4185616963 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83e40cf4a0 0x7f83e40c4a40 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:39.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.459+0000 7f83ea2e8700 1 -- 172.21.15.16:0/4185616963 >> 172.21.15.16:0/4185616963 conn(0x7f83e40b1fd0 msgr2=0x7f83e40b0220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:39.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.459+0000 7f83ea2e8700 1 -- 172.21.15.16:0/4185616963 shutdown_connections 2023-10-16T12:11:39.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.459+0000 7f83ea2e8700 1 -- 172.21.15.16:0/4185616963 wait complete. 2023-10-16T12:11:39.464 INFO:teuthology.orchestra.run.smithi016.stderr:listed 0 entries 2023-10-16T12:11:39.473 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2023-10-16T12:11:39.474 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:39.474 DEBUG:teuthology.orchestra.run.smithi049:> sudo umount /home/ubuntu/cephtest/mnt.1 -f 2023-10-16T12:11:39.531 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi049.front.sepia.ceph.com 2023-10-16T12:11:39.531 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:39.531 DEBUG:teuthology.orchestra.run.smithi049:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-10-16T12:11:39.554 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:11:39.554 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:11:39.572 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:11:39.573 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:11:39.630 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:11:39.630 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:11:39.688 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:11:39.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.850+0000 7fb02772d700 1 Processor -- start 2023-10-16T12:11:39.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.850+0000 7fb02772d700 1 -- start start 2023-10-16T12:11:39.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.851+0000 7fb02772d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb02015b9f0 0x7fb0200b29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:39.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.851+0000 7fb02772d700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fb0200cf280 0x7fb02006ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:39.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.851+0000 7fb02772d700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0200c3490 con 0x7fb02006d940 2023-10-16T12:11:39.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.851+0000 7fb02772d700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb02006e300 con 0x7fb0200cf280 2023-10-16T12:11:39.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.851+0000 7fb02772d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0200cdad0 con 0x7fb02015b9f0 2023-10-16T12:11:39.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.852+0000 7fb0254c9700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb02015b9f0 0x7fb0200b29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:39.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.852+0000 7fb025cca700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fb0200cf280 0x7fb02006ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:39.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.852+0000 7fb0254c9700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb02015b9f0 0x7fb0200b29a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:51338/0 (socket says 172.21.15.16:51338) 2023-10-16T12:11:39.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.852+0000 7fb025cca700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fb0200cf280 0x7fb02006ddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:33914/0 (socket says 172.21.15.16:33914) 2023-10-16T12:11:39.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.852+0000 7fb024cc8700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7fb02006d940 0x7fb0200ceb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:59536/0 (socket says 172.21.15.16:59536) 2023-10-16T12:11:39.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.852+0000 7fb0254c9700 1 -- 172.21.15.16:0/1922839598 learned_addr learned my addr 172.21.15.16:0/1922839598 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:39.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.852+0000 7fb025cca700 1 -- 172.21.15.16:0/1922839598 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb02015b9f0 msgr2=0x7fb0200b29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:39.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.852+0000 7fb025cca700 1 --2- 172.21.15.16:0/1922839598 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb02015b9f0 0x7fb0200b29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:39.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.852+0000 7fb025cca700 1 -- 172.21.15.16:0/1922839598 >> v1:172.21.15.49:6789/0 conn(0x7fb02006d940 legacy=0x7fb0200ceb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:39.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.852+0000 7fb025cca700 1 -- 172.21.15.16:0/1922839598 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0200cddb0 con 0x7fb0200cf280 2023-10-16T12:11:39.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.852+0000 7fb0254c9700 1 --2- 172.21.15.16:0/1922839598 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb02015b9f0 0x7fb0200b29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:11:39.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.852+0000 7fb025cca700 1 --2- 172.21.15.16:0/1922839598 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb0200cf280 0x7fb02006ddc0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb018009d90 tx=0x7fb01800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f8ba392f311d317 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:39.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.852+0000 7fb017fff700 1 -- 172.21.15.16:0/1922839598 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb01800e070 con 0x7fb0200cf280 2023-10-16T12:11:39.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.853+0000 7fb017fff700 1 -- 172.21.15.16:0/1922839598 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb01800bb20 con 0x7fb0200cf280 2023-10-16T12:11:39.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.853+0000 7fb017fff700 1 -- 172.21.15.16:0/1922839598 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb018012430 con 0x7fb0200cf280 2023-10-16T12:11:39.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.853+0000 7fb02772d700 1 -- 172.21.15.16:0/1922839598 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb0200cf280 msgr2=0x7fb02006ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:39.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.853+0000 7fb02772d700 1 --2- 172.21.15.16:0/1922839598 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb0200cf280 0x7fb02006ddc0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb018009d90 tx=0x7fb01800b330 comp rx=0 tx=0).stop 2023-10-16T12:11:39.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.853+0000 7fb02772d700 1 -- 172.21.15.16:0/1922839598 shutdown_connections 2023-10-16T12:11:39.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.853+0000 7fb02772d700 1 --2- 172.21.15.16:0/1922839598 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb0200cf280 0x7fb02006ddc0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:39.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.853+0000 7fb02772d700 1 --2- 172.21.15.16:0/1922839598 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb02015b9f0 0x7fb0200b29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:39.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.853+0000 7fb02772d700 1 -- 172.21.15.16:0/1922839598 >> 172.21.15.16:0/1922839598 conn(0x7fb0200cc5b0 msgr2=0x7fb0200cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:39.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.853+0000 7fb02772d700 1 -- 172.21.15.16:0/1922839598 shutdown_connections 2023-10-16T12:11:39.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.853+0000 7fb02772d700 1 -- 172.21.15.16:0/1922839598 wait complete. 2023-10-16T12:11:39.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.854+0000 7fb02772d700 1 Processor -- start 2023-10-16T12:11:39.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.854+0000 7fb02772d700 1 -- start start 2023-10-16T12:11:39.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.855+0000 7fb02772d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb02006d940 0x7fb02014b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:39.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.855+0000 7fb02772d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0200cf280 0x7fb0200c5960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:39.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.855+0000 7fb02772d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb02015c5b0 0x7fb0200c6d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:39.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.855+0000 7fb02772d700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0200c7980 con 0x7fb0200cf280 2023-10-16T12:11:39.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.855+0000 7fb02772d700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb02014b550 con 0x7fb02006d940 2023-10-16T12:11:39.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.855+0000 7fb02772d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0200c4670 con 0x7fb02015c5b0 2023-10-16T12:11:39.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.855+0000 7fb024cc8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0200cf280 0x7fb0200c5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:39.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.855+0000 7fb024cc8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0200cf280 0x7fb0200c5960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34856/0 (socket says 172.21.15.16:34856) 2023-10-16T12:11:39.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.855+0000 7fb024cc8700 1 -- 172.21.15.16:0/1616510617 learned_addr learned my addr 172.21.15.16:0/1616510617 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:39.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.855+0000 7fb0254c9700 1 --2- 172.21.15.16:0/1616510617 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb02006d940 0x7fb02014b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:39.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.855+0000 7fb025cca700 1 --2- 172.21.15.16:0/1616510617 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb02015c5b0 0x7fb0200c6d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:39.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.856+0000 7fb024cc8700 1 -- 172.21.15.16:0/1616510617 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb02015c5b0 msgr2=0x7fb0200c6d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:39.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.856+0000 7fb024cc8700 1 --2- 172.21.15.16:0/1616510617 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb02015c5b0 0x7fb0200c6d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:39.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.856+0000 7fb024cc8700 1 -- 172.21.15.16:0/1616510617 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb02006d940 msgr2=0x7fb02014b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:39.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.856+0000 7fb024cc8700 1 --2- 172.21.15.16:0/1616510617 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb02006d940 0x7fb02014b010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:39.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.856+0000 7fb024cc8700 1 -- 172.21.15.16:0/1616510617 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb01c009550 con 0x7fb0200cf280 2023-10-16T12:11:39.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.856+0000 7fb024cc8700 1 --2- 172.21.15.16:0/1616510617 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0200cf280 0x7fb0200c5960 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fb01000c500 tx=0x7fb01000c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:39.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.856+0000 7fb015ffb700 1 -- 172.21.15.16:0/1616510617 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb010005cb0 con 0x7fb0200cf280 2023-10-16T12:11:39.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.856+0000 7fb015ffb700 1 -- 172.21.15.16:0/1616510617 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb010005e10 con 0x7fb0200cf280 2023-10-16T12:11:39.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.856+0000 7fb02772d700 1 -- 172.21.15.16:0/1616510617 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb018009a70 con 0x7fb0200cf280 2023-10-16T12:11:39.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.856+0000 7fb015ffb700 1 -- 172.21.15.16:0/1616510617 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0100072d0 con 0x7fb0200cf280 2023-10-16T12:11:39.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.856+0000 7fb02772d700 1 -- 172.21.15.16:0/1616510617 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0200c4ba0 con 0x7fb0200cf280 2023-10-16T12:11:39.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.857+0000 7fb02772d700 1 -- 172.21.15.16:0/1616510617 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb02004ebb0 con 0x7fb0200cf280 2023-10-16T12:11:39.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.858+0000 7fb015ffb700 1 -- 172.21.15.16:0/1616510617 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fb010007430 con 0x7fb0200cf280 2023-10-16T12:11:39.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.858+0000 7fb015ffb700 1 --2- 172.21.15.16:0/1616510617 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fb00406af00 0x7fb00406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:39.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.858+0000 7fb015ffb700 1 -- 172.21.15.16:0/1616510617 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fb010088c90 con 0x7fb0200cf280 2023-10-16T12:11:39.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.858+0000 7fb0254c9700 1 --2- 172.21.15.16:0/1616510617 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fb00406af00 0x7fb00406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:39.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.859+0000 7fb0254c9700 1 --2- 172.21.15.16:0/1616510617 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fb00406af00 0x7fb00406d3c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fb01c006d30 tx=0x7fb01c0092c0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:39.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:39.863+0000 7fb015ffb700 1 -- 172.21.15.16:0/1616510617 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb010053910 con 0x7fb0200cf280 2023-10-16T12:11:40.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.066+0000 7fb02772d700 1 -- 172.21.15.16:0/1616510617 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb0200ce340 con 0x7fb0200cf280 2023-10-16T12:11:40.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.067+0000 7fb015ffb700 1 -- 172.21.15.16:0/1616510617 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18629 (secure 0 0 0) 0x7fb010056f30 con 0x7fb0200cf280 2023-10-16T12:11:40.070 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:11:40.071 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:11:28.241865+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:10:36.935773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T12:10:38.004870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-16T12:10:41.672025+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"3.2b","mappings":[{"from":0,"to":2}]},{"pgid":"3.38","mappings":[{"from":3,"to":1}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]},{"pgid":"4.1b","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:11:40.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.069+0000 7fb02772d700 1 -- 172.21.15.16:0/1616510617 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fb00406af00 msgr2=0x7fb00406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:40.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.069+0000 7fb02772d700 1 --2- 172.21.15.16:0/1616510617 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fb00406af00 0x7fb00406d3c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fb01c006d30 tx=0x7fb01c0092c0 comp rx=0 tx=0).stop 2023-10-16T12:11:40.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.069+0000 7fb02772d700 1 -- 172.21.15.16:0/1616510617 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0200cf280 msgr2=0x7fb0200c5960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:40.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.069+0000 7fb02772d700 1 --2- 172.21.15.16:0/1616510617 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0200cf280 0x7fb0200c5960 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fb01000c500 tx=0x7fb01000c810 comp rx=0 tx=0).stop 2023-10-16T12:11:40.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.069+0000 7fb02772d700 1 -- 172.21.15.16:0/1616510617 shutdown_connections 2023-10-16T12:11:40.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.069+0000 7fb02772d700 1 --2- 172.21.15.16:0/1616510617 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fb00406af00 0x7fb00406d3c0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:40.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.069+0000 7fb02772d700 1 --2- 172.21.15.16:0/1616510617 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb02015c5b0 0x7fb0200c6d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:40.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.070+0000 7fb02772d700 1 --2- 172.21.15.16:0/1616510617 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb02006d940 0x7fb02014b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:40.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.070+0000 7fb02772d700 1 --2- 172.21.15.16:0/1616510617 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0200cf280 0x7fb0200c5960 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:40.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.070+0000 7fb02772d700 1 -- 172.21.15.16:0/1616510617 >> 172.21.15.16:0/1616510617 conn(0x7fb0200cc5b0 msgr2=0x7fb02006bb50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:40.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.070+0000 7fb02772d700 1 -- 172.21.15.16:0/1616510617 shutdown_connections 2023-10-16T12:11:40.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.070+0000 7fb02772d700 1 -- 172.21.15.16:0/1616510617 wait complete. 2023-10-16T12:11:40.085 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T12:10:36.935773+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T12:10:38.004870+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-16T12:10:41.672025+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-16T12:11:40.086 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:11:40.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.211+0000 7f9ae1c1f700 1 Processor -- start 2023-10-16T12:11:40.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.211+0000 7f9ae1c1f700 1 -- start start 2023-10-16T12:11:40.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.211+0000 7f9ae1c1f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9adc072b30 0x7f9adc071a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:40.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.211+0000 7f9ae1c1f700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f9adc15bb40 0x7f9adc06e310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:40.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.211+0000 7f9ae1c1f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9adc0caa40 0x7f9adc0cae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:40.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.211+0000 7f9ae1c1f700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9adc0cb3d0 con 0x7f9adc072b30 2023-10-16T12:11:40.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.212+0000 7f9ae1c1f700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9adc15c040 con 0x7f9adc15bb40 2023-10-16T12:11:40.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.212+0000 7f9ae1c1f700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9adc15c320 con 0x7f9adc0caa40 2023-10-16T12:11:40.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.212+0000 7f9adb7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9adc072b30 0x7f9adc071a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:40.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.212+0000 7f9adb7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9adc072b30 0x7f9adc071a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34862/0 (socket says 172.21.15.16:34862) 2023-10-16T12:11:40.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.212+0000 7f9adb7fe700 1 -- 172.21.15.16:0/2280927380 learned_addr learned my addr 172.21.15.16:0/2280927380 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:40.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.212+0000 7f9adb7fe700 1 -- 172.21.15.16:0/2280927380 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9adc0caa40 msgr2=0x7f9adc0cae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:40.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.212+0000 7f9adaffd700 1 --2- 172.21.15.16:0/2280927380 >> v2:172.21.15.49:3300/0 conn(0x7f9adc15bb40 0x7f9adc06e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:40.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.212+0000 7f9adbfff700 1 --2- 172.21.15.16:0/2280927380 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9adc0caa40 0x7f9adc0cae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:40.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.212+0000 7f9adb7fe700 1 --2- 172.21.15.16:0/2280927380 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9adc0caa40 0x7f9adc0cae90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:40.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.212+0000 7f9adb7fe700 1 -- 172.21.15.16:0/2280927380 >> v2:172.21.15.49:3300/0 conn(0x7f9adc15bb40 msgr2=0x7f9adc06e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:40.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.212+0000 7f9adb7fe700 1 --2- 172.21.15.16:0/2280927380 >> v2:172.21.15.49:3300/0 conn(0x7f9adc15bb40 0x7f9adc06e310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:40.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.212+0000 7f9adb7fe700 1 -- 172.21.15.16:0/2280927380 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9adc15c600 con 0x7f9adc072b30 2023-10-16T12:11:40.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.212+0000 7f9adaffd700 1 --2- 172.21.15.16:0/2280927380 >> v2:172.21.15.49:3300/0 conn(0x7f9adc15bb40 0x7f9adc06e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:11:40.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.212+0000 7f9adb7fe700 1 --2- 172.21.15.16:0/2280927380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9adc072b30 0x7f9adc071a70 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f9adc06b880 tx=0x7f9acc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea0c63d1fead12ea server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:40.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.212+0000 7f9ada7fc700 1 -- 172.21.15.16:0/2280927380 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9acc00b870 con 0x7f9adc072b30 2023-10-16T12:11:40.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.212+0000 7f9ada7fc700 1 -- 172.21.15.16:0/2280927380 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9acc00b9d0 con 0x7f9adc072b30 2023-10-16T12:11:40.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.212+0000 7f9ada7fc700 1 -- 172.21.15.16:0/2280927380 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9acc00bd60 con 0x7f9adc072b30 2023-10-16T12:11:40.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.213+0000 7f9ae1c1f700 1 -- 172.21.15.16:0/2280927380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9adc072b30 msgr2=0x7f9adc071a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:40.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.213+0000 7f9ae1c1f700 1 --2- 172.21.15.16:0/2280927380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9adc072b30 0x7f9adc071a70 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f9adc06b880 tx=0x7f9acc009bd0 comp rx=0 tx=0).stop 2023-10-16T12:11:40.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.213+0000 7f9ae1c1f700 1 -- 172.21.15.16:0/2280927380 shutdown_connections 2023-10-16T12:11:40.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.213+0000 7f9ae1c1f700 1 --2- 172.21.15.16:0/2280927380 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9adc0caa40 0x7f9adc0cae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:40.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.213+0000 7f9ae1c1f700 1 --2- 172.21.15.16:0/2280927380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9adc072b30 0x7f9adc071a70 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:40.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.213+0000 7f9ae1c1f700 1 --2- 172.21.15.16:0/2280927380 >> v2:172.21.15.49:3300/0 conn(0x7f9adc15bb40 0x7f9adc06e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:40.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.213+0000 7f9ae1c1f700 1 -- 172.21.15.16:0/2280927380 >> 172.21.15.16:0/2280927380 conn(0x7f9adc0b1f40 msgr2=0x7f9adc0b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:40.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.213+0000 7f9ae1c1f700 1 -- 172.21.15.16:0/2280927380 shutdown_connections 2023-10-16T12:11:40.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.213+0000 7f9ae1c1f700 1 -- 172.21.15.16:0/2280927380 wait complete. 2023-10-16T12:11:40.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.214+0000 7f9ae1c1f700 1 Processor -- start 2023-10-16T12:11:40.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.214+0000 7f9ae1c1f700 1 -- start start 2023-10-16T12:11:40.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.215+0000 7f9ae1c1f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9adc072b30 0x7f9adc0b5200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:40.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.215+0000 7f9ae1c1f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9adc0caa40 0x7f9adc1528b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:40.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.215+0000 7f9ae1c1f700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9adc15bb40 0x7f9adc0d0380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:40.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.215+0000 7f9ae1c1f700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9adc0cfc20 con 0x7f9adc072b30 2023-10-16T12:11:40.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.215+0000 7f9ae1c1f700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9adc0c5a40 con 0x7f9adc15bb40 2023-10-16T12:11:40.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.215+0000 7f9ae1c1f700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9adc0b5740 con 0x7f9adc0caa40 2023-10-16T12:11:40.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.215+0000 7f9adb7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9adc072b30 0x7f9adc0b5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:40.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.215+0000 7f9adb7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9adc072b30 0x7f9adc0b5200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34868/0 (socket says 172.21.15.16:34868) 2023-10-16T12:11:40.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.215+0000 7f9adb7fe700 1 -- 172.21.15.16:0/1147385816 learned_addr learned my addr 172.21.15.16:0/1147385816 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:40.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.215+0000 7f9adbfff700 1 --2- 172.21.15.16:0/1147385816 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9adc15bb40 0x7f9adc0d0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:40.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.215+0000 7f9adaffd700 1 --2- 172.21.15.16:0/1147385816 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9adc0caa40 0x7f9adc1528b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:40.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.215+0000 7f9adb7fe700 1 -- 172.21.15.16:0/1147385816 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9adc0caa40 msgr2=0x7f9adc1528b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:40.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.215+0000 7f9adb7fe700 1 --2- 172.21.15.16:0/1147385816 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9adc0caa40 0x7f9adc1528b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:40.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.215+0000 7f9adb7fe700 1 -- 172.21.15.16:0/1147385816 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9adc15bb40 msgr2=0x7f9adc0d0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:40.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.215+0000 7f9adb7fe700 1 --2- 172.21.15.16:0/1147385816 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9adc15bb40 0x7f9adc0d0380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:40.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.215+0000 7f9adb7fe700 1 -- 172.21.15.16:0/1147385816 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9acc009550 con 0x7f9adc072b30 2023-10-16T12:11:40.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.215+0000 7f9adbfff700 1 --2- 172.21.15.16:0/1147385816 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9adc15bb40 0x7f9adc0d0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:11:40.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.216+0000 7f9adb7fe700 1 --2- 172.21.15.16:0/1147385816 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9adc072b30 0x7f9adc0b5200 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f9acc014040 tx=0x7f9acc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:40.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.216+0000 7f9ac3fff700 1 -- 172.21.15.16:0/1147385816 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9acc004520 con 0x7f9adc072b30 2023-10-16T12:11:40.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.216+0000 7f9ac3fff700 1 -- 172.21.15.16:0/1147385816 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9acc01e070 con 0x7f9adc072b30 2023-10-16T12:11:40.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.216+0000 7f9ae1c1f700 1 -- 172.21.15.16:0/1147385816 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9adc0b5970 con 0x7f9adc072b30 2023-10-16T12:11:40.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.216+0000 7f9ac3fff700 1 -- 172.21.15.16:0/1147385816 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9acc0038e0 con 0x7f9adc072b30 2023-10-16T12:11:40.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.216+0000 7f9ae1c1f700 1 -- 172.21.15.16:0/1147385816 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9adc0cf560 con 0x7f9adc072b30 2023-10-16T12:11:40.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.217+0000 7f9ae1c1f700 1 -- 172.21.15.16:0/1147385816 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9adc04ebb0 con 0x7f9adc072b30 2023-10-16T12:11:40.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.217+0000 7f9ac3fff700 1 -- 172.21.15.16:0/1147385816 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f9acc003a40 con 0x7f9adc072b30 2023-10-16T12:11:40.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.218+0000 7f9ac3fff700 1 --2- 172.21.15.16:0/1147385816 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f9ac806af00 0x7f9ac806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:40.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.218+0000 7f9ac3fff700 1 -- 172.21.15.16:0/1147385816 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f9acc020070 con 0x7f9adc072b30 2023-10-16T12:11:40.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.218+0000 7f9adaffd700 1 --2- 172.21.15.16:0/1147385816 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f9ac806af00 0x7f9ac806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:40.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.218+0000 7f9adaffd700 1 --2- 172.21.15.16:0/1147385816 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f9ac806af00 0x7f9ac806d3c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9ad0006fd0 tx=0x7f9ad0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:40.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.222+0000 7f9ac3fff700 1 -- 172.21.15.16:0/1147385816 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9acc055d50 con 0x7f9adc072b30 2023-10-16T12:11:40.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.426+0000 7f9ae1c1f700 1 -- 172.21.15.16:0/1147385816 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9adc0b2d40 con 0x7f9adc072b30 2023-10-16T12:11:40.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.427+0000 7f9ac3fff700 1 -- 172.21.15.16:0/1147385816 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9acc015020 con 0x7f9adc072b30 2023-10-16T12:11:40.430 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T12:11:40.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.429+0000 7f9ae1c1f700 1 -- 172.21.15.16:0/1147385816 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f9ac806af00 msgr2=0x7f9ac806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:40.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.429+0000 7f9ae1c1f700 1 --2- 172.21.15.16:0/1147385816 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f9ac806af00 0x7f9ac806d3c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9ad0006fd0 tx=0x7f9ad0008040 comp rx=0 tx=0).stop 2023-10-16T12:11:40.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.429+0000 7f9ae1c1f700 1 -- 172.21.15.16:0/1147385816 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9adc072b30 msgr2=0x7f9adc0b5200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:40.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.429+0000 7f9ae1c1f700 1 --2- 172.21.15.16:0/1147385816 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9adc072b30 0x7f9adc0b5200 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f9acc014040 tx=0x7f9acc004750 comp rx=0 tx=0).stop 2023-10-16T12:11:40.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.429+0000 7f9ae1c1f700 1 -- 172.21.15.16:0/1147385816 shutdown_connections 2023-10-16T12:11:40.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.429+0000 7f9ae1c1f700 1 --2- 172.21.15.16:0/1147385816 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f9ac806af00 0x7f9ac806d3c0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:40.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.429+0000 7f9ae1c1f700 1 --2- 172.21.15.16:0/1147385816 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9adc15bb40 0x7f9adc0d0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:40.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.429+0000 7f9ae1c1f700 1 --2- 172.21.15.16:0/1147385816 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9adc072b30 0x7f9adc0b5200 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:40.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.429+0000 7f9ae1c1f700 1 --2- 172.21.15.16:0/1147385816 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9adc0caa40 0x7f9adc1528b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:40.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.429+0000 7f9ae1c1f700 1 -- 172.21.15.16:0/1147385816 >> 172.21.15.16:0/1147385816 conn(0x7f9adc0b1f40 msgr2=0x7f9adc0b17f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:40.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.429+0000 7f9ae1c1f700 1 -- 172.21.15.16:0/1147385816 shutdown_connections 2023-10-16T12:11:40.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.429+0000 7f9ae1c1f700 1 -- 172.21.15.16:0/1147385816 wait complete. 2023-10-16T12:11:40.444 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T12:11:40.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.569+0000 7f003c911700 1 Processor -- start 2023-10-16T12:11:40.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.569+0000 7f003c911700 1 -- start start 2023-10-16T12:11:40.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.570+0000 7f003c911700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f003806a7e0 0x7f003806ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:40.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.570+0000 7f003c911700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f003815ecb0 con 0x7f003806e570 2023-10-16T12:11:40.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.570+0000 7f003c911700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f003815b6a0 con 0x7f003815b2c0 2023-10-16T12:11:40.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.570+0000 7f003c911700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f003806b190 con 0x7f003806a7e0 2023-10-16T12:11:40.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.570+0000 7f0035d9b700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f003806e570 0x7f003815e5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:52384/0 (socket says 172.21.15.16:52384) 2023-10-16T12:11:40.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.570+0000 7f0035d9b700 1 -- 172.21.15.16:0/3866395680 learned_addr learned my addr 172.21.15.16:0/3866395680 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:40.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.570+0000 7f0036d9d700 1 --2- 172.21.15.16:0/3866395680 >> v2:172.21.15.49:3300/0 conn(0x7f003806a7e0 0x7f003806ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:40.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.570+0000 7f003559a700 1 -- 172.21.15.16:0/3866395680 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1690695930 0 0) 0x7f003815ecb0 con 0x7f003806e570 2023-10-16T12:11:40.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.570+0000 7f003559a700 1 -- 172.21.15.16:0/3866395680 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00200035b0 con 0x7f003806e570 2023-10-16T12:11:40.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.570+0000 7f003559a700 1 -- 172.21.15.16:0/3866395680 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2629432796 0 0) 0x7f003815b6a0 con 0x7f003815b2c0 2023-10-16T12:11:40.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.570+0000 7f003559a700 1 -- 172.21.15.16:0/3866395680 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f003815ecb0 con 0x7f003815b2c0 2023-10-16T12:11:40.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.570+0000 7f003559a700 1 -- 172.21.15.16:0/3866395680 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4264842013 0 0) 0x7f00200035b0 con 0x7f003806e570 2023-10-16T12:11:40.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.570+0000 7f003559a700 1 -- 172.21.15.16:0/3866395680 >> v2:172.21.15.49:3300/0 conn(0x7f003806a7e0 msgr2=0x7f003806ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:40.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.570+0000 7f003559a700 1 --2- 172.21.15.16:0/3866395680 >> v2:172.21.15.49:3300/0 conn(0x7f003806a7e0 0x7f003806ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:40.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.570+0000 7f003559a700 1 -- 172.21.15.16:0/3866395680 >> v1:172.21.15.49:6789/0 conn(0x7f003815b2c0 legacy=0x7f0038161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:40.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.570+0000 7f003559a700 1 -- 172.21.15.16:0/3866395680 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0038161b20 con 0x7f003806e570 2023-10-16T12:11:40.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.570+0000 7f003559a700 1 -- 172.21.15.16:0/3866395680 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f002c001670 con 0x7f003806e570 2023-10-16T12:11:40.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.570+0000 7f003559a700 1 -- 172.21.15.16:0/3866395680 >> v1:172.21.15.16:6789/0 conn(0x7f003806e570 legacy=0x7f003815e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:40.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.571+0000 7f003559a700 1 --2- 172.21.15.16:0/3866395680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0020003b90 0x7f0020003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:40.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.571+0000 7f003559a700 1 --2- 172.21.15.16:0/3866395680 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0020004850 0x7f0020004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:40.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.571+0000 7f003559a700 1 --2- 172.21.15.16:0/3866395680 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0020005210 0x7f002000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:40.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.571+0000 7f003559a700 1 -- 172.21.15.16:0/3866395680 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f003806b190 con 0x7f0020003b90 2023-10-16T12:11:40.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.571+0000 7f003559a700 1 -- 172.21.15.16:0/3866395680 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f002000c140 con 0x7f0020005210 2023-10-16T12:11:40.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.571+0000 7f003559a700 1 -- 172.21.15.16:0/3866395680 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f002000c300 con 0x7f0020004850 2023-10-16T12:11:40.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.571+0000 7f0036d9d700 1 --2- 172.21.15.16:0/3866395680 >> v2:172.21.15.49:3300/0 conn(0x7f003806a7e0 0x7f003806ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:11:40.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.571+0000 7f003659c700 1 --2- 172.21.15.16:0/3866395680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0020003b90 0x7f0020003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:40.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.571+0000 7f0035d9b700 1 --2- 172.21.15.16:0/3866395680 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0020004850 0x7f0020004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:40.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.571+0000 7f0036d9d700 1 --2- 172.21.15.16:0/3866395680 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0020005210 0x7f002000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:40.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.571+0000 7f003659c700 1 -- 172.21.15.16:0/3866395680 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0020004850 msgr2=0x7f0020004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:40.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.571+0000 7f003659c700 1 --2- 172.21.15.16:0/3866395680 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0020004850 0x7f0020004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:40.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.571+0000 7f003659c700 1 -- 172.21.15.16:0/3866395680 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0020005210 msgr2=0x7f002000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:40.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.571+0000 7f003659c700 1 --2- 172.21.15.16:0/3866395680 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0020005210 0x7f002000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:40.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.571+0000 7f003659c700 1 -- 172.21.15.16:0/3866395680 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00200035b0 con 0x7f0020003b90 2023-10-16T12:11:40.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.571+0000 7f0036d9d700 1 -- 172.21.15.16:0/3866395680 reap_dead start 2023-10-16T12:11:40.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.571+0000 7f003659c700 1 --2- 172.21.15.16:0/3866395680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0020003b90 0x7f0020003f90 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f0028001930 tx=0x7f0020005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cdd534f799e1f84 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:40.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.571+0000 7f003559a700 1 -- 172.21.15.16:0/3866395680 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00280052d0 con 0x7f0020003b90 2023-10-16T12:11:40.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.571+0000 7f003559a700 1 -- 172.21.15.16:0/3866395680 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0028005450 con 0x7f0020003b90 2023-10-16T12:11:40.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.571+0000 7f003559a700 1 -- 172.21.15.16:0/3866395680 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00280038c0 con 0x7f0020003b90 2023-10-16T12:11:40.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.572+0000 7f003c911700 1 -- 172.21.15.16:0/3866395680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0020003b90 msgr2=0x7f0020003f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:40.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.572+0000 7f003c911700 1 --2- 172.21.15.16:0/3866395680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0020003b90 0x7f0020003f90 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f0028001930 tx=0x7f0020005880 comp rx=0 tx=0).stop 2023-10-16T12:11:40.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.572+0000 7f003c911700 1 -- 172.21.15.16:0/3866395680 shutdown_connections 2023-10-16T12:11:40.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.572+0000 7f003c911700 1 --2- 172.21.15.16:0/3866395680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0020003b90 0x7f0020003f90 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:40.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.572+0000 7f003c911700 1 -- 172.21.15.16:0/3866395680 >> 172.21.15.16:0/3866395680 conn(0x7f00380b1f40 msgr2=0x7f00380b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:40.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.572+0000 7f003c911700 1 -- 172.21.15.16:0/3866395680 shutdown_connections 2023-10-16T12:11:40.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.572+0000 7f003c911700 1 -- 172.21.15.16:0/3866395680 wait complete. 2023-10-16T12:11:40.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.573+0000 7f003c911700 1 Processor -- start 2023-10-16T12:11:40.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.573+0000 7f003c911700 1 -- start start 2023-10-16T12:11:40.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.574+0000 7f003c911700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0020003b90 0x7f00380ad7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:40.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.574+0000 7f003c911700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f00380ac8d0 0x7f00380aef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:40.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.574+0000 7f003c911700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00380af460 0x7f00380c74c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:40.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.574+0000 7f003c911700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00380aea90 con 0x7f00380af460 2023-10-16T12:11:40.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.574+0000 7f003c911700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f003815a2b0 con 0x7f0020003b90 2023-10-16T12:11:40.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.574+0000 7f003c911700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00380d0bd0 con 0x7f00380ac8d0 2023-10-16T12:11:40.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.574+0000 7f0036d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00380af460 0x7f00380c74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:40.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.574+0000 7f0036d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00380af460 0x7f00380c74c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34892/0 (socket says 172.21.15.16:34892) 2023-10-16T12:11:40.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.574+0000 7f0036d9d700 1 -- 172.21.15.16:0/447162029 learned_addr learned my addr 172.21.15.16:0/447162029 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:40.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.574+0000 7f003659c700 1 --2- 172.21.15.16:0/447162029 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0020003b90 0x7f00380ad7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:40.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.574+0000 7f0035d9b700 1 --2- 172.21.15.16:0/447162029 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f00380ac8d0 0x7f00380aef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:40.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.574+0000 7f0036d9d700 1 -- 172.21.15.16:0/447162029 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f00380ac8d0 msgr2=0x7f00380aef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:40.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.574+0000 7f0036d9d700 1 --2- 172.21.15.16:0/447162029 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f00380ac8d0 0x7f00380aef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:40.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.574+0000 7f0036d9d700 1 -- 172.21.15.16:0/447162029 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0020003b90 msgr2=0x7f00380ad7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:40.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.574+0000 7f0036d9d700 1 --2- 172.21.15.16:0/447162029 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0020003b90 0x7f00380ad7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:40.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.574+0000 7f0036d9d700 1 -- 172.21.15.16:0/447162029 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00380ade80 con 0x7f00380af460 2023-10-16T12:11:40.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.575+0000 7f0036d9d700 1 --2- 172.21.15.16:0/447162029 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00380af460 0x7f00380c74c0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f00380cdf10 tx=0x7f00300114b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:40.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.575+0000 7f0026ffd700 1 -- 172.21.15.16:0/447162029 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f003000e070 con 0x7f00380af460 2023-10-16T12:11:40.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.575+0000 7f0026ffd700 1 -- 172.21.15.16:0/447162029 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0030003cf0 con 0x7f00380af460 2023-10-16T12:11:40.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.575+0000 7f0026ffd700 1 -- 172.21.15.16:0/447162029 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0030003ea0 con 0x7f00380af460 2023-10-16T12:11:40.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.575+0000 7f003c911700 1 -- 172.21.15.16:0/447162029 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00380ae160 con 0x7f00380af460 2023-10-16T12:11:40.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.575+0000 7f003c911700 1 -- 172.21.15.16:0/447162029 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00380ae720 con 0x7f00380af460 2023-10-16T12:11:40.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.576+0000 7f003c911700 1 -- 172.21.15.16:0/447162029 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00380ae8e0 con 0x7f00380af460 2023-10-16T12:11:40.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.576+0000 7f0026ffd700 1 -- 172.21.15.16:0/447162029 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f0030011960 con 0x7f00380af460 2023-10-16T12:11:40.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.577+0000 7f0026ffd700 1 --2- 172.21.15.16:0/447162029 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f001806af00 0x7f001806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:40.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.577+0000 7f0026ffd700 1 -- 172.21.15.16:0/447162029 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f0030089eb0 con 0x7f00380af460 2023-10-16T12:11:40.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.577+0000 7f003659c700 1 --2- 172.21.15.16:0/447162029 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f001806af00 0x7f001806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:40.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.577+0000 7f003659c700 1 --2- 172.21.15.16:0/447162029 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f001806af00 0x7f001806d3c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f0028005040 tx=0x7f0028009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:40.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.581+0000 7f0026ffd700 1 -- 172.21.15.16:0/447162029 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0030054ba0 con 0x7f00380af460 2023-10-16T12:11:40.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.783+0000 7f003c911700 1 -- 172.21.15.16:0/447162029 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f003804ebb0 con 0x7f00380af460 2023-10-16T12:11:40.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.784+0000 7f0026ffd700 1 -- 172.21.15.16:0/447162029 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7f00300581c0 con 0x7f00380af460 2023-10-16T12:11:40.787 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-10-16T12:11:40.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.786+0000 7f003c911700 1 -- 172.21.15.16:0/447162029 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f001806af00 msgr2=0x7f001806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:40.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.786+0000 7f003c911700 1 --2- 172.21.15.16:0/447162029 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f001806af00 0x7f001806d3c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f0028005040 tx=0x7f0028009030 comp rx=0 tx=0).stop 2023-10-16T12:11:40.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.786+0000 7f003c911700 1 -- 172.21.15.16:0/447162029 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00380af460 msgr2=0x7f00380c74c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:40.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.786+0000 7f003c911700 1 --2- 172.21.15.16:0/447162029 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00380af460 0x7f00380c74c0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f00380cdf10 tx=0x7f00300114b0 comp rx=0 tx=0).stop 2023-10-16T12:11:40.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.786+0000 7f003c911700 1 -- 172.21.15.16:0/447162029 shutdown_connections 2023-10-16T12:11:40.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.786+0000 7f003c911700 1 --2- 172.21.15.16:0/447162029 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f001806af00 0x7f001806d3c0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:40.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.786+0000 7f003c911700 1 --2- 172.21.15.16:0/447162029 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00380af460 0x7f00380c74c0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:40.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.786+0000 7f003c911700 1 --2- 172.21.15.16:0/447162029 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0020003b90 0x7f00380ad7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:40.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.786+0000 7f003c911700 1 --2- 172.21.15.16:0/447162029 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f00380ac8d0 0x7f00380aef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:40.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.786+0000 7f003c911700 1 -- 172.21.15.16:0/447162029 >> 172.21.15.16:0/447162029 conn(0x7f00380b1f40 msgr2=0x7f003815e4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:40.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.786+0000 7f003c911700 1 -- 172.21.15.16:0/447162029 shutdown_connections 2023-10-16T12:11:40.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.786+0000 7f003c911700 1 -- 172.21.15.16:0/447162029 wait complete. 2023-10-16T12:11:40.801 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T12:11:40.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.926+0000 7f49f63ef700 1 Processor -- start 2023-10-16T12:11:40.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.927+0000 7f49f63ef700 1 -- start start 2023-10-16T12:11:40.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.927+0000 7f49f63ef700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f49f00cb880 0x7f49f00cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:40.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.927+0000 7f49f63ef700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f49f006da60 0x7f49f006def0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:40.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.927+0000 7f49f63ef700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49f00cef70 con 0x7f49f015b070 2023-10-16T12:11:40.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.927+0000 7f49f63ef700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49f006e430 con 0x7f49f006da60 2023-10-16T12:11:40.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.927+0000 7f49f63ef700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49f00ce360 con 0x7f49f00cb880 2023-10-16T12:11:40.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.928+0000 7f49effff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f49f015b070 0x7f49f015b450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:52402/0 (socket says 172.21.15.16:52402) 2023-10-16T12:11:40.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.928+0000 7f49effff700 1 -- 172.21.15.16:0/1162257777 learned_addr learned my addr 172.21.15.16:0/1162257777 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:40.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.928+0000 7f49f498c700 1 --2- 172.21.15.16:0/1162257777 >> v2:172.21.15.16:3300/0 conn(0x7f49f006da60 0x7f49f006def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:40.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.928+0000 7f49eeffd700 1 -- 172.21.15.16:0/1162257777 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4180396718 0 0) 0x7f49f00cef70 con 0x7f49f015b070 2023-10-16T12:11:40.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.928+0000 7f49eeffd700 1 -- 172.21.15.16:0/1162257777 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49d80035b0 con 0x7f49f015b070 2023-10-16T12:11:40.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.928+0000 7f49ef7fe700 1 --2- 172.21.15.16:0/1162257777 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f49f00cb880 0x7f49f00cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:40.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.928+0000 7f49f498c700 1 -- 172.21.15.16:0/1162257777 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f49f00cb880 msgr2=0x7f49f00cbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:40.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.928+0000 7f49f498c700 1 --2- 172.21.15.16:0/1162257777 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f49f00cb880 0x7f49f00cbd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:40.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.928+0000 7f49f498c700 1 -- 172.21.15.16:0/1162257777 >> v1:172.21.15.16:6789/0 conn(0x7f49f015b070 legacy=0x7f49f015b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:40.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.928+0000 7f49f498c700 1 -- 172.21.15.16:0/1162257777 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49f00ce640 con 0x7f49f006da60 2023-10-16T12:11:40.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.928+0000 7f49ef7fe700 1 --2- 172.21.15.16:0/1162257777 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f49f00cb880 0x7f49f00cbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:11:40.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.928+0000 7f49f498c700 1 --2- 172.21.15.16:0/1162257777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49f006da60 0x7f49f006def0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f49e8009d90 tx=0x7f49e800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a61afce9d352225 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:40.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.928+0000 7f49eeffd700 1 -- 172.21.15.16:0/1162257777 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49e800e070 con 0x7f49f006da60 2023-10-16T12:11:40.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.928+0000 7f49eeffd700 1 -- 172.21.15.16:0/1162257777 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49e800bb20 con 0x7f49f006da60 2023-10-16T12:11:40.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.928+0000 7f49eeffd700 1 -- 172.21.15.16:0/1162257777 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49e8012430 con 0x7f49f006da60 2023-10-16T12:11:40.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.929+0000 7f49f63ef700 1 -- 172.21.15.16:0/1162257777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49f006da60 msgr2=0x7f49f006def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:40.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.929+0000 7f49f63ef700 1 --2- 172.21.15.16:0/1162257777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49f006da60 0x7f49f006def0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f49e8009d90 tx=0x7f49e800b330 comp rx=0 tx=0).stop 2023-10-16T12:11:40.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.929+0000 7f49f63ef700 1 -- 172.21.15.16:0/1162257777 shutdown_connections 2023-10-16T12:11:40.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.929+0000 7f49f63ef700 1 --2- 172.21.15.16:0/1162257777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49f006da60 0x7f49f006def0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:40.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.929+0000 7f49f63ef700 1 --2- 172.21.15.16:0/1162257777 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f49f00cb880 0x7f49f00cbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:40.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.929+0000 7f49f63ef700 1 -- 172.21.15.16:0/1162257777 >> 172.21.15.16:0/1162257777 conn(0x7f49f00b1f40 msgr2=0x7f49f00b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:40.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.929+0000 7f49f63ef700 1 -- 172.21.15.16:0/1162257777 shutdown_connections 2023-10-16T12:11:40.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.929+0000 7f49f63ef700 1 -- 172.21.15.16:0/1162257777 wait complete. 2023-10-16T12:11:40.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.930+0000 7f49f63ef700 1 Processor -- start 2023-10-16T12:11:40.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.931+0000 7f49f63ef700 1 -- start start 2023-10-16T12:11:40.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.932+0000 7f49f63ef700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f49f006da60 0x7f49f00c9330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:40.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.932+0000 7f49f63ef700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49f00cb880 0x7f49f01f4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:40.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.932+0000 7f49f63ef700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f49f015b070 0x7f49f00c6900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:40.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.932+0000 7f49f63ef700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49f01f5130 con 0x7f49f00cb880 2023-10-16T12:11:40.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.932+0000 7f49f63ef700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49f00ca620 con 0x7f49f006da60 2023-10-16T12:11:40.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.932+0000 7f49f63ef700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49f01f0b00 con 0x7f49f015b070 2023-10-16T12:11:40.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.932+0000 7f49ef7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49f00cb880 0x7f49f01f4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:40.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.932+0000 7f49ef7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49f00cb880 0x7f49f01f4bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34904/0 (socket says 172.21.15.16:34904) 2023-10-16T12:11:40.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.932+0000 7f49ef7fe700 1 -- 172.21.15.16:0/1190474126 learned_addr learned my addr 172.21.15.16:0/1190474126 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:40.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.932+0000 7f49effff700 1 --2- 172.21.15.16:0/1190474126 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f49f006da60 0x7f49f00c9330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:40.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.932+0000 7f49ef7fe700 1 -- 172.21.15.16:0/1190474126 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f49f015b070 msgr2=0x7f49f00c6900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:40.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.932+0000 7f49f498c700 1 --2- 172.21.15.16:0/1190474126 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f49f015b070 0x7f49f00c6900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:40.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.932+0000 7f49ef7fe700 1 --2- 172.21.15.16:0/1190474126 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f49f015b070 0x7f49f00c6900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:40.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.932+0000 7f49ef7fe700 1 -- 172.21.15.16:0/1190474126 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f49f006da60 msgr2=0x7f49f00c9330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:40.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.932+0000 7f49ef7fe700 1 --2- 172.21.15.16:0/1190474126 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f49f006da60 0x7f49f00c9330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:40.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.932+0000 7f49ef7fe700 1 -- 172.21.15.16:0/1190474126 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49e8009a70 con 0x7f49f00cb880 2023-10-16T12:11:40.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.932+0000 7f49effff700 1 --2- 172.21.15.16:0/1190474126 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f49f006da60 0x7f49f00c9330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:11:40.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.932+0000 7f49f498c700 1 --2- 172.21.15.16:0/1190474126 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f49f015b070 0x7f49f00c6900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:11:40.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.932+0000 7f49ef7fe700 1 --2- 172.21.15.16:0/1190474126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49f00cb880 0x7f49f01f4bf0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f49e400c750 tx=0x7f49e400cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:40.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.932+0000 7f49ecff9700 1 -- 172.21.15.16:0/1190474126 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49e4011800 con 0x7f49f00cb880 2023-10-16T12:11:40.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.932+0000 7f49ecff9700 1 -- 172.21.15.16:0/1190474126 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49e4011960 con 0x7f49f00cb880 2023-10-16T12:11:40.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.932+0000 7f49f63ef700 1 -- 172.21.15.16:0/1190474126 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49f00c9a10 con 0x7f49f00cb880 2023-10-16T12:11:40.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.932+0000 7f49ecff9700 1 -- 172.21.15.16:0/1190474126 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49e4011cd0 con 0x7f49f00cb880 2023-10-16T12:11:40.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.932+0000 7f49f63ef700 1 -- 172.21.15.16:0/1190474126 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49f00c9fd0 con 0x7f49f00cb880 2023-10-16T12:11:40.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.933+0000 7f49f63ef700 1 -- 172.21.15.16:0/1190474126 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49f00ca7b0 con 0x7f49f00cb880 2023-10-16T12:11:40.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.934+0000 7f49ecff9700 1 -- 172.21.15.16:0/1190474126 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f49e4021350 con 0x7f49f00cb880 2023-10-16T12:11:40.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.934+0000 7f49ecff9700 1 --2- 172.21.15.16:0/1190474126 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f49dc06af00 0x7f49dc06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:40.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.935+0000 7f49effff700 1 --2- 172.21.15.16:0/1190474126 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f49dc06af00 0x7f49dc06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:40.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.935+0000 7f49ecff9700 1 -- 172.21.15.16:0/1190474126 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f49e4089930 con 0x7f49f00cb880 2023-10-16T12:11:40.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.935+0000 7f49effff700 1 --2- 172.21.15.16:0/1190474126 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f49dc06af00 0x7f49dc06d3c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f49e00011e0 tx=0x7f49e0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:40.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:40.939+0000 7f49ecff9700 1 -- 172.21.15.16:0/1190474126 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f49e4054620 con 0x7f49f00cb880 2023-10-16T12:11:41.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.141+0000 7f49f63ef700 1 -- 172.21.15.16:0/1190474126 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f49f004ebb0 con 0x7f49f00cb880 2023-10-16T12:11:41.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.141+0000 7f49ecff9700 1 -- 172.21.15.16:0/1190474126 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7f49e405e070 con 0x7f49f00cb880 2023-10-16T12:11:41.144 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-10-16T12:11:41.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.143+0000 7f49f63ef700 1 -- 172.21.15.16:0/1190474126 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f49dc06af00 msgr2=0x7f49dc06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:41.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.143+0000 7f49f63ef700 1 --2- 172.21.15.16:0/1190474126 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f49dc06af00 0x7f49dc06d3c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f49e00011e0 tx=0x7f49e0009040 comp rx=0 tx=0).stop 2023-10-16T12:11:41.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.143+0000 7f49f63ef700 1 -- 172.21.15.16:0/1190474126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49f00cb880 msgr2=0x7f49f01f4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:41.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.143+0000 7f49f63ef700 1 --2- 172.21.15.16:0/1190474126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49f00cb880 0x7f49f01f4bf0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f49e400c750 tx=0x7f49e400cb10 comp rx=0 tx=0).stop 2023-10-16T12:11:41.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.143+0000 7f49f63ef700 1 -- 172.21.15.16:0/1190474126 shutdown_connections 2023-10-16T12:11:41.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.143+0000 7f49f63ef700 1 --2- 172.21.15.16:0/1190474126 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f49dc06af00 0x7f49dc06d3c0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:41.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.143+0000 7f49f63ef700 1 --2- 172.21.15.16:0/1190474126 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f49f015b070 0x7f49f00c6900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:41.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.143+0000 7f49f63ef700 1 --2- 172.21.15.16:0/1190474126 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f49f006da60 0x7f49f00c9330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:41.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.143+0000 7f49f63ef700 1 --2- 172.21.15.16:0/1190474126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49f00cb880 0x7f49f01f4bf0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:41.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.143+0000 7f49f63ef700 1 -- 172.21.15.16:0/1190474126 >> 172.21.15.16:0/1190474126 conn(0x7f49f00b1f40 msgr2=0x7f49f01544a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:41.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.144+0000 7f49f63ef700 1 -- 172.21.15.16:0/1190474126 shutdown_connections 2023-10-16T12:11:41.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.144+0000 7f49f63ef700 1 -- 172.21.15.16:0/1190474126 wait complete. 2023-10-16T12:11:41.158 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-16T12:11:41.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.285+0000 7f4af22a8700 1 Processor -- start 2023-10-16T12:11:41.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.285+0000 7f4af22a8700 1 -- start start 2023-10-16T12:11:41.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.286+0000 7f4af22a8700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4aec06e170 0x7f4aec0cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:41.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.286+0000 7f4af22a8700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4aec0cef70 con 0x7f4aec15b070 2023-10-16T12:11:41.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.286+0000 7f4af22a8700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4aec0ce360 con 0x7f4aec0cb880 2023-10-16T12:11:41.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.286+0000 7f4af22a8700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4aec0ce640 con 0x7f4aec06e170 2023-10-16T12:11:41.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.286+0000 7f4aebfff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f4aec15b070 0x7f4aec15b450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:52416/0 (socket says 172.21.15.16:52416) 2023-10-16T12:11:41.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.286+0000 7f4aebfff700 1 -- 172.21.15.16:0/400435213 learned_addr learned my addr 172.21.15.16:0/400435213 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:41.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.286+0000 7f4af0845700 1 --2- 172.21.15.16:0/400435213 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4aec06e170 0x7f4aec0cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:41.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.286+0000 7f4aeaffd700 1 -- 172.21.15.16:0/400435213 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1478438057 0 0) 0x7f4aec0cef70 con 0x7f4aec15b070 2023-10-16T12:11:41.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.286+0000 7f4aeaffd700 1 -- 172.21.15.16:0/400435213 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ad80035b0 con 0x7f4aec15b070 2023-10-16T12:11:41.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.286+0000 7f4aeaffd700 1 -- 172.21.15.16:0/400435213 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1857613901 0 0) 0x7f4aec0ce360 con 0x7f4aec0cb880 2023-10-16T12:11:41.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.286+0000 7f4aeaffd700 1 -- 172.21.15.16:0/400435213 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4aec0cef70 con 0x7f4aec0cb880 2023-10-16T12:11:41.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.286+0000 7f4af0845700 1 -- 172.21.15.16:0/400435213 >> v1:172.21.15.49:6789/0 conn(0x7f4aec0cb880 legacy=0x7f4aec06da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:41.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.286+0000 7f4af0845700 1 -- 172.21.15.16:0/400435213 >> v1:172.21.15.16:6789/0 conn(0x7f4aec15b070 legacy=0x7f4aec15b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:41.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.286+0000 7f4aeaffd700 1 -- 172.21.15.16:0/400435213 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1797163661 0 0) 0x7f4ad80035b0 con 0x7f4aec15b070 2023-10-16T12:11:41.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.286+0000 7f4af0845700 1 -- 172.21.15.16:0/400435213 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aec0ce920 con 0x7f4aec06e170 2023-10-16T12:11:41.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.287+0000 7f4af0845700 1 --2- 172.21.15.16:0/400435213 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4aec06e170 0x7f4aec0cbd00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4ae4009dc0 tx=0x7f4ae400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=541786c2d8c41a39 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:41.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.287+0000 7f4aeaffd700 1 -- 172.21.15.16:0/400435213 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ae400e070 con 0x7f4aec06e170 2023-10-16T12:11:41.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.287+0000 7f4aeaffd700 1 -- 172.21.15.16:0/400435213 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ae4003a20 con 0x7f4aec06e170 2023-10-16T12:11:41.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.287+0000 7f4aeaffd700 1 -- 172.21.15.16:0/400435213 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ad4001670 con 0x7f4aec06e170 2023-10-16T12:11:41.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.287+0000 7f4af22a8700 1 -- 172.21.15.16:0/400435213 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4aec06e170 msgr2=0x7f4aec0cbd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:41.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.287+0000 7f4af22a8700 1 --2- 172.21.15.16:0/400435213 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4aec06e170 0x7f4aec0cbd00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4ae4009dc0 tx=0x7f4ae400b330 comp rx=0 tx=0).stop 2023-10-16T12:11:41.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.287+0000 7f4af22a8700 1 -- 172.21.15.16:0/400435213 shutdown_connections 2023-10-16T12:11:41.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.287+0000 7f4af22a8700 1 --2- 172.21.15.16:0/400435213 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4aec06e170 0x7f4aec0cbd00 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:41.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.287+0000 7f4af22a8700 1 -- 172.21.15.16:0/400435213 >> 172.21.15.16:0/400435213 conn(0x7f4aec0b1f40 msgr2=0x7f4aec0b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:41.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.287+0000 7f4af22a8700 1 -- 172.21.15.16:0/400435213 shutdown_connections 2023-10-16T12:11:41.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.288+0000 7f4af22a8700 1 -- 172.21.15.16:0/400435213 wait complete. 2023-10-16T12:11:41.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.289+0000 7f4af22a8700 1 Processor -- start 2023-10-16T12:11:41.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.289+0000 7f4af22a8700 1 -- start start 2023-10-16T12:11:41.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.290+0000 7f4af22a8700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4aec06e170 0x7f4aec15d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:41.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.290+0000 7f4af22a8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4aec0cb880 0x7f4aec15c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:41.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.290+0000 7f4af22a8700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4aec15b070 0x7f4aec1554e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:41.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.290+0000 7f4af22a8700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aec152e90 con 0x7f4aec0cb880 2023-10-16T12:11:41.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.290+0000 7f4af22a8700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aec0b3dc0 con 0x7f4aec15b070 2023-10-16T12:11:41.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.290+0000 7f4af22a8700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4aec0b31b0 con 0x7f4aec06e170 2023-10-16T12:11:41.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.290+0000 7f4aeb7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4aec0cb880 0x7f4aec15c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:41.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.290+0000 7f4aeb7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4aec0cb880 0x7f4aec15c710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34916/0 (socket says 172.21.15.16:34916) 2023-10-16T12:11:41.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.290+0000 7f4aeb7fe700 1 -- 172.21.15.16:0/1247792235 learned_addr learned my addr 172.21.15.16:0/1247792235 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:41.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.290+0000 7f4af0845700 1 --2- 172.21.15.16:0/1247792235 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4aec15b070 0x7f4aec1554e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:41.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.290+0000 7f4aebfff700 1 --2- 172.21.15.16:0/1247792235 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4aec06e170 0x7f4aec15d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:41.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.290+0000 7f4aeb7fe700 1 -- 172.21.15.16:0/1247792235 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4aec06e170 msgr2=0x7f4aec15d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:41.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.290+0000 7f4aeb7fe700 1 --2- 172.21.15.16:0/1247792235 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4aec06e170 0x7f4aec15d790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:41.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.290+0000 7f4aeb7fe700 1 -- 172.21.15.16:0/1247792235 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4aec15b070 msgr2=0x7f4aec1554e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:41.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.290+0000 7f4aeb7fe700 1 --2- 172.21.15.16:0/1247792235 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4aec15b070 0x7f4aec1554e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:41.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.290+0000 7f4aeb7fe700 1 -- 172.21.15.16:0/1247792235 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ae4009a70 con 0x7f4aec0cb880 2023-10-16T12:11:41.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.290+0000 7f4af0845700 1 --2- 172.21.15.16:0/1247792235 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4aec15b070 0x7f4aec1554e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:11:41.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.290+0000 7f4aeb7fe700 1 --2- 172.21.15.16:0/1247792235 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4aec0cb880 0x7f4aec15c710 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f4adc00e940 tx=0x7f4adc00ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:41.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.290+0000 7f4ae8ff9700 1 -- 172.21.15.16:0/1247792235 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4adc0052d0 con 0x7f4aec0cb880 2023-10-16T12:11:41.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.290+0000 7f4ae8ff9700 1 -- 172.21.15.16:0/1247792235 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4adc005430 con 0x7f4aec0cb880 2023-10-16T12:11:41.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.290+0000 7f4af22a8700 1 -- 172.21.15.16:0/1247792235 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4aec0b3490 con 0x7f4aec0cb880 2023-10-16T12:11:41.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.290+0000 7f4ae8ff9700 1 -- 172.21.15.16:0/1247792235 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4adc00cc30 con 0x7f4aec0cb880 2023-10-16T12:11:41.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.290+0000 7f4af22a8700 1 -- 172.21.15.16:0/1247792235 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4aec0b3a50 con 0x7f4aec0cb880 2023-10-16T12:11:41.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.291+0000 7f4af22a8700 1 -- 172.21.15.16:0/1247792235 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4aec0b3f50 con 0x7f4aec0cb880 2023-10-16T12:11:41.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.292+0000 7f4ae8ff9700 1 -- 172.21.15.16:0/1247792235 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f4adc003680 con 0x7f4aec0cb880 2023-10-16T12:11:41.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.292+0000 7f4ae8ff9700 1 --2- 172.21.15.16:0/1247792235 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f4acc06af00 0x7f4acc06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:41.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.292+0000 7f4ae8ff9700 1 -- 172.21.15.16:0/1247792235 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f4adc08bb60 con 0x7f4aec0cb880 2023-10-16T12:11:41.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.292+0000 7f4aebfff700 1 --2- 172.21.15.16:0/1247792235 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f4acc06af00 0x7f4acc06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:41.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.293+0000 7f4aebfff700 1 --2- 172.21.15.16:0/1247792235 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f4acc06af00 0x7f4acc06d3c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f4ad4000f60 tx=0x7f4ad4001490 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:41.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.297+0000 7f4ae8ff9700 1 -- 172.21.15.16:0/1247792235 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4adc0568e0 con 0x7f4aec0cb880 2023-10-16T12:11:41.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.502+0000 7f4af22a8700 1 -- 172.21.15.16:0/1247792235 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f4aec0b3c10 con 0x7f4aec0cb880 2023-10-16T12:11:41.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.503+0000 7f4ae8ff9700 1 -- 172.21.15.16:0/1247792235 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f4adc01f020 con 0x7f4aec0cb880 2023-10-16T12:11:41.506 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-10-16T12:11:41.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.505+0000 7f4af22a8700 1 -- 172.21.15.16:0/1247792235 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f4acc06af00 msgr2=0x7f4acc06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:41.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.505+0000 7f4af22a8700 1 --2- 172.21.15.16:0/1247792235 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f4acc06af00 0x7f4acc06d3c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f4ad4000f60 tx=0x7f4ad4001490 comp rx=0 tx=0).stop 2023-10-16T12:11:41.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.505+0000 7f4af22a8700 1 -- 172.21.15.16:0/1247792235 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4aec0cb880 msgr2=0x7f4aec15c710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:41.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.505+0000 7f4af22a8700 1 --2- 172.21.15.16:0/1247792235 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4aec0cb880 0x7f4aec15c710 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f4adc00e940 tx=0x7f4adc00ed00 comp rx=0 tx=0).stop 2023-10-16T12:11:41.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.505+0000 7f4af22a8700 1 -- 172.21.15.16:0/1247792235 shutdown_connections 2023-10-16T12:11:41.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.505+0000 7f4af22a8700 1 --2- 172.21.15.16:0/1247792235 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f4acc06af00 0x7f4acc06d3c0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:41.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.505+0000 7f4af22a8700 1 --2- 172.21.15.16:0/1247792235 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4aec15b070 0x7f4aec1554e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:41.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.505+0000 7f4af22a8700 1 --2- 172.21.15.16:0/1247792235 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4aec06e170 0x7f4aec15d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:41.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.505+0000 7f4af22a8700 1 --2- 172.21.15.16:0/1247792235 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4aec0cb880 0x7f4aec15c710 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:41.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.505+0000 7f4af22a8700 1 -- 172.21.15.16:0/1247792235 >> 172.21.15.16:0/1247792235 conn(0x7f4aec0b1f40 msgr2=0x7f4aec153040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:41.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.505+0000 7f4af22a8700 1 -- 172.21.15.16:0/1247792235 shutdown_connections 2023-10-16T12:11:41.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.505+0000 7f4af22a8700 1 -- 172.21.15.16:0/1247792235 wait complete. 2023-10-16T12:11:41.520 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T12:11:41.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.644+0000 7f445b683700 1 Processor -- start 2023-10-16T12:11:41.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.644+0000 7f445b683700 1 -- start start 2023-10-16T12:11:41.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.645+0000 7f445b683700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f445415b280 0x7f44541613e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:41.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.645+0000 7f445b683700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4454161c30 con 0x7f44540cc8a0 2023-10-16T12:11:41.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.645+0000 7f445b683700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4454161f10 con 0x7f445406a7e0 2023-10-16T12:11:41.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.645+0000 7f445b683700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44541621f0 con 0x7f445415b280 2023-10-16T12:11:41.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.645+0000 7f4458c1e700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f44540cc8a0 0x7f445415e700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:52428/0 (socket says 172.21.15.16:52428) 2023-10-16T12:11:41.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.645+0000 7f4458c1e700 1 -- 172.21.15.16:0/1664722488 learned_addr learned my addr 172.21.15.16:0/1664722488 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:41.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.645+0000 7f445941f700 1 --2- 172.21.15.16:0/1664722488 >> v2:172.21.15.49:3300/0 conn(0x7f445415b280 0x7f44541613e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:41.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.645+0000 7f444bfff700 1 -- 172.21.15.16:0/1664722488 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2313203702 0 0) 0x7f4454161c30 con 0x7f44540cc8a0 2023-10-16T12:11:41.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.645+0000 7f444bfff700 1 -- 172.21.15.16:0/1664722488 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44400035b0 con 0x7f44540cc8a0 2023-10-16T12:11:41.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.645+0000 7f444bfff700 1 -- 172.21.15.16:0/1664722488 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2261181391 0 0) 0x7f4454161f10 con 0x7f445406a7e0 2023-10-16T12:11:41.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.645+0000 7f444bfff700 1 -- 172.21.15.16:0/1664722488 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4454161c30 con 0x7f445406a7e0 2023-10-16T12:11:41.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.646+0000 7f445941f700 1 -- 172.21.15.16:0/1664722488 >> v1:172.21.15.49:6789/0 conn(0x7f445406a7e0 legacy=0x7f44540d02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:41.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.646+0000 7f445941f700 1 -- 172.21.15.16:0/1664722488 >> v1:172.21.15.16:6789/0 conn(0x7f44540cc8a0 legacy=0x7f445415e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:41.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.646+0000 7f445941f700 1 -- 172.21.15.16:0/1664722488 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f445415df00 con 0x7f445415b280 2023-10-16T12:11:41.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.646+0000 7f445941f700 1 --2- 172.21.15.16:0/1664722488 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f445415b280 0x7f44541613e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f4450009840 tx=0x7f4450009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8069f727d81cf72e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:41.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.646+0000 7f444bfff700 1 -- 172.21.15.16:0/1664722488 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4450011830 con 0x7f445415b280 2023-10-16T12:11:41.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.646+0000 7f444bfff700 1 -- 172.21.15.16:0/1664722488 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4450011990 con 0x7f445415b280 2023-10-16T12:11:41.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.646+0000 7f444bfff700 1 -- 172.21.15.16:0/1664722488 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4450011d20 con 0x7f445415b280 2023-10-16T12:11:41.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.646+0000 7f445b683700 1 -- 172.21.15.16:0/1664722488 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f445415b280 msgr2=0x7f44541613e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:41.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.647+0000 7f445b683700 1 --2- 172.21.15.16:0/1664722488 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f445415b280 0x7f44541613e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f4450009840 tx=0x7f4450009b50 comp rx=0 tx=0).stop 2023-10-16T12:11:41.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.647+0000 7f445b683700 1 -- 172.21.15.16:0/1664722488 shutdown_connections 2023-10-16T12:11:41.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.647+0000 7f445b683700 1 --2- 172.21.15.16:0/1664722488 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f445415b280 0x7f44541613e0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:41.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.647+0000 7f445b683700 1 -- 172.21.15.16:0/1664722488 >> 172.21.15.16:0/1664722488 conn(0x7f44540b1fd0 msgr2=0x7f44540b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:41.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.647+0000 7f445b683700 1 -- 172.21.15.16:0/1664722488 shutdown_connections 2023-10-16T12:11:41.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.647+0000 7f445b683700 1 -- 172.21.15.16:0/1664722488 wait complete. 2023-10-16T12:11:41.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.648+0000 7f445b683700 1 Processor -- start 2023-10-16T12:11:41.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.648+0000 7f445b683700 1 -- start start 2023-10-16T12:11:41.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.649+0000 7f445b683700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f445406a7e0 0x7f44540c68c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:41.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.649+0000 7f445b683700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44540cc8a0 0x7f4454157d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:41.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.649+0000 7f445b683700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f445415b280 0x7f44540c4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:41.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.649+0000 7f445b683700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44540c6430 con 0x7f44540cc8a0 2023-10-16T12:11:41.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.649+0000 7f445b683700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44540c5820 con 0x7f445406a7e0 2023-10-16T12:11:41.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.649+0000 7f445b683700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44540c5b00 con 0x7f445415b280 2023-10-16T12:11:41.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.649+0000 7f4458c1e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44540cc8a0 0x7f4454157d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:41.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.649+0000 7f4458c1e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44540cc8a0 0x7f4454157d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34930/0 (socket says 172.21.15.16:34930) 2023-10-16T12:11:41.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.649+0000 7f4458c1e700 1 -- 172.21.15.16:0/3481758373 learned_addr learned my addr 172.21.15.16:0/3481758373 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:41.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.649+0000 7f445941f700 1 --2- 172.21.15.16:0/3481758373 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f445406a7e0 0x7f44540c68c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:41.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.649+0000 7f4459c20700 1 --2- 172.21.15.16:0/3481758373 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f445415b280 0x7f44540c4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:41.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.649+0000 7f4458c1e700 1 -- 172.21.15.16:0/3481758373 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f445415b280 msgr2=0x7f44540c4eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:41.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.649+0000 7f4458c1e700 1 --2- 172.21.15.16:0/3481758373 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f445415b280 0x7f44540c4eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:41.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.649+0000 7f4458c1e700 1 -- 172.21.15.16:0/3481758373 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f445406a7e0 msgr2=0x7f44540c68c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:41.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.649+0000 7f4458c1e700 1 --2- 172.21.15.16:0/3481758373 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f445406a7e0 0x7f44540c68c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:41.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.649+0000 7f4458c1e700 1 -- 172.21.15.16:0/3481758373 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44500094a0 con 0x7f44540cc8a0 2023-10-16T12:11:41.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.649+0000 7f445941f700 1 --2- 172.21.15.16:0/3481758373 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f445406a7e0 0x7f44540c68c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:11:41.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.649+0000 7f4458c1e700 1 --2- 172.21.15.16:0/3481758373 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44540cc8a0 0x7f4454157d70 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f4444005910 tx=0x7f444400ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:41.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.649+0000 7f4449ffb700 1 -- 172.21.15.16:0/3481758373 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4444005d70 con 0x7f44540cc8a0 2023-10-16T12:11:41.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.649+0000 7f4449ffb700 1 -- 172.21.15.16:0/3481758373 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44440070d0 con 0x7f44540cc8a0 2023-10-16T12:11:41.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.649+0000 7f445b683700 1 -- 172.21.15.16:0/3481758373 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44540c5de0 con 0x7f44540cc8a0 2023-10-16T12:11:41.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.649+0000 7f4449ffb700 1 -- 172.21.15.16:0/3481758373 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4444007480 con 0x7f44540cc8a0 2023-10-16T12:11:41.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.650+0000 7f445b683700 1 -- 172.21.15.16:0/3481758373 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44541582b0 con 0x7f44540cc8a0 2023-10-16T12:11:41.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.651+0000 7f445b683700 1 -- 172.21.15.16:0/3481758373 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4454158440 con 0x7f44540cc8a0 2023-10-16T12:11:41.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.651+0000 7f4449ffb700 1 -- 172.21.15.16:0/3481758373 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f44440075e0 con 0x7f44540cc8a0 2023-10-16T12:11:41.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.651+0000 7f4449ffb700 1 --2- 172.21.15.16:0/3481758373 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f443806af00 0x7f443806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:41.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.652+0000 7f4449ffb700 1 -- 172.21.15.16:0/3481758373 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f4444088fd0 con 0x7f44540cc8a0 2023-10-16T12:11:41.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.652+0000 7f445941f700 1 --2- 172.21.15.16:0/3481758373 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f443806af00 0x7f443806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:41.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.652+0000 7f445941f700 1 --2- 172.21.15.16:0/3481758373 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f443806af00 0x7f443806d3c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f4450009fd0 tx=0x7f445001c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:41.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.656+0000 7f4449ffb700 1 -- 172.21.15.16:0/3481758373 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4444053c20 con 0x7f44540cc8a0 2023-10-16T12:11:41.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.876+0000 7f445b683700 1 -- 172.21.15.16:0/3481758373 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f445404ebb0 con 0x7f44540cc8a0 2023-10-16T12:11:41.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.877+0000 7f4449ffb700 1 -- 172.21.15.16:0/3481758373 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) v1 ==== 80+0+0 (secure 0 0 0) 0x7f4444017080 con 0x7f44540cc8a0 2023-10-16T12:11:41.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.879+0000 7f445b683700 1 -- 172.21.15.16:0/3481758373 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f443806af00 msgr2=0x7f443806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:41.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.879+0000 7f445b683700 1 --2- 172.21.15.16:0/3481758373 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f443806af00 0x7f443806d3c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f4450009fd0 tx=0x7f445001c040 comp rx=0 tx=0).stop 2023-10-16T12:11:41.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.879+0000 7f445b683700 1 -- 172.21.15.16:0/3481758373 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44540cc8a0 msgr2=0x7f4454157d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:41.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.879+0000 7f445b683700 1 --2- 172.21.15.16:0/3481758373 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44540cc8a0 0x7f4454157d70 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f4444005910 tx=0x7f444400ca90 comp rx=0 tx=0).stop 2023-10-16T12:11:41.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.879+0000 7f445b683700 1 -- 172.21.15.16:0/3481758373 shutdown_connections 2023-10-16T12:11:41.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.879+0000 7f445b683700 1 --2- 172.21.15.16:0/3481758373 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f443806af00 0x7f443806d3c0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:41.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.879+0000 7f445b683700 1 --2- 172.21.15.16:0/3481758373 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f445415b280 0x7f44540c4eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:41.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.879+0000 7f445b683700 1 --2- 172.21.15.16:0/3481758373 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f445406a7e0 0x7f44540c68c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:41.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.879+0000 7f445b683700 1 --2- 172.21.15.16:0/3481758373 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44540cc8a0 0x7f4454157d70 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:41.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.879+0000 7f445b683700 1 -- 172.21.15.16:0/3481758373 >> 172.21.15.16:0/3481758373 conn(0x7f44540b1fd0 msgr2=0x7f4454161150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:41.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.879+0000 7f445b683700 1 -- 172.21.15.16:0/3481758373 shutdown_connections 2023-10-16T12:11:41.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:41.879+0000 7f445b683700 1 -- 172.21.15.16:0/3481758373 wait complete. 2023-10-16T12:11:41.885 INFO:teuthology.orchestra.run.smithi016.stderr:listed 0 entries 2023-10-16T12:11:41.894 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.2... 2023-10-16T12:11:41.895 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:41.896 DEBUG:teuthology.orchestra.run.smithi016:> sudo umount /home/ubuntu/cephtest/mnt.2 -f 2023-10-16T12:11:41.937 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2023-10-16T12:11:41.937 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:41.938 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-10-16T12:11:41.997 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T12:11:41.999 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:11:42.021 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T12:11:42.022 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:11:42.083 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T12:11:42.084 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:11:42.145 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:11:42.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.268+0000 7f40d8411700 1 Processor -- start 2023-10-16T12:11:42.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.268+0000 7f40d8411700 1 -- start start 2023-10-16T12:11:42.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.269+0000 7f40d8411700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f40d00cf4a0 0x7f40d00726e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:42.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.269+0000 7f40d8411700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f40d00cf880 0x7f40d00b52f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:42.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.269+0000 7f40d8411700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40d015bc80 con 0x7f40d00b5830 2023-10-16T12:11:42.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.269+0000 7f40d8411700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40d0072c20 con 0x7f40d00cf4a0 2023-10-16T12:11:42.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.269+0000 7f40d8411700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40d015b070 con 0x7f40d00cf880 2023-10-16T12:11:42.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.269+0000 7f40d59ac700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f40d00cf880 0x7f40d00b52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:42.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.269+0000 7f40d59ac700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f40d00cf880 0x7f40d00b52f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:51426/0 (socket says 172.21.15.16:51426) 2023-10-16T12:11:42.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.269+0000 7f40d59ac700 1 -- 172.21.15.16:0/3540671818 learned_addr learned my addr 172.21.15.16:0/3540671818 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:42.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.270+0000 7f40d61ad700 1 --2- 172.21.15.16:0/3540671818 >> v2:172.21.15.49:3300/0 conn(0x7f40d00cf4a0 0x7f40d00726e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:42.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.270+0000 7f40d59ac700 1 -- 172.21.15.16:0/3540671818 >> v2:172.21.15.49:3300/0 conn(0x7f40d00cf4a0 msgr2=0x7f40d00726e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:42.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.270+0000 7f40d59ac700 1 --2- 172.21.15.16:0/3540671818 >> v2:172.21.15.49:3300/0 conn(0x7f40d00cf4a0 0x7f40d00726e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:42.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.270+0000 7f40d59ac700 1 -- 172.21.15.16:0/3540671818 >> v1:172.21.15.49:6789/0 conn(0x7f40d00b5830 legacy=0x7f40d015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:42.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.270+0000 7f40d51ab700 1 -- 172.21.15.16:0/3540671818 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3234492416 0 0) 0x7f40d015bc80 con 0x7f40d00b5830 2023-10-16T12:11:42.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.270+0000 7f40d59ac700 1 -- 172.21.15.16:0/3540671818 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40d015b350 con 0x7f40d00cf880 2023-10-16T12:11:42.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.270+0000 7f40d59ac700 1 --2- 172.21.15.16:0/3540671818 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f40d00cf880 0x7f40d00b52f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f40cc0098f0 tx=0x7f40cc009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4852953d2a6b4a4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:42.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.270+0000 7f40d51ab700 1 -- 172.21.15.16:0/3540671818 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40cc011830 con 0x7f40d00cf880 2023-10-16T12:11:42.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.270+0000 7f40d51ab700 1 -- 172.21.15.16:0/3540671818 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40cc011990 con 0x7f40d00cf880 2023-10-16T12:11:42.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.270+0000 7f40d51ab700 1 -- 172.21.15.16:0/3540671818 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40cc011d20 con 0x7f40d00cf880 2023-10-16T12:11:42.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.271+0000 7f40d8411700 1 -- 172.21.15.16:0/3540671818 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f40d00cf880 msgr2=0x7f40d00b52f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:42.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.271+0000 7f40d8411700 1 --2- 172.21.15.16:0/3540671818 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f40d00cf880 0x7f40d00b52f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f40cc0098f0 tx=0x7f40cc009c00 comp rx=0 tx=0).stop 2023-10-16T12:11:42.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.271+0000 7f40d8411700 1 -- 172.21.15.16:0/3540671818 shutdown_connections 2023-10-16T12:11:42.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.271+0000 7f40d8411700 1 --2- 172.21.15.16:0/3540671818 >> v2:172.21.15.49:3300/0 conn(0x7f40d00cf4a0 0x7f40d00726e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:42.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.271+0000 7f40d8411700 1 --2- 172.21.15.16:0/3540671818 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f40d00cf880 0x7f40d00b52f0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:42.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.271+0000 7f40d8411700 1 -- 172.21.15.16:0/3540671818 >> 172.21.15.16:0/3540671818 conn(0x7f40d00b1fd0 msgr2=0x7f40d00b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:42.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.271+0000 7f40d8411700 1 -- 172.21.15.16:0/3540671818 shutdown_connections 2023-10-16T12:11:42.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.271+0000 7f40d8411700 1 -- 172.21.15.16:0/3540671818 wait complete. 2023-10-16T12:11:42.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.272+0000 7f40d8411700 1 Processor -- start 2023-10-16T12:11:42.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.272+0000 7f40d8411700 1 -- start start 2023-10-16T12:11:42.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.273+0000 7f40d8411700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f40d00b5830 0x7f40d01514a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:42.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.273+0000 7f40d8411700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f40d00cf4a0 0x7f40d00c3f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:42.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.273+0000 7f40d8411700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40d00cf880 0x7f40d0152760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:42.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.273+0000 7f40d8411700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40d00c4480 con 0x7f40d00cf880 2023-10-16T12:11:42.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.273+0000 7f40d8411700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40d00c4690 con 0x7f40d00cf4a0 2023-10-16T12:11:42.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.273+0000 7f40d8411700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40d00c4970 con 0x7f40d00b5830 2023-10-16T12:11:42.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.273+0000 7f40d69ae700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40d00cf880 0x7f40d0152760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:42.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.273+0000 7f40d69ae700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40d00cf880 0x7f40d0152760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34946/0 (socket says 172.21.15.16:34946) 2023-10-16T12:11:42.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.273+0000 7f40d61ad700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f40d00b5830 0x7f40d01514a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:42.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.273+0000 7f40d69ae700 1 -- 172.21.15.16:0/1154329998 learned_addr learned my addr 172.21.15.16:0/1154329998 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:42.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.273+0000 7f40d59ac700 1 --2- 172.21.15.16:0/1154329998 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f40d00cf4a0 0x7f40d00c3f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:42.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.273+0000 7f40d69ae700 1 -- 172.21.15.16:0/1154329998 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f40d00b5830 msgr2=0x7f40d01514a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:42.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.273+0000 7f40d69ae700 1 --2- 172.21.15.16:0/1154329998 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f40d00b5830 0x7f40d01514a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:42.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.273+0000 7f40d69ae700 1 -- 172.21.15.16:0/1154329998 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f40d00cf4a0 msgr2=0x7f40d00c3f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:42.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.273+0000 7f40d69ae700 1 --2- 172.21.15.16:0/1154329998 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f40d00cf4a0 0x7f40d00c3f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:42.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.273+0000 7f40d69ae700 1 -- 172.21.15.16:0/1154329998 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40cc009550 con 0x7f40d00cf880 2023-10-16T12:11:42.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.274+0000 7f40d69ae700 1 --2- 172.21.15.16:0/1154329998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40d00cf880 0x7f40d0152760 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f40c8010c20 tx=0x7f40c8010f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:42.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.274+0000 7f40c6ffd700 1 -- 172.21.15.16:0/1154329998 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40c8003e30 con 0x7f40d00cf880 2023-10-16T12:11:42.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.274+0000 7f40c6ffd700 1 -- 172.21.15.16:0/1154329998 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40c800a690 con 0x7f40d00cf880 2023-10-16T12:11:42.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.274+0000 7f40d8411700 1 -- 172.21.15.16:0/1154329998 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40d00c4c50 con 0x7f40d00cf880 2023-10-16T12:11:42.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.274+0000 7f40c6ffd700 1 -- 172.21.15.16:0/1154329998 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40c800aa40 con 0x7f40d00cf880 2023-10-16T12:11:42.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.274+0000 7f40d8411700 1 -- 172.21.15.16:0/1154329998 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40d00c5210 con 0x7f40d00cf880 2023-10-16T12:11:42.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.275+0000 7f40d8411700 1 -- 172.21.15.16:0/1154329998 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40d004ebb0 con 0x7f40d00cf880 2023-10-16T12:11:42.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.276+0000 7f40c6ffd700 1 -- 172.21.15.16:0/1154329998 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f40c800aba0 con 0x7f40d00cf880 2023-10-16T12:11:42.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.276+0000 7f40c6ffd700 1 --2- 172.21.15.16:0/1154329998 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f40b406af00 0x7f40b406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:42.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.276+0000 7f40d61ad700 1 --2- 172.21.15.16:0/1154329998 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f40b406af00 0x7f40b406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:42.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.276+0000 7f40c6ffd700 1 -- 172.21.15.16:0/1154329998 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f40c808d710 con 0x7f40d00cf880 2023-10-16T12:11:42.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.277+0000 7f40d61ad700 1 --2- 172.21.15.16:0/1154329998 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f40b406af00 0x7f40b406d3c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f40c0009be0 tx=0x7f40c0009210 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:42.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.282+0000 7f40c6ffd700 1 -- 172.21.15.16:0/1154329998 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f40c8058380 con 0x7f40d00cf880 2023-10-16T12:11:42.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.487+0000 7f40d8411700 1 -- 172.21.15.16:0/1154329998 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f40d0151f60 con 0x7f40d00cf880 2023-10-16T12:11:42.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.488+0000 7f40c6ffd700 1 -- 172.21.15.16:0/1154329998 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18629 (secure 0 0 0) 0x7f40c805b9a0 con 0x7f40d00cf880 2023-10-16T12:11:42.492 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:11:42.492 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:11:28.241865+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:10:36.935773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T12:10:38.004870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-16T12:10:41.672025+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"3.2b","mappings":[{"from":0,"to":2}]},{"pgid":"3.38","mappings":[{"from":3,"to":1}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]},{"pgid":"4.1b","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:11:42.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.490+0000 7f40d8411700 1 -- 172.21.15.16:0/1154329998 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f40b406af00 msgr2=0x7f40b406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:42.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.490+0000 7f40d8411700 1 --2- 172.21.15.16:0/1154329998 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f40b406af00 0x7f40b406d3c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f40c0009be0 tx=0x7f40c0009210 comp rx=0 tx=0).stop 2023-10-16T12:11:42.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.490+0000 7f40d8411700 1 -- 172.21.15.16:0/1154329998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40d00cf880 msgr2=0x7f40d0152760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:42.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.490+0000 7f40d8411700 1 --2- 172.21.15.16:0/1154329998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40d00cf880 0x7f40d0152760 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f40c8010c20 tx=0x7f40c8010f30 comp rx=0 tx=0).stop 2023-10-16T12:11:42.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.490+0000 7f40d8411700 1 -- 172.21.15.16:0/1154329998 shutdown_connections 2023-10-16T12:11:42.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.490+0000 7f40d8411700 1 --2- 172.21.15.16:0/1154329998 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f40b406af00 0x7f40b406d3c0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:42.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.490+0000 7f40d8411700 1 --2- 172.21.15.16:0/1154329998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40d00cf880 0x7f40d0152760 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:42.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.490+0000 7f40d8411700 1 --2- 172.21.15.16:0/1154329998 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f40d00b5830 0x7f40d01514a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:42.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.490+0000 7f40d8411700 1 --2- 172.21.15.16:0/1154329998 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f40d00cf4a0 0x7f40d00c3f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:42.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.490+0000 7f40d8411700 1 -- 172.21.15.16:0/1154329998 >> 172.21.15.16:0/1154329998 conn(0x7f40d00b1fd0 msgr2=0x7f40d00ce730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:42.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.490+0000 7f40d8411700 1 -- 172.21.15.16:0/1154329998 shutdown_connections 2023-10-16T12:11:42.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.490+0000 7f40d8411700 1 -- 172.21.15.16:0/1154329998 wait complete. 2023-10-16T12:11:42.505 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T12:10:36.935773+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T12:10:38.004870+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-16T12:10:41.672025+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-16T12:11:42.506 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:11:42.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.632+0000 7f68bed9f700 1 Processor -- start 2023-10-16T12:11:42.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.632+0000 7f68bed9f700 1 -- start start 2023-10-16T12:11:42.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.633+0000 7f68bed9f700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f68b815b490 0x7f68b806b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:42.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.633+0000 7f68bed9f700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68b806b6e0 con 0x7f68b80cd7a0 2023-10-16T12:11:42.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.633+0000 7f68bed9f700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68b8161320 con 0x7f68b80ce4e0 2023-10-16T12:11:42.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.633+0000 7f68bed9f700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68b8161fa0 con 0x7f68b815b490 2023-10-16T12:11:42.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.633+0000 7f68bd59c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f68b80cd7a0 0x7f68b80cddd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:52458/0 (socket says 172.21.15.16:52458) 2023-10-16T12:11:42.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.633+0000 7f68bd59c700 1 -- 172.21.15.16:0/3812361313 learned_addr learned my addr 172.21.15.16:0/3812361313 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:42.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.633+0000 7f68bdd9d700 1 --2- 172.21.15.16:0/3812361313 >> v2:172.21.15.49:3300/0 conn(0x7f68b815b490 0x7f68b806b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:42.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.633+0000 7f68bcd9b700 1 -- 172.21.15.16:0/3812361313 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 879173135 0 0) 0x7f68b806b6e0 con 0x7f68b80cd7a0 2023-10-16T12:11:42.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.633+0000 7f68bcd9b700 1 -- 172.21.15.16:0/3812361313 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68a40035b0 con 0x7f68b80cd7a0 2023-10-16T12:11:42.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.633+0000 7f68bcd9b700 1 -- 172.21.15.16:0/3812361313 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2160650291 0 0) 0x7f68a40035b0 con 0x7f68b80cd7a0 2023-10-16T12:11:42.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.633+0000 7f68bcd9b700 1 -- 172.21.15.16:0/3812361313 >> v2:172.21.15.49:3300/0 conn(0x7f68b815b490 msgr2=0x7f68b806b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:42.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.633+0000 7f68bcd9b700 1 --2- 172.21.15.16:0/3812361313 >> v2:172.21.15.49:3300/0 conn(0x7f68b815b490 0x7f68b806b1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:42.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.633+0000 7f68bcd9b700 1 -- 172.21.15.16:0/3812361313 >> v1:172.21.15.49:6789/0 conn(0x7f68b80ce4e0 legacy=0x7f68b815dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:42.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.633+0000 7f68bcd9b700 1 -- 172.21.15.16:0/3812361313 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68b8161600 con 0x7f68b80cd7a0 2023-10-16T12:11:42.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.633+0000 7f68bcd9b700 1 -- 172.21.15.16:0/3812361313 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f68a8001670 con 0x7f68b80cd7a0 2023-10-16T12:11:42.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.633+0000 7f68bcd9b700 1 -- 172.21.15.16:0/3812361313 >> v1:172.21.15.16:6789/0 conn(0x7f68b80cd7a0 legacy=0x7f68b80cddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:42.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.634+0000 7f68bcd9b700 1 --2- 172.21.15.16:0/3812361313 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f68a4003a00 0x7f68a4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:42.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.634+0000 7f68bcd9b700 1 --2- 172.21.15.16:0/3812361313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68a4004400 0x7f68a40048d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:42.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.634+0000 7f68bcd9b700 1 --2- 172.21.15.16:0/3812361313 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f68a4005170 0x7f68a400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:42.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.634+0000 7f68bcd9b700 1 -- 172.21.15.16:0/3812361313 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68b8161fa0 con 0x7f68a4004400 2023-10-16T12:11:42.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.634+0000 7f68bcd9b700 1 -- 172.21.15.16:0/3812361313 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68a400c0b0 con 0x7f68a4003a00 2023-10-16T12:11:42.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.634+0000 7f68bcd9b700 1 -- 172.21.15.16:0/3812361313 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68a400c270 con 0x7f68a4005170 2023-10-16T12:11:42.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.634+0000 7f68bd59c700 1 --2- 172.21.15.16:0/3812361313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68a4004400 0x7f68a40048d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:42.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.634+0000 7f68bdd9d700 1 --2- 172.21.15.16:0/3812361313 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f68a4003a00 0x7f68a4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:42.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.634+0000 7f68be59e700 1 --2- 172.21.15.16:0/3812361313 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f68a4005170 0x7f68a400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:42.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.634+0000 7f68bd59c700 1 -- 172.21.15.16:0/3812361313 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f68a4005170 msgr2=0x7f68a400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:42.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.634+0000 7f68bd59c700 1 --2- 172.21.15.16:0/3812361313 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f68a4005170 0x7f68a400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:42.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.634+0000 7f68bd59c700 1 -- 172.21.15.16:0/3812361313 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f68a4003a00 msgr2=0x7f68a4003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:42.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.634+0000 7f68bd59c700 1 --2- 172.21.15.16:0/3812361313 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f68a4003a00 0x7f68a4003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:42.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.634+0000 7f68be59e700 1 -- 172.21.15.16:0/3812361313 reap_dead start 2023-10-16T12:11:42.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.634+0000 7f68bd59c700 1 -- 172.21.15.16:0/3812361313 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68a40035b0 con 0x7f68a4004400 2023-10-16T12:11:42.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.634+0000 7f68bd59c700 1 --2- 172.21.15.16:0/3812361313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68a4004400 0x7f68a40048d0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f68a800c240 tx=0x7f68a4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15b7943bb7ebe546 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:42.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.634+0000 7f68bcd9b700 1 -- 172.21.15.16:0/3812361313 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68a800cd00 con 0x7f68a4004400 2023-10-16T12:11:42.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.634+0000 7f68bcd9b700 1 -- 172.21.15.16:0/3812361313 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68a800ce60 con 0x7f68a4004400 2023-10-16T12:11:42.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.634+0000 7f68bcd9b700 1 -- 172.21.15.16:0/3812361313 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68a8005b80 con 0x7f68a4004400 2023-10-16T12:11:42.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.635+0000 7f68bed9f700 1 -- 172.21.15.16:0/3812361313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68a4004400 msgr2=0x7f68a40048d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:42.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.635+0000 7f68bed9f700 1 --2- 172.21.15.16:0/3812361313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68a4004400 0x7f68a40048d0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f68a800c240 tx=0x7f68a4005880 comp rx=0 tx=0).stop 2023-10-16T12:11:42.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.635+0000 7f68bed9f700 1 -- 172.21.15.16:0/3812361313 shutdown_connections 2023-10-16T12:11:42.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.635+0000 7f68bed9f700 1 --2- 172.21.15.16:0/3812361313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68a4004400 0x7f68a40048d0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:42.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.635+0000 7f68bed9f700 1 -- 172.21.15.16:0/3812361313 >> 172.21.15.16:0/3812361313 conn(0x7f68b80ccc80 msgr2=0x7f68b80cd090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:42.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.635+0000 7f68bed9f700 1 -- 172.21.15.16:0/3812361313 shutdown_connections 2023-10-16T12:11:42.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.635+0000 7f68bed9f700 1 -- 172.21.15.16:0/3812361313 wait complete. 2023-10-16T12:11:42.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.636+0000 7f68bed9f700 1 Processor -- start 2023-10-16T12:11:42.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.636+0000 7f68bed9f700 1 -- start start 2023-10-16T12:11:42.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.637+0000 7f68bed9f700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f68a4004400 0x7f68b80ae220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:42.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.637+0000 7f68bed9f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f68b80c9d50 0x7f68b80b2eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:42.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.637+0000 7f68bed9f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68b8148120 0x7f68b81485d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:42.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.637+0000 7f68bed9f700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68b8146200 con 0x7f68b8148120 2023-10-16T12:11:42.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.637+0000 7f68bed9f700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68b8148b10 con 0x7f68a4004400 2023-10-16T12:11:42.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.637+0000 7f68bed9f700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68b80ae910 con 0x7f68b80c9d50 2023-10-16T12:11:42.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.637+0000 7f68be59e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68b8148120 0x7f68b81485d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:42.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.637+0000 7f68be59e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68b8148120 0x7f68b81485d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34966/0 (socket says 172.21.15.16:34966) 2023-10-16T12:11:42.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.637+0000 7f68be59e700 1 -- 172.21.15.16:0/452740306 learned_addr learned my addr 172.21.15.16:0/452740306 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:42.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.637+0000 7f68bd59c700 1 --2- 172.21.15.16:0/452740306 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f68b80c9d50 0x7f68b80b2eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:42.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.637+0000 7f68bdd9d700 1 --2- 172.21.15.16:0/452740306 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f68a4004400 0x7f68b80ae220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:42.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.638+0000 7f68be59e700 1 -- 172.21.15.16:0/452740306 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f68b80c9d50 msgr2=0x7f68b80b2eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:42.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.638+0000 7f68be59e700 1 --2- 172.21.15.16:0/452740306 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f68b80c9d50 0x7f68b80b2eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:42.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.638+0000 7f68be59e700 1 -- 172.21.15.16:0/452740306 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f68a4004400 msgr2=0x7f68b80ae220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:42.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.638+0000 7f68be59e700 1 --2- 172.21.15.16:0/452740306 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f68a4004400 0x7f68b80ae220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:42.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.638+0000 7f68be59e700 1 -- 172.21.15.16:0/452740306 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68b80aebf0 con 0x7f68b8148120 2023-10-16T12:11:42.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.638+0000 7f68bdd9d700 1 --2- 172.21.15.16:0/452740306 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f68a4004400 0x7f68b80ae220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:11:42.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.638+0000 7f68be59e700 1 --2- 172.21.15.16:0/452740306 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68b8148120 0x7f68b81485d0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f68b8071420 tx=0x7f68b000d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:42.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.638+0000 7f68ae7fc700 1 -- 172.21.15.16:0/452740306 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68b0003cd0 con 0x7f68b8148120 2023-10-16T12:11:42.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.638+0000 7f68ae7fc700 1 -- 172.21.15.16:0/452740306 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68b0003e30 con 0x7f68b8148120 2023-10-16T12:11:42.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.638+0000 7f68bed9f700 1 -- 172.21.15.16:0/452740306 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68b80aeed0 con 0x7f68b8148120 2023-10-16T12:11:42.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.638+0000 7f68ae7fc700 1 -- 172.21.15.16:0/452740306 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68b000dbd0 con 0x7f68b8148120 2023-10-16T12:11:42.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.638+0000 7f68bed9f700 1 -- 172.21.15.16:0/452740306 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68b80ca1d0 con 0x7f68b8148120 2023-10-16T12:11:42.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.639+0000 7f68bed9f700 1 -- 172.21.15.16:0/452740306 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68b8146340 con 0x7f68b8148120 2023-10-16T12:11:42.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.640+0000 7f68ae7fc700 1 -- 172.21.15.16:0/452740306 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f68b000dd30 con 0x7f68b8148120 2023-10-16T12:11:42.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.640+0000 7f68ae7fc700 1 --2- 172.21.15.16:0/452740306 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f689c06af00 0x7f689c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:42.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.640+0000 7f68ae7fc700 1 -- 172.21.15.16:0/452740306 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f68b0089db0 con 0x7f68b8148120 2023-10-16T12:11:42.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.640+0000 7f68bdd9d700 1 --2- 172.21.15.16:0/452740306 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f689c06af00 0x7f689c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:42.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.640+0000 7f68bdd9d700 1 --2- 172.21.15.16:0/452740306 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f689c06af00 0x7f689c06d3c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f68b8050550 tx=0x7f68b4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:42.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.644+0000 7f68ae7fc700 1 -- 172.21.15.16:0/452740306 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f68b8146340 con 0x7f68b8148120 2023-10-16T12:11:42.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.844+0000 7f68bed9f700 1 -- 172.21.15.16:0/452740306 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f68b8146340 con 0x7f68b8148120 2023-10-16T12:11:42.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.844+0000 7f68ae7fc700 1 -- 172.21.15.16:0/452740306 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7f68b8146340 con 0x7f68b8148120 2023-10-16T12:11:42.848 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T12:11:42.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.846+0000 7f68bed9f700 1 -- 172.21.15.16:0/452740306 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f689c06af00 msgr2=0x7f689c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:42.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.846+0000 7f68bed9f700 1 --2- 172.21.15.16:0/452740306 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f689c06af00 0x7f689c06d3c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f68b8050550 tx=0x7f68b4008040 comp rx=0 tx=0).stop 2023-10-16T12:11:42.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.846+0000 7f68bed9f700 1 -- 172.21.15.16:0/452740306 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68b8148120 msgr2=0x7f68b81485d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:42.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.846+0000 7f68bed9f700 1 --2- 172.21.15.16:0/452740306 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68b8148120 0x7f68b81485d0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f68b8071420 tx=0x7f68b000d430 comp rx=0 tx=0).stop 2023-10-16T12:11:42.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.846+0000 7f68bed9f700 1 -- 172.21.15.16:0/452740306 shutdown_connections 2023-10-16T12:11:42.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.846+0000 7f68bed9f700 1 --2- 172.21.15.16:0/452740306 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f689c06af00 0x7f689c06d3c0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:42.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.846+0000 7f68bed9f700 1 --2- 172.21.15.16:0/452740306 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68b8148120 0x7f68b81485d0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:42.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.846+0000 7f68bed9f700 1 --2- 172.21.15.16:0/452740306 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f68a4004400 0x7f68b80ae220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:42.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.846+0000 7f68bed9f700 1 --2- 172.21.15.16:0/452740306 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f68b80c9d50 0x7f68b80b2eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:42.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.846+0000 7f68bed9f700 1 -- 172.21.15.16:0/452740306 >> 172.21.15.16:0/452740306 conn(0x7f68b80ccc80 msgr2=0x7f68b815a330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:42.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.846+0000 7f68bed9f700 1 -- 172.21.15.16:0/452740306 shutdown_connections 2023-10-16T12:11:42.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.846+0000 7f68bed9f700 1 -- 172.21.15.16:0/452740306 wait complete. 2023-10-16T12:11:42.861 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T12:11:42.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.987+0000 7f3848f62700 1 Processor -- start 2023-10-16T12:11:42.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.987+0000 7f3848f62700 1 -- start start 2023-10-16T12:11:42.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.988+0000 7f3848f62700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f384406a7e0 0x7f384406ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:42.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.988+0000 7f3848f62700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f384415ecb0 con 0x7f384415b2c0 2023-10-16T12:11:42.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.988+0000 7f3848f62700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f384415b6a0 con 0x7f384406e570 2023-10-16T12:11:42.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.988+0000 7f3848f62700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f384406b1b0 con 0x7f384406a7e0 2023-10-16T12:11:42.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.988+0000 7f384259c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f384415b2c0 0x7f3844161340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:52476/0 (socket says 172.21.15.16:52476) 2023-10-16T12:11:42.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.988+0000 7f384259c700 1 -- 172.21.15.16:0/1424971537 learned_addr learned my addr 172.21.15.16:0/1424971537 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:42.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.988+0000 7f3842d9d700 1 --2- 172.21.15.16:0/1424971537 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f384406a7e0 0x7f384406ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:42.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.988+0000 7f384159a700 1 -- 172.21.15.16:0/1424971537 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1915310326 0 0) 0x7f384415ecb0 con 0x7f384415b2c0 2023-10-16T12:11:42.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.988+0000 7f384159a700 1 -- 172.21.15.16:0/1424971537 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f382c0035b0 con 0x7f384415b2c0 2023-10-16T12:11:42.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.988+0000 7f384159a700 1 -- 172.21.15.16:0/1424971537 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3548091069 0 0) 0x7f384415b6a0 con 0x7f384406e570 2023-10-16T12:11:42.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.988+0000 7f384159a700 1 -- 172.21.15.16:0/1424971537 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f384415ecb0 con 0x7f384406e570 2023-10-16T12:11:42.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.988+0000 7f384159a700 1 -- 172.21.15.16:0/1424971537 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2818247824 0 0) 0x7f382c0035b0 con 0x7f384415b2c0 2023-10-16T12:11:42.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.988+0000 7f3842d9d700 1 -- 172.21.15.16:0/1424971537 >> v1:172.21.15.49:6789/0 conn(0x7f384406e570 legacy=0x7f384415e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:42.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.988+0000 7f3842d9d700 1 -- 172.21.15.16:0/1424971537 >> v1:172.21.15.16:6789/0 conn(0x7f384415b2c0 legacy=0x7f3844161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:42.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.988+0000 7f3842d9d700 1 -- 172.21.15.16:0/1424971537 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3844161b20 con 0x7f384406a7e0 2023-10-16T12:11:42.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.989+0000 7f3842d9d700 1 --2- 172.21.15.16:0/1424971537 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f384406a7e0 0x7f384406ac70 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f383c009dc0 tx=0x7f383c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b0599166cf583d8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:42.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.989+0000 7f384159a700 1 -- 172.21.15.16:0/1424971537 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f383c00e070 con 0x7f384406a7e0 2023-10-16T12:11:42.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.989+0000 7f384159a700 1 -- 172.21.15.16:0/1424971537 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f383c003a20 con 0x7f384406a7e0 2023-10-16T12:11:42.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.989+0000 7f384159a700 1 -- 172.21.15.16:0/1424971537 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3838001670 con 0x7f384406a7e0 2023-10-16T12:11:42.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.989+0000 7f3848f62700 1 -- 172.21.15.16:0/1424971537 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f384406a7e0 msgr2=0x7f384406ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:42.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.989+0000 7f3848f62700 1 --2- 172.21.15.16:0/1424971537 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f384406a7e0 0x7f384406ac70 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f383c009dc0 tx=0x7f383c00b330 comp rx=0 tx=0).stop 2023-10-16T12:11:42.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.989+0000 7f3848f62700 1 -- 172.21.15.16:0/1424971537 shutdown_connections 2023-10-16T12:11:42.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.989+0000 7f3848f62700 1 --2- 172.21.15.16:0/1424971537 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f384406a7e0 0x7f384406ac70 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:42.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.989+0000 7f3848f62700 1 -- 172.21.15.16:0/1424971537 >> 172.21.15.16:0/1424971537 conn(0x7f38440b1f40 msgr2=0x7f38440b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:42.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.989+0000 7f3848f62700 1 -- 172.21.15.16:0/1424971537 shutdown_connections 2023-10-16T12:11:42.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.990+0000 7f3848f62700 1 -- 172.21.15.16:0/1424971537 wait complete. 2023-10-16T12:11:43.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.991+0000 7f3848f62700 1 Processor -- start 2023-10-16T12:11:43.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.991+0000 7f3848f62700 1 -- start start 2023-10-16T12:11:43.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.992+0000 7f3848f62700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f384406a7e0 0x7f38441f27b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:43.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.992+0000 7f3848f62700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f384406e570 0x7f3844159b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:43.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.992+0000 7f3848f62700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f384415b2c0 0x7f38440c74c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:43.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.992+0000 7f3848f62700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38440b3b20 con 0x7f384406e570 2023-10-16T12:11:43.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.992+0000 7f3848f62700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38440b2f10 con 0x7f384406a7e0 2023-10-16T12:11:43.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.992+0000 7f3848f62700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38440b31f0 con 0x7f384415b2c0 2023-10-16T12:11:43.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.992+0000 7f3841d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f384406e570 0x7f3844159b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:43.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.992+0000 7f3841d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f384406e570 0x7f3844159b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34978/0 (socket says 172.21.15.16:34978) 2023-10-16T12:11:43.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.992+0000 7f3841d9b700 1 -- 172.21.15.16:0/2290980110 learned_addr learned my addr 172.21.15.16:0/2290980110 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:43.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.992+0000 7f384259c700 1 --2- 172.21.15.16:0/2290980110 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f384406a7e0 0x7f38441f27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:43.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.992+0000 7f3842d9d700 1 --2- 172.21.15.16:0/2290980110 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f384415b2c0 0x7f38440c74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:43.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.992+0000 7f3841d9b700 1 -- 172.21.15.16:0/2290980110 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f384415b2c0 msgr2=0x7f38440c74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:43.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.992+0000 7f3841d9b700 1 --2- 172.21.15.16:0/2290980110 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f384415b2c0 0x7f38440c74c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:43.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.992+0000 7f3841d9b700 1 -- 172.21.15.16:0/2290980110 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f384406a7e0 msgr2=0x7f38441f27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:43.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.992+0000 7f3841d9b700 1 --2- 172.21.15.16:0/2290980110 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f384406a7e0 0x7f38441f27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:43.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.992+0000 7f3841d9b700 1 -- 172.21.15.16:0/2290980110 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f383c009a70 con 0x7f384406e570 2023-10-16T12:11:43.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.992+0000 7f3841d9b700 1 --2- 172.21.15.16:0/2290980110 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f384406e570 0x7f3844159b50 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f383400c520 tx=0x7f383400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:43.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.992+0000 7f3832ffd700 1 -- 172.21.15.16:0/2290980110 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38340052d0 con 0x7f384406e570 2023-10-16T12:11:43.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.992+0000 7f3832ffd700 1 -- 172.21.15.16:0/2290980110 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3834005430 con 0x7f384406e570 2023-10-16T12:11:43.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.992+0000 7f3848f62700 1 -- 172.21.15.16:0/2290980110 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38440b34d0 con 0x7f384406e570 2023-10-16T12:11:43.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.993+0000 7f3832ffd700 1 -- 172.21.15.16:0/2290980110 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38340038c0 con 0x7f384406e570 2023-10-16T12:11:43.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.993+0000 7f3848f62700 1 -- 172.21.15.16:0/2290980110 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38441f2cf0 con 0x7f384406e570 2023-10-16T12:11:43.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.994+0000 7f3848f62700 1 -- 172.21.15.16:0/2290980110 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38440b3c90 con 0x7f384406e570 2023-10-16T12:11:43.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.994+0000 7f3832ffd700 1 -- 172.21.15.16:0/2290980110 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f3834003a20 con 0x7f384406e570 2023-10-16T12:11:43.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.994+0000 7f3832ffd700 1 --2- 172.21.15.16:0/2290980110 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f382406af00 0x7f382406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:43.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.995+0000 7f384259c700 1 --2- 172.21.15.16:0/2290980110 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f382406af00 0x7f382406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:43.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.995+0000 7f3832ffd700 1 -- 172.21.15.16:0/2290980110 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f3834088ea0 con 0x7f384406e570 2023-10-16T12:11:43.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.995+0000 7f384259c700 1 --2- 172.21.15.16:0/2290980110 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f382406af00 0x7f382406d3c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f38380058d0 tx=0x7f3838001430 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:43.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:42.999+0000 7f3832ffd700 1 -- 172.21.15.16:0/2290980110 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3834053c20 con 0x7f384406e570 2023-10-16T12:11:43.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.201+0000 7f3848f62700 1 -- 172.21.15.16:0/2290980110 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f384404ebb0 con 0x7f384406e570 2023-10-16T12:11:43.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.201+0000 7f3832ffd700 1 -- 172.21.15.16:0/2290980110 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3834017020 con 0x7f384406e570 2023-10-16T12:11:43.204 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-10-16T12:11:43.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.203+0000 7f3848f62700 1 -- 172.21.15.16:0/2290980110 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f382406af00 msgr2=0x7f382406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:43.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.203+0000 7f3848f62700 1 --2- 172.21.15.16:0/2290980110 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f382406af00 0x7f382406d3c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f38380058d0 tx=0x7f3838001430 comp rx=0 tx=0).stop 2023-10-16T12:11:43.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.203+0000 7f3848f62700 1 -- 172.21.15.16:0/2290980110 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f384406e570 msgr2=0x7f3844159b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:43.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.203+0000 7f3848f62700 1 --2- 172.21.15.16:0/2290980110 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f384406e570 0x7f3844159b50 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f383400c520 tx=0x7f383400c8e0 comp rx=0 tx=0).stop 2023-10-16T12:11:43.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.203+0000 7f3848f62700 1 -- 172.21.15.16:0/2290980110 shutdown_connections 2023-10-16T12:11:43.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.203+0000 7f3848f62700 1 --2- 172.21.15.16:0/2290980110 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f382406af00 0x7f382406d3c0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:43.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.203+0000 7f3848f62700 1 --2- 172.21.15.16:0/2290980110 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f384415b2c0 0x7f38440c74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:43.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.203+0000 7f3848f62700 1 --2- 172.21.15.16:0/2290980110 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f384406a7e0 0x7f38441f27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:43.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.203+0000 7f3848f62700 1 --2- 172.21.15.16:0/2290980110 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f384406e570 0x7f3844159b50 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:43.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.203+0000 7f3848f62700 1 -- 172.21.15.16:0/2290980110 >> 172.21.15.16:0/2290980110 conn(0x7f38440b1f40 msgr2=0x7f38440cb290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:43.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.203+0000 7f3848f62700 1 -- 172.21.15.16:0/2290980110 shutdown_connections 2023-10-16T12:11:43.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.203+0000 7f3848f62700 1 -- 172.21.15.16:0/2290980110 wait complete. 2023-10-16T12:11:43.219 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T12:11:43.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.344+0000 7fee4f9a0700 1 Processor -- start 2023-10-16T12:11:43.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.344+0000 7fee4f9a0700 1 -- start start 2023-10-16T12:11:43.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.345+0000 7fee4f9a0700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fee4806e570 0x7fee4815e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:43.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.345+0000 7fee4f9a0700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee4815b6a0 con 0x7fee4815b2c0 2023-10-16T12:11:43.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.345+0000 7fee4f9a0700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee48161b20 con 0x7fee4806a7e0 2023-10-16T12:11:43.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.345+0000 7fee4f9a0700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee48161e00 con 0x7fee4806e570 2023-10-16T12:11:43.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.345+0000 7fee4d73c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fee4815b2c0 0x7fee48161340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:52490/0 (socket says 172.21.15.16:52490) 2023-10-16T12:11:43.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.345+0000 7fee4d73c700 1 -- 172.21.15.16:0/1425600245 learned_addr learned my addr 172.21.15.16:0/1425600245 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:43.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.345+0000 7fee4cf3b700 1 --2- 172.21.15.16:0/1425600245 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fee4806e570 0x7fee4815e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:43.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.345+0000 7fee3ffff700 1 -- 172.21.15.16:0/1425600245 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2531547345 0 0) 0x7fee4815b6a0 con 0x7fee4815b2c0 2023-10-16T12:11:43.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.346+0000 7fee3ffff700 1 -- 172.21.15.16:0/1425600245 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee340035b0 con 0x7fee4815b2c0 2023-10-16T12:11:43.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.346+0000 7fee3ffff700 1 -- 172.21.15.16:0/1425600245 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3894472840 0 0) 0x7fee340035b0 con 0x7fee4815b2c0 2023-10-16T12:11:43.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.346+0000 7fee3ffff700 1 -- 172.21.15.16:0/1425600245 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fee4806e570 msgr2=0x7fee4815e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:43.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.346+0000 7fee3ffff700 1 --2- 172.21.15.16:0/1425600245 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fee4806e570 0x7fee4815e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:43.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.346+0000 7fee3ffff700 1 -- 172.21.15.16:0/1425600245 >> v1:172.21.15.49:6789/0 conn(0x7fee4806a7e0 legacy=0x7fee48071a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:43.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.346+0000 7fee3ffff700 1 -- 172.21.15.16:0/1425600245 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee481620e0 con 0x7fee4815b2c0 2023-10-16T12:11:43.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.346+0000 7fee3ffff700 1 -- 172.21.15.16:0/1425600245 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7fee38001670 con 0x7fee4815b2c0 2023-10-16T12:11:43.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.346+0000 7fee3ffff700 1 -- 172.21.15.16:0/1425600245 >> v1:172.21.15.16:6789/0 conn(0x7fee4815b2c0 legacy=0x7fee48161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:43.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.346+0000 7fee3ffff700 1 --2- 172.21.15.16:0/1425600245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee34003a30 0x7fee34003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:43.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.346+0000 7fee3ffff700 1 --2- 172.21.15.16:0/1425600245 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fee340043e0 0x7fee340048b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:43.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.346+0000 7fee3ffff700 1 --2- 172.21.15.16:0/1425600245 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fee4806e570 0x7fee340051b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:43.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.346+0000 7fee3ffff700 1 -- 172.21.15.16:0/1425600245 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee48161e00 con 0x7fee34003a30 2023-10-16T12:11:43.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.346+0000 7fee3ffff700 1 -- 172.21.15.16:0/1425600245 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee3400bac0 con 0x7fee340043e0 2023-10-16T12:11:43.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.346+0000 7fee3ffff700 1 -- 172.21.15.16:0/1425600245 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee3400bc80 con 0x7fee4806e570 2023-10-16T12:11:43.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.346+0000 7fee4d73c700 1 --2- 172.21.15.16:0/1425600245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee34003a30 0x7fee34003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:43.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.346+0000 7fee4cf3b700 1 --2- 172.21.15.16:0/1425600245 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fee340043e0 0x7fee340048b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:43.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.346+0000 7fee4df3d700 1 --2- 172.21.15.16:0/1425600245 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fee4806e570 0x7fee340051b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:43.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.346+0000 7fee4d73c700 1 -- 172.21.15.16:0/1425600245 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fee4806e570 msgr2=0x7fee340051b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:43.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.346+0000 7fee4d73c700 1 --2- 172.21.15.16:0/1425600245 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fee4806e570 0x7fee340051b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:43.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.346+0000 7fee4d73c700 1 -- 172.21.15.16:0/1425600245 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fee340043e0 msgr2=0x7fee340048b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:43.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.346+0000 7fee4d73c700 1 --2- 172.21.15.16:0/1425600245 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fee340043e0 0x7fee340048b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:43.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.346+0000 7fee4d73c700 1 -- 172.21.15.16:0/1425600245 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee340035b0 con 0x7fee34003a30 2023-10-16T12:11:43.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.347+0000 7fee4d73c700 1 --2- 172.21.15.16:0/1425600245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee34003a30 0x7fee34003e10 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fee3800c2f0 tx=0x7fee34005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff0abefc38da423c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:43.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.347+0000 7fee3ffff700 1 -- 172.21.15.16:0/1425600245 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee3800ce30 con 0x7fee34003a30 2023-10-16T12:11:43.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.347+0000 7fee3ffff700 1 -- 172.21.15.16:0/1425600245 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee38005960 con 0x7fee34003a30 2023-10-16T12:11:43.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.347+0000 7fee3ffff700 1 -- 172.21.15.16:0/1425600245 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee38005cf0 con 0x7fee34003a30 2023-10-16T12:11:43.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.347+0000 7fee4f9a0700 1 -- 172.21.15.16:0/1425600245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee34003a30 msgr2=0x7fee34003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:43.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.347+0000 7fee4f9a0700 1 --2- 172.21.15.16:0/1425600245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee34003a30 0x7fee34003e10 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fee3800c2f0 tx=0x7fee34005880 comp rx=0 tx=0).stop 2023-10-16T12:11:43.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.347+0000 7fee4df3d700 1 -- 172.21.15.16:0/1425600245 reap_dead start 2023-10-16T12:11:43.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.347+0000 7fee4f9a0700 1 -- 172.21.15.16:0/1425600245 shutdown_connections 2023-10-16T12:11:43.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.347+0000 7fee4f9a0700 1 -- 172.21.15.16:0/1425600245 >> 172.21.15.16:0/1425600245 conn(0x7fee480b1f40 msgr2=0x7fee480b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:43.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.347+0000 7fee4f9a0700 1 -- 172.21.15.16:0/1425600245 shutdown_connections 2023-10-16T12:11:43.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.347+0000 7fee4f9a0700 1 -- 172.21.15.16:0/1425600245 wait complete. 2023-10-16T12:11:43.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.349+0000 7fee4f9a0700 1 Processor -- start 2023-10-16T12:11:43.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.349+0000 7fee4f9a0700 1 -- start start 2023-10-16T12:11:43.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.349+0000 7fee4f9a0700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fee34003a30 0x7fee481f0600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:43.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.349+0000 7fee4f9a0700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fee48154370 0x7fee48155230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:43.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.349+0000 7fee4f9a0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee480cce00 0x7fee480cd280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:43.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.349+0000 7fee4f9a0700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee481f4140 con 0x7fee480cce00 2023-10-16T12:11:43.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.349+0000 7fee4f9a0700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee480cd7c0 con 0x7fee48154370 2023-10-16T12:11:43.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.349+0000 7fee4f9a0700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee481f0b40 con 0x7fee34003a30 2023-10-16T12:11:43.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.350+0000 7fee4df3d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee480cce00 0x7fee480cd280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:43.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.350+0000 7fee4df3d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee480cce00 0x7fee480cd280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35000/0 (socket says 172.21.15.16:35000) 2023-10-16T12:11:43.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.350+0000 7fee4df3d700 1 -- 172.21.15.16:0/1908017941 learned_addr learned my addr 172.21.15.16:0/1908017941 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:43.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.350+0000 7fee4cf3b700 1 --2- 172.21.15.16:0/1908017941 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fee48154370 0x7fee48155230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:43.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.350+0000 7fee4d73c700 1 --2- 172.21.15.16:0/1908017941 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fee34003a30 0x7fee481f0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:43.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.350+0000 7fee4df3d700 1 -- 172.21.15.16:0/1908017941 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fee34003a30 msgr2=0x7fee481f0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:43.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.350+0000 7fee4df3d700 1 --2- 172.21.15.16:0/1908017941 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fee34003a30 0x7fee481f0600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:43.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.350+0000 7fee4df3d700 1 -- 172.21.15.16:0/1908017941 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fee48154370 msgr2=0x7fee48155230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:43.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.350+0000 7fee4df3d700 1 --2- 172.21.15.16:0/1908017941 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fee48154370 0x7fee48155230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:43.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.350+0000 7fee4df3d700 1 -- 172.21.15.16:0/1908017941 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee481f35c0 con 0x7fee480cce00 2023-10-16T12:11:43.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.350+0000 7fee4df3d700 1 --2- 172.21.15.16:0/1908017941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee480cce00 0x7fee480cd280 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fee4806b430 tx=0x7fee4000d660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:43.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.350+0000 7fee3dffb700 1 -- 172.21.15.16:0/1908017941 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee4000db60 con 0x7fee480cce00 2023-10-16T12:11:43.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.350+0000 7fee3dffb700 1 -- 172.21.15.16:0/1908017941 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee4000dcc0 con 0x7fee480cce00 2023-10-16T12:11:43.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.350+0000 7fee4f9a0700 1 -- 172.21.15.16:0/1908017941 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee481f38a0 con 0x7fee480cce00 2023-10-16T12:11:43.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.350+0000 7fee3dffb700 1 -- 172.21.15.16:0/1908017941 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee40003a00 con 0x7fee480cce00 2023-10-16T12:11:43.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.351+0000 7fee4f9a0700 1 -- 172.21.15.16:0/1908017941 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee481f3e60 con 0x7fee480cce00 2023-10-16T12:11:43.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.351+0000 7fee4f9a0700 1 -- 172.21.15.16:0/1908017941 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee4804ebb0 con 0x7fee480cce00 2023-10-16T12:11:43.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.352+0000 7fee3dffb700 1 -- 172.21.15.16:0/1908017941 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fee40003b60 con 0x7fee480cce00 2023-10-16T12:11:43.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.352+0000 7fee3dffb700 1 --2- 172.21.15.16:0/1908017941 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fee2c06af00 0x7fee2c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:43.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.352+0000 7fee3dffb700 1 -- 172.21.15.16:0/1908017941 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fee40089c50 con 0x7fee480cce00 2023-10-16T12:11:43.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.352+0000 7fee4d73c700 1 --2- 172.21.15.16:0/1908017941 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fee2c06af00 0x7fee2c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:43.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.353+0000 7fee4d73c700 1 --2- 172.21.15.16:0/1908017941 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fee2c06af00 0x7fee2c06d3c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fee38001930 tx=0x7fee38007000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:43.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.357+0000 7fee3dffb700 1 -- 172.21.15.16:0/1908017941 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fee400549f0 con 0x7fee480cce00 2023-10-16T12:11:43.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.557+0000 7fee4f9a0700 1 -- 172.21.15.16:0/1908017941 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fee480b6370 con 0x7fee480cce00 2023-10-16T12:11:43.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.558+0000 7fee3dffb700 1 -- 172.21.15.16:0/1908017941 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7fee40058010 con 0x7fee480cce00 2023-10-16T12:11:43.561 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-10-16T12:11:43.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.560+0000 7fee4f9a0700 1 -- 172.21.15.16:0/1908017941 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fee2c06af00 msgr2=0x7fee2c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:43.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.560+0000 7fee4f9a0700 1 --2- 172.21.15.16:0/1908017941 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fee2c06af00 0x7fee2c06d3c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fee38001930 tx=0x7fee38007000 comp rx=0 tx=0).stop 2023-10-16T12:11:43.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.560+0000 7fee4f9a0700 1 -- 172.21.15.16:0/1908017941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee480cce00 msgr2=0x7fee480cd280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:43.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.560+0000 7fee4f9a0700 1 --2- 172.21.15.16:0/1908017941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee480cce00 0x7fee480cd280 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fee4806b430 tx=0x7fee4000d660 comp rx=0 tx=0).stop 2023-10-16T12:11:43.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.560+0000 7fee4f9a0700 1 -- 172.21.15.16:0/1908017941 shutdown_connections 2023-10-16T12:11:43.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.560+0000 7fee4f9a0700 1 --2- 172.21.15.16:0/1908017941 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fee2c06af00 0x7fee2c06d3c0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:43.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.560+0000 7fee4f9a0700 1 --2- 172.21.15.16:0/1908017941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee480cce00 0x7fee480cd280 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:43.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.560+0000 7fee4f9a0700 1 --2- 172.21.15.16:0/1908017941 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fee34003a30 0x7fee481f0600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:43.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.560+0000 7fee4f9a0700 1 --2- 172.21.15.16:0/1908017941 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fee48154370 0x7fee48155230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:43.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.560+0000 7fee4f9a0700 1 -- 172.21.15.16:0/1908017941 >> 172.21.15.16:0/1908017941 conn(0x7fee480b1f40 msgr2=0x7fee4815be90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:43.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.560+0000 7fee4f9a0700 1 -- 172.21.15.16:0/1908017941 shutdown_connections 2023-10-16T12:11:43.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.560+0000 7fee4f9a0700 1 -- 172.21.15.16:0/1908017941 wait complete. 2023-10-16T12:11:43.575 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-16T12:11:43.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.703+0000 7fcbaee67700 1 Processor -- start 2023-10-16T12:11:43.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.703+0000 7fcbaee67700 1 -- start start 2023-10-16T12:11:43.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.703+0000 7fcbaee67700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcba806a7e0 0x7fcba806ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:43.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.703+0000 7fcbaee67700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcba815ecb0 con 0x7fcba806e570 2023-10-16T12:11:43.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.703+0000 7fcbaee67700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcba815b6a0 con 0x7fcba815b2c0 2023-10-16T12:11:43.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.703+0000 7fcbaee67700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcba806b1b0 con 0x7fcba806a7e0 2023-10-16T12:11:43.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.704+0000 7fcba7fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fcba806e570 0x7fcba815e5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:52512/0 (socket says 172.21.15.16:52512) 2023-10-16T12:11:43.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.704+0000 7fcba7fff700 1 -- 172.21.15.16:0/867855326 learned_addr learned my addr 172.21.15.16:0/867855326 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:43.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.704+0000 7fcbad404700 1 --2- 172.21.15.16:0/867855326 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcba806a7e0 0x7fcba806ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:43.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.704+0000 7fcba77fe700 1 -- 172.21.15.16:0/867855326 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4021104064 0 0) 0x7fcba815ecb0 con 0x7fcba806e570 2023-10-16T12:11:43.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.704+0000 7fcba77fe700 1 -- 172.21.15.16:0/867855326 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb980035b0 con 0x7fcba806e570 2023-10-16T12:11:43.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.704+0000 7fcba77fe700 1 -- 172.21.15.16:0/867855326 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 937518121 0 0) 0x7fcba815b6a0 con 0x7fcba815b2c0 2023-10-16T12:11:43.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.704+0000 7fcba77fe700 1 -- 172.21.15.16:0/867855326 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcba815ecb0 con 0x7fcba815b2c0 2023-10-16T12:11:43.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.704+0000 7fcba77fe700 1 -- 172.21.15.16:0/867855326 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2701883403 0 0) 0x7fcb980035b0 con 0x7fcba806e570 2023-10-16T12:11:43.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.704+0000 7fcba77fe700 1 -- 172.21.15.16:0/867855326 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcba806a7e0 msgr2=0x7fcba806ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:43.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.704+0000 7fcba77fe700 1 --2- 172.21.15.16:0/867855326 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcba806a7e0 0x7fcba806ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:43.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.704+0000 7fcba77fe700 1 -- 172.21.15.16:0/867855326 >> v1:172.21.15.49:6789/0 conn(0x7fcba815b2c0 legacy=0x7fcba8161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:43.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.704+0000 7fcba77fe700 1 -- 172.21.15.16:0/867855326 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcba8161b20 con 0x7fcba806e570 2023-10-16T12:11:43.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.704+0000 7fcba77fe700 1 -- 172.21.15.16:0/867855326 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7fcb94001670 con 0x7fcba806e570 2023-10-16T12:11:43.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.704+0000 7fcba77fe700 1 -- 172.21.15.16:0/867855326 >> v1:172.21.15.16:6789/0 conn(0x7fcba806e570 legacy=0x7fcba815e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:43.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.704+0000 7fcba77fe700 1 --2- 172.21.15.16:0/867855326 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcb98003bc0 0x7fcb98003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:43.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.704+0000 7fcba77fe700 1 --2- 172.21.15.16:0/867855326 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcb98004850 0x7fcb98004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:43.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.704+0000 7fcba77fe700 1 --2- 172.21.15.16:0/867855326 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb98005210 0x7fcb9800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:43.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.704+0000 7fcba77fe700 1 -- 172.21.15.16:0/867855326 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcba806b1b0 con 0x7fcb98005210 2023-10-16T12:11:43.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.704+0000 7fcba77fe700 1 -- 172.21.15.16:0/867855326 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb9800c080 con 0x7fcb98004850 2023-10-16T12:11:43.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.704+0000 7fcba77fe700 1 -- 172.21.15.16:0/867855326 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb9800c240 con 0x7fcb98003bc0 2023-10-16T12:11:43.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.704+0000 7fcbad404700 1 --2- 172.21.15.16:0/867855326 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcba806a7e0 0x7fcba806ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:11:43.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.704+0000 7fcbad404700 1 --2- 172.21.15.16:0/867855326 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb98005210 0x7fcb9800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:43.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.704+0000 7fcba7fff700 1 --2- 172.21.15.16:0/867855326 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcb98004850 0x7fcb98004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:43.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.704+0000 7fcbacc03700 1 --2- 172.21.15.16:0/867855326 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcb98003bc0 0x7fcb98003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:43.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.705+0000 7fcbad404700 1 -- 172.21.15.16:0/867855326 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcb98003bc0 msgr2=0x7fcb98003fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:43.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.705+0000 7fcbad404700 1 --2- 172.21.15.16:0/867855326 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcb98003bc0 0x7fcb98003fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:43.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.705+0000 7fcbad404700 1 -- 172.21.15.16:0/867855326 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcb98004850 msgr2=0x7fcb98004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:43.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.705+0000 7fcbad404700 1 --2- 172.21.15.16:0/867855326 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcb98004850 0x7fcb98004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:43.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.705+0000 7fcbad404700 1 -- 172.21.15.16:0/867855326 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb980035b0 con 0x7fcb98005210 2023-10-16T12:11:43.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.705+0000 7fcbad404700 1 --2- 172.21.15.16:0/867855326 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb98005210 0x7fcb9800ba10 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fcba806abc0 tx=0x7fcb98005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e93ce4bf14ab9d97 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:43.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.705+0000 7fcba77fe700 1 -- 172.21.15.16:0/867855326 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcba0003c20 con 0x7fcb98005210 2023-10-16T12:11:43.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.705+0000 7fcba77fe700 1 -- 172.21.15.16:0/867855326 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcba000f040 con 0x7fcb98005210 2023-10-16T12:11:43.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.705+0000 7fcba77fe700 1 -- 172.21.15.16:0/867855326 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcba000ea60 con 0x7fcb98005210 2023-10-16T12:11:43.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.706+0000 7fcbaee67700 1 -- 172.21.15.16:0/867855326 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb98005210 msgr2=0x7fcb9800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:43.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.706+0000 7fcbaee67700 1 --2- 172.21.15.16:0/867855326 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb98005210 0x7fcb9800ba10 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fcba806abc0 tx=0x7fcb98005880 comp rx=0 tx=0).stop 2023-10-16T12:11:43.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.706+0000 7fcbad404700 1 -- 172.21.15.16:0/867855326 reap_dead start 2023-10-16T12:11:43.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.706+0000 7fcbaee67700 1 -- 172.21.15.16:0/867855326 shutdown_connections 2023-10-16T12:11:43.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.706+0000 7fcbaee67700 1 -- 172.21.15.16:0/867855326 >> 172.21.15.16:0/867855326 conn(0x7fcba80b1f40 msgr2=0x7fcba80b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:43.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.706+0000 7fcbaee67700 1 -- 172.21.15.16:0/867855326 shutdown_connections 2023-10-16T12:11:43.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.706+0000 7fcbaee67700 1 -- 172.21.15.16:0/867855326 wait complete. 2023-10-16T12:11:43.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.707+0000 7fcbaee67700 1 Processor -- start 2023-10-16T12:11:43.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.707+0000 7fcbaee67700 1 -- start start 2023-10-16T12:11:43.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.708+0000 7fcbaee67700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcb98005210 0x7fcba80b33b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:43.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.708+0000 7fcbaee67700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcba81f82d0 0x7fcba81fa0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:43.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.708+0000 7fcbaee67700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcba80b5720 0x7fcba81f7260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:43.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.708+0000 7fcbaee67700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcba80b61e0 con 0x7fcba81f82d0 2023-10-16T12:11:43.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.708+0000 7fcbaee67700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcba81fa5e0 con 0x7fcba80b5720 2023-10-16T12:11:43.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.708+0000 7fcbaee67700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcba80b5f00 con 0x7fcb98005210 2023-10-16T12:11:43.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.708+0000 7fcba7fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcba81f82d0 0x7fcba81fa0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:43.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.708+0000 7fcba7fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcba81f82d0 0x7fcba81fa0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35018/0 (socket says 172.21.15.16:35018) 2023-10-16T12:11:43.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.708+0000 7fcba7fff700 1 -- 172.21.15.16:0/4100076836 learned_addr learned my addr 172.21.15.16:0/4100076836 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:43.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.708+0000 7fcbacc03700 1 --2- 172.21.15.16:0/4100076836 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcb98005210 0x7fcba80b33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:43.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.708+0000 7fcbad404700 1 --2- 172.21.15.16:0/4100076836 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcba80b5720 0x7fcba81f7260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:43.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.708+0000 7fcba7fff700 1 -- 172.21.15.16:0/4100076836 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcb98005210 msgr2=0x7fcba80b33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:43.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.708+0000 7fcba7fff700 1 --2- 172.21.15.16:0/4100076836 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcb98005210 0x7fcba80b33b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:43.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.708+0000 7fcba7fff700 1 -- 172.21.15.16:0/4100076836 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcba80b5720 msgr2=0x7fcba81f7260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:43.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.708+0000 7fcba7fff700 1 --2- 172.21.15.16:0/4100076836 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcba80b5720 0x7fcba81f7260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:43.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.708+0000 7fcba7fff700 1 -- 172.21.15.16:0/4100076836 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcba80c4db0 con 0x7fcba81f82d0 2023-10-16T12:11:43.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.708+0000 7fcba7fff700 1 --2- 172.21.15.16:0/4100076836 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcba81f82d0 0x7fcba81fa0a0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fcb94001c40 tx=0x7fcb940075f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:43.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.709+0000 7fcba57fa700 1 -- 172.21.15.16:0/4100076836 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb94001080 con 0x7fcba81f82d0 2023-10-16T12:11:43.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.709+0000 7fcba57fa700 1 -- 172.21.15.16:0/4100076836 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb94011580 con 0x7fcba81f82d0 2023-10-16T12:11:43.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.709+0000 7fcba57fa700 1 -- 172.21.15.16:0/4100076836 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb94011930 con 0x7fcba81f82d0 2023-10-16T12:11:43.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.709+0000 7fcbaee67700 1 -- 172.21.15.16:0/4100076836 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcba80c5090 con 0x7fcba81f82d0 2023-10-16T12:11:43.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.709+0000 7fcbaee67700 1 -- 172.21.15.16:0/4100076836 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcba81f8750 con 0x7fcba81f82d0 2023-10-16T12:11:43.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.710+0000 7fcbaee67700 1 -- 172.21.15.16:0/4100076836 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcba80c4c10 con 0x7fcba81f82d0 2023-10-16T12:11:43.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.710+0000 7fcba57fa700 1 -- 172.21.15.16:0/4100076836 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fcb94011a90 con 0x7fcba81f82d0 2023-10-16T12:11:43.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.710+0000 7fcba57fa700 1 --2- 172.21.15.16:0/4100076836 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fcb8c06af00 0x7fcb8c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:43.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.711+0000 7fcba57fa700 1 -- 172.21.15.16:0/4100076836 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fcb94089930 con 0x7fcba81f82d0 2023-10-16T12:11:43.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.711+0000 7fcbacc03700 1 --2- 172.21.15.16:0/4100076836 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fcb8c06af00 0x7fcb8c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:43.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.711+0000 7fcbacc03700 1 --2- 172.21.15.16:0/4100076836 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fcb8c06af00 0x7fcb8c06d3c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c007fd0 tx=0x7fcb9c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:43.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.715+0000 7fcba57fa700 1 -- 172.21.15.16:0/4100076836 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcba80c4c10 con 0x7fcba81f82d0 2023-10-16T12:11:43.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.916+0000 7fcbaee67700 1 -- 172.21.15.16:0/4100076836 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fcba80c4c10 con 0x7fcba81f82d0 2023-10-16T12:11:43.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.917+0000 7fcba57fa700 1 -- 172.21.15.16:0/4100076836 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7fcba80c4c10 con 0x7fcba81f82d0 2023-10-16T12:11:43.920 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-10-16T12:11:43.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.919+0000 7fcbaee67700 1 -- 172.21.15.16:0/4100076836 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fcb8c06af00 msgr2=0x7fcb8c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:43.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.919+0000 7fcbaee67700 1 --2- 172.21.15.16:0/4100076836 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fcb8c06af00 0x7fcb8c06d3c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c007fd0 tx=0x7fcb9c009040 comp rx=0 tx=0).stop 2023-10-16T12:11:43.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.919+0000 7fcbaee67700 1 -- 172.21.15.16:0/4100076836 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcba81f82d0 msgr2=0x7fcba81fa0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:43.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.919+0000 7fcbaee67700 1 --2- 172.21.15.16:0/4100076836 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcba81f82d0 0x7fcba81fa0a0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fcb94001c40 tx=0x7fcb940075f0 comp rx=0 tx=0).stop 2023-10-16T12:11:43.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.919+0000 7fcbaee67700 1 -- 172.21.15.16:0/4100076836 shutdown_connections 2023-10-16T12:11:43.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.919+0000 7fcbaee67700 1 --2- 172.21.15.16:0/4100076836 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fcb8c06af00 0x7fcb8c06d3c0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:43.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.919+0000 7fcbaee67700 1 --2- 172.21.15.16:0/4100076836 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcba80b5720 0x7fcba81f7260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:43.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.919+0000 7fcbaee67700 1 --2- 172.21.15.16:0/4100076836 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcb98005210 0x7fcba80b33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:43.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.919+0000 7fcbaee67700 1 --2- 172.21.15.16:0/4100076836 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcba81f82d0 0x7fcba81fa0a0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:43.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.919+0000 7fcbaee67700 1 -- 172.21.15.16:0/4100076836 >> 172.21.15.16:0/4100076836 conn(0x7fcba80b1f40 msgr2=0x7fcba815b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:43.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.919+0000 7fcbaee67700 1 -- 172.21.15.16:0/4100076836 shutdown_connections 2023-10-16T12:11:43.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:43.919+0000 7fcbaee67700 1 -- 172.21.15.16:0/4100076836 wait complete. 2023-10-16T12:11:43.934 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T12:11:44.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.058+0000 7fb049506700 1 Processor -- start 2023-10-16T12:11:44.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.058+0000 7fb049506700 1 -- start start 2023-10-16T12:11:44.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.059+0000 7fb049506700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fb0440cf4a0 0x7fb0440726e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:44.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.059+0000 7fb049506700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb04415bc80 con 0x7fb0440cf880 2023-10-16T12:11:44.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.059+0000 7fb049506700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0440b5e70 con 0x7fb0440b5a00 2023-10-16T12:11:44.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.059+0000 7fb049506700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb044072c20 con 0x7fb0440cf4a0 2023-10-16T12:11:44.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.059+0000 7fb0427fc700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb0440cf880 0x7fb0440b52f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:52532/0 (socket says 172.21.15.16:52532) 2023-10-16T12:11:44.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.059+0000 7fb0427fc700 1 -- 172.21.15.16:0/2980325899 learned_addr learned my addr 172.21.15.16:0/2980325899 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:44.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.060+0000 7fb042ffd700 1 --2- 172.21.15.16:0/2980325899 >> v2:172.21.15.49:3300/0 conn(0x7fb0440cf4a0 0x7fb0440726e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:44.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.060+0000 7fb041ffb700 1 -- 172.21.15.16:0/2980325899 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3562997718 0 0) 0x7fb04415bc80 con 0x7fb0440cf880 2023-10-16T12:11:44.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.060+0000 7fb041ffb700 1 -- 172.21.15.16:0/2980325899 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb02c0035b0 con 0x7fb0440cf880 2023-10-16T12:11:44.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.060+0000 7fb041ffb700 1 -- 172.21.15.16:0/2980325899 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2566574683 0 0) 0x7fb0440b5e70 con 0x7fb0440b5a00 2023-10-16T12:11:44.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.060+0000 7fb041ffb700 1 -- 172.21.15.16:0/2980325899 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb04415bc80 con 0x7fb0440b5a00 2023-10-16T12:11:44.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.060+0000 7fb041ffb700 1 -- 172.21.15.16:0/2980325899 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2495023592 0 0) 0x7fb02c0035b0 con 0x7fb0440cf880 2023-10-16T12:11:44.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.060+0000 7fb041ffb700 1 -- 172.21.15.16:0/2980325899 >> v2:172.21.15.49:3300/0 conn(0x7fb0440cf4a0 msgr2=0x7fb0440726e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:44.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.060+0000 7fb041ffb700 1 --2- 172.21.15.16:0/2980325899 >> v2:172.21.15.49:3300/0 conn(0x7fb0440cf4a0 0x7fb0440726e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:44.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.060+0000 7fb041ffb700 1 -- 172.21.15.16:0/2980325899 >> v1:172.21.15.49:6789/0 conn(0x7fb0440b5a00 legacy=0x7fb04415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:44.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.060+0000 7fb041ffb700 1 -- 172.21.15.16:0/2980325899 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb04415b070 con 0x7fb0440cf880 2023-10-16T12:11:44.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.060+0000 7fb041ffb700 1 -- 172.21.15.16:0/2980325899 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7fb034001670 con 0x7fb0440cf880 2023-10-16T12:11:44.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.060+0000 7fb041ffb700 1 -- 172.21.15.16:0/2980325899 >> v1:172.21.15.16:6789/0 conn(0x7fb0440cf880 legacy=0x7fb0440b52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:44.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.060+0000 7fb041ffb700 1 --2- 172.21.15.16:0/2980325899 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb02c003b90 0x7fb02c003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:44.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.060+0000 7fb041ffb700 1 --2- 172.21.15.16:0/2980325899 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb02c004850 0x7fb02c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:44.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.060+0000 7fb041ffb700 1 --2- 172.21.15.16:0/2980325899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb02c005210 0x7fb02c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:44.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.060+0000 7fb041ffb700 1 -- 172.21.15.16:0/2980325899 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb044072c20 con 0x7fb02c005210 2023-10-16T12:11:44.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.060+0000 7fb041ffb700 1 -- 172.21.15.16:0/2980325899 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb02c00c090 con 0x7fb02c004850 2023-10-16T12:11:44.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.060+0000 7fb041ffb700 1 -- 172.21.15.16:0/2980325899 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb02c00c250 con 0x7fb02c003b90 2023-10-16T12:11:44.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.060+0000 7fb0437fe700 1 --2- 172.21.15.16:0/2980325899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb02c005210 0x7fb02c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:44.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.060+0000 7fb042ffd700 1 --2- 172.21.15.16:0/2980325899 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb02c003b90 0x7fb02c003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:44.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.060+0000 7fb0427fc700 1 --2- 172.21.15.16:0/2980325899 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb02c004850 0x7fb02c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:44.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.060+0000 7fb0437fe700 1 -- 172.21.15.16:0/2980325899 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb02c003b90 msgr2=0x7fb02c003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:44.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.060+0000 7fb0437fe700 1 --2- 172.21.15.16:0/2980325899 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb02c003b90 0x7fb02c003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:44.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.060+0000 7fb0437fe700 1 -- 172.21.15.16:0/2980325899 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb02c004850 msgr2=0x7fb02c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:44.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.060+0000 7fb0437fe700 1 --2- 172.21.15.16:0/2980325899 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb02c004850 0x7fb02c004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:44.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.060+0000 7fb0437fe700 1 -- 172.21.15.16:0/2980325899 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb02c0035b0 con 0x7fb02c005210 2023-10-16T12:11:44.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.061+0000 7fb0437fe700 1 -- 172.21.15.16:0/2980325899 reap_dead start 2023-10-16T12:11:44.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.061+0000 7fb0437fe700 1 --2- 172.21.15.16:0/2980325899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb02c005210 0x7fb02c00ba10 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fb03c002050 tx=0x7fb02c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5d016ef09e6bc5d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:44.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.061+0000 7fb041ffb700 1 -- 172.21.15.16:0/2980325899 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb03c00ce60 con 0x7fb02c005210 2023-10-16T12:11:44.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.061+0000 7fb041ffb700 1 -- 172.21.15.16:0/2980325899 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb03c003a00 con 0x7fb02c005210 2023-10-16T12:11:44.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.061+0000 7fb041ffb700 1 -- 172.21.15.16:0/2980325899 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb03c003dd0 con 0x7fb02c005210 2023-10-16T12:11:44.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.061+0000 7fb049506700 1 -- 172.21.15.16:0/2980325899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb02c005210 msgr2=0x7fb02c00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:44.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.061+0000 7fb049506700 1 --2- 172.21.15.16:0/2980325899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb02c005210 0x7fb02c00ba10 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fb03c002050 tx=0x7fb02c005880 comp rx=0 tx=0).stop 2023-10-16T12:11:44.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.061+0000 7fb049506700 1 -- 172.21.15.16:0/2980325899 shutdown_connections 2023-10-16T12:11:44.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.061+0000 7fb049506700 1 --2- 172.21.15.16:0/2980325899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb02c005210 0x7fb02c00ba10 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:44.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.061+0000 7fb049506700 1 -- 172.21.15.16:0/2980325899 >> 172.21.15.16:0/2980325899 conn(0x7fb0440b1fd0 msgr2=0x7fb0440b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:44.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.061+0000 7fb049506700 1 -- 172.21.15.16:0/2980325899 shutdown_connections 2023-10-16T12:11:44.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.061+0000 7fb049506700 1 -- 172.21.15.16:0/2980325899 wait complete. 2023-10-16T12:11:44.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.063+0000 7fb049506700 1 Processor -- start 2023-10-16T12:11:44.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.063+0000 7fb049506700 1 -- start start 2023-10-16T12:11:44.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.063+0000 7fb049506700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb02c005210 0x7fb044150a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:44.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.063+0000 7fb049506700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0441e1c90 0x7fb0441e2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:44.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.063+0000 7fb049506700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb0440c5a20 0x7fb0440c5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:44.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.063+0000 7fb049506700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0440c63f0 con 0x7fb0441e1c90 2023-10-16T12:11:44.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.063+0000 7fb049506700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0441e3080 con 0x7fb02c005210 2023-10-16T12:11:44.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.063+0000 7fb049506700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb044150f50 con 0x7fb0440c5a20 2023-10-16T12:11:44.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.063+0000 7fb0427fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0441e1c90 0x7fb0441e2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:44.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.064+0000 7fb0427fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0441e1c90 0x7fb0441e2110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35038/0 (socket says 172.21.15.16:35038) 2023-10-16T12:11:44.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.064+0000 7fb0427fc700 1 -- 172.21.15.16:0/1773858490 learned_addr learned my addr 172.21.15.16:0/1773858490 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:44.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.064+0000 7fb042ffd700 1 --2- 172.21.15.16:0/1773858490 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb02c005210 0x7fb044150a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:44.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.064+0000 7fb0437fe700 1 --2- 172.21.15.16:0/1773858490 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb0440c5a20 0x7fb0440c5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:44.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.064+0000 7fb0427fc700 1 -- 172.21.15.16:0/1773858490 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb0440c5a20 msgr2=0x7fb0440c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:44.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.064+0000 7fb0427fc700 1 --2- 172.21.15.16:0/1773858490 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb0440c5a20 0x7fb0440c5eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:44.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.064+0000 7fb0427fc700 1 -- 172.21.15.16:0/1773858490 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb02c005210 msgr2=0x7fb044150a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:44.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.064+0000 7fb0427fc700 1 --2- 172.21.15.16:0/1773858490 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb02c005210 0x7fb044150a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:44.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.064+0000 7fb0427fc700 1 -- 172.21.15.16:0/1773858490 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0441e27e0 con 0x7fb0441e1c90 2023-10-16T12:11:44.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.064+0000 7fb0427fc700 1 --2- 172.21.15.16:0/1773858490 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0441e1c90 0x7fb0441e2110 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fb034007340 tx=0x7fb034007650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:44.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.064+0000 7fb02b7fe700 1 -- 172.21.15.16:0/1773858490 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb03400f660 con 0x7fb0441e1c90 2023-10-16T12:11:44.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.064+0000 7fb02b7fe700 1 -- 172.21.15.16:0/1773858490 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb03400f800 con 0x7fb0441e1c90 2023-10-16T12:11:44.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.064+0000 7fb049506700 1 -- 172.21.15.16:0/1773858490 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0441e2ac0 con 0x7fb0441e1c90 2023-10-16T12:11:44.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.064+0000 7fb02b7fe700 1 -- 172.21.15.16:0/1773858490 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb03400fbb0 con 0x7fb0441e1c90 2023-10-16T12:11:44.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.064+0000 7fb049506700 1 -- 172.21.15.16:0/1773858490 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0440c40a0 con 0x7fb0441e1c90 2023-10-16T12:11:44.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.065+0000 7fb049506700 1 -- 172.21.15.16:0/1773858490 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb044073580 con 0x7fb0441e1c90 2023-10-16T12:11:44.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.066+0000 7fb02b7fe700 1 -- 172.21.15.16:0/1773858490 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fb03400fd10 con 0x7fb0441e1c90 2023-10-16T12:11:44.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.066+0000 7fb02b7fe700 1 --2- 172.21.15.16:0/1773858490 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fb03006af00 0x7fb03006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:44.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.066+0000 7fb02b7fe700 1 -- 172.21.15.16:0/1773858490 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fb034089780 con 0x7fb0441e1c90 2023-10-16T12:11:44.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.066+0000 7fb042ffd700 1 --2- 172.21.15.16:0/1773858490 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fb03006af00 0x7fb03006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:44.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.066+0000 7fb042ffd700 1 --2- 172.21.15.16:0/1773858490 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fb03006af00 0x7fb03006d3c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fb038009be0 tx=0x7fb038009210 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:44.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.071+0000 7fb02b7fe700 1 -- 172.21.15.16:0/1773858490 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb0340545b0 con 0x7fb0441e1c90 2023-10-16T12:11:44.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.294+0000 7fb049506700 1 -- 172.21.15.16:0/1773858490 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fb0440cff00 con 0x7fb0441e1c90 2023-10-16T12:11:44.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.294+0000 7fb02b7fe700 1 -- 172.21.15.16:0/1773858490 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) v1 ==== 80+0+0 (secure 0 0 0) 0x7fb034057bd0 con 0x7fb0441e1c90 2023-10-16T12:11:44.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.297+0000 7fb049506700 1 -- 172.21.15.16:0/1773858490 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fb03006af00 msgr2=0x7fb03006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:44.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.297+0000 7fb049506700 1 --2- 172.21.15.16:0/1773858490 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fb03006af00 0x7fb03006d3c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fb038009be0 tx=0x7fb038009210 comp rx=0 tx=0).stop 2023-10-16T12:11:44.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.297+0000 7fb049506700 1 -- 172.21.15.16:0/1773858490 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0441e1c90 msgr2=0x7fb0441e2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:44.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.297+0000 7fb049506700 1 --2- 172.21.15.16:0/1773858490 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0441e1c90 0x7fb0441e2110 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fb034007340 tx=0x7fb034007650 comp rx=0 tx=0).stop 2023-10-16T12:11:44.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.297+0000 7fb049506700 1 -- 172.21.15.16:0/1773858490 shutdown_connections 2023-10-16T12:11:44.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.297+0000 7fb049506700 1 --2- 172.21.15.16:0/1773858490 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fb03006af00 0x7fb03006d3c0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:44.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.297+0000 7fb049506700 1 --2- 172.21.15.16:0/1773858490 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb0440c5a20 0x7fb0440c5eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:44.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.297+0000 7fb049506700 1 --2- 172.21.15.16:0/1773858490 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb02c005210 0x7fb044150a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:44.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.297+0000 7fb049506700 1 --2- 172.21.15.16:0/1773858490 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0441e1c90 0x7fb0441e2110 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:44.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.297+0000 7fb049506700 1 -- 172.21.15.16:0/1773858490 >> 172.21.15.16:0/1773858490 conn(0x7fb0440b1fd0 msgr2=0x7fb044151550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:44.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.297+0000 7fb049506700 1 -- 172.21.15.16:0/1773858490 shutdown_connections 2023-10-16T12:11:44.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.297+0000 7fb049506700 1 -- 172.21.15.16:0/1773858490 wait complete. 2023-10-16T12:11:44.303 INFO:teuthology.orchestra.run.smithi016.stderr:listed 0 entries 2023-10-16T12:11:44.312 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.3... 2023-10-16T12:11:44.312 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:44.313 DEBUG:teuthology.orchestra.run.smithi049:> sudo umount /home/ubuntu/cephtest/mnt.3 -f 2023-10-16T12:11:44.361 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi049.front.sepia.ceph.com 2023-10-16T12:11:44.362 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:11:44.362 DEBUG:teuthology.orchestra.run.smithi049:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-10-16T12:11:44.382 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-16T12:11:44.382 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:11:44.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.505+0000 7f9cc44b6700 1 Processor -- start 2023-10-16T12:11:44.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.505+0000 7f9cc44b6700 1 -- start start 2023-10-16T12:11:44.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.506+0000 7f9cc44b6700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f9cbc15b630 0x7f9cbc0cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:44.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.506+0000 7f9cc44b6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9cbc0ce4f0 0x7f9cbc0cfca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:44.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.506+0000 7f9cc44b6700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cbc0cc2a0 con 0x7f9cbc15ba10 2023-10-16T12:11:44.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.506+0000 7f9cc44b6700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cbc0cdb20 con 0x7f9cbc0ce4f0 2023-10-16T12:11:44.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.506+0000 7f9cc44b6700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cbc0cc540 con 0x7f9cbc15b630 2023-10-16T12:11:44.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.506+0000 7f9cc2a53700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9cbc0ce4f0 0x7f9cbc0cfca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:44.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.506+0000 7f9cc2a53700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9cbc0ce4f0 0x7f9cbc0cfca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35048/0 (socket says 172.21.15.16:35048) 2023-10-16T12:11:44.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.506+0000 7f9cc2a53700 1 -- 172.21.15.16:0/2288642225 learned_addr learned my addr 172.21.15.16:0/2288642225 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:44.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.506+0000 7f9cc2a53700 1 -- 172.21.15.16:0/2288642225 >> v2:172.21.15.49:3300/0 conn(0x7f9cbc15b630 msgr2=0x7f9cbc0cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:44.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.506+0000 7f9cc2252700 1 --2- 172.21.15.16:0/2288642225 >> v2:172.21.15.49:3300/0 conn(0x7f9cbc15b630 0x7f9cbc0cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:44.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.506+0000 7f9cc2a53700 1 --2- 172.21.15.16:0/2288642225 >> v2:172.21.15.49:3300/0 conn(0x7f9cbc15b630 0x7f9cbc0cd5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:44.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.506+0000 7f9cc2a53700 1 -- 172.21.15.16:0/2288642225 >> v1:172.21.15.49:6789/0 conn(0x7f9cbc15ba10 legacy=0x7f9cbc0cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:44.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.506+0000 7f9cc2a53700 1 -- 172.21.15.16:0/2288642225 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cbc0cc820 con 0x7f9cbc0ce4f0 2023-10-16T12:11:44.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.506+0000 7f9cc2a53700 1 --2- 172.21.15.16:0/2288642225 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9cbc0ce4f0 0x7f9cbc0cfca0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f9cbc0cefb0 tx=0x7f9cb400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1990a84c69f0277d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:44.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.507+0000 7f9cc1250700 1 -- 172.21.15.16:0/2288642225 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cb400e070 con 0x7f9cbc0ce4f0 2023-10-16T12:11:44.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.507+0000 7f9cc1250700 1 -- 172.21.15.16:0/2288642225 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cb4003a20 con 0x7f9cbc0ce4f0 2023-10-16T12:11:44.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.507+0000 7f9cc1250700 1 -- 172.21.15.16:0/2288642225 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cb4003dd0 con 0x7f9cbc0ce4f0 2023-10-16T12:11:44.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.507+0000 7f9cc44b6700 1 -- 172.21.15.16:0/2288642225 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9cbc0ce4f0 msgr2=0x7f9cbc0cfca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:44.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.507+0000 7f9cc44b6700 1 --2- 172.21.15.16:0/2288642225 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9cbc0ce4f0 0x7f9cbc0cfca0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f9cbc0cefb0 tx=0x7f9cb400b330 comp rx=0 tx=0).stop 2023-10-16T12:11:44.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.507+0000 7f9cc44b6700 1 -- 172.21.15.16:0/2288642225 shutdown_connections 2023-10-16T12:11:44.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.507+0000 7f9cc44b6700 1 --2- 172.21.15.16:0/2288642225 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9cbc0ce4f0 0x7f9cbc0cfca0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:44.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.507+0000 7f9cc44b6700 1 --2- 172.21.15.16:0/2288642225 >> v2:172.21.15.49:3300/0 conn(0x7f9cbc15b630 0x7f9cbc0cd5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:44.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.507+0000 7f9cc44b6700 1 -- 172.21.15.16:0/2288642225 >> 172.21.15.16:0/2288642225 conn(0x7f9cbc0d14b0 msgr2=0x7f9cbc0d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:44.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.507+0000 7f9cc44b6700 1 -- 172.21.15.16:0/2288642225 shutdown_connections 2023-10-16T12:11:44.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.507+0000 7f9cc44b6700 1 -- 172.21.15.16:0/2288642225 wait complete. 2023-10-16T12:11:44.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.509+0000 7f9cc44b6700 1 Processor -- start 2023-10-16T12:11:44.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.509+0000 7f9cc44b6700 1 -- start start 2023-10-16T12:11:44.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.509+0000 7f9cc44b6700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9cbc0ce4f0 0x7f9cbc0c6c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:44.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.509+0000 7f9cc44b6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9cbc15b630 0x7f9cbc1eff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:44.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.509+0000 7f9cc44b6700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9cbc15ba10 0x7f9cbc1f0450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:44.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.509+0000 7f9cc44b6700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cbc0c8e80 con 0x7f9cbc15b630 2023-10-16T12:11:44.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.509+0000 7f9cc44b6700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cbc14d5e0 con 0x7f9cbc15ba10 2023-10-16T12:11:44.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.509+0000 7f9cc44b6700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cbc0c8ba0 con 0x7f9cbc0ce4f0 2023-10-16T12:11:44.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.509+0000 7f9cc1a51700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9cbc15b630 0x7f9cbc1eff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:44.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.509+0000 7f9cc1a51700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9cbc15b630 0x7f9cbc1eff10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35052/0 (socket says 172.21.15.16:35052) 2023-10-16T12:11:44.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.509+0000 7f9cc1a51700 1 -- 172.21.15.16:0/3170121594 learned_addr learned my addr 172.21.15.16:0/3170121594 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:44.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.510+0000 7f9cc2252700 1 --2- 172.21.15.16:0/3170121594 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9cbc0ce4f0 0x7f9cbc0c6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:44.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.510+0000 7f9cc2a53700 1 --2- 172.21.15.16:0/3170121594 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9cbc15ba10 0x7f9cbc1f0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:44.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.510+0000 7f9cc1a51700 1 -- 172.21.15.16:0/3170121594 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9cbc0ce4f0 msgr2=0x7f9cbc0c6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:44.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.510+0000 7f9cc1a51700 1 --2- 172.21.15.16:0/3170121594 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9cbc0ce4f0 0x7f9cbc0c6c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:44.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.510+0000 7f9cc1a51700 1 -- 172.21.15.16:0/3170121594 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9cbc15ba10 msgr2=0x7f9cbc1f0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:44.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.510+0000 7f9cc1a51700 1 --2- 172.21.15.16:0/3170121594 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9cbc15ba10 0x7f9cbc1f0450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:44.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.510+0000 7f9cc1a51700 1 -- 172.21.15.16:0/3170121594 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cb4009b30 con 0x7f9cbc15b630 2023-10-16T12:11:44.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.510+0000 7f9cc2a53700 1 --2- 172.21.15.16:0/3170121594 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9cbc15ba10 0x7f9cbc1f0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:11:44.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.510+0000 7f9cc1a51700 1 --2- 172.21.15.16:0/3170121594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9cbc15b630 0x7f9cbc1eff10 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f9cb800c4a0 tx=0x7f9cb800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:44.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.510+0000 7f9cb2ffd700 1 -- 172.21.15.16:0/3170121594 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cb80054e0 con 0x7f9cbc15b630 2023-10-16T12:11:44.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.510+0000 7f9cb2ffd700 1 -- 172.21.15.16:0/3170121594 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cb8003680 con 0x7f9cbc15b630 2023-10-16T12:11:44.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.510+0000 7f9cc44b6700 1 -- 172.21.15.16:0/3170121594 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cbc14a290 con 0x7f9cbc15b630 2023-10-16T12:11:44.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.510+0000 7f9cb2ffd700 1 -- 172.21.15.16:0/3170121594 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cb8003a30 con 0x7f9cbc15b630 2023-10-16T12:11:44.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.510+0000 7f9cc44b6700 1 -- 172.21.15.16:0/3170121594 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cbc14a850 con 0x7f9cbc15b630 2023-10-16T12:11:44.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.511+0000 7f9cc44b6700 1 -- 172.21.15.16:0/3170121594 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cbc14a0f0 con 0x7f9cbc15b630 2023-10-16T12:11:44.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.512+0000 7f9cb2ffd700 1 -- 172.21.15.16:0/3170121594 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f9cb8003b90 con 0x7f9cbc15b630 2023-10-16T12:11:44.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.512+0000 7f9cb2ffd700 1 --2- 172.21.15.16:0/3170121594 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f9ca006af00 0x7f9ca006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:44.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.512+0000 7f9cb2ffd700 1 -- 172.21.15.16:0/3170121594 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f9cb8088e30 con 0x7f9cbc15b630 2023-10-16T12:11:44.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.512+0000 7f9cc2252700 1 --2- 172.21.15.16:0/3170121594 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f9ca006af00 0x7f9ca006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:44.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.513+0000 7f9cc2252700 1 --2- 172.21.15.16:0/3170121594 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f9ca006af00 0x7f9ca006d3c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f9cac006fd0 tx=0x7f9cac008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:44.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.517+0000 7f9cb2ffd700 1 -- 172.21.15.16:0/3170121594 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9cbc14a0f0 con 0x7f9cbc15b630 2023-10-16T12:11:44.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.768+0000 7f9cc44b6700 1 -- 172.21.15.16:0/3170121594 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9cbc14a0f0 con 0x7f9cbc15b630 2023-10-16T12:11:44.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.769+0000 7f9cb2ffd700 1 -- 172.21.15.16:0/3170121594 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4909 (secure 0 0 0) 0x7f9cbc14a0f0 con 0x7f9cbc15b630 2023-10-16T12:11:44.772 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:11:44.772 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/2979266800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":2979266800},{"type":"v1","addr":"172.21.15.49:6834","nonce":2979266800}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4251,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/1398336801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1398336801},{"type":"v1","addr":"172.21.15.16:6836","nonce":1398336801}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4254,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3168304671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":3168304671},{"type":"v1","addr":"172.21.15.16:6837","nonce":3168304671}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:10:39.030243+0000","modified":"2023-10-16T12:11:02.593358+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4269":{"gid":4269,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.49:6835/2892536404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":2892536404},{"type":"v1","addr":"172.21.15.49:6835","nonce":2892536404}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T12:11:44.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.771+0000 7f9cc44b6700 1 -- 172.21.15.16:0/3170121594 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f9ca006af00 msgr2=0x7f9ca006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:44.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.771+0000 7f9cc44b6700 1 --2- 172.21.15.16:0/3170121594 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f9ca006af00 0x7f9ca006d3c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f9cac006fd0 tx=0x7f9cac008040 comp rx=0 tx=0).stop 2023-10-16T12:11:44.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.771+0000 7f9cc44b6700 1 -- 172.21.15.16:0/3170121594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9cbc15b630 msgr2=0x7f9cbc1eff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:44.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.771+0000 7f9cc44b6700 1 --2- 172.21.15.16:0/3170121594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9cbc15b630 0x7f9cbc1eff10 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f9cb800c4a0 tx=0x7f9cb800c860 comp rx=0 tx=0).stop 2023-10-16T12:11:44.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.771+0000 7f9cc44b6700 1 -- 172.21.15.16:0/3170121594 shutdown_connections 2023-10-16T12:11:44.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.771+0000 7f9cc44b6700 1 --2- 172.21.15.16:0/3170121594 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f9ca006af00 0x7f9ca006d3c0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:44.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.771+0000 7f9cc44b6700 1 --2- 172.21.15.16:0/3170121594 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9cbc15ba10 0x7f9cbc1f0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:44.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.771+0000 7f9cc44b6700 1 --2- 172.21.15.16:0/3170121594 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9cbc0ce4f0 0x7f9cbc0c6c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:44.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.771+0000 7f9cc44b6700 1 --2- 172.21.15.16:0/3170121594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9cbc15b630 0x7f9cbc1eff10 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:44.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.771+0000 7f9cc44b6700 1 -- 172.21.15.16:0/3170121594 >> 172.21.15.16:0/3170121594 conn(0x7f9cbc0d14b0 msgr2=0x7f9cbc0b52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:44.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.771+0000 7f9cc44b6700 1 -- 172.21.15.16:0/3170121594 shutdown_connections 2023-10-16T12:11:44.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.771+0000 7f9cc44b6700 1 -- 172.21.15.16:0/3170121594 wait complete. 2023-10-16T12:11:44.776 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 9 2023-10-16T12:11:44.786 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:11:44.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.908+0000 7f097b3a3700 1 Processor -- start 2023-10-16T12:11:44.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.908+0000 7f097b3a3700 1 -- start start 2023-10-16T12:11:44.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.909+0000 7f097b3a3700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f097415b280 0x7f09741613e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:44.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.909+0000 7f097b3a3700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f097406a7e0 0x7f097406ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:44.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.909+0000 7f097b3a3700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f097406b150 con 0x7f09740cc8a0 2023-10-16T12:11:44.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.909+0000 7f097b3a3700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0974161c30 con 0x7f097415b280 2023-10-16T12:11:44.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.909+0000 7f097b3a3700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0974161f10 con 0x7f097406a7e0 2023-10-16T12:11:44.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.909+0000 7f097913f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f097415b280 0x7f09741613e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:44.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.909+0000 7f097913f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f097415b280 0x7f09741613e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35058/0 (socket says 172.21.15.16:35058) 2023-10-16T12:11:44.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.909+0000 7f097913f700 1 -- 172.21.15.16:0/608438002 learned_addr learned my addr 172.21.15.16:0/608438002 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:44.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.909+0000 7f0979940700 1 --2- 172.21.15.16:0/608438002 >> v2:172.21.15.49:3300/0 conn(0x7f097406a7e0 0x7f097406ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:44.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.910+0000 7f097913f700 1 -- 172.21.15.16:0/608438002 >> v2:172.21.15.49:3300/0 conn(0x7f097406a7e0 msgr2=0x7f097406ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:44.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.910+0000 7f097913f700 1 --2- 172.21.15.16:0/608438002 >> v2:172.21.15.49:3300/0 conn(0x7f097406a7e0 0x7f097406ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:44.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.910+0000 7f097913f700 1 -- 172.21.15.16:0/608438002 >> v1:172.21.15.49:6789/0 conn(0x7f09740cc8a0 legacy=0x7f097415e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:44.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.910+0000 7f097913f700 1 -- 172.21.15.16:0/608438002 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09741621f0 con 0x7f097415b280 2023-10-16T12:11:44.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.910+0000 7f097913f700 1 --2- 172.21.15.16:0/608438002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f097415b280 0x7f09741613e0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f09740cbed0 tx=0x7f0964009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1078d91c9a4091b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:44.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.910+0000 7f096bfff700 1 -- 172.21.15.16:0/608438002 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0964011830 con 0x7f097415b280 2023-10-16T12:11:44.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.910+0000 7f096bfff700 1 -- 172.21.15.16:0/608438002 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0964011990 con 0x7f097415b280 2023-10-16T12:11:44.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.910+0000 7f096bfff700 1 -- 172.21.15.16:0/608438002 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0964011d20 con 0x7f097415b280 2023-10-16T12:11:44.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.910+0000 7f097b3a3700 1 -- 172.21.15.16:0/608438002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f097415b280 msgr2=0x7f09741613e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:44.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.910+0000 7f097b3a3700 1 --2- 172.21.15.16:0/608438002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f097415b280 0x7f09741613e0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f09740cbed0 tx=0x7f0964009bd0 comp rx=0 tx=0).stop 2023-10-16T12:11:44.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.911+0000 7f097b3a3700 1 -- 172.21.15.16:0/608438002 shutdown_connections 2023-10-16T12:11:44.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.911+0000 7f097b3a3700 1 --2- 172.21.15.16:0/608438002 >> v2:172.21.15.49:3300/0 conn(0x7f097406a7e0 0x7f097406ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:44.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.911+0000 7f097b3a3700 1 --2- 172.21.15.16:0/608438002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f097415b280 0x7f09741613e0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:44.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.911+0000 7f097b3a3700 1 -- 172.21.15.16:0/608438002 >> 172.21.15.16:0/608438002 conn(0x7f09740b1fd0 msgr2=0x7f09740b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:44.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.911+0000 7f097b3a3700 1 -- 172.21.15.16:0/608438002 shutdown_connections 2023-10-16T12:11:44.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.911+0000 7f097b3a3700 1 -- 172.21.15.16:0/608438002 wait complete. 2023-10-16T12:11:44.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.912+0000 7f097b3a3700 1 Processor -- start 2023-10-16T12:11:44.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.912+0000 7f097b3a3700 1 -- start start 2023-10-16T12:11:44.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.913+0000 7f097b3a3700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f097406a7e0 0x7f09740c4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:44.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.913+0000 7f097b3a3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09740cc8a0 0x7f09741590f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:44.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.913+0000 7f097b3a3700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f097415b280 0x7f0974159630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:44.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.913+0000 7f097b3a3700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09740c50f0 con 0x7f09740cc8a0 2023-10-16T12:11:44.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.913+0000 7f097b3a3700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09740c40f0 con 0x7f097415b280 2023-10-16T12:11:44.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.913+0000 7f097b3a3700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0974149e20 con 0x7f097406a7e0 2023-10-16T12:11:44.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.913+0000 7f097893e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09740cc8a0 0x7f09741590f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:44.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.913+0000 7f097893e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09740cc8a0 0x7f09741590f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35064/0 (socket says 172.21.15.16:35064) 2023-10-16T12:11:44.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.913+0000 7f097893e700 1 -- 172.21.15.16:0/2058229427 learned_addr learned my addr 172.21.15.16:0/2058229427 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:44.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.913+0000 7f097913f700 1 --2- 172.21.15.16:0/2058229427 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f097406a7e0 0x7f09740c4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:44.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.913+0000 7f0979940700 1 --2- 172.21.15.16:0/2058229427 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f097415b280 0x7f0974159630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:44.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.913+0000 7f097893e700 1 -- 172.21.15.16:0/2058229427 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f097406a7e0 msgr2=0x7f09740c4bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:44.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.913+0000 7f097893e700 1 --2- 172.21.15.16:0/2058229427 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f097406a7e0 0x7f09740c4bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:44.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.913+0000 7f097893e700 1 -- 172.21.15.16:0/2058229427 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f097415b280 msgr2=0x7f0974159630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:44.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.913+0000 7f097893e700 1 --2- 172.21.15.16:0/2058229427 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f097415b280 0x7f0974159630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:44.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.913+0000 7f097893e700 1 -- 172.21.15.16:0/2058229427 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0964009550 con 0x7f09740cc8a0 2023-10-16T12:11:44.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.913+0000 7f097893e700 1 --2- 172.21.15.16:0/2058229427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09740cc8a0 0x7f09741590f0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f097000c4b0 tx=0x7f097000c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:44.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.913+0000 7f0969ffb700 1 -- 172.21.15.16:0/2058229427 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09700052d0 con 0x7f09740cc8a0 2023-10-16T12:11:44.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.913+0000 7f0969ffb700 1 -- 172.21.15.16:0/2058229427 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0970005430 con 0x7f09740cc8a0 2023-10-16T12:11:44.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.913+0000 7f097b3a3700 1 -- 172.21.15.16:0/2058229427 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09740c43b0 con 0x7f09740cc8a0 2023-10-16T12:11:44.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.913+0000 7f0969ffb700 1 -- 172.21.15.16:0/2058229427 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09700038c0 con 0x7f09740cc8a0 2023-10-16T12:11:44.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.914+0000 7f097b3a3700 1 -- 172.21.15.16:0/2058229427 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09740c8c90 con 0x7f09740cc8a0 2023-10-16T12:11:44.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.914+0000 7f097b3a3700 1 -- 172.21.15.16:0/2058229427 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f097404ebb0 con 0x7f09740cc8a0 2023-10-16T12:11:44.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.915+0000 7f0969ffb700 1 -- 172.21.15.16:0/2058229427 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f0970003a20 con 0x7f09740cc8a0 2023-10-16T12:11:44.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.915+0000 7f0969ffb700 1 --2- 172.21.15.16:0/2058229427 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f095806af00 0x7f095806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:44.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.916+0000 7f097913f700 1 --2- 172.21.15.16:0/2058229427 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f095806af00 0x7f095806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:44.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.916+0000 7f0969ffb700 1 -- 172.21.15.16:0/2058229427 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f0970088c90 con 0x7f09740cc8a0 2023-10-16T12:11:44.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.916+0000 7f097913f700 1 --2- 172.21.15.16:0/2058229427 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f095806af00 0x7f095806d3c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f0964009c10 tx=0x7f0964008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:44.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:44.920+0000 7f0969ffb700 1 -- 172.21.15.16:0/2058229427 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0970053a10 con 0x7f09740cc8a0 2023-10-16T12:11:45.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.125+0000 7f097b3a3700 1 -- 172.21.15.16:0/2058229427 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f09740ce530 con 0x7f09740cc8a0 2023-10-16T12:11:45.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.126+0000 7f0969ffb700 1 -- 172.21.15.16:0/2058229427 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18629 (secure 0 0 0) 0x7f0970017090 con 0x7f09740cc8a0 2023-10-16T12:11:45.129 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:11:45.130 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:11:28.241865+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:10:36.935773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T12:10:38.004870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-16T12:10:41.672025+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"3.2b","mappings":[{"from":0,"to":2}]},{"pgid":"3.38","mappings":[{"from":3,"to":1}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]},{"pgid":"4.1b","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:11:45.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.128+0000 7f097b3a3700 1 -- 172.21.15.16:0/2058229427 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f095806af00 msgr2=0x7f095806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:45.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.128+0000 7f097b3a3700 1 --2- 172.21.15.16:0/2058229427 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f095806af00 0x7f095806d3c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f0964009c10 tx=0x7f0964008030 comp rx=0 tx=0).stop 2023-10-16T12:11:45.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.128+0000 7f097b3a3700 1 -- 172.21.15.16:0/2058229427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09740cc8a0 msgr2=0x7f09741590f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:45.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.128+0000 7f097b3a3700 1 --2- 172.21.15.16:0/2058229427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09740cc8a0 0x7f09741590f0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f097000c4b0 tx=0x7f097000c7c0 comp rx=0 tx=0).stop 2023-10-16T12:11:45.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.128+0000 7f097b3a3700 1 -- 172.21.15.16:0/2058229427 shutdown_connections 2023-10-16T12:11:45.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.128+0000 7f097b3a3700 1 --2- 172.21.15.16:0/2058229427 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f095806af00 0x7f095806d3c0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:45.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.128+0000 7f097b3a3700 1 --2- 172.21.15.16:0/2058229427 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f097415b280 0x7f0974159630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:45.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.128+0000 7f097b3a3700 1 --2- 172.21.15.16:0/2058229427 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f097406a7e0 0x7f09740c4bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:45.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.128+0000 7f097b3a3700 1 --2- 172.21.15.16:0/2058229427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09740cc8a0 0x7f09741590f0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:45.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.128+0000 7f097b3a3700 1 -- 172.21.15.16:0/2058229427 >> 172.21.15.16:0/2058229427 conn(0x7f09740b1fd0 msgr2=0x7f09741605d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:45.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.129+0000 7f097b3a3700 1 -- 172.21.15.16:0/2058229427 shutdown_connections 2023-10-16T12:11:45.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.129+0000 7f097b3a3700 1 -- 172.21.15.16:0/2058229427 wait complete. 2023-10-16T12:11:45.143 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T12:10:36.935773+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T12:10:38.004870+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-16T12:10:41.672025+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-16T12:11:45.144 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:11:45.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.270+0000 7f7a2704e700 1 Processor -- start 2023-10-16T12:11:45.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.270+0000 7f7a2704e700 1 -- start start 2023-10-16T12:11:45.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.270+0000 7f7a2704e700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f7a20159de0 0x7f7a2015a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:45.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.271+0000 7f7a2704e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7a200cf4c0 0x7f7a2006aac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:45.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.271+0000 7f7a2704e700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a2015dda0 con 0x7f7a2006b000 2023-10-16T12:11:45.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.271+0000 7f7a2704e700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a2015ea20 con 0x7f7a20159de0 2023-10-16T12:11:45.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.271+0000 7f7a2704e700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2015e080 con 0x7f7a200cf4c0 2023-10-16T12:11:45.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.271+0000 7f7a255eb700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f7a2006b000 0x7f7a2006b4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:52578/0 (socket says 172.21.15.16:52578) 2023-10-16T12:11:45.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.271+0000 7f7a255eb700 1 -- 172.21.15.16:0/2796250988 learned_addr learned my addr 172.21.15.16:0/2796250988 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:45.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.271+0000 7f7a1f7fe700 1 -- 172.21.15.16:0/2796250988 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4166439458 0 0) 0x7f7a2015dda0 con 0x7f7a2006b000 2023-10-16T12:11:45.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.271+0000 7f7a1f7fe700 1 -- 172.21.15.16:0/2796250988 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a100046c0 con 0x7f7a2006b000 2023-10-16T12:11:45.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.271+0000 7f7a1ffff700 1 --2- 172.21.15.16:0/2796250988 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7a200cf4c0 0x7f7a2006aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:45.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.271+0000 7f7a24dea700 1 --2- 172.21.15.16:0/2796250988 >> v2:172.21.15.49:3300/0 conn(0x7f7a20159de0 0x7f7a2015a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:45.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.271+0000 7f7a1f7fe700 1 -- 172.21.15.16:0/2796250988 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2365478209 0 0) 0x7f7a100046c0 con 0x7f7a2006b000 2023-10-16T12:11:45.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.271+0000 7f7a1f7fe700 1 -- 172.21.15.16:0/2796250988 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7a200cf4c0 msgr2=0x7f7a2006aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:45.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.271+0000 7f7a1f7fe700 1 --2- 172.21.15.16:0/2796250988 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7a200cf4c0 0x7f7a2006aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:45.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.271+0000 7f7a1f7fe700 1 -- 172.21.15.16:0/2796250988 >> v2:172.21.15.49:3300/0 conn(0x7f7a20159de0 msgr2=0x7f7a2015a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:45.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.271+0000 7f7a1f7fe700 1 --2- 172.21.15.16:0/2796250988 >> v2:172.21.15.49:3300/0 conn(0x7f7a20159de0 0x7f7a2015a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:45.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.271+0000 7f7a1f7fe700 1 -- 172.21.15.16:0/2796250988 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a2015e360 con 0x7f7a2006b000 2023-10-16T12:11:45.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.271+0000 7f7a1f7fe700 1 -- 172.21.15.16:0/2796250988 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f7a180019f0 con 0x7f7a2006b000 2023-10-16T12:11:45.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.271+0000 7f7a1f7fe700 1 -- 172.21.15.16:0/2796250988 >> v1:172.21.15.16:6789/0 conn(0x7f7a2006b000 legacy=0x7f7a2006b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:45.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.271+0000 7f7a1f7fe700 1 --2- 172.21.15.16:0/2796250988 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7a200cf4c0 0x7f7a10004b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:45.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.271+0000 7f7a1f7fe700 1 --2- 172.21.15.16:0/2796250988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a10005090 0x7f7a10005960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:45.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.271+0000 7f7a1f7fe700 1 --2- 172.21.15.16:0/2796250988 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7a10005ea0 0x7f7a10006320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:45.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.271+0000 7f7a1f7fe700 1 -- 172.21.15.16:0/2796250988 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2015ea20 con 0x7f7a10005090 2023-10-16T12:11:45.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.271+0000 7f7a1f7fe700 1 -- 172.21.15.16:0/2796250988 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2015e080 con 0x7f7a10005ea0 2023-10-16T12:11:45.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.271+0000 7f7a1f7fe700 1 -- 172.21.15.16:0/2796250988 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1000cc20 con 0x7f7a200cf4c0 2023-10-16T12:11:45.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.271+0000 7f7a1ffff700 1 --2- 172.21.15.16:0/2796250988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a10005090 0x7f7a10005960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:45.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.271+0000 7f7a24dea700 1 --2- 172.21.15.16:0/2796250988 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7a200cf4c0 0x7f7a10004b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:45.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.272+0000 7f7a255eb700 1 --2- 172.21.15.16:0/2796250988 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7a10005ea0 0x7f7a10006320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:45.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.272+0000 7f7a1ffff700 1 -- 172.21.15.16:0/2796250988 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7a200cf4c0 msgr2=0x7f7a10004b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:45.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.272+0000 7f7a1ffff700 1 --2- 172.21.15.16:0/2796250988 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7a200cf4c0 0x7f7a10004b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:45.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.272+0000 7f7a1ffff700 1 -- 172.21.15.16:0/2796250988 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7a10005ea0 msgr2=0x7f7a10006320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:45.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.272+0000 7f7a1ffff700 1 --2- 172.21.15.16:0/2796250988 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7a10005ea0 0x7f7a10006320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:45.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.272+0000 7f7a1ffff700 1 -- 172.21.15.16:0/2796250988 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a100046c0 con 0x7f7a10005090 2023-10-16T12:11:45.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.272+0000 7f7a1ffff700 1 --2- 172.21.15.16:0/2796250988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a10005090 0x7f7a10005960 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c009b90 tx=0x7f7a10006990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdaf8d57aa15b333 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:45.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.272+0000 7f7a1f7fe700 1 -- 172.21.15.16:0/2796250988 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a0c00e6c0 con 0x7f7a10005090 2023-10-16T12:11:45.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.272+0000 7f7a1f7fe700 1 -- 172.21.15.16:0/2796250988 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a0c00e820 con 0x7f7a10005090 2023-10-16T12:11:45.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.272+0000 7f7a1f7fe700 1 -- 172.21.15.16:0/2796250988 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a0c00eb90 con 0x7f7a10005090 2023-10-16T12:11:45.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.273+0000 7f7a2704e700 1 -- 172.21.15.16:0/2796250988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a10005090 msgr2=0x7f7a10005960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:45.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.273+0000 7f7a2704e700 1 --2- 172.21.15.16:0/2796250988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a10005090 0x7f7a10005960 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c009b90 tx=0x7f7a10006990 comp rx=0 tx=0).stop 2023-10-16T12:11:45.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.273+0000 7f7a255eb700 1 -- 172.21.15.16:0/2796250988 reap_dead start 2023-10-16T12:11:45.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.273+0000 7f7a2704e700 1 -- 172.21.15.16:0/2796250988 shutdown_connections 2023-10-16T12:11:45.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.273+0000 7f7a2704e700 1 -- 172.21.15.16:0/2796250988 >> 172.21.15.16:0/2796250988 conn(0x7f7a200cc5e0 msgr2=0x7f7a200cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:45.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.273+0000 7f7a2704e700 1 -- 172.21.15.16:0/2796250988 shutdown_connections 2023-10-16T12:11:45.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.273+0000 7f7a2704e700 1 -- 172.21.15.16:0/2796250988 wait complete. 2023-10-16T12:11:45.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.274+0000 7f7a2704e700 1 Processor -- start 2023-10-16T12:11:45.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.274+0000 7f7a2704e700 1 -- start start 2023-10-16T12:11:45.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.275+0000 7f7a2704e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7a10005090 0x7f7a200b51d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:45.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.275+0000 7f7a2704e700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7a200c8b00 0x7f7a201f1c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:45.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.275+0000 7f7a2704e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a201eddb0 0x7f7a200c6c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:45.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.275+0000 7f7a2704e700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a200c9b00 con 0x7f7a201eddb0 2023-10-16T12:11:45.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.275+0000 7f7a2704e700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a200b5710 con 0x7f7a200c8b00 2023-10-16T12:11:45.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.275+0000 7f7a2704e700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a200c8f80 con 0x7f7a10005090 2023-10-16T12:11:45.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.275+0000 7f7a255eb700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a201eddb0 0x7f7a200c6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:45.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.275+0000 7f7a255eb700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a201eddb0 0x7f7a200c6c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35084/0 (socket says 172.21.15.16:35084) 2023-10-16T12:11:45.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.275+0000 7f7a255eb700 1 -- 172.21.15.16:0/364786112 learned_addr learned my addr 172.21.15.16:0/364786112 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:45.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.275+0000 7f7a24dea700 1 --2- 172.21.15.16:0/364786112 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7a10005090 0x7f7a200b51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:45.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.275+0000 7f7a1ffff700 1 --2- 172.21.15.16:0/364786112 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7a200c8b00 0x7f7a201f1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:45.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.275+0000 7f7a255eb700 1 -- 172.21.15.16:0/364786112 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7a10005090 msgr2=0x7f7a200b51d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:45.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.275+0000 7f7a255eb700 1 --2- 172.21.15.16:0/364786112 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7a10005090 0x7f7a200b51d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:45.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.275+0000 7f7a255eb700 1 -- 172.21.15.16:0/364786112 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7a200c8b00 msgr2=0x7f7a201f1c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:45.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.275+0000 7f7a255eb700 1 --2- 172.21.15.16:0/364786112 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7a200c8b00 0x7f7a201f1c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:45.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.275+0000 7f7a255eb700 1 -- 172.21.15.16:0/364786112 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a200c9260 con 0x7f7a201eddb0 2023-10-16T12:11:45.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.275+0000 7f7a255eb700 1 --2- 172.21.15.16:0/364786112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a201eddb0 0x7f7a200c6c30 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f7a18001330 tx=0x7f7a1800a640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:45.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.276+0000 7f7a1d7fa700 1 -- 172.21.15.16:0/364786112 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a1800ad00 con 0x7f7a201eddb0 2023-10-16T12:11:45.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.276+0000 7f7a1d7fa700 1 -- 172.21.15.16:0/364786112 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a180133f0 con 0x7f7a201eddb0 2023-10-16T12:11:45.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.276+0000 7f7a2704e700 1 -- 172.21.15.16:0/364786112 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a200c9540 con 0x7f7a201eddb0 2023-10-16T12:11:45.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.276+0000 7f7a1d7fa700 1 -- 172.21.15.16:0/364786112 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a180137a0 con 0x7f7a201eddb0 2023-10-16T12:11:45.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.276+0000 7f7a2704e700 1 -- 172.21.15.16:0/364786112 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a200b1220 con 0x7f7a201eddb0 2023-10-16T12:11:45.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.277+0000 7f7a2704e700 1 -- 172.21.15.16:0/364786112 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a2004ebb0 con 0x7f7a201eddb0 2023-10-16T12:11:45.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.277+0000 7f7a1d7fa700 1 -- 172.21.15.16:0/364786112 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f7a18013900 con 0x7f7a201eddb0 2023-10-16T12:11:45.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.278+0000 7f7a1d7fa700 1 --2- 172.21.15.16:0/364786112 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f7a0406aef0 0x7f7a0406d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:45.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.278+0000 7f7a24dea700 1 --2- 172.21.15.16:0/364786112 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f7a0406aef0 0x7f7a0406d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:45.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.278+0000 7f7a1d7fa700 1 -- 172.21.15.16:0/364786112 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f7a1808a1d0 con 0x7f7a201eddb0 2023-10-16T12:11:45.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.278+0000 7f7a24dea700 1 --2- 172.21.15.16:0/364786112 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f7a0406aef0 0x7f7a0406d3b0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f7a14009a80 tx=0x7f7a14008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:45.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.282+0000 7f7a1d7fa700 1 -- 172.21.15.16:0/364786112 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7a18054ec0 con 0x7f7a201eddb0 2023-10-16T12:11:45.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.483+0000 7f7a2704e700 1 -- 172.21.15.16:0/364786112 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7a200ce3d0 con 0x7f7a201eddb0 2023-10-16T12:11:45.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.483+0000 7f7a1d7fa700 1 -- 172.21.15.16:0/364786112 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7a180584e0 con 0x7f7a201eddb0 2023-10-16T12:11:45.487 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T12:11:45.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.485+0000 7f7a2704e700 1 -- 172.21.15.16:0/364786112 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f7a0406aef0 msgr2=0x7f7a0406d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:45.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.485+0000 7f7a2704e700 1 --2- 172.21.15.16:0/364786112 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f7a0406aef0 0x7f7a0406d3b0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f7a14009a80 tx=0x7f7a14008040 comp rx=0 tx=0).stop 2023-10-16T12:11:45.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.485+0000 7f7a2704e700 1 -- 172.21.15.16:0/364786112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a201eddb0 msgr2=0x7f7a200c6c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:45.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.485+0000 7f7a2704e700 1 --2- 172.21.15.16:0/364786112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a201eddb0 0x7f7a200c6c30 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f7a18001330 tx=0x7f7a1800a640 comp rx=0 tx=0).stop 2023-10-16T12:11:45.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.485+0000 7f7a2704e700 1 -- 172.21.15.16:0/364786112 shutdown_connections 2023-10-16T12:11:45.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.485+0000 7f7a2704e700 1 --2- 172.21.15.16:0/364786112 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f7a0406aef0 0x7f7a0406d3b0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:45.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.485+0000 7f7a2704e700 1 --2- 172.21.15.16:0/364786112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a201eddb0 0x7f7a200c6c30 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:45.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.485+0000 7f7a2704e700 1 --2- 172.21.15.16:0/364786112 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7a10005090 0x7f7a200b51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:45.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.485+0000 7f7a2704e700 1 --2- 172.21.15.16:0/364786112 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7a200c8b00 0x7f7a201f1c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:45.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.485+0000 7f7a2704e700 1 -- 172.21.15.16:0/364786112 >> 172.21.15.16:0/364786112 conn(0x7f7a200cc5e0 msgr2=0x7f7a200cdb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:45.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.486+0000 7f7a2704e700 1 -- 172.21.15.16:0/364786112 shutdown_connections 2023-10-16T12:11:45.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.486+0000 7f7a2704e700 1 -- 172.21.15.16:0/364786112 wait complete. 2023-10-16T12:11:45.501 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T12:11:45.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.626+0000 7f69290e7700 1 Processor -- start 2023-10-16T12:11:45.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.626+0000 7f69290e7700 1 -- start start 2023-10-16T12:11:45.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.627+0000 7f69290e7700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f692415b2c0 0x7f6924161340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:45.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.627+0000 7f69290e7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f692406e570 0x7f692415e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:45.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.627+0000 7f69290e7700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6924161880 con 0x7f692406a7e0 2023-10-16T12:11:45.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.627+0000 7f69290e7700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f692415b6a0 con 0x7f692406e570 2023-10-16T12:11:45.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.627+0000 7f69290e7700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6924161b20 con 0x7f692415b2c0 2023-10-16T12:11:45.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.627+0000 7f692259c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f692406e570 0x7f692415e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:45.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.627+0000 7f692259c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f692406e570 0x7f692415e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35090/0 (socket says 172.21.15.16:35090) 2023-10-16T12:11:45.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.627+0000 7f692259c700 1 -- 172.21.15.16:0/1455149771 learned_addr learned my addr 172.21.15.16:0/1455149771 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:45.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.627+0000 7f6922d9d700 1 --2- 172.21.15.16:0/1455149771 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f692415b2c0 0x7f6924161340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:45.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.627+0000 7f692259c700 1 -- 172.21.15.16:0/1455149771 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f692415b2c0 msgr2=0x7f6924161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:45.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.627+0000 7f692259c700 1 --2- 172.21.15.16:0/1455149771 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f692415b2c0 0x7f6924161340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:45.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.627+0000 7f692259c700 1 -- 172.21.15.16:0/1455149771 >> v1:172.21.15.49:6789/0 conn(0x7f692406a7e0 legacy=0x7f6924071a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:45.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.627+0000 7f692259c700 1 -- 172.21.15.16:0/1455149771 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6924161e00 con 0x7f692406e570 2023-10-16T12:11:45.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.627+0000 7f6922d9d700 1 --2- 172.21.15.16:0/1455149771 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f692415b2c0 0x7f6924161340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:11:45.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.628+0000 7f692259c700 1 --2- 172.21.15.16:0/1455149771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f692406e570 0x7f692415e5a0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f69140098f0 tx=0x7f6914009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2545d69482feadf5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:45.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.628+0000 7f6921d9b700 1 -- 172.21.15.16:0/1455149771 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6914004230 con 0x7f692406e570 2023-10-16T12:11:45.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.628+0000 7f6921d9b700 1 -- 172.21.15.16:0/1455149771 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6914004390 con 0x7f692406e570 2023-10-16T12:11:45.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.628+0000 7f6921d9b700 1 -- 172.21.15.16:0/1455149771 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69140036a0 con 0x7f692406e570 2023-10-16T12:11:45.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.628+0000 7f69290e7700 1 -- 172.21.15.16:0/1455149771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f692406e570 msgr2=0x7f692415e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:45.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.628+0000 7f69290e7700 1 --2- 172.21.15.16:0/1455149771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f692406e570 0x7f692415e5a0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f69140098f0 tx=0x7f6914009c00 comp rx=0 tx=0).stop 2023-10-16T12:11:45.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.628+0000 7f69290e7700 1 -- 172.21.15.16:0/1455149771 shutdown_connections 2023-10-16T12:11:45.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.628+0000 7f69290e7700 1 --2- 172.21.15.16:0/1455149771 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f692415b2c0 0x7f6924161340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:45.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.628+0000 7f69290e7700 1 --2- 172.21.15.16:0/1455149771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f692406e570 0x7f692415e5a0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:45.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.628+0000 7f69290e7700 1 -- 172.21.15.16:0/1455149771 >> 172.21.15.16:0/1455149771 conn(0x7f69240b1f40 msgr2=0x7f69240b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:45.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.628+0000 7f69290e7700 1 -- 172.21.15.16:0/1455149771 shutdown_connections 2023-10-16T12:11:45.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.628+0000 7f69290e7700 1 -- 172.21.15.16:0/1455149771 wait complete. 2023-10-16T12:11:45.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.630+0000 7f69290e7700 1 Processor -- start 2023-10-16T12:11:45.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.630+0000 7f69290e7700 1 -- start start 2023-10-16T12:11:45.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.631+0000 7f69290e7700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f692406a7e0 0x7f69240b17f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:45.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.631+0000 7f69290e7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f692406e570 0x7f69240b5750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:45.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.631+0000 7f69290e7700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f692415b2c0 0x7f69241f2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:45.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.631+0000 7f69290e7700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69240b4910 con 0x7f692406e570 2023-10-16T12:11:45.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.631+0000 7f69290e7700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69240acc10 con 0x7f692406a7e0 2023-10-16T12:11:45.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.631+0000 7f69290e7700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69240b39f0 con 0x7f692415b2c0 2023-10-16T12:11:45.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.631+0000 7f692259c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f692406e570 0x7f69240b5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:45.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.631+0000 7f692259c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f692406e570 0x7f69240b5750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35098/0 (socket says 172.21.15.16:35098) 2023-10-16T12:11:45.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.631+0000 7f692259c700 1 -- 172.21.15.16:0/3275153328 learned_addr learned my addr 172.21.15.16:0/3275153328 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:45.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.631+0000 7f6922d9d700 1 --2- 172.21.15.16:0/3275153328 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f692406a7e0 0x7f69240b17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:45.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.631+0000 7f692359e700 1 --2- 172.21.15.16:0/3275153328 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f692415b2c0 0x7f69241f2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:45.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.631+0000 7f692259c700 1 -- 172.21.15.16:0/3275153328 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f692415b2c0 msgr2=0x7f69241f2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:45.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.631+0000 7f692259c700 1 --2- 172.21.15.16:0/3275153328 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f692415b2c0 0x7f69241f2cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:45.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.631+0000 7f692259c700 1 -- 172.21.15.16:0/3275153328 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f692406a7e0 msgr2=0x7f69240b17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:45.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.631+0000 7f692259c700 1 --2- 172.21.15.16:0/3275153328 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f692406a7e0 0x7f69240b17f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:45.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.631+0000 7f692259c700 1 -- 172.21.15.16:0/3275153328 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6914009550 con 0x7f692406e570 2023-10-16T12:11:45.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.631+0000 7f6922d9d700 1 --2- 172.21.15.16:0/3275153328 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f692406a7e0 0x7f69240b17f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:11:45.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.631+0000 7f692259c700 1 --2- 172.21.15.16:0/3275153328 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f692406e570 0x7f69240b5750 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f6914009130 tx=0x7f6914004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:45.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.631+0000 7f690b7fe700 1 -- 172.21.15.16:0/3275153328 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6914003e30 con 0x7f692406e570 2023-10-16T12:11:45.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.631+0000 7f690b7fe700 1 -- 172.21.15.16:0/3275153328 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f691401f070 con 0x7f692406e570 2023-10-16T12:11:45.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.632+0000 7f690b7fe700 1 -- 172.21.15.16:0/3275153328 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f691400b8a0 con 0x7f692406e570 2023-10-16T12:11:45.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.632+0000 7f69290e7700 1 -- 172.21.15.16:0/3275153328 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69240b1d30 con 0x7f692406e570 2023-10-16T12:11:45.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.632+0000 7f69290e7700 1 -- 172.21.15.16:0/3275153328 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69240b3f70 con 0x7f692406e570 2023-10-16T12:11:45.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.633+0000 7f69290e7700 1 -- 172.21.15.16:0/3275153328 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69240b4a50 con 0x7f692406e570 2023-10-16T12:11:45.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.633+0000 7f690b7fe700 1 -- 172.21.15.16:0/3275153328 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f6914021030 con 0x7f692406e570 2023-10-16T12:11:45.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.633+0000 7f690b7fe700 1 --2- 172.21.15.16:0/3275153328 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f691006af00 0x7f691006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:45.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.633+0000 7f690b7fe700 1 -- 172.21.15.16:0/3275153328 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f691408b060 con 0x7f692406e570 2023-10-16T12:11:45.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.633+0000 7f6922d9d700 1 --2- 172.21.15.16:0/3275153328 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f691006af00 0x7f691006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:45.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.634+0000 7f6922d9d700 1 --2- 172.21.15.16:0/3275153328 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f691006af00 0x7f691006d3c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f69180089b0 tx=0x7f6918008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:45.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.638+0000 7f690b7fe700 1 -- 172.21.15.16:0/3275153328 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f69240b4a50 con 0x7f692406e570 2023-10-16T12:11:45.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.841+0000 7f69290e7700 1 -- 172.21.15.16:0/3275153328 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f69240b4a50 con 0x7f692406e570 2023-10-16T12:11:45.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.841+0000 7f690b7fe700 1 -- 172.21.15.16:0/3275153328 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7f69240b4a50 con 0x7f692406e570 2023-10-16T12:11:45.845 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-10-16T12:11:45.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.843+0000 7f69290e7700 1 -- 172.21.15.16:0/3275153328 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f691006af00 msgr2=0x7f691006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:45.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.843+0000 7f69290e7700 1 --2- 172.21.15.16:0/3275153328 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f691006af00 0x7f691006d3c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f69180089b0 tx=0x7f6918008040 comp rx=0 tx=0).stop 2023-10-16T12:11:45.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.843+0000 7f69290e7700 1 -- 172.21.15.16:0/3275153328 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f692406e570 msgr2=0x7f69240b5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:45.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.843+0000 7f69290e7700 1 --2- 172.21.15.16:0/3275153328 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f692406e570 0x7f69240b5750 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f6914009130 tx=0x7f6914004750 comp rx=0 tx=0).stop 2023-10-16T12:11:45.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.844+0000 7f69290e7700 1 -- 172.21.15.16:0/3275153328 shutdown_connections 2023-10-16T12:11:45.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.844+0000 7f69290e7700 1 --2- 172.21.15.16:0/3275153328 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f691006af00 0x7f691006d3c0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:45.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.844+0000 7f69290e7700 1 --2- 172.21.15.16:0/3275153328 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f692415b2c0 0x7f69241f2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:45.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.844+0000 7f69290e7700 1 --2- 172.21.15.16:0/3275153328 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f692406a7e0 0x7f69240b17f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:45.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.844+0000 7f69290e7700 1 --2- 172.21.15.16:0/3275153328 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f692406e570 0x7f69240b5750 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:45.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.844+0000 7f69290e7700 1 -- 172.21.15.16:0/3275153328 >> 172.21.15.16:0/3275153328 conn(0x7f69240b1f40 msgr2=0x7f69240cc3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:45.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.844+0000 7f69290e7700 1 -- 172.21.15.16:0/3275153328 shutdown_connections 2023-10-16T12:11:45.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.844+0000 7f69290e7700 1 -- 172.21.15.16:0/3275153328 wait complete. 2023-10-16T12:11:45.859 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T12:11:45.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.984+0000 7f083f371700 1 Processor -- start 2023-10-16T12:11:45.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.984+0000 7f083f371700 1 -- start start 2023-10-16T12:11:45.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.985+0000 7f083f371700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f083806aef0 0x7f08380ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:45.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.985+0000 7f083f371700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f083806da30 0x7f08380adf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:45.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.985+0000 7f083f371700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08381613c0 con 0x7f0838160fe0 2023-10-16T12:11:45.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.985+0000 7f083f371700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08380cf680 con 0x7f083806da30 2023-10-16T12:11:45.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.985+0000 7f083f371700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08380adb00 con 0x7f083806aef0 2023-10-16T12:11:45.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.985+0000 7f083d90e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f083806da30 0x7f08380adf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:45.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.985+0000 7f083d10d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0838160fe0 0x7f083806a7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:52610/0 (socket says 172.21.15.16:52610) 2023-10-16T12:11:45.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.985+0000 7f083d90e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f083806da30 0x7f08380adf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35108/0 (socket says 172.21.15.16:35108) 2023-10-16T12:11:45.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.985+0000 7f083d10d700 1 -- 172.21.15.16:0/1193767660 learned_addr learned my addr 172.21.15.16:0/1193767660 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:45.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.985+0000 7f083c90c700 1 --2- 172.21.15.16:0/1193767660 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f083806aef0 0x7f08380ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:45.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.985+0000 7f083d90e700 1 -- 172.21.15.16:0/1193767660 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f083806aef0 msgr2=0x7f08380ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:45.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.985+0000 7f083d90e700 1 --2- 172.21.15.16:0/1193767660 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f083806aef0 0x7f08380ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:45.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.985+0000 7f083d90e700 1 -- 172.21.15.16:0/1193767660 >> v1:172.21.15.16:6789/0 conn(0x7f0838160fe0 legacy=0x7f083806a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:45.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.985+0000 7f083d90e700 1 -- 172.21.15.16:0/1193767660 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08380acef0 con 0x7f083806da30 2023-10-16T12:11:45.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.986+0000 7f083d90e700 1 --2- 172.21.15.16:0/1193767660 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f083806da30 0x7f08380adf90 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f0830009d90 tx=0x7f083000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75f5b87ad67005cc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:45.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.986+0000 7f082ffff700 1 -- 172.21.15.16:0/1193767660 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f083000e070 con 0x7f083806da30 2023-10-16T12:11:45.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.986+0000 7f082ffff700 1 -- 172.21.15.16:0/1193767660 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f083000bb20 con 0x7f083806da30 2023-10-16T12:11:45.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.986+0000 7f082ffff700 1 -- 172.21.15.16:0/1193767660 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0830012430 con 0x7f083806da30 2023-10-16T12:11:45.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.986+0000 7f083f371700 1 -- 172.21.15.16:0/1193767660 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f083806da30 msgr2=0x7f08380adf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:45.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.986+0000 7f083f371700 1 --2- 172.21.15.16:0/1193767660 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f083806da30 0x7f08380adf90 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f0830009d90 tx=0x7f083000b330 comp rx=0 tx=0).stop 2023-10-16T12:11:45.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.986+0000 7f083f371700 1 -- 172.21.15.16:0/1193767660 shutdown_connections 2023-10-16T12:11:45.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.986+0000 7f083f371700 1 --2- 172.21.15.16:0/1193767660 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f083806da30 0x7f08380adf90 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:45.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.986+0000 7f083f371700 1 --2- 172.21.15.16:0/1193767660 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f083806aef0 0x7f08380ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:45.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.986+0000 7f083f371700 1 -- 172.21.15.16:0/1193767660 >> 172.21.15.16:0/1193767660 conn(0x7f08380b1ed0 msgr2=0x7f08380b22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:45.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.986+0000 7f083f371700 1 -- 172.21.15.16:0/1193767660 shutdown_connections 2023-10-16T12:11:45.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.987+0000 7f083f371700 1 -- 172.21.15.16:0/1193767660 wait complete. 2023-10-16T12:11:45.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.988+0000 7f083f371700 1 Processor -- start 2023-10-16T12:11:45.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.988+0000 7f083f371700 1 -- start start 2023-10-16T12:11:45.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.988+0000 7f083f371700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f083806aef0 0x7f08380b0b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:45.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.988+0000 7f083f371700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f083806da30 0x7f083815c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:45.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.988+0000 7f083f371700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0838160fe0 0x7f08381f25f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:45.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.989+0000 7f083f371700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08380c3c60 con 0x7f083806da30 2023-10-16T12:11:45.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.989+0000 7f083f371700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08380b4fa0 con 0x7f0838160fe0 2023-10-16T12:11:45.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.989+0000 7f083f371700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08380c3ed0 con 0x7f083806aef0 2023-10-16T12:11:45.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.989+0000 7f083c90c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f083806da30 0x7f083815c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:45.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.989+0000 7f083c90c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f083806da30 0x7f083815c390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35112/0 (socket says 172.21.15.16:35112) 2023-10-16T12:11:45.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.989+0000 7f083c90c700 1 -- 172.21.15.16:0/3874613822 learned_addr learned my addr 172.21.15.16:0/3874613822 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:45.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.989+0000 7f083d10d700 1 --2- 172.21.15.16:0/3874613822 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f083806aef0 0x7f08380b0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:45.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.989+0000 7f083d90e700 1 --2- 172.21.15.16:0/3874613822 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0838160fe0 0x7f08381f25f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:45.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.989+0000 7f083c90c700 1 -- 172.21.15.16:0/3874613822 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f083806aef0 msgr2=0x7f08380b0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:45.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.989+0000 7f083c90c700 1 --2- 172.21.15.16:0/3874613822 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f083806aef0 0x7f08380b0b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:45.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.989+0000 7f083c90c700 1 -- 172.21.15.16:0/3874613822 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0838160fe0 msgr2=0x7f08381f25f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:45.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.989+0000 7f083c90c700 1 --2- 172.21.15.16:0/3874613822 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0838160fe0 0x7f08381f25f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:46.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.989+0000 7f083c90c700 1 -- 172.21.15.16:0/3874613822 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0830009a70 con 0x7f083806da30 2023-10-16T12:11:46.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.989+0000 7f083c90c700 1 --2- 172.21.15.16:0/3874613822 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f083806da30 0x7f083815c390 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f083400e870 tx=0x7f083400ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:46.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.989+0000 7f082dffb700 1 -- 172.21.15.16:0/3874613822 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0834009590 con 0x7f083806da30 2023-10-16T12:11:46.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.989+0000 7f082dffb700 1 -- 172.21.15.16:0/3874613822 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f083400ee70 con 0x7f083806da30 2023-10-16T12:11:46.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.989+0000 7f083f371700 1 -- 172.21.15.16:0/3874613822 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08380c41b0 con 0x7f083806da30 2023-10-16T12:11:46.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.989+0000 7f082dffb700 1 -- 172.21.15.16:0/3874613822 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f083400ca50 con 0x7f083806da30 2023-10-16T12:11:46.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.989+0000 7f083f371700 1 -- 172.21.15.16:0/3874613822 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08380b5280 con 0x7f083806da30 2023-10-16T12:11:46.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.991+0000 7f083f371700 1 -- 172.21.15.16:0/3874613822 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f083804ebb0 con 0x7f083806da30 2023-10-16T12:11:46.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.991+0000 7f082dffb700 1 -- 172.21.15.16:0/3874613822 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f083400cbb0 con 0x7f083806da30 2023-10-16T12:11:46.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.991+0000 7f082dffb700 1 --2- 172.21.15.16:0/3874613822 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f081c06af00 0x7f081c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:46.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.992+0000 7f083d10d700 1 --2- 172.21.15.16:0/3874613822 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f081c06af00 0x7f081c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:46.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.992+0000 7f082dffb700 1 -- 172.21.15.16:0/3874613822 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f0834089910 con 0x7f083806da30 2023-10-16T12:11:46.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.992+0000 7f083d10d700 1 --2- 172.21.15.16:0/3874613822 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f081c06af00 0x7f081c06d3c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f08280010e0 tx=0x7f0828009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:46.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:45.996+0000 7f082dffb700 1 -- 172.21.15.16:0/3874613822 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0834054600 con 0x7f083806da30 2023-10-16T12:11:46.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.201+0000 7f083f371700 1 -- 172.21.15.16:0/3874613822 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f08380cfa40 con 0x7f083806da30 2023-10-16T12:11:46.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.201+0000 7f082dffb700 1 -- 172.21.15.16:0/3874613822 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7f083400ce60 con 0x7f083806da30 2023-10-16T12:11:46.204 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-10-16T12:11:46.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.203+0000 7f083f371700 1 -- 172.21.15.16:0/3874613822 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f081c06af00 msgr2=0x7f081c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:46.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.203+0000 7f083f371700 1 --2- 172.21.15.16:0/3874613822 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f081c06af00 0x7f081c06d3c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f08280010e0 tx=0x7f0828009040 comp rx=0 tx=0).stop 2023-10-16T12:11:46.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.203+0000 7f083f371700 1 -- 172.21.15.16:0/3874613822 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f083806da30 msgr2=0x7f083815c390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:46.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.203+0000 7f083f371700 1 --2- 172.21.15.16:0/3874613822 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f083806da30 0x7f083815c390 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f083400e870 tx=0x7f083400ec30 comp rx=0 tx=0).stop 2023-10-16T12:11:46.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.203+0000 7f083f371700 1 -- 172.21.15.16:0/3874613822 shutdown_connections 2023-10-16T12:11:46.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.203+0000 7f083f371700 1 --2- 172.21.15.16:0/3874613822 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f081c06af00 0x7f081c06d3c0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:46.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.203+0000 7f083f371700 1 --2- 172.21.15.16:0/3874613822 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0838160fe0 0x7f08381f25f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:46.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.203+0000 7f083f371700 1 --2- 172.21.15.16:0/3874613822 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f083806aef0 0x7f08380b0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:46.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.203+0000 7f083f371700 1 --2- 172.21.15.16:0/3874613822 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f083806da30 0x7f083815c390 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:46.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.203+0000 7f083f371700 1 -- 172.21.15.16:0/3874613822 >> 172.21.15.16:0/3874613822 conn(0x7f08380b1ed0 msgr2=0x7f083815fa70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:46.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.203+0000 7f083f371700 1 -- 172.21.15.16:0/3874613822 shutdown_connections 2023-10-16T12:11:46.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.203+0000 7f083f371700 1 -- 172.21.15.16:0/3874613822 wait complete. 2023-10-16T12:11:46.218 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-16T12:11:46.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.344+0000 7f81cbfff700 1 Processor -- start 2023-10-16T12:11:46.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.344+0000 7f81cbfff700 1 -- start start 2023-10-16T12:11:46.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.345+0000 7f81cbfff700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f81cc15e600 0x7f81cc06dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:46.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.345+0000 7f81cbfff700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81cc06cbe0 con 0x7f81cc0cb900 2023-10-16T12:11:46.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.345+0000 7f81cbfff700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81cc06d740 con 0x7f81cc0aeea0 2023-10-16T12:11:46.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.345+0000 7f81cbfff700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81cc0af280 con 0x7f81cc15e600 2023-10-16T12:11:46.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.345+0000 7f81ca7fc700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f81cc0cb900 0x7f81cc15def0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:52626/0 (socket says 172.21.15.16:52626) 2023-10-16T12:11:46.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.345+0000 7f81ca7fc700 1 -- 172.21.15.16:0/741024584 learned_addr learned my addr 172.21.15.16:0/741024584 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:46.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.345+0000 7f81cb7fe700 1 --2- 172.21.15.16:0/741024584 >> v2:172.21.15.49:3300/0 conn(0x7f81cc15e600 0x7f81cc06dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:46.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.345+0000 7f81c9ffb700 1 -- 172.21.15.16:0/741024584 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2385420546 0 0) 0x7f81cc06cbe0 con 0x7f81cc0cb900 2023-10-16T12:11:46.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.345+0000 7f81c9ffb700 1 -- 172.21.15.16:0/741024584 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81b80035b0 con 0x7f81cc0cb900 2023-10-16T12:11:46.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.345+0000 7f81c9ffb700 1 -- 172.21.15.16:0/741024584 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3024195967 0 0) 0x7f81cc06d740 con 0x7f81cc0aeea0 2023-10-16T12:11:46.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.345+0000 7f81c9ffb700 1 -- 172.21.15.16:0/741024584 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81cc06cbe0 con 0x7f81cc0aeea0 2023-10-16T12:11:46.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.345+0000 7f81c9ffb700 1 -- 172.21.15.16:0/741024584 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2152324887 0 0) 0x7f81b80035b0 con 0x7f81cc0cb900 2023-10-16T12:11:46.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.345+0000 7f81cb7fe700 1 -- 172.21.15.16:0/741024584 >> v1:172.21.15.49:6789/0 conn(0x7f81cc0aeea0 legacy=0x7f81cc0cd550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:46.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.345+0000 7f81cb7fe700 1 -- 172.21.15.16:0/741024584 >> v1:172.21.15.16:6789/0 conn(0x7f81cc0cb900 legacy=0x7f81cc15def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:46.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.345+0000 7f81cb7fe700 1 -- 172.21.15.16:0/741024584 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81cc0cdc60 con 0x7f81cc15e600 2023-10-16T12:11:46.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.346+0000 7f81cb7fe700 1 --2- 172.21.15.16:0/741024584 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f81cc15e600 0x7f81cc06dbd0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f81c4009d90 tx=0x7f81c400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b6c13e5736c59472 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:46.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.346+0000 7f81c9ffb700 1 -- 172.21.15.16:0/741024584 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81c400e070 con 0x7f81cc15e600 2023-10-16T12:11:46.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.346+0000 7f81c9ffb700 1 -- 172.21.15.16:0/741024584 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81c4003a20 con 0x7f81cc15e600 2023-10-16T12:11:46.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.346+0000 7f81c9ffb700 1 -- 172.21.15.16:0/741024584 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81bc001670 con 0x7f81cc15e600 2023-10-16T12:11:46.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.346+0000 7f81cbfff700 1 -- 172.21.15.16:0/741024584 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f81cc15e600 msgr2=0x7f81cc06dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:46.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.346+0000 7f81cbfff700 1 --2- 172.21.15.16:0/741024584 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f81cc15e600 0x7f81cc06dbd0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f81c4009d90 tx=0x7f81c400b330 comp rx=0 tx=0).stop 2023-10-16T12:11:46.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.346+0000 7f81cbfff700 1 -- 172.21.15.16:0/741024584 shutdown_connections 2023-10-16T12:11:46.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.346+0000 7f81cbfff700 1 --2- 172.21.15.16:0/741024584 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f81cc15e600 0x7f81cc06dbd0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:46.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.346+0000 7f81cbfff700 1 -- 172.21.15.16:0/741024584 >> 172.21.15.16:0/741024584 conn(0x7f81cc0b1470 msgr2=0x7f81cc0b1880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:46.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.346+0000 7f81cbfff700 1 -- 172.21.15.16:0/741024584 shutdown_connections 2023-10-16T12:11:46.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.346+0000 7f81cbfff700 1 -- 172.21.15.16:0/741024584 wait complete. 2023-10-16T12:11:46.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.348+0000 7f81cbfff700 1 Processor -- start 2023-10-16T12:11:46.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.348+0000 7f81cbfff700 1 -- start start 2023-10-16T12:11:46.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.349+0000 7f81cbfff700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f81cc0aeea0 0x7f81cc0b32b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:46.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.349+0000 7f81cbfff700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f81cc0cb900 0x7f81cc0adc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:46.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.349+0000 7f81cbfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81cc15e600 0x7f81cc15caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:46.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.349+0000 7f81cbfff700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81cc0ae1a0 con 0x7f81cc15e600 2023-10-16T12:11:46.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.349+0000 7f81cbfff700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81cc0c6e30 con 0x7f81cc0cb900 2023-10-16T12:11:46.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.349+0000 7f81cbfff700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81cc0c7110 con 0x7f81cc0aeea0 2023-10-16T12:11:46.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.349+0000 7f81cb7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81cc15e600 0x7f81cc15caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:46.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.349+0000 7f81cb7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81cc15e600 0x7f81cc15caf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35128/0 (socket says 172.21.15.16:35128) 2023-10-16T12:11:46.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.349+0000 7f81cb7fe700 1 -- 172.21.15.16:0/1652990873 learned_addr learned my addr 172.21.15.16:0/1652990873 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:46.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.349+0000 7f81ca7fc700 1 --2- 172.21.15.16:0/1652990873 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f81cc0cb900 0x7f81cc0adc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:46.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.349+0000 7f81caffd700 1 --2- 172.21.15.16:0/1652990873 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f81cc0aeea0 0x7f81cc0b32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:46.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.349+0000 7f81cb7fe700 1 -- 172.21.15.16:0/1652990873 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f81cc0aeea0 msgr2=0x7f81cc0b32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:46.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.349+0000 7f81cb7fe700 1 --2- 172.21.15.16:0/1652990873 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f81cc0aeea0 0x7f81cc0b32b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:46.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.349+0000 7f81cb7fe700 1 -- 172.21.15.16:0/1652990873 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f81cc0cb900 msgr2=0x7f81cc0adc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:46.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.349+0000 7f81cb7fe700 1 --2- 172.21.15.16:0/1652990873 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f81cc0cb900 0x7f81cc0adc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:46.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.349+0000 7f81cb7fe700 1 -- 172.21.15.16:0/1652990873 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81c4009a70 con 0x7f81cc15e600 2023-10-16T12:11:46.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.349+0000 7f81caffd700 1 --2- 172.21.15.16:0/1652990873 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f81cc0aeea0 0x7f81cc0b32b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:11:46.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.349+0000 7f81cb7fe700 1 --2- 172.21.15.16:0/1652990873 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81cc15e600 0x7f81cc15caf0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f81c4015040 tx=0x7f81c4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:46.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.349+0000 7f81c37fe700 1 -- 172.21.15.16:0/1652990873 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81c4004540 con 0x7f81cc15e600 2023-10-16T12:11:46.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.350+0000 7f81c37fe700 1 -- 172.21.15.16:0/1652990873 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81c40046a0 con 0x7f81cc15e600 2023-10-16T12:11:46.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.350+0000 7f81cbfff700 1 -- 172.21.15.16:0/1652990873 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81cc0c73f0 con 0x7f81cc15e600 2023-10-16T12:11:46.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.350+0000 7f81c37fe700 1 -- 172.21.15.16:0/1652990873 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81c4012430 con 0x7f81cc15e600 2023-10-16T12:11:46.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.350+0000 7f81cbfff700 1 -- 172.21.15.16:0/1652990873 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81cc0c6a50 con 0x7f81cc15e600 2023-10-16T12:11:46.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.351+0000 7f81cbfff700 1 -- 172.21.15.16:0/1652990873 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81cc0c8120 con 0x7f81cc15e600 2023-10-16T12:11:46.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.351+0000 7f81c37fe700 1 -- 172.21.15.16:0/1652990873 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f81c4012590 con 0x7f81cc15e600 2023-10-16T12:11:46.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.351+0000 7f81c37fe700 1 --2- 172.21.15.16:0/1652990873 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f81ac06af00 0x7f81ac06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:46.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.352+0000 7f81c37fe700 1 -- 172.21.15.16:0/1652990873 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f81c408ae40 con 0x7f81cc15e600 2023-10-16T12:11:46.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.352+0000 7f81caffd700 1 --2- 172.21.15.16:0/1652990873 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f81ac06af00 0x7f81ac06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:46.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.352+0000 7f81caffd700 1 --2- 172.21.15.16:0/1652990873 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f81ac06af00 0x7f81ac06d3c0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f81b40011e0 tx=0x7f81b4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:46.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.356+0000 7f81c37fe700 1 -- 172.21.15.16:0/1652990873 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f81cc0c8120 con 0x7f81cc15e600 2023-10-16T12:11:46.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.560+0000 7f81cbfff700 1 -- 172.21.15.16:0/1652990873 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f81cc0c8120 con 0x7f81cc15e600 2023-10-16T12:11:46.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.561+0000 7f81c37fe700 1 -- 172.21.15.16:0/1652990873 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f81cc0c8120 con 0x7f81cc15e600 2023-10-16T12:11:46.564 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-10-16T12:11:46.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.563+0000 7f81cbfff700 1 -- 172.21.15.16:0/1652990873 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f81ac06af00 msgr2=0x7f81ac06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:46.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.563+0000 7f81cbfff700 1 --2- 172.21.15.16:0/1652990873 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f81ac06af00 0x7f81ac06d3c0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f81b40011e0 tx=0x7f81b4009040 comp rx=0 tx=0).stop 2023-10-16T12:11:46.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.563+0000 7f81cbfff700 1 -- 172.21.15.16:0/1652990873 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81cc15e600 msgr2=0x7f81cc15caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:46.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.563+0000 7f81cbfff700 1 --2- 172.21.15.16:0/1652990873 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81cc15e600 0x7f81cc15caf0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f81c4015040 tx=0x7f81c4004ad0 comp rx=0 tx=0).stop 2023-10-16T12:11:46.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.563+0000 7f81cbfff700 1 -- 172.21.15.16:0/1652990873 shutdown_connections 2023-10-16T12:11:46.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.563+0000 7f81cbfff700 1 --2- 172.21.15.16:0/1652990873 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f81ac06af00 0x7f81ac06d3c0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:46.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.563+0000 7f81cbfff700 1 --2- 172.21.15.16:0/1652990873 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81cc15e600 0x7f81cc15caf0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:46.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.563+0000 7f81cbfff700 1 --2- 172.21.15.16:0/1652990873 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f81cc0aeea0 0x7f81cc0b32b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:46.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.563+0000 7f81cbfff700 1 --2- 172.21.15.16:0/1652990873 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f81cc0cb900 0x7f81cc0adc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:46.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.563+0000 7f81cbfff700 1 -- 172.21.15.16:0/1652990873 >> 172.21.15.16:0/1652990873 conn(0x7f81cc0b1470 msgr2=0x7f81cc071bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:46.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.563+0000 7f81cbfff700 1 -- 172.21.15.16:0/1652990873 shutdown_connections 2023-10-16T12:11:46.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.563+0000 7f81cbfff700 1 -- 172.21.15.16:0/1652990873 wait complete. 2023-10-16T12:11:46.578 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:11:46.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.699+0000 7f64dc717700 1 Processor -- start 2023-10-16T12:11:46.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.699+0000 7f64dc717700 1 -- start start 2023-10-16T12:11:46.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.700+0000 7f64dc717700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f64d415b280 0x7f64d41613e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:46.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.700+0000 7f64dc717700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f64d406a7e0 0x7f64d406ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:46.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.700+0000 7f64dc717700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64d4161c30 con 0x7f64d40cc8a0 2023-10-16T12:11:46.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.700+0000 7f64dc717700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64d406b190 con 0x7f64d415b280 2023-10-16T12:11:46.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.700+0000 7f64dc717700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64d4161f10 con 0x7f64d406a7e0 2023-10-16T12:11:46.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.700+0000 7f64da4b3700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f64d415b280 0x7f64d41613e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:46.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.700+0000 7f64da4b3700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f64d415b280 0x7f64d41613e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35132/0 (socket says 172.21.15.16:35132) 2023-10-16T12:11:46.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.700+0000 7f64da4b3700 1 -- 172.21.15.16:0/3284162658 learned_addr learned my addr 172.21.15.16:0/3284162658 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:46.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.701+0000 7f64dacb4700 1 --2- 172.21.15.16:0/3284162658 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f64d406a7e0 0x7f64d406ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:46.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.701+0000 7f64da4b3700 1 -- 172.21.15.16:0/3284162658 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f64d406a7e0 msgr2=0x7f64d406ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:46.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.701+0000 7f64da4b3700 1 --2- 172.21.15.16:0/3284162658 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f64d406a7e0 0x7f64d406ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:46.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.701+0000 7f64da4b3700 1 -- 172.21.15.16:0/3284162658 >> v1:172.21.15.49:6789/0 conn(0x7f64d40cc8a0 legacy=0x7f64d415e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:46.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.701+0000 7f64da4b3700 1 -- 172.21.15.16:0/3284162658 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64d41621f0 con 0x7f64d415b280 2023-10-16T12:11:46.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.701+0000 7f64dacb4700 1 --2- 172.21.15.16:0/3284162658 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f64d406a7e0 0x7f64d406ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:11:46.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.701+0000 7f64da4b3700 1 --2- 172.21.15.16:0/3284162658 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64d415b280 0x7f64d41613e0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f64d00098f0 tx=0x7f64d0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8d0d77b2aa5fb80 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:46.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.701+0000 7f64d94b1700 1 -- 172.21.15.16:0/3284162658 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64d000b8a0 con 0x7f64d415b280 2023-10-16T12:11:46.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.701+0000 7f64d94b1700 1 -- 172.21.15.16:0/3284162658 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64d000ba00 con 0x7f64d415b280 2023-10-16T12:11:46.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.701+0000 7f64d94b1700 1 -- 172.21.15.16:0/3284162658 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64d000bd90 con 0x7f64d415b280 2023-10-16T12:11:46.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.702+0000 7f64dc717700 1 -- 172.21.15.16:0/3284162658 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64d415b280 msgr2=0x7f64d41613e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:46.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.702+0000 7f64dc717700 1 --2- 172.21.15.16:0/3284162658 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64d415b280 0x7f64d41613e0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f64d00098f0 tx=0x7f64d0009c00 comp rx=0 tx=0).stop 2023-10-16T12:11:46.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.702+0000 7f64dc717700 1 -- 172.21.15.16:0/3284162658 shutdown_connections 2023-10-16T12:11:46.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.702+0000 7f64dc717700 1 --2- 172.21.15.16:0/3284162658 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f64d406a7e0 0x7f64d406ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:46.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.702+0000 7f64dc717700 1 --2- 172.21.15.16:0/3284162658 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64d415b280 0x7f64d41613e0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:46.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.702+0000 7f64dc717700 1 -- 172.21.15.16:0/3284162658 >> 172.21.15.16:0/3284162658 conn(0x7f64d40b1fd0 msgr2=0x7f64d40b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:46.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.702+0000 7f64dc717700 1 -- 172.21.15.16:0/3284162658 shutdown_connections 2023-10-16T12:11:46.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.702+0000 7f64dc717700 1 -- 172.21.15.16:0/3284162658 wait complete. 2023-10-16T12:11:46.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.703+0000 7f64dc717700 1 Processor -- start 2023-10-16T12:11:46.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.703+0000 7f64dc717700 1 -- start start 2023-10-16T12:11:46.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.704+0000 7f64dc717700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64d406a7e0 0x7f64d40c4980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:46.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.704+0000 7f64dc717700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f64d40cc8a0 0x7f64d40c62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:46.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.704+0000 7f64dc717700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f64d415b280 0x7f64d4157d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:46.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.704+0000 7f64dc717700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64d41f2ea0 con 0x7f64d406a7e0 2023-10-16T12:11:46.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.704+0000 7f64dc717700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64d40c4ec0 con 0x7f64d40cc8a0 2023-10-16T12:11:46.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.704+0000 7f64dc717700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64d40c6820 con 0x7f64d415b280 2023-10-16T12:11:46.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.704+0000 7f64da4b3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64d406a7e0 0x7f64d40c4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:46.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.704+0000 7f64da4b3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64d406a7e0 0x7f64d40c4980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35138/0 (socket says 172.21.15.16:35138) 2023-10-16T12:11:46.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.704+0000 7f64da4b3700 1 -- 172.21.15.16:0/2779153392 learned_addr learned my addr 172.21.15.16:0/2779153392 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:46.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.704+0000 7f64dacb4700 1 --2- 172.21.15.16:0/2779153392 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f64d415b280 0x7f64d4157d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:46.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.704+0000 7f64d9cb2700 1 --2- 172.21.15.16:0/2779153392 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f64d40cc8a0 0x7f64d40c62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:46.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.704+0000 7f64da4b3700 1 -- 172.21.15.16:0/2779153392 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f64d415b280 msgr2=0x7f64d4157d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:46.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.704+0000 7f64da4b3700 1 --2- 172.21.15.16:0/2779153392 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f64d415b280 0x7f64d4157d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:46.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.704+0000 7f64da4b3700 1 -- 172.21.15.16:0/2779153392 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f64d40cc8a0 msgr2=0x7f64d40c62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:46.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.704+0000 7f64da4b3700 1 --2- 172.21.15.16:0/2779153392 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f64d40cc8a0 0x7f64d40c62e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:46.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.704+0000 7f64da4b3700 1 -- 172.21.15.16:0/2779153392 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64d0009550 con 0x7f64d406a7e0 2023-10-16T12:11:46.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.704+0000 7f64dacb4700 1 --2- 172.21.15.16:0/2779153392 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f64d415b280 0x7f64d4157d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:11:46.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.704+0000 7f64da4b3700 1 --2- 172.21.15.16:0/2779153392 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64d406a7e0 0x7f64d40c4980 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f64d0009130 tx=0x7f64d0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:46.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.705+0000 7f64c6ffd700 1 -- 172.21.15.16:0/2779153392 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64d00036a0 con 0x7f64d406a7e0 2023-10-16T12:11:46.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.705+0000 7f64c6ffd700 1 -- 172.21.15.16:0/2779153392 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64d001e070 con 0x7f64d406a7e0 2023-10-16T12:11:46.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.705+0000 7f64dc717700 1 -- 172.21.15.16:0/2779153392 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64d41582b0 con 0x7f64d406a7e0 2023-10-16T12:11:46.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.705+0000 7f64c6ffd700 1 -- 172.21.15.16:0/2779153392 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64d0003a60 con 0x7f64d406a7e0 2023-10-16T12:11:46.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.705+0000 7f64dc717700 1 -- 172.21.15.16:0/2779153392 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64d40c6d50 con 0x7f64d406a7e0 2023-10-16T12:11:46.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.706+0000 7f64dc717700 1 -- 172.21.15.16:0/2779153392 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64d404ebb0 con 0x7f64d406a7e0 2023-10-16T12:11:46.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.706+0000 7f64c6ffd700 1 -- 172.21.15.16:0/2779153392 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f64d0003bc0 con 0x7f64d406a7e0 2023-10-16T12:11:46.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.707+0000 7f64c6ffd700 1 --2- 172.21.15.16:0/2779153392 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f64b806af00 0x7f64b806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:46.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.707+0000 7f64d9cb2700 1 --2- 172.21.15.16:0/2779153392 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f64b806af00 0x7f64b806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:46.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.707+0000 7f64c6ffd700 1 -- 172.21.15.16:0/2779153392 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f64d0020070 con 0x7f64d406a7e0 2023-10-16T12:11:46.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.707+0000 7f64d9cb2700 1 --2- 172.21.15.16:0/2779153392 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f64b806af00 0x7f64b806d3c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f64d415a4f0 tx=0x7f64c8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:46.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.711+0000 7f64c6ffd700 1 -- 172.21.15.16:0/2779153392 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f64d0055c80 con 0x7f64d406a7e0 2023-10-16T12:11:46.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.957+0000 7f64dc717700 1 -- 172.21.15.16:0/2779153392 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f64d40ca2f0 con 0x7f64d406a7e0 2023-10-16T12:11:46.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.958+0000 7f64c6ffd700 1 -- 172.21.15.16:0/2779153392 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4909 (secure 0 0 0) 0x7f64d0015020 con 0x7f64d406a7e0 2023-10-16T12:11:46.961 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:11:46.961 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/2979266800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":2979266800},{"type":"v1","addr":"172.21.15.49:6834","nonce":2979266800}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4251,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/1398336801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1398336801},{"type":"v1","addr":"172.21.15.16:6836","nonce":1398336801}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4254,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3168304671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":3168304671},{"type":"v1","addr":"172.21.15.16:6837","nonce":3168304671}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:10:39.030243+0000","modified":"2023-10-16T12:11:02.593358+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4269":{"gid":4269,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.49:6835/2892536404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":2892536404},{"type":"v1","addr":"172.21.15.49:6835","nonce":2892536404}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T12:11:46.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.960+0000 7f64dc717700 1 -- 172.21.15.16:0/2779153392 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f64b806af00 msgr2=0x7f64b806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:46.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.960+0000 7f64dc717700 1 --2- 172.21.15.16:0/2779153392 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f64b806af00 0x7f64b806d3c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f64d415a4f0 tx=0x7f64c8009040 comp rx=0 tx=0).stop 2023-10-16T12:11:46.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.960+0000 7f64dc717700 1 -- 172.21.15.16:0/2779153392 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64d406a7e0 msgr2=0x7f64d40c4980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:46.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.960+0000 7f64dc717700 1 --2- 172.21.15.16:0/2779153392 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64d406a7e0 0x7f64d40c4980 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f64d0009130 tx=0x7f64d0004750 comp rx=0 tx=0).stop 2023-10-16T12:11:46.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.960+0000 7f64dc717700 1 -- 172.21.15.16:0/2779153392 shutdown_connections 2023-10-16T12:11:46.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.960+0000 7f64dc717700 1 --2- 172.21.15.16:0/2779153392 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f64b806af00 0x7f64b806d3c0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:46.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.960+0000 7f64dc717700 1 --2- 172.21.15.16:0/2779153392 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f64d415b280 0x7f64d4157d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:46.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.960+0000 7f64dc717700 1 --2- 172.21.15.16:0/2779153392 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64d406a7e0 0x7f64d40c4980 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:46.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.960+0000 7f64dc717700 1 --2- 172.21.15.16:0/2779153392 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f64d40cc8a0 0x7f64d40c62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:46.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.960+0000 7f64dc717700 1 -- 172.21.15.16:0/2779153392 >> 172.21.15.16:0/2779153392 conn(0x7f64d40b1fd0 msgr2=0x7f64d40d0270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:46.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.960+0000 7f64dc717700 1 -- 172.21.15.16:0/2779153392 shutdown_connections 2023-10-16T12:11:46.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:46.960+0000 7f64dc717700 1 -- 172.21.15.16:0/2779153392 wait complete. 2023-10-16T12:11:46.966 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 9 2023-10-16T12:11:46.975 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:11:47.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.097+0000 7f0d64409700 1 Processor -- start 2023-10-16T12:11:47.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.097+0000 7f0d64409700 1 -- start start 2023-10-16T12:11:47.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.098+0000 7f0d64409700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f0d5c15d750 0x7f0d5c0cf920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:47.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.098+0000 7f0d64409700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0d5c15db30 0x7f0d5c06c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:47.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.098+0000 7f0d64409700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d5c0d0070 con 0x7f0d5c161490 2023-10-16T12:11:47.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.098+0000 7f0d64409700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d5c0cfe60 con 0x7f0d5c15d750 2023-10-16T12:11:47.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.098+0000 7f0d64409700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d5c0d0350 con 0x7f0d5c15db30 2023-10-16T12:11:47.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.099+0000 7f0d621a5700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f0d5c15d750 0x7f0d5c0cf920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:47.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.099+0000 7f0d619a4700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0d5c15db30 0x7f0d5c06c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:47.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.099+0000 7f0d629a6700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7f0d5c161490 0x7f0d5c0b2060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:59838/0 (socket says 172.21.15.16:59838) 2023-10-16T12:11:47.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.099+0000 7f0d629a6700 1 -- 172.21.15.16:0/494156864 learned_addr learned my addr 172.21.15.16:0/494156864 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:47.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.099+0000 7f0d619a4700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0d5c15db30 0x7f0d5c06c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:51636/0 (socket says 172.21.15.16:51636) 2023-10-16T12:11:47.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.099+0000 7f0d621a5700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f0d5c15d750 0x7f0d5c0cf920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:34208/0 (socket says 172.21.15.16:34208) 2023-10-16T12:11:47.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.099+0000 7f0d611a3700 1 -- 172.21.15.16:0/494156864 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 455449951 0 0) 0x7f0d5c0d0070 con 0x7f0d5c161490 2023-10-16T12:11:47.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.099+0000 7f0d611a3700 1 -- 172.21.15.16:0/494156864 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d480035b0 con 0x7f0d5c161490 2023-10-16T12:11:47.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.099+0000 7f0d621a5700 1 -- 172.21.15.16:0/494156864 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0d5c15db30 msgr2=0x7f0d5c06c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:47.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.099+0000 7f0d621a5700 1 --2- 172.21.15.16:0/494156864 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0d5c15db30 0x7f0d5c06c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:47.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.099+0000 7f0d621a5700 1 -- 172.21.15.16:0/494156864 >> v1:172.21.15.49:6789/0 conn(0x7f0d5c161490 legacy=0x7f0d5c0b2060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:47.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.099+0000 7f0d621a5700 1 -- 172.21.15.16:0/494156864 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d5c0d0630 con 0x7f0d5c15d750 2023-10-16T12:11:47.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.099+0000 7f0d621a5700 1 --2- 172.21.15.16:0/494156864 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0d5c15d750 0x7f0d5c0cf920 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f0d4c009820 tx=0x7f0d4c009b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ba8c1a439df33033 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:47.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.099+0000 7f0d611a3700 1 -- 172.21.15.16:0/494156864 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d4c005e30 con 0x7f0d5c15d750 2023-10-16T12:11:47.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.099+0000 7f0d611a3700 1 -- 172.21.15.16:0/494156864 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d4c006030 con 0x7f0d5c15d750 2023-10-16T12:11:47.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.099+0000 7f0d611a3700 1 -- 172.21.15.16:0/494156864 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d4c0063c0 con 0x7f0d5c15d750 2023-10-16T12:11:47.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.100+0000 7f0d64409700 1 -- 172.21.15.16:0/494156864 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0d5c15d750 msgr2=0x7f0d5c0cf920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:47.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.100+0000 7f0d64409700 1 --2- 172.21.15.16:0/494156864 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0d5c15d750 0x7f0d5c0cf920 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f0d4c009820 tx=0x7f0d4c009b30 comp rx=0 tx=0).stop 2023-10-16T12:11:47.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.100+0000 7f0d64409700 1 -- 172.21.15.16:0/494156864 shutdown_connections 2023-10-16T12:11:47.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.100+0000 7f0d64409700 1 --2- 172.21.15.16:0/494156864 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0d5c15d750 0x7f0d5c0cf920 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:47.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.100+0000 7f0d64409700 1 --2- 172.21.15.16:0/494156864 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0d5c15db30 0x7f0d5c06c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:47.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.100+0000 7f0d64409700 1 -- 172.21.15.16:0/494156864 >> 172.21.15.16:0/494156864 conn(0x7f0d5c0d14b0 msgr2=0x7f0d5c0d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:47.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.100+0000 7f0d64409700 1 -- 172.21.15.16:0/494156864 shutdown_connections 2023-10-16T12:11:47.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.100+0000 7f0d64409700 1 -- 172.21.15.16:0/494156864 wait complete. 2023-10-16T12:11:47.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.101+0000 7f0d64409700 1 Processor -- start 2023-10-16T12:11:47.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.101+0000 7f0d64409700 1 -- start start 2023-10-16T12:11:47.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.102+0000 7f0d64409700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0d5c15d750 0x7f0d5c15ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:47.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.102+0000 7f0d64409700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d5c15db30 0x7f0d5c09bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:47.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.102+0000 7f0d64409700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0d5c161490 0x7f0d5c15eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:47.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.102+0000 7f0d64409700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d5c09cbb0 con 0x7f0d5c15db30 2023-10-16T12:11:47.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.102+0000 7f0d64409700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d5c1f2d80 con 0x7f0d5c15d750 2023-10-16T12:11:47.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.102+0000 7f0d64409700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d5c09c8f0 con 0x7f0d5c161490 2023-10-16T12:11:47.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.102+0000 7f0d619a4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d5c15db30 0x7f0d5c09bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:47.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.102+0000 7f0d619a4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d5c15db30 0x7f0d5c09bf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35154/0 (socket says 172.21.15.16:35154) 2023-10-16T12:11:47.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.102+0000 7f0d619a4700 1 -- 172.21.15.16:0/1844973442 learned_addr learned my addr 172.21.15.16:0/1844973442 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:47.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.102+0000 7f0d621a5700 1 --2- 172.21.15.16:0/1844973442 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0d5c15d750 0x7f0d5c15ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:47.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.102+0000 7f0d629a6700 1 --2- 172.21.15.16:0/1844973442 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0d5c161490 0x7f0d5c15eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:47.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.102+0000 7f0d619a4700 1 -- 172.21.15.16:0/1844973442 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0d5c161490 msgr2=0x7f0d5c15eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:47.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.102+0000 7f0d619a4700 1 --2- 172.21.15.16:0/1844973442 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0d5c161490 0x7f0d5c15eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:47.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.102+0000 7f0d619a4700 1 -- 172.21.15.16:0/1844973442 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0d5c15d750 msgr2=0x7f0d5c15ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:47.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.102+0000 7f0d619a4700 1 --2- 172.21.15.16:0/1844973442 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0d5c15d750 0x7f0d5c15ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:47.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.102+0000 7f0d619a4700 1 -- 172.21.15.16:0/1844973442 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d4c009430 con 0x7f0d5c15db30 2023-10-16T12:11:47.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.102+0000 7f0d619a4700 1 --2- 172.21.15.16:0/1844973442 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d5c15db30 0x7f0d5c09bf00 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f0d5800e980 tx=0x7f0d5800ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:47.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.103+0000 7f0d52ffd700 1 -- 172.21.15.16:0/1844973442 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d58009710 con 0x7f0d5c15db30 2023-10-16T12:11:47.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.103+0000 7f0d52ffd700 1 -- 172.21.15.16:0/1844973442 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d5800c690 con 0x7f0d5c15db30 2023-10-16T12:11:47.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.103+0000 7f0d64409700 1 -- 172.21.15.16:0/1844973442 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d5c1f2fd0 con 0x7f0d5c15db30 2023-10-16T12:11:47.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.103+0000 7f0d52ffd700 1 -- 172.21.15.16:0/1844973442 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d5800ca40 con 0x7f0d5c15db30 2023-10-16T12:11:47.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.103+0000 7f0d64409700 1 -- 172.21.15.16:0/1844973442 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d5c1f3590 con 0x7f0d5c15db30 2023-10-16T12:11:47.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.104+0000 7f0d64409700 1 -- 172.21.15.16:0/1844973442 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d5c1f37b0 con 0x7f0d5c15db30 2023-10-16T12:11:47.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.104+0000 7f0d52ffd700 1 -- 172.21.15.16:0/1844973442 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f0d5800cba0 con 0x7f0d5c15db30 2023-10-16T12:11:47.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.104+0000 7f0d52ffd700 1 --2- 172.21.15.16:0/1844973442 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f0d4006af00 0x7f0d4006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:47.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.104+0000 7f0d52ffd700 1 -- 172.21.15.16:0/1844973442 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f0d58089930 con 0x7f0d5c15db30 2023-10-16T12:11:47.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.105+0000 7f0d621a5700 1 --2- 172.21.15.16:0/1844973442 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f0d4006af00 0x7f0d4006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:47.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.105+0000 7f0d621a5700 1 --2- 172.21.15.16:0/1844973442 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f0d4006af00 0x7f0d4006d3c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f0d4c000c00 tx=0x7f0d4c008070 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:47.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.109+0000 7f0d52ffd700 1 -- 172.21.15.16:0/1844973442 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0d58054620 con 0x7f0d5c15db30 2023-10-16T12:11:47.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.315+0000 7f0d64409700 1 -- 172.21.15.16:0/1844973442 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0d5c04ebb0 con 0x7f0d5c15db30 2023-10-16T12:11:47.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.316+0000 7f0d52ffd700 1 -- 172.21.15.16:0/1844973442 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18629 (secure 0 0 0) 0x7f0d5805e030 con 0x7f0d5c15db30 2023-10-16T12:11:47.319 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:11:47.320 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:11:28.241865+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:10:36.935773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T12:10:38.004870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-16T12:10:41.672025+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"3.2b","mappings":[{"from":0,"to":2}]},{"pgid":"3.38","mappings":[{"from":3,"to":1}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]},{"pgid":"4.1b","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:11:47.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.318+0000 7f0d64409700 1 -- 172.21.15.16:0/1844973442 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f0d4006af00 msgr2=0x7f0d4006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:47.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.318+0000 7f0d64409700 1 --2- 172.21.15.16:0/1844973442 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f0d4006af00 0x7f0d4006d3c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f0d4c000c00 tx=0x7f0d4c008070 comp rx=0 tx=0).stop 2023-10-16T12:11:47.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.318+0000 7f0d64409700 1 -- 172.21.15.16:0/1844973442 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d5c15db30 msgr2=0x7f0d5c09bf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:47.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.318+0000 7f0d64409700 1 --2- 172.21.15.16:0/1844973442 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d5c15db30 0x7f0d5c09bf00 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f0d5800e980 tx=0x7f0d5800ec90 comp rx=0 tx=0).stop 2023-10-16T12:11:47.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.318+0000 7f0d64409700 1 -- 172.21.15.16:0/1844973442 shutdown_connections 2023-10-16T12:11:47.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.318+0000 7f0d64409700 1 --2- 172.21.15.16:0/1844973442 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f0d4006af00 0x7f0d4006d3c0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:47.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.318+0000 7f0d64409700 1 --2- 172.21.15.16:0/1844973442 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0d5c161490 0x7f0d5c15eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:47.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.318+0000 7f0d64409700 1 --2- 172.21.15.16:0/1844973442 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0d5c15d750 0x7f0d5c15ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:47.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.318+0000 7f0d64409700 1 --2- 172.21.15.16:0/1844973442 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d5c15db30 0x7f0d5c09bf00 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:47.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.318+0000 7f0d64409700 1 -- 172.21.15.16:0/1844973442 >> 172.21.15.16:0/1844973442 conn(0x7f0d5c0d14b0 msgr2=0x7f0d5c0d1890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:47.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.318+0000 7f0d64409700 1 -- 172.21.15.16:0/1844973442 shutdown_connections 2023-10-16T12:11:47.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.318+0000 7f0d64409700 1 -- 172.21.15.16:0/1844973442 wait complete. 2023-10-16T12:11:47.334 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-16T12:11:47.334 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:11:47.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.456+0000 7f73b49f7700 1 Processor -- start 2023-10-16T12:11:47.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.457+0000 7f73b49f7700 1 -- start start 2023-10-16T12:11:47.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.457+0000 7f73b49f7700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f73b00ce010 0x7f73b00cef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:47.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.457+0000 7f73b49f7700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73b0072a50 con 0x7f73b00d0ae0 2023-10-16T12:11:47.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.457+0000 7f73b49f7700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73b00cba30 con 0x7f73b015a5c0 2023-10-16T12:11:47.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.457+0000 7f73b49f7700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73b00cf440 con 0x7f73b00ce010 2023-10-16T12:11:47.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.458+0000 7f73add9b700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f73b00d0ae0 0x7f73b00cc1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:52666/0 (socket says 172.21.15.16:52666) 2023-10-16T12:11:47.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.458+0000 7f73add9b700 1 -- 172.21.15.16:0/2636215020 learned_addr learned my addr 172.21.15.16:0/2636215020 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:47.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.458+0000 7f73aed9d700 1 --2- 172.21.15.16:0/2636215020 >> v2:172.21.15.49:3300/0 conn(0x7f73b00ce010 0x7f73b00cef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:47.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.458+0000 7f73ad59a700 1 -- 172.21.15.16:0/2636215020 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3679700484 0 0) 0x7f73b0072a50 con 0x7f73b00d0ae0 2023-10-16T12:11:47.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.458+0000 7f73ad59a700 1 -- 172.21.15.16:0/2636215020 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73980057c0 con 0x7f73b00d0ae0 2023-10-16T12:11:47.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.458+0000 7f73ad59a700 1 -- 172.21.15.16:0/2636215020 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 352532700 0 0) 0x7f73980057c0 con 0x7f73b00d0ae0 2023-10-16T12:11:47.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.458+0000 7f73ad59a700 1 -- 172.21.15.16:0/2636215020 >> v2:172.21.15.49:3300/0 conn(0x7f73b00ce010 msgr2=0x7f73b00cef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:47.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.458+0000 7f73ad59a700 1 --2- 172.21.15.16:0/2636215020 >> v2:172.21.15.49:3300/0 conn(0x7f73b00ce010 0x7f73b00cef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:47.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.458+0000 7f73ad59a700 1 -- 172.21.15.16:0/2636215020 >> v1:172.21.15.49:6789/0 conn(0x7f73b015a5c0 legacy=0x7f73b00d03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:47.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.458+0000 7f73ad59a700 1 -- 172.21.15.16:0/2636215020 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73b00cf630 con 0x7f73b00d0ae0 2023-10-16T12:11:47.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.458+0000 7f73ad59a700 1 -- 172.21.15.16:0/2636215020 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f73a0001670 con 0x7f73b00d0ae0 2023-10-16T12:11:47.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.458+0000 7f73ad59a700 1 -- 172.21.15.16:0/2636215020 >> v1:172.21.15.16:6789/0 conn(0x7f73b00d0ae0 legacy=0x7f73b00cc1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:47.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.458+0000 7f73ad59a700 1 --2- 172.21.15.16:0/2636215020 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7398005c10 0x7f7398006030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:47.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.458+0000 7f73ad59a700 1 --2- 172.21.15.16:0/2636215020 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7398006a60 0x7f7398006ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:47.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.458+0000 7f73ad59a700 1 --2- 172.21.15.16:0/2636215020 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7398007420 0x7f739800dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:47.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.458+0000 7f73ad59a700 1 -- 172.21.15.16:0/2636215020 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f739800e290 con 0x7f7398007420 2023-10-16T12:11:47.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.458+0000 7f73ad59a700 1 -- 172.21.15.16:0/2636215020 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f739800e400 con 0x7f7398005c10 2023-10-16T12:11:47.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.458+0000 7f73ad59a700 1 -- 172.21.15.16:0/2636215020 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f739800e5c0 con 0x7f7398006a60 2023-10-16T12:11:47.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.458+0000 7f73aed9d700 1 --2- 172.21.15.16:0/2636215020 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7398007420 0x7f739800dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:47.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.458+0000 7f73ae59c700 1 --2- 172.21.15.16:0/2636215020 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7398005c10 0x7f7398006030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:47.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.458+0000 7f73add9b700 1 --2- 172.21.15.16:0/2636215020 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7398006a60 0x7f7398006ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:47.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.459+0000 7f73aed9d700 1 -- 172.21.15.16:0/2636215020 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7398006a60 msgr2=0x7f7398006ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:47.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.459+0000 7f73aed9d700 1 --2- 172.21.15.16:0/2636215020 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7398006a60 0x7f7398006ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:47.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.459+0000 7f73aed9d700 1 -- 172.21.15.16:0/2636215020 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7398005c10 msgr2=0x7f7398006030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:47.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.459+0000 7f73aed9d700 1 --2- 172.21.15.16:0/2636215020 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7398005c10 0x7f7398006030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:47.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.459+0000 7f73aed9d700 1 -- 172.21.15.16:0/2636215020 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73980057c0 con 0x7f7398007420 2023-10-16T12:11:47.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.459+0000 7f73aed9d700 1 -- 172.21.15.16:0/2636215020 reap_dead start 2023-10-16T12:11:47.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.459+0000 7f73aed9d700 1 --2- 172.21.15.16:0/2636215020 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7398007420 0x7f739800dc20 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f73a8009fc0 tx=0x7f7398007a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7886bf9fcb06da5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:47.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.459+0000 7f73ad59a700 1 -- 172.21.15.16:0/2636215020 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a8003a20 con 0x7f7398007420 2023-10-16T12:11:47.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.459+0000 7f73ad59a700 1 -- 172.21.15.16:0/2636215020 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73a800f040 con 0x7f7398007420 2023-10-16T12:11:47.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.459+0000 7f73ad59a700 1 -- 172.21.15.16:0/2636215020 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a8003de0 con 0x7f7398007420 2023-10-16T12:11:47.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.460+0000 7f73b49f7700 1 -- 172.21.15.16:0/2636215020 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7398007420 msgr2=0x7f739800dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:47.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.460+0000 7f73b49f7700 1 --2- 172.21.15.16:0/2636215020 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7398007420 0x7f739800dc20 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f73a8009fc0 tx=0x7f7398007a90 comp rx=0 tx=0).stop 2023-10-16T12:11:47.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.460+0000 7f73b49f7700 1 -- 172.21.15.16:0/2636215020 shutdown_connections 2023-10-16T12:11:47.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.460+0000 7f73b49f7700 1 --2- 172.21.15.16:0/2636215020 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7398007420 0x7f739800dc20 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:47.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.460+0000 7f73b49f7700 1 -- 172.21.15.16:0/2636215020 >> 172.21.15.16:0/2636215020 conn(0x7f73b00b1fd0 msgr2=0x7f73b00b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:47.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.460+0000 7f73b49f7700 1 -- 172.21.15.16:0/2636215020 shutdown_connections 2023-10-16T12:11:47.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.460+0000 7f73b49f7700 1 -- 172.21.15.16:0/2636215020 wait complete. 2023-10-16T12:11:47.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.461+0000 7f73b49f7700 1 Processor -- start 2023-10-16T12:11:47.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.461+0000 7f73b49f7700 1 -- start start 2023-10-16T12:11:47.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.462+0000 7f73b49f7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7398007420 0x7f73b01f3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:47.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.462+0000 7f73b49f7700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f73b00b5a90 0x7f73b00cd340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:47.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.462+0000 7f73b49f7700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f73b0157060 0x7f73b0157510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:47.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.462+0000 7f73b49f7700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73b01f3b90 con 0x7f7398007420 2023-10-16T12:11:47.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.462+0000 7f73b49f7700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73b01f6240 con 0x7f73b0157060 2023-10-16T12:11:47.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.462+0000 7f73b49f7700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73b01f6520 con 0x7f73b00b5a90 2023-10-16T12:11:47.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.462+0000 7f73ae59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7398007420 0x7f73b01f3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:47.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.462+0000 7f73ae59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7398007420 0x7f73b01f3650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35172/0 (socket says 172.21.15.16:35172) 2023-10-16T12:11:47.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.462+0000 7f73ae59c700 1 -- 172.21.15.16:0/110112913 learned_addr learned my addr 172.21.15.16:0/110112913 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:47.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.462+0000 7f73aed9d700 1 --2- 172.21.15.16:0/110112913 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f73b0157060 0x7f73b0157510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:47.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.462+0000 7f73add9b700 1 --2- 172.21.15.16:0/110112913 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f73b00b5a90 0x7f73b00cd340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:47.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.462+0000 7f73ae59c700 1 -- 172.21.15.16:0/110112913 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f73b00b5a90 msgr2=0x7f73b00cd340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:47.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.462+0000 7f73ae59c700 1 --2- 172.21.15.16:0/110112913 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f73b00b5a90 0x7f73b00cd340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:47.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.462+0000 7f73ae59c700 1 -- 172.21.15.16:0/110112913 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f73b0157060 msgr2=0x7f73b0157510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:47.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.462+0000 7f73ae59c700 1 --2- 172.21.15.16:0/110112913 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f73b0157060 0x7f73b0157510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:47.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.462+0000 7f73ae59c700 1 -- 172.21.15.16:0/110112913 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73b01f6800 con 0x7f7398007420 2023-10-16T12:11:47.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.462+0000 7f73ae59c700 1 --2- 172.21.15.16:0/110112913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7398007420 0x7f73b01f3650 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f73a400f820 tx=0x7f73a400fb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:47.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.463+0000 7f739effd700 1 -- 172.21.15.16:0/110112913 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a40052e0 con 0x7f7398007420 2023-10-16T12:11:47.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.463+0000 7f739effd700 1 -- 172.21.15.16:0/110112913 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73a4005440 con 0x7f7398007420 2023-10-16T12:11:47.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.463+0000 7f73b49f7700 1 -- 172.21.15.16:0/110112913 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73b01f6ae0 con 0x7f7398007420 2023-10-16T12:11:47.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.463+0000 7f739effd700 1 -- 172.21.15.16:0/110112913 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a40038a0 con 0x7f7398007420 2023-10-16T12:11:47.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.463+0000 7f73b49f7700 1 -- 172.21.15.16:0/110112913 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73b00ae8e0 con 0x7f7398007420 2023-10-16T12:11:47.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.464+0000 7f73b49f7700 1 -- 172.21.15.16:0/110112913 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73b006b550 con 0x7f7398007420 2023-10-16T12:11:47.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.464+0000 7f739effd700 1 -- 172.21.15.16:0/110112913 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f73a4003a00 con 0x7f7398007420 2023-10-16T12:11:47.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.464+0000 7f739effd700 1 --2- 172.21.15.16:0/110112913 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f739006af00 0x7f739006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:47.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.464+0000 7f739effd700 1 -- 172.21.15.16:0/110112913 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f73a4089520 con 0x7f7398007420 2023-10-16T12:11:47.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.465+0000 7f73add9b700 1 --2- 172.21.15.16:0/110112913 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f739006af00 0x7f739006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:47.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.465+0000 7f73add9b700 1 --2- 172.21.15.16:0/110112913 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f739006af00 0x7f739006d3c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f73b00b1dd0 tx=0x7f73a0000ed0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:47.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.469+0000 7f739effd700 1 -- 172.21.15.16:0/110112913 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f73a40542a0 con 0x7f7398007420 2023-10-16T12:11:47.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.724+0000 7f73b49f7700 1 -- 172.21.15.16:0/110112913 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f73b0071980 con 0x7f7398007420 2023-10-16T12:11:47.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.724+0000 7f739effd700 1 -- 172.21.15.16:0/110112913 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4909 (secure 0 0 0) 0x7f73a4003cb0 con 0x7f7398007420 2023-10-16T12:11:47.727 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:11:47.728 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/2979266800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":2979266800},{"type":"v1","addr":"172.21.15.49:6834","nonce":2979266800}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4251,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/1398336801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1398336801},{"type":"v1","addr":"172.21.15.16:6836","nonce":1398336801}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4254,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3168304671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":3168304671},{"type":"v1","addr":"172.21.15.16:6837","nonce":3168304671}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:10:39.030243+0000","modified":"2023-10-16T12:11:02.593358+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4269":{"gid":4269,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.49:6835/2892536404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":2892536404},{"type":"v1","addr":"172.21.15.49:6835","nonce":2892536404}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T12:11:47.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.727+0000 7f73b49f7700 1 -- 172.21.15.16:0/110112913 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f739006af00 msgr2=0x7f739006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:47.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.727+0000 7f73b49f7700 1 --2- 172.21.15.16:0/110112913 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f739006af00 0x7f739006d3c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f73b00b1dd0 tx=0x7f73a0000ed0 comp rx=0 tx=0).stop 2023-10-16T12:11:47.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.727+0000 7f73b49f7700 1 -- 172.21.15.16:0/110112913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7398007420 msgr2=0x7f73b01f3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:47.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.727+0000 7f73b49f7700 1 --2- 172.21.15.16:0/110112913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7398007420 0x7f73b01f3650 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f73a400f820 tx=0x7f73a400fb30 comp rx=0 tx=0).stop 2023-10-16T12:11:47.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.727+0000 7f73b49f7700 1 -- 172.21.15.16:0/110112913 shutdown_connections 2023-10-16T12:11:47.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.727+0000 7f73b49f7700 1 --2- 172.21.15.16:0/110112913 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f739006af00 0x7f739006d3c0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:47.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.727+0000 7f73b49f7700 1 --2- 172.21.15.16:0/110112913 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f73b0157060 0x7f73b0157510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:47.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.727+0000 7f73b49f7700 1 --2- 172.21.15.16:0/110112913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7398007420 0x7f73b01f3650 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:47.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.727+0000 7f73b49f7700 1 --2- 172.21.15.16:0/110112913 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f73b00b5a90 0x7f73b00cd340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:47.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.727+0000 7f73b49f7700 1 -- 172.21.15.16:0/110112913 >> 172.21.15.16:0/110112913 conn(0x7f73b00b1fd0 msgr2=0x7f73b00afd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:47.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.727+0000 7f73b49f7700 1 -- 172.21.15.16:0/110112913 shutdown_connections 2023-10-16T12:11:47.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.727+0000 7f73b49f7700 1 -- 172.21.15.16:0/110112913 wait complete. 2023-10-16T12:11:47.732 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 9 2023-10-16T12:11:47.742 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:11:47.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.866+0000 7f4d6509c700 1 Processor -- start 2023-10-16T12:11:47.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.866+0000 7f4d6509c700 1 -- start start 2023-10-16T12:11:47.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.866+0000 7f4d6509c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4d600cc8a0 0x7f4d6015e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:47.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.866+0000 7f4d6509c700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f4d6006a7e0 0x7f4d6006ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:47.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.866+0000 7f4d6509c700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d6006b150 con 0x7f4d6015b280 2023-10-16T12:11:47.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.866+0000 7f4d6509c700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d60161c30 con 0x7f4d600cc8a0 2023-10-16T12:11:47.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.866+0000 7f4d6509c700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d60161f10 con 0x7f4d6006a7e0 2023-10-16T12:11:47.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.867+0000 7f4d5e59c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4d600cc8a0 0x7f4d6015e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:47.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.867+0000 7f4d5e59c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4d600cc8a0 0x7f4d6015e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35180/0 (socket says 172.21.15.16:35180) 2023-10-16T12:11:47.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.867+0000 7f4d5e59c700 1 -- 172.21.15.16:0/4028475470 learned_addr learned my addr 172.21.15.16:0/4028475470 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:47.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.867+0000 7f4d5f59e700 1 --2- 172.21.15.16:0/4028475470 >> v2:172.21.15.49:3300/0 conn(0x7f4d6006a7e0 0x7f4d6006ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:47.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.867+0000 7f4d5e59c700 1 -- 172.21.15.16:0/4028475470 >> v2:172.21.15.49:3300/0 conn(0x7f4d6006a7e0 msgr2=0x7f4d6006ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:47.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.867+0000 7f4d5e59c700 1 --2- 172.21.15.16:0/4028475470 >> v2:172.21.15.49:3300/0 conn(0x7f4d6006a7e0 0x7f4d6006ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:47.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.867+0000 7f4d5e59c700 1 -- 172.21.15.16:0/4028475470 >> v1:172.21.15.49:6789/0 conn(0x7f4d6015b280 legacy=0x7f4d601613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:47.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.867+0000 7f4d5e59c700 1 -- 172.21.15.16:0/4028475470 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d601621f0 con 0x7f4d600cc8a0 2023-10-16T12:11:47.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.867+0000 7f4d5e59c700 1 --2- 172.21.15.16:0/4028475470 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d600cc8a0 0x7f4d6015e700 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f4d600cbed0 tx=0x7f4d48009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7b75a3bce304784 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:47.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.867+0000 7f4d5dd9b700 1 -- 172.21.15.16:0/4028475470 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d48004230 con 0x7f4d600cc8a0 2023-10-16T12:11:47.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.867+0000 7f4d5dd9b700 1 -- 172.21.15.16:0/4028475470 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d48004390 con 0x7f4d600cc8a0 2023-10-16T12:11:47.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.867+0000 7f4d5dd9b700 1 -- 172.21.15.16:0/4028475470 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d480036a0 con 0x7f4d600cc8a0 2023-10-16T12:11:47.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.868+0000 7f4d6509c700 1 -- 172.21.15.16:0/4028475470 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d600cc8a0 msgr2=0x7f4d6015e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:47.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.868+0000 7f4d6509c700 1 --2- 172.21.15.16:0/4028475470 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d600cc8a0 0x7f4d6015e700 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f4d600cbed0 tx=0x7f4d48009bd0 comp rx=0 tx=0).stop 2023-10-16T12:11:47.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.868+0000 7f4d6509c700 1 -- 172.21.15.16:0/4028475470 shutdown_connections 2023-10-16T12:11:47.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.868+0000 7f4d6509c700 1 --2- 172.21.15.16:0/4028475470 >> v2:172.21.15.49:3300/0 conn(0x7f4d6006a7e0 0x7f4d6006ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:47.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.868+0000 7f4d6509c700 1 --2- 172.21.15.16:0/4028475470 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d600cc8a0 0x7f4d6015e700 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:47.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.868+0000 7f4d6509c700 1 -- 172.21.15.16:0/4028475470 >> 172.21.15.16:0/4028475470 conn(0x7f4d600b1fd0 msgr2=0x7f4d600b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:47.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.868+0000 7f4d6509c700 1 -- 172.21.15.16:0/4028475470 shutdown_connections 2023-10-16T12:11:47.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.868+0000 7f4d6509c700 1 -- 172.21.15.16:0/4028475470 wait complete. 2023-10-16T12:11:47.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.869+0000 7f4d6509c700 1 Processor -- start 2023-10-16T12:11:47.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.869+0000 7f4d6509c700 1 -- start start 2023-10-16T12:11:47.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.870+0000 7f4d6509c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4d6006a7e0 0x7f4d600af420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:47.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.870+0000 7f4d6509c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d600cc8a0 0x7f4d600c3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:47.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.870+0000 7f4d6509c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4d6015b280 0x7f4d60151c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:47.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.870+0000 7f4d6509c700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d600c4ff0 con 0x7f4d600cc8a0 2023-10-16T12:11:47.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.870+0000 7f4d6509c700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d600aef20 con 0x7f4d6006a7e0 2023-10-16T12:11:47.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.870+0000 7f4d6509c700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d600c53a0 con 0x7f4d6015b280 2023-10-16T12:11:47.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.870+0000 7f4d5e59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d600cc8a0 0x7f4d600c3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:47.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.870+0000 7f4d5e59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d600cc8a0 0x7f4d600c3f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35186/0 (socket says 172.21.15.16:35186) 2023-10-16T12:11:47.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.870+0000 7f4d5e59c700 1 -- 172.21.15.16:0/2571229303 learned_addr learned my addr 172.21.15.16:0/2571229303 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:47.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.870+0000 7f4d5f59e700 1 --2- 172.21.15.16:0/2571229303 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4d6015b280 0x7f4d60151c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:47.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.870+0000 7f4d5ed9d700 1 --2- 172.21.15.16:0/2571229303 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4d6006a7e0 0x7f4d600af420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:47.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.870+0000 7f4d5e59c700 1 -- 172.21.15.16:0/2571229303 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4d6015b280 msgr2=0x7f4d60151c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:47.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.870+0000 7f4d5e59c700 1 --2- 172.21.15.16:0/2571229303 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4d6015b280 0x7f4d60151c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:47.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.870+0000 7f4d5e59c700 1 -- 172.21.15.16:0/2571229303 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4d6006a7e0 msgr2=0x7f4d600af420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:47.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.870+0000 7f4d5e59c700 1 --2- 172.21.15.16:0/2571229303 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4d6006a7e0 0x7f4d600af420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:47.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.871+0000 7f4d5e59c700 1 -- 172.21.15.16:0/2571229303 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d48009550 con 0x7f4d600cc8a0 2023-10-16T12:11:47.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.871+0000 7f4d5e59c700 1 --2- 172.21.15.16:0/2571229303 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d600cc8a0 0x7f4d600c3f70 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f4d48009130 tx=0x7f4d48004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:47.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.871+0000 7f4d577fe700 1 -- 172.21.15.16:0/2571229303 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d48003b90 con 0x7f4d600cc8a0 2023-10-16T12:11:47.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.871+0000 7f4d577fe700 1 -- 172.21.15.16:0/2571229303 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d4801f070 con 0x7f4d600cc8a0 2023-10-16T12:11:47.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.871+0000 7f4d6509c700 1 -- 172.21.15.16:0/2571229303 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d600c44b0 con 0x7f4d600cc8a0 2023-10-16T12:11:47.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.871+0000 7f4d577fe700 1 -- 172.21.15.16:0/2571229303 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d48011630 con 0x7f4d600cc8a0 2023-10-16T12:11:47.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.871+0000 7f4d6509c700 1 -- 172.21.15.16:0/2571229303 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d600c4a30 con 0x7f4d600cc8a0 2023-10-16T12:11:47.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.872+0000 7f4d6509c700 1 -- 172.21.15.16:0/2571229303 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d600c5530 con 0x7f4d600cc8a0 2023-10-16T12:11:47.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.873+0000 7f4d577fe700 1 -- 172.21.15.16:0/2571229303 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f4d48021030 con 0x7f4d600cc8a0 2023-10-16T12:11:47.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.873+0000 7f4d577fe700 1 --2- 172.21.15.16:0/2571229303 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f4d4006af00 0x7f4d4006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:47.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.873+0000 7f4d577fe700 1 -- 172.21.15.16:0/2571229303 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f4d4808ae30 con 0x7f4d600cc8a0 2023-10-16T12:11:47.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.873+0000 7f4d5ed9d700 1 --2- 172.21.15.16:0/2571229303 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f4d4006af00 0x7f4d4006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:47.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.873+0000 7f4d5ed9d700 1 --2- 172.21.15.16:0/2571229303 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f4d4006af00 0x7f4d4006d3c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4d6006b090 tx=0x7f4d500090e0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:47.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:47.877+0000 7f4d577fe700 1 -- 172.21.15.16:0/2571229303 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4d48055b30 con 0x7f4d600cc8a0 2023-10-16T12:11:48.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.127+0000 7f4d6509c700 1 -- 172.21.15.16:0/2571229303 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4d600c5170 con 0x7f4d600cc8a0 2023-10-16T12:11:48.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.127+0000 7f4d577fe700 1 -- 172.21.15.16:0/2571229303 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4909 (secure 0 0 0) 0x7f4d48059150 con 0x7f4d600cc8a0 2023-10-16T12:11:48.131 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:11:48.131 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/2979266800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":2979266800},{"type":"v1","addr":"172.21.15.49:6834","nonce":2979266800}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4251,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/1398336801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1398336801},{"type":"v1","addr":"172.21.15.16:6836","nonce":1398336801}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4254,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3168304671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":3168304671},{"type":"v1","addr":"172.21.15.16:6837","nonce":3168304671}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:10:39.030243+0000","modified":"2023-10-16T12:11:02.593358+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4269":{"gid":4269,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.49:6835/2892536404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":2892536404},{"type":"v1","addr":"172.21.15.49:6835","nonce":2892536404}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T12:11:48.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.129+0000 7f4d6509c700 1 -- 172.21.15.16:0/2571229303 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f4d4006af00 msgr2=0x7f4d4006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:48.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.129+0000 7f4d6509c700 1 --2- 172.21.15.16:0/2571229303 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f4d4006af00 0x7f4d4006d3c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4d6006b090 tx=0x7f4d500090e0 comp rx=0 tx=0).stop 2023-10-16T12:11:48.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.129+0000 7f4d6509c700 1 -- 172.21.15.16:0/2571229303 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d600cc8a0 msgr2=0x7f4d600c3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:48.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.129+0000 7f4d6509c700 1 --2- 172.21.15.16:0/2571229303 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d600cc8a0 0x7f4d600c3f70 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f4d48009130 tx=0x7f4d48004750 comp rx=0 tx=0).stop 2023-10-16T12:11:48.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.129+0000 7f4d6509c700 1 -- 172.21.15.16:0/2571229303 shutdown_connections 2023-10-16T12:11:48.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.129+0000 7f4d6509c700 1 --2- 172.21.15.16:0/2571229303 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f4d4006af00 0x7f4d4006d3c0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:48.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.129+0000 7f4d6509c700 1 --2- 172.21.15.16:0/2571229303 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4d6015b280 0x7f4d60151c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:48.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.129+0000 7f4d6509c700 1 --2- 172.21.15.16:0/2571229303 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4d6006a7e0 0x7f4d600af420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:48.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.129+0000 7f4d6509c700 1 --2- 172.21.15.16:0/2571229303 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d600cc8a0 0x7f4d600c3f70 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:48.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.129+0000 7f4d6509c700 1 -- 172.21.15.16:0/2571229303 >> 172.21.15.16:0/2571229303 conn(0x7f4d600b1fd0 msgr2=0x7f4d6015eeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:48.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.129+0000 7f4d6509c700 1 -- 172.21.15.16:0/2571229303 shutdown_connections 2023-10-16T12:11:48.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.129+0000 7f4d6509c700 1 -- 172.21.15.16:0/2571229303 wait complete. 2023-10-16T12:11:48.135 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 9 2023-10-16T12:11:48.144 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:11:48.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.267+0000 7fd17ac65700 1 Processor -- start 2023-10-16T12:11:48.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.267+0000 7fd17ac65700 1 -- start start 2023-10-16T12:11:48.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.268+0000 7fd17ac65700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd1740d03b0 0x7fd1740cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:48.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.268+0000 7fd17ac65700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1740cc2a0 con 0x7fd1740ce4f0 2023-10-16T12:11:48.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.268+0000 7fd17ac65700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1740cc540 con 0x7fd17415b630 2023-10-16T12:11:48.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.268+0000 7fd17ac65700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1740cdb20 con 0x7fd1740d03b0 2023-10-16T12:11:48.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.268+0000 7fd173fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd1740ce4f0 0x7fd1740cfca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:52698/0 (socket says 172.21.15.16:52698) 2023-10-16T12:11:48.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.268+0000 7fd173fff700 1 -- 172.21.15.16:0/3488295284 learned_addr learned my addr 172.21.15.16:0/3488295284 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:48.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.268+0000 7fd179202700 1 --2- 172.21.15.16:0/3488295284 >> v2:172.21.15.16:3300/0 conn(0x7fd1740d03b0 0x7fd1740cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:48.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.268+0000 7fd1737fe700 1 -- 172.21.15.16:0/3488295284 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2198806593 0 0) 0x7fd1740cc2a0 con 0x7fd1740ce4f0 2023-10-16T12:11:48.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.268+0000 7fd1737fe700 1 -- 172.21.15.16:0/3488295284 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1640035b0 con 0x7fd1740ce4f0 2023-10-16T12:11:48.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.268+0000 7fd179202700 1 -- 172.21.15.16:0/3488295284 >> v1:172.21.15.49:6789/0 conn(0x7fd17415b630 legacy=0x7fd1740cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:48.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.268+0000 7fd179202700 1 -- 172.21.15.16:0/3488295284 >> v1:172.21.15.16:6789/0 conn(0x7fd1740ce4f0 legacy=0x7fd1740cfca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:48.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.268+0000 7fd179202700 1 -- 172.21.15.16:0/3488295284 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1740cc820 con 0x7fd1740d03b0 2023-10-16T12:11:48.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.269+0000 7fd179202700 1 --2- 172.21.15.16:0/3488295284 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1740d03b0 0x7fd1740cd5e0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fd16c009d90 tx=0x7fd16c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=836107b320ff250c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:48.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.269+0000 7fd1737fe700 1 -- 172.21.15.16:0/3488295284 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd16c00e070 con 0x7fd1740d03b0 2023-10-16T12:11:48.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.269+0000 7fd1737fe700 1 -- 172.21.15.16:0/3488295284 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd16c00bb20 con 0x7fd1740d03b0 2023-10-16T12:11:48.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.269+0000 7fd1737fe700 1 -- 172.21.15.16:0/3488295284 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd16c00be90 con 0x7fd1740d03b0 2023-10-16T12:11:48.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.269+0000 7fd17ac65700 1 -- 172.21.15.16:0/3488295284 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1740d03b0 msgr2=0x7fd1740cd5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:48.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.269+0000 7fd17ac65700 1 --2- 172.21.15.16:0/3488295284 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1740d03b0 0x7fd1740cd5e0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fd16c009d90 tx=0x7fd16c00b330 comp rx=0 tx=0).stop 2023-10-16T12:11:48.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.269+0000 7fd17ac65700 1 -- 172.21.15.16:0/3488295284 shutdown_connections 2023-10-16T12:11:48.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.269+0000 7fd17ac65700 1 --2- 172.21.15.16:0/3488295284 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1740d03b0 0x7fd1740cd5e0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:48.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.269+0000 7fd17ac65700 1 -- 172.21.15.16:0/3488295284 >> 172.21.15.16:0/3488295284 conn(0x7fd1740d14b0 msgr2=0x7fd1740d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:48.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.269+0000 7fd17ac65700 1 -- 172.21.15.16:0/3488295284 shutdown_connections 2023-10-16T12:11:48.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.269+0000 7fd17ac65700 1 -- 172.21.15.16:0/3488295284 wait complete. 2023-10-16T12:11:48.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.271+0000 7fd17ac65700 1 Processor -- start 2023-10-16T12:11:48.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.271+0000 7fd17ac65700 1 -- start start 2023-10-16T12:11:48.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.272+0000 7fd17ac65700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd1740ce4f0 0x7fd174154b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:48.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.272+0000 7fd17ac65700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd1740d03b0 0x7fd1740b4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:48.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.272+0000 7fd17ac65700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd17415b630 0x7fd1740b4840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:48.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.272+0000 7fd17ac65700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1741f6900 con 0x7fd17415b630 2023-10-16T12:11:48.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.272+0000 7fd17ac65700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1741f6be0 con 0x7fd1740d03b0 2023-10-16T12:11:48.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.272+0000 7fd17ac65700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1741f6ec0 con 0x7fd1740ce4f0 2023-10-16T12:11:48.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.272+0000 7fd179202700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd17415b630 0x7fd1740b4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:48.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.272+0000 7fd179202700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd17415b630 0x7fd1740b4840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35202/0 (socket says 172.21.15.16:35202) 2023-10-16T12:11:48.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.272+0000 7fd179202700 1 -- 172.21.15.16:0/4103779123 learned_addr learned my addr 172.21.15.16:0/4103779123 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:48.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.272+0000 7fd173fff700 1 --2- 172.21.15.16:0/4103779123 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd1740d03b0 0x7fd1740b4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:48.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.272+0000 7fd178a01700 1 --2- 172.21.15.16:0/4103779123 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd1740ce4f0 0x7fd174154b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:48.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.272+0000 7fd179202700 1 -- 172.21.15.16:0/4103779123 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd1740ce4f0 msgr2=0x7fd174154b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:48.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.272+0000 7fd179202700 1 --2- 172.21.15.16:0/4103779123 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd1740ce4f0 0x7fd174154b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:48.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.272+0000 7fd179202700 1 -- 172.21.15.16:0/4103779123 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd1740d03b0 msgr2=0x7fd1740b4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:48.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.272+0000 7fd179202700 1 --2- 172.21.15.16:0/4103779123 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd1740d03b0 0x7fd1740b4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:48.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.272+0000 7fd179202700 1 -- 172.21.15.16:0/4103779123 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd16c009a70 con 0x7fd17415b630 2023-10-16T12:11:48.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.272+0000 7fd173fff700 1 --2- 172.21.15.16:0/4103779123 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd1740d03b0 0x7fd1740b4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:11:48.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.272+0000 7fd179202700 1 --2- 172.21.15.16:0/4103779123 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd17415b630 0x7fd1740b4840 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fd16c00b370 tx=0x7fd16c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:48.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.272+0000 7fd1717fa700 1 -- 172.21.15.16:0/4103779123 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd16c004960 con 0x7fd17415b630 2023-10-16T12:11:48.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.272+0000 7fd1717fa700 1 -- 172.21.15.16:0/4103779123 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd16c020070 con 0x7fd17415b630 2023-10-16T12:11:48.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.272+0000 7fd17ac65700 1 -- 172.21.15.16:0/4103779123 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1741f71a0 con 0x7fd17415b630 2023-10-16T12:11:48.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.272+0000 7fd1717fa700 1 -- 172.21.15.16:0/4103779123 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd16c0276c0 con 0x7fd17415b630 2023-10-16T12:11:48.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.272+0000 7fd17ac65700 1 -- 172.21.15.16:0/4103779123 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd174155050 con 0x7fd17415b630 2023-10-16T12:11:48.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.273+0000 7fd17ac65700 1 -- 172.21.15.16:0/4103779123 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1741f7660 con 0x7fd17415b630 2023-10-16T12:11:48.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.274+0000 7fd1717fa700 1 -- 172.21.15.16:0/4103779123 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fd16c012430 con 0x7fd17415b630 2023-10-16T12:11:48.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.274+0000 7fd1717fa700 1 --2- 172.21.15.16:0/4103779123 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fd15806af00 0x7fd15806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:48.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.274+0000 7fd1717fa700 1 -- 172.21.15.16:0/4103779123 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fd16c022070 con 0x7fd17415b630 2023-10-16T12:11:48.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.274+0000 7fd178a01700 1 --2- 172.21.15.16:0/4103779123 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fd15806af00 0x7fd15806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:48.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.275+0000 7fd178a01700 1 --2- 172.21.15.16:0/4103779123 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fd15806af00 0x7fd15806d3c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fd168001670 tx=0x7fd168009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:48.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.279+0000 7fd1717fa700 1 -- 172.21.15.16:0/4103779123 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd16c0568b0 con 0x7fd17415b630 2023-10-16T12:11:48.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.483+0000 7fd17ac65700 1 -- 172.21.15.16:0/4103779123 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd17404ebb0 con 0x7fd17415b630 2023-10-16T12:11:48.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.485+0000 7fd1717fa700 1 -- 172.21.15.16:0/4103779123 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18629 (secure 0 0 0) 0x7fd16c059ed0 con 0x7fd17415b630 2023-10-16T12:11:48.488 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:11:48.488 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:11:28.241865+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:10:36.935773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T12:10:38.004870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-16T12:10:41.672025+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"3.2b","mappings":[{"from":0,"to":2}]},{"pgid":"3.38","mappings":[{"from":3,"to":1}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]},{"pgid":"4.1b","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:11:48.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.487+0000 7fd17ac65700 1 -- 172.21.15.16:0/4103779123 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fd15806af00 msgr2=0x7fd15806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:48.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.487+0000 7fd17ac65700 1 --2- 172.21.15.16:0/4103779123 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fd15806af00 0x7fd15806d3c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fd168001670 tx=0x7fd168009040 comp rx=0 tx=0).stop 2023-10-16T12:11:48.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.487+0000 7fd17ac65700 1 -- 172.21.15.16:0/4103779123 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd17415b630 msgr2=0x7fd1740b4840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:48.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.487+0000 7fd17ac65700 1 --2- 172.21.15.16:0/4103779123 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd17415b630 0x7fd1740b4840 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fd16c00b370 tx=0x7fd16c004ad0 comp rx=0 tx=0).stop 2023-10-16T12:11:48.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.487+0000 7fd17ac65700 1 -- 172.21.15.16:0/4103779123 shutdown_connections 2023-10-16T12:11:48.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.487+0000 7fd17ac65700 1 --2- 172.21.15.16:0/4103779123 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fd15806af00 0x7fd15806d3c0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:48.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.487+0000 7fd17ac65700 1 --2- 172.21.15.16:0/4103779123 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd17415b630 0x7fd1740b4840 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:48.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.487+0000 7fd17ac65700 1 --2- 172.21.15.16:0/4103779123 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd1740ce4f0 0x7fd174154b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:48.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.487+0000 7fd17ac65700 1 --2- 172.21.15.16:0/4103779123 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd1740d03b0 0x7fd1740b4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:48.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.487+0000 7fd17ac65700 1 -- 172.21.15.16:0/4103779123 >> 172.21.15.16:0/4103779123 conn(0x7fd1740d14b0 msgr2=0x7fd17415da20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:48.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.487+0000 7fd17ac65700 1 -- 172.21.15.16:0/4103779123 shutdown_connections 2023-10-16T12:11:48.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.487+0000 7fd17ac65700 1 -- 172.21.15.16:0/4103779123 wait complete. 2023-10-16T12:11:48.502 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-16T12:11:48.545 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:11:48.548 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:11:48.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.627+0000 7ff522c12700 1 Processor -- start 2023-10-16T12:11:48.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.627+0000 7ff522c12700 1 -- start start 2023-10-16T12:11:48.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.628+0000 7ff522c12700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff51c0cc8a0 0x7ff51c15e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:48.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.628+0000 7ff522c12700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7ff51c06a7e0 0x7ff51c06ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:48.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.628+0000 7ff522c12700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff51c06b150 con 0x7ff51c15b280 2023-10-16T12:11:48.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.628+0000 7ff522c12700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff51c161c30 con 0x7ff51c0cc8a0 2023-10-16T12:11:48.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.628+0000 7ff522c12700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff51c161f10 con 0x7ff51c06a7e0 2023-10-16T12:11:48.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.628+0000 7ff51bfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff51c0cc8a0 0x7ff51c15e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:48.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.628+0000 7ff51bfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff51c0cc8a0 0x7ff51c15e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35208/0 (socket says 172.21.15.16:35208) 2023-10-16T12:11:48.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.628+0000 7ff51bfff700 1 -- 172.21.15.16:0/3009164206 learned_addr learned my addr 172.21.15.16:0/3009164206 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:48.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.628+0000 7ff5211af700 1 --2- 172.21.15.16:0/3009164206 >> v2:172.21.15.49:3300/0 conn(0x7ff51c06a7e0 0x7ff51c06ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:48.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.628+0000 7ff51bfff700 1 -- 172.21.15.16:0/3009164206 >> v2:172.21.15.49:3300/0 conn(0x7ff51c06a7e0 msgr2=0x7ff51c06ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:48.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.628+0000 7ff51bfff700 1 --2- 172.21.15.16:0/3009164206 >> v2:172.21.15.49:3300/0 conn(0x7ff51c06a7e0 0x7ff51c06ac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:48.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.628+0000 7ff51bfff700 1 -- 172.21.15.16:0/3009164206 >> v1:172.21.15.49:6789/0 conn(0x7ff51c15b280 legacy=0x7ff51c1613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:48.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.629+0000 7ff51bfff700 1 -- 172.21.15.16:0/3009164206 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff51c1621f0 con 0x7ff51c0cc8a0 2023-10-16T12:11:48.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.629+0000 7ff51bfff700 1 --2- 172.21.15.16:0/3009164206 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff51c0cc8a0 0x7ff51c15e700 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7ff51c0cbed0 tx=0x7ff508009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49dabf704932f8ba server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:48.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.629+0000 7ff51b7fe700 1 -- 172.21.15.16:0/3009164206 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff508004230 con 0x7ff51c0cc8a0 2023-10-16T12:11:48.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.629+0000 7ff51b7fe700 1 -- 172.21.15.16:0/3009164206 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff508004390 con 0x7ff51c0cc8a0 2023-10-16T12:11:48.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.629+0000 7ff51b7fe700 1 -- 172.21.15.16:0/3009164206 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5080036a0 con 0x7ff51c0cc8a0 2023-10-16T12:11:48.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.629+0000 7ff522c12700 1 -- 172.21.15.16:0/3009164206 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff51c0cc8a0 msgr2=0x7ff51c15e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:48.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.629+0000 7ff522c12700 1 --2- 172.21.15.16:0/3009164206 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff51c0cc8a0 0x7ff51c15e700 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7ff51c0cbed0 tx=0x7ff508009bd0 comp rx=0 tx=0).stop 2023-10-16T12:11:48.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.629+0000 7ff522c12700 1 -- 172.21.15.16:0/3009164206 shutdown_connections 2023-10-16T12:11:48.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.629+0000 7ff522c12700 1 --2- 172.21.15.16:0/3009164206 >> v2:172.21.15.49:3300/0 conn(0x7ff51c06a7e0 0x7ff51c06ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:48.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.629+0000 7ff522c12700 1 --2- 172.21.15.16:0/3009164206 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff51c0cc8a0 0x7ff51c15e700 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:48.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.629+0000 7ff522c12700 1 -- 172.21.15.16:0/3009164206 >> 172.21.15.16:0/3009164206 conn(0x7ff51c0b1fd0 msgr2=0x7ff51c0b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:48.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.629+0000 7ff522c12700 1 -- 172.21.15.16:0/3009164206 shutdown_connections 2023-10-16T12:11:48.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.630+0000 7ff522c12700 1 -- 172.21.15.16:0/3009164206 wait complete. 2023-10-16T12:11:48.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.631+0000 7ff522c12700 1 Processor -- start 2023-10-16T12:11:48.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.631+0000 7ff522c12700 1 -- start start 2023-10-16T12:11:48.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.632+0000 7ff522c12700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff51c06a7e0 0x7ff51c1f3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:48.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.632+0000 7ff522c12700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff51c0cc8a0 0x7ff51c071bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:48.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.632+0000 7ff522c12700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff51c15b280 0x7ff51c072130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:48.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.632+0000 7ff522c12700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff51c0af680 con 0x7ff51c06a7e0 2023-10-16T12:11:48.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.632+0000 7ff522c12700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff51c0aea00 con 0x7ff51c0cc8a0 2023-10-16T12:11:48.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.632+0000 7ff522c12700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff51c0aece0 con 0x7ff51c15b280 2023-10-16T12:11:48.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.632+0000 7ff5209ae700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff51c06a7e0 0x7ff51c1f3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:48.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.632+0000 7ff5209ae700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff51c06a7e0 0x7ff51c1f3970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35212/0 (socket says 172.21.15.16:35212) 2023-10-16T12:11:48.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.632+0000 7ff5209ae700 1 -- 172.21.15.16:0/3938306409 learned_addr learned my addr 172.21.15.16:0/3938306409 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:48.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.632+0000 7ff51bfff700 1 --2- 172.21.15.16:0/3938306409 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff51c0cc8a0 0x7ff51c071bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:48.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.632+0000 7ff5211af700 1 --2- 172.21.15.16:0/3938306409 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff51c15b280 0x7ff51c072130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:48.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.632+0000 7ff5209ae700 1 -- 172.21.15.16:0/3938306409 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff51c15b280 msgr2=0x7ff51c072130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:48.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.632+0000 7ff5209ae700 1 --2- 172.21.15.16:0/3938306409 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff51c15b280 0x7ff51c072130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:48.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.632+0000 7ff5209ae700 1 -- 172.21.15.16:0/3938306409 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff51c0cc8a0 msgr2=0x7ff51c071bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:48.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.632+0000 7ff5209ae700 1 --2- 172.21.15.16:0/3938306409 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff51c0cc8a0 0x7ff51c071bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:48.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.632+0000 7ff5209ae700 1 -- 172.21.15.16:0/3938306409 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff508009550 con 0x7ff51c06a7e0 2023-10-16T12:11:48.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.632+0000 7ff5211af700 1 --2- 172.21.15.16:0/3938306409 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff51c15b280 0x7ff51c072130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:11:48.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.632+0000 7ff5209ae700 1 --2- 172.21.15.16:0/3938306409 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff51c06a7e0 0x7ff51c1f3970 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7ff51000b4a0 tx=0x7ff51000b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:48.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.632+0000 7ff5197fa700 1 -- 172.21.15.16:0/3938306409 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff510011830 con 0x7ff51c06a7e0 2023-10-16T12:11:48.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.632+0000 7ff5197fa700 1 -- 172.21.15.16:0/3938306409 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff510011990 con 0x7ff51c06a7e0 2023-10-16T12:11:48.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.632+0000 7ff522c12700 1 -- 172.21.15.16:0/3938306409 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff51c0aefc0 con 0x7ff51c06a7e0 2023-10-16T12:11:48.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.632+0000 7ff5197fa700 1 -- 172.21.15.16:0/3938306409 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff510011d40 con 0x7ff51c06a7e0 2023-10-16T12:11:48.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.633+0000 7ff522c12700 1 -- 172.21.15.16:0/3938306409 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff51c0adc50 con 0x7ff51c06a7e0 2023-10-16T12:11:48.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.633+0000 7ff522c12700 1 -- 172.21.15.16:0/3938306409 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff51c04ebb0 con 0x7ff51c06a7e0 2023-10-16T12:11:48.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.634+0000 7ff5197fa700 1 -- 172.21.15.16:0/3938306409 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7ff51001a400 con 0x7ff51c06a7e0 2023-10-16T12:11:48.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.634+0000 7ff5197fa700 1 --2- 172.21.15.16:0/3938306409 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7ff50006af00 0x7ff50006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:48.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.635+0000 7ff5197fa700 1 -- 172.21.15.16:0/3938306409 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7ff510089650 con 0x7ff51c06a7e0 2023-10-16T12:11:48.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.635+0000 7ff51bfff700 1 --2- 172.21.15.16:0/3938306409 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7ff50006af00 0x7ff50006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:48.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.635+0000 7ff51bfff700 1 --2- 172.21.15.16:0/3938306409 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7ff50006af00 0x7ff50006d3c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7ff51c0cbed0 tx=0x7ff508008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:48.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.639+0000 7ff5197fa700 1 -- 172.21.15.16:0/3938306409 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff5100543d0 con 0x7ff51c06a7e0 2023-10-16T12:11:48.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.890+0000 7ff522c12700 1 -- 172.21.15.16:0/3938306409 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7ff51c0b2600 con 0x7ff51c06a7e0 2023-10-16T12:11:48.893 INFO:tasks.ceph.mon.a.smithi016.stderr:2023-10-16T12:11:48.890+0000 7f9fc8535700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-16T12:11:48.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.892+0000 7ff5197fa700 1 -- 172.21.15.16:0/3938306409 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v10) v1 ==== 161+0+0 (secure 0 0 0) 0x7ff5100579f0 con 0x7ff51c06a7e0 2023-10-16T12:11:48.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.895+0000 7ff522c12700 1 -- 172.21.15.16:0/3938306409 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7ff50006af00 msgr2=0x7ff50006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:48.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.895+0000 7ff522c12700 1 --2- 172.21.15.16:0/3938306409 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7ff50006af00 0x7ff50006d3c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7ff51c0cbed0 tx=0x7ff508008030 comp rx=0 tx=0).stop 2023-10-16T12:11:48.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.895+0000 7ff522c12700 1 -- 172.21.15.16:0/3938306409 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff51c06a7e0 msgr2=0x7ff51c1f3970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:48.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.895+0000 7ff522c12700 1 --2- 172.21.15.16:0/3938306409 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff51c06a7e0 0x7ff51c1f3970 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7ff51000b4a0 tx=0x7ff51000b860 comp rx=0 tx=0).stop 2023-10-16T12:11:48.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.895+0000 7ff522c12700 1 -- 172.21.15.16:0/3938306409 shutdown_connections 2023-10-16T12:11:48.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.895+0000 7ff522c12700 1 --2- 172.21.15.16:0/3938306409 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7ff50006af00 0x7ff50006d3c0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:48.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.895+0000 7ff522c12700 1 --2- 172.21.15.16:0/3938306409 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff51c15b280 0x7ff51c072130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:48.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.895+0000 7ff522c12700 1 --2- 172.21.15.16:0/3938306409 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff51c06a7e0 0x7ff51c1f3970 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:48.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.895+0000 7ff522c12700 1 --2- 172.21.15.16:0/3938306409 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff51c0cc8a0 0x7ff51c071bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:48.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.895+0000 7ff522c12700 1 -- 172.21.15.16:0/3938306409 >> 172.21.15.16:0/3938306409 conn(0x7ff51c0b1fd0 msgr2=0x7ff51c0cb770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:48.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.895+0000 7ff522c12700 1 -- 172.21.15.16:0/3938306409 shutdown_connections 2023-10-16T12:11:48.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:48.895+0000 7ff522c12700 1 -- 172.21.15.16:0/3938306409 wait complete. 2023-10-16T12:11:48.901 INFO:teuthology.orchestra.run.smithi016.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-16T12:11:48.902 INFO:tasks.ceph.mds.d.smithi049.stderr: -5932> 2023-10-16T12:10:43.387+0000 7fd982013700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T12:11:48.911 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-16T12:11:48.935 INFO:tasks.ceph.mds.d.smithi049.stdout:starting mds.d at 2023-10-16T12:11:49.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.074+0000 7fcc28362700 1 Processor -- start 2023-10-16T12:11:49.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.074+0000 7fcc28362700 1 -- start start 2023-10-16T12:11:49.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.075+0000 7fcc28362700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fcc2015b2c0 0x7fcc20161340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:49.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.075+0000 7fcc28362700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc2015ecb0 con 0x7fcc2006a7e0 2023-10-16T12:11:49.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.075+0000 7fcc28362700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc20161880 con 0x7fcc2006e570 2023-10-16T12:11:49.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.075+0000 7fcc28362700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc2015b6a0 con 0x7fcc2015b2c0 2023-10-16T12:11:49.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.075+0000 7fcc268ff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fcc2006a7e0 0x7fcc20071a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:52730/0 (socket says 172.21.15.16:52730) 2023-10-16T12:11:49.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.075+0000 7fcc268ff700 1 -- 172.21.15.16:0/4198851403 learned_addr learned my addr 172.21.15.16:0/4198851403 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:49.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.075+0000 7fcc260fe700 1 --2- 172.21.15.16:0/4198851403 >> v2:172.21.15.49:3300/0 conn(0x7fcc2015b2c0 0x7fcc20161340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:49.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.076+0000 7fcc250fc700 1 -- 172.21.15.16:0/4198851403 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 958360268 0 0) 0x7fcc2015ecb0 con 0x7fcc2006a7e0 2023-10-16T12:11:49.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.076+0000 7fcc250fc700 1 -- 172.21.15.16:0/4198851403 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc0c0035b0 con 0x7fcc2006a7e0 2023-10-16T12:11:49.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.076+0000 7fcc250fc700 1 -- 172.21.15.16:0/4198851403 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 704951283 0 0) 0x7fcc0c0035b0 con 0x7fcc2006a7e0 2023-10-16T12:11:49.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.076+0000 7fcc250fc700 1 -- 172.21.15.16:0/4198851403 >> v2:172.21.15.49:3300/0 conn(0x7fcc2015b2c0 msgr2=0x7fcc20161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:49.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.076+0000 7fcc250fc700 1 --2- 172.21.15.16:0/4198851403 >> v2:172.21.15.49:3300/0 conn(0x7fcc2015b2c0 0x7fcc20161340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:49.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.076+0000 7fcc250fc700 1 -- 172.21.15.16:0/4198851403 >> v1:172.21.15.49:6789/0 conn(0x7fcc2006e570 legacy=0x7fcc2015e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:49.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.076+0000 7fcc250fc700 1 -- 172.21.15.16:0/4198851403 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc20161b20 con 0x7fcc2006a7e0 2023-10-16T12:11:49.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.076+0000 7fcc250fc700 1 -- 172.21.15.16:0/4198851403 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7fcc180019f0 con 0x7fcc2006a7e0 2023-10-16T12:11:49.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.076+0000 7fcc250fc700 1 -- 172.21.15.16:0/4198851403 >> v1:172.21.15.16:6789/0 conn(0x7fcc2006a7e0 legacy=0x7fcc20071a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:49.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.076+0000 7fcc250fc700 1 --2- 172.21.15.16:0/4198851403 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcc0c003a00 0x7fcc0c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:49.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.076+0000 7fcc250fc700 1 --2- 172.21.15.16:0/4198851403 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcc0c004850 0x7fcc0c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:49.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.076+0000 7fcc250fc700 1 --2- 172.21.15.16:0/4198851403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc0c005210 0x7fcc0c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:49.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.076+0000 7fcc250fc700 1 -- 172.21.15.16:0/4198851403 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc2015b6a0 con 0x7fcc0c005210 2023-10-16T12:11:49.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.076+0000 7fcc250fc700 1 -- 172.21.15.16:0/4198851403 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc0c00c1b0 con 0x7fcc0c004850 2023-10-16T12:11:49.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.076+0000 7fcc250fc700 1 -- 172.21.15.16:0/4198851403 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc0c00c370 con 0x7fcc0c003a00 2023-10-16T12:11:49.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.076+0000 7fcc260fe700 1 --2- 172.21.15.16:0/4198851403 >> v2:172.21.15.49:3300/0 conn(0x7fcc2015b2c0 0x7fcc20161340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:11:49.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.076+0000 7fcc268ff700 1 --2- 172.21.15.16:0/4198851403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc0c005210 0x7fcc0c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:49.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.076+0000 7fcc258fd700 1 --2- 172.21.15.16:0/4198851403 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcc0c004850 0x7fcc0c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:49.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.076+0000 7fcc260fe700 1 --2- 172.21.15.16:0/4198851403 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcc0c003a00 0x7fcc0c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:49.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.076+0000 7fcc268ff700 1 -- 172.21.15.16:0/4198851403 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcc0c003a00 msgr2=0x7fcc0c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:49.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.076+0000 7fcc268ff700 1 --2- 172.21.15.16:0/4198851403 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcc0c003a00 0x7fcc0c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:49.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.076+0000 7fcc268ff700 1 -- 172.21.15.16:0/4198851403 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcc0c004850 msgr2=0x7fcc0c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:49.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.076+0000 7fcc268ff700 1 --2- 172.21.15.16:0/4198851403 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcc0c004850 0x7fcc0c004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:49.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.076+0000 7fcc268ff700 1 -- 172.21.15.16:0/4198851403 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc0c0035b0 con 0x7fcc0c005210 2023-10-16T12:11:49.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.077+0000 7fcc268ff700 1 -- 172.21.15.16:0/4198851403 reap_dead start 2023-10-16T12:11:49.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.077+0000 7fcc268ff700 1 --2- 172.21.15.16:0/4198851403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc0c005210 0x7fcc0c00ba10 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fcc1800a830 tx=0x7fcc0c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=152d7c7ce5da4a82 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:49.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.077+0000 7fcc250fc700 1 -- 172.21.15.16:0/4198851403 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc1800a2f0 con 0x7fcc0c005210 2023-10-16T12:11:49.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.077+0000 7fcc250fc700 1 -- 172.21.15.16:0/4198851403 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc18012620 con 0x7fcc0c005210 2023-10-16T12:11:49.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.077+0000 7fcc250fc700 1 -- 172.21.15.16:0/4198851403 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc180129d0 con 0x7fcc0c005210 2023-10-16T12:11:49.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.077+0000 7fcc28362700 1 -- 172.21.15.16:0/4198851403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc0c005210 msgr2=0x7fcc0c00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:49.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.077+0000 7fcc28362700 1 --2- 172.21.15.16:0/4198851403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc0c005210 0x7fcc0c00ba10 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fcc1800a830 tx=0x7fcc0c005880 comp rx=0 tx=0).stop 2023-10-16T12:11:49.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.077+0000 7fcc28362700 1 -- 172.21.15.16:0/4198851403 shutdown_connections 2023-10-16T12:11:49.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.077+0000 7fcc28362700 1 --2- 172.21.15.16:0/4198851403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc0c005210 0x7fcc0c00ba10 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:49.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.077+0000 7fcc28362700 1 -- 172.21.15.16:0/4198851403 >> 172.21.15.16:0/4198851403 conn(0x7fcc200b1f40 msgr2=0x7fcc200b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:49.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.077+0000 7fcc28362700 1 -- 172.21.15.16:0/4198851403 shutdown_connections 2023-10-16T12:11:49.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.077+0000 7fcc28362700 1 -- 172.21.15.16:0/4198851403 wait complete. 2023-10-16T12:11:49.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.079+0000 7fcc28362700 1 Processor -- start 2023-10-16T12:11:49.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.079+0000 7fcc28362700 1 -- start start 2023-10-16T12:11:49.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.079+0000 7fcc28362700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcc0c005210 0x7fcc201f2b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:49.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.079+0000 7fcc28362700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc20159df0 0x7fcc20158240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:49.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.079+0000 7fcc28362700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcc2015caa0 0x7fcc20157240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:49.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.079+0000 7fcc28362700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc200c96e0 con 0x7fcc20159df0 2023-10-16T12:11:49.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.079+0000 7fcc28362700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc200c74c0 con 0x7fcc2015caa0 2023-10-16T12:11:49.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.079+0000 7fcc28362700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc201f3140 con 0x7fcc0c005210 2023-10-16T12:11:49.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.079+0000 7fcc258fd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc20159df0 0x7fcc20158240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:49.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.079+0000 7fcc258fd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc20159df0 0x7fcc20158240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35234/0 (socket says 172.21.15.16:35234) 2023-10-16T12:11:49.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.080+0000 7fcc258fd700 1 -- 172.21.15.16:0/360862577 learned_addr learned my addr 172.21.15.16:0/360862577 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:49.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.080+0000 7fcc260fe700 1 --2- 172.21.15.16:0/360862577 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcc0c005210 0x7fcc201f2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:49.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.080+0000 7fcc268ff700 1 --2- 172.21.15.16:0/360862577 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcc2015caa0 0x7fcc20157240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:49.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.080+0000 7fcc258fd700 1 -- 172.21.15.16:0/360862577 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcc0c005210 msgr2=0x7fcc201f2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:49.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.080+0000 7fcc258fd700 1 --2- 172.21.15.16:0/360862577 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcc0c005210 0x7fcc201f2b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:49.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.080+0000 7fcc258fd700 1 -- 172.21.15.16:0/360862577 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcc2015caa0 msgr2=0x7fcc20157240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:49.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.080+0000 7fcc258fd700 1 --2- 172.21.15.16:0/360862577 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcc2015caa0 0x7fcc20157240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:49.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.080+0000 7fcc258fd700 1 -- 172.21.15.16:0/360862577 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc201f3420 con 0x7fcc20159df0 2023-10-16T12:11:49.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.080+0000 7fcc268ff700 1 --2- 172.21.15.16:0/360862577 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcc2015caa0 0x7fcc20157240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:11:49.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.080+0000 7fcc258fd700 1 --2- 172.21.15.16:0/360862577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc20159df0 0x7fcc20158240 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fcc1c00f820 tx=0x7fcc1c00fbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:49.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.080+0000 7fcc16ffd700 1 -- 172.21.15.16:0/360862577 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc1c0052e0 con 0x7fcc20159df0 2023-10-16T12:11:49.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.080+0000 7fcc16ffd700 1 -- 172.21.15.16:0/360862577 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc1c005460 con 0x7fcc20159df0 2023-10-16T12:11:49.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.080+0000 7fcc28362700 1 -- 172.21.15.16:0/360862577 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc201f3700 con 0x7fcc20159df0 2023-10-16T12:11:49.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.080+0000 7fcc16ffd700 1 -- 172.21.15.16:0/360862577 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc1c0038c0 con 0x7fcc20159df0 2023-10-16T12:11:49.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.080+0000 7fcc28362700 1 -- 172.21.15.16:0/360862577 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc20157f70 con 0x7fcc20159df0 2023-10-16T12:11:49.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.081+0000 7fcc28362700 1 -- 172.21.15.16:0/360862577 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc2004ebb0 con 0x7fcc20159df0 2023-10-16T12:11:49.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.082+0000 7fcc16ffd700 1 -- 172.21.15.16:0/360862577 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fcc1c003a20 con 0x7fcc20159df0 2023-10-16T12:11:49.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.082+0000 7fcc16ffd700 1 --2- 172.21.15.16:0/360862577 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fcc0406af00 0x7fcc0406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:49.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.082+0000 7fcc16ffd700 1 -- 172.21.15.16:0/360862577 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6749+0+0 (secure 0 0 0) 0x7fcc1c0896d0 con 0x7fcc20159df0 2023-10-16T12:11:49.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.082+0000 7fcc260fe700 1 --2- 172.21.15.16:0/360862577 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fcc0406af00 0x7fcc0406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:49.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.082+0000 7fcc260fe700 1 --2- 172.21.15.16:0/360862577 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fcc0406af00 0x7fcc0406d3c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fcc10009cb0 tx=0x7fcc10008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:49.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.086+0000 7fcc16ffd700 1 -- 172.21.15.16:0/360862577 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcc1c054400 con 0x7fcc20159df0 2023-10-16T12:11:49.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.342+0000 7fcc28362700 1 -- 172.21.15.16:0/360862577 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fcc200b6370 con 0x7fcc20159df0 2023-10-16T12:11:49.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.896+0000 7fcc16ffd700 1 -- 172.21.15.16:0/360862577 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v11) v1 ==== 104+0+0 (secure 0 0 0) 0x7fcc1c057a20 con 0x7fcc20159df0 2023-10-16T12:11:49.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.899+0000 7fcc28362700 1 -- 172.21.15.16:0/360862577 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fcc0406af00 msgr2=0x7fcc0406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:49.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.899+0000 7fcc28362700 1 --2- 172.21.15.16:0/360862577 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fcc0406af00 0x7fcc0406d3c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fcc10009cb0 tx=0x7fcc10008040 comp rx=0 tx=0).stop 2023-10-16T12:11:49.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.899+0000 7fcc28362700 1 -- 172.21.15.16:0/360862577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc20159df0 msgr2=0x7fcc20158240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:49.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.899+0000 7fcc28362700 1 --2- 172.21.15.16:0/360862577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc20159df0 0x7fcc20158240 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fcc1c00f820 tx=0x7fcc1c00fbe0 comp rx=0 tx=0).stop 2023-10-16T12:11:49.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.899+0000 7fcc28362700 1 -- 172.21.15.16:0/360862577 shutdown_connections 2023-10-16T12:11:49.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.899+0000 7fcc28362700 1 --2- 172.21.15.16:0/360862577 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fcc0406af00 0x7fcc0406d3c0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:49.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.899+0000 7fcc28362700 1 --2- 172.21.15.16:0/360862577 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcc2015caa0 0x7fcc20157240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:49.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.899+0000 7fcc28362700 1 --2- 172.21.15.16:0/360862577 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcc0c005210 0x7fcc201f2b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:49.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.899+0000 7fcc28362700 1 --2- 172.21.15.16:0/360862577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc20159df0 0x7fcc20158240 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:49.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.899+0000 7fcc28362700 1 -- 172.21.15.16:0/360862577 >> 172.21.15.16:0/360862577 conn(0x7fcc200b1f40 msgr2=0x7fcc200cbae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:49.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.899+0000 7fcc28362700 1 -- 172.21.15.16:0/360862577 shutdown_connections 2023-10-16T12:11:49.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:49.899+0000 7fcc28362700 1 -- 172.21.15.16:0/360862577 wait complete. 2023-10-16T12:11:49.918 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-16T12:11:50.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:50.053+0000 7fc56d0ab700 1 Processor -- start 2023-10-16T12:11:50.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:50.053+0000 7fc56d0ab700 1 -- start start 2023-10-16T12:11:50.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:50.053+0000 7fc56d0ab700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc56806d980 0x7fc56815efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:50.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:50.053+0000 7fc56d0ab700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc56806a7e0 0x7fc56806ac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:50.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:50.053+0000 7fc56d0ab700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5680cdeb0 con 0x7fc568160e60 2023-10-16T12:11:50.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:50.053+0000 7fc56d0ab700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5681613e0 con 0x7fc56806d980 2023-10-16T12:11:50.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:50.053+0000 7fc56d0ab700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc56806b160 con 0x7fc56806a7e0 2023-10-16T12:11:50.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:50.054+0000 7fc56659c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc56806d980 0x7fc56815efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:50.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:50.054+0000 7fc56659c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc56806d980 0x7fc56815efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35242/0 (socket says 172.21.15.16:35242) 2023-10-16T12:11:50.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:50.054+0000 7fc56659c700 1 -- 172.21.15.16:0/2064838036 learned_addr learned my addr 172.21.15.16:0/2064838036 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:50.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:50.054+0000 7fc56759e700 1 --2- 172.21.15.16:0/2064838036 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc56806a7e0 0x7fc56806ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:50.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:50.054+0000 7fc56659c700 1 -- 172.21.15.16:0/2064838036 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc56806a7e0 msgr2=0x7fc56806ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:50.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:50.054+0000 7fc56659c700 1 --2- 172.21.15.16:0/2064838036 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc56806a7e0 0x7fc56806ac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:50.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:50.054+0000 7fc56659c700 1 -- 172.21.15.16:0/2064838036 >> v1:172.21.15.49:6789/0 conn(0x7fc568160e60 legacy=0x7fc56815e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:50.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:50.054+0000 7fc56659c700 1 -- 172.21.15.16:0/2064838036 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5681582d0 con 0x7fc56806d980 2023-10-16T12:11:50.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:50.054+0000 7fc56659c700 1 --2- 172.21.15.16:0/2064838036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc56806d980 0x7fc56815efb0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fc5580098f0 tx=0x7fc558009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=325f5fdb5d9fbeeb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:50.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:50.054+0000 7fc565d9b700 1 -- 172.21.15.16:0/2064838036 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc558004230 con 0x7fc56806d980 2023-10-16T12:11:50.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:50.054+0000 7fc565d9b700 1 -- 172.21.15.16:0/2064838036 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc558004390 con 0x7fc56806d980 2023-10-16T12:11:50.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:50.054+0000 7fc565d9b700 1 -- 172.21.15.16:0/2064838036 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5580036a0 con 0x7fc56806d980 2023-10-16T12:11:50.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:50.055+0000 7fc56d0ab700 1 -- 172.21.15.16:0/2064838036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc56806d980 msgr2=0x7fc56815efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:50.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:50.055+0000 7fc56d0ab700 1 --2- 172.21.15.16:0/2064838036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc56806d980 0x7fc56815efb0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fc5580098f0 tx=0x7fc558009c00 comp rx=0 tx=0).stop 2023-10-16T12:11:50.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:50.055+0000 7fc56d0ab700 1 -- 172.21.15.16:0/2064838036 shutdown_connections 2023-10-16T12:11:50.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:50.055+0000 7fc56d0ab700 1 --2- 172.21.15.16:0/2064838036 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc56806a7e0 0x7fc56806ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:50.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:50.055+0000 7fc56d0ab700 1 --2- 172.21.15.16:0/2064838036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc56806d980 0x7fc56815efb0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:50.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:50.055+0000 7fc56d0ab700 1 -- 172.21.15.16:0/2064838036 >> 172.21.15.16:0/2064838036 conn(0x7fc56806f430 msgr2=0x7fc56806f810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:50.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:50.055+0000 7fc56d0ab700 1 -- 172.21.15.16:0/2064838036 shutdown_connections 2023-10-16T12:11:50.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:50.055+0000 7fc56d0ab700 1 -- 172.21.15.16:0/2064838036 wait complete. 2023-10-16T12:11:50.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:50.056+0000 7fc56d0ab700 1 Processor -- start 2023-10-16T12:11:50.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:50.056+0000 7fc56d0ab700 1 -- start start 2023-10-16T12:11:50.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:50.057+0000 7fc56d0ab700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc56806a7e0 0x7fc5680ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:50.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:50.057+0000 7fc56d0ab700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc56806d980 0x7fc5681f6c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:50.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:50.057+0000 7fc56d0ab700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc568160e60 0x7fc5681f7160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:50.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:50.057+0000 7fc56d0ab700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc568154ae0 con 0x7fc56806d980 2023-10-16T12:11:50.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:50.057+0000 7fc56d0ab700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5681547e0 con 0x7fc56806a7e0 2023-10-16T12:11:50.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:50.057+0000 7fc56d0ab700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5680b3040 con 0x7fc568160e60 2023-10-16T12:11:50.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:50.057+0000 7fc56659c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc56806d980 0x7fc5681f6c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:50.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:50.057+0000 7fc56659c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc56806d980 0x7fc5681f6c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35248/0 (socket says 172.21.15.16:35248) 2023-10-16T12:11:50.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:50.057+0000 7fc56659c700 1 -- 172.21.15.16:0/2540692991 learned_addr learned my addr 172.21.15.16:0/2540692991 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:50.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:50.057+0000 7fc56759e700 1 --2- 172.21.15.16:0/2540692991 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc568160e60 0x7fc5681f7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:50.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:50.057+0000 7fc566d9d700 1 --2- 172.21.15.16:0/2540692991 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc56806a7e0 0x7fc5680ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:50.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:50.057+0000 7fc56659c700 1 -- 172.21.15.16:0/2540692991 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc568160e60 msgr2=0x7fc5681f7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:50.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:50.057+0000 7fc56659c700 1 --2- 172.21.15.16:0/2540692991 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc568160e60 0x7fc5681f7160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:50.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:50.057+0000 7fc56659c700 1 -- 172.21.15.16:0/2540692991 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc56806a7e0 msgr2=0x7fc5680ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:50.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:50.057+0000 7fc56659c700 1 --2- 172.21.15.16:0/2540692991 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc56806a7e0 0x7fc5680ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:50.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:50.058+0000 7fc56659c700 1 -- 172.21.15.16:0/2540692991 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc558009550 con 0x7fc56806d980 2023-10-16T12:11:50.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:50.058+0000 7fc566d9d700 1 --2- 172.21.15.16:0/2540692991 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc56806a7e0 0x7fc5680ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:11:50.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:50.058+0000 7fc56659c700 1 --2- 172.21.15.16:0/2540692991 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc56806d980 0x7fc5681f6c20 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fc558004000 tx=0x7fc558004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:50.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:50.058+0000 7fc54f7fe700 1 -- 172.21.15.16:0/2540692991 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc558003d80 con 0x7fc56806d980 2023-10-16T12:11:50.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:50.058+0000 7fc54f7fe700 1 -- 172.21.15.16:0/2540692991 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc55801f070 con 0x7fc56806d980 2023-10-16T12:11:50.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:50.058+0000 7fc56d0ab700 1 -- 172.21.15.16:0/2540692991 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5680af270 con 0x7fc56806d980 2023-10-16T12:11:50.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:50.058+0000 7fc54f7fe700 1 -- 172.21.15.16:0/2540692991 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc55800b8a0 con 0x7fc56806d980 2023-10-16T12:11:50.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:50.058+0000 7fc56d0ab700 1 -- 172.21.15.16:0/2540692991 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5680b59b0 con 0x7fc56806d980 2023-10-16T12:11:50.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:50.059+0000 7fc56d0ab700 1 -- 172.21.15.16:0/2540692991 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc56804ec20 con 0x7fc56806d980 2023-10-16T12:11:50.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:50.059+0000 7fc54f7fe700 1 -- 172.21.15.16:0/2540692991 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fc558021030 con 0x7fc56806d980 2023-10-16T12:11:50.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:50.060+0000 7fc54f7fe700 1 --2- 172.21.15.16:0/2540692991 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fc55406af00 0x7fc55406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:50.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:50.060+0000 7fc54f7fe700 1 -- 172.21.15.16:0/2540692991 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6749+0+0 (secure 0 0 0) 0x7fc55808af10 con 0x7fc56806d980 2023-10-16T12:11:50.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:50.060+0000 7fc566d9d700 1 --2- 172.21.15.16:0/2540692991 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fc55406af00 0x7fc55406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:50.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:50.060+0000 7fc566d9d700 1 --2- 172.21.15.16:0/2540692991 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fc55406af00 0x7fc55406d3c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc55c009930 tx=0x7fc55c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:50.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:50.064+0000 7fc54f7fe700 1 -- 172.21.15.16:0/2540692991 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc558055c40 con 0x7fc56806d980 2023-10-16T12:11:50.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:50.266+0000 7fc56d0ab700 1 -- 172.21.15.16:0/2540692991 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc56804ec20 con 0x7fc56806d980 2023-10-16T12:11:50.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:50.899+0000 7fc54f7fe700 1 -- 172.21.15.16:0/2540692991 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v29) v1 ==== 181+0+0 (secure 0 0 0) 0x7fc558059260 con 0x7fc56806d980 2023-10-16T12:11:50.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:50.904+0000 7fc56d0ab700 1 -- 172.21.15.16:0/2540692991 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fc55406af00 msgr2=0x7fc55406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:50.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:50.904+0000 7fc56d0ab700 1 --2- 172.21.15.16:0/2540692991 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fc55406af00 0x7fc55406d3c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc55c009930 tx=0x7fc55c008040 comp rx=0 tx=0).stop 2023-10-16T12:11:50.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:50.904+0000 7fc56d0ab700 1 -- 172.21.15.16:0/2540692991 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc56806d980 msgr2=0x7fc5681f6c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:50.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:50.904+0000 7fc56d0ab700 1 --2- 172.21.15.16:0/2540692991 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc56806d980 0x7fc5681f6c20 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fc558004000 tx=0x7fc558004750 comp rx=0 tx=0).stop 2023-10-16T12:11:50.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:50.905+0000 7fc56d0ab700 1 -- 172.21.15.16:0/2540692991 shutdown_connections 2023-10-16T12:11:50.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:50.905+0000 7fc56d0ab700 1 --2- 172.21.15.16:0/2540692991 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fc55406af00 0x7fc55406d3c0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:50.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:50.905+0000 7fc56d0ab700 1 --2- 172.21.15.16:0/2540692991 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc568160e60 0x7fc5681f7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:50.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:50.905+0000 7fc56d0ab700 1 --2- 172.21.15.16:0/2540692991 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc56806a7e0 0x7fc5680ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:50.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:50.905+0000 7fc56d0ab700 1 --2- 172.21.15.16:0/2540692991 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc56806d980 0x7fc5681f6c20 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:50.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:50.905+0000 7fc56d0ab700 1 -- 172.21.15.16:0/2540692991 >> 172.21.15.16:0/2540692991 conn(0x7fc56806f430 msgr2=0x7fc5681507a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:50.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:50.905+0000 7fc56d0ab700 1 -- 172.21.15.16:0/2540692991 shutdown_connections 2023-10-16T12:11:50.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:50.905+0000 7fc56d0ab700 1 -- 172.21.15.16:0/2540692991 wait complete. 2023-10-16T12:11:50.911 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' removed 2023-10-16T12:11:50.923 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-16T12:11:51.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:51.054+0000 7f3b419ea700 1 Processor -- start 2023-10-16T12:11:51.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:51.054+0000 7f3b419ea700 1 -- start start 2023-10-16T12:11:51.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:51.055+0000 7f3b419ea700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3b3c06d980 0x7f3b3c15efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:51.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:51.055+0000 7f3b419ea700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b3c0cdeb0 con 0x7f3b3c06a7e0 2023-10-16T12:11:51.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:51.055+0000 7f3b419ea700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b3c1613e0 con 0x7f3b3c160e60 2023-10-16T12:11:51.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:51.055+0000 7f3b419ea700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b3c1582d0 con 0x7f3b3c06d980 2023-10-16T12:11:51.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:51.055+0000 7f3b3a7fc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3b3c06d980 0x7f3b3c15efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:51.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:51.055+0000 7f3b3a7fc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3b3c06d980 0x7f3b3c15efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35256/0 (socket says 172.21.15.16:35256) 2023-10-16T12:11:51.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:51.055+0000 7f3b3b7fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f3b3c06a7e0 0x7f3b3c161870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:52764/0 (socket says 172.21.15.16:52764) 2023-10-16T12:11:51.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:51.055+0000 7f3b3a7fc700 1 -- 172.21.15.16:0/3237009365 learned_addr learned my addr 172.21.15.16:0/3237009365 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:51.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:51.056+0000 7f3b3a7fc700 1 -- 172.21.15.16:0/3237009365 >> v1:172.21.15.49:6789/0 conn(0x7f3b3c160e60 legacy=0x7f3b3c15e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:51.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:51.056+0000 7f3b3a7fc700 1 -- 172.21.15.16:0/3237009365 >> v1:172.21.15.16:6789/0 conn(0x7f3b3c06a7e0 legacy=0x7f3b3c161870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:51.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:51.056+0000 7f3b39ffb700 1 -- 172.21.15.16:0/3237009365 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2013733377 0 0) 0x7f3b3c0cdeb0 con 0x7f3b3c06a7e0 2023-10-16T12:11:51.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:51.056+0000 7f3b3a7fc700 1 -- 172.21.15.16:0/3237009365 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b3c0ce100 con 0x7f3b3c06d980 2023-10-16T12:11:51.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:51.056+0000 7f3b3a7fc700 1 --2- 172.21.15.16:0/3237009365 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b3c06d980 0x7f3b3c15efb0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f3b300098f0 tx=0x7f3b30009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8647a4edfa1dcc14 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:51.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:51.056+0000 7f3b39ffb700 1 -- 172.21.15.16:0/3237009365 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b3000b8a0 con 0x7f3b3c06d980 2023-10-16T12:11:51.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:51.056+0000 7f3b39ffb700 1 -- 172.21.15.16:0/3237009365 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b3000ba00 con 0x7f3b3c06d980 2023-10-16T12:11:51.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:51.056+0000 7f3b39ffb700 1 -- 172.21.15.16:0/3237009365 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b3000bd90 con 0x7f3b3c06d980 2023-10-16T12:11:51.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:51.057+0000 7f3b419ea700 1 -- 172.21.15.16:0/3237009365 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b3c06d980 msgr2=0x7f3b3c15efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:51.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:51.057+0000 7f3b419ea700 1 --2- 172.21.15.16:0/3237009365 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b3c06d980 0x7f3b3c15efb0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f3b300098f0 tx=0x7f3b30009c00 comp rx=0 tx=0).stop 2023-10-16T12:11:51.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:51.057+0000 7f3b419ea700 1 -- 172.21.15.16:0/3237009365 shutdown_connections 2023-10-16T12:11:51.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:51.057+0000 7f3b419ea700 1 --2- 172.21.15.16:0/3237009365 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b3c06d980 0x7f3b3c15efb0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:51.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:51.057+0000 7f3b419ea700 1 -- 172.21.15.16:0/3237009365 >> 172.21.15.16:0/3237009365 conn(0x7f3b3c06f430 msgr2=0x7f3b3c06f810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:51.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:51.057+0000 7f3b419ea700 1 -- 172.21.15.16:0/3237009365 shutdown_connections 2023-10-16T12:11:51.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:51.057+0000 7f3b419ea700 1 -- 172.21.15.16:0/3237009365 wait complete. 2023-10-16T12:11:51.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:51.058+0000 7f3b419ea700 1 Processor -- start 2023-10-16T12:11:51.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:51.058+0000 7f3b419ea700 1 -- start start 2023-10-16T12:11:51.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:51.059+0000 7f3b419ea700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3b3c06a7e0 0x7f3b3c0c2f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:51.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:51.059+0000 7f3b419ea700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b3c06d980 0x7f3b3c0b3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:51.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:51.059+0000 7f3b419ea700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3b3c160e60 0x7f3b3c0aef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:51.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:51.059+0000 7f3b419ea700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b3c0c5750 con 0x7f3b3c06d980 2023-10-16T12:11:51.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:51.059+0000 7f3b419ea700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b3c0c3470 con 0x7f3b3c06a7e0 2023-10-16T12:11:51.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:51.059+0000 7f3b419ea700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b3c0c53b0 con 0x7f3b3c160e60 2023-10-16T12:11:51.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:51.059+0000 7f3b3a7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b3c06d980 0x7f3b3c0b3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:51.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:51.059+0000 7f3b3a7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b3c06d980 0x7f3b3c0b3910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35262/0 (socket says 172.21.15.16:35262) 2023-10-16T12:11:51.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:51.059+0000 7f3b3a7fc700 1 -- 172.21.15.16:0/1932579942 learned_addr learned my addr 172.21.15.16:0/1932579942 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:51.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:51.059+0000 7f3b3affd700 1 --2- 172.21.15.16:0/1932579942 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3b3c06a7e0 0x7f3b3c0c2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:51.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:51.059+0000 7f3b3b7fe700 1 --2- 172.21.15.16:0/1932579942 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3b3c160e60 0x7f3b3c0aef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:51.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:51.059+0000 7f3b3a7fc700 1 -- 172.21.15.16:0/1932579942 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3b3c160e60 msgr2=0x7f3b3c0aef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:51.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:51.059+0000 7f3b3a7fc700 1 --2- 172.21.15.16:0/1932579942 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3b3c160e60 0x7f3b3c0aef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:51.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:51.059+0000 7f3b3a7fc700 1 -- 172.21.15.16:0/1932579942 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3b3c06a7e0 msgr2=0x7f3b3c0c2f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:51.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:51.059+0000 7f3b3a7fc700 1 --2- 172.21.15.16:0/1932579942 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3b3c06a7e0 0x7f3b3c0c2f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:51.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:51.059+0000 7f3b3a7fc700 1 -- 172.21.15.16:0/1932579942 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b30009550 con 0x7f3b3c06d980 2023-10-16T12:11:51.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:51.059+0000 7f3b3a7fc700 1 --2- 172.21.15.16:0/1932579942 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b3c06d980 0x7f3b3c0b3910 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f3b3c06c8d0 tx=0x7f3b30004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:51.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:51.060+0000 7f3b23fff700 1 -- 172.21.15.16:0/1932579942 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b3001a900 con 0x7f3b3c06d980 2023-10-16T12:11:51.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:51.060+0000 7f3b23fff700 1 -- 172.21.15.16:0/1932579942 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b3001f070 con 0x7f3b3c06d980 2023-10-16T12:11:51.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:51.060+0000 7f3b23fff700 1 -- 172.21.15.16:0/1932579942 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b3001ac60 con 0x7f3b3c06d980 2023-10-16T12:11:51.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:51.060+0000 7f3b419ea700 1 -- 172.21.15.16:0/1932579942 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b3c0b4280 con 0x7f3b3c06d980 2023-10-16T12:11:51.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:51.060+0000 7f3b419ea700 1 -- 172.21.15.16:0/1932579942 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b3c0b4840 con 0x7f3b3c06d980 2023-10-16T12:11:51.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:51.061+0000 7f3b419ea700 1 -- 172.21.15.16:0/1932579942 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b3c0c5890 con 0x7f3b3c06d980 2023-10-16T12:11:51.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:51.061+0000 7f3b23fff700 1 -- 172.21.15.16:0/1932579942 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f3b3001b430 con 0x7f3b3c06d980 2023-10-16T12:11:51.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:51.061+0000 7f3b23fff700 1 --2- 172.21.15.16:0/1932579942 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f3b2806af00 0x7f3b2806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:51.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:51.062+0000 7f3b23fff700 1 -- 172.21.15.16:0/1932579942 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6311+0+0 (secure 0 0 0) 0x7f3b30021070 con 0x7f3b3c06d980 2023-10-16T12:11:51.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:51.062+0000 7f3b3affd700 1 --2- 172.21.15.16:0/1932579942 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f3b2806af00 0x7f3b2806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:51.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:51.062+0000 7f3b3affd700 1 --2- 172.21.15.16:0/1932579942 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f3b2806af00 0x7f3b2806d3c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3b3c0626c0 tx=0x7f3b2c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:51.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:51.066+0000 7f3b23fff700 1 -- 172.21.15.16:0/1932579942 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3b3c0c5890 con 0x7f3b3c06d980 2023-10-16T12:11:51.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:51.269+0000 7f3b419ea700 1 -- 172.21.15.16:0/1932579942 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f3b3c0c5890 con 0x7f3b3c06d980 2023-10-16T12:11:52.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:52.057+0000 7f3b23fff700 1 -- 172.21.15.16:0/1932579942 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v30) v1 ==== 169+0+0 (secure 0 0 0) 0x7f3b3c0c5890 con 0x7f3b3c06d980 2023-10-16T12:11:52.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:52.060+0000 7f3b419ea700 1 -- 172.21.15.16:0/1932579942 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f3b2806af00 msgr2=0x7f3b2806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:52.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:52.060+0000 7f3b419ea700 1 --2- 172.21.15.16:0/1932579942 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f3b2806af00 0x7f3b2806d3c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3b3c0626c0 tx=0x7f3b2c009040 comp rx=0 tx=0).stop 2023-10-16T12:11:52.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:52.060+0000 7f3b419ea700 1 -- 172.21.15.16:0/1932579942 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b3c06d980 msgr2=0x7f3b3c0b3910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:52.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:52.060+0000 7f3b419ea700 1 --2- 172.21.15.16:0/1932579942 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b3c06d980 0x7f3b3c0b3910 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f3b3c06c8d0 tx=0x7f3b30004750 comp rx=0 tx=0).stop 2023-10-16T12:11:52.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:52.061+0000 7f3b419ea700 1 -- 172.21.15.16:0/1932579942 shutdown_connections 2023-10-16T12:11:52.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:52.061+0000 7f3b419ea700 1 --2- 172.21.15.16:0/1932579942 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f3b2806af00 0x7f3b2806d3c0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:52.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:52.061+0000 7f3b419ea700 1 --2- 172.21.15.16:0/1932579942 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3b3c160e60 0x7f3b3c0aef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:52.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:52.061+0000 7f3b419ea700 1 --2- 172.21.15.16:0/1932579942 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3b3c06a7e0 0x7f3b3c0c2f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:52.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:52.061+0000 7f3b419ea700 1 --2- 172.21.15.16:0/1932579942 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b3c06d980 0x7f3b3c0b3910 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:52.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:52.061+0000 7f3b419ea700 1 -- 172.21.15.16:0/1932579942 >> 172.21.15.16:0/1932579942 conn(0x7f3b3c06f430 msgr2=0x7f3b3c156160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:52.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:52.061+0000 7f3b419ea700 1 -- 172.21.15.16:0/1932579942 shutdown_connections 2023-10-16T12:11:52.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:52.062+0000 7f3b419ea700 1 -- 172.21.15.16:0/1932579942 wait complete. 2023-10-16T12:11:52.066 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' removed 2023-10-16T12:11:52.079 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2023-10-16T12:11:52.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:52.222+0000 7fbf42016700 1 Processor -- start 2023-10-16T12:11:52.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:52.222+0000 7fbf42016700 1 -- start start 2023-10-16T12:11:52.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:52.223+0000 7fbf42016700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbf3c15a3d0 0x7fbf3c06d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:52.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:52.223+0000 7fbf42016700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbf3c15e0b0 0x7fbf3c0ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:52.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:52.223+0000 7fbf42016700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf3c0cb870 con 0x7fbf3c0cf430 2023-10-16T12:11:52.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:52.223+0000 7fbf42016700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf3c15f0c0 con 0x7fbf3c15e0b0 2023-10-16T12:11:52.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:52.223+0000 7fbf42016700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf3c072990 con 0x7fbf3c15a3d0 2023-10-16T12:11:52.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:52.223+0000 7fbf3affd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbf3c15e0b0 0x7fbf3c0ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:52.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:52.223+0000 7fbf3bfff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fbf3c0cf430 0x7fbf3c0cdeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:52778/0 (socket says 172.21.15.16:52778) 2023-10-16T12:11:52.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:52.223+0000 7fbf3bfff700 1 -- 172.21.15.16:0/3377146243 learned_addr learned my addr 172.21.15.16:0/3377146243 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:52.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:52.223+0000 7fbf3b7fe700 1 --2- 172.21.15.16:0/3377146243 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbf3c15a3d0 0x7fbf3c06d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:52.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:52.223+0000 7fbf3affd700 1 -- 172.21.15.16:0/3377146243 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbf3c15a3d0 msgr2=0x7fbf3c06d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:52.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:52.223+0000 7fbf3affd700 1 --2- 172.21.15.16:0/3377146243 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbf3c15a3d0 0x7fbf3c06d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:52.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:52.223+0000 7fbf3affd700 1 -- 172.21.15.16:0/3377146243 >> v1:172.21.15.16:6789/0 conn(0x7fbf3c0cf430 legacy=0x7fbf3c0cdeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:52.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:52.223+0000 7fbf3affd700 1 -- 172.21.15.16:0/3377146243 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf3c06e040 con 0x7fbf3c15e0b0 2023-10-16T12:11:52.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:52.223+0000 7fbf3a7fc700 1 -- 172.21.15.16:0/3377146243 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4244193615 0 0) 0x7fbf3c0cb870 con 0x7fbf3c0cf430 2023-10-16T12:11:52.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:52.223+0000 7fbf3affd700 1 --2- 172.21.15.16:0/3377146243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf3c15e0b0 0x7fbf3c0ceef0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c009840 tx=0x7fbf2c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d6d16b1942667c3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:52.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:52.224+0000 7fbf3a7fc700 1 -- 172.21.15.16:0/3377146243 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf2c011830 con 0x7fbf3c15e0b0 2023-10-16T12:11:52.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:52.224+0000 7fbf3a7fc700 1 -- 172.21.15.16:0/3377146243 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf2c011990 con 0x7fbf3c15e0b0 2023-10-16T12:11:52.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:52.224+0000 7fbf3a7fc700 1 -- 172.21.15.16:0/3377146243 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf2c011d20 con 0x7fbf3c15e0b0 2023-10-16T12:11:52.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:52.224+0000 7fbf42016700 1 -- 172.21.15.16:0/3377146243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf3c15e0b0 msgr2=0x7fbf3c0ceef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:52.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:52.224+0000 7fbf42016700 1 --2- 172.21.15.16:0/3377146243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf3c15e0b0 0x7fbf3c0ceef0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c009840 tx=0x7fbf2c009b50 comp rx=0 tx=0).stop 2023-10-16T12:11:52.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:52.224+0000 7fbf42016700 1 -- 172.21.15.16:0/3377146243 shutdown_connections 2023-10-16T12:11:52.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:52.224+0000 7fbf42016700 1 --2- 172.21.15.16:0/3377146243 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbf3c15a3d0 0x7fbf3c06d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:52.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:52.224+0000 7fbf42016700 1 --2- 172.21.15.16:0/3377146243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf3c15e0b0 0x7fbf3c0ceef0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:52.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:52.224+0000 7fbf42016700 1 -- 172.21.15.16:0/3377146243 >> 172.21.15.16:0/3377146243 conn(0x7fbf3c06f430 msgr2=0x7fbf3c06f810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:52.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:52.224+0000 7fbf42016700 1 -- 172.21.15.16:0/3377146243 shutdown_connections 2023-10-16T12:11:52.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:52.224+0000 7fbf42016700 1 -- 172.21.15.16:0/3377146243 wait complete. 2023-10-16T12:11:52.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:52.225+0000 7fbf42016700 1 Processor -- start 2023-10-16T12:11:52.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:52.226+0000 7fbf42016700 1 -- start start 2023-10-16T12:11:52.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:52.226+0000 7fbf42016700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf3c0cf430 0x7fbf3c0c9650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:52.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:52.226+0000 7fbf42016700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbf3c15a3d0 0x7fbf3c14d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:52.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:52.226+0000 7fbf42016700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbf3c15e0b0 0x7fbf3c14ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:52.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:52.226+0000 7fbf42016700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf3c0c7f40 con 0x7fbf3c0cf430 2023-10-16T12:11:52.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:52.226+0000 7fbf42016700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf3c14dc10 con 0x7fbf3c15a3d0 2023-10-16T12:11:52.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:52.226+0000 7fbf42016700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf3c1f4d10 con 0x7fbf3c15e0b0 2023-10-16T12:11:52.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:52.226+0000 7fbf3b7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf3c0cf430 0x7fbf3c0c9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:52.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:52.226+0000 7fbf3b7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf3c0cf430 0x7fbf3c0c9650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35278/0 (socket says 172.21.15.16:35278) 2023-10-16T12:11:52.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:52.226+0000 7fbf3b7fe700 1 -- 172.21.15.16:0/2189015313 learned_addr learned my addr 172.21.15.16:0/2189015313 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:52.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:52.227+0000 7fbf3affd700 1 --2- 172.21.15.16:0/2189015313 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbf3c15a3d0 0x7fbf3c14d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:52.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:52.227+0000 7fbf3b7fe700 1 -- 172.21.15.16:0/2189015313 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbf3c15e0b0 msgr2=0x7fbf3c14ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:52.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:52.227+0000 7fbf3b7fe700 1 --2- 172.21.15.16:0/2189015313 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbf3c15e0b0 0x7fbf3c14ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:52.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:52.227+0000 7fbf3b7fe700 1 -- 172.21.15.16:0/2189015313 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbf3c15a3d0 msgr2=0x7fbf3c14d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:52.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:52.227+0000 7fbf3b7fe700 1 --2- 172.21.15.16:0/2189015313 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbf3c15a3d0 0x7fbf3c14d6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:52.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:52.227+0000 7fbf3b7fe700 1 -- 172.21.15.16:0/2189015313 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf2c0094a0 con 0x7fbf3c0cf430 2023-10-16T12:11:52.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:52.227+0000 7fbf3b7fe700 1 --2- 172.21.15.16:0/2189015313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf3c0cf430 0x7fbf3c0c9650 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fbf3000e840 tx=0x7fbf3000ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:52.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:52.227+0000 7fbf23fff700 1 -- 172.21.15.16:0/2189015313 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf30013610 con 0x7fbf3c0cf430 2023-10-16T12:11:52.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:52.227+0000 7fbf23fff700 1 -- 172.21.15.16:0/2189015313 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf30013770 con 0x7fbf3c0cf430 2023-10-16T12:11:52.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:52.227+0000 7fbf42016700 1 -- 172.21.15.16:0/2189015313 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf3c0c7610 con 0x7fbf3c0cf430 2023-10-16T12:11:52.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:52.227+0000 7fbf23fff700 1 -- 172.21.15.16:0/2189015313 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf30013b20 con 0x7fbf3c0cf430 2023-10-16T12:11:52.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:52.227+0000 7fbf42016700 1 -- 172.21.15.16:0/2189015313 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf3c0c7bd0 con 0x7fbf3c0cf430 2023-10-16T12:11:52.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:52.228+0000 7fbf42016700 1 -- 172.21.15.16:0/2189015313 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf3c04ec20 con 0x7fbf3c0cf430 2023-10-16T12:11:52.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:52.229+0000 7fbf23fff700 1 -- 172.21.15.16:0/2189015313 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fbf30013c80 con 0x7fbf3c0cf430 2023-10-16T12:11:52.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:52.229+0000 7fbf23fff700 1 --2- 172.21.15.16:0/2189015313 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fbf2806af00 0x7fbf2806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:52.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:52.229+0000 7fbf3affd700 1 --2- 172.21.15.16:0/2189015313 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fbf2806af00 0x7fbf2806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:52.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:52.229+0000 7fbf23fff700 1 -- 172.21.15.16:0/2189015313 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5781+0+0 (secure 0 0 0) 0x7fbf300892a0 con 0x7fbf3c0cf430 2023-10-16T12:11:52.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:52.230+0000 7fbf3affd700 1 --2- 172.21.15.16:0/2189015313 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fbf2806af00 0x7fbf2806d3c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fbf3c158460 tx=0x7fbf2c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:52.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:52.234+0000 7fbf23fff700 1 -- 172.21.15.16:0/2189015313 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbf30054390 con 0x7fbf3c0cf430 2023-10-16T12:11:52.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:52.438+0000 7fbf42016700 1 -- 172.21.15.16:0/2189015313 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fbf3c0b6260 con 0x7fbf3c0cf430 2023-10-16T12:11:53.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:53.058+0000 7fbf23fff700 1 -- 172.21.15.16:0/2189015313 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v31) v1 ==== 178+0+0 (secure 0 0 0) 0x7fbf300579b0 con 0x7fbf3c0cf430 2023-10-16T12:11:53.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:53.065+0000 7fbf42016700 1 -- 172.21.15.16:0/2189015313 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fbf2806af00 msgr2=0x7fbf2806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:53.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:53.065+0000 7fbf42016700 1 --2- 172.21.15.16:0/2189015313 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fbf2806af00 0x7fbf2806d3c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fbf3c158460 tx=0x7fbf2c008030 comp rx=0 tx=0).stop 2023-10-16T12:11:53.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:53.065+0000 7fbf42016700 1 -- 172.21.15.16:0/2189015313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf3c0cf430 msgr2=0x7fbf3c0c9650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:53.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:53.065+0000 7fbf42016700 1 --2- 172.21.15.16:0/2189015313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf3c0cf430 0x7fbf3c0c9650 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fbf3000e840 tx=0x7fbf3000ec00 comp rx=0 tx=0).stop 2023-10-16T12:11:53.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:53.065+0000 7fbf42016700 1 -- 172.21.15.16:0/2189015313 shutdown_connections 2023-10-16T12:11:53.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:53.065+0000 7fbf42016700 1 --2- 172.21.15.16:0/2189015313 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fbf2806af00 0x7fbf2806d3c0 secure :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fbf3c158460 tx=0x7fbf2c008030 comp rx=0 tx=0).stop 2023-10-16T12:11:53.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:53.065+0000 7fbf42016700 1 --2- 172.21.15.16:0/2189015313 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbf3c15e0b0 0x7fbf3c14ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:53.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:53.065+0000 7fbf42016700 1 --2- 172.21.15.16:0/2189015313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf3c0cf430 0x7fbf3c0c9650 secure :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fbf3000e840 tx=0x7fbf3000ec00 comp rx=0 tx=0).stop 2023-10-16T12:11:53.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:53.065+0000 7fbf42016700 1 --2- 172.21.15.16:0/2189015313 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbf3c15a3d0 0x7fbf3c14d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:53.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:53.065+0000 7fbf42016700 1 -- 172.21.15.16:0/2189015313 >> 172.21.15.16:0/2189015313 conn(0x7fbf3c06f430 msgr2=0x7fbf3c15e640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:53.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:53.066+0000 7fbf42016700 1 -- 172.21.15.16:0/2189015313 shutdown_connections 2023-10-16T12:11:53.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:53.066+0000 7fbf42016700 1 -- 172.21.15.16:0/2189015313 wait complete. 2023-10-16T12:11:53.071 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data_ec' removed 2023-10-16T12:11:53.086 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-16T12:11:53.086 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-16T12:11:53.087 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-16T12:11:53.087 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T12:11:53.087 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-16T12:11:53.087 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-16T12:11:53.088 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-16T12:11:53.088 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T12:11:53.088 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-16T12:11:53.089 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-16T12:11:53.089 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-16T12:11:53.089 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T12:11:53.090 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-16T12:11:53.090 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-16T12:11:53.090 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-16T12:11:53.090 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T12:11:53.097 INFO:tasks.ceph.mds.a:Stopped 2023-10-16T12:11:53.097 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-16T12:11:53.098 INFO:tasks.ceph.mds.c:Stopped 2023-10-16T12:11:53.098 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-16T12:11:53.101 INFO:tasks.ceph.mds.b:Stopped 2023-10-16T12:11:53.102 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-16T12:11:53.103 INFO:tasks.ceph.mds.d:Stopped 2023-10-16T12:11:53.103 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-16T12:11:53.139 INFO:tasks.ceph.mds.a:Started 2023-10-16T12:11:53.139 DEBUG:teuthology.parallel:result is None 2023-10-16T12:11:53.140 INFO:tasks.ceph.mds.c:Started 2023-10-16T12:11:53.141 DEBUG:teuthology.parallel:result is None 2023-10-16T12:11:53.144 INFO:tasks.ceph.mds.b:Started 2023-10-16T12:11:53.145 DEBUG:teuthology.parallel:result is None 2023-10-16T12:11:53.145 INFO:tasks.ceph.mds.d:Started 2023-10-16T12:11:53.146 DEBUG:teuthology.parallel:result is None 2023-10-16T12:11:53.146 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-16T12:11:53.226 INFO:tasks.ceph.mds.a.smithi016.stdout:starting mds.a at 2023-10-16T12:11:53.226 INFO:tasks.ceph.mds.c.smithi016.stdout:starting mds.c at 2023-10-16T12:11:53.237 INFO:tasks.ceph.mds.d.smithi049.stdout:starting mds.d at 2023-10-16T12:11:53.237 INFO:tasks.ceph.mds.b.smithi049.stdout:starting mds.b at 2023-10-16T12:11:53.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:53.280+0000 7f077bd99700 1 Processor -- start 2023-10-16T12:11:53.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:53.280+0000 7f077bd99700 1 -- start start 2023-10-16T12:11:53.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:53.281+0000 7f077bd99700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f07740d41c0 0x7f07740d0040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:53.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:53.281+0000 7f077bd99700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f07740d0580 0x7f077406c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:53.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:53.281+0000 7f077bd99700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f077406ba00 con 0x7f077415d8d0 2023-10-16T12:11:53.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:53.281+0000 7f077bd99700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07740d0a10 con 0x7f07740d0580 2023-10-16T12:11:53.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:53.281+0000 7f077bd99700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07740d26d0 con 0x7f07740d41c0 2023-10-16T12:11:53.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:53.281+0000 7f0779b35700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f077415d8d0 0x7f07740d1c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:52812/0 (socket says 172.21.15.16:52812) 2023-10-16T12:11:53.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:53.281+0000 7f0779b35700 1 -- 172.21.15.16:0/965766501 learned_addr learned my addr 172.21.15.16:0/965766501 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:53.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:53.281+0000 7f077a336700 1 --2- 172.21.15.16:0/965766501 >> v2:172.21.15.49:3300/0 conn(0x7f07740d0580 0x7f077406c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:53.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:53.281+0000 7f0779334700 1 --2- 172.21.15.16:0/965766501 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f07740d41c0 0x7f07740d0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:53.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:53.282+0000 7f0778b33700 1 -- 172.21.15.16:0/965766501 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3461313799 0 0) 0x7f077406ba00 con 0x7f077415d8d0 2023-10-16T12:11:53.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:53.282+0000 7f0778b33700 1 -- 172.21.15.16:0/965766501 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f075c0035b0 con 0x7f077415d8d0 2023-10-16T12:11:53.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:53.282+0000 7f077a336700 1 -- 172.21.15.16:0/965766501 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f07740d41c0 msgr2=0x7f07740d0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:53.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:53.282+0000 7f077a336700 1 --2- 172.21.15.16:0/965766501 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f07740d41c0 0x7f07740d0040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:53.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:53.282+0000 7f077a336700 1 -- 172.21.15.16:0/965766501 >> v1:172.21.15.16:6789/0 conn(0x7f077415d8d0 legacy=0x7f07740d1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:53.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:53.282+0000 7f077a336700 1 -- 172.21.15.16:0/965766501 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07740d23f0 con 0x7f07740d0580 2023-10-16T12:11:53.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:53.282+0000 7f0779334700 1 --2- 172.21.15.16:0/965766501 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f07740d41c0 0x7f07740d0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:11:53.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:53.282+0000 7f077a336700 1 --2- 172.21.15.16:0/965766501 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f07740d0580 0x7f077406c4a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f0764009870 tx=0x7f0764009b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fdf0757d73a50564 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:53.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:53.282+0000 7f0778b33700 1 -- 172.21.15.16:0/965766501 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0764011830 con 0x7f07740d0580 2023-10-16T12:11:53.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:53.282+0000 7f0778b33700 1 -- 172.21.15.16:0/965766501 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0764011990 con 0x7f07740d0580 2023-10-16T12:11:53.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:53.282+0000 7f0778b33700 1 -- 172.21.15.16:0/965766501 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0764011d20 con 0x7f07740d0580 2023-10-16T12:11:53.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:53.283+0000 7f077bd99700 1 -- 172.21.15.16:0/965766501 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f07740d0580 msgr2=0x7f077406c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:53.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:53.283+0000 7f077bd99700 1 --2- 172.21.15.16:0/965766501 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f07740d0580 0x7f077406c4a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f0764009870 tx=0x7f0764009b80 comp rx=0 tx=0).stop 2023-10-16T12:11:53.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:53.283+0000 7f077bd99700 1 -- 172.21.15.16:0/965766501 shutdown_connections 2023-10-16T12:11:53.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:53.283+0000 7f077bd99700 1 --2- 172.21.15.16:0/965766501 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f07740d0580 0x7f077406c4a0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:53.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:53.283+0000 7f077bd99700 1 --2- 172.21.15.16:0/965766501 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f07740d41c0 0x7f07740d0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:53.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:53.283+0000 7f077bd99700 1 -- 172.21.15.16:0/965766501 >> 172.21.15.16:0/965766501 conn(0x7f07740d3630 msgr2=0x7f07740d3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:53.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:53.283+0000 7f077bd99700 1 -- 172.21.15.16:0/965766501 shutdown_connections 2023-10-16T12:11:53.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:53.283+0000 7f077bd99700 1 -- 172.21.15.16:0/965766501 wait complete. 2023-10-16T12:11:53.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:53.284+0000 7f077bd99700 1 Processor -- start 2023-10-16T12:11:53.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:53.284+0000 7f077bd99700 1 -- start start 2023-10-16T12:11:53.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:53.285+0000 7f077bd99700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f07740d0580 0x7f07741e71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:53.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:53.285+0000 7f077bd99700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07740d41c0 0x7f077415a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:53.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:53.285+0000 7f077bd99700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f077415d8d0 0x7f07740c79c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:53.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:53.285+0000 7f077bd99700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07741e7ae0 con 0x7f07740d41c0 2023-10-16T12:11:53.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:53.285+0000 7f077bd99700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07741e7dc0 con 0x7f077415d8d0 2023-10-16T12:11:53.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:53.285+0000 7f077bd99700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07740c71c0 con 0x7f07740d0580 2023-10-16T12:11:53.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:53.285+0000 7f0779334700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07740d41c0 0x7f077415a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:53.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:53.285+0000 7f0779334700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07740d41c0 0x7f077415a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35314/0 (socket says 172.21.15.16:35314) 2023-10-16T12:11:53.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:53.285+0000 7f0779334700 1 -- 172.21.15.16:0/757690178 learned_addr learned my addr 172.21.15.16:0/757690178 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:53.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:53.285+0000 7f0779b35700 1 --2- 172.21.15.16:0/757690178 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f07740d0580 0x7f07741e71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:53.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:53.285+0000 7f077a336700 1 --2- 172.21.15.16:0/757690178 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f077415d8d0 0x7f07740c79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:53.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:53.285+0000 7f0779334700 1 -- 172.21.15.16:0/757690178 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f07740d0580 msgr2=0x7f07741e71e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:53.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:53.285+0000 7f0779334700 1 --2- 172.21.15.16:0/757690178 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f07740d0580 0x7f07741e71e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:53.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:53.285+0000 7f0779334700 1 -- 172.21.15.16:0/757690178 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f077415d8d0 msgr2=0x7f07740c79c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:53.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:53.285+0000 7f0779334700 1 --2- 172.21.15.16:0/757690178 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f077415d8d0 0x7f07740c79c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:53.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:53.285+0000 7f0779334700 1 -- 172.21.15.16:0/757690178 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07680094a0 con 0x7f07740d41c0 2023-10-16T12:11:53.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:53.285+0000 7f0779334700 1 --2- 172.21.15.16:0/757690178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07740d41c0 0x7f077415a210 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f076800ebe0 tx=0x7f076800eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:53.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:53.285+0000 7f07727fc700 1 -- 172.21.15.16:0/757690178 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f076800c850 con 0x7f07740d41c0 2023-10-16T12:11:53.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:53.286+0000 7f07727fc700 1 -- 172.21.15.16:0/757690178 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f076800c9b0 con 0x7f07740d41c0 2023-10-16T12:11:53.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:53.286+0000 7f077bd99700 1 -- 172.21.15.16:0/757690178 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07640094a0 con 0x7f07740d41c0 2023-10-16T12:11:53.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:53.286+0000 7f07727fc700 1 -- 172.21.15.16:0/757690178 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f076800cd40 con 0x7f07740d41c0 2023-10-16T12:11:53.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:53.286+0000 7f077bd99700 1 -- 172.21.15.16:0/757690178 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07741e80a0 con 0x7f07740d41c0 2023-10-16T12:11:53.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:53.287+0000 7f077bd99700 1 -- 172.21.15.16:0/757690178 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f077404ebb0 con 0x7f07740d41c0 2023-10-16T12:11:53.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:53.287+0000 7f07727fc700 1 -- 172.21.15.16:0/757690178 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f0768013470 con 0x7f07740d41c0 2023-10-16T12:11:53.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:53.287+0000 7f07727fc700 1 --2- 172.21.15.16:0/757690178 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f076006af00 0x7f076006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:53.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:53.288+0000 7f07727fc700 1 -- 172.21.15.16:0/757690178 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5263+0+0 (secure 0 0 0) 0x7f0768089440 con 0x7f07740d41c0 2023-10-16T12:11:53.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:53.288+0000 7f0779b35700 1 --2- 172.21.15.16:0/757690178 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f076006af00 0x7f076006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:53.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:53.288+0000 7f0779b35700 1 --2- 172.21.15.16:0/757690178 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f076006af00 0x7f076006d3c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f076c001560 tx=0x7f076c0019f0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:53.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:53.292+0000 7f07727fc700 1 -- 172.21.15.16:0/757690178 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f07680546b0 con 0x7f07740d41c0 2023-10-16T12:11:53.304 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2023-10-16T12:11:53.305 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~0s 2023-10-16T12:11:53.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:53.510+0000 7f077bd99700 1 -- 172.21.15.16:0/757690178 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f07740b12b0 con 0x7f07740d41c0 2023-10-16T12:11:54.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.079+0000 7f07727fc700 1 -- 172.21.15.16:0/757690178 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v33) v1 ==== 97+0+0 (secure 0 0 0) 0x7f076800cea0 con 0x7f07740d41c0 2023-10-16T12:11:54.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.083+0000 7f077bd99700 1 -- 172.21.15.16:0/757690178 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f076006af00 msgr2=0x7f076006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:54.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.083+0000 7f077bd99700 1 --2- 172.21.15.16:0/757690178 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f076006af00 0x7f076006d3c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f076c001560 tx=0x7f076c0019f0 comp rx=0 tx=0).stop 2023-10-16T12:11:54.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.083+0000 7f077bd99700 1 -- 172.21.15.16:0/757690178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07740d41c0 msgr2=0x7f077415a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:54.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.083+0000 7f077bd99700 1 --2- 172.21.15.16:0/757690178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07740d41c0 0x7f077415a210 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f076800ebe0 tx=0x7f076800eef0 comp rx=0 tx=0).stop 2023-10-16T12:11:54.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.083+0000 7f077bd99700 1 -- 172.21.15.16:0/757690178 shutdown_connections 2023-10-16T12:11:54.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.083+0000 7f077bd99700 1 --2- 172.21.15.16:0/757690178 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f076006af00 0x7f076006d3c0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:54.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.083+0000 7f077bd99700 1 --2- 172.21.15.16:0/757690178 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f077415d8d0 0x7f07740c79c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:54.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.083+0000 7f077bd99700 1 --2- 172.21.15.16:0/757690178 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f07740d0580 0x7f07741e71e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:54.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.083+0000 7f077bd99700 1 --2- 172.21.15.16:0/757690178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07740d41c0 0x7f077415a210 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:54.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.083+0000 7f077bd99700 1 -- 172.21.15.16:0/757690178 >> 172.21.15.16:0/757690178 conn(0x7f07740d3630 msgr2=0x7f077415cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:54.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.083+0000 7f077bd99700 1 -- 172.21.15.16:0/757690178 shutdown_connections 2023-10-16T12:11:54.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.083+0000 7f077bd99700 1 -- 172.21.15.16:0/757690178 wait complete. 2023-10-16T12:11:54.088 INFO:teuthology.orchestra.run.smithi016.stderr: removed all blocklist entries 2023-10-16T12:11:54.103 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-16T12:11:54.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.234+0000 7f19f50e0700 1 Processor -- start 2023-10-16T12:11:54.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.234+0000 7f19f50e0700 1 -- start start 2023-10-16T12:11:54.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.235+0000 7f19f50e0700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f19f00cbed0 0x7f19f006eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:54.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.235+0000 7f19f50e0700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19f006f0a0 con 0x7f19f006d350 2023-10-16T12:11:54.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.235+0000 7f19f50e0700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19f006f380 con 0x7f19f015a870 2023-10-16T12:11:54.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.235+0000 7f19f50e0700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19f015ac50 con 0x7f19f00cbed0 2023-10-16T12:11:54.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.235+0000 7f19ee59c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f19f006d350 0x7f19f00cde10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:52830/0 (socket says 172.21.15.16:52830) 2023-10-16T12:11:54.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.235+0000 7f19ee59c700 1 -- 172.21.15.16:0/1233007076 learned_addr learned my addr 172.21.15.16:0/1233007076 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:54.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.235+0000 7f19ef59e700 1 --2- 172.21.15.16:0/1233007076 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f19f00cbed0 0x7f19f006eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:54.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.235+0000 7f19edd9b700 1 -- 172.21.15.16:0/1233007076 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2895369416 0 0) 0x7f19f006f0a0 con 0x7f19f006d350 2023-10-16T12:11:54.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.235+0000 7f19edd9b700 1 -- 172.21.15.16:0/1233007076 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19d80035b0 con 0x7f19f006d350 2023-10-16T12:11:54.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.235+0000 7f19edd9b700 1 -- 172.21.15.16:0/1233007076 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1716745344 0 0) 0x7f19d80035b0 con 0x7f19f006d350 2023-10-16T12:11:54.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.235+0000 7f19edd9b700 1 -- 172.21.15.16:0/1233007076 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f19f00cbed0 msgr2=0x7f19f006eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:54.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.235+0000 7f19edd9b700 1 --2- 172.21.15.16:0/1233007076 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f19f00cbed0 0x7f19f006eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:54.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.235+0000 7f19edd9b700 1 -- 172.21.15.16:0/1233007076 >> v1:172.21.15.49:6789/0 conn(0x7f19f015a870 legacy=0x7f19f00cb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:54.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.235+0000 7f19edd9b700 1 -- 172.21.15.16:0/1233007076 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19f006f660 con 0x7f19f006d350 2023-10-16T12:11:54.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.235+0000 7f19edd9b700 1 -- 172.21.15.16:0/1233007076 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f19e4001670 con 0x7f19f006d350 2023-10-16T12:11:54.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.235+0000 7f19edd9b700 1 -- 172.21.15.16:0/1233007076 >> v1:172.21.15.16:6789/0 conn(0x7f19f006d350 legacy=0x7f19f00cde10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:54.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.235+0000 7f19edd9b700 1 --2- 172.21.15.16:0/1233007076 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f19d8003a30 0x7f19d8003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:54.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.235+0000 7f19edd9b700 1 --2- 172.21.15.16:0/1233007076 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19d8004390 0x7f19d8004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:54.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.235+0000 7f19edd9b700 1 --2- 172.21.15.16:0/1233007076 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f19d8005070 0x7f19d800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:54.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.235+0000 7f19edd9b700 1 -- 172.21.15.16:0/1233007076 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19f015ac50 con 0x7f19d8004390 2023-10-16T12:11:54.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.235+0000 7f19edd9b700 1 -- 172.21.15.16:0/1233007076 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19d800bf80 con 0x7f19d8005070 2023-10-16T12:11:54.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.235+0000 7f19edd9b700 1 -- 172.21.15.16:0/1233007076 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19d800c140 con 0x7f19d8003a30 2023-10-16T12:11:54.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.235+0000 7f19ef59e700 1 --2- 172.21.15.16:0/1233007076 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f19f00cbed0 0x7f19f006eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:11:54.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.236+0000 7f19ee59c700 1 --2- 172.21.15.16:0/1233007076 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19d8004390 0x7f19d8004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:54.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.236+0000 7f19eed9d700 1 --2- 172.21.15.16:0/1233007076 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f19d8003a30 0x7f19d8003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:54.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.236+0000 7f19ef59e700 1 --2- 172.21.15.16:0/1233007076 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f19d8005070 0x7f19d800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:54.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.236+0000 7f19ee59c700 1 -- 172.21.15.16:0/1233007076 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f19d8003a30 msgr2=0x7f19d8003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:54.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.236+0000 7f19ee59c700 1 --2- 172.21.15.16:0/1233007076 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f19d8003a30 0x7f19d8003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:54.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.236+0000 7f19ee59c700 1 -- 172.21.15.16:0/1233007076 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f19d8005070 msgr2=0x7f19d800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:54.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.236+0000 7f19ee59c700 1 --2- 172.21.15.16:0/1233007076 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f19d8005070 0x7f19d800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:54.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.236+0000 7f19ee59c700 1 -- 172.21.15.16:0/1233007076 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19d80035b0 con 0x7f19d8004390 2023-10-16T12:11:54.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.236+0000 7f19ee59c700 1 --2- 172.21.15.16:0/1233007076 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19d8004390 0x7f19d8004870 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f19e400c2f0 tx=0x7f19d8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe8f9ad391c11489 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:54.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.236+0000 7f19edd9b700 1 -- 172.21.15.16:0/1233007076 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e400cd70 con 0x7f19d8004390 2023-10-16T12:11:54.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.236+0000 7f19edd9b700 1 -- 172.21.15.16:0/1233007076 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19e4005960 con 0x7f19d8004390 2023-10-16T12:11:54.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.236+0000 7f19edd9b700 1 -- 172.21.15.16:0/1233007076 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e4005d10 con 0x7f19d8004390 2023-10-16T12:11:54.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.237+0000 7f19f50e0700 1 -- 172.21.15.16:0/1233007076 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19d8004390 msgr2=0x7f19d8004870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:54.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.237+0000 7f19f50e0700 1 --2- 172.21.15.16:0/1233007076 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19d8004390 0x7f19d8004870 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f19e400c2f0 tx=0x7f19d8005880 comp rx=0 tx=0).stop 2023-10-16T12:11:54.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.237+0000 7f19ef59e700 1 -- 172.21.15.16:0/1233007076 reap_dead start 2023-10-16T12:11:54.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.237+0000 7f19f50e0700 1 -- 172.21.15.16:0/1233007076 shutdown_connections 2023-10-16T12:11:54.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.237+0000 7f19f50e0700 1 -- 172.21.15.16:0/1233007076 >> 172.21.15.16:0/1233007076 conn(0x7f19f00cc5b0 msgr2=0x7f19f00cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:54.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.237+0000 7f19f50e0700 1 -- 172.21.15.16:0/1233007076 shutdown_connections 2023-10-16T12:11:54.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.237+0000 7f19f50e0700 1 -- 172.21.15.16:0/1233007076 wait complete. 2023-10-16T12:11:54.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.238+0000 7f19f50e0700 1 Processor -- start 2023-10-16T12:11:54.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.238+0000 7f19f50e0700 1 -- start start 2023-10-16T12:11:54.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.239+0000 7f19f50e0700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f19d8004390 0x7f19f014c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:54.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.239+0000 7f19f50e0700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f19f01e5380 0x7f19f01e5800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:54.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.239+0000 7f19f50e0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19f01e5d40 0x7f19f014e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:54.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.239+0000 7f19f50e0700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19f01e61f0 con 0x7f19f01e5d40 2023-10-16T12:11:54.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.239+0000 7f19f50e0700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19f014e120 con 0x7f19f01e5380 2023-10-16T12:11:54.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.239+0000 7f19f50e0700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19f00b31a0 con 0x7f19d8004390 2023-10-16T12:11:54.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.239+0000 7f19ef59e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19f01e5d40 0x7f19f014e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:54.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.239+0000 7f19ef59e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19f01e5d40 0x7f19f014e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35338/0 (socket says 172.21.15.16:35338) 2023-10-16T12:11:54.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.239+0000 7f19ef59e700 1 -- 172.21.15.16:0/1375460199 learned_addr learned my addr 172.21.15.16:0/1375460199 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:54.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.239+0000 7f19eed9d700 1 --2- 172.21.15.16:0/1375460199 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f19d8004390 0x7f19f014c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:54.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.239+0000 7f19ee59c700 1 --2- 172.21.15.16:0/1375460199 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f19f01e5380 0x7f19f01e5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:54.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.240+0000 7f19ef59e700 1 -- 172.21.15.16:0/1375460199 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f19d8004390 msgr2=0x7f19f014c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:54.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.240+0000 7f19ef59e700 1 --2- 172.21.15.16:0/1375460199 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f19d8004390 0x7f19f014c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:54.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.240+0000 7f19ef59e700 1 -- 172.21.15.16:0/1375460199 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f19f01e5380 msgr2=0x7f19f01e5800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:54.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.240+0000 7f19ef59e700 1 --2- 172.21.15.16:0/1375460199 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f19f01e5380 0x7f19f01e5800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:54.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.240+0000 7f19ef59e700 1 -- 172.21.15.16:0/1375460199 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19f00b3480 con 0x7f19f01e5d40 2023-10-16T12:11:54.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.240+0000 7f19ee59c700 1 --2- 172.21.15.16:0/1375460199 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f19f01e5380 0x7f19f01e5800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:11:54.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.240+0000 7f19ef59e700 1 --2- 172.21.15.16:0/1375460199 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19f01e5d40 0x7f19f014e620 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f19f00722d0 tx=0x7f19e8011790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:54.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.240+0000 7f19d77fe700 1 -- 172.21.15.16:0/1375460199 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e800e070 con 0x7f19f01e5d40 2023-10-16T12:11:54.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.240+0000 7f19d77fe700 1 -- 172.21.15.16:0/1375460199 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19e8003a20 con 0x7f19f01e5d40 2023-10-16T12:11:54.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.240+0000 7f19f50e0700 1 -- 172.21.15.16:0/1375460199 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19f00b3760 con 0x7f19f01e5d40 2023-10-16T12:11:54.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.240+0000 7f19d77fe700 1 -- 172.21.15.16:0/1375460199 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e8009ea0 con 0x7f19f01e5d40 2023-10-16T12:11:54.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.240+0000 7f19f50e0700 1 -- 172.21.15.16:0/1375460199 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19f014dd40 con 0x7f19f01e5d40 2023-10-16T12:11:54.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.241+0000 7f19f50e0700 1 -- 172.21.15.16:0/1375460199 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19f00b3c80 con 0x7f19f01e5d40 2023-10-16T12:11:54.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.241+0000 7f19d77fe700 1 -- 172.21.15.16:0/1375460199 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f19e8003bd0 con 0x7f19f01e5d40 2023-10-16T12:11:54.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.242+0000 7f19d77fe700 1 --2- 172.21.15.16:0/1375460199 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f19dc06af00 0x7f19dc06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:54.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.242+0000 7f19d77fe700 1 -- 172.21.15.16:0/1375460199 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f19e8089c70 con 0x7f19f01e5d40 2023-10-16T12:11:54.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.242+0000 7f19eed9d700 1 --2- 172.21.15.16:0/1375460199 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f19dc06af00 0x7f19dc06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:54.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.242+0000 7f19eed9d700 1 --2- 172.21.15.16:0/1375460199 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f19dc06af00 0x7f19dc06d3c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f19e0000c00 tx=0x7f19e0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:54.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.246+0000 7f19d77fe700 1 -- 172.21.15.16:0/1375460199 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f19f00b3c80 con 0x7f19f01e5d40 2023-10-16T12:11:54.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.534+0000 7f19f50e0700 1 -- 172.21.15.16:0/1375460199 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f19f00b3c80 con 0x7f19f01e5d40 2023-10-16T12:11:54.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.535+0000 7f19d77fe700 1 -- 172.21.15.16:0/1375460199 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v8) v1 ==== 80+0+6645 (secure 0 0 0) 0x7f19f00b3c80 con 0x7f19f01e5d40 2023-10-16T12:11:54.537 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:11:54.538 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-10-16T12:11:54.538 INFO:teuthology.orchestra.run.smithi016.stdout: "auth_dump": [ 2023-10-16T12:11:54.538 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:11:54.538 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.a", 2023-10-16T12:11:54.538 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOKC1l/rBtOhAA/ob5WUvpz8Glp4E+xPZOYA==", 2023-10-16T12:11:54.539 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:11:54.539 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T12:11:54.539 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-10-16T12:11:54.539 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-10-16T12:11:54.539 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:11:54.540 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:11:54.540 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:11:54.540 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:11:54.540 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.b", 2023-10-16T12:11:54.540 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1lKAVsCxAAvhXuLY6i1+Iz1yq5BVPMNQ==", 2023-10-16T12:11:54.541 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:11:54.541 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T12:11:54.541 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-10-16T12:11:54.541 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-10-16T12:11:54.541 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:11:54.542 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:11:54.542 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:11:54.542 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:11:54.542 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.c", 2023-10-16T12:11:54.542 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1lvBowBhAA3waJNYhdgs8PGCYqYNYUog==", 2023-10-16T12:11:54.543 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:11:54.543 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T12:11:54.543 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-10-16T12:11:54.543 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-10-16T12:11:54.545 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:11:54.545 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:11:54.545 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:11:54.545 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:11:54.545 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.d", 2023-10-16T12:11:54.546 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1lS/fnEhAAfNYgQ5BW86JMjYm/bBfnaQ==", 2023-10-16T12:11:54.546 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:11:54.546 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T12:11:54.546 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-10-16T12:11:54.546 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-10-16T12:11:54.547 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:11:54.547 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:11:54.547 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:11:54.547 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:11:54.547 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.0", 2023-10-16T12:11:54.547 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAQKC1lbxllOhAAZeLdp2GMl6HqQPaQjKlNqw==", 2023-10-16T12:11:54.548 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:11:54.548 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T12:11:54.548 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T12:11:54.548 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:11:54.548 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:11:54.549 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:11:54.549 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:11:54.549 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.1", 2023-10-16T12:11:54.549 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQATKC1lnU6KOhAAHxvlbmzBptf1keGgL+Rbkg==", 2023-10-16T12:11:54.549 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:11:54.550 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T12:11:54.550 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T12:11:54.550 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:11:54.550 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:11:54.550 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:11:54.551 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:11:54.551 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.2", 2023-10-16T12:11:54.551 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAXKC1l9jFyARAAr/NAwtWhBMut9sAAoDEGWg==", 2023-10-16T12:11:54.551 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:11:54.552 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T12:11:54.552 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T12:11:54.552 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:11:54.552 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:11:54.552 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:11:54.553 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:11:54.553 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.3", 2023-10-16T12:11:54.553 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAaKC1lbw/lARAAGJ/9DNb5/7kLFSq7vrLtXg==", 2023-10-16T12:11:54.553 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:11:54.553 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T12:11:54.553 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T12:11:54.554 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:11:54.554 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:11:54.554 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:11:54.554 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:11:54.554 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.4", 2023-10-16T12:11:54.555 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAeKC1lZb7jERAAYpvb1xcSli9CTq7L8ChLOA==", 2023-10-16T12:11:54.555 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:11:54.555 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T12:11:54.555 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T12:11:54.555 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:11:54.556 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:11:54.556 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:11:54.556 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:11:54.556 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.5", 2023-10-16T12:11:54.556 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAhKC1l/4HuEBAAqfnRqjknij9KES/fqiyIBQ==", 2023-10-16T12:11:54.557 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:11:54.557 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T12:11:54.557 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T12:11:54.557 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:11:54.557 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:11:54.558 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:11:54.558 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:11:54.558 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.6", 2023-10-16T12:11:54.558 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAkKC1lzoUkEBAA+Q82OiBMNag8+R316WQtUw==", 2023-10-16T12:11:54.558 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:11:54.559 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T12:11:54.559 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T12:11:54.559 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:11:54.559 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:11:54.559 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:11:54.560 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:11:54.560 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.7", 2023-10-16T12:11:54.560 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAnKC1ljUu4DxAAc+G4anJANC2GhLJ8wb6gYg==", 2023-10-16T12:11:54.560 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:11:54.560 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T12:11:54.561 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T12:11:54.561 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:11:54.561 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:11:54.561 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:11:54.561 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:11:54.561 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.0", 2023-10-16T12:11:54.562 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1l2l6PFxAA9QyXtBAHdGAXpwH0JfknuQ==", 2023-10-16T12:11:54.562 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:11:54.562 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T12:11:54.562 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-10-16T12:11:54.563 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-10-16T12:11:54.563 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-10-16T12:11:54.563 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:11:54.563 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:11:54.563 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:11:54.563 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.1", 2023-10-16T12:11:54.564 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1lAcHFIxAAcNt2bmv2R6j5L3DYkdw7eQ==", 2023-10-16T12:11:54.564 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:11:54.564 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T12:11:54.564 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-10-16T12:11:54.564 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-10-16T12:11:54.565 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-10-16T12:11:54.565 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:11:54.565 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:11:54.565 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:11:54.565 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.2", 2023-10-16T12:11:54.565 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1lA6wnGxAAdtaqYhtIJu8qENMrD8xvVA==", 2023-10-16T12:11:54.566 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:11:54.566 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T12:11:54.566 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-10-16T12:11:54.566 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-10-16T12:11:54.566 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-10-16T12:11:54.567 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:11:54.567 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:11:54.567 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:11:54.567 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.3", 2023-10-16T12:11:54.567 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1l/ddAJxAA3fNKKn9XkXas1BlDN+ggjw==", 2023-10-16T12:11:54.567 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:11:54.568 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T12:11:54.568 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-10-16T12:11:54.568 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-10-16T12:11:54.568 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-10-16T12:11:54.568 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:11:54.569 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:11:54.569 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:11:54.569 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.admin", 2023-10-16T12:11:54.569 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOKC1ld5jUHBAAc6esnCce3dSEZn3wVjhIlQ==", 2023-10-16T12:11:54.569 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:11:54.570 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2023-10-16T12:11:54.570 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-10-16T12:11:54.570 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-10-16T12:11:54.570 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:11:54.570 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:11:54.571 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:11:54.571 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:11:54.571 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.bootstrap-mds", 2023-10-16T12:11:54.571 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAtKC1lb51uEBAAGThaPw5ZXk9+gwfnsVLJJA==", 2023-10-16T12:11:54.571 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:11:54.572 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile bootstrap-mds" 2023-10-16T12:11:54.572 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:11:54.572 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:11:54.572 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:11:54.572 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.bootstrap-mgr", 2023-10-16T12:11:54.573 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAtKC1lxKhuEBAAgQiooUbCDkPnWFoi3qIM2A==", 2023-10-16T12:11:54.573 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:11:54.573 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile bootstrap-mgr" 2023-10-16T12:11:54.573 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:11:54.573 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:11:54.574 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:11:54.574 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.bootstrap-osd", 2023-10-16T12:11:54.574 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAtKC1lALFuEBAAkg2ova+RWzJw7onBUj3Hzg==", 2023-10-16T12:11:54.574 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:11:54.574 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile bootstrap-osd" 2023-10-16T12:11:54.575 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:11:54.575 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:11:54.575 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:11:54.575 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.bootstrap-rbd", 2023-10-16T12:11:54.575 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAtKC1lP7luEBAAux5UiRbo/8q3s4scU5u59w==", 2023-10-16T12:11:54.575 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:11:54.576 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile bootstrap-rbd" 2023-10-16T12:11:54.576 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:11:54.576 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:11:54.576 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:11:54.576 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.bootstrap-rbd-mirror", 2023-10-16T12:11:54.577 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAtKC1lYcFuEBAApzrbO9l71Mc8pUCQHOu/4Q==", 2023-10-16T12:11:54.577 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:11:54.577 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2023-10-16T12:11:54.577 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:11:54.577 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:11:54.578 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:11:54.578 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.bootstrap-rgw", 2023-10-16T12:11:54.578 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAtKC1lrMluEBAAvDIASHjILRtfR7U+K1gLLw==", 2023-10-16T12:11:54.578 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:11:54.578 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile bootstrap-rgw" 2023-10-16T12:11:54.578 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:11:54.579 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:11:54.579 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:11:54.579 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mgr.x", 2023-10-16T12:11:54.579 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOKC1l6b3vNhAAmj5xbfid2sZM2KfHmAf1Wg==", 2023-10-16T12:11:54.579 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:11:54.580 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2023-10-16T12:11:54.580 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile mgr", 2023-10-16T12:11:54.580 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:11:54.580 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:11:54.580 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:11:54.581 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:11:54.581 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mgr.y", 2023-10-16T12:11:54.581 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOKC1lLz2nMhAA8NqFYZ5nDKkzPBMrkwsVuQ==", 2023-10-16T12:11:54.581 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:11:54.581 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2023-10-16T12:11:54.582 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile mgr", 2023-10-16T12:11:54.582 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:11:54.582 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:11:54.582 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:11:54.582 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-10-16T12:11:54.583 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-10-16T12:11:54.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.537+0000 7f19f50e0700 1 -- 172.21.15.16:0/1375460199 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f19dc06af00 msgr2=0x7f19dc06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:54.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.537+0000 7f19f50e0700 1 --2- 172.21.15.16:0/1375460199 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f19dc06af00 0x7f19dc06d3c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f19e0000c00 tx=0x7f19e0009040 comp rx=0 tx=0).stop 2023-10-16T12:11:54.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.537+0000 7f19f50e0700 1 -- 172.21.15.16:0/1375460199 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19f01e5d40 msgr2=0x7f19f014e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:54.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.537+0000 7f19f50e0700 1 --2- 172.21.15.16:0/1375460199 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19f01e5d40 0x7f19f014e620 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f19f00722d0 tx=0x7f19e8011790 comp rx=0 tx=0).stop 2023-10-16T12:11:54.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.537+0000 7f19f50e0700 1 -- 172.21.15.16:0/1375460199 shutdown_connections 2023-10-16T12:11:54.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.537+0000 7f19f50e0700 1 --2- 172.21.15.16:0/1375460199 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f19dc06af00 0x7f19dc06d3c0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:54.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.537+0000 7f19f50e0700 1 --2- 172.21.15.16:0/1375460199 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19f01e5d40 0x7f19f014e620 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:54.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.537+0000 7f19f50e0700 1 --2- 172.21.15.16:0/1375460199 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f19d8004390 0x7f19f014c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:54.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.537+0000 7f19f50e0700 1 --2- 172.21.15.16:0/1375460199 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f19f01e5380 0x7f19f01e5800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:54.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.537+0000 7f19f50e0700 1 -- 172.21.15.16:0/1375460199 >> 172.21.15.16:0/1375460199 conn(0x7f19f00cc5b0 msgr2=0x7f19f0157eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:54.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.537+0000 7f19f50e0700 1 -- 172.21.15.16:0/1375460199 shutdown_connections 2023-10-16T12:11:54.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.537+0000 7f19f50e0700 1 -- 172.21.15.16:0/1375460199 wait complete. 2023-10-16T12:11:54.587 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2023-10-16T12:11:54.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.711+0000 7f4ec5f02700 1 Processor -- start 2023-10-16T12:11:54.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.711+0000 7f4ec5f02700 1 -- start start 2023-10-16T12:11:54.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.712+0000 7f4ec5f02700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4ec006aef0 0x7f4ec00b3480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:54.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.712+0000 7f4ec5f02700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4ec00b2be0 0x7f4ec00b39c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:54.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.712+0000 7f4ec5f02700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ec00d11c0 con 0x7f4ec015b860 2023-10-16T12:11:54.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.712+0000 7f4ec5f02700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ec00d03d0 con 0x7f4ec006aef0 2023-10-16T12:11:54.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.712+0000 7f4ec5f02700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ec00cd360 con 0x7f4ec00b2be0 2023-10-16T12:11:54.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.712+0000 7f4ebeffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4ec006aef0 0x7f4ec00b3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:54.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.712+0000 7f4ebeffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4ec006aef0 0x7f4ec00b3480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35344/0 (socket says 172.21.15.16:35344) 2023-10-16T12:11:54.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.712+0000 7f4ebeffd700 1 -- 172.21.15.16:0/3839483945 learned_addr learned my addr 172.21.15.16:0/3839483945 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:54.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.712+0000 7f4ebffff700 1 --2- 172.21.15.16:0/3839483945 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4ec00b2be0 0x7f4ec00b39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:54.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.712+0000 7f4ebeffd700 1 -- 172.21.15.16:0/3839483945 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4ec00b2be0 msgr2=0x7f4ec00b39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:54.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.712+0000 7f4ebeffd700 1 --2- 172.21.15.16:0/3839483945 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4ec00b2be0 0x7f4ec00b39c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:54.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.712+0000 7f4ebeffd700 1 -- 172.21.15.16:0/3839483945 >> v1:172.21.15.49:6789/0 conn(0x7f4ec015b860 legacy=0x7f4ec006a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:54.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.712+0000 7f4ebeffd700 1 -- 172.21.15.16:0/3839483945 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ec00d05b0 con 0x7f4ec006aef0 2023-10-16T12:11:54.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.713+0000 7f4ebeffd700 1 --2- 172.21.15.16:0/3839483945 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ec006aef0 0x7f4ec00b3480 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f4eb40098f0 tx=0x7f4eb4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12a85e470ce9ea50 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:54.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.713+0000 7f4ebe7fc700 1 -- 172.21.15.16:0/3839483945 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eb400b8a0 con 0x7f4ec006aef0 2023-10-16T12:11:54.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.713+0000 7f4ebe7fc700 1 -- 172.21.15.16:0/3839483945 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4eb400ba00 con 0x7f4ec006aef0 2023-10-16T12:11:54.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.713+0000 7f4ebe7fc700 1 -- 172.21.15.16:0/3839483945 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eb400bd90 con 0x7f4ec006aef0 2023-10-16T12:11:54.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.713+0000 7f4ec5f02700 1 -- 172.21.15.16:0/3839483945 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ec006aef0 msgr2=0x7f4ec00b3480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:54.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.713+0000 7f4ec5f02700 1 --2- 172.21.15.16:0/3839483945 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ec006aef0 0x7f4ec00b3480 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f4eb40098f0 tx=0x7f4eb4009c00 comp rx=0 tx=0).stop 2023-10-16T12:11:54.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.713+0000 7f4ec5f02700 1 -- 172.21.15.16:0/3839483945 shutdown_connections 2023-10-16T12:11:54.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.713+0000 7f4ec5f02700 1 --2- 172.21.15.16:0/3839483945 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4ec00b2be0 0x7f4ec00b39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:54.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.713+0000 7f4ec5f02700 1 --2- 172.21.15.16:0/3839483945 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ec006aef0 0x7f4ec00b3480 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:54.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.713+0000 7f4ec5f02700 1 -- 172.21.15.16:0/3839483945 >> 172.21.15.16:0/3839483945 conn(0x7f4ec00b1fd0 msgr2=0x7f4ec00b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:54.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.713+0000 7f4ec5f02700 1 -- 172.21.15.16:0/3839483945 shutdown_connections 2023-10-16T12:11:54.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.713+0000 7f4ec5f02700 1 -- 172.21.15.16:0/3839483945 wait complete. 2023-10-16T12:11:54.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.715+0000 7f4ec5f02700 1 Processor -- start 2023-10-16T12:11:54.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.715+0000 7f4ec5f02700 1 -- start start 2023-10-16T12:11:54.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.716+0000 7f4ec5f02700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ec006aef0 0x7f4ec00ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:54.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.716+0000 7f4ec5f02700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4ec00b2be0 0x7f4ec00c6830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:54.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.716+0000 7f4ec5f02700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4ec015b860 0x7f4ec01572d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:54.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.716+0000 7f4ec5f02700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ec00c6d70 con 0x7f4ec006aef0 2023-10-16T12:11:54.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.716+0000 7f4ec5f02700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ec00cf000 con 0x7f4ec015b860 2023-10-16T12:11:54.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.716+0000 7f4ec5f02700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ec00cf2e0 con 0x7f4ec00b2be0 2023-10-16T12:11:54.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.716+0000 7f4ebf7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ec006aef0 0x7f4ec00ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:54.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.716+0000 7f4ebf7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ec006aef0 0x7f4ec00ce9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35348/0 (socket says 172.21.15.16:35348) 2023-10-16T12:11:54.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.716+0000 7f4ebf7fe700 1 -- 172.21.15.16:0/123622633 learned_addr learned my addr 172.21.15.16:0/123622633 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:54.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.716+0000 7f4ebeffd700 1 --2- 172.21.15.16:0/123622633 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4ec00b2be0 0x7f4ec00c6830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:54.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.716+0000 7f4ebffff700 1 --2- 172.21.15.16:0/123622633 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4ec015b860 0x7f4ec01572d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:54.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.716+0000 7f4ebf7fe700 1 -- 172.21.15.16:0/123622633 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4ec00b2be0 msgr2=0x7f4ec00c6830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:54.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.716+0000 7f4ebf7fe700 1 --2- 172.21.15.16:0/123622633 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4ec00b2be0 0x7f4ec00c6830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:54.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.716+0000 7f4ebf7fe700 1 -- 172.21.15.16:0/123622633 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4ec015b860 msgr2=0x7f4ec01572d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:54.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.716+0000 7f4ebf7fe700 1 --2- 172.21.15.16:0/123622633 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4ec015b860 0x7f4ec01572d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:54.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.716+0000 7f4ebf7fe700 1 -- 172.21.15.16:0/123622633 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4eb4009550 con 0x7f4ec006aef0 2023-10-16T12:11:54.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.716+0000 7f4ebeffd700 1 --2- 172.21.15.16:0/123622633 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4ec00b2be0 0x7f4ec00c6830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:11:54.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.716+0000 7f4ebf7fe700 1 --2- 172.21.15.16:0/123622633 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ec006aef0 0x7f4ec00ce9f0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f4eb000c450 tx=0x7f4eb000c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:54.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.716+0000 7f4ea7fff700 1 -- 172.21.15.16:0/123622633 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eb00052d0 con 0x7f4ec006aef0 2023-10-16T12:11:54.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.716+0000 7f4ea7fff700 1 -- 172.21.15.16:0/123622633 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4eb0005430 con 0x7f4ec006aef0 2023-10-16T12:11:54.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.716+0000 7f4ec5f02700 1 -- 172.21.15.16:0/123622633 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ec00cf5c0 con 0x7f4ec006aef0 2023-10-16T12:11:54.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.716+0000 7f4ea7fff700 1 -- 172.21.15.16:0/123622633 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eb00038c0 con 0x7f4ec006aef0 2023-10-16T12:11:54.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.716+0000 7f4ec5f02700 1 -- 172.21.15.16:0/123622633 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ec00cfb80 con 0x7f4ec006aef0 2023-10-16T12:11:54.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.717+0000 7f4ec5f02700 1 -- 172.21.15.16:0/123622633 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ec004ebb0 con 0x7f4ec006aef0 2023-10-16T12:11:54.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.718+0000 7f4ea7fff700 1 -- 172.21.15.16:0/123622633 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f4eb0003a20 con 0x7f4ec006aef0 2023-10-16T12:11:54.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.718+0000 7f4ea7fff700 1 --2- 172.21.15.16:0/123622633 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f4eac06af00 0x7f4eac06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:54.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.718+0000 7f4ea7fff700 1 -- 172.21.15.16:0/123622633 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f4eb00887c0 con 0x7f4ec006aef0 2023-10-16T12:11:54.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.718+0000 7f4ebeffd700 1 --2- 172.21.15.16:0/123622633 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f4eac06af00 0x7f4eac06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:54.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.719+0000 7f4ebeffd700 1 --2- 172.21.15.16:0/123622633 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f4eac06af00 0x7f4eac06d3c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f4eb4009c40 tx=0x7f4eb4008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:54.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:54.723+0000 7f4ea7fff700 1 -- 172.21.15.16:0/123622633 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4eb00539e0 con 0x7f4ec006aef0 2023-10-16T12:11:55.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.014+0000 7f4ec5f02700 1 -- 172.21.15.16:0/123622633 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7f4ec00b6210 con 0x7f4ec006aef0 2023-10-16T12:11:55.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.016+0000 7f4ea7fff700 1 -- 172.21.15.16:0/123622633 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v9) v1 ==== 90+0+0 (secure 0 0 0) 0x7f4eb0057000 con 0x7f4ec006aef0 2023-10-16T12:11:55.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.018+0000 7f4ec5f02700 1 -- 172.21.15.16:0/123622633 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f4eac06af00 msgr2=0x7f4eac06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:55.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.018+0000 7f4ec5f02700 1 --2- 172.21.15.16:0/123622633 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f4eac06af00 0x7f4eac06d3c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f4eb4009c40 tx=0x7f4eb4008030 comp rx=0 tx=0).stop 2023-10-16T12:11:55.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.018+0000 7f4ec5f02700 1 -- 172.21.15.16:0/123622633 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ec006aef0 msgr2=0x7f4ec00ce9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:55.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.018+0000 7f4ec5f02700 1 --2- 172.21.15.16:0/123622633 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ec006aef0 0x7f4ec00ce9f0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f4eb000c450 tx=0x7f4eb000c810 comp rx=0 tx=0).stop 2023-10-16T12:11:55.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.018+0000 7f4ec5f02700 1 -- 172.21.15.16:0/123622633 shutdown_connections 2023-10-16T12:11:55.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.018+0000 7f4ec5f02700 1 --2- 172.21.15.16:0/123622633 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f4eac06af00 0x7f4eac06d3c0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:55.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.018+0000 7f4ec5f02700 1 --2- 172.21.15.16:0/123622633 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4ec015b860 0x7f4ec01572d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:55.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.018+0000 7f4ec5f02700 1 --2- 172.21.15.16:0/123622633 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ec006aef0 0x7f4ec00ce9f0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:55.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.018+0000 7f4ec5f02700 1 --2- 172.21.15.16:0/123622633 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4ec00b2be0 0x7f4ec00c6830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:55.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.018+0000 7f4ec5f02700 1 -- 172.21.15.16:0/123622633 >> 172.21.15.16:0/123622633 conn(0x7f4ec00b1fd0 msgr2=0x7f4ec00b4410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:55.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.018+0000 7f4ec5f02700 1 -- 172.21.15.16:0/123622633 shutdown_connections 2023-10-16T12:11:55.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.018+0000 7f4ec5f02700 1 -- 172.21.15.16:0/123622633 wait complete. 2023-10-16T12:11:55.032 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2023-10-16T12:11:55.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.160+0000 7f589293a700 1 Processor -- start 2023-10-16T12:11:55.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.160+0000 7f589293a700 1 -- start start 2023-10-16T12:11:55.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.161+0000 7f589293a700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f588c0ce870 0x7f588c0cb980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:55.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.161+0000 7f589293a700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f588c0cedb0 con 0x7f588c06aef0 2023-10-16T12:11:55.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.161+0000 7f589293a700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f588c0d12c0 con 0x7f588c15db40 2023-10-16T12:11:55.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.161+0000 7f589293a700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f588c15df20 con 0x7f588c0ce870 2023-10-16T12:11:55.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.161+0000 7f588b7fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f588c06aef0 0x7f588c0ce160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:52864/0 (socket says 172.21.15.16:52864) 2023-10-16T12:11:55.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.161+0000 7f588b7fe700 1 -- 172.21.15.16:0/1050748941 learned_addr learned my addr 172.21.15.16:0/1050748941 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:55.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.162+0000 7f5890ed7700 1 --2- 172.21.15.16:0/1050748941 >> v2:172.21.15.49:3300/0 conn(0x7f588c0ce870 0x7f588c0cb980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:55.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.162+0000 7f588affd700 1 -- 172.21.15.16:0/1050748941 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2313798973 0 0) 0x7f588c0cedb0 con 0x7f588c06aef0 2023-10-16T12:11:55.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.162+0000 7f588affd700 1 -- 172.21.15.16:0/1050748941 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58780035b0 con 0x7f588c06aef0 2023-10-16T12:11:55.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.162+0000 7f588affd700 1 -- 172.21.15.16:0/1050748941 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1839550954 0 0) 0x7f58780035b0 con 0x7f588c06aef0 2023-10-16T12:11:55.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.162+0000 7f588affd700 1 -- 172.21.15.16:0/1050748941 >> v2:172.21.15.49:3300/0 conn(0x7f588c0ce870 msgr2=0x7f588c0cb980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:55.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.162+0000 7f588affd700 1 --2- 172.21.15.16:0/1050748941 >> v2:172.21.15.49:3300/0 conn(0x7f588c0ce870 0x7f588c0cb980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:55.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.162+0000 7f588affd700 1 -- 172.21.15.16:0/1050748941 >> v1:172.21.15.49:6789/0 conn(0x7f588c15db40 legacy=0x7f588c06a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:55.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.162+0000 7f588affd700 1 -- 172.21.15.16:0/1050748941 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f588c0cabc0 con 0x7f588c06aef0 2023-10-16T12:11:55.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.162+0000 7f588affd700 1 -- 172.21.15.16:0/1050748941 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f5874001670 con 0x7f588c06aef0 2023-10-16T12:11:55.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.162+0000 7f588affd700 1 -- 172.21.15.16:0/1050748941 >> v1:172.21.15.16:6789/0 conn(0x7f588c06aef0 legacy=0x7f588c0ce160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:55.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.162+0000 7f588affd700 1 --2- 172.21.15.16:0/1050748941 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5878003a00 0x7f5878003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:55.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.162+0000 7f588affd700 1 --2- 172.21.15.16:0/1050748941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5878004850 0x7f5878004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:55.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.162+0000 7f588affd700 1 --2- 172.21.15.16:0/1050748941 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5878005330 0x7f587800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:55.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.162+0000 7f588affd700 1 -- 172.21.15.16:0/1050748941 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f588c15df20 con 0x7f5878004850 2023-10-16T12:11:55.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.162+0000 7f588affd700 1 -- 172.21.15.16:0/1050748941 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f587800c200 con 0x7f5878005330 2023-10-16T12:11:55.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.162+0000 7f588affd700 1 -- 172.21.15.16:0/1050748941 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f587800c3c0 con 0x7f5878003a00 2023-10-16T12:11:55.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.162+0000 7f588b7fe700 1 --2- 172.21.15.16:0/1050748941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5878004850 0x7f5878004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:55.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.162+0000 7f5890ed7700 1 --2- 172.21.15.16:0/1050748941 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5878005330 0x7f587800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:55.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.162+0000 7f588bfff700 1 --2- 172.21.15.16:0/1050748941 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5878003a00 0x7f5878003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:55.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.162+0000 7f588b7fe700 1 -- 172.21.15.16:0/1050748941 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5878003a00 msgr2=0x7f5878003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:55.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.162+0000 7f588b7fe700 1 --2- 172.21.15.16:0/1050748941 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5878003a00 0x7f5878003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:55.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.162+0000 7f588b7fe700 1 -- 172.21.15.16:0/1050748941 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5878005330 msgr2=0x7f587800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:55.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.162+0000 7f588b7fe700 1 --2- 172.21.15.16:0/1050748941 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5878005330 0x7f587800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:55.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.162+0000 7f588b7fe700 1 -- 172.21.15.16:0/1050748941 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58780035b0 con 0x7f5878004850 2023-10-16T12:11:55.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.162+0000 7f5890ed7700 1 --2- 172.21.15.16:0/1050748941 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5878005330 0x7f587800ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:11:55.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.162+0000 7f5890ed7700 1 -- 172.21.15.16:0/1050748941 reap_dead start 2023-10-16T12:11:55.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.162+0000 7f588b7fe700 1 --2- 172.21.15.16:0/1050748941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5878004850 0x7f5878004cd0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f587400c210 tx=0x7f5878005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f759bdaa6c3420 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:55.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.163+0000 7f588affd700 1 -- 172.21.15.16:0/1050748941 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5874007c10 con 0x7f5878004850 2023-10-16T12:11:55.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.163+0000 7f588affd700 1 -- 172.21.15.16:0/1050748941 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f587400cdd0 con 0x7f5878004850 2023-10-16T12:11:55.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.163+0000 7f588affd700 1 -- 172.21.15.16:0/1050748941 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5874005ba0 con 0x7f5878004850 2023-10-16T12:11:55.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.163+0000 7f589293a700 1 -- 172.21.15.16:0/1050748941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5878004850 msgr2=0x7f5878004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:55.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.163+0000 7f589293a700 1 --2- 172.21.15.16:0/1050748941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5878004850 0x7f5878004cd0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f587400c210 tx=0x7f5878005880 comp rx=0 tx=0).stop 2023-10-16T12:11:55.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.163+0000 7f589293a700 1 -- 172.21.15.16:0/1050748941 shutdown_connections 2023-10-16T12:11:55.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.163+0000 7f589293a700 1 --2- 172.21.15.16:0/1050748941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5878004850 0x7f5878004cd0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:55.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.163+0000 7f589293a700 1 -- 172.21.15.16:0/1050748941 >> 172.21.15.16:0/1050748941 conn(0x7f588c0b1fd0 msgr2=0x7f588c0b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:55.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.163+0000 7f589293a700 1 -- 172.21.15.16:0/1050748941 shutdown_connections 2023-10-16T12:11:55.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.163+0000 7f589293a700 1 -- 172.21.15.16:0/1050748941 wait complete. 2023-10-16T12:11:55.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.164+0000 7f589293a700 1 Processor -- start 2023-10-16T12:11:55.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.165+0000 7f589293a700 1 -- start start 2023-10-16T12:11:55.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.165+0000 7f589293a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5878004850 0x7f588c14fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:55.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.165+0000 7f589293a700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f588c158a40 0x7f588c0c8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:55.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.165+0000 7f589293a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f588c0c7510 0x7f588c0cd3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:55.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.165+0000 7f589293a700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f588c0c93f0 con 0x7f588c0c7510 2023-10-16T12:11:55.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.165+0000 7f589293a700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f588c0c7e10 con 0x7f588c158a40 2023-10-16T12:11:55.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.165+0000 7f589293a700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f588c0c80f0 con 0x7f5878004850 2023-10-16T12:11:55.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.165+0000 7f5890ed7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f588c0c7510 0x7f588c0cd3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:55.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.166+0000 7f5890ed7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f588c0c7510 0x7f588c0cd3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35374/0 (socket says 172.21.15.16:35374) 2023-10-16T12:11:55.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.166+0000 7f5890ed7700 1 -- 172.21.15.16:0/1286738532 learned_addr learned my addr 172.21.15.16:0/1286738532 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:55.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.166+0000 7f588bfff700 1 --2- 172.21.15.16:0/1286738532 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5878004850 0x7f588c14fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:55.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.166+0000 7f588b7fe700 1 --2- 172.21.15.16:0/1286738532 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f588c158a40 0x7f588c0c8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:55.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.166+0000 7f5890ed7700 1 -- 172.21.15.16:0/1286738532 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5878004850 msgr2=0x7f588c14fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:55.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.166+0000 7f5890ed7700 1 --2- 172.21.15.16:0/1286738532 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5878004850 0x7f588c14fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:55.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.166+0000 7f5890ed7700 1 -- 172.21.15.16:0/1286738532 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f588c158a40 msgr2=0x7f588c0c8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:55.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.166+0000 7f5890ed7700 1 --2- 172.21.15.16:0/1286738532 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f588c158a40 0x7f588c0c8eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:55.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.166+0000 7f5890ed7700 1 -- 172.21.15.16:0/1286738532 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f588c0c83d0 con 0x7f588c0c7510 2023-10-16T12:11:55.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.166+0000 7f5890ed7700 1 --2- 172.21.15.16:0/1286738532 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f588c0c7510 0x7f588c0cd3b0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f588c0aff10 tx=0x7f58840101c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:55.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.166+0000 7f5888ff9700 1 -- 172.21.15.16:0/1286738532 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f588401b070 con 0x7f588c0c7510 2023-10-16T12:11:55.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.166+0000 7f5888ff9700 1 -- 172.21.15.16:0/1286738532 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f588400f040 con 0x7f588c0c7510 2023-10-16T12:11:55.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.166+0000 7f5888ff9700 1 -- 172.21.15.16:0/1286738532 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f588400e050 con 0x7f588c0c7510 2023-10-16T12:11:55.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.166+0000 7f589293a700 1 -- 172.21.15.16:0/1286738532 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f588c0c86b0 con 0x7f588c0c7510 2023-10-16T12:11:55.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.166+0000 7f589293a700 1 -- 172.21.15.16:0/1286738532 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f588c0c79a0 con 0x7f588c0c7510 2023-10-16T12:11:55.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.167+0000 7f589293a700 1 -- 172.21.15.16:0/1286738532 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f588c04ebb0 con 0x7f588c0c7510 2023-10-16T12:11:55.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.168+0000 7f5888ff9700 1 -- 172.21.15.16:0/1286738532 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f5884010cf0 con 0x7f588c0c7510 2023-10-16T12:11:55.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.168+0000 7f5888ff9700 1 --2- 172.21.15.16:0/1286738532 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f586c06af00 0x7f586c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:55.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.168+0000 7f5888ff9700 1 -- 172.21.15.16:0/1286738532 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f5884089660 con 0x7f588c0c7510 2023-10-16T12:11:55.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.168+0000 7f588bfff700 1 --2- 172.21.15.16:0/1286738532 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f586c06af00 0x7f586c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:55.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.168+0000 7f588bfff700 1 --2- 172.21.15.16:0/1286738532 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f586c06af00 0x7f586c06d3c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f588c06aae0 tx=0x7f587c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:55.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.172+0000 7f5888ff9700 1 -- 172.21.15.16:0/1286738532 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5884054920 con 0x7f588c0c7510 2023-10-16T12:11:55.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.463+0000 7f589293a700 1 -- 172.21.15.16:0/1286738532 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7f588c0d0b00 con 0x7f588c0c7510 2023-10-16T12:11:55.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.465+0000 7f5888ff9700 1 -- 172.21.15.16:0/1286738532 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v10) v1 ==== 90+0+0 (secure 0 0 0) 0x7f5884057f40 con 0x7f588c0c7510 2023-10-16T12:11:55.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.467+0000 7f589293a700 1 -- 172.21.15.16:0/1286738532 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f586c06af00 msgr2=0x7f586c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:55.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.467+0000 7f589293a700 1 --2- 172.21.15.16:0/1286738532 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f586c06af00 0x7f586c06d3c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f588c06aae0 tx=0x7f587c009040 comp rx=0 tx=0).stop 2023-10-16T12:11:55.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.467+0000 7f589293a700 1 -- 172.21.15.16:0/1286738532 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f588c0c7510 msgr2=0x7f588c0cd3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:55.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.467+0000 7f589293a700 1 --2- 172.21.15.16:0/1286738532 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f588c0c7510 0x7f588c0cd3b0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f588c0aff10 tx=0x7f58840101c0 comp rx=0 tx=0).stop 2023-10-16T12:11:55.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.467+0000 7f589293a700 1 -- 172.21.15.16:0/1286738532 shutdown_connections 2023-10-16T12:11:55.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.467+0000 7f589293a700 1 --2- 172.21.15.16:0/1286738532 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f586c06af00 0x7f586c06d3c0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:55.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.467+0000 7f589293a700 1 --2- 172.21.15.16:0/1286738532 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f588c0c7510 0x7f588c0cd3b0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:55.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.467+0000 7f589293a700 1 --2- 172.21.15.16:0/1286738532 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5878004850 0x7f588c14fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:55.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.467+0000 7f589293a700 1 --2- 172.21.15.16:0/1286738532 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f588c158a40 0x7f588c0c8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:55.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.467+0000 7f589293a700 1 -- 172.21.15.16:0/1286738532 >> 172.21.15.16:0/1286738532 conn(0x7f588c0b1fd0 msgr2=0x7f588c0b14f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:55.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.468+0000 7f589293a700 1 -- 172.21.15.16:0/1286738532 shutdown_connections 2023-10-16T12:11:55.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.468+0000 7f589293a700 1 -- 172.21.15.16:0/1286738532 wait complete. 2023-10-16T12:11:55.482 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2023-10-16T12:11:55.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.607+0000 7f30a500b700 1 Processor -- start 2023-10-16T12:11:55.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.607+0000 7f30a500b700 1 -- start start 2023-10-16T12:11:55.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.608+0000 7f30a500b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f30a00d0ae0 0x7f30a00cef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:55.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.608+0000 7f30a500b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f30a00ce010 0x7f30a00cc1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:55.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.608+0000 7f30a500b700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30a0072a50 con 0x7f30a015a5c0 2023-10-16T12:11:55.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.608+0000 7f30a500b700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30a00cf440 con 0x7f30a00ce010 2023-10-16T12:11:55.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.608+0000 7f30a500b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30a00cba30 con 0x7f30a00d0ae0 2023-10-16T12:11:55.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.608+0000 7f309ed9d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f30a015a5c0 0x7f30a00d03d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:52884/0 (socket says 172.21.15.16:52884) 2023-10-16T12:11:55.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.608+0000 7f309ed9d700 1 -- 172.21.15.16:0/2268622630 learned_addr learned my addr 172.21.15.16:0/2268622630 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:55.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.608+0000 7f309f59e700 1 --2- 172.21.15.16:0/2268622630 >> v2:172.21.15.16:3300/0 conn(0x7f30a00ce010 0x7f30a00cc1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:55.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.608+0000 7f309e59c700 1 --2- 172.21.15.16:0/2268622630 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f30a00d0ae0 0x7f30a00cef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:55.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.608+0000 7f309dd9b700 1 -- 172.21.15.16:0/2268622630 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1573689630 0 0) 0x7f30a0072a50 con 0x7f30a015a5c0 2023-10-16T12:11:55.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.608+0000 7f309dd9b700 1 -- 172.21.15.16:0/2268622630 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30880035b0 con 0x7f30a015a5c0 2023-10-16T12:11:55.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.608+0000 7f309dd9b700 1 -- 172.21.15.16:0/2268622630 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 531736783 0 0) 0x7f30880035b0 con 0x7f30a015a5c0 2023-10-16T12:11:55.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.608+0000 7f309e59c700 1 -- 172.21.15.16:0/2268622630 >> v2:172.21.15.16:3300/0 conn(0x7f30a00ce010 msgr2=0x7f30a00cc1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:55.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.608+0000 7f309e59c700 1 --2- 172.21.15.16:0/2268622630 >> v2:172.21.15.16:3300/0 conn(0x7f30a00ce010 0x7f30a00cc1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:55.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.608+0000 7f309e59c700 1 -- 172.21.15.16:0/2268622630 >> v1:172.21.15.16:6789/0 conn(0x7f30a015a5c0 legacy=0x7f30a00d03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:55.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.608+0000 7f309e59c700 1 -- 172.21.15.16:0/2268622630 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30a00cf630 con 0x7f30a00d0ae0 2023-10-16T12:11:55.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.609+0000 7f309e59c700 1 --2- 172.21.15.16:0/2268622630 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f30a00d0ae0 0x7f30a00cef00 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3094009840 tx=0x7f3094009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=455cc2b904764239 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:55.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.609+0000 7f309dd9b700 1 -- 172.21.15.16:0/2268622630 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3090001670 con 0x7f30a00d0ae0 2023-10-16T12:11:55.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.609+0000 7f309dd9b700 1 -- 172.21.15.16:0/2268622630 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3094004030 con 0x7f30a00d0ae0 2023-10-16T12:11:55.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.609+0000 7f309dd9b700 1 -- 172.21.15.16:0/2268622630 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30940043c0 con 0x7f30a00d0ae0 2023-10-16T12:11:55.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.609+0000 7f30a500b700 1 -- 172.21.15.16:0/2268622630 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f30a00d0ae0 msgr2=0x7f30a00cef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:55.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.609+0000 7f30a500b700 1 --2- 172.21.15.16:0/2268622630 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f30a00d0ae0 0x7f30a00cef00 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3094009840 tx=0x7f3094009b50 comp rx=0 tx=0).stop 2023-10-16T12:11:55.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.609+0000 7f30a500b700 1 -- 172.21.15.16:0/2268622630 shutdown_connections 2023-10-16T12:11:55.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.609+0000 7f30a500b700 1 --2- 172.21.15.16:0/2268622630 >> v2:172.21.15.16:3300/0 conn(0x7f30a00ce010 0x7f30a00cc1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:55.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.609+0000 7f30a500b700 1 --2- 172.21.15.16:0/2268622630 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f30a00d0ae0 0x7f30a00cef00 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:55.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.609+0000 7f30a500b700 1 -- 172.21.15.16:0/2268622630 >> 172.21.15.16:0/2268622630 conn(0x7f30a00b1fd0 msgr2=0x7f30a00b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:55.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.609+0000 7f30a500b700 1 -- 172.21.15.16:0/2268622630 shutdown_connections 2023-10-16T12:11:55.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.610+0000 7f30a500b700 1 -- 172.21.15.16:0/2268622630 wait complete. 2023-10-16T12:11:55.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.611+0000 7f30a500b700 1 Processor -- start 2023-10-16T12:11:55.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.611+0000 7f30a500b700 1 -- start start 2023-10-16T12:11:55.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.612+0000 7f30a500b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30a00ce010 0x7f30a00c4210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:55.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.612+0000 7f30a500b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f30a00d0ae0 0x7f30a015c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:55.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.612+0000 7f30a500b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f30a015a5c0 0x7f30a015cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:55.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.612+0000 7f30a500b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30a01efdb0 con 0x7f30a00ce010 2023-10-16T12:11:55.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.612+0000 7f30a500b700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30a01f0370 con 0x7f30a015a5c0 2023-10-16T12:11:55.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.612+0000 7f30a500b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30a00c4b10 con 0x7f30a00d0ae0 2023-10-16T12:11:55.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.613+0000 7f309ed9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30a00ce010 0x7f30a00c4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:55.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.613+0000 7f309ed9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30a00ce010 0x7f30a00c4210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35384/0 (socket says 172.21.15.16:35384) 2023-10-16T12:11:55.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.613+0000 7f309ed9d700 1 -- 172.21.15.16:0/4179032870 learned_addr learned my addr 172.21.15.16:0/4179032870 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:55.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.613+0000 7f309e59c700 1 --2- 172.21.15.16:0/4179032870 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f30a00d0ae0 0x7f30a015c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:55.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.613+0000 7f309f59e700 1 --2- 172.21.15.16:0/4179032870 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f30a015a5c0 0x7f30a015cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:55.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.613+0000 7f309ed9d700 1 -- 172.21.15.16:0/4179032870 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f30a00d0ae0 msgr2=0x7f30a015c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:55.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.613+0000 7f309ed9d700 1 --2- 172.21.15.16:0/4179032870 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f30a00d0ae0 0x7f30a015c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:55.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.613+0000 7f309ed9d700 1 -- 172.21.15.16:0/4179032870 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f30a015a5c0 msgr2=0x7f30a015cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:55.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.613+0000 7f309ed9d700 1 --2- 172.21.15.16:0/4179032870 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f30a015a5c0 0x7f30a015cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:55.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.613+0000 7f309ed9d700 1 -- 172.21.15.16:0/4179032870 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30940094a0 con 0x7f30a00ce010 2023-10-16T12:11:55.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.613+0000 7f309ed9d700 1 --2- 172.21.15.16:0/4179032870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30a00ce010 0x7f30a00c4210 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f309000c750 tx=0x7f309000cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:55.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.613+0000 7f30877fe700 1 -- 172.21.15.16:0/4179032870 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3090004c80 con 0x7f30a00ce010 2023-10-16T12:11:55.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.613+0000 7f30877fe700 1 -- 172.21.15.16:0/4179032870 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3090004de0 con 0x7f30a00ce010 2023-10-16T12:11:55.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.613+0000 7f30a500b700 1 -- 172.21.15.16:0/4179032870 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30a00c4df0 con 0x7f30a00ce010 2023-10-16T12:11:55.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.613+0000 7f30877fe700 1 -- 172.21.15.16:0/4179032870 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30900072f0 con 0x7f30a00ce010 2023-10-16T12:11:55.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.613+0000 7f30a500b700 1 -- 172.21.15.16:0/4179032870 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30a00c4750 con 0x7f30a00ce010 2023-10-16T12:11:55.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.614+0000 7f30a500b700 1 -- 172.21.15.16:0/4179032870 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30a00c5030 con 0x7f30a00ce010 2023-10-16T12:11:55.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.615+0000 7f30877fe700 1 -- 172.21.15.16:0/4179032870 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f3090007450 con 0x7f30a00ce010 2023-10-16T12:11:55.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.615+0000 7f30877fe700 1 --2- 172.21.15.16:0/4179032870 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f308c06af00 0x7f308c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:55.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.615+0000 7f30877fe700 1 -- 172.21.15.16:0/4179032870 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f3090089230 con 0x7f30a00ce010 2023-10-16T12:11:55.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.615+0000 7f309e59c700 1 --2- 172.21.15.16:0/4179032870 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f308c06af00 0x7f308c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:55.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.616+0000 7f309e59c700 1 --2- 172.21.15.16:0/4179032870 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f308c06af00 0x7f308c06d3c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f309400a010 tx=0x7f3094013040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:55.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.620+0000 7f30877fe700 1 -- 172.21.15.16:0/4179032870 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f30900544f0 con 0x7f30a00ce010 2023-10-16T12:11:55.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.907+0000 7f30a500b700 1 -- 172.21.15.16:0/4179032870 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7f30a004ebb0 con 0x7f30a00ce010 2023-10-16T12:11:55.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.909+0000 7f30877fe700 1 -- 172.21.15.16:0/4179032870 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v11) v1 ==== 90+0+0 (secure 0 0 0) 0x7f3090007700 con 0x7f30a00ce010 2023-10-16T12:11:55.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.911+0000 7f30a500b700 1 -- 172.21.15.16:0/4179032870 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f308c06af00 msgr2=0x7f308c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:55.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.911+0000 7f30a500b700 1 --2- 172.21.15.16:0/4179032870 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f308c06af00 0x7f308c06d3c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f309400a010 tx=0x7f3094013040 comp rx=0 tx=0).stop 2023-10-16T12:11:55.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.911+0000 7f30a500b700 1 -- 172.21.15.16:0/4179032870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30a00ce010 msgr2=0x7f30a00c4210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:55.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.911+0000 7f30a500b700 1 --2- 172.21.15.16:0/4179032870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30a00ce010 0x7f30a00c4210 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f309000c750 tx=0x7f309000cb10 comp rx=0 tx=0).stop 2023-10-16T12:11:55.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.912+0000 7f30a500b700 1 -- 172.21.15.16:0/4179032870 shutdown_connections 2023-10-16T12:11:55.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.912+0000 7f30a500b700 1 --2- 172.21.15.16:0/4179032870 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f308c06af00 0x7f308c06d3c0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:55.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.912+0000 7f30a500b700 1 --2- 172.21.15.16:0/4179032870 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f30a015a5c0 0x7f30a015cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:55.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.912+0000 7f30a500b700 1 --2- 172.21.15.16:0/4179032870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30a00ce010 0x7f30a00c4210 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:55.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.912+0000 7f30a500b700 1 --2- 172.21.15.16:0/4179032870 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f30a00d0ae0 0x7f30a015c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:55.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.912+0000 7f30a500b700 1 -- 172.21.15.16:0/4179032870 >> 172.21.15.16:0/4179032870 conn(0x7f30a00b1fd0 msgr2=0x7f30a0158c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:55.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.912+0000 7f30a500b700 1 -- 172.21.15.16:0/4179032870 shutdown_connections 2023-10-16T12:11:55.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:55.912+0000 7f30a500b700 1 -- 172.21.15.16:0/4179032870 wait complete. 2023-10-16T12:11:55.925 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2023-10-16T12:11:56.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.049+0000 7f1f8030d700 1 Processor -- start 2023-10-16T12:11:56.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.049+0000 7f1f8030d700 1 -- start start 2023-10-16T12:11:56.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.050+0000 7f1f8030d700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f1f7815b860 0x7f1f7806a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:56.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.050+0000 7f1f8030d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1f780b2be0 0x7f1f7806ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:56.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.050+0000 7f1f8030d700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f780d11c0 con 0x7f1f780cd770 2023-10-16T12:11:56.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.050+0000 7f1f8030d700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f780d03d0 con 0x7f1f7815b860 2023-10-16T12:11:56.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.050+0000 7f1f8030d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f780cd360 con 0x7f1f780b2be0 2023-10-16T12:11:56.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.051+0000 7f1f7e8aa700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f1f780cd770 0x7f1f780719e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:52898/0 (socket says 172.21.15.16:52898) 2023-10-16T12:11:56.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.051+0000 7f1f7e8aa700 1 -- 172.21.15.16:0/3740890243 learned_addr learned my addr 172.21.15.16:0/3740890243 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:56.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.051+0000 7f1f7d0a7700 1 -- 172.21.15.16:0/3740890243 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4023258218 0 0) 0x7f1f780d11c0 con 0x7f1f780cd770 2023-10-16T12:11:56.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.051+0000 7f1f7d0a7700 1 -- 172.21.15.16:0/3740890243 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f640035b0 con 0x7f1f780cd770 2023-10-16T12:11:56.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.051+0000 7f1f7e0a9700 1 --2- 172.21.15.16:0/3740890243 >> v2:172.21.15.49:3300/0 conn(0x7f1f7815b860 0x7f1f7806a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:56.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.051+0000 7f1f7d8a8700 1 --2- 172.21.15.16:0/3740890243 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1f780b2be0 0x7f1f7806ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:56.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.051+0000 7f1f7d0a7700 1 -- 172.21.15.16:0/3740890243 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2560563464 0 0) 0x7f1f640035b0 con 0x7f1f780cd770 2023-10-16T12:11:56.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.051+0000 7f1f7d0a7700 1 -- 172.21.15.16:0/3740890243 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1f780b2be0 msgr2=0x7f1f7806ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:56.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.051+0000 7f1f7d0a7700 1 --2- 172.21.15.16:0/3740890243 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1f780b2be0 0x7f1f7806ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:56.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.051+0000 7f1f7d0a7700 1 -- 172.21.15.16:0/3740890243 >> v2:172.21.15.49:3300/0 conn(0x7f1f7815b860 msgr2=0x7f1f7806a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:56.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.051+0000 7f1f7d0a7700 1 --2- 172.21.15.16:0/3740890243 >> v2:172.21.15.49:3300/0 conn(0x7f1f7815b860 0x7f1f7806a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:56.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.051+0000 7f1f7d0a7700 1 -- 172.21.15.16:0/3740890243 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f780d05b0 con 0x7f1f780cd770 2023-10-16T12:11:56.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.051+0000 7f1f7d0a7700 1 -- 172.21.15.16:0/3740890243 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f1f700019f0 con 0x7f1f780cd770 2023-10-16T12:11:56.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.051+0000 7f1f7d0a7700 1 -- 172.21.15.16:0/3740890243 >> v1:172.21.15.16:6789/0 conn(0x7f1f780cd770 legacy=0x7f1f780719e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:56.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.051+0000 7f1f7d0a7700 1 --2- 172.21.15.16:0/3740890243 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1f64003a00 0x7f1f64003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:56.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.051+0000 7f1f7d0a7700 1 --2- 172.21.15.16:0/3740890243 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1f780b2be0 0x7f1f64004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:56.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.051+0000 7f1f7d0a7700 1 --2- 172.21.15.16:0/3740890243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f64004d90 0x7f1f64005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:56.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.051+0000 7f1f7d0a7700 1 -- 172.21.15.16:0/3740890243 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f780d03d0 con 0x7f1f64004d90 2023-10-16T12:11:56.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.051+0000 7f1f7d0a7700 1 -- 172.21.15.16:0/3740890243 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f780cd360 con 0x7f1f64003a00 2023-10-16T12:11:56.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.051+0000 7f1f7d0a7700 1 -- 172.21.15.16:0/3740890243 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f6400bac0 con 0x7f1f780b2be0 2023-10-16T12:11:56.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.051+0000 7f1f7e8aa700 1 --2- 172.21.15.16:0/3740890243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f64004d90 0x7f1f64005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:56.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.051+0000 7f1f7d8a8700 1 --2- 172.21.15.16:0/3740890243 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1f780b2be0 0x7f1f64004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:56.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.051+0000 7f1f7e0a9700 1 --2- 172.21.15.16:0/3740890243 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1f64003a00 0x7f1f64003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:56.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.052+0000 7f1f7e8aa700 1 -- 172.21.15.16:0/3740890243 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1f780b2be0 msgr2=0x7f1f64004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:56.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.052+0000 7f1f7e8aa700 1 --2- 172.21.15.16:0/3740890243 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1f780b2be0 0x7f1f64004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:56.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.052+0000 7f1f7e8aa700 1 -- 172.21.15.16:0/3740890243 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1f64003a00 msgr2=0x7f1f64003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:56.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.052+0000 7f1f7e8aa700 1 --2- 172.21.15.16:0/3740890243 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1f64003a00 0x7f1f64003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:56.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.052+0000 7f1f7e8aa700 1 -- 172.21.15.16:0/3740890243 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f640035b0 con 0x7f1f64004d90 2023-10-16T12:11:56.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.052+0000 7f1f7e8aa700 1 --2- 172.21.15.16:0/3740890243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f64004d90 0x7f1f64005210 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f1f7000a900 tx=0x7f1f64005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20f4d4a2cf954393 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:56.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.052+0000 7f1f7d0a7700 1 -- 172.21.15.16:0/3740890243 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f7000a340 con 0x7f1f64004d90 2023-10-16T12:11:56.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.052+0000 7f1f7d0a7700 1 -- 172.21.15.16:0/3740890243 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f70012620 con 0x7f1f64004d90 2023-10-16T12:11:56.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.052+0000 7f1f7d0a7700 1 -- 172.21.15.16:0/3740890243 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f700129d0 con 0x7f1f64004d90 2023-10-16T12:11:56.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.053+0000 7f1f8030d700 1 -- 172.21.15.16:0/3740890243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f64004d90 msgr2=0x7f1f64005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:56.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.053+0000 7f1f8030d700 1 --2- 172.21.15.16:0/3740890243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f64004d90 0x7f1f64005210 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f1f7000a900 tx=0x7f1f64005880 comp rx=0 tx=0).stop 2023-10-16T12:11:56.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.053+0000 7f1f7e8aa700 1 -- 172.21.15.16:0/3740890243 reap_dead start 2023-10-16T12:11:56.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.053+0000 7f1f8030d700 1 -- 172.21.15.16:0/3740890243 shutdown_connections 2023-10-16T12:11:56.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.053+0000 7f1f8030d700 1 -- 172.21.15.16:0/3740890243 >> 172.21.15.16:0/3740890243 conn(0x7f1f780b1fd0 msgr2=0x7f1f780b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:56.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.053+0000 7f1f8030d700 1 -- 172.21.15.16:0/3740890243 shutdown_connections 2023-10-16T12:11:56.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.053+0000 7f1f8030d700 1 -- 172.21.15.16:0/3740890243 wait complete. 2023-10-16T12:11:56.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.054+0000 7f1f8030d700 1 Processor -- start 2023-10-16T12:11:56.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.054+0000 7f1f8030d700 1 -- start start 2023-10-16T12:11:56.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.055+0000 7f1f8030d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1f64004d90 0x7f1f7815d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:56.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.055+0000 7f1f8030d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1f7815bc40 0x7f1f78158200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:56.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.055+0000 7f1f8030d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f7815e100 0x7f1f78158740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:56.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.055+0000 7f1f8030d700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f7815ddb0 con 0x7f1f7815e100 2023-10-16T12:11:56.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.055+0000 7f1f8030d700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f780ce010 con 0x7f1f64004d90 2023-10-16T12:11:56.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.055+0000 7f1f8030d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f780ce2f0 con 0x7f1f7815bc40 2023-10-16T12:11:56.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.055+0000 7f1f7e8aa700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f7815e100 0x7f1f78158740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:56.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.055+0000 7f1f7e8aa700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f7815e100 0x7f1f78158740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35408/0 (socket says 172.21.15.16:35408) 2023-10-16T12:11:56.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.055+0000 7f1f7e8aa700 1 -- 172.21.15.16:0/2199434104 learned_addr learned my addr 172.21.15.16:0/2199434104 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:56.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.055+0000 7f1f7d8a8700 1 --2- 172.21.15.16:0/2199434104 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1f7815bc40 0x7f1f78158200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:56.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.055+0000 7f1f7e0a9700 1 --2- 172.21.15.16:0/2199434104 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1f64004d90 0x7f1f7815d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:56.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.055+0000 7f1f7e8aa700 1 -- 172.21.15.16:0/2199434104 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1f7815bc40 msgr2=0x7f1f78158200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:56.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.055+0000 7f1f7e8aa700 1 --2- 172.21.15.16:0/2199434104 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1f7815bc40 0x7f1f78158200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:56.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.055+0000 7f1f7e8aa700 1 -- 172.21.15.16:0/2199434104 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1f64004d90 msgr2=0x7f1f7815d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:56.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.055+0000 7f1f7e8aa700 1 --2- 172.21.15.16:0/2199434104 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1f64004d90 0x7f1f7815d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:56.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.055+0000 7f1f7e8aa700 1 -- 172.21.15.16:0/2199434104 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f780ce5d0 con 0x7f1f7815e100 2023-10-16T12:11:56.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.055+0000 7f1f7e8aa700 1 --2- 172.21.15.16:0/2199434104 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f7815e100 0x7f1f78158740 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f1f70012000 tx=0x7f1f70001360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:56.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.056+0000 7f1f6effd700 1 -- 172.21.15.16:0/2199434104 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f7000a340 con 0x7f1f7815e100 2023-10-16T12:11:56.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.056+0000 7f1f6effd700 1 -- 172.21.15.16:0/2199434104 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f700019b0 con 0x7f1f7815e100 2023-10-16T12:11:56.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.056+0000 7f1f8030d700 1 -- 172.21.15.16:0/2199434104 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f780ce8b0 con 0x7f1f7815e100 2023-10-16T12:11:56.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.056+0000 7f1f6effd700 1 -- 172.21.15.16:0/2199434104 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f7001e640 con 0x7f1f7815e100 2023-10-16T12:11:56.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.056+0000 7f1f8030d700 1 -- 172.21.15.16:0/2199434104 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f780adaf0 con 0x7f1f7815e100 2023-10-16T12:11:56.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.057+0000 7f1f8030d700 1 -- 172.21.15.16:0/2199434104 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f780cedd0 con 0x7f1f7815e100 2023-10-16T12:11:56.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.057+0000 7f1f6effd700 1 -- 172.21.15.16:0/2199434104 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f1f70019030 con 0x7f1f7815e100 2023-10-16T12:11:56.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.057+0000 7f1f6effd700 1 --2- 172.21.15.16:0/2199434104 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f1f5c06af00 0x7f1f5c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:56.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.058+0000 7f1f6effd700 1 -- 172.21.15.16:0/2199434104 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f1f7008af60 con 0x7f1f7815e100 2023-10-16T12:11:56.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.058+0000 7f1f7e0a9700 1 --2- 172.21.15.16:0/2199434104 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f1f5c06af00 0x7f1f5c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:56.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.058+0000 7f1f7e0a9700 1 --2- 172.21.15.16:0/2199434104 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f1f5c06af00 0x7f1f5c06d3c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f1f68009a80 tx=0x7f1f68008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:56.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.062+0000 7f1f6effd700 1 -- 172.21.15.16:0/2199434104 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1f780cedd0 con 0x7f1f7815e100 2023-10-16T12:11:56.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.352+0000 7f1f8030d700 1 -- 172.21.15.16:0/2199434104 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7f1f780cedd0 con 0x7f1f7815e100 2023-10-16T12:11:56.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.354+0000 7f1f6effd700 1 -- 172.21.15.16:0/2199434104 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v12) v1 ==== 90+0+0 (secure 0 0 0) 0x7f1f780cedd0 con 0x7f1f7815e100 2023-10-16T12:11:56.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.356+0000 7f1f8030d700 1 -- 172.21.15.16:0/2199434104 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f1f5c06af00 msgr2=0x7f1f5c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:56.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.356+0000 7f1f8030d700 1 --2- 172.21.15.16:0/2199434104 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f1f5c06af00 0x7f1f5c06d3c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f1f68009a80 tx=0x7f1f68008040 comp rx=0 tx=0).stop 2023-10-16T12:11:56.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.356+0000 7f1f8030d700 1 -- 172.21.15.16:0/2199434104 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f7815e100 msgr2=0x7f1f78158740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:56.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.356+0000 7f1f8030d700 1 --2- 172.21.15.16:0/2199434104 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f7815e100 0x7f1f78158740 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f1f70012000 tx=0x7f1f70001360 comp rx=0 tx=0).stop 2023-10-16T12:11:56.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.356+0000 7f1f8030d700 1 -- 172.21.15.16:0/2199434104 shutdown_connections 2023-10-16T12:11:56.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.356+0000 7f1f8030d700 1 --2- 172.21.15.16:0/2199434104 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f1f5c06af00 0x7f1f5c06d3c0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:56.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.356+0000 7f1f8030d700 1 --2- 172.21.15.16:0/2199434104 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f7815e100 0x7f1f78158740 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:56.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.356+0000 7f1f8030d700 1 --2- 172.21.15.16:0/2199434104 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1f64004d90 0x7f1f7815d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:56.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.356+0000 7f1f8030d700 1 --2- 172.21.15.16:0/2199434104 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1f7815bc40 0x7f1f78158200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:56.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.356+0000 7f1f8030d700 1 -- 172.21.15.16:0/2199434104 >> 172.21.15.16:0/2199434104 conn(0x7f1f780b1fd0 msgr2=0x7f1f780b3380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:56.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.356+0000 7f1f8030d700 1 -- 172.21.15.16:0/2199434104 shutdown_connections 2023-10-16T12:11:56.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.356+0000 7f1f8030d700 1 -- 172.21.15.16:0/2199434104 wait complete. 2023-10-16T12:11:56.370 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2023-10-16T12:11:56.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.495+0000 7fe5461ee700 1 Processor -- start 2023-10-16T12:11:56.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.495+0000 7fe5461ee700 1 -- start start 2023-10-16T12:11:56.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.495+0000 7fe5461ee700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe54015db40 0x7fe5400cb980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:56.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.496+0000 7fe5461ee700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fe54006a7e0 0x7fe54006ac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:56.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.496+0000 7fe5461ee700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5400cedb0 con 0x7fe5400ce160 2023-10-16T12:11:56.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.496+0000 7fe5461ee700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe54006b1a0 con 0x7fe54015db40 2023-10-16T12:11:56.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.496+0000 7fe5461ee700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe54015df20 con 0x7fe54006a7e0 2023-10-16T12:11:56.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.496+0000 7fe53f7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe54015db40 0x7fe5400cb980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:56.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.496+0000 7fe53f7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe54015db40 0x7fe5400cb980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35412/0 (socket says 172.21.15.16:35412) 2023-10-16T12:11:56.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.496+0000 7fe53f7fe700 1 -- 172.21.15.16:0/723108658 learned_addr learned my addr 172.21.15.16:0/723108658 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:56.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.496+0000 7fe53ffff700 1 --1- 172.21.15.16:0/723108658 >> v1:172.21.15.16:6789/0 conn(0x7fe5400ce160 0x7fe5400ce5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:52922/0 (socket says 172.21.15.16:52922) 2023-10-16T12:11:56.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.496+0000 7fe53effd700 1 --2- 172.21.15.16:0/723108658 >> v2:172.21.15.49:3300/0 conn(0x7fe54006a7e0 0x7fe54006ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:56.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.496+0000 7fe53f7fe700 1 -- 172.21.15.16:0/723108658 >> v2:172.21.15.49:3300/0 conn(0x7fe54006a7e0 msgr2=0x7fe54006ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:56.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.496+0000 7fe53f7fe700 1 --2- 172.21.15.16:0/723108658 >> v2:172.21.15.49:3300/0 conn(0x7fe54006a7e0 0x7fe54006ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:56.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.496+0000 7fe53f7fe700 1 -- 172.21.15.16:0/723108658 >> v1:172.21.15.16:6789/0 conn(0x7fe5400ce160 legacy=0x7fe5400ce5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:56.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.496+0000 7fe53e7fc700 1 -- 172.21.15.16:0/723108658 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3991977649 0 0) 0x7fe5400cedb0 con 0x7fe5400ce160 2023-10-16T12:11:56.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.496+0000 7fe53f7fe700 1 -- 172.21.15.16:0/723108658 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5400d12c0 con 0x7fe54015db40 2023-10-16T12:11:56.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.496+0000 7fe53f7fe700 1 --2- 172.21.15.16:0/723108658 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe54015db40 0x7fe5400cb980 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fe540067660 tx=0x7fe530009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5941afc469efbe74 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:56.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.496+0000 7fe53e7fc700 1 -- 172.21.15.16:0/723108658 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe53000b870 con 0x7fe54015db40 2023-10-16T12:11:56.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.496+0000 7fe53e7fc700 1 -- 172.21.15.16:0/723108658 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe53000b9d0 con 0x7fe54015db40 2023-10-16T12:11:56.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.496+0000 7fe53e7fc700 1 -- 172.21.15.16:0/723108658 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe53000bd60 con 0x7fe54015db40 2023-10-16T12:11:56.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.497+0000 7fe5461ee700 1 -- 172.21.15.16:0/723108658 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe54015db40 msgr2=0x7fe5400cb980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:56.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.497+0000 7fe5461ee700 1 --2- 172.21.15.16:0/723108658 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe54015db40 0x7fe5400cb980 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fe540067660 tx=0x7fe530009bd0 comp rx=0 tx=0).stop 2023-10-16T12:11:56.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.497+0000 7fe5461ee700 1 -- 172.21.15.16:0/723108658 shutdown_connections 2023-10-16T12:11:56.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.497+0000 7fe5461ee700 1 --2- 172.21.15.16:0/723108658 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe54015db40 0x7fe5400cb980 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:56.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.497+0000 7fe5461ee700 1 --2- 172.21.15.16:0/723108658 >> v2:172.21.15.49:3300/0 conn(0x7fe54006a7e0 0x7fe54006ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:56.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.497+0000 7fe5461ee700 1 -- 172.21.15.16:0/723108658 >> 172.21.15.16:0/723108658 conn(0x7fe5400b1fd0 msgr2=0x7fe5400b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:56.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.497+0000 7fe5461ee700 1 -- 172.21.15.16:0/723108658 shutdown_connections 2023-10-16T12:11:56.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.497+0000 7fe5461ee700 1 -- 172.21.15.16:0/723108658 wait complete. 2023-10-16T12:11:56.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.498+0000 7fe5461ee700 1 Processor -- start 2023-10-16T12:11:56.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.498+0000 7fe5461ee700 1 -- start start 2023-10-16T12:11:56.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.499+0000 7fe5461ee700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe54006a7e0 0x7fe540156cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:56.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.499+0000 7fe5461ee700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe5400ce160 0x7fe54015e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:56.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.499+0000 7fe5461ee700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe54015db40 0x7fe5400c9520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:56.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.499+0000 7fe5461ee700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5400c8030 con 0x7fe54006a7e0 2023-10-16T12:11:56.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.499+0000 7fe5461ee700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5400c64c0 con 0x7fe54015db40 2023-10-16T12:11:56.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.499+0000 7fe5461ee700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5400c7690 con 0x7fe5400ce160 2023-10-16T12:11:56.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.499+0000 7fe53f7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe54006a7e0 0x7fe540156cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:56.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.499+0000 7fe53f7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe54006a7e0 0x7fe540156cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35418/0 (socket says 172.21.15.16:35418) 2023-10-16T12:11:56.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.499+0000 7fe53f7fe700 1 -- 172.21.15.16:0/2000667882 learned_addr learned my addr 172.21.15.16:0/2000667882 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:56.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.499+0000 7fe53ffff700 1 --2- 172.21.15.16:0/2000667882 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe54015db40 0x7fe5400c9520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:56.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.499+0000 7fe53effd700 1 --2- 172.21.15.16:0/2000667882 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe5400ce160 0x7fe54015e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:56.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.499+0000 7fe53f7fe700 1 -- 172.21.15.16:0/2000667882 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe5400ce160 msgr2=0x7fe54015e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:56.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.499+0000 7fe53f7fe700 1 --2- 172.21.15.16:0/2000667882 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe5400ce160 0x7fe54015e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:56.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.499+0000 7fe53f7fe700 1 -- 172.21.15.16:0/2000667882 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe54015db40 msgr2=0x7fe5400c9520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:56.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.499+0000 7fe53f7fe700 1 --2- 172.21.15.16:0/2000667882 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe54015db40 0x7fe5400c9520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:56.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.500+0000 7fe53f7fe700 1 -- 172.21.15.16:0/2000667882 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe530009550 con 0x7fe54006a7e0 2023-10-16T12:11:56.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.500+0000 7fe53f7fe700 1 --2- 172.21.15.16:0/2000667882 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe54006a7e0 0x7fe540156cd0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fe530009c10 tx=0x7fe530004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:56.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.500+0000 7fe527fff700 1 -- 172.21.15.16:0/2000667882 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe530004310 con 0x7fe54006a7e0 2023-10-16T12:11:56.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.500+0000 7fe527fff700 1 -- 172.21.15.16:0/2000667882 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe53001f070 con 0x7fe54006a7e0 2023-10-16T12:11:56.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.500+0000 7fe527fff700 1 -- 172.21.15.16:0/2000667882 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5300036a0 con 0x7fe54006a7e0 2023-10-16T12:11:56.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.500+0000 7fe5461ee700 1 -- 172.21.15.16:0/2000667882 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5400c7970 con 0x7fe54006a7e0 2023-10-16T12:11:56.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.500+0000 7fe5461ee700 1 -- 172.21.15.16:0/2000667882 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe540157210 con 0x7fe54006a7e0 2023-10-16T12:11:56.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.501+0000 7fe5461ee700 1 -- 172.21.15.16:0/2000667882 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5400caba0 con 0x7fe54006a7e0 2023-10-16T12:11:56.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.501+0000 7fe527fff700 1 -- 172.21.15.16:0/2000667882 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fe530021030 con 0x7fe54006a7e0 2023-10-16T12:11:56.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.502+0000 7fe527fff700 1 --2- 172.21.15.16:0/2000667882 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fe52c06af00 0x7fe52c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:56.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.502+0000 7fe527fff700 1 -- 172.21.15.16:0/2000667882 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fe53008a560 con 0x7fe54006a7e0 2023-10-16T12:11:56.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.502+0000 7fe53effd700 1 --2- 172.21.15.16:0/2000667882 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fe52c06af00 0x7fe52c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:56.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.502+0000 7fe53effd700 1 --2- 172.21.15.16:0/2000667882 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fe52c06af00 0x7fe52c06d3c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe54015bb80 tx=0x7fe534008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:56.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.506+0000 7fe527fff700 1 -- 172.21.15.16:0/2000667882 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe5400caba0 con 0x7fe54006a7e0 2023-10-16T12:11:56.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.792+0000 7fe5461ee700 1 -- 172.21.15.16:0/2000667882 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7fe5400c8170 con 0x7fe54006a7e0 2023-10-16T12:11:56.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.794+0000 7fe527fff700 1 -- 172.21.15.16:0/2000667882 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v13) v1 ==== 97+0+0 (secure 0 0 0) 0x7fe5400c8170 con 0x7fe54006a7e0 2023-10-16T12:11:56.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.796+0000 7fe5461ee700 1 -- 172.21.15.16:0/2000667882 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fe52c06af00 msgr2=0x7fe52c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:56.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.796+0000 7fe5461ee700 1 --2- 172.21.15.16:0/2000667882 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fe52c06af00 0x7fe52c06d3c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe54015bb80 tx=0x7fe534008040 comp rx=0 tx=0).stop 2023-10-16T12:11:56.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.796+0000 7fe5461ee700 1 -- 172.21.15.16:0/2000667882 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe54006a7e0 msgr2=0x7fe540156cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:56.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.796+0000 7fe5461ee700 1 --2- 172.21.15.16:0/2000667882 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe54006a7e0 0x7fe540156cd0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fe530009c10 tx=0x7fe530004750 comp rx=0 tx=0).stop 2023-10-16T12:11:56.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.796+0000 7fe5461ee700 1 -- 172.21.15.16:0/2000667882 shutdown_connections 2023-10-16T12:11:56.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.796+0000 7fe5461ee700 1 --2- 172.21.15.16:0/2000667882 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fe52c06af00 0x7fe52c06d3c0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:56.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.796+0000 7fe5461ee700 1 --2- 172.21.15.16:0/2000667882 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe54015db40 0x7fe5400c9520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:56.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.796+0000 7fe5461ee700 1 --2- 172.21.15.16:0/2000667882 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe54006a7e0 0x7fe540156cd0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:56.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.796+0000 7fe5461ee700 1 --2- 172.21.15.16:0/2000667882 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe5400ce160 0x7fe54015e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:56.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.796+0000 7fe5461ee700 1 -- 172.21.15.16:0/2000667882 >> 172.21.15.16:0/2000667882 conn(0x7fe5400b1fd0 msgr2=0x7fe54015c470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:56.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.797+0000 7fe5461ee700 1 -- 172.21.15.16:0/2000667882 shutdown_connections 2023-10-16T12:11:56.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.797+0000 7fe5461ee700 1 -- 172.21.15.16:0/2000667882 wait complete. 2023-10-16T12:11:56.810 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2023-10-16T12:11:56.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.934+0000 7f4e65153700 1 Processor -- start 2023-10-16T12:11:56.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.934+0000 7f4e65153700 1 -- start start 2023-10-16T12:11:56.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.935+0000 7f4e65153700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f4e6006aef0 0x7f4e600b3480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:56.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.935+0000 7f4e65153700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e600d11c0 con 0x7f4e6015b860 2023-10-16T12:11:56.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.935+0000 7f4e65153700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e600cd360 con 0x7f4e600b2be0 2023-10-16T12:11:56.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.935+0000 7f4e65153700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e600d03d0 con 0x7f4e6006aef0 2023-10-16T12:11:56.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.935+0000 7f4e5ed9d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f4e6015b860 0x7f4e6006a7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:52932/0 (socket says 172.21.15.16:52932) 2023-10-16T12:11:56.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.935+0000 7f4e5ed9d700 1 -- 172.21.15.16:0/956400530 learned_addr learned my addr 172.21.15.16:0/956400530 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:56.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.935+0000 7f4e5e59c700 1 --2- 172.21.15.16:0/956400530 >> v2:172.21.15.49:3300/0 conn(0x7f4e6006aef0 0x7f4e600b3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:56.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.935+0000 7f4e5dd9b700 1 -- 172.21.15.16:0/956400530 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1400653688 0 0) 0x7f4e600d11c0 con 0x7f4e6015b860 2023-10-16T12:11:56.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.935+0000 7f4e5dd9b700 1 -- 172.21.15.16:0/956400530 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e480035b0 con 0x7f4e6015b860 2023-10-16T12:11:56.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.936+0000 7f4e5dd9b700 1 -- 172.21.15.16:0/956400530 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4093792687 0 0) 0x7f4e480035b0 con 0x7f4e6015b860 2023-10-16T12:11:56.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.936+0000 7f4e5dd9b700 1 -- 172.21.15.16:0/956400530 >> v2:172.21.15.49:3300/0 conn(0x7f4e6006aef0 msgr2=0x7f4e600b3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:56.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.936+0000 7f4e5dd9b700 1 --2- 172.21.15.16:0/956400530 >> v2:172.21.15.49:3300/0 conn(0x7f4e6006aef0 0x7f4e600b3480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:56.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.936+0000 7f4e5dd9b700 1 -- 172.21.15.16:0/956400530 >> v1:172.21.15.49:6789/0 conn(0x7f4e600b2be0 legacy=0x7f4e600b39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:56.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.936+0000 7f4e5dd9b700 1 -- 172.21.15.16:0/956400530 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e600d05b0 con 0x7f4e6015b860 2023-10-16T12:11:56.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.936+0000 7f4e5dd9b700 1 -- 172.21.15.16:0/956400530 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f4e54001670 con 0x7f4e6015b860 2023-10-16T12:11:56.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.936+0000 7f4e5dd9b700 1 -- 172.21.15.16:0/956400530 >> v1:172.21.15.16:6789/0 conn(0x7f4e6015b860 legacy=0x7f4e6006a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:56.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.936+0000 7f4e5dd9b700 1 --2- 172.21.15.16:0/956400530 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4e48003a00 0x7f4e48003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:56.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.936+0000 7f4e5dd9b700 1 --2- 172.21.15.16:0/956400530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e48004850 0x7f4e48004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:56.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.936+0000 7f4e5dd9b700 1 --2- 172.21.15.16:0/956400530 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4e48005210 0x7f4e4800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:56.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.936+0000 7f4e5dd9b700 1 -- 172.21.15.16:0/956400530 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e600d03d0 con 0x7f4e48004850 2023-10-16T12:11:56.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.936+0000 7f4e5dd9b700 1 -- 172.21.15.16:0/956400530 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e4800c0b0 con 0x7f4e48005210 2023-10-16T12:11:56.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.936+0000 7f4e5dd9b700 1 -- 172.21.15.16:0/956400530 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e4800c270 con 0x7f4e48003a00 2023-10-16T12:11:56.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.936+0000 7f4e5e59c700 1 --2- 172.21.15.16:0/956400530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e48004850 0x7f4e48004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:56.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.936+0000 7f4e5ed9d700 1 --2- 172.21.15.16:0/956400530 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4e48003a00 0x7f4e48003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:56.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.936+0000 7f4e5f59e700 1 --2- 172.21.15.16:0/956400530 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4e48005210 0x7f4e4800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:56.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.936+0000 7f4e5e59c700 1 -- 172.21.15.16:0/956400530 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4e48003a00 msgr2=0x7f4e48003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:56.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.936+0000 7f4e5e59c700 1 --2- 172.21.15.16:0/956400530 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4e48003a00 0x7f4e48003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:56.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.936+0000 7f4e5e59c700 1 -- 172.21.15.16:0/956400530 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4e48005210 msgr2=0x7f4e4800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:56.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.936+0000 7f4e5e59c700 1 --2- 172.21.15.16:0/956400530 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4e48005210 0x7f4e4800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:56.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.936+0000 7f4e5e59c700 1 -- 172.21.15.16:0/956400530 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e480035b0 con 0x7f4e48004850 2023-10-16T12:11:56.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.936+0000 7f4e5f59e700 1 -- 172.21.15.16:0/956400530 reap_dead start 2023-10-16T12:11:56.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.936+0000 7f4e5e59c700 1 --2- 172.21.15.16:0/956400530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e48004850 0x7f4e48004cd0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f4e50009b60 tx=0x7f4e48005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=483fdde6910468d6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:56.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.937+0000 7f4e5dd9b700 1 -- 172.21.15.16:0/956400530 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e5000e6c0 con 0x7f4e48004850 2023-10-16T12:11:56.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.937+0000 7f4e5dd9b700 1 -- 172.21.15.16:0/956400530 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e5000e820 con 0x7f4e48004850 2023-10-16T12:11:56.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.937+0000 7f4e5dd9b700 1 -- 172.21.15.16:0/956400530 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e5000eb90 con 0x7f4e48004850 2023-10-16T12:11:56.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.937+0000 7f4e65153700 1 -- 172.21.15.16:0/956400530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e48004850 msgr2=0x7f4e48004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:56.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.937+0000 7f4e65153700 1 --2- 172.21.15.16:0/956400530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e48004850 0x7f4e48004cd0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f4e50009b60 tx=0x7f4e48005880 comp rx=0 tx=0).stop 2023-10-16T12:11:56.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.937+0000 7f4e65153700 1 -- 172.21.15.16:0/956400530 shutdown_connections 2023-10-16T12:11:56.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.937+0000 7f4e65153700 1 --2- 172.21.15.16:0/956400530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e48004850 0x7f4e48004cd0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:56.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.937+0000 7f4e65153700 1 -- 172.21.15.16:0/956400530 >> 172.21.15.16:0/956400530 conn(0x7f4e600b1fd0 msgr2=0x7f4e600b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:56.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.937+0000 7f4e65153700 1 -- 172.21.15.16:0/956400530 shutdown_connections 2023-10-16T12:11:56.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.937+0000 7f4e65153700 1 -- 172.21.15.16:0/956400530 wait complete. 2023-10-16T12:11:56.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.939+0000 7f4e65153700 1 Processor -- start 2023-10-16T12:11:56.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.939+0000 7f4e65153700 1 -- start start 2023-10-16T12:11:56.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.940+0000 7f4e65153700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4e48004850 0x7f4e6015d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:56.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.940+0000 7f4e65153700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e600c8610 0x7f4e600b52c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:56.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.940+0000 7f4e65153700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4e60158cb0 0x7f4e600b5800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:56.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.940+0000 7f4e65153700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e600c81f0 con 0x7f4e600c8610 2023-10-16T12:11:56.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.940+0000 7f4e65153700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e600c75b0 con 0x7f4e48004850 2023-10-16T12:11:56.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.940+0000 7f4e65153700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e600c7850 con 0x7f4e60158cb0 2023-10-16T12:11:56.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.940+0000 7f4e5e59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e600c8610 0x7f4e600b52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:56.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.940+0000 7f4e5e59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e600c8610 0x7f4e600b52c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35440/0 (socket says 172.21.15.16:35440) 2023-10-16T12:11:56.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.940+0000 7f4e5e59c700 1 -- 172.21.15.16:0/1492216035 learned_addr learned my addr 172.21.15.16:0/1492216035 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:56.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.940+0000 7f4e5ed9d700 1 --2- 172.21.15.16:0/1492216035 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4e48004850 0x7f4e6015d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:56.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.940+0000 7f4e5f59e700 1 --2- 172.21.15.16:0/1492216035 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4e60158cb0 0x7f4e600b5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:56.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.941+0000 7f4e5e59c700 1 -- 172.21.15.16:0/1492216035 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4e60158cb0 msgr2=0x7f4e600b5800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:56.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.941+0000 7f4e5e59c700 1 --2- 172.21.15.16:0/1492216035 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4e60158cb0 0x7f4e600b5800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:56.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.941+0000 7f4e5e59c700 1 -- 172.21.15.16:0/1492216035 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4e48004850 msgr2=0x7f4e6015d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:56.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.941+0000 7f4e5e59c700 1 --2- 172.21.15.16:0/1492216035 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4e48004850 0x7f4e6015d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:56.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.941+0000 7f4e5e59c700 1 -- 172.21.15.16:0/1492216035 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e600c7b30 con 0x7f4e600c8610 2023-10-16T12:11:56.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.941+0000 7f4e5ed9d700 1 --2- 172.21.15.16:0/1492216035 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4e48004850 0x7f4e6015d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:11:56.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.941+0000 7f4e5e59c700 1 --2- 172.21.15.16:0/1492216035 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e600c8610 0x7f4e600b52c0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f4e5000a010 tx=0x7f4e500047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:56.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.941+0000 7f4e477fe700 1 -- 172.21.15.16:0/1492216035 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e5000e6c0 con 0x7f4e600c8610 2023-10-16T12:11:56.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.941+0000 7f4e477fe700 1 -- 172.21.15.16:0/1492216035 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e50003ad0 con 0x7f4e600c8610 2023-10-16T12:11:56.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.941+0000 7f4e65153700 1 -- 172.21.15.16:0/1492216035 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e600c7e10 con 0x7f4e600c8610 2023-10-16T12:11:56.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.941+0000 7f4e477fe700 1 -- 172.21.15.16:0/1492216035 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e50003e80 con 0x7f4e600c8610 2023-10-16T12:11:56.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.941+0000 7f4e65153700 1 -- 172.21.15.16:0/1492216035 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e60149a90 con 0x7f4e600c8610 2023-10-16T12:11:56.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.942+0000 7f4e65153700 1 -- 172.21.15.16:0/1492216035 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e6004ebb0 con 0x7f4e600c8610 2023-10-16T12:11:56.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.943+0000 7f4e477fe700 1 -- 172.21.15.16:0/1492216035 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f4e50013470 con 0x7f4e600c8610 2023-10-16T12:11:56.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.943+0000 7f4e477fe700 1 --2- 172.21.15.16:0/1492216035 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f4e4c06af00 0x7f4e4c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:56.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.943+0000 7f4e477fe700 1 -- 172.21.15.16:0/1492216035 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f4e5008ad70 con 0x7f4e600c8610 2023-10-16T12:11:56.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.943+0000 7f4e5ed9d700 1 --2- 172.21.15.16:0/1492216035 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f4e4c06af00 0x7f4e4c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:56.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.943+0000 7f4e5ed9d700 1 --2- 172.21.15.16:0/1492216035 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f4e4c06af00 0x7f4e4c06d3c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f4e54000c00 tx=0x7f4e54009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:56.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:56.947+0000 7f4e477fe700 1 -- 172.21.15.16:0/1492216035 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4e500561e0 con 0x7f4e600c8610 2023-10-16T12:11:57.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.235+0000 7f4e65153700 1 -- 172.21.15.16:0/1492216035 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7f4e600b0280 con 0x7f4e600c8610 2023-10-16T12:11:57.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.237+0000 7f4e477fe700 1 -- 172.21.15.16:0/1492216035 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v14) v1 ==== 90+0+0 (secure 0 0 0) 0x7f4e500170a0 con 0x7f4e600c8610 2023-10-16T12:11:57.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.239+0000 7f4e65153700 1 -- 172.21.15.16:0/1492216035 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f4e4c06af00 msgr2=0x7f4e4c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:57.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.239+0000 7f4e65153700 1 --2- 172.21.15.16:0/1492216035 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f4e4c06af00 0x7f4e4c06d3c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f4e54000c00 tx=0x7f4e54009040 comp rx=0 tx=0).stop 2023-10-16T12:11:57.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.239+0000 7f4e65153700 1 -- 172.21.15.16:0/1492216035 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e600c8610 msgr2=0x7f4e600b52c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:57.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.239+0000 7f4e65153700 1 --2- 172.21.15.16:0/1492216035 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e600c8610 0x7f4e600b52c0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f4e5000a010 tx=0x7f4e500047c0 comp rx=0 tx=0).stop 2023-10-16T12:11:57.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.239+0000 7f4e65153700 1 -- 172.21.15.16:0/1492216035 shutdown_connections 2023-10-16T12:11:57.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.239+0000 7f4e65153700 1 --2- 172.21.15.16:0/1492216035 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f4e4c06af00 0x7f4e4c06d3c0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:57.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.239+0000 7f4e65153700 1 --2- 172.21.15.16:0/1492216035 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4e60158cb0 0x7f4e600b5800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:57.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.239+0000 7f4e65153700 1 --2- 172.21.15.16:0/1492216035 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4e48004850 0x7f4e6015d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:57.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.239+0000 7f4e65153700 1 --2- 172.21.15.16:0/1492216035 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e600c8610 0x7f4e600b52c0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:57.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.239+0000 7f4e65153700 1 -- 172.21.15.16:0/1492216035 >> 172.21.15.16:0/1492216035 conn(0x7f4e600b1fd0 msgr2=0x7f4e6006a7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:57.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.239+0000 7f4e65153700 1 -- 172.21.15.16:0/1492216035 shutdown_connections 2023-10-16T12:11:57.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.239+0000 7f4e65153700 1 -- 172.21.15.16:0/1492216035 wait complete. 2023-10-16T12:11:57.254 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:11:57.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.376+0000 7f6d0530a700 1 Processor -- start 2023-10-16T12:11:57.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.377+0000 7f6d0530a700 1 -- start start 2023-10-16T12:11:57.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.378+0000 7f6d0530a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6d000b3ba0 0x7f6d000b2da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:57.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.378+0000 7f6d0530a700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f6d0015a6e0 0x7f6d00070860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:57.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.378+0000 7f6d0530a700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d000b4060 con 0x7f6d0015a300 2023-10-16T12:11:57.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.378+0000 7f6d0530a700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d000b22c0 con 0x7f6d0015a6e0 2023-10-16T12:11:57.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.378+0000 7f6d0530a700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d000b25a0 con 0x7f6d000b3ba0 2023-10-16T12:11:57.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.378+0000 7f6cff7fe700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f6d0015a6e0 0x7f6d00070860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:57.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.378+0000 7f6cfe7fc700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6d000b3ba0 0x7f6d000b2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:57.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.378+0000 7f6cff7fe700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f6d0015a6e0 0x7f6d00070860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:34512/0 (socket says 172.21.15.16:34512) 2023-10-16T12:11:57.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.378+0000 7f6cfeffd700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7f6d0015a300 0x7f6d0015ea70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:60134/0 (socket says 172.21.15.16:60134) 2023-10-16T12:11:57.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.378+0000 7f6cfe7fc700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6d000b3ba0 0x7f6d000b2da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:51938/0 (socket says 172.21.15.16:51938) 2023-10-16T12:11:57.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.378+0000 7f6cff7fe700 1 -- 172.21.15.16:0/1132186912 learned_addr learned my addr 172.21.15.16:0/1132186912 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:57.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.378+0000 7f6cfe7fc700 1 -- 172.21.15.16:0/1132186912 >> v2:172.21.15.49:3300/0 conn(0x7f6d0015a6e0 msgr2=0x7f6d00070860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:57.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.378+0000 7f6cfe7fc700 1 --2- 172.21.15.16:0/1132186912 >> v2:172.21.15.49:3300/0 conn(0x7f6d0015a6e0 0x7f6d00070860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:57.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.378+0000 7f6cfe7fc700 1 -- 172.21.15.16:0/1132186912 >> v1:172.21.15.49:6789/0 conn(0x7f6d0015a300 legacy=0x7f6d0015ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:57.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.379+0000 7f6cfe7fc700 1 -- 172.21.15.16:0/1132186912 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d000b2880 con 0x7f6d000b3ba0 2023-10-16T12:11:57.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.379+0000 7f6cff7fe700 1 --2- 172.21.15.16:0/1132186912 >> v2:172.21.15.49:3300/0 conn(0x7f6d0015a6e0 0x7f6d00070860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:11:57.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.379+0000 7f6cfe7fc700 1 --2- 172.21.15.16:0/1132186912 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6d000b3ba0 0x7f6d000b2da0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6ce8009840 tx=0x7f6ce8009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a9e4ac1320652e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:57.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.379+0000 7f6cfdffb700 1 -- 172.21.15.16:0/1132186912 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce8011830 con 0x7f6d000b3ba0 2023-10-16T12:11:57.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.379+0000 7f6cfdffb700 1 -- 172.21.15.16:0/1132186912 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ce8011990 con 0x7f6d000b3ba0 2023-10-16T12:11:57.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.379+0000 7f6cfdffb700 1 -- 172.21.15.16:0/1132186912 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce8011d20 con 0x7f6d000b3ba0 2023-10-16T12:11:57.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.379+0000 7f6d0530a700 1 -- 172.21.15.16:0/1132186912 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6d000b3ba0 msgr2=0x7f6d000b2da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:57.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.379+0000 7f6d0530a700 1 --2- 172.21.15.16:0/1132186912 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6d000b3ba0 0x7f6d000b2da0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6ce8009840 tx=0x7f6ce8009b50 comp rx=0 tx=0).stop 2023-10-16T12:11:57.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.380+0000 7f6d0530a700 1 -- 172.21.15.16:0/1132186912 shutdown_connections 2023-10-16T12:11:57.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.380+0000 7f6d0530a700 1 --2- 172.21.15.16:0/1132186912 >> v2:172.21.15.49:3300/0 conn(0x7f6d0015a6e0 0x7f6d00070860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:57.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.380+0000 7f6d0530a700 1 --2- 172.21.15.16:0/1132186912 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6d000b3ba0 0x7f6d000b2da0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:57.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.380+0000 7f6d0530a700 1 -- 172.21.15.16:0/1132186912 >> 172.21.15.16:0/1132186912 conn(0x7f6d000b4cd0 msgr2=0x7f6d0006a7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:57.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.380+0000 7f6d0530a700 1 -- 172.21.15.16:0/1132186912 shutdown_connections 2023-10-16T12:11:57.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.380+0000 7f6d0530a700 1 -- 172.21.15.16:0/1132186912 wait complete. 2023-10-16T12:11:57.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.381+0000 7f6d0530a700 1 Processor -- start 2023-10-16T12:11:57.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.381+0000 7f6d0530a700 1 -- start start 2023-10-16T12:11:57.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.382+0000 7f6d0530a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6d000b3ba0 0x7f6d0013d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:57.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.382+0000 7f6d0530a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d0015a300 0x7f6d001f1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:57.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.382+0000 7f6d0530a700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6d0015a6e0 0x7f6d0015cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:57.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.382+0000 7f6d0530a700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d0013e010 con 0x7f6d0015a300 2023-10-16T12:11:57.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.382+0000 7f6d0530a700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d0013e2f0 con 0x7f6d0015a6e0 2023-10-16T12:11:57.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.382+0000 7f6d0530a700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d0013e5d0 con 0x7f6d000b3ba0 2023-10-16T12:11:57.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.382+0000 7f6cfe7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d0015a300 0x7f6d001f1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:57.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.382+0000 7f6cfe7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d0015a300 0x7f6d001f1930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35454/0 (socket says 172.21.15.16:35454) 2023-10-16T12:11:57.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.382+0000 7f6cfe7fc700 1 -- 172.21.15.16:0/2490960277 learned_addr learned my addr 172.21.15.16:0/2490960277 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:57.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.383+0000 7f6cff7fe700 1 --2- 172.21.15.16:0/2490960277 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6d0015a6e0 0x7f6d0015cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:57.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.383+0000 7f6cfeffd700 1 --2- 172.21.15.16:0/2490960277 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6d000b3ba0 0x7f6d0013d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:57.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.383+0000 7f6cfe7fc700 1 -- 172.21.15.16:0/2490960277 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6d000b3ba0 msgr2=0x7f6d0013d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:57.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.383+0000 7f6cfe7fc700 1 --2- 172.21.15.16:0/2490960277 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6d000b3ba0 0x7f6d0013d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:57.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.383+0000 7f6cfe7fc700 1 -- 172.21.15.16:0/2490960277 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6d0015a6e0 msgr2=0x7f6d0015cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:57.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.383+0000 7f6cfe7fc700 1 --2- 172.21.15.16:0/2490960277 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6d0015a6e0 0x7f6d0015cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:57.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.383+0000 7f6cfe7fc700 1 -- 172.21.15.16:0/2490960277 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cf8009b30 con 0x7f6d0015a300 2023-10-16T12:11:57.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.383+0000 7f6cff7fe700 1 --2- 172.21.15.16:0/2490960277 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6d0015a6e0 0x7f6d0015cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:11:57.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.383+0000 7f6cfe7fc700 1 --2- 172.21.15.16:0/2490960277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d0015a300 0x7f6d001f1930 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f6ce80090b0 tx=0x7f6ce8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:57.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.383+0000 7f6cf77fe700 1 -- 172.21.15.16:0/2490960277 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce80036a0 con 0x7f6d0015a300 2023-10-16T12:11:57.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.383+0000 7f6cf77fe700 1 -- 172.21.15.16:0/2490960277 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ce801f070 con 0x7f6d0015a300 2023-10-16T12:11:57.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.383+0000 7f6d0530a700 1 -- 172.21.15.16:0/2490960277 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ce80094a0 con 0x7f6d0015a300 2023-10-16T12:11:57.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.383+0000 7f6cf77fe700 1 -- 172.21.15.16:0/2490960277 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce8003a20 con 0x7f6d0015a300 2023-10-16T12:11:57.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.383+0000 7f6d0530a700 1 -- 172.21.15.16:0/2490960277 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d001f1100 con 0x7f6d0015a300 2023-10-16T12:11:57.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.384+0000 7f6d0530a700 1 -- 172.21.15.16:0/2490960277 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d0004ebb0 con 0x7f6d0015a300 2023-10-16T12:11:57.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.385+0000 7f6cf77fe700 1 -- 172.21.15.16:0/2490960277 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f6ce8003b80 con 0x7f6d0015a300 2023-10-16T12:11:57.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.385+0000 7f6cf77fe700 1 --2- 172.21.15.16:0/2490960277 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f6ce006af00 0x7f6ce006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:57.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.385+0000 7f6cf77fe700 1 -- 172.21.15.16:0/2490960277 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f6ce8021070 con 0x7f6d0015a300 2023-10-16T12:11:57.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.385+0000 7f6cfeffd700 1 --2- 172.21.15.16:0/2490960277 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f6ce006af00 0x7f6ce006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:57.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.386+0000 7f6cfeffd700 1 --2- 172.21.15.16:0/2490960277 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f6ce006af00 0x7f6ce006d3c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f6cf0000f60 tx=0x7f6cf0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:57.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.390+0000 7f6cf77fe700 1 -- 172.21.15.16:0/2490960277 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6ce8055fb0 con 0x7f6d0015a300 2023-10-16T12:11:57.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.592+0000 7f6d0530a700 1 -- 172.21.15.16:0/2490960277 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6d000b0b90 con 0x7f6d0015a300 2023-10-16T12:11:57.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.592+0000 7f6cf77fe700 1 -- 172.21.15.16:0/2490960277 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v35) v1 ==== 74+0+12954 (secure 0 0 0) 0x7f6ce802b090 con 0x7f6d0015a300 2023-10-16T12:11:57.594 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:11:57.595 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":35,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:11:54.086908+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:11:57.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.595+0000 7f6d0530a700 1 -- 172.21.15.16:0/2490960277 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f6ce006af00 msgr2=0x7f6ce006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:57.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.595+0000 7f6d0530a700 1 --2- 172.21.15.16:0/2490960277 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f6ce006af00 0x7f6ce006d3c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f6cf0000f60 tx=0x7f6cf0009040 comp rx=0 tx=0).stop 2023-10-16T12:11:57.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.595+0000 7f6d0530a700 1 -- 172.21.15.16:0/2490960277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d0015a300 msgr2=0x7f6d001f1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:57.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.595+0000 7f6d0530a700 1 --2- 172.21.15.16:0/2490960277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d0015a300 0x7f6d001f1930 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f6ce80090b0 tx=0x7f6ce8004750 comp rx=0 tx=0).stop 2023-10-16T12:11:57.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.595+0000 7f6d0530a700 1 -- 172.21.15.16:0/2490960277 shutdown_connections 2023-10-16T12:11:57.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.595+0000 7f6d0530a700 1 --2- 172.21.15.16:0/2490960277 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f6ce006af00 0x7f6ce006d3c0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:57.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.595+0000 7f6d0530a700 1 --2- 172.21.15.16:0/2490960277 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6d0015a6e0 0x7f6d0015cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:57.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.595+0000 7f6d0530a700 1 --2- 172.21.15.16:0/2490960277 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6d000b3ba0 0x7f6d0013d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:57.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.595+0000 7f6d0530a700 1 --2- 172.21.15.16:0/2490960277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d0015a300 0x7f6d001f1930 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:57.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.595+0000 7f6d0530a700 1 -- 172.21.15.16:0/2490960277 >> 172.21.15.16:0/2490960277 conn(0x7f6d000b4cd0 msgr2=0x7f6d0006a7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:57.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.595+0000 7f6d0530a700 1 -- 172.21.15.16:0/2490960277 shutdown_connections 2023-10-16T12:11:57.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.595+0000 7f6d0530a700 1 -- 172.21.15.16:0/2490960277 wait complete. 2023-10-16T12:11:57.609 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T12:11:57.610 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:11:57.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.735+0000 7f49cf1e8700 1 Processor -- start 2023-10-16T12:11:57.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.736+0000 7f49cf1e8700 1 -- start start 2023-10-16T12:11:57.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.736+0000 7f49cf1e8700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f49c8160fe0 0x7f49c80ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:57.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.736+0000 7f49cf1e8700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f49c806aef0 0x7f49c80adf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:57.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.736+0000 7f49cf1e8700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49c81613c0 con 0x7f49c806da30 2023-10-16T12:11:57.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.736+0000 7f49cf1e8700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49c80cf680 con 0x7f49c806aef0 2023-10-16T12:11:57.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.736+0000 7f49cf1e8700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49c80adb00 con 0x7f49c8160fe0 2023-10-16T12:11:57.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.736+0000 7f49cd785700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f49c806aef0 0x7f49c80adf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:57.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.736+0000 7f49c7fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f49c806da30 0x7f49c806a7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:52968/0 (socket says 172.21.15.16:52968) 2023-10-16T12:11:57.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.736+0000 7f49c7fff700 1 -- 172.21.15.16:0/1556011608 learned_addr learned my addr 172.21.15.16:0/1556011608 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:57.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.737+0000 7f49ccf84700 1 --2- 172.21.15.16:0/1556011608 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f49c8160fe0 0x7f49c80ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:57.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.737+0000 7f49cd785700 1 -- 172.21.15.16:0/1556011608 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f49c8160fe0 msgr2=0x7f49c80ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:57.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.737+0000 7f49cd785700 1 --2- 172.21.15.16:0/1556011608 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f49c8160fe0 0x7f49c80ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:57.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.737+0000 7f49cd785700 1 -- 172.21.15.16:0/1556011608 >> v1:172.21.15.16:6789/0 conn(0x7f49c806da30 legacy=0x7f49c806a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:57.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.737+0000 7f49cd785700 1 -- 172.21.15.16:0/1556011608 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49c80acef0 con 0x7f49c806aef0 2023-10-16T12:11:57.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.737+0000 7f49c77fe700 1 -- 172.21.15.16:0/1556011608 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 730354484 0 0) 0x7f49b4001110 con 0x7f49c806da30 2023-10-16T12:11:57.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.737+0000 7f49ccf84700 1 --2- 172.21.15.16:0/1556011608 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f49c8160fe0 0x7f49c80ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:11:57.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.737+0000 7f49cd785700 1 --2- 172.21.15.16:0/1556011608 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49c806aef0 0x7f49c80adf90 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f49c0009d90 tx=0x7f49c000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af6207935c396c3a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:57.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.737+0000 7f49c77fe700 1 -- 172.21.15.16:0/1556011608 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c000e070 con 0x7f49c806aef0 2023-10-16T12:11:57.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.737+0000 7f49c77fe700 1 -- 172.21.15.16:0/1556011608 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49c000bbd0 con 0x7f49c806aef0 2023-10-16T12:11:57.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.737+0000 7f49c77fe700 1 -- 172.21.15.16:0/1556011608 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c0012430 con 0x7f49c806aef0 2023-10-16T12:11:57.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.738+0000 7f49cf1e8700 1 -- 172.21.15.16:0/1556011608 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49c806aef0 msgr2=0x7f49c80adf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:57.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.738+0000 7f49cf1e8700 1 --2- 172.21.15.16:0/1556011608 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49c806aef0 0x7f49c80adf90 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f49c0009d90 tx=0x7f49c000b330 comp rx=0 tx=0).stop 2023-10-16T12:11:57.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.738+0000 7f49cf1e8700 1 -- 172.21.15.16:0/1556011608 shutdown_connections 2023-10-16T12:11:57.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.738+0000 7f49cf1e8700 1 --2- 172.21.15.16:0/1556011608 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49c806aef0 0x7f49c80adf90 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:57.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.738+0000 7f49cf1e8700 1 --2- 172.21.15.16:0/1556011608 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f49c8160fe0 0x7f49c80ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:57.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.738+0000 7f49cf1e8700 1 -- 172.21.15.16:0/1556011608 >> 172.21.15.16:0/1556011608 conn(0x7f49c80b1ed0 msgr2=0x7f49c80b22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:57.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.738+0000 7f49cf1e8700 1 -- 172.21.15.16:0/1556011608 shutdown_connections 2023-10-16T12:11:57.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.738+0000 7f49cf1e8700 1 -- 172.21.15.16:0/1556011608 wait complete. 2023-10-16T12:11:57.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.739+0000 7f49cf1e8700 1 Processor -- start 2023-10-16T12:11:57.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.739+0000 7f49cf1e8700 1 -- start start 2023-10-16T12:11:57.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.740+0000 7f49cf1e8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49c806aef0 0x7f49c80c49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:57.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.740+0000 7f49cf1e8700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f49c806da30 0x7f49c81f5f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:57.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.740+0000 7f49cf1e8700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f49c8160fe0 0x7f49c80c61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:57.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.740+0000 7f49cf1e8700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49c81e6d20 con 0x7f49c806aef0 2023-10-16T12:11:57.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.740+0000 7f49cf1e8700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49c81e6f90 con 0x7f49c806da30 2023-10-16T12:11:57.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.740+0000 7f49cf1e8700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49c81e7270 con 0x7f49c8160fe0 2023-10-16T12:11:57.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.741+0000 7f49ccf84700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49c806aef0 0x7f49c80c49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:57.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.741+0000 7f49ccf84700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49c806aef0 0x7f49c80c49f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35466/0 (socket says 172.21.15.16:35466) 2023-10-16T12:11:57.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.741+0000 7f49ccf84700 1 -- 172.21.15.16:0/1816767992 learned_addr learned my addr 172.21.15.16:0/1816767992 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:57.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.741+0000 7f49c7fff700 1 --2- 172.21.15.16:0/1816767992 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f49c806da30 0x7f49c81f5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:57.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.741+0000 7f49cd785700 1 --2- 172.21.15.16:0/1816767992 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f49c8160fe0 0x7f49c80c61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:57.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.741+0000 7f49ccf84700 1 -- 172.21.15.16:0/1816767992 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f49c8160fe0 msgr2=0x7f49c80c61f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:57.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.741+0000 7f49ccf84700 1 --2- 172.21.15.16:0/1816767992 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f49c8160fe0 0x7f49c80c61f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:57.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.741+0000 7f49ccf84700 1 -- 172.21.15.16:0/1816767992 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f49c806da30 msgr2=0x7f49c81f5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:57.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.741+0000 7f49ccf84700 1 --2- 172.21.15.16:0/1816767992 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f49c806da30 0x7f49c81f5f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:57.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.741+0000 7f49ccf84700 1 -- 172.21.15.16:0/1816767992 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49c0009a70 con 0x7f49c806aef0 2023-10-16T12:11:57.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.741+0000 7f49c7fff700 1 --2- 172.21.15.16:0/1816767992 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f49c806da30 0x7f49c81f5f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:11:57.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.741+0000 7f49cd785700 1 --2- 172.21.15.16:0/1816767992 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f49c8160fe0 0x7f49c80c61f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:11:57.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.741+0000 7f49ccf84700 1 --2- 172.21.15.16:0/1816767992 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49c806aef0 0x7f49c80c49f0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f49bc00c7f0 tx=0x7f49bc00cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:57.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.741+0000 7f49c57fa700 1 -- 172.21.15.16:0/1816767992 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49bc005c20 con 0x7f49c806aef0 2023-10-16T12:11:57.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.741+0000 7f49c57fa700 1 -- 172.21.15.16:0/1816767992 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49bc005d80 con 0x7f49c806aef0 2023-10-16T12:11:57.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.741+0000 7f49cf1e8700 1 -- 172.21.15.16:0/1816767992 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49c81e7550 con 0x7f49c806aef0 2023-10-16T12:11:57.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.741+0000 7f49c57fa700 1 -- 172.21.15.16:0/1816767992 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49bc011800 con 0x7f49c806aef0 2023-10-16T12:11:57.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.741+0000 7f49cf1e8700 1 -- 172.21.15.16:0/1816767992 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49c81f6470 con 0x7f49c806aef0 2023-10-16T12:11:57.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.742+0000 7f49cf1e8700 1 -- 172.21.15.16:0/1816767992 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49c81e7a70 con 0x7f49c806aef0 2023-10-16T12:11:57.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.743+0000 7f49c57fa700 1 -- 172.21.15.16:0/1816767992 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f49bc011960 con 0x7f49c806aef0 2023-10-16T12:11:57.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.743+0000 7f49c57fa700 1 --2- 172.21.15.16:0/1816767992 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f49ac06af00 0x7f49ac06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:57.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.743+0000 7f49c57fa700 1 -- 172.21.15.16:0/1816767992 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f49bc089280 con 0x7f49c806aef0 2023-10-16T12:11:57.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.743+0000 7f49c7fff700 1 --2- 172.21.15.16:0/1816767992 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f49ac06af00 0x7f49ac06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:57.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.744+0000 7f49c7fff700 1 --2- 172.21.15.16:0/1816767992 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f49ac06af00 0x7f49ac06d3c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f49c80ae780 tx=0x7f49b4007ae0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:57.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.748+0000 7f49c57fa700 1 -- 172.21.15.16:0/1816767992 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f49c81e7a70 con 0x7f49c806aef0 2023-10-16T12:11:57.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.952+0000 7f49cf1e8700 1 -- 172.21.15.16:0/1816767992 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f49c81e7a70 con 0x7f49c806aef0 2023-10-16T12:11:57.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.953+0000 7f49c57fa700 1 -- 172.21.15.16:0/1816767992 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v35) v1 ==== 93+0+10 (secure 0 0 0) 0x7f49bc01e020 con 0x7f49c806aef0 2023-10-16T12:11:57.955 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T12:11:57.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.955+0000 7f49cf1e8700 1 -- 172.21.15.16:0/1816767992 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f49ac06af00 msgr2=0x7f49ac06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:57.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.955+0000 7f49cf1e8700 1 --2- 172.21.15.16:0/1816767992 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f49ac06af00 0x7f49ac06d3c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f49c80ae780 tx=0x7f49b4007ae0 comp rx=0 tx=0).stop 2023-10-16T12:11:57.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.955+0000 7f49cf1e8700 1 -- 172.21.15.16:0/1816767992 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49c806aef0 msgr2=0x7f49c80c49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:57.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.955+0000 7f49cf1e8700 1 --2- 172.21.15.16:0/1816767992 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49c806aef0 0x7f49c80c49f0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f49bc00c7f0 tx=0x7f49bc00cbb0 comp rx=0 tx=0).stop 2023-10-16T12:11:57.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.955+0000 7f49cf1e8700 1 -- 172.21.15.16:0/1816767992 shutdown_connections 2023-10-16T12:11:57.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.955+0000 7f49cf1e8700 1 --2- 172.21.15.16:0/1816767992 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f49ac06af00 0x7f49ac06d3c0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:57.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.955+0000 7f49cf1e8700 1 --2- 172.21.15.16:0/1816767992 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f49c8160fe0 0x7f49c80c61f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:57.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.955+0000 7f49cf1e8700 1 --2- 172.21.15.16:0/1816767992 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49c806aef0 0x7f49c80c49f0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:57.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.955+0000 7f49cf1e8700 1 --2- 172.21.15.16:0/1816767992 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f49c806da30 0x7f49c81f5f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:57.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.955+0000 7f49cf1e8700 1 -- 172.21.15.16:0/1816767992 >> 172.21.15.16:0/1816767992 conn(0x7f49c80b1ed0 msgr2=0x7f49c80cde70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:57.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.955+0000 7f49cf1e8700 1 -- 172.21.15.16:0/1816767992 shutdown_connections 2023-10-16T12:11:57.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:57.955+0000 7f49cf1e8700 1 -- 172.21.15.16:0/1816767992 wait complete. 2023-10-16T12:11:57.969 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-16T12:11:58.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.092+0000 7f41afd3d700 1 Processor -- start 2023-10-16T12:11:58.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.092+0000 7f41afd3d700 1 -- start start 2023-10-16T12:11:58.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.093+0000 7f41afd3d700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f41a80d03b0 0x7f41a80cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:58.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.093+0000 7f41afd3d700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41a80cc2a0 con 0x7f41a815b630 2023-10-16T12:11:58.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.093+0000 7f41afd3d700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41a80cc540 con 0x7f41a80ce4f0 2023-10-16T12:11:58.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.093+0000 7f41afd3d700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41a80cdb20 con 0x7f41a80d03b0 2023-10-16T12:11:58.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.093+0000 7f41adad9700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f41a815b630 0x7f41a80cdde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:52980/0 (socket says 172.21.15.16:52980) 2023-10-16T12:11:58.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.093+0000 7f41adad9700 1 -- 172.21.15.16:0/1306375738 learned_addr learned my addr 172.21.15.16:0/1306375738 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:11:58.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.093+0000 7f41ae2da700 1 --2- 172.21.15.16:0/1306375738 >> v2:172.21.15.49:3300/0 conn(0x7f41a80d03b0 0x7f41a80cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:58.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.093+0000 7f41acad7700 1 -- 172.21.15.16:0/1306375738 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 734552821 0 0) 0x7f41a80cc2a0 con 0x7f41a815b630 2023-10-16T12:11:58.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.093+0000 7f41acad7700 1 -- 172.21.15.16:0/1306375738 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41940035b0 con 0x7f41a815b630 2023-10-16T12:11:58.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.093+0000 7f41acad7700 1 -- 172.21.15.16:0/1306375738 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1730302121 0 0) 0x7f41940035b0 con 0x7f41a815b630 2023-10-16T12:11:58.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.093+0000 7f41acad7700 1 -- 172.21.15.16:0/1306375738 >> v2:172.21.15.49:3300/0 conn(0x7f41a80d03b0 msgr2=0x7f41a80cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:58.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.094+0000 7f41acad7700 1 --2- 172.21.15.16:0/1306375738 >> v2:172.21.15.49:3300/0 conn(0x7f41a80d03b0 0x7f41a80cd5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:58.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.094+0000 7f41acad7700 1 -- 172.21.15.16:0/1306375738 >> v1:172.21.15.49:6789/0 conn(0x7f41a80ce4f0 legacy=0x7f41a80cfca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:58.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.094+0000 7f41acad7700 1 -- 172.21.15.16:0/1306375738 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41a80cc820 con 0x7f41a815b630 2023-10-16T12:11:58.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.094+0000 7f41acad7700 1 -- 172.21.15.16:0/1306375738 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f41a40017f0 con 0x7f41a815b630 2023-10-16T12:11:58.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.094+0000 7f41ae2da700 1 --2- 172.21.15.16:0/1306375738 >> v2:172.21.15.49:3300/0 conn(0x7f41a80d03b0 0x7f41a80cd5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:11:58.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.094+0000 7f41acad7700 1 -- 172.21.15.16:0/1306375738 >> v1:172.21.15.16:6789/0 conn(0x7f41a815b630 legacy=0x7f41a80cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:58.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.094+0000 7f41acad7700 1 --2- 172.21.15.16:0/1306375738 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4194003a00 0x7f4194003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:58.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.094+0000 7f41acad7700 1 --2- 172.21.15.16:0/1306375738 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4194004850 0x7f4194004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:58.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.094+0000 7f41acad7700 1 --2- 172.21.15.16:0/1306375738 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4194005210 0x7f419400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:58.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.094+0000 7f41acad7700 1 -- 172.21.15.16:0/1306375738 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f419400c180 con 0x7f4194003a00 2023-10-16T12:11:58.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.094+0000 7f41acad7700 1 -- 172.21.15.16:0/1306375738 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f419400c2f0 con 0x7f4194004850 2023-10-16T12:11:58.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.094+0000 7f41acad7700 1 -- 172.21.15.16:0/1306375738 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f419400c4b0 con 0x7f4194005210 2023-10-16T12:11:58.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.094+0000 7f41adad9700 1 --2- 172.21.15.16:0/1306375738 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4194003a00 0x7f4194003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:58.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.094+0000 7f41ad2d8700 1 --2- 172.21.15.16:0/1306375738 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4194004850 0x7f4194004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:58.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.094+0000 7f41ae2da700 1 --2- 172.21.15.16:0/1306375738 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4194005210 0x7f419400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:58.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.094+0000 7f41adad9700 1 -- 172.21.15.16:0/1306375738 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4194005210 msgr2=0x7f419400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:58.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.094+0000 7f41adad9700 1 --2- 172.21.15.16:0/1306375738 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4194005210 0x7f419400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:58.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.094+0000 7f41adad9700 1 -- 172.21.15.16:0/1306375738 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4194004850 msgr2=0x7f4194004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:58.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.094+0000 7f41adad9700 1 --2- 172.21.15.16:0/1306375738 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4194004850 0x7f4194004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:58.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.094+0000 7f41adad9700 1 -- 172.21.15.16:0/1306375738 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41940035b0 con 0x7f4194003a00 2023-10-16T12:11:58.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.094+0000 7f41ad2d8700 1 --2- 172.21.15.16:0/1306375738 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4194004850 0x7f4194004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:11:58.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.094+0000 7f41ae2da700 1 -- 172.21.15.16:0/1306375738 reap_dead start 2023-10-16T12:11:58.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.094+0000 7f41adad9700 1 --2- 172.21.15.16:0/1306375738 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4194003a00 0x7f4194003e20 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f41a400c240 tx=0x7f4194005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0088b5701c93207 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:58.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.094+0000 7f41acad7700 1 -- 172.21.15.16:0/1306375738 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41a400cde0 con 0x7f4194003a00 2023-10-16T12:11:58.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.094+0000 7f41acad7700 1 -- 172.21.15.16:0/1306375738 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41a4007b80 con 0x7f4194003a00 2023-10-16T12:11:58.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.094+0000 7f41acad7700 1 -- 172.21.15.16:0/1306375738 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41a40072f0 con 0x7f4194003a00 2023-10-16T12:11:58.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.095+0000 7f41afd3d700 1 -- 172.21.15.16:0/1306375738 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4194003a00 msgr2=0x7f4194003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:58.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.095+0000 7f41afd3d700 1 --2- 172.21.15.16:0/1306375738 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4194003a00 0x7f4194003e20 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f41a400c240 tx=0x7f4194005880 comp rx=0 tx=0).stop 2023-10-16T12:11:58.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.095+0000 7f41afd3d700 1 -- 172.21.15.16:0/1306375738 shutdown_connections 2023-10-16T12:11:58.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.095+0000 7f41afd3d700 1 --2- 172.21.15.16:0/1306375738 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4194003a00 0x7f4194003e20 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:58.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.095+0000 7f41afd3d700 1 -- 172.21.15.16:0/1306375738 >> 172.21.15.16:0/1306375738 conn(0x7f41a80d14b0 msgr2=0x7f41a80d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:58.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.095+0000 7f41afd3d700 1 -- 172.21.15.16:0/1306375738 shutdown_connections 2023-10-16T12:11:58.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.095+0000 7f41afd3d700 1 -- 172.21.15.16:0/1306375738 wait complete. 2023-10-16T12:11:58.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.096+0000 7f41afd3d700 1 Processor -- start 2023-10-16T12:11:58.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.096+0000 7f41afd3d700 1 -- start start 2023-10-16T12:11:58.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.097+0000 7f41afd3d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4194003a00 0x7f41a8151e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:58.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.097+0000 7f41afd3d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41a8151560 0x7f41a81f06e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:58.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.097+0000 7f41afd3d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f41a815ca60 0x7f41a815cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:58.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.097+0000 7f41afd3d700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41a815d420 con 0x7f41a8151560 2023-10-16T12:11:58.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.097+0000 7f41afd3d700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41a81526f0 con 0x7f4194003a00 2023-10-16T12:11:58.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.097+0000 7f41afd3d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41a8152410 con 0x7f41a815ca60 2023-10-16T12:11:58.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.097+0000 7f41ad2d8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41a8151560 0x7f41a81f06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:58.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.097+0000 7f41ad2d8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41a8151560 0x7f41a81f06e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35488/0 (socket says 172.21.15.16:35488) 2023-10-16T12:11:58.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.097+0000 7f41ad2d8700 1 -- 172.21.15.16:0/2378565285 learned_addr learned my addr 172.21.15.16:0/2378565285 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:58.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.097+0000 7f41adad9700 1 --2- 172.21.15.16:0/2378565285 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4194003a00 0x7f41a8151e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:58.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.097+0000 7f41ad2d8700 1 -- 172.21.15.16:0/2378565285 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f41a815ca60 msgr2=0x7f41a815cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:58.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.097+0000 7f41ae2da700 1 --2- 172.21.15.16:0/2378565285 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f41a815ca60 0x7f41a815cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:58.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.097+0000 7f41ad2d8700 1 --2- 172.21.15.16:0/2378565285 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f41a815ca60 0x7f41a815cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:58.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.097+0000 7f41ad2d8700 1 -- 172.21.15.16:0/2378565285 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4194003a00 msgr2=0x7f41a8151e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:58.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.097+0000 7f41ad2d8700 1 --2- 172.21.15.16:0/2378565285 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4194003a00 0x7f41a8151e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:58.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.098+0000 7f41ad2d8700 1 -- 172.21.15.16:0/2378565285 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41a80c9a50 con 0x7f41a8151560 2023-10-16T12:11:58.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.098+0000 7f41ad2d8700 1 --2- 172.21.15.16:0/2378565285 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41a8151560 0x7f41a81f06e0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f419800e460 tx=0x7f419800e820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:58.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.098+0000 7f419e7fc700 1 -- 172.21.15.16:0/2378565285 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41980052e0 con 0x7f41a8151560 2023-10-16T12:11:58.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.098+0000 7f419e7fc700 1 -- 172.21.15.16:0/2378565285 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4198005440 con 0x7f41a8151560 2023-10-16T12:11:58.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.098+0000 7f419e7fc700 1 -- 172.21.15.16:0/2378565285 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41980038c0 con 0x7f41a8151560 2023-10-16T12:11:58.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.098+0000 7f41afd3d700 1 -- 172.21.15.16:0/2378565285 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41a80c9d30 con 0x7f41a8151560 2023-10-16T12:11:58.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.098+0000 7f41afd3d700 1 -- 172.21.15.16:0/2378565285 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41a81519e0 con 0x7f41a8151560 2023-10-16T12:11:58.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.099+0000 7f41afd3d700 1 -- 172.21.15.16:0/2378565285 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41a80ca0a0 con 0x7f41a8151560 2023-10-16T12:11:58.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.100+0000 7f419e7fc700 1 -- 172.21.15.16:0/2378565285 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f4198003a20 con 0x7f41a8151560 2023-10-16T12:11:58.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.100+0000 7f419e7fc700 1 --2- 172.21.15.16:0/2378565285 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f418c06af00 0x7f418c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:58.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.100+0000 7f419e7fc700 1 -- 172.21.15.16:0/2378565285 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f4198088c40 con 0x7f41a8151560 2023-10-16T12:11:58.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.100+0000 7f41adad9700 1 --2- 172.21.15.16:0/2378565285 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f418c06af00 0x7f418c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:58.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.101+0000 7f41adad9700 1 --2- 172.21.15.16:0/2378565285 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f418c06af00 0x7f418c06d3c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f41a4007d30 tx=0x7f41a4007000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:58.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.104+0000 7f419e7fc700 1 -- 172.21.15.16:0/2378565285 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4198053ee0 con 0x7f41a8151560 2023-10-16T12:11:58.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.310+0000 7f41afd3d700 1 -- 172.21.15.16:0/2378565285 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f41a80c9880 con 0x7f41a8151560 2023-10-16T12:11:58.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.310+0000 7f419e7fc700 1 -- 172.21.15.16:0/2378565285 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v35) v1 ==== 84+0+65 (secure 0 0 0) 0x7f4198019060 con 0x7f41a8151560 2023-10-16T12:11:58.312 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:11:58.313 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2023-10-16T12:11:58.313 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:11:58.313 INFO:teuthology.orchestra.run.smithi016.stdout: "poolnum": 1, 2023-10-16T12:11:58.313 INFO:teuthology.orchestra.run.smithi016.stdout: "poolname": ".mgr" 2023-10-16T12:11:58.313 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:11:58.313 INFO:teuthology.orchestra.run.smithi016.stdout:] 2023-10-16T12:11:58.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.312+0000 7f41afd3d700 1 -- 172.21.15.16:0/2378565285 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f418c06af00 msgr2=0x7f418c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:58.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.312+0000 7f41afd3d700 1 --2- 172.21.15.16:0/2378565285 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f418c06af00 0x7f418c06d3c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f41a4007d30 tx=0x7f41a4007000 comp rx=0 tx=0).stop 2023-10-16T12:11:58.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.312+0000 7f41afd3d700 1 -- 172.21.15.16:0/2378565285 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41a8151560 msgr2=0x7f41a81f06e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:58.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.312+0000 7f41afd3d700 1 --2- 172.21.15.16:0/2378565285 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41a8151560 0x7f41a81f06e0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f419800e460 tx=0x7f419800e820 comp rx=0 tx=0).stop 2023-10-16T12:11:58.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.312+0000 7f41afd3d700 1 -- 172.21.15.16:0/2378565285 shutdown_connections 2023-10-16T12:11:58.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.312+0000 7f41afd3d700 1 --2- 172.21.15.16:0/2378565285 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f418c06af00 0x7f418c06d3c0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:58.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.312+0000 7f41afd3d700 1 --2- 172.21.15.16:0/2378565285 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f41a815ca60 0x7f41a815cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:58.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.312+0000 7f41afd3d700 1 --2- 172.21.15.16:0/2378565285 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4194003a00 0x7f41a8151e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:58.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.312+0000 7f41afd3d700 1 --2- 172.21.15.16:0/2378565285 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41a8151560 0x7f41a81f06e0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:58.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.312+0000 7f41afd3d700 1 -- 172.21.15.16:0/2378565285 >> 172.21.15.16:0/2378565285 conn(0x7f41a80d14b0 msgr2=0x7f41a80b62c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:58.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.312+0000 7f41afd3d700 1 -- 172.21.15.16:0/2378565285 shutdown_connections 2023-10-16T12:11:58.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.312+0000 7f41afd3d700 1 -- 172.21.15.16:0/2378565285 wait complete. 2023-10-16T12:11:58.326 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-16T12:11:58.327 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-16T12:11:58.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.454+0000 7fa2612f2700 1 Processor -- start 2023-10-16T12:11:58.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.454+0000 7fa2612f2700 1 -- start start 2023-10-16T12:11:58.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.455+0000 7fa2612f2700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa25c15d7a0 0x7fa25c0b7240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:58.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.455+0000 7fa2612f2700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa25c0b7780 0x7fa25c0ce260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:58.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.455+0000 7fa2612f2700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fa25c0ce7a0 0x7fa25c0b5010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:58.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.455+0000 7fa2612f2700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa25c0b8980 con 0x7fa25c15d7a0 2023-10-16T12:11:58.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.455+0000 7fa2612f2700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa25c0cec30 con 0x7fa25c0ce7a0 2023-10-16T12:11:58.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.455+0000 7fa2612f2700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa25c0b91b0 con 0x7fa25c0b7780 2023-10-16T12:11:58.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.455+0000 7fa25affd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa25c15d7a0 0x7fa25c0b7240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:58.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.455+0000 7fa25affd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa25c15d7a0 0x7fa25c0b7240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35494/0 (socket says 172.21.15.16:35494) 2023-10-16T12:11:58.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.455+0000 7fa25affd700 1 -- 172.21.15.16:0/3597087232 learned_addr learned my addr 172.21.15.16:0/3597087232 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:58.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.455+0000 7fa25a7fc700 1 --2- 172.21.15.16:0/3597087232 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa25c0b7780 0x7fa25c0ce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:58.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.455+0000 7fa25b7fe700 1 --2- 172.21.15.16:0/3597087232 >> v2:172.21.15.49:3300/0 conn(0x7fa25c0ce7a0 0x7fa25c0b5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:58.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.455+0000 7fa25affd700 1 -- 172.21.15.16:0/3597087232 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa25c0b7780 msgr2=0x7fa25c0ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:58.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.455+0000 7fa25affd700 1 --2- 172.21.15.16:0/3597087232 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa25c0b7780 0x7fa25c0ce260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:58.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.455+0000 7fa25affd700 1 -- 172.21.15.16:0/3597087232 >> v2:172.21.15.49:3300/0 conn(0x7fa25c0ce7a0 msgr2=0x7fa25c0b5010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:58.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.455+0000 7fa25affd700 1 --2- 172.21.15.16:0/3597087232 >> v2:172.21.15.49:3300/0 conn(0x7fa25c0ce7a0 0x7fa25c0b5010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:58.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.455+0000 7fa25affd700 1 -- 172.21.15.16:0/3597087232 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa25c0b7d00 con 0x7fa25c15d7a0 2023-10-16T12:11:58.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.455+0000 7fa25b7fe700 1 --2- 172.21.15.16:0/3597087232 >> v2:172.21.15.49:3300/0 conn(0x7fa25c0ce7a0 0x7fa25c0b5010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:11:58.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.455+0000 7fa25affd700 1 --2- 172.21.15.16:0/3597087232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa25c15d7a0 0x7fa25c0b7240 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fa25c0cdb70 tx=0x7fa250009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efa4b5dd0eb070d5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:58.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.456+0000 7fa259ffb700 1 -- 172.21.15.16:0/3597087232 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa25000b870 con 0x7fa25c15d7a0 2023-10-16T12:11:58.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.456+0000 7fa259ffb700 1 -- 172.21.15.16:0/3597087232 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa25000b9d0 con 0x7fa25c15d7a0 2023-10-16T12:11:58.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.456+0000 7fa259ffb700 1 -- 172.21.15.16:0/3597087232 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa25000bd60 con 0x7fa25c15d7a0 2023-10-16T12:11:58.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.456+0000 7fa2612f2700 1 -- 172.21.15.16:0/3597087232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa25c15d7a0 msgr2=0x7fa25c0b7240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:58.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.456+0000 7fa2612f2700 1 --2- 172.21.15.16:0/3597087232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa25c15d7a0 0x7fa25c0b7240 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fa25c0cdb70 tx=0x7fa250009bd0 comp rx=0 tx=0).stop 2023-10-16T12:11:58.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.456+0000 7fa2612f2700 1 -- 172.21.15.16:0/3597087232 shutdown_connections 2023-10-16T12:11:58.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.456+0000 7fa2612f2700 1 --2- 172.21.15.16:0/3597087232 >> v2:172.21.15.49:3300/0 conn(0x7fa25c0ce7a0 0x7fa25c0b5010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:58.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.456+0000 7fa2612f2700 1 --2- 172.21.15.16:0/3597087232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa25c15d7a0 0x7fa25c0b7240 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:58.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.456+0000 7fa2612f2700 1 --2- 172.21.15.16:0/3597087232 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa25c0b7780 0x7fa25c0ce260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:58.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.456+0000 7fa2612f2700 1 -- 172.21.15.16:0/3597087232 >> 172.21.15.16:0/3597087232 conn(0x7fa25c072330 msgr2=0x7fa25c072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:58.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.456+0000 7fa2612f2700 1 -- 172.21.15.16:0/3597087232 shutdown_connections 2023-10-16T12:11:58.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.456+0000 7fa2612f2700 1 -- 172.21.15.16:0/3597087232 wait complete. 2023-10-16T12:11:58.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.457+0000 7fa2612f2700 1 Processor -- start 2023-10-16T12:11:58.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.458+0000 7fa2612f2700 1 -- start start 2023-10-16T12:11:58.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.458+0000 7fa2612f2700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa25c0b7780 0x7fa25c0c8270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:58.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.458+0000 7fa2612f2700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa25c0ce7a0 0x7fa25c0b4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:58.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.458+0000 7fa2612f2700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa25c15d7a0 0x7fa25c0b4b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:58.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.458+0000 7fa2612f2700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa25c0c99e0 con 0x7fa25c0ce7a0 2023-10-16T12:11:58.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.458+0000 7fa2612f2700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa25c0c6ef0 con 0x7fa25c15d7a0 2023-10-16T12:11:58.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.458+0000 7fa2612f2700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa25c0c71d0 con 0x7fa25c0b7780 2023-10-16T12:11:58.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.458+0000 7fa25a7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa25c0ce7a0 0x7fa25c0b4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:58.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.458+0000 7fa25a7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa25c0ce7a0 0x7fa25c0b4610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35502/0 (socket says 172.21.15.16:35502) 2023-10-16T12:11:58.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.458+0000 7fa25a7fc700 1 -- 172.21.15.16:0/3408790339 learned_addr learned my addr 172.21.15.16:0/3408790339 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:58.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.458+0000 7fa25b7fe700 1 --2- 172.21.15.16:0/3408790339 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa25c15d7a0 0x7fa25c0b4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:58.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.459+0000 7fa25affd700 1 --2- 172.21.15.16:0/3408790339 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa25c0b7780 0x7fa25c0c8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:58.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.459+0000 7fa25a7fc700 1 -- 172.21.15.16:0/3408790339 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa25c0b7780 msgr2=0x7fa25c0c8270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:58.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.459+0000 7fa25a7fc700 1 --2- 172.21.15.16:0/3408790339 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa25c0b7780 0x7fa25c0c8270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:58.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.459+0000 7fa25a7fc700 1 -- 172.21.15.16:0/3408790339 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa25c15d7a0 msgr2=0x7fa25c0b4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:58.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.459+0000 7fa25a7fc700 1 --2- 172.21.15.16:0/3408790339 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa25c15d7a0 0x7fa25c0b4b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:58.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.459+0000 7fa25a7fc700 1 -- 172.21.15.16:0/3408790339 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa250009550 con 0x7fa25c0ce7a0 2023-10-16T12:11:58.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.459+0000 7fa25affd700 1 --2- 172.21.15.16:0/3408790339 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa25c0b7780 0x7fa25c0c8270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:11:58.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.459+0000 7fa25b7fe700 1 --2- 172.21.15.16:0/3408790339 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa25c15d7a0 0x7fa25c0b4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:11:58.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.459+0000 7fa25a7fc700 1 --2- 172.21.15.16:0/3408790339 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa25c0ce7a0 0x7fa25c0b4610 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fa24c00e810 tx=0x7fa24c00ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:58.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.459+0000 7fa2437fe700 1 -- 172.21.15.16:0/3408790339 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa24c013610 con 0x7fa25c0ce7a0 2023-10-16T12:11:58.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.459+0000 7fa2437fe700 1 -- 172.21.15.16:0/3408790339 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa24c013770 con 0x7fa25c0ce7a0 2023-10-16T12:11:58.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.459+0000 7fa2612f2700 1 -- 172.21.15.16:0/3408790339 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa25c0c74b0 con 0x7fa25c0ce7a0 2023-10-16T12:11:58.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.459+0000 7fa2437fe700 1 -- 172.21.15.16:0/3408790339 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa24c013b00 con 0x7fa25c0ce7a0 2023-10-16T12:11:58.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.459+0000 7fa2612f2700 1 -- 172.21.15.16:0/3408790339 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa25c0c7a70 con 0x7fa25c0ce7a0 2023-10-16T12:11:58.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.460+0000 7fa2612f2700 1 -- 172.21.15.16:0/3408790339 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa25c04ec20 con 0x7fa25c0ce7a0 2023-10-16T12:11:58.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.461+0000 7fa2437fe700 1 -- 172.21.15.16:0/3408790339 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fa24c013c60 con 0x7fa25c0ce7a0 2023-10-16T12:11:58.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.461+0000 7fa2437fe700 1 --2- 172.21.15.16:0/3408790339 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fa24806af00 0x7fa24806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:58.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.461+0000 7fa2437fe700 1 -- 172.21.15.16:0/3408790339 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fa24c089080 con 0x7fa25c0ce7a0 2023-10-16T12:11:58.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.461+0000 7fa25affd700 1 --2- 172.21.15.16:0/3408790339 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fa24806af00 0x7fa24806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:58.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.461+0000 7fa25affd700 1 --2- 172.21.15.16:0/3408790339 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fa24806af00 0x7fa24806d3c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fa250014040 tx=0x7fa250008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:58.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.465+0000 7fa2437fe700 1 -- 172.21.15.16:0/3408790339 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa24c054340 con 0x7fa25c0ce7a0 2023-10-16T12:11:58.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:58.670+0000 7fa2612f2700 1 -- 172.21.15.16:0/3408790339 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fa25c0b2570 con 0x7fa25c0ce7a0 2023-10-16T12:11:59.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:59.093+0000 7fa2437fe700 1 -- 172.21.15.16:0/3408790339 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v36) v1 ==== 138+0+0 (secure 0 0 0) 0x7fa24c057960 con 0x7fa25c0ce7a0 2023-10-16T12:11:59.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:59.096+0000 7fa2612f2700 1 -- 172.21.15.16:0/3408790339 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fa24806af00 msgr2=0x7fa24806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:59.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:59.096+0000 7fa2612f2700 1 --2- 172.21.15.16:0/3408790339 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fa24806af00 0x7fa24806d3c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fa250014040 tx=0x7fa250008030 comp rx=0 tx=0).stop 2023-10-16T12:11:59.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:59.096+0000 7fa2612f2700 1 -- 172.21.15.16:0/3408790339 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa25c0ce7a0 msgr2=0x7fa25c0b4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:59.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:59.096+0000 7fa2612f2700 1 --2- 172.21.15.16:0/3408790339 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa25c0ce7a0 0x7fa25c0b4610 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fa24c00e810 tx=0x7fa24c00ebd0 comp rx=0 tx=0).stop 2023-10-16T12:11:59.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:59.096+0000 7fa2612f2700 1 -- 172.21.15.16:0/3408790339 shutdown_connections 2023-10-16T12:11:59.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:59.096+0000 7fa2612f2700 1 --2- 172.21.15.16:0/3408790339 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fa24806af00 0x7fa24806d3c0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:59.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:59.096+0000 7fa2612f2700 1 --2- 172.21.15.16:0/3408790339 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa25c15d7a0 0x7fa25c0b4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:59.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:59.096+0000 7fa2612f2700 1 --2- 172.21.15.16:0/3408790339 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa25c0b7780 0x7fa25c0c8270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:59.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:59.096+0000 7fa2612f2700 1 --2- 172.21.15.16:0/3408790339 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa25c0ce7a0 0x7fa25c0b4610 secure :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fa24c00e810 tx=0x7fa24c00ebd0 comp rx=0 tx=0).stop 2023-10-16T12:11:59.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:59.096+0000 7fa2612f2700 1 -- 172.21.15.16:0/3408790339 >> 172.21.15.16:0/3408790339 conn(0x7fa25c072330 msgr2=0x7fa25c0d36c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:59.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:59.097+0000 7fa2612f2700 1 -- 172.21.15.16:0/3408790339 shutdown_connections 2023-10-16T12:11:59.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:59.097+0000 7fa2612f2700 1 -- 172.21.15.16:0/3408790339 wait complete. 2023-10-16T12:11:59.101 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' created 2023-10-16T12:11:59.115 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-16T12:11:59.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:59.249+0000 7f4bc7ce4700 1 Processor -- start 2023-10-16T12:11:59.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:59.249+0000 7f4bc7ce4700 1 -- start start 2023-10-16T12:11:59.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:59.250+0000 7f4bc7ce4700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4bc00cb880 0x7f4bc00cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:59.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:59.250+0000 7f4bc7ce4700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4bc006da60 0x7f4bc006def0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:59.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:59.250+0000 7f4bc7ce4700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bc00cef70 con 0x7f4bc015b070 2023-10-16T12:11:59.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:59.250+0000 7f4bc7ce4700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bc006e430 con 0x7f4bc00cb880 2023-10-16T12:11:59.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:59.250+0000 7f4bc7ce4700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc00ce360 con 0x7f4bc006da60 2023-10-16T12:11:59.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:59.250+0000 7f4bc527f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4bc00cb880 0x7f4bc00cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:59.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:59.250+0000 7f4bc527f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4bc00cb880 0x7f4bc00cbd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35508/0 (socket says 172.21.15.16:35508) 2023-10-16T12:11:59.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:59.250+0000 7f4bc527f700 1 -- 172.21.15.16:0/2696119617 learned_addr learned my addr 172.21.15.16:0/2696119617 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:59.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:59.251+0000 7f4bc6281700 1 --2- 172.21.15.16:0/2696119617 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4bc006da60 0x7f4bc006def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:59.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:59.251+0000 7f4bc527f700 1 -- 172.21.15.16:0/2696119617 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4bc006da60 msgr2=0x7f4bc006def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:11:59.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:59.251+0000 7f4bc527f700 1 --2- 172.21.15.16:0/2696119617 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4bc006da60 0x7f4bc006def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:59.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:59.251+0000 7f4bc527f700 1 -- 172.21.15.16:0/2696119617 >> v1:172.21.15.16:6789/0 conn(0x7f4bc015b070 legacy=0x7f4bc015b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:59.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:59.251+0000 7f4bc527f700 1 -- 172.21.15.16:0/2696119617 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bc00ce640 con 0x7f4bc00cb880 2023-10-16T12:11:59.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:59.251+0000 7f4bc4a7e700 1 -- 172.21.15.16:0/2696119617 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2538563611 0 0) 0x7f4bc00cef70 con 0x7f4bc015b070 2023-10-16T12:11:59.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:59.251+0000 7f4bc527f700 1 --2- 172.21.15.16:0/2696119617 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bc00cb880 0x7f4bc00cbd00 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc00bb90 tx=0x7f4bbc00bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=226f8ac1083f8569 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:59.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:59.251+0000 7f4bc4a7e700 1 -- 172.21.15.16:0/2696119617 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bbc00dd50 con 0x7f4bc00cb880 2023-10-16T12:11:59.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:59.251+0000 7f4bc4a7e700 1 -- 172.21.15.16:0/2696119617 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bbc010470 con 0x7f4bc00cb880 2023-10-16T12:11:59.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:59.251+0000 7f4bc4a7e700 1 -- 172.21.15.16:0/2696119617 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bbc010800 con 0x7f4bc00cb880 2023-10-16T12:11:59.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:59.252+0000 7f4bc7ce4700 1 -- 172.21.15.16:0/2696119617 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bc00cb880 msgr2=0x7f4bc00cbd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:59.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:59.252+0000 7f4bc7ce4700 1 --2- 172.21.15.16:0/2696119617 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bc00cb880 0x7f4bc00cbd00 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc00bb90 tx=0x7f4bbc00bea0 comp rx=0 tx=0).stop 2023-10-16T12:11:59.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:59.252+0000 7f4bc7ce4700 1 -- 172.21.15.16:0/2696119617 shutdown_connections 2023-10-16T12:11:59.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:59.252+0000 7f4bc7ce4700 1 --2- 172.21.15.16:0/2696119617 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4bc006da60 0x7f4bc006def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:59.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:59.252+0000 7f4bc7ce4700 1 --2- 172.21.15.16:0/2696119617 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bc00cb880 0x7f4bc00cbd00 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:59.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:59.252+0000 7f4bc7ce4700 1 -- 172.21.15.16:0/2696119617 >> 172.21.15.16:0/2696119617 conn(0x7f4bc00b1f40 msgr2=0x7f4bc00b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:11:59.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:59.252+0000 7f4bc7ce4700 1 -- 172.21.15.16:0/2696119617 shutdown_connections 2023-10-16T12:11:59.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:59.252+0000 7f4bc7ce4700 1 -- 172.21.15.16:0/2696119617 wait complete. 2023-10-16T12:11:59.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:59.253+0000 7f4bc7ce4700 1 Processor -- start 2023-10-16T12:11:59.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:59.253+0000 7f4bc7ce4700 1 -- start start 2023-10-16T12:11:59.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:59.254+0000 7f4bc7ce4700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4bc006da60 0x7f4bc01538a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:59.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:59.254+0000 7f4bc7ce4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bc00cb880 0x7f4bc01577a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:59.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:59.254+0000 7f4bc7ce4700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4bc015b070 0x7f4bc0158ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:59.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:59.254+0000 7f4bc7ce4700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc0158130 con 0x7f4bc00cb880 2023-10-16T12:11:59.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:59.254+0000 7f4bc7ce4700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc00aca70 con 0x7f4bc015b070 2023-10-16T12:11:59.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:59.254+0000 7f4bc7ce4700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc00acd50 con 0x7f4bc006da60 2023-10-16T12:11:59.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:59.254+0000 7f4bc527f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bc00cb880 0x7f4bc01577a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:59.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:59.254+0000 7f4bc527f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bc00cb880 0x7f4bc01577a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35514/0 (socket says 172.21.15.16:35514) 2023-10-16T12:11:59.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:59.255+0000 7f4bc527f700 1 -- 172.21.15.16:0/1267868610 learned_addr learned my addr 172.21.15.16:0/1267868610 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:11:59.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:59.255+0000 7f4bc5a80700 1 --2- 172.21.15.16:0/1267868610 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4bc006da60 0x7f4bc01538a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:59.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:59.255+0000 7f4bc527f700 1 -- 172.21.15.16:0/1267868610 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4bc006da60 msgr2=0x7f4bc01538a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:59.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:59.255+0000 7f4bc527f700 1 --2- 172.21.15.16:0/1267868610 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4bc006da60 0x7f4bc01538a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:59.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:59.255+0000 7f4bc527f700 1 -- 172.21.15.16:0/1267868610 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4bc015b070 msgr2=0x7f4bc0158ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:11:59.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:59.255+0000 7f4bc527f700 1 --2- 172.21.15.16:0/1267868610 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4bc015b070 0x7f4bc0158ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:11:59.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:59.255+0000 7f4bc527f700 1 -- 172.21.15.16:0/1267868610 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bbc00b7f0 con 0x7f4bc00cb880 2023-10-16T12:11:59.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:59.255+0000 7f4bc527f700 1 --2- 172.21.15.16:0/1267868610 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bc00cb880 0x7f4bc01577a0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc00bee0 tx=0x7f4bbc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:59.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:59.255+0000 7f4bb67fc700 1 -- 172.21.15.16:0/1267868610 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bbc003bf0 con 0x7f4bc00cb880 2023-10-16T12:11:59.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:59.255+0000 7f4bb67fc700 1 -- 172.21.15.16:0/1267868610 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bbc003d50 con 0x7f4bc00cb880 2023-10-16T12:11:59.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:59.255+0000 7f4bb67fc700 1 -- 172.21.15.16:0/1267868610 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bbc0107e0 con 0x7f4bc00cb880 2023-10-16T12:11:59.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:59.255+0000 7f4bc7ce4700 1 -- 172.21.15.16:0/1267868610 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bc00ad030 con 0x7f4bc00cb880 2023-10-16T12:11:59.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:59.255+0000 7f4bc7ce4700 1 -- 172.21.15.16:0/1267868610 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bc0153de0 con 0x7f4bc00cb880 2023-10-16T12:11:59.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:59.256+0000 7f4bc7ce4700 1 -- 172.21.15.16:0/1267868610 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bc00ad550 con 0x7f4bc00cb880 2023-10-16T12:11:59.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:59.257+0000 7f4bb67fc700 1 -- 172.21.15.16:0/1267868610 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f4bbc019400 con 0x7f4bc00cb880 2023-10-16T12:11:59.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:59.257+0000 7f4bb67fc700 1 --2- 172.21.15.16:0/1267868610 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f4ba406af00 0x7f4ba406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:11:59.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:59.257+0000 7f4bb67fc700 1 -- 172.21.15.16:0/1267868610 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5547+0+0 (secure 0 0 0) 0x7f4bbc08afa0 con 0x7f4bc00cb880 2023-10-16T12:11:59.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:59.257+0000 7f4bc5a80700 1 --2- 172.21.15.16:0/1267868610 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f4ba406af00 0x7f4ba406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:11:59.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:59.258+0000 7f4bc5a80700 1 --2- 172.21.15.16:0/1267868610 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f4ba406af00 0x7f4ba406d3c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f4bb0000f60 tx=0x7f4bb0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:11:59.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:59.262+0000 7f4bb67fc700 1 -- 172.21.15.16:0/1267868610 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4bbc056160 con 0x7f4bc00cb880 2023-10-16T12:11:59.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:11:59.464+0000 7f4bc7ce4700 1 -- 172.21.15.16:0/1267868610 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f4bc00ad550 con 0x7f4bc00cb880 2023-10-16T12:12:00.015 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b has been restored 2023-10-16T12:12:00.016 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d has been restored 2023-10-16T12:12:00.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:00.098+0000 7f4bb67fc700 1 -- 172.21.15.16:0/1267868610 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v37) v1 ==== 170+0+0 (secure 0 0 0) 0x7f4bc00ad550 con 0x7f4bc00cb880 2023-10-16T12:12:00.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:00.104+0000 7f4bc7ce4700 1 -- 172.21.15.16:0/1267868610 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f4ba406af00 msgr2=0x7f4ba406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:00.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:00.104+0000 7f4bc7ce4700 1 --2- 172.21.15.16:0/1267868610 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f4ba406af00 0x7f4ba406d3c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f4bb0000f60 tx=0x7f4bb0009040 comp rx=0 tx=0).stop 2023-10-16T12:12:00.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:00.104+0000 7f4bc7ce4700 1 -- 172.21.15.16:0/1267868610 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bc00cb880 msgr2=0x7f4bc01577a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:00.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:00.104+0000 7f4bc7ce4700 1 --2- 172.21.15.16:0/1267868610 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bc00cb880 0x7f4bc01577a0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc00bee0 tx=0x7f4bbc004750 comp rx=0 tx=0).stop 2023-10-16T12:12:00.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:00.105+0000 7f4bc7ce4700 1 -- 172.21.15.16:0/1267868610 shutdown_connections 2023-10-16T12:12:00.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:00.105+0000 7f4bc7ce4700 1 --2- 172.21.15.16:0/1267868610 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f4ba406af00 0x7f4ba406d3c0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:00.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:00.105+0000 7f4bc7ce4700 1 --2- 172.21.15.16:0/1267868610 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4bc015b070 0x7f4bc0158ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:00.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:00.105+0000 7f4bc7ce4700 1 --2- 172.21.15.16:0/1267868610 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4bc006da60 0x7f4bc01538a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:00.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:00.105+0000 7f4bc7ce4700 1 --2- 172.21.15.16:0/1267868610 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bc00cb880 0x7f4bc01577a0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:00.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:00.106+0000 7f4bc7ce4700 1 -- 172.21.15.16:0/1267868610 >> 172.21.15.16:0/1267868610 conn(0x7f4bc00b1f40 msgr2=0x7f4bc015a040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:00.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:00.106+0000 7f4bc7ce4700 1 -- 172.21.15.16:0/1267868610 shutdown_connections 2023-10-16T12:12:00.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:00.106+0000 7f4bc7ce4700 1 -- 172.21.15.16:0/1267868610 wait complete. 2023-10-16T12:12:00.119 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' created 2023-10-16T12:12:00.134 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-16T12:12:00.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:00.267+0000 7f8e15ae1700 1 Processor -- start 2023-10-16T12:12:00.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:00.267+0000 7f8e15ae1700 1 -- start start 2023-10-16T12:12:00.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:00.268+0000 7f8e15ae1700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f8e10072b30 0x7f8e10071a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:00.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:00.268+0000 7f8e15ae1700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8e1015bb40 0x7f8e1006e310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:00.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:00.268+0000 7f8e15ae1700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e1015c040 con 0x7f8e100caa40 2023-10-16T12:12:00.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:00.268+0000 7f8e15ae1700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e1015c320 con 0x7f8e10072b30 2023-10-16T12:12:00.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:00.268+0000 7f8e15ae1700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e1015c600 con 0x7f8e1015bb40 2023-10-16T12:12:00.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:00.268+0000 7f8e0ffff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f8e100caa40 0x7f8e100ccf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:53030/0 (socket says 172.21.15.16:53030) 2023-10-16T12:12:00.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:00.268+0000 7f8e0ffff700 1 -- 172.21.15.16:0/3959482167 learned_addr learned my addr 172.21.15.16:0/3959482167 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:12:00.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:00.268+0000 7f8e0effd700 1 --2- 172.21.15.16:0/3959482167 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8e1015bb40 0x7f8e1006e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:00.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:00.268+0000 7f8e0f7fe700 1 --2- 172.21.15.16:0/3959482167 >> v2:172.21.15.49:3300/0 conn(0x7f8e10072b30 0x7f8e10071a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:00.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:00.268+0000 7f8e0e7fc700 1 -- 172.21.15.16:0/3959482167 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3752416055 0 0) 0x7f8e1015c040 con 0x7f8e100caa40 2023-10-16T12:12:00.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:00.268+0000 7f8e0e7fc700 1 -- 172.21.15.16:0/3959482167 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8df80035b0 con 0x7f8e100caa40 2023-10-16T12:12:00.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:00.269+0000 7f8e0effd700 1 -- 172.21.15.16:0/3959482167 >> v2:172.21.15.49:3300/0 conn(0x7f8e10072b30 msgr2=0x7f8e10071a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:00.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:00.269+0000 7f8e0effd700 1 --2- 172.21.15.16:0/3959482167 >> v2:172.21.15.49:3300/0 conn(0x7f8e10072b30 0x7f8e10071a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:00.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:00.269+0000 7f8e0effd700 1 -- 172.21.15.16:0/3959482167 >> v1:172.21.15.16:6789/0 conn(0x7f8e100caa40 legacy=0x7f8e100ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:00.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:00.269+0000 7f8e0effd700 1 -- 172.21.15.16:0/3959482167 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e1015c8e0 con 0x7f8e1015bb40 2023-10-16T12:12:00.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:00.269+0000 7f8e0f7fe700 1 --2- 172.21.15.16:0/3959482167 >> v2:172.21.15.49:3300/0 conn(0x7f8e10072b30 0x7f8e10071a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:12:00.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:00.269+0000 7f8e0effd700 1 --2- 172.21.15.16:0/3959482167 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8e1015bb40 0x7f8e1006e310 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8e04009840 tx=0x7f8e04009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=22fa91b3d4fd9f90 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:00.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:00.269+0000 7f8e0e7fc700 1 -- 172.21.15.16:0/3959482167 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e04011830 con 0x7f8e1015bb40 2023-10-16T12:12:00.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:00.269+0000 7f8e0e7fc700 1 -- 172.21.15.16:0/3959482167 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e04011990 con 0x7f8e1015bb40 2023-10-16T12:12:00.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:00.269+0000 7f8e0e7fc700 1 -- 172.21.15.16:0/3959482167 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e04011d20 con 0x7f8e1015bb40 2023-10-16T12:12:00.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:00.270+0000 7f8e15ae1700 1 -- 172.21.15.16:0/3959482167 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8e1015bb40 msgr2=0x7f8e1006e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:00.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:00.270+0000 7f8e15ae1700 1 --2- 172.21.15.16:0/3959482167 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8e1015bb40 0x7f8e1006e310 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8e04009840 tx=0x7f8e04009b50 comp rx=0 tx=0).stop 2023-10-16T12:12:00.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:00.270+0000 7f8e15ae1700 1 -- 172.21.15.16:0/3959482167 shutdown_connections 2023-10-16T12:12:00.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:00.270+0000 7f8e15ae1700 1 --2- 172.21.15.16:0/3959482167 >> v2:172.21.15.49:3300/0 conn(0x7f8e10072b30 0x7f8e10071a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:00.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:00.270+0000 7f8e15ae1700 1 --2- 172.21.15.16:0/3959482167 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8e1015bb40 0x7f8e1006e310 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:00.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:00.270+0000 7f8e15ae1700 1 -- 172.21.15.16:0/3959482167 >> 172.21.15.16:0/3959482167 conn(0x7f8e100b1f40 msgr2=0x7f8e100b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:00.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:00.270+0000 7f8e15ae1700 1 -- 172.21.15.16:0/3959482167 shutdown_connections 2023-10-16T12:12:00.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:00.270+0000 7f8e15ae1700 1 -- 172.21.15.16:0/3959482167 wait complete. 2023-10-16T12:12:00.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:00.271+0000 7f8e15ae1700 1 Processor -- start 2023-10-16T12:12:00.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:00.271+0000 7f8e15ae1700 1 -- start start 2023-10-16T12:12:00.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:00.272+0000 7f8e15ae1700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8e10072b30 0x7f8e10154ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:00.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:00.272+0000 7f8e15ae1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e100caa40 0x7f8e101589b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:00.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:00.272+0000 7f8e15ae1700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8e1015bb40 0x7f8e100b3030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:00.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:00.272+0000 7f8e15ae1700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e100b5250 con 0x7f8e100caa40 2023-10-16T12:12:00.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:00.272+0000 7f8e15ae1700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e100b5510 con 0x7f8e10072b30 2023-10-16T12:12:00.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:00.272+0000 7f8e15ae1700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e100b57f0 con 0x7f8e1015bb40 2023-10-16T12:12:00.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:00.272+0000 7f8e0effd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e100caa40 0x7f8e101589b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:00.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:00.272+0000 7f8e0effd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e100caa40 0x7f8e101589b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35528/0 (socket says 172.21.15.16:35528) 2023-10-16T12:12:00.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:00.272+0000 7f8e0effd700 1 -- 172.21.15.16:0/975514841 learned_addr learned my addr 172.21.15.16:0/975514841 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:12:00.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:00.272+0000 7f8e0f7fe700 1 --2- 172.21.15.16:0/975514841 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8e10072b30 0x7f8e10154ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:00.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:00.272+0000 7f8e0ffff700 1 --2- 172.21.15.16:0/975514841 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8e1015bb40 0x7f8e100b3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:00.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:00.272+0000 7f8e0effd700 1 -- 172.21.15.16:0/975514841 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8e1015bb40 msgr2=0x7f8e100b3030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:00.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:00.272+0000 7f8e0effd700 1 --2- 172.21.15.16:0/975514841 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8e1015bb40 0x7f8e100b3030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:00.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:00.272+0000 7f8e0effd700 1 -- 172.21.15.16:0/975514841 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8e10072b30 msgr2=0x7f8e10154ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:00.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:00.272+0000 7f8e0effd700 1 --2- 172.21.15.16:0/975514841 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8e10072b30 0x7f8e10154ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:00.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:00.272+0000 7f8e0effd700 1 -- 172.21.15.16:0/975514841 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e00009a70 con 0x7f8e100caa40 2023-10-16T12:12:00.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:00.272+0000 7f8e0effd700 1 --2- 172.21.15.16:0/975514841 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e100caa40 0x7f8e101589b0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f8e040090b0 tx=0x7f8e04004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:00.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:00.273+0000 7f8df7fff700 1 -- 172.21.15.16:0/975514841 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e04004590 con 0x7f8e100caa40 2023-10-16T12:12:00.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:00.273+0000 7f8df7fff700 1 -- 172.21.15.16:0/975514841 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e0401f070 con 0x7f8e100caa40 2023-10-16T12:12:00.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:00.273+0000 7f8e15ae1700 1 -- 172.21.15.16:0/975514841 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e040094a0 con 0x7f8e100caa40 2023-10-16T12:12:00.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:00.273+0000 7f8df7fff700 1 -- 172.21.15.16:0/975514841 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e040038e0 con 0x7f8e100caa40 2023-10-16T12:12:00.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:00.273+0000 7f8e15ae1700 1 -- 172.21.15.16:0/975514841 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e100b5db0 con 0x7f8e100caa40 2023-10-16T12:12:00.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:00.274+0000 7f8e15ae1700 1 -- 172.21.15.16:0/975514841 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e1004ebb0 con 0x7f8e100caa40 2023-10-16T12:12:00.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:00.274+0000 7f8df7fff700 1 -- 172.21.15.16:0/975514841 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f8e04003a40 con 0x7f8e100caa40 2023-10-16T12:12:00.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:00.274+0000 7f8df7fff700 1 --2- 172.21.15.16:0/975514841 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f8dfc06af00 0x7f8dfc06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:00.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:00.275+0000 7f8df7fff700 1 -- 172.21.15.16:0/975514841 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5929+0+0 (secure 0 0 0) 0x7f8e04021070 con 0x7f8e100caa40 2023-10-16T12:12:00.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:00.275+0000 7f8e0f7fe700 1 --2- 172.21.15.16:0/975514841 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f8dfc06af00 0x7f8dfc06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:00.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:00.275+0000 7f8e0f7fe700 1 --2- 172.21.15.16:0/975514841 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f8dfc06af00 0x7f8dfc06d3c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f8e000114e0 tx=0x7f8e0000b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:00.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:00.279+0000 7f8df7fff700 1 -- 172.21.15.16:0/975514841 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8e04055e10 con 0x7f8e100caa40 2023-10-16T12:12:00.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:00.535+0000 7f8e15ae1700 1 -- 172.21.15.16:0/975514841 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f8e100b2d40 con 0x7f8e100caa40 2023-10-16T12:12:01.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.100+0000 7f8df7fff700 1 -- 172.21.15.16:0/975514841 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 5 and data pool 6 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7f8e040150c0 con 0x7f8e100caa40 2023-10-16T12:12:01.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.104+0000 7f8e15ae1700 1 -- 172.21.15.16:0/975514841 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f8dfc06af00 msgr2=0x7f8dfc06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:01.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.104+0000 7f8e15ae1700 1 --2- 172.21.15.16:0/975514841 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f8dfc06af00 0x7f8dfc06d3c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f8e000114e0 tx=0x7f8e0000b040 comp rx=0 tx=0).stop 2023-10-16T12:12:01.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.104+0000 7f8e15ae1700 1 -- 172.21.15.16:0/975514841 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e100caa40 msgr2=0x7f8e101589b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:01.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.104+0000 7f8e15ae1700 1 --2- 172.21.15.16:0/975514841 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e100caa40 0x7f8e101589b0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f8e040090b0 tx=0x7f8e04004750 comp rx=0 tx=0).stop 2023-10-16T12:12:01.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.104+0000 7f8e15ae1700 1 -- 172.21.15.16:0/975514841 shutdown_connections 2023-10-16T12:12:01.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.104+0000 7f8e15ae1700 1 --2- 172.21.15.16:0/975514841 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f8dfc06af00 0x7f8dfc06d3c0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:01.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.104+0000 7f8e15ae1700 1 --2- 172.21.15.16:0/975514841 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8e1015bb40 0x7f8e100b3030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:01.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.104+0000 7f8e15ae1700 1 --2- 172.21.15.16:0/975514841 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8e10072b30 0x7f8e10154ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:01.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.104+0000 7f8e15ae1700 1 --2- 172.21.15.16:0/975514841 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e100caa40 0x7f8e101589b0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:01.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.104+0000 7f8e15ae1700 1 -- 172.21.15.16:0/975514841 >> 172.21.15.16:0/975514841 conn(0x7f8e100b1f40 msgr2=0x7f8e100b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:01.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.104+0000 7f8e15ae1700 1 -- 172.21.15.16:0/975514841 shutdown_connections 2023-10-16T12:12:01.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.105+0000 7f8e15ae1700 1 -- 172.21.15.16:0/975514841 wait complete. 2023-10-16T12:12:01.109 INFO:teuthology.orchestra.run.smithi016.stderr:new fs with metadata pool 5 and data pool 6 2023-10-16T12:12:01.124 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:12:01.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.251+0000 7f1637b11700 1 Processor -- start 2023-10-16T12:12:01.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.251+0000 7f1637b11700 1 -- start start 2023-10-16T12:12:01.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.252+0000 7f1637b11700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f16300cdfa0 0x7f16300b2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:01.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.252+0000 7f1637b11700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f16300ce380 0x7f16300b2980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:01.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.252+0000 7f1637b11700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f16300d04a0 0x7f16300d08d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:01.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.252+0000 7f1637b11700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16300d1210 con 0x7f16300d04a0 2023-10-16T12:12:01.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.252+0000 7f1637b11700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16300cbc30 con 0x7f16300cdfa0 2023-10-16T12:12:01.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.252+0000 7f1637b11700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16300cbf10 con 0x7f16300ce380 2023-10-16T12:12:01.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.252+0000 7f16360ae700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f16300d04a0 0x7f16300d08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:01.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.252+0000 7f16360ae700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f16300d04a0 0x7f16300d08d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35538/0 (socket says 172.21.15.16:35538) 2023-10-16T12:12:01.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.252+0000 7f16360ae700 1 -- 172.21.15.16:0/1015029790 learned_addr learned my addr 172.21.15.16:0/1015029790 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:12:01.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.252+0000 7f16350ac700 1 --2- 172.21.15.16:0/1015029790 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f16300ce380 0x7f16300b2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:01.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.252+0000 7f16358ad700 1 --2- 172.21.15.16:0/1015029790 >> v2:172.21.15.49:3300/0 conn(0x7f16300cdfa0 0x7f16300b2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:01.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.252+0000 7f16360ae700 1 -- 172.21.15.16:0/1015029790 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f16300ce380 msgr2=0x7f16300b2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:01.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.252+0000 7f16360ae700 1 --2- 172.21.15.16:0/1015029790 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f16300ce380 0x7f16300b2980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:01.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.252+0000 7f16360ae700 1 -- 172.21.15.16:0/1015029790 >> v2:172.21.15.49:3300/0 conn(0x7f16300cdfa0 msgr2=0x7f16300b2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:01.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.252+0000 7f16360ae700 1 --2- 172.21.15.16:0/1015029790 >> v2:172.21.15.49:3300/0 conn(0x7f16300cdfa0 0x7f16300b2440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:01.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.252+0000 7f16360ae700 1 -- 172.21.15.16:0/1015029790 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16300cc1f0 con 0x7f16300d04a0 2023-10-16T12:12:01.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.253+0000 7f16360ae700 1 --2- 172.21.15.16:0/1015029790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16300d04a0 0x7f16300d08d0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f1630067350 tx=0x7f162800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8ab67d677b2884d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:01.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.253+0000 7f16348ab700 1 -- 172.21.15.16:0/1015029790 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f162800e070 con 0x7f16300d04a0 2023-10-16T12:12:01.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.253+0000 7f16348ab700 1 -- 172.21.15.16:0/1015029790 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f162800b9c0 con 0x7f16300d04a0 2023-10-16T12:12:01.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.253+0000 7f16348ab700 1 -- 172.21.15.16:0/1015029790 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f162800bd70 con 0x7f16300d04a0 2023-10-16T12:12:01.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.253+0000 7f1637b11700 1 -- 172.21.15.16:0/1015029790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16300d04a0 msgr2=0x7f16300d08d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:01.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.253+0000 7f1637b11700 1 --2- 172.21.15.16:0/1015029790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16300d04a0 0x7f16300d08d0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f1630067350 tx=0x7f162800b330 comp rx=0 tx=0).stop 2023-10-16T12:12:01.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.253+0000 7f1637b11700 1 -- 172.21.15.16:0/1015029790 shutdown_connections 2023-10-16T12:12:01.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.253+0000 7f1637b11700 1 --2- 172.21.15.16:0/1015029790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16300d04a0 0x7f16300d08d0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:01.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.253+0000 7f1637b11700 1 --2- 172.21.15.16:0/1015029790 >> v2:172.21.15.49:3300/0 conn(0x7f16300cdfa0 0x7f16300b2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:01.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.253+0000 7f1637b11700 1 --2- 172.21.15.16:0/1015029790 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f16300ce380 0x7f16300b2980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:01.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.253+0000 7f1637b11700 1 -- 172.21.15.16:0/1015029790 >> 172.21.15.16:0/1015029790 conn(0x7f16300d14b0 msgr2=0x7f16300d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:01.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.253+0000 7f1637b11700 1 -- 172.21.15.16:0/1015029790 shutdown_connections 2023-10-16T12:12:01.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.253+0000 7f1637b11700 1 -- 172.21.15.16:0/1015029790 wait complete. 2023-10-16T12:12:01.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.255+0000 7f1637b11700 1 Processor -- start 2023-10-16T12:12:01.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.255+0000 7f1637b11700 1 -- start start 2023-10-16T12:12:01.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.255+0000 7f1637b11700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f16300cdfa0 0x7f163014ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:01.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.255+0000 7f1637b11700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f16300ce380 0x7f163009b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:01.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.255+0000 7f1637b11700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16300d04a0 0x7f163009b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:01.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.255+0000 7f1637b11700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f163014f0f0 con 0x7f16300d04a0 2023-10-16T12:12:01.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.255+0000 7f1637b11700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16300c6d60 con 0x7f16300cdfa0 2023-10-16T12:12:01.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.255+0000 7f1637b11700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16300c7040 con 0x7f16300ce380 2023-10-16T12:12:01.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.255+0000 7f16360ae700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16300d04a0 0x7f163009b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:01.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.256+0000 7f16360ae700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16300d04a0 0x7f163009b670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35542/0 (socket says 172.21.15.16:35542) 2023-10-16T12:12:01.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.256+0000 7f16360ae700 1 -- 172.21.15.16:0/2049673701 learned_addr learned my addr 172.21.15.16:0/2049673701 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:12:01.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.256+0000 7f16358ad700 1 --2- 172.21.15.16:0/2049673701 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f16300cdfa0 0x7f163014ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:01.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.256+0000 7f16350ac700 1 --2- 172.21.15.16:0/2049673701 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f16300ce380 0x7f163009b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:01.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.256+0000 7f16360ae700 1 -- 172.21.15.16:0/2049673701 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f16300ce380 msgr2=0x7f163009b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:01.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.256+0000 7f16360ae700 1 --2- 172.21.15.16:0/2049673701 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f16300ce380 0x7f163009b130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:01.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.256+0000 7f16360ae700 1 -- 172.21.15.16:0/2049673701 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f16300cdfa0 msgr2=0x7f163014ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:01.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.256+0000 7f16360ae700 1 --2- 172.21.15.16:0/2049673701 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f16300cdfa0 0x7f163014ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:01.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.256+0000 7f16360ae700 1 -- 172.21.15.16:0/2049673701 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1628009b30 con 0x7f16300d04a0 2023-10-16T12:12:01.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.256+0000 7f16358ad700 1 --2- 172.21.15.16:0/2049673701 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f16300cdfa0 0x7f163014ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:12:01.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.256+0000 7f16350ac700 1 --2- 172.21.15.16:0/2049673701 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f16300ce380 0x7f163009b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:12:01.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.256+0000 7f16360ae700 1 --2- 172.21.15.16:0/2049673701 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16300d04a0 0x7f163009b670 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f162800b370 tx=0x7f1628004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:01.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.256+0000 7f16267fc700 1 -- 172.21.15.16:0/2049673701 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1628004310 con 0x7f16300d04a0 2023-10-16T12:12:01.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.256+0000 7f16267fc700 1 -- 172.21.15.16:0/2049673701 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1628004470 con 0x7f16300d04a0 2023-10-16T12:12:01.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.256+0000 7f1637b11700 1 -- 172.21.15.16:0/2049673701 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16300c7320 con 0x7f16300d04a0 2023-10-16T12:12:01.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.256+0000 7f16267fc700 1 -- 172.21.15.16:0/2049673701 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1628004820 con 0x7f16300d04a0 2023-10-16T12:12:01.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.256+0000 7f1637b11700 1 -- 172.21.15.16:0/2049673701 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16300c78e0 con 0x7f16300d04a0 2023-10-16T12:12:01.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.257+0000 7f1637b11700 1 -- 172.21.15.16:0/2049673701 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f163014f400 con 0x7f16300d04a0 2023-10-16T12:12:01.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.258+0000 7f16267fc700 1 -- 172.21.15.16:0/2049673701 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f162801b430 con 0x7f16300d04a0 2023-10-16T12:12:01.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.258+0000 7f16267fc700 1 --2- 172.21.15.16:0/2049673701 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f161406af00 0x7f161406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:01.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.258+0000 7f16267fc700 1 -- 172.21.15.16:0/2049673701 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f162808a9a0 con 0x7f16300d04a0 2023-10-16T12:12:01.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.258+0000 7f16358ad700 1 --2- 172.21.15.16:0/2049673701 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f161406af00 0x7f161406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:01.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.259+0000 7f16358ad700 1 --2- 172.21.15.16:0/2049673701 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f161406af00 0x7f161406d3c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f16300c7c50 tx=0x7f1620008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:01.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.263+0000 7f16267fc700 1 -- 172.21.15.16:0/2049673701 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f163014f400 con 0x7f16300d04a0 2023-10-16T12:12:01.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.466+0000 7f1637b11700 1 -- 172.21.15.16:0/2049673701 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f163014f400 con 0x7f16300d04a0 2023-10-16T12:12:01.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.467+0000 7f16267fc700 1 -- 172.21.15.16:0/2049673701 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) v1 ==== 74+0+16679 (secure 0 0 0) 0x7f163014f400 con 0x7f16300d04a0 2023-10-16T12:12:01.470 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:12:01.470 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":38,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:12:00.537206+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:11:58.700868+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2023-10-16T12:11:59.495384+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:12:01.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.469+0000 7f1637b11700 1 -- 172.21.15.16:0/2049673701 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f161406af00 msgr2=0x7f161406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:01.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.469+0000 7f1637b11700 1 --2- 172.21.15.16:0/2049673701 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f161406af00 0x7f161406d3c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f16300c7c50 tx=0x7f1620008040 comp rx=0 tx=0).stop 2023-10-16T12:12:01.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.469+0000 7f1637b11700 1 -- 172.21.15.16:0/2049673701 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16300d04a0 msgr2=0x7f163009b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:01.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.469+0000 7f1637b11700 1 --2- 172.21.15.16:0/2049673701 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16300d04a0 0x7f163009b670 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f162800b370 tx=0x7f1628004ad0 comp rx=0 tx=0).stop 2023-10-16T12:12:01.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.470+0000 7f1637b11700 1 -- 172.21.15.16:0/2049673701 shutdown_connections 2023-10-16T12:12:01.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.470+0000 7f1637b11700 1 --2- 172.21.15.16:0/2049673701 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f161406af00 0x7f161406d3c0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:01.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.470+0000 7f1637b11700 1 --2- 172.21.15.16:0/2049673701 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16300d04a0 0x7f163009b670 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:01.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.470+0000 7f1637b11700 1 --2- 172.21.15.16:0/2049673701 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f16300cdfa0 0x7f163014ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:01.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.470+0000 7f1637b11700 1 --2- 172.21.15.16:0/2049673701 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f16300ce380 0x7f163009b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:01.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.470+0000 7f1637b11700 1 -- 172.21.15.16:0/2049673701 >> 172.21.15.16:0/2049673701 conn(0x7f16300d14b0 msgr2=0x7f163015fbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:01.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.470+0000 7f1637b11700 1 -- 172.21.15.16:0/2049673701 shutdown_connections 2023-10-16T12:12:01.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.470+0000 7f1637b11700 1 -- 172.21.15.16:0/2049673701 wait complete. 2023-10-16T12:12:01.484 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:12:01.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.607+0000 7f717a20d700 1 Processor -- start 2023-10-16T12:12:01.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.607+0000 7f717a20d700 1 -- start start 2023-10-16T12:12:01.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.608+0000 7f717a20d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f71740cb190 0x7f71740ae520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:01.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.608+0000 7f717a20d700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f717406b1f0 con 0x7f717415e0b0 2023-10-16T12:12:01.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.608+0000 7f717a20d700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f717406b4b0 con 0x7f7174159c90 2023-10-16T12:12:01.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.608+0000 7f717a20d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f717406b790 con 0x7f71740cb190 2023-10-16T12:12:01.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.608+0000 7f7173fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f717415e0b0 0x7f71740aea60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:53056/0 (socket says 172.21.15.16:53056) 2023-10-16T12:12:01.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.608+0000 7f7173fff700 1 -- 172.21.15.16:0/1558259670 learned_addr learned my addr 172.21.15.16:0/1558259670 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:12:01.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.608+0000 7f7172ffd700 1 --2- 172.21.15.16:0/1558259670 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f71740cb190 0x7f71740ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:01.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.609+0000 7f71727fc700 1 -- 172.21.15.16:0/1558259670 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3189269368 0 0) 0x7f717406b1f0 con 0x7f717415e0b0 2023-10-16T12:12:01.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.609+0000 7f71727fc700 1 -- 172.21.15.16:0/1558259670 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71600035b0 con 0x7f717415e0b0 2023-10-16T12:12:01.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.609+0000 7f71727fc700 1 -- 172.21.15.16:0/1558259670 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2155746993 0 0) 0x7f717406b4b0 con 0x7f7174159c90 2023-10-16T12:12:01.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.609+0000 7f71727fc700 1 -- 172.21.15.16:0/1558259670 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f717406b1f0 con 0x7f7174159c90 2023-10-16T12:12:01.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.609+0000 7f71727fc700 1 -- 172.21.15.16:0/1558259670 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3661457636 0 0) 0x7f71600035b0 con 0x7f717415e0b0 2023-10-16T12:12:01.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.609+0000 7f71727fc700 1 -- 172.21.15.16:0/1558259670 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f71740cb190 msgr2=0x7f71740ae520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:01.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.609+0000 7f71727fc700 1 --2- 172.21.15.16:0/1558259670 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f71740cb190 0x7f71740ae520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:01.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.609+0000 7f71727fc700 1 -- 172.21.15.16:0/1558259670 >> v1:172.21.15.49:6789/0 conn(0x7f7174159c90 legacy=0x7f717415d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:01.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.609+0000 7f71727fc700 1 -- 172.21.15.16:0/1558259670 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f717406ba70 con 0x7f717415e0b0 2023-10-16T12:12:01.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.609+0000 7f71727fc700 1 -- 172.21.15.16:0/1558259670 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f716c0019f0 con 0x7f717415e0b0 2023-10-16T12:12:01.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.609+0000 7f71727fc700 1 -- 172.21.15.16:0/1558259670 >> v1:172.21.15.16:6789/0 conn(0x7f717415e0b0 legacy=0x7f71740aea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:01.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.609+0000 7f71727fc700 1 --2- 172.21.15.16:0/1558259670 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7160003bc0 0x7f7160003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:01.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.609+0000 7f71727fc700 1 --2- 172.21.15.16:0/1558259670 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7160004850 0x7f7160004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:01.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.609+0000 7f71727fc700 1 --2- 172.21.15.16:0/1558259670 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f71740cb190 0x7f7160005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:01.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.609+0000 7f71727fc700 1 -- 172.21.15.16:0/1558259670 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f717406b790 con 0x7f7160003bc0 2023-10-16T12:12:01.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.609+0000 7f71727fc700 1 -- 172.21.15.16:0/1558259670 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7160005bf0 con 0x7f7160004850 2023-10-16T12:12:01.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.609+0000 7f71727fc700 1 -- 172.21.15.16:0/1558259670 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7160005db0 con 0x7f71740cb190 2023-10-16T12:12:01.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.609+0000 7f71737fe700 1 --2- 172.21.15.16:0/1558259670 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7160003bc0 0x7f7160003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:01.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.609+0000 7f7172ffd700 1 --2- 172.21.15.16:0/1558259670 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7160004850 0x7f7160004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:01.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.609+0000 7f7173fff700 1 --2- 172.21.15.16:0/1558259670 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f71740cb190 0x7f7160005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:01.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.609+0000 7f71737fe700 1 -- 172.21.15.16:0/1558259670 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f71740cb190 msgr2=0x7f7160005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:01.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.609+0000 7f71737fe700 1 --2- 172.21.15.16:0/1558259670 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f71740cb190 0x7f7160005230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:01.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.609+0000 7f71737fe700 1 -- 172.21.15.16:0/1558259670 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7160004850 msgr2=0x7f7160004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:01.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.609+0000 7f71737fe700 1 --2- 172.21.15.16:0/1558259670 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7160004850 0x7f7160004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:01.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.609+0000 7f71737fe700 1 -- 172.21.15.16:0/1558259670 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71600035b0 con 0x7f7160003bc0 2023-10-16T12:12:01.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.610+0000 7f71737fe700 1 --2- 172.21.15.16:0/1558259670 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7160003bc0 0x7f7160003fc0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f7164001940 tx=0x7f7160005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35d0b433da8b6d88 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:01.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.610+0000 7f71727fc700 1 -- 172.21.15.16:0/1558259670 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7164001430 con 0x7f7160003bc0 2023-10-16T12:12:01.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.610+0000 7f71727fc700 1 -- 172.21.15.16:0/1558259670 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7164001d70 con 0x7f7160003bc0 2023-10-16T12:12:01.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.610+0000 7f71727fc700 1 -- 172.21.15.16:0/1558259670 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7164005510 con 0x7f7160003bc0 2023-10-16T12:12:01.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.610+0000 7f717a20d700 1 -- 172.21.15.16:0/1558259670 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7160003bc0 msgr2=0x7f7160003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:01.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.610+0000 7f717a20d700 1 --2- 172.21.15.16:0/1558259670 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7160003bc0 0x7f7160003fc0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f7164001940 tx=0x7f7160005880 comp rx=0 tx=0).stop 2023-10-16T12:12:01.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.610+0000 7f7173fff700 1 -- 172.21.15.16:0/1558259670 reap_dead start 2023-10-16T12:12:01.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.610+0000 7f717a20d700 1 -- 172.21.15.16:0/1558259670 shutdown_connections 2023-10-16T12:12:01.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.610+0000 7f717a20d700 1 -- 172.21.15.16:0/1558259670 >> 172.21.15.16:0/1558259670 conn(0x7f71740cc5b0 msgr2=0x7f71740cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:01.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.610+0000 7f717a20d700 1 -- 172.21.15.16:0/1558259670 shutdown_connections 2023-10-16T12:12:01.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.610+0000 7f717a20d700 1 -- 172.21.15.16:0/1558259670 wait complete. 2023-10-16T12:12:01.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.612+0000 7f717a20d700 1 Processor -- start 2023-10-16T12:12:01.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.612+0000 7f717a20d700 1 -- start start 2023-10-16T12:12:01.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.612+0000 7f717a20d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7160003bc0 0x7f71740ca4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:01.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.612+0000 7f717a20d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f71741f70a0 0x7f71740c5060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:01.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.612+0000 7f717a20d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f71741f6470 0x7f71741f68f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:01.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.612+0000 7f717a20d700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71740ae0c0 con 0x7f7160003bc0 2023-10-16T12:12:01.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.612+0000 7f717a20d700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71741f6e30 con 0x7f71741f6470 2023-10-16T12:12:01.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.612+0000 7f717a20d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71740ad480 con 0x7f71741f70a0 2023-10-16T12:12:01.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.613+0000 7f71737fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7160003bc0 0x7f71740ca4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:01.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.613+0000 7f71737fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7160003bc0 0x7f71740ca4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35560/0 (socket says 172.21.15.16:35560) 2023-10-16T12:12:01.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.613+0000 7f71737fe700 1 -- 172.21.15.16:0/1776672931 learned_addr learned my addr 172.21.15.16:0/1776672931 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:12:01.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.613+0000 7f7173fff700 1 --2- 172.21.15.16:0/1776672931 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f71741f6470 0x7f71741f68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:01.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.613+0000 7f7172ffd700 1 --2- 172.21.15.16:0/1776672931 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f71741f70a0 0x7f71740c5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:01.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.613+0000 7f71737fe700 1 -- 172.21.15.16:0/1776672931 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f71741f70a0 msgr2=0x7f71740c5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:01.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.613+0000 7f71737fe700 1 --2- 172.21.15.16:0/1776672931 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f71741f70a0 0x7f71740c5060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:01.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.613+0000 7f71737fe700 1 -- 172.21.15.16:0/1776672931 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f71741f6470 msgr2=0x7f71741f68f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:01.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.613+0000 7f71737fe700 1 --2- 172.21.15.16:0/1776672931 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f71741f6470 0x7f71741f68f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:01.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.613+0000 7f71737fe700 1 -- 172.21.15.16:0/1776672931 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71740af2f0 con 0x7f7160003bc0 2023-10-16T12:12:01.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.613+0000 7f7172ffd700 1 --2- 172.21.15.16:0/1776672931 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f71741f70a0 0x7f71740c5060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:12:01.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.613+0000 7f71737fe700 1 --2- 172.21.15.16:0/1776672931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7160003bc0 0x7f71740ca4f0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f716400c3c0 tx=0x7f71640012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:01.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.613+0000 7f716bfff700 1 -- 172.21.15.16:0/1776672931 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7164006c70 con 0x7f7160003bc0 2023-10-16T12:12:01.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.613+0000 7f716bfff700 1 -- 172.21.15.16:0/1776672931 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7164006dd0 con 0x7f7160003bc0 2023-10-16T12:12:01.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.613+0000 7f717a20d700 1 -- 172.21.15.16:0/1776672931 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71740caa30 con 0x7f7160003bc0 2023-10-16T12:12:01.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.613+0000 7f716bfff700 1 -- 172.21.15.16:0/1776672931 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7164007280 con 0x7f7160003bc0 2023-10-16T12:12:01.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.613+0000 7f717a20d700 1 -- 172.21.15.16:0/1776672931 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71740ada00 con 0x7f7160003bc0 2023-10-16T12:12:01.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.614+0000 7f717a20d700 1 -- 172.21.15.16:0/1776672931 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f717404ebb0 con 0x7f7160003bc0 2023-10-16T12:12:01.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.615+0000 7f716bfff700 1 -- 172.21.15.16:0/1776672931 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f71640073e0 con 0x7f7160003bc0 2023-10-16T12:12:01.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.615+0000 7f716bfff700 1 --2- 172.21.15.16:0/1776672931 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f715406af00 0x7f715406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:01.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.615+0000 7f716bfff700 1 -- 172.21.15.16:0/1776672931 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f716408a480 con 0x7f7160003bc0 2023-10-16T12:12:01.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.615+0000 7f7172ffd700 1 --2- 172.21.15.16:0/1776672931 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f715406af00 0x7f715406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:01.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.616+0000 7f7172ffd700 1 --2- 172.21.15.16:0/1776672931 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f715406af00 0x7f715406d3c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f71740c6620 tx=0x7f715c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:01.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.620+0000 7f716bfff700 1 -- 172.21.15.16:0/1776672931 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7164055480 con 0x7f7160003bc0 2023-10-16T12:12:01.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.824+0000 7f717a20d700 1 -- 172.21.15.16:0/1776672931 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f71740ce410 con 0x7f7160003bc0 2023-10-16T12:12:01.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.825+0000 7f716bfff700 1 -- 172.21.15.16:0/1776672931 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) v1 ==== 74+0+16679 (secure 0 0 0) 0x7f7164058aa0 con 0x7f7160003bc0 2023-10-16T12:12:01.827 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:12:01.827 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":38,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:12:00.537206+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:11:58.700868+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2023-10-16T12:11:59.495384+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:12:01.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.827+0000 7f717a20d700 1 -- 172.21.15.16:0/1776672931 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f715406af00 msgr2=0x7f715406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:01.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.827+0000 7f717a20d700 1 --2- 172.21.15.16:0/1776672931 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f715406af00 0x7f715406d3c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f71740c6620 tx=0x7f715c008040 comp rx=0 tx=0).stop 2023-10-16T12:12:01.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.827+0000 7f717a20d700 1 -- 172.21.15.16:0/1776672931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7160003bc0 msgr2=0x7f71740ca4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:01.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.827+0000 7f717a20d700 1 --2- 172.21.15.16:0/1776672931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7160003bc0 0x7f71740ca4f0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f716400c3c0 tx=0x7f71640012d0 comp rx=0 tx=0).stop 2023-10-16T12:12:01.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.827+0000 7f717a20d700 1 -- 172.21.15.16:0/1776672931 shutdown_connections 2023-10-16T12:12:01.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.827+0000 7f717a20d700 1 --2- 172.21.15.16:0/1776672931 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f715406af00 0x7f715406d3c0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:01.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.827+0000 7f717a20d700 1 --2- 172.21.15.16:0/1776672931 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f71741f6470 0x7f71741f68f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:01.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.827+0000 7f717a20d700 1 --2- 172.21.15.16:0/1776672931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7160003bc0 0x7f71740ca4f0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:01.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.827+0000 7f717a20d700 1 --2- 172.21.15.16:0/1776672931 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f71741f70a0 0x7f71740c5060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:01.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.827+0000 7f717a20d700 1 -- 172.21.15.16:0/1776672931 >> 172.21.15.16:0/1776672931 conn(0x7f71740cc5b0 msgr2=0x7f71740af5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:01.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.827+0000 7f717a20d700 1 -- 172.21.15.16:0/1776672931 shutdown_connections 2023-10-16T12:12:01.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.827+0000 7f717a20d700 1 -- 172.21.15.16:0/1776672931 wait complete. 2023-10-16T12:12:01.841 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-16T12:12:01.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.968+0000 7f98925f6700 1 Processor -- start 2023-10-16T12:12:01.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.968+0000 7f98925f6700 1 -- start start 2023-10-16T12:12:01.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.969+0000 7f98925f6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f988c06e570 0x7f988c15e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:01.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.969+0000 7f98925f6700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f988c06a7e0 0x7f988c06ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:01.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.969+0000 7f98925f6700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f988c15b6a0 con 0x7f988c15b2c0 2023-10-16T12:12:01.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.969+0000 7f98925f6700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f988c06b1b0 con 0x7f988c06e570 2023-10-16T12:12:01.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.969+0000 7f98925f6700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f988c161b20 con 0x7f988c06a7e0 2023-10-16T12:12:01.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.969+0000 7f988b7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f988c06e570 0x7f988c15e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:01.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.969+0000 7f988b7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f988c06e570 0x7f988c15e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35570/0 (socket says 172.21.15.16:35570) 2023-10-16T12:12:01.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.969+0000 7f988bfff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f988c15b2c0 0x7f988c161340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:53074/0 (socket says 172.21.15.16:53074) 2023-10-16T12:12:01.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.969+0000 7f988b7fe700 1 -- 172.21.15.16:0/709036324 learned_addr learned my addr 172.21.15.16:0/709036324 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:12:01.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.969+0000 7f9890b93700 1 --2- 172.21.15.16:0/709036324 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f988c06a7e0 0x7f988c06ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:01.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.969+0000 7f988b7fe700 1 -- 172.21.15.16:0/709036324 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f988c06a7e0 msgr2=0x7f988c06ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:01.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.969+0000 7f988b7fe700 1 --2- 172.21.15.16:0/709036324 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f988c06a7e0 0x7f988c06ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:01.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.969+0000 7f988b7fe700 1 -- 172.21.15.16:0/709036324 >> v1:172.21.15.16:6789/0 conn(0x7f988c15b2c0 legacy=0x7f988c161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:01.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.969+0000 7f988b7fe700 1 -- 172.21.15.16:0/709036324 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f988c161e00 con 0x7f988c06e570 2023-10-16T12:12:01.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.969+0000 7f988affd700 1 -- 172.21.15.16:0/709036324 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3909529722 0 0) 0x7f988c15b6a0 con 0x7f988c15b2c0 2023-10-16T12:12:01.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.969+0000 7f9890b93700 1 --2- 172.21.15.16:0/709036324 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f988c06a7e0 0x7f988c06ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:12:01.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.969+0000 7f988b7fe700 1 --2- 172.21.15.16:0/709036324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f988c06e570 0x7f988c15e5a0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f98800098f0 tx=0x7f9880009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2eefea68d64e30cc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:01.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.970+0000 7f988affd700 1 -- 172.21.15.16:0/709036324 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f988000b8a0 con 0x7f988c06e570 2023-10-16T12:12:01.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.970+0000 7f988affd700 1 -- 172.21.15.16:0/709036324 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f988000ba00 con 0x7f988c06e570 2023-10-16T12:12:01.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.970+0000 7f988affd700 1 -- 172.21.15.16:0/709036324 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f988000bd90 con 0x7f988c06e570 2023-10-16T12:12:01.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.970+0000 7f98925f6700 1 -- 172.21.15.16:0/709036324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f988c06e570 msgr2=0x7f988c15e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:01.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.970+0000 7f98925f6700 1 --2- 172.21.15.16:0/709036324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f988c06e570 0x7f988c15e5a0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f98800098f0 tx=0x7f9880009c00 comp rx=0 tx=0).stop 2023-10-16T12:12:01.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.970+0000 7f98925f6700 1 -- 172.21.15.16:0/709036324 shutdown_connections 2023-10-16T12:12:01.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.970+0000 7f98925f6700 1 --2- 172.21.15.16:0/709036324 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f988c06a7e0 0x7f988c06ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:01.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.970+0000 7f98925f6700 1 --2- 172.21.15.16:0/709036324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f988c06e570 0x7f988c15e5a0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:01.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.970+0000 7f98925f6700 1 -- 172.21.15.16:0/709036324 >> 172.21.15.16:0/709036324 conn(0x7f988c0b1f40 msgr2=0x7f988c0b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:01.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.970+0000 7f98925f6700 1 -- 172.21.15.16:0/709036324 shutdown_connections 2023-10-16T12:12:01.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.970+0000 7f98925f6700 1 -- 172.21.15.16:0/709036324 wait complete. 2023-10-16T12:12:01.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.971+0000 7f98925f6700 1 Processor -- start 2023-10-16T12:12:01.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.972+0000 7f98925f6700 1 -- start start 2023-10-16T12:12:01.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.972+0000 7f98925f6700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f988c06a7e0 0x7f988c0ca0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:01.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.972+0000 7f98925f6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f988c06e570 0x7f988c15ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:01.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.972+0000 7f98925f6700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f988c15b2c0 0x7f988c0cbba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:01.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.972+0000 7f98925f6700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f988c1611e0 con 0x7f988c06e570 2023-10-16T12:12:01.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.972+0000 7f98925f6700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f988c0ca610 con 0x7f988c15b2c0 2023-10-16T12:12:01.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.972+0000 7f98925f6700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f988c15fe60 con 0x7f988c06a7e0 2023-10-16T12:12:01.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.972+0000 7f988b7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f988c06e570 0x7f988c15ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:01.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.972+0000 7f988b7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f988c06e570 0x7f988c15ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35576/0 (socket says 172.21.15.16:35576) 2023-10-16T12:12:01.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.972+0000 7f988b7fe700 1 -- 172.21.15.16:0/3816477892 learned_addr learned my addr 172.21.15.16:0/3816477892 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:12:01.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.973+0000 7f9890b93700 1 --2- 172.21.15.16:0/3816477892 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f988c15b2c0 0x7f988c0cbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:01.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.973+0000 7f988b7fe700 1 -- 172.21.15.16:0/3816477892 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f988c06a7e0 msgr2=0x7f988c0ca0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:01.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.973+0000 7f988bfff700 1 --2- 172.21.15.16:0/3816477892 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f988c06a7e0 0x7f988c0ca0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:01.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.973+0000 7f988b7fe700 1 --2- 172.21.15.16:0/3816477892 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f988c06a7e0 0x7f988c0ca0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:01.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.973+0000 7f988b7fe700 1 -- 172.21.15.16:0/3816477892 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f988c15b2c0 msgr2=0x7f988c0cbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:01.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.973+0000 7f988b7fe700 1 --2- 172.21.15.16:0/3816477892 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f988c15b2c0 0x7f988c0cbba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:01.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.973+0000 7f988b7fe700 1 -- 172.21.15.16:0/3816477892 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9880009550 con 0x7f988c06e570 2023-10-16T12:12:01.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.973+0000 7f9890b93700 1 --2- 172.21.15.16:0/3816477892 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f988c15b2c0 0x7f988c0cbba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:12:01.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.973+0000 7f988b7fe700 1 --2- 172.21.15.16:0/3816477892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f988c06e570 0x7f988c15ef20 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f988c070620 tx=0x7f9880004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:01.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.973+0000 7f9888ff9700 1 -- 172.21.15.16:0/3816477892 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9880004380 con 0x7f988c06e570 2023-10-16T12:12:01.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.973+0000 7f9888ff9700 1 -- 172.21.15.16:0/3816477892 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f988001f070 con 0x7f988c06e570 2023-10-16T12:12:01.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.973+0000 7f9888ff9700 1 -- 172.21.15.16:0/3816477892 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98800036a0 con 0x7f988c06e570 2023-10-16T12:12:01.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.973+0000 7f98925f6700 1 -- 172.21.15.16:0/3816477892 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f988c160140 con 0x7f988c06e570 2023-10-16T12:12:01.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.973+0000 7f98925f6700 1 -- 172.21.15.16:0/3816477892 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f988c160700 con 0x7f988c06e570 2023-10-16T12:12:01.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.974+0000 7f98925f6700 1 -- 172.21.15.16:0/3816477892 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f988c0ced00 con 0x7f988c06e570 2023-10-16T12:12:01.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.975+0000 7f9888ff9700 1 -- 172.21.15.16:0/3816477892 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f9880021030 con 0x7f988c06e570 2023-10-16T12:12:01.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.975+0000 7f9888ff9700 1 --2- 172.21.15.16:0/3816477892 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f987806af00 0x7f987806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:01.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.975+0000 7f9888ff9700 1 -- 172.21.15.16:0/3816477892 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f988008aa70 con 0x7f988c06e570 2023-10-16T12:12:01.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.975+0000 7f988bfff700 1 --2- 172.21.15.16:0/3816477892 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f987806af00 0x7f987806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:01.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.975+0000 7f988bfff700 1 --2- 172.21.15.16:0/3816477892 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f987806af00 0x7f987806d3c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f987c000f60 tx=0x7f987c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:01.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:01.979+0000 7f9888ff9700 1 -- 172.21.15.16:0/3816477892 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9880055a70 con 0x7f988c06e570 2023-10-16T12:12:02.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:02.235+0000 7f98925f6700 1 -- 172.21.15.16:0/3816477892 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f988c072e10 con 0x7f988c06e570 2023-10-16T12:12:03.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.114+0000 7f9888ff9700 1 -- 172.21.15.16:0/3816477892 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) v1 ==== 118+0+0 (secure 0 0 0) 0x7f98800150b0 con 0x7f988c06e570 2023-10-16T12:12:03.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.117+0000 7f98925f6700 1 -- 172.21.15.16:0/3816477892 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f987806af00 msgr2=0x7f987806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:03.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.117+0000 7f98925f6700 1 --2- 172.21.15.16:0/3816477892 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f987806af00 0x7f987806d3c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f987c000f60 tx=0x7f987c009040 comp rx=0 tx=0).stop 2023-10-16T12:12:03.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.117+0000 7f98925f6700 1 -- 172.21.15.16:0/3816477892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f988c06e570 msgr2=0x7f988c15ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:03.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.117+0000 7f98925f6700 1 --2- 172.21.15.16:0/3816477892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f988c06e570 0x7f988c15ef20 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f988c070620 tx=0x7f9880004750 comp rx=0 tx=0).stop 2023-10-16T12:12:03.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.117+0000 7f98925f6700 1 -- 172.21.15.16:0/3816477892 shutdown_connections 2023-10-16T12:12:03.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.117+0000 7f98925f6700 1 --2- 172.21.15.16:0/3816477892 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f987806af00 0x7f987806d3c0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:03.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.117+0000 7f98925f6700 1 --2- 172.21.15.16:0/3816477892 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f988c15b2c0 0x7f988c0cbba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:03.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.117+0000 7f98925f6700 1 --2- 172.21.15.16:0/3816477892 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f988c06a7e0 0x7f988c0ca0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:03.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.117+0000 7f98925f6700 1 --2- 172.21.15.16:0/3816477892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f988c06e570 0x7f988c15ef20 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:03.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.117+0000 7f98925f6700 1 -- 172.21.15.16:0/3816477892 >> 172.21.15.16:0/3816477892 conn(0x7f988c0b1f40 msgr2=0x7f988c15bac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:03.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.117+0000 7f98925f6700 1 -- 172.21.15.16:0/3816477892 shutdown_connections 2023-10-16T12:12:03.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.118+0000 7f98925f6700 1 -- 172.21.15.16:0/3816477892 wait complete. 2023-10-16T12:12:03.134 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-16T12:12:03.135 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-16T12:12:03.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.267+0000 7fa0adb2d700 1 Processor -- start 2023-10-16T12:12:03.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.267+0000 7fa0adb2d700 1 -- start start 2023-10-16T12:12:03.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.268+0000 7fa0adb2d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa0a80cb880 0x7fa0a80cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:03.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.268+0000 7fa0adb2d700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fa0a806da60 0x7fa0a806def0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:03.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.268+0000 7fa0adb2d700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0a80cef70 con 0x7fa0a815b070 2023-10-16T12:12:03.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.268+0000 7fa0adb2d700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0a806e430 con 0x7fa0a806da60 2023-10-16T12:12:03.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.268+0000 7fa0adb2d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a80ce360 con 0x7fa0a80cb880 2023-10-16T12:12:03.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.268+0000 7fa0a77fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa0a815b070 0x7fa0a815b450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:53088/0 (socket says 172.21.15.16:53088) 2023-10-16T12:12:03.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.268+0000 7fa0a77fe700 1 -- 172.21.15.16:0/3919502004 learned_addr learned my addr 172.21.15.16:0/3919502004 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:12:03.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.269+0000 7fa0a67fc700 1 -- 172.21.15.16:0/3919502004 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1408798002 0 0) 0x7fa0a80cef70 con 0x7fa0a815b070 2023-10-16T12:12:03.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.269+0000 7fa0a7fff700 1 --2- 172.21.15.16:0/3919502004 >> v2:172.21.15.49:3300/0 conn(0x7fa0a806da60 0x7fa0a806def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:03.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.269+0000 7fa0a6ffd700 1 --2- 172.21.15.16:0/3919502004 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa0a80cb880 0x7fa0a80cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:03.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.269+0000 7fa0a67fc700 1 -- 172.21.15.16:0/3919502004 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0940057c0 con 0x7fa0a815b070 2023-10-16T12:12:03.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.269+0000 7fa0a67fc700 1 -- 172.21.15.16:0/3919502004 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4033760860 0 0) 0x7fa0940057c0 con 0x7fa0a815b070 2023-10-16T12:12:03.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.269+0000 7fa0a67fc700 1 -- 172.21.15.16:0/3919502004 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa0a80cb880 msgr2=0x7fa0a80cbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:03.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.269+0000 7fa0a67fc700 1 --2- 172.21.15.16:0/3919502004 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa0a80cb880 0x7fa0a80cbd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:03.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.269+0000 7fa0a67fc700 1 -- 172.21.15.16:0/3919502004 >> v2:172.21.15.49:3300/0 conn(0x7fa0a806da60 msgr2=0x7fa0a806def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:03.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.269+0000 7fa0a67fc700 1 --2- 172.21.15.16:0/3919502004 >> v2:172.21.15.49:3300/0 conn(0x7fa0a806da60 0x7fa0a806def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:03.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.269+0000 7fa0a67fc700 1 -- 172.21.15.16:0/3919502004 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0a80ce640 con 0x7fa0a815b070 2023-10-16T12:12:03.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.269+0000 7fa0a67fc700 1 -- 172.21.15.16:0/3919502004 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7fa090001670 con 0x7fa0a815b070 2023-10-16T12:12:03.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.269+0000 7fa0a67fc700 1 -- 172.21.15.16:0/3919502004 >> v1:172.21.15.16:6789/0 conn(0x7fa0a815b070 legacy=0x7fa0a815b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:03.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.269+0000 7fa0a67fc700 1 --2- 172.21.15.16:0/3919502004 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa0a80cb880 0x7fa094005c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:03.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.269+0000 7fa0a67fc700 1 --2- 172.21.15.16:0/3919502004 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa094006190 0x7fa094006a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:03.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.269+0000 7fa0a67fc700 1 --2- 172.21.15.16:0/3919502004 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa094006fa0 0x7fa094007420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:03.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.269+0000 7fa0a67fc700 1 -- 172.21.15.16:0/3919502004 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a806e430 con 0x7fa094006190 2023-10-16T12:12:03.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.269+0000 7fa0a67fc700 1 -- 172.21.15.16:0/3919502004 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a80ce360 con 0x7fa094006fa0 2023-10-16T12:12:03.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.269+0000 7fa0a67fc700 1 -- 172.21.15.16:0/3919502004 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa09400dc70 con 0x7fa0a80cb880 2023-10-16T12:12:03.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.269+0000 7fa0a6ffd700 1 --2- 172.21.15.16:0/3919502004 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa094006190 0x7fa094006a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:03.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.269+0000 7fa0a77fe700 1 --2- 172.21.15.16:0/3919502004 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa0a80cb880 0x7fa094005c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:03.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.269+0000 7fa0a7fff700 1 --2- 172.21.15.16:0/3919502004 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa094006fa0 0x7fa094007420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:03.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.270+0000 7fa0a6ffd700 1 -- 172.21.15.16:0/3919502004 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa0a80cb880 msgr2=0x7fa094005c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:03.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.270+0000 7fa0a6ffd700 1 --2- 172.21.15.16:0/3919502004 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa0a80cb880 0x7fa094005c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:03.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.270+0000 7fa0a6ffd700 1 -- 172.21.15.16:0/3919502004 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa094006fa0 msgr2=0x7fa094007420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:03.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.270+0000 7fa0a6ffd700 1 --2- 172.21.15.16:0/3919502004 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa094006fa0 0x7fa094007420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:03.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.270+0000 7fa0a6ffd700 1 -- 172.21.15.16:0/3919502004 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0940057c0 con 0x7fa094006190 2023-10-16T12:12:03.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.270+0000 7fa0a6ffd700 1 --2- 172.21.15.16:0/3919502004 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa094006190 0x7fa094006a60 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fa098009b90 tx=0x7fa094007a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d96625a81ccf8bb2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:03.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.270+0000 7fa0a67fc700 1 -- 172.21.15.16:0/3919502004 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa09800e6c0 con 0x7fa094006190 2023-10-16T12:12:03.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.270+0000 7fa0a67fc700 1 -- 172.21.15.16:0/3919502004 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa09800e820 con 0x7fa094006190 2023-10-16T12:12:03.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.270+0000 7fa0a67fc700 1 -- 172.21.15.16:0/3919502004 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa09800eb90 con 0x7fa094006190 2023-10-16T12:12:03.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.270+0000 7fa0adb2d700 1 -- 172.21.15.16:0/3919502004 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa094006190 msgr2=0x7fa094006a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:03.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.270+0000 7fa0adb2d700 1 --2- 172.21.15.16:0/3919502004 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa094006190 0x7fa094006a60 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fa098009b90 tx=0x7fa094007a90 comp rx=0 tx=0).stop 2023-10-16T12:12:03.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.270+0000 7fa0a7fff700 1 -- 172.21.15.16:0/3919502004 reap_dead start 2023-10-16T12:12:03.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.270+0000 7fa0adb2d700 1 -- 172.21.15.16:0/3919502004 shutdown_connections 2023-10-16T12:12:03.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.270+0000 7fa0adb2d700 1 -- 172.21.15.16:0/3919502004 >> 172.21.15.16:0/3919502004 conn(0x7fa0a80b1f40 msgr2=0x7fa0a80b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:03.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.271+0000 7fa0adb2d700 1 -- 172.21.15.16:0/3919502004 shutdown_connections 2023-10-16T12:12:03.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.271+0000 7fa0adb2d700 1 -- 172.21.15.16:0/3919502004 wait complete. 2023-10-16T12:12:03.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.272+0000 7fa0adb2d700 1 Processor -- start 2023-10-16T12:12:03.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.272+0000 7fa0adb2d700 1 -- start start 2023-10-16T12:12:03.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.273+0000 7fa0adb2d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa094006190 0x7fa0a81ef3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:03.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.273+0000 7fa0adb2d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa0a81528b0 0x7fa0a8157520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:03.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.273+0000 7fa0adb2d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa0a8157a60 0x7fa0a81f1f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:03.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.273+0000 7fa0adb2d700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a8157ee0 con 0x7fa094006190 2023-10-16T12:12:03.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.273+0000 7fa0adb2d700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a81ef920 con 0x7fa0a81528b0 2023-10-16T12:12:03.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.273+0000 7fa0adb2d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a80b3330 con 0x7fa0a8157a60 2023-10-16T12:12:03.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.273+0000 7fa0a77fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa094006190 0x7fa0a81ef3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:03.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.273+0000 7fa0a77fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa094006190 0x7fa0a81ef3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35594/0 (socket says 172.21.15.16:35594) 2023-10-16T12:12:03.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.273+0000 7fa0a77fe700 1 -- 172.21.15.16:0/568739297 learned_addr learned my addr 172.21.15.16:0/568739297 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:12:03.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.273+0000 7fa0a7fff700 1 --2- 172.21.15.16:0/568739297 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa0a8157a60 0x7fa0a81f1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:03.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.273+0000 7fa0a6ffd700 1 --2- 172.21.15.16:0/568739297 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa0a81528b0 0x7fa0a8157520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:03.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.273+0000 7fa0a77fe700 1 -- 172.21.15.16:0/568739297 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa0a8157a60 msgr2=0x7fa0a81f1f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:03.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.273+0000 7fa0a77fe700 1 --2- 172.21.15.16:0/568739297 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa0a8157a60 0x7fa0a81f1f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:03.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.273+0000 7fa0a77fe700 1 -- 172.21.15.16:0/568739297 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa0a81528b0 msgr2=0x7fa0a8157520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:03.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.273+0000 7fa0a77fe700 1 --2- 172.21.15.16:0/568739297 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa0a81528b0 0x7fa0a8157520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:03.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.273+0000 7fa0a77fe700 1 -- 172.21.15.16:0/568739297 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0a81557b0 con 0x7fa094006190 2023-10-16T12:12:03.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.273+0000 7fa0a77fe700 1 --2- 172.21.15.16:0/568739297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa094006190 0x7fa0a81ef3e0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fa090000fb0 tx=0x7fa0900075f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:03.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.273+0000 7fa09ffff700 1 -- 172.21.15.16:0/568739297 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa09000f5e0 con 0x7fa094006190 2023-10-16T12:12:03.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.273+0000 7fa09ffff700 1 -- 172.21.15.16:0/568739297 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa09000f760 con 0x7fa094006190 2023-10-16T12:12:03.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.273+0000 7fa0adb2d700 1 -- 172.21.15.16:0/568739297 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0a8155a90 con 0x7fa094006190 2023-10-16T12:12:03.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.273+0000 7fa09ffff700 1 -- 172.21.15.16:0/568739297 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa09000fb10 con 0x7fa094006190 2023-10-16T12:12:03.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.274+0000 7fa0adb2d700 1 -- 172.21.15.16:0/568739297 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0a8156050 con 0x7fa094006190 2023-10-16T12:12:03.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.274+0000 7fa0adb2d700 1 -- 172.21.15.16:0/568739297 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0a804ebb0 con 0x7fa094006190 2023-10-16T12:12:03.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.275+0000 7fa09ffff700 1 -- 172.21.15.16:0/568739297 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fa09000fc70 con 0x7fa094006190 2023-10-16T12:12:03.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.275+0000 7fa09ffff700 1 --2- 172.21.15.16:0/568739297 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fa08806af00 0x7fa08806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:03.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.275+0000 7fa09ffff700 1 -- 172.21.15.16:0/568739297 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fa090089670 con 0x7fa094006190 2023-10-16T12:12:03.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.275+0000 7fa0a6ffd700 1 --2- 172.21.15.16:0/568739297 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fa08806af00 0x7fa08806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:03.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.276+0000 7fa0a6ffd700 1 --2- 172.21.15.16:0/568739297 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fa08806af00 0x7fa08806d3c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fa0980090b0 tx=0x7fa09801f040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:03.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.280+0000 7fa09ffff700 1 -- 172.21.15.16:0/568739297 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa0900545f0 con 0x7fa094006190 2023-10-16T12:12:03.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:03.537+0000 7fa0adb2d700 1 -- 172.21.15.16:0/568739297 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fa0a80728a0 con 0x7fa094006190 2023-10-16T12:12:04.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.118+0000 7fa09ffff700 1 -- 172.21.15.16:0/568739297 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v18) v1 ==== 122+0+0 (secure 0 0 0) 0x7fa090057c10 con 0x7fa094006190 2023-10-16T12:12:04.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.121+0000 7fa0adb2d700 1 -- 172.21.15.16:0/568739297 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fa08806af00 msgr2=0x7fa08806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:04.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.121+0000 7fa0adb2d700 1 --2- 172.21.15.16:0/568739297 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fa08806af00 0x7fa08806d3c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fa0980090b0 tx=0x7fa09801f040 comp rx=0 tx=0).stop 2023-10-16T12:12:04.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.121+0000 7fa0adb2d700 1 -- 172.21.15.16:0/568739297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa094006190 msgr2=0x7fa0a81ef3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:04.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.121+0000 7fa0adb2d700 1 --2- 172.21.15.16:0/568739297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa094006190 0x7fa0a81ef3e0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fa090000fb0 tx=0x7fa0900075f0 comp rx=0 tx=0).stop 2023-10-16T12:12:04.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.121+0000 7fa0adb2d700 1 -- 172.21.15.16:0/568739297 shutdown_connections 2023-10-16T12:12:04.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.121+0000 7fa0adb2d700 1 --2- 172.21.15.16:0/568739297 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fa08806af00 0x7fa08806d3c0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:04.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.121+0000 7fa0adb2d700 1 --2- 172.21.15.16:0/568739297 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa0a8157a60 0x7fa0a81f1f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:04.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.121+0000 7fa0adb2d700 1 --2- 172.21.15.16:0/568739297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa094006190 0x7fa0a81ef3e0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:04.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.121+0000 7fa0adb2d700 1 --2- 172.21.15.16:0/568739297 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa0a81528b0 0x7fa0a8157520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:04.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.121+0000 7fa0adb2d700 1 -- 172.21.15.16:0/568739297 >> 172.21.15.16:0/568739297 conn(0x7fa0a80b1f40 msgr2=0x7fa0a80c6d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:04.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.121+0000 7fa0adb2d700 1 -- 172.21.15.16:0/568739297 shutdown_connections 2023-10-16T12:12:04.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.121+0000 7fa0adb2d700 1 -- 172.21.15.16:0/568739297 wait complete. 2023-10-16T12:12:04.138 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-16T12:12:04.138 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:12:04.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.268+0000 7f0fd5b53700 1 Processor -- start 2023-10-16T12:12:04.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.268+0000 7f0fd5b53700 1 -- start start 2023-10-16T12:12:04.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.269+0000 7f0fd5b53700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f0fd015a870 0x7f0fd006eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:04.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.269+0000 7f0fd5b53700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0fd006d350 0x7f0fd00cb7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:04.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.269+0000 7f0fd5b53700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0fd0066840 0x7f0fd00cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:04.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.269+0000 7f0fd5b53700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fd015ac50 con 0x7f0fd0066840 2023-10-16T12:12:04.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.269+0000 7f0fd5b53700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fd006f0a0 con 0x7f0fd015a870 2023-10-16T12:12:04.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.269+0000 7f0fd5b53700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd006f380 con 0x7f0fd006d350 2023-10-16T12:12:04.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.269+0000 7f0fcffff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0fd0066840 0x7f0fd00cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:04.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.269+0000 7f0fcffff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0fd0066840 0x7f0fd00cbd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35606/0 (socket says 172.21.15.16:35606) 2023-10-16T12:12:04.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.269+0000 7f0fcffff700 1 -- 172.21.15.16:0/1903088320 learned_addr learned my addr 172.21.15.16:0/1903088320 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:12:04.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.269+0000 7f0fcf7fe700 1 --2- 172.21.15.16:0/1903088320 >> v2:172.21.15.49:3300/0 conn(0x7f0fd015a870 0x7f0fd006eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:04.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.269+0000 7f0fceffd700 1 --2- 172.21.15.16:0/1903088320 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0fd006d350 0x7f0fd00cb7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:04.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.270+0000 7f0fcffff700 1 -- 172.21.15.16:0/1903088320 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0fd006d350 msgr2=0x7f0fd00cb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:04.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.270+0000 7f0fcffff700 1 --2- 172.21.15.16:0/1903088320 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0fd006d350 0x7f0fd00cb7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:04.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.270+0000 7f0fcffff700 1 -- 172.21.15.16:0/1903088320 >> v2:172.21.15.49:3300/0 conn(0x7f0fd015a870 msgr2=0x7f0fd006eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:04.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.270+0000 7f0fcffff700 1 --2- 172.21.15.16:0/1903088320 >> v2:172.21.15.49:3300/0 conn(0x7f0fd015a870 0x7f0fd006eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:04.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.270+0000 7f0fcffff700 1 -- 172.21.15.16:0/1903088320 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fd006f660 con 0x7f0fd0066840 2023-10-16T12:12:04.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.270+0000 7f0fcffff700 1 --2- 172.21.15.16:0/1903088320 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fd0066840 0x7f0fd00cbd00 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f0fd0072390 tx=0x7f0fc800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa76780305ac0cea server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:04.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.270+0000 7f0fce7fc700 1 -- 172.21.15.16:0/1903088320 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc800e070 con 0x7f0fd0066840 2023-10-16T12:12:04.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.270+0000 7f0fce7fc700 1 -- 172.21.15.16:0/1903088320 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fc800b910 con 0x7f0fd0066840 2023-10-16T12:12:04.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.270+0000 7f0fce7fc700 1 -- 172.21.15.16:0/1903088320 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc800bcc0 con 0x7f0fd0066840 2023-10-16T12:12:04.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.271+0000 7f0fd5b53700 1 -- 172.21.15.16:0/1903088320 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fd0066840 msgr2=0x7f0fd00cbd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:04.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.271+0000 7f0fd5b53700 1 --2- 172.21.15.16:0/1903088320 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fd0066840 0x7f0fd00cbd00 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f0fd0072390 tx=0x7f0fc800b330 comp rx=0 tx=0).stop 2023-10-16T12:12:04.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.271+0000 7f0fd5b53700 1 -- 172.21.15.16:0/1903088320 shutdown_connections 2023-10-16T12:12:04.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.271+0000 7f0fd5b53700 1 --2- 172.21.15.16:0/1903088320 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fd0066840 0x7f0fd00cbd00 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:04.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.271+0000 7f0fd5b53700 1 --2- 172.21.15.16:0/1903088320 >> v2:172.21.15.49:3300/0 conn(0x7f0fd015a870 0x7f0fd006eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:04.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.271+0000 7f0fd5b53700 1 --2- 172.21.15.16:0/1903088320 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0fd006d350 0x7f0fd00cb7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:04.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.271+0000 7f0fd5b53700 1 -- 172.21.15.16:0/1903088320 >> 172.21.15.16:0/1903088320 conn(0x7f0fd00cc5b0 msgr2=0x7f0fd00cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:04.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.271+0000 7f0fd5b53700 1 -- 172.21.15.16:0/1903088320 shutdown_connections 2023-10-16T12:12:04.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.271+0000 7f0fd5b53700 1 -- 172.21.15.16:0/1903088320 wait complete. 2023-10-16T12:12:04.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.272+0000 7f0fd5b53700 1 Processor -- start 2023-10-16T12:12:04.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.272+0000 7f0fd5b53700 1 -- start start 2023-10-16T12:12:04.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.273+0000 7f0fd5b53700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0fd0066840 0x7f0fd0155f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:04.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.273+0000 7f0fd5b53700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fd006d350 0x7f0fd01564c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:04.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.273+0000 7f0fd5b53700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0fd015a870 0x7f0fd0156a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:04.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.273+0000 7f0fd5b53700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd00b2790 con 0x7f0fd006d350 2023-10-16T12:12:04.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.273+0000 7f0fd5b53700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd00af270 con 0x7f0fd0066840 2023-10-16T12:12:04.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.273+0000 7f0fd5b53700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd00b29e0 con 0x7f0fd015a870 2023-10-16T12:12:04.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.273+0000 7f0fceffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fd006d350 0x7f0fd01564c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:04.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.273+0000 7f0fceffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fd006d350 0x7f0fd01564c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35610/0 (socket says 172.21.15.16:35610) 2023-10-16T12:12:04.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.273+0000 7f0fceffd700 1 -- 172.21.15.16:0/856323273 learned_addr learned my addr 172.21.15.16:0/856323273 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:12:04.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.273+0000 7f0fcf7fe700 1 --2- 172.21.15.16:0/856323273 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0fd0066840 0x7f0fd0155f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:04.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.273+0000 7f0fcffff700 1 --2- 172.21.15.16:0/856323273 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0fd015a870 0x7f0fd0156a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:04.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.273+0000 7f0fceffd700 1 -- 172.21.15.16:0/856323273 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0fd015a870 msgr2=0x7f0fd0156a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:04.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.273+0000 7f0fceffd700 1 --2- 172.21.15.16:0/856323273 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0fd015a870 0x7f0fd0156a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:04.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.273+0000 7f0fceffd700 1 -- 172.21.15.16:0/856323273 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0fd0066840 msgr2=0x7f0fd0155f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:04.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.273+0000 7f0fceffd700 1 --2- 172.21.15.16:0/856323273 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0fd0066840 0x7f0fd0155f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:04.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.273+0000 7f0fceffd700 1 -- 172.21.15.16:0/856323273 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fc8009b30 con 0x7f0fd006d350 2023-10-16T12:12:04.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.273+0000 7f0fceffd700 1 --2- 172.21.15.16:0/856323273 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fd006d350 0x7f0fd01564c0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f0fb800e930 tx=0x7f0fb800ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:04.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.274+0000 7f0fc7fff700 1 -- 172.21.15.16:0/856323273 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fb8013680 con 0x7f0fd006d350 2023-10-16T12:12:04.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.274+0000 7f0fc7fff700 1 -- 172.21.15.16:0/856323273 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fb80137e0 con 0x7f0fd006d350 2023-10-16T12:12:04.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.274+0000 7f0fd5b53700 1 -- 172.21.15.16:0/856323273 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fd00b2cc0 con 0x7f0fd006d350 2023-10-16T12:12:04.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.274+0000 7f0fc7fff700 1 -- 172.21.15.16:0/856323273 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fb8013b70 con 0x7f0fd006d350 2023-10-16T12:12:04.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.274+0000 7f0fd5b53700 1 -- 172.21.15.16:0/856323273 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fd0159240 con 0x7f0fd006d350 2023-10-16T12:12:04.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.275+0000 7f0fd5b53700 1 -- 172.21.15.16:0/856323273 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fd004ebb0 con 0x7f0fd006d350 2023-10-16T12:12:04.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.275+0000 7f0fc7fff700 1 -- 172.21.15.16:0/856323273 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f0fb8013cd0 con 0x7f0fd006d350 2023-10-16T12:12:04.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.275+0000 7f0fc7fff700 1 --2- 172.21.15.16:0/856323273 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f0fb006af00 0x7f0fb006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:04.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.276+0000 7f0fc7fff700 1 -- 172.21.15.16:0/856323273 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f0fb80896d0 con 0x7f0fd006d350 2023-10-16T12:12:04.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.276+0000 7f0fcf7fe700 1 --2- 172.21.15.16:0/856323273 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f0fb006af00 0x7f0fb006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:04.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.276+0000 7f0fcf7fe700 1 --2- 172.21.15.16:0/856323273 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f0fb006af00 0x7f0fb006d3c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f0fc0009b40 tx=0x7f0fc0009210 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:04.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.280+0000 7f0fc7fff700 1 -- 172.21.15.16:0/856323273 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0fb8054640 con 0x7f0fd006d350 2023-10-16T12:12:04.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.528+0000 7f0fd5b53700 1 -- 172.21.15.16:0/856323273 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0fd00af780 con 0x7f0fd006d350 2023-10-16T12:12:04.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.529+0000 7f0fc7fff700 1 -- 172.21.15.16:0/856323273 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+4910 (secure 0 0 0) 0x7f0fb8057c60 con 0x7f0fd006d350 2023-10-16T12:12:04.531 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:12:04.531 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":18,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5194,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/266676368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":266676368},{"type":"v1","addr":"172.21.15.49:6834","nonce":266676368}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5268,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/1460528697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":1460528697},{"type":"v1","addr":"172.21.15.16:6837","nonce":1460528697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/2374376752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2374376752},{"type":"v1","addr":"172.21.15.16:6836","nonce":2374376752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:12:00.537600+0000","modified":"2023-10-16T12:12:03.538619+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5283},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5283":{"gid":5283,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.49:6835/2788396302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":2788396302},{"type":"v1","addr":"172.21.15.49:6835","nonce":2788396302}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-16T12:12:04.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.531+0000 7f0fd5b53700 1 -- 172.21.15.16:0/856323273 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f0fb006af00 msgr2=0x7f0fb006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:04.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.531+0000 7f0fd5b53700 1 --2- 172.21.15.16:0/856323273 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f0fb006af00 0x7f0fb006d3c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f0fc0009b40 tx=0x7f0fc0009210 comp rx=0 tx=0).stop 2023-10-16T12:12:04.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.531+0000 7f0fd5b53700 1 -- 172.21.15.16:0/856323273 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fd006d350 msgr2=0x7f0fd01564c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:04.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.531+0000 7f0fd5b53700 1 --2- 172.21.15.16:0/856323273 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fd006d350 0x7f0fd01564c0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f0fb800e930 tx=0x7f0fb800ec40 comp rx=0 tx=0).stop 2023-10-16T12:12:04.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.531+0000 7f0fd5b53700 1 -- 172.21.15.16:0/856323273 shutdown_connections 2023-10-16T12:12:04.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.531+0000 7f0fd5b53700 1 --2- 172.21.15.16:0/856323273 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f0fb006af00 0x7f0fb006d3c0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:04.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.531+0000 7f0fd5b53700 1 --2- 172.21.15.16:0/856323273 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0fd015a870 0x7f0fd0156a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:04.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.531+0000 7f0fd5b53700 1 --2- 172.21.15.16:0/856323273 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0fd0066840 0x7f0fd0155f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:04.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.531+0000 7f0fd5b53700 1 --2- 172.21.15.16:0/856323273 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fd006d350 0x7f0fd01564c0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:04.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.531+0000 7f0fd5b53700 1 -- 172.21.15.16:0/856323273 >> 172.21.15.16:0/856323273 conn(0x7f0fd00cc5b0 msgr2=0x7f0fd0151800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:04.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.531+0000 7f0fd5b53700 1 -- 172.21.15.16:0/856323273 shutdown_connections 2023-10-16T12:12:04.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.531+0000 7f0fd5b53700 1 -- 172.21.15.16:0/856323273 wait complete. 2023-10-16T12:12:04.536 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 18 2023-10-16T12:12:04.545 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:12:04.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.669+0000 7f8782b20700 1 Processor -- start 2023-10-16T12:12:04.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.669+0000 7f8782b20700 1 -- start start 2023-10-16T12:12:04.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.670+0000 7f8782b20700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f877c15b280 0x7f877c1613e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:04.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.670+0000 7f8782b20700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f877c06a7e0 0x7f877c06ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:04.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.670+0000 7f8782b20700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f877c06b170 con 0x7f877c0cc8a0 2023-10-16T12:12:04.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.670+0000 7f8782b20700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f877c161c30 con 0x7f877c06a7e0 2023-10-16T12:12:04.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.670+0000 7f8782b20700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f877c161f10 con 0x7f877c15b280 2023-10-16T12:12:04.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.670+0000 7f877bfff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f877c0cc8a0 0x7f877c15e700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:53124/0 (socket says 172.21.15.16:53124) 2023-10-16T12:12:04.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.670+0000 7f877bfff700 1 -- 172.21.15.16:0/698391421 learned_addr learned my addr 172.21.15.16:0/698391421 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:12:04.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.670+0000 7f87808bc700 1 --2- 172.21.15.16:0/698391421 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f877c15b280 0x7f877c1613e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:04.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.670+0000 7f87810bd700 1 --2- 172.21.15.16:0/698391421 >> v2:172.21.15.49:3300/0 conn(0x7f877c06a7e0 0x7f877c06ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:04.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.670+0000 7f877b7fe700 1 -- 172.21.15.16:0/698391421 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1330124398 0 0) 0x7f8768001110 con 0x7f877c0cc8a0 2023-10-16T12:12:04.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.670+0000 7f877b7fe700 1 -- 172.21.15.16:0/698391421 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f876c0035b0 con 0x7f877c0cc8a0 2023-10-16T12:12:04.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.670+0000 7f877b7fe700 1 -- 172.21.15.16:0/698391421 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2640897541 0 0) 0x7f876c0035b0 con 0x7f877c0cc8a0 2023-10-16T12:12:04.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.670+0000 7f877b7fe700 1 -- 172.21.15.16:0/698391421 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f877c15b280 msgr2=0x7f877c1613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:04.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.670+0000 7f877b7fe700 1 --2- 172.21.15.16:0/698391421 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f877c15b280 0x7f877c1613e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:04.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.670+0000 7f877b7fe700 1 -- 172.21.15.16:0/698391421 >> v2:172.21.15.49:3300/0 conn(0x7f877c06a7e0 msgr2=0x7f877c06ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:04.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.670+0000 7f877b7fe700 1 --2- 172.21.15.16:0/698391421 >> v2:172.21.15.49:3300/0 conn(0x7f877c06a7e0 0x7f877c06ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:04.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.670+0000 7f877b7fe700 1 -- 172.21.15.16:0/698391421 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f877c1621f0 con 0x7f877c0cc8a0 2023-10-16T12:12:04.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.670+0000 7f877b7fe700 1 -- 172.21.15.16:0/698391421 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f8768001800 con 0x7f877c0cc8a0 2023-10-16T12:12:04.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.670+0000 7f877b7fe700 1 -- 172.21.15.16:0/698391421 >> v1:172.21.15.16:6789/0 conn(0x7f877c0cc8a0 legacy=0x7f877c15e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:04.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.671+0000 7f877b7fe700 1 --2- 172.21.15.16:0/698391421 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f876c003a00 0x7f876c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:04.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.671+0000 7f877b7fe700 1 --2- 172.21.15.16:0/698391421 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f876c004850 0x7f876c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:04.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.671+0000 7f877b7fe700 1 --2- 172.21.15.16:0/698391421 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f877c15b280 0x7f876c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:04.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.671+0000 7f877b7fe700 1 -- 172.21.15.16:0/698391421 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f877c161c30 con 0x7f876c003a00 2023-10-16T12:12:04.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.671+0000 7f877b7fe700 1 -- 172.21.15.16:0/698391421 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f877c161f10 con 0x7f876c004850 2023-10-16T12:12:04.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.671+0000 7f877b7fe700 1 -- 172.21.15.16:0/698391421 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f876c00ba10 con 0x7f877c15b280 2023-10-16T12:12:04.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.671+0000 7f87808bc700 1 --2- 172.21.15.16:0/698391421 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f876c003a00 0x7f876c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:04.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.671+0000 7f877bfff700 1 --2- 172.21.15.16:0/698391421 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f876c004850 0x7f876c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:04.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.671+0000 7f87810bd700 1 --2- 172.21.15.16:0/698391421 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f877c15b280 0x7f876c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:04.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.671+0000 7f87808bc700 1 -- 172.21.15.16:0/698391421 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f877c15b280 msgr2=0x7f876c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:04.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.671+0000 7f87808bc700 1 --2- 172.21.15.16:0/698391421 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f877c15b280 0x7f876c005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:04.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.671+0000 7f87808bc700 1 -- 172.21.15.16:0/698391421 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f876c004850 msgr2=0x7f876c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:04.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.671+0000 7f87808bc700 1 --2- 172.21.15.16:0/698391421 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f876c004850 0x7f876c004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:04.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.671+0000 7f87808bc700 1 -- 172.21.15.16:0/698391421 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f876c0035b0 con 0x7f876c003a00 2023-10-16T12:12:04.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.671+0000 7f877bfff700 1 --2- 172.21.15.16:0/698391421 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f876c004850 0x7f876c004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:12:04.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.671+0000 7f87808bc700 1 --2- 172.21.15.16:0/698391421 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f876c003a00 0x7f876c003e20 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f8770009b60 tx=0x7f876c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64c18438269a8fc8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:04.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.671+0000 7f877b7fe700 1 -- 172.21.15.16:0/698391421 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f877000e6c0 con 0x7f876c003a00 2023-10-16T12:12:04.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.671+0000 7f877b7fe700 1 -- 172.21.15.16:0/698391421 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f877000e820 con 0x7f876c003a00 2023-10-16T12:12:04.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.671+0000 7f877b7fe700 1 -- 172.21.15.16:0/698391421 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f877000eb90 con 0x7f876c003a00 2023-10-16T12:12:04.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.672+0000 7f8782b20700 1 -- 172.21.15.16:0/698391421 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f876c003a00 msgr2=0x7f876c003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:04.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.672+0000 7f8782b20700 1 --2- 172.21.15.16:0/698391421 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f876c003a00 0x7f876c003e20 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f8770009b60 tx=0x7f876c005880 comp rx=0 tx=0).stop 2023-10-16T12:12:04.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.672+0000 7f87810bd700 1 -- 172.21.15.16:0/698391421 reap_dead start 2023-10-16T12:12:04.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.672+0000 7f8782b20700 1 -- 172.21.15.16:0/698391421 shutdown_connections 2023-10-16T12:12:04.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.672+0000 7f8782b20700 1 -- 172.21.15.16:0/698391421 >> 172.21.15.16:0/698391421 conn(0x7f877c0b1fd0 msgr2=0x7f877c0b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:04.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.672+0000 7f8782b20700 1 -- 172.21.15.16:0/698391421 shutdown_connections 2023-10-16T12:12:04.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.672+0000 7f8782b20700 1 -- 172.21.15.16:0/698391421 wait complete. 2023-10-16T12:12:04.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.673+0000 7f8782b20700 1 Processor -- start 2023-10-16T12:12:04.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.673+0000 7f8782b20700 1 -- start start 2023-10-16T12:12:04.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.674+0000 7f8782b20700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f877c1f28c0 0x7f877c0ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:04.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.674+0000 7f8782b20700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f877c1f2ca0 0x7f877c0ace10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:04.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.674+0000 7f8782b20700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f877c14b1c0 0x7f877c15f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:04.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.674+0000 7f8782b20700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f877c1f3190 con 0x7f877c1f28c0 2023-10-16T12:12:04.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.674+0000 7f8782b20700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f877c1f3470 con 0x7f877c1f2ca0 2023-10-16T12:12:04.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.674+0000 7f8782b20700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f877c1f3750 con 0x7f877c14b1c0 2023-10-16T12:12:04.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.674+0000 7f87808bc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f877c1f28c0 0x7f877c0ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:04.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.674+0000 7f87808bc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f877c1f28c0 0x7f877c0ac8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35628/0 (socket says 172.21.15.16:35628) 2023-10-16T12:12:04.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.674+0000 7f87808bc700 1 -- 172.21.15.16:0/2162560418 learned_addr learned my addr 172.21.15.16:0/2162560418 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:12:04.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.674+0000 7f877bfff700 1 --2- 172.21.15.16:0/2162560418 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f877c1f2ca0 0x7f877c0ace10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:04.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.674+0000 7f87810bd700 1 --2- 172.21.15.16:0/2162560418 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f877c14b1c0 0x7f877c15f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:04.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.674+0000 7f87808bc700 1 -- 172.21.15.16:0/2162560418 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f877c14b1c0 msgr2=0x7f877c15f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:04.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.674+0000 7f87808bc700 1 --2- 172.21.15.16:0/2162560418 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f877c14b1c0 0x7f877c15f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:04.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.675+0000 7f87808bc700 1 -- 172.21.15.16:0/2162560418 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f877c1f2ca0 msgr2=0x7f877c0ace10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:04.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.675+0000 7f87808bc700 1 --2- 172.21.15.16:0/2162560418 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f877c1f2ca0 0x7f877c0ace10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:04.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.675+0000 7f87808bc700 1 -- 172.21.15.16:0/2162560418 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f877c1f3a30 con 0x7f877c1f28c0 2023-10-16T12:12:04.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.675+0000 7f87808bc700 1 --2- 172.21.15.16:0/2162560418 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f877c1f28c0 0x7f877c0ac8d0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f8770016000 tx=0x7f87700047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:04.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.675+0000 7f87797fa700 1 -- 172.21.15.16:0/2162560418 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f877000e6c0 con 0x7f877c1f28c0 2023-10-16T12:12:04.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.675+0000 7f87797fa700 1 -- 172.21.15.16:0/2162560418 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8770003ad0 con 0x7f877c1f28c0 2023-10-16T12:12:04.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.675+0000 7f8782b20700 1 -- 172.21.15.16:0/2162560418 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f877c1f3d10 con 0x7f877c1f28c0 2023-10-16T12:12:04.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.675+0000 7f87797fa700 1 -- 172.21.15.16:0/2162560418 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8770003e80 con 0x7f877c1f28c0 2023-10-16T12:12:04.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.675+0000 7f8782b20700 1 -- 172.21.15.16:0/2162560418 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f877c15fde0 con 0x7f877c1f28c0 2023-10-16T12:12:04.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.676+0000 7f8782b20700 1 -- 172.21.15.16:0/2162560418 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f877c04ebb0 con 0x7f877c1f28c0 2023-10-16T12:12:04.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.676+0000 7f87797fa700 1 -- 172.21.15.16:0/2162560418 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f8770013470 con 0x7f877c1f28c0 2023-10-16T12:12:04.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.677+0000 7f87797fa700 1 --2- 172.21.15.16:0/2162560418 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f876006af00 0x7f876006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:04.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.677+0000 7f87797fa700 1 -- 172.21.15.16:0/2162560418 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f877008afc0 con 0x7f877c1f28c0 2023-10-16T12:12:04.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.677+0000 7f877bfff700 1 --2- 172.21.15.16:0/2162560418 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f876006af00 0x7f876006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:04.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.677+0000 7f877bfff700 1 --2- 172.21.15.16:0/2162560418 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f876006af00 0x7f876006d3c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f8768000fb0 tx=0x7f8768009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:04.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.681+0000 7f87797fa700 1 -- 172.21.15.16:0/2162560418 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f87700bdfe0 con 0x7f877c1f28c0 2023-10-16T12:12:04.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.885+0000 7f8782b20700 1 -- 172.21.15.16:0/2162560418 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f877c0cd190 con 0x7f877c1f28c0 2023-10-16T12:12:04.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.886+0000 7f87797fa700 1 -- 172.21.15.16:0/2162560418 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) v1 ==== 74+0+16666 (secure 0 0 0) 0x7f87700560e0 con 0x7f877c1f28c0 2023-10-16T12:12:04.889 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:12:04.889 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":39,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:12:02.055689+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:11:58.700868+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2023-10-16T12:11:59.495384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:12:04.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.889+0000 7f8782b20700 1 -- 172.21.15.16:0/2162560418 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f876006af00 msgr2=0x7f876006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:04.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.889+0000 7f8782b20700 1 --2- 172.21.15.16:0/2162560418 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f876006af00 0x7f876006d3c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f8768000fb0 tx=0x7f8768009040 comp rx=0 tx=0).stop 2023-10-16T12:12:04.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.889+0000 7f8782b20700 1 -- 172.21.15.16:0/2162560418 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f877c1f28c0 msgr2=0x7f877c0ac8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:04.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.889+0000 7f8782b20700 1 --2- 172.21.15.16:0/2162560418 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f877c1f28c0 0x7f877c0ac8d0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f8770016000 tx=0x7f87700047c0 comp rx=0 tx=0).stop 2023-10-16T12:12:04.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.889+0000 7f8782b20700 1 -- 172.21.15.16:0/2162560418 shutdown_connections 2023-10-16T12:12:04.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.889+0000 7f8782b20700 1 --2- 172.21.15.16:0/2162560418 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f876006af00 0x7f876006d3c0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:04.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.889+0000 7f8782b20700 1 --2- 172.21.15.16:0/2162560418 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f877c14b1c0 0x7f877c15f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:04.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.889+0000 7f8782b20700 1 --2- 172.21.15.16:0/2162560418 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f877c1f28c0 0x7f877c0ac8d0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:04.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.889+0000 7f8782b20700 1 --2- 172.21.15.16:0/2162560418 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f877c1f2ca0 0x7f877c0ace10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:04.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.889+0000 7f8782b20700 1 -- 172.21.15.16:0/2162560418 >> 172.21.15.16:0/2162560418 conn(0x7f877c0b1fd0 msgr2=0x7f877c0d0270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:04.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.889+0000 7f8782b20700 1 -- 172.21.15.16:0/2162560418 shutdown_connections 2023-10-16T12:12:04.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:04.889+0000 7f8782b20700 1 -- 172.21.15.16:0/2162560418 wait complete. 2023-10-16T12:12:04.903 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-16T12:12:04.904 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T12:12:05.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.026+0000 7f94d9eeb700 1 Processor -- start 2023-10-16T12:12:05.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.027+0000 7f94d9eeb700 1 -- start start 2023-10-16T12:12:05.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.027+0000 7f94d9eeb700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f94d415b860 0x7f94d406a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:05.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.027+0000 7f94d9eeb700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f94d40b4490 0x7f94d406ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:05.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.027+0000 7f94d9eeb700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94d40d11c0 con 0x7f94d40b2be0 2023-10-16T12:12:05.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.027+0000 7f94d9eeb700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94d40d03d0 con 0x7f94d40b4490 2023-10-16T12:12:05.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.027+0000 7f94d9eeb700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94d40cd360 con 0x7f94d415b860 2023-10-16T12:12:05.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.028+0000 7f94d3fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f94d40b4490 0x7f94d406ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:05.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.028+0000 7f94d3fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f94d40b4490 0x7f94d406ad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35638/0 (socket says 172.21.15.16:35638) 2023-10-16T12:12:05.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.028+0000 7f94d3fff700 1 -- 172.21.15.16:0/1917174007 learned_addr learned my addr 172.21.15.16:0/1917174007 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:12:05.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.028+0000 7f94d37fe700 1 --2- 172.21.15.16:0/1917174007 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f94d415b860 0x7f94d406a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:05.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.028+0000 7f94d3fff700 1 -- 172.21.15.16:0/1917174007 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f94d415b860 msgr2=0x7f94d406a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:05.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.028+0000 7f94d3fff700 1 --2- 172.21.15.16:0/1917174007 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f94d415b860 0x7f94d406a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:05.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.028+0000 7f94d3fff700 1 -- 172.21.15.16:0/1917174007 >> v1:172.21.15.49:6789/0 conn(0x7f94d40b2be0 legacy=0x7f94d40719e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:05.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.028+0000 7f94d3fff700 1 -- 172.21.15.16:0/1917174007 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94d40d05b0 con 0x7f94d40b4490 2023-10-16T12:12:05.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.028+0000 7f94d3fff700 1 --2- 172.21.15.16:0/1917174007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94d40b4490 0x7f94d406ad20 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f94cc009e50 tx=0x7f94cc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb58605d60de80e3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:05.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.028+0000 7f94d27fc700 1 -- 172.21.15.16:0/1917174007 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94cc00e070 con 0x7f94d40b4490 2023-10-16T12:12:05.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.028+0000 7f94d27fc700 1 -- 172.21.15.16:0/1917174007 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94cc00b9c0 con 0x7f94d40b4490 2023-10-16T12:12:05.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.028+0000 7f94d27fc700 1 -- 172.21.15.16:0/1917174007 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94cc00bd70 con 0x7f94d40b4490 2023-10-16T12:12:05.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.029+0000 7f94d9eeb700 1 -- 172.21.15.16:0/1917174007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94d40b4490 msgr2=0x7f94d406ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:05.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.029+0000 7f94d9eeb700 1 --2- 172.21.15.16:0/1917174007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94d40b4490 0x7f94d406ad20 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f94cc009e50 tx=0x7f94cc00b330 comp rx=0 tx=0).stop 2023-10-16T12:12:05.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.029+0000 7f94d9eeb700 1 -- 172.21.15.16:0/1917174007 shutdown_connections 2023-10-16T12:12:05.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.029+0000 7f94d9eeb700 1 --2- 172.21.15.16:0/1917174007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94d40b4490 0x7f94d406ad20 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:05.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.029+0000 7f94d9eeb700 1 --2- 172.21.15.16:0/1917174007 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f94d415b860 0x7f94d406a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:05.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.029+0000 7f94d9eeb700 1 -- 172.21.15.16:0/1917174007 >> 172.21.15.16:0/1917174007 conn(0x7f94d40b1fd0 msgr2=0x7f94d40b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:05.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.029+0000 7f94d9eeb700 1 -- 172.21.15.16:0/1917174007 shutdown_connections 2023-10-16T12:12:05.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.029+0000 7f94d9eeb700 1 -- 172.21.15.16:0/1917174007 wait complete. 2023-10-16T12:12:05.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.030+0000 7f94d9eeb700 1 Processor -- start 2023-10-16T12:12:05.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.030+0000 7f94d9eeb700 1 -- start start 2023-10-16T12:12:05.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.031+0000 7f94d9eeb700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f94d40b2be0 0x7f94d4153230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:05.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.031+0000 7f94d9eeb700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94d40b4490 0x7f94d41526d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:05.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.031+0000 7f94d9eeb700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f94d415b860 0x7f94d40ad640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:05.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.031+0000 7f94d9eeb700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94d40c90c0 con 0x7f94d40b4490 2023-10-16T12:12:05.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.031+0000 7f94d9eeb700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94d40ce0b0 con 0x7f94d415b860 2023-10-16T12:12:05.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.031+0000 7f94d9eeb700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94d40ce390 con 0x7f94d40b2be0 2023-10-16T12:12:05.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.031+0000 7f94d2ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94d40b4490 0x7f94d41526d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:05.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.031+0000 7f94d2ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94d40b4490 0x7f94d41526d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35642/0 (socket says 172.21.15.16:35642) 2023-10-16T12:12:05.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.031+0000 7f94d2ffd700 1 -- 172.21.15.16:0/1614837365 learned_addr learned my addr 172.21.15.16:0/1614837365 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:12:05.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.031+0000 7f94d37fe700 1 --2- 172.21.15.16:0/1614837365 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f94d40b2be0 0x7f94d4153230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:05.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.031+0000 7f94d3fff700 1 --2- 172.21.15.16:0/1614837365 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f94d415b860 0x7f94d40ad640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:05.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.032+0000 7f94d2ffd700 1 -- 172.21.15.16:0/1614837365 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f94d40b2be0 msgr2=0x7f94d4153230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:05.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.032+0000 7f94d2ffd700 1 --2- 172.21.15.16:0/1614837365 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f94d40b2be0 0x7f94d4153230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:05.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.032+0000 7f94d2ffd700 1 -- 172.21.15.16:0/1614837365 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f94d415b860 msgr2=0x7f94d40ad640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:05.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.032+0000 7f94d2ffd700 1 --2- 172.21.15.16:0/1614837365 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f94d415b860 0x7f94d40ad640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:05.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.032+0000 7f94d2ffd700 1 -- 172.21.15.16:0/1614837365 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94cc009b30 con 0x7f94d40b4490 2023-10-16T12:12:05.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.032+0000 7f94d37fe700 1 --2- 172.21.15.16:0/1614837365 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f94d40b2be0 0x7f94d4153230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:12:05.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.032+0000 7f94d3fff700 1 --2- 172.21.15.16:0/1614837365 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f94d415b860 0x7f94d40ad640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:12:05.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.032+0000 7f94d2ffd700 1 --2- 172.21.15.16:0/1614837365 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94d40b4490 0x7f94d41526d0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f94c400c450 tx=0x7f94c400c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:05.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.032+0000 7f94cbfff700 1 -- 172.21.15.16:0/1614837365 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c40052d0 con 0x7f94d40b4490 2023-10-16T12:12:05.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.032+0000 7f94cbfff700 1 -- 172.21.15.16:0/1614837365 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94c4005430 con 0x7f94d40b4490 2023-10-16T12:12:05.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.032+0000 7f94d9eeb700 1 -- 172.21.15.16:0/1614837365 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94d40ce670 con 0x7f94d40b4490 2023-10-16T12:12:05.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.032+0000 7f94cbfff700 1 -- 172.21.15.16:0/1614837365 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c40038c0 con 0x7f94d40b4490 2023-10-16T12:12:05.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.032+0000 7f94d9eeb700 1 -- 172.21.15.16:0/1614837365 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94d40cec30 con 0x7f94d40b4490 2023-10-16T12:12:05.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.033+0000 7f94d9eeb700 1 -- 172.21.15.16:0/1614837365 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94d404ebb0 con 0x7f94d40b4490 2023-10-16T12:12:05.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.033+0000 7f94cbfff700 1 -- 172.21.15.16:0/1614837365 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f94c4003a20 con 0x7f94d40b4490 2023-10-16T12:12:05.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.034+0000 7f94cbfff700 1 --2- 172.21.15.16:0/1614837365 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f94b406af00 0x7f94b406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:05.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.034+0000 7f94cbfff700 1 -- 172.21.15.16:0/1614837365 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f94c4088a70 con 0x7f94d40b4490 2023-10-16T12:12:05.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.034+0000 7f94d37fe700 1 --2- 172.21.15.16:0/1614837365 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f94b406af00 0x7f94b406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:05.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.034+0000 7f94d37fe700 1 --2- 172.21.15.16:0/1614837365 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f94b406af00 0x7f94b406d3c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f94bc0099f0 tx=0x7f94bc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:05.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.039+0000 7f94cbfff700 1 -- 172.21.15.16:0/1614837365 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f94c40539e0 con 0x7f94d40b4490 2023-10-16T12:12:05.087 INFO:tasks.ceph.mds.b.smithi049.stderr:2023-10-16T12:12:05.085+0000 7fcc4bdf0700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T12:12:05.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.239+0000 7f94d9eeb700 1 -- 172.21.15.16:0/1614837365 --> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f94d40d0b70 con 0x7f94b406af00 2023-10-16T12:12:05.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.248+0000 7f94cbfff700 1 -- 172.21.15.16:0/1614837365 <== mgr.4104 v2:172.21.15.16:6832/365921436 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289939 (secure 0 0 0) 0x7f94bc009680 con 0x7f94b406af00 2023-10-16T12:12:05.253 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:12:05.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.251+0000 7f94d9eeb700 1 -- 172.21.15.16:0/1614837365 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f94b406af00 msgr2=0x7f94b406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:05.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.251+0000 7f94d9eeb700 1 --2- 172.21.15.16:0/1614837365 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f94b406af00 0x7f94b406d3c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f94bc0099f0 tx=0x7f94bc008040 comp rx=0 tx=0).stop 2023-10-16T12:12:05.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.251+0000 7f94d9eeb700 1 -- 172.21.15.16:0/1614837365 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94d40b4490 msgr2=0x7f94d41526d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:05.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.251+0000 7f94d9eeb700 1 --2- 172.21.15.16:0/1614837365 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94d40b4490 0x7f94d41526d0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f94c400c450 tx=0x7f94c400c810 comp rx=0 tx=0).stop 2023-10-16T12:12:05.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.251+0000 7f94d9eeb700 1 -- 172.21.15.16:0/1614837365 shutdown_connections 2023-10-16T12:12:05.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.251+0000 7f94d9eeb700 1 --2- 172.21.15.16:0/1614837365 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f94b406af00 0x7f94b406d3c0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:05.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.251+0000 7f94d9eeb700 1 --2- 172.21.15.16:0/1614837365 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f94d415b860 0x7f94d40ad640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:05.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.251+0000 7f94d9eeb700 1 --2- 172.21.15.16:0/1614837365 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f94d40b2be0 0x7f94d4153230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:05.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.251+0000 7f94d9eeb700 1 --2- 172.21.15.16:0/1614837365 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94d40b4490 0x7f94d41526d0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:05.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.251+0000 7f94d9eeb700 1 -- 172.21.15.16:0/1614837365 >> 172.21.15.16:0/1614837365 conn(0x7f94d40b1fd0 msgr2=0x7f94d40b0020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:05.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.251+0000 7f94d9eeb700 1 -- 172.21.15.16:0/1614837365 shutdown_connections 2023-10-16T12:12:05.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.251+0000 7f94d9eeb700 1 -- 172.21.15.16:0/1614837365 wait complete. 2023-10-16T12:12:05.257 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-10-16T12:12:05.265 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":79,"stamp":"2023-10-16T12:12:05.113754+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":97,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220808,"kb_used_data":4648,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754753912,"statfs":{"total":773094113280,"available":772868005888,"internally_reserved":0,"allocated":4759552,"data_stored":2683308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12740,"internal_metadata":221236796},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001232"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T12:12:02.059592+0000","last_change":"2023-10-16T12:12:01.084878+0000","last_active":"2023-10-16T12:12:02.059592+0000","last_peered":"2023-10-16T12:12:02.059592+0000","last_clean":"2023-10-16T12:12:02.059592+0000","last_became_active":"2023-10-16T12:12:01.084510+0000","last_became_peered":"2023-10-16T12:12:01.084510+0000","last_unstale":"2023-10-16T12:12:02.059592+0000","last_undegraded":"2023-10-16T12:12:02.059592+0000","last_fullsized":"2023-10-16T12:12:02.059592+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:12:52.469257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T12:12:02.059773+0000","last_change":"2023-10-16T12:12:01.073425+0000","last_active":"2023-10-16T12:12:02.059773+0000","last_peered":"2023-10-16T12:12:02.059773+0000","last_clean":"2023-10-16T12:12:02.059773+0000","last_became_active":"2023-10-16T12:12:01.072828+0000","last_became_peered":"2023-10-16T12:12:01.072828+0000","last_unstale":"2023-10-16T12:12:02.059773+0000","last_undegraded":"2023-10-16T12:12:02.059773+0000","last_fullsized":"2023-10-16T12:12:02.059773+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:59:23.612795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.084213+0000","last_change":"2023-10-16T12:12:01.082901+0000","last_active":"2023-10-16T12:12:01.084213+0000","last_peered":"2023-10-16T12:12:01.084213+0000","last_clean":"2023-10-16T12:12:01.084213+0000","last_became_active":"2023-10-16T12:12:01.082289+0000","last_became_peered":"2023-10-16T12:12:01.082289+0000","last_unstale":"2023-10-16T12:12:01.084213+0000","last_undegraded":"2023-10-16T12:12:01.084213+0000","last_fullsized":"2023-10-16T12:12:01.084213+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:36:12.640742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.072468+0000","last_change":"2023-10-16T12:12:01.069237+0000","last_active":"2023-10-16T12:12:01.072468+0000","last_peered":"2023-10-16T12:12:01.072468+0000","last_clean":"2023-10-16T12:12:01.072468+0000","last_became_active":"2023-10-16T12:12:01.068932+0000","last_became_peered":"2023-10-16T12:12:01.068932+0000","last_unstale":"2023-10-16T12:12:01.072468+0000","last_undegraded":"2023-10-16T12:12:01.072468+0000","last_fullsized":"2023-10-16T12:12:01.072468+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:57:57.533990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.071377+0000","last_change":"2023-10-16T12:12:01.068561+0000","last_active":"2023-10-16T12:12:01.071377+0000","last_peered":"2023-10-16T12:12:01.071377+0000","last_clean":"2023-10-16T12:12:01.071377+0000","last_became_active":"2023-10-16T12:12:01.068253+0000","last_became_peered":"2023-10-16T12:12:01.068253+0000","last_unstale":"2023-10-16T12:12:01.071377+0000","last_undegraded":"2023-10-16T12:12:01.071377+0000","last_fullsized":"2023-10-16T12:12:01.071377+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:30.033807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T12:12:02.060374+0000","last_change":"2023-10-16T12:12:01.101436+0000","last_active":"2023-10-16T12:12:02.060374+0000","last_peered":"2023-10-16T12:12:02.060374+0000","last_clean":"2023-10-16T12:12:02.060374+0000","last_became_active":"2023-10-16T12:12:01.101066+0000","last_became_peered":"2023-10-16T12:12:01.101066+0000","last_unstale":"2023-10-16T12:12:02.060374+0000","last_undegraded":"2023-10-16T12:12:02.060374+0000","last_fullsized":"2023-10-16T12:12:02.060374+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:50:00.190927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.081964+0000","last_change":"2023-10-16T12:12:01.078197+0000","last_active":"2023-10-16T12:12:01.081964+0000","last_peered":"2023-10-16T12:12:01.081964+0000","last_clean":"2023-10-16T12:12:01.081964+0000","last_became_active":"2023-10-16T12:12:01.077180+0000","last_became_peered":"2023-10-16T12:12:01.077180+0000","last_unstale":"2023-10-16T12:12:01.081964+0000","last_undegraded":"2023-10-16T12:12:01.081964+0000","last_fullsized":"2023-10-16T12:12:01.081964+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:07:14.264264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T12:12:02.059254+0000","last_change":"2023-10-16T12:12:01.079867+0000","last_active":"2023-10-16T12:12:02.059254+0000","last_peered":"2023-10-16T12:12:02.059254+0000","last_clean":"2023-10-16T12:12:02.059254+0000","last_became_active":"2023-10-16T12:12:01.079610+0000","last_became_peered":"2023-10-16T12:12:01.079610+0000","last_unstale":"2023-10-16T12:12:02.059254+0000","last_undegraded":"2023-10-16T12:12:02.059254+0000","last_fullsized":"2023-10-16T12:12:02.059254+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:33:26.016966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.084162+0000","last_change":"2023-10-16T12:12:01.082539+0000","last_active":"2023-10-16T12:12:01.084162+0000","last_peered":"2023-10-16T12:12:01.084162+0000","last_clean":"2023-10-16T12:12:01.084162+0000","last_became_active":"2023-10-16T12:12:01.082201+0000","last_became_peered":"2023-10-16T12:12:01.082201+0000","last_unstale":"2023-10-16T12:12:01.084162+0000","last_undegraded":"2023-10-16T12:12:01.084162+0000","last_fullsized":"2023-10-16T12:12:01.084162+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:53:52.073154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.081881+0000","last_change":"2023-10-16T12:12:01.078045+0000","last_active":"2023-10-16T12:12:01.081881+0000","last_peered":"2023-10-16T12:12:01.081881+0000","last_clean":"2023-10-16T12:12:01.081881+0000","last_became_active":"2023-10-16T12:12:01.077691+0000","last_became_peered":"2023-10-16T12:12:01.077691+0000","last_unstale":"2023-10-16T12:12:01.081881+0000","last_undegraded":"2023-10-16T12:12:01.081881+0000","last_fullsized":"2023-10-16T12:12:01.081881+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:48:55.890617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.081870+0000","last_change":"2023-10-16T12:12:01.078118+0000","last_active":"2023-10-16T12:12:01.081870+0000","last_peered":"2023-10-16T12:12:01.081870+0000","last_clean":"2023-10-16T12:12:01.081870+0000","last_became_active":"2023-10-16T12:12:01.077244+0000","last_became_peered":"2023-10-16T12:12:01.077244+0000","last_unstale":"2023-10-16T12:12:01.081870+0000","last_undegraded":"2023-10-16T12:12:01.081870+0000","last_fullsized":"2023-10-16T12:12:01.081870+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:56:57.470920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.081869+0000","last_change":"2023-10-16T12:12:01.068208+0000","last_active":"2023-10-16T12:12:01.081869+0000","last_peered":"2023-10-16T12:12:01.081869+0000","last_clean":"2023-10-16T12:12:01.081869+0000","last_became_active":"2023-10-16T12:12:01.067322+0000","last_became_peered":"2023-10-16T12:12:01.067322+0000","last_unstale":"2023-10-16T12:12:01.081869+0000","last_undegraded":"2023-10-16T12:12:01.081869+0000","last_fullsized":"2023-10-16T12:12:01.081869+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:37:29.394307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.081975+0000","last_change":"2023-10-16T12:12:01.077363+0000","last_active":"2023-10-16T12:12:01.081975+0000","last_peered":"2023-10-16T12:12:01.081975+0000","last_clean":"2023-10-16T12:12:01.081975+0000","last_became_active":"2023-10-16T12:12:01.077058+0000","last_became_peered":"2023-10-16T12:12:01.077058+0000","last_unstale":"2023-10-16T12:12:01.081975+0000","last_undegraded":"2023-10-16T12:12:01.081975+0000","last_fullsized":"2023-10-16T12:12:01.081975+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:39:58.395936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.078465+0000","last_change":"2023-10-16T12:12:01.073897+0000","last_active":"2023-10-16T12:12:01.078465+0000","last_peered":"2023-10-16T12:12:01.078465+0000","last_clean":"2023-10-16T12:12:01.078465+0000","last_became_active":"2023-10-16T12:12:01.073582+0000","last_became_peered":"2023-10-16T12:12:01.073582+0000","last_unstale":"2023-10-16T12:12:01.078465+0000","last_undegraded":"2023-10-16T12:12:01.078465+0000","last_fullsized":"2023-10-16T12:12:01.078465+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:37:29.394307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.081984+0000","last_change":"2023-10-16T12:12:01.078416+0000","last_active":"2023-10-16T12:12:01.081984+0000","last_peered":"2023-10-16T12:12:01.081984+0000","last_clean":"2023-10-16T12:12:01.081984+0000","last_became_active":"2023-10-16T12:12:01.077663+0000","last_became_peered":"2023-10-16T12:12:01.077663+0000","last_unstale":"2023-10-16T12:12:01.081984+0000","last_undegraded":"2023-10-16T12:12:01.081984+0000","last_fullsized":"2023-10-16T12:12:01.081984+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:24:05.341039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.082603+0000","last_change":"2023-10-16T12:12:01.079732+0000","last_active":"2023-10-16T12:12:01.082603+0000","last_peered":"2023-10-16T12:12:01.082603+0000","last_clean":"2023-10-16T12:12:01.082603+0000","last_became_active":"2023-10-16T12:12:01.079438+0000","last_became_peered":"2023-10-16T12:12:01.079438+0000","last_unstale":"2023-10-16T12:12:01.082603+0000","last_undegraded":"2023-10-16T12:12:01.082603+0000","last_fullsized":"2023-10-16T12:12:01.082603+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:39:58.395936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.082770+0000","last_change":"2023-10-16T12:12:01.080566+0000","last_active":"2023-10-16T12:12:01.082770+0000","last_peered":"2023-10-16T12:12:01.082770+0000","last_clean":"2023-10-16T12:12:01.082770+0000","last_became_active":"2023-10-16T12:12:01.080352+0000","last_became_peered":"2023-10-16T12:12:01.080352+0000","last_unstale":"2023-10-16T12:12:01.082770+0000","last_undegraded":"2023-10-16T12:12:01.082770+0000","last_fullsized":"2023-10-16T12:12:01.082770+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:24:05.341039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.082410+0000","last_change":"2023-10-16T12:12:01.076923+0000","last_active":"2023-10-16T12:12:01.082410+0000","last_peered":"2023-10-16T12:12:01.082410+0000","last_clean":"2023-10-16T12:12:01.082410+0000","last_became_active":"2023-10-16T12:12:01.076643+0000","last_became_peered":"2023-10-16T12:12:01.076643+0000","last_unstale":"2023-10-16T12:12:01.082410+0000","last_undegraded":"2023-10-16T12:12:01.082410+0000","last_fullsized":"2023-10-16T12:12:01.082410+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:36:12.640742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.072491+0000","last_change":"2023-10-16T12:12:01.068827+0000","last_active":"2023-10-16T12:12:01.072491+0000","last_peered":"2023-10-16T12:12:01.072491+0000","last_clean":"2023-10-16T12:12:01.072491+0000","last_became_active":"2023-10-16T12:12:01.068529+0000","last_became_peered":"2023-10-16T12:12:01.068529+0000","last_unstale":"2023-10-16T12:12:01.072491+0000","last_undegraded":"2023-10-16T12:12:01.072491+0000","last_fullsized":"2023-10-16T12:12:01.072491+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:00:10.087413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.071112+0000","last_change":"2023-10-16T12:12:01.068771+0000","last_active":"2023-10-16T12:12:01.071112+0000","last_peered":"2023-10-16T12:12:01.071112+0000","last_clean":"2023-10-16T12:12:01.071112+0000","last_became_active":"2023-10-16T12:12:01.068413+0000","last_became_peered":"2023-10-16T12:12:01.068413+0000","last_unstale":"2023-10-16T12:12:01.071112+0000","last_undegraded":"2023-10-16T12:12:01.071112+0000","last_fullsized":"2023-10-16T12:12:01.071112+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:37:29.394307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.104749+0000","last_change":"2023-10-16T12:12:01.103461+0000","last_active":"2023-10-16T12:12:01.104749+0000","last_peered":"2023-10-16T12:12:01.104749+0000","last_clean":"2023-10-16T12:12:01.104749+0000","last_became_active":"2023-10-16T12:12:01.103183+0000","last_became_peered":"2023-10-16T12:12:01.103183+0000","last_unstale":"2023-10-16T12:12:01.104749+0000","last_undegraded":"2023-10-16T12:12:01.104749+0000","last_fullsized":"2023-10-16T12:12:01.104749+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:53:52.073154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.078699+0000","last_change":"2023-10-16T12:12:01.074009+0000","last_active":"2023-10-16T12:12:01.078699+0000","last_peered":"2023-10-16T12:12:01.078699+0000","last_clean":"2023-10-16T12:12:01.078699+0000","last_became_active":"2023-10-16T12:12:01.073366+0000","last_became_peered":"2023-10-16T12:12:01.073366+0000","last_unstale":"2023-10-16T12:12:01.078699+0000","last_undegraded":"2023-10-16T12:12:01.078699+0000","last_fullsized":"2023-10-16T12:12:01.078699+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:04:27.812653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.075942+0000","last_change":"2023-10-16T12:12:01.072484+0000","last_active":"2023-10-16T12:12:01.075942+0000","last_peered":"2023-10-16T12:12:01.075942+0000","last_clean":"2023-10-16T12:12:01.075942+0000","last_became_active":"2023-10-16T12:12:01.069343+0000","last_became_peered":"2023-10-16T12:12:01.069343+0000","last_unstale":"2023-10-16T12:12:01.075942+0000","last_undegraded":"2023-10-16T12:12:01.075942+0000","last_fullsized":"2023-10-16T12:12:01.075942+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:07:14.264264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.068813+0000","last_change":"2023-10-16T12:12:01.065484+0000","last_active":"2023-10-16T12:12:01.068813+0000","last_peered":"2023-10-16T12:12:01.068813+0000","last_clean":"2023-10-16T12:12:01.068813+0000","last_became_active":"2023-10-16T12:12:01.065246+0000","last_became_peered":"2023-10-16T12:12:01.065246+0000","last_unstale":"2023-10-16T12:12:01.068813+0000","last_undegraded":"2023-10-16T12:12:01.068813+0000","last_fullsized":"2023-10-16T12:12:01.068813+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:05:42.985553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.072238+0000","last_change":"2023-10-16T12:12:01.069295+0000","last_active":"2023-10-16T12:12:01.072238+0000","last_peered":"2023-10-16T12:12:01.072238+0000","last_clean":"2023-10-16T12:12:01.072238+0000","last_became_active":"2023-10-16T12:12:01.068358+0000","last_became_peered":"2023-10-16T12:12:01.068358+0000","last_unstale":"2023-10-16T12:12:01.072238+0000","last_undegraded":"2023-10-16T12:12:01.072238+0000","last_fullsized":"2023-10-16T12:12:01.072238+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:07:14.264264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T12:12:02.059552+0000","last_change":"2023-10-16T12:12:01.101694+0000","last_active":"2023-10-16T12:12:02.059552+0000","last_peered":"2023-10-16T12:12:02.059552+0000","last_clean":"2023-10-16T12:12:02.059552+0000","last_became_active":"2023-10-16T12:12:01.101362+0000","last_became_peered":"2023-10-16T12:12:01.101362+0000","last_unstale":"2023-10-16T12:12:02.059552+0000","last_undegraded":"2023-10-16T12:12:02.059552+0000","last_fullsized":"2023-10-16T12:12:02.059552+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:19:37.391525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.067793+0000","last_change":"2023-10-16T12:12:01.066030+0000","last_active":"2023-10-16T12:12:01.067793+0000","last_peered":"2023-10-16T12:12:01.067793+0000","last_clean":"2023-10-16T12:12:01.067793+0000","last_became_active":"2023-10-16T12:12:01.065177+0000","last_became_peered":"2023-10-16T12:12:01.065177+0000","last_unstale":"2023-10-16T12:12:01.067793+0000","last_undegraded":"2023-10-16T12:12:01.067793+0000","last_fullsized":"2023-10-16T12:12:01.067793+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:17:07.743741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.081958+0000","last_change":"2023-10-16T12:12:01.078282+0000","last_active":"2023-10-16T12:12:01.081958+0000","last_peered":"2023-10-16T12:12:01.081958+0000","last_clean":"2023-10-16T12:12:01.081958+0000","last_became_active":"2023-10-16T12:12:01.077877+0000","last_became_peered":"2023-10-16T12:12:01.077877+0000","last_unstale":"2023-10-16T12:12:01.081958+0000","last_undegraded":"2023-10-16T12:12:01.081958+0000","last_fullsized":"2023-10-16T12:12:01.081958+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:19:37.391525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.067109+0000","last_change":"2023-10-16T12:12:01.065779+0000","last_active":"2023-10-16T12:12:01.067109+0000","last_peered":"2023-10-16T12:12:01.067109+0000","last_clean":"2023-10-16T12:12:01.067109+0000","last_became_active":"2023-10-16T12:12:01.065460+0000","last_became_peered":"2023-10-16T12:12:01.065460+0000","last_unstale":"2023-10-16T12:12:01.067109+0000","last_undegraded":"2023-10-16T12:12:01.067109+0000","last_fullsized":"2023-10-16T12:12:01.067109+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:57:03.304620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.076017+0000","last_change":"2023-10-16T12:12:01.074312+0000","last_active":"2023-10-16T12:12:01.076017+0000","last_peered":"2023-10-16T12:12:01.076017+0000","last_clean":"2023-10-16T12:12:01.076017+0000","last_became_active":"2023-10-16T12:12:01.074001+0000","last_became_peered":"2023-10-16T12:12:01.074001+0000","last_unstale":"2023-10-16T12:12:01.076017+0000","last_undegraded":"2023-10-16T12:12:01.076017+0000","last_fullsized":"2023-10-16T12:12:01.076017+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:04:27.812653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.075925+0000","last_change":"2023-10-16T12:12:01.069686+0000","last_active":"2023-10-16T12:12:01.075925+0000","last_peered":"2023-10-16T12:12:01.075925+0000","last_clean":"2023-10-16T12:12:01.075925+0000","last_became_active":"2023-10-16T12:12:01.068948+0000","last_became_peered":"2023-10-16T12:12:01.068948+0000","last_unstale":"2023-10-16T12:12:01.075925+0000","last_undegraded":"2023-10-16T12:12:01.075925+0000","last_fullsized":"2023-10-16T12:12:01.075925+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:57:03.304620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T12:12:02.059032+0000","last_change":"2023-10-16T12:12:01.070907+0000","last_active":"2023-10-16T12:12:02.059032+0000","last_peered":"2023-10-16T12:12:02.059032+0000","last_clean":"2023-10-16T12:12:02.059032+0000","last_became_active":"2023-10-16T12:12:01.070570+0000","last_became_peered":"2023-10-16T12:12:01.070570+0000","last_unstale":"2023-10-16T12:12:02.059032+0000","last_undegraded":"2023-10-16T12:12:02.059032+0000","last_fullsized":"2023-10-16T12:12:02.059032+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:30:46.876254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.082656+0000","last_change":"2023-10-16T12:12:01.079832+0000","last_active":"2023-10-16T12:12:01.082656+0000","last_peered":"2023-10-16T12:12:01.082656+0000","last_clean":"2023-10-16T12:12:01.082656+0000","last_became_active":"2023-10-16T12:12:01.079553+0000","last_became_peered":"2023-10-16T12:12:01.079553+0000","last_unstale":"2023-10-16T12:12:01.082656+0000","last_undegraded":"2023-10-16T12:12:01.082656+0000","last_fullsized":"2023-10-16T12:12:01.082656+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:30:46.876254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"38'4","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.113925+0000","last_change":"2023-10-16T12:12:00.124682+0000","last_active":"2023-10-16T12:12:01.113925+0000","last_peered":"2023-10-16T12:12:01.113925+0000","last_clean":"2023-10-16T12:12:01.113925+0000","last_became_active":"2023-10-16T12:12:00.124271+0000","last_became_peered":"2023-10-16T12:12:00.124271+0000","last_unstale":"2023-10-16T12:12:01.113925+0000","last_undegraded":"2023-10-16T12:12:01.113925+0000","last_fullsized":"2023-10-16T12:12:01.113925+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_clean_scrub_stamp":"2023-10-16T12:11:59.089272+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:31:07.635977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T12:12:02.059483+0000","last_change":"2023-10-16T12:12:01.101563+0000","last_active":"2023-10-16T12:12:02.059483+0000","last_peered":"2023-10-16T12:12:02.059483+0000","last_clean":"2023-10-16T12:12:02.059483+0000","last_became_active":"2023-10-16T12:12:01.101170+0000","last_became_peered":"2023-10-16T12:12:01.101170+0000","last_unstale":"2023-10-16T12:12:02.059483+0000","last_undegraded":"2023-10-16T12:12:02.059483+0000","last_fullsized":"2023-10-16T12:12:02.059483+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:56:03.080747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"38'1","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.114532+0000","last_change":"2023-10-16T12:12:00.120024+0000","last_active":"2023-10-16T12:12:01.114532+0000","last_peered":"2023-10-16T12:12:01.114532+0000","last_clean":"2023-10-16T12:12:01.114532+0000","last_became_active":"2023-10-16T12:12:00.119782+0000","last_became_peered":"2023-10-16T12:12:00.119782+0000","last_unstale":"2023-10-16T12:12:01.114532+0000","last_undegraded":"2023-10-16T12:12:01.114532+0000","last_fullsized":"2023-10-16T12:12:01.114532+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_clean_scrub_stamp":"2023-10-16T12:11:59.089272+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:56:56.465657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.078821+0000","last_change":"2023-10-16T12:12:01.075997+0000","last_active":"2023-10-16T12:12:01.078821+0000","last_peered":"2023-10-16T12:12:01.078821+0000","last_clean":"2023-10-16T12:12:01.078821+0000","last_became_active":"2023-10-16T12:12:01.075682+0000","last_became_peered":"2023-10-16T12:12:01.075682+0000","last_unstale":"2023-10-16T12:12:01.078821+0000","last_undegraded":"2023-10-16T12:12:01.078821+0000","last_fullsized":"2023-10-16T12:12:01.078821+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:53:52.073154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"38'2","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T12:12:02.059586+0000","last_change":"2023-10-16T12:12:00.124348+0000","last_active":"2023-10-16T12:12:02.059586+0000","last_peered":"2023-10-16T12:12:02.059586+0000","last_clean":"2023-10-16T12:12:02.059586+0000","last_became_active":"2023-10-16T12:12:00.124001+0000","last_became_peered":"2023-10-16T12:12:00.124001+0000","last_unstale":"2023-10-16T12:12:02.059586+0000","last_undegraded":"2023-10-16T12:12:02.059586+0000","last_fullsized":"2023-10-16T12:12:02.059586+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_clean_scrub_stamp":"2023-10-16T12:11:59.089272+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:56:56.465657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T12:12:02.059761+0000","last_change":"2023-10-16T12:12:01.073542+0000","last_active":"2023-10-16T12:12:02.059761+0000","last_peered":"2023-10-16T12:12:02.059761+0000","last_clean":"2023-10-16T12:12:02.059761+0000","last_became_active":"2023-10-16T12:12:01.073241+0000","last_became_peered":"2023-10-16T12:12:01.073241+0000","last_unstale":"2023-10-16T12:12:02.059761+0000","last_undegraded":"2023-10-16T12:12:02.059761+0000","last_fullsized":"2023-10-16T12:12:02.059761+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:55:56.284339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"38'2","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.113519+0000","last_change":"2023-10-16T12:12:00.121183+0000","last_active":"2023-10-16T12:12:01.113519+0000","last_peered":"2023-10-16T12:12:01.113519+0000","last_clean":"2023-10-16T12:12:01.113519+0000","last_became_active":"2023-10-16T12:12:00.120862+0000","last_became_peered":"2023-10-16T12:12:00.120862+0000","last_unstale":"2023-10-16T12:12:01.113519+0000","last_undegraded":"2023-10-16T12:12:01.113519+0000","last_fullsized":"2023-10-16T12:12:01.113519+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_clean_scrub_stamp":"2023-10-16T12:11:59.089272+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:24:04.335776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.081867+0000","last_change":"2023-10-16T12:12:01.068095+0000","last_active":"2023-10-16T12:12:01.081867+0000","last_peered":"2023-10-16T12:12:01.081867+0000","last_clean":"2023-10-16T12:12:01.081867+0000","last_became_active":"2023-10-16T12:12:01.067752+0000","last_became_peered":"2023-10-16T12:12:01.067752+0000","last_unstale":"2023-10-16T12:12:01.081867+0000","last_undegraded":"2023-10-16T12:12:01.081867+0000","last_fullsized":"2023-10-16T12:12:01.081867+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:17:07.743741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"38'1","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.112258+0000","last_change":"2023-10-16T12:12:00.116133+0000","last_active":"2023-10-16T12:12:01.112258+0000","last_peered":"2023-10-16T12:12:01.112258+0000","last_clean":"2023-10-16T12:12:01.112258+0000","last_became_active":"2023-10-16T12:12:00.115780+0000","last_became_peered":"2023-10-16T12:12:00.115780+0000","last_unstale":"2023-10-16T12:12:01.112258+0000","last_undegraded":"2023-10-16T12:12:01.112258+0000","last_fullsized":"2023-10-16T12:12:01.112258+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_clean_scrub_stamp":"2023-10-16T12:11:59.089272+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:59:39.837329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.104758+0000","last_change":"2023-10-16T12:12:01.103464+0000","last_active":"2023-10-16T12:12:01.104758+0000","last_peered":"2023-10-16T12:12:01.104758+0000","last_clean":"2023-10-16T12:12:01.104758+0000","last_became_active":"2023-10-16T12:12:01.103208+0000","last_became_peered":"2023-10-16T12:12:01.103208+0000","last_unstale":"2023-10-16T12:12:01.104758+0000","last_undegraded":"2023-10-16T12:12:01.104758+0000","last_fullsized":"2023-10-16T12:12:01.104758+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:31:08.641239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"38'1","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.115055+0000","last_change":"2023-10-16T12:12:00.125114+0000","last_active":"2023-10-16T12:12:01.115055+0000","last_peered":"2023-10-16T12:12:01.115055+0000","last_clean":"2023-10-16T12:12:01.115055+0000","last_became_active":"2023-10-16T12:12:00.124415+0000","last_became_peered":"2023-10-16T12:12:00.124415+0000","last_unstale":"2023-10-16T12:12:01.115055+0000","last_undegraded":"2023-10-16T12:12:01.115055+0000","last_fullsized":"2023-10-16T12:12:01.115055+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_clean_scrub_stamp":"2023-10-16T12:11:59.089272+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:59:39.837329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.082006+0000","last_change":"2023-10-16T12:12:01.079973+0000","last_active":"2023-10-16T12:12:01.082006+0000","last_peered":"2023-10-16T12:12:01.082006+0000","last_clean":"2023-10-16T12:12:01.082006+0000","last_became_active":"2023-10-16T12:12:01.079701+0000","last_became_peered":"2023-10-16T12:12:01.079701+0000","last_unstale":"2023-10-16T12:12:01.082006+0000","last_undegraded":"2023-10-16T12:12:01.082006+0000","last_fullsized":"2023-10-16T12:12:01.082006+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:39:58.395936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T12:12:02.059187+0000","last_change":"2023-10-16T12:12:00.125087+0000","last_active":"2023-10-16T12:12:02.059187+0000","last_peered":"2023-10-16T12:12:02.059187+0000","last_clean":"2023-10-16T12:12:02.059187+0000","last_became_active":"2023-10-16T12:12:00.124806+0000","last_became_peered":"2023-10-16T12:12:00.124806+0000","last_unstale":"2023-10-16T12:12:02.059187+0000","last_undegraded":"2023-10-16T12:12:02.059187+0000","last_fullsized":"2023-10-16T12:12:02.059187+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_clean_scrub_stamp":"2023-10-16T12:11:59.089272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:31:07.635977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T12:12:02.060308+0000","last_change":"2023-10-16T12:12:01.077197+0000","last_active":"2023-10-16T12:12:02.060308+0000","last_peered":"2023-10-16T12:12:02.060308+0000","last_clean":"2023-10-16T12:12:02.060308+0000","last_became_active":"2023-10-16T12:12:01.076891+0000","last_became_peered":"2023-10-16T12:12:01.076891+0000","last_unstale":"2023-10-16T12:12:02.060308+0000","last_undegraded":"2023-10-16T12:12:02.060308+0000","last_fullsized":"2023-10-16T12:12:02.060308+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:39:38.834105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.063853+0000","last_change":"2023-10-16T12:12:00.124172+0000","last_active":"2023-10-16T12:12:01.063853+0000","last_peered":"2023-10-16T12:12:01.063853+0000","last_clean":"2023-10-16T12:12:01.063853+0000","last_became_active":"2023-10-16T12:12:00.123703+0000","last_became_peered":"2023-10-16T12:12:00.123703+0000","last_unstale":"2023-10-16T12:12:01.063853+0000","last_undegraded":"2023-10-16T12:12:01.063853+0000","last_fullsized":"2023-10-16T12:12:01.063853+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_clean_scrub_stamp":"2023-10-16T12:11:59.089272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:31:07.635977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.072443+0000","last_change":"2023-10-16T12:12:01.069201+0000","last_active":"2023-10-16T12:12:01.072443+0000","last_peered":"2023-10-16T12:12:01.072443+0000","last_clean":"2023-10-16T12:12:01.072443+0000","last_became_active":"2023-10-16T12:12:01.068512+0000","last_became_peered":"2023-10-16T12:12:01.068512+0000","last_unstale":"2023-10-16T12:12:01.072443+0000","last_undegraded":"2023-10-16T12:12:01.072443+0000","last_fullsized":"2023-10-16T12:12:01.072443+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:17:15.506280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T12:12:02.059705+0000","last_change":"2023-10-16T12:12:00.118028+0000","last_active":"2023-10-16T12:12:02.059705+0000","last_peered":"2023-10-16T12:12:02.059705+0000","last_clean":"2023-10-16T12:12:02.059705+0000","last_became_active":"2023-10-16T12:12:00.117729+0000","last_became_peered":"2023-10-16T12:12:00.117729+0000","last_unstale":"2023-10-16T12:12:02.059705+0000","last_undegraded":"2023-10-16T12:12:02.059705+0000","last_fullsized":"2023-10-16T12:12:02.059705+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_clean_scrub_stamp":"2023-10-16T12:11:59.089272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:17:49.299356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.069963+0000","last_change":"2023-10-16T12:12:01.067600+0000","last_active":"2023-10-16T12:12:01.069963+0000","last_peered":"2023-10-16T12:12:01.069963+0000","last_clean":"2023-10-16T12:12:01.069963+0000","last_became_active":"2023-10-16T12:12:01.067310+0000","last_became_peered":"2023-10-16T12:12:01.067310+0000","last_unstale":"2023-10-16T12:12:01.069963+0000","last_undegraded":"2023-10-16T12:12:01.069963+0000","last_fullsized":"2023-10-16T12:12:01.069963+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:36:12.640742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.068666+0000","last_change":"2023-10-16T12:12:00.120031+0000","last_active":"2023-10-16T12:12:01.068666+0000","last_peered":"2023-10-16T12:12:01.068666+0000","last_clean":"2023-10-16T12:12:01.068666+0000","last_became_active":"2023-10-16T12:12:00.119643+0000","last_became_peered":"2023-10-16T12:12:00.119643+0000","last_unstale":"2023-10-16T12:12:01.068666+0000","last_undegraded":"2023-10-16T12:12:01.068666+0000","last_fullsized":"2023-10-16T12:12:01.068666+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_clean_scrub_stamp":"2023-10-16T12:11:59.089272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:56:16.294667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.069072+0000","last_change":"2023-10-16T12:12:01.067631+0000","last_active":"2023-10-16T12:12:01.069072+0000","last_peered":"2023-10-16T12:12:01.069072+0000","last_clean":"2023-10-16T12:12:01.069072+0000","last_became_active":"2023-10-16T12:12:01.067340+0000","last_became_peered":"2023-10-16T12:12:01.067340+0000","last_unstale":"2023-10-16T12:12:01.069072+0000","last_undegraded":"2023-10-16T12:12:01.069072+0000","last_fullsized":"2023-10-16T12:12:01.069072+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:24:05.341039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"38'3","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.112712+0000","last_change":"2023-10-16T12:12:00.126200+0000","last_active":"2023-10-16T12:12:01.112712+0000","last_peered":"2023-10-16T12:12:01.112712+0000","last_clean":"2023-10-16T12:12:01.112712+0000","last_became_active":"2023-10-16T12:12:00.125890+0000","last_became_peered":"2023-10-16T12:12:00.125890+0000","last_unstale":"2023-10-16T12:12:01.112712+0000","last_undegraded":"2023-10-16T12:12:01.112712+0000","last_fullsized":"2023-10-16T12:12:01.112712+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_clean_scrub_stamp":"2023-10-16T12:11:59.089272+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:30:45.870992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.078853+0000","last_change":"2023-10-16T12:12:01.076502+0000","last_active":"2023-10-16T12:12:01.078853+0000","last_peered":"2023-10-16T12:12:01.078853+0000","last_clean":"2023-10-16T12:12:01.078853+0000","last_became_active":"2023-10-16T12:12:01.076122+0000","last_became_peered":"2023-10-16T12:12:01.076122+0000","last_unstale":"2023-10-16T12:12:01.078853+0000","last_undegraded":"2023-10-16T12:12:01.078853+0000","last_fullsized":"2023-10-16T12:12:01.078853+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:07:14.264264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"38'6","reported_seq":18,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T12:12:02.059919+0000","last_change":"2023-10-16T12:12:00.120746+0000","last_active":"2023-10-16T12:12:02.059919+0000","last_peered":"2023-10-16T12:12:02.059919+0000","last_clean":"2023-10-16T12:12:02.059919+0000","last_became_active":"2023-10-16T12:12:00.119566+0000","last_became_peered":"2023-10-16T12:12:00.119566+0000","last_unstale":"2023-10-16T12:12:02.059919+0000","last_undegraded":"2023-10-16T12:12:02.059919+0000","last_fullsized":"2023-10-16T12:12:02.059919+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_clean_scrub_stamp":"2023-10-16T12:11:59.089272+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:43:24.590290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.084708+0000","last_change":"2023-10-16T12:12:01.082940+0000","last_active":"2023-10-16T12:12:01.084708+0000","last_peered":"2023-10-16T12:12:01.084708+0000","last_clean":"2023-10-16T12:12:01.084708+0000","last_became_active":"2023-10-16T12:12:01.082461+0000","last_became_peered":"2023-10-16T12:12:01.082461+0000","last_unstale":"2023-10-16T12:12:01.084708+0000","last_undegraded":"2023-10-16T12:12:01.084708+0000","last_fullsized":"2023-10-16T12:12:01.084708+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:31:08.641239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"38'2","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T12:12:02.060335+0000","last_change":"2023-10-16T12:12:00.119080+0000","last_active":"2023-10-16T12:12:02.060335+0000","last_peered":"2023-10-16T12:12:02.060335+0000","last_clean":"2023-10-16T12:12:02.060335+0000","last_became_active":"2023-10-16T12:12:00.118762+0000","last_became_peered":"2023-10-16T12:12:00.118762+0000","last_unstale":"2023-10-16T12:12:02.060335+0000","last_undegraded":"2023-10-16T12:12:02.060335+0000","last_fullsized":"2023-10-16T12:12:02.060335+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_clean_scrub_stamp":"2023-10-16T12:11:59.089272+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:58:57.536737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T12:12:02.059616+0000","last_change":"2023-10-16T12:12:01.084845+0000","last_active":"2023-10-16T12:12:02.059616+0000","last_peered":"2023-10-16T12:12:02.059616+0000","last_clean":"2023-10-16T12:12:02.059616+0000","last_became_active":"2023-10-16T12:12:01.084486+0000","last_became_peered":"2023-10-16T12:12:01.084486+0000","last_unstale":"2023-10-16T12:12:02.059616+0000","last_undegraded":"2023-10-16T12:12:02.059616+0000","last_fullsized":"2023-10-16T12:12:02.059616+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:06:55.894696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T12:12:02.060027+0000","last_change":"2023-10-16T12:12:00.117820+0000","last_active":"2023-10-16T12:12:02.060027+0000","last_peered":"2023-10-16T12:12:02.060027+0000","last_clean":"2023-10-16T12:12:02.060027+0000","last_became_active":"2023-10-16T12:12:00.117588+0000","last_became_peered":"2023-10-16T12:12:00.117588+0000","last_unstale":"2023-10-16T12:12:02.060027+0000","last_undegraded":"2023-10-16T12:12:02.060027+0000","last_fullsized":"2023-10-16T12:12:02.060027+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_clean_scrub_stamp":"2023-10-16T12:11:59.089272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:39:57.390674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.075347+0000","last_change":"2023-10-16T12:12:01.069077+0000","last_active":"2023-10-16T12:12:01.075347+0000","last_peered":"2023-10-16T12:12:01.075347+0000","last_clean":"2023-10-16T12:12:01.075347+0000","last_became_active":"2023-10-16T12:12:01.068766+0000","last_became_peered":"2023-10-16T12:12:01.068766+0000","last_unstale":"2023-10-16T12:12:01.075347+0000","last_undegraded":"2023-10-16T12:12:01.075347+0000","last_fullsized":"2023-10-16T12:12:01.075347+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:53:52.073154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.059512+0000","last_change":"2023-10-16T12:12:00.126570+0000","last_active":"2023-10-16T12:12:01.059512+0000","last_peered":"2023-10-16T12:12:01.059512+0000","last_clean":"2023-10-16T12:12:01.059512+0000","last_became_active":"2023-10-16T12:12:00.126279+0000","last_became_peered":"2023-10-16T12:12:00.126279+0000","last_unstale":"2023-10-16T12:12:01.059512+0000","last_undegraded":"2023-10-16T12:12:01.059512+0000","last_fullsized":"2023-10-16T12:12:01.059512+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_clean_scrub_stamp":"2023-10-16T12:11:59.089272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:55:02.904354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T12:12:02.060156+0000","last_change":"2023-10-16T12:12:01.084851+0000","last_active":"2023-10-16T12:12:02.060156+0000","last_peered":"2023-10-16T12:12:02.060156+0000","last_clean":"2023-10-16T12:12:02.060156+0000","last_became_active":"2023-10-16T12:12:01.084506+0000","last_became_peered":"2023-10-16T12:12:01.084506+0000","last_unstale":"2023-10-16T12:12:02.060156+0000","last_undegraded":"2023-10-16T12:12:02.060156+0000","last_fullsized":"2023-10-16T12:12:02.060156+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:21:25.584112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"38'8","reported_seq":19,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.116033+0000","last_change":"2023-10-16T12:12:00.125281+0000","last_active":"2023-10-16T12:12:01.116033+0000","last_peered":"2023-10-16T12:12:01.116033+0000","last_clean":"2023-10-16T12:12:01.116033+0000","last_became_active":"2023-10-16T12:12:00.124960+0000","last_became_peered":"2023-10-16T12:12:00.124960+0000","last_unstale":"2023-10-16T12:12:01.116033+0000","last_undegraded":"2023-10-16T12:12:01.116033+0000","last_fullsized":"2023-10-16T12:12:01.116033+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_clean_scrub_stamp":"2023-10-16T12:11:59.089272+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:37:57.399730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T12:12:02.059751+0000","last_change":"2023-10-16T12:12:01.073634+0000","last_active":"2023-10-16T12:12:02.059751+0000","last_peered":"2023-10-16T12:12:02.059751+0000","last_clean":"2023-10-16T12:12:02.059751+0000","last_became_active":"2023-10-16T12:12:01.073224+0000","last_became_peered":"2023-10-16T12:12:01.073224+0000","last_unstale":"2023-10-16T12:12:02.059751+0000","last_undegraded":"2023-10-16T12:12:02.059751+0000","last_fullsized":"2023-10-16T12:12:02.059751+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:10:18.638711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"38'1","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.114900+0000","last_change":"2023-10-16T12:12:00.125068+0000","last_active":"2023-10-16T12:12:01.114900+0000","last_peered":"2023-10-16T12:12:01.114900+0000","last_clean":"2023-10-16T12:12:01.114900+0000","last_became_active":"2023-10-16T12:12:00.124767+0000","last_became_peered":"2023-10-16T12:12:00.124767+0000","last_unstale":"2023-10-16T12:12:01.114900+0000","last_undegraded":"2023-10-16T12:12:01.114900+0000","last_fullsized":"2023-10-16T12:12:01.114900+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_clean_scrub_stamp":"2023-10-16T12:11:59.089272+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:49:59.185665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.104779+0000","last_change":"2023-10-16T12:12:01.103617+0000","last_active":"2023-10-16T12:12:01.104779+0000","last_peered":"2023-10-16T12:12:01.104779+0000","last_clean":"2023-10-16T12:12:01.104779+0000","last_became_active":"2023-10-16T12:12:01.103320+0000","last_became_peered":"2023-10-16T12:12:01.103320+0000","last_unstale":"2023-10-16T12:12:01.104779+0000","last_undegraded":"2023-10-16T12:12:01.104779+0000","last_fullsized":"2023-10-16T12:12:01.104779+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:37:58.404993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"38'2","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.115534+0000","last_change":"2023-10-16T12:12:00.124060+0000","last_active":"2023-10-16T12:12:01.115534+0000","last_peered":"2023-10-16T12:12:01.115534+0000","last_clean":"2023-10-16T12:12:01.115534+0000","last_became_active":"2023-10-16T12:12:00.123767+0000","last_became_peered":"2023-10-16T12:12:00.123767+0000","last_unstale":"2023-10-16T12:12:01.115534+0000","last_undegraded":"2023-10-16T12:12:01.115534+0000","last_fullsized":"2023-10-16T12:12:01.115534+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_clean_scrub_stamp":"2023-10-16T12:11:59.089272+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:59:39.837329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.078801+0000","last_change":"2023-10-16T12:12:01.074798+0000","last_active":"2023-10-16T12:12:01.078801+0000","last_peered":"2023-10-16T12:12:01.078801+0000","last_clean":"2023-10-16T12:12:01.078801+0000","last_became_active":"2023-10-16T12:12:01.074429+0000","last_became_peered":"2023-10-16T12:12:01.074429+0000","last_unstale":"2023-10-16T12:12:01.078801+0000","last_undegraded":"2023-10-16T12:12:01.078801+0000","last_fullsized":"2023-10-16T12:12:01.078801+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:57:03.304620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"38'1","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.113152+0000","last_change":"2023-10-16T12:12:00.122920+0000","last_active":"2023-10-16T12:12:01.113152+0000","last_peered":"2023-10-16T12:12:01.113152+0000","last_clean":"2023-10-16T12:12:01.113152+0000","last_became_active":"2023-10-16T12:12:00.122628+0000","last_became_peered":"2023-10-16T12:12:00.122628+0000","last_unstale":"2023-10-16T12:12:01.113152+0000","last_undegraded":"2023-10-16T12:12:01.113152+0000","last_fullsized":"2023-10-16T12:12:01.113152+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_clean_scrub_stamp":"2023-10-16T12:11:59.089272+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:05:22.800311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.075322+0000","last_change":"2023-10-16T12:12:01.074661+0000","last_active":"2023-10-16T12:12:01.075322+0000","last_peered":"2023-10-16T12:12:01.075322+0000","last_clean":"2023-10-16T12:12:01.075322+0000","last_became_active":"2023-10-16T12:12:01.074319+0000","last_became_peered":"2023-10-16T12:12:01.074319+0000","last_unstale":"2023-10-16T12:12:01.075322+0000","last_undegraded":"2023-10-16T12:12:01.075322+0000","last_fullsized":"2023-10-16T12:12:01.075322+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:41:53.032282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T12:12:02.059475+0000","last_change":"2023-10-16T12:12:00.124350+0000","last_active":"2023-10-16T12:12:02.059475+0000","last_peered":"2023-10-16T12:12:02.059475+0000","last_clean":"2023-10-16T12:12:02.059475+0000","last_became_active":"2023-10-16T12:12:00.123989+0000","last_became_peered":"2023-10-16T12:12:00.123989+0000","last_unstale":"2023-10-16T12:12:02.059475+0000","last_undegraded":"2023-10-16T12:12:02.059475+0000","last_fullsized":"2023-10-16T12:12:02.059475+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_clean_scrub_stamp":"2023-10-16T12:11:59.089272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:37:57.399730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.075959+0000","last_change":"2023-10-16T12:12:01.071010+0000","last_active":"2023-10-16T12:12:01.075959+0000","last_peered":"2023-10-16T12:12:01.075959+0000","last_clean":"2023-10-16T12:12:01.075959+0000","last_became_active":"2023-10-16T12:12:01.070742+0000","last_became_peered":"2023-10-16T12:12:01.070742+0000","last_unstale":"2023-10-16T12:12:01.075959+0000","last_undegraded":"2023-10-16T12:12:01.075959+0000","last_fullsized":"2023-10-16T12:12:01.075959+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:56:57.470920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"38'2","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.113064+0000","last_change":"2023-10-16T12:12:00.122643+0000","last_active":"2023-10-16T12:12:01.113064+0000","last_peered":"2023-10-16T12:12:01.113064+0000","last_clean":"2023-10-16T12:12:01.113064+0000","last_became_active":"2023-10-16T12:12:00.122364+0000","last_became_peered":"2023-10-16T12:12:00.122364+0000","last_unstale":"2023-10-16T12:12:01.113064+0000","last_undegraded":"2023-10-16T12:12:01.113064+0000","last_fullsized":"2023-10-16T12:12:01.113064+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_clean_scrub_stamp":"2023-10-16T12:11:59.089272+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:37:15.293764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.084123+0000","last_change":"2023-10-16T12:12:01.082270+0000","last_active":"2023-10-16T12:12:01.084123+0000","last_peered":"2023-10-16T12:12:01.084123+0000","last_clean":"2023-10-16T12:12:01.084123+0000","last_became_active":"2023-10-16T12:12:01.081910+0000","last_became_peered":"2023-10-16T12:12:01.081910+0000","last_unstale":"2023-10-16T12:12:01.084123+0000","last_undegraded":"2023-10-16T12:12:01.084123+0000","last_fullsized":"2023-10-16T12:12:01.084123+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:07:14.264264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.057632+0000","last_change":"2023-10-16T12:12:00.121951+0000","last_active":"2023-10-16T12:12:01.057632+0000","last_peered":"2023-10-16T12:12:01.057632+0000","last_clean":"2023-10-16T12:12:01.057632+0000","last_became_active":"2023-10-16T12:12:00.121686+0000","last_became_peered":"2023-10-16T12:12:00.121686+0000","last_unstale":"2023-10-16T12:12:01.057632+0000","last_undegraded":"2023-10-16T12:12:01.057632+0000","last_fullsized":"2023-10-16T12:12:01.057632+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_clean_scrub_stamp":"2023-10-16T12:11:59.089272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:30:17.984189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.081932+0000","last_change":"2023-10-16T12:12:01.078333+0000","last_active":"2023-10-16T12:12:01.081932+0000","last_peered":"2023-10-16T12:12:01.081932+0000","last_clean":"2023-10-16T12:12:01.081932+0000","last_became_active":"2023-10-16T12:12:01.077928+0000","last_became_peered":"2023-10-16T12:12:01.077928+0000","last_unstale":"2023-10-16T12:12:01.081932+0000","last_undegraded":"2023-10-16T12:12:01.081932+0000","last_fullsized":"2023-10-16T12:12:01.081932+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:13:01.102564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.065675+0000","last_change":"2023-10-16T12:12:00.121496+0000","last_active":"2023-10-16T12:12:01.065675+0000","last_peered":"2023-10-16T12:12:01.065675+0000","last_clean":"2023-10-16T12:12:01.065675+0000","last_became_active":"2023-10-16T12:12:00.121194+0000","last_became_peered":"2023-10-16T12:12:00.121194+0000","last_unstale":"2023-10-16T12:12:01.065675+0000","last_undegraded":"2023-10-16T12:12:01.065675+0000","last_fullsized":"2023-10-16T12:12:01.065675+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_clean_scrub_stamp":"2023-10-16T12:11:59.089272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:56:02.075484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T12:12:02.060229+0000","last_change":"2023-10-16T12:12:01.077074+0000","last_active":"2023-10-16T12:12:02.060229+0000","last_peered":"2023-10-16T12:12:02.060229+0000","last_clean":"2023-10-16T12:12:02.060229+0000","last_became_active":"2023-10-16T12:12:01.076578+0000","last_became_peered":"2023-10-16T12:12:01.076578+0000","last_unstale":"2023-10-16T12:12:02.060229+0000","last_undegraded":"2023-10-16T12:12:02.060229+0000","last_fullsized":"2023-10-16T12:12:02.060229+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:44:26.822860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.057966+0000","last_change":"2023-10-16T12:12:00.114664+0000","last_active":"2023-10-16T12:12:01.057966+0000","last_peered":"2023-10-16T12:12:01.057966+0000","last_clean":"2023-10-16T12:12:01.057966+0000","last_became_active":"2023-10-16T12:12:00.114399+0000","last_became_peered":"2023-10-16T12:12:00.114399+0000","last_unstale":"2023-10-16T12:12:01.057966+0000","last_undegraded":"2023-10-16T12:12:01.057966+0000","last_fullsized":"2023-10-16T12:12:01.057966+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_clean_scrub_stamp":"2023-10-16T12:11:59.089272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:03:57.508990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T12:12:02.059380+0000","last_change":"2023-10-16T12:12:01.101889+0000","last_active":"2023-10-16T12:12:02.059380+0000","last_peered":"2023-10-16T12:12:02.059380+0000","last_clean":"2023-10-16T12:12:02.059380+0000","last_became_active":"2023-10-16T12:12:01.101477+0000","last_became_peered":"2023-10-16T12:12:01.101477+0000","last_unstale":"2023-10-16T12:12:02.059380+0000","last_undegraded":"2023-10-16T12:12:02.059380+0000","last_fullsized":"2023-10-16T12:12:02.059380+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:56:17.299930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"38'2","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.112812+0000","last_change":"2023-10-16T12:12:00.126072+0000","last_active":"2023-10-16T12:12:01.112812+0000","last_peered":"2023-10-16T12:12:01.112812+0000","last_clean":"2023-10-16T12:12:01.112812+0000","last_became_active":"2023-10-16T12:12:00.125763+0000","last_became_peered":"2023-10-16T12:12:00.125763+0000","last_unstale":"2023-10-16T12:12:01.112812+0000","last_undegraded":"2023-10-16T12:12:01.112812+0000","last_fullsized":"2023-10-16T12:12:01.112812+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_clean_scrub_stamp":"2023-10-16T12:11:59.089272+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:58:32.014737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.077731+0000","last_change":"2023-10-16T12:12:01.069504+0000","last_active":"2023-10-16T12:12:01.077731+0000","last_peered":"2023-10-16T12:12:01.077731+0000","last_clean":"2023-10-16T12:12:01.077731+0000","last_became_active":"2023-10-16T12:12:01.069200+0000","last_became_peered":"2023-10-16T12:12:01.069200+0000","last_unstale":"2023-10-16T12:12:01.077731+0000","last_undegraded":"2023-10-16T12:12:01.077731+0000","last_fullsized":"2023-10-16T12:12:01.077731+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:56:17.299930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.057731+0000","last_change":"2023-10-16T12:12:00.112373+0000","last_active":"2023-10-16T12:12:01.057731+0000","last_peered":"2023-10-16T12:12:01.057731+0000","last_clean":"2023-10-16T12:12:01.057731+0000","last_became_active":"2023-10-16T12:12:00.112045+0000","last_became_peered":"2023-10-16T12:12:00.112045+0000","last_unstale":"2023-10-16T12:12:01.057731+0000","last_undegraded":"2023-10-16T12:12:01.057731+0000","last_fullsized":"2023-10-16T12:12:01.057731+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_clean_scrub_stamp":"2023-10-16T12:11:59.089272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:42:40.590200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T12:12:02.059859+0000","last_change":"2023-10-16T12:12:01.079305+0000","last_active":"2023-10-16T12:12:02.059859+0000","last_peered":"2023-10-16T12:12:02.059859+0000","last_clean":"2023-10-16T12:12:02.059859+0000","last_became_active":"2023-10-16T12:12:01.079092+0000","last_became_peered":"2023-10-16T12:12:01.079092+0000","last_unstale":"2023-10-16T12:12:02.059859+0000","last_undegraded":"2023-10-16T12:12:02.059859+0000","last_fullsized":"2023-10-16T12:12:02.059859+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:15:26.287918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.063493+0000","last_change":"2023-10-16T12:12:00.125702+0000","last_active":"2023-10-16T12:12:01.063493+0000","last_peered":"2023-10-16T12:12:01.063493+0000","last_clean":"2023-10-16T12:12:01.063493+0000","last_became_active":"2023-10-16T12:12:00.125419+0000","last_became_peered":"2023-10-16T12:12:00.125419+0000","last_unstale":"2023-10-16T12:12:01.063493+0000","last_undegraded":"2023-10-16T12:12:01.063493+0000","last_fullsized":"2023-10-16T12:12:01.063493+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_clean_scrub_stamp":"2023-10-16T12:11:59.089272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:39:57.390674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.084208+0000","last_change":"2023-10-16T12:12:01.082772+0000","last_active":"2023-10-16T12:12:01.084208+0000","last_peered":"2023-10-16T12:12:01.084208+0000","last_clean":"2023-10-16T12:12:01.084208+0000","last_became_active":"2023-10-16T12:12:01.082034+0000","last_became_peered":"2023-10-16T12:12:01.082034+0000","last_unstale":"2023-10-16T12:12:01.084208+0000","last_undegraded":"2023-10-16T12:12:01.084208+0000","last_fullsized":"2023-10-16T12:12:01.084208+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:57:03.304620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.063663+0000","last_change":"2023-10-16T12:12:00.115273+0000","last_active":"2023-10-16T12:12:01.063663+0000","last_peered":"2023-10-16T12:12:01.063663+0000","last_clean":"2023-10-16T12:12:01.063663+0000","last_became_active":"2023-10-16T12:12:00.114991+0000","last_became_peered":"2023-10-16T12:12:00.114991+0000","last_unstale":"2023-10-16T12:12:01.063663+0000","last_undegraded":"2023-10-16T12:12:01.063663+0000","last_fullsized":"2023-10-16T12:12:01.063663+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_clean_scrub_stamp":"2023-10-16T12:11:59.089272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:59:39.837329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T12:12:02.059176+0000","last_change":"2023-10-16T12:12:01.064260+0000","last_active":"2023-10-16T12:12:02.059176+0000","last_peered":"2023-10-16T12:12:02.059176+0000","last_clean":"2023-10-16T12:12:02.059176+0000","last_became_active":"2023-10-16T12:12:01.063980+0000","last_became_peered":"2023-10-16T12:12:01.063980+0000","last_unstale":"2023-10-16T12:12:02.059176+0000","last_undegraded":"2023-10-16T12:12:02.059176+0000","last_fullsized":"2023-10-16T12:12:02.059176+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:48:55.890617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.060214+0000","last_change":"2023-10-16T12:12:00.116949+0000","last_active":"2023-10-16T12:12:01.060214+0000","last_peered":"2023-10-16T12:12:01.060214+0000","last_clean":"2023-10-16T12:12:01.060214+0000","last_became_active":"2023-10-16T12:12:00.116619+0000","last_became_peered":"2023-10-16T12:12:00.116619+0000","last_unstale":"2023-10-16T12:12:01.060214+0000","last_undegraded":"2023-10-16T12:12:01.060214+0000","last_fullsized":"2023-10-16T12:12:01.060214+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_clean_scrub_stamp":"2023-10-16T12:11:59.089272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:24:04.335776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.078742+0000","last_change":"2023-10-16T12:12:01.074865+0000","last_active":"2023-10-16T12:12:01.078742+0000","last_peered":"2023-10-16T12:12:01.078742+0000","last_clean":"2023-10-16T12:12:01.078742+0000","last_became_active":"2023-10-16T12:12:01.074296+0000","last_became_peered":"2023-10-16T12:12:01.074296+0000","last_unstale":"2023-10-16T12:12:01.078742+0000","last_undegraded":"2023-10-16T12:12:01.078742+0000","last_fullsized":"2023-10-16T12:12:01.078742+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:17:07.743741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.063727+0000","last_change":"2023-10-16T12:12:00.121241+0000","last_active":"2023-10-16T12:12:01.063727+0000","last_peered":"2023-10-16T12:12:01.063727+0000","last_clean":"2023-10-16T12:12:01.063727+0000","last_became_active":"2023-10-16T12:12:00.120916+0000","last_became_peered":"2023-10-16T12:12:00.120916+0000","last_unstale":"2023-10-16T12:12:01.063727+0000","last_undegraded":"2023-10-16T12:12:01.063727+0000","last_fullsized":"2023-10-16T12:12:01.063727+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_clean_scrub_stamp":"2023-10-16T12:11:59.089272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:36:11.635480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.072309+0000","last_change":"2023-10-16T12:12:01.068857+0000","last_active":"2023-10-16T12:12:01.072309+0000","last_peered":"2023-10-16T12:12:01.072309+0000","last_clean":"2023-10-16T12:12:01.072309+0000","last_became_active":"2023-10-16T12:12:01.068538+0000","last_became_peered":"2023-10-16T12:12:01.068538+0000","last_unstale":"2023-10-16T12:12:01.072309+0000","last_undegraded":"2023-10-16T12:12:01.072309+0000","last_fullsized":"2023-10-16T12:12:01.072309+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:31:08.641239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T12:12:02.059338+0000","last_change":"2023-10-16T12:12:00.117949+0000","last_active":"2023-10-16T12:12:02.059338+0000","last_peered":"2023-10-16T12:12:02.059338+0000","last_clean":"2023-10-16T12:12:02.059338+0000","last_became_active":"2023-10-16T12:12:00.117507+0000","last_became_peered":"2023-10-16T12:12:00.117507+0000","last_unstale":"2023-10-16T12:12:02.059338+0000","last_undegraded":"2023-10-16T12:12:02.059338+0000","last_fullsized":"2023-10-16T12:12:02.059338+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_clean_scrub_stamp":"2023-10-16T12:11:59.089272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:13:00.097302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.082311+0000","last_change":"2023-10-16T12:12:01.079605+0000","last_active":"2023-10-16T12:12:01.082311+0000","last_peered":"2023-10-16T12:12:01.082311+0000","last_clean":"2023-10-16T12:12:01.082311+0000","last_became_active":"2023-10-16T12:12:01.079293+0000","last_became_peered":"2023-10-16T12:12:01.079293+0000","last_unstale":"2023-10-16T12:12:01.082311+0000","last_undegraded":"2023-10-16T12:12:01.082311+0000","last_fullsized":"2023-10-16T12:12:01.082311+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:56:03.080747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.063834+0000","last_change":"2023-10-16T12:12:00.122781+0000","last_active":"2023-10-16T12:12:01.063834+0000","last_peered":"2023-10-16T12:12:01.063834+0000","last_clean":"2023-10-16T12:12:01.063834+0000","last_became_active":"2023-10-16T12:12:00.122521+0000","last_became_peered":"2023-10-16T12:12:00.122521+0000","last_unstale":"2023-10-16T12:12:01.063834+0000","last_undegraded":"2023-10-16T12:12:01.063834+0000","last_fullsized":"2023-10-16T12:12:01.063834+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_clean_scrub_stamp":"2023-10-16T12:11:59.089272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:48:54.885355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":179,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-16T12:12:00.111128+0000","last_change":"2023-10-16T12:10:34.746820+0000","last_active":"2023-10-16T12:12:00.111128+0000","last_peered":"2023-10-16T12:12:00.111128+0000","last_clean":"2023-10-16T12:12:00.111128+0000","last_became_active":"2023-10-16T12:10:34.745574+0000","last_became_peered":"2023-10-16T12:10:34.745574+0000","last_unstale":"2023-10-16T12:12:00.111128+0000","last_undegraded":"2023-10-16T12:12:00.111128+0000","last_fullsized":"2023-10-16T12:12:00.111128+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_clean_scrub_stamp":"2023-10-16T12:10:33.577111+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:14.461986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574871,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28044,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343796,"statfs":{"total":96636764160,"available":96608047104,"internally_reserved":0,"allocated":1048576,"data_stored":779269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:11:34 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.318}]},{"osd":1,"last update":"Mon Oct 16 12:11:34 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.44400000000000001}]},{"osd":2,"last update":"Mon Oct 16 12:11:34 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.52200000000000002}]},{"osd":3,"last update":"Mon Oct 16 12:11:34 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.50600000000000001}]},{"osd":4,"last update":"Mon Oct 16 12:11:34 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.60099999999999998}]},{"osd":5,"last update":"Mon Oct 16 12:11:34 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.65200000000000002}]},{"osd":6,"last update":"Mon Oct 16 12:11:34 2023","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.628}]}]},{"osd":6,"up_from":13,"seq":55834574869,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27452,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344388,"statfs":{"total":96636764160,"available":96608653312,"internally_reserved":0,"allocated":442368,"data_stored":187801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:11:36 2023","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":5.9349999999999996,"5min":5.9349999999999996,"15min":5.9349999999999996},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":5.9489999999999998,"5min":5.9489999999999998,"15min":5.9489999999999998},"last":0.76700000000000002}]},{"osd":1,"last update":"Mon Oct 16 12:11:36 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.69299999999999995}]},{"osd":2,"last update":"Mon Oct 16 12:11:36 2023","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":8.0489999999999995,"5min":8.0489999999999995,"15min":8.0489999999999995},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":2.855,"5min":2.855,"15min":2.855},"last":0.80800000000000005}]},{"osd":3,"last update":"Mon Oct 16 12:11:36 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.78600000000000003}]},{"osd":4,"last update":"Mon Oct 16 12:11:36 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.66500000000000004}]},{"osd":5,"last update":"Mon Oct 16 12:11:36 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.58999999999999997}]},{"osd":7,"last update":"Mon Oct 16 12:11:36 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.61899999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574869,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27452,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344388,"statfs":{"total":96636764160,"available":96608653312,"internally_reserved":0,"allocated":442368,"data_stored":187309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:11:32 2023","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.37,"15min":0.37},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.318}]},{"osd":2,"last update":"Mon Oct 16 12:11:32 2023","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.253}]},{"osd":3,"last update":"Mon Oct 16 12:11:32 2023","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"last":0.23699999999999999},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.28299999999999997}]},{"osd":4,"last update":"Mon Oct 16 12:11:32 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.55100000000000005}]},{"osd":5,"last update":"Mon Oct 16 12:11:32 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.48699999999999999}]},{"osd":6,"last update":"Mon Oct 16 12:11:32 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.44500000000000001}]},{"osd":7,"last update":"Mon Oct 16 12:11:32 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.58899999999999997}]}]},{"osd":0,"up_from":13,"seq":55834574870,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28032,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343808,"statfs":{"total":96636764160,"available":96608059392,"internally_reserved":0,"allocated":1036288,"data_stored":777621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 12:11:33 2023","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.253},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"last":0.28899999999999998}]},{"osd":2,"last update":"Mon Oct 16 12:11:33 2023","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.32700000000000001}]},{"osd":3,"last update":"Mon Oct 16 12:11:33 2023","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.378,"15min":0.378},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.36899999999999999}]},{"osd":4,"last update":"Mon Oct 16 12:11:33 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.68000000000000005}]},{"osd":5,"last update":"Mon Oct 16 12:11:33 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.54800000000000004}]},{"osd":6,"last update":"Mon Oct 16 12:11:33 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.53000000000000003}]},{"osd":7,"last update":"Mon Oct 16 12:11:33 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":6.7300000000000004,"5min":6.7300000000000004,"15min":6.7300000000000004},"last":0.65900000000000003}]}]},{"osd":2,"up_from":13,"seq":55834574870,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27452,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344388,"statfs":{"total":96636764160,"available":96608653312,"internally_reserved":0,"allocated":442368,"data_stored":187289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:11:35 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"last":0.35199999999999998}]},{"osd":1,"last update":"Mon Oct 16 12:11:35 2023","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.378,"15min":0.378},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.312},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.33700000000000002}]},{"osd":3,"last update":"Mon Oct 16 12:11:35 2023","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.442,"5min":0.442,"15min":0.442},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.38500000000000001}]},{"osd":4,"last update":"Mon Oct 16 12:11:35 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.79900000000000004}]},{"osd":5,"last update":"Mon Oct 16 12:11:35 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.56000000000000005}]},{"osd":6,"last update":"Mon Oct 16 12:11:35 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.78200000000000003}]},{"osd":7,"last update":"Mon Oct 16 12:11:35 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.82199999999999995}]}]},{"osd":3,"up_from":13,"seq":55834574869,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27464,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344376,"statfs":{"total":96636764160,"available":96608641024,"internally_reserved":0,"allocated":454656,"data_stored":188865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:11:36 2023","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.36499999999999999}]},{"osd":1,"last update":"Mon Oct 16 12:11:36 2023","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"last":0.309}]},{"osd":2,"last update":"Mon Oct 16 12:11:36 2023","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.34899999999999998}]},{"osd":4,"last update":"Mon Oct 16 12:11:36 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.52600000000000002}]},{"osd":5,"last update":"Mon Oct 16 12:11:36 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.502},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.41399999999999998}]},{"osd":6,"last update":"Mon Oct 16 12:11:36 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.42699999999999999}]},{"osd":7,"last update":"Mon Oct 16 12:11:36 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.44},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.57899999999999996}]}]},{"osd":4,"up_from":13,"seq":55834574868,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27456,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344384,"statfs":{"total":96636764160,"available":96608649216,"internally_reserved":0,"allocated":446464,"data_stored":187823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:11:33 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.171,"5min":1.171,"15min":1.171},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.63900000000000001}]},{"osd":1,"last update":"Mon Oct 16 12:11:33 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.433},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.67600000000000005}]},{"osd":2,"last update":"Mon Oct 16 12:11:33 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.73299999999999998}]},{"osd":3,"last update":"Mon Oct 16 12:11:33 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.60999999999999999}]},{"osd":5,"last update":"Mon Oct 16 12:11:33 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.70399999999999996}]},{"osd":6,"last update":"Mon Oct 16 12:11:33 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.79100000000000004}]},{"osd":7,"last update":"Mon Oct 16 12:11:33 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.747},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.71999999999999997}]}]},{"osd":5,"up_from":13,"seq":55834574869,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27456,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344384,"statfs":{"total":96636764160,"available":96608649216,"internally_reserved":0,"allocated":446464,"data_stored":187331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:11:34 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.41699999999999998}]},{"osd":1,"last update":"Mon Oct 16 12:11:34 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.59699999999999998}]},{"osd":2,"last update":"Mon Oct 16 12:11:34 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.49099999999999999}]},{"osd":3,"last update":"Mon Oct 16 12:11:34 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.69499999999999995}]},{"osd":4,"last update":"Mon Oct 16 12:11:34 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.57499999999999996}]},{"osd":6,"last update":"Mon Oct 16 12:11:34 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.63}]},{"osd":7,"last update":"Mon Oct 16 12:11:34 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.74399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T12:12:05.271 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T12:12:05.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.428+0000 7fe934d05700 1 Processor -- start 2023-10-16T12:12:05.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.428+0000 7fe934d05700 1 -- start start 2023-10-16T12:12:05.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.429+0000 7fe934d05700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe9300cdfa0 0x7fe9300b2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:05.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.429+0000 7fe934d05700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe9300d0bb0 0x7fe9300b2980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:05.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.429+0000 7fe934d05700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9300d0fe0 con 0x7fe9300ce380 2023-10-16T12:12:05.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.429+0000 7fe934d05700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9300d1210 con 0x7fe9300d0bb0 2023-10-16T12:12:05.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.429+0000 7fe934d05700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9300cbc30 con 0x7fe9300cdfa0 2023-10-16T12:12:05.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.430+0000 7fe92ed9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe9300d0bb0 0x7fe9300b2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:05.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.430+0000 7fe92ed9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe9300d0bb0 0x7fe9300b2980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35652/0 (socket says 172.21.15.16:35652) 2023-10-16T12:12:05.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.430+0000 7fe92dd9b700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe9300ce380 0x7fe9300d04a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:53154/0 (socket says 172.21.15.16:53154) 2023-10-16T12:12:05.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.430+0000 7fe92ed9d700 1 -- 172.21.15.16:0/61790806 learned_addr learned my addr 172.21.15.16:0/61790806 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:12:05.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.430+0000 7fe92e59c700 1 --2- 172.21.15.16:0/61790806 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe9300cdfa0 0x7fe9300b2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:05.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.430+0000 7fe92ed9d700 1 -- 172.21.15.16:0/61790806 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe9300cdfa0 msgr2=0x7fe9300b2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:05.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.430+0000 7fe92ed9d700 1 --2- 172.21.15.16:0/61790806 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe9300cdfa0 0x7fe9300b2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:05.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.430+0000 7fe92ed9d700 1 -- 172.21.15.16:0/61790806 >> v1:172.21.15.16:6789/0 conn(0x7fe9300ce380 legacy=0x7fe9300d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:05.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.430+0000 7fe92ed9d700 1 -- 172.21.15.16:0/61790806 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9300cbf10 con 0x7fe9300d0bb0 2023-10-16T12:12:05.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.430+0000 7fe92d59a700 1 -- 172.21.15.16:0/61790806 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2993959571 0 0) 0x7fe9300d0fe0 con 0x7fe9300ce380 2023-10-16T12:12:05.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.430+0000 7fe92ed9d700 1 --2- 172.21.15.16:0/61790806 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9300d0bb0 0x7fe9300b2980 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fe928009e50 tx=0x7fe92800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aab7f1b6f9df4034 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:05.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.430+0000 7fe92d59a700 1 -- 172.21.15.16:0/61790806 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe92800e070 con 0x7fe9300d0bb0 2023-10-16T12:12:05.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.430+0000 7fe92d59a700 1 -- 172.21.15.16:0/61790806 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe92800b9c0 con 0x7fe9300d0bb0 2023-10-16T12:12:05.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.430+0000 7fe92d59a700 1 -- 172.21.15.16:0/61790806 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe92800bd70 con 0x7fe9300d0bb0 2023-10-16T12:12:05.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.431+0000 7fe934d05700 1 -- 172.21.15.16:0/61790806 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9300d0bb0 msgr2=0x7fe9300b2980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:05.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.431+0000 7fe934d05700 1 --2- 172.21.15.16:0/61790806 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9300d0bb0 0x7fe9300b2980 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fe928009e50 tx=0x7fe92800b330 comp rx=0 tx=0).stop 2023-10-16T12:12:05.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.431+0000 7fe934d05700 1 -- 172.21.15.16:0/61790806 shutdown_connections 2023-10-16T12:12:05.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.431+0000 7fe934d05700 1 --2- 172.21.15.16:0/61790806 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9300d0bb0 0x7fe9300b2980 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:05.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.431+0000 7fe934d05700 1 --2- 172.21.15.16:0/61790806 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe9300cdfa0 0x7fe9300b2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:05.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.431+0000 7fe934d05700 1 -- 172.21.15.16:0/61790806 >> 172.21.15.16:0/61790806 conn(0x7fe9300d14b0 msgr2=0x7fe9300d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:05.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.431+0000 7fe934d05700 1 -- 172.21.15.16:0/61790806 shutdown_connections 2023-10-16T12:12:05.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.431+0000 7fe934d05700 1 -- 172.21.15.16:0/61790806 wait complete. 2023-10-16T12:12:05.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.432+0000 7fe934d05700 1 Processor -- start 2023-10-16T12:12:05.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.432+0000 7fe934d05700 1 -- start start 2023-10-16T12:12:05.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.433+0000 7fe934d05700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe9300cdfa0 0x7fe930154530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:05.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.433+0000 7fe934d05700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe9300ce380 0x7fe9300c8580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:05.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.433+0000 7fe934d05700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9300d0bb0 0x7fe9300c59d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:05.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.433+0000 7fe934d05700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9300b5560 con 0x7fe9300d0bb0 2023-10-16T12:12:05.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.433+0000 7fe934d05700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9300b4950 con 0x7fe9300cdfa0 2023-10-16T12:12:05.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.433+0000 7fe934d05700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9300b4c30 con 0x7fe9300ce380 2023-10-16T12:12:05.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.433+0000 7fe92ed9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9300d0bb0 0x7fe9300c59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:05.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.433+0000 7fe92ed9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9300d0bb0 0x7fe9300c59d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35658/0 (socket says 172.21.15.16:35658) 2023-10-16T12:12:05.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.433+0000 7fe92ed9d700 1 -- 172.21.15.16:0/784068085 learned_addr learned my addr 172.21.15.16:0/784068085 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:12:05.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.433+0000 7fe92e59c700 1 --2- 172.21.15.16:0/784068085 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe9300cdfa0 0x7fe930154530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:05.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.433+0000 7fe92dd9b700 1 --2- 172.21.15.16:0/784068085 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe9300ce380 0x7fe9300c8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:05.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.434+0000 7fe92ed9d700 1 -- 172.21.15.16:0/784068085 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe9300ce380 msgr2=0x7fe9300c8580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:05.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.434+0000 7fe92ed9d700 1 --2- 172.21.15.16:0/784068085 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe9300ce380 0x7fe9300c8580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:05.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.434+0000 7fe92ed9d700 1 -- 172.21.15.16:0/784068085 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe9300cdfa0 msgr2=0x7fe930154530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:05.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.434+0000 7fe92ed9d700 1 --2- 172.21.15.16:0/784068085 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe9300cdfa0 0x7fe930154530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:05.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.434+0000 7fe92ed9d700 1 -- 172.21.15.16:0/784068085 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe928009b30 con 0x7fe9300d0bb0 2023-10-16T12:12:05.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.434+0000 7fe92ed9d700 1 --2- 172.21.15.16:0/784068085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9300d0bb0 0x7fe9300c59d0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fe92800b730 tx=0x7fe928004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:05.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.434+0000 7fe926ffd700 1 -- 172.21.15.16:0/784068085 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9280099b0 con 0x7fe9300d0bb0 2023-10-16T12:12:05.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.434+0000 7fe926ffd700 1 -- 172.21.15.16:0/784068085 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9280092e0 con 0x7fe9300d0bb0 2023-10-16T12:12:05.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.434+0000 7fe934d05700 1 -- 172.21.15.16:0/784068085 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9300b4f10 con 0x7fe9300d0bb0 2023-10-16T12:12:05.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.434+0000 7fe926ffd700 1 -- 172.21.15.16:0/784068085 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe92801b650 con 0x7fe9300d0bb0 2023-10-16T12:12:05.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.434+0000 7fe934d05700 1 -- 172.21.15.16:0/784068085 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe930154e30 con 0x7fe9300d0bb0 2023-10-16T12:12:05.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.435+0000 7fe934d05700 1 -- 172.21.15.16:0/784068085 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe93004ebb0 con 0x7fe9300d0bb0 2023-10-16T12:12:05.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.435+0000 7fe926ffd700 1 -- 172.21.15.16:0/784068085 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fe92801b7b0 con 0x7fe9300d0bb0 2023-10-16T12:12:05.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.436+0000 7fe926ffd700 1 --2- 172.21.15.16:0/784068085 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fe91006af00 0x7fe91006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:05.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.436+0000 7fe926ffd700 1 -- 172.21.15.16:0/784068085 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fe92808acc0 con 0x7fe9300d0bb0 2023-10-16T12:12:05.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.436+0000 7fe92e59c700 1 --2- 172.21.15.16:0/784068085 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fe91006af00 0x7fe91006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:05.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.436+0000 7fe92e59c700 1 --2- 172.21.15.16:0/784068085 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fe91006af00 0x7fe91006d3c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fe920009930 tx=0x7fe920008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:05.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.441+0000 7fe926ffd700 1 -- 172.21.15.16:0/784068085 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe928055cc0 con 0x7fe9300d0bb0 2023-10-16T12:12:05.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.640+0000 7fe934d05700 1 -- 172.21.15.16:0/784068085 --> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe9300cc4d0 con 0x7fe91006af00 2023-10-16T12:12:05.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.648+0000 7fe926ffd700 1 -- 172.21.15.16:0/784068085 <== mgr.4104 v2:172.21.15.16:6832/365921436 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289939 (secure 0 0 0) 0x7fe920009e00 con 0x7fe91006af00 2023-10-16T12:12:05.654 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:12:05.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.652+0000 7fe934d05700 1 -- 172.21.15.16:0/784068085 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fe91006af00 msgr2=0x7fe91006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:05.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.652+0000 7fe934d05700 1 --2- 172.21.15.16:0/784068085 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fe91006af00 0x7fe91006d3c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fe920009930 tx=0x7fe920008040 comp rx=0 tx=0).stop 2023-10-16T12:12:05.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.652+0000 7fe934d05700 1 -- 172.21.15.16:0/784068085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9300d0bb0 msgr2=0x7fe9300c59d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:05.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.652+0000 7fe934d05700 1 --2- 172.21.15.16:0/784068085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9300d0bb0 0x7fe9300c59d0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fe92800b730 tx=0x7fe928004ad0 comp rx=0 tx=0).stop 2023-10-16T12:12:05.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.652+0000 7fe934d05700 1 -- 172.21.15.16:0/784068085 shutdown_connections 2023-10-16T12:12:05.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.652+0000 7fe934d05700 1 --2- 172.21.15.16:0/784068085 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fe91006af00 0x7fe91006d3c0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:05.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.652+0000 7fe934d05700 1 --2- 172.21.15.16:0/784068085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9300d0bb0 0x7fe9300c59d0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:05.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.652+0000 7fe934d05700 1 --2- 172.21.15.16:0/784068085 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe9300cdfa0 0x7fe930154530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:05.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.652+0000 7fe934d05700 1 --2- 172.21.15.16:0/784068085 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe9300ce380 0x7fe9300c8580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:05.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.652+0000 7fe934d05700 1 -- 172.21.15.16:0/784068085 >> 172.21.15.16:0/784068085 conn(0x7fe9300d14b0 msgr2=0x7fe93006c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:05.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.652+0000 7fe934d05700 1 -- 172.21.15.16:0/784068085 shutdown_connections 2023-10-16T12:12:05.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.652+0000 7fe934d05700 1 -- 172.21.15.16:0/784068085 wait complete. 2023-10-16T12:12:05.658 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-10-16T12:12:05.666 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":79,"stamp":"2023-10-16T12:12:05.113754+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":97,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220808,"kb_used_data":4648,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754753912,"statfs":{"total":773094113280,"available":772868005888,"internally_reserved":0,"allocated":4759552,"data_stored":2683308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12740,"internal_metadata":221236796},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001232"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T12:12:02.059592+0000","last_change":"2023-10-16T12:12:01.084878+0000","last_active":"2023-10-16T12:12:02.059592+0000","last_peered":"2023-10-16T12:12:02.059592+0000","last_clean":"2023-10-16T12:12:02.059592+0000","last_became_active":"2023-10-16T12:12:01.084510+0000","last_became_peered":"2023-10-16T12:12:01.084510+0000","last_unstale":"2023-10-16T12:12:02.059592+0000","last_undegraded":"2023-10-16T12:12:02.059592+0000","last_fullsized":"2023-10-16T12:12:02.059592+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:12:52.469257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T12:12:02.059773+0000","last_change":"2023-10-16T12:12:01.073425+0000","last_active":"2023-10-16T12:12:02.059773+0000","last_peered":"2023-10-16T12:12:02.059773+0000","last_clean":"2023-10-16T12:12:02.059773+0000","last_became_active":"2023-10-16T12:12:01.072828+0000","last_became_peered":"2023-10-16T12:12:01.072828+0000","last_unstale":"2023-10-16T12:12:02.059773+0000","last_undegraded":"2023-10-16T12:12:02.059773+0000","last_fullsized":"2023-10-16T12:12:02.059773+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:59:23.612795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.084213+0000","last_change":"2023-10-16T12:12:01.082901+0000","last_active":"2023-10-16T12:12:01.084213+0000","last_peered":"2023-10-16T12:12:01.084213+0000","last_clean":"2023-10-16T12:12:01.084213+0000","last_became_active":"2023-10-16T12:12:01.082289+0000","last_became_peered":"2023-10-16T12:12:01.082289+0000","last_unstale":"2023-10-16T12:12:01.084213+0000","last_undegraded":"2023-10-16T12:12:01.084213+0000","last_fullsized":"2023-10-16T12:12:01.084213+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:36:12.640742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.072468+0000","last_change":"2023-10-16T12:12:01.069237+0000","last_active":"2023-10-16T12:12:01.072468+0000","last_peered":"2023-10-16T12:12:01.072468+0000","last_clean":"2023-10-16T12:12:01.072468+0000","last_became_active":"2023-10-16T12:12:01.068932+0000","last_became_peered":"2023-10-16T12:12:01.068932+0000","last_unstale":"2023-10-16T12:12:01.072468+0000","last_undegraded":"2023-10-16T12:12:01.072468+0000","last_fullsized":"2023-10-16T12:12:01.072468+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:57:57.533990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.071377+0000","last_change":"2023-10-16T12:12:01.068561+0000","last_active":"2023-10-16T12:12:01.071377+0000","last_peered":"2023-10-16T12:12:01.071377+0000","last_clean":"2023-10-16T12:12:01.071377+0000","last_became_active":"2023-10-16T12:12:01.068253+0000","last_became_peered":"2023-10-16T12:12:01.068253+0000","last_unstale":"2023-10-16T12:12:01.071377+0000","last_undegraded":"2023-10-16T12:12:01.071377+0000","last_fullsized":"2023-10-16T12:12:01.071377+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:30.033807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T12:12:02.060374+0000","last_change":"2023-10-16T12:12:01.101436+0000","last_active":"2023-10-16T12:12:02.060374+0000","last_peered":"2023-10-16T12:12:02.060374+0000","last_clean":"2023-10-16T12:12:02.060374+0000","last_became_active":"2023-10-16T12:12:01.101066+0000","last_became_peered":"2023-10-16T12:12:01.101066+0000","last_unstale":"2023-10-16T12:12:02.060374+0000","last_undegraded":"2023-10-16T12:12:02.060374+0000","last_fullsized":"2023-10-16T12:12:02.060374+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:50:00.190927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.081964+0000","last_change":"2023-10-16T12:12:01.078197+0000","last_active":"2023-10-16T12:12:01.081964+0000","last_peered":"2023-10-16T12:12:01.081964+0000","last_clean":"2023-10-16T12:12:01.081964+0000","last_became_active":"2023-10-16T12:12:01.077180+0000","last_became_peered":"2023-10-16T12:12:01.077180+0000","last_unstale":"2023-10-16T12:12:01.081964+0000","last_undegraded":"2023-10-16T12:12:01.081964+0000","last_fullsized":"2023-10-16T12:12:01.081964+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:07:14.264264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T12:12:02.059254+0000","last_change":"2023-10-16T12:12:01.079867+0000","last_active":"2023-10-16T12:12:02.059254+0000","last_peered":"2023-10-16T12:12:02.059254+0000","last_clean":"2023-10-16T12:12:02.059254+0000","last_became_active":"2023-10-16T12:12:01.079610+0000","last_became_peered":"2023-10-16T12:12:01.079610+0000","last_unstale":"2023-10-16T12:12:02.059254+0000","last_undegraded":"2023-10-16T12:12:02.059254+0000","last_fullsized":"2023-10-16T12:12:02.059254+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:33:26.016966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.084162+0000","last_change":"2023-10-16T12:12:01.082539+0000","last_active":"2023-10-16T12:12:01.084162+0000","last_peered":"2023-10-16T12:12:01.084162+0000","last_clean":"2023-10-16T12:12:01.084162+0000","last_became_active":"2023-10-16T12:12:01.082201+0000","last_became_peered":"2023-10-16T12:12:01.082201+0000","last_unstale":"2023-10-16T12:12:01.084162+0000","last_undegraded":"2023-10-16T12:12:01.084162+0000","last_fullsized":"2023-10-16T12:12:01.084162+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:53:52.073154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.081881+0000","last_change":"2023-10-16T12:12:01.078045+0000","last_active":"2023-10-16T12:12:01.081881+0000","last_peered":"2023-10-16T12:12:01.081881+0000","last_clean":"2023-10-16T12:12:01.081881+0000","last_became_active":"2023-10-16T12:12:01.077691+0000","last_became_peered":"2023-10-16T12:12:01.077691+0000","last_unstale":"2023-10-16T12:12:01.081881+0000","last_undegraded":"2023-10-16T12:12:01.081881+0000","last_fullsized":"2023-10-16T12:12:01.081881+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:48:55.890617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.081870+0000","last_change":"2023-10-16T12:12:01.078118+0000","last_active":"2023-10-16T12:12:01.081870+0000","last_peered":"2023-10-16T12:12:01.081870+0000","last_clean":"2023-10-16T12:12:01.081870+0000","last_became_active":"2023-10-16T12:12:01.077244+0000","last_became_peered":"2023-10-16T12:12:01.077244+0000","last_unstale":"2023-10-16T12:12:01.081870+0000","last_undegraded":"2023-10-16T12:12:01.081870+0000","last_fullsized":"2023-10-16T12:12:01.081870+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:56:57.470920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.081869+0000","last_change":"2023-10-16T12:12:01.068208+0000","last_active":"2023-10-16T12:12:01.081869+0000","last_peered":"2023-10-16T12:12:01.081869+0000","last_clean":"2023-10-16T12:12:01.081869+0000","last_became_active":"2023-10-16T12:12:01.067322+0000","last_became_peered":"2023-10-16T12:12:01.067322+0000","last_unstale":"2023-10-16T12:12:01.081869+0000","last_undegraded":"2023-10-16T12:12:01.081869+0000","last_fullsized":"2023-10-16T12:12:01.081869+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:37:29.394307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.081975+0000","last_change":"2023-10-16T12:12:01.077363+0000","last_active":"2023-10-16T12:12:01.081975+0000","last_peered":"2023-10-16T12:12:01.081975+0000","last_clean":"2023-10-16T12:12:01.081975+0000","last_became_active":"2023-10-16T12:12:01.077058+0000","last_became_peered":"2023-10-16T12:12:01.077058+0000","last_unstale":"2023-10-16T12:12:01.081975+0000","last_undegraded":"2023-10-16T12:12:01.081975+0000","last_fullsized":"2023-10-16T12:12:01.081975+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:39:58.395936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.078465+0000","last_change":"2023-10-16T12:12:01.073897+0000","last_active":"2023-10-16T12:12:01.078465+0000","last_peered":"2023-10-16T12:12:01.078465+0000","last_clean":"2023-10-16T12:12:01.078465+0000","last_became_active":"2023-10-16T12:12:01.073582+0000","last_became_peered":"2023-10-16T12:12:01.073582+0000","last_unstale":"2023-10-16T12:12:01.078465+0000","last_undegraded":"2023-10-16T12:12:01.078465+0000","last_fullsized":"2023-10-16T12:12:01.078465+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:37:29.394307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.081984+0000","last_change":"2023-10-16T12:12:01.078416+0000","last_active":"2023-10-16T12:12:01.081984+0000","last_peered":"2023-10-16T12:12:01.081984+0000","last_clean":"2023-10-16T12:12:01.081984+0000","last_became_active":"2023-10-16T12:12:01.077663+0000","last_became_peered":"2023-10-16T12:12:01.077663+0000","last_unstale":"2023-10-16T12:12:01.081984+0000","last_undegraded":"2023-10-16T12:12:01.081984+0000","last_fullsized":"2023-10-16T12:12:01.081984+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:24:05.341039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.082603+0000","last_change":"2023-10-16T12:12:01.079732+0000","last_active":"2023-10-16T12:12:01.082603+0000","last_peered":"2023-10-16T12:12:01.082603+0000","last_clean":"2023-10-16T12:12:01.082603+0000","last_became_active":"2023-10-16T12:12:01.079438+0000","last_became_peered":"2023-10-16T12:12:01.079438+0000","last_unstale":"2023-10-16T12:12:01.082603+0000","last_undegraded":"2023-10-16T12:12:01.082603+0000","last_fullsized":"2023-10-16T12:12:01.082603+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:39:58.395936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.082770+0000","last_change":"2023-10-16T12:12:01.080566+0000","last_active":"2023-10-16T12:12:01.082770+0000","last_peered":"2023-10-16T12:12:01.082770+0000","last_clean":"2023-10-16T12:12:01.082770+0000","last_became_active":"2023-10-16T12:12:01.080352+0000","last_became_peered":"2023-10-16T12:12:01.080352+0000","last_unstale":"2023-10-16T12:12:01.082770+0000","last_undegraded":"2023-10-16T12:12:01.082770+0000","last_fullsized":"2023-10-16T12:12:01.082770+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:24:05.341039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.082410+0000","last_change":"2023-10-16T12:12:01.076923+0000","last_active":"2023-10-16T12:12:01.082410+0000","last_peered":"2023-10-16T12:12:01.082410+0000","last_clean":"2023-10-16T12:12:01.082410+0000","last_became_active":"2023-10-16T12:12:01.076643+0000","last_became_peered":"2023-10-16T12:12:01.076643+0000","last_unstale":"2023-10-16T12:12:01.082410+0000","last_undegraded":"2023-10-16T12:12:01.082410+0000","last_fullsized":"2023-10-16T12:12:01.082410+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:36:12.640742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.072491+0000","last_change":"2023-10-16T12:12:01.068827+0000","last_active":"2023-10-16T12:12:01.072491+0000","last_peered":"2023-10-16T12:12:01.072491+0000","last_clean":"2023-10-16T12:12:01.072491+0000","last_became_active":"2023-10-16T12:12:01.068529+0000","last_became_peered":"2023-10-16T12:12:01.068529+0000","last_unstale":"2023-10-16T12:12:01.072491+0000","last_undegraded":"2023-10-16T12:12:01.072491+0000","last_fullsized":"2023-10-16T12:12:01.072491+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:00:10.087413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.071112+0000","last_change":"2023-10-16T12:12:01.068771+0000","last_active":"2023-10-16T12:12:01.071112+0000","last_peered":"2023-10-16T12:12:01.071112+0000","last_clean":"2023-10-16T12:12:01.071112+0000","last_became_active":"2023-10-16T12:12:01.068413+0000","last_became_peered":"2023-10-16T12:12:01.068413+0000","last_unstale":"2023-10-16T12:12:01.071112+0000","last_undegraded":"2023-10-16T12:12:01.071112+0000","last_fullsized":"2023-10-16T12:12:01.071112+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:37:29.394307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.104749+0000","last_change":"2023-10-16T12:12:01.103461+0000","last_active":"2023-10-16T12:12:01.104749+0000","last_peered":"2023-10-16T12:12:01.104749+0000","last_clean":"2023-10-16T12:12:01.104749+0000","last_became_active":"2023-10-16T12:12:01.103183+0000","last_became_peered":"2023-10-16T12:12:01.103183+0000","last_unstale":"2023-10-16T12:12:01.104749+0000","last_undegraded":"2023-10-16T12:12:01.104749+0000","last_fullsized":"2023-10-16T12:12:01.104749+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:53:52.073154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.078699+0000","last_change":"2023-10-16T12:12:01.074009+0000","last_active":"2023-10-16T12:12:01.078699+0000","last_peered":"2023-10-16T12:12:01.078699+0000","last_clean":"2023-10-16T12:12:01.078699+0000","last_became_active":"2023-10-16T12:12:01.073366+0000","last_became_peered":"2023-10-16T12:12:01.073366+0000","last_unstale":"2023-10-16T12:12:01.078699+0000","last_undegraded":"2023-10-16T12:12:01.078699+0000","last_fullsized":"2023-10-16T12:12:01.078699+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:04:27.812653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.075942+0000","last_change":"2023-10-16T12:12:01.072484+0000","last_active":"2023-10-16T12:12:01.075942+0000","last_peered":"2023-10-16T12:12:01.075942+0000","last_clean":"2023-10-16T12:12:01.075942+0000","last_became_active":"2023-10-16T12:12:01.069343+0000","last_became_peered":"2023-10-16T12:12:01.069343+0000","last_unstale":"2023-10-16T12:12:01.075942+0000","last_undegraded":"2023-10-16T12:12:01.075942+0000","last_fullsized":"2023-10-16T12:12:01.075942+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:07:14.264264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.068813+0000","last_change":"2023-10-16T12:12:01.065484+0000","last_active":"2023-10-16T12:12:01.068813+0000","last_peered":"2023-10-16T12:12:01.068813+0000","last_clean":"2023-10-16T12:12:01.068813+0000","last_became_active":"2023-10-16T12:12:01.065246+0000","last_became_peered":"2023-10-16T12:12:01.065246+0000","last_unstale":"2023-10-16T12:12:01.068813+0000","last_undegraded":"2023-10-16T12:12:01.068813+0000","last_fullsized":"2023-10-16T12:12:01.068813+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:05:42.985553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.072238+0000","last_change":"2023-10-16T12:12:01.069295+0000","last_active":"2023-10-16T12:12:01.072238+0000","last_peered":"2023-10-16T12:12:01.072238+0000","last_clean":"2023-10-16T12:12:01.072238+0000","last_became_active":"2023-10-16T12:12:01.068358+0000","last_became_peered":"2023-10-16T12:12:01.068358+0000","last_unstale":"2023-10-16T12:12:01.072238+0000","last_undegraded":"2023-10-16T12:12:01.072238+0000","last_fullsized":"2023-10-16T12:12:01.072238+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:07:14.264264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T12:12:02.059552+0000","last_change":"2023-10-16T12:12:01.101694+0000","last_active":"2023-10-16T12:12:02.059552+0000","last_peered":"2023-10-16T12:12:02.059552+0000","last_clean":"2023-10-16T12:12:02.059552+0000","last_became_active":"2023-10-16T12:12:01.101362+0000","last_became_peered":"2023-10-16T12:12:01.101362+0000","last_unstale":"2023-10-16T12:12:02.059552+0000","last_undegraded":"2023-10-16T12:12:02.059552+0000","last_fullsized":"2023-10-16T12:12:02.059552+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:19:37.391525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.067793+0000","last_change":"2023-10-16T12:12:01.066030+0000","last_active":"2023-10-16T12:12:01.067793+0000","last_peered":"2023-10-16T12:12:01.067793+0000","last_clean":"2023-10-16T12:12:01.067793+0000","last_became_active":"2023-10-16T12:12:01.065177+0000","last_became_peered":"2023-10-16T12:12:01.065177+0000","last_unstale":"2023-10-16T12:12:01.067793+0000","last_undegraded":"2023-10-16T12:12:01.067793+0000","last_fullsized":"2023-10-16T12:12:01.067793+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:17:07.743741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.081958+0000","last_change":"2023-10-16T12:12:01.078282+0000","last_active":"2023-10-16T12:12:01.081958+0000","last_peered":"2023-10-16T12:12:01.081958+0000","last_clean":"2023-10-16T12:12:01.081958+0000","last_became_active":"2023-10-16T12:12:01.077877+0000","last_became_peered":"2023-10-16T12:12:01.077877+0000","last_unstale":"2023-10-16T12:12:01.081958+0000","last_undegraded":"2023-10-16T12:12:01.081958+0000","last_fullsized":"2023-10-16T12:12:01.081958+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:19:37.391525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.067109+0000","last_change":"2023-10-16T12:12:01.065779+0000","last_active":"2023-10-16T12:12:01.067109+0000","last_peered":"2023-10-16T12:12:01.067109+0000","last_clean":"2023-10-16T12:12:01.067109+0000","last_became_active":"2023-10-16T12:12:01.065460+0000","last_became_peered":"2023-10-16T12:12:01.065460+0000","last_unstale":"2023-10-16T12:12:01.067109+0000","last_undegraded":"2023-10-16T12:12:01.067109+0000","last_fullsized":"2023-10-16T12:12:01.067109+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:57:03.304620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.076017+0000","last_change":"2023-10-16T12:12:01.074312+0000","last_active":"2023-10-16T12:12:01.076017+0000","last_peered":"2023-10-16T12:12:01.076017+0000","last_clean":"2023-10-16T12:12:01.076017+0000","last_became_active":"2023-10-16T12:12:01.074001+0000","last_became_peered":"2023-10-16T12:12:01.074001+0000","last_unstale":"2023-10-16T12:12:01.076017+0000","last_undegraded":"2023-10-16T12:12:01.076017+0000","last_fullsized":"2023-10-16T12:12:01.076017+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:04:27.812653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.075925+0000","last_change":"2023-10-16T12:12:01.069686+0000","last_active":"2023-10-16T12:12:01.075925+0000","last_peered":"2023-10-16T12:12:01.075925+0000","last_clean":"2023-10-16T12:12:01.075925+0000","last_became_active":"2023-10-16T12:12:01.068948+0000","last_became_peered":"2023-10-16T12:12:01.068948+0000","last_unstale":"2023-10-16T12:12:01.075925+0000","last_undegraded":"2023-10-16T12:12:01.075925+0000","last_fullsized":"2023-10-16T12:12:01.075925+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:57:03.304620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T12:12:02.059032+0000","last_change":"2023-10-16T12:12:01.070907+0000","last_active":"2023-10-16T12:12:02.059032+0000","last_peered":"2023-10-16T12:12:02.059032+0000","last_clean":"2023-10-16T12:12:02.059032+0000","last_became_active":"2023-10-16T12:12:01.070570+0000","last_became_peered":"2023-10-16T12:12:01.070570+0000","last_unstale":"2023-10-16T12:12:02.059032+0000","last_undegraded":"2023-10-16T12:12:02.059032+0000","last_fullsized":"2023-10-16T12:12:02.059032+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:30:46.876254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.082656+0000","last_change":"2023-10-16T12:12:01.079832+0000","last_active":"2023-10-16T12:12:01.082656+0000","last_peered":"2023-10-16T12:12:01.082656+0000","last_clean":"2023-10-16T12:12:01.082656+0000","last_became_active":"2023-10-16T12:12:01.079553+0000","last_became_peered":"2023-10-16T12:12:01.079553+0000","last_unstale":"2023-10-16T12:12:01.082656+0000","last_undegraded":"2023-10-16T12:12:01.082656+0000","last_fullsized":"2023-10-16T12:12:01.082656+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:30:46.876254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"38'4","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.113925+0000","last_change":"2023-10-16T12:12:00.124682+0000","last_active":"2023-10-16T12:12:01.113925+0000","last_peered":"2023-10-16T12:12:01.113925+0000","last_clean":"2023-10-16T12:12:01.113925+0000","last_became_active":"2023-10-16T12:12:00.124271+0000","last_became_peered":"2023-10-16T12:12:00.124271+0000","last_unstale":"2023-10-16T12:12:01.113925+0000","last_undegraded":"2023-10-16T12:12:01.113925+0000","last_fullsized":"2023-10-16T12:12:01.113925+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_clean_scrub_stamp":"2023-10-16T12:11:59.089272+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:31:07.635977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T12:12:02.059483+0000","last_change":"2023-10-16T12:12:01.101563+0000","last_active":"2023-10-16T12:12:02.059483+0000","last_peered":"2023-10-16T12:12:02.059483+0000","last_clean":"2023-10-16T12:12:02.059483+0000","last_became_active":"2023-10-16T12:12:01.101170+0000","last_became_peered":"2023-10-16T12:12:01.101170+0000","last_unstale":"2023-10-16T12:12:02.059483+0000","last_undegraded":"2023-10-16T12:12:02.059483+0000","last_fullsized":"2023-10-16T12:12:02.059483+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:56:03.080747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"38'1","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.114532+0000","last_change":"2023-10-16T12:12:00.120024+0000","last_active":"2023-10-16T12:12:01.114532+0000","last_peered":"2023-10-16T12:12:01.114532+0000","last_clean":"2023-10-16T12:12:01.114532+0000","last_became_active":"2023-10-16T12:12:00.119782+0000","last_became_peered":"2023-10-16T12:12:00.119782+0000","last_unstale":"2023-10-16T12:12:01.114532+0000","last_undegraded":"2023-10-16T12:12:01.114532+0000","last_fullsized":"2023-10-16T12:12:01.114532+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_clean_scrub_stamp":"2023-10-16T12:11:59.089272+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:56:56.465657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.078821+0000","last_change":"2023-10-16T12:12:01.075997+0000","last_active":"2023-10-16T12:12:01.078821+0000","last_peered":"2023-10-16T12:12:01.078821+0000","last_clean":"2023-10-16T12:12:01.078821+0000","last_became_active":"2023-10-16T12:12:01.075682+0000","last_became_peered":"2023-10-16T12:12:01.075682+0000","last_unstale":"2023-10-16T12:12:01.078821+0000","last_undegraded":"2023-10-16T12:12:01.078821+0000","last_fullsized":"2023-10-16T12:12:01.078821+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:53:52.073154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"38'2","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T12:12:02.059586+0000","last_change":"2023-10-16T12:12:00.124348+0000","last_active":"2023-10-16T12:12:02.059586+0000","last_peered":"2023-10-16T12:12:02.059586+0000","last_clean":"2023-10-16T12:12:02.059586+0000","last_became_active":"2023-10-16T12:12:00.124001+0000","last_became_peered":"2023-10-16T12:12:00.124001+0000","last_unstale":"2023-10-16T12:12:02.059586+0000","last_undegraded":"2023-10-16T12:12:02.059586+0000","last_fullsized":"2023-10-16T12:12:02.059586+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_clean_scrub_stamp":"2023-10-16T12:11:59.089272+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:56:56.465657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T12:12:02.059761+0000","last_change":"2023-10-16T12:12:01.073542+0000","last_active":"2023-10-16T12:12:02.059761+0000","last_peered":"2023-10-16T12:12:02.059761+0000","last_clean":"2023-10-16T12:12:02.059761+0000","last_became_active":"2023-10-16T12:12:01.073241+0000","last_became_peered":"2023-10-16T12:12:01.073241+0000","last_unstale":"2023-10-16T12:12:02.059761+0000","last_undegraded":"2023-10-16T12:12:02.059761+0000","last_fullsized":"2023-10-16T12:12:02.059761+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:55:56.284339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"38'2","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.113519+0000","last_change":"2023-10-16T12:12:00.121183+0000","last_active":"2023-10-16T12:12:01.113519+0000","last_peered":"2023-10-16T12:12:01.113519+0000","last_clean":"2023-10-16T12:12:01.113519+0000","last_became_active":"2023-10-16T12:12:00.120862+0000","last_became_peered":"2023-10-16T12:12:00.120862+0000","last_unstale":"2023-10-16T12:12:01.113519+0000","last_undegraded":"2023-10-16T12:12:01.113519+0000","last_fullsized":"2023-10-16T12:12:01.113519+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_clean_scrub_stamp":"2023-10-16T12:11:59.089272+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:24:04.335776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.081867+0000","last_change":"2023-10-16T12:12:01.068095+0000","last_active":"2023-10-16T12:12:01.081867+0000","last_peered":"2023-10-16T12:12:01.081867+0000","last_clean":"2023-10-16T12:12:01.081867+0000","last_became_active":"2023-10-16T12:12:01.067752+0000","last_became_peered":"2023-10-16T12:12:01.067752+0000","last_unstale":"2023-10-16T12:12:01.081867+0000","last_undegraded":"2023-10-16T12:12:01.081867+0000","last_fullsized":"2023-10-16T12:12:01.081867+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:17:07.743741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"38'1","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.112258+0000","last_change":"2023-10-16T12:12:00.116133+0000","last_active":"2023-10-16T12:12:01.112258+0000","last_peered":"2023-10-16T12:12:01.112258+0000","last_clean":"2023-10-16T12:12:01.112258+0000","last_became_active":"2023-10-16T12:12:00.115780+0000","last_became_peered":"2023-10-16T12:12:00.115780+0000","last_unstale":"2023-10-16T12:12:01.112258+0000","last_undegraded":"2023-10-16T12:12:01.112258+0000","last_fullsized":"2023-10-16T12:12:01.112258+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_clean_scrub_stamp":"2023-10-16T12:11:59.089272+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:59:39.837329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.104758+0000","last_change":"2023-10-16T12:12:01.103464+0000","last_active":"2023-10-16T12:12:01.104758+0000","last_peered":"2023-10-16T12:12:01.104758+0000","last_clean":"2023-10-16T12:12:01.104758+0000","last_became_active":"2023-10-16T12:12:01.103208+0000","last_became_peered":"2023-10-16T12:12:01.103208+0000","last_unstale":"2023-10-16T12:12:01.104758+0000","last_undegraded":"2023-10-16T12:12:01.104758+0000","last_fullsized":"2023-10-16T12:12:01.104758+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:31:08.641239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"38'1","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.115055+0000","last_change":"2023-10-16T12:12:00.125114+0000","last_active":"2023-10-16T12:12:01.115055+0000","last_peered":"2023-10-16T12:12:01.115055+0000","last_clean":"2023-10-16T12:12:01.115055+0000","last_became_active":"2023-10-16T12:12:00.124415+0000","last_became_peered":"2023-10-16T12:12:00.124415+0000","last_unstale":"2023-10-16T12:12:01.115055+0000","last_undegraded":"2023-10-16T12:12:01.115055+0000","last_fullsized":"2023-10-16T12:12:01.115055+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_clean_scrub_stamp":"2023-10-16T12:11:59.089272+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:59:39.837329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.082006+0000","last_change":"2023-10-16T12:12:01.079973+0000","last_active":"2023-10-16T12:12:01.082006+0000","last_peered":"2023-10-16T12:12:01.082006+0000","last_clean":"2023-10-16T12:12:01.082006+0000","last_became_active":"2023-10-16T12:12:01.079701+0000","last_became_peered":"2023-10-16T12:12:01.079701+0000","last_unstale":"2023-10-16T12:12:01.082006+0000","last_undegraded":"2023-10-16T12:12:01.082006+0000","last_fullsized":"2023-10-16T12:12:01.082006+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:39:58.395936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T12:12:02.059187+0000","last_change":"2023-10-16T12:12:00.125087+0000","last_active":"2023-10-16T12:12:02.059187+0000","last_peered":"2023-10-16T12:12:02.059187+0000","last_clean":"2023-10-16T12:12:02.059187+0000","last_became_active":"2023-10-16T12:12:00.124806+0000","last_became_peered":"2023-10-16T12:12:00.124806+0000","last_unstale":"2023-10-16T12:12:02.059187+0000","last_undegraded":"2023-10-16T12:12:02.059187+0000","last_fullsized":"2023-10-16T12:12:02.059187+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_clean_scrub_stamp":"2023-10-16T12:11:59.089272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:31:07.635977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T12:12:02.060308+0000","last_change":"2023-10-16T12:12:01.077197+0000","last_active":"2023-10-16T12:12:02.060308+0000","last_peered":"2023-10-16T12:12:02.060308+0000","last_clean":"2023-10-16T12:12:02.060308+0000","last_became_active":"2023-10-16T12:12:01.076891+0000","last_became_peered":"2023-10-16T12:12:01.076891+0000","last_unstale":"2023-10-16T12:12:02.060308+0000","last_undegraded":"2023-10-16T12:12:02.060308+0000","last_fullsized":"2023-10-16T12:12:02.060308+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:39:38.834105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.063853+0000","last_change":"2023-10-16T12:12:00.124172+0000","last_active":"2023-10-16T12:12:01.063853+0000","last_peered":"2023-10-16T12:12:01.063853+0000","last_clean":"2023-10-16T12:12:01.063853+0000","last_became_active":"2023-10-16T12:12:00.123703+0000","last_became_peered":"2023-10-16T12:12:00.123703+0000","last_unstale":"2023-10-16T12:12:01.063853+0000","last_undegraded":"2023-10-16T12:12:01.063853+0000","last_fullsized":"2023-10-16T12:12:01.063853+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_clean_scrub_stamp":"2023-10-16T12:11:59.089272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:31:07.635977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.072443+0000","last_change":"2023-10-16T12:12:01.069201+0000","last_active":"2023-10-16T12:12:01.072443+0000","last_peered":"2023-10-16T12:12:01.072443+0000","last_clean":"2023-10-16T12:12:01.072443+0000","last_became_active":"2023-10-16T12:12:01.068512+0000","last_became_peered":"2023-10-16T12:12:01.068512+0000","last_unstale":"2023-10-16T12:12:01.072443+0000","last_undegraded":"2023-10-16T12:12:01.072443+0000","last_fullsized":"2023-10-16T12:12:01.072443+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:17:15.506280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T12:12:02.059705+0000","last_change":"2023-10-16T12:12:00.118028+0000","last_active":"2023-10-16T12:12:02.059705+0000","last_peered":"2023-10-16T12:12:02.059705+0000","last_clean":"2023-10-16T12:12:02.059705+0000","last_became_active":"2023-10-16T12:12:00.117729+0000","last_became_peered":"2023-10-16T12:12:00.117729+0000","last_unstale":"2023-10-16T12:12:02.059705+0000","last_undegraded":"2023-10-16T12:12:02.059705+0000","last_fullsized":"2023-10-16T12:12:02.059705+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_clean_scrub_stamp":"2023-10-16T12:11:59.089272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:17:49.299356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.069963+0000","last_change":"2023-10-16T12:12:01.067600+0000","last_active":"2023-10-16T12:12:01.069963+0000","last_peered":"2023-10-16T12:12:01.069963+0000","last_clean":"2023-10-16T12:12:01.069963+0000","last_became_active":"2023-10-16T12:12:01.067310+0000","last_became_peered":"2023-10-16T12:12:01.067310+0000","last_unstale":"2023-10-16T12:12:01.069963+0000","last_undegraded":"2023-10-16T12:12:01.069963+0000","last_fullsized":"2023-10-16T12:12:01.069963+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:36:12.640742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.068666+0000","last_change":"2023-10-16T12:12:00.120031+0000","last_active":"2023-10-16T12:12:01.068666+0000","last_peered":"2023-10-16T12:12:01.068666+0000","last_clean":"2023-10-16T12:12:01.068666+0000","last_became_active":"2023-10-16T12:12:00.119643+0000","last_became_peered":"2023-10-16T12:12:00.119643+0000","last_unstale":"2023-10-16T12:12:01.068666+0000","last_undegraded":"2023-10-16T12:12:01.068666+0000","last_fullsized":"2023-10-16T12:12:01.068666+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_clean_scrub_stamp":"2023-10-16T12:11:59.089272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:56:16.294667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.069072+0000","last_change":"2023-10-16T12:12:01.067631+0000","last_active":"2023-10-16T12:12:01.069072+0000","last_peered":"2023-10-16T12:12:01.069072+0000","last_clean":"2023-10-16T12:12:01.069072+0000","last_became_active":"2023-10-16T12:12:01.067340+0000","last_became_peered":"2023-10-16T12:12:01.067340+0000","last_unstale":"2023-10-16T12:12:01.069072+0000","last_undegraded":"2023-10-16T12:12:01.069072+0000","last_fullsized":"2023-10-16T12:12:01.069072+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:24:05.341039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"38'3","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.112712+0000","last_change":"2023-10-16T12:12:00.126200+0000","last_active":"2023-10-16T12:12:01.112712+0000","last_peered":"2023-10-16T12:12:01.112712+0000","last_clean":"2023-10-16T12:12:01.112712+0000","last_became_active":"2023-10-16T12:12:00.125890+0000","last_became_peered":"2023-10-16T12:12:00.125890+0000","last_unstale":"2023-10-16T12:12:01.112712+0000","last_undegraded":"2023-10-16T12:12:01.112712+0000","last_fullsized":"2023-10-16T12:12:01.112712+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_clean_scrub_stamp":"2023-10-16T12:11:59.089272+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:30:45.870992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.078853+0000","last_change":"2023-10-16T12:12:01.076502+0000","last_active":"2023-10-16T12:12:01.078853+0000","last_peered":"2023-10-16T12:12:01.078853+0000","last_clean":"2023-10-16T12:12:01.078853+0000","last_became_active":"2023-10-16T12:12:01.076122+0000","last_became_peered":"2023-10-16T12:12:01.076122+0000","last_unstale":"2023-10-16T12:12:01.078853+0000","last_undegraded":"2023-10-16T12:12:01.078853+0000","last_fullsized":"2023-10-16T12:12:01.078853+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:07:14.264264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"38'6","reported_seq":18,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T12:12:02.059919+0000","last_change":"2023-10-16T12:12:00.120746+0000","last_active":"2023-10-16T12:12:02.059919+0000","last_peered":"2023-10-16T12:12:02.059919+0000","last_clean":"2023-10-16T12:12:02.059919+0000","last_became_active":"2023-10-16T12:12:00.119566+0000","last_became_peered":"2023-10-16T12:12:00.119566+0000","last_unstale":"2023-10-16T12:12:02.059919+0000","last_undegraded":"2023-10-16T12:12:02.059919+0000","last_fullsized":"2023-10-16T12:12:02.059919+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_clean_scrub_stamp":"2023-10-16T12:11:59.089272+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:43:24.590290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.084708+0000","last_change":"2023-10-16T12:12:01.082940+0000","last_active":"2023-10-16T12:12:01.084708+0000","last_peered":"2023-10-16T12:12:01.084708+0000","last_clean":"2023-10-16T12:12:01.084708+0000","last_became_active":"2023-10-16T12:12:01.082461+0000","last_became_peered":"2023-10-16T12:12:01.082461+0000","last_unstale":"2023-10-16T12:12:01.084708+0000","last_undegraded":"2023-10-16T12:12:01.084708+0000","last_fullsized":"2023-10-16T12:12:01.084708+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:31:08.641239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"38'2","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T12:12:02.060335+0000","last_change":"2023-10-16T12:12:00.119080+0000","last_active":"2023-10-16T12:12:02.060335+0000","last_peered":"2023-10-16T12:12:02.060335+0000","last_clean":"2023-10-16T12:12:02.060335+0000","last_became_active":"2023-10-16T12:12:00.118762+0000","last_became_peered":"2023-10-16T12:12:00.118762+0000","last_unstale":"2023-10-16T12:12:02.060335+0000","last_undegraded":"2023-10-16T12:12:02.060335+0000","last_fullsized":"2023-10-16T12:12:02.060335+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_clean_scrub_stamp":"2023-10-16T12:11:59.089272+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:58:57.536737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T12:12:02.059616+0000","last_change":"2023-10-16T12:12:01.084845+0000","last_active":"2023-10-16T12:12:02.059616+0000","last_peered":"2023-10-16T12:12:02.059616+0000","last_clean":"2023-10-16T12:12:02.059616+0000","last_became_active":"2023-10-16T12:12:01.084486+0000","last_became_peered":"2023-10-16T12:12:01.084486+0000","last_unstale":"2023-10-16T12:12:02.059616+0000","last_undegraded":"2023-10-16T12:12:02.059616+0000","last_fullsized":"2023-10-16T12:12:02.059616+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:06:55.894696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T12:12:02.060027+0000","last_change":"2023-10-16T12:12:00.117820+0000","last_active":"2023-10-16T12:12:02.060027+0000","last_peered":"2023-10-16T12:12:02.060027+0000","last_clean":"2023-10-16T12:12:02.060027+0000","last_became_active":"2023-10-16T12:12:00.117588+0000","last_became_peered":"2023-10-16T12:12:00.117588+0000","last_unstale":"2023-10-16T12:12:02.060027+0000","last_undegraded":"2023-10-16T12:12:02.060027+0000","last_fullsized":"2023-10-16T12:12:02.060027+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_clean_scrub_stamp":"2023-10-16T12:11:59.089272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:39:57.390674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.075347+0000","last_change":"2023-10-16T12:12:01.069077+0000","last_active":"2023-10-16T12:12:01.075347+0000","last_peered":"2023-10-16T12:12:01.075347+0000","last_clean":"2023-10-16T12:12:01.075347+0000","last_became_active":"2023-10-16T12:12:01.068766+0000","last_became_peered":"2023-10-16T12:12:01.068766+0000","last_unstale":"2023-10-16T12:12:01.075347+0000","last_undegraded":"2023-10-16T12:12:01.075347+0000","last_fullsized":"2023-10-16T12:12:01.075347+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:53:52.073154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.059512+0000","last_change":"2023-10-16T12:12:00.126570+0000","last_active":"2023-10-16T12:12:01.059512+0000","last_peered":"2023-10-16T12:12:01.059512+0000","last_clean":"2023-10-16T12:12:01.059512+0000","last_became_active":"2023-10-16T12:12:00.126279+0000","last_became_peered":"2023-10-16T12:12:00.126279+0000","last_unstale":"2023-10-16T12:12:01.059512+0000","last_undegraded":"2023-10-16T12:12:01.059512+0000","last_fullsized":"2023-10-16T12:12:01.059512+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_clean_scrub_stamp":"2023-10-16T12:11:59.089272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:55:02.904354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T12:12:02.060156+0000","last_change":"2023-10-16T12:12:01.084851+0000","last_active":"2023-10-16T12:12:02.060156+0000","last_peered":"2023-10-16T12:12:02.060156+0000","last_clean":"2023-10-16T12:12:02.060156+0000","last_became_active":"2023-10-16T12:12:01.084506+0000","last_became_peered":"2023-10-16T12:12:01.084506+0000","last_unstale":"2023-10-16T12:12:02.060156+0000","last_undegraded":"2023-10-16T12:12:02.060156+0000","last_fullsized":"2023-10-16T12:12:02.060156+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:21:25.584112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"38'8","reported_seq":19,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.116033+0000","last_change":"2023-10-16T12:12:00.125281+0000","last_active":"2023-10-16T12:12:01.116033+0000","last_peered":"2023-10-16T12:12:01.116033+0000","last_clean":"2023-10-16T12:12:01.116033+0000","last_became_active":"2023-10-16T12:12:00.124960+0000","last_became_peered":"2023-10-16T12:12:00.124960+0000","last_unstale":"2023-10-16T12:12:01.116033+0000","last_undegraded":"2023-10-16T12:12:01.116033+0000","last_fullsized":"2023-10-16T12:12:01.116033+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_clean_scrub_stamp":"2023-10-16T12:11:59.089272+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:37:57.399730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T12:12:02.059751+0000","last_change":"2023-10-16T12:12:01.073634+0000","last_active":"2023-10-16T12:12:02.059751+0000","last_peered":"2023-10-16T12:12:02.059751+0000","last_clean":"2023-10-16T12:12:02.059751+0000","last_became_active":"2023-10-16T12:12:01.073224+0000","last_became_peered":"2023-10-16T12:12:01.073224+0000","last_unstale":"2023-10-16T12:12:02.059751+0000","last_undegraded":"2023-10-16T12:12:02.059751+0000","last_fullsized":"2023-10-16T12:12:02.059751+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:10:18.638711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"38'1","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.114900+0000","last_change":"2023-10-16T12:12:00.125068+0000","last_active":"2023-10-16T12:12:01.114900+0000","last_peered":"2023-10-16T12:12:01.114900+0000","last_clean":"2023-10-16T12:12:01.114900+0000","last_became_active":"2023-10-16T12:12:00.124767+0000","last_became_peered":"2023-10-16T12:12:00.124767+0000","last_unstale":"2023-10-16T12:12:01.114900+0000","last_undegraded":"2023-10-16T12:12:01.114900+0000","last_fullsized":"2023-10-16T12:12:01.114900+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_clean_scrub_stamp":"2023-10-16T12:11:59.089272+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:49:59.185665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.104779+0000","last_change":"2023-10-16T12:12:01.103617+0000","last_active":"2023-10-16T12:12:01.104779+0000","last_peered":"2023-10-16T12:12:01.104779+0000","last_clean":"2023-10-16T12:12:01.104779+0000","last_became_active":"2023-10-16T12:12:01.103320+0000","last_became_peered":"2023-10-16T12:12:01.103320+0000","last_unstale":"2023-10-16T12:12:01.104779+0000","last_undegraded":"2023-10-16T12:12:01.104779+0000","last_fullsized":"2023-10-16T12:12:01.104779+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:37:58.404993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"38'2","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.115534+0000","last_change":"2023-10-16T12:12:00.124060+0000","last_active":"2023-10-16T12:12:01.115534+0000","last_peered":"2023-10-16T12:12:01.115534+0000","last_clean":"2023-10-16T12:12:01.115534+0000","last_became_active":"2023-10-16T12:12:00.123767+0000","last_became_peered":"2023-10-16T12:12:00.123767+0000","last_unstale":"2023-10-16T12:12:01.115534+0000","last_undegraded":"2023-10-16T12:12:01.115534+0000","last_fullsized":"2023-10-16T12:12:01.115534+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_clean_scrub_stamp":"2023-10-16T12:11:59.089272+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:59:39.837329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.078801+0000","last_change":"2023-10-16T12:12:01.074798+0000","last_active":"2023-10-16T12:12:01.078801+0000","last_peered":"2023-10-16T12:12:01.078801+0000","last_clean":"2023-10-16T12:12:01.078801+0000","last_became_active":"2023-10-16T12:12:01.074429+0000","last_became_peered":"2023-10-16T12:12:01.074429+0000","last_unstale":"2023-10-16T12:12:01.078801+0000","last_undegraded":"2023-10-16T12:12:01.078801+0000","last_fullsized":"2023-10-16T12:12:01.078801+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:57:03.304620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"38'1","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.113152+0000","last_change":"2023-10-16T12:12:00.122920+0000","last_active":"2023-10-16T12:12:01.113152+0000","last_peered":"2023-10-16T12:12:01.113152+0000","last_clean":"2023-10-16T12:12:01.113152+0000","last_became_active":"2023-10-16T12:12:00.122628+0000","last_became_peered":"2023-10-16T12:12:00.122628+0000","last_unstale":"2023-10-16T12:12:01.113152+0000","last_undegraded":"2023-10-16T12:12:01.113152+0000","last_fullsized":"2023-10-16T12:12:01.113152+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_clean_scrub_stamp":"2023-10-16T12:11:59.089272+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:05:22.800311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.075322+0000","last_change":"2023-10-16T12:12:01.074661+0000","last_active":"2023-10-16T12:12:01.075322+0000","last_peered":"2023-10-16T12:12:01.075322+0000","last_clean":"2023-10-16T12:12:01.075322+0000","last_became_active":"2023-10-16T12:12:01.074319+0000","last_became_peered":"2023-10-16T12:12:01.074319+0000","last_unstale":"2023-10-16T12:12:01.075322+0000","last_undegraded":"2023-10-16T12:12:01.075322+0000","last_fullsized":"2023-10-16T12:12:01.075322+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:41:53.032282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T12:12:02.059475+0000","last_change":"2023-10-16T12:12:00.124350+0000","last_active":"2023-10-16T12:12:02.059475+0000","last_peered":"2023-10-16T12:12:02.059475+0000","last_clean":"2023-10-16T12:12:02.059475+0000","last_became_active":"2023-10-16T12:12:00.123989+0000","last_became_peered":"2023-10-16T12:12:00.123989+0000","last_unstale":"2023-10-16T12:12:02.059475+0000","last_undegraded":"2023-10-16T12:12:02.059475+0000","last_fullsized":"2023-10-16T12:12:02.059475+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_clean_scrub_stamp":"2023-10-16T12:11:59.089272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:37:57.399730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.075959+0000","last_change":"2023-10-16T12:12:01.071010+0000","last_active":"2023-10-16T12:12:01.075959+0000","last_peered":"2023-10-16T12:12:01.075959+0000","last_clean":"2023-10-16T12:12:01.075959+0000","last_became_active":"2023-10-16T12:12:01.070742+0000","last_became_peered":"2023-10-16T12:12:01.070742+0000","last_unstale":"2023-10-16T12:12:01.075959+0000","last_undegraded":"2023-10-16T12:12:01.075959+0000","last_fullsized":"2023-10-16T12:12:01.075959+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:56:57.470920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"38'2","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.113064+0000","last_change":"2023-10-16T12:12:00.122643+0000","last_active":"2023-10-16T12:12:01.113064+0000","last_peered":"2023-10-16T12:12:01.113064+0000","last_clean":"2023-10-16T12:12:01.113064+0000","last_became_active":"2023-10-16T12:12:00.122364+0000","last_became_peered":"2023-10-16T12:12:00.122364+0000","last_unstale":"2023-10-16T12:12:01.113064+0000","last_undegraded":"2023-10-16T12:12:01.113064+0000","last_fullsized":"2023-10-16T12:12:01.113064+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_clean_scrub_stamp":"2023-10-16T12:11:59.089272+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:37:15.293764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.084123+0000","last_change":"2023-10-16T12:12:01.082270+0000","last_active":"2023-10-16T12:12:01.084123+0000","last_peered":"2023-10-16T12:12:01.084123+0000","last_clean":"2023-10-16T12:12:01.084123+0000","last_became_active":"2023-10-16T12:12:01.081910+0000","last_became_peered":"2023-10-16T12:12:01.081910+0000","last_unstale":"2023-10-16T12:12:01.084123+0000","last_undegraded":"2023-10-16T12:12:01.084123+0000","last_fullsized":"2023-10-16T12:12:01.084123+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:07:14.264264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.057632+0000","last_change":"2023-10-16T12:12:00.121951+0000","last_active":"2023-10-16T12:12:01.057632+0000","last_peered":"2023-10-16T12:12:01.057632+0000","last_clean":"2023-10-16T12:12:01.057632+0000","last_became_active":"2023-10-16T12:12:00.121686+0000","last_became_peered":"2023-10-16T12:12:00.121686+0000","last_unstale":"2023-10-16T12:12:01.057632+0000","last_undegraded":"2023-10-16T12:12:01.057632+0000","last_fullsized":"2023-10-16T12:12:01.057632+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_clean_scrub_stamp":"2023-10-16T12:11:59.089272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:30:17.984189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.081932+0000","last_change":"2023-10-16T12:12:01.078333+0000","last_active":"2023-10-16T12:12:01.081932+0000","last_peered":"2023-10-16T12:12:01.081932+0000","last_clean":"2023-10-16T12:12:01.081932+0000","last_became_active":"2023-10-16T12:12:01.077928+0000","last_became_peered":"2023-10-16T12:12:01.077928+0000","last_unstale":"2023-10-16T12:12:01.081932+0000","last_undegraded":"2023-10-16T12:12:01.081932+0000","last_fullsized":"2023-10-16T12:12:01.081932+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:13:01.102564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.065675+0000","last_change":"2023-10-16T12:12:00.121496+0000","last_active":"2023-10-16T12:12:01.065675+0000","last_peered":"2023-10-16T12:12:01.065675+0000","last_clean":"2023-10-16T12:12:01.065675+0000","last_became_active":"2023-10-16T12:12:00.121194+0000","last_became_peered":"2023-10-16T12:12:00.121194+0000","last_unstale":"2023-10-16T12:12:01.065675+0000","last_undegraded":"2023-10-16T12:12:01.065675+0000","last_fullsized":"2023-10-16T12:12:01.065675+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_clean_scrub_stamp":"2023-10-16T12:11:59.089272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:56:02.075484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T12:12:02.060229+0000","last_change":"2023-10-16T12:12:01.077074+0000","last_active":"2023-10-16T12:12:02.060229+0000","last_peered":"2023-10-16T12:12:02.060229+0000","last_clean":"2023-10-16T12:12:02.060229+0000","last_became_active":"2023-10-16T12:12:01.076578+0000","last_became_peered":"2023-10-16T12:12:01.076578+0000","last_unstale":"2023-10-16T12:12:02.060229+0000","last_undegraded":"2023-10-16T12:12:02.060229+0000","last_fullsized":"2023-10-16T12:12:02.060229+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:44:26.822860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.057966+0000","last_change":"2023-10-16T12:12:00.114664+0000","last_active":"2023-10-16T12:12:01.057966+0000","last_peered":"2023-10-16T12:12:01.057966+0000","last_clean":"2023-10-16T12:12:01.057966+0000","last_became_active":"2023-10-16T12:12:00.114399+0000","last_became_peered":"2023-10-16T12:12:00.114399+0000","last_unstale":"2023-10-16T12:12:01.057966+0000","last_undegraded":"2023-10-16T12:12:01.057966+0000","last_fullsized":"2023-10-16T12:12:01.057966+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_clean_scrub_stamp":"2023-10-16T12:11:59.089272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:03:57.508990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T12:12:02.059380+0000","last_change":"2023-10-16T12:12:01.101889+0000","last_active":"2023-10-16T12:12:02.059380+0000","last_peered":"2023-10-16T12:12:02.059380+0000","last_clean":"2023-10-16T12:12:02.059380+0000","last_became_active":"2023-10-16T12:12:01.101477+0000","last_became_peered":"2023-10-16T12:12:01.101477+0000","last_unstale":"2023-10-16T12:12:02.059380+0000","last_undegraded":"2023-10-16T12:12:02.059380+0000","last_fullsized":"2023-10-16T12:12:02.059380+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:56:17.299930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"38'2","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.112812+0000","last_change":"2023-10-16T12:12:00.126072+0000","last_active":"2023-10-16T12:12:01.112812+0000","last_peered":"2023-10-16T12:12:01.112812+0000","last_clean":"2023-10-16T12:12:01.112812+0000","last_became_active":"2023-10-16T12:12:00.125763+0000","last_became_peered":"2023-10-16T12:12:00.125763+0000","last_unstale":"2023-10-16T12:12:01.112812+0000","last_undegraded":"2023-10-16T12:12:01.112812+0000","last_fullsized":"2023-10-16T12:12:01.112812+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_clean_scrub_stamp":"2023-10-16T12:11:59.089272+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:58:32.014737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.077731+0000","last_change":"2023-10-16T12:12:01.069504+0000","last_active":"2023-10-16T12:12:01.077731+0000","last_peered":"2023-10-16T12:12:01.077731+0000","last_clean":"2023-10-16T12:12:01.077731+0000","last_became_active":"2023-10-16T12:12:01.069200+0000","last_became_peered":"2023-10-16T12:12:01.069200+0000","last_unstale":"2023-10-16T12:12:01.077731+0000","last_undegraded":"2023-10-16T12:12:01.077731+0000","last_fullsized":"2023-10-16T12:12:01.077731+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:56:17.299930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.057731+0000","last_change":"2023-10-16T12:12:00.112373+0000","last_active":"2023-10-16T12:12:01.057731+0000","last_peered":"2023-10-16T12:12:01.057731+0000","last_clean":"2023-10-16T12:12:01.057731+0000","last_became_active":"2023-10-16T12:12:00.112045+0000","last_became_peered":"2023-10-16T12:12:00.112045+0000","last_unstale":"2023-10-16T12:12:01.057731+0000","last_undegraded":"2023-10-16T12:12:01.057731+0000","last_fullsized":"2023-10-16T12:12:01.057731+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_clean_scrub_stamp":"2023-10-16T12:11:59.089272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:42:40.590200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T12:12:02.059859+0000","last_change":"2023-10-16T12:12:01.079305+0000","last_active":"2023-10-16T12:12:02.059859+0000","last_peered":"2023-10-16T12:12:02.059859+0000","last_clean":"2023-10-16T12:12:02.059859+0000","last_became_active":"2023-10-16T12:12:01.079092+0000","last_became_peered":"2023-10-16T12:12:01.079092+0000","last_unstale":"2023-10-16T12:12:02.059859+0000","last_undegraded":"2023-10-16T12:12:02.059859+0000","last_fullsized":"2023-10-16T12:12:02.059859+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:15:26.287918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.063493+0000","last_change":"2023-10-16T12:12:00.125702+0000","last_active":"2023-10-16T12:12:01.063493+0000","last_peered":"2023-10-16T12:12:01.063493+0000","last_clean":"2023-10-16T12:12:01.063493+0000","last_became_active":"2023-10-16T12:12:00.125419+0000","last_became_peered":"2023-10-16T12:12:00.125419+0000","last_unstale":"2023-10-16T12:12:01.063493+0000","last_undegraded":"2023-10-16T12:12:01.063493+0000","last_fullsized":"2023-10-16T12:12:01.063493+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_clean_scrub_stamp":"2023-10-16T12:11:59.089272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:39:57.390674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.084208+0000","last_change":"2023-10-16T12:12:01.082772+0000","last_active":"2023-10-16T12:12:01.084208+0000","last_peered":"2023-10-16T12:12:01.084208+0000","last_clean":"2023-10-16T12:12:01.084208+0000","last_became_active":"2023-10-16T12:12:01.082034+0000","last_became_peered":"2023-10-16T12:12:01.082034+0000","last_unstale":"2023-10-16T12:12:01.084208+0000","last_undegraded":"2023-10-16T12:12:01.084208+0000","last_fullsized":"2023-10-16T12:12:01.084208+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:57:03.304620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.063663+0000","last_change":"2023-10-16T12:12:00.115273+0000","last_active":"2023-10-16T12:12:01.063663+0000","last_peered":"2023-10-16T12:12:01.063663+0000","last_clean":"2023-10-16T12:12:01.063663+0000","last_became_active":"2023-10-16T12:12:00.114991+0000","last_became_peered":"2023-10-16T12:12:00.114991+0000","last_unstale":"2023-10-16T12:12:01.063663+0000","last_undegraded":"2023-10-16T12:12:01.063663+0000","last_fullsized":"2023-10-16T12:12:01.063663+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_clean_scrub_stamp":"2023-10-16T12:11:59.089272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:59:39.837329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T12:12:02.059176+0000","last_change":"2023-10-16T12:12:01.064260+0000","last_active":"2023-10-16T12:12:02.059176+0000","last_peered":"2023-10-16T12:12:02.059176+0000","last_clean":"2023-10-16T12:12:02.059176+0000","last_became_active":"2023-10-16T12:12:01.063980+0000","last_became_peered":"2023-10-16T12:12:01.063980+0000","last_unstale":"2023-10-16T12:12:02.059176+0000","last_undegraded":"2023-10-16T12:12:02.059176+0000","last_fullsized":"2023-10-16T12:12:02.059176+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:48:55.890617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.060214+0000","last_change":"2023-10-16T12:12:00.116949+0000","last_active":"2023-10-16T12:12:01.060214+0000","last_peered":"2023-10-16T12:12:01.060214+0000","last_clean":"2023-10-16T12:12:01.060214+0000","last_became_active":"2023-10-16T12:12:00.116619+0000","last_became_peered":"2023-10-16T12:12:00.116619+0000","last_unstale":"2023-10-16T12:12:01.060214+0000","last_undegraded":"2023-10-16T12:12:01.060214+0000","last_fullsized":"2023-10-16T12:12:01.060214+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_clean_scrub_stamp":"2023-10-16T12:11:59.089272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:24:04.335776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.078742+0000","last_change":"2023-10-16T12:12:01.074865+0000","last_active":"2023-10-16T12:12:01.078742+0000","last_peered":"2023-10-16T12:12:01.078742+0000","last_clean":"2023-10-16T12:12:01.078742+0000","last_became_active":"2023-10-16T12:12:01.074296+0000","last_became_peered":"2023-10-16T12:12:01.074296+0000","last_unstale":"2023-10-16T12:12:01.078742+0000","last_undegraded":"2023-10-16T12:12:01.078742+0000","last_fullsized":"2023-10-16T12:12:01.078742+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:17:07.743741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.063727+0000","last_change":"2023-10-16T12:12:00.121241+0000","last_active":"2023-10-16T12:12:01.063727+0000","last_peered":"2023-10-16T12:12:01.063727+0000","last_clean":"2023-10-16T12:12:01.063727+0000","last_became_active":"2023-10-16T12:12:00.120916+0000","last_became_peered":"2023-10-16T12:12:00.120916+0000","last_unstale":"2023-10-16T12:12:01.063727+0000","last_undegraded":"2023-10-16T12:12:01.063727+0000","last_fullsized":"2023-10-16T12:12:01.063727+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_clean_scrub_stamp":"2023-10-16T12:11:59.089272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:36:11.635480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.072309+0000","last_change":"2023-10-16T12:12:01.068857+0000","last_active":"2023-10-16T12:12:01.072309+0000","last_peered":"2023-10-16T12:12:01.072309+0000","last_clean":"2023-10-16T12:12:01.072309+0000","last_became_active":"2023-10-16T12:12:01.068538+0000","last_became_peered":"2023-10-16T12:12:01.068538+0000","last_unstale":"2023-10-16T12:12:01.072309+0000","last_undegraded":"2023-10-16T12:12:01.072309+0000","last_fullsized":"2023-10-16T12:12:01.072309+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:31:08.641239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-16T12:12:02.059338+0000","last_change":"2023-10-16T12:12:00.117949+0000","last_active":"2023-10-16T12:12:02.059338+0000","last_peered":"2023-10-16T12:12:02.059338+0000","last_clean":"2023-10-16T12:12:02.059338+0000","last_became_active":"2023-10-16T12:12:00.117507+0000","last_became_peered":"2023-10-16T12:12:00.117507+0000","last_unstale":"2023-10-16T12:12:02.059338+0000","last_undegraded":"2023-10-16T12:12:02.059338+0000","last_fullsized":"2023-10-16T12:12:02.059338+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_clean_scrub_stamp":"2023-10-16T12:11:59.089272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:13:00.097302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.082311+0000","last_change":"2023-10-16T12:12:01.079605+0000","last_active":"2023-10-16T12:12:01.082311+0000","last_peered":"2023-10-16T12:12:01.082311+0000","last_clean":"2023-10-16T12:12:01.082311+0000","last_became_active":"2023-10-16T12:12:01.079293+0000","last_became_peered":"2023-10-16T12:12:01.079293+0000","last_unstale":"2023-10-16T12:12:01.082311+0000","last_undegraded":"2023-10-16T12:12:01.082311+0000","last_fullsized":"2023-10-16T12:12:01.082311+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:12:00.094535+0000","last_clean_scrub_stamp":"2023-10-16T12:12:00.094535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:56:03.080747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-16T12:12:01.063834+0000","last_change":"2023-10-16T12:12:00.122781+0000","last_active":"2023-10-16T12:12:01.063834+0000","last_peered":"2023-10-16T12:12:01.063834+0000","last_clean":"2023-10-16T12:12:01.063834+0000","last_became_active":"2023-10-16T12:12:00.122521+0000","last_became_peered":"2023-10-16T12:12:00.122521+0000","last_unstale":"2023-10-16T12:12:01.063834+0000","last_undegraded":"2023-10-16T12:12:01.063834+0000","last_fullsized":"2023-10-16T12:12:01.063834+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:11:59.089272+0000","last_clean_scrub_stamp":"2023-10-16T12:11:59.089272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:48:54.885355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":179,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-16T12:12:00.111128+0000","last_change":"2023-10-16T12:10:34.746820+0000","last_active":"2023-10-16T12:12:00.111128+0000","last_peered":"2023-10-16T12:12:00.111128+0000","last_clean":"2023-10-16T12:12:00.111128+0000","last_became_active":"2023-10-16T12:10:34.745574+0000","last_became_peered":"2023-10-16T12:10:34.745574+0000","last_unstale":"2023-10-16T12:12:00.111128+0000","last_undegraded":"2023-10-16T12:12:00.111128+0000","last_fullsized":"2023-10-16T12:12:00.111128+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_clean_scrub_stamp":"2023-10-16T12:10:33.577111+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:14.461986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574871,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28044,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343796,"statfs":{"total":96636764160,"available":96608047104,"internally_reserved":0,"allocated":1048576,"data_stored":779269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:11:34 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.318}]},{"osd":1,"last update":"Mon Oct 16 12:11:34 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.44400000000000001}]},{"osd":2,"last update":"Mon Oct 16 12:11:34 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.52200000000000002}]},{"osd":3,"last update":"Mon Oct 16 12:11:34 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.50600000000000001}]},{"osd":4,"last update":"Mon Oct 16 12:11:34 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.60099999999999998}]},{"osd":5,"last update":"Mon Oct 16 12:11:34 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.65200000000000002}]},{"osd":6,"last update":"Mon Oct 16 12:11:34 2023","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.628}]}]},{"osd":6,"up_from":13,"seq":55834574869,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27452,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344388,"statfs":{"total":96636764160,"available":96608653312,"internally_reserved":0,"allocated":442368,"data_stored":187801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:11:36 2023","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":5.9349999999999996,"5min":5.9349999999999996,"15min":5.9349999999999996},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":5.9489999999999998,"5min":5.9489999999999998,"15min":5.9489999999999998},"last":0.76700000000000002}]},{"osd":1,"last update":"Mon Oct 16 12:11:36 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.69299999999999995}]},{"osd":2,"last update":"Mon Oct 16 12:11:36 2023","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":8.0489999999999995,"5min":8.0489999999999995,"15min":8.0489999999999995},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":2.855,"5min":2.855,"15min":2.855},"last":0.80800000000000005}]},{"osd":3,"last update":"Mon Oct 16 12:11:36 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.78600000000000003}]},{"osd":4,"last update":"Mon Oct 16 12:11:36 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.66500000000000004}]},{"osd":5,"last update":"Mon Oct 16 12:11:36 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.58999999999999997}]},{"osd":7,"last update":"Mon Oct 16 12:11:36 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.61899999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574869,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27452,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344388,"statfs":{"total":96636764160,"available":96608653312,"internally_reserved":0,"allocated":442368,"data_stored":187309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:11:32 2023","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.37,"15min":0.37},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.318}]},{"osd":2,"last update":"Mon Oct 16 12:11:32 2023","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.253}]},{"osd":3,"last update":"Mon Oct 16 12:11:32 2023","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"last":0.23699999999999999},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.28299999999999997}]},{"osd":4,"last update":"Mon Oct 16 12:11:32 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.55100000000000005}]},{"osd":5,"last update":"Mon Oct 16 12:11:32 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.48699999999999999}]},{"osd":6,"last update":"Mon Oct 16 12:11:32 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.44500000000000001}]},{"osd":7,"last update":"Mon Oct 16 12:11:32 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.58899999999999997}]}]},{"osd":0,"up_from":13,"seq":55834574870,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28032,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343808,"statfs":{"total":96636764160,"available":96608059392,"internally_reserved":0,"allocated":1036288,"data_stored":777621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 12:11:33 2023","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.253},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"last":0.28899999999999998}]},{"osd":2,"last update":"Mon Oct 16 12:11:33 2023","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.32700000000000001}]},{"osd":3,"last update":"Mon Oct 16 12:11:33 2023","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.378,"15min":0.378},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.36899999999999999}]},{"osd":4,"last update":"Mon Oct 16 12:11:33 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.68000000000000005}]},{"osd":5,"last update":"Mon Oct 16 12:11:33 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.54800000000000004}]},{"osd":6,"last update":"Mon Oct 16 12:11:33 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.53000000000000003}]},{"osd":7,"last update":"Mon Oct 16 12:11:33 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":6.7300000000000004,"5min":6.7300000000000004,"15min":6.7300000000000004},"last":0.65900000000000003}]}]},{"osd":2,"up_from":13,"seq":55834574870,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27452,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344388,"statfs":{"total":96636764160,"available":96608653312,"internally_reserved":0,"allocated":442368,"data_stored":187289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:11:35 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"last":0.35199999999999998}]},{"osd":1,"last update":"Mon Oct 16 12:11:35 2023","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.378,"15min":0.378},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.312},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.33700000000000002}]},{"osd":3,"last update":"Mon Oct 16 12:11:35 2023","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.442,"5min":0.442,"15min":0.442},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.38500000000000001}]},{"osd":4,"last update":"Mon Oct 16 12:11:35 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.79900000000000004}]},{"osd":5,"last update":"Mon Oct 16 12:11:35 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.56000000000000005}]},{"osd":6,"last update":"Mon Oct 16 12:11:35 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.78200000000000003}]},{"osd":7,"last update":"Mon Oct 16 12:11:35 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.82199999999999995}]}]},{"osd":3,"up_from":13,"seq":55834574869,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27464,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344376,"statfs":{"total":96636764160,"available":96608641024,"internally_reserved":0,"allocated":454656,"data_stored":188865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:11:36 2023","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.36499999999999999}]},{"osd":1,"last update":"Mon Oct 16 12:11:36 2023","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"last":0.309}]},{"osd":2,"last update":"Mon Oct 16 12:11:36 2023","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.34899999999999998}]},{"osd":4,"last update":"Mon Oct 16 12:11:36 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.52600000000000002}]},{"osd":5,"last update":"Mon Oct 16 12:11:36 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.502},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.41399999999999998}]},{"osd":6,"last update":"Mon Oct 16 12:11:36 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.42699999999999999}]},{"osd":7,"last update":"Mon Oct 16 12:11:36 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.44},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.57899999999999996}]}]},{"osd":4,"up_from":13,"seq":55834574868,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27456,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344384,"statfs":{"total":96636764160,"available":96608649216,"internally_reserved":0,"allocated":446464,"data_stored":187823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:11:33 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.171,"5min":1.171,"15min":1.171},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.63900000000000001}]},{"osd":1,"last update":"Mon Oct 16 12:11:33 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.433},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.67600000000000005}]},{"osd":2,"last update":"Mon Oct 16 12:11:33 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.73299999999999998}]},{"osd":3,"last update":"Mon Oct 16 12:11:33 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.60999999999999999}]},{"osd":5,"last update":"Mon Oct 16 12:11:33 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.70399999999999996}]},{"osd":6,"last update":"Mon Oct 16 12:11:33 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.79100000000000004}]},{"osd":7,"last update":"Mon Oct 16 12:11:33 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.747},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.71999999999999997}]}]},{"osd":5,"up_from":13,"seq":55834574869,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27456,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344384,"statfs":{"total":96636764160,"available":96608649216,"internally_reserved":0,"allocated":446464,"data_stored":187331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:11:34 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.41699999999999998}]},{"osd":1,"last update":"Mon Oct 16 12:11:34 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.59699999999999998}]},{"osd":2,"last update":"Mon Oct 16 12:11:34 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.49099999999999999}]},{"osd":3,"last update":"Mon Oct 16 12:11:34 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.69499999999999995}]},{"osd":4,"last update":"Mon Oct 16 12:11:34 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.57499999999999996}]},{"osd":6,"last update":"Mon Oct 16 12:11:34 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.63}]},{"osd":7,"last update":"Mon Oct 16 12:11:34 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.74399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T12:12:05.672 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-16T12:12:05.672 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-16T12:12:05.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.837+0000 7f5b6b526700 1 Processor -- start 2023-10-16T12:12:05.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.837+0000 7f5b6b526700 1 -- start start 2023-10-16T12:12:05.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.838+0000 7f5b6b526700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5b6415a640 0x7f5b640cb710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:05.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.838+0000 7f5b6b526700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f5b640c6770 0x7f5b640cbc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:05.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.838+0000 7f5b6b526700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5b6415e140 0x7f5b640ca1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:05.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.838+0000 7f5b6b526700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b6415aa20 con 0x7f5b6415a640 2023-10-16T12:12:05.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.838+0000 7f5b6b526700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b6406eb40 con 0x7f5b640c6770 2023-10-16T12:12:05.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.838+0000 7f5b6b526700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b6406ee20 con 0x7f5b6415e140 2023-10-16T12:12:05.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.838+0000 7f5b692c2700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5b6415a640 0x7f5b640cb710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:05.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.838+0000 7f5b692c2700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5b6415a640 0x7f5b640cb710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35662/0 (socket says 172.21.15.16:35662) 2023-10-16T12:12:05.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.838+0000 7f5b692c2700 1 -- 172.21.15.16:0/3494533752 learned_addr learned my addr 172.21.15.16:0/3494533752 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:12:05.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.838+0000 7f5b69ac3700 1 --2- 172.21.15.16:0/3494533752 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5b6415e140 0x7f5b640ca1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:05.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.838+0000 7f5b68ac1700 1 --2- 172.21.15.16:0/3494533752 >> v2:172.21.15.49:3300/0 conn(0x7f5b640c6770 0x7f5b640cbc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:05.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.838+0000 7f5b692c2700 1 -- 172.21.15.16:0/3494533752 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5b6415e140 msgr2=0x7f5b640ca1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:05.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.838+0000 7f5b692c2700 1 --2- 172.21.15.16:0/3494533752 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5b6415e140 0x7f5b640ca1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:05.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.838+0000 7f5b692c2700 1 -- 172.21.15.16:0/3494533752 >> v2:172.21.15.49:3300/0 conn(0x7f5b640c6770 msgr2=0x7f5b640cbc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:05.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.838+0000 7f5b692c2700 1 --2- 172.21.15.16:0/3494533752 >> v2:172.21.15.49:3300/0 conn(0x7f5b640c6770 0x7f5b640cbc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:05.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.838+0000 7f5b692c2700 1 -- 172.21.15.16:0/3494533752 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b6406f100 con 0x7f5b6415a640 2023-10-16T12:12:05.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.838+0000 7f5b68ac1700 1 --2- 172.21.15.16:0/3494533752 >> v2:172.21.15.49:3300/0 conn(0x7f5b640c6770 0x7f5b640cbc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:12:05.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.838+0000 7f5b692c2700 1 --2- 172.21.15.16:0/3494533752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b6415a640 0x7f5b640cb710 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f5b640b5cd0 tx=0x7f5b60009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59875f4826e506e4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:05.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.839+0000 7f5b5bfff700 1 -- 172.21.15.16:0/3494533752 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b60011830 con 0x7f5b6415a640 2023-10-16T12:12:05.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.839+0000 7f5b5bfff700 1 -- 172.21.15.16:0/3494533752 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b60011990 con 0x7f5b6415a640 2023-10-16T12:12:05.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.839+0000 7f5b5bfff700 1 -- 172.21.15.16:0/3494533752 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b60011d20 con 0x7f5b6415a640 2023-10-16T12:12:05.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.839+0000 7f5b6b526700 1 -- 172.21.15.16:0/3494533752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b6415a640 msgr2=0x7f5b640cb710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:05.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.839+0000 7f5b6b526700 1 --2- 172.21.15.16:0/3494533752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b6415a640 0x7f5b640cb710 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f5b640b5cd0 tx=0x7f5b60009bd0 comp rx=0 tx=0).stop 2023-10-16T12:12:05.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.839+0000 7f5b6b526700 1 -- 172.21.15.16:0/3494533752 shutdown_connections 2023-10-16T12:12:05.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.839+0000 7f5b6b526700 1 --2- 172.21.15.16:0/3494533752 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5b6415e140 0x7f5b640ca1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:05.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.839+0000 7f5b6b526700 1 --2- 172.21.15.16:0/3494533752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b6415a640 0x7f5b640cb710 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:05.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.839+0000 7f5b6b526700 1 --2- 172.21.15.16:0/3494533752 >> v2:172.21.15.49:3300/0 conn(0x7f5b640c6770 0x7f5b640cbc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:05.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.839+0000 7f5b6b526700 1 -- 172.21.15.16:0/3494533752 >> 172.21.15.16:0/3494533752 conn(0x7f5b640cc5e0 msgr2=0x7f5b640cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:05.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.839+0000 7f5b6b526700 1 -- 172.21.15.16:0/3494533752 shutdown_connections 2023-10-16T12:12:05.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.839+0000 7f5b6b526700 1 -- 172.21.15.16:0/3494533752 wait complete. 2023-10-16T12:12:05.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.841+0000 7f5b6b526700 1 Processor -- start 2023-10-16T12:12:05.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.841+0000 7f5b6b526700 1 -- start start 2023-10-16T12:12:05.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.842+0000 7f5b6b526700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b640c6770 0x7f5b64153380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:05.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.842+0000 7f5b6b526700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5b6415a640 0x7f5b640c5be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:05.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.842+0000 7f5b6b526700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5b6415e140 0x7f5b640c6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:05.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.842+0000 7f5b6b526700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b6415ad60 con 0x7f5b640c6770 2023-10-16T12:12:05.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.842+0000 7f5b6b526700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b6415b040 con 0x7f5b6415a640 2023-10-16T12:12:05.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.842+0000 7f5b6b526700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b6415b320 con 0x7f5b6415e140 2023-10-16T12:12:05.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.842+0000 7f5b692c2700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b640c6770 0x7f5b64153380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:05.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.842+0000 7f5b692c2700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b640c6770 0x7f5b64153380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35666/0 (socket says 172.21.15.16:35666) 2023-10-16T12:12:05.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.842+0000 7f5b692c2700 1 -- 172.21.15.16:0/2292967952 learned_addr learned my addr 172.21.15.16:0/2292967952 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:12:05.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.842+0000 7f5b68ac1700 1 --2- 172.21.15.16:0/2292967952 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5b6415a640 0x7f5b640c5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:05.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.842+0000 7f5b69ac3700 1 --2- 172.21.15.16:0/2292967952 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5b6415e140 0x7f5b640c6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:05.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.842+0000 7f5b692c2700 1 -- 172.21.15.16:0/2292967952 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5b6415e140 msgr2=0x7f5b640c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:05.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.842+0000 7f5b692c2700 1 --2- 172.21.15.16:0/2292967952 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5b6415e140 0x7f5b640c6120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:05.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.842+0000 7f5b692c2700 1 -- 172.21.15.16:0/2292967952 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5b6415a640 msgr2=0x7f5b640c5be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:05.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.842+0000 7f5b692c2700 1 --2- 172.21.15.16:0/2292967952 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5b6415a640 0x7f5b640c5be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:05.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.842+0000 7f5b692c2700 1 -- 172.21.15.16:0/2292967952 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b60009550 con 0x7f5b640c6770 2023-10-16T12:12:05.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.842+0000 7f5b68ac1700 1 --2- 172.21.15.16:0/2292967952 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5b6415a640 0x7f5b640c5be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:12:05.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.843+0000 7f5b692c2700 1 --2- 172.21.15.16:0/2292967952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b640c6770 0x7f5b64153380 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f5b60009c10 tx=0x7f5b60004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:05.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.843+0000 7f5b59ffb700 1 -- 172.21.15.16:0/2292967952 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b600036a0 con 0x7f5b640c6770 2023-10-16T12:12:05.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.843+0000 7f5b59ffb700 1 -- 172.21.15.16:0/2292967952 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b6001e070 con 0x7f5b640c6770 2023-10-16T12:12:05.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.843+0000 7f5b6b526700 1 -- 172.21.15.16:0/2292967952 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b6415b600 con 0x7f5b640c6770 2023-10-16T12:12:05.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.843+0000 7f5b59ffb700 1 -- 172.21.15.16:0/2292967952 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b60003a60 con 0x7f5b640c6770 2023-10-16T12:12:05.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.843+0000 7f5b6b526700 1 -- 172.21.15.16:0/2292967952 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b641538c0 con 0x7f5b640c6770 2023-10-16T12:12:05.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.844+0000 7f5b6b526700 1 -- 172.21.15.16:0/2292967952 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b64153ab0 con 0x7f5b640c6770 2023-10-16T12:12:05.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.845+0000 7f5b59ffb700 1 -- 172.21.15.16:0/2292967952 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f5b60003bc0 con 0x7f5b640c6770 2023-10-16T12:12:05.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.845+0000 7f5b59ffb700 1 --2- 172.21.15.16:0/2292967952 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f5b4806af00 0x7f5b4806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:05.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.845+0000 7f5b68ac1700 1 --2- 172.21.15.16:0/2292967952 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f5b4806af00 0x7f5b4806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:05.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.845+0000 7f5b59ffb700 1 -- 172.21.15.16:0/2292967952 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f5b6008a620 con 0x7f5b640c6770 2023-10-16T12:12:05.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.846+0000 7f5b68ac1700 1 --2- 172.21.15.16:0/2292967952 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f5b4806af00 0x7f5b4806d3c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f5b6415c6d0 tx=0x7f5b54008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:05.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:05.850+0000 7f5b59ffb700 1 -- 172.21.15.16:0/2292967952 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5b60055590 con 0x7f5b640c6770 2023-10-16T12:12:06.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.136+0000 7f5b6b526700 1 -- 172.21.15.16:0/2292967952 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f5b640665a0 con 0x7f5b640c6770 2023-10-16T12:12:06.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.138+0000 7f5b59ffb700 1 -- 172.21.15.16:0/2292967952 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v15) v1 ==== 190+0+147 (secure 0 0 0) 0x7f5b600200b0 con 0x7f5b640c6770 2023-10-16T12:12:06.140 INFO:teuthology.orchestra.run.smithi016.stdout:[client.0] 2023-10-16T12:12:06.140 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAPKC1l2l6PFxAA9QyXtBAHdGAXpwH0JfknuQ== 2023-10-16T12:12:06.140 INFO:teuthology.orchestra.run.smithi016.stdout: caps mds = "allow" 2023-10-16T12:12:06.140 INFO:teuthology.orchestra.run.smithi016.stdout: caps mon = "allow r" 2023-10-16T12:12:06.141 INFO:teuthology.orchestra.run.smithi016.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-16T12:12:06.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.140+0000 7f5b6b526700 1 -- 172.21.15.16:0/2292967952 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f5b4806af00 msgr2=0x7f5b4806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:06.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.140+0000 7f5b6b526700 1 --2- 172.21.15.16:0/2292967952 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f5b4806af00 0x7f5b4806d3c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f5b6415c6d0 tx=0x7f5b54008040 comp rx=0 tx=0).stop 2023-10-16T12:12:06.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.140+0000 7f5b6b526700 1 -- 172.21.15.16:0/2292967952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b640c6770 msgr2=0x7f5b64153380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:06.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.140+0000 7f5b6b526700 1 --2- 172.21.15.16:0/2292967952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b640c6770 0x7f5b64153380 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f5b60009c10 tx=0x7f5b60004750 comp rx=0 tx=0).stop 2023-10-16T12:12:06.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.140+0000 7f5b6b526700 1 -- 172.21.15.16:0/2292967952 shutdown_connections 2023-10-16T12:12:06.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.140+0000 7f5b6b526700 1 --2- 172.21.15.16:0/2292967952 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f5b4806af00 0x7f5b4806d3c0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:06.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.140+0000 7f5b6b526700 1 --2- 172.21.15.16:0/2292967952 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5b6415e140 0x7f5b640c6120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:06.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.140+0000 7f5b6b526700 1 --2- 172.21.15.16:0/2292967952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b640c6770 0x7f5b64153380 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:06.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.140+0000 7f5b6b526700 1 --2- 172.21.15.16:0/2292967952 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5b6415a640 0x7f5b640c5be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:06.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.140+0000 7f5b6b526700 1 -- 172.21.15.16:0/2292967952 >> 172.21.15.16:0/2292967952 conn(0x7f5b640cc5e0 msgr2=0x7f5b640afc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:06.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.140+0000 7f5b6b526700 1 -- 172.21.15.16:0/2292967952 shutdown_connections 2023-10-16T12:12:06.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.140+0000 7f5b6b526700 1 -- 172.21.15.16:0/2292967952 wait complete. 2023-10-16T12:12:06.145 INFO:teuthology.orchestra.run.smithi016.stderr:updated caps for client.0 2023-10-16T12:12:06.154 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:12:06.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.277+0000 7ff609592700 1 Processor -- start 2023-10-16T12:12:06.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.277+0000 7ff609592700 1 -- start start 2023-10-16T12:12:06.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.278+0000 7ff609592700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7ff60415b280 0x7ff6041613e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:06.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.278+0000 7ff609592700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff604161c30 con 0x7ff6040cc8a0 2023-10-16T12:12:06.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.278+0000 7ff609592700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff604161f10 con 0x7ff60406a7e0 2023-10-16T12:12:06.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.278+0000 7ff609592700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6041621f0 con 0x7ff60415b280 2023-10-16T12:12:06.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.278+0000 7ff6027fc700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff6040cc8a0 0x7ff60415e700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:53180/0 (socket says 172.21.15.16:53180) 2023-10-16T12:12:06.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.278+0000 7ff6027fc700 1 -- 172.21.15.16:0/989598161 learned_addr learned my addr 172.21.15.16:0/989598161 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:12:06.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.278+0000 7ff602ffd700 1 --2- 172.21.15.16:0/989598161 >> v2:172.21.15.49:3300/0 conn(0x7ff60415b280 0x7ff6041613e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:06.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.278+0000 7ff601ffb700 1 -- 172.21.15.16:0/989598161 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2865413427 0 0) 0x7ff604161c30 con 0x7ff6040cc8a0 2023-10-16T12:12:06.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.279+0000 7ff601ffb700 1 -- 172.21.15.16:0/989598161 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5f00035b0 con 0x7ff6040cc8a0 2023-10-16T12:12:06.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.279+0000 7ff601ffb700 1 -- 172.21.15.16:0/989598161 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1779532951 0 0) 0x7ff5f00035b0 con 0x7ff6040cc8a0 2023-10-16T12:12:06.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.279+0000 7ff601ffb700 1 -- 172.21.15.16:0/989598161 >> v2:172.21.15.49:3300/0 conn(0x7ff60415b280 msgr2=0x7ff6041613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:06.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.279+0000 7ff601ffb700 1 --2- 172.21.15.16:0/989598161 >> v2:172.21.15.49:3300/0 conn(0x7ff60415b280 0x7ff6041613e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:06.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.279+0000 7ff601ffb700 1 -- 172.21.15.16:0/989598161 >> v1:172.21.15.49:6789/0 conn(0x7ff60406a7e0 legacy=0x7ff6040d02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:06.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.279+0000 7ff601ffb700 1 -- 172.21.15.16:0/989598161 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff60415df00 con 0x7ff6040cc8a0 2023-10-16T12:12:06.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.279+0000 7ff601ffb700 1 -- 172.21.15.16:0/989598161 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7ff5ec001670 con 0x7ff6040cc8a0 2023-10-16T12:12:06.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.279+0000 7ff601ffb700 1 -- 172.21.15.16:0/989598161 >> v1:172.21.15.16:6789/0 conn(0x7ff6040cc8a0 legacy=0x7ff60415e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:06.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.279+0000 7ff601ffb700 1 --2- 172.21.15.16:0/989598161 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5f0003a00 0x7ff5f0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:06.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.279+0000 7ff601ffb700 1 --2- 172.21.15.16:0/989598161 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff5f0004850 0x7ff5f0004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:06.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.279+0000 7ff601ffb700 1 --2- 172.21.15.16:0/989598161 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff5f0005210 0x7ff5f000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:06.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.279+0000 7ff601ffb700 1 -- 172.21.15.16:0/989598161 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6041621f0 con 0x7ff5f0003a00 2023-10-16T12:12:06.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.279+0000 7ff601ffb700 1 -- 172.21.15.16:0/989598161 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f000c0b0 con 0x7ff5f0005210 2023-10-16T12:12:06.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.279+0000 7ff601ffb700 1 -- 172.21.15.16:0/989598161 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f000c270 con 0x7ff5f0004850 2023-10-16T12:12:06.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.279+0000 7ff602ffd700 1 --2- 172.21.15.16:0/989598161 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5f0003a00 0x7ff5f0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:06.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.279+0000 7ff6027fc700 1 --2- 172.21.15.16:0/989598161 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff5f0004850 0x7ff5f0004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:06.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.279+0000 7ff6037fe700 1 --2- 172.21.15.16:0/989598161 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff5f0005210 0x7ff5f000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:06.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.279+0000 7ff602ffd700 1 -- 172.21.15.16:0/989598161 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff5f0004850 msgr2=0x7ff5f0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:06.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.279+0000 7ff602ffd700 1 --2- 172.21.15.16:0/989598161 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff5f0004850 0x7ff5f0004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:06.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.279+0000 7ff602ffd700 1 -- 172.21.15.16:0/989598161 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff5f0005210 msgr2=0x7ff5f000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:06.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.279+0000 7ff602ffd700 1 --2- 172.21.15.16:0/989598161 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff5f0005210 0x7ff5f000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:06.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.279+0000 7ff602ffd700 1 -- 172.21.15.16:0/989598161 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5f00035b0 con 0x7ff5f0003a00 2023-10-16T12:12:06.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.279+0000 7ff6037fe700 1 --2- 172.21.15.16:0/989598161 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff5f0005210 0x7ff5f000ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:12:06.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.279+0000 7ff6037fe700 1 -- 172.21.15.16:0/989598161 reap_dead start 2023-10-16T12:12:06.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.279+0000 7ff602ffd700 1 --2- 172.21.15.16:0/989598161 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5f0003a00 0x7ff5f0003e20 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7ff5f4009b60 tx=0x7ff5f0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54a545070a19e672 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:06.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.280+0000 7ff601ffb700 1 -- 172.21.15.16:0/989598161 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f400e6c0 con 0x7ff5f0003a00 2023-10-16T12:12:06.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.280+0000 7ff601ffb700 1 -- 172.21.15.16:0/989598161 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5f400e820 con 0x7ff5f0003a00 2023-10-16T12:12:06.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.280+0000 7ff601ffb700 1 -- 172.21.15.16:0/989598161 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f400eb90 con 0x7ff5f0003a00 2023-10-16T12:12:06.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.280+0000 7ff609592700 1 -- 172.21.15.16:0/989598161 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5f0003a00 msgr2=0x7ff5f0003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:06.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.280+0000 7ff609592700 1 --2- 172.21.15.16:0/989598161 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5f0003a00 0x7ff5f0003e20 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7ff5f4009b60 tx=0x7ff5f0005880 comp rx=0 tx=0).stop 2023-10-16T12:12:06.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.280+0000 7ff609592700 1 -- 172.21.15.16:0/989598161 shutdown_connections 2023-10-16T12:12:06.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.280+0000 7ff609592700 1 --2- 172.21.15.16:0/989598161 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5f0003a00 0x7ff5f0003e20 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:06.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.280+0000 7ff609592700 1 -- 172.21.15.16:0/989598161 >> 172.21.15.16:0/989598161 conn(0x7ff6040b1fd0 msgr2=0x7ff6040b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:06.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.280+0000 7ff609592700 1 -- 172.21.15.16:0/989598161 shutdown_connections 2023-10-16T12:12:06.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.280+0000 7ff609592700 1 -- 172.21.15.16:0/989598161 wait complete. 2023-10-16T12:12:06.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.282+0000 7ff609592700 1 Processor -- start 2023-10-16T12:12:06.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.282+0000 7ff609592700 1 -- start start 2023-10-16T12:12:06.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.282+0000 7ff609592700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5f0003a00 0x7ff6040c7510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:06.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.282+0000 7ff609592700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff604159520 0x7ff6040c3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:06.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.282+0000 7ff609592700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff6040c44b0 0x7ff6040cb510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:06.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.282+0000 7ff609592700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6040c4dc0 con 0x7ff5f0003a00 2023-10-16T12:12:06.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.282+0000 7ff609592700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6040c7a50 con 0x7ff604159520 2023-10-16T12:12:06.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.282+0000 7ff609592700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6040c4ae0 con 0x7ff6040c44b0 2023-10-16T12:12:06.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.282+0000 7ff602ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5f0003a00 0x7ff6040c7510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:06.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.282+0000 7ff602ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5f0003a00 0x7ff6040c7510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35686/0 (socket says 172.21.15.16:35686) 2023-10-16T12:12:06.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.282+0000 7ff602ffd700 1 -- 172.21.15.16:0/3286534838 learned_addr learned my addr 172.21.15.16:0/3286534838 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:12:06.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.283+0000 7ff6027fc700 1 --2- 172.21.15.16:0/3286534838 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff604159520 0x7ff6040c3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:06.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.283+0000 7ff6037fe700 1 --2- 172.21.15.16:0/3286534838 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff6040c44b0 0x7ff6040cb510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:06.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.283+0000 7ff602ffd700 1 -- 172.21.15.16:0/3286534838 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff6040c44b0 msgr2=0x7ff6040cb510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:06.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.283+0000 7ff602ffd700 1 --2- 172.21.15.16:0/3286534838 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff6040c44b0 0x7ff6040cb510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:06.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.283+0000 7ff602ffd700 1 -- 172.21.15.16:0/3286534838 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff604159520 msgr2=0x7ff6040c3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:06.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.283+0000 7ff602ffd700 1 --2- 172.21.15.16:0/3286534838 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff604159520 0x7ff6040c3f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:06.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.283+0000 7ff602ffd700 1 -- 172.21.15.16:0/3286534838 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6040ad320 con 0x7ff5f0003a00 2023-10-16T12:12:06.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.283+0000 7ff602ffd700 1 --2- 172.21.15.16:0/3286534838 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5f0003a00 0x7ff6040c7510 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7ff5f000c080 tx=0x7ff5f40047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:06.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.283+0000 7ff5fb7fe700 1 -- 172.21.15.16:0/3286534838 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f400e6c0 con 0x7ff5f0003a00 2023-10-16T12:12:06.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.283+0000 7ff5fb7fe700 1 -- 172.21.15.16:0/3286534838 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5f40038c0 con 0x7ff5f0003a00 2023-10-16T12:12:06.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.283+0000 7ff609592700 1 -- 172.21.15.16:0/3286534838 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6040ad600 con 0x7ff5f0003a00 2023-10-16T12:12:06.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.283+0000 7ff5fb7fe700 1 -- 172.21.15.16:0/3286534838 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f4003c70 con 0x7ff5f0003a00 2023-10-16T12:12:06.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.283+0000 7ff609592700 1 -- 172.21.15.16:0/3286534838 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6040af5f0 con 0x7ff5f0003a00 2023-10-16T12:12:06.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.284+0000 7ff609592700 1 -- 172.21.15.16:0/3286534838 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6040adc50 con 0x7ff5f0003a00 2023-10-16T12:12:06.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.285+0000 7ff5fb7fe700 1 -- 172.21.15.16:0/3286534838 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7ff5f4013470 con 0x7ff5f0003a00 2023-10-16T12:12:06.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.285+0000 7ff5fb7fe700 1 --2- 172.21.15.16:0/3286534838 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7ff5e406af00 0x7ff5e406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:06.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.285+0000 7ff5fb7fe700 1 -- 172.21.15.16:0/3286534838 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7ff5f408ada0 con 0x7ff5f0003a00 2023-10-16T12:12:06.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.285+0000 7ff6027fc700 1 --2- 172.21.15.16:0/3286534838 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7ff5e406af00 0x7ff5e406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:06.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.286+0000 7ff6027fc700 1 --2- 172.21.15.16:0/3286534838 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7ff5e406af00 0x7ff5e406d3c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ff6040c38c0 tx=0x7ff5ec007040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:06.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.290+0000 7ff5fb7fe700 1 -- 172.21.15.16:0/3286534838 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff5f4055ec0 con 0x7ff5f0003a00 2023-10-16T12:12:06.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.540+0000 7ff609592700 1 -- 172.21.15.16:0/3286534838 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff604072d20 con 0x7ff5f0003a00 2023-10-16T12:12:06.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.541+0000 7ff5fb7fe700 1 -- 172.21.15.16:0/3286534838 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+4910 (secure 0 0 0) 0x7ff5f4017020 con 0x7ff5f0003a00 2023-10-16T12:12:06.543 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:12:06.543 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":18,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5194,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/266676368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":266676368},{"type":"v1","addr":"172.21.15.49:6834","nonce":266676368}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5268,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/1460528697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":1460528697},{"type":"v1","addr":"172.21.15.16:6837","nonce":1460528697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/2374376752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2374376752},{"type":"v1","addr":"172.21.15.16:6836","nonce":2374376752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:12:00.537600+0000","modified":"2023-10-16T12:12:03.538619+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5283},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5283":{"gid":5283,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.49:6835/2788396302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":2788396302},{"type":"v1","addr":"172.21.15.49:6835","nonce":2788396302}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-16T12:12:06.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.543+0000 7ff609592700 1 -- 172.21.15.16:0/3286534838 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7ff5e406af00 msgr2=0x7ff5e406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:06.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.543+0000 7ff609592700 1 --2- 172.21.15.16:0/3286534838 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7ff5e406af00 0x7ff5e406d3c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ff6040c38c0 tx=0x7ff5ec007040 comp rx=0 tx=0).stop 2023-10-16T12:12:06.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.543+0000 7ff609592700 1 -- 172.21.15.16:0/3286534838 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5f0003a00 msgr2=0x7ff6040c7510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:06.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.543+0000 7ff609592700 1 --2- 172.21.15.16:0/3286534838 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5f0003a00 0x7ff6040c7510 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7ff5f000c080 tx=0x7ff5f40047c0 comp rx=0 tx=0).stop 2023-10-16T12:12:06.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.543+0000 7ff609592700 1 -- 172.21.15.16:0/3286534838 shutdown_connections 2023-10-16T12:12:06.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.543+0000 7ff609592700 1 --2- 172.21.15.16:0/3286534838 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7ff5e406af00 0x7ff5e406d3c0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:06.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.543+0000 7ff609592700 1 --2- 172.21.15.16:0/3286534838 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff6040c44b0 0x7ff6040cb510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:06.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.543+0000 7ff609592700 1 --2- 172.21.15.16:0/3286534838 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5f0003a00 0x7ff6040c7510 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:06.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.543+0000 7ff609592700 1 --2- 172.21.15.16:0/3286534838 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff604159520 0x7ff6040c3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:06.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.543+0000 7ff609592700 1 -- 172.21.15.16:0/3286534838 >> 172.21.15.16:0/3286534838 conn(0x7ff6040b1fd0 msgr2=0x7ff6040b03b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:06.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.543+0000 7ff609592700 1 -- 172.21.15.16:0/3286534838 shutdown_connections 2023-10-16T12:12:06.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:06.543+0000 7ff609592700 1 -- 172.21.15.16:0/3286534838 wait complete. 2023-10-16T12:12:06.548 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 18 2023-10-16T12:12:06.959 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 18, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T12:12:00.537600+0000', 'modified': '2023-10-16T12:12:03.538619+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5283}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5283': {'gid': 5283, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.49:6835/2788396302', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6833', 'nonce': 2788396302}, {'type': 'v1', 'addr': '172.21.15.49:6835', 'nonce': 2788396302}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-16T12:12:06.959 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-16T12:12:06.960 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-10-16T12:12:07.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.089+0000 7f4d8de7e700 1 Processor -- start 2023-10-16T12:12:07.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.089+0000 7f4d8de7e700 1 -- start start 2023-10-16T12:12:07.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.090+0000 7f4d8de7e700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f4d8815b630 0x7f4d880cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:07.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.090+0000 7f4d8de7e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4d8815ba10 0x7f4d880cdde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:07.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.090+0000 7f4d8de7e700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d880ce770 con 0x7f4d880ce320 2023-10-16T12:12:07.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.090+0000 7f4d8de7e700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d880cdb20 con 0x7f4d8815b630 2023-10-16T12:12:07.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.090+0000 7f4d8de7e700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d880cc2a0 con 0x7f4d8815ba10 2023-10-16T12:12:07.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.090+0000 7f4d87fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f4d880ce320 0x7f4d880cfca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:53204/0 (socket says 172.21.15.16:53204) 2023-10-16T12:12:07.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.090+0000 7f4d87fff700 1 -- 172.21.15.16:0/3160367859 learned_addr learned my addr 172.21.15.16:0/3160367859 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:12:07.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.090+0000 7f4d86ffd700 1 --2- 172.21.15.16:0/3160367859 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4d8815ba10 0x7f4d880cdde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:07.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.090+0000 7f4d877fe700 1 --2- 172.21.15.16:0/3160367859 >> v2:172.21.15.49:3300/0 conn(0x7f4d8815b630 0x7f4d880cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:07.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.090+0000 7f4d867fc700 1 -- 172.21.15.16:0/3160367859 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3206373900 0 0) 0x7f4d80001490 con 0x7f4d880ce320 2023-10-16T12:12:07.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.090+0000 7f4d867fc700 1 -- 172.21.15.16:0/3160367859 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d740035b0 con 0x7f4d880ce320 2023-10-16T12:12:07.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.091+0000 7f4d86ffd700 1 -- 172.21.15.16:0/3160367859 >> v2:172.21.15.49:3300/0 conn(0x7f4d8815b630 msgr2=0x7f4d880cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:07.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.091+0000 7f4d86ffd700 1 --2- 172.21.15.16:0/3160367859 >> v2:172.21.15.49:3300/0 conn(0x7f4d8815b630 0x7f4d880cd5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:07.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.091+0000 7f4d867fc700 1 -- 172.21.15.16:0/3160367859 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2015479898 0 0) 0x7f4d740035b0 con 0x7f4d880ce320 2023-10-16T12:12:07.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.091+0000 7f4d86ffd700 1 -- 172.21.15.16:0/3160367859 >> v1:172.21.15.16:6789/0 conn(0x7f4d880ce320 legacy=0x7f4d880cfca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:07.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.091+0000 7f4d86ffd700 1 -- 172.21.15.16:0/3160367859 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d880cc540 con 0x7f4d8815ba10 2023-10-16T12:12:07.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.091+0000 7f4d86ffd700 1 --2- 172.21.15.16:0/3160367859 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4d8815ba10 0x7f4d880cdde0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4d78009840 tx=0x7f4d78009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a904e60a5d3bef2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:07.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.091+0000 7f4d867fc700 1 -- 172.21.15.16:0/3160367859 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d80001b80 con 0x7f4d8815ba10 2023-10-16T12:12:07.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.091+0000 7f4d867fc700 1 -- 172.21.15.16:0/3160367859 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d78004030 con 0x7f4d8815ba10 2023-10-16T12:12:07.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.091+0000 7f4d867fc700 1 -- 172.21.15.16:0/3160367859 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d780043c0 con 0x7f4d8815ba10 2023-10-16T12:12:07.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.092+0000 7f4d8de7e700 1 -- 172.21.15.16:0/3160367859 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4d8815ba10 msgr2=0x7f4d880cdde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:07.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.092+0000 7f4d8de7e700 1 --2- 172.21.15.16:0/3160367859 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4d8815ba10 0x7f4d880cdde0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4d78009840 tx=0x7f4d78009b50 comp rx=0 tx=0).stop 2023-10-16T12:12:07.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.092+0000 7f4d8de7e700 1 -- 172.21.15.16:0/3160367859 shutdown_connections 2023-10-16T12:12:07.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.092+0000 7f4d8de7e700 1 --2- 172.21.15.16:0/3160367859 >> v2:172.21.15.49:3300/0 conn(0x7f4d8815b630 0x7f4d880cd5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:07.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.092+0000 7f4d8de7e700 1 --2- 172.21.15.16:0/3160367859 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4d8815ba10 0x7f4d880cdde0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:07.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.092+0000 7f4d8de7e700 1 -- 172.21.15.16:0/3160367859 >> 172.21.15.16:0/3160367859 conn(0x7f4d880d14b0 msgr2=0x7f4d880d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:07.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.092+0000 7f4d8de7e700 1 -- 172.21.15.16:0/3160367859 shutdown_connections 2023-10-16T12:12:07.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.092+0000 7f4d8de7e700 1 -- 172.21.15.16:0/3160367859 wait complete. 2023-10-16T12:12:07.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.093+0000 7f4d8de7e700 1 Processor -- start 2023-10-16T12:12:07.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.093+0000 7f4d8de7e700 1 -- start start 2023-10-16T12:12:07.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.094+0000 7f4d8de7e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d880ce320 0x7f4d881579b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:07.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.094+0000 7f4d8de7e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4d8815b630 0x7f4d8806b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:07.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.094+0000 7f4d8de7e700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4d8815ba10 0x7f4d8815cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:07.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.094+0000 7f4d8de7e700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d880c7dc0 con 0x7f4d880ce320 2023-10-16T12:12:07.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.094+0000 7f4d8de7e700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d88157ef0 con 0x7f4d8815ba10 2023-10-16T12:12:07.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.094+0000 7f4d8de7e700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d881f37a0 con 0x7f4d8815b630 2023-10-16T12:12:07.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.094+0000 7f4d877fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d880ce320 0x7f4d881579b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:07.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.094+0000 7f4d877fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d880ce320 0x7f4d881579b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35700/0 (socket says 172.21.15.16:35700) 2023-10-16T12:12:07.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.094+0000 7f4d877fe700 1 -- 172.21.15.16:0/92049304 learned_addr learned my addr 172.21.15.16:0/92049304 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:12:07.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.094+0000 7f4d87fff700 1 --2- 172.21.15.16:0/92049304 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4d8815ba10 0x7f4d8815cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:07.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.094+0000 7f4d86ffd700 1 --2- 172.21.15.16:0/92049304 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4d8815b630 0x7f4d8806b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:07.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.094+0000 7f4d877fe700 1 -- 172.21.15.16:0/92049304 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4d8815b630 msgr2=0x7f4d8806b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:07.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.094+0000 7f4d877fe700 1 --2- 172.21.15.16:0/92049304 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4d8815b630 0x7f4d8806b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:07.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.094+0000 7f4d877fe700 1 -- 172.21.15.16:0/92049304 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4d8815ba10 msgr2=0x7f4d8815cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:07.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.094+0000 7f4d877fe700 1 --2- 172.21.15.16:0/92049304 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4d8815ba10 0x7f4d8815cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:07.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.094+0000 7f4d877fe700 1 -- 172.21.15.16:0/92049304 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d780094a0 con 0x7f4d880ce320 2023-10-16T12:12:07.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.094+0000 7f4d87fff700 1 --2- 172.21.15.16:0/92049304 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4d8815ba10 0x7f4d8815cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:12:07.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.094+0000 7f4d86ffd700 1 --2- 172.21.15.16:0/92049304 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4d8815b630 0x7f4d8806b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:12:07.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.094+0000 7f4d877fe700 1 --2- 172.21.15.16:0/92049304 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d880ce320 0x7f4d881579b0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f4d7000e900 tx=0x7f4d7000ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:07.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.095+0000 7f4d7ffff700 1 -- 172.21.15.16:0/92049304 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d70009650 con 0x7f4d880ce320 2023-10-16T12:12:07.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.095+0000 7f4d7ffff700 1 -- 172.21.15.16:0/92049304 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d7000c850 con 0x7f4d880ce320 2023-10-16T12:12:07.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.095+0000 7f4d8de7e700 1 -- 172.21.15.16:0/92049304 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d880c8620 con 0x7f4d880ce320 2023-10-16T12:12:07.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.095+0000 7f4d7ffff700 1 -- 172.21.15.16:0/92049304 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d7000cbc0 con 0x7f4d880ce320 2023-10-16T12:12:07.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.095+0000 7f4d8de7e700 1 -- 172.21.15.16:0/92049304 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d880c8be0 con 0x7f4d880ce320 2023-10-16T12:12:07.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.096+0000 7f4d7ffff700 1 -- 172.21.15.16:0/92049304 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f4d7000cd20 con 0x7f4d880ce320 2023-10-16T12:12:07.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.097+0000 7f4d7ffff700 1 --2- 172.21.15.16:0/92049304 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f4d6806af00 0x7f4d6806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:07.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.097+0000 7f4d7ffff700 1 -- 172.21.15.16:0/92049304 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f4d70089bc0 con 0x7f4d880ce320 2023-10-16T12:12:07.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.097+0000 7f4d86ffd700 1 --2- 172.21.15.16:0/92049304 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f4d6806af00 0x7f4d6806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:07.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.097+0000 7f4d86ffd700 1 --2- 172.21.15.16:0/92049304 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f4d6806af00 0x7f4d6806d3c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f4d8815e4f0 tx=0x7f4d78008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:07.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.099+0000 7f4d9a068b80 1 Processor -- start 2023-10-16T12:12:07.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.099+0000 7f4d9a068b80 1 -- start start 2023-10-16T12:12:07.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.100+0000 7f4d9a068b80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b804aaf220 0x55b804ab6a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:07.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.100+0000 7f4d9a068b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b804ab4630 0x55b804ab6fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:07.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.100+0000 7f4d9a068b80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55b804a8a010 0x55b804a8a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:07.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.100+0000 7f4d9a068b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b804a00c20 con 0x55b804ab4630 2023-10-16T12:12:07.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.100+0000 7f4d9a068b80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8049e5c10 con 0x55b804aaf220 2023-10-16T12:12:07.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.100+0000 7f4d9a068b80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b8049e3530 con 0x55b804a8a010 2023-10-16T12:12:07.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.100+0000 7f4d86ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b804ab4630 0x55b804ab6fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:07.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.100+0000 7f4d86ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b804ab4630 0x55b804ab6fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35710/0 (socket says 172.21.15.16:35710) 2023-10-16T12:12:07.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.100+0000 7f4d86ffd700 1 -- 172.21.15.16:0/1868316655 learned_addr learned my addr 172.21.15.16:0/1868316655 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:12:07.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.100+0000 7f4d87fff700 1 --2- 172.21.15.16:0/1868316655 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55b804a8a010 0x55b804a8a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:07.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.100+0000 7f4d877fe700 1 --2- 172.21.15.16:0/1868316655 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b804aaf220 0x55b804ab6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:07.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.100+0000 7f4d86ffd700 1 -- 172.21.15.16:0/1868316655 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55b804a8a010 msgr2=0x55b804a8a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:07.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.100+0000 7f4d86ffd700 1 --2- 172.21.15.16:0/1868316655 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55b804a8a010 0x55b804a8a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:07.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.100+0000 7f4d86ffd700 1 -- 172.21.15.16:0/1868316655 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b804aaf220 msgr2=0x55b804ab6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:07.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.100+0000 7f4d86ffd700 1 --2- 172.21.15.16:0/1868316655 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b804aaf220 0x55b804ab6a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:07.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.100+0000 7f4d86ffd700 1 -- 172.21.15.16:0/1868316655 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b80499c3a0 con 0x55b804ab4630 2023-10-16T12:12:07.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.100+0000 7f4d877fe700 1 --2- 172.21.15.16:0/1868316655 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b804aaf220 0x55b804ab6a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:12:07.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.100+0000 7f4d86ffd700 1 --2- 172.21.15.16:0/1868316655 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b804ab4630 0x55b804ab6fb0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f4d880cc3e0 tx=0x7f4d78010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2524d16493cb9df3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:07.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.100+0000 7f4d7d7fa700 1 -- 172.21.15.16:0/1868316655 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d78003d30 con 0x55b804ab4630 2023-10-16T12:12:07.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.100+0000 7f4d7d7fa700 1 -- 172.21.15.16:0/1868316655 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d7802b070 con 0x55b804ab4630 2023-10-16T12:12:07.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.100+0000 7f4d7d7fa700 1 -- 172.21.15.16:0/1868316655 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d78011830 con 0x55b804ab4630 2023-10-16T12:12:07.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.101+0000 7f4d9a068b80 1 -- 172.21.15.16:0/1868316655 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b804ab4630 msgr2=0x55b804ab6fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:07.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.101+0000 7f4d9a068b80 1 --2- 172.21.15.16:0/1868316655 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b804ab4630 0x55b804ab6fb0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f4d880cc3e0 tx=0x7f4d78010000 comp rx=0 tx=0).stop 2023-10-16T12:12:07.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.101+0000 7f4d9a068b80 1 -- 172.21.15.16:0/1868316655 shutdown_connections 2023-10-16T12:12:07.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.101+0000 7f4d9a068b80 1 --2- 172.21.15.16:0/1868316655 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55b804a8a010 0x55b804a8a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:07.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.101+0000 7f4d9a068b80 1 --2- 172.21.15.16:0/1868316655 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b804aaf220 0x55b804ab6a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:07.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.101+0000 7f4d9a068b80 1 --2- 172.21.15.16:0/1868316655 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b804ab4630 0x55b804ab6fb0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:07.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.101+0000 7f4d9a068b80 1 -- 172.21.15.16:0/1868316655 >> 172.21.15.16:0/1868316655 conn(0x55b8049b9d80 msgr2=0x55b804ad4ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:07.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.101+0000 7f4d9a068b80 1 -- 172.21.15.16:0/1868316655 shutdown_connections 2023-10-16T12:12:07.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.101+0000 7f4d9a068b80 1 -- 172.21.15.16:0/1868316655 wait complete. 2023-10-16T12:12:07.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.102+0000 7f4d9a068b80 1 -- start start 2023-10-16T12:12:07.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.102+0000 7f4d9a068b80 1 Processor -- start 2023-10-16T12:12:07.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.103+0000 7f4d9a068b80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55b804a8a010 0x55b804aae160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:07.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.103+0000 7f4d9a068b80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b804aaf220 0x55b804ae4d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:07.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.103+0000 7f4d9a068b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b804ab4630 0x55b804ab5aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:07.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.103+0000 7f4d9a068b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8049dee20 con 0x55b804ab4630 2023-10-16T12:12:07.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.103+0000 7f4d9a068b80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b80497b6c0 con 0x55b804aaf220 2023-10-16T12:12:07.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.103+0000 7f4d9a068b80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b80497b360 con 0x55b804a8a010 2023-10-16T12:12:07.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.103+0000 7f4d9a068b80 10 client.0 _ll_register_callbacks cb 0x55b8047d87b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T12:12:07.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.103+0000 7f4d87fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b804ab4630 0x55b804ab5aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:07.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.103+0000 7f4d87fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b804ab4630 0x55b804ab5aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35718/0 (socket says 172.21.15.16:35718) 2023-10-16T12:12:07.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.103+0000 7f4d87fff700 1 -- 172.21.15.16:0/3237478747 learned_addr learned my addr 172.21.15.16:0/3237478747 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:12:07.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.103+0000 7f4d877fe700 1 --2- 172.21.15.16:0/3237478747 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55b804a8a010 0x55b804aae160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:07.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.103+0000 7f4d86ffd700 1 --2- 172.21.15.16:0/3237478747 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b804aaf220 0x55b804ae4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:07.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.103+0000 7f4d87fff700 1 -- 172.21.15.16:0/3237478747 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55b804a8a010 msgr2=0x55b804aae160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:07.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.103+0000 7f4d87fff700 1 --2- 172.21.15.16:0/3237478747 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55b804a8a010 0x55b804aae160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:07.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.103+0000 7f4d87fff700 1 -- 172.21.15.16:0/3237478747 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b804aaf220 msgr2=0x55b804ae4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:07.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.103+0000 7f4d87fff700 1 --2- 172.21.15.16:0/3237478747 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b804aaf220 0x55b804ae4d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:07.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.103+0000 7f4d87fff700 1 -- 172.21.15.16:0/3237478747 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d78021050 con 0x55b804ab4630 2023-10-16T12:12:07.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.103+0000 7f4d87fff700 1 --2- 172.21.15.16:0/3237478747 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b804ab4630 0x55b804ab5aa0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f4d8000e830 tx=0x7f4d8000ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db49d147a971c83d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:07.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.103+0000 7f4d667fc700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2023-10-16T12:12:07.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.103+0000 7f4d667fc700 1 -- 172.21.15.16:0/3237478747 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d80018070 con 0x55b804ab4630 2023-10-16T12:12:07.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.104+0000 7f4d9a068b80 1 -- 172.21.15.16:0/3237478747 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b804a7f5b0 con 0x55b804ab4630 2023-10-16T12:12:07.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.104+0000 7f4d667fc700 1 -- 172.21.15.16:0/3237478747 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f4d80015860 con 0x55b804ab4630 2023-10-16T12:12:07.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.104+0000 7f4d667fc700 1 -- 172.21.15.16:0/3237478747 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=18) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4d80015a20 con 0x55b804ab4630 2023-10-16T12:12:07.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.104+0000 7f4d9a068b80 10 client.5454 fetch_fsmap learned FSMap version 18 2023-10-16T12:12:07.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.104+0000 7f4d9a068b80 1 -- 172.21.15.16:0/3237478747 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=18,monmap=2+}) v3 -- 0x55b804809d90 con 0x55b804ab4630 2023-10-16T12:12:07.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.104+0000 7f4d667fc700 1 -- 172.21.15.16:0/3237478747 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d80022020 con 0x55b804ab4630 2023-10-16T12:12:07.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.104+0000 7f4d667fc700 1 -- 172.21.15.16:0/3237478747 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 18) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f4d8001b030 con 0x55b804ab4630 2023-10-16T12:12:07.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.104+0000 7f4d9a068b80 10 client.5454 fetch_fsmap finished waiting for FSMap version 18 2023-10-16T12:12:07.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.104+0000 7f4d9a068b80 10 client.5454 resolve_mds: resolved name 'b' to daemon mds.b 2023-10-16T12:12:07.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.104+0000 7f4d9a068b80 20 client.5454 populate_metadata read hostname 'smithi016' 2023-10-16T12:12:07.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.104+0000 7f4d9a068b80 1 --2- 172.21.15.16:0/3237478747 >> [v2:172.21.15.49:6833/2788396302,v1:172.21.15.49:6835/2788396302] conn(0x55b804ab27a0 0x55b804ae0d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:07.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.104+0000 7f4d9a068b80 4 client.5454 mds_command: new command op to 5283 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T12:12:07.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.104+0000 7f4d9a068b80 1 -- 172.21.15.16:0/3237478747 --> [v2:172.21.15.49:6833/2788396302,v1:172.21.15.49:6835/2788396302] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b8049d7e00 con 0x55b804ab27a0 2023-10-16T12:12:07.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.104+0000 7f4d877fe700 1 --2- 172.21.15.16:0/3237478747 >> [v2:172.21.15.49:6833/2788396302,v1:172.21.15.49:6835/2788396302] conn(0x55b804ab27a0 0x55b804ae0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:07.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.105+0000 7f4d667fc700 10 client.5454 ms_handle_connect on v2:172.21.15.49:6833/2788396302 2023-10-16T12:12:07.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.105+0000 7f4d877fe700 1 --2- 172.21.15.16:0/3237478747 >> [v2:172.21.15.49:6833/2788396302,v1:172.21.15.49:6835/2788396302] conn(0x55b804ab27a0 0x55b804ae0d40 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f55b0a020273b863 server_cookie=4b1960a631cc359d in_seq=0 out_seq=0 2023-10-16T12:12:07.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.105+0000 7f4d667fc700 1 -- 172.21.15.16:0/3237478747 <== mds.0 v2:172.21.15.49:6833/2788396302 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55b8049d7e00 con 0x55b804ab27a0 2023-10-16T12:12:07.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.105+0000 7f4d667fc700 10 client.5454 handle_command_reply: tid=0 2023-10-16T12:12:07.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.105+0000 7f4d9a068b80 1 client.5454 shutdown 2023-10-16T12:12:07.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.105+0000 7f4d9a068b80 1 -- 172.21.15.16:0/3237478747 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b804ab4630 msgr2=0x55b804ab5aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:07.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.105+0000 7f4d9a068b80 1 --2- 172.21.15.16:0/3237478747 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b804ab4630 0x55b804ab5aa0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f4d8000e830 tx=0x7f4d8000ebf0 comp rx=0 tx=0).stop 2023-10-16T12:12:07.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.106+0000 7f4d9a068b80 1 -- 172.21.15.16:0/3237478747 shutdown_connections 2023-10-16T12:12:07.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.106+0000 7f4d9a068b80 1 --2- 172.21.15.16:0/3237478747 >> [v2:172.21.15.49:6833/2788396302,v1:172.21.15.49:6835/2788396302] conn(0x55b804ab27a0 0x55b804ae0d40 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:07.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.106+0000 7f4d9a068b80 1 --2- 172.21.15.16:0/3237478747 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b804ab4630 0x55b804ab5aa0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:07.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.106+0000 7f4d9a068b80 1 --2- 172.21.15.16:0/3237478747 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55b804a8a010 0x55b804aae160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:07.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.106+0000 7f4d9a068b80 1 --2- 172.21.15.16:0/3237478747 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b804aaf220 0x55b804ae4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:07.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.106+0000 7f4d9a068b80 1 -- 172.21.15.16:0/3237478747 >> 172.21.15.16:0/3237478747 conn(0x55b8049b9d80 msgr2=0x55b804ad1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:07.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.106+0000 7f4d667fc700 0 client.5454 ms_handle_reset on v2:172.21.15.49:6833/2788396302 2023-10-16T12:12:07.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.106+0000 7f4d9a068b80 1 -- 172.21.15.16:0/3237478747 shutdown_connections 2023-10-16T12:12:07.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.106+0000 7f4d9a068b80 1 -- 172.21.15.16:0/3237478747 wait complete. 2023-10-16T12:12:07.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.106+0000 7f4d9a068b80 20 client.5454 trim_cache size 0 max 16384 2023-10-16T12:12:07.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.124+0000 7f4d9a068b80 1 Processor -- start 2023-10-16T12:12:07.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.124+0000 7f4d9a068b80 1 -- start start 2023-10-16T12:12:07.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.125+0000 7f4d9a068b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b804a8a010 0x55b804a86a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:07.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.125+0000 7f4d9a068b80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55b804aaf220 0x55b804a86fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:07.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.125+0000 7f4d9a068b80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b804ab4630 0x55b804aba030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:07.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.125+0000 7f4d9a068b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b80485f8d0 con 0x55b804a8a010 2023-10-16T12:12:07.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.125+0000 7f4d9a068b80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b804a8e070 con 0x55b804ab4630 2023-10-16T12:12:07.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.125+0000 7f4d9a068b80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b804a694c0 con 0x55b804aaf220 2023-10-16T12:12:07.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.125+0000 7f4d877fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b804a8a010 0x55b804a86a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:07.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.125+0000 7f4d877fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b804a8a010 0x55b804a86a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35722/0 (socket says 172.21.15.16:35722) 2023-10-16T12:12:07.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.125+0000 7f4d877fe700 1 -- 172.21.15.16:0/490254570 learned_addr learned my addr 172.21.15.16:0/490254570 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:12:07.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.125+0000 7f4d87fff700 1 --2- 172.21.15.16:0/490254570 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b804ab4630 0x55b804aba030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:07.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.125+0000 7f4d86ffd700 1 --2- 172.21.15.16:0/490254570 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55b804aaf220 0x55b804a86fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:07.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.125+0000 7f4d877fe700 1 -- 172.21.15.16:0/490254570 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55b804aaf220 msgr2=0x55b804a86fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:07.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.125+0000 7f4d877fe700 1 --2- 172.21.15.16:0/490254570 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55b804aaf220 0x55b804a86fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:07.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.125+0000 7f4d877fe700 1 -- 172.21.15.16:0/490254570 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b804ab4630 msgr2=0x55b804aba030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:07.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.125+0000 7f4d877fe700 1 --2- 172.21.15.16:0/490254570 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b804ab4630 0x55b804aba030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:07.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.125+0000 7f4d877fe700 1 -- 172.21.15.16:0/490254570 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b8047f2c10 con 0x55b804a8a010 2023-10-16T12:12:07.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.126+0000 7f4d877fe700 1 --2- 172.21.15.16:0/490254570 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b804a8a010 0x55b804a86a90 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f4d70054c00 tx=0x7f4d70087de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=799046ad77e0fb97 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:07.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.126+0000 7f4d7dffb700 1 -- 172.21.15.16:0/490254570 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d70060a50 con 0x55b804a8a010 2023-10-16T12:12:07.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.126+0000 7f4d7dffb700 1 -- 172.21.15.16:0/490254570 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d70056af0 con 0x55b804a8a010 2023-10-16T12:12:07.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.126+0000 7f4d7dffb700 1 -- 172.21.15.16:0/490254570 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d700589b0 con 0x55b804a8a010 2023-10-16T12:12:07.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.127+0000 7f4d9a068b80 1 -- 172.21.15.16:0/490254570 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b804a8a010 msgr2=0x55b804a86a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:07.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.127+0000 7f4d9a068b80 1 --2- 172.21.15.16:0/490254570 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b804a8a010 0x55b804a86a90 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f4d70054c00 tx=0x7f4d70087de0 comp rx=0 tx=0).stop 2023-10-16T12:12:07.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.127+0000 7f4d9a068b80 1 -- 172.21.15.16:0/490254570 shutdown_connections 2023-10-16T12:12:07.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.127+0000 7f4d9a068b80 1 --2- 172.21.15.16:0/490254570 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b804ab4630 0x55b804aba030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:07.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.127+0000 7f4d9a068b80 1 --2- 172.21.15.16:0/490254570 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b804a8a010 0x55b804a86a90 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:07.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.127+0000 7f4d9a068b80 1 --2- 172.21.15.16:0/490254570 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55b804aaf220 0x55b804a86fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:07.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.127+0000 7f4d9a068b80 1 -- 172.21.15.16:0/490254570 >> 172.21.15.16:0/490254570 conn(0x55b8049b9d80 msgr2=0x55b804af37f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:07.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.127+0000 7f4d9a068b80 1 -- 172.21.15.16:0/490254570 shutdown_connections 2023-10-16T12:12:07.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.127+0000 7f4d9a068b80 1 -- 172.21.15.16:0/490254570 wait complete. 2023-10-16T12:12:07.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.128+0000 7f4d9a068b80 1 -- start start 2023-10-16T12:12:07.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.128+0000 7f4d9a068b80 1 Processor -- start 2023-10-16T12:12:07.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.128+0000 7f4d9a068b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b804a8a010 0x55b804acb650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:07.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.128+0000 7f4d9a068b80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b804aaf220 0x55b804b38a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:07.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.128+0000 7f4d9a068b80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55b804ab4630 0x55b804b38f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:07.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.128+0000 7f4d9a068b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b804acbee0 con 0x55b804a8a010 2023-10-16T12:12:07.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.128+0000 7f4d9a068b80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b804acd000 con 0x55b804aaf220 2023-10-16T12:12:07.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.128+0000 7f4d9a068b80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b804ae4f20 con 0x55b804ab4630 2023-10-16T12:12:07.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.129+0000 7f4d9a068b80 10 client.0 _ll_register_callbacks cb 0x7f4d8000e3f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T12:12:07.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.129+0000 7f4d877fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b804a8a010 0x55b804acb650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:07.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.129+0000 7f4d877fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b804a8a010 0x55b804acb650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35728/0 (socket says 172.21.15.16:35728) 2023-10-16T12:12:07.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.129+0000 7f4d877fe700 1 -- 172.21.15.16:0/1645797644 learned_addr learned my addr 172.21.15.16:0/1645797644 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:12:07.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.129+0000 7f4d87fff700 1 --2- 172.21.15.16:0/1645797644 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55b804ab4630 0x55b804b38f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:07.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.129+0000 7f4d86ffd700 1 --2- 172.21.15.16:0/1645797644 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b804aaf220 0x55b804b38a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:07.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.129+0000 7f4d877fe700 1 -- 172.21.15.16:0/1645797644 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55b804ab4630 msgr2=0x55b804b38f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:07.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.129+0000 7f4d877fe700 1 --2- 172.21.15.16:0/1645797644 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55b804ab4630 0x55b804b38f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:07.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.129+0000 7f4d877fe700 1 -- 172.21.15.16:0/1645797644 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b804aaf220 msgr2=0x55b804b38a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:07.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.129+0000 7f4d877fe700 1 --2- 172.21.15.16:0/1645797644 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b804aaf220 0x55b804b38a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:07.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.129+0000 7f4d877fe700 1 -- 172.21.15.16:0/1645797644 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d880c8be0 con 0x55b804a8a010 2023-10-16T12:12:07.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.129+0000 7f4d87fff700 1 --2- 172.21.15.16:0/1645797644 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55b804ab4630 0x55b804b38f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:12:07.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.129+0000 7f4d86ffd700 1 --2- 172.21.15.16:0/1645797644 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b804aaf220 0x55b804b38a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:12:07.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.129+0000 7f4d877fe700 1 --2- 172.21.15.16:0/1645797644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b804a8a010 0x55b804acb650 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f4d700586e0 tx=0x7f4d80001cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97a72b7ce195f03c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:07.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.129+0000 7f4d667fc700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2023-10-16T12:12:07.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.129+0000 7f4d667fc700 1 -- 172.21.15.16:0/1645797644 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d70057e40 con 0x55b804a8a010 2023-10-16T12:12:07.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.129+0000 7f4d667fc700 1 -- 172.21.15.16:0/1645797644 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f4d70060a50 con 0x55b804a8a010 2023-10-16T12:12:07.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.129+0000 7f4d9a068b80 1 -- 172.21.15.16:0/1645797644 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4d7802b070 con 0x55b804a8a010 2023-10-16T12:12:07.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.129+0000 7f4d667fc700 1 -- 172.21.15.16:0/1645797644 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=18) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4d700589b0 con 0x55b804a8a010 2023-10-16T12:12:07.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.129+0000 7f4d9a068b80 10 client.5460 fetch_fsmap learned FSMap version 18 2023-10-16T12:12:07.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.129+0000 7f4d9a068b80 1 -- 172.21.15.16:0/1645797644 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=18,monmap=2+}) v3 -- 0x55b804830200 con 0x55b804a8a010 2023-10-16T12:12:07.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.129+0000 7f4d667fc700 1 -- 172.21.15.16:0/1645797644 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d7802b070 con 0x55b804a8a010 2023-10-16T12:12:07.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.130+0000 7f4d667fc700 1 -- 172.21.15.16:0/1645797644 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 18) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f4d700605c0 con 0x55b804a8a010 2023-10-16T12:12:07.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.130+0000 7f4d9a068b80 10 client.5460 fetch_fsmap finished waiting for FSMap version 18 2023-10-16T12:12:07.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.130+0000 7f4d9a068b80 10 client.5460 resolve_mds: resolved name 'b' to daemon mds.b 2023-10-16T12:12:07.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.130+0000 7f4d9a068b80 20 client.5460 populate_metadata read hostname 'smithi016' 2023-10-16T12:12:07.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.130+0000 7f4d9a068b80 1 --2- 172.21.15.16:0/1645797644 >> [v2:172.21.15.49:6833/2788396302,v1:172.21.15.49:6835/2788396302] conn(0x55b804ab2440 0x55b804ab2820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:07.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.130+0000 7f4d9a068b80 4 client.5460 mds_command: new command op to 5283 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T12:12:07.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.130+0000 7f4d9a068b80 1 -- 172.21.15.16:0/1645797644 --> [v2:172.21.15.49:6833/2788396302,v1:172.21.15.49:6835/2788396302] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55b8049d7b20 con 0x55b804ab2440 2023-10-16T12:12:07.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.130+0000 7f4d86ffd700 1 --2- 172.21.15.16:0/1645797644 >> [v2:172.21.15.49:6833/2788396302,v1:172.21.15.49:6835/2788396302] conn(0x55b804ab2440 0x55b804ab2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:07.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.130+0000 7f4d667fc700 10 client.5460 ms_handle_connect on v2:172.21.15.49:6833/2788396302 2023-10-16T12:12:07.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.130+0000 7f4d86ffd700 1 --2- 172.21.15.16:0/1645797644 >> [v2:172.21.15.49:6833/2788396302,v1:172.21.15.49:6835/2788396302] conn(0x55b804ab2440 0x55b804ab2820 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f596649872f6eba1 server_cookie=64a9a977eac7e66 in_seq=0 out_seq=0 2023-10-16T12:12:07.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.131+0000 7f4d667fc700 1 -- 172.21.15.16:0/1645797644 <== mds.0 v2:172.21.15.49:6833/2788396302 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x55b8049d7b20 con 0x55b804ab2440 2023-10-16T12:12:07.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.131+0000 7f4d667fc700 10 client.5460 handle_command_reply: tid=0 2023-10-16T12:12:07.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.131+0000 7f4d9a068b80 1 client.5460 shutdown 2023-10-16T12:12:07.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.131+0000 7f4d9a068b80 1 -- 172.21.15.16:0/1645797644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b804a8a010 msgr2=0x55b804acb650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:07.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.131+0000 7f4d9a068b80 1 --2- 172.21.15.16:0/1645797644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b804a8a010 0x55b804acb650 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f4d700586e0 tx=0x7f4d80001cb0 comp rx=0 tx=0).stop 2023-10-16T12:12:07.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.131+0000 7f4d9a068b80 1 -- 172.21.15.16:0/1645797644 shutdown_connections 2023-10-16T12:12:07.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.131+0000 7f4d9a068b80 1 --2- 172.21.15.16:0/1645797644 >> [v2:172.21.15.49:6833/2788396302,v1:172.21.15.49:6835/2788396302] conn(0x55b804ab2440 0x55b804ab2820 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:07.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.131+0000 7f4d9a068b80 1 --2- 172.21.15.16:0/1645797644 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55b804ab4630 0x55b804b38f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:07.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.131+0000 7f4d667fc700 0 client.5460 ms_handle_reset on v2:172.21.15.49:6833/2788396302 2023-10-16T12:12:07.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.131+0000 7f4d9a068b80 1 --2- 172.21.15.16:0/1645797644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b804a8a010 0x55b804acb650 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:07.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.131+0000 7f4d9a068b80 1 --2- 172.21.15.16:0/1645797644 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b804aaf220 0x55b804b38a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:07.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.131+0000 7f4d9a068b80 1 -- 172.21.15.16:0/1645797644 >> 172.21.15.16:0/1645797644 conn(0x55b8049b9d80 msgr2=0x55b804ab3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:07.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.131+0000 7f4d9a068b80 1 -- 172.21.15.16:0/1645797644 shutdown_connections 2023-10-16T12:12:07.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.131+0000 7f4d9a068b80 1 -- 172.21.15.16:0/1645797644 wait complete. 2023-10-16T12:12:07.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.131+0000 7f4d9a068b80 20 client.5460 trim_cache size 0 max 16384 2023-10-16T12:12:07.148 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-10-16T12:12:07.148 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "5b28dfa1-b21f-474a-9b2e-9fb77af58254", 2023-10-16T12:12:07.148 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 0, 2023-10-16T12:12:07.149 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 5283, 2023-10-16T12:12:07.149 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2023-10-16T12:12:07.149 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2023-10-16T12:12:07.149 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2023-10-16T12:12:07.149 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 6.0295629589999997, 2023-10-16T12:12:07.150 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 18, 2023-10-16T12:12:07.150 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 39, 2023-10-16T12:12:07.150 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 39, 2023-10-16T12:12:07.150 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 13.893631672 2023-10-16T12:12:07.150 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-10-16T12:12:07.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.131+0000 7f4d8de7e700 1 -- 172.21.15.16:0/92049304 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f4d6806af00 msgr2=0x7f4d6806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:07.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.131+0000 7f4d8de7e700 1 --2- 172.21.15.16:0/92049304 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f4d6806af00 0x7f4d6806d3c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f4d8815e4f0 tx=0x7f4d78008030 comp rx=0 tx=0).stop 2023-10-16T12:12:07.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.131+0000 7f4d8de7e700 1 -- 172.21.15.16:0/92049304 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d880ce320 msgr2=0x7f4d881579b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:07.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.131+0000 7f4d8de7e700 1 --2- 172.21.15.16:0/92049304 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d880ce320 0x7f4d881579b0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f4d7000e900 tx=0x7f4d7000ecc0 comp rx=0 tx=0).stop 2023-10-16T12:12:07.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.132+0000 7f4d8de7e700 1 -- 172.21.15.16:0/92049304 shutdown_connections 2023-10-16T12:12:07.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.132+0000 7f4d8de7e700 1 --2- 172.21.15.16:0/92049304 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f4d6806af00 0x7f4d6806d3c0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:07.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.132+0000 7f4d8de7e700 1 --2- 172.21.15.16:0/92049304 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4d8815ba10 0x7f4d8815cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:07.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.132+0000 7f4d8de7e700 1 --2- 172.21.15.16:0/92049304 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d880ce320 0x7f4d881579b0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:07.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.132+0000 7f4d8de7e700 1 --2- 172.21.15.16:0/92049304 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4d8815b630 0x7f4d8806b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:07.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.132+0000 7f4d8de7e700 1 -- 172.21.15.16:0/92049304 >> 172.21.15.16:0/92049304 conn(0x7f4d880d14b0 msgr2=0x7f4d880b4710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:07.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.132+0000 7f4d8de7e700 1 -- 172.21.15.16:0/92049304 shutdown_connections 2023-10-16T12:12:07.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.132+0000 7f4d8de7e700 1 -- 172.21.15.16:0/92049304 wait complete. 2023-10-16T12:12:07.154 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T12:12:07.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.279+0000 7f4b1afc6700 1 Processor -- start 2023-10-16T12:12:07.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.279+0000 7f4b1afc6700 1 -- start start 2023-10-16T12:12:07.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.279+0000 7f4b1afc6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4b1415d750 0x7f4b140cf920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:07.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.279+0000 7f4b1afc6700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f4b14161490 0x7f4b14161870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:07.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.279+0000 7f4b1afc6700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b14161db0 con 0x7f4b1415db30 2023-10-16T12:12:07.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.279+0000 7f4b1afc6700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b140cfe60 con 0x7f4b1415d750 2023-10-16T12:12:07.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.279+0000 7f4b1afc6700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b140d0070 con 0x7f4b14161490 2023-10-16T12:12:07.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.280+0000 7f4b18d62700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4b1415d750 0x7f4b140cf920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:07.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.280+0000 7f4b18d62700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4b1415d750 0x7f4b140cf920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35736/0 (socket says 172.21.15.16:35736) 2023-10-16T12:12:07.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.280+0000 7f4b18d62700 1 -- 172.21.15.16:0/2728991555 learned_addr learned my addr 172.21.15.16:0/2728991555 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:12:07.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.280+0000 7f4b18d62700 1 -- 172.21.15.16:0/2728991555 >> v2:172.21.15.49:3300/0 conn(0x7f4b14161490 msgr2=0x7f4b14161870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:07.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.280+0000 7f4b19563700 1 --2- 172.21.15.16:0/2728991555 >> v2:172.21.15.49:3300/0 conn(0x7f4b14161490 0x7f4b14161870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:07.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.280+0000 7f4b18d62700 1 --2- 172.21.15.16:0/2728991555 >> v2:172.21.15.49:3300/0 conn(0x7f4b14161490 0x7f4b14161870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:07.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.280+0000 7f4b18d62700 1 -- 172.21.15.16:0/2728991555 >> v1:172.21.15.49:6789/0 conn(0x7f4b1415db30 legacy=0x7f4b1406c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:07.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.280+0000 7f4b18d62700 1 -- 172.21.15.16:0/2728991555 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b140d0350 con 0x7f4b1415d750 2023-10-16T12:12:07.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.280+0000 7f4b18d62700 1 --2- 172.21.15.16:0/2728991555 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b1415d750 0x7f4b140cf920 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f4b140cef80 tx=0x7f4b08009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1412058d860d59a4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:07.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.280+0000 7f4b137fe700 1 -- 172.21.15.16:0/2728991555 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b08005e30 con 0x7f4b1415d750 2023-10-16T12:12:07.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.280+0000 7f4b137fe700 1 -- 172.21.15.16:0/2728991555 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b08006030 con 0x7f4b1415d750 2023-10-16T12:12:07.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.280+0000 7f4b137fe700 1 -- 172.21.15.16:0/2728991555 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b080063e0 con 0x7f4b1415d750 2023-10-16T12:12:07.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.281+0000 7f4b1afc6700 1 -- 172.21.15.16:0/2728991555 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b1415d750 msgr2=0x7f4b140cf920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:07.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.281+0000 7f4b1afc6700 1 --2- 172.21.15.16:0/2728991555 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b1415d750 0x7f4b140cf920 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f4b140cef80 tx=0x7f4b08009bd0 comp rx=0 tx=0).stop 2023-10-16T12:12:07.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.281+0000 7f4b1afc6700 1 -- 172.21.15.16:0/2728991555 shutdown_connections 2023-10-16T12:12:07.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.281+0000 7f4b1afc6700 1 --2- 172.21.15.16:0/2728991555 >> v2:172.21.15.49:3300/0 conn(0x7f4b14161490 0x7f4b14161870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:07.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.281+0000 7f4b1afc6700 1 --2- 172.21.15.16:0/2728991555 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b1415d750 0x7f4b140cf920 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:07.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.281+0000 7f4b1afc6700 1 -- 172.21.15.16:0/2728991555 >> 172.21.15.16:0/2728991555 conn(0x7f4b140d14b0 msgr2=0x7f4b140d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:07.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.281+0000 7f4b1afc6700 1 -- 172.21.15.16:0/2728991555 shutdown_connections 2023-10-16T12:12:07.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.281+0000 7f4b1afc6700 1 -- 172.21.15.16:0/2728991555 wait complete. 2023-10-16T12:12:07.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.282+0000 7f4b1afc6700 1 Processor -- start 2023-10-16T12:12:07.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.283+0000 7f4b1afc6700 1 -- start start 2023-10-16T12:12:07.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.284+0000 7f4b1afc6700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4b1415d750 0x7f4b141f2c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:07.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.284+0000 7f4b1afc6700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4b1415db30 0x7f4b140b6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:07.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.284+0000 7f4b1afc6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b14161490 0x7f4b140b4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:07.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.284+0000 7f4b1afc6700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b140b73c0 con 0x7f4b14161490 2023-10-16T12:12:07.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.284+0000 7f4b1afc6700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b1414b190 con 0x7f4b1415db30 2023-10-16T12:12:07.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.284+0000 7f4b1afc6700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b140c5d10 con 0x7f4b1415d750 2023-10-16T12:12:07.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.284+0000 7f4b19563700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b14161490 0x7f4b140b4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:07.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.284+0000 7f4b19563700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b14161490 0x7f4b140b4300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35746/0 (socket says 172.21.15.16:35746) 2023-10-16T12:12:07.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.284+0000 7f4b19563700 1 -- 172.21.15.16:0/4067468223 learned_addr learned my addr 172.21.15.16:0/4067468223 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:12:07.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.284+0000 7f4b18d62700 1 --2- 172.21.15.16:0/4067468223 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4b1415d750 0x7f4b141f2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:07.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.284+0000 7f4b13fff700 1 --2- 172.21.15.16:0/4067468223 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4b1415db30 0x7f4b140b6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:07.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.284+0000 7f4b19563700 1 -- 172.21.15.16:0/4067468223 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4b1415d750 msgr2=0x7f4b141f2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:07.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.284+0000 7f4b19563700 1 --2- 172.21.15.16:0/4067468223 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4b1415d750 0x7f4b141f2c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:07.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.284+0000 7f4b19563700 1 -- 172.21.15.16:0/4067468223 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4b1415db30 msgr2=0x7f4b140b6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:07.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.284+0000 7f4b19563700 1 --2- 172.21.15.16:0/4067468223 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4b1415db30 0x7f4b140b6bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:07.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.284+0000 7f4b19563700 1 -- 172.21.15.16:0/4067468223 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b08009500 con 0x7f4b14161490 2023-10-16T12:12:07.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.284+0000 7f4b19563700 1 --2- 172.21.15.16:0/4067468223 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b14161490 0x7f4b140b4300 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f4b0c00cc30 tx=0x7f4b0c00cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:07.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.284+0000 7f4b117fa700 1 -- 172.21.15.16:0/4067468223 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b0c00aa50 con 0x7f4b14161490 2023-10-16T12:12:07.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.284+0000 7f4b117fa700 1 -- 172.21.15.16:0/4067468223 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b0c00abb0 con 0x7f4b14161490 2023-10-16T12:12:07.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.284+0000 7f4b1afc6700 1 -- 172.21.15.16:0/4067468223 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b141f3170 con 0x7f4b14161490 2023-10-16T12:12:07.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.284+0000 7f4b117fa700 1 -- 172.21.15.16:0/4067468223 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b0c003a20 con 0x7f4b14161490 2023-10-16T12:12:07.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.285+0000 7f4b1afc6700 1 -- 172.21.15.16:0/4067468223 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b140c53e0 con 0x7f4b14161490 2023-10-16T12:12:07.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.286+0000 7f4b1afc6700 1 -- 172.21.15.16:0/4067468223 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b140d1010 con 0x7f4b14161490 2023-10-16T12:12:07.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.286+0000 7f4b117fa700 1 -- 172.21.15.16:0/4067468223 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f4b0c00f3f0 con 0x7f4b14161490 2023-10-16T12:12:07.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.286+0000 7f4b117fa700 1 --2- 172.21.15.16:0/4067468223 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f4af806af00 0x7f4af806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:07.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.286+0000 7f4b117fa700 1 -- 172.21.15.16:0/4067468223 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f4b0c0898d0 con 0x7f4b14161490 2023-10-16T12:12:07.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.286+0000 7f4b18d62700 1 --2- 172.21.15.16:0/4067468223 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f4af806af00 0x7f4af806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:07.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.287+0000 7f4b18d62700 1 --2- 172.21.15.16:0/4067468223 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f4af806af00 0x7f4af806d3c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f4b140cef80 tx=0x7f4b08008070 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:07.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.291+0000 7f4b117fa700 1 -- 172.21.15.16:0/4067468223 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4b0c054720 con 0x7f4b14161490 2023-10-16T12:12:07.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.546+0000 7f4b1afc6700 1 -- 172.21.15.16:0/4067468223 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f4b1404ebb0 con 0x7f4b14161490 2023-10-16T12:12:07.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.546+0000 7f4b117fa700 1 -- 172.21.15.16:0/4067468223 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v18) v1 ==== 53+0+73 (secure 0 0 0) 0x7f4b0c01d020 con 0x7f4b14161490 2023-10-16T12:12:07.548 INFO:teuthology.orchestra.run.smithi016.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-16T12:12:07.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.548+0000 7f4b1afc6700 1 -- 172.21.15.16:0/4067468223 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f4af806af00 msgr2=0x7f4af806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:07.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.548+0000 7f4b1afc6700 1 --2- 172.21.15.16:0/4067468223 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f4af806af00 0x7f4af806d3c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f4b140cef80 tx=0x7f4b08008070 comp rx=0 tx=0).stop 2023-10-16T12:12:07.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.549+0000 7f4b1afc6700 1 -- 172.21.15.16:0/4067468223 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b14161490 msgr2=0x7f4b140b4300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:07.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.549+0000 7f4b1afc6700 1 --2- 172.21.15.16:0/4067468223 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b14161490 0x7f4b140b4300 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f4b0c00cc30 tx=0x7f4b0c00cf40 comp rx=0 tx=0).stop 2023-10-16T12:12:07.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.549+0000 7f4b1afc6700 1 -- 172.21.15.16:0/4067468223 shutdown_connections 2023-10-16T12:12:07.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.549+0000 7f4b1afc6700 1 --2- 172.21.15.16:0/4067468223 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f4af806af00 0x7f4af806d3c0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:07.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.549+0000 7f4b1afc6700 1 --2- 172.21.15.16:0/4067468223 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b14161490 0x7f4b140b4300 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:07.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.549+0000 7f4b1afc6700 1 --2- 172.21.15.16:0/4067468223 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4b1415d750 0x7f4b141f2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:07.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.549+0000 7f4b1afc6700 1 --2- 172.21.15.16:0/4067468223 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4b1415db30 0x7f4b140b6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:07.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.549+0000 7f4b1afc6700 1 -- 172.21.15.16:0/4067468223 >> 172.21.15.16:0/4067468223 conn(0x7f4b140d14b0 msgr2=0x7f4b1415fc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:07.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.549+0000 7f4b1afc6700 1 -- 172.21.15.16:0/4067468223 shutdown_connections 2023-10-16T12:12:07.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:07.549+0000 7f4b1afc6700 1 -- 172.21.15.16:0/4067468223 wait complete. 2023-10-16T12:12:07.563 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-16T12:12:07.563 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T12:12:07.563 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi016.front.sepia.ceph.com 2023-10-16T12:12:07.564 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-16T12:12:07.564 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-16T12:12:07.564 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-16T12:12:07.564 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-16T12:12:07.565 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T12:12:07.565 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T12:12:07.566 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:12:07.566 DEBUG:teuthology.orchestra.run.smithi016:> ip addr 2023-10-16T12:12:07.584 INFO:teuthology.orchestra.run.smithi016.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T12:12:07.584 INFO:teuthology.orchestra.run.smithi016.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T12:12:07.584 INFO:teuthology.orchestra.run.smithi016.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T12:12:07.584 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:12:07.584 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 ::1/128 scope host 2023-10-16T12:12:07.584 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:12:07.585 INFO:teuthology.orchestra.run.smithi016.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T12:12:07.585 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e0 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:12:07.585 INFO:teuthology.orchestra.run.smithi016.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T12:12:07.585 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e1 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:12:07.585 INFO:teuthology.orchestra.run.smithi016.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T12:12:07.586 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:64 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:12:07.586 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.21.15.16/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-16T12:12:07.586 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft 42209sec preferred_lft 42209sec 2023-10-16T12:12:07.586 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::ec4:7aff:febd:1564/64 scope link 2023-10-16T12:12:07.586 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:12:07.587 INFO:teuthology.orchestra.run.smithi016.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T12:12:07.587 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:65 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:12:07.587 INFO:teuthology.orchestra.run.smithi016.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T12:12:07.587 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 36:cf:ea:25:db:f7 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:12:07.587 INFO:teuthology.orchestra.run.smithi016.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-16T12:12:07.587 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:12:07.588 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::cfc:c5ff:fe52:a786/64 scope link 2023-10-16T12:12:07.588 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:12:07.588 INFO:teuthology.orchestra.run.smithi016.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T12:12:07.588 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 6e:62:9d:a4:43:23 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T12:12:07.588 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::6c62:9dff:fea4:4323/64 scope link 2023-10-16T12:12:07.588 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:12:07.589 INFO:teuthology.orchestra.run.smithi016.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T12:12:07.589 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 36:cf:ea:25:db:f7 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-16T12:12:07.589 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::34cf:eaff:fe25:dbf7/64 scope link 2023-10-16T12:12:07.589 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:12:07.589 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:12:07.590 DEBUG:teuthology.orchestra.run.smithi016:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T12:12:07.649 INFO:teuthology.orchestra.run.smithi016.stdout:1 2023-10-16T12:12:07.651 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:12:07.651 DEBUG:teuthology.orchestra.run.smithi016:> route 2023-10-16T12:12:07.710 INFO:teuthology.orchestra.run.smithi016.stdout:Kernel IP routing table 2023-10-16T12:12:07.710 INFO:teuthology.orchestra.run.smithi016.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T12:12:07.710 INFO:teuthology.orchestra.run.smithi016.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-16T12:12:07.710 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-16T12:12:07.711 INFO:teuthology.orchestra.run.smithi016.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T12:12:07.711 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:12:07.711 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec bash -c ' 2023-10-16T12:12:07.712 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2023-10-16T12:12:07.712 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-16T12:12:07.712 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-16T12:12:07.713 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-16T12:12:07.713 DEBUG:teuthology.orchestra.run.smithi016:> ') 2023-10-16T12:12:07.817 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:12:07.818 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2023-10-16T12:12:07.835 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-16T12:12:07.835 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-16T12:12:07.835 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T12:12:07.836 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:12:07.836 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set brx.0 up 2023-10-16T12:12:07.902 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T12:12:07.903 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-16T12:12:07.959 INFO:teuthology.orchestra.run.smithi016.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-16T12:12:07.960 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T12:12:07.960 DEBUG:teuthology.orchestra.run.smithi016:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-16T12:12:08.016 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-16T12:12:08.016 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:12:08.017 DEBUG:teuthology.orchestra.run.smithi016:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T12:12:08.103 INFO:teuthology.orchestra.run.smithi016.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T12:12:08.103 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T12:12:08.103 INFO:teuthology.orchestra.run.smithi016.stdout:invalid new device string format 2023-10-16T12:12:08.112 INFO:teuthology.orchestra.run.smithi016.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T12:12:08.112 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T12:12:08.112 INFO:teuthology.orchestra.run.smithi016.stdout:invalid new device string format 2023-10-16T12:12:08.113 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: resolved to: "172.21.15.16:3300,172.21.15.49:3300,172.21.15.49:3301" 2023-10-16T12:12:08.113 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T12:12:08.178 INFO:teuthology.orchestra.run.smithi016.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T12:12:08.178 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T12:12:08.179 INFO:teuthology.orchestra.run.smithi016.stdout:invalid new device string format 2023-10-16T12:12:08.179 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: resolved to: "172.21.15.16:3300,172.21.15.49:3300,172.21.15.49:3301" 2023-10-16T12:12:08.179 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T12:12:08.179 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: resolved to: "172.21.15.16:6789,172.21.15.49:6789,172.21.15.49:6790" 2023-10-16T12:12:08.179 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.16:6789,172.21.15.49:6789,172.21.15.49:6790:/ 2023-10-16T12:12:08.180 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=5b28dfa1-b21f-474a-9b2e-9fb77af58254" will pass to kernel 2023-10-16T12:12:08.181 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-16T12:12:08.181 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:12:08.181 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-16T12:12:08.209 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:12:08.210 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T12:12:08.283 INFO:teuthology.orchestra.run.smithi016.stdout:client5469 2023-10-16T12:12:08.283 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:12:08.284 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:12:08.341 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-16T12:12:08.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.505+0000 7fb1b9f2e700 1 Processor -- start 2023-10-16T12:12:08.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.505+0000 7fb1b9f2e700 1 -- start start 2023-10-16T12:12:08.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.505+0000 7fb1b9f2e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb1b40cc8a0 0x7fb1b415e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:08.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.505+0000 7fb1b9f2e700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1b4161c30 con 0x7fb1b415b280 2023-10-16T12:12:08.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.505+0000 7fb1b9f2e700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1b4161f10 con 0x7fb1b406a7e0 2023-10-16T12:12:08.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.505+0000 7fb1b9f2e700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1b41621f0 con 0x7fb1b40cc8a0 2023-10-16T12:12:08.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.506+0000 7fb1b2ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb1b40cc8a0 0x7fb1b415e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:08.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.506+0000 7fb1b37fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb1b415b280 0x7fb1b41613e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:53260/0 (socket says 172.21.15.16:53260) 2023-10-16T12:12:08.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.506+0000 7fb1b37fe700 1 -- 172.21.15.16:0/2361659255 learned_addr learned my addr 172.21.15.16:0/2361659255 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:12:08.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.506+0000 7fb1b2ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb1b40cc8a0 0x7fb1b415e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35756/0 (socket says 172.21.15.16:35756) 2023-10-16T12:12:08.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.506+0000 7fb1b2ffd700 1 -- 172.21.15.16:0/2361659255 >> v1:172.21.15.49:6789/0 conn(0x7fb1b406a7e0 legacy=0x7fb1b40d02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:08.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.506+0000 7fb1b2ffd700 1 -- 172.21.15.16:0/2361659255 >> v1:172.21.15.16:6789/0 conn(0x7fb1b415b280 legacy=0x7fb1b41613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:08.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.506+0000 7fb1b27fc700 1 -- 172.21.15.16:0/2361659255 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3786320173 0 0) 0x7fb1b4161c30 con 0x7fb1b415b280 2023-10-16T12:12:08.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.506+0000 7fb1b2ffd700 1 -- 172.21.15.16:0/2361659255 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1b415df00 con 0x7fb1b40cc8a0 2023-10-16T12:12:08.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.506+0000 7fb1b2ffd700 1 --2- 172.21.15.16:0/2361659255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1b40cc8a0 0x7fb1b415e700 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fb1a8009840 tx=0x7fb1a8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44f996722823204d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:08.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.506+0000 7fb1b27fc700 1 -- 172.21.15.16:0/2361659255 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1a8004230 con 0x7fb1b40cc8a0 2023-10-16T12:12:08.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.506+0000 7fb1b27fc700 1 -- 172.21.15.16:0/2361659255 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1a8004390 con 0x7fb1b40cc8a0 2023-10-16T12:12:08.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.506+0000 7fb1b27fc700 1 -- 172.21.15.16:0/2361659255 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1a80036a0 con 0x7fb1b40cc8a0 2023-10-16T12:12:08.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.507+0000 7fb1b9f2e700 1 -- 172.21.15.16:0/2361659255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1b40cc8a0 msgr2=0x7fb1b415e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:08.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.507+0000 7fb1b9f2e700 1 --2- 172.21.15.16:0/2361659255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1b40cc8a0 0x7fb1b415e700 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fb1a8009840 tx=0x7fb1a8009b50 comp rx=0 tx=0).stop 2023-10-16T12:12:08.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.507+0000 7fb1b9f2e700 1 -- 172.21.15.16:0/2361659255 shutdown_connections 2023-10-16T12:12:08.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.507+0000 7fb1b9f2e700 1 --2- 172.21.15.16:0/2361659255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1b40cc8a0 0x7fb1b415e700 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:08.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.507+0000 7fb1b9f2e700 1 -- 172.21.15.16:0/2361659255 >> 172.21.15.16:0/2361659255 conn(0x7fb1b40b1fd0 msgr2=0x7fb1b40b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:08.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.507+0000 7fb1b9f2e700 1 -- 172.21.15.16:0/2361659255 shutdown_connections 2023-10-16T12:12:08.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.507+0000 7fb1b9f2e700 1 -- 172.21.15.16:0/2361659255 wait complete. 2023-10-16T12:12:08.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.508+0000 7fb1b9f2e700 1 Processor -- start 2023-10-16T12:12:08.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.508+0000 7fb1b9f2e700 1 -- start start 2023-10-16T12:12:08.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.509+0000 7fb1b9f2e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb1b406a7e0 0x7fb1b40af4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:08.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.509+0000 7fb1b9f2e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1b40cc8a0 0x7fb1b40c5d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:08.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.509+0000 7fb1b9f2e700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb1b415b280 0x7fb1b40c6290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:08.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.509+0000 7fb1b9f2e700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1b4160dd0 con 0x7fb1b40cc8a0 2023-10-16T12:12:08.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.509+0000 7fb1b9f2e700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1b40afa10 con 0x7fb1b415b280 2023-10-16T12:12:08.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.509+0000 7fb1b9f2e700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1b40c67d0 con 0x7fb1b406a7e0 2023-10-16T12:12:08.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.509+0000 7fb1b2ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1b40cc8a0 0x7fb1b40c5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:08.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.509+0000 7fb1b2ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1b40cc8a0 0x7fb1b40c5d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35762/0 (socket says 172.21.15.16:35762) 2023-10-16T12:12:08.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.509+0000 7fb1b2ffd700 1 -- 172.21.15.16:0/3862524801 learned_addr learned my addr 172.21.15.16:0/3862524801 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:12:08.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.509+0000 7fb1b37fe700 1 --2- 172.21.15.16:0/3862524801 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb1b406a7e0 0x7fb1b40af4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:08.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.509+0000 7fb1b3fff700 1 --2- 172.21.15.16:0/3862524801 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb1b415b280 0x7fb1b40c6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:08.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.509+0000 7fb1b2ffd700 1 -- 172.21.15.16:0/3862524801 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb1b406a7e0 msgr2=0x7fb1b40af4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:08.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.509+0000 7fb1b2ffd700 1 --2- 172.21.15.16:0/3862524801 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb1b406a7e0 0x7fb1b40af4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:08.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.509+0000 7fb1b2ffd700 1 -- 172.21.15.16:0/3862524801 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb1b415b280 msgr2=0x7fb1b40c6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:08.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.509+0000 7fb1b2ffd700 1 --2- 172.21.15.16:0/3862524801 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb1b415b280 0x7fb1b40c6290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:08.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.509+0000 7fb1b2ffd700 1 -- 172.21.15.16:0/3862524801 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1a80094a0 con 0x7fb1b40cc8a0 2023-10-16T12:12:08.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.509+0000 7fb1b37fe700 1 --2- 172.21.15.16:0/3862524801 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb1b406a7e0 0x7fb1b40af4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:12:08.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.509+0000 7fb1b3fff700 1 --2- 172.21.15.16:0/3862524801 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb1b415b280 0x7fb1b40c6290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:12:08.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.510+0000 7fb1b2ffd700 1 --2- 172.21.15.16:0/3862524801 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1b40cc8a0 0x7fb1b40c5d50 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fb1a80090b0 tx=0x7fb1a8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:08.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.510+0000 7fb19bfff700 1 -- 172.21.15.16:0/3862524801 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1a8011830 con 0x7fb1b40cc8a0 2023-10-16T12:12:08.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.510+0000 7fb19bfff700 1 -- 172.21.15.16:0/3862524801 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1a801e070 con 0x7fb1b40cc8a0 2023-10-16T12:12:08.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.510+0000 7fb1b9f2e700 1 -- 172.21.15.16:0/3862524801 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1b40b0e30 con 0x7fb1b40cc8a0 2023-10-16T12:12:08.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.510+0000 7fb19bfff700 1 -- 172.21.15.16:0/3862524801 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1a8011bb0 con 0x7fb1b40cc8a0 2023-10-16T12:12:08.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.510+0000 7fb1b9f2e700 1 -- 172.21.15.16:0/3862524801 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1b415cce0 con 0x7fb1b40cc8a0 2023-10-16T12:12:08.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.511+0000 7fb1b9f2e700 1 -- 172.21.15.16:0/3862524801 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1b404ebb0 con 0x7fb1b40cc8a0 2023-10-16T12:12:08.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.511+0000 7fb19bfff700 1 -- 172.21.15.16:0/3862524801 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fb1a8011d10 con 0x7fb1b40cc8a0 2023-10-16T12:12:08.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.512+0000 7fb19bfff700 1 --2- 172.21.15.16:0/3862524801 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fb1a006af00 0x7fb1a006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:08.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.512+0000 7fb19bfff700 1 -- 172.21.15.16:0/3862524801 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fb1a808aab0 con 0x7fb1b40cc8a0 2023-10-16T12:12:08.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.512+0000 7fb1b37fe700 1 --2- 172.21.15.16:0/3862524801 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fb1a006af00 0x7fb1a006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:08.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.512+0000 7fb1b37fe700 1 --2- 172.21.15.16:0/3862524801 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fb1a006af00 0x7fb1a006d3c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fb1a4000f60 tx=0x7fb1a4007d30 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:08.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.516+0000 7fb19bfff700 1 -- 172.21.15.16:0/3862524801 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb1a8055a10 con 0x7fb1b40cc8a0 2023-10-16T12:12:08.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.748+0000 7fb1b9f2e700 1 -- 172.21.15.16:0/3862524801 --> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fb1b415ec40 con 0x7fb1a006af00 2023-10-16T12:12:08.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.750+0000 7fb19bfff700 1 -- 172.21.15.16:0/3862524801 <== mgr.4104 v2:172.21.15.16:6832/365921436 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fb1a4004870 con 0x7fb1a006af00 2023-10-16T12:12:08.752 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2023-10-16T12:12:08.752 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:12:08.752 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cephfs" 2023-10-16T12:12:08.753 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:12:08.753 INFO:teuthology.orchestra.run.smithi016.stdout:] 2023-10-16T12:12:08.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.752+0000 7fb1b9f2e700 1 -- 172.21.15.16:0/3862524801 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fb1a006af00 msgr2=0x7fb1a006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:08.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.752+0000 7fb1b9f2e700 1 --2- 172.21.15.16:0/3862524801 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fb1a006af00 0x7fb1a006d3c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fb1a4000f60 tx=0x7fb1a4007d30 comp rx=0 tx=0).stop 2023-10-16T12:12:08.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.752+0000 7fb1b9f2e700 1 -- 172.21.15.16:0/3862524801 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1b40cc8a0 msgr2=0x7fb1b40c5d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:08.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.752+0000 7fb1b9f2e700 1 --2- 172.21.15.16:0/3862524801 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1b40cc8a0 0x7fb1b40c5d50 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fb1a80090b0 tx=0x7fb1a8004750 comp rx=0 tx=0).stop 2023-10-16T12:12:08.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.752+0000 7fb1b9f2e700 1 -- 172.21.15.16:0/3862524801 shutdown_connections 2023-10-16T12:12:08.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.752+0000 7fb1b9f2e700 1 --2- 172.21.15.16:0/3862524801 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fb1a006af00 0x7fb1a006d3c0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:08.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.752+0000 7fb1b9f2e700 1 --2- 172.21.15.16:0/3862524801 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb1b415b280 0x7fb1b40c6290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:08.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.752+0000 7fb1b9f2e700 1 --2- 172.21.15.16:0/3862524801 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb1b406a7e0 0x7fb1b40af4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:08.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.752+0000 7fb1b9f2e700 1 --2- 172.21.15.16:0/3862524801 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1b40cc8a0 0x7fb1b40c5d50 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:08.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.752+0000 7fb1b9f2e700 1 -- 172.21.15.16:0/3862524801 >> 172.21.15.16:0/3862524801 conn(0x7fb1b40b1fd0 msgr2=0x7fb1b406abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:08.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.752+0000 7fb1b9f2e700 1 -- 172.21.15.16:0/3862524801 shutdown_connections 2023-10-16T12:12:08.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.752+0000 7fb1b9f2e700 1 -- 172.21.15.16:0/3862524801 wait complete. 2023-10-16T12:12:08.766 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-16T12:12:08.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.891+0000 7fa11a033700 1 Processor -- start 2023-10-16T12:12:08.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.891+0000 7fa11a033700 1 -- start start 2023-10-16T12:12:08.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.892+0000 7fa11a033700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fa11406aef0 0x7fa1140ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:08.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.892+0000 7fa11a033700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa11406da30 0x7fa1140adf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:08.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.892+0000 7fa11a033700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1141613c0 con 0x7fa114160fe0 2023-10-16T12:12:08.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.892+0000 7fa11a033700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1140cf680 con 0x7fa11406da30 2023-10-16T12:12:08.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.892+0000 7fa11a033700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1140adb00 con 0x7fa11406aef0 2023-10-16T12:12:08.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.892+0000 7fa113fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa11406da30 0x7fa1140adf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:08.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.892+0000 7fa1137fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa114160fe0 0x7fa11406a7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:53274/0 (socket says 172.21.15.16:53274) 2023-10-16T12:12:08.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.892+0000 7fa1137fe700 1 -- 172.21.15.16:0/1510170295 learned_addr learned my addr 172.21.15.16:0/1510170295 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:12:08.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.893+0000 7fa112ffd700 1 --2- 172.21.15.16:0/1510170295 >> v2:172.21.15.49:3300/0 conn(0x7fa11406aef0 0x7fa1140ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:08.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.893+0000 7fa113fff700 1 -- 172.21.15.16:0/1510170295 >> v2:172.21.15.49:3300/0 conn(0x7fa11406aef0 msgr2=0x7fa1140ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:08.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.893+0000 7fa113fff700 1 --2- 172.21.15.16:0/1510170295 >> v2:172.21.15.49:3300/0 conn(0x7fa11406aef0 0x7fa1140ac8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:08.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.893+0000 7fa113fff700 1 -- 172.21.15.16:0/1510170295 >> v1:172.21.15.16:6789/0 conn(0x7fa114160fe0 legacy=0x7fa11406a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:08.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.893+0000 7fa113fff700 1 -- 172.21.15.16:0/1510170295 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1140acef0 con 0x7fa11406da30 2023-10-16T12:12:08.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.893+0000 7fa1127fc700 1 -- 172.21.15.16:0/1510170295 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3216846095 0 0) 0x7fa0fc001290 con 0x7fa114160fe0 2023-10-16T12:12:08.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.893+0000 7fa113fff700 1 --2- 172.21.15.16:0/1510170295 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa11406da30 0x7fa1140adf90 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fa1140cf330 tx=0x7fa10c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d51609ad5f55ffb7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:08.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.893+0000 7fa1127fc700 1 -- 172.21.15.16:0/1510170295 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa10c00e070 con 0x7fa11406da30 2023-10-16T12:12:08.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.893+0000 7fa1127fc700 1 -- 172.21.15.16:0/1510170295 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa10c00bbd0 con 0x7fa11406da30 2023-10-16T12:12:08.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.893+0000 7fa1127fc700 1 -- 172.21.15.16:0/1510170295 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa10c012430 con 0x7fa11406da30 2023-10-16T12:12:08.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.893+0000 7fa11a033700 1 -- 172.21.15.16:0/1510170295 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa11406da30 msgr2=0x7fa1140adf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:08.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.893+0000 7fa11a033700 1 --2- 172.21.15.16:0/1510170295 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa11406da30 0x7fa1140adf90 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fa1140cf330 tx=0x7fa10c00b330 comp rx=0 tx=0).stop 2023-10-16T12:12:08.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.894+0000 7fa11a033700 1 -- 172.21.15.16:0/1510170295 shutdown_connections 2023-10-16T12:12:08.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.894+0000 7fa11a033700 1 --2- 172.21.15.16:0/1510170295 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa11406da30 0x7fa1140adf90 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:08.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.894+0000 7fa11a033700 1 --2- 172.21.15.16:0/1510170295 >> v2:172.21.15.49:3300/0 conn(0x7fa11406aef0 0x7fa1140ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:08.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.894+0000 7fa11a033700 1 -- 172.21.15.16:0/1510170295 >> 172.21.15.16:0/1510170295 conn(0x7fa1140b1ed0 msgr2=0x7fa1140b22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:08.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.894+0000 7fa11a033700 1 -- 172.21.15.16:0/1510170295 shutdown_connections 2023-10-16T12:12:08.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.894+0000 7fa11a033700 1 -- 172.21.15.16:0/1510170295 wait complete. 2023-10-16T12:12:08.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.895+0000 7fa11a033700 1 Processor -- start 2023-10-16T12:12:08.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.895+0000 7fa11a033700 1 -- start start 2023-10-16T12:12:08.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.896+0000 7fa11a033700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa11406aef0 0x7fa1140c3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:08.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.896+0000 7fa11a033700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa11406da30 0x7fa1140b3500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:08.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.896+0000 7fa11a033700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa114160fe0 0x7fa1140b52c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:08.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.896+0000 7fa11a033700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1141e6270 con 0x7fa11406da30 2023-10-16T12:12:08.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.896+0000 7fa11a033700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1140b5e10 con 0x7fa11406aef0 2023-10-16T12:12:08.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.896+0000 7fa11a033700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1141e6480 con 0x7fa114160fe0 2023-10-16T12:12:08.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.897+0000 7fa112ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa11406da30 0x7fa1140b3500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:08.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.897+0000 7fa112ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa11406da30 0x7fa1140b3500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35776/0 (socket says 172.21.15.16:35776) 2023-10-16T12:12:08.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.897+0000 7fa112ffd700 1 -- 172.21.15.16:0/3842302322 learned_addr learned my addr 172.21.15.16:0/3842302322 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:12:08.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.897+0000 7fa1137fe700 1 --2- 172.21.15.16:0/3842302322 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa11406aef0 0x7fa1140c3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:08.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.897+0000 7fa113fff700 1 --2- 172.21.15.16:0/3842302322 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa114160fe0 0x7fa1140b52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:08.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.897+0000 7fa112ffd700 1 -- 172.21.15.16:0/3842302322 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa114160fe0 msgr2=0x7fa1140b52c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:08.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.897+0000 7fa112ffd700 1 --2- 172.21.15.16:0/3842302322 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa114160fe0 0x7fa1140b52c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:08.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.897+0000 7fa112ffd700 1 -- 172.21.15.16:0/3842302322 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa11406aef0 msgr2=0x7fa1140c3bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:08.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.897+0000 7fa112ffd700 1 --2- 172.21.15.16:0/3842302322 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa11406aef0 0x7fa1140c3bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:08.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.897+0000 7fa112ffd700 1 -- 172.21.15.16:0/3842302322 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa10c009a70 con 0x7fa11406da30 2023-10-16T12:12:08.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.897+0000 7fa112ffd700 1 --2- 172.21.15.16:0/3842302322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa11406da30 0x7fa1140b3500 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fa10400c7f0 tx=0x7fa10400cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:08.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.897+0000 7fa10bfff700 1 -- 172.21.15.16:0/3842302322 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa104011800 con 0x7fa11406da30 2023-10-16T12:12:08.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.897+0000 7fa10bfff700 1 -- 172.21.15.16:0/3842302322 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa104011960 con 0x7fa11406da30 2023-10-16T12:12:08.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.897+0000 7fa11a033700 1 -- 172.21.15.16:0/3842302322 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1141e6760 con 0x7fa11406da30 2023-10-16T12:12:08.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.897+0000 7fa10bfff700 1 -- 172.21.15.16:0/3842302322 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa104011cf0 con 0x7fa11406da30 2023-10-16T12:12:08.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.897+0000 7fa11a033700 1 -- 172.21.15.16:0/3842302322 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa11415f960 con 0x7fa11406da30 2023-10-16T12:12:08.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.898+0000 7fa11a033700 1 -- 172.21.15.16:0/3842302322 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa11404ebb0 con 0x7fa11406da30 2023-10-16T12:12:08.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.899+0000 7fa10bfff700 1 -- 172.21.15.16:0/3842302322 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fa104018360 con 0x7fa11406da30 2023-10-16T12:12:08.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.899+0000 7fa10bfff700 1 --2- 172.21.15.16:0/3842302322 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fa0f406af00 0x7fa0f406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:08.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.899+0000 7fa1137fe700 1 --2- 172.21.15.16:0/3842302322 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fa0f406af00 0x7fa0f406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:08.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.899+0000 7fa10bfff700 1 -- 172.21.15.16:0/3842302322 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fa104089700 con 0x7fa11406da30 2023-10-16T12:12:08.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.900+0000 7fa1137fe700 1 --2- 172.21.15.16:0/3842302322 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fa0f406af00 0x7fa0f406d3c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa0fc001620 tx=0x7fa0fc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:08.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:08.904+0000 7fa10bfff700 1 -- 172.21.15.16:0/3842302322 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa104054680 con 0x7fa11406da30 2023-10-16T12:12:09.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:09.173+0000 7fa11a033700 1 -- 172.21.15.16:0/3842302322 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7fa11406ca50 con 0x7fa11406da30 2023-10-16T12:12:10.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.150+0000 7fa10bfff700 1 -- 172.21.15.16:0/3842302322 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v6) v1 ==== 92+0+0 (secure 0 0 0) 0x7fa104057ca0 con 0x7fa11406da30 2023-10-16T12:12:10.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.154+0000 7fa11a033700 1 -- 172.21.15.16:0/3842302322 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fa0f406af00 msgr2=0x7fa0f406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:10.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.154+0000 7fa11a033700 1 --2- 172.21.15.16:0/3842302322 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fa0f406af00 0x7fa0f406d3c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa0fc001620 tx=0x7fa0fc009040 comp rx=0 tx=0).stop 2023-10-16T12:12:10.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.154+0000 7fa11a033700 1 -- 172.21.15.16:0/3842302322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa11406da30 msgr2=0x7fa1140b3500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:10.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.154+0000 7fa11a033700 1 --2- 172.21.15.16:0/3842302322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa11406da30 0x7fa1140b3500 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fa10400c7f0 tx=0x7fa10400cbb0 comp rx=0 tx=0).stop 2023-10-16T12:12:10.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.154+0000 7fa11a033700 1 -- 172.21.15.16:0/3842302322 shutdown_connections 2023-10-16T12:12:10.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.154+0000 7fa11a033700 1 --2- 172.21.15.16:0/3842302322 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fa0f406af00 0x7fa0f406d3c0 secure :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa0fc001620 tx=0x7fa0fc009040 comp rx=0 tx=0).stop 2023-10-16T12:12:10.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.154+0000 7fa11a033700 1 --2- 172.21.15.16:0/3842302322 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa114160fe0 0x7fa1140b52c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:10.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.154+0000 7fa11a033700 1 --2- 172.21.15.16:0/3842302322 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa11406aef0 0x7fa1140c3bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:10.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.154+0000 7fa11a033700 1 --2- 172.21.15.16:0/3842302322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa11406da30 0x7fa1140b3500 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:10.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.154+0000 7fa11a033700 1 -- 172.21.15.16:0/3842302322 >> 172.21.15.16:0/3842302322 conn(0x7fa1140b1ed0 msgr2=0x7fa114153b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:10.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.155+0000 7fa11a033700 1 -- 172.21.15.16:0/3842302322 shutdown_connections 2023-10-16T12:12:10.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.155+0000 7fa11a033700 1 -- 172.21.15.16:0/3842302322 wait complete. 2023-10-16T12:12:10.174 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-16T12:12:10.303 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:12:10.302+0000 7f6483603200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T12:12:10.307 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:12:10.305+0000 7ffab6438200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T12:12:10.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.309+0000 7fcc744c1700 1 Processor -- start 2023-10-16T12:12:10.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.309+0000 7fcc744c1700 1 -- start start 2023-10-16T12:12:10.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.311+0000 7fcc744c1700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fcc6c0b8fa0 0x7fcc6c0b2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:10.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.311+0000 7fcc744c1700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcc6c0ce350 0x7fcc6c0ce780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:10.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.311+0000 7fcc744c1700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc6c0cecc0 con 0x7fcc6c15d8d0 2023-10-16T12:12:10.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.311+0000 7fcc744c1700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc6c0d0bb0 con 0x7fcc6c0b8fa0 2023-10-16T12:12:10.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.311+0000 7fcc744c1700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc6c15dcb0 con 0x7fcc6c0ce350 2023-10-16T12:12:10.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.311+0000 7fcc7225d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fcc6c15d8d0 0x7fcc6c0b6720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:53314/0 (socket says 172.21.15.16:53314) 2023-10-16T12:12:10.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.311+0000 7fcc7225d700 1 -- 172.21.15.16:0/1833658998 learned_addr learned my addr 172.21.15.16:0/1833658998 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:12:10.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.311+0000 7fcc72a5e700 1 --2- 172.21.15.16:0/1833658998 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcc6c0ce350 0x7fcc6c0ce780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:10.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.311+0000 7fcc71a5c700 1 --2- 172.21.15.16:0/1833658998 >> v2:172.21.15.49:3300/0 conn(0x7fcc6c0b8fa0 0x7fcc6c0b2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:10.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.311+0000 7fcc7125b700 1 -- 172.21.15.16:0/1833658998 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 504832982 0 0) 0x7fcc6c0cecc0 con 0x7fcc6c15d8d0 2023-10-16T12:12:10.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.311+0000 7fcc7125b700 1 -- 172.21.15.16:0/1833658998 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc580035b0 con 0x7fcc6c15d8d0 2023-10-16T12:12:10.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.311+0000 7fcc72a5e700 1 -- 172.21.15.16:0/1833658998 >> v2:172.21.15.49:3300/0 conn(0x7fcc6c0b8fa0 msgr2=0x7fcc6c0b2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:10.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.311+0000 7fcc72a5e700 1 --2- 172.21.15.16:0/1833658998 >> v2:172.21.15.49:3300/0 conn(0x7fcc6c0b8fa0 0x7fcc6c0b2830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:10.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.311+0000 7fcc72a5e700 1 -- 172.21.15.16:0/1833658998 >> v1:172.21.15.16:6789/0 conn(0x7fcc6c15d8d0 legacy=0x7fcc6c0b6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:10.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.311+0000 7fcc72a5e700 1 -- 172.21.15.16:0/1833658998 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc6c0b2f30 con 0x7fcc6c0ce350 2023-10-16T12:12:10.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.311+0000 7fcc71a5c700 1 --2- 172.21.15.16:0/1833658998 >> v2:172.21.15.49:3300/0 conn(0x7fcc6c0b8fa0 0x7fcc6c0b2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:12:10.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.312+0000 7fcc72a5e700 1 --2- 172.21.15.16:0/1833658998 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcc6c0ce350 0x7fcc6c0ce780 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fcc64009d90 tx=0x7fcc6400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b58746a4b259afe server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:10.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.312+0000 7fcc7125b700 1 -- 172.21.15.16:0/1833658998 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc6400e070 con 0x7fcc6c0ce350 2023-10-16T12:12:10.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.312+0000 7fcc7125b700 1 -- 172.21.15.16:0/1833658998 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc6400bb20 con 0x7fcc6c0ce350 2023-10-16T12:12:10.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.312+0000 7fcc7125b700 1 -- 172.21.15.16:0/1833658998 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc64012430 con 0x7fcc6c0ce350 2023-10-16T12:12:10.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.312+0000 7fcc744c1700 1 -- 172.21.15.16:0/1833658998 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcc6c0ce350 msgr2=0x7fcc6c0ce780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:10.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.312+0000 7fcc744c1700 1 --2- 172.21.15.16:0/1833658998 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcc6c0ce350 0x7fcc6c0ce780 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fcc64009d90 tx=0x7fcc6400b330 comp rx=0 tx=0).stop 2023-10-16T12:12:10.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.312+0000 7fcc744c1700 1 -- 172.21.15.16:0/1833658998 shutdown_connections 2023-10-16T12:12:10.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.312+0000 7fcc744c1700 1 --2- 172.21.15.16:0/1833658998 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcc6c0ce350 0x7fcc6c0ce780 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:10.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.312+0000 7fcc744c1700 1 --2- 172.21.15.16:0/1833658998 >> v2:172.21.15.49:3300/0 conn(0x7fcc6c0b8fa0 0x7fcc6c0b2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:10.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.312+0000 7fcc744c1700 1 -- 172.21.15.16:0/1833658998 >> 172.21.15.16:0/1833658998 conn(0x7fcc6c072330 msgr2=0x7fcc6c072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:10.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.313+0000 7fcc744c1700 1 -- 172.21.15.16:0/1833658998 shutdown_connections 2023-10-16T12:12:10.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.313+0000 7fcc744c1700 1 -- 172.21.15.16:0/1833658998 wait complete. 2023-10-16T12:12:10.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.314+0000 7fcc744c1700 1 Processor -- start 2023-10-16T12:12:10.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.314+0000 7fcc744c1700 1 -- start start 2023-10-16T12:12:10.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.314+0000 7fcc744c1700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcc6c0b8fa0 0x7fcc6c15a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:10.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.314+0000 7fcc744c1700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcc6c0ce350 0x7fcc6c0c7ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:10.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.314+0000 7fcc744c1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc6c15d8d0 0x7fcc6c159940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:10.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.314+0000 7fcc744c1700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc6c159e80 con 0x7fcc6c15d8d0 2023-10-16T12:12:10.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.315+0000 7fcc744c1700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc6c1539c0 con 0x7fcc6c0b8fa0 2023-10-16T12:12:10.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.315+0000 7fcc744c1700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc6c0caf80 con 0x7fcc6c0ce350 2023-10-16T12:12:10.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.315+0000 7fcc72a5e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc6c15d8d0 0x7fcc6c159940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:10.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.315+0000 7fcc72a5e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc6c15d8d0 0x7fcc6c159940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35818/0 (socket says 172.21.15.16:35818) 2023-10-16T12:12:10.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.315+0000 7fcc72a5e700 1 -- 172.21.15.16:0/3057068744 learned_addr learned my addr 172.21.15.16:0/3057068744 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:12:10.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.315+0000 7fcc7225d700 1 --2- 172.21.15.16:0/3057068744 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcc6c0b8fa0 0x7fcc6c15a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:10.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.315+0000 7fcc71a5c700 1 --2- 172.21.15.16:0/3057068744 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcc6c0ce350 0x7fcc6c0c7ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:10.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.315+0000 7fcc7225d700 1 -- 172.21.15.16:0/3057068744 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcc6c0ce350 msgr2=0x7fcc6c0c7ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:10.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.315+0000 7fcc7225d700 1 --2- 172.21.15.16:0/3057068744 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcc6c0ce350 0x7fcc6c0c7ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:10.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.315+0000 7fcc7225d700 1 -- 172.21.15.16:0/3057068744 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc6c15d8d0 msgr2=0x7fcc6c159940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:10.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.315+0000 7fcc7225d700 1 --2- 172.21.15.16:0/3057068744 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc6c15d8d0 0x7fcc6c159940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:10.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.315+0000 7fcc7225d700 1 -- 172.21.15.16:0/3057068744 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc5c0094a0 con 0x7fcc6c0b8fa0 2023-10-16T12:12:10.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.315+0000 7fcc72a5e700 1 --2- 172.21.15.16:0/3057068744 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc6c15d8d0 0x7fcc6c159940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:12:10.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.315+0000 7fcc7225d700 1 --2- 172.21.15.16:0/3057068744 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcc6c0b8fa0 0x7fcc6c15a000 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fcc6800c430 tx=0x7fcc6800c740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:10.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.315+0000 7fcc62ffd700 1 -- 172.21.15.16:0/3057068744 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc68005d70 con 0x7fcc6c0b8fa0 2023-10-16T12:12:10.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.315+0000 7fcc62ffd700 1 -- 172.21.15.16:0/3057068744 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc680070d0 con 0x7fcc6c0b8fa0 2023-10-16T12:12:10.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.315+0000 7fcc744c1700 1 -- 172.21.15.16:0/3057068744 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc64009a70 con 0x7fcc6c0b8fa0 2023-10-16T12:12:10.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.315+0000 7fcc62ffd700 1 -- 172.21.15.16:0/3057068744 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc68007460 con 0x7fcc6c0b8fa0 2023-10-16T12:12:10.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.315+0000 7fcc744c1700 1 -- 172.21.15.16:0/3057068744 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc6c1f3ce0 con 0x7fcc6c0b8fa0 2023-10-16T12:12:10.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.316+0000 7fcc744c1700 1 -- 172.21.15.16:0/3057068744 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc6c0c83e0 con 0x7fcc6c0b8fa0 2023-10-16T12:12:10.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.317+0000 7fcc62ffd700 1 -- 172.21.15.16:0/3057068744 <== mon.1 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 6) v1 ==== 89246+0+0 (secure 0 0 0) 0x7fcc680075c0 con 0x7fcc6c0b8fa0 2023-10-16T12:12:10.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.317+0000 7fcc62ffd700 1 --2- 172.21.15.16:0/3057068744 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fcc5006af50 0x7fcc5006d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:10.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.317+0000 7fcc71a5c700 1 -- 172.21.15.16:0/3057068744 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fcc5006af50 msgr2=0x7fcc5006d410 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/365921436 2023-10-16T12:12:10.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.317+0000 7fcc62ffd700 1 -- 172.21.15.16:0/3057068744 <== mon.1 v2:172.21.15.49:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fcc68088870 con 0x7fcc6c0b8fa0 2023-10-16T12:12:10.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.317+0000 7fcc71a5c700 1 --2- 172.21.15.16:0/3057068744 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fcc5006af50 0x7fcc5006d410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:12:10.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.322+0000 7fcc62ffd700 1 -- 172.21.15.16:0/3057068744 <== mon.1 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcc68053870 con 0x7fcc6c0b8fa0 2023-10-16T12:12:10.372 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:12:10.371+0000 7f6483603200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T12:12:10.377 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:12:10.376+0000 7ffab6438200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T12:12:10.447 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:12:10.446+0000 7f6483603200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T12:12:10.452 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:12:10.451+0000 7ffab6438200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T12:12:10.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.518+0000 7fcc71a5c700 1 -- 172.21.15.16:0/3057068744 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fcc5006af50 msgr2=0x7fcc5006d410 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/365921436 2023-10-16T12:12:10.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.518+0000 7fcc71a5c700 1 --2- 172.21.15.16:0/3057068744 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fcc5006af50 0x7fcc5006d410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:12:10.528 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:12:10.527+0000 7f6483603200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T12:12:10.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.529+0000 7fcc744c1700 1 -- 172.21.15.16:0/3057068744 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7fcc6c04ebb0 con 0x7fcc6c0b8fa0 2023-10-16T12:12:10.534 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:12:10.532+0000 7ffab6438200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T12:12:10.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.534+0000 7fcc62ffd700 1 -- 172.21.15.16:0/3057068744 <== mon.1 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v11)=0 v11) v1 ==== 138+0+0 (secure 0 0 0) 0x7fcc68056e90 con 0x7fcc6c0b8fa0 2023-10-16T12:12:10.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.536+0000 7fcc744c1700 1 -- 172.21.15.16:0/3057068744 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fcc5006af50 msgr2=0x7fcc5006d410 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:12:10.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.536+0000 7fcc744c1700 1 --2- 172.21.15.16:0/3057068744 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fcc5006af50 0x7fcc5006d410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:10.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.536+0000 7fcc744c1700 1 -- 172.21.15.16:0/3057068744 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcc6c0b8fa0 msgr2=0x7fcc6c15a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:10.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.536+0000 7fcc744c1700 1 --2- 172.21.15.16:0/3057068744 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcc6c0b8fa0 0x7fcc6c15a000 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fcc6800c430 tx=0x7fcc6800c740 comp rx=0 tx=0).stop 2023-10-16T12:12:10.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.536+0000 7fcc744c1700 1 -- 172.21.15.16:0/3057068744 shutdown_connections 2023-10-16T12:12:10.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.536+0000 7fcc744c1700 1 --2- 172.21.15.16:0/3057068744 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fcc5006af50 0x7fcc5006d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:10.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.536+0000 7fcc744c1700 1 --2- 172.21.15.16:0/3057068744 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc6c15d8d0 0x7fcc6c159940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:10.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.536+0000 7fcc744c1700 1 --2- 172.21.15.16:0/3057068744 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcc6c0b8fa0 0x7fcc6c15a000 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:10.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.536+0000 7fcc744c1700 1 --2- 172.21.15.16:0/3057068744 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcc6c0ce350 0x7fcc6c0c7ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:10.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.536+0000 7fcc744c1700 1 -- 172.21.15.16:0/3057068744 >> 172.21.15.16:0/3057068744 conn(0x7fcc6c072330 msgr2=0x7fcc6c15ccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:10.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.536+0000 7fcc744c1700 1 -- 172.21.15.16:0/3057068744 shutdown_connections 2023-10-16T12:12:10.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.536+0000 7fcc744c1700 1 -- 172.21.15.16:0/3057068744 wait complete. 2023-10-16T12:12:10.550 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-16T12:12:10.676 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:12:10.675+0000 7f6483603200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T12:12:10.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.680+0000 7f3c53385700 1 Processor -- start 2023-10-16T12:12:10.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.680+0000 7f3c53385700 1 -- start start 2023-10-16T12:12:10.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.681+0000 7f3c53385700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3c4c15b2c0 0x7f3c4c161340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:10.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.681+0000 7f3c53385700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f3c4c06e570 0x7f3c4c15e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:10.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.681+0000 7f3c53385700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3c4c06a7e0 0x7f3c4c06ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:10.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.681+0000 7f3c53385700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c4c06b1b0 con 0x7f3c4c06a7e0 2023-10-16T12:12:10.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.681+0000 7f3c53385700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c4c161880 con 0x7f3c4c06e570 2023-10-16T12:12:10.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.681+0000 7f3c53385700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c4c15b6a0 con 0x7f3c4c15b2c0 2023-10-16T12:12:10.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.681+0000 7f3c51922700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3c4c06a7e0 0x7f3c4c06ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:10.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.681+0000 7f3c51922700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3c4c06a7e0 0x7f3c4c06ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35872/0 (socket says 172.21.15.16:35872) 2023-10-16T12:12:10.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.681+0000 7f3c51922700 1 -- 172.21.15.16:0/3962429319 learned_addr learned my addr 172.21.15.16:0/3962429319 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:12:10.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.681+0000 7f3c50920700 1 --2- 172.21.15.16:0/3962429319 >> v2:172.21.15.49:3300/0 conn(0x7f3c4c06e570 0x7f3c4c15e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:10.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.681+0000 7f3c51121700 1 --2- 172.21.15.16:0/3962429319 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3c4c15b2c0 0x7f3c4c161340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:10.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.681+0000 7f3c51922700 1 -- 172.21.15.16:0/3962429319 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3c4c15b2c0 msgr2=0x7f3c4c161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:10.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.681+0000 7f3c51922700 1 --2- 172.21.15.16:0/3962429319 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3c4c15b2c0 0x7f3c4c161340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:10.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.681+0000 7f3c51922700 1 -- 172.21.15.16:0/3962429319 >> v2:172.21.15.49:3300/0 conn(0x7f3c4c06e570 msgr2=0x7f3c4c15e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:10.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.681+0000 7f3c51922700 1 --2- 172.21.15.16:0/3962429319 >> v2:172.21.15.49:3300/0 conn(0x7f3c4c06e570 0x7f3c4c15e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:10.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.681+0000 7f3c51922700 1 -- 172.21.15.16:0/3962429319 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c4c161b20 con 0x7f3c4c06a7e0 2023-10-16T12:12:10.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.681+0000 7f3c50920700 1 --2- 172.21.15.16:0/3962429319 >> v2:172.21.15.49:3300/0 conn(0x7f3c4c06e570 0x7f3c4c15e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:12:10.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.681+0000 7f3c51121700 1 --2- 172.21.15.16:0/3962429319 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3c4c15b2c0 0x7f3c4c161340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:12:10.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.681+0000 7f3c51922700 1 --2- 172.21.15.16:0/3962429319 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c4c06a7e0 0x7f3c4c06ac70 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f3c4c070410 tx=0x7f3c4400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef760f2e21e8b50f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:10.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.681+0000 7f3c43fff700 1 -- 172.21.15.16:0/3962429319 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c4400e070 con 0x7f3c4c06a7e0 2023-10-16T12:12:10.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.682+0000 7f3c43fff700 1 -- 172.21.15.16:0/3962429319 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c4400b910 con 0x7f3c4c06a7e0 2023-10-16T12:12:10.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.682+0000 7f3c43fff700 1 -- 172.21.15.16:0/3962429319 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c4400bcc0 con 0x7f3c4c06a7e0 2023-10-16T12:12:10.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.682+0000 7f3c53385700 1 -- 172.21.15.16:0/3962429319 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c4c06a7e0 msgr2=0x7f3c4c06ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:10.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.682+0000 7f3c53385700 1 --2- 172.21.15.16:0/3962429319 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c4c06a7e0 0x7f3c4c06ac70 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f3c4c070410 tx=0x7f3c4400b330 comp rx=0 tx=0).stop 2023-10-16T12:12:10.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.682+0000 7f3c53385700 1 -- 172.21.15.16:0/3962429319 shutdown_connections 2023-10-16T12:12:10.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.682+0000 7f3c53385700 1 --2- 172.21.15.16:0/3962429319 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c4c06a7e0 0x7f3c4c06ac70 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:10.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.682+0000 7f3c53385700 1 --2- 172.21.15.16:0/3962429319 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3c4c15b2c0 0x7f3c4c161340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:10.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.682+0000 7f3c53385700 1 --2- 172.21.15.16:0/3962429319 >> v2:172.21.15.49:3300/0 conn(0x7f3c4c06e570 0x7f3c4c15e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:10.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.682+0000 7f3c53385700 1 -- 172.21.15.16:0/3962429319 >> 172.21.15.16:0/3962429319 conn(0x7f3c4c0b1f40 msgr2=0x7f3c4c0b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:10.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.682+0000 7f3c53385700 1 -- 172.21.15.16:0/3962429319 shutdown_connections 2023-10-16T12:12:10.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.682+0000 7f3c53385700 1 -- 172.21.15.16:0/3962429319 wait complete. 2023-10-16T12:12:10.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.684+0000 7f3c53385700 1 Processor -- start 2023-10-16T12:12:10.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.684+0000 7f3c53385700 1 -- start start 2023-10-16T12:12:10.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.684+0000 7f3c53385700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c4c06a7e0 0x7f3c4c0c74c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:10.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.684+0000 7f3c53385700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3c4c06e570 0x7f3c4c15e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:10.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.684+0000 7f3c53385700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3c4c15b2c0 0x7f3c4c14fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:10.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.684+0000 7f3c53385700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c4c0b5370 con 0x7f3c4c06a7e0 2023-10-16T12:12:10.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.684+0000 7f3c53385700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c4c0b5650 con 0x7f3c4c15b2c0 2023-10-16T12:12:10.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.684+0000 7f3c53385700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c4c0c7b60 con 0x7f3c4c06e570 2023-10-16T12:12:10.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.684+0000 7f3c51121700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c4c06a7e0 0x7f3c4c0c74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:10.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.684+0000 7f3c51121700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c4c06a7e0 0x7f3c4c0c74c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35874/0 (socket says 172.21.15.16:35874) 2023-10-16T12:12:10.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.684+0000 7f3c51121700 1 -- 172.21.15.16:0/1984191717 learned_addr learned my addr 172.21.15.16:0/1984191717 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:12:10.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.685+0000 7f3c51922700 1 --2- 172.21.15.16:0/1984191717 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3c4c15b2c0 0x7f3c4c14fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:10.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.685+0000 7f3c50920700 1 --2- 172.21.15.16:0/1984191717 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3c4c06e570 0x7f3c4c15e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:10.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.685+0000 7f3c51121700 1 -- 172.21.15.16:0/1984191717 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3c4c06e570 msgr2=0x7f3c4c15e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:10.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.685+0000 7f3c51121700 1 --2- 172.21.15.16:0/1984191717 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3c4c06e570 0x7f3c4c15e4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:10.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.685+0000 7f3c51121700 1 -- 172.21.15.16:0/1984191717 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3c4c15b2c0 msgr2=0x7f3c4c14fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:10.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.685+0000 7f3c51121700 1 --2- 172.21.15.16:0/1984191717 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3c4c15b2c0 0x7f3c4c14fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:10.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.685+0000 7f3c51121700 1 -- 172.21.15.16:0/1984191717 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c3c0094a0 con 0x7f3c4c06a7e0 2023-10-16T12:12:10.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.685+0000 7f3c51121700 1 --2- 172.21.15.16:0/1984191717 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c4c06a7e0 0x7f3c4c0c74c0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f3c4800ebe0 tx=0x7f3c4800eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:10.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.685+0000 7f3c41ffb700 1 -- 172.21.15.16:0/1984191717 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c4800c850 con 0x7f3c4c06a7e0 2023-10-16T12:12:10.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.685+0000 7f3c41ffb700 1 -- 172.21.15.16:0/1984191717 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c4800c9b0 con 0x7f3c4c06a7e0 2023-10-16T12:12:10.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.685+0000 7f3c53385700 1 -- 172.21.15.16:0/1984191717 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c480094a0 con 0x7f3c4c06a7e0 2023-10-16T12:12:10.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.685+0000 7f3c41ffb700 1 -- 172.21.15.16:0/1984191717 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c4800cd40 con 0x7f3c4c06a7e0 2023-10-16T12:12:10.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.685+0000 7f3c53385700 1 -- 172.21.15.16:0/1984191717 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c44009b30 con 0x7f3c4c06a7e0 2023-10-16T12:12:10.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.686+0000 7f3c53385700 1 -- 172.21.15.16:0/1984191717 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c4c0c8490 con 0x7f3c4c06a7e0 2023-10-16T12:12:10.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.687+0000 7f3c41ffb700 1 -- 172.21.15.16:0/1984191717 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 6) v1 ==== 89246+0+0 (secure 0 0 0) 0x7f3c48013470 con 0x7f3c4c06a7e0 2023-10-16T12:12:10.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.687+0000 7f3c41ffb700 1 --2- 172.21.15.16:0/1984191717 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f3c3006af50 0x7f3c3006d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:10.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.687+0000 7f3c50920700 1 -- 172.21.15.16:0/1984191717 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f3c3006af50 msgr2=0x7f3c3006d410 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/365921436 2023-10-16T12:12:10.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.687+0000 7f3c41ffb700 1 -- 172.21.15.16:0/1984191717 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f3c48089ea0 con 0x7f3c4c06a7e0 2023-10-16T12:12:10.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.688+0000 7f3c50920700 1 --2- 172.21.15.16:0/1984191717 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f3c3006af50 0x7f3c3006d410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:12:10.698 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:12:10.688+0000 7ffab6438200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T12:12:10.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.692+0000 7f3c41ffb700 1 -- 172.21.15.16:0/1984191717 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3c48054ef0 con 0x7f3c4c06a7e0 2023-10-16T12:12:10.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.889+0000 7f3c50920700 1 -- 172.21.15.16:0/1984191717 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f3c3006af50 msgr2=0x7f3c3006d410 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/365921436 2023-10-16T12:12:10.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.889+0000 7f3c50920700 1 --2- 172.21.15.16:0/1984191717 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f3c3006af50 0x7f3c3006d410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:12:10.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.898+0000 7f3c53385700 1 -- 172.21.15.16:0/1984191717 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f3c4c0b54b0 con 0x7f3c4c06a7e0 2023-10-16T12:12:10.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.901+0000 7f3c41ffb700 1 -- 172.21.15.16:0/1984191717 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v12)=0 v12) v1 ==== 133+0+0 (secure 0 0 0) 0x7f3c4c0b54b0 con 0x7f3c4c06a7e0 2023-10-16T12:12:10.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.903+0000 7f3c53385700 1 -- 172.21.15.16:0/1984191717 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f3c3006af50 msgr2=0x7f3c3006d410 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:12:10.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.903+0000 7f3c53385700 1 --2- 172.21.15.16:0/1984191717 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f3c3006af50 0x7f3c3006d410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:10.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.903+0000 7f3c53385700 1 -- 172.21.15.16:0/1984191717 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c4c06a7e0 msgr2=0x7f3c4c0c74c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:10.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.903+0000 7f3c53385700 1 --2- 172.21.15.16:0/1984191717 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c4c06a7e0 0x7f3c4c0c74c0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f3c4800ebe0 tx=0x7f3c4800eef0 comp rx=0 tx=0).stop 2023-10-16T12:12:10.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.903+0000 7f3c53385700 1 -- 172.21.15.16:0/1984191717 shutdown_connections 2023-10-16T12:12:10.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.903+0000 7f3c53385700 1 --2- 172.21.15.16:0/1984191717 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7f3c3006af50 0x7f3c3006d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:10.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.903+0000 7f3c53385700 1 --2- 172.21.15.16:0/1984191717 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3c4c15b2c0 0x7f3c4c14fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:10.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.903+0000 7f3c53385700 1 --2- 172.21.15.16:0/1984191717 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c4c06a7e0 0x7f3c4c0c74c0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:10.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.903+0000 7f3c53385700 1 --2- 172.21.15.16:0/1984191717 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3c4c06e570 0x7f3c4c15e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:10.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.903+0000 7f3c53385700 1 -- 172.21.15.16:0/1984191717 >> 172.21.15.16:0/1984191717 conn(0x7f3c4c0b1f40 msgr2=0x7f3c4c0cce00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:10.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.903+0000 7f3c53385700 1 -- 172.21.15.16:0/1984191717 shutdown_connections 2023-10-16T12:12:10.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:10.903+0000 7f3c53385700 1 -- 172.21.15.16:0/1984191717 wait complete. 2023-10-16T12:12:10.918 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:12:10.918 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-10-16T12:12:10.942 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:12:10.942 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir0) 2023-10-16T12:12:11.005 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir0 1m 2023-10-16T12:12:11.019 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:12:11.018+0000 7f6483603200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T12:12:11.038 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:12:11.036+0000 7ffab6438200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T12:12:11.139 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:12:11.138+0000 7f6483603200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T12:12:11.161 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:12:11.159+0000 7ffab6438200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T12:12:11.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:11.167+0000 7fe1d8635700 1 Processor -- start 2023-10-16T12:12:11.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:11.167+0000 7fe1d8635700 1 -- start start 2023-10-16T12:12:11.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:11.168+0000 7fe1d8635700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fe1d015da10 0x7fe1d00d1520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:11.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:11.168+0000 7fe1d8635700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1d00d4540 con 0x7fe1d00d4060 2023-10-16T12:12:11.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:11.168+0000 7fe1d8635700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1d00ce7e0 con 0x7fe1d00d1d20 2023-10-16T12:12:11.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:11.168+0000 7fe1d8635700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1d00cf9b0 con 0x7fe1d015da10 2023-10-16T12:12:11.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:11.168+0000 7fe1d5bd0700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe1d00d4060 0x7fe1d00cf2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:53424/0 (socket says 172.21.15.16:53424) 2023-10-16T12:12:11.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:11.168+0000 7fe1d5bd0700 1 -- 172.21.15.16:0/4173612057 learned_addr learned my addr 172.21.15.16:0/4173612057 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:12:11.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:11.168+0000 7fe1d63d1700 1 --2- 172.21.15.16:0/4173612057 >> v2:172.21.15.49:3300/0 conn(0x7fe1d015da10 0x7fe1d00d1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:11.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:11.168+0000 7fe1d53cf700 1 -- 172.21.15.16:0/4173612057 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2755950531 0 0) 0x7fe1d00d4540 con 0x7fe1d00d4060 2023-10-16T12:12:11.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:11.168+0000 7fe1d53cf700 1 -- 172.21.15.16:0/4173612057 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1bc0035b0 con 0x7fe1d00d4060 2023-10-16T12:12:11.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:11.168+0000 7fe1d53cf700 1 -- 172.21.15.16:0/4173612057 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1400431397 0 0) 0x7fe1d00ce7e0 con 0x7fe1d00d1d20 2023-10-16T12:12:11.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:11.168+0000 7fe1d53cf700 1 -- 172.21.15.16:0/4173612057 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1d00d4540 con 0x7fe1d00d1d20 2023-10-16T12:12:11.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:11.168+0000 7fe1d63d1700 1 -- 172.21.15.16:0/4173612057 >> v1:172.21.15.49:6789/0 conn(0x7fe1d00d1d20 legacy=0x7fe1d006b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:11.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:11.168+0000 7fe1d63d1700 1 -- 172.21.15.16:0/4173612057 >> v1:172.21.15.16:6789/0 conn(0x7fe1d00d4060 legacy=0x7fe1d00cf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:11.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:11.168+0000 7fe1d63d1700 1 -- 172.21.15.16:0/4173612057 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1d00ceac0 con 0x7fe1d015da10 2023-10-16T12:12:11.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:11.169+0000 7fe1d63d1700 1 --2- 172.21.15.16:0/4173612057 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe1d015da10 0x7fe1d00d1520 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fe1cc009820 tx=0x7fe1cc009b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=13577561d85969d2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:11.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:11.169+0000 7fe1d53cf700 1 -- 172.21.15.16:0/4173612057 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1cc005e30 con 0x7fe1d015da10 2023-10-16T12:12:11.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:11.169+0000 7fe1d53cf700 1 -- 172.21.15.16:0/4173612057 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1cc006030 con 0x7fe1d015da10 2023-10-16T12:12:11.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:11.169+0000 7fe1d53cf700 1 -- 172.21.15.16:0/4173612057 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1cc0063c0 con 0x7fe1d015da10 2023-10-16T12:12:11.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:11.169+0000 7fe1d8635700 1 -- 172.21.15.16:0/4173612057 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe1d015da10 msgr2=0x7fe1d00d1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:11.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:11.169+0000 7fe1d8635700 1 --2- 172.21.15.16:0/4173612057 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe1d015da10 0x7fe1d00d1520 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fe1cc009820 tx=0x7fe1cc009b30 comp rx=0 tx=0).stop 2023-10-16T12:12:11.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:11.169+0000 7fe1d8635700 1 -- 172.21.15.16:0/4173612057 shutdown_connections 2023-10-16T12:12:11.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:11.170+0000 7fe1d8635700 1 --2- 172.21.15.16:0/4173612057 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe1d015da10 0x7fe1d00d1520 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:11.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:11.170+0000 7fe1d8635700 1 -- 172.21.15.16:0/4173612057 >> 172.21.15.16:0/4173612057 conn(0x7fe1d009c9f0 msgr2=0x7fe1d00d3740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:11.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:11.170+0000 7fe1d8635700 1 -- 172.21.15.16:0/4173612057 shutdown_connections 2023-10-16T12:12:11.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:11.170+0000 7fe1d8635700 1 -- 172.21.15.16:0/4173612057 wait complete. 2023-10-16T12:12:11.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:11.171+0000 7fe1d8635700 1 Processor -- start 2023-10-16T12:12:11.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:11.171+0000 7fe1d8635700 1 -- start start 2023-10-16T12:12:11.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:11.171+0000 7fe1d8635700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe1d00d1d20 0x7fe1d01f3c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:11.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:11.171+0000 7fe1d8635700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1d00d4060 0x7fe1d01f6600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:11.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:11.171+0000 7fe1d8635700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe1d015da10 0x7fe1d0151a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:11.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:11.172+0000 7fe1d8635700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d00b6400 con 0x7fe1d00d4060 2023-10-16T12:12:11.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:11.172+0000 7fe1d8635700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d01f6e50 con 0x7fe1d015da10 2023-10-16T12:12:11.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:11.172+0000 7fe1d8635700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d01f7130 con 0x7fe1d00d1d20 2023-10-16T12:12:11.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:11.172+0000 7fe1d5bd0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1d00d4060 0x7fe1d01f6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:11.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:11.172+0000 7fe1d5bd0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1d00d4060 0x7fe1d01f6600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35926/0 (socket says 172.21.15.16:35926) 2023-10-16T12:12:11.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:11.172+0000 7fe1d5bd0700 1 -- 172.21.15.16:0/51838310 learned_addr learned my addr 172.21.15.16:0/51838310 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:12:11.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:11.172+0000 7fe1d6bd2700 1 --2- 172.21.15.16:0/51838310 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe1d015da10 0x7fe1d0151a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:11.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:11.172+0000 7fe1d63d1700 1 --2- 172.21.15.16:0/51838310 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe1d00d1d20 0x7fe1d01f3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:11.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:11.172+0000 7fe1d5bd0700 1 -- 172.21.15.16:0/51838310 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe1d00d1d20 msgr2=0x7fe1d01f3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:11.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:11.172+0000 7fe1d5bd0700 1 --2- 172.21.15.16:0/51838310 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe1d00d1d20 0x7fe1d01f3c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:11.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:11.172+0000 7fe1d5bd0700 1 -- 172.21.15.16:0/51838310 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe1d015da10 msgr2=0x7fe1d0151a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:11.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:11.172+0000 7fe1d5bd0700 1 --2- 172.21.15.16:0/51838310 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe1d015da10 0x7fe1d0151a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:11.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:11.172+0000 7fe1d5bd0700 1 -- 172.21.15.16:0/51838310 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1cc009430 con 0x7fe1d00d4060 2023-10-16T12:12:11.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:11.172+0000 7fe1d6bd2700 1 --2- 172.21.15.16:0/51838310 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe1d015da10 0x7fe1d0151a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:12:11.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:11.172+0000 7fe1d5bd0700 1 --2- 172.21.15.16:0/51838310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1d00d4060 0x7fe1d01f6600 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fe1c000c500 tx=0x7fe1c000c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:11.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:11.172+0000 7fe1c6ffd700 1 -- 172.21.15.16:0/51838310 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1c0005d70 con 0x7fe1d00d4060 2023-10-16T12:12:11.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:11.172+0000 7fe1c6ffd700 1 -- 172.21.15.16:0/51838310 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1c00070d0 con 0x7fe1d00d4060 2023-10-16T12:12:11.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:11.172+0000 7fe1d8635700 1 -- 172.21.15.16:0/51838310 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1d00b6670 con 0x7fe1d00d4060 2023-10-16T12:12:11.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:11.172+0000 7fe1c6ffd700 1 -- 172.21.15.16:0/51838310 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1c0007480 con 0x7fe1d00d4060 2023-10-16T12:12:11.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:11.172+0000 7fe1d8635700 1 -- 172.21.15.16:0/51838310 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1d01f7410 con 0x7fe1d00d4060 2023-10-16T12:12:11.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:11.173+0000 7fe1d8635700 1 -- 172.21.15.16:0/51838310 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1d004ebb0 con 0x7fe1d00d4060 2023-10-16T12:12:11.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:11.174+0000 7fe1c6ffd700 1 -- 172.21.15.16:0/51838310 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 6) v1 ==== 89246+0+0 (secure 0 0 0) 0x7fe1c00075e0 con 0x7fe1d00d4060 2023-10-16T12:12:11.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:11.174+0000 7fe1c6ffd700 1 --2- 172.21.15.16:0/51838310 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fe1b406af50 0x7fe1b406d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:11.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:11.174+0000 7fe1d63d1700 1 -- 172.21.15.16:0/51838310 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fe1b406af50 msgr2=0x7fe1b406d410 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/365921436 2023-10-16T12:12:11.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:11.174+0000 7fe1c6ffd700 1 -- 172.21.15.16:0/51838310 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fe1c0088ba0 con 0x7fe1d00d4060 2023-10-16T12:12:11.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:11.174+0000 7fe1d63d1700 1 --2- 172.21.15.16:0/51838310 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fe1b406af50 0x7fe1b406d410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:12:11.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:11.179+0000 7fe1c6ffd700 1 -- 172.21.15.16:0/51838310 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe1c0053bd0 con 0x7fe1d00d4060 2023-10-16T12:12:11.216 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:12:11.215+0000 7f6483603200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T12:12:11.241 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:12:11.239+0000 7ffab6438200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T12:12:11.300 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:12:11.299+0000 7f6483603200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T12:12:11.326 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:12:11.325+0000 7ffab6438200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T12:12:11.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:11.375+0000 7fe1d63d1700 1 -- 172.21.15.16:0/51838310 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fe1b406af50 msgr2=0x7fe1b406d410 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/365921436 2023-10-16T12:12:11.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:11.375+0000 7fe1d63d1700 1 --2- 172.21.15.16:0/51838310 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fe1b406af50 0x7fe1b406d410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:12:11.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:11.411+0000 7fe1d8635700 1 -- 172.21.15.16:0/51838310 --> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir0", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fe1d00b2d30 con 0x7fe1b406af50 2023-10-16T12:12:11.685 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:12:11.684+0000 7f6483603200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T12:12:11.713 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:12:11.711+0000 7ffab6438200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T12:12:11.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:11.776+0000 7fe1d63d1700 1 -- 172.21.15.16:0/51838310 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fe1b406af50 msgr2=0x7fe1b406d410 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/365921436 2023-10-16T12:12:11.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:11.776+0000 7fe1d63d1700 1 --2- 172.21.15.16:0/51838310 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fe1b406af50 0x7fe1b406d410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-16T12:12:12.290 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:12:12.289+0000 7f6483603200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T12:12:12.318 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:12:12.317+0000 7ffab6438200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T12:12:12.362 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:12:12.361+0000 7f6483603200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T12:12:12.391 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:12:12.389+0000 7ffab6438200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T12:12:12.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:12.577+0000 7fe1d63d1700 1 -- 172.21.15.16:0/51838310 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fe1b406af50 msgr2=0x7fe1b406d410 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/365921436 2023-10-16T12:12:12.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:12.577+0000 7fe1d63d1700 1 --2- 172.21.15.16:0/51838310 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fe1b406af50 0x7fe1b406d410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-16T12:12:13.019 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:12:13.018+0000 7f6483603200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:12:13.053 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:12:13.052+0000 7ffab6438200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:12:13.099 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:12:13.097+0000 7f6483603200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T12:12:13.134 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:12:13.133+0000 7ffab6438200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T12:12:13.249 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:12:13.248+0000 7f6483603200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:12:13.284 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:12:13.283+0000 7ffab6438200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:12:13.485 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:12:13.483+0000 7f6483603200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T12:12:13.519 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:12:13.518+0000 7ffab6438200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T12:12:13.560 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:12:13.559+0000 7f6483603200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T12:12:13.595 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:12:13.594+0000 7ffab6438200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T12:12:13.834 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:12:13.832+0000 7f6483603200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T12:12:13.870 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:12:13.868+0000 7ffab6438200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T12:12:14.061 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:12:14.059+0000 7f6483603200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T12:12:14.098 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:12:14.096+0000 7ffab6438200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T12:12:14.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:14.178+0000 7fe1d63d1700 1 -- 172.21.15.16:0/51838310 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fe1b406af50 msgr2=0x7fe1b406d410 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/365921436 2023-10-16T12:12:14.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:14.178+0000 7fe1d63d1700 1 --2- 172.21.15.16:0/51838310 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fe1b406af50 0x7fe1b406d410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-10-16T12:12:14.215 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:12:14.213+0000 7f6483603200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T12:12:14.252 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:12:14.250+0000 7ffab6438200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T12:12:14.333 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:12:14.331+0000 7f6483603200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T12:12:14.370 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:12:14.368+0000 7ffab6438200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T12:12:14.493 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:12:14.491+0000 7f6483603200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T12:12:14.530 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:12:14.528+0000 7ffab6438200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T12:12:14.763 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:12:14.762+0000 7f6483603200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T12:12:14.800 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:12:14.798+0000 7ffab6438200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T12:12:14.928 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:12:14.926+0000 7f6483603200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T12:12:14.964 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:12:14.962+0000 7ffab6438200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T12:12:15.476 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:12:15.474+0000 7f6483603200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T12:12:15.513 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:12:15.511+0000 7ffab6438200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T12:12:15.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:15.532+0000 7fe1c6ffd700 1 -- 172.21.15.16:0/51838310 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 7) v1 ==== 88957+0+0 (secure 0 0 0) 0x7fe1c0053fa0 con 0x7fe1d00d4060 2023-10-16T12:12:15.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:15.532+0000 7fe1c6ffd700 1 -- 172.21.15.16:0/51838310 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fe1b406af50 msgr2=0x7fe1b406d410 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:12:15.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:15.532+0000 7fe1c6ffd700 1 --2- 172.21.15.16:0/51838310 >> [v2:172.21.15.16:6832/365921436,v1:172.21.15.16:6833/365921436] conn(0x7fe1b406af50 0x7fe1b406d410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:15.673 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:12:15.671+0000 7ffa29413700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:12:15.673 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:12:15.671+0000 7ffa29413700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:12:15.673 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:12:15.671+0000 7ffa29413700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:12:15.673 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:12:15.671+0000 7ffa29413700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:12:15.674 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:12:15.671+0000 7ffa29413700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:12:15.674 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:12:15.671+0000 7ffa2c459700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:12:15.674 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:12:15.671+0000 7ffa2c459700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:12:15.675 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:12:15.671+0000 7ffa2c459700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:12:15.675 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:12:15.671+0000 7ffa2c459700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:12:15.675 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:12:15.671+0000 7ffa2c459700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:12:16.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.551+0000 7fe1c6ffd700 1 -- 172.21.15.16:0/51838310 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mgrmap(e 8) v1 ==== 89084+0+0 (secure 0 0 0) 0x7fe1c0035dc0 con 0x7fe1d00d4060 2023-10-16T12:12:16.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.551+0000 7fe1c6ffd700 1 --2- 172.21.15.16:0/51838310 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fe1b4035b40 0x7fe1b4075320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:16.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.551+0000 7fe1c6ffd700 1 -- 172.21.15.16:0/51838310 --> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir0", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fe1b4035f20 con 0x7fe1b4035b40 2023-10-16T12:12:16.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.554+0000 7fe1d63d1700 1 --2- 172.21.15.16:0/51838310 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fe1b4035b40 0x7fe1b4075320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:16.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.558+0000 7fe1d63d1700 1 --2- 172.21.15.16:0/51838310 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fe1b4035b40 0x7fe1b4075320 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe1cc009fd0 tx=0x7fe1cc008070 comp rx=0 tx=0).ready entity=mgr.5499 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:16.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.626+0000 7fe1c6ffd700 1 -- 172.21.15.16:0/51838310 <== mgr.5499 v2:172.21.15.16:6832/1707126831 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fe1b4035f20 con 0x7fe1b4035b40 2023-10-16T12:12:16.628 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule set for path /snap_test_dir1/dir0 2023-10-16T12:12:16.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.628+0000 7fe1d8635700 1 -- 172.21.15.16:0/51838310 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fe1b4035b40 msgr2=0x7fe1b4075320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:16.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.628+0000 7fe1d8635700 1 --2- 172.21.15.16:0/51838310 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fe1b4035b40 0x7fe1b4075320 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe1cc009fd0 tx=0x7fe1cc008070 comp rx=0 tx=0).stop 2023-10-16T12:12:16.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.628+0000 7fe1d8635700 1 -- 172.21.15.16:0/51838310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1d00d4060 msgr2=0x7fe1d01f6600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:16.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.628+0000 7fe1d8635700 1 --2- 172.21.15.16:0/51838310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1d00d4060 0x7fe1d01f6600 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fe1c000c500 tx=0x7fe1c000c810 comp rx=0 tx=0).stop 2023-10-16T12:12:16.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.628+0000 7fe1d6bd2700 1 -- 172.21.15.16:0/51838310 reap_dead start 2023-10-16T12:12:16.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.628+0000 7fe1d8635700 1 -- 172.21.15.16:0/51838310 shutdown_connections 2023-10-16T12:12:16.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.628+0000 7fe1d8635700 1 -- 172.21.15.16:0/51838310 >> 172.21.15.16:0/51838310 conn(0x7fe1d009c9f0 msgr2=0x7fe1d015e170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:16.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.628+0000 7fe1d8635700 1 -- 172.21.15.16:0/51838310 shutdown_connections 2023-10-16T12:12:16.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.628+0000 7fe1d8635700 1 -- 172.21.15.16:0/51838310 wait complete. 2023-10-16T12:12:16.643 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir0 2023-10-16T12:12:16.643 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:12:16.644 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir1) 2023-10-16T12:12:16.663 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir1 1m 2023-10-16T12:12:16.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.831+0000 7f0310f1b700 1 Processor -- start 2023-10-16T12:12:16.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.831+0000 7f0310f1b700 1 -- start start 2023-10-16T12:12:16.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.832+0000 7f0310f1b700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f030c160fe0 0x7f030c0ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:16.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.832+0000 7f0310f1b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f030c06aef0 0x7f030c0adf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:16.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.832+0000 7f0310f1b700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f030c1613c0 con 0x7f030c06da30 2023-10-16T12:12:16.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.832+0000 7f0310f1b700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f030c0cf680 con 0x7f030c160fe0 2023-10-16T12:12:16.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.832+0000 7f0310f1b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f030c0adb00 con 0x7f030c06aef0 2023-10-16T12:12:16.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.832+0000 7f0309d9b700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f030c06da30 0x7f030c06a7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:53638/0 (socket says 172.21.15.16:53638) 2023-10-16T12:12:16.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.832+0000 7f0309d9b700 1 -- 172.21.15.16:0/4076914548 learned_addr learned my addr 172.21.15.16:0/4076914548 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:12:16.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.832+0000 7f030ad9d700 1 --2- 172.21.15.16:0/4076914548 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f030c06aef0 0x7f030c0adf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:16.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.832+0000 7f030a59c700 1 --2- 172.21.15.16:0/4076914548 >> v2:172.21.15.49:3300/0 conn(0x7f030c160fe0 0x7f030c0ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:16.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.832+0000 7f030959a700 1 -- 172.21.15.16:0/4076914548 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2947389806 0 0) 0x7f02fc001110 con 0x7f030c06da30 2023-10-16T12:12:16.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.832+0000 7f030959a700 1 -- 172.21.15.16:0/4076914548 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02f40035b0 con 0x7f030c06da30 2023-10-16T12:12:16.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.833+0000 7f030959a700 1 -- 172.21.15.16:0/4076914548 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2465851471 0 0) 0x7f02f40035b0 con 0x7f030c06da30 2023-10-16T12:12:16.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.833+0000 7f030959a700 1 -- 172.21.15.16:0/4076914548 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f030c06aef0 msgr2=0x7f030c0adf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:16.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.833+0000 7f030959a700 1 --2- 172.21.15.16:0/4076914548 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f030c06aef0 0x7f030c0adf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:16.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.833+0000 7f030959a700 1 -- 172.21.15.16:0/4076914548 >> v2:172.21.15.49:3300/0 conn(0x7f030c160fe0 msgr2=0x7f030c0ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:16.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.833+0000 7f030959a700 1 --2- 172.21.15.16:0/4076914548 >> v2:172.21.15.49:3300/0 conn(0x7f030c160fe0 0x7f030c0ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:16.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.833+0000 7f030959a700 1 -- 172.21.15.16:0/4076914548 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f030c0acef0 con 0x7f030c06da30 2023-10-16T12:12:16.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.833+0000 7f030959a700 1 -- 172.21.15.16:0/4076914548 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f02fc001800 con 0x7f030c06da30 2023-10-16T12:12:16.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.833+0000 7f030959a700 1 -- 172.21.15.16:0/4076914548 >> v1:172.21.15.16:6789/0 conn(0x7f030c06da30 legacy=0x7f030c06a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:16.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.833+0000 7f030959a700 1 --2- 172.21.15.16:0/4076914548 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02f4003a00 0x7f02f4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:16.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.833+0000 7f030959a700 1 --2- 172.21.15.16:0/4076914548 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f02f4004850 0x7f02f4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:16.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.833+0000 7f030959a700 1 --2- 172.21.15.16:0/4076914548 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f02f4005210 0x7f02f400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:16.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.833+0000 7f030959a700 1 -- 172.21.15.16:0/4076914548 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f030c0adb00 con 0x7f02f4003a00 2023-10-16T12:12:16.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.833+0000 7f030959a700 1 -- 172.21.15.16:0/4076914548 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02f400bf50 con 0x7f02f4005210 2023-10-16T12:12:16.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.833+0000 7f030959a700 1 -- 172.21.15.16:0/4076914548 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02f400c090 con 0x7f02f4004850 2023-10-16T12:12:16.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.833+0000 7f030ad9d700 1 --2- 172.21.15.16:0/4076914548 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f030c06aef0 0x7f030c0adf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:12:16.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.833+0000 7f030a59c700 1 --2- 172.21.15.16:0/4076914548 >> v2:172.21.15.49:3300/0 conn(0x7f030c160fe0 0x7f030c0ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:12:16.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.833+0000 7f0309d9b700 1 --2- 172.21.15.16:0/4076914548 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f02f4004850 0x7f02f4004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:16.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.833+0000 7f030a59c700 1 --2- 172.21.15.16:0/4076914548 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02f4003a00 0x7f02f4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:16.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.833+0000 7f030ad9d700 1 --2- 172.21.15.16:0/4076914548 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f02f4005210 0x7f02f400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:16.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.833+0000 7f0309d9b700 1 -- 172.21.15.16:0/4076914548 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f02f4005210 msgr2=0x7f02f400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:16.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.833+0000 7f0309d9b700 1 --2- 172.21.15.16:0/4076914548 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f02f4005210 0x7f02f400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:16.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.833+0000 7f0309d9b700 1 -- 172.21.15.16:0/4076914548 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02f4003a00 msgr2=0x7f02f4003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:16.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.833+0000 7f0309d9b700 1 --2- 172.21.15.16:0/4076914548 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02f4003a00 0x7f02f4003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:16.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.833+0000 7f0309d9b700 1 -- 172.21.15.16:0/4076914548 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02f40035b0 con 0x7f02f4004850 2023-10-16T12:12:16.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.833+0000 7f030a59c700 1 --2- 172.21.15.16:0/4076914548 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02f4003a00 0x7f02f4003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:12:16.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.834+0000 7f0309d9b700 1 --2- 172.21.15.16:0/4076914548 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f02f4004850 0x7f02f4004cd0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f02fc00c2f0 tx=0x7f02f4005880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f3b571bfcbabed9f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:16.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.834+0000 7f030959a700 1 -- 172.21.15.16:0/4076914548 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02fc00ce30 con 0x7f02f4004850 2023-10-16T12:12:16.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.834+0000 7f030959a700 1 -- 172.21.15.16:0/4076914548 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02fc005960 con 0x7f02f4004850 2023-10-16T12:12:16.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.834+0000 7f030959a700 1 -- 172.21.15.16:0/4076914548 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02fc005d10 con 0x7f02f4004850 2023-10-16T12:12:16.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.834+0000 7f0310f1b700 1 -- 172.21.15.16:0/4076914548 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f02f4004850 msgr2=0x7f02f4004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:16.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.834+0000 7f0310f1b700 1 --2- 172.21.15.16:0/4076914548 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f02f4004850 0x7f02f4004cd0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f02fc00c2f0 tx=0x7f02f4005880 comp rx=0 tx=0).stop 2023-10-16T12:12:16.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.834+0000 7f030ad9d700 1 -- 172.21.15.16:0/4076914548 reap_dead start 2023-10-16T12:12:16.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.834+0000 7f0310f1b700 1 -- 172.21.15.16:0/4076914548 shutdown_connections 2023-10-16T12:12:16.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.834+0000 7f0310f1b700 1 -- 172.21.15.16:0/4076914548 >> 172.21.15.16:0/4076914548 conn(0x7f030c0b1ed0 msgr2=0x7f030c0b22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:16.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.834+0000 7f0310f1b700 1 -- 172.21.15.16:0/4076914548 shutdown_connections 2023-10-16T12:12:16.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.834+0000 7f0310f1b700 1 -- 172.21.15.16:0/4076914548 wait complete. 2023-10-16T12:12:16.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.836+0000 7f0310f1b700 1 Processor -- start 2023-10-16T12:12:16.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.836+0000 7f0310f1b700 1 -- start start 2023-10-16T12:12:16.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.837+0000 7f0310f1b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f02f4004850 0x7f030c14a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:16.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.837+0000 7f0310f1b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f030c0aead0 0x7f030c0aef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:16.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.837+0000 7f0310f1b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f030c159570 0x7f030c1599f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:16.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.837+0000 7f0310f1b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f030c15a880 con 0x7f030c0aead0 2023-10-16T12:12:16.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.837+0000 7f0310f1b700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f030c0af490 con 0x7f030c159570 2023-10-16T12:12:16.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.837+0000 7f0310f1b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f030c14ac30 con 0x7f02f4004850 2023-10-16T12:12:16.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.837+0000 7f0309d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f030c0aead0 0x7f030c0aef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:16.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.837+0000 7f0309d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f030c0aead0 0x7f030c0aef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36144/0 (socket says 172.21.15.16:36144) 2023-10-16T12:12:16.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.837+0000 7f0309d9b700 1 -- 172.21.15.16:0/625990870 learned_addr learned my addr 172.21.15.16:0/625990870 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:12:16.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.837+0000 7f030a59c700 1 --2- 172.21.15.16:0/625990870 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f02f4004850 0x7f030c14a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:16.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.837+0000 7f030ad9d700 1 --2- 172.21.15.16:0/625990870 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f030c159570 0x7f030c1599f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:16.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.837+0000 7f0309d9b700 1 -- 172.21.15.16:0/625990870 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f02f4004850 msgr2=0x7f030c14a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:16.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.837+0000 7f0309d9b700 1 --2- 172.21.15.16:0/625990870 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f02f4004850 0x7f030c14a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:16.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.837+0000 7f0309d9b700 1 -- 172.21.15.16:0/625990870 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f030c159570 msgr2=0x7f030c1599f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:16.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.837+0000 7f0309d9b700 1 --2- 172.21.15.16:0/625990870 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f030c159570 0x7f030c1599f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:16.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.837+0000 7f0309d9b700 1 -- 172.21.15.16:0/625990870 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f030c15a2c0 con 0x7f030c0aead0 2023-10-16T12:12:16.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.837+0000 7f0309d9b700 1 --2- 172.21.15.16:0/625990870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f030c0aead0 0x7f030c0aef50 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f02fc00c8e0 tx=0x7f02fc001b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:16.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.837+0000 7f02faffd700 1 -- 172.21.15.16:0/625990870 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02fc001150 con 0x7f030c0aead0 2023-10-16T12:12:16.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.837+0000 7f02faffd700 1 -- 172.21.15.16:0/625990870 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02fc004870 con 0x7f030c0aead0 2023-10-16T12:12:16.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.837+0000 7f0310f1b700 1 -- 172.21.15.16:0/625990870 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f030c15a5a0 con 0x7f030c0aead0 2023-10-16T12:12:16.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.837+0000 7f02faffd700 1 -- 172.21.15.16:0/625990870 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02fc004c00 con 0x7f030c0aead0 2023-10-16T12:12:16.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.837+0000 7f0310f1b700 1 -- 172.21.15.16:0/625990870 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f030c1486d0 con 0x7f030c0aead0 2023-10-16T12:12:16.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.838+0000 7f0310f1b700 1 -- 172.21.15.16:0/625990870 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f030c148250 con 0x7f030c0aead0 2023-10-16T12:12:16.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.839+0000 7f02faffd700 1 -- 172.21.15.16:0/625990870 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f02fc005040 con 0x7f030c0aead0 2023-10-16T12:12:16.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.839+0000 7f02faffd700 1 --2- 172.21.15.16:0/625990870 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f02ec06ada0 0x7f02ec06d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:16.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.839+0000 7f02faffd700 1 -- 172.21.15.16:0/625990870 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7f02fc08aa90 con 0x7f030c0aead0 2023-10-16T12:12:16.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.839+0000 7f030a59c700 1 --2- 172.21.15.16:0/625990870 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f02ec06ada0 0x7f02ec06d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:16.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.840+0000 7f030a59c700 1 --2- 172.21.15.16:0/625990870 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f02ec06ada0 0x7f02ec06d260 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0300009ef0 tx=0x7f03000091c0 comp rx=0 tx=0).ready entity=mgr.5499 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:16.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:16.844+0000 7f02faffd700 1 -- 172.21.15.16:0/625990870 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f030c148250 con 0x7f030c0aead0 2023-10-16T12:12:17.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.078+0000 7f0310f1b700 1 -- 172.21.15.16:0/625990870 --> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f030c0ad4b0 con 0x7f02ec06ada0 2023-10-16T12:12:17.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.133+0000 7f02faffd700 1 -- 172.21.15.16:0/625990870 <== mgr.5499 v2:172.21.15.16:6832/1707126831 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f0300019070 con 0x7f02ec06ada0 2023-10-16T12:12:17.135 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule set for path /snap_test_dir1/dir1 2023-10-16T12:12:17.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.135+0000 7f0310f1b700 1 -- 172.21.15.16:0/625990870 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f02ec06ada0 msgr2=0x7f02ec06d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:17.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.135+0000 7f0310f1b700 1 --2- 172.21.15.16:0/625990870 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f02ec06ada0 0x7f02ec06d260 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0300009ef0 tx=0x7f03000091c0 comp rx=0 tx=0).stop 2023-10-16T12:12:17.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.135+0000 7f0310f1b700 1 -- 172.21.15.16:0/625990870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f030c0aead0 msgr2=0x7f030c0aef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:17.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.135+0000 7f0310f1b700 1 --2- 172.21.15.16:0/625990870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f030c0aead0 0x7f030c0aef50 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f02fc00c8e0 tx=0x7f02fc001b90 comp rx=0 tx=0).stop 2023-10-16T12:12:17.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.135+0000 7f0310f1b700 1 -- 172.21.15.16:0/625990870 shutdown_connections 2023-10-16T12:12:17.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.135+0000 7f0310f1b700 1 --2- 172.21.15.16:0/625990870 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f02ec06ada0 0x7f02ec06d260 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:17.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.135+0000 7f0310f1b700 1 --2- 172.21.15.16:0/625990870 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f030c159570 0x7f030c1599f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:17.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.135+0000 7f0310f1b700 1 --2- 172.21.15.16:0/625990870 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f02f4004850 0x7f030c14a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:17.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.135+0000 7f0310f1b700 1 --2- 172.21.15.16:0/625990870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f030c0aead0 0x7f030c0aef50 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:17.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.135+0000 7f0310f1b700 1 -- 172.21.15.16:0/625990870 >> 172.21.15.16:0/625990870 conn(0x7f030c0b1ed0 msgr2=0x7f030c0ce990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:17.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.135+0000 7f0310f1b700 1 -- 172.21.15.16:0/625990870 shutdown_connections 2023-10-16T12:12:17.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.135+0000 7f0310f1b700 1 -- 172.21.15.16:0/625990870 wait complete. 2023-10-16T12:12:17.149 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir1 2023-10-16T12:12:17.150 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:12:17.150 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir2) 2023-10-16T12:12:17.169 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir2 1m 2023-10-16T12:12:17.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.335+0000 7fd1dedba700 1 Processor -- start 2023-10-16T12:12:17.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.335+0000 7fd1dedba700 1 -- start start 2023-10-16T12:12:17.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.336+0000 7fd1dedba700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd1d815b2c0 0x7fd1d8161340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:17.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.336+0000 7fd1dedba700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fd1d806e570 0x7fd1d815e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:17.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.336+0000 7fd1dedba700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd1d806a7e0 0x7fd1d806ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:17.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.336+0000 7fd1dedba700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1d806b1b0 con 0x7fd1d806a7e0 2023-10-16T12:12:17.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.336+0000 7fd1dedba700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1d8161880 con 0x7fd1d806e570 2023-10-16T12:12:17.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.336+0000 7fd1dedba700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d815b6a0 con 0x7fd1d815b2c0 2023-10-16T12:12:17.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.336+0000 7fd1dd357700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd1d806a7e0 0x7fd1d806ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:17.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.336+0000 7fd1dd357700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd1d806a7e0 0x7fd1d806ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36154/0 (socket says 172.21.15.16:36154) 2023-10-16T12:12:17.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.336+0000 7fd1dd357700 1 -- 172.21.15.16:0/1637448058 learned_addr learned my addr 172.21.15.16:0/1637448058 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:12:17.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.337+0000 7fd1dcb56700 1 --2- 172.21.15.16:0/1637448058 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd1d815b2c0 0x7fd1d8161340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:17.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.337+0000 7fd1d7fff700 1 --2- 172.21.15.16:0/1637448058 >> v2:172.21.15.49:3300/0 conn(0x7fd1d806e570 0x7fd1d815e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:17.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.337+0000 7fd1dd357700 1 -- 172.21.15.16:0/1637448058 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd1d815b2c0 msgr2=0x7fd1d8161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:17.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.337+0000 7fd1dd357700 1 --2- 172.21.15.16:0/1637448058 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd1d815b2c0 0x7fd1d8161340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:17.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.337+0000 7fd1dd357700 1 -- 172.21.15.16:0/1637448058 >> v2:172.21.15.49:3300/0 conn(0x7fd1d806e570 msgr2=0x7fd1d815e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:17.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.337+0000 7fd1dd357700 1 --2- 172.21.15.16:0/1637448058 >> v2:172.21.15.49:3300/0 conn(0x7fd1d806e570 0x7fd1d815e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:17.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.337+0000 7fd1dd357700 1 -- 172.21.15.16:0/1637448058 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1d8161b20 con 0x7fd1d806a7e0 2023-10-16T12:12:17.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.337+0000 7fd1dd357700 1 --2- 172.21.15.16:0/1637448058 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1d806a7e0 0x7fd1d806ac70 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fd1d8070410 tx=0x7fd1d000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4704aef86ce50cc7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:17.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.337+0000 7fd1d77fe700 1 -- 172.21.15.16:0/1637448058 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1d000e070 con 0x7fd1d806a7e0 2023-10-16T12:12:17.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.337+0000 7fd1d77fe700 1 -- 172.21.15.16:0/1637448058 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1d000b910 con 0x7fd1d806a7e0 2023-10-16T12:12:17.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.337+0000 7fd1d77fe700 1 -- 172.21.15.16:0/1637448058 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1d000bcc0 con 0x7fd1d806a7e0 2023-10-16T12:12:17.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.338+0000 7fd1dedba700 1 -- 172.21.15.16:0/1637448058 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1d806a7e0 msgr2=0x7fd1d806ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:17.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.338+0000 7fd1dedba700 1 --2- 172.21.15.16:0/1637448058 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1d806a7e0 0x7fd1d806ac70 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fd1d8070410 tx=0x7fd1d000b330 comp rx=0 tx=0).stop 2023-10-16T12:12:17.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.338+0000 7fd1dedba700 1 -- 172.21.15.16:0/1637448058 shutdown_connections 2023-10-16T12:12:17.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.338+0000 7fd1dedba700 1 --2- 172.21.15.16:0/1637448058 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1d806a7e0 0x7fd1d806ac70 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:17.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.338+0000 7fd1dedba700 1 --2- 172.21.15.16:0/1637448058 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd1d815b2c0 0x7fd1d8161340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:17.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.338+0000 7fd1dedba700 1 --2- 172.21.15.16:0/1637448058 >> v2:172.21.15.49:3300/0 conn(0x7fd1d806e570 0x7fd1d815e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:17.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.338+0000 7fd1dedba700 1 -- 172.21.15.16:0/1637448058 >> 172.21.15.16:0/1637448058 conn(0x7fd1d80b1f40 msgr2=0x7fd1d80b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:17.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.338+0000 7fd1dedba700 1 -- 172.21.15.16:0/1637448058 shutdown_connections 2023-10-16T12:12:17.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.338+0000 7fd1dedba700 1 -- 172.21.15.16:0/1637448058 wait complete. 2023-10-16T12:12:17.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.339+0000 7fd1dedba700 1 Processor -- start 2023-10-16T12:12:17.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.339+0000 7fd1dedba700 1 -- start start 2023-10-16T12:12:17.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.340+0000 7fd1dedba700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd1d806a7e0 0x7fd1d81f3090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:17.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.340+0000 7fd1dedba700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd1d806e570 0x7fd1d815fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:17.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.340+0000 7fd1dedba700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1d815b2c0 0x7fd1d80d0510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:17.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.340+0000 7fd1dedba700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d81f40a0 con 0x7fd1d815b2c0 2023-10-16T12:12:17.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.340+0000 7fd1dedba700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d81f35d0 con 0x7fd1d806a7e0 2023-10-16T12:12:17.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.340+0000 7fd1dedba700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d80b4190 con 0x7fd1d806e570 2023-10-16T12:12:17.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.340+0000 7fd1dd357700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1d815b2c0 0x7fd1d80d0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:17.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.340+0000 7fd1dd357700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1d815b2c0 0x7fd1d80d0510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36160/0 (socket says 172.21.15.16:36160) 2023-10-16T12:12:17.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.340+0000 7fd1dd357700 1 -- 172.21.15.16:0/1855956186 learned_addr learned my addr 172.21.15.16:0/1855956186 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:12:17.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.340+0000 7fd1d7fff700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd1d806e570 0x7fd1d815fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:17.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.340+0000 7fd1dcb56700 1 --2- 172.21.15.16:0/1855956186 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd1d806a7e0 0x7fd1d81f3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:17.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.340+0000 7fd1dd357700 1 -- 172.21.15.16:0/1855956186 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd1d806e570 msgr2=0x7fd1d815fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:17.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.340+0000 7fd1dd357700 1 --2- 172.21.15.16:0/1855956186 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd1d806e570 0x7fd1d815fe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:17.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.340+0000 7fd1dd357700 1 -- 172.21.15.16:0/1855956186 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd1d806a7e0 msgr2=0x7fd1d81f3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:17.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.340+0000 7fd1dd357700 1 --2- 172.21.15.16:0/1855956186 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd1d806a7e0 0x7fd1d81f3090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:17.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.340+0000 7fd1dd357700 1 -- 172.21.15.16:0/1855956186 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1d0009b30 con 0x7fd1d815b2c0 2023-10-16T12:12:17.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.341+0000 7fd1dd357700 1 --2- 172.21.15.16:0/1855956186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1d815b2c0 0x7fd1d80d0510 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fd1d000b680 tx=0x7fd1d0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:17.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.341+0000 7fd1d57fa700 1 -- 172.21.15.16:0/1855956186 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1d00099b0 con 0x7fd1d815b2c0 2023-10-16T12:12:17.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.341+0000 7fd1d57fa700 1 -- 172.21.15.16:0/1855956186 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1d000ce80 con 0x7fd1d815b2c0 2023-10-16T12:12:17.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.341+0000 7fd1dedba700 1 -- 172.21.15.16:0/1855956186 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1d81f3800 con 0x7fd1d815b2c0 2023-10-16T12:12:17.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.341+0000 7fd1d57fa700 1 -- 172.21.15.16:0/1855956186 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1d0004490 con 0x7fd1d815b2c0 2023-10-16T12:12:17.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.341+0000 7fd1dedba700 1 -- 172.21.15.16:0/1855956186 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1d81f3dc0 con 0x7fd1d815b2c0 2023-10-16T12:12:17.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.342+0000 7fd1dedba700 1 -- 172.21.15.16:0/1855956186 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1d80b3f60 con 0x7fd1d815b2c0 2023-10-16T12:12:17.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.342+0000 7fd1d57fa700 1 -- 172.21.15.16:0/1855956186 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) v1 ==== 89084+0+0 (secure 0 0 0) 0x7fd1d00045f0 con 0x7fd1d815b2c0 2023-10-16T12:12:17.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.343+0000 7fd1d57fa700 1 --2- 172.21.15.16:0/1855956186 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fd1bc06ada0 0x7fd1bc06d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:17.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.343+0000 7fd1dcb56700 1 --2- 172.21.15.16:0/1855956186 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fd1bc06ada0 0x7fd1bc06d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:17.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.343+0000 7fd1d57fa700 1 -- 172.21.15.16:0/1855956186 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7fd1d008a430 con 0x7fd1d815b2c0 2023-10-16T12:12:17.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.343+0000 7fd1dcb56700 1 --2- 172.21.15.16:0/1855956186 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fd1bc06ada0 0x7fd1bc06d260 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd1d81536f0 tx=0x7fd1cc009210 comp rx=0 tx=0).ready entity=mgr.5499 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:17.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.348+0000 7fd1d57fa700 1 -- 172.21.15.16:0/1855956186 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd1d80b3f60 con 0x7fd1d815b2c0 2023-10-16T12:12:17.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.557+0000 7fd1d57fa700 1 -- 172.21.15.16:0/1855956186 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 9) v1 ==== 89349+0+0 (secure 0 0 0) 0x7fd1d00559b0 con 0x7fd1d815b2c0 2023-10-16T12:12:17.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.584+0000 7fd1dedba700 1 -- 172.21.15.16:0/1855956186 --> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir2", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fd1d81620e0 con 0x7fd1bc06ada0 2023-10-16T12:12:17.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.646+0000 7fd1d57fa700 1 -- 172.21.15.16:0/1855956186 <== mgr.5499 v2:172.21.15.16:6832/1707126831 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fd1cc0098d0 con 0x7fd1bc06ada0 2023-10-16T12:12:17.649 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule set for path /snap_test_dir1/dir2 2023-10-16T12:12:17.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.649+0000 7fd1dedba700 1 -- 172.21.15.16:0/1855956186 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fd1bc06ada0 msgr2=0x7fd1bc06d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:17.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.649+0000 7fd1dedba700 1 --2- 172.21.15.16:0/1855956186 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fd1bc06ada0 0x7fd1bc06d260 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd1d81536f0 tx=0x7fd1cc009210 comp rx=0 tx=0).stop 2023-10-16T12:12:17.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.649+0000 7fd1dedba700 1 -- 172.21.15.16:0/1855956186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1d815b2c0 msgr2=0x7fd1d80d0510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:17.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.649+0000 7fd1dedba700 1 --2- 172.21.15.16:0/1855956186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1d815b2c0 0x7fd1d80d0510 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fd1d000b680 tx=0x7fd1d0004ad0 comp rx=0 tx=0).stop 2023-10-16T12:12:17.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.649+0000 7fd1dedba700 1 -- 172.21.15.16:0/1855956186 shutdown_connections 2023-10-16T12:12:17.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.649+0000 7fd1dedba700 1 --2- 172.21.15.16:0/1855956186 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fd1bc06ada0 0x7fd1bc06d260 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:17.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.649+0000 7fd1dedba700 1 --2- 172.21.15.16:0/1855956186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1d815b2c0 0x7fd1d80d0510 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:17.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.649+0000 7fd1dedba700 1 --2- 172.21.15.16:0/1855956186 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd1d806a7e0 0x7fd1d81f3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:17.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.649+0000 7fd1dedba700 1 --2- 172.21.15.16:0/1855956186 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd1d806e570 0x7fd1d815fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:17.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.649+0000 7fd1dedba700 1 -- 172.21.15.16:0/1855956186 >> 172.21.15.16:0/1855956186 conn(0x7fd1d80b1f40 msgr2=0x7fd1d80726b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:17.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.649+0000 7fd1dedba700 1 -- 172.21.15.16:0/1855956186 shutdown_connections 2023-10-16T12:12:17.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.649+0000 7fd1dedba700 1 -- 172.21.15.16:0/1855956186 wait complete. 2023-10-16T12:12:17.663 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir2 2023-10-16T12:12:17.663 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:12:17.664 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir3) 2023-10-16T12:12:17.682 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir3 1m 2023-10-16T12:12:17.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.848+0000 7f4db33e7700 1 Processor -- start 2023-10-16T12:12:17.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.849+0000 7f4db33e7700 1 -- start start 2023-10-16T12:12:17.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.849+0000 7f4db33e7700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f4dac15b2c0 0x7f4dac161340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:17.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.849+0000 7f4db33e7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4dac06e570 0x7f4dac15e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:17.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.849+0000 7f4db33e7700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dac161880 con 0x7f4dac06a7e0 2023-10-16T12:12:17.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.849+0000 7f4db33e7700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dac15b6a0 con 0x7f4dac06e570 2023-10-16T12:12:17.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.849+0000 7f4db33e7700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dac161b20 con 0x7f4dac15b2c0 2023-10-16T12:12:17.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.850+0000 7f4db0982700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4dac06e570 0x7f4dac15e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:17.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.850+0000 7f4db0982700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4dac06e570 0x7f4dac15e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36184/0 (socket says 172.21.15.16:36184) 2023-10-16T12:12:17.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.850+0000 7f4db0982700 1 -- 172.21.15.16:0/513829038 learned_addr learned my addr 172.21.15.16:0/513829038 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:12:17.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.850+0000 7f4db1183700 1 --2- 172.21.15.16:0/513829038 >> v2:172.21.15.49:3300/0 conn(0x7f4dac15b2c0 0x7f4dac161340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:17.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.850+0000 7f4db0982700 1 -- 172.21.15.16:0/513829038 >> v2:172.21.15.49:3300/0 conn(0x7f4dac15b2c0 msgr2=0x7f4dac161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:17.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.850+0000 7f4db0982700 1 --2- 172.21.15.16:0/513829038 >> v2:172.21.15.49:3300/0 conn(0x7f4dac15b2c0 0x7f4dac161340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:17.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.850+0000 7f4db0982700 1 -- 172.21.15.16:0/513829038 >> v1:172.21.15.49:6789/0 conn(0x7f4dac06a7e0 legacy=0x7f4dac071a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:17.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.850+0000 7f4db0982700 1 -- 172.21.15.16:0/513829038 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dac161e00 con 0x7f4dac06e570 2023-10-16T12:12:17.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.850+0000 7f4db0982700 1 --2- 172.21.15.16:0/513829038 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dac06e570 0x7f4dac15e5a0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f4dac06b4a0 tx=0x7f4d9c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dfbf8e87b3749e4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:17.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.850+0000 7f4da3fff700 1 -- 172.21.15.16:0/513829038 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d9c004230 con 0x7f4dac06e570 2023-10-16T12:12:17.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.850+0000 7f4da3fff700 1 -- 172.21.15.16:0/513829038 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d9c004390 con 0x7f4dac06e570 2023-10-16T12:12:17.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.850+0000 7f4da3fff700 1 -- 172.21.15.16:0/513829038 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d9c0036a0 con 0x7f4dac06e570 2023-10-16T12:12:17.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.851+0000 7f4db33e7700 1 -- 172.21.15.16:0/513829038 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dac06e570 msgr2=0x7f4dac15e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:17.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.851+0000 7f4db33e7700 1 --2- 172.21.15.16:0/513829038 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dac06e570 0x7f4dac15e5a0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f4dac06b4a0 tx=0x7f4d9c009bd0 comp rx=0 tx=0).stop 2023-10-16T12:12:17.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.851+0000 7f4db33e7700 1 -- 172.21.15.16:0/513829038 shutdown_connections 2023-10-16T12:12:17.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.851+0000 7f4db33e7700 1 --2- 172.21.15.16:0/513829038 >> v2:172.21.15.49:3300/0 conn(0x7f4dac15b2c0 0x7f4dac161340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:17.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.851+0000 7f4db33e7700 1 --2- 172.21.15.16:0/513829038 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dac06e570 0x7f4dac15e5a0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:17.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.851+0000 7f4db33e7700 1 -- 172.21.15.16:0/513829038 >> 172.21.15.16:0/513829038 conn(0x7f4dac0b1f40 msgr2=0x7f4dac0b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:17.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.851+0000 7f4db33e7700 1 -- 172.21.15.16:0/513829038 shutdown_connections 2023-10-16T12:12:17.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.851+0000 7f4db33e7700 1 -- 172.21.15.16:0/513829038 wait complete. 2023-10-16T12:12:17.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.852+0000 7f4db33e7700 1 Processor -- start 2023-10-16T12:12:17.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.852+0000 7f4db33e7700 1 -- start start 2023-10-16T12:12:17.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.853+0000 7f4db33e7700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4dac06a7e0 0x7f4dac157970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:17.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.853+0000 7f4db33e7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dac06e570 0x7f4dac0c7ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:17.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.853+0000 7f4db33e7700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4dac15b2c0 0x7f4dac15fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:17.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.853+0000 7f4db33e7700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dac0c8ca0 con 0x7f4dac06e570 2023-10-16T12:12:17.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.853+0000 7f4db33e7700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dac0b6170 con 0x7f4dac15b2c0 2023-10-16T12:12:17.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.853+0000 7f4db33e7700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dac0b58d0 con 0x7f4dac06a7e0 2023-10-16T12:12:17.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.853+0000 7f4db0982700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dac06e570 0x7f4dac0c7ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:17.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.853+0000 7f4db0982700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dac06e570 0x7f4dac0c7ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36186/0 (socket says 172.21.15.16:36186) 2023-10-16T12:12:17.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.853+0000 7f4db0982700 1 -- 172.21.15.16:0/1891523139 learned_addr learned my addr 172.21.15.16:0/1891523139 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:12:17.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.853+0000 7f4db1183700 1 --2- 172.21.15.16:0/1891523139 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4dac06a7e0 0x7f4dac157970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:17.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.853+0000 7f4db1984700 1 --2- 172.21.15.16:0/1891523139 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4dac15b2c0 0x7f4dac15fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:17.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.853+0000 7f4db0982700 1 -- 172.21.15.16:0/1891523139 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4dac06a7e0 msgr2=0x7f4dac157970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:17.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.853+0000 7f4db0982700 1 --2- 172.21.15.16:0/1891523139 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4dac06a7e0 0x7f4dac157970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:17.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.853+0000 7f4db0982700 1 -- 172.21.15.16:0/1891523139 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4dac15b2c0 msgr2=0x7f4dac15fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:17.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.853+0000 7f4db0982700 1 --2- 172.21.15.16:0/1891523139 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4dac15b2c0 0x7f4dac15fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:17.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.853+0000 7f4db0982700 1 -- 172.21.15.16:0/1891523139 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d9c009550 con 0x7f4dac06e570 2023-10-16T12:12:17.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.853+0000 7f4db1183700 1 --2- 172.21.15.16:0/1891523139 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4dac06a7e0 0x7f4dac157970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:12:17.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.853+0000 7f4db1984700 1 --2- 172.21.15.16:0/1891523139 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4dac15b2c0 0x7f4dac15fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:12:17.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.853+0000 7f4db0982700 1 --2- 172.21.15.16:0/1891523139 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dac06e570 0x7f4dac0c7ed0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f4dac070620 tx=0x7f4d9c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:17.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.853+0000 7f4da1ffb700 1 -- 172.21.15.16:0/1891523139 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d9c003d80 con 0x7f4dac06e570 2023-10-16T12:12:17.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.854+0000 7f4da1ffb700 1 -- 172.21.15.16:0/1891523139 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d9c01f070 con 0x7f4dac06e570 2023-10-16T12:12:17.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.854+0000 7f4da1ffb700 1 -- 172.21.15.16:0/1891523139 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d9c011870 con 0x7f4dac06e570 2023-10-16T12:12:17.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.854+0000 7f4db33e7700 1 -- 172.21.15.16:0/1891523139 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dac0b5bb0 con 0x7f4dac06e570 2023-10-16T12:12:17.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.854+0000 7f4db33e7700 1 -- 172.21.15.16:0/1891523139 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dac157eb0 con 0x7f4dac06e570 2023-10-16T12:12:17.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.855+0000 7f4db33e7700 1 -- 172.21.15.16:0/1891523139 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dac0c8e10 con 0x7f4dac06e570 2023-10-16T12:12:17.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.855+0000 7f4da1ffb700 1 -- 172.21.15.16:0/1891523139 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 9) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f4d9c021030 con 0x7f4dac06e570 2023-10-16T12:12:17.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.855+0000 7f4da1ffb700 1 --2- 172.21.15.16:0/1891523139 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f4d9006b070 0x7f4d9006d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:17.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.856+0000 7f4da1ffb700 1 -- 172.21.15.16:0/1891523139 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7f4d9c08b160 con 0x7f4dac06e570 2023-10-16T12:12:17.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.856+0000 7f4db1183700 1 --2- 172.21.15.16:0/1891523139 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f4d9006b070 0x7f4d9006d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:17.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.856+0000 7f4db1183700 1 --2- 172.21.15.16:0/1891523139 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f4d9006b070 0x7f4d9006d530 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4dac15bbf0 tx=0x7f4da8008040 comp rx=0 tx=0).ready entity=mgr.5499 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:17.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:17.860+0000 7f4da1ffb700 1 -- 172.21.15.16:0/1891523139 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4d9c056070 con 0x7f4dac06e570 2023-10-16T12:12:18.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.090+0000 7f4db33e7700 1 -- 172.21.15.16:0/1891523139 --> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir3", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f4dac15e080 con 0x7f4d9006b070 2023-10-16T12:12:18.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.146+0000 7f4da1ffb700 1 -- 172.21.15.16:0/1891523139 <== mgr.5499 v2:172.21.15.16:6832/1707126831 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f4da800a2d0 con 0x7f4d9006b070 2023-10-16T12:12:18.149 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule set for path /snap_test_dir1/dir3 2023-10-16T12:12:18.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.149+0000 7f4db33e7700 1 -- 172.21.15.16:0/1891523139 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f4d9006b070 msgr2=0x7f4d9006d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:18.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.149+0000 7f4db33e7700 1 --2- 172.21.15.16:0/1891523139 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f4d9006b070 0x7f4d9006d530 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4dac15bbf0 tx=0x7f4da8008040 comp rx=0 tx=0).stop 2023-10-16T12:12:18.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.149+0000 7f4db33e7700 1 -- 172.21.15.16:0/1891523139 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dac06e570 msgr2=0x7f4dac0c7ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:18.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.149+0000 7f4db33e7700 1 --2- 172.21.15.16:0/1891523139 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dac06e570 0x7f4dac0c7ed0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f4dac070620 tx=0x7f4d9c004750 comp rx=0 tx=0).stop 2023-10-16T12:12:18.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.149+0000 7f4db33e7700 1 -- 172.21.15.16:0/1891523139 shutdown_connections 2023-10-16T12:12:18.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.149+0000 7f4db33e7700 1 --2- 172.21.15.16:0/1891523139 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f4d9006b070 0x7f4d9006d530 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:18.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.149+0000 7f4db33e7700 1 --2- 172.21.15.16:0/1891523139 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4dac15b2c0 0x7f4dac15fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:18.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.149+0000 7f4db33e7700 1 --2- 172.21.15.16:0/1891523139 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4dac06a7e0 0x7f4dac157970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:18.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.149+0000 7f4db33e7700 1 --2- 172.21.15.16:0/1891523139 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dac06e570 0x7f4dac0c7ed0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:18.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.149+0000 7f4db33e7700 1 -- 172.21.15.16:0/1891523139 >> 172.21.15.16:0/1891523139 conn(0x7f4dac0b1f40 msgr2=0x7f4dac160b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:18.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.149+0000 7f4db33e7700 1 -- 172.21.15.16:0/1891523139 shutdown_connections 2023-10-16T12:12:18.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.149+0000 7f4db33e7700 1 -- 172.21.15.16:0/1891523139 wait complete. 2023-10-16T12:12:18.163 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir3 2023-10-16T12:12:18.163 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:12:18.163 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir4) 2023-10-16T12:12:18.182 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir4 1m 2023-10-16T12:12:18.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.350+0000 7fc52a801700 1 Processor -- start 2023-10-16T12:12:18.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.350+0000 7fc52a801700 1 -- start start 2023-10-16T12:12:18.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.351+0000 7fc52a801700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fc5240cb880 0x7fc5240cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:18.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.351+0000 7fc52a801700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc52406da60 0x7fc52406def0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:18.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.351+0000 7fc52a801700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5240cef70 con 0x7fc52415b070 2023-10-16T12:12:18.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.351+0000 7fc52a801700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc52406e430 con 0x7fc52406da60 2023-10-16T12:12:18.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.351+0000 7fc52a801700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5240ce360 con 0x7fc5240cb880 2023-10-16T12:12:18.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.351+0000 7fc528d9e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc52406da60 0x7fc52406def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:18.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.351+0000 7fc528d9e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc52406da60 0x7fc52406def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36198/0 (socket says 172.21.15.16:36198) 2023-10-16T12:12:18.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.351+0000 7fc528d9e700 1 -- 172.21.15.16:0/2470192381 learned_addr learned my addr 172.21.15.16:0/2470192381 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:12:18.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.351+0000 7fc5237fe700 1 --2- 172.21.15.16:0/2470192381 >> v2:172.21.15.49:3300/0 conn(0x7fc5240cb880 0x7fc5240cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:18.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.351+0000 7fc528d9e700 1 -- 172.21.15.16:0/2470192381 >> v2:172.21.15.49:3300/0 conn(0x7fc5240cb880 msgr2=0x7fc5240cbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:18.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.351+0000 7fc528d9e700 1 --2- 172.21.15.16:0/2470192381 >> v2:172.21.15.49:3300/0 conn(0x7fc5240cb880 0x7fc5240cbd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:18.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.351+0000 7fc528d9e700 1 -- 172.21.15.16:0/2470192381 >> v1:172.21.15.49:6789/0 conn(0x7fc52415b070 legacy=0x7fc52415b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:18.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.351+0000 7fc528d9e700 1 -- 172.21.15.16:0/2470192381 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5240ce640 con 0x7fc52406da60 2023-10-16T12:12:18.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.351+0000 7fc5237fe700 1 --2- 172.21.15.16:0/2470192381 >> v2:172.21.15.49:3300/0 conn(0x7fc5240cb880 0x7fc5240cbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:12:18.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.351+0000 7fc528d9e700 1 --2- 172.21.15.16:0/2470192381 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc52406da60 0x7fc52406def0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fc524070500 tx=0x7fc51c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e081be5349f95d5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:18.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.352+0000 7fc522ffd700 1 -- 172.21.15.16:0/2470192381 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc51c00e070 con 0x7fc52406da60 2023-10-16T12:12:18.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.352+0000 7fc522ffd700 1 -- 172.21.15.16:0/2470192381 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc51c00b9c0 con 0x7fc52406da60 2023-10-16T12:12:18.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.352+0000 7fc522ffd700 1 -- 172.21.15.16:0/2470192381 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc51c00bd70 con 0x7fc52406da60 2023-10-16T12:12:18.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.352+0000 7fc52a801700 1 -- 172.21.15.16:0/2470192381 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc52406da60 msgr2=0x7fc52406def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:18.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.352+0000 7fc52a801700 1 --2- 172.21.15.16:0/2470192381 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc52406da60 0x7fc52406def0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fc524070500 tx=0x7fc51c00b330 comp rx=0 tx=0).stop 2023-10-16T12:12:18.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.352+0000 7fc52a801700 1 -- 172.21.15.16:0/2470192381 shutdown_connections 2023-10-16T12:12:18.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.352+0000 7fc52a801700 1 --2- 172.21.15.16:0/2470192381 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc52406da60 0x7fc52406def0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:18.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.352+0000 7fc52a801700 1 --2- 172.21.15.16:0/2470192381 >> v2:172.21.15.49:3300/0 conn(0x7fc5240cb880 0x7fc5240cbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:18.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.352+0000 7fc52a801700 1 -- 172.21.15.16:0/2470192381 >> 172.21.15.16:0/2470192381 conn(0x7fc5240b1f40 msgr2=0x7fc5240b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:18.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.352+0000 7fc52a801700 1 -- 172.21.15.16:0/2470192381 shutdown_connections 2023-10-16T12:12:18.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.352+0000 7fc52a801700 1 -- 172.21.15.16:0/2470192381 wait complete. 2023-10-16T12:12:18.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.354+0000 7fc52a801700 1 Processor -- start 2023-10-16T12:12:18.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.354+0000 7fc52a801700 1 -- start start 2023-10-16T12:12:18.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.355+0000 7fc52a801700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc52406da60 0x7fc524152580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:18.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.355+0000 7fc52a801700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc5240cb880 0x7fc5240c6110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:18.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.355+0000 7fc52a801700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc52415b070 0x7fc5240c6650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:18.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.355+0000 7fc52a801700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5240ccc60 con 0x7fc52415b070 2023-10-16T12:12:18.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.355+0000 7fc52a801700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc524152ac0 con 0x7fc5240cb880 2023-10-16T12:12:18.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.355+0000 7fc52a801700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc524155330 con 0x7fc52406da60 2023-10-16T12:12:18.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.355+0000 7fc528d9e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc52415b070 0x7fc5240c6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:18.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.355+0000 7fc528d9e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc52415b070 0x7fc5240c6650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36204/0 (socket says 172.21.15.16:36204) 2023-10-16T12:12:18.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.355+0000 7fc528d9e700 1 -- 172.21.15.16:0/2948855939 learned_addr learned my addr 172.21.15.16:0/2948855939 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:12:18.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.355+0000 7fc5237fe700 1 --2- 172.21.15.16:0/2948855939 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc5240cb880 0x7fc5240c6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:18.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.355+0000 7fc523fff700 1 --2- 172.21.15.16:0/2948855939 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc52406da60 0x7fc524152580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:18.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.356+0000 7fc528d9e700 1 -- 172.21.15.16:0/2948855939 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc52406da60 msgr2=0x7fc524152580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:18.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.356+0000 7fc528d9e700 1 --2- 172.21.15.16:0/2948855939 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc52406da60 0x7fc524152580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:18.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.356+0000 7fc528d9e700 1 -- 172.21.15.16:0/2948855939 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc5240cb880 msgr2=0x7fc5240c6110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:18.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.356+0000 7fc528d9e700 1 --2- 172.21.15.16:0/2948855939 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc5240cb880 0x7fc5240c6110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:18.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.356+0000 7fc528d9e700 1 -- 172.21.15.16:0/2948855939 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc51c009b30 con 0x7fc52415b070 2023-10-16T12:12:18.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.356+0000 7fc528d9e700 1 --2- 172.21.15.16:0/2948855939 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc52415b070 0x7fc5240c6650 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fc524070620 tx=0x7fc51c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:18.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.356+0000 7fc520ff9700 1 -- 172.21.15.16:0/2948855939 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc51c004880 con 0x7fc52415b070 2023-10-16T12:12:18.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.356+0000 7fc520ff9700 1 -- 172.21.15.16:0/2948855939 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc51c0099b0 con 0x7fc52415b070 2023-10-16T12:12:18.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.356+0000 7fc52a801700 1 -- 172.21.15.16:0/2948855939 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc524155610 con 0x7fc52415b070 2023-10-16T12:12:18.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.356+0000 7fc520ff9700 1 -- 172.21.15.16:0/2948855939 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc51c01b670 con 0x7fc52415b070 2023-10-16T12:12:18.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.356+0000 7fc52a801700 1 -- 172.21.15.16:0/2948855939 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc524155bd0 con 0x7fc52415b070 2023-10-16T12:12:18.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.357+0000 7fc52a801700 1 -- 172.21.15.16:0/2948855939 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc52404ebb0 con 0x7fc52415b070 2023-10-16T12:12:18.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.357+0000 7fc520ff9700 1 -- 172.21.15.16:0/2948855939 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 9) v1 ==== 89349+0+0 (secure 0 0 0) 0x7fc51c01b7d0 con 0x7fc52415b070 2023-10-16T12:12:18.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.358+0000 7fc520ff9700 1 --2- 172.21.15.16:0/2948855939 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fc50406b070 0x7fc50406d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:18.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.358+0000 7fc520ff9700 1 -- 172.21.15.16:0/2948855939 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7fc51c08b2c0 con 0x7fc52415b070 2023-10-16T12:12:18.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.358+0000 7fc523fff700 1 --2- 172.21.15.16:0/2948855939 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fc50406b070 0x7fc50406d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:18.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.358+0000 7fc523fff700 1 --2- 172.21.15.16:0/2948855939 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fc50406b070 0x7fc50406d530 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc524071c50 tx=0x7fc50c009040 comp rx=0 tx=0).ready entity=mgr.5499 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:18.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.362+0000 7fc520ff9700 1 -- 172.21.15.16:0/2948855939 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc51c055fa0 con 0x7fc52415b070 2023-10-16T12:12:18.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.593+0000 7fc52a801700 1 -- 172.21.15.16:0/2948855939 --> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir4", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fc5240cec00 con 0x7fc50406b070 2023-10-16T12:12:18.621 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:12:18.623 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:12:18.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.653+0000 7fc520ff9700 1 -- 172.21.15.16:0/2948855939 <== mgr.5499 v2:172.21.15.16:6832/1707126831 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fc50c001d60 con 0x7fc50406b070 2023-10-16T12:12:18.655 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule set for path /snap_test_dir1/dir4 2023-10-16T12:12:18.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.655+0000 7fc52a801700 1 -- 172.21.15.16:0/2948855939 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fc50406b070 msgr2=0x7fc50406d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:18.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.655+0000 7fc52a801700 1 --2- 172.21.15.16:0/2948855939 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fc50406b070 0x7fc50406d530 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc524071c50 tx=0x7fc50c009040 comp rx=0 tx=0).stop 2023-10-16T12:12:18.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.655+0000 7fc52a801700 1 -- 172.21.15.16:0/2948855939 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc52415b070 msgr2=0x7fc5240c6650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:18.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.655+0000 7fc52a801700 1 --2- 172.21.15.16:0/2948855939 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc52415b070 0x7fc5240c6650 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fc524070620 tx=0x7fc51c004ad0 comp rx=0 tx=0).stop 2023-10-16T12:12:18.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.655+0000 7fc52a801700 1 -- 172.21.15.16:0/2948855939 shutdown_connections 2023-10-16T12:12:18.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.655+0000 7fc52a801700 1 --2- 172.21.15.16:0/2948855939 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fc50406b070 0x7fc50406d530 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:18.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.655+0000 7fc52a801700 1 --2- 172.21.15.16:0/2948855939 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc52415b070 0x7fc5240c6650 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:18.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.655+0000 7fc52a801700 1 --2- 172.21.15.16:0/2948855939 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc52406da60 0x7fc524152580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:18.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.655+0000 7fc52a801700 1 --2- 172.21.15.16:0/2948855939 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc5240cb880 0x7fc5240c6110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:18.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.655+0000 7fc52a801700 1 -- 172.21.15.16:0/2948855939 >> 172.21.15.16:0/2948855939 conn(0x7fc5240b1f40 msgr2=0x7fc5240cf250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:18.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.655+0000 7fc52a801700 1 -- 172.21.15.16:0/2948855939 shutdown_connections 2023-10-16T12:12:18.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.655+0000 7fc52a801700 1 -- 172.21.15.16:0/2948855939 wait complete. 2023-10-16T12:12:18.669 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir4 2023-10-16T12:12:18.670 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:12:18.670 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir5) 2023-10-16T12:12:18.728 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir5 1m 2023-10-16T12:12:18.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.894+0000 7f882bd7f700 1 Processor -- start 2023-10-16T12:12:18.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.894+0000 7f882bd7f700 1 -- start start 2023-10-16T12:12:18.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.895+0000 7f882bd7f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f882415b070 0x7f882415b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:18.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.895+0000 7f882bd7f700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f88240cb880 0x7f88240cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:18.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.895+0000 7f882bd7f700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88240cef70 con 0x7f882406da60 2023-10-16T12:12:18.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.895+0000 7f882bd7f700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88240ce360 con 0x7f88240cb880 2023-10-16T12:12:18.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.895+0000 7f882bd7f700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88240ce640 con 0x7f882415b070 2023-10-16T12:12:18.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.895+0000 7f882a31c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f882406da60 0x7f88240cf400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:53718/0 (socket says 172.21.15.16:53718) 2023-10-16T12:12:18.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.895+0000 7f882a31c700 1 -- 172.21.15.16:0/2615316000 learned_addr learned my addr 172.21.15.16:0/2615316000 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:12:18.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.895+0000 7f8829b1b700 1 --2- 172.21.15.16:0/2615316000 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f882415b070 0x7f882415b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:18.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.895+0000 7f882931a700 1 --2- 172.21.15.16:0/2615316000 >> v2:172.21.15.49:3300/0 conn(0x7f88240cb880 0x7f88240cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:18.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.896+0000 7f8828b19700 1 -- 172.21.15.16:0/2615316000 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2621590215 0 0) 0x7f88240cef70 con 0x7f882406da60 2023-10-16T12:12:18.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.896+0000 7f8828b19700 1 -- 172.21.15.16:0/2615316000 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88100035b0 con 0x7f882406da60 2023-10-16T12:12:18.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.896+0000 7f8828b19700 1 -- 172.21.15.16:0/2615316000 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2113664844 0 0) 0x7f88100035b0 con 0x7f882406da60 2023-10-16T12:12:18.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.896+0000 7f8828b19700 1 -- 172.21.15.16:0/2615316000 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f882415b070 msgr2=0x7f882415b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:18.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.896+0000 7f8828b19700 1 --2- 172.21.15.16:0/2615316000 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f882415b070 0x7f882415b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:18.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.896+0000 7f8828b19700 1 -- 172.21.15.16:0/2615316000 >> v2:172.21.15.49:3300/0 conn(0x7f88240cb880 msgr2=0x7f88240cbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:18.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.896+0000 7f8828b19700 1 --2- 172.21.15.16:0/2615316000 >> v2:172.21.15.49:3300/0 conn(0x7f88240cb880 0x7f88240cbd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:18.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.896+0000 7f8828b19700 1 -- 172.21.15.16:0/2615316000 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88240ce920 con 0x7f882406da60 2023-10-16T12:12:18.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.896+0000 7f8828b19700 1 -- 172.21.15.16:0/2615316000 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f881c0019f0 con 0x7f882406da60 2023-10-16T12:12:18.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.896+0000 7f8828b19700 1 -- 172.21.15.16:0/2615316000 >> v1:172.21.15.16:6789/0 conn(0x7f882406da60 legacy=0x7f88240cf400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:18.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.896+0000 7f8828b19700 1 --2- 172.21.15.16:0/2615316000 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8810003a00 0x7f8810003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:18.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.896+0000 7f8828b19700 1 --2- 172.21.15.16:0/2615316000 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f882415b070 0x7f8810004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:18.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.896+0000 7f8828b19700 1 --2- 172.21.15.16:0/2615316000 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8810004d90 0x7f8810005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:18.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.896+0000 7f8828b19700 1 -- 172.21.15.16:0/2615316000 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88240ce360 con 0x7f8810003a00 2023-10-16T12:12:18.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.896+0000 7f8828b19700 1 -- 172.21.15.16:0/2615316000 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88240ce640 con 0x7f8810004d90 2023-10-16T12:12:18.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.896+0000 7f8828b19700 1 -- 172.21.15.16:0/2615316000 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f881000ba10 con 0x7f882415b070 2023-10-16T12:12:18.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.896+0000 7f8829b1b700 1 --2- 172.21.15.16:0/2615316000 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8810003a00 0x7f8810003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:18.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.896+0000 7f882931a700 1 --2- 172.21.15.16:0/2615316000 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f882415b070 0x7f8810004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:18.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.896+0000 7f882a31c700 1 --2- 172.21.15.16:0/2615316000 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8810004d90 0x7f8810005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:18.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.896+0000 7f8829b1b700 1 -- 172.21.15.16:0/2615316000 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f882415b070 msgr2=0x7f8810004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:18.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.896+0000 7f8829b1b700 1 --2- 172.21.15.16:0/2615316000 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f882415b070 0x7f8810004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:18.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.896+0000 7f8829b1b700 1 -- 172.21.15.16:0/2615316000 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8810004d90 msgr2=0x7f8810005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:18.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.896+0000 7f8829b1b700 1 --2- 172.21.15.16:0/2615316000 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8810004d90 0x7f8810005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:18.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.896+0000 7f8829b1b700 1 -- 172.21.15.16:0/2615316000 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88100035b0 con 0x7f8810003a00 2023-10-16T12:12:18.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.896+0000 7f8829b1b700 1 --2- 172.21.15.16:0/2615316000 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8810003a00 0x7f8810003e20 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f8820009b60 tx=0x7f8810005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5412e16bf6d0ca0e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:18.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.897+0000 7f8828b19700 1 -- 172.21.15.16:0/2615316000 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f882000e6c0 con 0x7f8810003a00 2023-10-16T12:12:18.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.897+0000 7f8828b19700 1 -- 172.21.15.16:0/2615316000 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f882000e820 con 0x7f8810003a00 2023-10-16T12:12:18.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.897+0000 7f8828b19700 1 -- 172.21.15.16:0/2615316000 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f882000eb90 con 0x7f8810003a00 2023-10-16T12:12:18.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.897+0000 7f882bd7f700 1 -- 172.21.15.16:0/2615316000 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8810003a00 msgr2=0x7f8810003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:18.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.897+0000 7f882bd7f700 1 --2- 172.21.15.16:0/2615316000 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8810003a00 0x7f8810003e20 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f8820009b60 tx=0x7f8810005880 comp rx=0 tx=0).stop 2023-10-16T12:12:18.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.897+0000 7f882a31c700 1 -- 172.21.15.16:0/2615316000 reap_dead start 2023-10-16T12:12:18.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.897+0000 7f882bd7f700 1 -- 172.21.15.16:0/2615316000 shutdown_connections 2023-10-16T12:12:18.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.897+0000 7f882bd7f700 1 -- 172.21.15.16:0/2615316000 >> 172.21.15.16:0/2615316000 conn(0x7f88240b1f40 msgr2=0x7f88240b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:18.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.897+0000 7f882bd7f700 1 -- 172.21.15.16:0/2615316000 shutdown_connections 2023-10-16T12:12:18.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.897+0000 7f882bd7f700 1 -- 172.21.15.16:0/2615316000 wait complete. 2023-10-16T12:12:18.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.899+0000 7f882bd7f700 1 Processor -- start 2023-10-16T12:12:18.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.899+0000 7f882bd7f700 1 -- start start 2023-10-16T12:12:18.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.899+0000 7f882bd7f700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8810003a00 0x7f88240c6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:18.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.899+0000 7f882bd7f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88240ccc30 0x7f88240c8510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:18.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.899+0000 7f882bd7f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f88240cd0b0 0x7f88240c8a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:18.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.899+0000 7f882bd7f700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88240b3480 con 0x7f88240ccc30 2023-10-16T12:12:18.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.899+0000 7f882bd7f700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88240b3760 con 0x7f8810003a00 2023-10-16T12:12:18.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.899+0000 7f882bd7f700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88240b3a40 con 0x7f88240cd0b0 2023-10-16T12:12:18.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.899+0000 7f882931a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88240ccc30 0x7f88240c8510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:18.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.900+0000 7f882931a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88240ccc30 0x7f88240c8510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36222/0 (socket says 172.21.15.16:36222) 2023-10-16T12:12:18.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.900+0000 7f882931a700 1 -- 172.21.15.16:0/3339357590 learned_addr learned my addr 172.21.15.16:0/3339357590 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:12:18.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.900+0000 7f882a31c700 1 --2- 172.21.15.16:0/3339357590 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f88240cd0b0 0x7f88240c8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:18.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.900+0000 7f8829b1b700 1 --2- 172.21.15.16:0/3339357590 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8810003a00 0x7f88240c6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:18.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.900+0000 7f882931a700 1 -- 172.21.15.16:0/3339357590 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f88240cd0b0 msgr2=0x7f88240c8a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:18.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.900+0000 7f882931a700 1 --2- 172.21.15.16:0/3339357590 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f88240cd0b0 0x7f88240c8a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:18.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.900+0000 7f882931a700 1 -- 172.21.15.16:0/3339357590 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8810003a00 msgr2=0x7f88240c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:18.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.900+0000 7f882931a700 1 --2- 172.21.15.16:0/3339357590 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8810003a00 0x7f88240c6bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:18.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.900+0000 7f882931a700 1 -- 172.21.15.16:0/3339357590 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88240b3d20 con 0x7f88240ccc30 2023-10-16T12:12:18.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.900+0000 7f882931a700 1 --2- 172.21.15.16:0/3339357590 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88240ccc30 0x7f88240c8510 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f8818011aa0 tx=0x7f8818011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:18.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.900+0000 7f88167fc700 1 -- 172.21.15.16:0/3339357590 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f881800f9e0 con 0x7f88240ccc30 2023-10-16T12:12:18.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.900+0000 7f88167fc700 1 -- 172.21.15.16:0/3339357590 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f881800fb40 con 0x7f88240ccc30 2023-10-16T12:12:18.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.900+0000 7f882bd7f700 1 -- 172.21.15.16:0/3339357590 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88240b4000 con 0x7f88240ccc30 2023-10-16T12:12:18.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.900+0000 7f88167fc700 1 -- 172.21.15.16:0/3339357590 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8818014440 con 0x7f88240ccc30 2023-10-16T12:12:18.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.900+0000 7f882bd7f700 1 -- 172.21.15.16:0/3339357590 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88240b4520 con 0x7f88240ccc30 2023-10-16T12:12:18.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.901+0000 7f882bd7f700 1 -- 172.21.15.16:0/3339357590 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f882404ebb0 con 0x7f88240ccc30 2023-10-16T12:12:18.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.902+0000 7f88167fc700 1 -- 172.21.15.16:0/3339357590 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 9) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f88180145a0 con 0x7f88240ccc30 2023-10-16T12:12:18.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.902+0000 7f88167fc700 1 --2- 172.21.15.16:0/3339357590 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f880806b070 0x7f880806d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:18.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.902+0000 7f88167fc700 1 -- 172.21.15.16:0/3339357590 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7f881808a180 con 0x7f88240ccc30 2023-10-16T12:12:18.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.902+0000 7f8829b1b700 1 --2- 172.21.15.16:0/3339357590 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f880806b070 0x7f880806d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:18.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.903+0000 7f8829b1b700 1 --2- 172.21.15.16:0/3339357590 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f880806b070 0x7f880806d530 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f88200090b0 tx=0x7f8820008030 comp rx=0 tx=0).ready entity=mgr.5499 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:18.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:18.907+0000 7f88167fc700 1 -- 172.21.15.16:0/3339357590 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8818054ee0 con 0x7f88240ccc30 2023-10-16T12:12:19.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.138+0000 7f882bd7f700 1 -- 172.21.15.16:0/3339357590 --> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir5", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f88240728d0 con 0x7f880806b070 2023-10-16T12:12:19.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.198+0000 7f88167fc700 1 -- 172.21.15.16:0/3339357590 <== mgr.5499 v2:172.21.15.16:6832/1707126831 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f882000ea10 con 0x7f880806b070 2023-10-16T12:12:19.200 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule set for path /snap_test_dir1/dir5 2023-10-16T12:12:19.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.200+0000 7f882bd7f700 1 -- 172.21.15.16:0/3339357590 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f880806b070 msgr2=0x7f880806d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:19.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.200+0000 7f882bd7f700 1 --2- 172.21.15.16:0/3339357590 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f880806b070 0x7f880806d530 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f88200090b0 tx=0x7f8820008030 comp rx=0 tx=0).stop 2023-10-16T12:12:19.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.200+0000 7f882bd7f700 1 -- 172.21.15.16:0/3339357590 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88240ccc30 msgr2=0x7f88240c8510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:19.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.200+0000 7f882bd7f700 1 --2- 172.21.15.16:0/3339357590 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88240ccc30 0x7f88240c8510 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f8818011aa0 tx=0x7f8818011e60 comp rx=0 tx=0).stop 2023-10-16T12:12:19.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.200+0000 7f882bd7f700 1 -- 172.21.15.16:0/3339357590 shutdown_connections 2023-10-16T12:12:19.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.200+0000 7f882bd7f700 1 --2- 172.21.15.16:0/3339357590 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f880806b070 0x7f880806d530 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:19.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.200+0000 7f882bd7f700 1 --2- 172.21.15.16:0/3339357590 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f88240cd0b0 0x7f88240c8a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:19.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.200+0000 7f882bd7f700 1 --2- 172.21.15.16:0/3339357590 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8810003a00 0x7f88240c6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:19.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.200+0000 7f882bd7f700 1 --2- 172.21.15.16:0/3339357590 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88240ccc30 0x7f88240c8510 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:19.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.200+0000 7f882bd7f700 1 -- 172.21.15.16:0/3339357590 >> 172.21.15.16:0/3339357590 conn(0x7f88240b1f40 msgr2=0x7f88240b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:19.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.200+0000 7f882bd7f700 1 -- 172.21.15.16:0/3339357590 shutdown_connections 2023-10-16T12:12:19.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.200+0000 7f882bd7f700 1 -- 172.21.15.16:0/3339357590 wait complete. 2023-10-16T12:12:19.214 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir5 2023-10-16T12:12:19.214 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:12:19.215 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir6) 2023-10-16T12:12:19.234 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir6 1m 2023-10-16T12:12:19.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.400+0000 7f817ac22700 1 Processor -- start 2023-10-16T12:12:19.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.400+0000 7f817ac22700 1 -- start start 2023-10-16T12:12:19.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.401+0000 7f817ac22700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f817415d8d0 0x7f81740b2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:19.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.401+0000 7f817ac22700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f81740b8fa0 0x7f81740b6720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:19.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.401+0000 7f817ac22700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81740b2f30 con 0x7f81740b6c60 2023-10-16T12:12:19.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.401+0000 7f817ac22700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81740d0bb0 con 0x7f817415d8d0 2023-10-16T12:12:19.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.401+0000 7f817ac22700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f817415dcb0 con 0x7f81740b8fa0 2023-10-16T12:12:19.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.401+0000 7f81789be700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f817415d8d0 0x7f81740b2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:19.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.401+0000 7f81789be700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f817415d8d0 0x7f81740b2830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36228/0 (socket says 172.21.15.16:36228) 2023-10-16T12:12:19.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.401+0000 7f81789be700 1 -- 172.21.15.16:0/1851314412 learned_addr learned my addr 172.21.15.16:0/1851314412 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:12:19.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.401+0000 7f8173fff700 1 --2- 172.21.15.16:0/1851314412 >> v2:172.21.15.49:3300/0 conn(0x7f81740b8fa0 0x7f81740b6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:19.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.402+0000 7f81789be700 1 -- 172.21.15.16:0/1851314412 >> v2:172.21.15.49:3300/0 conn(0x7f81740b8fa0 msgr2=0x7f81740b6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:19.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.402+0000 7f81789be700 1 --2- 172.21.15.16:0/1851314412 >> v2:172.21.15.49:3300/0 conn(0x7f81740b8fa0 0x7f81740b6720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:19.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.402+0000 7f81789be700 1 -- 172.21.15.16:0/1851314412 >> v1:172.21.15.16:6789/0 conn(0x7f81740b6c60 legacy=0x7f81740ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:19.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.402+0000 7f81789be700 1 -- 172.21.15.16:0/1851314412 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81740d1c40 con 0x7f817415d8d0 2023-10-16T12:12:19.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.402+0000 7f81789be700 1 --2- 172.21.15.16:0/1851314412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f817415d8d0 0x7f81740b2830 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f81740cdc10 tx=0x7f8168009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1a6b8e80ad3e4a9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:19.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.402+0000 7f81737fe700 1 -- 172.21.15.16:0/1851314412 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8168011830 con 0x7f817415d8d0 2023-10-16T12:12:19.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.402+0000 7f81737fe700 1 -- 172.21.15.16:0/1851314412 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8168011990 con 0x7f817415d8d0 2023-10-16T12:12:19.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.402+0000 7f81737fe700 1 -- 172.21.15.16:0/1851314412 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8168011d20 con 0x7f817415d8d0 2023-10-16T12:12:19.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.402+0000 7f817ac22700 1 -- 172.21.15.16:0/1851314412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f817415d8d0 msgr2=0x7f81740b2830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:19.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.402+0000 7f817ac22700 1 --2- 172.21.15.16:0/1851314412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f817415d8d0 0x7f81740b2830 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f81740cdc10 tx=0x7f8168009bd0 comp rx=0 tx=0).stop 2023-10-16T12:12:19.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.402+0000 7f817ac22700 1 -- 172.21.15.16:0/1851314412 shutdown_connections 2023-10-16T12:12:19.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.402+0000 7f817ac22700 1 --2- 172.21.15.16:0/1851314412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f817415d8d0 0x7f81740b2830 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:19.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.402+0000 7f817ac22700 1 --2- 172.21.15.16:0/1851314412 >> v2:172.21.15.49:3300/0 conn(0x7f81740b8fa0 0x7f81740b6720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:19.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.402+0000 7f817ac22700 1 -- 172.21.15.16:0/1851314412 >> 172.21.15.16:0/1851314412 conn(0x7f8174072330 msgr2=0x7f8174072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:19.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.403+0000 7f817ac22700 1 -- 172.21.15.16:0/1851314412 shutdown_connections 2023-10-16T12:12:19.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.403+0000 7f817ac22700 1 -- 172.21.15.16:0/1851314412 wait complete. 2023-10-16T12:12:19.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.404+0000 7f817ac22700 1 Processor -- start 2023-10-16T12:12:19.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.404+0000 7f817ac22700 1 -- start start 2023-10-16T12:12:19.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.405+0000 7f817ac22700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81740b6c60 0x7f817415a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:19.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.405+0000 7f817ac22700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f81740b8fa0 0x7f8174156db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:19.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.405+0000 7f817ac22700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f817415d8d0 0x7f81740b7040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:19.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.405+0000 7f817ac22700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81740cc1a0 con 0x7f81740b6c60 2023-10-16T12:12:19.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.405+0000 7f817ac22700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81740cac00 con 0x7f817415d8d0 2023-10-16T12:12:19.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.405+0000 7f817ac22700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81740caee0 con 0x7f81740b8fa0 2023-10-16T12:12:19.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.405+0000 7f81789be700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81740b6c60 0x7f817415a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:19.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.405+0000 7f81789be700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81740b6c60 0x7f817415a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36234/0 (socket says 172.21.15.16:36234) 2023-10-16T12:12:19.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.405+0000 7f81789be700 1 -- 172.21.15.16:0/3166102223 learned_addr learned my addr 172.21.15.16:0/3166102223 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:12:19.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.405+0000 7f81791bf700 1 --2- 172.21.15.16:0/3166102223 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f817415d8d0 0x7f81740b7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:19.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.405+0000 7f8173fff700 1 --2- 172.21.15.16:0/3166102223 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f81740b8fa0 0x7f8174156db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:19.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.405+0000 7f81789be700 1 -- 172.21.15.16:0/3166102223 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f81740b8fa0 msgr2=0x7f8174156db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:19.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.405+0000 7f81789be700 1 --2- 172.21.15.16:0/3166102223 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f81740b8fa0 0x7f8174156db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:19.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.405+0000 7f81789be700 1 -- 172.21.15.16:0/3166102223 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f817415d8d0 msgr2=0x7f81740b7040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:19.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.405+0000 7f81789be700 1 --2- 172.21.15.16:0/3166102223 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f817415d8d0 0x7f81740b7040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:19.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.405+0000 7f81789be700 1 -- 172.21.15.16:0/3166102223 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8168009550 con 0x7f81740b6c60 2023-10-16T12:12:19.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.405+0000 7f8173fff700 1 --2- 172.21.15.16:0/3166102223 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f81740b8fa0 0x7f8174156db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:12:19.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.405+0000 7f81789be700 1 --2- 172.21.15.16:0/3166102223 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81740b6c60 0x7f817415a220 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f8168000c00 tx=0x7f8168004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:19.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.405+0000 7f81717fa700 1 -- 172.21.15.16:0/3166102223 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8168004570 con 0x7f81740b6c60 2023-10-16T12:12:19.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.405+0000 7f81717fa700 1 -- 172.21.15.16:0/3166102223 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f816801f070 con 0x7f81740b6c60 2023-10-16T12:12:19.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.405+0000 7f817ac22700 1 -- 172.21.15.16:0/3166102223 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81740cb1c0 con 0x7f81740b6c60 2023-10-16T12:12:19.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.405+0000 7f81717fa700 1 -- 172.21.15.16:0/3166102223 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81680038e0 con 0x7f81740b6c60 2023-10-16T12:12:19.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.405+0000 7f817ac22700 1 -- 172.21.15.16:0/3166102223 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81741f2fb0 con 0x7f81740b6c60 2023-10-16T12:12:19.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.406+0000 7f817ac22700 1 -- 172.21.15.16:0/3166102223 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f817404ebb0 con 0x7f81740b6c60 2023-10-16T12:12:19.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.407+0000 7f81717fa700 1 -- 172.21.15.16:0/3166102223 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 9) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f8168021030 con 0x7f81740b6c60 2023-10-16T12:12:19.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.407+0000 7f81717fa700 1 --2- 172.21.15.16:0/3166102223 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f815806b070 0x7f815806d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:19.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.407+0000 7f81717fa700 1 -- 172.21.15.16:0/3166102223 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7f816808aeb0 con 0x7f81740b6c60 2023-10-16T12:12:19.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.407+0000 7f8173fff700 1 --2- 172.21.15.16:0/3166102223 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f815806b070 0x7f815806d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:19.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.408+0000 7f8173fff700 1 --2- 172.21.15.16:0/3166102223 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f815806b070 0x7f815806d530 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8160009ac0 tx=0x7f8160008040 comp rx=0 tx=0).ready entity=mgr.5499 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:19.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.412+0000 7f81717fa700 1 -- 172.21.15.16:0/3166102223 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8168055ca0 con 0x7f81740b6c60 2023-10-16T12:12:19.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.563+0000 7f81717fa700 1 -- 172.21.15.16:0/3166102223 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 10) v1 ==== 89405+0+0 (secure 0 0 0) 0x7f8168056070 con 0x7f81740b6c60 2023-10-16T12:12:19.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.644+0000 7f817ac22700 1 -- 172.21.15.16:0/3166102223 --> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir6", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f81741f3140 con 0x7f815806b070 2023-10-16T12:12:19.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.700+0000 7f81717fa700 1 -- 172.21.15.16:0/3166102223 <== mgr.5499 v2:172.21.15.16:6832/1707126831 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f8160009730 con 0x7f815806b070 2023-10-16T12:12:19.702 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule set for path /snap_test_dir1/dir6 2023-10-16T12:12:19.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.702+0000 7f817ac22700 1 -- 172.21.15.16:0/3166102223 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f815806b070 msgr2=0x7f815806d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:19.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.702+0000 7f817ac22700 1 --2- 172.21.15.16:0/3166102223 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f815806b070 0x7f815806d530 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8160009ac0 tx=0x7f8160008040 comp rx=0 tx=0).stop 2023-10-16T12:12:19.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.702+0000 7f817ac22700 1 -- 172.21.15.16:0/3166102223 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81740b6c60 msgr2=0x7f817415a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:19.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.702+0000 7f817ac22700 1 --2- 172.21.15.16:0/3166102223 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81740b6c60 0x7f817415a220 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f8168000c00 tx=0x7f8168004750 comp rx=0 tx=0).stop 2023-10-16T12:12:19.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.703+0000 7f817ac22700 1 -- 172.21.15.16:0/3166102223 shutdown_connections 2023-10-16T12:12:19.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.703+0000 7f817ac22700 1 --2- 172.21.15.16:0/3166102223 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f815806b070 0x7f815806d530 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:19.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.703+0000 7f817ac22700 1 --2- 172.21.15.16:0/3166102223 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f817415d8d0 0x7f81740b7040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:19.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.703+0000 7f817ac22700 1 --2- 172.21.15.16:0/3166102223 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81740b6c60 0x7f817415a220 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:19.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.703+0000 7f817ac22700 1 --2- 172.21.15.16:0/3166102223 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f81740b8fa0 0x7f8174156db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:19.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.703+0000 7f817ac22700 1 -- 172.21.15.16:0/3166102223 >> 172.21.15.16:0/3166102223 conn(0x7f8174072330 msgr2=0x7f817415cca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:19.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.703+0000 7f817ac22700 1 -- 172.21.15.16:0/3166102223 shutdown_connections 2023-10-16T12:12:19.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.703+0000 7f817ac22700 1 -- 172.21.15.16:0/3166102223 wait complete. 2023-10-16T12:12:19.717 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir6 2023-10-16T12:12:19.717 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:12:19.717 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir7) 2023-10-16T12:12:19.736 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir7 1m 2023-10-16T12:12:19.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.903+0000 7fe2d17a2700 1 Processor -- start 2023-10-16T12:12:19.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.903+0000 7fe2d17a2700 1 -- start start 2023-10-16T12:12:19.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.904+0000 7fe2d17a2700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe2cc15b360 0x7fe2cc070cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:19.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.904+0000 7fe2d17a2700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fe2cc0b3400 0x7fe2cc0cd580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:19.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.904+0000 7fe2d17a2700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2cc06f6e0 con 0x7fe2cc0b40c0 2023-10-16T12:12:19.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.904+0000 7fe2d17a2700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2cc0cb8c0 con 0x7fe2cc15b360 2023-10-16T12:12:19.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.904+0000 7fe2d17a2700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2cc0d0c00 con 0x7fe2cc0b3400 2023-10-16T12:12:19.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.904+0000 7fe2cbfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe2cc15b360 0x7fe2cc070cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:19.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.904+0000 7fe2cbfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe2cc15b360 0x7fe2cc070cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36242/0 (socket says 172.21.15.16:36242) 2023-10-16T12:12:19.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.904+0000 7fe2cbfff700 1 -- 172.21.15.16:0/1492621910 learned_addr learned my addr 172.21.15.16:0/1492621910 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:12:19.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.904+0000 7fe2d0fa1700 1 --2- 172.21.15.16:0/1492621910 >> v2:172.21.15.49:3300/0 conn(0x7fe2cc0b3400 0x7fe2cc0cd580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:19.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.904+0000 7fe2cbfff700 1 -- 172.21.15.16:0/1492621910 >> v2:172.21.15.49:3300/0 conn(0x7fe2cc0b3400 msgr2=0x7fe2cc0cd580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:19.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.904+0000 7fe2cbfff700 1 --2- 172.21.15.16:0/1492621910 >> v2:172.21.15.49:3300/0 conn(0x7fe2cc0b3400 0x7fe2cc0cd580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:19.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.904+0000 7fe2cbfff700 1 -- 172.21.15.16:0/1492621910 >> v1:172.21.15.49:6789/0 conn(0x7fe2cc0b40c0 legacy=0x7fe2cc0b2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:19.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.904+0000 7fe2cbfff700 1 -- 172.21.15.16:0/1492621910 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2cc0cbe40 con 0x7fe2cc15b360 2023-10-16T12:12:19.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.904+0000 7fe2cbfff700 1 --2- 172.21.15.16:0/1492621910 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2cc15b360 0x7fe2cc070cc0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc071460 tx=0x7fe2bc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d58f37f40526a2d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:19.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.904+0000 7fe2caffd700 1 -- 172.21.15.16:0/1492621910 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2bc004230 con 0x7fe2cc15b360 2023-10-16T12:12:19.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.904+0000 7fe2caffd700 1 -- 172.21.15.16:0/1492621910 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2bc004390 con 0x7fe2cc15b360 2023-10-16T12:12:19.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.904+0000 7fe2caffd700 1 -- 172.21.15.16:0/1492621910 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2bc0036a0 con 0x7fe2cc15b360 2023-10-16T12:12:19.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.905+0000 7fe2d17a2700 1 -- 172.21.15.16:0/1492621910 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2cc15b360 msgr2=0x7fe2cc070cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:19.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.905+0000 7fe2d17a2700 1 --2- 172.21.15.16:0/1492621910 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2cc15b360 0x7fe2cc070cc0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc071460 tx=0x7fe2bc009bd0 comp rx=0 tx=0).stop 2023-10-16T12:12:19.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.905+0000 7fe2d17a2700 1 -- 172.21.15.16:0/1492621910 shutdown_connections 2023-10-16T12:12:19.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.905+0000 7fe2d17a2700 1 --2- 172.21.15.16:0/1492621910 >> v2:172.21.15.49:3300/0 conn(0x7fe2cc0b3400 0x7fe2cc0cd580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:19.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.905+0000 7fe2d17a2700 1 --2- 172.21.15.16:0/1492621910 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2cc15b360 0x7fe2cc070cc0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:19.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.905+0000 7fe2d17a2700 1 -- 172.21.15.16:0/1492621910 >> 172.21.15.16:0/1492621910 conn(0x7fe2cc0cdcf0 msgr2=0x7fe2cc0ce100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:19.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.905+0000 7fe2d17a2700 1 -- 172.21.15.16:0/1492621910 shutdown_connections 2023-10-16T12:12:19.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.905+0000 7fe2d17a2700 1 -- 172.21.15.16:0/1492621910 wait complete. 2023-10-16T12:12:19.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.906+0000 7fe2d17a2700 1 Processor -- start 2023-10-16T12:12:19.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.906+0000 7fe2d17a2700 1 -- start start 2023-10-16T12:12:19.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.907+0000 7fe2d17a2700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe2cc0b3400 0x7fe2cc0c9300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:19.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.907+0000 7fe2d17a2700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2cc0b40c0 0x7fe2cc0afea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:19.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.907+0000 7fe2d17a2700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe2cc15b360 0x7fe2cc0c8340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:19.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.907+0000 7fe2d17a2700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2cc1541a0 con 0x7fe2cc0b40c0 2023-10-16T12:12:19.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.907+0000 7fe2d17a2700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2cc06d290 con 0x7fe2cc15b360 2023-10-16T12:12:19.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.907+0000 7fe2d17a2700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2cc06d570 con 0x7fe2cc0b3400 2023-10-16T12:12:19.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.907+0000 7fe2cb7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2cc0b40c0 0x7fe2cc0afea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:19.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.907+0000 7fe2cb7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2cc0b40c0 0x7fe2cc0afea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36250/0 (socket says 172.21.15.16:36250) 2023-10-16T12:12:19.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.907+0000 7fe2cb7fe700 1 -- 172.21.15.16:0/1918290822 learned_addr learned my addr 172.21.15.16:0/1918290822 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:12:19.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.907+0000 7fe2cbfff700 1 --2- 172.21.15.16:0/1918290822 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe2cc0b3400 0x7fe2cc0c9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:19.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.907+0000 7fe2d0fa1700 1 --2- 172.21.15.16:0/1918290822 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe2cc15b360 0x7fe2cc0c8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:19.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.908+0000 7fe2cb7fe700 1 -- 172.21.15.16:0/1918290822 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe2cc0b3400 msgr2=0x7fe2cc0c9300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:19.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.908+0000 7fe2cb7fe700 1 --2- 172.21.15.16:0/1918290822 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe2cc0b3400 0x7fe2cc0c9300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:19.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.908+0000 7fe2cb7fe700 1 -- 172.21.15.16:0/1918290822 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe2cc15b360 msgr2=0x7fe2cc0c8340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:19.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.908+0000 7fe2cb7fe700 1 --2- 172.21.15.16:0/1918290822 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe2cc15b360 0x7fe2cc0c8340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:19.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.908+0000 7fe2cb7fe700 1 -- 172.21.15.16:0/1918290822 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2bc009550 con 0x7fe2cc0b40c0 2023-10-16T12:12:19.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.908+0000 7fe2cbfff700 1 --2- 172.21.15.16:0/1918290822 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe2cc0b3400 0x7fe2cc0c9300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:12:19.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.908+0000 7fe2d0fa1700 1 --2- 172.21.15.16:0/1918290822 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe2cc15b360 0x7fe2cc0c8340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:12:19.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.908+0000 7fe2cb7fe700 1 --2- 172.21.15.16:0/1918290822 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2cc0b40c0 0x7fe2cc0afea0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fe2c000c4d0 tx=0x7fe2c000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:19.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.908+0000 7fe2c8ff9700 1 -- 172.21.15.16:0/1918290822 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c0011790 con 0x7fe2cc0b40c0 2023-10-16T12:12:19.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.908+0000 7fe2c8ff9700 1 -- 172.21.15.16:0/1918290822 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2c00118f0 con 0x7fe2cc0b40c0 2023-10-16T12:12:19.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.908+0000 7fe2d17a2700 1 -- 172.21.15.16:0/1918290822 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2cc154410 con 0x7fe2cc0b40c0 2023-10-16T12:12:19.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.908+0000 7fe2c8ff9700 1 -- 172.21.15.16:0/1918290822 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c0011ca0 con 0x7fe2cc0b40c0 2023-10-16T12:12:19.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.908+0000 7fe2d17a2700 1 -- 172.21.15.16:0/1918290822 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2cc06d850 con 0x7fe2cc0b40c0 2023-10-16T12:12:19.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.909+0000 7fe2d17a2700 1 -- 172.21.15.16:0/1918290822 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2cc154630 con 0x7fe2cc0b40c0 2023-10-16T12:12:19.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.909+0000 7fe2c8ff9700 1 -- 172.21.15.16:0/1918290822 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 10) v1 ==== 89405+0+0 (secure 0 0 0) 0x7fe2c001c460 con 0x7fe2cc0b40c0 2023-10-16T12:12:19.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.910+0000 7fe2c8ff9700 1 --2- 172.21.15.16:0/1918290822 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fe2b806b100 0x7fe2b806d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:19.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.910+0000 7fe2c8ff9700 1 -- 172.21.15.16:0/1918290822 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7fe2c0089a70 con 0x7fe2cc0b40c0 2023-10-16T12:12:19.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.910+0000 7fe2cbfff700 1 --2- 172.21.15.16:0/1918290822 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fe2b806b100 0x7fe2b806d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:19.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.910+0000 7fe2cbfff700 1 --2- 172.21.15.16:0/1918290822 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fe2b806b100 0x7fe2b806d5c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe2bc009130 tx=0x7fe2bc008030 comp rx=0 tx=0).ready entity=mgr.5499 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:19.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:19.914+0000 7fe2c8ff9700 1 -- 172.21.15.16:0/1918290822 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe2c0054660 con 0x7fe2cc0b40c0 2023-10-16T12:12:20.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.144+0000 7fe2d17a2700 1 -- 172.21.15.16:0/1918290822 --> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir7", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fe2cc0cbad0 con 0x7fe2b806b100 2023-10-16T12:12:20.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.206+0000 7fe2c8ff9700 1 -- 172.21.15.16:0/1918290822 <== mgr.5499 v2:172.21.15.16:6832/1707126831 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fe2bc0041b0 con 0x7fe2b806b100 2023-10-16T12:12:20.208 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule set for path /snap_test_dir1/dir7 2023-10-16T12:12:20.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.208+0000 7fe2d17a2700 1 -- 172.21.15.16:0/1918290822 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fe2b806b100 msgr2=0x7fe2b806d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:20.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.208+0000 7fe2d17a2700 1 --2- 172.21.15.16:0/1918290822 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fe2b806b100 0x7fe2b806d5c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe2bc009130 tx=0x7fe2bc008030 comp rx=0 tx=0).stop 2023-10-16T12:12:20.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.208+0000 7fe2d17a2700 1 -- 172.21.15.16:0/1918290822 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2cc0b40c0 msgr2=0x7fe2cc0afea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:20.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.208+0000 7fe2d17a2700 1 --2- 172.21.15.16:0/1918290822 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2cc0b40c0 0x7fe2cc0afea0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fe2c000c4d0 tx=0x7fe2c000c890 comp rx=0 tx=0).stop 2023-10-16T12:12:20.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.208+0000 7fe2d17a2700 1 -- 172.21.15.16:0/1918290822 shutdown_connections 2023-10-16T12:12:20.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.208+0000 7fe2d17a2700 1 --2- 172.21.15.16:0/1918290822 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fe2b806b100 0x7fe2b806d5c0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:20.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.208+0000 7fe2d17a2700 1 --2- 172.21.15.16:0/1918290822 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe2cc15b360 0x7fe2cc0c8340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:20.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.208+0000 7fe2d17a2700 1 --2- 172.21.15.16:0/1918290822 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe2cc0b3400 0x7fe2cc0c9300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:20.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.208+0000 7fe2d17a2700 1 --2- 172.21.15.16:0/1918290822 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2cc0b40c0 0x7fe2cc0afea0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:20.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.208+0000 7fe2d17a2700 1 -- 172.21.15.16:0/1918290822 >> 172.21.15.16:0/1918290822 conn(0x7fe2cc0cdcf0 msgr2=0x7fe2cc0b0890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:20.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.209+0000 7fe2d17a2700 1 -- 172.21.15.16:0/1918290822 shutdown_connections 2023-10-16T12:12:20.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.209+0000 7fe2d17a2700 1 -- 172.21.15.16:0/1918290822 wait complete. 2023-10-16T12:12:20.222 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir7 2023-10-16T12:12:20.223 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:12:20.223 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir8) 2023-10-16T12:12:20.242 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir8 1m 2023-10-16T12:12:20.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.411+0000 7f22a1688700 1 Processor -- start 2023-10-16T12:12:20.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.411+0000 7f22a1688700 1 -- start start 2023-10-16T12:12:20.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.411+0000 7f22a1688700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f229c0cb100 0x7f229c0ce150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:20.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.411+0000 7f22a1688700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f229c0cef10 con 0x7f229c0ce690 2023-10-16T12:12:20.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.411+0000 7f22a1688700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f229c0cf1f0 con 0x7f229c157e30 2023-10-16T12:12:20.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.411+0000 7f22a1688700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f229c158210 con 0x7f229c0cb100 2023-10-16T12:12:20.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.412+0000 7f229b7fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f229c0ce690 0x7f229c06ada0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:53766/0 (socket says 172.21.15.16:53766) 2023-10-16T12:12:20.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.412+0000 7f229b7fe700 1 -- 172.21.15.16:0/1821038843 learned_addr learned my addr 172.21.15.16:0/1821038843 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:12:20.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.412+0000 7f229a7fc700 1 --2- 172.21.15.16:0/1821038843 >> v2:172.21.15.49:3300/0 conn(0x7f229c0cb100 0x7f229c0ce150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:20.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.412+0000 7f2299ffb700 1 -- 172.21.15.16:0/1821038843 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 540016052 0 0) 0x7f229c0cef10 con 0x7f229c0ce690 2023-10-16T12:12:20.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.412+0000 7f2299ffb700 1 -- 172.21.15.16:0/1821038843 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22840035b0 con 0x7f229c0ce690 2023-10-16T12:12:20.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.412+0000 7f2299ffb700 1 -- 172.21.15.16:0/1821038843 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1917523430 0 0) 0x7f22840035b0 con 0x7f229c0ce690 2023-10-16T12:12:20.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.412+0000 7f2299ffb700 1 -- 172.21.15.16:0/1821038843 >> v2:172.21.15.49:3300/0 conn(0x7f229c0cb100 msgr2=0x7f229c0ce150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:20.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.412+0000 7f2299ffb700 1 --2- 172.21.15.16:0/1821038843 >> v2:172.21.15.49:3300/0 conn(0x7f229c0cb100 0x7f229c0ce150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:20.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.412+0000 7f2299ffb700 1 -- 172.21.15.16:0/1821038843 >> v1:172.21.15.49:6789/0 conn(0x7f229c157e30 legacy=0x7f229c0ca9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:20.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.412+0000 7f2299ffb700 1 -- 172.21.15.16:0/1821038843 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f229c0cf4d0 con 0x7f229c0ce690 2023-10-16T12:12:20.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.412+0000 7f2299ffb700 1 -- 172.21.15.16:0/1821038843 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f22940019f0 con 0x7f229c0ce690 2023-10-16T12:12:20.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.412+0000 7f2299ffb700 1 -- 172.21.15.16:0/1821038843 >> v1:172.21.15.16:6789/0 conn(0x7f229c0ce690 legacy=0x7f229c06ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:20.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.412+0000 7f2299ffb700 1 --2- 172.21.15.16:0/1821038843 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2284003a00 0x7f2284003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:20.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.412+0000 7f2299ffb700 1 --2- 172.21.15.16:0/1821038843 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2284004850 0x7f2284004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:20.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.412+0000 7f2299ffb700 1 --2- 172.21.15.16:0/1821038843 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2284005210 0x7f228400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:20.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.412+0000 7f2299ffb700 1 -- 172.21.15.16:0/1821038843 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f229c158210 con 0x7f2284005210 2023-10-16T12:12:20.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.412+0000 7f2299ffb700 1 -- 172.21.15.16:0/1821038843 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f228400c0b0 con 0x7f2284004850 2023-10-16T12:12:20.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.412+0000 7f2299ffb700 1 -- 172.21.15.16:0/1821038843 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f228400c270 con 0x7f2284003a00 2023-10-16T12:12:20.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.412+0000 7f229a7fc700 1 --2- 172.21.15.16:0/1821038843 >> v2:172.21.15.49:3300/0 conn(0x7f229c0cb100 0x7f229c0ce150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:12:20.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.412+0000 7f229b7fe700 1 --2- 172.21.15.16:0/1821038843 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2284005210 0x7f228400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:20.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.412+0000 7f229affd700 1 --2- 172.21.15.16:0/1821038843 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2284003a00 0x7f2284003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:20.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.413+0000 7f229a7fc700 1 --2- 172.21.15.16:0/1821038843 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2284004850 0x7f2284004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:20.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.413+0000 7f229b7fe700 1 -- 172.21.15.16:0/1821038843 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2284003a00 msgr2=0x7f2284003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:20.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.413+0000 7f229b7fe700 1 --2- 172.21.15.16:0/1821038843 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2284003a00 0x7f2284003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:20.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.413+0000 7f229b7fe700 1 -- 172.21.15.16:0/1821038843 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2284004850 msgr2=0x7f2284004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:20.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.413+0000 7f229b7fe700 1 --2- 172.21.15.16:0/1821038843 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2284004850 0x7f2284004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:20.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.413+0000 7f229b7fe700 1 -- 172.21.15.16:0/1821038843 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22840035b0 con 0x7f2284005210 2023-10-16T12:12:20.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.413+0000 7f229affd700 1 --2- 172.21.15.16:0/1821038843 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2284003a00 0x7f2284003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:12:20.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.413+0000 7f229b7fe700 1 -- 172.21.15.16:0/1821038843 reap_dead start 2023-10-16T12:12:20.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.413+0000 7f229b7fe700 1 --2- 172.21.15.16:0/1821038843 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2284005210 0x7f228400ba10 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f229400a830 tx=0x7f2284005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6eff677947c8cfb9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:20.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.413+0000 7f2299ffb700 1 -- 172.21.15.16:0/1821038843 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f229400a2f0 con 0x7f2284005210 2023-10-16T12:12:20.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.413+0000 7f2299ffb700 1 -- 172.21.15.16:0/1821038843 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2294012620 con 0x7f2284005210 2023-10-16T12:12:20.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.413+0000 7f2299ffb700 1 -- 172.21.15.16:0/1821038843 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22940129d0 con 0x7f2284005210 2023-10-16T12:12:20.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.413+0000 7f22a1688700 1 -- 172.21.15.16:0/1821038843 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2284005210 msgr2=0x7f228400ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:20.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.413+0000 7f22a1688700 1 --2- 172.21.15.16:0/1821038843 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2284005210 0x7f228400ba10 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f229400a830 tx=0x7f2284005880 comp rx=0 tx=0).stop 2023-10-16T12:12:20.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.413+0000 7f22a1688700 1 -- 172.21.15.16:0/1821038843 shutdown_connections 2023-10-16T12:12:20.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.413+0000 7f22a1688700 1 --2- 172.21.15.16:0/1821038843 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2284005210 0x7f228400ba10 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:20.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.413+0000 7f22a1688700 1 -- 172.21.15.16:0/1821038843 >> 172.21.15.16:0/1821038843 conn(0x7f229c0cc5c0 msgr2=0x7f229c0cc9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:20.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.414+0000 7f22a1688700 1 -- 172.21.15.16:0/1821038843 shutdown_connections 2023-10-16T12:12:20.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.414+0000 7f22a1688700 1 -- 172.21.15.16:0/1821038843 wait complete. 2023-10-16T12:12:20.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.415+0000 7f22a1688700 1 Processor -- start 2023-10-16T12:12:20.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.415+0000 7f22a1688700 1 -- start start 2023-10-16T12:12:20.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.416+0000 7f22a1688700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2284005210 0x7f229c155130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:20.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.416+0000 7f22a1688700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f229c1603a0 0x7f229c160960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:20.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.416+0000 7f22a1688700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f229c0ada50 0x7f229c0aded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:20.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.416+0000 7f22a1688700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f229c0b4200 con 0x7f229c1603a0 2023-10-16T12:12:20.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.416+0000 7f22a1688700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f229c155670 con 0x7f2284005210 2023-10-16T12:12:20.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.416+0000 7f22a1688700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f229c0ae410 con 0x7f229c0ada50 2023-10-16T12:12:20.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.416+0000 7f229a7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f229c1603a0 0x7f229c160960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:20.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.416+0000 7f229a7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f229c1603a0 0x7f229c160960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36270/0 (socket says 172.21.15.16:36270) 2023-10-16T12:12:20.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.416+0000 7f229a7fc700 1 -- 172.21.15.16:0/3066326544 learned_addr learned my addr 172.21.15.16:0/3066326544 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:12:20.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.416+0000 7f229b7fe700 1 --2- 172.21.15.16:0/3066326544 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f229c0ada50 0x7f229c0aded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:20.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.416+0000 7f229affd700 1 --2- 172.21.15.16:0/3066326544 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2284005210 0x7f229c155130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:20.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.416+0000 7f229a7fc700 1 -- 172.21.15.16:0/3066326544 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f229c0ada50 msgr2=0x7f229c0aded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:20.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.416+0000 7f229a7fc700 1 --2- 172.21.15.16:0/3066326544 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f229c0ada50 0x7f229c0aded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:20.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.416+0000 7f229a7fc700 1 -- 172.21.15.16:0/3066326544 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2284005210 msgr2=0x7f229c155130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:20.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.416+0000 7f229a7fc700 1 --2- 172.21.15.16:0/3066326544 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2284005210 0x7f229c155130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:20.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.416+0000 7f229a7fc700 1 -- 172.21.15.16:0/3066326544 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f229c160ea0 con 0x7f229c1603a0 2023-10-16T12:12:20.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.416+0000 7f229affd700 1 --2- 172.21.15.16:0/3066326544 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2284005210 0x7f229c155130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:12:20.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.416+0000 7f229a7fc700 1 --2- 172.21.15.16:0/3066326544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f229c1603a0 0x7f229c160960 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f2290010a30 tx=0x7f2290010d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:20.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.416+0000 7f22837fe700 1 -- 172.21.15.16:0/3066326544 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f229000e870 con 0x7f229c1603a0 2023-10-16T12:12:20.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.416+0000 7f22837fe700 1 -- 172.21.15.16:0/3066326544 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f229000e9d0 con 0x7f229c1603a0 2023-10-16T12:12:20.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.416+0000 7f22a1688700 1 -- 172.21.15.16:0/3066326544 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f229c161ce0 con 0x7f229c1603a0 2023-10-16T12:12:20.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.416+0000 7f22837fe700 1 -- 172.21.15.16:0/3066326544 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f229000ed60 con 0x7f229c1603a0 2023-10-16T12:12:20.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.416+0000 7f22a1688700 1 -- 172.21.15.16:0/3066326544 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f229c1613b0 con 0x7f229c1603a0 2023-10-16T12:12:20.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.417+0000 7f22a1688700 1 -- 172.21.15.16:0/3066326544 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f229c158350 con 0x7f229c1603a0 2023-10-16T12:12:20.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.418+0000 7f22837fe700 1 -- 172.21.15.16:0/3066326544 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 10) v1 ==== 89405+0+0 (secure 0 0 0) 0x7f2290016460 con 0x7f229c1603a0 2023-10-16T12:12:20.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.418+0000 7f22837fe700 1 --2- 172.21.15.16:0/3066326544 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f228806b100 0x7f228806d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:20.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.418+0000 7f22837fe700 1 -- 172.21.15.16:0/3066326544 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7f2290089d30 con 0x7f229c1603a0 2023-10-16T12:12:20.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.418+0000 7f229affd700 1 --2- 172.21.15.16:0/3066326544 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f228806b100 0x7f228806d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:20.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.419+0000 7f229affd700 1 --2- 172.21.15.16:0/3066326544 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f228806b100 0x7f228806d5c0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f228c000c00 tx=0x7f228c009040 comp rx=0 tx=0).ready entity=mgr.5499 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:20.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.423+0000 7f22837fe700 1 -- 172.21.15.16:0/3066326544 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f229c158350 con 0x7f229c1603a0 2023-10-16T12:12:20.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.654+0000 7f22a1688700 1 -- 172.21.15.16:0/3066326544 --> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir8", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f229c06a880 con 0x7f228806b100 2023-10-16T12:12:20.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.713+0000 7f22837fe700 1 -- 172.21.15.16:0/3066326544 <== mgr.5499 v2:172.21.15.16:6832/1707126831 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f228c003e80 con 0x7f228806b100 2023-10-16T12:12:20.715 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule set for path /snap_test_dir1/dir8 2023-10-16T12:12:20.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.715+0000 7f22a1688700 1 -- 172.21.15.16:0/3066326544 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f228806b100 msgr2=0x7f228806d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:20.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.715+0000 7f22a1688700 1 --2- 172.21.15.16:0/3066326544 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f228806b100 0x7f228806d5c0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f228c000c00 tx=0x7f228c009040 comp rx=0 tx=0).stop 2023-10-16T12:12:20.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.715+0000 7f22a1688700 1 -- 172.21.15.16:0/3066326544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f229c1603a0 msgr2=0x7f229c160960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:20.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.715+0000 7f22a1688700 1 --2- 172.21.15.16:0/3066326544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f229c1603a0 0x7f229c160960 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f2290010a30 tx=0x7f2290010d40 comp rx=0 tx=0).stop 2023-10-16T12:12:20.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.715+0000 7f22a1688700 1 -- 172.21.15.16:0/3066326544 shutdown_connections 2023-10-16T12:12:20.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.715+0000 7f22a1688700 1 --2- 172.21.15.16:0/3066326544 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f228806b100 0x7f228806d5c0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:20.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.715+0000 7f22a1688700 1 --2- 172.21.15.16:0/3066326544 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f229c0ada50 0x7f229c0aded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:20.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.715+0000 7f22a1688700 1 --2- 172.21.15.16:0/3066326544 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2284005210 0x7f229c155130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:20.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.715+0000 7f22a1688700 1 --2- 172.21.15.16:0/3066326544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f229c1603a0 0x7f229c160960 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:20.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.715+0000 7f22a1688700 1 -- 172.21.15.16:0/3066326544 >> 172.21.15.16:0/3066326544 conn(0x7f229c0cc5c0 msgr2=0x7f229c15dd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:20.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.715+0000 7f22a1688700 1 -- 172.21.15.16:0/3066326544 shutdown_connections 2023-10-16T12:12:20.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.715+0000 7f22a1688700 1 -- 172.21.15.16:0/3066326544 wait complete. 2023-10-16T12:12:20.729 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir8 2023-10-16T12:12:20.730 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:12:20.730 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir9) 2023-10-16T12:12:20.749 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir9 1m 2023-10-16T12:12:20.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.914+0000 7f40e8dd3700 1 Processor -- start 2023-10-16T12:12:20.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.914+0000 7f40e8dd3700 1 -- start start 2023-10-16T12:12:20.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.915+0000 7f40e8dd3700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f40e40ce6a0 0x7f40e406c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:20.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.915+0000 7f40e8dd3700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f40e40d2110 0x7f40e406c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:20.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.915+0000 7f40e8dd3700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f40e40b2620 0x7f40e40b0f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:20.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.915+0000 7f40e8dd3700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40e40b20f0 con 0x7f40e40ce6a0 2023-10-16T12:12:20.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.915+0000 7f40e8dd3700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40e40d03d0 con 0x7f40e40b2620 2023-10-16T12:12:20.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.915+0000 7f40e8dd3700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40e40b2340 con 0x7f40e40d2110 2023-10-16T12:12:20.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.915+0000 7f40e259c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f40e40ce6a0 0x7f40e406c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:20.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.916+0000 7f40e259c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f40e40ce6a0 0x7f40e406c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36276/0 (socket says 172.21.15.16:36276) 2023-10-16T12:12:20.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.916+0000 7f40e259c700 1 -- 172.21.15.16:0/2624755319 learned_addr learned my addr 172.21.15.16:0/2624755319 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:12:20.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.916+0000 7f40e1d9b700 1 --2- 172.21.15.16:0/2624755319 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f40e40d2110 0x7f40e406c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:20.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.916+0000 7f40e2d9d700 1 --2- 172.21.15.16:0/2624755319 >> v2:172.21.15.49:3300/0 conn(0x7f40e40b2620 0x7f40e40b0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:20.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.916+0000 7f40e259c700 1 -- 172.21.15.16:0/2624755319 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f40e40d2110 msgr2=0x7f40e406c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:20.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.916+0000 7f40e259c700 1 --2- 172.21.15.16:0/2624755319 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f40e40d2110 0x7f40e406c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:20.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.916+0000 7f40e259c700 1 -- 172.21.15.16:0/2624755319 >> v2:172.21.15.49:3300/0 conn(0x7f40e40b2620 msgr2=0x7f40e40b0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:20.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.916+0000 7f40e259c700 1 --2- 172.21.15.16:0/2624755319 >> v2:172.21.15.49:3300/0 conn(0x7f40e40b2620 0x7f40e40b0f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:20.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.916+0000 7f40e259c700 1 -- 172.21.15.16:0/2624755319 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40e406bb00 con 0x7f40e40ce6a0 2023-10-16T12:12:20.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.916+0000 7f40e259c700 1 --2- 172.21.15.16:0/2624755319 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40e40ce6a0 0x7f40e406c300 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f40e408bcb0 tx=0x7f40d4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51c4912c391d16d6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:20.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.916+0000 7f40e159a700 1 -- 172.21.15.16:0/2624755319 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40d400b870 con 0x7f40e40ce6a0 2023-10-16T12:12:20.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.916+0000 7f40e159a700 1 -- 172.21.15.16:0/2624755319 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40d400b9d0 con 0x7f40e40ce6a0 2023-10-16T12:12:20.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.916+0000 7f40e159a700 1 -- 172.21.15.16:0/2624755319 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40d400bd60 con 0x7f40e40ce6a0 2023-10-16T12:12:20.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.917+0000 7f40e8dd3700 1 -- 172.21.15.16:0/2624755319 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40e40ce6a0 msgr2=0x7f40e406c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:20.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.917+0000 7f40e8dd3700 1 --2- 172.21.15.16:0/2624755319 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40e40ce6a0 0x7f40e406c300 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f40e408bcb0 tx=0x7f40d4009bd0 comp rx=0 tx=0).stop 2023-10-16T12:12:20.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.917+0000 7f40e8dd3700 1 -- 172.21.15.16:0/2624755319 shutdown_connections 2023-10-16T12:12:20.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.917+0000 7f40e8dd3700 1 --2- 172.21.15.16:0/2624755319 >> v2:172.21.15.49:3300/0 conn(0x7f40e40b2620 0x7f40e40b0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:20.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.917+0000 7f40e8dd3700 1 --2- 172.21.15.16:0/2624755319 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40e40ce6a0 0x7f40e406c300 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:20.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.917+0000 7f40e8dd3700 1 --2- 172.21.15.16:0/2624755319 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f40e40d2110 0x7f40e406c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:20.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.917+0000 7f40e8dd3700 1 -- 172.21.15.16:0/2624755319 >> 172.21.15.16:0/2624755319 conn(0x7f40e40d15f0 msgr2=0x7f40e40d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:20.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.917+0000 7f40e8dd3700 1 -- 172.21.15.16:0/2624755319 shutdown_connections 2023-10-16T12:12:20.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.917+0000 7f40e8dd3700 1 -- 172.21.15.16:0/2624755319 wait complete. 2023-10-16T12:12:20.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.918+0000 7f40e8dd3700 1 Processor -- start 2023-10-16T12:12:20.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.918+0000 7f40e8dd3700 1 -- start start 2023-10-16T12:12:20.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.919+0000 7f40e8dd3700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f40e40b2620 0x7f40e40c5120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:20.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.919+0000 7f40e8dd3700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f40e40ce6a0 0x7f40e41f3be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:20.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.919+0000 7f40e8dd3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40e40d2110 0x7f40e41f05d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:20.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.919+0000 7f40e8dd3700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40e40c5660 con 0x7f40e40d2110 2023-10-16T12:12:20.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.919+0000 7f40e8dd3700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40e40c5890 con 0x7f40e40ce6a0 2023-10-16T12:12:20.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.919+0000 7f40e8dd3700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40e40c5b70 con 0x7f40e40b2620 2023-10-16T12:12:20.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.919+0000 7f40e2d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40e40d2110 0x7f40e41f05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:20.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.919+0000 7f40e2d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40e40d2110 0x7f40e41f05d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36286/0 (socket says 172.21.15.16:36286) 2023-10-16T12:12:20.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.919+0000 7f40e2d9d700 1 -- 172.21.15.16:0/2543616387 learned_addr learned my addr 172.21.15.16:0/2543616387 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:12:20.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.919+0000 7f40e259c700 1 --2- 172.21.15.16:0/2543616387 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f40e40b2620 0x7f40e40c5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:20.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.919+0000 7f40e1d9b700 1 --2- 172.21.15.16:0/2543616387 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f40e40ce6a0 0x7f40e41f3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:20.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.919+0000 7f40e2d9d700 1 -- 172.21.15.16:0/2543616387 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f40e40b2620 msgr2=0x7f40e40c5120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:20.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.919+0000 7f40e2d9d700 1 --2- 172.21.15.16:0/2543616387 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f40e40b2620 0x7f40e40c5120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:20.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.919+0000 7f40e2d9d700 1 -- 172.21.15.16:0/2543616387 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f40e40ce6a0 msgr2=0x7f40e41f3be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:20.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.919+0000 7f40e2d9d700 1 --2- 172.21.15.16:0/2543616387 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f40e40ce6a0 0x7f40e41f3be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:20.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.919+0000 7f40e2d9d700 1 -- 172.21.15.16:0/2543616387 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40d4009550 con 0x7f40e40d2110 2023-10-16T12:12:20.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.919+0000 7f40e1d9b700 1 --2- 172.21.15.16:0/2543616387 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f40e40ce6a0 0x7f40e41f3be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:12:20.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.919+0000 7f40e259c700 1 --2- 172.21.15.16:0/2543616387 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f40e40b2620 0x7f40e40c5120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:12:20.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.920+0000 7f40e2d9d700 1 --2- 172.21.15.16:0/2543616387 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40e40d2110 0x7f40e41f05d0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f40dc007e60 tx=0x7f40dc00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:20.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.920+0000 7f40daffd700 1 -- 172.21.15.16:0/2543616387 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40dc00ede0 con 0x7f40e40d2110 2023-10-16T12:12:20.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.920+0000 7f40daffd700 1 -- 172.21.15.16:0/2543616387 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40dc014420 con 0x7f40e40d2110 2023-10-16T12:12:20.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.920+0000 7f40e8dd3700 1 -- 172.21.15.16:0/2543616387 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40e40c5e50 con 0x7f40e40d2110 2023-10-16T12:12:20.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.920+0000 7f40daffd700 1 -- 172.21.15.16:0/2543616387 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40dc0147d0 con 0x7f40e40d2110 2023-10-16T12:12:20.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.920+0000 7f40e8dd3700 1 -- 172.21.15.16:0/2543616387 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40e40c6410 con 0x7f40e40d2110 2023-10-16T12:12:20.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.921+0000 7f40e8dd3700 1 -- 172.21.15.16:0/2543616387 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40e404ebb0 con 0x7f40e40d2110 2023-10-16T12:12:20.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.921+0000 7f40daffd700 1 -- 172.21.15.16:0/2543616387 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 10) v1 ==== 89405+0+0 (secure 0 0 0) 0x7f40dc014930 con 0x7f40e40d2110 2023-10-16T12:12:20.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.922+0000 7f40daffd700 1 --2- 172.21.15.16:0/2543616387 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f40c406b100 0x7f40c406d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:20.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.922+0000 7f40daffd700 1 -- 172.21.15.16:0/2543616387 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7f40dc08a2a0 con 0x7f40e40d2110 2023-10-16T12:12:20.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.922+0000 7f40e259c700 1 --2- 172.21.15.16:0/2543616387 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f40c406b100 0x7f40c406d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:20.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.922+0000 7f40e259c700 1 --2- 172.21.15.16:0/2543616387 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f40c406b100 0x7f40c406d5c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f40d4000c00 tx=0x7f40d4008030 comp rx=0 tx=0).ready entity=mgr.5499 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:20.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:20.926+0000 7f40daffd700 1 -- 172.21.15.16:0/2543616387 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f40dc054f20 con 0x7f40e40d2110 2023-10-16T12:12:21.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.157+0000 7f40e8dd3700 1 -- 172.21.15.16:0/2543616387 --> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir9", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f40e40d0690 con 0x7f40c406b100 2023-10-16T12:12:21.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.214+0000 7f40daffd700 1 -- 172.21.15.16:0/2543616387 <== mgr.5499 v2:172.21.15.16:6832/1707126831 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f40d400bbc0 con 0x7f40c406b100 2023-10-16T12:12:21.216 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule set for path /snap_test_dir1/dir9 2023-10-16T12:12:21.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.216+0000 7f40e8dd3700 1 -- 172.21.15.16:0/2543616387 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f40c406b100 msgr2=0x7f40c406d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:21.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.216+0000 7f40e8dd3700 1 --2- 172.21.15.16:0/2543616387 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f40c406b100 0x7f40c406d5c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f40d4000c00 tx=0x7f40d4008030 comp rx=0 tx=0).stop 2023-10-16T12:12:21.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.216+0000 7f40e8dd3700 1 -- 172.21.15.16:0/2543616387 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40e40d2110 msgr2=0x7f40e41f05d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:21.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.216+0000 7f40e8dd3700 1 --2- 172.21.15.16:0/2543616387 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40e40d2110 0x7f40e41f05d0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f40dc007e60 tx=0x7f40dc00e3c0 comp rx=0 tx=0).stop 2023-10-16T12:12:21.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.216+0000 7f40e8dd3700 1 -- 172.21.15.16:0/2543616387 shutdown_connections 2023-10-16T12:12:21.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.216+0000 7f40e8dd3700 1 --2- 172.21.15.16:0/2543616387 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f40c406b100 0x7f40c406d5c0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:21.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.216+0000 7f40e8dd3700 1 --2- 172.21.15.16:0/2543616387 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40e40d2110 0x7f40e41f05d0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:21.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.216+0000 7f40e8dd3700 1 --2- 172.21.15.16:0/2543616387 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f40e40b2620 0x7f40e40c5120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:21.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.216+0000 7f40e8dd3700 1 --2- 172.21.15.16:0/2543616387 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f40e40ce6a0 0x7f40e41f3be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:21.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.216+0000 7f40e8dd3700 1 -- 172.21.15.16:0/2543616387 >> 172.21.15.16:0/2543616387 conn(0x7f40e40d15f0 msgr2=0x7f40e40b07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:21.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.216+0000 7f40e8dd3700 1 -- 172.21.15.16:0/2543616387 shutdown_connections 2023-10-16T12:12:21.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.216+0000 7f40e8dd3700 1 -- 172.21.15.16:0/2543616387 wait complete. 2023-10-16T12:12:21.230 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir9 2023-10-16T12:12:21.230 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir0 1m 2023-10-16T12:12:21.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.357+0000 7f62b29c9700 1 Processor -- start 2023-10-16T12:12:21.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.357+0000 7f62b29c9700 1 -- start start 2023-10-16T12:12:21.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.357+0000 7f62b29c9700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f62ac15ec80 0x7f62ac0cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:21.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.357+0000 7f62b29c9700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f62ac0d2110 0x7f62ac0cd1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:21.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.357+0000 7f62b29c9700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62ac0b2440 con 0x7f62ac15f060 2023-10-16T12:12:21.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.357+0000 7f62b29c9700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62ac0b2c50 con 0x7f62ac0d2110 2023-10-16T12:12:21.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.357+0000 7f62b29c9700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62ac0d00e0 con 0x7f62ac15ec80 2023-10-16T12:12:21.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.358+0000 7f62ab7fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f62ac15f060 0x7f62ac0b4440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:53796/0 (socket says 172.21.15.16:53796) 2023-10-16T12:12:21.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.358+0000 7f62ab7fe700 1 -- 172.21.15.16:0/625938847 learned_addr learned my addr 172.21.15.16:0/625938847 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:12:21.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.358+0000 7f62b0f66700 1 --2- 172.21.15.16:0/625938847 >> v2:172.21.15.49:3300/0 conn(0x7f62ac0d2110 0x7f62ac0cd1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:21.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.358+0000 7f62abfff700 1 --2- 172.21.15.16:0/625938847 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f62ac15ec80 0x7f62ac0cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:21.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.358+0000 7f62aaffd700 1 -- 172.21.15.16:0/625938847 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2125408616 0 0) 0x7f62ac0b2440 con 0x7f62ac15f060 2023-10-16T12:12:21.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.358+0000 7f62aaffd700 1 -- 172.21.15.16:0/625938847 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62940035b0 con 0x7f62ac15f060 2023-10-16T12:12:21.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.358+0000 7f62aaffd700 1 -- 172.21.15.16:0/625938847 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2600572395 0 0) 0x7f62940035b0 con 0x7f62ac15f060 2023-10-16T12:12:21.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.358+0000 7f62aaffd700 1 -- 172.21.15.16:0/625938847 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f62ac15ec80 msgr2=0x7f62ac0cdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:21.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.358+0000 7f62aaffd700 1 --2- 172.21.15.16:0/625938847 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f62ac15ec80 0x7f62ac0cdec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:21.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.358+0000 7f62aaffd700 1 -- 172.21.15.16:0/625938847 >> v2:172.21.15.49:3300/0 conn(0x7f62ac0d2110 msgr2=0x7f62ac0cd1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:21.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.358+0000 7f62aaffd700 1 --2- 172.21.15.16:0/625938847 >> v2:172.21.15.49:3300/0 conn(0x7f62ac0d2110 0x7f62ac0cd1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:21.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.358+0000 7f62aaffd700 1 -- 172.21.15.16:0/625938847 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62ac0cef60 con 0x7f62ac15f060 2023-10-16T12:12:21.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.358+0000 7f62b0f66700 1 --2- 172.21.15.16:0/625938847 >> v2:172.21.15.49:3300/0 conn(0x7f62ac0d2110 0x7f62ac0cd1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:12:21.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.358+0000 7f62abfff700 1 --2- 172.21.15.16:0/625938847 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f62ac15ec80 0x7f62ac0cdec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:12:21.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.358+0000 7f62aaffd700 1 -- 172.21.15.16:0/625938847 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f62a0001670 con 0x7f62ac15f060 2023-10-16T12:12:21.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.359+0000 7f62aaffd700 1 -- 172.21.15.16:0/625938847 >> v1:172.21.15.16:6789/0 conn(0x7f62ac15f060 legacy=0x7f62ac0b4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:21.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.359+0000 7f62aaffd700 1 --2- 172.21.15.16:0/625938847 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6294003a00 0x7f6294003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:21.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.359+0000 7f62aaffd700 1 --2- 172.21.15.16:0/625938847 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f62ac15ec80 0x7f6294004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:21.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.359+0000 7f62aaffd700 1 --2- 172.21.15.16:0/625938847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6294004d90 0x7f6294005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:21.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.359+0000 7f62aaffd700 1 -- 172.21.15.16:0/625938847 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62ac0b2c50 con 0x7f6294004d90 2023-10-16T12:12:21.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.359+0000 7f62aaffd700 1 -- 172.21.15.16:0/625938847 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62ac0d00e0 con 0x7f6294003a00 2023-10-16T12:12:21.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.359+0000 7f62aaffd700 1 -- 172.21.15.16:0/625938847 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f629400ba10 con 0x7f62ac15ec80 2023-10-16T12:12:21.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.359+0000 7f62b0f66700 1 --2- 172.21.15.16:0/625938847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6294004d90 0x7f6294005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:21.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.359+0000 7f62ab7fe700 1 --2- 172.21.15.16:0/625938847 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f62ac15ec80 0x7f6294004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:21.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.359+0000 7f62abfff700 1 --2- 172.21.15.16:0/625938847 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6294003a00 0x7f6294003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:21.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.359+0000 7f62b0f66700 1 -- 172.21.15.16:0/625938847 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f62ac15ec80 msgr2=0x7f6294004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:21.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.359+0000 7f62b0f66700 1 --2- 172.21.15.16:0/625938847 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f62ac15ec80 0x7f6294004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:21.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.359+0000 7f62b0f66700 1 -- 172.21.15.16:0/625938847 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6294003a00 msgr2=0x7f6294003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:21.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.359+0000 7f62b0f66700 1 --2- 172.21.15.16:0/625938847 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6294003a00 0x7f6294003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:21.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.359+0000 7f62b0f66700 1 -- 172.21.15.16:0/625938847 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f629c0094a0 con 0x7f6294004d90 2023-10-16T12:12:21.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.359+0000 7f62b0f66700 1 --2- 172.21.15.16:0/625938847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6294004d90 0x7f6294005210 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f62a400e350 tx=0x7f6294005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2657d16a05c03e5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:21.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.359+0000 7f62aaffd700 1 -- 172.21.15.16:0/625938847 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62a400edf0 con 0x7f6294004d90 2023-10-16T12:12:21.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.359+0000 7f62aaffd700 1 -- 172.21.15.16:0/625938847 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62a400f040 con 0x7f6294004d90 2023-10-16T12:12:21.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.359+0000 7f62aaffd700 1 -- 172.21.15.16:0/625938847 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62a40146a0 con 0x7f6294004d90 2023-10-16T12:12:21.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.360+0000 7f62b29c9700 1 -- 172.21.15.16:0/625938847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6294004d90 msgr2=0x7f6294005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:21.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.360+0000 7f62b29c9700 1 --2- 172.21.15.16:0/625938847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6294004d90 0x7f6294005210 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f62a400e350 tx=0x7f6294005880 comp rx=0 tx=0).stop 2023-10-16T12:12:21.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.360+0000 7f62b0f66700 1 -- 172.21.15.16:0/625938847 reap_dead start 2023-10-16T12:12:21.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.360+0000 7f62b29c9700 1 -- 172.21.15.16:0/625938847 shutdown_connections 2023-10-16T12:12:21.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.360+0000 7f62b29c9700 1 -- 172.21.15.16:0/625938847 >> 172.21.15.16:0/625938847 conn(0x7f62ac0d15f0 msgr2=0x7f62ac0d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:21.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.360+0000 7f62b29c9700 1 -- 172.21.15.16:0/625938847 shutdown_connections 2023-10-16T12:12:21.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.360+0000 7f62b29c9700 1 -- 172.21.15.16:0/625938847 wait complete. 2023-10-16T12:12:21.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.361+0000 7f62b29c9700 1 Processor -- start 2023-10-16T12:12:21.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.361+0000 7f62b29c9700 1 -- start start 2023-10-16T12:12:21.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.362+0000 7f62b29c9700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6294004d90 0x7f62ac14b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:21.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.362+0000 7f62b29c9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62ac1495a0 0x7f62ac149de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:21.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.362+0000 7f62b29c9700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f62ac14a320 0x7f62ac14be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:21.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.362+0000 7f62b29c9700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62ac14b3d0 con 0x7f62ac1495a0 2023-10-16T12:12:21.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.362+0000 7f62b29c9700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62ac14a7c0 con 0x7f62ac14a320 2023-10-16T12:12:21.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.362+0000 7f62b29c9700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62ac14aaa0 con 0x7f6294004d90 2023-10-16T12:12:21.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.362+0000 7f62ab7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62ac1495a0 0x7f62ac149de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:21.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.362+0000 7f62ab7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62ac1495a0 0x7f62ac149de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36302/0 (socket says 172.21.15.16:36302) 2023-10-16T12:12:21.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.362+0000 7f62ab7fe700 1 -- 172.21.15.16:0/3400214156 learned_addr learned my addr 172.21.15.16:0/3400214156 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:12:21.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.362+0000 7f62abfff700 1 --2- 172.21.15.16:0/3400214156 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6294004d90 0x7f62ac14b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:21.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.362+0000 7f62b0f66700 1 --2- 172.21.15.16:0/3400214156 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f62ac14a320 0x7f62ac14be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:21.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.363+0000 7f62ab7fe700 1 -- 172.21.15.16:0/3400214156 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6294004d90 msgr2=0x7f62ac14b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:21.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.363+0000 7f62ab7fe700 1 --2- 172.21.15.16:0/3400214156 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6294004d90 0x7f62ac14b860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:21.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.363+0000 7f62ab7fe700 1 -- 172.21.15.16:0/3400214156 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f62ac14a320 msgr2=0x7f62ac14be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:21.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.363+0000 7f62ab7fe700 1 --2- 172.21.15.16:0/3400214156 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f62ac14a320 0x7f62ac14be30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:21.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.363+0000 7f62ab7fe700 1 -- 172.21.15.16:0/3400214156 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62ac14ad80 con 0x7f62ac1495a0 2023-10-16T12:12:21.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.363+0000 7f62ab7fe700 1 --2- 172.21.15.16:0/3400214156 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62ac1495a0 0x7f62ac149de0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f62a0001050 tx=0x7f62a0007640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:21.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.363+0000 7f62a8ff9700 1 -- 172.21.15.16:0/3400214156 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62a000f6c0 con 0x7f62ac1495a0 2023-10-16T12:12:21.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.363+0000 7f62a8ff9700 1 -- 172.21.15.16:0/3400214156 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62a000f840 con 0x7f62ac1495a0 2023-10-16T12:12:21.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.363+0000 7f62a8ff9700 1 -- 172.21.15.16:0/3400214156 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62a000fbf0 con 0x7f62ac1495a0 2023-10-16T12:12:21.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.363+0000 7f62b29c9700 1 -- 172.21.15.16:0/3400214156 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62ac14b060 con 0x7f62ac1495a0 2023-10-16T12:12:21.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.363+0000 7f62b29c9700 1 -- 172.21.15.16:0/3400214156 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62ac15c020 con 0x7f62ac1495a0 2023-10-16T12:12:21.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.364+0000 7f62b29c9700 1 -- 172.21.15.16:0/3400214156 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62ac04ebb0 con 0x7f62ac1495a0 2023-10-16T12:12:21.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.364+0000 7f62a8ff9700 1 -- 172.21.15.16:0/3400214156 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 10) v1 ==== 89405+0+0 (secure 0 0 0) 0x7f62a000fd50 con 0x7f62ac1495a0 2023-10-16T12:12:21.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.365+0000 7f62a8ff9700 1 --2- 172.21.15.16:0/3400214156 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f629806b100 0x7f629806d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:21.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.365+0000 7f62a8ff9700 1 -- 172.21.15.16:0/3400214156 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7f62a0089c20 con 0x7f62ac1495a0 2023-10-16T12:12:21.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.365+0000 7f62abfff700 1 --2- 172.21.15.16:0/3400214156 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f629806b100 0x7f629806d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:21.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.365+0000 7f62abfff700 1 --2- 172.21.15.16:0/3400214156 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f629806b100 0x7f629806d5c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f629c009470 tx=0x7f629c009210 comp rx=0 tx=0).ready entity=mgr.5499 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:21.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.370+0000 7f62a8ff9700 1 -- 172.21.15.16:0/3400214156 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f62a0054810 con 0x7f62ac1495a0 2023-10-16T12:12:21.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.605+0000 7f62b29c9700 1 -- 172.21.15.16:0/3400214156 --> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir0", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f62ac15c240 con 0x7f629806b100 2023-10-16T12:12:21.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.653+0000 7f62a8ff9700 1 -- 172.21.15.16:0/3400214156 <== mgr.5499 v2:172.21.15.16:6832/1707126831 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f62ac15c240 con 0x7f629806b100 2023-10-16T12:12:21.655 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule activated for path /snap_test_dir1/dir0 2023-10-16T12:12:21.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.655+0000 7f62b29c9700 1 -- 172.21.15.16:0/3400214156 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f629806b100 msgr2=0x7f629806d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:21.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.655+0000 7f62b29c9700 1 --2- 172.21.15.16:0/3400214156 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f629806b100 0x7f629806d5c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f629c009470 tx=0x7f629c009210 comp rx=0 tx=0).stop 2023-10-16T12:12:21.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.655+0000 7f62b29c9700 1 -- 172.21.15.16:0/3400214156 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62ac1495a0 msgr2=0x7f62ac149de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:21.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.655+0000 7f62b29c9700 1 --2- 172.21.15.16:0/3400214156 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62ac1495a0 0x7f62ac149de0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f62a0001050 tx=0x7f62a0007640 comp rx=0 tx=0).stop 2023-10-16T12:12:21.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.655+0000 7f62b29c9700 1 -- 172.21.15.16:0/3400214156 shutdown_connections 2023-10-16T12:12:21.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.655+0000 7f62b29c9700 1 --2- 172.21.15.16:0/3400214156 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f629806b100 0x7f629806d5c0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:21.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.655+0000 7f62b29c9700 1 --2- 172.21.15.16:0/3400214156 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f62ac14a320 0x7f62ac14be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:21.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.655+0000 7f62b29c9700 1 --2- 172.21.15.16:0/3400214156 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6294004d90 0x7f62ac14b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:21.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.655+0000 7f62b29c9700 1 --2- 172.21.15.16:0/3400214156 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62ac1495a0 0x7f62ac149de0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:21.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.655+0000 7f62b29c9700 1 -- 172.21.15.16:0/3400214156 >> 172.21.15.16:0/3400214156 conn(0x7f62ac0d15f0 msgr2=0x7f62ac15c930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:21.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.655+0000 7f62b29c9700 1 -- 172.21.15.16:0/3400214156 shutdown_connections 2023-10-16T12:12:21.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.655+0000 7f62b29c9700 1 -- 172.21.15.16:0/3400214156 wait complete. 2023-10-16T12:12:21.669 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir0 2023-10-16T12:12:21.670 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir1 1m 2023-10-16T12:12:21.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.796+0000 7f50f3b7a700 1 Processor -- start 2023-10-16T12:12:21.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.797+0000 7f50f3b7a700 1 -- start start 2023-10-16T12:12:21.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.797+0000 7f50f3b7a700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f50ec0b3ad0 0x7f50ec0717b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:21.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.797+0000 7f50f3b7a700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50ec072350 con 0x7f50ec0cf8d0 2023-10-16T12:12:21.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.797+0000 7f50f3b7a700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50ec071d90 con 0x7f50ec1577a0 2023-10-16T12:12:21.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.797+0000 7f50f3b7a700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50ec072070 con 0x7f50ec0b3ad0 2023-10-16T12:12:21.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.798+0000 7f50f1115700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f50ec0cf8d0 0x7f50ec0710a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:53814/0 (socket says 172.21.15.16:53814) 2023-10-16T12:12:21.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.798+0000 7f50f1115700 1 -- 172.21.15.16:0/3177563044 learned_addr learned my addr 172.21.15.16:0/3177563044 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:12:21.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.798+0000 7f50f2117700 1 --2- 172.21.15.16:0/3177563044 >> v2:172.21.15.49:3300/0 conn(0x7f50ec0b3ad0 0x7f50ec0717b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:21.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.798+0000 7f50f0914700 1 -- 172.21.15.16:0/3177563044 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3846981280 0 0) 0x7f50ec072350 con 0x7f50ec0cf8d0 2023-10-16T12:12:21.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.798+0000 7f50f0914700 1 -- 172.21.15.16:0/3177563044 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50d80034d0 con 0x7f50ec0cf8d0 2023-10-16T12:12:21.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.798+0000 7f50f0914700 1 -- 172.21.15.16:0/3177563044 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2960108444 0 0) 0x7f50d80034d0 con 0x7f50ec0cf8d0 2023-10-16T12:12:21.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.798+0000 7f50f0914700 1 -- 172.21.15.16:0/3177563044 >> v2:172.21.15.49:3300/0 conn(0x7f50ec0b3ad0 msgr2=0x7f50ec0717b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:21.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.798+0000 7f50f0914700 1 --2- 172.21.15.16:0/3177563044 >> v2:172.21.15.49:3300/0 conn(0x7f50ec0b3ad0 0x7f50ec0717b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:21.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.798+0000 7f50f0914700 1 -- 172.21.15.16:0/3177563044 >> v1:172.21.15.49:6789/0 conn(0x7f50ec1577a0 legacy=0x7f50ec0cf1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:21.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.798+0000 7f50f0914700 1 -- 172.21.15.16:0/3177563044 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50ec0b22c0 con 0x7f50ec0cf8d0 2023-10-16T12:12:21.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.798+0000 7f50f0914700 1 -- 172.21.15.16:0/3177563044 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f50e8001670 con 0x7f50ec0cf8d0 2023-10-16T12:12:21.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.798+0000 7f50f0914700 1 -- 172.21.15.16:0/3177563044 >> v1:172.21.15.16:6789/0 conn(0x7f50ec0cf8d0 legacy=0x7f50ec0710a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:21.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.798+0000 7f50f0914700 1 --2- 172.21.15.16:0/3177563044 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f50d8003af0 0x7f50d8003f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:21.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.798+0000 7f50f0914700 1 --2- 172.21.15.16:0/3177563044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50d80088e0 0x7f50d800ad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:21.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.798+0000 7f50f0914700 1 --2- 172.21.15.16:0/3177563044 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f50d800b5b0 0x7f50d800da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:21.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.798+0000 7f50f0914700 1 -- 172.21.15.16:0/3177563044 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50ec072070 con 0x7f50d80088e0 2023-10-16T12:12:21.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.798+0000 7f50f0914700 1 -- 172.21.15.16:0/3177563044 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50d800e350 con 0x7f50d8003af0 2023-10-16T12:12:21.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.798+0000 7f50f0914700 1 -- 172.21.15.16:0/3177563044 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50d800e510 con 0x7f50d800b5b0 2023-10-16T12:12:21.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.798+0000 7f50f1115700 1 --2- 172.21.15.16:0/3177563044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50d80088e0 0x7f50d800ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:21.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.798+0000 7f50f1916700 1 --2- 172.21.15.16:0/3177563044 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f50d8003af0 0x7f50d8003f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:21.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.798+0000 7f50f2117700 1 --2- 172.21.15.16:0/3177563044 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f50d800b5b0 0x7f50d800da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:21.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.799+0000 7f50f1115700 1 -- 172.21.15.16:0/3177563044 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f50d800b5b0 msgr2=0x7f50d800da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:21.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.799+0000 7f50f1115700 1 --2- 172.21.15.16:0/3177563044 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f50d800b5b0 0x7f50d800da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:21.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.799+0000 7f50f1115700 1 -- 172.21.15.16:0/3177563044 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f50d8003af0 msgr2=0x7f50d8003f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:21.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.799+0000 7f50f1115700 1 --2- 172.21.15.16:0/3177563044 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f50d8003af0 0x7f50d8003f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:21.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.799+0000 7f50f1115700 1 -- 172.21.15.16:0/3177563044 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50d80034d0 con 0x7f50d80088e0 2023-10-16T12:12:21.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.799+0000 7f50f2117700 1 -- 172.21.15.16:0/3177563044 reap_dead start 2023-10-16T12:12:21.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.799+0000 7f50f1115700 1 --2- 172.21.15.16:0/3177563044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50d80088e0 0x7f50d800ad10 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f50e800c210 tx=0x7f50d8003780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8b730d222f29f03 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:21.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.799+0000 7f50f0914700 1 -- 172.21.15.16:0/3177563044 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50e8007c10 con 0x7f50d80088e0 2023-10-16T12:12:21.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.799+0000 7f50f0914700 1 -- 172.21.15.16:0/3177563044 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50e800cdb0 con 0x7f50d80088e0 2023-10-16T12:12:21.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.799+0000 7f50f0914700 1 -- 172.21.15.16:0/3177563044 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50e8005ba0 con 0x7f50d80088e0 2023-10-16T12:12:21.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.799+0000 7f50f3b7a700 1 -- 172.21.15.16:0/3177563044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50d80088e0 msgr2=0x7f50d800ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:21.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.799+0000 7f50f3b7a700 1 --2- 172.21.15.16:0/3177563044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50d80088e0 0x7f50d800ad10 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f50e800c210 tx=0x7f50d8003780 comp rx=0 tx=0).stop 2023-10-16T12:12:21.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.799+0000 7f50f3b7a700 1 -- 172.21.15.16:0/3177563044 shutdown_connections 2023-10-16T12:12:21.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.799+0000 7f50f3b7a700 1 --2- 172.21.15.16:0/3177563044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50d80088e0 0x7f50d800ad10 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:21.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.800+0000 7f50f3b7a700 1 -- 172.21.15.16:0/3177563044 >> 172.21.15.16:0/3177563044 conn(0x7f50ec06a7e0 msgr2=0x7f50ec06abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:21.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.800+0000 7f50f3b7a700 1 -- 172.21.15.16:0/3177563044 shutdown_connections 2023-10-16T12:12:21.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.800+0000 7f50f3b7a700 1 -- 172.21.15.16:0/3177563044 wait complete. 2023-10-16T12:12:21.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.801+0000 7f50f3b7a700 1 Processor -- start 2023-10-16T12:12:21.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.801+0000 7f50f3b7a700 1 -- start start 2023-10-16T12:12:21.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.802+0000 7f50f3b7a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50d80088e0 0x7f50ec1f2580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:21.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.802+0000 7f50f3b7a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f50ec0cb160 0x7f50ec15da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:21.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.802+0000 7f50f3b7a700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f50ec150810 0x7f50ec150ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:21.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.802+0000 7f50f3b7a700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50ec153750 con 0x7f50d80088e0 2023-10-16T12:12:21.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.802+0000 7f50f3b7a700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50ec1f2ac0 con 0x7f50ec150810 2023-10-16T12:12:21.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.802+0000 7f50f3b7a700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50ec152ad0 con 0x7f50ec0cb160 2023-10-16T12:12:21.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.802+0000 7f50f1916700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50d80088e0 0x7f50ec1f2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:21.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.802+0000 7f50f1916700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50d80088e0 0x7f50ec1f2580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36320/0 (socket says 172.21.15.16:36320) 2023-10-16T12:12:21.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.802+0000 7f50f1916700 1 -- 172.21.15.16:0/2850163368 learned_addr learned my addr 172.21.15.16:0/2850163368 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:12:21.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.802+0000 7f50f1115700 1 --2- 172.21.15.16:0/2850163368 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f50ec0cb160 0x7f50ec15da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:21.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.802+0000 7f50f2117700 1 --2- 172.21.15.16:0/2850163368 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f50ec150810 0x7f50ec150ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:21.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.802+0000 7f50f1916700 1 -- 172.21.15.16:0/2850163368 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f50ec0cb160 msgr2=0x7f50ec15da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:21.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.802+0000 7f50f1916700 1 --2- 172.21.15.16:0/2850163368 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f50ec0cb160 0x7f50ec15da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:21.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.802+0000 7f50f1916700 1 -- 172.21.15.16:0/2850163368 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f50ec150810 msgr2=0x7f50ec150ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:21.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.802+0000 7f50f1916700 1 --2- 172.21.15.16:0/2850163368 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f50ec150810 0x7f50ec150ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:21.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.802+0000 7f50f1916700 1 -- 172.21.15.16:0/2850163368 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50ec152db0 con 0x7f50d80088e0 2023-10-16T12:12:21.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.802+0000 7f50f1115700 1 --2- 172.21.15.16:0/2850163368 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f50ec0cb160 0x7f50ec15da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:12:21.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.802+0000 7f50f2117700 1 --2- 172.21.15.16:0/2850163368 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f50ec150810 0x7f50ec150ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:12:21.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.802+0000 7f50f1916700 1 --2- 172.21.15.16:0/2850163368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50d80088e0 0x7f50ec1f2580 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f50dc00f820 tx=0x7f50dc00fbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:21.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.802+0000 7f50e27fc700 1 -- 172.21.15.16:0/2850163368 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50dc0052e0 con 0x7f50d80088e0 2023-10-16T12:12:21.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.802+0000 7f50e27fc700 1 -- 172.21.15.16:0/2850163368 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50dc005460 con 0x7f50d80088e0 2023-10-16T12:12:21.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.802+0000 7f50e27fc700 1 -- 172.21.15.16:0/2850163368 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50dc0038c0 con 0x7f50d80088e0 2023-10-16T12:12:21.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.802+0000 7f50f3b7a700 1 -- 172.21.15.16:0/2850163368 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50ec153090 con 0x7f50d80088e0 2023-10-16T12:12:21.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.802+0000 7f50f3b7a700 1 -- 172.21.15.16:0/2850163368 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50ec1511e0 con 0x7f50d80088e0 2023-10-16T12:12:21.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.803+0000 7f50f3b7a700 1 -- 172.21.15.16:0/2850163368 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50ec153890 con 0x7f50d80088e0 2023-10-16T12:12:21.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.804+0000 7f50e27fc700 1 -- 172.21.15.16:0/2850163368 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 10) v1 ==== 89405+0+0 (secure 0 0 0) 0x7f50dc003a20 con 0x7f50d80088e0 2023-10-16T12:12:21.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.804+0000 7f50e27fc700 1 --2- 172.21.15.16:0/2850163368 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f50d006b100 0x7f50d006d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:21.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.805+0000 7f50f1115700 1 --2- 172.21.15.16:0/2850163368 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f50d006b100 0x7f50d006d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:21.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.805+0000 7f50e27fc700 1 -- 172.21.15.16:0/2850163368 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7f50dc089930 con 0x7f50d80088e0 2023-10-16T12:12:21.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.805+0000 7f50f1115700 1 --2- 172.21.15.16:0/2850163368 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f50d006b100 0x7f50d006d5c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f50e8001930 tx=0x7f50e8007000 comp rx=0 tx=0).ready entity=mgr.5499 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:21.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:21.809+0000 7f50e27fc700 1 -- 172.21.15.16:0/2850163368 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f50ec153890 con 0x7f50d80088e0 2023-10-16T12:12:22.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.043+0000 7f50f3b7a700 1 -- 172.21.15.16:0/2850163368 --> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir1", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f50ec0b25a0 con 0x7f50d006b100 2023-10-16T12:12:22.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.090+0000 7f50e27fc700 1 -- 172.21.15.16:0/2850163368 <== mgr.5499 v2:172.21.15.16:6832/1707126831 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f50e8007bf0 con 0x7f50d006b100 2023-10-16T12:12:22.092 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule activated for path /snap_test_dir1/dir1 2023-10-16T12:12:22.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.092+0000 7f50f3b7a700 1 -- 172.21.15.16:0/2850163368 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f50d006b100 msgr2=0x7f50d006d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:22.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.092+0000 7f50f3b7a700 1 --2- 172.21.15.16:0/2850163368 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f50d006b100 0x7f50d006d5c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f50e8001930 tx=0x7f50e8007000 comp rx=0 tx=0).stop 2023-10-16T12:12:22.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.092+0000 7f50f3b7a700 1 -- 172.21.15.16:0/2850163368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50d80088e0 msgr2=0x7f50ec1f2580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:22.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.092+0000 7f50f3b7a700 1 --2- 172.21.15.16:0/2850163368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50d80088e0 0x7f50ec1f2580 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f50dc00f820 tx=0x7f50dc00fbe0 comp rx=0 tx=0).stop 2023-10-16T12:12:22.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.092+0000 7f50f3b7a700 1 -- 172.21.15.16:0/2850163368 shutdown_connections 2023-10-16T12:12:22.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.092+0000 7f50f3b7a700 1 --2- 172.21.15.16:0/2850163368 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f50d006b100 0x7f50d006d5c0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:22.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.092+0000 7f50f3b7a700 1 --2- 172.21.15.16:0/2850163368 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f50ec150810 0x7f50ec150ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:22.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.092+0000 7f50f3b7a700 1 --2- 172.21.15.16:0/2850163368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50d80088e0 0x7f50ec1f2580 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:22.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.092+0000 7f50f3b7a700 1 --2- 172.21.15.16:0/2850163368 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f50ec0cb160 0x7f50ec15da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:22.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.092+0000 7f50f3b7a700 1 -- 172.21.15.16:0/2850163368 >> 172.21.15.16:0/2850163368 conn(0x7f50ec06a7e0 msgr2=0x7f50ec06abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:22.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.092+0000 7f50f3b7a700 1 -- 172.21.15.16:0/2850163368 shutdown_connections 2023-10-16T12:12:22.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.092+0000 7f50f3b7a700 1 -- 172.21.15.16:0/2850163368 wait complete. 2023-10-16T12:12:22.106 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir1 2023-10-16T12:12:22.106 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir2 1m 2023-10-16T12:12:22.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.232+0000 7f185fb95700 1 Processor -- start 2023-10-16T12:12:22.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.232+0000 7f185fb95700 1 -- start start 2023-10-16T12:12:22.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.233+0000 7f185fb95700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1858072b30 0x7f1858071a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:22.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.233+0000 7f185fb95700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f185815c040 con 0x7f18580caa40 2023-10-16T12:12:22.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.233+0000 7f185fb95700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f185815c320 con 0x7f185815bb40 2023-10-16T12:12:22.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.233+0000 7f185fb95700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f185806ea20 con 0x7f1858072b30 2023-10-16T12:12:22.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.233+0000 7f185e132700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f18580caa40 0x7f18580ccf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:53836/0 (socket says 172.21.15.16:53836) 2023-10-16T12:12:22.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.233+0000 7f185e132700 1 -- 172.21.15.16:0/3994281123 learned_addr learned my addr 172.21.15.16:0/3994281123 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:12:22.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.233+0000 7f185d931700 1 --2- 172.21.15.16:0/3994281123 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1858072b30 0x7f1858071a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:22.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.234+0000 7f185c92f700 1 -- 172.21.15.16:0/3994281123 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2283936711 0 0) 0x7f185815c040 con 0x7f18580caa40 2023-10-16T12:12:22.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.234+0000 7f185c92f700 1 -- 172.21.15.16:0/3994281123 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18440035b0 con 0x7f18580caa40 2023-10-16T12:12:22.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.234+0000 7f185c92f700 1 -- 172.21.15.16:0/3994281123 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2376205345 0 0) 0x7f185815c320 con 0x7f185815bb40 2023-10-16T12:12:22.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.234+0000 7f185c92f700 1 -- 172.21.15.16:0/3994281123 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f185815c040 con 0x7f185815bb40 2023-10-16T12:12:22.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.234+0000 7f185d931700 1 -- 172.21.15.16:0/3994281123 >> v1:172.21.15.49:6789/0 conn(0x7f185815bb40 legacy=0x7f185806e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:22.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.234+0000 7f185d931700 1 -- 172.21.15.16:0/3994281123 >> v1:172.21.15.16:6789/0 conn(0x7f18580caa40 legacy=0x7f18580ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:22.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.234+0000 7f185d931700 1 -- 172.21.15.16:0/3994281123 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f185815c600 con 0x7f1858072b30 2023-10-16T12:12:22.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.234+0000 7f185d931700 1 --2- 172.21.15.16:0/3994281123 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1858072b30 0x7f1858071a70 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1848009870 tx=0x7f1848009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=55bd64edb3665558 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:22.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.234+0000 7f185c92f700 1 -- 172.21.15.16:0/3994281123 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1848011830 con 0x7f1858072b30 2023-10-16T12:12:22.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.234+0000 7f185c92f700 1 -- 172.21.15.16:0/3994281123 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1848011990 con 0x7f1858072b30 2023-10-16T12:12:22.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.234+0000 7f185c92f700 1 -- 172.21.15.16:0/3994281123 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1848011d20 con 0x7f1858072b30 2023-10-16T12:12:22.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.235+0000 7f185fb95700 1 -- 172.21.15.16:0/3994281123 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1858072b30 msgr2=0x7f1858071a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:22.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.235+0000 7f185fb95700 1 --2- 172.21.15.16:0/3994281123 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1858072b30 0x7f1858071a70 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1848009870 tx=0x7f1848009b80 comp rx=0 tx=0).stop 2023-10-16T12:12:22.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.235+0000 7f185fb95700 1 -- 172.21.15.16:0/3994281123 shutdown_connections 2023-10-16T12:12:22.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.235+0000 7f185fb95700 1 --2- 172.21.15.16:0/3994281123 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1858072b30 0x7f1858071a70 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:22.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.235+0000 7f185fb95700 1 -- 172.21.15.16:0/3994281123 >> 172.21.15.16:0/3994281123 conn(0x7f18580b1f40 msgr2=0x7f18580b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:22.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.235+0000 7f185fb95700 1 -- 172.21.15.16:0/3994281123 shutdown_connections 2023-10-16T12:12:22.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.235+0000 7f185fb95700 1 -- 172.21.15.16:0/3994281123 wait complete. 2023-10-16T12:12:22.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.236+0000 7f185fb95700 1 Processor -- start 2023-10-16T12:12:22.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.236+0000 7f185fb95700 1 -- start start 2023-10-16T12:12:22.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.237+0000 7f185fb95700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1858072b30 0x7f1858152a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:22.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.237+0000 7f185fb95700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f18580caa40 0x7f1858156100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:22.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.237+0000 7f185fb95700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f185815bb40 0x7f18580c4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:22.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.237+0000 7f185fb95700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1858159300 con 0x7f185815bb40 2023-10-16T12:12:22.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.237+0000 7f185fb95700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f185815e550 con 0x7f18580caa40 2023-10-16T12:12:22.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.237+0000 7f185fb95700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18580b45e0 con 0x7f1858072b30 2023-10-16T12:12:22.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.237+0000 7f185e132700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f185815bb40 0x7f18580c4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:22.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.237+0000 7f185e132700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f185815bb40 0x7f18580c4f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36336/0 (socket says 172.21.15.16:36336) 2023-10-16T12:12:22.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.237+0000 7f185e132700 1 -- 172.21.15.16:0/1272849832 learned_addr learned my addr 172.21.15.16:0/1272849832 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:12:22.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.237+0000 7f185d931700 1 --2- 172.21.15.16:0/1272849832 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1858072b30 0x7f1858152a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:22.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.237+0000 7f185d130700 1 --2- 172.21.15.16:0/1272849832 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f18580caa40 0x7f1858156100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:22.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.237+0000 7f185e132700 1 -- 172.21.15.16:0/1272849832 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1858072b30 msgr2=0x7f1858152a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:22.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.237+0000 7f185e132700 1 --2- 172.21.15.16:0/1272849832 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1858072b30 0x7f1858152a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:22.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.237+0000 7f185e132700 1 -- 172.21.15.16:0/1272849832 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f18580caa40 msgr2=0x7f1858156100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:22.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.237+0000 7f185e132700 1 --2- 172.21.15.16:0/1272849832 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f18580caa40 0x7f1858156100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:22.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.237+0000 7f185e132700 1 -- 172.21.15.16:0/1272849832 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18480094a0 con 0x7f185815bb40 2023-10-16T12:12:22.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.237+0000 7f185d130700 1 --2- 172.21.15.16:0/1272849832 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f18580caa40 0x7f1858156100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:12:22.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.238+0000 7f185e132700 1 --2- 172.21.15.16:0/1272849832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f185815bb40 0x7f18580c4f70 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f185000cbd0 tx=0x7f185000cee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:22.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.238+0000 7f184e7fc700 1 -- 172.21.15.16:0/1272849832 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f185000aa00 con 0x7f185815bb40 2023-10-16T12:12:22.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.238+0000 7f184e7fc700 1 -- 172.21.15.16:0/1272849832 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f185000ab60 con 0x7f185815bb40 2023-10-16T12:12:22.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.238+0000 7f185fb95700 1 -- 172.21.15.16:0/1272849832 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18580b48c0 con 0x7f185815bb40 2023-10-16T12:12:22.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.238+0000 7f184e7fc700 1 -- 172.21.15.16:0/1272849832 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1850012400 con 0x7f185815bb40 2023-10-16T12:12:22.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.238+0000 7f185fb95700 1 -- 172.21.15.16:0/1272849832 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18580b4e80 con 0x7f185815bb40 2023-10-16T12:12:22.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.239+0000 7f185fb95700 1 -- 172.21.15.16:0/1272849832 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f185804ebb0 con 0x7f185815bb40 2023-10-16T12:12:22.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.239+0000 7f184e7fc700 1 -- 172.21.15.16:0/1272849832 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 10) v1 ==== 89405+0+0 (secure 0 0 0) 0x7f185001b3b0 con 0x7f185815bb40 2023-10-16T12:12:22.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.240+0000 7f184e7fc700 1 --2- 172.21.15.16:0/1272849832 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f183c06b100 0x7f183c06d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:22.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.240+0000 7f185d931700 1 --2- 172.21.15.16:0/1272849832 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f183c06b100 0x7f183c06d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:22.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.240+0000 7f184e7fc700 1 -- 172.21.15.16:0/1272849832 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7f18500898a0 con 0x7f185815bb40 2023-10-16T12:12:22.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.240+0000 7f185d931700 1 --2- 172.21.15.16:0/1272849832 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f183c06b100 0x7f183c06d5c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f18480090b0 tx=0x7f184801c040 comp rx=0 tx=0).ready entity=mgr.5499 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:22.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.244+0000 7f184e7fc700 1 -- 172.21.15.16:0/1272849832 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1850054480 con 0x7f185815bb40 2023-10-16T12:12:22.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.477+0000 7f185fb95700 1 -- 172.21.15.16:0/1272849832 --> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir2", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f18580cc9f0 con 0x7f183c06b100 2023-10-16T12:12:22.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.526+0000 7f184e7fc700 1 -- 172.21.15.16:0/1272849832 <== mgr.5499 v2:172.21.15.16:6832/1707126831 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f1848011b00 con 0x7f183c06b100 2023-10-16T12:12:22.528 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule activated for path /snap_test_dir1/dir2 2023-10-16T12:12:22.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.528+0000 7f185fb95700 1 -- 172.21.15.16:0/1272849832 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f183c06b100 msgr2=0x7f183c06d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:22.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.528+0000 7f185fb95700 1 --2- 172.21.15.16:0/1272849832 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f183c06b100 0x7f183c06d5c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f18480090b0 tx=0x7f184801c040 comp rx=0 tx=0).stop 2023-10-16T12:12:22.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.528+0000 7f185fb95700 1 -- 172.21.15.16:0/1272849832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f185815bb40 msgr2=0x7f18580c4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:22.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.528+0000 7f185fb95700 1 --2- 172.21.15.16:0/1272849832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f185815bb40 0x7f18580c4f70 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f185000cbd0 tx=0x7f185000cee0 comp rx=0 tx=0).stop 2023-10-16T12:12:22.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.528+0000 7f185fb95700 1 -- 172.21.15.16:0/1272849832 shutdown_connections 2023-10-16T12:12:22.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.528+0000 7f185fb95700 1 --2- 172.21.15.16:0/1272849832 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f183c06b100 0x7f183c06d5c0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:22.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.528+0000 7f185fb95700 1 --2- 172.21.15.16:0/1272849832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f185815bb40 0x7f18580c4f70 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:22.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.528+0000 7f185fb95700 1 --2- 172.21.15.16:0/1272849832 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1858072b30 0x7f1858152a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:22.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.528+0000 7f185fb95700 1 --2- 172.21.15.16:0/1272849832 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f18580caa40 0x7f1858156100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:22.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.528+0000 7f185fb95700 1 -- 172.21.15.16:0/1272849832 >> 172.21.15.16:0/1272849832 conn(0x7f18580b1f40 msgr2=0x7f18581596d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:22.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.528+0000 7f185fb95700 1 -- 172.21.15.16:0/1272849832 shutdown_connections 2023-10-16T12:12:22.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.528+0000 7f185fb95700 1 -- 172.21.15.16:0/1272849832 wait complete. 2023-10-16T12:12:22.542 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir2 2023-10-16T12:12:22.542 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir3 1m 2023-10-16T12:12:22.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.668+0000 7f1997a58700 1 Processor -- start 2023-10-16T12:12:22.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.669+0000 7f1997a58700 1 -- start start 2023-10-16T12:12:22.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.669+0000 7f1997a58700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f19900d4060 0x7f19900d1520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:22.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.669+0000 7f1997a58700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f19900d1d20 0x7f19900d21b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:22.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.670+0000 7f1997a58700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19900d4540 con 0x7f199015da10 2023-10-16T12:12:22.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.670+0000 7f1997a58700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19900cf9b0 con 0x7f19900d4060 2023-10-16T12:12:22.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.670+0000 7f1997a58700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19900d26f0 con 0x7f19900d1d20 2023-10-16T12:12:22.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.670+0000 7f1995ff5700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f19900d1d20 0x7f19900d21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:22.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.670+0000 7f19957f4700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7f199015da10 0x7f19900cf2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:32796/0 (socket says 172.21.15.16:32796) 2023-10-16T12:12:22.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.670+0000 7f19957f4700 1 -- 172.21.15.16:0/2723028831 learned_addr learned my addr 172.21.15.16:0/2723028831 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:12:22.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.670+0000 7f1994ff3700 1 --2- 172.21.15.16:0/2723028831 >> v2:172.21.15.49:3300/0 conn(0x7f19900d4060 0x7f19900d1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:22.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.670+0000 7f1983fff700 1 -- 172.21.15.16:0/2723028831 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3212836814 0 0) 0x7f19900d4540 con 0x7f199015da10 2023-10-16T12:12:22.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.670+0000 7f1983fff700 1 -- 172.21.15.16:0/2723028831 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f197c0035b0 con 0x7f199015da10 2023-10-16T12:12:22.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.670+0000 7f1995ff5700 1 -- 172.21.15.16:0/2723028831 >> v2:172.21.15.49:3300/0 conn(0x7f19900d4060 msgr2=0x7f19900d1520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:22.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.670+0000 7f1995ff5700 1 --2- 172.21.15.16:0/2723028831 >> v2:172.21.15.49:3300/0 conn(0x7f19900d4060 0x7f19900d1520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:22.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.670+0000 7f1995ff5700 1 -- 172.21.15.16:0/2723028831 >> v1:172.21.15.49:6789/0 conn(0x7f199015da10 legacy=0x7f19900cf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:22.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.670+0000 7f1995ff5700 1 -- 172.21.15.16:0/2723028831 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19900ce7e0 con 0x7f19900d1d20 2023-10-16T12:12:22.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.670+0000 7f1994ff3700 1 --2- 172.21.15.16:0/2723028831 >> v2:172.21.15.49:3300/0 conn(0x7f19900d4060 0x7f19900d1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:12:22.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.670+0000 7f1995ff5700 1 --2- 172.21.15.16:0/2723028831 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f19900d1d20 0x7f19900d21b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1988009d90 tx=0x7f198800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cafa5a04c871b37e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:22.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.671+0000 7f1983fff700 1 -- 172.21.15.16:0/2723028831 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f198800e070 con 0x7f19900d1d20 2023-10-16T12:12:22.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.671+0000 7f1983fff700 1 -- 172.21.15.16:0/2723028831 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f198800bb20 con 0x7f19900d1d20 2023-10-16T12:12:22.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.671+0000 7f1983fff700 1 -- 172.21.15.16:0/2723028831 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f198800be90 con 0x7f19900d1d20 2023-10-16T12:12:22.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.671+0000 7f1997a58700 1 -- 172.21.15.16:0/2723028831 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f19900d1d20 msgr2=0x7f19900d21b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:22.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.671+0000 7f1997a58700 1 --2- 172.21.15.16:0/2723028831 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f19900d1d20 0x7f19900d21b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1988009d90 tx=0x7f198800b330 comp rx=0 tx=0).stop 2023-10-16T12:12:22.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.671+0000 7f1997a58700 1 -- 172.21.15.16:0/2723028831 shutdown_connections 2023-10-16T12:12:22.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.671+0000 7f1997a58700 1 --2- 172.21.15.16:0/2723028831 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f19900d1d20 0x7f19900d21b0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:22.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.671+0000 7f1997a58700 1 --2- 172.21.15.16:0/2723028831 >> v2:172.21.15.49:3300/0 conn(0x7f19900d4060 0x7f19900d1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:22.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.671+0000 7f1997a58700 1 -- 172.21.15.16:0/2723028831 >> 172.21.15.16:0/2723028831 conn(0x7f199009c9f0 msgr2=0x7f19900d3740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:22.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.671+0000 7f1997a58700 1 -- 172.21.15.16:0/2723028831 shutdown_connections 2023-10-16T12:12:22.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.671+0000 7f1997a58700 1 -- 172.21.15.16:0/2723028831 wait complete. 2023-10-16T12:12:22.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.673+0000 7f1997a58700 1 Processor -- start 2023-10-16T12:12:22.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.673+0000 7f1997a58700 1 -- start start 2023-10-16T12:12:22.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.673+0000 7f1997a58700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f19900d1d20 0x7f19900b6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:22.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.673+0000 7f1997a58700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19900d4060 0x7f19900b5c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:22.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.673+0000 7f1997a58700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f199015da10 0x7f19900b88d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:22.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.673+0000 7f1997a58700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19901f7170 con 0x7f19900d4060 2023-10-16T12:12:22.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.673+0000 7f1997a58700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19901f76a0 con 0x7f19900d1d20 2023-10-16T12:12:22.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.673+0000 7f1997a58700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19900b32d0 con 0x7f199015da10 2023-10-16T12:12:22.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.674+0000 7f1994ff3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19900d4060 0x7f19900b5c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:22.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.674+0000 7f1994ff3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19900d4060 0x7f19900b5c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36348/0 (socket says 172.21.15.16:36348) 2023-10-16T12:12:22.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.674+0000 7f1994ff3700 1 -- 172.21.15.16:0/3517720940 learned_addr learned my addr 172.21.15.16:0/3517720940 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:12:22.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.674+0000 7f1995ff5700 1 --2- 172.21.15.16:0/3517720940 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f199015da10 0x7f19900b88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:22.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.674+0000 7f19957f4700 1 --2- 172.21.15.16:0/3517720940 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f19900d1d20 0x7f19900b6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:22.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.674+0000 7f1994ff3700 1 -- 172.21.15.16:0/3517720940 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f199015da10 msgr2=0x7f19900b88d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:22.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.674+0000 7f1994ff3700 1 --2- 172.21.15.16:0/3517720940 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f199015da10 0x7f19900b88d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:22.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.674+0000 7f1994ff3700 1 -- 172.21.15.16:0/3517720940 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f19900d1d20 msgr2=0x7f19900b6360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:22.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.674+0000 7f1994ff3700 1 --2- 172.21.15.16:0/3517720940 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f19900d1d20 0x7f19900b6360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:22.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.674+0000 7f1994ff3700 1 -- 172.21.15.16:0/3517720940 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19840094a0 con 0x7f19900d4060 2023-10-16T12:12:22.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.674+0000 7f1994ff3700 1 --2- 172.21.15.16:0/3517720940 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19900d4060 0x7f19900b5c80 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f198400ebe0 tx=0x7f1984009e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:22.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.674+0000 7f1981ffb700 1 -- 172.21.15.16:0/3517720940 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f198400c8b0 con 0x7f19900d4060 2023-10-16T12:12:22.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.674+0000 7f1981ffb700 1 -- 172.21.15.16:0/3517720940 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f198400ca10 con 0x7f19900d4060 2023-10-16T12:12:22.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.674+0000 7f1997a58700 1 -- 172.21.15.16:0/3517720940 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1988009a70 con 0x7f19900d4060 2023-10-16T12:12:22.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.674+0000 7f1981ffb700 1 -- 172.21.15.16:0/3517720940 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f198400cda0 con 0x7f19900d4060 2023-10-16T12:12:22.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.674+0000 7f1997a58700 1 -- 172.21.15.16:0/3517720940 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19900b3570 con 0x7f19900d4060 2023-10-16T12:12:22.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.675+0000 7f1997a58700 1 -- 172.21.15.16:0/3517720940 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f199004ebb0 con 0x7f19900d4060 2023-10-16T12:12:22.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.676+0000 7f1981ffb700 1 -- 172.21.15.16:0/3517720940 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 10) v1 ==== 89405+0+0 (secure 0 0 0) 0x7f1984013470 con 0x7f19900d4060 2023-10-16T12:12:22.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.676+0000 7f1981ffb700 1 --2- 172.21.15.16:0/3517720940 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f197406b100 0x7f197406d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:22.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.676+0000 7f1981ffb700 1 -- 172.21.15.16:0/3517720940 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7f1984089d80 con 0x7f19900d4060 2023-10-16T12:12:22.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.676+0000 7f19957f4700 1 --2- 172.21.15.16:0/3517720940 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f197406b100 0x7f197406d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:22.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.677+0000 7f19957f4700 1 --2- 172.21.15.16:0/3517720940 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f197406b100 0x7f197406d5c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f198c007fd0 tx=0x7f198c009040 comp rx=0 tx=0).ready entity=mgr.5499 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:22.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.681+0000 7f1981ffb700 1 -- 172.21.15.16:0/3517720940 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1984054960 con 0x7f19900d4060 2023-10-16T12:12:22.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.912+0000 7f1997a58700 1 -- 172.21.15.16:0/3517720940 --> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir3", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f19900ceda0 con 0x7f197406b100 2023-10-16T12:12:22.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.962+0000 7f1981ffb700 1 -- 172.21.15.16:0/3517720940 <== mgr.5499 v2:172.21.15.16:6832/1707126831 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f198c001380 con 0x7f197406b100 2023-10-16T12:12:22.964 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule activated for path /snap_test_dir1/dir3 2023-10-16T12:12:22.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.964+0000 7f1997a58700 1 -- 172.21.15.16:0/3517720940 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f197406b100 msgr2=0x7f197406d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:22.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.964+0000 7f1997a58700 1 --2- 172.21.15.16:0/3517720940 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f197406b100 0x7f197406d5c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f198c007fd0 tx=0x7f198c009040 comp rx=0 tx=0).stop 2023-10-16T12:12:22.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.964+0000 7f1997a58700 1 -- 172.21.15.16:0/3517720940 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19900d4060 msgr2=0x7f19900b5c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:22.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.964+0000 7f1997a58700 1 --2- 172.21.15.16:0/3517720940 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19900d4060 0x7f19900b5c80 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f198400ebe0 tx=0x7f1984009e00 comp rx=0 tx=0).stop 2023-10-16T12:12:22.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.964+0000 7f1997a58700 1 -- 172.21.15.16:0/3517720940 shutdown_connections 2023-10-16T12:12:22.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.964+0000 7f1997a58700 1 --2- 172.21.15.16:0/3517720940 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f197406b100 0x7f197406d5c0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:22.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.964+0000 7f1997a58700 1 --2- 172.21.15.16:0/3517720940 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f199015da10 0x7f19900b88d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:22.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.964+0000 7f1997a58700 1 --2- 172.21.15.16:0/3517720940 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f19900d1d20 0x7f19900b6360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:22.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.964+0000 7f1997a58700 1 --2- 172.21.15.16:0/3517720940 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19900d4060 0x7f19900b5c80 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:22.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.964+0000 7f1997a58700 1 -- 172.21.15.16:0/3517720940 >> 172.21.15.16:0/3517720940 conn(0x7f199009c9f0 msgr2=0x7f19900cf1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:22.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.964+0000 7f1997a58700 1 -- 172.21.15.16:0/3517720940 shutdown_connections 2023-10-16T12:12:22.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:22.964+0000 7f1997a58700 1 -- 172.21.15.16:0/3517720940 wait complete. 2023-10-16T12:12:22.978 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir3 2023-10-16T12:12:22.979 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir4 1m 2023-10-16T12:12:23.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.105+0000 7f7584d1b700 1 Processor -- start 2023-10-16T12:12:23.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.105+0000 7f7584d1b700 1 -- start start 2023-10-16T12:12:23.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.106+0000 7f7584d1b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f75800d2110 0x7f75800ae1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:23.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.106+0000 7f7584d1b700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75800b20f0 con 0x7f75800ce6a0 2023-10-16T12:12:23.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.106+0000 7f7584d1b700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75800d03d0 con 0x7f758006ca10 2023-10-16T12:12:23.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.106+0000 7f7584d1b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75800b2340 con 0x7f75800d2110 2023-10-16T12:12:23.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.106+0000 7f757e59c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f75800ce6a0 0x7f758006c300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:53860/0 (socket says 172.21.15.16:53860) 2023-10-16T12:12:23.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.106+0000 7f757e59c700 1 -- 172.21.15.16:0/2824650061 learned_addr learned my addr 172.21.15.16:0/2824650061 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:12:23.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.107+0000 7f757dd9b700 1 --2- 172.21.15.16:0/2824650061 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f75800d2110 0x7f75800ae1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:23.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.107+0000 7f757d59a700 1 -- 172.21.15.16:0/2824650061 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 541993867 0 0) 0x7f75800b20f0 con 0x7f75800ce6a0 2023-10-16T12:12:23.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.107+0000 7f757d59a700 1 -- 172.21.15.16:0/2824650061 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75680035b0 con 0x7f75800ce6a0 2023-10-16T12:12:23.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.107+0000 7f757d59a700 1 -- 172.21.15.16:0/2824650061 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3037702606 0 0) 0x7f75680035b0 con 0x7f75800ce6a0 2023-10-16T12:12:23.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.107+0000 7f757d59a700 1 -- 172.21.15.16:0/2824650061 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f75800d2110 msgr2=0x7f75800ae1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:23.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.107+0000 7f757d59a700 1 --2- 172.21.15.16:0/2824650061 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f75800d2110 0x7f75800ae1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:23.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.107+0000 7f757d59a700 1 -- 172.21.15.16:0/2824650061 >> v1:172.21.15.49:6789/0 conn(0x7f758006ca10 legacy=0x7f75800ae720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:23.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.107+0000 7f757d59a700 1 -- 172.21.15.16:0/2824650061 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f758006bb00 con 0x7f75800ce6a0 2023-10-16T12:12:23.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.107+0000 7f757d59a700 1 -- 172.21.15.16:0/2824650061 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f7570001670 con 0x7f75800ce6a0 2023-10-16T12:12:23.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.107+0000 7f757d59a700 1 -- 172.21.15.16:0/2824650061 >> v1:172.21.15.16:6789/0 conn(0x7f75800ce6a0 legacy=0x7f758006c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:23.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.107+0000 7f757d59a700 1 --2- 172.21.15.16:0/2824650061 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7568003a30 0x7f7568003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:23.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.107+0000 7f757d59a700 1 --2- 172.21.15.16:0/2824650061 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7568004850 0x7f7568004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:23.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.107+0000 7f757d59a700 1 --2- 172.21.15.16:0/2824650061 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f75800d2110 0x7f7568005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:23.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.107+0000 7f757d59a700 1 -- 172.21.15.16:0/2824650061 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75800b2340 con 0x7f7568004850 2023-10-16T12:12:23.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.107+0000 7f757d59a700 1 -- 172.21.15.16:0/2824650061 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f756800bac0 con 0x7f7568003a30 2023-10-16T12:12:23.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.107+0000 7f757d59a700 1 -- 172.21.15.16:0/2824650061 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f756800bc80 con 0x7f75800d2110 2023-10-16T12:12:23.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.107+0000 7f757dd9b700 1 --2- 172.21.15.16:0/2824650061 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7568004850 0x7f7568004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:23.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.107+0000 7f757e59c700 1 --2- 172.21.15.16:0/2824650061 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7568003a30 0x7f7568003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:23.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.107+0000 7f757ed9d700 1 --2- 172.21.15.16:0/2824650061 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f75800d2110 0x7f7568005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:23.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.107+0000 7f757dd9b700 1 -- 172.21.15.16:0/2824650061 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f75800d2110 msgr2=0x7f7568005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:23.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.107+0000 7f757dd9b700 1 --2- 172.21.15.16:0/2824650061 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f75800d2110 0x7f7568005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:23.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.107+0000 7f757dd9b700 1 -- 172.21.15.16:0/2824650061 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7568003a30 msgr2=0x7f7568003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:23.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.107+0000 7f757dd9b700 1 --2- 172.21.15.16:0/2824650061 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7568003a30 0x7f7568003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:23.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.108+0000 7f757dd9b700 1 -- 172.21.15.16:0/2824650061 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75680035b0 con 0x7f7568004850 2023-10-16T12:12:23.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.108+0000 7f757dd9b700 1 --2- 172.21.15.16:0/2824650061 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7568004850 0x7f7568004cd0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f7574009b30 tx=0x7f7568005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3142cb896bb224e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:23.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.108+0000 7f757d59a700 1 -- 172.21.15.16:0/2824650061 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f757400e6c0 con 0x7f7568004850 2023-10-16T12:12:23.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.108+0000 7f757d59a700 1 -- 172.21.15.16:0/2824650061 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f757400e820 con 0x7f7568004850 2023-10-16T12:12:23.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.108+0000 7f757d59a700 1 -- 172.21.15.16:0/2824650061 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f757400eb90 con 0x7f7568004850 2023-10-16T12:12:23.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.108+0000 7f7584d1b700 1 -- 172.21.15.16:0/2824650061 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7568004850 msgr2=0x7f7568004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:23.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.108+0000 7f7584d1b700 1 --2- 172.21.15.16:0/2824650061 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7568004850 0x7f7568004cd0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f7574009b30 tx=0x7f7568005880 comp rx=0 tx=0).stop 2023-10-16T12:12:23.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.108+0000 7f757ed9d700 1 -- 172.21.15.16:0/2824650061 reap_dead start 2023-10-16T12:12:23.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.108+0000 7f7584d1b700 1 -- 172.21.15.16:0/2824650061 shutdown_connections 2023-10-16T12:12:23.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.108+0000 7f7584d1b700 1 -- 172.21.15.16:0/2824650061 >> 172.21.15.16:0/2824650061 conn(0x7f75800d15f0 msgr2=0x7f75800d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:23.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.108+0000 7f7584d1b700 1 -- 172.21.15.16:0/2824650061 shutdown_connections 2023-10-16T12:12:23.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.109+0000 7f7584d1b700 1 -- 172.21.15.16:0/2824650061 wait complete. 2023-10-16T12:12:23.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.110+0000 7f7584d1b700 1 Processor -- start 2023-10-16T12:12:23.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.110+0000 7f7584d1b700 1 -- start start 2023-10-16T12:12:23.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.111+0000 7f7584d1b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7568004850 0x7f75800b6c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:23.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.111+0000 7f7584d1b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7580155ee0 0x7f7580152970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:23.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.111+0000 7f7584d1b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75800b4c20 0x7f75800b50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:23.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.111+0000 7f7584d1b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75800b55e0 con 0x7f75800b4c20 2023-10-16T12:12:23.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.111+0000 7f7584d1b700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75801f31c0 con 0x7f7568004850 2023-10-16T12:12:23.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.111+0000 7f7584d1b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75801f34a0 con 0x7f7580155ee0 2023-10-16T12:12:23.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.111+0000 7f757ed9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75800b4c20 0x7f75800b50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:23.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.111+0000 7f757ed9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75800b4c20 0x7f75800b50a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36370/0 (socket says 172.21.15.16:36370) 2023-10-16T12:12:23.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.111+0000 7f757ed9d700 1 -- 172.21.15.16:0/2498187693 learned_addr learned my addr 172.21.15.16:0/2498187693 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:12:23.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.111+0000 7f757dd9b700 1 --2- 172.21.15.16:0/2498187693 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7580155ee0 0x7f7580152970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:23.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.111+0000 7f757e59c700 1 --2- 172.21.15.16:0/2498187693 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7568004850 0x7f75800b6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:23.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.111+0000 7f757ed9d700 1 -- 172.21.15.16:0/2498187693 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7580155ee0 msgr2=0x7f7580152970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:23.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.111+0000 7f757ed9d700 1 --2- 172.21.15.16:0/2498187693 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7580155ee0 0x7f7580152970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:23.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.111+0000 7f757ed9d700 1 -- 172.21.15.16:0/2498187693 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7568004850 msgr2=0x7f75800b6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:23.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.111+0000 7f757ed9d700 1 --2- 172.21.15.16:0/2498187693 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7568004850 0x7f75800b6c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:23.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.111+0000 7f757ed9d700 1 -- 172.21.15.16:0/2498187693 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75801f3780 con 0x7f75800b4c20 2023-10-16T12:12:23.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.111+0000 7f757ed9d700 1 --2- 172.21.15.16:0/2498187693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75800b4c20 0x7f75800b50a0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f7580072af0 tx=0x7f757800d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:23.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.111+0000 7f756effd700 1 -- 172.21.15.16:0/2498187693 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7578003c10 con 0x7f75800b4c20 2023-10-16T12:12:23.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.111+0000 7f756effd700 1 -- 172.21.15.16:0/2498187693 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7578003d90 con 0x7f75800b4c20 2023-10-16T12:12:23.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.111+0000 7f7584d1b700 1 -- 172.21.15.16:0/2498187693 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75801f3a60 con 0x7f75800b4c20 2023-10-16T12:12:23.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.111+0000 7f756effd700 1 -- 172.21.15.16:0/2498187693 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f757800dbd0 con 0x7f75800b4c20 2023-10-16T12:12:23.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.111+0000 7f7584d1b700 1 -- 172.21.15.16:0/2498187693 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75801f2eb0 con 0x7f75800b4c20 2023-10-16T12:12:23.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.112+0000 7f7584d1b700 1 -- 172.21.15.16:0/2498187693 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f758004ebb0 con 0x7f75800b4c20 2023-10-16T12:12:23.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.113+0000 7f756effd700 1 -- 172.21.15.16:0/2498187693 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 10) v1 ==== 89405+0+0 (secure 0 0 0) 0x7f757800dd30 con 0x7f75800b4c20 2023-10-16T12:12:23.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.113+0000 7f756effd700 1 --2- 172.21.15.16:0/2498187693 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f756006b0d0 0x7f756006d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:23.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.113+0000 7f756effd700 1 -- 172.21.15.16:0/2498187693 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7f7578089dd0 con 0x7f75800b4c20 2023-10-16T12:12:23.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.113+0000 7f757e59c700 1 --2- 172.21.15.16:0/2498187693 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f756006b0d0 0x7f756006d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:23.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.114+0000 7f757e59c700 1 --2- 172.21.15.16:0/2498187693 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f756006b0d0 0x7f756006d590 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7570000c00 tx=0x7f7570009040 comp rx=0 tx=0).ready entity=mgr.5499 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:23.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.118+0000 7f756effd700 1 -- 172.21.15.16:0/2498187693 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7578054a50 con 0x7f75800b4c20 2023-10-16T12:12:23.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.355+0000 7f7584d1b700 1 -- 172.21.15.16:0/2498187693 --> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir4", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f75800d0690 con 0x7f756006b0d0 2023-10-16T12:12:23.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.400+0000 7f756effd700 1 -- 172.21.15.16:0/2498187693 <== mgr.5499 v2:172.21.15.16:6832/1707126831 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f7570001660 con 0x7f756006b0d0 2023-10-16T12:12:23.403 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule activated for path /snap_test_dir1/dir4 2023-10-16T12:12:23.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.403+0000 7f7584d1b700 1 -- 172.21.15.16:0/2498187693 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f756006b0d0 msgr2=0x7f756006d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:23.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.403+0000 7f7584d1b700 1 --2- 172.21.15.16:0/2498187693 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f756006b0d0 0x7f756006d590 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7570000c00 tx=0x7f7570009040 comp rx=0 tx=0).stop 2023-10-16T12:12:23.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.403+0000 7f7584d1b700 1 -- 172.21.15.16:0/2498187693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75800b4c20 msgr2=0x7f75800b50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:23.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.403+0000 7f7584d1b700 1 --2- 172.21.15.16:0/2498187693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75800b4c20 0x7f75800b50a0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f7580072af0 tx=0x7f757800d430 comp rx=0 tx=0).stop 2023-10-16T12:12:23.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.403+0000 7f7584d1b700 1 -- 172.21.15.16:0/2498187693 shutdown_connections 2023-10-16T12:12:23.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.403+0000 7f7584d1b700 1 --2- 172.21.15.16:0/2498187693 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f756006b0d0 0x7f756006d590 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:23.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.403+0000 7f7584d1b700 1 --2- 172.21.15.16:0/2498187693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75800b4c20 0x7f75800b50a0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:23.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.403+0000 7f7584d1b700 1 --2- 172.21.15.16:0/2498187693 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7568004850 0x7f75800b6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:23.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.403+0000 7f7584d1b700 1 --2- 172.21.15.16:0/2498187693 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7580155ee0 0x7f7580152970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:23.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.403+0000 7f7584d1b700 1 -- 172.21.15.16:0/2498187693 >> 172.21.15.16:0/2498187693 conn(0x7f75800d15f0 msgr2=0x7f75800afaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:23.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.403+0000 7f7584d1b700 1 -- 172.21.15.16:0/2498187693 shutdown_connections 2023-10-16T12:12:23.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.403+0000 7f7584d1b700 1 -- 172.21.15.16:0/2498187693 wait complete. 2023-10-16T12:12:23.417 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir4 2023-10-16T12:12:23.417 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir5 1m 2023-10-16T12:12:23.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.544+0000 7f5455810700 1 Processor -- start 2023-10-16T12:12:23.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.544+0000 7f5455810700 1 -- start start 2023-10-16T12:12:23.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.544+0000 7f5455810700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f545015d8d0 0x7f54500b2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:23.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.544+0000 7f5455810700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f54500b8fa0 0x7f54500b6720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:23.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.544+0000 7f5455810700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54500b2f30 con 0x7f54500b6c60 2023-10-16T12:12:23.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.545+0000 7f5455810700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54500d0bb0 con 0x7f54500b8fa0 2023-10-16T12:12:23.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.545+0000 7f5455810700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f545015dcb0 con 0x7f545015d8d0 2023-10-16T12:12:23.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.545+0000 7f544e7fc700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f54500b8fa0 0x7f54500b6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:23.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.545+0000 7f544effd700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f545015d8d0 0x7f54500b2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:23.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.545+0000 7f544effd700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f545015d8d0 0x7f54500b2830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:52864/0 (socket says 172.21.15.16:52864) 2023-10-16T12:12:23.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.545+0000 7f544e7fc700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f54500b8fa0 0x7f54500b6720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:35438/0 (socket says 172.21.15.16:35438) 2023-10-16T12:12:23.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.545+0000 7f544effd700 1 -- 172.21.15.16:0/2358309508 learned_addr learned my addr 172.21.15.16:0/2358309508 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:12:23.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.545+0000 7f544f7fe700 1 --1- 172.21.15.16:0/2358309508 >> v1:172.21.15.49:6789/0 conn(0x7f54500b6c60 0x7f54500ce350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:32834/0 (socket says 172.21.15.16:32834) 2023-10-16T12:12:23.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.545+0000 7f544dffb700 1 -- 172.21.15.16:0/2358309508 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2192992762 0 0) 0x7f54500b2f30 con 0x7f54500b6c60 2023-10-16T12:12:23.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.545+0000 7f544dffb700 1 -- 172.21.15.16:0/2358309508 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54380035b0 con 0x7f54500b6c60 2023-10-16T12:12:23.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.545+0000 7f544e7fc700 1 -- 172.21.15.16:0/2358309508 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f545015d8d0 msgr2=0x7f54500b2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:23.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.545+0000 7f544e7fc700 1 --2- 172.21.15.16:0/2358309508 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f545015d8d0 0x7f54500b2830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:23.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.545+0000 7f544e7fc700 1 -- 172.21.15.16:0/2358309508 >> v1:172.21.15.49:6789/0 conn(0x7f54500b6c60 legacy=0x7f54500ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:23.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.545+0000 7f544e7fc700 1 -- 172.21.15.16:0/2358309508 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54500d1c40 con 0x7f54500b8fa0 2023-10-16T12:12:23.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.545+0000 7f544effd700 1 --2- 172.21.15.16:0/2358309508 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f545015d8d0 0x7f54500b2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:12:23.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.545+0000 7f544e7fc700 1 --2- 172.21.15.16:0/2358309508 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f54500b8fa0 0x7f54500b6720 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f5444009840 tx=0x7f5444009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ed6faeb2c8cd4ed server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:23.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.546+0000 7f544dffb700 1 -- 172.21.15.16:0/2358309508 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5444011830 con 0x7f54500b8fa0 2023-10-16T12:12:23.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.546+0000 7f544dffb700 1 -- 172.21.15.16:0/2358309508 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5444011990 con 0x7f54500b8fa0 2023-10-16T12:12:23.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.546+0000 7f544dffb700 1 -- 172.21.15.16:0/2358309508 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5444011d20 con 0x7f54500b8fa0 2023-10-16T12:12:23.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.546+0000 7f5455810700 1 -- 172.21.15.16:0/2358309508 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f54500b8fa0 msgr2=0x7f54500b6720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:23.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.546+0000 7f5455810700 1 --2- 172.21.15.16:0/2358309508 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f54500b8fa0 0x7f54500b6720 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f5444009840 tx=0x7f5444009b50 comp rx=0 tx=0).stop 2023-10-16T12:12:23.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.546+0000 7f5455810700 1 -- 172.21.15.16:0/2358309508 shutdown_connections 2023-10-16T12:12:23.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.546+0000 7f5455810700 1 --2- 172.21.15.16:0/2358309508 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f545015d8d0 0x7f54500b2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:23.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.546+0000 7f5455810700 1 --2- 172.21.15.16:0/2358309508 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f54500b8fa0 0x7f54500b6720 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:23.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.546+0000 7f5455810700 1 -- 172.21.15.16:0/2358309508 >> 172.21.15.16:0/2358309508 conn(0x7f5450072330 msgr2=0x7f5450072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:23.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.546+0000 7f5455810700 1 -- 172.21.15.16:0/2358309508 shutdown_connections 2023-10-16T12:12:23.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.546+0000 7f5455810700 1 -- 172.21.15.16:0/2358309508 wait complete. 2023-10-16T12:12:23.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.548+0000 7f5455810700 1 Processor -- start 2023-10-16T12:12:23.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.548+0000 7f5455810700 1 -- start start 2023-10-16T12:12:23.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.549+0000 7f5455810700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f54500b6c60 0x7f54501f0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:23.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.549+0000 7f5455810700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f54500b8fa0 0x7f54501f27e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:23.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.549+0000 7f5455810700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f545015d8d0 0x7f545015a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:23.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.549+0000 7f5455810700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54500aab80 con 0x7f545015d8d0 2023-10-16T12:12:23.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.549+0000 7f5455810700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54501f45a0 con 0x7f54500b8fa0 2023-10-16T12:12:23.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.549+0000 7f5455810700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54501f4880 con 0x7f54500b6c60 2023-10-16T12:12:23.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.549+0000 7f544f7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f545015d8d0 0x7f545015a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:23.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.549+0000 7f544f7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f545015d8d0 0x7f545015a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36384/0 (socket says 172.21.15.16:36384) 2023-10-16T12:12:23.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.549+0000 7f544f7fe700 1 -- 172.21.15.16:0/1807633905 learned_addr learned my addr 172.21.15.16:0/1807633905 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:12:23.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.549+0000 7f544effd700 1 --2- 172.21.15.16:0/1807633905 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f54500b6c60 0x7f54501f0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:23.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.549+0000 7f544e7fc700 1 --2- 172.21.15.16:0/1807633905 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f54500b8fa0 0x7f54501f27e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:23.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.550+0000 7f544f7fe700 1 -- 172.21.15.16:0/1807633905 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f54500b6c60 msgr2=0x7f54501f0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:23.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.550+0000 7f544f7fe700 1 --2- 172.21.15.16:0/1807633905 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f54500b6c60 0x7f54501f0ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:23.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.550+0000 7f544f7fe700 1 -- 172.21.15.16:0/1807633905 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f54500b8fa0 msgr2=0x7f54501f27e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:23.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.550+0000 7f544f7fe700 1 --2- 172.21.15.16:0/1807633905 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f54500b8fa0 0x7f54501f27e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:23.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.550+0000 7f544f7fe700 1 -- 172.21.15.16:0/1807633905 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5440009550 con 0x7f545015d8d0 2023-10-16T12:12:23.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.550+0000 7f544effd700 1 --2- 172.21.15.16:0/1807633905 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f54500b6c60 0x7f54501f0ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:12:23.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.550+0000 7f544f7fe700 1 --2- 172.21.15.16:0/1807633905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f545015d8d0 0x7f545015a730 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f544800ca80 tx=0x7f544800ce40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:23.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.550+0000 7f5437fff700 1 -- 172.21.15.16:0/1807633905 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f544800aaa0 con 0x7f545015d8d0 2023-10-16T12:12:23.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.550+0000 7f5437fff700 1 -- 172.21.15.16:0/1807633905 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f544800ac00 con 0x7f545015d8d0 2023-10-16T12:12:23.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.550+0000 7f5437fff700 1 -- 172.21.15.16:0/1807633905 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5448007070 con 0x7f545015d8d0 2023-10-16T12:12:23.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.550+0000 7f5455810700 1 -- 172.21.15.16:0/1807633905 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54440094a0 con 0x7f545015d8d0 2023-10-16T12:12:23.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.550+0000 7f5455810700 1 -- 172.21.15.16:0/1807633905 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54501f4e40 con 0x7f545015d8d0 2023-10-16T12:12:23.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.551+0000 7f5455810700 1 -- 172.21.15.16:0/1807633905 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f545004ebb0 con 0x7f545015d8d0 2023-10-16T12:12:23.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.551+0000 7f5437fff700 1 -- 172.21.15.16:0/1807633905 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 10) v1 ==== 89405+0+0 (secure 0 0 0) 0x7f54480071d0 con 0x7f545015d8d0 2023-10-16T12:12:23.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.552+0000 7f5437fff700 1 --2- 172.21.15.16:0/1807633905 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f543c06b100 0x7f543c06d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:23.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.552+0000 7f5437fff700 1 -- 172.21.15.16:0/1807633905 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7f5448089920 con 0x7f545015d8d0 2023-10-16T12:12:23.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.552+0000 7f544effd700 1 --2- 172.21.15.16:0/1807633905 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f543c06b100 0x7f543c06d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:23.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.552+0000 7f544effd700 1 --2- 172.21.15.16:0/1807633905 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f543c06b100 0x7f543c06d5c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f545015a6a0 tx=0x7f5440008040 comp rx=0 tx=0).ready entity=mgr.5499 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:23.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.556+0000 7f5437fff700 1 -- 172.21.15.16:0/1807633905 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f54480545a0 con 0x7f545015d8d0 2023-10-16T12:12:23.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.790+0000 7f5455810700 1 -- 172.21.15.16:0/1807633905 --> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir5", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f54500d21c0 con 0x7f543c06b100 2023-10-16T12:12:23.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.838+0000 7f5437fff700 1 -- 172.21.15.16:0/1807633905 <== mgr.5499 v2:172.21.15.16:6832/1707126831 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f5440009a20 con 0x7f543c06b100 2023-10-16T12:12:23.840 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule activated for path /snap_test_dir1/dir5 2023-10-16T12:12:23.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.840+0000 7f5455810700 1 -- 172.21.15.16:0/1807633905 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f543c06b100 msgr2=0x7f543c06d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:23.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.840+0000 7f5455810700 1 --2- 172.21.15.16:0/1807633905 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f543c06b100 0x7f543c06d5c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f545015a6a0 tx=0x7f5440008040 comp rx=0 tx=0).stop 2023-10-16T12:12:23.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.840+0000 7f5455810700 1 -- 172.21.15.16:0/1807633905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f545015d8d0 msgr2=0x7f545015a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:23.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.840+0000 7f5455810700 1 --2- 172.21.15.16:0/1807633905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f545015d8d0 0x7f545015a730 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f544800ca80 tx=0x7f544800ce40 comp rx=0 tx=0).stop 2023-10-16T12:12:23.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.840+0000 7f5455810700 1 -- 172.21.15.16:0/1807633905 shutdown_connections 2023-10-16T12:12:23.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.840+0000 7f5455810700 1 --2- 172.21.15.16:0/1807633905 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f543c06b100 0x7f543c06d5c0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:23.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.840+0000 7f5455810700 1 --2- 172.21.15.16:0/1807633905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f545015d8d0 0x7f545015a730 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:23.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.840+0000 7f5455810700 1 --2- 172.21.15.16:0/1807633905 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f54500b6c60 0x7f54501f0ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:23.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.840+0000 7f5455810700 1 --2- 172.21.15.16:0/1807633905 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f54500b8fa0 0x7f54501f27e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:23.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.840+0000 7f5455810700 1 -- 172.21.15.16:0/1807633905 >> 172.21.15.16:0/1807633905 conn(0x7f5450072330 msgr2=0x7f545015cc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:23.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.840+0000 7f5455810700 1 -- 172.21.15.16:0/1807633905 shutdown_connections 2023-10-16T12:12:23.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.840+0000 7f5455810700 1 -- 172.21.15.16:0/1807633905 wait complete. 2023-10-16T12:12:23.854 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir5 2023-10-16T12:12:23.855 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir6 1m 2023-10-16T12:12:23.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.980+0000 7fe79be82700 1 Processor -- start 2023-10-16T12:12:23.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.980+0000 7fe79be82700 1 -- start start 2023-10-16T12:12:23.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.981+0000 7fe79be82700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe79415b2c0 0x7fe794161340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:23.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.981+0000 7fe79be82700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fe79406a7e0 0x7fe79406ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:23.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.981+0000 7fe79be82700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe79415ecb0 con 0x7fe79406e570 2023-10-16T12:12:23.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.981+0000 7fe79be82700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe79406b190 con 0x7fe79415b2c0 2023-10-16T12:12:23.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.981+0000 7fe79be82700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe794161880 con 0x7fe79406a7e0 2023-10-16T12:12:23.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.981+0000 7fe799c1e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe79415b2c0 0x7fe794161340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:23.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.981+0000 7fe799c1e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe79415b2c0 0x7fe794161340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36388/0 (socket says 172.21.15.16:36388) 2023-10-16T12:12:23.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.981+0000 7fe799c1e700 1 -- 172.21.15.16:0/1660454176 learned_addr learned my addr 172.21.15.16:0/1660454176 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:12:23.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.982+0000 7fe79a41f700 1 --2- 172.21.15.16:0/1660454176 >> v2:172.21.15.49:3300/0 conn(0x7fe79406a7e0 0x7fe79406ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:23.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.982+0000 7fe799c1e700 1 -- 172.21.15.16:0/1660454176 >> v2:172.21.15.49:3300/0 conn(0x7fe79406a7e0 msgr2=0x7fe79406ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:23.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.982+0000 7fe799c1e700 1 --2- 172.21.15.16:0/1660454176 >> v2:172.21.15.49:3300/0 conn(0x7fe79406a7e0 0x7fe79406ac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:23.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.982+0000 7fe799c1e700 1 -- 172.21.15.16:0/1660454176 >> v1:172.21.15.49:6789/0 conn(0x7fe79406e570 legacy=0x7fe79415e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:23.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.982+0000 7fe799c1e700 1 -- 172.21.15.16:0/1660454176 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe79415b6a0 con 0x7fe79415b2c0 2023-10-16T12:12:23.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.982+0000 7fe799c1e700 1 --2- 172.21.15.16:0/1660454176 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe79415b2c0 0x7fe794161340 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fe79406b880 tx=0x7fe784009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e58bcaea6342233b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:23.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.982+0000 7fe798c1c700 1 -- 172.21.15.16:0/1660454176 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe784004230 con 0x7fe79415b2c0 2023-10-16T12:12:23.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.982+0000 7fe798c1c700 1 -- 172.21.15.16:0/1660454176 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe784004390 con 0x7fe79415b2c0 2023-10-16T12:12:23.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.982+0000 7fe798c1c700 1 -- 172.21.15.16:0/1660454176 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7840036a0 con 0x7fe79415b2c0 2023-10-16T12:12:23.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.983+0000 7fe79be82700 1 -- 172.21.15.16:0/1660454176 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe79415b2c0 msgr2=0x7fe794161340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:23.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.983+0000 7fe79be82700 1 --2- 172.21.15.16:0/1660454176 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe79415b2c0 0x7fe794161340 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fe79406b880 tx=0x7fe784009bd0 comp rx=0 tx=0).stop 2023-10-16T12:12:23.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.983+0000 7fe79be82700 1 -- 172.21.15.16:0/1660454176 shutdown_connections 2023-10-16T12:12:23.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.983+0000 7fe79be82700 1 --2- 172.21.15.16:0/1660454176 >> v2:172.21.15.49:3300/0 conn(0x7fe79406a7e0 0x7fe79406ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:23.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.983+0000 7fe79be82700 1 --2- 172.21.15.16:0/1660454176 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe79415b2c0 0x7fe794161340 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:23.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.983+0000 7fe79be82700 1 -- 172.21.15.16:0/1660454176 >> 172.21.15.16:0/1660454176 conn(0x7fe7940b1f40 msgr2=0x7fe7940b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:23.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.983+0000 7fe79be82700 1 -- 172.21.15.16:0/1660454176 shutdown_connections 2023-10-16T12:12:23.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.983+0000 7fe79be82700 1 -- 172.21.15.16:0/1660454176 wait complete. 2023-10-16T12:12:23.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.984+0000 7fe79be82700 1 Processor -- start 2023-10-16T12:12:23.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.984+0000 7fe79be82700 1 -- start start 2023-10-16T12:12:23.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.985+0000 7fe79be82700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe79406a7e0 0x7fe7940c9e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:23.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.985+0000 7fe79be82700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe79406e570 0x7fe7941f2a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:23.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.985+0000 7fe79be82700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe79415b2c0 0x7fe7940b2f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:23.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.985+0000 7fe79be82700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7941f2f40 con 0x7fe79406a7e0 2023-10-16T12:12:23.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.985+0000 7fe79be82700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7940b4330 con 0x7fe79415b2c0 2023-10-16T12:12:23.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.985+0000 7fe79be82700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7940b3450 con 0x7fe79406e570 2023-10-16T12:12:23.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.985+0000 7fe799c1e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe79406a7e0 0x7fe7940c9e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:23.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.985+0000 7fe799c1e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe79406a7e0 0x7fe7940c9e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36392/0 (socket says 172.21.15.16:36392) 2023-10-16T12:12:23.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.985+0000 7fe799c1e700 1 -- 172.21.15.16:0/146230535 learned_addr learned my addr 172.21.15.16:0/146230535 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:12:23.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.985+0000 7fe79a41f700 1 --2- 172.21.15.16:0/146230535 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe79415b2c0 0x7fe7940b2f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:23.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.985+0000 7fe799c1e700 1 -- 172.21.15.16:0/146230535 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe79406e570 msgr2=0x7fe7941f2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:23.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.985+0000 7fe79941d700 1 --2- 172.21.15.16:0/146230535 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe79406e570 0x7fe7941f2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:23.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.985+0000 7fe799c1e700 1 --2- 172.21.15.16:0/146230535 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe79406e570 0x7fe7941f2a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:23.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.985+0000 7fe799c1e700 1 -- 172.21.15.16:0/146230535 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe79415b2c0 msgr2=0x7fe7940b2f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:23.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.985+0000 7fe799c1e700 1 --2- 172.21.15.16:0/146230535 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe79415b2c0 0x7fe7940b2f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:23.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.985+0000 7fe799c1e700 1 -- 172.21.15.16:0/146230535 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe784009550 con 0x7fe79406a7e0 2023-10-16T12:12:23.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.985+0000 7fe79941d700 1 --2- 172.21.15.16:0/146230535 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe79406e570 0x7fe7941f2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:12:23.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.985+0000 7fe79a41f700 1 --2- 172.21.15.16:0/146230535 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe79415b2c0 0x7fe7940b2f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:12:23.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.985+0000 7fe799c1e700 1 --2- 172.21.15.16:0/146230535 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe79406a7e0 0x7fe7940c9e80 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fe784009130 tx=0x7fe784004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:23.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.985+0000 7fe78a7fc700 1 -- 172.21.15.16:0/146230535 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe784011630 con 0x7fe79406a7e0 2023-10-16T12:12:23.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.985+0000 7fe78a7fc700 1 -- 172.21.15.16:0/146230535 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe784011790 con 0x7fe79406a7e0 2023-10-16T12:12:23.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.985+0000 7fe78a7fc700 1 -- 172.21.15.16:0/146230535 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe784011b40 con 0x7fe79406a7e0 2023-10-16T12:12:23.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.986+0000 7fe79be82700 1 -- 172.21.15.16:0/146230535 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7940b3720 con 0x7fe79406a7e0 2023-10-16T12:12:23.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.986+0000 7fe79be82700 1 -- 172.21.15.16:0/146230535 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7940b3ce0 con 0x7fe79406a7e0 2023-10-16T12:12:23.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.987+0000 7fe79be82700 1 -- 172.21.15.16:0/146230535 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe79404ebb0 con 0x7fe79406a7e0 2023-10-16T12:12:23.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.987+0000 7fe78a7fc700 1 -- 172.21.15.16:0/146230535 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 10) v1 ==== 89405+0+0 (secure 0 0 0) 0x7fe784011ca0 con 0x7fe79406a7e0 2023-10-16T12:12:23.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.987+0000 7fe78a7fc700 1 --2- 172.21.15.16:0/146230535 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fe77806b130 0x7fe77806d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:23.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.987+0000 7fe78a7fc700 1 -- 172.21.15.16:0/146230535 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7fe78408b170 con 0x7fe79406a7e0 2023-10-16T12:12:23.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.987+0000 7fe79941d700 1 --2- 172.21.15.16:0/146230535 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fe77806b130 0x7fe77806d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:23.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.988+0000 7fe79941d700 1 --2- 172.21.15.16:0/146230535 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fe77806b130 0x7fe77806d5f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe790006fd0 tx=0x7fe790008040 comp rx=0 tx=0).ready entity=mgr.5499 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:23.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:23.992+0000 7fe78a7fc700 1 -- 172.21.15.16:0/146230535 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe784055df0 con 0x7fe79406a7e0 2023-10-16T12:12:24.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.228+0000 7fe79be82700 1 -- 172.21.15.16:0/146230535 --> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir6", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fe794161e00 con 0x7fe77806b130 2023-10-16T12:12:24.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.277+0000 7fe78a7fc700 1 -- 172.21.15.16:0/146230535 <== mgr.5499 v2:172.21.15.16:6832/1707126831 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fe790013420 con 0x7fe77806b130 2023-10-16T12:12:24.279 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule activated for path /snap_test_dir1/dir6 2023-10-16T12:12:24.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.279+0000 7fe79be82700 1 -- 172.21.15.16:0/146230535 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fe77806b130 msgr2=0x7fe77806d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:24.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.279+0000 7fe79be82700 1 --2- 172.21.15.16:0/146230535 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fe77806b130 0x7fe77806d5f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe790006fd0 tx=0x7fe790008040 comp rx=0 tx=0).stop 2023-10-16T12:12:24.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.279+0000 7fe79be82700 1 -- 172.21.15.16:0/146230535 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe79406a7e0 msgr2=0x7fe7940c9e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:24.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.279+0000 7fe79be82700 1 --2- 172.21.15.16:0/146230535 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe79406a7e0 0x7fe7940c9e80 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fe784009130 tx=0x7fe784004750 comp rx=0 tx=0).stop 2023-10-16T12:12:24.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.279+0000 7fe79be82700 1 -- 172.21.15.16:0/146230535 shutdown_connections 2023-10-16T12:12:24.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.279+0000 7fe79be82700 1 --2- 172.21.15.16:0/146230535 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fe77806b130 0x7fe77806d5f0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:24.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.279+0000 7fe79be82700 1 --2- 172.21.15.16:0/146230535 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe79415b2c0 0x7fe7940b2f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:24.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.279+0000 7fe79be82700 1 --2- 172.21.15.16:0/146230535 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe79406a7e0 0x7fe7940c9e80 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:24.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.279+0000 7fe79be82700 1 --2- 172.21.15.16:0/146230535 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe79406e570 0x7fe7941f2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:24.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.279+0000 7fe79be82700 1 -- 172.21.15.16:0/146230535 >> 172.21.15.16:0/146230535 conn(0x7fe7940b1f40 msgr2=0x7fe79415b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:24.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.279+0000 7fe79be82700 1 -- 172.21.15.16:0/146230535 shutdown_connections 2023-10-16T12:12:24.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.279+0000 7fe79be82700 1 -- 172.21.15.16:0/146230535 wait complete. 2023-10-16T12:12:24.293 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir6 2023-10-16T12:12:24.293 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir7 1m 2023-10-16T12:12:24.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.420+0000 7f4c68b1e700 1 Processor -- start 2023-10-16T12:12:24.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.420+0000 7f4c68b1e700 1 -- start start 2023-10-16T12:12:24.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.421+0000 7f4c68b1e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4c640cb880 0x7f4c640cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:24.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.421+0000 7f4c68b1e700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c640cef70 con 0x7f4c6415b070 2023-10-16T12:12:24.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.421+0000 7f4c68b1e700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c640ce360 con 0x7f4c6406da60 2023-10-16T12:12:24.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.421+0000 7f4c68b1e700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c640ce640 con 0x7f4c640cb880 2023-10-16T12:12:24.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.421+0000 7f4c61d9b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4c640cb880 0x7f4c640cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:24.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.421+0000 7f4c61d9b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4c640cb880 0x7f4c640cbd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36402/0 (socket says 172.21.15.16:36402) 2023-10-16T12:12:24.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.421+0000 7f4c61d9b700 1 -- 172.21.15.16:0/3885022842 learned_addr learned my addr 172.21.15.16:0/3885022842 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:12:24.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.421+0000 7f4c61d9b700 1 -- 172.21.15.16:0/3885022842 >> v1:172.21.15.49:6789/0 conn(0x7f4c6406da60 legacy=0x7f4c640cf400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:24.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.421+0000 7f4c61d9b700 1 -- 172.21.15.16:0/3885022842 >> v1:172.21.15.16:6789/0 conn(0x7f4c6415b070 legacy=0x7f4c6415b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:24.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.421+0000 7f4c61d9b700 1 -- 172.21.15.16:0/3885022842 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c640ce920 con 0x7f4c640cb880 2023-10-16T12:12:24.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.421+0000 7f4c61d9b700 1 --2- 172.21.15.16:0/3885022842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c640cb880 0x7f4c640cbd00 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c00bb90 tx=0x7f4c4c00bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa9ff0a12a6237b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:24.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.421+0000 7f4c6159a700 1 -- 172.21.15.16:0/3885022842 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4c00dd10 con 0x7f4c640cb880 2023-10-16T12:12:24.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.421+0000 7f4c6159a700 1 -- 172.21.15.16:0/3885022842 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c4c00de70 con 0x7f4c640cb880 2023-10-16T12:12:24.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.421+0000 7f4c6159a700 1 -- 172.21.15.16:0/3885022842 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4c004230 con 0x7f4c640cb880 2023-10-16T12:12:24.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.422+0000 7f4c68b1e700 1 -- 172.21.15.16:0/3885022842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c640cb880 msgr2=0x7f4c640cbd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:24.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.422+0000 7f4c68b1e700 1 --2- 172.21.15.16:0/3885022842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c640cb880 0x7f4c640cbd00 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c00bb90 tx=0x7f4c4c00bea0 comp rx=0 tx=0).stop 2023-10-16T12:12:24.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.422+0000 7f4c68b1e700 1 -- 172.21.15.16:0/3885022842 shutdown_connections 2023-10-16T12:12:24.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.422+0000 7f4c68b1e700 1 --2- 172.21.15.16:0/3885022842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c640cb880 0x7f4c640cbd00 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:24.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.422+0000 7f4c68b1e700 1 -- 172.21.15.16:0/3885022842 >> 172.21.15.16:0/3885022842 conn(0x7f4c640b1f40 msgr2=0x7f4c640b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:24.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.422+0000 7f4c68b1e700 1 -- 172.21.15.16:0/3885022842 shutdown_connections 2023-10-16T12:12:24.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.422+0000 7f4c68b1e700 1 -- 172.21.15.16:0/3885022842 wait complete. 2023-10-16T12:12:24.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.423+0000 7f4c68b1e700 1 Processor -- start 2023-10-16T12:12:24.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.423+0000 7f4c68b1e700 1 -- start start 2023-10-16T12:12:24.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.424+0000 7f4c68b1e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c6406da60 0x7f4c64150820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:24.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.424+0000 7f4c68b1e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4c640cb880 0x7f4c640b3b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:24.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.424+0000 7f4c68b1e700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4c6415b070 0x7f4c641525d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:24.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.424+0000 7f4c68b1e700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c6415cbc0 con 0x7f4c6406da60 2023-10-16T12:12:24.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.424+0000 7f4c68b1e700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c6415cea0 con 0x7f4c6415b070 2023-10-16T12:12:24.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.424+0000 7f4c68b1e700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c640b3030 con 0x7f4c640cb880 2023-10-16T12:12:24.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.424+0000 7f4c6259c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c6406da60 0x7f4c64150820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:24.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.424+0000 7f4c6259c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c6406da60 0x7f4c64150820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36406/0 (socket says 172.21.15.16:36406) 2023-10-16T12:12:24.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.424+0000 7f4c6259c700 1 -- 172.21.15.16:0/958969138 learned_addr learned my addr 172.21.15.16:0/958969138 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:12:24.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.424+0000 7f4c62d9d700 1 --2- 172.21.15.16:0/958969138 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4c6415b070 0x7f4c641525d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:24.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.424+0000 7f4c61d9b700 1 --2- 172.21.15.16:0/958969138 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4c640cb880 0x7f4c640b3b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:24.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.425+0000 7f4c6259c700 1 -- 172.21.15.16:0/958969138 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4c640cb880 msgr2=0x7f4c640b3b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:24.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.425+0000 7f4c6259c700 1 --2- 172.21.15.16:0/958969138 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4c640cb880 0x7f4c640b3b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:24.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.425+0000 7f4c6259c700 1 -- 172.21.15.16:0/958969138 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4c6415b070 msgr2=0x7f4c641525d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:24.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.425+0000 7f4c6259c700 1 --2- 172.21.15.16:0/958969138 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4c6415b070 0x7f4c641525d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:24.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.425+0000 7f4c6259c700 1 -- 172.21.15.16:0/958969138 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c4c00b7f0 con 0x7f4c6406da60 2023-10-16T12:12:24.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.425+0000 7f4c6259c700 1 --2- 172.21.15.16:0/958969138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c6406da60 0x7f4c64150820 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f4c5400c430 tx=0x7f4c5400c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:24.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.425+0000 7f4c5affd700 1 -- 172.21.15.16:0/958969138 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c540052d0 con 0x7f4c6406da60 2023-10-16T12:12:24.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.425+0000 7f4c5affd700 1 -- 172.21.15.16:0/958969138 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c54005450 con 0x7f4c6406da60 2023-10-16T12:12:24.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.425+0000 7f4c68b1e700 1 -- 172.21.15.16:0/958969138 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c640b3310 con 0x7f4c6406da60 2023-10-16T12:12:24.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.425+0000 7f4c5affd700 1 -- 172.21.15.16:0/958969138 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c54006aa0 con 0x7f4c6406da60 2023-10-16T12:12:24.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.425+0000 7f4c68b1e700 1 -- 172.21.15.16:0/958969138 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c64150d60 con 0x7f4c6406da60 2023-10-16T12:12:24.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.426+0000 7f4c68b1e700 1 -- 172.21.15.16:0/958969138 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c640b3830 con 0x7f4c6406da60 2023-10-16T12:12:24.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.426+0000 7f4c5affd700 1 -- 172.21.15.16:0/958969138 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 10) v1 ==== 89405+0+0 (secure 0 0 0) 0x7f4c54003680 con 0x7f4c6406da60 2023-10-16T12:12:24.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.427+0000 7f4c5affd700 1 --2- 172.21.15.16:0/958969138 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f4c4406b130 0x7f4c4406d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:24.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.427+0000 7f4c5affd700 1 -- 172.21.15.16:0/958969138 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7f4c54089150 con 0x7f4c6406da60 2023-10-16T12:12:24.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.427+0000 7f4c61d9b700 1 --2- 172.21.15.16:0/958969138 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f4c4406b130 0x7f4c4406d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:24.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.427+0000 7f4c61d9b700 1 --2- 172.21.15.16:0/958969138 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f4c4406b130 0x7f4c4406d5f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c00b340 tx=0x7f4c4c013000 comp rx=0 tx=0).ready entity=mgr.5499 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:24.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.431+0000 7f4c5affd700 1 -- 172.21.15.16:0/958969138 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4c54053d90 con 0x7f4c6406da60 2023-10-16T12:12:24.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.666+0000 7f4c68b1e700 1 -- 172.21.15.16:0/958969138 --> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir7", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f4c640728d0 con 0x7f4c4406b130 2023-10-16T12:12:24.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.713+0000 7f4c5affd700 1 -- 172.21.15.16:0/958969138 <== mgr.5499 v2:172.21.15.16:6832/1707126831 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f4c4c021040 con 0x7f4c4406b130 2023-10-16T12:12:24.715 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule activated for path /snap_test_dir1/dir7 2023-10-16T12:12:24.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.715+0000 7f4c68b1e700 1 -- 172.21.15.16:0/958969138 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f4c4406b130 msgr2=0x7f4c4406d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:24.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.715+0000 7f4c68b1e700 1 --2- 172.21.15.16:0/958969138 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f4c4406b130 0x7f4c4406d5f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c00b340 tx=0x7f4c4c013000 comp rx=0 tx=0).stop 2023-10-16T12:12:24.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.715+0000 7f4c68b1e700 1 -- 172.21.15.16:0/958969138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c6406da60 msgr2=0x7f4c64150820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:24.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.715+0000 7f4c68b1e700 1 --2- 172.21.15.16:0/958969138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c6406da60 0x7f4c64150820 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f4c5400c430 tx=0x7f4c5400c7f0 comp rx=0 tx=0).stop 2023-10-16T12:12:24.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.715+0000 7f4c68b1e700 1 -- 172.21.15.16:0/958969138 shutdown_connections 2023-10-16T12:12:24.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.715+0000 7f4c68b1e700 1 --2- 172.21.15.16:0/958969138 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f4c4406b130 0x7f4c4406d5f0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:24.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.715+0000 7f4c68b1e700 1 --2- 172.21.15.16:0/958969138 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4c6415b070 0x7f4c641525d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:24.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.715+0000 7f4c68b1e700 1 --2- 172.21.15.16:0/958969138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c6406da60 0x7f4c64150820 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:24.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.715+0000 7f4c68b1e700 1 --2- 172.21.15.16:0/958969138 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4c640cb880 0x7f4c640b3b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:24.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.715+0000 7f4c68b1e700 1 -- 172.21.15.16:0/958969138 >> 172.21.15.16:0/958969138 conn(0x7f4c640b1f40 msgr2=0x7f4c641586f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:24.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.715+0000 7f4c68b1e700 1 -- 172.21.15.16:0/958969138 shutdown_connections 2023-10-16T12:12:24.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.715+0000 7f4c68b1e700 1 -- 172.21.15.16:0/958969138 wait complete. 2023-10-16T12:12:24.729 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir7 2023-10-16T12:12:24.730 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir8 1m 2023-10-16T12:12:24.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.856+0000 7f7805419700 1 Processor -- start 2023-10-16T12:12:24.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.856+0000 7f7805419700 1 -- start start 2023-10-16T12:12:24.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.857+0000 7f7805419700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f780015b070 0x7f780015b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:24.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.857+0000 7f7805419700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f780006e170 0x7f78000cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:24.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.857+0000 7f7805419700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78000cef70 con 0x7f78000cb880 2023-10-16T12:12:24.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.857+0000 7f7805419700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78000ce360 con 0x7f780015b070 2023-10-16T12:12:24.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.857+0000 7f7805419700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78000ce640 con 0x7f780006e170 2023-10-16T12:12:24.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.857+0000 7f77feffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f780015b070 0x7f780015b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:24.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.857+0000 7f77feffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f780015b070 0x7f780015b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36414/0 (socket says 172.21.15.16:36414) 2023-10-16T12:12:24.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.857+0000 7f77feffd700 1 -- 172.21.15.16:0/626735343 learned_addr learned my addr 172.21.15.16:0/626735343 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:12:24.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.857+0000 7f77f67fc700 1 --1- 172.21.15.16:0/626735343 >> v1:172.21.15.16:6789/0 conn(0x7f78000cb880 0x7f780006da60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:53920/0 (socket says 172.21.15.16:53920) 2023-10-16T12:12:24.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.857+0000 7f77ff7fe700 1 --2- 172.21.15.16:0/626735343 >> v2:172.21.15.49:3300/0 conn(0x7f780006e170 0x7f78000cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:24.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.857+0000 7f77feffd700 1 -- 172.21.15.16:0/626735343 >> v2:172.21.15.49:3300/0 conn(0x7f780006e170 msgr2=0x7f78000cbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:24.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.857+0000 7f77feffd700 1 --2- 172.21.15.16:0/626735343 >> v2:172.21.15.49:3300/0 conn(0x7f780006e170 0x7f78000cbd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:24.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.857+0000 7f77feffd700 1 -- 172.21.15.16:0/626735343 >> v1:172.21.15.16:6789/0 conn(0x7f78000cb880 legacy=0x7f780006da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:24.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.858+0000 7f77feffd700 1 -- 172.21.15.16:0/626735343 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78000ce920 con 0x7f780015b070 2023-10-16T12:12:24.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.857+0000 7f77fe7fc700 1 -- 172.21.15.16:0/626735343 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3791818758 0 0) 0x7f78000cef70 con 0x7f78000cb880 2023-10-16T12:12:24.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.858+0000 7f77feffd700 1 --2- 172.21.15.16:0/626735343 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f780015b070 0x7f780015b450 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f780015aad0 tx=0x7f77f0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=836c4c19e8415e38 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:24.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.858+0000 7f77fe7fc700 1 -- 172.21.15.16:0/626735343 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77f000b870 con 0x7f780015b070 2023-10-16T12:12:24.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.858+0000 7f77fe7fc700 1 -- 172.21.15.16:0/626735343 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77f000b9d0 con 0x7f780015b070 2023-10-16T12:12:24.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.858+0000 7f77fe7fc700 1 -- 172.21.15.16:0/626735343 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77f000bd60 con 0x7f780015b070 2023-10-16T12:12:24.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.858+0000 7f7805419700 1 -- 172.21.15.16:0/626735343 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f780015b070 msgr2=0x7f780015b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:24.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.858+0000 7f7805419700 1 --2- 172.21.15.16:0/626735343 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f780015b070 0x7f780015b450 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f780015aad0 tx=0x7f77f0009bd0 comp rx=0 tx=0).stop 2023-10-16T12:12:24.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.858+0000 7f7805419700 1 -- 172.21.15.16:0/626735343 shutdown_connections 2023-10-16T12:12:24.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.858+0000 7f7805419700 1 --2- 172.21.15.16:0/626735343 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f780015b070 0x7f780015b450 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:24.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.858+0000 7f7805419700 1 --2- 172.21.15.16:0/626735343 >> v2:172.21.15.49:3300/0 conn(0x7f780006e170 0x7f78000cbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:24.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.858+0000 7f7805419700 1 -- 172.21.15.16:0/626735343 >> 172.21.15.16:0/626735343 conn(0x7f78000b1f40 msgr2=0x7f78000b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:24.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.858+0000 7f7805419700 1 -- 172.21.15.16:0/626735343 shutdown_connections 2023-10-16T12:12:24.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.858+0000 7f7805419700 1 -- 172.21.15.16:0/626735343 wait complete. 2023-10-16T12:12:24.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.860+0000 7f7805419700 1 Processor -- start 2023-10-16T12:12:24.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.860+0000 7f7805419700 1 -- start start 2023-10-16T12:12:24.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.861+0000 7f7805419700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f780006e170 0x7f78000c9a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:24.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.861+0000 7f7805419700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f78000cb880 0x7f78000cc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:24.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.861+0000 7f7805419700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f780015b070 0x7f78001535c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:24.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.861+0000 7f7805419700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78001584f0 con 0x7f780015b070 2023-10-16T12:12:24.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.861+0000 7f7805419700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78000b5370 con 0x7f780006e170 2023-10-16T12:12:24.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.861+0000 7f7805419700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78000b5650 con 0x7f78000cb880 2023-10-16T12:12:24.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.861+0000 7f77ff7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f780015b070 0x7f78001535c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:24.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.861+0000 7f77ff7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f780015b070 0x7f78001535c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36422/0 (socket says 172.21.15.16:36422) 2023-10-16T12:12:24.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.861+0000 7f77ff7fe700 1 -- 172.21.15.16:0/2673397520 learned_addr learned my addr 172.21.15.16:0/2673397520 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:12:24.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.861+0000 7f77feffd700 1 --2- 172.21.15.16:0/2673397520 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f780006e170 0x7f78000c9a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:24.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.861+0000 7f77f67fc700 1 --2- 172.21.15.16:0/2673397520 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f78000cb880 0x7f78000cc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:24.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.862+0000 7f77ff7fe700 1 -- 172.21.15.16:0/2673397520 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f78000cb880 msgr2=0x7f78000cc7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:24.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.862+0000 7f77ff7fe700 1 --2- 172.21.15.16:0/2673397520 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f78000cb880 0x7f78000cc7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:24.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.862+0000 7f77ff7fe700 1 -- 172.21.15.16:0/2673397520 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f780006e170 msgr2=0x7f78000c9a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:24.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.862+0000 7f77ff7fe700 1 --2- 172.21.15.16:0/2673397520 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f780006e170 0x7f78000c9a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:24.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.862+0000 7f77ff7fe700 1 -- 172.21.15.16:0/2673397520 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77f0009550 con 0x7f780015b070 2023-10-16T12:12:24.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.862+0000 7f77f67fc700 1 --2- 172.21.15.16:0/2673397520 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f78000cb880 0x7f78000cc7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:12:24.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.862+0000 7f77ff7fe700 1 --2- 172.21.15.16:0/2673397520 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f780015b070 0x7f78001535c0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f77f800cd20 tx=0x7f77f800a570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:24.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.862+0000 7f77f7fff700 1 -- 172.21.15.16:0/2673397520 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77f800ce70 con 0x7f780015b070 2023-10-16T12:12:24.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.862+0000 7f77f7fff700 1 -- 172.21.15.16:0/2673397520 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77f8003a20 con 0x7f780015b070 2023-10-16T12:12:24.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.862+0000 7f77f7fff700 1 -- 172.21.15.16:0/2673397520 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77f8003db0 con 0x7f780015b070 2023-10-16T12:12:24.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.862+0000 7f7805419700 1 -- 172.21.15.16:0/2673397520 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78000b5930 con 0x7f780015b070 2023-10-16T12:12:24.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.862+0000 7f7805419700 1 -- 172.21.15.16:0/2673397520 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78000c9fb0 con 0x7f780015b070 2023-10-16T12:12:24.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.863+0000 7f7805419700 1 -- 172.21.15.16:0/2673397520 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78000b5e50 con 0x7f780015b070 2023-10-16T12:12:24.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.863+0000 7f77f7fff700 1 -- 172.21.15.16:0/2673397520 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 10) v1 ==== 89405+0+0 (secure 0 0 0) 0x7f77f8003f10 con 0x7f780015b070 2023-10-16T12:12:24.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.864+0000 7f77f7fff700 1 --2- 172.21.15.16:0/2673397520 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f77e006b100 0x7f77e006d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:24.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.864+0000 7f77f7fff700 1 -- 172.21.15.16:0/2673397520 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7f77f8012070 con 0x7f780015b070 2023-10-16T12:12:24.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.864+0000 7f77feffd700 1 --2- 172.21.15.16:0/2673397520 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f77e006b100 0x7f77e006d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:24.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.864+0000 7f77feffd700 1 --2- 172.21.15.16:0/2673397520 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f77e006b100 0x7f77e006d5c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f77f0014040 tx=0x7f77f0008030 comp rx=0 tx=0).ready entity=mgr.5499 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:24.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:24.868+0000 7f77f7fff700 1 -- 172.21.15.16:0/2673397520 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f78000b5e50 con 0x7f780015b070 2023-10-16T12:12:25.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.105+0000 7f7805419700 1 -- 172.21.15.16:0/2673397520 --> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir8", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f78000728d0 con 0x7f77e006b100 2023-10-16T12:12:25.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.152+0000 7f77f7fff700 1 -- 172.21.15.16:0/2673397520 <== mgr.5499 v2:172.21.15.16:6832/1707126831 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f77f000bb40 con 0x7f77e006b100 2023-10-16T12:12:25.154 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule activated for path /snap_test_dir1/dir8 2023-10-16T12:12:25.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.154+0000 7f7805419700 1 -- 172.21.15.16:0/2673397520 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f77e006b100 msgr2=0x7f77e006d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:25.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.154+0000 7f7805419700 1 --2- 172.21.15.16:0/2673397520 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f77e006b100 0x7f77e006d5c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f77f0014040 tx=0x7f77f0008030 comp rx=0 tx=0).stop 2023-10-16T12:12:25.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.154+0000 7f7805419700 1 -- 172.21.15.16:0/2673397520 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f780015b070 msgr2=0x7f78001535c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:25.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.154+0000 7f7805419700 1 --2- 172.21.15.16:0/2673397520 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f780015b070 0x7f78001535c0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f77f800cd20 tx=0x7f77f800a570 comp rx=0 tx=0).stop 2023-10-16T12:12:25.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.154+0000 7f7805419700 1 -- 172.21.15.16:0/2673397520 shutdown_connections 2023-10-16T12:12:25.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.154+0000 7f7805419700 1 --2- 172.21.15.16:0/2673397520 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f77e006b100 0x7f77e006d5c0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:25.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.154+0000 7f7805419700 1 --2- 172.21.15.16:0/2673397520 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f780015b070 0x7f78001535c0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:25.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.154+0000 7f7805419700 1 --2- 172.21.15.16:0/2673397520 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f780006e170 0x7f78000c9a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:25.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.154+0000 7f7805419700 1 --2- 172.21.15.16:0/2673397520 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f78000cb880 0x7f78000cc7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:25.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.154+0000 7f7805419700 1 -- 172.21.15.16:0/2673397520 >> 172.21.15.16:0/2673397520 conn(0x7f78000b1f40 msgr2=0x7f78000b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:25.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.154+0000 7f7805419700 1 -- 172.21.15.16:0/2673397520 shutdown_connections 2023-10-16T12:12:25.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.154+0000 7f7805419700 1 -- 172.21.15.16:0/2673397520 wait complete. 2023-10-16T12:12:25.168 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir8 2023-10-16T12:12:25.169 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir9 1m 2023-10-16T12:12:25.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.295+0000 7fae1a545700 1 Processor -- start 2023-10-16T12:12:25.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.295+0000 7fae1a545700 1 -- start start 2023-10-16T12:12:25.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.296+0000 7fae1a545700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fae1406e170 0x7fae140cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:25.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.296+0000 7fae1a545700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae140cef70 con 0x7fae140cb880 2023-10-16T12:12:25.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.296+0000 7fae1a545700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae140ce360 con 0x7fae1415b070 2023-10-16T12:12:25.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.296+0000 7fae1a545700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae140ce640 con 0x7fae1406e170 2023-10-16T12:12:25.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.296+0000 7fae137fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fae140cb880 0x7fae1406da60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:53932/0 (socket says 172.21.15.16:53932) 2023-10-16T12:12:25.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.296+0000 7fae137fe700 1 -- 172.21.15.16:0/2475313672 learned_addr learned my addr 172.21.15.16:0/2475313672 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:12:25.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.296+0000 7fae18ae2700 1 --2- 172.21.15.16:0/2475313672 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fae1406e170 0x7fae140cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:25.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.296+0000 7fae12ffd700 1 -- 172.21.15.16:0/2475313672 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3156207471 0 0) 0x7fae140cef70 con 0x7fae140cb880 2023-10-16T12:12:25.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.297+0000 7fae12ffd700 1 -- 172.21.15.16:0/2475313672 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae000035b0 con 0x7fae140cb880 2023-10-16T12:12:25.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.297+0000 7fae12ffd700 1 -- 172.21.15.16:0/2475313672 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2190021592 0 0) 0x7fae140ce360 con 0x7fae1415b070 2023-10-16T12:12:25.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.297+0000 7fae12ffd700 1 -- 172.21.15.16:0/2475313672 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae140cef70 con 0x7fae1415b070 2023-10-16T12:12:25.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.297+0000 7fae12ffd700 1 -- 172.21.15.16:0/2475313672 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3015373211 0 0) 0x7fae000035b0 con 0x7fae140cb880 2023-10-16T12:12:25.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.297+0000 7fae12ffd700 1 -- 172.21.15.16:0/2475313672 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fae1406e170 msgr2=0x7fae140cbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:25.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.297+0000 7fae12ffd700 1 --2- 172.21.15.16:0/2475313672 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fae1406e170 0x7fae140cbd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:25.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.297+0000 7fae12ffd700 1 -- 172.21.15.16:0/2475313672 >> v1:172.21.15.49:6789/0 conn(0x7fae1415b070 legacy=0x7fae1415b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:25.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.297+0000 7fae12ffd700 1 -- 172.21.15.16:0/2475313672 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae140ce920 con 0x7fae140cb880 2023-10-16T12:12:25.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.297+0000 7fae12ffd700 1 -- 172.21.15.16:0/2475313672 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7fadfc0015f0 con 0x7fae140cb880 2023-10-16T12:12:25.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.297+0000 7fae18ae2700 1 --2- 172.21.15.16:0/2475313672 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fae1406e170 0x7fae140cbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:12:25.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.297+0000 7fae12ffd700 1 -- 172.21.15.16:0/2475313672 >> v1:172.21.15.16:6789/0 conn(0x7fae140cb880 legacy=0x7fae1406da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:25.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.297+0000 7fae12ffd700 1 --2- 172.21.15.16:0/2475313672 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae00003bc0 0x7fae00003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:25.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.297+0000 7fae12ffd700 1 --2- 172.21.15.16:0/2475313672 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fae00004850 0x7fae00004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:25.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.297+0000 7fae12ffd700 1 --2- 172.21.15.16:0/2475313672 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fae1406e170 0x7fae00005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:25.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.297+0000 7fae12ffd700 1 -- 172.21.15.16:0/2475313672 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae140ce640 con 0x7fae00003bc0 2023-10-16T12:12:25.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.297+0000 7fae12ffd700 1 -- 172.21.15.16:0/2475313672 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae0000bb50 con 0x7fae00004850 2023-10-16T12:12:25.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.297+0000 7fae12ffd700 1 -- 172.21.15.16:0/2475313672 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae0000bd10 con 0x7fae1406e170 2023-10-16T12:12:25.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.297+0000 7fae13fff700 1 --2- 172.21.15.16:0/2475313672 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae00003bc0 0x7fae00003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:25.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.297+0000 7fae137fe700 1 --2- 172.21.15.16:0/2475313672 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fae00004850 0x7fae00004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:25.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.297+0000 7fae18ae2700 1 --2- 172.21.15.16:0/2475313672 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fae1406e170 0x7fae00005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:25.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.297+0000 7fae13fff700 1 -- 172.21.15.16:0/2475313672 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fae1406e170 msgr2=0x7fae00005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:25.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.297+0000 7fae13fff700 1 --2- 172.21.15.16:0/2475313672 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fae1406e170 0x7fae00005230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:25.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.297+0000 7fae13fff700 1 -- 172.21.15.16:0/2475313672 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fae00004850 msgr2=0x7fae00004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:12:25.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.297+0000 7fae13fff700 1 --2- 172.21.15.16:0/2475313672 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fae00004850 0x7fae00004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:25.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.297+0000 7fae13fff700 1 -- 172.21.15.16:0/2475313672 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae0c009a70 con 0x7fae00003bc0 2023-10-16T12:12:25.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.298+0000 7fae13fff700 1 --2- 172.21.15.16:0/2475313672 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae00003bc0 0x7fae00003fc0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fae04001980 tx=0x7fae00005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=608efcce14b428d6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:25.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.298+0000 7fae12ffd700 1 -- 172.21.15.16:0/2475313672 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae04001470 con 0x7fae00003bc0 2023-10-16T12:12:25.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.298+0000 7fae12ffd700 1 -- 172.21.15.16:0/2475313672 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae040052d0 con 0x7fae00003bc0 2023-10-16T12:12:25.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.298+0000 7fae12ffd700 1 -- 172.21.15.16:0/2475313672 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae04003680 con 0x7fae00003bc0 2023-10-16T12:12:25.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.298+0000 7fae1a545700 1 -- 172.21.15.16:0/2475313672 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae00003bc0 msgr2=0x7fae00003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:25.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.298+0000 7fae1a545700 1 --2- 172.21.15.16:0/2475313672 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae00003bc0 0x7fae00003fc0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fae04001980 tx=0x7fae00005880 comp rx=0 tx=0).stop 2023-10-16T12:12:25.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.298+0000 7fae18ae2700 1 -- 172.21.15.16:0/2475313672 reap_dead start 2023-10-16T12:12:25.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.298+0000 7fae1a545700 1 -- 172.21.15.16:0/2475313672 shutdown_connections 2023-10-16T12:12:25.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.298+0000 7fae1a545700 1 -- 172.21.15.16:0/2475313672 >> 172.21.15.16:0/2475313672 conn(0x7fae140b1f40 msgr2=0x7fae140b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:25.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.298+0000 7fae1a545700 1 -- 172.21.15.16:0/2475313672 shutdown_connections 2023-10-16T12:12:25.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.298+0000 7fae1a545700 1 -- 172.21.15.16:0/2475313672 wait complete. 2023-10-16T12:12:25.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.299+0000 7fae1a545700 1 Processor -- start 2023-10-16T12:12:25.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.300+0000 7fae1a545700 1 -- start start 2023-10-16T12:12:25.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.300+0000 7fae1a545700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae00003bc0 0x7fae140cf8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:25.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.300+0000 7fae1a545700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fae140b5a50 0x7fae140c6000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:25.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.300+0000 7fae1a545700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fae14159160 0x7fae140c9a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:25.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.300+0000 7fae1a545700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae140b5ed0 con 0x7fae00003bc0 2023-10-16T12:12:25.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.300+0000 7fae1a545700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae140c6540 con 0x7fae14159160 2023-10-16T12:12:25.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.300+0000 7fae1a545700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae14159920 con 0x7fae140b5a50 2023-10-16T12:12:25.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.301+0000 7fae13fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae00003bc0 0x7fae140cf8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:25.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.301+0000 7fae13fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae00003bc0 0x7fae140cf8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36438/0 (socket says 172.21.15.16:36438) 2023-10-16T12:12:25.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.301+0000 7fae13fff700 1 -- 172.21.15.16:0/234671820 learned_addr learned my addr 172.21.15.16:0/234671820 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:12:25.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.301+0000 7fae137fe700 1 --2- 172.21.15.16:0/234671820 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fae140b5a50 0x7fae140c6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:25.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.301+0000 7fae18ae2700 1 --2- 172.21.15.16:0/234671820 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fae14159160 0x7fae140c9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:25.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.301+0000 7fae13fff700 1 -- 172.21.15.16:0/234671820 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fae140b5a50 msgr2=0x7fae140c6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:25.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.301+0000 7fae13fff700 1 --2- 172.21.15.16:0/234671820 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fae140b5a50 0x7fae140c6000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:25.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.301+0000 7fae13fff700 1 -- 172.21.15.16:0/234671820 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fae14159160 msgr2=0x7fae140c9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:25.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.301+0000 7fae13fff700 1 --2- 172.21.15.16:0/234671820 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fae14159160 0x7fae140c9a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:25.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.301+0000 7fae13fff700 1 -- 172.21.15.16:0/234671820 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae14159c00 con 0x7fae00003bc0 2023-10-16T12:12:25.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.301+0000 7fae13fff700 1 --2- 172.21.15.16:0/234671820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae00003bc0 0x7fae140cf8c0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fae0400c3c0 tx=0x7fae04001310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:25.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.301+0000 7fae10ff9700 1 -- 172.21.15.16:0/234671820 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae04001410 con 0x7fae00003bc0 2023-10-16T12:12:25.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.301+0000 7fae10ff9700 1 -- 172.21.15.16:0/234671820 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae04007040 con 0x7fae00003bc0 2023-10-16T12:12:25.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.301+0000 7fae10ff9700 1 -- 172.21.15.16:0/234671820 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae040073f0 con 0x7fae00003bc0 2023-10-16T12:12:25.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.301+0000 7fae1a545700 1 -- 172.21.15.16:0/234671820 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae140b5250 con 0x7fae00003bc0 2023-10-16T12:12:25.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.301+0000 7fae1a545700 1 -- 172.21.15.16:0/234671820 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae140b4f40 con 0x7fae00003bc0 2023-10-16T12:12:25.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.302+0000 7fae1a545700 1 -- 172.21.15.16:0/234671820 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae140b5e30 con 0x7fae00003bc0 2023-10-16T12:12:25.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.303+0000 7fae10ff9700 1 -- 172.21.15.16:0/234671820 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 10) v1 ==== 89405+0+0 (secure 0 0 0) 0x7fae04007550 con 0x7fae00003bc0 2023-10-16T12:12:25.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.303+0000 7fae10ff9700 1 --2- 172.21.15.16:0/234671820 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fadf406b100 0x7fadf406d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:12:25.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.303+0000 7fae10ff9700 1 -- 172.21.15.16:0/234671820 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7fae0408ad10 con 0x7fae00003bc0 2023-10-16T12:12:25.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.303+0000 7fae137fe700 1 --2- 172.21.15.16:0/234671820 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fadf406b100 0x7fadf406d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:12:25.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.304+0000 7fae137fe700 1 --2- 172.21.15.16:0/234671820 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fadf406b100 0x7fadf406d5c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fadfc000c00 tx=0x7fadfc009040 comp rx=0 tx=0).ready entity=mgr.5499 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:12:25.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.308+0000 7fae10ff9700 1 -- 172.21.15.16:0/234671820 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fae04055990 con 0x7fae00003bc0 2023-10-16T12:12:25.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.543+0000 7fae1a545700 1 -- 172.21.15.16:0/234671820 --> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir9", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fae140728d0 con 0x7fadf406b100 2023-10-16T12:12:25.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.590+0000 7fae10ff9700 1 -- 172.21.15.16:0/234671820 <== mgr.5499 v2:172.21.15.16:6832/1707126831 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fadfc007e70 con 0x7fadf406b100 2023-10-16T12:12:25.592 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule activated for path /snap_test_dir1/dir9 2023-10-16T12:12:25.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.593+0000 7fae1a545700 1 -- 172.21.15.16:0/234671820 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fadf406b100 msgr2=0x7fadf406d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:25.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.593+0000 7fae1a545700 1 --2- 172.21.15.16:0/234671820 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fadf406b100 0x7fadf406d5c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fadfc000c00 tx=0x7fadfc009040 comp rx=0 tx=0).stop 2023-10-16T12:12:25.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.593+0000 7fae1a545700 1 -- 172.21.15.16:0/234671820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae00003bc0 msgr2=0x7fae140cf8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:12:25.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.593+0000 7fae1a545700 1 --2- 172.21.15.16:0/234671820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae00003bc0 0x7fae140cf8c0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fae0400c3c0 tx=0x7fae04001310 comp rx=0 tx=0).stop 2023-10-16T12:12:25.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.593+0000 7fae1a545700 1 -- 172.21.15.16:0/234671820 shutdown_connections 2023-10-16T12:12:25.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.593+0000 7fae1a545700 1 --2- 172.21.15.16:0/234671820 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fadf406b100 0x7fadf406d5c0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:25.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.593+0000 7fae1a545700 1 --2- 172.21.15.16:0/234671820 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fae14159160 0x7fae140c9a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:25.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.593+0000 7fae1a545700 1 --2- 172.21.15.16:0/234671820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae00003bc0 0x7fae140cf8c0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:25.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.593+0000 7fae1a545700 1 --2- 172.21.15.16:0/234671820 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fae140b5a50 0x7fae140c6000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:12:25.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.593+0000 7fae1a545700 1 -- 172.21.15.16:0/234671820 >> 172.21.15.16:0/234671820 conn(0x7fae140b1f40 msgr2=0x7fae141547e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:12:25.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.593+0000 7fae1a545700 1 -- 172.21.15.16:0/234671820 shutdown_connections 2023-10-16T12:12:25.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:12:25.593+0000 7fae1a545700 1 -- 172.21.15.16:0/234671820 wait complete. 2023-10-16T12:12:25.607 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir9 2023-10-16T12:12:48.688 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:12:48.692 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:13:00.103 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:13:00.100+0000 7ffa1bbf8700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:13:00.103 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:13:00.100+0000 7ffa1bbf8700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:13:00.103 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:13:00.100+0000 7ffa1bbf8700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:13:00.103 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:13:00.100+0000 7ffa1bbf8700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:13:00.104 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:13:00.100+0000 7ffa1bbf8700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:13:18.769 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:13:18.773 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:13:48.845 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:13:48.849 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:14:18.910 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:14:18.913 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:14:48.975 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:14:48.979 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:15:19.055 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:15:19.058 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:15:49.120 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:15:49.124 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:16:19.190 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:16:19.193 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:16:49.272 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:16:49.276 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:17:00.825 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:17:00.822+0000 7ffa14369700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:17:00.825 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:17:00.822+0000 7ffa14369700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:17:00.826 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:17:00.822+0000 7ffa14369700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:17:00.826 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:17:00.822+0000 7ffa14369700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:17:00.826 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:17:00.822+0000 7ffa14369700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:17:19.336 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:17:19.339 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:17:49.401 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:17:49.404 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:18:19.465 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:18:19.468 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:18:49.536 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:18:49.539 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:19:19.607 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:19:19.610 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:19:49.678 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:19:49.682 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:20:19.750 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:20:19.754 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:20:49.841 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:20:49.845 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:21:19.910 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:21:19.913 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:21:49.990 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:21:49.993 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:22:20.062 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:22:20.065 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:22:50.134 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:22:50.138 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:23:19.608 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir0 1m 2023-10-16T12:23:19.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:19.749+0000 7f35b402a700 1 Processor -- start 2023-10-16T12:23:19.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:19.749+0000 7f35b402a700 1 -- start start 2023-10-16T12:23:19.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:19.750+0000 7f35b402a700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f35ac15d8d0 0x7f35ac0b2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:19.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:19.750+0000 7f35b402a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f35ac0ce350 0x7f35ac0ce780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:19.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:19.750+0000 7f35b402a700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35ac0cecc0 con 0x7f35ac0b8fa0 2023-10-16T12:23:19.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:19.750+0000 7f35b402a700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35ac0d0bb0 con 0x7f35ac15d8d0 2023-10-16T12:23:19.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:19.750+0000 7f35b402a700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35ac15dcb0 con 0x7f35ac0ce350 2023-10-16T12:23:19.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:19.750+0000 7f35b1dc6700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f35ac15d8d0 0x7f35ac0b2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:19.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:19.750+0000 7f35b25c7700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f35ac0ce350 0x7f35ac0ce780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:19.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:19.750+0000 7f35b1dc6700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f35ac15d8d0 0x7f35ac0b2830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:35548/0 (socket says 172.21.15.16:35548) 2023-10-16T12:23:19.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:19.750+0000 7f35b1dc6700 1 -- 172.21.15.16:0/521758888 learned_addr learned my addr 172.21.15.16:0/521758888 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:19.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:19.750+0000 7f35b25c7700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f35ac0ce350 0x7f35ac0ce780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:52980/0 (socket says 172.21.15.16:52980) 2023-10-16T12:23:19.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:19.750+0000 7f35b15c5700 1 --1- 172.21.15.16:0/521758888 >> v1:172.21.15.49:6789/0 conn(0x7f35ac0b8fa0 0x7f35ac0b6720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:32942/0 (socket says 172.21.15.16:32942) 2023-10-16T12:23:19.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:19.751+0000 7f35b25c7700 1 -- 172.21.15.16:0/521758888 >> v2:172.21.15.49:3300/0 conn(0x7f35ac15d8d0 msgr2=0x7f35ac0b2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:19.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:19.751+0000 7f35b25c7700 1 --2- 172.21.15.16:0/521758888 >> v2:172.21.15.49:3300/0 conn(0x7f35ac15d8d0 0x7f35ac0b2830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:19.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:19.751+0000 7f35b25c7700 1 -- 172.21.15.16:0/521758888 >> v1:172.21.15.49:6789/0 conn(0x7f35ac0b8fa0 legacy=0x7f35ac0b6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:19.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:19.751+0000 7f35b25c7700 1 -- 172.21.15.16:0/521758888 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35ac0b2f30 con 0x7f35ac0ce350 2023-10-16T12:23:19.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:19.751+0000 7f35b0dc4700 1 -- 172.21.15.16:0/521758888 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3478324732 0 0) 0x7f35ac0cecc0 con 0x7f35ac0b8fa0 2023-10-16T12:23:19.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:19.751+0000 7f35b1dc6700 1 --2- 172.21.15.16:0/521758888 >> v2:172.21.15.49:3300/0 conn(0x7f35ac15d8d0 0x7f35ac0b2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:23:19.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:19.751+0000 7f35b25c7700 1 --2- 172.21.15.16:0/521758888 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f35ac0ce350 0x7f35ac0ce780 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f35a4009d90 tx=0x7f35a400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=739aa8858ef2c5ea server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:19.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:19.751+0000 7f35b0dc4700 1 -- 172.21.15.16:0/521758888 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35a400e070 con 0x7f35ac0ce350 2023-10-16T12:23:19.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:19.751+0000 7f35b0dc4700 1 -- 172.21.15.16:0/521758888 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35a400bb20 con 0x7f35ac0ce350 2023-10-16T12:23:19.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:19.751+0000 7f35b0dc4700 1 -- 172.21.15.16:0/521758888 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35a4012430 con 0x7f35ac0ce350 2023-10-16T12:23:19.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:19.752+0000 7f35b402a700 1 -- 172.21.15.16:0/521758888 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f35ac0ce350 msgr2=0x7f35ac0ce780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:19.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:19.752+0000 7f35b402a700 1 --2- 172.21.15.16:0/521758888 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f35ac0ce350 0x7f35ac0ce780 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f35a4009d90 tx=0x7f35a400b330 comp rx=0 tx=0).stop 2023-10-16T12:23:19.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:19.752+0000 7f35b402a700 1 -- 172.21.15.16:0/521758888 shutdown_connections 2023-10-16T12:23:19.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:19.752+0000 7f35b402a700 1 --2- 172.21.15.16:0/521758888 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f35ac0ce350 0x7f35ac0ce780 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:19.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:19.752+0000 7f35b402a700 1 --2- 172.21.15.16:0/521758888 >> v2:172.21.15.49:3300/0 conn(0x7f35ac15d8d0 0x7f35ac0b2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:19.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:19.752+0000 7f35b402a700 1 -- 172.21.15.16:0/521758888 >> 172.21.15.16:0/521758888 conn(0x7f35ac072330 msgr2=0x7f35ac072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:19.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:19.752+0000 7f35b402a700 1 -- 172.21.15.16:0/521758888 shutdown_connections 2023-10-16T12:23:19.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:19.752+0000 7f35b402a700 1 -- 172.21.15.16:0/521758888 wait complete. 2023-10-16T12:23:19.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:19.753+0000 7f35b402a700 1 Processor -- start 2023-10-16T12:23:19.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:19.753+0000 7f35b402a700 1 -- start start 2023-10-16T12:23:19.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:19.754+0000 7f35b402a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35ac0b8fa0 0x7f35ac0ccf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:19.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:19.754+0000 7f35b402a700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f35ac0ce350 0x7f35ac0c97c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:19.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:19.754+0000 7f35b402a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f35ac15d8d0 0x7f35ac15ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:19.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:19.754+0000 7f35b402a700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35ac0c8790 con 0x7f35ac0b8fa0 2023-10-16T12:23:19.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:19.754+0000 7f35b402a700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35ac0c8a00 con 0x7f35ac0ce350 2023-10-16T12:23:19.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:19.754+0000 7f35b402a700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35ac0c8ce0 con 0x7f35ac15d8d0 2023-10-16T12:23:19.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:19.755+0000 7f35b1dc6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35ac0b8fa0 0x7f35ac0ccf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:19.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:19.755+0000 7f35b1dc6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35ac0b8fa0 0x7f35ac0ccf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36492/0 (socket says 172.21.15.16:36492) 2023-10-16T12:23:19.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:19.755+0000 7f35b1dc6700 1 -- 172.21.15.16:0/2403697857 learned_addr learned my addr 172.21.15.16:0/2403697857 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:19.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:19.755+0000 7f35b15c5700 1 --2- 172.21.15.16:0/2403697857 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f35ac0ce350 0x7f35ac0c97c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:19.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:19.755+0000 7f35b25c7700 1 --2- 172.21.15.16:0/2403697857 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f35ac15d8d0 0x7f35ac15ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:19.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:19.755+0000 7f35b1dc6700 1 -- 172.21.15.16:0/2403697857 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f35ac15d8d0 msgr2=0x7f35ac15ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:19.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:19.755+0000 7f35b1dc6700 1 --2- 172.21.15.16:0/2403697857 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f35ac15d8d0 0x7f35ac15ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:19.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:19.755+0000 7f35b1dc6700 1 -- 172.21.15.16:0/2403697857 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f35ac0ce350 msgr2=0x7f35ac0c97c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:19.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:19.755+0000 7f35b1dc6700 1 --2- 172.21.15.16:0/2403697857 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f35ac0ce350 0x7f35ac0c97c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:19.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:19.755+0000 7f35b1dc6700 1 -- 172.21.15.16:0/2403697857 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35a8009550 con 0x7f35ac0b8fa0 2023-10-16T12:23:19.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:19.755+0000 7f35b15c5700 1 --2- 172.21.15.16:0/2403697857 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f35ac0ce350 0x7f35ac0c97c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:23:19.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:19.755+0000 7f35b1dc6700 1 --2- 172.21.15.16:0/2403697857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35ac0b8fa0 0x7f35ac0ccf10 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f35a800e9b0 tx=0x7f35a800ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:19.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:19.755+0000 7f35a27fc700 1 -- 172.21.15.16:0/2403697857 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35a800c850 con 0x7f35ac0b8fa0 2023-10-16T12:23:19.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:19.755+0000 7f35a27fc700 1 -- 172.21.15.16:0/2403697857 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35a800c9b0 con 0x7f35ac0b8fa0 2023-10-16T12:23:19.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:19.755+0000 7f35b402a700 1 -- 172.21.15.16:0/2403697857 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35a4009a70 con 0x7f35ac0b8fa0 2023-10-16T12:23:19.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:19.755+0000 7f35a27fc700 1 -- 172.21.15.16:0/2403697857 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35a800cd20 con 0x7f35ac0b8fa0 2023-10-16T12:23:19.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:19.755+0000 7f35b402a700 1 -- 172.21.15.16:0/2403697857 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35ac0c92a0 con 0x7f35ac0b8fa0 2023-10-16T12:23:19.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:19.756+0000 7f35b402a700 1 -- 172.21.15.16:0/2403697857 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35ac0c9610 con 0x7f35ac0b8fa0 2023-10-16T12:23:19.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:19.757+0000 7f35a27fc700 1 -- 172.21.15.16:0/2403697857 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f35a8013470 con 0x7f35ac0b8fa0 2023-10-16T12:23:19.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:19.757+0000 7f35a27fc700 1 --2- 172.21.15.16:0/2403697857 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f359006b100 0x7f359006d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:19.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:19.757+0000 7f35a27fc700 1 -- 172.21.15.16:0/2403697857 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f35a8089d40 con 0x7f35ac0b8fa0 2023-10-16T12:23:19.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:19.757+0000 7f35b15c5700 1 --2- 172.21.15.16:0/2403697857 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f359006b100 0x7f359006d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:19.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:19.758+0000 7f35b15c5700 1 --2- 172.21.15.16:0/2403697857 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f359006b100 0x7f359006d5c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f35ac0ca820 tx=0x7f359c007d30 comp rx=0 tx=0).ready entity=mgr.5499 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:19.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:19.762+0000 7f35a27fc700 1 -- 172.21.15.16:0/2403697857 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f35a8054910 con 0x7f35ac0b8fa0 2023-10-16T12:23:19.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:19.996+0000 7f35b402a700 1 -- 172.21.15.16:0/2403697857 --> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir0", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f35ac0d1ee0 con 0x7f359006b100 2023-10-16T12:23:20.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.066+0000 7f35a27fc700 1 -- 172.21.15.16:0/2403697857 <== mgr.5499 v2:172.21.15.16:6832/1707126831 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f359c004870 con 0x7f359006b100 2023-10-16T12:23:20.069 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule deactivated for path /snap_test_dir1/dir0 2023-10-16T12:23:20.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.068+0000 7f35b402a700 1 -- 172.21.15.16:0/2403697857 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f359006b100 msgr2=0x7f359006d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:20.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.068+0000 7f35b402a700 1 --2- 172.21.15.16:0/2403697857 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f359006b100 0x7f359006d5c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f35ac0ca820 tx=0x7f359c007d30 comp rx=0 tx=0).stop 2023-10-16T12:23:20.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.068+0000 7f35b402a700 1 -- 172.21.15.16:0/2403697857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35ac0b8fa0 msgr2=0x7f35ac0ccf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:20.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.068+0000 7f35b402a700 1 --2- 172.21.15.16:0/2403697857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35ac0b8fa0 0x7f35ac0ccf10 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f35a800e9b0 tx=0x7f35a800ed70 comp rx=0 tx=0).stop 2023-10-16T12:23:20.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.068+0000 7f35b402a700 1 -- 172.21.15.16:0/2403697857 shutdown_connections 2023-10-16T12:23:20.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.068+0000 7f35b402a700 1 --2- 172.21.15.16:0/2403697857 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f359006b100 0x7f359006d5c0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:20.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.068+0000 7f35b402a700 1 --2- 172.21.15.16:0/2403697857 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f35ac15d8d0 0x7f35ac15ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:20.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.068+0000 7f35b402a700 1 --2- 172.21.15.16:0/2403697857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35ac0b8fa0 0x7f35ac0ccf10 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:20.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.068+0000 7f35b402a700 1 --2- 172.21.15.16:0/2403697857 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f35ac0ce350 0x7f35ac0c97c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:20.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.068+0000 7f35b402a700 1 -- 172.21.15.16:0/2403697857 >> 172.21.15.16:0/2403697857 conn(0x7f35ac072330 msgr2=0x7f35ac15dec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:20.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.068+0000 7f35b402a700 1 -- 172.21.15.16:0/2403697857 shutdown_connections 2023-10-16T12:23:20.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.068+0000 7f35b402a700 1 -- 172.21.15.16:0/2403697857 wait complete. 2023-10-16T12:23:20.083 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir0 2023-10-16T12:23:20.083 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir1 1m 2023-10-16T12:23:20.195 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:23:20.198 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:23:20.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.210+0000 7f97e3171700 1 Processor -- start 2023-10-16T12:23:20.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.210+0000 7f97e3171700 1 -- start start 2023-10-16T12:23:20.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.210+0000 7f97e3171700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f97dc15b2c0 0x7f97dc161340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:20.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.210+0000 7f97e3171700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97dc15ecb0 con 0x7f97dc06e570 2023-10-16T12:23:20.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.210+0000 7f97e3171700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97dc161880 con 0x7f97dc06a7e0 2023-10-16T12:23:20.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.210+0000 7f97e3171700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97dc15b6a0 con 0x7f97dc15b2c0 2023-10-16T12:23:20.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.211+0000 7f97dbfff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f97dc06e570 0x7f97dc15e5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54008/0 (socket says 172.21.15.16:54008) 2023-10-16T12:23:20.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.211+0000 7f97dbfff700 1 -- 172.21.15.16:0/1070438101 learned_addr learned my addr 172.21.15.16:0/1070438101 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:23:20.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.211+0000 7f97e0f0d700 1 --2- 172.21.15.16:0/1070438101 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f97dc15b2c0 0x7f97dc161340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:20.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.211+0000 7f97db7fe700 1 -- 172.21.15.16:0/1070438101 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 82469289 0 0) 0x7f97dc15ecb0 con 0x7f97dc06e570 2023-10-16T12:23:20.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.211+0000 7f97db7fe700 1 -- 172.21.15.16:0/1070438101 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97cc0035b0 con 0x7f97dc06e570 2023-10-16T12:23:20.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.211+0000 7f97db7fe700 1 -- 172.21.15.16:0/1070438101 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 141947446 0 0) 0x7f97dc161880 con 0x7f97dc06a7e0 2023-10-16T12:23:20.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.211+0000 7f97e0f0d700 1 -- 172.21.15.16:0/1070438101 >> v1:172.21.15.49:6789/0 conn(0x7f97dc06a7e0 legacy=0x7f97dc071a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:20.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.211+0000 7f97e0f0d700 1 -- 172.21.15.16:0/1070438101 >> v1:172.21.15.16:6789/0 conn(0x7f97dc06e570 legacy=0x7f97dc15e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:20.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.211+0000 7f97e0f0d700 1 -- 172.21.15.16:0/1070438101 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97dc161b20 con 0x7f97dc15b2c0 2023-10-16T12:23:20.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.211+0000 7f97e0f0d700 1 --2- 172.21.15.16:0/1070438101 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f97dc15b2c0 0x7f97dc161340 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f97d0009870 tx=0x7f97d0009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=29ade77709b77b73 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:20.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.212+0000 7f97db7fe700 1 -- 172.21.15.16:0/1070438101 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97c8001670 con 0x7f97dc15b2c0 2023-10-16T12:23:20.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.212+0000 7f97db7fe700 1 -- 172.21.15.16:0/1070438101 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97d0004030 con 0x7f97dc15b2c0 2023-10-16T12:23:20.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.212+0000 7f97db7fe700 1 -- 172.21.15.16:0/1070438101 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d00043a0 con 0x7f97dc15b2c0 2023-10-16T12:23:20.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.212+0000 7f97e3171700 1 -- 172.21.15.16:0/1070438101 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f97dc15b2c0 msgr2=0x7f97dc161340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:20.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.212+0000 7f97e3171700 1 --2- 172.21.15.16:0/1070438101 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f97dc15b2c0 0x7f97dc161340 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f97d0009870 tx=0x7f97d0009b80 comp rx=0 tx=0).stop 2023-10-16T12:23:20.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.212+0000 7f97e3171700 1 -- 172.21.15.16:0/1070438101 shutdown_connections 2023-10-16T12:23:20.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.212+0000 7f97e3171700 1 --2- 172.21.15.16:0/1070438101 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f97dc15b2c0 0x7f97dc161340 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:20.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.212+0000 7f97e3171700 1 -- 172.21.15.16:0/1070438101 >> 172.21.15.16:0/1070438101 conn(0x7f97dc0b1f40 msgr2=0x7f97dc0b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:20.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.212+0000 7f97e3171700 1 -- 172.21.15.16:0/1070438101 shutdown_connections 2023-10-16T12:23:20.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.212+0000 7f97e3171700 1 -- 172.21.15.16:0/1070438101 wait complete. 2023-10-16T12:23:20.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.214+0000 7f97e3171700 1 Processor -- start 2023-10-16T12:23:20.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.214+0000 7f97e3171700 1 -- start start 2023-10-16T12:23:20.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.215+0000 7f97e3171700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f97dc06a7e0 0x7f97dc0c6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:20.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.215+0000 7f97e3171700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97dc06e570 0x7f97dc1611b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:20.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.215+0000 7f97e3171700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f97dc15b2c0 0x7f97dc148e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:20.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.215+0000 7f97e3171700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97dc160d20 con 0x7f97dc06e570 2023-10-16T12:23:20.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.215+0000 7f97e3171700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97dc15fe90 con 0x7f97dc15b2c0 2023-10-16T12:23:20.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.215+0000 7f97e3171700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97dc160110 con 0x7f97dc06a7e0 2023-10-16T12:23:20.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.215+0000 7f97dbfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97dc06e570 0x7f97dc1611b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:20.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.215+0000 7f97dbfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97dc06e570 0x7f97dc1611b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36508/0 (socket says 172.21.15.16:36508) 2023-10-16T12:23:20.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.215+0000 7f97e0f0d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f97dc06a7e0 0x7f97dc0c6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:20.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.215+0000 7f97dbfff700 1 -- 172.21.15.16:0/1279607607 learned_addr learned my addr 172.21.15.16:0/1279607607 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:20.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.215+0000 7f97e170e700 1 --2- 172.21.15.16:0/1279607607 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f97dc15b2c0 0x7f97dc148e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:20.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.215+0000 7f97dbfff700 1 -- 172.21.15.16:0/1279607607 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f97dc06a7e0 msgr2=0x7f97dc0c6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:20.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.215+0000 7f97dbfff700 1 --2- 172.21.15.16:0/1279607607 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f97dc06a7e0 0x7f97dc0c6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:20.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.215+0000 7f97dbfff700 1 -- 172.21.15.16:0/1279607607 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f97dc15b2c0 msgr2=0x7f97dc148e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:20.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.215+0000 7f97dbfff700 1 --2- 172.21.15.16:0/1279607607 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f97dc15b2c0 0x7f97dc148e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:20.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.215+0000 7f97dbfff700 1 -- 172.21.15.16:0/1279607607 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97d00094a0 con 0x7f97dc06e570 2023-10-16T12:23:20.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.215+0000 7f97dbfff700 1 --2- 172.21.15.16:0/1279607607 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97dc06e570 0x7f97dc1611b0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f97c800c780 tx=0x7f97c800ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:20.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.216+0000 7f97d97fa700 1 -- 172.21.15.16:0/1279607607 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97c8004c80 con 0x7f97dc06e570 2023-10-16T12:23:20.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.216+0000 7f97d97fa700 1 -- 172.21.15.16:0/1279607607 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97c8004de0 con 0x7f97dc06e570 2023-10-16T12:23:20.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.216+0000 7f97d97fa700 1 -- 172.21.15.16:0/1279607607 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97c8007310 con 0x7f97dc06e570 2023-10-16T12:23:20.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.216+0000 7f97e3171700 1 -- 172.21.15.16:0/1279607607 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97dc1603f0 con 0x7f97dc06e570 2023-10-16T12:23:20.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.216+0000 7f97e3171700 1 -- 172.21.15.16:0/1279607607 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97dc1609b0 con 0x7f97dc06e570 2023-10-16T12:23:20.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.217+0000 7f97e3171700 1 -- 172.21.15.16:0/1279607607 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97dc04ebb0 con 0x7f97dc06e570 2023-10-16T12:23:20.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.218+0000 7f97d97fa700 1 -- 172.21.15.16:0/1279607607 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f97c8007470 con 0x7f97dc06e570 2023-10-16T12:23:20.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.218+0000 7f97d97fa700 1 --2- 172.21.15.16:0/1279607607 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f97c006b100 0x7f97c006d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:20.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.218+0000 7f97d97fa700 1 -- 172.21.15.16:0/1279607607 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f97c80899b0 con 0x7f97dc06e570 2023-10-16T12:23:20.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.218+0000 7f97e0f0d700 1 --2- 172.21.15.16:0/1279607607 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f97c006b100 0x7f97c006d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:20.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.219+0000 7f97e0f0d700 1 --2- 172.21.15.16:0/1279607607 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f97c006b100 0x7f97c006d5c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f97d0000c00 tx=0x7f97d0008000 comp rx=0 tx=0).ready entity=mgr.5499 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:20.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.223+0000 7f97d97fa700 1 -- 172.21.15.16:0/1279607607 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f97c80544e0 con 0x7f97dc06e570 2023-10-16T12:23:20.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.460+0000 7f97e3171700 1 -- 172.21.15.16:0/1279607607 --> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir1", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f97dc1620e0 con 0x7f97c006b100 2023-10-16T12:23:20.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.526+0000 7f97d97fa700 1 -- 172.21.15.16:0/1279607607 <== mgr.5499 v2:172.21.15.16:6832/1707126831 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f97d0006c60 con 0x7f97c006b100 2023-10-16T12:23:20.530 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule deactivated for path /snap_test_dir1/dir1 2023-10-16T12:23:20.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.529+0000 7f97e3171700 1 -- 172.21.15.16:0/1279607607 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f97c006b100 msgr2=0x7f97c006d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:20.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.529+0000 7f97e3171700 1 --2- 172.21.15.16:0/1279607607 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f97c006b100 0x7f97c006d5c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f97d0000c00 tx=0x7f97d0008000 comp rx=0 tx=0).stop 2023-10-16T12:23:20.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.529+0000 7f97e3171700 1 -- 172.21.15.16:0/1279607607 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97dc06e570 msgr2=0x7f97dc1611b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:20.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.529+0000 7f97e3171700 1 --2- 172.21.15.16:0/1279607607 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97dc06e570 0x7f97dc1611b0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f97c800c780 tx=0x7f97c800ca90 comp rx=0 tx=0).stop 2023-10-16T12:23:20.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.529+0000 7f97e3171700 1 -- 172.21.15.16:0/1279607607 shutdown_connections 2023-10-16T12:23:20.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.529+0000 7f97e3171700 1 --2- 172.21.15.16:0/1279607607 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f97c006b100 0x7f97c006d5c0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:20.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.529+0000 7f97e3171700 1 --2- 172.21.15.16:0/1279607607 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f97dc15b2c0 0x7f97dc148e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:20.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.529+0000 7f97e3171700 1 --2- 172.21.15.16:0/1279607607 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f97dc06a7e0 0x7f97dc0c6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:20.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.529+0000 7f97e3171700 1 --2- 172.21.15.16:0/1279607607 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97dc06e570 0x7f97dc1611b0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:20.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.529+0000 7f97e3171700 1 -- 172.21.15.16:0/1279607607 >> 172.21.15.16:0/1279607607 conn(0x7f97dc0b1f40 msgr2=0x7f97dc15b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:20.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.529+0000 7f97e3171700 1 -- 172.21.15.16:0/1279607607 shutdown_connections 2023-10-16T12:23:20.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.529+0000 7f97e3171700 1 -- 172.21.15.16:0/1279607607 wait complete. 2023-10-16T12:23:20.544 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir1 2023-10-16T12:23:20.544 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir2 1m 2023-10-16T12:23:20.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.670+0000 7f8133215700 1 Processor -- start 2023-10-16T12:23:20.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.670+0000 7f8133215700 1 -- start start 2023-10-16T12:23:20.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.670+0000 7f8133215700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f812c15b2c0 0x7f812c161340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:20.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.670+0000 7f8133215700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f812c06e570 0x7f812c15e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:20.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.670+0000 7f8133215700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f812c06a7e0 0x7f812c06ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:20.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.670+0000 7f8133215700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f812c06b1b0 con 0x7f812c15b2c0 2023-10-16T12:23:20.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.670+0000 7f8133215700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f812c161880 con 0x7f812c06a7e0 2023-10-16T12:23:20.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.670+0000 7f8133215700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f812c15b6a0 con 0x7f812c06e570 2023-10-16T12:23:20.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.671+0000 7f8130fb1700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f812c15b2c0 0x7f812c161340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:20.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.671+0000 7f8130fb1700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f812c15b2c0 0x7f812c161340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36514/0 (socket says 172.21.15.16:36514) 2023-10-16T12:23:20.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.671+0000 7f8130fb1700 1 -- 172.21.15.16:0/2710011112 learned_addr learned my addr 172.21.15.16:0/2710011112 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:20.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.671+0000 7f812bfff700 1 --2- 172.21.15.16:0/2710011112 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f812c06e570 0x7f812c15e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:20.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.671+0000 7f81317b2700 1 --2- 172.21.15.16:0/2710011112 >> v2:172.21.15.49:3300/0 conn(0x7f812c06a7e0 0x7f812c06ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:20.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.671+0000 7f8130fb1700 1 -- 172.21.15.16:0/2710011112 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f812c06e570 msgr2=0x7f812c15e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:20.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.671+0000 7f8130fb1700 1 --2- 172.21.15.16:0/2710011112 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f812c06e570 0x7f812c15e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:20.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.671+0000 7f8130fb1700 1 -- 172.21.15.16:0/2710011112 >> v2:172.21.15.49:3300/0 conn(0x7f812c06a7e0 msgr2=0x7f812c06ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:20.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.671+0000 7f8130fb1700 1 --2- 172.21.15.16:0/2710011112 >> v2:172.21.15.49:3300/0 conn(0x7f812c06a7e0 0x7f812c06ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:20.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.671+0000 7f8130fb1700 1 -- 172.21.15.16:0/2710011112 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f812c161b20 con 0x7f812c15b2c0 2023-10-16T12:23:20.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.671+0000 7f812bfff700 1 --2- 172.21.15.16:0/2710011112 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f812c06e570 0x7f812c15e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:23:20.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.671+0000 7f8130fb1700 1 --2- 172.21.15.16:0/2710011112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f812c15b2c0 0x7f812c161340 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f812c0ca970 tx=0x7f8120009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d79487895264475 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:20.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.671+0000 7f812b7fe700 1 -- 172.21.15.16:0/2710011112 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f812000b870 con 0x7f812c15b2c0 2023-10-16T12:23:20.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.671+0000 7f812b7fe700 1 -- 172.21.15.16:0/2710011112 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f812000b9d0 con 0x7f812c15b2c0 2023-10-16T12:23:20.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.671+0000 7f812b7fe700 1 -- 172.21.15.16:0/2710011112 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f812000bd60 con 0x7f812c15b2c0 2023-10-16T12:23:20.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.672+0000 7f8133215700 1 -- 172.21.15.16:0/2710011112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f812c15b2c0 msgr2=0x7f812c161340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:20.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.672+0000 7f8133215700 1 --2- 172.21.15.16:0/2710011112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f812c15b2c0 0x7f812c161340 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f812c0ca970 tx=0x7f8120009bd0 comp rx=0 tx=0).stop 2023-10-16T12:23:20.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.672+0000 7f8133215700 1 -- 172.21.15.16:0/2710011112 shutdown_connections 2023-10-16T12:23:20.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.672+0000 7f8133215700 1 --2- 172.21.15.16:0/2710011112 >> v2:172.21.15.49:3300/0 conn(0x7f812c06a7e0 0x7f812c06ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:20.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.672+0000 7f8133215700 1 --2- 172.21.15.16:0/2710011112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f812c15b2c0 0x7f812c161340 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:20.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.672+0000 7f8133215700 1 --2- 172.21.15.16:0/2710011112 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f812c06e570 0x7f812c15e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:20.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.672+0000 7f8133215700 1 -- 172.21.15.16:0/2710011112 >> 172.21.15.16:0/2710011112 conn(0x7f812c0b1f40 msgr2=0x7f812c0b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:20.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.672+0000 7f8133215700 1 -- 172.21.15.16:0/2710011112 shutdown_connections 2023-10-16T12:23:20.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.672+0000 7f8133215700 1 -- 172.21.15.16:0/2710011112 wait complete. 2023-10-16T12:23:20.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.673+0000 7f8133215700 1 Processor -- start 2023-10-16T12:23:20.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.673+0000 7f8133215700 1 -- start start 2023-10-16T12:23:20.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.674+0000 7f8133215700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f812c06a7e0 0x7f812c1f3e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:20.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.674+0000 7f8133215700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f812c06e570 0x7f812c0c9bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:20.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.674+0000 7f8133215700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f812c15b2c0 0x7f812c155220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:20.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.674+0000 7f8133215700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f812c0ca130 con 0x7f812c06a7e0 2023-10-16T12:23:20.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.674+0000 7f8133215700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f812c1f68d0 con 0x7f812c15b2c0 2023-10-16T12:23:20.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.674+0000 7f8133215700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f812c155760 con 0x7f812c06e570 2023-10-16T12:23:20.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.674+0000 7f8130fb1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f812c06a7e0 0x7f812c1f3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:20.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.674+0000 7f8130fb1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f812c06a7e0 0x7f812c1f3e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36520/0 (socket says 172.21.15.16:36520) 2023-10-16T12:23:20.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.674+0000 7f8130fb1700 1 -- 172.21.15.16:0/3986601037 learned_addr learned my addr 172.21.15.16:0/3986601037 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:20.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.674+0000 7f812bfff700 1 --2- 172.21.15.16:0/3986601037 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f812c06e570 0x7f812c0c9bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:20.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.674+0000 7f8130fb1700 1 -- 172.21.15.16:0/3986601037 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f812c06e570 msgr2=0x7f812c0c9bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:20.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.674+0000 7f8130fb1700 1 --2- 172.21.15.16:0/3986601037 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f812c06e570 0x7f812c0c9bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:20.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.674+0000 7f8130fb1700 1 -- 172.21.15.16:0/3986601037 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f812c15b2c0 msgr2=0x7f812c155220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:20.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.674+0000 7f81317b2700 1 --2- 172.21.15.16:0/3986601037 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f812c15b2c0 0x7f812c155220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:20.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.674+0000 7f8130fb1700 1 --2- 172.21.15.16:0/3986601037 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f812c15b2c0 0x7f812c155220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:20.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.674+0000 7f8130fb1700 1 -- 172.21.15.16:0/3986601037 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8120009550 con 0x7f812c06a7e0 2023-10-16T12:23:20.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.674+0000 7f812bfff700 1 --2- 172.21.15.16:0/3986601037 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f812c06e570 0x7f812c0c9bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:23:20.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.675+0000 7f8130fb1700 1 --2- 172.21.15.16:0/3986601037 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f812c06a7e0 0x7f812c1f3e60 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f812c070620 tx=0x7f8120004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:20.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.675+0000 7f81297fa700 1 -- 172.21.15.16:0/3986601037 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8120004540 con 0x7f812c06a7e0 2023-10-16T12:23:20.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.675+0000 7f81297fa700 1 -- 172.21.15.16:0/3986601037 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81200036a0 con 0x7f812c06a7e0 2023-10-16T12:23:20.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.675+0000 7f8133215700 1 -- 172.21.15.16:0/3986601037 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f812c15f640 con 0x7f812c06a7e0 2023-10-16T12:23:20.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.675+0000 7f81297fa700 1 -- 172.21.15.16:0/3986601037 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8120003a50 con 0x7f812c06a7e0 2023-10-16T12:23:20.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.675+0000 7f8133215700 1 -- 172.21.15.16:0/3986601037 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f812c1f6e00 con 0x7f812c06a7e0 2023-10-16T12:23:20.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.676+0000 7f8133215700 1 -- 172.21.15.16:0/3986601037 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f812c04ebb0 con 0x7f812c06a7e0 2023-10-16T12:23:20.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.676+0000 7f81297fa700 1 -- 172.21.15.16:0/3986601037 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f8120021070 con 0x7f812c06a7e0 2023-10-16T12:23:20.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.677+0000 7f81297fa700 1 --2- 172.21.15.16:0/3986601037 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f811006b100 0x7f811006d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:20.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.677+0000 7f81297fa700 1 -- 172.21.15.16:0/3986601037 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f812008b300 con 0x7f812c06a7e0 2023-10-16T12:23:20.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.677+0000 7f812bfff700 1 --2- 172.21.15.16:0/3986601037 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f811006b100 0x7f811006d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:20.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.677+0000 7f812bfff700 1 --2- 172.21.15.16:0/3986601037 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f811006b100 0x7f811006d5c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f812c0b19e0 tx=0x7f8118008040 comp rx=0 tx=0).ready entity=mgr.5499 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:20.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.681+0000 7f81297fa700 1 -- 172.21.15.16:0/3986601037 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8120055ee0 con 0x7f812c06a7e0 2023-10-16T12:23:20.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.919+0000 7f8133215700 1 -- 172.21.15.16:0/3986601037 --> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir2", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f812c1620e0 con 0x7f811006b100 2023-10-16T12:23:20.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.986+0000 7f81297fa700 1 -- 172.21.15.16:0/3986601037 <== mgr.5499 v2:172.21.15.16:6832/1707126831 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f8118009510 con 0x7f811006b100 2023-10-16T12:23:20.989 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule deactivated for path /snap_test_dir1/dir2 2023-10-16T12:23:20.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.988+0000 7f8133215700 1 -- 172.21.15.16:0/3986601037 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f811006b100 msgr2=0x7f811006d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:20.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.988+0000 7f8133215700 1 --2- 172.21.15.16:0/3986601037 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f811006b100 0x7f811006d5c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f812c0b19e0 tx=0x7f8118008040 comp rx=0 tx=0).stop 2023-10-16T12:23:20.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.988+0000 7f8133215700 1 -- 172.21.15.16:0/3986601037 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f812c06a7e0 msgr2=0x7f812c1f3e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:20.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.988+0000 7f8133215700 1 --2- 172.21.15.16:0/3986601037 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f812c06a7e0 0x7f812c1f3e60 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f812c070620 tx=0x7f8120004750 comp rx=0 tx=0).stop 2023-10-16T12:23:20.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.989+0000 7f8133215700 1 -- 172.21.15.16:0/3986601037 shutdown_connections 2023-10-16T12:23:20.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.989+0000 7f8133215700 1 --2- 172.21.15.16:0/3986601037 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f811006b100 0x7f811006d5c0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:20.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.989+0000 7f8133215700 1 --2- 172.21.15.16:0/3986601037 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f812c15b2c0 0x7f812c155220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:20.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.989+0000 7f8133215700 1 --2- 172.21.15.16:0/3986601037 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f812c06a7e0 0x7f812c1f3e60 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:20.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.989+0000 7f8133215700 1 --2- 172.21.15.16:0/3986601037 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f812c06e570 0x7f812c0c9bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:20.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.989+0000 7f8133215700 1 -- 172.21.15.16:0/3986601037 >> 172.21.15.16:0/3986601037 conn(0x7f812c0b1f40 msgr2=0x7f812c0cc3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:20.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.989+0000 7f8133215700 1 -- 172.21.15.16:0/3986601037 shutdown_connections 2023-10-16T12:23:20.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:20.989+0000 7f8133215700 1 -- 172.21.15.16:0/3986601037 wait complete. 2023-10-16T12:23:21.003 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir2 2023-10-16T12:23:21.004 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir3 1m 2023-10-16T12:23:21.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.130+0000 7fd0beefd700 1 Processor -- start 2023-10-16T12:23:21.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.130+0000 7fd0beefd700 1 -- start start 2023-10-16T12:23:21.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.130+0000 7fd0beefd700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fd0b815ec80 0x7fd0b80cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:21.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.130+0000 7fd0beefd700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd0b815f060 0x7fd0b80ce400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:21.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.130+0000 7fd0beefd700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0b80b2440 con 0x7fd0b80d2110 2023-10-16T12:23:21.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.130+0000 7fd0beefd700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0b80b2c50 con 0x7fd0b815ec80 2023-10-16T12:23:21.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.130+0000 7fd0beefd700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b80d00e0 con 0x7fd0b815f060 2023-10-16T12:23:21.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.131+0000 7fd0b7fff700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd0b815f060 0x7fd0b80ce400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:21.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.131+0000 7fd0bcc99700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fd0b815ec80 0x7fd0b80cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:21.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.131+0000 7fd0bcc99700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fd0b815ec80 0x7fd0b80cdec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:35590/0 (socket says 172.21.15.16:35590) 2023-10-16T12:23:21.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.131+0000 7fd0b7fff700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd0b815f060 0x7fd0b80ce400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:53022/0 (socket says 172.21.15.16:53022) 2023-10-16T12:23:21.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.131+0000 7fd0bd49a700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7fd0b80d2110 0x7fd0b80b4440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:32986/0 (socket says 172.21.15.16:32986) 2023-10-16T12:23:21.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.131+0000 7fd0bcc99700 1 -- 172.21.15.16:0/3643655079 learned_addr learned my addr 172.21.15.16:0/3643655079 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:21.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.131+0000 7fd0bcc99700 1 -- 172.21.15.16:0/3643655079 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd0b815f060 msgr2=0x7fd0b80ce400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:21.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.131+0000 7fd0bcc99700 1 --2- 172.21.15.16:0/3643655079 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd0b815f060 0x7fd0b80ce400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:21.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.131+0000 7fd0bcc99700 1 -- 172.21.15.16:0/3643655079 >> v1:172.21.15.49:6789/0 conn(0x7fd0b80d2110 legacy=0x7fd0b80b4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:21.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.131+0000 7fd0bcc99700 1 -- 172.21.15.16:0/3643655079 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0b80cef60 con 0x7fd0b815ec80 2023-10-16T12:23:21.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.131+0000 7fd0b77fe700 1 -- 172.21.15.16:0/3643655079 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 843024362 0 0) 0x7fd0b80b2440 con 0x7fd0b80d2110 2023-10-16T12:23:21.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.131+0000 7fd0b7fff700 1 --2- 172.21.15.16:0/3643655079 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd0b815f060 0x7fd0b80ce400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:23:21.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.131+0000 7fd0bcc99700 1 --2- 172.21.15.16:0/3643655079 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd0b815ec80 0x7fd0b80cdec0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd0ac0098f0 tx=0x7fd0ac009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1c93ed53f534f53f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:21.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.131+0000 7fd0b77fe700 1 -- 172.21.15.16:0/3643655079 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0ac011830 con 0x7fd0b815ec80 2023-10-16T12:23:21.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.132+0000 7fd0b77fe700 1 -- 172.21.15.16:0/3643655079 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0ac011990 con 0x7fd0b815ec80 2023-10-16T12:23:21.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.132+0000 7fd0b77fe700 1 -- 172.21.15.16:0/3643655079 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0ac011d20 con 0x7fd0b815ec80 2023-10-16T12:23:21.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.132+0000 7fd0beefd700 1 -- 172.21.15.16:0/3643655079 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd0b815ec80 msgr2=0x7fd0b80cdec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:21.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.132+0000 7fd0beefd700 1 --2- 172.21.15.16:0/3643655079 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd0b815ec80 0x7fd0b80cdec0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd0ac0098f0 tx=0x7fd0ac009c00 comp rx=0 tx=0).stop 2023-10-16T12:23:21.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.132+0000 7fd0beefd700 1 -- 172.21.15.16:0/3643655079 shutdown_connections 2023-10-16T12:23:21.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.132+0000 7fd0beefd700 1 --2- 172.21.15.16:0/3643655079 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd0b815ec80 0x7fd0b80cdec0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:21.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.132+0000 7fd0beefd700 1 --2- 172.21.15.16:0/3643655079 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd0b815f060 0x7fd0b80ce400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:21.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.132+0000 7fd0beefd700 1 -- 172.21.15.16:0/3643655079 >> 172.21.15.16:0/3643655079 conn(0x7fd0b80d15f0 msgr2=0x7fd0b80d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:21.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.132+0000 7fd0beefd700 1 -- 172.21.15.16:0/3643655079 shutdown_connections 2023-10-16T12:23:21.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.132+0000 7fd0beefd700 1 -- 172.21.15.16:0/3643655079 wait complete. 2023-10-16T12:23:21.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.133+0000 7fd0beefd700 1 Processor -- start 2023-10-16T12:23:21.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.133+0000 7fd0beefd700 1 -- start start 2023-10-16T12:23:21.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.134+0000 7fd0beefd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0b80d2110 0x7fd0b81e4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:21.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.134+0000 7fd0beefd700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd0b815ec80 0x7fd0b815a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:21.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.134+0000 7fd0beefd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd0b815f060 0x7fd0b81f6580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:21.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.134+0000 7fd0beefd700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b81573b0 con 0x7fd0b80d2110 2023-10-16T12:23:21.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.134+0000 7fd0beefd700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b81e5400 con 0x7fd0b815f060 2023-10-16T12:23:21.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.134+0000 7fd0beefd700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b80c61d0 con 0x7fd0b815ec80 2023-10-16T12:23:21.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.134+0000 7fd0bcc99700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0b80d2110 0x7fd0b81e4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:21.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.134+0000 7fd0bcc99700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0b80d2110 0x7fd0b81e4ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36534/0 (socket says 172.21.15.16:36534) 2023-10-16T12:23:21.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.134+0000 7fd0bcc99700 1 -- 172.21.15.16:0/1419936775 learned_addr learned my addr 172.21.15.16:0/1419936775 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:21.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.134+0000 7fd0bd49a700 1 --2- 172.21.15.16:0/1419936775 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd0b815f060 0x7fd0b81f6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:21.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.135+0000 7fd0b7fff700 1 --2- 172.21.15.16:0/1419936775 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd0b815ec80 0x7fd0b815a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:21.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.135+0000 7fd0bcc99700 1 -- 172.21.15.16:0/1419936775 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd0b815ec80 msgr2=0x7fd0b815a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:21.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.135+0000 7fd0bcc99700 1 --2- 172.21.15.16:0/1419936775 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd0b815ec80 0x7fd0b815a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:21.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.135+0000 7fd0bcc99700 1 -- 172.21.15.16:0/1419936775 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd0b815f060 msgr2=0x7fd0b81f6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:21.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.135+0000 7fd0bcc99700 1 --2- 172.21.15.16:0/1419936775 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd0b815f060 0x7fd0b81f6580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:21.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.135+0000 7fd0bcc99700 1 -- 172.21.15.16:0/1419936775 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0a40094a0 con 0x7fd0b80d2110 2023-10-16T12:23:21.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.135+0000 7fd0bcc99700 1 --2- 172.21.15.16:0/1419936775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0b80d2110 0x7fd0b81e4ec0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fd0b8067580 tx=0x7fd0ac004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:21.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.135+0000 7fd0b57fa700 1 -- 172.21.15.16:0/1419936775 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0ac004480 con 0x7fd0b80d2110 2023-10-16T12:23:21.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.135+0000 7fd0b57fa700 1 -- 172.21.15.16:0/1419936775 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0ac01f070 con 0x7fd0b80d2110 2023-10-16T12:23:21.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.135+0000 7fd0b57fa700 1 -- 172.21.15.16:0/1419936775 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0ac0038c0 con 0x7fd0b80d2110 2023-10-16T12:23:21.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.135+0000 7fd0beefd700 1 -- 172.21.15.16:0/1419936775 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0ac009550 con 0x7fd0b80d2110 2023-10-16T12:23:21.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.135+0000 7fd0beefd700 1 -- 172.21.15.16:0/1419936775 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0b806ce30 con 0x7fd0b80d2110 2023-10-16T12:23:21.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.136+0000 7fd0beefd700 1 -- 172.21.15.16:0/1419936775 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0b804ebb0 con 0x7fd0b80d2110 2023-10-16T12:23:21.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.137+0000 7fd0b57fa700 1 -- 172.21.15.16:0/1419936775 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7fd0ac021030 con 0x7fd0b80d2110 2023-10-16T12:23:21.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.137+0000 7fd0b57fa700 1 --2- 172.21.15.16:0/1419936775 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fd09c06b100 0x7fd09c06d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:21.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.137+0000 7fd0b57fa700 1 -- 172.21.15.16:0/1419936775 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7fd0ac08afa0 con 0x7fd0b80d2110 2023-10-16T12:23:21.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.137+0000 7fd0b7fff700 1 --2- 172.21.15.16:0/1419936775 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fd09c06b100 0x7fd09c06d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:21.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.138+0000 7fd0b7fff700 1 --2- 172.21.15.16:0/1419936775 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fd09c06b100 0x7fd09c06d5c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd0b809f530 tx=0x7fd0a4008040 comp rx=0 tx=0).ready entity=mgr.5499 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:21.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.142+0000 7fd0b57fa700 1 -- 172.21.15.16:0/1419936775 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd0ac0bdfe0 con 0x7fd0b80d2110 2023-10-16T12:23:21.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.377+0000 7fd0beefd700 1 -- 172.21.15.16:0/1419936775 --> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir3", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fd0b80b2970 con 0x7fd09c06b100 2023-10-16T12:23:21.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.446+0000 7fd0b57fa700 1 -- 172.21.15.16:0/1419936775 <== mgr.5499 v2:172.21.15.16:6832/1707126831 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fd0a40098b0 con 0x7fd09c06b100 2023-10-16T12:23:21.449 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule deactivated for path /snap_test_dir1/dir3 2023-10-16T12:23:21.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.448+0000 7fd0beefd700 1 -- 172.21.15.16:0/1419936775 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fd09c06b100 msgr2=0x7fd09c06d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:21.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.448+0000 7fd0beefd700 1 --2- 172.21.15.16:0/1419936775 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fd09c06b100 0x7fd09c06d5c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd0b809f530 tx=0x7fd0a4008040 comp rx=0 tx=0).stop 2023-10-16T12:23:21.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.448+0000 7fd0beefd700 1 -- 172.21.15.16:0/1419936775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0b80d2110 msgr2=0x7fd0b81e4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:21.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.448+0000 7fd0beefd700 1 --2- 172.21.15.16:0/1419936775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0b80d2110 0x7fd0b81e4ec0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fd0b8067580 tx=0x7fd0ac004750 comp rx=0 tx=0).stop 2023-10-16T12:23:21.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.448+0000 7fd0beefd700 1 -- 172.21.15.16:0/1419936775 shutdown_connections 2023-10-16T12:23:21.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.448+0000 7fd0beefd700 1 --2- 172.21.15.16:0/1419936775 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fd09c06b100 0x7fd09c06d5c0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:21.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.448+0000 7fd0beefd700 1 --2- 172.21.15.16:0/1419936775 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd0b815f060 0x7fd0b81f6580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:21.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.448+0000 7fd0beefd700 1 --2- 172.21.15.16:0/1419936775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0b80d2110 0x7fd0b81e4ec0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:21.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.448+0000 7fd0beefd700 1 --2- 172.21.15.16:0/1419936775 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd0b815ec80 0x7fd0b815a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:21.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.448+0000 7fd0beefd700 1 -- 172.21.15.16:0/1419936775 >> 172.21.15.16:0/1419936775 conn(0x7fd0b80d15f0 msgr2=0x7fd0b80cc190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:21.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.448+0000 7fd0beefd700 1 -- 172.21.15.16:0/1419936775 shutdown_connections 2023-10-16T12:23:21.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.448+0000 7fd0beefd700 1 -- 172.21.15.16:0/1419936775 wait complete. 2023-10-16T12:23:21.463 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir3 2023-10-16T12:23:21.464 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir4 1m 2023-10-16T12:23:21.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.588+0000 7fe79c7b6700 1 Processor -- start 2023-10-16T12:23:21.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.588+0000 7fe79c7b6700 1 -- start start 2023-10-16T12:23:21.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.589+0000 7fe79c7b6700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe79415bb40 0x7fe794071a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:21.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.589+0000 7fe79c7b6700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fe7940caa40 0x7fe7940cae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:21.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.589+0000 7fe79c7b6700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7940cb3b0 con 0x7fe794072b30 2023-10-16T12:23:21.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.589+0000 7fe79c7b6700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe79406ea20 con 0x7fe7940caa40 2023-10-16T12:23:21.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.589+0000 7fe79c7b6700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe79415c040 con 0x7fe79415bb40 2023-10-16T12:23:21.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.590+0000 7fe79a552700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe794072b30 0x7fe79406e310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54048/0 (socket says 172.21.15.16:54048) 2023-10-16T12:23:21.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.590+0000 7fe79a552700 1 -- 172.21.15.16:0/1482035585 learned_addr learned my addr 172.21.15.16:0/1482035585 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:23:21.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.590+0000 7fe799d51700 1 --2- 172.21.15.16:0/1482035585 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe79415bb40 0x7fe794071a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:21.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.590+0000 7fe79ad53700 1 --2- 172.21.15.16:0/1482035585 >> v2:172.21.15.49:3300/0 conn(0x7fe7940caa40 0x7fe7940cae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:21.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.590+0000 7fe799550700 1 -- 172.21.15.16:0/1482035585 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1417788519 0 0) 0x7fe7940cb3b0 con 0x7fe794072b30 2023-10-16T12:23:21.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.590+0000 7fe799550700 1 -- 172.21.15.16:0/1482035585 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7800035b0 con 0x7fe794072b30 2023-10-16T12:23:21.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.590+0000 7fe799550700 1 -- 172.21.15.16:0/1482035585 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 747853815 0 0) 0x7fe7800035b0 con 0x7fe794072b30 2023-10-16T12:23:21.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.590+0000 7fe799550700 1 -- 172.21.15.16:0/1482035585 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe79415bb40 msgr2=0x7fe794071a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:21.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.590+0000 7fe799550700 1 --2- 172.21.15.16:0/1482035585 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe79415bb40 0x7fe794071a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:21.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.590+0000 7fe799550700 1 -- 172.21.15.16:0/1482035585 >> v2:172.21.15.49:3300/0 conn(0x7fe7940caa40 msgr2=0x7fe7940cae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:21.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.590+0000 7fe799550700 1 --2- 172.21.15.16:0/1482035585 >> v2:172.21.15.49:3300/0 conn(0x7fe7940caa40 0x7fe7940cae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:21.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.590+0000 7fe799550700 1 -- 172.21.15.16:0/1482035585 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe79415c320 con 0x7fe794072b30 2023-10-16T12:23:21.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.590+0000 7fe79ad53700 1 --2- 172.21.15.16:0/1482035585 >> v2:172.21.15.49:3300/0 conn(0x7fe7940caa40 0x7fe7940cae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:23:21.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.590+0000 7fe799550700 1 -- 172.21.15.16:0/1482035585 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7fe790001670 con 0x7fe794072b30 2023-10-16T12:23:21.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.590+0000 7fe799550700 1 -- 172.21.15.16:0/1482035585 >> v1:172.21.15.16:6789/0 conn(0x7fe794072b30 legacy=0x7fe79406e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:21.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.590+0000 7fe799550700 1 --2- 172.21.15.16:0/1482035585 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe79415bb40 0x7fe780003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:21.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.590+0000 7fe799550700 1 --2- 172.21.15.16:0/1482035585 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe780003f80 0x7fe780004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:21.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.590+0000 7fe799550700 1 --2- 172.21.15.16:0/1482035585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe780004d90 0x7fe780005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:21.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.590+0000 7fe799550700 1 -- 172.21.15.16:0/1482035585 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe79406ea20 con 0x7fe780004d90 2023-10-16T12:23:21.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.590+0000 7fe799550700 1 -- 172.21.15.16:0/1482035585 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe79415c040 con 0x7fe780003f80 2023-10-16T12:23:21.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.590+0000 7fe799550700 1 -- 172.21.15.16:0/1482035585 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe78000ba10 con 0x7fe79415bb40 2023-10-16T12:23:21.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.590+0000 7fe79ad53700 1 --2- 172.21.15.16:0/1482035585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe780004d90 0x7fe780005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:21.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.590+0000 7fe79a552700 1 --2- 172.21.15.16:0/1482035585 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe79415bb40 0x7fe780003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:21.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.590+0000 7fe799d51700 1 --2- 172.21.15.16:0/1482035585 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe780003f80 0x7fe780004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:21.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.591+0000 7fe79ad53700 1 -- 172.21.15.16:0/1482035585 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe79415bb40 msgr2=0x7fe780003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:21.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.591+0000 7fe79ad53700 1 --2- 172.21.15.16:0/1482035585 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe79415bb40 0x7fe780003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:21.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.591+0000 7fe79ad53700 1 -- 172.21.15.16:0/1482035585 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe780003f80 msgr2=0x7fe780004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:21.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.591+0000 7fe79ad53700 1 --2- 172.21.15.16:0/1482035585 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe780003f80 0x7fe780004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:21.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.591+0000 7fe79ad53700 1 -- 172.21.15.16:0/1482035585 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7800035b0 con 0x7fe780004d90 2023-10-16T12:23:21.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.591+0000 7fe79ad53700 1 --2- 172.21.15.16:0/1482035585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe780004d90 0x7fe780005210 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fe78c00e350 tx=0x7fe780005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3a7e8a21a978e6c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:21.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.591+0000 7fe799550700 1 -- 172.21.15.16:0/1482035585 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe78c00ed90 con 0x7fe780004d90 2023-10-16T12:23:21.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.591+0000 7fe799550700 1 -- 172.21.15.16:0/1482035585 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe78c00f040 con 0x7fe780004d90 2023-10-16T12:23:21.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.591+0000 7fe799550700 1 -- 172.21.15.16:0/1482035585 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe78c0146a0 con 0x7fe780004d90 2023-10-16T12:23:21.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.591+0000 7fe79c7b6700 1 -- 172.21.15.16:0/1482035585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe780004d90 msgr2=0x7fe780005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:21.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.591+0000 7fe79c7b6700 1 --2- 172.21.15.16:0/1482035585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe780004d90 0x7fe780005210 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fe78c00e350 tx=0x7fe780005880 comp rx=0 tx=0).stop 2023-10-16T12:23:21.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.591+0000 7fe79ad53700 1 -- 172.21.15.16:0/1482035585 reap_dead start 2023-10-16T12:23:21.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.592+0000 7fe79c7b6700 1 -- 172.21.15.16:0/1482035585 shutdown_connections 2023-10-16T12:23:21.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.592+0000 7fe79c7b6700 1 -- 172.21.15.16:0/1482035585 >> 172.21.15.16:0/1482035585 conn(0x7fe7940b1f40 msgr2=0x7fe7940b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:21.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.592+0000 7fe79c7b6700 1 -- 172.21.15.16:0/1482035585 shutdown_connections 2023-10-16T12:23:21.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.592+0000 7fe79c7b6700 1 -- 172.21.15.16:0/1482035585 wait complete. 2023-10-16T12:23:21.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.593+0000 7fe79c7b6700 1 Processor -- start 2023-10-16T12:23:21.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.593+0000 7fe79c7b6700 1 -- start start 2023-10-16T12:23:21.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.594+0000 7fe79c7b6700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe780004d90 0x7fe79415a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:21.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.594+0000 7fe79c7b6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7940b5530 0x7fe7940c9900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:21.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.594+0000 7fe79c7b6700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe79414d2c0 0x7fe79414d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:21.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.594+0000 7fe79c7b6700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe794156100 con 0x7fe7940b5530 2023-10-16T12:23:21.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.594+0000 7fe79c7b6700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7941566c0 con 0x7fe79414d2c0 2023-10-16T12:23:21.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.594+0000 7fe79c7b6700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe79414dc70 con 0x7fe780004d90 2023-10-16T12:23:21.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.594+0000 7fe799d51700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7940b5530 0x7fe7940c9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:21.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.594+0000 7fe799d51700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7940b5530 0x7fe7940c9900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36556/0 (socket says 172.21.15.16:36556) 2023-10-16T12:23:21.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.594+0000 7fe799d51700 1 -- 172.21.15.16:0/1949875765 learned_addr learned my addr 172.21.15.16:0/1949875765 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:21.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.594+0000 7fe79a552700 1 --2- 172.21.15.16:0/1949875765 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe780004d90 0x7fe79415a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:21.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.594+0000 7fe79ad53700 1 --2- 172.21.15.16:0/1949875765 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe79414d2c0 0x7fe79414d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:21.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.594+0000 7fe799d51700 1 -- 172.21.15.16:0/1949875765 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe780004d90 msgr2=0x7fe79415a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:21.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.594+0000 7fe799d51700 1 --2- 172.21.15.16:0/1949875765 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe780004d90 0x7fe79415a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:21.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.594+0000 7fe799d51700 1 -- 172.21.15.16:0/1949875765 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe79414d2c0 msgr2=0x7fe79414d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:21.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.594+0000 7fe799d51700 1 --2- 172.21.15.16:0/1949875765 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe79414d2c0 0x7fe79414d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:21.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.594+0000 7fe799d51700 1 -- 172.21.15.16:0/1949875765 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe79415aca0 con 0x7fe7940b5530 2023-10-16T12:23:21.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.594+0000 7fe799d51700 1 --2- 172.21.15.16:0/1949875765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7940b5530 0x7fe7940c9900 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fe7840119a0 tx=0x7fe784011cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:21.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.594+0000 7fe78affd700 1 -- 172.21.15.16:0/1949875765 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe78400f780 con 0x7fe7940b5530 2023-10-16T12:23:21.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.594+0000 7fe78affd700 1 -- 172.21.15.16:0/1949875765 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe78400f8e0 con 0x7fe7940b5530 2023-10-16T12:23:21.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.594+0000 7fe79c7b6700 1 -- 172.21.15.16:0/1949875765 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe79415af80 con 0x7fe7940b5530 2023-10-16T12:23:21.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.594+0000 7fe78affd700 1 -- 172.21.15.16:0/1949875765 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe78400fc70 con 0x7fe7940b5530 2023-10-16T12:23:21.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.595+0000 7fe79c7b6700 1 -- 172.21.15.16:0/1949875765 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe79415a950 con 0x7fe7940b5530 2023-10-16T12:23:21.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.595+0000 7fe79c7b6700 1 -- 172.21.15.16:0/1949875765 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe794156800 con 0x7fe7940b5530 2023-10-16T12:23:21.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.596+0000 7fe78affd700 1 -- 172.21.15.16:0/1949875765 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7fe784014440 con 0x7fe7940b5530 2023-10-16T12:23:21.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.596+0000 7fe78affd700 1 --2- 172.21.15.16:0/1949875765 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fe77806b100 0x7fe77806d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:21.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.596+0000 7fe78affd700 1 -- 172.21.15.16:0/1949875765 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7fe78408a160 con 0x7fe7940b5530 2023-10-16T12:23:21.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.596+0000 7fe79a552700 1 --2- 172.21.15.16:0/1949875765 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fe77806b100 0x7fe77806d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:21.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.597+0000 7fe79a552700 1 --2- 172.21.15.16:0/1949875765 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fe77806b100 0x7fe77806d5c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe7940ca230 tx=0x7fe790007980 comp rx=0 tx=0).ready entity=mgr.5499 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:21.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.601+0000 7fe78affd700 1 -- 172.21.15.16:0/1949875765 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe794156800 con 0x7fe7940b5530 2023-10-16T12:23:21.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.835+0000 7fe79c7b6700 1 -- 172.21.15.16:0/1949875765 --> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir4", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fe79415c8e0 con 0x7fe77806b100 2023-10-16T12:23:21.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.903+0000 7fe78affd700 1 -- 172.21.15.16:0/1949875765 <== mgr.5499 v2:172.21.15.16:6832/1707126831 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fe790007120 con 0x7fe77806b100 2023-10-16T12:23:21.907 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule deactivated for path /snap_test_dir1/dir4 2023-10-16T12:23:21.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.905+0000 7fe79c7b6700 1 -- 172.21.15.16:0/1949875765 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fe77806b100 msgr2=0x7fe77806d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:21.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.905+0000 7fe79c7b6700 1 --2- 172.21.15.16:0/1949875765 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fe77806b100 0x7fe77806d5c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe7940ca230 tx=0x7fe790007980 comp rx=0 tx=0).stop 2023-10-16T12:23:21.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.905+0000 7fe79c7b6700 1 -- 172.21.15.16:0/1949875765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7940b5530 msgr2=0x7fe7940c9900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:21.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.905+0000 7fe79c7b6700 1 --2- 172.21.15.16:0/1949875765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7940b5530 0x7fe7940c9900 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fe7840119a0 tx=0x7fe784011cb0 comp rx=0 tx=0).stop 2023-10-16T12:23:21.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.905+0000 7fe79c7b6700 1 -- 172.21.15.16:0/1949875765 shutdown_connections 2023-10-16T12:23:21.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.905+0000 7fe79c7b6700 1 --2- 172.21.15.16:0/1949875765 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fe77806b100 0x7fe77806d5c0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:21.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.905+0000 7fe79c7b6700 1 --2- 172.21.15.16:0/1949875765 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe79414d2c0 0x7fe79414d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:21.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.905+0000 7fe79c7b6700 1 --2- 172.21.15.16:0/1949875765 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe780004d90 0x7fe79415a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:21.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.905+0000 7fe79c7b6700 1 --2- 172.21.15.16:0/1949875765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7940b5530 0x7fe7940c9900 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:21.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.905+0000 7fe79c7b6700 1 -- 172.21.15.16:0/1949875765 >> 172.21.15.16:0/1949875765 conn(0x7fe7940b1f40 msgr2=0x7fe7940c3f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:21.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.905+0000 7fe79c7b6700 1 -- 172.21.15.16:0/1949875765 shutdown_connections 2023-10-16T12:23:21.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:21.906+0000 7fe79c7b6700 1 -- 172.21.15.16:0/1949875765 wait complete. 2023-10-16T12:23:21.920 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir4 2023-10-16T12:23:21.921 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir5 1m 2023-10-16T12:23:22.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.045+0000 7f3fd1086700 1 Processor -- start 2023-10-16T12:23:22.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.046+0000 7f3fd1086700 1 -- start start 2023-10-16T12:23:22.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.047+0000 7f3fd1086700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f3fcc15b070 0x7f3fcc15b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:22.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.047+0000 7f3fd1086700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3fcc06e170 0x7f3fcc0cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:22.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.047+0000 7f3fd1086700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fcc0cef70 con 0x7f3fcc0cb880 2023-10-16T12:23:22.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.047+0000 7f3fd1086700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fcc0ce360 con 0x7f3fcc15b070 2023-10-16T12:23:22.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.047+0000 7f3fd1086700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fcc0ce640 con 0x7f3fcc06e170 2023-10-16T12:23:22.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.047+0000 7f3fcad9d700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f3fcc15b070 0x7f3fcc15b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:22.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.047+0000 7f3fcb59e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3fcc06e170 0x7f3fcc0cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:22.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.047+0000 7f3fcad9d700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f3fcc15b070 0x7f3fcc15b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:35624/0 (socket says 172.21.15.16:35624) 2023-10-16T12:23:22.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.047+0000 7f3fcad9d700 1 -- 172.21.15.16:0/1445726316 learned_addr learned my addr 172.21.15.16:0/1445726316 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:22.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.048+0000 7f3fcb59e700 1 -- 172.21.15.16:0/1445726316 >> v2:172.21.15.49:3300/0 conn(0x7f3fcc15b070 msgr2=0x7f3fcc15b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:22.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.048+0000 7f3fcb59e700 1 --2- 172.21.15.16:0/1445726316 >> v2:172.21.15.49:3300/0 conn(0x7f3fcc15b070 0x7f3fcc15b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:22.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.048+0000 7f3fcb59e700 1 -- 172.21.15.16:0/1445726316 >> v1:172.21.15.49:6789/0 conn(0x7f3fcc0cb880 legacy=0x7f3fcc06da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:22.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.048+0000 7f3fcb59e700 1 -- 172.21.15.16:0/1445726316 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fcc0ce920 con 0x7f3fcc06e170 2023-10-16T12:23:22.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.048+0000 7f3fcad9d700 1 --2- 172.21.15.16:0/1445726316 >> v2:172.21.15.49:3300/0 conn(0x7f3fcc15b070 0x7f3fcc15b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:23:22.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.048+0000 7f3fcb59e700 1 --2- 172.21.15.16:0/1445726316 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3fcc06e170 0x7f3fcc0cbd00 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3fc4009d90 tx=0x7f3fc400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b06a79a17b19412 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:22.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.048+0000 7f3fc9d9b700 1 -- 172.21.15.16:0/1445726316 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fc400e070 con 0x7f3fcc06e170 2023-10-16T12:23:22.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.048+0000 7f3fc9d9b700 1 -- 172.21.15.16:0/1445726316 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fc400bb20 con 0x7f3fcc06e170 2023-10-16T12:23:22.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.048+0000 7f3fc9d9b700 1 -- 172.21.15.16:0/1445726316 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fc4012430 con 0x7f3fcc06e170 2023-10-16T12:23:22.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.048+0000 7f3fd1086700 1 -- 172.21.15.16:0/1445726316 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3fcc06e170 msgr2=0x7f3fcc0cbd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:22.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.048+0000 7f3fd1086700 1 --2- 172.21.15.16:0/1445726316 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3fcc06e170 0x7f3fcc0cbd00 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3fc4009d90 tx=0x7f3fc400b330 comp rx=0 tx=0).stop 2023-10-16T12:23:22.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.049+0000 7f3fd1086700 1 -- 172.21.15.16:0/1445726316 shutdown_connections 2023-10-16T12:23:22.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.049+0000 7f3fd1086700 1 --2- 172.21.15.16:0/1445726316 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3fcc06e170 0x7f3fcc0cbd00 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:22.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.049+0000 7f3fd1086700 1 --2- 172.21.15.16:0/1445726316 >> v2:172.21.15.49:3300/0 conn(0x7f3fcc15b070 0x7f3fcc15b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:22.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.049+0000 7f3fd1086700 1 -- 172.21.15.16:0/1445726316 >> 172.21.15.16:0/1445726316 conn(0x7f3fcc0b1f40 msgr2=0x7f3fcc0b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:22.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.049+0000 7f3fd1086700 1 -- 172.21.15.16:0/1445726316 shutdown_connections 2023-10-16T12:23:22.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.049+0000 7f3fd1086700 1 -- 172.21.15.16:0/1445726316 wait complete. 2023-10-16T12:23:22.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.050+0000 7f3fd1086700 1 Processor -- start 2023-10-16T12:23:22.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.050+0000 7f3fd1086700 1 -- start start 2023-10-16T12:23:22.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.051+0000 7f3fd1086700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3fcc06e170 0x7f3fcc149e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:22.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.051+0000 7f3fd1086700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3fcc0cb880 0x7f3fcc159740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:22.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.051+0000 7f3fd1086700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3fcc15b070 0x7f3fcc0ca8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:22.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.051+0000 7f3fd1086700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fcc0c7cf0 con 0x7f3fcc15b070 2023-10-16T12:23:22.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.051+0000 7f3fd1086700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fcc0c7a10 con 0x7f3fcc06e170 2023-10-16T12:23:22.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.051+0000 7f3fd1086700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fcc0c9dc0 con 0x7f3fcc0cb880 2023-10-16T12:23:22.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.051+0000 7f3fcb59e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3fcc15b070 0x7f3fcc0ca8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:22.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.051+0000 7f3fcb59e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3fcc15b070 0x7f3fcc0ca8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36572/0 (socket says 172.21.15.16:36572) 2023-10-16T12:23:22.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.051+0000 7f3fcb59e700 1 -- 172.21.15.16:0/1160075939 learned_addr learned my addr 172.21.15.16:0/1160075939 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:22.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.051+0000 7f3fcad9d700 1 --2- 172.21.15.16:0/1160075939 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3fcc06e170 0x7f3fcc149e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:22.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.051+0000 7f3fca59c700 1 --2- 172.21.15.16:0/1160075939 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3fcc0cb880 0x7f3fcc159740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:22.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.051+0000 7f3fcad9d700 1 -- 172.21.15.16:0/1160075939 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3fcc0cb880 msgr2=0x7f3fcc159740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:22.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.051+0000 7f3fcad9d700 1 --2- 172.21.15.16:0/1160075939 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3fcc0cb880 0x7f3fcc159740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:22.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.051+0000 7f3fcad9d700 1 -- 172.21.15.16:0/1160075939 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3fcc15b070 msgr2=0x7f3fcc0ca8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:22.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.051+0000 7f3fcad9d700 1 --2- 172.21.15.16:0/1160075939 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3fcc15b070 0x7f3fcc0ca8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:22.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.051+0000 7f3fcad9d700 1 -- 172.21.15.16:0/1160075939 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fbc009550 con 0x7f3fcc06e170 2023-10-16T12:23:22.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.051+0000 7f3fcb59e700 1 --2- 172.21.15.16:0/1160075939 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3fcc15b070 0x7f3fcc0ca8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:23:22.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.052+0000 7f3fcad9d700 1 --2- 172.21.15.16:0/1160075939 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3fcc06e170 0x7f3fcc149e00 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f3fbc00e980 tx=0x7f3fbc00ec90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:22.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.052+0000 7f3fc37fe700 1 -- 172.21.15.16:0/1160075939 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fbc004030 con 0x7f3fcc06e170 2023-10-16T12:23:22.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.052+0000 7f3fc37fe700 1 -- 172.21.15.16:0/1160075939 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fbc004190 con 0x7f3fcc06e170 2023-10-16T12:23:22.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.052+0000 7f3fd1086700 1 -- 172.21.15.16:0/1160075939 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fc4009a70 con 0x7f3fcc06e170 2023-10-16T12:23:22.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.052+0000 7f3fc37fe700 1 -- 172.21.15.16:0/1160075939 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fbc004520 con 0x7f3fcc06e170 2023-10-16T12:23:22.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.052+0000 7f3fd1086700 1 -- 172.21.15.16:0/1160075939 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fcc0ca380 con 0x7f3fcc06e170 2023-10-16T12:23:22.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.053+0000 7f3fd1086700 1 -- 172.21.15.16:0/1160075939 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fcc0c9c20 con 0x7f3fcc06e170 2023-10-16T12:23:22.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.053+0000 7f3fc37fe700 1 -- 172.21.15.16:0/1160075939 <== mon.1 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f3fbc0036a0 con 0x7f3fcc06e170 2023-10-16T12:23:22.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.054+0000 7f3fc37fe700 1 --2- 172.21.15.16:0/1160075939 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f3fac06b100 0x7f3fac06d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:22.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.054+0000 7f3fc37fe700 1 -- 172.21.15.16:0/1160075939 <== mon.1 v2:172.21.15.49:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f3fbc089e20 con 0x7f3fcc06e170 2023-10-16T12:23:22.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.054+0000 7f3fca59c700 1 --2- 172.21.15.16:0/1160075939 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f3fac06b100 0x7f3fac06d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:22.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.054+0000 7f3fca59c700 1 --2- 172.21.15.16:0/1160075939 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f3fac06b100 0x7f3fac06d5c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3fcc1f4290 tx=0x7f3fb4009040 comp rx=0 tx=0).ready entity=mgr.5499 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:22.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.059+0000 7f3fc37fe700 1 -- 172.21.15.16:0/1160075939 <== mon.1 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3fbc054b60 con 0x7f3fcc06e170 2023-10-16T12:23:22.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.292+0000 7f3fd1086700 1 -- 172.21.15.16:0/1160075939 --> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir5", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f3fcc0728d0 con 0x7f3fac06b100 2023-10-16T12:23:22.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.363+0000 7f3fc37fe700 1 -- 172.21.15.16:0/1160075939 <== mgr.5499 v2:172.21.15.16:6832/1707126831 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f3fb4001d60 con 0x7f3fac06b100 2023-10-16T12:23:22.366 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule deactivated for path /snap_test_dir1/dir5 2023-10-16T12:23:22.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.365+0000 7f3fd1086700 1 -- 172.21.15.16:0/1160075939 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f3fac06b100 msgr2=0x7f3fac06d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:22.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.365+0000 7f3fd1086700 1 --2- 172.21.15.16:0/1160075939 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f3fac06b100 0x7f3fac06d5c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3fcc1f4290 tx=0x7f3fb4009040 comp rx=0 tx=0).stop 2023-10-16T12:23:22.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.365+0000 7f3fd1086700 1 -- 172.21.15.16:0/1160075939 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3fcc06e170 msgr2=0x7f3fcc149e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:22.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.365+0000 7f3fd1086700 1 --2- 172.21.15.16:0/1160075939 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3fcc06e170 0x7f3fcc149e00 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f3fbc00e980 tx=0x7f3fbc00ec90 comp rx=0 tx=0).stop 2023-10-16T12:23:22.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.366+0000 7f3fd1086700 1 -- 172.21.15.16:0/1160075939 shutdown_connections 2023-10-16T12:23:22.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.366+0000 7f3fd1086700 1 --2- 172.21.15.16:0/1160075939 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f3fac06b100 0x7f3fac06d5c0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:22.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.366+0000 7f3fd1086700 1 --2- 172.21.15.16:0/1160075939 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3fcc15b070 0x7f3fcc0ca8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:22.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.366+0000 7f3fd1086700 1 --2- 172.21.15.16:0/1160075939 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3fcc06e170 0x7f3fcc149e00 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:22.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.366+0000 7f3fd1086700 1 --2- 172.21.15.16:0/1160075939 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3fcc0cb880 0x7f3fcc159740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:22.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.366+0000 7f3fd1086700 1 -- 172.21.15.16:0/1160075939 >> 172.21.15.16:0/1160075939 conn(0x7f3fcc0b1f40 msgr2=0x7f3fcc0b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:22.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.366+0000 7f3fd1086700 1 -- 172.21.15.16:0/1160075939 shutdown_connections 2023-10-16T12:23:22.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.366+0000 7f3fd1086700 1 -- 172.21.15.16:0/1160075939 wait complete. 2023-10-16T12:23:22.380 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir5 2023-10-16T12:23:22.381 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir6 1m 2023-10-16T12:23:22.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.506+0000 7fe3ec513700 1 Processor -- start 2023-10-16T12:23:22.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.506+0000 7fe3ec513700 1 -- start start 2023-10-16T12:23:22.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.507+0000 7fe3ec513700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fe3e415b2c0 0x7fe3e4161340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:22.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.507+0000 7fe3ec513700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe3e406a7e0 0x7fe3e406ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:22.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.507+0000 7fe3ec513700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3e415ecb0 con 0x7fe3e406e570 2023-10-16T12:23:22.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.507+0000 7fe3ec513700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3e406b190 con 0x7fe3e406a7e0 2023-10-16T12:23:22.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.507+0000 7fe3ec513700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3e4161880 con 0x7fe3e415b2c0 2023-10-16T12:23:22.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.507+0000 7fe3eaab0700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe3e406a7e0 0x7fe3e406ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:22.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.507+0000 7fe3e9aae700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe3e406e570 0x7fe3e415e5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54084/0 (socket says 172.21.15.16:54084) 2023-10-16T12:23:22.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.507+0000 7fe3e9aae700 1 -- 172.21.15.16:0/4233059052 learned_addr learned my addr 172.21.15.16:0/4233059052 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:23:22.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.507+0000 7fe3ea2af700 1 --2- 172.21.15.16:0/4233059052 >> v2:172.21.15.49:3300/0 conn(0x7fe3e415b2c0 0x7fe3e4161340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:22.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.507+0000 7fe3eaab0700 1 -- 172.21.15.16:0/4233059052 >> v2:172.21.15.49:3300/0 conn(0x7fe3e415b2c0 msgr2=0x7fe3e4161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:22.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.507+0000 7fe3eaab0700 1 --2- 172.21.15.16:0/4233059052 >> v2:172.21.15.49:3300/0 conn(0x7fe3e415b2c0 0x7fe3e4161340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:22.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.507+0000 7fe3eaab0700 1 -- 172.21.15.16:0/4233059052 >> v1:172.21.15.16:6789/0 conn(0x7fe3e406e570 legacy=0x7fe3e415e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:22.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.507+0000 7fe3eaab0700 1 -- 172.21.15.16:0/4233059052 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3e415b6a0 con 0x7fe3e406a7e0 2023-10-16T12:23:22.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.507+0000 7fe3e92ad700 1 -- 172.21.15.16:0/4233059052 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 366975136 0 0) 0x7fe3e415ecb0 con 0x7fe3e406e570 2023-10-16T12:23:22.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.507+0000 7fe3eaab0700 1 --2- 172.21.15.16:0/4233059052 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3e406a7e0 0x7fe3e406ac50 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fe3e406b4a0 tx=0x7fe3dc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2110703c782fca8e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:22.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.507+0000 7fe3e92ad700 1 -- 172.21.15.16:0/4233059052 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3dc00e070 con 0x7fe3e406a7e0 2023-10-16T12:23:22.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.507+0000 7fe3e92ad700 1 -- 172.21.15.16:0/4233059052 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3dc00bbd0 con 0x7fe3e406a7e0 2023-10-16T12:23:22.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.507+0000 7fe3e92ad700 1 -- 172.21.15.16:0/4233059052 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3dc012430 con 0x7fe3e406a7e0 2023-10-16T12:23:22.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.508+0000 7fe3ec513700 1 -- 172.21.15.16:0/4233059052 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3e406a7e0 msgr2=0x7fe3e406ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:22.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.508+0000 7fe3ec513700 1 --2- 172.21.15.16:0/4233059052 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3e406a7e0 0x7fe3e406ac50 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fe3e406b4a0 tx=0x7fe3dc00b330 comp rx=0 tx=0).stop 2023-10-16T12:23:22.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.508+0000 7fe3ec513700 1 -- 172.21.15.16:0/4233059052 shutdown_connections 2023-10-16T12:23:22.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.508+0000 7fe3ec513700 1 --2- 172.21.15.16:0/4233059052 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3e406a7e0 0x7fe3e406ac50 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:22.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.508+0000 7fe3ec513700 1 --2- 172.21.15.16:0/4233059052 >> v2:172.21.15.49:3300/0 conn(0x7fe3e415b2c0 0x7fe3e4161340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:22.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.508+0000 7fe3ec513700 1 -- 172.21.15.16:0/4233059052 >> 172.21.15.16:0/4233059052 conn(0x7fe3e40b1f40 msgr2=0x7fe3e40b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:22.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.508+0000 7fe3ec513700 1 -- 172.21.15.16:0/4233059052 shutdown_connections 2023-10-16T12:23:22.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.508+0000 7fe3ec513700 1 -- 172.21.15.16:0/4233059052 wait complete. 2023-10-16T12:23:22.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.509+0000 7fe3ec513700 1 Processor -- start 2023-10-16T12:23:22.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.509+0000 7fe3ec513700 1 -- start start 2023-10-16T12:23:22.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.510+0000 7fe3ec513700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe3e406a7e0 0x7fe3e40b2f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:22.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.510+0000 7fe3ec513700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3e406e570 0x7fe3e4157870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:22.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.510+0000 7fe3ec513700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe3e415b2c0 0x7fe3e40c4be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:22.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.510+0000 7fe3ec513700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3e40c8bb0 con 0x7fe3e406e570 2023-10-16T12:23:22.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.510+0000 7fe3ec513700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3e40b3450 con 0x7fe3e406a7e0 2023-10-16T12:23:22.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.510+0000 7fe3ec513700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3e40b3790 con 0x7fe3e415b2c0 2023-10-16T12:23:22.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.510+0000 7fe3e9aae700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3e406e570 0x7fe3e4157870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:22.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.510+0000 7fe3e9aae700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3e406e570 0x7fe3e4157870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36584/0 (socket says 172.21.15.16:36584) 2023-10-16T12:23:22.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.510+0000 7fe3e9aae700 1 -- 172.21.15.16:0/1219394178 learned_addr learned my addr 172.21.15.16:0/1219394178 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:22.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.510+0000 7fe3eaab0700 1 --2- 172.21.15.16:0/1219394178 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe3e415b2c0 0x7fe3e40c4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:22.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.510+0000 7fe3ea2af700 1 --2- 172.21.15.16:0/1219394178 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe3e406a7e0 0x7fe3e40b2f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:22.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.511+0000 7fe3e9aae700 1 -- 172.21.15.16:0/1219394178 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe3e415b2c0 msgr2=0x7fe3e40c4be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:22.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.511+0000 7fe3e9aae700 1 --2- 172.21.15.16:0/1219394178 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe3e415b2c0 0x7fe3e40c4be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:22.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.511+0000 7fe3e9aae700 1 -- 172.21.15.16:0/1219394178 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe3e406a7e0 msgr2=0x7fe3e40b2f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:22.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.511+0000 7fe3e9aae700 1 --2- 172.21.15.16:0/1219394178 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe3e406a7e0 0x7fe3e40b2f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:22.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.511+0000 7fe3e9aae700 1 -- 172.21.15.16:0/1219394178 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3dc009a70 con 0x7fe3e406e570 2023-10-16T12:23:22.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.511+0000 7fe3ea2af700 1 --2- 172.21.15.16:0/1219394178 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe3e406a7e0 0x7fe3e40b2f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:23:22.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.511+0000 7fe3e9aae700 1 --2- 172.21.15.16:0/1219394178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3e406e570 0x7fe3e4157870 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fe3d400c550 tx=0x7fe3d400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:22.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.511+0000 7fe3daffd700 1 -- 172.21.15.16:0/1219394178 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3d4004c80 con 0x7fe3e406e570 2023-10-16T12:23:22.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.511+0000 7fe3daffd700 1 -- 172.21.15.16:0/1219394178 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3d4004e00 con 0x7fe3e406e570 2023-10-16T12:23:22.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.511+0000 7fe3ec513700 1 -- 172.21.15.16:0/1219394178 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3e40c8560 con 0x7fe3e406e570 2023-10-16T12:23:22.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.511+0000 7fe3daffd700 1 -- 172.21.15.16:0/1219394178 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3d4007310 con 0x7fe3e406e570 2023-10-16T12:23:22.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.511+0000 7fe3ec513700 1 -- 172.21.15.16:0/1219394178 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3e40b3cc0 con 0x7fe3e406e570 2023-10-16T12:23:22.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.512+0000 7fe3ec513700 1 -- 172.21.15.16:0/1219394178 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3e404ebb0 con 0x7fe3e406e570 2023-10-16T12:23:22.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.512+0000 7fe3daffd700 1 -- 172.21.15.16:0/1219394178 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7fe3d4007470 con 0x7fe3e406e570 2023-10-16T12:23:22.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.513+0000 7fe3daffd700 1 --2- 172.21.15.16:0/1219394178 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fe3c806b100 0x7fe3c806d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:22.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.513+0000 7fe3daffd700 1 -- 172.21.15.16:0/1219394178 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7fe3d40898b0 con 0x7fe3e406e570 2023-10-16T12:23:22.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.513+0000 7fe3ea2af700 1 --2- 172.21.15.16:0/1219394178 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fe3c806b100 0x7fe3c806d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:22.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.513+0000 7fe3ea2af700 1 --2- 172.21.15.16:0/1219394178 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fe3c806b100 0x7fe3c806d5c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe3e4160fa0 tx=0x7fe3e0008040 comp rx=0 tx=0).ready entity=mgr.5499 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:22.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.517+0000 7fe3daffd700 1 -- 172.21.15.16:0/1219394178 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe3d40543e0 con 0x7fe3e406e570 2023-10-16T12:23:22.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.754+0000 7fe3ec513700 1 -- 172.21.15.16:0/1219394178 --> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir6", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fe3e4161e00 con 0x7fe3c806b100 2023-10-16T12:23:22.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.821+0000 7fe3daffd700 1 -- 172.21.15.16:0/1219394178 <== mgr.5499 v2:172.21.15.16:6832/1707126831 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fe3e00072a0 con 0x7fe3c806b100 2023-10-16T12:23:22.824 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule deactivated for path /snap_test_dir1/dir6 2023-10-16T12:23:22.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.823+0000 7fe3ec513700 1 -- 172.21.15.16:0/1219394178 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fe3c806b100 msgr2=0x7fe3c806d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:22.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.823+0000 7fe3ec513700 1 --2- 172.21.15.16:0/1219394178 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fe3c806b100 0x7fe3c806d5c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe3e4160fa0 tx=0x7fe3e0008040 comp rx=0 tx=0).stop 2023-10-16T12:23:22.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.823+0000 7fe3ec513700 1 -- 172.21.15.16:0/1219394178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3e406e570 msgr2=0x7fe3e4157870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:22.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.823+0000 7fe3ec513700 1 --2- 172.21.15.16:0/1219394178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3e406e570 0x7fe3e4157870 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fe3d400c550 tx=0x7fe3d400c860 comp rx=0 tx=0).stop 2023-10-16T12:23:22.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.823+0000 7fe3ec513700 1 -- 172.21.15.16:0/1219394178 shutdown_connections 2023-10-16T12:23:22.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.823+0000 7fe3ec513700 1 --2- 172.21.15.16:0/1219394178 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fe3c806b100 0x7fe3c806d5c0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:22.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.823+0000 7fe3ec513700 1 --2- 172.21.15.16:0/1219394178 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe3e415b2c0 0x7fe3e40c4be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:22.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.823+0000 7fe3ec513700 1 --2- 172.21.15.16:0/1219394178 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe3e406a7e0 0x7fe3e40b2f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:22.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.823+0000 7fe3ec513700 1 --2- 172.21.15.16:0/1219394178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3e406e570 0x7fe3e4157870 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:22.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.823+0000 7fe3ec513700 1 -- 172.21.15.16:0/1219394178 >> 172.21.15.16:0/1219394178 conn(0x7fe3e40b1f40 msgr2=0x7fe3e415c440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:22.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.823+0000 7fe3ec513700 1 -- 172.21.15.16:0/1219394178 shutdown_connections 2023-10-16T12:23:22.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.823+0000 7fe3ec513700 1 -- 172.21.15.16:0/1219394178 wait complete. 2023-10-16T12:23:22.838 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir6 2023-10-16T12:23:22.838 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir7 1m 2023-10-16T12:23:22.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.963+0000 7fa816740700 1 Processor -- start 2023-10-16T12:23:22.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.964+0000 7fa816740700 1 -- start start 2023-10-16T12:23:22.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.964+0000 7fa816740700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa810160fe0 0x7fa8100ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:22.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.964+0000 7fa816740700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa81006da30 0x7fa81006a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:22.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.964+0000 7fa816740700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8101613c0 con 0x7fa81006ad20 2023-10-16T12:23:22.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.964+0000 7fa816740700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8100cf680 con 0x7fa810160fe0 2023-10-16T12:23:22.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.964+0000 7fa816740700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa81006b1b0 con 0x7fa81006da30 2023-10-16T12:23:22.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.964+0000 7fa80ffff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa810160fe0 0x7fa8100ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:22.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.964+0000 7fa80ffff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa810160fe0 0x7fa8100ac8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36590/0 (socket says 172.21.15.16:36590) 2023-10-16T12:23:22.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.964+0000 7fa80ffff700 1 -- 172.21.15.16:0/1612141545 learned_addr learned my addr 172.21.15.16:0/1612141545 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:22.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.965+0000 7fa80f7fe700 1 --2- 172.21.15.16:0/1612141545 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa81006da30 0x7fa81006a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:22.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.965+0000 7fa80ffff700 1 -- 172.21.15.16:0/1612141545 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa81006da30 msgr2=0x7fa81006a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:22.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.965+0000 7fa80ffff700 1 --2- 172.21.15.16:0/1612141545 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa81006da30 0x7fa81006a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:22.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.965+0000 7fa80ffff700 1 -- 172.21.15.16:0/1612141545 >> v1:172.21.15.49:6789/0 conn(0x7fa81006ad20 legacy=0x7fa810071a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:22.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.965+0000 7fa80ffff700 1 -- 172.21.15.16:0/1612141545 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8100adb00 con 0x7fa810160fe0 2023-10-16T12:23:22.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.965+0000 7fa80ffff700 1 --2- 172.21.15.16:0/1612141545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa810160fe0 0x7fa8100ac8d0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fa800006fd0 tx=0x7fa800009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d745fd850b0406b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:22.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.965+0000 7fa80effd700 1 -- 172.21.15.16:0/1612141545 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa800005e30 con 0x7fa810160fe0 2023-10-16T12:23:22.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.965+0000 7fa80effd700 1 -- 172.21.15.16:0/1612141545 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa800006030 con 0x7fa810160fe0 2023-10-16T12:23:22.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.965+0000 7fa80effd700 1 -- 172.21.15.16:0/1612141545 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8000063c0 con 0x7fa810160fe0 2023-10-16T12:23:22.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.966+0000 7fa816740700 1 -- 172.21.15.16:0/1612141545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa810160fe0 msgr2=0x7fa8100ac8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:22.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.966+0000 7fa816740700 1 --2- 172.21.15.16:0/1612141545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa810160fe0 0x7fa8100ac8d0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fa800006fd0 tx=0x7fa800009bd0 comp rx=0 tx=0).stop 2023-10-16T12:23:22.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.966+0000 7fa816740700 1 -- 172.21.15.16:0/1612141545 shutdown_connections 2023-10-16T12:23:22.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.966+0000 7fa816740700 1 --2- 172.21.15.16:0/1612141545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa810160fe0 0x7fa8100ac8d0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:22.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.966+0000 7fa816740700 1 --2- 172.21.15.16:0/1612141545 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa81006da30 0x7fa81006a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:22.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.966+0000 7fa816740700 1 -- 172.21.15.16:0/1612141545 >> 172.21.15.16:0/1612141545 conn(0x7fa8100b1ed0 msgr2=0x7fa8100b22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:22.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.966+0000 7fa816740700 1 -- 172.21.15.16:0/1612141545 shutdown_connections 2023-10-16T12:23:22.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.966+0000 7fa816740700 1 -- 172.21.15.16:0/1612141545 wait complete. 2023-10-16T12:23:22.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.967+0000 7fa816740700 1 Processor -- start 2023-10-16T12:23:22.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.967+0000 7fa816740700 1 -- start start 2023-10-16T12:23:22.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.968+0000 7fa816740700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa81006ad20 0x7fa8100b5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:22.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.968+0000 7fa816740700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa81006da30 0x7fa81014da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:22.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.968+0000 7fa816740700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa810160fe0 0x7fa8100b3150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:22.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.968+0000 7fa816740700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa81014df40 con 0x7fa810160fe0 2023-10-16T12:23:22.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.968+0000 7fa816740700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa81014b070 con 0x7fa81006ad20 2023-10-16T12:23:22.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.968+0000 7fa816740700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8100c5670 con 0x7fa81006da30 2023-10-16T12:23:22.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.968+0000 7fa814cdd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa810160fe0 0x7fa8100b3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:22.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.968+0000 7fa814cdd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa810160fe0 0x7fa8100b3150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36598/0 (socket says 172.21.15.16:36598) 2023-10-16T12:23:22.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.968+0000 7fa814cdd700 1 -- 172.21.15.16:0/2147680628 learned_addr learned my addr 172.21.15.16:0/2147680628 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:22.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.968+0000 7fa80ffff700 1 --2- 172.21.15.16:0/2147680628 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa81006ad20 0x7fa8100b5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:22.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.968+0000 7fa80f7fe700 1 --2- 172.21.15.16:0/2147680628 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa81006da30 0x7fa81014da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:22.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.968+0000 7fa814cdd700 1 -- 172.21.15.16:0/2147680628 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa81006da30 msgr2=0x7fa81014da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:22.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.968+0000 7fa814cdd700 1 --2- 172.21.15.16:0/2147680628 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa81006da30 0x7fa81014da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:22.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.968+0000 7fa814cdd700 1 -- 172.21.15.16:0/2147680628 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa81006ad20 msgr2=0x7fa8100b5b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:22.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.968+0000 7fa814cdd700 1 --2- 172.21.15.16:0/2147680628 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa81006ad20 0x7fa8100b5b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:22.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.968+0000 7fa814cdd700 1 -- 172.21.15.16:0/2147680628 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa800009500 con 0x7fa810160fe0 2023-10-16T12:23:22.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.968+0000 7fa814cdd700 1 --2- 172.21.15.16:0/2147680628 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa810160fe0 0x7fa8100b3150 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fa80800cb90 tx=0x7fa80800cea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:22.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.969+0000 7fa80cff9700 1 -- 172.21.15.16:0/2147680628 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa80800a9c0 con 0x7fa810160fe0 2023-10-16T12:23:22.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.969+0000 7fa80cff9700 1 -- 172.21.15.16:0/2147680628 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa80800ab20 con 0x7fa810160fe0 2023-10-16T12:23:22.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.969+0000 7fa816740700 1 -- 172.21.15.16:0/2147680628 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8100c4d40 con 0x7fa810160fe0 2023-10-16T12:23:22.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.969+0000 7fa80cff9700 1 -- 172.21.15.16:0/2147680628 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa808012420 con 0x7fa810160fe0 2023-10-16T12:23:22.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.969+0000 7fa816740700 1 -- 172.21.15.16:0/2147680628 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8100c5300 con 0x7fa810160fe0 2023-10-16T12:23:22.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.970+0000 7fa816740700 1 -- 172.21.15.16:0/2147680628 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8100c5800 con 0x7fa810160fe0 2023-10-16T12:23:22.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.970+0000 7fa80cff9700 1 -- 172.21.15.16:0/2147680628 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7fa808012580 con 0x7fa810160fe0 2023-10-16T12:23:22.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.970+0000 7fa80cff9700 1 --2- 172.21.15.16:0/2147680628 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fa7f006b100 0x7fa7f006d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:22.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.971+0000 7fa80cff9700 1 -- 172.21.15.16:0/2147680628 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7fa80808a0f0 con 0x7fa810160fe0 2023-10-16T12:23:22.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.971+0000 7fa80ffff700 1 --2- 172.21.15.16:0/2147680628 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fa7f006b100 0x7fa7f006d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:22.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.971+0000 7fa80ffff700 1 --2- 172.21.15.16:0/2147680628 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fa7f006b100 0x7fa7f006d5c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa800013040 tx=0x7fa800008070 comp rx=0 tx=0).ready entity=mgr.5499 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:22.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:22.975+0000 7fa80cff9700 1 -- 172.21.15.16:0/2147680628 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa808054d80 con 0x7fa810160fe0 2023-10-16T12:23:23.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.206+0000 7fa816740700 1 -- 172.21.15.16:0/2147680628 --> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir7", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fa8100ad1d0 con 0x7fa7f006b100 2023-10-16T12:23:23.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.277+0000 7fa80cff9700 1 -- 172.21.15.16:0/2147680628 <== mgr.5499 v2:172.21.15.16:6832/1707126831 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fa800006c20 con 0x7fa7f006b100 2023-10-16T12:23:23.280 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule deactivated for path /snap_test_dir1/dir7 2023-10-16T12:23:23.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.279+0000 7fa816740700 1 -- 172.21.15.16:0/2147680628 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fa7f006b100 msgr2=0x7fa7f006d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:23.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.279+0000 7fa816740700 1 --2- 172.21.15.16:0/2147680628 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fa7f006b100 0x7fa7f006d5c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa800013040 tx=0x7fa800008070 comp rx=0 tx=0).stop 2023-10-16T12:23:23.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.279+0000 7fa816740700 1 -- 172.21.15.16:0/2147680628 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa810160fe0 msgr2=0x7fa8100b3150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:23.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.279+0000 7fa816740700 1 --2- 172.21.15.16:0/2147680628 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa810160fe0 0x7fa8100b3150 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fa80800cb90 tx=0x7fa80800cea0 comp rx=0 tx=0).stop 2023-10-16T12:23:23.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.279+0000 7fa816740700 1 -- 172.21.15.16:0/2147680628 shutdown_connections 2023-10-16T12:23:23.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.279+0000 7fa816740700 1 --2- 172.21.15.16:0/2147680628 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fa7f006b100 0x7fa7f006d5c0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:23.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.279+0000 7fa816740700 1 --2- 172.21.15.16:0/2147680628 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa810160fe0 0x7fa8100b3150 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:23.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.279+0000 7fa816740700 1 --2- 172.21.15.16:0/2147680628 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa81006ad20 0x7fa8100b5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:23.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.279+0000 7fa816740700 1 --2- 172.21.15.16:0/2147680628 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa81006da30 0x7fa81014da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:23.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.279+0000 7fa816740700 1 -- 172.21.15.16:0/2147680628 >> 172.21.15.16:0/2147680628 conn(0x7fa8100b1ed0 msgr2=0x7fa8100af1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:23.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.279+0000 7fa816740700 1 -- 172.21.15.16:0/2147680628 shutdown_connections 2023-10-16T12:23:23.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.279+0000 7fa816740700 1 -- 172.21.15.16:0/2147680628 wait complete. 2023-10-16T12:23:23.294 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir7 2023-10-16T12:23:23.294 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir8 1m 2023-10-16T12:23:23.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.421+0000 7f6be41ac700 1 Processor -- start 2023-10-16T12:23:23.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.421+0000 7f6be41ac700 1 -- start start 2023-10-16T12:23:23.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.421+0000 7f6be41ac700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f6bdc15b070 0x7f6bdc15b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:23.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.421+0000 7f6be41ac700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6bdc0cb880 0x7f6bdc0cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:23.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.421+0000 7f6be41ac700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bdc0cef70 con 0x7f6bdc06da60 2023-10-16T12:23:23.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.421+0000 7f6be41ac700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bdc0ce360 con 0x7f6bdc15b070 2023-10-16T12:23:23.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.421+0000 7f6be41ac700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bdc0ce640 con 0x7f6bdc0cb880 2023-10-16T12:23:23.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.421+0000 7f6be1747700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6bdc0cb880 0x7f6bdc0cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:23.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.421+0000 7f6be1f48700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f6bdc15b070 0x7f6bdc15b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:23.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.421+0000 7f6be2749700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7f6bdc06da60 0x7f6bdc0cf400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:33062/0 (socket says 172.21.15.16:33062) 2023-10-16T12:23:23.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.421+0000 7f6be2749700 1 -- 172.21.15.16:0/360269939 learned_addr learned my addr 172.21.15.16:0/360269939 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:23:23.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.421+0000 7f6be1747700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6bdc0cb880 0x7f6bdc0cbd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:53094/0 (socket says 172.21.15.16:53094) 2023-10-16T12:23:23.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.421+0000 7f6be0f46700 1 -- 172.21.15.16:0/360269939 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2725013372 0 0) 0x7f6bdc0cef70 con 0x7f6bdc06da60 2023-10-16T12:23:23.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.421+0000 7f6be0f46700 1 -- 172.21.15.16:0/360269939 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bc80035b0 con 0x7f6bdc06da60 2023-10-16T12:23:23.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.422+0000 7f6be1747700 1 -- 172.21.15.16:0/360269939 >> v2:172.21.15.49:3300/0 conn(0x7f6bdc15b070 msgr2=0x7f6bdc15b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:23.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.422+0000 7f6be1747700 1 --2- 172.21.15.16:0/360269939 >> v2:172.21.15.49:3300/0 conn(0x7f6bdc15b070 0x7f6bdc15b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:23.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.422+0000 7f6be1747700 1 -- 172.21.15.16:0/360269939 >> v1:172.21.15.49:6789/0 conn(0x7f6bdc06da60 legacy=0x7f6bdc0cf400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:23.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.422+0000 7f6be1747700 1 -- 172.21.15.16:0/360269939 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bdc0ce920 con 0x7f6bdc0cb880 2023-10-16T12:23:23.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.422+0000 7f6be1f48700 1 --2- 172.21.15.16:0/360269939 >> v2:172.21.15.49:3300/0 conn(0x7f6bdc15b070 0x7f6bdc15b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:23:23.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.422+0000 7f6be1747700 1 --2- 172.21.15.16:0/360269939 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6bdc0cb880 0x7f6bdc0cbd00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc009840 tx=0x7f6bcc009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f758d126f2b73a20 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:23.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.422+0000 7f6be0f46700 1 -- 172.21.15.16:0/360269939 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bcc011830 con 0x7f6bdc0cb880 2023-10-16T12:23:23.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.422+0000 7f6be0f46700 1 -- 172.21.15.16:0/360269939 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bcc011990 con 0x7f6bdc0cb880 2023-10-16T12:23:23.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.422+0000 7f6be0f46700 1 -- 172.21.15.16:0/360269939 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bcc011d20 con 0x7f6bdc0cb880 2023-10-16T12:23:23.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.422+0000 7f6be41ac700 1 -- 172.21.15.16:0/360269939 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6bdc0cb880 msgr2=0x7f6bdc0cbd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:23.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.422+0000 7f6be41ac700 1 --2- 172.21.15.16:0/360269939 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6bdc0cb880 0x7f6bdc0cbd00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc009840 tx=0x7f6bcc009b50 comp rx=0 tx=0).stop 2023-10-16T12:23:23.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.423+0000 7f6be41ac700 1 -- 172.21.15.16:0/360269939 shutdown_connections 2023-10-16T12:23:23.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.423+0000 7f6be41ac700 1 --2- 172.21.15.16:0/360269939 >> v2:172.21.15.49:3300/0 conn(0x7f6bdc15b070 0x7f6bdc15b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:23.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.423+0000 7f6be41ac700 1 --2- 172.21.15.16:0/360269939 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6bdc0cb880 0x7f6bdc0cbd00 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:23.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.423+0000 7f6be41ac700 1 -- 172.21.15.16:0/360269939 >> 172.21.15.16:0/360269939 conn(0x7f6bdc0b1f40 msgr2=0x7f6bdc0b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:23.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.423+0000 7f6be41ac700 1 -- 172.21.15.16:0/360269939 shutdown_connections 2023-10-16T12:23:23.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.423+0000 7f6be41ac700 1 -- 172.21.15.16:0/360269939 wait complete. 2023-10-16T12:23:23.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.424+0000 7f6be41ac700 1 Processor -- start 2023-10-16T12:23:23.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.424+0000 7f6be41ac700 1 -- start start 2023-10-16T12:23:23.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.426+0000 7f6be41ac700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6bdc06da60 0x7f6bdc1f4040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:23.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.426+0000 7f6be41ac700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bdc0cb880 0x7f6bdc155f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:23.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.426+0000 7f6be41ac700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6bdc15b070 0x7f6bdc158d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:23.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.426+0000 7f6be41ac700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bdc155b70 con 0x7f6bdc0cb880 2023-10-16T12:23:23.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.426+0000 7f6be41ac700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bdc1f4f00 con 0x7f6bdc15b070 2023-10-16T12:23:23.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.426+0000 7f6be41ac700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bdc1f4940 con 0x7f6bdc06da60 2023-10-16T12:23:23.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.426+0000 7f6be1747700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bdc0cb880 0x7f6bdc155f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:23.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.426+0000 7f6be1747700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bdc0cb880 0x7f6bdc155f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36610/0 (socket says 172.21.15.16:36610) 2023-10-16T12:23:23.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.426+0000 7f6be1747700 1 -- 172.21.15.16:0/2272351766 learned_addr learned my addr 172.21.15.16:0/2272351766 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:23.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.426+0000 7f6be1f48700 1 --2- 172.21.15.16:0/2272351766 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6bdc06da60 0x7f6bdc1f4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:23.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.426+0000 7f6be2749700 1 --2- 172.21.15.16:0/2272351766 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6bdc15b070 0x7f6bdc158d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:23.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.426+0000 7f6be1747700 1 -- 172.21.15.16:0/2272351766 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6bdc06da60 msgr2=0x7f6bdc1f4040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:23.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.426+0000 7f6be1747700 1 --2- 172.21.15.16:0/2272351766 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6bdc06da60 0x7f6bdc1f4040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:23.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.426+0000 7f6be1747700 1 -- 172.21.15.16:0/2272351766 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6bdc15b070 msgr2=0x7f6bdc158d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:23.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.426+0000 7f6be1747700 1 --2- 172.21.15.16:0/2272351766 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6bdc15b070 0x7f6bdc158d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:23.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.426+0000 7f6be1747700 1 -- 172.21.15.16:0/2272351766 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bd80094a0 con 0x7f6bdc0cb880 2023-10-16T12:23:23.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.426+0000 7f6be1747700 1 --2- 172.21.15.16:0/2272351766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bdc0cb880 0x7f6bdc155f90 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc0090b0 tx=0x7f6bcc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:23.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.426+0000 7f6bd27fc700 1 -- 172.21.15.16:0/2272351766 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bcc004590 con 0x7f6bdc0cb880 2023-10-16T12:23:23.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.426+0000 7f6bd27fc700 1 -- 172.21.15.16:0/2272351766 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bcc01f070 con 0x7f6bdc0cb880 2023-10-16T12:23:23.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.426+0000 7f6be41ac700 1 -- 172.21.15.16:0/2272351766 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bcc0094a0 con 0x7f6bdc0cb880 2023-10-16T12:23:23.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.426+0000 7f6bd27fc700 1 -- 172.21.15.16:0/2272351766 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bcc0038e0 con 0x7f6bdc0cb880 2023-10-16T12:23:23.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.426+0000 7f6be41ac700 1 -- 172.21.15.16:0/2272351766 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bdc155790 con 0x7f6bdc0cb880 2023-10-16T12:23:23.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.426+0000 7f6be41ac700 1 -- 172.21.15.16:0/2272351766 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bdc155cb0 con 0x7f6bdc0cb880 2023-10-16T12:23:23.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.427+0000 7f6bd27fc700 1 -- 172.21.15.16:0/2272351766 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f6bcc003a40 con 0x7f6bdc0cb880 2023-10-16T12:23:23.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.427+0000 7f6bd27fc700 1 --2- 172.21.15.16:0/2272351766 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f6bc006b100 0x7f6bc006d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:23.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.427+0000 7f6bd27fc700 1 -- 172.21.15.16:0/2272351766 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f6bcc021070 con 0x7f6bdc0cb880 2023-10-16T12:23:23.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.427+0000 7f6be1f48700 1 --2- 172.21.15.16:0/2272351766 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f6bc006b100 0x7f6bc006d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:23.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.428+0000 7f6be1f48700 1 --2- 172.21.15.16:0/2272351766 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f6bc006b100 0x7f6bc006d5c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6bdc159b90 tx=0x7f6bd8009210 comp rx=0 tx=0).ready entity=mgr.5499 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:23.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.433+0000 7f6bd27fc700 1 -- 172.21.15.16:0/2272351766 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6bdc155cb0 con 0x7f6bdc0cb880 2023-10-16T12:23:23.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.664+0000 7f6be41ac700 1 -- 172.21.15.16:0/2272351766 --> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir8", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f6bdc0728d0 con 0x7f6bc006b100 2023-10-16T12:23:23.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.732+0000 7f6bd27fc700 1 -- 172.21.15.16:0/2272351766 <== mgr.5499 v2:172.21.15.16:6832/1707126831 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f6bd8009de0 con 0x7f6bc006b100 2023-10-16T12:23:23.735 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule deactivated for path /snap_test_dir1/dir8 2023-10-16T12:23:23.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.734+0000 7f6be41ac700 1 -- 172.21.15.16:0/2272351766 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f6bc006b100 msgr2=0x7f6bc006d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:23.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.734+0000 7f6be41ac700 1 --2- 172.21.15.16:0/2272351766 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f6bc006b100 0x7f6bc006d5c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6bdc159b90 tx=0x7f6bd8009210 comp rx=0 tx=0).stop 2023-10-16T12:23:23.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.734+0000 7f6be41ac700 1 -- 172.21.15.16:0/2272351766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bdc0cb880 msgr2=0x7f6bdc155f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:23.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.734+0000 7f6be41ac700 1 --2- 172.21.15.16:0/2272351766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bdc0cb880 0x7f6bdc155f90 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc0090b0 tx=0x7f6bcc004750 comp rx=0 tx=0).stop 2023-10-16T12:23:23.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.734+0000 7f6be41ac700 1 -- 172.21.15.16:0/2272351766 shutdown_connections 2023-10-16T12:23:23.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.734+0000 7f6be41ac700 1 --2- 172.21.15.16:0/2272351766 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f6bc006b100 0x7f6bc006d5c0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:23.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.734+0000 7f6be41ac700 1 --2- 172.21.15.16:0/2272351766 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6bdc15b070 0x7f6bdc158d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:23.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.734+0000 7f6be41ac700 1 --2- 172.21.15.16:0/2272351766 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6bdc06da60 0x7f6bdc1f4040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:23.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.734+0000 7f6be41ac700 1 --2- 172.21.15.16:0/2272351766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bdc0cb880 0x7f6bdc155f90 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:23.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.734+0000 7f6be41ac700 1 -- 172.21.15.16:0/2272351766 >> 172.21.15.16:0/2272351766 conn(0x7f6bdc0b1f40 msgr2=0x7f6bdc06def0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:23.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.735+0000 7f6be41ac700 1 -- 172.21.15.16:0/2272351766 shutdown_connections 2023-10-16T12:23:23.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.735+0000 7f6be41ac700 1 -- 172.21.15.16:0/2272351766 wait complete. 2023-10-16T12:23:23.749 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir8 2023-10-16T12:23:23.750 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir9 1m 2023-10-16T12:23:23.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.875+0000 7feab3bca700 1 Processor -- start 2023-10-16T12:23:23.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.875+0000 7feab3bca700 1 -- start start 2023-10-16T12:23:23.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.876+0000 7feab3bca700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7feaac15b070 0x7feaac15b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:23.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.876+0000 7feab3bca700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaac0cef70 con 0x7feaac06e170 2023-10-16T12:23:23.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.876+0000 7feab3bca700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaac0ce360 con 0x7feaac0cb880 2023-10-16T12:23:23.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.876+0000 7feab3bca700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feaac0ce640 con 0x7feaac15b070 2023-10-16T12:23:23.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.876+0000 7feab2167700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7feaac06e170 0x7feaac0cf400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54126/0 (socket says 172.21.15.16:54126) 2023-10-16T12:23:23.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.876+0000 7feab2167700 1 -- 172.21.15.16:0/3615920664 learned_addr learned my addr 172.21.15.16:0/3615920664 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:23:23.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.876+0000 7feab1966700 1 --2- 172.21.15.16:0/3615920664 >> v2:172.21.15.49:3300/0 conn(0x7feaac15b070 0x7feaac15b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:23.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.877+0000 7feab0964700 1 -- 172.21.15.16:0/3615920664 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2413575886 0 0) 0x7feaac0cef70 con 0x7feaac06e170 2023-10-16T12:23:23.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.877+0000 7feab0964700 1 -- 172.21.15.16:0/3615920664 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea980035b0 con 0x7feaac06e170 2023-10-16T12:23:23.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.877+0000 7feab1966700 1 -- 172.21.15.16:0/3615920664 >> v1:172.21.15.49:6789/0 conn(0x7feaac0cb880 legacy=0x7feaac06da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:23.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.877+0000 7feab1966700 1 -- 172.21.15.16:0/3615920664 >> v1:172.21.15.16:6789/0 conn(0x7feaac06e170 legacy=0x7feaac0cf400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:23.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.877+0000 7feab1966700 1 -- 172.21.15.16:0/3615920664 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaac0ce920 con 0x7feaac15b070 2023-10-16T12:23:23.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.877+0000 7feab1966700 1 --2- 172.21.15.16:0/3615920664 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feaac15b070 0x7feaac15b450 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7feaa8009840 tx=0x7feaa8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf4601e99d3a1d08 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:23.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.877+0000 7feab0964700 1 -- 172.21.15.16:0/3615920664 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaa8011830 con 0x7feaac15b070 2023-10-16T12:23:23.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.877+0000 7feab0964700 1 -- 172.21.15.16:0/3615920664 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaa8011990 con 0x7feaac15b070 2023-10-16T12:23:23.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.877+0000 7feab0964700 1 -- 172.21.15.16:0/3615920664 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaa8011d00 con 0x7feaac15b070 2023-10-16T12:23:23.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.878+0000 7feab3bca700 1 -- 172.21.15.16:0/3615920664 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feaac15b070 msgr2=0x7feaac15b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:23.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.878+0000 7feab3bca700 1 --2- 172.21.15.16:0/3615920664 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feaac15b070 0x7feaac15b450 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7feaa8009840 tx=0x7feaa8009b50 comp rx=0 tx=0).stop 2023-10-16T12:23:23.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.878+0000 7feab3bca700 1 -- 172.21.15.16:0/3615920664 shutdown_connections 2023-10-16T12:23:23.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.878+0000 7feab3bca700 1 --2- 172.21.15.16:0/3615920664 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feaac15b070 0x7feaac15b450 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:23.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.878+0000 7feab3bca700 1 -- 172.21.15.16:0/3615920664 >> 172.21.15.16:0/3615920664 conn(0x7feaac0b1f40 msgr2=0x7feaac0b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:23.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.878+0000 7feab3bca700 1 -- 172.21.15.16:0/3615920664 shutdown_connections 2023-10-16T12:23:23.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.878+0000 7feab3bca700 1 -- 172.21.15.16:0/3615920664 wait complete. 2023-10-16T12:23:23.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.880+0000 7feab3bca700 1 Processor -- start 2023-10-16T12:23:23.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.880+0000 7feab3bca700 1 -- start start 2023-10-16T12:23:23.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.881+0000 7feab3bca700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7feaac06e170 0x7feaac0cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:23.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.881+0000 7feab3bca700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feaac0cb880 0x7feaac154a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:23.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.881+0000 7feab3bca700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feaac15b070 0x7feaac0b5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:23.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.881+0000 7feab3bca700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaac155190 con 0x7feaac0cb880 2023-10-16T12:23:23.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.881+0000 7feab3bca700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaac0cc2e0 con 0x7feaac15b070 2023-10-16T12:23:23.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.881+0000 7feab3bca700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feaac0cc5c0 con 0x7feaac06e170 2023-10-16T12:23:23.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.881+0000 7feab1165700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feaac0cb880 0x7feaac154a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:23.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.881+0000 7feab1165700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feaac0cb880 0x7feaac154a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36624/0 (socket says 172.21.15.16:36624) 2023-10-16T12:23:23.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.881+0000 7feab1966700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7feaac06e170 0x7feaac0cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:23.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.881+0000 7feab1165700 1 -- 172.21.15.16:0/2858251764 learned_addr learned my addr 172.21.15.16:0/2858251764 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:23.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.881+0000 7feab2167700 1 --2- 172.21.15.16:0/2858251764 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feaac15b070 0x7feaac0b5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:23.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.881+0000 7feab1165700 1 -- 172.21.15.16:0/2858251764 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7feaac06e170 msgr2=0x7feaac0cbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:23.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.881+0000 7feab1165700 1 --2- 172.21.15.16:0/2858251764 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7feaac06e170 0x7feaac0cbd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:23.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.881+0000 7feab1165700 1 -- 172.21.15.16:0/2858251764 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feaac15b070 msgr2=0x7feaac0b5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:23.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.881+0000 7feab1165700 1 --2- 172.21.15.16:0/2858251764 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feaac15b070 0x7feaac0b5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:23.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.881+0000 7feab1165700 1 -- 172.21.15.16:0/2858251764 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaa80094a0 con 0x7feaac0cb880 2023-10-16T12:23:23.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.881+0000 7feab1165700 1 --2- 172.21.15.16:0/2858251764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feaac0cb880 0x7feaac154a60 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fea9c00c4e0 tx=0x7fea9c00c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:23.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.881+0000 7feaa27fc700 1 -- 172.21.15.16:0/2858251764 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea9c0052d0 con 0x7feaac0cb880 2023-10-16T12:23:23.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.881+0000 7feaa27fc700 1 -- 172.21.15.16:0/2858251764 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea9c005430 con 0x7feaac0cb880 2023-10-16T12:23:23.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.881+0000 7feab3bca700 1 -- 172.21.15.16:0/2858251764 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feaac0cc8a0 con 0x7feaac0cb880 2023-10-16T12:23:23.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.881+0000 7feaa27fc700 1 -- 172.21.15.16:0/2858251764 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea9c0038c0 con 0x7feaac0cb880 2023-10-16T12:23:23.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.881+0000 7feab3bca700 1 -- 172.21.15.16:0/2858251764 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feaac0cce60 con 0x7feaac0cb880 2023-10-16T12:23:23.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.882+0000 7feab3bca700 1 -- 172.21.15.16:0/2858251764 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feaac04ebb0 con 0x7feaac0cb880 2023-10-16T12:23:23.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.882+0000 7feaa27fc700 1 -- 172.21.15.16:0/2858251764 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7fea9c003a20 con 0x7feaac0cb880 2023-10-16T12:23:23.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.882+0000 7feaa27fc700 1 --2- 172.21.15.16:0/2858251764 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fea9006b100 0x7fea9006d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:23.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.883+0000 7feaa27fc700 1 -- 172.21.15.16:0/2858251764 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7fea9c088ff0 con 0x7feaac0cb880 2023-10-16T12:23:23.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.883+0000 7feab1966700 1 --2- 172.21.15.16:0/2858251764 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fea9006b100 0x7fea9006d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:23.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.883+0000 7feab1966700 1 --2- 172.21.15.16:0/2858251764 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fea9006b100 0x7fea9006d5c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7feaa8009fd0 tx=0x7feaa801c040 comp rx=0 tx=0).ready entity=mgr.5499 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:23.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:23.887+0000 7feaa27fc700 1 -- 172.21.15.16:0/2858251764 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fea9c053b20 con 0x7feaac0cb880 2023-10-16T12:23:24.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.123+0000 7feab3bca700 1 -- 172.21.15.16:0/2858251764 --> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir9", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7feaac0728d0 con 0x7fea9006b100 2023-10-16T12:23:24.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.191+0000 7feaa27fc700 1 -- 172.21.15.16:0/2858251764 <== mgr.5499 v2:172.21.15.16:6832/1707126831 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7feaa8011730 con 0x7fea9006b100 2023-10-16T12:23:24.194 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule deactivated for path /snap_test_dir1/dir9 2023-10-16T12:23:24.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.193+0000 7feab3bca700 1 -- 172.21.15.16:0/2858251764 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fea9006b100 msgr2=0x7fea9006d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:24.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.193+0000 7feab3bca700 1 --2- 172.21.15.16:0/2858251764 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fea9006b100 0x7fea9006d5c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7feaa8009fd0 tx=0x7feaa801c040 comp rx=0 tx=0).stop 2023-10-16T12:23:24.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.193+0000 7feab3bca700 1 -- 172.21.15.16:0/2858251764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feaac0cb880 msgr2=0x7feaac154a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:24.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.193+0000 7feab3bca700 1 --2- 172.21.15.16:0/2858251764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feaac0cb880 0x7feaac154a60 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fea9c00c4e0 tx=0x7fea9c00c7f0 comp rx=0 tx=0).stop 2023-10-16T12:23:24.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.193+0000 7feab3bca700 1 -- 172.21.15.16:0/2858251764 shutdown_connections 2023-10-16T12:23:24.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.193+0000 7feab3bca700 1 --2- 172.21.15.16:0/2858251764 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fea9006b100 0x7fea9006d5c0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:24.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.193+0000 7feab3bca700 1 --2- 172.21.15.16:0/2858251764 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feaac15b070 0x7feaac0b5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:24.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.193+0000 7feab3bca700 1 --2- 172.21.15.16:0/2858251764 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7feaac06e170 0x7feaac0cbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:24.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.193+0000 7feab3bca700 1 --2- 172.21.15.16:0/2858251764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feaac0cb880 0x7feaac154a60 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:24.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.193+0000 7feab3bca700 1 -- 172.21.15.16:0/2858251764 >> 172.21.15.16:0/2858251764 conn(0x7feaac0b1f40 msgr2=0x7feaac0b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:24.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.193+0000 7feab3bca700 1 -- 172.21.15.16:0/2858251764 shutdown_connections 2023-10-16T12:23:24.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.193+0000 7feab3bca700 1 -- 172.21.15.16:0/2858251764 wait complete. 2023-10-16T12:23:24.207 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir9 2023-10-16T12:23:24.208 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:24.208 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir0/.snap) 2023-10-16T12:23:24.227 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_13_00_UTC 2023-10-16T12:23:24.227 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_14_00_UTC 2023-10-16T12:23:24.227 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_15_00_UTC 2023-10-16T12:23:24.228 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_16_00_UTC 2023-10-16T12:23:24.228 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_17_00_UTC 2023-10-16T12:23:24.228 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_18_00_UTC 2023-10-16T12:23:24.228 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_19_00_UTC 2023-10-16T12:23:24.228 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_20_00_UTC 2023-10-16T12:23:24.228 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_21_00_UTC 2023-10-16T12:23:24.229 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_22_00_UTC 2023-10-16T12:23:24.229 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_23_00_UTC 2023-10-16T12:23:24.230 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-16-12_13_00_UTC', 'scheduled-2023-10-16-12_14_00_UTC', 'scheduled-2023-10-16-12_15_00_UTC', 'scheduled-2023-10-16-12_16_00_UTC', 'scheduled-2023-10-16-12_17_00_UTC', 'scheduled-2023-10-16-12_18_00_UTC', 'scheduled-2023-10-16-12_19_00_UTC', 'scheduled-2023-10-16-12_20_00_UTC', 'scheduled-2023-10-16-12_21_00_UTC', 'scheduled-2023-10-16-12_22_00_UTC', 'scheduled-2023-10-16-12_23_00_UTC'] 2023-10-16T12:23:24.230 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir0 2023-10-16T12:23:24.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.395+0000 7faf3d742700 1 Processor -- start 2023-10-16T12:23:24.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.395+0000 7faf3d742700 1 -- start start 2023-10-16T12:23:24.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.396+0000 7faf3d742700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7faf38071ad0 0x7faf380b0db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:24.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.396+0000 7faf3d742700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7faf380b12f0 0x7faf380b24c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:24.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.396+0000 7faf3d742700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf3815bab0 con 0x7faf3815afc0 2023-10-16T12:23:24.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.396+0000 7faf3d742700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf3815f080 con 0x7faf380b12f0 2023-10-16T12:23:24.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.396+0000 7faf3d742700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf380b1720 con 0x7faf38071ad0 2023-10-16T12:23:24.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.396+0000 7faf377fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7faf380b12f0 0x7faf380b24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:24.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.396+0000 7faf377fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7faf380b12f0 0x7faf380b24c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36634/0 (socket says 172.21.15.16:36634) 2023-10-16T12:23:24.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.396+0000 7faf377fe700 1 -- 172.21.15.16:0/1489511372 learned_addr learned my addr 172.21.15.16:0/1489511372 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:24.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.396+0000 7faf367fc700 1 --2- 172.21.15.16:0/1489511372 >> v2:172.21.15.49:3300/0 conn(0x7faf38071ad0 0x7faf380b0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:24.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.396+0000 7faf377fe700 1 -- 172.21.15.16:0/1489511372 >> v2:172.21.15.49:3300/0 conn(0x7faf38071ad0 msgr2=0x7faf380b0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:24.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.396+0000 7faf377fe700 1 --2- 172.21.15.16:0/1489511372 >> v2:172.21.15.49:3300/0 conn(0x7faf38071ad0 0x7faf380b0db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:24.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.396+0000 7faf377fe700 1 -- 172.21.15.16:0/1489511372 >> v1:172.21.15.49:6789/0 conn(0x7faf3815afc0 legacy=0x7faf3815b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:24.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.396+0000 7faf377fe700 1 -- 172.21.15.16:0/1489511372 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf3815e500 con 0x7faf380b12f0 2023-10-16T12:23:24.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.396+0000 7faf367fc700 1 --2- 172.21.15.16:0/1489511372 >> v2:172.21.15.49:3300/0 conn(0x7faf38071ad0 0x7faf380b0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:23:24.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.397+0000 7faf377fe700 1 --2- 172.21.15.16:0/1489511372 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf380b12f0 0x7faf380b24c0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7faf380cb890 tx=0x7faf3000d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40ab5a7b60d6617c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:24.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.397+0000 7faf35ffb700 1 -- 172.21.15.16:0/1489511372 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf30010070 con 0x7faf380b12f0 2023-10-16T12:23:24.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.397+0000 7faf35ffb700 1 -- 172.21.15.16:0/1489511372 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf3000dd50 con 0x7faf380b12f0 2023-10-16T12:23:24.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.397+0000 7faf35ffb700 1 -- 172.21.15.16:0/1489511372 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf3000b310 con 0x7faf380b12f0 2023-10-16T12:23:24.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.397+0000 7faf3d742700 1 -- 172.21.15.16:0/1489511372 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf380b12f0 msgr2=0x7faf380b24c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:24.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.397+0000 7faf3d742700 1 --2- 172.21.15.16:0/1489511372 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf380b12f0 0x7faf380b24c0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7faf380cb890 tx=0x7faf3000d430 comp rx=0 tx=0).stop 2023-10-16T12:23:24.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.397+0000 7faf3d742700 1 -- 172.21.15.16:0/1489511372 shutdown_connections 2023-10-16T12:23:24.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.397+0000 7faf3d742700 1 --2- 172.21.15.16:0/1489511372 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf380b12f0 0x7faf380b24c0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:24.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.397+0000 7faf3d742700 1 --2- 172.21.15.16:0/1489511372 >> v2:172.21.15.49:3300/0 conn(0x7faf38071ad0 0x7faf380b0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:24.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.397+0000 7faf3d742700 1 -- 172.21.15.16:0/1489511372 >> 172.21.15.16:0/1489511372 conn(0x7faf3806a7e0 msgr2=0x7faf3806abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:24.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.397+0000 7faf3d742700 1 -- 172.21.15.16:0/1489511372 shutdown_connections 2023-10-16T12:23:24.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.397+0000 7faf3d742700 1 -- 172.21.15.16:0/1489511372 wait complete. 2023-10-16T12:23:24.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.399+0000 7faf3d742700 1 Processor -- start 2023-10-16T12:23:24.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.399+0000 7faf3d742700 1 -- start start 2023-10-16T12:23:24.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.399+0000 7faf3d742700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faf38071ad0 0x7faf381531a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:24.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.399+0000 7faf3d742700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf380b12f0 0x7faf381536e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:24.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.399+0000 7faf3d742700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faf3815afc0 0x7faf38155350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:24.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.399+0000 7faf3d742700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf381560a0 con 0x7faf380b12f0 2023-10-16T12:23:24.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.399+0000 7faf3d742700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf38156380 con 0x7faf3815afc0 2023-10-16T12:23:24.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.399+0000 7faf3d742700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf38156660 con 0x7faf38071ad0 2023-10-16T12:23:24.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.400+0000 7faf367fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf380b12f0 0x7faf381536e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:24.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.400+0000 7faf367fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf380b12f0 0x7faf381536e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36638/0 (socket says 172.21.15.16:36638) 2023-10-16T12:23:24.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.400+0000 7faf367fc700 1 -- 172.21.15.16:0/1903551040 learned_addr learned my addr 172.21.15.16:0/1903551040 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:24.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.400+0000 7faf377fe700 1 --2- 172.21.15.16:0/1903551040 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faf3815afc0 0x7faf38155350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:24.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.400+0000 7faf36ffd700 1 --2- 172.21.15.16:0/1903551040 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faf38071ad0 0x7faf381531a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:24.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.400+0000 7faf367fc700 1 -- 172.21.15.16:0/1903551040 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faf38071ad0 msgr2=0x7faf381531a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:24.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.400+0000 7faf367fc700 1 --2- 172.21.15.16:0/1903551040 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faf38071ad0 0x7faf381531a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:24.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.400+0000 7faf367fc700 1 -- 172.21.15.16:0/1903551040 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faf3815afc0 msgr2=0x7faf38155350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:24.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.400+0000 7faf367fc700 1 --2- 172.21.15.16:0/1903551040 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faf3815afc0 0x7faf38155350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:24.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.400+0000 7faf367fc700 1 -- 172.21.15.16:0/1903551040 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf3000bd40 con 0x7faf380b12f0 2023-10-16T12:23:24.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.400+0000 7faf377fe700 1 --2- 172.21.15.16:0/1903551040 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faf3815afc0 0x7faf38155350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:23:24.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.400+0000 7faf367fc700 1 --2- 172.21.15.16:0/1903551040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf380b12f0 0x7faf381536e0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7faf2c00c750 tx=0x7faf2c00cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:24.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.400+0000 7faf1f7fe700 1 -- 172.21.15.16:0/1903551040 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf2c011800 con 0x7faf380b12f0 2023-10-16T12:23:24.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.400+0000 7faf1f7fe700 1 -- 172.21.15.16:0/1903551040 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf2c011960 con 0x7faf380b12f0 2023-10-16T12:23:24.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.400+0000 7faf1f7fe700 1 -- 172.21.15.16:0/1903551040 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf2c011cd0 con 0x7faf380b12f0 2023-10-16T12:23:24.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.400+0000 7faf3d742700 1 -- 172.21.15.16:0/1903551040 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf38156940 con 0x7faf380b12f0 2023-10-16T12:23:24.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.400+0000 7faf3d742700 1 -- 172.21.15.16:0/1903551040 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf38140f40 con 0x7faf380b12f0 2023-10-16T12:23:24.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.401+0000 7faf3d742700 1 -- 172.21.15.16:0/1903551040 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf3815c990 con 0x7faf380b12f0 2023-10-16T12:23:24.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.402+0000 7faf1f7fe700 1 -- 172.21.15.16:0/1903551040 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7faf2c021350 con 0x7faf380b12f0 2023-10-16T12:23:24.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.402+0000 7faf1f7fe700 1 --2- 172.21.15.16:0/1903551040 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7faf2406b100 0x7faf2406d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:24.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.402+0000 7faf1f7fe700 1 -- 172.21.15.16:0/1903551040 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7faf2c089c50 con 0x7faf380b12f0 2023-10-16T12:23:24.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.402+0000 7faf36ffd700 1 --2- 172.21.15.16:0/1903551040 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7faf2406b100 0x7faf2406d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:24.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.403+0000 7faf36ffd700 1 --2- 172.21.15.16:0/1903551040 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7faf2406b100 0x7faf2406d5c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7faf28009870 tx=0x7faf28008040 comp rx=0 tx=0).ready entity=mgr.5499 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:24.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.407+0000 7faf1f7fe700 1 -- 172.21.15.16:0/1903551040 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7faf2c08f020 con 0x7faf380b12f0 2023-10-16T12:23:24.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.633+0000 7faf3d742700 1 -- 172.21.15.16:0/1903551040 --> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir0", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7faf3815eac0 con 0x7faf2406b100 2023-10-16T12:23:24.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.642+0000 7faf1f7fe700 1 -- 172.21.15.16:0/1903551040 <== mgr.5499 v2:172.21.15.16:6832/1707126831 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7faf28009460 con 0x7faf2406b100 2023-10-16T12:23:24.644 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:23:24.644 INFO:teuthology.orchestra.run.smithi016.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir0", "rel_path": "/snap_test_dir1/dir0", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T12:12:16", "first": "2023-10-16T12:13:00", "last": "2023-10-16T12:23:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-16T12:23:24.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.644+0000 7faf3d742700 1 -- 172.21.15.16:0/1903551040 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7faf2406b100 msgr2=0x7faf2406d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:24.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.644+0000 7faf3d742700 1 --2- 172.21.15.16:0/1903551040 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7faf2406b100 0x7faf2406d5c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7faf28009870 tx=0x7faf28008040 comp rx=0 tx=0).stop 2023-10-16T12:23:24.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.644+0000 7faf3d742700 1 -- 172.21.15.16:0/1903551040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf380b12f0 msgr2=0x7faf381536e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:24.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.644+0000 7faf3d742700 1 --2- 172.21.15.16:0/1903551040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf380b12f0 0x7faf381536e0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7faf2c00c750 tx=0x7faf2c00cb10 comp rx=0 tx=0).stop 2023-10-16T12:23:24.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.644+0000 7faf3d742700 1 -- 172.21.15.16:0/1903551040 shutdown_connections 2023-10-16T12:23:24.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.644+0000 7faf3d742700 1 --2- 172.21.15.16:0/1903551040 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7faf2406b100 0x7faf2406d5c0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:24.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.644+0000 7faf3d742700 1 --2- 172.21.15.16:0/1903551040 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faf3815afc0 0x7faf38155350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:24.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.644+0000 7faf3d742700 1 --2- 172.21.15.16:0/1903551040 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faf38071ad0 0x7faf381531a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:24.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.644+0000 7faf3d742700 1 --2- 172.21.15.16:0/1903551040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf380b12f0 0x7faf381536e0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:24.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.644+0000 7faf3d742700 1 -- 172.21.15.16:0/1903551040 >> 172.21.15.16:0/1903551040 conn(0x7faf3806a7e0 msgr2=0x7faf3806abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:24.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.644+0000 7faf3d742700 1 -- 172.21.15.16:0/1903551040 shutdown_connections 2023-10-16T12:23:24.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.644+0000 7faf3d742700 1 -- 172.21.15.16:0/1903551040 wait complete. 2023-10-16T12:23:24.658 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir0", "rel_path": "/snap_test_dir1/dir0", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T12:12:16", "first": "2023-10-16T12:13:00", "last": "2023-10-16T12:23:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-16T12:23:24.658 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir0', 'rel_path': '/snap_test_dir1/dir0', 'schedule': '1m', 'retention': {}, 'start': '2023-10-16T00:00:00', 'created': '2023-10-16T12:12:16', 'first': '2023-10-16T12:13:00', 'last': '2023-10-16T12:23:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-10-16T12:23:24.658 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-10-16T12:23:24.659 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-10-16T12:23:24.659 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:24.659 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir1/.snap) 2023-10-16T12:23:24.678 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_13_00_UTC 2023-10-16T12:23:24.678 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_14_00_UTC 2023-10-16T12:23:24.678 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_15_00_UTC 2023-10-16T12:23:24.678 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_16_00_UTC 2023-10-16T12:23:24.679 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_17_00_UTC 2023-10-16T12:23:24.679 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_18_00_UTC 2023-10-16T12:23:24.679 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_19_00_UTC 2023-10-16T12:23:24.679 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_20_00_UTC 2023-10-16T12:23:24.679 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_21_00_UTC 2023-10-16T12:23:24.679 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_22_00_UTC 2023-10-16T12:23:24.680 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_23_00_UTC 2023-10-16T12:23:24.680 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-16-12_13_00_UTC', 'scheduled-2023-10-16-12_14_00_UTC', 'scheduled-2023-10-16-12_15_00_UTC', 'scheduled-2023-10-16-12_16_00_UTC', 'scheduled-2023-10-16-12_17_00_UTC', 'scheduled-2023-10-16-12_18_00_UTC', 'scheduled-2023-10-16-12_19_00_UTC', 'scheduled-2023-10-16-12_20_00_UTC', 'scheduled-2023-10-16-12_21_00_UTC', 'scheduled-2023-10-16-12_22_00_UTC', 'scheduled-2023-10-16-12_23_00_UTC'] 2023-10-16T12:23:24.680 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir1 2023-10-16T12:23:24.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.842+0000 7f3f10113700 1 Processor -- start 2023-10-16T12:23:24.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.843+0000 7f3f10113700 1 -- start start 2023-10-16T12:23:24.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.843+0000 7f3f10113700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3f0815b2c0 0x7f3f08161340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:24.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.843+0000 7f3f10113700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f0815ecb0 con 0x7f3f0806a7e0 2023-10-16T12:23:24.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.843+0000 7f3f10113700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f08161880 con 0x7f3f0806e570 2023-10-16T12:23:24.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.843+0000 7f3f10113700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f0815b6a0 con 0x7f3f0815b2c0 2023-10-16T12:23:24.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.844+0000 7f3f0deaf700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3f0815b2c0 0x7f3f08161340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:24.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.844+0000 7f3f0deaf700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3f0815b2c0 0x7f3f08161340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36644/0 (socket says 172.21.15.16:36644) 2023-10-16T12:23:24.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.844+0000 7f3f0e6b0700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f3f0806a7e0 0x7f3f08071a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54154/0 (socket says 172.21.15.16:54154) 2023-10-16T12:23:24.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.844+0000 7f3f0deaf700 1 -- 172.21.15.16:0/2425421550 learned_addr learned my addr 172.21.15.16:0/2425421550 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:24.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.844+0000 7f3f0deaf700 1 -- 172.21.15.16:0/2425421550 >> v1:172.21.15.49:6789/0 conn(0x7f3f0806e570 legacy=0x7f3f0815e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:24.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.844+0000 7f3f0deaf700 1 -- 172.21.15.16:0/2425421550 >> v1:172.21.15.16:6789/0 conn(0x7f3f0806a7e0 legacy=0x7f3f08071a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:24.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.844+0000 7f3f0deaf700 1 -- 172.21.15.16:0/2425421550 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f08161b20 con 0x7f3f0815b2c0 2023-10-16T12:23:24.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.844+0000 7f3f0deaf700 1 --2- 172.21.15.16:0/2425421550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f0815b2c0 0x7f3f08161340 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f3f040098f0 tx=0x7f3f04009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bc2de999e3e392e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:24.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.844+0000 7f3f0cead700 1 -- 172.21.15.16:0/2425421550 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f04004230 con 0x7f3f0815b2c0 2023-10-16T12:23:24.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.844+0000 7f3f0cead700 1 -- 172.21.15.16:0/2425421550 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f04004390 con 0x7f3f0815b2c0 2023-10-16T12:23:24.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.844+0000 7f3f0cead700 1 -- 172.21.15.16:0/2425421550 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f040036a0 con 0x7f3f0815b2c0 2023-10-16T12:23:24.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.845+0000 7f3f10113700 1 -- 172.21.15.16:0/2425421550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f0815b2c0 msgr2=0x7f3f08161340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:24.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.845+0000 7f3f10113700 1 --2- 172.21.15.16:0/2425421550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f0815b2c0 0x7f3f08161340 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f3f040098f0 tx=0x7f3f04009c00 comp rx=0 tx=0).stop 2023-10-16T12:23:24.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.845+0000 7f3f10113700 1 -- 172.21.15.16:0/2425421550 shutdown_connections 2023-10-16T12:23:24.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.845+0000 7f3f10113700 1 --2- 172.21.15.16:0/2425421550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f0815b2c0 0x7f3f08161340 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:24.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.845+0000 7f3f10113700 1 -- 172.21.15.16:0/2425421550 >> 172.21.15.16:0/2425421550 conn(0x7f3f080b1f40 msgr2=0x7f3f080b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:24.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.845+0000 7f3f10113700 1 -- 172.21.15.16:0/2425421550 shutdown_connections 2023-10-16T12:23:24.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.845+0000 7f3f10113700 1 -- 172.21.15.16:0/2425421550 wait complete. 2023-10-16T12:23:24.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.846+0000 7f3f10113700 1 Processor -- start 2023-10-16T12:23:24.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.846+0000 7f3f10113700 1 -- start start 2023-10-16T12:23:24.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.847+0000 7f3f10113700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3f0806a7e0 0x7f3f08154bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:24.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.847+0000 7f3f10113700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f0806e570 0x7f3f08160960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:24.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.847+0000 7f3f10113700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3f0815b2c0 0x7f3f080c7f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:24.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.847+0000 7f3f10113700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f080ca760 con 0x7f3f0806e570 2023-10-16T12:23:24.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.847+0000 7f3f10113700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f081f4b90 con 0x7f3f0806a7e0 2023-10-16T12:23:24.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.847+0000 7f3f10113700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f080ca480 con 0x7f3f0815b2c0 2023-10-16T12:23:24.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.847+0000 7f3f0d6ae700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f0806e570 0x7f3f08160960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:24.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.847+0000 7f3f0d6ae700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f0806e570 0x7f3f08160960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36654/0 (socket says 172.21.15.16:36654) 2023-10-16T12:23:24.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.847+0000 7f3f0d6ae700 1 -- 172.21.15.16:0/3177655029 learned_addr learned my addr 172.21.15.16:0/3177655029 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:24.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.847+0000 7f3f0deaf700 1 --2- 172.21.15.16:0/3177655029 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3f0806a7e0 0x7f3f08154bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:24.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.847+0000 7f3f0e6b0700 1 --2- 172.21.15.16:0/3177655029 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3f0815b2c0 0x7f3f080c7f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:24.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.847+0000 7f3f0d6ae700 1 -- 172.21.15.16:0/3177655029 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3f0815b2c0 msgr2=0x7f3f080c7f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:24.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.847+0000 7f3f0d6ae700 1 --2- 172.21.15.16:0/3177655029 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3f0815b2c0 0x7f3f080c7f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:24.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.847+0000 7f3f0d6ae700 1 -- 172.21.15.16:0/3177655029 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3f0806a7e0 msgr2=0x7f3f08154bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:24.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.847+0000 7f3f0d6ae700 1 --2- 172.21.15.16:0/3177655029 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3f0806a7e0 0x7f3f08154bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:24.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.847+0000 7f3f0d6ae700 1 -- 172.21.15.16:0/3177655029 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f04009550 con 0x7f3f0806e570 2023-10-16T12:23:24.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.847+0000 7f3f0d6ae700 1 --2- 172.21.15.16:0/3177655029 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f0806e570 0x7f3f08160960 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f3efc00c500 tx=0x7f3efc00c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:24.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.848+0000 7f3efa7fc700 1 -- 172.21.15.16:0/3177655029 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3efc011790 con 0x7f3f0806e570 2023-10-16T12:23:24.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.848+0000 7f3efa7fc700 1 -- 172.21.15.16:0/3177655029 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3efc0118f0 con 0x7f3f0806e570 2023-10-16T12:23:24.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.848+0000 7f3efa7fc700 1 -- 172.21.15.16:0/3177655029 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3efc011ca0 con 0x7f3f0806e570 2023-10-16T12:23:24.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.848+0000 7f3f10113700 1 -- 172.21.15.16:0/3177655029 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f081f4540 con 0x7f3f0806e570 2023-10-16T12:23:24.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.848+0000 7f3f10113700 1 -- 172.21.15.16:0/3177655029 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f08160ea0 con 0x7f3f0806e570 2023-10-16T12:23:24.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.849+0000 7f3f10113700 1 -- 172.21.15.16:0/3177655029 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f080cc9f0 con 0x7f3f0806e570 2023-10-16T12:23:24.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.849+0000 7f3efa7fc700 1 -- 172.21.15.16:0/3177655029 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f3efc01c460 con 0x7f3f0806e570 2023-10-16T12:23:24.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.850+0000 7f3efa7fc700 1 --2- 172.21.15.16:0/3177655029 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f3eec06b100 0x7f3eec06d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:24.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.850+0000 7f3f0deaf700 1 --2- 172.21.15.16:0/3177655029 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f3eec06b100 0x7f3eec06d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:24.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.850+0000 7f3efa7fc700 1 -- 172.21.15.16:0/3177655029 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f3efc0898f0 con 0x7f3f0806e570 2023-10-16T12:23:24.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.850+0000 7f3f0deaf700 1 --2- 172.21.15.16:0/3177655029 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f3eec06b100 0x7f3eec06d5c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3f04009130 tx=0x7f3f04008030 comp rx=0 tx=0).ready entity=mgr.5499 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:24.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:24.854+0000 7f3efa7fc700 1 -- 172.21.15.16:0/3177655029 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3efc0544b0 con 0x7f3f0806e570 2023-10-16T12:23:25.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.087+0000 7f3f10113700 1 -- 172.21.15.16:0/3177655029 --> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3f081620e0 con 0x7f3eec06b100 2023-10-16T12:23:25.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.095+0000 7f3efa7fc700 1 -- 172.21.15.16:0/3177655029 <== mgr.5499 v2:172.21.15.16:6832/1707126831 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f3f04006d00 con 0x7f3eec06b100 2023-10-16T12:23:25.097 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:23:25.097 INFO:teuthology.orchestra.run.smithi016.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir1", "rel_path": "/snap_test_dir1/dir1", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T12:12:17", "first": "2023-10-16T12:13:00", "last": "2023-10-16T12:23:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-16T12:23:25.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.097+0000 7f3f10113700 1 -- 172.21.15.16:0/3177655029 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f3eec06b100 msgr2=0x7f3eec06d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:25.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.097+0000 7f3f10113700 1 --2- 172.21.15.16:0/3177655029 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f3eec06b100 0x7f3eec06d5c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3f04009130 tx=0x7f3f04008030 comp rx=0 tx=0).stop 2023-10-16T12:23:25.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.097+0000 7f3f10113700 1 -- 172.21.15.16:0/3177655029 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f0806e570 msgr2=0x7f3f08160960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:25.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.097+0000 7f3f10113700 1 --2- 172.21.15.16:0/3177655029 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f0806e570 0x7f3f08160960 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f3efc00c500 tx=0x7f3efc00c810 comp rx=0 tx=0).stop 2023-10-16T12:23:25.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.097+0000 7f3f10113700 1 -- 172.21.15.16:0/3177655029 shutdown_connections 2023-10-16T12:23:25.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.097+0000 7f3f10113700 1 --2- 172.21.15.16:0/3177655029 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f3eec06b100 0x7f3eec06d5c0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:25.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.097+0000 7f3f10113700 1 --2- 172.21.15.16:0/3177655029 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3f0815b2c0 0x7f3f080c7f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:25.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.097+0000 7f3f10113700 1 --2- 172.21.15.16:0/3177655029 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3f0806a7e0 0x7f3f08154bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:25.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.097+0000 7f3f10113700 1 --2- 172.21.15.16:0/3177655029 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f0806e570 0x7f3f08160960 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:25.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.097+0000 7f3f10113700 1 -- 172.21.15.16:0/3177655029 >> 172.21.15.16:0/3177655029 conn(0x7f3f080b1f40 msgr2=0x7f3f0815e530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:25.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.097+0000 7f3f10113700 1 -- 172.21.15.16:0/3177655029 shutdown_connections 2023-10-16T12:23:25.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.097+0000 7f3f10113700 1 -- 172.21.15.16:0/3177655029 wait complete. 2023-10-16T12:23:25.111 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir1", "rel_path": "/snap_test_dir1/dir1", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T12:12:17", "first": "2023-10-16T12:13:00", "last": "2023-10-16T12:23:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-16T12:23:25.111 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir1', 'rel_path': '/snap_test_dir1/dir1', 'schedule': '1m', 'retention': {}, 'start': '2023-10-16T00:00:00', 'created': '2023-10-16T12:12:17', 'first': '2023-10-16T12:13:00', 'last': '2023-10-16T12:23:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-10-16T12:23:25.112 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-10-16T12:23:25.112 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-10-16T12:23:25.112 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:25.113 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir2/.snap) 2023-10-16T12:23:25.130 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_13_00_UTC 2023-10-16T12:23:25.130 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_14_00_UTC 2023-10-16T12:23:25.131 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_15_00_UTC 2023-10-16T12:23:25.131 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_16_00_UTC 2023-10-16T12:23:25.131 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_17_00_UTC 2023-10-16T12:23:25.131 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_18_00_UTC 2023-10-16T12:23:25.131 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_19_00_UTC 2023-10-16T12:23:25.132 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_20_00_UTC 2023-10-16T12:23:25.132 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_21_00_UTC 2023-10-16T12:23:25.132 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_22_00_UTC 2023-10-16T12:23:25.132 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_23_00_UTC 2023-10-16T12:23:25.133 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-16-12_13_00_UTC', 'scheduled-2023-10-16-12_14_00_UTC', 'scheduled-2023-10-16-12_15_00_UTC', 'scheduled-2023-10-16-12_16_00_UTC', 'scheduled-2023-10-16-12_17_00_UTC', 'scheduled-2023-10-16-12_18_00_UTC', 'scheduled-2023-10-16-12_19_00_UTC', 'scheduled-2023-10-16-12_20_00_UTC', 'scheduled-2023-10-16-12_21_00_UTC', 'scheduled-2023-10-16-12_22_00_UTC', 'scheduled-2023-10-16-12_23_00_UTC'] 2023-10-16T12:23:25.133 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir2 2023-10-16T12:23:25.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.300+0000 7f6e799d9700 1 Processor -- start 2023-10-16T12:23:25.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.300+0000 7f6e799d9700 1 -- start start 2023-10-16T12:23:25.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.300+0000 7f6e799d9700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6e7415d8d0 0x7f6e740b2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:25.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.301+0000 7f6e799d9700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6e740ce350 0x7f6e740ce780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:25.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.301+0000 7f6e799d9700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e740cecc0 con 0x7f6e740b8fa0 2023-10-16T12:23:25.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.301+0000 7f6e799d9700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e740d0bb0 con 0x7f6e7415d8d0 2023-10-16T12:23:25.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.301+0000 7f6e799d9700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e7415dcb0 con 0x7f6e740ce350 2023-10-16T12:23:25.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.301+0000 7f6e72ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6e7415d8d0 0x7f6e740b2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:25.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.301+0000 7f6e72ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6e7415d8d0 0x7f6e740b2830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36658/0 (socket says 172.21.15.16:36658) 2023-10-16T12:23:25.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.301+0000 7f6e72ffd700 1 -- 172.21.15.16:0/1592506139 learned_addr learned my addr 172.21.15.16:0/1592506139 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:25.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.301+0000 7f6e72ffd700 1 -- 172.21.15.16:0/1592506139 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6e740ce350 msgr2=0x7f6e740ce780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:25.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.301+0000 7f6e737fe700 1 --2- 172.21.15.16:0/1592506139 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6e740ce350 0x7f6e740ce780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:25.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.301+0000 7f6e72ffd700 1 --2- 172.21.15.16:0/1592506139 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6e740ce350 0x7f6e740ce780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:25.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.301+0000 7f6e72ffd700 1 -- 172.21.15.16:0/1592506139 >> v1:172.21.15.49:6789/0 conn(0x7f6e740b8fa0 legacy=0x7f6e740b6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:25.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.301+0000 7f6e72ffd700 1 -- 172.21.15.16:0/1592506139 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e740b2f30 con 0x7f6e7415d8d0 2023-10-16T12:23:25.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.301+0000 7f6e72ffd700 1 --2- 172.21.15.16:0/1592506139 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e7415d8d0 0x7f6e740b2830 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f6e640098f0 tx=0x7f6e64009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1f2f32a02ac2ffc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:25.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.301+0000 7f6e71ffb700 1 -- 172.21.15.16:0/1592506139 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e64004230 con 0x7f6e7415d8d0 2023-10-16T12:23:25.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.301+0000 7f6e71ffb700 1 -- 172.21.15.16:0/1592506139 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e64004390 con 0x7f6e7415d8d0 2023-10-16T12:23:25.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.301+0000 7f6e71ffb700 1 -- 172.21.15.16:0/1592506139 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e640036a0 con 0x7f6e7415d8d0 2023-10-16T12:23:25.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.302+0000 7f6e799d9700 1 -- 172.21.15.16:0/1592506139 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e7415d8d0 msgr2=0x7f6e740b2830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:25.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.302+0000 7f6e799d9700 1 --2- 172.21.15.16:0/1592506139 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e7415d8d0 0x7f6e740b2830 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f6e640098f0 tx=0x7f6e64009c00 comp rx=0 tx=0).stop 2023-10-16T12:23:25.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.302+0000 7f6e799d9700 1 -- 172.21.15.16:0/1592506139 shutdown_connections 2023-10-16T12:23:25.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.302+0000 7f6e799d9700 1 --2- 172.21.15.16:0/1592506139 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6e740ce350 0x7f6e740ce780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:25.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.302+0000 7f6e799d9700 1 --2- 172.21.15.16:0/1592506139 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e7415d8d0 0x7f6e740b2830 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:25.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.302+0000 7f6e799d9700 1 -- 172.21.15.16:0/1592506139 >> 172.21.15.16:0/1592506139 conn(0x7f6e74072330 msgr2=0x7f6e74072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:25.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.302+0000 7f6e799d9700 1 -- 172.21.15.16:0/1592506139 shutdown_connections 2023-10-16T12:23:25.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.302+0000 7f6e799d9700 1 -- 172.21.15.16:0/1592506139 wait complete. 2023-10-16T12:23:25.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.303+0000 7f6e799d9700 1 Processor -- start 2023-10-16T12:23:25.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.303+0000 7f6e799d9700 1 -- start start 2023-10-16T12:23:25.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.304+0000 7f6e799d9700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6e740b8fa0 0x7f6e740b8000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:25.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.304+0000 7f6e799d9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e740ce350 0x7f6e74156c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:25.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.304+0000 7f6e799d9700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6e7415d8d0 0x7f6e740d28c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:25.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.304+0000 7f6e799d9700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e740b8540 con 0x7f6e740ce350 2023-10-16T12:23:25.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.304+0000 7f6e799d9700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e740c94b0 con 0x7f6e7415d8d0 2023-10-16T12:23:25.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.304+0000 7f6e799d9700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e740c9790 con 0x7f6e740b8fa0 2023-10-16T12:23:25.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.304+0000 7f6e727fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e740ce350 0x7f6e74156c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:25.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.304+0000 7f6e727fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e740ce350 0x7f6e74156c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36666/0 (socket says 172.21.15.16:36666) 2023-10-16T12:23:25.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.304+0000 7f6e727fc700 1 -- 172.21.15.16:0/3175348267 learned_addr learned my addr 172.21.15.16:0/3175348267 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:25.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.304+0000 7f6e737fe700 1 --2- 172.21.15.16:0/3175348267 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6e7415d8d0 0x7f6e740d28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:25.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.304+0000 7f6e727fc700 1 -- 172.21.15.16:0/3175348267 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6e740b8fa0 msgr2=0x7f6e740b8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:25.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.304+0000 7f6e727fc700 1 --2- 172.21.15.16:0/3175348267 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6e740b8fa0 0x7f6e740b8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:25.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.304+0000 7f6e727fc700 1 -- 172.21.15.16:0/3175348267 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6e7415d8d0 msgr2=0x7f6e740d28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:25.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.304+0000 7f6e727fc700 1 --2- 172.21.15.16:0/3175348267 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6e7415d8d0 0x7f6e740d28c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:25.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.304+0000 7f6e727fc700 1 -- 172.21.15.16:0/3175348267 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e64009550 con 0x7f6e740ce350 2023-10-16T12:23:25.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.305+0000 7f6e727fc700 1 --2- 172.21.15.16:0/3175348267 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e740ce350 0x7f6e74156c90 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f6e6800b4a0 tx=0x7f6e6800b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:25.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.305+0000 7f6e5bfff700 1 -- 172.21.15.16:0/3175348267 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e68011830 con 0x7f6e740ce350 2023-10-16T12:23:25.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.305+0000 7f6e5bfff700 1 -- 172.21.15.16:0/3175348267 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e68011990 con 0x7f6e740ce350 2023-10-16T12:23:25.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.305+0000 7f6e799d9700 1 -- 172.21.15.16:0/3175348267 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e740c9a70 con 0x7f6e740ce350 2023-10-16T12:23:25.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.305+0000 7f6e5bfff700 1 -- 172.21.15.16:0/3175348267 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e68011d40 con 0x7f6e740ce350 2023-10-16T12:23:25.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.305+0000 7f6e799d9700 1 -- 172.21.15.16:0/3175348267 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e741571d0 con 0x7f6e740ce350 2023-10-16T12:23:25.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.306+0000 7f6e799d9700 1 -- 172.21.15.16:0/3175348267 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e7404ebb0 con 0x7f6e740ce350 2023-10-16T12:23:25.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.306+0000 7f6e5bfff700 1 -- 172.21.15.16:0/3175348267 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f6e6801a400 con 0x7f6e740ce350 2023-10-16T12:23:25.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.307+0000 7f6e5bfff700 1 --2- 172.21.15.16:0/3175348267 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f6e6006b100 0x7f6e6006d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:25.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.307+0000 7f6e5bfff700 1 -- 172.21.15.16:0/3175348267 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f6e680899c0 con 0x7f6e740ce350 2023-10-16T12:23:25.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.307+0000 7f6e72ffd700 1 --2- 172.21.15.16:0/3175348267 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f6e6006b100 0x7f6e6006d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:25.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.307+0000 7f6e72ffd700 1 --2- 172.21.15.16:0/3175348267 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f6e6006b100 0x7f6e6006d5c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6e7415f6d0 tx=0x7f6e64010000 comp rx=0 tx=0).ready entity=mgr.5499 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:25.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.311+0000 7f6e5bfff700 1 -- 172.21.15.16:0/3175348267 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6e680544f0 con 0x7f6e740ce350 2023-10-16T12:23:25.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.543+0000 7f6e799d9700 1 -- 172.21.15.16:0/3175348267 --> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir2", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6e740d1ee0 con 0x7f6e6006b100 2023-10-16T12:23:25.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.552+0000 7f6e5bfff700 1 -- 172.21.15.16:0/3175348267 <== mgr.5499 v2:172.21.15.16:6832/1707126831 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f6e64004130 con 0x7f6e6006b100 2023-10-16T12:23:25.554 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:23:25.554 INFO:teuthology.orchestra.run.smithi016.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir2", "rel_path": "/snap_test_dir1/dir2", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T12:12:17", "first": "2023-10-16T12:13:00", "last": "2023-10-16T12:23:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-16T12:23:25.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.554+0000 7f6e799d9700 1 -- 172.21.15.16:0/3175348267 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f6e6006b100 msgr2=0x7f6e6006d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:25.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.554+0000 7f6e799d9700 1 --2- 172.21.15.16:0/3175348267 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f6e6006b100 0x7f6e6006d5c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6e7415f6d0 tx=0x7f6e64010000 comp rx=0 tx=0).stop 2023-10-16T12:23:25.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.554+0000 7f6e799d9700 1 -- 172.21.15.16:0/3175348267 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e740ce350 msgr2=0x7f6e74156c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:25.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.554+0000 7f6e799d9700 1 --2- 172.21.15.16:0/3175348267 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e740ce350 0x7f6e74156c90 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f6e6800b4a0 tx=0x7f6e6800b860 comp rx=0 tx=0).stop 2023-10-16T12:23:25.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.554+0000 7f6e799d9700 1 -- 172.21.15.16:0/3175348267 shutdown_connections 2023-10-16T12:23:25.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.554+0000 7f6e799d9700 1 --2- 172.21.15.16:0/3175348267 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f6e6006b100 0x7f6e6006d5c0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:25.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.554+0000 7f6e799d9700 1 --2- 172.21.15.16:0/3175348267 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6e7415d8d0 0x7f6e740d28c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:25.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.554+0000 7f6e799d9700 1 --2- 172.21.15.16:0/3175348267 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6e740b8fa0 0x7f6e740b8000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:25.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.554+0000 7f6e799d9700 1 --2- 172.21.15.16:0/3175348267 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e740ce350 0x7f6e74156c90 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:25.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.554+0000 7f6e799d9700 1 -- 172.21.15.16:0/3175348267 >> 172.21.15.16:0/3175348267 conn(0x7f6e74072330 msgr2=0x7f6e740b66b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:25.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.554+0000 7f6e799d9700 1 -- 172.21.15.16:0/3175348267 shutdown_connections 2023-10-16T12:23:25.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.554+0000 7f6e799d9700 1 -- 172.21.15.16:0/3175348267 wait complete. 2023-10-16T12:23:25.568 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir2", "rel_path": "/snap_test_dir1/dir2", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T12:12:17", "first": "2023-10-16T12:13:00", "last": "2023-10-16T12:23:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-16T12:23:25.568 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir2', 'rel_path': '/snap_test_dir1/dir2', 'schedule': '1m', 'retention': {}, 'start': '2023-10-16T00:00:00', 'created': '2023-10-16T12:12:17', 'first': '2023-10-16T12:13:00', 'last': '2023-10-16T12:23:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-10-16T12:23:25.569 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-10-16T12:23:25.569 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-10-16T12:23:25.569 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:25.570 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir3/.snap) 2023-10-16T12:23:25.588 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_13_00_UTC 2023-10-16T12:23:25.589 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_14_00_UTC 2023-10-16T12:23:25.589 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_15_00_UTC 2023-10-16T12:23:25.589 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_16_00_UTC 2023-10-16T12:23:25.589 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_17_00_UTC 2023-10-16T12:23:25.589 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_18_00_UTC 2023-10-16T12:23:25.590 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_19_00_UTC 2023-10-16T12:23:25.590 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_20_00_UTC 2023-10-16T12:23:25.590 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_21_00_UTC 2023-10-16T12:23:25.590 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_22_00_UTC 2023-10-16T12:23:25.590 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_23_00_UTC 2023-10-16T12:23:25.591 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-16-12_13_00_UTC', 'scheduled-2023-10-16-12_14_00_UTC', 'scheduled-2023-10-16-12_15_00_UTC', 'scheduled-2023-10-16-12_16_00_UTC', 'scheduled-2023-10-16-12_17_00_UTC', 'scheduled-2023-10-16-12_18_00_UTC', 'scheduled-2023-10-16-12_19_00_UTC', 'scheduled-2023-10-16-12_20_00_UTC', 'scheduled-2023-10-16-12_21_00_UTC', 'scheduled-2023-10-16-12_22_00_UTC', 'scheduled-2023-10-16-12_23_00_UTC'] 2023-10-16T12:23:25.591 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir3 2023-10-16T12:23:25.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.753+0000 7f867154c700 1 Processor -- start 2023-10-16T12:23:25.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.753+0000 7f867154c700 1 -- start start 2023-10-16T12:23:25.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.754+0000 7f867154c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f866c15d8d0 0x7f866c0b2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:25.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.754+0000 7f867154c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f866c0ce350 0x7f866c0ce7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:25.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.754+0000 7f867154c700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f866c0cece0 con 0x7f866c0b8fa0 2023-10-16T12:23:25.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.754+0000 7f867154c700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f866c0d0bb0 con 0x7f866c0ce350 2023-10-16T12:23:25.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.754+0000 7f867154c700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f866c15dcb0 con 0x7f866c15d8d0 2023-10-16T12:23:25.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.754+0000 7f866b7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f866c0ce350 0x7f866c0ce7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:25.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.754+0000 7f866b7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f866c0ce350 0x7f866c0ce7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36676/0 (socket says 172.21.15.16:36676) 2023-10-16T12:23:25.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.754+0000 7f866b7fe700 1 -- 172.21.15.16:0/1926936289 learned_addr learned my addr 172.21.15.16:0/1926936289 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:25.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.754+0000 7f866b7fe700 1 -- 172.21.15.16:0/1926936289 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f866c15d8d0 msgr2=0x7f866c0b2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:25.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.754+0000 7f866affd700 1 --2- 172.21.15.16:0/1926936289 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f866c15d8d0 0x7f866c0b2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:25.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.755+0000 7f866b7fe700 1 --2- 172.21.15.16:0/1926936289 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f866c15d8d0 0x7f866c0b2830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:25.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.755+0000 7f866b7fe700 1 -- 172.21.15.16:0/1926936289 >> v1:172.21.15.49:6789/0 conn(0x7f866c0b8fa0 legacy=0x7f866c0b6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:25.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.755+0000 7f866b7fe700 1 -- 172.21.15.16:0/1926936289 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f866c0b2f30 con 0x7f866c0ce350 2023-10-16T12:23:25.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.755+0000 7f866b7fe700 1 --2- 172.21.15.16:0/1926936289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f866c0ce350 0x7f866c0ce7a0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f8664009e50 tx=0x7f866400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e76c9cc379a11a88 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:25.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.755+0000 7f8669ffb700 1 -- 172.21.15.16:0/1926936289 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f866400e070 con 0x7f866c0ce350 2023-10-16T12:23:25.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.755+0000 7f8669ffb700 1 -- 172.21.15.16:0/1926936289 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8664003a20 con 0x7f866c0ce350 2023-10-16T12:23:25.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.755+0000 7f8669ffb700 1 -- 172.21.15.16:0/1926936289 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8664003dd0 con 0x7f866c0ce350 2023-10-16T12:23:25.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.755+0000 7f867154c700 1 -- 172.21.15.16:0/1926936289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f866c0ce350 msgr2=0x7f866c0ce7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:25.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.755+0000 7f867154c700 1 --2- 172.21.15.16:0/1926936289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f866c0ce350 0x7f866c0ce7a0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f8664009e50 tx=0x7f866400b330 comp rx=0 tx=0).stop 2023-10-16T12:23:25.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.755+0000 7f867154c700 1 -- 172.21.15.16:0/1926936289 shutdown_connections 2023-10-16T12:23:25.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.755+0000 7f867154c700 1 --2- 172.21.15.16:0/1926936289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f866c0ce350 0x7f866c0ce7a0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:25.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.755+0000 7f867154c700 1 --2- 172.21.15.16:0/1926936289 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f866c15d8d0 0x7f866c0b2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:25.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.755+0000 7f867154c700 1 -- 172.21.15.16:0/1926936289 >> 172.21.15.16:0/1926936289 conn(0x7f866c072330 msgr2=0x7f866c072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:25.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.755+0000 7f867154c700 1 -- 172.21.15.16:0/1926936289 shutdown_connections 2023-10-16T12:23:25.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.756+0000 7f867154c700 1 -- 172.21.15.16:0/1926936289 wait complete. 2023-10-16T12:23:25.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.757+0000 7f867154c700 1 Processor -- start 2023-10-16T12:23:25.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.757+0000 7f867154c700 1 -- start start 2023-10-16T12:23:25.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.758+0000 7f867154c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f866c0b8fa0 0x7f866c1567b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:25.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.758+0000 7f867154c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f866c0ce350 0x7f866c1579e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:25.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.758+0000 7f867154c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f866c15d8d0 0x7f866c15a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:25.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.758+0000 7f867154c700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f866c0cb810 con 0x7f866c15d8d0 2023-10-16T12:23:25.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.758+0000 7f867154c700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f866c0cab90 con 0x7f866c0ce350 2023-10-16T12:23:25.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.758+0000 7f867154c700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f866c0cae70 con 0x7f866c0b8fa0 2023-10-16T12:23:25.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.758+0000 7f866b7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f866c15d8d0 0x7f866c15a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:25.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.758+0000 7f866b7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f866c15d8d0 0x7f866c15a3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36682/0 (socket says 172.21.15.16:36682) 2023-10-16T12:23:25.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.758+0000 7f866b7fe700 1 -- 172.21.15.16:0/4082465195 learned_addr learned my addr 172.21.15.16:0/4082465195 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:25.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.758+0000 7f866affd700 1 --2- 172.21.15.16:0/4082465195 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f866c0b8fa0 0x7f866c1567b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:25.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.758+0000 7f866b7fe700 1 -- 172.21.15.16:0/4082465195 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f866c0b8fa0 msgr2=0x7f866c1567b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:25.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.758+0000 7f866b7fe700 1 --2- 172.21.15.16:0/4082465195 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f866c0b8fa0 0x7f866c1567b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:25.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.758+0000 7f866a7fc700 1 --2- 172.21.15.16:0/4082465195 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f866c0ce350 0x7f866c1579e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:25.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.758+0000 7f866b7fe700 1 -- 172.21.15.16:0/4082465195 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f866c0ce350 msgr2=0x7f866c1579e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:25.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.758+0000 7f866b7fe700 1 --2- 172.21.15.16:0/4082465195 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f866c0ce350 0x7f866c1579e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:25.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.758+0000 7f866b7fe700 1 -- 172.21.15.16:0/4082465195 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8664009b30 con 0x7f866c15d8d0 2023-10-16T12:23:25.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.758+0000 7f866b7fe700 1 --2- 172.21.15.16:0/4082465195 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f866c15d8d0 0x7f866c15a3f0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f866400b730 tx=0x7f8664004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:25.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.758+0000 7f86537fe700 1 -- 172.21.15.16:0/4082465195 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8664004440 con 0x7f866c15d8d0 2023-10-16T12:23:25.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.758+0000 7f86537fe700 1 -- 172.21.15.16:0/4082465195 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86640045c0 con 0x7f866c15d8d0 2023-10-16T12:23:25.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.758+0000 7f867154c700 1 -- 172.21.15.16:0/4082465195 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f866c0cb150 con 0x7f866c15d8d0 2023-10-16T12:23:25.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.758+0000 7f86537fe700 1 -- 172.21.15.16:0/4082465195 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8664004970 con 0x7f866c15d8d0 2023-10-16T12:23:25.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.758+0000 7f867154c700 1 -- 172.21.15.16:0/4082465195 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f866c156cf0 con 0x7f866c15d8d0 2023-10-16T12:23:25.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.759+0000 7f867154c700 1 -- 172.21.15.16:0/4082465195 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f866c04ebb0 con 0x7f866c15d8d0 2023-10-16T12:23:25.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.760+0000 7f86537fe700 1 -- 172.21.15.16:0/4082465195 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f8664019030 con 0x7f866c15d8d0 2023-10-16T12:23:25.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.760+0000 7f86537fe700 1 --2- 172.21.15.16:0/4082465195 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f865806b100 0x7f865806d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:25.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.760+0000 7f86537fe700 1 -- 172.21.15.16:0/4082465195 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f866408ad00 con 0x7f866c15d8d0 2023-10-16T12:23:25.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.760+0000 7f866affd700 1 --2- 172.21.15.16:0/4082465195 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f865806b100 0x7f865806d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:25.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.761+0000 7f866affd700 1 --2- 172.21.15.16:0/4082465195 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f865806b100 0x7f865806d5c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f866000b840 tx=0x7f8660008040 comp rx=0 tx=0).ready entity=mgr.5499 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:25.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.765+0000 7f86537fe700 1 -- 172.21.15.16:0/4082465195 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f86640557d0 con 0x7f866c15d8d0 2023-10-16T12:23:25.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.991+0000 7f867154c700 1 -- 172.21.15.16:0/4082465195 --> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir3", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f866c0d1ee0 con 0x7f865806b100 2023-10-16T12:23:26.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:25.998+0000 7f86537fe700 1 -- 172.21.15.16:0/4082465195 <== mgr.5499 v2:172.21.15.16:6832/1707126831 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f866000c040 con 0x7f865806b100 2023-10-16T12:23:26.000 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:23:26.001 INFO:teuthology.orchestra.run.smithi016.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir3", "rel_path": "/snap_test_dir1/dir3", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T12:12:18", "first": "2023-10-16T12:13:00", "last": "2023-10-16T12:23:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-16T12:23:26.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.000+0000 7f867154c700 1 -- 172.21.15.16:0/4082465195 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f865806b100 msgr2=0x7f865806d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:26.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.000+0000 7f867154c700 1 --2- 172.21.15.16:0/4082465195 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f865806b100 0x7f865806d5c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f866000b840 tx=0x7f8660008040 comp rx=0 tx=0).stop 2023-10-16T12:23:26.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.000+0000 7f867154c700 1 -- 172.21.15.16:0/4082465195 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f866c15d8d0 msgr2=0x7f866c15a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:26.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.000+0000 7f867154c700 1 --2- 172.21.15.16:0/4082465195 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f866c15d8d0 0x7f866c15a3f0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f866400b730 tx=0x7f8664004ad0 comp rx=0 tx=0).stop 2023-10-16T12:23:26.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.000+0000 7f867154c700 1 -- 172.21.15.16:0/4082465195 shutdown_connections 2023-10-16T12:23:26.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.000+0000 7f867154c700 1 --2- 172.21.15.16:0/4082465195 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f865806b100 0x7f865806d5c0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:26.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.000+0000 7f867154c700 1 --2- 172.21.15.16:0/4082465195 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f866c15d8d0 0x7f866c15a3f0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:26.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.000+0000 7f867154c700 1 --2- 172.21.15.16:0/4082465195 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f866c0b8fa0 0x7f866c1567b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:26.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.000+0000 7f867154c700 1 --2- 172.21.15.16:0/4082465195 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f866c0ce350 0x7f866c1579e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:26.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.000+0000 7f867154c700 1 -- 172.21.15.16:0/4082465195 >> 172.21.15.16:0/4082465195 conn(0x7f866c072330 msgr2=0x7f866c15deb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:26.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.001+0000 7f867154c700 1 -- 172.21.15.16:0/4082465195 shutdown_connections 2023-10-16T12:23:26.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.001+0000 7f867154c700 1 -- 172.21.15.16:0/4082465195 wait complete. 2023-10-16T12:23:26.014 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir3", "rel_path": "/snap_test_dir1/dir3", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T12:12:18", "first": "2023-10-16T12:13:00", "last": "2023-10-16T12:23:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-16T12:23:26.015 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir3', 'rel_path': '/snap_test_dir1/dir3', 'schedule': '1m', 'retention': {}, 'start': '2023-10-16T00:00:00', 'created': '2023-10-16T12:12:18', 'first': '2023-10-16T12:13:00', 'last': '2023-10-16T12:23:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-10-16T12:23:26.015 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-10-16T12:23:26.016 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-10-16T12:23:26.016 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:26.016 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir4/.snap) 2023-10-16T12:23:26.035 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_13_00_UTC 2023-10-16T12:23:26.035 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_14_00_UTC 2023-10-16T12:23:26.035 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_15_00_UTC 2023-10-16T12:23:26.035 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_16_00_UTC 2023-10-16T12:23:26.036 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_17_00_UTC 2023-10-16T12:23:26.036 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_18_00_UTC 2023-10-16T12:23:26.036 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_19_00_UTC 2023-10-16T12:23:26.036 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_20_00_UTC 2023-10-16T12:23:26.036 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_21_00_UTC 2023-10-16T12:23:26.037 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_22_00_UTC 2023-10-16T12:23:26.037 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_23_00_UTC 2023-10-16T12:23:26.037 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-16-12_13_00_UTC', 'scheduled-2023-10-16-12_14_00_UTC', 'scheduled-2023-10-16-12_15_00_UTC', 'scheduled-2023-10-16-12_16_00_UTC', 'scheduled-2023-10-16-12_17_00_UTC', 'scheduled-2023-10-16-12_18_00_UTC', 'scheduled-2023-10-16-12_19_00_UTC', 'scheduled-2023-10-16-12_20_00_UTC', 'scheduled-2023-10-16-12_21_00_UTC', 'scheduled-2023-10-16-12_22_00_UTC', 'scheduled-2023-10-16-12_23_00_UTC'] 2023-10-16T12:23:26.037 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir4 2023-10-16T12:23:26.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.200+0000 7fd6c9583700 1 Processor -- start 2023-10-16T12:23:26.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.200+0000 7fd6c9583700 1 -- start start 2023-10-16T12:23:26.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.200+0000 7fd6c9583700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fd6c415d8d0 0x7fd6c40b2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:26.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.200+0000 7fd6c9583700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd6c40ce350 0x7fd6c40ce780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:26.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.200+0000 7fd6c9583700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6c40cecc0 con 0x7fd6c40b8fa0 2023-10-16T12:23:26.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.200+0000 7fd6c9583700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6c40d0bb0 con 0x7fd6c40ce350 2023-10-16T12:23:26.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.200+0000 7fd6c9583700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6c415dcb0 con 0x7fd6c415d8d0 2023-10-16T12:23:26.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.201+0000 7fd6c37fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd6c40ce350 0x7fd6c40ce780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:26.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.201+0000 7fd6c37fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd6c40ce350 0x7fd6c40ce780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36690/0 (socket says 172.21.15.16:36690) 2023-10-16T12:23:26.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.201+0000 7fd6c37fe700 1 -- 172.21.15.16:0/4039096403 learned_addr learned my addr 172.21.15.16:0/4039096403 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:26.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.201+0000 7fd6c2ffd700 1 --2- 172.21.15.16:0/4039096403 >> v2:172.21.15.49:3300/0 conn(0x7fd6c415d8d0 0x7fd6c40b2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:26.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.201+0000 7fd6c37fe700 1 -- 172.21.15.16:0/4039096403 >> v2:172.21.15.49:3300/0 conn(0x7fd6c415d8d0 msgr2=0x7fd6c40b2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:26.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.201+0000 7fd6c37fe700 1 --2- 172.21.15.16:0/4039096403 >> v2:172.21.15.49:3300/0 conn(0x7fd6c415d8d0 0x7fd6c40b2830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:26.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.201+0000 7fd6c37fe700 1 -- 172.21.15.16:0/4039096403 >> v1:172.21.15.49:6789/0 conn(0x7fd6c40b8fa0 legacy=0x7fd6c40b6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:26.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.201+0000 7fd6c37fe700 1 -- 172.21.15.16:0/4039096403 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6c40b2f30 con 0x7fd6c40ce350 2023-10-16T12:23:26.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.201+0000 7fd6c2ffd700 1 --2- 172.21.15.16:0/4039096403 >> v2:172.21.15.49:3300/0 conn(0x7fd6c415d8d0 0x7fd6c40b2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:23:26.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.201+0000 7fd6c37fe700 1 --2- 172.21.15.16:0/4039096403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6c40ce350 0x7fd6c40ce780 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fd6c40d0dd0 tx=0x7fd6bc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d70300e1305512b9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:26.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.201+0000 7fd6c1ffb700 1 -- 172.21.15.16:0/4039096403 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6bc00e070 con 0x7fd6c40ce350 2023-10-16T12:23:26.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.201+0000 7fd6c1ffb700 1 -- 172.21.15.16:0/4039096403 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6bc003a20 con 0x7fd6c40ce350 2023-10-16T12:23:26.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.201+0000 7fd6c1ffb700 1 -- 172.21.15.16:0/4039096403 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6bc003dd0 con 0x7fd6c40ce350 2023-10-16T12:23:26.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.202+0000 7fd6c9583700 1 -- 172.21.15.16:0/4039096403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6c40ce350 msgr2=0x7fd6c40ce780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:26.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.202+0000 7fd6c9583700 1 --2- 172.21.15.16:0/4039096403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6c40ce350 0x7fd6c40ce780 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fd6c40d0dd0 tx=0x7fd6bc00b330 comp rx=0 tx=0).stop 2023-10-16T12:23:26.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.202+0000 7fd6c9583700 1 -- 172.21.15.16:0/4039096403 shutdown_connections 2023-10-16T12:23:26.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.202+0000 7fd6c9583700 1 --2- 172.21.15.16:0/4039096403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6c40ce350 0x7fd6c40ce780 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:26.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.202+0000 7fd6c9583700 1 --2- 172.21.15.16:0/4039096403 >> v2:172.21.15.49:3300/0 conn(0x7fd6c415d8d0 0x7fd6c40b2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:26.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.202+0000 7fd6c9583700 1 -- 172.21.15.16:0/4039096403 >> 172.21.15.16:0/4039096403 conn(0x7fd6c4072330 msgr2=0x7fd6c4072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:26.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.202+0000 7fd6c9583700 1 -- 172.21.15.16:0/4039096403 shutdown_connections 2023-10-16T12:23:26.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.202+0000 7fd6c9583700 1 -- 172.21.15.16:0/4039096403 wait complete. 2023-10-16T12:23:26.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.204+0000 7fd6c9583700 1 Processor -- start 2023-10-16T12:23:26.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.204+0000 7fd6c9583700 1 -- start start 2023-10-16T12:23:26.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.205+0000 7fd6c9583700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6c40b8fa0 0x7fd6c4153710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:26.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.205+0000 7fd6c9583700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd6c40ce350 0x7fd6c41f32a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:26.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.205+0000 7fd6c9583700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd6c415d8d0 0x7fd6c40c79e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:26.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.205+0000 7fd6c9583700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6c41f4240 con 0x7fd6c40b8fa0 2023-10-16T12:23:26.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.205+0000 7fd6c9583700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6c40ca020 con 0x7fd6c40ce350 2023-10-16T12:23:26.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.205+0000 7fd6c9583700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6c41f3c80 con 0x7fd6c415d8d0 2023-10-16T12:23:26.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.205+0000 7fd6c2ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6c40b8fa0 0x7fd6c4153710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:26.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.205+0000 7fd6c2ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6c40b8fa0 0x7fd6c4153710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36692/0 (socket says 172.21.15.16:36692) 2023-10-16T12:23:26.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.205+0000 7fd6c2ffd700 1 -- 172.21.15.16:0/4028528188 learned_addr learned my addr 172.21.15.16:0/4028528188 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:26.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.205+0000 7fd6c27fc700 1 --2- 172.21.15.16:0/4028528188 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd6c40ce350 0x7fd6c41f32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:26.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.205+0000 7fd6c37fe700 1 --2- 172.21.15.16:0/4028528188 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd6c415d8d0 0x7fd6c40c79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:26.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.205+0000 7fd6c2ffd700 1 -- 172.21.15.16:0/4028528188 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd6c415d8d0 msgr2=0x7fd6c40c79e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:26.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.205+0000 7fd6c2ffd700 1 --2- 172.21.15.16:0/4028528188 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd6c415d8d0 0x7fd6c40c79e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:26.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.205+0000 7fd6c2ffd700 1 -- 172.21.15.16:0/4028528188 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd6c40ce350 msgr2=0x7fd6c41f32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:26.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.205+0000 7fd6c2ffd700 1 --2- 172.21.15.16:0/4028528188 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd6c40ce350 0x7fd6c41f32a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:26.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.205+0000 7fd6c2ffd700 1 -- 172.21.15.16:0/4028528188 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6bc009b30 con 0x7fd6c40b8fa0 2023-10-16T12:23:26.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.205+0000 7fd6c2ffd700 1 --2- 172.21.15.16:0/4028528188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6c40b8fa0 0x7fd6c4153710 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fd6b800c7f0 tx=0x7fd6b800cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:26.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.205+0000 7fd6ab7fe700 1 -- 172.21.15.16:0/4028528188 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6b8011800 con 0x7fd6c40b8fa0 2023-10-16T12:23:26.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.205+0000 7fd6ab7fe700 1 -- 172.21.15.16:0/4028528188 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6b8011960 con 0x7fd6c40b8fa0 2023-10-16T12:23:26.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.205+0000 7fd6c9583700 1 -- 172.21.15.16:0/4028528188 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6c41f3f60 con 0x7fd6c40b8fa0 2023-10-16T12:23:26.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.205+0000 7fd6ab7fe700 1 -- 172.21.15.16:0/4028528188 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6b8011d10 con 0x7fd6c40b8fa0 2023-10-16T12:23:26.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.205+0000 7fd6c9583700 1 -- 172.21.15.16:0/4028528188 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6c40c74c0 con 0x7fd6c40b8fa0 2023-10-16T12:23:26.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.206+0000 7fd6c9583700 1 -- 172.21.15.16:0/4028528188 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6c40c7040 con 0x7fd6c40b8fa0 2023-10-16T12:23:26.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.207+0000 7fd6ab7fe700 1 -- 172.21.15.16:0/4028528188 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7fd6b8018360 con 0x7fd6c40b8fa0 2023-10-16T12:23:26.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.207+0000 7fd6ab7fe700 1 --2- 172.21.15.16:0/4028528188 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fd6b006b100 0x7fd6b006d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:26.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.207+0000 7fd6ab7fe700 1 -- 172.21.15.16:0/4028528188 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7fd6b8089c90 con 0x7fd6c40b8fa0 2023-10-16T12:23:26.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.207+0000 7fd6c27fc700 1 --2- 172.21.15.16:0/4028528188 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fd6b006b100 0x7fd6b006d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:26.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.208+0000 7fd6c27fc700 1 --2- 172.21.15.16:0/4028528188 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fd6b006b100 0x7fd6b006d5c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd6c415a0d0 tx=0x7fd6b4008040 comp rx=0 tx=0).ready entity=mgr.5499 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:26.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.212+0000 7fd6ab7fe700 1 -- 172.21.15.16:0/4028528188 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd6c40c7040 con 0x7fd6c40b8fa0 2023-10-16T12:23:26.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.440+0000 7fd6c9583700 1 -- 172.21.15.16:0/4028528188 --> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir4", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd6c40d1ee0 con 0x7fd6b006b100 2023-10-16T12:23:26.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.449+0000 7fd6ab7fe700 1 -- 172.21.15.16:0/4028528188 <== mgr.5499 v2:172.21.15.16:6832/1707126831 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7fd6b4009460 con 0x7fd6b006b100 2023-10-16T12:23:26.451 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:23:26.451 INFO:teuthology.orchestra.run.smithi016.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir4", "rel_path": "/snap_test_dir1/dir4", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T12:12:18", "first": "2023-10-16T12:13:00", "last": "2023-10-16T12:23:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-16T12:23:26.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.451+0000 7fd6c9583700 1 -- 172.21.15.16:0/4028528188 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fd6b006b100 msgr2=0x7fd6b006d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:26.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.451+0000 7fd6c9583700 1 --2- 172.21.15.16:0/4028528188 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fd6b006b100 0x7fd6b006d5c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd6c415a0d0 tx=0x7fd6b4008040 comp rx=0 tx=0).stop 2023-10-16T12:23:26.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.451+0000 7fd6c9583700 1 -- 172.21.15.16:0/4028528188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6c40b8fa0 msgr2=0x7fd6c4153710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:26.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.451+0000 7fd6c9583700 1 --2- 172.21.15.16:0/4028528188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6c40b8fa0 0x7fd6c4153710 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fd6b800c7f0 tx=0x7fd6b800cbb0 comp rx=0 tx=0).stop 2023-10-16T12:23:26.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.451+0000 7fd6c9583700 1 -- 172.21.15.16:0/4028528188 shutdown_connections 2023-10-16T12:23:26.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.451+0000 7fd6c9583700 1 --2- 172.21.15.16:0/4028528188 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fd6b006b100 0x7fd6b006d5c0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:26.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.451+0000 7fd6c9583700 1 --2- 172.21.15.16:0/4028528188 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd6c415d8d0 0x7fd6c40c79e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:26.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.451+0000 7fd6c9583700 1 --2- 172.21.15.16:0/4028528188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6c40b8fa0 0x7fd6c4153710 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:26.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.451+0000 7fd6c9583700 1 --2- 172.21.15.16:0/4028528188 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd6c40ce350 0x7fd6c41f32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:26.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.451+0000 7fd6c9583700 1 -- 172.21.15.16:0/4028528188 >> 172.21.15.16:0/4028528188 conn(0x7fd6c4072330 msgr2=0x7fd6c40b22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:26.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.451+0000 7fd6c9583700 1 -- 172.21.15.16:0/4028528188 shutdown_connections 2023-10-16T12:23:26.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.451+0000 7fd6c9583700 1 -- 172.21.15.16:0/4028528188 wait complete. 2023-10-16T12:23:26.465 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir4", "rel_path": "/snap_test_dir1/dir4", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T12:12:18", "first": "2023-10-16T12:13:00", "last": "2023-10-16T12:23:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-16T12:23:26.465 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir4', 'rel_path': '/snap_test_dir1/dir4', 'schedule': '1m', 'retention': {}, 'start': '2023-10-16T00:00:00', 'created': '2023-10-16T12:12:18', 'first': '2023-10-16T12:13:00', 'last': '2023-10-16T12:23:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-10-16T12:23:26.466 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-10-16T12:23:26.466 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-10-16T12:23:26.466 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:26.467 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir5/.snap) 2023-10-16T12:23:26.485 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_13_00_UTC 2023-10-16T12:23:26.486 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_14_00_UTC 2023-10-16T12:23:26.486 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_15_00_UTC 2023-10-16T12:23:26.486 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_16_00_UTC 2023-10-16T12:23:26.486 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_17_00_UTC 2023-10-16T12:23:26.486 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_18_00_UTC 2023-10-16T12:23:26.487 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_19_00_UTC 2023-10-16T12:23:26.487 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_20_00_UTC 2023-10-16T12:23:26.487 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_21_00_UTC 2023-10-16T12:23:26.487 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_22_00_UTC 2023-10-16T12:23:26.487 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_23_00_UTC 2023-10-16T12:23:26.488 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-16-12_13_00_UTC', 'scheduled-2023-10-16-12_14_00_UTC', 'scheduled-2023-10-16-12_15_00_UTC', 'scheduled-2023-10-16-12_16_00_UTC', 'scheduled-2023-10-16-12_17_00_UTC', 'scheduled-2023-10-16-12_18_00_UTC', 'scheduled-2023-10-16-12_19_00_UTC', 'scheduled-2023-10-16-12_20_00_UTC', 'scheduled-2023-10-16-12_21_00_UTC', 'scheduled-2023-10-16-12_22_00_UTC', 'scheduled-2023-10-16-12_23_00_UTC'] 2023-10-16T12:23:26.488 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir5 2023-10-16T12:23:26.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.651+0000 7f0f320f9700 1 Processor -- start 2023-10-16T12:23:26.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.651+0000 7f0f320f9700 1 -- start start 2023-10-16T12:23:26.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.651+0000 7f0f320f9700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f0f2c15b2c0 0x7f0f2c161340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:26.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.651+0000 7f0f320f9700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f2c15ecb0 con 0x7f0f2c06a7e0 2023-10-16T12:23:26.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.651+0000 7f0f320f9700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f2c161880 con 0x7f0f2c06e570 2023-10-16T12:23:26.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.651+0000 7f0f320f9700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f2c15b6a0 con 0x7f0f2c15b2c0 2023-10-16T12:23:26.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.652+0000 7f0f2bfff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0f2c06a7e0 0x7f0f2c071a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54206/0 (socket says 172.21.15.16:54206) 2023-10-16T12:23:26.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.652+0000 7f0f2bfff700 1 -- 172.21.15.16:0/3000858395 learned_addr learned my addr 172.21.15.16:0/3000858395 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:23:26.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.652+0000 7f0f2b7fe700 1 --2- 172.21.15.16:0/3000858395 >> v2:172.21.15.49:3300/0 conn(0x7f0f2c15b2c0 0x7f0f2c161340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:26.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.652+0000 7f0f2a7fc700 1 -- 172.21.15.16:0/3000858395 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1512027786 0 0) 0x7f0f2c15ecb0 con 0x7f0f2c06a7e0 2023-10-16T12:23:26.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.652+0000 7f0f2a7fc700 1 -- 172.21.15.16:0/3000858395 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f180035b0 con 0x7f0f2c06a7e0 2023-10-16T12:23:26.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.652+0000 7f0f2a7fc700 1 -- 172.21.15.16:0/3000858395 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2398500374 0 0) 0x7f0f180035b0 con 0x7f0f2c06a7e0 2023-10-16T12:23:26.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.652+0000 7f0f2a7fc700 1 -- 172.21.15.16:0/3000858395 >> v2:172.21.15.49:3300/0 conn(0x7f0f2c15b2c0 msgr2=0x7f0f2c161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:26.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.652+0000 7f0f2a7fc700 1 --2- 172.21.15.16:0/3000858395 >> v2:172.21.15.49:3300/0 conn(0x7f0f2c15b2c0 0x7f0f2c161340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:26.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.652+0000 7f0f2a7fc700 1 -- 172.21.15.16:0/3000858395 >> v1:172.21.15.49:6789/0 conn(0x7f0f2c06e570 legacy=0x7f0f2c15e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:26.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.652+0000 7f0f2a7fc700 1 -- 172.21.15.16:0/3000858395 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f2c161b20 con 0x7f0f2c06a7e0 2023-10-16T12:23:26.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.653+0000 7f0f2a7fc700 1 -- 172.21.15.16:0/3000858395 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f0f24001670 con 0x7f0f2c06a7e0 2023-10-16T12:23:26.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.653+0000 7f0f2a7fc700 1 -- 172.21.15.16:0/3000858395 >> v1:172.21.15.16:6789/0 conn(0x7f0f2c06a7e0 legacy=0x7f0f2c071a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:26.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.653+0000 7f0f2a7fc700 1 --2- 172.21.15.16:0/3000858395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f18003a00 0x7f0f18003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:26.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.653+0000 7f0f2a7fc700 1 --2- 172.21.15.16:0/3000858395 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f18004400 0x7f0f180048d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:26.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.653+0000 7f0f2a7fc700 1 --2- 172.21.15.16:0/3000858395 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f18005170 0x7f0f1800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:26.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.653+0000 7f0f2a7fc700 1 -- 172.21.15.16:0/3000858395 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f2c15b6a0 con 0x7f0f18003a00 2023-10-16T12:23:26.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.653+0000 7f0f2a7fc700 1 -- 172.21.15.16:0/3000858395 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1800c0b0 con 0x7f0f18005170 2023-10-16T12:23:26.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.653+0000 7f0f2a7fc700 1 -- 172.21.15.16:0/3000858395 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1800c270 con 0x7f0f18004400 2023-10-16T12:23:26.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.653+0000 7f0f2b7fe700 1 --2- 172.21.15.16:0/3000858395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f18003a00 0x7f0f18003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:26.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.653+0000 7f0f2bfff700 1 --2- 172.21.15.16:0/3000858395 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f18005170 0x7f0f1800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:26.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.653+0000 7f0f2affd700 1 --2- 172.21.15.16:0/3000858395 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f18004400 0x7f0f180048d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:26.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.653+0000 7f0f2b7fe700 1 -- 172.21.15.16:0/3000858395 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f18004400 msgr2=0x7f0f180048d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:26.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.653+0000 7f0f2b7fe700 1 --2- 172.21.15.16:0/3000858395 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f18004400 0x7f0f180048d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:26.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.653+0000 7f0f2b7fe700 1 -- 172.21.15.16:0/3000858395 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f18005170 msgr2=0x7f0f1800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:26.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.653+0000 7f0f2b7fe700 1 --2- 172.21.15.16:0/3000858395 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f18005170 0x7f0f1800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:26.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.653+0000 7f0f2b7fe700 1 -- 172.21.15.16:0/3000858395 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f180035b0 con 0x7f0f18003a00 2023-10-16T12:23:26.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.653+0000 7f0f2bfff700 1 --2- 172.21.15.16:0/3000858395 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f18005170 0x7f0f1800ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:23:26.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.653+0000 7f0f2bfff700 1 -- 172.21.15.16:0/3000858395 reap_dead start 2023-10-16T12:23:26.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.653+0000 7f0f2b7fe700 1 --2- 172.21.15.16:0/3000858395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f18003a00 0x7f0f18003de0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c009fc0 tx=0x7f0f18005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f64de7210fce8152 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:26.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.653+0000 7f0f2a7fc700 1 -- 172.21.15.16:0/3000858395 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f1c00ec30 con 0x7f0f18003a00 2023-10-16T12:23:26.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.653+0000 7f0f2a7fc700 1 -- 172.21.15.16:0/3000858395 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f1c00f040 con 0x7f0f18003a00 2023-10-16T12:23:26.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.653+0000 7f0f2a7fc700 1 -- 172.21.15.16:0/3000858395 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f1c014480 con 0x7f0f18003a00 2023-10-16T12:23:26.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.654+0000 7f0f320f9700 1 -- 172.21.15.16:0/3000858395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f18003a00 msgr2=0x7f0f18003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:26.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.654+0000 7f0f320f9700 1 --2- 172.21.15.16:0/3000858395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f18003a00 0x7f0f18003de0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c009fc0 tx=0x7f0f18005880 comp rx=0 tx=0).stop 2023-10-16T12:23:26.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.654+0000 7f0f320f9700 1 -- 172.21.15.16:0/3000858395 shutdown_connections 2023-10-16T12:23:26.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.654+0000 7f0f320f9700 1 --2- 172.21.15.16:0/3000858395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f18003a00 0x7f0f18003de0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:26.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.654+0000 7f0f320f9700 1 -- 172.21.15.16:0/3000858395 >> 172.21.15.16:0/3000858395 conn(0x7f0f2c0b1f40 msgr2=0x7f0f2c0b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:26.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.654+0000 7f0f320f9700 1 -- 172.21.15.16:0/3000858395 shutdown_connections 2023-10-16T12:23:26.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.654+0000 7f0f320f9700 1 -- 172.21.15.16:0/3000858395 wait complete. 2023-10-16T12:23:26.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.655+0000 7f0f320f9700 1 Processor -- start 2023-10-16T12:23:26.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.655+0000 7f0f320f9700 1 -- start start 2023-10-16T12:23:26.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.656+0000 7f0f320f9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f18003a00 0x7f0f2c1f2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:26.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.656+0000 7f0f320f9700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f2c0c8770 0x7f0f2c15c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:26.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.656+0000 7f0f320f9700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f2c0c8bf0 0x7f0f2c15cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:26.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.656+0000 7f0f320f9700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f2c0c9d40 con 0x7f0f18003a00 2023-10-16T12:23:26.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.656+0000 7f0f320f9700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f2c1f3040 con 0x7f0f2c0c8bf0 2023-10-16T12:23:26.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.656+0000 7f0f320f9700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f2c0c9130 con 0x7f0f2c0c8770 2023-10-16T12:23:26.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.656+0000 7f0f2b7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f18003a00 0x7f0f2c1f2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:26.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.656+0000 7f0f2b7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f18003a00 0x7f0f2c1f2b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36712/0 (socket says 172.21.15.16:36712) 2023-10-16T12:23:26.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.656+0000 7f0f2b7fe700 1 -- 172.21.15.16:0/2467092185 learned_addr learned my addr 172.21.15.16:0/2467092185 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:26.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.656+0000 7f0f2bfff700 1 --2- 172.21.15.16:0/2467092185 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f2c0c8bf0 0x7f0f2c15cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:26.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.656+0000 7f0f2affd700 1 --2- 172.21.15.16:0/2467092185 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f2c0c8770 0x7f0f2c15c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:26.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.657+0000 7f0f2b7fe700 1 -- 172.21.15.16:0/2467092185 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f2c0c8770 msgr2=0x7f0f2c15c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:26.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.657+0000 7f0f2b7fe700 1 --2- 172.21.15.16:0/2467092185 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f2c0c8770 0x7f0f2c15c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:26.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.657+0000 7f0f2b7fe700 1 -- 172.21.15.16:0/2467092185 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f2c0c8bf0 msgr2=0x7f0f2c15cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:26.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.657+0000 7f0f2b7fe700 1 --2- 172.21.15.16:0/2467092185 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f2c0c8bf0 0x7f0f2c15cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:26.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.657+0000 7f0f2b7fe700 1 -- 172.21.15.16:0/2467092185 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f2c0c9410 con 0x7f0f18003a00 2023-10-16T12:23:26.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.657+0000 7f0f2b7fe700 1 --2- 172.21.15.16:0/2467092185 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f18003a00 0x7f0f2c1f2b00 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c00ef30 tx=0x7f0f1c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:26.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.657+0000 7f0f23fff700 1 -- 172.21.15.16:0/2467092185 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f1c004450 con 0x7f0f18003a00 2023-10-16T12:23:26.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.657+0000 7f0f23fff700 1 -- 172.21.15.16:0/2467092185 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f1c003e90 con 0x7f0f18003a00 2023-10-16T12:23:26.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.657+0000 7f0f320f9700 1 -- 172.21.15.16:0/2467092185 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f2c0c96f0 con 0x7f0f18003a00 2023-10-16T12:23:26.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.657+0000 7f0f23fff700 1 -- 172.21.15.16:0/2467092185 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f1c0047f0 con 0x7f0f18003a00 2023-10-16T12:23:26.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.657+0000 7f0f320f9700 1 -- 172.21.15.16:0/2467092185 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f2c0ca1d0 con 0x7f0f18003a00 2023-10-16T12:23:26.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.658+0000 7f0f320f9700 1 -- 172.21.15.16:0/2467092185 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f2c04ebb0 con 0x7f0f18003a00 2023-10-16T12:23:26.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.658+0000 7f0f23fff700 1 -- 172.21.15.16:0/2467092185 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f0f1c014480 con 0x7f0f18003a00 2023-10-16T12:23:26.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.659+0000 7f0f23fff700 1 --2- 172.21.15.16:0/2467092185 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f0f0c06b100 0x7f0f0c06d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:26.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.659+0000 7f0f23fff700 1 -- 172.21.15.16:0/2467092185 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f0f1c08ba70 con 0x7f0f18003a00 2023-10-16T12:23:26.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.659+0000 7f0f2affd700 1 --2- 172.21.15.16:0/2467092185 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f0f0c06b100 0x7f0f0c06d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:26.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.659+0000 7f0f2affd700 1 --2- 172.21.15.16:0/2467092185 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f0f0c06b100 0x7f0f0c06d5c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0f14007fd0 tx=0x7f0f14009040 comp rx=0 tx=0).ready entity=mgr.5499 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:26.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.663+0000 7f0f23fff700 1 -- 172.21.15.16:0/2467092185 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0f1c0565c0 con 0x7f0f18003a00 2023-10-16T12:23:26.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.891+0000 7f0f320f9700 1 -- 172.21.15.16:0/2467092185 --> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir5", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0f2c161e00 con 0x7f0f0c06b100 2023-10-16T12:23:26.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.899+0000 7f0f23fff700 1 -- 172.21.15.16:0/2467092185 <== mgr.5499 v2:172.21.15.16:6832/1707126831 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f0f140015e0 con 0x7f0f0c06b100 2023-10-16T12:23:26.901 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:23:26.901 INFO:teuthology.orchestra.run.smithi016.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir5", "rel_path": "/snap_test_dir1/dir5", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T12:12:19", "first": "2023-10-16T12:13:00", "last": "2023-10-16T12:23:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-16T12:23:26.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.901+0000 7f0f320f9700 1 -- 172.21.15.16:0/2467092185 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f0f0c06b100 msgr2=0x7f0f0c06d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:26.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.901+0000 7f0f320f9700 1 --2- 172.21.15.16:0/2467092185 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f0f0c06b100 0x7f0f0c06d5c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0f14007fd0 tx=0x7f0f14009040 comp rx=0 tx=0).stop 2023-10-16T12:23:26.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.901+0000 7f0f320f9700 1 -- 172.21.15.16:0/2467092185 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f18003a00 msgr2=0x7f0f2c1f2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:26.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.901+0000 7f0f320f9700 1 --2- 172.21.15.16:0/2467092185 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f18003a00 0x7f0f2c1f2b00 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c00ef30 tx=0x7f0f1c009750 comp rx=0 tx=0).stop 2023-10-16T12:23:26.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.901+0000 7f0f320f9700 1 -- 172.21.15.16:0/2467092185 shutdown_connections 2023-10-16T12:23:26.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.901+0000 7f0f320f9700 1 --2- 172.21.15.16:0/2467092185 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f0f0c06b100 0x7f0f0c06d5c0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:26.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.901+0000 7f0f320f9700 1 --2- 172.21.15.16:0/2467092185 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f2c0c8bf0 0x7f0f2c15cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:26.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.901+0000 7f0f320f9700 1 --2- 172.21.15.16:0/2467092185 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f18003a00 0x7f0f2c1f2b00 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:26.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.901+0000 7f0f320f9700 1 --2- 172.21.15.16:0/2467092185 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f2c0c8770 0x7f0f2c15c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:26.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.901+0000 7f0f320f9700 1 -- 172.21.15.16:0/2467092185 >> 172.21.15.16:0/2467092185 conn(0x7f0f2c0b1f40 msgr2=0x7f0f2c0b1780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:26.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.901+0000 7f0f320f9700 1 -- 172.21.15.16:0/2467092185 shutdown_connections 2023-10-16T12:23:26.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:26.901+0000 7f0f320f9700 1 -- 172.21.15.16:0/2467092185 wait complete. 2023-10-16T12:23:26.915 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir5", "rel_path": "/snap_test_dir1/dir5", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T12:12:19", "first": "2023-10-16T12:13:00", "last": "2023-10-16T12:23:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-16T12:23:26.916 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir5', 'rel_path': '/snap_test_dir1/dir5', 'schedule': '1m', 'retention': {}, 'start': '2023-10-16T00:00:00', 'created': '2023-10-16T12:12:19', 'first': '2023-10-16T12:13:00', 'last': '2023-10-16T12:23:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-10-16T12:23:26.916 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-10-16T12:23:26.916 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-10-16T12:23:26.916 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:26.917 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir6/.snap) 2023-10-16T12:23:26.936 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_13_00_UTC 2023-10-16T12:23:26.936 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_14_00_UTC 2023-10-16T12:23:26.937 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_15_00_UTC 2023-10-16T12:23:26.937 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_16_00_UTC 2023-10-16T12:23:26.937 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_17_00_UTC 2023-10-16T12:23:26.937 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_18_00_UTC 2023-10-16T12:23:26.938 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_19_00_UTC 2023-10-16T12:23:26.938 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_20_00_UTC 2023-10-16T12:23:26.938 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_21_00_UTC 2023-10-16T12:23:26.938 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_22_00_UTC 2023-10-16T12:23:26.938 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_23_00_UTC 2023-10-16T12:23:26.939 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-16-12_13_00_UTC', 'scheduled-2023-10-16-12_14_00_UTC', 'scheduled-2023-10-16-12_15_00_UTC', 'scheduled-2023-10-16-12_16_00_UTC', 'scheduled-2023-10-16-12_17_00_UTC', 'scheduled-2023-10-16-12_18_00_UTC', 'scheduled-2023-10-16-12_19_00_UTC', 'scheduled-2023-10-16-12_20_00_UTC', 'scheduled-2023-10-16-12_21_00_UTC', 'scheduled-2023-10-16-12_22_00_UTC', 'scheduled-2023-10-16-12_23_00_UTC'] 2023-10-16T12:23:26.939 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir6 2023-10-16T12:23:27.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.102+0000 7fa3d96bb700 1 Processor -- start 2023-10-16T12:23:27.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.102+0000 7fa3d96bb700 1 -- start start 2023-10-16T12:23:27.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.103+0000 7fa3d96bb700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fa3d415d8d0 0x7fa3d40b2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:27.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.103+0000 7fa3d96bb700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa3d40b8fa0 0x7fa3d40b6720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:27.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.103+0000 7fa3d96bb700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3d40b2f30 con 0x7fa3d40b6c60 2023-10-16T12:23:27.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.103+0000 7fa3d96bb700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3d40d0bb0 con 0x7fa3d415d8d0 2023-10-16T12:23:27.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.103+0000 7fa3d96bb700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d415dcb0 con 0x7fa3d40b8fa0 2023-10-16T12:23:27.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.103+0000 7fa3d2ffd700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fa3d415d8d0 0x7fa3d40b2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:27.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.103+0000 7fa3d27fc700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa3d40b8fa0 0x7fa3d40b6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:27.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.103+0000 7fa3d37fe700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7fa3d40b6c60 0x7fa3d40ce350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:33180/0 (socket says 172.21.15.16:33180) 2023-10-16T12:23:27.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.103+0000 7fa3d37fe700 1 -- 172.21.15.16:0/1628601290 learned_addr learned my addr 172.21.15.16:0/1628601290 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:23:27.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.103+0000 7fa3d2ffd700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fa3d415d8d0 0x7fa3d40b2830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:35782/0 (socket says 172.21.15.16:35782) 2023-10-16T12:23:27.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.103+0000 7fa3d27fc700 1 --2- 172.21.15.16:0/1628601290 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa3d40b8fa0 0x7fa3d40b6720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:53212/0 (socket says 172.21.15.16:53212) 2023-10-16T12:23:27.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.103+0000 7fa3d2ffd700 1 -- 172.21.15.16:0/1628601290 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa3d40b8fa0 msgr2=0x7fa3d40b6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:27.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.103+0000 7fa3d2ffd700 1 --2- 172.21.15.16:0/1628601290 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa3d40b8fa0 0x7fa3d40b6720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:27.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.103+0000 7fa3d2ffd700 1 -- 172.21.15.16:0/1628601290 >> v1:172.21.15.49:6789/0 conn(0x7fa3d40b6c60 legacy=0x7fa3d40ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:27.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.103+0000 7fa3d2ffd700 1 -- 172.21.15.16:0/1628601290 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3d40d1c40 con 0x7fa3d415d8d0 2023-10-16T12:23:27.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.104+0000 7fa3d2ffd700 1 --2- 172.21.15.16:0/1628601290 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa3d415d8d0 0x7fa3d40b2830 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fa3c8009840 tx=0x7fa3c8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c92668fb4cd6285f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:27.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.104+0000 7fa3d1ffb700 1 -- 172.21.15.16:0/1628601290 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c8011830 con 0x7fa3d415d8d0 2023-10-16T12:23:27.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.104+0000 7fa3d1ffb700 1 -- 172.21.15.16:0/1628601290 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3c8011990 con 0x7fa3d415d8d0 2023-10-16T12:23:27.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.104+0000 7fa3d1ffb700 1 -- 172.21.15.16:0/1628601290 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c8011d20 con 0x7fa3d415d8d0 2023-10-16T12:23:27.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.104+0000 7fa3d96bb700 1 -- 172.21.15.16:0/1628601290 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa3d415d8d0 msgr2=0x7fa3d40b2830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:27.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.104+0000 7fa3d96bb700 1 --2- 172.21.15.16:0/1628601290 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa3d415d8d0 0x7fa3d40b2830 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fa3c8009840 tx=0x7fa3c8009b50 comp rx=0 tx=0).stop 2023-10-16T12:23:27.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.104+0000 7fa3d96bb700 1 -- 172.21.15.16:0/1628601290 shutdown_connections 2023-10-16T12:23:27.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.104+0000 7fa3d96bb700 1 --2- 172.21.15.16:0/1628601290 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa3d415d8d0 0x7fa3d40b2830 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:27.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.104+0000 7fa3d96bb700 1 --2- 172.21.15.16:0/1628601290 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa3d40b8fa0 0x7fa3d40b6720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:27.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.104+0000 7fa3d96bb700 1 -- 172.21.15.16:0/1628601290 >> 172.21.15.16:0/1628601290 conn(0x7fa3d4072330 msgr2=0x7fa3d4072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:27.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.105+0000 7fa3d96bb700 1 -- 172.21.15.16:0/1628601290 shutdown_connections 2023-10-16T12:23:27.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.105+0000 7fa3d96bb700 1 -- 172.21.15.16:0/1628601290 wait complete. 2023-10-16T12:23:27.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.106+0000 7fa3d96bb700 1 Processor -- start 2023-10-16T12:23:27.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.106+0000 7fa3d96bb700 1 -- start start 2023-10-16T12:23:27.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.107+0000 7fa3d96bb700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3d40b6c60 0x7fa3d4152990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:27.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.107+0000 7fa3d96bb700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa3d40b8fa0 0x7fa3d40b4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:27.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.107+0000 7fa3d96bb700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa3d415d8d0 0x7fa3d4154da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:27.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.107+0000 7fa3d96bb700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d40b4a90 con 0x7fa3d40b6c60 2023-10-16T12:23:27.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.107+0000 7fa3d96bb700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d4152ed0 con 0x7fa3d415d8d0 2023-10-16T12:23:27.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.107+0000 7fa3d96bb700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d40cb8f0 con 0x7fa3d40b8fa0 2023-10-16T12:23:27.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.107+0000 7fa3d2ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3d40b6c60 0x7fa3d4152990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:27.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.107+0000 7fa3d2ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3d40b6c60 0x7fa3d4152990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36726/0 (socket says 172.21.15.16:36726) 2023-10-16T12:23:27.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.107+0000 7fa3d2ffd700 1 -- 172.21.15.16:0/4083748380 learned_addr learned my addr 172.21.15.16:0/4083748380 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:27.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.107+0000 7fa3d37fe700 1 --2- 172.21.15.16:0/4083748380 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa3d415d8d0 0x7fa3d4154da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:27.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.107+0000 7fa3d27fc700 1 --2- 172.21.15.16:0/4083748380 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa3d40b8fa0 0x7fa3d40b4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:27.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.107+0000 7fa3d2ffd700 1 -- 172.21.15.16:0/4083748380 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa3d40b8fa0 msgr2=0x7fa3d40b4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:27.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.107+0000 7fa3d2ffd700 1 --2- 172.21.15.16:0/4083748380 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa3d40b8fa0 0x7fa3d40b4010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:27.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.107+0000 7fa3d2ffd700 1 -- 172.21.15.16:0/4083748380 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa3d415d8d0 msgr2=0x7fa3d4154da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:27.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.107+0000 7fa3d2ffd700 1 --2- 172.21.15.16:0/4083748380 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa3d415d8d0 0x7fa3d4154da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:27.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.107+0000 7fa3d2ffd700 1 -- 172.21.15.16:0/4083748380 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3c80094a0 con 0x7fa3d40b6c60 2023-10-16T12:23:27.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.107+0000 7fa3d2ffd700 1 --2- 172.21.15.16:0/4083748380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3d40b6c60 0x7fa3d4152990 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fa3c8009fd0 tx=0x7fa3c8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:27.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.107+0000 7fa3bb7fe700 1 -- 172.21.15.16:0/4083748380 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c80036a0 con 0x7fa3d40b6c60 2023-10-16T12:23:27.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.107+0000 7fa3bb7fe700 1 -- 172.21.15.16:0/4083748380 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3c801f070 con 0x7fa3d40b6c60 2023-10-16T12:23:27.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.107+0000 7fa3d96bb700 1 -- 172.21.15.16:0/4083748380 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3d40cbbd0 con 0x7fa3d40b6c60 2023-10-16T12:23:27.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.107+0000 7fa3bb7fe700 1 -- 172.21.15.16:0/4083748380 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c8003a60 con 0x7fa3d40b6c60 2023-10-16T12:23:27.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.107+0000 7fa3d96bb700 1 -- 172.21.15.16:0/4083748380 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3d40cc190 con 0x7fa3d40b6c60 2023-10-16T12:23:27.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.108+0000 7fa3d96bb700 1 -- 172.21.15.16:0/4083748380 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3d40d1ec0 con 0x7fa3d40b6c60 2023-10-16T12:23:27.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.109+0000 7fa3bb7fe700 1 -- 172.21.15.16:0/4083748380 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7fa3c8003bc0 con 0x7fa3d40b6c60 2023-10-16T12:23:27.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.109+0000 7fa3bb7fe700 1 --2- 172.21.15.16:0/4083748380 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fa3c006b100 0x7fa3c006d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:27.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.110+0000 7fa3d27fc700 1 --2- 172.21.15.16:0/4083748380 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fa3c006b100 0x7fa3c006d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:27.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.110+0000 7fa3bb7fe700 1 -- 172.21.15.16:0/4083748380 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7fa3c8021070 con 0x7fa3d40b6c60 2023-10-16T12:23:27.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.110+0000 7fa3d27fc700 1 --2- 172.21.15.16:0/4083748380 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fa3c006b100 0x7fa3c006d5c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa3d40cd460 tx=0x7fa3c4008040 comp rx=0 tx=0).ready entity=mgr.5499 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:27.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.114+0000 7fa3bb7fe700 1 -- 172.21.15.16:0/4083748380 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa3d40d1ec0 con 0x7fa3d40b6c60 2023-10-16T12:23:27.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.346+0000 7fa3d96bb700 1 -- 172.21.15.16:0/4083748380 --> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir6", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa3d40d21c0 con 0x7fa3c006b100 2023-10-16T12:23:27.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.354+0000 7fa3bb7fe700 1 -- 172.21.15.16:0/4083748380 <== mgr.5499 v2:172.21.15.16:6832/1707126831 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7fa3c4009660 con 0x7fa3c006b100 2023-10-16T12:23:27.357 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:23:27.357 INFO:teuthology.orchestra.run.smithi016.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir6", "rel_path": "/snap_test_dir1/dir6", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T12:12:19", "first": "2023-10-16T12:13:00", "last": "2023-10-16T12:23:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-16T12:23:27.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.357+0000 7fa3d96bb700 1 -- 172.21.15.16:0/4083748380 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fa3c006b100 msgr2=0x7fa3c006d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:27.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.357+0000 7fa3d96bb700 1 --2- 172.21.15.16:0/4083748380 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fa3c006b100 0x7fa3c006d5c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa3d40cd460 tx=0x7fa3c4008040 comp rx=0 tx=0).stop 2023-10-16T12:23:27.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.357+0000 7fa3d96bb700 1 -- 172.21.15.16:0/4083748380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3d40b6c60 msgr2=0x7fa3d4152990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:27.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.357+0000 7fa3d96bb700 1 --2- 172.21.15.16:0/4083748380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3d40b6c60 0x7fa3d4152990 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fa3c8009fd0 tx=0x7fa3c8004750 comp rx=0 tx=0).stop 2023-10-16T12:23:27.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.357+0000 7fa3d96bb700 1 -- 172.21.15.16:0/4083748380 shutdown_connections 2023-10-16T12:23:27.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.357+0000 7fa3d96bb700 1 --2- 172.21.15.16:0/4083748380 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fa3c006b100 0x7fa3c006d5c0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:27.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.357+0000 7fa3d96bb700 1 --2- 172.21.15.16:0/4083748380 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa3d415d8d0 0x7fa3d4154da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:27.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.357+0000 7fa3d96bb700 1 --2- 172.21.15.16:0/4083748380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3d40b6c60 0x7fa3d4152990 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:27.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.357+0000 7fa3d96bb700 1 --2- 172.21.15.16:0/4083748380 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa3d40b8fa0 0x7fa3d40b4010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:27.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.357+0000 7fa3d96bb700 1 -- 172.21.15.16:0/4083748380 >> 172.21.15.16:0/4083748380 conn(0x7fa3d4072330 msgr2=0x7fa3d415ddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:27.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.357+0000 7fa3d96bb700 1 -- 172.21.15.16:0/4083748380 shutdown_connections 2023-10-16T12:23:27.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.357+0000 7fa3d96bb700 1 -- 172.21.15.16:0/4083748380 wait complete. 2023-10-16T12:23:27.371 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir6", "rel_path": "/snap_test_dir1/dir6", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T12:12:19", "first": "2023-10-16T12:13:00", "last": "2023-10-16T12:23:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-16T12:23:27.371 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir6', 'rel_path': '/snap_test_dir1/dir6', 'schedule': '1m', 'retention': {}, 'start': '2023-10-16T00:00:00', 'created': '2023-10-16T12:12:19', 'first': '2023-10-16T12:13:00', 'last': '2023-10-16T12:23:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-10-16T12:23:27.372 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-10-16T12:23:27.372 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-10-16T12:23:27.372 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:27.373 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir7/.snap) 2023-10-16T12:23:27.392 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_13_00_UTC 2023-10-16T12:23:27.392 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_14_00_UTC 2023-10-16T12:23:27.392 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_15_00_UTC 2023-10-16T12:23:27.392 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_16_00_UTC 2023-10-16T12:23:27.392 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_17_00_UTC 2023-10-16T12:23:27.393 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_18_00_UTC 2023-10-16T12:23:27.393 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_19_00_UTC 2023-10-16T12:23:27.393 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_20_00_UTC 2023-10-16T12:23:27.393 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_21_00_UTC 2023-10-16T12:23:27.393 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_22_00_UTC 2023-10-16T12:23:27.394 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_23_00_UTC 2023-10-16T12:23:27.394 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-16-12_13_00_UTC', 'scheduled-2023-10-16-12_14_00_UTC', 'scheduled-2023-10-16-12_15_00_UTC', 'scheduled-2023-10-16-12_16_00_UTC', 'scheduled-2023-10-16-12_17_00_UTC', 'scheduled-2023-10-16-12_18_00_UTC', 'scheduled-2023-10-16-12_19_00_UTC', 'scheduled-2023-10-16-12_20_00_UTC', 'scheduled-2023-10-16-12_21_00_UTC', 'scheduled-2023-10-16-12_22_00_UTC', 'scheduled-2023-10-16-12_23_00_UTC'] 2023-10-16T12:23:27.394 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir7 2023-10-16T12:23:27.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.557+0000 7f88570ba700 1 Processor -- start 2023-10-16T12:23:27.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.557+0000 7f88570ba700 1 -- start start 2023-10-16T12:23:27.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.557+0000 7f88570ba700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8850158050 0x7f88500cdf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:27.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.557+0000 7f88570ba700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f88500ce470 0x7f885006da30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:27.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.557+0000 7f88570ba700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f885006eab0 con 0x7f885006df70 2023-10-16T12:23:27.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.557+0000 7f88570ba700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f885006e4f0 con 0x7f88500ce470 2023-10-16T12:23:27.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.557+0000 7f88570ba700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f885006e7d0 con 0x7f8850158050 2023-10-16T12:23:27.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.558+0000 7f8855657700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f885006df70 0x7f885015e1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54244/0 (socket says 172.21.15.16:54244) 2023-10-16T12:23:27.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.558+0000 7f8855657700 1 -- 172.21.15.16:0/2062192462 learned_addr learned my addr 172.21.15.16:0/2062192462 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:23:27.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.558+0000 7f8854e56700 1 --2- 172.21.15.16:0/2062192462 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8850158050 0x7f88500cdf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:27.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.558+0000 7f884ffff700 1 --2- 172.21.15.16:0/2062192462 >> v2:172.21.15.49:3300/0 conn(0x7f88500ce470 0x7f885006da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:27.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.558+0000 7f884f7fe700 1 -- 172.21.15.16:0/2062192462 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 566240147 0 0) 0x7f885006eab0 con 0x7f885006df70 2023-10-16T12:23:27.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.558+0000 7f884f7fe700 1 -- 172.21.15.16:0/2062192462 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88400035b0 con 0x7f885006df70 2023-10-16T12:23:27.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.558+0000 7f884f7fe700 1 -- 172.21.15.16:0/2062192462 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2176029877 0 0) 0x7f88400035b0 con 0x7f885006df70 2023-10-16T12:23:27.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.558+0000 7f884f7fe700 1 -- 172.21.15.16:0/2062192462 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8850158050 msgr2=0x7f88500cdf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:27.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.558+0000 7f884f7fe700 1 --2- 172.21.15.16:0/2062192462 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8850158050 0x7f88500cdf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:27.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.558+0000 7f884f7fe700 1 -- 172.21.15.16:0/2062192462 >> v2:172.21.15.49:3300/0 conn(0x7f88500ce470 msgr2=0x7f885006da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:27.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.558+0000 7f884f7fe700 1 --2- 172.21.15.16:0/2062192462 >> v2:172.21.15.49:3300/0 conn(0x7f88500ce470 0x7f885006da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:27.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.558+0000 7f884f7fe700 1 -- 172.21.15.16:0/2062192462 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88500cb9d0 con 0x7f885006df70 2023-10-16T12:23:27.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.558+0000 7f884f7fe700 1 -- 172.21.15.16:0/2062192462 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f88480019f0 con 0x7f885006df70 2023-10-16T12:23:27.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.558+0000 7f884f7fe700 1 -- 172.21.15.16:0/2062192462 >> v1:172.21.15.16:6789/0 conn(0x7f885006df70 legacy=0x7f885015e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:27.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.558+0000 7f884f7fe700 1 --2- 172.21.15.16:0/2062192462 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8850158050 0x7f8840003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:27.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.558+0000 7f884f7fe700 1 --2- 172.21.15.16:0/2062192462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8840003f80 0x7f8840004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:27.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.558+0000 7f884f7fe700 1 --2- 172.21.15.16:0/2062192462 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8840004d90 0x7f8840005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:27.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.558+0000 7f884f7fe700 1 -- 172.21.15.16:0/2062192462 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f885006e4f0 con 0x7f8840003f80 2023-10-16T12:23:27.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.558+0000 7f884f7fe700 1 -- 172.21.15.16:0/2062192462 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f885006e7d0 con 0x7f8840004d90 2023-10-16T12:23:27.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.558+0000 7f884f7fe700 1 -- 172.21.15.16:0/2062192462 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f884000ba10 con 0x7f8850158050 2023-10-16T12:23:27.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.558+0000 7f884ffff700 1 --2- 172.21.15.16:0/2062192462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8840003f80 0x7f8840004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:27.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.558+0000 7f8854e56700 1 --2- 172.21.15.16:0/2062192462 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8850158050 0x7f8840003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:27.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.559+0000 7f8855657700 1 --2- 172.21.15.16:0/2062192462 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8840004d90 0x7f8840005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:27.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.559+0000 7f884ffff700 1 -- 172.21.15.16:0/2062192462 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8850158050 msgr2=0x7f8840003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:27.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.559+0000 7f884ffff700 1 --2- 172.21.15.16:0/2062192462 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8850158050 0x7f8840003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:27.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.559+0000 7f884ffff700 1 -- 172.21.15.16:0/2062192462 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8840004d90 msgr2=0x7f8840005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:27.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.559+0000 7f884ffff700 1 --2- 172.21.15.16:0/2062192462 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8840004d90 0x7f8840005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:27.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.559+0000 7f884ffff700 1 -- 172.21.15.16:0/2062192462 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88400035b0 con 0x7f8840003f80 2023-10-16T12:23:27.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.559+0000 7f884ffff700 1 --2- 172.21.15.16:0/2062192462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8840003f80 0x7f8840004850 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f883c009b30 tx=0x7f8840005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=761ee0c1f1f1ff7c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:27.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.559+0000 7f884f7fe700 1 -- 172.21.15.16:0/2062192462 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f883c00e660 con 0x7f8840003f80 2023-10-16T12:23:27.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.559+0000 7f884f7fe700 1 -- 172.21.15.16:0/2062192462 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f883c00e7c0 con 0x7f8840003f80 2023-10-16T12:23:27.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.559+0000 7f884f7fe700 1 -- 172.21.15.16:0/2062192462 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f883c00eb30 con 0x7f8840003f80 2023-10-16T12:23:27.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.560+0000 7f88570ba700 1 -- 172.21.15.16:0/2062192462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8840003f80 msgr2=0x7f8840004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:27.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.560+0000 7f88570ba700 1 --2- 172.21.15.16:0/2062192462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8840003f80 0x7f8840004850 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f883c009b30 tx=0x7f8840005880 comp rx=0 tx=0).stop 2023-10-16T12:23:27.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.560+0000 7f8855657700 1 -- 172.21.15.16:0/2062192462 reap_dead start 2023-10-16T12:23:27.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.560+0000 7f88570ba700 1 -- 172.21.15.16:0/2062192462 shutdown_connections 2023-10-16T12:23:27.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.560+0000 7f88570ba700 1 -- 172.21.15.16:0/2062192462 >> 172.21.15.16:0/2062192462 conn(0x7f88500b1ed0 msgr2=0x7f88500b22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:27.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.560+0000 7f88570ba700 1 -- 172.21.15.16:0/2062192462 shutdown_connections 2023-10-16T12:23:27.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.560+0000 7f88570ba700 1 -- 172.21.15.16:0/2062192462 wait complete. 2023-10-16T12:23:27.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.561+0000 7f88570ba700 1 Processor -- start 2023-10-16T12:23:27.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.561+0000 7f88570ba700 1 -- start start 2023-10-16T12:23:27.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.562+0000 7f88570ba700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8840003f80 0x7f885014cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:27.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.562+0000 7f88570ba700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8850151350 0x7f88501517d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:27.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.562+0000 7f88570ba700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88500cc0d0 0x7f88500cc550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:27.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.562+0000 7f88570ba700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88500c79e0 con 0x7f88500cc0d0 2023-10-16T12:23:27.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.562+0000 7f88570ba700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f885014d2c0 con 0x7f8850151350 2023-10-16T12:23:27.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.562+0000 7f88570ba700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8850151d10 con 0x7f8840003f80 2023-10-16T12:23:27.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.562+0000 7f8855657700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88500cc0d0 0x7f88500cc550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:27.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.562+0000 7f8855657700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88500cc0d0 0x7f88500cc550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36750/0 (socket says 172.21.15.16:36750) 2023-10-16T12:23:27.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.562+0000 7f8855657700 1 -- 172.21.15.16:0/282394415 learned_addr learned my addr 172.21.15.16:0/282394415 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:27.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.562+0000 7f8854e56700 1 --2- 172.21.15.16:0/282394415 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8840003f80 0x7f885014cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:27.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.562+0000 7f884ffff700 1 --2- 172.21.15.16:0/282394415 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8850151350 0x7f88501517d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:27.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.562+0000 7f8855657700 1 -- 172.21.15.16:0/282394415 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8840003f80 msgr2=0x7f885014cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:27.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.562+0000 7f8855657700 1 --2- 172.21.15.16:0/282394415 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8840003f80 0x7f885014cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:27.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.562+0000 7f8855657700 1 -- 172.21.15.16:0/282394415 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8850151350 msgr2=0x7f88501517d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:27.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.562+0000 7f8855657700 1 --2- 172.21.15.16:0/282394415 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8850151350 0x7f88501517d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:27.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.562+0000 7f8855657700 1 -- 172.21.15.16:0/282394415 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88500ccd90 con 0x7f88500cc0d0 2023-10-16T12:23:27.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.562+0000 7f8854e56700 1 --2- 172.21.15.16:0/282394415 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8840003f80 0x7f885014cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:23:27.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.562+0000 7f8855657700 1 --2- 172.21.15.16:0/282394415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88500cc0d0 0x7f88500cc550 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f8850073480 tx=0x7f884800a640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:27.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.563+0000 7f884d7fa700 1 -- 172.21.15.16:0/282394415 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f884800ad50 con 0x7f88500cc0d0 2023-10-16T12:23:27.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.563+0000 7f884d7fa700 1 -- 172.21.15.16:0/282394415 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88480133f0 con 0x7f88500cc0d0 2023-10-16T12:23:27.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.563+0000 7f884d7fa700 1 -- 172.21.15.16:0/282394415 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88480137a0 con 0x7f88500cc0d0 2023-10-16T12:23:27.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.563+0000 7f88570ba700 1 -- 172.21.15.16:0/282394415 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88500cd070 con 0x7f88500cc0d0 2023-10-16T12:23:27.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.563+0000 7f88570ba700 1 -- 172.21.15.16:0/282394415 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88500c7600 con 0x7f88500cc0d0 2023-10-16T12:23:27.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.564+0000 7f88570ba700 1 -- 172.21.15.16:0/282394415 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88500c7b20 con 0x7f88500cc0d0 2023-10-16T12:23:27.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.564+0000 7f884d7fa700 1 -- 172.21.15.16:0/282394415 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f8848013900 con 0x7f88500cc0d0 2023-10-16T12:23:27.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.565+0000 7f884d7fa700 1 --2- 172.21.15.16:0/282394415 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f883406b100 0x7f883406d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:27.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.565+0000 7f8854e56700 1 --2- 172.21.15.16:0/282394415 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f883406b100 0x7f883406d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:27.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.565+0000 7f884d7fa700 1 -- 172.21.15.16:0/282394415 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f884808a6d0 con 0x7f88500cc0d0 2023-10-16T12:23:27.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.565+0000 7f8854e56700 1 --2- 172.21.15.16:0/282394415 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f883406b100 0x7f883406d5c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8844009b60 tx=0x7f8844008040 comp rx=0 tx=0).ready entity=mgr.5499 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:27.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.569+0000 7f884d7fa700 1 -- 172.21.15.16:0/282394415 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f88500c7b20 con 0x7f88500cc0d0 2023-10-16T12:23:27.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.798+0000 7f88570ba700 1 -- 172.21.15.16:0/282394415 --> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir7", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f88500cbcb0 con 0x7f883406b100 2023-10-16T12:23:27.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.806+0000 7f884d7fa700 1 -- 172.21.15.16:0/282394415 <== mgr.5499 v2:172.21.15.16:6832/1707126831 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f88440097f0 con 0x7f883406b100 2023-10-16T12:23:27.808 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:23:27.808 INFO:teuthology.orchestra.run.smithi016.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir7", "rel_path": "/snap_test_dir1/dir7", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T12:12:20", "first": "2023-10-16T12:13:00", "last": "2023-10-16T12:23:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-16T12:23:27.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.808+0000 7f88570ba700 1 -- 172.21.15.16:0/282394415 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f883406b100 msgr2=0x7f883406d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:27.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.808+0000 7f88570ba700 1 --2- 172.21.15.16:0/282394415 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f883406b100 0x7f883406d5c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8844009b60 tx=0x7f8844008040 comp rx=0 tx=0).stop 2023-10-16T12:23:27.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.808+0000 7f88570ba700 1 -- 172.21.15.16:0/282394415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88500cc0d0 msgr2=0x7f88500cc550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:27.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.808+0000 7f88570ba700 1 --2- 172.21.15.16:0/282394415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88500cc0d0 0x7f88500cc550 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f8850073480 tx=0x7f884800a640 comp rx=0 tx=0).stop 2023-10-16T12:23:27.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.808+0000 7f88570ba700 1 -- 172.21.15.16:0/282394415 shutdown_connections 2023-10-16T12:23:27.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.808+0000 7f88570ba700 1 --2- 172.21.15.16:0/282394415 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f883406b100 0x7f883406d5c0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:27.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.808+0000 7f88570ba700 1 --2- 172.21.15.16:0/282394415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88500cc0d0 0x7f88500cc550 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:27.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.808+0000 7f88570ba700 1 --2- 172.21.15.16:0/282394415 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8840003f80 0x7f885014cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:27.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.808+0000 7f88570ba700 1 --2- 172.21.15.16:0/282394415 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8850151350 0x7f88501517d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:27.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.808+0000 7f88570ba700 1 -- 172.21.15.16:0/282394415 >> 172.21.15.16:0/282394415 conn(0x7f88500b1ed0 msgr2=0x7f8850158430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:27.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.808+0000 7f88570ba700 1 -- 172.21.15.16:0/282394415 shutdown_connections 2023-10-16T12:23:27.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:27.808+0000 7f88570ba700 1 -- 172.21.15.16:0/282394415 wait complete. 2023-10-16T12:23:27.822 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir7", "rel_path": "/snap_test_dir1/dir7", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T12:12:20", "first": "2023-10-16T12:13:00", "last": "2023-10-16T12:23:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-16T12:23:27.822 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir7', 'rel_path': '/snap_test_dir1/dir7', 'schedule': '1m', 'retention': {}, 'start': '2023-10-16T00:00:00', 'created': '2023-10-16T12:12:20', 'first': '2023-10-16T12:13:00', 'last': '2023-10-16T12:23:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-10-16T12:23:27.822 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-10-16T12:23:27.823 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-10-16T12:23:27.823 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:27.823 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir8/.snap) 2023-10-16T12:23:27.842 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_13_00_UTC 2023-10-16T12:23:27.842 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_14_00_UTC 2023-10-16T12:23:27.843 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_15_00_UTC 2023-10-16T12:23:27.843 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_16_00_UTC 2023-10-16T12:23:27.843 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_17_00_UTC 2023-10-16T12:23:27.843 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_18_00_UTC 2023-10-16T12:23:27.843 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_19_00_UTC 2023-10-16T12:23:27.844 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_20_00_UTC 2023-10-16T12:23:27.844 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_21_00_UTC 2023-10-16T12:23:27.844 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_22_00_UTC 2023-10-16T12:23:27.844 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_23_00_UTC 2023-10-16T12:23:27.844 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-16-12_13_00_UTC', 'scheduled-2023-10-16-12_14_00_UTC', 'scheduled-2023-10-16-12_15_00_UTC', 'scheduled-2023-10-16-12_16_00_UTC', 'scheduled-2023-10-16-12_17_00_UTC', 'scheduled-2023-10-16-12_18_00_UTC', 'scheduled-2023-10-16-12_19_00_UTC', 'scheduled-2023-10-16-12_20_00_UTC', 'scheduled-2023-10-16-12_21_00_UTC', 'scheduled-2023-10-16-12_22_00_UTC', 'scheduled-2023-10-16-12_23_00_UTC'] 2023-10-16T12:23:27.845 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir8 2023-10-16T12:23:28.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.007+0000 7f4260587700 1 Processor -- start 2023-10-16T12:23:28.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.007+0000 7f4260587700 1 -- start start 2023-10-16T12:23:28.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.008+0000 7f4260587700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f425815b2c0 0x7f4258161340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:28.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.008+0000 7f4260587700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f425806e570 0x7f425815e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:28.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.008+0000 7f4260587700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4258161880 con 0x7f425806a7e0 2023-10-16T12:23:28.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.008+0000 7f4260587700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f425815b6a0 con 0x7f425806e570 2023-10-16T12:23:28.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.008+0000 7f4260587700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4258161b20 con 0x7f425815b2c0 2023-10-16T12:23:28.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.008+0000 7f425db22700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f425806e570 0x7f425815e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:28.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.008+0000 7f425db22700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f425806e570 0x7f425815e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36754/0 (socket says 172.21.15.16:36754) 2023-10-16T12:23:28.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.008+0000 7f425db22700 1 -- 172.21.15.16:0/1174070292 learned_addr learned my addr 172.21.15.16:0/1174070292 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:28.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.009+0000 7f425e323700 1 --2- 172.21.15.16:0/1174070292 >> v2:172.21.15.49:3300/0 conn(0x7f425815b2c0 0x7f4258161340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:28.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.009+0000 7f425db22700 1 -- 172.21.15.16:0/1174070292 >> v2:172.21.15.49:3300/0 conn(0x7f425815b2c0 msgr2=0x7f4258161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:28.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.009+0000 7f425db22700 1 --2- 172.21.15.16:0/1174070292 >> v2:172.21.15.49:3300/0 conn(0x7f425815b2c0 0x7f4258161340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:28.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.009+0000 7f425db22700 1 -- 172.21.15.16:0/1174070292 >> v1:172.21.15.49:6789/0 conn(0x7f425806a7e0 legacy=0x7f4258071a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:28.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.009+0000 7f425db22700 1 -- 172.21.15.16:0/1174070292 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4258161e00 con 0x7f425806e570 2023-10-16T12:23:28.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.009+0000 7f425db22700 1 --2- 172.21.15.16:0/1174070292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f425806e570 0x7f425815e5a0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f425806b4a0 tx=0x7f4254009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=148e6134cce68f0f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:28.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.009+0000 7f425d321700 1 -- 172.21.15.16:0/1174070292 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4254011830 con 0x7f425806e570 2023-10-16T12:23:28.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.009+0000 7f425d321700 1 -- 172.21.15.16:0/1174070292 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4254011990 con 0x7f425806e570 2023-10-16T12:23:28.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.009+0000 7f425d321700 1 -- 172.21.15.16:0/1174070292 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4254011d20 con 0x7f425806e570 2023-10-16T12:23:28.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.010+0000 7f4260587700 1 -- 172.21.15.16:0/1174070292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f425806e570 msgr2=0x7f425815e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:28.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.010+0000 7f4260587700 1 --2- 172.21.15.16:0/1174070292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f425806e570 0x7f425815e5a0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f425806b4a0 tx=0x7f4254009bd0 comp rx=0 tx=0).stop 2023-10-16T12:23:28.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.010+0000 7f4260587700 1 -- 172.21.15.16:0/1174070292 shutdown_connections 2023-10-16T12:23:28.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.010+0000 7f4260587700 1 --2- 172.21.15.16:0/1174070292 >> v2:172.21.15.49:3300/0 conn(0x7f425815b2c0 0x7f4258161340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:28.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.010+0000 7f4260587700 1 --2- 172.21.15.16:0/1174070292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f425806e570 0x7f425815e5a0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:28.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.010+0000 7f4260587700 1 -- 172.21.15.16:0/1174070292 >> 172.21.15.16:0/1174070292 conn(0x7f42580b1f40 msgr2=0x7f42580b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:28.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.010+0000 7f4260587700 1 -- 172.21.15.16:0/1174070292 shutdown_connections 2023-10-16T12:23:28.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.010+0000 7f4260587700 1 -- 172.21.15.16:0/1174070292 wait complete. 2023-10-16T12:23:28.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.011+0000 7f4260587700 1 Processor -- start 2023-10-16T12:23:28.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.011+0000 7f4260587700 1 -- start start 2023-10-16T12:23:28.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.012+0000 7f4260587700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f425806a7e0 0x7f42581f5c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:28.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.012+0000 7f4260587700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f425806e570 0x7f42581f2de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:28.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.012+0000 7f4260587700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f425815b2c0 0x7f4258159ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:28.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.012+0000 7f4260587700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42581f61b0 con 0x7f425806a7e0 2023-10-16T12:23:28.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.012+0000 7f4260587700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42580ca540 con 0x7f425815b2c0 2023-10-16T12:23:28.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.012+0000 7f4260587700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42580c9f80 con 0x7f425806e570 2023-10-16T12:23:28.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.012+0000 7f425e323700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f425806a7e0 0x7f42581f5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:28.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.012+0000 7f425e323700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f425806a7e0 0x7f42581f5c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36758/0 (socket says 172.21.15.16:36758) 2023-10-16T12:23:28.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.012+0000 7f425e323700 1 -- 172.21.15.16:0/1717848232 learned_addr learned my addr 172.21.15.16:0/1717848232 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:28.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.012+0000 7f425eb24700 1 --2- 172.21.15.16:0/1717848232 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f425815b2c0 0x7f4258159ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:28.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.012+0000 7f425db22700 1 --2- 172.21.15.16:0/1717848232 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f425806e570 0x7f42581f2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:28.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.012+0000 7f425e323700 1 -- 172.21.15.16:0/1717848232 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f425806e570 msgr2=0x7f42581f2de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:28.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.012+0000 7f425e323700 1 --2- 172.21.15.16:0/1717848232 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f425806e570 0x7f42581f2de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:28.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.012+0000 7f425e323700 1 -- 172.21.15.16:0/1717848232 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f425815b2c0 msgr2=0x7f4258159ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:28.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.012+0000 7f425e323700 1 --2- 172.21.15.16:0/1717848232 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f425815b2c0 0x7f4258159ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:28.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.012+0000 7f425e323700 1 -- 172.21.15.16:0/1717848232 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4254009550 con 0x7f425806a7e0 2023-10-16T12:23:28.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.012+0000 7f425e323700 1 --2- 172.21.15.16:0/1717848232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f425806a7e0 0x7f42581f5c70 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f424800e9b0 tx=0x7f424800ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:28.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.012+0000 7f424effd700 1 -- 172.21.15.16:0/1717848232 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4248009710 con 0x7f425806a7e0 2023-10-16T12:23:28.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.012+0000 7f424effd700 1 -- 172.21.15.16:0/1717848232 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f424800c850 con 0x7f425806a7e0 2023-10-16T12:23:28.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.012+0000 7f4260587700 1 -- 172.21.15.16:0/1717848232 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42580ca260 con 0x7f425806a7e0 2023-10-16T12:23:28.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.012+0000 7f424effd700 1 -- 172.21.15.16:0/1717848232 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f424800cbc0 con 0x7f425806a7e0 2023-10-16T12:23:28.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.012+0000 7f4260587700 1 -- 172.21.15.16:0/1717848232 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42580b4d40 con 0x7f425806a7e0 2023-10-16T12:23:28.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.013+0000 7f4260587700 1 -- 172.21.15.16:0/1717848232 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42580b50b0 con 0x7f425806a7e0 2023-10-16T12:23:28.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.014+0000 7f424effd700 1 -- 172.21.15.16:0/1717848232 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f424800cd20 con 0x7f425806a7e0 2023-10-16T12:23:28.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.014+0000 7f424effd700 1 --2- 172.21.15.16:0/1717848232 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f423c06b100 0x7f423c06d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:28.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.014+0000 7f424effd700 1 -- 172.21.15.16:0/1717848232 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f4248089c30 con 0x7f425806a7e0 2023-10-16T12:23:28.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.014+0000 7f425db22700 1 --2- 172.21.15.16:0/1717848232 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f423c06b100 0x7f423c06d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:28.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.015+0000 7f425db22700 1 --2- 172.21.15.16:0/1717848232 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f423c06b100 0x7f423c06d5c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4254009c10 tx=0x7f4254008030 comp rx=0 tx=0).ready entity=mgr.5499 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:28.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.019+0000 7f424effd700 1 -- 172.21.15.16:0/1717848232 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4248054890 con 0x7f425806a7e0 2023-10-16T12:23:28.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.251+0000 7f4260587700 1 -- 172.21.15.16:0/1717848232 --> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir8", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f425815e080 con 0x7f423c06b100 2023-10-16T12:23:28.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.259+0000 7f424effd700 1 -- 172.21.15.16:0/1717848232 <== mgr.5499 v2:172.21.15.16:6832/1707126831 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f4254011b80 con 0x7f423c06b100 2023-10-16T12:23:28.261 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:23:28.261 INFO:teuthology.orchestra.run.smithi016.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir8", "rel_path": "/snap_test_dir1/dir8", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T12:12:20", "first": "2023-10-16T12:13:00", "last": "2023-10-16T12:23:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-16T12:23:28.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.260+0000 7f4260587700 1 -- 172.21.15.16:0/1717848232 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f423c06b100 msgr2=0x7f423c06d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:28.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.260+0000 7f4260587700 1 --2- 172.21.15.16:0/1717848232 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f423c06b100 0x7f423c06d5c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4254009c10 tx=0x7f4254008030 comp rx=0 tx=0).stop 2023-10-16T12:23:28.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.260+0000 7f4260587700 1 -- 172.21.15.16:0/1717848232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f425806a7e0 msgr2=0x7f42581f5c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:28.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.260+0000 7f4260587700 1 --2- 172.21.15.16:0/1717848232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f425806a7e0 0x7f42581f5c70 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f424800e9b0 tx=0x7f424800ed70 comp rx=0 tx=0).stop 2023-10-16T12:23:28.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.261+0000 7f4260587700 1 -- 172.21.15.16:0/1717848232 shutdown_connections 2023-10-16T12:23:28.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.261+0000 7f4260587700 1 --2- 172.21.15.16:0/1717848232 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f423c06b100 0x7f423c06d5c0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:28.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.261+0000 7f4260587700 1 --2- 172.21.15.16:0/1717848232 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f425815b2c0 0x7f4258159ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:28.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.261+0000 7f4260587700 1 --2- 172.21.15.16:0/1717848232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f425806a7e0 0x7f42581f5c70 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:28.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.261+0000 7f4260587700 1 --2- 172.21.15.16:0/1717848232 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f425806e570 0x7f42581f2de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:28.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.261+0000 7f4260587700 1 -- 172.21.15.16:0/1717848232 >> 172.21.15.16:0/1717848232 conn(0x7f42580b1f40 msgr2=0x7f425815b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:28.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.261+0000 7f4260587700 1 -- 172.21.15.16:0/1717848232 shutdown_connections 2023-10-16T12:23:28.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.261+0000 7f4260587700 1 -- 172.21.15.16:0/1717848232 wait complete. 2023-10-16T12:23:28.274 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir8", "rel_path": "/snap_test_dir1/dir8", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T12:12:20", "first": "2023-10-16T12:13:00", "last": "2023-10-16T12:23:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-16T12:23:28.275 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir8', 'rel_path': '/snap_test_dir1/dir8', 'schedule': '1m', 'retention': {}, 'start': '2023-10-16T00:00:00', 'created': '2023-10-16T12:12:20', 'first': '2023-10-16T12:13:00', 'last': '2023-10-16T12:23:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-10-16T12:23:28.275 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-10-16T12:23:28.276 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-10-16T12:23:28.276 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:28.276 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir9/.snap) 2023-10-16T12:23:28.295 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_13_00_UTC 2023-10-16T12:23:28.295 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_14_00_UTC 2023-10-16T12:23:28.295 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_15_00_UTC 2023-10-16T12:23:28.296 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_16_00_UTC 2023-10-16T12:23:28.296 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_17_00_UTC 2023-10-16T12:23:28.296 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_18_00_UTC 2023-10-16T12:23:28.296 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_19_00_UTC 2023-10-16T12:23:28.296 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_20_00_UTC 2023-10-16T12:23:28.297 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_21_00_UTC 2023-10-16T12:23:28.297 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_22_00_UTC 2023-10-16T12:23:28.297 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_23_00_UTC 2023-10-16T12:23:28.297 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-16-12_13_00_UTC', 'scheduled-2023-10-16-12_14_00_UTC', 'scheduled-2023-10-16-12_15_00_UTC', 'scheduled-2023-10-16-12_16_00_UTC', 'scheduled-2023-10-16-12_17_00_UTC', 'scheduled-2023-10-16-12_18_00_UTC', 'scheduled-2023-10-16-12_19_00_UTC', 'scheduled-2023-10-16-12_20_00_UTC', 'scheduled-2023-10-16-12_21_00_UTC', 'scheduled-2023-10-16-12_22_00_UTC', 'scheduled-2023-10-16-12_23_00_UTC'] 2023-10-16T12:23:28.298 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir9 2023-10-16T12:23:28.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.460+0000 7f5f44912700 1 Processor -- start 2023-10-16T12:23:28.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.460+0000 7f5f44912700 1 -- start start 2023-10-16T12:23:28.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.461+0000 7f5f44912700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5f400ce6a0 0x7f5f4006c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:28.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.461+0000 7f5f44912700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f5f400d2110 0x7f5f4006c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:28.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.461+0000 7f5f44912700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f400b20f0 con 0x7f5f400b2620 2023-10-16T12:23:28.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.461+0000 7f5f44912700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f400af1f0 con 0x7f5f400d2110 2023-10-16T12:23:28.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.461+0000 7f5f44912700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f400d03d0 con 0x7f5f400ce6a0 2023-10-16T12:23:28.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.462+0000 7f5f3ed9d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f5f400b2620 0x7f5f400b0f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54276/0 (socket says 172.21.15.16:54276) 2023-10-16T12:23:28.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.462+0000 7f5f3ed9d700 1 -- 172.21.15.16:0/1756703646 learned_addr learned my addr 172.21.15.16:0/1756703646 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:23:28.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.462+0000 7f5f3e59c700 1 --2- 172.21.15.16:0/1756703646 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5f400ce6a0 0x7f5f4006c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:28.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.462+0000 7f5f3dd9b700 1 --2- 172.21.15.16:0/1756703646 >> v2:172.21.15.49:3300/0 conn(0x7f5f400d2110 0x7f5f4006c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:28.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.462+0000 7f5f3d59a700 1 -- 172.21.15.16:0/1756703646 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 995909140 0 0) 0x7f5f400b20f0 con 0x7f5f400b2620 2023-10-16T12:23:28.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.462+0000 7f5f3d59a700 1 -- 172.21.15.16:0/1756703646 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f2c0035b0 con 0x7f5f400b2620 2023-10-16T12:23:28.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.462+0000 7f5f3d59a700 1 -- 172.21.15.16:0/1756703646 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3379470707 0 0) 0x7f5f2c0035b0 con 0x7f5f400b2620 2023-10-16T12:23:28.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.462+0000 7f5f3d59a700 1 -- 172.21.15.16:0/1756703646 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5f400ce6a0 msgr2=0x7f5f4006c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:28.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.462+0000 7f5f3d59a700 1 --2- 172.21.15.16:0/1756703646 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5f400ce6a0 0x7f5f4006c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:28.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.462+0000 7f5f3d59a700 1 -- 172.21.15.16:0/1756703646 >> v2:172.21.15.49:3300/0 conn(0x7f5f400d2110 msgr2=0x7f5f4006c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:28.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.462+0000 7f5f3d59a700 1 --2- 172.21.15.16:0/1756703646 >> v2:172.21.15.49:3300/0 conn(0x7f5f400d2110 0x7f5f4006c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:28.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.462+0000 7f5f3d59a700 1 -- 172.21.15.16:0/1756703646 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f400b2340 con 0x7f5f400b2620 2023-10-16T12:23:28.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.462+0000 7f5f3d59a700 1 -- 172.21.15.16:0/1756703646 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f5f380019f0 con 0x7f5f400b2620 2023-10-16T12:23:28.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.462+0000 7f5f3d59a700 1 -- 172.21.15.16:0/1756703646 >> v1:172.21.15.16:6789/0 conn(0x7f5f400b2620 legacy=0x7f5f400b0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:28.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.462+0000 7f5f3d59a700 1 --2- 172.21.15.16:0/1756703646 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5f400ce6a0 0x7f5f2c003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:28.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.462+0000 7f5f3d59a700 1 --2- 172.21.15.16:0/1756703646 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5f2c003f80 0x7f5f2c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:28.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.462+0000 7f5f3d59a700 1 --2- 172.21.15.16:0/1756703646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f2c004d90 0x7f5f2c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:28.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.462+0000 7f5f3d59a700 1 -- 172.21.15.16:0/1756703646 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f400af1f0 con 0x7f5f2c004d90 2023-10-16T12:23:28.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.462+0000 7f5f3d59a700 1 -- 172.21.15.16:0/1756703646 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f400d03d0 con 0x7f5f2c003f80 2023-10-16T12:23:28.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.462+0000 7f5f3d59a700 1 -- 172.21.15.16:0/1756703646 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f2c00ba10 con 0x7f5f400ce6a0 2023-10-16T12:23:28.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.462+0000 7f5f3ed9d700 1 --2- 172.21.15.16:0/1756703646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f2c004d90 0x7f5f2c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:28.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.462+0000 7f5f3dd9b700 1 --2- 172.21.15.16:0/1756703646 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5f2c003f80 0x7f5f2c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:28.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.462+0000 7f5f3e59c700 1 --2- 172.21.15.16:0/1756703646 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5f400ce6a0 0x7f5f2c003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:28.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.463+0000 7f5f3ed9d700 1 -- 172.21.15.16:0/1756703646 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5f400ce6a0 msgr2=0x7f5f2c003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:28.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.463+0000 7f5f3ed9d700 1 --2- 172.21.15.16:0/1756703646 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5f400ce6a0 0x7f5f2c003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:28.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.463+0000 7f5f3ed9d700 1 -- 172.21.15.16:0/1756703646 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5f2c003f80 msgr2=0x7f5f2c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:28.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.463+0000 7f5f3ed9d700 1 --2- 172.21.15.16:0/1756703646 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5f2c003f80 0x7f5f2c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:28.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.463+0000 7f5f3ed9d700 1 -- 172.21.15.16:0/1756703646 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f2c0035b0 con 0x7f5f2c004d90 2023-10-16T12:23:28.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.463+0000 7f5f3e59c700 1 --2- 172.21.15.16:0/1756703646 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5f400ce6a0 0x7f5f2c003a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:23:28.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.463+0000 7f5f3dd9b700 1 --2- 172.21.15.16:0/1756703646 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5f2c003f80 0x7f5f2c004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:23:28.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.463+0000 7f5f3ed9d700 1 --2- 172.21.15.16:0/1756703646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f2c004d90 0x7f5f2c005210 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f5f3800a860 tx=0x7f5f2c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c45570e5797b6cb8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:28.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.463+0000 7f5f3d59a700 1 -- 172.21.15.16:0/1756703646 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f38012600 con 0x7f5f2c004d90 2023-10-16T12:23:28.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.463+0000 7f5f3d59a700 1 -- 172.21.15.16:0/1756703646 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f38012780 con 0x7f5f2c004d90 2023-10-16T12:23:28.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.463+0000 7f5f3d59a700 1 -- 172.21.15.16:0/1756703646 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f38012b30 con 0x7f5f2c004d90 2023-10-16T12:23:28.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.463+0000 7f5f44912700 1 -- 172.21.15.16:0/1756703646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f2c004d90 msgr2=0x7f5f2c005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:28.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.463+0000 7f5f44912700 1 --2- 172.21.15.16:0/1756703646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f2c004d90 0x7f5f2c005210 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f5f3800a860 tx=0x7f5f2c005880 comp rx=0 tx=0).stop 2023-10-16T12:23:28.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.463+0000 7f5f3ed9d700 1 -- 172.21.15.16:0/1756703646 reap_dead start 2023-10-16T12:23:28.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.464+0000 7f5f44912700 1 -- 172.21.15.16:0/1756703646 shutdown_connections 2023-10-16T12:23:28.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.464+0000 7f5f44912700 1 -- 172.21.15.16:0/1756703646 >> 172.21.15.16:0/1756703646 conn(0x7f5f400d15f0 msgr2=0x7f5f400d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:28.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.464+0000 7f5f44912700 1 -- 172.21.15.16:0/1756703646 shutdown_connections 2023-10-16T12:23:28.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.464+0000 7f5f44912700 1 -- 172.21.15.16:0/1756703646 wait complete. 2023-10-16T12:23:28.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.465+0000 7f5f44912700 1 Processor -- start 2023-10-16T12:23:28.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.465+0000 7f5f44912700 1 -- start start 2023-10-16T12:23:28.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.466+0000 7f5f44912700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5f2c004d90 0x7f5f401f37c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:28.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.466+0000 7f5f44912700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5f4015f870 0x7f5f4014f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:28.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.466+0000 7f5f44912700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f400c9c20 0x7f5f400ca0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:28.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.466+0000 7f5f44912700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f400b60d0 con 0x7f5f400c9c20 2023-10-16T12:23:28.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.466+0000 7f5f44912700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f400ca5e0 con 0x7f5f2c004d90 2023-10-16T12:23:28.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.466+0000 7f5f44912700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f401f3d00 con 0x7f5f4015f870 2023-10-16T12:23:28.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.466+0000 7f5f3ed9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f400c9c20 0x7f5f400ca0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:28.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.466+0000 7f5f3ed9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f400c9c20 0x7f5f400ca0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36782/0 (socket says 172.21.15.16:36782) 2023-10-16T12:23:28.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.466+0000 7f5f3ed9d700 1 -- 172.21.15.16:0/1891860890 learned_addr learned my addr 172.21.15.16:0/1891860890 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:28.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.466+0000 7f5f3e59c700 1 --2- 172.21.15.16:0/1891860890 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5f2c004d90 0x7f5f401f37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:28.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.466+0000 7f5f3dd9b700 1 --2- 172.21.15.16:0/1891860890 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5f4015f870 0x7f5f4014f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:28.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.466+0000 7f5f3ed9d700 1 -- 172.21.15.16:0/1891860890 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5f4015f870 msgr2=0x7f5f4014f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:28.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.466+0000 7f5f3ed9d700 1 --2- 172.21.15.16:0/1891860890 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5f4015f870 0x7f5f4014f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:28.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.466+0000 7f5f3ed9d700 1 -- 172.21.15.16:0/1891860890 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5f2c004d90 msgr2=0x7f5f401f37c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:28.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.466+0000 7f5f3ed9d700 1 --2- 172.21.15.16:0/1891860890 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5f2c004d90 0x7f5f401f37c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:28.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.466+0000 7f5f3ed9d700 1 -- 172.21.15.16:0/1891860890 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f400b5450 con 0x7f5f400c9c20 2023-10-16T12:23:28.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.467+0000 7f5f3ed9d700 1 --2- 172.21.15.16:0/1891860890 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f400c9c20 0x7f5f400ca0a0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f5f38012000 tx=0x7f5f38001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:28.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.467+0000 7f5f36ffd700 1 -- 172.21.15.16:0/1891860890 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f3801e600 con 0x7f5f400c9c20 2023-10-16T12:23:28.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.467+0000 7f5f36ffd700 1 -- 172.21.15.16:0/1891860890 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f3801e760 con 0x7f5f400c9c20 2023-10-16T12:23:28.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.467+0000 7f5f44912700 1 -- 172.21.15.16:0/1891860890 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f400b5730 con 0x7f5f400c9c20 2023-10-16T12:23:28.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.467+0000 7f5f36ffd700 1 -- 172.21.15.16:0/1891860890 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f3801eb10 con 0x7f5f400c9c20 2023-10-16T12:23:28.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.467+0000 7f5f44912700 1 -- 172.21.15.16:0/1891860890 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f400b5cf0 con 0x7f5f400c9c20 2023-10-16T12:23:28.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.468+0000 7f5f44912700 1 -- 172.21.15.16:0/1891860890 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f400b6210 con 0x7f5f400c9c20 2023-10-16T12:23:28.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.468+0000 7f5f36ffd700 1 -- 172.21.15.16:0/1891860890 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f5f38019030 con 0x7f5f400c9c20 2023-10-16T12:23:28.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.469+0000 7f5f36ffd700 1 --2- 172.21.15.16:0/1891860890 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f5f2006b100 0x7f5f2006d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:28.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.469+0000 7f5f36ffd700 1 -- 172.21.15.16:0/1891860890 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f5f3808b740 con 0x7f5f400c9c20 2023-10-16T12:23:28.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.469+0000 7f5f3e59c700 1 --2- 172.21.15.16:0/1891860890 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f5f2006b100 0x7f5f2006d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:28.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.469+0000 7f5f3e59c700 1 --2- 172.21.15.16:0/1891860890 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f5f2006b100 0x7f5f2006d5c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5f400c8290 tx=0x7f5f30008040 comp rx=0 tx=0).ready entity=mgr.5499 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:28.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.473+0000 7f5f36ffd700 1 -- 172.21.15.16:0/1891860890 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5f400b6210 con 0x7f5f400c9c20 2023-10-16T12:23:28.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.704+0000 7f5f44912700 1 -- 172.21.15.16:0/1891860890 --> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir9", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5f4006bde0 con 0x7f5f2006b100 2023-10-16T12:23:28.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.712+0000 7f5f36ffd700 1 -- 172.21.15.16:0/1891860890 <== mgr.5499 v2:172.21.15.16:6832/1707126831 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f5f30013890 con 0x7f5f2006b100 2023-10-16T12:23:28.714 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:23:28.714 INFO:teuthology.orchestra.run.smithi016.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir9", "rel_path": "/snap_test_dir1/dir9", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T12:12:21", "first": "2023-10-16T12:13:00", "last": "2023-10-16T12:23:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-16T12:23:28.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.714+0000 7f5f44912700 1 -- 172.21.15.16:0/1891860890 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f5f2006b100 msgr2=0x7f5f2006d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:28.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.714+0000 7f5f44912700 1 --2- 172.21.15.16:0/1891860890 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f5f2006b100 0x7f5f2006d5c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5f400c8290 tx=0x7f5f30008040 comp rx=0 tx=0).stop 2023-10-16T12:23:28.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.714+0000 7f5f44912700 1 -- 172.21.15.16:0/1891860890 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f400c9c20 msgr2=0x7f5f400ca0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:28.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.714+0000 7f5f44912700 1 --2- 172.21.15.16:0/1891860890 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f400c9c20 0x7f5f400ca0a0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f5f38012000 tx=0x7f5f38001d80 comp rx=0 tx=0).stop 2023-10-16T12:23:28.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.714+0000 7f5f44912700 1 -- 172.21.15.16:0/1891860890 shutdown_connections 2023-10-16T12:23:28.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.714+0000 7f5f44912700 1 --2- 172.21.15.16:0/1891860890 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f5f2006b100 0x7f5f2006d5c0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:28.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.714+0000 7f5f44912700 1 --2- 172.21.15.16:0/1891860890 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f400c9c20 0x7f5f400ca0a0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:28.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.714+0000 7f5f44912700 1 --2- 172.21.15.16:0/1891860890 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5f2c004d90 0x7f5f401f37c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:28.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.714+0000 7f5f44912700 1 --2- 172.21.15.16:0/1891860890 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5f4015f870 0x7f5f4014f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:28.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.714+0000 7f5f44912700 1 -- 172.21.15.16:0/1891860890 >> 172.21.15.16:0/1891860890 conn(0x7f5f400d15f0 msgr2=0x7f5f400702e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:28.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.714+0000 7f5f44912700 1 -- 172.21.15.16:0/1891860890 shutdown_connections 2023-10-16T12:23:28.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.714+0000 7f5f44912700 1 -- 172.21.15.16:0/1891860890 wait complete. 2023-10-16T12:23:28.728 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir9", "rel_path": "/snap_test_dir1/dir9", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T12:12:21", "first": "2023-10-16T12:13:00", "last": "2023-10-16T12:23:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-16T12:23:28.729 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir9', 'rel_path': '/snap_test_dir1/dir9', 'schedule': '1m', 'retention': {}, 'start': '2023-10-16T00:00:00', 'created': '2023-10-16T12:12:21', 'first': '2023-10-16T12:13:00', 'last': '2023-10-16T12:23:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-10-16T12:23:28.729 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-10-16T12:23:28.729 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-10-16T12:23:28.730 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir0 1m 2023-10-16T12:23:28.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.855+0000 7fb8742b1700 1 Processor -- start 2023-10-16T12:23:28.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.856+0000 7fb8742b1700 1 -- start start 2023-10-16T12:23:28.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.856+0000 7fb8742b1700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb86c15b2c0 0x7fb86c161340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:28.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.856+0000 7fb8742b1700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fb86c06e570 0x7fb86c15e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:28.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.856+0000 7fb8742b1700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb86c161880 con 0x7fb86c06a7e0 2023-10-16T12:23:28.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.856+0000 7fb8742b1700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb86c15b6a0 con 0x7fb86c15b2c0 2023-10-16T12:23:28.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.856+0000 7fb8742b1700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb86c161b20 con 0x7fb86c06e570 2023-10-16T12:23:28.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.857+0000 7fb87204d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb86c15b2c0 0x7fb86c161340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:28.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.857+0000 7fb87204d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb86c15b2c0 0x7fb86c161340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36786/0 (socket says 172.21.15.16:36786) 2023-10-16T12:23:28.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.857+0000 7fb87204d700 1 -- 172.21.15.16:0/1340008243 learned_addr learned my addr 172.21.15.16:0/1340008243 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:28.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.857+0000 7fb87184c700 1 --2- 172.21.15.16:0/1340008243 >> v2:172.21.15.49:3300/0 conn(0x7fb86c06e570 0x7fb86c15e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:28.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.857+0000 7fb87204d700 1 -- 172.21.15.16:0/1340008243 >> v2:172.21.15.49:3300/0 conn(0x7fb86c06e570 msgr2=0x7fb86c15e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:28.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.857+0000 7fb87204d700 1 --2- 172.21.15.16:0/1340008243 >> v2:172.21.15.49:3300/0 conn(0x7fb86c06e570 0x7fb86c15e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:28.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.857+0000 7fb87204d700 1 -- 172.21.15.16:0/1340008243 >> v1:172.21.15.49:6789/0 conn(0x7fb86c06a7e0 legacy=0x7fb86c071a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:28.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.857+0000 7fb87204d700 1 -- 172.21.15.16:0/1340008243 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb86c161e00 con 0x7fb86c15b2c0 2023-10-16T12:23:28.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.857+0000 7fb87204d700 1 --2- 172.21.15.16:0/1340008243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb86c15b2c0 0x7fb86c161340 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fb86c070500 tx=0x7fb868009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f151b50c914b9f3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:28.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.857+0000 7fb87104b700 1 -- 172.21.15.16:0/1340008243 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb868004230 con 0x7fb86c15b2c0 2023-10-16T12:23:28.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.857+0000 7fb87104b700 1 -- 172.21.15.16:0/1340008243 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb868004390 con 0x7fb86c15b2c0 2023-10-16T12:23:28.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.857+0000 7fb87104b700 1 -- 172.21.15.16:0/1340008243 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8680036a0 con 0x7fb86c15b2c0 2023-10-16T12:23:28.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.858+0000 7fb8742b1700 1 -- 172.21.15.16:0/1340008243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb86c15b2c0 msgr2=0x7fb86c161340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:28.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.858+0000 7fb8742b1700 1 --2- 172.21.15.16:0/1340008243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb86c15b2c0 0x7fb86c161340 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fb86c070500 tx=0x7fb868009bd0 comp rx=0 tx=0).stop 2023-10-16T12:23:28.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.858+0000 7fb8742b1700 1 -- 172.21.15.16:0/1340008243 shutdown_connections 2023-10-16T12:23:28.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.858+0000 7fb8742b1700 1 --2- 172.21.15.16:0/1340008243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb86c15b2c0 0x7fb86c161340 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:28.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.858+0000 7fb8742b1700 1 --2- 172.21.15.16:0/1340008243 >> v2:172.21.15.49:3300/0 conn(0x7fb86c06e570 0x7fb86c15e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:28.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.858+0000 7fb8742b1700 1 -- 172.21.15.16:0/1340008243 >> 172.21.15.16:0/1340008243 conn(0x7fb86c0b1f40 msgr2=0x7fb86c0b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:28.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.858+0000 7fb8742b1700 1 -- 172.21.15.16:0/1340008243 shutdown_connections 2023-10-16T12:23:28.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.858+0000 7fb8742b1700 1 -- 172.21.15.16:0/1340008243 wait complete. 2023-10-16T12:23:28.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.859+0000 7fb8742b1700 1 Processor -- start 2023-10-16T12:23:28.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.859+0000 7fb8742b1700 1 -- start start 2023-10-16T12:23:28.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.860+0000 7fb8742b1700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb86c06a7e0 0x7fb86c1579e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:28.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.860+0000 7fb8742b1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb86c06e570 0x7fb86c0c8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:28.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.860+0000 7fb8742b1700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb86c15b2c0 0x7fb86c15f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:28.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.860+0000 7fb8742b1700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb86c0ca390 con 0x7fb86c06e570 2023-10-16T12:23:28.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.860+0000 7fb8742b1700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb86c0c9810 con 0x7fb86c15b2c0 2023-10-16T12:23:28.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.860+0000 7fb8742b1700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb86c0c9af0 con 0x7fb86c06a7e0 2023-10-16T12:23:28.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.860+0000 7fb87184c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb86c06e570 0x7fb86c0c8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:28.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.860+0000 7fb87184c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb86c06e570 0x7fb86c0c8dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36794/0 (socket says 172.21.15.16:36794) 2023-10-16T12:23:28.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.860+0000 7fb87184c700 1 -- 172.21.15.16:0/801325345 learned_addr learned my addr 172.21.15.16:0/801325345 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:28.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.860+0000 7fb87204d700 1 --2- 172.21.15.16:0/801325345 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb86c06a7e0 0x7fb86c1579e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:28.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.860+0000 7fb87284e700 1 --2- 172.21.15.16:0/801325345 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb86c15b2c0 0x7fb86c15f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:28.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.860+0000 7fb87184c700 1 -- 172.21.15.16:0/801325345 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb86c06a7e0 msgr2=0x7fb86c1579e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:28.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.860+0000 7fb87184c700 1 --2- 172.21.15.16:0/801325345 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb86c06a7e0 0x7fb86c1579e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:28.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.860+0000 7fb87184c700 1 -- 172.21.15.16:0/801325345 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb86c15b2c0 msgr2=0x7fb86c15f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:28.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.860+0000 7fb87184c700 1 --2- 172.21.15.16:0/801325345 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb86c15b2c0 0x7fb86c15f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:28.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.860+0000 7fb87184c700 1 -- 172.21.15.16:0/801325345 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb868009550 con 0x7fb86c06e570 2023-10-16T12:23:28.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.860+0000 7fb87284e700 1 --2- 172.21.15.16:0/801325345 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb86c15b2c0 0x7fb86c15f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:23:28.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.860+0000 7fb87184c700 1 --2- 172.21.15.16:0/801325345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb86c06e570 0x7fb86c0c8dc0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fb85c00c7f0 tx=0x7fb85c00cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:28.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.861+0000 7fb862ffd700 1 -- 172.21.15.16:0/801325345 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb85c004220 con 0x7fb86c06e570 2023-10-16T12:23:28.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.861+0000 7fb862ffd700 1 -- 172.21.15.16:0/801325345 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb85c004380 con 0x7fb86c06e570 2023-10-16T12:23:28.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.861+0000 7fb8742b1700 1 -- 172.21.15.16:0/801325345 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb86c0c9dd0 con 0x7fb86c06e570 2023-10-16T12:23:28.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.861+0000 7fb862ffd700 1 -- 172.21.15.16:0/801325345 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb85c0036a0 con 0x7fb86c06e570 2023-10-16T12:23:28.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.861+0000 7fb8742b1700 1 -- 172.21.15.16:0/801325345 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb86c157f20 con 0x7fb86c06e570 2023-10-16T12:23:28.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.862+0000 7fb8742b1700 1 -- 172.21.15.16:0/801325345 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb86c04ebb0 con 0x7fb86c06e570 2023-10-16T12:23:28.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.862+0000 7fb862ffd700 1 -- 172.21.15.16:0/801325345 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7fb85c003800 con 0x7fb86c06e570 2023-10-16T12:23:28.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.862+0000 7fb862ffd700 1 --2- 172.21.15.16:0/801325345 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fb85006b100 0x7fb85006d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:28.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.863+0000 7fb862ffd700 1 -- 172.21.15.16:0/801325345 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7fb85c089840 con 0x7fb86c06e570 2023-10-16T12:23:28.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.863+0000 7fb87204d700 1 --2- 172.21.15.16:0/801325345 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fb85006b100 0x7fb85006d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:28.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.863+0000 7fb87204d700 1 --2- 172.21.15.16:0/801325345 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fb85006b100 0x7fb85006d5c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb868009130 tx=0x7fb868008030 comp rx=0 tx=0).ready entity=mgr.5499 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:28.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:28.867+0000 7fb862ffd700 1 -- 172.21.15.16:0/801325345 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb85c054310 con 0x7fb86c06e570 2023-10-16T12:23:29.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:29.097+0000 7fb8742b1700 1 -- 172.21.15.16:0/801325345 --> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir0", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fb86c15e080 con 0x7fb85006b100 2023-10-16T12:23:29.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:29.175+0000 7fb862ffd700 1 -- 172.21.15.16:0/801325345 <== mgr.5499 v2:172.21.15.16:6832/1707126831 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7fb8680041e0 con 0x7fb85006b100 2023-10-16T12:23:29.177 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule removed for path /snap_test_dir1/dir0 2023-10-16T12:23:29.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:29.177+0000 7fb8742b1700 1 -- 172.21.15.16:0/801325345 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fb85006b100 msgr2=0x7fb85006d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:29.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:29.177+0000 7fb8742b1700 1 --2- 172.21.15.16:0/801325345 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fb85006b100 0x7fb85006d5c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb868009130 tx=0x7fb868008030 comp rx=0 tx=0).stop 2023-10-16T12:23:29.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:29.177+0000 7fb8742b1700 1 -- 172.21.15.16:0/801325345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb86c06e570 msgr2=0x7fb86c0c8dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:29.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:29.177+0000 7fb8742b1700 1 --2- 172.21.15.16:0/801325345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb86c06e570 0x7fb86c0c8dc0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fb85c00c7f0 tx=0x7fb85c00cb00 comp rx=0 tx=0).stop 2023-10-16T12:23:29.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:29.177+0000 7fb8742b1700 1 -- 172.21.15.16:0/801325345 shutdown_connections 2023-10-16T12:23:29.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:29.177+0000 7fb8742b1700 1 --2- 172.21.15.16:0/801325345 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fb85006b100 0x7fb85006d5c0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:29.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:29.177+0000 7fb8742b1700 1 --2- 172.21.15.16:0/801325345 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb86c15b2c0 0x7fb86c15f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:29.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:29.177+0000 7fb8742b1700 1 --2- 172.21.15.16:0/801325345 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb86c06a7e0 0x7fb86c1579e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:29.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:29.177+0000 7fb8742b1700 1 --2- 172.21.15.16:0/801325345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb86c06e570 0x7fb86c0c8dc0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:29.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:29.177+0000 7fb8742b1700 1 -- 172.21.15.16:0/801325345 >> 172.21.15.16:0/801325345 conn(0x7fb86c0b1f40 msgr2=0x7fb86c0cb6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:29.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:29.178+0000 7fb8742b1700 1 -- 172.21.15.16:0/801325345 shutdown_connections 2023-10-16T12:23:29.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:29.178+0000 7fb8742b1700 1 -- 172.21.15.16:0/801325345 wait complete. 2023-10-16T12:23:29.191 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir0 2023-10-16T12:23:29.192 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:29.192 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir0/.snap) 2023-10-16T12:23:29.210 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_13_00_UTC 2023-10-16T12:23:29.210 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_14_00_UTC 2023-10-16T12:23:29.211 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_15_00_UTC 2023-10-16T12:23:29.211 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_16_00_UTC 2023-10-16T12:23:29.211 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_17_00_UTC 2023-10-16T12:23:29.211 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_18_00_UTC 2023-10-16T12:23:29.211 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_19_00_UTC 2023-10-16T12:23:29.212 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_20_00_UTC 2023-10-16T12:23:29.212 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_21_00_UTC 2023-10-16T12:23:29.212 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_22_00_UTC 2023-10-16T12:23:29.212 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_23_00_UTC 2023-10-16T12:23:29.212 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-16-12_13_00_UTC 2023-10-16T12:23:29.213 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:29.213 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-16-12_13_00_UTC) 2023-10-16T12:23:29.275 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-16-12_14_00_UTC 2023-10-16T12:23:29.275 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:29.275 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-16-12_14_00_UTC) 2023-10-16T12:23:29.338 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-16-12_15_00_UTC 2023-10-16T12:23:29.338 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:29.339 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-16-12_15_00_UTC) 2023-10-16T12:23:29.402 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-16-12_16_00_UTC 2023-10-16T12:23:29.402 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:29.402 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-16-12_16_00_UTC) 2023-10-16T12:23:29.465 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-16-12_17_00_UTC 2023-10-16T12:23:29.466 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:29.466 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-16-12_17_00_UTC) 2023-10-16T12:23:29.529 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-16-12_18_00_UTC 2023-10-16T12:23:29.530 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:29.530 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-16-12_18_00_UTC) 2023-10-16T12:23:29.594 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-16-12_19_00_UTC 2023-10-16T12:23:29.594 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:29.595 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-16-12_19_00_UTC) 2023-10-16T12:23:29.656 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-16-12_20_00_UTC 2023-10-16T12:23:29.657 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:29.657 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-16-12_20_00_UTC) 2023-10-16T12:23:29.720 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-16-12_21_00_UTC 2023-10-16T12:23:29.721 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:29.721 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-16-12_21_00_UTC) 2023-10-16T12:23:29.784 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-16-12_22_00_UTC 2023-10-16T12:23:29.785 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:29.785 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-16-12_22_00_UTC) 2023-10-16T12:23:29.848 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-16-12_23_00_UTC 2023-10-16T12:23:29.849 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:29.849 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-16-12_23_00_UTC) 2023-10-16T12:23:29.913 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:29.913 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0) 2023-10-16T12:23:29.973 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir1 1m 2023-10-16T12:23:30.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:30.138+0000 7f3932ee4700 1 Processor -- start 2023-10-16T12:23:30.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:30.138+0000 7f3932ee4700 1 -- start start 2023-10-16T12:23:30.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:30.138+0000 7f3932ee4700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f392c072b30 0x7f392c071a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:30.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:30.138+0000 7f3932ee4700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f392c0caa40 0x7f392c0cae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:30.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:30.138+0000 7f3932ee4700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f392c0cb390 con 0x7f392c15bb40 2023-10-16T12:23:30.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:30.138+0000 7f3932ee4700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f392c06ea20 con 0x7f392c072b30 2023-10-16T12:23:30.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:30.138+0000 7f3932ee4700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f392c15c040 con 0x7f392c0caa40 2023-10-16T12:23:30.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:30.139+0000 7f3930c80700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f392c072b30 0x7f392c071a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:30.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:30.139+0000 7f392bfff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f392c15bb40 0x7f392c06e310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54306/0 (socket says 172.21.15.16:54306) 2023-10-16T12:23:30.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:30.139+0000 7f392bfff700 1 -- 172.21.15.16:0/4179025871 learned_addr learned my addr 172.21.15.16:0/4179025871 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:23:30.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:30.139+0000 7f3930c80700 1 -- 172.21.15.16:0/4179025871 >> v2:172.21.15.49:3300/0 conn(0x7f392c0caa40 msgr2=0x7f392c0cae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:30.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:30.139+0000 7f3930c80700 1 --2- 172.21.15.16:0/4179025871 >> v2:172.21.15.49:3300/0 conn(0x7f392c0caa40 0x7f392c0cae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:30.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:30.139+0000 7f3930c80700 1 -- 172.21.15.16:0/4179025871 >> v1:172.21.15.16:6789/0 conn(0x7f392c15bb40 legacy=0x7f392c06e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:30.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:30.139+0000 7f3930c80700 1 -- 172.21.15.16:0/4179025871 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f392c15c320 con 0x7f392c072b30 2023-10-16T12:23:30.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:30.139+0000 7f3930c80700 1 --2- 172.21.15.16:0/4179025871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f392c072b30 0x7f392c071a70 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f392c159680 tx=0x7f3920009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24344acf39663545 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:30.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:30.139+0000 7f392b7fe700 1 -- 172.21.15.16:0/4179025871 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3920011830 con 0x7f392c072b30 2023-10-16T12:23:30.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:30.139+0000 7f392b7fe700 1 -- 172.21.15.16:0/4179025871 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3920011990 con 0x7f392c072b30 2023-10-16T12:23:30.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:30.139+0000 7f392b7fe700 1 -- 172.21.15.16:0/4179025871 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3920011d20 con 0x7f392c072b30 2023-10-16T12:23:30.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:30.140+0000 7f3932ee4700 1 -- 172.21.15.16:0/4179025871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f392c072b30 msgr2=0x7f392c071a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:30.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:30.140+0000 7f3932ee4700 1 --2- 172.21.15.16:0/4179025871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f392c072b30 0x7f392c071a70 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f392c159680 tx=0x7f3920009b20 comp rx=0 tx=0).stop 2023-10-16T12:23:30.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:30.140+0000 7f3932ee4700 1 -- 172.21.15.16:0/4179025871 shutdown_connections 2023-10-16T12:23:30.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:30.140+0000 7f3932ee4700 1 --2- 172.21.15.16:0/4179025871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f392c072b30 0x7f392c071a70 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:30.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:30.140+0000 7f3932ee4700 1 --2- 172.21.15.16:0/4179025871 >> v2:172.21.15.49:3300/0 conn(0x7f392c0caa40 0x7f392c0cae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:30.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:30.140+0000 7f3932ee4700 1 -- 172.21.15.16:0/4179025871 >> 172.21.15.16:0/4179025871 conn(0x7f392c0b1f40 msgr2=0x7f392c0b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:30.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:30.140+0000 7f3932ee4700 1 -- 172.21.15.16:0/4179025871 shutdown_connections 2023-10-16T12:23:30.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:30.140+0000 7f3932ee4700 1 -- 172.21.15.16:0/4179025871 wait complete. 2023-10-16T12:23:30.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:30.141+0000 7f3932ee4700 1 Processor -- start 2023-10-16T12:23:30.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:30.141+0000 7f3932ee4700 1 -- start start 2023-10-16T12:23:30.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:30.142+0000 7f3932ee4700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f392c072b30 0x7f392c0ad2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:30.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:30.142+0000 7f3932ee4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f392c0caa40 0x7f392c152b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:30.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:30.142+0000 7f3932ee4700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f392c15bb40 0x7f392c0d0380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:30.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:30.142+0000 7f3932ee4700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f392c15d1f0 con 0x7f392c0caa40 2023-10-16T12:23:30.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:30.142+0000 7f3932ee4700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f392c0ad820 con 0x7f392c15bb40 2023-10-16T12:23:30.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:30.142+0000 7f3932ee4700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f392c0adad0 con 0x7f392c072b30 2023-10-16T12:23:30.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:30.142+0000 7f392bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f392c0caa40 0x7f392c152b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:30.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:30.142+0000 7f392bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f392c0caa40 0x7f392c152b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36806/0 (socket says 172.21.15.16:36806) 2023-10-16T12:23:30.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:30.142+0000 7f392bfff700 1 -- 172.21.15.16:0/3497497554 learned_addr learned my addr 172.21.15.16:0/3497497554 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:30.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:30.142+0000 7f3931481700 1 --2- 172.21.15.16:0/3497497554 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f392c15bb40 0x7f392c0d0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:30.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:30.142+0000 7f3930c80700 1 --2- 172.21.15.16:0/3497497554 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f392c072b30 0x7f392c0ad2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:30.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:30.142+0000 7f392bfff700 1 -- 172.21.15.16:0/3497497554 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f392c072b30 msgr2=0x7f392c0ad2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:30.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:30.142+0000 7f392bfff700 1 --2- 172.21.15.16:0/3497497554 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f392c072b30 0x7f392c0ad2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:30.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:30.142+0000 7f392bfff700 1 -- 172.21.15.16:0/3497497554 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f392c15bb40 msgr2=0x7f392c0d0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:30.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:30.142+0000 7f392bfff700 1 --2- 172.21.15.16:0/3497497554 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f392c15bb40 0x7f392c0d0380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:30.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:30.142+0000 7f392bfff700 1 -- 172.21.15.16:0/3497497554 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39200094a0 con 0x7f392c0caa40 2023-10-16T12:23:30.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:30.142+0000 7f3931481700 1 --2- 172.21.15.16:0/3497497554 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f392c15bb40 0x7f392c0d0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:23:30.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:30.143+0000 7f392bfff700 1 --2- 172.21.15.16:0/3497497554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f392c0caa40 0x7f392c152b40 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f391800c500 tx=0x7f391800c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:30.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:30.143+0000 7f39297fa700 1 -- 172.21.15.16:0/3497497554 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3918005d70 con 0x7f392c0caa40 2023-10-16T12:23:30.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:30.143+0000 7f39297fa700 1 -- 172.21.15.16:0/3497497554 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39180070d0 con 0x7f392c0caa40 2023-10-16T12:23:30.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:30.143+0000 7f3932ee4700 1 -- 172.21.15.16:0/3497497554 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f392c0addb0 con 0x7f392c0caa40 2023-10-16T12:23:30.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:30.143+0000 7f39297fa700 1 -- 172.21.15.16:0/3497497554 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3918007440 con 0x7f392c0caa40 2023-10-16T12:23:30.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:30.143+0000 7f3932ee4700 1 -- 172.21.15.16:0/3497497554 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f392c0ae370 con 0x7f392c0caa40 2023-10-16T12:23:30.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:30.144+0000 7f3932ee4700 1 -- 172.21.15.16:0/3497497554 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f392c15d360 con 0x7f392c0caa40 2023-10-16T12:23:30.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:30.144+0000 7f39297fa700 1 -- 172.21.15.16:0/3497497554 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f39180075a0 con 0x7f392c0caa40 2023-10-16T12:23:30.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:30.144+0000 7f39297fa700 1 --2- 172.21.15.16:0/3497497554 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f391006b100 0x7f391006d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:30.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:30.145+0000 7f39297fa700 1 -- 172.21.15.16:0/3497497554 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f3918088e50 con 0x7f392c0caa40 2023-10-16T12:23:30.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:30.145+0000 7f3930c80700 1 --2- 172.21.15.16:0/3497497554 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f391006b100 0x7f391006d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:30.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:30.145+0000 7f3930c80700 1 --2- 172.21.15.16:0/3497497554 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f391006b100 0x7f391006d5c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3920000c00 tx=0x7f392001c040 comp rx=0 tx=0).ready entity=mgr.5499 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:30.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:30.149+0000 7f39297fa700 1 -- 172.21.15.16:0/3497497554 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3918053b60 con 0x7f392c0caa40 2023-10-16T12:23:30.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:30.383+0000 7f3932ee4700 1 -- 172.21.15.16:0/3497497554 --> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir1", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f392c15c8e0 con 0x7f391006b100 2023-10-16T12:23:30.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:30.460+0000 7f39297fa700 1 -- 172.21.15.16:0/3497497554 <== mgr.5499 v2:172.21.15.16:6832/1707126831 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f3920006c60 con 0x7f391006b100 2023-10-16T12:23:30.462 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule removed for path /snap_test_dir1/dir1 2023-10-16T12:23:30.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:30.463+0000 7f3932ee4700 1 -- 172.21.15.16:0/3497497554 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f391006b100 msgr2=0x7f391006d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:30.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:30.463+0000 7f3932ee4700 1 --2- 172.21.15.16:0/3497497554 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f391006b100 0x7f391006d5c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3920000c00 tx=0x7f392001c040 comp rx=0 tx=0).stop 2023-10-16T12:23:30.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:30.463+0000 7f3932ee4700 1 -- 172.21.15.16:0/3497497554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f392c0caa40 msgr2=0x7f392c152b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:30.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:30.463+0000 7f3932ee4700 1 --2- 172.21.15.16:0/3497497554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f392c0caa40 0x7f392c152b40 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f391800c500 tx=0x7f391800c810 comp rx=0 tx=0).stop 2023-10-16T12:23:30.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:30.463+0000 7f3932ee4700 1 -- 172.21.15.16:0/3497497554 shutdown_connections 2023-10-16T12:23:30.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:30.463+0000 7f3932ee4700 1 --2- 172.21.15.16:0/3497497554 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f391006b100 0x7f391006d5c0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:30.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:30.463+0000 7f3932ee4700 1 --2- 172.21.15.16:0/3497497554 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f392c15bb40 0x7f392c0d0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:30.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:30.463+0000 7f3932ee4700 1 --2- 172.21.15.16:0/3497497554 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f392c072b30 0x7f392c0ad2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:30.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:30.463+0000 7f3932ee4700 1 --2- 172.21.15.16:0/3497497554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f392c0caa40 0x7f392c152b40 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:30.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:30.463+0000 7f3932ee4700 1 -- 172.21.15.16:0/3497497554 >> 172.21.15.16:0/3497497554 conn(0x7f392c0b1f40 msgr2=0x7f392c1599c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:30.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:30.463+0000 7f3932ee4700 1 -- 172.21.15.16:0/3497497554 shutdown_connections 2023-10-16T12:23:30.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:30.463+0000 7f3932ee4700 1 -- 172.21.15.16:0/3497497554 wait complete. 2023-10-16T12:23:30.476 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir1 2023-10-16T12:23:30.477 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:30.477 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir1/.snap) 2023-10-16T12:23:30.496 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_13_00_UTC 2023-10-16T12:23:30.496 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_14_00_UTC 2023-10-16T12:23:30.496 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_15_00_UTC 2023-10-16T12:23:30.496 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_16_00_UTC 2023-10-16T12:23:30.496 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_17_00_UTC 2023-10-16T12:23:30.497 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_18_00_UTC 2023-10-16T12:23:30.497 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_19_00_UTC 2023-10-16T12:23:30.497 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_20_00_UTC 2023-10-16T12:23:30.497 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_21_00_UTC 2023-10-16T12:23:30.497 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_22_00_UTC 2023-10-16T12:23:30.497 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_23_00_UTC 2023-10-16T12:23:30.498 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-16-12_13_00_UTC 2023-10-16T12:23:30.498 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:30.498 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-16-12_13_00_UTC) 2023-10-16T12:23:30.561 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-16-12_14_00_UTC 2023-10-16T12:23:30.561 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:30.561 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-16-12_14_00_UTC) 2023-10-16T12:23:30.624 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-16-12_15_00_UTC 2023-10-16T12:23:30.624 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:30.624 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-16-12_15_00_UTC) 2023-10-16T12:23:30.688 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-16-12_16_00_UTC 2023-10-16T12:23:30.688 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:30.689 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-16-12_16_00_UTC) 2023-10-16T12:23:30.751 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-16-12_17_00_UTC 2023-10-16T12:23:30.752 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:30.752 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-16-12_17_00_UTC) 2023-10-16T12:23:30.816 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-16-12_18_00_UTC 2023-10-16T12:23:30.816 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:30.817 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-16-12_18_00_UTC) 2023-10-16T12:23:30.880 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-16-12_19_00_UTC 2023-10-16T12:23:30.880 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:30.881 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-16-12_19_00_UTC) 2023-10-16T12:23:30.944 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-16-12_20_00_UTC 2023-10-16T12:23:30.944 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:30.945 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-16-12_20_00_UTC) 2023-10-16T12:23:31.007 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-16-12_21_00_UTC 2023-10-16T12:23:31.008 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:31.008 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-16-12_21_00_UTC) 2023-10-16T12:23:31.071 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-16-12_22_00_UTC 2023-10-16T12:23:31.071 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:31.071 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-16-12_22_00_UTC) 2023-10-16T12:23:31.134 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-16-12_23_00_UTC 2023-10-16T12:23:31.134 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:31.135 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-16-12_23_00_UTC) 2023-10-16T12:23:31.198 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:31.198 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1) 2023-10-16T12:23:31.260 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir2 1m 2023-10-16T12:23:31.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.428+0000 7ff88eb46700 1 Processor -- start 2023-10-16T12:23:31.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.428+0000 7ff88eb46700 1 -- start start 2023-10-16T12:23:31.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.428+0000 7ff88eb46700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7ff88815da10 0x7ff8880d1520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:31.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.429+0000 7ff88eb46700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff8880d1d20 0x7ff8880d21b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:31.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.429+0000 7ff88eb46700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8880d4540 con 0x7ff8880d4060 2023-10-16T12:23:31.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.429+0000 7ff88eb46700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8880cf9b0 con 0x7ff88815da10 2023-10-16T12:23:31.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.429+0000 7ff88eb46700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8880d26f0 con 0x7ff8880d1d20 2023-10-16T12:23:31.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.429+0000 7ff88d0e3700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff8880d1d20 0x7ff8880d21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:31.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.429+0000 7ff88c8e2700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7ff88815da10 0x7ff8880d1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:31.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.429+0000 7ff88d0e3700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff8880d1d20 0x7ff8880d21b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:53306/0 (socket says 172.21.15.16:53306) 2023-10-16T12:23:31.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.429+0000 7ff88c8e2700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7ff88815da10 0x7ff8880d1520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:35874/0 (socket says 172.21.15.16:35874) 2023-10-16T12:23:31.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.429+0000 7ff887fff700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7ff8880d4060 0x7ff8880cf2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:33268/0 (socket says 172.21.15.16:33268) 2023-10-16T12:23:31.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.429+0000 7ff88d0e3700 1 -- 172.21.15.16:0/1595595268 learned_addr learned my addr 172.21.15.16:0/1595595268 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:31.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.429+0000 7ff88c8e2700 1 -- 172.21.15.16:0/1595595268 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff8880d1d20 msgr2=0x7ff8880d21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:31.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.429+0000 7ff88c8e2700 1 --2- 172.21.15.16:0/1595595268 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff8880d1d20 0x7ff8880d21b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:31.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.429+0000 7ff88c8e2700 1 -- 172.21.15.16:0/1595595268 >> v1:172.21.15.49:6789/0 conn(0x7ff8880d4060 legacy=0x7ff8880cf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:31.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.429+0000 7ff88c8e2700 1 -- 172.21.15.16:0/1595595268 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8880ce7e0 con 0x7ff88815da10 2023-10-16T12:23:31.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.429+0000 7ff8877fe700 1 -- 172.21.15.16:0/1595595268 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 85130670 0 0) 0x7ff8880d4540 con 0x7ff8880d4060 2023-10-16T12:23:31.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.429+0000 7ff88d0e3700 1 --2- 172.21.15.16:0/1595595268 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff8880d1d20 0x7ff8880d21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:23:31.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.430+0000 7ff88c8e2700 1 --2- 172.21.15.16:0/1595595268 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff88815da10 0x7ff8880d1520 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ff87c009820 tx=0x7ff87c009b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6958146e85efbee server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:31.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.430+0000 7ff8877fe700 1 -- 172.21.15.16:0/1595595268 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff87c005e30 con 0x7ff88815da10 2023-10-16T12:23:31.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.430+0000 7ff8877fe700 1 -- 172.21.15.16:0/1595595268 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff87c006030 con 0x7ff88815da10 2023-10-16T12:23:31.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.430+0000 7ff8877fe700 1 -- 172.21.15.16:0/1595595268 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff87c0063c0 con 0x7ff88815da10 2023-10-16T12:23:31.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.430+0000 7ff88eb46700 1 -- 172.21.15.16:0/1595595268 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff88815da10 msgr2=0x7ff8880d1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:31.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.430+0000 7ff88eb46700 1 --2- 172.21.15.16:0/1595595268 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff88815da10 0x7ff8880d1520 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ff87c009820 tx=0x7ff87c009b30 comp rx=0 tx=0).stop 2023-10-16T12:23:31.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.430+0000 7ff88eb46700 1 -- 172.21.15.16:0/1595595268 shutdown_connections 2023-10-16T12:23:31.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.430+0000 7ff88eb46700 1 --2- 172.21.15.16:0/1595595268 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff8880d1d20 0x7ff8880d21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:31.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.430+0000 7ff88eb46700 1 --2- 172.21.15.16:0/1595595268 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff88815da10 0x7ff8880d1520 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:31.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.430+0000 7ff88eb46700 1 -- 172.21.15.16:0/1595595268 >> 172.21.15.16:0/1595595268 conn(0x7ff88809c9f0 msgr2=0x7ff8880d3740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:31.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.430+0000 7ff88eb46700 1 -- 172.21.15.16:0/1595595268 shutdown_connections 2023-10-16T12:23:31.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.430+0000 7ff88eb46700 1 -- 172.21.15.16:0/1595595268 wait complete. 2023-10-16T12:23:31.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.431+0000 7ff88eb46700 1 Processor -- start 2023-10-16T12:23:31.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.432+0000 7ff88eb46700 1 -- start start 2023-10-16T12:23:31.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.432+0000 7ff88eb46700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff8880d1d20 0x7ff88809f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:31.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.432+0000 7ff88eb46700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff8880d4060 0x7ff888153bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:31.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.432+0000 7ff88eb46700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff88815da10 0x7ff8880b6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:31.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.432+0000 7ff88eb46700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff888154100 con 0x7ff88815da10 2023-10-16T12:23:31.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.432+0000 7ff88eb46700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff88809f980 con 0x7ff8880d4060 2023-10-16T12:23:31.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.432+0000 7ff88eb46700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff88809fc60 con 0x7ff8880d1d20 2023-10-16T12:23:31.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.432+0000 7ff88d0e3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff88815da10 0x7ff8880b6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:31.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.433+0000 7ff88d0e3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff88815da10 0x7ff8880b6ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36822/0 (socket says 172.21.15.16:36822) 2023-10-16T12:23:31.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.433+0000 7ff88d0e3700 1 -- 172.21.15.16:0/554806374 learned_addr learned my addr 172.21.15.16:0/554806374 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:31.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.433+0000 7ff887fff700 1 --2- 172.21.15.16:0/554806374 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff8880d4060 0x7ff888153bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:31.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.433+0000 7ff88c8e2700 1 --2- 172.21.15.16:0/554806374 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff8880d1d20 0x7ff88809f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:31.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.433+0000 7ff88d0e3700 1 -- 172.21.15.16:0/554806374 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff8880d1d20 msgr2=0x7ff88809f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:31.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.433+0000 7ff88d0e3700 1 --2- 172.21.15.16:0/554806374 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff8880d1d20 0x7ff88809f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:31.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.433+0000 7ff88d0e3700 1 -- 172.21.15.16:0/554806374 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff8880d4060 msgr2=0x7ff888153bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:31.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.433+0000 7ff88d0e3700 1 --2- 172.21.15.16:0/554806374 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff8880d4060 0x7ff888153bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:31.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.433+0000 7ff88d0e3700 1 -- 172.21.15.16:0/554806374 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff880009b30 con 0x7ff88815da10 2023-10-16T12:23:31.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.433+0000 7ff88c8e2700 1 --2- 172.21.15.16:0/554806374 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff8880d1d20 0x7ff88809f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:23:31.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.433+0000 7ff887fff700 1 --2- 172.21.15.16:0/554806374 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff8880d4060 0x7ff888153bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:23:31.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.433+0000 7ff88d0e3700 1 --2- 172.21.15.16:0/554806374 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff88815da10 0x7ff8880b6ba0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7ff88000efc0 tx=0x7ff880009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:31.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.433+0000 7ff8857fa700 1 -- 172.21.15.16:0/554806374 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff880018070 con 0x7ff88815da10 2023-10-16T12:23:31.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.433+0000 7ff8857fa700 1 -- 172.21.15.16:0/554806374 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff88000f040 con 0x7ff88815da10 2023-10-16T12:23:31.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.433+0000 7ff88eb46700 1 -- 172.21.15.16:0/554806374 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff87c009430 con 0x7ff88815da10 2023-10-16T12:23:31.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.433+0000 7ff8857fa700 1 -- 172.21.15.16:0/554806374 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff880003c40 con 0x7ff88815da10 2023-10-16T12:23:31.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.433+0000 7ff88eb46700 1 -- 172.21.15.16:0/554806374 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8880a0220 con 0x7ff88815da10 2023-10-16T12:23:31.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.434+0000 7ff88eb46700 1 -- 172.21.15.16:0/554806374 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff888067870 con 0x7ff88815da10 2023-10-16T12:23:31.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.435+0000 7ff8857fa700 1 -- 172.21.15.16:0/554806374 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7ff88000c9e0 con 0x7ff88815da10 2023-10-16T12:23:31.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.435+0000 7ff8857fa700 1 --2- 172.21.15.16:0/554806374 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7ff86c06b100 0x7ff86c06d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:31.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.435+0000 7ff8857fa700 1 -- 172.21.15.16:0/554806374 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7ff880089f80 con 0x7ff88815da10 2023-10-16T12:23:31.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.435+0000 7ff88c8e2700 1 --2- 172.21.15.16:0/554806374 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7ff86c06b100 0x7ff86c06d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:31.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.435+0000 7ff88c8e2700 1 --2- 172.21.15.16:0/554806374 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7ff86c06b100 0x7ff86c06d5c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff87c013040 tx=0x7ff87c008070 comp rx=0 tx=0).ready entity=mgr.5499 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:31.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.440+0000 7ff8857fa700 1 -- 172.21.15.16:0/554806374 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff8800bcfe0 con 0x7ff88815da10 2023-10-16T12:23:31.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.672+0000 7ff88eb46700 1 -- 172.21.15.16:0/554806374 --> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir2", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7ff8880ceda0 con 0x7ff86c06b100 2023-10-16T12:23:31.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.750+0000 7ff8857fa700 1 -- 172.21.15.16:0/554806374 <== mgr.5499 v2:172.21.15.16:6832/1707126831 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7ff87c021070 con 0x7ff86c06b100 2023-10-16T12:23:31.752 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule removed for path /snap_test_dir1/dir2 2023-10-16T12:23:31.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.752+0000 7ff88eb46700 1 -- 172.21.15.16:0/554806374 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7ff86c06b100 msgr2=0x7ff86c06d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:31.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.752+0000 7ff88eb46700 1 --2- 172.21.15.16:0/554806374 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7ff86c06b100 0x7ff86c06d5c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff87c013040 tx=0x7ff87c008070 comp rx=0 tx=0).stop 2023-10-16T12:23:31.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.752+0000 7ff88eb46700 1 -- 172.21.15.16:0/554806374 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff88815da10 msgr2=0x7ff8880b6ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:31.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.752+0000 7ff88eb46700 1 --2- 172.21.15.16:0/554806374 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff88815da10 0x7ff8880b6ba0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7ff88000efc0 tx=0x7ff880009f90 comp rx=0 tx=0).stop 2023-10-16T12:23:31.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.752+0000 7ff88eb46700 1 -- 172.21.15.16:0/554806374 shutdown_connections 2023-10-16T12:23:31.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.752+0000 7ff88eb46700 1 --2- 172.21.15.16:0/554806374 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7ff86c06b100 0x7ff86c06d5c0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:31.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.752+0000 7ff88eb46700 1 --2- 172.21.15.16:0/554806374 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff88815da10 0x7ff8880b6ba0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:31.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.752+0000 7ff88eb46700 1 --2- 172.21.15.16:0/554806374 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff8880d1d20 0x7ff88809f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:31.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.752+0000 7ff88eb46700 1 --2- 172.21.15.16:0/554806374 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff8880d4060 0x7ff888153bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:31.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.752+0000 7ff88eb46700 1 -- 172.21.15.16:0/554806374 >> 172.21.15.16:0/554806374 conn(0x7ff88809c9f0 msgr2=0x7ff88815df30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:31.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.752+0000 7ff88eb46700 1 -- 172.21.15.16:0/554806374 shutdown_connections 2023-10-16T12:23:31.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:31.752+0000 7ff88eb46700 1 -- 172.21.15.16:0/554806374 wait complete. 2023-10-16T12:23:31.766 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir2 2023-10-16T12:23:31.766 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:31.767 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir2/.snap) 2023-10-16T12:23:31.785 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_13_00_UTC 2023-10-16T12:23:31.785 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_14_00_UTC 2023-10-16T12:23:31.785 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_15_00_UTC 2023-10-16T12:23:31.786 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_16_00_UTC 2023-10-16T12:23:31.786 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_17_00_UTC 2023-10-16T12:23:31.786 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_18_00_UTC 2023-10-16T12:23:31.786 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_19_00_UTC 2023-10-16T12:23:31.786 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_20_00_UTC 2023-10-16T12:23:31.787 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_21_00_UTC 2023-10-16T12:23:31.787 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_22_00_UTC 2023-10-16T12:23:31.787 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_23_00_UTC 2023-10-16T12:23:31.788 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-16-12_13_00_UTC 2023-10-16T12:23:31.789 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:31.789 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-16-12_13_00_UTC) 2023-10-16T12:23:31.853 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-16-12_14_00_UTC 2023-10-16T12:23:31.854 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:31.854 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-16-12_14_00_UTC) 2023-10-16T12:23:31.916 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-16-12_15_00_UTC 2023-10-16T12:23:31.917 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:31.917 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-16-12_15_00_UTC) 2023-10-16T12:23:31.980 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-16-12_16_00_UTC 2023-10-16T12:23:31.980 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:31.980 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-16-12_16_00_UTC) 2023-10-16T12:23:32.043 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-16-12_17_00_UTC 2023-10-16T12:23:32.043 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:32.044 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-16-12_17_00_UTC) 2023-10-16T12:23:32.106 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-16-12_18_00_UTC 2023-10-16T12:23:32.107 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:32.107 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-16-12_18_00_UTC) 2023-10-16T12:23:32.170 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-16-12_19_00_UTC 2023-10-16T12:23:32.170 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:32.171 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-16-12_19_00_UTC) 2023-10-16T12:23:32.234 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-16-12_20_00_UTC 2023-10-16T12:23:32.234 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:32.235 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-16-12_20_00_UTC) 2023-10-16T12:23:32.299 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-16-12_21_00_UTC 2023-10-16T12:23:32.300 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:32.300 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-16-12_21_00_UTC) 2023-10-16T12:23:32.363 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-16-12_22_00_UTC 2023-10-16T12:23:32.363 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:32.364 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-16-12_22_00_UTC) 2023-10-16T12:23:32.427 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-16-12_23_00_UTC 2023-10-16T12:23:32.427 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:32.428 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-16-12_23_00_UTC) 2023-10-16T12:23:32.491 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:32.491 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2) 2023-10-16T12:23:32.552 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir3 1m 2023-10-16T12:23:32.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:32.717+0000 7f16ab381700 1 Processor -- start 2023-10-16T12:23:32.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:32.717+0000 7f16ab381700 1 -- start start 2023-10-16T12:23:32.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:32.718+0000 7f16ab381700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f16a415ec80 0x7f16a40cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:32.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:32.718+0000 7f16ab381700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f16a40d2110 0x7f16a40cd1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:32.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:32.718+0000 7f16ab381700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16a40b2440 con 0x7f16a415f060 2023-10-16T12:23:32.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:32.718+0000 7f16ab381700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16a40b2c50 con 0x7f16a40d2110 2023-10-16T12:23:32.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:32.718+0000 7f16ab381700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16a40d00e0 con 0x7f16a415ec80 2023-10-16T12:23:32.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:32.718+0000 7f16a911d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f16a415ec80 0x7f16a40cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:32.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:32.719+0000 7f16a991e700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f16a40d2110 0x7f16a40cd1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:32.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:32.719+0000 7f16a911d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f16a415ec80 0x7f16a40cdec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:53316/0 (socket says 172.21.15.16:53316) 2023-10-16T12:23:32.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:32.719+0000 7f16a911d700 1 -- 172.21.15.16:0/3087965637 learned_addr learned my addr 172.21.15.16:0/3087965637 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:32.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:32.719+0000 7f16a891c700 1 --1- 172.21.15.16:0/3087965637 >> v1:172.21.15.49:6789/0 conn(0x7f16a415f060 0x7f16a40b4440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:33282/0 (socket says 172.21.15.16:33282) 2023-10-16T12:23:32.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:32.719+0000 7f16a911d700 1 -- 172.21.15.16:0/3087965637 >> v2:172.21.15.49:3300/0 conn(0x7f16a40d2110 msgr2=0x7f16a40cd1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:32.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:32.719+0000 7f16a911d700 1 --2- 172.21.15.16:0/3087965637 >> v2:172.21.15.49:3300/0 conn(0x7f16a40d2110 0x7f16a40cd1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:32.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:32.719+0000 7f16a911d700 1 -- 172.21.15.16:0/3087965637 >> v1:172.21.15.49:6789/0 conn(0x7f16a415f060 legacy=0x7f16a40b4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:32.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:32.719+0000 7f16a911d700 1 -- 172.21.15.16:0/3087965637 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16a40cef60 con 0x7f16a415ec80 2023-10-16T12:23:32.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:32.719+0000 7f16a991e700 1 --2- 172.21.15.16:0/3087965637 >> v2:172.21.15.49:3300/0 conn(0x7f16a40d2110 0x7f16a40cd1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:23:32.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:32.719+0000 7f16a911d700 1 --2- 172.21.15.16:0/3087965637 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f16a415ec80 0x7f16a40cdec0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f16a00098f0 tx=0x7f16a0009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a4e67cc41c7b12fe server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:32.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:32.719+0000 7f169bfff700 1 -- 172.21.15.16:0/3087965637 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16a0011830 con 0x7f16a415ec80 2023-10-16T12:23:32.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:32.719+0000 7f169bfff700 1 -- 172.21.15.16:0/3087965637 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16a0011990 con 0x7f16a415ec80 2023-10-16T12:23:32.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:32.719+0000 7f169bfff700 1 -- 172.21.15.16:0/3087965637 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16a0011d20 con 0x7f16a415ec80 2023-10-16T12:23:32.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:32.720+0000 7f16ab381700 1 -- 172.21.15.16:0/3087965637 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f16a415ec80 msgr2=0x7f16a40cdec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:32.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:32.720+0000 7f16ab381700 1 --2- 172.21.15.16:0/3087965637 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f16a415ec80 0x7f16a40cdec0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f16a00098f0 tx=0x7f16a0009c00 comp rx=0 tx=0).stop 2023-10-16T12:23:32.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:32.720+0000 7f16ab381700 1 -- 172.21.15.16:0/3087965637 shutdown_connections 2023-10-16T12:23:32.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:32.720+0000 7f16ab381700 1 --2- 172.21.15.16:0/3087965637 >> v2:172.21.15.49:3300/0 conn(0x7f16a40d2110 0x7f16a40cd1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:32.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:32.720+0000 7f16ab381700 1 --2- 172.21.15.16:0/3087965637 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f16a415ec80 0x7f16a40cdec0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:32.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:32.720+0000 7f16ab381700 1 -- 172.21.15.16:0/3087965637 >> 172.21.15.16:0/3087965637 conn(0x7f16a40d15f0 msgr2=0x7f16a40d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:32.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:32.720+0000 7f16ab381700 1 -- 172.21.15.16:0/3087965637 shutdown_connections 2023-10-16T12:23:32.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:32.720+0000 7f16ab381700 1 -- 172.21.15.16:0/3087965637 wait complete. 2023-10-16T12:23:32.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:32.721+0000 7f16ab381700 1 Processor -- start 2023-10-16T12:23:32.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:32.721+0000 7f16ab381700 1 -- start start 2023-10-16T12:23:32.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:32.722+0000 7f16ab381700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f16a40d2110 0x7f16a415b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:32.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:32.722+0000 7f16ab381700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f16a415ec80 0x7f16a40ca4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:32.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:32.722+0000 7f16ab381700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16a415f060 0x7f16a40c98e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:32.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:32.722+0000 7f16ab381700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16a415be40 con 0x7f16a415f060 2023-10-16T12:23:32.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:32.722+0000 7f16ab381700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16a41f44e0 con 0x7f16a40d2110 2023-10-16T12:23:32.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:32.722+0000 7f16ab381700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16a415c440 con 0x7f16a415ec80 2023-10-16T12:23:32.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:32.722+0000 7f16a991e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16a415f060 0x7f16a40c98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:32.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:32.722+0000 7f16a991e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16a415f060 0x7f16a40c98e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36836/0 (socket says 172.21.15.16:36836) 2023-10-16T12:23:32.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:32.722+0000 7f16a991e700 1 -- 172.21.15.16:0/4258225708 learned_addr learned my addr 172.21.15.16:0/4258225708 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:32.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:32.722+0000 7f16a891c700 1 --2- 172.21.15.16:0/4258225708 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f16a415ec80 0x7f16a40ca4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:32.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:32.722+0000 7f16a911d700 1 --2- 172.21.15.16:0/4258225708 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f16a40d2110 0x7f16a415b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:32.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:32.722+0000 7f16a991e700 1 -- 172.21.15.16:0/4258225708 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f16a415ec80 msgr2=0x7f16a40ca4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:32.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:32.722+0000 7f16a991e700 1 --2- 172.21.15.16:0/4258225708 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f16a415ec80 0x7f16a40ca4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:32.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:32.722+0000 7f16a991e700 1 -- 172.21.15.16:0/4258225708 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f16a40d2110 msgr2=0x7f16a415b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:32.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:32.722+0000 7f16a991e700 1 --2- 172.21.15.16:0/4258225708 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f16a40d2110 0x7f16a415b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:32.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:32.722+0000 7f16a991e700 1 -- 172.21.15.16:0/4258225708 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f169c009a70 con 0x7f16a415f060 2023-10-16T12:23:32.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:32.723+0000 7f16a991e700 1 --2- 172.21.15.16:0/4258225708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16a415f060 0x7f16a40c98e0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f169c00efc0 tx=0x7f169c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:32.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:32.723+0000 7f1699ffb700 1 -- 172.21.15.16:0/4258225708 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f169c00ca90 con 0x7f16a415f060 2023-10-16T12:23:32.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:32.723+0000 7f1699ffb700 1 -- 172.21.15.16:0/4258225708 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f169c00f040 con 0x7f16a415f060 2023-10-16T12:23:32.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:32.723+0000 7f16ab381700 1 -- 172.21.15.16:0/4258225708 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16a0009550 con 0x7f16a415f060 2023-10-16T12:23:32.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:32.723+0000 7f1699ffb700 1 -- 172.21.15.16:0/4258225708 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f169c00ce10 con 0x7f16a415f060 2023-10-16T12:23:32.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:32.723+0000 7f16ab381700 1 -- 172.21.15.16:0/4258225708 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16a41f1740 con 0x7f16a415f060 2023-10-16T12:23:32.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:32.724+0000 7f16ab381700 1 -- 172.21.15.16:0/4258225708 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16a41f4700 con 0x7f16a415f060 2023-10-16T12:23:32.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:32.724+0000 7f1699ffb700 1 -- 172.21.15.16:0/4258225708 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f169c014480 con 0x7f16a415f060 2023-10-16T12:23:32.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:32.725+0000 7f1699ffb700 1 --2- 172.21.15.16:0/4258225708 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f168806b100 0x7f168806d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:32.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:32.725+0000 7f1699ffb700 1 -- 172.21.15.16:0/4258225708 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f169c08a290 con 0x7f16a415f060 2023-10-16T12:23:32.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:32.725+0000 7f16a911d700 1 --2- 172.21.15.16:0/4258225708 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f168806b100 0x7f168806d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:32.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:32.725+0000 7f16a911d700 1 --2- 172.21.15.16:0/4258225708 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f168806b100 0x7f168806d5c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f16a0009c40 tx=0x7f16a0008030 comp rx=0 tx=0).ready entity=mgr.5499 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:32.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:32.729+0000 7f1699ffb700 1 -- 172.21.15.16:0/4258225708 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f169c054de0 con 0x7f16a415f060 2023-10-16T12:23:32.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:32.957+0000 7f16ab381700 1 -- 172.21.15.16:0/4258225708 --> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir3", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f16a40b2970 con 0x7f168806b100 2023-10-16T12:23:33.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:33.040+0000 7f1699ffb700 1 -- 172.21.15.16:0/4258225708 <== mgr.5499 v2:172.21.15.16:6832/1707126831 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f16a0011630 con 0x7f168806b100 2023-10-16T12:23:33.042 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule removed for path /snap_test_dir1/dir3 2023-10-16T12:23:33.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:33.042+0000 7f16ab381700 1 -- 172.21.15.16:0/4258225708 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f168806b100 msgr2=0x7f168806d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:33.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:33.042+0000 7f16ab381700 1 --2- 172.21.15.16:0/4258225708 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f168806b100 0x7f168806d5c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f16a0009c40 tx=0x7f16a0008030 comp rx=0 tx=0).stop 2023-10-16T12:23:33.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:33.042+0000 7f16ab381700 1 -- 172.21.15.16:0/4258225708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16a415f060 msgr2=0x7f16a40c98e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:33.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:33.042+0000 7f16ab381700 1 --2- 172.21.15.16:0/4258225708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16a415f060 0x7f16a40c98e0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f169c00efc0 tx=0x7f169c00c3a0 comp rx=0 tx=0).stop 2023-10-16T12:23:33.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:33.042+0000 7f16ab381700 1 -- 172.21.15.16:0/4258225708 shutdown_connections 2023-10-16T12:23:33.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:33.042+0000 7f16ab381700 1 --2- 172.21.15.16:0/4258225708 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f168806b100 0x7f168806d5c0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:33.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:33.042+0000 7f16ab381700 1 --2- 172.21.15.16:0/4258225708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16a415f060 0x7f16a40c98e0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:33.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:33.042+0000 7f16ab381700 1 --2- 172.21.15.16:0/4258225708 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f16a40d2110 0x7f16a415b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:33.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:33.042+0000 7f16ab381700 1 --2- 172.21.15.16:0/4258225708 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f16a415ec80 0x7f16a40ca4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:33.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:33.042+0000 7f16ab381700 1 -- 172.21.15.16:0/4258225708 >> 172.21.15.16:0/4258225708 conn(0x7f16a40d15f0 msgr2=0x7f16a4153ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:33.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:33.042+0000 7f16ab381700 1 -- 172.21.15.16:0/4258225708 shutdown_connections 2023-10-16T12:23:33.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:33.042+0000 7f16ab381700 1 -- 172.21.15.16:0/4258225708 wait complete. 2023-10-16T12:23:33.056 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir3 2023-10-16T12:23:33.057 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:33.057 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir3/.snap) 2023-10-16T12:23:33.075 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_13_00_UTC 2023-10-16T12:23:33.075 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_14_00_UTC 2023-10-16T12:23:33.076 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_15_00_UTC 2023-10-16T12:23:33.076 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_16_00_UTC 2023-10-16T12:23:33.076 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_17_00_UTC 2023-10-16T12:23:33.076 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_18_00_UTC 2023-10-16T12:23:33.076 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_19_00_UTC 2023-10-16T12:23:33.076 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_20_00_UTC 2023-10-16T12:23:33.077 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_21_00_UTC 2023-10-16T12:23:33.077 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_22_00_UTC 2023-10-16T12:23:33.077 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_23_00_UTC 2023-10-16T12:23:33.078 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-16-12_13_00_UTC 2023-10-16T12:23:33.078 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:33.078 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-16-12_13_00_UTC) 2023-10-16T12:23:33.142 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-16-12_14_00_UTC 2023-10-16T12:23:33.143 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:33.143 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-16-12_14_00_UTC) 2023-10-16T12:23:33.206 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-16-12_15_00_UTC 2023-10-16T12:23:33.206 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:33.207 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-16-12_15_00_UTC) 2023-10-16T12:23:33.271 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-16-12_16_00_UTC 2023-10-16T12:23:33.271 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:33.272 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-16-12_16_00_UTC) 2023-10-16T12:23:33.335 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-16-12_17_00_UTC 2023-10-16T12:23:33.335 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:33.336 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-16-12_17_00_UTC) 2023-10-16T12:23:33.444 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-16-12_18_00_UTC 2023-10-16T12:23:33.445 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:33.445 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-16-12_18_00_UTC) 2023-10-16T12:23:33.516 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-16-12_19_00_UTC 2023-10-16T12:23:33.517 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:33.517 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-16-12_19_00_UTC) 2023-10-16T12:23:33.553 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-16-12_20_00_UTC 2023-10-16T12:23:33.553 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:33.554 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-16-12_20_00_UTC) 2023-10-16T12:23:33.619 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-16-12_21_00_UTC 2023-10-16T12:23:33.620 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:33.620 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-16-12_21_00_UTC) 2023-10-16T12:23:33.687 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-16-12_22_00_UTC 2023-10-16T12:23:33.687 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:33.687 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-16-12_22_00_UTC) 2023-10-16T12:23:33.749 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-16-12_23_00_UTC 2023-10-16T12:23:33.750 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:33.750 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-16-12_23_00_UTC) 2023-10-16T12:23:33.812 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:33.812 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3) 2023-10-16T12:23:33.871 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir4 1m 2023-10-16T12:23:34.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:34.047+0000 7fceac396700 1 Processor -- start 2023-10-16T12:23:34.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:34.047+0000 7fceac396700 1 -- start start 2023-10-16T12:23:34.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:34.048+0000 7fceac396700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fcea415b790 0x7fcea40cda60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:34.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:34.048+0000 7fceac396700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcea415bb70 0x7fcea406d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:34.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:34.048+0000 7fceac396700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcea40ce0e0 con 0x7fcea406dea0 2023-10-16T12:23:34.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:34.048+0000 7fceac396700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcea406e2d0 con 0x7fcea415bb70 2023-10-16T12:23:34.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:34.048+0000 7fceac396700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcea406a7e0 con 0x7fcea415b790 2023-10-16T12:23:34.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:34.048+0000 7fcea9931700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcea415bb70 0x7fcea406d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:34.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:34.048+0000 7fceaa933700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fcea406dea0 0x7fcea415f1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54350/0 (socket says 172.21.15.16:54350) 2023-10-16T12:23:34.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:34.048+0000 7fcea9931700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcea415bb70 0x7fcea406d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36842/0 (socket says 172.21.15.16:36842) 2023-10-16T12:23:34.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:34.048+0000 7fceaa933700 1 -- 172.21.15.16:0/1160596161 learned_addr learned my addr 172.21.15.16:0/1160596161 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:23:34.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:34.048+0000 7fceaa132700 1 --2- 172.21.15.16:0/1160596161 >> v2:172.21.15.49:3300/0 conn(0x7fcea415b790 0x7fcea40cda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:34.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:34.049+0000 7fcea9931700 1 -- 172.21.15.16:0/1160596161 >> v2:172.21.15.49:3300/0 conn(0x7fcea415b790 msgr2=0x7fcea40cda60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:34.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:34.049+0000 7fcea9931700 1 --2- 172.21.15.16:0/1160596161 >> v2:172.21.15.49:3300/0 conn(0x7fcea415b790 0x7fcea40cda60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:34.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:34.049+0000 7fcea9931700 1 -- 172.21.15.16:0/1160596161 >> v1:172.21.15.16:6789/0 conn(0x7fcea406dea0 legacy=0x7fcea415f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:34.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:34.049+0000 7fcea9931700 1 -- 172.21.15.16:0/1160596161 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcea415c410 con 0x7fcea415bb70 2023-10-16T12:23:34.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:34.049+0000 7fcea9130700 1 -- 172.21.15.16:0/1160596161 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 594999740 0 0) 0x7fcea40ce0e0 con 0x7fcea406dea0 2023-10-16T12:23:34.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:34.049+0000 7fcea9931700 1 --2- 172.21.15.16:0/1160596161 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcea415bb70 0x7fcea406d960 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fcea4072220 tx=0x7fcea0009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a97cec14eaedb81d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:34.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:34.049+0000 7fcea9130700 1 -- 172.21.15.16:0/1160596161 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea0011830 con 0x7fcea415bb70 2023-10-16T12:23:34.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:34.049+0000 7fcea9130700 1 -- 172.21.15.16:0/1160596161 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcea0011990 con 0x7fcea415bb70 2023-10-16T12:23:34.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:34.049+0000 7fcea9130700 1 -- 172.21.15.16:0/1160596161 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea0011d20 con 0x7fcea415bb70 2023-10-16T12:23:34.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:34.049+0000 7fceac396700 1 -- 172.21.15.16:0/1160596161 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcea415bb70 msgr2=0x7fcea406d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:34.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:34.049+0000 7fceac396700 1 --2- 172.21.15.16:0/1160596161 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcea415bb70 0x7fcea406d960 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fcea4072220 tx=0x7fcea0009b20 comp rx=0 tx=0).stop 2023-10-16T12:23:34.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:34.050+0000 7fceac396700 1 -- 172.21.15.16:0/1160596161 shutdown_connections 2023-10-16T12:23:34.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:34.050+0000 7fceac396700 1 --2- 172.21.15.16:0/1160596161 >> v2:172.21.15.49:3300/0 conn(0x7fcea415b790 0x7fcea40cda60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:34.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:34.050+0000 7fceac396700 1 --2- 172.21.15.16:0/1160596161 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcea415bb70 0x7fcea406d960 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:34.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:34.050+0000 7fceac396700 1 -- 172.21.15.16:0/1160596161 >> 172.21.15.16:0/1160596161 conn(0x7fcea40cc5e0 msgr2=0x7fcea40cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:34.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:34.050+0000 7fceac396700 1 -- 172.21.15.16:0/1160596161 shutdown_connections 2023-10-16T12:23:34.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:34.050+0000 7fceac396700 1 -- 172.21.15.16:0/1160596161 wait complete. 2023-10-16T12:23:34.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:34.051+0000 7fceac396700 1 Processor -- start 2023-10-16T12:23:34.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:34.051+0000 7fceac396700 1 -- start start 2023-10-16T12:23:34.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:34.052+0000 7fceac396700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcea406dea0 0x7fcea41504c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:34.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:34.052+0000 7fceac396700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcea415b790 0x7fcea40cabc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:34.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:34.052+0000 7fceac396700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcea415bb70 0x7fcea41538c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:34.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:34.052+0000 7fceac396700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea40c37e0 con 0x7fcea415bb70 2023-10-16T12:23:34.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:34.052+0000 7fceac396700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea40cb530 con 0x7fcea406dea0 2023-10-16T12:23:34.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:34.052+0000 7fceac396700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcea40cb810 con 0x7fcea415b790 2023-10-16T12:23:34.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:34.053+0000 7fceaa132700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcea406dea0 0x7fcea41504c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:34.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:34.053+0000 7fceaa933700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcea415bb70 0x7fcea41538c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:34.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:34.053+0000 7fceaa933700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcea415bb70 0x7fcea41538c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36848/0 (socket says 172.21.15.16:36848) 2023-10-16T12:23:34.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:34.053+0000 7fceaa132700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcea406dea0 0x7fcea41504c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:35908/0 (socket says 172.21.15.16:35908) 2023-10-16T12:23:34.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:34.053+0000 7fceaa933700 1 -- 172.21.15.16:0/4088610642 learned_addr learned my addr 172.21.15.16:0/4088610642 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:34.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:34.053+0000 7fcea9931700 1 --2- 172.21.15.16:0/4088610642 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcea415b790 0x7fcea40cabc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:34.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:34.053+0000 7fceaa933700 1 -- 172.21.15.16:0/4088610642 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcea415b790 msgr2=0x7fcea40cabc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:34.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:34.053+0000 7fceaa933700 1 --2- 172.21.15.16:0/4088610642 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcea415b790 0x7fcea40cabc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:34.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:34.053+0000 7fceaa933700 1 -- 172.21.15.16:0/4088610642 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcea406dea0 msgr2=0x7fcea41504c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:34.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:34.053+0000 7fceaa933700 1 --2- 172.21.15.16:0/4088610642 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcea406dea0 0x7fcea41504c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:34.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:34.053+0000 7fceaa933700 1 -- 172.21.15.16:0/4088610642 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcea00094a0 con 0x7fcea415bb70 2023-10-16T12:23:34.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:34.054+0000 7fceaa933700 1 --2- 172.21.15.16:0/4088610642 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcea415bb70 0x7fcea41538c0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fce9c00cba0 tx=0x7fce9c00ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:34.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:34.054+0000 7fce9affd700 1 -- 172.21.15.16:0/4088610642 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce9c00aaa0 con 0x7fcea415bb70 2023-10-16T12:23:34.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:34.055+0000 7fce9affd700 1 -- 172.21.15.16:0/4088610642 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce9c00ac20 con 0x7fcea415bb70 2023-10-16T12:23:34.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:34.055+0000 7fce9affd700 1 -- 172.21.15.16:0/4088610642 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce9c0070d0 con 0x7fcea415bb70 2023-10-16T12:23:34.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:34.055+0000 7fceac396700 1 -- 172.21.15.16:0/4088610642 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcea40cbaf0 con 0x7fcea415bb70 2023-10-16T12:23:34.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:34.055+0000 7fceac396700 1 -- 172.21.15.16:0/4088610642 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcea40cb100 con 0x7fcea415bb70 2023-10-16T12:23:34.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:34.057+0000 7fceac396700 1 -- 172.21.15.16:0/4088610642 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcea40cb2f0 con 0x7fcea415bb70 2023-10-16T12:23:34.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:34.057+0000 7fce9affd700 1 -- 172.21.15.16:0/4088610642 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7fce9c007230 con 0x7fcea415bb70 2023-10-16T12:23:34.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:34.058+0000 7fce9affd700 1 --2- 172.21.15.16:0/4088610642 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fce8806b100 0x7fce8806d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:34.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:34.058+0000 7fce9affd700 1 -- 172.21.15.16:0/4088610642 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 7396+0+0 (secure 0 0 0) 0x7fce9c0899f0 con 0x7fcea415bb70 2023-10-16T12:23:34.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:34.066+0000 7fceaa132700 1 --2- 172.21.15.16:0/4088610642 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fce8806b100 0x7fce8806d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:34.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:34.066+0000 7fce9affd700 1 -- 172.21.15.16:0/4088610642 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcea40cb2f0 con 0x7fcea415bb70 2023-10-16T12:23:34.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:34.066+0000 7fceaa132700 1 --2- 172.21.15.16:0/4088610642 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fce8806b100 0x7fce8806d5c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fce94009be0 tx=0x7fce94009210 comp rx=0 tx=0).ready entity=mgr.5499 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:34.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:34.338+0000 7fceac396700 1 -- 172.21.15.16:0/4088610642 --> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir4", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fcea415c9d0 con 0x7fce8806b100 2023-10-16T12:23:34.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:34.414+0000 7fce9affd700 1 -- 172.21.15.16:0/4088610642 <== mgr.5499 v2:172.21.15.16:6832/1707126831 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7fce94004040 con 0x7fce8806b100 2023-10-16T12:23:34.416 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule removed for path /snap_test_dir1/dir4 2023-10-16T12:23:34.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:34.416+0000 7fceac396700 1 -- 172.21.15.16:0/4088610642 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fce8806b100 msgr2=0x7fce8806d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:34.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:34.416+0000 7fceac396700 1 --2- 172.21.15.16:0/4088610642 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fce8806b100 0x7fce8806d5c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fce94009be0 tx=0x7fce94009210 comp rx=0 tx=0).stop 2023-10-16T12:23:34.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:34.416+0000 7fceac396700 1 -- 172.21.15.16:0/4088610642 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcea415bb70 msgr2=0x7fcea41538c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:34.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:34.416+0000 7fceac396700 1 --2- 172.21.15.16:0/4088610642 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcea415bb70 0x7fcea41538c0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fce9c00cba0 tx=0x7fce9c00ceb0 comp rx=0 tx=0).stop 2023-10-16T12:23:34.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:34.416+0000 7fceac396700 1 -- 172.21.15.16:0/4088610642 shutdown_connections 2023-10-16T12:23:34.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:34.416+0000 7fceac396700 1 --2- 172.21.15.16:0/4088610642 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fce8806b100 0x7fce8806d5c0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:34.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:34.416+0000 7fceac396700 1 --2- 172.21.15.16:0/4088610642 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcea415bb70 0x7fcea41538c0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:34.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:34.416+0000 7fceac396700 1 --2- 172.21.15.16:0/4088610642 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcea406dea0 0x7fcea41504c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:34.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:34.416+0000 7fceac396700 1 --2- 172.21.15.16:0/4088610642 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcea415b790 0x7fcea40cabc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:34.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:34.416+0000 7fceac396700 1 -- 172.21.15.16:0/4088610642 >> 172.21.15.16:0/4088610642 conn(0x7fcea40cc5e0 msgr2=0x7fcea40cf2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:34.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:34.417+0000 7fceac396700 1 -- 172.21.15.16:0/4088610642 shutdown_connections 2023-10-16T12:23:34.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:34.417+0000 7fceac396700 1 -- 172.21.15.16:0/4088610642 wait complete. 2023-10-16T12:23:34.430 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir4 2023-10-16T12:23:34.431 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:34.431 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir4/.snap) 2023-10-16T12:23:34.451 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_13_00_UTC 2023-10-16T12:23:34.451 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_14_00_UTC 2023-10-16T12:23:34.451 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_15_00_UTC 2023-10-16T12:23:34.451 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_16_00_UTC 2023-10-16T12:23:34.452 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_17_00_UTC 2023-10-16T12:23:34.452 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_18_00_UTC 2023-10-16T12:23:34.452 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_19_00_UTC 2023-10-16T12:23:34.452 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_20_00_UTC 2023-10-16T12:23:34.452 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_21_00_UTC 2023-10-16T12:23:34.453 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_22_00_UTC 2023-10-16T12:23:34.453 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_23_00_UTC 2023-10-16T12:23:34.453 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-16-12_13_00_UTC 2023-10-16T12:23:34.454 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:34.454 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-16-12_13_00_UTC) 2023-10-16T12:23:34.518 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-16-12_14_00_UTC 2023-10-16T12:23:34.519 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:34.519 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-16-12_14_00_UTC) 2023-10-16T12:23:34.581 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-16-12_15_00_UTC 2023-10-16T12:23:34.582 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:34.582 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-16-12_15_00_UTC) 2023-10-16T12:23:34.645 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-16-12_16_00_UTC 2023-10-16T12:23:34.645 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:34.646 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-16-12_16_00_UTC) 2023-10-16T12:23:34.707 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-16-12_17_00_UTC 2023-10-16T12:23:34.708 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:34.708 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-16-12_17_00_UTC) 2023-10-16T12:23:34.770 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-16-12_18_00_UTC 2023-10-16T12:23:34.770 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:34.771 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-16-12_18_00_UTC) 2023-10-16T12:23:34.832 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-16-12_19_00_UTC 2023-10-16T12:23:34.832 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:34.833 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-16-12_19_00_UTC) 2023-10-16T12:23:34.895 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-16-12_20_00_UTC 2023-10-16T12:23:34.895 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:34.896 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-16-12_20_00_UTC) 2023-10-16T12:23:34.958 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-16-12_21_00_UTC 2023-10-16T12:23:34.959 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:34.959 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-16-12_21_00_UTC) 2023-10-16T12:23:35.029 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-16-12_22_00_UTC 2023-10-16T12:23:35.030 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:35.030 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-16-12_22_00_UTC) 2023-10-16T12:23:35.092 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-16-12_23_00_UTC 2023-10-16T12:23:35.093 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:35.093 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-16-12_23_00_UTC) 2023-10-16T12:23:35.156 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:35.156 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4) 2023-10-16T12:23:35.216 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir5 1m 2023-10-16T12:23:35.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.403+0000 7f2710e8b700 1 Processor -- start 2023-10-16T12:23:35.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.403+0000 7f2710e8b700 1 -- start start 2023-10-16T12:23:35.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.404+0000 7f2710e8b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f270c15ec80 0x7f270c0cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:35.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.404+0000 7f2710e8b700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f270c15f060 0x7f270c0ce400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:35.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.404+0000 7f2710e8b700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f270c0b2440 con 0x7f270c0d2110 2023-10-16T12:23:35.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.404+0000 7f2710e8b700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f270c0b2c50 con 0x7f270c15f060 2023-10-16T12:23:35.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.404+0000 7f2710e8b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f270c0d00e0 con 0x7f270c15ec80 2023-10-16T12:23:35.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.405+0000 7f2709d9b700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f270c15f060 0x7f270c0ce400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:35.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.405+0000 7f270a59c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f270c15ec80 0x7f270c0cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:35.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.405+0000 7f270ad9d700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7f270c0d2110 0x7f270c0b4440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:33312/0 (socket says 172.21.15.16:33312) 2023-10-16T12:23:35.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.405+0000 7f270ad9d700 1 -- 172.21.15.16:0/4044311564 learned_addr learned my addr 172.21.15.16:0/4044311564 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:23:35.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.405+0000 7f2709d9b700 1 --2- 172.21.15.16:0/4044311564 >> v2:172.21.15.49:3300/0 conn(0x7f270c15f060 0x7f270c0ce400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:35916/0 (socket says 172.21.15.16:35916) 2023-10-16T12:23:35.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.405+0000 7f270959a700 1 -- 172.21.15.16:0/4044311564 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 128248279 0 0) 0x7f270c0b2440 con 0x7f270c0d2110 2023-10-16T12:23:35.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.405+0000 7f270959a700 1 -- 172.21.15.16:0/4044311564 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26f40035b0 con 0x7f270c0d2110 2023-10-16T12:23:35.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.405+0000 7f270a59c700 1 -- 172.21.15.16:0/4044311564 >> v2:172.21.15.49:3300/0 conn(0x7f270c15f060 msgr2=0x7f270c0ce400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:35.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.405+0000 7f270a59c700 1 --2- 172.21.15.16:0/4044311564 >> v2:172.21.15.49:3300/0 conn(0x7f270c15f060 0x7f270c0ce400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:35.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.405+0000 7f270a59c700 1 -- 172.21.15.16:0/4044311564 >> v1:172.21.15.49:6789/0 conn(0x7f270c0d2110 legacy=0x7f270c0b4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:35.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.405+0000 7f270a59c700 1 -- 172.21.15.16:0/4044311564 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f270c0cef60 con 0x7f270c15ec80 2023-10-16T12:23:35.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.405+0000 7f2709d9b700 1 --2- 172.21.15.16:0/4044311564 >> v2:172.21.15.49:3300/0 conn(0x7f270c15f060 0x7f270c0ce400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:23:35.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.405+0000 7f270a59c700 1 --2- 172.21.15.16:0/4044311564 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f270c15ec80 0x7f270c0cdec0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f26fc009840 tx=0x7f26fc009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76a6e5abb3c729ac server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:35.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.405+0000 7f270959a700 1 -- 172.21.15.16:0/4044311564 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26fc011830 con 0x7f270c15ec80 2023-10-16T12:23:35.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.405+0000 7f270959a700 1 -- 172.21.15.16:0/4044311564 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26fc011990 con 0x7f270c15ec80 2023-10-16T12:23:35.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.405+0000 7f270959a700 1 -- 172.21.15.16:0/4044311564 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26fc011d20 con 0x7f270c15ec80 2023-10-16T12:23:35.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.406+0000 7f2710e8b700 1 -- 172.21.15.16:0/4044311564 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f270c15ec80 msgr2=0x7f270c0cdec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:35.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.406+0000 7f2710e8b700 1 --2- 172.21.15.16:0/4044311564 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f270c15ec80 0x7f270c0cdec0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f26fc009840 tx=0x7f26fc009b50 comp rx=0 tx=0).stop 2023-10-16T12:23:35.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.406+0000 7f2710e8b700 1 -- 172.21.15.16:0/4044311564 shutdown_connections 2023-10-16T12:23:35.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.406+0000 7f2710e8b700 1 --2- 172.21.15.16:0/4044311564 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f270c15ec80 0x7f270c0cdec0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:35.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.406+0000 7f2710e8b700 1 --2- 172.21.15.16:0/4044311564 >> v2:172.21.15.49:3300/0 conn(0x7f270c15f060 0x7f270c0ce400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:35.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.406+0000 7f2710e8b700 1 -- 172.21.15.16:0/4044311564 >> 172.21.15.16:0/4044311564 conn(0x7f270c0d15f0 msgr2=0x7f270c0d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:35.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.406+0000 7f2710e8b700 1 -- 172.21.15.16:0/4044311564 shutdown_connections 2023-10-16T12:23:35.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.406+0000 7f2710e8b700 1 -- 172.21.15.16:0/4044311564 wait complete. 2023-10-16T12:23:35.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.407+0000 7f2710e8b700 1 Processor -- start 2023-10-16T12:23:35.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.407+0000 7f2710e8b700 1 -- start start 2023-10-16T12:23:35.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.408+0000 7f2710e8b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f270c0d2110 0x7f270c159960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:35.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.408+0000 7f2710e8b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f270c15ec80 0x7f270c06d050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:35.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.408+0000 7f2710e8b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f270c15f060 0x7f270c06d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:35.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.408+0000 7f2710e8b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f270c06c7e0 con 0x7f270c0d2110 2023-10-16T12:23:35.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.408+0000 7f2710e8b700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f270c06b8f0 con 0x7f270c15ec80 2023-10-16T12:23:35.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.408+0000 7f2710e8b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f270c06bbd0 con 0x7f270c15f060 2023-10-16T12:23:35.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.408+0000 7f270a59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f270c0d2110 0x7f270c159960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:35.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.408+0000 7f270a59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f270c0d2110 0x7f270c159960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36860/0 (socket says 172.21.15.16:36860) 2023-10-16T12:23:35.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.408+0000 7f270a59c700 1 -- 172.21.15.16:0/657247858 learned_addr learned my addr 172.21.15.16:0/657247858 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:35.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.408+0000 7f270ad9d700 1 --2- 172.21.15.16:0/657247858 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f270c15f060 0x7f270c06d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:35.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.408+0000 7f2709d9b700 1 --2- 172.21.15.16:0/657247858 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f270c15ec80 0x7f270c06d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:35.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.409+0000 7f270a59c700 1 -- 172.21.15.16:0/657247858 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f270c15f060 msgr2=0x7f270c06d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:35.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.409+0000 7f270a59c700 1 --2- 172.21.15.16:0/657247858 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f270c15f060 0x7f270c06d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:35.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.409+0000 7f270a59c700 1 -- 172.21.15.16:0/657247858 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f270c15ec80 msgr2=0x7f270c06d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:35.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.409+0000 7f270a59c700 1 --2- 172.21.15.16:0/657247858 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f270c15ec80 0x7f270c06d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:35.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.409+0000 7f270a59c700 1 -- 172.21.15.16:0/657247858 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27000094a0 con 0x7f270c0d2110 2023-10-16T12:23:35.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.409+0000 7f270ad9d700 1 --2- 172.21.15.16:0/657247858 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f270c15f060 0x7f270c06d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:23:35.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.409+0000 7f270a59c700 1 --2- 172.21.15.16:0/657247858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f270c0d2110 0x7f270c159960 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f26fc0090b0 tx=0x7f26fc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:35.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.409+0000 7f26faffd700 1 -- 172.21.15.16:0/657247858 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26fc0036a0 con 0x7f270c0d2110 2023-10-16T12:23:35.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.409+0000 7f26faffd700 1 -- 172.21.15.16:0/657247858 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26fc01f070 con 0x7f270c0d2110 2023-10-16T12:23:35.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.409+0000 7f2710e8b700 1 -- 172.21.15.16:0/657247858 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26fc0094a0 con 0x7f270c0d2110 2023-10-16T12:23:35.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.409+0000 7f26faffd700 1 -- 172.21.15.16:0/657247858 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26fc003a40 con 0x7f270c0d2110 2023-10-16T12:23:35.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.409+0000 7f2710e8b700 1 -- 172.21.15.16:0/657247858 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f270c06c190 con 0x7f270c0d2110 2023-10-16T12:23:35.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.410+0000 7f2710e8b700 1 -- 172.21.15.16:0/657247858 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f270c04ebb0 con 0x7f270c0d2110 2023-10-16T12:23:35.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.411+0000 7f26faffd700 1 -- 172.21.15.16:0/657247858 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f26fc003ba0 con 0x7f270c0d2110 2023-10-16T12:23:35.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.411+0000 7f26faffd700 1 --2- 172.21.15.16:0/657247858 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f26ec06b100 0x7f26ec06d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:35.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.411+0000 7f26faffd700 1 -- 172.21.15.16:0/657247858 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6980+0+0 (secure 0 0 0) 0x7f26fc021070 con 0x7f270c0d2110 2023-10-16T12:23:35.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.411+0000 7f2709d9b700 1 --2- 172.21.15.16:0/657247858 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f26ec06b100 0x7f26ec06d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:35.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.411+0000 7f2709d9b700 1 --2- 172.21.15.16:0/657247858 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f26ec06b100 0x7f26ec06d5c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f270c158e10 tx=0x7f2700008040 comp rx=0 tx=0).ready entity=mgr.5499 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:35.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.415+0000 7f26faffd700 1 -- 172.21.15.16:0/657247858 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f26fc056070 con 0x7f270c0d2110 2023-10-16T12:23:35.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.652+0000 7f2710e8b700 1 -- 172.21.15.16:0/657247858 --> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir5", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f270c159080 con 0x7f26ec06b100 2023-10-16T12:23:35.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.733+0000 7f26faffd700 1 -- 172.21.15.16:0/657247858 <== mgr.5499 v2:172.21.15.16:6832/1707126831 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f270c159080 con 0x7f26ec06b100 2023-10-16T12:23:35.735 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule removed for path /snap_test_dir1/dir5 2023-10-16T12:23:35.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.735+0000 7f2710e8b700 1 -- 172.21.15.16:0/657247858 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f26ec06b100 msgr2=0x7f26ec06d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:35.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.735+0000 7f2710e8b700 1 --2- 172.21.15.16:0/657247858 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f26ec06b100 0x7f26ec06d5c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f270c158e10 tx=0x7f2700008040 comp rx=0 tx=0).stop 2023-10-16T12:23:35.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.735+0000 7f2710e8b700 1 -- 172.21.15.16:0/657247858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f270c0d2110 msgr2=0x7f270c159960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:35.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.735+0000 7f2710e8b700 1 --2- 172.21.15.16:0/657247858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f270c0d2110 0x7f270c159960 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f26fc0090b0 tx=0x7f26fc004750 comp rx=0 tx=0).stop 2023-10-16T12:23:35.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.735+0000 7f2710e8b700 1 -- 172.21.15.16:0/657247858 shutdown_connections 2023-10-16T12:23:35.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.735+0000 7f2710e8b700 1 --2- 172.21.15.16:0/657247858 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f26ec06b100 0x7f26ec06d5c0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:35.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.735+0000 7f2710e8b700 1 --2- 172.21.15.16:0/657247858 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f270c15f060 0x7f270c06d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:35.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.735+0000 7f2710e8b700 1 --2- 172.21.15.16:0/657247858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f270c0d2110 0x7f270c159960 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:35.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.735+0000 7f2710e8b700 1 --2- 172.21.15.16:0/657247858 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f270c15ec80 0x7f270c06d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:35.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.735+0000 7f2710e8b700 1 -- 172.21.15.16:0/657247858 >> 172.21.15.16:0/657247858 conn(0x7f270c0d15f0 msgr2=0x7f270c0cc190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:35.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.735+0000 7f2710e8b700 1 -- 172.21.15.16:0/657247858 shutdown_connections 2023-10-16T12:23:35.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:35.735+0000 7f2710e8b700 1 -- 172.21.15.16:0/657247858 wait complete. 2023-10-16T12:23:35.749 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir5 2023-10-16T12:23:35.749 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:35.750 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir5/.snap) 2023-10-16T12:23:35.768 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_13_00_UTC 2023-10-16T12:23:35.769 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_14_00_UTC 2023-10-16T12:23:35.769 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_15_00_UTC 2023-10-16T12:23:35.769 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_16_00_UTC 2023-10-16T12:23:35.769 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_17_00_UTC 2023-10-16T12:23:35.769 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_18_00_UTC 2023-10-16T12:23:35.770 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_19_00_UTC 2023-10-16T12:23:35.770 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_20_00_UTC 2023-10-16T12:23:35.770 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_21_00_UTC 2023-10-16T12:23:35.770 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_22_00_UTC 2023-10-16T12:23:35.770 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_23_00_UTC 2023-10-16T12:23:35.771 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-16-12_13_00_UTC 2023-10-16T12:23:35.771 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:35.771 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-16-12_13_00_UTC) 2023-10-16T12:23:35.833 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-16-12_14_00_UTC 2023-10-16T12:23:35.833 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:35.833 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-16-12_14_00_UTC) 2023-10-16T12:23:35.895 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-16-12_15_00_UTC 2023-10-16T12:23:35.896 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:35.896 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-16-12_15_00_UTC) 2023-10-16T12:23:35.959 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-16-12_16_00_UTC 2023-10-16T12:23:35.960 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:35.960 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-16-12_16_00_UTC) 2023-10-16T12:23:36.024 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-16-12_17_00_UTC 2023-10-16T12:23:36.024 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:36.024 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-16-12_17_00_UTC) 2023-10-16T12:23:36.093 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-16-12_18_00_UTC 2023-10-16T12:23:36.093 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:36.094 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-16-12_18_00_UTC) 2023-10-16T12:23:36.157 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-16-12_19_00_UTC 2023-10-16T12:23:36.158 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:36.158 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-16-12_19_00_UTC) 2023-10-16T12:23:36.223 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-16-12_20_00_UTC 2023-10-16T12:23:36.223 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:36.224 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-16-12_20_00_UTC) 2023-10-16T12:23:36.286 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-16-12_21_00_UTC 2023-10-16T12:23:36.287 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:36.287 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-16-12_21_00_UTC) 2023-10-16T12:23:36.350 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-16-12_22_00_UTC 2023-10-16T12:23:36.350 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:36.351 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-16-12_22_00_UTC) 2023-10-16T12:23:36.417 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-16-12_23_00_UTC 2023-10-16T12:23:36.418 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:36.418 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-16-12_23_00_UTC) 2023-10-16T12:23:36.481 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:36.481 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5) 2023-10-16T12:23:36.544 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir6 1m 2023-10-16T12:23:36.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:36.710+0000 7ff211792700 1 Processor -- start 2023-10-16T12:23:36.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:36.710+0000 7ff211792700 1 -- start start 2023-10-16T12:23:36.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:36.711+0000 7ff211792700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7ff20c15b2c0 0x7ff20c161340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:36.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:36.711+0000 7ff211792700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff20c06e570 0x7ff20c15e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:36.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:36.711+0000 7ff211792700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff20c06a7e0 0x7ff20c06ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:36.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:36.711+0000 7ff211792700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff20c06b1b0 con 0x7ff20c06a7e0 2023-10-16T12:23:36.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:36.711+0000 7ff211792700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff20c161880 con 0x7ff20c15b2c0 2023-10-16T12:23:36.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:36.711+0000 7ff211792700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff20c15b6a0 con 0x7ff20c06e570 2023-10-16T12:23:36.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:36.711+0000 7ff20b7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff20c06a7e0 0x7ff20c06ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:36.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:36.711+0000 7ff20b7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff20c06a7e0 0x7ff20c06ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36870/0 (socket says 172.21.15.16:36870) 2023-10-16T12:23:36.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:36.711+0000 7ff20affd700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7ff20c15b2c0 0x7ff20c161340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:36.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:36.711+0000 7ff20a7fc700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff20c06e570 0x7ff20c15e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:36.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:36.711+0000 7ff20b7fe700 1 -- 172.21.15.16:0/2780765295 learned_addr learned my addr 172.21.15.16:0/2780765295 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:36.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:36.712+0000 7ff20b7fe700 1 -- 172.21.15.16:0/2780765295 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff20c06e570 msgr2=0x7ff20c15e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:36.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:36.712+0000 7ff20b7fe700 1 --2- 172.21.15.16:0/2780765295 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff20c06e570 0x7ff20c15e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:36.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:36.712+0000 7ff20b7fe700 1 -- 172.21.15.16:0/2780765295 >> v2:172.21.15.49:3300/0 conn(0x7ff20c15b2c0 msgr2=0x7ff20c161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:36.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:36.712+0000 7ff20b7fe700 1 --2- 172.21.15.16:0/2780765295 >> v2:172.21.15.49:3300/0 conn(0x7ff20c15b2c0 0x7ff20c161340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:36.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:36.712+0000 7ff20b7fe700 1 -- 172.21.15.16:0/2780765295 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff20c161b20 con 0x7ff20c06a7e0 2023-10-16T12:23:36.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:36.712+0000 7ff20b7fe700 1 --2- 172.21.15.16:0/2780765295 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff20c06a7e0 0x7ff20c06ac70 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7ff20c06b4a0 tx=0x7ff20400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=923f84dd475c2216 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:36.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:36.712+0000 7ff209ffb700 1 -- 172.21.15.16:0/2780765295 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff20400e070 con 0x7ff20c06a7e0 2023-10-16T12:23:36.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:36.712+0000 7ff209ffb700 1 -- 172.21.15.16:0/2780765295 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff20400b910 con 0x7ff20c06a7e0 2023-10-16T12:23:36.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:36.712+0000 7ff209ffb700 1 -- 172.21.15.16:0/2780765295 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff20400bcc0 con 0x7ff20c06a7e0 2023-10-16T12:23:36.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:36.713+0000 7ff211792700 1 -- 172.21.15.16:0/2780765295 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff20c06a7e0 msgr2=0x7ff20c06ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:36.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:36.713+0000 7ff211792700 1 --2- 172.21.15.16:0/2780765295 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff20c06a7e0 0x7ff20c06ac70 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7ff20c06b4a0 tx=0x7ff20400b330 comp rx=0 tx=0).stop 2023-10-16T12:23:36.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:36.713+0000 7ff211792700 1 -- 172.21.15.16:0/2780765295 shutdown_connections 2023-10-16T12:23:36.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:36.713+0000 7ff211792700 1 --2- 172.21.15.16:0/2780765295 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff20c06a7e0 0x7ff20c06ac70 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:36.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:36.713+0000 7ff211792700 1 --2- 172.21.15.16:0/2780765295 >> v2:172.21.15.49:3300/0 conn(0x7ff20c15b2c0 0x7ff20c161340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:36.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:36.713+0000 7ff211792700 1 --2- 172.21.15.16:0/2780765295 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff20c06e570 0x7ff20c15e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:36.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:36.713+0000 7ff211792700 1 -- 172.21.15.16:0/2780765295 >> 172.21.15.16:0/2780765295 conn(0x7ff20c0b1f40 msgr2=0x7ff20c0b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:36.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:36.713+0000 7ff211792700 1 -- 172.21.15.16:0/2780765295 shutdown_connections 2023-10-16T12:23:36.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:36.713+0000 7ff211792700 1 -- 172.21.15.16:0/2780765295 wait complete. 2023-10-16T12:23:36.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:36.714+0000 7ff211792700 1 Processor -- start 2023-10-16T12:23:36.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:36.714+0000 7ff211792700 1 -- start start 2023-10-16T12:23:36.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:36.715+0000 7ff211792700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff20c06a7e0 0x7ff20c160d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:36.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:36.715+0000 7ff211792700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff20c06e570 0x7ff20c1f67a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:36.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:36.715+0000 7ff211792700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff20c15b2c0 0x7ff20c14f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:36.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:36.715+0000 7ff211792700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff20c0b55c0 con 0x7ff20c15b2c0 2023-10-16T12:23:36.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:36.715+0000 7ff211792700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff20c0b46f0 con 0x7ff20c06a7e0 2023-10-16T12:23:36.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:36.715+0000 7ff211792700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff20c0b59e0 con 0x7ff20c06e570 2023-10-16T12:23:36.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:36.715+0000 7ff20b7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff20c15b2c0 0x7ff20c14f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:36.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:36.715+0000 7ff20b7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff20c15b2c0 0x7ff20c14f3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36876/0 (socket says 172.21.15.16:36876) 2023-10-16T12:23:36.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:36.715+0000 7ff20a7fc700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff20c06e570 0x7ff20c1f67a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:36.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:36.715+0000 7ff20b7fe700 1 -- 172.21.15.16:0/833233041 learned_addr learned my addr 172.21.15.16:0/833233041 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:36.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:36.715+0000 7ff20affd700 1 --2- 172.21.15.16:0/833233041 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff20c06a7e0 0x7ff20c160d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:36.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:36.715+0000 7ff20b7fe700 1 -- 172.21.15.16:0/833233041 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff20c06e570 msgr2=0x7ff20c1f67a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:36.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:36.715+0000 7ff20b7fe700 1 --2- 172.21.15.16:0/833233041 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff20c06e570 0x7ff20c1f67a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:36.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:36.715+0000 7ff20b7fe700 1 -- 172.21.15.16:0/833233041 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff20c06a7e0 msgr2=0x7ff20c160d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:36.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:36.715+0000 7ff20b7fe700 1 --2- 172.21.15.16:0/833233041 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff20c06a7e0 0x7ff20c160d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:36.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:36.715+0000 7ff20b7fe700 1 -- 172.21.15.16:0/833233041 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff204009b30 con 0x7ff20c15b2c0 2023-10-16T12:23:36.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:36.716+0000 7ff20b7fe700 1 --2- 172.21.15.16:0/833233041 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff20c15b2c0 0x7ff20c14f3c0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7ff20400b680 tx=0x7ff204004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:36.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:36.716+0000 7ff1f37fe700 1 -- 172.21.15.16:0/833233041 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2040099b0 con 0x7ff20c15b2c0 2023-10-16T12:23:36.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:36.716+0000 7ff1f37fe700 1 -- 172.21.15.16:0/833233041 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff20400ce80 con 0x7ff20c15b2c0 2023-10-16T12:23:36.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:36.716+0000 7ff1f37fe700 1 -- 172.21.15.16:0/833233041 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff204004490 con 0x7ff20c15b2c0 2023-10-16T12:23:36.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:36.716+0000 7ff211792700 1 -- 172.21.15.16:0/833233041 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff20c0b4940 con 0x7ff20c15b2c0 2023-10-16T12:23:36.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:36.716+0000 7ff211792700 1 -- 172.21.15.16:0/833233041 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff20c0b4f00 con 0x7ff20c15b2c0 2023-10-16T12:23:36.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:36.717+0000 7ff211792700 1 -- 172.21.15.16:0/833233041 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff20c0b5700 con 0x7ff20c15b2c0 2023-10-16T12:23:36.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:36.717+0000 7ff1f37fe700 1 -- 172.21.15.16:0/833233041 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7ff2040045f0 con 0x7ff20c15b2c0 2023-10-16T12:23:36.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:36.717+0000 7ff1f37fe700 1 --2- 172.21.15.16:0/833233041 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7ff1f806b130 0x7ff1f806d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:36.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:36.718+0000 7ff1f37fe700 1 -- 172.21.15.16:0/833233041 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 7060+0+0 (secure 0 0 0) 0x7ff20408af40 con 0x7ff20c15b2c0 2023-10-16T12:23:36.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:36.718+0000 7ff20affd700 1 --2- 172.21.15.16:0/833233041 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7ff1f806b130 0x7ff1f806d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:36.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:36.718+0000 7ff20affd700 1 --2- 172.21.15.16:0/833233041 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7ff1f806b130 0x7ff1f806d5f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff200009b60 tx=0x7ff200009210 comp rx=0 tx=0).ready entity=mgr.5499 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:36.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:36.722+0000 7ff1f37fe700 1 -- 172.21.15.16:0/833233041 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff20c0b5700 con 0x7ff20c15b2c0 2023-10-16T12:23:36.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:36.986+0000 7ff211792700 1 -- 172.21.15.16:0/833233041 --> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir6", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7ff20c1620e0 con 0x7ff1f806b130 2023-10-16T12:23:37.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:37.055+0000 7ff1f37fe700 1 -- 172.21.15.16:0/833233041 <== mgr.5499 v2:172.21.15.16:6832/1707126831 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7ff2000098d0 con 0x7ff1f806b130 2023-10-16T12:23:37.058 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule removed for path /snap_test_dir1/dir6 2023-10-16T12:23:37.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:37.057+0000 7ff211792700 1 -- 172.21.15.16:0/833233041 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7ff1f806b130 msgr2=0x7ff1f806d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:37.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:37.057+0000 7ff211792700 1 --2- 172.21.15.16:0/833233041 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7ff1f806b130 0x7ff1f806d5f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff200009b60 tx=0x7ff200009210 comp rx=0 tx=0).stop 2023-10-16T12:23:37.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:37.057+0000 7ff211792700 1 -- 172.21.15.16:0/833233041 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff20c15b2c0 msgr2=0x7ff20c14f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:37.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:37.057+0000 7ff211792700 1 --2- 172.21.15.16:0/833233041 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff20c15b2c0 0x7ff20c14f3c0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7ff20400b680 tx=0x7ff204004ad0 comp rx=0 tx=0).stop 2023-10-16T12:23:37.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:37.058+0000 7ff211792700 1 -- 172.21.15.16:0/833233041 shutdown_connections 2023-10-16T12:23:37.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:37.058+0000 7ff211792700 1 --2- 172.21.15.16:0/833233041 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7ff1f806b130 0x7ff1f806d5f0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:37.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:37.058+0000 7ff211792700 1 --2- 172.21.15.16:0/833233041 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff20c15b2c0 0x7ff20c14f3c0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:37.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:37.058+0000 7ff211792700 1 --2- 172.21.15.16:0/833233041 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff20c06a7e0 0x7ff20c160d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:37.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:37.058+0000 7ff211792700 1 --2- 172.21.15.16:0/833233041 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff20c06e570 0x7ff20c1f67a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:37.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:37.058+0000 7ff211792700 1 -- 172.21.15.16:0/833233041 >> 172.21.15.16:0/833233041 conn(0x7ff20c0b1f40 msgr2=0x7ff20c0cce00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:37.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:37.058+0000 7ff211792700 1 -- 172.21.15.16:0/833233041 shutdown_connections 2023-10-16T12:23:37.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:37.058+0000 7ff211792700 1 -- 172.21.15.16:0/833233041 wait complete. 2023-10-16T12:23:37.071 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir6 2023-10-16T12:23:37.072 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:37.072 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir6/.snap) 2023-10-16T12:23:37.091 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_13_00_UTC 2023-10-16T12:23:37.091 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_14_00_UTC 2023-10-16T12:23:37.092 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_15_00_UTC 2023-10-16T12:23:37.092 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_16_00_UTC 2023-10-16T12:23:37.092 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_17_00_UTC 2023-10-16T12:23:37.092 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_18_00_UTC 2023-10-16T12:23:37.092 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_19_00_UTC 2023-10-16T12:23:37.093 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_20_00_UTC 2023-10-16T12:23:37.093 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_21_00_UTC 2023-10-16T12:23:37.093 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_22_00_UTC 2023-10-16T12:23:37.093 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_23_00_UTC 2023-10-16T12:23:37.093 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-16-12_13_00_UTC 2023-10-16T12:23:37.094 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:37.094 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-16-12_13_00_UTC) 2023-10-16T12:23:37.158 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-16-12_14_00_UTC 2023-10-16T12:23:37.158 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:37.159 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-16-12_14_00_UTC) 2023-10-16T12:23:37.223 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-16-12_15_00_UTC 2023-10-16T12:23:37.223 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:37.223 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-16-12_15_00_UTC) 2023-10-16T12:23:37.286 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-16-12_16_00_UTC 2023-10-16T12:23:37.287 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:37.287 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-16-12_16_00_UTC) 2023-10-16T12:23:37.351 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-16-12_17_00_UTC 2023-10-16T12:23:37.351 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:37.352 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-16-12_17_00_UTC) 2023-10-16T12:23:37.420 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-16-12_18_00_UTC 2023-10-16T12:23:37.420 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:37.421 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-16-12_18_00_UTC) 2023-10-16T12:23:37.484 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-16-12_19_00_UTC 2023-10-16T12:23:37.484 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:37.484 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-16-12_19_00_UTC) 2023-10-16T12:23:37.547 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-16-12_20_00_UTC 2023-10-16T12:23:37.547 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:37.547 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-16-12_20_00_UTC) 2023-10-16T12:23:37.610 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-16-12_21_00_UTC 2023-10-16T12:23:37.611 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:37.611 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-16-12_21_00_UTC) 2023-10-16T12:23:37.674 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-16-12_22_00_UTC 2023-10-16T12:23:37.674 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:37.675 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-16-12_22_00_UTC) 2023-10-16T12:23:37.736 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-16-12_23_00_UTC 2023-10-16T12:23:37.737 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:37.737 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-16-12_23_00_UTC) 2023-10-16T12:23:37.800 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:37.801 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6) 2023-10-16T12:23:37.860 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir7 1m 2023-10-16T12:23:38.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.030+0000 7f063a5bc700 1 Processor -- start 2023-10-16T12:23:38.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.030+0000 7f063a5bc700 1 -- start start 2023-10-16T12:23:38.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.031+0000 7f063a5bc700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f06340ce5d0 0x7f06340cc190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:38.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.031+0000 7f063a5bc700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f063415f060 0x7f06340cc6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:38.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.031+0000 7f063a5bc700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06340b2440 con 0x7f063415ec80 2023-10-16T12:23:38.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.031+0000 7f063a5bc700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06340b2c50 con 0x7f063415f060 2023-10-16T12:23:38.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.031+0000 7f063a5bc700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06340d00e0 con 0x7f06340ce5d0 2023-10-16T12:23:38.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.031+0000 7f0633fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f063415ec80 0x7f06340cdec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54386/0 (socket says 172.21.15.16:54386) 2023-10-16T12:23:38.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.031+0000 7f0633fff700 1 -- 172.21.15.16:0/3887539596 learned_addr learned my addr 172.21.15.16:0/3887539596 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:23:38.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.031+0000 7f06337fe700 1 --2- 172.21.15.16:0/3887539596 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f06340ce5d0 0x7f06340cc190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:38.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.031+0000 7f0638b59700 1 --2- 172.21.15.16:0/3887539596 >> v2:172.21.15.49:3300/0 conn(0x7f063415f060 0x7f06340cc6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:38.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.032+0000 7f0632ffd700 1 -- 172.21.15.16:0/3887539596 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 115528698 0 0) 0x7f06340b2440 con 0x7f063415ec80 2023-10-16T12:23:38.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.032+0000 7f0632ffd700 1 -- 172.21.15.16:0/3887539596 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f061c0035b0 con 0x7f063415ec80 2023-10-16T12:23:38.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.032+0000 7f0632ffd700 1 -- 172.21.15.16:0/3887539596 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 382212061 0 0) 0x7f061c0035b0 con 0x7f063415ec80 2023-10-16T12:23:38.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.032+0000 7f0638b59700 1 -- 172.21.15.16:0/3887539596 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f06340ce5d0 msgr2=0x7f06340cc190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:38.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.032+0000 7f0638b59700 1 --2- 172.21.15.16:0/3887539596 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f06340ce5d0 0x7f06340cc190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:38.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.032+0000 7f0638b59700 1 -- 172.21.15.16:0/3887539596 >> v1:172.21.15.16:6789/0 conn(0x7f063415ec80 legacy=0x7f06340cdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:38.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.032+0000 7f0638b59700 1 -- 172.21.15.16:0/3887539596 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06340cef60 con 0x7f063415f060 2023-10-16T12:23:38.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.032+0000 7f06337fe700 1 --2- 172.21.15.16:0/3887539596 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f06340ce5d0 0x7f06340cc190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:23:38.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.032+0000 7f0638b59700 1 --2- 172.21.15.16:0/3887539596 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f063415f060 0x7f06340cc6d0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f062c009d90 tx=0x7f062c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fcc58936a654a451 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:38.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.032+0000 7f0632ffd700 1 -- 172.21.15.16:0/3887539596 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f062c00e070 con 0x7f063415f060 2023-10-16T12:23:38.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.032+0000 7f0632ffd700 1 -- 172.21.15.16:0/3887539596 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f062c003a20 con 0x7f063415f060 2023-10-16T12:23:38.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.032+0000 7f0632ffd700 1 -- 172.21.15.16:0/3887539596 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0628001670 con 0x7f063415f060 2023-10-16T12:23:38.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.033+0000 7f063a5bc700 1 -- 172.21.15.16:0/3887539596 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f063415f060 msgr2=0x7f06340cc6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:38.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.033+0000 7f063a5bc700 1 --2- 172.21.15.16:0/3887539596 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f063415f060 0x7f06340cc6d0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f062c009d90 tx=0x7f062c00b330 comp rx=0 tx=0).stop 2023-10-16T12:23:38.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.033+0000 7f063a5bc700 1 -- 172.21.15.16:0/3887539596 shutdown_connections 2023-10-16T12:23:38.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.033+0000 7f063a5bc700 1 --2- 172.21.15.16:0/3887539596 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f063415f060 0x7f06340cc6d0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:38.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.033+0000 7f063a5bc700 1 --2- 172.21.15.16:0/3887539596 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f06340ce5d0 0x7f06340cc190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:38.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.033+0000 7f063a5bc700 1 -- 172.21.15.16:0/3887539596 >> 172.21.15.16:0/3887539596 conn(0x7f06340d15f0 msgr2=0x7f06340d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:38.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.033+0000 7f063a5bc700 1 -- 172.21.15.16:0/3887539596 shutdown_connections 2023-10-16T12:23:38.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.033+0000 7f063a5bc700 1 -- 172.21.15.16:0/3887539596 wait complete. 2023-10-16T12:23:38.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.034+0000 7f063a5bc700 1 Processor -- start 2023-10-16T12:23:38.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.034+0000 7f063a5bc700 1 -- start start 2023-10-16T12:23:38.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.035+0000 7f063a5bc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06340ce5d0 0x7f063414e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:38.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.035+0000 7f063a5bc700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f063415ec80 0x7f063406ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:38.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.035+0000 7f063a5bc700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f063415f060 0x7f063406f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:38.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.035+0000 7f063a5bc700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06340c8070 con 0x7f06340ce5d0 2023-10-16T12:23:38.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.035+0000 7f063a5bc700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f063414e920 con 0x7f063415f060 2023-10-16T12:23:38.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.035+0000 7f063a5bc700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f063406f770 con 0x7f063415ec80 2023-10-16T12:23:38.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.035+0000 7f0633fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06340ce5d0 0x7f063414e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:38.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.035+0000 7f0633fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06340ce5d0 0x7f063414e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36886/0 (socket says 172.21.15.16:36886) 2023-10-16T12:23:38.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.035+0000 7f0633fff700 1 -- 172.21.15.16:0/1438536515 learned_addr learned my addr 172.21.15.16:0/1438536515 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:38.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.035+0000 7f06337fe700 1 --2- 172.21.15.16:0/1438536515 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f063415ec80 0x7f063406ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:38.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.035+0000 7f0638b59700 1 --2- 172.21.15.16:0/1438536515 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f063415f060 0x7f063406f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:38.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.035+0000 7f0633fff700 1 -- 172.21.15.16:0/1438536515 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f063415ec80 msgr2=0x7f063406ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:38.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.035+0000 7f0633fff700 1 --2- 172.21.15.16:0/1438536515 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f063415ec80 0x7f063406ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:38.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.035+0000 7f0633fff700 1 -- 172.21.15.16:0/1438536515 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f063415f060 msgr2=0x7f063406f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:38.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.035+0000 7f0633fff700 1 --2- 172.21.15.16:0/1438536515 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f063415f060 0x7f063406f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:38.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.035+0000 7f0633fff700 1 -- 172.21.15.16:0/1438536515 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06240094a0 con 0x7f06340ce5d0 2023-10-16T12:23:38.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.036+0000 7f0633fff700 1 --2- 172.21.15.16:0/1438536515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06340ce5d0 0x7f063414e3e0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f062800c780 tx=0x7f062800cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:38.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.036+0000 7f0630ff9700 1 -- 172.21.15.16:0/1438536515 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0628004c80 con 0x7f06340ce5d0 2023-10-16T12:23:38.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.036+0000 7f0630ff9700 1 -- 172.21.15.16:0/1438536515 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0628004de0 con 0x7f06340ce5d0 2023-10-16T12:23:38.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.036+0000 7f063a5bc700 1 -- 172.21.15.16:0/1438536515 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f062c009a70 con 0x7f06340ce5d0 2023-10-16T12:23:38.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.036+0000 7f0630ff9700 1 -- 172.21.15.16:0/1438536515 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06280072f0 con 0x7f06340ce5d0 2023-10-16T12:23:38.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.036+0000 7f063a5bc700 1 -- 172.21.15.16:0/1438536515 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f063406fd30 con 0x7f06340ce5d0 2023-10-16T12:23:38.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.037+0000 7f063a5bc700 1 -- 172.21.15.16:0/1438536515 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f063404ebb0 con 0x7f06340ce5d0 2023-10-16T12:23:38.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.037+0000 7f0630ff9700 1 -- 172.21.15.16:0/1438536515 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f0628007450 con 0x7f06340ce5d0 2023-10-16T12:23:38.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.038+0000 7f0630ff9700 1 --2- 172.21.15.16:0/1438536515 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f062006b100 0x7f062006d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:38.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.038+0000 7f0630ff9700 1 -- 172.21.15.16:0/1438536515 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6996+0+0 (secure 0 0 0) 0x7f06280891e0 con 0x7f06340ce5d0 2023-10-16T12:23:38.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.038+0000 7f06337fe700 1 --2- 172.21.15.16:0/1438536515 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f062006b100 0x7f062006d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:38.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.038+0000 7f06337fe700 1 --2- 172.21.15.16:0/1438536515 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f062006b100 0x7f062006d5c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f06340cf2e0 tx=0x7f0624008040 comp rx=0 tx=0).ready entity=mgr.5499 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:38.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.042+0000 7f0630ff9700 1 -- 172.21.15.16:0/1438536515 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0628053a70 con 0x7f06340ce5d0 2023-10-16T12:23:38.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.301+0000 7f063a5bc700 1 -- 172.21.15.16:0/1438536515 --> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir7", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f06340b2970 con 0x7f062006b100 2023-10-16T12:23:38.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.401+0000 7f0630ff9700 1 -- 172.21.15.16:0/1438536515 <== mgr.5499 v2:172.21.15.16:6832/1707126831 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f0624009970 con 0x7f062006b100 2023-10-16T12:23:38.403 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule removed for path /snap_test_dir1/dir7 2023-10-16T12:23:38.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.404+0000 7f063a5bc700 1 -- 172.21.15.16:0/1438536515 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f062006b100 msgr2=0x7f062006d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:38.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.404+0000 7f063a5bc700 1 --2- 172.21.15.16:0/1438536515 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f062006b100 0x7f062006d5c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f06340cf2e0 tx=0x7f0624008040 comp rx=0 tx=0).stop 2023-10-16T12:23:38.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.404+0000 7f063a5bc700 1 -- 172.21.15.16:0/1438536515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06340ce5d0 msgr2=0x7f063414e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:38.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.404+0000 7f063a5bc700 1 --2- 172.21.15.16:0/1438536515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06340ce5d0 0x7f063414e3e0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f062800c780 tx=0x7f062800cb40 comp rx=0 tx=0).stop 2023-10-16T12:23:38.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.404+0000 7f063a5bc700 1 -- 172.21.15.16:0/1438536515 shutdown_connections 2023-10-16T12:23:38.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.404+0000 7f063a5bc700 1 --2- 172.21.15.16:0/1438536515 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f062006b100 0x7f062006d5c0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:38.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.404+0000 7f063a5bc700 1 --2- 172.21.15.16:0/1438536515 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f063415f060 0x7f063406f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:38.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.404+0000 7f063a5bc700 1 --2- 172.21.15.16:0/1438536515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06340ce5d0 0x7f063414e3e0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:38.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.404+0000 7f063a5bc700 1 --2- 172.21.15.16:0/1438536515 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f063415ec80 0x7f063406ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:38.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.404+0000 7f063a5bc700 1 -- 172.21.15.16:0/1438536515 >> 172.21.15.16:0/1438536515 conn(0x7f06340d15f0 msgr2=0x7f06340cc190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:38.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.404+0000 7f063a5bc700 1 -- 172.21.15.16:0/1438536515 shutdown_connections 2023-10-16T12:23:38.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:38.404+0000 7f063a5bc700 1 -- 172.21.15.16:0/1438536515 wait complete. 2023-10-16T12:23:38.423 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir7 2023-10-16T12:23:38.424 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:38.424 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir7/.snap) 2023-10-16T12:23:38.469 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_13_00_UTC 2023-10-16T12:23:38.469 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_14_00_UTC 2023-10-16T12:23:38.469 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_15_00_UTC 2023-10-16T12:23:38.470 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_16_00_UTC 2023-10-16T12:23:38.470 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_17_00_UTC 2023-10-16T12:23:38.470 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_18_00_UTC 2023-10-16T12:23:38.470 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_19_00_UTC 2023-10-16T12:23:38.470 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_20_00_UTC 2023-10-16T12:23:38.471 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_21_00_UTC 2023-10-16T12:23:38.471 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_22_00_UTC 2023-10-16T12:23:38.471 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_23_00_UTC 2023-10-16T12:23:38.472 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-16-12_13_00_UTC 2023-10-16T12:23:38.472 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:38.473 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-16-12_13_00_UTC) 2023-10-16T12:23:38.538 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-16-12_14_00_UTC 2023-10-16T12:23:38.539 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:38.539 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-16-12_14_00_UTC) 2023-10-16T12:23:38.602 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-16-12_15_00_UTC 2023-10-16T12:23:38.602 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:38.603 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-16-12_15_00_UTC) 2023-10-16T12:23:38.664 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-16-12_16_00_UTC 2023-10-16T12:23:38.665 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:38.665 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-16-12_16_00_UTC) 2023-10-16T12:23:38.727 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-16-12_17_00_UTC 2023-10-16T12:23:38.727 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:38.728 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-16-12_17_00_UTC) 2023-10-16T12:23:38.790 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-16-12_18_00_UTC 2023-10-16T12:23:38.790 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:38.791 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-16-12_18_00_UTC) 2023-10-16T12:23:38.854 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-16-12_19_00_UTC 2023-10-16T12:23:38.855 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:38.855 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-16-12_19_00_UTC) 2023-10-16T12:23:38.918 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-16-12_20_00_UTC 2023-10-16T12:23:38.918 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:38.918 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-16-12_20_00_UTC) 2023-10-16T12:23:38.981 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-16-12_21_00_UTC 2023-10-16T12:23:38.981 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:38.982 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-16-12_21_00_UTC) 2023-10-16T12:23:39.045 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-16-12_22_00_UTC 2023-10-16T12:23:39.045 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:39.046 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-16-12_22_00_UTC) 2023-10-16T12:23:39.108 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-16-12_23_00_UTC 2023-10-16T12:23:39.108 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:39.109 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-16-12_23_00_UTC) 2023-10-16T12:23:39.172 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:39.173 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7) 2023-10-16T12:23:39.233 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir8 1m 2023-10-16T12:23:39.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.403+0000 7f3d3b3f8700 1 Processor -- start 2023-10-16T12:23:39.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.404+0000 7f3d3b3f8700 1 -- start start 2023-10-16T12:23:39.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.404+0000 7f3d3b3f8700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3d340cb880 0x7f3d340cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:39.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.404+0000 7f3d3b3f8700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d340cef70 con 0x7f3d3406da60 2023-10-16T12:23:39.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.404+0000 7f3d3b3f8700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d340ce360 con 0x7f3d3415b070 2023-10-16T12:23:39.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.404+0000 7f3d3b3f8700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d340ce640 con 0x7f3d340cb880 2023-10-16T12:23:39.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.404+0000 7f3d39995700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f3d3406da60 0x7f3d340cf400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54404/0 (socket says 172.21.15.16:54404) 2023-10-16T12:23:39.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.404+0000 7f3d39995700 1 -- 172.21.15.16:0/311394036 learned_addr learned my addr 172.21.15.16:0/311394036 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:23:39.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.405+0000 7f3d38993700 1 --2- 172.21.15.16:0/311394036 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3d340cb880 0x7f3d340cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:39.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.405+0000 7f3d2bfff700 1 -- 172.21.15.16:0/311394036 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3186521375 0 0) 0x7f3d340cef70 con 0x7f3d3406da60 2023-10-16T12:23:39.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.405+0000 7f3d2bfff700 1 -- 172.21.15.16:0/311394036 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d200035b0 con 0x7f3d3406da60 2023-10-16T12:23:39.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.405+0000 7f3d2bfff700 1 -- 172.21.15.16:0/311394036 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2532170694 0 0) 0x7f3d200035b0 con 0x7f3d3406da60 2023-10-16T12:23:39.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.405+0000 7f3d2bfff700 1 -- 172.21.15.16:0/311394036 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3d340cb880 msgr2=0x7f3d340cbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:39.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.405+0000 7f3d2bfff700 1 --2- 172.21.15.16:0/311394036 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3d340cb880 0x7f3d340cbd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:39.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.405+0000 7f3d2bfff700 1 -- 172.21.15.16:0/311394036 >> v1:172.21.15.49:6789/0 conn(0x7f3d3415b070 legacy=0x7f3d3415b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:39.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.405+0000 7f3d2bfff700 1 -- 172.21.15.16:0/311394036 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d340ce920 con 0x7f3d3406da60 2023-10-16T12:23:39.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.405+0000 7f3d2bfff700 1 -- 172.21.15.16:0/311394036 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f3d2c0019f0 con 0x7f3d3406da60 2023-10-16T12:23:39.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.405+0000 7f3d2bfff700 1 -- 172.21.15.16:0/311394036 >> v1:172.21.15.16:6789/0 conn(0x7f3d3406da60 legacy=0x7f3d340cf400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:39.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.405+0000 7f3d2bfff700 1 --2- 172.21.15.16:0/311394036 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3d20003a30 0x7f3d20003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:39.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.405+0000 7f3d2bfff700 1 --2- 172.21.15.16:0/311394036 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3d340cb880 0x7f3d20004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:39.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.405+0000 7f3d2bfff700 1 --2- 172.21.15.16:0/311394036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d20004d90 0x7f3d20005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:39.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.405+0000 7f3d2bfff700 1 -- 172.21.15.16:0/311394036 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d340ce640 con 0x7f3d20004d90 2023-10-16T12:23:39.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.405+0000 7f3d2bfff700 1 -- 172.21.15.16:0/311394036 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d2000bb40 con 0x7f3d20003a30 2023-10-16T12:23:39.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.405+0000 7f3d2bfff700 1 -- 172.21.15.16:0/311394036 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d2000bd00 con 0x7f3d340cb880 2023-10-16T12:23:39.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.405+0000 7f3d39995700 1 --2- 172.21.15.16:0/311394036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d20004d90 0x7f3d20005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:39.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.405+0000 7f3d38993700 1 --2- 172.21.15.16:0/311394036 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3d340cb880 0x7f3d20004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:39.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.405+0000 7f3d39194700 1 --2- 172.21.15.16:0/311394036 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3d20003a30 0x7f3d20003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:39.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.406+0000 7f3d39995700 1 -- 172.21.15.16:0/311394036 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3d340cb880 msgr2=0x7f3d20004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:39.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.406+0000 7f3d39995700 1 --2- 172.21.15.16:0/311394036 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3d340cb880 0x7f3d20004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:39.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.406+0000 7f3d39995700 1 -- 172.21.15.16:0/311394036 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3d20003a30 msgr2=0x7f3d20003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:39.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.406+0000 7f3d39995700 1 --2- 172.21.15.16:0/311394036 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3d20003a30 0x7f3d20003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:39.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.406+0000 7f3d39995700 1 -- 172.21.15.16:0/311394036 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d200035b0 con 0x7f3d20004d90 2023-10-16T12:23:39.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.406+0000 7f3d39995700 1 --2- 172.21.15.16:0/311394036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d20004d90 0x7f3d20005210 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c00a830 tx=0x7f3d20005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a7e3adee09dd059 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:39.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.406+0000 7f3d2bfff700 1 -- 172.21.15.16:0/311394036 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d2c00a2f0 con 0x7f3d20004d90 2023-10-16T12:23:39.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.406+0000 7f3d2bfff700 1 -- 172.21.15.16:0/311394036 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d2c012620 con 0x7f3d20004d90 2023-10-16T12:23:39.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.406+0000 7f3d2bfff700 1 -- 172.21.15.16:0/311394036 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d2c0129d0 con 0x7f3d20004d90 2023-10-16T12:23:39.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.406+0000 7f3d3b3f8700 1 -- 172.21.15.16:0/311394036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d20004d90 msgr2=0x7f3d20005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:39.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.406+0000 7f3d3b3f8700 1 --2- 172.21.15.16:0/311394036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d20004d90 0x7f3d20005210 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c00a830 tx=0x7f3d20005880 comp rx=0 tx=0).stop 2023-10-16T12:23:39.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.406+0000 7f3d39995700 1 -- 172.21.15.16:0/311394036 reap_dead start 2023-10-16T12:23:39.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.406+0000 7f3d3b3f8700 1 -- 172.21.15.16:0/311394036 shutdown_connections 2023-10-16T12:23:39.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.406+0000 7f3d3b3f8700 1 -- 172.21.15.16:0/311394036 >> 172.21.15.16:0/311394036 conn(0x7f3d340b1f40 msgr2=0x7f3d340b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:39.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.406+0000 7f3d3b3f8700 1 -- 172.21.15.16:0/311394036 shutdown_connections 2023-10-16T12:23:39.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.407+0000 7f3d3b3f8700 1 -- 172.21.15.16:0/311394036 wait complete. 2023-10-16T12:23:39.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.408+0000 7f3d3b3f8700 1 Processor -- start 2023-10-16T12:23:39.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.408+0000 7f3d3b3f8700 1 -- start start 2023-10-16T12:23:39.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.408+0000 7f3d3b3f8700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3d20004d90 0x7f3d3414fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:39.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.408+0000 7f3d3b3f8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d340b3030 0x7f3d340b45d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:39.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.408+0000 7f3d3b3f8700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3d3415d1c0 0x7f3d34158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:39.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.408+0000 7f3d3b3f8700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d340b41b0 con 0x7f3d340b3030 2023-10-16T12:23:39.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.408+0000 7f3d3b3f8700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d341501e0 con 0x7f3d3415d1c0 2023-10-16T12:23:39.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.408+0000 7f3d3b3f8700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d340b4b10 con 0x7f3d20004d90 2023-10-16T12:23:39.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.409+0000 7f3d38993700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d340b3030 0x7f3d340b45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:39.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.409+0000 7f3d38993700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d340b3030 0x7f3d340b45d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36908/0 (socket says 172.21.15.16:36908) 2023-10-16T12:23:39.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.409+0000 7f3d38993700 1 -- 172.21.15.16:0/1362741180 learned_addr learned my addr 172.21.15.16:0/1362741180 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:39.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.409+0000 7f3d39194700 1 --2- 172.21.15.16:0/1362741180 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3d20004d90 0x7f3d3414fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:39.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.409+0000 7f3d39995700 1 --2- 172.21.15.16:0/1362741180 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3d3415d1c0 0x7f3d34158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:39.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.409+0000 7f3d38993700 1 -- 172.21.15.16:0/1362741180 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3d20004d90 msgr2=0x7f3d3414fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:39.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.409+0000 7f3d38993700 1 --2- 172.21.15.16:0/1362741180 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3d20004d90 0x7f3d3414fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:39.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.409+0000 7f3d38993700 1 -- 172.21.15.16:0/1362741180 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3d3415d1c0 msgr2=0x7f3d34158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:39.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.409+0000 7f3d38993700 1 --2- 172.21.15.16:0/1362741180 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3d3415d1c0 0x7f3d34158ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:39.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.409+0000 7f3d38993700 1 -- 172.21.15.16:0/1362741180 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d340b3530 con 0x7f3d340b3030 2023-10-16T12:23:39.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.409+0000 7f3d39194700 1 --2- 172.21.15.16:0/1362741180 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3d20004d90 0x7f3d3414fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:23:39.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.409+0000 7f3d39995700 1 --2- 172.21.15.16:0/1362741180 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3d3415d1c0 0x7f3d34158ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:23:39.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.409+0000 7f3d38993700 1 --2- 172.21.15.16:0/1362741180 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d340b3030 0x7f3d340b45d0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f3d30011bb0 tx=0x7f3d30009f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:39.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.409+0000 7f3d29ffb700 1 -- 172.21.15.16:0/1362741180 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d30016070 con 0x7f3d340b3030 2023-10-16T12:23:39.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.409+0000 7f3d29ffb700 1 -- 172.21.15.16:0/1362741180 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d3000eaa0 con 0x7f3d340b3030 2023-10-16T12:23:39.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.409+0000 7f3d29ffb700 1 -- 172.21.15.16:0/1362741180 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d3000ec10 con 0x7f3d340b3030 2023-10-16T12:23:39.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.409+0000 7f3d3b3f8700 1 -- 172.21.15.16:0/1362741180 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d340b3810 con 0x7f3d340b3030 2023-10-16T12:23:39.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.409+0000 7f3d3b3f8700 1 -- 172.21.15.16:0/1362741180 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d340b3dd0 con 0x7f3d340b3030 2023-10-16T12:23:39.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.411+0000 7f3d3b3f8700 1 -- 172.21.15.16:0/1362741180 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d3404ebb0 con 0x7f3d340b3030 2023-10-16T12:23:39.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.411+0000 7f3d29ffb700 1 -- 172.21.15.16:0/1362741180 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f3d3001a410 con 0x7f3d340b3030 2023-10-16T12:23:39.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.411+0000 7f3d29ffb700 1 --2- 172.21.15.16:0/1362741180 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f3d1806b100 0x7f3d1806d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:39.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.411+0000 7f3d29ffb700 1 -- 172.21.15.16:0/1362741180 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6884+0+0 (secure 0 0 0) 0x7f3d3008b6f0 con 0x7f3d340b3030 2023-10-16T12:23:39.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.412+0000 7f3d39194700 1 --2- 172.21.15.16:0/1362741180 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f3d1806b100 0x7f3d1806d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:39.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.412+0000 7f3d39194700 1 --2- 172.21.15.16:0/1362741180 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f3d1806b100 0x7f3d1806d5c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3d340b6120 tx=0x7f3d24009040 comp rx=0 tx=0).ready entity=mgr.5499 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:39.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.416+0000 7f3d29ffb700 1 -- 172.21.15.16:0/1362741180 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3d30056280 con 0x7f3d340b3030 2023-10-16T12:23:39.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.647+0000 7f3d3b3f8700 1 -- 172.21.15.16:0/1362741180 --> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir8", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f3d340728d0 con 0x7f3d1806b100 2023-10-16T12:23:39.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.734+0000 7f3d29ffb700 1 -- 172.21.15.16:0/1362741180 <== mgr.5499 v2:172.21.15.16:6832/1707126831 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f3d24005380 con 0x7f3d1806b100 2023-10-16T12:23:39.736 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule removed for path /snap_test_dir1/dir8 2023-10-16T12:23:39.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.736+0000 7f3d3b3f8700 1 -- 172.21.15.16:0/1362741180 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f3d1806b100 msgr2=0x7f3d1806d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:39.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.736+0000 7f3d3b3f8700 1 --2- 172.21.15.16:0/1362741180 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f3d1806b100 0x7f3d1806d5c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3d340b6120 tx=0x7f3d24009040 comp rx=0 tx=0).stop 2023-10-16T12:23:39.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.736+0000 7f3d3b3f8700 1 -- 172.21.15.16:0/1362741180 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d340b3030 msgr2=0x7f3d340b45d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:39.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.736+0000 7f3d3b3f8700 1 --2- 172.21.15.16:0/1362741180 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d340b3030 0x7f3d340b45d0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f3d30011bb0 tx=0x7f3d30009f70 comp rx=0 tx=0).stop 2023-10-16T12:23:39.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.736+0000 7f3d3b3f8700 1 -- 172.21.15.16:0/1362741180 shutdown_connections 2023-10-16T12:23:39.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.736+0000 7f3d3b3f8700 1 --2- 172.21.15.16:0/1362741180 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f3d1806b100 0x7f3d1806d5c0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:39.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.736+0000 7f3d3b3f8700 1 --2- 172.21.15.16:0/1362741180 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3d3415d1c0 0x7f3d34158ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:39.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.736+0000 7f3d3b3f8700 1 --2- 172.21.15.16:0/1362741180 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3d20004d90 0x7f3d3414fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:39.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.736+0000 7f3d3b3f8700 1 --2- 172.21.15.16:0/1362741180 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d340b3030 0x7f3d340b45d0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:39.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.736+0000 7f3d3b3f8700 1 -- 172.21.15.16:0/1362741180 >> 172.21.15.16:0/1362741180 conn(0x7f3d340b1f40 msgr2=0x7f3d341548d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:39.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.736+0000 7f3d3b3f8700 1 -- 172.21.15.16:0/1362741180 shutdown_connections 2023-10-16T12:23:39.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:39.736+0000 7f3d3b3f8700 1 -- 172.21.15.16:0/1362741180 wait complete. 2023-10-16T12:23:39.750 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir8 2023-10-16T12:23:39.750 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:39.750 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir8/.snap) 2023-10-16T12:23:39.769 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_13_00_UTC 2023-10-16T12:23:39.770 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_14_00_UTC 2023-10-16T12:23:39.770 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_15_00_UTC 2023-10-16T12:23:39.770 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_16_00_UTC 2023-10-16T12:23:39.770 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_17_00_UTC 2023-10-16T12:23:39.770 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_18_00_UTC 2023-10-16T12:23:39.770 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_19_00_UTC 2023-10-16T12:23:39.771 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_20_00_UTC 2023-10-16T12:23:39.771 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_21_00_UTC 2023-10-16T12:23:39.771 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_22_00_UTC 2023-10-16T12:23:39.771 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_23_00_UTC 2023-10-16T12:23:39.771 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-16-12_13_00_UTC 2023-10-16T12:23:39.772 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:39.772 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-16-12_13_00_UTC) 2023-10-16T12:23:39.834 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-16-12_14_00_UTC 2023-10-16T12:23:39.835 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:39.835 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-16-12_14_00_UTC) 2023-10-16T12:23:39.899 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-16-12_15_00_UTC 2023-10-16T12:23:39.899 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:39.900 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-16-12_15_00_UTC) 2023-10-16T12:23:39.962 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-16-12_16_00_UTC 2023-10-16T12:23:39.963 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:39.963 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-16-12_16_00_UTC) 2023-10-16T12:23:40.027 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-16-12_17_00_UTC 2023-10-16T12:23:40.027 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:40.028 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-16-12_17_00_UTC) 2023-10-16T12:23:40.092 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-16-12_18_00_UTC 2023-10-16T12:23:40.092 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:40.093 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-16-12_18_00_UTC) 2023-10-16T12:23:40.155 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-16-12_19_00_UTC 2023-10-16T12:23:40.155 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:40.156 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-16-12_19_00_UTC) 2023-10-16T12:23:40.219 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-16-12_20_00_UTC 2023-10-16T12:23:40.220 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:40.220 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-16-12_20_00_UTC) 2023-10-16T12:23:40.283 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-16-12_21_00_UTC 2023-10-16T12:23:40.284 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:40.284 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-16-12_21_00_UTC) 2023-10-16T12:23:40.348 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-16-12_22_00_UTC 2023-10-16T12:23:40.348 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:40.348 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-16-12_22_00_UTC) 2023-10-16T12:23:40.431 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-16-12_23_00_UTC 2023-10-16T12:23:40.432 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:40.432 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-16-12_23_00_UTC) 2023-10-16T12:23:40.467 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:40.467 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8) 2023-10-16T12:23:40.527 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir9 1m 2023-10-16T12:23:40.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:40.697+0000 7f66d691e700 1 Processor -- start 2023-10-16T12:23:40.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:40.697+0000 7f66d691e700 1 -- start start 2023-10-16T12:23:40.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:40.698+0000 7f66d691e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f66d015b070 0x7f66d015b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:40.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:40.698+0000 7f66d691e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f66d006e170 0x7f66d00cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:40.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:40.698+0000 7f66d691e700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66d00cef70 con 0x7f66d00cb880 2023-10-16T12:23:40.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:40.698+0000 7f66d691e700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66d00ce360 con 0x7f66d015b070 2023-10-16T12:23:40.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:40.698+0000 7f66d691e700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66d00ce640 con 0x7f66d006e170 2023-10-16T12:23:40.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:40.698+0000 7f66cffff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f66d015b070 0x7f66d015b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:40.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:40.698+0000 7f66cf7fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f66d00cb880 0x7f66d006da60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54422/0 (socket says 172.21.15.16:54422) 2023-10-16T12:23:40.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:40.698+0000 7f66cf7fe700 1 -- 172.21.15.16:0/756385627 learned_addr learned my addr 172.21.15.16:0/756385627 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:23:40.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:40.698+0000 7f66d4ebb700 1 --2- 172.21.15.16:0/756385627 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f66d006e170 0x7f66d00cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:40.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:40.698+0000 7f66cffff700 1 -- 172.21.15.16:0/756385627 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f66d006e170 msgr2=0x7f66d00cbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:40.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:40.698+0000 7f66cffff700 1 --2- 172.21.15.16:0/756385627 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f66d006e170 0x7f66d00cbd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:40.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:40.698+0000 7f66cffff700 1 -- 172.21.15.16:0/756385627 >> v1:172.21.15.16:6789/0 conn(0x7f66d00cb880 legacy=0x7f66d006da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:40.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:40.698+0000 7f66cffff700 1 -- 172.21.15.16:0/756385627 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66d00ce920 con 0x7f66d015b070 2023-10-16T12:23:40.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:40.698+0000 7f66ceffd700 1 -- 172.21.15.16:0/756385627 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2924876500 0 0) 0x7f66d00cef70 con 0x7f66d00cb880 2023-10-16T12:23:40.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:40.698+0000 7f66cffff700 1 --2- 172.21.15.16:0/756385627 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66d015b070 0x7f66d015b450 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f66c0009840 tx=0x7f66c0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b1bbc348755af52 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:40.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:40.698+0000 7f66ceffd700 1 -- 172.21.15.16:0/756385627 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66c0011830 con 0x7f66d015b070 2023-10-16T12:23:40.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:40.699+0000 7f66ceffd700 1 -- 172.21.15.16:0/756385627 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66c0011990 con 0x7f66d015b070 2023-10-16T12:23:40.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:40.699+0000 7f66ceffd700 1 -- 172.21.15.16:0/756385627 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66c0011d20 con 0x7f66d015b070 2023-10-16T12:23:40.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:40.699+0000 7f66d691e700 1 -- 172.21.15.16:0/756385627 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66d015b070 msgr2=0x7f66d015b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:40.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:40.699+0000 7f66d691e700 1 --2- 172.21.15.16:0/756385627 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66d015b070 0x7f66d015b450 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f66c0009840 tx=0x7f66c0009b50 comp rx=0 tx=0).stop 2023-10-16T12:23:40.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:40.699+0000 7f66d691e700 1 -- 172.21.15.16:0/756385627 shutdown_connections 2023-10-16T12:23:40.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:40.699+0000 7f66d691e700 1 --2- 172.21.15.16:0/756385627 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f66d006e170 0x7f66d00cbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:40.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:40.699+0000 7f66d691e700 1 --2- 172.21.15.16:0/756385627 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66d015b070 0x7f66d015b450 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:40.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:40.699+0000 7f66d691e700 1 -- 172.21.15.16:0/756385627 >> 172.21.15.16:0/756385627 conn(0x7f66d00b1f40 msgr2=0x7f66d00b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:40.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:40.699+0000 7f66d691e700 1 -- 172.21.15.16:0/756385627 shutdown_connections 2023-10-16T12:23:40.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:40.699+0000 7f66d691e700 1 -- 172.21.15.16:0/756385627 wait complete. 2023-10-16T12:23:40.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:40.700+0000 7f66d691e700 1 Processor -- start 2023-10-16T12:23:40.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:40.700+0000 7f66d691e700 1 -- start start 2023-10-16T12:23:40.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:40.701+0000 7f66d691e700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f66d006e170 0x7f66d0153ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:40.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:40.701+0000 7f66d691e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f66d00cb880 0x7f66d01eff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:40.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:40.701+0000 7f66d691e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66d015b070 0x7f66d0156500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:40.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:40.701+0000 7f66d691e700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66d0154220 con 0x7f66d015b070 2023-10-16T12:23:40.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:40.701+0000 7f66d691e700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66d00b34c0 con 0x7f66d006e170 2023-10-16T12:23:40.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:40.701+0000 7f66d691e700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66d00b37a0 con 0x7f66d00cb880 2023-10-16T12:23:40.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:40.701+0000 7f66d4ebb700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66d015b070 0x7f66d0156500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:40.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:40.701+0000 7f66d4ebb700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66d015b070 0x7f66d0156500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36924/0 (socket says 172.21.15.16:36924) 2023-10-16T12:23:40.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:40.701+0000 7f66d4ebb700 1 -- 172.21.15.16:0/1542876922 learned_addr learned my addr 172.21.15.16:0/1542876922 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:40.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:40.702+0000 7f66cf7fe700 1 --2- 172.21.15.16:0/1542876922 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f66d00cb880 0x7f66d01eff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:40.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:40.702+0000 7f66cffff700 1 --2- 172.21.15.16:0/1542876922 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f66d006e170 0x7f66d0153ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:40.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:40.702+0000 7f66d4ebb700 1 -- 172.21.15.16:0/1542876922 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f66d00cb880 msgr2=0x7f66d01eff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:40.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:40.702+0000 7f66d4ebb700 1 --2- 172.21.15.16:0/1542876922 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f66d00cb880 0x7f66d01eff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:40.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:40.702+0000 7f66d4ebb700 1 -- 172.21.15.16:0/1542876922 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f66d006e170 msgr2=0x7f66d0153ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:40.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:40.702+0000 7f66d4ebb700 1 --2- 172.21.15.16:0/1542876922 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f66d006e170 0x7f66d0153ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:40.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:40.702+0000 7f66d4ebb700 1 -- 172.21.15.16:0/1542876922 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66c00094a0 con 0x7f66d015b070 2023-10-16T12:23:40.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:40.702+0000 7f66cffff700 1 --2- 172.21.15.16:0/1542876922 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f66d006e170 0x7f66d0153ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:23:40.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:40.702+0000 7f66d4ebb700 1 --2- 172.21.15.16:0/1542876922 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66d015b070 0x7f66d0156500 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f66c800efc0 tx=0x7f66c800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:40.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:40.702+0000 7f66ccff9700 1 -- 172.21.15.16:0/1542876922 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66c8009ea0 con 0x7f66d015b070 2023-10-16T12:23:40.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:40.702+0000 7f66ccff9700 1 -- 172.21.15.16:0/1542876922 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66c800f040 con 0x7f66d015b070 2023-10-16T12:23:40.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:40.702+0000 7f66d691e700 1 -- 172.21.15.16:0/1542876922 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66d00b3a80 con 0x7f66d015b070 2023-10-16T12:23:40.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:40.702+0000 7f66ccff9700 1 -- 172.21.15.16:0/1542876922 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66c8014480 con 0x7f66d015b070 2023-10-16T12:23:40.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:40.702+0000 7f66d691e700 1 -- 172.21.15.16:0/1542876922 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66d01f3500 con 0x7f66d015b070 2023-10-16T12:23:40.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:40.703+0000 7f66d691e700 1 -- 172.21.15.16:0/1542876922 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66d004ebb0 con 0x7f66d015b070 2023-10-16T12:23:40.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:40.703+0000 7f66ccff9700 1 -- 172.21.15.16:0/1542876922 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f66c80145e0 con 0x7f66d015b070 2023-10-16T12:23:40.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:40.704+0000 7f66ccff9700 1 --2- 172.21.15.16:0/1542876922 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f66bc06b100 0x7f66bc06d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:40.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:40.704+0000 7f66ccff9700 1 -- 172.21.15.16:0/1542876922 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6868+0+0 (secure 0 0 0) 0x7f66c808a730 con 0x7f66d015b070 2023-10-16T12:23:40.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:40.704+0000 7f66cffff700 1 --2- 172.21.15.16:0/1542876922 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f66bc06b100 0x7f66bc06d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:40.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:40.704+0000 7f66cffff700 1 --2- 172.21.15.16:0/1542876922 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f66bc06b100 0x7f66bc06d5c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f66c00090b0 tx=0x7f66c001c040 comp rx=0 tx=0).ready entity=mgr.5499 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:40.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:40.709+0000 7f66ccff9700 1 -- 172.21.15.16:0/1542876922 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f66c8055050 con 0x7f66d015b070 2023-10-16T12:23:40.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:40.940+0000 7f66d691e700 1 -- 172.21.15.16:0/1542876922 --> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir9", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f66d00728d0 con 0x7f66bc06b100 2023-10-16T12:23:41.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:41.016+0000 7f66ccff9700 1 -- 172.21.15.16:0/1542876922 <== mgr.5499 v2:172.21.15.16:6832/1707126831 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f66c0011c00 con 0x7f66bc06b100 2023-10-16T12:23:41.019 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule removed for path /snap_test_dir1/dir9 2023-10-16T12:23:41.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:41.019+0000 7f66d691e700 1 -- 172.21.15.16:0/1542876922 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f66bc06b100 msgr2=0x7f66bc06d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:41.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:41.019+0000 7f66d691e700 1 --2- 172.21.15.16:0/1542876922 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f66bc06b100 0x7f66bc06d5c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f66c00090b0 tx=0x7f66c001c040 comp rx=0 tx=0).stop 2023-10-16T12:23:41.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:41.019+0000 7f66d691e700 1 -- 172.21.15.16:0/1542876922 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66d015b070 msgr2=0x7f66d0156500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:41.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:41.019+0000 7f66d691e700 1 --2- 172.21.15.16:0/1542876922 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66d015b070 0x7f66d0156500 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f66c800efc0 tx=0x7f66c800c3a0 comp rx=0 tx=0).stop 2023-10-16T12:23:41.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:41.019+0000 7f66d691e700 1 -- 172.21.15.16:0/1542876922 shutdown_connections 2023-10-16T12:23:41.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:41.019+0000 7f66d691e700 1 --2- 172.21.15.16:0/1542876922 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f66bc06b100 0x7f66bc06d5c0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:41.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:41.019+0000 7f66d691e700 1 --2- 172.21.15.16:0/1542876922 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66d015b070 0x7f66d0156500 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:41.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:41.019+0000 7f66d691e700 1 --2- 172.21.15.16:0/1542876922 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f66d006e170 0x7f66d0153ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:41.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:41.019+0000 7f66d691e700 1 --2- 172.21.15.16:0/1542876922 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f66d00cb880 0x7f66d01eff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:41.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:41.019+0000 7f66d691e700 1 -- 172.21.15.16:0/1542876922 >> 172.21.15.16:0/1542876922 conn(0x7f66d00b1f40 msgr2=0x7f66d00c69b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:41.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:41.019+0000 7f66d691e700 1 -- 172.21.15.16:0/1542876922 shutdown_connections 2023-10-16T12:23:41.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:41.019+0000 7f66d691e700 1 -- 172.21.15.16:0/1542876922 wait complete. 2023-10-16T12:23:41.033 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir9 2023-10-16T12:23:41.033 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:41.034 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir9/.snap) 2023-10-16T12:23:41.052 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_13_00_UTC 2023-10-16T12:23:41.052 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_14_00_UTC 2023-10-16T12:23:41.053 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_15_00_UTC 2023-10-16T12:23:41.053 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_16_00_UTC 2023-10-16T12:23:41.053 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_17_00_UTC 2023-10-16T12:23:41.053 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_18_00_UTC 2023-10-16T12:23:41.053 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_19_00_UTC 2023-10-16T12:23:41.054 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_20_00_UTC 2023-10-16T12:23:41.054 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_21_00_UTC 2023-10-16T12:23:41.054 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_22_00_UTC 2023-10-16T12:23:41.054 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_23_00_UTC 2023-10-16T12:23:41.054 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-16-12_13_00_UTC 2023-10-16T12:23:41.055 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:41.055 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-16-12_13_00_UTC) 2023-10-16T12:23:41.118 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-16-12_14_00_UTC 2023-10-16T12:23:41.119 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:41.119 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-16-12_14_00_UTC) 2023-10-16T12:23:41.181 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-16-12_15_00_UTC 2023-10-16T12:23:41.181 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:41.182 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-16-12_15_00_UTC) 2023-10-16T12:23:41.270 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-16-12_16_00_UTC 2023-10-16T12:23:41.271 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:41.271 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-16-12_16_00_UTC) 2023-10-16T12:23:41.294 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-16-12_17_00_UTC 2023-10-16T12:23:41.294 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:41.295 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-16-12_17_00_UTC) 2023-10-16T12:23:41.358 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-16-12_18_00_UTC 2023-10-16T12:23:41.359 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:41.359 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-16-12_18_00_UTC) 2023-10-16T12:23:41.421 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-16-12_19_00_UTC 2023-10-16T12:23:41.422 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:41.422 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-16-12_19_00_UTC) 2023-10-16T12:23:41.484 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-16-12_20_00_UTC 2023-10-16T12:23:41.484 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:41.485 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-16-12_20_00_UTC) 2023-10-16T12:23:41.546 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-16-12_21_00_UTC 2023-10-16T12:23:41.547 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:41.547 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-16-12_21_00_UTC) 2023-10-16T12:23:41.609 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-16-12_22_00_UTC 2023-10-16T12:23:41.609 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:41.609 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-16-12_22_00_UTC) 2023-10-16T12:23:41.673 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-16-12_23_00_UTC 2023-10-16T12:23:41.673 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:41.673 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-16-12_23_00_UTC) 2023-10-16T12:23:41.736 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:41.736 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9) 2023-10-16T12:23:41.796 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-16T12:23:41.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:41.966+0000 7ffb464e5700 1 Processor -- start 2023-10-16T12:23:41.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:41.966+0000 7ffb464e5700 1 -- start start 2023-10-16T12:23:41.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:41.967+0000 7ffb464e5700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7ffb400ce6a0 0x7ffb4006c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:41.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:41.967+0000 7ffb464e5700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb400b20f0 con 0x7ffb400d2110 2023-10-16T12:23:41.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:41.967+0000 7ffb464e5700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb400af1f0 con 0x7ffb400b2620 2023-10-16T12:23:41.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:41.967+0000 7ffb464e5700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb400b1d70 con 0x7ffb400ce6a0 2023-10-16T12:23:41.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:41.967+0000 7ffb3f7fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ffb400d2110 0x7ffb400b0f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54436/0 (socket says 172.21.15.16:54436) 2023-10-16T12:23:41.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:41.967+0000 7ffb3f7fe700 1 -- 172.21.15.16:0/3353390998 learned_addr learned my addr 172.21.15.16:0/3353390998 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:23:41.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:41.967+0000 7ffb3ffff700 1 --2- 172.21.15.16:0/3353390998 >> v2:172.21.15.49:3300/0 conn(0x7ffb400ce6a0 0x7ffb4006c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:41.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:41.967+0000 7ffb3effd700 1 -- 172.21.15.16:0/3353390998 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 475760982 0 0) 0x7ffb400b20f0 con 0x7ffb400d2110 2023-10-16T12:23:41.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:41.967+0000 7ffb3effd700 1 -- 172.21.15.16:0/3353390998 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb2c0035b0 con 0x7ffb400d2110 2023-10-16T12:23:41.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:41.968+0000 7ffb3effd700 1 -- 172.21.15.16:0/3353390998 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3704601972 0 0) 0x7ffb400af1f0 con 0x7ffb400b2620 2023-10-16T12:23:41.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:41.968+0000 7ffb3effd700 1 -- 172.21.15.16:0/3353390998 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb400b20f0 con 0x7ffb400b2620 2023-10-16T12:23:41.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:41.968+0000 7ffb3effd700 1 -- 172.21.15.16:0/3353390998 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3118919481 0 0) 0x7ffb2c0035b0 con 0x7ffb400d2110 2023-10-16T12:23:41.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:41.968+0000 7ffb3ffff700 1 -- 172.21.15.16:0/3353390998 >> v1:172.21.15.49:6789/0 conn(0x7ffb400b2620 legacy=0x7ffb400b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:41.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:41.968+0000 7ffb3ffff700 1 -- 172.21.15.16:0/3353390998 >> v1:172.21.15.16:6789/0 conn(0x7ffb400d2110 legacy=0x7ffb400b0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:41.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:41.968+0000 7ffb3ffff700 1 -- 172.21.15.16:0/3353390998 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb400d03d0 con 0x7ffb400ce6a0 2023-10-16T12:23:41.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:41.968+0000 7ffb3ffff700 1 --2- 172.21.15.16:0/3353390998 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ffb400ce6a0 0x7ffb4006c300 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7ffb30009840 tx=0x7ffb30009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2deffd971624452 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:41.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:41.968+0000 7ffb3effd700 1 -- 172.21.15.16:0/3353390998 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb28001670 con 0x7ffb400ce6a0 2023-10-16T12:23:41.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:41.968+0000 7ffb3effd700 1 -- 172.21.15.16:0/3353390998 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb30004030 con 0x7ffb400ce6a0 2023-10-16T12:23:41.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:41.968+0000 7ffb3effd700 1 -- 172.21.15.16:0/3353390998 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb300043c0 con 0x7ffb400ce6a0 2023-10-16T12:23:41.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:41.969+0000 7ffb464e5700 1 -- 172.21.15.16:0/3353390998 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ffb400ce6a0 msgr2=0x7ffb4006c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:41.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:41.969+0000 7ffb464e5700 1 --2- 172.21.15.16:0/3353390998 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ffb400ce6a0 0x7ffb4006c300 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7ffb30009840 tx=0x7ffb30009b50 comp rx=0 tx=0).stop 2023-10-16T12:23:41.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:41.969+0000 7ffb464e5700 1 -- 172.21.15.16:0/3353390998 shutdown_connections 2023-10-16T12:23:41.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:41.969+0000 7ffb464e5700 1 --2- 172.21.15.16:0/3353390998 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ffb400ce6a0 0x7ffb4006c300 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:41.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:41.969+0000 7ffb464e5700 1 -- 172.21.15.16:0/3353390998 >> 172.21.15.16:0/3353390998 conn(0x7ffb400d15f0 msgr2=0x7ffb400d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:41.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:41.969+0000 7ffb464e5700 1 -- 172.21.15.16:0/3353390998 shutdown_connections 2023-10-16T12:23:41.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:41.969+0000 7ffb464e5700 1 -- 172.21.15.16:0/3353390998 wait complete. 2023-10-16T12:23:41.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:41.970+0000 7ffb464e5700 1 Processor -- start 2023-10-16T12:23:41.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:41.970+0000 7ffb464e5700 1 -- start start 2023-10-16T12:23:41.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:41.971+0000 7ffb464e5700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ffb400b2620 0x7ffb401f6e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:41.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:41.971+0000 7ffb464e5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb400ce6a0 0x7ffb401f25a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:41.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:41.971+0000 7ffb464e5700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ffb400d2110 0x7ffb4015f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:41.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:41.971+0000 7ffb464e5700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb401f7340 con 0x7ffb400ce6a0 2023-10-16T12:23:41.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:41.971+0000 7ffb464e5700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb400b4ab0 con 0x7ffb400d2110 2023-10-16T12:23:41.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:41.971+0000 7ffb464e5700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb400b4d90 con 0x7ffb400b2620 2023-10-16T12:23:41.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:41.971+0000 7ffb3f7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb400ce6a0 0x7ffb401f25a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:41.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:41.971+0000 7ffb3ffff700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ffb400b2620 0x7ffb401f6e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:41.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:41.971+0000 7ffb3f7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb400ce6a0 0x7ffb401f25a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36936/0 (socket says 172.21.15.16:36936) 2023-10-16T12:23:41.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:41.971+0000 7ffb3f7fe700 1 -- 172.21.15.16:0/2249653297 learned_addr learned my addr 172.21.15.16:0/2249653297 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:41.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:41.971+0000 7ffb44a82700 1 --2- 172.21.15.16:0/2249653297 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ffb400d2110 0x7ffb4015f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:41.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:41.971+0000 7ffb3ffff700 1 -- 172.21.15.16:0/2249653297 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ffb400d2110 msgr2=0x7ffb4015f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:41.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:41.971+0000 7ffb3ffff700 1 --2- 172.21.15.16:0/2249653297 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ffb400d2110 0x7ffb4015f2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:41.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:41.971+0000 7ffb3ffff700 1 -- 172.21.15.16:0/2249653297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb400ce6a0 msgr2=0x7ffb401f25a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:41.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:41.971+0000 7ffb3ffff700 1 --2- 172.21.15.16:0/2249653297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb400ce6a0 0x7ffb401f25a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:41.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:41.971+0000 7ffb3ffff700 1 -- 172.21.15.16:0/2249653297 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb300094a0 con 0x7ffb400b2620 2023-10-16T12:23:41.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:41.971+0000 7ffb3f7fe700 1 --2- 172.21.15.16:0/2249653297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb400ce6a0 0x7ffb401f25a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:23:41.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:41.972+0000 7ffb3ffff700 1 --2- 172.21.15.16:0/2249653297 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ffb400b2620 0x7ffb401f6e00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ffb30015000 tx=0x7ffb30004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:41.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:41.972+0000 7ffb3cff9700 1 -- 172.21.15.16:0/2249653297 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb30004570 con 0x7ffb400b2620 2023-10-16T12:23:41.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:41.972+0000 7ffb3cff9700 1 -- 172.21.15.16:0/2249653297 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb3001f070 con 0x7ffb400b2620 2023-10-16T12:23:41.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:41.972+0000 7ffb3cff9700 1 -- 172.21.15.16:0/2249653297 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb30011830 con 0x7ffb400b2620 2023-10-16T12:23:41.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:41.972+0000 7ffb464e5700 1 -- 172.21.15.16:0/2249653297 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb400b5070 con 0x7ffb400b2620 2023-10-16T12:23:41.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:41.972+0000 7ffb464e5700 1 -- 172.21.15.16:0/2249653297 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb400b5630 con 0x7ffb400b2620 2023-10-16T12:23:41.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:41.973+0000 7ffb464e5700 1 -- 172.21.15.16:0/2249653297 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb4004ebb0 con 0x7ffb400b2620 2023-10-16T12:23:41.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:41.973+0000 7ffb3cff9700 1 -- 172.21.15.16:0/2249653297 <== mon.2 v2:172.21.15.49:3301/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7ffb30011990 con 0x7ffb400b2620 2023-10-16T12:23:41.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:41.974+0000 7ffb3cff9700 1 --2- 172.21.15.16:0/2249653297 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7ffb2006b100 0x7ffb2006d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:41.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:41.974+0000 7ffb3cff9700 1 -- 172.21.15.16:0/2249653297 <== mon.2 v2:172.21.15.49:3301/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 7376+0+0 (secure 0 0 0) 0x7ffb30021070 con 0x7ffb400b2620 2023-10-16T12:23:41.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:41.974+0000 7ffb3f7fe700 1 --2- 172.21.15.16:0/2249653297 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7ffb2006b100 0x7ffb2006d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:41.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:41.974+0000 7ffb3f7fe700 1 --2- 172.21.15.16:0/2249653297 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7ffb2006b100 0x7ffb2006d5c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ffb28001eb0 tx=0x7ffb28001a20 comp rx=0 tx=0).ready entity=mgr.5499 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:41.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:41.979+0000 7ffb3cff9700 1 -- 172.21.15.16:0/2249653297 <== mon.2 v2:172.21.15.49:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ffb30055f60 con 0x7ffb400b2620 2023-10-16T12:23:42.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:42.250+0000 7ffb464e5700 1 -- 172.21.15.16:0/2249653297 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7ffb400b4ed0 con 0x7ffb400b2620 2023-10-16T12:23:43.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.209+0000 7ffb3cff9700 1 -- 172.21.15.16:0/2249653297 <== mon.2 v2:172.21.15.49:3301/0 7 ==== mgrmap(e 14) v1 ==== 89400+0+0 (secure 0 0 0) 0x7ffb30056330 con 0x7ffb400b2620 2023-10-16T12:23:43.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.209+0000 7ffb3cff9700 1 -- 172.21.15.16:0/2249653297 <== mon.2 v2:172.21.15.49:3301/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v14) v1 ==== 93+0+0 (secure 0 0 0) 0x7ffb30052780 con 0x7ffb400b2620 2023-10-16T12:23:43.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.212+0000 7ffb464e5700 1 -- 172.21.15.16:0/2249653297 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7ffb2006b100 msgr2=0x7ffb2006d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:43.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.212+0000 7ffb464e5700 1 --2- 172.21.15.16:0/2249653297 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7ffb2006b100 0x7ffb2006d5c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ffb28001eb0 tx=0x7ffb28001a20 comp rx=0 tx=0).stop 2023-10-16T12:23:43.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.212+0000 7ffb464e5700 1 -- 172.21.15.16:0/2249653297 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ffb400b2620 msgr2=0x7ffb401f6e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:43.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.212+0000 7ffb464e5700 1 --2- 172.21.15.16:0/2249653297 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ffb400b2620 0x7ffb401f6e00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ffb30015000 tx=0x7ffb30004750 comp rx=0 tx=0).stop 2023-10-16T12:23:43.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.212+0000 7ffb464e5700 1 -- 172.21.15.16:0/2249653297 shutdown_connections 2023-10-16T12:23:43.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.212+0000 7ffb464e5700 1 --2- 172.21.15.16:0/2249653297 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7ffb2006b100 0x7ffb2006d5c0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:43.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.212+0000 7ffb464e5700 1 --2- 172.21.15.16:0/2249653297 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ffb400d2110 0x7ffb4015f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:43.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.212+0000 7ffb464e5700 1 --2- 172.21.15.16:0/2249653297 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ffb400b2620 0x7ffb401f6e00 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:43.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.212+0000 7ffb464e5700 1 --2- 172.21.15.16:0/2249653297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb400ce6a0 0x7ffb401f25a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:43.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.212+0000 7ffb464e5700 1 -- 172.21.15.16:0/2249653297 >> 172.21.15.16:0/2249653297 conn(0x7ffb400d15f0 msgr2=0x7ffb400cccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:43.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.213+0000 7ffb464e5700 1 -- 172.21.15.16:0/2249653297 shutdown_connections 2023-10-16T12:23:43.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.213+0000 7ffb464e5700 1 -- 172.21.15.16:0/2249653297 wait complete. 2023-10-16T12:23:43.238 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T12:23:43.239 DEBUG:teuthology.orchestra.run.smithi016:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T12:23:43.316 DEBUG:teuthology.orchestra.run.smithi049:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T12:23:43.356 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T12:23:43.357 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:23:43.357 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:23:43.375 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:23:43.376 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:23:43.392 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:23:43.390+0000 7fb59d613200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T12:23:43.395 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:23:43.393+0000 7f98e4adb200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T12:23:43.412 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:23:43.461 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:23:43.459+0000 7fb59d613200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T12:23:43.464 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:23:43.462+0000 7f98e4adb200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T12:23:43.536 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:23:43.534+0000 7fb59d613200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T12:23:43.541 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:23:43.539+0000 7f98e4adb200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T12:23:43.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.577+0000 7fcd6816b700 1 Processor -- start 2023-10-16T12:23:43.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.577+0000 7fcd6816b700 1 -- start start 2023-10-16T12:23:43.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.578+0000 7fcd6816b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcd6015d900 0x7fcd600d2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:43.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.578+0000 7fcd6816b700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fcd600ce9f0 0x7fcd600cdc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:43.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.578+0000 7fcd6816b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcd600d2860 0x7fcd600d2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:43.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.578+0000 7fcd6816b700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd600d3230 con 0x7fcd600d2860 2023-10-16T12:23:43.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.578+0000 7fcd6816b700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd600ce1a0 con 0x7fcd600ce9f0 2023-10-16T12:23:43.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.578+0000 7fcd6816b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd600cee70 con 0x7fcd6015d900 2023-10-16T12:23:43.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.578+0000 7fcd66708700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcd600d2860 0x7fcd600d2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:43.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.578+0000 7fcd66708700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcd600d2860 0x7fcd600d2cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36988/0 (socket says 172.21.15.16:36988) 2023-10-16T12:23:43.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.578+0000 7fcd66708700 1 -- 172.21.15.16:0/964649456 learned_addr learned my addr 172.21.15.16:0/964649456 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:43.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.578+0000 7fcd65706700 1 --2- 172.21.15.16:0/964649456 >> v2:172.21.15.49:3300/0 conn(0x7fcd600ce9f0 0x7fcd600cdc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:43.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.578+0000 7fcd65f07700 1 --2- 172.21.15.16:0/964649456 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcd6015d900 0x7fcd600d2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:43.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.578+0000 7fcd66708700 1 -- 172.21.15.16:0/964649456 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcd6015d900 msgr2=0x7fcd600d2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:43.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.578+0000 7fcd66708700 1 --2- 172.21.15.16:0/964649456 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcd6015d900 0x7fcd600d2040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:43.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.578+0000 7fcd66708700 1 -- 172.21.15.16:0/964649456 >> v2:172.21.15.49:3300/0 conn(0x7fcd600ce9f0 msgr2=0x7fcd600cdc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:43.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.578+0000 7fcd66708700 1 --2- 172.21.15.16:0/964649456 >> v2:172.21.15.49:3300/0 conn(0x7fcd600ce9f0 0x7fcd600cdc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:43.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.578+0000 7fcd66708700 1 -- 172.21.15.16:0/964649456 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd600ce400 con 0x7fcd600d2860 2023-10-16T12:23:43.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.578+0000 7fcd65706700 1 --2- 172.21.15.16:0/964649456 >> v2:172.21.15.49:3300/0 conn(0x7fcd600ce9f0 0x7fcd600cdc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:23:43.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.579+0000 7fcd66708700 1 --2- 172.21.15.16:0/964649456 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd600d2860 0x7fcd600d2cf0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fcd600b6220 tx=0x7fcd5800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcb48986ee92a4c7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:43.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.579+0000 7fcd64f05700 1 -- 172.21.15.16:0/964649456 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd5800e070 con 0x7fcd600d2860 2023-10-16T12:23:43.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.579+0000 7fcd64f05700 1 -- 172.21.15.16:0/964649456 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd5800b910 con 0x7fcd600d2860 2023-10-16T12:23:43.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.579+0000 7fcd64f05700 1 -- 172.21.15.16:0/964649456 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd5800bcc0 con 0x7fcd600d2860 2023-10-16T12:23:43.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.579+0000 7fcd6816b700 1 -- 172.21.15.16:0/964649456 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd600d2860 msgr2=0x7fcd600d2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:43.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.579+0000 7fcd6816b700 1 --2- 172.21.15.16:0/964649456 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd600d2860 0x7fcd600d2cf0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fcd600b6220 tx=0x7fcd5800b330 comp rx=0 tx=0).stop 2023-10-16T12:23:43.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.579+0000 7fcd6816b700 1 -- 172.21.15.16:0/964649456 shutdown_connections 2023-10-16T12:23:43.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.579+0000 7fcd6816b700 1 --2- 172.21.15.16:0/964649456 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd600d2860 0x7fcd600d2cf0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:43.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.579+0000 7fcd6816b700 1 --2- 172.21.15.16:0/964649456 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcd6015d900 0x7fcd600d2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:43.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.579+0000 7fcd6816b700 1 --2- 172.21.15.16:0/964649456 >> v2:172.21.15.49:3300/0 conn(0x7fcd600ce9f0 0x7fcd600cdc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:43.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.580+0000 7fcd6816b700 1 -- 172.21.15.16:0/964649456 >> 172.21.15.16:0/964649456 conn(0x7fcd60072330 msgr2=0x7fcd60072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:43.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.580+0000 7fcd6816b700 1 -- 172.21.15.16:0/964649456 shutdown_connections 2023-10-16T12:23:43.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.580+0000 7fcd6816b700 1 -- 172.21.15.16:0/964649456 wait complete. 2023-10-16T12:23:43.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.581+0000 7fcd6816b700 1 Processor -- start 2023-10-16T12:23:43.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.581+0000 7fcd6816b700 1 -- start start 2023-10-16T12:23:43.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.582+0000 7fcd6816b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcd600ce9f0 0x7fcd60151160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:43.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.582+0000 7fcd6816b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcd600d2860 0x7fcd6015ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:43.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.582+0000 7fcd6816b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd6015d900 0x7fcd6015f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:43.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.582+0000 7fcd6816b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd6015f980 con 0x7fcd6015d900 2023-10-16T12:23:43.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.582+0000 7fcd6816b700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd601f3df0 con 0x7fcd600ce9f0 2023-10-16T12:23:43.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.582+0000 7fcd6816b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd601f40d0 con 0x7fcd600d2860 2023-10-16T12:23:43.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.582+0000 7fcd66708700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd6015d900 0x7fcd6015f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:43.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.582+0000 7fcd65f07700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcd600ce9f0 0x7fcd60151160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:43.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.582+0000 7fcd66708700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd6015d900 0x7fcd6015f350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36994/0 (socket says 172.21.15.16:36994) 2023-10-16T12:23:43.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.582+0000 7fcd65706700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcd600d2860 0x7fcd6015ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:43.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.582+0000 7fcd66708700 1 -- 172.21.15.16:0/3590729447 learned_addr learned my addr 172.21.15.16:0/3590729447 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:43.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.583+0000 7fcd66708700 1 -- 172.21.15.16:0/3590729447 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcd600d2860 msgr2=0x7fcd6015ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:43.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.583+0000 7fcd66708700 1 --2- 172.21.15.16:0/3590729447 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcd600d2860 0x7fcd6015ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:43.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.583+0000 7fcd66708700 1 -- 172.21.15.16:0/3590729447 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcd600ce9f0 msgr2=0x7fcd60151160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:43.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.583+0000 7fcd66708700 1 --2- 172.21.15.16:0/3590729447 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcd600ce9f0 0x7fcd60151160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:43.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.583+0000 7fcd66708700 1 -- 172.21.15.16:0/3590729447 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd500094a0 con 0x7fcd6015d900 2023-10-16T12:23:43.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.583+0000 7fcd65f07700 1 --2- 172.21.15.16:0/3590729447 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcd600ce9f0 0x7fcd60151160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:23:43.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.583+0000 7fcd66708700 1 --2- 172.21.15.16:0/3590729447 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd6015d900 0x7fcd6015f350 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fcd600d0f20 tx=0x7fcd58004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:43.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.583+0000 7fcd567fc700 1 -- 172.21.15.16:0/3590729447 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd580099b0 con 0x7fcd6015d900 2023-10-16T12:23:43.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.583+0000 7fcd567fc700 1 -- 172.21.15.16:0/3590729447 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd5800ce80 con 0x7fcd6015d900 2023-10-16T12:23:43.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.583+0000 7fcd567fc700 1 -- 172.21.15.16:0/3590729447 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd58004490 con 0x7fcd6015d900 2023-10-16T12:23:43.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.583+0000 7fcd6816b700 1 -- 172.21.15.16:0/3590729447 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd58009b30 con 0x7fcd6015d900 2023-10-16T12:23:43.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.583+0000 7fcd6816b700 1 -- 172.21.15.16:0/3590729447 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd601f4690 con 0x7fcd6015d900 2023-10-16T12:23:43.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.584+0000 7fcd6816b700 1 -- 172.21.15.16:0/3590729447 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd6004ebb0 con 0x7fcd6015d900 2023-10-16T12:23:43.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.585+0000 7fcd567fc700 1 -- 172.21.15.16:0/3590729447 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 14) v1 ==== 89400+0+0 (secure 0 0 0) 0x7fcd580045f0 con 0x7fcd6015d900 2023-10-16T12:23:43.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.585+0000 7fcd567fc700 1 --2- 172.21.15.16:0/3590729447 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fcd4406b0b0 0x7fcd4406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:43.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.585+0000 7fcd65f07700 1 -- 172.21.15.16:0/3590729447 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fcd4406b0b0 msgr2=0x7fcd4406d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1707126831 2023-10-16T12:23:43.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.585+0000 7fcd567fc700 1 -- 172.21.15.16:0/3590729447 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6980+0+0 (secure 0 0 0) 0x7fcd5808b260 con 0x7fcd6015d900 2023-10-16T12:23:43.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.585+0000 7fcd65f07700 1 --2- 172.21.15.16:0/3590729447 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fcd4406b0b0 0x7fcd4406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:23:43.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.590+0000 7fcd567fc700 1 -- 172.21.15.16:0/3590729447 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcd58055a80 con 0x7fcd6015d900 2023-10-16T12:23:43.617 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:23:43.615+0000 7fb59d613200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T12:23:43.626 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:23:43.625+0000 7f98e4adb200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T12:23:43.779 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:23:43.777+0000 7fb59d613200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T12:23:43.781 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:23:43.779+0000 7f98e4adb200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T12:23:43.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.786+0000 7fcd65f07700 1 -- 172.21.15.16:0/3590729447 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fcd4406b0b0 msgr2=0x7fcd4406d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1707126831 2023-10-16T12:23:43.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.786+0000 7fcd65f07700 1 --2- 172.21.15.16:0/3590729447 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fcd4406b0b0 0x7fcd4406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:23:43.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.800+0000 7fcd6816b700 1 -- 172.21.15.16:0/3590729447 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcd6015e1c0 con 0x7fcd6015d900 2023-10-16T12:23:43.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.801+0000 7fcd567fc700 1 -- 172.21.15.16:0/3590729447 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v53) v1 ==== 74+0+18058 (secure 0 0 0) 0x7fcd580590a0 con 0x7fcd6015d900 2023-10-16T12:23:43.804 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:23:43.804 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":53,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:23:43.201579+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:11:58.700868+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2023-10-16T12:11:59.495384+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":220,"snap_epoch":53,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.7","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/1300148187":"2023-10-17T12:12:15.525647+0000","172.21.15.16:0/675356848":"2023-10-17T12:12:15.525647+0000","172.21.15.16:6832/365921436":"2023-10-17T12:12:15.525647+0000","172.21.15.16:0/3547353312":"2023-10-17T12:12:15.525647+0000","172.21.15.16:0/1893346008":"2023-10-17T12:12:15.525647+0000","172.21.15.16:6833/365921436":"2023-10-17T12:12:15.525647+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":3},{"begin":7,"length":1},{"begin":9,"length":2},{"begin":14,"length":3},{"begin":18,"length":1},{"begin":20,"length":2},{"begin":24,"length":3},{"begin":28,"length":1},{"begin":30,"length":3},{"begin":35,"length":3},{"begin":39,"length":1},{"begin":41,"length":3},{"begin":46,"length":3},{"begin":50,"length":1},{"begin":52,"length":2},{"begin":56,"length":3},{"begin":60,"length":1},{"begin":63,"length":2},{"begin":67,"length":3},{"begin":71,"length":2},{"begin":74,"length":2},{"begin":78,"length":3},{"begin":83,"length":1},{"begin":85,"length":2},{"begin":89,"length":3},{"begin":93,"length":1},{"begin":95,"length":2},{"begin":99,"length":4},{"begin":106,"length":3},{"begin":110,"length":2},{"begin":155,"length":66}]}],"new_removed_snaps":[{"pool":6,"snaps":[{"begin":18,"length":1},{"begin":28,"length":1},{"begin":39,"length":1},{"begin":50,"length":1},{"begin":60,"length":1},{"begin":71,"length":2},{"begin":83,"length":1},{"begin":93,"length":1},{"begin":212,"length":9}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:23:43.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.804+0000 7fcd6816b700 1 -- 172.21.15.16:0/3590729447 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fcd4406b0b0 msgr2=0x7fcd4406d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:23:43.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.804+0000 7fcd6816b700 1 --2- 172.21.15.16:0/3590729447 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fcd4406b0b0 0x7fcd4406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:43.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.804+0000 7fcd6816b700 1 -- 172.21.15.16:0/3590729447 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd6015d900 msgr2=0x7fcd6015f350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:43.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.804+0000 7fcd6816b700 1 --2- 172.21.15.16:0/3590729447 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd6015d900 0x7fcd6015f350 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fcd600d0f20 tx=0x7fcd58004ad0 comp rx=0 tx=0).stop 2023-10-16T12:23:43.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.804+0000 7fcd6816b700 1 -- 172.21.15.16:0/3590729447 shutdown_connections 2023-10-16T12:23:43.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.804+0000 7fcd6816b700 1 --2- 172.21.15.16:0/3590729447 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fcd4406b0b0 0x7fcd4406d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:43.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.804+0000 7fcd6816b700 1 --2- 172.21.15.16:0/3590729447 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd6015d900 0x7fcd6015f350 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:43.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.804+0000 7fcd6816b700 1 --2- 172.21.15.16:0/3590729447 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcd600ce9f0 0x7fcd60151160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:43.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.804+0000 7fcd6816b700 1 --2- 172.21.15.16:0/3590729447 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcd600d2860 0x7fcd6015ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:43.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.804+0000 7fcd6816b700 1 -- 172.21.15.16:0/3590729447 >> 172.21.15.16:0/3590729447 conn(0x7fcd60072330 msgr2=0x7fcd60158280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:43.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.804+0000 7fcd6816b700 1 -- 172.21.15.16:0/3590729447 shutdown_connections 2023-10-16T12:23:43.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.804+0000 7fcd6816b700 1 -- 172.21.15.16:0/3590729447 wait complete. 2023-10-16T12:23:43.819 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T12:11:58.700868+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T12:11:59.495384+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '53', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 220, 'snap_epoch': 53, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T12:23:43.820 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:23:43.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.985+0000 7f7d28c8c700 1 Processor -- start 2023-10-16T12:23:43.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.985+0000 7f7d28c8c700 1 -- start start 2023-10-16T12:23:43.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.986+0000 7f7d28c8c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7d2406e570 0x7f7d2415e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:43.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.986+0000 7f7d28c8c700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d2415b6a0 con 0x7f7d2406a7e0 2023-10-16T12:23:43.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.986+0000 7f7d28c8c700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d24161b20 con 0x7f7d2415b2c0 2023-10-16T12:23:43.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.986+0000 7f7d28c8c700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d24161e00 con 0x7f7d2406e570 2023-10-16T12:23:43.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.987+0000 7f7d21d9b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7d2406e570 0x7f7d2415e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:43.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.987+0000 7f7d22d9d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f7d2406a7e0 0x7f7d24071a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54526/0 (socket says 172.21.15.16:54526) 2023-10-16T12:23:43.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.987+0000 7f7d21d9b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7d2406e570 0x7f7d2415e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37018/0 (socket says 172.21.15.16:37018) 2023-10-16T12:23:43.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.987+0000 7f7d22d9d700 1 -- 172.21.15.16:0/135125531 learned_addr learned my addr 172.21.15.16:0/135125531 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:23:43.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.987+0000 7f7d21d9b700 1 -- 172.21.15.16:0/135125531 >> v1:172.21.15.49:6789/0 conn(0x7f7d2415b2c0 legacy=0x7f7d24161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:43.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.987+0000 7f7d21d9b700 1 -- 172.21.15.16:0/135125531 >> v1:172.21.15.16:6789/0 conn(0x7f7d2406a7e0 legacy=0x7f7d24071a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:43.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.987+0000 7f7d21d9b700 1 -- 172.21.15.16:0/135125531 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d241620e0 con 0x7f7d2406e570 2023-10-16T12:23:43.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.987+0000 7f7d21d9b700 1 --2- 172.21.15.16:0/135125531 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d2406e570 0x7f7d2415e5a0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f7d14009840 tx=0x7f7d14009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e4281b387995d32 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:43.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.987+0000 7f7d2159a700 1 -- 172.21.15.16:0/135125531 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d14011830 con 0x7f7d2406e570 2023-10-16T12:23:43.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.987+0000 7f7d2159a700 1 -- 172.21.15.16:0/135125531 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d14011990 con 0x7f7d2406e570 2023-10-16T12:23:43.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.987+0000 7f7d2159a700 1 -- 172.21.15.16:0/135125531 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d14011d20 con 0x7f7d2406e570 2023-10-16T12:23:43.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.988+0000 7f7d28c8c700 1 -- 172.21.15.16:0/135125531 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d2406e570 msgr2=0x7f7d2415e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:43.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.988+0000 7f7d28c8c700 1 --2- 172.21.15.16:0/135125531 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d2406e570 0x7f7d2415e5a0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f7d14009840 tx=0x7f7d14009b50 comp rx=0 tx=0).stop 2023-10-16T12:23:43.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.988+0000 7f7d28c8c700 1 -- 172.21.15.16:0/135125531 shutdown_connections 2023-10-16T12:23:43.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.988+0000 7f7d28c8c700 1 --2- 172.21.15.16:0/135125531 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d2406e570 0x7f7d2415e5a0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:43.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.988+0000 7f7d28c8c700 1 -- 172.21.15.16:0/135125531 >> 172.21.15.16:0/135125531 conn(0x7f7d240b1f40 msgr2=0x7f7d240b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:43.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.988+0000 7f7d28c8c700 1 -- 172.21.15.16:0/135125531 shutdown_connections 2023-10-16T12:23:43.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.988+0000 7f7d28c8c700 1 -- 172.21.15.16:0/135125531 wait complete. 2023-10-16T12:23:43.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.989+0000 7f7d28c8c700 1 Processor -- start 2023-10-16T12:23:43.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.989+0000 7f7d28c8c700 1 -- start start 2023-10-16T12:23:43.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.990+0000 7f7d28c8c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d2406a7e0 0x7f7d240b5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:43.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.990+0000 7f7d28c8c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7d2406e570 0x7f7d241f64a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:43.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.990+0000 7f7d28c8c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7d2415b2c0 0x7f7d241f69e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:43.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.990+0000 7f7d28c8c700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d24154610 con 0x7f7d2406a7e0 2023-10-16T12:23:43.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.990+0000 7f7d28c8c700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d24159eb0 con 0x7f7d2406e570 2023-10-16T12:23:43.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.990+0000 7f7d28c8c700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d240c7c20 con 0x7f7d2415b2c0 2023-10-16T12:23:43.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.990+0000 7f7d2259c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d2406a7e0 0x7f7d240b5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:43.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.990+0000 7f7d2259c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d2406a7e0 0x7f7d240b5730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37022/0 (socket says 172.21.15.16:37022) 2023-10-16T12:23:43.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.990+0000 7f7d2259c700 1 -- 172.21.15.16:0/1834805773 learned_addr learned my addr 172.21.15.16:0/1834805773 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:43.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.990+0000 7f7d21d9b700 1 --2- 172.21.15.16:0/1834805773 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7d2406e570 0x7f7d241f64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:43.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.990+0000 7f7d22d9d700 1 --2- 172.21.15.16:0/1834805773 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7d2415b2c0 0x7f7d241f69e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:43.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.991+0000 7f7d2259c700 1 -- 172.21.15.16:0/1834805773 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7d2415b2c0 msgr2=0x7f7d241f69e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:43.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.991+0000 7f7d2259c700 1 --2- 172.21.15.16:0/1834805773 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7d2415b2c0 0x7f7d241f69e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:43.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.991+0000 7f7d2259c700 1 -- 172.21.15.16:0/1834805773 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7d2406e570 msgr2=0x7f7d241f64a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:43.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.991+0000 7f7d2259c700 1 --2- 172.21.15.16:0/1834805773 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7d2406e570 0x7f7d241f64a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:43.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.991+0000 7f7d2259c700 1 -- 172.21.15.16:0/1834805773 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d140094a0 con 0x7f7d2406a7e0 2023-10-16T12:23:44.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.991+0000 7f7d2259c700 1 --2- 172.21.15.16:0/1834805773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d2406a7e0 0x7f7d240b5730 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f7d1800c450 tx=0x7f7d1800c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:44.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.991+0000 7f7d12ffd700 1 -- 172.21.15.16:0/1834805773 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d180052d0 con 0x7f7d2406a7e0 2023-10-16T12:23:44.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.991+0000 7f7d12ffd700 1 -- 172.21.15.16:0/1834805773 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d18005430 con 0x7f7d2406a7e0 2023-10-16T12:23:44.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.991+0000 7f7d28c8c700 1 -- 172.21.15.16:0/1834805773 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d240c7660 con 0x7f7d2406a7e0 2023-10-16T12:23:44.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.991+0000 7f7d12ffd700 1 -- 172.21.15.16:0/1834805773 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d180038c0 con 0x7f7d2406a7e0 2023-10-16T12:23:44.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.991+0000 7f7d28c8c700 1 -- 172.21.15.16:0/1834805773 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d240b5c70 con 0x7f7d2406a7e0 2023-10-16T12:23:44.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.992+0000 7f7d28c8c700 1 -- 172.21.15.16:0/1834805773 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d2415a070 con 0x7f7d2406a7e0 2023-10-16T12:23:44.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.993+0000 7f7d12ffd700 1 -- 172.21.15.16:0/1834805773 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 14) v1 ==== 89400+0+0 (secure 0 0 0) 0x7f7d18003a20 con 0x7f7d2406a7e0 2023-10-16T12:23:44.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.993+0000 7f7d12ffd700 1 --2- 172.21.15.16:0/1834805773 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f7d0406b0b0 0x7f7d0406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:44.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.993+0000 7f7d21d9b700 1 -- 172.21.15.16:0/1834805773 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f7d0406b0b0 msgr2=0x7f7d0406d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1707126831 2023-10-16T12:23:44.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.993+0000 7f7d12ffd700 1 -- 172.21.15.16:0/1834805773 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6980+0+0 (secure 0 0 0) 0x7f7d18089130 con 0x7f7d2406a7e0 2023-10-16T12:23:44.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.993+0000 7f7d21d9b700 1 --2- 172.21.15.16:0/1834805773 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f7d0406b0b0 0x7f7d0406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:23:44.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:43.998+0000 7f7d12ffd700 1 -- 172.21.15.16:0/1834805773 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7d18053a60 con 0x7f7d2406a7e0 2023-10-16T12:23:44.121 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:23:44.119+0000 7fb59d613200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T12:23:44.152 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:23:44.150+0000 7f98e4adb200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T12:23:44.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.194+0000 7f7d21d9b700 1 -- 172.21.15.16:0/1834805773 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f7d0406b0b0 msgr2=0x7f7d0406d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1707126831 2023-10-16T12:23:44.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.194+0000 7f7d21d9b700 1 --2- 172.21.15.16:0/1834805773 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f7d0406b0b0 0x7f7d0406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:23:44.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.222+0000 7f7d28c8c700 1 -- 172.21.15.16:0/1834805773 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7d2404ebb0 con 0x7f7d2406a7e0 2023-10-16T12:23:44.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.222+0000 7f7d12ffd700 1 -- 172.21.15.16:0/1834805773 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v53) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7d1800ce60 con 0x7f7d2406a7e0 2023-10-16T12:23:44.225 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T12:23:44.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.225+0000 7f7d28c8c700 1 -- 172.21.15.16:0/1834805773 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f7d0406b0b0 msgr2=0x7f7d0406d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:23:44.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.225+0000 7f7d28c8c700 1 --2- 172.21.15.16:0/1834805773 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f7d0406b0b0 0x7f7d0406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:44.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.225+0000 7f7d28c8c700 1 -- 172.21.15.16:0/1834805773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d2406a7e0 msgr2=0x7f7d240b5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:44.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.225+0000 7f7d28c8c700 1 --2- 172.21.15.16:0/1834805773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d2406a7e0 0x7f7d240b5730 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f7d1800c450 tx=0x7f7d1800c760 comp rx=0 tx=0).stop 2023-10-16T12:23:44.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.225+0000 7f7d28c8c700 1 -- 172.21.15.16:0/1834805773 shutdown_connections 2023-10-16T12:23:44.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.225+0000 7f7d28c8c700 1 --2- 172.21.15.16:0/1834805773 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f7d0406b0b0 0x7f7d0406d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:44.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.225+0000 7f7d28c8c700 1 --2- 172.21.15.16:0/1834805773 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7d2415b2c0 0x7f7d241f69e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:44.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.225+0000 7f7d28c8c700 1 --2- 172.21.15.16:0/1834805773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d2406a7e0 0x7f7d240b5730 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:44.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.225+0000 7f7d28c8c700 1 --2- 172.21.15.16:0/1834805773 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7d2406e570 0x7f7d241f64a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:44.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.225+0000 7f7d28c8c700 1 -- 172.21.15.16:0/1834805773 >> 172.21.15.16:0/1834805773 conn(0x7f7d240b1f40 msgr2=0x7f7d240cc190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:44.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.225+0000 7f7d28c8c700 1 -- 172.21.15.16:0/1834805773 shutdown_connections 2023-10-16T12:23:44.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.225+0000 7f7d28c8c700 1 -- 172.21.15.16:0/1834805773 wait complete. 2023-10-16T12:23:44.241 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T12:23:44.245 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:23:44.243+0000 7fb59d613200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T12:23:44.278 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:23:44.276+0000 7f98e4adb200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T12:23:44.322 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:23:44.320+0000 7fb59d613200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T12:23:44.356 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:23:44.355+0000 7f98e4adb200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T12:23:44.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.373+0000 7fc1aa81d700 1 Processor -- start 2023-10-16T12:23:44.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.373+0000 7fc1aa81d700 1 -- start start 2023-10-16T12:23:44.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.374+0000 7fc1aa81d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc1a4159e40 0x7fc1a415d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:44.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.374+0000 7fc1aa81d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc1a40ae6a0 0x7fc1a40cc800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:44.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.374+0000 7fc1aa81d700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1a40b21c0 con 0x7fc1a40cf110 2023-10-16T12:23:44.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.374+0000 7fc1aa81d700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1a415a220 con 0x7fc1a40ae6a0 2023-10-16T12:23:44.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.374+0000 7fc1aa81d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a40ccd40 con 0x7fc1a4159e40 2023-10-16T12:23:44.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.375+0000 7fc1a37fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc1a40ae6a0 0x7fc1a40cc800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:44.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.375+0000 7fc1a37fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc1a40ae6a0 0x7fc1a40cc800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37034/0 (socket says 172.21.15.16:37034) 2023-10-16T12:23:44.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.375+0000 7fc1a37fe700 1 -- 172.21.15.16:0/4277862847 learned_addr learned my addr 172.21.15.16:0/4277862847 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:44.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.375+0000 7fc1a3fff700 1 --2- 172.21.15.16:0/4277862847 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc1a4159e40 0x7fc1a415d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:44.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.375+0000 7fc1a37fe700 1 -- 172.21.15.16:0/4277862847 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc1a4159e40 msgr2=0x7fc1a415d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:44.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.375+0000 7fc1a37fe700 1 --2- 172.21.15.16:0/4277862847 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc1a4159e40 0x7fc1a415d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:44.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.375+0000 7fc1a37fe700 1 -- 172.21.15.16:0/4277862847 >> v1:172.21.15.49:6789/0 conn(0x7fc1a40cf110 legacy=0x7fc1a40b2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:44.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.375+0000 7fc1a37fe700 1 -- 172.21.15.16:0/4277862847 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1a415df10 con 0x7fc1a40ae6a0 2023-10-16T12:23:44.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.375+0000 7fc1a37fe700 1 --2- 172.21.15.16:0/4277862847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1a40ae6a0 0x7fc1a40cc800 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fc19800bb90 tx=0x7fc19800bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e51b3f56ab4d4fa5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:44.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.375+0000 7fc1a2ffd700 1 -- 172.21.15.16:0/4277862847 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc19800dc20 con 0x7fc1a40ae6a0 2023-10-16T12:23:44.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.375+0000 7fc1a2ffd700 1 -- 172.21.15.16:0/4277862847 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc19800dd80 con 0x7fc1a40ae6a0 2023-10-16T12:23:44.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.375+0000 7fc1a2ffd700 1 -- 172.21.15.16:0/4277862847 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc198017630 con 0x7fc1a40ae6a0 2023-10-16T12:23:44.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.376+0000 7fc1aa81d700 1 -- 172.21.15.16:0/4277862847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1a40ae6a0 msgr2=0x7fc1a40cc800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:44.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.376+0000 7fc1aa81d700 1 --2- 172.21.15.16:0/4277862847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1a40ae6a0 0x7fc1a40cc800 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fc19800bb90 tx=0x7fc19800bea0 comp rx=0 tx=0).stop 2023-10-16T12:23:44.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.376+0000 7fc1aa81d700 1 -- 172.21.15.16:0/4277862847 shutdown_connections 2023-10-16T12:23:44.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.376+0000 7fc1aa81d700 1 --2- 172.21.15.16:0/4277862847 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc1a4159e40 0x7fc1a415d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:44.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.376+0000 7fc1aa81d700 1 --2- 172.21.15.16:0/4277862847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1a40ae6a0 0x7fc1a40cc800 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:44.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.376+0000 7fc1aa81d700 1 -- 172.21.15.16:0/4277862847 >> 172.21.15.16:0/4277862847 conn(0x7fc1a406a7e0 msgr2=0x7fc1a406abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:44.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.376+0000 7fc1aa81d700 1 -- 172.21.15.16:0/4277862847 shutdown_connections 2023-10-16T12:23:44.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.376+0000 7fc1aa81d700 1 -- 172.21.15.16:0/4277862847 wait complete. 2023-10-16T12:23:44.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.378+0000 7fc1aa81d700 1 Processor -- start 2023-10-16T12:23:44.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.378+0000 7fc1aa81d700 1 -- start start 2023-10-16T12:23:44.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.380+0000 7fc1aa81d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1a40ae6a0 0x7fc1a41501a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:44.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.380+0000 7fc1aa81d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc1a40cf110 0x7fc1a415b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:44.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.380+0000 7fc1aa81d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc1a4159e40 0x7fc1a40b3700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:44.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.380+0000 7fc1aa81d700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a41506e0 con 0x7fc1a40ae6a0 2023-10-16T12:23:44.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.380+0000 7fc1aa81d700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a40b0340 con 0x7fc1a4159e40 2023-10-16T12:23:44.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.380+0000 7fc1aa81d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a40b0620 con 0x7fc1a40cf110 2023-10-16T12:23:44.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.380+0000 7fc1a3fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1a40ae6a0 0x7fc1a41501a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:44.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.380+0000 7fc1a3fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1a40ae6a0 0x7fc1a41501a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37038/0 (socket says 172.21.15.16:37038) 2023-10-16T12:23:44.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.380+0000 7fc1a37fe700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc1a40cf110 0x7fc1a415b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:44.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.380+0000 7fc1a3fff700 1 -- 172.21.15.16:0/1699037185 learned_addr learned my addr 172.21.15.16:0/1699037185 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:44.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.380+0000 7fc1a8dba700 1 --2- 172.21.15.16:0/1699037185 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc1a4159e40 0x7fc1a40b3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:44.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.380+0000 7fc1a3fff700 1 -- 172.21.15.16:0/1699037185 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc1a40cf110 msgr2=0x7fc1a415b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:44.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.380+0000 7fc1a3fff700 1 --2- 172.21.15.16:0/1699037185 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc1a40cf110 0x7fc1a415b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:44.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.380+0000 7fc1a3fff700 1 -- 172.21.15.16:0/1699037185 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc1a4159e40 msgr2=0x7fc1a40b3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:44.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.380+0000 7fc1a3fff700 1 --2- 172.21.15.16:0/1699037185 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc1a4159e40 0x7fc1a40b3700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:44.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.380+0000 7fc1a3fff700 1 -- 172.21.15.16:0/1699037185 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc19800b7f0 con 0x7fc1a40ae6a0 2023-10-16T12:23:44.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.380+0000 7fc1a3fff700 1 --2- 172.21.15.16:0/1699037185 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1a40ae6a0 0x7fc1a41501a0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fc19400e9b0 tx=0x7fc19400ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:44.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.380+0000 7fc1a0ff9700 1 -- 172.21.15.16:0/1699037185 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc194009710 con 0x7fc1a40ae6a0 2023-10-16T12:23:44.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.381+0000 7fc1a0ff9700 1 -- 172.21.15.16:0/1699037185 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc19400ee50 con 0x7fc1a40ae6a0 2023-10-16T12:23:44.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.381+0000 7fc1aa81d700 1 -- 172.21.15.16:0/1699037185 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1a40b0900 con 0x7fc1a40ae6a0 2023-10-16T12:23:44.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.381+0000 7fc1a0ff9700 1 -- 172.21.15.16:0/1699037185 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc19400ca50 con 0x7fc1a40ae6a0 2023-10-16T12:23:44.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.381+0000 7fc1aa81d700 1 -- 172.21.15.16:0/1699037185 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1a415b0f0 con 0x7fc1a40ae6a0 2023-10-16T12:23:44.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.381+0000 7fc1aa81d700 1 -- 172.21.15.16:0/1699037185 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1a415b740 con 0x7fc1a40ae6a0 2023-10-16T12:23:44.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.382+0000 7fc1a0ff9700 1 -- 172.21.15.16:0/1699037185 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 14) v1 ==== 89400+0+0 (secure 0 0 0) 0x7fc19400cbb0 con 0x7fc1a40ae6a0 2023-10-16T12:23:44.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.383+0000 7fc1a0ff9700 1 --2- 172.21.15.16:0/1699037185 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fc19006b0b0 0x7fc19006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:44.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.383+0000 7fc1a37fe700 1 -- 172.21.15.16:0/1699037185 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fc19006b0b0 msgr2=0x7fc19006d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1707126831 2023-10-16T12:23:44.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.383+0000 7fc1a0ff9700 1 -- 172.21.15.16:0/1699037185 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6980+0+0 (secure 0 0 0) 0x7fc194089dc0 con 0x7fc1a40ae6a0 2023-10-16T12:23:44.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.383+0000 7fc1a37fe700 1 --2- 172.21.15.16:0/1699037185 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fc19006b0b0 0x7fc19006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:23:44.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.387+0000 7fc1a0ff9700 1 -- 172.21.15.16:0/1699037185 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc1940546f0 con 0x7fc1a40ae6a0 2023-10-16T12:23:44.406 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:23:44.404+0000 7fb59d613200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T12:23:44.446 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:23:44.444+0000 7f98e4adb200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T12:23:44.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.584+0000 7fc1a37fe700 1 -- 172.21.15.16:0/1699037185 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fc19006b0b0 msgr2=0x7fc19006d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1707126831 2023-10-16T12:23:44.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.584+0000 7fc1a37fe700 1 --2- 172.21.15.16:0/1699037185 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fc19006b0b0 0x7fc19006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:23:44.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.593+0000 7fc1aa81d700 1 -- 172.21.15.16:0/1699037185 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc1a4071230 con 0x7fc1a40ae6a0 2023-10-16T12:23:44.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.593+0000 7fc1a0ff9700 1 -- 172.21.15.16:0/1699037185 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v53) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc194021020 con 0x7fc1a40ae6a0 2023-10-16T12:23:44.595 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-10-16T12:23:44.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.595+0000 7fc1aa81d700 1 -- 172.21.15.16:0/1699037185 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fc19006b0b0 msgr2=0x7fc19006d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:23:44.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.595+0000 7fc1aa81d700 1 --2- 172.21.15.16:0/1699037185 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fc19006b0b0 0x7fc19006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:44.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.595+0000 7fc1aa81d700 1 -- 172.21.15.16:0/1699037185 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1a40ae6a0 msgr2=0x7fc1a41501a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:44.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.595+0000 7fc1aa81d700 1 --2- 172.21.15.16:0/1699037185 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1a40ae6a0 0x7fc1a41501a0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fc19400e9b0 tx=0x7fc19400ecc0 comp rx=0 tx=0).stop 2023-10-16T12:23:44.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.595+0000 7fc1aa81d700 1 -- 172.21.15.16:0/1699037185 shutdown_connections 2023-10-16T12:23:44.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.595+0000 7fc1aa81d700 1 --2- 172.21.15.16:0/1699037185 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7fc19006b0b0 0x7fc19006d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:44.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.596+0000 7fc1aa81d700 1 --2- 172.21.15.16:0/1699037185 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc1a4159e40 0x7fc1a40b3700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:44.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.596+0000 7fc1aa81d700 1 --2- 172.21.15.16:0/1699037185 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1a40ae6a0 0x7fc1a41501a0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:44.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.596+0000 7fc1aa81d700 1 --2- 172.21.15.16:0/1699037185 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc1a40cf110 0x7fc1a415b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:44.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.596+0000 7fc1aa81d700 1 -- 172.21.15.16:0/1699037185 >> 172.21.15.16:0/1699037185 conn(0x7fc1a406a7e0 msgr2=0x7fc1a4157130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:44.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.596+0000 7fc1aa81d700 1 -- 172.21.15.16:0/1699037185 shutdown_connections 2023-10-16T12:23:44.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.596+0000 7fc1aa81d700 1 -- 172.21.15.16:0/1699037185 wait complete. 2023-10-16T12:23:44.610 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T12:23:44.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.778+0000 7f4ab57ca700 1 Processor -- start 2023-10-16T12:23:44.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.778+0000 7f4ab57ca700 1 -- start start 2023-10-16T12:23:44.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.779+0000 7f4ab57ca700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f4ab00ce5d0 0x7f4ab00cc190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:44.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.779+0000 7f4ab57ca700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4ab015f060 0x7f4ab00cc6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:44.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.779+0000 7f4ab57ca700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ab00b2440 con 0x7f4ab015ec80 2023-10-16T12:23:44.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.779+0000 7f4ab57ca700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ab00b2c50 con 0x7f4ab00ce5d0 2023-10-16T12:23:44.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.779+0000 7f4ab57ca700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ab00d00e0 con 0x7f4ab015f060 2023-10-16T12:23:44.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.779+0000 7f4aae7fc700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f4ab00ce5d0 0x7f4ab00cc190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:44.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.779+0000 7f4aaeffd700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7f4ab015ec80 0x7f4ab00cdec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:33518/0 (socket says 172.21.15.16:33518) 2023-10-16T12:23:44.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.779+0000 7f4aaeffd700 1 -- 172.21.15.16:0/142886835 learned_addr learned my addr 172.21.15.16:0/142886835 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:23:44.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.779+0000 7f4aaf7fe700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4ab015f060 0x7f4ab00cc6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:44.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.779+0000 7f4aadffb700 1 -- 172.21.15.16:0/142886835 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 627462869 0 0) 0x7f4ab00b2440 con 0x7f4ab015ec80 2023-10-16T12:23:44.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.779+0000 7f4aadffb700 1 -- 172.21.15.16:0/142886835 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a980035b0 con 0x7f4ab015ec80 2023-10-16T12:23:44.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.779+0000 7f4aaf7fe700 1 -- 172.21.15.16:0/142886835 >> v2:172.21.15.49:3300/0 conn(0x7f4ab00ce5d0 msgr2=0x7f4ab00cc190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:44.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.779+0000 7f4aaf7fe700 1 --2- 172.21.15.16:0/142886835 >> v2:172.21.15.49:3300/0 conn(0x7f4ab00ce5d0 0x7f4ab00cc190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:44.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.779+0000 7f4aaf7fe700 1 -- 172.21.15.16:0/142886835 >> v1:172.21.15.49:6789/0 conn(0x7f4ab015ec80 legacy=0x7f4ab00cdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:44.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.780+0000 7f4aaf7fe700 1 -- 172.21.15.16:0/142886835 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ab00cef60 con 0x7f4ab015f060 2023-10-16T12:23:44.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.780+0000 7f4aae7fc700 1 --2- 172.21.15.16:0/142886835 >> v2:172.21.15.49:3300/0 conn(0x7f4ab00ce5d0 0x7f4ab00cc190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:23:44.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.780+0000 7f4aaf7fe700 1 --2- 172.21.15.16:0/142886835 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4ab015f060 0x7f4ab00cc6d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4aa8009d90 tx=0x7f4aa800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fbe94a9b32a0f7fc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:44.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.780+0000 7f4aadffb700 1 -- 172.21.15.16:0/142886835 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa800e070 con 0x7f4ab015f060 2023-10-16T12:23:44.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.780+0000 7f4aadffb700 1 -- 172.21.15.16:0/142886835 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4aa800bb20 con 0x7f4ab015f060 2023-10-16T12:23:44.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.780+0000 7f4aadffb700 1 -- 172.21.15.16:0/142886835 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa8012430 con 0x7f4ab015f060 2023-10-16T12:23:44.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.780+0000 7f4ab57ca700 1 -- 172.21.15.16:0/142886835 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4ab015f060 msgr2=0x7f4ab00cc6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:44.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.780+0000 7f4ab57ca700 1 --2- 172.21.15.16:0/142886835 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4ab015f060 0x7f4ab00cc6d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4aa8009d90 tx=0x7f4aa800b330 comp rx=0 tx=0).stop 2023-10-16T12:23:44.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.781+0000 7f4ab57ca700 1 -- 172.21.15.16:0/142886835 shutdown_connections 2023-10-16T12:23:44.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.781+0000 7f4ab57ca700 1 --2- 172.21.15.16:0/142886835 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4ab015f060 0x7f4ab00cc6d0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:44.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.781+0000 7f4ab57ca700 1 --2- 172.21.15.16:0/142886835 >> v2:172.21.15.49:3300/0 conn(0x7f4ab00ce5d0 0x7f4ab00cc190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:44.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.781+0000 7f4ab57ca700 1 -- 172.21.15.16:0/142886835 >> 172.21.15.16:0/142886835 conn(0x7f4ab00d15f0 msgr2=0x7f4ab00d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:44.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.781+0000 7f4ab57ca700 1 -- 172.21.15.16:0/142886835 shutdown_connections 2023-10-16T12:23:44.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.781+0000 7f4ab57ca700 1 -- 172.21.15.16:0/142886835 wait complete. 2023-10-16T12:23:44.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.782+0000 7f4ab57ca700 1 Processor -- start 2023-10-16T12:23:44.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.782+0000 7f4ab57ca700 1 -- start start 2023-10-16T12:23:44.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.783+0000 7f4ab57ca700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4ab00ce5d0 0x7f4ab01527e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:44.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.783+0000 7f4ab57ca700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4ab015ec80 0x7f4ab00c78e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:44.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.783+0000 7f4ab57ca700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ab015f060 0x7f4ab006f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:44.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.783+0000 7f4ab57ca700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ab00cae70 con 0x7f4ab015f060 2023-10-16T12:23:44.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.783+0000 7f4ab57ca700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ab00ca260 con 0x7f4ab00ce5d0 2023-10-16T12:23:44.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.783+0000 7f4ab57ca700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ab00ca540 con 0x7f4ab015ec80 2023-10-16T12:23:44.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.783+0000 7f4aaf7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ab015f060 0x7f4ab006f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:44.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.783+0000 7f4aae7fc700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4ab015ec80 0x7f4ab00c78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:44.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.783+0000 7f4aaeffd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4ab00ce5d0 0x7f4ab01527e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:44.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.783+0000 7f4aaf7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ab015f060 0x7f4ab006f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37072/0 (socket says 172.21.15.16:37072) 2023-10-16T12:23:44.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.783+0000 7f4aaf7fe700 1 -- 172.21.15.16:0/3047556412 learned_addr learned my addr 172.21.15.16:0/3047556412 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:44.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.783+0000 7f4aaf7fe700 1 -- 172.21.15.16:0/3047556412 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4ab015ec80 msgr2=0x7f4ab00c78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:44.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.783+0000 7f4aaf7fe700 1 --2- 172.21.15.16:0/3047556412 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4ab015ec80 0x7f4ab00c78e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:44.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.783+0000 7f4aaf7fe700 1 -- 172.21.15.16:0/3047556412 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4ab00ce5d0 msgr2=0x7f4ab01527e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:44.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.783+0000 7f4aaf7fe700 1 --2- 172.21.15.16:0/3047556412 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4ab00ce5d0 0x7f4ab01527e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:44.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.783+0000 7f4aaf7fe700 1 -- 172.21.15.16:0/3047556412 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aa40094a0 con 0x7f4ab015f060 2023-10-16T12:23:44.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.783+0000 7f4aaf7fe700 1 --2- 172.21.15.16:0/3047556412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ab015f060 0x7f4ab006f020 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f4aa8014040 tx=0x7f4aa8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:44.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.784+0000 7f4a977fe700 1 -- 172.21.15.16:0/3047556412 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa80129f0 con 0x7f4ab015f060 2023-10-16T12:23:44.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.784+0000 7f4a977fe700 1 -- 172.21.15.16:0/3047556412 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4aa8020070 con 0x7f4ab015f060 2023-10-16T12:23:44.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.784+0000 7f4a977fe700 1 -- 172.21.15.16:0/3047556412 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa8012dd0 con 0x7f4ab015f060 2023-10-16T12:23:44.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.784+0000 7f4ab57ca700 1 -- 172.21.15.16:0/3047556412 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4aa8009a70 con 0x7f4ab015f060 2023-10-16T12:23:44.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.784+0000 7f4ab57ca700 1 -- 172.21.15.16:0/3047556412 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ab00cab00 con 0x7f4ab015f060 2023-10-16T12:23:44.795 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:23:44.789+0000 7fb59d613200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T12:23:44.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.784+0000 7f4ab57ca700 1 -- 172.21.15.16:0/3047556412 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ab00cafe0 con 0x7f4ab015f060 2023-10-16T12:23:44.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.785+0000 7f4a977fe700 1 -- 172.21.15.16:0/3047556412 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 14) v1 ==== 89400+0+0 (secure 0 0 0) 0x7f4aa8004040 con 0x7f4ab015f060 2023-10-16T12:23:44.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.785+0000 7f4a977fe700 1 --2- 172.21.15.16:0/3047556412 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f4a9c06b0b0 0x7f4a9c06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:44.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.786+0000 7f4aaeffd700 1 -- 172.21.15.16:0/3047556412 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f4a9c06b0b0 msgr2=0x7f4a9c06d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1707126831 2023-10-16T12:23:44.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.786+0000 7f4a977fe700 1 -- 172.21.15.16:0/3047556412 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6980+0+0 (secure 0 0 0) 0x7f4aa8022070 con 0x7f4ab015f060 2023-10-16T12:23:44.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.786+0000 7f4aaeffd700 1 --2- 172.21.15.16:0/3047556412 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f4a9c06b0b0 0x7f4a9c06d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:23:44.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.790+0000 7f4a977fe700 1 -- 172.21.15.16:0/3047556412 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4aa8056720 con 0x7f4ab015f060 2023-10-16T12:23:44.840 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:23:44.838+0000 7f98e4adb200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T12:23:44.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.987+0000 7f4aaeffd700 1 -- 172.21.15.16:0/3047556412 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f4a9c06b0b0 msgr2=0x7f4a9c06d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1707126831 2023-10-16T12:23:44.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.987+0000 7f4aaeffd700 1 --2- 172.21.15.16:0/3047556412 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f4a9c06b0b0 0x7f4a9c06d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:23:44.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.993+0000 7f4ab57ca700 1 -- 172.21.15.16:0/3047556412 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f4ab004ebb0 con 0x7f4ab015f060 2023-10-16T12:23:44.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.994+0000 7f4a977fe700 1 -- 172.21.15.16:0/3047556412 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v53) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4aa802c0a0 con 0x7f4ab015f060 2023-10-16T12:23:44.996 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-10-16T12:23:44.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.996+0000 7f4ab57ca700 1 -- 172.21.15.16:0/3047556412 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f4a9c06b0b0 msgr2=0x7f4a9c06d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:23:44.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.996+0000 7f4ab57ca700 1 --2- 172.21.15.16:0/3047556412 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f4a9c06b0b0 0x7f4a9c06d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:44.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.996+0000 7f4ab57ca700 1 -- 172.21.15.16:0/3047556412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ab015f060 msgr2=0x7f4ab006f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:44.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.996+0000 7f4ab57ca700 1 --2- 172.21.15.16:0/3047556412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ab015f060 0x7f4ab006f020 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f4aa8014040 tx=0x7f4aa8004ad0 comp rx=0 tx=0).stop 2023-10-16T12:23:44.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.996+0000 7f4ab57ca700 1 -- 172.21.15.16:0/3047556412 shutdown_connections 2023-10-16T12:23:44.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.996+0000 7f4ab57ca700 1 --2- 172.21.15.16:0/3047556412 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f4a9c06b0b0 0x7f4a9c06d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:44.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.996+0000 7f4ab57ca700 1 --2- 172.21.15.16:0/3047556412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ab015f060 0x7f4ab006f020 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:45.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.996+0000 7f4ab57ca700 1 --2- 172.21.15.16:0/3047556412 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4ab00ce5d0 0x7f4ab01527e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:45.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.996+0000 7f4ab57ca700 1 --2- 172.21.15.16:0/3047556412 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4ab015ec80 0x7f4ab00c78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:45.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.996+0000 7f4ab57ca700 1 -- 172.21.15.16:0/3047556412 >> 172.21.15.16:0/3047556412 conn(0x7f4ab00d15f0 msgr2=0x7f4ab00cf1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:45.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.996+0000 7f4ab57ca700 1 -- 172.21.15.16:0/3047556412 shutdown_connections 2023-10-16T12:23:45.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:44.996+0000 7f4ab57ca700 1 -- 172.21.15.16:0/3047556412 wait complete. 2023-10-16T12:23:45.010 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T12:23:45.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.138+0000 7f054e8e4700 1 Processor -- start 2023-10-16T12:23:45.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.138+0000 7f054e8e4700 1 -- start start 2023-10-16T12:23:45.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.139+0000 7f054e8e4700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f05480cc8a0 0x7f054815e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:45.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.139+0000 7f054e8e4700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f054806a7e0 0x7f054806ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:45.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.139+0000 7f054e8e4700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f054806b150 con 0x7f054815b280 2023-10-16T12:23:45.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.139+0000 7f054e8e4700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0548161c30 con 0x7f054806a7e0 2023-10-16T12:23:45.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.139+0000 7f054e8e4700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0548161f10 con 0x7f05480cc8a0 2023-10-16T12:23:45.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.139+0000 7f054ce81700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f054806a7e0 0x7f054806ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:45.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.139+0000 7f0547fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f054815b280 0x7f05481613e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54584/0 (socket says 172.21.15.16:54584) 2023-10-16T12:23:45.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.139+0000 7f0547fff700 1 -- 172.21.15.16:0/3449987322 learned_addr learned my addr 172.21.15.16:0/3449987322 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:23:45.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.139+0000 7f05477fe700 1 --2- 172.21.15.16:0/3449987322 >> v2:172.21.15.49:3300/0 conn(0x7f05480cc8a0 0x7f054815e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:45.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.140+0000 7f054ce81700 1 -- 172.21.15.16:0/3449987322 >> v2:172.21.15.49:3300/0 conn(0x7f05480cc8a0 msgr2=0x7f054815e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:45.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.140+0000 7f054ce81700 1 --2- 172.21.15.16:0/3449987322 >> v2:172.21.15.49:3300/0 conn(0x7f05480cc8a0 0x7f054815e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:45.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.140+0000 7f054ce81700 1 -- 172.21.15.16:0/3449987322 >> v1:172.21.15.16:6789/0 conn(0x7f054815b280 legacy=0x7f05481613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:45.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.140+0000 7f0546ffd700 1 -- 172.21.15.16:0/3449987322 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 335976844 0 0) 0x7f054806b150 con 0x7f054815b280 2023-10-16T12:23:45.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.140+0000 7f054ce81700 1 -- 172.21.15.16:0/3449987322 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05481621f0 con 0x7f054806a7e0 2023-10-16T12:23:45.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.140+0000 7f054ce81700 1 --2- 172.21.15.16:0/3449987322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f054806a7e0 0x7f054806ac10 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f0548071970 tx=0x7f054000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54259f6993746de8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:45.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.140+0000 7f0546ffd700 1 -- 172.21.15.16:0/3449987322 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f054000e070 con 0x7f054806a7e0 2023-10-16T12:23:45.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.140+0000 7f0546ffd700 1 -- 172.21.15.16:0/3449987322 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f054000bb20 con 0x7f054806a7e0 2023-10-16T12:23:45.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.140+0000 7f0546ffd700 1 -- 172.21.15.16:0/3449987322 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0540012430 con 0x7f054806a7e0 2023-10-16T12:23:45.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.141+0000 7f054e8e4700 1 -- 172.21.15.16:0/3449987322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f054806a7e0 msgr2=0x7f054806ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:45.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.141+0000 7f054e8e4700 1 --2- 172.21.15.16:0/3449987322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f054806a7e0 0x7f054806ac10 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f0548071970 tx=0x7f054000b330 comp rx=0 tx=0).stop 2023-10-16T12:23:45.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.141+0000 7f054e8e4700 1 -- 172.21.15.16:0/3449987322 shutdown_connections 2023-10-16T12:23:45.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.141+0000 7f054e8e4700 1 --2- 172.21.15.16:0/3449987322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f054806a7e0 0x7f054806ac10 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:45.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.141+0000 7f054e8e4700 1 --2- 172.21.15.16:0/3449987322 >> v2:172.21.15.49:3300/0 conn(0x7f05480cc8a0 0x7f054815e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:45.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.141+0000 7f054e8e4700 1 -- 172.21.15.16:0/3449987322 >> 172.21.15.16:0/3449987322 conn(0x7f05480b1fd0 msgr2=0x7f05480b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:45.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.141+0000 7f054e8e4700 1 -- 172.21.15.16:0/3449987322 shutdown_connections 2023-10-16T12:23:45.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.141+0000 7f054e8e4700 1 -- 172.21.15.16:0/3449987322 wait complete. 2023-10-16T12:23:45.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.142+0000 7f054e8e4700 1 Processor -- start 2023-10-16T12:23:45.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.142+0000 7f054e8e4700 1 -- start start 2023-10-16T12:23:45.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.143+0000 7f054e8e4700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f054806a7e0 0x7f05481562c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:45.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.143+0000 7f054e8e4700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f05480cc8a0 0x7f05480c3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:45.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.143+0000 7f054e8e4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f054815b280 0x7f05480c44b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:45.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.143+0000 7f054e8e4700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0548157a70 con 0x7f054815b280 2023-10-16T12:23:45.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.143+0000 7f054e8e4700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f054815cfe0 con 0x7f054806a7e0 2023-10-16T12:23:45.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.143+0000 7f054e8e4700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f054815cd00 con 0x7f05480cc8a0 2023-10-16T12:23:45.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.143+0000 7f054ce81700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f054815b280 0x7f05480c44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:45.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.143+0000 7f054ce81700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f054815b280 0x7f05480c44b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37088/0 (socket says 172.21.15.16:37088) 2023-10-16T12:23:45.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.143+0000 7f054ce81700 1 -- 172.21.15.16:0/3595982879 learned_addr learned my addr 172.21.15.16:0/3595982879 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:45.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.143+0000 7f0547fff700 1 --2- 172.21.15.16:0/3595982879 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f054806a7e0 0x7f05481562c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:45.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.143+0000 7f05477fe700 1 --2- 172.21.15.16:0/3595982879 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f05480cc8a0 0x7f05480c3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:45.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.143+0000 7f054ce81700 1 -- 172.21.15.16:0/3595982879 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f05480cc8a0 msgr2=0x7f05480c3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:45.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.143+0000 7f054ce81700 1 --2- 172.21.15.16:0/3595982879 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f05480cc8a0 0x7f05480c3f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:45.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.143+0000 7f054ce81700 1 -- 172.21.15.16:0/3595982879 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f054806a7e0 msgr2=0x7f05481562c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:45.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.143+0000 7f054ce81700 1 --2- 172.21.15.16:0/3595982879 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f054806a7e0 0x7f05481562c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:45.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.143+0000 7f054ce81700 1 -- 172.21.15.16:0/3595982879 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0540009a70 con 0x7f054815b280 2023-10-16T12:23:45.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.144+0000 7f054ce81700 1 --2- 172.21.15.16:0/3595982879 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f054815b280 0x7f05480c44b0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f0540014040 tx=0x7f0540004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:45.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.144+0000 7f0544ff9700 1 -- 172.21.15.16:0/3595982879 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0540012430 con 0x7f054815b280 2023-10-16T12:23:45.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.144+0000 7f0544ff9700 1 -- 172.21.15.16:0/3595982879 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0540020070 con 0x7f054815b280 2023-10-16T12:23:45.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.144+0000 7f054e8e4700 1 -- 172.21.15.16:0/3595982879 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0548157420 con 0x7f054815b280 2023-10-16T12:23:45.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.144+0000 7f0544ff9700 1 -- 172.21.15.16:0/3595982879 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0540004280 con 0x7f054815b280 2023-10-16T12:23:45.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.144+0000 7f054e8e4700 1 -- 172.21.15.16:0/3595982879 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0548156800 con 0x7f054815b280 2023-10-16T12:23:45.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.145+0000 7f054e8e4700 1 -- 172.21.15.16:0/3595982879 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0548157c20 con 0x7f054815b280 2023-10-16T12:23:45.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.145+0000 7f0544ff9700 1 -- 172.21.15.16:0/3595982879 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 14) v1 ==== 89400+0+0 (secure 0 0 0) 0x7f05400043e0 con 0x7f054815b280 2023-10-16T12:23:45.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.146+0000 7f0544ff9700 1 --2- 172.21.15.16:0/3595982879 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f053406b0b0 0x7f053406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:45.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.146+0000 7f0547fff700 1 -- 172.21.15.16:0/3595982879 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f053406b0b0 msgr2=0x7f053406d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1707126831 2023-10-16T12:23:45.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.146+0000 7f0544ff9700 1 -- 172.21.15.16:0/3595982879 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6980+0+0 (secure 0 0 0) 0x7f0540022070 con 0x7f054815b280 2023-10-16T12:23:45.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.146+0000 7f0547fff700 1 --2- 172.21.15.16:0/3595982879 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f053406b0b0 0x7f053406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:23:45.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.154+0000 7f0544ff9700 1 -- 172.21.15.16:0/3595982879 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0548157c20 con 0x7f054815b280 2023-10-16T12:23:45.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.347+0000 7f0547fff700 1 -- 172.21.15.16:0/3595982879 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f053406b0b0 msgr2=0x7f053406d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1707126831 2023-10-16T12:23:45.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.347+0000 7f0547fff700 1 --2- 172.21.15.16:0/3595982879 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f053406b0b0 0x7f053406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:23:45.397 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:23:45.395+0000 7fb59d613200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T12:23:45.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.422+0000 7f054e8e4700 1 -- 172.21.15.16:0/3595982879 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f0548156990 con 0x7f054815b280 2023-10-16T12:23:45.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.422+0000 7f0544ff9700 1 -- 172.21.15.16:0/3595982879 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 6 entries v53) v1 ==== 80+0+351 (secure 0 0 0) 0x7f05400566c0 con 0x7f054815b280 2023-10-16T12:23:45.424 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/1300148187 2023-10-17T12:12:15.525647+0000 2023-10-16T12:23:45.424 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/675356848 2023-10-17T12:12:15.525647+0000 2023-10-16T12:23:45.425 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6832/365921436 2023-10-17T12:12:15.525647+0000 2023-10-16T12:23:45.425 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3547353312 2023-10-17T12:12:15.525647+0000 2023-10-16T12:23:45.425 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/1893346008 2023-10-17T12:12:15.525647+0000 2023-10-16T12:23:45.425 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6833/365921436 2023-10-17T12:12:15.525647+0000 2023-10-16T12:23:45.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.424+0000 7f054e8e4700 1 -- 172.21.15.16:0/3595982879 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f053406b0b0 msgr2=0x7f053406d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:23:45.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.424+0000 7f054e8e4700 1 --2- 172.21.15.16:0/3595982879 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f053406b0b0 0x7f053406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:45.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.424+0000 7f054e8e4700 1 -- 172.21.15.16:0/3595982879 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f054815b280 msgr2=0x7f05480c44b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:45.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.424+0000 7f054e8e4700 1 --2- 172.21.15.16:0/3595982879 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f054815b280 0x7f05480c44b0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f0540014040 tx=0x7f0540004ad0 comp rx=0 tx=0).stop 2023-10-16T12:23:45.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.424+0000 7f054e8e4700 1 -- 172.21.15.16:0/3595982879 shutdown_connections 2023-10-16T12:23:45.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.424+0000 7f054e8e4700 1 --2- 172.21.15.16:0/3595982879 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f053406b0b0 0x7f053406d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:45.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.424+0000 7f054e8e4700 1 --2- 172.21.15.16:0/3595982879 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f054815b280 0x7f05480c44b0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:45.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.424+0000 7f054e8e4700 1 --2- 172.21.15.16:0/3595982879 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f054806a7e0 0x7f05481562c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:45.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.424+0000 7f054e8e4700 1 --2- 172.21.15.16:0/3595982879 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f05480cc8a0 0x7f05480c3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:45.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.424+0000 7f054e8e4700 1 -- 172.21.15.16:0/3595982879 >> 172.21.15.16:0/3595982879 conn(0x7f05480b1fd0 msgr2=0x7f054815a4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:45.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.424+0000 7f054e8e4700 1 -- 172.21.15.16:0/3595982879 shutdown_connections 2023-10-16T12:23:45.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.424+0000 7f054e8e4700 1 -- 172.21.15.16:0/3595982879 wait complete. 2023-10-16T12:23:45.429 INFO:teuthology.orchestra.run.smithi016.stderr:listed 6 entries 2023-10-16T12:23:45.440 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-16T12:23:45.440 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:45.440 DEBUG:teuthology.orchestra.run.smithi016:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-16T12:23:45.469 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:23:45.467+0000 7fb59d613200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T12:23:45.483 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:23:45.482+0000 7f98e4adb200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T12:23:45.487 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2023-10-16T12:23:45.488 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:23:45.488 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-16T12:23:45.545 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T12:23:45.546 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:23:45.559 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:23:45.557+0000 7f98e4adb200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T12:23:45.564 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284204k,nr_inodes=4071051,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=753 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260540k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T12:23:45.564 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:23:45.564 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:23:45.582 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16287364k,nr_inodes=4071841,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20251 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3261172k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T12:23:45.582 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T12:23:45.583 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:23:45.621 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284204k,nr_inodes=4071051,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=753 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260540k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T12:23:45.622 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:23:45.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.746+0000 7f6e28f6b700 1 Processor -- start 2023-10-16T12:23:45.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.746+0000 7f6e28f6b700 1 -- start start 2023-10-16T12:23:45.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.747+0000 7f6e28f6b700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f6e240ce4f0 0x7f6e240d0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:45.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.747+0000 7f6e28f6b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6e2415ba10 0x7f6e240cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:45.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.747+0000 7f6e28f6b700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e240b0970 con 0x7f6e2415b630 2023-10-16T12:23:45.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.747+0000 7f6e28f6b700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e240cd690 con 0x7f6e240ce4f0 2023-10-16T12:23:45.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.747+0000 7f6e28f6b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e240cd970 con 0x7f6e2415ba10 2023-10-16T12:23:45.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.747+0000 7f6e22d9d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6e2415ba10 0x7f6e240cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:45.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.747+0000 7f6e21d9b700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f6e240ce4f0 0x7f6e240d0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:45.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.747+0000 7f6e21d9b700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f6e240ce4f0 0x7f6e240d0380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:36158/0 (socket says 172.21.15.16:36158) 2023-10-16T12:23:45.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.747+0000 7f6e2259c700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7f6e2415b630 0x7f6e240cdde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:33550/0 (socket says 172.21.15.16:33550) 2023-10-16T12:23:45.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.747+0000 7f6e22d9d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6e2415ba10 0x7f6e240cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:53588/0 (socket says 172.21.15.16:53588) 2023-10-16T12:23:45.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.747+0000 7f6e21d9b700 1 -- 172.21.15.16:0/2690253133 learned_addr learned my addr 172.21.15.16:0/2690253133 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:45.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.748+0000 7f6e2159a700 1 -- 172.21.15.16:0/2690253133 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1797692261 0 0) 0x7f6e240b0970 con 0x7f6e2415b630 2023-10-16T12:23:45.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.748+0000 7f6e22d9d700 1 -- 172.21.15.16:0/2690253133 >> v2:172.21.15.49:3300/0 conn(0x7f6e240ce4f0 msgr2=0x7f6e240d0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:45.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.748+0000 7f6e22d9d700 1 --2- 172.21.15.16:0/2690253133 >> v2:172.21.15.49:3300/0 conn(0x7f6e240ce4f0 0x7f6e240d0380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:45.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.748+0000 7f6e22d9d700 1 -- 172.21.15.16:0/2690253133 >> v1:172.21.15.49:6789/0 conn(0x7f6e2415b630 legacy=0x7f6e240cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:45.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.748+0000 7f6e22d9d700 1 -- 172.21.15.16:0/2690253133 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e240b0c30 con 0x7f6e2415ba10 2023-10-16T12:23:45.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.748+0000 7f6e21d9b700 1 --2- 172.21.15.16:0/2690253133 >> v2:172.21.15.49:3300/0 conn(0x7f6e240ce4f0 0x7f6e240d0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:23:45.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.748+0000 7f6e22d9d700 1 --2- 172.21.15.16:0/2690253133 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6e2415ba10 0x7f6e240cf290 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6e1c009d90 tx=0x7f6e1c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=526c9846e551b4ab server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:45.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.748+0000 7f6e2159a700 1 -- 172.21.15.16:0/2690253133 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e1c00e070 con 0x7f6e2415ba10 2023-10-16T12:23:45.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.748+0000 7f6e2159a700 1 -- 172.21.15.16:0/2690253133 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e1c00bb20 con 0x7f6e2415ba10 2023-10-16T12:23:45.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.748+0000 7f6e2159a700 1 -- 172.21.15.16:0/2690253133 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e1c012430 con 0x7f6e2415ba10 2023-10-16T12:23:45.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.749+0000 7f6e28f6b700 1 -- 172.21.15.16:0/2690253133 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6e2415ba10 msgr2=0x7f6e240cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:45.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.749+0000 7f6e28f6b700 1 --2- 172.21.15.16:0/2690253133 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6e2415ba10 0x7f6e240cf290 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6e1c009d90 tx=0x7f6e1c00b330 comp rx=0 tx=0).stop 2023-10-16T12:23:45.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.749+0000 7f6e28f6b700 1 -- 172.21.15.16:0/2690253133 shutdown_connections 2023-10-16T12:23:45.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.749+0000 7f6e28f6b700 1 --2- 172.21.15.16:0/2690253133 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6e2415ba10 0x7f6e240cf290 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:45.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.749+0000 7f6e28f6b700 1 --2- 172.21.15.16:0/2690253133 >> v2:172.21.15.49:3300/0 conn(0x7f6e240ce4f0 0x7f6e240d0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:45.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.749+0000 7f6e28f6b700 1 -- 172.21.15.16:0/2690253133 >> 172.21.15.16:0/2690253133 conn(0x7f6e240d14b0 msgr2=0x7f6e240d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:45.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.749+0000 7f6e28f6b700 1 -- 172.21.15.16:0/2690253133 shutdown_connections 2023-10-16T12:23:45.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.749+0000 7f6e28f6b700 1 -- 172.21.15.16:0/2690253133 wait complete. 2023-10-16T12:23:45.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.751+0000 7f6e28f6b700 1 Processor -- start 2023-10-16T12:23:45.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.751+0000 7f6e28f6b700 1 -- start start 2023-10-16T12:23:45.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.752+0000 7f6e28f6b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6e240ce4f0 0x7f6e24150a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:45.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.752+0000 7f6e28f6b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e2415b630 0x7f6e2415de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:45.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.752+0000 7f6e28f6b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6e2415ba10 0x7f6e240af1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:45.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.752+0000 7f6e28f6b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e2415e380 con 0x7f6e2415b630 2023-10-16T12:23:45.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.752+0000 7f6e28f6b700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e2415d080 con 0x7f6e2415ba10 2023-10-16T12:23:45.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.752+0000 7f6e28f6b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e2415d360 con 0x7f6e240ce4f0 2023-10-16T12:23:45.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.752+0000 7f6e21d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e2415b630 0x7f6e2415de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:45.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.752+0000 7f6e2259c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6e240ce4f0 0x7f6e24150a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:45.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.752+0000 7f6e21d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e2415b630 0x7f6e2415de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37102/0 (socket says 172.21.15.16:37102) 2023-10-16T12:23:45.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.752+0000 7f6e21d9b700 1 -- 172.21.15.16:0/1794119365 learned_addr learned my addr 172.21.15.16:0/1794119365 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:45.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.752+0000 7f6e22d9d700 1 --2- 172.21.15.16:0/1794119365 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6e2415ba10 0x7f6e240af1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:45.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.752+0000 7f6e22d9d700 1 -- 172.21.15.16:0/1794119365 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6e240ce4f0 msgr2=0x7f6e24150a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:45.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.752+0000 7f6e22d9d700 1 --2- 172.21.15.16:0/1794119365 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6e240ce4f0 0x7f6e24150a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:45.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.752+0000 7f6e22d9d700 1 -- 172.21.15.16:0/1794119365 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e2415b630 msgr2=0x7f6e2415de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:45.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.752+0000 7f6e22d9d700 1 --2- 172.21.15.16:0/1794119365 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e2415b630 0x7f6e2415de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:45.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.752+0000 7f6e22d9d700 1 -- 172.21.15.16:0/1794119365 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e18009550 con 0x7f6e2415ba10 2023-10-16T12:23:45.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.752+0000 7f6e2259c700 1 --2- 172.21.15.16:0/1794119365 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6e240ce4f0 0x7f6e24150a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:23:45.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.752+0000 7f6e22d9d700 1 --2- 172.21.15.16:0/1794119365 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6e2415ba10 0x7f6e240af1a0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f6e1c016000 tx=0x7f6e1c004ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:45.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.753+0000 7f6e12ffd700 1 -- 172.21.15.16:0/1794119365 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e1c012430 con 0x7f6e2415ba10 2023-10-16T12:23:45.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.753+0000 7f6e12ffd700 1 -- 172.21.15.16:0/1794119365 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e1c020070 con 0x7f6e2415ba10 2023-10-16T12:23:45.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.753+0000 7f6e12ffd700 1 -- 172.21.15.16:0/1794119365 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e1c004280 con 0x7f6e2415ba10 2023-10-16T12:23:45.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.753+0000 7f6e28f6b700 1 -- 172.21.15.16:0/1794119365 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e1c009a70 con 0x7f6e2415ba10 2023-10-16T12:23:45.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.753+0000 7f6e28f6b700 1 -- 172.21.15.16:0/1794119365 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e2415d920 con 0x7f6e2415ba10 2023-10-16T12:23:45.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.754+0000 7f6e28f6b700 1 -- 172.21.15.16:0/1794119365 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e2404ebb0 con 0x7f6e2415ba10 2023-10-16T12:23:45.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.754+0000 7f6e12ffd700 1 -- 172.21.15.16:0/1794119365 <== mon.1 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 14) v1 ==== 89400+0+0 (secure 0 0 0) 0x7f6e1c0043e0 con 0x7f6e2415ba10 2023-10-16T12:23:45.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.755+0000 7f6e12ffd700 1 --2- 172.21.15.16:0/1794119365 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f6e0406b0b0 0x7f6e0406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:45.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.755+0000 7f6e2259c700 1 -- 172.21.15.16:0/1794119365 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f6e0406b0b0 msgr2=0x7f6e0406d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1707126831 2023-10-16T12:23:45.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.755+0000 7f6e12ffd700 1 -- 172.21.15.16:0/1794119365 <== mon.1 v2:172.21.15.49:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6980+0+0 (secure 0 0 0) 0x7f6e1c022070 con 0x7f6e2415ba10 2023-10-16T12:23:45.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.755+0000 7f6e2259c700 1 --2- 172.21.15.16:0/1794119365 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f6e0406b0b0 0x7f6e0406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:23:45.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.759+0000 7f6e12ffd700 1 -- 172.21.15.16:0/1794119365 <== mon.1 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6e1c056330 con 0x7f6e2415ba10 2023-10-16T12:23:45.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.956+0000 7f6e2259c700 1 -- 172.21.15.16:0/1794119365 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f6e0406b0b0 msgr2=0x7f6e0406d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1707126831 2023-10-16T12:23:45.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:45.956+0000 7f6e2259c700 1 --2- 172.21.15.16:0/1794119365 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f6e0406b0b0 0x7f6e0406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:23:46.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.009+0000 7f6e28f6b700 1 -- 172.21.15.16:0/1794119365 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6e2415d1c0 con 0x7f6e2415ba10 2023-10-16T12:23:46.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.010+0000 7f6e12ffd700 1 -- 172.21.15.16:0/1794119365 <== mon.1 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+4910 (secure 0 0 0) 0x7f6e2415d1c0 con 0x7f6e2415ba10 2023-10-16T12:23:46.013 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:23:46.013 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":18,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5194,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/266676368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":266676368},{"type":"v1","addr":"172.21.15.49:6834","nonce":266676368}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5268,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/1460528697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":1460528697},{"type":"v1","addr":"172.21.15.16:6837","nonce":1460528697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/2374376752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2374376752},{"type":"v1","addr":"172.21.15.16:6836","nonce":2374376752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:12:00.537600+0000","modified":"2023-10-16T12:12:03.538619+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5283},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5283":{"gid":5283,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.49:6835/2788396302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":2788396302},{"type":"v1","addr":"172.21.15.49:6835","nonce":2788396302}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-16T12:23:46.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.012+0000 7f6e28f6b700 1 -- 172.21.15.16:0/1794119365 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f6e0406b0b0 msgr2=0x7f6e0406d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:23:46.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.012+0000 7f6e28f6b700 1 --2- 172.21.15.16:0/1794119365 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f6e0406b0b0 0x7f6e0406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:46.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.012+0000 7f6e28f6b700 1 -- 172.21.15.16:0/1794119365 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6e2415ba10 msgr2=0x7f6e240af1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:46.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.012+0000 7f6e28f6b700 1 --2- 172.21.15.16:0/1794119365 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6e2415ba10 0x7f6e240af1a0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f6e1c016000 tx=0x7f6e1c004ad0 comp rx=0 tx=0).stop 2023-10-16T12:23:46.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.012+0000 7f6e28f6b700 1 -- 172.21.15.16:0/1794119365 shutdown_connections 2023-10-16T12:23:46.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.012+0000 7f6e28f6b700 1 --2- 172.21.15.16:0/1794119365 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f6e0406b0b0 0x7f6e0406d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:46.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.012+0000 7f6e28f6b700 1 --2- 172.21.15.16:0/1794119365 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6e2415ba10 0x7f6e240af1a0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:46.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.012+0000 7f6e28f6b700 1 --2- 172.21.15.16:0/1794119365 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6e240ce4f0 0x7f6e24150a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:46.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.012+0000 7f6e28f6b700 1 --2- 172.21.15.16:0/1794119365 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e2415b630 0x7f6e2415de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:46.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.012+0000 7f6e28f6b700 1 -- 172.21.15.16:0/1794119365 >> 172.21.15.16:0/1794119365 conn(0x7f6e240d14b0 msgr2=0x7f6e240b1330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:46.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.012+0000 7f6e28f6b700 1 -- 172.21.15.16:0/1794119365 shutdown_connections 2023-10-16T12:23:46.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.012+0000 7f6e28f6b700 1 -- 172.21.15.16:0/1794119365 wait complete. 2023-10-16T12:23:46.020 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 18 2023-10-16T12:23:46.027 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:23:46.133 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:23:46.132+0000 7fb59d613200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:23:46.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.157+0000 7f2399154700 1 Processor -- start 2023-10-16T12:23:46.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.157+0000 7f2399154700 1 -- start start 2023-10-16T12:23:46.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.158+0000 7f2399154700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f23940ce4f0 0x7f23940d0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:46.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.158+0000 7f2399154700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f239415ba10 0x7f23940cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:46.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.158+0000 7f2399154700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23940b0970 con 0x7f239415b630 2023-10-16T12:23:46.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.158+0000 7f2399154700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23940cd690 con 0x7f239415ba10 2023-10-16T12:23:46.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.158+0000 7f2399154700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23940cd970 con 0x7f23940ce4f0 2023-10-16T12:23:46.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.158+0000 7f239359e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f239415ba10 0x7f23940cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:46.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.158+0000 7f239359e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f239415ba10 0x7f23940cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37114/0 (socket says 172.21.15.16:37114) 2023-10-16T12:23:46.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.158+0000 7f239359e700 1 -- 172.21.15.16:0/4042001287 learned_addr learned my addr 172.21.15.16:0/4042001287 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:46.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.158+0000 7f239259c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f23940ce4f0 0x7f23940d0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:46.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.158+0000 7f239359e700 1 -- 172.21.15.16:0/4042001287 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f23940ce4f0 msgr2=0x7f23940d0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:46.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.158+0000 7f239359e700 1 --2- 172.21.15.16:0/4042001287 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f23940ce4f0 0x7f23940d0380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:46.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.158+0000 7f239359e700 1 -- 172.21.15.16:0/4042001287 >> v1:172.21.15.49:6789/0 conn(0x7f239415b630 legacy=0x7f23940cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:46.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.158+0000 7f2391d9b700 1 -- 172.21.15.16:0/4042001287 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1057493568 0 0) 0x7f23940b0970 con 0x7f239415b630 2023-10-16T12:23:46.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.158+0000 7f239359e700 1 -- 172.21.15.16:0/4042001287 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23940b0c30 con 0x7f239415ba10 2023-10-16T12:23:46.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.158+0000 7f239359e700 1 --2- 172.21.15.16:0/4042001287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f239415ba10 0x7f23940cf290 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f238c009e50 tx=0x7f238c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ba826faca8f7ca4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:46.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.159+0000 7f2391d9b700 1 -- 172.21.15.16:0/4042001287 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f238c00e070 con 0x7f239415ba10 2023-10-16T12:23:46.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.159+0000 7f2391d9b700 1 -- 172.21.15.16:0/4042001287 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f238c00b910 con 0x7f239415ba10 2023-10-16T12:23:46.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.159+0000 7f2391d9b700 1 -- 172.21.15.16:0/4042001287 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f238c00bcc0 con 0x7f239415ba10 2023-10-16T12:23:46.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.159+0000 7f2399154700 1 -- 172.21.15.16:0/4042001287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f239415ba10 msgr2=0x7f23940cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:46.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.159+0000 7f2399154700 1 --2- 172.21.15.16:0/4042001287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f239415ba10 0x7f23940cf290 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f238c009e50 tx=0x7f238c00b330 comp rx=0 tx=0).stop 2023-10-16T12:23:46.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.159+0000 7f2399154700 1 -- 172.21.15.16:0/4042001287 shutdown_connections 2023-10-16T12:23:46.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.159+0000 7f2399154700 1 --2- 172.21.15.16:0/4042001287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f239415ba10 0x7f23940cf290 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:46.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.159+0000 7f2399154700 1 --2- 172.21.15.16:0/4042001287 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f23940ce4f0 0x7f23940d0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:46.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.159+0000 7f2399154700 1 -- 172.21.15.16:0/4042001287 >> 172.21.15.16:0/4042001287 conn(0x7f23940d14b0 msgr2=0x7f23940d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:46.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.159+0000 7f2399154700 1 -- 172.21.15.16:0/4042001287 shutdown_connections 2023-10-16T12:23:46.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.159+0000 7f2399154700 1 -- 172.21.15.16:0/4042001287 wait complete. 2023-10-16T12:23:46.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.161+0000 7f2399154700 1 Processor -- start 2023-10-16T12:23:46.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.161+0000 7f2399154700 1 -- start start 2023-10-16T12:23:46.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.161+0000 7f2399154700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f23940ce4f0 0x7f23941509b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:46.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.161+0000 7f2399154700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f239415b630 0x7f239415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:46.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.162+0000 7f2399154700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f239415ba10 0x7f23941564f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:46.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.162+0000 7f2399154700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f239406d930 con 0x7f239415b630 2023-10-16T12:23:46.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.162+0000 7f2399154700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2394155730 con 0x7f23940ce4f0 2023-10-16T12:23:46.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.162+0000 7f2399154700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2394155a10 con 0x7f239415ba10 2023-10-16T12:23:46.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.162+0000 7f239259c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f239415b630 0x7f239415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:46.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.162+0000 7f239259c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f239415b630 0x7f239415bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37118/0 (socket says 172.21.15.16:37118) 2023-10-16T12:23:46.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.162+0000 7f239259c700 1 -- 172.21.15.16:0/2716917554 learned_addr learned my addr 172.21.15.16:0/2716917554 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:46.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.162+0000 7f2392d9d700 1 --2- 172.21.15.16:0/2716917554 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f23940ce4f0 0x7f23941509b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:46.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.162+0000 7f239359e700 1 --2- 172.21.15.16:0/2716917554 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f239415ba10 0x7f23941564f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:46.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.162+0000 7f239259c700 1 -- 172.21.15.16:0/2716917554 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f239415ba10 msgr2=0x7f23941564f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:46.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.162+0000 7f239259c700 1 --2- 172.21.15.16:0/2716917554 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f239415ba10 0x7f23941564f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:46.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.162+0000 7f239259c700 1 -- 172.21.15.16:0/2716917554 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f23940ce4f0 msgr2=0x7f23941509b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:46.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.162+0000 7f239259c700 1 --2- 172.21.15.16:0/2716917554 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f23940ce4f0 0x7f23941509b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:46.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.162+0000 7f239259c700 1 -- 172.21.15.16:0/2716917554 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f238c009b30 con 0x7f239415b630 2023-10-16T12:23:46.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.162+0000 7f239259c700 1 --2- 172.21.15.16:0/2716917554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f239415b630 0x7f239415bee0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f238800e870 tx=0x7f238800eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:46.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.162+0000 7f237b7fe700 1 -- 172.21.15.16:0/2716917554 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2388009590 con 0x7f239415b630 2023-10-16T12:23:46.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.162+0000 7f237b7fe700 1 -- 172.21.15.16:0/2716917554 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2388013680 con 0x7f239415b630 2023-10-16T12:23:46.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.162+0000 7f2399154700 1 -- 172.21.15.16:0/2716917554 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2394155cf0 con 0x7f239415b630 2023-10-16T12:23:46.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.162+0000 7f237b7fe700 1 -- 172.21.15.16:0/2716917554 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23880139f0 con 0x7f239415b630 2023-10-16T12:23:46.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.162+0000 7f2399154700 1 -- 172.21.15.16:0/2716917554 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f239406b960 con 0x7f239415b630 2023-10-16T12:23:46.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.163+0000 7f2399154700 1 -- 172.21.15.16:0/2716917554 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2394156340 con 0x7f239415b630 2023-10-16T12:23:46.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.164+0000 7f237b7fe700 1 -- 172.21.15.16:0/2716917554 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 14) v1 ==== 89400+0+0 (secure 0 0 0) 0x7f2388013b50 con 0x7f239415b630 2023-10-16T12:23:46.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.164+0000 7f237b7fe700 1 --2- 172.21.15.16:0/2716917554 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f238006b0b0 0x7f238006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:46.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.164+0000 7f2392d9d700 1 -- 172.21.15.16:0/2716917554 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f238006b0b0 msgr2=0x7f238006d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1707126831 2023-10-16T12:23:46.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.164+0000 7f237b7fe700 1 -- 172.21.15.16:0/2716917554 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6980+0+0 (secure 0 0 0) 0x7f2388089c20 con 0x7f239415b630 2023-10-16T12:23:46.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.164+0000 7f2392d9d700 1 --2- 172.21.15.16:0/2716917554 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f238006b0b0 0x7f238006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:23:46.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.169+0000 7f237b7fe700 1 -- 172.21.15.16:0/2716917554 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2388054550 con 0x7f239415b630 2023-10-16T12:23:46.215 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:23:46.214+0000 7fb59d613200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T12:23:46.240 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:23:46.238+0000 7f98e4adb200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:23:46.322 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:23:46.320+0000 7f98e4adb200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T12:23:46.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.365+0000 7f2392d9d700 1 -- 172.21.15.16:0/2716917554 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f238006b0b0 msgr2=0x7f238006d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1707126831 2023-10-16T12:23:46.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.365+0000 7f2392d9d700 1 --2- 172.21.15.16:0/2716917554 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f238006b0b0 0x7f238006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:23:46.367 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:23:46.366+0000 7fb59d613200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:23:46.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.384+0000 7f2399154700 1 -- 172.21.15.16:0/2716917554 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f239404ebb0 con 0x7f239415b630 2023-10-16T12:23:46.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.385+0000 7f237b7fe700 1 -- 172.21.15.16:0/2716917554 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v54) v1 ==== 74+0+18157 (secure 0 0 0) 0x7f2388021090 con 0x7f239415b630 2023-10-16T12:23:46.388 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:23:46.388 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":54,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:23:46.199586+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:11:58.700868+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2023-10-16T12:11:59.495384+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":220,"snap_epoch":53,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.7","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/1300148187":"2023-10-17T12:12:15.525647+0000","172.21.15.16:0/675356848":"2023-10-17T12:12:15.525647+0000","172.21.15.16:6832/365921436":"2023-10-17T12:12:15.525647+0000","172.21.15.16:0/3547353312":"2023-10-17T12:12:15.525647+0000","172.21.15.16:0/1893346008":"2023-10-17T12:12:15.525647+0000","172.21.15.16:6833/365921436":"2023-10-17T12:12:15.525647+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":7,"length":1},{"begin":14,"length":1},{"begin":18,"length":1},{"begin":24,"length":1},{"begin":28,"length":1},{"begin":35,"length":1},{"begin":39,"length":1},{"begin":46,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":67,"length":1},{"begin":71,"length":2},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":89,"length":1},{"begin":93,"length":1},{"begin":99,"length":1},{"begin":106,"length":1},{"begin":110,"length":1},{"begin":199,"length":22}]}],"new_removed_snaps":[],"new_purged_snaps":[{"pool":6,"snaps":[{"begin":4,"length":2},{"begin":9,"length":2},{"begin":15,"length":2},{"begin":20,"length":2},{"begin":25,"length":2},{"begin":30,"length":3},{"begin":36,"length":2},{"begin":41,"length":3},{"begin":47,"length":2},{"begin":52,"length":2},{"begin":57,"length":2},{"begin":63,"length":2},{"begin":68,"length":2},{"begin":74,"length":2},{"begin":79,"length":2},{"begin":85,"length":2},{"begin":90,"length":2},{"begin":95,"length":2},{"begin":100,"length":3},{"begin":107,"length":2},{"begin":111,"length":1},{"begin":155,"length":44}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:23:46.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.387+0000 7f2399154700 1 -- 172.21.15.16:0/2716917554 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f238006b0b0 msgr2=0x7f238006d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:23:46.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.387+0000 7f2399154700 1 --2- 172.21.15.16:0/2716917554 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f238006b0b0 0x7f238006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:46.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.387+0000 7f2399154700 1 -- 172.21.15.16:0/2716917554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f239415b630 msgr2=0x7f239415bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:46.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.387+0000 7f2399154700 1 --2- 172.21.15.16:0/2716917554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f239415b630 0x7f239415bee0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f238800e870 tx=0x7f238800eb80 comp rx=0 tx=0).stop 2023-10-16T12:23:46.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.388+0000 7f2399154700 1 -- 172.21.15.16:0/2716917554 shutdown_connections 2023-10-16T12:23:46.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.388+0000 7f2399154700 1 --2- 172.21.15.16:0/2716917554 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f238006b0b0 0x7f238006d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:46.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.388+0000 7f2399154700 1 --2- 172.21.15.16:0/2716917554 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f239415ba10 0x7f23941564f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:46.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.388+0000 7f2399154700 1 --2- 172.21.15.16:0/2716917554 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f23940ce4f0 0x7f23941509b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:46.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.388+0000 7f2399154700 1 --2- 172.21.15.16:0/2716917554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f239415b630 0x7f239415bee0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:46.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.388+0000 7f2399154700 1 -- 172.21.15.16:0/2716917554 >> 172.21.15.16:0/2716917554 conn(0x7f23940d14b0 msgr2=0x7f23940af2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:46.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.388+0000 7f2399154700 1 -- 172.21.15.16:0/2716917554 shutdown_connections 2023-10-16T12:23:46.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.388+0000 7f2399154700 1 -- 172.21.15.16:0/2716917554 wait complete. 2023-10-16T12:23:46.402 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T12:11:58.700868+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T12:11:59.495384+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '53', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 220, 'snap_epoch': 53, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T12:23:46.403 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:23:46.479 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:23:46.477+0000 7f98e4adb200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:23:46.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.530+0000 7f403b7ed700 1 Processor -- start 2023-10-16T12:23:46.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.531+0000 7f403b7ed700 1 -- start start 2023-10-16T12:23:46.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.531+0000 7f403b7ed700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f403406beb0 0x7f40340ce890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:46.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.531+0000 7f403b7ed700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f403406aa10 con 0x7f4034160ee0 2023-10-16T12:23:46.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.531+0000 7f403b7ed700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40340ceed0 con 0x7f40340af210 2023-10-16T12:23:46.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.531+0000 7f403b7ed700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40340cf770 con 0x7f403406beb0 2023-10-16T12:23:46.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.532+0000 7f4039589700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f4034160ee0 0x7f40340aeb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54646/0 (socket says 172.21.15.16:54646) 2023-10-16T12:23:46.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.532+0000 7f4039589700 1 -- 172.21.15.16:0/3056364467 learned_addr learned my addr 172.21.15.16:0/3056364467 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:23:46.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.532+0000 7f4039d8a700 1 --2- 172.21.15.16:0/3056364467 >> v2:172.21.15.49:3300/0 conn(0x7f403406beb0 0x7f40340ce890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:46.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.532+0000 7f402bfff700 1 -- 172.21.15.16:0/3056364467 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4145760797 0 0) 0x7f403406aa10 con 0x7f4034160ee0 2023-10-16T12:23:46.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.532+0000 7f402bfff700 1 -- 172.21.15.16:0/3056364467 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40200035b0 con 0x7f4034160ee0 2023-10-16T12:23:46.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.532+0000 7f4039d8a700 1 -- 172.21.15.16:0/3056364467 >> v1:172.21.15.49:6789/0 conn(0x7f40340af210 legacy=0x7f40340ce180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:46.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.532+0000 7f4039d8a700 1 -- 172.21.15.16:0/3056364467 >> v1:172.21.15.16:6789/0 conn(0x7f4034160ee0 legacy=0x7f40340aeb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:46.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.532+0000 7f4039d8a700 1 -- 172.21.15.16:0/3056364467 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40340cf1b0 con 0x7f403406beb0 2023-10-16T12:23:46.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.532+0000 7f4039d8a700 1 --2- 172.21.15.16:0/3056364467 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f403406beb0 0x7f40340ce890 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f402c009d90 tx=0x7f402c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b3df1b8929aad05 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:46.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.532+0000 7f402bfff700 1 -- 172.21.15.16:0/3056364467 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f402c00e070 con 0x7f403406beb0 2023-10-16T12:23:46.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.532+0000 7f402bfff700 1 -- 172.21.15.16:0/3056364467 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f402c00bb20 con 0x7f403406beb0 2023-10-16T12:23:46.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.532+0000 7f402bfff700 1 -- 172.21.15.16:0/3056364467 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f402c012430 con 0x7f403406beb0 2023-10-16T12:23:46.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.533+0000 7f403b7ed700 1 -- 172.21.15.16:0/3056364467 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f403406beb0 msgr2=0x7f40340ce890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:46.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.533+0000 7f403b7ed700 1 --2- 172.21.15.16:0/3056364467 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f403406beb0 0x7f40340ce890 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f402c009d90 tx=0x7f402c00b330 comp rx=0 tx=0).stop 2023-10-16T12:23:46.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.533+0000 7f403b7ed700 1 -- 172.21.15.16:0/3056364467 shutdown_connections 2023-10-16T12:23:46.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.533+0000 7f403b7ed700 1 --2- 172.21.15.16:0/3056364467 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f403406beb0 0x7f40340ce890 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:46.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.533+0000 7f403b7ed700 1 -- 172.21.15.16:0/3056364467 >> 172.21.15.16:0/3056364467 conn(0x7f40340cc5e0 msgr2=0x7f40340cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:46.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.533+0000 7f403b7ed700 1 -- 172.21.15.16:0/3056364467 shutdown_connections 2023-10-16T12:23:46.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.533+0000 7f403b7ed700 1 -- 172.21.15.16:0/3056364467 wait complete. 2023-10-16T12:23:46.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.535+0000 7f403b7ed700 1 Processor -- start 2023-10-16T12:23:46.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.535+0000 7f403b7ed700 1 -- start start 2023-10-16T12:23:46.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.535+0000 7f403b7ed700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f403406beb0 0x7f403414f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:46.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.535+0000 7f403b7ed700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f40340af210 0x7f40340c7640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:46.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.535+0000 7f403b7ed700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4034160ee0 0x7f40340af960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:46.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.535+0000 7f403b7ed700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4034159700 con 0x7f403406beb0 2023-10-16T12:23:46.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.535+0000 7f403b7ed700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40341599e0 con 0x7f4034160ee0 2023-10-16T12:23:46.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.535+0000 7f403b7ed700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4034159cc0 con 0x7f40340af210 2023-10-16T12:23:46.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.535+0000 7f4039589700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f403406beb0 0x7f403414f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:46.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.536+0000 7f4039589700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f403406beb0 0x7f403414f180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37146/0 (socket says 172.21.15.16:37146) 2023-10-16T12:23:46.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.536+0000 7f4039589700 1 -- 172.21.15.16:0/1215840894 learned_addr learned my addr 172.21.15.16:0/1215840894 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:46.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.536+0000 7f4039d8a700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4034160ee0 0x7f40340af960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:46.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.536+0000 7f4038d88700 1 --2- 172.21.15.16:0/1215840894 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f40340af210 0x7f40340c7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:46.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.536+0000 7f4039589700 1 -- 172.21.15.16:0/1215840894 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f40340af210 msgr2=0x7f40340c7640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:46.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.536+0000 7f4039589700 1 --2- 172.21.15.16:0/1215840894 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f40340af210 0x7f40340c7640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:46.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.536+0000 7f4039589700 1 -- 172.21.15.16:0/1215840894 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4034160ee0 msgr2=0x7f40340af960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:46.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.536+0000 7f4039589700 1 --2- 172.21.15.16:0/1215840894 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4034160ee0 0x7f40340af960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:46.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.536+0000 7f4039589700 1 -- 172.21.15.16:0/1215840894 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f402c009a70 con 0x7f403406beb0 2023-10-16T12:23:46.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.536+0000 7f4039d8a700 1 --2- 172.21.15.16:0/1215840894 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4034160ee0 0x7f40340af960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:23:46.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.536+0000 7f4039589700 1 --2- 172.21.15.16:0/1215840894 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f403406beb0 0x7f403414f180 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f4030001eb0 tx=0x7f403000c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:46.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.536+0000 7f4029ffb700 1 -- 172.21.15.16:0/1215840894 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4030005d70 con 0x7f403406beb0 2023-10-16T12:23:46.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.536+0000 7f4029ffb700 1 -- 172.21.15.16:0/1215840894 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40300070d0 con 0x7f403406beb0 2023-10-16T12:23:46.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.536+0000 7f403b7ed700 1 -- 172.21.15.16:0/1215840894 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4034159fa0 con 0x7f403406beb0 2023-10-16T12:23:46.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.536+0000 7f4029ffb700 1 -- 172.21.15.16:0/1215840894 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4030007460 con 0x7f403406beb0 2023-10-16T12:23:46.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.537+0000 7f403b7ed700 1 -- 172.21.15.16:0/1215840894 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f403414f6c0 con 0x7f403406beb0 2023-10-16T12:23:46.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.538+0000 7f403b7ed700 1 -- 172.21.15.16:0/1215840894 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f403404ebb0 con 0x7f403406beb0 2023-10-16T12:23:46.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.538+0000 7f4029ffb700 1 -- 172.21.15.16:0/1215840894 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 14) v1 ==== 89400+0+0 (secure 0 0 0) 0x7f40300075c0 con 0x7f403406beb0 2023-10-16T12:23:46.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.538+0000 7f4029ffb700 1 --2- 172.21.15.16:0/1215840894 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f401806b0b0 0x7f401806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:46.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.538+0000 7f4038d88700 1 -- 172.21.15.16:0/1215840894 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f401806b0b0 msgr2=0x7f401806d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1707126831 2023-10-16T12:23:46.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.538+0000 7f4029ffb700 1 -- 172.21.15.16:0/1215840894 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 7044+0+0 (secure 0 0 0) 0x7f40300892a0 con 0x7f403406beb0 2023-10-16T12:23:46.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.538+0000 7f4038d88700 1 --2- 172.21.15.16:0/1215840894 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f401806b0b0 0x7f401806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:23:46.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.543+0000 7f4029ffb700 1 -- 172.21.15.16:0/1215840894 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4030053b10 con 0x7f403406beb0 2023-10-16T12:23:46.602 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:23:46.601+0000 7fb59d613200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T12:23:46.678 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:23:46.677+0000 7fb59d613200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T12:23:46.721 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:23:46.719+0000 7f98e4adb200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T12:23:46.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.739+0000 7f4038d88700 1 -- 172.21.15.16:0/1215840894 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f401806b0b0 msgr2=0x7f401806d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1707126831 2023-10-16T12:23:46.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.739+0000 7f4038d88700 1 --2- 172.21.15.16:0/1215840894 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f401806b0b0 0x7f401806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:23:46.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.747+0000 7f403b7ed700 1 -- 172.21.15.16:0/1215840894 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f40340c5a30 con 0x7f403406beb0 2023-10-16T12:23:46.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.748+0000 7f4029ffb700 1 -- 172.21.15.16:0/1215840894 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v54) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4030057130 con 0x7f403406beb0 2023-10-16T12:23:46.750 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T12:23:46.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.750+0000 7f403b7ed700 1 -- 172.21.15.16:0/1215840894 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f401806b0b0 msgr2=0x7f401806d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:23:46.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.750+0000 7f403b7ed700 1 --2- 172.21.15.16:0/1215840894 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f401806b0b0 0x7f401806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:46.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.750+0000 7f403b7ed700 1 -- 172.21.15.16:0/1215840894 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f403406beb0 msgr2=0x7f403414f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:46.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.750+0000 7f403b7ed700 1 --2- 172.21.15.16:0/1215840894 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f403406beb0 0x7f403414f180 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f4030001eb0 tx=0x7f403000c7b0 comp rx=0 tx=0).stop 2023-10-16T12:23:46.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.750+0000 7f403b7ed700 1 -- 172.21.15.16:0/1215840894 shutdown_connections 2023-10-16T12:23:46.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.750+0000 7f403b7ed700 1 --2- 172.21.15.16:0/1215840894 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f401806b0b0 0x7f401806d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:46.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.750+0000 7f403b7ed700 1 --2- 172.21.15.16:0/1215840894 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4034160ee0 0x7f40340af960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:46.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.750+0000 7f403b7ed700 1 --2- 172.21.15.16:0/1215840894 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f403406beb0 0x7f403414f180 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:46.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.750+0000 7f403b7ed700 1 --2- 172.21.15.16:0/1215840894 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f40340af210 0x7f40340c7640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:46.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.750+0000 7f403b7ed700 1 -- 172.21.15.16:0/1215840894 >> 172.21.15.16:0/1215840894 conn(0x7f40340cc5e0 msgr2=0x7f40340b0aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:46.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.750+0000 7f403b7ed700 1 -- 172.21.15.16:0/1215840894 shutdown_connections 2023-10-16T12:23:46.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.750+0000 7f403b7ed700 1 -- 172.21.15.16:0/1215840894 wait complete. 2023-10-16T12:23:46.764 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T12:23:46.804 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:23:46.802+0000 7f98e4adb200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T12:23:46.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.894+0000 7f5020037700 1 Processor -- start 2023-10-16T12:23:46.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.894+0000 7f5020037700 1 -- start start 2023-10-16T12:23:46.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.895+0000 7f5020037700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f501815f060 0x7f50180cc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:46.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.895+0000 7f5020037700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50180b2440 con 0x7f50180ce5d0 2023-10-16T12:23:46.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.895+0000 7f5020037700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50180d00e0 con 0x7f501815ec80 2023-10-16T12:23:46.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.895+0000 7f5020037700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50180b2c50 con 0x7f501815f060 2023-10-16T12:23:46.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.895+0000 7f501e5d4700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f501815f060 0x7f50180cc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:46.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.895+0000 7f501e5d4700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f501815f060 0x7f50180cc8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37160/0 (socket says 172.21.15.16:37160) 2023-10-16T12:23:46.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.895+0000 7f501d5d2700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f50180ce5d0 0x7f50180cc190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54662/0 (socket says 172.21.15.16:54662) 2023-10-16T12:23:46.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.895+0000 7f501e5d4700 1 -- 172.21.15.16:0/848939138 learned_addr learned my addr 172.21.15.16:0/848939138 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:46.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.895+0000 7f501e5d4700 1 -- 172.21.15.16:0/848939138 >> v1:172.21.15.49:6789/0 conn(0x7f501815ec80 legacy=0x7f50180cdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:46.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.895+0000 7f501e5d4700 1 -- 172.21.15.16:0/848939138 >> v1:172.21.15.16:6789/0 conn(0x7f50180ce5d0 legacy=0x7f50180cc190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:46.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.895+0000 7f501cdd1700 1 -- 172.21.15.16:0/848939138 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2553245367 0 0) 0x7f50180b2440 con 0x7f50180ce5d0 2023-10-16T12:23:46.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.895+0000 7f501e5d4700 1 -- 172.21.15.16:0/848939138 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50180cef60 con 0x7f501815f060 2023-10-16T12:23:46.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.895+0000 7f501e5d4700 1 --2- 172.21.15.16:0/848939138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f501815f060 0x7f50180cc8a0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f5010009e80 tx=0x7f501000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d461d87576101cce server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:46.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.895+0000 7f501cdd1700 1 -- 172.21.15.16:0/848939138 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f501000e070 con 0x7f501815f060 2023-10-16T12:23:46.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.896+0000 7f501cdd1700 1 -- 172.21.15.16:0/848939138 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f501000b980 con 0x7f501815f060 2023-10-16T12:23:46.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.896+0000 7f501cdd1700 1 -- 172.21.15.16:0/848939138 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f501000bd30 con 0x7f501815f060 2023-10-16T12:23:46.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.896+0000 7f5020037700 1 -- 172.21.15.16:0/848939138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f501815f060 msgr2=0x7f50180cc8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:46.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.896+0000 7f5020037700 1 --2- 172.21.15.16:0/848939138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f501815f060 0x7f50180cc8a0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f5010009e80 tx=0x7f501000b330 comp rx=0 tx=0).stop 2023-10-16T12:23:46.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.896+0000 7f5020037700 1 -- 172.21.15.16:0/848939138 shutdown_connections 2023-10-16T12:23:46.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.896+0000 7f5020037700 1 --2- 172.21.15.16:0/848939138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f501815f060 0x7f50180cc8a0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:46.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.896+0000 7f5020037700 1 -- 172.21.15.16:0/848939138 >> 172.21.15.16:0/848939138 conn(0x7f50180d15f0 msgr2=0x7f50180d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:46.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.896+0000 7f5020037700 1 -- 172.21.15.16:0/848939138 shutdown_connections 2023-10-16T12:23:46.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.896+0000 7f5020037700 1 -- 172.21.15.16:0/848939138 wait complete. 2023-10-16T12:23:46.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.898+0000 7f5020037700 1 Processor -- start 2023-10-16T12:23:46.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.898+0000 7f5020037700 1 -- start start 2023-10-16T12:23:46.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.898+0000 7f5020037700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50180ce5d0 0x7f50181f5d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:46.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.898+0000 7f5020037700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f501815ec80 0x7f50180c86c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:46.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.898+0000 7f5020037700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f501815f060 0x7f50180c5100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:46.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.898+0000 7f5020037700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50181533e0 con 0x7f50180ce5d0 2023-10-16T12:23:46.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.898+0000 7f5020037700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f501815b700 con 0x7f501815ec80 2023-10-16T12:23:46.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.898+0000 7f5020037700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f501815ce60 con 0x7f501815f060 2023-10-16T12:23:46.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.898+0000 7f501ddd3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50180ce5d0 0x7f50181f5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:46.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.898+0000 7f501ddd3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50180ce5d0 0x7f50181f5d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37162/0 (socket says 172.21.15.16:37162) 2023-10-16T12:23:46.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.899+0000 7f501ddd3700 1 -- 172.21.15.16:0/1343568449 learned_addr learned my addr 172.21.15.16:0/1343568449 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:46.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.899+0000 7f501d5d2700 1 --2- 172.21.15.16:0/1343568449 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f501815ec80 0x7f50180c86c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:46.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.899+0000 7f501e5d4700 1 --2- 172.21.15.16:0/1343568449 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f501815f060 0x7f50180c5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:46.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.899+0000 7f501ddd3700 1 -- 172.21.15.16:0/1343568449 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f501815f060 msgr2=0x7f50180c5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:46.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.899+0000 7f501ddd3700 1 --2- 172.21.15.16:0/1343568449 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f501815f060 0x7f50180c5100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:46.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.899+0000 7f501ddd3700 1 -- 172.21.15.16:0/1343568449 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f501815ec80 msgr2=0x7f50180c86c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:46.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.899+0000 7f501ddd3700 1 --2- 172.21.15.16:0/1343568449 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f501815ec80 0x7f50180c86c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:46.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.899+0000 7f501ddd3700 1 -- 172.21.15.16:0/1343568449 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5010009b30 con 0x7f50180ce5d0 2023-10-16T12:23:46.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.899+0000 7f501ddd3700 1 --2- 172.21.15.16:0/1343568449 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50180ce5d0 0x7f50181f5d00 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f501400c4e0 tx=0x7f501400c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:46.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.899+0000 7f500a7fc700 1 -- 172.21.15.16:0/1343568449 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50140052d0 con 0x7f50180ce5d0 2023-10-16T12:23:46.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.899+0000 7f500a7fc700 1 -- 172.21.15.16:0/1343568449 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5014005430 con 0x7f50180ce5d0 2023-10-16T12:23:46.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.899+0000 7f5020037700 1 -- 172.21.15.16:0/1343568449 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50180c5c70 con 0x7f50180ce5d0 2023-10-16T12:23:46.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.899+0000 7f500a7fc700 1 -- 172.21.15.16:0/1343568449 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50140038c0 con 0x7f50180ce5d0 2023-10-16T12:23:46.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.899+0000 7f5020037700 1 -- 172.21.15.16:0/1343568449 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f501815b0b0 con 0x7f50180ce5d0 2023-10-16T12:23:46.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.900+0000 7f5020037700 1 -- 172.21.15.16:0/1343568449 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f501804ebb0 con 0x7f50180ce5d0 2023-10-16T12:23:46.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.901+0000 7f500a7fc700 1 -- 172.21.15.16:0/1343568449 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 14) v1 ==== 89400+0+0 (secure 0 0 0) 0x7f5014003a20 con 0x7f50180ce5d0 2023-10-16T12:23:46.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.901+0000 7f500a7fc700 1 --2- 172.21.15.16:0/1343568449 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f4ffc06b0b0 0x7f4ffc06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:46.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.901+0000 7f501d5d2700 1 -- 172.21.15.16:0/1343568449 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f4ffc06b0b0 msgr2=0x7f4ffc06d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1707126831 2023-10-16T12:23:46.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.901+0000 7f500a7fc700 1 -- 172.21.15.16:0/1343568449 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 7044+0+0 (secure 0 0 0) 0x7f5014089140 con 0x7f50180ce5d0 2023-10-16T12:23:46.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.901+0000 7f501d5d2700 1 --2- 172.21.15.16:0/1343568449 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f4ffc06b0b0 0x7f4ffc06d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:23:46.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:46.906+0000 7f500a7fc700 1 -- 172.21.15.16:0/1343568449 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5014053ac0 con 0x7f50180ce5d0 2023-10-16T12:23:46.952 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:23:46.951+0000 7fb59d613200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T12:23:47.085 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:23:47.083+0000 7f98e4adb200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T12:23:47.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.102+0000 7f501d5d2700 1 -- 172.21.15.16:0/1343568449 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f4ffc06b0b0 msgr2=0x7f4ffc06d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1707126831 2023-10-16T12:23:47.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.102+0000 7f501d5d2700 1 --2- 172.21.15.16:0/1343568449 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f4ffc06b0b0 0x7f4ffc06d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:23:47.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.109+0000 7f5020037700 1 -- 172.21.15.16:0/1343568449 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f501806fec0 con 0x7f50180ce5d0 2023-10-16T12:23:47.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.110+0000 7f500a7fc700 1 -- 172.21.15.16:0/1343568449 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v54) v1 ==== 104+0+11 (secure 0 0 0) 0x7f50140570e0 con 0x7f50180ce5d0 2023-10-16T12:23:47.112 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-10-16T12:23:47.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.112+0000 7f5020037700 1 -- 172.21.15.16:0/1343568449 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f4ffc06b0b0 msgr2=0x7f4ffc06d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:23:47.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.112+0000 7f5020037700 1 --2- 172.21.15.16:0/1343568449 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f4ffc06b0b0 0x7f4ffc06d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:47.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.112+0000 7f5020037700 1 -- 172.21.15.16:0/1343568449 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50180ce5d0 msgr2=0x7f50181f5d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:47.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.112+0000 7f5020037700 1 --2- 172.21.15.16:0/1343568449 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50180ce5d0 0x7f50181f5d00 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f501400c4e0 tx=0x7f501400c7f0 comp rx=0 tx=0).stop 2023-10-16T12:23:47.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.112+0000 7f5020037700 1 -- 172.21.15.16:0/1343568449 shutdown_connections 2023-10-16T12:23:47.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.112+0000 7f5020037700 1 --2- 172.21.15.16:0/1343568449 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f4ffc06b0b0 0x7f4ffc06d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:47.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.112+0000 7f5020037700 1 --2- 172.21.15.16:0/1343568449 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f501815f060 0x7f50180c5100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:47.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.112+0000 7f5020037700 1 --2- 172.21.15.16:0/1343568449 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50180ce5d0 0x7f50181f5d00 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:47.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.112+0000 7f5020037700 1 --2- 172.21.15.16:0/1343568449 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f501815ec80 0x7f50180c86c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:47.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.112+0000 7f5020037700 1 -- 172.21.15.16:0/1343568449 >> 172.21.15.16:0/1343568449 conn(0x7f50180d15f0 msgr2=0x7f501815dbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:47.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.112+0000 7f5020037700 1 -- 172.21.15.16:0/1343568449 shutdown_connections 2023-10-16T12:23:47.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.112+0000 7f5020037700 1 -- 172.21.15.16:0/1343568449 wait complete. 2023-10-16T12:23:47.126 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T12:23:47.179 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:23:47.178+0000 7fb59d613200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T12:23:47.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.295+0000 7efd6221e700 1 Processor -- start 2023-10-16T12:23:47.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.296+0000 7efd6221e700 1 -- start start 2023-10-16T12:23:47.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.297+0000 7efd6221e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efd5c0cbe20 0x7efd5c0cda60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:47.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.297+0000 7efd6221e700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd5c15aa20 con 0x7efd5c15a640 2023-10-16T12:23:47.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.297+0000 7efd6221e700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd5c06eb40 con 0x7efd5c0c6770 2023-10-16T12:23:47.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.297+0000 7efd6221e700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd5c06ee20 con 0x7efd5c0cbe20 2023-10-16T12:23:47.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.297+0000 7efd5b7fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7efd5c15a640 0x7efd5c0cb710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54678/0 (socket says 172.21.15.16:54678) 2023-10-16T12:23:47.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.297+0000 7efd5b7fe700 1 -- 172.21.15.16:0/2136088601 learned_addr learned my addr 172.21.15.16:0/2136088601 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:23:47.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.297+0000 7efd5affd700 1 --2- 172.21.15.16:0/2136088601 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efd5c0cbe20 0x7efd5c0cda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:47.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.297+0000 7efd5a7fc700 1 -- 172.21.15.16:0/2136088601 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3571380111 0 0) 0x7efd4c001110 con 0x7efd5c15a640 2023-10-16T12:23:47.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.297+0000 7efd5a7fc700 1 -- 172.21.15.16:0/2136088601 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd440035b0 con 0x7efd5c15a640 2023-10-16T12:23:47.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.297+0000 7efd5a7fc700 1 -- 172.21.15.16:0/2136088601 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4106664020 0 0) 0x7efd5c06eb40 con 0x7efd5c0c6770 2023-10-16T12:23:47.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.297+0000 7efd5a7fc700 1 -- 172.21.15.16:0/2136088601 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd4c001110 con 0x7efd5c0c6770 2023-10-16T12:23:47.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.297+0000 7efd5affd700 1 -- 172.21.15.16:0/2136088601 >> v1:172.21.15.49:6789/0 conn(0x7efd5c0c6770 legacy=0x7efd5c0cdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:47.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.297+0000 7efd5affd700 1 -- 172.21.15.16:0/2136088601 >> v1:172.21.15.16:6789/0 conn(0x7efd5c15a640 legacy=0x7efd5c0cb710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:47.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.297+0000 7efd5affd700 1 -- 172.21.15.16:0/2136088601 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd5c06f100 con 0x7efd5c0cbe20 2023-10-16T12:23:47.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.298+0000 7efd5affd700 1 --2- 172.21.15.16:0/2136088601 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efd5c0cbe20 0x7efd5c0cda60 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7efd50009870 tx=0x7efd50009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=949c66397d875d1a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:47.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.298+0000 7efd5a7fc700 1 -- 172.21.15.16:0/2136088601 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd50011830 con 0x7efd5c0cbe20 2023-10-16T12:23:47.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.298+0000 7efd5a7fc700 1 -- 172.21.15.16:0/2136088601 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd50011990 con 0x7efd5c0cbe20 2023-10-16T12:23:47.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.298+0000 7efd5a7fc700 1 -- 172.21.15.16:0/2136088601 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd50011d20 con 0x7efd5c0cbe20 2023-10-16T12:23:47.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.298+0000 7efd6221e700 1 -- 172.21.15.16:0/2136088601 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efd5c0cbe20 msgr2=0x7efd5c0cda60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:47.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.298+0000 7efd6221e700 1 --2- 172.21.15.16:0/2136088601 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efd5c0cbe20 0x7efd5c0cda60 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7efd50009870 tx=0x7efd50009b80 comp rx=0 tx=0).stop 2023-10-16T12:23:47.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.299+0000 7efd6221e700 1 -- 172.21.15.16:0/2136088601 shutdown_connections 2023-10-16T12:23:47.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.299+0000 7efd6221e700 1 --2- 172.21.15.16:0/2136088601 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efd5c0cbe20 0x7efd5c0cda60 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:47.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.299+0000 7efd6221e700 1 -- 172.21.15.16:0/2136088601 >> 172.21.15.16:0/2136088601 conn(0x7efd5c0cc5e0 msgr2=0x7efd5c0cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:47.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.299+0000 7efd6221e700 1 -- 172.21.15.16:0/2136088601 shutdown_connections 2023-10-16T12:23:47.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.299+0000 7efd6221e700 1 -- 172.21.15.16:0/2136088601 wait complete. 2023-10-16T12:23:47.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.300+0000 7efd6221e700 1 Processor -- start 2023-10-16T12:23:47.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.301+0000 7efd6221e700 1 -- start start 2023-10-16T12:23:47.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.301+0000 7efd6221e700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efd5c0c6770 0x7efd5c151010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:47.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.301+0000 7efd6221e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd5c0cbe20 0x7efd5c0cb1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:47.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.301+0000 7efd6221e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efd5c15a640 0x7efd5c0cb700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:47.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.302+0000 7efd6221e700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd5c0c5c80 con 0x7efd5c0cbe20 2023-10-16T12:23:47.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.302+0000 7efd6221e700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd5c0c37c0 con 0x7efd5c0c6770 2023-10-16T12:23:47.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.302+0000 7efd6221e700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd5c0c3aa0 con 0x7efd5c15a640 2023-10-16T12:23:47.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.302+0000 7efd5affd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd5c0cbe20 0x7efd5c0cb1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:47.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.302+0000 7efd5affd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd5c0cbe20 0x7efd5c0cb1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37180/0 (socket says 172.21.15.16:37180) 2023-10-16T12:23:47.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.302+0000 7efd5b7fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efd5c0c6770 0x7efd5c151010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:47.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.302+0000 7efd5affd700 1 -- 172.21.15.16:0/706364956 learned_addr learned my addr 172.21.15.16:0/706364956 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:47.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.302+0000 7efd5bfff700 1 --2- 172.21.15.16:0/706364956 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efd5c15a640 0x7efd5c0cb700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:47.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.302+0000 7efd5affd700 1 -- 172.21.15.16:0/706364956 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efd5c15a640 msgr2=0x7efd5c0cb700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:47.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.302+0000 7efd5affd700 1 --2- 172.21.15.16:0/706364956 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efd5c15a640 0x7efd5c0cb700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:47.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.302+0000 7efd5affd700 1 -- 172.21.15.16:0/706364956 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efd5c0c6770 msgr2=0x7efd5c151010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:47.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.302+0000 7efd5affd700 1 --2- 172.21.15.16:0/706364956 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efd5c0c6770 0x7efd5c151010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:47.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.302+0000 7efd5affd700 1 -- 172.21.15.16:0/706364956 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd500094a0 con 0x7efd5c0cbe20 2023-10-16T12:23:47.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.302+0000 7efd5affd700 1 --2- 172.21.15.16:0/706364956 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd5c0cbe20 0x7efd5c0cb1c0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7efd50009bc0 tx=0x7efd50004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:47.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.302+0000 7efd43fff700 1 -- 172.21.15.16:0/706364956 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd5001a760 con 0x7efd5c0cbe20 2023-10-16T12:23:47.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.302+0000 7efd43fff700 1 -- 172.21.15.16:0/706364956 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd5001a8c0 con 0x7efd5c0cbe20 2023-10-16T12:23:47.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.302+0000 7efd6221e700 1 -- 172.21.15.16:0/706364956 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd5c0c3d80 con 0x7efd5c0cbe20 2023-10-16T12:23:47.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.302+0000 7efd43fff700 1 -- 172.21.15.16:0/706364956 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd5001ac70 con 0x7efd5c0cbe20 2023-10-16T12:23:47.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.302+0000 7efd6221e700 1 -- 172.21.15.16:0/706364956 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd5c151550 con 0x7efd5c0cbe20 2023-10-16T12:23:47.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.303+0000 7efd6221e700 1 -- 172.21.15.16:0/706364956 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd5c0c42a0 con 0x7efd5c0cbe20 2023-10-16T12:23:47.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.305+0000 7efd43fff700 1 -- 172.21.15.16:0/706364956 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 14) v1 ==== 89400+0+0 (secure 0 0 0) 0x7efd50022070 con 0x7efd5c0cbe20 2023-10-16T12:23:47.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.305+0000 7efd43fff700 1 --2- 172.21.15.16:0/706364956 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7efd4806b0b0 0x7efd4806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:47.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.305+0000 7efd5b7fe700 1 -- 172.21.15.16:0/706364956 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7efd4806b0b0 msgr2=0x7efd4806d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1707126831 2023-10-16T12:23:47.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.305+0000 7efd43fff700 1 -- 172.21.15.16:0/706364956 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6740+0+0 (secure 0 0 0) 0x7efd5008aee0 con 0x7efd5c0cbe20 2023-10-16T12:23:47.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.305+0000 7efd5b7fe700 1 --2- 172.21.15.16:0/706364956 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7efd4806b0b0 0x7efd4806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:23:47.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.309+0000 7efd43fff700 1 -- 172.21.15.16:0/706364956 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7efd50055970 con 0x7efd5c0cbe20 2023-10-16T12:23:47.323 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:23:47.322+0000 7f98e4adb200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T12:23:47.337 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:23:47.336+0000 7fb59d613200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T12:23:47.454 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:23:47.453+0000 7fb59d613200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T12:23:47.491 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:23:47.489+0000 7f98e4adb200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T12:23:47.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.506+0000 7efd5b7fe700 1 -- 172.21.15.16:0/706364956 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7efd4806b0b0 msgr2=0x7efd4806d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1707126831 2023-10-16T12:23:47.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.506+0000 7efd5b7fe700 1 --2- 172.21.15.16:0/706364956 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7efd4806b0b0 0x7efd4806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:23:47.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.527+0000 7efd6221e700 1 -- 172.21.15.16:0/706364956 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7efd5c0c42a0 con 0x7efd5c0cbe20 2023-10-16T12:23:47.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.527+0000 7efd43fff700 1 -- 172.21.15.16:0/706364956 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v55) v1 ==== 100+0+11 (secure 0 0 0) 0x7efd5c0c42a0 con 0x7efd5c0cbe20 2023-10-16T12:23:47.530 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-10-16T12:23:47.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.530+0000 7efd6221e700 1 -- 172.21.15.16:0/706364956 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7efd4806b0b0 msgr2=0x7efd4806d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:23:47.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.530+0000 7efd6221e700 1 --2- 172.21.15.16:0/706364956 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7efd4806b0b0 0x7efd4806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:47.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.530+0000 7efd6221e700 1 -- 172.21.15.16:0/706364956 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd5c0cbe20 msgr2=0x7efd5c0cb1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:47.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.530+0000 7efd6221e700 1 --2- 172.21.15.16:0/706364956 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd5c0cbe20 0x7efd5c0cb1c0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7efd50009bc0 tx=0x7efd50004750 comp rx=0 tx=0).stop 2023-10-16T12:23:47.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.530+0000 7efd6221e700 1 -- 172.21.15.16:0/706364956 shutdown_connections 2023-10-16T12:23:47.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.530+0000 7efd6221e700 1 --2- 172.21.15.16:0/706364956 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7efd4806b0b0 0x7efd4806d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:47.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.530+0000 7efd6221e700 1 --2- 172.21.15.16:0/706364956 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efd5c15a640 0x7efd5c0cb700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:47.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.530+0000 7efd6221e700 1 --2- 172.21.15.16:0/706364956 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efd5c0c6770 0x7efd5c151010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:47.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.530+0000 7efd6221e700 1 --2- 172.21.15.16:0/706364956 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd5c0cbe20 0x7efd5c0cb1c0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:47.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.530+0000 7efd6221e700 1 -- 172.21.15.16:0/706364956 >> 172.21.15.16:0/706364956 conn(0x7efd5c0cc5e0 msgr2=0x7efd5c154f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:47.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.530+0000 7efd6221e700 1 -- 172.21.15.16:0/706364956 shutdown_connections 2023-10-16T12:23:47.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.530+0000 7efd6221e700 1 -- 172.21.15.16:0/706364956 wait complete. 2023-10-16T12:23:47.545 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:23:47.613 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:23:47.611+0000 7f98e4adb200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T12:23:47.614 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:23:47.612+0000 7fb59d613200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T12:23:47.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.669+0000 7ff219535700 1 Processor -- start 2023-10-16T12:23:47.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.669+0000 7ff219535700 1 -- start start 2023-10-16T12:23:47.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.670+0000 7ff219535700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff21415b630 0x7ff2140cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:47.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.670+0000 7ff219535700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7ff2140ce4f0 0x7ff2140cfca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:47.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.670+0000 7ff219535700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2140cc2a0 con 0x7ff21415ba10 2023-10-16T12:23:47.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.670+0000 7ff219535700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2140cdb20 con 0x7ff2140ce4f0 2023-10-16T12:23:47.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.670+0000 7ff219535700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2140cc540 con 0x7ff21415b630 2023-10-16T12:23:47.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.670+0000 7ff2127fc700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff21415ba10 0x7ff2140cdde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54694/0 (socket says 172.21.15.16:54694) 2023-10-16T12:23:47.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.670+0000 7ff2127fc700 1 -- 172.21.15.16:0/2684930307 learned_addr learned my addr 172.21.15.16:0/2684930307 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:23:47.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.670+0000 7ff2137fe700 1 --2- 172.21.15.16:0/2684930307 >> v2:172.21.15.49:3300/0 conn(0x7ff2140ce4f0 0x7ff2140cfca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:47.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.670+0000 7ff212ffd700 1 --2- 172.21.15.16:0/2684930307 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff21415b630 0x7ff2140cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:47.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.671+0000 7ff211ffb700 1 -- 172.21.15.16:0/2684930307 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4016333582 0 0) 0x7ff2140cc2a0 con 0x7ff21415ba10 2023-10-16T12:23:47.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.671+0000 7ff211ffb700 1 -- 172.21.15.16:0/2684930307 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1fc0035b0 con 0x7ff21415ba10 2023-10-16T12:23:47.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.671+0000 7ff2137fe700 1 -- 172.21.15.16:0/2684930307 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff21415b630 msgr2=0x7ff2140cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:47.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.671+0000 7ff2137fe700 1 --2- 172.21.15.16:0/2684930307 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff21415b630 0x7ff2140cd5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:47.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.671+0000 7ff2137fe700 1 -- 172.21.15.16:0/2684930307 >> v1:172.21.15.16:6789/0 conn(0x7ff21415ba10 legacy=0x7ff2140cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:47.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.671+0000 7ff2137fe700 1 -- 172.21.15.16:0/2684930307 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2140cc820 con 0x7ff2140ce4f0 2023-10-16T12:23:47.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.671+0000 7ff212ffd700 1 --2- 172.21.15.16:0/2684930307 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff21415b630 0x7ff2140cd5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:23:47.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.671+0000 7ff2137fe700 1 --2- 172.21.15.16:0/2684930307 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff2140ce4f0 0x7ff2140cfca0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ff20c009d90 tx=0x7ff20c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=57266e765ba3f1bf server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:47.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.671+0000 7ff211ffb700 1 -- 172.21.15.16:0/2684930307 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff20c00e070 con 0x7ff2140ce4f0 2023-10-16T12:23:47.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.671+0000 7ff211ffb700 1 -- 172.21.15.16:0/2684930307 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff20c00bb20 con 0x7ff2140ce4f0 2023-10-16T12:23:47.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.671+0000 7ff211ffb700 1 -- 172.21.15.16:0/2684930307 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff20c012430 con 0x7ff2140ce4f0 2023-10-16T12:23:47.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.672+0000 7ff219535700 1 -- 172.21.15.16:0/2684930307 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff2140ce4f0 msgr2=0x7ff2140cfca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:47.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.672+0000 7ff219535700 1 --2- 172.21.15.16:0/2684930307 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff2140ce4f0 0x7ff2140cfca0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ff20c009d90 tx=0x7ff20c00b330 comp rx=0 tx=0).stop 2023-10-16T12:23:47.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.672+0000 7ff219535700 1 -- 172.21.15.16:0/2684930307 shutdown_connections 2023-10-16T12:23:47.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.672+0000 7ff219535700 1 --2- 172.21.15.16:0/2684930307 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff21415b630 0x7ff2140cd5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:47.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.672+0000 7ff219535700 1 --2- 172.21.15.16:0/2684930307 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff2140ce4f0 0x7ff2140cfca0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:47.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.672+0000 7ff219535700 1 -- 172.21.15.16:0/2684930307 >> 172.21.15.16:0/2684930307 conn(0x7ff2140d14b0 msgr2=0x7ff2140d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:47.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.672+0000 7ff219535700 1 -- 172.21.15.16:0/2684930307 shutdown_connections 2023-10-16T12:23:47.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.672+0000 7ff219535700 1 -- 172.21.15.16:0/2684930307 wait complete. 2023-10-16T12:23:47.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.673+0000 7ff219535700 1 Processor -- start 2023-10-16T12:23:47.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.673+0000 7ff219535700 1 -- start start 2023-10-16T12:23:47.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.674+0000 7ff219535700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff2140ce4f0 0x7ff2141509b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:47.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.674+0000 7ff219535700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff21415b630 0x7ff21415dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:47.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.674+0000 7ff219535700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff21415ba10 0x7ff2140c7470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:47.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.674+0000 7ff219535700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff21415ec20 con 0x7ff21415b630 2023-10-16T12:23:47.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.674+0000 7ff219535700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff21415d040 con 0x7ff2140ce4f0 2023-10-16T12:23:47.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.674+0000 7ff219535700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2140c58b0 con 0x7ff21415ba10 2023-10-16T12:23:47.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.674+0000 7ff2127fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff21415b630 0x7ff21415dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:47.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.674+0000 7ff2127fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff21415b630 0x7ff21415dba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37196/0 (socket says 172.21.15.16:37196) 2023-10-16T12:23:47.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.674+0000 7ff212ffd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff2140ce4f0 0x7ff2141509b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:47.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.674+0000 7ff2127fc700 1 -- 172.21.15.16:0/339973607 learned_addr learned my addr 172.21.15.16:0/339973607 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:47.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.674+0000 7ff2137fe700 1 --2- 172.21.15.16:0/339973607 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff21415ba10 0x7ff2140c7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:47.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.675+0000 7ff2127fc700 1 -- 172.21.15.16:0/339973607 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff21415ba10 msgr2=0x7ff2140c7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:47.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.675+0000 7ff2127fc700 1 --2- 172.21.15.16:0/339973607 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff21415ba10 0x7ff2140c7470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:47.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.675+0000 7ff2127fc700 1 -- 172.21.15.16:0/339973607 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff2140ce4f0 msgr2=0x7ff2141509b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:47.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.675+0000 7ff2127fc700 1 --2- 172.21.15.16:0/339973607 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff2140ce4f0 0x7ff2141509b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:47.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.675+0000 7ff2127fc700 1 -- 172.21.15.16:0/339973607 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff204009430 con 0x7ff21415b630 2023-10-16T12:23:47.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.675+0000 7ff2127fc700 1 --2- 172.21.15.16:0/339973607 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff21415b630 0x7ff21415dba0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7ff20800c730 tx=0x7ff20800ca40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:47.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.675+0000 7ff1fb7fe700 1 -- 172.21.15.16:0/339973607 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff208005d70 con 0x7ff21415b630 2023-10-16T12:23:47.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.675+0000 7ff1fb7fe700 1 -- 172.21.15.16:0/339973607 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2080070d0 con 0x7ff21415b630 2023-10-16T12:23:47.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.675+0000 7ff1fb7fe700 1 -- 172.21.15.16:0/339973607 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff208007440 con 0x7ff21415b630 2023-10-16T12:23:47.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.675+0000 7ff219535700 1 -- 172.21.15.16:0/339973607 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff20c009a70 con 0x7ff21415b630 2023-10-16T12:23:47.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.675+0000 7ff219535700 1 -- 172.21.15.16:0/339973607 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2140c5b50 con 0x7ff21415b630 2023-10-16T12:23:47.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.677+0000 7ff219535700 1 -- 172.21.15.16:0/339973607 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2140c95d0 con 0x7ff21415b630 2023-10-16T12:23:47.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.677+0000 7ff1fb7fe700 1 -- 172.21.15.16:0/339973607 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 14) v1 ==== 89400+0+0 (secure 0 0 0) 0x7ff2080075a0 con 0x7ff21415b630 2023-10-16T12:23:47.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.677+0000 7ff1fb7fe700 1 --2- 172.21.15.16:0/339973607 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7ff20006b0b0 0x7ff20006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:47.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.677+0000 7ff212ffd700 1 -- 172.21.15.16:0/339973607 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7ff20006b0b0 msgr2=0x7ff20006d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1707126831 2023-10-16T12:23:47.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.677+0000 7ff1fb7fe700 1 -- 172.21.15.16:0/339973607 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6740+0+0 (secure 0 0 0) 0x7ff208088f90 con 0x7ff21415b630 2023-10-16T12:23:47.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.677+0000 7ff212ffd700 1 --2- 172.21.15.16:0/339973607 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7ff20006b0b0 0x7ff20006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:23:47.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.682+0000 7ff1fb7fe700 1 -- 172.21.15.16:0/339973607 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff2080bbfe0 con 0x7ff21415b630 2023-10-16T12:23:47.792 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:23:47.790+0000 7f98e4adb200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T12:23:47.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.878+0000 7ff212ffd700 1 -- 172.21.15.16:0/339973607 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7ff20006b0b0 msgr2=0x7ff20006d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1707126831 2023-10-16T12:23:47.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.878+0000 7ff212ffd700 1 --2- 172.21.15.16:0/339973607 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7ff20006b0b0 0x7ff20006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:23:47.885 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:23:47.884+0000 7fb59d613200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T12:23:47.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.952+0000 7ff219535700 1 -- 172.21.15.16:0/339973607 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff21404ebb0 con 0x7ff21415b630 2023-10-16T12:23:47.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.953+0000 7ff1fb7fe700 1 -- 172.21.15.16:0/339973607 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+4910 (secure 0 0 0) 0x7ff208017090 con 0x7ff21415b630 2023-10-16T12:23:47.955 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:23:47.955 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":18,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5194,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/266676368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":266676368},{"type":"v1","addr":"172.21.15.49:6834","nonce":266676368}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5268,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/1460528697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":1460528697},{"type":"v1","addr":"172.21.15.16:6837","nonce":1460528697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/2374376752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2374376752},{"type":"v1","addr":"172.21.15.16:6836","nonce":2374376752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:12:00.537600+0000","modified":"2023-10-16T12:12:03.538619+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5283},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5283":{"gid":5283,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.49:6835/2788396302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":2788396302},{"type":"v1","addr":"172.21.15.49:6835","nonce":2788396302}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-16T12:23:47.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.956+0000 7ff219535700 1 -- 172.21.15.16:0/339973607 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7ff20006b0b0 msgr2=0x7ff20006d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:23:47.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.956+0000 7ff219535700 1 --2- 172.21.15.16:0/339973607 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7ff20006b0b0 0x7ff20006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:47.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.957+0000 7ff219535700 1 -- 172.21.15.16:0/339973607 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff21415b630 msgr2=0x7ff21415dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:47.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.957+0000 7ff219535700 1 --2- 172.21.15.16:0/339973607 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff21415b630 0x7ff21415dba0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7ff20800c730 tx=0x7ff20800ca40 comp rx=0 tx=0).stop 2023-10-16T12:23:47.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.957+0000 7ff219535700 1 -- 172.21.15.16:0/339973607 shutdown_connections 2023-10-16T12:23:47.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.957+0000 7ff219535700 1 --2- 172.21.15.16:0/339973607 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7ff20006b0b0 0x7ff20006d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:47.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.957+0000 7ff219535700 1 --2- 172.21.15.16:0/339973607 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff21415ba10 0x7ff2140c7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:47.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.957+0000 7ff219535700 1 --2- 172.21.15.16:0/339973607 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff2140ce4f0 0x7ff2141509b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:47.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.957+0000 7ff219535700 1 --2- 172.21.15.16:0/339973607 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff21415b630 0x7ff21415dba0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:47.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.957+0000 7ff219535700 1 -- 172.21.15.16:0/339973607 >> 172.21.15.16:0/339973607 conn(0x7ff2140d14b0 msgr2=0x7ff2140d1890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:47.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.957+0000 7ff219535700 1 -- 172.21.15.16:0/339973607 shutdown_connections 2023-10-16T12:23:47.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:47.957+0000 7ff219535700 1 -- 172.21.15.16:0/339973607 wait complete. 2023-10-16T12:23:47.961 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 18 2023-10-16T12:23:47.978 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:23:48.054 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:23:48.053+0000 7fb59d613200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T12:23:48.075 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:23:48.073+0000 7f98e4adb200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T12:23:48.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.102+0000 7f8f5ea8d700 1 Processor -- start 2023-10-16T12:23:48.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.102+0000 7f8f5ea8d700 1 -- start start 2023-10-16T12:23:48.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.103+0000 7f8f5ea8d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8f5815d8d0 0x7f8f580d1c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:48.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.103+0000 7f8f5ea8d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8f580d41c0 0x7f8f580d0040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:48.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.103+0000 7f8f5ea8d700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f5806ba00 con 0x7f8f580d0580 2023-10-16T12:23:48.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.103+0000 7f8f5ea8d700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f580d0a10 con 0x7f8f5815d8d0 2023-10-16T12:23:48.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.103+0000 7f8f5ea8d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f580d2150 con 0x7f8f580d41c0 2023-10-16T12:23:48.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.103+0000 7f8f5c829700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8f5815d8d0 0x7f8f580d1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:48.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.103+0000 7f8f5c829700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8f5815d8d0 0x7f8f580d1c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37204/0 (socket says 172.21.15.16:37204) 2023-10-16T12:23:48.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.103+0000 7f8f5d02a700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f8f580d0580 0x7f8f580ce240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54714/0 (socket says 172.21.15.16:54714) 2023-10-16T12:23:48.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.103+0000 7f8f5c829700 1 -- 172.21.15.16:0/1519679764 learned_addr learned my addr 172.21.15.16:0/1519679764 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:48.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.104+0000 7f8f57fff700 1 --2- 172.21.15.16:0/1519679764 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8f580d41c0 0x7f8f580d0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:48.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.104+0000 7f8f5c829700 1 -- 172.21.15.16:0/1519679764 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8f580d41c0 msgr2=0x7f8f580d0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:48.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.104+0000 7f8f5c829700 1 --2- 172.21.15.16:0/1519679764 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8f580d41c0 0x7f8f580d0040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:48.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.104+0000 7f8f5c829700 1 -- 172.21.15.16:0/1519679764 >> v1:172.21.15.16:6789/0 conn(0x7f8f580d0580 legacy=0x7f8f580ce240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:48.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.104+0000 7f8f5c829700 1 -- 172.21.15.16:0/1519679764 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f580d26d0 con 0x7f8f5815d8d0 2023-10-16T12:23:48.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.104+0000 7f8f577fe700 1 -- 172.21.15.16:0/1519679764 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2173965283 0 0) 0x7f8f5806ba00 con 0x7f8f580d0580 2023-10-16T12:23:48.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.104+0000 7f8f5c829700 1 --2- 172.21.15.16:0/1519679764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f5815d8d0 0x7f8f580d1c10 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f8f4c0098f0 tx=0x7f8f4c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65525aa8ee346007 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:48.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.104+0000 7f8f577fe700 1 -- 172.21.15.16:0/1519679764 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f4c011830 con 0x7f8f5815d8d0 2023-10-16T12:23:48.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.104+0000 7f8f577fe700 1 -- 172.21.15.16:0/1519679764 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f4c011990 con 0x7f8f5815d8d0 2023-10-16T12:23:48.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.104+0000 7f8f577fe700 1 -- 172.21.15.16:0/1519679764 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f4c011d20 con 0x7f8f5815d8d0 2023-10-16T12:23:48.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.105+0000 7f8f5ea8d700 1 -- 172.21.15.16:0/1519679764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f5815d8d0 msgr2=0x7f8f580d1c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:48.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.105+0000 7f8f5ea8d700 1 --2- 172.21.15.16:0/1519679764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f5815d8d0 0x7f8f580d1c10 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f8f4c0098f0 tx=0x7f8f4c009c00 comp rx=0 tx=0).stop 2023-10-16T12:23:48.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.105+0000 7f8f5ea8d700 1 -- 172.21.15.16:0/1519679764 shutdown_connections 2023-10-16T12:23:48.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.105+0000 7f8f5ea8d700 1 --2- 172.21.15.16:0/1519679764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f5815d8d0 0x7f8f580d1c10 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:48.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.105+0000 7f8f5ea8d700 1 --2- 172.21.15.16:0/1519679764 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8f580d41c0 0x7f8f580d0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:48.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.105+0000 7f8f5ea8d700 1 -- 172.21.15.16:0/1519679764 >> 172.21.15.16:0/1519679764 conn(0x7f8f580d3630 msgr2=0x7f8f580d3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:48.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.105+0000 7f8f5ea8d700 1 -- 172.21.15.16:0/1519679764 shutdown_connections 2023-10-16T12:23:48.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.105+0000 7f8f5ea8d700 1 -- 172.21.15.16:0/1519679764 wait complete. 2023-10-16T12:23:48.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.106+0000 7f8f5ea8d700 1 Processor -- start 2023-10-16T12:23:48.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.107+0000 7f8f5ea8d700 1 -- start start 2023-10-16T12:23:48.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.108+0000 7f8f5ea8d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8f580d0580 0x7f8f5815bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:48.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.108+0000 7f8f5ea8d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8f580d41c0 0x7f8f581e6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:48.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.108+0000 7f8f5ea8d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f5815d8d0 0x7f8f581e7420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:48.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.108+0000 7f8f5ea8d700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f5815c200 con 0x7f8f5815d8d0 2023-10-16T12:23:48.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.108+0000 7f8f5ea8d700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f5806d500 con 0x7f8f580d41c0 2023-10-16T12:23:48.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.108+0000 7f8f5ea8d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f580ccf90 con 0x7f8f580d0580 2023-10-16T12:23:48.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.108+0000 7f8f5d02a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f5815d8d0 0x7f8f581e7420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:48.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.108+0000 7f8f5d02a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f5815d8d0 0x7f8f581e7420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37214/0 (socket says 172.21.15.16:37214) 2023-10-16T12:23:48.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.108+0000 7f8f5d02a700 1 -- 172.21.15.16:0/2575437235 learned_addr learned my addr 172.21.15.16:0/2575437235 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:48.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.108+0000 7f8f5c829700 1 --2- 172.21.15.16:0/2575437235 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8f580d0580 0x7f8f5815bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:48.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.108+0000 7f8f57fff700 1 --2- 172.21.15.16:0/2575437235 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8f580d41c0 0x7f8f581e6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:48.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.108+0000 7f8f5d02a700 1 -- 172.21.15.16:0/2575437235 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8f580d0580 msgr2=0x7f8f5815bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:48.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.108+0000 7f8f5d02a700 1 --2- 172.21.15.16:0/2575437235 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8f580d0580 0x7f8f5815bcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:48.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.108+0000 7f8f5d02a700 1 -- 172.21.15.16:0/2575437235 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8f580d41c0 msgr2=0x7f8f581e6ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:48.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.108+0000 7f8f5d02a700 1 --2- 172.21.15.16:0/2575437235 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8f580d41c0 0x7f8f581e6ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:48.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.108+0000 7f8f5d02a700 1 -- 172.21.15.16:0/2575437235 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f4c009550 con 0x7f8f5815d8d0 2023-10-16T12:23:48.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.108+0000 7f8f5d02a700 1 --2- 172.21.15.16:0/2575437235 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f5815d8d0 0x7f8f581e7420 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f8f5000cc70 tx=0x7f8f5000cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:48.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.108+0000 7f8f557fa700 1 -- 172.21.15.16:0/2575437235 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f5000aac0 con 0x7f8f5815d8d0 2023-10-16T12:23:48.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.108+0000 7f8f557fa700 1 -- 172.21.15.16:0/2575437235 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f5000ac40 con 0x7f8f5815d8d0 2023-10-16T12:23:48.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.108+0000 7f8f5ea8d700 1 -- 172.21.15.16:0/2575437235 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f580cc380 con 0x7f8f5815d8d0 2023-10-16T12:23:48.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.109+0000 7f8f557fa700 1 -- 172.21.15.16:0/2575437235 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f500070d0 con 0x7f8f5815d8d0 2023-10-16T12:23:48.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.109+0000 7f8f5ea8d700 1 -- 172.21.15.16:0/2575437235 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f580cc940 con 0x7f8f5815d8d0 2023-10-16T12:23:48.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.110+0000 7f8f5ea8d700 1 -- 172.21.15.16:0/2575437235 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f5804ebb0 con 0x7f8f5815d8d0 2023-10-16T12:23:48.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.110+0000 7f8f557fa700 1 -- 172.21.15.16:0/2575437235 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 14) v1 ==== 89400+0+0 (secure 0 0 0) 0x7f8f50007250 con 0x7f8f5815d8d0 2023-10-16T12:23:48.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.110+0000 7f8f557fa700 1 --2- 172.21.15.16:0/2575437235 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f8f3c06b0b0 0x7f8f3c06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:48.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.110+0000 7f8f5c829700 1 -- 172.21.15.16:0/2575437235 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f8f3c06b0b0 msgr2=0x7f8f3c06d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1707126831 2023-10-16T12:23:48.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.110+0000 7f8f557fa700 1 -- 172.21.15.16:0/2575437235 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6740+0+0 (secure 0 0 0) 0x7f8f50089ad0 con 0x7f8f5815d8d0 2023-10-16T12:23:48.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.110+0000 7f8f5c829700 1 --2- 172.21.15.16:0/2575437235 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f8f3c06b0b0 0x7f8f3c06d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:23:48.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.115+0000 7f8f557fa700 1 -- 172.21.15.16:0/2575437235 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8f500544f0 con 0x7f8f5815d8d0 2023-10-16T12:23:48.256 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:23:48.254+0000 7f98e4adb200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T12:23:48.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.311+0000 7f8f5c829700 1 -- 172.21.15.16:0/2575437235 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f8f3c06b0b0 msgr2=0x7f8f3c06d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1707126831 2023-10-16T12:23:48.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.311+0000 7f8f5c829700 1 --2- 172.21.15.16:0/2575437235 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f8f3c06b0b0 0x7f8f3c06d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:23:48.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.335+0000 7f8f5ea8d700 1 -- 172.21.15.16:0/2575437235 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8f580cf340 con 0x7f8f5815d8d0 2023-10-16T12:23:48.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.337+0000 7f8f557fa700 1 -- 172.21.15.16:0/2575437235 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) v1 ==== 74+0+17701 (secure 0 0 0) 0x7f8f50018020 con 0x7f8f5815d8d0 2023-10-16T12:23:48.339 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:23:48.339 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":55,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:23:47.201562+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:11:58.700868+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2023-10-16T12:11:59.495384+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":221,"snap_epoch":55,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.7","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/1300148187":"2023-10-17T12:12:15.525647+0000","172.21.15.16:0/675356848":"2023-10-17T12:12:15.525647+0000","172.21.15.16:6832/365921436":"2023-10-17T12:12:15.525647+0000","172.21.15.16:0/3547353312":"2023-10-17T12:12:15.525647+0000","172.21.15.16:0/1893346008":"2023-10-17T12:12:15.525647+0000","172.21.15.16:6833/365921436":"2023-10-17T12:12:15.525647+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":7,"length":1},{"begin":14,"length":1},{"begin":18,"length":1},{"begin":24,"length":1},{"begin":28,"length":1},{"begin":35,"length":1},{"begin":39,"length":1},{"begin":46,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":67,"length":1},{"begin":71,"length":2},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":89,"length":1},{"begin":93,"length":1},{"begin":99,"length":1},{"begin":104,"length":1},{"begin":106,"length":1},{"begin":110,"length":1},{"begin":199,"length":23}]}],"new_removed_snaps":[{"pool":6,"snaps":[{"begin":104,"length":1},{"begin":221,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:23:48.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.339+0000 7f8f5ea8d700 1 -- 172.21.15.16:0/2575437235 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f8f3c06b0b0 msgr2=0x7f8f3c06d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:23:48.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.339+0000 7f8f5ea8d700 1 --2- 172.21.15.16:0/2575437235 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f8f3c06b0b0 0x7f8f3c06d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:48.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.339+0000 7f8f5ea8d700 1 -- 172.21.15.16:0/2575437235 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f5815d8d0 msgr2=0x7f8f581e7420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:48.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.339+0000 7f8f5ea8d700 1 --2- 172.21.15.16:0/2575437235 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f5815d8d0 0x7f8f581e7420 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f8f5000cc70 tx=0x7f8f5000cf80 comp rx=0 tx=0).stop 2023-10-16T12:23:48.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.339+0000 7f8f5ea8d700 1 -- 172.21.15.16:0/2575437235 shutdown_connections 2023-10-16T12:23:48.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.339+0000 7f8f5ea8d700 1 --2- 172.21.15.16:0/2575437235 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f8f3c06b0b0 0x7f8f3c06d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:48.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.339+0000 7f8f5ea8d700 1 --2- 172.21.15.16:0/2575437235 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f5815d8d0 0x7f8f581e7420 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:48.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.339+0000 7f8f5ea8d700 1 --2- 172.21.15.16:0/2575437235 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8f580d0580 0x7f8f5815bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:48.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.339+0000 7f8f5ea8d700 1 --2- 172.21.15.16:0/2575437235 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8f580d41c0 0x7f8f581e6ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:48.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.339+0000 7f8f5ea8d700 1 -- 172.21.15.16:0/2575437235 >> 172.21.15.16:0/2575437235 conn(0x7f8f580d3630 msgr2=0x7f8f5815c8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:48.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.339+0000 7f8f5ea8d700 1 -- 172.21.15.16:0/2575437235 shutdown_connections 2023-10-16T12:23:48.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.339+0000 7f8f5ea8d700 1 -- 172.21.15.16:0/2575437235 wait complete. 2023-10-16T12:23:48.353 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-16T12:23:48.354 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:23:48.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.478+0000 7f12f7044700 1 Processor -- start 2023-10-16T12:23:48.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.478+0000 7f12f7044700 1 -- start start 2023-10-16T12:23:48.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.479+0000 7f12f7044700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f12f00cf880 0x7f12f00b52f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:48.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.479+0000 7f12f7044700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f12f00b5830 0x7f12f015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:48.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.479+0000 7f12f7044700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12f015bc80 con 0x7f12f00cf4a0 2023-10-16T12:23:48.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.479+0000 7f12f7044700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12f015b070 con 0x7f12f00b5830 2023-10-16T12:23:48.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.479+0000 7f12f7044700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12f015b350 con 0x7f12f00cf880 2023-10-16T12:23:48.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.479+0000 7f12f55e1700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f12f00b5830 0x7f12f015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:48.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.479+0000 7f12f55e1700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f12f00b5830 0x7f12f015a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37224/0 (socket says 172.21.15.16:37224) 2023-10-16T12:23:48.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.479+0000 7f12f55e1700 1 -- 172.21.15.16:0/1206078863 learned_addr learned my addr 172.21.15.16:0/1206078863 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:48.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.479+0000 7f12effff700 1 --2- 172.21.15.16:0/1206078863 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f12f00cf880 0x7f12f00b52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:48.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.479+0000 7f12f55e1700 1 -- 172.21.15.16:0/1206078863 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f12f00cf880 msgr2=0x7f12f00b52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:48.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.479+0000 7f12f55e1700 1 --2- 172.21.15.16:0/1206078863 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f12f00cf880 0x7f12f00b52f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:48.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.479+0000 7f12f55e1700 1 -- 172.21.15.16:0/1206078863 >> v1:172.21.15.49:6789/0 conn(0x7f12f00cf4a0 legacy=0x7f12f00726e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:48.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.479+0000 7f12f55e1700 1 -- 172.21.15.16:0/1206078863 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12f015b630 con 0x7f12f00b5830 2023-10-16T12:23:48.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.479+0000 7f12f55e1700 1 --2- 172.21.15.16:0/1206078863 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12f00b5830 0x7f12f015a090 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f12e8009e50 tx=0x7f12e800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e35838ecdca53532 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:48.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.480+0000 7f12ef7fe700 1 -- 172.21.15.16:0/1206078863 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e800e070 con 0x7f12f00b5830 2023-10-16T12:23:48.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.480+0000 7f12ef7fe700 1 -- 172.21.15.16:0/1206078863 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12e800b910 con 0x7f12f00b5830 2023-10-16T12:23:48.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.480+0000 7f12ef7fe700 1 -- 172.21.15.16:0/1206078863 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e800bcc0 con 0x7f12f00b5830 2023-10-16T12:23:48.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.480+0000 7f12f7044700 1 -- 172.21.15.16:0/1206078863 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12f00b5830 msgr2=0x7f12f015a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:48.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.480+0000 7f12f7044700 1 --2- 172.21.15.16:0/1206078863 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12f00b5830 0x7f12f015a090 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f12e8009e50 tx=0x7f12e800b330 comp rx=0 tx=0).stop 2023-10-16T12:23:48.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.480+0000 7f12f7044700 1 -- 172.21.15.16:0/1206078863 shutdown_connections 2023-10-16T12:23:48.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.480+0000 7f12f7044700 1 --2- 172.21.15.16:0/1206078863 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12f00b5830 0x7f12f015a090 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:48.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.480+0000 7f12f7044700 1 --2- 172.21.15.16:0/1206078863 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f12f00cf880 0x7f12f00b52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:48.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.480+0000 7f12f7044700 1 -- 172.21.15.16:0/1206078863 >> 172.21.15.16:0/1206078863 conn(0x7f12f00b1fd0 msgr2=0x7f12f00b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:48.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.480+0000 7f12f7044700 1 -- 172.21.15.16:0/1206078863 shutdown_connections 2023-10-16T12:23:48.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.480+0000 7f12f7044700 1 -- 172.21.15.16:0/1206078863 wait complete. 2023-10-16T12:23:48.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.482+0000 7f12f7044700 1 Processor -- start 2023-10-16T12:23:48.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.482+0000 7f12f7044700 1 -- start start 2023-10-16T12:23:48.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.482+0000 7f12f7044700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f12f00b5830 0x7f12f0156370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:48.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.483+0000 7f12f7044700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12f00cf4a0 0x7f12f0155a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:48.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.483+0000 7f12f7044700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f12f00cf880 0x7f12f015cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:48.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.483+0000 7f12f7044700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12f01f4180 con 0x7f12f00cf4a0 2023-10-16T12:23:48.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.483+0000 7f12f7044700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12f0155f70 con 0x7f12f00b5830 2023-10-16T12:23:48.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.483+0000 7f12f7044700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12f00c6ce0 con 0x7f12f00cf880 2023-10-16T12:23:48.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.483+0000 7f12effff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12f00cf4a0 0x7f12f0155a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:48.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.483+0000 7f12effff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12f00cf4a0 0x7f12f0155a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37228/0 (socket says 172.21.15.16:37228) 2023-10-16T12:23:48.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.483+0000 7f12effff700 1 -- 172.21.15.16:0/3394091644 learned_addr learned my addr 172.21.15.16:0/3394091644 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:48.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.483+0000 7f12f55e1700 1 --2- 172.21.15.16:0/3394091644 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f12f00cf880 0x7f12f015cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:48.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.483+0000 7f12f4de0700 1 --2- 172.21.15.16:0/3394091644 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f12f00b5830 0x7f12f0156370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:48.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.483+0000 7f12f55e1700 1 -- 172.21.15.16:0/3394091644 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f12f00b5830 msgr2=0x7f12f0156370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:48.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.483+0000 7f12f55e1700 1 --2- 172.21.15.16:0/3394091644 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f12f00b5830 0x7f12f0156370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:48.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.483+0000 7f12f55e1700 1 -- 172.21.15.16:0/3394091644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12f00cf4a0 msgr2=0x7f12f0155a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:48.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.483+0000 7f12f55e1700 1 --2- 172.21.15.16:0/3394091644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12f00cf4a0 0x7f12f0155a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:48.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.483+0000 7f12f55e1700 1 -- 172.21.15.16:0/3394091644 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12e8009b30 con 0x7f12f00cf880 2023-10-16T12:23:48.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.483+0000 7f12effff700 1 --2- 172.21.15.16:0/3394091644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12f00cf4a0 0x7f12f0155a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:23:48.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.483+0000 7f12f55e1700 1 --2- 172.21.15.16:0/3394091644 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f12f00cf880 0x7f12f015cb80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f12e8015000 tx=0x7f12e8004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:48.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.483+0000 7f12ed7fa700 1 -- 172.21.15.16:0/3394091644 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e8003ea0 con 0x7f12f00cf880 2023-10-16T12:23:48.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.483+0000 7f12ed7fa700 1 -- 172.21.15.16:0/3394091644 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12e80048e0 con 0x7f12f00cf880 2023-10-16T12:23:48.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.483+0000 7f12f7044700 1 -- 172.21.15.16:0/3394091644 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12f01f4360 con 0x7f12f00cf880 2023-10-16T12:23:48.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.483+0000 7f12ed7fa700 1 -- 172.21.15.16:0/3394091644 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e801c680 con 0x7f12f00cf880 2023-10-16T12:23:48.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.484+0000 7f12f7044700 1 -- 172.21.15.16:0/3394091644 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12f01f4920 con 0x7f12f00cf880 2023-10-16T12:23:48.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.484+0000 7f12f7044700 1 -- 172.21.15.16:0/3394091644 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12f004ebb0 con 0x7f12f00cf880 2023-10-16T12:23:48.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.485+0000 7f12ed7fa700 1 -- 172.21.15.16:0/3394091644 <== mon.2 v2:172.21.15.49:3301/0 4 ==== mgrmap(e 14) v1 ==== 89400+0+0 (secure 0 0 0) 0x7f12e801c7e0 con 0x7f12f00cf880 2023-10-16T12:23:48.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.485+0000 7f12ed7fa700 1 --2- 172.21.15.16:0/3394091644 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f12d406b0e0 0x7f12d406d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:48.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.485+0000 7f12f4de0700 1 -- 172.21.15.16:0/3394091644 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f12d406b0e0 msgr2=0x7f12d406d5a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1707126831 2023-10-16T12:23:48.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.485+0000 7f12ed7fa700 1 -- 172.21.15.16:0/3394091644 <== mon.2 v2:172.21.15.49:3301/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6740+0+0 (secure 0 0 0) 0x7f12e808b610 con 0x7f12f00cf880 2023-10-16T12:23:48.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.485+0000 7f12f4de0700 1 --2- 172.21.15.16:0/3394091644 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f12d406b0e0 0x7f12d406d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:23:48.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.490+0000 7f12ed7fa700 1 -- 172.21.15.16:0/3394091644 <== mon.2 v2:172.21.15.49:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f12e8055f90 con 0x7f12f00cf880 2023-10-16T12:23:48.605 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:23:48.604+0000 7fb59d613200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T12:23:48.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.686+0000 7f12f4de0700 1 -- 172.21.15.16:0/3394091644 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f12d406b0e0 msgr2=0x7f12d406d5a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1707126831 2023-10-16T12:23:48.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.686+0000 7f12f4de0700 1 --2- 172.21.15.16:0/3394091644 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f12d406b0e0 0x7f12d406d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:23:48.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.738+0000 7f12f7044700 1 -- 172.21.15.16:0/3394091644 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f12f006b6d0 con 0x7f12f00cf880 2023-10-16T12:23:48.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.739+0000 7f12ed7fa700 1 -- 172.21.15.16:0/3394091644 <== mon.2 v2:172.21.15.49:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+4910 (secure 0 0 0) 0x7f12e80595b0 con 0x7f12f00cf880 2023-10-16T12:23:48.741 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:23:48.741 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":18,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5194,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/266676368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":266676368},{"type":"v1","addr":"172.21.15.49:6834","nonce":266676368}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5268,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/1460528697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":1460528697},{"type":"v1","addr":"172.21.15.16:6837","nonce":1460528697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/2374376752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2374376752},{"type":"v1","addr":"172.21.15.16:6836","nonce":2374376752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:12:00.537600+0000","modified":"2023-10-16T12:12:03.538619+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5283},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5283":{"gid":5283,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.49:6835/2788396302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":2788396302},{"type":"v1","addr":"172.21.15.49:6835","nonce":2788396302}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-16T12:23:48.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.741+0000 7f12f7044700 1 -- 172.21.15.16:0/3394091644 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f12d406b0e0 msgr2=0x7f12d406d5a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:23:48.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.741+0000 7f12f7044700 1 --2- 172.21.15.16:0/3394091644 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f12d406b0e0 0x7f12d406d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:48.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.741+0000 7f12f7044700 1 -- 172.21.15.16:0/3394091644 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f12f00cf880 msgr2=0x7f12f015cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:48.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.741+0000 7f12f7044700 1 --2- 172.21.15.16:0/3394091644 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f12f00cf880 0x7f12f015cb80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f12e8015000 tx=0x7f12e8004ad0 comp rx=0 tx=0).stop 2023-10-16T12:23:48.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.741+0000 7f12f7044700 1 -- 172.21.15.16:0/3394091644 shutdown_connections 2023-10-16T12:23:48.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.741+0000 7f12f7044700 1 --2- 172.21.15.16:0/3394091644 >> [v2:172.21.15.16:6832/1707126831,v1:172.21.15.16:6833/1707126831] conn(0x7f12d406b0e0 0x7f12d406d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:48.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.741+0000 7f12f7044700 1 --2- 172.21.15.16:0/3394091644 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f12f00cf880 0x7f12f015cb80 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:48.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.741+0000 7f12f7044700 1 --2- 172.21.15.16:0/3394091644 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f12f00b5830 0x7f12f0156370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:48.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.741+0000 7f12f7044700 1 --2- 172.21.15.16:0/3394091644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12f00cf4a0 0x7f12f0155a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:48.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.741+0000 7f12f7044700 1 -- 172.21.15.16:0/3394091644 >> 172.21.15.16:0/3394091644 conn(0x7f12f00b1fd0 msgr2=0x7f12f00b0ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:48.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.741+0000 7f12f7044700 1 -- 172.21.15.16:0/3394091644 shutdown_connections 2023-10-16T12:23:48.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.741+0000 7f12f7044700 1 -- 172.21.15.16:0/3394091644 wait complete. 2023-10-16T12:23:48.746 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 18 2023-10-16T12:23:48.756 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:23:48.829 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:23:48.827+0000 7f98e4adb200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T12:23:48.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.886+0000 7f4dd5da9700 1 Processor -- start 2023-10-16T12:23:48.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.886+0000 7f4dd5da9700 1 -- start start 2023-10-16T12:23:48.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.887+0000 7f4dd5da9700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f4dd015b630 0x7f4dd00d0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:48.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.887+0000 7f4dd5da9700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4dd015ba10 0x7f4dd00cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:48.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.887+0000 7f4dd5da9700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4dd00cdde0 0x7f4dd00ce230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:48.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.887+0000 7f4dd5da9700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dd00ce770 con 0x7f4dd00cdde0 2023-10-16T12:23:48.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.887+0000 7f4dd5da9700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dd00b0970 con 0x7f4dd015b630 2023-10-16T12:23:48.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.887+0000 7f4dd5da9700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd00cd690 con 0x7f4dd015ba10 2023-10-16T12:23:48.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.887+0000 7f4dcffff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4dd00cdde0 0x7f4dd00ce230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:48.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.887+0000 7f4dcffff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4dd00cdde0 0x7f4dd00ce230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37258/0 (socket says 172.21.15.16:37258) 2023-10-16T12:23:48.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.887+0000 7f4dcffff700 1 -- 172.21.15.16:0/3344912914 learned_addr learned my addr 172.21.15.16:0/3344912914 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:48.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.887+0000 7f4dcf7fe700 1 --2- 172.21.15.16:0/3344912914 >> v2:172.21.15.49:3300/0 conn(0x7f4dd015b630 0x7f4dd00d0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:48.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.887+0000 7f4dceffd700 1 --2- 172.21.15.16:0/3344912914 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4dd015ba10 0x7f4dd00cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:48.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.887+0000 7f4dcffff700 1 -- 172.21.15.16:0/3344912914 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4dd015ba10 msgr2=0x7f4dd00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:48.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.887+0000 7f4dcffff700 1 --2- 172.21.15.16:0/3344912914 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4dd015ba10 0x7f4dd00cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:48.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.887+0000 7f4dcffff700 1 -- 172.21.15.16:0/3344912914 >> v2:172.21.15.49:3300/0 conn(0x7f4dd015b630 msgr2=0x7f4dd00d0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:48.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.887+0000 7f4dcffff700 1 --2- 172.21.15.16:0/3344912914 >> v2:172.21.15.49:3300/0 conn(0x7f4dd015b630 0x7f4dd00d0380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:48.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.887+0000 7f4dcffff700 1 -- 172.21.15.16:0/3344912914 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dd00cd970 con 0x7f4dd00cdde0 2023-10-16T12:23:48.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.888+0000 7f4dcffff700 1 --2- 172.21.15.16:0/3344912914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dd00cdde0 0x7f4dd00ce230 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f4dd0067350 tx=0x7f4dc800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f08e85ae77c1991 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:48.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.888+0000 7f4dce7fc700 1 -- 172.21.15.16:0/3344912914 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc800e070 con 0x7f4dd00cdde0 2023-10-16T12:23:48.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.888+0000 7f4dce7fc700 1 -- 172.21.15.16:0/3344912914 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dc800b910 con 0x7f4dd00cdde0 2023-10-16T12:23:48.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.888+0000 7f4dce7fc700 1 -- 172.21.15.16:0/3344912914 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc800bcc0 con 0x7f4dd00cdde0 2023-10-16T12:23:48.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.888+0000 7f4dd5da9700 1 -- 172.21.15.16:0/3344912914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dd00cdde0 msgr2=0x7f4dd00ce230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:48.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.888+0000 7f4dd5da9700 1 --2- 172.21.15.16:0/3344912914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dd00cdde0 0x7f4dd00ce230 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f4dd0067350 tx=0x7f4dc800b330 comp rx=0 tx=0).stop 2023-10-16T12:23:48.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.888+0000 7f4dd5da9700 1 -- 172.21.15.16:0/3344912914 shutdown_connections 2023-10-16T12:23:48.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.888+0000 7f4dd5da9700 1 --2- 172.21.15.16:0/3344912914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dd00cdde0 0x7f4dd00ce230 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:48.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.888+0000 7f4dd5da9700 1 --2- 172.21.15.16:0/3344912914 >> v2:172.21.15.49:3300/0 conn(0x7f4dd015b630 0x7f4dd00d0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:48.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.888+0000 7f4dd5da9700 1 --2- 172.21.15.16:0/3344912914 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4dd015ba10 0x7f4dd00cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:48.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.888+0000 7f4dd5da9700 1 -- 172.21.15.16:0/3344912914 >> 172.21.15.16:0/3344912914 conn(0x7f4dd00d14b0 msgr2=0x7f4dd00d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:48.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.889+0000 7f4dd5da9700 1 -- 172.21.15.16:0/3344912914 shutdown_connections 2023-10-16T12:23:48.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.889+0000 7f4dd5da9700 1 -- 172.21.15.16:0/3344912914 wait complete. 2023-10-16T12:23:48.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.890+0000 7f4dd5da9700 1 Processor -- start 2023-10-16T12:23:48.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.890+0000 7f4dd5da9700 1 -- start start 2023-10-16T12:23:48.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.891+0000 7f4dd5da9700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4dd00cdde0 0x7f4dd00b6550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:48.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.891+0000 7f4dd5da9700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4dd015b630 0x7f4dd015e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:48.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.891+0000 7f4dd5da9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dd015ba10 0x7f4dd00af070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:48.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.891+0000 7f4dd5da9700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd00c94a0 con 0x7f4dd015ba10 2023-10-16T12:23:48.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.891+0000 7f4dd5da9700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd015ded0 con 0x7f4dd015b630 2023-10-16T12:23:48.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.891+0000 7f4dd5da9700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd00b6a90 con 0x7f4dd00cdde0 2023-10-16T12:23:48.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.891+0000 7f4dcffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dd015ba10 0x7f4dd00af070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:48.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.891+0000 7f4dcf7fe700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4dd00cdde0 0x7f4dd00b6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:48.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.891+0000 7f4dcf7fe700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4dd00cdde0 0x7f4dd00b6550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:53756/0 (socket says 172.21.15.16:53756) 2023-10-16T12:23:48.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.891+0000 7f4dcffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dd015ba10 0x7f4dd00af070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37270/0 (socket says 172.21.15.16:37270) 2023-10-16T12:23:48.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.891+0000 7f4dcf7fe700 1 -- 172.21.15.16:0/1638997405 learned_addr learned my addr 172.21.15.16:0/1638997405 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:48.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.891+0000 7f4dceffd700 1 --2- 172.21.15.16:0/1638997405 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4dd015b630 0x7f4dd015e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:48.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.892+0000 7f4dcf7fe700 1 -- 172.21.15.16:0/1638997405 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4dd015b630 msgr2=0x7f4dd015e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:48.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.892+0000 7f4dcf7fe700 1 --2- 172.21.15.16:0/1638997405 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4dd015b630 0x7f4dd015e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:48.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.892+0000 7f4dcf7fe700 1 -- 172.21.15.16:0/1638997405 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dd015ba10 msgr2=0x7f4dd00af070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:48.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.892+0000 7f4dcf7fe700 1 --2- 172.21.15.16:0/1638997405 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dd015ba10 0x7f4dd00af070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:48.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.892+0000 7f4dcf7fe700 1 -- 172.21.15.16:0/1638997405 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dc8009b30 con 0x7f4dd00cdde0 2023-10-16T12:23:48.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.892+0000 7f4dceffd700 1 --2- 172.21.15.16:0/1638997405 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4dd015b630 0x7f4dd015e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:23:48.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.892+0000 7f4dcffff700 1 --2- 172.21.15.16:0/1638997405 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dd015ba10 0x7f4dd00af070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:23:48.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.892+0000 7f4dcf7fe700 1 --2- 172.21.15.16:0/1638997405 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4dd00cdde0 0x7f4dd00b6550 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4dc000e960 tx=0x7f4dc000ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:48.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.892+0000 7f4db7fff700 1 -- 172.21.15.16:0/1638997405 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc0013680 con 0x7f4dd00cdde0 2023-10-16T12:23:48.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.892+0000 7f4db7fff700 1 -- 172.21.15.16:0/1638997405 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dc00137e0 con 0x7f4dd00cdde0 2023-10-16T12:23:48.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.892+0000 7f4dd5da9700 1 -- 172.21.15.16:0/1638997405 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dd00c8580 con 0x7f4dd00cdde0 2023-10-16T12:23:48.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.892+0000 7f4db7fff700 1 -- 172.21.15.16:0/1638997405 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc0013b70 con 0x7f4dd00cdde0 2023-10-16T12:23:48.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.892+0000 7f4dd5da9700 1 -- 172.21.15.16:0/1638997405 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dd00c8b00 con 0x7f4dd00cdde0 2023-10-16T12:23:48.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.893+0000 7f4dd5da9700 1 -- 172.21.15.16:0/1638997405 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dd004ebb0 con 0x7f4dd00cdde0 2023-10-16T12:23:48.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.894+0000 7f4db7fff700 1 -- 172.21.15.16:0/1638997405 <== mon.2 v2:172.21.15.49:3301/0 4 ==== mgrmap(e 15) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f4dc0013cd0 con 0x7f4dd00cdde0 2023-10-16T12:23:48.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.894+0000 7f4db7fff700 1 -- 172.21.15.16:0/1638997405 <== mon.2 v2:172.21.15.49:3301/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 7083+0+0 (secure 0 0 0) 0x7f4dc0089580 con 0x7f4dd00cdde0 2023-10-16T12:23:48.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:48.899+0000 7f4db7fff700 1 -- 172.21.15.16:0/1638997405 <== mon.2 v2:172.21.15.49:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4dc0054520 con 0x7f4dd00cdde0 2023-10-16T12:23:48.914 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:23:48.912+0000 7f985d2c1700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:23:48.914 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:23:48.912+0000 7f985d2c1700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:23:48.915 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:23:48.912+0000 7f985d2c1700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:23:48.915 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:23:48.912+0000 7f985d2c1700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:23:48.915 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:23:48.912+0000 7f985d2c1700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:23:48.915 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:23:48.912+0000 7f98602c7700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:23:48.916 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:23:48.912+0000 7f98602c7700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:23:48.916 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:23:48.912+0000 7f98602c7700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:23:48.916 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:23:48.912+0000 7f98602c7700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:23:48.916 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:23:48.912+0000 7f98602c7700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:23:49.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.156+0000 7f4dd5da9700 1 -- 172.21.15.16:0/1638997405 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4dd006b5d0 con 0x7f4dd00cdde0 2023-10-16T12:23:49.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.157+0000 7f4db7fff700 1 -- 172.21.15.16:0/1638997405 <== mon.2 v2:172.21.15.49:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+4910 (secure 0 0 0) 0x7f4dc0057b40 con 0x7f4dd00cdde0 2023-10-16T12:23:49.159 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:23:49.159 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":18,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5194,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/266676368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":266676368},{"type":"v1","addr":"172.21.15.49:6834","nonce":266676368}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5268,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/1460528697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":1460528697},{"type":"v1","addr":"172.21.15.16:6837","nonce":1460528697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/2374376752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2374376752},{"type":"v1","addr":"172.21.15.16:6836","nonce":2374376752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:12:00.537600+0000","modified":"2023-10-16T12:12:03.538619+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5283},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5283":{"gid":5283,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.49:6835/2788396302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":2788396302},{"type":"v1","addr":"172.21.15.49:6835","nonce":2788396302}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-16T12:23:49.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.159+0000 7f4dd5da9700 1 -- 172.21.15.16:0/1638997405 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4dd00cdde0 msgr2=0x7f4dd00b6550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:49.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.159+0000 7f4dd5da9700 1 --2- 172.21.15.16:0/1638997405 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4dd00cdde0 0x7f4dd00b6550 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4dc000e960 tx=0x7f4dc000ec70 comp rx=0 tx=0).stop 2023-10-16T12:23:49.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.159+0000 7f4dd5da9700 1 -- 172.21.15.16:0/1638997405 shutdown_connections 2023-10-16T12:23:49.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.159+0000 7f4dd5da9700 1 --2- 172.21.15.16:0/1638997405 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dd015ba10 0x7f4dd00af070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:49.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.159+0000 7f4dd5da9700 1 --2- 172.21.15.16:0/1638997405 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4dd00cdde0 0x7f4dd00b6550 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:49.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.159+0000 7f4dd5da9700 1 --2- 172.21.15.16:0/1638997405 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4dd015b630 0x7f4dd015e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:49.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.159+0000 7f4dd5da9700 1 -- 172.21.15.16:0/1638997405 >> 172.21.15.16:0/1638997405 conn(0x7f4dd00d14b0 msgr2=0x7f4dd015a920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:49.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.159+0000 7f4dd5da9700 1 -- 172.21.15.16:0/1638997405 shutdown_connections 2023-10-16T12:23:49.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.159+0000 7f4dd5da9700 1 -- 172.21.15.16:0/1638997405 wait complete. 2023-10-16T12:23:49.163 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 18 2023-10-16T12:23:49.173 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:23:49.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.297+0000 7fc123324700 1 Processor -- start 2023-10-16T12:23:49.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.297+0000 7fc123324700 1 -- start start 2023-10-16T12:23:49.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.298+0000 7fc123324700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc11c0cf880 0x7fc11c0b52f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:49.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.298+0000 7fc123324700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc11c15bc80 con 0x7fc11c0b5830 2023-10-16T12:23:49.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.298+0000 7fc123324700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc11c15b070 con 0x7fc11c0cf4a0 2023-10-16T12:23:49.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.298+0000 7fc123324700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc11c15b350 con 0x7fc11c0cf880 2023-10-16T12:23:49.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.298+0000 7fc1208bf700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc11c0cf880 0x7fc11c0b52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:49.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.298+0000 7fc1218c1700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc11c0b5830 0x7fc11c15a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54830/0 (socket says 172.21.15.16:54830) 2023-10-16T12:23:49.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.298+0000 7fc1218c1700 1 -- 172.21.15.16:0/3213025533 learned_addr learned my addr 172.21.15.16:0/3213025533 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:23:49.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.299+0000 7fc1208bf700 1 -- 172.21.15.16:0/3213025533 >> v1:172.21.15.49:6789/0 conn(0x7fc11c0cf4a0 legacy=0x7fc11c0726e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:49.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.299+0000 7fc1208bf700 1 -- 172.21.15.16:0/3213025533 >> v1:172.21.15.16:6789/0 conn(0x7fc11c0b5830 legacy=0x7fc11c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:49.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.299+0000 7fc113fff700 1 -- 172.21.15.16:0/3213025533 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3848278906 0 0) 0x7fc11c15bc80 con 0x7fc11c0b5830 2023-10-16T12:23:49.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.299+0000 7fc1208bf700 1 -- 172.21.15.16:0/3213025533 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc11c15b630 con 0x7fc11c0cf880 2023-10-16T12:23:49.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.299+0000 7fc1208bf700 1 --2- 172.21.15.16:0/3213025533 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc11c0cf880 0x7fc11c0b52f0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fc118009840 tx=0x7fc118009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4cf41aad2864316 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:49.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.299+0000 7fc113fff700 1 -- 172.21.15.16:0/3213025533 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc118011830 con 0x7fc11c0cf880 2023-10-16T12:23:49.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.299+0000 7fc113fff700 1 -- 172.21.15.16:0/3213025533 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc118011990 con 0x7fc11c0cf880 2023-10-16T12:23:49.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.299+0000 7fc113fff700 1 -- 172.21.15.16:0/3213025533 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc118011d20 con 0x7fc11c0cf880 2023-10-16T12:23:49.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.299+0000 7fc123324700 1 -- 172.21.15.16:0/3213025533 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc11c0cf880 msgr2=0x7fc11c0b52f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:49.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.299+0000 7fc123324700 1 --2- 172.21.15.16:0/3213025533 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc11c0cf880 0x7fc11c0b52f0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fc118009840 tx=0x7fc118009b50 comp rx=0 tx=0).stop 2023-10-16T12:23:49.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.299+0000 7fc123324700 1 -- 172.21.15.16:0/3213025533 shutdown_connections 2023-10-16T12:23:49.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.300+0000 7fc123324700 1 --2- 172.21.15.16:0/3213025533 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc11c0cf880 0x7fc11c0b52f0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:49.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.300+0000 7fc123324700 1 -- 172.21.15.16:0/3213025533 >> 172.21.15.16:0/3213025533 conn(0x7fc11c0b1fd0 msgr2=0x7fc11c0b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:49.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.300+0000 7fc123324700 1 -- 172.21.15.16:0/3213025533 shutdown_connections 2023-10-16T12:23:49.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.300+0000 7fc123324700 1 -- 172.21.15.16:0/3213025533 wait complete. 2023-10-16T12:23:49.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.301+0000 7fc123324700 1 Processor -- start 2023-10-16T12:23:49.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.301+0000 7fc123324700 1 -- start start 2023-10-16T12:23:49.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.302+0000 7fc123324700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc11c0b5830 0x7fc11c0a8f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:49.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.302+0000 7fc123324700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc11c0cf4a0 0x7fc11c1f1200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:49.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.302+0000 7fc123324700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc11c0cf880 0x7fc11c0c5960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:49.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.302+0000 7fc123324700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc11c1f2210 con 0x7fc11c0cf4a0 2023-10-16T12:23:49.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.302+0000 7fc123324700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc11c1f1c50 con 0x7fc11c0b5830 2023-10-16T12:23:49.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.302+0000 7fc123324700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc11c1f1f30 con 0x7fc11c0cf880 2023-10-16T12:23:49.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.302+0000 7fc1208bf700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc11c0cf4a0 0x7fc11c1f1200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:49.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.302+0000 7fc1208bf700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc11c0cf4a0 0x7fc11c1f1200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37328/0 (socket says 172.21.15.16:37328) 2023-10-16T12:23:49.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.302+0000 7fc1208bf700 1 -- 172.21.15.16:0/82658346 learned_addr learned my addr 172.21.15.16:0/82658346 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:49.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.302+0000 7fc1210c0700 1 --2- 172.21.15.16:0/82658346 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc11c0b5830 0x7fc11c0a8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:49.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.302+0000 7fc1218c1700 1 --2- 172.21.15.16:0/82658346 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc11c0cf880 0x7fc11c0c5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:49.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.303+0000 7fc1208bf700 1 -- 172.21.15.16:0/82658346 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc11c0cf880 msgr2=0x7fc11c0c5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:49.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.303+0000 7fc1208bf700 1 --2- 172.21.15.16:0/82658346 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc11c0cf880 0x7fc11c0c5960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:49.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.303+0000 7fc1208bf700 1 -- 172.21.15.16:0/82658346 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc11c0b5830 msgr2=0x7fc11c0a8f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:49.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.303+0000 7fc1208bf700 1 --2- 172.21.15.16:0/82658346 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc11c0b5830 0x7fc11c0a8f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:49.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.303+0000 7fc1208bf700 1 -- 172.21.15.16:0/82658346 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1180094a0 con 0x7fc11c0cf4a0 2023-10-16T12:23:49.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.303+0000 7fc1210c0700 1 --2- 172.21.15.16:0/82658346 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc11c0b5830 0x7fc11c0a8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:23:49.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.303+0000 7fc1208bf700 1 --2- 172.21.15.16:0/82658346 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc11c0cf4a0 0x7fc11c1f1200 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fc118014040 tx=0x7fc118004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:49.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.303+0000 7fc111ffb700 1 -- 172.21.15.16:0/82658346 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc11801a900 con 0x7fc11c0cf4a0 2023-10-16T12:23:49.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.303+0000 7fc111ffb700 1 -- 172.21.15.16:0/82658346 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc11801aa60 con 0x7fc11c0cf4a0 2023-10-16T12:23:49.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.303+0000 7fc123324700 1 -- 172.21.15.16:0/82658346 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc11c0c5160 con 0x7fc11c0cf4a0 2023-10-16T12:23:49.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.303+0000 7fc111ffb700 1 -- 172.21.15.16:0/82658346 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc11801add0 con 0x7fc11c0cf4a0 2023-10-16T12:23:49.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.303+0000 7fc123324700 1 -- 172.21.15.16:0/82658346 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc11c0c3290 con 0x7fc11c0cf4a0 2023-10-16T12:23:49.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.304+0000 7fc123324700 1 -- 172.21.15.16:0/82658346 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc11c0c4fc0 con 0x7fc11c0cf4a0 2023-10-16T12:23:49.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.305+0000 7fc111ffb700 1 -- 172.21.15.16:0/82658346 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 15) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fc118022070 con 0x7fc11c0cf4a0 2023-10-16T12:23:49.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.305+0000 7fc111ffb700 1 -- 172.21.15.16:0/82658346 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 7083+0+0 (secure 0 0 0) 0x7fc11808a8a0 con 0x7fc11c0cf4a0 2023-10-16T12:23:49.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.310+0000 7fc111ffb700 1 -- 172.21.15.16:0/82658346 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc11c0c4fc0 con 0x7fc11c0cf4a0 2023-10-16T12:23:49.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.515+0000 7fc123324700 1 -- 172.21.15.16:0/82658346 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc11c15b8f0 con 0x7fc11c0cf4a0 2023-10-16T12:23:49.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.516+0000 7fc111ffb700 1 -- 172.21.15.16:0/82658346 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) v1 ==== 74+0+18194 (secure 0 0 0) 0x7fc11c15b8f0 con 0x7fc11c0cf4a0 2023-10-16T12:23:49.519 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:23:49.519 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":56,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:23:48.841881+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:11:58.700868+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2023-10-16T12:11:59.495384+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":221,"snap_epoch":55,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.7","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/1886125045":"2023-10-17T12:23:48.841815+0000","172.21.15.16:0/2241612272":"2023-10-17T12:23:48.841815+0000","172.21.15.16:0/2779790691":"2023-10-17T12:23:48.841815+0000","172.21.15.16:0/2158072405":"2023-10-17T12:23:48.841815+0000","172.21.15.16:6833/1707126831":"2023-10-17T12:23:48.841815+0000","172.21.15.16:0/1802260224":"2023-10-17T12:23:48.841815+0000","172.21.15.16:6833/365921436":"2023-10-17T12:12:15.525647+0000","172.21.15.16:0/1893346008":"2023-10-17T12:12:15.525647+0000","172.21.15.16:6832/1707126831":"2023-10-17T12:23:48.841815+0000","172.21.15.16:0/3547353312":"2023-10-17T12:12:15.525647+0000","172.21.15.16:6832/365921436":"2023-10-17T12:12:15.525647+0000","172.21.15.16:0/109549027":"2023-10-17T12:23:48.841815+0000","172.21.15.16:0/3754942648":"2023-10-17T12:23:48.841815+0000","172.21.15.16:0/675356848":"2023-10-17T12:12:15.525647+0000","172.21.15.16:0/1300148187":"2023-10-17T12:12:15.525647+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":7,"length":1},{"begin":14,"length":1},{"begin":18,"length":1},{"begin":24,"length":1},{"begin":28,"length":1},{"begin":35,"length":1},{"begin":39,"length":1},{"begin":46,"length":1},{"begin":50,"length":1},{"begin":56,"length":1},{"begin":60,"length":1},{"begin":67,"length":1},{"begin":71,"length":2},{"begin":78,"length":1},{"begin":83,"length":1},{"begin":89,"length":1},{"begin":93,"length":1},{"begin":99,"length":1},{"begin":104,"length":1},{"begin":106,"length":1},{"begin":110,"length":1},{"begin":199,"length":23}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:23:49.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.518+0000 7fc123324700 1 -- 172.21.15.16:0/82658346 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc11c0cf4a0 msgr2=0x7fc11c1f1200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:49.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.518+0000 7fc123324700 1 --2- 172.21.15.16:0/82658346 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc11c0cf4a0 0x7fc11c1f1200 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fc118014040 tx=0x7fc118004750 comp rx=0 tx=0).stop 2023-10-16T12:23:49.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.519+0000 7fc123324700 1 -- 172.21.15.16:0/82658346 shutdown_connections 2023-10-16T12:23:49.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.519+0000 7fc123324700 1 --2- 172.21.15.16:0/82658346 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc11c0cf880 0x7fc11c0c5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:49.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.519+0000 7fc123324700 1 --2- 172.21.15.16:0/82658346 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc11c0b5830 0x7fc11c0a8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:49.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.519+0000 7fc123324700 1 --2- 172.21.15.16:0/82658346 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc11c0cf4a0 0x7fc11c1f1200 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:49.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.519+0000 7fc123324700 1 -- 172.21.15.16:0/82658346 >> 172.21.15.16:0/82658346 conn(0x7fc11c0b1fd0 msgr2=0x7fc11c0b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:49.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.519+0000 7fc123324700 1 -- 172.21.15.16:0/82658346 shutdown_connections 2023-10-16T12:23:49.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.519+0000 7fc123324700 1 -- 172.21.15.16:0/82658346 wait complete. 2023-10-16T12:23:49.533 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-16T12:23:49.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.657+0000 7fd36a379700 1 Processor -- start 2023-10-16T12:23:49.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.657+0000 7fd36a379700 1 -- start start 2023-10-16T12:23:49.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.658+0000 7fd36a379700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fd36415b860 0x7fd36406a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:49.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.658+0000 7fd36a379700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3640d11c0 con 0x7fd3640b2be0 2023-10-16T12:23:49.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.658+0000 7fd36a379700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd364072800 con 0x7fd3640b4490 2023-10-16T12:23:49.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.658+0000 7fd36a379700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3640d03d0 con 0x7fd36415b860 2023-10-16T12:23:49.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.658+0000 7fd3637fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd3640b2be0 0x7fd3640719e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54838/0 (socket says 172.21.15.16:54838) 2023-10-16T12:23:49.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.658+0000 7fd3637fe700 1 -- 172.21.15.16:0/3526099803 learned_addr learned my addr 172.21.15.16:0/3526099803 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:23:49.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.659+0000 7fd363fff700 1 --2- 172.21.15.16:0/3526099803 >> v2:172.21.15.49:3300/0 conn(0x7fd36415b860 0x7fd36406a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:49.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.659+0000 7fd362ffd700 1 -- 172.21.15.16:0/3526099803 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 217127156 0 0) 0x7fd3640d11c0 con 0x7fd3640b2be0 2023-10-16T12:23:49.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.659+0000 7fd362ffd700 1 -- 172.21.15.16:0/3526099803 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3500035b0 con 0x7fd3640b2be0 2023-10-16T12:23:49.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.659+0000 7fd362ffd700 1 -- 172.21.15.16:0/3526099803 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3868835131 0 0) 0x7fd3500035b0 con 0x7fd3640b2be0 2023-10-16T12:23:49.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.659+0000 7fd362ffd700 1 -- 172.21.15.16:0/3526099803 >> v2:172.21.15.49:3300/0 conn(0x7fd36415b860 msgr2=0x7fd36406a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:49.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.659+0000 7fd362ffd700 1 --2- 172.21.15.16:0/3526099803 >> v2:172.21.15.49:3300/0 conn(0x7fd36415b860 0x7fd36406a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:49.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.659+0000 7fd362ffd700 1 -- 172.21.15.16:0/3526099803 >> v1:172.21.15.49:6789/0 conn(0x7fd3640b4490 legacy=0x7fd3640720f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:49.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.659+0000 7fd362ffd700 1 -- 172.21.15.16:0/3526099803 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3640cd360 con 0x7fd3640b2be0 2023-10-16T12:23:49.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.659+0000 7fd362ffd700 1 -- 172.21.15.16:0/3526099803 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7fd34c001670 con 0x7fd3640b2be0 2023-10-16T12:23:49.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.659+0000 7fd362ffd700 1 -- 172.21.15.16:0/3526099803 >> v1:172.21.15.16:6789/0 conn(0x7fd3640b2be0 legacy=0x7fd3640719e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:49.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.659+0000 7fd362ffd700 1 --2- 172.21.15.16:0/3526099803 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd350003a00 0x7fd350003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:49.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.659+0000 7fd362ffd700 1 --2- 172.21.15.16:0/3526099803 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd350004850 0x7fd350004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:49.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.659+0000 7fd362ffd700 1 --2- 172.21.15.16:0/3526099803 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd350005210 0x7fd35000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:49.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.659+0000 7fd362ffd700 1 -- 172.21.15.16:0/3526099803 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3640d03d0 con 0x7fd350003a00 2023-10-16T12:23:49.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.659+0000 7fd362ffd700 1 -- 172.21.15.16:0/3526099803 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd35000c0b0 con 0x7fd350004850 2023-10-16T12:23:49.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.659+0000 7fd362ffd700 1 -- 172.21.15.16:0/3526099803 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd35000c270 con 0x7fd350005210 2023-10-16T12:23:49.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.659+0000 7fd363fff700 1 --2- 172.21.15.16:0/3526099803 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd350003a00 0x7fd350003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:49.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.659+0000 7fd3637fe700 1 --2- 172.21.15.16:0/3526099803 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd350004850 0x7fd350004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:49.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.659+0000 7fd368916700 1 --2- 172.21.15.16:0/3526099803 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd350005210 0x7fd35000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:49.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.659+0000 7fd363fff700 1 -- 172.21.15.16:0/3526099803 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd350005210 msgr2=0x7fd35000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:49.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.659+0000 7fd363fff700 1 --2- 172.21.15.16:0/3526099803 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd350005210 0x7fd35000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:49.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.659+0000 7fd363fff700 1 -- 172.21.15.16:0/3526099803 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd350004850 msgr2=0x7fd350004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:49.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.659+0000 7fd363fff700 1 --2- 172.21.15.16:0/3526099803 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd350004850 0x7fd350004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:49.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.659+0000 7fd363fff700 1 -- 172.21.15.16:0/3526099803 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3500035b0 con 0x7fd350003a00 2023-10-16T12:23:49.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.659+0000 7fd368916700 1 -- 172.21.15.16:0/3526099803 reap_dead start 2023-10-16T12:23:49.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.660+0000 7fd363fff700 1 --2- 172.21.15.16:0/3526099803 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd350003a00 0x7fd350003e20 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fd354009b60 tx=0x7fd350005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60fa4427993995a3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:49.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.660+0000 7fd362ffd700 1 -- 172.21.15.16:0/3526099803 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd35400e6c0 con 0x7fd350003a00 2023-10-16T12:23:49.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.660+0000 7fd362ffd700 1 -- 172.21.15.16:0/3526099803 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd35400e820 con 0x7fd350003a00 2023-10-16T12:23:49.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.660+0000 7fd362ffd700 1 -- 172.21.15.16:0/3526099803 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd35400ebb0 con 0x7fd350003a00 2023-10-16T12:23:49.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.660+0000 7fd36a379700 1 -- 172.21.15.16:0/3526099803 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd350003a00 msgr2=0x7fd350003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:49.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.660+0000 7fd36a379700 1 --2- 172.21.15.16:0/3526099803 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd350003a00 0x7fd350003e20 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fd354009b60 tx=0x7fd350005880 comp rx=0 tx=0).stop 2023-10-16T12:23:49.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.660+0000 7fd36a379700 1 -- 172.21.15.16:0/3526099803 shutdown_connections 2023-10-16T12:23:49.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.660+0000 7fd36a379700 1 --2- 172.21.15.16:0/3526099803 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd350003a00 0x7fd350003e20 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:49.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.660+0000 7fd36a379700 1 -- 172.21.15.16:0/3526099803 >> 172.21.15.16:0/3526099803 conn(0x7fd3640b1fd0 msgr2=0x7fd3640b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:49.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.660+0000 7fd36a379700 1 -- 172.21.15.16:0/3526099803 shutdown_connections 2023-10-16T12:23:49.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.660+0000 7fd36a379700 1 -- 172.21.15.16:0/3526099803 wait complete. 2023-10-16T12:23:49.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.662+0000 7fd36a379700 1 Processor -- start 2023-10-16T12:23:49.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.662+0000 7fd36a379700 1 -- start start 2023-10-16T12:23:49.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.662+0000 7fd36a379700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd350003a00 0x7fd3640c7510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:49.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.662+0000 7fd36a379700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd36415dbf0 0x7fd3640c9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:49.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.662+0000 7fd36a379700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd36415cdc0 0x7fd3641f4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:49.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.662+0000 7fd36a379700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3641f4290 con 0x7fd36415dbf0 2023-10-16T12:23:49.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.662+0000 7fd36a379700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3640c8050 con 0x7fd36415cdc0 2023-10-16T12:23:49.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.662+0000 7fd36a379700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3640c8330 con 0x7fd350003a00 2023-10-16T12:23:49.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.663+0000 7fd3637fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd36415dbf0 0x7fd3640c9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:49.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.663+0000 7fd3637fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd36415dbf0 0x7fd3640c9ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37346/0 (socket says 172.21.15.16:37346) 2023-10-16T12:23:49.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.663+0000 7fd3637fe700 1 -- 172.21.15.16:0/389993701 learned_addr learned my addr 172.21.15.16:0/389993701 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:49.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.663+0000 7fd368916700 1 --2- 172.21.15.16:0/389993701 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd36415cdc0 0x7fd3641f4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:49.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.663+0000 7fd363fff700 1 --2- 172.21.15.16:0/389993701 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd350003a00 0x7fd3640c7510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:49.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.663+0000 7fd3637fe700 1 -- 172.21.15.16:0/389993701 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd350003a00 msgr2=0x7fd3640c7510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:49.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.663+0000 7fd3637fe700 1 --2- 172.21.15.16:0/389993701 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd350003a00 0x7fd3640c7510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:49.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.663+0000 7fd3637fe700 1 -- 172.21.15.16:0/389993701 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd36415cdc0 msgr2=0x7fd3641f4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:49.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.663+0000 7fd3637fe700 1 --2- 172.21.15.16:0/389993701 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd36415cdc0 0x7fd3641f4720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:49.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.663+0000 7fd3637fe700 1 -- 172.21.15.16:0/389993701 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3640c8610 con 0x7fd36415dbf0 2023-10-16T12:23:49.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.663+0000 7fd363fff700 1 --2- 172.21.15.16:0/389993701 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd350003a00 0x7fd3640c7510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:23:49.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.663+0000 7fd3637fe700 1 --2- 172.21.15.16:0/389993701 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd36415dbf0 0x7fd3640c9ed0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fd34c007290 tx=0x7fd34c0075a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:49.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.663+0000 7fd360ff9700 1 -- 172.21.15.16:0/389993701 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd34c00f660 con 0x7fd36415dbf0 2023-10-16T12:23:49.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.663+0000 7fd360ff9700 1 -- 172.21.15.16:0/389993701 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd34c00f7e0 con 0x7fd36415dbf0 2023-10-16T12:23:49.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.663+0000 7fd36a379700 1 -- 172.21.15.16:0/389993701 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3640ca410 con 0x7fd36415dbf0 2023-10-16T12:23:49.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.663+0000 7fd360ff9700 1 -- 172.21.15.16:0/389993701 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd34c00fb90 con 0x7fd36415dbf0 2023-10-16T12:23:49.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.663+0000 7fd36a379700 1 -- 172.21.15.16:0/389993701 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3640c88f0 con 0x7fd36415dbf0 2023-10-16T12:23:49.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.664+0000 7fd36a379700 1 -- 172.21.15.16:0/389993701 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd36404ebb0 con 0x7fd36415dbf0 2023-10-16T12:23:49.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.665+0000 7fd360ff9700 1 -- 172.21.15.16:0/389993701 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 15) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fd34c00fcf0 con 0x7fd36415dbf0 2023-10-16T12:23:49.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.665+0000 7fd360ff9700 1 -- 172.21.15.16:0/389993701 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 7083+0+0 (secure 0 0 0) 0x7fd34c089560 con 0x7fd36415dbf0 2023-10-16T12:23:49.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.670+0000 7fd360ff9700 1 -- 172.21.15.16:0/389993701 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd34c054500 con 0x7fd36415dbf0 2023-10-16T12:23:49.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.851+0000 7fd360ff9700 1 -- 172.21.15.16:0/389993701 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 16) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fd34c053fb0 con 0x7fd36415dbf0 2023-10-16T12:23:49.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.852+0000 7fd360ff9700 1 --2- 172.21.15.16:0/389993701 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fd344071100 0x7fd3440734f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:49.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.853+0000 7fd363fff700 1 --2- 172.21.15.16:0/389993701 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fd344071100 0x7fd3440734f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:49.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.855+0000 7fd363fff700 1 --2- 172.21.15.16:0/389993701 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fd344071100 0x7fd3440734f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd3540090b0 tx=0x7fd35401e040 comp rx=0 tx=0).ready entity=mgr.5874 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:49.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.924+0000 7fd36a379700 1 -- 172.21.15.16:0/389993701 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fd364068ad0 con 0x7fd36415dbf0 2023-10-16T12:23:49.927 INFO:tasks.ceph.mon.a.smithi016.stderr:2023-10-16T12:23:49.925+0000 7f9fc8535700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-16T12:23:49.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.926+0000 7fd360ff9700 1 -- 172.21.15.16:0/389993701 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v19) v1 ==== 161+0+0 (secure 0 0 0) 0x7fd34c070410 con 0x7fd36415dbf0 2023-10-16T12:23:49.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.929+0000 7fd36a379700 1 -- 172.21.15.16:0/389993701 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fd344071100 msgr2=0x7fd3440734f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:49.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.929+0000 7fd36a379700 1 --2- 172.21.15.16:0/389993701 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fd344071100 0x7fd3440734f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd3540090b0 tx=0x7fd35401e040 comp rx=0 tx=0).stop 2023-10-16T12:23:49.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.929+0000 7fd36a379700 1 -- 172.21.15.16:0/389993701 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd36415dbf0 msgr2=0x7fd3640c9ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:49.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.929+0000 7fd36a379700 1 --2- 172.21.15.16:0/389993701 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd36415dbf0 0x7fd3640c9ed0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fd34c007290 tx=0x7fd34c0075a0 comp rx=0 tx=0).stop 2023-10-16T12:23:49.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.929+0000 7fd36a379700 1 -- 172.21.15.16:0/389993701 shutdown_connections 2023-10-16T12:23:49.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.929+0000 7fd36a379700 1 --2- 172.21.15.16:0/389993701 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fd344071100 0x7fd3440734f0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:49.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.929+0000 7fd36a379700 1 --2- 172.21.15.16:0/389993701 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd36415cdc0 0x7fd3641f4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:49.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.929+0000 7fd36a379700 1 --2- 172.21.15.16:0/389993701 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd350003a00 0x7fd3640c7510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:49.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.929+0000 7fd36a379700 1 --2- 172.21.15.16:0/389993701 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd36415dbf0 0x7fd3640c9ed0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:49.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.929+0000 7fd36a379700 1 -- 172.21.15.16:0/389993701 >> 172.21.15.16:0/389993701 conn(0x7fd3640b1fd0 msgr2=0x7fd3640b49e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:49.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.929+0000 7fd36a379700 1 -- 172.21.15.16:0/389993701 shutdown_connections 2023-10-16T12:23:49.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:49.930+0000 7fd36a379700 1 -- 172.21.15.16:0/389993701 wait complete. 2023-10-16T12:23:49.934 INFO:teuthology.orchestra.run.smithi016.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-16T12:23:49.944 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-16T12:23:49.970 INFO:tasks.ceph.mds.b.smithi049.stdout:starting mds.b at 2023-10-16T12:23:50.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:50.069+0000 7f1b90a0d700 1 Processor -- start 2023-10-16T12:23:50.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:50.069+0000 7f1b90a0d700 1 -- start start 2023-10-16T12:23:50.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:50.070+0000 7f1b90a0d700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f1b8c15b2c0 0x7f1b8c161340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:50.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:50.070+0000 7f1b90a0d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1b8c06a7e0 0x7f1b8c06ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:50.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:50.070+0000 7f1b90a0d700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b8c15ecb0 con 0x7f1b8c06e570 2023-10-16T12:23:50.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:50.070+0000 7f1b90a0d700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b8c06b1b0 con 0x7f1b8c15b2c0 2023-10-16T12:23:50.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:50.070+0000 7f1b90a0d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b8c161880 con 0x7f1b8c06a7e0 2023-10-16T12:23:50.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:50.070+0000 7f1b8ad9d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1b8c06a7e0 0x7f1b8c06ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:50.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:50.071+0000 7f1b8a59c700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f1b8c15b2c0 0x7f1b8c161340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:50.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:50.071+0000 7f1b8ad9d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1b8c06a7e0 0x7f1b8c06ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:53866/0 (socket says 172.21.15.16:53866) 2023-10-16T12:23:50.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:50.071+0000 7f1b89d9b700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7f1b8c06e570 0x7f1b8c15e5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:33830/0 (socket says 172.21.15.16:33830) 2023-10-16T12:23:50.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:50.071+0000 7f1b8ad9d700 1 -- 172.21.15.16:0/1815657458 learned_addr learned my addr 172.21.15.16:0/1815657458 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:50.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:50.071+0000 7f1b8ad9d700 1 -- 172.21.15.16:0/1815657458 >> v2:172.21.15.49:3300/0 conn(0x7f1b8c15b2c0 msgr2=0x7f1b8c161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:50.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:50.071+0000 7f1b8ad9d700 1 --2- 172.21.15.16:0/1815657458 >> v2:172.21.15.49:3300/0 conn(0x7f1b8c15b2c0 0x7f1b8c161340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:50.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:50.071+0000 7f1b8ad9d700 1 -- 172.21.15.16:0/1815657458 >> v1:172.21.15.49:6789/0 conn(0x7f1b8c06e570 legacy=0x7f1b8c15e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:50.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:50.071+0000 7f1b8ad9d700 1 -- 172.21.15.16:0/1815657458 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b8c15b6a0 con 0x7f1b8c06a7e0 2023-10-16T12:23:50.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:50.071+0000 7f1b8ad9d700 1 --2- 172.21.15.16:0/1815657458 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1b8c06a7e0 0x7f1b8c06ac70 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1b84009e50 tx=0x7f1b8400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ae26e4223e14a8c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:50.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:50.071+0000 7f1b8959a700 1 -- 172.21.15.16:0/1815657458 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b8400e070 con 0x7f1b8c06a7e0 2023-10-16T12:23:50.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:50.071+0000 7f1b8959a700 1 -- 172.21.15.16:0/1815657458 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b8400b910 con 0x7f1b8c06a7e0 2023-10-16T12:23:50.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:50.071+0000 7f1b8959a700 1 -- 172.21.15.16:0/1815657458 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b8400bcc0 con 0x7f1b8c06a7e0 2023-10-16T12:23:50.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:50.072+0000 7f1b90a0d700 1 -- 172.21.15.16:0/1815657458 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1b8c06a7e0 msgr2=0x7f1b8c06ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:50.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:50.072+0000 7f1b90a0d700 1 --2- 172.21.15.16:0/1815657458 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1b8c06a7e0 0x7f1b8c06ac70 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1b84009e50 tx=0x7f1b8400b330 comp rx=0 tx=0).stop 2023-10-16T12:23:50.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:50.072+0000 7f1b90a0d700 1 -- 172.21.15.16:0/1815657458 shutdown_connections 2023-10-16T12:23:50.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:50.072+0000 7f1b90a0d700 1 --2- 172.21.15.16:0/1815657458 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1b8c06a7e0 0x7f1b8c06ac70 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:50.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:50.072+0000 7f1b90a0d700 1 --2- 172.21.15.16:0/1815657458 >> v2:172.21.15.49:3300/0 conn(0x7f1b8c15b2c0 0x7f1b8c161340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:50.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:50.072+0000 7f1b90a0d700 1 -- 172.21.15.16:0/1815657458 >> 172.21.15.16:0/1815657458 conn(0x7f1b8c0b1f40 msgr2=0x7f1b8c0b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:50.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:50.072+0000 7f1b90a0d700 1 -- 172.21.15.16:0/1815657458 shutdown_connections 2023-10-16T12:23:50.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:50.072+0000 7f1b90a0d700 1 -- 172.21.15.16:0/1815657458 wait complete. 2023-10-16T12:23:50.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:50.074+0000 7f1b90a0d700 1 Processor -- start 2023-10-16T12:23:50.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:50.074+0000 7f1b90a0d700 1 -- start start 2023-10-16T12:23:50.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:50.075+0000 7f1b90a0d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b8c06a7e0 0x7f1b8c1f70f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:50.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:50.075+0000 7f1b90a0d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1b8c06e570 0x7f1b8c0c8660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:50.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:50.075+0000 7f1b90a0d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1b8c15b2c0 0x7f1b8c159190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:50.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:50.075+0000 7f1b90a0d700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b8c0b61c0 con 0x7f1b8c06a7e0 2023-10-16T12:23:50.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:50.075+0000 7f1b90a0d700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b8c0b5c00 con 0x7f1b8c06e570 2023-10-16T12:23:50.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:50.075+0000 7f1b90a0d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b8c0b5ee0 con 0x7f1b8c15b2c0 2023-10-16T12:23:50.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:50.075+0000 7f1b8a59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b8c06a7e0 0x7f1b8c1f70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:50.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:50.075+0000 7f1b8a59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b8c06a7e0 0x7f1b8c1f70f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37378/0 (socket says 172.21.15.16:37378) 2023-10-16T12:23:50.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:50.075+0000 7f1b8a59c700 1 -- 172.21.15.16:0/2081836158 learned_addr learned my addr 172.21.15.16:0/2081836158 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:50.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:50.075+0000 7f1b89d9b700 1 --2- 172.21.15.16:0/2081836158 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1b8c06e570 0x7f1b8c0c8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:50.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:50.075+0000 7f1b8ad9d700 1 --2- 172.21.15.16:0/2081836158 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1b8c15b2c0 0x7f1b8c159190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:50.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:50.075+0000 7f1b8a59c700 1 -- 172.21.15.16:0/2081836158 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1b8c15b2c0 msgr2=0x7f1b8c159190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:50.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:50.075+0000 7f1b8a59c700 1 --2- 172.21.15.16:0/2081836158 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1b8c15b2c0 0x7f1b8c159190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:50.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:50.075+0000 7f1b8a59c700 1 -- 172.21.15.16:0/2081836158 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1b8c06e570 msgr2=0x7f1b8c0c8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:50.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:50.075+0000 7f1b8a59c700 1 --2- 172.21.15.16:0/2081836158 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1b8c06e570 0x7f1b8c0c8660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:50.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:50.075+0000 7f1b8a59c700 1 -- 172.21.15.16:0/2081836158 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b84009b30 con 0x7f1b8c06a7e0 2023-10-16T12:23:50.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:50.075+0000 7f1b8ad9d700 1 --2- 172.21.15.16:0/2081836158 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1b8c15b2c0 0x7f1b8c159190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:23:50.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:50.075+0000 7f1b8a59c700 1 --2- 172.21.15.16:0/2081836158 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b8c06a7e0 0x7f1b8c1f70f0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f1b8000e9b0 tx=0x7f1b8000ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:50.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:50.076+0000 7f1b7affd700 1 -- 172.21.15.16:0/2081836158 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b80009710 con 0x7f1b8c06a7e0 2023-10-16T12:23:50.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:50.076+0000 7f1b7affd700 1 -- 172.21.15.16:0/2081836158 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b8000ee50 con 0x7f1b8c06a7e0 2023-10-16T12:23:50.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:50.076+0000 7f1b90a0d700 1 -- 172.21.15.16:0/2081836158 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b8c1f68f0 con 0x7f1b8c06a7e0 2023-10-16T12:23:50.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:50.076+0000 7f1b7affd700 1 -- 172.21.15.16:0/2081836158 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b8000ca50 con 0x7f1b8c06a7e0 2023-10-16T12:23:50.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:50.076+0000 7f1b90a0d700 1 -- 172.21.15.16:0/2081836158 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b8c1f7630 con 0x7f1b8c06a7e0 2023-10-16T12:23:50.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:50.076+0000 7f1b90a0d700 1 -- 172.21.15.16:0/2081836158 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b8c1f6f40 con 0x7f1b8c06a7e0 2023-10-16T12:23:50.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:50.077+0000 7f1b7affd700 1 -- 172.21.15.16:0/2081836158 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f1b8000cbb0 con 0x7f1b8c06a7e0 2023-10-16T12:23:50.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:50.077+0000 7f1b7affd700 1 --2- 172.21.15.16:0/2081836158 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f1b6c06ad50 0x7f1b6c06d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:50.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:50.078+0000 7f1b7affd700 1 -- 172.21.15.16:0/2081836158 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 7169+0+0 (secure 0 0 0) 0x7f1b80089860 con 0x7f1b8c06a7e0 2023-10-16T12:23:50.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:50.078+0000 7f1b89d9b700 1 --2- 172.21.15.16:0/2081836158 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f1b6c06ad50 0x7f1b6c06d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:50.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:50.078+0000 7f1b89d9b700 1 --2- 172.21.15.16:0/2081836158 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f1b6c06ad50 0x7f1b6c06d210 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c0ccba0 tx=0x7f1b7c007ed0 comp rx=0 tx=0).ready entity=mgr.5874 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:50.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:50.082+0000 7f1b7affd700 1 -- 172.21.15.16:0/2081836158 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1b800545c0 con 0x7f1b8c06a7e0 2023-10-16T12:23:50.262 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:23:50.265 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:23:50.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:50.334+0000 7f1b90a0d700 1 -- 172.21.15.16:0/2081836158 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f1b8c04ebb0 con 0x7f1b8c06a7e0 2023-10-16T12:23:50.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:50.927+0000 7f1b7affd700 1 -- 172.21.15.16:0/2081836158 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v20) v1 ==== 104+0+0 (secure 0 0 0) 0x7f1b8000ce60 con 0x7f1b8c06a7e0 2023-10-16T12:23:50.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:50.930+0000 7f1b90a0d700 1 -- 172.21.15.16:0/2081836158 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f1b6c06ad50 msgr2=0x7f1b6c06d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:50.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:50.930+0000 7f1b90a0d700 1 --2- 172.21.15.16:0/2081836158 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f1b6c06ad50 0x7f1b6c06d210 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c0ccba0 tx=0x7f1b7c007ed0 comp rx=0 tx=0).stop 2023-10-16T12:23:50.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:50.930+0000 7f1b90a0d700 1 -- 172.21.15.16:0/2081836158 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b8c06a7e0 msgr2=0x7f1b8c1f70f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:50.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:50.930+0000 7f1b90a0d700 1 --2- 172.21.15.16:0/2081836158 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b8c06a7e0 0x7f1b8c1f70f0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f1b8000e9b0 tx=0x7f1b8000ecc0 comp rx=0 tx=0).stop 2023-10-16T12:23:50.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:50.930+0000 7f1b90a0d700 1 -- 172.21.15.16:0/2081836158 shutdown_connections 2023-10-16T12:23:50.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:50.930+0000 7f1b90a0d700 1 --2- 172.21.15.16:0/2081836158 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f1b6c06ad50 0x7f1b6c06d210 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:50.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:50.930+0000 7f1b90a0d700 1 --2- 172.21.15.16:0/2081836158 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1b8c15b2c0 0x7f1b8c159190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:50.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:50.930+0000 7f1b90a0d700 1 --2- 172.21.15.16:0/2081836158 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b8c06a7e0 0x7f1b8c1f70f0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:50.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:50.930+0000 7f1b90a0d700 1 --2- 172.21.15.16:0/2081836158 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1b8c06e570 0x7f1b8c0c8660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:50.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:50.930+0000 7f1b90a0d700 1 -- 172.21.15.16:0/2081836158 >> 172.21.15.16:0/2081836158 conn(0x7f1b8c0b1f40 msgr2=0x7f1b8c15e4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:50.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:50.931+0000 7f1b90a0d700 1 -- 172.21.15.16:0/2081836158 shutdown_connections 2023-10-16T12:23:50.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:50.931+0000 7f1b90a0d700 1 -- 172.21.15.16:0/2081836158 wait complete. 2023-10-16T12:23:50.948 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-16T12:23:51.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:51.124+0000 7f2343e8e700 1 Processor -- start 2023-10-16T12:23:51.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:51.124+0000 7f2343e8e700 1 -- start start 2023-10-16T12:23:51.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:51.125+0000 7f2343e8e700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f233c155a60 0x7f233c15fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:51.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:51.125+0000 7f2343e8e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f233c0cba40 0x7f233c0cbe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:51.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:51.125+0000 7f2343e8e700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f233c0cc3a0 con 0x7f233c155e40 2023-10-16T12:23:51.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:51.125+0000 7f2343e8e700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f233c15f240 con 0x7f233c0cba40 2023-10-16T12:23:51.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:51.125+0000 7f2343e8e700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f233c160190 con 0x7f233c155a60 2023-10-16T12:23:51.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:51.125+0000 7f2341429700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f233c155e40 0x7f233c15bc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54910/0 (socket says 172.21.15.16:54910) 2023-10-16T12:23:51.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:51.125+0000 7f2341429700 1 -- 172.21.15.16:0/1197176580 learned_addr learned my addr 172.21.15.16:0/1197176580 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:23:51.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:51.125+0000 7f234242b700 1 --2- 172.21.15.16:0/1197176580 >> v2:172.21.15.16:3300/0 conn(0x7f233c0cba40 0x7f233c0cbe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:51.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:51.126+0000 7f234242b700 1 -- 172.21.15.16:0/1197176580 >> v2:172.21.15.49:3300/0 conn(0x7f233c155a60 msgr2=0x7f233c15fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:51.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:51.126+0000 7f2341c2a700 1 --2- 172.21.15.16:0/1197176580 >> v2:172.21.15.49:3300/0 conn(0x7f233c155a60 0x7f233c15fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:51.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:51.126+0000 7f234242b700 1 --2- 172.21.15.16:0/1197176580 >> v2:172.21.15.49:3300/0 conn(0x7f233c155a60 0x7f233c15fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:51.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:51.126+0000 7f234242b700 1 -- 172.21.15.16:0/1197176580 >> v1:172.21.15.16:6789/0 conn(0x7f233c155e40 legacy=0x7f233c15bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:51.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:51.126+0000 7f234242b700 1 -- 172.21.15.16:0/1197176580 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f233c15c780 con 0x7f233c0cba40 2023-10-16T12:23:51.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:51.126+0000 7f2340c28700 1 -- 172.21.15.16:0/1197176580 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3527774709 0 0) 0x7f233c0cc3a0 con 0x7f233c155e40 2023-10-16T12:23:51.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:51.126+0000 7f2341c2a700 1 --2- 172.21.15.16:0/1197176580 >> v2:172.21.15.49:3300/0 conn(0x7f233c155a60 0x7f233c15fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:23:51.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:51.126+0000 7f234242b700 1 --2- 172.21.15.16:0/1197176580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f233c0cba40 0x7f233c0cbe60 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f233c0d24f0 tx=0x7f233400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b9ea6832b11ca7f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:51.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:51.126+0000 7f2340c28700 1 -- 172.21.15.16:0/1197176580 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f233400e070 con 0x7f233c0cba40 2023-10-16T12:23:51.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:51.126+0000 7f2340c28700 1 -- 172.21.15.16:0/1197176580 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f233400bbd0 con 0x7f233c0cba40 2023-10-16T12:23:51.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:51.126+0000 7f2340c28700 1 -- 172.21.15.16:0/1197176580 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2334012430 con 0x7f233c0cba40 2023-10-16T12:23:51.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:51.127+0000 7f2343e8e700 1 -- 172.21.15.16:0/1197176580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f233c0cba40 msgr2=0x7f233c0cbe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:51.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:51.127+0000 7f2343e8e700 1 --2- 172.21.15.16:0/1197176580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f233c0cba40 0x7f233c0cbe60 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f233c0d24f0 tx=0x7f233400b330 comp rx=0 tx=0).stop 2023-10-16T12:23:51.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:51.127+0000 7f2343e8e700 1 -- 172.21.15.16:0/1197176580 shutdown_connections 2023-10-16T12:23:51.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:51.127+0000 7f2343e8e700 1 --2- 172.21.15.16:0/1197176580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f233c0cba40 0x7f233c0cbe60 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:51.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:51.127+0000 7f2343e8e700 1 --2- 172.21.15.16:0/1197176580 >> v2:172.21.15.49:3300/0 conn(0x7f233c155a60 0x7f233c15fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:51.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:51.127+0000 7f2343e8e700 1 -- 172.21.15.16:0/1197176580 >> 172.21.15.16:0/1197176580 conn(0x7f233c06a7e0 msgr2=0x7f233c06abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:51.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:51.127+0000 7f2343e8e700 1 -- 172.21.15.16:0/1197176580 shutdown_connections 2023-10-16T12:23:51.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:51.127+0000 7f2343e8e700 1 -- 172.21.15.16:0/1197176580 wait complete. 2023-10-16T12:23:51.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:51.128+0000 7f2343e8e700 1 Processor -- start 2023-10-16T12:23:51.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:51.128+0000 7f2343e8e700 1 -- start start 2023-10-16T12:23:51.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:51.129+0000 7f2343e8e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f233c0cba40 0x7f233c151920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:51.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:51.129+0000 7f2343e8e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f233c155a60 0x7f233c157310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:51.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:51.129+0000 7f2343e8e700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f233c155e40 0x7f233c15a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:51.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:51.129+0000 7f2343e8e700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f233c1580b0 con 0x7f233c155a60 2023-10-16T12:23:51.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:51.129+0000 7f2343e8e700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f233c158390 con 0x7f233c155e40 2023-10-16T12:23:51.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:51.129+0000 7f2343e8e700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f233c158670 con 0x7f233c0cba40 2023-10-16T12:23:51.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:51.129+0000 7f2341429700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f233c155a60 0x7f233c157310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:51.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:51.129+0000 7f2341429700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f233c155a60 0x7f233c157310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37412/0 (socket says 172.21.15.16:37412) 2023-10-16T12:23:51.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:51.129+0000 7f2341429700 1 -- 172.21.15.16:0/1478288150 learned_addr learned my addr 172.21.15.16:0/1478288150 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:51.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:51.129+0000 7f2341c2a700 1 --2- 172.21.15.16:0/1478288150 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f233c0cba40 0x7f233c151920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:51.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:51.129+0000 7f234242b700 1 --2- 172.21.15.16:0/1478288150 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f233c155e40 0x7f233c15a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:51.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:51.129+0000 7f2341429700 1 -- 172.21.15.16:0/1478288150 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f233c0cba40 msgr2=0x7f233c151920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:51.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:51.129+0000 7f2341429700 1 --2- 172.21.15.16:0/1478288150 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f233c0cba40 0x7f233c151920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:51.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:51.129+0000 7f2341429700 1 -- 172.21.15.16:0/1478288150 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f233c155e40 msgr2=0x7f233c15a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:51.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:51.129+0000 7f2341429700 1 --2- 172.21.15.16:0/1478288150 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f233c155e40 0x7f233c15a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:51.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:51.129+0000 7f2341429700 1 -- 172.21.15.16:0/1478288150 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2334009a70 con 0x7f233c155a60 2023-10-16T12:23:51.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:51.129+0000 7f2341c2a700 1 --2- 172.21.15.16:0/1478288150 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f233c0cba40 0x7f233c151920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:23:51.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:51.129+0000 7f2341429700 1 --2- 172.21.15.16:0/1478288150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f233c155a60 0x7f233c157310 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f233000c550 tx=0x7f233000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:51.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:51.130+0000 7f232e7fc700 1 -- 172.21.15.16:0/1478288150 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2330004c80 con 0x7f233c155a60 2023-10-16T12:23:51.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:51.130+0000 7f232e7fc700 1 -- 172.21.15.16:0/1478288150 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2330004e00 con 0x7f233c155a60 2023-10-16T12:23:51.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:51.130+0000 7f232e7fc700 1 -- 172.21.15.16:0/1478288150 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2330007310 con 0x7f233c155a60 2023-10-16T12:23:51.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:51.130+0000 7f2343e8e700 1 -- 172.21.15.16:0/1478288150 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f233c158950 con 0x7f233c155a60 2023-10-16T12:23:51.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:51.130+0000 7f2343e8e700 1 -- 172.21.15.16:0/1478288150 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f233c158f10 con 0x7f233c155a60 2023-10-16T12:23:51.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:51.131+0000 7f2343e8e700 1 -- 172.21.15.16:0/1478288150 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f233c04ec20 con 0x7f233c155a60 2023-10-16T12:23:51.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:51.131+0000 7f232e7fc700 1 -- 172.21.15.16:0/1478288150 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f2330007470 con 0x7f233c155a60 2023-10-16T12:23:51.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:51.132+0000 7f232e7fc700 1 --2- 172.21.15.16:0/1478288150 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f232006ad50 0x7f232006d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:51.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:51.132+0000 7f2341c2a700 1 --2- 172.21.15.16:0/1478288150 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f232006ad50 0x7f232006d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:51.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:51.132+0000 7f232e7fc700 1 -- 172.21.15.16:0/1478288150 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 7169+0+0 (secure 0 0 0) 0x7f2330089490 con 0x7f233c155a60 2023-10-16T12:23:51.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:51.132+0000 7f2341c2a700 1 --2- 172.21.15.16:0/1478288150 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f232006ad50 0x7f232006d210 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2338006fd0 tx=0x7f2338008040 comp rx=0 tx=0).ready entity=mgr.5874 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:51.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:51.136+0000 7f232e7fc700 1 -- 172.21.15.16:0/1478288150 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f23300542c0 con 0x7f233c155a60 2023-10-16T12:23:51.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:51.345+0000 7f2343e8e700 1 -- 172.21.15.16:0/1478288150 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f233c0b3b10 con 0x7f233c155a60 2023-10-16T12:23:51.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:51.859+0000 7f232e7fc700 1 -- 172.21.15.16:0/1478288150 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 17) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f233005e080 con 0x7f233c155a60 2023-10-16T12:23:52.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:52.221+0000 7f232e7fc700 1 -- 172.21.15.16:0/1478288150 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v59) v1 ==== 181+0+0 (secure 0 0 0) 0x7f2330073990 con 0x7f233c155a60 2023-10-16T12:23:52.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:52.226+0000 7f2343e8e700 1 -- 172.21.15.16:0/1478288150 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f232006ad50 msgr2=0x7f232006d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:52.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:52.226+0000 7f2343e8e700 1 --2- 172.21.15.16:0/1478288150 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f232006ad50 0x7f232006d210 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2338006fd0 tx=0x7f2338008040 comp rx=0 tx=0).stop 2023-10-16T12:23:52.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:52.226+0000 7f2343e8e700 1 -- 172.21.15.16:0/1478288150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f233c155a60 msgr2=0x7f233c157310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:52.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:52.226+0000 7f2343e8e700 1 --2- 172.21.15.16:0/1478288150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f233c155a60 0x7f233c157310 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f233000c550 tx=0x7f233000c910 comp rx=0 tx=0).stop 2023-10-16T12:23:52.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:52.226+0000 7f2343e8e700 1 -- 172.21.15.16:0/1478288150 shutdown_connections 2023-10-16T12:23:52.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:52.226+0000 7f2343e8e700 1 --2- 172.21.15.16:0/1478288150 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f232006ad50 0x7f232006d210 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:52.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:52.226+0000 7f2343e8e700 1 --2- 172.21.15.16:0/1478288150 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f233c155e40 0x7f233c15a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:52.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:52.226+0000 7f2343e8e700 1 --2- 172.21.15.16:0/1478288150 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f233c0cba40 0x7f233c151920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:52.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:52.226+0000 7f2343e8e700 1 --2- 172.21.15.16:0/1478288150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f233c155a60 0x7f233c157310 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:52.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:52.226+0000 7f2343e8e700 1 -- 172.21.15.16:0/1478288150 >> 172.21.15.16:0/1478288150 conn(0x7f233c06a7e0 msgr2=0x7f233c0d17b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:52.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:52.227+0000 7f2343e8e700 1 -- 172.21.15.16:0/1478288150 shutdown_connections 2023-10-16T12:23:52.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:52.227+0000 7f2343e8e700 1 -- 172.21.15.16:0/1478288150 wait complete. 2023-10-16T12:23:52.236 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' removed 2023-10-16T12:23:52.245 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-16T12:23:52.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:52.378+0000 7fba5f3c0700 1 Processor -- start 2023-10-16T12:23:52.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:52.379+0000 7fba5f3c0700 1 -- start start 2023-10-16T12:23:52.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:52.379+0000 7fba5f3c0700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fba5806d980 0x7fba5815efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:52.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:52.379+0000 7fba5f3c0700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fba5806a7e0 0x7fba5806ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:52.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:52.379+0000 7fba5f3c0700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba580cdeb0 con 0x7fba58160e60 2023-10-16T12:23:52.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:52.379+0000 7fba5f3c0700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba5806b190 con 0x7fba5806d980 2023-10-16T12:23:52.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:52.379+0000 7fba5f3c0700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba581613e0 con 0x7fba5806a7e0 2023-10-16T12:23:52.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:52.380+0000 7fba5c95b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fba5806d980 0x7fba5815efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:52.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:52.380+0000 7fba5c95b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fba5806d980 0x7fba5815efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37418/0 (socket says 172.21.15.16:37418) 2023-10-16T12:23:52.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:52.380+0000 7fba5c95b700 1 -- 172.21.15.16:0/2848781664 learned_addr learned my addr 172.21.15.16:0/2848781664 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:52.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:52.380+0000 7fba5c95b700 1 -- 172.21.15.16:0/2848781664 >> v2:172.21.15.49:3300/0 conn(0x7fba5806a7e0 msgr2=0x7fba5806ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:52.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:52.380+0000 7fba5c95b700 1 --2- 172.21.15.16:0/2848781664 >> v2:172.21.15.49:3300/0 conn(0x7fba5806a7e0 0x7fba5806ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:52.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:52.380+0000 7fba5c95b700 1 -- 172.21.15.16:0/2848781664 >> v1:172.21.15.16:6789/0 conn(0x7fba58160e60 legacy=0x7fba5815e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:52.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:52.380+0000 7fba5c95b700 1 -- 172.21.15.16:0/2848781664 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba581582d0 con 0x7fba5806d980 2023-10-16T12:23:52.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:52.380+0000 7fba5c95b700 1 --2- 172.21.15.16:0/2848781664 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba5806d980 0x7fba5815efb0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fba580cf5a0 tx=0x7fba54009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=126b96d5e1497068 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:52.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:52.380+0000 7fba4ffff700 1 -- 172.21.15.16:0/2848781664 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba5400b870 con 0x7fba5806d980 2023-10-16T12:23:52.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:52.380+0000 7fba4ffff700 1 -- 172.21.15.16:0/2848781664 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba5400b9d0 con 0x7fba5806d980 2023-10-16T12:23:52.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:52.380+0000 7fba4ffff700 1 -- 172.21.15.16:0/2848781664 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba5400bd60 con 0x7fba5806d980 2023-10-16T12:23:52.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:52.381+0000 7fba5f3c0700 1 -- 172.21.15.16:0/2848781664 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba5806d980 msgr2=0x7fba5815efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:52.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:52.381+0000 7fba5f3c0700 1 --2- 172.21.15.16:0/2848781664 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba5806d980 0x7fba5815efb0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fba580cf5a0 tx=0x7fba54009bd0 comp rx=0 tx=0).stop 2023-10-16T12:23:52.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:52.381+0000 7fba5f3c0700 1 -- 172.21.15.16:0/2848781664 shutdown_connections 2023-10-16T12:23:52.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:52.381+0000 7fba5f3c0700 1 --2- 172.21.15.16:0/2848781664 >> v2:172.21.15.49:3300/0 conn(0x7fba5806a7e0 0x7fba5806ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:52.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:52.381+0000 7fba5f3c0700 1 --2- 172.21.15.16:0/2848781664 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba5806d980 0x7fba5815efb0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:52.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:52.381+0000 7fba5f3c0700 1 -- 172.21.15.16:0/2848781664 >> 172.21.15.16:0/2848781664 conn(0x7fba5806f430 msgr2=0x7fba5806f810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:52.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:52.381+0000 7fba5f3c0700 1 -- 172.21.15.16:0/2848781664 shutdown_connections 2023-10-16T12:23:52.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:52.381+0000 7fba5f3c0700 1 -- 172.21.15.16:0/2848781664 wait complete. 2023-10-16T12:23:52.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:52.382+0000 7fba5f3c0700 1 Processor -- start 2023-10-16T12:23:52.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:52.382+0000 7fba5f3c0700 1 -- start start 2023-10-16T12:23:52.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:52.383+0000 7fba5f3c0700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fba5806a7e0 0x7fba5814cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:52.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:52.383+0000 7fba5f3c0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba5806d980 0x7fba580c63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:52.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:52.383+0000 7fba5f3c0700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fba58160e60 0x7fba5806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:52.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:52.383+0000 7fba5f3c0700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba580c6910 con 0x7fba5806d980 2023-10-16T12:23:52.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:52.383+0000 7fba5f3c0700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba580c6bc0 con 0x7fba5806a7e0 2023-10-16T12:23:52.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:52.383+0000 7fba5f3c0700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba580aead0 con 0x7fba58160e60 2023-10-16T12:23:52.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:52.383+0000 7fba5c95b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba5806d980 0x7fba580c63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:52.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:52.383+0000 7fba5c95b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba5806d980 0x7fba580c63d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37426/0 (socket says 172.21.15.16:37426) 2023-10-16T12:23:52.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:52.383+0000 7fba5c95b700 1 -- 172.21.15.16:0/1623375361 learned_addr learned my addr 172.21.15.16:0/1623375361 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:52.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:52.383+0000 7fba5d15c700 1 --2- 172.21.15.16:0/1623375361 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fba5806a7e0 0x7fba5814cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:52.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:52.383+0000 7fba5d95d700 1 --2- 172.21.15.16:0/1623375361 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fba58160e60 0x7fba5806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:52.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:52.383+0000 7fba5c95b700 1 -- 172.21.15.16:0/1623375361 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fba58160e60 msgr2=0x7fba5806e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:52.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:52.383+0000 7fba5c95b700 1 --2- 172.21.15.16:0/1623375361 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fba58160e60 0x7fba5806e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:52.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:52.383+0000 7fba5c95b700 1 -- 172.21.15.16:0/1623375361 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fba5806a7e0 msgr2=0x7fba5814cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:52.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:52.383+0000 7fba5c95b700 1 --2- 172.21.15.16:0/1623375361 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fba5806a7e0 0x7fba5814cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:52.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:52.383+0000 7fba5c95b700 1 -- 172.21.15.16:0/1623375361 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba54009550 con 0x7fba5806d980 2023-10-16T12:23:52.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:52.383+0000 7fba5c95b700 1 --2- 172.21.15.16:0/1623375361 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba5806d980 0x7fba580c63d0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fba54013000 tx=0x7fba54004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:52.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:52.383+0000 7fba4dffb700 1 -- 172.21.15.16:0/1623375361 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba54004310 con 0x7fba5806d980 2023-10-16T12:23:52.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:52.384+0000 7fba4dffb700 1 -- 172.21.15.16:0/1623375361 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba5401f070 con 0x7fba5806d980 2023-10-16T12:23:52.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:52.384+0000 7fba4dffb700 1 -- 172.21.15.16:0/1623375361 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba540036a0 con 0x7fba5806d980 2023-10-16T12:23:52.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:52.384+0000 7fba5f3c0700 1 -- 172.21.15.16:0/1623375361 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba580aedb0 con 0x7fba5806d980 2023-10-16T12:23:52.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:52.384+0000 7fba5f3c0700 1 -- 172.21.15.16:0/1623375361 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba580af370 con 0x7fba5806d980 2023-10-16T12:23:52.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:52.385+0000 7fba5f3c0700 1 -- 172.21.15.16:0/1623375361 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba5804ec20 con 0x7fba5806d980 2023-10-16T12:23:52.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:52.385+0000 7fba4dffb700 1 -- 172.21.15.16:0/1623375361 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fba54021030 con 0x7fba5806d980 2023-10-16T12:23:52.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:52.386+0000 7fba4dffb700 1 --2- 172.21.15.16:0/1623375361 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fba3c06af90 0x7fba3c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:52.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:52.386+0000 7fba4dffb700 1 -- 172.21.15.16:0/1623375361 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6351+0+0 (secure 0 0 0) 0x7fba5408abf0 con 0x7fba5806d980 2023-10-16T12:23:52.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:52.386+0000 7fba5d15c700 1 --2- 172.21.15.16:0/1623375361 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fba3c06af90 0x7fba3c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:52.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:52.386+0000 7fba5d15c700 1 --2- 172.21.15.16:0/1623375361 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fba3c06af90 0x7fba3c06d450 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fba48007fd0 tx=0x7fba48009040 comp rx=0 tx=0).ready entity=mgr.5874 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:52.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:52.390+0000 7fba4dffb700 1 -- 172.21.15.16:0/1623375361 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fba54055ac0 con 0x7fba5806d980 2023-10-16T12:23:52.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:52.594+0000 7fba5f3c0700 1 -- 172.21.15.16:0/1623375361 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fba580b6260 con 0x7fba5806d980 2023-10-16T12:23:53.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.232+0000 7fba4dffb700 1 -- 172.21.15.16:0/1623375361 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v60) v1 ==== 169+0+0 (secure 0 0 0) 0x7fba540590e0 con 0x7fba5806d980 2023-10-16T12:23:53.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.238+0000 7fba5f3c0700 1 -- 172.21.15.16:0/1623375361 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fba3c06af90 msgr2=0x7fba3c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:53.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.238+0000 7fba5f3c0700 1 --2- 172.21.15.16:0/1623375361 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fba3c06af90 0x7fba3c06d450 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fba48007fd0 tx=0x7fba48009040 comp rx=0 tx=0).stop 2023-10-16T12:23:53.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.238+0000 7fba5f3c0700 1 -- 172.21.15.16:0/1623375361 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba5806d980 msgr2=0x7fba580c63d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:53.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.238+0000 7fba5f3c0700 1 --2- 172.21.15.16:0/1623375361 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba5806d980 0x7fba580c63d0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fba54013000 tx=0x7fba54004750 comp rx=0 tx=0).stop 2023-10-16T12:23:53.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.241+0000 7fba5f3c0700 1 -- 172.21.15.16:0/1623375361 shutdown_connections 2023-10-16T12:23:53.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.241+0000 7fba5f3c0700 1 --2- 172.21.15.16:0/1623375361 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fba3c06af90 0x7fba3c06d450 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:53.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.242+0000 7fba5f3c0700 1 --2- 172.21.15.16:0/1623375361 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fba58160e60 0x7fba5806e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:53.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.242+0000 7fba5f3c0700 1 --2- 172.21.15.16:0/1623375361 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fba5806a7e0 0x7fba5814cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:53.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.242+0000 7fba5f3c0700 1 --2- 172.21.15.16:0/1623375361 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba5806d980 0x7fba580c63d0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:53.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.242+0000 7fba5f3c0700 1 -- 172.21.15.16:0/1623375361 >> 172.21.15.16:0/1623375361 conn(0x7fba5806f430 msgr2=0x7fba5815f6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:53.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.242+0000 7fba5f3c0700 1 -- 172.21.15.16:0/1623375361 shutdown_connections 2023-10-16T12:23:53.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.243+0000 7fba5f3c0700 1 -- 172.21.15.16:0/1623375361 wait complete. 2023-10-16T12:23:53.249 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' removed 2023-10-16T12:23:53.264 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-16T12:23:53.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.394+0000 7f1ec9279700 1 Processor -- start 2023-10-16T12:23:53.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.394+0000 7f1ec9279700 1 -- start start 2023-10-16T12:23:53.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.395+0000 7f1ec9279700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f1ec40b8fa0 0x7f1ec40b2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:53.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.395+0000 7f1ec9279700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1ec40ce350 0x7f1ec40ce780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:53.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.395+0000 7f1ec9279700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ec40cecc0 con 0x7f1ec415d8d0 2023-10-16T12:23:53.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.395+0000 7f1ec9279700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ec40d0bb0 con 0x7f1ec40b8fa0 2023-10-16T12:23:53.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.395+0000 7f1ec9279700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec415dcb0 con 0x7f1ec40ce350 2023-10-16T12:23:53.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.395+0000 7f1ec37fe700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1ec40ce350 0x7f1ec40ce780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:53.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.395+0000 7f1ec27fc700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f1ec40b8fa0 0x7f1ec40b2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:53.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.395+0000 7f1ec2ffd700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7f1ec415d8d0 0x7f1ec40b6720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:33888/0 (socket says 172.21.15.16:33888) 2023-10-16T12:23:53.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.395+0000 7f1ec2ffd700 1 -- 172.21.15.16:0/1503930726 learned_addr learned my addr 172.21.15.16:0/1503930726 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:23:53.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.396+0000 7f1ec1ffb700 1 -- 172.21.15.16:0/1503930726 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2478153109 0 0) 0x7f1ec40cecc0 con 0x7f1ec415d8d0 2023-10-16T12:23:53.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.396+0000 7f1ec1ffb700 1 -- 172.21.15.16:0/1503930726 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1eac0035b0 con 0x7f1ec415d8d0 2023-10-16T12:23:53.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.396+0000 7f1ec37fe700 1 -- 172.21.15.16:0/1503930726 >> v2:172.21.15.49:3300/0 conn(0x7f1ec40b8fa0 msgr2=0x7f1ec40b2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:53.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.396+0000 7f1ec37fe700 1 --2- 172.21.15.16:0/1503930726 >> v2:172.21.15.49:3300/0 conn(0x7f1ec40b8fa0 0x7f1ec40b2830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:53.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.396+0000 7f1ec37fe700 1 -- 172.21.15.16:0/1503930726 >> v1:172.21.15.49:6789/0 conn(0x7f1ec415d8d0 legacy=0x7f1ec40b6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:53.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.396+0000 7f1ec37fe700 1 -- 172.21.15.16:0/1503930726 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ec40b2f30 con 0x7f1ec40ce350 2023-10-16T12:23:53.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.396+0000 7f1ec27fc700 1 --2- 172.21.15.16:0/1503930726 >> v2:172.21.15.49:3300/0 conn(0x7f1ec40b8fa0 0x7f1ec40b2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:23:53.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.396+0000 7f1ec37fe700 1 --2- 172.21.15.16:0/1503930726 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1ec40ce350 0x7f1ec40ce780 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc009d90 tx=0x7f1ebc00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae925737d8f19e72 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:53.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.396+0000 7f1ec1ffb700 1 -- 172.21.15.16:0/1503930726 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ebc00e070 con 0x7f1ec40ce350 2023-10-16T12:23:53.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.396+0000 7f1ec1ffb700 1 -- 172.21.15.16:0/1503930726 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ebc00bb20 con 0x7f1ec40ce350 2023-10-16T12:23:53.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.396+0000 7f1ec1ffb700 1 -- 172.21.15.16:0/1503930726 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ebc00be90 con 0x7f1ec40ce350 2023-10-16T12:23:53.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.397+0000 7f1ec9279700 1 -- 172.21.15.16:0/1503930726 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1ec40ce350 msgr2=0x7f1ec40ce780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:53.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.397+0000 7f1ec9279700 1 --2- 172.21.15.16:0/1503930726 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1ec40ce350 0x7f1ec40ce780 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc009d90 tx=0x7f1ebc00b330 comp rx=0 tx=0).stop 2023-10-16T12:23:53.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.397+0000 7f1ec9279700 1 -- 172.21.15.16:0/1503930726 shutdown_connections 2023-10-16T12:23:53.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.397+0000 7f1ec9279700 1 --2- 172.21.15.16:0/1503930726 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1ec40ce350 0x7f1ec40ce780 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:53.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.397+0000 7f1ec9279700 1 --2- 172.21.15.16:0/1503930726 >> v2:172.21.15.49:3300/0 conn(0x7f1ec40b8fa0 0x7f1ec40b2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:53.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.397+0000 7f1ec9279700 1 -- 172.21.15.16:0/1503930726 >> 172.21.15.16:0/1503930726 conn(0x7f1ec4072330 msgr2=0x7f1ec4072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:53.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.397+0000 7f1ec9279700 1 -- 172.21.15.16:0/1503930726 shutdown_connections 2023-10-16T12:23:53.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.397+0000 7f1ec9279700 1 -- 172.21.15.16:0/1503930726 wait complete. 2023-10-16T12:23:53.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.398+0000 7f1ec9279700 1 Processor -- start 2023-10-16T12:23:53.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.398+0000 7f1ec9279700 1 -- start start 2023-10-16T12:23:53.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.399+0000 7f1ec9279700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1ec40b8fa0 0x7f1ec40c7040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:53.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.399+0000 7f1ec9279700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ec40ce350 0x7f1ec41f65b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:53.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.399+0000 7f1ec9279700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1ec415d8d0 0x7f1ec41f6af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:53.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.399+0000 7f1ec9279700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec4149ce0 con 0x7f1ec40ce350 2023-10-16T12:23:53.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.399+0000 7f1ec9279700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec415eff0 con 0x7f1ec415d8d0 2023-10-16T12:23:53.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.399+0000 7f1ec9279700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec40c9b30 con 0x7f1ec40b8fa0 2023-10-16T12:23:53.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.399+0000 7f1ec27fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ec40ce350 0x7f1ec41f65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:53.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.399+0000 7f1ec27fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ec40ce350 0x7f1ec41f65b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37440/0 (socket says 172.21.15.16:37440) 2023-10-16T12:23:53.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.399+0000 7f1ec27fc700 1 -- 172.21.15.16:0/2849251936 learned_addr learned my addr 172.21.15.16:0/2849251936 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:53.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.399+0000 7f1ec2ffd700 1 --2- 172.21.15.16:0/2849251936 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1ec40b8fa0 0x7f1ec40c7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:53.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.399+0000 7f1ec37fe700 1 --2- 172.21.15.16:0/2849251936 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1ec415d8d0 0x7f1ec41f6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:53.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.399+0000 7f1ec27fc700 1 -- 172.21.15.16:0/2849251936 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1ec40b8fa0 msgr2=0x7f1ec40c7040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:53.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.399+0000 7f1ec27fc700 1 --2- 172.21.15.16:0/2849251936 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1ec40b8fa0 0x7f1ec40c7040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:53.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.399+0000 7f1ec27fc700 1 -- 172.21.15.16:0/2849251936 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1ec415d8d0 msgr2=0x7f1ec41f6af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:53.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.399+0000 7f1ec27fc700 1 --2- 172.21.15.16:0/2849251936 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1ec415d8d0 0x7f1ec41f6af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:53.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.399+0000 7f1ec27fc700 1 -- 172.21.15.16:0/2849251936 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eb40094a0 con 0x7f1ec40ce350 2023-10-16T12:23:53.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.400+0000 7f1ec27fc700 1 --2- 172.21.15.16:0/2849251936 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ec40ce350 0x7f1ec41f65b0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f1eb400ebe0 tx=0x7f1eb400eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:53.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.400+0000 7f1eab7fe700 1 -- 172.21.15.16:0/2849251936 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb4009760 con 0x7f1ec40ce350 2023-10-16T12:23:53.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.400+0000 7f1eab7fe700 1 -- 172.21.15.16:0/2849251936 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1eb400c850 con 0x7f1ec40ce350 2023-10-16T12:23:53.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.400+0000 7f1ec9279700 1 -- 172.21.15.16:0/2849251936 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ebc009a70 con 0x7f1ec40ce350 2023-10-16T12:23:53.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.400+0000 7f1eab7fe700 1 -- 172.21.15.16:0/2849251936 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb400cbe0 con 0x7f1ec40ce350 2023-10-16T12:23:53.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.400+0000 7f1ec9279700 1 -- 172.21.15.16:0/2849251936 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ec415f240 con 0x7f1ec40ce350 2023-10-16T12:23:53.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.401+0000 7f1ec9279700 1 -- 172.21.15.16:0/2849251936 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ec404ebb0 con 0x7f1ec40ce350 2023-10-16T12:23:53.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.401+0000 7f1eab7fe700 1 -- 172.21.15.16:0/2849251936 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1eb400cd40 con 0x7f1ec40ce350 2023-10-16T12:23:53.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.402+0000 7f1eab7fe700 1 --2- 172.21.15.16:0/2849251936 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f1eb006b020 0x7f1eb006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:53.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.402+0000 7f1eab7fe700 1 -- 172.21.15.16:0/2849251936 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5908+0+0 (secure 0 0 0) 0x7f1eb40897b0 con 0x7f1ec40ce350 2023-10-16T12:23:53.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.402+0000 7f1ec2ffd700 1 --2- 172.21.15.16:0/2849251936 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f1eb006b020 0x7f1eb006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:53.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.402+0000 7f1ec2ffd700 1 --2- 172.21.15.16:0/2849251936 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f1eb006b020 0x7f1eb006d4e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1eb8007fd0 tx=0x7f1eb8009040 comp rx=0 tx=0).ready entity=mgr.5874 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:53.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.406+0000 7f1eab7fe700 1 -- 172.21.15.16:0/2849251936 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1eb4054710 con 0x7f1ec40ce350 2023-10-16T12:23:53.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.613+0000 7f1ec9279700 1 -- 172.21.15.16:0/2849251936 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f1ec40d2490 con 0x7f1ec40ce350 2023-10-16T12:23:53.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.616+0000 7f1eab7fe700 1 -- 172.21.15.16:0/2849251936 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v13) v1 ==== 115+0+0 (secure 0 0 0) 0x7f1eb4057d30 con 0x7f1ec40ce350 2023-10-16T12:23:53.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.619+0000 7f1ec9279700 1 -- 172.21.15.16:0/2849251936 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f1eb006b020 msgr2=0x7f1eb006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:53.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.619+0000 7f1ec9279700 1 --2- 172.21.15.16:0/2849251936 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f1eb006b020 0x7f1eb006d4e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1eb8007fd0 tx=0x7f1eb8009040 comp rx=0 tx=0).stop 2023-10-16T12:23:53.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.619+0000 7f1ec9279700 1 -- 172.21.15.16:0/2849251936 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ec40ce350 msgr2=0x7f1ec41f65b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:53.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.619+0000 7f1ec9279700 1 --2- 172.21.15.16:0/2849251936 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ec40ce350 0x7f1ec41f65b0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f1eb400ebe0 tx=0x7f1eb400eef0 comp rx=0 tx=0).stop 2023-10-16T12:23:53.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.619+0000 7f1ec9279700 1 -- 172.21.15.16:0/2849251936 shutdown_connections 2023-10-16T12:23:53.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.619+0000 7f1ec9279700 1 --2- 172.21.15.16:0/2849251936 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f1eb006b020 0x7f1eb006d4e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:53.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.619+0000 7f1ec9279700 1 --2- 172.21.15.16:0/2849251936 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1ec415d8d0 0x7f1ec41f6af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:53.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.619+0000 7f1ec9279700 1 --2- 172.21.15.16:0/2849251936 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1ec40b8fa0 0x7f1ec40c7040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:53.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.619+0000 7f1ec9279700 1 --2- 172.21.15.16:0/2849251936 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ec40ce350 0x7f1ec41f65b0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:53.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.619+0000 7f1ec9279700 1 -- 172.21.15.16:0/2849251936 >> 172.21.15.16:0/2849251936 conn(0x7f1ec4072330 msgr2=0x7f1ec415cca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:53.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.619+0000 7f1ec9279700 1 -- 172.21.15.16:0/2849251936 shutdown_connections 2023-10-16T12:23:53.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.619+0000 7f1ec9279700 1 -- 172.21.15.16:0/2849251936 wait complete. 2023-10-16T12:23:53.634 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-16T12:23:53.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.759+0000 7fef1b64b700 1 Processor -- start 2023-10-16T12:23:53.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.759+0000 7fef1b64b700 1 -- start start 2023-10-16T12:23:53.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.760+0000 7fef1b64b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fef1415b2c0 0x7fef14161340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:53.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.760+0000 7fef1b64b700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fef1406e570 0x7fef1415e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:53.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.760+0000 7fef1b64b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fef1406a7e0 0x7fef1406ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:53.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.760+0000 7fef1b64b700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef1406b1b0 con 0x7fef1406a7e0 2023-10-16T12:23:53.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.760+0000 7fef1b64b700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef14161880 con 0x7fef1406e570 2023-10-16T12:23:53.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.760+0000 7fef1b64b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef1415b6a0 con 0x7fef1415b2c0 2023-10-16T12:23:53.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.760+0000 7fef19be8700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fef1406a7e0 0x7fef1406ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:53.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.760+0000 7fef19be8700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fef1406a7e0 0x7fef1406ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37450/0 (socket says 172.21.15.16:37450) 2023-10-16T12:23:53.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.760+0000 7fef19be8700 1 -- 172.21.15.16:0/404100417 learned_addr learned my addr 172.21.15.16:0/404100417 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:53.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.760+0000 7fef193e7700 1 --2- 172.21.15.16:0/404100417 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fef1415b2c0 0x7fef14161340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:53.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.760+0000 7fef18be6700 1 --2- 172.21.15.16:0/404100417 >> v2:172.21.15.49:3300/0 conn(0x7fef1406e570 0x7fef1415e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:53.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.760+0000 7fef19be8700 1 -- 172.21.15.16:0/404100417 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fef1415b2c0 msgr2=0x7fef14161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:53.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.760+0000 7fef19be8700 1 --2- 172.21.15.16:0/404100417 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fef1415b2c0 0x7fef14161340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:53.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.760+0000 7fef19be8700 1 -- 172.21.15.16:0/404100417 >> v2:172.21.15.49:3300/0 conn(0x7fef1406e570 msgr2=0x7fef1415e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:53.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.760+0000 7fef19be8700 1 --2- 172.21.15.16:0/404100417 >> v2:172.21.15.49:3300/0 conn(0x7fef1406e570 0x7fef1415e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:53.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.760+0000 7fef19be8700 1 -- 172.21.15.16:0/404100417 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef14161b20 con 0x7fef1406a7e0 2023-10-16T12:23:53.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.760+0000 7fef193e7700 1 --2- 172.21.15.16:0/404100417 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fef1415b2c0 0x7fef14161340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:23:53.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.761+0000 7fef19be8700 1 --2- 172.21.15.16:0/404100417 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef1406a7e0 0x7fef1406ac70 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fef14070410 tx=0x7fef0c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51c399a7ba2cbd08 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:53.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.761+0000 7fef0bfff700 1 -- 172.21.15.16:0/404100417 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef0c00e070 con 0x7fef1406a7e0 2023-10-16T12:23:53.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.761+0000 7fef0bfff700 1 -- 172.21.15.16:0/404100417 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef0c00b9c0 con 0x7fef1406a7e0 2023-10-16T12:23:53.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.761+0000 7fef0bfff700 1 -- 172.21.15.16:0/404100417 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef0c00bd70 con 0x7fef1406a7e0 2023-10-16T12:23:53.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.761+0000 7fef1b64b700 1 -- 172.21.15.16:0/404100417 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef1406a7e0 msgr2=0x7fef1406ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:53.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.761+0000 7fef1b64b700 1 --2- 172.21.15.16:0/404100417 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef1406a7e0 0x7fef1406ac70 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fef14070410 tx=0x7fef0c00b330 comp rx=0 tx=0).stop 2023-10-16T12:23:53.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.761+0000 7fef1b64b700 1 -- 172.21.15.16:0/404100417 shutdown_connections 2023-10-16T12:23:53.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.761+0000 7fef1b64b700 1 --2- 172.21.15.16:0/404100417 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef1406a7e0 0x7fef1406ac70 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:53.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.761+0000 7fef1b64b700 1 --2- 172.21.15.16:0/404100417 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fef1415b2c0 0x7fef14161340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:53.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.761+0000 7fef1b64b700 1 --2- 172.21.15.16:0/404100417 >> v2:172.21.15.49:3300/0 conn(0x7fef1406e570 0x7fef1415e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:53.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.761+0000 7fef1b64b700 1 -- 172.21.15.16:0/404100417 >> 172.21.15.16:0/404100417 conn(0x7fef140b1f40 msgr2=0x7fef140b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:53.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.762+0000 7fef1b64b700 1 -- 172.21.15.16:0/404100417 shutdown_connections 2023-10-16T12:23:53.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.762+0000 7fef1b64b700 1 -- 172.21.15.16:0/404100417 wait complete. 2023-10-16T12:23:53.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.763+0000 7fef1b64b700 1 Processor -- start 2023-10-16T12:23:53.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.763+0000 7fef1b64b700 1 -- start start 2023-10-16T12:23:53.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.763+0000 7fef1b64b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fef1406a7e0 0x7fef140cbcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:53.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.763+0000 7fef1b64b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef1406e570 0x7fef140c9c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:53.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.763+0000 7fef1b64b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fef1415b2c0 0x7fef14157700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:53.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.764+0000 7fef1b64b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef140b3b00 con 0x7fef1406e570 2023-10-16T12:23:53.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.764+0000 7fef1b64b700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef140b2f10 con 0x7fef1415b2c0 2023-10-16T12:23:53.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.764+0000 7fef1b64b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef140b3160 con 0x7fef1406a7e0 2023-10-16T12:23:53.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.764+0000 7fef18be6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef1406e570 0x7fef140c9c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:53.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.764+0000 7fef18be6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef1406e570 0x7fef140c9c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37454/0 (socket says 172.21.15.16:37454) 2023-10-16T12:23:53.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.764+0000 7fef18be6700 1 -- 172.21.15.16:0/3215201852 learned_addr learned my addr 172.21.15.16:0/3215201852 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:53.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.764+0000 7fef193e7700 1 --2- 172.21.15.16:0/3215201852 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fef1406a7e0 0x7fef140cbcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:53.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.764+0000 7fef19be8700 1 --2- 172.21.15.16:0/3215201852 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fef1415b2c0 0x7fef14157700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:53.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.764+0000 7fef18be6700 1 -- 172.21.15.16:0/3215201852 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fef1406a7e0 msgr2=0x7fef140cbcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:53.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.764+0000 7fef18be6700 1 --2- 172.21.15.16:0/3215201852 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fef1406a7e0 0x7fef140cbcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:53.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.764+0000 7fef18be6700 1 -- 172.21.15.16:0/3215201852 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fef1415b2c0 msgr2=0x7fef14157700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:53.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.764+0000 7fef18be6700 1 --2- 172.21.15.16:0/3215201852 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fef1415b2c0 0x7fef14157700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:53.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.764+0000 7fef18be6700 1 -- 172.21.15.16:0/3215201852 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef0c009b30 con 0x7fef1406e570 2023-10-16T12:23:53.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.764+0000 7fef18be6700 1 --2- 172.21.15.16:0/3215201852 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef1406e570 0x7fef140c9c70 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fef0400d800 tx=0x7fef0400dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:53.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.764+0000 7fef09ffb700 1 -- 172.21.15.16:0/3215201852 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef040125f0 con 0x7fef1406e570 2023-10-16T12:23:53.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.764+0000 7fef09ffb700 1 -- 172.21.15.16:0/3215201852 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef04012750 con 0x7fef1406e570 2023-10-16T12:23:53.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.764+0000 7fef1b64b700 1 -- 172.21.15.16:0/3215201852 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef140b3440 con 0x7fef1406e570 2023-10-16T12:23:53.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.764+0000 7fef09ffb700 1 -- 172.21.15.16:0/3215201852 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef04012b00 con 0x7fef1406e570 2023-10-16T12:23:53.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.764+0000 7fef1b64b700 1 -- 172.21.15.16:0/3215201852 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef140cc210 con 0x7fef1406e570 2023-10-16T12:23:53.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.765+0000 7fef1b64b700 1 -- 172.21.15.16:0/3215201852 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef1404ebb0 con 0x7fef1406e570 2023-10-16T12:23:53.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.766+0000 7fef09ffb700 1 -- 172.21.15.16:0/3215201852 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fef04012c60 con 0x7fef1406e570 2023-10-16T12:23:53.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.766+0000 7fef09ffb700 1 --2- 172.21.15.16:0/3215201852 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7feef806b020 0x7feef806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:53.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.766+0000 7fef09ffb700 1 -- 172.21.15.16:0/3215201852 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5908+0+0 (secure 0 0 0) 0x7fef04089b40 con 0x7fef1406e570 2023-10-16T12:23:53.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.766+0000 7fef193e7700 1 --2- 172.21.15.16:0/3215201852 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7feef806b020 0x7feef806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:53.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.767+0000 7fef193e7700 1 --2- 172.21.15.16:0/3215201852 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7feef806b020 0x7feef806d4e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fef140b5bc0 tx=0x7fef10008040 comp rx=0 tx=0).ready entity=mgr.5874 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:53.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.771+0000 7fef09ffb700 1 -- 172.21.15.16:0/3215201852 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fef04054aa0 con 0x7fef1406e570 2023-10-16T12:23:53.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.980+0000 7fef1b64b700 1 -- 172.21.15.16:0/3215201852 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fef140b6370 con 0x7fef1406e570 2023-10-16T12:23:53.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.984+0000 7fef09ffb700 1 -- 172.21.15.16:0/3215201852 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v14) v1 ==== 104+0+0 (secure 0 0 0) 0x7fef040580c0 con 0x7fef1406e570 2023-10-16T12:23:53.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.986+0000 7fef1b64b700 1 -- 172.21.15.16:0/3215201852 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7feef806b020 msgr2=0x7feef806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:53.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.986+0000 7fef1b64b700 1 --2- 172.21.15.16:0/3215201852 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7feef806b020 0x7feef806d4e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fef140b5bc0 tx=0x7fef10008040 comp rx=0 tx=0).stop 2023-10-16T12:23:53.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.986+0000 7fef1b64b700 1 -- 172.21.15.16:0/3215201852 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef1406e570 msgr2=0x7fef140c9c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:53.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.986+0000 7fef1b64b700 1 --2- 172.21.15.16:0/3215201852 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef1406e570 0x7fef140c9c70 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fef0400d800 tx=0x7fef0400dbc0 comp rx=0 tx=0).stop 2023-10-16T12:23:53.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.986+0000 7fef1b64b700 1 -- 172.21.15.16:0/3215201852 shutdown_connections 2023-10-16T12:23:53.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.986+0000 7fef1b64b700 1 --2- 172.21.15.16:0/3215201852 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7feef806b020 0x7feef806d4e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:53.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.986+0000 7fef1b64b700 1 --2- 172.21.15.16:0/3215201852 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fef1415b2c0 0x7fef14157700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:53.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.986+0000 7fef1b64b700 1 --2- 172.21.15.16:0/3215201852 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fef1406a7e0 0x7fef140cbcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:53.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.986+0000 7fef1b64b700 1 --2- 172.21.15.16:0/3215201852 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef1406e570 0x7fef140c9c70 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:53.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.986+0000 7fef1b64b700 1 -- 172.21.15.16:0/3215201852 >> 172.21.15.16:0/3215201852 conn(0x7fef140b1f40 msgr2=0x7fef1415b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:53.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.986+0000 7fef1b64b700 1 -- 172.21.15.16:0/3215201852 shutdown_connections 2023-10-16T12:23:53.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:53.986+0000 7fef1b64b700 1 -- 172.21.15.16:0/3215201852 wait complete. 2023-10-16T12:23:54.000 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-16T12:23:54.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.125+0000 7f33be98f700 1 Processor -- start 2023-10-16T12:23:54.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.125+0000 7f33be98f700 1 -- start start 2023-10-16T12:23:54.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.126+0000 7f33be98f700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f33b815b070 0x7f33b815b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:54.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.126+0000 7f33be98f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f33b80cb880 0x7f33b80cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:54.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.126+0000 7f33be98f700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33b80cef70 con 0x7f33b806da60 2023-10-16T12:23:54.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.126+0000 7f33be98f700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33b80ce360 con 0x7f33b815b070 2023-10-16T12:23:54.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.126+0000 7f33be98f700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33b80ce640 con 0x7f33b80cb880 2023-10-16T12:23:54.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.126+0000 7f33b7fff700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f33b815b070 0x7f33b815b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:54.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.126+0000 7f33bcf2c700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7f33b806da60 0x7f33b80cf400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:33920/0 (socket says 172.21.15.16:33920) 2023-10-16T12:23:54.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.126+0000 7f33bcf2c700 1 -- 172.21.15.16:0/3382920871 learned_addr learned my addr 172.21.15.16:0/3382920871 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:23:54.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.127+0000 7f33b77fe700 1 --2- 172.21.15.16:0/3382920871 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f33b80cb880 0x7f33b80cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:54.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.127+0000 7f33b6ffd700 1 -- 172.21.15.16:0/3382920871 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3145014922 0 0) 0x7f33b80cef70 con 0x7f33b806da60 2023-10-16T12:23:54.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.127+0000 7f33b6ffd700 1 -- 172.21.15.16:0/3382920871 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33a00035b0 con 0x7f33b806da60 2023-10-16T12:23:54.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.127+0000 7f33b77fe700 1 -- 172.21.15.16:0/3382920871 >> v2:172.21.15.49:3300/0 conn(0x7f33b815b070 msgr2=0x7f33b815b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:54.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.127+0000 7f33b77fe700 1 --2- 172.21.15.16:0/3382920871 >> v2:172.21.15.49:3300/0 conn(0x7f33b815b070 0x7f33b815b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:54.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.127+0000 7f33b77fe700 1 -- 172.21.15.16:0/3382920871 >> v1:172.21.15.49:6789/0 conn(0x7f33b806da60 legacy=0x7f33b80cf400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:54.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.127+0000 7f33b77fe700 1 -- 172.21.15.16:0/3382920871 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33b80ce920 con 0x7f33b80cb880 2023-10-16T12:23:54.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.127+0000 7f33b7fff700 1 --2- 172.21.15.16:0/3382920871 >> v2:172.21.15.49:3300/0 conn(0x7f33b815b070 0x7f33b815b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:23:54.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.127+0000 7f33b77fe700 1 --2- 172.21.15.16:0/3382920871 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f33b80cb880 0x7f33b80cbd00 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f33ac009840 tx=0x7f33ac009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88dee90821b0cc40 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:54.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.127+0000 7f33b6ffd700 1 -- 172.21.15.16:0/3382920871 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33ac011830 con 0x7f33b80cb880 2023-10-16T12:23:54.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.127+0000 7f33b6ffd700 1 -- 172.21.15.16:0/3382920871 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33ac011990 con 0x7f33b80cb880 2023-10-16T12:23:54.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.127+0000 7f33b6ffd700 1 -- 172.21.15.16:0/3382920871 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33ac011d20 con 0x7f33b80cb880 2023-10-16T12:23:54.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.128+0000 7f33be98f700 1 -- 172.21.15.16:0/3382920871 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f33b80cb880 msgr2=0x7f33b80cbd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:54.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.128+0000 7f33be98f700 1 --2- 172.21.15.16:0/3382920871 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f33b80cb880 0x7f33b80cbd00 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f33ac009840 tx=0x7f33ac009b50 comp rx=0 tx=0).stop 2023-10-16T12:23:54.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.128+0000 7f33be98f700 1 -- 172.21.15.16:0/3382920871 shutdown_connections 2023-10-16T12:23:54.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.128+0000 7f33be98f700 1 --2- 172.21.15.16:0/3382920871 >> v2:172.21.15.49:3300/0 conn(0x7f33b815b070 0x7f33b815b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:54.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.128+0000 7f33be98f700 1 --2- 172.21.15.16:0/3382920871 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f33b80cb880 0x7f33b80cbd00 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:54.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.128+0000 7f33be98f700 1 -- 172.21.15.16:0/3382920871 >> 172.21.15.16:0/3382920871 conn(0x7f33b80b1f40 msgr2=0x7f33b80b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:54.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.128+0000 7f33be98f700 1 -- 172.21.15.16:0/3382920871 shutdown_connections 2023-10-16T12:23:54.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.128+0000 7f33be98f700 1 -- 172.21.15.16:0/3382920871 wait complete. 2023-10-16T12:23:54.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.129+0000 7f33be98f700 1 Processor -- start 2023-10-16T12:23:54.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.129+0000 7f33be98f700 1 -- start start 2023-10-16T12:23:54.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.130+0000 7f33be98f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f33b806da60 0x7f33b815d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:54.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.130+0000 7f33be98f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33b80cb880 0x7f33b81f6c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:54.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.130+0000 7f33be98f700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f33b815b070 0x7f33b8154730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:54.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.130+0000 7f33be98f700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33b815dbb0 con 0x7f33b80cb880 2023-10-16T12:23:54.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.130+0000 7f33be98f700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33b815d110 con 0x7f33b815b070 2023-10-16T12:23:54.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.130+0000 7f33be98f700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33b81f71b0 con 0x7f33b806da60 2023-10-16T12:23:54.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.130+0000 7f33b77fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33b80cb880 0x7f33b81f6c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:54.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.130+0000 7f33b77fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33b80cb880 0x7f33b81f6c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37468/0 (socket says 172.21.15.16:37468) 2023-10-16T12:23:54.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.130+0000 7f33b77fe700 1 -- 172.21.15.16:0/817329905 learned_addr learned my addr 172.21.15.16:0/817329905 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:54.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.130+0000 7f33b7fff700 1 --2- 172.21.15.16:0/817329905 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f33b806da60 0x7f33b815d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:54.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.130+0000 7f33bcf2c700 1 --2- 172.21.15.16:0/817329905 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f33b815b070 0x7f33b8154730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:54.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.131+0000 7f33b77fe700 1 -- 172.21.15.16:0/817329905 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f33b806da60 msgr2=0x7f33b815d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:54.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.131+0000 7f33b77fe700 1 --2- 172.21.15.16:0/817329905 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f33b806da60 0x7f33b815d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:54.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.131+0000 7f33b77fe700 1 -- 172.21.15.16:0/817329905 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f33b815b070 msgr2=0x7f33b8154730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:54.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.131+0000 7f33b77fe700 1 --2- 172.21.15.16:0/817329905 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f33b815b070 0x7f33b8154730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:54.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.131+0000 7f33b77fe700 1 -- 172.21.15.16:0/817329905 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33a80094a0 con 0x7f33b80cb880 2023-10-16T12:23:54.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.131+0000 7f33b77fe700 1 --2- 172.21.15.16:0/817329905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33b80cb880 0x7f33b81f6c70 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f33ac000c00 tx=0x7f33ac004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:54.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.131+0000 7f33b4ff9700 1 -- 172.21.15.16:0/817329905 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33ac004590 con 0x7f33b80cb880 2023-10-16T12:23:54.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.131+0000 7f33b4ff9700 1 -- 172.21.15.16:0/817329905 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33ac01f070 con 0x7f33b80cb880 2023-10-16T12:23:54.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.131+0000 7f33be98f700 1 -- 172.21.15.16:0/817329905 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33ac0094a0 con 0x7f33b80cb880 2023-10-16T12:23:54.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.131+0000 7f33b4ff9700 1 -- 172.21.15.16:0/817329905 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33ac0038e0 con 0x7f33b80cb880 2023-10-16T12:23:54.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.131+0000 7f33be98f700 1 -- 172.21.15.16:0/817329905 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33b8155330 con 0x7f33b80cb880 2023-10-16T12:23:54.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.132+0000 7f33be98f700 1 -- 172.21.15.16:0/817329905 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33b804ebb0 con 0x7f33b80cb880 2023-10-16T12:23:54.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.132+0000 7f33b4ff9700 1 -- 172.21.15.16:0/817329905 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f33ac003a40 con 0x7f33b80cb880 2023-10-16T12:23:54.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.133+0000 7f33b4ff9700 1 --2- 172.21.15.16:0/817329905 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f33a406b020 0x7f33a406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:54.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.133+0000 7f33b4ff9700 1 -- 172.21.15.16:0/817329905 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5908+0+0 (secure 0 0 0) 0x7f33ac021070 con 0x7f33b80cb880 2023-10-16T12:23:54.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.133+0000 7f33b7fff700 1 --2- 172.21.15.16:0/817329905 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f33a406b020 0x7f33a406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:54.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.133+0000 7f33b7fff700 1 --2- 172.21.15.16:0/817329905 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f33a406b020 0x7f33a406d4e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f33a8009cd0 tx=0x7f33a8009210 comp rx=0 tx=0).ready entity=mgr.5874 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:54.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.137+0000 7f33b4ff9700 1 -- 172.21.15.16:0/817329905 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f33ac055e30 con 0x7f33b80cb880 2023-10-16T12:23:54.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.342+0000 7f33be98f700 1 -- 172.21.15.16:0/817329905 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f33b80668b0 con 0x7f33b80cb880 2023-10-16T12:23:54.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.345+0000 7f33b4ff9700 1 -- 172.21.15.16:0/817329905 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v15) v1 ==== 120+0+0 (secure 0 0 0) 0x7f33ac059450 con 0x7f33b80cb880 2023-10-16T12:23:54.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.347+0000 7f33be98f700 1 -- 172.21.15.16:0/817329905 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f33a406b020 msgr2=0x7f33a406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:54.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.347+0000 7f33be98f700 1 --2- 172.21.15.16:0/817329905 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f33a406b020 0x7f33a406d4e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f33a8009cd0 tx=0x7f33a8009210 comp rx=0 tx=0).stop 2023-10-16T12:23:54.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.347+0000 7f33be98f700 1 -- 172.21.15.16:0/817329905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33b80cb880 msgr2=0x7f33b81f6c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:54.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.347+0000 7f33be98f700 1 --2- 172.21.15.16:0/817329905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33b80cb880 0x7f33b81f6c70 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f33ac000c00 tx=0x7f33ac004750 comp rx=0 tx=0).stop 2023-10-16T12:23:54.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.347+0000 7f33be98f700 1 -- 172.21.15.16:0/817329905 shutdown_connections 2023-10-16T12:23:54.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.347+0000 7f33be98f700 1 --2- 172.21.15.16:0/817329905 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f33a406b020 0x7f33a406d4e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:54.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.347+0000 7f33be98f700 1 --2- 172.21.15.16:0/817329905 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f33b815b070 0x7f33b8154730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:54.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.347+0000 7f33be98f700 1 --2- 172.21.15.16:0/817329905 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f33b806da60 0x7f33b815d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:54.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.347+0000 7f33be98f700 1 --2- 172.21.15.16:0/817329905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33b80cb880 0x7f33b81f6c70 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:54.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.347+0000 7f33be98f700 1 -- 172.21.15.16:0/817329905 >> 172.21.15.16:0/817329905 conn(0x7f33b80b1f40 msgr2=0x7f33b80d0d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:54.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.347+0000 7f33be98f700 1 -- 172.21.15.16:0/817329905 shutdown_connections 2023-10-16T12:23:54.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.347+0000 7f33be98f700 1 -- 172.21.15.16:0/817329905 wait complete. 2023-10-16T12:23:54.362 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates' 2023-10-16T12:23:54.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.486+0000 7f33db3c0700 1 Processor -- start 2023-10-16T12:23:54.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.486+0000 7f33db3c0700 1 -- start start 2023-10-16T12:23:54.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.487+0000 7f33db3c0700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f33d40cf880 0x7f33d40b52f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:54.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.487+0000 7f33db3c0700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f33d40b5830 0x7f33d415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:54.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.487+0000 7f33db3c0700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33d415bc80 con 0x7f33d40cf4a0 2023-10-16T12:23:54.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.487+0000 7f33db3c0700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33d415b070 con 0x7f33d40cf880 2023-10-16T12:23:54.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.487+0000 7f33db3c0700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33d415b350 con 0x7f33d40b5830 2023-10-16T12:23:54.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.487+0000 7f33d895b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f33d40cf880 0x7f33d40b52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:54.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.487+0000 7f33d895b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f33d40cf880 0x7f33d40b52f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37476/0 (socket says 172.21.15.16:37476) 2023-10-16T12:23:54.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.487+0000 7f33d895b700 1 -- 172.21.15.16:0/84028587 learned_addr learned my addr 172.21.15.16:0/84028587 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:54.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.487+0000 7f33d995d700 1 --2- 172.21.15.16:0/84028587 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f33d40b5830 0x7f33d415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:54.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.487+0000 7f33d895b700 1 -- 172.21.15.16:0/84028587 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f33d40b5830 msgr2=0x7f33d415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:54.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.487+0000 7f33d895b700 1 --2- 172.21.15.16:0/84028587 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f33d40b5830 0x7f33d415a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:54.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.487+0000 7f33d895b700 1 -- 172.21.15.16:0/84028587 >> v1:172.21.15.49:6789/0 conn(0x7f33d40cf4a0 legacy=0x7f33d40726e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:54.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.487+0000 7f33d895b700 1 -- 172.21.15.16:0/84028587 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33d415b630 con 0x7f33d40cf880 2023-10-16T12:23:54.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.487+0000 7f33d895b700 1 --2- 172.21.15.16:0/84028587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33d40cf880 0x7f33d40b52f0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f33c400bb90 tx=0x7f33c400bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b16d47303509fae server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:54.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.487+0000 7f33cbfff700 1 -- 172.21.15.16:0/84028587 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33c400dd10 con 0x7f33d40cf880 2023-10-16T12:23:54.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.487+0000 7f33cbfff700 1 -- 172.21.15.16:0/84028587 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33c400de70 con 0x7f33d40cf880 2023-10-16T12:23:54.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.487+0000 7f33cbfff700 1 -- 172.21.15.16:0/84028587 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33c4004230 con 0x7f33d40cf880 2023-10-16T12:23:54.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.488+0000 7f33db3c0700 1 -- 172.21.15.16:0/84028587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33d40cf880 msgr2=0x7f33d40b52f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:54.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.488+0000 7f33db3c0700 1 --2- 172.21.15.16:0/84028587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33d40cf880 0x7f33d40b52f0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f33c400bb90 tx=0x7f33c400bea0 comp rx=0 tx=0).stop 2023-10-16T12:23:54.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.488+0000 7f33db3c0700 1 -- 172.21.15.16:0/84028587 shutdown_connections 2023-10-16T12:23:54.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.488+0000 7f33db3c0700 1 --2- 172.21.15.16:0/84028587 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f33d40b5830 0x7f33d415a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:54.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.488+0000 7f33db3c0700 1 --2- 172.21.15.16:0/84028587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33d40cf880 0x7f33d40b52f0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:54.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.488+0000 7f33db3c0700 1 -- 172.21.15.16:0/84028587 >> 172.21.15.16:0/84028587 conn(0x7f33d40b1fd0 msgr2=0x7f33d40b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:54.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.488+0000 7f33db3c0700 1 -- 172.21.15.16:0/84028587 shutdown_connections 2023-10-16T12:23:54.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.488+0000 7f33db3c0700 1 -- 172.21.15.16:0/84028587 wait complete. 2023-10-16T12:23:54.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.489+0000 7f33db3c0700 1 Processor -- start 2023-10-16T12:23:54.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.489+0000 7f33db3c0700 1 -- start start 2023-10-16T12:23:54.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.490+0000 7f33db3c0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33d40b5830 0x7f33d40ad8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:54.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.490+0000 7f33db3c0700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f33d40cf4a0 0x7f33d41506b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:54.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.490+0000 7f33db3c0700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f33d40cf880 0x7f33d40b0410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:54.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.490+0000 7f33db3c0700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33d40ac8d0 con 0x7f33d40b5830 2023-10-16T12:23:54.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.490+0000 7f33db3c0700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33d40ad450 con 0x7f33d40cf880 2023-10-16T12:23:54.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.490+0000 7f33db3c0700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33d40acb20 con 0x7f33d40cf4a0 2023-10-16T12:23:54.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.490+0000 7f33d915c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33d40b5830 0x7f33d40ad8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:54.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.490+0000 7f33d915c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33d40b5830 0x7f33d40ad8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37480/0 (socket says 172.21.15.16:37480) 2023-10-16T12:23:54.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.490+0000 7f33d915c700 1 -- 172.21.15.16:0/2068101258 learned_addr learned my addr 172.21.15.16:0/2068101258 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:54.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.490+0000 7f33d895b700 1 --2- 172.21.15.16:0/2068101258 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f33d40cf4a0 0x7f33d41506b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:54.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.490+0000 7f33d915c700 1 -- 172.21.15.16:0/2068101258 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f33d40cf4a0 msgr2=0x7f33d41506b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:54.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.490+0000 7f33d915c700 1 --2- 172.21.15.16:0/2068101258 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f33d40cf4a0 0x7f33d41506b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:54.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.490+0000 7f33d915c700 1 -- 172.21.15.16:0/2068101258 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f33d40cf880 msgr2=0x7f33d40b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:54.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.490+0000 7f33d915c700 1 --2- 172.21.15.16:0/2068101258 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f33d40cf880 0x7f33d40b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:54.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.490+0000 7f33d915c700 1 -- 172.21.15.16:0/2068101258 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33c400b7f0 con 0x7f33d40b5830 2023-10-16T12:23:54.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.490+0000 7f33d915c700 1 --2- 172.21.15.16:0/2068101258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33d40b5830 0x7f33d40ad8e0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f33d000c500 tx=0x7f33d000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:54.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.491+0000 7f33c9ffb700 1 -- 172.21.15.16:0/2068101258 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33d0011790 con 0x7f33d40b5830 2023-10-16T12:23:54.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.491+0000 7f33c9ffb700 1 -- 172.21.15.16:0/2068101258 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33d00118f0 con 0x7f33d40b5830 2023-10-16T12:23:54.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.491+0000 7f33db3c0700 1 -- 172.21.15.16:0/2068101258 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33d40ace00 con 0x7f33d40b5830 2023-10-16T12:23:54.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.491+0000 7f33c9ffb700 1 -- 172.21.15.16:0/2068101258 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33d0011ca0 con 0x7f33d40b5830 2023-10-16T12:23:54.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.491+0000 7f33db3c0700 1 -- 172.21.15.16:0/2068101258 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33d4150bf0 con 0x7f33d40b5830 2023-10-16T12:23:54.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.492+0000 7f33db3c0700 1 -- 172.21.15.16:0/2068101258 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33d404ebb0 con 0x7f33d40b5830 2023-10-16T12:23:54.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.492+0000 7f33c9ffb700 1 -- 172.21.15.16:0/2068101258 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f33d001c460 con 0x7f33d40b5830 2023-10-16T12:23:54.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.493+0000 7f33c9ffb700 1 --2- 172.21.15.16:0/2068101258 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f33b806b020 0x7f33b806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:54.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.493+0000 7f33c9ffb700 1 -- 172.21.15.16:0/2068101258 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5908+0+0 (secure 0 0 0) 0x7f33d0089810 con 0x7f33d40b5830 2023-10-16T12:23:54.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.493+0000 7f33d895b700 1 --2- 172.21.15.16:0/2068101258 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f33b806b020 0x7f33b806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:54.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.493+0000 7f33d895b700 1 --2- 172.21.15.16:0/2068101258 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f33b806b020 0x7f33b806d4e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f33d41f6880 tx=0x7f33c4013000 comp rx=0 tx=0).ready entity=mgr.5874 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:54.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.497+0000 7f33c9ffb700 1 -- 172.21.15.16:0/2068101258 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f33d00545b0 con 0x7f33d40b5830 2023-10-16T12:23:54.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:54.784+0000 7f33db3c0700 1 -- 172.21.15.16:0/2068101258 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]} v 0) v1 -- 0x7f33d40b01f0 con 0x7f33d40b5830 2023-10-16T12:23:55.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.235+0000 7f33c9ffb700 1 -- 172.21.15.16:0/2068101258 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]}]=0 v478) v1 ==== 158+0+0 (secure 0 0 0) 0x7f33d0057bd0 con 0x7f33d40b5830 2023-10-16T12:23:55.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.237+0000 7f33db3c0700 1 -- 172.21.15.16:0/2068101258 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f33b806b020 msgr2=0x7f33b806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:55.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.237+0000 7f33db3c0700 1 --2- 172.21.15.16:0/2068101258 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f33b806b020 0x7f33b806d4e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f33d41f6880 tx=0x7f33c4013000 comp rx=0 tx=0).stop 2023-10-16T12:23:55.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.237+0000 7f33db3c0700 1 -- 172.21.15.16:0/2068101258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33d40b5830 msgr2=0x7f33d40ad8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:55.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.237+0000 7f33db3c0700 1 --2- 172.21.15.16:0/2068101258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33d40b5830 0x7f33d40ad8e0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f33d000c500 tx=0x7f33d000c8c0 comp rx=0 tx=0).stop 2023-10-16T12:23:55.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.237+0000 7f33db3c0700 1 -- 172.21.15.16:0/2068101258 shutdown_connections 2023-10-16T12:23:55.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.237+0000 7f33db3c0700 1 --2- 172.21.15.16:0/2068101258 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f33b806b020 0x7f33b806d4e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:55.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.237+0000 7f33db3c0700 1 --2- 172.21.15.16:0/2068101258 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f33d40cf880 0x7f33d40b0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:55.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.237+0000 7f33db3c0700 1 --2- 172.21.15.16:0/2068101258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33d40b5830 0x7f33d40ad8e0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:55.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.237+0000 7f33db3c0700 1 --2- 172.21.15.16:0/2068101258 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f33d40cf4a0 0x7f33d41506b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:55.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.237+0000 7f33db3c0700 1 -- 172.21.15.16:0/2068101258 >> 172.21.15.16:0/2068101258 conn(0x7f33d40b1fd0 msgr2=0x7f33d415cd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:55.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.238+0000 7f33db3c0700 1 -- 172.21.15.16:0/2068101258 shutdown_connections 2023-10-16T12:23:55.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.238+0000 7f33db3c0700 1 -- 172.21.15.16:0/2068101258 wait complete. 2023-10-16T12:23:55.254 INFO:tasks.cephfs_test_runner:Test concurrent snap creates in same file-system without db issues ... ok 2023-10-16T12:23:55.254 INFO:tasks.cephfs_test_runner:Starting test: test_multi_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test exisitence of multiple scheduled snapshots 2023-10-16T12:23:55.254 INFO:tasks.cephfs_test_runner:test_multi_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-16T12:23:55.255 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:23:55.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.383+0000 7f361b04c700 1 Processor -- start 2023-10-16T12:23:55.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.383+0000 7f361b04c700 1 -- start start 2023-10-16T12:23:55.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.383+0000 7f361b04c700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f36140cf4a0 0x7f36140726e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:55.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.383+0000 7f361b04c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f36140b5a00 0x7f361415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:55.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.384+0000 7f361b04c700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f361415bc80 con 0x7f36140cf880 2023-10-16T12:23:55.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.384+0000 7f361b04c700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36140b5e70 con 0x7f36140b5a00 2023-10-16T12:23:55.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.384+0000 7f361b04c700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3614072c20 con 0x7f36140cf4a0 2023-10-16T12:23:55.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.384+0000 7f36195e9700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f36140b5a00 0x7f361415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:55.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.384+0000 7f36195e9700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f36140b5a00 0x7f361415a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37492/0 (socket says 172.21.15.16:37492) 2023-10-16T12:23:55.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.384+0000 7f36195e9700 1 -- 172.21.15.16:0/1445006088 learned_addr learned my addr 172.21.15.16:0/1445006088 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:55.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.384+0000 7f3618de8700 1 --2- 172.21.15.16:0/1445006088 >> v2:172.21.15.49:3300/0 conn(0x7f36140cf4a0 0x7f36140726e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:55.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.384+0000 7f36195e9700 1 -- 172.21.15.16:0/1445006088 >> v2:172.21.15.49:3300/0 conn(0x7f36140cf4a0 msgr2=0x7f36140726e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:55.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.384+0000 7f36195e9700 1 --2- 172.21.15.16:0/1445006088 >> v2:172.21.15.49:3300/0 conn(0x7f36140cf4a0 0x7f36140726e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:55.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.384+0000 7f36195e9700 1 -- 172.21.15.16:0/1445006088 >> v1:172.21.15.16:6789/0 conn(0x7f36140cf880 legacy=0x7f36140b52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:55.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.384+0000 7f36137fe700 1 -- 172.21.15.16:0/1445006088 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 438856994 0 0) 0x7f361415bc80 con 0x7f36140cf880 2023-10-16T12:23:55.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.384+0000 7f36195e9700 1 -- 172.21.15.16:0/1445006088 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f361415b070 con 0x7f36140b5a00 2023-10-16T12:23:55.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.384+0000 7f3618de8700 1 --2- 172.21.15.16:0/1445006088 >> v2:172.21.15.49:3300/0 conn(0x7f36140cf4a0 0x7f36140726e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:23:55.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.384+0000 7f36195e9700 1 --2- 172.21.15.16:0/1445006088 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36140b5a00 0x7f361415a090 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f36140b2d20 tx=0x7f360c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c0bb044d69503d6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:55.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.384+0000 7f36137fe700 1 -- 172.21.15.16:0/1445006088 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f360c00e070 con 0x7f36140b5a00 2023-10-16T12:23:55.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.384+0000 7f36137fe700 1 -- 172.21.15.16:0/1445006088 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f360c00b9c0 con 0x7f36140b5a00 2023-10-16T12:23:55.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.384+0000 7f36137fe700 1 -- 172.21.15.16:0/1445006088 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f360c00bd70 con 0x7f36140b5a00 2023-10-16T12:23:55.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.385+0000 7f361b04c700 1 -- 172.21.15.16:0/1445006088 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36140b5a00 msgr2=0x7f361415a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:55.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.385+0000 7f361b04c700 1 --2- 172.21.15.16:0/1445006088 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36140b5a00 0x7f361415a090 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f36140b2d20 tx=0x7f360c00b330 comp rx=0 tx=0).stop 2023-10-16T12:23:55.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.385+0000 7f361b04c700 1 -- 172.21.15.16:0/1445006088 shutdown_connections 2023-10-16T12:23:55.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.385+0000 7f361b04c700 1 --2- 172.21.15.16:0/1445006088 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36140b5a00 0x7f361415a090 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:55.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.385+0000 7f361b04c700 1 --2- 172.21.15.16:0/1445006088 >> v2:172.21.15.49:3300/0 conn(0x7f36140cf4a0 0x7f36140726e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:55.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.385+0000 7f361b04c700 1 -- 172.21.15.16:0/1445006088 >> 172.21.15.16:0/1445006088 conn(0x7f36140b1fd0 msgr2=0x7f36140b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:55.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.385+0000 7f361b04c700 1 -- 172.21.15.16:0/1445006088 shutdown_connections 2023-10-16T12:23:55.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.385+0000 7f361b04c700 1 -- 172.21.15.16:0/1445006088 wait complete. 2023-10-16T12:23:55.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.386+0000 7f361b04c700 1 Processor -- start 2023-10-16T12:23:55.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.386+0000 7f361b04c700 1 -- start start 2023-10-16T12:23:55.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.387+0000 7f361b04c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f36140b5a00 0x7f36140c69f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:55.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.387+0000 7f361b04c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36140cf4a0 0x7f36140b2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:55.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.387+0000 7f361b04c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f36140cf880 0x7f36140c3300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:55.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.387+0000 7f361b04c700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36140c6f30 con 0x7f36140cf4a0 2023-10-16T12:23:55.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.387+0000 7f361b04c700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36140b3a10 con 0x7f36140cf880 2023-10-16T12:23:55.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.387+0000 7f361b04c700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36140b3cf0 con 0x7f36140b5a00 2023-10-16T12:23:55.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.387+0000 7f3613fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36140cf4a0 0x7f36140b2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:55.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.387+0000 7f3613fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36140cf4a0 0x7f36140b2fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37494/0 (socket says 172.21.15.16:37494) 2023-10-16T12:23:55.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.387+0000 7f3613fff700 1 -- 172.21.15.16:0/2546515957 learned_addr learned my addr 172.21.15.16:0/2546515957 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:55.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.388+0000 7f3618de8700 1 --2- 172.21.15.16:0/2546515957 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f36140b5a00 0x7f36140c69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:55.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.388+0000 7f36195e9700 1 --2- 172.21.15.16:0/2546515957 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f36140cf880 0x7f36140c3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:55.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.388+0000 7f3613fff700 1 -- 172.21.15.16:0/2546515957 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f36140b5a00 msgr2=0x7f36140c69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:55.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.388+0000 7f3613fff700 1 --2- 172.21.15.16:0/2546515957 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f36140b5a00 0x7f36140c69f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:55.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.388+0000 7f3613fff700 1 -- 172.21.15.16:0/2546515957 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f36140cf880 msgr2=0x7f36140c3300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:55.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.388+0000 7f3613fff700 1 --2- 172.21.15.16:0/2546515957 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f36140cf880 0x7f36140c3300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:55.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.388+0000 7f3613fff700 1 -- 172.21.15.16:0/2546515957 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f360c009b30 con 0x7f36140cf4a0 2023-10-16T12:23:55.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.388+0000 7f3618de8700 1 --2- 172.21.15.16:0/2546515957 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f36140b5a00 0x7f36140c69f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:23:55.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.388+0000 7f36195e9700 1 --2- 172.21.15.16:0/2546515957 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f36140cf880 0x7f36140c3300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:23:55.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.388+0000 7f3613fff700 1 --2- 172.21.15.16:0/2546515957 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36140cf4a0 0x7f36140b2fc0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f360000c550 tx=0x7f360000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:55.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.388+0000 7f36117fa700 1 -- 172.21.15.16:0/2546515957 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36000054e0 con 0x7f36140cf4a0 2023-10-16T12:23:55.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.388+0000 7f36117fa700 1 -- 172.21.15.16:0/2546515957 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3600003680 con 0x7f36140cf4a0 2023-10-16T12:23:55.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.388+0000 7f361b04c700 1 -- 172.21.15.16:0/2546515957 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36140b3fd0 con 0x7f36140cf4a0 2023-10-16T12:23:55.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.388+0000 7f36117fa700 1 -- 172.21.15.16:0/2546515957 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3600003a30 con 0x7f36140cf4a0 2023-10-16T12:23:55.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.388+0000 7f361b04c700 1 -- 172.21.15.16:0/2546515957 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36140b4590 con 0x7f36140cf4a0 2023-10-16T12:23:55.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.389+0000 7f361b04c700 1 -- 172.21.15.16:0/2546515957 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f361404ebb0 con 0x7f36140cf4a0 2023-10-16T12:23:55.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.389+0000 7f36117fa700 1 -- 172.21.15.16:0/2546515957 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3600003b90 con 0x7f36140cf4a0 2023-10-16T12:23:55.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.390+0000 7f36117fa700 1 --2- 172.21.15.16:0/2546515957 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f35f806b020 0x7f35f806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:55.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.390+0000 7f36117fa700 1 -- 172.21.15.16:0/2546515957 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5908+0+0 (secure 0 0 0) 0x7f3600088d60 con 0x7f36140cf4a0 2023-10-16T12:23:55.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.390+0000 7f3618de8700 1 --2- 172.21.15.16:0/2546515957 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f35f806b020 0x7f35f806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:55.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.390+0000 7f3618de8700 1 --2- 172.21.15.16:0/2546515957 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f35f806b020 0x7f35f806d4e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f36080089b0 tx=0x7f3608008040 comp rx=0 tx=0).ready entity=mgr.5874 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:55.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.394+0000 7f36117fa700 1 -- 172.21.15.16:0/2546515957 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3600053ba0 con 0x7f36140cf4a0 2023-10-16T12:23:55.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.597+0000 7f361b04c700 1 -- 172.21.15.16:0/2546515957 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f36140b4750 con 0x7f36140cf4a0 2023-10-16T12:23:55.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.598+0000 7f36117fa700 1 -- 172.21.15.16:0/2546515957 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v60) v1 ==== 74+0+14021 (secure 0 0 0) 0x7f3600017030 con 0x7f36140cf4a0 2023-10-16T12:23:55.600 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:23:55.600 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":60,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:23:53.229025+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.49:6835/2788396302":"2023-10-17T12:23:49.925846+0000","172.21.15.16:6832/365921436":"2023-10-17T12:12:15.525647+0000","172.21.15.16:0/3547353312":"2023-10-17T12:12:15.525647+0000","172.21.15.16:6832/1707126831":"2023-10-17T12:23:48.841815+0000","172.21.15.16:0/1893346008":"2023-10-17T12:12:15.525647+0000","172.21.15.16:6833/365921436":"2023-10-17T12:12:15.525647+0000","172.21.15.16:0/1802260224":"2023-10-17T12:23:48.841815+0000","172.21.15.16:0/675356848":"2023-10-17T12:12:15.525647+0000","172.21.15.16:6833/1707126831":"2023-10-17T12:23:48.841815+0000","172.21.15.16:0/1300148187":"2023-10-17T12:12:15.525647+0000","172.21.15.16:0/2158072405":"2023-10-17T12:23:48.841815+0000","172.21.15.16:0/109549027":"2023-10-17T12:23:48.841815+0000","172.21.15.16:0/2779790691":"2023-10-17T12:23:48.841815+0000","172.21.15.16:0/2241612272":"2023-10-17T12:23:48.841815+0000","172.21.15.16:0/1886125045":"2023-10-17T12:23:48.841815+0000","172.21.15.16:0/3754942648":"2023-10-17T12:23:48.841815+0000","172.21.15.49:6833/2788396302":"2023-10-17T12:23:49.925846+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:23:55.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.600+0000 7f361b04c700 1 -- 172.21.15.16:0/2546515957 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f35f806b020 msgr2=0x7f35f806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:55.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.600+0000 7f361b04c700 1 --2- 172.21.15.16:0/2546515957 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f35f806b020 0x7f35f806d4e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f36080089b0 tx=0x7f3608008040 comp rx=0 tx=0).stop 2023-10-16T12:23:55.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.600+0000 7f361b04c700 1 -- 172.21.15.16:0/2546515957 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36140cf4a0 msgr2=0x7f36140b2fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:55.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.600+0000 7f361b04c700 1 --2- 172.21.15.16:0/2546515957 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36140cf4a0 0x7f36140b2fc0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f360000c550 tx=0x7f360000c910 comp rx=0 tx=0).stop 2023-10-16T12:23:55.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.600+0000 7f361b04c700 1 -- 172.21.15.16:0/2546515957 shutdown_connections 2023-10-16T12:23:55.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.600+0000 7f361b04c700 1 --2- 172.21.15.16:0/2546515957 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f35f806b020 0x7f35f806d4e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:55.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.600+0000 7f361b04c700 1 --2- 172.21.15.16:0/2546515957 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f36140cf880 0x7f36140c3300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:55.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.600+0000 7f361b04c700 1 --2- 172.21.15.16:0/2546515957 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f36140b5a00 0x7f36140c69f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:55.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.600+0000 7f361b04c700 1 --2- 172.21.15.16:0/2546515957 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36140cf4a0 0x7f36140b2fc0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:55.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.600+0000 7f361b04c700 1 -- 172.21.15.16:0/2546515957 >> 172.21.15.16:0/2546515957 conn(0x7f36140b1fd0 msgr2=0x7f36140ce4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:55.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.600+0000 7f361b04c700 1 -- 172.21.15.16:0/2546515957 shutdown_connections 2023-10-16T12:23:55.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.600+0000 7f361b04c700 1 -- 172.21.15.16:0/2546515957 wait complete. 2023-10-16T12:23:55.614 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T12:23:55.615 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:23:55.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.740+0000 7f17efbce700 1 Processor -- start 2023-10-16T12:23:55.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.741+0000 7f17efbce700 1 -- start start 2023-10-16T12:23:55.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.741+0000 7f17efbce700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f17e806e570 0x7f17e815e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:55.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.741+0000 7f17efbce700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17e815b6a0 con 0x7f17e806a7e0 2023-10-16T12:23:55.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.741+0000 7f17efbce700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17e8161b20 con 0x7f17e815b2c0 2023-10-16T12:23:55.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.741+0000 7f17efbce700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17e8161e00 con 0x7f17e806e570 2023-10-16T12:23:55.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.741+0000 7f17ee16b700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f17e806a7e0 0x7f17e8071a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:55012/0 (socket says 172.21.15.16:55012) 2023-10-16T12:23:55.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.741+0000 7f17ee16b700 1 -- 172.21.15.16:0/960186232 learned_addr learned my addr 172.21.15.16:0/960186232 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:23:55.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.742+0000 7f17ed169700 1 --2- 172.21.15.16:0/960186232 >> v2:172.21.15.49:3300/0 conn(0x7f17e806e570 0x7f17e815e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:55.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.742+0000 7f17ec968700 1 -- 172.21.15.16:0/960186232 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3367778362 0 0) 0x7f17e815b6a0 con 0x7f17e806a7e0 2023-10-16T12:23:55.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.742+0000 7f17ec968700 1 -- 172.21.15.16:0/960186232 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17d40035b0 con 0x7f17e806a7e0 2023-10-16T12:23:55.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.742+0000 7f17ec968700 1 -- 172.21.15.16:0/960186232 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1785170071 0 0) 0x7f17e8161b20 con 0x7f17e815b2c0 2023-10-16T12:23:55.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.742+0000 7f17ec968700 1 -- 172.21.15.16:0/960186232 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17e815b6a0 con 0x7f17e815b2c0 2023-10-16T12:23:55.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.742+0000 7f17ec968700 1 -- 172.21.15.16:0/960186232 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 878952468 0 0) 0x7f17d40035b0 con 0x7f17e806a7e0 2023-10-16T12:23:55.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.742+0000 7f17ed169700 1 -- 172.21.15.16:0/960186232 >> v1:172.21.15.49:6789/0 conn(0x7f17e815b2c0 legacy=0x7f17e8161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:55.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.742+0000 7f17ed169700 1 -- 172.21.15.16:0/960186232 >> v1:172.21.15.16:6789/0 conn(0x7f17e806a7e0 legacy=0x7f17e8071a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:55.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.742+0000 7f17ed169700 1 -- 172.21.15.16:0/960186232 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17e81620e0 con 0x7f17e806e570 2023-10-16T12:23:55.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.742+0000 7f17ed169700 1 --2- 172.21.15.16:0/960186232 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f17e806e570 0x7f17e815e5a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f17e4009840 tx=0x7f17e4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=26efb55aeb18a88b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:55.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.742+0000 7f17ec968700 1 -- 172.21.15.16:0/960186232 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17e00019f0 con 0x7f17e806e570 2023-10-16T12:23:55.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.742+0000 7f17ec968700 1 -- 172.21.15.16:0/960186232 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17e4004030 con 0x7f17e806e570 2023-10-16T12:23:55.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.742+0000 7f17ec968700 1 -- 172.21.15.16:0/960186232 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17e40043c0 con 0x7f17e806e570 2023-10-16T12:23:55.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.743+0000 7f17efbce700 1 -- 172.21.15.16:0/960186232 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f17e806e570 msgr2=0x7f17e815e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:55.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.743+0000 7f17efbce700 1 --2- 172.21.15.16:0/960186232 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f17e806e570 0x7f17e815e5a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f17e4009840 tx=0x7f17e4009b50 comp rx=0 tx=0).stop 2023-10-16T12:23:55.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.743+0000 7f17efbce700 1 -- 172.21.15.16:0/960186232 shutdown_connections 2023-10-16T12:23:55.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.743+0000 7f17efbce700 1 --2- 172.21.15.16:0/960186232 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f17e806e570 0x7f17e815e5a0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:55.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.743+0000 7f17efbce700 1 -- 172.21.15.16:0/960186232 >> 172.21.15.16:0/960186232 conn(0x7f17e80b1f40 msgr2=0x7f17e80b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:55.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.743+0000 7f17efbce700 1 -- 172.21.15.16:0/960186232 shutdown_connections 2023-10-16T12:23:55.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.743+0000 7f17efbce700 1 -- 172.21.15.16:0/960186232 wait complete. 2023-10-16T12:23:55.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.744+0000 7f17efbce700 1 Processor -- start 2023-10-16T12:23:55.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.744+0000 7f17efbce700 1 -- start start 2023-10-16T12:23:55.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.745+0000 7f17efbce700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f17e806a7e0 0x7f17e80c6a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:55.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.745+0000 7f17efbce700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f17e806e570 0x7f17e80c5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:55.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.745+0000 7f17efbce700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17e815b2c0 0x7f17e80c7e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:55.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.745+0000 7f17efbce700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17e80c3b90 con 0x7f17e815b2c0 2023-10-16T12:23:55.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.745+0000 7f17efbce700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17e8159e40 con 0x7f17e806e570 2023-10-16T12:23:55.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.745+0000 7f17efbce700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17e80c3e30 con 0x7f17e806a7e0 2023-10-16T12:23:55.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.745+0000 7f17ee16b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17e815b2c0 0x7f17e80c7e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:55.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.745+0000 7f17ee16b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17e815b2c0 0x7f17e80c7e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37512/0 (socket says 172.21.15.16:37512) 2023-10-16T12:23:55.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.745+0000 7f17ee16b700 1 -- 172.21.15.16:0/1329550972 learned_addr learned my addr 172.21.15.16:0/1329550972 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:55.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.745+0000 7f17ed169700 1 --2- 172.21.15.16:0/1329550972 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f17e806e570 0x7f17e80c5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:55.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.745+0000 7f17ed96a700 1 --2- 172.21.15.16:0/1329550972 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f17e806a7e0 0x7f17e80c6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:55.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.745+0000 7f17ee16b700 1 -- 172.21.15.16:0/1329550972 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f17e806a7e0 msgr2=0x7f17e80c6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:55.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.745+0000 7f17ee16b700 1 --2- 172.21.15.16:0/1329550972 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f17e806a7e0 0x7f17e80c6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:55.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.745+0000 7f17ee16b700 1 -- 172.21.15.16:0/1329550972 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f17e806e570 msgr2=0x7f17e80c5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:55.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.745+0000 7f17ee16b700 1 --2- 172.21.15.16:0/1329550972 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f17e806e570 0x7f17e80c5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:55.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.745+0000 7f17ee16b700 1 -- 172.21.15.16:0/1329550972 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17e40094a0 con 0x7f17e815b2c0 2023-10-16T12:23:55.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.745+0000 7f17ed96a700 1 --2- 172.21.15.16:0/1329550972 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f17e806a7e0 0x7f17e80c6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:23:55.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.746+0000 7f17ee16b700 1 --2- 172.21.15.16:0/1329550972 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17e815b2c0 0x7f17e80c7e40 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f17e000cc50 tx=0x7f17e000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:55.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.746+0000 7f17de7fc700 1 -- 172.21.15.16:0/1329550972 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17e000ab80 con 0x7f17e815b2c0 2023-10-16T12:23:55.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.746+0000 7f17efbce700 1 -- 172.21.15.16:0/1329550972 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17e80c6420 con 0x7f17e815b2c0 2023-10-16T12:23:55.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.746+0000 7f17de7fc700 1 -- 172.21.15.16:0/1329550972 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17e000ace0 con 0x7f17e815b2c0 2023-10-16T12:23:55.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.746+0000 7f17de7fc700 1 -- 172.21.15.16:0/1329550972 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17e0007310 con 0x7f17e815b2c0 2023-10-16T12:23:55.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.746+0000 7f17efbce700 1 -- 172.21.15.16:0/1329550972 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17e815a090 con 0x7f17e815b2c0 2023-10-16T12:23:55.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.747+0000 7f17efbce700 1 -- 172.21.15.16:0/1329550972 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17e815ea60 con 0x7f17e815b2c0 2023-10-16T12:23:55.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.747+0000 7f17de7fc700 1 -- 172.21.15.16:0/1329550972 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f17e0007470 con 0x7f17e815b2c0 2023-10-16T12:23:55.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.748+0000 7f17de7fc700 1 --2- 172.21.15.16:0/1329550972 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f17cc06b020 0x7f17cc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:55.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.748+0000 7f17de7fc700 1 -- 172.21.15.16:0/1329550972 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5908+0+0 (secure 0 0 0) 0x7f17e0089390 con 0x7f17e815b2c0 2023-10-16T12:23:55.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.748+0000 7f17ed96a700 1 --2- 172.21.15.16:0/1329550972 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f17cc06b020 0x7f17cc06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:55.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.748+0000 7f17ed96a700 1 --2- 172.21.15.16:0/1329550972 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f17cc06b020 0x7f17cc06d4e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f17d80011e0 tx=0x7f17d8009040 comp rx=0 tx=0).ready entity=mgr.5874 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:55.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.752+0000 7f17de7fc700 1 -- 172.21.15.16:0/1329550972 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f17e0054270 con 0x7f17e815b2c0 2023-10-16T12:23:55.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.956+0000 7f17efbce700 1 -- 172.21.15.16:0/1329550972 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f17e804ebb0 con 0x7f17e815b2c0 2023-10-16T12:23:55.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.956+0000 7f17de7fc700 1 -- 172.21.15.16:0/1329550972 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v60) v1 ==== 93+0+10 (secure 0 0 0) 0x7f17e0057890 con 0x7f17e815b2c0 2023-10-16T12:23:55.959 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T12:23:55.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.959+0000 7f17efbce700 1 -- 172.21.15.16:0/1329550972 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f17cc06b020 msgr2=0x7f17cc06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:55.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.959+0000 7f17efbce700 1 --2- 172.21.15.16:0/1329550972 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f17cc06b020 0x7f17cc06d4e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f17d80011e0 tx=0x7f17d8009040 comp rx=0 tx=0).stop 2023-10-16T12:23:55.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.959+0000 7f17efbce700 1 -- 172.21.15.16:0/1329550972 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17e815b2c0 msgr2=0x7f17e80c7e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:55.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.959+0000 7f17efbce700 1 --2- 172.21.15.16:0/1329550972 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17e815b2c0 0x7f17e80c7e40 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f17e000cc50 tx=0x7f17e000cf60 comp rx=0 tx=0).stop 2023-10-16T12:23:55.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.959+0000 7f17efbce700 1 -- 172.21.15.16:0/1329550972 shutdown_connections 2023-10-16T12:23:55.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.959+0000 7f17efbce700 1 --2- 172.21.15.16:0/1329550972 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f17cc06b020 0x7f17cc06d4e0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:55.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.959+0000 7f17efbce700 1 --2- 172.21.15.16:0/1329550972 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17e815b2c0 0x7f17e80c7e40 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:55.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.959+0000 7f17efbce700 1 --2- 172.21.15.16:0/1329550972 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f17e806a7e0 0x7f17e80c6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:55.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.959+0000 7f17efbce700 1 --2- 172.21.15.16:0/1329550972 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f17e806e570 0x7f17e80c5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:55.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.959+0000 7f17efbce700 1 -- 172.21.15.16:0/1329550972 >> 172.21.15.16:0/1329550972 conn(0x7f17e80b1f40 msgr2=0x7f17e80cc1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:55.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.959+0000 7f17efbce700 1 -- 172.21.15.16:0/1329550972 shutdown_connections 2023-10-16T12:23:55.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:55.959+0000 7f17efbce700 1 -- 172.21.15.16:0/1329550972 wait complete. 2023-10-16T12:23:55.973 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule' 2023-10-16T12:23:56.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:56.097+0000 7fbe9d46b700 1 Processor -- start 2023-10-16T12:23:56.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:56.097+0000 7fbe9d46b700 1 -- start start 2023-10-16T12:23:56.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:56.098+0000 7fbe9d46b700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fbe980cbed0 0x7fbe9806eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:56.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:56.098+0000 7fbe9d46b700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe9806f0a0 con 0x7fbe9815a870 2023-10-16T12:23:56.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:56.098+0000 7fbe9d46b700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe9806f380 con 0x7fbe9806d350 2023-10-16T12:23:56.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:56.098+0000 7fbe9d46b700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe9815ac50 con 0x7fbe980cbed0 2023-10-16T12:23:56.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:56.098+0000 7fbe96ffd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fbe9815a870 0x7fbe980cb7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:55022/0 (socket says 172.21.15.16:55022) 2023-10-16T12:23:56.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:56.098+0000 7fbe96ffd700 1 -- 172.21.15.16:0/3452792664 learned_addr learned my addr 172.21.15.16:0/3452792664 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:23:56.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:56.098+0000 7fbe977fe700 1 --2- 172.21.15.16:0/3452792664 >> v2:172.21.15.49:3300/0 conn(0x7fbe980cbed0 0x7fbe9806eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:56.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:56.099+0000 7fbe95ffb700 1 -- 172.21.15.16:0/3452792664 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 685878459 0 0) 0x7fbe9806f0a0 con 0x7fbe9815a870 2023-10-16T12:23:56.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:56.099+0000 7fbe95ffb700 1 -- 172.21.15.16:0/3452792664 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe800035b0 con 0x7fbe9815a870 2023-10-16T12:23:56.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:56.099+0000 7fbe95ffb700 1 -- 172.21.15.16:0/3452792664 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2894771738 0 0) 0x7fbe9806f380 con 0x7fbe9806d350 2023-10-16T12:23:56.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:56.099+0000 7fbe95ffb700 1 -- 172.21.15.16:0/3452792664 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe9806f0a0 con 0x7fbe9806d350 2023-10-16T12:23:56.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:56.099+0000 7fbe977fe700 1 -- 172.21.15.16:0/3452792664 >> v1:172.21.15.49:6789/0 conn(0x7fbe9806d350 legacy=0x7fbe980cde10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:56.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:56.099+0000 7fbe977fe700 1 -- 172.21.15.16:0/3452792664 >> v1:172.21.15.16:6789/0 conn(0x7fbe9815a870 legacy=0x7fbe980cb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:56.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:56.099+0000 7fbe977fe700 1 -- 172.21.15.16:0/3452792664 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe9806f660 con 0x7fbe980cbed0 2023-10-16T12:23:56.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:56.099+0000 7fbe977fe700 1 --2- 172.21.15.16:0/3452792664 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbe980cbed0 0x7fbe9806eaa0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fbe90009d90 tx=0x7fbe9000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9dd115889c77c207 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:56.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:56.099+0000 7fbe95ffb700 1 -- 172.21.15.16:0/3452792664 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe9000e070 con 0x7fbe980cbed0 2023-10-16T12:23:56.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:56.099+0000 7fbe95ffb700 1 -- 172.21.15.16:0/3452792664 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe9000bb20 con 0x7fbe980cbed0 2023-10-16T12:23:56.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:56.099+0000 7fbe95ffb700 1 -- 172.21.15.16:0/3452792664 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe9000be90 con 0x7fbe980cbed0 2023-10-16T12:23:56.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:56.100+0000 7fbe9d46b700 1 -- 172.21.15.16:0/3452792664 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbe980cbed0 msgr2=0x7fbe9806eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:56.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:56.100+0000 7fbe9d46b700 1 --2- 172.21.15.16:0/3452792664 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbe980cbed0 0x7fbe9806eaa0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fbe90009d90 tx=0x7fbe9000b330 comp rx=0 tx=0).stop 2023-10-16T12:23:56.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:56.100+0000 7fbe9d46b700 1 -- 172.21.15.16:0/3452792664 shutdown_connections 2023-10-16T12:23:56.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:56.100+0000 7fbe9d46b700 1 --2- 172.21.15.16:0/3452792664 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbe980cbed0 0x7fbe9806eaa0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:56.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:56.100+0000 7fbe9d46b700 1 -- 172.21.15.16:0/3452792664 >> 172.21.15.16:0/3452792664 conn(0x7fbe980cc5b0 msgr2=0x7fbe980cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:56.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:56.100+0000 7fbe9d46b700 1 -- 172.21.15.16:0/3452792664 shutdown_connections 2023-10-16T12:23:56.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:56.100+0000 7fbe9d46b700 1 -- 172.21.15.16:0/3452792664 wait complete. 2023-10-16T12:23:56.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:56.101+0000 7fbe9d46b700 1 Processor -- start 2023-10-16T12:23:56.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:56.102+0000 7fbe9d46b700 1 -- start start 2023-10-16T12:23:56.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:56.102+0000 7fbe9d46b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbe9806d350 0x7fbe9815bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:56.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:56.102+0000 7fbe9d46b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe980cbed0 0x7fbe98154810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:56.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:56.102+0000 7fbe9d46b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbe9815a870 0x7fbe9814bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:56.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:56.102+0000 7fbe9d46b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9815c0a0 con 0x7fbe980cbed0 2023-10-16T12:23:56.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:56.102+0000 7fbe9d46b700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe98154d50 con 0x7fbe9806d350 2023-10-16T12:23:56.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:56.103+0000 7fbe9d46b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe98153770 con 0x7fbe9815a870 2023-10-16T12:23:56.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:56.103+0000 7fbe967fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe980cbed0 0x7fbe98154810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:56.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:56.103+0000 7fbe967fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe980cbed0 0x7fbe98154810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37524/0 (socket says 172.21.15.16:37524) 2023-10-16T12:23:56.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:56.103+0000 7fbe967fc700 1 -- 172.21.15.16:0/1666216658 learned_addr learned my addr 172.21.15.16:0/1666216658 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:56.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:56.103+0000 7fbe96ffd700 1 --2- 172.21.15.16:0/1666216658 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbe9806d350 0x7fbe9815bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:56.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:56.103+0000 7fbe977fe700 1 --2- 172.21.15.16:0/1666216658 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbe9815a870 0x7fbe9814bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:56.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:56.103+0000 7fbe967fc700 1 -- 172.21.15.16:0/1666216658 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbe9815a870 msgr2=0x7fbe9814bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:56.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:56.103+0000 7fbe967fc700 1 --2- 172.21.15.16:0/1666216658 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbe9815a870 0x7fbe9814bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:56.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:56.103+0000 7fbe967fc700 1 -- 172.21.15.16:0/1666216658 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbe9806d350 msgr2=0x7fbe9815bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:56.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:56.103+0000 7fbe967fc700 1 --2- 172.21.15.16:0/1666216658 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbe9806d350 0x7fbe9815bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:56.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:56.103+0000 7fbe967fc700 1 -- 172.21.15.16:0/1666216658 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe90009a70 con 0x7fbe980cbed0 2023-10-16T12:23:56.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:56.103+0000 7fbe977fe700 1 --2- 172.21.15.16:0/1666216658 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbe9815a870 0x7fbe9814bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:23:56.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:56.103+0000 7fbe967fc700 1 --2- 172.21.15.16:0/1666216658 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe980cbed0 0x7fbe98154810 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fbe8c00c520 tx=0x7fbe8c00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:56.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:56.103+0000 7fbe7f7fe700 1 -- 172.21.15.16:0/1666216658 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe8c0052d0 con 0x7fbe980cbed0 2023-10-16T12:23:56.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:56.103+0000 7fbe7f7fe700 1 -- 172.21.15.16:0/1666216658 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe8c005430 con 0x7fbe980cbed0 2023-10-16T12:23:56.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:56.103+0000 7fbe9d46b700 1 -- 172.21.15.16:0/1666216658 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe98153a50 con 0x7fbe980cbed0 2023-10-16T12:23:56.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:56.103+0000 7fbe7f7fe700 1 -- 172.21.15.16:0/1666216658 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe8c0038c0 con 0x7fbe980cbed0 2023-10-16T12:23:56.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:56.103+0000 7fbe9d46b700 1 -- 172.21.15.16:0/1666216658 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe98154010 con 0x7fbe980cbed0 2023-10-16T12:23:56.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:56.104+0000 7fbe9d46b700 1 -- 172.21.15.16:0/1666216658 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe9804ebb0 con 0x7fbe980cbed0 2023-10-16T12:23:56.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:56.105+0000 7fbe7f7fe700 1 -- 172.21.15.16:0/1666216658 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fbe8c003a20 con 0x7fbe980cbed0 2023-10-16T12:23:56.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:56.105+0000 7fbe7f7fe700 1 --2- 172.21.15.16:0/1666216658 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fbe8406b050 0x7fbe8406d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:56.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:56.105+0000 7fbe7f7fe700 1 -- 172.21.15.16:0/1666216658 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5908+0+0 (secure 0 0 0) 0x7fbe8c088d50 con 0x7fbe980cbed0 2023-10-16T12:23:56.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:56.105+0000 7fbe96ffd700 1 --2- 172.21.15.16:0/1666216658 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fbe8406b050 0x7fbe8406d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:56.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:56.106+0000 7fbe96ffd700 1 --2- 172.21.15.16:0/1666216658 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fbe8406b050 0x7fbe8406d510 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fbe880011e0 tx=0x7fbe88009040 comp rx=0 tx=0).ready entity=mgr.5874 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:56.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:56.110+0000 7fbe7f7fe700 1 -- 172.21.15.16:0/1666216658 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbe8c08c030 con 0x7fbe980cbed0 2023-10-16T12:23:56.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:56.399+0000 7fbe9d46b700 1 -- 172.21.15.16:0/1666216658 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]} v 0) v1 -- 0x7fbe980cf320 con 0x7fbe980cbed0 2023-10-16T12:23:57.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.241+0000 7fbe7f7fe700 1 -- 172.21.15.16:0/1666216658 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]}]=0 v480) v1 ==== 157+0+0 (secure 0 0 0) 0x7fbe8c053c20 con 0x7fbe980cbed0 2023-10-16T12:23:57.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.244+0000 7fbe9d46b700 1 -- 172.21.15.16:0/1666216658 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fbe8406b050 msgr2=0x7fbe8406d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:57.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.244+0000 7fbe9d46b700 1 --2- 172.21.15.16:0/1666216658 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fbe8406b050 0x7fbe8406d510 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fbe880011e0 tx=0x7fbe88009040 comp rx=0 tx=0).stop 2023-10-16T12:23:57.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.244+0000 7fbe9d46b700 1 -- 172.21.15.16:0/1666216658 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe980cbed0 msgr2=0x7fbe98154810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:57.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.244+0000 7fbe9d46b700 1 --2- 172.21.15.16:0/1666216658 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe980cbed0 0x7fbe98154810 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fbe8c00c520 tx=0x7fbe8c00c8e0 comp rx=0 tx=0).stop 2023-10-16T12:23:57.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.244+0000 7fbe9d46b700 1 -- 172.21.15.16:0/1666216658 shutdown_connections 2023-10-16T12:23:57.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.244+0000 7fbe9d46b700 1 --2- 172.21.15.16:0/1666216658 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fbe8406b050 0x7fbe8406d510 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:57.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.244+0000 7fbe9d46b700 1 --2- 172.21.15.16:0/1666216658 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbe9815a870 0x7fbe9814bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:57.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.244+0000 7fbe9d46b700 1 --2- 172.21.15.16:0/1666216658 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbe9806d350 0x7fbe9815bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:57.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.244+0000 7fbe9d46b700 1 --2- 172.21.15.16:0/1666216658 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe980cbed0 0x7fbe98154810 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:57.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.244+0000 7fbe9d46b700 1 -- 172.21.15.16:0/1666216658 >> 172.21.15.16:0/1666216658 conn(0x7fbe980cc5b0 msgr2=0x7fbe980ceb50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:57.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.244+0000 7fbe9d46b700 1 -- 172.21.15.16:0/1666216658 shutdown_connections 2023-10-16T12:23:57.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.244+0000 7fbe9d46b700 1 -- 172.21.15.16:0/1666216658 wait complete. 2023-10-16T12:23:57.261 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-16T12:23:57.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.395+0000 7fc0a1574700 1 Processor -- start 2023-10-16T12:23:57.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.395+0000 7fc0a1574700 1 -- start start 2023-10-16T12:23:57.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.396+0000 7fc0a1574700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc09c15d8d0 0x7fc09c0b2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:57.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.396+0000 7fc0a1574700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc09c0b8fa0 0x7fc09c0b6720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:57.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.396+0000 7fc0a1574700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc09c0b2f30 con 0x7fc09c0b6c60 2023-10-16T12:23:57.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.396+0000 7fc0a1574700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc09c0d0bb0 con 0x7fc09c0b8fa0 2023-10-16T12:23:57.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.396+0000 7fc0a1574700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc09c15dcb0 con 0x7fc09c15d8d0 2023-10-16T12:23:57.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.396+0000 7fc09a7fc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc09c0b8fa0 0x7fc09c0b6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:57.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.396+0000 7fc09a7fc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc09c0b8fa0 0x7fc09c0b6720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37532/0 (socket says 172.21.15.16:37532) 2023-10-16T12:23:57.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.396+0000 7fc09a7fc700 1 -- 172.21.15.16:0/2080576412 learned_addr learned my addr 172.21.15.16:0/2080576412 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:57.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.396+0000 7fc09a7fc700 1 -- 172.21.15.16:0/2080576412 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc09c15d8d0 msgr2=0x7fc09c0b2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:57.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.396+0000 7fc09a7fc700 1 --2- 172.21.15.16:0/2080576412 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc09c15d8d0 0x7fc09c0b2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:57.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.396+0000 7fc09a7fc700 1 -- 172.21.15.16:0/2080576412 >> v1:172.21.15.16:6789/0 conn(0x7fc09c0b6c60 legacy=0x7fc09c0ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:57.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.396+0000 7fc09a7fc700 1 -- 172.21.15.16:0/2080576412 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc09c0d1c40 con 0x7fc09c0b8fa0 2023-10-16T12:23:57.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.396+0000 7fc09a7fc700 1 --2- 172.21.15.16:0/2080576412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc09c0b8fa0 0x7fc09c0b6720 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fc0900098f0 tx=0x7fc090009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=438e9854ff2cd377 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:57.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.396+0000 7fc099ffb700 1 -- 172.21.15.16:0/2080576412 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc09000b8a0 con 0x7fc09c0b8fa0 2023-10-16T12:23:57.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.396+0000 7fc099ffb700 1 -- 172.21.15.16:0/2080576412 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc09000ba00 con 0x7fc09c0b8fa0 2023-10-16T12:23:57.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.396+0000 7fc099ffb700 1 -- 172.21.15.16:0/2080576412 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc09000bd90 con 0x7fc09c0b8fa0 2023-10-16T12:23:57.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.397+0000 7fc0a1574700 1 -- 172.21.15.16:0/2080576412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc09c0b8fa0 msgr2=0x7fc09c0b6720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:57.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.397+0000 7fc0a1574700 1 --2- 172.21.15.16:0/2080576412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc09c0b8fa0 0x7fc09c0b6720 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fc0900098f0 tx=0x7fc090009c00 comp rx=0 tx=0).stop 2023-10-16T12:23:57.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.397+0000 7fc0a1574700 1 -- 172.21.15.16:0/2080576412 shutdown_connections 2023-10-16T12:23:57.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.397+0000 7fc0a1574700 1 --2- 172.21.15.16:0/2080576412 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc09c15d8d0 0x7fc09c0b2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:57.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.397+0000 7fc0a1574700 1 --2- 172.21.15.16:0/2080576412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc09c0b8fa0 0x7fc09c0b6720 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:57.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.397+0000 7fc0a1574700 1 -- 172.21.15.16:0/2080576412 >> 172.21.15.16:0/2080576412 conn(0x7fc09c072330 msgr2=0x7fc09c072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:57.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.397+0000 7fc0a1574700 1 -- 172.21.15.16:0/2080576412 shutdown_connections 2023-10-16T12:23:57.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.397+0000 7fc0a1574700 1 -- 172.21.15.16:0/2080576412 wait complete. 2023-10-16T12:23:57.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.398+0000 7fc0a1574700 1 Processor -- start 2023-10-16T12:23:57.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.398+0000 7fc0a1574700 1 -- start start 2023-10-16T12:23:57.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.399+0000 7fc0a1574700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc09c0b6c60 0x7fc09c0cacf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:57.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.399+0000 7fc0a1574700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc09c0b8fa0 0x7fc09c158d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:57.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.399+0000 7fc0a1574700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc09c15d8d0 0x7fc09c0ccd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:57.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.399+0000 7fc0a1574700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc09c0cb230 con 0x7fc09c0b8fa0 2023-10-16T12:23:57.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.399+0000 7fc0a1574700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc09c0cb460 con 0x7fc09c0b6c60 2023-10-16T12:23:57.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.399+0000 7fc0a1574700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc09c0cb740 con 0x7fc09c15d8d0 2023-10-16T12:23:57.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.399+0000 7fc09a7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc09c0b8fa0 0x7fc09c158d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:57.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.399+0000 7fc09a7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc09c0b8fa0 0x7fc09c158d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37538/0 (socket says 172.21.15.16:37538) 2023-10-16T12:23:57.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.399+0000 7fc09a7fc700 1 -- 172.21.15.16:0/3402887511 learned_addr learned my addr 172.21.15.16:0/3402887511 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:57.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.399+0000 7fc09a7fc700 1 -- 172.21.15.16:0/3402887511 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc09c15d8d0 msgr2=0x7fc09c0ccd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:57.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.399+0000 7fc09a7fc700 1 --2- 172.21.15.16:0/3402887511 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc09c15d8d0 0x7fc09c0ccd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:57.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.399+0000 7fc09a7fc700 1 -- 172.21.15.16:0/3402887511 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc09c0b6c60 msgr2=0x7fc09c0cacf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:57.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.399+0000 7fc09affd700 1 --2- 172.21.15.16:0/3402887511 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc09c0b6c60 0x7fc09c0cacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:57.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.399+0000 7fc09a7fc700 1 --2- 172.21.15.16:0/3402887511 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc09c0b6c60 0x7fc09c0cacf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:57.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.400+0000 7fc09a7fc700 1 -- 172.21.15.16:0/3402887511 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc090009550 con 0x7fc09c0b8fa0 2023-10-16T12:23:57.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.400+0000 7fc09a7fc700 1 --2- 172.21.15.16:0/3402887511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc09c0b8fa0 0x7fc09c158d20 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fc090014040 tx=0x7fc090004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:57.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.400+0000 7fc0837fe700 1 -- 172.21.15.16:0/3402887511 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0900038a0 con 0x7fc09c0b8fa0 2023-10-16T12:23:57.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.400+0000 7fc0837fe700 1 -- 172.21.15.16:0/3402887511 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc09001e070 con 0x7fc09c0b8fa0 2023-10-16T12:23:57.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.400+0000 7fc0a1574700 1 -- 172.21.15.16:0/3402887511 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc09c0c82b0 con 0x7fc09c0b8fa0 2023-10-16T12:23:57.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.400+0000 7fc0837fe700 1 -- 172.21.15.16:0/3402887511 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc090003c60 con 0x7fc09c0b8fa0 2023-10-16T12:23:57.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.400+0000 7fc0a1574700 1 -- 172.21.15.16:0/3402887511 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc09c159260 con 0x7fc09c0b8fa0 2023-10-16T12:23:57.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.401+0000 7fc0a1574700 1 -- 172.21.15.16:0/3402887511 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc09c0b8b30 con 0x7fc09c0b8fa0 2023-10-16T12:23:57.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.401+0000 7fc0837fe700 1 -- 172.21.15.16:0/3402887511 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc090011420 con 0x7fc09c0b8fa0 2023-10-16T12:23:57.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.402+0000 7fc0837fe700 1 --2- 172.21.15.16:0/3402887511 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fc08806b020 0x7fc08806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:57.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.402+0000 7fc0837fe700 1 -- 172.21.15.16:0/3402887511 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5908+0+0 (secure 0 0 0) 0x7fc090020070 con 0x7fc09c0b8fa0 2023-10-16T12:23:57.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.402+0000 7fc09affd700 1 --2- 172.21.15.16:0/3402887511 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fc08806b020 0x7fc08806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:57.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.402+0000 7fc09affd700 1 --2- 172.21.15.16:0/3402887511 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fc08806b020 0x7fc08806d4e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc09c0c9f10 tx=0x7fc08c008040 comp rx=0 tx=0).ready entity=mgr.5874 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:57.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.406+0000 7fc0837fe700 1 -- 172.21.15.16:0/3402887511 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc090055cd0 con 0x7fc09c0b8fa0 2023-10-16T12:23:57.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.612+0000 7fc0a1574700 1 -- 172.21.15.16:0/3402887511 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fc09c0cb900 con 0x7fc09c0b8fa0 2023-10-16T12:23:57.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.615+0000 7fc0837fe700 1 -- 172.21.15.16:0/3402887511 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v16)=0 v16) v1 ==== 122+0+0 (secure 0 0 0) 0x7fc0900592f0 con 0x7fc09c0b8fa0 2023-10-16T12:23:57.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.617+0000 7fc0a1574700 1 -- 172.21.15.16:0/3402887511 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fc08806b020 msgr2=0x7fc08806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:57.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.617+0000 7fc0a1574700 1 --2- 172.21.15.16:0/3402887511 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fc08806b020 0x7fc08806d4e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc09c0c9f10 tx=0x7fc08c008040 comp rx=0 tx=0).stop 2023-10-16T12:23:57.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.618+0000 7fc0a1574700 1 -- 172.21.15.16:0/3402887511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc09c0b8fa0 msgr2=0x7fc09c158d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:57.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.618+0000 7fc0a1574700 1 --2- 172.21.15.16:0/3402887511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc09c0b8fa0 0x7fc09c158d20 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fc090014040 tx=0x7fc090004750 comp rx=0 tx=0).stop 2023-10-16T12:23:57.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.618+0000 7fc0a1574700 1 -- 172.21.15.16:0/3402887511 shutdown_connections 2023-10-16T12:23:57.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.618+0000 7fc0a1574700 1 --2- 172.21.15.16:0/3402887511 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fc08806b020 0x7fc08806d4e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:57.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.618+0000 7fc0a1574700 1 --2- 172.21.15.16:0/3402887511 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc09c15d8d0 0x7fc09c0ccd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:57.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.618+0000 7fc0a1574700 1 --2- 172.21.15.16:0/3402887511 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc09c0b6c60 0x7fc09c0cacf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:57.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.618+0000 7fc0a1574700 1 --2- 172.21.15.16:0/3402887511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc09c0b8fa0 0x7fc09c158d20 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:57.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.618+0000 7fc0a1574700 1 -- 172.21.15.16:0/3402887511 >> 172.21.15.16:0/3402887511 conn(0x7fc09c072330 msgr2=0x7fc09c15de20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:57.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.618+0000 7fc0a1574700 1 -- 172.21.15.16:0/3402887511 shutdown_connections 2023-10-16T12:23:57.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.618+0000 7fc0a1574700 1 -- 172.21.15.16:0/3402887511 wait complete. 2023-10-16T12:23:57.632 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T12:23:57.632 DEBUG:teuthology.orchestra.run.smithi016:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T12:23:57.683 DEBUG:teuthology.orchestra.run.smithi049:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T12:23:57.735 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T12:23:57.735 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:23:57.735 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:23:57.753 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16287364k,nr_inodes=4071841,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20251 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3261172k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T12:23:57.753 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T12:23:57.754 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:23:57.774 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284204k,nr_inodes=4071051,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=753 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260540k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T12:23:57.774 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:23:57.775 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:23:57.811 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16287364k,nr_inodes=4071841,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20251 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3261172k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T12:23:57.812 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T12:23:57.812 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:23:57.833 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284204k,nr_inodes=4071051,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=753 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260540k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T12:23:57.834 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-16T12:23:57.834 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:23:57.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.974+0000 7f151fcd5700 1 Processor -- start 2023-10-16T12:23:57.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.975+0000 7f151fcd5700 1 -- start start 2023-10-16T12:23:57.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.975+0000 7f151fcd5700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f15180ce4f0 0x7f15180d0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:57.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.975+0000 7f151fcd5700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f151815ba10 0x7f15180cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:57.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.975+0000 7f151fcd5700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15180b0970 con 0x7f151815b630 2023-10-16T12:23:57.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.975+0000 7f151fcd5700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15180cd690 con 0x7f151815ba10 2023-10-16T12:23:57.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.975+0000 7f151fcd5700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15180cd970 con 0x7f15180ce4f0 2023-10-16T12:23:57.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.976+0000 7f151d270700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f15180ce4f0 0x7f15180d0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:57.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.976+0000 7f151e272700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f151815ba10 0x7f15180cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:57.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.976+0000 7f151da71700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7f151815b630 0x7f15180cdde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:34000/0 (socket says 172.21.15.16:34000) 2023-10-16T12:23:57.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.976+0000 7f151da71700 1 -- 172.21.15.16:0/3689717301 learned_addr learned my addr 172.21.15.16:0/3689717301 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:23:57.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.976+0000 7f151e272700 1 -- 172.21.15.16:0/3689717301 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f15180ce4f0 msgr2=0x7f15180d0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:57.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.976+0000 7f151e272700 1 --2- 172.21.15.16:0/3689717301 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f15180ce4f0 0x7f15180d0380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:57.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.976+0000 7f151e272700 1 -- 172.21.15.16:0/3689717301 >> v1:172.21.15.49:6789/0 conn(0x7f151815b630 legacy=0x7f15180cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:57.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.976+0000 7f151e272700 1 -- 172.21.15.16:0/3689717301 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15180b0c30 con 0x7f151815ba10 2023-10-16T12:23:57.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.976+0000 7f151ca6f700 1 -- 172.21.15.16:0/3689717301 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1679706830 0 0) 0x7f15180b0970 con 0x7f151815b630 2023-10-16T12:23:57.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.976+0000 7f151d270700 1 --2- 172.21.15.16:0/3689717301 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f15180ce4f0 0x7f15180d0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:23:57.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.976+0000 7f151e272700 1 --2- 172.21.15.16:0/3689717301 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f151815ba10 0x7f15180cf290 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f1510009d90 tx=0x7f151000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a176e33182a2b39f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:57.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.977+0000 7f151ca6f700 1 -- 172.21.15.16:0/3689717301 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f151000e070 con 0x7f151815ba10 2023-10-16T12:23:57.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.977+0000 7f151ca6f700 1 -- 172.21.15.16:0/3689717301 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f151000bb20 con 0x7f151815ba10 2023-10-16T12:23:57.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.977+0000 7f151ca6f700 1 -- 172.21.15.16:0/3689717301 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1510012430 con 0x7f151815ba10 2023-10-16T12:23:57.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.977+0000 7f151fcd5700 1 -- 172.21.15.16:0/3689717301 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f151815ba10 msgr2=0x7f15180cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:57.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.977+0000 7f151fcd5700 1 --2- 172.21.15.16:0/3689717301 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f151815ba10 0x7f15180cf290 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f1510009d90 tx=0x7f151000b330 comp rx=0 tx=0).stop 2023-10-16T12:23:57.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.977+0000 7f151fcd5700 1 -- 172.21.15.16:0/3689717301 shutdown_connections 2023-10-16T12:23:57.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.977+0000 7f151fcd5700 1 --2- 172.21.15.16:0/3689717301 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f151815ba10 0x7f15180cf290 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:57.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.977+0000 7f151fcd5700 1 --2- 172.21.15.16:0/3689717301 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f15180ce4f0 0x7f15180d0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:57.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.977+0000 7f151fcd5700 1 -- 172.21.15.16:0/3689717301 >> 172.21.15.16:0/3689717301 conn(0x7f15180d14b0 msgr2=0x7f15180d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:57.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.977+0000 7f151fcd5700 1 -- 172.21.15.16:0/3689717301 shutdown_connections 2023-10-16T12:23:57.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.977+0000 7f151fcd5700 1 -- 172.21.15.16:0/3689717301 wait complete. 2023-10-16T12:23:57.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.978+0000 7f151fcd5700 1 Processor -- start 2023-10-16T12:23:57.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.978+0000 7f151fcd5700 1 -- start start 2023-10-16T12:23:57.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.979+0000 7f151fcd5700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f15180ce4f0 0x7f151806c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:57.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.979+0000 7f151fcd5700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f151815b630 0x7f1518157330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:57.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.979+0000 7f151fcd5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f151815ba10 0x7f15180c5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:57.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.979+0000 7f151fcd5700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f151806d160 con 0x7f151815ba10 2023-10-16T12:23:57.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.979+0000 7f151fcd5700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f151806d900 con 0x7f15180ce4f0 2023-10-16T12:23:57.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.979+0000 7f151fcd5700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f151806d340 con 0x7f151815b630 2023-10-16T12:23:57.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.979+0000 7f151e272700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f151815ba10 0x7f15180c5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:57.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.979+0000 7f151e272700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f151815ba10 0x7f15180c5030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37554/0 (socket says 172.21.15.16:37554) 2023-10-16T12:23:57.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.979+0000 7f151e272700 1 -- 172.21.15.16:0/3166765671 learned_addr learned my addr 172.21.15.16:0/3166765671 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:57.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.979+0000 7f151da71700 1 --2- 172.21.15.16:0/3166765671 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f15180ce4f0 0x7f151806c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:57.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.980+0000 7f151d270700 1 --2- 172.21.15.16:0/3166765671 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f151815b630 0x7f1518157330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:57.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.980+0000 7f151e272700 1 -- 172.21.15.16:0/3166765671 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f151815b630 msgr2=0x7f1518157330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:57.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.980+0000 7f151e272700 1 --2- 172.21.15.16:0/3166765671 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f151815b630 0x7f1518157330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:57.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.980+0000 7f151e272700 1 -- 172.21.15.16:0/3166765671 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f15180ce4f0 msgr2=0x7f151806c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:57.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.980+0000 7f151e272700 1 --2- 172.21.15.16:0/3166765671 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f15180ce4f0 0x7f151806c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:57.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.980+0000 7f151e272700 1 -- 172.21.15.16:0/3166765671 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f150c0094a0 con 0x7f151815ba10 2023-10-16T12:23:57.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.980+0000 7f151da71700 1 --2- 172.21.15.16:0/3166765671 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f15180ce4f0 0x7f151806c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:23:57.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.980+0000 7f151e272700 1 --2- 172.21.15.16:0/3166765671 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f151815ba10 0x7f15180c5030 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f1518072680 tx=0x7f1510004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:57.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.980+0000 7f150a7fc700 1 -- 172.21.15.16:0/3166765671 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1510012430 con 0x7f151815ba10 2023-10-16T12:23:57.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.980+0000 7f150a7fc700 1 -- 172.21.15.16:0/3166765671 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1510020070 con 0x7f151815ba10 2023-10-16T12:23:57.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.980+0000 7f151fcd5700 1 -- 172.21.15.16:0/3166765671 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1510009a70 con 0x7f151815ba10 2023-10-16T12:23:57.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.980+0000 7f150a7fc700 1 -- 172.21.15.16:0/3166765671 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1510004280 con 0x7f151815ba10 2023-10-16T12:23:57.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.980+0000 7f151fcd5700 1 -- 172.21.15.16:0/3166765671 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1518159590 con 0x7f151815ba10 2023-10-16T12:23:57.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.981+0000 7f151fcd5700 1 -- 172.21.15.16:0/3166765671 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f151804ebb0 con 0x7f151815ba10 2023-10-16T12:23:57.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.981+0000 7f150a7fc700 1 -- 172.21.15.16:0/3166765671 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f15100043e0 con 0x7f151815ba10 2023-10-16T12:23:57.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.982+0000 7f150a7fc700 1 --2- 172.21.15.16:0/3166765671 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f14fc06b020 0x7f14fc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:57.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.982+0000 7f150a7fc700 1 -- 172.21.15.16:0/3166765671 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5908+0+0 (secure 0 0 0) 0x7f1510022070 con 0x7f151815ba10 2023-10-16T12:23:57.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.982+0000 7f151da71700 1 --2- 172.21.15.16:0/3166765671 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f14fc06b020 0x7f14fc06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:57.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.982+0000 7f151da71700 1 --2- 172.21.15.16:0/3166765671 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f14fc06b020 0x7f14fc06d4e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f1514000f60 tx=0x7f1514007ca0 comp rx=0 tx=0).ready entity=mgr.5874 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:57.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:57.986+0000 7f150a7fc700 1 -- 172.21.15.16:0/3166765671 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1510056480 con 0x7f151815ba10 2023-10-16T12:23:58.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:58.238+0000 7f151fcd5700 1 -- 172.21.15.16:0/3166765671 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f151806b5d0 con 0x7f151815ba10 2023-10-16T12:23:58.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:58.238+0000 7f150a7fc700 1 -- 172.21.15.16:0/3166765671 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+3696 (secure 0 0 0) 0x7f1510059aa0 con 0x7f151815ba10 2023-10-16T12:23:58.240 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:23:58.241 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":20,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5194,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/266676368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":266676368},{"type":"v1","addr":"172.21.15.49:6834","nonce":266676368}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5268,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/1460528697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":1460528697},{"type":"v1","addr":"172.21.15.16:6837","nonce":1460528697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/2374376752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2374376752},{"type":"v1","addr":"172.21.15.16:6836","nonce":2374376752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5857,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/2844420015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":2844420015},{"type":"v1","addr":"172.21.15.49:6835","nonce":2844420015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[]} 2023-10-16T12:23:58.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:58.240+0000 7f151fcd5700 1 -- 172.21.15.16:0/3166765671 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f14fc06b020 msgr2=0x7f14fc06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:58.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:58.240+0000 7f151fcd5700 1 --2- 172.21.15.16:0/3166765671 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f14fc06b020 0x7f14fc06d4e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f1514000f60 tx=0x7f1514007ca0 comp rx=0 tx=0).stop 2023-10-16T12:23:58.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:58.240+0000 7f151fcd5700 1 -- 172.21.15.16:0/3166765671 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f151815ba10 msgr2=0x7f15180c5030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:58.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:58.240+0000 7f151fcd5700 1 --2- 172.21.15.16:0/3166765671 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f151815ba10 0x7f15180c5030 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f1518072680 tx=0x7f1510004ad0 comp rx=0 tx=0).stop 2023-10-16T12:23:58.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:58.240+0000 7f151fcd5700 1 -- 172.21.15.16:0/3166765671 shutdown_connections 2023-10-16T12:23:58.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:58.240+0000 7f151fcd5700 1 --2- 172.21.15.16:0/3166765671 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f14fc06b020 0x7f14fc06d4e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:58.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:58.240+0000 7f151fcd5700 1 --2- 172.21.15.16:0/3166765671 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f151815ba10 0x7f15180c5030 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:58.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:58.240+0000 7f151fcd5700 1 --2- 172.21.15.16:0/3166765671 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f15180ce4f0 0x7f151806c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:58.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:58.240+0000 7f151fcd5700 1 --2- 172.21.15.16:0/3166765671 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f151815b630 0x7f1518157330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:58.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:58.240+0000 7f151fcd5700 1 -- 172.21.15.16:0/3166765671 >> 172.21.15.16:0/3166765671 conn(0x7f15180d14b0 msgr2=0x7f15180686c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:58.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:58.240+0000 7f151fcd5700 1 -- 172.21.15.16:0/3166765671 shutdown_connections 2023-10-16T12:23:58.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:58.240+0000 7f151fcd5700 1 -- 172.21.15.16:0/3166765671 wait complete. 2023-10-16T12:23:58.245 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 20 2023-10-16T12:23:58.255 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-16T12:23:58.255 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-16T12:23:58.256 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-16T12:23:58.256 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T12:23:58.256 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-16T12:23:58.256 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-16T12:23:58.257 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-16T12:23:58.257 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T12:23:58.257 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-16T12:23:58.258 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-16T12:23:58.258 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-16T12:23:58.258 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T12:23:58.259 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-16T12:23:58.259 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-16T12:23:58.259 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-16T12:23:58.259 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T12:23:58.267 INFO:tasks.ceph.mds.a:Stopped 2023-10-16T12:23:58.267 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-16T12:23:58.268 INFO:tasks.ceph.mds.c:Stopped 2023-10-16T12:23:58.268 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-16T12:23:58.269 INFO:tasks.ceph.mds.b:Stopped 2023-10-16T12:23:58.270 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-16T12:23:58.271 INFO:tasks.ceph.mds.d:Stopped 2023-10-16T12:23:58.271 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-16T12:23:58.309 INFO:tasks.ceph.mds.a:Started 2023-10-16T12:23:58.309 DEBUG:teuthology.parallel:result is None 2023-10-16T12:23:58.310 INFO:tasks.ceph.mds.c:Started 2023-10-16T12:23:58.310 DEBUG:teuthology.parallel:result is None 2023-10-16T12:23:58.313 INFO:tasks.ceph.mds.b:Started 2023-10-16T12:23:58.313 DEBUG:teuthology.parallel:result is None 2023-10-16T12:23:58.314 INFO:tasks.ceph.mds.d:Started 2023-10-16T12:23:58.314 DEBUG:teuthology.parallel:result is None 2023-10-16T12:23:58.315 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-16T12:23:58.393 INFO:tasks.ceph.mds.a.smithi016.stdout:starting mds.a at 2023-10-16T12:23:58.393 INFO:tasks.ceph.mds.c.smithi016.stdout:starting mds.c at 2023-10-16T12:23:58.404 INFO:tasks.ceph.mds.b.smithi049.stdout:starting mds.b at 2023-10-16T12:23:58.405 INFO:tasks.ceph.mds.d.smithi049.stdout:starting mds.d at 2023-10-16T12:23:58.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:58.446+0000 7f1ee558e700 1 Processor -- start 2023-10-16T12:23:58.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:58.446+0000 7f1ee558e700 1 -- start start 2023-10-16T12:23:58.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:58.447+0000 7f1ee558e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1ee015b630 0x7f1ee00d0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:58.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:58.447+0000 7f1ee558e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1ee00ce4f0 0x7f1ee00cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:58.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:58.447+0000 7f1ee558e700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ee00b0970 con 0x7f1ee015ba10 2023-10-16T12:23:58.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:58.447+0000 7f1ee558e700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ee00cd690 con 0x7f1ee015b630 2023-10-16T12:23:58.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:58.447+0000 7f1ee558e700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee00cd970 con 0x7f1ee00ce4f0 2023-10-16T12:23:58.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:58.447+0000 7f1edeffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1ee015b630 0x7f1ee00d0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:58.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:58.447+0000 7f1edeffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1ee015b630 0x7f1ee00d0380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37582/0 (socket says 172.21.15.16:37582) 2023-10-16T12:23:58.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:58.447+0000 7f1edeffd700 1 -- 172.21.15.16:0/2622210342 learned_addr learned my addr 172.21.15.16:0/2622210342 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:58.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:58.447+0000 7f1edf7fe700 1 --2- 172.21.15.16:0/2622210342 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1ee00ce4f0 0x7f1ee00cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:58.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:58.447+0000 7f1edeffd700 1 -- 172.21.15.16:0/2622210342 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1ee00ce4f0 msgr2=0x7f1ee00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:58.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:58.447+0000 7f1edeffd700 1 --2- 172.21.15.16:0/2622210342 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1ee00ce4f0 0x7f1ee00cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:58.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:58.447+0000 7f1edeffd700 1 -- 172.21.15.16:0/2622210342 >> v1:172.21.15.16:6789/0 conn(0x7f1ee015ba10 legacy=0x7f1ee00cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:58.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:58.447+0000 7f1eddffb700 1 -- 172.21.15.16:0/2622210342 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3405169125 0 0) 0x7f1ee00b0970 con 0x7f1ee015ba10 2023-10-16T12:23:58.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:58.447+0000 7f1edeffd700 1 -- 172.21.15.16:0/2622210342 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ee00b0c30 con 0x7f1ee015b630 2023-10-16T12:23:58.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:58.448+0000 7f1edeffd700 1 --2- 172.21.15.16:0/2622210342 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ee015b630 0x7f1ee00d0380 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f1ed00098f0 tx=0x7f1ed0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da76d0ee7345e105 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:58.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:58.448+0000 7f1eddffb700 1 -- 172.21.15.16:0/2622210342 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ed0011830 con 0x7f1ee015b630 2023-10-16T12:23:58.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:58.448+0000 7f1eddffb700 1 -- 172.21.15.16:0/2622210342 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ed0011990 con 0x7f1ee015b630 2023-10-16T12:23:58.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:58.448+0000 7f1eddffb700 1 -- 172.21.15.16:0/2622210342 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ed0011d20 con 0x7f1ee015b630 2023-10-16T12:23:58.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:58.448+0000 7f1ee558e700 1 -- 172.21.15.16:0/2622210342 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ee015b630 msgr2=0x7f1ee00d0380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:58.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:58.448+0000 7f1ee558e700 1 --2- 172.21.15.16:0/2622210342 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ee015b630 0x7f1ee00d0380 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f1ed00098f0 tx=0x7f1ed0009c00 comp rx=0 tx=0).stop 2023-10-16T12:23:58.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:58.448+0000 7f1ee558e700 1 -- 172.21.15.16:0/2622210342 shutdown_connections 2023-10-16T12:23:58.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:58.448+0000 7f1ee558e700 1 --2- 172.21.15.16:0/2622210342 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1ee00ce4f0 0x7f1ee00cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:58.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:58.448+0000 7f1ee558e700 1 --2- 172.21.15.16:0/2622210342 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ee015b630 0x7f1ee00d0380 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:58.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:58.448+0000 7f1ee558e700 1 -- 172.21.15.16:0/2622210342 >> 172.21.15.16:0/2622210342 conn(0x7f1ee00d14b0 msgr2=0x7f1ee00d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:58.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:58.448+0000 7f1ee558e700 1 -- 172.21.15.16:0/2622210342 shutdown_connections 2023-10-16T12:23:58.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:58.448+0000 7f1ee558e700 1 -- 172.21.15.16:0/2622210342 wait complete. 2023-10-16T12:23:58.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:58.450+0000 7f1ee558e700 1 Processor -- start 2023-10-16T12:23:58.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:58.450+0000 7f1ee558e700 1 -- start start 2023-10-16T12:23:58.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:58.450+0000 7f1ee558e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1ee00ce4f0 0x7f1ee01f1740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:58.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:58.450+0000 7f1ee558e700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1ee015b630 0x7f1ee00b4570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:58.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:58.451+0000 7f1ee558e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ee015ba10 0x7f1ee01f0120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:58.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:58.451+0000 7f1ee558e700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee01f1c80 con 0x7f1ee015ba10 2023-10-16T12:23:58.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:58.451+0000 7f1ee558e700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee00b6960 con 0x7f1ee015b630 2023-10-16T12:23:58.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:58.451+0000 7f1ee558e700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee00b5a70 con 0x7f1ee00ce4f0 2023-10-16T12:23:58.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:58.451+0000 7f1edf7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ee015ba10 0x7f1ee01f0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:58.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:58.451+0000 7f1edf7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ee015ba10 0x7f1ee01f0120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37592/0 (socket says 172.21.15.16:37592) 2023-10-16T12:23:58.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:58.451+0000 7f1edf7fe700 1 -- 172.21.15.16:0/135811825 learned_addr learned my addr 172.21.15.16:0/135811825 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:58.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:58.451+0000 7f1edeffd700 1 --2- 172.21.15.16:0/135811825 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1ee00ce4f0 0x7f1ee01f1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:58.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:58.451+0000 7f1edf7fe700 1 -- 172.21.15.16:0/135811825 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1ee00ce4f0 msgr2=0x7f1ee01f1740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:58.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:58.451+0000 7f1ede7fc700 1 --2- 172.21.15.16:0/135811825 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1ee015b630 0x7f1ee00b4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:58.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:58.451+0000 7f1edf7fe700 1 --2- 172.21.15.16:0/135811825 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1ee00ce4f0 0x7f1ee01f1740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:58.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:58.451+0000 7f1edf7fe700 1 -- 172.21.15.16:0/135811825 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1ee015b630 msgr2=0x7f1ee00b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:58.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:58.451+0000 7f1edf7fe700 1 --2- 172.21.15.16:0/135811825 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1ee015b630 0x7f1ee00b4570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:58.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:58.451+0000 7f1edf7fe700 1 -- 172.21.15.16:0/135811825 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ed0009550 con 0x7f1ee015ba10 2023-10-16T12:23:58.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:58.451+0000 7f1ede7fc700 1 --2- 172.21.15.16:0/135811825 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1ee015b630 0x7f1ee00b4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:23:58.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:58.451+0000 7f1edf7fe700 1 --2- 172.21.15.16:0/135811825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ee015ba10 0x7f1ee01f0120 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f1ed800efc0 tx=0x7f1ed800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:58.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:58.451+0000 7f1ed77fe700 1 -- 172.21.15.16:0/135811825 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ed800cd40 con 0x7f1ee015ba10 2023-10-16T12:23:58.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:58.451+0000 7f1ed77fe700 1 -- 172.21.15.16:0/135811825 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ed800f040 con 0x7f1ee015ba10 2023-10-16T12:23:58.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:58.451+0000 7f1ee558e700 1 -- 172.21.15.16:0/135811825 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ee00b5d50 con 0x7f1ee015ba10 2023-10-16T12:23:58.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:58.451+0000 7f1ed77fe700 1 -- 172.21.15.16:0/135811825 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ed80146c0 con 0x7f1ee015ba10 2023-10-16T12:23:58.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:58.451+0000 7f1ee558e700 1 -- 172.21.15.16:0/135811825 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ee00b6310 con 0x7f1ee015ba10 2023-10-16T12:23:58.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:58.452+0000 7f1ee558e700 1 -- 172.21.15.16:0/135811825 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ee004ebb0 con 0x7f1ee015ba10 2023-10-16T12:23:58.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:58.453+0000 7f1ed77fe700 1 -- 172.21.15.16:0/135811825 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1ed8014820 con 0x7f1ee015ba10 2023-10-16T12:23:58.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:58.453+0000 7f1ed77fe700 1 --2- 172.21.15.16:0/135811825 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f1ec006b020 0x7f1ec006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:58.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:58.453+0000 7f1ed77fe700 1 -- 172.21.15.16:0/135811825 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5908+0+0 (secure 0 0 0) 0x7f1ed808a530 con 0x7f1ee015ba10 2023-10-16T12:23:58.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:58.453+0000 7f1edeffd700 1 --2- 172.21.15.16:0/135811825 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f1ec006b020 0x7f1ec006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:58.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:58.454+0000 7f1edeffd700 1 --2- 172.21.15.16:0/135811825 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f1ec006b020 0x7f1ec006d4e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1ed0009c40 tx=0x7f1ed0008030 comp rx=0 tx=0).ready entity=mgr.5874 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:58.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:58.458+0000 7f1ed77fe700 1 -- 172.21.15.16:0/135811825 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1ed8055380 con 0x7f1ee015ba10 2023-10-16T12:23:58.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:58.677+0000 7f1ee558e700 1 -- 172.21.15.16:0/135811825 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f1ee006b5d0 con 0x7f1ee015ba10 2023-10-16T12:23:59.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.253+0000 7f1ed77fe700 1 -- 172.21.15.16:0/135811825 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v62) v1 ==== 97+0+0 (secure 0 0 0) 0x7f1ed800cea0 con 0x7f1ee015ba10 2023-10-16T12:23:59.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.256+0000 7f1ee558e700 1 -- 172.21.15.16:0/135811825 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f1ec006b020 msgr2=0x7f1ec006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:59.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.256+0000 7f1ee558e700 1 --2- 172.21.15.16:0/135811825 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f1ec006b020 0x7f1ec006d4e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1ed0009c40 tx=0x7f1ed0008030 comp rx=0 tx=0).stop 2023-10-16T12:23:59.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.256+0000 7f1ee558e700 1 -- 172.21.15.16:0/135811825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ee015ba10 msgr2=0x7f1ee01f0120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:59.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.256+0000 7f1ee558e700 1 --2- 172.21.15.16:0/135811825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ee015ba10 0x7f1ee01f0120 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f1ed800efc0 tx=0x7f1ed800c3a0 comp rx=0 tx=0).stop 2023-10-16T12:23:59.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.256+0000 7f1ee558e700 1 -- 172.21.15.16:0/135811825 shutdown_connections 2023-10-16T12:23:59.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.256+0000 7f1ee558e700 1 --2- 172.21.15.16:0/135811825 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f1ec006b020 0x7f1ec006d4e0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:59.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.256+0000 7f1ee558e700 1 --2- 172.21.15.16:0/135811825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ee015ba10 0x7f1ee01f0120 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:59.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.256+0000 7f1ee558e700 1 --2- 172.21.15.16:0/135811825 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1ee00ce4f0 0x7f1ee01f1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:59.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.256+0000 7f1ee558e700 1 --2- 172.21.15.16:0/135811825 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1ee015b630 0x7f1ee00b4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:59.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.256+0000 7f1ee558e700 1 -- 172.21.15.16:0/135811825 >> 172.21.15.16:0/135811825 conn(0x7f1ee00d14b0 msgr2=0x7f1ee006eba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:59.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.256+0000 7f1ee558e700 1 -- 172.21.15.16:0/135811825 shutdown_connections 2023-10-16T12:23:59.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.256+0000 7f1ee558e700 1 -- 172.21.15.16:0/135811825 wait complete. 2023-10-16T12:23:59.261 INFO:teuthology.orchestra.run.smithi016.stderr: removed all blocklist entries 2023-10-16T12:23:59.275 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-16T12:23:59.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.405+0000 7fb48dc76700 1 Processor -- start 2023-10-16T12:23:59.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.405+0000 7fb48dc76700 1 -- start start 2023-10-16T12:23:59.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.406+0000 7fb48dc76700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb48815b280 0x7fb4881613e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:59.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.406+0000 7fb48dc76700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb4880cc8a0 0x7fb48815e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:59.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.406+0000 7fb48dc76700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fb48806a7e0 0x7fb48806ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:59.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.406+0000 7fb48dc76700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb48806b170 con 0x7fb48815b280 2023-10-16T12:23:59.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.406+0000 7fb48dc76700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb488161c30 con 0x7fb48806a7e0 2023-10-16T12:23:59.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.406+0000 7fb48dc76700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb488161f10 con 0x7fb4880cc8a0 2023-10-16T12:23:59.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.406+0000 7fb4877fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb48815b280 0x7fb4881613e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:59.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.406+0000 7fb4877fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb48815b280 0x7fb4881613e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37596/0 (socket says 172.21.15.16:37596) 2023-10-16T12:23:59.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.406+0000 7fb4877fe700 1 -- 172.21.15.16:0/2805632445 learned_addr learned my addr 172.21.15.16:0/2805632445 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:59.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.406+0000 7fb487fff700 1 --2- 172.21.15.16:0/2805632445 >> v2:172.21.15.49:3300/0 conn(0x7fb48806a7e0 0x7fb48806ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:59.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.406+0000 7fb486ffd700 1 --2- 172.21.15.16:0/2805632445 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb4880cc8a0 0x7fb48815e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:59.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.406+0000 7fb4877fe700 1 -- 172.21.15.16:0/2805632445 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb4880cc8a0 msgr2=0x7fb48815e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:59.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.406+0000 7fb4877fe700 1 --2- 172.21.15.16:0/2805632445 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb4880cc8a0 0x7fb48815e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:59.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.406+0000 7fb4877fe700 1 -- 172.21.15.16:0/2805632445 >> v2:172.21.15.49:3300/0 conn(0x7fb48806a7e0 msgr2=0x7fb48806ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:59.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.406+0000 7fb4877fe700 1 --2- 172.21.15.16:0/2805632445 >> v2:172.21.15.49:3300/0 conn(0x7fb48806a7e0 0x7fb48806ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:59.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.406+0000 7fb4877fe700 1 -- 172.21.15.16:0/2805632445 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4881621f0 con 0x7fb48815b280 2023-10-16T12:23:59.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.406+0000 7fb4877fe700 1 --2- 172.21.15.16:0/2805632445 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb48815b280 0x7fb4881613e0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fb488071970 tx=0x7fb47c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b14bb7f47f080d3a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:59.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.406+0000 7fb4867fc700 1 -- 172.21.15.16:0/2805632445 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb47c00b870 con 0x7fb48815b280 2023-10-16T12:23:59.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.406+0000 7fb4867fc700 1 -- 172.21.15.16:0/2805632445 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb47c00b9d0 con 0x7fb48815b280 2023-10-16T12:23:59.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.406+0000 7fb4867fc700 1 -- 172.21.15.16:0/2805632445 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb47c00bd60 con 0x7fb48815b280 2023-10-16T12:23:59.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.407+0000 7fb48dc76700 1 -- 172.21.15.16:0/2805632445 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb48815b280 msgr2=0x7fb4881613e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:59.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.407+0000 7fb48dc76700 1 --2- 172.21.15.16:0/2805632445 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb48815b280 0x7fb4881613e0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fb488071970 tx=0x7fb47c009bd0 comp rx=0 tx=0).stop 2023-10-16T12:23:59.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.407+0000 7fb48dc76700 1 -- 172.21.15.16:0/2805632445 shutdown_connections 2023-10-16T12:23:59.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.407+0000 7fb48dc76700 1 --2- 172.21.15.16:0/2805632445 >> v2:172.21.15.49:3300/0 conn(0x7fb48806a7e0 0x7fb48806ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:59.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.407+0000 7fb48dc76700 1 --2- 172.21.15.16:0/2805632445 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb48815b280 0x7fb4881613e0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:59.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.407+0000 7fb48dc76700 1 --2- 172.21.15.16:0/2805632445 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb4880cc8a0 0x7fb48815e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:59.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.407+0000 7fb48dc76700 1 -- 172.21.15.16:0/2805632445 >> 172.21.15.16:0/2805632445 conn(0x7fb4880b1fd0 msgr2=0x7fb4880b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:59.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.407+0000 7fb48dc76700 1 -- 172.21.15.16:0/2805632445 shutdown_connections 2023-10-16T12:23:59.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.407+0000 7fb48dc76700 1 -- 172.21.15.16:0/2805632445 wait complete. 2023-10-16T12:23:59.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.408+0000 7fb48dc76700 1 Processor -- start 2023-10-16T12:23:59.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.408+0000 7fb48dc76700 1 -- start start 2023-10-16T12:23:59.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.409+0000 7fb48dc76700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb48806a7e0 0x7fb48815c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:59.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.409+0000 7fb48dc76700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb4880cc8a0 0x7fb4881f3160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:59.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.409+0000 7fb48dc76700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb48815b280 0x7fb4880c5690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:59.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.409+0000 7fb48dc76700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb488160170 con 0x7fb48806a7e0 2023-10-16T12:23:59.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.409+0000 7fb48dc76700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb48815f4f0 con 0x7fb4880cc8a0 2023-10-16T12:23:59.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.409+0000 7fb48dc76700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb48815f7d0 con 0x7fb48815b280 2023-10-16T12:23:59.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.409+0000 7fb4877fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb48806a7e0 0x7fb48815c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:59.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.409+0000 7fb4877fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb48806a7e0 0x7fb48815c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37602/0 (socket says 172.21.15.16:37602) 2023-10-16T12:23:59.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.409+0000 7fb4877fe700 1 -- 172.21.15.16:0/402071532 learned_addr learned my addr 172.21.15.16:0/402071532 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:59.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.410+0000 7fb487fff700 1 --2- 172.21.15.16:0/402071532 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb48815b280 0x7fb4880c5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:59.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.410+0000 7fb486ffd700 1 --2- 172.21.15.16:0/402071532 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb4880cc8a0 0x7fb4881f3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:59.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.410+0000 7fb4877fe700 1 -- 172.21.15.16:0/402071532 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb48815b280 msgr2=0x7fb4880c5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:59.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.410+0000 7fb4877fe700 1 --2- 172.21.15.16:0/402071532 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb48815b280 0x7fb4880c5690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:59.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.410+0000 7fb4877fe700 1 -- 172.21.15.16:0/402071532 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb4880cc8a0 msgr2=0x7fb4881f3160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:59.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.410+0000 7fb4877fe700 1 --2- 172.21.15.16:0/402071532 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb4880cc8a0 0x7fb4881f3160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:59.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.410+0000 7fb4877fe700 1 -- 172.21.15.16:0/402071532 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb47c009550 con 0x7fb48806a7e0 2023-10-16T12:23:59.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.410+0000 7fb4877fe700 1 --2- 172.21.15.16:0/402071532 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb48806a7e0 0x7fb48815c6f0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fb47c000c00 tx=0x7fb47c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:59.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.410+0000 7fb46ffff700 1 -- 172.21.15.16:0/402071532 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb47c004520 con 0x7fb48806a7e0 2023-10-16T12:23:59.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.410+0000 7fb46ffff700 1 -- 172.21.15.16:0/402071532 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb47c01e070 con 0x7fb48806a7e0 2023-10-16T12:23:59.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.410+0000 7fb48dc76700 1 -- 172.21.15.16:0/402071532 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb48815fab0 con 0x7fb48806a7e0 2023-10-16T12:23:59.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.410+0000 7fb46ffff700 1 -- 172.21.15.16:0/402071532 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb47c0038c0 con 0x7fb48806a7e0 2023-10-16T12:23:59.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.410+0000 7fb48dc76700 1 -- 172.21.15.16:0/402071532 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb48815cc30 con 0x7fb48806a7e0 2023-10-16T12:23:59.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.411+0000 7fb46ffff700 1 -- 172.21.15.16:0/402071532 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb47c003a20 con 0x7fb48806a7e0 2023-10-16T12:23:59.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.412+0000 7fb48dc76700 1 -- 172.21.15.16:0/402071532 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb488160590 con 0x7fb48806a7e0 2023-10-16T12:23:59.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.412+0000 7fb46ffff700 1 --2- 172.21.15.16:0/402071532 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fb47406b020 0x7fb47406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:59.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.412+0000 7fb46ffff700 1 -- 172.21.15.16:0/402071532 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fb47c020070 con 0x7fb48806a7e0 2023-10-16T12:23:59.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.412+0000 7fb486ffd700 1 --2- 172.21.15.16:0/402071532 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fb47406b020 0x7fb47406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:59.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.412+0000 7fb486ffd700 1 --2- 172.21.15.16:0/402071532 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fb47406b020 0x7fb47406d4e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb48815c1c0 tx=0x7fb478008040 comp rx=0 tx=0).ready entity=mgr.5874 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:59.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.417+0000 7fb46ffff700 1 -- 172.21.15.16:0/402071532 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb488160590 con 0x7fb48806a7e0 2023-10-16T12:23:59.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.700+0000 7fb48dc76700 1 -- 172.21.15.16:0/402071532 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fb488160590 con 0x7fb48806a7e0 2023-10-16T12:23:59.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.700+0000 7fb46ffff700 1 -- 172.21.15.16:0/402071532 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v17) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fb488160590 con 0x7fb48806a7e0 2023-10-16T12:23:59.703 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:23:59.703 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-10-16T12:23:59.704 INFO:teuthology.orchestra.run.smithi016.stdout: "auth_dump": [ 2023-10-16T12:23:59.704 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:23:59.704 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.a", 2023-10-16T12:23:59.704 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOKC1l/rBtOhAA/ob5WUvpz8Glp4E+xPZOYA==", 2023-10-16T12:23:59.705 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:23:59.705 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T12:23:59.705 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-10-16T12:23:59.705 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-10-16T12:23:59.705 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:23:59.706 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:23:59.706 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:23:59.706 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:23:59.706 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.b", 2023-10-16T12:23:59.707 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1lKAVsCxAAvhXuLY6i1+Iz1yq5BVPMNQ==", 2023-10-16T12:23:59.707 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:23:59.707 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T12:23:59.708 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-10-16T12:23:59.708 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-10-16T12:23:59.708 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:23:59.709 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:23:59.709 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:23:59.709 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:23:59.709 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.c", 2023-10-16T12:23:59.709 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1lvBowBhAA3waJNYhdgs8PGCYqYNYUog==", 2023-10-16T12:23:59.710 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:23:59.710 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T12:23:59.710 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-10-16T12:23:59.710 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-10-16T12:23:59.710 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:23:59.711 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:23:59.711 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:23:59.711 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:23:59.711 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.d", 2023-10-16T12:23:59.712 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1lS/fnEhAAfNYgQ5BW86JMjYm/bBfnaQ==", 2023-10-16T12:23:59.712 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:23:59.712 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T12:23:59.712 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-10-16T12:23:59.712 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-10-16T12:23:59.713 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:23:59.713 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:23:59.713 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:23:59.713 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:23:59.713 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.0", 2023-10-16T12:23:59.713 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAQKC1lbxllOhAAZeLdp2GMl6HqQPaQjKlNqw==", 2023-10-16T12:23:59.714 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:23:59.714 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T12:23:59.714 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T12:23:59.714 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:23:59.714 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:23:59.715 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:23:59.715 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:23:59.715 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.1", 2023-10-16T12:23:59.715 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQATKC1lnU6KOhAAHxvlbmzBptf1keGgL+Rbkg==", 2023-10-16T12:23:59.715 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:23:59.716 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T12:23:59.716 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T12:23:59.716 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:23:59.716 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:23:59.716 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:23:59.717 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:23:59.717 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.2", 2023-10-16T12:23:59.717 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAXKC1l9jFyARAAr/NAwtWhBMut9sAAoDEGWg==", 2023-10-16T12:23:59.717 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:23:59.718 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T12:23:59.718 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T12:23:59.718 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:23:59.718 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:23:59.718 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:23:59.719 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:23:59.719 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.3", 2023-10-16T12:23:59.719 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAaKC1lbw/lARAAGJ/9DNb5/7kLFSq7vrLtXg==", 2023-10-16T12:23:59.719 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:23:59.719 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T12:23:59.720 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T12:23:59.720 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:23:59.720 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:23:59.720 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:23:59.720 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:23:59.721 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.4", 2023-10-16T12:23:59.721 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAeKC1lZb7jERAAYpvb1xcSli9CTq7L8ChLOA==", 2023-10-16T12:23:59.721 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:23:59.721 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T12:23:59.721 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T12:23:59.722 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:23:59.722 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:23:59.722 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:23:59.722 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:23:59.722 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.5", 2023-10-16T12:23:59.723 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAhKC1l/4HuEBAAqfnRqjknij9KES/fqiyIBQ==", 2023-10-16T12:23:59.723 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:23:59.723 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T12:23:59.723 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T12:23:59.723 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:23:59.724 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:23:59.724 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:23:59.724 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:23:59.724 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.6", 2023-10-16T12:23:59.724 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAkKC1lzoUkEBAA+Q82OiBMNag8+R316WQtUw==", 2023-10-16T12:23:59.725 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:23:59.725 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T12:23:59.725 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T12:23:59.725 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:23:59.725 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:23:59.726 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:23:59.726 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:23:59.726 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.7", 2023-10-16T12:23:59.726 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAnKC1ljUu4DxAAc+G4anJANC2GhLJ8wb6gYg==", 2023-10-16T12:23:59.726 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:23:59.727 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T12:23:59.727 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T12:23:59.727 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:23:59.727 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:23:59.727 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:23:59.728 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:23:59.728 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.0", 2023-10-16T12:23:59.728 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1l2l6PFxAA9QyXtBAHdGAXpwH0JfknuQ==", 2023-10-16T12:23:59.728 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:23:59.729 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T12:23:59.729 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow r", 2023-10-16T12:23:59.729 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-16T12:23:59.729 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:23:59.729 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:23:59.730 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:23:59.730 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.1", 2023-10-16T12:23:59.730 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1lAcHFIxAAcNt2bmv2R6j5L3DYkdw7eQ==", 2023-10-16T12:23:59.730 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:23:59.730 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T12:23:59.731 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-10-16T12:23:59.731 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-10-16T12:23:59.731 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-10-16T12:23:59.731 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:23:59.731 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:23:59.732 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:23:59.732 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.2", 2023-10-16T12:23:59.732 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1lA6wnGxAAdtaqYhtIJu8qENMrD8xvVA==", 2023-10-16T12:23:59.732 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:23:59.732 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T12:23:59.732 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-10-16T12:23:59.733 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-10-16T12:23:59.733 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-10-16T12:23:59.733 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:23:59.733 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:23:59.733 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:23:59.734 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.3", 2023-10-16T12:23:59.734 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1l/ddAJxAA3fNKKn9XkXas1BlDN+ggjw==", 2023-10-16T12:23:59.734 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:23:59.734 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T12:23:59.734 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-10-16T12:23:59.735 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-10-16T12:23:59.735 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-10-16T12:23:59.735 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:23:59.735 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:23:59.735 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:23:59.736 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.admin", 2023-10-16T12:23:59.736 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOKC1ld5jUHBAAc6esnCce3dSEZn3wVjhIlQ==", 2023-10-16T12:23:59.736 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:23:59.736 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2023-10-16T12:23:59.736 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-10-16T12:23:59.737 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-10-16T12:23:59.737 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:23:59.737 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:23:59.737 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:23:59.737 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:23:59.738 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mgr.x", 2023-10-16T12:23:59.738 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOKC1l6b3vNhAAmj5xbfid2sZM2KfHmAf1Wg==", 2023-10-16T12:23:59.738 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:23:59.738 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2023-10-16T12:23:59.738 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile mgr", 2023-10-16T12:23:59.739 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:23:59.739 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:23:59.739 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:23:59.739 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:23:59.740 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mgr.y", 2023-10-16T12:23:59.740 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOKC1lLz2nMhAA8NqFYZ5nDKkzPBMrkwsVuQ==", 2023-10-16T12:23:59.740 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:23:59.740 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2023-10-16T12:23:59.740 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile mgr", 2023-10-16T12:23:59.740 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:23:59.741 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:23:59.741 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:23:59.741 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-10-16T12:23:59.741 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-10-16T12:23:59.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.702+0000 7fb48dc76700 1 -- 172.21.15.16:0/402071532 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fb47406b020 msgr2=0x7fb47406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:59.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.702+0000 7fb48dc76700 1 --2- 172.21.15.16:0/402071532 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fb47406b020 0x7fb47406d4e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb48815c1c0 tx=0x7fb478008040 comp rx=0 tx=0).stop 2023-10-16T12:23:59.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.702+0000 7fb48dc76700 1 -- 172.21.15.16:0/402071532 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb48806a7e0 msgr2=0x7fb48815c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:59.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.702+0000 7fb48dc76700 1 --2- 172.21.15.16:0/402071532 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb48806a7e0 0x7fb48815c6f0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fb47c000c00 tx=0x7fb47c004750 comp rx=0 tx=0).stop 2023-10-16T12:23:59.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.702+0000 7fb48dc76700 1 -- 172.21.15.16:0/402071532 shutdown_connections 2023-10-16T12:23:59.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.702+0000 7fb48dc76700 1 --2- 172.21.15.16:0/402071532 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fb47406b020 0x7fb47406d4e0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:59.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.702+0000 7fb48dc76700 1 --2- 172.21.15.16:0/402071532 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb48815b280 0x7fb4880c5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:59.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.702+0000 7fb48dc76700 1 --2- 172.21.15.16:0/402071532 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb48806a7e0 0x7fb48815c6f0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:59.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.702+0000 7fb48dc76700 1 --2- 172.21.15.16:0/402071532 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb4880cc8a0 0x7fb4881f3160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:59.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.702+0000 7fb48dc76700 1 -- 172.21.15.16:0/402071532 >> 172.21.15.16:0/402071532 conn(0x7fb4880b1fd0 msgr2=0x7fb48815a900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:59.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.703+0000 7fb48dc76700 1 -- 172.21.15.16:0/402071532 shutdown_connections 2023-10-16T12:23:59.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.703+0000 7fb48dc76700 1 -- 172.21.15.16:0/402071532 wait complete. 2023-10-16T12:23:59.746 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:23:59.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.869+0000 7fb102c15700 1 Processor -- start 2023-10-16T12:23:59.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.869+0000 7fb102c15700 1 -- start start 2023-10-16T12:23:59.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.870+0000 7fb102c15700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fb0fc15a5c0 0x7fb0fc0cef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:59.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.870+0000 7fb102c15700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb0fc0d0ae0 0x7fb0fc0cc1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:59.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.870+0000 7fb102c15700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0fc072a50 con 0x7fb0fc0ce010 2023-10-16T12:23:59.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.870+0000 7fb102c15700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0fc0cf440 con 0x7fb0fc0d0ae0 2023-10-16T12:23:59.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.870+0000 7fb102c15700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0fc0cba30 con 0x7fb0fc15a5c0 2023-10-16T12:23:59.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.870+0000 7fb1011b2700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb0fc0d0ae0 0x7fb0fc0cc1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:59.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.870+0000 7fb1011b2700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb0fc0d0ae0 0x7fb0fc0cc1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37614/0 (socket says 172.21.15.16:37614) 2023-10-16T12:23:59.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.870+0000 7fb1011b2700 1 -- 172.21.15.16:0/1510206133 learned_addr learned my addr 172.21.15.16:0/1510206133 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:59.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.870+0000 7fb1009b1700 1 --2- 172.21.15.16:0/1510206133 >> v2:172.21.15.49:3300/0 conn(0x7fb0fc15a5c0 0x7fb0fc0cef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:59.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.870+0000 7fb1011b2700 1 -- 172.21.15.16:0/1510206133 >> v2:172.21.15.49:3300/0 conn(0x7fb0fc15a5c0 msgr2=0x7fb0fc0cef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:23:59.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.870+0000 7fb1011b2700 1 --2- 172.21.15.16:0/1510206133 >> v2:172.21.15.49:3300/0 conn(0x7fb0fc15a5c0 0x7fb0fc0cef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:59.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.870+0000 7fb1011b2700 1 -- 172.21.15.16:0/1510206133 >> v1:172.21.15.49:6789/0 conn(0x7fb0fc0ce010 legacy=0x7fb0fc0d03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:59.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.870+0000 7fb1011b2700 1 -- 172.21.15.16:0/1510206133 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0fc0cf630 con 0x7fb0fc0d0ae0 2023-10-16T12:23:59.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.870+0000 7fb1011b2700 1 --2- 172.21.15.16:0/1510206133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0fc0d0ae0 0x7fb0fc0cc1f0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc0b2ce0 tx=0x7fb0f400d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9da6f0ea7f8c108a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:59.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.871+0000 7fb0fb7fe700 1 -- 172.21.15.16:0/1510206133 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0f4010070 con 0x7fb0fc0d0ae0 2023-10-16T12:23:59.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.871+0000 7fb0fb7fe700 1 -- 172.21.15.16:0/1510206133 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0f400dd90 con 0x7fb0fc0d0ae0 2023-10-16T12:23:59.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.871+0000 7fb0fb7fe700 1 -- 172.21.15.16:0/1510206133 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0f400b310 con 0x7fb0fc0d0ae0 2023-10-16T12:23:59.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.871+0000 7fb102c15700 1 -- 172.21.15.16:0/1510206133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0fc0d0ae0 msgr2=0x7fb0fc0cc1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:59.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.871+0000 7fb102c15700 1 --2- 172.21.15.16:0/1510206133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0fc0d0ae0 0x7fb0fc0cc1f0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc0b2ce0 tx=0x7fb0f400d430 comp rx=0 tx=0).stop 2023-10-16T12:23:59.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.871+0000 7fb102c15700 1 -- 172.21.15.16:0/1510206133 shutdown_connections 2023-10-16T12:23:59.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.871+0000 7fb102c15700 1 --2- 172.21.15.16:0/1510206133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0fc0d0ae0 0x7fb0fc0cc1f0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:59.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.871+0000 7fb102c15700 1 --2- 172.21.15.16:0/1510206133 >> v2:172.21.15.49:3300/0 conn(0x7fb0fc15a5c0 0x7fb0fc0cef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:59.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.871+0000 7fb102c15700 1 -- 172.21.15.16:0/1510206133 >> 172.21.15.16:0/1510206133 conn(0x7fb0fc0b1fd0 msgr2=0x7fb0fc0b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:23:59.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.871+0000 7fb102c15700 1 -- 172.21.15.16:0/1510206133 shutdown_connections 2023-10-16T12:23:59.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.871+0000 7fb102c15700 1 -- 172.21.15.16:0/1510206133 wait complete. 2023-10-16T12:23:59.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.872+0000 7fb102c15700 1 Processor -- start 2023-10-16T12:23:59.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.872+0000 7fb102c15700 1 -- start start 2023-10-16T12:23:59.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.873+0000 7fb102c15700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0fc0ce010 0x7fb0fc0b30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:59.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.873+0000 7fb102c15700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb0fc0d0ae0 0x7fb0fc154030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:59.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.873+0000 7fb102c15700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb0fc15a5c0 0x7fb0fc0b4c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:59.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.873+0000 7fb102c15700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0fc0b41b0 con 0x7fb0fc0ce010 2023-10-16T12:23:59.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.873+0000 7fb102c15700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0fc0c8270 con 0x7fb0fc0d0ae0 2023-10-16T12:23:59.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.873+0000 7fb102c15700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0fc0b3910 con 0x7fb0fc15a5c0 2023-10-16T12:23:59.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.873+0000 7fb1009b1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0fc0ce010 0x7fb0fc0b30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:59.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.873+0000 7fb1009b1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0fc0ce010 0x7fb0fc0b30c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37616/0 (socket says 172.21.15.16:37616) 2023-10-16T12:23:59.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.873+0000 7fb1009b1700 1 -- 172.21.15.16:0/2868588324 learned_addr learned my addr 172.21.15.16:0/2868588324 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:23:59.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.874+0000 7fb1011b2700 1 --2- 172.21.15.16:0/2868588324 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb0fc15a5c0 0x7fb0fc0b4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:59.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.874+0000 7fb0fbfff700 1 --2- 172.21.15.16:0/2868588324 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb0fc0d0ae0 0x7fb0fc154030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:59.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.874+0000 7fb1009b1700 1 -- 172.21.15.16:0/2868588324 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb0fc15a5c0 msgr2=0x7fb0fc0b4c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:59.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.874+0000 7fb1009b1700 1 --2- 172.21.15.16:0/2868588324 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb0fc15a5c0 0x7fb0fc0b4c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:59.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.874+0000 7fb1009b1700 1 -- 172.21.15.16:0/2868588324 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb0fc0d0ae0 msgr2=0x7fb0fc154030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:23:59.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.874+0000 7fb1009b1700 1 --2- 172.21.15.16:0/2868588324 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb0fc0d0ae0 0x7fb0fc154030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:23:59.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.874+0000 7fb1009b1700 1 -- 172.21.15.16:0/2868588324 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0f400bd40 con 0x7fb0fc0ce010 2023-10-16T12:23:59.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.874+0000 7fb1009b1700 1 --2- 172.21.15.16:0/2868588324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0fc0ce010 0x7fb0fc0b30c0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fb0f000d800 tx=0x7fb0f000dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:59.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.874+0000 7fb0f97fa700 1 -- 172.21.15.16:0/2868588324 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0f00125f0 con 0x7fb0fc0ce010 2023-10-16T12:23:59.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.874+0000 7fb0f97fa700 1 -- 172.21.15.16:0/2868588324 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0f0012750 con 0x7fb0fc0ce010 2023-10-16T12:23:59.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.874+0000 7fb102c15700 1 -- 172.21.15.16:0/2868588324 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0fc0b3bf0 con 0x7fb0fc0ce010 2023-10-16T12:23:59.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.874+0000 7fb0f97fa700 1 -- 172.21.15.16:0/2868588324 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0f0012b00 con 0x7fb0fc0ce010 2023-10-16T12:23:59.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.874+0000 7fb102c15700 1 -- 172.21.15.16:0/2868588324 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0fc0c5160 con 0x7fb0fc0ce010 2023-10-16T12:23:59.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.875+0000 7fb102c15700 1 -- 172.21.15.16:0/2868588324 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0fc0c73e0 con 0x7fb0fc0ce010 2023-10-16T12:23:59.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.875+0000 7fb0f97fa700 1 -- 172.21.15.16:0/2868588324 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb0f0012c60 con 0x7fb0fc0ce010 2023-10-16T12:23:59.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.876+0000 7fb0f97fa700 1 --2- 172.21.15.16:0/2868588324 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fb0e006b020 0x7fb0e006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:23:59.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.876+0000 7fb0f97fa700 1 -- 172.21.15.16:0/2868588324 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fb0f00898f0 con 0x7fb0fc0ce010 2023-10-16T12:23:59.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.876+0000 7fb0fbfff700 1 --2- 172.21.15.16:0/2868588324 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fb0e006b020 0x7fb0e006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:23:59.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.876+0000 7fb0fbfff700 1 --2- 172.21.15.16:0/2868588324 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fb0e006b020 0x7fb0e006d4e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc15daa0 tx=0x7fb0e8009040 comp rx=0 tx=0).ready entity=mgr.5874 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:23:59.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:23:59.880+0000 7fb0f97fa700 1 -- 172.21.15.16:0/2868588324 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb0f0054aa0 con 0x7fb0fc0ce010 2023-10-16T12:24:00.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.085+0000 7fb102c15700 1 -- 172.21.15.16:0/2868588324 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb0fc0c54d0 con 0x7fb0fc0ce010 2023-10-16T12:24:00.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.086+0000 7fb0f97fa700 1 -- 172.21.15.16:0/2868588324 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v64) v1 ==== 74+0+12954 (secure 0 0 0) 0x7fb0f00580c0 con 0x7fb0fc0ce010 2023-10-16T12:24:00.088 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:24:00.089 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":64,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:23:59.260642+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:24:00.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.088+0000 7fb102c15700 1 -- 172.21.15.16:0/2868588324 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fb0e006b020 msgr2=0x7fb0e006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:00.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.088+0000 7fb102c15700 1 --2- 172.21.15.16:0/2868588324 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fb0e006b020 0x7fb0e006d4e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc15daa0 tx=0x7fb0e8009040 comp rx=0 tx=0).stop 2023-10-16T12:24:00.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.088+0000 7fb102c15700 1 -- 172.21.15.16:0/2868588324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0fc0ce010 msgr2=0x7fb0fc0b30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:00.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.088+0000 7fb102c15700 1 --2- 172.21.15.16:0/2868588324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0fc0ce010 0x7fb0fc0b30c0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fb0f000d800 tx=0x7fb0f000dbc0 comp rx=0 tx=0).stop 2023-10-16T12:24:00.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.088+0000 7fb102c15700 1 -- 172.21.15.16:0/2868588324 shutdown_connections 2023-10-16T12:24:00.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.088+0000 7fb102c15700 1 --2- 172.21.15.16:0/2868588324 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fb0e006b020 0x7fb0e006d4e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:00.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.088+0000 7fb102c15700 1 --2- 172.21.15.16:0/2868588324 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb0fc15a5c0 0x7fb0fc0b4c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:00.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.088+0000 7fb102c15700 1 --2- 172.21.15.16:0/2868588324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0fc0ce010 0x7fb0fc0b30c0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:00.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.088+0000 7fb102c15700 1 --2- 172.21.15.16:0/2868588324 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb0fc0d0ae0 0x7fb0fc154030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:00.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.088+0000 7fb102c15700 1 -- 172.21.15.16:0/2868588324 >> 172.21.15.16:0/2868588324 conn(0x7fb0fc0b1fd0 msgr2=0x7fb0fc06a7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:24:00.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.088+0000 7fb102c15700 1 -- 172.21.15.16:0/2868588324 shutdown_connections 2023-10-16T12:24:00.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.088+0000 7fb102c15700 1 -- 172.21.15.16:0/2868588324 wait complete. 2023-10-16T12:24:00.103 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T12:24:00.103 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:24:00.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.229+0000 7f94d07df700 1 Processor -- start 2023-10-16T12:24:00.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.229+0000 7f94d07df700 1 -- start start 2023-10-16T12:24:00.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.230+0000 7f94d07df700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f94c806a7e0 0x7f94c806ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:00.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.230+0000 7f94d07df700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94c815ecb0 con 0x7f94c806e570 2023-10-16T12:24:00.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.230+0000 7f94d07df700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94c815b6a0 con 0x7f94c815b2c0 2023-10-16T12:24:00.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.230+0000 7f94d07df700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94c806b1b0 con 0x7f94c806a7e0 2023-10-16T12:24:00.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.231+0000 7f94cdd7a700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f94c806e570 0x7f94c815e5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:55130/0 (socket says 172.21.15.16:55130) 2023-10-16T12:24:00.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.231+0000 7f94cdd7a700 1 -- 172.21.15.16:0/1260726281 learned_addr learned my addr 172.21.15.16:0/1260726281 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:24:00.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.231+0000 7f94ced7c700 1 --2- 172.21.15.16:0/1260726281 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f94c806a7e0 0x7f94c806ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:00.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.231+0000 7f94cd579700 1 -- 172.21.15.16:0/1260726281 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2560521764 0 0) 0x7f94c815ecb0 con 0x7f94c806e570 2023-10-16T12:24:00.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.231+0000 7f94cd579700 1 -- 172.21.15.16:0/1260726281 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94b40035b0 con 0x7f94c806e570 2023-10-16T12:24:00.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.231+0000 7f94cd579700 1 -- 172.21.15.16:0/1260726281 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3204499674 0 0) 0x7f94c815b6a0 con 0x7f94c815b2c0 2023-10-16T12:24:00.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.231+0000 7f94cd579700 1 -- 172.21.15.16:0/1260726281 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94c815ecb0 con 0x7f94c815b2c0 2023-10-16T12:24:00.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.231+0000 7f94cd579700 1 -- 172.21.15.16:0/1260726281 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3957703862 0 0) 0x7f94b40035b0 con 0x7f94c806e570 2023-10-16T12:24:00.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.231+0000 7f94cd579700 1 -- 172.21.15.16:0/1260726281 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f94c806a7e0 msgr2=0x7f94c806ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:24:00.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.231+0000 7f94cd579700 1 --2- 172.21.15.16:0/1260726281 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f94c806a7e0 0x7f94c806ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:00.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.231+0000 7f94cd579700 1 -- 172.21.15.16:0/1260726281 >> v1:172.21.15.49:6789/0 conn(0x7f94c815b2c0 legacy=0x7f94c8161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:00.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.231+0000 7f94cd579700 1 -- 172.21.15.16:0/1260726281 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94c8161b20 con 0x7f94c806e570 2023-10-16T12:24:00.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.231+0000 7f94cd579700 1 -- 172.21.15.16:0/1260726281 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f94bc001670 con 0x7f94c806e570 2023-10-16T12:24:00.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.231+0000 7f94cd579700 1 -- 172.21.15.16:0/1260726281 >> v1:172.21.15.16:6789/0 conn(0x7f94c806e570 legacy=0x7f94c815e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:00.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.231+0000 7f94cd579700 1 --2- 172.21.15.16:0/1260726281 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f94c806a7e0 0x7f94b4003bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:00.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.231+0000 7f94cd579700 1 --2- 172.21.15.16:0/1260726281 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94b4004120 0x7f94b4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:00.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.231+0000 7f94cd579700 1 --2- 172.21.15.16:0/1260726281 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f94b4004d90 0x7f94b4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:00.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.231+0000 7f94cd579700 1 -- 172.21.15.16:0/1260726281 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94c806b1b0 con 0x7f94b4004120 2023-10-16T12:24:00.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.231+0000 7f94cd579700 1 -- 172.21.15.16:0/1260726281 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94b400bba0 con 0x7f94b4004d90 2023-10-16T12:24:00.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.231+0000 7f94cd579700 1 -- 172.21.15.16:0/1260726281 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94b400bd30 con 0x7f94c806a7e0 2023-10-16T12:24:00.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.231+0000 7f94cdd7a700 1 --2- 172.21.15.16:0/1260726281 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94b4004120 0x7f94b4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:00.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.231+0000 7f94ce57b700 1 --2- 172.21.15.16:0/1260726281 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f94c806a7e0 0x7f94b4003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:00.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.231+0000 7f94ced7c700 1 --2- 172.21.15.16:0/1260726281 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f94b4004d90 0x7f94b4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:00.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.232+0000 7f94cdd7a700 1 -- 172.21.15.16:0/1260726281 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f94c806a7e0 msgr2=0x7f94b4003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:24:00.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.232+0000 7f94cdd7a700 1 --2- 172.21.15.16:0/1260726281 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f94c806a7e0 0x7f94b4003bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:00.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.232+0000 7f94cdd7a700 1 -- 172.21.15.16:0/1260726281 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f94b4004d90 msgr2=0x7f94b4005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:24:00.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.232+0000 7f94cdd7a700 1 --2- 172.21.15.16:0/1260726281 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f94b4004d90 0x7f94b4005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:00.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.232+0000 7f94cdd7a700 1 -- 172.21.15.16:0/1260726281 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94b40035b0 con 0x7f94b4004120 2023-10-16T12:24:00.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.232+0000 7f94ced7c700 1 --2- 172.21.15.16:0/1260726281 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f94b4004d90 0x7f94b4005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:24:00.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.232+0000 7f94cdd7a700 1 --2- 172.21.15.16:0/1260726281 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94b4004120 0x7f94b4004850 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f94bc00c210 tx=0x7f94b4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c138c65d67ee4491 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:24:00.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.232+0000 7f94cd579700 1 -- 172.21.15.16:0/1260726281 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94bc00cd50 con 0x7f94b4004120 2023-10-16T12:24:00.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.232+0000 7f94cd579700 1 -- 172.21.15.16:0/1260726281 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94bc005960 con 0x7f94b4004120 2023-10-16T12:24:00.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.232+0000 7f94cd579700 1 -- 172.21.15.16:0/1260726281 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94bc005d10 con 0x7f94b4004120 2023-10-16T12:24:00.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.232+0000 7f94d07df700 1 -- 172.21.15.16:0/1260726281 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94b4004120 msgr2=0x7f94b4004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:00.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.232+0000 7f94d07df700 1 --2- 172.21.15.16:0/1260726281 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94b4004120 0x7f94b4004850 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f94bc00c210 tx=0x7f94b4005880 comp rx=0 tx=0).stop 2023-10-16T12:24:00.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.232+0000 7f94ced7c700 1 -- 172.21.15.16:0/1260726281 reap_dead start 2023-10-16T12:24:00.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.233+0000 7f94d07df700 1 -- 172.21.15.16:0/1260726281 shutdown_connections 2023-10-16T12:24:00.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.233+0000 7f94d07df700 1 -- 172.21.15.16:0/1260726281 >> 172.21.15.16:0/1260726281 conn(0x7f94c80b1f40 msgr2=0x7f94c80b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:24:00.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.233+0000 7f94d07df700 1 -- 172.21.15.16:0/1260726281 shutdown_connections 2023-10-16T12:24:00.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.233+0000 7f94d07df700 1 -- 172.21.15.16:0/1260726281 wait complete. 2023-10-16T12:24:00.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.234+0000 7f94d07df700 1 Processor -- start 2023-10-16T12:24:00.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.234+0000 7f94d07df700 1 -- start start 2023-10-16T12:24:00.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.235+0000 7f94d07df700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f94b4004120 0x7f94c815d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:00.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.235+0000 7f94d07df700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f94c8157a90 0x7f94c80b3a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:00.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.235+0000 7f94d07df700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94c80b3f70 0x7f94c81f6950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:00.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.235+0000 7f94d07df700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94c80b4420 con 0x7f94c80b3f70 2023-10-16T12:24:00.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.235+0000 7f94d07df700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94c81f4520 con 0x7f94c8157a90 2023-10-16T12:24:00.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.235+0000 7f94d07df700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94c815d840 con 0x7f94b4004120 2023-10-16T12:24:00.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.235+0000 7f94ced7c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94c80b3f70 0x7f94c81f6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:00.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.235+0000 7f94ced7c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94c80b3f70 0x7f94c81f6950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37638/0 (socket says 172.21.15.16:37638) 2023-10-16T12:24:00.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.235+0000 7f94ced7c700 1 -- 172.21.15.16:0/3878223002 learned_addr learned my addr 172.21.15.16:0/3878223002 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:24:00.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.235+0000 7f94cdd7a700 1 --2- 172.21.15.16:0/3878223002 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f94c8157a90 0x7f94c80b3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:00.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.235+0000 7f94ce57b700 1 --2- 172.21.15.16:0/3878223002 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f94b4004120 0x7f94c815d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:00.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.235+0000 7f94ced7c700 1 -- 172.21.15.16:0/3878223002 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f94b4004120 msgr2=0x7f94c815d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:00.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.235+0000 7f94ced7c700 1 --2- 172.21.15.16:0/3878223002 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f94b4004120 0x7f94c815d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:00.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.235+0000 7f94ced7c700 1 -- 172.21.15.16:0/3878223002 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f94c8157a90 msgr2=0x7f94c80b3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:00.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.235+0000 7f94ced7c700 1 --2- 172.21.15.16:0/3878223002 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f94c8157a90 0x7f94c80b3a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:00.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.235+0000 7f94ced7c700 1 -- 172.21.15.16:0/3878223002 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94c81f4d80 con 0x7f94c80b3f70 2023-10-16T12:24:00.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.235+0000 7f94cdd7a700 1 --2- 172.21.15.16:0/3878223002 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f94c8157a90 0x7f94c80b3a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:24:00.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.235+0000 7f94ced7c700 1 --2- 172.21.15.16:0/3878223002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94c80b3f70 0x7f94c81f6950 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f94c806b430 tx=0x7f94c00104a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:24:00.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.235+0000 7f94baffd700 1 -- 172.21.15.16:0/3878223002 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c001b070 con 0x7f94c80b3f70 2023-10-16T12:24:00.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.235+0000 7f94baffd700 1 -- 172.21.15.16:0/3878223002 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94c000f040 con 0x7f94c80b3f70 2023-10-16T12:24:00.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.235+0000 7f94baffd700 1 -- 172.21.15.16:0/3878223002 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c000e050 con 0x7f94c80b3f70 2023-10-16T12:24:00.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.235+0000 7f94d07df700 1 -- 172.21.15.16:0/3878223002 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94c81f4150 con 0x7f94c80b3f70 2023-10-16T12:24:00.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.236+0000 7f94d07df700 1 -- 172.21.15.16:0/3878223002 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94c81f4a10 con 0x7f94c80b3f70 2023-10-16T12:24:00.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.237+0000 7f94d07df700 1 -- 172.21.15.16:0/3878223002 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94c804ebb0 con 0x7f94c80b3f70 2023-10-16T12:24:00.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.237+0000 7f94baffd700 1 -- 172.21.15.16:0/3878223002 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f94c0015440 con 0x7f94c80b3f70 2023-10-16T12:24:00.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.237+0000 7f94baffd700 1 --2- 172.21.15.16:0/3878223002 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f94ac06b020 0x7f94ac06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:00.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.237+0000 7f94baffd700 1 -- 172.21.15.16:0/3878223002 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f94c0089610 con 0x7f94c80b3f70 2023-10-16T12:24:00.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.238+0000 7f94ce57b700 1 --2- 172.21.15.16:0/3878223002 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f94ac06b020 0x7f94ac06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:00.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.238+0000 7f94ce57b700 1 --2- 172.21.15.16:0/3878223002 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f94ac06b020 0x7f94ac06d4e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f94c4007fd0 tx=0x7f94c4009040 comp rx=0 tx=0).ready entity=mgr.5874 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:24:00.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.242+0000 7f94baffd700 1 -- 172.21.15.16:0/3878223002 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f94c0054630 con 0x7f94c80b3f70 2023-10-16T12:24:00.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.447+0000 7f94d07df700 1 -- 172.21.15.16:0/3878223002 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f94c806e2e0 con 0x7f94c80b3f70 2023-10-16T12:24:00.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.448+0000 7f94baffd700 1 -- 172.21.15.16:0/3878223002 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v64) v1 ==== 93+0+10 (secure 0 0 0) 0x7f94c00890a0 con 0x7f94c80b3f70 2023-10-16T12:24:00.450 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T12:24:00.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.450+0000 7f94d07df700 1 -- 172.21.15.16:0/3878223002 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f94ac06b020 msgr2=0x7f94ac06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:00.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.450+0000 7f94d07df700 1 --2- 172.21.15.16:0/3878223002 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f94ac06b020 0x7f94ac06d4e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f94c4007fd0 tx=0x7f94c4009040 comp rx=0 tx=0).stop 2023-10-16T12:24:00.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.450+0000 7f94d07df700 1 -- 172.21.15.16:0/3878223002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94c80b3f70 msgr2=0x7f94c81f6950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:00.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.450+0000 7f94d07df700 1 --2- 172.21.15.16:0/3878223002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94c80b3f70 0x7f94c81f6950 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f94c806b430 tx=0x7f94c00104a0 comp rx=0 tx=0).stop 2023-10-16T12:24:00.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.450+0000 7f94d07df700 1 -- 172.21.15.16:0/3878223002 shutdown_connections 2023-10-16T12:24:00.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.450+0000 7f94d07df700 1 --2- 172.21.15.16:0/3878223002 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f94ac06b020 0x7f94ac06d4e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:00.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.450+0000 7f94d07df700 1 --2- 172.21.15.16:0/3878223002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94c80b3f70 0x7f94c81f6950 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:00.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.450+0000 7f94d07df700 1 --2- 172.21.15.16:0/3878223002 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f94b4004120 0x7f94c815d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:00.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.450+0000 7f94d07df700 1 --2- 172.21.15.16:0/3878223002 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f94c8157a90 0x7f94c80b3a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:00.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.450+0000 7f94d07df700 1 -- 172.21.15.16:0/3878223002 >> 172.21.15.16:0/3878223002 conn(0x7f94c80b1f40 msgr2=0x7f94c80cb3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:24:00.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.450+0000 7f94d07df700 1 -- 172.21.15.16:0/3878223002 shutdown_connections 2023-10-16T12:24:00.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.450+0000 7f94d07df700 1 -- 172.21.15.16:0/3878223002 wait complete. 2023-10-16T12:24:00.464 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-16T12:24:00.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.587+0000 7f6236dd7700 1 Processor -- start 2023-10-16T12:24:00.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.587+0000 7f6236dd7700 1 -- start start 2023-10-16T12:24:00.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.588+0000 7f6236dd7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f623006a7e0 0x7f623006ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:00.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.588+0000 7f6236dd7700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f623006b150 con 0x7f62300cc8a0 2023-10-16T12:24:00.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.588+0000 7f6236dd7700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6230161c30 con 0x7f623015b280 2023-10-16T12:24:00.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.588+0000 7f6236dd7700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6230161f10 con 0x7f623006a7e0 2023-10-16T12:24:00.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.588+0000 7f6235374700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f623006a7e0 0x7f623006ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:00.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.588+0000 7f6235374700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f623006a7e0 0x7f623006ac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37652/0 (socket says 172.21.15.16:37652) 2023-10-16T12:24:00.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.588+0000 7f6235374700 1 -- 172.21.15.16:0/2258222012 learned_addr learned my addr 172.21.15.16:0/2258222012 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:24:00.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.589+0000 7f6235374700 1 -- 172.21.15.16:0/2258222012 >> v1:172.21.15.49:6789/0 conn(0x7f623015b280 legacy=0x7f62301613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:00.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.589+0000 7f6235374700 1 -- 172.21.15.16:0/2258222012 >> v1:172.21.15.16:6789/0 conn(0x7f62300cc8a0 legacy=0x7f623015e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:00.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.589+0000 7f6235374700 1 -- 172.21.15.16:0/2258222012 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62301621f0 con 0x7f623006a7e0 2023-10-16T12:24:00.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.589+0000 7f6235374700 1 --2- 172.21.15.16:0/2258222012 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f623006a7e0 0x7f623006ac10 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f6228009e50 tx=0x7f622800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=500f6a960800ea14 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:24:00.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.589+0000 7f622f7fe700 1 -- 172.21.15.16:0/2258222012 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f622800e070 con 0x7f623006a7e0 2023-10-16T12:24:00.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.589+0000 7f622f7fe700 1 -- 172.21.15.16:0/2258222012 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f622800b9c0 con 0x7f623006a7e0 2023-10-16T12:24:00.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.589+0000 7f622f7fe700 1 -- 172.21.15.16:0/2258222012 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f622800bd70 con 0x7f623006a7e0 2023-10-16T12:24:00.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.589+0000 7f6236dd7700 1 -- 172.21.15.16:0/2258222012 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f623006a7e0 msgr2=0x7f623006ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:00.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.589+0000 7f6236dd7700 1 --2- 172.21.15.16:0/2258222012 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f623006a7e0 0x7f623006ac10 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f6228009e50 tx=0x7f622800b330 comp rx=0 tx=0).stop 2023-10-16T12:24:00.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.589+0000 7f6236dd7700 1 -- 172.21.15.16:0/2258222012 shutdown_connections 2023-10-16T12:24:00.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.589+0000 7f6236dd7700 1 --2- 172.21.15.16:0/2258222012 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f623006a7e0 0x7f623006ac10 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:00.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.589+0000 7f6236dd7700 1 -- 172.21.15.16:0/2258222012 >> 172.21.15.16:0/2258222012 conn(0x7f62300b1fd0 msgr2=0x7f62300b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:24:00.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.590+0000 7f6236dd7700 1 -- 172.21.15.16:0/2258222012 shutdown_connections 2023-10-16T12:24:00.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.590+0000 7f6236dd7700 1 -- 172.21.15.16:0/2258222012 wait complete. 2023-10-16T12:24:00.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.591+0000 7f6236dd7700 1 Processor -- start 2023-10-16T12:24:00.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.591+0000 7f6236dd7700 1 -- start start 2023-10-16T12:24:00.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.592+0000 7f6236dd7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f623006a7e0 0x7f62301588d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:00.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.592+0000 7f6236dd7700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f62300cc8a0 0x7f62300c83b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:00.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.592+0000 7f6236dd7700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f623015b280 0x7f623015b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:00.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.592+0000 7f6236dd7700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62300c6020 con 0x7f623006a7e0 2023-10-16T12:24:00.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.592+0000 7f6236dd7700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62300c56f0 con 0x7f62300cc8a0 2023-10-16T12:24:00.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.592+0000 7f6236dd7700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62300c59d0 con 0x7f623015b280 2023-10-16T12:24:00.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.592+0000 7f6234b73700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f623006a7e0 0x7f62301588d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:00.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.592+0000 7f6234b73700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f623006a7e0 0x7f62301588d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37654/0 (socket says 172.21.15.16:37654) 2023-10-16T12:24:00.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.592+0000 7f6234b73700 1 -- 172.21.15.16:0/1355269114 learned_addr learned my addr 172.21.15.16:0/1355269114 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:24:00.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.593+0000 7f622ffff700 1 --2- 172.21.15.16:0/1355269114 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f62300cc8a0 0x7f62300c83b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:00.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.593+0000 7f6235374700 1 --2- 172.21.15.16:0/1355269114 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f623015b280 0x7f623015b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:00.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.593+0000 7f6234b73700 1 -- 172.21.15.16:0/1355269114 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f623015b280 msgr2=0x7f623015b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:00.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.593+0000 7f6234b73700 1 --2- 172.21.15.16:0/1355269114 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f623015b280 0x7f623015b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:00.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.593+0000 7f6234b73700 1 -- 172.21.15.16:0/1355269114 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f62300cc8a0 msgr2=0x7f62300c83b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:00.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.593+0000 7f6234b73700 1 --2- 172.21.15.16:0/1355269114 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f62300cc8a0 0x7f62300c83b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:00.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.593+0000 7f6234b73700 1 -- 172.21.15.16:0/1355269114 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6228009b30 con 0x7f623006a7e0 2023-10-16T12:24:00.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.593+0000 7f622ffff700 1 --2- 172.21.15.16:0/1355269114 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f62300cc8a0 0x7f62300c83b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:24:00.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.593+0000 7f6234b73700 1 --2- 172.21.15.16:0/1355269114 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f623006a7e0 0x7f62301588d0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f622400c450 tx=0x7f622400c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:24:00.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.593+0000 7f622d7fa700 1 -- 172.21.15.16:0/1355269114 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62240052d0 con 0x7f623006a7e0 2023-10-16T12:24:00.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.593+0000 7f622d7fa700 1 -- 172.21.15.16:0/1355269114 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6224005430 con 0x7f623006a7e0 2023-10-16T12:24:00.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.593+0000 7f6236dd7700 1 -- 172.21.15.16:0/1355269114 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62300c5cb0 con 0x7f623006a7e0 2023-10-16T12:24:00.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.593+0000 7f622d7fa700 1 -- 172.21.15.16:0/1355269114 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62240038c0 con 0x7f623006a7e0 2023-10-16T12:24:00.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.593+0000 7f6236dd7700 1 -- 172.21.15.16:0/1355269114 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6230158e10 con 0x7f623006a7e0 2023-10-16T12:24:00.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.594+0000 7f6236dd7700 1 -- 172.21.15.16:0/1355269114 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f623004ebb0 con 0x7f623006a7e0 2023-10-16T12:24:00.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.595+0000 7f622d7fa700 1 -- 172.21.15.16:0/1355269114 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6224003a20 con 0x7f623006a7e0 2023-10-16T12:24:00.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.595+0000 7f622d7fa700 1 --2- 172.21.15.16:0/1355269114 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f621406b020 0x7f621406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:00.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.595+0000 7f622d7fa700 1 -- 172.21.15.16:0/1355269114 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f6224088950 con 0x7f623006a7e0 2023-10-16T12:24:00.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.595+0000 7f622ffff700 1 --2- 172.21.15.16:0/1355269114 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f621406b020 0x7f621406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:00.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.595+0000 7f622ffff700 1 --2- 172.21.15.16:0/1355269114 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f621406b020 0x7f621406d4e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f62300afa20 tx=0x7f621c009040 comp rx=0 tx=0).ready entity=mgr.5874 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:24:00.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.600+0000 7f622d7fa700 1 -- 172.21.15.16:0/1355269114 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6224053b00 con 0x7f623006a7e0 2023-10-16T12:24:00.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.809+0000 7f6236dd7700 1 -- 172.21.15.16:0/1355269114 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f62300cc050 con 0x7f623006a7e0 2023-10-16T12:24:00.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.810+0000 7f622d7fa700 1 -- 172.21.15.16:0/1355269114 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v64) v1 ==== 84+0+65 (secure 0 0 0) 0x7f6224017090 con 0x7f623006a7e0 2023-10-16T12:24:00.812 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:24:00.813 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2023-10-16T12:24:00.813 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:24:00.813 INFO:teuthology.orchestra.run.smithi016.stdout: "poolnum": 1, 2023-10-16T12:24:00.813 INFO:teuthology.orchestra.run.smithi016.stdout: "poolname": ".mgr" 2023-10-16T12:24:00.814 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:24:00.814 INFO:teuthology.orchestra.run.smithi016.stdout:] 2023-10-16T12:24:00.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.812+0000 7f6236dd7700 1 -- 172.21.15.16:0/1355269114 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f621406b020 msgr2=0x7f621406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:00.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.812+0000 7f6236dd7700 1 --2- 172.21.15.16:0/1355269114 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f621406b020 0x7f621406d4e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f62300afa20 tx=0x7f621c009040 comp rx=0 tx=0).stop 2023-10-16T12:24:00.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.812+0000 7f6236dd7700 1 -- 172.21.15.16:0/1355269114 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f623006a7e0 msgr2=0x7f62301588d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:00.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.812+0000 7f6236dd7700 1 --2- 172.21.15.16:0/1355269114 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f623006a7e0 0x7f62301588d0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f622400c450 tx=0x7f622400c810 comp rx=0 tx=0).stop 2023-10-16T12:24:00.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.812+0000 7f6236dd7700 1 -- 172.21.15.16:0/1355269114 shutdown_connections 2023-10-16T12:24:00.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.812+0000 7f6236dd7700 1 --2- 172.21.15.16:0/1355269114 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f621406b020 0x7f621406d4e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:00.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.812+0000 7f6236dd7700 1 --2- 172.21.15.16:0/1355269114 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f623015b280 0x7f623015b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:00.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.812+0000 7f6236dd7700 1 --2- 172.21.15.16:0/1355269114 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f623006a7e0 0x7f62301588d0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:00.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.812+0000 7f6236dd7700 1 --2- 172.21.15.16:0/1355269114 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f62300cc8a0 0x7f62300c83b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:00.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.812+0000 7f6236dd7700 1 -- 172.21.15.16:0/1355269114 >> 172.21.15.16:0/1355269114 conn(0x7f62300b1fd0 msgr2=0x7f62300b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:24:00.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.812+0000 7f6236dd7700 1 -- 172.21.15.16:0/1355269114 shutdown_connections 2023-10-16T12:24:00.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.812+0000 7f6236dd7700 1 -- 172.21.15.16:0/1355269114 wait complete. 2023-10-16T12:24:00.826 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-16T12:24:00.827 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-16T12:24:00.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.954+0000 7f7d8a76b700 1 Processor -- start 2023-10-16T12:24:00.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.954+0000 7f7d8a76b700 1 -- start start 2023-10-16T12:24:00.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.955+0000 7f7d8a76b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7d84160e60 0x7f7d8415e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:00.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.955+0000 7f7d8a76b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7d8406a7e0 0x7f7d8406abc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:00.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.955+0000 7f7d8a76b700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d840cdeb0 con 0x7f7d8406d980 2023-10-16T12:24:00.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.955+0000 7f7d8a76b700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d8415f6c0 con 0x7f7d8406a7e0 2023-10-16T12:24:00.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.955+0000 7f7d8a76b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d841613e0 con 0x7f7d84160e60 2023-10-16T12:24:00.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.955+0000 7f7d88d08700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7d8406a7e0 0x7f7d8406abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:00.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.955+0000 7f7d88d08700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7d8406a7e0 0x7f7d8406abc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37666/0 (socket says 172.21.15.16:37666) 2023-10-16T12:24:00.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.955+0000 7f7d88d08700 1 -- 172.21.15.16:0/1079856117 learned_addr learned my addr 172.21.15.16:0/1079856117 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:24:00.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.956+0000 7f7d83fff700 1 --2- 172.21.15.16:0/1079856117 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7d84160e60 0x7f7d8415e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:00.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.956+0000 7f7d88d08700 1 -- 172.21.15.16:0/1079856117 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7d84160e60 msgr2=0x7f7d8415e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:24:00.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.956+0000 7f7d88d08700 1 --2- 172.21.15.16:0/1079856117 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7d84160e60 0x7f7d8415e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:00.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.956+0000 7f7d88d08700 1 -- 172.21.15.16:0/1079856117 >> v1:172.21.15.16:6789/0 conn(0x7f7d8406d980 legacy=0x7f7d8415efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:00.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.956+0000 7f7d88d08700 1 -- 172.21.15.16:0/1079856117 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d841582d0 con 0x7f7d8406a7e0 2023-10-16T12:24:00.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.956+0000 7f7d88d08700 1 --2- 172.21.15.16:0/1079856117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d8406a7e0 0x7f7d8406abc0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f7d7c009e50 tx=0x7f7d7c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0d556c3a91412e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:24:00.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.956+0000 7f7d82ffd700 1 -- 172.21.15.16:0/1079856117 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d7c00e070 con 0x7f7d8406a7e0 2023-10-16T12:24:00.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.956+0000 7f7d82ffd700 1 -- 172.21.15.16:0/1079856117 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d7c00b9c0 con 0x7f7d8406a7e0 2023-10-16T12:24:00.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.956+0000 7f7d82ffd700 1 -- 172.21.15.16:0/1079856117 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d7c00bd70 con 0x7f7d8406a7e0 2023-10-16T12:24:00.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.957+0000 7f7d8a76b700 1 -- 172.21.15.16:0/1079856117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d8406a7e0 msgr2=0x7f7d8406abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:00.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.957+0000 7f7d8a76b700 1 --2- 172.21.15.16:0/1079856117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d8406a7e0 0x7f7d8406abc0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f7d7c009e50 tx=0x7f7d7c00b330 comp rx=0 tx=0).stop 2023-10-16T12:24:00.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.957+0000 7f7d8a76b700 1 -- 172.21.15.16:0/1079856117 shutdown_connections 2023-10-16T12:24:00.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.957+0000 7f7d8a76b700 1 --2- 172.21.15.16:0/1079856117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d8406a7e0 0x7f7d8406abc0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:00.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.957+0000 7f7d8a76b700 1 --2- 172.21.15.16:0/1079856117 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7d84160e60 0x7f7d8415e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:00.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.957+0000 7f7d8a76b700 1 -- 172.21.15.16:0/1079856117 >> 172.21.15.16:0/1079856117 conn(0x7f7d8406f430 msgr2=0x7f7d8406f810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:24:00.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.957+0000 7f7d8a76b700 1 -- 172.21.15.16:0/1079856117 shutdown_connections 2023-10-16T12:24:00.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.957+0000 7f7d8a76b700 1 -- 172.21.15.16:0/1079856117 wait complete. 2023-10-16T12:24:00.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.958+0000 7f7d8a76b700 1 Processor -- start 2023-10-16T12:24:00.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.958+0000 7f7d8a76b700 1 -- start start 2023-10-16T12:24:00.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.959+0000 7f7d8a76b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7d8406a7e0 0x7f7d840b0160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:00.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.959+0000 7f7d8a76b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7d8406d980 0x7f7d84156770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:00.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.959+0000 7f7d8a76b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d84160e60 0x7f7d84160070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:00.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.959+0000 7f7d8a76b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d841e7b60 con 0x7f7d84160e60 2023-10-16T12:24:00.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.959+0000 7f7d8a76b700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d840c5010 con 0x7f7d8406a7e0 2023-10-16T12:24:00.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.959+0000 7f7d8a76b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d840b06a0 con 0x7f7d8406d980 2023-10-16T12:24:00.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.959+0000 7f7d88d08700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d84160e60 0x7f7d84160070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:00.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.959+0000 7f7d88d08700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d84160e60 0x7f7d84160070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37672/0 (socket says 172.21.15.16:37672) 2023-10-16T12:24:00.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.959+0000 7f7d88d08700 1 -- 172.21.15.16:0/2025606514 learned_addr learned my addr 172.21.15.16:0/2025606514 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:24:00.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.959+0000 7f7d83fff700 1 --2- 172.21.15.16:0/2025606514 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7d8406a7e0 0x7f7d840b0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:00.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.959+0000 7f7d837fe700 1 --2- 172.21.15.16:0/2025606514 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7d8406d980 0x7f7d84156770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:00.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.959+0000 7f7d88d08700 1 -- 172.21.15.16:0/2025606514 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7d8406d980 msgr2=0x7f7d84156770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:00.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.959+0000 7f7d88d08700 1 --2- 172.21.15.16:0/2025606514 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7d8406d980 0x7f7d84156770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:00.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.959+0000 7f7d88d08700 1 -- 172.21.15.16:0/2025606514 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7d8406a7e0 msgr2=0x7f7d840b0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:00.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.959+0000 7f7d88d08700 1 --2- 172.21.15.16:0/2025606514 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7d8406a7e0 0x7f7d840b0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:00.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.959+0000 7f7d88d08700 1 -- 172.21.15.16:0/2025606514 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d7c009b30 con 0x7f7d84160e60 2023-10-16T12:24:00.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.959+0000 7f7d837fe700 1 --2- 172.21.15.16:0/2025606514 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7d8406d980 0x7f7d84156770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:24:00.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.959+0000 7f7d88d08700 1 --2- 172.21.15.16:0/2025606514 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d84160e60 0x7f7d84160070 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f7d8406c8d0 tx=0x7f7d7c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:24:00.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.960+0000 7f7d80ff9700 1 -- 172.21.15.16:0/2025606514 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d7c0048b0 con 0x7f7d84160e60 2023-10-16T12:24:00.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.960+0000 7f7d80ff9700 1 -- 172.21.15.16:0/2025606514 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d7c0092e0 con 0x7f7d84160e60 2023-10-16T12:24:00.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.960+0000 7f7d80ff9700 1 -- 172.21.15.16:0/2025606514 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d7c01c660 con 0x7f7d84160e60 2023-10-16T12:24:00.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.960+0000 7f7d8a76b700 1 -- 172.21.15.16:0/2025606514 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d840b5420 con 0x7f7d84160e60 2023-10-16T12:24:00.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.960+0000 7f7d8a76b700 1 -- 172.21.15.16:0/2025606514 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d840c4d30 con 0x7f7d84160e60 2023-10-16T12:24:00.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.961+0000 7f7d8a76b700 1 -- 172.21.15.16:0/2025606514 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d841e7d10 con 0x7f7d84160e60 2023-10-16T12:24:00.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.961+0000 7f7d80ff9700 1 -- 172.21.15.16:0/2025606514 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7d7c01c7c0 con 0x7f7d84160e60 2023-10-16T12:24:00.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.962+0000 7f7d80ff9700 1 --2- 172.21.15.16:0/2025606514 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f7d6406b020 0x7f7d6406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:00.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.962+0000 7f7d80ff9700 1 -- 172.21.15.16:0/2025606514 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f7d7c08b280 con 0x7f7d84160e60 2023-10-16T12:24:00.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.962+0000 7f7d83fff700 1 --2- 172.21.15.16:0/2025606514 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f7d6406b020 0x7f7d6406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:00.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.962+0000 7f7d83fff700 1 --2- 172.21.15.16:0/2025606514 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f7d6406b020 0x7f7d6406d4e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7d74006b20 tx=0x7f7d74008040 comp rx=0 tx=0).ready entity=mgr.5874 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:24:00.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:00.966+0000 7f7d80ff9700 1 -- 172.21.15.16:0/2025606514 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7d841e7d10 con 0x7f7d84160e60 2023-10-16T12:24:01.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:01.169+0000 7f7d8a76b700 1 -- 172.21.15.16:0/2025606514 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f7d8404ec20 con 0x7f7d84160e60 2023-10-16T12:24:01.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:01.256+0000 7f7d80ff9700 1 -- 172.21.15.16:0/2025606514 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v65) v1 ==== 138+0+0 (secure 0 0 0) 0x7f7d7c0563b0 con 0x7f7d84160e60 2023-10-16T12:24:01.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:01.259+0000 7f7d8a76b700 1 -- 172.21.15.16:0/2025606514 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f7d6406b020 msgr2=0x7f7d6406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:01.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:01.259+0000 7f7d8a76b700 1 --2- 172.21.15.16:0/2025606514 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f7d6406b020 0x7f7d6406d4e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7d74006b20 tx=0x7f7d74008040 comp rx=0 tx=0).stop 2023-10-16T12:24:01.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:01.259+0000 7f7d8a76b700 1 -- 172.21.15.16:0/2025606514 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d84160e60 msgr2=0x7f7d84160070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:01.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:01.259+0000 7f7d8a76b700 1 --2- 172.21.15.16:0/2025606514 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d84160e60 0x7f7d84160070 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f7d8406c8d0 tx=0x7f7d7c004ad0 comp rx=0 tx=0).stop 2023-10-16T12:24:01.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:01.259+0000 7f7d8a76b700 1 -- 172.21.15.16:0/2025606514 shutdown_connections 2023-10-16T12:24:01.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:01.259+0000 7f7d8a76b700 1 --2- 172.21.15.16:0/2025606514 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f7d6406b020 0x7f7d6406d4e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:01.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:01.259+0000 7f7d8a76b700 1 --2- 172.21.15.16:0/2025606514 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d84160e60 0x7f7d84160070 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:01.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:01.259+0000 7f7d8a76b700 1 --2- 172.21.15.16:0/2025606514 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7d8406a7e0 0x7f7d840b0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:01.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:01.259+0000 7f7d8a76b700 1 --2- 172.21.15.16:0/2025606514 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7d8406d980 0x7f7d84156770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:01.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:01.259+0000 7f7d8a76b700 1 -- 172.21.15.16:0/2025606514 >> 172.21.15.16:0/2025606514 conn(0x7f7d8406f430 msgr2=0x7f7d8415ef90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:24:01.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:01.259+0000 7f7d8a76b700 1 -- 172.21.15.16:0/2025606514 shutdown_connections 2023-10-16T12:24:01.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:01.259+0000 7f7d8a76b700 1 -- 172.21.15.16:0/2025606514 wait complete. 2023-10-16T12:24:01.266 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' created 2023-10-16T12:24:01.278 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-16T12:24:01.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:01.408+0000 7f74ae8f2700 1 Processor -- start 2023-10-16T12:24:01.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:01.408+0000 7f74ae8f2700 1 -- start start 2023-10-16T12:24:01.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:01.408+0000 7f74ae8f2700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f74a815b2c0 0x7f74a8161340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:01.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:01.408+0000 7f74ae8f2700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f74a806a7e0 0x7f74a806ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:01.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:01.408+0000 7f74ae8f2700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74a815ecb0 con 0x7f74a806e570 2023-10-16T12:24:01.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:01.408+0000 7f74ae8f2700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74a806b190 con 0x7f74a806a7e0 2023-10-16T12:24:01.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:01.408+0000 7f74ae8f2700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74a8161880 con 0x7f74a815b2c0 2023-10-16T12:24:01.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:01.409+0000 7f74a77fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f74a806e570 0x7f74a815e5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:55184/0 (socket says 172.21.15.16:55184) 2023-10-16T12:24:01.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:01.409+0000 7f74a77fe700 1 -- 172.21.15.16:0/1061499199 learned_addr learned my addr 172.21.15.16:0/1061499199 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:24:01.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:01.409+0000 7f74ace8f700 1 --2- 172.21.15.16:0/1061499199 >> v2:172.21.15.16:3300/0 conn(0x7f74a806a7e0 0x7f74a806ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:01.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:01.409+0000 7f74a7fff700 1 --2- 172.21.15.16:0/1061499199 >> v2:172.21.15.49:3300/0 conn(0x7f74a815b2c0 0x7f74a8161340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:01.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:01.409+0000 7f74ace8f700 1 -- 172.21.15.16:0/1061499199 >> v2:172.21.15.49:3300/0 conn(0x7f74a815b2c0 msgr2=0x7f74a8161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:24:01.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:01.409+0000 7f74ace8f700 1 --2- 172.21.15.16:0/1061499199 >> v2:172.21.15.49:3300/0 conn(0x7f74a815b2c0 0x7f74a8161340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:01.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:01.409+0000 7f74ace8f700 1 -- 172.21.15.16:0/1061499199 >> v1:172.21.15.16:6789/0 conn(0x7f74a806e570 legacy=0x7f74a815e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:01.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:01.409+0000 7f74ace8f700 1 -- 172.21.15.16:0/1061499199 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74a815b6a0 con 0x7f74a806a7e0 2023-10-16T12:24:01.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:01.409+0000 7f74a6ffd700 1 -- 172.21.15.16:0/1061499199 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3596805085 0 0) 0x7f74a815ecb0 con 0x7f74a806e570 2023-10-16T12:24:01.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:01.409+0000 7f74ace8f700 1 --2- 172.21.15.16:0/1061499199 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74a806a7e0 0x7f74a806ac50 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f74a806b4a0 tx=0x7f74a000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3a08ef1959751 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:24:01.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:01.409+0000 7f74a6ffd700 1 -- 172.21.15.16:0/1061499199 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74a000e070 con 0x7f74a806a7e0 2023-10-16T12:24:01.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:01.409+0000 7f74a6ffd700 1 -- 172.21.15.16:0/1061499199 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74a000bbd0 con 0x7f74a806a7e0 2023-10-16T12:24:01.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:01.409+0000 7f74a6ffd700 1 -- 172.21.15.16:0/1061499199 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74a0012430 con 0x7f74a806a7e0 2023-10-16T12:24:01.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:01.410+0000 7f74ae8f2700 1 -- 172.21.15.16:0/1061499199 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74a806a7e0 msgr2=0x7f74a806ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:01.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:01.410+0000 7f74ae8f2700 1 --2- 172.21.15.16:0/1061499199 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74a806a7e0 0x7f74a806ac50 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f74a806b4a0 tx=0x7f74a000b330 comp rx=0 tx=0).stop 2023-10-16T12:24:01.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:01.410+0000 7f74ae8f2700 1 -- 172.21.15.16:0/1061499199 shutdown_connections 2023-10-16T12:24:01.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:01.410+0000 7f74ae8f2700 1 --2- 172.21.15.16:0/1061499199 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74a806a7e0 0x7f74a806ac50 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:01.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:01.410+0000 7f74ae8f2700 1 --2- 172.21.15.16:0/1061499199 >> v2:172.21.15.49:3300/0 conn(0x7f74a815b2c0 0x7f74a8161340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:01.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:01.410+0000 7f74ae8f2700 1 -- 172.21.15.16:0/1061499199 >> 172.21.15.16:0/1061499199 conn(0x7f74a80b1f40 msgr2=0x7f74a80b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:24:01.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:01.410+0000 7f74ae8f2700 1 -- 172.21.15.16:0/1061499199 shutdown_connections 2023-10-16T12:24:01.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:01.410+0000 7f74ae8f2700 1 -- 172.21.15.16:0/1061499199 wait complete. 2023-10-16T12:24:01.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:01.411+0000 7f74ae8f2700 1 Processor -- start 2023-10-16T12:24:01.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:01.411+0000 7f74ae8f2700 1 -- start start 2023-10-16T12:24:01.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:01.412+0000 7f74ae8f2700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74a806a7e0 0x7f74a80b2f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:01.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:01.412+0000 7f74ae8f2700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f74a806e570 0x7f74a8157870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:01.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:01.412+0000 7f74ae8f2700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f74a815b2c0 0x7f74a80c4be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:01.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:01.412+0000 7f74ae8f2700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74a80c8bb0 con 0x7f74a806a7e0 2023-10-16T12:24:01.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:01.412+0000 7f74ae8f2700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74a80b3450 con 0x7f74a806e570 2023-10-16T12:24:01.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:01.412+0000 7f74ae8f2700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74a80b3790 con 0x7f74a815b2c0 2023-10-16T12:24:01.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:01.412+0000 7f74a7fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74a806a7e0 0x7f74a80b2f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:01.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:01.412+0000 7f74a7fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74a806a7e0 0x7f74a80b2f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37682/0 (socket says 172.21.15.16:37682) 2023-10-16T12:24:01.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:01.412+0000 7f74a7fff700 1 -- 172.21.15.16:0/2835859401 learned_addr learned my addr 172.21.15.16:0/2835859401 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:24:01.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:01.412+0000 7f74a77fe700 1 --2- 172.21.15.16:0/2835859401 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f74a806e570 0x7f74a8157870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:01.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:01.412+0000 7f74ace8f700 1 --2- 172.21.15.16:0/2835859401 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f74a815b2c0 0x7f74a80c4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:01.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:01.412+0000 7f74a7fff700 1 -- 172.21.15.16:0/2835859401 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f74a815b2c0 msgr2=0x7f74a80c4be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:01.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:01.412+0000 7f74a7fff700 1 --2- 172.21.15.16:0/2835859401 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f74a815b2c0 0x7f74a80c4be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:01.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:01.413+0000 7f74a7fff700 1 -- 172.21.15.16:0/2835859401 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f74a806e570 msgr2=0x7f74a8157870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:01.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:01.413+0000 7f74a7fff700 1 --2- 172.21.15.16:0/2835859401 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f74a806e570 0x7f74a8157870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:01.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:01.413+0000 7f74a7fff700 1 -- 172.21.15.16:0/2835859401 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74a0009a70 con 0x7f74a806a7e0 2023-10-16T12:24:01.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:01.413+0000 7f74a7fff700 1 --2- 172.21.15.16:0/2835859401 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74a806a7e0 0x7f74a80b2f10 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f749800d800 tx=0x7f749800db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:24:01.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:01.413+0000 7f74a4ff9700 1 -- 172.21.15.16:0/2835859401 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74980125f0 con 0x7f74a806a7e0 2023-10-16T12:24:01.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:01.413+0000 7f74a4ff9700 1 -- 172.21.15.16:0/2835859401 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7498012750 con 0x7f74a806a7e0 2023-10-16T12:24:01.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:01.413+0000 7f74ae8f2700 1 -- 172.21.15.16:0/2835859401 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74a80c8560 con 0x7f74a806a7e0 2023-10-16T12:24:01.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:01.413+0000 7f74a4ff9700 1 -- 172.21.15.16:0/2835859401 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7498012b00 con 0x7f74a806a7e0 2023-10-16T12:24:01.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:01.413+0000 7f74ae8f2700 1 -- 172.21.15.16:0/2835859401 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74a80b3cc0 con 0x7f74a806a7e0 2023-10-16T12:24:01.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:01.414+0000 7f74ae8f2700 1 -- 172.21.15.16:0/2835859401 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74a80c8e20 con 0x7f74a806a7e0 2023-10-16T12:24:01.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:01.415+0000 7f74a4ff9700 1 -- 172.21.15.16:0/2835859401 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7498012c60 con 0x7f74a806a7e0 2023-10-16T12:24:01.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:01.415+0000 7f74a4ff9700 1 --2- 172.21.15.16:0/2835859401 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f749406b020 0x7f749406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:01.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:01.415+0000 7f74a4ff9700 1 -- 172.21.15.16:0/2835859401 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5547+0+0 (secure 0 0 0) 0x7f749808a280 con 0x7f74a806a7e0 2023-10-16T12:24:01.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:01.415+0000 7f74a77fe700 1 --2- 172.21.15.16:0/2835859401 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f749406b020 0x7f749406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:01.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:01.416+0000 7f74a77fe700 1 --2- 172.21.15.16:0/2835859401 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f749406b020 0x7f749406d4e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f74a80ca650 tx=0x7f749c007e70 comp rx=0 tx=0).ready entity=mgr.5874 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:24:01.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:01.420+0000 7f74a4ff9700 1 -- 172.21.15.16:0/2835859401 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f74980552a0 con 0x7f74a806a7e0 2023-10-16T12:24:01.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:01.621+0000 7f74ae8f2700 1 -- 172.21.15.16:0/2835859401 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f74a804ebb0 con 0x7f74a806a7e0 2023-10-16T12:24:02.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.263+0000 7f74a4ff9700 1 -- 172.21.15.16:0/2835859401 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v66) v1 ==== 170+0+0 (secure 0 0 0) 0x7f74980588c0 con 0x7f74a806a7e0 2023-10-16T12:24:02.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.269+0000 7f74ae8f2700 1 -- 172.21.15.16:0/2835859401 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f749406b020 msgr2=0x7f749406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:02.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.269+0000 7f74ae8f2700 1 --2- 172.21.15.16:0/2835859401 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f749406b020 0x7f749406d4e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f74a80ca650 tx=0x7f749c007e70 comp rx=0 tx=0).stop 2023-10-16T12:24:02.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.269+0000 7f74ae8f2700 1 -- 172.21.15.16:0/2835859401 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74a806a7e0 msgr2=0x7f74a80b2f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:02.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.269+0000 7f74ae8f2700 1 --2- 172.21.15.16:0/2835859401 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74a806a7e0 0x7f74a80b2f10 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f749800d800 tx=0x7f749800db10 comp rx=0 tx=0).stop 2023-10-16T12:24:02.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.271+0000 7f74ae8f2700 1 -- 172.21.15.16:0/2835859401 shutdown_connections 2023-10-16T12:24:02.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.271+0000 7f74ae8f2700 1 --2- 172.21.15.16:0/2835859401 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f749406b020 0x7f749406d4e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:02.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.271+0000 7f74ae8f2700 1 --2- 172.21.15.16:0/2835859401 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f74a815b2c0 0x7f74a80c4be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:02.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.271+0000 7f74ae8f2700 1 --2- 172.21.15.16:0/2835859401 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74a806a7e0 0x7f74a80b2f10 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:02.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.271+0000 7f74ae8f2700 1 --2- 172.21.15.16:0/2835859401 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f74a806e570 0x7f74a8157870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:02.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.271+0000 7f74ae8f2700 1 -- 172.21.15.16:0/2835859401 >> 172.21.15.16:0/2835859401 conn(0x7f74a80b1f40 msgr2=0x7f74a815c440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:24:02.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.274+0000 7f74ae8f2700 1 -- 172.21.15.16:0/2835859401 shutdown_connections 2023-10-16T12:24:02.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.275+0000 7f74ae8f2700 1 -- 172.21.15.16:0/2835859401 wait complete. 2023-10-16T12:24:02.280 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' created 2023-10-16T12:24:02.300 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-16T12:24:02.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.431+0000 7f871dda2700 1 Processor -- start 2023-10-16T12:24:02.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.431+0000 7f871dda2700 1 -- start start 2023-10-16T12:24:02.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.432+0000 7f871dda2700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f87180cd7a0 0x7f87180cddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:02.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.432+0000 7f871dda2700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87180ce740 con 0x7f87180ce310 2023-10-16T12:24:02.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.432+0000 7f871dda2700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8718161320 con 0x7f871815b490 2023-10-16T12:24:02.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.432+0000 7f871dda2700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8718161fa0 con 0x7f87180cd7a0 2023-10-16T12:24:02.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.432+0000 7f871d5a1700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f87180ce310 0x7f871815dc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:55198/0 (socket says 172.21.15.16:55198) 2023-10-16T12:24:02.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.432+0000 7f871d5a1700 1 -- 172.21.15.16:0/3514882042 learned_addr learned my addr 172.21.15.16:0/3514882042 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:24:02.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.433+0000 7f87177fe700 1 -- 172.21.15.16:0/3514882042 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4188504927 0 0) 0x7f87180ce740 con 0x7f87180ce310 2023-10-16T12:24:02.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.433+0000 7f8717fff700 1 --2- 172.21.15.16:0/3514882042 >> v2:172.21.15.49:3300/0 conn(0x7f87180cd7a0 0x7f87180cddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:02.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.433+0000 7f87177fe700 1 -- 172.21.15.16:0/3514882042 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87080035b0 con 0x7f87180ce310 2023-10-16T12:24:02.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.433+0000 7f87177fe700 1 -- 172.21.15.16:0/3514882042 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2385277267 0 0) 0x7f87080035b0 con 0x7f87180ce310 2023-10-16T12:24:02.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.433+0000 7f87177fe700 1 -- 172.21.15.16:0/3514882042 >> v2:172.21.15.49:3300/0 conn(0x7f87180cd7a0 msgr2=0x7f87180cddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:24:02.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.433+0000 7f87177fe700 1 --2- 172.21.15.16:0/3514882042 >> v2:172.21.15.49:3300/0 conn(0x7f87180cd7a0 0x7f87180cddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:02.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.433+0000 7f87177fe700 1 -- 172.21.15.16:0/3514882042 >> v1:172.21.15.49:6789/0 conn(0x7f871815b490 legacy=0x7f871806b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:02.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.433+0000 7f87177fe700 1 -- 172.21.15.16:0/3514882042 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8718161600 con 0x7f87180ce310 2023-10-16T12:24:02.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.433+0000 7f87177fe700 1 -- 172.21.15.16:0/3514882042 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f87100019f0 con 0x7f87180ce310 2023-10-16T12:24:02.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.433+0000 7f87177fe700 1 -- 172.21.15.16:0/3514882042 >> v1:172.21.15.16:6789/0 conn(0x7f87180ce310 legacy=0x7f871815dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:02.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.433+0000 7f87177fe700 1 --2- 172.21.15.16:0/3514882042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8708003a00 0x7f8708003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:02.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.433+0000 7f87177fe700 1 --2- 172.21.15.16:0/3514882042 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8708004850 0x7f8708004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:02.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.433+0000 7f87177fe700 1 --2- 172.21.15.16:0/3514882042 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8708005210 0x7f870800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:02.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.433+0000 7f87177fe700 1 -- 172.21.15.16:0/3514882042 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8718161fa0 con 0x7f8708003a00 2023-10-16T12:24:02.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.433+0000 7f87177fe700 1 -- 172.21.15.16:0/3514882042 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f870800c1b0 con 0x7f8708004850 2023-10-16T12:24:02.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.433+0000 7f87177fe700 1 -- 172.21.15.16:0/3514882042 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f870800c370 con 0x7f8708005210 2023-10-16T12:24:02.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.433+0000 7f871cda0700 1 --2- 172.21.15.16:0/3514882042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8708003a00 0x7f8708003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:02.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.433+0000 7f8717fff700 1 --2- 172.21.15.16:0/3514882042 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8708004850 0x7f8708004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:02.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.433+0000 7f871d5a1700 1 --2- 172.21.15.16:0/3514882042 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8708005210 0x7f870800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:02.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.433+0000 7f871cda0700 1 -- 172.21.15.16:0/3514882042 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8708005210 msgr2=0x7f870800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:24:02.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.433+0000 7f871cda0700 1 --2- 172.21.15.16:0/3514882042 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8708005210 0x7f870800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:02.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.433+0000 7f871cda0700 1 -- 172.21.15.16:0/3514882042 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8708004850 msgr2=0x7f8708004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:24:02.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.433+0000 7f871cda0700 1 --2- 172.21.15.16:0/3514882042 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8708004850 0x7f8708004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:02.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.433+0000 7f871cda0700 1 -- 172.21.15.16:0/3514882042 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87080035b0 con 0x7f8708003a00 2023-10-16T12:24:02.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.433+0000 7f871d5a1700 1 --2- 172.21.15.16:0/3514882042 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8708005210 0x7f870800ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:24:02.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.433+0000 7f871d5a1700 1 -- 172.21.15.16:0/3514882042 reap_dead start 2023-10-16T12:24:02.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.434+0000 7f871cda0700 1 --2- 172.21.15.16:0/3514882042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8708003a00 0x7f8708003e20 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f870c0017c0 tx=0x7f8708005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf79121fed49028e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:24:02.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.434+0000 7f87177fe700 1 -- 172.21.15.16:0/3514882042 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f870c0052d0 con 0x7f8708003a00 2023-10-16T12:24:02.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.434+0000 7f87177fe700 1 -- 172.21.15.16:0/3514882042 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f870c005430 con 0x7f8708003a00 2023-10-16T12:24:02.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.434+0000 7f87177fe700 1 -- 172.21.15.16:0/3514882042 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f870c0038c0 con 0x7f8708003a00 2023-10-16T12:24:02.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.434+0000 7f871dda2700 1 -- 172.21.15.16:0/3514882042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8708003a00 msgr2=0x7f8708003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:02.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.434+0000 7f871dda2700 1 --2- 172.21.15.16:0/3514882042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8708003a00 0x7f8708003e20 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f870c0017c0 tx=0x7f8708005880 comp rx=0 tx=0).stop 2023-10-16T12:24:02.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.434+0000 7f871dda2700 1 -- 172.21.15.16:0/3514882042 shutdown_connections 2023-10-16T12:24:02.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.434+0000 7f871dda2700 1 --2- 172.21.15.16:0/3514882042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8708003a00 0x7f8708003e20 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:02.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.434+0000 7f871dda2700 1 -- 172.21.15.16:0/3514882042 >> 172.21.15.16:0/3514882042 conn(0x7f87180ccc80 msgr2=0x7f87180cd090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:24:02.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.434+0000 7f871dda2700 1 -- 172.21.15.16:0/3514882042 shutdown_connections 2023-10-16T12:24:02.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.434+0000 7f871dda2700 1 -- 172.21.15.16:0/3514882042 wait complete. 2023-10-16T12:24:02.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.436+0000 7f871dda2700 1 Processor -- start 2023-10-16T12:24:02.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.436+0000 7f871dda2700 1 -- start start 2023-10-16T12:24:02.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.436+0000 7f871dda2700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8708003a00 0x7f8718148180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:02.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.436+0000 7f871dda2700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87180c5e00 0x7f87181486c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:02.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.437+0000 7f871dda2700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f87181f6e10 0x7f87181f7290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:02.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.437+0000 7f871dda2700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8718148c00 con 0x7f87180c5e00 2023-10-16T12:24:02.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.437+0000 7f871dda2700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87180b3450 con 0x7f8708003a00 2023-10-16T12:24:02.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.437+0000 7f871dda2700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87180b2e20 con 0x7f87181f6e10 2023-10-16T12:24:02.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.437+0000 7f8717fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87180c5e00 0x7f87181486c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:02.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.437+0000 7f8717fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87180c5e00 0x7f87181486c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37704/0 (socket says 172.21.15.16:37704) 2023-10-16T12:24:02.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.437+0000 7f8717fff700 1 -- 172.21.15.16:0/402843299 learned_addr learned my addr 172.21.15.16:0/402843299 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:24:02.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.437+0000 7f871cda0700 1 --2- 172.21.15.16:0/402843299 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8708003a00 0x7f8718148180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:02.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.437+0000 7f8717fff700 1 -- 172.21.15.16:0/402843299 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f87181f6e10 msgr2=0x7f87181f7290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:02.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.437+0000 7f8717fff700 1 --2- 172.21.15.16:0/402843299 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f87181f6e10 0x7f87181f7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:02.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.437+0000 7f8717fff700 1 -- 172.21.15.16:0/402843299 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8708003a00 msgr2=0x7f8718148180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:02.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.437+0000 7f8717fff700 1 --2- 172.21.15.16:0/402843299 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8708003a00 0x7f8718148180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:02.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.437+0000 7f8717fff700 1 -- 172.21.15.16:0/402843299 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87180c6370 con 0x7f87180c5e00 2023-10-16T12:24:02.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.437+0000 7f8717fff700 1 --2- 172.21.15.16:0/402843299 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87180c5e00 0x7f87181486c0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f8704011a50 tx=0x7f8704011e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:24:02.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.437+0000 7f87157fa700 1 -- 172.21.15.16:0/402843299 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f870400fa00 con 0x7f87180c5e00 2023-10-16T12:24:02.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.437+0000 7f87157fa700 1 -- 172.21.15.16:0/402843299 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f870400fb60 con 0x7f87180c5e00 2023-10-16T12:24:02.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.437+0000 7f871dda2700 1 -- 172.21.15.16:0/402843299 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87180c6650 con 0x7f87180c5e00 2023-10-16T12:24:02.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.437+0000 7f87157fa700 1 -- 172.21.15.16:0/402843299 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8704014440 con 0x7f87180c5e00 2023-10-16T12:24:02.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.437+0000 7f871dda2700 1 -- 172.21.15.16:0/402843299 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8718152960 con 0x7f87180c5e00 2023-10-16T12:24:02.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.438+0000 7f871dda2700 1 -- 172.21.15.16:0/402843299 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f871806fd40 con 0x7f87180c5e00 2023-10-16T12:24:02.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.439+0000 7f87157fa700 1 -- 172.21.15.16:0/402843299 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f870401d400 con 0x7f87180c5e00 2023-10-16T12:24:02.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.439+0000 7f87157fa700 1 --2- 172.21.15.16:0/402843299 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f86fc06b050 0x7f86fc06d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:02.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.439+0000 7f87157fa700 1 -- 172.21.15.16:0/402843299 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5929+0+0 (secure 0 0 0) 0x7f870408a520 con 0x7f87180c5e00 2023-10-16T12:24:02.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.439+0000 7f871cda0700 1 --2- 172.21.15.16:0/402843299 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f86fc06b050 0x7f86fc06d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:02.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.440+0000 7f871cda0700 1 --2- 172.21.15.16:0/402843299 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f86fc06b050 0x7f86fc06d510 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f870c005040 tx=0x7f870c009030 comp rx=0 tx=0).ready entity=mgr.5874 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:24:02.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.444+0000 7f87157fa700 1 -- 172.21.15.16:0/402843299 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8704055450 con 0x7f87180c5e00 2023-10-16T12:24:02.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:02.693+0000 7f871dda2700 1 -- 172.21.15.16:0/402843299 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f8718161b90 con 0x7f87180c5e00 2023-10-16T12:24:03.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.277+0000 7f87157fa700 1 -- 172.21.15.16:0/402843299 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 7 and data pool 8 v23) v1 ==== 172+0+0 (secure 0 0 0) 0x7f8704058a70 con 0x7f87180c5e00 2023-10-16T12:24:03.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.284+0000 7f871dda2700 1 -- 172.21.15.16:0/402843299 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f86fc06b050 msgr2=0x7f86fc06d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:03.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.284+0000 7f871dda2700 1 --2- 172.21.15.16:0/402843299 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f86fc06b050 0x7f86fc06d510 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f870c005040 tx=0x7f870c009030 comp rx=0 tx=0).stop 2023-10-16T12:24:03.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.284+0000 7f871dda2700 1 -- 172.21.15.16:0/402843299 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87180c5e00 msgr2=0x7f87181486c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:03.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.284+0000 7f871dda2700 1 --2- 172.21.15.16:0/402843299 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87180c5e00 0x7f87181486c0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f8704011a50 tx=0x7f8704011e10 comp rx=0 tx=0).stop 2023-10-16T12:24:03.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.285+0000 7f871dda2700 1 -- 172.21.15.16:0/402843299 shutdown_connections 2023-10-16T12:24:03.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.285+0000 7f871dda2700 1 --2- 172.21.15.16:0/402843299 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f86fc06b050 0x7f86fc06d510 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:03.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.285+0000 7f871dda2700 1 --2- 172.21.15.16:0/402843299 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f87181f6e10 0x7f87181f7290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:03.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.285+0000 7f871dda2700 1 --2- 172.21.15.16:0/402843299 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8708003a00 0x7f8718148180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:03.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.285+0000 7f871dda2700 1 --2- 172.21.15.16:0/402843299 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87180c5e00 0x7f87181486c0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:03.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.285+0000 7f871dda2700 1 -- 172.21.15.16:0/402843299 >> 172.21.15.16:0/402843299 conn(0x7f87180ccc80 msgr2=0x7f87180ca830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:24:03.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.291+0000 7f871dda2700 1 -- 172.21.15.16:0/402843299 shutdown_connections 2023-10-16T12:24:03.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.292+0000 7f871dda2700 1 -- 172.21.15.16:0/402843299 wait complete. 2023-10-16T12:24:03.299 INFO:teuthology.orchestra.run.smithi016.stderr:new fs with metadata pool 7 and data pool 8 2023-10-16T12:24:03.316 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:24:03.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.444+0000 7fa016510700 1 Processor -- start 2023-10-16T12:24:03.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.444+0000 7fa016510700 1 -- start start 2023-10-16T12:24:03.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.445+0000 7fa016510700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa01015d230 0x7fa0100b5a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:03.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.445+0000 7fa016510700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa01015bc80 con 0x7fa0100cf880 2023-10-16T12:24:03.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.445+0000 7fa016510700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa01015b070 con 0x7fa0100cf4a0 2023-10-16T12:24:03.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.445+0000 7fa016510700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa01015b350 con 0x7fa01015d230 2023-10-16T12:24:03.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.445+0000 7fa00f7fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa0100cf880 0x7fa0100b52f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:55216/0 (socket says 172.21.15.16:55216) 2023-10-16T12:24:03.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.445+0000 7fa00f7fe700 1 -- 172.21.15.16:0/2142992203 learned_addr learned my addr 172.21.15.16:0/2142992203 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:24:03.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.446+0000 7fa014aad700 1 --2- 172.21.15.16:0/2142992203 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa01015d230 0x7fa0100b5a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:03.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.446+0000 7fa00effd700 1 -- 172.21.15.16:0/2142992203 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3393141378 0 0) 0x7fa01015bc80 con 0x7fa0100cf880 2023-10-16T12:24:03.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.446+0000 7fa00effd700 1 -- 172.21.15.16:0/2142992203 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ff80035b0 con 0x7fa0100cf880 2023-10-16T12:24:03.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.446+0000 7fa00effd700 1 -- 172.21.15.16:0/2142992203 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3008169587 0 0) 0x7f9ff80035b0 con 0x7fa0100cf880 2023-10-16T12:24:03.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.446+0000 7fa00effd700 1 -- 172.21.15.16:0/2142992203 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa01015d230 msgr2=0x7fa0100b5a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:24:03.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.446+0000 7fa00effd700 1 --2- 172.21.15.16:0/2142992203 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa01015d230 0x7fa0100b5a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:03.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.446+0000 7fa00effd700 1 -- 172.21.15.16:0/2142992203 >> v1:172.21.15.49:6789/0 conn(0x7fa0100cf4a0 legacy=0x7fa0100726e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:03.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.446+0000 7fa00effd700 1 -- 172.21.15.16:0/2142992203 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa01015b630 con 0x7fa0100cf880 2023-10-16T12:24:03.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.446+0000 7fa00effd700 1 -- 172.21.15.16:0/2142992203 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7fa004001670 con 0x7fa0100cf880 2023-10-16T12:24:03.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.446+0000 7fa00effd700 1 -- 172.21.15.16:0/2142992203 >> v1:172.21.15.16:6789/0 conn(0x7fa0100cf880 legacy=0x7fa0100b52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:03.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.446+0000 7fa00effd700 1 --2- 172.21.15.16:0/2142992203 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa01015d230 0x7f9ff8003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:03.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.446+0000 7fa00effd700 1 --2- 172.21.15.16:0/2142992203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ff8003fb0 0x7f9ff8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:03.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.446+0000 7fa00effd700 1 --2- 172.21.15.16:0/2142992203 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9ff8004d90 0x7f9ff8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:03.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.446+0000 7fa00effd700 1 -- 172.21.15.16:0/2142992203 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa01015b350 con 0x7f9ff8003fb0 2023-10-16T12:24:03.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.446+0000 7fa00effd700 1 -- 172.21.15.16:0/2142992203 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff800baf0 con 0x7f9ff8004d90 2023-10-16T12:24:03.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.446+0000 7fa00effd700 1 -- 172.21.15.16:0/2142992203 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff800bcb0 con 0x7fa01015d230 2023-10-16T12:24:03.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.446+0000 7fa00f7fe700 1 --2- 172.21.15.16:0/2142992203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ff8003fb0 0x7f9ff8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:03.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.446+0000 7fa00ffff700 1 --2- 172.21.15.16:0/2142992203 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa01015d230 0x7f9ff8003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:03.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.446+0000 7fa014aad700 1 --2- 172.21.15.16:0/2142992203 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9ff8004d90 0x7f9ff8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:03.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.446+0000 7fa00f7fe700 1 -- 172.21.15.16:0/2142992203 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa01015d230 msgr2=0x7f9ff8003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:24:03.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.446+0000 7fa00f7fe700 1 --2- 172.21.15.16:0/2142992203 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa01015d230 0x7f9ff8003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:03.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.446+0000 7fa00f7fe700 1 -- 172.21.15.16:0/2142992203 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9ff8004d90 msgr2=0x7f9ff8005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:24:03.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.446+0000 7fa00f7fe700 1 --2- 172.21.15.16:0/2142992203 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9ff8004d90 0x7f9ff8005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:03.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.446+0000 7fa00f7fe700 1 -- 172.21.15.16:0/2142992203 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ff80035b0 con 0x7f9ff8003fb0 2023-10-16T12:24:03.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.447+0000 7fa00f7fe700 1 --2- 172.21.15.16:0/2142992203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ff8003fb0 0x7f9ff8004850 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fa00400c240 tx=0x7f9ff8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59e86197671e1f1d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:24:03.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.447+0000 7fa00effd700 1 -- 172.21.15.16:0/2142992203 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa004007c10 con 0x7f9ff8003fb0 2023-10-16T12:24:03.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.447+0000 7fa00effd700 1 -- 172.21.15.16:0/2142992203 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa00400ce00 con 0x7f9ff8003fb0 2023-10-16T12:24:03.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.447+0000 7fa00effd700 1 -- 172.21.15.16:0/2142992203 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa004005ba0 con 0x7f9ff8003fb0 2023-10-16T12:24:03.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.447+0000 7fa016510700 1 -- 172.21.15.16:0/2142992203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ff8003fb0 msgr2=0x7f9ff8004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:03.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.447+0000 7fa016510700 1 --2- 172.21.15.16:0/2142992203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ff8003fb0 0x7f9ff8004850 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fa00400c240 tx=0x7f9ff8005880 comp rx=0 tx=0).stop 2023-10-16T12:24:03.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.447+0000 7fa014aad700 1 -- 172.21.15.16:0/2142992203 reap_dead start 2023-10-16T12:24:03.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.447+0000 7fa016510700 1 -- 172.21.15.16:0/2142992203 shutdown_connections 2023-10-16T12:24:03.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.447+0000 7fa016510700 1 -- 172.21.15.16:0/2142992203 >> 172.21.15.16:0/2142992203 conn(0x7fa0100b1fd0 msgr2=0x7fa0100b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:24:03.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.447+0000 7fa016510700 1 -- 172.21.15.16:0/2142992203 shutdown_connections 2023-10-16T12:24:03.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.447+0000 7fa016510700 1 -- 172.21.15.16:0/2142992203 wait complete. 2023-10-16T12:24:03.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.448+0000 7fa016510700 1 Processor -- start 2023-10-16T12:24:03.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.449+0000 7fa016510700 1 -- start start 2023-10-16T12:24:03.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.449+0000 7fa016510700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ff8003fb0 0x7fa010158810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:03.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.449+0000 7fa016510700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa0100c5290 0x7fa0100c9520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:03.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.449+0000 7fa016510700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa0100c5710 0x7fa0100c9a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:03.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.449+0000 7fa016510700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa010158d50 con 0x7f9ff8003fb0 2023-10-16T12:24:03.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.449+0000 7fa016510700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0100c5ca0 con 0x7fa0100c5710 2023-10-16T12:24:03.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.449+0000 7fa016510700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0100c5f80 con 0x7fa0100c5290 2023-10-16T12:24:03.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.450+0000 7fa00ffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ff8003fb0 0x7fa010158810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:03.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.450+0000 7fa00ffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ff8003fb0 0x7fa010158810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37722/0 (socket says 172.21.15.16:37722) 2023-10-16T12:24:03.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.450+0000 7fa00ffff700 1 -- 172.21.15.16:0/1270290509 learned_addr learned my addr 172.21.15.16:0/1270290509 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:24:03.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.450+0000 7fa00f7fe700 1 --2- 172.21.15.16:0/1270290509 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa0100c5290 0x7fa0100c9520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:03.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.450+0000 7fa014aad700 1 --2- 172.21.15.16:0/1270290509 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa0100c5710 0x7fa0100c9a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:03.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.450+0000 7fa00ffff700 1 -- 172.21.15.16:0/1270290509 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa0100c5290 msgr2=0x7fa0100c9520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:03.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.450+0000 7fa00ffff700 1 --2- 172.21.15.16:0/1270290509 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa0100c5290 0x7fa0100c9520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:03.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.450+0000 7fa00ffff700 1 -- 172.21.15.16:0/1270290509 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa0100c5710 msgr2=0x7fa0100c9a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:03.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.450+0000 7fa00ffff700 1 --2- 172.21.15.16:0/1270290509 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa0100c5710 0x7fa0100c9a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:03.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.450+0000 7fa00ffff700 1 -- 172.21.15.16:0/1270290509 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0100c6260 con 0x7f9ff8003fb0 2023-10-16T12:24:03.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.450+0000 7fa00ffff700 1 --2- 172.21.15.16:0/1270290509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ff8003fb0 0x7fa010158810 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fa00000f7f0 tx=0x7fa00000fbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:24:03.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.450+0000 7fa00cff9700 1 -- 172.21.15.16:0/1270290509 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0000052e0 con 0x7f9ff8003fb0 2023-10-16T12:24:03.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.450+0000 7fa00cff9700 1 -- 172.21.15.16:0/1270290509 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa000005440 con 0x7f9ff8003fb0 2023-10-16T12:24:03.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.450+0000 7fa016510700 1 -- 172.21.15.16:0/1270290509 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0100c6540 con 0x7f9ff8003fb0 2023-10-16T12:24:03.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.450+0000 7fa00cff9700 1 -- 172.21.15.16:0/1270290509 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0000038c0 con 0x7f9ff8003fb0 2023-10-16T12:24:03.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.450+0000 7fa016510700 1 -- 172.21.15.16:0/1270290509 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0100c6d40 con 0x7f9ff8003fb0 2023-10-16T12:24:03.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.451+0000 7fa016510700 1 -- 172.21.15.16:0/1270290509 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0100c6b90 con 0x7f9ff8003fb0 2023-10-16T12:24:03.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.452+0000 7fa00cff9700 1 -- 172.21.15.16:0/1270290509 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa000003a20 con 0x7f9ff8003fb0 2023-10-16T12:24:03.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.452+0000 7fa00cff9700 1 --2- 172.21.15.16:0/1270290509 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f9ffc06b020 0x7f9ffc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:03.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.452+0000 7fa00cff9700 1 -- 172.21.15.16:0/1270290509 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fa000089710 con 0x7f9ff8003fb0 2023-10-16T12:24:03.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.452+0000 7fa00f7fe700 1 --2- 172.21.15.16:0/1270290509 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f9ffc06b020 0x7f9ffc06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:03.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.453+0000 7fa00f7fe700 1 --2- 172.21.15.16:0/1270290509 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f9ffc06b020 0x7f9ffc06d4e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa004000c00 tx=0x7fa004007000 comp rx=0 tx=0).ready entity=mgr.5874 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:24:03.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.457+0000 7fa00cff9700 1 -- 172.21.15.16:0/1270290509 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa000054570 con 0x7f9ff8003fb0 2023-10-16T12:24:03.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.661+0000 7fa016510700 1 -- 172.21.15.16:0/1270290509 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa0100c6f00 con 0x7f9ff8003fb0 2023-10-16T12:24:03.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.662+0000 7fa00cff9700 1 -- 172.21.15.16:0/1270290509 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v67) v1 ==== 74+0+16679 (secure 0 0 0) 0x7fa000057b90 con 0x7f9ff8003fb0 2023-10-16T12:24:03.665 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:24:03.665 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":67,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:24:02.694894+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:24:01.200494+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-10-16T12:24:01.652885+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:24:03.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.665+0000 7fa016510700 1 -- 172.21.15.16:0/1270290509 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f9ffc06b020 msgr2=0x7f9ffc06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:03.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.665+0000 7fa016510700 1 --2- 172.21.15.16:0/1270290509 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f9ffc06b020 0x7f9ffc06d4e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa004000c00 tx=0x7fa004007000 comp rx=0 tx=0).stop 2023-10-16T12:24:03.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.665+0000 7fa016510700 1 -- 172.21.15.16:0/1270290509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ff8003fb0 msgr2=0x7fa010158810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:03.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.665+0000 7fa016510700 1 --2- 172.21.15.16:0/1270290509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ff8003fb0 0x7fa010158810 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fa00000f7f0 tx=0x7fa00000fbb0 comp rx=0 tx=0).stop 2023-10-16T12:24:03.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.665+0000 7fa016510700 1 -- 172.21.15.16:0/1270290509 shutdown_connections 2023-10-16T12:24:03.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.665+0000 7fa016510700 1 --2- 172.21.15.16:0/1270290509 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f9ffc06b020 0x7f9ffc06d4e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:03.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.665+0000 7fa016510700 1 --2- 172.21.15.16:0/1270290509 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa0100c5710 0x7fa0100c9a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:03.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.665+0000 7fa016510700 1 --2- 172.21.15.16:0/1270290509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ff8003fb0 0x7fa010158810 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:03.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.665+0000 7fa016510700 1 --2- 172.21.15.16:0/1270290509 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa0100c5290 0x7fa0100c9520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:03.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.665+0000 7fa016510700 1 -- 172.21.15.16:0/1270290509 >> 172.21.15.16:0/1270290509 conn(0x7fa0100b1fd0 msgr2=0x7fa0100cd520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:24:03.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.665+0000 7fa016510700 1 -- 172.21.15.16:0/1270290509 shutdown_connections 2023-10-16T12:24:03.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.665+0000 7fa016510700 1 -- 172.21.15.16:0/1270290509 wait complete. 2023-10-16T12:24:03.679 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:24:03.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.802+0000 7f4fa15c3700 1 Processor -- start 2023-10-16T12:24:03.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.802+0000 7f4fa15c3700 1 -- start start 2023-10-16T12:24:03.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.803+0000 7f4fa15c3700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4f9c159c90 0x7f4f9c15d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:03.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.803+0000 7f4fa15c3700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f9c06b1f0 con 0x7f4f9c0cb190 2023-10-16T12:24:03.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.803+0000 7f4fa15c3700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f9c06b4b0 con 0x7f4f9c15a070 2023-10-16T12:24:03.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.803+0000 7f4fa15c3700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f9c0af340 con 0x7f4f9c159c90 2023-10-16T12:24:03.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.803+0000 7f4f9b7fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f4f9c0cb190 0x7f4f9c0aec30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:55240/0 (socket says 172.21.15.16:55240) 2023-10-16T12:24:03.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.803+0000 7f4f9b7fe700 1 -- 172.21.15.16:0/4188236369 learned_addr learned my addr 172.21.15.16:0/4188236369 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:24:03.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.803+0000 7f4f9affd700 1 --2- 172.21.15.16:0/4188236369 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4f9c159c90 0x7f4f9c15d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:03.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.804+0000 7f4f99ffb700 1 -- 172.21.15.16:0/4188236369 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3839064183 0 0) 0x7f4f9c06b1f0 con 0x7f4f9c0cb190 2023-10-16T12:24:03.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.804+0000 7f4f99ffb700 1 -- 172.21.15.16:0/4188236369 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f840046c0 con 0x7f4f9c0cb190 2023-10-16T12:24:03.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.804+0000 7f4f99ffb700 1 -- 172.21.15.16:0/4188236369 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2752351233 0 0) 0x7f4f9c06b4b0 con 0x7f4f9c15a070 2023-10-16T12:24:03.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.804+0000 7f4f99ffb700 1 -- 172.21.15.16:0/4188236369 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f9c06b1f0 con 0x7f4f9c15a070 2023-10-16T12:24:03.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.804+0000 7f4f9affd700 1 -- 172.21.15.16:0/4188236369 >> v1:172.21.15.49:6789/0 conn(0x7f4f9c15a070 legacy=0x7f4f9c0ae520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:03.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.804+0000 7f4f9affd700 1 -- 172.21.15.16:0/4188236369 >> v1:172.21.15.16:6789/0 conn(0x7f4f9c0cb190 legacy=0x7f4f9c0aec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:03.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.804+0000 7f4f9affd700 1 -- 172.21.15.16:0/4188236369 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f9c06b790 con 0x7f4f9c159c90 2023-10-16T12:24:03.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.804+0000 7f4f9affd700 1 --2- 172.21.15.16:0/4188236369 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4f9c159c90 0x7f4f9c15d9a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4f8c009870 tx=0x7f4f8c009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c665f9ca88503008 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:24:03.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.804+0000 7f4f99ffb700 1 -- 172.21.15.16:0/4188236369 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f8c011830 con 0x7f4f9c159c90 2023-10-16T12:24:03.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.804+0000 7f4f99ffb700 1 -- 172.21.15.16:0/4188236369 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f8c011990 con 0x7f4f9c159c90 2023-10-16T12:24:03.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.804+0000 7f4f99ffb700 1 -- 172.21.15.16:0/4188236369 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f8c011d20 con 0x7f4f9c159c90 2023-10-16T12:24:03.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.805+0000 7f4fa15c3700 1 -- 172.21.15.16:0/4188236369 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4f9c159c90 msgr2=0x7f4f9c15d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:03.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.805+0000 7f4fa15c3700 1 --2- 172.21.15.16:0/4188236369 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4f9c159c90 0x7f4f9c15d9a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4f8c009870 tx=0x7f4f8c009b80 comp rx=0 tx=0).stop 2023-10-16T12:24:03.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.805+0000 7f4fa15c3700 1 -- 172.21.15.16:0/4188236369 shutdown_connections 2023-10-16T12:24:03.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.805+0000 7f4fa15c3700 1 --2- 172.21.15.16:0/4188236369 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4f9c159c90 0x7f4f9c15d9a0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:03.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.805+0000 7f4fa15c3700 1 -- 172.21.15.16:0/4188236369 >> 172.21.15.16:0/4188236369 conn(0x7f4f9c0cc5b0 msgr2=0x7f4f9c0cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:24:03.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.805+0000 7f4fa15c3700 1 -- 172.21.15.16:0/4188236369 shutdown_connections 2023-10-16T12:24:03.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.805+0000 7f4fa15c3700 1 -- 172.21.15.16:0/4188236369 wait complete. 2023-10-16T12:24:03.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.807+0000 7f4fa15c3700 1 Processor -- start 2023-10-16T12:24:03.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.807+0000 7f4fa15c3700 1 -- start start 2023-10-16T12:24:03.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.807+0000 7f4fa15c3700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4f9c0cb190 0x7f4f9c1501f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:03.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.808+0000 7f4fa15c3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f9c159c90 0x7f4f9c0ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:03.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.808+0000 7f4fa15c3700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4f9c15a070 0x7f4f9c0ace10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:03.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.808+0000 7f4fa15c3700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f9c150730 con 0x7f4f9c159c90 2023-10-16T12:24:03.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.808+0000 7f4fa15c3700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f9c0b56f0 con 0x7f4f9c0cb190 2023-10-16T12:24:03.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.808+0000 7f4fa15c3700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f9c0ad440 con 0x7f4f9c15a070 2023-10-16T12:24:03.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.808+0000 7f4f9a7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f9c159c90 0x7f4f9c0ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:03.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.808+0000 7f4f9a7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f9c159c90 0x7f4f9c0ac8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37738/0 (socket says 172.21.15.16:37738) 2023-10-16T12:24:03.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.808+0000 7f4f9a7fc700 1 -- 172.21.15.16:0/2321139831 learned_addr learned my addr 172.21.15.16:0/2321139831 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:24:03.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.808+0000 7f4f9affd700 1 --2- 172.21.15.16:0/2321139831 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4f9c0cb190 0x7f4f9c1501f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:03.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.808+0000 7f4f9b7fe700 1 --2- 172.21.15.16:0/2321139831 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4f9c15a070 0x7f4f9c0ace10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:03.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.808+0000 7f4f9a7fc700 1 -- 172.21.15.16:0/2321139831 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4f9c15a070 msgr2=0x7f4f9c0ace10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:03.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.808+0000 7f4f9a7fc700 1 --2- 172.21.15.16:0/2321139831 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4f9c15a070 0x7f4f9c0ace10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:03.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.808+0000 7f4f9a7fc700 1 -- 172.21.15.16:0/2321139831 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4f9c0cb190 msgr2=0x7f4f9c1501f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:03.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.808+0000 7f4f9a7fc700 1 --2- 172.21.15.16:0/2321139831 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4f9c0cb190 0x7f4f9c1501f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:03.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.808+0000 7f4f9a7fc700 1 -- 172.21.15.16:0/2321139831 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f8c0094a0 con 0x7f4f9c159c90 2023-10-16T12:24:03.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.808+0000 7f4f9b7fe700 1 --2- 172.21.15.16:0/2321139831 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4f9c15a070 0x7f4f9c0ace10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:24:03.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.808+0000 7f4f9a7fc700 1 --2- 172.21.15.16:0/2321139831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f9c159c90 0x7f4f9c0ac8d0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f4f9000c4d0 tx=0x7f4f9000c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:24:03.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.808+0000 7f4f837fe700 1 -- 172.21.15.16:0/2321139831 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f900052d0 con 0x7f4f9c159c90 2023-10-16T12:24:03.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.808+0000 7f4f837fe700 1 -- 172.21.15.16:0/2321139831 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f90005430 con 0x7f4f9c159c90 2023-10-16T12:24:03.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.808+0000 7f4fa15c3700 1 -- 172.21.15.16:0/2321139831 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f9c0ad720 con 0x7f4f9c159c90 2023-10-16T12:24:03.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.808+0000 7f4f837fe700 1 -- 172.21.15.16:0/2321139831 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f90003880 con 0x7f4f9c159c90 2023-10-16T12:24:03.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.808+0000 7f4fa15c3700 1 -- 172.21.15.16:0/2321139831 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f9c0adce0 con 0x7f4f9c159c90 2023-10-16T12:24:03.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.809+0000 7f4fa15c3700 1 -- 172.21.15.16:0/2321139831 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f9c04ebb0 con 0x7f4f9c159c90 2023-10-16T12:24:03.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.810+0000 7f4f837fe700 1 -- 172.21.15.16:0/2321139831 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4f900039e0 con 0x7f4f9c159c90 2023-10-16T12:24:03.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.810+0000 7f4f837fe700 1 --2- 172.21.15.16:0/2321139831 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f4f8806b020 0x7f4f8806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:03.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.810+0000 7f4f837fe700 1 -- 172.21.15.16:0/2321139831 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f4f90088bd0 con 0x7f4f9c159c90 2023-10-16T12:24:03.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.810+0000 7f4f9affd700 1 --2- 172.21.15.16:0/2321139831 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f4f8806b020 0x7f4f8806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:03.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.811+0000 7f4f9affd700 1 --2- 172.21.15.16:0/2321139831 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f4f8806b020 0x7f4f8806d4e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4f8c0090b0 tx=0x7f4f8c01c040 comp rx=0 tx=0).ready entity=mgr.5874 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:24:03.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:03.815+0000 7f4f837fe700 1 -- 172.21.15.16:0/2321139831 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4f90053a60 con 0x7f4f9c159c90 2023-10-16T12:24:04.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:04.018+0000 7f4fa15c3700 1 -- 172.21.15.16:0/2321139831 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4f9c0ce410 con 0x7f4f9c159c90 2023-10-16T12:24:04.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:04.019+0000 7f4f837fe700 1 -- 172.21.15.16:0/2321139831 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v67) v1 ==== 74+0+16679 (secure 0 0 0) 0x7f4f90057080 con 0x7f4f9c159c90 2023-10-16T12:24:04.021 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:24:04.022 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":67,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:24:02.694894+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:24:01.200494+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-10-16T12:24:01.652885+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:24:04.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:04.021+0000 7f4fa15c3700 1 -- 172.21.15.16:0/2321139831 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f4f8806b020 msgr2=0x7f4f8806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:04.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:04.021+0000 7f4fa15c3700 1 --2- 172.21.15.16:0/2321139831 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f4f8806b020 0x7f4f8806d4e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4f8c0090b0 tx=0x7f4f8c01c040 comp rx=0 tx=0).stop 2023-10-16T12:24:04.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:04.021+0000 7f4fa15c3700 1 -- 172.21.15.16:0/2321139831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f9c159c90 msgr2=0x7f4f9c0ac8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:04.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:04.021+0000 7f4fa15c3700 1 --2- 172.21.15.16:0/2321139831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f9c159c90 0x7f4f9c0ac8d0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f4f9000c4d0 tx=0x7f4f9000c7e0 comp rx=0 tx=0).stop 2023-10-16T12:24:04.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:04.021+0000 7f4fa15c3700 1 -- 172.21.15.16:0/2321139831 shutdown_connections 2023-10-16T12:24:04.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:04.021+0000 7f4fa15c3700 1 --2- 172.21.15.16:0/2321139831 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f4f8806b020 0x7f4f8806d4e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:04.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:04.021+0000 7f4fa15c3700 1 --2- 172.21.15.16:0/2321139831 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4f9c15a070 0x7f4f9c0ace10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:04.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:04.021+0000 7f4fa15c3700 1 --2- 172.21.15.16:0/2321139831 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4f9c0cb190 0x7f4f9c1501f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:04.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:04.021+0000 7f4fa15c3700 1 --2- 172.21.15.16:0/2321139831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f9c159c90 0x7f4f9c0ac8d0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:04.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:04.021+0000 7f4fa15c3700 1 -- 172.21.15.16:0/2321139831 >> 172.21.15.16:0/2321139831 conn(0x7f4f9c0cc5b0 msgr2=0x7f4f9c158f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:24:04.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:04.021+0000 7f4fa15c3700 1 -- 172.21.15.16:0/2321139831 shutdown_connections 2023-10-16T12:24:04.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:04.021+0000 7f4fa15c3700 1 -- 172.21.15.16:0/2321139831 wait complete. 2023-10-16T12:24:04.036 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-16T12:24:04.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:04.162+0000 7fde81d4b700 1 Processor -- start 2023-10-16T12:24:04.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:04.162+0000 7fde81d4b700 1 -- start start 2023-10-16T12:24:04.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:04.163+0000 7fde81d4b700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fde7c15b2c0 0x7fde7c161340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:04.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:04.163+0000 7fde81d4b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fde7c06a7e0 0x7fde7c06ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:04.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:04.163+0000 7fde81d4b700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde7c15ecb0 con 0x7fde7c06e570 2023-10-16T12:24:04.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:04.163+0000 7fde81d4b700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde7c06b1b0 con 0x7fde7c15b2c0 2023-10-16T12:24:04.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:04.163+0000 7fde81d4b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde7c161880 con 0x7fde7c06a7e0 2023-10-16T12:24:04.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:04.163+0000 7fde7b7fe700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fde7c15b2c0 0x7fde7c161340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:04.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:04.163+0000 7fde7bfff700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fde7c06a7e0 0x7fde7c06ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:04.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:04.163+0000 7fde7b7fe700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fde7c15b2c0 0x7fde7c161340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:36806/0 (socket says 172.21.15.16:36806) 2023-10-16T12:24:04.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:04.163+0000 7fde7b7fe700 1 -- 172.21.15.16:0/654220440 learned_addr learned my addr 172.21.15.16:0/654220440 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:24:04.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:04.164+0000 7fde7bfff700 1 -- 172.21.15.16:0/654220440 >> v2:172.21.15.49:3300/0 conn(0x7fde7c15b2c0 msgr2=0x7fde7c161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:24:04.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:04.164+0000 7fde7bfff700 1 --2- 172.21.15.16:0/654220440 >> v2:172.21.15.49:3300/0 conn(0x7fde7c15b2c0 0x7fde7c161340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:04.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:04.164+0000 7fde7bfff700 1 -- 172.21.15.16:0/654220440 >> v1:172.21.15.49:6789/0 conn(0x7fde7c06e570 legacy=0x7fde7c15e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:04.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:04.164+0000 7fde7bfff700 1 -- 172.21.15.16:0/654220440 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde7c15b6a0 con 0x7fde7c06a7e0 2023-10-16T12:24:04.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:04.164+0000 7fde7a7fc700 1 -- 172.21.15.16:0/654220440 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 553589436 0 0) 0x7fde7c15ecb0 con 0x7fde7c06e570 2023-10-16T12:24:04.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:04.164+0000 7fde7b7fe700 1 --2- 172.21.15.16:0/654220440 >> v2:172.21.15.49:3300/0 conn(0x7fde7c15b2c0 0x7fde7c161340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:24:04.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:04.164+0000 7fde7bfff700 1 --2- 172.21.15.16:0/654220440 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fde7c06a7e0 0x7fde7c06ac70 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fde74009d90 tx=0x7fde7400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=423745bd6cabad8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:24:04.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:04.164+0000 7fde7a7fc700 1 -- 172.21.15.16:0/654220440 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde7400e070 con 0x7fde7c06a7e0 2023-10-16T12:24:04.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:04.164+0000 7fde7a7fc700 1 -- 172.21.15.16:0/654220440 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde7400bb20 con 0x7fde7c06a7e0 2023-10-16T12:24:04.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:04.164+0000 7fde7a7fc700 1 -- 172.21.15.16:0/654220440 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde74012430 con 0x7fde7c06a7e0 2023-10-16T12:24:04.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:04.165+0000 7fde81d4b700 1 -- 172.21.15.16:0/654220440 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fde7c06a7e0 msgr2=0x7fde7c06ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:04.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:04.165+0000 7fde81d4b700 1 --2- 172.21.15.16:0/654220440 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fde7c06a7e0 0x7fde7c06ac70 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fde74009d90 tx=0x7fde7400b330 comp rx=0 tx=0).stop 2023-10-16T12:24:04.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:04.165+0000 7fde81d4b700 1 -- 172.21.15.16:0/654220440 shutdown_connections 2023-10-16T12:24:04.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:04.165+0000 7fde81d4b700 1 --2- 172.21.15.16:0/654220440 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fde7c06a7e0 0x7fde7c06ac70 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:04.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:04.165+0000 7fde81d4b700 1 --2- 172.21.15.16:0/654220440 >> v2:172.21.15.49:3300/0 conn(0x7fde7c15b2c0 0x7fde7c161340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:04.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:04.165+0000 7fde81d4b700 1 -- 172.21.15.16:0/654220440 >> 172.21.15.16:0/654220440 conn(0x7fde7c0b1f40 msgr2=0x7fde7c0b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:24:04.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:04.165+0000 7fde81d4b700 1 -- 172.21.15.16:0/654220440 shutdown_connections 2023-10-16T12:24:04.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:04.165+0000 7fde81d4b700 1 -- 172.21.15.16:0/654220440 wait complete. 2023-10-16T12:24:04.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:04.166+0000 7fde81d4b700 1 Processor -- start 2023-10-16T12:24:04.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:04.166+0000 7fde81d4b700 1 -- start start 2023-10-16T12:24:04.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:04.167+0000 7fde81d4b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fde7c06a7e0 0x7fde7c159c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:04.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:04.167+0000 7fde81d4b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fde7c06e570 0x7fde7c0c9cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:04.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:04.167+0000 7fde81d4b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde7c15b2c0 0x7fde7c0c7e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:04.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:04.167+0000 7fde81d4b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde7c1f4220 con 0x7fde7c15b2c0 2023-10-16T12:24:04.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:04.167+0000 7fde81d4b700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde7c0c7750 con 0x7fde7c06e570 2023-10-16T12:24:04.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:04.167+0000 7fde81d4b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde7c1f3610 con 0x7fde7c06a7e0 2023-10-16T12:24:04.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:04.167+0000 7fde7bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde7c15b2c0 0x7fde7c0c7e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:04.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:04.167+0000 7fde7bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde7c15b2c0 0x7fde7c0c7e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37754/0 (socket says 172.21.15.16:37754) 2023-10-16T12:24:04.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:04.167+0000 7fde7bfff700 1 -- 172.21.15.16:0/602949824 learned_addr learned my addr 172.21.15.16:0/602949824 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:24:04.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:04.167+0000 7fde7b7fe700 1 --2- 172.21.15.16:0/602949824 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fde7c06a7e0 0x7fde7c159c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:04.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:04.167+0000 7fde7affd700 1 --2- 172.21.15.16:0/602949824 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fde7c06e570 0x7fde7c0c9cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:04.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:04.167+0000 7fde7bfff700 1 -- 172.21.15.16:0/602949824 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fde7c06a7e0 msgr2=0x7fde7c159c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:04.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:04.167+0000 7fde7bfff700 1 --2- 172.21.15.16:0/602949824 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fde7c06a7e0 0x7fde7c159c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:04.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:04.168+0000 7fde7bfff700 1 -- 172.21.15.16:0/602949824 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fde7c06e570 msgr2=0x7fde7c0c9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:04.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:04.168+0000 7fde7bfff700 1 --2- 172.21.15.16:0/602949824 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fde7c06e570 0x7fde7c0c9cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:04.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:04.168+0000 7fde7bfff700 1 -- 172.21.15.16:0/602949824 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde6c009550 con 0x7fde7c15b2c0 2023-10-16T12:24:04.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:04.168+0000 7fde7bfff700 1 --2- 172.21.15.16:0/602949824 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde7c15b2c0 0x7fde7c0c7e60 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fde74014040 tx=0x7fde74004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:24:04.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:04.168+0000 7fde73fff700 1 -- 172.21.15.16:0/602949824 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde74012430 con 0x7fde7c15b2c0 2023-10-16T12:24:04.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:04.168+0000 7fde73fff700 1 -- 172.21.15.16:0/602949824 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde74020070 con 0x7fde7c15b2c0 2023-10-16T12:24:04.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:04.168+0000 7fde81d4b700 1 -- 172.21.15.16:0/602949824 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde74009a70 con 0x7fde7c15b2c0 2023-10-16T12:24:04.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:04.168+0000 7fde73fff700 1 -- 172.21.15.16:0/602949824 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde74004280 con 0x7fde7c15b2c0 2023-10-16T12:24:04.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:04.168+0000 7fde81d4b700 1 -- 172.21.15.16:0/602949824 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde7c1f3bd0 con 0x7fde7c15b2c0 2023-10-16T12:24:04.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:04.169+0000 7fde81d4b700 1 -- 172.21.15.16:0/602949824 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde7c0ca680 con 0x7fde7c15b2c0 2023-10-16T12:24:04.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:04.169+0000 7fde73fff700 1 -- 172.21.15.16:0/602949824 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fde740043e0 con 0x7fde7c15b2c0 2023-10-16T12:24:04.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:04.170+0000 7fde73fff700 1 --2- 172.21.15.16:0/602949824 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fde5c06b020 0x7fde5c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:04.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:04.170+0000 7fde73fff700 1 -- 172.21.15.16:0/602949824 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fde74022070 con 0x7fde7c15b2c0 2023-10-16T12:24:04.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:04.170+0000 7fde7b7fe700 1 --2- 172.21.15.16:0/602949824 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fde5c06b020 0x7fde5c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:04.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:04.170+0000 7fde7b7fe700 1 --2- 172.21.15.16:0/602949824 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fde5c06b020 0x7fde5c06d4e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fde6c006d30 tx=0x7fde6c0092c0 comp rx=0 tx=0).ready entity=mgr.5874 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:24:04.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:04.174+0000 7fde73fff700 1 -- 172.21.15.16:0/602949824 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fde74056290 con 0x7fde7c15b2c0 2023-10-16T12:24:04.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:04.430+0000 7fde81d4b700 1 -- 172.21.15.16:0/602949824 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fde7c0679f0 con 0x7fde7c15b2c0 2023-10-16T12:24:05.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.287+0000 7fde73fff700 1 -- 172.21.15.16:0/602949824 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v26) v1 ==== 118+0+0 (secure 0 0 0) 0x7fde740598b0 con 0x7fde7c15b2c0 2023-10-16T12:24:05.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.290+0000 7fde81d4b700 1 -- 172.21.15.16:0/602949824 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fde5c06b020 msgr2=0x7fde5c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:05.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.290+0000 7fde81d4b700 1 --2- 172.21.15.16:0/602949824 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fde5c06b020 0x7fde5c06d4e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fde6c006d30 tx=0x7fde6c0092c0 comp rx=0 tx=0).stop 2023-10-16T12:24:05.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.290+0000 7fde81d4b700 1 -- 172.21.15.16:0/602949824 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde7c15b2c0 msgr2=0x7fde7c0c7e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:05.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.290+0000 7fde81d4b700 1 --2- 172.21.15.16:0/602949824 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde7c15b2c0 0x7fde7c0c7e60 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fde74014040 tx=0x7fde74004ad0 comp rx=0 tx=0).stop 2023-10-16T12:24:05.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.290+0000 7fde81d4b700 1 -- 172.21.15.16:0/602949824 shutdown_connections 2023-10-16T12:24:05.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.290+0000 7fde81d4b700 1 --2- 172.21.15.16:0/602949824 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fde5c06b020 0x7fde5c06d4e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:05.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.290+0000 7fde81d4b700 1 --2- 172.21.15.16:0/602949824 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde7c15b2c0 0x7fde7c0c7e60 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:05.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.290+0000 7fde81d4b700 1 --2- 172.21.15.16:0/602949824 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fde7c06a7e0 0x7fde7c159c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:05.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.290+0000 7fde81d4b700 1 --2- 172.21.15.16:0/602949824 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fde7c06e570 0x7fde7c0c9cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:05.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.290+0000 7fde81d4b700 1 -- 172.21.15.16:0/602949824 >> 172.21.15.16:0/602949824 conn(0x7fde7c0b1f40 msgr2=0x7fde7c0d0030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:24:05.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.290+0000 7fde81d4b700 1 -- 172.21.15.16:0/602949824 shutdown_connections 2023-10-16T12:24:05.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.290+0000 7fde81d4b700 1 -- 172.21.15.16:0/602949824 wait complete. 2023-10-16T12:24:05.307 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-16T12:24:05.308 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-16T12:24:05.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.441+0000 7fc44fda4700 1 Processor -- start 2023-10-16T12:24:05.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.441+0000 7fc44fda4700 1 -- start start 2023-10-16T12:24:05.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.442+0000 7fc44fda4700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fc44815b2c0 0x7fc448161340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:05.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.442+0000 7fc44fda4700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc44806e570 0x7fc44815e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:05.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.442+0000 7fc44fda4700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc448161880 con 0x7fc44806a7e0 2023-10-16T12:24:05.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.442+0000 7fc44fda4700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc44815b6a0 con 0x7fc44815b2c0 2023-10-16T12:24:05.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.442+0000 7fc44fda4700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc448161b20 con 0x7fc44806e570 2023-10-16T12:24:05.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.442+0000 7fc44e341700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc44806a7e0 0x7fc448071a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:55268/0 (socket says 172.21.15.16:55268) 2023-10-16T12:24:05.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.442+0000 7fc44e341700 1 -- 172.21.15.16:0/2604015158 learned_addr learned my addr 172.21.15.16:0/2604015158 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:24:05.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.442+0000 7fc44db40700 1 --2- 172.21.15.16:0/2604015158 >> v2:172.21.15.49:3300/0 conn(0x7fc44815b2c0 0x7fc448161340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:05.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.442+0000 7fc44cb3e700 1 -- 172.21.15.16:0/2604015158 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3811584600 0 0) 0x7fc448161880 con 0x7fc44806a7e0 2023-10-16T12:24:05.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.442+0000 7fc44d33f700 1 --2- 172.21.15.16:0/2604015158 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc44806e570 0x7fc44815e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:05.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.442+0000 7fc44cb3e700 1 -- 172.21.15.16:0/2604015158 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4340035b0 con 0x7fc44806a7e0 2023-10-16T12:24:05.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.443+0000 7fc44cb3e700 1 -- 172.21.15.16:0/2604015158 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2047471868 0 0) 0x7fc4340035b0 con 0x7fc44806a7e0 2023-10-16T12:24:05.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.443+0000 7fc44cb3e700 1 -- 172.21.15.16:0/2604015158 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc44806e570 msgr2=0x7fc44815e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:24:05.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.443+0000 7fc44cb3e700 1 --2- 172.21.15.16:0/2604015158 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc44806e570 0x7fc44815e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:05.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.443+0000 7fc44cb3e700 1 -- 172.21.15.16:0/2604015158 >> v2:172.21.15.49:3300/0 conn(0x7fc44815b2c0 msgr2=0x7fc448161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:24:05.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.443+0000 7fc44cb3e700 1 --2- 172.21.15.16:0/2604015158 >> v2:172.21.15.49:3300/0 conn(0x7fc44815b2c0 0x7fc448161340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:05.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.443+0000 7fc44cb3e700 1 -- 172.21.15.16:0/2604015158 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc448161e00 con 0x7fc44806a7e0 2023-10-16T12:24:05.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.443+0000 7fc44cb3e700 1 -- 172.21.15.16:0/2604015158 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7fc4400019f0 con 0x7fc44806a7e0 2023-10-16T12:24:05.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.443+0000 7fc44cb3e700 1 -- 172.21.15.16:0/2604015158 >> v1:172.21.15.16:6789/0 conn(0x7fc44806a7e0 legacy=0x7fc448071a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:05.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.443+0000 7fc44cb3e700 1 --2- 172.21.15.16:0/2604015158 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc434003a00 0x7fc434003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:05.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.443+0000 7fc44cb3e700 1 --2- 172.21.15.16:0/2604015158 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc434004850 0x7fc434004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:05.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.443+0000 7fc44cb3e700 1 --2- 172.21.15.16:0/2604015158 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc44806e570 0x7fc434005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:05.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.443+0000 7fc44cb3e700 1 -- 172.21.15.16:0/2604015158 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc44815b6a0 con 0x7fc434003a00 2023-10-16T12:24:05.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.443+0000 7fc44cb3e700 1 -- 172.21.15.16:0/2604015158 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc448161b20 con 0x7fc434004850 2023-10-16T12:24:05.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.443+0000 7fc44cb3e700 1 -- 172.21.15.16:0/2604015158 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc43400ba10 con 0x7fc44806e570 2023-10-16T12:24:05.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.443+0000 7fc44db40700 1 --2- 172.21.15.16:0/2604015158 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc434003a00 0x7fc434003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:05.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.443+0000 7fc44d33f700 1 --2- 172.21.15.16:0/2604015158 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc434004850 0x7fc434004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:05.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.443+0000 7fc44e341700 1 --2- 172.21.15.16:0/2604015158 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc44806e570 0x7fc434005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:05.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.443+0000 7fc44db40700 1 -- 172.21.15.16:0/2604015158 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc44806e570 msgr2=0x7fc434005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:24:05.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.443+0000 7fc44db40700 1 --2- 172.21.15.16:0/2604015158 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc44806e570 0x7fc434005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:05.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.443+0000 7fc44db40700 1 -- 172.21.15.16:0/2604015158 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc434004850 msgr2=0x7fc434004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:24:05.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.443+0000 7fc44db40700 1 --2- 172.21.15.16:0/2604015158 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc434004850 0x7fc434004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:05.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.443+0000 7fc44db40700 1 -- 172.21.15.16:0/2604015158 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4340035b0 con 0x7fc434003a00 2023-10-16T12:24:05.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.443+0000 7fc44db40700 1 --2- 172.21.15.16:0/2604015158 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc434003a00 0x7fc434003e20 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fc438009b90 tx=0x7fc434005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33da34d1a61b318b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:24:05.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.443+0000 7fc44cb3e700 1 -- 172.21.15.16:0/2604015158 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc43800e6c0 con 0x7fc434003a00 2023-10-16T12:24:05.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.443+0000 7fc44cb3e700 1 -- 172.21.15.16:0/2604015158 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc43800e820 con 0x7fc434003a00 2023-10-16T12:24:05.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.444+0000 7fc44cb3e700 1 -- 172.21.15.16:0/2604015158 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc43800eb90 con 0x7fc434003a00 2023-10-16T12:24:05.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.444+0000 7fc44fda4700 1 -- 172.21.15.16:0/2604015158 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc434003a00 msgr2=0x7fc434003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:05.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.444+0000 7fc44fda4700 1 --2- 172.21.15.16:0/2604015158 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc434003a00 0x7fc434003e20 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fc438009b90 tx=0x7fc434005880 comp rx=0 tx=0).stop 2023-10-16T12:24:05.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.444+0000 7fc44e341700 1 -- 172.21.15.16:0/2604015158 reap_dead start 2023-10-16T12:24:05.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.444+0000 7fc44fda4700 1 -- 172.21.15.16:0/2604015158 shutdown_connections 2023-10-16T12:24:05.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.444+0000 7fc44fda4700 1 -- 172.21.15.16:0/2604015158 >> 172.21.15.16:0/2604015158 conn(0x7fc4480b1f40 msgr2=0x7fc4480b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:24:05.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.444+0000 7fc44fda4700 1 -- 172.21.15.16:0/2604015158 shutdown_connections 2023-10-16T12:24:05.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.444+0000 7fc44fda4700 1 -- 172.21.15.16:0/2604015158 wait complete. 2023-10-16T12:24:05.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.445+0000 7fc44fda4700 1 Processor -- start 2023-10-16T12:24:05.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.445+0000 7fc44fda4700 1 -- start start 2023-10-16T12:24:05.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.446+0000 7fc44fda4700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc434003a00 0x7fc44814ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:05.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.446+0000 7fc44fda4700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc44814f660 0x7fc4480b2f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:05.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.446+0000 7fc44fda4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4480b3450 0x7fc4480b38d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:05.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.446+0000 7fc44fda4700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4480c7dd0 con 0x7fc4480b3450 2023-10-16T12:24:05.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.446+0000 7fc44fda4700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4480c74c0 con 0x7fc434003a00 2023-10-16T12:24:05.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.446+0000 7fc44fda4700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc448152730 con 0x7fc44814f660 2023-10-16T12:24:05.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.446+0000 7fc44e341700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4480b3450 0x7fc4480b38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:05.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.446+0000 7fc44e341700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4480b3450 0x7fc4480b38d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37774/0 (socket says 172.21.15.16:37774) 2023-10-16T12:24:05.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.446+0000 7fc44e341700 1 -- 172.21.15.16:0/3110269339 learned_addr learned my addr 172.21.15.16:0/3110269339 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:24:05.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.447+0000 7fc44d33f700 1 --2- 172.21.15.16:0/3110269339 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc44814f660 0x7fc4480b2f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:05.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.447+0000 7fc44db40700 1 --2- 172.21.15.16:0/3110269339 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc434003a00 0x7fc44814ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:05.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.447+0000 7fc44e341700 1 -- 172.21.15.16:0/3110269339 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc44814f660 msgr2=0x7fc4480b2f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:05.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.447+0000 7fc44e341700 1 --2- 172.21.15.16:0/3110269339 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc44814f660 0x7fc4480b2f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:05.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.447+0000 7fc44e341700 1 -- 172.21.15.16:0/3110269339 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc434003a00 msgr2=0x7fc44814ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:05.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.447+0000 7fc44e341700 1 --2- 172.21.15.16:0/3110269339 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc434003a00 0x7fc44814ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:05.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.447+0000 7fc44e341700 1 -- 172.21.15.16:0/3110269339 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4480c7710 con 0x7fc4480b3450 2023-10-16T12:24:05.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.447+0000 7fc44d33f700 1 --2- 172.21.15.16:0/3110269339 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc44814f660 0x7fc4480b2f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:24:05.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.447+0000 7fc44db40700 1 --2- 172.21.15.16:0/3110269339 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc434003a00 0x7fc44814ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:24:05.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.447+0000 7fc44e341700 1 --2- 172.21.15.16:0/3110269339 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4480b3450 0x7fc4480b38d0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fc440001330 tx=0x7fc44000a640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:24:05.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.447+0000 7fc43e7fc700 1 -- 172.21.15.16:0/3110269339 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4400133f0 con 0x7fc4480b3450 2023-10-16T12:24:05.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.447+0000 7fc43e7fc700 1 -- 172.21.15.16:0/3110269339 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc440013590 con 0x7fc4480b3450 2023-10-16T12:24:05.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.447+0000 7fc44fda4700 1 -- 172.21.15.16:0/3110269339 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4480c79f0 con 0x7fc4480b3450 2023-10-16T12:24:05.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.447+0000 7fc43e7fc700 1 -- 172.21.15.16:0/3110269339 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc440013940 con 0x7fc4480b3450 2023-10-16T12:24:05.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.447+0000 7fc44fda4700 1 -- 172.21.15.16:0/3110269339 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc448151220 con 0x7fc4480b3450 2023-10-16T12:24:05.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.448+0000 7fc44fda4700 1 -- 172.21.15.16:0/3110269339 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4480c7f10 con 0x7fc4480b3450 2023-10-16T12:24:05.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.449+0000 7fc43e7fc700 1 -- 172.21.15.16:0/3110269339 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc440013aa0 con 0x7fc4480b3450 2023-10-16T12:24:05.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.449+0000 7fc43e7fc700 1 --2- 172.21.15.16:0/3110269339 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fc42c06b020 0x7fc42c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:05.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.449+0000 7fc43e7fc700 1 -- 172.21.15.16:0/3110269339 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fc44008a240 con 0x7fc4480b3450 2023-10-16T12:24:05.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.449+0000 7fc44db40700 1 --2- 172.21.15.16:0/3110269339 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fc42c06b020 0x7fc42c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:05.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.449+0000 7fc44db40700 1 --2- 172.21.15.16:0/3110269339 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fc42c06b020 0x7fc42c06d4e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc438009b30 tx=0x7fc438008030 comp rx=0 tx=0).ready entity=mgr.5874 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:24:05.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.453+0000 7fc43e7fc700 1 -- 172.21.15.16:0/3110269339 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc4480c7f10 con 0x7fc4480b3450 2023-10-16T12:24:05.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:05.702+0000 7fc44fda4700 1 -- 172.21.15.16:0/3110269339 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fc4480c7f10 con 0x7fc4480b3450 2023-10-16T12:24:06.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.288+0000 7fc43e7fc700 1 -- 172.21.15.16:0/3110269339 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v27) v1 ==== 122+0+0 (secure 0 0 0) 0x7fc4480c7f10 con 0x7fc4480b3450 2023-10-16T12:24:06.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.291+0000 7fc44fda4700 1 -- 172.21.15.16:0/3110269339 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fc42c06b020 msgr2=0x7fc42c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:06.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.291+0000 7fc44fda4700 1 --2- 172.21.15.16:0/3110269339 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fc42c06b020 0x7fc42c06d4e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc438009b30 tx=0x7fc438008030 comp rx=0 tx=0).stop 2023-10-16T12:24:06.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.291+0000 7fc44fda4700 1 -- 172.21.15.16:0/3110269339 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4480b3450 msgr2=0x7fc4480b38d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:06.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.291+0000 7fc44fda4700 1 --2- 172.21.15.16:0/3110269339 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4480b3450 0x7fc4480b38d0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fc440001330 tx=0x7fc44000a640 comp rx=0 tx=0).stop 2023-10-16T12:24:06.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.291+0000 7fc44fda4700 1 -- 172.21.15.16:0/3110269339 shutdown_connections 2023-10-16T12:24:06.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.291+0000 7fc44fda4700 1 --2- 172.21.15.16:0/3110269339 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fc42c06b020 0x7fc42c06d4e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:06.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.291+0000 7fc44fda4700 1 --2- 172.21.15.16:0/3110269339 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4480b3450 0x7fc4480b38d0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:06.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.291+0000 7fc44fda4700 1 --2- 172.21.15.16:0/3110269339 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc434003a00 0x7fc44814ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:06.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.291+0000 7fc44fda4700 1 --2- 172.21.15.16:0/3110269339 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc44814f660 0x7fc4480b2f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:06.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.291+0000 7fc44fda4700 1 -- 172.21.15.16:0/3110269339 >> 172.21.15.16:0/3110269339 conn(0x7fc4480b1f40 msgr2=0x7fc44815b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:24:06.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.291+0000 7fc44fda4700 1 -- 172.21.15.16:0/3110269339 shutdown_connections 2023-10-16T12:24:06.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.291+0000 7fc44fda4700 1 -- 172.21.15.16:0/3110269339 wait complete. 2023-10-16T12:24:06.308 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-16T12:24:06.308 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:24:06.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.439+0000 7f242ef82700 1 Processor -- start 2023-10-16T12:24:06.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.439+0000 7f242ef82700 1 -- start start 2023-10-16T12:24:06.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.440+0000 7f242ef82700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f242815b280 0x7f24281613e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:06.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.440+0000 7f242ef82700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f24280cc8a0 0x7f242815e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:06.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.440+0000 7f242ef82700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2428161c30 con 0x7f242806a7e0 2023-10-16T12:24:06.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.440+0000 7f242ef82700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2428161f10 con 0x7f242815b280 2023-10-16T12:24:06.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.440+0000 7f242ef82700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24281621f0 con 0x7f24280cc8a0 2023-10-16T12:24:06.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.440+0000 7f242cd1e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f242815b280 0x7f24281613e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:06.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.440+0000 7f242cd1e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f242815b280 0x7f24281613e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37778/0 (socket says 172.21.15.16:37778) 2023-10-16T12:24:06.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.440+0000 7f242cd1e700 1 -- 172.21.15.16:0/814831400 learned_addr learned my addr 172.21.15.16:0/814831400 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:24:06.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.440+0000 7f2427fff700 1 --2- 172.21.15.16:0/814831400 >> v2:172.21.15.49:3300/0 conn(0x7f24280cc8a0 0x7f242815e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:06.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.441+0000 7f242cd1e700 1 -- 172.21.15.16:0/814831400 >> v2:172.21.15.49:3300/0 conn(0x7f24280cc8a0 msgr2=0x7f242815e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:24:06.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.441+0000 7f242cd1e700 1 --2- 172.21.15.16:0/814831400 >> v2:172.21.15.49:3300/0 conn(0x7f24280cc8a0 0x7f242815e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:06.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.441+0000 7f242cd1e700 1 -- 172.21.15.16:0/814831400 >> v1:172.21.15.16:6789/0 conn(0x7f242806a7e0 legacy=0x7f24280d02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:06.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.441+0000 7f242cd1e700 1 -- 172.21.15.16:0/814831400 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f242815df00 con 0x7f242815b280 2023-10-16T12:24:06.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.441+0000 7f242cd1e700 1 --2- 172.21.15.16:0/814831400 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f242815b280 0x7f24281613e0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f2428071970 tx=0x7f241c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2f7af21da3a7f21 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:24:06.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.441+0000 7f24277fe700 1 -- 172.21.15.16:0/814831400 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f241c00b870 con 0x7f242815b280 2023-10-16T12:24:06.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.441+0000 7f24277fe700 1 -- 172.21.15.16:0/814831400 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f241c00b9d0 con 0x7f242815b280 2023-10-16T12:24:06.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.441+0000 7f24277fe700 1 -- 172.21.15.16:0/814831400 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f241c00bd60 con 0x7f242815b280 2023-10-16T12:24:06.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.441+0000 7f242ef82700 1 -- 172.21.15.16:0/814831400 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f242815b280 msgr2=0x7f24281613e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:06.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.441+0000 7f242ef82700 1 --2- 172.21.15.16:0/814831400 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f242815b280 0x7f24281613e0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f2428071970 tx=0x7f241c009bd0 comp rx=0 tx=0).stop 2023-10-16T12:24:06.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.441+0000 7f242ef82700 1 -- 172.21.15.16:0/814831400 shutdown_connections 2023-10-16T12:24:06.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.441+0000 7f242ef82700 1 --2- 172.21.15.16:0/814831400 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f242815b280 0x7f24281613e0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:06.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.441+0000 7f242ef82700 1 --2- 172.21.15.16:0/814831400 >> v2:172.21.15.49:3300/0 conn(0x7f24280cc8a0 0x7f242815e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:06.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.441+0000 7f242ef82700 1 -- 172.21.15.16:0/814831400 >> 172.21.15.16:0/814831400 conn(0x7f24280b1fd0 msgr2=0x7f24280b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:24:06.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.442+0000 7f242ef82700 1 -- 172.21.15.16:0/814831400 shutdown_connections 2023-10-16T12:24:06.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.442+0000 7f242ef82700 1 -- 172.21.15.16:0/814831400 wait complete. 2023-10-16T12:24:06.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.443+0000 7f242ef82700 1 Processor -- start 2023-10-16T12:24:06.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.443+0000 7f242ef82700 1 -- start start 2023-10-16T12:24:06.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.444+0000 7f242ef82700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f242806a7e0 0x7f24281f4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:06.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.444+0000 7f242ef82700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f24280cc8a0 0x7f242814e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:06.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.444+0000 7f242ef82700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f242815b280 0x7f242806ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:06.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.444+0000 7f242ef82700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24280cad20 con 0x7f242815b280 2023-10-16T12:24:06.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.444+0000 7f242ef82700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f242814ef00 con 0x7f242806a7e0 2023-10-16T12:24:06.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.444+0000 7f242ef82700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24280c7510 con 0x7f24280cc8a0 2023-10-16T12:24:06.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.444+0000 7f242d51f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f242815b280 0x7f242806ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:06.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.444+0000 7f242d51f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f242815b280 0x7f242806ac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37788/0 (socket says 172.21.15.16:37788) 2023-10-16T12:24:06.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.444+0000 7f242d51f700 1 -- 172.21.15.16:0/4194369479 learned_addr learned my addr 172.21.15.16:0/4194369479 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:24:06.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.444+0000 7f242cd1e700 1 --2- 172.21.15.16:0/4194369479 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f242806a7e0 0x7f24281f4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:06.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.444+0000 7f2427fff700 1 --2- 172.21.15.16:0/4194369479 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f24280cc8a0 0x7f242814e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:06.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.444+0000 7f242d51f700 1 -- 172.21.15.16:0/4194369479 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f24280cc8a0 msgr2=0x7f242814e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:06.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.444+0000 7f242d51f700 1 --2- 172.21.15.16:0/4194369479 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f24280cc8a0 0x7f242814e9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:06.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.444+0000 7f242d51f700 1 -- 172.21.15.16:0/4194369479 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f242806a7e0 msgr2=0x7f24281f4090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:06.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.444+0000 7f242d51f700 1 --2- 172.21.15.16:0/4194369479 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f242806a7e0 0x7f24281f4090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:06.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.444+0000 7f242d51f700 1 -- 172.21.15.16:0/4194369479 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f241c009550 con 0x7f242815b280 2023-10-16T12:24:06.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.444+0000 7f242d51f700 1 --2- 172.21.15.16:0/4194369479 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f242815b280 0x7f242806ac10 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f242000cb50 tx=0x7f242000ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:24:06.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.444+0000 7f24257fa700 1 -- 172.21.15.16:0/4194369479 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2420003c00 con 0x7f242815b280 2023-10-16T12:24:06.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.444+0000 7f24257fa700 1 -- 172.21.15.16:0/4194369479 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2420003d60 con 0x7f242815b280 2023-10-16T12:24:06.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.444+0000 7f242ef82700 1 -- 172.21.15.16:0/4194369479 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24280c77f0 con 0x7f242815b280 2023-10-16T12:24:06.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.444+0000 7f24257fa700 1 -- 172.21.15.16:0/4194369479 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f242000aa20 con 0x7f242815b280 2023-10-16T12:24:06.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.445+0000 7f242ef82700 1 -- 172.21.15.16:0/4194369479 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24280c7db0 con 0x7f242815b280 2023-10-16T12:24:06.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.445+0000 7f242ef82700 1 -- 172.21.15.16:0/4194369479 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f242804ebb0 con 0x7f242815b280 2023-10-16T12:24:06.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.446+0000 7f24257fa700 1 -- 172.21.15.16:0/4194369479 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f242000ab80 con 0x7f242815b280 2023-10-16T12:24:06.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.446+0000 7f24257fa700 1 --2- 172.21.15.16:0/4194369479 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f240c06b020 0x7f240c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:06.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.446+0000 7f24257fa700 1 -- 172.21.15.16:0/4194369479 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f2420089930 con 0x7f242815b280 2023-10-16T12:24:06.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.446+0000 7f242cd1e700 1 --2- 172.21.15.16:0/4194369479 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f240c06b020 0x7f240c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:06.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.447+0000 7f242cd1e700 1 --2- 172.21.15.16:0/4194369479 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f240c06b020 0x7f240c06d4e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f241c000c00 tx=0x7f241c008030 comp rx=0 tx=0).ready entity=mgr.5874 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:24:06.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.451+0000 7f24257fa700 1 -- 172.21.15.16:0/4194369479 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f24200546f0 con 0x7f242815b280 2023-10-16T12:24:06.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.698+0000 7f242ef82700 1 -- 172.21.15.16:0/4194369479 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2428072870 con 0x7f242815b280 2023-10-16T12:24:06.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.698+0000 7f24257fa700 1 -- 172.21.15.16:0/4194369479 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+4913 (secure 0 0 0) 0x7f2420057d10 con 0x7f242815b280 2023-10-16T12:24:06.701 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:24:06.701 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5947,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/1103417847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":1103417847},{"type":"v1","addr":"172.21.15.49:6835","nonce":1103417847}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":6096,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/1380792179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1380792179},{"type":"v1","addr":"172.21.15.16:6836","nonce":1380792179}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":6099,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2238585443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":2238585443},{"type":"v1","addr":"172.21.15.16:6837","nonce":2238585443}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:24:02.695223+0000","modified":"2023-10-16T12:24:05.704039+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6111},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6111":{"gid":6111,"name":"b","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.49:6834/3246624425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":3246624425},{"type":"v1","addr":"172.21.15.49:6834","nonce":3246624425}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-10-16T12:24:06.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.701+0000 7f242ef82700 1 -- 172.21.15.16:0/4194369479 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f240c06b020 msgr2=0x7f240c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:06.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.701+0000 7f242ef82700 1 --2- 172.21.15.16:0/4194369479 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f240c06b020 0x7f240c06d4e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f241c000c00 tx=0x7f241c008030 comp rx=0 tx=0).stop 2023-10-16T12:24:06.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.701+0000 7f242ef82700 1 -- 172.21.15.16:0/4194369479 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f242815b280 msgr2=0x7f242806ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:06.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.701+0000 7f242ef82700 1 --2- 172.21.15.16:0/4194369479 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f242815b280 0x7f242806ac10 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f242000cb50 tx=0x7f242000ce60 comp rx=0 tx=0).stop 2023-10-16T12:24:06.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.701+0000 7f242ef82700 1 -- 172.21.15.16:0/4194369479 shutdown_connections 2023-10-16T12:24:06.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.701+0000 7f242ef82700 1 --2- 172.21.15.16:0/4194369479 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f240c06b020 0x7f240c06d4e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:06.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.701+0000 7f242ef82700 1 --2- 172.21.15.16:0/4194369479 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f242815b280 0x7f242806ac10 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:06.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.701+0000 7f242ef82700 1 --2- 172.21.15.16:0/4194369479 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f242806a7e0 0x7f24281f4090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:06.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.701+0000 7f242ef82700 1 --2- 172.21.15.16:0/4194369479 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f24280cc8a0 0x7f242814e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:06.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.701+0000 7f242ef82700 1 -- 172.21.15.16:0/4194369479 >> 172.21.15.16:0/4194369479 conn(0x7f24280b1fd0 msgr2=0x7f2428159cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:24:06.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.701+0000 7f242ef82700 1 -- 172.21.15.16:0/4194369479 shutdown_connections 2023-10-16T12:24:06.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.701+0000 7f242ef82700 1 -- 172.21.15.16:0/4194369479 wait complete. 2023-10-16T12:24:06.705 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 27 2023-10-16T12:24:06.715 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:24:06.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.837+0000 7f403aade700 1 Processor -- start 2023-10-16T12:24:06.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.837+0000 7f403aade700 1 -- start start 2023-10-16T12:24:06.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.838+0000 7f403aade700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f403415b280 0x7f40341613e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:06.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.838+0000 7f403aade700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f40340cc8a0 0x7f403415e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:06.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.838+0000 7f403aade700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4034161c30 con 0x7f403406a7e0 2023-10-16T12:24:06.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.838+0000 7f403aade700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4034161f10 con 0x7f40340cc8a0 2023-10-16T12:24:06.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.838+0000 7f403aade700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40341621f0 con 0x7f403415b280 2023-10-16T12:24:06.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.838+0000 7f4033fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f40340cc8a0 0x7f403415e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:06.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.838+0000 7f4033fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f40340cc8a0 0x7f403415e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37794/0 (socket says 172.21.15.16:37794) 2023-10-16T12:24:06.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.838+0000 7f4033fff700 1 -- 172.21.15.16:0/2285944265 learned_addr learned my addr 172.21.15.16:0/2285944265 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:24:06.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.839+0000 7f403887a700 1 --2- 172.21.15.16:0/2285944265 >> v2:172.21.15.49:3300/0 conn(0x7f403415b280 0x7f40341613e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:06.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.839+0000 7f4033fff700 1 -- 172.21.15.16:0/2285944265 >> v2:172.21.15.49:3300/0 conn(0x7f403415b280 msgr2=0x7f40341613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:24:06.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.839+0000 7f4033fff700 1 --2- 172.21.15.16:0/2285944265 >> v2:172.21.15.49:3300/0 conn(0x7f403415b280 0x7f40341613e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:06.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.839+0000 7f4033fff700 1 -- 172.21.15.16:0/2285944265 >> v1:172.21.15.49:6789/0 conn(0x7f403406a7e0 legacy=0x7f40340d02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:24:06.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.839+0000 7f4033fff700 1 -- 172.21.15.16:0/2285944265 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f403415df00 con 0x7f40340cc8a0 2023-10-16T12:24:06.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.839+0000 7f403887a700 1 --2- 172.21.15.16:0/2285944265 >> v2:172.21.15.49:3300/0 conn(0x7f403415b280 0x7f40341613e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:24:06.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.839+0000 7f4033fff700 1 --2- 172.21.15.16:0/2285944265 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40340cc8a0 0x7f403415e700 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f403406b8a0 tx=0x7f4020009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8aabbfb9f8ec366 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:24:06.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.839+0000 7f40337fe700 1 -- 172.21.15.16:0/2285944265 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4020004230 con 0x7f40340cc8a0 2023-10-16T12:24:06.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.839+0000 7f40337fe700 1 -- 172.21.15.16:0/2285944265 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4020004390 con 0x7f40340cc8a0 2023-10-16T12:24:06.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.839+0000 7f40337fe700 1 -- 172.21.15.16:0/2285944265 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40200036a0 con 0x7f40340cc8a0 2023-10-16T12:24:06.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.839+0000 7f403aade700 1 -- 172.21.15.16:0/2285944265 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40340cc8a0 msgr2=0x7f403415e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:06.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.839+0000 7f403aade700 1 --2- 172.21.15.16:0/2285944265 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40340cc8a0 0x7f403415e700 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f403406b8a0 tx=0x7f4020009bd0 comp rx=0 tx=0).stop 2023-10-16T12:24:06.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.840+0000 7f403aade700 1 -- 172.21.15.16:0/2285944265 shutdown_connections 2023-10-16T12:24:06.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.840+0000 7f403aade700 1 --2- 172.21.15.16:0/2285944265 >> v2:172.21.15.49:3300/0 conn(0x7f403415b280 0x7f40341613e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:06.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.840+0000 7f403aade700 1 --2- 172.21.15.16:0/2285944265 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40340cc8a0 0x7f403415e700 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:06.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.840+0000 7f403aade700 1 -- 172.21.15.16:0/2285944265 >> 172.21.15.16:0/2285944265 conn(0x7f40340b1fd0 msgr2=0x7f40340b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:24:06.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.840+0000 7f403aade700 1 -- 172.21.15.16:0/2285944265 shutdown_connections 2023-10-16T12:24:06.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.840+0000 7f403aade700 1 -- 172.21.15.16:0/2285944265 wait complete. 2023-10-16T12:24:06.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.841+0000 7f403aade700 1 Processor -- start 2023-10-16T12:24:06.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.841+0000 7f403aade700 1 -- start start 2023-10-16T12:24:06.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.842+0000 7f403aade700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f403406a7e0 0x7f40340c9f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:06.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.842+0000 7f403aade700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f40340cc8a0 0x7f40341f2870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:06.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.842+0000 7f403aade700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f403415b280 0x7f40340c7510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:06.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.842+0000 7f403aade700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40340c7cf0 con 0x7f403406a7e0 2023-10-16T12:24:06.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.842+0000 7f403aade700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f403415cf30 con 0x7f40340cc8a0 2023-10-16T12:24:06.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.842+0000 7f403aade700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40341f2db0 con 0x7f403415b280 2023-10-16T12:24:06.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.842+0000 7f403887a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f403406a7e0 0x7f40340c9f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:06.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.842+0000 7f403887a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f403406a7e0 0x7f40340c9f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37798/0 (socket says 172.21.15.16:37798) 2023-10-16T12:24:06.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.842+0000 7f403887a700 1 -- 172.21.15.16:0/2041683808 learned_addr learned my addr 172.21.15.16:0/2041683808 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:24:06.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.842+0000 7f4033fff700 1 --2- 172.21.15.16:0/2041683808 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f40340cc8a0 0x7f40341f2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:06.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.842+0000 7f403907b700 1 --2- 172.21.15.16:0/2041683808 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f403415b280 0x7f40340c7510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:06.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.842+0000 7f403887a700 1 -- 172.21.15.16:0/2041683808 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f403415b280 msgr2=0x7f40340c7510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:06.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.842+0000 7f403887a700 1 --2- 172.21.15.16:0/2041683808 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f403415b280 0x7f40340c7510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:06.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.842+0000 7f403887a700 1 -- 172.21.15.16:0/2041683808 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f40340cc8a0 msgr2=0x7f40341f2870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:06.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.842+0000 7f403887a700 1 --2- 172.21.15.16:0/2041683808 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f40340cc8a0 0x7f40341f2870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:06.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.842+0000 7f403887a700 1 -- 172.21.15.16:0/2041683808 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4020009550 con 0x7f403406a7e0 2023-10-16T12:24:06.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.842+0000 7f403907b700 1 --2- 172.21.15.16:0/2041683808 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f403415b280 0x7f40340c7510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:24:06.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.842+0000 7f403887a700 1 --2- 172.21.15.16:0/2041683808 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f403406a7e0 0x7f40340c9f30 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f402800c780 tx=0x7f402800cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:24:06.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.842+0000 7f40317fa700 1 -- 172.21.15.16:0/2041683808 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4028011800 con 0x7f403406a7e0 2023-10-16T12:24:06.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.842+0000 7f40317fa700 1 -- 172.21.15.16:0/2041683808 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4028011960 con 0x7f403406a7e0 2023-10-16T12:24:06.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.842+0000 7f403aade700 1 -- 172.21.15.16:0/2041683808 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40340c7a50 con 0x7f403406a7e0 2023-10-16T12:24:06.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.842+0000 7f40317fa700 1 -- 172.21.15.16:0/2041683808 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4028011cd0 con 0x7f403406a7e0 2023-10-16T12:24:06.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.842+0000 7f403aade700 1 -- 172.21.15.16:0/2041683808 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4034158740 con 0x7f403406a7e0 2023-10-16T12:24:06.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.843+0000 7f403aade700 1 -- 172.21.15.16:0/2041683808 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f403404ebb0 con 0x7f403406a7e0 2023-10-16T12:24:06.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.844+0000 7f40317fa700 1 -- 172.21.15.16:0/2041683808 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4028021350 con 0x7f403406a7e0 2023-10-16T12:24:06.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.844+0000 7f40317fa700 1 --2- 172.21.15.16:0/2041683808 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f401806b020 0x7f401806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:06.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.844+0000 7f40317fa700 1 -- 172.21.15.16:0/2041683808 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f40280899c0 con 0x7f403406a7e0 2023-10-16T12:24:06.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.844+0000 7f4033fff700 1 --2- 172.21.15.16:0/2041683808 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f401806b020 0x7f401806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:06.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.845+0000 7f4033fff700 1 --2- 172.21.15.16:0/2041683808 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f401806b020 0x7f401806d4e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f403406b8a0 tx=0x7f4020008030 comp rx=0 tx=0).ready entity=mgr.5874 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:24:06.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:06.849+0000 7f40317fa700 1 -- 172.21.15.16:0/2041683808 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4028054690 con 0x7f403406a7e0 2023-10-16T12:24:07.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.053+0000 7f403aade700 1 -- 172.21.15.16:0/2041683808 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f40340cb380 con 0x7f403406a7e0 2023-10-16T12:24:07.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.055+0000 7f40317fa700 1 -- 172.21.15.16:0/2041683808 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v68) v1 ==== 74+0+16666 (secure 0 0 0) 0x7f4028057cb0 con 0x7f403406a7e0 2023-10-16T12:24:07.057 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:24:07.057 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":68,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:24:04.278212+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:24:01.200494+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-10-16T12:24:01.652885+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:24:07.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.057+0000 7f403aade700 1 -- 172.21.15.16:0/2041683808 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f401806b020 msgr2=0x7f401806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:07.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.057+0000 7f403aade700 1 --2- 172.21.15.16:0/2041683808 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f401806b020 0x7f401806d4e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f403406b8a0 tx=0x7f4020008030 comp rx=0 tx=0).stop 2023-10-16T12:24:07.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.057+0000 7f403aade700 1 -- 172.21.15.16:0/2041683808 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f403406a7e0 msgr2=0x7f40340c9f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:07.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.057+0000 7f403aade700 1 --2- 172.21.15.16:0/2041683808 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f403406a7e0 0x7f40340c9f30 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f402800c780 tx=0x7f402800cb40 comp rx=0 tx=0).stop 2023-10-16T12:24:07.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.057+0000 7f403aade700 1 -- 172.21.15.16:0/2041683808 shutdown_connections 2023-10-16T12:24:07.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.057+0000 7f403aade700 1 --2- 172.21.15.16:0/2041683808 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f401806b020 0x7f401806d4e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:07.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.057+0000 7f403aade700 1 --2- 172.21.15.16:0/2041683808 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f403415b280 0x7f40340c7510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:07.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.057+0000 7f403aade700 1 --2- 172.21.15.16:0/2041683808 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f403406a7e0 0x7f40340c9f30 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:07.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.057+0000 7f403aade700 1 --2- 172.21.15.16:0/2041683808 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f40340cc8a0 0x7f40341f2870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:07.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.057+0000 7f403aade700 1 -- 172.21.15.16:0/2041683808 >> 172.21.15.16:0/2041683808 conn(0x7f40340b1fd0 msgr2=0x7f40340cd960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:24:07.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.057+0000 7f403aade700 1 -- 172.21.15.16:0/2041683808 shutdown_connections 2023-10-16T12:24:07.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.057+0000 7f403aade700 1 -- 172.21.15.16:0/2041683808 wait complete. 2023-10-16T12:24:07.072 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-16T12:24:07.072 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T12:24:07.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.195+0000 7fc74a175700 1 Processor -- start 2023-10-16T12:24:07.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.195+0000 7fc74a175700 1 -- start start 2023-10-16T12:24:07.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.196+0000 7fc74a175700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc74406aef0 0x7fc7440b3480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:07.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.196+0000 7fc74a175700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc7440b2be0 0x7fc7440b39c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:07.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.196+0000 7fc74a175700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7440d11c0 con 0x7fc74415b860 2023-10-16T12:24:07.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.196+0000 7fc74a175700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7440d03d0 con 0x7fc74406aef0 2023-10-16T12:24:07.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.196+0000 7fc74a175700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7440cd360 con 0x7fc7440b2be0 2023-10-16T12:24:07.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.196+0000 7fc742ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc74406aef0 0x7fc7440b3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:07.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.196+0000 7fc742ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc74406aef0 0x7fc7440b3480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37806/0 (socket says 172.21.15.16:37806) 2023-10-16T12:24:07.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.196+0000 7fc742ffd700 1 -- 172.21.15.16:0/2963873555 learned_addr learned my addr 172.21.15.16:0/2963873555 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:24:07.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.196+0000 7fc743fff700 1 --2- 172.21.15.16:0/2963873555 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc7440b2be0 0x7fc7440b39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:07.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.196+0000 7fc742ffd700 1 -- 172.21.15.16:0/2963873555 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc7440b2be0 msgr2=0x7fc7440b39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:24:07.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.196+0000 7fc742ffd700 1 --2- 172.21.15.16:0/2963873555 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc7440b2be0 0x7fc7440b39c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:07.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.196+0000 7fc742ffd700 1 -- 172.21.15.16:0/2963873555 >> v1:172.21.15.16:6789/0 conn(0x7fc74415b860 legacy=0x7fc74406a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:07.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.196+0000 7fc742ffd700 1 -- 172.21.15.16:0/2963873555 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7440d05b0 con 0x7fc74406aef0 2023-10-16T12:24:07.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.197+0000 7fc742ffd700 1 --2- 172.21.15.16:0/2963873555 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc74406aef0 0x7fc7440b3480 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fc72c0098f0 tx=0x7fc72c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f754de20ec36e860 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:24:07.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.197+0000 7fc7427fc700 1 -- 172.21.15.16:0/2963873555 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc72c00b8a0 con 0x7fc74406aef0 2023-10-16T12:24:07.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.197+0000 7fc7427fc700 1 -- 172.21.15.16:0/2963873555 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc72c00ba00 con 0x7fc74406aef0 2023-10-16T12:24:07.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.197+0000 7fc7427fc700 1 -- 172.21.15.16:0/2963873555 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc72c00bd90 con 0x7fc74406aef0 2023-10-16T12:24:07.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.197+0000 7fc74a175700 1 -- 172.21.15.16:0/2963873555 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc74406aef0 msgr2=0x7fc7440b3480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:07.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.197+0000 7fc74a175700 1 --2- 172.21.15.16:0/2963873555 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc74406aef0 0x7fc7440b3480 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fc72c0098f0 tx=0x7fc72c009c00 comp rx=0 tx=0).stop 2023-10-16T12:24:07.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.197+0000 7fc74a175700 1 -- 172.21.15.16:0/2963873555 shutdown_connections 2023-10-16T12:24:07.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.197+0000 7fc74a175700 1 --2- 172.21.15.16:0/2963873555 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc7440b2be0 0x7fc7440b39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:07.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.197+0000 7fc74a175700 1 --2- 172.21.15.16:0/2963873555 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc74406aef0 0x7fc7440b3480 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:07.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.197+0000 7fc74a175700 1 -- 172.21.15.16:0/2963873555 >> 172.21.15.16:0/2963873555 conn(0x7fc7440b1fd0 msgr2=0x7fc7440b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:24:07.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.197+0000 7fc74a175700 1 -- 172.21.15.16:0/2963873555 shutdown_connections 2023-10-16T12:24:07.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.197+0000 7fc74a175700 1 -- 172.21.15.16:0/2963873555 wait complete. 2023-10-16T12:24:07.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.199+0000 7fc74a175700 1 Processor -- start 2023-10-16T12:24:07.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.199+0000 7fc74a175700 1 -- start start 2023-10-16T12:24:07.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.200+0000 7fc74a175700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc74406aef0 0x7fc7440af080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:07.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.200+0000 7fc74a175700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7440b2be0 0x7fc7440c8650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:07.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.200+0000 7fc74a175700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc74415b860 0x7fc7441f0a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:07.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.200+0000 7fc74a175700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc74415ecb0 con 0x7fc7440b2be0 2023-10-16T12:24:07.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.200+0000 7fc74a175700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7440c81c0 con 0x7fc74406aef0 2023-10-16T12:24:07.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.200+0000 7fc74a175700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7440af5c0 con 0x7fc74415b860 2023-10-16T12:24:07.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.200+0000 7fc742ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7440b2be0 0x7fc7440c8650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:07.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.200+0000 7fc742ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7440b2be0 0x7fc7440c8650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37814/0 (socket says 172.21.15.16:37814) 2023-10-16T12:24:07.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.200+0000 7fc742ffd700 1 -- 172.21.15.16:0/2013226410 learned_addr learned my addr 172.21.15.16:0/2013226410 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:24:07.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.200+0000 7fc743fff700 1 --2- 172.21.15.16:0/2013226410 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc74415b860 0x7fc7441f0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:07.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.200+0000 7fc7437fe700 1 --2- 172.21.15.16:0/2013226410 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc74406aef0 0x7fc7440af080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:07.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.200+0000 7fc742ffd700 1 -- 172.21.15.16:0/2013226410 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc74415b860 msgr2=0x7fc7441f0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:07.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.200+0000 7fc742ffd700 1 --2- 172.21.15.16:0/2013226410 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc74415b860 0x7fc7441f0a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:07.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.200+0000 7fc742ffd700 1 -- 172.21.15.16:0/2013226410 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc74406aef0 msgr2=0x7fc7440af080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:07.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.200+0000 7fc742ffd700 1 --2- 172.21.15.16:0/2013226410 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc74406aef0 0x7fc7440af080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:07.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.200+0000 7fc742ffd700 1 -- 172.21.15.16:0/2013226410 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc72c009550 con 0x7fc7440b2be0 2023-10-16T12:24:07.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.200+0000 7fc743fff700 1 --2- 172.21.15.16:0/2013226410 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc74415b860 0x7fc7441f0a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:24:07.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.200+0000 7fc7437fe700 1 --2- 172.21.15.16:0/2013226410 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc74406aef0 0x7fc7440af080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:24:07.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.200+0000 7fc742ffd700 1 --2- 172.21.15.16:0/2013226410 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7440b2be0 0x7fc7440c8650 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fc72c000c00 tx=0x7fc72c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:24:07.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.200+0000 7fc73bfff700 1 -- 172.21.15.16:0/2013226410 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc72c004570 con 0x7fc7440b2be0 2023-10-16T12:24:07.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.200+0000 7fc73bfff700 1 -- 172.21.15.16:0/2013226410 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc72c01f070 con 0x7fc7440b2be0 2023-10-16T12:24:07.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.200+0000 7fc74a175700 1 -- 172.21.15.16:0/2013226410 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7440c75b0 con 0x7fc7440b2be0 2023-10-16T12:24:07.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.200+0000 7fc73bfff700 1 -- 172.21.15.16:0/2013226410 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc72c0038e0 con 0x7fc7440b2be0 2023-10-16T12:24:07.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.200+0000 7fc74a175700 1 -- 172.21.15.16:0/2013226410 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7440c7b70 con 0x7fc7440b2be0 2023-10-16T12:24:07.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.201+0000 7fc74a175700 1 -- 172.21.15.16:0/2013226410 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7440c8330 con 0x7fc7440b2be0 2023-10-16T12:24:07.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.202+0000 7fc73bfff700 1 -- 172.21.15.16:0/2013226410 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc72c021030 con 0x7fc7440b2be0 2023-10-16T12:24:07.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.202+0000 7fc73bfff700 1 --2- 172.21.15.16:0/2013226410 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fc72406b020 0x7fc72406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:07.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.202+0000 7fc73bfff700 1 -- 172.21.15.16:0/2013226410 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fc72c08ae00 con 0x7fc7440b2be0 2023-10-16T12:24:07.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.202+0000 7fc7437fe700 1 --2- 172.21.15.16:0/2013226410 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fc72406b020 0x7fc72406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:07.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.203+0000 7fc7437fe700 1 --2- 172.21.15.16:0/2013226410 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fc72406b020 0x7fc72406d4e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc734000f60 tx=0x7fc734009040 comp rx=0 tx=0).ready entity=mgr.5874 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:24:07.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.207+0000 7fc73bfff700 1 -- 172.21.15.16:0/2013226410 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc72c055c70 con 0x7fc7440b2be0 2023-10-16T12:24:07.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.408+0000 7fc74a175700 1 -- 172.21.15.16:0/2013226410 --> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc7440d0b70 con 0x7fc72406b020 2023-10-16T12:24:07.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.417+0000 7fc73bfff700 1 -- 172.21.15.16:0/2013226410 <== mgr.5874 v2:172.21.15.16:6832/3870937344 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289204 (secure 0 0 0) 0x7fc734014420 con 0x7fc72406b020 2023-10-16T12:24:07.422 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:24:07.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.420+0000 7fc74a175700 1 -- 172.21.15.16:0/2013226410 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fc72406b020 msgr2=0x7fc72406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:07.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.420+0000 7fc74a175700 1 --2- 172.21.15.16:0/2013226410 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fc72406b020 0x7fc72406d4e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc734000f60 tx=0x7fc734009040 comp rx=0 tx=0).stop 2023-10-16T12:24:07.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.420+0000 7fc74a175700 1 -- 172.21.15.16:0/2013226410 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7440b2be0 msgr2=0x7fc7440c8650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:07.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.420+0000 7fc74a175700 1 --2- 172.21.15.16:0/2013226410 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7440b2be0 0x7fc7440c8650 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fc72c000c00 tx=0x7fc72c004750 comp rx=0 tx=0).stop 2023-10-16T12:24:07.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.420+0000 7fc74a175700 1 -- 172.21.15.16:0/2013226410 shutdown_connections 2023-10-16T12:24:07.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.420+0000 7fc74a175700 1 --2- 172.21.15.16:0/2013226410 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fc72406b020 0x7fc72406d4e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:07.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.420+0000 7fc74a175700 1 --2- 172.21.15.16:0/2013226410 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc74415b860 0x7fc7441f0a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:07.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.420+0000 7fc74a175700 1 --2- 172.21.15.16:0/2013226410 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc74406aef0 0x7fc7440af080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:07.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.420+0000 7fc74a175700 1 --2- 172.21.15.16:0/2013226410 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7440b2be0 0x7fc7440c8650 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:07.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.420+0000 7fc74a175700 1 -- 172.21.15.16:0/2013226410 >> 172.21.15.16:0/2013226410 conn(0x7fc7440b1fd0 msgr2=0x7fc7440b46f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:24:07.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.420+0000 7fc74a175700 1 -- 172.21.15.16:0/2013226410 shutdown_connections 2023-10-16T12:24:07.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.420+0000 7fc74a175700 1 -- 172.21.15.16:0/2013226410 wait complete. 2023-10-16T12:24:07.427 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-10-16T12:24:07.434 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-10-16T12:24:06.861116+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":243,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":177,"ondisk_log_size":177,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":154,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":336096,"kb_used_data":7520,"kb_used_omap":12,"kb_used_meta":328435,"kb_avail":754638624,"statfs":{"total":773094113280,"available":772749950976,"internally_reserved":0,"allocated":7700480,"data_stored":4355100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12740,"internal_metadata":336318012},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.602838"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619129+0000","last_change":"2023-10-16T12:24:03.287328+0000","last_active":"2023-10-16T12:24:04.619129+0000","last_peered":"2023-10-16T12:24:04.619129+0000","last_clean":"2023-10-16T12:24:04.619129+0000","last_became_active":"2023-10-16T12:24:03.287047+0000","last_became_peered":"2023-10-16T12:24:03.287047+0000","last_unstale":"2023-10-16T12:24:04.619129+0000","last_undegraded":"2023-10-16T12:24:04.619129+0000","last_fullsized":"2023-10-16T12:24:04.619129+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:32:22.452723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.620012+0000","last_change":"2023-10-16T12:24:03.282048+0000","last_active":"2023-10-16T12:24:04.620012+0000","last_peered":"2023-10-16T12:24:04.620012+0000","last_clean":"2023-10-16T12:24:04.620012+0000","last_became_active":"2023-10-16T12:24:03.281058+0000","last_became_peered":"2023-10-16T12:24:03.281058+0000","last_unstale":"2023-10-16T12:24:04.620012+0000","last_undegraded":"2023-10-16T12:24:04.620012+0000","last_fullsized":"2023-10-16T12:24:04.620012+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:22:10.364603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619402+0000","last_change":"2023-10-16T12:24:03.282806+0000","last_active":"2023-10-16T12:24:04.619402+0000","last_peered":"2023-10-16T12:24:04.619402+0000","last_clean":"2023-10-16T12:24:04.619402+0000","last_became_active":"2023-10-16T12:24:03.281883+0000","last_became_peered":"2023-10-16T12:24:03.281883+0000","last_unstale":"2023-10-16T12:24:04.619402+0000","last_undegraded":"2023-10-16T12:24:04.619402+0000","last_fullsized":"2023-10-16T12:24:04.619402+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:53:46.533436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619939+0000","last_change":"2023-10-16T12:24:03.288881+0000","last_active":"2023-10-16T12:24:04.619939+0000","last_peered":"2023-10-16T12:24:04.619939+0000","last_clean":"2023-10-16T12:24:04.619939+0000","last_became_active":"2023-10-16T12:24:03.288601+0000","last_became_peered":"2023-10-16T12:24:03.288601+0000","last_unstale":"2023-10-16T12:24:04.619939+0000","last_undegraded":"2023-10-16T12:24:04.619939+0000","last_fullsized":"2023-10-16T12:24:04.619939+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:53:46.533436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.620874+0000","last_change":"2023-10-16T12:24:03.280171+0000","last_active":"2023-10-16T12:24:04.620874+0000","last_peered":"2023-10-16T12:24:04.620874+0000","last_clean":"2023-10-16T12:24:04.620874+0000","last_became_active":"2023-10-16T12:24:03.279833+0000","last_became_peered":"2023-10-16T12:24:03.279833+0000","last_unstale":"2023-10-16T12:24:04.620874+0000","last_undegraded":"2023-10-16T12:24:04.620874+0000","last_fullsized":"2023-10-16T12:24:04.620874+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:00:44.935777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.618748+0000","last_change":"2023-10-16T12:24:03.288544+0000","last_active":"2023-10-16T12:24:04.618748+0000","last_peered":"2023-10-16T12:24:04.618748+0000","last_clean":"2023-10-16T12:24:04.618748+0000","last_became_active":"2023-10-16T12:24:03.288182+0000","last_became_peered":"2023-10-16T12:24:03.288182+0000","last_unstale":"2023-10-16T12:24:04.618748+0000","last_undegraded":"2023-10-16T12:24:04.618748+0000","last_fullsized":"2023-10-16T12:24:04.618748+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:17:39.834485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619487+0000","last_change":"2023-10-16T12:24:03.296406+0000","last_active":"2023-10-16T12:24:04.619487+0000","last_peered":"2023-10-16T12:24:04.619487+0000","last_clean":"2023-10-16T12:24:04.619487+0000","last_became_active":"2023-10-16T12:24:03.292117+0000","last_became_peered":"2023-10-16T12:24:03.292117+0000","last_unstale":"2023-10-16T12:24:04.619487+0000","last_undegraded":"2023-10-16T12:24:04.619487+0000","last_fullsized":"2023-10-16T12:24:04.619487+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:20:27.553468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.620307+0000","last_change":"2023-10-16T12:24:03.289998+0000","last_active":"2023-10-16T12:24:04.620307+0000","last_peered":"2023-10-16T12:24:04.620307+0000","last_clean":"2023-10-16T12:24:04.620307+0000","last_became_active":"2023-10-16T12:24:03.286418+0000","last_became_peered":"2023-10-16T12:24:03.286418+0000","last_unstale":"2023-10-16T12:24:04.620307+0000","last_undegraded":"2023-10-16T12:24:04.620307+0000","last_fullsized":"2023-10-16T12:24:04.620307+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:27:33.449783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619290+0000","last_change":"2023-10-16T12:24:03.282172+0000","last_active":"2023-10-16T12:24:04.619290+0000","last_peered":"2023-10-16T12:24:04.619290+0000","last_clean":"2023-10-16T12:24:04.619290+0000","last_became_active":"2023-10-16T12:24:03.281533+0000","last_became_peered":"2023-10-16T12:24:03.281533+0000","last_unstale":"2023-10-16T12:24:04.619290+0000","last_undegraded":"2023-10-16T12:24:04.619290+0000","last_fullsized":"2023-10-16T12:24:04.619290+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:23:16.330749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619420+0000","last_change":"2023-10-16T12:24:03.294968+0000","last_active":"2023-10-16T12:24:04.619420+0000","last_peered":"2023-10-16T12:24:04.619420+0000","last_clean":"2023-10-16T12:24:04.619420+0000","last_became_active":"2023-10-16T12:24:03.294681+0000","last_became_peered":"2023-10-16T12:24:03.294681+0000","last_unstale":"2023-10-16T12:24:04.619420+0000","last_undegraded":"2023-10-16T12:24:04.619420+0000","last_fullsized":"2023-10-16T12:24:04.619420+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:44:10.111021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.284448+0000","last_change":"2023-10-16T12:24:03.282888+0000","last_active":"2023-10-16T12:24:04.284448+0000","last_peered":"2023-10-16T12:24:04.284448+0000","last_clean":"2023-10-16T12:24:04.284448+0000","last_became_active":"2023-10-16T12:24:03.282599+0000","last_became_peered":"2023-10-16T12:24:03.282599+0000","last_unstale":"2023-10-16T12:24:04.284448+0000","last_undegraded":"2023-10-16T12:24:04.284448+0000","last_fullsized":"2023-10-16T12:24:04.284448+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:19.839274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619357+0000","last_change":"2023-10-16T12:24:03.291963+0000","last_active":"2023-10-16T12:24:04.619357+0000","last_peered":"2023-10-16T12:24:04.619357+0000","last_clean":"2023-10-16T12:24:04.619357+0000","last_became_active":"2023-10-16T12:24:03.291646+0000","last_became_peered":"2023-10-16T12:24:03.291646+0000","last_unstale":"2023-10-16T12:24:04.619357+0000","last_undegraded":"2023-10-16T12:24:04.619357+0000","last_fullsized":"2023-10-16T12:24:04.619357+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:10:24.643281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.618600+0000","last_change":"2023-10-16T12:24:03.294342+0000","last_active":"2023-10-16T12:24:04.618600+0000","last_peered":"2023-10-16T12:24:04.618600+0000","last_clean":"2023-10-16T12:24:04.618600+0000","last_became_active":"2023-10-16T12:24:03.293965+0000","last_became_peered":"2023-10-16T12:24:03.293965+0000","last_unstale":"2023-10-16T12:24:04.618600+0000","last_undegraded":"2023-10-16T12:24:04.618600+0000","last_fullsized":"2023-10-16T12:24:04.618600+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:29:08.023383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619390+0000","last_change":"2023-10-16T12:24:03.283959+0000","last_active":"2023-10-16T12:24:04.619390+0000","last_peered":"2023-10-16T12:24:04.619390+0000","last_clean":"2023-10-16T12:24:04.619390+0000","last_became_active":"2023-10-16T12:24:03.283638+0000","last_became_peered":"2023-10-16T12:24:03.283638+0000","last_unstale":"2023-10-16T12:24:04.619390+0000","last_undegraded":"2023-10-16T12:24:04.619390+0000","last_fullsized":"2023-10-16T12:24:04.619390+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:19:28.302798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619185+0000","last_change":"2023-10-16T12:24:03.293257+0000","last_active":"2023-10-16T12:24:04.619185+0000","last_peered":"2023-10-16T12:24:04.619185+0000","last_clean":"2023-10-16T12:24:04.619185+0000","last_became_active":"2023-10-16T12:24:03.292696+0000","last_became_peered":"2023-10-16T12:24:03.292696+0000","last_unstale":"2023-10-16T12:24:04.619185+0000","last_undegraded":"2023-10-16T12:24:04.619185+0000","last_fullsized":"2023-10-16T12:24:04.619185+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:19:28.302798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.620014+0000","last_change":"2023-10-16T12:24:03.294302+0000","last_active":"2023-10-16T12:24:04.620014+0000","last_peered":"2023-10-16T12:24:04.620014+0000","last_clean":"2023-10-16T12:24:04.620014+0000","last_became_active":"2023-10-16T12:24:03.294002+0000","last_became_peered":"2023-10-16T12:24:03.294002+0000","last_unstale":"2023-10-16T12:24:04.620014+0000","last_undegraded":"2023-10-16T12:24:04.620014+0000","last_fullsized":"2023-10-16T12:24:04.620014+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:00:44.935777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.620489+0000","last_change":"2023-10-16T12:24:03.279125+0000","last_active":"2023-10-16T12:24:04.620489+0000","last_peered":"2023-10-16T12:24:04.620489+0000","last_clean":"2023-10-16T12:24:04.620489+0000","last_became_active":"2023-10-16T12:24:03.278676+0000","last_became_peered":"2023-10-16T12:24:03.278676+0000","last_unstale":"2023-10-16T12:24:04.620489+0000","last_undegraded":"2023-10-16T12:24:04.620489+0000","last_fullsized":"2023-10-16T12:24:04.620489+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:10:24.643281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.620205+0000","last_change":"2023-10-16T12:24:03.282365+0000","last_active":"2023-10-16T12:24:04.620205+0000","last_peered":"2023-10-16T12:24:04.620205+0000","last_clean":"2023-10-16T12:24:04.620205+0000","last_became_active":"2023-10-16T12:24:03.282052+0000","last_became_peered":"2023-10-16T12:24:03.282052+0000","last_unstale":"2023-10-16T12:24:04.620205+0000","last_undegraded":"2023-10-16T12:24:04.620205+0000","last_fullsized":"2023-10-16T12:24:04.620205+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:35:43.782093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619586+0000","last_change":"2023-10-16T12:24:03.279495+0000","last_active":"2023-10-16T12:24:04.619586+0000","last_peered":"2023-10-16T12:24:04.619586+0000","last_clean":"2023-10-16T12:24:04.619586+0000","last_became_active":"2023-10-16T12:24:03.279196+0000","last_became_peered":"2023-10-16T12:24:03.279196+0000","last_unstale":"2023-10-16T12:24:04.619586+0000","last_undegraded":"2023-10-16T12:24:04.619586+0000","last_fullsized":"2023-10-16T12:24:04.619586+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:38:55.619256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.285527+0000","last_change":"2023-10-16T12:24:03.296079+0000","last_active":"2023-10-16T12:24:04.285527+0000","last_peered":"2023-10-16T12:24:04.285527+0000","last_clean":"2023-10-16T12:24:04.285527+0000","last_became_active":"2023-10-16T12:24:03.295803+0000","last_became_peered":"2023-10-16T12:24:03.295803+0000","last_unstale":"2023-10-16T12:24:04.285527+0000","last_undegraded":"2023-10-16T12:24:04.285527+0000","last_fullsized":"2023-10-16T12:24:04.285527+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:46:56.368712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.285659+0000","last_change":"2023-10-16T12:24:03.284801+0000","last_active":"2023-10-16T12:24:04.285659+0000","last_peered":"2023-10-16T12:24:04.285659+0000","last_clean":"2023-10-16T12:24:04.285659+0000","last_became_active":"2023-10-16T12:24:03.284466+0000","last_became_peered":"2023-10-16T12:24:03.284466+0000","last_unstale":"2023-10-16T12:24:04.285659+0000","last_undegraded":"2023-10-16T12:24:04.285659+0000","last_fullsized":"2023-10-16T12:24:04.285659+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:02:01.654872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.618738+0000","last_change":"2023-10-16T12:24:03.285544+0000","last_active":"2023-10-16T12:24:04.618738+0000","last_peered":"2023-10-16T12:24:04.618738+0000","last_clean":"2023-10-16T12:24:04.618738+0000","last_became_active":"2023-10-16T12:24:03.285245+0000","last_became_peered":"2023-10-16T12:24:03.285245+0000","last_unstale":"2023-10-16T12:24:04.618738+0000","last_undegraded":"2023-10-16T12:24:04.618738+0000","last_fullsized":"2023-10-16T12:24:04.618738+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:40:56.018881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.284266+0000","last_change":"2023-10-16T12:24:03.287309+0000","last_active":"2023-10-16T12:24:04.284266+0000","last_peered":"2023-10-16T12:24:04.284266+0000","last_clean":"2023-10-16T12:24:04.284266+0000","last_became_active":"2023-10-16T12:24:03.286997+0000","last_became_peered":"2023-10-16T12:24:03.286997+0000","last_unstale":"2023-10-16T12:24:04.284266+0000","last_undegraded":"2023-10-16T12:24:04.284266+0000","last_fullsized":"2023-10-16T12:24:04.284266+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:31:06.779111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.618603+0000","last_change":"2023-10-16T12:24:03.291789+0000","last_active":"2023-10-16T12:24:04.618603+0000","last_peered":"2023-10-16T12:24:04.618603+0000","last_clean":"2023-10-16T12:24:04.618603+0000","last_became_active":"2023-10-16T12:24:03.291471+0000","last_became_peered":"2023-10-16T12:24:03.291471+0000","last_unstale":"2023-10-16T12:24:04.618603+0000","last_undegraded":"2023-10-16T12:24:04.618603+0000","last_fullsized":"2023-10-16T12:24:04.618603+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:11:40.533026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.620413+0000","last_change":"2023-10-16T12:24:03.290830+0000","last_active":"2023-10-16T12:24:04.620413+0000","last_peered":"2023-10-16T12:24:04.620413+0000","last_clean":"2023-10-16T12:24:04.620413+0000","last_became_active":"2023-10-16T12:24:03.290443+0000","last_became_peered":"2023-10-16T12:24:03.290443+0000","last_unstale":"2023-10-16T12:24:04.620413+0000","last_undegraded":"2023-10-16T12:24:04.620413+0000","last_fullsized":"2023-10-16T12:24:04.620413+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:40:56.018881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619490+0000","last_change":"2023-10-16T12:24:03.290691+0000","last_active":"2023-10-16T12:24:04.619490+0000","last_peered":"2023-10-16T12:24:04.619490+0000","last_clean":"2023-10-16T12:24:04.619490+0000","last_became_active":"2023-10-16T12:24:03.290405+0000","last_became_peered":"2023-10-16T12:24:03.290405+0000","last_unstale":"2023-10-16T12:24:04.619490+0000","last_undegraded":"2023-10-16T12:24:04.619490+0000","last_fullsized":"2023-10-16T12:24:04.619490+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:01:16.802661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619265+0000","last_change":"2023-10-16T12:24:03.303054+0000","last_active":"2023-10-16T12:24:04.619265+0000","last_peered":"2023-10-16T12:24:04.619265+0000","last_clean":"2023-10-16T12:24:04.619265+0000","last_became_active":"2023-10-16T12:24:03.302758+0000","last_became_peered":"2023-10-16T12:24:03.302758+0000","last_unstale":"2023-10-16T12:24:04.619265+0000","last_undegraded":"2023-10-16T12:24:04.619265+0000","last_fullsized":"2023-10-16T12:24:04.619265+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:44:10.478322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.285824+0000","last_change":"2023-10-16T12:24:03.290932+0000","last_active":"2023-10-16T12:24:04.285824+0000","last_peered":"2023-10-16T12:24:04.285824+0000","last_clean":"2023-10-16T12:24:04.285824+0000","last_became_active":"2023-10-16T12:24:03.290639+0000","last_became_peered":"2023-10-16T12:24:03.290639+0000","last_unstale":"2023-10-16T12:24:04.285824+0000","last_undegraded":"2023-10-16T12:24:04.285824+0000","last_fullsized":"2023-10-16T12:24:04.285824+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:10:20.088811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619456+0000","last_change":"2023-10-16T12:24:03.294572+0000","last_active":"2023-10-16T12:24:04.619456+0000","last_peered":"2023-10-16T12:24:04.619456+0000","last_clean":"2023-10-16T12:24:04.619456+0000","last_became_active":"2023-10-16T12:24:03.291752+0000","last_became_peered":"2023-10-16T12:24:03.291752+0000","last_unstale":"2023-10-16T12:24:04.619456+0000","last_undegraded":"2023-10-16T12:24:04.619456+0000","last_fullsized":"2023-10-16T12:24:04.619456+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:35:43.093148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.620108+0000","last_change":"2023-10-16T12:24:03.293658+0000","last_active":"2023-10-16T12:24:04.620108+0000","last_peered":"2023-10-16T12:24:04.620108+0000","last_clean":"2023-10-16T12:24:04.620108+0000","last_became_active":"2023-10-16T12:24:03.292954+0000","last_became_peered":"2023-10-16T12:24:03.292954+0000","last_unstale":"2023-10-16T12:24:04.620108+0000","last_undegraded":"2023-10-16T12:24:04.620108+0000","last_fullsized":"2023-10-16T12:24:04.620108+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:38:55.619256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619326+0000","last_change":"2023-10-16T12:24:03.296212+0000","last_active":"2023-10-16T12:24:04.619326+0000","last_peered":"2023-10-16T12:24:04.619326+0000","last_clean":"2023-10-16T12:24:04.619326+0000","last_became_active":"2023-10-16T12:24:03.292558+0000","last_became_peered":"2023-10-16T12:24:03.292558+0000","last_unstale":"2023-10-16T12:24:04.619326+0000","last_undegraded":"2023-10-16T12:24:04.619326+0000","last_fullsized":"2023-10-16T12:24:04.619326+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:01:17.363320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.618938+0000","last_change":"2023-10-16T12:24:03.290176+0000","last_active":"2023-10-16T12:24:04.618938+0000","last_peered":"2023-10-16T12:24:04.618938+0000","last_clean":"2023-10-16T12:24:04.618938+0000","last_became_active":"2023-10-16T12:24:03.289565+0000","last_became_peered":"2023-10-16T12:24:03.289565+0000","last_unstale":"2023-10-16T12:24:04.618938+0000","last_undegraded":"2023-10-16T12:24:04.618938+0000","last_fullsized":"2023-10-16T12:24:04.618938+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:53:46.533436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.618832+0000","last_change":"2023-10-16T12:24:03.281732+0000","last_active":"2023-10-16T12:24:04.618832+0000","last_peered":"2023-10-16T12:24:04.618832+0000","last_clean":"2023-10-16T12:24:04.618832+0000","last_became_active":"2023-10-16T12:24:03.280747+0000","last_became_peered":"2023-10-16T12:24:03.280747+0000","last_unstale":"2023-10-16T12:24:04.618832+0000","last_undegraded":"2023-10-16T12:24:04.618832+0000","last_fullsized":"2023-10-16T12:24:04.618832+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:01:17.363320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"67'4","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.285479+0000","last_change":"2023-10-16T12:24:02.284432+0000","last_active":"2023-10-16T12:24:04.285479+0000","last_peered":"2023-10-16T12:24:04.285479+0000","last_clean":"2023-10-16T12:24:04.285479+0000","last_became_active":"2023-10-16T12:24:02.284129+0000","last_became_peered":"2023-10-16T12:24:02.284129+0000","last_unstale":"2023-10-16T12:24:04.285479+0000","last_undegraded":"2023-10-16T12:24:04.285479+0000","last_fullsized":"2023-10-16T12:24:04.285479+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_clean_scrub_stamp":"2023-10-16T12:24:01.254337+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:17:59.015657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.620037+0000","last_change":"2023-10-16T12:24:03.290379+0000","last_active":"2023-10-16T12:24:04.620037+0000","last_peered":"2023-10-16T12:24:04.620037+0000","last_clean":"2023-10-16T12:24:04.620037+0000","last_became_active":"2023-10-16T12:24:03.290042+0000","last_became_peered":"2023-10-16T12:24:03.290042+0000","last_unstale":"2023-10-16T12:24:04.620037+0000","last_undegraded":"2023-10-16T12:24:04.620037+0000","last_fullsized":"2023-10-16T12:24:04.620037+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:55:39.961058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"67'1","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619517+0000","last_change":"2023-10-16T12:24:02.287250+0000","last_active":"2023-10-16T12:24:04.619517+0000","last_peered":"2023-10-16T12:24:04.619517+0000","last_clean":"2023-10-16T12:24:04.619517+0000","last_became_active":"2023-10-16T12:24:02.287010+0000","last_became_peered":"2023-10-16T12:24:02.287010+0000","last_unstale":"2023-10-16T12:24:04.619517+0000","last_undegraded":"2023-10-16T12:24:04.619517+0000","last_fullsized":"2023-10-16T12:24:04.619517+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_clean_scrub_stamp":"2023-10-16T12:24:01.254337+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:46:55.364974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.285416+0000","last_change":"2023-10-16T12:24:03.290251+0000","last_active":"2023-10-16T12:24:04.285416+0000","last_peered":"2023-10-16T12:24:04.285416+0000","last_clean":"2023-10-16T12:24:04.285416+0000","last_became_active":"2023-10-16T12:24:03.289955+0000","last_became_peered":"2023-10-16T12:24:03.289955+0000","last_unstale":"2023-10-16T12:24:04.285416+0000","last_undegraded":"2023-10-16T12:24:04.285416+0000","last_fullsized":"2023-10-16T12:24:04.285416+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:41:19.510518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"67'2","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619053+0000","last_change":"2023-10-16T12:24:02.288769+0000","last_active":"2023-10-16T12:24:04.619053+0000","last_peered":"2023-10-16T12:24:04.619053+0000","last_clean":"2023-10-16T12:24:04.619053+0000","last_became_active":"2023-10-16T12:24:02.284978+0000","last_became_peered":"2023-10-16T12:24:02.284978+0000","last_unstale":"2023-10-16T12:24:04.619053+0000","last_undegraded":"2023-10-16T12:24:04.619053+0000","last_fullsized":"2023-10-16T12:24:04.619053+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_clean_scrub_stamp":"2023-10-16T12:24:01.254337+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:52:48.050577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.620334+0000","last_change":"2023-10-16T12:24:03.290027+0000","last_active":"2023-10-16T12:24:04.620334+0000","last_peered":"2023-10-16T12:24:04.620334+0000","last_clean":"2023-10-16T12:24:04.620334+0000","last_became_active":"2023-10-16T12:24:03.286282+0000","last_became_peered":"2023-10-16T12:24:03.286282+0000","last_unstale":"2023-10-16T12:24:04.620334+0000","last_undegraded":"2023-10-16T12:24:04.620334+0000","last_fullsized":"2023-10-16T12:24:04.620334+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:31:00.012476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"67'2","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619574+0000","last_change":"2023-10-16T12:24:02.287316+0000","last_active":"2023-10-16T12:24:04.619574+0000","last_peered":"2023-10-16T12:24:04.619574+0000","last_clean":"2023-10-16T12:24:04.619574+0000","last_became_active":"2023-10-16T12:24:02.287089+0000","last_became_peered":"2023-10-16T12:24:02.287089+0000","last_unstale":"2023-10-16T12:24:04.619574+0000","last_undegraded":"2023-10-16T12:24:04.619574+0000","last_fullsized":"2023-10-16T12:24:04.619574+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_clean_scrub_stamp":"2023-10-16T12:24:01.254337+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:01:16.359582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.621061+0000","last_change":"2023-10-16T12:24:03.282668+0000","last_active":"2023-10-16T12:24:04.621061+0000","last_peered":"2023-10-16T12:24:04.621061+0000","last_clean":"2023-10-16T12:24:04.621061+0000","last_became_active":"2023-10-16T12:24:03.282375+0000","last_became_peered":"2023-10-16T12:24:03.282375+0000","last_unstale":"2023-10-16T12:24:04.621061+0000","last_undegraded":"2023-10-16T12:24:04.621061+0000","last_fullsized":"2023-10-16T12:24:04.621061+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:18:54.252334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"67'1","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619563+0000","last_change":"2023-10-16T12:24:02.284422+0000","last_active":"2023-10-16T12:24:04.619563+0000","last_peered":"2023-10-16T12:24:04.619563+0000","last_clean":"2023-10-16T12:24:04.619563+0000","last_became_active":"2023-10-16T12:24:02.284140+0000","last_became_peered":"2023-10-16T12:24:02.284140+0000","last_unstale":"2023-10-16T12:24:04.619563+0000","last_undegraded":"2023-10-16T12:24:04.619563+0000","last_fullsized":"2023-10-16T12:24:04.619563+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_clean_scrub_stamp":"2023-10-16T12:24:01.254337+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:01:17.574797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.620006+0000","last_change":"2023-10-16T12:24:03.287876+0000","last_active":"2023-10-16T12:24:04.620006+0000","last_peered":"2023-10-16T12:24:04.620006+0000","last_clean":"2023-10-16T12:24:04.620006+0000","last_became_active":"2023-10-16T12:24:03.287461+0000","last_became_peered":"2023-10-16T12:24:03.287461+0000","last_unstale":"2023-10-16T12:24:04.620006+0000","last_undegraded":"2023-10-16T12:24:04.620006+0000","last_fullsized":"2023-10-16T12:24:04.620006+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:20:27.553468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"67'1","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.620663+0000","last_change":"2023-10-16T12:24:02.283692+0000","last_active":"2023-10-16T12:24:04.620663+0000","last_peered":"2023-10-16T12:24:04.620663+0000","last_clean":"2023-10-16T12:24:04.620663+0000","last_became_active":"2023-10-16T12:24:02.283241+0000","last_became_peered":"2023-10-16T12:24:02.283241+0000","last_unstale":"2023-10-16T12:24:04.620663+0000","last_undegraded":"2023-10-16T12:24:04.620663+0000","last_fullsized":"2023-10-16T12:24:04.620663+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_clean_scrub_stamp":"2023-10-16T12:24:01.254337+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:46:01.346151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619279+0000","last_change":"2023-10-16T12:24:03.283956+0000","last_active":"2023-10-16T12:24:04.619279+0000","last_peered":"2023-10-16T12:24:04.619279+0000","last_clean":"2023-10-16T12:24:04.619279+0000","last_became_active":"2023-10-16T12:24:03.281947+0000","last_became_peered":"2023-10-16T12:24:03.281947+0000","last_unstale":"2023-10-16T12:24:04.619279+0000","last_undegraded":"2023-10-16T12:24:04.619279+0000","last_fullsized":"2023-10-16T12:24:04.619279+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:06:14.168954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.285303+0000","last_change":"2023-10-16T12:24:02.281241+0000","last_active":"2023-10-16T12:24:04.285303+0000","last_peered":"2023-10-16T12:24:04.285303+0000","last_clean":"2023-10-16T12:24:04.285303+0000","last_became_active":"2023-10-16T12:24:02.280918+0000","last_became_peered":"2023-10-16T12:24:02.280918+0000","last_unstale":"2023-10-16T12:24:04.285303+0000","last_undegraded":"2023-10-16T12:24:04.285303+0000","last_fullsized":"2023-10-16T12:24:04.285303+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_clean_scrub_stamp":"2023-10-16T12:24:01.254337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:37:34.622976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619989+0000","last_change":"2023-10-16T12:24:03.282828+0000","last_active":"2023-10-16T12:24:04.619989+0000","last_peered":"2023-10-16T12:24:04.619989+0000","last_clean":"2023-10-16T12:24:04.619989+0000","last_became_active":"2023-10-16T12:24:03.282547+0000","last_became_peered":"2023-10-16T12:24:03.282547+0000","last_unstale":"2023-10-16T12:24:04.619989+0000","last_undegraded":"2023-10-16T12:24:04.619989+0000","last_fullsized":"2023-10-16T12:24:04.619989+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:35:43.093148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619673+0000","last_change":"2023-10-16T12:24:02.283508+0000","last_active":"2023-10-16T12:24:04.619673+0000","last_peered":"2023-10-16T12:24:04.619673+0000","last_clean":"2023-10-16T12:24:04.619673+0000","last_became_active":"2023-10-16T12:24:02.283119+0000","last_became_peered":"2023-10-16T12:24:02.283119+0000","last_unstale":"2023-10-16T12:24:04.619673+0000","last_undegraded":"2023-10-16T12:24:04.619673+0000","last_fullsized":"2023-10-16T12:24:04.619673+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_clean_scrub_stamp":"2023-10-16T12:24:01.254337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:39:17.482962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.620632+0000","last_change":"2023-10-16T12:24:03.290637+0000","last_active":"2023-10-16T12:24:04.620632+0000","last_peered":"2023-10-16T12:24:04.620632+0000","last_clean":"2023-10-16T12:24:04.620632+0000","last_became_active":"2023-10-16T12:24:03.288515+0000","last_became_peered":"2023-10-16T12:24:03.288515+0000","last_unstale":"2023-10-16T12:24:04.620632+0000","last_undegraded":"2023-10-16T12:24:04.620632+0000","last_fullsized":"2023-10-16T12:24:04.620632+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:44:10.111021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.620125+0000","last_change":"2023-10-16T12:24:02.280970+0000","last_active":"2023-10-16T12:24:04.620125+0000","last_peered":"2023-10-16T12:24:04.620125+0000","last_clean":"2023-10-16T12:24:04.620125+0000","last_became_active":"2023-10-16T12:24:02.280680+0000","last_became_peered":"2023-10-16T12:24:02.280680+0000","last_unstale":"2023-10-16T12:24:04.620125+0000","last_undegraded":"2023-10-16T12:24:04.620125+0000","last_fullsized":"2023-10-16T12:24:04.620125+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_clean_scrub_stamp":"2023-10-16T12:24:01.254337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:11:28.470877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619490+0000","last_change":"2023-10-16T12:24:03.282651+0000","last_active":"2023-10-16T12:24:04.619490+0000","last_peered":"2023-10-16T12:24:04.619490+0000","last_clean":"2023-10-16T12:24:04.619490+0000","last_became_active":"2023-10-16T12:24:03.282359+0000","last_became_peered":"2023-10-16T12:24:03.282359+0000","last_unstale":"2023-10-16T12:24:04.619490+0000","last_undegraded":"2023-10-16T12:24:04.619490+0000","last_fullsized":"2023-10-16T12:24:04.619490+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:38:55.619256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.621147+0000","last_change":"2023-10-16T12:24:02.278079+0000","last_active":"2023-10-16T12:24:04.621147+0000","last_peered":"2023-10-16T12:24:04.621147+0000","last_clean":"2023-10-16T12:24:04.621147+0000","last_became_active":"2023-10-16T12:24:02.277761+0000","last_became_peered":"2023-10-16T12:24:02.277761+0000","last_unstale":"2023-10-16T12:24:04.621147+0000","last_undegraded":"2023-10-16T12:24:04.621147+0000","last_fullsized":"2023-10-16T12:24:04.621147+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_clean_scrub_stamp":"2023-10-16T12:24:01.254337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:44:09.474584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619686+0000","last_change":"2023-10-16T12:24:03.291531+0000","last_active":"2023-10-16T12:24:04.619686+0000","last_peered":"2023-10-16T12:24:04.619686+0000","last_clean":"2023-10-16T12:24:04.619686+0000","last_became_active":"2023-10-16T12:24:03.291228+0000","last_became_peered":"2023-10-16T12:24:03.291228+0000","last_unstale":"2023-10-16T12:24:04.619686+0000","last_undegraded":"2023-10-16T12:24:04.619686+0000","last_fullsized":"2023-10-16T12:24:04.619686+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:35:43.782093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"67'3","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.620095+0000","last_change":"2023-10-16T12:24:02.284853+0000","last_active":"2023-10-16T12:24:04.620095+0000","last_peered":"2023-10-16T12:24:04.620095+0000","last_clean":"2023-10-16T12:24:04.620095+0000","last_became_active":"2023-10-16T12:24:02.284438+0000","last_became_peered":"2023-10-16T12:24:02.284438+0000","last_unstale":"2023-10-16T12:24:04.620095+0000","last_undegraded":"2023-10-16T12:24:04.620095+0000","last_fullsized":"2023-10-16T12:24:04.620095+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_clean_scrub_stamp":"2023-10-16T12:24:01.254337+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:50:44.843718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.621005+0000","last_change":"2023-10-16T12:24:03.287948+0000","last_active":"2023-10-16T12:24:04.621005+0000","last_peered":"2023-10-16T12:24:04.621005+0000","last_clean":"2023-10-16T12:24:04.621005+0000","last_became_active":"2023-10-16T12:24:03.287637+0000","last_became_peered":"2023-10-16T12:24:03.287637+0000","last_unstale":"2023-10-16T12:24:04.621005+0000","last_undegraded":"2023-10-16T12:24:04.621005+0000","last_fullsized":"2023-10-16T12:24:04.621005+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:53:46.741709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"67'6","reported_seq":18,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619098+0000","last_change":"2023-10-16T12:24:02.288074+0000","last_active":"2023-10-16T12:24:04.619098+0000","last_peered":"2023-10-16T12:24:04.619098+0000","last_clean":"2023-10-16T12:24:04.619098+0000","last_became_active":"2023-10-16T12:24:02.285745+0000","last_became_peered":"2023-10-16T12:24:02.285745+0000","last_unstale":"2023-10-16T12:24:04.619098+0000","last_undegraded":"2023-10-16T12:24:04.619098+0000","last_fullsized":"2023-10-16T12:24:04.619098+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_clean_scrub_stamp":"2023-10-16T12:24:01.254337+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:35:44.718016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619449+0000","last_change":"2023-10-16T12:24:03.283374+0000","last_active":"2023-10-16T12:24:04.619449+0000","last_peered":"2023-10-16T12:24:04.619449+0000","last_clean":"2023-10-16T12:24:04.619449+0000","last_became_active":"2023-10-16T12:24:03.283035+0000","last_became_peered":"2023-10-16T12:24:03.283035+0000","last_unstale":"2023-10-16T12:24:04.619449+0000","last_undegraded":"2023-10-16T12:24:04.619449+0000","last_fullsized":"2023-10-16T12:24:04.619449+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:22:10.364603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"67'2","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619678+0000","last_change":"2023-10-16T12:24:02.278013+0000","last_active":"2023-10-16T12:24:04.619678+0000","last_peered":"2023-10-16T12:24:04.619678+0000","last_clean":"2023-10-16T12:24:04.619678+0000","last_became_active":"2023-10-16T12:24:02.277687+0000","last_became_peered":"2023-10-16T12:24:02.277687+0000","last_unstale":"2023-10-16T12:24:04.619678+0000","last_undegraded":"2023-10-16T12:24:04.619678+0000","last_fullsized":"2023-10-16T12:24:04.619678+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_clean_scrub_stamp":"2023-10-16T12:24:01.254337+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:46:03.012648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619996+0000","last_change":"2023-10-16T12:24:03.294356+0000","last_active":"2023-10-16T12:24:04.619996+0000","last_peered":"2023-10-16T12:24:04.619996+0000","last_clean":"2023-10-16T12:24:04.619996+0000","last_became_active":"2023-10-16T12:24:03.291626+0000","last_became_peered":"2023-10-16T12:24:03.291626+0000","last_unstale":"2023-10-16T12:24:04.619996+0000","last_undegraded":"2023-10-16T12:24:04.619996+0000","last_fullsized":"2023-10-16T12:24:04.619996+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:13:00.102553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.285540+0000","last_change":"2023-10-16T12:24:02.284975+0000","last_active":"2023-10-16T12:24:04.285540+0000","last_peered":"2023-10-16T12:24:04.285540+0000","last_clean":"2023-10-16T12:24:04.285540+0000","last_became_active":"2023-10-16T12:24:02.284577+0000","last_became_peered":"2023-10-16T12:24:02.284577+0000","last_unstale":"2023-10-16T12:24:04.285540+0000","last_undegraded":"2023-10-16T12:24:04.285540+0000","last_fullsized":"2023-10-16T12:24:04.285540+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_clean_scrub_stamp":"2023-10-16T12:24:01.254337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:47:13.219274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619125+0000","last_change":"2023-10-16T12:24:03.283843+0000","last_active":"2023-10-16T12:24:04.619125+0000","last_peered":"2023-10-16T12:24:04.619125+0000","last_clean":"2023-10-16T12:24:04.619125+0000","last_became_active":"2023-10-16T12:24:03.283531+0000","last_became_peered":"2023-10-16T12:24:03.283531+0000","last_unstale":"2023-10-16T12:24:04.619125+0000","last_undegraded":"2023-10-16T12:24:04.619125+0000","last_fullsized":"2023-10-16T12:24:04.619125+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:29:28.797038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619199+0000","last_change":"2023-10-16T12:24:02.285342+0000","last_active":"2023-10-16T12:24:04.619199+0000","last_peered":"2023-10-16T12:24:04.619199+0000","last_clean":"2023-10-16T12:24:04.619199+0000","last_became_active":"2023-10-16T12:24:02.285024+0000","last_became_peered":"2023-10-16T12:24:02.285024+0000","last_unstale":"2023-10-16T12:24:04.619199+0000","last_undegraded":"2023-10-16T12:24:04.619199+0000","last_fullsized":"2023-10-16T12:24:04.619199+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_clean_scrub_stamp":"2023-10-16T12:24:01.254337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:29:25.193985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.620299+0000","last_change":"2023-10-16T12:24:03.278722+0000","last_active":"2023-10-16T12:24:04.620299+0000","last_peered":"2023-10-16T12:24:04.620299+0000","last_clean":"2023-10-16T12:24:04.620299+0000","last_became_active":"2023-10-16T12:24:03.278416+0000","last_became_peered":"2023-10-16T12:24:03.278416+0000","last_unstale":"2023-10-16T12:24:04.620299+0000","last_undegraded":"2023-10-16T12:24:04.620299+0000","last_fullsized":"2023-10-16T12:24:04.620299+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:17:41.719369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"67'8","reported_seq":20,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.620216+0000","last_change":"2023-10-16T12:24:02.288564+0000","last_active":"2023-10-16T12:24:04.620216+0000","last_peered":"2023-10-16T12:24:04.620216+0000","last_clean":"2023-10-16T12:24:04.620216+0000","last_became_active":"2023-10-16T12:24:02.288271+0000","last_became_peered":"2023-10-16T12:24:02.288271+0000","last_unstale":"2023-10-16T12:24:04.620216+0000","last_undegraded":"2023-10-16T12:24:04.620216+0000","last_fullsized":"2023-10-16T12:24:04.620216+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_clean_scrub_stamp":"2023-10-16T12:24:01.254337+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:35:42.778355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.620039+0000","last_change":"2023-10-16T12:24:03.286609+0000","last_active":"2023-10-16T12:24:04.620039+0000","last_peered":"2023-10-16T12:24:04.620039+0000","last_clean":"2023-10-16T12:24:04.620039+0000","last_became_active":"2023-10-16T12:24:03.285733+0000","last_became_peered":"2023-10-16T12:24:03.285733+0000","last_unstale":"2023-10-16T12:24:04.620039+0000","last_undegraded":"2023-10-16T12:24:04.620039+0000","last_fullsized":"2023-10-16T12:24:04.620039+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:29:38.700767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"67'1","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.620152+0000","last_change":"2023-10-16T12:24:02.285817+0000","last_active":"2023-10-16T12:24:04.620152+0000","last_peered":"2023-10-16T12:24:04.620152+0000","last_clean":"2023-10-16T12:24:04.620152+0000","last_became_active":"2023-10-16T12:24:02.285507+0000","last_became_peered":"2023-10-16T12:24:02.285507+0000","last_unstale":"2023-10-16T12:24:04.620152+0000","last_undegraded":"2023-10-16T12:24:04.620152+0000","last_fullsized":"2023-10-16T12:24:04.620152+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_clean_scrub_stamp":"2023-10-16T12:24:01.254337+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:49:19.404409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.618777+0000","last_change":"2023-10-16T12:24:03.300237+0000","last_active":"2023-10-16T12:24:04.618777+0000","last_peered":"2023-10-16T12:24:04.618777+0000","last_clean":"2023-10-16T12:24:04.618777+0000","last_became_active":"2023-10-16T12:24:03.299944+0000","last_became_peered":"2023-10-16T12:24:03.299944+0000","last_unstale":"2023-10-16T12:24:04.618777+0000","last_undegraded":"2023-10-16T12:24:04.618777+0000","last_fullsized":"2023-10-16T12:24:04.618777+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:46:02.349889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"67'2","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.284544+0000","last_change":"2023-10-16T12:24:02.282861+0000","last_active":"2023-10-16T12:24:04.284544+0000","last_peered":"2023-10-16T12:24:04.284544+0000","last_clean":"2023-10-16T12:24:04.284544+0000","last_became_active":"2023-10-16T12:24:02.280447+0000","last_became_peered":"2023-10-16T12:24:02.280447+0000","last_unstale":"2023-10-16T12:24:04.284544+0000","last_undegraded":"2023-10-16T12:24:04.284544+0000","last_fullsized":"2023-10-16T12:24:04.284544+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_clean_scrub_stamp":"2023-10-16T12:24:01.254337+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:47:24.974456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.620691+0000","last_change":"2023-10-16T12:24:03.287961+0000","last_active":"2023-10-16T12:24:04.620691+0000","last_peered":"2023-10-16T12:24:04.620691+0000","last_clean":"2023-10-16T12:24:04.620691+0000","last_became_active":"2023-10-16T12:24:03.287623+0000","last_became_peered":"2023-10-16T12:24:03.287623+0000","last_unstale":"2023-10-16T12:24:04.620691+0000","last_undegraded":"2023-10-16T12:24:04.620691+0000","last_fullsized":"2023-10-16T12:24:04.620691+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:39:29.404835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"67'1","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.285717+0000","last_change":"2023-10-16T12:24:02.292400+0000","last_active":"2023-10-16T12:24:04.285717+0000","last_peered":"2023-10-16T12:24:04.285717+0000","last_clean":"2023-10-16T12:24:04.285717+0000","last_became_active":"2023-10-16T12:24:02.292091+0000","last_became_peered":"2023-10-16T12:24:02.292091+0000","last_unstale":"2023-10-16T12:24:04.285717+0000","last_undegraded":"2023-10-16T12:24:04.285717+0000","last_fullsized":"2023-10-16T12:24:04.285717+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_clean_scrub_stamp":"2023-10-16T12:24:01.254337+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:45:21.707886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619749+0000","last_change":"2023-10-16T12:24:03.290136+0000","last_active":"2023-10-16T12:24:04.619749+0000","last_peered":"2023-10-16T12:24:04.619749+0000","last_clean":"2023-10-16T12:24:04.619749+0000","last_became_active":"2023-10-16T12:24:03.289843+0000","last_became_peered":"2023-10-16T12:24:03.289843+0000","last_unstale":"2023-10-16T12:24:04.619749+0000","last_undegraded":"2023-10-16T12:24:04.619749+0000","last_fullsized":"2023-10-16T12:24:04.619749+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:39:29.404835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.285926+0000","last_change":"2023-10-16T12:24:02.286330+0000","last_active":"2023-10-16T12:24:04.285926+0000","last_peered":"2023-10-16T12:24:04.285926+0000","last_clean":"2023-10-16T12:24:04.285926+0000","last_became_active":"2023-10-16T12:24:02.285890+0000","last_became_peered":"2023-10-16T12:24:02.285890+0000","last_unstale":"2023-10-16T12:24:04.285926+0000","last_undegraded":"2023-10-16T12:24:04.285926+0000","last_fullsized":"2023-10-16T12:24:04.285926+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_clean_scrub_stamp":"2023-10-16T12:24:01.254337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:20:10.698580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619243+0000","last_change":"2023-10-16T12:24:03.289934+0000","last_active":"2023-10-16T12:24:04.619243+0000","last_peered":"2023-10-16T12:24:04.619243+0000","last_clean":"2023-10-16T12:24:04.619243+0000","last_became_active":"2023-10-16T12:24:03.289537+0000","last_became_peered":"2023-10-16T12:24:03.289537+0000","last_unstale":"2023-10-16T12:24:04.619243+0000","last_undegraded":"2023-10-16T12:24:04.619243+0000","last_fullsized":"2023-10-16T12:24:04.619243+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:20:27.553468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"67'2","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619850+0000","last_change":"2023-10-16T12:24:02.283343+0000","last_active":"2023-10-16T12:24:04.619850+0000","last_peered":"2023-10-16T12:24:04.619850+0000","last_clean":"2023-10-16T12:24:04.619850+0000","last_became_active":"2023-10-16T12:24:02.283027+0000","last_became_peered":"2023-10-16T12:24:02.283027+0000","last_unstale":"2023-10-16T12:24:04.619850+0000","last_undegraded":"2023-10-16T12:24:04.619850+0000","last_fullsized":"2023-10-16T12:24:04.619850+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_clean_scrub_stamp":"2023-10-16T12:24:01.254337+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:45:43.529561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.285623+0000","last_change":"2023-10-16T12:24:03.295797+0000","last_active":"2023-10-16T12:24:04.285623+0000","last_peered":"2023-10-16T12:24:04.285623+0000","last_clean":"2023-10-16T12:24:04.285623+0000","last_became_active":"2023-10-16T12:24:03.295515+0000","last_became_peered":"2023-10-16T12:24:03.295515+0000","last_unstale":"2023-10-16T12:24:04.285623+0000","last_undegraded":"2023-10-16T12:24:04.285623+0000","last_fullsized":"2023-10-16T12:24:04.285623+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:45:44.533299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.618847+0000","last_change":"2023-10-16T12:24:02.288599+0000","last_active":"2023-10-16T12:24:04.618847+0000","last_peered":"2023-10-16T12:24:04.618847+0000","last_clean":"2023-10-16T12:24:04.618847+0000","last_became_active":"2023-10-16T12:24:02.288292+0000","last_became_peered":"2023-10-16T12:24:02.288292+0000","last_unstale":"2023-10-16T12:24:04.618847+0000","last_undegraded":"2023-10-16T12:24:04.618847+0000","last_fullsized":"2023-10-16T12:24:04.618847+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_clean_scrub_stamp":"2023-10-16T12:24:01.254337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:55:38.957320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619944+0000","last_change":"2023-10-16T12:24:03.289842+0000","last_active":"2023-10-16T12:24:04.619944+0000","last_peered":"2023-10-16T12:24:04.619944+0000","last_clean":"2023-10-16T12:24:04.619944+0000","last_became_active":"2023-10-16T12:24:03.289551+0000","last_became_peered":"2023-10-16T12:24:03.289551+0000","last_unstale":"2023-10-16T12:24:04.619944+0000","last_undegraded":"2023-10-16T12:24:04.619944+0000","last_fullsized":"2023-10-16T12:24:04.619944+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:35:45.721754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619592+0000","last_change":"2023-10-16T12:24:02.289353+0000","last_active":"2023-10-16T12:24:04.619592+0000","last_peered":"2023-10-16T12:24:04.619592+0000","last_clean":"2023-10-16T12:24:04.619592+0000","last_became_active":"2023-10-16T12:24:02.289061+0000","last_became_peered":"2023-10-16T12:24:02.289061+0000","last_unstale":"2023-10-16T12:24:04.619592+0000","last_undegraded":"2023-10-16T12:24:04.619592+0000","last_fullsized":"2023-10-16T12:24:04.619592+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_clean_scrub_stamp":"2023-10-16T12:24:01.254337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:29:25.193985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619273+0000","last_change":"2023-10-16T12:24:03.288935+0000","last_active":"2023-10-16T12:24:04.619273+0000","last_peered":"2023-10-16T12:24:04.619273+0000","last_clean":"2023-10-16T12:24:04.619273+0000","last_became_active":"2023-10-16T12:24:03.288478+0000","last_became_peered":"2023-10-16T12:24:03.288478+0000","last_unstale":"2023-10-16T12:24:04.619273+0000","last_undegraded":"2023-10-16T12:24:04.619273+0000","last_fullsized":"2023-10-16T12:24:04.619273+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:13:47.577433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.284201+0000","last_change":"2023-10-16T12:24:02.274586+0000","last_active":"2023-10-16T12:24:04.284201+0000","last_peered":"2023-10-16T12:24:04.284201+0000","last_clean":"2023-10-16T12:24:04.284201+0000","last_became_active":"2023-10-16T12:24:02.274278+0000","last_became_peered":"2023-10-16T12:24:02.274278+0000","last_unstale":"2023-10-16T12:24:04.284201+0000","last_undegraded":"2023-10-16T12:24:04.284201+0000","last_fullsized":"2023-10-16T12:24:04.284201+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_clean_scrub_stamp":"2023-10-16T12:24:01.254337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:44:43.398711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619022+0000","last_change":"2023-10-16T12:24:03.289984+0000","last_active":"2023-10-16T12:24:04.619022+0000","last_peered":"2023-10-16T12:24:04.619022+0000","last_clean":"2023-10-16T12:24:04.619022+0000","last_became_active":"2023-10-16T12:24:03.289140+0000","last_became_peered":"2023-10-16T12:24:03.289140+0000","last_unstale":"2023-10-16T12:24:04.619022+0000","last_undegraded":"2023-10-16T12:24:04.619022+0000","last_fullsized":"2023-10-16T12:24:04.619022+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:35:43.093148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"67'2","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.285602+0000","last_change":"2023-10-16T12:24:02.284553+0000","last_active":"2023-10-16T12:24:04.285602+0000","last_peered":"2023-10-16T12:24:04.285602+0000","last_clean":"2023-10-16T12:24:04.285602+0000","last_became_active":"2023-10-16T12:24:02.284261+0000","last_became_peered":"2023-10-16T12:24:02.284261+0000","last_unstale":"2023-10-16T12:24:04.285602+0000","last_undegraded":"2023-10-16T12:24:04.285602+0000","last_fullsized":"2023-10-16T12:24:04.285602+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_clean_scrub_stamp":"2023-10-16T12:24:01.254337+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:04:01.752804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.618867+0000","last_change":"2023-10-16T12:24:03.285078+0000","last_active":"2023-10-16T12:24:04.618867+0000","last_peered":"2023-10-16T12:24:04.618867+0000","last_clean":"2023-10-16T12:24:04.618867+0000","last_became_active":"2023-10-16T12:24:03.284779+0000","last_became_peered":"2023-10-16T12:24:03.284779+0000","last_unstale":"2023-10-16T12:24:04.618867+0000","last_undegraded":"2023-10-16T12:24:04.618867+0000","last_fullsized":"2023-10-16T12:24:04.618867+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:12:34.419931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.620121+0000","last_change":"2023-10-16T12:24:02.277744+0000","last_active":"2023-10-16T12:24:04.620121+0000","last_peered":"2023-10-16T12:24:04.620121+0000","last_clean":"2023-10-16T12:24:04.620121+0000","last_became_active":"2023-10-16T12:24:02.277436+0000","last_became_peered":"2023-10-16T12:24:02.277436+0000","last_unstale":"2023-10-16T12:24:04.620121+0000","last_undegraded":"2023-10-16T12:24:04.620121+0000","last_fullsized":"2023-10-16T12:24:04.620121+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_clean_scrub_stamp":"2023-10-16T12:24:01.254337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:52:27.639535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619793+0000","last_change":"2023-10-16T12:24:03.281663+0000","last_active":"2023-10-16T12:24:04.619793+0000","last_peered":"2023-10-16T12:24:04.619793+0000","last_clean":"2023-10-16T12:24:04.619793+0000","last_became_active":"2023-10-16T12:24:03.281228+0000","last_became_peered":"2023-10-16T12:24:03.281228+0000","last_unstale":"2023-10-16T12:24:04.619793+0000","last_undegraded":"2023-10-16T12:24:04.619793+0000","last_fullsized":"2023-10-16T12:24:04.619793+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:05:25.768970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619724+0000","last_change":"2023-10-16T12:24:02.285672+0000","last_active":"2023-10-16T12:24:04.619724+0000","last_peered":"2023-10-16T12:24:04.619724+0000","last_clean":"2023-10-16T12:24:04.619724+0000","last_became_active":"2023-10-16T12:24:02.285337+0000","last_became_peered":"2023-10-16T12:24:02.285337+0000","last_unstale":"2023-10-16T12:24:04.619724+0000","last_undegraded":"2023-10-16T12:24:04.619724+0000","last_fullsized":"2023-10-16T12:24:04.619724+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_clean_scrub_stamp":"2023-10-16T12:24:01.254337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:35:48.332026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619518+0000","last_change":"2023-10-16T12:24:03.290847+0000","last_active":"2023-10-16T12:24:04.619518+0000","last_peered":"2023-10-16T12:24:04.619518+0000","last_clean":"2023-10-16T12:24:04.619518+0000","last_became_active":"2023-10-16T12:24:03.290512+0000","last_became_peered":"2023-10-16T12:24:03.290512+0000","last_unstale":"2023-10-16T12:24:04.619518+0000","last_undegraded":"2023-10-16T12:24:04.619518+0000","last_fullsized":"2023-10-16T12:24:04.619518+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:22:10.364603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619891+0000","last_change":"2023-10-16T12:24:02.285124+0000","last_active":"2023-10-16T12:24:04.619891+0000","last_peered":"2023-10-16T12:24:04.619891+0000","last_clean":"2023-10-16T12:24:04.619891+0000","last_became_active":"2023-10-16T12:24:02.284841+0000","last_became_peered":"2023-10-16T12:24:02.284841+0000","last_unstale":"2023-10-16T12:24:04.619891+0000","last_undegraded":"2023-10-16T12:24:04.619891+0000","last_fullsized":"2023-10-16T12:24:04.619891+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_clean_scrub_stamp":"2023-10-16T12:24:01.254337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:38:54.615518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619529+0000","last_change":"2023-10-16T12:24:03.283921+0000","last_active":"2023-10-16T12:24:04.619529+0000","last_peered":"2023-10-16T12:24:04.619529+0000","last_clean":"2023-10-16T12:24:04.619529+0000","last_became_active":"2023-10-16T12:24:03.282680+0000","last_became_peered":"2023-10-16T12:24:03.282680+0000","last_unstale":"2023-10-16T12:24:04.619529+0000","last_undegraded":"2023-10-16T12:24:04.619529+0000","last_fullsized":"2023-10-16T12:24:04.619529+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:35:43.782093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.618950+0000","last_change":"2023-10-16T12:24:02.291490+0000","last_active":"2023-10-16T12:24:04.618950+0000","last_peered":"2023-10-16T12:24:04.618950+0000","last_clean":"2023-10-16T12:24:04.618950+0000","last_became_active":"2023-10-16T12:24:02.291245+0000","last_became_peered":"2023-10-16T12:24:02.291245+0000","last_unstale":"2023-10-16T12:24:04.618950+0000","last_undegraded":"2023-10-16T12:24:04.618950+0000","last_fullsized":"2023-10-16T12:24:04.618950+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_clean_scrub_stamp":"2023-10-16T12:24:01.254337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:12:59.098815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619849+0000","last_change":"2023-10-16T12:24:03.295916+0000","last_active":"2023-10-16T12:24:04.619849+0000","last_peered":"2023-10-16T12:24:04.619849+0000","last_clean":"2023-10-16T12:24:04.619849+0000","last_became_active":"2023-10-16T12:24:03.292053+0000","last_became_peered":"2023-10-16T12:24:03.292053+0000","last_unstale":"2023-10-16T12:24:04.619849+0000","last_undegraded":"2023-10-16T12:24:04.619849+0000","last_fullsized":"2023-10-16T12:24:04.619849+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:55:39.961058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.620604+0000","last_change":"2023-10-16T12:24:02.279222+0000","last_active":"2023-10-16T12:24:04.620604+0000","last_peered":"2023-10-16T12:24:04.620604+0000","last_clean":"2023-10-16T12:24:04.620604+0000","last_became_active":"2023-10-16T12:24:02.278927+0000","last_became_peered":"2023-10-16T12:24:02.278927+0000","last_unstale":"2023-10-16T12:24:04.620604+0000","last_undegraded":"2023-10-16T12:24:04.620604+0000","last_fullsized":"2023-10-16T12:24:04.620604+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_clean_scrub_stamp":"2023-10-16T12:24:01.254337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:13:46.573695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619437+0000","last_change":"2023-10-16T12:24:03.284059+0000","last_active":"2023-10-16T12:24:04.619437+0000","last_peered":"2023-10-16T12:24:04.619437+0000","last_clean":"2023-10-16T12:24:04.619437+0000","last_became_active":"2023-10-16T12:24:03.281887+0000","last_became_peered":"2023-10-16T12:24:03.281887+0000","last_unstale":"2023-10-16T12:24:04.619437+0000","last_undegraded":"2023-10-16T12:24:04.619437+0000","last_fullsized":"2023-10-16T12:24:04.619437+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:17:58.552410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619055+0000","last_change":"2023-10-16T12:24:02.286415+0000","last_active":"2023-10-16T12:24:04.619055+0000","last_peered":"2023-10-16T12:24:04.619055+0000","last_clean":"2023-10-16T12:24:04.619055+0000","last_became_active":"2023-10-16T12:24:02.286073+0000","last_became_peered":"2023-10-16T12:24:02.286073+0000","last_unstale":"2023-10-16T12:24:04.619055+0000","last_undegraded":"2023-10-16T12:24:04.619055+0000","last_fullsized":"2023-10-16T12:24:04.619055+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_clean_scrub_stamp":"2023-10-16T12:24:01.254337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:19:27.299060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.285553+0000","last_change":"2023-10-16T12:24:03.291580+0000","last_active":"2023-10-16T12:24:04.285553+0000","last_peered":"2023-10-16T12:24:04.285553+0000","last_clean":"2023-10-16T12:24:04.285553+0000","last_became_active":"2023-10-16T12:24:03.291216+0000","last_became_peered":"2023-10-16T12:24:03.291216+0000","last_unstale":"2023-10-16T12:24:04.285553+0000","last_undegraded":"2023-10-16T12:24:04.285553+0000","last_fullsized":"2023-10-16T12:24:04.285553+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:11:10.623582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619969+0000","last_change":"2023-10-16T12:24:02.278058+0000","last_active":"2023-10-16T12:24:04.619969+0000","last_peered":"2023-10-16T12:24:04.619969+0000","last_clean":"2023-10-16T12:24:04.619969+0000","last_became_active":"2023-10-16T12:24:02.277736+0000","last_became_peered":"2023-10-16T12:24:02.277736+0000","last_unstale":"2023-10-16T12:24:04.619969+0000","last_undegraded":"2023-10-16T12:24:04.619969+0000","last_fullsized":"2023-10-16T12:24:04.619969+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_clean_scrub_stamp":"2023-10-16T12:24:01.254337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:50:44.843718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"57'139","reported_seq":291,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:24:03.272411+0000","last_change":"2023-10-16T12:10:34.746820+0000","last_active":"2023-10-16T12:24:03.272411+0000","last_peered":"2023-10-16T12:24:03.272411+0000","last_clean":"2023-10-16T12:24:03.272411+0000","last_became_active":"2023-10-16T12:10:34.745574+0000","last_became_peered":"2023-10-16T12:10:34.745574+0000","last_unstale":"2023-10-16T12:24:03.272411+0000","last_undegraded":"2023-10-16T12:24:03.272411+0000","last_fullsized":"2023-10-16T12:24:03.272411+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_clean_scrub_stamp":"2023-10-16T12:10:33.577111+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:14.461986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575014,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58920,"kb_used_data":1372,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312920,"statfs":{"total":96636764160,"available":96576430080,"internally_reserved":0,"allocated":1404928,"data_stored":985030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58915275},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:23:57 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.60899999999999999,"15min":0.57399999999999995},"min":{"1min":0.313,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.72599999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.59399999999999997,"15min":0.57299999999999995},"min":{"1min":0.374,"5min":0.35299999999999998,"15min":0.312},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.49399999999999999}]},{"osd":1,"last update":"Mon Oct 16 12:23:57 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.61899999999999999,"15min":0.58999999999999997},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.247},"max":{"1min":0.76200000000000001,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.59799999999999998,"15min":0.58699999999999997},"min":{"1min":0.373,"5min":0.33200000000000002,"15min":0.32400000000000001},"max":{"1min":0.82999999999999996,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.58299999999999996}]},{"osd":2,"last update":"Mon Oct 16 12:23:57 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.67300000000000004,"15min":0.64000000000000001},"min":{"1min":0.41599999999999998,"5min":0.378,"15min":0.36099999999999999},"max":{"1min":0.81799999999999995,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.68799999999999994,"15min":0.65700000000000003},"min":{"1min":0.41099999999999998,"5min":0.376,"15min":0.36699999999999999},"max":{"1min":0.85899999999999999,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.47299999999999998}]},{"osd":3,"last update":"Mon Oct 16 12:23:57 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.624,"15min":0.61299999999999999},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.34899999999999998},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":1.1000000000000001},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.64300000000000002,"15min":0.63400000000000001},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.2509999999999999,"5min":1.2509999999999999,"15min":1.2509999999999999},"last":0.43099999999999999}]},{"osd":4,"last update":"Mon Oct 16 12:23:57 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.52900000000000003,"15min":0.54000000000000004},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.28499999999999998},"max":{"1min":1.383,"5min":1.383,"15min":1.383},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.629,"5min":0.61699999999999999,"15min":0.60999999999999999},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.307},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.54800000000000004}]},{"osd":5,"last update":"Mon Oct 16 12:23:57 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.53100000000000003,"15min":0.54700000000000004},"min":{"1min":0.314,"5min":0.314,"15min":0.29799999999999999},"max":{"1min":1.3109999999999999,"5min":1.3109999999999999,"15min":1.3109999999999999},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.32400000000000001},"max":{"1min":1.4359999999999999,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":0.56499999999999995}]},{"osd":6,"last update":"Mon Oct 16 12:23:57 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.628,"15min":0.624},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.36399999999999999},"max":{"1min":0.80800000000000005,"5min":0.91200000000000003,"15min":0.91400000000000003},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.63900000000000001,"15min":0.63200000000000001},"min":{"1min":0.46300000000000002,"5min":0.34599999999999997,"15min":0.32300000000000001},"max":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.59899999999999998}]}]},{"osd":6,"up_from":13,"seq":55834575012,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31984,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339856,"statfs":{"total":96636764160,"available":96604012544,"internally_reserved":0,"allocated":823296,"data_stored":401810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:23:51 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.61899999999999999,"15min":0.65500000000000003},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.26900000000000002},"max":{"1min":1.196,"5min":1.196,"15min":5.9349999999999996},"last":0.747},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.629,"15min":0.66900000000000004},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.997,"5min":0.997,"15min":5.9489999999999998},"last":0.75800000000000001}]},{"osd":1,"last update":"Mon Oct 16 12:23:51 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.57599999999999996,"15min":0.57299999999999995},"min":{"1min":0.32800000000000001,"5min":0.30499999999999999,"15min":0.29399999999999998},"max":{"1min":0.83899999999999997,"5min":0.875,"15min":0.875},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.60299999999999998,"15min":0.59499999999999997},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.56599999999999995}]},{"osd":2,"last update":"Mon Oct 16 12:23:51 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.59999999999999998,"15min":0.66700000000000004},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.34499999999999997},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":8.0489999999999995},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.60299999999999998,"15min":0.623},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":2.855},"last":0.70399999999999996}]},{"osd":3,"last update":"Mon Oct 16 12:23:51 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.63700000000000001,"15min":0.63300000000000001},"min":{"1min":0.38100000000000001,"5min":0.38,"15min":0.307},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":1.083},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.68899999999999995,"15min":0.67700000000000005},"min":{"1min":0.42799999999999999,"5min":0.40899999999999997,"15min":0.36599999999999999},"max":{"1min":1.3080000000000001,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.66600000000000004}]},{"osd":4,"last update":"Mon Oct 16 12:23:51 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52000000000000002,"15min":0.52200000000000002},"min":{"1min":0.372,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.83899999999999997,"5min":0.85599999999999998,"15min":0.88200000000000001},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.60699999999999998,"15min":0.59499999999999997},"min":{"1min":0.40300000000000002,"5min":0.318,"15min":0.30199999999999999},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.65000000000000002}]},{"osd":5,"last update":"Mon Oct 16 12:23:51 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.58799999999999997,"15min":0.58299999999999996},"min":{"1min":0.41699999999999998,"5min":0.34799999999999998,"15min":0.30099999999999999},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.63800000000000001,"15min":0.63100000000000001},"min":{"1min":0.38700000000000001,"5min":0.35399999999999998,"15min":0.311},"max":{"1min":1.3540000000000001,"5min":1.3540000000000001,"15min":1.3540000000000001},"last":0.63300000000000001}]},{"osd":7,"last update":"Mon Oct 16 12:23:51 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.59699999999999998,"15min":0.58199999999999996},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.63400000000000001,"15min":0.629},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.60199999999999998}]}]},{"osd":1,"up_from":13,"seq":55834575012,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31972,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339868,"statfs":{"total":96636764160,"available":96604024832,"internally_reserved":0,"allocated":811008,"data_stored":394698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":31914441},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:23:55 2023","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.38,"15min":0.377},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.186},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.66000000000000003},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.46899999999999997,"15min":0.45100000000000001},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.81699999999999995},"last":1.0189999999999999}]},{"osd":2,"last update":"Mon Oct 16 12:23:55 2023","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.439,"15min":0.42599999999999999},"min":{"1min":0.25,"5min":0.25,"15min":0.215},"max":{"1min":0.69199999999999995,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.47699999999999998,"15min":0.45400000000000001},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.24399999999999999},"max":{"1min":0.61199999999999999,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.67200000000000004}]},{"osd":3,"last update":"Mon Oct 16 12:23:55 2023","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.40899999999999997,"15min":0.39500000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.23599999999999999},"max":{"1min":0.53600000000000003,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.98799999999999999},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.45200000000000001,"15min":0.45100000000000001},"min":{"1min":0.252,"5min":0.252,"15min":0.22900000000000001},"max":{"1min":0.60499999999999998,"5min":0.73299999999999998,"15min":1.127},"last":0.91200000000000003}]},{"osd":4,"last update":"Mon Oct 16 12:23:55 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.64100000000000001,"15min":0.628},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.85499999999999998,"5min":0.93500000000000005,"15min":0.95599999999999996},"last":1.0409999999999999},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.66000000000000003,"15min":0.63500000000000001},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.34699999999999998},"max":{"1min":0.75600000000000001,"5min":0.98299999999999998,"15min":1.089},"last":0.48699999999999999}]},{"osd":5,"last update":"Mon Oct 16 12:23:55 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.68200000000000005,"15min":0.66900000000000004},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.28499999999999998},"max":{"1min":0.93400000000000005,"5min":1,"15min":1.052},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.69499999999999995,"15min":0.67800000000000005},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.32500000000000001},"max":{"1min":0.90300000000000002,"5min":0.96699999999999997,"15min":1.0649999999999999},"last":1.0620000000000001}]},{"osd":6,"last update":"Mon Oct 16 12:23:55 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.65900000000000003,"15min":0.65000000000000002},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.94999999999999996,"5min":1.0109999999999999,"15min":1.0129999999999999},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.67700000000000005,"15min":0.65600000000000003},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.80700000000000005,"5min":1.048,"15min":1.0609999999999999},"last":0.749}]},{"osd":7,"last update":"Mon Oct 16 12:23:55 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.70599999999999996,"15min":0.68300000000000005},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.82899999999999996,"5min":0.98499999999999999,"15min":1.1499999999999999},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.70999999999999996,"15min":0.69999999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.91700000000000004,"5min":1.02,"15min":1.167},"last":0.55200000000000005}]}]},{"osd":0,"up_from":13,"seq":55834575013,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32560,"kb_used_data":1372,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339280,"statfs":{"total":96636764160,"available":96603422720,"internally_reserved":0,"allocated":1404928,"data_stored":984962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 12:23:50 2023","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.379,"15min":0.377},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.55600000000000005,"5min":0.64200000000000002,"15min":0.73699999999999999},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.42099999999999999,"15min":0.40799999999999997},"min":{"1min":0.25800000000000001,"5min":0.24299999999999999,"15min":0.23200000000000001},"max":{"1min":0.54800000000000004,"5min":0.60799999999999998,"15min":0.73099999999999998},"last":0.34899999999999998}]},{"osd":2,"last update":"Mon Oct 16 12:23:50 2023","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.42399999999999999,"15min":0.41899999999999998},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.25800000000000001},"max":{"1min":0.55100000000000005,"5min":0.65600000000000003,"15min":0.67200000000000004},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.44,"5min":0.45200000000000001,"15min":0.44},"min":{"1min":0.29599999999999999,"5min":0.29499999999999998,"15min":0.27900000000000003},"max":{"1min":0.67600000000000005,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.307}]},{"osd":3,"last update":"Mon Oct 16 12:23:50 2023","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.42499999999999999,"15min":0.41199999999999998},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.23699999999999999},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.48199999999999998,"15min":0.47699999999999998},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.27600000000000002},"max":{"1min":0.63,"5min":0.878,"15min":0.878},"last":0.371}]},{"osd":4,"last update":"Mon Oct 16 12:23:50 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.64600000000000002,"15min":0.63900000000000001},"min":{"1min":0.39600000000000002,"5min":0.377,"15min":0.36699999999999999},"max":{"1min":0.80200000000000005,"5min":0.90700000000000003,"15min":1.0960000000000001},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.65700000000000003,"15min":0.64700000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.77200000000000002,"5min":0.94999999999999996,"15min":1.1719999999999999},"last":0.57799999999999996}]},{"osd":5,"last update":"Mon Oct 16 12:23:50 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.65300000000000002,"15min":0.63900000000000001},"min":{"1min":0.374,"5min":0.374,"15min":0.34399999999999997},"max":{"1min":0.90100000000000002,"5min":1,"15min":1.127},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.67000000000000004,"15min":0.67300000000000004},"min":{"1min":0.39200000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.82299999999999995,"5min":1.0069999999999999,"15min":1.1160000000000001},"last":0.58999999999999997}]},{"osd":6,"last update":"Mon Oct 16 12:23:50 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.68000000000000005,"15min":0.67500000000000004},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.751,"5min":1.0409999999999999,"15min":1.079},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.68999999999999995,"15min":0.68300000000000005},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.93600000000000005,"5min":0.98299999999999998,"15min":1.1499999999999999},"last":0.52000000000000002}]},{"osd":7,"last update":"Mon Oct 16 12:23:50 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.69599999999999995,"15min":0.68500000000000005},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.40699999999999997},"max":{"1min":0.88400000000000001,"5min":0.999,"15min":1.0349999999999999},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.627,"5min":0.69299999999999995,"15min":0.753},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41299999999999998},"max":{"1min":0.83799999999999997,"5min":1.0329999999999999,"15min":6.7300000000000004},"last":0.625}]}]},{"osd":2,"up_from":13,"seq":55834575013,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58360,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313480,"statfs":{"total":96636764160,"available":96577003520,"internally_reserved":0,"allocated":827392,"data_stored":402770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:23:49 2023","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.38700000000000001,"15min":0.376},"min":{"1min":0.222,"5min":0.218,"15min":0.21199999999999999},"max":{"1min":0.56999999999999995,"5min":0.60299999999999998,"15min":0.73799999999999999},"last":0.44},{"interface":"front","average":{"1min":0.441,"5min":0.45100000000000001,"15min":0.439},"min":{"1min":0.29899999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.53800000000000003}]},{"osd":1,"last update":"Mon Oct 16 12:23:49 2023","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.47399999999999998,"15min":0.45200000000000001},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.254},"max":{"1min":0.63700000000000001,"5min":0.68200000000000005,"15min":0.71199999999999997},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.44500000000000001,"15min":0.43099999999999999},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.246},"max":{"1min":0.63100000000000001,"5min":0.69699999999999995,"15min":0.76500000000000001},"last":0.502}]},{"osd":3,"last update":"Mon Oct 16 12:23:49 2023","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.41799999999999998,"15min":0.40699999999999997},"min":{"1min":0.27900000000000003,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.51600000000000001,"15min":0.505},"min":{"1min":0.318,"5min":0.318,"15min":0.246},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.36399999999999999}]},{"osd":4,"last update":"Mon Oct 16 12:23:49 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.67900000000000005,"15min":0.68100000000000005},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":1.3839999999999999},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.68999999999999995,"15min":0.68200000000000005},"min":{"1min":0.46400000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.72799999999999998}]},{"osd":5,"last update":"Mon Oct 16 12:23:49 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.67400000000000004,"15min":0.65300000000000002},"min":{"1min":0.40200000000000002,"5min":0.312,"15min":0.312},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":1.0800000000000001},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.66800000000000004,"15min":0.66000000000000003},"min":{"1min":0.41299999999999998,"5min":0.38600000000000001,"15min":0.317},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":1.331}]},{"osd":6,"last update":"Mon Oct 16 12:23:49 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.65500000000000003,"15min":0.64100000000000001},"min":{"1min":0.439,"5min":0.42499999999999999,"15min":0.34699999999999998},"max":{"1min":0.97399999999999998,"5min":1.036,"15min":1.046},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.628,"5min":0.68700000000000006,"15min":0.67300000000000004},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.004,"5min":1.004,"15min":1.0389999999999999},"last":0.81699999999999995}]},{"osd":7,"last update":"Mon Oct 16 12:23:49 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.93899999999999995,"5min":1.016,"15min":1.026},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.627,"5min":0.68899999999999995,"15min":0.68300000000000005},"min":{"1min":0.442,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.92100000000000004,"5min":0.96299999999999997,"15min":1.0509999999999999},"last":0.84899999999999998}]}]},{"osd":3,"up_from":13,"seq":55834575012,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36148,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335692,"statfs":{"total":96636764160,"available":96599748608,"internally_reserved":0,"allocated":819200,"data_stored":396172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:23:57 2023","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.40799999999999997,"15min":0.38300000000000001},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.216},"max":{"1min":0.60699999999999998,"5min":0.66900000000000004,"15min":0.67700000000000005},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.41599999999999998,"15min":0.39700000000000002},"min":{"1min":0.25900000000000001,"5min":0.255,"15min":0.22600000000000001},"max":{"1min":0.63100000000000001,"5min":0.64400000000000002,"15min":0.73099999999999998},"last":0.35399999999999998}]},{"osd":1,"last update":"Mon Oct 16 12:23:57 2023","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.42699999999999999,"15min":0.39900000000000002},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.53700000000000003,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.317},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.47299999999999998,"15min":0.44500000000000001},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.25700000000000001},"max":{"1min":0.65700000000000003,"5min":0.747,"15min":0.747},"last":0.47399999999999998}]},{"osd":2,"last update":"Mon Oct 16 12:23:57 2023","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.47499999999999998,"15min":0.45500000000000002},"min":{"1min":0.307,"5min":0.30499999999999999,"15min":0.26900000000000002},"max":{"1min":0.63500000000000001,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.51300000000000001,"15min":0.47899999999999998},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.27200000000000002},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.41699999999999998}]},{"osd":4,"last update":"Mon Oct 16 12:23:57 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.66100000000000003,"15min":0.63500000000000001},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.377},"max":{"1min":0.92700000000000005,"5min":1.026,"15min":1.1539999999999999},"last":0.438},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.66800000000000004,"15min":0.65800000000000003},"min":{"1min":0.44900000000000001,"5min":0.42999999999999999,"15min":0.40200000000000002},"max":{"1min":0.95699999999999996,"5min":1.0409999999999999,"15min":1.234},"last":0.45800000000000002}]},{"osd":5,"last update":"Mon Oct 16 12:23:57 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.69699999999999995,"15min":0.67200000000000004},"min":{"1min":0.443,"5min":0.443,"15min":0.41199999999999998},"max":{"1min":0.97199999999999998,"5min":0.996,"15min":1.194},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.68000000000000005,"15min":0.65800000000000003},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.89600000000000002,"5min":0.97399999999999998,"15min":1.202},"last":0.57299999999999995}]},{"osd":6,"last update":"Mon Oct 16 12:23:57 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.68700000000000006,"15min":0.66200000000000003},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.94499999999999995,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.72399999999999998,"15min":0.69499999999999995},"min":{"1min":0.51500000000000001,"5min":0.47399999999999998,"15min":0.42699999999999999},"max":{"1min":0.96899999999999997,"5min":1.0409999999999999,"15min":1.1559999999999999},"last":0.59499999999999997}]},{"osd":7,"last update":"Mon Oct 16 12:23:57 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.69999999999999996,"15min":0.68100000000000005},"min":{"1min":0.435,"5min":0.435,"15min":0.38},"max":{"1min":0.94199999999999995,"5min":1.014,"15min":1.2430000000000001},"last":0.623},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.72399999999999998,"15min":0.69699999999999995},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.42699999999999999},"max":{"1min":1.004,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.60899999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575011,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31980,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339860,"statfs":{"total":96636764160,"available":96604016640,"internally_reserved":0,"allocated":806912,"data_stored":395120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:23:56 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.59799999999999998,"15min":0.58399999999999996},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.79300000000000004,"5min":0.90200000000000002,"15min":1.171},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.60299999999999998,"15min":0.58399999999999996},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.69899999999999995}]},{"osd":1,"last update":"Mon Oct 16 12:23:56 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.61299999999999999,"15min":0.60399999999999998},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.28499999999999998},"max":{"1min":0.71999999999999997,"5min":0.89600000000000002,"15min":0.92600000000000005},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.625,"15min":0.61299999999999999},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.33600000000000002},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.71699999999999997}]},{"osd":2,"last update":"Mon Oct 16 12:23:56 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.625,"15min":0.61799999999999999},"min":{"1min":0.35399999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":1.012},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.629,"15min":0.621},"min":{"1min":0.38800000000000001,"5min":0.38600000000000001,"15min":0.34100000000000003},"max":{"1min":0.84699999999999998,"5min":0.873,"15min":0.873},"last":0.68700000000000006}]},{"osd":3,"last update":"Mon Oct 16 12:23:56 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.61599999999999999,"15min":0.61699999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.79500000000000004,"5min":0.878,"15min":1.3779999999999999},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.63800000000000001,"15min":0.64200000000000002},"min":{"1min":0.42099999999999999,"5min":0.39700000000000002,"15min":0.37},"max":{"1min":0.87,"5min":0.91900000000000004,"15min":0.97199999999999998},"last":0.61299999999999999}]},{"osd":5,"last update":"Mon Oct 16 12:23:56 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.49299999999999999,"15min":0.51500000000000001},"min":{"1min":0.34000000000000002,"5min":0.29799999999999999,"15min":0.27600000000000002},"max":{"1min":0.69499999999999995,"5min":0.75600000000000001,"15min":0.85899999999999999},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52300000000000002,"15min":0.54000000000000004},"min":{"1min":0.316,"5min":0.316,"15min":0.29599999999999999},"max":{"1min":0.74399999999999999,"5min":0.79100000000000004,"15min":0.94199999999999995},"last":0.65200000000000002}]},{"osd":6,"last update":"Mon Oct 16 12:23:56 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.55200000000000005,"15min":0.57599999999999996},"min":{"1min":0.39400000000000002,"5min":0.32500000000000001,"15min":0.28699999999999998},"max":{"1min":0.84499999999999997,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.63400000000000001,"15min":0.64700000000000002},"min":{"1min":0.47299999999999998,"5min":0.34799999999999998,"15min":0.32000000000000001},"max":{"1min":0.91300000000000003,"5min":0.91500000000000004,"15min":1.1180000000000001},"last":0.752}]},{"osd":7,"last update":"Mon Oct 16 12:23:56 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.58199999999999996,"15min":0.59399999999999997},"min":{"1min":0.379,"5min":0.372,"15min":0.34100000000000003},"max":{"1min":0.745,"5min":0.83599999999999997,"15min":0.89500000000000002},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.63100000000000001,"15min":0.64600000000000002},"min":{"1min":0.48199999999999998,"5min":0.38400000000000001,"15min":0.375},"max":{"1min":0.872,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.82099999999999995}]}]},{"osd":5,"up_from":13,"seq":55834575012,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54172,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317668,"statfs":{"total":96636764160,"available":96581292032,"internally_reserved":0,"allocated":802816,"data_stored":394538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:23:52 2023","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.59799999999999998,"15min":0.57999999999999996},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.82399999999999995,"5min":0.873,"15min":0.873},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.308,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.83899999999999997,"5min":0.86799999999999999,"15min":0.88800000000000001},"last":0.313}]},{"osd":1,"last update":"Mon Oct 16 12:23:52 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.61199999999999999,"15min":0.61099999999999999},"min":{"1min":0.32600000000000001,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.78900000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.622,"15min":0.61799999999999999},"min":{"1min":0.39200000000000002,"5min":0.32800000000000001,"15min":0.28799999999999998},"max":{"1min":0.85699999999999998,"5min":0.89000000000000001,"15min":0.95199999999999996},"last":0.35099999999999998}]},{"osd":2,"last update":"Mon Oct 16 12:23:52 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.63400000000000001,"15min":0.625},"min":{"1min":0.35099999999999998,"5min":0.29099999999999998,"15min":0.26400000000000001},"max":{"1min":0.83699999999999997,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.497},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.38200000000000001,"5min":0.378,"15min":0.36499999999999999},"max":{"1min":0.76800000000000002,"5min":0.84599999999999997,"15min":0.89700000000000002},"last":0.63900000000000001}]},{"osd":3,"last update":"Mon Oct 16 12:23:52 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.64600000000000002,"15min":0.64300000000000002},"min":{"1min":0.45800000000000002,"5min":0.371,"15min":0.36099999999999999},"max":{"1min":0.73799999999999999,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.65100000000000002,"15min":0.66200000000000003},"min":{"1min":0.47499999999999998,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.80700000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.59999999999999998}]},{"osd":4,"last update":"Mon Oct 16 12:23:52 2023","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.51500000000000001,"15min":0.52900000000000003},"min":{"1min":0.33100000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.70099999999999996,"5min":0.80200000000000005,"15min":1.0089999999999999},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.58099999999999996,"15min":0.58899999999999997},"min":{"1min":0.40699999999999997,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.347,"5min":1.347,"15min":1.347},"last":0.66000000000000003}]},{"osd":6,"last update":"Mon Oct 16 12:23:52 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.54400000000000004,"15min":0.57199999999999995},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.71099999999999997,"5min":0.93500000000000005,"15min":0.98699999999999999},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.60599999999999998,"15min":0.61399999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.35999999999999999},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.95999999999999996},"last":0.747}]},{"osd":7,"last update":"Mon Oct 16 12:23:52 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.622,"15min":0.63400000000000001},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.78100000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.65900000000000003,"15min":0.67400000000000004},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.73199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T12:24:07.441 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T12:24:07.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.598+0000 7f6c595af700 1 Processor -- start 2023-10-16T12:24:07.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.598+0000 7f6c595af700 1 -- start start 2023-10-16T12:24:07.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.598+0000 7f6c595af700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6c5406aef0 0x7f6c540cb980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:07.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.598+0000 7f6c595af700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6c540ce160 0x7f6c540ce610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:07.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.598+0000 7f6c595af700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c540cedb0 con 0x7f6c5415db40 2023-10-16T12:24:07.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.598+0000 7f6c595af700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c5415df20 con 0x7f6c540ce160 2023-10-16T12:24:07.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.598+0000 7f6c595af700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c540d12c0 con 0x7f6c5406aef0 2023-10-16T12:24:07.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.599+0000 7f6c537fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6c540ce160 0x7f6c540ce610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:07.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.599+0000 7f6c537fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6c540ce160 0x7f6c540ce610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37824/0 (socket says 172.21.15.16:37824) 2023-10-16T12:24:07.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.599+0000 7f6c537fe700 1 -- 172.21.15.16:0/760996408 learned_addr learned my addr 172.21.15.16:0/760996408 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:24:07.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.599+0000 7f6c527fc700 1 --2- 172.21.15.16:0/760996408 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6c5406aef0 0x7f6c540cb980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:07.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.599+0000 7f6c537fe700 1 -- 172.21.15.16:0/760996408 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6c5406aef0 msgr2=0x7f6c540cb980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:24:07.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.599+0000 7f6c537fe700 1 --2- 172.21.15.16:0/760996408 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6c5406aef0 0x7f6c540cb980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:07.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.599+0000 7f6c537fe700 1 -- 172.21.15.16:0/760996408 >> v1:172.21.15.16:6789/0 conn(0x7f6c5415db40 legacy=0x7f6c5406a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:07.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.599+0000 7f6c537fe700 1 -- 172.21.15.16:0/760996408 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c540ceb50 con 0x7f6c540ce160 2023-10-16T12:24:07.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.599+0000 7f6c51ffb700 1 -- 172.21.15.16:0/760996408 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 314635585 0 0) 0x7f6c540cedb0 con 0x7f6c5415db40 2023-10-16T12:24:07.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.599+0000 7f6c527fc700 1 --2- 172.21.15.16:0/760996408 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6c5406aef0 0x7f6c540cb980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:24:07.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.599+0000 7f6c537fe700 1 --2- 172.21.15.16:0/760996408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c540ce160 0x7f6c540ce610 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f6c4c009e50 tx=0x7f6c4c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b0a0d4715b3daf8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:24:07.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.599+0000 7f6c51ffb700 1 -- 172.21.15.16:0/760996408 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c4c00e070 con 0x7f6c540ce160 2023-10-16T12:24:07.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.599+0000 7f6c51ffb700 1 -- 172.21.15.16:0/760996408 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c4c00b9c0 con 0x7f6c540ce160 2023-10-16T12:24:07.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.599+0000 7f6c51ffb700 1 -- 172.21.15.16:0/760996408 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c4c00bd70 con 0x7f6c540ce160 2023-10-16T12:24:07.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.600+0000 7f6c595af700 1 -- 172.21.15.16:0/760996408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c540ce160 msgr2=0x7f6c540ce610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:07.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.600+0000 7f6c595af700 1 --2- 172.21.15.16:0/760996408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c540ce160 0x7f6c540ce610 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f6c4c009e50 tx=0x7f6c4c00b330 comp rx=0 tx=0).stop 2023-10-16T12:24:07.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.600+0000 7f6c595af700 1 -- 172.21.15.16:0/760996408 shutdown_connections 2023-10-16T12:24:07.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.600+0000 7f6c595af700 1 --2- 172.21.15.16:0/760996408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c540ce160 0x7f6c540ce610 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:07.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.600+0000 7f6c595af700 1 --2- 172.21.15.16:0/760996408 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6c5406aef0 0x7f6c540cb980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:07.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.600+0000 7f6c595af700 1 -- 172.21.15.16:0/760996408 >> 172.21.15.16:0/760996408 conn(0x7f6c540b1fd0 msgr2=0x7f6c540b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:24:07.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.600+0000 7f6c595af700 1 -- 172.21.15.16:0/760996408 shutdown_connections 2023-10-16T12:24:07.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.600+0000 7f6c595af700 1 -- 172.21.15.16:0/760996408 wait complete. 2023-10-16T12:24:07.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.601+0000 7f6c595af700 1 Processor -- start 2023-10-16T12:24:07.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.601+0000 7f6c595af700 1 -- start start 2023-10-16T12:24:07.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.602+0000 7f6c595af700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6c5406aef0 0x7f6c5414e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:07.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.602+0000 7f6c595af700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c540ce160 0x7f6c541e7320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:07.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.602+0000 7f6c595af700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6c5415db40 0x7f6c541e7860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:07.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.602+0000 7f6c595af700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c5415b100 con 0x7f6c540ce160 2023-10-16T12:24:07.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.602+0000 7f6c595af700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c540c4b80 con 0x7f6c5406aef0 2023-10-16T12:24:07.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.602+0000 7f6c595af700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c540c4e60 con 0x7f6c5415db40 2023-10-16T12:24:07.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.602+0000 7f6c527fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c540ce160 0x7f6c541e7320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:07.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.602+0000 7f6c527fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c540ce160 0x7f6c541e7320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37828/0 (socket says 172.21.15.16:37828) 2023-10-16T12:24:07.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.602+0000 7f6c527fc700 1 -- 172.21.15.16:0/2744648076 learned_addr learned my addr 172.21.15.16:0/2744648076 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:24:07.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.602+0000 7f6c537fe700 1 --2- 172.21.15.16:0/2744648076 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6c5415db40 0x7f6c541e7860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:07.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.602+0000 7f6c52ffd700 1 --2- 172.21.15.16:0/2744648076 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6c5406aef0 0x7f6c5414e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:07.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.602+0000 7f6c527fc700 1 -- 172.21.15.16:0/2744648076 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6c5415db40 msgr2=0x7f6c541e7860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:07.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.603+0000 7f6c527fc700 1 --2- 172.21.15.16:0/2744648076 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6c5415db40 0x7f6c541e7860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:07.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.603+0000 7f6c527fc700 1 -- 172.21.15.16:0/2744648076 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6c5406aef0 msgr2=0x7f6c5414e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:07.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.603+0000 7f6c527fc700 1 --2- 172.21.15.16:0/2744648076 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6c5406aef0 0x7f6c5414e790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:07.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.603+0000 7f6c527fc700 1 -- 172.21.15.16:0/2744648076 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c4c009b30 con 0x7f6c540ce160 2023-10-16T12:24:07.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.603+0000 7f6c537fe700 1 --2- 172.21.15.16:0/2744648076 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6c5415db40 0x7f6c541e7860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:24:07.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.603+0000 7f6c527fc700 1 --2- 172.21.15.16:0/2744648076 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c540ce160 0x7f6c541e7320 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f6c4800c7f0 tx=0x7f6c4800cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:24:07.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.603+0000 7f6c3b7fe700 1 -- 172.21.15.16:0/2744648076 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c48011800 con 0x7f6c540ce160 2023-10-16T12:24:07.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.603+0000 7f6c3b7fe700 1 -- 172.21.15.16:0/2744648076 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c48011960 con 0x7f6c540ce160 2023-10-16T12:24:07.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.603+0000 7f6c595af700 1 -- 172.21.15.16:0/2744648076 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c540c5140 con 0x7f6c540ce160 2023-10-16T12:24:07.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.603+0000 7f6c3b7fe700 1 -- 172.21.15.16:0/2744648076 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c48011d10 con 0x7f6c540ce160 2023-10-16T12:24:07.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.603+0000 7f6c595af700 1 -- 172.21.15.16:0/2744648076 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c540c5700 con 0x7f6c540ce160 2023-10-16T12:24:07.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.604+0000 7f6c595af700 1 -- 172.21.15.16:0/2744648076 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c540c5920 con 0x7f6c540ce160 2023-10-16T12:24:07.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.604+0000 7f6c3b7fe700 1 -- 172.21.15.16:0/2744648076 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6c48018360 con 0x7f6c540ce160 2023-10-16T12:24:07.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.605+0000 7f6c3b7fe700 1 --2- 172.21.15.16:0/2744648076 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f6c4006b020 0x7f6c4006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:07.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.605+0000 7f6c3b7fe700 1 -- 172.21.15.16:0/2744648076 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f6c480899a0 con 0x7f6c540ce160 2023-10-16T12:24:07.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.605+0000 7f6c52ffd700 1 --2- 172.21.15.16:0/2744648076 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f6c4006b020 0x7f6c4006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:07.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.605+0000 7f6c52ffd700 1 --2- 172.21.15.16:0/2744648076 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f6c4006b020 0x7f6c4006d4e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6c44000f60 tx=0x7f6c44009040 comp rx=0 tx=0).ready entity=mgr.5874 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:24:07.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.609+0000 7f6c3b7fe700 1 -- 172.21.15.16:0/2744648076 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6c480546f0 con 0x7f6c540ce160 2023-10-16T12:24:07.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.807+0000 7f6c595af700 1 -- 172.21.15.16:0/2744648076 --> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6c540c4fd0 con 0x7f6c4006b020 2023-10-16T12:24:07.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.815+0000 7f6c3b7fe700 1 -- 172.21.15.16:0/2744648076 <== mgr.5874 v2:172.21.15.16:6832/3870937344 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289204 (secure 0 0 0) 0x7f6c540c4fd0 con 0x7f6c4006b020 2023-10-16T12:24:07.820 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:24:07.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.818+0000 7f6c595af700 1 -- 172.21.15.16:0/2744648076 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f6c4006b020 msgr2=0x7f6c4006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:07.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.818+0000 7f6c595af700 1 --2- 172.21.15.16:0/2744648076 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f6c4006b020 0x7f6c4006d4e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6c44000f60 tx=0x7f6c44009040 comp rx=0 tx=0).stop 2023-10-16T12:24:07.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.818+0000 7f6c595af700 1 -- 172.21.15.16:0/2744648076 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c540ce160 msgr2=0x7f6c541e7320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:07.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.818+0000 7f6c595af700 1 --2- 172.21.15.16:0/2744648076 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c540ce160 0x7f6c541e7320 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f6c4800c7f0 tx=0x7f6c4800cbb0 comp rx=0 tx=0).stop 2023-10-16T12:24:07.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.818+0000 7f6c595af700 1 -- 172.21.15.16:0/2744648076 shutdown_connections 2023-10-16T12:24:07.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.818+0000 7f6c595af700 1 --2- 172.21.15.16:0/2744648076 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f6c4006b020 0x7f6c4006d4e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:07.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.818+0000 7f6c595af700 1 --2- 172.21.15.16:0/2744648076 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6c5415db40 0x7f6c541e7860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:07.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.818+0000 7f6c595af700 1 --2- 172.21.15.16:0/2744648076 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6c5406aef0 0x7f6c5414e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:07.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.818+0000 7f6c595af700 1 --2- 172.21.15.16:0/2744648076 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c540ce160 0x7f6c541e7320 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:07.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.818+0000 7f6c595af700 1 -- 172.21.15.16:0/2744648076 >> 172.21.15.16:0/2744648076 conn(0x7f6c540b1fd0 msgr2=0x7f6c5415b580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:24:07.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.818+0000 7f6c595af700 1 -- 172.21.15.16:0/2744648076 shutdown_connections 2023-10-16T12:24:07.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:07.819+0000 7f6c595af700 1 -- 172.21.15.16:0/2744648076 wait complete. 2023-10-16T12:24:07.825 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-10-16T12:24:07.833 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-10-16T12:24:06.861116+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":243,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":177,"ondisk_log_size":177,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":154,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":336096,"kb_used_data":7520,"kb_used_omap":12,"kb_used_meta":328435,"kb_avail":754638624,"statfs":{"total":773094113280,"available":772749950976,"internally_reserved":0,"allocated":7700480,"data_stored":4355100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12740,"internal_metadata":336318012},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.602838"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619129+0000","last_change":"2023-10-16T12:24:03.287328+0000","last_active":"2023-10-16T12:24:04.619129+0000","last_peered":"2023-10-16T12:24:04.619129+0000","last_clean":"2023-10-16T12:24:04.619129+0000","last_became_active":"2023-10-16T12:24:03.287047+0000","last_became_peered":"2023-10-16T12:24:03.287047+0000","last_unstale":"2023-10-16T12:24:04.619129+0000","last_undegraded":"2023-10-16T12:24:04.619129+0000","last_fullsized":"2023-10-16T12:24:04.619129+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:32:22.452723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.620012+0000","last_change":"2023-10-16T12:24:03.282048+0000","last_active":"2023-10-16T12:24:04.620012+0000","last_peered":"2023-10-16T12:24:04.620012+0000","last_clean":"2023-10-16T12:24:04.620012+0000","last_became_active":"2023-10-16T12:24:03.281058+0000","last_became_peered":"2023-10-16T12:24:03.281058+0000","last_unstale":"2023-10-16T12:24:04.620012+0000","last_undegraded":"2023-10-16T12:24:04.620012+0000","last_fullsized":"2023-10-16T12:24:04.620012+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:22:10.364603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619402+0000","last_change":"2023-10-16T12:24:03.282806+0000","last_active":"2023-10-16T12:24:04.619402+0000","last_peered":"2023-10-16T12:24:04.619402+0000","last_clean":"2023-10-16T12:24:04.619402+0000","last_became_active":"2023-10-16T12:24:03.281883+0000","last_became_peered":"2023-10-16T12:24:03.281883+0000","last_unstale":"2023-10-16T12:24:04.619402+0000","last_undegraded":"2023-10-16T12:24:04.619402+0000","last_fullsized":"2023-10-16T12:24:04.619402+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:53:46.533436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619939+0000","last_change":"2023-10-16T12:24:03.288881+0000","last_active":"2023-10-16T12:24:04.619939+0000","last_peered":"2023-10-16T12:24:04.619939+0000","last_clean":"2023-10-16T12:24:04.619939+0000","last_became_active":"2023-10-16T12:24:03.288601+0000","last_became_peered":"2023-10-16T12:24:03.288601+0000","last_unstale":"2023-10-16T12:24:04.619939+0000","last_undegraded":"2023-10-16T12:24:04.619939+0000","last_fullsized":"2023-10-16T12:24:04.619939+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:53:46.533436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.620874+0000","last_change":"2023-10-16T12:24:03.280171+0000","last_active":"2023-10-16T12:24:04.620874+0000","last_peered":"2023-10-16T12:24:04.620874+0000","last_clean":"2023-10-16T12:24:04.620874+0000","last_became_active":"2023-10-16T12:24:03.279833+0000","last_became_peered":"2023-10-16T12:24:03.279833+0000","last_unstale":"2023-10-16T12:24:04.620874+0000","last_undegraded":"2023-10-16T12:24:04.620874+0000","last_fullsized":"2023-10-16T12:24:04.620874+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:00:44.935777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.618748+0000","last_change":"2023-10-16T12:24:03.288544+0000","last_active":"2023-10-16T12:24:04.618748+0000","last_peered":"2023-10-16T12:24:04.618748+0000","last_clean":"2023-10-16T12:24:04.618748+0000","last_became_active":"2023-10-16T12:24:03.288182+0000","last_became_peered":"2023-10-16T12:24:03.288182+0000","last_unstale":"2023-10-16T12:24:04.618748+0000","last_undegraded":"2023-10-16T12:24:04.618748+0000","last_fullsized":"2023-10-16T12:24:04.618748+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:17:39.834485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619487+0000","last_change":"2023-10-16T12:24:03.296406+0000","last_active":"2023-10-16T12:24:04.619487+0000","last_peered":"2023-10-16T12:24:04.619487+0000","last_clean":"2023-10-16T12:24:04.619487+0000","last_became_active":"2023-10-16T12:24:03.292117+0000","last_became_peered":"2023-10-16T12:24:03.292117+0000","last_unstale":"2023-10-16T12:24:04.619487+0000","last_undegraded":"2023-10-16T12:24:04.619487+0000","last_fullsized":"2023-10-16T12:24:04.619487+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:20:27.553468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.620307+0000","last_change":"2023-10-16T12:24:03.289998+0000","last_active":"2023-10-16T12:24:04.620307+0000","last_peered":"2023-10-16T12:24:04.620307+0000","last_clean":"2023-10-16T12:24:04.620307+0000","last_became_active":"2023-10-16T12:24:03.286418+0000","last_became_peered":"2023-10-16T12:24:03.286418+0000","last_unstale":"2023-10-16T12:24:04.620307+0000","last_undegraded":"2023-10-16T12:24:04.620307+0000","last_fullsized":"2023-10-16T12:24:04.620307+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:27:33.449783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619290+0000","last_change":"2023-10-16T12:24:03.282172+0000","last_active":"2023-10-16T12:24:04.619290+0000","last_peered":"2023-10-16T12:24:04.619290+0000","last_clean":"2023-10-16T12:24:04.619290+0000","last_became_active":"2023-10-16T12:24:03.281533+0000","last_became_peered":"2023-10-16T12:24:03.281533+0000","last_unstale":"2023-10-16T12:24:04.619290+0000","last_undegraded":"2023-10-16T12:24:04.619290+0000","last_fullsized":"2023-10-16T12:24:04.619290+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:23:16.330749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619420+0000","last_change":"2023-10-16T12:24:03.294968+0000","last_active":"2023-10-16T12:24:04.619420+0000","last_peered":"2023-10-16T12:24:04.619420+0000","last_clean":"2023-10-16T12:24:04.619420+0000","last_became_active":"2023-10-16T12:24:03.294681+0000","last_became_peered":"2023-10-16T12:24:03.294681+0000","last_unstale":"2023-10-16T12:24:04.619420+0000","last_undegraded":"2023-10-16T12:24:04.619420+0000","last_fullsized":"2023-10-16T12:24:04.619420+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:44:10.111021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.284448+0000","last_change":"2023-10-16T12:24:03.282888+0000","last_active":"2023-10-16T12:24:04.284448+0000","last_peered":"2023-10-16T12:24:04.284448+0000","last_clean":"2023-10-16T12:24:04.284448+0000","last_became_active":"2023-10-16T12:24:03.282599+0000","last_became_peered":"2023-10-16T12:24:03.282599+0000","last_unstale":"2023-10-16T12:24:04.284448+0000","last_undegraded":"2023-10-16T12:24:04.284448+0000","last_fullsized":"2023-10-16T12:24:04.284448+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:19.839274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619357+0000","last_change":"2023-10-16T12:24:03.291963+0000","last_active":"2023-10-16T12:24:04.619357+0000","last_peered":"2023-10-16T12:24:04.619357+0000","last_clean":"2023-10-16T12:24:04.619357+0000","last_became_active":"2023-10-16T12:24:03.291646+0000","last_became_peered":"2023-10-16T12:24:03.291646+0000","last_unstale":"2023-10-16T12:24:04.619357+0000","last_undegraded":"2023-10-16T12:24:04.619357+0000","last_fullsized":"2023-10-16T12:24:04.619357+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:10:24.643281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.618600+0000","last_change":"2023-10-16T12:24:03.294342+0000","last_active":"2023-10-16T12:24:04.618600+0000","last_peered":"2023-10-16T12:24:04.618600+0000","last_clean":"2023-10-16T12:24:04.618600+0000","last_became_active":"2023-10-16T12:24:03.293965+0000","last_became_peered":"2023-10-16T12:24:03.293965+0000","last_unstale":"2023-10-16T12:24:04.618600+0000","last_undegraded":"2023-10-16T12:24:04.618600+0000","last_fullsized":"2023-10-16T12:24:04.618600+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:29:08.023383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619390+0000","last_change":"2023-10-16T12:24:03.283959+0000","last_active":"2023-10-16T12:24:04.619390+0000","last_peered":"2023-10-16T12:24:04.619390+0000","last_clean":"2023-10-16T12:24:04.619390+0000","last_became_active":"2023-10-16T12:24:03.283638+0000","last_became_peered":"2023-10-16T12:24:03.283638+0000","last_unstale":"2023-10-16T12:24:04.619390+0000","last_undegraded":"2023-10-16T12:24:04.619390+0000","last_fullsized":"2023-10-16T12:24:04.619390+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:19:28.302798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619185+0000","last_change":"2023-10-16T12:24:03.293257+0000","last_active":"2023-10-16T12:24:04.619185+0000","last_peered":"2023-10-16T12:24:04.619185+0000","last_clean":"2023-10-16T12:24:04.619185+0000","last_became_active":"2023-10-16T12:24:03.292696+0000","last_became_peered":"2023-10-16T12:24:03.292696+0000","last_unstale":"2023-10-16T12:24:04.619185+0000","last_undegraded":"2023-10-16T12:24:04.619185+0000","last_fullsized":"2023-10-16T12:24:04.619185+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:19:28.302798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.620014+0000","last_change":"2023-10-16T12:24:03.294302+0000","last_active":"2023-10-16T12:24:04.620014+0000","last_peered":"2023-10-16T12:24:04.620014+0000","last_clean":"2023-10-16T12:24:04.620014+0000","last_became_active":"2023-10-16T12:24:03.294002+0000","last_became_peered":"2023-10-16T12:24:03.294002+0000","last_unstale":"2023-10-16T12:24:04.620014+0000","last_undegraded":"2023-10-16T12:24:04.620014+0000","last_fullsized":"2023-10-16T12:24:04.620014+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:00:44.935777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.620489+0000","last_change":"2023-10-16T12:24:03.279125+0000","last_active":"2023-10-16T12:24:04.620489+0000","last_peered":"2023-10-16T12:24:04.620489+0000","last_clean":"2023-10-16T12:24:04.620489+0000","last_became_active":"2023-10-16T12:24:03.278676+0000","last_became_peered":"2023-10-16T12:24:03.278676+0000","last_unstale":"2023-10-16T12:24:04.620489+0000","last_undegraded":"2023-10-16T12:24:04.620489+0000","last_fullsized":"2023-10-16T12:24:04.620489+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:10:24.643281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.620205+0000","last_change":"2023-10-16T12:24:03.282365+0000","last_active":"2023-10-16T12:24:04.620205+0000","last_peered":"2023-10-16T12:24:04.620205+0000","last_clean":"2023-10-16T12:24:04.620205+0000","last_became_active":"2023-10-16T12:24:03.282052+0000","last_became_peered":"2023-10-16T12:24:03.282052+0000","last_unstale":"2023-10-16T12:24:04.620205+0000","last_undegraded":"2023-10-16T12:24:04.620205+0000","last_fullsized":"2023-10-16T12:24:04.620205+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:35:43.782093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619586+0000","last_change":"2023-10-16T12:24:03.279495+0000","last_active":"2023-10-16T12:24:04.619586+0000","last_peered":"2023-10-16T12:24:04.619586+0000","last_clean":"2023-10-16T12:24:04.619586+0000","last_became_active":"2023-10-16T12:24:03.279196+0000","last_became_peered":"2023-10-16T12:24:03.279196+0000","last_unstale":"2023-10-16T12:24:04.619586+0000","last_undegraded":"2023-10-16T12:24:04.619586+0000","last_fullsized":"2023-10-16T12:24:04.619586+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:38:55.619256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.285527+0000","last_change":"2023-10-16T12:24:03.296079+0000","last_active":"2023-10-16T12:24:04.285527+0000","last_peered":"2023-10-16T12:24:04.285527+0000","last_clean":"2023-10-16T12:24:04.285527+0000","last_became_active":"2023-10-16T12:24:03.295803+0000","last_became_peered":"2023-10-16T12:24:03.295803+0000","last_unstale":"2023-10-16T12:24:04.285527+0000","last_undegraded":"2023-10-16T12:24:04.285527+0000","last_fullsized":"2023-10-16T12:24:04.285527+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:46:56.368712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.285659+0000","last_change":"2023-10-16T12:24:03.284801+0000","last_active":"2023-10-16T12:24:04.285659+0000","last_peered":"2023-10-16T12:24:04.285659+0000","last_clean":"2023-10-16T12:24:04.285659+0000","last_became_active":"2023-10-16T12:24:03.284466+0000","last_became_peered":"2023-10-16T12:24:03.284466+0000","last_unstale":"2023-10-16T12:24:04.285659+0000","last_undegraded":"2023-10-16T12:24:04.285659+0000","last_fullsized":"2023-10-16T12:24:04.285659+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:02:01.654872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.618738+0000","last_change":"2023-10-16T12:24:03.285544+0000","last_active":"2023-10-16T12:24:04.618738+0000","last_peered":"2023-10-16T12:24:04.618738+0000","last_clean":"2023-10-16T12:24:04.618738+0000","last_became_active":"2023-10-16T12:24:03.285245+0000","last_became_peered":"2023-10-16T12:24:03.285245+0000","last_unstale":"2023-10-16T12:24:04.618738+0000","last_undegraded":"2023-10-16T12:24:04.618738+0000","last_fullsized":"2023-10-16T12:24:04.618738+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:40:56.018881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.284266+0000","last_change":"2023-10-16T12:24:03.287309+0000","last_active":"2023-10-16T12:24:04.284266+0000","last_peered":"2023-10-16T12:24:04.284266+0000","last_clean":"2023-10-16T12:24:04.284266+0000","last_became_active":"2023-10-16T12:24:03.286997+0000","last_became_peered":"2023-10-16T12:24:03.286997+0000","last_unstale":"2023-10-16T12:24:04.284266+0000","last_undegraded":"2023-10-16T12:24:04.284266+0000","last_fullsized":"2023-10-16T12:24:04.284266+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:31:06.779111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.618603+0000","last_change":"2023-10-16T12:24:03.291789+0000","last_active":"2023-10-16T12:24:04.618603+0000","last_peered":"2023-10-16T12:24:04.618603+0000","last_clean":"2023-10-16T12:24:04.618603+0000","last_became_active":"2023-10-16T12:24:03.291471+0000","last_became_peered":"2023-10-16T12:24:03.291471+0000","last_unstale":"2023-10-16T12:24:04.618603+0000","last_undegraded":"2023-10-16T12:24:04.618603+0000","last_fullsized":"2023-10-16T12:24:04.618603+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:11:40.533026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.620413+0000","last_change":"2023-10-16T12:24:03.290830+0000","last_active":"2023-10-16T12:24:04.620413+0000","last_peered":"2023-10-16T12:24:04.620413+0000","last_clean":"2023-10-16T12:24:04.620413+0000","last_became_active":"2023-10-16T12:24:03.290443+0000","last_became_peered":"2023-10-16T12:24:03.290443+0000","last_unstale":"2023-10-16T12:24:04.620413+0000","last_undegraded":"2023-10-16T12:24:04.620413+0000","last_fullsized":"2023-10-16T12:24:04.620413+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:40:56.018881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619490+0000","last_change":"2023-10-16T12:24:03.290691+0000","last_active":"2023-10-16T12:24:04.619490+0000","last_peered":"2023-10-16T12:24:04.619490+0000","last_clean":"2023-10-16T12:24:04.619490+0000","last_became_active":"2023-10-16T12:24:03.290405+0000","last_became_peered":"2023-10-16T12:24:03.290405+0000","last_unstale":"2023-10-16T12:24:04.619490+0000","last_undegraded":"2023-10-16T12:24:04.619490+0000","last_fullsized":"2023-10-16T12:24:04.619490+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:01:16.802661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619265+0000","last_change":"2023-10-16T12:24:03.303054+0000","last_active":"2023-10-16T12:24:04.619265+0000","last_peered":"2023-10-16T12:24:04.619265+0000","last_clean":"2023-10-16T12:24:04.619265+0000","last_became_active":"2023-10-16T12:24:03.302758+0000","last_became_peered":"2023-10-16T12:24:03.302758+0000","last_unstale":"2023-10-16T12:24:04.619265+0000","last_undegraded":"2023-10-16T12:24:04.619265+0000","last_fullsized":"2023-10-16T12:24:04.619265+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:44:10.478322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.285824+0000","last_change":"2023-10-16T12:24:03.290932+0000","last_active":"2023-10-16T12:24:04.285824+0000","last_peered":"2023-10-16T12:24:04.285824+0000","last_clean":"2023-10-16T12:24:04.285824+0000","last_became_active":"2023-10-16T12:24:03.290639+0000","last_became_peered":"2023-10-16T12:24:03.290639+0000","last_unstale":"2023-10-16T12:24:04.285824+0000","last_undegraded":"2023-10-16T12:24:04.285824+0000","last_fullsized":"2023-10-16T12:24:04.285824+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:10:20.088811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619456+0000","last_change":"2023-10-16T12:24:03.294572+0000","last_active":"2023-10-16T12:24:04.619456+0000","last_peered":"2023-10-16T12:24:04.619456+0000","last_clean":"2023-10-16T12:24:04.619456+0000","last_became_active":"2023-10-16T12:24:03.291752+0000","last_became_peered":"2023-10-16T12:24:03.291752+0000","last_unstale":"2023-10-16T12:24:04.619456+0000","last_undegraded":"2023-10-16T12:24:04.619456+0000","last_fullsized":"2023-10-16T12:24:04.619456+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:35:43.093148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.620108+0000","last_change":"2023-10-16T12:24:03.293658+0000","last_active":"2023-10-16T12:24:04.620108+0000","last_peered":"2023-10-16T12:24:04.620108+0000","last_clean":"2023-10-16T12:24:04.620108+0000","last_became_active":"2023-10-16T12:24:03.292954+0000","last_became_peered":"2023-10-16T12:24:03.292954+0000","last_unstale":"2023-10-16T12:24:04.620108+0000","last_undegraded":"2023-10-16T12:24:04.620108+0000","last_fullsized":"2023-10-16T12:24:04.620108+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:38:55.619256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619326+0000","last_change":"2023-10-16T12:24:03.296212+0000","last_active":"2023-10-16T12:24:04.619326+0000","last_peered":"2023-10-16T12:24:04.619326+0000","last_clean":"2023-10-16T12:24:04.619326+0000","last_became_active":"2023-10-16T12:24:03.292558+0000","last_became_peered":"2023-10-16T12:24:03.292558+0000","last_unstale":"2023-10-16T12:24:04.619326+0000","last_undegraded":"2023-10-16T12:24:04.619326+0000","last_fullsized":"2023-10-16T12:24:04.619326+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:01:17.363320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.618938+0000","last_change":"2023-10-16T12:24:03.290176+0000","last_active":"2023-10-16T12:24:04.618938+0000","last_peered":"2023-10-16T12:24:04.618938+0000","last_clean":"2023-10-16T12:24:04.618938+0000","last_became_active":"2023-10-16T12:24:03.289565+0000","last_became_peered":"2023-10-16T12:24:03.289565+0000","last_unstale":"2023-10-16T12:24:04.618938+0000","last_undegraded":"2023-10-16T12:24:04.618938+0000","last_fullsized":"2023-10-16T12:24:04.618938+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:53:46.533436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.618832+0000","last_change":"2023-10-16T12:24:03.281732+0000","last_active":"2023-10-16T12:24:04.618832+0000","last_peered":"2023-10-16T12:24:04.618832+0000","last_clean":"2023-10-16T12:24:04.618832+0000","last_became_active":"2023-10-16T12:24:03.280747+0000","last_became_peered":"2023-10-16T12:24:03.280747+0000","last_unstale":"2023-10-16T12:24:04.618832+0000","last_undegraded":"2023-10-16T12:24:04.618832+0000","last_fullsized":"2023-10-16T12:24:04.618832+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:01:17.363320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"67'4","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.285479+0000","last_change":"2023-10-16T12:24:02.284432+0000","last_active":"2023-10-16T12:24:04.285479+0000","last_peered":"2023-10-16T12:24:04.285479+0000","last_clean":"2023-10-16T12:24:04.285479+0000","last_became_active":"2023-10-16T12:24:02.284129+0000","last_became_peered":"2023-10-16T12:24:02.284129+0000","last_unstale":"2023-10-16T12:24:04.285479+0000","last_undegraded":"2023-10-16T12:24:04.285479+0000","last_fullsized":"2023-10-16T12:24:04.285479+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_clean_scrub_stamp":"2023-10-16T12:24:01.254337+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:17:59.015657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.620037+0000","last_change":"2023-10-16T12:24:03.290379+0000","last_active":"2023-10-16T12:24:04.620037+0000","last_peered":"2023-10-16T12:24:04.620037+0000","last_clean":"2023-10-16T12:24:04.620037+0000","last_became_active":"2023-10-16T12:24:03.290042+0000","last_became_peered":"2023-10-16T12:24:03.290042+0000","last_unstale":"2023-10-16T12:24:04.620037+0000","last_undegraded":"2023-10-16T12:24:04.620037+0000","last_fullsized":"2023-10-16T12:24:04.620037+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:55:39.961058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"67'1","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619517+0000","last_change":"2023-10-16T12:24:02.287250+0000","last_active":"2023-10-16T12:24:04.619517+0000","last_peered":"2023-10-16T12:24:04.619517+0000","last_clean":"2023-10-16T12:24:04.619517+0000","last_became_active":"2023-10-16T12:24:02.287010+0000","last_became_peered":"2023-10-16T12:24:02.287010+0000","last_unstale":"2023-10-16T12:24:04.619517+0000","last_undegraded":"2023-10-16T12:24:04.619517+0000","last_fullsized":"2023-10-16T12:24:04.619517+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_clean_scrub_stamp":"2023-10-16T12:24:01.254337+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:46:55.364974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.285416+0000","last_change":"2023-10-16T12:24:03.290251+0000","last_active":"2023-10-16T12:24:04.285416+0000","last_peered":"2023-10-16T12:24:04.285416+0000","last_clean":"2023-10-16T12:24:04.285416+0000","last_became_active":"2023-10-16T12:24:03.289955+0000","last_became_peered":"2023-10-16T12:24:03.289955+0000","last_unstale":"2023-10-16T12:24:04.285416+0000","last_undegraded":"2023-10-16T12:24:04.285416+0000","last_fullsized":"2023-10-16T12:24:04.285416+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:41:19.510518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"67'2","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619053+0000","last_change":"2023-10-16T12:24:02.288769+0000","last_active":"2023-10-16T12:24:04.619053+0000","last_peered":"2023-10-16T12:24:04.619053+0000","last_clean":"2023-10-16T12:24:04.619053+0000","last_became_active":"2023-10-16T12:24:02.284978+0000","last_became_peered":"2023-10-16T12:24:02.284978+0000","last_unstale":"2023-10-16T12:24:04.619053+0000","last_undegraded":"2023-10-16T12:24:04.619053+0000","last_fullsized":"2023-10-16T12:24:04.619053+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_clean_scrub_stamp":"2023-10-16T12:24:01.254337+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:52:48.050577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.620334+0000","last_change":"2023-10-16T12:24:03.290027+0000","last_active":"2023-10-16T12:24:04.620334+0000","last_peered":"2023-10-16T12:24:04.620334+0000","last_clean":"2023-10-16T12:24:04.620334+0000","last_became_active":"2023-10-16T12:24:03.286282+0000","last_became_peered":"2023-10-16T12:24:03.286282+0000","last_unstale":"2023-10-16T12:24:04.620334+0000","last_undegraded":"2023-10-16T12:24:04.620334+0000","last_fullsized":"2023-10-16T12:24:04.620334+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:31:00.012476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"67'2","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619574+0000","last_change":"2023-10-16T12:24:02.287316+0000","last_active":"2023-10-16T12:24:04.619574+0000","last_peered":"2023-10-16T12:24:04.619574+0000","last_clean":"2023-10-16T12:24:04.619574+0000","last_became_active":"2023-10-16T12:24:02.287089+0000","last_became_peered":"2023-10-16T12:24:02.287089+0000","last_unstale":"2023-10-16T12:24:04.619574+0000","last_undegraded":"2023-10-16T12:24:04.619574+0000","last_fullsized":"2023-10-16T12:24:04.619574+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_clean_scrub_stamp":"2023-10-16T12:24:01.254337+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:01:16.359582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.621061+0000","last_change":"2023-10-16T12:24:03.282668+0000","last_active":"2023-10-16T12:24:04.621061+0000","last_peered":"2023-10-16T12:24:04.621061+0000","last_clean":"2023-10-16T12:24:04.621061+0000","last_became_active":"2023-10-16T12:24:03.282375+0000","last_became_peered":"2023-10-16T12:24:03.282375+0000","last_unstale":"2023-10-16T12:24:04.621061+0000","last_undegraded":"2023-10-16T12:24:04.621061+0000","last_fullsized":"2023-10-16T12:24:04.621061+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:18:54.252334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"67'1","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619563+0000","last_change":"2023-10-16T12:24:02.284422+0000","last_active":"2023-10-16T12:24:04.619563+0000","last_peered":"2023-10-16T12:24:04.619563+0000","last_clean":"2023-10-16T12:24:04.619563+0000","last_became_active":"2023-10-16T12:24:02.284140+0000","last_became_peered":"2023-10-16T12:24:02.284140+0000","last_unstale":"2023-10-16T12:24:04.619563+0000","last_undegraded":"2023-10-16T12:24:04.619563+0000","last_fullsized":"2023-10-16T12:24:04.619563+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_clean_scrub_stamp":"2023-10-16T12:24:01.254337+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:01:17.574797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.620006+0000","last_change":"2023-10-16T12:24:03.287876+0000","last_active":"2023-10-16T12:24:04.620006+0000","last_peered":"2023-10-16T12:24:04.620006+0000","last_clean":"2023-10-16T12:24:04.620006+0000","last_became_active":"2023-10-16T12:24:03.287461+0000","last_became_peered":"2023-10-16T12:24:03.287461+0000","last_unstale":"2023-10-16T12:24:04.620006+0000","last_undegraded":"2023-10-16T12:24:04.620006+0000","last_fullsized":"2023-10-16T12:24:04.620006+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:20:27.553468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"67'1","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.620663+0000","last_change":"2023-10-16T12:24:02.283692+0000","last_active":"2023-10-16T12:24:04.620663+0000","last_peered":"2023-10-16T12:24:04.620663+0000","last_clean":"2023-10-16T12:24:04.620663+0000","last_became_active":"2023-10-16T12:24:02.283241+0000","last_became_peered":"2023-10-16T12:24:02.283241+0000","last_unstale":"2023-10-16T12:24:04.620663+0000","last_undegraded":"2023-10-16T12:24:04.620663+0000","last_fullsized":"2023-10-16T12:24:04.620663+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_clean_scrub_stamp":"2023-10-16T12:24:01.254337+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:46:01.346151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619279+0000","last_change":"2023-10-16T12:24:03.283956+0000","last_active":"2023-10-16T12:24:04.619279+0000","last_peered":"2023-10-16T12:24:04.619279+0000","last_clean":"2023-10-16T12:24:04.619279+0000","last_became_active":"2023-10-16T12:24:03.281947+0000","last_became_peered":"2023-10-16T12:24:03.281947+0000","last_unstale":"2023-10-16T12:24:04.619279+0000","last_undegraded":"2023-10-16T12:24:04.619279+0000","last_fullsized":"2023-10-16T12:24:04.619279+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:06:14.168954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.285303+0000","last_change":"2023-10-16T12:24:02.281241+0000","last_active":"2023-10-16T12:24:04.285303+0000","last_peered":"2023-10-16T12:24:04.285303+0000","last_clean":"2023-10-16T12:24:04.285303+0000","last_became_active":"2023-10-16T12:24:02.280918+0000","last_became_peered":"2023-10-16T12:24:02.280918+0000","last_unstale":"2023-10-16T12:24:04.285303+0000","last_undegraded":"2023-10-16T12:24:04.285303+0000","last_fullsized":"2023-10-16T12:24:04.285303+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_clean_scrub_stamp":"2023-10-16T12:24:01.254337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:37:34.622976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619989+0000","last_change":"2023-10-16T12:24:03.282828+0000","last_active":"2023-10-16T12:24:04.619989+0000","last_peered":"2023-10-16T12:24:04.619989+0000","last_clean":"2023-10-16T12:24:04.619989+0000","last_became_active":"2023-10-16T12:24:03.282547+0000","last_became_peered":"2023-10-16T12:24:03.282547+0000","last_unstale":"2023-10-16T12:24:04.619989+0000","last_undegraded":"2023-10-16T12:24:04.619989+0000","last_fullsized":"2023-10-16T12:24:04.619989+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:35:43.093148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619673+0000","last_change":"2023-10-16T12:24:02.283508+0000","last_active":"2023-10-16T12:24:04.619673+0000","last_peered":"2023-10-16T12:24:04.619673+0000","last_clean":"2023-10-16T12:24:04.619673+0000","last_became_active":"2023-10-16T12:24:02.283119+0000","last_became_peered":"2023-10-16T12:24:02.283119+0000","last_unstale":"2023-10-16T12:24:04.619673+0000","last_undegraded":"2023-10-16T12:24:04.619673+0000","last_fullsized":"2023-10-16T12:24:04.619673+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_clean_scrub_stamp":"2023-10-16T12:24:01.254337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:39:17.482962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.620632+0000","last_change":"2023-10-16T12:24:03.290637+0000","last_active":"2023-10-16T12:24:04.620632+0000","last_peered":"2023-10-16T12:24:04.620632+0000","last_clean":"2023-10-16T12:24:04.620632+0000","last_became_active":"2023-10-16T12:24:03.288515+0000","last_became_peered":"2023-10-16T12:24:03.288515+0000","last_unstale":"2023-10-16T12:24:04.620632+0000","last_undegraded":"2023-10-16T12:24:04.620632+0000","last_fullsized":"2023-10-16T12:24:04.620632+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:44:10.111021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.620125+0000","last_change":"2023-10-16T12:24:02.280970+0000","last_active":"2023-10-16T12:24:04.620125+0000","last_peered":"2023-10-16T12:24:04.620125+0000","last_clean":"2023-10-16T12:24:04.620125+0000","last_became_active":"2023-10-16T12:24:02.280680+0000","last_became_peered":"2023-10-16T12:24:02.280680+0000","last_unstale":"2023-10-16T12:24:04.620125+0000","last_undegraded":"2023-10-16T12:24:04.620125+0000","last_fullsized":"2023-10-16T12:24:04.620125+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_clean_scrub_stamp":"2023-10-16T12:24:01.254337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:11:28.470877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619490+0000","last_change":"2023-10-16T12:24:03.282651+0000","last_active":"2023-10-16T12:24:04.619490+0000","last_peered":"2023-10-16T12:24:04.619490+0000","last_clean":"2023-10-16T12:24:04.619490+0000","last_became_active":"2023-10-16T12:24:03.282359+0000","last_became_peered":"2023-10-16T12:24:03.282359+0000","last_unstale":"2023-10-16T12:24:04.619490+0000","last_undegraded":"2023-10-16T12:24:04.619490+0000","last_fullsized":"2023-10-16T12:24:04.619490+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:38:55.619256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.621147+0000","last_change":"2023-10-16T12:24:02.278079+0000","last_active":"2023-10-16T12:24:04.621147+0000","last_peered":"2023-10-16T12:24:04.621147+0000","last_clean":"2023-10-16T12:24:04.621147+0000","last_became_active":"2023-10-16T12:24:02.277761+0000","last_became_peered":"2023-10-16T12:24:02.277761+0000","last_unstale":"2023-10-16T12:24:04.621147+0000","last_undegraded":"2023-10-16T12:24:04.621147+0000","last_fullsized":"2023-10-16T12:24:04.621147+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_clean_scrub_stamp":"2023-10-16T12:24:01.254337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:44:09.474584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619686+0000","last_change":"2023-10-16T12:24:03.291531+0000","last_active":"2023-10-16T12:24:04.619686+0000","last_peered":"2023-10-16T12:24:04.619686+0000","last_clean":"2023-10-16T12:24:04.619686+0000","last_became_active":"2023-10-16T12:24:03.291228+0000","last_became_peered":"2023-10-16T12:24:03.291228+0000","last_unstale":"2023-10-16T12:24:04.619686+0000","last_undegraded":"2023-10-16T12:24:04.619686+0000","last_fullsized":"2023-10-16T12:24:04.619686+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:35:43.782093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"67'3","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.620095+0000","last_change":"2023-10-16T12:24:02.284853+0000","last_active":"2023-10-16T12:24:04.620095+0000","last_peered":"2023-10-16T12:24:04.620095+0000","last_clean":"2023-10-16T12:24:04.620095+0000","last_became_active":"2023-10-16T12:24:02.284438+0000","last_became_peered":"2023-10-16T12:24:02.284438+0000","last_unstale":"2023-10-16T12:24:04.620095+0000","last_undegraded":"2023-10-16T12:24:04.620095+0000","last_fullsized":"2023-10-16T12:24:04.620095+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_clean_scrub_stamp":"2023-10-16T12:24:01.254337+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:50:44.843718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.621005+0000","last_change":"2023-10-16T12:24:03.287948+0000","last_active":"2023-10-16T12:24:04.621005+0000","last_peered":"2023-10-16T12:24:04.621005+0000","last_clean":"2023-10-16T12:24:04.621005+0000","last_became_active":"2023-10-16T12:24:03.287637+0000","last_became_peered":"2023-10-16T12:24:03.287637+0000","last_unstale":"2023-10-16T12:24:04.621005+0000","last_undegraded":"2023-10-16T12:24:04.621005+0000","last_fullsized":"2023-10-16T12:24:04.621005+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:53:46.741709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"67'6","reported_seq":18,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619098+0000","last_change":"2023-10-16T12:24:02.288074+0000","last_active":"2023-10-16T12:24:04.619098+0000","last_peered":"2023-10-16T12:24:04.619098+0000","last_clean":"2023-10-16T12:24:04.619098+0000","last_became_active":"2023-10-16T12:24:02.285745+0000","last_became_peered":"2023-10-16T12:24:02.285745+0000","last_unstale":"2023-10-16T12:24:04.619098+0000","last_undegraded":"2023-10-16T12:24:04.619098+0000","last_fullsized":"2023-10-16T12:24:04.619098+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_clean_scrub_stamp":"2023-10-16T12:24:01.254337+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:35:44.718016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619449+0000","last_change":"2023-10-16T12:24:03.283374+0000","last_active":"2023-10-16T12:24:04.619449+0000","last_peered":"2023-10-16T12:24:04.619449+0000","last_clean":"2023-10-16T12:24:04.619449+0000","last_became_active":"2023-10-16T12:24:03.283035+0000","last_became_peered":"2023-10-16T12:24:03.283035+0000","last_unstale":"2023-10-16T12:24:04.619449+0000","last_undegraded":"2023-10-16T12:24:04.619449+0000","last_fullsized":"2023-10-16T12:24:04.619449+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:22:10.364603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"67'2","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619678+0000","last_change":"2023-10-16T12:24:02.278013+0000","last_active":"2023-10-16T12:24:04.619678+0000","last_peered":"2023-10-16T12:24:04.619678+0000","last_clean":"2023-10-16T12:24:04.619678+0000","last_became_active":"2023-10-16T12:24:02.277687+0000","last_became_peered":"2023-10-16T12:24:02.277687+0000","last_unstale":"2023-10-16T12:24:04.619678+0000","last_undegraded":"2023-10-16T12:24:04.619678+0000","last_fullsized":"2023-10-16T12:24:04.619678+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_clean_scrub_stamp":"2023-10-16T12:24:01.254337+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:46:03.012648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619996+0000","last_change":"2023-10-16T12:24:03.294356+0000","last_active":"2023-10-16T12:24:04.619996+0000","last_peered":"2023-10-16T12:24:04.619996+0000","last_clean":"2023-10-16T12:24:04.619996+0000","last_became_active":"2023-10-16T12:24:03.291626+0000","last_became_peered":"2023-10-16T12:24:03.291626+0000","last_unstale":"2023-10-16T12:24:04.619996+0000","last_undegraded":"2023-10-16T12:24:04.619996+0000","last_fullsized":"2023-10-16T12:24:04.619996+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:13:00.102553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.285540+0000","last_change":"2023-10-16T12:24:02.284975+0000","last_active":"2023-10-16T12:24:04.285540+0000","last_peered":"2023-10-16T12:24:04.285540+0000","last_clean":"2023-10-16T12:24:04.285540+0000","last_became_active":"2023-10-16T12:24:02.284577+0000","last_became_peered":"2023-10-16T12:24:02.284577+0000","last_unstale":"2023-10-16T12:24:04.285540+0000","last_undegraded":"2023-10-16T12:24:04.285540+0000","last_fullsized":"2023-10-16T12:24:04.285540+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_clean_scrub_stamp":"2023-10-16T12:24:01.254337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:47:13.219274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619125+0000","last_change":"2023-10-16T12:24:03.283843+0000","last_active":"2023-10-16T12:24:04.619125+0000","last_peered":"2023-10-16T12:24:04.619125+0000","last_clean":"2023-10-16T12:24:04.619125+0000","last_became_active":"2023-10-16T12:24:03.283531+0000","last_became_peered":"2023-10-16T12:24:03.283531+0000","last_unstale":"2023-10-16T12:24:04.619125+0000","last_undegraded":"2023-10-16T12:24:04.619125+0000","last_fullsized":"2023-10-16T12:24:04.619125+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:29:28.797038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619199+0000","last_change":"2023-10-16T12:24:02.285342+0000","last_active":"2023-10-16T12:24:04.619199+0000","last_peered":"2023-10-16T12:24:04.619199+0000","last_clean":"2023-10-16T12:24:04.619199+0000","last_became_active":"2023-10-16T12:24:02.285024+0000","last_became_peered":"2023-10-16T12:24:02.285024+0000","last_unstale":"2023-10-16T12:24:04.619199+0000","last_undegraded":"2023-10-16T12:24:04.619199+0000","last_fullsized":"2023-10-16T12:24:04.619199+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_clean_scrub_stamp":"2023-10-16T12:24:01.254337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:29:25.193985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.620299+0000","last_change":"2023-10-16T12:24:03.278722+0000","last_active":"2023-10-16T12:24:04.620299+0000","last_peered":"2023-10-16T12:24:04.620299+0000","last_clean":"2023-10-16T12:24:04.620299+0000","last_became_active":"2023-10-16T12:24:03.278416+0000","last_became_peered":"2023-10-16T12:24:03.278416+0000","last_unstale":"2023-10-16T12:24:04.620299+0000","last_undegraded":"2023-10-16T12:24:04.620299+0000","last_fullsized":"2023-10-16T12:24:04.620299+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:17:41.719369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"67'8","reported_seq":20,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.620216+0000","last_change":"2023-10-16T12:24:02.288564+0000","last_active":"2023-10-16T12:24:04.620216+0000","last_peered":"2023-10-16T12:24:04.620216+0000","last_clean":"2023-10-16T12:24:04.620216+0000","last_became_active":"2023-10-16T12:24:02.288271+0000","last_became_peered":"2023-10-16T12:24:02.288271+0000","last_unstale":"2023-10-16T12:24:04.620216+0000","last_undegraded":"2023-10-16T12:24:04.620216+0000","last_fullsized":"2023-10-16T12:24:04.620216+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_clean_scrub_stamp":"2023-10-16T12:24:01.254337+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:35:42.778355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.620039+0000","last_change":"2023-10-16T12:24:03.286609+0000","last_active":"2023-10-16T12:24:04.620039+0000","last_peered":"2023-10-16T12:24:04.620039+0000","last_clean":"2023-10-16T12:24:04.620039+0000","last_became_active":"2023-10-16T12:24:03.285733+0000","last_became_peered":"2023-10-16T12:24:03.285733+0000","last_unstale":"2023-10-16T12:24:04.620039+0000","last_undegraded":"2023-10-16T12:24:04.620039+0000","last_fullsized":"2023-10-16T12:24:04.620039+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:29:38.700767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"67'1","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.620152+0000","last_change":"2023-10-16T12:24:02.285817+0000","last_active":"2023-10-16T12:24:04.620152+0000","last_peered":"2023-10-16T12:24:04.620152+0000","last_clean":"2023-10-16T12:24:04.620152+0000","last_became_active":"2023-10-16T12:24:02.285507+0000","last_became_peered":"2023-10-16T12:24:02.285507+0000","last_unstale":"2023-10-16T12:24:04.620152+0000","last_undegraded":"2023-10-16T12:24:04.620152+0000","last_fullsized":"2023-10-16T12:24:04.620152+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_clean_scrub_stamp":"2023-10-16T12:24:01.254337+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:49:19.404409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.618777+0000","last_change":"2023-10-16T12:24:03.300237+0000","last_active":"2023-10-16T12:24:04.618777+0000","last_peered":"2023-10-16T12:24:04.618777+0000","last_clean":"2023-10-16T12:24:04.618777+0000","last_became_active":"2023-10-16T12:24:03.299944+0000","last_became_peered":"2023-10-16T12:24:03.299944+0000","last_unstale":"2023-10-16T12:24:04.618777+0000","last_undegraded":"2023-10-16T12:24:04.618777+0000","last_fullsized":"2023-10-16T12:24:04.618777+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:46:02.349889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"67'2","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.284544+0000","last_change":"2023-10-16T12:24:02.282861+0000","last_active":"2023-10-16T12:24:04.284544+0000","last_peered":"2023-10-16T12:24:04.284544+0000","last_clean":"2023-10-16T12:24:04.284544+0000","last_became_active":"2023-10-16T12:24:02.280447+0000","last_became_peered":"2023-10-16T12:24:02.280447+0000","last_unstale":"2023-10-16T12:24:04.284544+0000","last_undegraded":"2023-10-16T12:24:04.284544+0000","last_fullsized":"2023-10-16T12:24:04.284544+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_clean_scrub_stamp":"2023-10-16T12:24:01.254337+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:47:24.974456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.620691+0000","last_change":"2023-10-16T12:24:03.287961+0000","last_active":"2023-10-16T12:24:04.620691+0000","last_peered":"2023-10-16T12:24:04.620691+0000","last_clean":"2023-10-16T12:24:04.620691+0000","last_became_active":"2023-10-16T12:24:03.287623+0000","last_became_peered":"2023-10-16T12:24:03.287623+0000","last_unstale":"2023-10-16T12:24:04.620691+0000","last_undegraded":"2023-10-16T12:24:04.620691+0000","last_fullsized":"2023-10-16T12:24:04.620691+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:39:29.404835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"67'1","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.285717+0000","last_change":"2023-10-16T12:24:02.292400+0000","last_active":"2023-10-16T12:24:04.285717+0000","last_peered":"2023-10-16T12:24:04.285717+0000","last_clean":"2023-10-16T12:24:04.285717+0000","last_became_active":"2023-10-16T12:24:02.292091+0000","last_became_peered":"2023-10-16T12:24:02.292091+0000","last_unstale":"2023-10-16T12:24:04.285717+0000","last_undegraded":"2023-10-16T12:24:04.285717+0000","last_fullsized":"2023-10-16T12:24:04.285717+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_clean_scrub_stamp":"2023-10-16T12:24:01.254337+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:45:21.707886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619749+0000","last_change":"2023-10-16T12:24:03.290136+0000","last_active":"2023-10-16T12:24:04.619749+0000","last_peered":"2023-10-16T12:24:04.619749+0000","last_clean":"2023-10-16T12:24:04.619749+0000","last_became_active":"2023-10-16T12:24:03.289843+0000","last_became_peered":"2023-10-16T12:24:03.289843+0000","last_unstale":"2023-10-16T12:24:04.619749+0000","last_undegraded":"2023-10-16T12:24:04.619749+0000","last_fullsized":"2023-10-16T12:24:04.619749+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:39:29.404835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.285926+0000","last_change":"2023-10-16T12:24:02.286330+0000","last_active":"2023-10-16T12:24:04.285926+0000","last_peered":"2023-10-16T12:24:04.285926+0000","last_clean":"2023-10-16T12:24:04.285926+0000","last_became_active":"2023-10-16T12:24:02.285890+0000","last_became_peered":"2023-10-16T12:24:02.285890+0000","last_unstale":"2023-10-16T12:24:04.285926+0000","last_undegraded":"2023-10-16T12:24:04.285926+0000","last_fullsized":"2023-10-16T12:24:04.285926+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_clean_scrub_stamp":"2023-10-16T12:24:01.254337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:20:10.698580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619243+0000","last_change":"2023-10-16T12:24:03.289934+0000","last_active":"2023-10-16T12:24:04.619243+0000","last_peered":"2023-10-16T12:24:04.619243+0000","last_clean":"2023-10-16T12:24:04.619243+0000","last_became_active":"2023-10-16T12:24:03.289537+0000","last_became_peered":"2023-10-16T12:24:03.289537+0000","last_unstale":"2023-10-16T12:24:04.619243+0000","last_undegraded":"2023-10-16T12:24:04.619243+0000","last_fullsized":"2023-10-16T12:24:04.619243+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:20:27.553468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"67'2","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619850+0000","last_change":"2023-10-16T12:24:02.283343+0000","last_active":"2023-10-16T12:24:04.619850+0000","last_peered":"2023-10-16T12:24:04.619850+0000","last_clean":"2023-10-16T12:24:04.619850+0000","last_became_active":"2023-10-16T12:24:02.283027+0000","last_became_peered":"2023-10-16T12:24:02.283027+0000","last_unstale":"2023-10-16T12:24:04.619850+0000","last_undegraded":"2023-10-16T12:24:04.619850+0000","last_fullsized":"2023-10-16T12:24:04.619850+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_clean_scrub_stamp":"2023-10-16T12:24:01.254337+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:45:43.529561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.285623+0000","last_change":"2023-10-16T12:24:03.295797+0000","last_active":"2023-10-16T12:24:04.285623+0000","last_peered":"2023-10-16T12:24:04.285623+0000","last_clean":"2023-10-16T12:24:04.285623+0000","last_became_active":"2023-10-16T12:24:03.295515+0000","last_became_peered":"2023-10-16T12:24:03.295515+0000","last_unstale":"2023-10-16T12:24:04.285623+0000","last_undegraded":"2023-10-16T12:24:04.285623+0000","last_fullsized":"2023-10-16T12:24:04.285623+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:45:44.533299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.618847+0000","last_change":"2023-10-16T12:24:02.288599+0000","last_active":"2023-10-16T12:24:04.618847+0000","last_peered":"2023-10-16T12:24:04.618847+0000","last_clean":"2023-10-16T12:24:04.618847+0000","last_became_active":"2023-10-16T12:24:02.288292+0000","last_became_peered":"2023-10-16T12:24:02.288292+0000","last_unstale":"2023-10-16T12:24:04.618847+0000","last_undegraded":"2023-10-16T12:24:04.618847+0000","last_fullsized":"2023-10-16T12:24:04.618847+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_clean_scrub_stamp":"2023-10-16T12:24:01.254337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:55:38.957320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619944+0000","last_change":"2023-10-16T12:24:03.289842+0000","last_active":"2023-10-16T12:24:04.619944+0000","last_peered":"2023-10-16T12:24:04.619944+0000","last_clean":"2023-10-16T12:24:04.619944+0000","last_became_active":"2023-10-16T12:24:03.289551+0000","last_became_peered":"2023-10-16T12:24:03.289551+0000","last_unstale":"2023-10-16T12:24:04.619944+0000","last_undegraded":"2023-10-16T12:24:04.619944+0000","last_fullsized":"2023-10-16T12:24:04.619944+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:35:45.721754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619592+0000","last_change":"2023-10-16T12:24:02.289353+0000","last_active":"2023-10-16T12:24:04.619592+0000","last_peered":"2023-10-16T12:24:04.619592+0000","last_clean":"2023-10-16T12:24:04.619592+0000","last_became_active":"2023-10-16T12:24:02.289061+0000","last_became_peered":"2023-10-16T12:24:02.289061+0000","last_unstale":"2023-10-16T12:24:04.619592+0000","last_undegraded":"2023-10-16T12:24:04.619592+0000","last_fullsized":"2023-10-16T12:24:04.619592+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_clean_scrub_stamp":"2023-10-16T12:24:01.254337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:29:25.193985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619273+0000","last_change":"2023-10-16T12:24:03.288935+0000","last_active":"2023-10-16T12:24:04.619273+0000","last_peered":"2023-10-16T12:24:04.619273+0000","last_clean":"2023-10-16T12:24:04.619273+0000","last_became_active":"2023-10-16T12:24:03.288478+0000","last_became_peered":"2023-10-16T12:24:03.288478+0000","last_unstale":"2023-10-16T12:24:04.619273+0000","last_undegraded":"2023-10-16T12:24:04.619273+0000","last_fullsized":"2023-10-16T12:24:04.619273+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:13:47.577433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.284201+0000","last_change":"2023-10-16T12:24:02.274586+0000","last_active":"2023-10-16T12:24:04.284201+0000","last_peered":"2023-10-16T12:24:04.284201+0000","last_clean":"2023-10-16T12:24:04.284201+0000","last_became_active":"2023-10-16T12:24:02.274278+0000","last_became_peered":"2023-10-16T12:24:02.274278+0000","last_unstale":"2023-10-16T12:24:04.284201+0000","last_undegraded":"2023-10-16T12:24:04.284201+0000","last_fullsized":"2023-10-16T12:24:04.284201+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_clean_scrub_stamp":"2023-10-16T12:24:01.254337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:44:43.398711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619022+0000","last_change":"2023-10-16T12:24:03.289984+0000","last_active":"2023-10-16T12:24:04.619022+0000","last_peered":"2023-10-16T12:24:04.619022+0000","last_clean":"2023-10-16T12:24:04.619022+0000","last_became_active":"2023-10-16T12:24:03.289140+0000","last_became_peered":"2023-10-16T12:24:03.289140+0000","last_unstale":"2023-10-16T12:24:04.619022+0000","last_undegraded":"2023-10-16T12:24:04.619022+0000","last_fullsized":"2023-10-16T12:24:04.619022+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:35:43.093148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"67'2","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.285602+0000","last_change":"2023-10-16T12:24:02.284553+0000","last_active":"2023-10-16T12:24:04.285602+0000","last_peered":"2023-10-16T12:24:04.285602+0000","last_clean":"2023-10-16T12:24:04.285602+0000","last_became_active":"2023-10-16T12:24:02.284261+0000","last_became_peered":"2023-10-16T12:24:02.284261+0000","last_unstale":"2023-10-16T12:24:04.285602+0000","last_undegraded":"2023-10-16T12:24:04.285602+0000","last_fullsized":"2023-10-16T12:24:04.285602+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_clean_scrub_stamp":"2023-10-16T12:24:01.254337+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:04:01.752804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.618867+0000","last_change":"2023-10-16T12:24:03.285078+0000","last_active":"2023-10-16T12:24:04.618867+0000","last_peered":"2023-10-16T12:24:04.618867+0000","last_clean":"2023-10-16T12:24:04.618867+0000","last_became_active":"2023-10-16T12:24:03.284779+0000","last_became_peered":"2023-10-16T12:24:03.284779+0000","last_unstale":"2023-10-16T12:24:04.618867+0000","last_undegraded":"2023-10-16T12:24:04.618867+0000","last_fullsized":"2023-10-16T12:24:04.618867+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:12:34.419931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.620121+0000","last_change":"2023-10-16T12:24:02.277744+0000","last_active":"2023-10-16T12:24:04.620121+0000","last_peered":"2023-10-16T12:24:04.620121+0000","last_clean":"2023-10-16T12:24:04.620121+0000","last_became_active":"2023-10-16T12:24:02.277436+0000","last_became_peered":"2023-10-16T12:24:02.277436+0000","last_unstale":"2023-10-16T12:24:04.620121+0000","last_undegraded":"2023-10-16T12:24:04.620121+0000","last_fullsized":"2023-10-16T12:24:04.620121+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_clean_scrub_stamp":"2023-10-16T12:24:01.254337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:52:27.639535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619793+0000","last_change":"2023-10-16T12:24:03.281663+0000","last_active":"2023-10-16T12:24:04.619793+0000","last_peered":"2023-10-16T12:24:04.619793+0000","last_clean":"2023-10-16T12:24:04.619793+0000","last_became_active":"2023-10-16T12:24:03.281228+0000","last_became_peered":"2023-10-16T12:24:03.281228+0000","last_unstale":"2023-10-16T12:24:04.619793+0000","last_undegraded":"2023-10-16T12:24:04.619793+0000","last_fullsized":"2023-10-16T12:24:04.619793+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:05:25.768970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619724+0000","last_change":"2023-10-16T12:24:02.285672+0000","last_active":"2023-10-16T12:24:04.619724+0000","last_peered":"2023-10-16T12:24:04.619724+0000","last_clean":"2023-10-16T12:24:04.619724+0000","last_became_active":"2023-10-16T12:24:02.285337+0000","last_became_peered":"2023-10-16T12:24:02.285337+0000","last_unstale":"2023-10-16T12:24:04.619724+0000","last_undegraded":"2023-10-16T12:24:04.619724+0000","last_fullsized":"2023-10-16T12:24:04.619724+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_clean_scrub_stamp":"2023-10-16T12:24:01.254337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:35:48.332026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619518+0000","last_change":"2023-10-16T12:24:03.290847+0000","last_active":"2023-10-16T12:24:04.619518+0000","last_peered":"2023-10-16T12:24:04.619518+0000","last_clean":"2023-10-16T12:24:04.619518+0000","last_became_active":"2023-10-16T12:24:03.290512+0000","last_became_peered":"2023-10-16T12:24:03.290512+0000","last_unstale":"2023-10-16T12:24:04.619518+0000","last_undegraded":"2023-10-16T12:24:04.619518+0000","last_fullsized":"2023-10-16T12:24:04.619518+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:22:10.364603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619891+0000","last_change":"2023-10-16T12:24:02.285124+0000","last_active":"2023-10-16T12:24:04.619891+0000","last_peered":"2023-10-16T12:24:04.619891+0000","last_clean":"2023-10-16T12:24:04.619891+0000","last_became_active":"2023-10-16T12:24:02.284841+0000","last_became_peered":"2023-10-16T12:24:02.284841+0000","last_unstale":"2023-10-16T12:24:04.619891+0000","last_undegraded":"2023-10-16T12:24:04.619891+0000","last_fullsized":"2023-10-16T12:24:04.619891+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_clean_scrub_stamp":"2023-10-16T12:24:01.254337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:38:54.615518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619529+0000","last_change":"2023-10-16T12:24:03.283921+0000","last_active":"2023-10-16T12:24:04.619529+0000","last_peered":"2023-10-16T12:24:04.619529+0000","last_clean":"2023-10-16T12:24:04.619529+0000","last_became_active":"2023-10-16T12:24:03.282680+0000","last_became_peered":"2023-10-16T12:24:03.282680+0000","last_unstale":"2023-10-16T12:24:04.619529+0000","last_undegraded":"2023-10-16T12:24:04.619529+0000","last_fullsized":"2023-10-16T12:24:04.619529+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:35:43.782093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.618950+0000","last_change":"2023-10-16T12:24:02.291490+0000","last_active":"2023-10-16T12:24:04.618950+0000","last_peered":"2023-10-16T12:24:04.618950+0000","last_clean":"2023-10-16T12:24:04.618950+0000","last_became_active":"2023-10-16T12:24:02.291245+0000","last_became_peered":"2023-10-16T12:24:02.291245+0000","last_unstale":"2023-10-16T12:24:04.618950+0000","last_undegraded":"2023-10-16T12:24:04.618950+0000","last_fullsized":"2023-10-16T12:24:04.618950+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_clean_scrub_stamp":"2023-10-16T12:24:01.254337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:12:59.098815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619849+0000","last_change":"2023-10-16T12:24:03.295916+0000","last_active":"2023-10-16T12:24:04.619849+0000","last_peered":"2023-10-16T12:24:04.619849+0000","last_clean":"2023-10-16T12:24:04.619849+0000","last_became_active":"2023-10-16T12:24:03.292053+0000","last_became_peered":"2023-10-16T12:24:03.292053+0000","last_unstale":"2023-10-16T12:24:04.619849+0000","last_undegraded":"2023-10-16T12:24:04.619849+0000","last_fullsized":"2023-10-16T12:24:04.619849+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:55:39.961058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.620604+0000","last_change":"2023-10-16T12:24:02.279222+0000","last_active":"2023-10-16T12:24:04.620604+0000","last_peered":"2023-10-16T12:24:04.620604+0000","last_clean":"2023-10-16T12:24:04.620604+0000","last_became_active":"2023-10-16T12:24:02.278927+0000","last_became_peered":"2023-10-16T12:24:02.278927+0000","last_unstale":"2023-10-16T12:24:04.620604+0000","last_undegraded":"2023-10-16T12:24:04.620604+0000","last_fullsized":"2023-10-16T12:24:04.620604+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_clean_scrub_stamp":"2023-10-16T12:24:01.254337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:13:46.573695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619437+0000","last_change":"2023-10-16T12:24:03.284059+0000","last_active":"2023-10-16T12:24:04.619437+0000","last_peered":"2023-10-16T12:24:04.619437+0000","last_clean":"2023-10-16T12:24:04.619437+0000","last_became_active":"2023-10-16T12:24:03.281887+0000","last_became_peered":"2023-10-16T12:24:03.281887+0000","last_unstale":"2023-10-16T12:24:04.619437+0000","last_undegraded":"2023-10-16T12:24:04.619437+0000","last_fullsized":"2023-10-16T12:24:04.619437+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:17:58.552410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619055+0000","last_change":"2023-10-16T12:24:02.286415+0000","last_active":"2023-10-16T12:24:04.619055+0000","last_peered":"2023-10-16T12:24:04.619055+0000","last_clean":"2023-10-16T12:24:04.619055+0000","last_became_active":"2023-10-16T12:24:02.286073+0000","last_became_peered":"2023-10-16T12:24:02.286073+0000","last_unstale":"2023-10-16T12:24:04.619055+0000","last_undegraded":"2023-10-16T12:24:04.619055+0000","last_fullsized":"2023-10-16T12:24:04.619055+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_clean_scrub_stamp":"2023-10-16T12:24:01.254337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:19:27.299060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":11,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.285553+0000","last_change":"2023-10-16T12:24:03.291580+0000","last_active":"2023-10-16T12:24:04.285553+0000","last_peered":"2023-10-16T12:24:04.285553+0000","last_clean":"2023-10-16T12:24:04.285553+0000","last_became_active":"2023-10-16T12:24:03.291216+0000","last_became_peered":"2023-10-16T12:24:03.291216+0000","last_unstale":"2023-10-16T12:24:04.285553+0000","last_undegraded":"2023-10-16T12:24:04.285553+0000","last_fullsized":"2023-10-16T12:24:04.285553+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:02.258075+0000","last_clean_scrub_stamp":"2023-10-16T12:24:02.258075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:11:10.623582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-16T12:24:04.619969+0000","last_change":"2023-10-16T12:24:02.278058+0000","last_active":"2023-10-16T12:24:04.619969+0000","last_peered":"2023-10-16T12:24:04.619969+0000","last_clean":"2023-10-16T12:24:04.619969+0000","last_became_active":"2023-10-16T12:24:02.277736+0000","last_became_peered":"2023-10-16T12:24:02.277736+0000","last_unstale":"2023-10-16T12:24:04.619969+0000","last_undegraded":"2023-10-16T12:24:04.619969+0000","last_fullsized":"2023-10-16T12:24:04.619969+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:24:01.254337+0000","last_clean_scrub_stamp":"2023-10-16T12:24:01.254337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:50:44.843718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"57'139","reported_seq":291,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T12:24:03.272411+0000","last_change":"2023-10-16T12:10:34.746820+0000","last_active":"2023-10-16T12:24:03.272411+0000","last_peered":"2023-10-16T12:24:03.272411+0000","last_clean":"2023-10-16T12:24:03.272411+0000","last_became_active":"2023-10-16T12:10:34.745574+0000","last_became_peered":"2023-10-16T12:10:34.745574+0000","last_unstale":"2023-10-16T12:24:03.272411+0000","last_undegraded":"2023-10-16T12:24:03.272411+0000","last_fullsized":"2023-10-16T12:24:03.272411+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_clean_scrub_stamp":"2023-10-16T12:10:33.577111+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:14.461986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575014,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58920,"kb_used_data":1372,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312920,"statfs":{"total":96636764160,"available":96576430080,"internally_reserved":0,"allocated":1404928,"data_stored":985030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58915275},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:23:57 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.60899999999999999,"15min":0.57399999999999995},"min":{"1min":0.313,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.72599999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.59399999999999997,"15min":0.57299999999999995},"min":{"1min":0.374,"5min":0.35299999999999998,"15min":0.312},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.49399999999999999}]},{"osd":1,"last update":"Mon Oct 16 12:23:57 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.61899999999999999,"15min":0.58999999999999997},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.247},"max":{"1min":0.76200000000000001,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.59799999999999998,"15min":0.58699999999999997},"min":{"1min":0.373,"5min":0.33200000000000002,"15min":0.32400000000000001},"max":{"1min":0.82999999999999996,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.58299999999999996}]},{"osd":2,"last update":"Mon Oct 16 12:23:57 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.67300000000000004,"15min":0.64000000000000001},"min":{"1min":0.41599999999999998,"5min":0.378,"15min":0.36099999999999999},"max":{"1min":0.81799999999999995,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.68799999999999994,"15min":0.65700000000000003},"min":{"1min":0.41099999999999998,"5min":0.376,"15min":0.36699999999999999},"max":{"1min":0.85899999999999999,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.47299999999999998}]},{"osd":3,"last update":"Mon Oct 16 12:23:57 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.624,"15min":0.61299999999999999},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.34899999999999998},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":1.1000000000000001},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.64300000000000002,"15min":0.63400000000000001},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.2509999999999999,"5min":1.2509999999999999,"15min":1.2509999999999999},"last":0.43099999999999999}]},{"osd":4,"last update":"Mon Oct 16 12:23:57 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.52900000000000003,"15min":0.54000000000000004},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.28499999999999998},"max":{"1min":1.383,"5min":1.383,"15min":1.383},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.629,"5min":0.61699999999999999,"15min":0.60999999999999999},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.307},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.54800000000000004}]},{"osd":5,"last update":"Mon Oct 16 12:23:57 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.53100000000000003,"15min":0.54700000000000004},"min":{"1min":0.314,"5min":0.314,"15min":0.29799999999999999},"max":{"1min":1.3109999999999999,"5min":1.3109999999999999,"15min":1.3109999999999999},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.32400000000000001},"max":{"1min":1.4359999999999999,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":0.56499999999999995}]},{"osd":6,"last update":"Mon Oct 16 12:23:57 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.628,"15min":0.624},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.36399999999999999},"max":{"1min":0.80800000000000005,"5min":0.91200000000000003,"15min":0.91400000000000003},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.63900000000000001,"15min":0.63200000000000001},"min":{"1min":0.46300000000000002,"5min":0.34599999999999997,"15min":0.32300000000000001},"max":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.59899999999999998}]}]},{"osd":6,"up_from":13,"seq":55834575012,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31984,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339856,"statfs":{"total":96636764160,"available":96604012544,"internally_reserved":0,"allocated":823296,"data_stored":401810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:23:51 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.61899999999999999,"15min":0.65500000000000003},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.26900000000000002},"max":{"1min":1.196,"5min":1.196,"15min":5.9349999999999996},"last":0.747},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.629,"15min":0.66900000000000004},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.997,"5min":0.997,"15min":5.9489999999999998},"last":0.75800000000000001}]},{"osd":1,"last update":"Mon Oct 16 12:23:51 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.57599999999999996,"15min":0.57299999999999995},"min":{"1min":0.32800000000000001,"5min":0.30499999999999999,"15min":0.29399999999999998},"max":{"1min":0.83899999999999997,"5min":0.875,"15min":0.875},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.60299999999999998,"15min":0.59499999999999997},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.56599999999999995}]},{"osd":2,"last update":"Mon Oct 16 12:23:51 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.59999999999999998,"15min":0.66700000000000004},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.34499999999999997},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":8.0489999999999995},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.60299999999999998,"15min":0.623},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":2.855},"last":0.70399999999999996}]},{"osd":3,"last update":"Mon Oct 16 12:23:51 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.63700000000000001,"15min":0.63300000000000001},"min":{"1min":0.38100000000000001,"5min":0.38,"15min":0.307},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":1.083},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.68899999999999995,"15min":0.67700000000000005},"min":{"1min":0.42799999999999999,"5min":0.40899999999999997,"15min":0.36599999999999999},"max":{"1min":1.3080000000000001,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.66600000000000004}]},{"osd":4,"last update":"Mon Oct 16 12:23:51 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52000000000000002,"15min":0.52200000000000002},"min":{"1min":0.372,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.83899999999999997,"5min":0.85599999999999998,"15min":0.88200000000000001},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.60699999999999998,"15min":0.59499999999999997},"min":{"1min":0.40300000000000002,"5min":0.318,"15min":0.30199999999999999},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.65000000000000002}]},{"osd":5,"last update":"Mon Oct 16 12:23:51 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.58799999999999997,"15min":0.58299999999999996},"min":{"1min":0.41699999999999998,"5min":0.34799999999999998,"15min":0.30099999999999999},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.63800000000000001,"15min":0.63100000000000001},"min":{"1min":0.38700000000000001,"5min":0.35399999999999998,"15min":0.311},"max":{"1min":1.3540000000000001,"5min":1.3540000000000001,"15min":1.3540000000000001},"last":0.63300000000000001}]},{"osd":7,"last update":"Mon Oct 16 12:23:51 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.59699999999999998,"15min":0.58199999999999996},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.63400000000000001,"15min":0.629},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.60199999999999998}]}]},{"osd":1,"up_from":13,"seq":55834575012,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31972,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339868,"statfs":{"total":96636764160,"available":96604024832,"internally_reserved":0,"allocated":811008,"data_stored":394698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":31914441},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:23:55 2023","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.38,"15min":0.377},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.186},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.66000000000000003},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.46899999999999997,"15min":0.45100000000000001},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.81699999999999995},"last":1.0189999999999999}]},{"osd":2,"last update":"Mon Oct 16 12:23:55 2023","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.439,"15min":0.42599999999999999},"min":{"1min":0.25,"5min":0.25,"15min":0.215},"max":{"1min":0.69199999999999995,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.47699999999999998,"15min":0.45400000000000001},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.24399999999999999},"max":{"1min":0.61199999999999999,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.67200000000000004}]},{"osd":3,"last update":"Mon Oct 16 12:23:55 2023","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.40899999999999997,"15min":0.39500000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.23599999999999999},"max":{"1min":0.53600000000000003,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.98799999999999999},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.45200000000000001,"15min":0.45100000000000001},"min":{"1min":0.252,"5min":0.252,"15min":0.22900000000000001},"max":{"1min":0.60499999999999998,"5min":0.73299999999999998,"15min":1.127},"last":0.91200000000000003}]},{"osd":4,"last update":"Mon Oct 16 12:23:55 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.64100000000000001,"15min":0.628},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.85499999999999998,"5min":0.93500000000000005,"15min":0.95599999999999996},"last":1.0409999999999999},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.66000000000000003,"15min":0.63500000000000001},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.34699999999999998},"max":{"1min":0.75600000000000001,"5min":0.98299999999999998,"15min":1.089},"last":0.48699999999999999}]},{"osd":5,"last update":"Mon Oct 16 12:23:55 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.68200000000000005,"15min":0.66900000000000004},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.28499999999999998},"max":{"1min":0.93400000000000005,"5min":1,"15min":1.052},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.69499999999999995,"15min":0.67800000000000005},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.32500000000000001},"max":{"1min":0.90300000000000002,"5min":0.96699999999999997,"15min":1.0649999999999999},"last":1.0620000000000001}]},{"osd":6,"last update":"Mon Oct 16 12:23:55 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.65900000000000003,"15min":0.65000000000000002},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.94999999999999996,"5min":1.0109999999999999,"15min":1.0129999999999999},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.67700000000000005,"15min":0.65600000000000003},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.80700000000000005,"5min":1.048,"15min":1.0609999999999999},"last":0.749}]},{"osd":7,"last update":"Mon Oct 16 12:23:55 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.70599999999999996,"15min":0.68300000000000005},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.82899999999999996,"5min":0.98499999999999999,"15min":1.1499999999999999},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.70999999999999996,"15min":0.69999999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.91700000000000004,"5min":1.02,"15min":1.167},"last":0.55200000000000005}]}]},{"osd":0,"up_from":13,"seq":55834575013,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32560,"kb_used_data":1372,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339280,"statfs":{"total":96636764160,"available":96603422720,"internally_reserved":0,"allocated":1404928,"data_stored":984962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 12:23:50 2023","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.379,"15min":0.377},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.55600000000000005,"5min":0.64200000000000002,"15min":0.73699999999999999},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.42099999999999999,"15min":0.40799999999999997},"min":{"1min":0.25800000000000001,"5min":0.24299999999999999,"15min":0.23200000000000001},"max":{"1min":0.54800000000000004,"5min":0.60799999999999998,"15min":0.73099999999999998},"last":0.34899999999999998}]},{"osd":2,"last update":"Mon Oct 16 12:23:50 2023","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.42399999999999999,"15min":0.41899999999999998},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.25800000000000001},"max":{"1min":0.55100000000000005,"5min":0.65600000000000003,"15min":0.67200000000000004},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.44,"5min":0.45200000000000001,"15min":0.44},"min":{"1min":0.29599999999999999,"5min":0.29499999999999998,"15min":0.27900000000000003},"max":{"1min":0.67600000000000005,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.307}]},{"osd":3,"last update":"Mon Oct 16 12:23:50 2023","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.42499999999999999,"15min":0.41199999999999998},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.23699999999999999},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.48199999999999998,"15min":0.47699999999999998},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.27600000000000002},"max":{"1min":0.63,"5min":0.878,"15min":0.878},"last":0.371}]},{"osd":4,"last update":"Mon Oct 16 12:23:50 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.64600000000000002,"15min":0.63900000000000001},"min":{"1min":0.39600000000000002,"5min":0.377,"15min":0.36699999999999999},"max":{"1min":0.80200000000000005,"5min":0.90700000000000003,"15min":1.0960000000000001},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.65700000000000003,"15min":0.64700000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.77200000000000002,"5min":0.94999999999999996,"15min":1.1719999999999999},"last":0.57799999999999996}]},{"osd":5,"last update":"Mon Oct 16 12:23:50 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.65300000000000002,"15min":0.63900000000000001},"min":{"1min":0.374,"5min":0.374,"15min":0.34399999999999997},"max":{"1min":0.90100000000000002,"5min":1,"15min":1.127},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.67000000000000004,"15min":0.67300000000000004},"min":{"1min":0.39200000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.82299999999999995,"5min":1.0069999999999999,"15min":1.1160000000000001},"last":0.58999999999999997}]},{"osd":6,"last update":"Mon Oct 16 12:23:50 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.68000000000000005,"15min":0.67500000000000004},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.751,"5min":1.0409999999999999,"15min":1.079},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.68999999999999995,"15min":0.68300000000000005},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.93600000000000005,"5min":0.98299999999999998,"15min":1.1499999999999999},"last":0.52000000000000002}]},{"osd":7,"last update":"Mon Oct 16 12:23:50 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.69599999999999995,"15min":0.68500000000000005},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.40699999999999997},"max":{"1min":0.88400000000000001,"5min":0.999,"15min":1.0349999999999999},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.627,"5min":0.69299999999999995,"15min":0.753},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41299999999999998},"max":{"1min":0.83799999999999997,"5min":1.0329999999999999,"15min":6.7300000000000004},"last":0.625}]}]},{"osd":2,"up_from":13,"seq":55834575013,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58360,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313480,"statfs":{"total":96636764160,"available":96577003520,"internally_reserved":0,"allocated":827392,"data_stored":402770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:23:49 2023","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.38700000000000001,"15min":0.376},"min":{"1min":0.222,"5min":0.218,"15min":0.21199999999999999},"max":{"1min":0.56999999999999995,"5min":0.60299999999999998,"15min":0.73799999999999999},"last":0.44},{"interface":"front","average":{"1min":0.441,"5min":0.45100000000000001,"15min":0.439},"min":{"1min":0.29899999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.53800000000000003}]},{"osd":1,"last update":"Mon Oct 16 12:23:49 2023","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.47399999999999998,"15min":0.45200000000000001},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.254},"max":{"1min":0.63700000000000001,"5min":0.68200000000000005,"15min":0.71199999999999997},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.44500000000000001,"15min":0.43099999999999999},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.246},"max":{"1min":0.63100000000000001,"5min":0.69699999999999995,"15min":0.76500000000000001},"last":0.502}]},{"osd":3,"last update":"Mon Oct 16 12:23:49 2023","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.41799999999999998,"15min":0.40699999999999997},"min":{"1min":0.27900000000000003,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.51600000000000001,"15min":0.505},"min":{"1min":0.318,"5min":0.318,"15min":0.246},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.36399999999999999}]},{"osd":4,"last update":"Mon Oct 16 12:23:49 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.67900000000000005,"15min":0.68100000000000005},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":1.3839999999999999},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.68999999999999995,"15min":0.68200000000000005},"min":{"1min":0.46400000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.72799999999999998}]},{"osd":5,"last update":"Mon Oct 16 12:23:49 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.67400000000000004,"15min":0.65300000000000002},"min":{"1min":0.40200000000000002,"5min":0.312,"15min":0.312},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":1.0800000000000001},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.66800000000000004,"15min":0.66000000000000003},"min":{"1min":0.41299999999999998,"5min":0.38600000000000001,"15min":0.317},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":1.331}]},{"osd":6,"last update":"Mon Oct 16 12:23:49 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.65500000000000003,"15min":0.64100000000000001},"min":{"1min":0.439,"5min":0.42499999999999999,"15min":0.34699999999999998},"max":{"1min":0.97399999999999998,"5min":1.036,"15min":1.046},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.628,"5min":0.68700000000000006,"15min":0.67300000000000004},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.004,"5min":1.004,"15min":1.0389999999999999},"last":0.81699999999999995}]},{"osd":7,"last update":"Mon Oct 16 12:23:49 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.93899999999999995,"5min":1.016,"15min":1.026},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.627,"5min":0.68899999999999995,"15min":0.68300000000000005},"min":{"1min":0.442,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.92100000000000004,"5min":0.96299999999999997,"15min":1.0509999999999999},"last":0.84899999999999998}]}]},{"osd":3,"up_from":13,"seq":55834575012,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36148,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335692,"statfs":{"total":96636764160,"available":96599748608,"internally_reserved":0,"allocated":819200,"data_stored":396172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:23:57 2023","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.40799999999999997,"15min":0.38300000000000001},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.216},"max":{"1min":0.60699999999999998,"5min":0.66900000000000004,"15min":0.67700000000000005},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.41599999999999998,"15min":0.39700000000000002},"min":{"1min":0.25900000000000001,"5min":0.255,"15min":0.22600000000000001},"max":{"1min":0.63100000000000001,"5min":0.64400000000000002,"15min":0.73099999999999998},"last":0.35399999999999998}]},{"osd":1,"last update":"Mon Oct 16 12:23:57 2023","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.42699999999999999,"15min":0.39900000000000002},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.53700000000000003,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.317},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.47299999999999998,"15min":0.44500000000000001},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.25700000000000001},"max":{"1min":0.65700000000000003,"5min":0.747,"15min":0.747},"last":0.47399999999999998}]},{"osd":2,"last update":"Mon Oct 16 12:23:57 2023","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.47499999999999998,"15min":0.45500000000000002},"min":{"1min":0.307,"5min":0.30499999999999999,"15min":0.26900000000000002},"max":{"1min":0.63500000000000001,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.51300000000000001,"15min":0.47899999999999998},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.27200000000000002},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.41699999999999998}]},{"osd":4,"last update":"Mon Oct 16 12:23:57 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.66100000000000003,"15min":0.63500000000000001},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.377},"max":{"1min":0.92700000000000005,"5min":1.026,"15min":1.1539999999999999},"last":0.438},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.66800000000000004,"15min":0.65800000000000003},"min":{"1min":0.44900000000000001,"5min":0.42999999999999999,"15min":0.40200000000000002},"max":{"1min":0.95699999999999996,"5min":1.0409999999999999,"15min":1.234},"last":0.45800000000000002}]},{"osd":5,"last update":"Mon Oct 16 12:23:57 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.69699999999999995,"15min":0.67200000000000004},"min":{"1min":0.443,"5min":0.443,"15min":0.41199999999999998},"max":{"1min":0.97199999999999998,"5min":0.996,"15min":1.194},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.68000000000000005,"15min":0.65800000000000003},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.89600000000000002,"5min":0.97399999999999998,"15min":1.202},"last":0.57299999999999995}]},{"osd":6,"last update":"Mon Oct 16 12:23:57 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.68700000000000006,"15min":0.66200000000000003},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.94499999999999995,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.72399999999999998,"15min":0.69499999999999995},"min":{"1min":0.51500000000000001,"5min":0.47399999999999998,"15min":0.42699999999999999},"max":{"1min":0.96899999999999997,"5min":1.0409999999999999,"15min":1.1559999999999999},"last":0.59499999999999997}]},{"osd":7,"last update":"Mon Oct 16 12:23:57 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.69999999999999996,"15min":0.68100000000000005},"min":{"1min":0.435,"5min":0.435,"15min":0.38},"max":{"1min":0.94199999999999995,"5min":1.014,"15min":1.2430000000000001},"last":0.623},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.72399999999999998,"15min":0.69699999999999995},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.42699999999999999},"max":{"1min":1.004,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.60899999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575011,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31980,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339860,"statfs":{"total":96636764160,"available":96604016640,"internally_reserved":0,"allocated":806912,"data_stored":395120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:23:56 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.59799999999999998,"15min":0.58399999999999996},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.79300000000000004,"5min":0.90200000000000002,"15min":1.171},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.60299999999999998,"15min":0.58399999999999996},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.69899999999999995}]},{"osd":1,"last update":"Mon Oct 16 12:23:56 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.61299999999999999,"15min":0.60399999999999998},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.28499999999999998},"max":{"1min":0.71999999999999997,"5min":0.89600000000000002,"15min":0.92600000000000005},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.625,"15min":0.61299999999999999},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.33600000000000002},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.71699999999999997}]},{"osd":2,"last update":"Mon Oct 16 12:23:56 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.625,"15min":0.61799999999999999},"min":{"1min":0.35399999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":1.012},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.629,"15min":0.621},"min":{"1min":0.38800000000000001,"5min":0.38600000000000001,"15min":0.34100000000000003},"max":{"1min":0.84699999999999998,"5min":0.873,"15min":0.873},"last":0.68700000000000006}]},{"osd":3,"last update":"Mon Oct 16 12:23:56 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.61599999999999999,"15min":0.61699999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.79500000000000004,"5min":0.878,"15min":1.3779999999999999},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.63800000000000001,"15min":0.64200000000000002},"min":{"1min":0.42099999999999999,"5min":0.39700000000000002,"15min":0.37},"max":{"1min":0.87,"5min":0.91900000000000004,"15min":0.97199999999999998},"last":0.61299999999999999}]},{"osd":5,"last update":"Mon Oct 16 12:23:56 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.49299999999999999,"15min":0.51500000000000001},"min":{"1min":0.34000000000000002,"5min":0.29799999999999999,"15min":0.27600000000000002},"max":{"1min":0.69499999999999995,"5min":0.75600000000000001,"15min":0.85899999999999999},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52300000000000002,"15min":0.54000000000000004},"min":{"1min":0.316,"5min":0.316,"15min":0.29599999999999999},"max":{"1min":0.74399999999999999,"5min":0.79100000000000004,"15min":0.94199999999999995},"last":0.65200000000000002}]},{"osd":6,"last update":"Mon Oct 16 12:23:56 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.55200000000000005,"15min":0.57599999999999996},"min":{"1min":0.39400000000000002,"5min":0.32500000000000001,"15min":0.28699999999999998},"max":{"1min":0.84499999999999997,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.63400000000000001,"15min":0.64700000000000002},"min":{"1min":0.47299999999999998,"5min":0.34799999999999998,"15min":0.32000000000000001},"max":{"1min":0.91300000000000003,"5min":0.91500000000000004,"15min":1.1180000000000001},"last":0.752}]},{"osd":7,"last update":"Mon Oct 16 12:23:56 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.58199999999999996,"15min":0.59399999999999997},"min":{"1min":0.379,"5min":0.372,"15min":0.34100000000000003},"max":{"1min":0.745,"5min":0.83599999999999997,"15min":0.89500000000000002},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.63100000000000001,"15min":0.64600000000000002},"min":{"1min":0.48199999999999998,"5min":0.38400000000000001,"15min":0.375},"max":{"1min":0.872,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.82099999999999995}]}]},{"osd":5,"up_from":13,"seq":55834575012,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54172,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317668,"statfs":{"total":96636764160,"available":96581292032,"internally_reserved":0,"allocated":802816,"data_stored":394538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:23:52 2023","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.59799999999999998,"15min":0.57999999999999996},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.82399999999999995,"5min":0.873,"15min":0.873},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.308,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.83899999999999997,"5min":0.86799999999999999,"15min":0.88800000000000001},"last":0.313}]},{"osd":1,"last update":"Mon Oct 16 12:23:52 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.61199999999999999,"15min":0.61099999999999999},"min":{"1min":0.32600000000000001,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.78900000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.622,"15min":0.61799999999999999},"min":{"1min":0.39200000000000002,"5min":0.32800000000000001,"15min":0.28799999999999998},"max":{"1min":0.85699999999999998,"5min":0.89000000000000001,"15min":0.95199999999999996},"last":0.35099999999999998}]},{"osd":2,"last update":"Mon Oct 16 12:23:52 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.63400000000000001,"15min":0.625},"min":{"1min":0.35099999999999998,"5min":0.29099999999999998,"15min":0.26400000000000001},"max":{"1min":0.83699999999999997,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.497},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.38200000000000001,"5min":0.378,"15min":0.36499999999999999},"max":{"1min":0.76800000000000002,"5min":0.84599999999999997,"15min":0.89700000000000002},"last":0.63900000000000001}]},{"osd":3,"last update":"Mon Oct 16 12:23:52 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.64600000000000002,"15min":0.64300000000000002},"min":{"1min":0.45800000000000002,"5min":0.371,"15min":0.36099999999999999},"max":{"1min":0.73799999999999999,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.65100000000000002,"15min":0.66200000000000003},"min":{"1min":0.47499999999999998,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.80700000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.59999999999999998}]},{"osd":4,"last update":"Mon Oct 16 12:23:52 2023","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.51500000000000001,"15min":0.52900000000000003},"min":{"1min":0.33100000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.70099999999999996,"5min":0.80200000000000005,"15min":1.0089999999999999},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.58099999999999996,"15min":0.58899999999999997},"min":{"1min":0.40699999999999997,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.347,"5min":1.347,"15min":1.347},"last":0.66000000000000003}]},{"osd":6,"last update":"Mon Oct 16 12:23:52 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.54400000000000004,"15min":0.57199999999999995},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.71099999999999997,"5min":0.93500000000000005,"15min":0.98699999999999999},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.60599999999999998,"15min":0.61399999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.35999999999999999},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.95999999999999996},"last":0.747}]},{"osd":7,"last update":"Mon Oct 16 12:23:52 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.622,"15min":0.63400000000000001},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.78100000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.65900000000000003,"15min":0.67400000000000004},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.73199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T12:24:07.838 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-16T12:24:07.839 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-16T12:24:08.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.003+0000 7fd244bec700 1 Processor -- start 2023-10-16T12:24:08.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.003+0000 7fd244bec700 1 -- start start 2023-10-16T12:24:08.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.004+0000 7fd244bec700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fd2400cf4c0 0x7fd2400ca1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:08.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.004+0000 7fd244bec700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd24006bad0 0x7fd2400ca6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:08.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.004+0000 7fd244bec700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd24015dda0 con 0x7fd240159de0 2023-10-16T12:24:08.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.004+0000 7fd244bec700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd24015ea20 con 0x7fd2400cf4c0 2023-10-16T12:24:08.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.004+0000 7fd244bec700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd24015e080 con 0x7fd24006bad0 2023-10-16T12:24:08.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.004+0000 7fd23e59c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd240159de0 0x7fd24015a1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:55340/0 (socket says 172.21.15.16:55340) 2023-10-16T12:24:08.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.004+0000 7fd23e59c700 1 -- 172.21.15.16:0/276257180 learned_addr learned my addr 172.21.15.16:0/276257180 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:24:08.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.004+0000 7fd23dd9b700 1 --2- 172.21.15.16:0/276257180 >> v2:172.21.15.49:3300/0 conn(0x7fd2400cf4c0 0x7fd2400ca1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:08.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.004+0000 7fd23ed9d700 1 --2- 172.21.15.16:0/276257180 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd24006bad0 0x7fd2400ca6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:08.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.004+0000 7fd23d59a700 1 -- 172.21.15.16:0/276257180 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3018563997 0 0) 0x7fd24015dda0 con 0x7fd240159de0 2023-10-16T12:24:08.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.004+0000 7fd23d59a700 1 -- 172.21.15.16:0/276257180 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2280035b0 con 0x7fd240159de0 2023-10-16T12:24:08.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.005+0000 7fd23d59a700 1 -- 172.21.15.16:0/276257180 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2777059452 0 0) 0x7fd2280035b0 con 0x7fd240159de0 2023-10-16T12:24:08.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.005+0000 7fd23d59a700 1 -- 172.21.15.16:0/276257180 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd24006bad0 msgr2=0x7fd2400ca6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:24:08.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.005+0000 7fd23d59a700 1 --2- 172.21.15.16:0/276257180 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd24006bad0 0x7fd2400ca6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:08.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.005+0000 7fd23d59a700 1 -- 172.21.15.16:0/276257180 >> v2:172.21.15.49:3300/0 conn(0x7fd2400cf4c0 msgr2=0x7fd2400ca1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:24:08.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.005+0000 7fd23d59a700 1 --2- 172.21.15.16:0/276257180 >> v2:172.21.15.49:3300/0 conn(0x7fd2400cf4c0 0x7fd2400ca1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:08.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.005+0000 7fd23d59a700 1 -- 172.21.15.16:0/276257180 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd24015e360 con 0x7fd240159de0 2023-10-16T12:24:08.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.005+0000 7fd23d59a700 1 -- 172.21.15.16:0/276257180 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7fd230001670 con 0x7fd240159de0 2023-10-16T12:24:08.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.005+0000 7fd23d59a700 1 -- 172.21.15.16:0/276257180 >> v1:172.21.15.16:6789/0 conn(0x7fd240159de0 legacy=0x7fd24015a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:08.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.005+0000 7fd23d59a700 1 --2- 172.21.15.16:0/276257180 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd24006bad0 0x7fd228003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:08.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.005+0000 7fd23d59a700 1 --2- 172.21.15.16:0/276257180 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd228003f80 0x7fd228004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:08.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.005+0000 7fd23d59a700 1 --2- 172.21.15.16:0/276257180 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd228004d90 0x7fd228005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:08.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.005+0000 7fd23d59a700 1 -- 172.21.15.16:0/276257180 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd24015ea20 con 0x7fd228004d90 2023-10-16T12:24:08.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.005+0000 7fd23d59a700 1 -- 172.21.15.16:0/276257180 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd24015e080 con 0x7fd228003f80 2023-10-16T12:24:08.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.005+0000 7fd23d59a700 1 -- 172.21.15.16:0/276257180 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd22800dc70 con 0x7fd24006bad0 2023-10-16T12:24:08.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.005+0000 7fd23ed9d700 1 --2- 172.21.15.16:0/276257180 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd228004d90 0x7fd228005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:08.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.005+0000 7fd23e59c700 1 --2- 172.21.15.16:0/276257180 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd24006bad0 0x7fd228003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:08.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.005+0000 7fd23dd9b700 1 --2- 172.21.15.16:0/276257180 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd228003f80 0x7fd228004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:08.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.005+0000 7fd23ed9d700 1 -- 172.21.15.16:0/276257180 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd24006bad0 msgr2=0x7fd228003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:24:08.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.005+0000 7fd23ed9d700 1 --2- 172.21.15.16:0/276257180 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd24006bad0 0x7fd228003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:08.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.005+0000 7fd23ed9d700 1 -- 172.21.15.16:0/276257180 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd228003f80 msgr2=0x7fd228004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:24:08.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.005+0000 7fd23ed9d700 1 --2- 172.21.15.16:0/276257180 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd228003f80 0x7fd228004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:08.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.005+0000 7fd23ed9d700 1 -- 172.21.15.16:0/276257180 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2280035b0 con 0x7fd228004d90 2023-10-16T12:24:08.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.005+0000 7fd23ed9d700 1 --2- 172.21.15.16:0/276257180 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd228004d90 0x7fd228005210 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fd238009fc0 tx=0x7fd228005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43871fb22a488b51 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:24:08.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.005+0000 7fd23d59a700 1 -- 172.21.15.16:0/276257180 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd23800ec30 con 0x7fd228004d90 2023-10-16T12:24:08.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.005+0000 7fd23d59a700 1 -- 172.21.15.16:0/276257180 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd23800f040 con 0x7fd228004d90 2023-10-16T12:24:08.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.005+0000 7fd23d59a700 1 -- 172.21.15.16:0/276257180 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd238014480 con 0x7fd228004d90 2023-10-16T12:24:08.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.006+0000 7fd244bec700 1 -- 172.21.15.16:0/276257180 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd228004d90 msgr2=0x7fd228005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:08.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.006+0000 7fd244bec700 1 --2- 172.21.15.16:0/276257180 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd228004d90 0x7fd228005210 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fd238009fc0 tx=0x7fd228005880 comp rx=0 tx=0).stop 2023-10-16T12:24:08.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.006+0000 7fd23ed9d700 1 -- 172.21.15.16:0/276257180 reap_dead start 2023-10-16T12:24:08.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.006+0000 7fd244bec700 1 -- 172.21.15.16:0/276257180 shutdown_connections 2023-10-16T12:24:08.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.006+0000 7fd244bec700 1 -- 172.21.15.16:0/276257180 >> 172.21.15.16:0/276257180 conn(0x7fd2400cc5e0 msgr2=0x7fd2400cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:24:08.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.006+0000 7fd244bec700 1 -- 172.21.15.16:0/276257180 shutdown_connections 2023-10-16T12:24:08.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.006+0000 7fd244bec700 1 -- 172.21.15.16:0/276257180 wait complete. 2023-10-16T12:24:08.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.007+0000 7fd244bec700 1 Processor -- start 2023-10-16T12:24:08.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.008+0000 7fd244bec700 1 -- start start 2023-10-16T12:24:08.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.008+0000 7fd244bec700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd228004d90 0x7fd2400c57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:08.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.008+0000 7fd244bec700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd24014c6b0 0x7fd240156650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:08.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.008+0000 7fd244bec700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd2400b3940 0x7fd24006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:08.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.008+0000 7fd244bec700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2400c5d00 con 0x7fd228004d90 2023-10-16T12:24:08.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.008+0000 7fd244bec700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2400b28a0 con 0x7fd24014c6b0 2023-10-16T12:24:08.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.008+0000 7fd244bec700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2400b2b80 con 0x7fd2400b3940 2023-10-16T12:24:08.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.008+0000 7fd23e59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd228004d90 0x7fd2400c57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:08.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.008+0000 7fd23e59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd228004d90 0x7fd2400c57c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37846/0 (socket says 172.21.15.16:37846) 2023-10-16T12:24:08.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.008+0000 7fd23e59c700 1 -- 172.21.15.16:0/2032675031 learned_addr learned my addr 172.21.15.16:0/2032675031 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:24:08.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.009+0000 7fd23dd9b700 1 --2- 172.21.15.16:0/2032675031 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd24014c6b0 0x7fd240156650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:08.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.009+0000 7fd23ed9d700 1 --2- 172.21.15.16:0/2032675031 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd2400b3940 0x7fd24006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:08.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.009+0000 7fd23e59c700 1 -- 172.21.15.16:0/2032675031 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd2400b3940 msgr2=0x7fd24006e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:08.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.009+0000 7fd23e59c700 1 --2- 172.21.15.16:0/2032675031 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd2400b3940 0x7fd24006e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:08.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.009+0000 7fd23e59c700 1 -- 172.21.15.16:0/2032675031 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd24014c6b0 msgr2=0x7fd240156650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:08.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.009+0000 7fd23e59c700 1 --2- 172.21.15.16:0/2032675031 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd24014c6b0 0x7fd240156650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:08.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.009+0000 7fd23e59c700 1 -- 172.21.15.16:0/2032675031 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2400b2e60 con 0x7fd228004d90 2023-10-16T12:24:08.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.009+0000 7fd23dd9b700 1 --2- 172.21.15.16:0/2032675031 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd24014c6b0 0x7fd240156650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:24:08.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.009+0000 7fd23ed9d700 1 --2- 172.21.15.16:0/2032675031 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd2400b3940 0x7fd24006e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:24:08.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.009+0000 7fd23e59c700 1 --2- 172.21.15.16:0/2032675031 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd228004d90 0x7fd2400c57c0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fd230001cc0 tx=0x7fd2300076a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:24:08.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.009+0000 7fd22effd700 1 -- 172.21.15.16:0/2032675031 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd23000f8a0 con 0x7fd228004d90 2023-10-16T12:24:08.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.009+0000 7fd22effd700 1 -- 172.21.15.16:0/2032675031 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd23000fa40 con 0x7fd228004d90 2023-10-16T12:24:08.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.009+0000 7fd244bec700 1 -- 172.21.15.16:0/2032675031 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2400b3140 con 0x7fd228004d90 2023-10-16T12:24:08.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.009+0000 7fd22effd700 1 -- 172.21.15.16:0/2032675031 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd23000fdf0 con 0x7fd228004d90 2023-10-16T12:24:08.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.009+0000 7fd244bec700 1 -- 172.21.15.16:0/2032675031 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd240156b90 con 0x7fd228004d90 2023-10-16T12:24:08.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.010+0000 7fd244bec700 1 -- 172.21.15.16:0/2032675031 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd24015dda0 con 0x7fd228004d90 2023-10-16T12:24:08.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.011+0000 7fd22effd700 1 -- 172.21.15.16:0/2032675031 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd23001d460 con 0x7fd228004d90 2023-10-16T12:24:08.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.011+0000 7fd22effd700 1 --2- 172.21.15.16:0/2032675031 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fd22006b020 0x7fd22006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:08.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.011+0000 7fd23dd9b700 1 --2- 172.21.15.16:0/2032675031 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fd22006b020 0x7fd22006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:08.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.011+0000 7fd22effd700 1 -- 172.21.15.16:0/2032675031 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fd230089a50 con 0x7fd228004d90 2023-10-16T12:24:08.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.012+0000 7fd23dd9b700 1 --2- 172.21.15.16:0/2032675031 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fd22006b020 0x7fd22006d4e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd24014e0b0 tx=0x7fd234008040 comp rx=0 tx=0).ready entity=mgr.5874 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:24:08.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.016+0000 7fd22effd700 1 -- 172.21.15.16:0/2032675031 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd2300548b0 con 0x7fd228004d90 2023-10-16T12:24:08.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.305+0000 7fd244bec700 1 -- 172.21.15.16:0/2032675031 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fd24004ebb0 con 0x7fd228004d90 2023-10-16T12:24:08.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.307+0000 7fd22effd700 1 -- 172.21.15.16:0/2032675031 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v18) v1 ==== 190+0+147 (secure 0 0 0) 0x7fd230057ed0 con 0x7fd228004d90 2023-10-16T12:24:08.309 INFO:teuthology.orchestra.run.smithi016.stdout:[client.0] 2023-10-16T12:24:08.309 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAPKC1l2l6PFxAA9QyXtBAHdGAXpwH0JfknuQ== 2023-10-16T12:24:08.309 INFO:teuthology.orchestra.run.smithi016.stdout: caps mds = "allow" 2023-10-16T12:24:08.309 INFO:teuthology.orchestra.run.smithi016.stdout: caps mon = "allow r" 2023-10-16T12:24:08.310 INFO:teuthology.orchestra.run.smithi016.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-16T12:24:08.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.309+0000 7fd244bec700 1 -- 172.21.15.16:0/2032675031 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fd22006b020 msgr2=0x7fd22006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:08.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.309+0000 7fd244bec700 1 --2- 172.21.15.16:0/2032675031 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fd22006b020 0x7fd22006d4e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd24014e0b0 tx=0x7fd234008040 comp rx=0 tx=0).stop 2023-10-16T12:24:08.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.309+0000 7fd244bec700 1 -- 172.21.15.16:0/2032675031 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd228004d90 msgr2=0x7fd2400c57c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:08.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.309+0000 7fd244bec700 1 --2- 172.21.15.16:0/2032675031 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd228004d90 0x7fd2400c57c0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fd230001cc0 tx=0x7fd2300076a0 comp rx=0 tx=0).stop 2023-10-16T12:24:08.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.309+0000 7fd244bec700 1 -- 172.21.15.16:0/2032675031 shutdown_connections 2023-10-16T12:24:08.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.309+0000 7fd244bec700 1 --2- 172.21.15.16:0/2032675031 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fd22006b020 0x7fd22006d4e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:08.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.309+0000 7fd244bec700 1 --2- 172.21.15.16:0/2032675031 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd2400b3940 0x7fd24006e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:08.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.309+0000 7fd244bec700 1 --2- 172.21.15.16:0/2032675031 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd228004d90 0x7fd2400c57c0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:08.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.309+0000 7fd244bec700 1 --2- 172.21.15.16:0/2032675031 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd24014c6b0 0x7fd240156650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:08.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.309+0000 7fd244bec700 1 -- 172.21.15.16:0/2032675031 >> 172.21.15.16:0/2032675031 conn(0x7fd2400cc5e0 msgr2=0x7fd240156f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:24:08.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.309+0000 7fd244bec700 1 -- 172.21.15.16:0/2032675031 shutdown_connections 2023-10-16T12:24:08.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.309+0000 7fd244bec700 1 -- 172.21.15.16:0/2032675031 wait complete. 2023-10-16T12:24:08.314 INFO:teuthology.orchestra.run.smithi016.stderr:updated caps for client.0 2023-10-16T12:24:08.323 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:24:08.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.446+0000 7f06c7708700 1 Processor -- start 2023-10-16T12:24:08.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.447+0000 7f06c7708700 1 -- start start 2023-10-16T12:24:08.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.447+0000 7f06c7708700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f06c00cf880 0x7f06c00b52f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:08.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.447+0000 7f06c7708700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06c015bc80 con 0x7f06c00b5830 2023-10-16T12:24:08.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.447+0000 7f06c7708700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06c015b070 con 0x7f06c00cf4a0 2023-10-16T12:24:08.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.447+0000 7f06c7708700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06c015b350 con 0x7f06c00cf880 2023-10-16T12:24:08.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.447+0000 7f06c5ca5700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f06c00b5830 0x7f06c015a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:55364/0 (socket says 172.21.15.16:55364) 2023-10-16T12:24:08.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.447+0000 7f06c5ca5700 1 -- 172.21.15.16:0/411646249 learned_addr learned my addr 172.21.15.16:0/411646249 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:24:08.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.448+0000 7f06c4ca3700 1 --2- 172.21.15.16:0/411646249 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f06c00cf880 0x7f06c00b52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:08.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.448+0000 7f06b3fff700 1 -- 172.21.15.16:0/411646249 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3459376842 0 0) 0x7f06c015bc80 con 0x7f06c00b5830 2023-10-16T12:24:08.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.448+0000 7f06b3fff700 1 -- 172.21.15.16:0/411646249 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06ac0035b0 con 0x7f06c00b5830 2023-10-16T12:24:08.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.448+0000 7f06b3fff700 1 -- 172.21.15.16:0/411646249 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1895599260 0 0) 0x7f06ac0035b0 con 0x7f06c00b5830 2023-10-16T12:24:08.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.448+0000 7f06b3fff700 1 -- 172.21.15.16:0/411646249 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f06c00cf880 msgr2=0x7f06c00b52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:24:08.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.448+0000 7f06b3fff700 1 --2- 172.21.15.16:0/411646249 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f06c00cf880 0x7f06c00b52f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:08.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.448+0000 7f06b3fff700 1 -- 172.21.15.16:0/411646249 >> v1:172.21.15.49:6789/0 conn(0x7f06c00cf4a0 legacy=0x7f06c00726e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:08.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.448+0000 7f06b3fff700 1 -- 172.21.15.16:0/411646249 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06c015b630 con 0x7f06c00b5830 2023-10-16T12:24:08.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.448+0000 7f06b3fff700 1 -- 172.21.15.16:0/411646249 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f06b80019f0 con 0x7f06c00b5830 2023-10-16T12:24:08.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.448+0000 7f06b3fff700 1 -- 172.21.15.16:0/411646249 >> v1:172.21.15.16:6789/0 conn(0x7f06c00b5830 legacy=0x7f06c015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:08.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.448+0000 7f06b3fff700 1 --2- 172.21.15.16:0/411646249 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f06ac003a30 0x7f06ac003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:08.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.448+0000 7f06b3fff700 1 --2- 172.21.15.16:0/411646249 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06ac0043e0 0x7f06ac0048b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:08.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.448+0000 7f06b3fff700 1 --2- 172.21.15.16:0/411646249 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f06c00cf880 0x7f06ac005110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:08.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.448+0000 7f06b3fff700 1 -- 172.21.15.16:0/411646249 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06c015b350 con 0x7f06ac0043e0 2023-10-16T12:24:08.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.448+0000 7f06b3fff700 1 -- 172.21.15.16:0/411646249 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06ac00ba40 con 0x7f06ac003a30 2023-10-16T12:24:08.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.448+0000 7f06b3fff700 1 -- 172.21.15.16:0/411646249 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06ac00bc00 con 0x7f06c00cf880 2023-10-16T12:24:08.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.448+0000 7f06c4ca3700 1 --2- 172.21.15.16:0/411646249 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06ac0043e0 0x7f06ac0048b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:08.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.448+0000 7f06c54a4700 1 --2- 172.21.15.16:0/411646249 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f06ac003a30 0x7f06ac003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:08.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.448+0000 7f06c5ca5700 1 --2- 172.21.15.16:0/411646249 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f06c00cf880 0x7f06ac005110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:08.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.449+0000 7f06c4ca3700 1 -- 172.21.15.16:0/411646249 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f06c00cf880 msgr2=0x7f06ac005110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:24:08.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.449+0000 7f06c4ca3700 1 --2- 172.21.15.16:0/411646249 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f06c00cf880 0x7f06ac005110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:08.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.449+0000 7f06c4ca3700 1 -- 172.21.15.16:0/411646249 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f06ac003a30 msgr2=0x7f06ac003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:24:08.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.449+0000 7f06c4ca3700 1 --2- 172.21.15.16:0/411646249 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f06ac003a30 0x7f06ac003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:08.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.449+0000 7f06c4ca3700 1 -- 172.21.15.16:0/411646249 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06ac0035b0 con 0x7f06ac0043e0 2023-10-16T12:24:08.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.449+0000 7f06c4ca3700 1 --2- 172.21.15.16:0/411646249 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06ac0043e0 0x7f06ac0048b0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f06b4009b60 tx=0x7f06ac005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1010a75d2aa7ced server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:24:08.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.449+0000 7f06b3fff700 1 -- 172.21.15.16:0/411646249 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06b400e6c0 con 0x7f06ac0043e0 2023-10-16T12:24:08.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.449+0000 7f06b3fff700 1 -- 172.21.15.16:0/411646249 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06b400e820 con 0x7f06ac0043e0 2023-10-16T12:24:08.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.449+0000 7f06b3fff700 1 -- 172.21.15.16:0/411646249 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06b400eb90 con 0x7f06ac0043e0 2023-10-16T12:24:08.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.449+0000 7f06c7708700 1 -- 172.21.15.16:0/411646249 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06ac0043e0 msgr2=0x7f06ac0048b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:08.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.449+0000 7f06c7708700 1 --2- 172.21.15.16:0/411646249 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06ac0043e0 0x7f06ac0048b0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f06b4009b60 tx=0x7f06ac005880 comp rx=0 tx=0).stop 2023-10-16T12:24:08.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.449+0000 7f06c5ca5700 1 -- 172.21.15.16:0/411646249 reap_dead start 2023-10-16T12:24:08.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.449+0000 7f06c7708700 1 -- 172.21.15.16:0/411646249 shutdown_connections 2023-10-16T12:24:08.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.449+0000 7f06c7708700 1 -- 172.21.15.16:0/411646249 >> 172.21.15.16:0/411646249 conn(0x7f06c00b1fd0 msgr2=0x7f06c00b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:24:08.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.450+0000 7f06c7708700 1 -- 172.21.15.16:0/411646249 shutdown_connections 2023-10-16T12:24:08.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.450+0000 7f06c7708700 1 -- 172.21.15.16:0/411646249 wait complete. 2023-10-16T12:24:08.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.451+0000 7f06c7708700 1 Processor -- start 2023-10-16T12:24:08.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.451+0000 7f06c7708700 1 -- start start 2023-10-16T12:24:08.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.451+0000 7f06c7708700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f06ac0043e0 0x7f06c0151e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:08.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.452+0000 7f06c7708700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f06c00c9f80 0x7f06c00b4140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:08.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.452+0000 7f06c7708700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06c00c5030 0x7f06c00b4680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:08.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.452+0000 7f06c7708700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06c00c58e0 con 0x7f06c00c5030 2023-10-16T12:24:08.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.452+0000 7f06c7708700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06c01563a0 con 0x7f06ac0043e0 2023-10-16T12:24:08.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.452+0000 7f06c7708700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06c0156bb0 con 0x7f06c00c9f80 2023-10-16T12:24:08.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.452+0000 7f06c5ca5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06c00c5030 0x7f06c00b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:08.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.452+0000 7f06c5ca5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06c00c5030 0x7f06c00b4680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37870/0 (socket says 172.21.15.16:37870) 2023-10-16T12:24:08.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.452+0000 7f06c5ca5700 1 -- 172.21.15.16:0/2302939868 learned_addr learned my addr 172.21.15.16:0/2302939868 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:24:08.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.452+0000 7f06c4ca3700 1 --2- 172.21.15.16:0/2302939868 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f06c00c9f80 0x7f06c00b4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:08.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.452+0000 7f06c54a4700 1 --2- 172.21.15.16:0/2302939868 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f06ac0043e0 0x7f06c0151e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:08.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.452+0000 7f06c5ca5700 1 -- 172.21.15.16:0/2302939868 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f06c00c9f80 msgr2=0x7f06c00b4140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:08.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.452+0000 7f06c5ca5700 1 --2- 172.21.15.16:0/2302939868 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f06c00c9f80 0x7f06c00b4140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:08.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.452+0000 7f06c5ca5700 1 -- 172.21.15.16:0/2302939868 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f06ac0043e0 msgr2=0x7f06c0151e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:08.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.452+0000 7f06c5ca5700 1 --2- 172.21.15.16:0/2302939868 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f06ac0043e0 0x7f06c0151e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:08.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.452+0000 7f06c5ca5700 1 -- 172.21.15.16:0/2302939868 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06c01565f0 con 0x7f06c00c5030 2023-10-16T12:24:08.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.452+0000 7f06c54a4700 1 --2- 172.21.15.16:0/2302939868 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f06ac0043e0 0x7f06c0151e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:24:08.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.452+0000 7f06c5ca5700 1 --2- 172.21.15.16:0/2302939868 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06c00c5030 0x7f06c00b4680 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f06b8000f80 tx=0x7f06b800a640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:24:08.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.452+0000 7f06b1ffb700 1 -- 172.21.15.16:0/2302939868 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06b800ae10 con 0x7f06c00c5030 2023-10-16T12:24:08.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.452+0000 7f06b1ffb700 1 -- 172.21.15.16:0/2302939868 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06b80133f0 con 0x7f06c00c5030 2023-10-16T12:24:08.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.452+0000 7f06b1ffb700 1 -- 172.21.15.16:0/2302939868 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06b80137a0 con 0x7f06c00c5030 2023-10-16T12:24:08.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.452+0000 7f06c7708700 1 -- 172.21.15.16:0/2302939868 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06c01568d0 con 0x7f06c00c5030 2023-10-16T12:24:08.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.453+0000 7f06c7708700 1 -- 172.21.15.16:0/2302939868 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06c00ca840 con 0x7f06c00c5030 2023-10-16T12:24:08.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.453+0000 7f06c7708700 1 -- 172.21.15.16:0/2302939868 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06c00c5a20 con 0x7f06c00c5030 2023-10-16T12:24:08.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.454+0000 7f06b1ffb700 1 -- 172.21.15.16:0/2302939868 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f06b8013900 con 0x7f06c00c5030 2023-10-16T12:24:08.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.454+0000 7f06b1ffb700 1 --2- 172.21.15.16:0/2302939868 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f06a406b020 0x7f06a406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:08.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.454+0000 7f06b1ffb700 1 -- 172.21.15.16:0/2302939868 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f06b808a460 con 0x7f06c00c5030 2023-10-16T12:24:08.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.454+0000 7f06c54a4700 1 --2- 172.21.15.16:0/2302939868 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f06a406b020 0x7f06a406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:08.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.455+0000 7f06c54a4700 1 --2- 172.21.15.16:0/2302939868 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f06a406b020 0x7f06a406d4e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f06bc000c00 tx=0x7f06bc009040 comp rx=0 tx=0).ready entity=mgr.5874 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:24:08.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.459+0000 7f06b1ffb700 1 -- 172.21.15.16:0/2302939868 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f06b8055350 con 0x7f06c00c5030 2023-10-16T12:24:08.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.707+0000 7f06c7708700 1 -- 172.21.15.16:0/2302939868 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f06c004ebb0 con 0x7f06c00c5030 2023-10-16T12:24:08.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.708+0000 7f06b1ffb700 1 -- 172.21.15.16:0/2302939868 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+4913 (secure 0 0 0) 0x7f06b8058970 con 0x7f06c00c5030 2023-10-16T12:24:08.710 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:24:08.711 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5947,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/1103417847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":1103417847},{"type":"v1","addr":"172.21.15.49:6835","nonce":1103417847}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":6096,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/1380792179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1380792179},{"type":"v1","addr":"172.21.15.16:6836","nonce":1380792179}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":6099,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2238585443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":2238585443},{"type":"v1","addr":"172.21.15.16:6837","nonce":2238585443}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:24:02.695223+0000","modified":"2023-10-16T12:24:05.704039+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6111},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6111":{"gid":6111,"name":"b","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.49:6834/3246624425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":3246624425},{"type":"v1","addr":"172.21.15.49:6834","nonce":3246624425}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-10-16T12:24:08.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.710+0000 7f06c7708700 1 -- 172.21.15.16:0/2302939868 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f06a406b020 msgr2=0x7f06a406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:08.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.710+0000 7f06c7708700 1 --2- 172.21.15.16:0/2302939868 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f06a406b020 0x7f06a406d4e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f06bc000c00 tx=0x7f06bc009040 comp rx=0 tx=0).stop 2023-10-16T12:24:08.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.710+0000 7f06c7708700 1 -- 172.21.15.16:0/2302939868 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06c00c5030 msgr2=0x7f06c00b4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:08.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.710+0000 7f06c7708700 1 --2- 172.21.15.16:0/2302939868 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06c00c5030 0x7f06c00b4680 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f06b8000f80 tx=0x7f06b800a640 comp rx=0 tx=0).stop 2023-10-16T12:24:08.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.710+0000 7f06c7708700 1 -- 172.21.15.16:0/2302939868 shutdown_connections 2023-10-16T12:24:08.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.710+0000 7f06c7708700 1 --2- 172.21.15.16:0/2302939868 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f06a406b020 0x7f06a406d4e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:08.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.710+0000 7f06c7708700 1 --2- 172.21.15.16:0/2302939868 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06c00c5030 0x7f06c00b4680 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:08.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.710+0000 7f06c7708700 1 --2- 172.21.15.16:0/2302939868 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f06ac0043e0 0x7f06c0151e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:08.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.710+0000 7f06c7708700 1 --2- 172.21.15.16:0/2302939868 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f06c00c9f80 0x7f06c00b4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:08.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.710+0000 7f06c7708700 1 -- 172.21.15.16:0/2302939868 >> 172.21.15.16:0/2302939868 conn(0x7f06c00b1fd0 msgr2=0x7f06c00b03c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:24:08.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.710+0000 7f06c7708700 1 -- 172.21.15.16:0/2302939868 shutdown_connections 2023-10-16T12:24:08.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:08.710+0000 7f06c7708700 1 -- 172.21.15.16:0/2302939868 wait complete. 2023-10-16T12:24:08.716 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 27 2023-10-16T12:24:09.127 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T12:24:02.695223+0000', 'modified': '2023-10-16T12:24:05.704039+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6111}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6111': {'gid': 6111, 'name': 'b', 'rank': 0, 'incarnation': 23, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.49:6834/3246624425', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6832', 'nonce': 3246624425}, {'type': 'v1', 'addr': '172.21.15.49:6834', 'nonce': 3246624425}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [8], 'metadata_pool': 7, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-16T12:24:09.128 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-16T12:24:09.128 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-10-16T12:24:09.258 INFO:tasks.ceph.mds.b.smithi049.stderr:2023-10-16T12:24:09.256+0000 7f6237f99700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T12:24:09.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.263+0000 7f0cce8f4700 1 Processor -- start 2023-10-16T12:24:09.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.263+0000 7f0cce8f4700 1 -- start start 2023-10-16T12:24:09.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.264+0000 7f0cce8f4700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0cc815b280 0x7f0cc81613e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:09.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.264+0000 7f0cce8f4700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f0cc80cc8a0 0x7f0cc815e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:09.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.264+0000 7f0cce8f4700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cc8161c30 con 0x7f0cc806a7e0 2023-10-16T12:24:09.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.264+0000 7f0cce8f4700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cc8161f10 con 0x7f0cc80cc8a0 2023-10-16T12:24:09.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.264+0000 7f0cce8f4700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cc81621f0 con 0x7f0cc815b280 2023-10-16T12:24:09.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.264+0000 7f0ccce91700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0cc806a7e0 0x7f0cc80d02b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:55384/0 (socket says 172.21.15.16:55384) 2023-10-16T12:24:09.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.264+0000 7f0ccce91700 1 -- 172.21.15.16:0/140771469 learned_addr learned my addr 172.21.15.16:0/140771469 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:24:09.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.264+0000 7f0cc77fe700 1 --2- 172.21.15.16:0/140771469 >> v2:172.21.15.49:3300/0 conn(0x7f0cc80cc8a0 0x7f0cc815e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:09.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.264+0000 7f0cc7fff700 1 --2- 172.21.15.16:0/140771469 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0cc815b280 0x7f0cc81613e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:09.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.264+0000 7f0cc6ffd700 1 -- 172.21.15.16:0/140771469 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3186302926 0 0) 0x7f0cc8161c30 con 0x7f0cc806a7e0 2023-10-16T12:24:09.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.264+0000 7f0cc6ffd700 1 -- 172.21.15.16:0/140771469 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cb00035b0 con 0x7f0cc806a7e0 2023-10-16T12:24:09.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.264+0000 7f0cc6ffd700 1 -- 172.21.15.16:0/140771469 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1362918810 0 0) 0x7f0cb00035b0 con 0x7f0cc806a7e0 2023-10-16T12:24:09.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.264+0000 7f0cc6ffd700 1 -- 172.21.15.16:0/140771469 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0cc815b280 msgr2=0x7f0cc81613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:24:09.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.264+0000 7f0cc6ffd700 1 --2- 172.21.15.16:0/140771469 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0cc815b280 0x7f0cc81613e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:09.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.264+0000 7f0cc6ffd700 1 -- 172.21.15.16:0/140771469 >> v2:172.21.15.49:3300/0 conn(0x7f0cc80cc8a0 msgr2=0x7f0cc815e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:24:09.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.264+0000 7f0cc6ffd700 1 --2- 172.21.15.16:0/140771469 >> v2:172.21.15.49:3300/0 conn(0x7f0cc80cc8a0 0x7f0cc815e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:09.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.264+0000 7f0cc6ffd700 1 -- 172.21.15.16:0/140771469 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cc815df00 con 0x7f0cc806a7e0 2023-10-16T12:24:09.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.264+0000 7f0cc6ffd700 1 -- 172.21.15.16:0/140771469 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f0cc00019f0 con 0x7f0cc806a7e0 2023-10-16T12:24:09.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.264+0000 7f0cc6ffd700 1 -- 172.21.15.16:0/140771469 >> v1:172.21.15.16:6789/0 conn(0x7f0cc806a7e0 legacy=0x7f0cc80d02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:09.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.264+0000 7f0cc6ffd700 1 --2- 172.21.15.16:0/140771469 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0cb0003a00 0x7f0cb0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:09.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.264+0000 7f0cc6ffd700 1 --2- 172.21.15.16:0/140771469 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0cb0004850 0x7f0cb0004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:09.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.264+0000 7f0cc6ffd700 1 --2- 172.21.15.16:0/140771469 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0cc815b280 0x7f0cb0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:09.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.265+0000 7f0cc6ffd700 1 -- 172.21.15.16:0/140771469 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cc8161f10 con 0x7f0cb0004850 2023-10-16T12:24:09.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.265+0000 7f0cc6ffd700 1 -- 172.21.15.16:0/140771469 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cc81621f0 con 0x7f0cb0003a00 2023-10-16T12:24:09.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.265+0000 7f0cc6ffd700 1 -- 172.21.15.16:0/140771469 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb000ba10 con 0x7f0cc815b280 2023-10-16T12:24:09.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.265+0000 7f0cc77fe700 1 --2- 172.21.15.16:0/140771469 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0cb0004850 0x7f0cb0004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:09.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.265+0000 7f0cc7fff700 1 --2- 172.21.15.16:0/140771469 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0cb0003a00 0x7f0cb0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:09.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.265+0000 7f0ccce91700 1 --2- 172.21.15.16:0/140771469 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0cc815b280 0x7f0cb0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:09.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.265+0000 7f0cc77fe700 1 -- 172.21.15.16:0/140771469 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0cc815b280 msgr2=0x7f0cb0005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:24:09.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.265+0000 7f0cc77fe700 1 --2- 172.21.15.16:0/140771469 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0cc815b280 0x7f0cb0005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:09.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.265+0000 7f0cc77fe700 1 -- 172.21.15.16:0/140771469 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0cb0003a00 msgr2=0x7f0cb0003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:24:09.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.265+0000 7f0cc77fe700 1 --2- 172.21.15.16:0/140771469 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0cb0003a00 0x7f0cb0003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:09.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.265+0000 7f0cc77fe700 1 -- 172.21.15.16:0/140771469 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cb00035b0 con 0x7f0cb0004850 2023-10-16T12:24:09.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.265+0000 7f0ccce91700 1 --2- 172.21.15.16:0/140771469 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0cc815b280 0x7f0cb0005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:24:09.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.265+0000 7f0cc77fe700 1 --2- 172.21.15.16:0/140771469 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0cb0004850 0x7f0cb0004cd0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc009b60 tx=0x7f0cb0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d60e04010c33a87 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:24:09.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.265+0000 7f0cc6ffd700 1 -- 172.21.15.16:0/140771469 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cbc00e6c0 con 0x7f0cb0004850 2023-10-16T12:24:09.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.265+0000 7f0cc6ffd700 1 -- 172.21.15.16:0/140771469 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cbc00e820 con 0x7f0cb0004850 2023-10-16T12:24:09.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.265+0000 7f0cc6ffd700 1 -- 172.21.15.16:0/140771469 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cbc00eb90 con 0x7f0cb0004850 2023-10-16T12:24:09.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.266+0000 7f0cce8f4700 1 -- 172.21.15.16:0/140771469 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0cb0004850 msgr2=0x7f0cb0004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:09.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.266+0000 7f0cce8f4700 1 --2- 172.21.15.16:0/140771469 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0cb0004850 0x7f0cb0004cd0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc009b60 tx=0x7f0cb0005880 comp rx=0 tx=0).stop 2023-10-16T12:24:09.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.266+0000 7f0ccce91700 1 -- 172.21.15.16:0/140771469 reap_dead start 2023-10-16T12:24:09.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.266+0000 7f0cce8f4700 1 -- 172.21.15.16:0/140771469 shutdown_connections 2023-10-16T12:24:09.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.266+0000 7f0cce8f4700 1 -- 172.21.15.16:0/140771469 >> 172.21.15.16:0/140771469 conn(0x7f0cc80b1fd0 msgr2=0x7f0cc80b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:24:09.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.266+0000 7f0cce8f4700 1 -- 172.21.15.16:0/140771469 shutdown_connections 2023-10-16T12:24:09.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.266+0000 7f0cce8f4700 1 -- 172.21.15.16:0/140771469 wait complete. 2023-10-16T12:24:09.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.267+0000 7f0cce8f4700 1 Processor -- start 2023-10-16T12:24:09.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.267+0000 7f0cce8f4700 1 -- start start 2023-10-16T12:24:09.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.268+0000 7f0cce8f4700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0cb0004850 0x7f0cc81f2880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:09.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.268+0000 7f0cce8f4700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0cc80c4760 0x7f0cc80c4be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:09.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.268+0000 7f0cce8f4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0cc8157c80 0x7f0cc8158110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:09.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.268+0000 7f0cce8f4700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cc80c5120 con 0x7f0cc8157c80 2023-10-16T12:24:09.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.268+0000 7f0cce8f4700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cc81571f0 con 0x7f0cb0004850 2023-10-16T12:24:09.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.268+0000 7f0cce8f4700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cc8159510 con 0x7f0cc80c4760 2023-10-16T12:24:09.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.268+0000 7f0ccce91700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0cc8157c80 0x7f0cc8158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:09.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.268+0000 7f0ccce91700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0cc8157c80 0x7f0cc8158110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37890/0 (socket says 172.21.15.16:37890) 2023-10-16T12:24:09.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.268+0000 7f0ccce91700 1 -- 172.21.15.16:0/3833565862 learned_addr learned my addr 172.21.15.16:0/3833565862 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:24:09.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.268+0000 7f0cc7fff700 1 --2- 172.21.15.16:0/3833565862 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0cb0004850 0x7f0cc81f2880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:09.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.268+0000 7f0cc77fe700 1 --2- 172.21.15.16:0/3833565862 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0cc80c4760 0x7f0cc80c4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:09.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.268+0000 7f0ccce91700 1 -- 172.21.15.16:0/3833565862 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0cc80c4760 msgr2=0x7f0cc80c4be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:09.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.268+0000 7f0ccce91700 1 --2- 172.21.15.16:0/3833565862 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0cc80c4760 0x7f0cc80c4be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:09.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.268+0000 7f0ccce91700 1 -- 172.21.15.16:0/3833565862 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0cb0004850 msgr2=0x7f0cc81f2880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:09.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.268+0000 7f0ccce91700 1 --2- 172.21.15.16:0/3833565862 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0cb0004850 0x7f0cc81f2880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:09.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.268+0000 7f0ccce91700 1 -- 172.21.15.16:0/3833565862 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cc8158650 con 0x7f0cc8157c80 2023-10-16T12:24:09.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.269+0000 7f0ccce91700 1 --2- 172.21.15.16:0/3833565862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0cc8157c80 0x7f0cc8158110 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f0cc806b8a0 tx=0x7f0cc000e440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:24:09.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.269+0000 7f0cc4ff9700 1 -- 172.21.15.16:0/3833565862 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cc0017070 con 0x7f0cc8157c80 2023-10-16T12:24:09.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.269+0000 7f0cc4ff9700 1 -- 172.21.15.16:0/3833565862 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cc000ec10 con 0x7f0cc8157c80 2023-10-16T12:24:09.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.269+0000 7f0cce8f4700 1 -- 172.21.15.16:0/3833565862 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cc8158900 con 0x7f0cc8157c80 2023-10-16T12:24:09.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.269+0000 7f0cc4ff9700 1 -- 172.21.15.16:0/3833565862 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cc000eda0 con 0x7f0cc8157c80 2023-10-16T12:24:09.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.269+0000 7f0cce8f4700 1 -- 172.21.15.16:0/3833565862 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cc8158ec0 con 0x7f0cc8157c80 2023-10-16T12:24:09.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.272+0000 7f0cc4ff9700 1 -- 172.21.15.16:0/3833565862 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0cc0013420 con 0x7f0cc8157c80 2023-10-16T12:24:09.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.272+0000 7f0cc4ff9700 1 --2- 172.21.15.16:0/3833565862 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f0cb406b020 0x7f0cb406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:09.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.272+0000 7f0cc4ff9700 1 -- 172.21.15.16:0/3833565862 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f0cc00895b0 con 0x7f0cc8157c80 2023-10-16T12:24:09.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.273+0000 7f0cc7fff700 1 --2- 172.21.15.16:0/3833565862 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f0cb406b020 0x7f0cb406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:09.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.273+0000 7f0cdaadeb80 1 Processor -- start 2023-10-16T12:24:09.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.273+0000 7f0cdaadeb80 1 -- start start 2023-10-16T12:24:09.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.273+0000 7f0cc7fff700 1 --2- 172.21.15.16:0/3833565862 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f0cb406b020 0x7f0cb406d4e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0cb8009b80 tx=0x7f0cb8009210 comp rx=0 tx=0).ready entity=mgr.5874 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:24:09.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.274+0000 7f0cdaadeb80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55582df08220 0x55582df0fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:09.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.274+0000 7f0cdaadeb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55582df0d630 0x55582df0ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:09.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.274+0000 7f0cdaadeb80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55582dee3010 0x55582dee36c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:09.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.274+0000 7f0cdaadeb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55582de59c20 con 0x55582df0d630 2023-10-16T12:24:09.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.274+0000 7f0cdaadeb80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x55582de3ec10 con 0x55582dee3010 2023-10-16T12:24:09.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.274+0000 7f0cdaadeb80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x55582de3c530 con 0x55582df08220 2023-10-16T12:24:09.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.274+0000 7f0cc77fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55582df0d630 0x55582df0ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:09.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.274+0000 7f0cc77fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55582df0d630 0x55582df0ffb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37896/0 (socket says 172.21.15.16:37896) 2023-10-16T12:24:09.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.274+0000 7f0cc77fe700 1 -- 172.21.15.16:0/2445594988 learned_addr learned my addr 172.21.15.16:0/2445594988 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:24:09.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.274+0000 7f0ccce91700 1 --2- 172.21.15.16:0/2445594988 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55582dee3010 0x55582dee36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:09.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.274+0000 7f0cc7fff700 1 --2- 172.21.15.16:0/2445594988 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55582df08220 0x55582df0fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:09.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.274+0000 7f0cc77fe700 1 -- 172.21.15.16:0/2445594988 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55582df08220 msgr2=0x55582df0fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:24:09.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.274+0000 7f0cc77fe700 1 --2- 172.21.15.16:0/2445594988 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55582df08220 0x55582df0fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:09.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.274+0000 7f0cc77fe700 1 -- 172.21.15.16:0/2445594988 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55582dee3010 msgr2=0x55582dee36c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:24:09.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.274+0000 7f0cc77fe700 1 --2- 172.21.15.16:0/2445594988 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55582dee3010 0x55582dee36c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:09.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.274+0000 7f0cc77fe700 1 -- 172.21.15.16:0/2445594988 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55582ddf53a0 con 0x55582df0d630 2023-10-16T12:24:09.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.274+0000 7f0ccce91700 1 --2- 172.21.15.16:0/2445594988 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55582dee3010 0x55582dee36c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:24:09.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.274+0000 7f0cc77fe700 1 --2- 172.21.15.16:0/2445594988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55582df0d630 0x55582df0ffb0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc00a010 tx=0x7f0cbc008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef7ff0f9334edac7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:24:09.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.274+0000 7f0cadffb700 1 -- 172.21.15.16:0/2445594988 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cbc022040 con 0x55582df0d630 2023-10-16T12:24:09.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.274+0000 7f0cadffb700 1 -- 172.21.15.16:0/2445594988 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cbc00eb90 con 0x55582df0d630 2023-10-16T12:24:09.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.274+0000 7f0cadffb700 1 -- 172.21.15.16:0/2445594988 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cbc00e6c0 con 0x55582df0d630 2023-10-16T12:24:09.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.275+0000 7f0cdaadeb80 1 -- 172.21.15.16:0/2445594988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55582df0d630 msgr2=0x55582df0ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:09.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.275+0000 7f0cdaadeb80 1 --2- 172.21.15.16:0/2445594988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55582df0d630 0x55582df0ffb0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc00a010 tx=0x7f0cbc008030 comp rx=0 tx=0).stop 2023-10-16T12:24:09.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.275+0000 7f0cdaadeb80 1 -- 172.21.15.16:0/2445594988 shutdown_connections 2023-10-16T12:24:09.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.275+0000 7f0cdaadeb80 1 --2- 172.21.15.16:0/2445594988 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55582dee3010 0x55582dee36c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:09.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.275+0000 7f0cdaadeb80 1 --2- 172.21.15.16:0/2445594988 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55582df08220 0x55582df0fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:09.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.275+0000 7f0cdaadeb80 1 --2- 172.21.15.16:0/2445594988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55582df0d630 0x55582df0ffb0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:09.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.275+0000 7f0cdaadeb80 1 -- 172.21.15.16:0/2445594988 >> 172.21.15.16:0/2445594988 conn(0x55582de12d80 msgr2=0x55582df2dff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:24:09.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.275+0000 7f0cdaadeb80 1 -- 172.21.15.16:0/2445594988 shutdown_connections 2023-10-16T12:24:09.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.275+0000 7f0cdaadeb80 1 -- 172.21.15.16:0/2445594988 wait complete. 2023-10-16T12:24:09.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.276+0000 7f0cdaadeb80 1 -- start start 2023-10-16T12:24:09.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.276+0000 7f0cdaadeb80 1 Processor -- start 2023-10-16T12:24:09.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.277+0000 7f0cdaadeb80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55582dee3010 0x55582df39b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:09.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.277+0000 7f0cdaadeb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55582df08220 0x55582df08600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:09.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.277+0000 7f0cdaadeb80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55582df0d630 0x55582df12fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:09.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.277+0000 7f0cdaadeb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55582de37e20 con 0x55582df08220 2023-10-16T12:24:09.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.277+0000 7f0cdaadeb80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x55582ddd46c0 con 0x55582dee3010 2023-10-16T12:24:09.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.277+0000 7f0cdaadeb80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x55582ddd4360 con 0x55582df0d630 2023-10-16T12:24:09.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.277+0000 7f0cdaadeb80 10 client.0 _ll_register_callbacks cb 0x55582dc317b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T12:24:09.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.277+0000 7f0cc77fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55582df08220 0x55582df08600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:09.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.277+0000 7f0cc77fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55582df08220 0x55582df08600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37902/0 (socket says 172.21.15.16:37902) 2023-10-16T12:24:09.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.277+0000 7f0cc77fe700 1 -- 172.21.15.16:0/3290414368 learned_addr learned my addr 172.21.15.16:0/3290414368 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:24:09.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.277+0000 7f0ccce91700 1 --2- 172.21.15.16:0/3290414368 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55582df0d630 0x55582df12fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:09.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.277+0000 7f0cc7fff700 1 --2- 172.21.15.16:0/3290414368 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55582dee3010 0x55582df39b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:09.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.277+0000 7f0cc77fe700 1 -- 172.21.15.16:0/3290414368 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55582df0d630 msgr2=0x55582df12fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:24:09.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.277+0000 7f0cc77fe700 1 --2- 172.21.15.16:0/3290414368 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55582df0d630 0x55582df12fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:09.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.277+0000 7f0cc77fe700 1 -- 172.21.15.16:0/3290414368 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55582dee3010 msgr2=0x55582df39b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:24:09.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.277+0000 7f0cc77fe700 1 --2- 172.21.15.16:0/3290414368 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55582dee3010 0x55582df39b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:09.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.277+0000 7f0cc77fe700 1 -- 172.21.15.16:0/3290414368 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cb00035b0 con 0x55582df08220 2023-10-16T12:24:09.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.277+0000 7f0ccce91700 1 --2- 172.21.15.16:0/3290414368 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55582df0d630 0x55582df12fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:24:09.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.277+0000 7f0cc77fe700 1 --2- 172.21.15.16:0/3290414368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55582df08220 0x55582df08600 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f0cb00046a0 tx=0x7f0cbc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f94cfe387678f05 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:24:09.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.277+0000 7f0ca6ffd700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2023-10-16T12:24:09.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.277+0000 7f0ca6ffd700 1 -- 172.21.15.16:0/3290414368 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cbc022070 con 0x55582df08220 2023-10-16T12:24:09.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.277+0000 7f0ca6ffd700 1 -- 172.21.15.16:0/3290414368 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(68..68 src has 1..68) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f0cbc004240 con 0x55582df08220 2023-10-16T12:24:09.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.277+0000 7f0cdaadeb80 1 -- 172.21.15.16:0/3290414368 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55582ded85b0 con 0x55582df08220 2023-10-16T12:24:09.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.278+0000 7f0ca6ffd700 1 -- 172.21.15.16:0/3290414368 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0cbc004400 con 0x55582df08220 2023-10-16T12:24:09.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.278+0000 7f0cdaadeb80 10 client.6249 fetch_fsmap learned FSMap version 27 2023-10-16T12:24:09.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.278+0000 7f0cdaadeb80 1 -- 172.21.15.16:0/3290414368 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=2+}) v3 -- 0x55582dc62d90 con 0x55582df08220 2023-10-16T12:24:09.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.278+0000 7f0ca6ffd700 1 -- 172.21.15.16:0/3290414368 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55582ded85b0 con 0x55582df08220 2023-10-16T12:24:09.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.278+0000 7f0ca6ffd700 1 -- 172.21.15.16:0/3290414368 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f0cbc038070 con 0x55582df08220 2023-10-16T12:24:09.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.278+0000 7f0cdaadeb80 10 client.6249 fetch_fsmap finished waiting for FSMap version 27 2023-10-16T12:24:09.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.278+0000 7f0cdaadeb80 10 client.6249 resolve_mds: resolved name 'b' to daemon mds.b 2023-10-16T12:24:09.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.278+0000 7f0cdaadeb80 20 client.6249 populate_metadata read hostname 'smithi016' 2023-10-16T12:24:09.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.278+0000 7f0cdaadeb80 1 --2- 172.21.15.16:0/3290414368 >> [v2:172.21.15.49:6832/3246624425,v1:172.21.15.49:6834/3246624425] conn(0x55582deda130 0x55582deda510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:09.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.278+0000 7f0cdaadeb80 4 client.6249 mds_command: new command op to 6111 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T12:24:09.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.278+0000 7f0cdaadeb80 1 -- 172.21.15.16:0/3290414368 --> [v2:172.21.15.49:6832/3246624425,v1:172.21.15.49:6834/3246624425] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55582de30e00 con 0x55582deda130 2023-10-16T12:24:09.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.278+0000 7f0cc7fff700 1 --2- 172.21.15.16:0/3290414368 >> [v2:172.21.15.49:6832/3246624425,v1:172.21.15.49:6834/3246624425] conn(0x55582deda130 0x55582deda510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:09.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.279+0000 7f0ca6ffd700 10 client.6249 ms_handle_connect on v2:172.21.15.49:6832/3246624425 2023-10-16T12:24:09.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.279+0000 7f0cc7fff700 1 --2- 172.21.15.16:0/3290414368 >> [v2:172.21.15.49:6832/3246624425,v1:172.21.15.49:6834/3246624425] conn(0x55582deda130 0x55582deda510 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cd3143d005f46d17 server_cookie=8e448e5634acd9d in_seq=0 out_seq=0 2023-10-16T12:24:09.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.279+0000 7f0ca6ffd700 1 -- 172.21.15.16:0/3290414368 <== mds.0 v2:172.21.15.49:6832/3246624425 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55582de30e00 con 0x55582deda130 2023-10-16T12:24:09.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.279+0000 7f0ca6ffd700 10 client.6249 handle_command_reply: tid=0 2023-10-16T12:24:09.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.279+0000 7f0cdaadeb80 1 client.6249 shutdown 2023-10-16T12:24:09.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.279+0000 7f0cdaadeb80 1 -- 172.21.15.16:0/3290414368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55582df08220 msgr2=0x55582df08600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:09.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.279+0000 7f0cdaadeb80 1 --2- 172.21.15.16:0/3290414368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55582df08220 0x55582df08600 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f0cb00046a0 tx=0x7f0cbc004750 comp rx=0 tx=0).stop 2023-10-16T12:24:09.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.280+0000 7f0cdaadeb80 1 -- 172.21.15.16:0/3290414368 shutdown_connections 2023-10-16T12:24:09.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.280+0000 7f0cdaadeb80 1 --2- 172.21.15.16:0/3290414368 >> [v2:172.21.15.49:6832/3246624425,v1:172.21.15.49:6834/3246624425] conn(0x55582deda130 0x55582deda510 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:09.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.280+0000 7f0cdaadeb80 1 --2- 172.21.15.16:0/3290414368 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55582df0d630 0x55582df12fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:09.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.280+0000 7f0cdaadeb80 1 --2- 172.21.15.16:0/3290414368 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55582dee3010 0x55582df39b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:09.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.280+0000 7f0ca6ffd700 0 client.6249 ms_handle_reset on v2:172.21.15.49:6832/3246624425 2023-10-16T12:24:09.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.280+0000 7f0cdaadeb80 1 --2- 172.21.15.16:0/3290414368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55582df08220 0x55582df08600 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:09.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.280+0000 7f0cdaadeb80 1 -- 172.21.15.16:0/3290414368 >> 172.21.15.16:0/3290414368 conn(0x55582de12d80 msgr2=0x55582df2b4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:24:09.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.280+0000 7f0cdaadeb80 1 -- 172.21.15.16:0/3290414368 shutdown_connections 2023-10-16T12:24:09.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.280+0000 7f0cdaadeb80 1 -- 172.21.15.16:0/3290414368 wait complete. 2023-10-16T12:24:09.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.280+0000 7f0cdaadeb80 20 client.6249 trim_cache size 0 max 16384 2023-10-16T12:24:09.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.298+0000 7f0cdaadeb80 1 Processor -- start 2023-10-16T12:24:09.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.298+0000 7f0cdaadeb80 1 -- start start 2023-10-16T12:24:09.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.299+0000 7f0cdaadeb80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55582dee3010 0x55582df24420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:09.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.299+0000 7f0cdaadeb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55582df0d630 0x55582df4c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:09.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.299+0000 7f0cdaadeb80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55582df0b080 0x55582dee3620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:09.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.299+0000 7f0cdaadeb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55582dcb88d0 con 0x55582df0d630 2023-10-16T12:24:09.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.299+0000 7f0cdaadeb80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x55582dee1670 con 0x55582dee3010 2023-10-16T12:24:09.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.299+0000 7f0cdaadeb80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x55582df4ca00 con 0x55582df0b080 2023-10-16T12:24:09.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.299+0000 7f0cc77fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55582df0d630 0x55582df4c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:09.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.299+0000 7f0cc77fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55582df0d630 0x55582df4c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37908/0 (socket says 172.21.15.16:37908) 2023-10-16T12:24:09.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.299+0000 7f0cc77fe700 1 -- 172.21.15.16:0/1313616057 learned_addr learned my addr 172.21.15.16:0/1313616057 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:24:09.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.299+0000 7f0cc7fff700 1 --2- 172.21.15.16:0/1313616057 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55582dee3010 0x55582df24420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:09.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.299+0000 7f0ccce91700 1 --2- 172.21.15.16:0/1313616057 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55582df0b080 0x55582dee3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:09.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.299+0000 7f0cc77fe700 1 -- 172.21.15.16:0/1313616057 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55582df0b080 msgr2=0x55582dee3620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:24:09.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.299+0000 7f0cc77fe700 1 --2- 172.21.15.16:0/1313616057 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55582df0b080 0x55582dee3620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:09.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.299+0000 7f0cc77fe700 1 -- 172.21.15.16:0/1313616057 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55582dee3010 msgr2=0x55582df24420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:24:09.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.299+0000 7f0cc77fe700 1 --2- 172.21.15.16:0/1313616057 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55582dee3010 0x55582df24420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:09.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.299+0000 7f0cc77fe700 1 -- 172.21.15.16:0/1313616057 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55582dee1f20 con 0x55582df0d630 2023-10-16T12:24:09.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.299+0000 7f0cc7fff700 1 --2- 172.21.15.16:0/1313616057 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55582dee3010 0x55582df24420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:24:09.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.299+0000 7f0cc77fe700 1 --2- 172.21.15.16:0/1313616057 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55582df0d630 0x55582df4c4c0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc003a30 tx=0x7f0cbc02e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfc2270d681e9800 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:24:09.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.299+0000 7f0cae7fc700 1 -- 172.21.15.16:0/1313616057 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cbc00e990 con 0x55582df0d630 2023-10-16T12:24:09.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.299+0000 7f0cae7fc700 1 -- 172.21.15.16:0/1313616057 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cbc006c60 con 0x55582df0d630 2023-10-16T12:24:09.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.299+0000 7f0cae7fc700 1 -- 172.21.15.16:0/1313616057 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cbc004240 con 0x55582df0d630 2023-10-16T12:24:09.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.300+0000 7f0cdaadeb80 1 -- 172.21.15.16:0/1313616057 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55582df0d630 msgr2=0x55582df4c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:09.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.300+0000 7f0cdaadeb80 1 --2- 172.21.15.16:0/1313616057 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55582df0d630 0x55582df4c4c0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc003a30 tx=0x7f0cbc02e000 comp rx=0 tx=0).stop 2023-10-16T12:24:09.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.300+0000 7f0cdaadeb80 1 -- 172.21.15.16:0/1313616057 shutdown_connections 2023-10-16T12:24:09.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.300+0000 7f0cdaadeb80 1 --2- 172.21.15.16:0/1313616057 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55582df0b080 0x55582dee3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:09.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.300+0000 7f0cdaadeb80 1 --2- 172.21.15.16:0/1313616057 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55582dee3010 0x55582df24420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:09.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.300+0000 7f0cdaadeb80 1 --2- 172.21.15.16:0/1313616057 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55582df0d630 0x55582df4c4c0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:09.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.300+0000 7f0cdaadeb80 1 -- 172.21.15.16:0/1313616057 >> 172.21.15.16:0/1313616057 conn(0x55582de12d80 msgr2=0x55582df2e420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:24:09.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.300+0000 7f0cdaadeb80 1 -- 172.21.15.16:0/1313616057 shutdown_connections 2023-10-16T12:24:09.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.300+0000 7f0cdaadeb80 1 -- 172.21.15.16:0/1313616057 wait complete. 2023-10-16T12:24:09.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.301+0000 7f0cdaadeb80 1 -- start start 2023-10-16T12:24:09.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.301+0000 7f0cdaadeb80 1 Processor -- start 2023-10-16T12:24:09.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.302+0000 7f0cdaadeb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55582dee3010 0x55582df90e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:09.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.302+0000 7f0cdaadeb80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55582df0b080 0x55582dee5920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:09.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.302+0000 7f0cdaadeb80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55582df0d630 0x55582dee5e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:09.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.302+0000 7f0cdaadeb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55582df27410 con 0x55582dee3010 2023-10-16T12:24:09.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.302+0000 7f0cdaadeb80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x55582df91370 con 0x55582df0d630 2023-10-16T12:24:09.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.302+0000 7f0cdaadeb80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x55582dee33f0 con 0x55582df0b080 2023-10-16T12:24:09.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.302+0000 7f0cdaadeb80 10 client.0 _ll_register_callbacks cb 0x7f0cbc02a070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T12:24:09.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.302+0000 7f0cc7fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55582dee3010 0x55582df90e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:09.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.302+0000 7f0cc7fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55582dee3010 0x55582df90e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37912/0 (socket says 172.21.15.16:37912) 2023-10-16T12:24:09.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.302+0000 7f0cc7fff700 1 -- 172.21.15.16:0/1961192622 learned_addr learned my addr 172.21.15.16:0/1961192622 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:24:09.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.302+0000 7f0cc77fe700 1 --2- 172.21.15.16:0/1961192622 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55582df0b080 0x55582dee5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:09.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.302+0000 7f0ccce91700 1 --2- 172.21.15.16:0/1961192622 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55582df0d630 0x55582dee5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:09.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.302+0000 7f0cc7fff700 1 -- 172.21.15.16:0/1961192622 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55582df0b080 msgr2=0x55582dee5920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:24:09.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.302+0000 7f0cc7fff700 1 --2- 172.21.15.16:0/1961192622 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55582df0b080 0x55582dee5920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:09.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.302+0000 7f0cc7fff700 1 -- 172.21.15.16:0/1961192622 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55582df0d630 msgr2=0x55582dee5e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:24:09.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.302+0000 7f0cc7fff700 1 --2- 172.21.15.16:0/1961192622 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55582df0d630 0x55582dee5e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:09.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.302+0000 7f0cc7fff700 1 -- 172.21.15.16:0/1961192622 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55582dc62d90 con 0x55582dee3010 2023-10-16T12:24:09.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.302+0000 7f0cc77fe700 1 --2- 172.21.15.16:0/1961192622 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55582df0b080 0x55582dee5920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:24:09.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.302+0000 7f0cc7fff700 1 --2- 172.21.15.16:0/1961192622 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55582dee3010 0x55582df90e30 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x55582df3a930 tx=0x7f0cb8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59190e0f937160f8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:24:09.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.302+0000 7f0ca6ffd700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2023-10-16T12:24:09.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.302+0000 7f0ca6ffd700 1 -- 172.21.15.16:0/1961192622 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cb8003e40 con 0x55582dee3010 2023-10-16T12:24:09.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.302+0000 7f0ca6ffd700 1 -- 172.21.15.16:0/1961192622 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(68..68 src has 1..68) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f0cb802a460 con 0x55582dee3010 2023-10-16T12:24:09.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.302+0000 7f0cdaadeb80 1 -- 172.21.15.16:0/1961192622 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0cbc00eb90 con 0x55582dee3010 2023-10-16T12:24:09.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.303+0000 7f0ca6ffd700 1 -- 172.21.15.16:0/1961192622 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0cb802a7a0 con 0x55582dee3010 2023-10-16T12:24:09.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.303+0000 7f0cdaadeb80 10 client.6255 fetch_fsmap learned FSMap version 27 2023-10-16T12:24:09.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.303+0000 7f0cdaadeb80 1 -- 172.21.15.16:0/1961192622 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=2+}) v3 -- 0x55582dc4bc10 con 0x55582dee3010 2023-10-16T12:24:09.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.303+0000 7f0ca6ffd700 1 -- 172.21.15.16:0/1961192622 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cbc00eb90 con 0x55582dee3010 2023-10-16T12:24:09.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.303+0000 7f0ca6ffd700 1 -- 172.21.15.16:0/1961192622 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 27) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f0cb802a900 con 0x55582dee3010 2023-10-16T12:24:09.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.303+0000 7f0cdaadeb80 10 client.6255 fetch_fsmap finished waiting for FSMap version 27 2023-10-16T12:24:09.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.303+0000 7f0cdaadeb80 10 client.6255 resolve_mds: resolved name 'b' to daemon mds.b 2023-10-16T12:24:09.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.303+0000 7f0cdaadeb80 20 client.6255 populate_metadata read hostname 'smithi016' 2023-10-16T12:24:09.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.303+0000 7f0cdaadeb80 1 --2- 172.21.15.16:0/1961192622 >> [v2:172.21.15.49:6832/3246624425,v1:172.21.15.49:6834/3246624425] conn(0x55582dedfef0 0x55582dee02d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:09.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.303+0000 7f0cdaadeb80 4 client.6255 mds_command: new command op to 6111 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T12:24:09.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.303+0000 7f0cdaadeb80 1 -- 172.21.15.16:0/1961192622 --> [v2:172.21.15.49:6832/3246624425,v1:172.21.15.49:6834/3246624425] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55582de30b20 con 0x55582dedfef0 2023-10-16T12:24:09.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.303+0000 7f0cc77fe700 1 --2- 172.21.15.16:0/1961192622 >> [v2:172.21.15.49:6832/3246624425,v1:172.21.15.49:6834/3246624425] conn(0x55582dedfef0 0x55582dee02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:09.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.304+0000 7f0cc77fe700 1 --2- 172.21.15.16:0/1961192622 >> [v2:172.21.15.49:6832/3246624425,v1:172.21.15.49:6834/3246624425] conn(0x55582dedfef0 0x55582dee02d0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9dc067d00ec02121 server_cookie=cd9e4bfdf0a854d3 in_seq=0 out_seq=0 2023-10-16T12:24:09.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.304+0000 7f0ca6ffd700 10 client.6255 ms_handle_connect on v2:172.21.15.49:6832/3246624425 2023-10-16T12:24:09.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.304+0000 7f0ca6ffd700 1 -- 172.21.15.16:0/1961192622 <== mds.0 v2:172.21.15.49:6832/3246624425 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+323 (crc 0 0 0) 0x55582de30b20 con 0x55582dedfef0 2023-10-16T12:24:09.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.304+0000 7f0ca6ffd700 10 client.6255 handle_command_reply: tid=0 2023-10-16T12:24:09.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.304+0000 7f0cdaadeb80 1 client.6255 shutdown 2023-10-16T12:24:09.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.304+0000 7f0cdaadeb80 1 -- 172.21.15.16:0/1961192622 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55582dee3010 msgr2=0x55582df90e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:09.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.304+0000 7f0cdaadeb80 1 --2- 172.21.15.16:0/1961192622 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55582dee3010 0x55582df90e30 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x55582df3a930 tx=0x7f0cb8004750 comp rx=0 tx=0).stop 2023-10-16T12:24:09.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.304+0000 7f0cdaadeb80 1 -- 172.21.15.16:0/1961192622 shutdown_connections 2023-10-16T12:24:09.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.304+0000 7f0cdaadeb80 1 --2- 172.21.15.16:0/1961192622 >> [v2:172.21.15.49:6832/3246624425,v1:172.21.15.49:6834/3246624425] conn(0x55582dedfef0 0x55582dee02d0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:09.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.304+0000 7f0cdaadeb80 1 --2- 172.21.15.16:0/1961192622 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55582df0d630 0x55582dee5e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:09.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.304+0000 7f0ca6ffd700 0 client.6255 ms_handle_reset on v2:172.21.15.49:6832/3246624425 2023-10-16T12:24:09.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.304+0000 7f0cdaadeb80 1 --2- 172.21.15.16:0/1961192622 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55582dee3010 0x55582df90e30 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:09.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.304+0000 7f0cdaadeb80 1 --2- 172.21.15.16:0/1961192622 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55582df0b080 0x55582dee5920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:09.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.304+0000 7f0cdaadeb80 1 -- 172.21.15.16:0/1961192622 >> 172.21.15.16:0/1961192622 conn(0x55582de12d80 msgr2=0x55582df12b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:24:09.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.304+0000 7f0cdaadeb80 1 -- 172.21.15.16:0/1961192622 shutdown_connections 2023-10-16T12:24:09.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.304+0000 7f0cdaadeb80 1 -- 172.21.15.16:0/1961192622 wait complete. 2023-10-16T12:24:09.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.304+0000 7f0cdaadeb80 20 client.6255 trim_cache size 0 max 16384 2023-10-16T12:24:09.330 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-10-16T12:24:09.331 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "5b28dfa1-b21f-474a-9b2e-9fb77af58254", 2023-10-16T12:24:09.331 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 0, 2023-10-16T12:24:09.331 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 6111, 2023-10-16T12:24:09.331 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2023-10-16T12:24:09.331 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2023-10-16T12:24:09.332 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2023-10-16T12:24:09.332 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 6.02424465, 2023-10-16T12:24:09.332 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 27, 2023-10-16T12:24:09.332 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 68, 2023-10-16T12:24:09.332 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 68, 2023-10-16T12:24:09.332 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 10.899668222000001 2023-10-16T12:24:09.333 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-10-16T12:24:09.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.305+0000 7f0cce8f4700 1 -- 172.21.15.16:0/3833565862 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f0cb406b020 msgr2=0x7f0cb406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:09.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.305+0000 7f0cce8f4700 1 --2- 172.21.15.16:0/3833565862 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f0cb406b020 0x7f0cb406d4e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0cb8009b80 tx=0x7f0cb8009210 comp rx=0 tx=0).stop 2023-10-16T12:24:09.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.305+0000 7f0cce8f4700 1 -- 172.21.15.16:0/3833565862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0cc8157c80 msgr2=0x7f0cc8158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:09.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.305+0000 7f0cce8f4700 1 --2- 172.21.15.16:0/3833565862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0cc8157c80 0x7f0cc8158110 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f0cc806b8a0 tx=0x7f0cc000e440 comp rx=0 tx=0).stop 2023-10-16T12:24:09.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.305+0000 7f0cce8f4700 1 -- 172.21.15.16:0/3833565862 shutdown_connections 2023-10-16T12:24:09.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.305+0000 7f0cce8f4700 1 --2- 172.21.15.16:0/3833565862 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f0cb406b020 0x7f0cb406d4e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:09.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.305+0000 7f0cce8f4700 1 --2- 172.21.15.16:0/3833565862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0cc8157c80 0x7f0cc8158110 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:09.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.305+0000 7f0cce8f4700 1 --2- 172.21.15.16:0/3833565862 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0cb0004850 0x7f0cc81f2880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:09.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.305+0000 7f0cce8f4700 1 --2- 172.21.15.16:0/3833565862 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0cc80c4760 0x7f0cc80c4be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:09.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.305+0000 7f0cce8f4700 1 -- 172.21.15.16:0/3833565862 >> 172.21.15.16:0/3833565862 conn(0x7f0cc80b1fd0 msgr2=0x7f0cc806abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:24:09.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.305+0000 7f0cce8f4700 1 -- 172.21.15.16:0/3833565862 shutdown_connections 2023-10-16T12:24:09.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.305+0000 7f0cce8f4700 1 -- 172.21.15.16:0/3833565862 wait complete. 2023-10-16T12:24:09.337 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T12:24:09.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.460+0000 7fe798cd2700 1 Processor -- start 2023-10-16T12:24:09.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.460+0000 7fe798cd2700 1 -- start start 2023-10-16T12:24:09.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.461+0000 7fe798cd2700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7940cb280 0x7fe79406d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:09.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.461+0000 7fe798cd2700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe79406b1f0 0x7fe79406b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:09.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.461+0000 7fe798cd2700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe79406c150 con 0x7fe794160eb0 2023-10-16T12:24:09.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.461+0000 7fe798cd2700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7940af370 con 0x7fe79406b1f0 2023-10-16T12:24:09.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.461+0000 7fe798cd2700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe79406bbe0 con 0x7fe7940cb280 2023-10-16T12:24:09.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.461+0000 7fe792d9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe79406b1f0 0x7fe79406b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:09.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.462+0000 7fe792d9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe79406b1f0 0x7fe79406b6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37924/0 (socket says 172.21.15.16:37924) 2023-10-16T12:24:09.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.462+0000 7fe792d9d700 1 -- 172.21.15.16:0/3341265336 learned_addr learned my addr 172.21.15.16:0/3341265336 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:24:09.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.462+0000 7fe791d9b700 1 --2- 172.21.15.16:0/3341265336 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7940cb280 0x7fe79406d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:09.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.462+0000 7fe792d9d700 1 -- 172.21.15.16:0/3341265336 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7940cb280 msgr2=0x7fe79406d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:24:09.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.462+0000 7fe792d9d700 1 --2- 172.21.15.16:0/3341265336 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7940cb280 0x7fe79406d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:09.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.462+0000 7fe792d9d700 1 -- 172.21.15.16:0/3341265336 >> v1:172.21.15.49:6789/0 conn(0x7fe794160eb0 legacy=0x7fe7940cab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:09.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.462+0000 7fe792d9d700 1 -- 172.21.15.16:0/3341265336 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe79406be70 con 0x7fe79406b1f0 2023-10-16T12:24:09.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.462+0000 7fe79159a700 1 -- 172.21.15.16:0/3341265336 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2104994457 0 0) 0x7fe79406c150 con 0x7fe794160eb0 2023-10-16T12:24:09.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.462+0000 7fe792d9d700 1 --2- 172.21.15.16:0/3341265336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe79406b1f0 0x7fe79406b6a0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fe78c009e50 tx=0x7fe78c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=649aaa4939da438a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:24:09.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.462+0000 7fe79159a700 1 -- 172.21.15.16:0/3341265336 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe78c00e070 con 0x7fe79406b1f0 2023-10-16T12:24:09.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.462+0000 7fe79159a700 1 -- 172.21.15.16:0/3341265336 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe78c00b910 con 0x7fe79406b1f0 2023-10-16T12:24:09.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.462+0000 7fe79159a700 1 -- 172.21.15.16:0/3341265336 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe78c00bcc0 con 0x7fe79406b1f0 2023-10-16T12:24:09.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.463+0000 7fe798cd2700 1 -- 172.21.15.16:0/3341265336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe79406b1f0 msgr2=0x7fe79406b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:09.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.463+0000 7fe798cd2700 1 --2- 172.21.15.16:0/3341265336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe79406b1f0 0x7fe79406b6a0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fe78c009e50 tx=0x7fe78c00b330 comp rx=0 tx=0).stop 2023-10-16T12:24:09.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.463+0000 7fe798cd2700 1 -- 172.21.15.16:0/3341265336 shutdown_connections 2023-10-16T12:24:09.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.463+0000 7fe798cd2700 1 --2- 172.21.15.16:0/3341265336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe79406b1f0 0x7fe79406b6a0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:09.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.463+0000 7fe798cd2700 1 --2- 172.21.15.16:0/3341265336 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7940cb280 0x7fe79406d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:09.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.463+0000 7fe798cd2700 1 -- 172.21.15.16:0/3341265336 >> 172.21.15.16:0/3341265336 conn(0x7fe7940cc5b0 msgr2=0x7fe7940cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:24:09.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.463+0000 7fe798cd2700 1 -- 172.21.15.16:0/3341265336 shutdown_connections 2023-10-16T12:24:09.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.463+0000 7fe798cd2700 1 -- 172.21.15.16:0/3341265336 wait complete. 2023-10-16T12:24:09.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.464+0000 7fe798cd2700 1 Processor -- start 2023-10-16T12:24:09.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.464+0000 7fe798cd2700 1 -- start start 2023-10-16T12:24:09.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.465+0000 7fe798cd2700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe79406b1f0 0x7fe7941534d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:09.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.465+0000 7fe798cd2700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7940cb280 0x7fe79414ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:09.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.465+0000 7fe798cd2700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe794160eb0 0x7fe794150900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:09.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.465+0000 7fe798cd2700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7940adfd0 con 0x7fe794160eb0 2023-10-16T12:24:09.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.465+0000 7fe798cd2700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe794150510 con 0x7fe79406b1f0 2023-10-16T12:24:09.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.465+0000 7fe798cd2700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7940b36f0 con 0x7fe7940cb280 2023-10-16T12:24:09.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.465+0000 7fe792d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe794160eb0 0x7fe794150900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:09.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.465+0000 7fe792d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe794160eb0 0x7fe794150900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37930/0 (socket says 172.21.15.16:37930) 2023-10-16T12:24:09.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.465+0000 7fe792d9d700 1 -- 172.21.15.16:0/1412380237 learned_addr learned my addr 172.21.15.16:0/1412380237 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:24:09.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.465+0000 7fe79259c700 1 --2- 172.21.15.16:0/1412380237 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe79406b1f0 0x7fe7941534d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:09.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.465+0000 7fe791d9b700 1 --2- 172.21.15.16:0/1412380237 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7940cb280 0x7fe79414ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:09.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.466+0000 7fe792d9d700 1 -- 172.21.15.16:0/1412380237 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7940cb280 msgr2=0x7fe79414ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:09.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.466+0000 7fe792d9d700 1 --2- 172.21.15.16:0/1412380237 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7940cb280 0x7fe79414ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:09.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.466+0000 7fe792d9d700 1 -- 172.21.15.16:0/1412380237 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe79406b1f0 msgr2=0x7fe7941534d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:09.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.466+0000 7fe792d9d700 1 --2- 172.21.15.16:0/1412380237 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe79406b1f0 0x7fe7941534d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:09.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.466+0000 7fe792d9d700 1 -- 172.21.15.16:0/1412380237 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe78c009b30 con 0x7fe794160eb0 2023-10-16T12:24:09.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.466+0000 7fe792d9d700 1 --2- 172.21.15.16:0/1412380237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe794160eb0 0x7fe794150900 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fe794072670 tx=0x7fe78c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:24:09.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.466+0000 7fe78affd700 1 -- 172.21.15.16:0/1412380237 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe78c003ea0 con 0x7fe794160eb0 2023-10-16T12:24:09.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.466+0000 7fe78affd700 1 -- 172.21.15.16:0/1412380237 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe78c004920 con 0x7fe794160eb0 2023-10-16T12:24:09.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.466+0000 7fe798cd2700 1 -- 172.21.15.16:0/1412380237 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7940c8dd0 con 0x7fe794160eb0 2023-10-16T12:24:09.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.466+0000 7fe78affd700 1 -- 172.21.15.16:0/1412380237 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe78c01c680 con 0x7fe794160eb0 2023-10-16T12:24:09.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.466+0000 7fe798cd2700 1 -- 172.21.15.16:0/1412380237 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7940c9390 con 0x7fe794160eb0 2023-10-16T12:24:09.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.467+0000 7fe798cd2700 1 -- 172.21.15.16:0/1412380237 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7940c8c10 con 0x7fe794160eb0 2023-10-16T12:24:09.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.467+0000 7fe78affd700 1 -- 172.21.15.16:0/1412380237 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe78c01c7e0 con 0x7fe794160eb0 2023-10-16T12:24:09.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.468+0000 7fe78affd700 1 --2- 172.21.15.16:0/1412380237 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fe77406b020 0x7fe77406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:09.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.468+0000 7fe78affd700 1 -- 172.21.15.16:0/1412380237 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fe78c08aa80 con 0x7fe794160eb0 2023-10-16T12:24:09.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.468+0000 7fe79259c700 1 --2- 172.21.15.16:0/1412380237 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fe77406b020 0x7fe77406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:09.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.468+0000 7fe79259c700 1 --2- 172.21.15.16:0/1412380237 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fe77406b020 0x7fe77406d4e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe784007fd0 tx=0x7fe784009040 comp rx=0 tx=0).ready entity=mgr.5874 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:24:09.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.472+0000 7fe78affd700 1 -- 172.21.15.16:0/1412380237 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe78c055740 con 0x7fe794160eb0 2023-10-16T12:24:09.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.723+0000 7fe798cd2700 1 -- 172.21.15.16:0/1412380237 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fe7940c95b0 con 0x7fe794160eb0 2023-10-16T12:24:09.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.723+0000 7fe78affd700 1 -- 172.21.15.16:0/1412380237 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v27) v1 ==== 53+0+73 (secure 0 0 0) 0x7fe78c058d60 con 0x7fe794160eb0 2023-10-16T12:24:09.726 INFO:teuthology.orchestra.run.smithi016.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-16T12:24:09.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.725+0000 7fe798cd2700 1 -- 172.21.15.16:0/1412380237 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fe77406b020 msgr2=0x7fe77406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:09.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.725+0000 7fe798cd2700 1 --2- 172.21.15.16:0/1412380237 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fe77406b020 0x7fe77406d4e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe784007fd0 tx=0x7fe784009040 comp rx=0 tx=0).stop 2023-10-16T12:24:09.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.725+0000 7fe798cd2700 1 -- 172.21.15.16:0/1412380237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe794160eb0 msgr2=0x7fe794150900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:09.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.725+0000 7fe798cd2700 1 --2- 172.21.15.16:0/1412380237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe794160eb0 0x7fe794150900 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fe794072670 tx=0x7fe78c004ad0 comp rx=0 tx=0).stop 2023-10-16T12:24:09.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.726+0000 7fe798cd2700 1 -- 172.21.15.16:0/1412380237 shutdown_connections 2023-10-16T12:24:09.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.726+0000 7fe798cd2700 1 --2- 172.21.15.16:0/1412380237 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fe77406b020 0x7fe77406d4e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:09.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.726+0000 7fe798cd2700 1 --2- 172.21.15.16:0/1412380237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe794160eb0 0x7fe794150900 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:09.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.726+0000 7fe798cd2700 1 --2- 172.21.15.16:0/1412380237 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe79406b1f0 0x7fe7941534d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:09.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.726+0000 7fe798cd2700 1 --2- 172.21.15.16:0/1412380237 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7940cb280 0x7fe79414ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:09.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.726+0000 7fe798cd2700 1 -- 172.21.15.16:0/1412380237 >> 172.21.15.16:0/1412380237 conn(0x7fe7940cc5b0 msgr2=0x7fe79415fee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:24:09.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.726+0000 7fe798cd2700 1 -- 172.21.15.16:0/1412380237 shutdown_connections 2023-10-16T12:24:09.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:09.726+0000 7fe798cd2700 1 -- 172.21.15.16:0/1412380237 wait complete. 2023-10-16T12:24:09.740 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-16T12:24:09.740 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T12:24:09.740 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi016.front.sepia.ceph.com 2023-10-16T12:24:09.741 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-16T12:24:09.741 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-16T12:24:09.741 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-16T12:24:09.742 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-16T12:24:09.742 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T12:24:09.742 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T12:24:09.743 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:24:09.743 DEBUG:teuthology.orchestra.run.smithi016:> ip addr 2023-10-16T12:24:09.761 INFO:teuthology.orchestra.run.smithi016.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T12:24:09.761 INFO:teuthology.orchestra.run.smithi016.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T12:24:09.761 INFO:teuthology.orchestra.run.smithi016.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T12:24:09.761 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:24:09.762 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 ::1/128 scope host 2023-10-16T12:24:09.762 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:24:09.762 INFO:teuthology.orchestra.run.smithi016.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T12:24:09.762 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e0 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:24:09.762 INFO:teuthology.orchestra.run.smithi016.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T12:24:09.763 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e1 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:24:09.763 INFO:teuthology.orchestra.run.smithi016.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T12:24:09.763 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:64 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:24:09.763 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.21.15.16/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-16T12:24:09.763 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft 41486sec preferred_lft 41486sec 2023-10-16T12:24:09.763 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::ec4:7aff:febd:1564/64 scope link 2023-10-16T12:24:09.764 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:24:09.764 INFO:teuthology.orchestra.run.smithi016.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T12:24:09.764 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:65 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:24:09.764 INFO:teuthology.orchestra.run.smithi016.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T12:24:09.764 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 36:cf:ea:25:db:f7 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:24:09.765 INFO:teuthology.orchestra.run.smithi016.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-16T12:24:09.765 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:24:09.765 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::cfc:c5ff:fe52:a786/64 scope link 2023-10-16T12:24:09.765 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:24:09.765 INFO:teuthology.orchestra.run.smithi016.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T12:24:09.765 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 6e:62:9d:a4:43:23 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T12:24:09.766 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::6c62:9dff:fea4:4323/64 scope link 2023-10-16T12:24:09.766 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:24:09.766 INFO:teuthology.orchestra.run.smithi016.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T12:24:09.766 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 36:cf:ea:25:db:f7 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-16T12:24:09.766 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::34cf:eaff:fe25:dbf7/64 scope link 2023-10-16T12:24:09.767 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:24:09.767 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:24:09.767 DEBUG:teuthology.orchestra.run.smithi016:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T12:24:09.827 INFO:teuthology.orchestra.run.smithi016.stdout:1 2023-10-16T12:24:09.828 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:24:09.828 DEBUG:teuthology.orchestra.run.smithi016:> route 2023-10-16T12:24:09.887 INFO:teuthology.orchestra.run.smithi016.stdout:Kernel IP routing table 2023-10-16T12:24:09.887 INFO:teuthology.orchestra.run.smithi016.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T12:24:09.888 INFO:teuthology.orchestra.run.smithi016.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-16T12:24:09.888 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-16T12:24:09.888 INFO:teuthology.orchestra.run.smithi016.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T12:24:09.888 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:24:09.889 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec bash -c ' 2023-10-16T12:24:09.889 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2023-10-16T12:24:09.889 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-16T12:24:09.890 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-16T12:24:09.890 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-16T12:24:09.890 DEBUG:teuthology.orchestra.run.smithi016:> ') 2023-10-16T12:24:09.994 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:24:09.995 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2023-10-16T12:24:10.012 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-16T12:24:10.012 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-16T12:24:10.013 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T12:24:10.013 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:24:10.013 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set brx.0 up 2023-10-16T12:24:10.080 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T12:24:10.080 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-16T12:24:10.138 INFO:teuthology.orchestra.run.smithi016.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-16T12:24:10.138 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T12:24:10.138 DEBUG:teuthology.orchestra.run.smithi016:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-16T12:24:10.194 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-16T12:24:10.195 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:24:10.195 DEBUG:teuthology.orchestra.run.smithi016:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T12:24:10.281 INFO:teuthology.orchestra.run.smithi016.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T12:24:10.282 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T12:24:10.282 INFO:teuthology.orchestra.run.smithi016.stdout:invalid new device string format 2023-10-16T12:24:10.291 INFO:teuthology.orchestra.run.smithi016.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T12:24:10.292 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T12:24:10.292 INFO:teuthology.orchestra.run.smithi016.stdout:invalid new device string format 2023-10-16T12:24:10.292 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: resolved to: "172.21.15.16:3300,172.21.15.49:3300,172.21.15.49:3301" 2023-10-16T12:24:10.292 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T12:24:10.326 INFO:teuthology.orchestra.run.smithi016.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T12:24:10.326 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T12:24:10.326 INFO:teuthology.orchestra.run.smithi016.stdout:invalid new device string format 2023-10-16T12:24:10.326 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: resolved to: "172.21.15.16:3300,172.21.15.49:3300,172.21.15.49:3301" 2023-10-16T12:24:10.327 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T12:24:10.327 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: resolved to: "172.21.15.16:6789,172.21.15.49:6789,172.21.15.49:6790" 2023-10-16T12:24:10.327 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.16:6789,172.21.15.49:6789,172.21.15.49:6790:/ 2023-10-16T12:24:10.327 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=5b28dfa1-b21f-474a-9b2e-9fb77af58254" will pass to kernel 2023-10-16T12:24:10.333 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-16T12:24:10.334 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:24:10.334 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-16T12:24:10.370 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:24:10.371 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T12:24:10.468 INFO:teuthology.orchestra.run.smithi016.stdout:client6264 2023-10-16T12:24:10.468 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:24:10.469 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:24:10.486 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-16T12:24:10.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.651+0000 7f321d21b700 1 Processor -- start 2023-10-16T12:24:10.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.651+0000 7f321d21b700 1 -- start start 2023-10-16T12:24:10.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.652+0000 7f321d21b700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f3218159c10 0x7f321806c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:10.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.652+0000 7f321d21b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f321806cd40 0x7f32180b3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:10.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.652+0000 7f321d21b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f32180b3b80 0x7f3218070e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:10.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.652+0000 7f321d21b700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32180b2b20 con 0x7f32180b3b80 2023-10-16T12:24:10.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.652+0000 7f321d21b700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f321806d1a0 con 0x7f3218159c10 2023-10-16T12:24:10.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.652+0000 7f321d21b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32180b21c0 con 0x7f321806cd40 2023-10-16T12:24:10.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.652+0000 7f321759e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f32180b3b80 0x7f3218070e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:10.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.652+0000 7f321759e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f32180b3b80 0x7f3218070e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37942/0 (socket says 172.21.15.16:37942) 2023-10-16T12:24:10.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.652+0000 7f321759e700 1 -- 172.21.15.16:0/3396328383 learned_addr learned my addr 172.21.15.16:0/3396328383 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:24:10.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.652+0000 7f321659c700 1 --2- 172.21.15.16:0/3396328383 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f321806cd40 0x7f32180b3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:10.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.652+0000 7f3216d9d700 1 --2- 172.21.15.16:0/3396328383 >> v2:172.21.15.49:3300/0 conn(0x7f3218159c10 0x7f321806c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:10.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.652+0000 7f321759e700 1 -- 172.21.15.16:0/3396328383 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f321806cd40 msgr2=0x7f32180b3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:24:10.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.652+0000 7f321759e700 1 --2- 172.21.15.16:0/3396328383 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f321806cd40 0x7f32180b3640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:10.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.652+0000 7f321759e700 1 -- 172.21.15.16:0/3396328383 >> v2:172.21.15.49:3300/0 conn(0x7f3218159c10 msgr2=0x7f321806c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:24:10.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.652+0000 7f321759e700 1 --2- 172.21.15.16:0/3396328383 >> v2:172.21.15.49:3300/0 conn(0x7f3218159c10 0x7f321806c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:10.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.652+0000 7f321759e700 1 -- 172.21.15.16:0/3396328383 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32180b2460 con 0x7f32180b3b80 2023-10-16T12:24:10.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.652+0000 7f321759e700 1 --2- 172.21.15.16:0/3396328383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32180b3b80 0x7f3218070e30 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f321806eb50 tx=0x7f321000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=104b55ddc2b7be8a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:24:10.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.652+0000 7f3215d9b700 1 -- 172.21.15.16:0/3396328383 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f321000e070 con 0x7f32180b3b80 2023-10-16T12:24:10.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.652+0000 7f3215d9b700 1 -- 172.21.15.16:0/3396328383 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f321000b9c0 con 0x7f32180b3b80 2023-10-16T12:24:10.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.652+0000 7f3215d9b700 1 -- 172.21.15.16:0/3396328383 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f321000bd70 con 0x7f32180b3b80 2023-10-16T12:24:10.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.653+0000 7f321d21b700 1 -- 172.21.15.16:0/3396328383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32180b3b80 msgr2=0x7f3218070e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:10.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.653+0000 7f321d21b700 1 --2- 172.21.15.16:0/3396328383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32180b3b80 0x7f3218070e30 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f321806eb50 tx=0x7f321000b330 comp rx=0 tx=0).stop 2023-10-16T12:24:10.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.653+0000 7f321d21b700 1 -- 172.21.15.16:0/3396328383 shutdown_connections 2023-10-16T12:24:10.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.653+0000 7f321d21b700 1 --2- 172.21.15.16:0/3396328383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32180b3b80 0x7f3218070e30 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:10.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.653+0000 7f321d21b700 1 --2- 172.21.15.16:0/3396328383 >> v2:172.21.15.49:3300/0 conn(0x7f3218159c10 0x7f321806c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:10.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.653+0000 7f321d21b700 1 --2- 172.21.15.16:0/3396328383 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f321806cd40 0x7f32180b3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:10.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.653+0000 7f321d21b700 1 -- 172.21.15.16:0/3396328383 >> 172.21.15.16:0/3396328383 conn(0x7f32180b4cd0 msgr2=0x7f321806a7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:24:10.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.653+0000 7f321d21b700 1 -- 172.21.15.16:0/3396328383 shutdown_connections 2023-10-16T12:24:10.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.653+0000 7f321d21b700 1 -- 172.21.15.16:0/3396328383 wait complete. 2023-10-16T12:24:10.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.654+0000 7f321d21b700 1 Processor -- start 2023-10-16T12:24:10.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.655+0000 7f321d21b700 1 -- start start 2023-10-16T12:24:10.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.655+0000 7f321d21b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f321806cd40 0x7f3218152e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:10.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.655+0000 7f321d21b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f32180b3b80 0x7f3218155730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:10.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.655+0000 7f321d21b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3218159c10 0x7f32180710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:10.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.655+0000 7f321d21b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3218156e00 con 0x7f3218159c10 2023-10-16T12:24:10.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.655+0000 7f321d21b700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3218156110 con 0x7f321806cd40 2023-10-16T12:24:10.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.655+0000 7f321d21b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32181563f0 con 0x7f32180b3b80 2023-10-16T12:24:10.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.655+0000 7f321759e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3218159c10 0x7f32180710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:10.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.655+0000 7f321759e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3218159c10 0x7f32180710f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37948/0 (socket says 172.21.15.16:37948) 2023-10-16T12:24:10.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.655+0000 7f321759e700 1 -- 172.21.15.16:0/1825436794 learned_addr learned my addr 172.21.15.16:0/1825436794 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:24:10.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.656+0000 7f321659c700 1 --2- 172.21.15.16:0/1825436794 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f32180b3b80 0x7f3218155730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:10.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.656+0000 7f3216d9d700 1 --2- 172.21.15.16:0/1825436794 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f321806cd40 0x7f3218152e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:10.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.656+0000 7f321759e700 1 -- 172.21.15.16:0/1825436794 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f32180b3b80 msgr2=0x7f3218155730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:10.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.656+0000 7f321759e700 1 --2- 172.21.15.16:0/1825436794 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f32180b3b80 0x7f3218155730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:10.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.656+0000 7f321759e700 1 -- 172.21.15.16:0/1825436794 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f321806cd40 msgr2=0x7f3218152e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:10.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.656+0000 7f321759e700 1 --2- 172.21.15.16:0/1825436794 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f321806cd40 0x7f3218152e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:10.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.656+0000 7f321759e700 1 -- 172.21.15.16:0/1825436794 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3210009b30 con 0x7f3218159c10 2023-10-16T12:24:10.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.656+0000 7f321759e700 1 --2- 172.21.15.16:0/1825436794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3218159c10 0x7f32180710f0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f321806d470 tx=0x7f3210004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:24:10.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.656+0000 7f31ff7fe700 1 -- 172.21.15.16:0/1825436794 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3210004250 con 0x7f3218159c10 2023-10-16T12:24:10.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.656+0000 7f31ff7fe700 1 -- 172.21.15.16:0/1825436794 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32100043d0 con 0x7f3218159c10 2023-10-16T12:24:10.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.656+0000 7f31ff7fe700 1 -- 172.21.15.16:0/1825436794 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3210004780 con 0x7f3218159c10 2023-10-16T12:24:10.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.656+0000 7f321d21b700 1 -- 172.21.15.16:0/1825436794 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32181566d0 con 0x7f3218159c10 2023-10-16T12:24:10.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.656+0000 7f321d21b700 1 -- 172.21.15.16:0/1825436794 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3218153380 con 0x7f3218159c10 2023-10-16T12:24:10.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.657+0000 7f321d21b700 1 -- 172.21.15.16:0/1825436794 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3218156f40 con 0x7f3218159c10 2023-10-16T12:24:10.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.658+0000 7f31ff7fe700 1 -- 172.21.15.16:0/1825436794 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f321001c440 con 0x7f3218159c10 2023-10-16T12:24:10.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.658+0000 7f31ff7fe700 1 --2- 172.21.15.16:0/1825436794 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f320406b020 0x7f320406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:10.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.658+0000 7f31ff7fe700 1 -- 172.21.15.16:0/1825436794 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f321008b1e0 con 0x7f3218159c10 2023-10-16T12:24:10.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.658+0000 7f3216d9d700 1 --2- 172.21.15.16:0/1825436794 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f320406b020 0x7f320406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:10.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.659+0000 7f3216d9d700 1 --2- 172.21.15.16:0/1825436794 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f320406b020 0x7f320406d4e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3208009930 tx=0x7f3208008040 comp rx=0 tx=0).ready entity=mgr.5874 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:24:10.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.662+0000 7f31ff7fe700 1 -- 172.21.15.16:0/1825436794 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3210055fa0 con 0x7f3218159c10 2023-10-16T12:24:10.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.897+0000 7f321d21b700 1 -- 172.21.15.16:0/1825436794 --> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f32180b43c0 con 0x7f320406b020 2023-10-16T12:24:10.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.898+0000 7f31ff7fe700 1 -- 172.21.15.16:0/1825436794 <== mgr.5874 v2:172.21.15.16:6832/3870937344 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f3208009e00 con 0x7f320406b020 2023-10-16T12:24:10.901 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2023-10-16T12:24:10.901 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:24:10.901 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cephfs" 2023-10-16T12:24:10.901 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:24:10.901 INFO:teuthology.orchestra.run.smithi016.stdout:] 2023-10-16T12:24:10.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.900+0000 7f321d21b700 1 -- 172.21.15.16:0/1825436794 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f320406b020 msgr2=0x7f320406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:10.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.900+0000 7f321d21b700 1 --2- 172.21.15.16:0/1825436794 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f320406b020 0x7f320406d4e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3208009930 tx=0x7f3208008040 comp rx=0 tx=0).stop 2023-10-16T12:24:10.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.901+0000 7f321d21b700 1 -- 172.21.15.16:0/1825436794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3218159c10 msgr2=0x7f32180710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:10.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.901+0000 7f321d21b700 1 --2- 172.21.15.16:0/1825436794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3218159c10 0x7f32180710f0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f321806d470 tx=0x7f3210004ad0 comp rx=0 tx=0).stop 2023-10-16T12:24:10.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.901+0000 7f321d21b700 1 -- 172.21.15.16:0/1825436794 shutdown_connections 2023-10-16T12:24:10.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.901+0000 7f321d21b700 1 --2- 172.21.15.16:0/1825436794 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f320406b020 0x7f320406d4e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:10.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.901+0000 7f321d21b700 1 --2- 172.21.15.16:0/1825436794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3218159c10 0x7f32180710f0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:10.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.901+0000 7f321d21b700 1 --2- 172.21.15.16:0/1825436794 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f321806cd40 0x7f3218152e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:10.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.901+0000 7f321d21b700 1 --2- 172.21.15.16:0/1825436794 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f32180b3b80 0x7f3218155730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:10.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.901+0000 7f321d21b700 1 -- 172.21.15.16:0/1825436794 >> 172.21.15.16:0/1825436794 conn(0x7f32180b4cd0 msgr2=0x7f32180acb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:24:10.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.901+0000 7f321d21b700 1 -- 172.21.15.16:0/1825436794 shutdown_connections 2023-10-16T12:24:10.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:10.901+0000 7f321d21b700 1 -- 172.21.15.16:0/1825436794 wait complete. 2023-10-16T12:24:10.915 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-16T12:24:11.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:11.040+0000 7fb37094e700 1 Processor -- start 2023-10-16T12:24:11.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:11.040+0000 7fb37094e700 1 -- start start 2023-10-16T12:24:11.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:11.041+0000 7fb37094e700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fb36c15b2c0 0x7fb36c161340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:11.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:11.041+0000 7fb37094e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb36c06a7e0 0x7fb36c06ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:11.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:11.041+0000 7fb37094e700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb36c15ecb0 con 0x7fb36c06e570 2023-10-16T12:24:11.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:11.041+0000 7fb37094e700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb36c06b190 con 0x7fb36c06a7e0 2023-10-16T12:24:11.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:11.041+0000 7fb37094e700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb36c161880 con 0x7fb36c15b2c0 2023-10-16T12:24:11.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:11.041+0000 7fb36ad9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb36c06a7e0 0x7fb36c06ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:11.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:11.041+0000 7fb36ad9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb36c06a7e0 0x7fb36c06ac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37956/0 (socket says 172.21.15.16:37956) 2023-10-16T12:24:11.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:11.041+0000 7fb369d9b700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb36c06e570 0x7fb36c15e5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:55460/0 (socket says 172.21.15.16:55460) 2023-10-16T12:24:11.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:11.041+0000 7fb36ad9d700 1 -- 172.21.15.16:0/3557701335 learned_addr learned my addr 172.21.15.16:0/3557701335 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:24:11.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:11.041+0000 7fb36a59c700 1 --2- 172.21.15.16:0/3557701335 >> v2:172.21.15.49:3300/0 conn(0x7fb36c15b2c0 0x7fb36c161340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:11.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:11.041+0000 7fb36ad9d700 1 -- 172.21.15.16:0/3557701335 >> v2:172.21.15.49:3300/0 conn(0x7fb36c15b2c0 msgr2=0x7fb36c161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:24:11.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:11.041+0000 7fb36ad9d700 1 --2- 172.21.15.16:0/3557701335 >> v2:172.21.15.49:3300/0 conn(0x7fb36c15b2c0 0x7fb36c161340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:11.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:11.041+0000 7fb36ad9d700 1 -- 172.21.15.16:0/3557701335 >> v1:172.21.15.16:6789/0 conn(0x7fb36c06e570 legacy=0x7fb36c15e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:11.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:11.041+0000 7fb36ad9d700 1 -- 172.21.15.16:0/3557701335 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb36c15b6a0 con 0x7fb36c06a7e0 2023-10-16T12:24:11.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:11.041+0000 7fb36959a700 1 -- 172.21.15.16:0/3557701335 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3353219883 0 0) 0x7fb36c15ecb0 con 0x7fb36c06e570 2023-10-16T12:24:11.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:11.041+0000 7fb36a59c700 1 --2- 172.21.15.16:0/3557701335 >> v2:172.21.15.49:3300/0 conn(0x7fb36c15b2c0 0x7fb36c161340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:24:11.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:11.042+0000 7fb36ad9d700 1 --2- 172.21.15.16:0/3557701335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb36c06a7e0 0x7fb36c06ac50 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fb36c06b4a0 tx=0x7fb36400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77f85987c39c3c9e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:24:11.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:11.042+0000 7fb36959a700 1 -- 172.21.15.16:0/3557701335 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb36400e070 con 0x7fb36c06a7e0 2023-10-16T12:24:11.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:11.042+0000 7fb36959a700 1 -- 172.21.15.16:0/3557701335 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb36400b9c0 con 0x7fb36c06a7e0 2023-10-16T12:24:11.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:11.042+0000 7fb36959a700 1 -- 172.21.15.16:0/3557701335 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb36400bd70 con 0x7fb36c06a7e0 2023-10-16T12:24:11.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:11.042+0000 7fb37094e700 1 -- 172.21.15.16:0/3557701335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb36c06a7e0 msgr2=0x7fb36c06ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:11.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:11.042+0000 7fb37094e700 1 --2- 172.21.15.16:0/3557701335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb36c06a7e0 0x7fb36c06ac50 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fb36c06b4a0 tx=0x7fb36400b330 comp rx=0 tx=0).stop 2023-10-16T12:24:11.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:11.042+0000 7fb37094e700 1 -- 172.21.15.16:0/3557701335 shutdown_connections 2023-10-16T12:24:11.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:11.042+0000 7fb37094e700 1 --2- 172.21.15.16:0/3557701335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb36c06a7e0 0x7fb36c06ac50 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:11.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:11.042+0000 7fb37094e700 1 --2- 172.21.15.16:0/3557701335 >> v2:172.21.15.49:3300/0 conn(0x7fb36c15b2c0 0x7fb36c161340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:11.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:11.042+0000 7fb37094e700 1 -- 172.21.15.16:0/3557701335 >> 172.21.15.16:0/3557701335 conn(0x7fb36c0b1f40 msgr2=0x7fb36c0b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:24:11.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:11.042+0000 7fb37094e700 1 -- 172.21.15.16:0/3557701335 shutdown_connections 2023-10-16T12:24:11.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:11.042+0000 7fb37094e700 1 -- 172.21.15.16:0/3557701335 wait complete. 2023-10-16T12:24:11.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:11.044+0000 7fb37094e700 1 Processor -- start 2023-10-16T12:24:11.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:11.044+0000 7fb37094e700 1 -- start start 2023-10-16T12:24:11.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:11.044+0000 7fb37094e700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb36c06a7e0 0x7fb36c0b3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:11.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:11.044+0000 7fb37094e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb36c06e570 0x7fb36c0c9d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:11.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:11.044+0000 7fb37094e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb36c15b2c0 0x7fb36c0c6050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:11.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:11.045+0000 7fb37094e700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb36c1f6920 con 0x7fb36c15b2c0 2023-10-16T12:24:11.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:11.045+0000 7fb37094e700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb36c1f6c00 con 0x7fb36c06a7e0 2023-10-16T12:24:11.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:11.045+0000 7fb37094e700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb36c1f6ee0 con 0x7fb36c06e570 2023-10-16T12:24:11.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:11.045+0000 7fb36ad9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb36c15b2c0 0x7fb36c0c6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:11.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:11.045+0000 7fb36ad9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb36c15b2c0 0x7fb36c0c6050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37962/0 (socket says 172.21.15.16:37962) 2023-10-16T12:24:11.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:11.045+0000 7fb36ad9d700 1 -- 172.21.15.16:0/2613783071 learned_addr learned my addr 172.21.15.16:0/2613783071 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:24:11.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:11.045+0000 7fb36ad9d700 1 -- 172.21.15.16:0/2613783071 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb36c06e570 msgr2=0x7fb36c0c9d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:11.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:11.045+0000 7fb36ad9d700 1 --2- 172.21.15.16:0/2613783071 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb36c06e570 0x7fb36c0c9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:11.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:11.045+0000 7fb36ad9d700 1 -- 172.21.15.16:0/2613783071 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb36c06a7e0 msgr2=0x7fb36c0b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:11.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:11.045+0000 7fb36ad9d700 1 --2- 172.21.15.16:0/2613783071 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb36c06a7e0 0x7fb36c0b3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:11.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:11.045+0000 7fb36ad9d700 1 -- 172.21.15.16:0/2613783071 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb364009b30 con 0x7fb36c15b2c0 2023-10-16T12:24:11.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:11.045+0000 7fb36ad9d700 1 --2- 172.21.15.16:0/2613783071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb36c15b2c0 0x7fb36c0c6050 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fb36c070620 tx=0x7fb364004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:24:11.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:11.045+0000 7fb35affd700 1 -- 172.21.15.16:0/2613783071 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3640099b0 con 0x7fb36c15b2c0 2023-10-16T12:24:11.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:11.045+0000 7fb35affd700 1 -- 172.21.15.16:0/2613783071 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3640092e0 con 0x7fb36c15b2c0 2023-10-16T12:24:11.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:11.045+0000 7fb37094e700 1 -- 172.21.15.16:0/2613783071 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb36c1f71c0 con 0x7fb36c15b2c0 2023-10-16T12:24:11.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:11.045+0000 7fb35affd700 1 -- 172.21.15.16:0/2613783071 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb36401b630 con 0x7fb36c15b2c0 2023-10-16T12:24:11.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:11.045+0000 7fb37094e700 1 -- 172.21.15.16:0/2613783071 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb36c0ca290 con 0x7fb36c15b2c0 2023-10-16T12:24:11.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:11.046+0000 7fb37094e700 1 -- 172.21.15.16:0/2613783071 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb36c04ebb0 con 0x7fb36c15b2c0 2023-10-16T12:24:11.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:11.047+0000 7fb35affd700 1 -- 172.21.15.16:0/2613783071 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb36401b790 con 0x7fb36c15b2c0 2023-10-16T12:24:11.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:11.047+0000 7fb35affd700 1 --2- 172.21.15.16:0/2613783071 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fb34c06b020 0x7fb34c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:11.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:11.047+0000 7fb35affd700 1 -- 172.21.15.16:0/2613783071 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fb36408aff0 con 0x7fb36c15b2c0 2023-10-16T12:24:11.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:11.047+0000 7fb36a59c700 1 --2- 172.21.15.16:0/2613783071 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fb34c06b020 0x7fb34c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:11.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:11.048+0000 7fb36a59c700 1 --2- 172.21.15.16:0/2613783071 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fb34c06b020 0x7fb34c06d4e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb3600098f0 tx=0x7fb360008040 comp rx=0 tx=0).ready entity=mgr.5874 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:24:11.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:11.052+0000 7fb35affd700 1 -- 172.21.15.16:0/2613783071 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb364055d50 con 0x7fb36c15b2c0 2023-10-16T12:24:11.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:11.318+0000 7fb37094e700 1 -- 172.21.15.16:0/2613783071 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7fb36c0cba60 con 0x7fb36c15b2c0 2023-10-16T12:24:12.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.314+0000 7fb35affd700 1 -- 172.21.15.16:0/2613783071 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) v1 ==== 92+0+0 (secure 0 0 0) 0x7fb364059370 con 0x7fb36c15b2c0 2023-10-16T12:24:12.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.317+0000 7fb37094e700 1 -- 172.21.15.16:0/2613783071 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fb34c06b020 msgr2=0x7fb34c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:12.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.317+0000 7fb37094e700 1 --2- 172.21.15.16:0/2613783071 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fb34c06b020 0x7fb34c06d4e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb3600098f0 tx=0x7fb360008040 comp rx=0 tx=0).stop 2023-10-16T12:24:12.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.317+0000 7fb37094e700 1 -- 172.21.15.16:0/2613783071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb36c15b2c0 msgr2=0x7fb36c0c6050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:12.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.317+0000 7fb37094e700 1 --2- 172.21.15.16:0/2613783071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb36c15b2c0 0x7fb36c0c6050 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fb36c070620 tx=0x7fb364004ad0 comp rx=0 tx=0).stop 2023-10-16T12:24:12.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.318+0000 7fb37094e700 1 -- 172.21.15.16:0/2613783071 shutdown_connections 2023-10-16T12:24:12.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.318+0000 7fb37094e700 1 --2- 172.21.15.16:0/2613783071 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fb34c06b020 0x7fb34c06d4e0 secure :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb3600098f0 tx=0x7fb360008040 comp rx=0 tx=0).stop 2023-10-16T12:24:12.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.318+0000 7fb37094e700 1 --2- 172.21.15.16:0/2613783071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb36c15b2c0 0x7fb36c0c6050 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:12.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.318+0000 7fb37094e700 1 --2- 172.21.15.16:0/2613783071 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb36c06a7e0 0x7fb36c0b3ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:12.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.318+0000 7fb37094e700 1 --2- 172.21.15.16:0/2613783071 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb36c06e570 0x7fb36c0c9d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:12.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.318+0000 7fb37094e700 1 -- 172.21.15.16:0/2613783071 >> 172.21.15.16:0/2613783071 conn(0x7fb36c0b1f40 msgr2=0x7fb36c0cbd40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:24:12.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.319+0000 7fb37094e700 1 -- 172.21.15.16:0/2613783071 shutdown_connections 2023-10-16T12:24:12.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.319+0000 7fb37094e700 1 -- 172.21.15.16:0/2613783071 wait complete. 2023-10-16T12:24:12.337 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-16T12:24:12.466 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:24:12.464+0000 7f5794f0d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T12:24:12.497 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:24:12.482+0000 7f1670f01200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T12:24:12.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.483+0000 7f2c59b33700 1 Processor -- start 2023-10-16T12:24:12.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.483+0000 7f2c59b33700 1 -- start start 2023-10-16T12:24:12.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.483+0000 7f2c59b33700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f2c5408a640 0x7f2c5416eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:12.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.483+0000 7f2c59b33700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2c5408aa20 0x7f2c540cbda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:12.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.483+0000 7f2c59b33700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2c540708b0 0x7f2c540cf1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:12.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.483+0000 7f2c59b33700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c540b2870 con 0x7f2c5408aa20 2023-10-16T12:24:12.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.483+0000 7f2c59b33700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c5416aa80 con 0x7f2c5408a640 2023-10-16T12:24:12.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.483+0000 7f2c59b33700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c5416f030 con 0x7f2c540708b0 2023-10-16T12:24:12.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.484+0000 7f2c52ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2c5408aa20 0x7f2c540cbda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:12.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.484+0000 7f2c52ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2c5408aa20 0x7f2c540cbda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37992/0 (socket says 172.21.15.16:37992) 2023-10-16T12:24:12.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.484+0000 7f2c52ffd700 1 -- 172.21.15.16:0/1009246363 learned_addr learned my addr 172.21.15.16:0/1009246363 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:24:12.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.484+0000 7f2c53fff700 1 --2- 172.21.15.16:0/1009246363 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2c540708b0 0x7f2c540cf1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:12.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.484+0000 7f2c537fe700 1 --2- 172.21.15.16:0/1009246363 >> v2:172.21.15.49:3300/0 conn(0x7f2c5408a640 0x7f2c5416eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:12.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.495+0000 7f2c52ffd700 1 -- 172.21.15.16:0/1009246363 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2c540708b0 msgr2=0x7f2c540cf1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:24:12.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.495+0000 7f2c52ffd700 1 --2- 172.21.15.16:0/1009246363 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2c540708b0 0x7f2c540cf1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:12.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.495+0000 7f2c52ffd700 1 -- 172.21.15.16:0/1009246363 >> v2:172.21.15.49:3300/0 conn(0x7f2c5408a640 msgr2=0x7f2c5416eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:24:12.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.495+0000 7f2c52ffd700 1 --2- 172.21.15.16:0/1009246363 >> v2:172.21.15.49:3300/0 conn(0x7f2c5408a640 0x7f2c5416eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:12.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.495+0000 7f2c52ffd700 1 -- 172.21.15.16:0/1009246363 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c54071570 con 0x7f2c5408aa20 2023-10-16T12:24:12.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.495+0000 7f2c53fff700 1 --2- 172.21.15.16:0/1009246363 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2c540708b0 0x7f2c540cf1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:24:12.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.496+0000 7f2c52ffd700 1 --2- 172.21.15.16:0/1009246363 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c5408aa20 0x7f2c540cbda0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f2c4800b9d0 tx=0x7f2c4800bce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f79fd4445df0e4e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:24:12.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.496+0000 7f2c527fc700 1 -- 172.21.15.16:0/1009246363 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c4800cbe0 con 0x7f2c5408aa20 2023-10-16T12:24:12.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.496+0000 7f2c527fc700 1 -- 172.21.15.16:0/1009246363 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c4800cd40 con 0x7f2c5408aa20 2023-10-16T12:24:12.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.496+0000 7f2c527fc700 1 -- 172.21.15.16:0/1009246363 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c4800f680 con 0x7f2c5408aa20 2023-10-16T12:24:12.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.496+0000 7f2c59b33700 1 -- 172.21.15.16:0/1009246363 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c5408aa20 msgr2=0x7f2c540cbda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:12.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.496+0000 7f2c59b33700 1 --2- 172.21.15.16:0/1009246363 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c5408aa20 0x7f2c540cbda0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f2c4800b9d0 tx=0x7f2c4800bce0 comp rx=0 tx=0).stop 2023-10-16T12:24:12.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.496+0000 7f2c59b33700 1 -- 172.21.15.16:0/1009246363 shutdown_connections 2023-10-16T12:24:12.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.496+0000 7f2c59b33700 1 --2- 172.21.15.16:0/1009246363 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2c540708b0 0x7f2c540cf1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:12.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.496+0000 7f2c59b33700 1 --2- 172.21.15.16:0/1009246363 >> v2:172.21.15.49:3300/0 conn(0x7f2c5408a640 0x7f2c5416eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:12.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.496+0000 7f2c59b33700 1 --2- 172.21.15.16:0/1009246363 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c5408aa20 0x7f2c540cbda0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:12.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.496+0000 7f2c59b33700 1 -- 172.21.15.16:0/1009246363 >> 172.21.15.16:0/1009246363 conn(0x7f2c5406c980 msgr2=0x7f2c5406cd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:24:12.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.497+0000 7f2c59b33700 1 -- 172.21.15.16:0/1009246363 shutdown_connections 2023-10-16T12:24:12.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.497+0000 7f2c59b33700 1 -- 172.21.15.16:0/1009246363 wait complete. 2023-10-16T12:24:12.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.498+0000 7f2c59b33700 1 Processor -- start 2023-10-16T12:24:12.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.498+0000 7f2c59b33700 1 -- start start 2023-10-16T12:24:12.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.499+0000 7f2c59b33700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c540708b0 0x7f2c540c4af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:12.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.499+0000 7f2c59b33700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2c5408a640 0x7f2c54089940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:12.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.499+0000 7f2c59b33700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2c5408aa20 0x7f2c540c73c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:12.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.499+0000 7f2c59b33700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c540c9490 con 0x7f2c540708b0 2023-10-16T12:24:12.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.499+0000 7f2c59b33700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c541681d0 con 0x7f2c5408aa20 2023-10-16T12:24:12.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.499+0000 7f2c59b33700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c54167ef0 con 0x7f2c5408a640 2023-10-16T12:24:12.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.499+0000 7f2c537fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c540708b0 0x7f2c540c4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:12.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.499+0000 7f2c537fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c540708b0 0x7f2c540c4af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37998/0 (socket says 172.21.15.16:37998) 2023-10-16T12:24:12.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.499+0000 7f2c52ffd700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2c5408a640 0x7f2c54089940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:12.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.499+0000 7f2c537fe700 1 -- 172.21.15.16:0/3292468554 learned_addr learned my addr 172.21.15.16:0/3292468554 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:24:12.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.499+0000 7f2c53fff700 1 --2- 172.21.15.16:0/3292468554 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2c5408aa20 0x7f2c540c73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:12.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.499+0000 7f2c52ffd700 1 -- 172.21.15.16:0/3292468554 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2c5408aa20 msgr2=0x7f2c540c73c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:12.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.499+0000 7f2c52ffd700 1 --2- 172.21.15.16:0/3292468554 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2c5408aa20 0x7f2c540c73c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:12.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.499+0000 7f2c52ffd700 1 -- 172.21.15.16:0/3292468554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c540708b0 msgr2=0x7f2c540c4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:12.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.499+0000 7f2c52ffd700 1 --2- 172.21.15.16:0/3292468554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c540708b0 0x7f2c540c4af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:12.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.499+0000 7f2c52ffd700 1 -- 172.21.15.16:0/3292468554 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c4c007840 con 0x7f2c5408a640 2023-10-16T12:24:12.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.499+0000 7f2c52ffd700 1 --2- 172.21.15.16:0/3292468554 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2c5408a640 0x7f2c54089940 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2c48016000 tx=0x7f2c48004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:24:12.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.500+0000 7f2c3bfff700 1 -- 172.21.15.16:0/3292468554 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c4801c070 con 0x7f2c5408a640 2023-10-16T12:24:12.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.500+0000 7f2c3bfff700 1 -- 172.21.15.16:0/3292468554 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c4800be70 con 0x7f2c5408a640 2023-10-16T12:24:12.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.500+0000 7f2c3bfff700 1 -- 172.21.15.16:0/3292468554 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c48004270 con 0x7f2c5408a640 2023-10-16T12:24:12.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.500+0000 7f2c59b33700 1 -- 172.21.15.16:0/3292468554 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c4800b600 con 0x7f2c5408a640 2023-10-16T12:24:12.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.500+0000 7f2c59b33700 1 -- 172.21.15.16:0/3292468554 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c540c9980 con 0x7f2c5408a640 2023-10-16T12:24:12.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.501+0000 7f2c59b33700 1 -- 172.21.15.16:0/3292468554 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c5404ebb0 con 0x7f2c5408a640 2023-10-16T12:24:12.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.501+0000 7f2c3bfff700 1 -- 172.21.15.16:0/3292468554 <== mon.2 v2:172.21.15.49:3301/0 4 ==== mgrmap(e 19) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f2c480043d0 con 0x7f2c5408a640 2023-10-16T12:24:12.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.501+0000 7f2c3bfff700 1 --2- 172.21.15.16:0/3292468554 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f2c4006b070 0x7f2c4006d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:12.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.502+0000 7f2c537fe700 1 -- 172.21.15.16:0/3292468554 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f2c4006b070 msgr2=0x7f2c4006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3870937344 2023-10-16T12:24:12.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.502+0000 7f2c3bfff700 1 -- 172.21.15.16:0/3292468554 <== mon.2 v2:172.21.15.49:3301/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f2c4808ac40 con 0x7f2c5408a640 2023-10-16T12:24:12.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.502+0000 7f2c537fe700 1 --2- 172.21.15.16:0/3292468554 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f2c4006b070 0x7f2c4006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:24:12.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.507+0000 7f2c3bfff700 1 -- 172.21.15.16:0/3292468554 <== mon.2 v2:172.21.15.49:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2c48055a20 con 0x7f2c5408a640 2023-10-16T12:24:12.536 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:24:12.534+0000 7f5794f0d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T12:24:12.555 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:24:12.553+0000 7f1670f01200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T12:24:12.611 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:24:12.609+0000 7f5794f0d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T12:24:12.630 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:24:12.628+0000 7f1670f01200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T12:24:12.691 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:24:12.689+0000 7f5794f0d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T12:24:12.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.703+0000 7f2c537fe700 1 -- 172.21.15.16:0/3292468554 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f2c4006b070 msgr2=0x7f2c4006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3870937344 2023-10-16T12:24:12.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.703+0000 7f2c537fe700 1 --2- 172.21.15.16:0/3292468554 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f2c4006b070 0x7f2c4006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:24:12.712 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:24:12.710+0000 7f1670f01200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T12:24:12.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.712+0000 7f2c59b33700 1 -- 172.21.15.16:0/3292468554 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f2c540b4280 con 0x7f2c5408a640 2023-10-16T12:24:12.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.717+0000 7f2c3bfff700 1 -- 172.21.15.16:0/3292468554 <== mon.2 v2:172.21.15.49:3301/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v17)=0 v17) v1 ==== 138+0+0 (secure 0 0 0) 0x7f2c48059040 con 0x7f2c5408a640 2023-10-16T12:24:12.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.719+0000 7f2c59b33700 1 -- 172.21.15.16:0/3292468554 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f2c4006b070 msgr2=0x7f2c4006d530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:24:12.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.719+0000 7f2c59b33700 1 --2- 172.21.15.16:0/3292468554 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f2c4006b070 0x7f2c4006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:12.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.719+0000 7f2c59b33700 1 -- 172.21.15.16:0/3292468554 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2c5408a640 msgr2=0x7f2c54089940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:12.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.719+0000 7f2c59b33700 1 --2- 172.21.15.16:0/3292468554 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2c5408a640 0x7f2c54089940 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2c48016000 tx=0x7f2c48004750 comp rx=0 tx=0).stop 2023-10-16T12:24:12.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.719+0000 7f2c59b33700 1 -- 172.21.15.16:0/3292468554 shutdown_connections 2023-10-16T12:24:12.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.719+0000 7f2c59b33700 1 --2- 172.21.15.16:0/3292468554 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7f2c4006b070 0x7f2c4006d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:12.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.719+0000 7f2c59b33700 1 --2- 172.21.15.16:0/3292468554 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2c5408aa20 0x7f2c540c73c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:12.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.719+0000 7f2c59b33700 1 --2- 172.21.15.16:0/3292468554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c540708b0 0x7f2c540c4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:12.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.719+0000 7f2c59b33700 1 --2- 172.21.15.16:0/3292468554 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2c5408a640 0x7f2c54089940 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:12.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.719+0000 7f2c59b33700 1 -- 172.21.15.16:0/3292468554 >> 172.21.15.16:0/3292468554 conn(0x7f2c5406c980 msgr2=0x7f2c540cf1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:24:12.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.719+0000 7f2c59b33700 1 -- 172.21.15.16:0/3292468554 shutdown_connections 2023-10-16T12:24:12.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.719+0000 7f2c59b33700 1 -- 172.21.15.16:0/3292468554 wait complete. 2023-10-16T12:24:12.733 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-16T12:24:12.839 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:24:12.837+0000 7f5794f0d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T12:24:12.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.861+0000 7fc53b5ae700 1 Processor -- start 2023-10-16T12:24:12.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.862+0000 7fc53b5ae700 1 -- start start 2023-10-16T12:24:12.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.862+0000 7fc53b5ae700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc5340cb880 0x7fc5340cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:12.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.862+0000 7fc53b5ae700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fc53406da60 0x7fc53406def0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:12.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.862+0000 7fc53b5ae700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5340cef70 con 0x7fc53415b070 2023-10-16T12:24:12.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.862+0000 7fc53b5ae700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc53406e430 con 0x7fc5340cb880 2023-10-16T12:24:12.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.862+0000 7fc53b5ae700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5340ce360 con 0x7fc53406da60 2023-10-16T12:24:12.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.863+0000 7fc538b49700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc5340cb880 0x7fc5340cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:12.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.863+0000 7fc538b49700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc5340cb880 0x7fc5340cbd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38054/0 (socket says 172.21.15.16:38054) 2023-10-16T12:24:12.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.863+0000 7fc538b49700 1 -- 172.21.15.16:0/915178248 learned_addr learned my addr 172.21.15.16:0/915178248 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:24:12.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.863+0000 7fc539b4b700 1 --2- 172.21.15.16:0/915178248 >> v2:172.21.15.49:3300/0 conn(0x7fc53406da60 0x7fc53406def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:12.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.863+0000 7fc538b49700 1 -- 172.21.15.16:0/915178248 >> v2:172.21.15.49:3300/0 conn(0x7fc53406da60 msgr2=0x7fc53406def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:24:12.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.863+0000 7fc538b49700 1 --2- 172.21.15.16:0/915178248 >> v2:172.21.15.49:3300/0 conn(0x7fc53406da60 0x7fc53406def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:12.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.863+0000 7fc538b49700 1 -- 172.21.15.16:0/915178248 >> v1:172.21.15.49:6789/0 conn(0x7fc53415b070 legacy=0x7fc53415b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:12.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.863+0000 7fc538b49700 1 -- 172.21.15.16:0/915178248 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5340ce640 con 0x7fc5340cb880 2023-10-16T12:24:12.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.863+0000 7fc538b49700 1 --2- 172.21.15.16:0/915178248 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5340cb880 0x7fc5340cbd00 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fc53406edc0 tx=0x7fc530009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8abc5fdf7f856ef5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:24:12.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.863+0000 7fc52bfff700 1 -- 172.21.15.16:0/915178248 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc530011830 con 0x7fc5340cb880 2023-10-16T12:24:12.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.863+0000 7fc52bfff700 1 -- 172.21.15.16:0/915178248 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc530011990 con 0x7fc5340cb880 2023-10-16T12:24:12.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.863+0000 7fc52bfff700 1 -- 172.21.15.16:0/915178248 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc530011d20 con 0x7fc5340cb880 2023-10-16T12:24:12.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.864+0000 7fc53b5ae700 1 -- 172.21.15.16:0/915178248 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5340cb880 msgr2=0x7fc5340cbd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:12.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.864+0000 7fc53b5ae700 1 --2- 172.21.15.16:0/915178248 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5340cb880 0x7fc5340cbd00 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fc53406edc0 tx=0x7fc530009bd0 comp rx=0 tx=0).stop 2023-10-16T12:24:12.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.864+0000 7fc53b5ae700 1 -- 172.21.15.16:0/915178248 shutdown_connections 2023-10-16T12:24:12.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.864+0000 7fc53b5ae700 1 --2- 172.21.15.16:0/915178248 >> v2:172.21.15.49:3300/0 conn(0x7fc53406da60 0x7fc53406def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:12.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.864+0000 7fc53b5ae700 1 --2- 172.21.15.16:0/915178248 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5340cb880 0x7fc5340cbd00 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:12.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.864+0000 7fc53b5ae700 1 -- 172.21.15.16:0/915178248 >> 172.21.15.16:0/915178248 conn(0x7fc5340b1f40 msgr2=0x7fc5340b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:24:12.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.864+0000 7fc53b5ae700 1 -- 172.21.15.16:0/915178248 shutdown_connections 2023-10-16T12:24:12.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.864+0000 7fc53b5ae700 1 -- 172.21.15.16:0/915178248 wait complete. 2023-10-16T12:24:12.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.865+0000 7fc53b5ae700 1 Processor -- start 2023-10-16T12:24:12.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.865+0000 7fc53b5ae700 1 -- start start 2023-10-16T12:24:12.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.866+0000 7fc53b5ae700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc53406da60 0x7fc534159b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:12.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.866+0000 7fc53b5ae700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc5340cb880 0x7fc534154530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:12.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.866+0000 7fc53b5ae700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc53415b070 0x7fc534154a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:12.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.866+0000 7fc53b5ae700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5340b5a70 con 0x7fc53415b070 2023-10-16T12:24:12.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.866+0000 7fc53b5ae700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5340b4e60 con 0x7fc5340cb880 2023-10-16T12:24:12.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.866+0000 7fc53b5ae700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5340b5140 con 0x7fc53406da60 2023-10-16T12:24:12.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.866+0000 7fc539b4b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc53415b070 0x7fc534154a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:12.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.866+0000 7fc539b4b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc53415b070 0x7fc534154a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38062/0 (socket says 172.21.15.16:38062) 2023-10-16T12:24:12.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.866+0000 7fc539b4b700 1 -- 172.21.15.16:0/3605861391 learned_addr learned my addr 172.21.15.16:0/3605861391 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:24:12.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.866+0000 7fc53934a700 1 --2- 172.21.15.16:0/3605861391 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc53406da60 0x7fc534159b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:12.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.866+0000 7fc538b49700 1 --2- 172.21.15.16:0/3605861391 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc5340cb880 0x7fc534154530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:12.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.866+0000 7fc539b4b700 1 -- 172.21.15.16:0/3605861391 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc53406da60 msgr2=0x7fc534159b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:12.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.866+0000 7fc539b4b700 1 --2- 172.21.15.16:0/3605861391 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc53406da60 0x7fc534159b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:12.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.866+0000 7fc539b4b700 1 -- 172.21.15.16:0/3605861391 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc5340cb880 msgr2=0x7fc534154530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:12.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.866+0000 7fc539b4b700 1 --2- 172.21.15.16:0/3605861391 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc5340cb880 0x7fc534154530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:12.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.866+0000 7fc539b4b700 1 -- 172.21.15.16:0/3605861391 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc530009550 con 0x7fc53415b070 2023-10-16T12:24:12.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.867+0000 7fc539b4b700 1 --2- 172.21.15.16:0/3605861391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc53415b070 0x7fc534154a70 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fc52c00efc0 tx=0x7fc52c00c0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:24:12.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.867+0000 7fc529ffb700 1 -- 172.21.15.16:0/3605861391 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc52c00cad0 con 0x7fc53415b070 2023-10-16T12:24:12.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.867+0000 7fc529ffb700 1 -- 172.21.15.16:0/3605861391 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc52c00f040 con 0x7fc53415b070 2023-10-16T12:24:12.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.867+0000 7fc53b5ae700 1 -- 172.21.15.16:0/3605861391 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5340b5420 con 0x7fc53415b070 2023-10-16T12:24:12.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.867+0000 7fc529ffb700 1 -- 172.21.15.16:0/3605861391 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc52c00ce70 con 0x7fc53415b070 2023-10-16T12:24:12.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.867+0000 7fc53b5ae700 1 -- 172.21.15.16:0/3605861391 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc53415a060 con 0x7fc53415b070 2023-10-16T12:24:12.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.868+0000 7fc53b5ae700 1 -- 172.21.15.16:0/3605861391 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5340b5be0 con 0x7fc53415b070 2023-10-16T12:24:12.877 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:24:12.868+0000 7f1670f01200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T12:24:12.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.868+0000 7fc529ffb700 1 -- 172.21.15.16:0/3605861391 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 89348+0+0 (secure 0 0 0) 0x7fc52c014480 con 0x7fc53415b070 2023-10-16T12:24:12.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.869+0000 7fc529ffb700 1 --2- 172.21.15.16:0/3605861391 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fc51806b070 0x7fc51806d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:12.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.869+0000 7fc53934a700 1 -- 172.21.15.16:0/3605861391 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fc51806b070 msgr2=0x7fc51806d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3870937344 2023-10-16T12:24:12.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.869+0000 7fc529ffb700 1 -- 172.21.15.16:0/3605861391 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fc52c08a150 con 0x7fc53415b070 2023-10-16T12:24:12.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.869+0000 7fc53934a700 1 --2- 172.21.15.16:0/3605861391 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fc51806b070 0x7fc51806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:24:12.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:12.873+0000 7fc529ffb700 1 -- 172.21.15.16:0/3605861391 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc52c054ea0 con 0x7fc53415b070 2023-10-16T12:24:13.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.070+0000 7fc53934a700 1 -- 172.21.15.16:0/3605861391 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fc51806b070 msgr2=0x7fc51806d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3870937344 2023-10-16T12:24:13.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.070+0000 7fc53934a700 1 --2- 172.21.15.16:0/3605861391 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fc51806b070 0x7fc51806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:24:13.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.079+0000 7fc53b5ae700 1 -- 172.21.15.16:0/3605861391 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fc53404ebb0 con 0x7fc53415b070 2023-10-16T12:24:13.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.082+0000 7fc529ffb700 1 -- 172.21.15.16:0/3605861391 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v18)=0 v18) v1 ==== 133+0+0 (secure 0 0 0) 0x7fc52c0584c0 con 0x7fc53415b070 2023-10-16T12:24:13.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.084+0000 7fc53b5ae700 1 -- 172.21.15.16:0/3605861391 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fc51806b070 msgr2=0x7fc51806d530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:24:13.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.084+0000 7fc53b5ae700 1 --2- 172.21.15.16:0/3605861391 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fc51806b070 0x7fc51806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:13.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.084+0000 7fc53b5ae700 1 -- 172.21.15.16:0/3605861391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc53415b070 msgr2=0x7fc534154a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:13.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.084+0000 7fc53b5ae700 1 --2- 172.21.15.16:0/3605861391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc53415b070 0x7fc534154a70 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fc52c00efc0 tx=0x7fc52c00c0c0 comp rx=0 tx=0).stop 2023-10-16T12:24:13.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.084+0000 7fc53b5ae700 1 -- 172.21.15.16:0/3605861391 shutdown_connections 2023-10-16T12:24:13.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.084+0000 7fc53b5ae700 1 --2- 172.21.15.16:0/3605861391 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fc51806b070 0x7fc51806d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:13.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.084+0000 7fc53b5ae700 1 --2- 172.21.15.16:0/3605861391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc53415b070 0x7fc534154a70 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:13.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.084+0000 7fc53b5ae700 1 --2- 172.21.15.16:0/3605861391 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc53406da60 0x7fc534159b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:13.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.084+0000 7fc53b5ae700 1 --2- 172.21.15.16:0/3605861391 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc5340cb880 0x7fc534154530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:13.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.084+0000 7fc53b5ae700 1 -- 172.21.15.16:0/3605861391 >> 172.21.15.16:0/3605861391 conn(0x7fc5340b1f40 msgr2=0x7fc5340b17a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:24:13.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.084+0000 7fc53b5ae700 1 -- 172.21.15.16:0/3605861391 shutdown_connections 2023-10-16T12:24:13.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.084+0000 7fc53b5ae700 1 -- 172.21.15.16:0/3605861391 wait complete. 2023-10-16T12:24:13.099 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:24:13.099 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-10-16T12:24:13.121 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1m 2023-10-16T12:24:13.179 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:24:13.177+0000 7f5794f0d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T12:24:13.221 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:24:13.219+0000 7f1670f01200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T12:24:13.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.290+0000 7fc9336f1700 1 Processor -- start 2023-10-16T12:24:13.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.290+0000 7fc9336f1700 1 -- start start 2023-10-16T12:24:13.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.291+0000 7fc9336f1700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc92c072b30 0x7fc92c071a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:13.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.291+0000 7fc9336f1700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fc92c15bb40 0x7fc92c06e310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:13.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.291+0000 7fc9336f1700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc92c15c040 con 0x7fc92c0caa40 2023-10-16T12:24:13.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.291+0000 7fc9336f1700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc92c15c320 con 0x7fc92c072b30 2023-10-16T12:24:13.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.291+0000 7fc9336f1700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc92c15c600 con 0x7fc92c15bb40 2023-10-16T12:24:13.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.291+0000 7fc93148d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc92c072b30 0x7fc92c071a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:13.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.291+0000 7fc931c8e700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc92c0caa40 0x7fc92c0cae50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:55622/0 (socket says 172.21.15.16:55622) 2023-10-16T12:24:13.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.291+0000 7fc93148d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc92c072b30 0x7fc92c071a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38112/0 (socket says 172.21.15.16:38112) 2023-10-16T12:24:13.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.291+0000 7fc931c8e700 1 -- 172.21.15.16:0/3503606103 learned_addr learned my addr 172.21.15.16:0/3503606103 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:24:13.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.291+0000 7fc930c8c700 1 --2- 172.21.15.16:0/3503606103 >> v2:172.21.15.49:3300/0 conn(0x7fc92c15bb40 0x7fc92c06e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:13.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.291+0000 7fc93148d700 1 -- 172.21.15.16:0/3503606103 >> v2:172.21.15.49:3300/0 conn(0x7fc92c15bb40 msgr2=0x7fc92c06e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:24:13.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.291+0000 7fc93148d700 1 --2- 172.21.15.16:0/3503606103 >> v2:172.21.15.49:3300/0 conn(0x7fc92c15bb40 0x7fc92c06e310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:13.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.291+0000 7fc93148d700 1 -- 172.21.15.16:0/3503606103 >> v1:172.21.15.16:6789/0 conn(0x7fc92c0caa40 legacy=0x7fc92c0cae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:13.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.291+0000 7fc93148d700 1 -- 172.21.15.16:0/3503606103 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc92c15c8e0 con 0x7fc92c072b30 2023-10-16T12:24:13.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.292+0000 7fc93148d700 1 --2- 172.21.15.16:0/3503606103 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc92c072b30 0x7fc92c071a70 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fc92c06b880 tx=0x7fc928009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6278172fea95af7a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:24:13.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.292+0000 7fc923fff700 1 -- 172.21.15.16:0/3503606103 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc928011830 con 0x7fc92c072b30 2023-10-16T12:24:13.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.292+0000 7fc923fff700 1 -- 172.21.15.16:0/3503606103 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc928011990 con 0x7fc92c072b30 2023-10-16T12:24:13.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.292+0000 7fc923fff700 1 -- 172.21.15.16:0/3503606103 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc928011d20 con 0x7fc92c072b30 2023-10-16T12:24:13.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.292+0000 7fc9336f1700 1 -- 172.21.15.16:0/3503606103 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc92c072b30 msgr2=0x7fc92c071a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:13.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.292+0000 7fc9336f1700 1 --2- 172.21.15.16:0/3503606103 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc92c072b30 0x7fc92c071a70 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fc92c06b880 tx=0x7fc928009b20 comp rx=0 tx=0).stop 2023-10-16T12:24:13.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.292+0000 7fc9336f1700 1 -- 172.21.15.16:0/3503606103 shutdown_connections 2023-10-16T12:24:13.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.292+0000 7fc9336f1700 1 --2- 172.21.15.16:0/3503606103 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc92c072b30 0x7fc92c071a70 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:13.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.292+0000 7fc9336f1700 1 --2- 172.21.15.16:0/3503606103 >> v2:172.21.15.49:3300/0 conn(0x7fc92c15bb40 0x7fc92c06e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:13.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.292+0000 7fc9336f1700 1 -- 172.21.15.16:0/3503606103 >> 172.21.15.16:0/3503606103 conn(0x7fc92c0b1f40 msgr2=0x7fc92c0b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:24:13.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.293+0000 7fc9336f1700 1 -- 172.21.15.16:0/3503606103 shutdown_connections 2023-10-16T12:24:13.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.293+0000 7fc9336f1700 1 -- 172.21.15.16:0/3503606103 wait complete. 2023-10-16T12:24:13.301 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:24:13.297+0000 7f5794f0d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T12:24:13.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.294+0000 7fc9336f1700 1 Processor -- start 2023-10-16T12:24:13.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.294+0000 7fc9336f1700 1 -- start start 2023-10-16T12:24:13.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.295+0000 7fc9336f1700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc92c072b30 0x7fc92c1474f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:13.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.295+0000 7fc9336f1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc92c0caa40 0x7fc92c1e5d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:13.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.295+0000 7fc9336f1700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc92c15bb40 0x7fc92c0c4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:13.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.295+0000 7fc9336f1700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc92c0c96a0 con 0x7fc92c0caa40 2023-10-16T12:24:13.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.295+0000 7fc9336f1700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc92c1e6630 con 0x7fc92c072b30 2023-10-16T12:24:13.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.295+0000 7fc9336f1700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc92c1e6910 con 0x7fc92c15bb40 2023-10-16T12:24:13.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.295+0000 7fc930c8c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc92c0caa40 0x7fc92c1e5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:13.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.295+0000 7fc930c8c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc92c0caa40 0x7fc92c1e5d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38120/0 (socket says 172.21.15.16:38120) 2023-10-16T12:24:13.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.295+0000 7fc930c8c700 1 -- 172.21.15.16:0/2078162749 learned_addr learned my addr 172.21.15.16:0/2078162749 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:24:13.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.295+0000 7fc93148d700 1 --2- 172.21.15.16:0/2078162749 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc92c072b30 0x7fc92c1474f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:13.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.295+0000 7fc931c8e700 1 --2- 172.21.15.16:0/2078162749 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc92c15bb40 0x7fc92c0c4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:13.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.295+0000 7fc930c8c700 1 -- 172.21.15.16:0/2078162749 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc92c15bb40 msgr2=0x7fc92c0c4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:13.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.295+0000 7fc930c8c700 1 --2- 172.21.15.16:0/2078162749 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc92c15bb40 0x7fc92c0c4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:13.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.295+0000 7fc930c8c700 1 -- 172.21.15.16:0/2078162749 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc92c072b30 msgr2=0x7fc92c1474f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:13.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.295+0000 7fc930c8c700 1 --2- 172.21.15.16:0/2078162749 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc92c072b30 0x7fc92c1474f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:13.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.295+0000 7fc930c8c700 1 -- 172.21.15.16:0/2078162749 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9280094a0 con 0x7fc92c0caa40 2023-10-16T12:24:13.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.295+0000 7fc930c8c700 1 --2- 172.21.15.16:0/2078162749 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc92c0caa40 0x7fc92c1e5d70 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fc91c00e840 tx=0x7fc91c00eb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:24:13.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.295+0000 7fc921ffb700 1 -- 172.21.15.16:0/2078162749 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc91c009590 con 0x7fc92c0caa40 2023-10-16T12:24:13.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.295+0000 7fc921ffb700 1 -- 172.21.15.16:0/2078162749 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc91c013680 con 0x7fc92c0caa40 2023-10-16T12:24:13.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.296+0000 7fc9336f1700 1 -- 172.21.15.16:0/2078162749 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc92c1e6bf0 con 0x7fc92c0caa40 2023-10-16T12:24:13.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.296+0000 7fc921ffb700 1 -- 172.21.15.16:0/2078162749 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc91c013a30 con 0x7fc92c0caa40 2023-10-16T12:24:13.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.296+0000 7fc9336f1700 1 -- 172.21.15.16:0/2078162749 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc92c1e62b0 con 0x7fc92c0caa40 2023-10-16T12:24:13.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.297+0000 7fc9336f1700 1 -- 172.21.15.16:0/2078162749 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc92c0c9910 con 0x7fc92c0caa40 2023-10-16T12:24:13.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.297+0000 7fc921ffb700 1 -- 172.21.15.16:0/2078162749 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 89348+0+0 (secure 0 0 0) 0x7fc91c013b90 con 0x7fc92c0caa40 2023-10-16T12:24:13.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.298+0000 7fc921ffb700 1 --2- 172.21.15.16:0/2078162749 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fc91006b070 0x7fc91006d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:13.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.298+0000 7fc93148d700 1 -- 172.21.15.16:0/2078162749 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fc91006b070 msgr2=0x7fc91006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3870937344 2023-10-16T12:24:13.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.298+0000 7fc921ffb700 1 -- 172.21.15.16:0/2078162749 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fc91c0896a0 con 0x7fc92c0caa40 2023-10-16T12:24:13.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.298+0000 7fc93148d700 1 --2- 172.21.15.16:0/2078162749 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fc91006b070 0x7fc91006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:24:13.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.302+0000 7fc921ffb700 1 -- 172.21.15.16:0/2078162749 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc91c054570 con 0x7fc92c0caa40 2023-10-16T12:24:13.345 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:24:13.343+0000 7f1670f01200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T12:24:13.375 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:24:13.373+0000 7f5794f0d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T12:24:13.423 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:24:13.421+0000 7f1670f01200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T12:24:13.459 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:24:13.457+0000 7f5794f0d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T12:24:13.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.499+0000 7fc93148d700 1 -- 172.21.15.16:0/2078162749 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fc91006b070 msgr2=0x7fc91006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3870937344 2023-10-16T12:24:13.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.499+0000 7fc93148d700 1 --2- 172.21.15.16:0/2078162749 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fc91006b070 0x7fc91006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:24:13.507 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:24:13.506+0000 7f1670f01200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T12:24:13.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.537+0000 7fc9336f1700 1 -- 172.21.15.16:0/2078162749 --> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fc92c0c9430 con 0x7fc91006b070 2023-10-16T12:24:13.843 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:24:13.841+0000 7f5794f0d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T12:24:13.894 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:24:13.892+0000 7f1670f01200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T12:24:13.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.900+0000 7fc93148d700 1 -- 172.21.15.16:0/2078162749 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fc91006b070 msgr2=0x7fc91006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3870937344 2023-10-16T12:24:13.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:13.900+0000 7fc93148d700 1 --2- 172.21.15.16:0/2078162749 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fc91006b070 0x7fc91006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-16T12:24:14.447 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:24:14.445+0000 7f5794f0d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T12:24:14.500 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:24:14.498+0000 7f1670f01200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T12:24:14.519 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:24:14.517+0000 7f5794f0d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T12:24:14.573 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:24:14.571+0000 7f1670f01200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T12:24:14.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:14.701+0000 7fc93148d700 1 -- 172.21.15.16:0/2078162749 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fc91006b070 msgr2=0x7fc91006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3870937344 2023-10-16T12:24:14.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:14.701+0000 7fc93148d700 1 --2- 172.21.15.16:0/2078162749 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fc91006b070 0x7fc91006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-16T12:24:15.181 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:24:15.179+0000 7f5794f0d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:24:15.237 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:24:15.235+0000 7f1670f01200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:24:15.261 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:24:15.259+0000 7f5794f0d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T12:24:15.319 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:24:15.317+0000 7f1670f01200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T12:24:15.412 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:24:15.409+0000 7f5794f0d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:24:15.471 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:24:15.469+0000 7f1670f01200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:24:15.646 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:24:15.644+0000 7f5794f0d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T12:24:15.708 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:24:15.706+0000 7f1670f01200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T12:24:15.721 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:24:15.720+0000 7f5794f0d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T12:24:15.785 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:24:15.784+0000 7f1670f01200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T12:24:15.996 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:24:15.994+0000 7f5794f0d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T12:24:16.060 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:24:16.058+0000 7f1670f01200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T12:24:16.224 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:24:16.222+0000 7f5794f0d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T12:24:16.292 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:24:16.290+0000 7f1670f01200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T12:24:16.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:16.302+0000 7fc93148d700 1 -- 172.21.15.16:0/2078162749 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fc91006b070 msgr2=0x7fc91006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3870937344 2023-10-16T12:24:16.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:16.302+0000 7fc93148d700 1 --2- 172.21.15.16:0/2078162749 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fc91006b070 0x7fc91006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-10-16T12:24:16.378 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:24:16.376+0000 7f5794f0d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T12:24:16.448 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:24:16.446+0000 7f1670f01200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T12:24:16.495 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:24:16.493+0000 7f5794f0d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T12:24:16.566 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:24:16.564+0000 7f1670f01200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T12:24:16.654 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:24:16.652+0000 7f5794f0d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T12:24:16.728 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:24:16.726+0000 7f1670f01200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T12:24:16.936 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:24:16.934+0000 7f5794f0d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T12:24:16.999 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:24:16.997+0000 7f1670f01200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T12:24:17.100 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:24:17.098+0000 7f5794f0d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T12:24:17.167 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:24:17.165+0000 7f1670f01200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T12:24:17.647 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:24:17.645+0000 7f5794f0d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T12:24:17.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:17.714+0000 7fc921ffb700 1 -- 172.21.15.16:0/2078162749 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 20) v1 ==== 89348+0+0 (secure 0 0 0) 0x7fc91c054940 con 0x7fc92c0caa40 2023-10-16T12:24:17.724 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:24:17.721+0000 7f1670f01200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T12:24:17.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:17.741+0000 7fc921ffb700 1 -- 172.21.15.16:0/2078162749 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mgrmap(e 21) v1 ==== 88957+0+0 (secure 0 0 0) 0x7fc91c050c00 con 0x7fc92c0caa40 2023-10-16T12:24:17.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:17.741+0000 7fc921ffb700 1 -- 172.21.15.16:0/2078162749 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fc91006b070 msgr2=0x7fc91006d530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:24:17.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:17.741+0000 7fc921ffb700 1 --2- 172.21.15.16:0/2078162749 >> [v2:172.21.15.16:6832/3870937344,v1:172.21.15.16:6833/3870937344] conn(0x7fc91006b070 0x7fc91006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:17.889 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:24:17.886+0000 7f15e3edc700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:24:17.890 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:24:17.886+0000 7f15e3edc700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:24:17.890 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:24:17.886+0000 7f15e3edc700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:24:17.890 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:24:17.886+0000 7f15e3edc700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:24:17.890 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:24:17.886+0000 7f15e3edc700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:24:17.891 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:24:17.886+0000 7f15e6ee2700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:24:17.891 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:24:17.886+0000 7f15e6ee2700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:24:17.891 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:24:17.886+0000 7f15e6ee2700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:24:17.891 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:24:17.886+0000 7f15e6ee2700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:24:17.891 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:24:17.886+0000 7f15e6ee2700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:24:18.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:18.745+0000 7fc921ffb700 1 -- 172.21.15.16:0/2078162749 <== mon.0 v2:172.21.15.16:3300/0 9 ==== mgrmap(e 22) v1 ==== 89084+0+0 (secure 0 0 0) 0x7fc91c055210 con 0x7fc92c0caa40 2023-10-16T12:24:18.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:18.745+0000 7fc921ffb700 1 --2- 172.21.15.16:0/2078162749 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7fc910027990 0x7fc910027d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:18.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:18.745+0000 7fc921ffb700 1 -- 172.21.15.16:0/2078162749 --> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fc92c0c9430 con 0x7fc910027990 2023-10-16T12:24:18.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:18.747+0000 7fc93148d700 1 --2- 172.21.15.16:0/2078162749 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7fc910027990 0x7fc910027d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:18.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:18.750+0000 7fc93148d700 1 --2- 172.21.15.16:0/2078162749 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7fc910027990 0x7fc910027d70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fc9280090b0 tx=0x7fc928008030 comp rx=0 tx=0).ready entity=mgr.6291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:24:18.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:18.808+0000 7fc921ffb700 1 -- 172.21.15.16:0/2078162749 <== mgr.6291 v2:172.21.15.16:6832/4075855741 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fc92c0c9430 con 0x7fc910027990 2023-10-16T12:24:18.811 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule set for path snap_test_dir1 2023-10-16T12:24:18.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:18.811+0000 7fc9336f1700 1 -- 172.21.15.16:0/2078162749 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7fc910027990 msgr2=0x7fc910027d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:18.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:18.811+0000 7fc9336f1700 1 --2- 172.21.15.16:0/2078162749 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7fc910027990 0x7fc910027d70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fc9280090b0 tx=0x7fc928008030 comp rx=0 tx=0).stop 2023-10-16T12:24:18.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:18.811+0000 7fc9336f1700 1 -- 172.21.15.16:0/2078162749 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc92c0caa40 msgr2=0x7fc92c1e5d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:18.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:18.811+0000 7fc9336f1700 1 --2- 172.21.15.16:0/2078162749 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc92c0caa40 0x7fc92c1e5d70 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fc91c00e840 tx=0x7fc91c00eb50 comp rx=0 tx=0).stop 2023-10-16T12:24:18.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:18.811+0000 7fc931c8e700 1 -- 172.21.15.16:0/2078162749 reap_dead start 2023-10-16T12:24:18.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:18.811+0000 7fc9336f1700 1 -- 172.21.15.16:0/2078162749 shutdown_connections 2023-10-16T12:24:18.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:18.811+0000 7fc9336f1700 1 -- 172.21.15.16:0/2078162749 >> 172.21.15.16:0/2078162749 conn(0x7fc92c0b1f40 msgr2=0x7fc92c0686c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:24:18.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:18.811+0000 7fc9336f1700 1 -- 172.21.15.16:0/2078162749 shutdown_connections 2023-10-16T12:24:18.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:18.811+0000 7fc9336f1700 1 -- 172.21.15.16:0/2078162749 wait complete. 2023-10-16T12:24:18.827 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2023-10-16T12:24:18.827 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 2m 2023-10-16T12:24:18.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:18.954+0000 7f6426d1d700 1 Processor -- start 2023-10-16T12:24:18.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:18.954+0000 7f6426d1d700 1 -- start start 2023-10-16T12:24:18.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:18.955+0000 7f6426d1d700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f642015b2c0 0x7f6420161340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:18.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:18.955+0000 7f6426d1d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f642006e570 0x7f642015e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:18.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:18.955+0000 7f6426d1d700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6420161880 con 0x7f642006a7e0 2023-10-16T12:24:18.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:18.955+0000 7f6426d1d700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f642015b6a0 con 0x7f642006e570 2023-10-16T12:24:18.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:18.955+0000 7f6426d1d700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6420161b20 con 0x7f642015b2c0 2023-10-16T12:24:18.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:18.955+0000 7f641ffff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f642006e570 0x7f642015e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:18.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:18.955+0000 7f641ffff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f642006e570 0x7f642015e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38314/0 (socket says 172.21.15.16:38314) 2023-10-16T12:24:18.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:18.955+0000 7f641ffff700 1 -- 172.21.15.16:0/882982575 learned_addr learned my addr 172.21.15.16:0/882982575 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:24:18.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:18.955+0000 7f6424ab9700 1 --2- 172.21.15.16:0/882982575 >> v2:172.21.15.49:3300/0 conn(0x7f642015b2c0 0x7f6420161340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:18.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:18.955+0000 7f641ffff700 1 -- 172.21.15.16:0/882982575 >> v2:172.21.15.49:3300/0 conn(0x7f642015b2c0 msgr2=0x7f6420161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:24:18.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:18.955+0000 7f641ffff700 1 --2- 172.21.15.16:0/882982575 >> v2:172.21.15.49:3300/0 conn(0x7f642015b2c0 0x7f6420161340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:18.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:18.955+0000 7f641ffff700 1 -- 172.21.15.16:0/882982575 >> v1:172.21.15.16:6789/0 conn(0x7f642006a7e0 legacy=0x7f6420071a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:18.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:18.955+0000 7f641ffff700 1 -- 172.21.15.16:0/882982575 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6420161e00 con 0x7f642006e570 2023-10-16T12:24:18.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:18.956+0000 7f641ffff700 1 --2- 172.21.15.16:0/882982575 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f642006e570 0x7f642015e5a0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f642006b4a0 tx=0x7f640c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa1295b82bff2ad4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:24:18.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:18.956+0000 7f641f7fe700 1 -- 172.21.15.16:0/882982575 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f640c00b870 con 0x7f642006e570 2023-10-16T12:24:18.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:18.956+0000 7f641f7fe700 1 -- 172.21.15.16:0/882982575 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f640c00b9d0 con 0x7f642006e570 2023-10-16T12:24:18.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:18.956+0000 7f641f7fe700 1 -- 172.21.15.16:0/882982575 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f640c00bd60 con 0x7f642006e570 2023-10-16T12:24:18.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:18.956+0000 7f6426d1d700 1 -- 172.21.15.16:0/882982575 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f642006e570 msgr2=0x7f642015e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:18.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:18.956+0000 7f6426d1d700 1 --2- 172.21.15.16:0/882982575 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f642006e570 0x7f642015e5a0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f642006b4a0 tx=0x7f640c009bd0 comp rx=0 tx=0).stop 2023-10-16T12:24:18.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:18.956+0000 7f6426d1d700 1 -- 172.21.15.16:0/882982575 shutdown_connections 2023-10-16T12:24:18.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:18.956+0000 7f6426d1d700 1 --2- 172.21.15.16:0/882982575 >> v2:172.21.15.49:3300/0 conn(0x7f642015b2c0 0x7f6420161340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:18.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:18.956+0000 7f6426d1d700 1 --2- 172.21.15.16:0/882982575 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f642006e570 0x7f642015e5a0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:18.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:18.956+0000 7f6426d1d700 1 -- 172.21.15.16:0/882982575 >> 172.21.15.16:0/882982575 conn(0x7f64200b1f40 msgr2=0x7f64200b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:24:18.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:18.956+0000 7f6426d1d700 1 -- 172.21.15.16:0/882982575 shutdown_connections 2023-10-16T12:24:18.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:18.956+0000 7f6426d1d700 1 -- 172.21.15.16:0/882982575 wait complete. 2023-10-16T12:24:18.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:18.958+0000 7f6426d1d700 1 Processor -- start 2023-10-16T12:24:18.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:18.958+0000 7f6426d1d700 1 -- start start 2023-10-16T12:24:18.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:18.958+0000 7f6426d1d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f642006a7e0 0x7f64200c7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:18.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:18.958+0000 7f6426d1d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f642006e570 0x7f642014d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:18.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:18.959+0000 7f6426d1d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f642015b2c0 0x7f64201f35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:18.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:18.959+0000 7f6426d1d700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64200c84e0 con 0x7f642015b2c0 2023-10-16T12:24:18.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:18.959+0000 7f6426d1d700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64201f4120 con 0x7f642006e570 2023-10-16T12:24:18.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:18.959+0000 7f6426d1d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64201f4400 con 0x7f642006a7e0 2023-10-16T12:24:18.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:18.959+0000 7f64252ba700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f642015b2c0 0x7f64201f35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:18.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:18.959+0000 7f64252ba700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f642015b2c0 0x7f64201f35e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38322/0 (socket says 172.21.15.16:38322) 2023-10-16T12:24:18.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:18.959+0000 7f64252ba700 1 -- 172.21.15.16:0/1067300628 learned_addr learned my addr 172.21.15.16:0/1067300628 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:24:18.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:18.959+0000 7f6424ab9700 1 --2- 172.21.15.16:0/1067300628 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f642006a7e0 0x7f64200c7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:18.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:18.959+0000 7f641ffff700 1 --2- 172.21.15.16:0/1067300628 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f642006e570 0x7f642014d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:18.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:18.959+0000 7f64252ba700 1 -- 172.21.15.16:0/1067300628 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f642006a7e0 msgr2=0x7f64200c7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:18.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:18.959+0000 7f64252ba700 1 --2- 172.21.15.16:0/1067300628 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f642006a7e0 0x7f64200c7fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:18.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:18.959+0000 7f64252ba700 1 -- 172.21.15.16:0/1067300628 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f642006e570 msgr2=0x7f642014d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:18.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:18.959+0000 7f64252ba700 1 --2- 172.21.15.16:0/1067300628 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f642006e570 0x7f642014d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:18.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:18.959+0000 7f64252ba700 1 -- 172.21.15.16:0/1067300628 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f640c009550 con 0x7f642015b2c0 2023-10-16T12:24:18.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:18.959+0000 7f641ffff700 1 --2- 172.21.15.16:0/1067300628 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f642006e570 0x7f642014d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:24:18.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:18.959+0000 7f64252ba700 1 --2- 172.21.15.16:0/1067300628 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f642015b2c0 0x7f64201f35e0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f641800cb50 tx=0x7f641800cf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:24:18.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:18.959+0000 7f641d7fa700 1 -- 172.21.15.16:0/1067300628 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6418003c00 con 0x7f642015b2c0 2023-10-16T12:24:18.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:18.959+0000 7f641d7fa700 1 -- 172.21.15.16:0/1067300628 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6418003d60 con 0x7f642015b2c0 2023-10-16T12:24:18.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:18.959+0000 7f6426d1d700 1 -- 172.21.15.16:0/1067300628 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64200c74c0 con 0x7f642015b2c0 2023-10-16T12:24:18.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:18.959+0000 7f641d7fa700 1 -- 172.21.15.16:0/1067300628 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f641800ab70 con 0x7f642015b2c0 2023-10-16T12:24:18.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:18.959+0000 7f6426d1d700 1 -- 172.21.15.16:0/1067300628 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64200c7a80 con 0x7f642015b2c0 2023-10-16T12:24:18.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:18.961+0000 7f6426d1d700 1 -- 172.21.15.16:0/1067300628 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f642004ebb0 con 0x7f642015b2c0 2023-10-16T12:24:18.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:18.961+0000 7f641d7fa700 1 -- 172.21.15.16:0/1067300628 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 22) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f641800acd0 con 0x7f642015b2c0 2023-10-16T12:24:18.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:18.961+0000 7f641d7fa700 1 --2- 172.21.15.16:0/1067300628 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f640406ada0 0x7f640406d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:18.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:18.961+0000 7f641d7fa700 1 -- 172.21.15.16:0/1067300628 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f6418089550 con 0x7f642015b2c0 2023-10-16T12:24:18.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:18.961+0000 7f6424ab9700 1 --2- 172.21.15.16:0/1067300628 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f640406ada0 0x7f640406d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:18.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:18.962+0000 7f6424ab9700 1 --2- 172.21.15.16:0/1067300628 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f640406ada0 0x7f640406d260 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f64200d1010 tx=0x7f6414008040 comp rx=0 tx=0).ready entity=mgr.6291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:24:18.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:18.966+0000 7f641d7fa700 1 -- 172.21.15.16:0/1067300628 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f64180546a0 con 0x7f642015b2c0 2023-10-16T12:24:19.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.198+0000 7f6426d1d700 1 -- 172.21.15.16:0/1067300628 --> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "2m", "target": ["mon-mgr", ""]}) v1 -- 0x7f642015e080 con 0x7f640406ada0 2023-10-16T12:24:19.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.248+0000 7f641d7fa700 1 -- 172.21.15.16:0/1067300628 <== mgr.6291 v2:172.21.15.16:6832/4075855741 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f64140095c0 con 0x7f640406ada0 2023-10-16T12:24:19.252 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule set for path snap_test_dir1 2023-10-16T12:24:19.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.250+0000 7f6426d1d700 1 -- 172.21.15.16:0/1067300628 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f640406ada0 msgr2=0x7f640406d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:19.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.250+0000 7f6426d1d700 1 --2- 172.21.15.16:0/1067300628 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f640406ada0 0x7f640406d260 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f64200d1010 tx=0x7f6414008040 comp rx=0 tx=0).stop 2023-10-16T12:24:19.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.251+0000 7f6426d1d700 1 -- 172.21.15.16:0/1067300628 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f642015b2c0 msgr2=0x7f64201f35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:19.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.251+0000 7f6426d1d700 1 --2- 172.21.15.16:0/1067300628 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f642015b2c0 0x7f64201f35e0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f641800cb50 tx=0x7f641800cf10 comp rx=0 tx=0).stop 2023-10-16T12:24:19.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.251+0000 7f6426d1d700 1 -- 172.21.15.16:0/1067300628 shutdown_connections 2023-10-16T12:24:19.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.251+0000 7f6426d1d700 1 --2- 172.21.15.16:0/1067300628 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f640406ada0 0x7f640406d260 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:19.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.251+0000 7f6426d1d700 1 --2- 172.21.15.16:0/1067300628 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f642015b2c0 0x7f64201f35e0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:19.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.251+0000 7f6426d1d700 1 --2- 172.21.15.16:0/1067300628 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f642006a7e0 0x7f64200c7fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:19.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.251+0000 7f6426d1d700 1 --2- 172.21.15.16:0/1067300628 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f642006e570 0x7f642014d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:19.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.251+0000 7f6426d1d700 1 -- 172.21.15.16:0/1067300628 >> 172.21.15.16:0/1067300628 conn(0x7f64200b1f40 msgr2=0x7f6420071970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:24:19.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.251+0000 7f6426d1d700 1 -- 172.21.15.16:0/1067300628 shutdown_connections 2023-10-16T12:24:19.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.251+0000 7f6426d1d700 1 -- 172.21.15.16:0/1067300628 wait complete. 2023-10-16T12:24:19.266 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2023-10-16T12:24:19.266 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2023-10-16-12_25_00 in ~41s... 2023-10-16T12:24:19.267 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2023-10-16-12_26_00 in ~101s... 2023-10-16T12:24:19.267 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m', '2m'], expected_retention: [] 2023-10-16T12:24:19.267 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json snap_test_dir1 2023-10-16T12:24:19.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.394+0000 7f8ac5552700 1 Processor -- start 2023-10-16T12:24:19.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.394+0000 7f8ac5552700 1 -- start start 2023-10-16T12:24:19.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.395+0000 7f8ac5552700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f8ac015d8d0 0x7f8ac00b2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:19.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.395+0000 7f8ac5552700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8ac00ce350 0x7f8ac00ce780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:19.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.395+0000 7f8ac5552700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ac00cecc0 con 0x7f8ac00b8fa0 2023-10-16T12:24:19.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.395+0000 7f8ac5552700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ac00d0bb0 con 0x7f8ac00ce350 2023-10-16T12:24:19.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.395+0000 7f8ac5552700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ac015dcb0 con 0x7f8ac015d8d0 2023-10-16T12:24:19.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.395+0000 7f8abf7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8ac00ce350 0x7f8ac00ce780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:19.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.395+0000 7f8abf7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8ac00ce350 0x7f8ac00ce780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38330/0 (socket says 172.21.15.16:38330) 2023-10-16T12:24:19.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.395+0000 7f8abf7fe700 1 -- 172.21.15.16:0/3172186443 learned_addr learned my addr 172.21.15.16:0/3172186443 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:24:19.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.395+0000 7f8abeffd700 1 --2- 172.21.15.16:0/3172186443 >> v2:172.21.15.49:3300/0 conn(0x7f8ac015d8d0 0x7f8ac00b2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:19.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.395+0000 7f8abf7fe700 1 -- 172.21.15.16:0/3172186443 >> v2:172.21.15.49:3300/0 conn(0x7f8ac015d8d0 msgr2=0x7f8ac00b2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:24:19.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.395+0000 7f8abf7fe700 1 --2- 172.21.15.16:0/3172186443 >> v2:172.21.15.49:3300/0 conn(0x7f8ac015d8d0 0x7f8ac00b2830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:19.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.395+0000 7f8abf7fe700 1 -- 172.21.15.16:0/3172186443 >> v1:172.21.15.49:6789/0 conn(0x7f8ac00b8fa0 legacy=0x7f8ac00b6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:24:19.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.395+0000 7f8abf7fe700 1 -- 172.21.15.16:0/3172186443 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ac00b2f30 con 0x7f8ac00ce350 2023-10-16T12:24:19.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.395+0000 7f8abf7fe700 1 --2- 172.21.15.16:0/3172186443 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ac00ce350 0x7f8ac00ce780 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f8ac00d0dd0 tx=0x7f8ab800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8bd0a90c3105daa server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:24:19.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.396+0000 7f8abdffb700 1 -- 172.21.15.16:0/3172186443 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ab800e070 con 0x7f8ac00ce350 2023-10-16T12:24:19.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.396+0000 7f8abdffb700 1 -- 172.21.15.16:0/3172186443 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ab8003a20 con 0x7f8ac00ce350 2023-10-16T12:24:19.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.396+0000 7f8abdffb700 1 -- 172.21.15.16:0/3172186443 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ab8003dd0 con 0x7f8ac00ce350 2023-10-16T12:24:19.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.396+0000 7f8ac5552700 1 -- 172.21.15.16:0/3172186443 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ac00ce350 msgr2=0x7f8ac00ce780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:19.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.396+0000 7f8ac5552700 1 --2- 172.21.15.16:0/3172186443 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ac00ce350 0x7f8ac00ce780 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f8ac00d0dd0 tx=0x7f8ab800b330 comp rx=0 tx=0).stop 2023-10-16T12:24:19.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.396+0000 7f8ac5552700 1 -- 172.21.15.16:0/3172186443 shutdown_connections 2023-10-16T12:24:19.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.396+0000 7f8ac5552700 1 --2- 172.21.15.16:0/3172186443 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ac00ce350 0x7f8ac00ce780 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:19.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.396+0000 7f8ac5552700 1 --2- 172.21.15.16:0/3172186443 >> v2:172.21.15.49:3300/0 conn(0x7f8ac015d8d0 0x7f8ac00b2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:19.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.396+0000 7f8ac5552700 1 -- 172.21.15.16:0/3172186443 >> 172.21.15.16:0/3172186443 conn(0x7f8ac0072330 msgr2=0x7f8ac0072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:24:19.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.396+0000 7f8ac5552700 1 -- 172.21.15.16:0/3172186443 shutdown_connections 2023-10-16T12:24:19.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.396+0000 7f8ac5552700 1 -- 172.21.15.16:0/3172186443 wait complete. 2023-10-16T12:24:19.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.398+0000 7f8ac5552700 1 Processor -- start 2023-10-16T12:24:19.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.398+0000 7f8ac5552700 1 -- start start 2023-10-16T12:24:19.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.398+0000 7f8ac5552700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8ac00b8fa0 0x7f8ac01f3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:19.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.398+0000 7f8ac5552700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ac00ce350 0x7f8ac00c8f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:19.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.398+0000 7f8ac5552700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8ac015d8d0 0x7f8ac01547b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:19.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.398+0000 7f8ac5552700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac01f4100 con 0x7f8ac00ce350 2023-10-16T12:24:19.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.398+0000 7f8ac5552700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac00c73c0 con 0x7f8ac00b8fa0 2023-10-16T12:24:19.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.398+0000 7f8ac5552700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac00c76a0 con 0x7f8ac015d8d0 2023-10-16T12:24:19.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.398+0000 7f8abe7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ac00ce350 0x7f8ac00c8f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:19.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.398+0000 7f8abe7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ac00ce350 0x7f8ac00c8f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38332/0 (socket says 172.21.15.16:38332) 2023-10-16T12:24:19.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.398+0000 7f8abe7fc700 1 -- 172.21.15.16:0/4277039186 learned_addr learned my addr 172.21.15.16:0/4277039186 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:24:19.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.399+0000 7f8abf7fe700 1 --2- 172.21.15.16:0/4277039186 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8ac015d8d0 0x7f8ac01547b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:19.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.399+0000 7f8abe7fc700 1 -- 172.21.15.16:0/4277039186 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8ac015d8d0 msgr2=0x7f8ac01547b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:19.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.399+0000 7f8abe7fc700 1 --2- 172.21.15.16:0/4277039186 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8ac015d8d0 0x7f8ac01547b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:19.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.399+0000 7f8abe7fc700 1 -- 172.21.15.16:0/4277039186 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8ac00b8fa0 msgr2=0x7f8ac01f3bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:19.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.399+0000 7f8abe7fc700 1 --2- 172.21.15.16:0/4277039186 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8ac00b8fa0 0x7f8ac01f3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:19.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.399+0000 7f8abe7fc700 1 -- 172.21.15.16:0/4277039186 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ab8009b30 con 0x7f8ac00ce350 2023-10-16T12:24:19.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.399+0000 7f8abe7fc700 1 --2- 172.21.15.16:0/4277039186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ac00ce350 0x7f8ac00c8f30 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f8ab000b470 tx=0x7f8ab000b830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:24:19.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.399+0000 7f8aa77fe700 1 -- 172.21.15.16:0/4277039186 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ab0011830 con 0x7f8ac00ce350 2023-10-16T12:24:19.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.399+0000 7f8aa77fe700 1 -- 172.21.15.16:0/4277039186 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ab0011990 con 0x7f8ac00ce350 2023-10-16T12:24:19.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.399+0000 7f8ac5552700 1 -- 172.21.15.16:0/4277039186 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ac00c7980 con 0x7f8ac00ce350 2023-10-16T12:24:19.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.399+0000 7f8aa77fe700 1 -- 172.21.15.16:0/4277039186 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ab0011d40 con 0x7f8ac00ce350 2023-10-16T12:24:19.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.399+0000 7f8ac5552700 1 -- 172.21.15.16:0/4277039186 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ac00c7f40 con 0x7f8ac00ce350 2023-10-16T12:24:19.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.400+0000 7f8ac5552700 1 -- 172.21.15.16:0/4277039186 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ac00c8160 con 0x7f8ac00ce350 2023-10-16T12:24:19.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.401+0000 7f8aa77fe700 1 -- 172.21.15.16:0/4277039186 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 22) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f8ab001a400 con 0x7f8ac00ce350 2023-10-16T12:24:19.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.401+0000 7f8aa77fe700 1 --2- 172.21.15.16:0/4277039186 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f8aac06ada0 0x7f8aac06d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:24:19.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.401+0000 7f8aa77fe700 1 -- 172.21.15.16:0/4277039186 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f8ab00895a0 con 0x7f8ac00ce350 2023-10-16T12:24:19.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.401+0000 7f8abeffd700 1 --2- 172.21.15.16:0/4277039186 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f8aac06ada0 0x7f8aac06d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:24:19.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.401+0000 7f8abeffd700 1 --2- 172.21.15.16:0/4277039186 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f8aac06ada0 0x7f8aac06d260 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f8ab400a890 tx=0x7f8ab4008040 comp rx=0 tx=0).ready entity=mgr.6291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:24:19.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.406+0000 7f8aa77fe700 1 -- 172.21.15.16:0/4277039186 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8ab00546f0 con 0x7f8ac00ce350 2023-10-16T12:24:19.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.635+0000 7f8ac5552700 1 -- 172.21.15.16:0/4277039186 --> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8ac00d1ee0 con 0x7f8aac06ada0 2023-10-16T12:24:19.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.645+0000 7f8aa77fe700 1 -- 172.21.15.16:0/4277039186 <== mgr.6291 v2:172.21.15.16:6832/4075855741 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+76 (secure 0 0 0) 0x7f8ab4013420 con 0x7f8aac06ada0 2023-10-16T12:24:19.648 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:24:19.648 INFO:teuthology.orchestra.run.smithi016.stdout:{"path": "snap_test_dir1", "schedule": ["1m", "2m"], "retention": [{}, {}]} 2023-10-16T12:24:19.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.647+0000 7f8ac5552700 1 -- 172.21.15.16:0/4277039186 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f8aac06ada0 msgr2=0x7f8aac06d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:19.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.647+0000 7f8ac5552700 1 --2- 172.21.15.16:0/4277039186 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f8aac06ada0 0x7f8aac06d260 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f8ab400a890 tx=0x7f8ab4008040 comp rx=0 tx=0).stop 2023-10-16T12:24:19.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.647+0000 7f8ac5552700 1 -- 172.21.15.16:0/4277039186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ac00ce350 msgr2=0x7f8ac00c8f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:24:19.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.647+0000 7f8ac5552700 1 --2- 172.21.15.16:0/4277039186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ac00ce350 0x7f8ac00c8f30 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f8ab000b470 tx=0x7f8ab000b830 comp rx=0 tx=0).stop 2023-10-16T12:24:19.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.647+0000 7f8ac5552700 1 -- 172.21.15.16:0/4277039186 shutdown_connections 2023-10-16T12:24:19.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.647+0000 7f8ac5552700 1 --2- 172.21.15.16:0/4277039186 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f8aac06ada0 0x7f8aac06d260 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:19.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.647+0000 7f8ac5552700 1 --2- 172.21.15.16:0/4277039186 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8ac015d8d0 0x7f8ac01547b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:19.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.647+0000 7f8ac5552700 1 --2- 172.21.15.16:0/4277039186 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8ac00b8fa0 0x7f8ac01f3bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:19.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.647+0000 7f8ac5552700 1 --2- 172.21.15.16:0/4277039186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ac00ce350 0x7f8ac00c8f30 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:24:19.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.647+0000 7f8ac5552700 1 -- 172.21.15.16:0/4277039186 >> 172.21.15.16:0/4277039186 conn(0x7f8ac0072330 msgr2=0x7f8ac00d2f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:24:19.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.647+0000 7f8ac5552700 1 -- 172.21.15.16:0/4277039186 shutdown_connections 2023-10-16T12:24:19.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:24:19.647+0000 7f8ac5552700 1 -- 172.21.15.16:0/4277039186 wait complete. 2023-10-16T12:24:19.662 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1m", "2m"], "retention": [{}, {}]} 2023-10-16T12:24:19.663 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1m', '2m'], 'retention': [{}, {}]} 2023-10-16T12:24:19.663 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:24:19.664 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:24:19.682 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:24:19.683 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:24:19.683 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:24:20.331 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:24:20.334 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:24:20.684 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:24:20.685 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:24:20.708 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:24:20.709 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:24:20.709 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:24:21.710 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:24:21.711 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:24:21.733 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:24:21.734 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:24:21.734 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:24:22.735 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:24:22.736 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:24:22.761 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:24:22.761 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:24:22.762 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:24:23.763 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:24:23.763 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:24:23.786 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:24:23.786 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:24:23.787 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:24:24.788 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:24:24.788 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:24:24.811 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:24:24.811 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:24:24.812 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:24:25.812 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:24:25.813 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:24:25.836 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:24:25.836 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:24:25.836 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:24:26.837 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:24:26.838 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:24:26.861 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:24:26.861 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:24:26.861 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:24:27.863 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:24:27.863 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:24:27.886 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:24:27.886 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:24:27.887 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:24:28.888 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:24:28.888 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:24:28.911 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:24:28.911 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:24:28.912 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:24:29.913 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:24:29.913 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:24:29.936 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:24:29.936 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:24:29.937 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:24:30.937 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:24:30.938 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:24:30.961 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:24:30.961 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:24:30.961 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:24:31.963 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:24:31.963 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:24:31.986 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:24:31.986 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:24:31.987 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:24:32.988 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:24:32.989 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:24:33.011 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:24:33.011 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:24:33.012 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:24:34.013 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:24:34.014 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:24:34.036 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:24:34.037 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:24:34.037 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:24:35.038 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:24:35.039 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:24:35.062 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:24:35.062 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:24:35.062 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:24:36.064 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:24:36.064 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:24:36.087 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:24:36.087 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:24:36.087 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:24:37.089 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:24:37.089 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:24:37.112 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:24:37.113 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:24:37.113 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:24:38.114 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:24:38.114 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:24:38.137 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:24:38.138 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:24:38.138 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:24:39.139 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:24:39.140 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:24:39.162 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:24:39.163 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:24:39.163 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:24:40.164 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:24:40.165 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:24:40.188 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:24:40.188 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:24:40.188 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:24:41.189 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:24:41.190 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:24:41.212 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:24:41.212 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:24:41.213 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:24:42.214 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:24:42.215 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:24:42.238 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:24:42.238 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:24:42.239 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:24:43.240 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:24:43.240 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:24:43.264 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:24:43.264 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:24:43.264 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:24:44.265 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:24:44.266 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:24:44.288 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:24:44.289 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:24:44.289 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:24:45.290 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:24:45.290 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:24:45.313 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:24:45.314 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:24:45.314 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:24:46.316 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:24:46.316 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:24:46.339 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:24:46.340 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:24:46.340 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:24:47.341 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:24:47.342 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:24:47.364 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:24:47.365 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:24:47.365 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:24:48.366 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:24:48.367 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:24:48.390 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:24:48.390 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:24:48.390 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:24:49.391 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:24:49.392 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:24:49.414 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:24:49.415 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:24:49.415 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:24:50.402 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:24:50.405 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:24:50.415 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:24:50.416 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:24:50.468 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:24:50.469 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:24:50.469 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:24:51.471 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:24:51.471 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:24:51.495 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:24:51.495 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:24:51.495 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:24:52.496 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:24:52.497 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:24:52.520 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:24:52.520 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:24:52.521 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:24:53.521 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:24:53.522 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:24:53.544 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:24:53.545 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:24:53.545 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:24:54.546 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:24:54.547 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:24:54.570 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:24:54.571 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:24:54.571 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:24:55.572 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:24:55.572 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:24:55.595 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:24:55.595 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:24:55.595 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:24:56.596 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:24:56.597 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:24:56.620 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:24:56.620 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:24:56.620 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:24:57.621 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:24:57.621 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:24:57.644 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:24:57.645 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:24:57.645 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:24:58.646 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:24:58.647 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:24:58.670 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:24:58.670 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:24:58.671 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:24:59.672 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:24:59.673 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:24:59.695 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:24:59.696 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:24:59.696 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:25:00.262 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:25:00.260+0000 7f15d5ec0700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:25:00.262 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:25:00.260+0000 7f15d5ec0700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:25:00.263 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:25:00.260+0000 7f15d5ec0700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:25:00.263 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:25:00.260+0000 7f15d5ec0700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:25:00.263 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:25:00.260+0000 7f15d5ec0700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:25:00.697 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:25:00.698 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:25:00.720 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_25_00_UTC 2023-10-16T12:25:00.722 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_25_00_UTC'} 2023-10-16T12:25:00.722 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2023-10-16-12_25_00_UTC'} 2023-10-16T12:25:00.723 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:25:00.723 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2023-10-16-12_25_00_UTC'] 2023-10-16T12:25:00.723 DEBUG:tasks.cephfs.test_snap_schedules:exec=1697459059.2667265, now = 1697459100.723907, timo = 41 2023-10-16T12:25:01.725 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:25:01.726 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:25:01.749 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_25_00_UTC 2023-10-16T12:25:01.749 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_25_00_UTC'} 2023-10-16T12:25:01.749 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:25:01.750 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:25:02.751 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:25:02.751 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:25:02.773 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_25_00_UTC 2023-10-16T12:25:02.774 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_25_00_UTC'} 2023-10-16T12:25:02.775 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:25:02.775 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:25:03.776 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:25:03.777 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:25:03.800 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_25_00_UTC 2023-10-16T12:25:03.800 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_25_00_UTC'} 2023-10-16T12:25:03.800 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:25:03.801 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:25:04.802 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:25:04.802 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:25:04.824 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_25_00_UTC 2023-10-16T12:25:04.825 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_25_00_UTC'} 2023-10-16T12:25:04.826 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:25:04.826 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:25:05.827 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:25:05.827 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:25:05.849 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_25_00_UTC 2023-10-16T12:25:05.850 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_25_00_UTC'} 2023-10-16T12:25:05.851 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:25:05.851 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:25:06.852 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:25:06.853 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:25:06.875 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_25_00_UTC 2023-10-16T12:25:06.876 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_25_00_UTC'} 2023-10-16T12:25:06.876 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:25:06.876 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:25:07.878 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:25:07.878 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:25:07.900 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_25_00_UTC 2023-10-16T12:25:07.901 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_25_00_UTC'} 2023-10-16T12:25:07.901 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:25:07.902 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:25:08.903 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:25:08.903 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:25:08.926 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_25_00_UTC 2023-10-16T12:25:08.927 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_25_00_UTC'} 2023-10-16T12:25:08.927 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:25:08.927 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:25:09.928 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:25:09.929 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:25:09.951 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_25_00_UTC 2023-10-16T12:25:09.952 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_25_00_UTC'} 2023-10-16T12:25:09.952 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:25:09.952 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:25:10.953 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:25:10.954 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:25:10.977 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_25_00_UTC 2023-10-16T12:25:10.977 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_25_00_UTC'} 2023-10-16T12:25:10.977 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:25:10.978 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:25:11.979 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:25:11.980 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:25:12.003 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_25_00_UTC 2023-10-16T12:25:12.003 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_25_00_UTC'} 2023-10-16T12:25:12.004 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:25:12.004 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:25:13.005 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:25:13.006 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:25:13.028 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_25_00_UTC 2023-10-16T12:25:13.029 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_25_00_UTC'} 2023-10-16T12:25:13.029 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:25:13.030 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:25:14.031 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:25:14.031 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:25:14.053 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_25_00_UTC 2023-10-16T12:25:14.054 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_25_00_UTC'} 2023-10-16T12:25:14.055 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:25:14.055 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:25:15.056 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:25:15.057 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:25:15.079 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_25_00_UTC 2023-10-16T12:25:15.080 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_25_00_UTC'} 2023-10-16T12:25:15.080 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:25:15.081 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:25:16.082 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:25:16.083 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:25:16.105 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_25_00_UTC 2023-10-16T12:25:16.106 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_25_00_UTC'} 2023-10-16T12:25:16.106 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:25:16.107 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:25:17.108 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:25:17.109 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:25:17.131 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_25_00_UTC 2023-10-16T12:25:17.132 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_25_00_UTC'} 2023-10-16T12:25:17.132 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:25:17.133 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:25:18.134 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:25:18.134 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:25:18.157 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_25_00_UTC 2023-10-16T12:25:18.158 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_25_00_UTC'} 2023-10-16T12:25:18.158 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:25:18.159 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:25:19.160 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:25:19.160 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:25:19.182 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_25_00_UTC 2023-10-16T12:25:19.183 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_25_00_UTC'} 2023-10-16T12:25:19.183 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:25:19.184 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:25:20.185 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:25:20.186 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:25:20.208 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_25_00_UTC 2023-10-16T12:25:20.209 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_25_00_UTC'} 2023-10-16T12:25:20.210 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:25:20.210 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:25:20.469 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:25:20.472 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:25:21.211 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:25:21.212 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:25:21.234 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_25_00_UTC 2023-10-16T12:25:21.235 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_25_00_UTC'} 2023-10-16T12:25:21.235 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:25:21.235 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:25:22.236 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:25:22.237 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:25:22.259 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_25_00_UTC 2023-10-16T12:25:22.260 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_25_00_UTC'} 2023-10-16T12:25:22.260 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:25:22.261 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:25:23.262 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:25:23.262 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:25:23.284 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_25_00_UTC 2023-10-16T12:25:23.285 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_25_00_UTC'} 2023-10-16T12:25:23.285 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:25:23.285 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:25:24.287 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:25:24.287 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:25:24.309 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_25_00_UTC 2023-10-16T12:25:24.310 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_25_00_UTC'} 2023-10-16T12:25:24.310 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:25:24.310 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:25:25.311 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:25:25.312 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:25:25.334 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_25_00_UTC 2023-10-16T12:25:25.335 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_25_00_UTC'} 2023-10-16T12:25:25.335 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:25:25.335 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:25:26.337 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:25:26.337 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:25:26.360 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_25_00_UTC 2023-10-16T12:25:26.360 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_25_00_UTC'} 2023-10-16T12:25:26.360 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:25:26.361 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:25:27.361 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:25:27.362 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:25:27.384 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_25_00_UTC 2023-10-16T12:25:27.385 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_25_00_UTC'} 2023-10-16T12:25:27.385 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:25:27.385 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:25:28.387 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:25:28.387 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:25:28.409 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_25_00_UTC 2023-10-16T12:25:28.411 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_25_00_UTC'} 2023-10-16T12:25:28.411 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:25:28.411 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:25:29.412 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:25:29.413 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:25:29.435 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_25_00_UTC 2023-10-16T12:25:29.436 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_25_00_UTC'} 2023-10-16T12:25:29.436 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:25:29.437 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:25:30.438 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:25:30.439 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:25:30.460 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_25_00_UTC 2023-10-16T12:25:30.461 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_25_00_UTC'} 2023-10-16T12:25:30.462 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:25:30.462 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:25:31.464 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:25:31.465 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:25:31.487 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_25_00_UTC 2023-10-16T12:25:31.488 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_25_00_UTC'} 2023-10-16T12:25:31.488 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:25:31.488 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:25:32.489 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:25:32.490 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:25:32.512 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_25_00_UTC 2023-10-16T12:25:32.512 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_25_00_UTC'} 2023-10-16T12:25:32.512 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:25:32.513 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:25:33.514 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:25:33.515 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:25:33.537 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_25_00_UTC 2023-10-16T12:25:33.538 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_25_00_UTC'} 2023-10-16T12:25:33.538 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:25:33.539 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:25:34.540 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:25:34.541 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:25:34.562 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_25_00_UTC 2023-10-16T12:25:34.564 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_25_00_UTC'} 2023-10-16T12:25:34.564 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:25:34.564 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:25:35.565 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:25:35.566 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:25:35.588 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_25_00_UTC 2023-10-16T12:25:35.589 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_25_00_UTC'} 2023-10-16T12:25:35.589 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:25:35.589 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:25:36.590 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:25:36.591 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:25:36.613 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_25_00_UTC 2023-10-16T12:25:36.614 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_25_00_UTC'} 2023-10-16T12:25:36.615 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:25:36.615 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:25:37.616 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:25:37.616 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:25:37.638 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_25_00_UTC 2023-10-16T12:25:37.639 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_25_00_UTC'} 2023-10-16T12:25:37.640 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:25:37.640 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:25:38.641 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:25:38.641 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:25:38.663 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_25_00_UTC 2023-10-16T12:25:38.664 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_25_00_UTC'} 2023-10-16T12:25:38.664 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:25:38.664 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:25:39.666 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:25:39.666 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:25:39.688 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_25_00_UTC 2023-10-16T12:25:39.689 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_25_00_UTC'} 2023-10-16T12:25:39.690 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:25:39.690 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:25:40.691 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:25:40.692 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:25:40.713 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_25_00_UTC 2023-10-16T12:25:40.715 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_25_00_UTC'} 2023-10-16T12:25:40.715 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:25:40.715 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:25:41.717 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:25:41.717 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:25:41.739 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_25_00_UTC 2023-10-16T12:25:41.740 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_25_00_UTC'} 2023-10-16T12:25:41.740 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:25:41.741 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:25:42.742 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:25:42.742 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:25:42.764 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_25_00_UTC 2023-10-16T12:25:42.765 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_25_00_UTC'} 2023-10-16T12:25:42.765 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:25:42.766 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:25:43.766 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:25:43.767 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:25:43.789 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_25_00_UTC 2023-10-16T12:25:43.790 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_25_00_UTC'} 2023-10-16T12:25:43.790 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:25:43.790 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:25:44.791 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:25:44.792 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:25:44.814 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_25_00_UTC 2023-10-16T12:25:44.815 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_25_00_UTC'} 2023-10-16T12:25:44.815 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:25:44.815 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:25:45.817 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:25:45.817 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:25:45.839 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_25_00_UTC 2023-10-16T12:25:45.840 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_25_00_UTC'} 2023-10-16T12:25:45.840 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:25:45.841 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:25:46.842 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:25:46.842 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:25:46.864 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_25_00_UTC 2023-10-16T12:25:46.865 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_25_00_UTC'} 2023-10-16T12:25:46.866 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:25:46.866 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:25:47.867 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:25:47.868 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:25:47.891 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_25_00_UTC 2023-10-16T12:25:47.892 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_25_00_UTC'} 2023-10-16T12:25:47.892 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:25:47.893 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:25:48.893 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:25:48.894 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:25:48.916 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_25_00_UTC 2023-10-16T12:25:48.917 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_25_00_UTC'} 2023-10-16T12:25:48.917 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:25:48.918 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:25:49.919 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:25:49.919 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:25:49.941 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_25_00_UTC 2023-10-16T12:25:49.942 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_25_00_UTC'} 2023-10-16T12:25:49.943 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:25:49.943 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:25:50.554 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:25:50.557 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:25:50.944 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:25:50.945 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:25:50.967 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_25_00_UTC 2023-10-16T12:25:50.968 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_25_00_UTC'} 2023-10-16T12:25:50.968 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:25:50.969 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:25:51.969 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:25:51.970 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:25:51.992 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_25_00_UTC 2023-10-16T12:25:51.993 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_25_00_UTC'} 2023-10-16T12:25:51.993 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:25:51.994 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:25:52.995 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:25:52.995 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:25:53.017 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_25_00_UTC 2023-10-16T12:25:53.018 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_25_00_UTC'} 2023-10-16T12:25:53.019 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:25:53.019 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:25:54.020 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:25:54.021 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:25:54.043 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_25_00_UTC 2023-10-16T12:25:54.044 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_25_00_UTC'} 2023-10-16T12:25:54.044 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:25:54.045 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:25:55.046 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:25:55.046 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:25:55.068 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_25_00_UTC 2023-10-16T12:25:55.069 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_25_00_UTC'} 2023-10-16T12:25:55.070 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:25:55.070 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:25:56.071 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:25:56.071 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:25:56.093 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_25_00_UTC 2023-10-16T12:25:56.094 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_25_00_UTC'} 2023-10-16T12:25:56.094 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:25:56.095 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:25:57.095 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:25:57.096 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:25:57.118 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_25_00_UTC 2023-10-16T12:25:57.119 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_25_00_UTC'} 2023-10-16T12:25:57.119 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:25:57.120 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:25:58.121 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:25:58.122 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:25:58.143 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_25_00_UTC 2023-10-16T12:25:58.144 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_25_00_UTC'} 2023-10-16T12:25:58.145 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:25:58.145 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:25:59.146 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:25:59.147 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:25:59.169 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_25_00_UTC 2023-10-16T12:25:59.170 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_25_00_UTC'} 2023-10-16T12:25:59.170 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:25:59.171 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:26:00.172 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:26:00.172 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:26:00.195 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_25_00_UTC 2023-10-16T12:26:00.195 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_25_00_UTC'} 2023-10-16T12:26:00.195 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:26:00.195 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:26:01.197 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:26:01.197 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:26:01.219 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_25_00_UTC 2023-10-16T12:26:01.220 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_26_00_UTC 2023-10-16T12:26:01.221 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_26_00_UTC', 'scheduled-2023-10-16-12_25_00_UTC'} 2023-10-16T12:26:01.221 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2023-10-16-12_26_00_UTC'} 2023-10-16T12:26:01.221 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:26:01.222 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2023-10-16-12_26_00_UTC'] 2023-10-16T12:26:01.222 DEBUG:tasks.cephfs.test_snap_schedules:exec=1697459059.2667265, now = 1697459161.2224343, timo = 101 2023-10-16T12:26:02.223 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2023-10-16T12:26:02.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:02.365+0000 7f7292547700 1 Processor -- start 2023-10-16T12:26:02.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:02.366+0000 7f7292547700 1 -- start start 2023-10-16T12:26:02.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:02.366+0000 7f7292547700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f728c0d4060 0x7f728c0d1520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:02.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:02.366+0000 7f7292547700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f728c0d1d20 0x7f728c0d21b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:02.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:02.366+0000 7f7292547700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f728c0d4540 con 0x7f728c15da10 2023-10-16T12:26:02.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:02.366+0000 7f7292547700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f728c0cf9b0 con 0x7f728c0d1d20 2023-10-16T12:26:02.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:02.366+0000 7f7292547700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f728c0d26f0 con 0x7f728c0d4060 2023-10-16T12:26:02.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:02.367+0000 7f7290ae4700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f728c0d1d20 0x7f728c0d21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:02.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:02.367+0000 7f7290ae4700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f728c0d1d20 0x7f728c0d21b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38372/0 (socket says 172.21.15.16:38372) 2023-10-16T12:26:02.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:02.367+0000 7f7290ae4700 1 -- 172.21.15.16:0/2899948851 learned_addr learned my addr 172.21.15.16:0/2899948851 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:02.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:02.367+0000 7f728b7fe700 1 --2- 172.21.15.16:0/2899948851 >> v2:172.21.15.49:3300/0 conn(0x7f728c0d4060 0x7f728c0d1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:02.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:02.367+0000 7f7290ae4700 1 -- 172.21.15.16:0/2899948851 >> v2:172.21.15.49:3300/0 conn(0x7f728c0d4060 msgr2=0x7f728c0d1520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:02.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:02.367+0000 7f7290ae4700 1 --2- 172.21.15.16:0/2899948851 >> v2:172.21.15.49:3300/0 conn(0x7f728c0d4060 0x7f728c0d1520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:02.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:02.367+0000 7f7290ae4700 1 -- 172.21.15.16:0/2899948851 >> v1:172.21.15.49:6789/0 conn(0x7f728c15da10 legacy=0x7f728c0cf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:02.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:02.367+0000 7f7290ae4700 1 -- 172.21.15.16:0/2899948851 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f728c0ce7e0 con 0x7f728c0d1d20 2023-10-16T12:26:02.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:02.367+0000 7f7290ae4700 1 --2- 172.21.15.16:0/2899948851 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f728c0d1d20 0x7f728c0d21b0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f728c0cdf70 tx=0x7f728400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ae176d844fe9850 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:02.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:02.367+0000 7f728affd700 1 -- 172.21.15.16:0/2899948851 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f728400e070 con 0x7f728c0d1d20 2023-10-16T12:26:02.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:02.367+0000 7f728affd700 1 -- 172.21.15.16:0/2899948851 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7284003a20 con 0x7f728c0d1d20 2023-10-16T12:26:02.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:02.367+0000 7f728affd700 1 -- 172.21.15.16:0/2899948851 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7284003dd0 con 0x7f728c0d1d20 2023-10-16T12:26:02.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:02.368+0000 7f7292547700 1 -- 172.21.15.16:0/2899948851 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f728c0d1d20 msgr2=0x7f728c0d21b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:02.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:02.368+0000 7f7292547700 1 --2- 172.21.15.16:0/2899948851 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f728c0d1d20 0x7f728c0d21b0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f728c0cdf70 tx=0x7f728400b330 comp rx=0 tx=0).stop 2023-10-16T12:26:02.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:02.368+0000 7f7292547700 1 -- 172.21.15.16:0/2899948851 shutdown_connections 2023-10-16T12:26:02.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:02.368+0000 7f7292547700 1 --2- 172.21.15.16:0/2899948851 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f728c0d1d20 0x7f728c0d21b0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:02.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:02.368+0000 7f7292547700 1 --2- 172.21.15.16:0/2899948851 >> v2:172.21.15.49:3300/0 conn(0x7f728c0d4060 0x7f728c0d1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:02.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:02.368+0000 7f7292547700 1 -- 172.21.15.16:0/2899948851 >> 172.21.15.16:0/2899948851 conn(0x7f728c09c9f0 msgr2=0x7f728c0d3740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:02.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:02.368+0000 7f7292547700 1 -- 172.21.15.16:0/2899948851 shutdown_connections 2023-10-16T12:26:02.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:02.368+0000 7f7292547700 1 -- 172.21.15.16:0/2899948851 wait complete. 2023-10-16T12:26:02.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:02.369+0000 7f7292547700 1 Processor -- start 2023-10-16T12:26:02.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:02.369+0000 7f7292547700 1 -- start start 2023-10-16T12:26:02.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:02.370+0000 7f7292547700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f728c0d1d20 0x7f728c155560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:02.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:02.370+0000 7f7292547700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f728c0d4060 0x7f728c0b6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:02.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:02.370+0000 7f7292547700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f728c15da10 0x7f728c15fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:02.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:02.370+0000 7f7292547700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f728c0b73c0 con 0x7f728c0d4060 2023-10-16T12:26:02.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:02.370+0000 7f7292547700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f728c0b81a0 con 0x7f728c15da10 2023-10-16T12:26:02.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:02.370+0000 7f7292547700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f728c0b6e00 con 0x7f728c0d1d20 2023-10-16T12:26:02.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:02.370+0000 7f728b7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f728c0d4060 0x7f728c0b6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:02.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:02.370+0000 7f728b7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f728c0d4060 0x7f728c0b6360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38378/0 (socket says 172.21.15.16:38378) 2023-10-16T12:26:02.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:02.370+0000 7f728b7fe700 1 -- 172.21.15.16:0/3906887475 learned_addr learned my addr 172.21.15.16:0/3906887475 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:02.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:02.370+0000 7f7290ae4700 1 --2- 172.21.15.16:0/3906887475 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f728c15da10 0x7f728c15fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:02.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:02.370+0000 7f728bfff700 1 --2- 172.21.15.16:0/3906887475 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f728c0d1d20 0x7f728c155560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:02.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:02.370+0000 7f728b7fe700 1 -- 172.21.15.16:0/3906887475 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f728c0d1d20 msgr2=0x7f728c155560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:02.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:02.370+0000 7f728b7fe700 1 --2- 172.21.15.16:0/3906887475 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f728c0d1d20 0x7f728c155560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:02.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:02.370+0000 7f728b7fe700 1 -- 172.21.15.16:0/3906887475 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f728c15da10 msgr2=0x7f728c15fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:02.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:02.370+0000 7f728b7fe700 1 --2- 172.21.15.16:0/3906887475 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f728c15da10 0x7f728c15fd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:02.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:02.371+0000 7f728b7fe700 1 -- 172.21.15.16:0/3906887475 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7284009b30 con 0x7f728c0d4060 2023-10-16T12:26:02.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:02.371+0000 7f728b7fe700 1 --2- 172.21.15.16:0/3906887475 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f728c0d4060 0x7f728c0b6360 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f727400e840 tx=0x7f727400ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:02.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:02.371+0000 7f7288ff9700 1 -- 172.21.15.16:0/3906887475 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7274013610 con 0x7f728c0d4060 2023-10-16T12:26:02.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:02.371+0000 7f7288ff9700 1 -- 172.21.15.16:0/3906887475 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7274013770 con 0x7f728c0d4060 2023-10-16T12:26:02.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:02.371+0000 7f7292547700 1 -- 172.21.15.16:0/3906887475 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f728c0b70e0 con 0x7f728c0d4060 2023-10-16T12:26:02.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:02.371+0000 7f7288ff9700 1 -- 172.21.15.16:0/3906887475 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7274013b20 con 0x7f728c0d4060 2023-10-16T12:26:02.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:02.371+0000 7f7292547700 1 -- 172.21.15.16:0/3906887475 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f728c155aa0 con 0x7f728c0d4060 2023-10-16T12:26:02.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:02.372+0000 7f7292547700 1 -- 172.21.15.16:0/3906887475 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f728c0b7c20 con 0x7f728c0d4060 2023-10-16T12:26:02.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:02.372+0000 7f7288ff9700 1 -- 172.21.15.16:0/3906887475 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) v1 ==== 89360+0+0 (secure 0 0 0) 0x7f7274013c80 con 0x7f728c0d4060 2023-10-16T12:26:02.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:02.373+0000 7f7288ff9700 1 --2- 172.21.15.16:0/3906887475 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f726c06b070 0x7f726c06d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:02.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:02.373+0000 7f7288ff9700 1 -- 172.21.15.16:0/3906887475 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f7274089850 con 0x7f728c0d4060 2023-10-16T12:26:02.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:02.373+0000 7f728bfff700 1 --2- 172.21.15.16:0/3906887475 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f726c06b070 0x7f726c06d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:02.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:02.373+0000 7f728bfff700 1 --2- 172.21.15.16:0/3906887475 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f726c06b070 0x7f726c06d530 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f727c00a7d0 tx=0x7f727c008040 comp rx=0 tx=0).ready entity=mgr.6291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:02.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:02.378+0000 7f7288ff9700 1 -- 172.21.15.16:0/3906887475 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f728c0b7c20 con 0x7f728c0d4060 2023-10-16T12:26:02.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:02.609+0000 7f7292547700 1 -- 172.21.15.16:0/3906887475 --> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f728c0ceda0 con 0x7f726c06b070 2023-10-16T12:26:02.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:02.672+0000 7f7288ff9700 1 -- 172.21.15.16:0/3906887475 <== mgr.6291 v2:172.21.15.16:6832/4075855741 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f727c013400 con 0x7f726c06b070 2023-10-16T12:26:02.674 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule removed for path snap_test_dir1 2023-10-16T12:26:02.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:02.674+0000 7f7292547700 1 -- 172.21.15.16:0/3906887475 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f726c06b070 msgr2=0x7f726c06d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:02.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:02.674+0000 7f7292547700 1 --2- 172.21.15.16:0/3906887475 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f726c06b070 0x7f726c06d530 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f727c00a7d0 tx=0x7f727c008040 comp rx=0 tx=0).stop 2023-10-16T12:26:02.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:02.674+0000 7f7292547700 1 -- 172.21.15.16:0/3906887475 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f728c0d4060 msgr2=0x7f728c0b6360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:02.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:02.674+0000 7f7292547700 1 --2- 172.21.15.16:0/3906887475 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f728c0d4060 0x7f728c0b6360 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f727400e840 tx=0x7f727400ec00 comp rx=0 tx=0).stop 2023-10-16T12:26:02.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:02.674+0000 7f7292547700 1 -- 172.21.15.16:0/3906887475 shutdown_connections 2023-10-16T12:26:02.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:02.674+0000 7f7292547700 1 --2- 172.21.15.16:0/3906887475 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f726c06b070 0x7f726c06d530 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:02.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:02.674+0000 7f7292547700 1 --2- 172.21.15.16:0/3906887475 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f728c15da10 0x7f728c15fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:02.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:02.674+0000 7f7292547700 1 --2- 172.21.15.16:0/3906887475 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f728c0d1d20 0x7f728c155560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:02.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:02.674+0000 7f7292547700 1 --2- 172.21.15.16:0/3906887475 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f728c0d4060 0x7f728c0b6360 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:02.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:02.674+0000 7f7292547700 1 -- 172.21.15.16:0/3906887475 >> 172.21.15.16:0/3906887475 conn(0x7f728c09c9f0 msgr2=0x7f728c15de80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:02.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:02.675+0000 7f7292547700 1 -- 172.21.15.16:0/3906887475 shutdown_connections 2023-10-16T12:26:02.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:02.675+0000 7f7292547700 1 -- 172.21.15.16:0/3906887475 wait complete. 2023-10-16T12:26:02.688 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2023-10-16T12:26:02.689 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:26:02.689 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:26:02.707 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_25_00_UTC 2023-10-16T12:26:02.707 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_26_00_UTC 2023-10-16T12:26:02.707 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-12_25_00_UTC 2023-10-16T12:26:02.707 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:26:02.708 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-12_25_00_UTC) 2023-10-16T12:26:02.768 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-12_26_00_UTC 2023-10-16T12:26:02.769 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:26:02.769 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-12_26_00_UTC) 2023-10-16T12:26:02.830 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:26:02.831 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2023-10-16T12:26:02.891 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-16T12:26:03.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.056+0000 7f2dc192a700 1 Processor -- start 2023-10-16T12:26:03.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.056+0000 7f2dc192a700 1 -- start start 2023-10-16T12:26:03.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.057+0000 7f2dc192a700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f2dbc15d8d0 0x7f2dbc0b2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:03.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.057+0000 7f2dc192a700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dbc0b2f30 con 0x7f2dbc0b8fa0 2023-10-16T12:26:03.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.057+0000 7f2dc192a700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dbc0d1c40 con 0x7f2dbc0ce350 2023-10-16T12:26:03.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.057+0000 7f2dc192a700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dbc0d0bb0 con 0x7f2dbc15d8d0 2023-10-16T12:26:03.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.057+0000 7f2dba7fc700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2dbc0b8fa0 0x7f2dbc0b6720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:55892/0 (socket says 172.21.15.16:55892) 2023-10-16T12:26:03.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.057+0000 7f2dba7fc700 1 -- 172.21.15.16:0/2407419647 learned_addr learned my addr 172.21.15.16:0/2407419647 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:26:03.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.057+0000 7f2dbaffd700 1 --2- 172.21.15.16:0/2407419647 >> v2:172.21.15.49:3300/0 conn(0x7f2dbc15d8d0 0x7f2dbc0b2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:03.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.057+0000 7f2db9ffb700 1 -- 172.21.15.16:0/2407419647 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2092107775 0 0) 0x7f2dbc0b2f30 con 0x7f2dbc0b8fa0 2023-10-16T12:26:03.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.057+0000 7f2db9ffb700 1 -- 172.21.15.16:0/2407419647 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2da40035b0 con 0x7f2dbc0b8fa0 2023-10-16T12:26:03.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.058+0000 7f2db9ffb700 1 -- 172.21.15.16:0/2407419647 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3126151577 0 0) 0x7f2da40035b0 con 0x7f2dbc0b8fa0 2023-10-16T12:26:03.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.058+0000 7f2db9ffb700 1 -- 172.21.15.16:0/2407419647 >> v2:172.21.15.49:3300/0 conn(0x7f2dbc15d8d0 msgr2=0x7f2dbc0b2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:03.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.058+0000 7f2db9ffb700 1 --2- 172.21.15.16:0/2407419647 >> v2:172.21.15.49:3300/0 conn(0x7f2dbc15d8d0 0x7f2dbc0b2830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:03.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.058+0000 7f2db9ffb700 1 -- 172.21.15.16:0/2407419647 >> v1:172.21.15.49:6789/0 conn(0x7f2dbc0ce350 legacy=0x7f2dbc0d29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:03.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.058+0000 7f2db9ffb700 1 -- 172.21.15.16:0/2407419647 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dbc0d1ee0 con 0x7f2dbc0b8fa0 2023-10-16T12:26:03.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.058+0000 7f2db9ffb700 1 -- 172.21.15.16:0/2407419647 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f2dac001670 con 0x7f2dbc0b8fa0 2023-10-16T12:26:03.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.058+0000 7f2db9ffb700 1 -- 172.21.15.16:0/2407419647 >> v1:172.21.15.16:6789/0 conn(0x7f2dbc0b8fa0 legacy=0x7f2dbc0b6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:03.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.058+0000 7f2db9ffb700 1 --2- 172.21.15.16:0/2407419647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2da4003a00 0x7f2da4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:03.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.058+0000 7f2db9ffb700 1 --2- 172.21.15.16:0/2407419647 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2da4004850 0x7f2da4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:03.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.058+0000 7f2db9ffb700 1 --2- 172.21.15.16:0/2407419647 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2da4005210 0x7f2da400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:03.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.058+0000 7f2db9ffb700 1 -- 172.21.15.16:0/2407419647 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dbc0d0bb0 con 0x7f2da4003a00 2023-10-16T12:26:03.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.058+0000 7f2db9ffb700 1 -- 172.21.15.16:0/2407419647 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2da400c1b0 con 0x7f2da4004850 2023-10-16T12:26:03.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.058+0000 7f2db9ffb700 1 -- 172.21.15.16:0/2407419647 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2da400c370 con 0x7f2da4005210 2023-10-16T12:26:03.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.058+0000 7f2dbaffd700 1 --2- 172.21.15.16:0/2407419647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2da4003a00 0x7f2da4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:03.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.058+0000 7f2dba7fc700 1 --2- 172.21.15.16:0/2407419647 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2da4004850 0x7f2da4004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:03.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.058+0000 7f2dbb7fe700 1 --2- 172.21.15.16:0/2407419647 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2da4005210 0x7f2da400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:03.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.058+0000 7f2dbaffd700 1 -- 172.21.15.16:0/2407419647 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2da4005210 msgr2=0x7f2da400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:03.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.058+0000 7f2dbaffd700 1 --2- 172.21.15.16:0/2407419647 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2da4005210 0x7f2da400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:03.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.058+0000 7f2dbaffd700 1 -- 172.21.15.16:0/2407419647 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2da4004850 msgr2=0x7f2da4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:03.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.058+0000 7f2dbaffd700 1 --2- 172.21.15.16:0/2407419647 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2da4004850 0x7f2da4004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:03.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.058+0000 7f2dbaffd700 1 -- 172.21.15.16:0/2407419647 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2da40035b0 con 0x7f2da4003a00 2023-10-16T12:26:03.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.058+0000 7f2dbb7fe700 1 -- 172.21.15.16:0/2407419647 reap_dead start 2023-10-16T12:26:03.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.058+0000 7f2dbaffd700 1 --2- 172.21.15.16:0/2407419647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2da4003a00 0x7f2da4003e20 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f2db0009b60 tx=0x7f2da4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69bc3a39e258d7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:03.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.058+0000 7f2db9ffb700 1 -- 172.21.15.16:0/2407419647 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2db000e6c0 con 0x7f2da4003a00 2023-10-16T12:26:03.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.058+0000 7f2db9ffb700 1 -- 172.21.15.16:0/2407419647 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2db000e820 con 0x7f2da4003a00 2023-10-16T12:26:03.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.058+0000 7f2db9ffb700 1 -- 172.21.15.16:0/2407419647 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2db000ebb0 con 0x7f2da4003a00 2023-10-16T12:26:03.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.059+0000 7f2dc192a700 1 -- 172.21.15.16:0/2407419647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2da4003a00 msgr2=0x7f2da4003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:03.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.059+0000 7f2dc192a700 1 --2- 172.21.15.16:0/2407419647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2da4003a00 0x7f2da4003e20 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f2db0009b60 tx=0x7f2da4005880 comp rx=0 tx=0).stop 2023-10-16T12:26:03.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.059+0000 7f2dc192a700 1 -- 172.21.15.16:0/2407419647 shutdown_connections 2023-10-16T12:26:03.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.059+0000 7f2dc192a700 1 --2- 172.21.15.16:0/2407419647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2da4003a00 0x7f2da4003e20 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:03.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.059+0000 7f2dc192a700 1 -- 172.21.15.16:0/2407419647 >> 172.21.15.16:0/2407419647 conn(0x7f2dbc072330 msgr2=0x7f2dbc072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:03.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.059+0000 7f2dc192a700 1 -- 172.21.15.16:0/2407419647 shutdown_connections 2023-10-16T12:26:03.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.059+0000 7f2dc192a700 1 -- 172.21.15.16:0/2407419647 wait complete. 2023-10-16T12:26:03.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.061+0000 7f2dc192a700 1 Processor -- start 2023-10-16T12:26:03.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.061+0000 7f2dc192a700 1 -- start start 2023-10-16T12:26:03.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.062+0000 7f2dc192a700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2da4003a00 0x7f2dbc0b1a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:03.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.062+0000 7f2dc192a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2dbc15ed00 0x7f2dbc0cc710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:03.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.062+0000 7f2dc192a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2dbc15f180 0x7f2dbc0ccc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:03.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.062+0000 7f2dc192a700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dbc1602f0 con 0x7f2dbc15f180 2023-10-16T12:26:03.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.062+0000 7f2dc192a700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dbc15f6e0 con 0x7f2da4003a00 2023-10-16T12:26:03.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.062+0000 7f2dc192a700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dbc15f9c0 con 0x7f2dbc15ed00 2023-10-16T12:26:03.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.062+0000 7f2dbb7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2dbc15f180 0x7f2dbc0ccc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:03.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.062+0000 7f2dbb7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2dbc15f180 0x7f2dbc0ccc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38400/0 (socket says 172.21.15.16:38400) 2023-10-16T12:26:03.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.062+0000 7f2dbb7fe700 1 -- 172.21.15.16:0/3165465921 learned_addr learned my addr 172.21.15.16:0/3165465921 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:03.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.062+0000 7f2dbaffd700 1 --2- 172.21.15.16:0/3165465921 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2da4003a00 0x7f2dbc0b1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:03.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.062+0000 7f2dba7fc700 1 --2- 172.21.15.16:0/3165465921 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2dbc15ed00 0x7f2dbc0cc710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:03.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.062+0000 7f2dbb7fe700 1 -- 172.21.15.16:0/3165465921 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2dbc15ed00 msgr2=0x7f2dbc0cc710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:03.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.062+0000 7f2dbb7fe700 1 --2- 172.21.15.16:0/3165465921 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2dbc15ed00 0x7f2dbc0cc710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:03.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.062+0000 7f2dbb7fe700 1 -- 172.21.15.16:0/3165465921 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2da4003a00 msgr2=0x7f2dbc0b1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:03.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.062+0000 7f2dbb7fe700 1 --2- 172.21.15.16:0/3165465921 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2da4003a00 0x7f2dbc0b1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:03.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.062+0000 7f2dbb7fe700 1 -- 172.21.15.16:0/3165465921 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dbc15fca0 con 0x7f2dbc15f180 2023-10-16T12:26:03.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.062+0000 7f2dba7fc700 1 --2- 172.21.15.16:0/3165465921 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2dbc15ed00 0x7f2dbc0cc710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:26:03.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.062+0000 7f2dbb7fe700 1 --2- 172.21.15.16:0/3165465921 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2dbc15f180 0x7f2dbc0ccc50 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f2dbc0cdef0 tx=0x7f2db400ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:03.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.063+0000 7f2da3fff700 1 -- 172.21.15.16:0/3165465921 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2db4003c10 con 0x7f2dbc15f180 2023-10-16T12:26:03.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.063+0000 7f2da3fff700 1 -- 172.21.15.16:0/3165465921 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2db4003d70 con 0x7f2dbc15f180 2023-10-16T12:26:03.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.063+0000 7f2da3fff700 1 -- 172.21.15.16:0/3165465921 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2db4005310 con 0x7f2dbc15f180 2023-10-16T12:26:03.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.063+0000 7f2dc192a700 1 -- 172.21.15.16:0/3165465921 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dbc15ff80 con 0x7f2dbc15f180 2023-10-16T12:26:03.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.063+0000 7f2dc192a700 1 -- 172.21.15.16:0/3165465921 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dbc1f32b0 con 0x7f2dbc15f180 2023-10-16T12:26:03.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.064+0000 7f2dc192a700 1 -- 172.21.15.16:0/3165465921 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dbc04ebb0 con 0x7f2dbc15f180 2023-10-16T12:26:03.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.064+0000 7f2da3fff700 1 -- 172.21.15.16:0/3165465921 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) v1 ==== 89360+0+0 (secure 0 0 0) 0x7f2db4005470 con 0x7f2dbc15f180 2023-10-16T12:26:03.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.065+0000 7f2da3fff700 1 --2- 172.21.15.16:0/3165465921 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f2da806b070 0x7f2da806d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:03.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.065+0000 7f2da3fff700 1 -- 172.21.15.16:0/3165465921 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f2db4089900 con 0x7f2dbc15f180 2023-10-16T12:26:03.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.065+0000 7f2dbaffd700 1 --2- 172.21.15.16:0/3165465921 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f2da806b070 0x7f2da806d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:03.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.065+0000 7f2dbaffd700 1 --2- 172.21.15.16:0/3165465921 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f2da806b070 0x7f2da806d530 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2db00090b0 tx=0x7f2db0008030 comp rx=0 tx=0).ready entity=mgr.6291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:03.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.069+0000 7f2da3fff700 1 -- 172.21.15.16:0/3165465921 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2db40545d0 con 0x7f2dbc15f180 2023-10-16T12:26:03.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:03.338+0000 7f2dc192a700 1 -- 172.21.15.16:0/3165465921 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f2dbc0b5f80 con 0x7f2dbc15f180 2023-10-16T12:26:04.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.274+0000 7f2da3fff700 1 -- 172.21.15.16:0/3165465921 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v27) v1 ==== 93+0+0 (secure 0 0 0) 0x7f2db4057bf0 con 0x7f2dbc15f180 2023-10-16T12:26:04.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.285+0000 7f2da3fff700 1 -- 172.21.15.16:0/3165465921 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mgrmap(e 27) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f2db4087b50 con 0x7f2dbc15f180 2023-10-16T12:26:04.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.288+0000 7f2dc192a700 1 -- 172.21.15.16:0/3165465921 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f2da806b070 msgr2=0x7f2da806d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:04.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.288+0000 7f2dc192a700 1 --2- 172.21.15.16:0/3165465921 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f2da806b070 0x7f2da806d530 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2db00090b0 tx=0x7f2db0008030 comp rx=0 tx=0).stop 2023-10-16T12:26:04.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.288+0000 7f2dc192a700 1 -- 172.21.15.16:0/3165465921 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2dbc15f180 msgr2=0x7f2dbc0ccc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:04.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.288+0000 7f2dc192a700 1 --2- 172.21.15.16:0/3165465921 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2dbc15f180 0x7f2dbc0ccc50 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f2dbc0cdef0 tx=0x7f2db400ee00 comp rx=0 tx=0).stop 2023-10-16T12:26:04.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.288+0000 7f2dc192a700 1 -- 172.21.15.16:0/3165465921 shutdown_connections 2023-10-16T12:26:04.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.288+0000 7f2dc192a700 1 --2- 172.21.15.16:0/3165465921 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f2da806b070 0x7f2da806d530 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2db00090b0 tx=0x7f2db0008030 comp rx=0 tx=0).stop 2023-10-16T12:26:04.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.288+0000 7f2dc192a700 1 --2- 172.21.15.16:0/3165465921 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2dbc15f180 0x7f2dbc0ccc50 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:04.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.288+0000 7f2dc192a700 1 --2- 172.21.15.16:0/3165465921 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2da4003a00 0x7f2dbc0b1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:04.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.288+0000 7f2dc192a700 1 --2- 172.21.15.16:0/3165465921 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2dbc15ed00 0x7f2dbc0cc710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:04.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.288+0000 7f2dc192a700 1 -- 172.21.15.16:0/3165465921 >> 172.21.15.16:0/3165465921 conn(0x7f2dbc072330 msgr2=0x7f2dbc0b39d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:04.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.292+0000 7f2dc192a700 1 -- 172.21.15.16:0/3165465921 shutdown_connections 2023-10-16T12:26:04.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.292+0000 7f2dc192a700 1 -- 172.21.15.16:0/3165465921 wait complete. 2023-10-16T12:26:04.309 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T12:26:04.309 DEBUG:teuthology.orchestra.run.smithi016:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T12:26:04.351 DEBUG:teuthology.orchestra.run.smithi049:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T12:26:04.403 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T12:26:04.403 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:26:04.404 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:26:04.422 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:26:04.423 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:26:04.430 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:26:04.428+0000 7fab230f8200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T12:26:04.439 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:04.437+0000 7fd5cc790200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T12:26:04.457 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:26:04.499 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:26:04.497+0000 7fab230f8200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T12:26:04.509 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:04.507+0000 7fd5cc790200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T12:26:04.575 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:26:04.573+0000 7fab230f8200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T12:26:04.586 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:04.585+0000 7fd5cc790200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T12:26:04.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.622+0000 7efc261c1700 1 Processor -- start 2023-10-16T12:26:04.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.623+0000 7efc261c1700 1 -- start start 2023-10-16T12:26:04.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.623+0000 7efc261c1700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efc2015d8d0 0x7efc200d1c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:04.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.623+0000 7efc261c1700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc2006ba00 con 0x7efc200d0750 2023-10-16T12:26:04.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.623+0000 7efc261c1700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc200d2150 con 0x7efc200d41c0 2023-10-16T12:26:04.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.623+0000 7efc261c1700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc200d26d0 con 0x7efc2015d8d0 2023-10-16T12:26:04.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.624+0000 7efc1ffff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7efc200d0750 0x7efc2006c4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:55996/0 (socket says 172.21.15.16:55996) 2023-10-16T12:26:04.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.624+0000 7efc1ffff700 1 -- 172.21.15.16:0/355867775 learned_addr learned my addr 172.21.15.16:0/355867775 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:26:04.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.624+0000 7efc1f7fe700 1 --2- 172.21.15.16:0/355867775 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efc2015d8d0 0x7efc200d1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:04.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.624+0000 7efc1e7fc700 1 -- 172.21.15.16:0/355867775 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1642731836 0 0) 0x7efc2006ba00 con 0x7efc200d0750 2023-10-16T12:26:04.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.624+0000 7efc1e7fc700 1 -- 172.21.15.16:0/355867775 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc080035b0 con 0x7efc200d0750 2023-10-16T12:26:04.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.624+0000 7efc1e7fc700 1 -- 172.21.15.16:0/355867775 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3628619915 0 0) 0x7efc200d2150 con 0x7efc200d41c0 2023-10-16T12:26:04.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.624+0000 7efc1e7fc700 1 -- 172.21.15.16:0/355867775 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc2006ba00 con 0x7efc200d41c0 2023-10-16T12:26:04.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.624+0000 7efc1f7fe700 1 -- 172.21.15.16:0/355867775 >> v1:172.21.15.49:6789/0 conn(0x7efc200d41c0 legacy=0x7efc200d0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:04.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.624+0000 7efc1f7fe700 1 -- 172.21.15.16:0/355867775 >> v1:172.21.15.16:6789/0 conn(0x7efc200d0750 legacy=0x7efc2006c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:04.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.624+0000 7efc1f7fe700 1 -- 172.21.15.16:0/355867775 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc200d23f0 con 0x7efc2015d8d0 2023-10-16T12:26:04.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.624+0000 7efc1f7fe700 1 --2- 172.21.15.16:0/355867775 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efc2015d8d0 0x7efc200d1c10 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7efc10009870 tx=0x7efc10009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d831a987e81f0790 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:04.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.624+0000 7efc1e7fc700 1 -- 172.21.15.16:0/355867775 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc10011830 con 0x7efc2015d8d0 2023-10-16T12:26:04.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.624+0000 7efc1e7fc700 1 -- 172.21.15.16:0/355867775 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc10011990 con 0x7efc2015d8d0 2023-10-16T12:26:04.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.624+0000 7efc1e7fc700 1 -- 172.21.15.16:0/355867775 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc10011d20 con 0x7efc2015d8d0 2023-10-16T12:26:04.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.625+0000 7efc261c1700 1 -- 172.21.15.16:0/355867775 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efc2015d8d0 msgr2=0x7efc200d1c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:04.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.625+0000 7efc261c1700 1 --2- 172.21.15.16:0/355867775 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efc2015d8d0 0x7efc200d1c10 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7efc10009870 tx=0x7efc10009b80 comp rx=0 tx=0).stop 2023-10-16T12:26:04.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.625+0000 7efc261c1700 1 -- 172.21.15.16:0/355867775 shutdown_connections 2023-10-16T12:26:04.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.625+0000 7efc261c1700 1 --2- 172.21.15.16:0/355867775 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efc2015d8d0 0x7efc200d1c10 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:04.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.625+0000 7efc261c1700 1 -- 172.21.15.16:0/355867775 >> 172.21.15.16:0/355867775 conn(0x7efc200d3630 msgr2=0x7efc200d3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:04.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.625+0000 7efc261c1700 1 -- 172.21.15.16:0/355867775 shutdown_connections 2023-10-16T12:26:04.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.625+0000 7efc261c1700 1 -- 172.21.15.16:0/355867775 wait complete. 2023-10-16T12:26:04.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.626+0000 7efc261c1700 1 Processor -- start 2023-10-16T12:26:04.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.627+0000 7efc261c1700 1 -- start start 2023-10-16T12:26:04.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.627+0000 7efc261c1700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efc200d0750 0x7efc2009fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:04.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.627+0000 7efc261c1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc200d41c0 0x7efc2009d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:04.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.627+0000 7efc261c1700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efc2015d8d0 0x7efc201f2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:04.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.627+0000 7efc261c1700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc200a1200 con 0x7efc200d41c0 2023-10-16T12:26:04.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.627+0000 7efc261c1700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc200a0580 con 0x7efc200d0750 2023-10-16T12:26:04.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.627+0000 7efc261c1700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc200a0860 con 0x7efc2015d8d0 2023-10-16T12:26:04.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.627+0000 7efc1effd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc200d41c0 0x7efc2009d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:04.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.627+0000 7efc1effd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc200d41c0 0x7efc2009d580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38494/0 (socket says 172.21.15.16:38494) 2023-10-16T12:26:04.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.627+0000 7efc1effd700 1 -- 172.21.15.16:0/20783317 learned_addr learned my addr 172.21.15.16:0/20783317 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:04.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.628+0000 7efc1f7fe700 1 --2- 172.21.15.16:0/20783317 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efc200d0750 0x7efc2009fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:04.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.628+0000 7efc1effd700 1 -- 172.21.15.16:0/20783317 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efc2015d8d0 msgr2=0x7efc201f2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:04.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.628+0000 7efc1effd700 1 --2- 172.21.15.16:0/20783317 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efc2015d8d0 0x7efc201f2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:04.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.628+0000 7efc1effd700 1 -- 172.21.15.16:0/20783317 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efc200d0750 msgr2=0x7efc2009fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:04.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.628+0000 7efc1effd700 1 --2- 172.21.15.16:0/20783317 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efc200d0750 0x7efc2009fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:04.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.628+0000 7efc1effd700 1 -- 172.21.15.16:0/20783317 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc100094a0 con 0x7efc200d41c0 2023-10-16T12:26:04.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.628+0000 7efc1effd700 1 --2- 172.21.15.16:0/20783317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc200d41c0 0x7efc2009d580 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7efc1400c520 tx=0x7efc1400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:04.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.628+0000 7efc07fff700 1 -- 172.21.15.16:0/20783317 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc140052d0 con 0x7efc200d41c0 2023-10-16T12:26:04.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.628+0000 7efc07fff700 1 -- 172.21.15.16:0/20783317 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc14005430 con 0x7efc200d41c0 2023-10-16T12:26:04.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.628+0000 7efc261c1700 1 -- 172.21.15.16:0/20783317 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc200a0b40 con 0x7efc200d41c0 2023-10-16T12:26:04.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.628+0000 7efc07fff700 1 -- 172.21.15.16:0/20783317 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc140038c0 con 0x7efc200d41c0 2023-10-16T12:26:04.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.628+0000 7efc261c1700 1 -- 172.21.15.16:0/20783317 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc200a0270 con 0x7efc200d41c0 2023-10-16T12:26:04.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.629+0000 7efc261c1700 1 -- 172.21.15.16:0/20783317 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc2004ebb0 con 0x7efc200d41c0 2023-10-16T12:26:04.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.630+0000 7efc07fff700 1 -- 172.21.15.16:0/20783317 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 27) v1 ==== 89343+0+0 (secure 0 0 0) 0x7efc14003a20 con 0x7efc200d41c0 2023-10-16T12:26:04.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.630+0000 7efc07fff700 1 --2- 172.21.15.16:0/20783317 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7efc0c06b020 0x7efc0c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:04.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.630+0000 7efc1f7fe700 1 -- 172.21.15.16:0/20783317 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7efc0c06b020 msgr2=0x7efc0c06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/4075855741 2023-10-16T12:26:04.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.631+0000 7efc07fff700 1 -- 172.21.15.16:0/20783317 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6519+0+0 (secure 0 0 0) 0x7efc140890c0 con 0x7efc200d41c0 2023-10-16T12:26:04.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.631+0000 7efc1f7fe700 1 --2- 172.21.15.16:0/20783317 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7efc0c06b020 0x7efc0c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:26:04.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.639+0000 7efc07fff700 1 -- 172.21.15.16:0/20783317 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7efc14053cb0 con 0x7efc200d41c0 2023-10-16T12:26:04.658 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:26:04.656+0000 7fab230f8200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T12:26:04.675 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:04.674+0000 7fd5cc790200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T12:26:04.809 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:26:04.807+0000 7fab230f8200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T12:26:04.826 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:04.824+0000 7fd5cc790200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T12:26:04.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.832+0000 7efc1f7fe700 1 -- 172.21.15.16:0/20783317 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7efc0c06b020 msgr2=0x7efc0c06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/4075855741 2023-10-16T12:26:04.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.832+0000 7efc1f7fe700 1 --2- 172.21.15.16:0/20783317 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7efc0c06b020 0x7efc0c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:26:04.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.848+0000 7efc261c1700 1 -- 172.21.15.16:0/20783317 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efc200d1410 con 0x7efc200d41c0 2023-10-16T12:26:04.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.849+0000 7efc07fff700 1 -- 172.21.15.16:0/20783317 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) v1 ==== 74+0+17359 (secure 0 0 0) 0x7efc140572d0 con 0x7efc200d41c0 2023-10-16T12:26:04.851 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:26:04.851 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":72,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:26:04.269379+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:24:01.200494+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-10-16T12:24:01.652885+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":4,"snap_epoch":72,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.14","mappings":[{"from":0,"to":6}]},{"pgid":"8.1d","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/2103819518":"2023-10-17T12:24:17.735687+0000","172.21.15.16:0/224613758":"2023-10-17T12:24:17.735687+0000","172.21.15.16:0/877922677":"2023-10-17T12:24:17.735687+0000","172.21.15.16:0/1135331895":"2023-10-17T12:24:17.735687+0000","172.21.15.16:0/1214553139":"2023-10-17T12:24:17.735687+0000","172.21.15.16:0/1240820252":"2023-10-17T12:24:17.735687+0000","172.21.15.16:6833/3870937344":"2023-10-17T12:24:17.735687+0000","172.21.15.16:6832/3870937344":"2023-10-17T12:24:17.735687+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":1},{"begin":4,"length":1}]}],"new_removed_snaps":[{"pool":8,"snaps":[{"begin":2,"length":1},{"begin":4,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:26:04.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.851+0000 7efc261c1700 1 -- 172.21.15.16:0/20783317 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7efc0c06b020 msgr2=0x7efc0c06d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:26:04.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.851+0000 7efc261c1700 1 --2- 172.21.15.16:0/20783317 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7efc0c06b020 0x7efc0c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:04.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.851+0000 7efc261c1700 1 -- 172.21.15.16:0/20783317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc200d41c0 msgr2=0x7efc2009d580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:04.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.851+0000 7efc261c1700 1 --2- 172.21.15.16:0/20783317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc200d41c0 0x7efc2009d580 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7efc1400c520 tx=0x7efc1400c8e0 comp rx=0 tx=0).stop 2023-10-16T12:26:04.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.851+0000 7efc261c1700 1 -- 172.21.15.16:0/20783317 shutdown_connections 2023-10-16T12:26:04.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.851+0000 7efc261c1700 1 --2- 172.21.15.16:0/20783317 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7efc0c06b020 0x7efc0c06d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:04.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.851+0000 7efc261c1700 1 --2- 172.21.15.16:0/20783317 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efc2015d8d0 0x7efc201f2170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:04.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.851+0000 7efc261c1700 1 --2- 172.21.15.16:0/20783317 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efc200d0750 0x7efc2009fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:04.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.851+0000 7efc261c1700 1 --2- 172.21.15.16:0/20783317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc200d41c0 0x7efc2009d580 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:04.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.851+0000 7efc261c1700 1 -- 172.21.15.16:0/20783317 >> 172.21.15.16:0/20783317 conn(0x7efc200d3630 msgr2=0x7efc2015bc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:04.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.852+0000 7efc261c1700 1 -- 172.21.15.16:0/20783317 shutdown_connections 2023-10-16T12:26:04.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:04.852+0000 7efc261c1700 1 -- 172.21.15.16:0/20783317 wait complete. 2023-10-16T12:26:04.866 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T12:24:01.200494+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T12:24:01.652885+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 4, 'snap_epoch': 72, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T12:26:04.867 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:26:05.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.035+0000 7feaedba8700 1 Processor -- start 2023-10-16T12:26:05.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.035+0000 7feaedba8700 1 -- start start 2023-10-16T12:26:05.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.036+0000 7feaedba8700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7feae80ce6a0 0x7feae806c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:05.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.036+0000 7feaedba8700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7feae80b2620 0x7feae806c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:05.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.036+0000 7feaedba8700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feae80b20f0 con 0x7feae80d2110 2023-10-16T12:26:05.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.036+0000 7feaedba8700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feae80af1f0 con 0x7feae80b2620 2023-10-16T12:26:05.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.036+0000 7feaedba8700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feae80d03d0 con 0x7feae80ce6a0 2023-10-16T12:26:05.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.036+0000 7feadffff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7feae80d2110 0x7feae80b0f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:56052/0 (socket says 172.21.15.16:56052) 2023-10-16T12:26:05.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.036+0000 7feadffff700 1 -- 172.21.15.16:0/2882524966 learned_addr learned my addr 172.21.15.16:0/2882524966 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:26:05.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.036+0000 7feae77fe700 1 --2- 172.21.15.16:0/2882524966 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7feae80ce6a0 0x7feae806c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:05.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.036+0000 7feae7fff700 1 --2- 172.21.15.16:0/2882524966 >> v2:172.21.15.49:3300/0 conn(0x7feae80b2620 0x7feae806c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:05.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.036+0000 7feae6ffd700 1 -- 172.21.15.16:0/2882524966 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1799518461 0 0) 0x7feae80b20f0 con 0x7feae80d2110 2023-10-16T12:26:05.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.036+0000 7feae6ffd700 1 -- 172.21.15.16:0/2882524966 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fead40035b0 con 0x7feae80d2110 2023-10-16T12:26:05.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.036+0000 7feae77fe700 1 -- 172.21.15.16:0/2882524966 >> v2:172.21.15.49:3300/0 conn(0x7feae80b2620 msgr2=0x7feae806c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:05.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.036+0000 7feae77fe700 1 --2- 172.21.15.16:0/2882524966 >> v2:172.21.15.49:3300/0 conn(0x7feae80b2620 0x7feae806c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:05.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.036+0000 7feae77fe700 1 -- 172.21.15.16:0/2882524966 >> v1:172.21.15.16:6789/0 conn(0x7feae80d2110 legacy=0x7feae80b0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:05.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.036+0000 7feae77fe700 1 -- 172.21.15.16:0/2882524966 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feae80b2340 con 0x7feae80ce6a0 2023-10-16T12:26:05.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.036+0000 7feae7fff700 1 --2- 172.21.15.16:0/2882524966 >> v2:172.21.15.49:3300/0 conn(0x7feae80b2620 0x7feae806c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:26:05.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.036+0000 7feae77fe700 1 --2- 172.21.15.16:0/2882524966 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7feae80ce6a0 0x7feae806c300 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fead8009840 tx=0x7fead8009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=18c810440a27daba server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:05.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.037+0000 7feae6ffd700 1 -- 172.21.15.16:0/2882524966 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead8011830 con 0x7feae80ce6a0 2023-10-16T12:26:05.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.037+0000 7feae6ffd700 1 -- 172.21.15.16:0/2882524966 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fead8011990 con 0x7feae80ce6a0 2023-10-16T12:26:05.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.037+0000 7feae6ffd700 1 -- 172.21.15.16:0/2882524966 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead8011d00 con 0x7feae80ce6a0 2023-10-16T12:26:05.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.037+0000 7feaedba8700 1 -- 172.21.15.16:0/2882524966 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7feae80ce6a0 msgr2=0x7feae806c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:05.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.037+0000 7feaedba8700 1 --2- 172.21.15.16:0/2882524966 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7feae80ce6a0 0x7feae806c300 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fead8009840 tx=0x7fead8009b50 comp rx=0 tx=0).stop 2023-10-16T12:26:05.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.037+0000 7feaedba8700 1 -- 172.21.15.16:0/2882524966 shutdown_connections 2023-10-16T12:26:05.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.037+0000 7feaedba8700 1 --2- 172.21.15.16:0/2882524966 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7feae80ce6a0 0x7feae806c300 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:05.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.037+0000 7feaedba8700 1 --2- 172.21.15.16:0/2882524966 >> v2:172.21.15.49:3300/0 conn(0x7feae80b2620 0x7feae806c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:05.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.037+0000 7feaedba8700 1 -- 172.21.15.16:0/2882524966 >> 172.21.15.16:0/2882524966 conn(0x7feae80d15f0 msgr2=0x7feae80d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:05.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.038+0000 7feaedba8700 1 -- 172.21.15.16:0/2882524966 shutdown_connections 2023-10-16T12:26:05.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.038+0000 7feaedba8700 1 -- 172.21.15.16:0/2882524966 wait complete. 2023-10-16T12:26:05.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.039+0000 7feaedba8700 1 Processor -- start 2023-10-16T12:26:05.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.039+0000 7feaedba8700 1 -- start start 2023-10-16T12:26:05.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.039+0000 7feaedba8700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7feae80b2620 0x7feae80cabc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:05.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.039+0000 7feaedba8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feae80ce6a0 0x7feae81f25f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:05.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.039+0000 7feaedba8700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feae80d2110 0x7feae814f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:05.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.039+0000 7feaedba8700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae8155fd0 con 0x7feae80ce6a0 2023-10-16T12:26:05.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.039+0000 7feaedba8700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae80cb100 con 0x7feae80d2110 2023-10-16T12:26:05.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.039+0000 7feaedba8700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feae80ca120 con 0x7feae80b2620 2023-10-16T12:26:05.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.040+0000 7feadffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feae80ce6a0 0x7feae81f25f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:05.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.040+0000 7feadffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feae80ce6a0 0x7feae81f25f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38554/0 (socket says 172.21.15.16:38554) 2023-10-16T12:26:05.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.040+0000 7feae7fff700 1 --2- 172.21.15.16:0/816945609 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feae80d2110 0x7feae814f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:05.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.040+0000 7feadffff700 1 -- 172.21.15.16:0/816945609 learned_addr learned my addr 172.21.15.16:0/816945609 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:05.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.040+0000 7feae77fe700 1 --2- 172.21.15.16:0/816945609 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7feae80b2620 0x7feae80cabc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:05.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.040+0000 7feadffff700 1 -- 172.21.15.16:0/816945609 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7feae80b2620 msgr2=0x7feae80cabc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:05.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.040+0000 7feadffff700 1 --2- 172.21.15.16:0/816945609 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7feae80b2620 0x7feae80cabc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:05.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.040+0000 7feadffff700 1 -- 172.21.15.16:0/816945609 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feae80d2110 msgr2=0x7feae814f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:05.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.040+0000 7feadffff700 1 --2- 172.21.15.16:0/816945609 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feae80d2110 0x7feae814f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:05.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.040+0000 7feadffff700 1 -- 172.21.15.16:0/816945609 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feae0009a70 con 0x7feae80ce6a0 2023-10-16T12:26:05.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.040+0000 7feae77fe700 1 --2- 172.21.15.16:0/816945609 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7feae80b2620 0x7feae80cabc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:26:05.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.040+0000 7feae7fff700 1 --2- 172.21.15.16:0/816945609 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feae80d2110 0x7feae814f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:26:05.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.040+0000 7feadffff700 1 --2- 172.21.15.16:0/816945609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feae80ce6a0 0x7feae81f25f0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fead000c420 tx=0x7fead000c730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:05.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.040+0000 7feae4ff9700 1 -- 172.21.15.16:0/816945609 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead0005d70 con 0x7feae80ce6a0 2023-10-16T12:26:05.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.040+0000 7feae4ff9700 1 -- 172.21.15.16:0/816945609 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fead000ce20 con 0x7feae80ce6a0 2023-10-16T12:26:05.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.040+0000 7feaedba8700 1 -- 172.21.15.16:0/816945609 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fead80094a0 con 0x7feae80ce6a0 2023-10-16T12:26:05.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.040+0000 7feae4ff9700 1 -- 172.21.15.16:0/816945609 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead00072d0 con 0x7feae80ce6a0 2023-10-16T12:26:05.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.040+0000 7feaedba8700 1 -- 172.21.15.16:0/816945609 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feae80ca3c0 con 0x7feae80ce6a0 2023-10-16T12:26:05.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.041+0000 7feaedba8700 1 -- 172.21.15.16:0/816945609 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feae804ebb0 con 0x7feae80ce6a0 2023-10-16T12:26:05.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.042+0000 7feae4ff9700 1 -- 172.21.15.16:0/816945609 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 27) v1 ==== 89343+0+0 (secure 0 0 0) 0x7fead0007430 con 0x7feae80ce6a0 2023-10-16T12:26:05.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.042+0000 7feae4ff9700 1 --2- 172.21.15.16:0/816945609 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7feac806b050 0x7feac806d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:05.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.042+0000 7feae77fe700 1 -- 172.21.15.16:0/816945609 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7feac806b050 msgr2=0x7feac806d510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/4075855741 2023-10-16T12:26:05.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.042+0000 7feae4ff9700 1 -- 172.21.15.16:0/816945609 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6519+0+0 (secure 0 0 0) 0x7fead0088c30 con 0x7feae80ce6a0 2023-10-16T12:26:05.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.042+0000 7feae77fe700 1 --2- 172.21.15.16:0/816945609 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7feac806b050 0x7feac806d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:26:05.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.047+0000 7feae4ff9700 1 -- 172.21.15.16:0/816945609 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fead0053950 con 0x7feae80ce6a0 2023-10-16T12:26:05.151 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:26:05.149+0000 7fab230f8200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T12:26:05.177 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:05.175+0000 7fd5cc790200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T12:26:05.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.243+0000 7feae77fe700 1 -- 172.21.15.16:0/816945609 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7feac806b050 msgr2=0x7feac806d510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/4075855741 2023-10-16T12:26:05.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.243+0000 7feae77fe700 1 --2- 172.21.15.16:0/816945609 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7feac806b050 0x7feac806d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:26:05.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.251+0000 7feaedba8700 1 -- 172.21.15.16:0/816945609 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7feae8161040 con 0x7feae80ce6a0 2023-10-16T12:26:05.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.251+0000 7feae4ff9700 1 -- 172.21.15.16:0/816945609 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v72) v1 ==== 93+0+10 (secure 0 0 0) 0x7fead0017020 con 0x7feae80ce6a0 2023-10-16T12:26:05.253 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T12:26:05.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.254+0000 7feaedba8700 1 -- 172.21.15.16:0/816945609 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7feac806b050 msgr2=0x7feac806d510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:26:05.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.254+0000 7feaedba8700 1 --2- 172.21.15.16:0/816945609 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7feac806b050 0x7feac806d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:05.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.254+0000 7feaedba8700 1 -- 172.21.15.16:0/816945609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feae80ce6a0 msgr2=0x7feae81f25f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:05.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.254+0000 7feaedba8700 1 --2- 172.21.15.16:0/816945609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feae80ce6a0 0x7feae81f25f0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fead000c420 tx=0x7fead000c730 comp rx=0 tx=0).stop 2023-10-16T12:26:05.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.254+0000 7feaedba8700 1 -- 172.21.15.16:0/816945609 shutdown_connections 2023-10-16T12:26:05.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.254+0000 7feaedba8700 1 --2- 172.21.15.16:0/816945609 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7feac806b050 0x7feac806d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:05.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.254+0000 7feaedba8700 1 --2- 172.21.15.16:0/816945609 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feae80d2110 0x7feae814f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:05.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.254+0000 7feaedba8700 1 --2- 172.21.15.16:0/816945609 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7feae80b2620 0x7feae80cabc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:05.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.254+0000 7feaedba8700 1 --2- 172.21.15.16:0/816945609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feae80ce6a0 0x7feae81f25f0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:05.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.254+0000 7feaedba8700 1 -- 172.21.15.16:0/816945609 >> 172.21.15.16:0/816945609 conn(0x7feae80d15f0 msgr2=0x7feae80cdc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:05.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.254+0000 7feaedba8700 1 -- 172.21.15.16:0/816945609 shutdown_connections 2023-10-16T12:26:05.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.254+0000 7feaedba8700 1 -- 172.21.15.16:0/816945609 wait complete. 2023-10-16T12:26:05.269 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T12:26:05.270 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:26:05.268+0000 7fab230f8200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T12:26:05.303 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:05.301+0000 7fd5cc790200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T12:26:05.350 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:26:05.349+0000 7fab230f8200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T12:26:05.382 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:05.380+0000 7fd5cc790200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T12:26:05.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.402+0000 7fe372278700 1 Processor -- start 2023-10-16T12:26:05.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.402+0000 7fe372278700 1 -- start start 2023-10-16T12:26:05.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.403+0000 7fe372278700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe36c15ec80 0x7fe36c0cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:05.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.403+0000 7fe372278700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe36c0d2110 0x7fe36c0cd1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:05.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.403+0000 7fe372278700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe36c0b2440 con 0x7fe36c15f060 2023-10-16T12:26:05.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.403+0000 7fe372278700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe36c0b2c50 con 0x7fe36c0d2110 2023-10-16T12:26:05.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.403+0000 7fe372278700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe36c0d00e0 con 0x7fe36c15ec80 2023-10-16T12:26:05.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.403+0000 7fe370815700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe36c0d2110 0x7fe36c0cd1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:05.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.403+0000 7fe370815700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe36c0d2110 0x7fe36c0cd1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38566/0 (socket says 172.21.15.16:38566) 2023-10-16T12:26:05.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.403+0000 7fe370815700 1 -- 172.21.15.16:0/616613749 learned_addr learned my addr 172.21.15.16:0/616613749 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:05.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.403+0000 7fe36bfff700 1 --2- 172.21.15.16:0/616613749 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe36c15ec80 0x7fe36c0cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:05.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.403+0000 7fe370815700 1 -- 172.21.15.16:0/616613749 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe36c15ec80 msgr2=0x7fe36c0cdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:05.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.403+0000 7fe370815700 1 --2- 172.21.15.16:0/616613749 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe36c15ec80 0x7fe36c0cdec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:05.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.403+0000 7fe370815700 1 -- 172.21.15.16:0/616613749 >> v1:172.21.15.16:6789/0 conn(0x7fe36c15f060 legacy=0x7fe36c0b4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:05.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.403+0000 7fe370815700 1 -- 172.21.15.16:0/616613749 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe36c0cef60 con 0x7fe36c0d2110 2023-10-16T12:26:05.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.403+0000 7fe370815700 1 --2- 172.21.15.16:0/616613749 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe36c0d2110 0x7fe36c0cd1a0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fe364009e50 tx=0x7fe36400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fb4f446434f120e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:05.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.404+0000 7fe36affd700 1 -- 172.21.15.16:0/616613749 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe36400e070 con 0x7fe36c0d2110 2023-10-16T12:26:05.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.404+0000 7fe36affd700 1 -- 172.21.15.16:0/616613749 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe36400b910 con 0x7fe36c0d2110 2023-10-16T12:26:05.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.404+0000 7fe36affd700 1 -- 172.21.15.16:0/616613749 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe36400bcc0 con 0x7fe36c0d2110 2023-10-16T12:26:05.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.404+0000 7fe372278700 1 -- 172.21.15.16:0/616613749 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe36c0d2110 msgr2=0x7fe36c0cd1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:05.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.404+0000 7fe372278700 1 --2- 172.21.15.16:0/616613749 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe36c0d2110 0x7fe36c0cd1a0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fe364009e50 tx=0x7fe36400b330 comp rx=0 tx=0).stop 2023-10-16T12:26:05.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.404+0000 7fe372278700 1 -- 172.21.15.16:0/616613749 shutdown_connections 2023-10-16T12:26:05.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.404+0000 7fe372278700 1 --2- 172.21.15.16:0/616613749 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe36c0d2110 0x7fe36c0cd1a0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:05.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.404+0000 7fe372278700 1 --2- 172.21.15.16:0/616613749 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe36c15ec80 0x7fe36c0cdec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:05.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.404+0000 7fe372278700 1 -- 172.21.15.16:0/616613749 >> 172.21.15.16:0/616613749 conn(0x7fe36c0d15f0 msgr2=0x7fe36c0d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:05.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.404+0000 7fe372278700 1 -- 172.21.15.16:0/616613749 shutdown_connections 2023-10-16T12:26:05.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.404+0000 7fe372278700 1 -- 172.21.15.16:0/616613749 wait complete. 2023-10-16T12:26:05.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.405+0000 7fe372278700 1 Processor -- start 2023-10-16T12:26:05.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.406+0000 7fe372278700 1 -- start start 2023-10-16T12:26:05.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.406+0000 7fe372278700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe36c0d2110 0x7fe36c0c53a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:05.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.406+0000 7fe372278700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe36c15ec80 0x7fe36c14d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:05.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.406+0000 7fe372278700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe36c15f060 0x7fe36c15d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:05.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.406+0000 7fe372278700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe36c0c86c0 con 0x7fe36c15ec80 2023-10-16T12:26:05.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.406+0000 7fe372278700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe36c0c8910 con 0x7fe36c0d2110 2023-10-16T12:26:05.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.406+0000 7fe372278700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe36c0c8bf0 con 0x7fe36c15f060 2023-10-16T12:26:05.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.406+0000 7fe36b7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe36c15ec80 0x7fe36c14d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:05.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.407+0000 7fe36b7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe36c15ec80 0x7fe36c14d8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38568/0 (socket says 172.21.15.16:38568) 2023-10-16T12:26:05.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.407+0000 7fe36b7fe700 1 -- 172.21.15.16:0/4138156171 learned_addr learned my addr 172.21.15.16:0/4138156171 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:05.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.407+0000 7fe370815700 1 --2- 172.21.15.16:0/4138156171 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe36c15f060 0x7fe36c15d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:05.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.407+0000 7fe36bfff700 1 --2- 172.21.15.16:0/4138156171 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe36c0d2110 0x7fe36c0c53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:05.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.407+0000 7fe36b7fe700 1 -- 172.21.15.16:0/4138156171 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe36c15f060 msgr2=0x7fe36c15d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:05.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.407+0000 7fe36b7fe700 1 --2- 172.21.15.16:0/4138156171 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe36c15f060 0x7fe36c15d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:05.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.407+0000 7fe36b7fe700 1 -- 172.21.15.16:0/4138156171 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe36c0d2110 msgr2=0x7fe36c0c53a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:05.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.407+0000 7fe36b7fe700 1 --2- 172.21.15.16:0/4138156171 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe36c0d2110 0x7fe36c0c53a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:05.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.407+0000 7fe36b7fe700 1 -- 172.21.15.16:0/4138156171 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe364009b30 con 0x7fe36c15ec80 2023-10-16T12:26:05.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.407+0000 7fe36b7fe700 1 --2- 172.21.15.16:0/4138156171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe36c15ec80 0x7fe36c14d8b0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fe36000c420 tx=0x7fe36000c730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:05.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.407+0000 7fe368ff9700 1 -- 172.21.15.16:0/4138156171 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3600052d0 con 0x7fe36c15ec80 2023-10-16T12:26:05.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.407+0000 7fe368ff9700 1 -- 172.21.15.16:0/4138156171 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe360005430 con 0x7fe36c15ec80 2023-10-16T12:26:05.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.407+0000 7fe372278700 1 -- 172.21.15.16:0/4138156171 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe36c0c8ed0 con 0x7fe36c15ec80 2023-10-16T12:26:05.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.407+0000 7fe368ff9700 1 -- 172.21.15.16:0/4138156171 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3600038c0 con 0x7fe36c15ec80 2023-10-16T12:26:05.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.407+0000 7fe372278700 1 -- 172.21.15.16:0/4138156171 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe36c14ddf0 con 0x7fe36c15ec80 2023-10-16T12:26:05.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.409+0000 7fe372278700 1 -- 172.21.15.16:0/4138156171 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe36c0c93f0 con 0x7fe36c15ec80 2023-10-16T12:26:05.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.409+0000 7fe368ff9700 1 -- 172.21.15.16:0/4138156171 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 27) v1 ==== 89343+0+0 (secure 0 0 0) 0x7fe360003a20 con 0x7fe36c15ec80 2023-10-16T12:26:05.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.410+0000 7fe368ff9700 1 --2- 172.21.15.16:0/4138156171 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7fe35806b020 0x7fe35806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:05.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.410+0000 7fe36bfff700 1 -- 172.21.15.16:0/4138156171 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7fe35806b020 msgr2=0x7fe35806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/4075855741 2023-10-16T12:26:05.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.410+0000 7fe368ff9700 1 -- 172.21.15.16:0/4138156171 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6503+0+0 (secure 0 0 0) 0x7fe360088d50 con 0x7fe36c15ec80 2023-10-16T12:26:05.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.410+0000 7fe36bfff700 1 --2- 172.21.15.16:0/4138156171 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7fe35806b020 0x7fe35806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:26:05.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.414+0000 7fe368ff9700 1 -- 172.21.15.16:0/4138156171 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe36c0c93f0 con 0x7fe36c15ec80 2023-10-16T12:26:05.434 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:26:05.433+0000 7fab230f8200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T12:26:05.470 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:05.468+0000 7fd5cc790200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T12:26:05.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.611+0000 7fe36bfff700 1 -- 172.21.15.16:0/4138156171 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7fe35806b020 msgr2=0x7fe35806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/4075855741 2023-10-16T12:26:05.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.611+0000 7fe36bfff700 1 --2- 172.21.15.16:0/4138156171 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7fe35806b020 0x7fe35806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:26:05.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.618+0000 7fe372278700 1 -- 172.21.15.16:0/4138156171 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe36c0c93f0 con 0x7fe36c15ec80 2023-10-16T12:26:05.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.618+0000 7fe368ff9700 1 -- 172.21.15.16:0/4138156171 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v73) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe360017020 con 0x7fe36c15ec80 2023-10-16T12:26:05.620 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-10-16T12:26:05.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.620+0000 7fe372278700 1 -- 172.21.15.16:0/4138156171 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7fe35806b020 msgr2=0x7fe35806d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:26:05.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.620+0000 7fe372278700 1 --2- 172.21.15.16:0/4138156171 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7fe35806b020 0x7fe35806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:05.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.620+0000 7fe372278700 1 -- 172.21.15.16:0/4138156171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe36c15ec80 msgr2=0x7fe36c14d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:05.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.620+0000 7fe372278700 1 --2- 172.21.15.16:0/4138156171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe36c15ec80 0x7fe36c14d8b0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fe36000c420 tx=0x7fe36000c730 comp rx=0 tx=0).stop 2023-10-16T12:26:05.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.620+0000 7fe372278700 1 -- 172.21.15.16:0/4138156171 shutdown_connections 2023-10-16T12:26:05.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.620+0000 7fe372278700 1 --2- 172.21.15.16:0/4138156171 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7fe35806b020 0x7fe35806d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:05.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.620+0000 7fe372278700 1 --2- 172.21.15.16:0/4138156171 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe36c15f060 0x7fe36c15d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:05.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.620+0000 7fe372278700 1 --2- 172.21.15.16:0/4138156171 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe36c0d2110 0x7fe36c0c53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:05.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.620+0000 7fe372278700 1 --2- 172.21.15.16:0/4138156171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe36c15ec80 0x7fe36c14d8b0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:05.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.620+0000 7fe372278700 1 -- 172.21.15.16:0/4138156171 >> 172.21.15.16:0/4138156171 conn(0x7fe36c0d15f0 msgr2=0x7fe36c0b4170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:05.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.620+0000 7fe372278700 1 -- 172.21.15.16:0/4138156171 shutdown_connections 2023-10-16T12:26:05.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.620+0000 7fe372278700 1 -- 172.21.15.16:0/4138156171 wait complete. 2023-10-16T12:26:05.635 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T12:26:05.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.802+0000 7f7b5cf7a700 1 Processor -- start 2023-10-16T12:26:05.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.802+0000 7f7b5cf7a700 1 -- start start 2023-10-16T12:26:05.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.803+0000 7f7b5cf7a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7b5815da10 0x7f7b580d1520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:05.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.803+0000 7f7b5cf7a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7b580d1d20 0x7f7b580d2130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:05.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.803+0000 7f7b5cf7a700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b580d2670 con 0x7f7b580d4060 2023-10-16T12:26:05.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.803+0000 7f7b5cf7a700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b580cf9b0 con 0x7f7b580d1d20 2023-10-16T12:26:05.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.803+0000 7f7b5cf7a700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b580d4540 con 0x7f7b5815da10 2023-10-16T12:26:05.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.803+0000 7f7b56d9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7b580d1d20 0x7f7b580d2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:05.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.803+0000 7f7b56d9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7b580d1d20 0x7f7b580d2130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38606/0 (socket says 172.21.15.16:38606) 2023-10-16T12:26:05.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.803+0000 7f7b56d9d700 1 -- 172.21.15.16:0/2285600781 learned_addr learned my addr 172.21.15.16:0/2285600781 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:05.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.803+0000 7f7b5659c700 1 --2- 172.21.15.16:0/2285600781 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7b5815da10 0x7f7b580d1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:05.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.804+0000 7f7b56d9d700 1 -- 172.21.15.16:0/2285600781 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7b5815da10 msgr2=0x7f7b580d1520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:05.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.804+0000 7f7b56d9d700 1 --2- 172.21.15.16:0/2285600781 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7b5815da10 0x7f7b580d1520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:05.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.804+0000 7f7b56d9d700 1 -- 172.21.15.16:0/2285600781 >> v1:172.21.15.16:6789/0 conn(0x7f7b580d4060 legacy=0x7f7b580cf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:05.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.804+0000 7f7b56d9d700 1 -- 172.21.15.16:0/2285600781 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b580ce7e0 con 0x7f7b580d1d20 2023-10-16T12:26:05.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.804+0000 7f7b56d9d700 1 --2- 172.21.15.16:0/2285600781 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b580d1d20 0x7f7b580d2130 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f7b50009e50 tx=0x7f7b5000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=820db2ea9a86083b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:05.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.804+0000 7f7b5559a700 1 -- 172.21.15.16:0/2285600781 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b5000e070 con 0x7f7b580d1d20 2023-10-16T12:26:05.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.804+0000 7f7b5559a700 1 -- 172.21.15.16:0/2285600781 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b5000b910 con 0x7f7b580d1d20 2023-10-16T12:26:05.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.804+0000 7f7b5559a700 1 -- 172.21.15.16:0/2285600781 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b5000bcc0 con 0x7f7b580d1d20 2023-10-16T12:26:05.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.805+0000 7f7b5cf7a700 1 -- 172.21.15.16:0/2285600781 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b580d1d20 msgr2=0x7f7b580d2130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:05.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.805+0000 7f7b5cf7a700 1 --2- 172.21.15.16:0/2285600781 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b580d1d20 0x7f7b580d2130 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f7b50009e50 tx=0x7f7b5000b330 comp rx=0 tx=0).stop 2023-10-16T12:26:05.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.805+0000 7f7b5cf7a700 1 -- 172.21.15.16:0/2285600781 shutdown_connections 2023-10-16T12:26:05.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.805+0000 7f7b5cf7a700 1 --2- 172.21.15.16:0/2285600781 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b580d1d20 0x7f7b580d2130 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:05.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.805+0000 7f7b5cf7a700 1 --2- 172.21.15.16:0/2285600781 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7b5815da10 0x7f7b580d1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:05.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.805+0000 7f7b5cf7a700 1 -- 172.21.15.16:0/2285600781 >> 172.21.15.16:0/2285600781 conn(0x7f7b5809c9f0 msgr2=0x7f7b580d3740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:05.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.805+0000 7f7b5cf7a700 1 -- 172.21.15.16:0/2285600781 shutdown_connections 2023-10-16T12:26:05.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.805+0000 7f7b5cf7a700 1 -- 172.21.15.16:0/2285600781 wait complete. 2023-10-16T12:26:05.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.806+0000 7f7b5cf7a700 1 Processor -- start 2023-10-16T12:26:05.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.806+0000 7f7b5cf7a700 1 -- start start 2023-10-16T12:26:05.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.807+0000 7f7b5cf7a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b580d1d20 0x7f7b5815b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:05.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.807+0000 7f7b5cf7a700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7b580d4060 0x7f7b580cb170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:05.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.807+0000 7f7b5cf7a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7b5815da10 0x7f7b580cb6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:05.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.807+0000 7f7b5cf7a700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b581f1310 con 0x7f7b580d1d20 2023-10-16T12:26:05.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.807+0000 7f7b5cf7a700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b581f3e30 con 0x7f7b580d4060 2023-10-16T12:26:05.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.807+0000 7f7b5cf7a700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b581f3b50 con 0x7f7b5815da10 2023-10-16T12:26:05.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.807+0000 7f7b5659c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b580d1d20 0x7f7b5815b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:05.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.807+0000 7f7b5659c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b580d1d20 0x7f7b5815b070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38608/0 (socket says 172.21.15.16:38608) 2023-10-16T12:26:05.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.807+0000 7f7b5659c700 1 -- 172.21.15.16:0/6641418 learned_addr learned my addr 172.21.15.16:0/6641418 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:05.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.807+0000 7f7b56d9d700 1 --2- 172.21.15.16:0/6641418 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7b5815da10 0x7f7b580cb6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:05.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.807+0000 7f7b55d9b700 1 --2- 172.21.15.16:0/6641418 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7b580d4060 0x7f7b580cb170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:05.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.807+0000 7f7b5659c700 1 -- 172.21.15.16:0/6641418 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7b5815da10 msgr2=0x7f7b580cb6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:05.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.807+0000 7f7b5659c700 1 --2- 172.21.15.16:0/6641418 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7b5815da10 0x7f7b580cb6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:05.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.807+0000 7f7b5659c700 1 -- 172.21.15.16:0/6641418 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7b580d4060 msgr2=0x7f7b580cb170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:05.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.807+0000 7f7b5659c700 1 --2- 172.21.15.16:0/6641418 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7b580d4060 0x7f7b580cb170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:05.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.807+0000 7f7b5659c700 1 -- 172.21.15.16:0/6641418 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b50009b30 con 0x7f7b580d1d20 2023-10-16T12:26:05.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.808+0000 7f7b5659c700 1 --2- 172.21.15.16:0/6641418 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b580d1d20 0x7f7b5815b070 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f7b4800e900 tx=0x7f7b4800ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:05.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.808+0000 7f7b46ffd700 1 -- 172.21.15.16:0/6641418 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b48013680 con 0x7f7b580d1d20 2023-10-16T12:26:05.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.808+0000 7f7b46ffd700 1 -- 172.21.15.16:0/6641418 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b480137e0 con 0x7f7b580d1d20 2023-10-16T12:26:05.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.808+0000 7f7b5cf7a700 1 -- 172.21.15.16:0/6641418 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b581f0c50 con 0x7f7b580d1d20 2023-10-16T12:26:05.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.808+0000 7f7b46ffd700 1 -- 172.21.15.16:0/6641418 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b48013b50 con 0x7f7b580d1d20 2023-10-16T12:26:05.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.808+0000 7f7b5cf7a700 1 -- 172.21.15.16:0/6641418 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b5815b5b0 con 0x7f7b580d1d20 2023-10-16T12:26:05.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.809+0000 7f7b5cf7a700 1 -- 172.21.15.16:0/6641418 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b5804ebb0 con 0x7f7b580d1d20 2023-10-16T12:26:05.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.809+0000 7f7b46ffd700 1 -- 172.21.15.16:0/6641418 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 27) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f7b48013cb0 con 0x7f7b580d1d20 2023-10-16T12:26:05.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.810+0000 7f7b46ffd700 1 --2- 172.21.15.16:0/6641418 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f7b3806b020 0x7f7b3806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:05.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.810+0000 7f7b55d9b700 1 -- 172.21.15.16:0/6641418 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f7b3806b020 msgr2=0x7f7b3806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/4075855741 2023-10-16T12:26:05.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.810+0000 7f7b46ffd700 1 -- 172.21.15.16:0/6641418 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6503+0+0 (secure 0 0 0) 0x7f7b48089ce0 con 0x7f7b580d1d20 2023-10-16T12:26:05.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.810+0000 7f7b55d9b700 1 --2- 172.21.15.16:0/6641418 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f7b3806b020 0x7f7b3806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:26:05.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:05.814+0000 7f7b46ffd700 1 -- 172.21.15.16:0/6641418 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7b48054970 con 0x7f7b580d1d20 2023-10-16T12:26:05.827 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:26:05.826+0000 7fab230f8200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T12:26:05.871 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:05.869+0000 7fd5cc790200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T12:26:06.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.011+0000 7f7b55d9b700 1 -- 172.21.15.16:0/6641418 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f7b3806b020 msgr2=0x7f7b3806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/4075855741 2023-10-16T12:26:06.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.011+0000 7f7b55d9b700 1 --2- 172.21.15.16:0/6641418 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f7b3806b020 0x7f7b3806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:26:06.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.020+0000 7f7b5cf7a700 1 -- 172.21.15.16:0/6641418 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7b580d1b00 con 0x7f7b580d1d20 2023-10-16T12:26:06.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.021+0000 7f7b46ffd700 1 -- 172.21.15.16:0/6641418 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v73) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7b480170a0 con 0x7f7b580d1d20 2023-10-16T12:26:06.023 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-10-16T12:26:06.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.023+0000 7f7b5cf7a700 1 -- 172.21.15.16:0/6641418 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f7b3806b020 msgr2=0x7f7b3806d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:26:06.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.023+0000 7f7b5cf7a700 1 --2- 172.21.15.16:0/6641418 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f7b3806b020 0x7f7b3806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:06.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.023+0000 7f7b5cf7a700 1 -- 172.21.15.16:0/6641418 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b580d1d20 msgr2=0x7f7b5815b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:06.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.023+0000 7f7b5cf7a700 1 --2- 172.21.15.16:0/6641418 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b580d1d20 0x7f7b5815b070 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f7b4800e900 tx=0x7f7b4800ec10 comp rx=0 tx=0).stop 2023-10-16T12:26:06.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.023+0000 7f7b5cf7a700 1 -- 172.21.15.16:0/6641418 shutdown_connections 2023-10-16T12:26:06.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.023+0000 7f7b5cf7a700 1 --2- 172.21.15.16:0/6641418 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f7b3806b020 0x7f7b3806d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:06.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.023+0000 7f7b5cf7a700 1 --2- 172.21.15.16:0/6641418 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7b5815da10 0x7f7b580cb6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:06.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.023+0000 7f7b5cf7a700 1 --2- 172.21.15.16:0/6641418 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b580d1d20 0x7f7b5815b070 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:06.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.023+0000 7f7b5cf7a700 1 --2- 172.21.15.16:0/6641418 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7b580d4060 0x7f7b580cb170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:06.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.023+0000 7f7b5cf7a700 1 -- 172.21.15.16:0/6641418 >> 172.21.15.16:0/6641418 conn(0x7f7b5809c9f0 msgr2=0x7f7b5815e270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:06.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.023+0000 7f7b5cf7a700 1 -- 172.21.15.16:0/6641418 shutdown_connections 2023-10-16T12:26:06.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.023+0000 7f7b5cf7a700 1 -- 172.21.15.16:0/6641418 wait complete. 2023-10-16T12:26:06.037 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T12:26:06.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.163+0000 7f587d2ac700 1 Processor -- start 2023-10-16T12:26:06.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.163+0000 7f587d2ac700 1 -- start start 2023-10-16T12:26:06.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.163+0000 7f587d2ac700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f587815a5c0 0x7f58780cef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:06.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.163+0000 7f587d2ac700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5878072a50 con 0x7f58780ce010 2023-10-16T12:26:06.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.163+0000 7f587d2ac700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58780cba30 con 0x7f58780d0ae0 2023-10-16T12:26:06.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.163+0000 7f587d2ac700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58780cf440 con 0x7f587815a5c0 2023-10-16T12:26:06.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.164+0000 7f58767fc700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f58780ce010 0x7f58780d03d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:56124/0 (socket says 172.21.15.16:56124) 2023-10-16T12:26:06.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.164+0000 7f58767fc700 1 -- 172.21.15.16:0/1456079545 learned_addr learned my addr 172.21.15.16:0/1456079545 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:26:06.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.164+0000 7f5876ffd700 1 --2- 172.21.15.16:0/1456079545 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f587815a5c0 0x7f58780cef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:06.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.164+0000 7f5875ffb700 1 -- 172.21.15.16:0/1456079545 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 534721347 0 0) 0x7f5878072a50 con 0x7f58780ce010 2023-10-16T12:26:06.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.164+0000 7f5875ffb700 1 -- 172.21.15.16:0/1456079545 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58600035b0 con 0x7f58780ce010 2023-10-16T12:26:06.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.164+0000 7f5875ffb700 1 -- 172.21.15.16:0/1456079545 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3161438781 0 0) 0x7f58780cba30 con 0x7f58780d0ae0 2023-10-16T12:26:06.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.164+0000 7f5875ffb700 1 -- 172.21.15.16:0/1456079545 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5878072a50 con 0x7f58780d0ae0 2023-10-16T12:26:06.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.164+0000 7f5876ffd700 1 -- 172.21.15.16:0/1456079545 >> v1:172.21.15.49:6789/0 conn(0x7f58780d0ae0 legacy=0x7f58780cc1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:06.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.164+0000 7f5876ffd700 1 -- 172.21.15.16:0/1456079545 >> v1:172.21.15.16:6789/0 conn(0x7f58780ce010 legacy=0x7f58780d03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:06.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.164+0000 7f5876ffd700 1 -- 172.21.15.16:0/1456079545 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58780cf630 con 0x7f587815a5c0 2023-10-16T12:26:06.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.164+0000 7f5876ffd700 1 --2- 172.21.15.16:0/1456079545 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f587815a5c0 0x7f58780cef00 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5868009870 tx=0x7f5868009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=780e082fe240b144 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:06.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.164+0000 7f5875ffb700 1 -- 172.21.15.16:0/1456079545 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5868004230 con 0x7f587815a5c0 2023-10-16T12:26:06.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.165+0000 7f5875ffb700 1 -- 172.21.15.16:0/1456079545 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5868004390 con 0x7f587815a5c0 2023-10-16T12:26:06.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.165+0000 7f5875ffb700 1 -- 172.21.15.16:0/1456079545 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58680036a0 con 0x7f587815a5c0 2023-10-16T12:26:06.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.165+0000 7f587d2ac700 1 -- 172.21.15.16:0/1456079545 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f587815a5c0 msgr2=0x7f58780cef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:06.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.165+0000 7f587d2ac700 1 --2- 172.21.15.16:0/1456079545 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f587815a5c0 0x7f58780cef00 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5868009870 tx=0x7f5868009b80 comp rx=0 tx=0).stop 2023-10-16T12:26:06.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.165+0000 7f587d2ac700 1 -- 172.21.15.16:0/1456079545 shutdown_connections 2023-10-16T12:26:06.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.165+0000 7f587d2ac700 1 --2- 172.21.15.16:0/1456079545 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f587815a5c0 0x7f58780cef00 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:06.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.165+0000 7f587d2ac700 1 -- 172.21.15.16:0/1456079545 >> 172.21.15.16:0/1456079545 conn(0x7f58780b1fd0 msgr2=0x7f58780b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:06.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.165+0000 7f587d2ac700 1 -- 172.21.15.16:0/1456079545 shutdown_connections 2023-10-16T12:26:06.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.165+0000 7f587d2ac700 1 -- 172.21.15.16:0/1456079545 wait complete. 2023-10-16T12:26:06.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.167+0000 7f587d2ac700 1 Processor -- start 2023-10-16T12:26:06.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.167+0000 7f587d2ac700 1 -- start start 2023-10-16T12:26:06.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.167+0000 7f587d2ac700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58780ce010 0x7f5878153530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:06.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.167+0000 7f587d2ac700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f58780d0ae0 0x7f5878153b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:06.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.167+0000 7f587d2ac700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f587815a5c0 0x7f58781595d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:06.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.167+0000 7f587d2ac700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5878154eb0 con 0x7f58780ce010 2023-10-16T12:26:06.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.167+0000 7f587d2ac700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5878154610 con 0x7f587815a5c0 2023-10-16T12:26:06.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.167+0000 7f587d2ac700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58781548f0 con 0x7f58780d0ae0 2023-10-16T12:26:06.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.167+0000 7f5876ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58780ce010 0x7f5878153530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:06.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.167+0000 7f5876ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58780ce010 0x7f5878153530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38624/0 (socket says 172.21.15.16:38624) 2023-10-16T12:26:06.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.168+0000 7f5876ffd700 1 -- 172.21.15.16:0/3022030367 learned_addr learned my addr 172.21.15.16:0/3022030367 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:06.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.168+0000 7f58767fc700 1 --2- 172.21.15.16:0/3022030367 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f58780d0ae0 0x7f5878153b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:06.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.168+0000 7f58777fe700 1 --2- 172.21.15.16:0/3022030367 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f587815a5c0 0x7f58781595d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:06.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.168+0000 7f58767fc700 1 -- 172.21.15.16:0/3022030367 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f587815a5c0 msgr2=0x7f58781595d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:06.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.168+0000 7f58767fc700 1 --2- 172.21.15.16:0/3022030367 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f587815a5c0 0x7f58781595d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:06.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.168+0000 7f58767fc700 1 -- 172.21.15.16:0/3022030367 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58780ce010 msgr2=0x7f5878153530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:06.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.168+0000 7f58767fc700 1 --2- 172.21.15.16:0/3022030367 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58780ce010 0x7f5878153530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:06.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.168+0000 7f58767fc700 1 -- 172.21.15.16:0/3022030367 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58680094a0 con 0x7f58780d0ae0 2023-10-16T12:26:06.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.168+0000 7f5876ffd700 1 --2- 172.21.15.16:0/3022030367 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58780ce010 0x7f5878153530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:26:06.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.168+0000 7f58767fc700 1 --2- 172.21.15.16:0/3022030367 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f58780d0ae0 0x7f5878153b50 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f586c00c490 tx=0x7f586c00c7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:06.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.168+0000 7f585f7fe700 1 -- 172.21.15.16:0/3022030367 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f586c005d70 con 0x7f58780d0ae0 2023-10-16T12:26:06.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.168+0000 7f585f7fe700 1 -- 172.21.15.16:0/3022030367 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f586c0070d0 con 0x7f58780d0ae0 2023-10-16T12:26:06.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.168+0000 7f587d2ac700 1 -- 172.21.15.16:0/3022030367 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5878154bd0 con 0x7f58780d0ae0 2023-10-16T12:26:06.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.168+0000 7f585f7fe700 1 -- 172.21.15.16:0/3022030367 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f586c007440 con 0x7f58780d0ae0 2023-10-16T12:26:06.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.168+0000 7f587d2ac700 1 -- 172.21.15.16:0/3022030367 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58780c80c0 con 0x7f58780d0ae0 2023-10-16T12:26:06.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.170+0000 7f587d2ac700 1 -- 172.21.15.16:0/3022030367 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f587815c3f0 con 0x7f58780d0ae0 2023-10-16T12:26:06.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.170+0000 7f585f7fe700 1 -- 172.21.15.16:0/3022030367 <== mon.2 v2:172.21.15.49:3301/0 4 ==== mgrmap(e 27) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f586c0075a0 con 0x7f58780d0ae0 2023-10-16T12:26:06.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.170+0000 7f585f7fe700 1 --2- 172.21.15.16:0/3022030367 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f586406b020 0x7f586406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:06.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.170+0000 7f5876ffd700 1 -- 172.21.15.16:0/3022030367 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f586406b020 msgr2=0x7f586406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/4075855741 2023-10-16T12:26:06.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.170+0000 7f585f7fe700 1 -- 172.21.15.16:0/3022030367 <== mon.2 v2:172.21.15.49:3301/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6503+0+0 (secure 0 0 0) 0x7f586c088d70 con 0x7f58780d0ae0 2023-10-16T12:26:06.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.170+0000 7f5876ffd700 1 --2- 172.21.15.16:0/3022030367 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f586406b020 0x7f586406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:26:06.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.175+0000 7f585f7fe700 1 -- 172.21.15.16:0/3022030367 <== mon.2 v2:172.21.15.49:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f587815c3f0 con 0x7f58780d0ae0 2023-10-16T12:26:06.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.371+0000 7f5876ffd700 1 -- 172.21.15.16:0/3022030367 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f586406b020 msgr2=0x7f586406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/4075855741 2023-10-16T12:26:06.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.371+0000 7f5876ffd700 1 --2- 172.21.15.16:0/3022030367 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f586406b020 0x7f586406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:26:06.433 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:26:06.432+0000 7fab230f8200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T12:26:06.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.441+0000 7f587d2ac700 1 -- 172.21.15.16:0/3022030367 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f587815c3f0 con 0x7f58780d0ae0 2023-10-16T12:26:06.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.441+0000 7f585f7fe700 1 -- 172.21.15.16:0/3022030367 <== mon.2 v2:172.21.15.49:3301/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v73) v1 ==== 80+0+468 (secure 0 0 0) 0x7f587815c3f0 con 0x7f58780d0ae0 2023-10-16T12:26:06.444 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2103819518 2023-10-17T12:24:17.735687+0000 2023-10-16T12:26:06.444 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/224613758 2023-10-17T12:24:17.735687+0000 2023-10-16T12:26:06.444 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/877922677 2023-10-17T12:24:17.735687+0000 2023-10-16T12:26:06.444 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/1135331895 2023-10-17T12:24:17.735687+0000 2023-10-16T12:26:06.444 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/1214553139 2023-10-17T12:24:17.735687+0000 2023-10-16T12:26:06.445 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/1240820252 2023-10-17T12:24:17.735687+0000 2023-10-16T12:26:06.445 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6833/3870937344 2023-10-17T12:24:17.735687+0000 2023-10-16T12:26:06.445 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6832/3870937344 2023-10-17T12:24:17.735687+0000 2023-10-16T12:26:06.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.445+0000 7f587d2ac700 1 -- 172.21.15.16:0/3022030367 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f586406b020 msgr2=0x7f586406d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:26:06.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.445+0000 7f587d2ac700 1 --2- 172.21.15.16:0/3022030367 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f586406b020 0x7f586406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:06.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.445+0000 7f587d2ac700 1 -- 172.21.15.16:0/3022030367 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f58780d0ae0 msgr2=0x7f5878153b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:06.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.445+0000 7f587d2ac700 1 --2- 172.21.15.16:0/3022030367 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f58780d0ae0 0x7f5878153b50 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f586c00c490 tx=0x7f586c00c7a0 comp rx=0 tx=0).stop 2023-10-16T12:26:06.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.445+0000 7f587d2ac700 1 -- 172.21.15.16:0/3022030367 shutdown_connections 2023-10-16T12:26:06.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.445+0000 7f587d2ac700 1 --2- 172.21.15.16:0/3022030367 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f586406b020 0x7f586406d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:06.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.445+0000 7f587d2ac700 1 --2- 172.21.15.16:0/3022030367 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f587815a5c0 0x7f58781595d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:06.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.445+0000 7f587d2ac700 1 --2- 172.21.15.16:0/3022030367 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58780ce010 0x7f5878153530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:06.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.445+0000 7f587d2ac700 1 --2- 172.21.15.16:0/3022030367 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f58780d0ae0 0x7f5878153b50 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:06.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.445+0000 7f587d2ac700 1 -- 172.21.15.16:0/3022030367 >> 172.21.15.16:0/3022030367 conn(0x7f58780b1fd0 msgr2=0x7f58780ce4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:06.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.445+0000 7f587d2ac700 1 -- 172.21.15.16:0/3022030367 shutdown_connections 2023-10-16T12:26:06.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.445+0000 7f587d2ac700 1 -- 172.21.15.16:0/3022030367 wait complete. 2023-10-16T12:26:06.450 INFO:teuthology.orchestra.run.smithi016.stderr:listed 8 entries 2023-10-16T12:26:06.464 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-16T12:26:06.464 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:26:06.465 DEBUG:teuthology.orchestra.run.smithi016:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-16T12:26:06.494 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:06.493+0000 7fd5cc790200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T12:26:06.505 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:26:06.504+0000 7fab230f8200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T12:26:06.516 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2023-10-16T12:26:06.516 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:26:06.517 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-16T12:26:06.571 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:06.570+0000 7fd5cc790200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T12:26:06.572 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T12:26:06.573 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:26:06.591 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284204k,nr_inodes=4071051,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=753 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260540k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T12:26:06.591 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:26:06.592 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:26:06.630 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16287364k,nr_inodes=4071841,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20251 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3261172k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T12:26:06.630 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T12:26:06.631 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:26:06.648 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284204k,nr_inodes=4071051,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=753 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260540k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T12:26:06.649 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:26:06.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.794+0000 7fcccfe38700 1 Processor -- start 2023-10-16T12:26:06.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.795+0000 7fcccfe38700 1 -- start start 2023-10-16T12:26:06.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.795+0000 7fcccfe38700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fccc80b2b50 0x7fccc80d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:06.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.795+0000 7fcccfe38700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fccc80ce380 0x7fccc80d09e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:06.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.795+0000 7fcccfe38700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccc80d1210 con 0x7fccc80cdfa0 2023-10-16T12:26:06.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.795+0000 7fcccfe38700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccc80cbc30 con 0x7fccc80ce380 2023-10-16T12:26:06.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.795+0000 7fcccfe38700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccc80cbf10 con 0x7fccc80b2b50 2023-10-16T12:26:06.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.796+0000 7fcccdbd4700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fccc80cdfa0 0x7fccc80b2440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:56140/0 (socket says 172.21.15.16:56140) 2023-10-16T12:26:06.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.796+0000 7fcccdbd4700 1 -- 172.21.15.16:0/2398933440 learned_addr learned my addr 172.21.15.16:0/2398933440 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:26:06.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.796+0000 7fccce3d5700 1 --2- 172.21.15.16:0/2398933440 >> v2:172.21.15.49:3300/0 conn(0x7fccc80ce380 0x7fccc80d09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:06.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.796+0000 7fcccd3d3700 1 --2- 172.21.15.16:0/2398933440 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fccc80b2b50 0x7fccc80d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:06.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.796+0000 7fccccbd2700 1 -- 172.21.15.16:0/2398933440 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 229361261 0 0) 0x7fccc80d1210 con 0x7fccc80cdfa0 2023-10-16T12:26:06.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.796+0000 7fccccbd2700 1 -- 172.21.15.16:0/2398933440 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccb40035b0 con 0x7fccc80cdfa0 2023-10-16T12:26:06.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.796+0000 7fccccbd2700 1 -- 172.21.15.16:0/2398933440 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3515533246 0 0) 0x7fccb40035b0 con 0x7fccc80cdfa0 2023-10-16T12:26:06.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.796+0000 7fccccbd2700 1 -- 172.21.15.16:0/2398933440 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fccc80b2b50 msgr2=0x7fccc80d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:06.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.796+0000 7fccccbd2700 1 --2- 172.21.15.16:0/2398933440 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fccc80b2b50 0x7fccc80d04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:06.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.796+0000 7fccccbd2700 1 -- 172.21.15.16:0/2398933440 >> v2:172.21.15.49:3300/0 conn(0x7fccc80ce380 msgr2=0x7fccc80d09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:06.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.796+0000 7fccccbd2700 1 --2- 172.21.15.16:0/2398933440 >> v2:172.21.15.49:3300/0 conn(0x7fccc80ce380 0x7fccc80d09e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:06.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.796+0000 7fccccbd2700 1 -- 172.21.15.16:0/2398933440 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccc80cc1f0 con 0x7fccc80cdfa0 2023-10-16T12:26:06.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.796+0000 7fccccbd2700 1 -- 172.21.15.16:0/2398933440 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7fccc4001670 con 0x7fccc80cdfa0 2023-10-16T12:26:06.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.796+0000 7fccccbd2700 1 -- 172.21.15.16:0/2398933440 >> v1:172.21.15.16:6789/0 conn(0x7fccc80cdfa0 legacy=0x7fccc80b2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:06.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.796+0000 7fccccbd2700 1 --2- 172.21.15.16:0/2398933440 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccb4003a00 0x7fccb4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:06.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.796+0000 7fccccbd2700 1 --2- 172.21.15.16:0/2398933440 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fccb4004850 0x7fccb4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:06.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.796+0000 7fccccbd2700 1 --2- 172.21.15.16:0/2398933440 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fccb4005210 0x7fccb400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:06.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.796+0000 7fccccbd2700 1 -- 172.21.15.16:0/2398933440 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccc80cbc30 con 0x7fccb4003a00 2023-10-16T12:26:06.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.796+0000 7fccccbd2700 1 -- 172.21.15.16:0/2398933440 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccc80cbf10 con 0x7fccb4004850 2023-10-16T12:26:06.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.796+0000 7fccccbd2700 1 -- 172.21.15.16:0/2398933440 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccb400bf80 con 0x7fccb4005210 2023-10-16T12:26:06.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.796+0000 7fccce3d5700 1 --2- 172.21.15.16:0/2398933440 >> v2:172.21.15.49:3300/0 conn(0x7fccc80ce380 0x7fccc80d09e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:26:06.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.796+0000 7fcccd3d3700 1 --2- 172.21.15.16:0/2398933440 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fccc80b2b50 0x7fccc80d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:26:06.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.796+0000 7fcccdbd4700 1 --2- 172.21.15.16:0/2398933440 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccb4003a00 0x7fccb4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:06.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.797+0000 7fccce3d5700 1 --2- 172.21.15.16:0/2398933440 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fccb4005210 0x7fccb400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:06.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.797+0000 7fcccd3d3700 1 --2- 172.21.15.16:0/2398933440 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fccb4004850 0x7fccb4004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:06.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.797+0000 7fcccdbd4700 1 -- 172.21.15.16:0/2398933440 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fccb4005210 msgr2=0x7fccb400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:06.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.797+0000 7fcccdbd4700 1 --2- 172.21.15.16:0/2398933440 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fccb4005210 0x7fccb400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:06.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.797+0000 7fcccdbd4700 1 -- 172.21.15.16:0/2398933440 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fccb4004850 msgr2=0x7fccb4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:06.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.797+0000 7fcccdbd4700 1 --2- 172.21.15.16:0/2398933440 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fccb4004850 0x7fccb4004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:06.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.797+0000 7fcccdbd4700 1 -- 172.21.15.16:0/2398933440 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccb40035b0 con 0x7fccb4003a00 2023-10-16T12:26:06.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.797+0000 7fcccd3d3700 1 --2- 172.21.15.16:0/2398933440 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fccb4004850 0x7fccb4004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:26:06.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.797+0000 7fcccdbd4700 1 --2- 172.21.15.16:0/2398933440 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccb4003a00 0x7fccb4003e20 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fccc400c320 tx=0x7fccb4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=536bde134fd3884e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:06.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.797+0000 7fccccbd2700 1 -- 172.21.15.16:0/2398933440 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc400ce00 con 0x7fccb4003a00 2023-10-16T12:26:06.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.797+0000 7fccccbd2700 1 -- 172.21.15.16:0/2398933440 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccc4005960 con 0x7fccb4003a00 2023-10-16T12:26:06.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.797+0000 7fccccbd2700 1 -- 172.21.15.16:0/2398933440 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc4005d10 con 0x7fccb4003a00 2023-10-16T12:26:06.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.798+0000 7fcccfe38700 1 -- 172.21.15.16:0/2398933440 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccb4003a00 msgr2=0x7fccb4003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:06.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.798+0000 7fcccfe38700 1 --2- 172.21.15.16:0/2398933440 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccb4003a00 0x7fccb4003e20 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fccc400c320 tx=0x7fccb4005880 comp rx=0 tx=0).stop 2023-10-16T12:26:06.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.798+0000 7fccce3d5700 1 -- 172.21.15.16:0/2398933440 reap_dead start 2023-10-16T12:26:06.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.798+0000 7fcccfe38700 1 -- 172.21.15.16:0/2398933440 shutdown_connections 2023-10-16T12:26:06.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.798+0000 7fcccfe38700 1 -- 172.21.15.16:0/2398933440 >> 172.21.15.16:0/2398933440 conn(0x7fccc80d14b0 msgr2=0x7fccc80d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:06.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.798+0000 7fcccfe38700 1 -- 172.21.15.16:0/2398933440 shutdown_connections 2023-10-16T12:26:06.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.798+0000 7fcccfe38700 1 -- 172.21.15.16:0/2398933440 wait complete. 2023-10-16T12:26:06.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.799+0000 7fcccfe38700 1 Processor -- start 2023-10-16T12:26:06.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.799+0000 7fcccfe38700 1 -- start start 2023-10-16T12:26:06.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.800+0000 7fcccfe38700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fccb4003a00 0x7fccc81f6b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:06.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.800+0000 7fcccfe38700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccc80caf60 0x7fccc80cfb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:06.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.800+0000 7fcccfe38700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fccc8151520 0x7fccc81519b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:06.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.800+0000 7fcccfe38700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccc8151ef0 con 0x7fccc80caf60 2023-10-16T12:26:06.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.800+0000 7fcccfe38700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccc80c8930 con 0x7fccc8151520 2023-10-16T12:26:06.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.800+0000 7fcccfe38700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccc80c8c10 con 0x7fccb4003a00 2023-10-16T12:26:06.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.800+0000 7fcccd3d3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccc80caf60 0x7fccc80cfb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:06.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.800+0000 7fcccd3d3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccc80caf60 0x7fccc80cfb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38648/0 (socket says 172.21.15.16:38648) 2023-10-16T12:26:06.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.800+0000 7fcccd3d3700 1 -- 172.21.15.16:0/2838989445 learned_addr learned my addr 172.21.15.16:0/2838989445 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:06.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.800+0000 7fccce3d5700 1 --2- 172.21.15.16:0/2838989445 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fccc8151520 0x7fccc81519b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:06.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.800+0000 7fcccdbd4700 1 --2- 172.21.15.16:0/2838989445 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fccb4003a00 0x7fccc81f6b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:06.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.800+0000 7fcccd3d3700 1 -- 172.21.15.16:0/2838989445 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fccb4003a00 msgr2=0x7fccc81f6b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:06.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.800+0000 7fcccd3d3700 1 --2- 172.21.15.16:0/2838989445 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fccb4003a00 0x7fccc81f6b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:06.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.800+0000 7fcccd3d3700 1 -- 172.21.15.16:0/2838989445 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fccc8151520 msgr2=0x7fccc81519b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:06.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.800+0000 7fcccd3d3700 1 --2- 172.21.15.16:0/2838989445 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fccc8151520 0x7fccc81519b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:06.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.800+0000 7fcccd3d3700 1 -- 172.21.15.16:0/2838989445 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccc80c8ef0 con 0x7fccc80caf60 2023-10-16T12:26:06.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.801+0000 7fcccd3d3700 1 --2- 172.21.15.16:0/2838989445 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccc80caf60 0x7fccc80cfb50 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fccb80109e0 tx=0x7fccb8010cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:06.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.801+0000 7fccbe7fc700 1 -- 172.21.15.16:0/2838989445 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb8004030 con 0x7fccc80caf60 2023-10-16T12:26:06.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.801+0000 7fccbe7fc700 1 -- 172.21.15.16:0/2838989445 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccb8004190 con 0x7fccc80caf60 2023-10-16T12:26:06.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.801+0000 7fccbe7fc700 1 -- 172.21.15.16:0/2838989445 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb8004540 con 0x7fccc80caf60 2023-10-16T12:26:06.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.801+0000 7fcccfe38700 1 -- 172.21.15.16:0/2838989445 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccc80c91d0 con 0x7fccc80caf60 2023-10-16T12:26:06.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.801+0000 7fcccfe38700 1 -- 172.21.15.16:0/2838989445 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccc80d0090 con 0x7fccc80caf60 2023-10-16T12:26:06.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.802+0000 7fcccfe38700 1 -- 172.21.15.16:0/2838989445 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccc804ebb0 con 0x7fccc80caf60 2023-10-16T12:26:06.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.802+0000 7fccbe7fc700 1 -- 172.21.15.16:0/2838989445 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 27) v1 ==== 89343+0+0 (secure 0 0 0) 0x7fccb80038b0 con 0x7fccc80caf60 2023-10-16T12:26:06.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.803+0000 7fccbe7fc700 1 --2- 172.21.15.16:0/2838989445 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7fccac06b020 0x7fccac06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:06.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.803+0000 7fcccdbd4700 1 -- 172.21.15.16:0/2838989445 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7fccac06b020 msgr2=0x7fccac06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/4075855741 2023-10-16T12:26:06.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.803+0000 7fccbe7fc700 1 -- 172.21.15.16:0/2838989445 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6503+0+0 (secure 0 0 0) 0x7fccb808a230 con 0x7fccc80caf60 2023-10-16T12:26:06.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.803+0000 7fcccdbd4700 1 --2- 172.21.15.16:0/2838989445 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7fccac06b020 0x7fccac06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:26:06.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:06.808+0000 7fccbe7fc700 1 -- 172.21.15.16:0/2838989445 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fccb8054eb0 con 0x7fccc80caf60 2023-10-16T12:26:07.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.004+0000 7fcccdbd4700 1 -- 172.21.15.16:0/2838989445 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7fccac06b020 msgr2=0x7fccac06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/4075855741 2023-10-16T12:26:07.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.004+0000 7fcccdbd4700 1 --2- 172.21.15.16:0/2838989445 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7fccac06b020 0x7fccac06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:26:07.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.058+0000 7fcccfe38700 1 -- 172.21.15.16:0/2838989445 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fccc806d190 con 0x7fccc80caf60 2023-10-16T12:26:07.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.058+0000 7fccbe7fc700 1 -- 172.21.15.16:0/2838989445 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+4913 (secure 0 0 0) 0x7fccb80584d0 con 0x7fccc80caf60 2023-10-16T12:26:07.060 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:26:07.060 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5947,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/1103417847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":1103417847},{"type":"v1","addr":"172.21.15.49:6835","nonce":1103417847}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":6096,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/1380792179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1380792179},{"type":"v1","addr":"172.21.15.16:6836","nonce":1380792179}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":6099,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2238585443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":2238585443},{"type":"v1","addr":"172.21.15.16:6837","nonce":2238585443}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:24:02.695223+0000","modified":"2023-10-16T12:24:05.704039+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6111},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6111":{"gid":6111,"name":"b","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.49:6834/3246624425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":3246624425},{"type":"v1","addr":"172.21.15.49:6834","nonce":3246624425}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-10-16T12:26:07.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.060+0000 7fcccfe38700 1 -- 172.21.15.16:0/2838989445 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7fccac06b020 msgr2=0x7fccac06d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:26:07.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.060+0000 7fcccfe38700 1 --2- 172.21.15.16:0/2838989445 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7fccac06b020 0x7fccac06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:07.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.060+0000 7fcccfe38700 1 -- 172.21.15.16:0/2838989445 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccc80caf60 msgr2=0x7fccc80cfb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:07.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.060+0000 7fcccfe38700 1 --2- 172.21.15.16:0/2838989445 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccc80caf60 0x7fccc80cfb50 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fccb80109e0 tx=0x7fccb8010cf0 comp rx=0 tx=0).stop 2023-10-16T12:26:07.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.060+0000 7fcccfe38700 1 -- 172.21.15.16:0/2838989445 shutdown_connections 2023-10-16T12:26:07.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.060+0000 7fcccfe38700 1 --2- 172.21.15.16:0/2838989445 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7fccac06b020 0x7fccac06d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:07.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.060+0000 7fcccfe38700 1 --2- 172.21.15.16:0/2838989445 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fccc8151520 0x7fccc81519b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:07.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.060+0000 7fcccfe38700 1 --2- 172.21.15.16:0/2838989445 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fccb4003a00 0x7fccc81f6b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:07.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.060+0000 7fcccfe38700 1 --2- 172.21.15.16:0/2838989445 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccc80caf60 0x7fccc80cfb50 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:07.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.060+0000 7fcccfe38700 1 -- 172.21.15.16:0/2838989445 >> 172.21.15.16:0/2838989445 conn(0x7fccc80d14b0 msgr2=0x7fccc806eba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:07.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.060+0000 7fcccfe38700 1 -- 172.21.15.16:0/2838989445 shutdown_connections 2023-10-16T12:26:07.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.060+0000 7fcccfe38700 1 -- 172.21.15.16:0/2838989445 wait complete. 2023-10-16T12:26:07.065 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 27 2023-10-16T12:26:07.075 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:26:07.168 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:26:07.167+0000 7fab230f8200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:26:07.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.203+0000 7f47d7088700 1 Processor -- start 2023-10-16T12:26:07.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.203+0000 7f47d7088700 1 -- start start 2023-10-16T12:26:07.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.204+0000 7f47d7088700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f47d00ce4f0 0x7f47d00d0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:07.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.204+0000 7f47d7088700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f47d015ba10 0x7f47d00cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:07.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.204+0000 7f47d7088700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47d00b0970 con 0x7f47d015b630 2023-10-16T12:26:07.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.204+0000 7f47d7088700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47d00cd690 con 0x7f47d015ba10 2023-10-16T12:26:07.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.204+0000 7f47d7088700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47d00cd970 con 0x7f47d00ce4f0 2023-10-16T12:26:07.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.204+0000 7f47d5625700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f47d015ba10 0x7f47d00cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:07.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.204+0000 7f47d4e24700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f47d015b630 0x7f47d00cdde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:56162/0 (socket says 172.21.15.16:56162) 2023-10-16T12:26:07.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.204+0000 7f47d4e24700 1 -- 172.21.15.16:0/645171253 learned_addr learned my addr 172.21.15.16:0/645171253 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:26:07.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.204+0000 7f47cffff700 1 --2- 172.21.15.16:0/645171253 >> v2:172.21.15.49:3300/0 conn(0x7f47d00ce4f0 0x7f47d00d0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:07.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.204+0000 7f47d5625700 1 -- 172.21.15.16:0/645171253 >> v2:172.21.15.49:3300/0 conn(0x7f47d00ce4f0 msgr2=0x7f47d00d0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:07.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.204+0000 7f47d5625700 1 --2- 172.21.15.16:0/645171253 >> v2:172.21.15.49:3300/0 conn(0x7f47d00ce4f0 0x7f47d00d0380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:07.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.204+0000 7f47d5625700 1 -- 172.21.15.16:0/645171253 >> v1:172.21.15.16:6789/0 conn(0x7f47d015b630 legacy=0x7f47d00cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:07.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.204+0000 7f47cf7fe700 1 -- 172.21.15.16:0/645171253 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 962477362 0 0) 0x7f47d00b0970 con 0x7f47d015b630 2023-10-16T12:26:07.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.204+0000 7f47d5625700 1 -- 172.21.15.16:0/645171253 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47d00b0c30 con 0x7f47d015ba10 2023-10-16T12:26:07.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.204+0000 7f47d5625700 1 --2- 172.21.15.16:0/645171253 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47d015ba10 0x7f47d00cf290 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f47d00d2460 tx=0x7f47c800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=324b3bc189540152 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:07.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.205+0000 7f47cf7fe700 1 -- 172.21.15.16:0/645171253 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47c800e070 con 0x7f47d015ba10 2023-10-16T12:26:07.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.205+0000 7f47cf7fe700 1 -- 172.21.15.16:0/645171253 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47c800bb20 con 0x7f47d015ba10 2023-10-16T12:26:07.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.205+0000 7f47cf7fe700 1 -- 172.21.15.16:0/645171253 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47c8012430 con 0x7f47d015ba10 2023-10-16T12:26:07.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.205+0000 7f47d7088700 1 -- 172.21.15.16:0/645171253 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47d015ba10 msgr2=0x7f47d00cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:07.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.205+0000 7f47d7088700 1 --2- 172.21.15.16:0/645171253 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47d015ba10 0x7f47d00cf290 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f47d00d2460 tx=0x7f47c800b330 comp rx=0 tx=0).stop 2023-10-16T12:26:07.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.205+0000 7f47d7088700 1 -- 172.21.15.16:0/645171253 shutdown_connections 2023-10-16T12:26:07.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.205+0000 7f47d7088700 1 --2- 172.21.15.16:0/645171253 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47d015ba10 0x7f47d00cf290 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:07.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.205+0000 7f47d7088700 1 --2- 172.21.15.16:0/645171253 >> v2:172.21.15.49:3300/0 conn(0x7f47d00ce4f0 0x7f47d00d0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:07.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.205+0000 7f47d7088700 1 -- 172.21.15.16:0/645171253 >> 172.21.15.16:0/645171253 conn(0x7f47d00d14b0 msgr2=0x7f47d00d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:07.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.205+0000 7f47d7088700 1 -- 172.21.15.16:0/645171253 shutdown_connections 2023-10-16T12:26:07.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.205+0000 7f47d7088700 1 -- 172.21.15.16:0/645171253 wait complete. 2023-10-16T12:26:07.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.206+0000 7f47d7088700 1 Processor -- start 2023-10-16T12:26:07.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.207+0000 7f47d7088700 1 -- start start 2023-10-16T12:26:07.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.207+0000 7f47d7088700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47d00ce4f0 0x7f47d0150ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:07.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.207+0000 7f47d7088700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f47d015b630 0x7f47d00b4d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:07.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.207+0000 7f47d7088700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f47d015ba10 0x7f47d015e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:07.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.207+0000 7f47d7088700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47d00b6450 con 0x7f47d00ce4f0 2023-10-16T12:26:07.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.207+0000 7f47d7088700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47d0151010 con 0x7f47d015ba10 2023-10-16T12:26:07.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.207+0000 7f47d7088700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47d00b6870 con 0x7f47d015b630 2023-10-16T12:26:07.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.208+0000 7f47d4e24700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47d00ce4f0 0x7f47d0150ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:07.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.208+0000 7f47d4e24700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47d00ce4f0 0x7f47d0150ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38662/0 (socket says 172.21.15.16:38662) 2023-10-16T12:26:07.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.208+0000 7f47d4e24700 1 -- 172.21.15.16:0/1537623402 learned_addr learned my addr 172.21.15.16:0/1537623402 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:07.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.208+0000 7f47d5625700 1 --2- 172.21.15.16:0/1537623402 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f47d015ba10 0x7f47d015e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:07.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.208+0000 7f47cffff700 1 --2- 172.21.15.16:0/1537623402 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f47d015b630 0x7f47d00b4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:07.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.208+0000 7f47d4e24700 1 -- 172.21.15.16:0/1537623402 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f47d015b630 msgr2=0x7f47d00b4d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:07.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.208+0000 7f47d4e24700 1 --2- 172.21.15.16:0/1537623402 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f47d015b630 0x7f47d00b4d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:07.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.208+0000 7f47d4e24700 1 -- 172.21.15.16:0/1537623402 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f47d015ba10 msgr2=0x7f47d015e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:07.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.208+0000 7f47d4e24700 1 --2- 172.21.15.16:0/1537623402 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f47d015ba10 0x7f47d015e120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:07.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.208+0000 7f47d4e24700 1 -- 172.21.15.16:0/1537623402 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47c8009a70 con 0x7f47d00ce4f0 2023-10-16T12:26:07.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.208+0000 7f47d4e24700 1 --2- 172.21.15.16:0/1537623402 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47d00ce4f0 0x7f47d0150ad0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f47c40011e0 tx=0x7f47c400ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:07.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.208+0000 7f47cd7fa700 1 -- 172.21.15.16:0/1537623402 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47c4004c80 con 0x7f47d00ce4f0 2023-10-16T12:26:07.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.208+0000 7f47cd7fa700 1 -- 172.21.15.16:0/1537623402 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47c4004e00 con 0x7f47d00ce4f0 2023-10-16T12:26:07.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.208+0000 7f47d7088700 1 -- 172.21.15.16:0/1537623402 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47d00b57d0 con 0x7f47d00ce4f0 2023-10-16T12:26:07.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.208+0000 7f47cd7fa700 1 -- 172.21.15.16:0/1537623402 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47c4007310 con 0x7f47d00ce4f0 2023-10-16T12:26:07.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.208+0000 7f47d7088700 1 -- 172.21.15.16:0/1537623402 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47d00b5d90 con 0x7f47d00ce4f0 2023-10-16T12:26:07.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.209+0000 7f47d7088700 1 -- 172.21.15.16:0/1537623402 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47d004ebb0 con 0x7f47d00ce4f0 2023-10-16T12:26:07.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.210+0000 7f47cd7fa700 1 -- 172.21.15.16:0/1537623402 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 27) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f47c4007470 con 0x7f47d00ce4f0 2023-10-16T12:26:07.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.210+0000 7f47cd7fa700 1 --2- 172.21.15.16:0/1537623402 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f47b406b020 0x7f47b406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:07.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.210+0000 7f47cffff700 1 -- 172.21.15.16:0/1537623402 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f47b406b020 msgr2=0x7f47b406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/4075855741 2023-10-16T12:26:07.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.210+0000 7f47cd7fa700 1 -- 172.21.15.16:0/1537623402 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6503+0+0 (secure 0 0 0) 0x7f47c40897a0 con 0x7f47d00ce4f0 2023-10-16T12:26:07.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.210+0000 7f47cffff700 1 --2- 172.21.15.16:0/1537623402 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f47b406b020 0x7f47b406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:26:07.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.215+0000 7f47cd7fa700 1 -- 172.21.15.16:0/1537623402 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f47c40543a0 con 0x7f47d00ce4f0 2023-10-16T12:26:07.250 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:26:07.249+0000 7fab230f8200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T12:26:07.250 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:07.249+0000 7fd5cc790200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:26:07.334 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:07.332+0000 7fd5cc790200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T12:26:07.400 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:26:07.399+0000 7fab230f8200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:26:07.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.411+0000 7f47cffff700 1 -- 172.21.15.16:0/1537623402 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f47b406b020 msgr2=0x7f47b406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/4075855741 2023-10-16T12:26:07.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.411+0000 7f47cffff700 1 --2- 172.21.15.16:0/1537623402 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f47b406b020 0x7f47b406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:26:07.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.422+0000 7f47d7088700 1 -- 172.21.15.16:0/1537623402 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f47d006b5d0 con 0x7f47d00ce4f0 2023-10-16T12:26:07.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.423+0000 7f47cd7fa700 1 -- 172.21.15.16:0/1537623402 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v73) v1 ==== 74+0+17336 (secure 0 0 0) 0x7f47c40579c0 con 0x7f47d00ce4f0 2023-10-16T12:26:07.426 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:26:07.426 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":73,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:26:05.274972+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:24:01.200494+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-10-16T12:24:01.652885+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":73,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.14","mappings":[{"from":0,"to":6}]},{"pgid":"8.1d","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/2103819518":"2023-10-17T12:24:17.735687+0000","172.21.15.16:0/224613758":"2023-10-17T12:24:17.735687+0000","172.21.15.16:0/877922677":"2023-10-17T12:24:17.735687+0000","172.21.15.16:0/1135331895":"2023-10-17T12:24:17.735687+0000","172.21.15.16:0/1214553139":"2023-10-17T12:24:17.735687+0000","172.21.15.16:0/1240820252":"2023-10-17T12:24:17.735687+0000","172.21.15.16:6833/3870937344":"2023-10-17T12:24:17.735687+0000","172.21.15.16:6832/3870937344":"2023-10-17T12:24:17.735687+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[{"pool":8,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:26:07.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.425+0000 7f47d7088700 1 -- 172.21.15.16:0/1537623402 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f47b406b020 msgr2=0x7f47b406d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:26:07.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.425+0000 7f47d7088700 1 --2- 172.21.15.16:0/1537623402 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f47b406b020 0x7f47b406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:07.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.425+0000 7f47d7088700 1 -- 172.21.15.16:0/1537623402 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47d00ce4f0 msgr2=0x7f47d0150ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:07.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.425+0000 7f47d7088700 1 --2- 172.21.15.16:0/1537623402 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47d00ce4f0 0x7f47d0150ad0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f47c40011e0 tx=0x7f47c400ca30 comp rx=0 tx=0).stop 2023-10-16T12:26:07.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.426+0000 7f47d7088700 1 -- 172.21.15.16:0/1537623402 shutdown_connections 2023-10-16T12:26:07.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.426+0000 7f47d7088700 1 --2- 172.21.15.16:0/1537623402 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f47b406b020 0x7f47b406d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:07.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.426+0000 7f47d7088700 1 --2- 172.21.15.16:0/1537623402 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f47d015ba10 0x7f47d015e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:07.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.426+0000 7f47d7088700 1 --2- 172.21.15.16:0/1537623402 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47d00ce4f0 0x7f47d0150ad0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:07.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.426+0000 7f47d7088700 1 --2- 172.21.15.16:0/1537623402 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f47d015b630 0x7f47d00b4d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:07.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.426+0000 7f47d7088700 1 -- 172.21.15.16:0/1537623402 >> 172.21.15.16:0/1537623402 conn(0x7f47d00d14b0 msgr2=0x7f47d015a940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:07.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.426+0000 7f47d7088700 1 -- 172.21.15.16:0/1537623402 shutdown_connections 2023-10-16T12:26:07.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.426+0000 7f47d7088700 1 -- 172.21.15.16:0/1537623402 wait complete. 2023-10-16T12:26:07.440 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T12:24:01.200494+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T12:24:01.652885+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '73', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 5, 'snap_epoch': 73, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T12:26:07.441 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:26:07.491 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:07.489+0000 7fd5cc790200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:26:07.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.571+0000 7ff9b3269700 1 Processor -- start 2023-10-16T12:26:07.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.572+0000 7ff9b3269700 1 -- start start 2023-10-16T12:26:07.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.572+0000 7ff9b3269700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff9ac15b2c0 0x7ff9ac161340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:07.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.572+0000 7ff9b3269700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7ff9ac06e570 0x7ff9ac15e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:07.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.572+0000 7ff9b3269700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff9ac06a7e0 0x7ff9ac06ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:07.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.572+0000 7ff9b3269700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9ac06b1b0 con 0x7ff9ac06a7e0 2023-10-16T12:26:07.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.572+0000 7ff9b3269700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9ac161880 con 0x7ff9ac06e570 2023-10-16T12:26:07.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.572+0000 7ff9b3269700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9ac15b6a0 con 0x7ff9ac15b2c0 2023-10-16T12:26:07.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.573+0000 7ff9b1806700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff9ac06a7e0 0x7ff9ac06ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:07.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.573+0000 7ff9b1806700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff9ac06a7e0 0x7ff9ac06ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38720/0 (socket says 172.21.15.16:38720) 2023-10-16T12:26:07.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.573+0000 7ff9b1806700 1 -- 172.21.15.16:0/75842197 learned_addr learned my addr 172.21.15.16:0/75842197 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:07.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.573+0000 7ff9b1005700 1 --2- 172.21.15.16:0/75842197 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff9ac15b2c0 0x7ff9ac161340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:07.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.573+0000 7ff9b0804700 1 --2- 172.21.15.16:0/75842197 >> v2:172.21.15.49:3300/0 conn(0x7ff9ac06e570 0x7ff9ac15e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:07.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.573+0000 7ff9b1806700 1 -- 172.21.15.16:0/75842197 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff9ac15b2c0 msgr2=0x7ff9ac161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:07.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.573+0000 7ff9b1806700 1 --2- 172.21.15.16:0/75842197 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff9ac15b2c0 0x7ff9ac161340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:07.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.573+0000 7ff9b1806700 1 -- 172.21.15.16:0/75842197 >> v2:172.21.15.49:3300/0 conn(0x7ff9ac06e570 msgr2=0x7ff9ac15e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:07.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.573+0000 7ff9b1806700 1 --2- 172.21.15.16:0/75842197 >> v2:172.21.15.49:3300/0 conn(0x7ff9ac06e570 0x7ff9ac15e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:07.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.573+0000 7ff9b1806700 1 -- 172.21.15.16:0/75842197 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9ac161b20 con 0x7ff9ac06a7e0 2023-10-16T12:26:07.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.573+0000 7ff9b1806700 1 --2- 172.21.15.16:0/75842197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9ac06a7e0 0x7ff9ac06ac70 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7ff9ac070410 tx=0x7ff9a400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=900cefeb42853be1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:07.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.573+0000 7ff9a3fff700 1 -- 172.21.15.16:0/75842197 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a400e070 con 0x7ff9ac06a7e0 2023-10-16T12:26:07.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.573+0000 7ff9a3fff700 1 -- 172.21.15.16:0/75842197 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9a400b910 con 0x7ff9ac06a7e0 2023-10-16T12:26:07.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.573+0000 7ff9a3fff700 1 -- 172.21.15.16:0/75842197 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a400bcc0 con 0x7ff9ac06a7e0 2023-10-16T12:26:07.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.574+0000 7ff9b3269700 1 -- 172.21.15.16:0/75842197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9ac06a7e0 msgr2=0x7ff9ac06ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:07.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.574+0000 7ff9b3269700 1 --2- 172.21.15.16:0/75842197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9ac06a7e0 0x7ff9ac06ac70 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7ff9ac070410 tx=0x7ff9a400b330 comp rx=0 tx=0).stop 2023-10-16T12:26:07.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.574+0000 7ff9b3269700 1 -- 172.21.15.16:0/75842197 shutdown_connections 2023-10-16T12:26:07.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.574+0000 7ff9b3269700 1 --2- 172.21.15.16:0/75842197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9ac06a7e0 0x7ff9ac06ac70 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:07.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.574+0000 7ff9b3269700 1 --2- 172.21.15.16:0/75842197 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff9ac15b2c0 0x7ff9ac161340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:07.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.574+0000 7ff9b3269700 1 --2- 172.21.15.16:0/75842197 >> v2:172.21.15.49:3300/0 conn(0x7ff9ac06e570 0x7ff9ac15e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:07.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.574+0000 7ff9b3269700 1 -- 172.21.15.16:0/75842197 >> 172.21.15.16:0/75842197 conn(0x7ff9ac0b1f40 msgr2=0x7ff9ac0b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:07.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.574+0000 7ff9b3269700 1 -- 172.21.15.16:0/75842197 shutdown_connections 2023-10-16T12:26:07.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.574+0000 7ff9b3269700 1 -- 172.21.15.16:0/75842197 wait complete. 2023-10-16T12:26:07.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.575+0000 7ff9b3269700 1 Processor -- start 2023-10-16T12:26:07.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.575+0000 7ff9b3269700 1 -- start start 2023-10-16T12:26:07.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.576+0000 7ff9b3269700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff9ac06a7e0 0x7ff9ac14f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:07.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.576+0000 7ff9b3269700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff9ac06e570 0x7ff9ac160a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:07.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.576+0000 7ff9b3269700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9ac15b2c0 0x7ff9ac160f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:07.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.576+0000 7ff9b3269700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9ac1f3180 con 0x7ff9ac15b2c0 2023-10-16T12:26:07.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.576+0000 7ff9b3269700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9ac14fec0 con 0x7ff9ac06a7e0 2023-10-16T12:26:07.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.576+0000 7ff9b3269700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9ac1f3370 con 0x7ff9ac06e570 2023-10-16T12:26:07.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.576+0000 7ff9b1806700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9ac15b2c0 0x7ff9ac160f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:07.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.576+0000 7ff9b1806700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9ac15b2c0 0x7ff9ac160f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38726/0 (socket says 172.21.15.16:38726) 2023-10-16T12:26:07.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.576+0000 7ff9b0804700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff9ac06e570 0x7ff9ac160a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:07.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.576+0000 7ff9b1005700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff9ac06a7e0 0x7ff9ac14f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:07.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.576+0000 7ff9b1806700 1 -- 172.21.15.16:0/2456959593 learned_addr learned my addr 172.21.15.16:0/2456959593 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:07.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.577+0000 7ff9b1806700 1 -- 172.21.15.16:0/2456959593 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff9ac06e570 msgr2=0x7ff9ac160a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:07.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.577+0000 7ff9b1806700 1 --2- 172.21.15.16:0/2456959593 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff9ac06e570 0x7ff9ac160a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:07.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.577+0000 7ff9b1806700 1 -- 172.21.15.16:0/2456959593 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff9ac06a7e0 msgr2=0x7ff9ac14f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:07.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.577+0000 7ff9b1806700 1 --2- 172.21.15.16:0/2456959593 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff9ac06a7e0 0x7ff9ac14f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:07.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.577+0000 7ff9b1806700 1 -- 172.21.15.16:0/2456959593 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9a4009b30 con 0x7ff9ac15b2c0 2023-10-16T12:26:07.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.577+0000 7ff9b0804700 1 --2- 172.21.15.16:0/2456959593 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff9ac06e570 0x7ff9ac160a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:26:07.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.577+0000 7ff9b1806700 1 --2- 172.21.15.16:0/2456959593 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9ac15b2c0 0x7ff9ac160f40 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7ff9ac070620 tx=0x7ff9a4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:07.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.577+0000 7ff9a1ffb700 1 -- 172.21.15.16:0/2456959593 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a40099b0 con 0x7ff9ac15b2c0 2023-10-16T12:26:07.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.577+0000 7ff9a1ffb700 1 -- 172.21.15.16:0/2456959593 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9a40092e0 con 0x7ff9ac15b2c0 2023-10-16T12:26:07.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.577+0000 7ff9b3269700 1 -- 172.21.15.16:0/2456959593 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9ac1f3610 con 0x7ff9ac15b2c0 2023-10-16T12:26:07.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.577+0000 7ff9a1ffb700 1 -- 172.21.15.16:0/2456959593 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a4004470 con 0x7ff9ac15b2c0 2023-10-16T12:26:07.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.577+0000 7ff9b3269700 1 -- 172.21.15.16:0/2456959593 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9ac1f3bd0 con 0x7ff9ac15b2c0 2023-10-16T12:26:07.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.578+0000 7ff9b3269700 1 -- 172.21.15.16:0/2456959593 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9ac04ebb0 con 0x7ff9ac15b2c0 2023-10-16T12:26:07.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.579+0000 7ff9a1ffb700 1 -- 172.21.15.16:0/2456959593 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 27) v1 ==== 89343+0+0 (secure 0 0 0) 0x7ff9a40045d0 con 0x7ff9ac15b2c0 2023-10-16T12:26:07.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.579+0000 7ff9a1ffb700 1 --2- 172.21.15.16:0/2456959593 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7ff99006b020 0x7ff99006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:07.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.579+0000 7ff9b1005700 1 -- 172.21.15.16:0/2456959593 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7ff99006b020 msgr2=0x7ff99006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/4075855741 2023-10-16T12:26:07.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.579+0000 7ff9a1ffb700 1 -- 172.21.15.16:0/2456959593 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6503+0+0 (secure 0 0 0) 0x7ff9a408ab90 con 0x7ff9ac15b2c0 2023-10-16T12:26:07.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.579+0000 7ff9b1005700 1 --2- 172.21.15.16:0/2456959593 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7ff99006b020 0x7ff99006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:26:07.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.584+0000 7ff9a1ffb700 1 -- 172.21.15.16:0/2456959593 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff9a40591a0 con 0x7ff9ac15b2c0 2023-10-16T12:26:07.636 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:26:07.635+0000 7fab230f8200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T12:26:07.713 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:26:07.712+0000 7fab230f8200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T12:26:07.736 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:07.734+0000 7fd5cc790200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T12:26:07.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.780+0000 7ff9b1005700 1 -- 172.21.15.16:0/2456959593 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7ff99006b020 msgr2=0x7ff99006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/4075855741 2023-10-16T12:26:07.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.780+0000 7ff9b1005700 1 --2- 172.21.15.16:0/2456959593 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7ff99006b020 0x7ff99006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:26:07.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.785+0000 7ff9b3269700 1 -- 172.21.15.16:0/2456959593 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff9ac0b6370 con 0x7ff9ac15b2c0 2023-10-16T12:26:07.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.786+0000 7ff9a1ffb700 1 -- 172.21.15.16:0/2456959593 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v73) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff9a4029020 con 0x7ff9ac15b2c0 2023-10-16T12:26:07.788 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T12:26:07.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.788+0000 7ff9b3269700 1 -- 172.21.15.16:0/2456959593 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7ff99006b020 msgr2=0x7ff99006d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:26:07.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.788+0000 7ff9b3269700 1 --2- 172.21.15.16:0/2456959593 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7ff99006b020 0x7ff99006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:07.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.788+0000 7ff9b3269700 1 -- 172.21.15.16:0/2456959593 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9ac15b2c0 msgr2=0x7ff9ac160f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:07.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.788+0000 7ff9b3269700 1 --2- 172.21.15.16:0/2456959593 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9ac15b2c0 0x7ff9ac160f40 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7ff9ac070620 tx=0x7ff9a4004ad0 comp rx=0 tx=0).stop 2023-10-16T12:26:07.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.788+0000 7ff9b3269700 1 -- 172.21.15.16:0/2456959593 shutdown_connections 2023-10-16T12:26:07.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.788+0000 7ff9b3269700 1 --2- 172.21.15.16:0/2456959593 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7ff99006b020 0x7ff99006d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:07.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.788+0000 7ff9b3269700 1 --2- 172.21.15.16:0/2456959593 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9ac15b2c0 0x7ff9ac160f40 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:07.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.788+0000 7ff9b3269700 1 --2- 172.21.15.16:0/2456959593 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff9ac06a7e0 0x7ff9ac14f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:07.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.788+0000 7ff9b3269700 1 --2- 172.21.15.16:0/2456959593 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff9ac06e570 0x7ff9ac160a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:07.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.788+0000 7ff9b3269700 1 -- 172.21.15.16:0/2456959593 >> 172.21.15.16:0/2456959593 conn(0x7ff9ac0b1f40 msgr2=0x7ff9ac15b880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:07.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.788+0000 7ff9b3269700 1 -- 172.21.15.16:0/2456959593 shutdown_connections 2023-10-16T12:26:07.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.788+0000 7ff9b3269700 1 -- 172.21.15.16:0/2456959593 wait complete. 2023-10-16T12:26:07.803 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T12:26:07.815 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:07.814+0000 7fd5cc790200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T12:26:07.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.935+0000 7f335dcf1700 1 Processor -- start 2023-10-16T12:26:07.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.935+0000 7f335dcf1700 1 -- start start 2023-10-16T12:26:07.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.936+0000 7f335dcf1700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f335815b070 0x7f335815b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:07.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.936+0000 7f335dcf1700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f335806e170 0x7f33580cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:07.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.936+0000 7f335dcf1700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33580cef70 con 0x7f33580cb880 2023-10-16T12:26:07.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.936+0000 7f335dcf1700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33580ce360 con 0x7f335806e170 2023-10-16T12:26:07.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.936+0000 7f335dcf1700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33580ce640 con 0x7f335815b070 2023-10-16T12:26:07.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.936+0000 7f3357fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f335806e170 0x7f33580cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:07.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.936+0000 7f3357fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f335806e170 0x7f33580cbd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38736/0 (socket says 172.21.15.16:38736) 2023-10-16T12:26:07.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.936+0000 7f3357fff700 1 -- 172.21.15.16:0/1425937257 learned_addr learned my addr 172.21.15.16:0/1425937257 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:07.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.936+0000 7f33577fe700 1 --2- 172.21.15.16:0/1425937257 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f335815b070 0x7f335815b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:07.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.937+0000 7f3357fff700 1 -- 172.21.15.16:0/1425937257 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f335815b070 msgr2=0x7f335815b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:07.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.937+0000 7f3357fff700 1 --2- 172.21.15.16:0/1425937257 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f335815b070 0x7f335815b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:07.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.937+0000 7f3357fff700 1 -- 172.21.15.16:0/1425937257 >> v1:172.21.15.49:6789/0 conn(0x7f33580cb880 legacy=0x7f335806da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:07.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.937+0000 7f33567fc700 1 -- 172.21.15.16:0/1425937257 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2348909212 0 0) 0x7f33580cef70 con 0x7f33580cb880 2023-10-16T12:26:07.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.937+0000 7f3357fff700 1 -- 172.21.15.16:0/1425937257 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33580ce920 con 0x7f335806e170 2023-10-16T12:26:07.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.937+0000 7f3357fff700 1 --2- 172.21.15.16:0/1425937257 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f335806e170 0x7f33580cbd00 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f335000bb90 tx=0x7f335000bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48b75524d5919b3c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:07.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.937+0000 7f33567fc700 1 -- 172.21.15.16:0/1425937257 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f335000dd30 con 0x7f335806e170 2023-10-16T12:26:07.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.937+0000 7f33567fc700 1 -- 172.21.15.16:0/1425937257 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f335000de90 con 0x7f335806e170 2023-10-16T12:26:07.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.937+0000 7f33567fc700 1 -- 172.21.15.16:0/1425937257 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3350010690 con 0x7f335806e170 2023-10-16T12:26:07.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.938+0000 7f335dcf1700 1 -- 172.21.15.16:0/1425937257 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f335806e170 msgr2=0x7f33580cbd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:07.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.938+0000 7f335dcf1700 1 --2- 172.21.15.16:0/1425937257 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f335806e170 0x7f33580cbd00 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f335000bb90 tx=0x7f335000bea0 comp rx=0 tx=0).stop 2023-10-16T12:26:07.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.938+0000 7f335dcf1700 1 -- 172.21.15.16:0/1425937257 shutdown_connections 2023-10-16T12:26:07.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.938+0000 7f335dcf1700 1 --2- 172.21.15.16:0/1425937257 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f335806e170 0x7f33580cbd00 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:07.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.938+0000 7f335dcf1700 1 --2- 172.21.15.16:0/1425937257 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f335815b070 0x7f335815b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:07.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.938+0000 7f335dcf1700 1 -- 172.21.15.16:0/1425937257 >> 172.21.15.16:0/1425937257 conn(0x7f33580b1f40 msgr2=0x7f33580b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:07.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.938+0000 7f335dcf1700 1 -- 172.21.15.16:0/1425937257 shutdown_connections 2023-10-16T12:26:07.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.938+0000 7f335dcf1700 1 -- 172.21.15.16:0/1425937257 wait complete. 2023-10-16T12:26:07.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.939+0000 7f335dcf1700 1 Processor -- start 2023-10-16T12:26:07.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.939+0000 7f335dcf1700 1 -- start start 2023-10-16T12:26:07.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.940+0000 7f335dcf1700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f335806e170 0x7f33580b3e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:07.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.940+0000 7f335dcf1700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f33580cb880 0x7f3358146630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:07.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.940+0000 7f335dcf1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f335815b070 0x7f33580b0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:07.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.940+0000 7f335dcf1700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f335815db00 con 0x7f335815b070 2023-10-16T12:26:07.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.940+0000 7f335dcf1700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33580b30d0 con 0x7f335806e170 2023-10-16T12:26:07.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.940+0000 7f335dcf1700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33580b33b0 con 0x7f33580cb880 2023-10-16T12:26:07.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.940+0000 7f3357fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f335815b070 0x7f33580b0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:07.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.940+0000 7f33577fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f335806e170 0x7f33580b3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:07.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.940+0000 7f3356ffd700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f33580cb880 0x7f3358146630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:07.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.940+0000 7f3357fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f335815b070 0x7f33580b0eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38742/0 (socket says 172.21.15.16:38742) 2023-10-16T12:26:07.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.940+0000 7f3357fff700 1 -- 172.21.15.16:0/2086827912 learned_addr learned my addr 172.21.15.16:0/2086827912 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:07.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.941+0000 7f3356ffd700 1 -- 172.21.15.16:0/2086827912 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f335806e170 msgr2=0x7f33580b3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:07.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.941+0000 7f3356ffd700 1 --2- 172.21.15.16:0/2086827912 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f335806e170 0x7f33580b3e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:07.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.941+0000 7f3356ffd700 1 -- 172.21.15.16:0/2086827912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f335815b070 msgr2=0x7f33580b0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:07.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.941+0000 7f3356ffd700 1 --2- 172.21.15.16:0/2086827912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f335815b070 0x7f33580b0eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:07.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.941+0000 7f3356ffd700 1 -- 172.21.15.16:0/2086827912 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f335000b7f0 con 0x7f33580cb880 2023-10-16T12:26:07.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.941+0000 7f3356ffd700 1 --2- 172.21.15.16:0/2086827912 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f33580cb880 0x7f3358146630 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f334c00c520 tx=0x7f334c00c830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:07.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.941+0000 7f333ffff700 1 -- 172.21.15.16:0/2086827912 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f334c0054e0 con 0x7f33580cb880 2023-10-16T12:26:07.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.941+0000 7f333ffff700 1 -- 172.21.15.16:0/2086827912 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f334c003680 con 0x7f33580cb880 2023-10-16T12:26:07.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.941+0000 7f335dcf1700 1 -- 172.21.15.16:0/2086827912 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33580b3690 con 0x7f33580cb880 2023-10-16T12:26:07.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.941+0000 7f333ffff700 1 -- 172.21.15.16:0/2086827912 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f334c003a30 con 0x7f33580cb880 2023-10-16T12:26:07.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.941+0000 7f335dcf1700 1 -- 172.21.15.16:0/2086827912 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3358146b70 con 0x7f33580cb880 2023-10-16T12:26:07.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.942+0000 7f335dcf1700 1 -- 172.21.15.16:0/2086827912 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3358146d30 con 0x7f33580cb880 2023-10-16T12:26:07.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.942+0000 7f333ffff700 1 -- 172.21.15.16:0/2086827912 <== mon.2 v2:172.21.15.49:3301/0 4 ==== mgrmap(e 27) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f334c003b90 con 0x7f33580cb880 2023-10-16T12:26:07.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.943+0000 7f333ffff700 1 --2- 172.21.15.16:0/2086827912 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f334406b050 0x7f334406d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:07.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.943+0000 7f33577fe700 1 -- 172.21.15.16:0/2086827912 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f334406b050 msgr2=0x7f334406d510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/4075855741 2023-10-16T12:26:07.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.943+0000 7f333ffff700 1 -- 172.21.15.16:0/2086827912 <== mon.2 v2:172.21.15.49:3301/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6503+0+0 (secure 0 0 0) 0x7f334c088ce0 con 0x7f33580cb880 2023-10-16T12:26:07.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.943+0000 7f33577fe700 1 --2- 172.21.15.16:0/2086827912 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f334406b050 0x7f334406d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:26:07.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:07.948+0000 7f333ffff700 1 -- 172.21.15.16:0/2086827912 <== mon.2 v2:172.21.15.49:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f334c053980 con 0x7f33580cb880 2023-10-16T12:26:07.988 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:26:07.987+0000 7fab230f8200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T12:26:08.096 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:08.094+0000 7fd5cc790200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T12:26:08.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.144+0000 7f33577fe700 1 -- 172.21.15.16:0/2086827912 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f334406b050 msgr2=0x7f334406d510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/4075855741 2023-10-16T12:26:08.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.144+0000 7f33577fe700 1 --2- 172.21.15.16:0/2086827912 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f334406b050 0x7f334406d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:26:08.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.151+0000 7f335dcf1700 1 -- 172.21.15.16:0/2086827912 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f33580b3bb0 con 0x7f33580cb880 2023-10-16T12:26:08.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.151+0000 7f333ffff700 1 -- 172.21.15.16:0/2086827912 <== mon.2 v2:172.21.15.49:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v73) v1 ==== 104+0+11 (secure 0 0 0) 0x7f33580b3bb0 con 0x7f33580cb880 2023-10-16T12:26:08.153 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-10-16T12:26:08.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.153+0000 7f335dcf1700 1 -- 172.21.15.16:0/2086827912 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f334406b050 msgr2=0x7f334406d510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:26:08.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.153+0000 7f335dcf1700 1 --2- 172.21.15.16:0/2086827912 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f334406b050 0x7f334406d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:08.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.154+0000 7f335dcf1700 1 -- 172.21.15.16:0/2086827912 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f33580cb880 msgr2=0x7f3358146630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:08.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.154+0000 7f335dcf1700 1 --2- 172.21.15.16:0/2086827912 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f33580cb880 0x7f3358146630 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f334c00c520 tx=0x7f334c00c830 comp rx=0 tx=0).stop 2023-10-16T12:26:08.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.154+0000 7f335dcf1700 1 -- 172.21.15.16:0/2086827912 shutdown_connections 2023-10-16T12:26:08.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.154+0000 7f335dcf1700 1 --2- 172.21.15.16:0/2086827912 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f334406b050 0x7f334406d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:08.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.154+0000 7f335dcf1700 1 --2- 172.21.15.16:0/2086827912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f335815b070 0x7f33580b0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:08.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.154+0000 7f335dcf1700 1 --2- 172.21.15.16:0/2086827912 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f335806e170 0x7f33580b3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:08.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.154+0000 7f335dcf1700 1 --2- 172.21.15.16:0/2086827912 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f33580cb880 0x7f3358146630 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:08.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.154+0000 7f335dcf1700 1 -- 172.21.15.16:0/2086827912 >> 172.21.15.16:0/2086827912 conn(0x7f33580b1f40 msgr2=0x7f33581548f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:08.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.154+0000 7f335dcf1700 1 -- 172.21.15.16:0/2086827912 shutdown_connections 2023-10-16T12:26:08.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.154+0000 7f335dcf1700 1 -- 172.21.15.16:0/2086827912 wait complete. 2023-10-16T12:26:08.168 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T12:26:08.217 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:26:08.216+0000 7fab230f8200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T12:26:08.335 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:08.333+0000 7fd5cc790200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T12:26:08.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.336+0000 7f362bd7f700 1 Processor -- start 2023-10-16T12:26:08.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.336+0000 7f362bd7f700 1 -- start start 2023-10-16T12:26:08.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.337+0000 7f362bd7f700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f362415ec80 0x7f36240cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:08.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.337+0000 7f362bd7f700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36240b2440 con 0x7f36240d2110 2023-10-16T12:26:08.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.337+0000 7f362bd7f700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36240d00e0 con 0x7f362415f060 2023-10-16T12:26:08.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.337+0000 7f362bd7f700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36240b2c50 con 0x7f362415ec80 2023-10-16T12:26:08.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.337+0000 7f362a31c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f36240d2110 0x7f36240b4b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:56258/0 (socket says 172.21.15.16:56258) 2023-10-16T12:26:08.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.337+0000 7f362a31c700 1 -- 172.21.15.16:0/1676773810 learned_addr learned my addr 172.21.15.16:0/1676773810 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:26:08.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.337+0000 7f3629b1b700 1 --2- 172.21.15.16:0/1676773810 >> v2:172.21.15.49:3300/0 conn(0x7f362415ec80 0x7f36240cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:08.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.338+0000 7f3628b19700 1 -- 172.21.15.16:0/1676773810 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2550603095 0 0) 0x7f36240b2440 con 0x7f36240d2110 2023-10-16T12:26:08.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.338+0000 7f3628b19700 1 -- 172.21.15.16:0/1676773810 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36100035b0 con 0x7f36240d2110 2023-10-16T12:26:08.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.338+0000 7f3629b1b700 1 -- 172.21.15.16:0/1676773810 >> v1:172.21.15.49:6789/0 conn(0x7f362415f060 legacy=0x7f36240b4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:08.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.338+0000 7f3628b19700 1 -- 172.21.15.16:0/1676773810 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4195448569 0 0) 0x7f36240d00e0 con 0x7f362415f060 2023-10-16T12:26:08.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.338+0000 7f3629b1b700 1 -- 172.21.15.16:0/1676773810 >> v1:172.21.15.16:6789/0 conn(0x7f36240d2110 legacy=0x7f36240b4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:08.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.338+0000 7f3629b1b700 1 -- 172.21.15.16:0/1676773810 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36240cef60 con 0x7f362415ec80 2023-10-16T12:26:08.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.338+0000 7f3629b1b700 1 --2- 172.21.15.16:0/1676773810 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f362415ec80 0x7f36240cdec0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f3620009840 tx=0x7f3620009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6447b86b301bbf09 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:08.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.338+0000 7f3628b19700 1 -- 172.21.15.16:0/1676773810 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3620011830 con 0x7f362415ec80 2023-10-16T12:26:08.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.338+0000 7f3628b19700 1 -- 172.21.15.16:0/1676773810 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3620011990 con 0x7f362415ec80 2023-10-16T12:26:08.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.338+0000 7f3628b19700 1 -- 172.21.15.16:0/1676773810 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3620011d20 con 0x7f362415ec80 2023-10-16T12:26:08.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.339+0000 7f362bd7f700 1 -- 172.21.15.16:0/1676773810 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f362415ec80 msgr2=0x7f36240cdec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:08.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.339+0000 7f362bd7f700 1 --2- 172.21.15.16:0/1676773810 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f362415ec80 0x7f36240cdec0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f3620009840 tx=0x7f3620009b50 comp rx=0 tx=0).stop 2023-10-16T12:26:08.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.339+0000 7f362bd7f700 1 -- 172.21.15.16:0/1676773810 shutdown_connections 2023-10-16T12:26:08.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.339+0000 7f362bd7f700 1 --2- 172.21.15.16:0/1676773810 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f362415ec80 0x7f36240cdec0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:08.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.339+0000 7f362bd7f700 1 -- 172.21.15.16:0/1676773810 >> 172.21.15.16:0/1676773810 conn(0x7f36240d15f0 msgr2=0x7f36240d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:08.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.339+0000 7f362bd7f700 1 -- 172.21.15.16:0/1676773810 shutdown_connections 2023-10-16T12:26:08.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.339+0000 7f362bd7f700 1 -- 172.21.15.16:0/1676773810 wait complete. 2023-10-16T12:26:08.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.340+0000 7f362bd7f700 1 Processor -- start 2023-10-16T12:26:08.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.340+0000 7f362bd7f700 1 -- start start 2023-10-16T12:26:08.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.341+0000 7f362bd7f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f36240d2110 0x7f36241f1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:08.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.341+0000 7f362bd7f700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f362415ec80 0x7f36240c6c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:08.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.341+0000 7f362bd7f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f362415f060 0x7f36240c7a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:08.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.341+0000 7f362bd7f700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36241f2110 con 0x7f362415f060 2023-10-16T12:26:08.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.341+0000 7f362bd7f700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f362415ce80 con 0x7f362415ec80 2023-10-16T12:26:08.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.341+0000 7f362bd7f700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36241f1870 con 0x7f36240d2110 2023-10-16T12:26:08.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.341+0000 7f362a31c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f362415f060 0x7f36240c7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:08.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.341+0000 7f362a31c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f362415f060 0x7f36240c7a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38758/0 (socket says 172.21.15.16:38758) 2023-10-16T12:26:08.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.341+0000 7f362a31c700 1 -- 172.21.15.16:0/1425942398 learned_addr learned my addr 172.21.15.16:0/1425942398 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:08.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.341+0000 7f3629b1b700 1 --2- 172.21.15.16:0/1425942398 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f36240d2110 0x7f36241f1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:08.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.341+0000 7f362931a700 1 --2- 172.21.15.16:0/1425942398 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f362415ec80 0x7f36240c6c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:08.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.341+0000 7f362a31c700 1 -- 172.21.15.16:0/1425942398 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f36240d2110 msgr2=0x7f36241f1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:08.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.341+0000 7f362a31c700 1 --2- 172.21.15.16:0/1425942398 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f36240d2110 0x7f36241f1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:08.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.341+0000 7f362a31c700 1 -- 172.21.15.16:0/1425942398 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f362415ec80 msgr2=0x7f36240c6c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:08.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.341+0000 7f362a31c700 1 --2- 172.21.15.16:0/1425942398 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f362415ec80 0x7f36240c6c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:08.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.341+0000 7f362a31c700 1 -- 172.21.15.16:0/1425942398 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36200094a0 con 0x7f362415f060 2023-10-16T12:26:08.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.341+0000 7f362a31c700 1 --2- 172.21.15.16:0/1425942398 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f362415f060 0x7f36240c7a30 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f361c00cb40 tx=0x7f361c00ce50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:08.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.341+0000 7f361a7fc700 1 -- 172.21.15.16:0/1425942398 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f361c00a9e0 con 0x7f362415f060 2023-10-16T12:26:08.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.342+0000 7f361a7fc700 1 -- 172.21.15.16:0/1425942398 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f361c00ab40 con 0x7f362415f060 2023-10-16T12:26:08.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.342+0000 7f361a7fc700 1 -- 172.21.15.16:0/1425942398 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f361c012400 con 0x7f362415f060 2023-10-16T12:26:08.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.342+0000 7f362bd7f700 1 -- 172.21.15.16:0/1425942398 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36241f1b50 con 0x7f362415f060 2023-10-16T12:26:08.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.342+0000 7f362bd7f700 1 -- 172.21.15.16:0/1425942398 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36240c5230 con 0x7f362415f060 2023-10-16T12:26:08.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.342+0000 7f362bd7f700 1 -- 172.21.15.16:0/1425942398 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36240d0550 con 0x7f362415f060 2023-10-16T12:26:08.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.343+0000 7f361a7fc700 1 -- 172.21.15.16:0/1425942398 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 27) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f361c01b3b0 con 0x7f362415f060 2023-10-16T12:26:08.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.344+0000 7f361a7fc700 1 --2- 172.21.15.16:0/1425942398 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f360806b020 0x7f360806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:08.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.344+0000 7f3629b1b700 1 -- 172.21.15.16:0/1425942398 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f360806b020 msgr2=0x7f360806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/4075855741 2023-10-16T12:26:08.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.344+0000 7f361a7fc700 1 -- 172.21.15.16:0/1425942398 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6503+0+0 (secure 0 0 0) 0x7f361c089900 con 0x7f362415f060 2023-10-16T12:26:08.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.344+0000 7f3629b1b700 1 --2- 172.21.15.16:0/1425942398 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f360806b020 0x7f360806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:26:08.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.348+0000 7f361a7fc700 1 -- 172.21.15.16:0/1425942398 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f361c0547a0 con 0x7f362415f060 2023-10-16T12:26:08.372 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:26:08.371+0000 7fab230f8200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T12:26:08.491 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:26:08.490+0000 7fab230f8200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T12:26:08.497 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:08.495+0000 7fd5cc790200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T12:26:08.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.545+0000 7f3629b1b700 1 -- 172.21.15.16:0/1425942398 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f360806b020 msgr2=0x7f360806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/4075855741 2023-10-16T12:26:08.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.545+0000 7f3629b1b700 1 --2- 172.21.15.16:0/1425942398 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f360806b020 0x7f360806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:26:08.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.555+0000 7f362bd7f700 1 -- 172.21.15.16:0/1425942398 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f362404ebb0 con 0x7f362415f060 2023-10-16T12:26:08.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.555+0000 7f361a7fc700 1 -- 172.21.15.16:0/1425942398 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v73) v1 ==== 100+0+11 (secure 0 0 0) 0x7f361c057dc0 con 0x7f362415f060 2023-10-16T12:26:08.557 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-10-16T12:26:08.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.557+0000 7f362bd7f700 1 -- 172.21.15.16:0/1425942398 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f360806b020 msgr2=0x7f360806d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:26:08.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.557+0000 7f362bd7f700 1 --2- 172.21.15.16:0/1425942398 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f360806b020 0x7f360806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:08.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.557+0000 7f362bd7f700 1 -- 172.21.15.16:0/1425942398 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f362415f060 msgr2=0x7f36240c7a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:08.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.557+0000 7f362bd7f700 1 --2- 172.21.15.16:0/1425942398 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f362415f060 0x7f36240c7a30 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f361c00cb40 tx=0x7f361c00ce50 comp rx=0 tx=0).stop 2023-10-16T12:26:08.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.557+0000 7f362bd7f700 1 -- 172.21.15.16:0/1425942398 shutdown_connections 2023-10-16T12:26:08.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.557+0000 7f362bd7f700 1 --2- 172.21.15.16:0/1425942398 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f360806b020 0x7f360806d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:08.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.557+0000 7f362bd7f700 1 --2- 172.21.15.16:0/1425942398 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f362415f060 0x7f36240c7a30 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:08.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.557+0000 7f362bd7f700 1 --2- 172.21.15.16:0/1425942398 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f36240d2110 0x7f36241f1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:08.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.557+0000 7f362bd7f700 1 --2- 172.21.15.16:0/1425942398 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f362415ec80 0x7f36240c6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:08.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.557+0000 7f362bd7f700 1 -- 172.21.15.16:0/1425942398 >> 172.21.15.16:0/1425942398 conn(0x7f36240d15f0 msgr2=0x7f36240b3950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:08.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.557+0000 7f362bd7f700 1 -- 172.21.15.16:0/1425942398 shutdown_connections 2023-10-16T12:26:08.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.557+0000 7f362bd7f700 1 -- 172.21.15.16:0/1425942398 wait complete. 2023-10-16T12:26:08.572 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:26:08.619 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:08.618+0000 7fd5cc790200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T12:26:08.653 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:26:08.651+0000 7fab230f8200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T12:26:08.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.696+0000 7f1e8826b700 1 Processor -- start 2023-10-16T12:26:08.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.696+0000 7f1e8826b700 1 -- start start 2023-10-16T12:26:08.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.697+0000 7f1e8826b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1e8015d230 0x7f1e800b5a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:08.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.697+0000 7f1e8826b700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e8015bc80 con 0x7f1e800cf880 2023-10-16T12:26:08.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.697+0000 7f1e8826b700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e8015b070 con 0x7f1e800cf4a0 2023-10-16T12:26:08.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.697+0000 7f1e8826b700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e8015b350 con 0x7f1e8015d230 2023-10-16T12:26:08.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.698+0000 7f1e85806700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f1e800cf880 0x7f1e800b52f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:56270/0 (socket says 172.21.15.16:56270) 2023-10-16T12:26:08.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.698+0000 7f1e86808700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1e8015d230 0x7f1e800b5a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:08.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.698+0000 7f1e85806700 1 -- 172.21.15.16:0/3895591607 learned_addr learned my addr 172.21.15.16:0/3895591607 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:26:08.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.698+0000 7f1e86808700 1 -- 172.21.15.16:0/3895591607 >> v1:172.21.15.49:6789/0 conn(0x7f1e800cf4a0 legacy=0x7f1e800726e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:08.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.698+0000 7f1e86808700 1 -- 172.21.15.16:0/3895591607 >> v1:172.21.15.16:6789/0 conn(0x7f1e800cf880 legacy=0x7f1e800b52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:08.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.698+0000 7f1e85005700 1 -- 172.21.15.16:0/3895591607 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2075363861 0 0) 0x7f1e8015bc80 con 0x7f1e800cf880 2023-10-16T12:26:08.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.698+0000 7f1e86808700 1 -- 172.21.15.16:0/3895591607 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e8015b630 con 0x7f1e8015d230 2023-10-16T12:26:08.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.698+0000 7f1e86808700 1 --2- 172.21.15.16:0/3895591607 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e8015d230 0x7f1e800b5a00 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f1e78009d90 tx=0x7f1e7800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35b1cea48224a36d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:08.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.698+0000 7f1e85005700 1 -- 172.21.15.16:0/3895591607 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e7800e070 con 0x7f1e8015d230 2023-10-16T12:26:08.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.698+0000 7f1e85005700 1 -- 172.21.15.16:0/3895591607 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e7800bbd0 con 0x7f1e8015d230 2023-10-16T12:26:08.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.698+0000 7f1e85005700 1 -- 172.21.15.16:0/3895591607 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e78012430 con 0x7f1e8015d230 2023-10-16T12:26:08.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.699+0000 7f1e8826b700 1 -- 172.21.15.16:0/3895591607 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e8015d230 msgr2=0x7f1e800b5a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:08.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.699+0000 7f1e8826b700 1 --2- 172.21.15.16:0/3895591607 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e8015d230 0x7f1e800b5a00 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f1e78009d90 tx=0x7f1e7800b330 comp rx=0 tx=0).stop 2023-10-16T12:26:08.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.699+0000 7f1e8826b700 1 -- 172.21.15.16:0/3895591607 shutdown_connections 2023-10-16T12:26:08.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.699+0000 7f1e8826b700 1 --2- 172.21.15.16:0/3895591607 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e8015d230 0x7f1e800b5a00 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:08.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.699+0000 7f1e8826b700 1 -- 172.21.15.16:0/3895591607 >> 172.21.15.16:0/3895591607 conn(0x7f1e800b1fd0 msgr2=0x7f1e800b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:08.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.699+0000 7f1e8826b700 1 -- 172.21.15.16:0/3895591607 shutdown_connections 2023-10-16T12:26:08.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.699+0000 7f1e8826b700 1 -- 172.21.15.16:0/3895591607 wait complete. 2023-10-16T12:26:08.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.700+0000 7f1e8826b700 1 Processor -- start 2023-10-16T12:26:08.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.700+0000 7f1e8826b700 1 -- start start 2023-10-16T12:26:08.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.701+0000 7f1e8826b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1e800cf4a0 0x7f1e80150870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:08.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.701+0000 7f1e8826b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e800cf880 0x7f1e800ab1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:08.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.701+0000 7f1e8826b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1e8015d230 0x7f1e800ab730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:08.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.701+0000 7f1e8826b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e80150db0 con 0x7f1e800cf880 2023-10-16T12:26:08.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.701+0000 7f1e8826b700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e800c8f40 con 0x7f1e8015d230 2023-10-16T12:26:08.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.701+0000 7f1e8826b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e800a8db0 con 0x7f1e800cf4a0 2023-10-16T12:26:08.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.701+0000 7f1e85806700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e800cf880 0x7f1e800ab1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:08.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.701+0000 7f1e85806700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e800cf880 0x7f1e800ab1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38772/0 (socket says 172.21.15.16:38772) 2023-10-16T12:26:08.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.701+0000 7f1e85806700 1 -- 172.21.15.16:0/2749291019 learned_addr learned my addr 172.21.15.16:0/2749291019 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:08.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.701+0000 7f1e86808700 1 --2- 172.21.15.16:0/2749291019 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1e8015d230 0x7f1e800ab730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:08.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.701+0000 7f1e86007700 1 --2- 172.21.15.16:0/2749291019 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1e800cf4a0 0x7f1e80150870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:08.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.702+0000 7f1e85806700 1 -- 172.21.15.16:0/2749291019 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1e800cf4a0 msgr2=0x7f1e80150870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:08.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.702+0000 7f1e85806700 1 --2- 172.21.15.16:0/2749291019 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1e800cf4a0 0x7f1e80150870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:08.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.702+0000 7f1e85806700 1 -- 172.21.15.16:0/2749291019 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1e8015d230 msgr2=0x7f1e800ab730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:08.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.702+0000 7f1e85806700 1 --2- 172.21.15.16:0/2749291019 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1e8015d230 0x7f1e800ab730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:08.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.702+0000 7f1e85806700 1 -- 172.21.15.16:0/2749291019 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e78009a70 con 0x7f1e800cf880 2023-10-16T12:26:08.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.702+0000 7f1e86808700 1 --2- 172.21.15.16:0/2749291019 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1e8015d230 0x7f1e800ab730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:26:08.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.702+0000 7f1e85806700 1 --2- 172.21.15.16:0/2749291019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e800cf880 0x7f1e800ab1f0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f1e7000c520 tx=0x7f1e7000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:08.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.702+0000 7f1e76ffd700 1 -- 172.21.15.16:0/2749291019 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e70004c80 con 0x7f1e800cf880 2023-10-16T12:26:08.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.702+0000 7f1e76ffd700 1 -- 172.21.15.16:0/2749291019 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e70004e00 con 0x7f1e800cf880 2023-10-16T12:26:08.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.702+0000 7f1e8826b700 1 -- 172.21.15.16:0/2749291019 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e800c91c0 con 0x7f1e800cf880 2023-10-16T12:26:08.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.702+0000 7f1e76ffd700 1 -- 172.21.15.16:0/2749291019 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e70007310 con 0x7f1e800cf880 2023-10-16T12:26:08.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.702+0000 7f1e8826b700 1 -- 172.21.15.16:0/2749291019 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e800c9780 con 0x7f1e800cf880 2023-10-16T12:26:08.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.703+0000 7f1e8826b700 1 -- 172.21.15.16:0/2749291019 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e800c99a0 con 0x7f1e800cf880 2023-10-16T12:26:08.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.704+0000 7f1e76ffd700 1 -- 172.21.15.16:0/2749291019 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 27) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f1e70007470 con 0x7f1e800cf880 2023-10-16T12:26:08.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.704+0000 7f1e76ffd700 1 --2- 172.21.15.16:0/2749291019 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f1e6406b020 0x7f1e6406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:08.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.704+0000 7f1e86007700 1 -- 172.21.15.16:0/2749291019 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f1e6406b020 msgr2=0x7f1e6406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/4075855741 2023-10-16T12:26:08.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.704+0000 7f1e76ffd700 1 -- 172.21.15.16:0/2749291019 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6503+0+0 (secure 0 0 0) 0x7f1e70089660 con 0x7f1e800cf880 2023-10-16T12:26:08.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.704+0000 7f1e86007700 1 --2- 172.21.15.16:0/2749291019 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f1e6406b020 0x7f1e6406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:26:08.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.708+0000 7f1e76ffd700 1 -- 172.21.15.16:0/2749291019 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1e70054370 con 0x7f1e800cf880 2023-10-16T12:26:08.788 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:08.786+0000 7fd5cc790200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T12:26:08.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.905+0000 7f1e86007700 1 -- 172.21.15.16:0/2749291019 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f1e6406b020 msgr2=0x7f1e6406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/4075855741 2023-10-16T12:26:08.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.905+0000 7f1e86007700 1 --2- 172.21.15.16:0/2749291019 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f1e6406b020 0x7f1e6406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:26:08.922 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:26:08.921+0000 7fab230f8200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T12:26:08.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.958+0000 7f1e8826b700 1 -- 172.21.15.16:0/2749291019 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1e800726e0 con 0x7f1e800cf880 2023-10-16T12:26:08.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.959+0000 7f1e76ffd700 1 -- 172.21.15.16:0/2749291019 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+4913 (secure 0 0 0) 0x7f1e70057990 con 0x7f1e800cf880 2023-10-16T12:26:08.961 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:26:08.961 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5947,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/1103417847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":1103417847},{"type":"v1","addr":"172.21.15.49:6835","nonce":1103417847}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":6096,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/1380792179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1380792179},{"type":"v1","addr":"172.21.15.16:6836","nonce":1380792179}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":6099,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2238585443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":2238585443},{"type":"v1","addr":"172.21.15.16:6837","nonce":2238585443}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:24:02.695223+0000","modified":"2023-10-16T12:24:05.704039+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6111},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6111":{"gid":6111,"name":"b","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.49:6834/3246624425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":3246624425},{"type":"v1","addr":"172.21.15.49:6834","nonce":3246624425}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-10-16T12:26:08.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.961+0000 7f1e8826b700 1 -- 172.21.15.16:0/2749291019 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f1e6406b020 msgr2=0x7f1e6406d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:26:08.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.961+0000 7f1e8826b700 1 --2- 172.21.15.16:0/2749291019 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f1e6406b020 0x7f1e6406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:08.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.961+0000 7f1e8826b700 1 -- 172.21.15.16:0/2749291019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e800cf880 msgr2=0x7f1e800ab1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:08.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.961+0000 7f1e8826b700 1 --2- 172.21.15.16:0/2749291019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e800cf880 0x7f1e800ab1f0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f1e7000c520 tx=0x7f1e7000c830 comp rx=0 tx=0).stop 2023-10-16T12:26:08.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.961+0000 7f1e8826b700 1 -- 172.21.15.16:0/2749291019 shutdown_connections 2023-10-16T12:26:08.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.961+0000 7f1e8826b700 1 --2- 172.21.15.16:0/2749291019 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f1e6406b020 0x7f1e6406d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:08.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.961+0000 7f1e8826b700 1 --2- 172.21.15.16:0/2749291019 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1e8015d230 0x7f1e800ab730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:08.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.961+0000 7f1e8826b700 1 --2- 172.21.15.16:0/2749291019 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1e800cf4a0 0x7f1e80150870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:08.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.961+0000 7f1e8826b700 1 --2- 172.21.15.16:0/2749291019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e800cf880 0x7f1e800ab1f0 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:08.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.961+0000 7f1e8826b700 1 -- 172.21.15.16:0/2749291019 >> 172.21.15.16:0/2749291019 conn(0x7f1e800b1fd0 msgr2=0x7f1e80159cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:08.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.961+0000 7f1e8826b700 1 -- 172.21.15.16:0/2749291019 shutdown_connections 2023-10-16T12:26:08.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:08.962+0000 7f1e8826b700 1 -- 172.21.15.16:0/2749291019 wait complete. 2023-10-16T12:26:08.966 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 27 2023-10-16T12:26:08.976 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:26:09.064 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:09.062+0000 7fd5cc790200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T12:26:09.086 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:26:09.085+0000 7fab230f8200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T12:26:09.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.100+0000 7f5aab206700 1 Processor -- start 2023-10-16T12:26:09.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.100+0000 7f5aab206700 1 -- start start 2023-10-16T12:26:09.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.101+0000 7f5aab206700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5aa40b2b50 0x7f5aa40d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:09.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.101+0000 7f5aab206700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f5aa40ce380 0x7f5aa40d09e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:09.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.101+0000 7f5aab206700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5aa40d1210 con 0x7f5aa40cdfa0 2023-10-16T12:26:09.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.101+0000 7f5aab206700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5aa40cbc30 con 0x7f5aa40ce380 2023-10-16T12:26:09.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.101+0000 7f5aab206700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa40cbf10 con 0x7f5aa40b2b50 2023-10-16T12:26:09.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.101+0000 7f5aa97a3700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f5aa40ce380 0x7f5aa40d09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:09.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.101+0000 7f5aa3fff700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5aa40b2b50 0x7f5aa40d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:09.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.101+0000 7f5aa97a3700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f5aa40ce380 0x7f5aa40d09e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:37846/0 (socket says 172.21.15.16:37846) 2023-10-16T12:26:09.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.101+0000 7f5aa97a3700 1 -- 172.21.15.16:0/2098964240 learned_addr learned my addr 172.21.15.16:0/2098964240 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:09.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.101+0000 7f5aa8fa2700 1 --1- 172.21.15.16:0/2098964240 >> v1:172.21.15.49:6789/0 conn(0x7f5aa40cdfa0 0x7f5aa40b2440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:35236/0 (socket says 172.21.15.16:35236) 2023-10-16T12:26:09.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.102+0000 7f5aa97a3700 1 -- 172.21.15.16:0/2098964240 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5aa40b2b50 msgr2=0x7f5aa40d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:09.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.102+0000 7f5aa97a3700 1 --2- 172.21.15.16:0/2098964240 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5aa40b2b50 0x7f5aa40d04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:09.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.102+0000 7f5aa97a3700 1 -- 172.21.15.16:0/2098964240 >> v1:172.21.15.49:6789/0 conn(0x7f5aa40cdfa0 legacy=0x7f5aa40b2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:09.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.102+0000 7f5aa97a3700 1 -- 172.21.15.16:0/2098964240 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aa40cc1f0 con 0x7f5aa40ce380 2023-10-16T12:26:09.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.102+0000 7f5aa37fe700 1 -- 172.21.15.16:0/2098964240 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4079243708 0 0) 0x7f5aa40d1210 con 0x7f5aa40cdfa0 2023-10-16T12:26:09.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.102+0000 7f5aa97a3700 1 --2- 172.21.15.16:0/2098964240 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5aa40ce380 0x7f5aa40d09e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f5a9c009e50 tx=0x7f5a9c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9b9566b26cbf723b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:09.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.102+0000 7f5aa37fe700 1 -- 172.21.15.16:0/2098964240 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a9c00e070 con 0x7f5aa40ce380 2023-10-16T12:26:09.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.102+0000 7f5aa37fe700 1 -- 172.21.15.16:0/2098964240 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a9c00b910 con 0x7f5aa40ce380 2023-10-16T12:26:09.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.102+0000 7f5aa37fe700 1 -- 172.21.15.16:0/2098964240 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a9c00bcc0 con 0x7f5aa40ce380 2023-10-16T12:26:09.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.103+0000 7f5aab206700 1 -- 172.21.15.16:0/2098964240 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5aa40ce380 msgr2=0x7f5aa40d09e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:09.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.103+0000 7f5aab206700 1 --2- 172.21.15.16:0/2098964240 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5aa40ce380 0x7f5aa40d09e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f5a9c009e50 tx=0x7f5a9c00b330 comp rx=0 tx=0).stop 2023-10-16T12:26:09.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.103+0000 7f5aab206700 1 -- 172.21.15.16:0/2098964240 shutdown_connections 2023-10-16T12:26:09.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.103+0000 7f5aab206700 1 --2- 172.21.15.16:0/2098964240 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5aa40ce380 0x7f5aa40d09e0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:09.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.103+0000 7f5aab206700 1 --2- 172.21.15.16:0/2098964240 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5aa40b2b50 0x7f5aa40d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:09.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.103+0000 7f5aab206700 1 -- 172.21.15.16:0/2098964240 >> 172.21.15.16:0/2098964240 conn(0x7f5aa40d14b0 msgr2=0x7f5aa40d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:09.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.103+0000 7f5aab206700 1 -- 172.21.15.16:0/2098964240 shutdown_connections 2023-10-16T12:26:09.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.103+0000 7f5aab206700 1 -- 172.21.15.16:0/2098964240 wait complete. 2023-10-16T12:26:09.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.104+0000 7f5aab206700 1 Processor -- start 2023-10-16T12:26:09.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.104+0000 7f5aab206700 1 -- start start 2023-10-16T12:26:09.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.105+0000 7f5aab206700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5aa40b2b50 0x7f5aa40c4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:09.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.105+0000 7f5aab206700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5aa40cdfa0 0x7f5aa415e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:09.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.105+0000 7f5aab206700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5aa40ce380 0x7f5aa415ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:09.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.105+0000 7f5aab206700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa4155230 con 0x7f5aa40cdfa0 2023-10-16T12:26:09.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.105+0000 7f5aab206700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa40c80e0 con 0x7f5aa40b2b50 2023-10-16T12:26:09.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.105+0000 7f5aab206700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa41e6ba0 con 0x7f5aa40ce380 2023-10-16T12:26:09.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.106+0000 7f5aa3fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5aa40cdfa0 0x7f5aa415e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:09.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.106+0000 7f5aa8fa2700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5aa40b2b50 0x7f5aa40c4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:09.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.106+0000 7f5aa97a3700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5aa40ce380 0x7f5aa415ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:09.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.106+0000 7f5aa3fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5aa40cdfa0 0x7f5aa415e9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38788/0 (socket says 172.21.15.16:38788) 2023-10-16T12:26:09.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.106+0000 7f5aa8fa2700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5aa40b2b50 0x7f5aa40c4fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:37848/0 (socket says 172.21.15.16:37848) 2023-10-16T12:26:09.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.106+0000 7f5aa3fff700 1 -- 172.21.15.16:0/2807665478 learned_addr learned my addr 172.21.15.16:0/2807665478 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:09.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.106+0000 7f5aa8fa2700 1 -- 172.21.15.16:0/2807665478 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5aa40ce380 msgr2=0x7f5aa415ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:09.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.106+0000 7f5aa8fa2700 1 --2- 172.21.15.16:0/2807665478 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5aa40ce380 0x7f5aa415ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:09.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.106+0000 7f5aa8fa2700 1 -- 172.21.15.16:0/2807665478 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5aa40cdfa0 msgr2=0x7f5aa415e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:09.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.106+0000 7f5aa8fa2700 1 --2- 172.21.15.16:0/2807665478 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5aa40cdfa0 0x7f5aa415e9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:09.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.106+0000 7f5aa8fa2700 1 -- 172.21.15.16:0/2807665478 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a9c009b30 con 0x7f5aa40b2b50 2023-10-16T12:26:09.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.106+0000 7f5aa8fa2700 1 --2- 172.21.15.16:0/2807665478 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5aa40b2b50 0x7f5aa40c4fe0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f5a9800c4b0 tx=0x7f5a9800c7c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:09.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.106+0000 7f5aa17fa700 1 -- 172.21.15.16:0/2807665478 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a98004c80 con 0x7f5aa40b2b50 2023-10-16T12:26:09.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.106+0000 7f5aa17fa700 1 -- 172.21.15.16:0/2807665478 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a98004e00 con 0x7f5aa40b2b50 2023-10-16T12:26:09.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.106+0000 7f5aab206700 1 -- 172.21.15.16:0/2807665478 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5aa415f440 con 0x7f5aa40b2b50 2023-10-16T12:26:09.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.106+0000 7f5aa17fa700 1 -- 172.21.15.16:0/2807665478 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a98007310 con 0x7f5aa40b2b50 2023-10-16T12:26:09.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.106+0000 7f5aab206700 1 -- 172.21.15.16:0/2807665478 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5aa41e6270 con 0x7f5aa40b2b50 2023-10-16T12:26:09.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.107+0000 7f5aab206700 1 -- 172.21.15.16:0/2807665478 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5aa404ebb0 con 0x7f5aa40b2b50 2023-10-16T12:26:09.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.108+0000 7f5aa17fa700 1 -- 172.21.15.16:0/2807665478 <== mon.1 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 27) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f5a98007470 con 0x7f5aa40b2b50 2023-10-16T12:26:09.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.108+0000 7f5aa17fa700 1 --2- 172.21.15.16:0/2807665478 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f5a8806b020 0x7f5a8806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:09.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.108+0000 7f5aa3fff700 1 -- 172.21.15.16:0/2807665478 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f5a8806b020 msgr2=0x7f5a8806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/4075855741 2023-10-16T12:26:09.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.108+0000 7f5aa17fa700 1 -- 172.21.15.16:0/2807665478 <== mon.1 v2:172.21.15.49:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6503+0+0 (secure 0 0 0) 0x7f5a98089560 con 0x7f5aa40b2b50 2023-10-16T12:26:09.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.108+0000 7f5aa3fff700 1 --2- 172.21.15.16:0/2807665478 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f5a8806b020 0x7f5a8806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:26:09.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.113+0000 7f5aa17fa700 1 -- 172.21.15.16:0/2807665478 <== mon.1 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5a98054330 con 0x7f5aa40b2b50 2023-10-16T12:26:09.238 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:09.236+0000 7fd5cc790200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T12:26:09.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.309+0000 7f5aa3fff700 1 -- 172.21.15.16:0/2807665478 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f5a8806b020 msgr2=0x7f5a8806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/4075855741 2023-10-16T12:26:09.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.309+0000 7f5aa3fff700 1 --2- 172.21.15.16:0/2807665478 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f5a8806b020 0x7f5a8806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:26:09.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.320+0000 7f5aab206700 1 -- 172.21.15.16:0/2807665478 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5aa40ccd70 con 0x7f5aa40b2b50 2023-10-16T12:26:09.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.322+0000 7f5aa17fa700 1 -- 172.21.15.16:0/2807665478 <== mon.1 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v73) v1 ==== 74+0+17336 (secure 0 0 0) 0x7f5a98057950 con 0x7f5aa40b2b50 2023-10-16T12:26:09.324 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:26:09.324 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":73,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:26:05.274972+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:24:01.200494+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-10-16T12:24:01.652885+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":73,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.14","mappings":[{"from":0,"to":6}]},{"pgid":"8.1d","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/2103819518":"2023-10-17T12:24:17.735687+0000","172.21.15.16:0/224613758":"2023-10-17T12:24:17.735687+0000","172.21.15.16:0/877922677":"2023-10-17T12:24:17.735687+0000","172.21.15.16:0/1135331895":"2023-10-17T12:24:17.735687+0000","172.21.15.16:0/1214553139":"2023-10-17T12:24:17.735687+0000","172.21.15.16:0/1240820252":"2023-10-17T12:24:17.735687+0000","172.21.15.16:6833/3870937344":"2023-10-17T12:24:17.735687+0000","172.21.15.16:6832/3870937344":"2023-10-17T12:24:17.735687+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[{"pool":8,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:26:09.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.324+0000 7f5aab206700 1 -- 172.21.15.16:0/2807665478 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f5a8806b020 msgr2=0x7f5a8806d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:26:09.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.324+0000 7f5aab206700 1 --2- 172.21.15.16:0/2807665478 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f5a8806b020 0x7f5a8806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:09.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.324+0000 7f5aab206700 1 -- 172.21.15.16:0/2807665478 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5aa40b2b50 msgr2=0x7f5aa40c4fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:09.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.324+0000 7f5aab206700 1 --2- 172.21.15.16:0/2807665478 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5aa40b2b50 0x7f5aa40c4fe0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f5a9800c4b0 tx=0x7f5a9800c7c0 comp rx=0 tx=0).stop 2023-10-16T12:26:09.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.324+0000 7f5aab206700 1 -- 172.21.15.16:0/2807665478 shutdown_connections 2023-10-16T12:26:09.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.324+0000 7f5aab206700 1 --2- 172.21.15.16:0/2807665478 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f5a8806b020 0x7f5a8806d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:09.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.324+0000 7f5aab206700 1 --2- 172.21.15.16:0/2807665478 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5aa40ce380 0x7f5aa415ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:09.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.324+0000 7f5aab206700 1 --2- 172.21.15.16:0/2807665478 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5aa40b2b50 0x7f5aa40c4fe0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:09.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.324+0000 7f5aab206700 1 --2- 172.21.15.16:0/2807665478 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5aa40cdfa0 0x7f5aa415e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:09.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.324+0000 7f5aab206700 1 -- 172.21.15.16:0/2807665478 >> 172.21.15.16:0/2807665478 conn(0x7f5aa40d14b0 msgr2=0x7f5aa40cd290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:09.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.324+0000 7f5aab206700 1 -- 172.21.15.16:0/2807665478 shutdown_connections 2023-10-16T12:26:09.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.324+0000 7f5aab206700 1 -- 172.21.15.16:0/2807665478 wait complete. 2023-10-16T12:26:09.339 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-16T12:26:09.339 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:26:09.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.463+0000 7f5ddeb98700 1 Processor -- start 2023-10-16T12:26:09.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.463+0000 7f5ddeb98700 1 -- start start 2023-10-16T12:26:09.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.464+0000 7f5ddeb98700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5dd815b760 0x7f5dd806a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:09.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.464+0000 7f5ddeb98700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5dd80b2be0 0x7f5dd806ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:09.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.464+0000 7f5ddeb98700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dd80d03b0 con 0x7f5dd80cd6e0 2023-10-16T12:26:09.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.464+0000 7f5ddeb98700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dd815bb40 con 0x7f5dd80b2be0 2023-10-16T12:26:09.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.464+0000 7f5ddeb98700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd80cb760 con 0x7f5dd815b760 2023-10-16T12:26:09.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.464+0000 7f5dd7fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5dd80b2be0 0x7f5dd806ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:09.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.464+0000 7f5ddd135700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f5dd80cd6e0 0x7f5dd80719e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:56306/0 (socket says 172.21.15.16:56306) 2023-10-16T12:26:09.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.464+0000 7f5dd7fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5dd80b2be0 0x7f5dd806ad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38798/0 (socket says 172.21.15.16:38798) 2023-10-16T12:26:09.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.464+0000 7f5ddd135700 1 -- 172.21.15.16:0/2798475837 learned_addr learned my addr 172.21.15.16:0/2798475837 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:26:09.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.464+0000 7f5ddc934700 1 --2- 172.21.15.16:0/2798475837 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5dd815b760 0x7f5dd806a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:09.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.464+0000 7f5dd7fff700 1 -- 172.21.15.16:0/2798475837 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5dd815b760 msgr2=0x7f5dd806a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:09.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.465+0000 7f5dd7fff700 1 --2- 172.21.15.16:0/2798475837 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5dd815b760 0x7f5dd806a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:09.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.465+0000 7f5dd7fff700 1 -- 172.21.15.16:0/2798475837 >> v1:172.21.15.16:6789/0 conn(0x7f5dd80cd6e0 legacy=0x7f5dd80719e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:09.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.465+0000 7f5dd7fff700 1 -- 172.21.15.16:0/2798475837 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dd80d0600 con 0x7f5dd80b2be0 2023-10-16T12:26:09.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.465+0000 7f5dd77fe700 1 -- 172.21.15.16:0/2798475837 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1104117182 0 0) 0x7f5dd80d03b0 con 0x7f5dd80cd6e0 2023-10-16T12:26:09.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.465+0000 7f5dd7fff700 1 --2- 172.21.15.16:0/2798475837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5dd80b2be0 0x7f5dd806ad20 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f5dc4009840 tx=0x7f5dc4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a9bbfc044270738 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:09.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.465+0000 7f5dd77fe700 1 -- 172.21.15.16:0/2798475837 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dc4011830 con 0x7f5dd80b2be0 2023-10-16T12:26:09.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.465+0000 7f5dd77fe700 1 -- 172.21.15.16:0/2798475837 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dc4011990 con 0x7f5dd80b2be0 2023-10-16T12:26:09.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.465+0000 7f5dd77fe700 1 -- 172.21.15.16:0/2798475837 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dc4011d20 con 0x7f5dd80b2be0 2023-10-16T12:26:09.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.466+0000 7f5ddeb98700 1 -- 172.21.15.16:0/2798475837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5dd80b2be0 msgr2=0x7f5dd806ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:09.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.466+0000 7f5ddeb98700 1 --2- 172.21.15.16:0/2798475837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5dd80b2be0 0x7f5dd806ad20 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f5dc4009840 tx=0x7f5dc4009b50 comp rx=0 tx=0).stop 2023-10-16T12:26:09.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.466+0000 7f5ddeb98700 1 -- 172.21.15.16:0/2798475837 shutdown_connections 2023-10-16T12:26:09.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.466+0000 7f5ddeb98700 1 --2- 172.21.15.16:0/2798475837 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5dd815b760 0x7f5dd806a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:09.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.466+0000 7f5ddeb98700 1 --2- 172.21.15.16:0/2798475837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5dd80b2be0 0x7f5dd806ad20 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:09.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.466+0000 7f5ddeb98700 1 -- 172.21.15.16:0/2798475837 >> 172.21.15.16:0/2798475837 conn(0x7f5dd80b1fd0 msgr2=0x7f5dd80b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:09.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.466+0000 7f5ddeb98700 1 -- 172.21.15.16:0/2798475837 shutdown_connections 2023-10-16T12:26:09.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.466+0000 7f5ddeb98700 1 -- 172.21.15.16:0/2798475837 wait complete. 2023-10-16T12:26:09.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.467+0000 7f5ddeb98700 1 Processor -- start 2023-10-16T12:26:09.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.467+0000 7f5ddeb98700 1 -- start start 2023-10-16T12:26:09.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.468+0000 7f5ddeb98700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5dd80b2be0 0x7f5dd80c6ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:09.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.468+0000 7f5ddeb98700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5dd80cd6e0 0x7f5dd815cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:09.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.468+0000 7f5ddeb98700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5dd815b760 0x7f5dd815d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:09.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.468+0000 7f5ddeb98700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd81f2080 con 0x7f5dd815b760 2023-10-16T12:26:09.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.468+0000 7f5ddeb98700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd81f5250 con 0x7f5dd80b2be0 2023-10-16T12:26:09.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.468+0000 7f5ddeb98700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd81f2290 con 0x7f5dd80cd6e0 2023-10-16T12:26:09.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.468+0000 7f5ddd135700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5dd815b760 0x7f5dd815d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:09.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.468+0000 7f5ddd135700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5dd815b760 0x7f5dd815d2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38806/0 (socket says 172.21.15.16:38806) 2023-10-16T12:26:09.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.468+0000 7f5ddd135700 1 -- 172.21.15.16:0/2442377084 learned_addr learned my addr 172.21.15.16:0/2442377084 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:09.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.468+0000 7f5ddc934700 1 --2- 172.21.15.16:0/2442377084 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5dd80b2be0 0x7f5dd80c6ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:09.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.468+0000 7f5dd7fff700 1 --2- 172.21.15.16:0/2442377084 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5dd80cd6e0 0x7f5dd815cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:09.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.468+0000 7f5ddd135700 1 -- 172.21.15.16:0/2442377084 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5dd80cd6e0 msgr2=0x7f5dd815cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:09.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.468+0000 7f5ddd135700 1 --2- 172.21.15.16:0/2442377084 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5dd80cd6e0 0x7f5dd815cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:09.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.468+0000 7f5ddd135700 1 -- 172.21.15.16:0/2442377084 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5dd80b2be0 msgr2=0x7f5dd80c6ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:09.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.468+0000 7f5ddd135700 1 --2- 172.21.15.16:0/2442377084 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5dd80b2be0 0x7f5dd80c6ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:09.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.468+0000 7f5ddd135700 1 -- 172.21.15.16:0/2442377084 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dc40094a0 con 0x7f5dd815b760 2023-10-16T12:26:09.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.469+0000 7f5ddd135700 1 --2- 172.21.15.16:0/2442377084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5dd815b760 0x7f5dd815d2a0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f5dd000cb90 tx=0x7f5dd000cea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:09.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.469+0000 7f5dd57fa700 1 -- 172.21.15.16:0/2442377084 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dd000aaa0 con 0x7f5dd815b760 2023-10-16T12:26:09.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.469+0000 7f5dd57fa700 1 -- 172.21.15.16:0/2442377084 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dd000ac20 con 0x7f5dd815b760 2023-10-16T12:26:09.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.469+0000 7f5dd57fa700 1 -- 172.21.15.16:0/2442377084 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dd00070d0 con 0x7f5dd815b760 2023-10-16T12:26:09.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.469+0000 7f5ddeb98700 1 -- 172.21.15.16:0/2442377084 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dd81f2570 con 0x7f5dd815b760 2023-10-16T12:26:09.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.469+0000 7f5ddeb98700 1 -- 172.21.15.16:0/2442377084 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dd80c71e0 con 0x7f5dd815b760 2023-10-16T12:26:09.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.470+0000 7f5dd57fa700 1 -- 172.21.15.16:0/2442377084 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 27) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f5dd0007230 con 0x7f5dd815b760 2023-10-16T12:26:09.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.470+0000 7f5ddeb98700 1 -- 172.21.15.16:0/2442377084 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dd81f2a90 con 0x7f5dd815b760 2023-10-16T12:26:09.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.471+0000 7f5dd57fa700 1 --2- 172.21.15.16:0/2442377084 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f5dbc06b020 0x7f5dbc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:09.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.471+0000 7f5ddc934700 1 -- 172.21.15.16:0/2442377084 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f5dbc06b020 msgr2=0x7f5dbc06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/4075855741 2023-10-16T12:26:09.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.471+0000 7f5dd57fa700 1 -- 172.21.15.16:0/2442377084 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6503+0+0 (secure 0 0 0) 0x7f5dd00894f0 con 0x7f5dd815b760 2023-10-16T12:26:09.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.471+0000 7f5ddc934700 1 --2- 172.21.15.16:0/2442377084 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f5dbc06b020 0x7f5dbc06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:26:09.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.476+0000 7f5dd57fa700 1 -- 172.21.15.16:0/2442377084 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5dd81f2a90 con 0x7f5dd815b760 2023-10-16T12:26:09.638 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:26:09.637+0000 7fab230f8200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T12:26:09.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.672+0000 7f5ddc934700 1 -- 172.21.15.16:0/2442377084 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f5dbc06b020 msgr2=0x7f5dbc06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/4075855741 2023-10-16T12:26:09.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.672+0000 7f5ddc934700 1 --2- 172.21.15.16:0/2442377084 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f5dbc06b020 0x7f5dbc06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:26:09.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.727+0000 7f5ddeb98700 1 -- 172.21.15.16:0/2442377084 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5dd81f2a90 con 0x7f5dd815b760 2023-10-16T12:26:09.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.727+0000 7f5dd57fa700 1 -- 172.21.15.16:0/2442377084 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+4913 (secure 0 0 0) 0x7f5dd81f2a90 con 0x7f5dd815b760 2023-10-16T12:26:09.730 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:26:09.730 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5947,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/1103417847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":1103417847},{"type":"v1","addr":"172.21.15.49:6835","nonce":1103417847}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":6096,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/1380792179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1380792179},{"type":"v1","addr":"172.21.15.16:6836","nonce":1380792179}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":6099,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2238585443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":2238585443},{"type":"v1","addr":"172.21.15.16:6837","nonce":2238585443}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:24:02.695223+0000","modified":"2023-10-16T12:24:05.704039+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6111},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6111":{"gid":6111,"name":"b","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.49:6834/3246624425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":3246624425},{"type":"v1","addr":"172.21.15.49:6834","nonce":3246624425}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-10-16T12:26:09.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.730+0000 7f5ddeb98700 1 -- 172.21.15.16:0/2442377084 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f5dbc06b020 msgr2=0x7f5dbc06d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:26:09.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.730+0000 7f5ddeb98700 1 --2- 172.21.15.16:0/2442377084 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f5dbc06b020 0x7f5dbc06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:09.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.730+0000 7f5ddeb98700 1 -- 172.21.15.16:0/2442377084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5dd815b760 msgr2=0x7f5dd815d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:09.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.730+0000 7f5ddeb98700 1 --2- 172.21.15.16:0/2442377084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5dd815b760 0x7f5dd815d2a0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f5dd000cb90 tx=0x7f5dd000cea0 comp rx=0 tx=0).stop 2023-10-16T12:26:09.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.730+0000 7f5ddeb98700 1 -- 172.21.15.16:0/2442377084 shutdown_connections 2023-10-16T12:26:09.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.730+0000 7f5ddeb98700 1 --2- 172.21.15.16:0/2442377084 >> [v2:172.21.15.16:6832/4075855741,v1:172.21.15.16:6833/4075855741] conn(0x7f5dbc06b020 0x7f5dbc06d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:09.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.730+0000 7f5ddeb98700 1 --2- 172.21.15.16:0/2442377084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5dd815b760 0x7f5dd815d2a0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:09.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.730+0000 7f5ddeb98700 1 --2- 172.21.15.16:0/2442377084 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5dd80b2be0 0x7f5dd80c6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:09.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.730+0000 7f5ddeb98700 1 --2- 172.21.15.16:0/2442377084 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5dd80cd6e0 0x7f5dd815cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:09.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.730+0000 7f5ddeb98700 1 -- 172.21.15.16:0/2442377084 >> 172.21.15.16:0/2442377084 conn(0x7f5dd80b1fd0 msgr2=0x7f5dd81e5980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:09.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.730+0000 7f5ddeb98700 1 -- 172.21.15.16:0/2442377084 shutdown_connections 2023-10-16T12:26:09.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.730+0000 7f5ddeb98700 1 -- 172.21.15.16:0/2442377084 wait complete. 2023-10-16T12:26:09.734 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 27 2023-10-16T12:26:09.744 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:26:09.807 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:09.805+0000 7fd5cc790200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T12:26:09.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.877+0000 7ff8845c4700 1 Processor -- start 2023-10-16T12:26:09.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.877+0000 7ff8845c4700 1 -- start start 2023-10-16T12:26:09.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.878+0000 7ff8845c4700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff87c0cf880 0x7ff87c0b52f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:09.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.878+0000 7ff8845c4700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff87c15bc80 con 0x7ff87c0cf4a0 2023-10-16T12:26:09.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.878+0000 7ff8845c4700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff87c15b070 con 0x7ff87c0b5830 2023-10-16T12:26:09.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.878+0000 7ff8845c4700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff87c15b350 con 0x7ff87c0cf880 2023-10-16T12:26:09.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.878+0000 7ff882360700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff87c0cf4a0 0x7ff87c0726e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:56350/0 (socket says 172.21.15.16:56350) 2023-10-16T12:26:09.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.878+0000 7ff882360700 1 -- 172.21.15.16:0/3889804348 learned_addr learned my addr 172.21.15.16:0/3889804348 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:26:09.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.878+0000 7ff881b5f700 1 --2- 172.21.15.16:0/3889804348 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff87c0cf880 0x7ff87c0b52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:09.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.879+0000 7ff881b5f700 1 -- 172.21.15.16:0/3889804348 >> v1:172.21.15.49:6789/0 conn(0x7ff87c0b5830 legacy=0x7ff87c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:09.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.879+0000 7ff881b5f700 1 -- 172.21.15.16:0/3889804348 >> v1:172.21.15.16:6789/0 conn(0x7ff87c0cf4a0 legacy=0x7ff87c0726e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:09.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.879+0000 7ff88135e700 1 -- 172.21.15.16:0/3889804348 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3750507931 0 0) 0x7ff87c15b070 con 0x7ff87c0b5830 2023-10-16T12:26:09.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.879+0000 7ff881b5f700 1 -- 172.21.15.16:0/3889804348 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff87c15b630 con 0x7ff87c0cf880 2023-10-16T12:26:09.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.879+0000 7ff881b5f700 1 --2- 172.21.15.16:0/3889804348 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff87c0cf880 0x7ff87c0b52f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ff878009870 tx=0x7ff878009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87a530fc3dd37386 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:09.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.879+0000 7ff88135e700 1 -- 172.21.15.16:0/3889804348 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff878011830 con 0x7ff87c0cf880 2023-10-16T12:26:09.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.879+0000 7ff88135e700 1 -- 172.21.15.16:0/3889804348 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff878011990 con 0x7ff87c0cf880 2023-10-16T12:26:09.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.879+0000 7ff88135e700 1 -- 172.21.15.16:0/3889804348 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff878011d20 con 0x7ff87c0cf880 2023-10-16T12:26:09.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.880+0000 7ff8845c4700 1 -- 172.21.15.16:0/3889804348 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff87c0cf880 msgr2=0x7ff87c0b52f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:09.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.880+0000 7ff8845c4700 1 --2- 172.21.15.16:0/3889804348 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff87c0cf880 0x7ff87c0b52f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ff878009870 tx=0x7ff878009b80 comp rx=0 tx=0).stop 2023-10-16T12:26:09.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.880+0000 7ff8845c4700 1 -- 172.21.15.16:0/3889804348 shutdown_connections 2023-10-16T12:26:09.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.880+0000 7ff8845c4700 1 --2- 172.21.15.16:0/3889804348 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff87c0cf880 0x7ff87c0b52f0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:09.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.880+0000 7ff8845c4700 1 -- 172.21.15.16:0/3889804348 >> 172.21.15.16:0/3889804348 conn(0x7ff87c0b1fd0 msgr2=0x7ff87c0b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:09.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.880+0000 7ff8845c4700 1 -- 172.21.15.16:0/3889804348 shutdown_connections 2023-10-16T12:26:09.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.880+0000 7ff8845c4700 1 -- 172.21.15.16:0/3889804348 wait complete. 2023-10-16T12:26:09.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.881+0000 7ff8845c4700 1 Processor -- start 2023-10-16T12:26:09.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.881+0000 7ff8845c4700 1 -- start start 2023-10-16T12:26:09.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.882+0000 7ff8845c4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff87c0b5830 0x7ff87c152b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:09.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.882+0000 7ff8845c4700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff87c0cf4a0 0x7ff87c0c5510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:09.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.882+0000 7ff8845c4700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff87c0cf880 0x7ff87c06a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:09.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.882+0000 7ff8845c4700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff87c0c6ba0 con 0x7ff87c0b5830 2023-10-16T12:26:09.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.882+0000 7ff8845c4700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff87c0c5f90 con 0x7ff87c0cf880 2023-10-16T12:26:09.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.882+0000 7ff8845c4700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff87c0c6270 con 0x7ff87c0cf4a0 2023-10-16T12:26:09.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.882+0000 7ff882360700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff87c0b5830 0x7ff87c152b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:09.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.882+0000 7ff882360700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff87c0b5830 0x7ff87c152b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38862/0 (socket says 172.21.15.16:38862) 2023-10-16T12:26:09.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.882+0000 7ff882360700 1 -- 172.21.15.16:0/670232487 learned_addr learned my addr 172.21.15.16:0/670232487 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:09.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.882+0000 7ff881b5f700 1 --2- 172.21.15.16:0/670232487 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff87c0cf4a0 0x7ff87c0c5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:09.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.882+0000 7ff882b61700 1 --2- 172.21.15.16:0/670232487 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff87c0cf880 0x7ff87c06a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:09.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.883+0000 7ff881b5f700 1 -- 172.21.15.16:0/670232487 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff87c0cf880 msgr2=0x7ff87c06a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:09.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.883+0000 7ff881b5f700 1 --2- 172.21.15.16:0/670232487 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff87c0cf880 0x7ff87c06a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:09.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.883+0000 7ff881b5f700 1 -- 172.21.15.16:0/670232487 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff87c0b5830 msgr2=0x7ff87c152b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:09.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.883+0000 7ff881b5f700 1 --2- 172.21.15.16:0/670232487 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff87c0b5830 0x7ff87c152b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:09.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.883+0000 7ff881b5f700 1 -- 172.21.15.16:0/670232487 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8780094a0 con 0x7ff87c0cf4a0 2023-10-16T12:26:09.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.883+0000 7ff882360700 1 --2- 172.21.15.16:0/670232487 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff87c0b5830 0x7ff87c152b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:26:09.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.883+0000 7ff881b5f700 1 --2- 172.21.15.16:0/670232487 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff87c0cf4a0 0x7ff87c0c5510 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff878016000 tx=0x7ff878004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:09.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.883+0000 7ff872ffd700 1 -- 172.21.15.16:0/670232487 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8780036a0 con 0x7ff87c0cf4a0 2023-10-16T12:26:09.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.883+0000 7ff872ffd700 1 -- 172.21.15.16:0/670232487 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff87801f070 con 0x7ff87c0cf4a0 2023-10-16T12:26:09.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.883+0000 7ff872ffd700 1 -- 172.21.15.16:0/670232487 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff878003a40 con 0x7ff87c0cf4a0 2023-10-16T12:26:09.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.883+0000 7ff8845c4700 1 -- 172.21.15.16:0/670232487 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff87c0c6550 con 0x7ff87c0cf4a0 2023-10-16T12:26:09.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.883+0000 7ff8845c4700 1 -- 172.21.15.16:0/670232487 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff87c1530d0 con 0x7ff87c0cf4a0 2023-10-16T12:26:09.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.884+0000 7ff8845c4700 1 -- 172.21.15.16:0/670232487 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff87c04ebb0 con 0x7ff87c0cf4a0 2023-10-16T12:26:09.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.885+0000 7ff872ffd700 1 -- 172.21.15.16:0/670232487 <== mon.2 v2:172.21.15.49:3301/0 4 ==== mgrmap(e 28) v1 ==== 88940+0+0 (secure 0 0 0) 0x7ff878003ba0 con 0x7ff87c0cf4a0 2023-10-16T12:26:09.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.885+0000 7ff872ffd700 1 -- 172.21.15.16:0/670232487 <== mon.2 v2:172.21.15.49:3301/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6803+0+0 (secure 0 0 0) 0x7ff878021070 con 0x7ff87c0cf4a0 2023-10-16T12:26:09.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:09.890+0000 7ff872ffd700 1 -- 172.21.15.16:0/670232487 <== mon.2 v2:172.21.15.49:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff878055af0 con 0x7ff87c0cf4a0 2023-10-16T12:26:09.895 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:09.891+0000 7fd547fbc700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:26:09.895 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:09.891+0000 7fd547fbc700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:26:09.895 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:09.891+0000 7fd547fbc700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:26:09.895 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:09.891+0000 7fd547fbc700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:26:09.895 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:09.891+0000 7fd547fbc700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:26:09.896 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:09.891+0000 7fd544fb6700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:26:09.896 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:09.891+0000 7fd544fb6700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:26:09.896 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:09.891+0000 7fd544fb6700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:26:09.896 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:09.891+0000 7fd544fb6700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:26:09.896 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:09.891+0000 7fd544fb6700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:26:10.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.142+0000 7ff8845c4700 1 -- 172.21.15.16:0/670232487 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff87c0c6ce0 con 0x7ff87c0cf4a0 2023-10-16T12:26:10.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.143+0000 7ff872ffd700 1 -- 172.21.15.16:0/670232487 <== mon.2 v2:172.21.15.49:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+4913 (secure 0 0 0) 0x7ff878015090 con 0x7ff87c0cf4a0 2023-10-16T12:26:10.145 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:26:10.146 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5947,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/1103417847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":1103417847},{"type":"v1","addr":"172.21.15.49:6835","nonce":1103417847}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":6096,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/1380792179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1380792179},{"type":"v1","addr":"172.21.15.16:6836","nonce":1380792179}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":6099,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2238585443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":2238585443},{"type":"v1","addr":"172.21.15.16:6837","nonce":2238585443}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:24:02.695223+0000","modified":"2023-10-16T12:24:05.704039+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6111},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6111":{"gid":6111,"name":"b","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.49:6834/3246624425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":3246624425},{"type":"v1","addr":"172.21.15.49:6834","nonce":3246624425}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-10-16T12:26:10.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.146+0000 7ff8845c4700 1 -- 172.21.15.16:0/670232487 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff87c0cf4a0 msgr2=0x7ff87c0c5510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:10.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.146+0000 7ff8845c4700 1 --2- 172.21.15.16:0/670232487 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff87c0cf4a0 0x7ff87c0c5510 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff878016000 tx=0x7ff878004750 comp rx=0 tx=0).stop 2023-10-16T12:26:10.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.146+0000 7ff8845c4700 1 -- 172.21.15.16:0/670232487 shutdown_connections 2023-10-16T12:26:10.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.146+0000 7ff8845c4700 1 --2- 172.21.15.16:0/670232487 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff87c0cf880 0x7ff87c06a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:10.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.146+0000 7ff8845c4700 1 --2- 172.21.15.16:0/670232487 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff87c0b5830 0x7ff87c152b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:10.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.146+0000 7ff8845c4700 1 --2- 172.21.15.16:0/670232487 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff87c0cf4a0 0x7ff87c0c5510 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:10.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.146+0000 7ff8845c4700 1 -- 172.21.15.16:0/670232487 >> 172.21.15.16:0/670232487 conn(0x7ff87c0b1fd0 msgr2=0x7ff87c0afed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:10.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.146+0000 7ff8845c4700 1 -- 172.21.15.16:0/670232487 shutdown_connections 2023-10-16T12:26:10.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.146+0000 7ff8845c4700 1 -- 172.21.15.16:0/670232487 wait complete. 2023-10-16T12:26:10.150 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 27 2023-10-16T12:26:10.160 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:26:10.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.283+0000 7f1733e5c700 1 Processor -- start 2023-10-16T12:26:10.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.283+0000 7f1733e5c700 1 -- start start 2023-10-16T12:26:10.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.284+0000 7f1733e5c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f172c15d900 0x7f172c0d2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:10.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.284+0000 7f1733e5c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f172c0d2f70 0x7f172c0cdc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:10.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.284+0000 7f1733e5c700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f172c0ce1a0 con 0x7f172c0ce9f0 2023-10-16T12:26:10.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.284+0000 7f1733e5c700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f172c0cee70 con 0x7f172c0d2f70 2023-10-16T12:26:10.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.284+0000 7f1733e5c700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f172c0ce400 con 0x7f172c15d900 2023-10-16T12:26:10.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.284+0000 7f17323f9700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f172c0d2f70 0x7f172c0cdc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:10.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.284+0000 7f17323f9700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f172c0d2f70 0x7f172c0cdc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38900/0 (socket says 172.21.15.16:38900) 2023-10-16T12:26:10.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.284+0000 7f17323f9700 1 -- 172.21.15.16:0/2835661720 learned_addr learned my addr 172.21.15.16:0/2835661720 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:10.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.284+0000 7f1731bf8700 1 --2- 172.21.15.16:0/2835661720 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f172c15d900 0x7f172c0d2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:10.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.285+0000 7f17323f9700 1 -- 172.21.15.16:0/2835661720 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f172c15d900 msgr2=0x7f172c0d2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:10.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.285+0000 7f17323f9700 1 --2- 172.21.15.16:0/2835661720 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f172c15d900 0x7f172c0d2040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:10.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.285+0000 7f17323f9700 1 -- 172.21.15.16:0/2835661720 >> v1:172.21.15.49:6789/0 conn(0x7f172c0ce9f0 legacy=0x7f172c0d2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:10.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.285+0000 7f17323f9700 1 -- 172.21.15.16:0/2835661720 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f172c15dce0 con 0x7f172c0d2f70 2023-10-16T12:26:10.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.285+0000 7f1730bf6700 1 -- 172.21.15.16:0/2835661720 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3751649285 0 0) 0x7f172c0ce1a0 con 0x7f172c0ce9f0 2023-10-16T12:26:10.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.285+0000 7f17323f9700 1 --2- 172.21.15.16:0/2835661720 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f172c0d2f70 0x7f172c0cdc60 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f1724009e50 tx=0x7f172400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b21464f6e9925841 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:10.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.285+0000 7f1730bf6700 1 -- 172.21.15.16:0/2835661720 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f172400e070 con 0x7f172c0d2f70 2023-10-16T12:26:10.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.285+0000 7f1730bf6700 1 -- 172.21.15.16:0/2835661720 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f172400b910 con 0x7f172c0d2f70 2023-10-16T12:26:10.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.285+0000 7f1730bf6700 1 -- 172.21.15.16:0/2835661720 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f172400bcc0 con 0x7f172c0d2f70 2023-10-16T12:26:10.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.286+0000 7f1733e5c700 1 -- 172.21.15.16:0/2835661720 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f172c0d2f70 msgr2=0x7f172c0cdc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:10.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.286+0000 7f1733e5c700 1 --2- 172.21.15.16:0/2835661720 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f172c0d2f70 0x7f172c0cdc60 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f1724009e50 tx=0x7f172400b330 comp rx=0 tx=0).stop 2023-10-16T12:26:10.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.286+0000 7f1733e5c700 1 -- 172.21.15.16:0/2835661720 shutdown_connections 2023-10-16T12:26:10.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.286+0000 7f1733e5c700 1 --2- 172.21.15.16:0/2835661720 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f172c0d2f70 0x7f172c0cdc60 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:10.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.286+0000 7f1733e5c700 1 --2- 172.21.15.16:0/2835661720 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f172c15d900 0x7f172c0d2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:10.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.286+0000 7f1733e5c700 1 -- 172.21.15.16:0/2835661720 >> 172.21.15.16:0/2835661720 conn(0x7f172c072330 msgr2=0x7f172c072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:10.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.286+0000 7f1733e5c700 1 -- 172.21.15.16:0/2835661720 shutdown_connections 2023-10-16T12:26:10.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.286+0000 7f1733e5c700 1 -- 172.21.15.16:0/2835661720 wait complete. 2023-10-16T12:26:10.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.287+0000 7f1733e5c700 1 Processor -- start 2023-10-16T12:26:10.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.287+0000 7f1733e5c700 1 -- start start 2023-10-16T12:26:10.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.288+0000 7f1733e5c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f172c0ce9f0 0x7f172c1f2780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:10.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.288+0000 7f1733e5c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f172c0d2f70 0x7f172c1f6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:10.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.288+0000 7f1733e5c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f172c15d900 0x7f172c15ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:10.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.288+0000 7f1733e5c700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f172c15f480 con 0x7f172c0ce9f0 2023-10-16T12:26:10.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.288+0000 7f1733e5c700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f172c1f6d20 con 0x7f172c0d2f70 2023-10-16T12:26:10.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.288+0000 7f1733e5c700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f172c1f7000 con 0x7f172c15d900 2023-10-16T12:26:10.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.288+0000 7f1731bf8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f172c0ce9f0 0x7f172c1f2780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:10.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.288+0000 7f17313f7700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f172c0d2f70 0x7f172c1f6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:10.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.288+0000 7f1731bf8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f172c0ce9f0 0x7f172c1f2780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38902/0 (socket says 172.21.15.16:38902) 2023-10-16T12:26:10.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.288+0000 7f1731bf8700 1 -- 172.21.15.16:0/2764852656 learned_addr learned my addr 172.21.15.16:0/2764852656 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:10.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.289+0000 7f17323f9700 1 --2- 172.21.15.16:0/2764852656 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f172c15d900 0x7f172c15ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:10.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.289+0000 7f1731bf8700 1 -- 172.21.15.16:0/2764852656 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f172c15d900 msgr2=0x7f172c15ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:10.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.289+0000 7f1731bf8700 1 --2- 172.21.15.16:0/2764852656 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f172c15d900 0x7f172c15ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:10.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.289+0000 7f1731bf8700 1 -- 172.21.15.16:0/2764852656 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f172c0d2f70 msgr2=0x7f172c1f6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:10.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.289+0000 7f1731bf8700 1 --2- 172.21.15.16:0/2764852656 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f172c0d2f70 0x7f172c1f6420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:10.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.289+0000 7f1731bf8700 1 -- 172.21.15.16:0/2764852656 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1724009b30 con 0x7f172c0ce9f0 2023-10-16T12:26:10.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.289+0000 7f1731bf8700 1 --2- 172.21.15.16:0/2764852656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f172c0ce9f0 0x7f172c1f2780 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f172800e9b0 tx=0x7f172800ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:10.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.289+0000 7f171e7fc700 1 -- 172.21.15.16:0/2764852656 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1728009710 con 0x7f172c0ce9f0 2023-10-16T12:26:10.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.289+0000 7f171e7fc700 1 -- 172.21.15.16:0/2764852656 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f172800ee50 con 0x7f172c0ce9f0 2023-10-16T12:26:10.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.289+0000 7f171e7fc700 1 -- 172.21.15.16:0/2764852656 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f172800ca50 con 0x7f172c0ce9f0 2023-10-16T12:26:10.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.289+0000 7f1733e5c700 1 -- 172.21.15.16:0/2764852656 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f172c15f110 con 0x7f172c0ce9f0 2023-10-16T12:26:10.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.289+0000 7f1733e5c700 1 -- 172.21.15.16:0/2764852656 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f172c1f72e0 con 0x7f172c0ce9f0 2023-10-16T12:26:10.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.290+0000 7f1733e5c700 1 -- 172.21.15.16:0/2764852656 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f172c04ebb0 con 0x7f172c0ce9f0 2023-10-16T12:26:10.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.291+0000 7f171e7fc700 1 -- 172.21.15.16:0/2764852656 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 28) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f172800cbb0 con 0x7f172c0ce9f0 2023-10-16T12:26:10.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.291+0000 7f171e7fc700 1 -- 172.21.15.16:0/2764852656 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6803+0+0 (secure 0 0 0) 0x7f1728089490 con 0x7f172c0ce9f0 2023-10-16T12:26:10.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.296+0000 7f171e7fc700 1 -- 172.21.15.16:0/2764852656 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1728054540 con 0x7f172c0ce9f0 2023-10-16T12:26:10.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.501+0000 7f1733e5c700 1 -- 172.21.15.16:0/2764852656 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f172c157ba0 con 0x7f172c0ce9f0 2023-10-16T12:26:10.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.502+0000 7f171e7fc700 1 -- 172.21.15.16:0/2764852656 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v74) v1 ==== 74+0+17769 (secure 0 0 0) 0x7f172800ce60 con 0x7f172c0ce9f0 2023-10-16T12:26:10.505 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:26:10.505 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":74,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:26:09.819682+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:24:01.200494+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-10-16T12:24:01.652885+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":73,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.14","mappings":[{"from":0,"to":6}]},{"pgid":"8.1d","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/1907743445":"2023-10-17T12:26:09.819640+0000","172.21.15.16:0/1948269493":"2023-10-17T12:26:09.819640+0000","172.21.15.16:0/1785161614":"2023-10-17T12:26:09.819640+0000","172.21.15.16:0/491384204":"2023-10-17T12:26:09.819640+0000","172.21.15.16:6833/4075855741":"2023-10-17T12:26:09.819640+0000","172.21.15.16:6832/4075855741":"2023-10-17T12:26:09.819640+0000","172.21.15.16:0/754053746":"2023-10-17T12:26:09.819640+0000","172.21.15.16:0/2103819518":"2023-10-17T12:24:17.735687+0000","172.21.15.16:0/224613758":"2023-10-17T12:24:17.735687+0000","172.21.15.16:0/877922677":"2023-10-17T12:24:17.735687+0000","172.21.15.16:0/864162051":"2023-10-17T12:26:09.819640+0000","172.21.15.16:0/1135331895":"2023-10-17T12:24:17.735687+0000","172.21.15.16:0/1214553139":"2023-10-17T12:24:17.735687+0000","172.21.15.16:0/1240820252":"2023-10-17T12:24:17.735687+0000","172.21.15.16:6833/3870937344":"2023-10-17T12:24:17.735687+0000","172.21.15.16:6832/3870937344":"2023-10-17T12:24:17.735687+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:26:10.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.504+0000 7f1733e5c700 1 -- 172.21.15.16:0/2764852656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f172c0ce9f0 msgr2=0x7f172c1f2780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:10.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.504+0000 7f1733e5c700 1 --2- 172.21.15.16:0/2764852656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f172c0ce9f0 0x7f172c1f2780 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f172800e9b0 tx=0x7f172800ecc0 comp rx=0 tx=0).stop 2023-10-16T12:26:10.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.505+0000 7f1733e5c700 1 -- 172.21.15.16:0/2764852656 shutdown_connections 2023-10-16T12:26:10.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.505+0000 7f1733e5c700 1 --2- 172.21.15.16:0/2764852656 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f172c15d900 0x7f172c15ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:10.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.505+0000 7f1733e5c700 1 --2- 172.21.15.16:0/2764852656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f172c0ce9f0 0x7f172c1f2780 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:10.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.505+0000 7f1733e5c700 1 --2- 172.21.15.16:0/2764852656 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f172c0d2f70 0x7f172c1f6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:10.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.505+0000 7f1733e5c700 1 -- 172.21.15.16:0/2764852656 >> 172.21.15.16:0/2764852656 conn(0x7f172c072330 msgr2=0x7f172c15c790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:10.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.505+0000 7f1733e5c700 1 -- 172.21.15.16:0/2764852656 shutdown_connections 2023-10-16T12:26:10.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.505+0000 7f1733e5c700 1 -- 172.21.15.16:0/2764852656 wait complete. 2023-10-16T12:26:10.519 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-16T12:26:10.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.643+0000 7fd176edf700 1 Processor -- start 2023-10-16T12:26:10.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.644+0000 7fd176edf700 1 -- start start 2023-10-16T12:26:10.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.644+0000 7fd176edf700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd17015d230 0x7fd1700b5a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:10.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.644+0000 7fd176edf700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd17015bc80 con 0x7fd1700cf880 2023-10-16T12:26:10.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.644+0000 7fd176edf700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd17015b070 con 0x7fd1700cf4a0 2023-10-16T12:26:10.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.644+0000 7fd176edf700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd17015b350 con 0x7fd17015d230 2023-10-16T12:26:10.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.644+0000 7fd16ffff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd1700cf880 0x7fd1700b52f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:56414/0 (socket says 172.21.15.16:56414) 2023-10-16T12:26:10.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.644+0000 7fd16ffff700 1 -- 172.21.15.16:0/3653119911 learned_addr learned my addr 172.21.15.16:0/3653119911 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:26:10.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.644+0000 7fd17547c700 1 --2- 172.21.15.16:0/3653119911 >> v2:172.21.15.16:3300/0 conn(0x7fd17015d230 0x7fd1700b5a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:10.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.645+0000 7fd16f7fe700 1 -- 172.21.15.16:0/3653119911 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4071969847 0 0) 0x7fd17015bc80 con 0x7fd1700cf880 2023-10-16T12:26:10.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.645+0000 7fd16f7fe700 1 -- 172.21.15.16:0/3653119911 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1600035b0 con 0x7fd1700cf880 2023-10-16T12:26:10.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.645+0000 7fd17547c700 1 -- 172.21.15.16:0/3653119911 >> v1:172.21.15.49:6789/0 conn(0x7fd1700cf4a0 legacy=0x7fd1700726e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:10.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.645+0000 7fd17547c700 1 -- 172.21.15.16:0/3653119911 >> v1:172.21.15.16:6789/0 conn(0x7fd1700cf880 legacy=0x7fd1700b52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:10.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.645+0000 7fd17547c700 1 -- 172.21.15.16:0/3653119911 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd17015b630 con 0x7fd17015d230 2023-10-16T12:26:10.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.645+0000 7fd17547c700 1 --2- 172.21.15.16:0/3653119911 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd17015d230 0x7fd1700b5a00 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fd168009d90 tx=0x7fd16800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da621d42d5592edd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:10.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.645+0000 7fd16f7fe700 1 -- 172.21.15.16:0/3653119911 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd16800e070 con 0x7fd17015d230 2023-10-16T12:26:10.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.645+0000 7fd16f7fe700 1 -- 172.21.15.16:0/3653119911 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd16800bb20 con 0x7fd17015d230 2023-10-16T12:26:10.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.645+0000 7fd16f7fe700 1 -- 172.21.15.16:0/3653119911 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd168012430 con 0x7fd17015d230 2023-10-16T12:26:10.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.646+0000 7fd176edf700 1 -- 172.21.15.16:0/3653119911 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd17015d230 msgr2=0x7fd1700b5a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:10.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.646+0000 7fd176edf700 1 --2- 172.21.15.16:0/3653119911 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd17015d230 0x7fd1700b5a00 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fd168009d90 tx=0x7fd16800b330 comp rx=0 tx=0).stop 2023-10-16T12:26:10.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.646+0000 7fd176edf700 1 -- 172.21.15.16:0/3653119911 shutdown_connections 2023-10-16T12:26:10.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.646+0000 7fd176edf700 1 --2- 172.21.15.16:0/3653119911 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd17015d230 0x7fd1700b5a00 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:10.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.646+0000 7fd176edf700 1 -- 172.21.15.16:0/3653119911 >> 172.21.15.16:0/3653119911 conn(0x7fd1700b1fd0 msgr2=0x7fd1700b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:10.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.646+0000 7fd176edf700 1 -- 172.21.15.16:0/3653119911 shutdown_connections 2023-10-16T12:26:10.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.646+0000 7fd176edf700 1 -- 172.21.15.16:0/3653119911 wait complete. 2023-10-16T12:26:10.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.647+0000 7fd176edf700 1 Processor -- start 2023-10-16T12:26:10.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.647+0000 7fd176edf700 1 -- start start 2023-10-16T12:26:10.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.648+0000 7fd176edf700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd1700cf4a0 0x7fd1700c6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:10.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.648+0000 7fd176edf700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1700cf880 0x7fd1700b0470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:10.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.648+0000 7fd176edf700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd17015d230 0x7fd1700b09b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:10.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.648+0000 7fd176edf700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1700c9310 con 0x7fd1700cf880 2023-10-16T12:26:10.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.648+0000 7fd176edf700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1700c8700 con 0x7fd1700cf4a0 2023-10-16T12:26:10.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.648+0000 7fd176edf700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1700c89e0 con 0x7fd17015d230 2023-10-16T12:26:10.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.648+0000 7fd16ffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1700cf880 0x7fd1700b0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:10.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.648+0000 7fd16ffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1700cf880 0x7fd1700b0470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38916/0 (socket says 172.21.15.16:38916) 2023-10-16T12:26:10.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.648+0000 7fd16ffff700 1 -- 172.21.15.16:0/1792320298 learned_addr learned my addr 172.21.15.16:0/1792320298 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:10.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.648+0000 7fd174c7b700 1 --2- 172.21.15.16:0/1792320298 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd1700cf4a0 0x7fd1700c6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:10.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.648+0000 7fd17547c700 1 --2- 172.21.15.16:0/1792320298 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd17015d230 0x7fd1700b09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:10.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.648+0000 7fd16ffff700 1 -- 172.21.15.16:0/1792320298 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd17015d230 msgr2=0x7fd1700b09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:10.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.648+0000 7fd16ffff700 1 --2- 172.21.15.16:0/1792320298 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd17015d230 0x7fd1700b09b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:10.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.648+0000 7fd16ffff700 1 -- 172.21.15.16:0/1792320298 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd1700cf4a0 msgr2=0x7fd1700c6100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:10.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.648+0000 7fd16ffff700 1 --2- 172.21.15.16:0/1792320298 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd1700cf4a0 0x7fd1700c6100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:10.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.648+0000 7fd16ffff700 1 -- 172.21.15.16:0/1792320298 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd168009a70 con 0x7fd1700cf880 2023-10-16T12:26:10.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.649+0000 7fd16ffff700 1 --2- 172.21.15.16:0/1792320298 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1700cf880 0x7fd1700b0470 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fd15c00c700 tx=0x7fd15c00ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:10.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.649+0000 7fd16d7fa700 1 -- 172.21.15.16:0/1792320298 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd15c005d70 con 0x7fd1700cf880 2023-10-16T12:26:10.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.649+0000 7fd16d7fa700 1 -- 172.21.15.16:0/1792320298 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd15c0070d0 con 0x7fd1700cf880 2023-10-16T12:26:10.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.649+0000 7fd16d7fa700 1 -- 172.21.15.16:0/1792320298 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd15c007460 con 0x7fd1700cf880 2023-10-16T12:26:10.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.649+0000 7fd176edf700 1 -- 172.21.15.16:0/1792320298 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1700c8cc0 con 0x7fd1700cf880 2023-10-16T12:26:10.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.649+0000 7fd176edf700 1 -- 172.21.15.16:0/1792320298 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1700c6640 con 0x7fd1700cf880 2023-10-16T12:26:10.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.650+0000 7fd176edf700 1 -- 172.21.15.16:0/1792320298 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd17004ebb0 con 0x7fd1700cf880 2023-10-16T12:26:10.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.650+0000 7fd16d7fa700 1 -- 172.21.15.16:0/1792320298 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 28) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fd15c0075c0 con 0x7fd1700cf880 2023-10-16T12:26:10.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.651+0000 7fd16d7fa700 1 -- 172.21.15.16:0/1792320298 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6803+0+0 (secure 0 0 0) 0x7fd15c088a00 con 0x7fd1700cf880 2023-10-16T12:26:10.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.655+0000 7fd16d7fa700 1 -- 172.21.15.16:0/1792320298 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd15c0539c0 con 0x7fd1700cf880 2023-10-16T12:26:10.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.829+0000 7fd16d7fa700 1 -- 172.21.15.16:0/1792320298 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 29) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fd15c053470 con 0x7fd1700cf880 2023-10-16T12:26:10.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.829+0000 7fd16d7fa700 1 --2- 172.21.15.16:0/1792320298 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7fd154070cb0 0x7fd1540730a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:10.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.830+0000 7fd174c7b700 1 --2- 172.21.15.16:0/1792320298 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7fd154070cb0 0x7fd1540730a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:10.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.832+0000 7fd174c7b700 1 --2- 172.21.15.16:0/1792320298 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7fd154070cb0 0x7fd1540730a0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fd164007fd0 tx=0x7fd164009040 comp rx=0 tx=0).ready entity=mgr.6396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:10.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.915+0000 7fd176edf700 1 -- 172.21.15.16:0/1792320298 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fd1700b0090 con 0x7fd1700cf880 2023-10-16T12:26:10.917 INFO:tasks.ceph.mon.a.smithi016.stderr:2023-10-16T12:26:10.915+0000 7f9fc8535700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-16T12:26:10.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.916+0000 7fd16d7fa700 1 -- 172.21.15.16:0/1792320298 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v28) v1 ==== 161+0+0 (secure 0 0 0) 0x7fd15c086e60 con 0x7fd1700cf880 2023-10-16T12:26:10.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.919+0000 7fd176edf700 1 -- 172.21.15.16:0/1792320298 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7fd154070cb0 msgr2=0x7fd1540730a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:10.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.919+0000 7fd176edf700 1 --2- 172.21.15.16:0/1792320298 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7fd154070cb0 0x7fd1540730a0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fd164007fd0 tx=0x7fd164009040 comp rx=0 tx=0).stop 2023-10-16T12:26:10.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.919+0000 7fd176edf700 1 -- 172.21.15.16:0/1792320298 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1700cf880 msgr2=0x7fd1700b0470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:10.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.919+0000 7fd176edf700 1 --2- 172.21.15.16:0/1792320298 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1700cf880 0x7fd1700b0470 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fd15c00c700 tx=0x7fd15c00ca10 comp rx=0 tx=0).stop 2023-10-16T12:26:10.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.919+0000 7fd176edf700 1 -- 172.21.15.16:0/1792320298 shutdown_connections 2023-10-16T12:26:10.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.919+0000 7fd176edf700 1 --2- 172.21.15.16:0/1792320298 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7fd154070cb0 0x7fd1540730a0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:10.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.919+0000 7fd176edf700 1 --2- 172.21.15.16:0/1792320298 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd17015d230 0x7fd1700b09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:10.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.919+0000 7fd176edf700 1 --2- 172.21.15.16:0/1792320298 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd1700cf4a0 0x7fd1700c6100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:10.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.919+0000 7fd176edf700 1 --2- 172.21.15.16:0/1792320298 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1700cf880 0x7fd1700b0470 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:10.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.919+0000 7fd176edf700 1 -- 172.21.15.16:0/1792320298 >> 172.21.15.16:0/1792320298 conn(0x7fd1700b1fd0 msgr2=0x7fd17006d9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:10.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.919+0000 7fd176edf700 1 -- 172.21.15.16:0/1792320298 shutdown_connections 2023-10-16T12:26:10.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:10.919+0000 7fd176edf700 1 -- 172.21.15.16:0/1792320298 wait complete. 2023-10-16T12:26:10.924 INFO:teuthology.orchestra.run.smithi016.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-16T12:26:10.934 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-16T12:26:10.959 INFO:tasks.ceph.mds.b.smithi049.stdout:starting mds.b at 2023-10-16T12:26:11.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:11.059+0000 7f75b68c1700 1 Processor -- start 2023-10-16T12:26:11.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:11.059+0000 7f75b68c1700 1 -- start start 2023-10-16T12:26:11.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:11.059+0000 7f75b68c1700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f75b015ec80 0x7f75b00cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:11.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:11.059+0000 7f75b68c1700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f75b015f060 0x7f75b00ce400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:11.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:11.059+0000 7f75b68c1700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75b00b2440 con 0x7f75b00d2110 2023-10-16T12:26:11.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:11.059+0000 7f75b68c1700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75b00b2c50 con 0x7f75b015ec80 2023-10-16T12:26:11.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:11.059+0000 7f75b68c1700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75b00d00e0 con 0x7f75b015f060 2023-10-16T12:26:11.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:11.060+0000 7f75affff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f75b015ec80 0x7f75b00cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:11.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:11.060+0000 7f75b4e5e700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f75b00d2110 0x7f75b00b4440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:56454/0 (socket says 172.21.15.16:56454) 2023-10-16T12:26:11.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:11.060+0000 7f75b4e5e700 1 -- 172.21.15.16:0/1467437044 learned_addr learned my addr 172.21.15.16:0/1467437044 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:26:11.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:11.060+0000 7f75af7fe700 1 --2- 172.21.15.16:0/1467437044 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f75b015f060 0x7f75b00ce400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:11.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:11.060+0000 7f75affff700 1 -- 172.21.15.16:0/1467437044 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f75b015f060 msgr2=0x7f75b00ce400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:11.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:11.060+0000 7f75affff700 1 --2- 172.21.15.16:0/1467437044 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f75b015f060 0x7f75b00ce400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:11.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:11.060+0000 7f75affff700 1 -- 172.21.15.16:0/1467437044 >> v1:172.21.15.16:6789/0 conn(0x7f75b00d2110 legacy=0x7f75b00b4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:11.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:11.060+0000 7f75aeffd700 1 -- 172.21.15.16:0/1467437044 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 741439234 0 0) 0x7f75b00b2440 con 0x7f75b00d2110 2023-10-16T12:26:11.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:11.060+0000 7f75affff700 1 -- 172.21.15.16:0/1467437044 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75b00cef60 con 0x7f75b015ec80 2023-10-16T12:26:11.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:11.060+0000 7f75affff700 1 --2- 172.21.15.16:0/1467437044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75b015ec80 0x7f75b00cdec0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f75a0009840 tx=0x7f75a0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70eaabc4b9bcf05e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:11.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:11.060+0000 7f75aeffd700 1 -- 172.21.15.16:0/1467437044 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75a0011830 con 0x7f75b015ec80 2023-10-16T12:26:11.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:11.060+0000 7f75aeffd700 1 -- 172.21.15.16:0/1467437044 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75a0011990 con 0x7f75b015ec80 2023-10-16T12:26:11.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:11.060+0000 7f75aeffd700 1 -- 172.21.15.16:0/1467437044 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75a0011d20 con 0x7f75b015ec80 2023-10-16T12:26:11.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:11.061+0000 7f75b68c1700 1 -- 172.21.15.16:0/1467437044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75b015ec80 msgr2=0x7f75b00cdec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:11.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:11.061+0000 7f75b68c1700 1 --2- 172.21.15.16:0/1467437044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75b015ec80 0x7f75b00cdec0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f75a0009840 tx=0x7f75a0009b50 comp rx=0 tx=0).stop 2023-10-16T12:26:11.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:11.061+0000 7f75b68c1700 1 -- 172.21.15.16:0/1467437044 shutdown_connections 2023-10-16T12:26:11.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:11.061+0000 7f75b68c1700 1 --2- 172.21.15.16:0/1467437044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75b015ec80 0x7f75b00cdec0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:11.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:11.061+0000 7f75b68c1700 1 --2- 172.21.15.16:0/1467437044 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f75b015f060 0x7f75b00ce400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:11.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:11.061+0000 7f75b68c1700 1 -- 172.21.15.16:0/1467437044 >> 172.21.15.16:0/1467437044 conn(0x7f75b00d15f0 msgr2=0x7f75b00d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:11.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:11.061+0000 7f75b68c1700 1 -- 172.21.15.16:0/1467437044 shutdown_connections 2023-10-16T12:26:11.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:11.061+0000 7f75b68c1700 1 -- 172.21.15.16:0/1467437044 wait complete. 2023-10-16T12:26:11.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:11.062+0000 7f75b68c1700 1 Processor -- start 2023-10-16T12:26:11.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:11.062+0000 7f75b68c1700 1 -- start start 2023-10-16T12:26:11.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:11.063+0000 7f75b68c1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75b00d2110 0x7f75b0154440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:11.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:11.063+0000 7f75b68c1700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f75b015ec80 0x7f75b01e41a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:11.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:11.063+0000 7f75b68c1700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f75b015f060 0x7f75b01e4a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:11.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:11.063+0000 7f75b68c1700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75b015a620 con 0x7f75b00d2110 2023-10-16T12:26:11.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:11.063+0000 7f75b68c1700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75b01e46e0 con 0x7f75b015ec80 2023-10-16T12:26:11.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:11.063+0000 7f75b68c1700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75b0159730 con 0x7f75b015f060 2023-10-16T12:26:11.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:11.063+0000 7f75affff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75b00d2110 0x7f75b0154440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:11.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:11.063+0000 7f75affff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75b00d2110 0x7f75b0154440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38952/0 (socket says 172.21.15.16:38952) 2023-10-16T12:26:11.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:11.063+0000 7f75affff700 1 -- 172.21.15.16:0/211456960 learned_addr learned my addr 172.21.15.16:0/211456960 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:11.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:11.063+0000 7f75af7fe700 1 --2- 172.21.15.16:0/211456960 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f75b015ec80 0x7f75b01e41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:11.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:11.063+0000 7f75b4e5e700 1 --2- 172.21.15.16:0/211456960 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f75b015f060 0x7f75b01e4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:11.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:11.063+0000 7f75affff700 1 -- 172.21.15.16:0/211456960 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f75b015f060 msgr2=0x7f75b01e4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:11.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:11.063+0000 7f75affff700 1 --2- 172.21.15.16:0/211456960 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f75b015f060 0x7f75b01e4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:11.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:11.063+0000 7f75affff700 1 -- 172.21.15.16:0/211456960 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f75b015ec80 msgr2=0x7f75b01e41a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:11.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:11.063+0000 7f75affff700 1 --2- 172.21.15.16:0/211456960 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f75b015ec80 0x7f75b01e41a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:11.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:11.063+0000 7f75affff700 1 -- 172.21.15.16:0/211456960 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75a00094a0 con 0x7f75b00d2110 2023-10-16T12:26:11.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:11.064+0000 7f75b4e5e700 1 --2- 172.21.15.16:0/211456960 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f75b015f060 0x7f75b01e4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:26:11.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:11.064+0000 7f75affff700 1 --2- 172.21.15.16:0/211456960 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75b00d2110 0x7f75b0154440 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f75a00090b0 tx=0x7f75a0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:11.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:11.064+0000 7f75acff9700 1 -- 172.21.15.16:0/211456960 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75a00038a0 con 0x7f75b00d2110 2023-10-16T12:26:11.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:11.064+0000 7f75acff9700 1 -- 172.21.15.16:0/211456960 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75a001f070 con 0x7f75b00d2110 2023-10-16T12:26:11.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:11.064+0000 7f75b68c1700 1 -- 172.21.15.16:0/211456960 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75b0159a10 con 0x7f75b00d2110 2023-10-16T12:26:11.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:11.064+0000 7f75acff9700 1 -- 172.21.15.16:0/211456960 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75a0003c20 con 0x7f75b00d2110 2023-10-16T12:26:11.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:11.064+0000 7f75b68c1700 1 -- 172.21.15.16:0/211456960 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75b0159fd0 con 0x7f75b00d2110 2023-10-16T12:26:11.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:11.065+0000 7f75b68c1700 1 -- 172.21.15.16:0/211456960 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75b004ebb0 con 0x7f75b00d2110 2023-10-16T12:26:11.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:11.065+0000 7f75acff9700 1 -- 172.21.15.16:0/211456960 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 29) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f75a001a430 con 0x7f75b00d2110 2023-10-16T12:26:11.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:11.066+0000 7f75acff9700 1 --2- 172.21.15.16:0/211456960 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f759c06ad50 0x7f759c06d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:11.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:11.066+0000 7f75acff9700 1 -- 172.21.15.16:0/211456960 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6889+0+0 (secure 0 0 0) 0x7f75a0021070 con 0x7f75b00d2110 2023-10-16T12:26:11.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:11.066+0000 7f75af7fe700 1 --2- 172.21.15.16:0/211456960 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f759c06ad50 0x7f759c06d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:11.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:11.066+0000 7f75af7fe700 1 --2- 172.21.15.16:0/211456960 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f759c06ad50 0x7f759c06d210 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f75b00c53b0 tx=0x7f75a4008040 comp rx=0 tx=0).ready entity=mgr.6396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:11.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:11.070+0000 7f75acff9700 1 -- 172.21.15.16:0/211456960 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f75a0055f90 con 0x7f75b00d2110 2023-10-16T12:26:11.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:11.326+0000 7f75b68c1700 1 -- 172.21.15.16:0/211456960 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f75b0071360 con 0x7f75b00d2110 2023-10-16T12:26:11.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:11.918+0000 7f75acff9700 1 -- 172.21.15.16:0/211456960 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v29) v1 ==== 104+0+0 (secure 0 0 0) 0x7f75a00150a0 con 0x7f75b00d2110 2023-10-16T12:26:11.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:11.921+0000 7f75b68c1700 1 -- 172.21.15.16:0/211456960 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f759c06ad50 msgr2=0x7f759c06d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:11.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:11.921+0000 7f75b68c1700 1 --2- 172.21.15.16:0/211456960 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f759c06ad50 0x7f759c06d210 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f75b00c53b0 tx=0x7f75a4008040 comp rx=0 tx=0).stop 2023-10-16T12:26:11.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:11.921+0000 7f75b68c1700 1 -- 172.21.15.16:0/211456960 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75b00d2110 msgr2=0x7f75b0154440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:11.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:11.921+0000 7f75b68c1700 1 --2- 172.21.15.16:0/211456960 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75b00d2110 0x7f75b0154440 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f75a00090b0 tx=0x7f75a0004750 comp rx=0 tx=0).stop 2023-10-16T12:26:11.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:11.921+0000 7f75b68c1700 1 -- 172.21.15.16:0/211456960 shutdown_connections 2023-10-16T12:26:11.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:11.921+0000 7f75b68c1700 1 --2- 172.21.15.16:0/211456960 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f759c06ad50 0x7f759c06d210 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:11.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:11.921+0000 7f75b68c1700 1 --2- 172.21.15.16:0/211456960 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f75b015f060 0x7f75b01e4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:11.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:11.921+0000 7f75b68c1700 1 --2- 172.21.15.16:0/211456960 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75b00d2110 0x7f75b0154440 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:11.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:11.921+0000 7f75b68c1700 1 --2- 172.21.15.16:0/211456960 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f75b015ec80 0x7f75b01e41a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:11.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:11.921+0000 7f75b68c1700 1 -- 172.21.15.16:0/211456960 >> 172.21.15.16:0/211456960 conn(0x7f75b00d15f0 msgr2=0x7f75b00cc190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:11.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:11.921+0000 7f75b68c1700 1 -- 172.21.15.16:0/211456960 shutdown_connections 2023-10-16T12:26:11.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:11.921+0000 7f75b68c1700 1 -- 172.21.15.16:0/211456960 wait complete. 2023-10-16T12:26:11.938 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-16T12:26:12.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.071+0000 7f4187ea6700 1 Processor -- start 2023-10-16T12:26:12.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.071+0000 7f4187ea6700 1 -- start start 2023-10-16T12:26:12.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.072+0000 7f4187ea6700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4180160e60 0x7f418015e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:12.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.072+0000 7f4187ea6700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f418006d980 0x7f418015efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:12.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.072+0000 7f4187ea6700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41800cdeb0 con 0x7f418006a7e0 2023-10-16T12:26:12.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.072+0000 7f4187ea6700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41801613e0 con 0x7f418006d980 2023-10-16T12:26:12.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.072+0000 7f4187ea6700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41801582d0 con 0x7f4180160e60 2023-10-16T12:26:12.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.072+0000 7f4186443700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f418006a7e0 0x7f418006abc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:56482/0 (socket says 172.21.15.16:56482) 2023-10-16T12:26:12.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.072+0000 7f4186443700 1 -- 172.21.15.16:0/975926578 learned_addr learned my addr 172.21.15.16:0/975926578 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:26:12.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.072+0000 7f4185441700 1 --2- 172.21.15.16:0/975926578 >> v2:172.21.15.49:3300/0 conn(0x7f418006d980 0x7f418015efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:12.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.072+0000 7f4185c42700 1 --2- 172.21.15.16:0/975926578 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4180160e60 0x7f418015e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:12.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.072+0000 7f4184c40700 1 -- 172.21.15.16:0/975926578 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 769102870 0 0) 0x7f41800cdeb0 con 0x7f418006a7e0 2023-10-16T12:26:12.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.072+0000 7f4184c40700 1 -- 172.21.15.16:0/975926578 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f416c0035b0 con 0x7f418006a7e0 2023-10-16T12:26:12.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.073+0000 7f4184c40700 1 -- 172.21.15.16:0/975926578 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3972635398 0 0) 0x7f416c0035b0 con 0x7f418006a7e0 2023-10-16T12:26:12.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.073+0000 7f4184c40700 1 -- 172.21.15.16:0/975926578 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4180160e60 msgr2=0x7f418015e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:12.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.073+0000 7f4184c40700 1 --2- 172.21.15.16:0/975926578 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4180160e60 0x7f418015e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:12.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.073+0000 7f4184c40700 1 -- 172.21.15.16:0/975926578 >> v2:172.21.15.49:3300/0 conn(0x7f418006d980 msgr2=0x7f418015efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:12.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.073+0000 7f4184c40700 1 --2- 172.21.15.16:0/975926578 >> v2:172.21.15.49:3300/0 conn(0x7f418006d980 0x7f418015efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:12.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.073+0000 7f4184c40700 1 -- 172.21.15.16:0/975926578 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f418015edb0 con 0x7f418006a7e0 2023-10-16T12:26:12.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.073+0000 7f4184c40700 1 -- 172.21.15.16:0/975926578 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f41780019f0 con 0x7f418006a7e0 2023-10-16T12:26:12.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.073+0000 7f4184c40700 1 -- 172.21.15.16:0/975926578 >> v1:172.21.15.16:6789/0 conn(0x7f418006a7e0 legacy=0x7f418006abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:12.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.073+0000 7f4184c40700 1 --2- 172.21.15.16:0/975926578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f416c003a00 0x7f416c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:12.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.073+0000 7f4184c40700 1 --2- 172.21.15.16:0/975926578 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f416c004850 0x7f416c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:12.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.073+0000 7f4184c40700 1 --2- 172.21.15.16:0/975926578 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4180160e60 0x7f416c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:12.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.073+0000 7f4184c40700 1 -- 172.21.15.16:0/975926578 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41801613e0 con 0x7f416c003a00 2023-10-16T12:26:12.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.073+0000 7f4184c40700 1 -- 172.21.15.16:0/975926578 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41801582d0 con 0x7f416c004850 2023-10-16T12:26:12.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.073+0000 7f4184c40700 1 -- 172.21.15.16:0/975926578 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f416c00ba10 con 0x7f4180160e60 2023-10-16T12:26:12.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.073+0000 7f4185c42700 1 --2- 172.21.15.16:0/975926578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f416c003a00 0x7f416c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:12.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.073+0000 7f4186443700 1 --2- 172.21.15.16:0/975926578 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4180160e60 0x7f416c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:12.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.073+0000 7f4185441700 1 --2- 172.21.15.16:0/975926578 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f416c004850 0x7f416c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:12.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.073+0000 7f4185c42700 1 -- 172.21.15.16:0/975926578 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4180160e60 msgr2=0x7f416c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:12.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.073+0000 7f4185c42700 1 --2- 172.21.15.16:0/975926578 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4180160e60 0x7f416c005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:12.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.073+0000 7f4185c42700 1 -- 172.21.15.16:0/975926578 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f416c004850 msgr2=0x7f416c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:12.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.073+0000 7f4185c42700 1 --2- 172.21.15.16:0/975926578 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f416c004850 0x7f416c004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:12.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.073+0000 7f4185c42700 1 -- 172.21.15.16:0/975926578 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f416c0035b0 con 0x7f416c003a00 2023-10-16T12:26:12.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.073+0000 7f4185441700 1 --2- 172.21.15.16:0/975926578 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f416c004850 0x7f416c004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:26:12.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.073+0000 7f4185c42700 1 --2- 172.21.15.16:0/975926578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f416c003a00 0x7f416c003e20 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f4170009b60 tx=0x7f416c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dcf729a259e4b85 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:12.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.073+0000 7f4184c40700 1 -- 172.21.15.16:0/975926578 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f417000e6c0 con 0x7f416c003a00 2023-10-16T12:26:12.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.074+0000 7f4184c40700 1 -- 172.21.15.16:0/975926578 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f417000e820 con 0x7f416c003a00 2023-10-16T12:26:12.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.074+0000 7f4184c40700 1 -- 172.21.15.16:0/975926578 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f417000eb90 con 0x7f416c003a00 2023-10-16T12:26:12.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.074+0000 7f4187ea6700 1 -- 172.21.15.16:0/975926578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f416c003a00 msgr2=0x7f416c003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:12.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.074+0000 7f4187ea6700 1 --2- 172.21.15.16:0/975926578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f416c003a00 0x7f416c003e20 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f4170009b60 tx=0x7f416c005880 comp rx=0 tx=0).stop 2023-10-16T12:26:12.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.074+0000 7f4186443700 1 -- 172.21.15.16:0/975926578 reap_dead start 2023-10-16T12:26:12.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.074+0000 7f4187ea6700 1 -- 172.21.15.16:0/975926578 shutdown_connections 2023-10-16T12:26:12.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.074+0000 7f4187ea6700 1 -- 172.21.15.16:0/975926578 >> 172.21.15.16:0/975926578 conn(0x7f418006f430 msgr2=0x7f418006f810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:12.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.074+0000 7f4187ea6700 1 -- 172.21.15.16:0/975926578 shutdown_connections 2023-10-16T12:26:12.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.074+0000 7f4187ea6700 1 -- 172.21.15.16:0/975926578 wait complete. 2023-10-16T12:26:12.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.075+0000 7f4187ea6700 1 Processor -- start 2023-10-16T12:26:12.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.075+0000 7f4187ea6700 1 -- start start 2023-10-16T12:26:12.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.076+0000 7f4187ea6700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f416c003a00 0x7f41800c6760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:12.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.076+0000 7f4187ea6700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f41800ac8d0 0x7f41801e64f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:12.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.076+0000 7f4187ea6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41801e6a30 0x7f41801e6ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:12.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.076+0000 7f4187ea6700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41800ad120 con 0x7f41801e6a30 2023-10-16T12:26:12.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.076+0000 7f4187ea6700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41800b4070 con 0x7f416c003a00 2023-10-16T12:26:12.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.076+0000 7f4187ea6700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41800ace40 con 0x7f41800ac8d0 2023-10-16T12:26:12.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.076+0000 7f4186443700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41801e6a30 0x7f41801e6ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:12.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.076+0000 7f4186443700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41801e6a30 0x7f41801e6ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38992/0 (socket says 172.21.15.16:38992) 2023-10-16T12:26:12.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.076+0000 7f4186443700 1 -- 172.21.15.16:0/3549319974 learned_addr learned my addr 172.21.15.16:0/3549319974 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:12.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.076+0000 7f4185c42700 1 --2- 172.21.15.16:0/3549319974 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f416c003a00 0x7f41800c6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:12.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.076+0000 7f4185441700 1 --2- 172.21.15.16:0/3549319974 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f41800ac8d0 0x7f41801e64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:12.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.077+0000 7f4186443700 1 -- 172.21.15.16:0/3549319974 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f41800ac8d0 msgr2=0x7f41801e64f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:12.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.077+0000 7f4186443700 1 --2- 172.21.15.16:0/3549319974 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f41800ac8d0 0x7f41801e64f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:12.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.077+0000 7f4186443700 1 -- 172.21.15.16:0/3549319974 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f416c003a00 msgr2=0x7f41800c6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:12.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.077+0000 7f4186443700 1 --2- 172.21.15.16:0/3549319974 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f416c003a00 0x7f41800c6760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:12.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.077+0000 7f4186443700 1 -- 172.21.15.16:0/3549319974 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41800b3a20 con 0x7f41801e6a30 2023-10-16T12:26:12.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.077+0000 7f4186443700 1 --2- 172.21.15.16:0/3549319974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41801e6a30 0x7f41801e6ec0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f41800cdd20 tx=0x7f417800a600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:12.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.077+0000 7f41767fc700 1 -- 172.21.15.16:0/3549319974 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f417800ac60 con 0x7f41801e6a30 2023-10-16T12:26:12.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.077+0000 7f41767fc700 1 -- 172.21.15.16:0/3549319974 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f417800ade0 con 0x7f41801e6a30 2023-10-16T12:26:12.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.077+0000 7f4187ea6700 1 -- 172.21.15.16:0/3549319974 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41800b3d00 con 0x7f41801e6a30 2023-10-16T12:26:12.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.077+0000 7f41767fc700 1 -- 172.21.15.16:0/3549319974 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4178013630 con 0x7f41801e6a30 2023-10-16T12:26:12.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.077+0000 7f4187ea6700 1 -- 172.21.15.16:0/3549319974 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41800ae790 con 0x7f41801e6a30 2023-10-16T12:26:12.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.078+0000 7f4187ea6700 1 -- 172.21.15.16:0/3549319974 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41800b3880 con 0x7f41801e6a30 2023-10-16T12:26:12.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.079+0000 7f41767fc700 1 -- 172.21.15.16:0/3549319974 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 29) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f4178013790 con 0x7f41801e6a30 2023-10-16T12:26:12.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.079+0000 7f41767fc700 1 --2- 172.21.15.16:0/3549319974 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f416406ad50 0x7f416406d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:12.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.079+0000 7f41767fc700 1 -- 172.21.15.16:0/3549319974 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6889+0+0 (secure 0 0 0) 0x7f4178089e30 con 0x7f41801e6a30 2023-10-16T12:26:12.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.079+0000 7f4185c42700 1 --2- 172.21.15.16:0/3549319974 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f416406ad50 0x7f416406d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:12.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.080+0000 7f4185c42700 1 --2- 172.21.15.16:0/3549319974 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f416406ad50 0x7f416406d210 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f41700090b0 tx=0x7f4170008030 comp rx=0 tx=0).ready entity=mgr.6396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:12.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.084+0000 7f41767fc700 1 -- 172.21.15.16:0/3549319974 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f41800b3880 con 0x7f41801e6a30 2023-10-16T12:26:12.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.291+0000 7f4187ea6700 1 -- 172.21.15.16:0/3549319974 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f41800b3880 con 0x7f41801e6a30 2023-10-16T12:26:12.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.836+0000 7f41767fc700 1 -- 172.21.15.16:0/3549319974 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 30) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f4178055150 con 0x7f41801e6a30 2023-10-16T12:26:12.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.919+0000 7f41767fc700 1 -- 172.21.15.16:0/3549319974 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v76) v1 ==== 181+0+0 (secure 0 0 0) 0x7f41780737c0 con 0x7f41801e6a30 2023-10-16T12:26:12.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.924+0000 7f4187ea6700 1 -- 172.21.15.16:0/3549319974 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f416406ad50 msgr2=0x7f416406d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:12.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.924+0000 7f4187ea6700 1 --2- 172.21.15.16:0/3549319974 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f416406ad50 0x7f416406d210 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f41700090b0 tx=0x7f4170008030 comp rx=0 tx=0).stop 2023-10-16T12:26:12.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.924+0000 7f4187ea6700 1 -- 172.21.15.16:0/3549319974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41801e6a30 msgr2=0x7f41801e6ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:12.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.924+0000 7f4187ea6700 1 --2- 172.21.15.16:0/3549319974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41801e6a30 0x7f41801e6ec0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f41800cdd20 tx=0x7f417800a600 comp rx=0 tx=0).stop 2023-10-16T12:26:12.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.926+0000 7f4187ea6700 1 -- 172.21.15.16:0/3549319974 shutdown_connections 2023-10-16T12:26:12.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.926+0000 7f4187ea6700 1 --2- 172.21.15.16:0/3549319974 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f416406ad50 0x7f416406d210 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:12.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.926+0000 7f4187ea6700 1 --2- 172.21.15.16:0/3549319974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41801e6a30 0x7f41801e6ec0 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:12.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.926+0000 7f4187ea6700 1 --2- 172.21.15.16:0/3549319974 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f416c003a00 0x7f41800c6760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:12.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.926+0000 7f4187ea6700 1 --2- 172.21.15.16:0/3549319974 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f41800ac8d0 0x7f41801e64f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:12.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.926+0000 7f4187ea6700 1 -- 172.21.15.16:0/3549319974 >> 172.21.15.16:0/3549319974 conn(0x7f418006f430 msgr2=0x7f41800afd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:12.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.927+0000 7f4187ea6700 1 -- 172.21.15.16:0/3549319974 shutdown_connections 2023-10-16T12:26:12.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:12.927+0000 7f4187ea6700 1 -- 172.21.15.16:0/3549319974 wait complete. 2023-10-16T12:26:12.932 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' removed 2023-10-16T12:26:12.945 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-16T12:26:13.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:13.078+0000 7f31c865b700 1 Processor -- start 2023-10-16T12:26:13.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:13.079+0000 7f31c865b700 1 -- start start 2023-10-16T12:26:13.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:13.079+0000 7f31c865b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f31c015d890 0x7f31c00ce5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:13.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:13.079+0000 7f31c865b700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31c00b6800 con 0x7f31c00d06b0 2023-10-16T12:26:13.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:13.079+0000 7f31c865b700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31c00d2580 con 0x7f31c00ceb20 2023-10-16T12:26:13.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:13.079+0000 7f31c865b700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31c00d33e0 con 0x7f31c015d890 2023-10-16T12:26:13.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:13.080+0000 7f31c63f7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f31c015d890 0x7f31c00ce5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:13.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:13.080+0000 7f31c63f7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f31c015d890 0x7f31c00ce5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38996/0 (socket says 172.21.15.16:38996) 2023-10-16T12:26:13.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:13.080+0000 7f31c63f7700 1 -- 172.21.15.16:0/524528352 learned_addr learned my addr 172.21.15.16:0/524528352 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:13.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:13.080+0000 7f31c63f7700 1 -- 172.21.15.16:0/524528352 >> v1:172.21.15.49:6789/0 conn(0x7f31c00ceb20 legacy=0x7f31c00cffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:13.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:13.080+0000 7f31c63f7700 1 -- 172.21.15.16:0/524528352 >> v1:172.21.15.16:6789/0 conn(0x7f31c00d06b0 legacy=0x7f31c00b6b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:13.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:13.080+0000 7f31c63f7700 1 -- 172.21.15.16:0/524528352 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31c00d27d0 con 0x7f31c015d890 2023-10-16T12:26:13.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:13.080+0000 7f31c63f7700 1 --2- 172.21.15.16:0/524528352 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31c015d890 0x7f31c00ce5e0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f31b00098f0 tx=0x7f31b0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfe17fefdbabaaa server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:13.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:13.080+0000 7f31c53f5700 1 -- 172.21.15.16:0/524528352 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31b000b8a0 con 0x7f31c015d890 2023-10-16T12:26:13.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:13.080+0000 7f31c53f5700 1 -- 172.21.15.16:0/524528352 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31b000ba00 con 0x7f31c015d890 2023-10-16T12:26:13.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:13.080+0000 7f31c53f5700 1 -- 172.21.15.16:0/524528352 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31b000bd90 con 0x7f31c015d890 2023-10-16T12:26:13.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:13.081+0000 7f31c865b700 1 -- 172.21.15.16:0/524528352 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31c015d890 msgr2=0x7f31c00ce5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:13.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:13.081+0000 7f31c865b700 1 --2- 172.21.15.16:0/524528352 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31c015d890 0x7f31c00ce5e0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f31b00098f0 tx=0x7f31b0009c00 comp rx=0 tx=0).stop 2023-10-16T12:26:13.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:13.081+0000 7f31c865b700 1 -- 172.21.15.16:0/524528352 shutdown_connections 2023-10-16T12:26:13.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:13.081+0000 7f31c865b700 1 --2- 172.21.15.16:0/524528352 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31c015d890 0x7f31c00ce5e0 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:13.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:13.081+0000 7f31c865b700 1 -- 172.21.15.16:0/524528352 >> 172.21.15.16:0/524528352 conn(0x7f31c0072360 msgr2=0x7f31c0072770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:13.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:13.081+0000 7f31c865b700 1 -- 172.21.15.16:0/524528352 shutdown_connections 2023-10-16T12:26:13.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:13.081+0000 7f31c865b700 1 -- 172.21.15.16:0/524528352 wait complete. 2023-10-16T12:26:13.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:13.082+0000 7f31c865b700 1 Processor -- start 2023-10-16T12:26:13.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:13.082+0000 7f31c865b700 1 -- start start 2023-10-16T12:26:13.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:13.083+0000 7f31c865b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31c00ceb20 0x7f31c01579a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:13.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:13.083+0000 7f31c865b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f31c00d06b0 0x7f31c0153ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:13.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:13.083+0000 7f31c865b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f31c015d890 0x7f31c00cd830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:13.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:13.083+0000 7f31c865b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31c00cd5a0 con 0x7f31c00ceb20 2023-10-16T12:26:13.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:13.083+0000 7f31c865b700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31c0157ee0 con 0x7f31c00d06b0 2023-10-16T12:26:13.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:13.083+0000 7f31c865b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31c00ca8a0 con 0x7f31c015d890 2023-10-16T12:26:13.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:13.083+0000 7f31c63f7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31c00ceb20 0x7f31c01579a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:13.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:13.083+0000 7f31c63f7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31c00ceb20 0x7f31c01579a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39000/0 (socket says 172.21.15.16:39000) 2023-10-16T12:26:13.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:13.083+0000 7f31c63f7700 1 -- 172.21.15.16:0/1364432074 learned_addr learned my addr 172.21.15.16:0/1364432074 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:13.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:13.083+0000 7f31c5bf6700 1 --2- 172.21.15.16:0/1364432074 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f31c00d06b0 0x7f31c0153ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:13.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:13.083+0000 7f31c6bf8700 1 --2- 172.21.15.16:0/1364432074 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f31c015d890 0x7f31c00cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:13.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:13.083+0000 7f31c63f7700 1 -- 172.21.15.16:0/1364432074 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f31c015d890 msgr2=0x7f31c00cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:13.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:13.083+0000 7f31c63f7700 1 --2- 172.21.15.16:0/1364432074 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f31c015d890 0x7f31c00cd830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:13.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:13.083+0000 7f31c63f7700 1 -- 172.21.15.16:0/1364432074 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f31c00d06b0 msgr2=0x7f31c0153ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:13.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:13.083+0000 7f31c63f7700 1 --2- 172.21.15.16:0/1364432074 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f31c00d06b0 0x7f31c0153ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:13.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:13.083+0000 7f31c63f7700 1 -- 172.21.15.16:0/1364432074 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31b0009550 con 0x7f31c00ceb20 2023-10-16T12:26:13.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:13.083+0000 7f31c5bf6700 1 --2- 172.21.15.16:0/1364432074 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f31c00d06b0 0x7f31c0153ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:26:13.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:13.083+0000 7f31c63f7700 1 --2- 172.21.15.16:0/1364432074 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31c00ceb20 0x7f31c01579a0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f31b0009130 tx=0x7f31b0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:13.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:13.084+0000 7f31b6ffd700 1 -- 172.21.15.16:0/1364432074 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31b00036a0 con 0x7f31c00ceb20 2023-10-16T12:26:13.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:13.084+0000 7f31b6ffd700 1 -- 172.21.15.16:0/1364432074 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31b001e070 con 0x7f31c00ceb20 2023-10-16T12:26:13.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:13.084+0000 7f31c865b700 1 -- 172.21.15.16:0/1364432074 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31c00cab80 con 0x7f31c00ceb20 2023-10-16T12:26:13.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:13.084+0000 7f31b6ffd700 1 -- 172.21.15.16:0/1364432074 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31b0003a60 con 0x7f31c00ceb20 2023-10-16T12:26:13.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:13.084+0000 7f31c865b700 1 -- 172.21.15.16:0/1364432074 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31c00cb140 con 0x7f31c00ceb20 2023-10-16T12:26:13.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:13.085+0000 7f31c865b700 1 -- 172.21.15.16:0/1364432074 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31c004ec20 con 0x7f31c00ceb20 2023-10-16T12:26:13.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:13.085+0000 7f31b6ffd700 1 -- 172.21.15.16:0/1364432074 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 30) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f31b0003bc0 con 0x7f31c00ceb20 2023-10-16T12:26:13.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:13.085+0000 7f31b6ffd700 1 --2- 172.21.15.16:0/1364432074 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f31a406af90 0x7f31a406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:13.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:13.086+0000 7f31b6ffd700 1 -- 172.21.15.16:0/1364432074 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6451+0+0 (secure 0 0 0) 0x7f31b0020070 con 0x7f31c00ceb20 2023-10-16T12:26:13.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:13.086+0000 7f31c5bf6700 1 --2- 172.21.15.16:0/1364432074 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f31a406af90 0x7f31a406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:13.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:13.086+0000 7f31c5bf6700 1 --2- 172.21.15.16:0/1364432074 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f31a406af90 0x7f31a406d450 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f31c015a940 tx=0x7f31bc009040 comp rx=0 tx=0).ready entity=mgr.6396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:13.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:13.090+0000 7f31b6ffd700 1 -- 172.21.15.16:0/1364432074 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f31b0055cf0 con 0x7f31c00ceb20 2023-10-16T12:26:13.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:13.291+0000 7f31c865b700 1 -- 172.21.15.16:0/1364432074 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f31c00b8720 con 0x7f31c00ceb20 2023-10-16T12:26:13.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:13.921+0000 7f31b6ffd700 1 -- 172.21.15.16:0/1364432074 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v77) v1 ==== 169+0+0 (secure 0 0 0) 0x7f31b0015020 con 0x7f31c00ceb20 2023-10-16T12:26:13.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:13.927+0000 7f31c865b700 1 -- 172.21.15.16:0/1364432074 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f31a406af90 msgr2=0x7f31a406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:13.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:13.927+0000 7f31c865b700 1 --2- 172.21.15.16:0/1364432074 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f31a406af90 0x7f31a406d450 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f31c015a940 tx=0x7f31bc009040 comp rx=0 tx=0).stop 2023-10-16T12:26:13.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:13.927+0000 7f31c865b700 1 -- 172.21.15.16:0/1364432074 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31c00ceb20 msgr2=0x7f31c01579a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:13.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:13.927+0000 7f31c865b700 1 --2- 172.21.15.16:0/1364432074 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31c00ceb20 0x7f31c01579a0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f31b0009130 tx=0x7f31b0004750 comp rx=0 tx=0).stop 2023-10-16T12:26:13.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:13.927+0000 7f31c865b700 1 -- 172.21.15.16:0/1364432074 shutdown_connections 2023-10-16T12:26:13.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:13.927+0000 7f31c865b700 1 --2- 172.21.15.16:0/1364432074 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f31a406af90 0x7f31a406d450 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:13.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:13.927+0000 7f31c865b700 1 --2- 172.21.15.16:0/1364432074 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f31c015d890 0x7f31c00cd830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:13.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:13.927+0000 7f31c865b700 1 --2- 172.21.15.16:0/1364432074 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31c00ceb20 0x7f31c01579a0 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:13.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:13.927+0000 7f31c865b700 1 --2- 172.21.15.16:0/1364432074 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f31c00d06b0 0x7f31c0153ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:13.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:13.927+0000 7f31c865b700 1 -- 172.21.15.16:0/1364432074 >> 172.21.15.16:0/1364432074 conn(0x7f31c0072360 msgr2=0x7f31c00b6b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:13.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:13.928+0000 7f31c865b700 1 -- 172.21.15.16:0/1364432074 shutdown_connections 2023-10-16T12:26:13.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:13.928+0000 7f31c865b700 1 -- 172.21.15.16:0/1364432074 wait complete. 2023-10-16T12:26:13.933 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' removed 2023-10-16T12:26:13.948 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-16T12:26:14.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.084+0000 7ff330755700 1 Processor -- start 2023-10-16T12:26:14.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.084+0000 7ff330755700 1 -- start start 2023-10-16T12:26:14.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.085+0000 7ff330755700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7ff32815a640 0x7ff3280cb710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:14.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.085+0000 7ff330755700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff3280c6770 0x7ff3280cbc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:14.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.085+0000 7ff330755700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff32815aa20 con 0x7ff32815e140 2023-10-16T12:26:14.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.085+0000 7ff330755700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff32806eb40 con 0x7ff32815a640 2023-10-16T12:26:14.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.085+0000 7ff330755700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff32806ee20 con 0x7ff3280c6770 2023-10-16T12:26:14.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.085+0000 7ff32ecf2700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff32815e140 0x7ff3280ca1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:56518/0 (socket says 172.21.15.16:56518) 2023-10-16T12:26:14.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.085+0000 7ff32ecf2700 1 -- 172.21.15.16:0/1564791334 learned_addr learned my addr 172.21.15.16:0/1564791334 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:26:14.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.085+0000 7ff32d4ef700 1 -- 172.21.15.16:0/1564791334 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1416176674 0 0) 0x7ff320001490 con 0x7ff32815e140 2023-10-16T12:26:14.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.085+0000 7ff32d4ef700 1 -- 172.21.15.16:0/1564791334 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3140035b0 con 0x7ff32815e140 2023-10-16T12:26:14.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.085+0000 7ff32e4f1700 1 --2- 172.21.15.16:0/1564791334 >> v2:172.21.15.49:3300/0 conn(0x7ff32815a640 0x7ff3280cb710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:14.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.085+0000 7ff32dcf0700 1 --2- 172.21.15.16:0/1564791334 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff3280c6770 0x7ff3280cbc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:14.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.086+0000 7ff32d4ef700 1 -- 172.21.15.16:0/1564791334 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2033104791 0 0) 0x7ff3140035b0 con 0x7ff32815e140 2023-10-16T12:26:14.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.086+0000 7ff32d4ef700 1 -- 172.21.15.16:0/1564791334 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff3280c6770 msgr2=0x7ff3280cbc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:14.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.086+0000 7ff32d4ef700 1 --2- 172.21.15.16:0/1564791334 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff3280c6770 0x7ff3280cbc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:14.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.086+0000 7ff32d4ef700 1 -- 172.21.15.16:0/1564791334 >> v2:172.21.15.49:3300/0 conn(0x7ff32815a640 msgr2=0x7ff3280cb710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:14.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.086+0000 7ff32d4ef700 1 --2- 172.21.15.16:0/1564791334 >> v2:172.21.15.49:3300/0 conn(0x7ff32815a640 0x7ff3280cb710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:14.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.086+0000 7ff32d4ef700 1 -- 172.21.15.16:0/1564791334 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff32806f100 con 0x7ff32815e140 2023-10-16T12:26:14.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.086+0000 7ff32d4ef700 1 -- 172.21.15.16:0/1564791334 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7ff320001b80 con 0x7ff32815e140 2023-10-16T12:26:14.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.086+0000 7ff32d4ef700 1 -- 172.21.15.16:0/1564791334 >> v1:172.21.15.16:6789/0 conn(0x7ff32815e140 legacy=0x7ff3280ca1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:14.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.086+0000 7ff32d4ef700 1 --2- 172.21.15.16:0/1564791334 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff3280c6770 0x7ff314003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:14.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.086+0000 7ff32d4ef700 1 --2- 172.21.15.16:0/1564791334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff314003f80 0x7ff314004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:14.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.086+0000 7ff32d4ef700 1 --2- 172.21.15.16:0/1564791334 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff314004d90 0x7ff314005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:14.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.086+0000 7ff32d4ef700 1 -- 172.21.15.16:0/1564791334 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff32806eb40 con 0x7ff314003f80 2023-10-16T12:26:14.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.086+0000 7ff32d4ef700 1 -- 172.21.15.16:0/1564791334 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff32806ee20 con 0x7ff314004d90 2023-10-16T12:26:14.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.086+0000 7ff32d4ef700 1 -- 172.21.15.16:0/1564791334 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff31400ba10 con 0x7ff3280c6770 2023-10-16T12:26:14.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.086+0000 7ff32dcf0700 1 --2- 172.21.15.16:0/1564791334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff314003f80 0x7ff314004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:14.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.086+0000 7ff32e4f1700 1 --2- 172.21.15.16:0/1564791334 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff3280c6770 0x7ff314003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:14.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.086+0000 7ff32ecf2700 1 --2- 172.21.15.16:0/1564791334 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff314004d90 0x7ff314005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:14.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.086+0000 7ff32dcf0700 1 -- 172.21.15.16:0/1564791334 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff3280c6770 msgr2=0x7ff314003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:14.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.086+0000 7ff32dcf0700 1 --2- 172.21.15.16:0/1564791334 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff3280c6770 0x7ff314003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:14.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.086+0000 7ff32dcf0700 1 -- 172.21.15.16:0/1564791334 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff314004d90 msgr2=0x7ff314005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:14.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.086+0000 7ff32dcf0700 1 --2- 172.21.15.16:0/1564791334 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff314004d90 0x7ff314005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:14.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.086+0000 7ff32dcf0700 1 -- 172.21.15.16:0/1564791334 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3140035b0 con 0x7ff314003f80 2023-10-16T12:26:14.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.086+0000 7ff32dcf0700 1 --2- 172.21.15.16:0/1564791334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff314003f80 0x7ff314004850 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7ff324009b30 tx=0x7ff314005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=242e33a219c77b3d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:14.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.086+0000 7ff32d4ef700 1 -- 172.21.15.16:0/1564791334 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff32400e660 con 0x7ff314003f80 2023-10-16T12:26:14.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.087+0000 7ff32d4ef700 1 -- 172.21.15.16:0/1564791334 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff32400e7c0 con 0x7ff314003f80 2023-10-16T12:26:14.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.087+0000 7ff32d4ef700 1 -- 172.21.15.16:0/1564791334 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff32400eb30 con 0x7ff314003f80 2023-10-16T12:26:14.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.087+0000 7ff330755700 1 -- 172.21.15.16:0/1564791334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff314003f80 msgr2=0x7ff314004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:14.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.087+0000 7ff330755700 1 --2- 172.21.15.16:0/1564791334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff314003f80 0x7ff314004850 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7ff324009b30 tx=0x7ff314005880 comp rx=0 tx=0).stop 2023-10-16T12:26:14.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.087+0000 7ff32ecf2700 1 -- 172.21.15.16:0/1564791334 reap_dead start 2023-10-16T12:26:14.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.087+0000 7ff330755700 1 -- 172.21.15.16:0/1564791334 shutdown_connections 2023-10-16T12:26:14.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.087+0000 7ff330755700 1 -- 172.21.15.16:0/1564791334 >> 172.21.15.16:0/1564791334 conn(0x7ff3280cc5e0 msgr2=0x7ff3280cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:14.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.087+0000 7ff330755700 1 -- 172.21.15.16:0/1564791334 shutdown_connections 2023-10-16T12:26:14.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.087+0000 7ff330755700 1 -- 172.21.15.16:0/1564791334 wait complete. 2023-10-16T12:26:14.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.089+0000 7ff330755700 1 Processor -- start 2023-10-16T12:26:14.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.089+0000 7ff330755700 1 -- start start 2023-10-16T12:26:14.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.090+0000 7ff330755700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff314003f80 0x7ff3281539c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:14.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.090+0000 7ff330755700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff3280b3cd0 0x7ff3280b4150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:14.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.090+0000 7ff330755700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff32815d100 0x7ff32815d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:14.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.090+0000 7ff330755700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3280b53d0 con 0x7ff32815d100 2023-10-16T12:26:14.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.090+0000 7ff330755700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff328153f00 con 0x7ff314003f80 2023-10-16T12:26:14.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.090+0000 7ff330755700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3280b4850 con 0x7ff3280b3cd0 2023-10-16T12:26:14.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.090+0000 7ff32ecf2700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff32815d100 0x7ff32815d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:14.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.090+0000 7ff32ecf2700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff32815d100 0x7ff32815d5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39024/0 (socket says 172.21.15.16:39024) 2023-10-16T12:26:14.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.090+0000 7ff32ecf2700 1 -- 172.21.15.16:0/1157520821 learned_addr learned my addr 172.21.15.16:0/1157520821 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:14.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.090+0000 7ff32dcf0700 1 --2- 172.21.15.16:0/1157520821 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff3280b3cd0 0x7ff3280b4150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:14.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.090+0000 7ff32e4f1700 1 --2- 172.21.15.16:0/1157520821 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff314003f80 0x7ff3281539c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:14.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.090+0000 7ff32ecf2700 1 -- 172.21.15.16:0/1157520821 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff3280b3cd0 msgr2=0x7ff3280b4150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:14.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.090+0000 7ff32ecf2700 1 --2- 172.21.15.16:0/1157520821 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff3280b3cd0 0x7ff3280b4150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:14.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.090+0000 7ff32ecf2700 1 -- 172.21.15.16:0/1157520821 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff314003f80 msgr2=0x7ff3281539c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:14.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.090+0000 7ff32ecf2700 1 --2- 172.21.15.16:0/1157520821 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff314003f80 0x7ff3281539c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:14.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.090+0000 7ff32ecf2700 1 -- 172.21.15.16:0/1157520821 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3280b4b30 con 0x7ff32815d100 2023-10-16T12:26:14.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.090+0000 7ff32dcf0700 1 --2- 172.21.15.16:0/1157520821 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff3280b3cd0 0x7ff3280b4150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:26:14.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.090+0000 7ff32ecf2700 1 --2- 172.21.15.16:0/1157520821 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff32815d100 0x7ff32815d5b0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7ff3280666c0 tx=0x7ff320007340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:14.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.091+0000 7ff31effd700 1 -- 172.21.15.16:0/1157520821 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff32000ae50 con 0x7ff32815d100 2023-10-16T12:26:14.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.091+0000 7ff31effd700 1 -- 172.21.15.16:0/1157520821 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff320007a50 con 0x7ff32815d100 2023-10-16T12:26:14.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.091+0000 7ff330755700 1 -- 172.21.15.16:0/1157520821 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3280b4e10 con 0x7ff32815d100 2023-10-16T12:26:14.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.091+0000 7ff31effd700 1 -- 172.21.15.16:0/1157520821 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff32001b6b0 con 0x7ff32815d100 2023-10-16T12:26:14.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.091+0000 7ff330755700 1 -- 172.21.15.16:0/1157520821 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff328154380 con 0x7ff32815d100 2023-10-16T12:26:14.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.092+0000 7ff330755700 1 -- 172.21.15.16:0/1157520821 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff32804ebb0 con 0x7ff32815d100 2023-10-16T12:26:14.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.092+0000 7ff31effd700 1 -- 172.21.15.16:0/1157520821 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 30) v1 ==== 89275+0+0 (secure 0 0 0) 0x7ff320018070 con 0x7ff32815d100 2023-10-16T12:26:14.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.093+0000 7ff31effd700 1 --2- 172.21.15.16:0/1157520821 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7ff30c06af90 0x7ff30c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:14.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.093+0000 7ff31effd700 1 -- 172.21.15.16:0/1157520821 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5979+0+0 (secure 0 0 0) 0x7ff32008bb10 con 0x7ff32815d100 2023-10-16T12:26:14.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.093+0000 7ff32e4f1700 1 --2- 172.21.15.16:0/1157520821 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7ff30c06af90 0x7ff30c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:14.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.093+0000 7ff32e4f1700 1 --2- 172.21.15.16:0/1157520821 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7ff30c06af90 0x7ff30c06d450 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff318009ce0 tx=0x7ff318009210 comp rx=0 tx=0).ready entity=mgr.6396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:14.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.097+0000 7ff31effd700 1 -- 172.21.15.16:0/1157520821 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff320056b50 con 0x7ff32815d100 2023-10-16T12:26:14.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.301+0000 7ff330755700 1 -- 172.21.15.16:0/1157520821 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7ff3280c5c70 con 0x7ff32815d100 2023-10-16T12:26:14.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.304+0000 7ff31effd700 1 -- 172.21.15.16:0/1157520821 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v19) v1 ==== 115+0+0 (secure 0 0 0) 0x7ff320020070 con 0x7ff32815d100 2023-10-16T12:26:14.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.307+0000 7ff330755700 1 -- 172.21.15.16:0/1157520821 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7ff30c06af90 msgr2=0x7ff30c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:14.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.307+0000 7ff330755700 1 --2- 172.21.15.16:0/1157520821 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7ff30c06af90 0x7ff30c06d450 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff318009ce0 tx=0x7ff318009210 comp rx=0 tx=0).stop 2023-10-16T12:26:14.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.307+0000 7ff330755700 1 -- 172.21.15.16:0/1157520821 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff32815d100 msgr2=0x7ff32815d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:14.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.307+0000 7ff330755700 1 --2- 172.21.15.16:0/1157520821 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff32815d100 0x7ff32815d5b0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7ff3280666c0 tx=0x7ff320007340 comp rx=0 tx=0).stop 2023-10-16T12:26:14.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.307+0000 7ff330755700 1 -- 172.21.15.16:0/1157520821 shutdown_connections 2023-10-16T12:26:14.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.307+0000 7ff330755700 1 --2- 172.21.15.16:0/1157520821 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7ff30c06af90 0x7ff30c06d450 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:14.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.307+0000 7ff330755700 1 --2- 172.21.15.16:0/1157520821 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff32815d100 0x7ff32815d5b0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:14.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.307+0000 7ff330755700 1 --2- 172.21.15.16:0/1157520821 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff314003f80 0x7ff3281539c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:14.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.307+0000 7ff330755700 1 --2- 172.21.15.16:0/1157520821 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff3280b3cd0 0x7ff3280b4150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:14.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.307+0000 7ff330755700 1 -- 172.21.15.16:0/1157520821 >> 172.21.15.16:0/1157520821 conn(0x7ff3280cc5e0 msgr2=0x7ff3281596f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:14.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.307+0000 7ff330755700 1 -- 172.21.15.16:0/1157520821 shutdown_connections 2023-10-16T12:26:14.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.307+0000 7ff330755700 1 -- 172.21.15.16:0/1157520821 wait complete. 2023-10-16T12:26:14.325 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-16T12:26:14.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.451+0000 7fbc506be700 1 Processor -- start 2023-10-16T12:26:14.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.451+0000 7fbc506be700 1 -- start start 2023-10-16T12:26:14.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.452+0000 7fbc506be700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbc4815b070 0x7fbc4815b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:14.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.452+0000 7fbc506be700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fbc480cb880 0x7fbc480cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:14.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.452+0000 7fbc506be700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc480cef70 con 0x7fbc4806da60 2023-10-16T12:26:14.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.452+0000 7fbc506be700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc480ce360 con 0x7fbc480cb880 2023-10-16T12:26:14.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.452+0000 7fbc506be700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc480ce640 con 0x7fbc4815b070 2023-10-16T12:26:14.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.452+0000 7fbc4e45a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbc4815b070 0x7fbc4815b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:14.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.452+0000 7fbc4dc59700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fbc480cb880 0x7fbc480cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:14.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.452+0000 7fbc4dc59700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fbc480cb880 0x7fbc480cbd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:38090/0 (socket says 172.21.15.16:38090) 2023-10-16T12:26:14.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.452+0000 7fbc4e45a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbc4815b070 0x7fbc4815b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:55518/0 (socket says 172.21.15.16:55518) 2023-10-16T12:26:14.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.452+0000 7fbc4dc59700 1 -- 172.21.15.16:0/1747054885 learned_addr learned my addr 172.21.15.16:0/1747054885 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:14.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.452+0000 7fbc4dc59700 1 -- 172.21.15.16:0/1747054885 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbc4815b070 msgr2=0x7fbc4815b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:14.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.452+0000 7fbc4dc59700 1 --2- 172.21.15.16:0/1747054885 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbc4815b070 0x7fbc4815b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:14.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.452+0000 7fbc4dc59700 1 -- 172.21.15.16:0/1747054885 >> v1:172.21.15.49:6789/0 conn(0x7fbc4806da60 legacy=0x7fbc480cf400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:14.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.452+0000 7fbc4dc59700 1 -- 172.21.15.16:0/1747054885 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc480ce920 con 0x7fbc480cb880 2023-10-16T12:26:14.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.453+0000 7fbc4dc59700 1 --2- 172.21.15.16:0/1747054885 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbc480cb880 0x7fbc480cbd00 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fbc380098f0 tx=0x7fbc38009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e5938e72deab344 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:14.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.453+0000 7fbc4d458700 1 -- 172.21.15.16:0/1747054885 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc38011830 con 0x7fbc480cb880 2023-10-16T12:26:14.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.453+0000 7fbc4d458700 1 -- 172.21.15.16:0/1747054885 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc38011990 con 0x7fbc480cb880 2023-10-16T12:26:14.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.453+0000 7fbc4d458700 1 -- 172.21.15.16:0/1747054885 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc38011d20 con 0x7fbc480cb880 2023-10-16T12:26:14.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.453+0000 7fbc506be700 1 -- 172.21.15.16:0/1747054885 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbc480cb880 msgr2=0x7fbc480cbd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:14.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.453+0000 7fbc506be700 1 --2- 172.21.15.16:0/1747054885 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbc480cb880 0x7fbc480cbd00 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fbc380098f0 tx=0x7fbc38009c00 comp rx=0 tx=0).stop 2023-10-16T12:26:14.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.453+0000 7fbc506be700 1 -- 172.21.15.16:0/1747054885 shutdown_connections 2023-10-16T12:26:14.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.453+0000 7fbc506be700 1 --2- 172.21.15.16:0/1747054885 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbc4815b070 0x7fbc4815b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:14.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.453+0000 7fbc506be700 1 --2- 172.21.15.16:0/1747054885 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbc480cb880 0x7fbc480cbd00 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:14.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.453+0000 7fbc506be700 1 -- 172.21.15.16:0/1747054885 >> 172.21.15.16:0/1747054885 conn(0x7fbc480b1f40 msgr2=0x7fbc480b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:14.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.453+0000 7fbc506be700 1 -- 172.21.15.16:0/1747054885 shutdown_connections 2023-10-16T12:26:14.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.453+0000 7fbc506be700 1 -- 172.21.15.16:0/1747054885 wait complete. 2023-10-16T12:26:14.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.455+0000 7fbc506be700 1 Processor -- start 2023-10-16T12:26:14.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.455+0000 7fbc506be700 1 -- start start 2023-10-16T12:26:14.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.455+0000 7fbc506be700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbc4806da60 0x7fbc48152e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:14.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.455+0000 7fbc506be700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbc480cb880 0x7fbc4815bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:14.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.455+0000 7fbc506be700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc4815b070 0x7fbc481586f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:14.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.455+0000 7fbc506be700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc4815c620 con 0x7fbc4815b070 2023-10-16T12:26:14.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.455+0000 7fbc506be700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc4815c900 con 0x7fbc4806da60 2023-10-16T12:26:14.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.455+0000 7fbc506be700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc4815cbe0 con 0x7fbc480cb880 2023-10-16T12:26:14.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.456+0000 7fbc4ec5b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc4815b070 0x7fbc481586f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:14.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.456+0000 7fbc4ec5b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc4815b070 0x7fbc481586f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39034/0 (socket says 172.21.15.16:39034) 2023-10-16T12:26:14.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.456+0000 7fbc4ec5b700 1 -- 172.21.15.16:0/1491120759 learned_addr learned my addr 172.21.15.16:0/1491120759 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:14.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.456+0000 7fbc4dc59700 1 --2- 172.21.15.16:0/1491120759 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbc480cb880 0x7fbc4815bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:14.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.456+0000 7fbc4e45a700 1 --2- 172.21.15.16:0/1491120759 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbc4806da60 0x7fbc48152e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:14.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.456+0000 7fbc4ec5b700 1 -- 172.21.15.16:0/1491120759 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbc480cb880 msgr2=0x7fbc4815bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:14.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.456+0000 7fbc4ec5b700 1 --2- 172.21.15.16:0/1491120759 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbc480cb880 0x7fbc4815bbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:14.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.456+0000 7fbc4ec5b700 1 -- 172.21.15.16:0/1491120759 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbc4806da60 msgr2=0x7fbc48152e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:14.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.456+0000 7fbc4ec5b700 1 --2- 172.21.15.16:0/1491120759 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbc4806da60 0x7fbc48152e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:14.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.456+0000 7fbc4ec5b700 1 -- 172.21.15.16:0/1491120759 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc38009550 con 0x7fbc4815b070 2023-10-16T12:26:14.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.456+0000 7fbc4e45a700 1 --2- 172.21.15.16:0/1491120759 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbc4806da60 0x7fbc48152e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:26:14.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.456+0000 7fbc4ec5b700 1 --2- 172.21.15.16:0/1491120759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc4815b070 0x7fbc481586f0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fbc4000cc50 tx=0x7fbc4000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:14.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.456+0000 7fbc3effd700 1 -- 172.21.15.16:0/1491120759 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc40003c00 con 0x7fbc4815b070 2023-10-16T12:26:14.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.456+0000 7fbc3effd700 1 -- 172.21.15.16:0/1491120759 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc40003d60 con 0x7fbc4815b070 2023-10-16T12:26:14.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.456+0000 7fbc3effd700 1 -- 172.21.15.16:0/1491120759 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc4000aab0 con 0x7fbc4815b070 2023-10-16T12:26:14.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.456+0000 7fbc506be700 1 -- 172.21.15.16:0/1491120759 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc4815cec0 con 0x7fbc4815b070 2023-10-16T12:26:14.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.456+0000 7fbc506be700 1 -- 172.21.15.16:0/1491120759 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc48153340 con 0x7fbc4815b070 2023-10-16T12:26:14.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.457+0000 7fbc506be700 1 -- 172.21.15.16:0/1491120759 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc4804ebb0 con 0x7fbc4815b070 2023-10-16T12:26:14.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.458+0000 7fbc3effd700 1 -- 172.21.15.16:0/1491120759 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 31) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fbc4000ac10 con 0x7fbc4815b070 2023-10-16T12:26:14.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.458+0000 7fbc3effd700 1 --2- 172.21.15.16:0/1491120759 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7fbc2c06b020 0x7fbc2c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:14.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.458+0000 7fbc3effd700 1 -- 172.21.15.16:0/1491120759 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5979+0+0 (secure 0 0 0) 0x7fbc400897b0 con 0x7fbc4815b070 2023-10-16T12:26:14.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.458+0000 7fbc4e45a700 1 --2- 172.21.15.16:0/1491120759 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7fbc2c06b020 0x7fbc2c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:14.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.459+0000 7fbc4e45a700 1 --2- 172.21.15.16:0/1491120759 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7fbc2c06b020 0x7fbc2c06d4e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fbc44009f30 tx=0x7fbc44009210 comp rx=0 tx=0).ready entity=mgr.6396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:14.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.463+0000 7fbc3effd700 1 -- 172.21.15.16:0/1491120759 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbc400546d0 con 0x7fbc4815b070 2023-10-16T12:26:14.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.667+0000 7fbc506be700 1 -- 172.21.15.16:0/1491120759 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fbc480c6a70 con 0x7fbc4815b070 2023-10-16T12:26:14.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.670+0000 7fbc3effd700 1 -- 172.21.15.16:0/1491120759 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v20) v1 ==== 104+0+0 (secure 0 0 0) 0x7fbc40057cf0 con 0x7fbc4815b070 2023-10-16T12:26:14.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.673+0000 7fbc506be700 1 -- 172.21.15.16:0/1491120759 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7fbc2c06b020 msgr2=0x7fbc2c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:14.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.673+0000 7fbc506be700 1 --2- 172.21.15.16:0/1491120759 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7fbc2c06b020 0x7fbc2c06d4e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fbc44009f30 tx=0x7fbc44009210 comp rx=0 tx=0).stop 2023-10-16T12:26:14.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.673+0000 7fbc506be700 1 -- 172.21.15.16:0/1491120759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc4815b070 msgr2=0x7fbc481586f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:14.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.673+0000 7fbc506be700 1 --2- 172.21.15.16:0/1491120759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc4815b070 0x7fbc481586f0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fbc4000cc50 tx=0x7fbc4000cf60 comp rx=0 tx=0).stop 2023-10-16T12:26:14.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.673+0000 7fbc506be700 1 -- 172.21.15.16:0/1491120759 shutdown_connections 2023-10-16T12:26:14.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.673+0000 7fbc506be700 1 --2- 172.21.15.16:0/1491120759 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7fbc2c06b020 0x7fbc2c06d4e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:14.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.673+0000 7fbc506be700 1 --2- 172.21.15.16:0/1491120759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc4815b070 0x7fbc481586f0 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:14.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.673+0000 7fbc506be700 1 --2- 172.21.15.16:0/1491120759 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbc4806da60 0x7fbc48152e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:14.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.673+0000 7fbc506be700 1 --2- 172.21.15.16:0/1491120759 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbc480cb880 0x7fbc4815bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:14.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.673+0000 7fbc506be700 1 -- 172.21.15.16:0/1491120759 >> 172.21.15.16:0/1491120759 conn(0x7fbc480b1f40 msgr2=0x7fbc481548f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:14.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.673+0000 7fbc506be700 1 -- 172.21.15.16:0/1491120759 shutdown_connections 2023-10-16T12:26:14.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.673+0000 7fbc506be700 1 -- 172.21.15.16:0/1491120759 wait complete. 2023-10-16T12:26:14.687 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-16T12:26:14.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.812+0000 7ff32972e700 1 Processor -- start 2023-10-16T12:26:14.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.812+0000 7ff32972e700 1 -- start start 2023-10-16T12:26:14.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.813+0000 7ff32972e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff32415d8d0 0x7ff3240b2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:14.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.813+0000 7ff32972e700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3240b2f30 con 0x7ff3240b8fa0 2023-10-16T12:26:14.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.813+0000 7ff32972e700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3240d1c40 con 0x7ff3240ce350 2023-10-16T12:26:14.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.813+0000 7ff32972e700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3240d0bb0 con 0x7ff32415d8d0 2023-10-16T12:26:14.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.813+0000 7ff322ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff32415d8d0 0x7ff3240b2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:14.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.813+0000 7ff322ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff32415d8d0 0x7ff3240b2830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39040/0 (socket says 172.21.15.16:39040) 2023-10-16T12:26:14.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.813+0000 7ff322ffd700 1 -- 172.21.15.16:0/676231495 learned_addr learned my addr 172.21.15.16:0/676231495 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:14.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.813+0000 7ff322ffd700 1 -- 172.21.15.16:0/676231495 >> v1:172.21.15.49:6789/0 conn(0x7ff3240ce350 legacy=0x7ff3240d29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:14.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.813+0000 7ff322ffd700 1 -- 172.21.15.16:0/676231495 >> v1:172.21.15.16:6789/0 conn(0x7ff3240b8fa0 legacy=0x7ff3240b6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:14.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.813+0000 7ff322ffd700 1 -- 172.21.15.16:0/676231495 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3240d1ee0 con 0x7ff32415d8d0 2023-10-16T12:26:14.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.813+0000 7ff322ffd700 1 --2- 172.21.15.16:0/676231495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff32415d8d0 0x7ff3240b2830 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7ff314009e50 tx=0x7ff31400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1d298ceab149ac8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:14.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.814+0000 7ff321ffb700 1 -- 172.21.15.16:0/676231495 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff31400e070 con 0x7ff32415d8d0 2023-10-16T12:26:14.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.814+0000 7ff321ffb700 1 -- 172.21.15.16:0/676231495 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff31400b9c0 con 0x7ff32415d8d0 2023-10-16T12:26:14.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.814+0000 7ff321ffb700 1 -- 172.21.15.16:0/676231495 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff31400bd70 con 0x7ff32415d8d0 2023-10-16T12:26:14.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.814+0000 7ff32972e700 1 -- 172.21.15.16:0/676231495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff32415d8d0 msgr2=0x7ff3240b2830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:14.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.814+0000 7ff32972e700 1 --2- 172.21.15.16:0/676231495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff32415d8d0 0x7ff3240b2830 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7ff314009e50 tx=0x7ff31400b330 comp rx=0 tx=0).stop 2023-10-16T12:26:14.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.814+0000 7ff32972e700 1 -- 172.21.15.16:0/676231495 shutdown_connections 2023-10-16T12:26:14.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.814+0000 7ff32972e700 1 --2- 172.21.15.16:0/676231495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff32415d8d0 0x7ff3240b2830 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:14.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.814+0000 7ff32972e700 1 -- 172.21.15.16:0/676231495 >> 172.21.15.16:0/676231495 conn(0x7ff324072330 msgr2=0x7ff324072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:14.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.814+0000 7ff32972e700 1 -- 172.21.15.16:0/676231495 shutdown_connections 2023-10-16T12:26:14.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.814+0000 7ff32972e700 1 -- 172.21.15.16:0/676231495 wait complete. 2023-10-16T12:26:14.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.816+0000 7ff32972e700 1 Processor -- start 2023-10-16T12:26:14.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.816+0000 7ff32972e700 1 -- start start 2023-10-16T12:26:14.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.816+0000 7ff32972e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3240b8fa0 0x7ff3240caf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:14.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.816+0000 7ff32972e700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff3240ce350 0x7ff324154c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:14.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.816+0000 7ff32972e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff32415d8d0 0x7ff324160450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:14.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.816+0000 7ff32972e700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff32415dcb0 con 0x7ff3240b8fa0 2023-10-16T12:26:14.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.816+0000 7ff32972e700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3240cb4c0 con 0x7ff3240ce350 2023-10-16T12:26:14.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.816+0000 7ff32972e700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3241551b0 con 0x7ff32415d8d0 2023-10-16T12:26:14.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.816+0000 7ff322ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3240b8fa0 0x7ff3240caf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:14.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.816+0000 7ff322ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3240b8fa0 0x7ff3240caf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39046/0 (socket says 172.21.15.16:39046) 2023-10-16T12:26:14.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.816+0000 7ff322ffd700 1 -- 172.21.15.16:0/2956476492 learned_addr learned my addr 172.21.15.16:0/2956476492 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:14.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.817+0000 7ff3237fe700 1 --2- 172.21.15.16:0/2956476492 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff32415d8d0 0x7ff324160450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:14.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.817+0000 7ff322ffd700 1 -- 172.21.15.16:0/2956476492 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff32415d8d0 msgr2=0x7ff324160450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:14.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.817+0000 7ff3227fc700 1 --2- 172.21.15.16:0/2956476492 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff3240ce350 0x7ff324154c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:14.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.817+0000 7ff322ffd700 1 --2- 172.21.15.16:0/2956476492 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff32415d8d0 0x7ff324160450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:14.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.817+0000 7ff322ffd700 1 -- 172.21.15.16:0/2956476492 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff3240ce350 msgr2=0x7ff324154c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:14.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.817+0000 7ff322ffd700 1 --2- 172.21.15.16:0/2956476492 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff3240ce350 0x7ff324154c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:14.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.817+0000 7ff322ffd700 1 -- 172.21.15.16:0/2956476492 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff314009b30 con 0x7ff3240b8fa0 2023-10-16T12:26:14.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.817+0000 7ff3237fe700 1 --2- 172.21.15.16:0/2956476492 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff32415d8d0 0x7ff324160450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:26:14.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.817+0000 7ff3227fc700 1 --2- 172.21.15.16:0/2956476492 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff3240ce350 0x7ff324154c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:26:14.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.817+0000 7ff322ffd700 1 --2- 172.21.15.16:0/2956476492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3240b8fa0 0x7ff3240caf80 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7ff31400b730 tx=0x7ff314004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:14.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.817+0000 7ff31b7fe700 1 -- 172.21.15.16:0/2956476492 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3140099b0 con 0x7ff3240b8fa0 2023-10-16T12:26:14.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.817+0000 7ff31b7fe700 1 -- 172.21.15.16:0/2956476492 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3140092e0 con 0x7ff3240b8fa0 2023-10-16T12:26:14.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.817+0000 7ff32972e700 1 -- 172.21.15.16:0/2956476492 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff324160990 con 0x7ff3240b8fa0 2023-10-16T12:26:14.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.817+0000 7ff31b7fe700 1 -- 172.21.15.16:0/2956476492 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff31401b630 con 0x7ff3240b8fa0 2023-10-16T12:26:14.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.817+0000 7ff32972e700 1 -- 172.21.15.16:0/2956476492 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3240c9380 con 0x7ff3240b8fa0 2023-10-16T12:26:14.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.818+0000 7ff32972e700 1 -- 172.21.15.16:0/2956476492 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff32404ebb0 con 0x7ff3240b8fa0 2023-10-16T12:26:14.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.818+0000 7ff31b7fe700 1 -- 172.21.15.16:0/2956476492 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 31) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff31401b790 con 0x7ff3240b8fa0 2023-10-16T12:26:14.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.819+0000 7ff31b7fe700 1 --2- 172.21.15.16:0/2956476492 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7ff30406b020 0x7ff30406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:14.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.819+0000 7ff31b7fe700 1 -- 172.21.15.16:0/2956476492 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5979+0+0 (secure 0 0 0) 0x7ff31408b0b0 con 0x7ff3240b8fa0 2023-10-16T12:26:14.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.819+0000 7ff3227fc700 1 --2- 172.21.15.16:0/2956476492 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7ff30406b020 0x7ff30406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:14.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.819+0000 7ff3227fc700 1 --2- 172.21.15.16:0/2956476492 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7ff30406b020 0x7ff30406d4e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff3240b8200 tx=0x7ff30c009040 comp rx=0 tx=0).ready entity=mgr.6396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:14.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:14.823+0000 7ff31b7fe700 1 -- 172.21.15.16:0/2956476492 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff314055e40 con 0x7ff3240b8fa0 2023-10-16T12:26:15.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.028+0000 7ff32972e700 1 -- 172.21.15.16:0/2956476492 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7ff3240d2e20 con 0x7ff3240b8fa0 2023-10-16T12:26:15.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.031+0000 7ff31b7fe700 1 -- 172.21.15.16:0/2956476492 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v21) v1 ==== 120+0+0 (secure 0 0 0) 0x7ff314059460 con 0x7ff3240b8fa0 2023-10-16T12:26:15.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.034+0000 7ff32972e700 1 -- 172.21.15.16:0/2956476492 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7ff30406b020 msgr2=0x7ff30406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:15.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.034+0000 7ff32972e700 1 --2- 172.21.15.16:0/2956476492 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7ff30406b020 0x7ff30406d4e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff3240b8200 tx=0x7ff30c009040 comp rx=0 tx=0).stop 2023-10-16T12:26:15.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.034+0000 7ff32972e700 1 -- 172.21.15.16:0/2956476492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3240b8fa0 msgr2=0x7ff3240caf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:15.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.034+0000 7ff32972e700 1 --2- 172.21.15.16:0/2956476492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3240b8fa0 0x7ff3240caf80 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7ff31400b730 tx=0x7ff314004ad0 comp rx=0 tx=0).stop 2023-10-16T12:26:15.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.034+0000 7ff32972e700 1 -- 172.21.15.16:0/2956476492 shutdown_connections 2023-10-16T12:26:15.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.034+0000 7ff32972e700 1 --2- 172.21.15.16:0/2956476492 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7ff30406b020 0x7ff30406d4e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:15.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.034+0000 7ff32972e700 1 --2- 172.21.15.16:0/2956476492 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff32415d8d0 0x7ff324160450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:15.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.034+0000 7ff32972e700 1 --2- 172.21.15.16:0/2956476492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3240b8fa0 0x7ff3240caf80 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:15.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.034+0000 7ff32972e700 1 --2- 172.21.15.16:0/2956476492 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff3240ce350 0x7ff324154c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:15.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.034+0000 7ff32972e700 1 -- 172.21.15.16:0/2956476492 >> 172.21.15.16:0/2956476492 conn(0x7ff324072330 msgr2=0x7ff324072710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:15.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.034+0000 7ff32972e700 1 -- 172.21.15.16:0/2956476492 shutdown_connections 2023-10-16T12:26:15.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.034+0000 7ff32972e700 1 -- 172.21.15.16:0/2956476492 wait complete. 2023-10-16T12:26:15.048 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule' 2023-10-16T12:26:15.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.171+0000 7f4042969700 1 Processor -- start 2023-10-16T12:26:15.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.171+0000 7f4042969700 1 -- start start 2023-10-16T12:26:15.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.172+0000 7f4042969700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f403c15b630 0x7f403c0cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:15.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.172+0000 7f4042969700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f403c0cc2a0 con 0x7f403c0ce4f0 2023-10-16T12:26:15.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.172+0000 7f4042969700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f403c0cc540 con 0x7f403c15ba10 2023-10-16T12:26:15.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.172+0000 7f4042969700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f403c0cdb20 con 0x7f403c15b630 2023-10-16T12:26:15.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.172+0000 7f4040f06700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f403c0ce4f0 0x7f403c0cfca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:56564/0 (socket says 172.21.15.16:56564) 2023-10-16T12:26:15.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.172+0000 7f4040f06700 1 -- 172.21.15.16:0/78738386 learned_addr learned my addr 172.21.15.16:0/78738386 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:26:15.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.173+0000 7f403bfff700 1 --2- 172.21.15.16:0/78738386 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f403c15b630 0x7f403c0cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:15.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.173+0000 7f403affd700 1 -- 172.21.15.16:0/78738386 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3334538553 0 0) 0x7f403c0cc2a0 con 0x7f403c0ce4f0 2023-10-16T12:26:15.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.173+0000 7f403affd700 1 -- 172.21.15.16:0/78738386 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40240035b0 con 0x7f403c0ce4f0 2023-10-16T12:26:15.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.173+0000 7f403affd700 1 -- 172.21.15.16:0/78738386 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2976616195 0 0) 0x7f403c0cc540 con 0x7f403c15ba10 2023-10-16T12:26:15.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.173+0000 7f403affd700 1 -- 172.21.15.16:0/78738386 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f403c0cc2a0 con 0x7f403c15ba10 2023-10-16T12:26:15.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.173+0000 7f403affd700 1 -- 172.21.15.16:0/78738386 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1474199862 0 0) 0x7f40240035b0 con 0x7f403c0ce4f0 2023-10-16T12:26:15.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.173+0000 7f403affd700 1 -- 172.21.15.16:0/78738386 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f403c15b630 msgr2=0x7f403c0cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:15.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.173+0000 7f403affd700 1 --2- 172.21.15.16:0/78738386 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f403c15b630 0x7f403c0cd5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:15.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.173+0000 7f403affd700 1 -- 172.21.15.16:0/78738386 >> v1:172.21.15.49:6789/0 conn(0x7f403c15ba10 legacy=0x7f403c0cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:15.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.173+0000 7f403affd700 1 -- 172.21.15.16:0/78738386 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f403c0cc820 con 0x7f403c0ce4f0 2023-10-16T12:26:15.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.173+0000 7f403affd700 1 -- 172.21.15.16:0/78738386 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f40340019f0 con 0x7f403c0ce4f0 2023-10-16T12:26:15.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.173+0000 7f403affd700 1 -- 172.21.15.16:0/78738386 >> v1:172.21.15.16:6789/0 conn(0x7f403c0ce4f0 legacy=0x7f403c0cfca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:15.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.173+0000 7f403affd700 1 --2- 172.21.15.16:0/78738386 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4024003bc0 0x7f4024003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:15.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.173+0000 7f403affd700 1 --2- 172.21.15.16:0/78738386 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f403c15b630 0x7f4024004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:15.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.173+0000 7f403affd700 1 --2- 172.21.15.16:0/78738386 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4024004d90 0x7f4024005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:15.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.173+0000 7f403affd700 1 -- 172.21.15.16:0/78738386 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f402400bba0 con 0x7f4024004d90 2023-10-16T12:26:15.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.173+0000 7f403affd700 1 -- 172.21.15.16:0/78738386 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f402400bd10 con 0x7f4024003bc0 2023-10-16T12:26:15.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.173+0000 7f403affd700 1 -- 172.21.15.16:0/78738386 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f402400bed0 con 0x7f403c15b630 2023-10-16T12:26:15.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.173+0000 7f4040f06700 1 --2- 172.21.15.16:0/78738386 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4024004d90 0x7f4024005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:15.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.173+0000 7f403bfff700 1 --2- 172.21.15.16:0/78738386 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4024003bc0 0x7f4024003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:15.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.173+0000 7f403b7fe700 1 --2- 172.21.15.16:0/78738386 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f403c15b630 0x7f4024004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:15.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.173+0000 7f4040f06700 1 -- 172.21.15.16:0/78738386 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f403c15b630 msgr2=0x7f4024004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:15.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.173+0000 7f4040f06700 1 --2- 172.21.15.16:0/78738386 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f403c15b630 0x7f4024004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:15.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.173+0000 7f4040f06700 1 -- 172.21.15.16:0/78738386 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4024003bc0 msgr2=0x7f4024003fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:15.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.173+0000 7f4040f06700 1 --2- 172.21.15.16:0/78738386 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4024003bc0 0x7f4024003fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:15.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.173+0000 7f4040f06700 1 -- 172.21.15.16:0/78738386 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40240035b0 con 0x7f4024004d90 2023-10-16T12:26:15.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.174+0000 7f4040f06700 1 --2- 172.21.15.16:0/78738386 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4024004d90 0x7f4024005210 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f403c0cc090 tx=0x7f4024005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba2475b476c82fcd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:15.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.174+0000 7f403affd700 1 -- 172.21.15.16:0/78738386 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4034012600 con 0x7f4024004d90 2023-10-16T12:26:15.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.174+0000 7f403affd700 1 -- 172.21.15.16:0/78738386 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4034012780 con 0x7f4024004d90 2023-10-16T12:26:15.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.174+0000 7f403affd700 1 -- 172.21.15.16:0/78738386 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4034012b30 con 0x7f4024004d90 2023-10-16T12:26:15.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.174+0000 7f4042969700 1 -- 172.21.15.16:0/78738386 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4024004d90 msgr2=0x7f4024005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:15.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.174+0000 7f4042969700 1 --2- 172.21.15.16:0/78738386 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4024004d90 0x7f4024005210 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f403c0cc090 tx=0x7f4024005880 comp rx=0 tx=0).stop 2023-10-16T12:26:15.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.174+0000 7f4040f06700 1 -- 172.21.15.16:0/78738386 reap_dead start 2023-10-16T12:26:15.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.174+0000 7f4042969700 1 -- 172.21.15.16:0/78738386 shutdown_connections 2023-10-16T12:26:15.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.174+0000 7f4042969700 1 -- 172.21.15.16:0/78738386 >> 172.21.15.16:0/78738386 conn(0x7f403c0d14b0 msgr2=0x7f403c0d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:15.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.174+0000 7f4042969700 1 -- 172.21.15.16:0/78738386 shutdown_connections 2023-10-16T12:26:15.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.174+0000 7f4042969700 1 -- 172.21.15.16:0/78738386 wait complete. 2023-10-16T12:26:15.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.176+0000 7f4042969700 1 Processor -- start 2023-10-16T12:26:15.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.176+0000 7f4042969700 1 -- start start 2023-10-16T12:26:15.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.176+0000 7f4042969700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4024004d90 0x7f403c1f2690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:15.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.176+0000 7f4042969700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f403c0b6560 0x7f403c0c95d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:15.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.176+0000 7f4042969700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f403c0b4300 0x7f403c0b4780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:15.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.176+0000 7f4042969700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f403c0b4cc0 con 0x7f403c0b4300 2023-10-16T12:26:15.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.177+0000 7f4042969700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f403c0caf70 con 0x7f4024004d90 2023-10-16T12:26:15.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.177+0000 7f4042969700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f403c0caae0 con 0x7f403c0b6560 2023-10-16T12:26:15.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.177+0000 7f4040f06700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f403c0b4300 0x7f403c0b4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:15.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.177+0000 7f4040f06700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f403c0b4300 0x7f403c0b4780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39070/0 (socket says 172.21.15.16:39070) 2023-10-16T12:26:15.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.177+0000 7f4040f06700 1 -- 172.21.15.16:0/2123893459 learned_addr learned my addr 172.21.15.16:0/2123893459 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:15.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.177+0000 7f403bfff700 1 --2- 172.21.15.16:0/2123893459 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4024004d90 0x7f403c1f2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:15.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.177+0000 7f403b7fe700 1 --2- 172.21.15.16:0/2123893459 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f403c0b6560 0x7f403c0c95d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:15.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.177+0000 7f4040f06700 1 -- 172.21.15.16:0/2123893459 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f403c0b6560 msgr2=0x7f403c0c95d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:15.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.177+0000 7f4040f06700 1 --2- 172.21.15.16:0/2123893459 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f403c0b6560 0x7f403c0c95d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:15.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.177+0000 7f4040f06700 1 -- 172.21.15.16:0/2123893459 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4024004d90 msgr2=0x7f403c1f2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:15.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.177+0000 7f4040f06700 1 --2- 172.21.15.16:0/2123893459 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4024004d90 0x7f403c1f2690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:15.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.177+0000 7f4040f06700 1 -- 172.21.15.16:0/2123893459 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f403c0c9ed0 con 0x7f403c0b4300 2023-10-16T12:26:15.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.177+0000 7f4040f06700 1 --2- 172.21.15.16:0/2123893459 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f403c0b4300 0x7f403c0b4780 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f4034012000 tx=0x7f4034001d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:15.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.177+0000 7f4038ff9700 1 -- 172.21.15.16:0/2123893459 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40340019b0 con 0x7f403c0b4300 2023-10-16T12:26:15.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.177+0000 7f4038ff9700 1 -- 172.21.15.16:0/2123893459 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4034012610 con 0x7f403c0b4300 2023-10-16T12:26:15.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.177+0000 7f4042969700 1 -- 172.21.15.16:0/2123893459 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f403c0ca1b0 con 0x7f403c0b4300 2023-10-16T12:26:15.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.177+0000 7f4038ff9700 1 -- 172.21.15.16:0/2123893459 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f403401e640 con 0x7f403c0b4300 2023-10-16T12:26:15.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.177+0000 7f4042969700 1 -- 172.21.15.16:0/2123893459 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f403c0ca770 con 0x7f403c0b4300 2023-10-16T12:26:15.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.178+0000 7f4042969700 1 -- 172.21.15.16:0/2123893459 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f403c0cac90 con 0x7f403c0b4300 2023-10-16T12:26:15.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.179+0000 7f4038ff9700 1 -- 172.21.15.16:0/2123893459 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 31) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4034019030 con 0x7f403c0b4300 2023-10-16T12:26:15.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.179+0000 7f4038ff9700 1 --2- 172.21.15.16:0/2123893459 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f402806b050 0x7f402806d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:15.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.179+0000 7f4038ff9700 1 -- 172.21.15.16:0/2123893459 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5979+0+0 (secure 0 0 0) 0x7f403408b420 con 0x7f403c0b4300 2023-10-16T12:26:15.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.179+0000 7f403bfff700 1 --2- 172.21.15.16:0/2123893459 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f402806b050 0x7f402806d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:15.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.180+0000 7f403bfff700 1 --2- 172.21.15.16:0/2123893459 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f402806b050 0x7f402806d510 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f402c009c40 tx=0x7f402c0091c0 comp rx=0 tx=0).ready entity=mgr.6396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:15.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.184+0000 7f4038ff9700 1 -- 172.21.15.16:0/2123893459 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f403c0cac90 con 0x7f403c0b4300 2023-10-16T12:26:15.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.475+0000 7f4042969700 1 -- 172.21.15.16:0/2123893459 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]} v 0) v1 -- 0x7f403c0cac50 con 0x7f403c0b4300 2023-10-16T12:26:15.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.923+0000 7f4038ff9700 1 -- 172.21.15.16:0/2123893459 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]}]=0 v567) v1 ==== 154+0+0 (secure 0 0 0) 0x7f4034056340 con 0x7f403c0b4300 2023-10-16T12:26:15.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.926+0000 7f4042969700 1 -- 172.21.15.16:0/2123893459 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f402806b050 msgr2=0x7f402806d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:15.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.926+0000 7f4042969700 1 --2- 172.21.15.16:0/2123893459 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f402806b050 0x7f402806d510 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f402c009c40 tx=0x7f402c0091c0 comp rx=0 tx=0).stop 2023-10-16T12:26:15.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.926+0000 7f4042969700 1 -- 172.21.15.16:0/2123893459 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f403c0b4300 msgr2=0x7f403c0b4780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:15.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.926+0000 7f4042969700 1 --2- 172.21.15.16:0/2123893459 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f403c0b4300 0x7f403c0b4780 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f4034012000 tx=0x7f4034001d00 comp rx=0 tx=0).stop 2023-10-16T12:26:15.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.926+0000 7f4042969700 1 -- 172.21.15.16:0/2123893459 shutdown_connections 2023-10-16T12:26:15.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.926+0000 7f4042969700 1 --2- 172.21.15.16:0/2123893459 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f402806b050 0x7f402806d510 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:15.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.926+0000 7f4042969700 1 --2- 172.21.15.16:0/2123893459 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f403c0b4300 0x7f403c0b4780 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:15.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.926+0000 7f4042969700 1 --2- 172.21.15.16:0/2123893459 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4024004d90 0x7f403c1f2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:15.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.926+0000 7f4042969700 1 --2- 172.21.15.16:0/2123893459 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f403c0b6560 0x7f403c0c95d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:15.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.926+0000 7f4042969700 1 -- 172.21.15.16:0/2123893459 >> 172.21.15.16:0/2123893459 conn(0x7f403c0d14b0 msgr2=0x7f403c15df10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:15.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.926+0000 7f4042969700 1 -- 172.21.15.16:0/2123893459 shutdown_connections 2023-10-16T12:26:15.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:15.926+0000 7f4042969700 1 -- 172.21.15.16:0/2123893459 wait complete. 2023-10-16T12:26:15.942 INFO:tasks.cephfs_test_runner:Test exisitence of multiple scheduled snapshots ... ok 2023-10-16T12:26:15.943 INFO:tasks.cephfs_test_runner:Starting test: test_non_existent_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test listing non-existing snap schedules failure 2023-10-16T12:26:15.943 INFO:tasks.cephfs_test_runner:test_non_existent_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-16T12:26:15.943 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:26:16.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.070+0000 7f01bf004700 1 Processor -- start 2023-10-16T12:26:16.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.070+0000 7f01bf004700 1 -- start start 2023-10-16T12:26:16.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.071+0000 7f01bf004700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f01b806aef0 0x7f01b80cb980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:16.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.071+0000 7f01bf004700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f01b80ce160 0x7f01b80ce610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:16.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.071+0000 7f01bf004700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01b80cedb0 con 0x7f01b815db40 2023-10-16T12:26:16.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.071+0000 7f01bf004700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01b815df20 con 0x7f01b80ce160 2023-10-16T12:26:16.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.071+0000 7f01bf004700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01b80d12c0 con 0x7f01b806aef0 2023-10-16T12:26:16.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.071+0000 7f01bcda0700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f01b815db40 0x7f01b806a7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:56580/0 (socket says 172.21.15.16:56580) 2023-10-16T12:26:16.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.071+0000 7f01bcda0700 1 -- 172.21.15.16:0/1078590786 learned_addr learned my addr 172.21.15.16:0/1078590786 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:26:16.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.071+0000 7f01bd5a1700 1 --2- 172.21.15.16:0/1078590786 >> v2:172.21.15.16:3300/0 conn(0x7f01b80ce160 0x7f01b80ce610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:16.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.072+0000 7f01bd5a1700 1 -- 172.21.15.16:0/1078590786 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f01b806aef0 msgr2=0x7f01b80cb980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:16.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.072+0000 7f01b7fff700 1 --2- 172.21.15.16:0/1078590786 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f01b806aef0 0x7f01b80cb980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:16.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.072+0000 7f01bd5a1700 1 --2- 172.21.15.16:0/1078590786 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f01b806aef0 0x7f01b80cb980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:16.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.072+0000 7f01bd5a1700 1 -- 172.21.15.16:0/1078590786 >> v1:172.21.15.16:6789/0 conn(0x7f01b815db40 legacy=0x7f01b806a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:16.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.072+0000 7f01bd5a1700 1 -- 172.21.15.16:0/1078590786 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01b80ceb50 con 0x7f01b80ce160 2023-10-16T12:26:16.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.072+0000 7f01b77fe700 1 -- 172.21.15.16:0/1078590786 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1950213851 0 0) 0x7f01b80cedb0 con 0x7f01b815db40 2023-10-16T12:26:16.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.072+0000 7f01bd5a1700 1 --2- 172.21.15.16:0/1078590786 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01b80ce160 0x7f01b80ce610 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f01b0009d90 tx=0x7f01b000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92c0ba5c0b1554ab server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:16.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.072+0000 7f01b77fe700 1 -- 172.21.15.16:0/1078590786 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b000e070 con 0x7f01b80ce160 2023-10-16T12:26:16.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.072+0000 7f01b77fe700 1 -- 172.21.15.16:0/1078590786 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01b000bbd0 con 0x7f01b80ce160 2023-10-16T12:26:16.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.072+0000 7f01b77fe700 1 -- 172.21.15.16:0/1078590786 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b0012430 con 0x7f01b80ce160 2023-10-16T12:26:16.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.073+0000 7f01bf004700 1 -- 172.21.15.16:0/1078590786 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01b80ce160 msgr2=0x7f01b80ce610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:16.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.073+0000 7f01bf004700 1 --2- 172.21.15.16:0/1078590786 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01b80ce160 0x7f01b80ce610 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f01b0009d90 tx=0x7f01b000b330 comp rx=0 tx=0).stop 2023-10-16T12:26:16.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.073+0000 7f01bf004700 1 -- 172.21.15.16:0/1078590786 shutdown_connections 2023-10-16T12:26:16.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.073+0000 7f01bf004700 1 --2- 172.21.15.16:0/1078590786 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01b80ce160 0x7f01b80ce610 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:16.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.073+0000 7f01bf004700 1 --2- 172.21.15.16:0/1078590786 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f01b806aef0 0x7f01b80cb980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:16.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.073+0000 7f01bf004700 1 -- 172.21.15.16:0/1078590786 >> 172.21.15.16:0/1078590786 conn(0x7f01b80b1fd0 msgr2=0x7f01b80b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:16.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.073+0000 7f01bf004700 1 -- 172.21.15.16:0/1078590786 shutdown_connections 2023-10-16T12:26:16.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.073+0000 7f01bf004700 1 -- 172.21.15.16:0/1078590786 wait complete. 2023-10-16T12:26:16.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.074+0000 7f01bf004700 1 Processor -- start 2023-10-16T12:26:16.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.074+0000 7f01bf004700 1 -- start start 2023-10-16T12:26:16.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.075+0000 7f01bf004700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01b806aef0 0x7f01b815b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:16.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.075+0000 7f01bf004700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f01b80ce160 0x7f01b80c4120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:16.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.075+0000 7f01bf004700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f01b815db40 0x7f01b814e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:16.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.075+0000 7f01bf004700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01b815aed0 con 0x7f01b806aef0 2023-10-16T12:26:16.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.075+0000 7f01bf004700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01b814ee00 con 0x7f01b815db40 2023-10-16T12:26:16.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.075+0000 7f01bf004700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01b80c4800 con 0x7f01b80ce160 2023-10-16T12:26:16.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.075+0000 7f01bcda0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01b806aef0 0x7f01b815b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:16.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.075+0000 7f01bcda0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01b806aef0 0x7f01b815b070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39080/0 (socket says 172.21.15.16:39080) 2023-10-16T12:26:16.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.075+0000 7f01bcda0700 1 -- 172.21.15.16:0/1704349602 learned_addr learned my addr 172.21.15.16:0/1704349602 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:16.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.075+0000 7f01bd5a1700 1 --2- 172.21.15.16:0/1704349602 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f01b815db40 0x7f01b814e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:16.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.075+0000 7f01b7fff700 1 --2- 172.21.15.16:0/1704349602 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f01b80ce160 0x7f01b80c4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:16.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.075+0000 7f01bcda0700 1 -- 172.21.15.16:0/1704349602 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f01b80ce160 msgr2=0x7f01b80c4120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:16.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.075+0000 7f01bcda0700 1 --2- 172.21.15.16:0/1704349602 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f01b80ce160 0x7f01b80c4120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:16.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.075+0000 7f01bcda0700 1 -- 172.21.15.16:0/1704349602 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f01b815db40 msgr2=0x7f01b814e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:16.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.075+0000 7f01bcda0700 1 --2- 172.21.15.16:0/1704349602 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f01b815db40 0x7f01b814e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:16.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.075+0000 7f01bcda0700 1 -- 172.21.15.16:0/1704349602 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01b0009a70 con 0x7f01b806aef0 2023-10-16T12:26:16.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.075+0000 7f01b7fff700 1 --2- 172.21.15.16:0/1704349602 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f01b80ce160 0x7f01b80c4120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:26:16.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.075+0000 7f01bd5a1700 1 --2- 172.21.15.16:0/1704349602 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f01b815db40 0x7f01b814e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:26:16.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.076+0000 7f01bcda0700 1 --2- 172.21.15.16:0/1704349602 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01b806aef0 0x7f01b815b070 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f01ac0011e0 tx=0x7f01ac00cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:16.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.076+0000 7f01b57fa700 1 -- 172.21.15.16:0/1704349602 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01ac004c80 con 0x7f01b806aef0 2023-10-16T12:26:16.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.076+0000 7f01b57fa700 1 -- 172.21.15.16:0/1704349602 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01ac004e00 con 0x7f01b806aef0 2023-10-16T12:26:16.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.076+0000 7f01bf004700 1 -- 172.21.15.16:0/1704349602 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01b80c4ae0 con 0x7f01b806aef0 2023-10-16T12:26:16.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.076+0000 7f01b57fa700 1 -- 172.21.15.16:0/1704349602 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01ac007310 con 0x7f01b806aef0 2023-10-16T12:26:16.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.076+0000 7f01bf004700 1 -- 172.21.15.16:0/1704349602 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01b80c50a0 con 0x7f01b806aef0 2023-10-16T12:26:16.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.077+0000 7f01bf004700 1 -- 172.21.15.16:0/1704349602 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01b804ebb0 con 0x7f01b806aef0 2023-10-16T12:26:16.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.077+0000 7f01b57fa700 1 -- 172.21.15.16:0/1704349602 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 31) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f01ac007470 con 0x7f01b806aef0 2023-10-16T12:26:16.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.078+0000 7f01b57fa700 1 --2- 172.21.15.16:0/1704349602 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f019c06b020 0x7f019c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:16.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.078+0000 7f01b57fa700 1 -- 172.21.15.16:0/1704349602 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5979+0+0 (secure 0 0 0) 0x7f01ac0896a0 con 0x7f01b806aef0 2023-10-16T12:26:16.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.078+0000 7f01b7fff700 1 --2- 172.21.15.16:0/1704349602 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f019c06b020 0x7f019c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:16.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.078+0000 7f01b7fff700 1 --2- 172.21.15.16:0/1704349602 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f019c06b020 0x7f019c06d4e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f01b80ca0c0 tx=0x7f01a4008040 comp rx=0 tx=0).ready entity=mgr.6396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:16.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.082+0000 7f01b57fa700 1 -- 172.21.15.16:0/1704349602 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f01ac054530 con 0x7f01b806aef0 2023-10-16T12:26:16.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.287+0000 7f01bf004700 1 -- 172.21.15.16:0/1704349602 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f01b806ca50 con 0x7f01b806aef0 2023-10-16T12:26:16.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.288+0000 7f01b57fa700 1 -- 172.21.15.16:0/1704349602 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) v1 ==== 74+0+14125 (secure 0 0 0) 0x7f01ac057b50 con 0x7f01b806aef0 2023-10-16T12:26:16.290 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:26:16.290 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":77,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:26:13.920489+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.49:6834/3246624425":"2023-10-17T12:26:10.916164+0000","172.21.15.49:6832/3246624425":"2023-10-17T12:26:10.916164+0000","172.21.15.16:0/1907743445":"2023-10-17T12:26:09.819640+0000","172.21.15.16:0/1948269493":"2023-10-17T12:26:09.819640+0000","172.21.15.16:0/1785161614":"2023-10-17T12:26:09.819640+0000","172.21.15.16:0/491384204":"2023-10-17T12:26:09.819640+0000","172.21.15.16:6833/4075855741":"2023-10-17T12:26:09.819640+0000","172.21.15.16:6832/4075855741":"2023-10-17T12:26:09.819640+0000","172.21.15.16:0/754053746":"2023-10-17T12:26:09.819640+0000","172.21.15.16:0/2103819518":"2023-10-17T12:24:17.735687+0000","172.21.15.16:0/224613758":"2023-10-17T12:24:17.735687+0000","172.21.15.16:0/877922677":"2023-10-17T12:24:17.735687+0000","172.21.15.16:0/864162051":"2023-10-17T12:26:09.819640+0000","172.21.15.16:0/1135331895":"2023-10-17T12:24:17.735687+0000","172.21.15.16:0/1214553139":"2023-10-17T12:24:17.735687+0000","172.21.15.16:0/1240820252":"2023-10-17T12:24:17.735687+0000","172.21.15.16:6833/3870937344":"2023-10-17T12:24:17.735687+0000","172.21.15.16:6832/3870937344":"2023-10-17T12:24:17.735687+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:26:16.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.290+0000 7f01bf004700 1 -- 172.21.15.16:0/1704349602 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f019c06b020 msgr2=0x7f019c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:16.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.290+0000 7f01bf004700 1 --2- 172.21.15.16:0/1704349602 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f019c06b020 0x7f019c06d4e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f01b80ca0c0 tx=0x7f01a4008040 comp rx=0 tx=0).stop 2023-10-16T12:26:16.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.290+0000 7f01bf004700 1 -- 172.21.15.16:0/1704349602 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01b806aef0 msgr2=0x7f01b815b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:16.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.290+0000 7f01bf004700 1 --2- 172.21.15.16:0/1704349602 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01b806aef0 0x7f01b815b070 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f01ac0011e0 tx=0x7f01ac00cae0 comp rx=0 tx=0).stop 2023-10-16T12:26:16.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.290+0000 7f01bf004700 1 -- 172.21.15.16:0/1704349602 shutdown_connections 2023-10-16T12:26:16.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.290+0000 7f01bf004700 1 --2- 172.21.15.16:0/1704349602 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f019c06b020 0x7f019c06d4e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:16.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.290+0000 7f01bf004700 1 --2- 172.21.15.16:0/1704349602 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f01b815db40 0x7f01b814e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:16.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.290+0000 7f01bf004700 1 --2- 172.21.15.16:0/1704349602 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01b806aef0 0x7f01b815b070 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:16.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.290+0000 7f01bf004700 1 --2- 172.21.15.16:0/1704349602 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f01b80ce160 0x7f01b80c4120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:16.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.290+0000 7f01bf004700 1 -- 172.21.15.16:0/1704349602 >> 172.21.15.16:0/1704349602 conn(0x7f01b80b1fd0 msgr2=0x7f01b80cc1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:16.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.290+0000 7f01bf004700 1 -- 172.21.15.16:0/1704349602 shutdown_connections 2023-10-16T12:26:16.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.290+0000 7f01bf004700 1 -- 172.21.15.16:0/1704349602 wait complete. 2023-10-16T12:26:16.304 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T12:26:16.305 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:26:16.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.431+0000 7faaae331700 1 Processor -- start 2023-10-16T12:26:16.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.431+0000 7faaae331700 1 -- start start 2023-10-16T12:26:16.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.432+0000 7faaae331700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faaa815b070 0x7faaa815b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:16.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.432+0000 7faaae331700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7faaa806e170 0x7faaa80cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:16.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.432+0000 7faaae331700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faaa80cef70 con 0x7faaa80cb880 2023-10-16T12:26:16.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.432+0000 7faaae331700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faaa80ce360 con 0x7faaa806e170 2023-10-16T12:26:16.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.432+0000 7faaae331700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faaa80ce640 con 0x7faaa815b070 2023-10-16T12:26:16.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.432+0000 7faaa7fff700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faaa815b070 0x7faaa815b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:16.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.432+0000 7faaac8ce700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7faaa806e170 0x7faaa80cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:16.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.432+0000 7faaa77fe700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7faaa80cb880 0x7faaa806da60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:35546/0 (socket says 172.21.15.16:35546) 2023-10-16T12:26:16.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.432+0000 7faaa77fe700 1 -- 172.21.15.16:0/2034351756 learned_addr learned my addr 172.21.15.16:0/2034351756 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:26:16.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.432+0000 7faaa6ffd700 1 -- 172.21.15.16:0/2034351756 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3917722954 0 0) 0x7faaa80cef70 con 0x7faaa80cb880 2023-10-16T12:26:16.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.433+0000 7faaa6ffd700 1 -- 172.21.15.16:0/2034351756 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa900035b0 con 0x7faaa80cb880 2023-10-16T12:26:16.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.433+0000 7faaa7fff700 1 -- 172.21.15.16:0/2034351756 >> v2:172.21.15.49:3300/0 conn(0x7faaa806e170 msgr2=0x7faaa80cbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:16.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.433+0000 7faaa7fff700 1 --2- 172.21.15.16:0/2034351756 >> v2:172.21.15.49:3300/0 conn(0x7faaa806e170 0x7faaa80cbd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:16.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.433+0000 7faaa7fff700 1 -- 172.21.15.16:0/2034351756 >> v1:172.21.15.49:6789/0 conn(0x7faaa80cb880 legacy=0x7faaa806da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:16.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.433+0000 7faaa7fff700 1 -- 172.21.15.16:0/2034351756 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaa80ce920 con 0x7faaa815b070 2023-10-16T12:26:16.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.433+0000 7faaac8ce700 1 --2- 172.21.15.16:0/2034351756 >> v2:172.21.15.49:3300/0 conn(0x7faaa806e170 0x7faaa80cbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:26:16.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.433+0000 7faaa7fff700 1 --2- 172.21.15.16:0/2034351756 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faaa815b070 0x7faaa815b450 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7faa98009840 tx=0x7faa98009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da1b66691156c590 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:16.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.433+0000 7faaa6ffd700 1 -- 172.21.15.16:0/2034351756 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa98011830 con 0x7faaa815b070 2023-10-16T12:26:16.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.433+0000 7faaa6ffd700 1 -- 172.21.15.16:0/2034351756 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa98011990 con 0x7faaa815b070 2023-10-16T12:26:16.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.433+0000 7faaa6ffd700 1 -- 172.21.15.16:0/2034351756 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa98011d20 con 0x7faaa815b070 2023-10-16T12:26:16.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.434+0000 7faaae331700 1 -- 172.21.15.16:0/2034351756 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faaa815b070 msgr2=0x7faaa815b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:16.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.434+0000 7faaae331700 1 --2- 172.21.15.16:0/2034351756 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faaa815b070 0x7faaa815b450 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7faa98009840 tx=0x7faa98009b50 comp rx=0 tx=0).stop 2023-10-16T12:26:16.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.434+0000 7faaae331700 1 -- 172.21.15.16:0/2034351756 shutdown_connections 2023-10-16T12:26:16.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.434+0000 7faaae331700 1 --2- 172.21.15.16:0/2034351756 >> v2:172.21.15.49:3300/0 conn(0x7faaa806e170 0x7faaa80cbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:16.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.434+0000 7faaae331700 1 --2- 172.21.15.16:0/2034351756 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faaa815b070 0x7faaa815b450 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:16.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.434+0000 7faaae331700 1 -- 172.21.15.16:0/2034351756 >> 172.21.15.16:0/2034351756 conn(0x7faaa80b1f40 msgr2=0x7faaa80b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:16.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.434+0000 7faaae331700 1 -- 172.21.15.16:0/2034351756 shutdown_connections 2023-10-16T12:26:16.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.434+0000 7faaae331700 1 -- 172.21.15.16:0/2034351756 wait complete. 2023-10-16T12:26:16.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.435+0000 7faaae331700 1 Processor -- start 2023-10-16T12:26:16.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.435+0000 7faaae331700 1 -- start start 2023-10-16T12:26:16.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.436+0000 7faaae331700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faaa806e170 0x7faaa8155400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:16.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.436+0000 7faaae331700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faaa80cb880 0x7faaa81f42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:16.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.436+0000 7faaae331700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faaa815b070 0x7faaa81f4800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:16.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.436+0000 7faaae331700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaa815c9c0 con 0x7faaa806e170 2023-10-16T12:26:16.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.436+0000 7faaae331700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaa815cca0 con 0x7faaa80cb880 2023-10-16T12:26:16.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.436+0000 7faaae331700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faaa815cf80 con 0x7faaa815b070 2023-10-16T12:26:16.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.436+0000 7faaa7fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faaa806e170 0x7faaa8155400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:16.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.436+0000 7faaa7fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faaa806e170 0x7faaa8155400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39094/0 (socket says 172.21.15.16:39094) 2023-10-16T12:26:16.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.436+0000 7faaa7fff700 1 -- 172.21.15.16:0/3209110226 learned_addr learned my addr 172.21.15.16:0/3209110226 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:16.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.436+0000 7faaa77fe700 1 --2- 172.21.15.16:0/3209110226 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faaa80cb880 0x7faaa81f42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:16.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.436+0000 7faaac8ce700 1 --2- 172.21.15.16:0/3209110226 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faaa815b070 0x7faaa81f4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:16.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.436+0000 7faaa7fff700 1 -- 172.21.15.16:0/3209110226 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faaa815b070 msgr2=0x7faaa81f4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:16.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.436+0000 7faaa7fff700 1 --2- 172.21.15.16:0/3209110226 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faaa815b070 0x7faaa81f4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:16.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.436+0000 7faaa7fff700 1 -- 172.21.15.16:0/3209110226 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faaa80cb880 msgr2=0x7faaa81f42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:16.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.436+0000 7faaa7fff700 1 --2- 172.21.15.16:0/3209110226 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faaa80cb880 0x7faaa81f42c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:16.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.436+0000 7faaa7fff700 1 -- 172.21.15.16:0/3209110226 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaa0009a70 con 0x7faaa806e170 2023-10-16T12:26:16.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.436+0000 7faaa7fff700 1 --2- 172.21.15.16:0/3209110226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faaa806e170 0x7faaa8155400 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7faa98000c00 tx=0x7faa98004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:16.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.437+0000 7faaa4ff9700 1 -- 172.21.15.16:0/3209110226 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa98004550 con 0x7faaa806e170 2023-10-16T12:26:16.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.437+0000 7faaa4ff9700 1 -- 172.21.15.16:0/3209110226 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa9801f070 con 0x7faaa806e170 2023-10-16T12:26:16.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.437+0000 7faaa4ff9700 1 -- 172.21.15.16:0/3209110226 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa980038c0 con 0x7faaa806e170 2023-10-16T12:26:16.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.437+0000 7faaae331700 1 -- 172.21.15.16:0/3209110226 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa980094a0 con 0x7faaa806e170 2023-10-16T12:26:16.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.437+0000 7faaae331700 1 -- 172.21.15.16:0/3209110226 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faaa815c5e0 con 0x7faaa806e170 2023-10-16T12:26:16.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.438+0000 7faaae331700 1 -- 172.21.15.16:0/3209110226 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faaa80b39c0 con 0x7faaa806e170 2023-10-16T12:26:16.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.438+0000 7faaa4ff9700 1 -- 172.21.15.16:0/3209110226 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 31) v1 ==== 89331+0+0 (secure 0 0 0) 0x7faa98003a20 con 0x7faaa806e170 2023-10-16T12:26:16.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.438+0000 7faaa4ff9700 1 --2- 172.21.15.16:0/3209110226 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7faa9406b020 0x7faa9406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:16.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.439+0000 7faaa4ff9700 1 -- 172.21.15.16:0/3209110226 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5979+0+0 (secure 0 0 0) 0x7faa98021070 con 0x7faaa806e170 2023-10-16T12:26:16.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.439+0000 7faaa77fe700 1 --2- 172.21.15.16:0/3209110226 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7faa9406b020 0x7faa9406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:16.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.439+0000 7faaa77fe700 1 --2- 172.21.15.16:0/3209110226 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7faa9406b020 0x7faa9406d4e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7faa9c007fd0 tx=0x7faa9c007dc0 comp rx=0 tx=0).ready entity=mgr.6396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:16.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.443+0000 7faaa4ff9700 1 -- 172.21.15.16:0/3209110226 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7faa98055eb0 con 0x7faaa806e170 2023-10-16T12:26:16.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.642+0000 7faaae331700 1 -- 172.21.15.16:0/3209110226 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7faaa815c770 con 0x7faaa806e170 2023-10-16T12:26:16.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.643+0000 7faaa4ff9700 1 -- 172.21.15.16:0/3209110226 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v77) v1 ==== 93+0+10 (secure 0 0 0) 0x7faa9802b0a0 con 0x7faaa806e170 2023-10-16T12:26:16.645 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T12:26:16.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.645+0000 7faaae331700 1 -- 172.21.15.16:0/3209110226 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7faa9406b020 msgr2=0x7faa9406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:16.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.645+0000 7faaae331700 1 --2- 172.21.15.16:0/3209110226 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7faa9406b020 0x7faa9406d4e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7faa9c007fd0 tx=0x7faa9c007dc0 comp rx=0 tx=0).stop 2023-10-16T12:26:16.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.645+0000 7faaae331700 1 -- 172.21.15.16:0/3209110226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faaa806e170 msgr2=0x7faaa8155400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:16.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.645+0000 7faaae331700 1 --2- 172.21.15.16:0/3209110226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faaa806e170 0x7faaa8155400 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7faa98000c00 tx=0x7faa98004750 comp rx=0 tx=0).stop 2023-10-16T12:26:16.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.645+0000 7faaae331700 1 -- 172.21.15.16:0/3209110226 shutdown_connections 2023-10-16T12:26:16.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.645+0000 7faaae331700 1 --2- 172.21.15.16:0/3209110226 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7faa9406b020 0x7faa9406d4e0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:16.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.645+0000 7faaae331700 1 --2- 172.21.15.16:0/3209110226 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faaa815b070 0x7faaa81f4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:16.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.645+0000 7faaae331700 1 --2- 172.21.15.16:0/3209110226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faaa806e170 0x7faaa8155400 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:16.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.645+0000 7faaae331700 1 --2- 172.21.15.16:0/3209110226 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faaa80cb880 0x7faaa81f42c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:16.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.645+0000 7faaae331700 1 -- 172.21.15.16:0/3209110226 >> 172.21.15.16:0/3209110226 conn(0x7faaa80b1f40 msgr2=0x7faaa80d0e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:16.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.645+0000 7faaae331700 1 -- 172.21.15.16:0/3209110226 shutdown_connections 2023-10-16T12:26:16.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.645+0000 7faaae331700 1 -- 172.21.15.16:0/3209110226 wait complete. 2023-10-16T12:26:16.659 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule' 2023-10-16T12:26:16.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.783+0000 7f957831c700 1 Processor -- start 2023-10-16T12:26:16.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.783+0000 7f957831c700 1 -- start start 2023-10-16T12:26:16.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.784+0000 7f957831c700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f957006aef0 0x7f95700b3480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:16.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.784+0000 7f957831c700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95700d11c0 con 0x7f95700b2be0 2023-10-16T12:26:16.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.784+0000 7f957831c700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95700cd360 con 0x7f957015b860 2023-10-16T12:26:16.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.784+0000 7f957831c700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95700d03d0 con 0x7f957006aef0 2023-10-16T12:26:16.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.784+0000 7f95768b9700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f95700b2be0 0x7f95700b39c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:56612/0 (socket says 172.21.15.16:56612) 2023-10-16T12:26:16.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.784+0000 7f95768b9700 1 -- 172.21.15.16:0/3681291516 learned_addr learned my addr 172.21.15.16:0/3681291516 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:26:16.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.784+0000 7f95758b7700 1 --2- 172.21.15.16:0/3681291516 >> v2:172.21.15.49:3300/0 conn(0x7f957006aef0 0x7f95700b3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:16.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.784+0000 7f95750b6700 1 -- 172.21.15.16:0/3681291516 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1830766564 0 0) 0x7f95700d11c0 con 0x7f95700b2be0 2023-10-16T12:26:16.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.784+0000 7f95750b6700 1 -- 172.21.15.16:0/3681291516 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f955c0035b0 con 0x7f95700b2be0 2023-10-16T12:26:16.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.785+0000 7f95750b6700 1 -- 172.21.15.16:0/3681291516 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3310361373 0 0) 0x7f95700cd360 con 0x7f957015b860 2023-10-16T12:26:16.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.785+0000 7f95750b6700 1 -- 172.21.15.16:0/3681291516 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95700d11c0 con 0x7f957015b860 2023-10-16T12:26:16.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.785+0000 7f95758b7700 1 -- 172.21.15.16:0/3681291516 >> v1:172.21.15.49:6789/0 conn(0x7f957015b860 legacy=0x7f957006a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:16.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.785+0000 7f95758b7700 1 -- 172.21.15.16:0/3681291516 >> v1:172.21.15.16:6789/0 conn(0x7f95700b2be0 legacy=0x7f95700b39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:16.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.785+0000 7f95750b6700 1 -- 172.21.15.16:0/3681291516 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1667088378 0 0) 0x7f955c0035b0 con 0x7f95700b2be0 2023-10-16T12:26:16.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.785+0000 7f95758b7700 1 -- 172.21.15.16:0/3681291516 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95700d05b0 con 0x7f957006aef0 2023-10-16T12:26:16.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.785+0000 7f95758b7700 1 --2- 172.21.15.16:0/3681291516 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f957006aef0 0x7f95700b3480 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f9560009840 tx=0x7f9560009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f477b15fea627af server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:16.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.785+0000 7f95750b6700 1 -- 172.21.15.16:0/3681291516 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95680019f0 con 0x7f957006aef0 2023-10-16T12:26:16.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.785+0000 7f95750b6700 1 -- 172.21.15.16:0/3681291516 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9560004030 con 0x7f957006aef0 2023-10-16T12:26:16.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.785+0000 7f95750b6700 1 -- 172.21.15.16:0/3681291516 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95600043a0 con 0x7f957006aef0 2023-10-16T12:26:16.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.786+0000 7f957831c700 1 -- 172.21.15.16:0/3681291516 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f957006aef0 msgr2=0x7f95700b3480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:16.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.786+0000 7f957831c700 1 --2- 172.21.15.16:0/3681291516 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f957006aef0 0x7f95700b3480 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f9560009840 tx=0x7f9560009b50 comp rx=0 tx=0).stop 2023-10-16T12:26:16.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.786+0000 7f957831c700 1 -- 172.21.15.16:0/3681291516 shutdown_connections 2023-10-16T12:26:16.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.786+0000 7f957831c700 1 --2- 172.21.15.16:0/3681291516 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f957006aef0 0x7f95700b3480 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:16.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.786+0000 7f957831c700 1 -- 172.21.15.16:0/3681291516 >> 172.21.15.16:0/3681291516 conn(0x7f95700b1fd0 msgr2=0x7f95700b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:16.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.786+0000 7f957831c700 1 -- 172.21.15.16:0/3681291516 shutdown_connections 2023-10-16T12:26:16.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.786+0000 7f957831c700 1 -- 172.21.15.16:0/3681291516 wait complete. 2023-10-16T12:26:16.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.787+0000 7f957831c700 1 Processor -- start 2023-10-16T12:26:16.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.787+0000 7f957831c700 1 -- start start 2023-10-16T12:26:16.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.788+0000 7f957831c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f957006aef0 0x7f95701509a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:16.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.788+0000 7f957831c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95700b2be0 0x7f95700ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:16.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.788+0000 7f957831c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f957015b860 0x7f95700b4390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:16.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.788+0000 7f957831c700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95700add50 con 0x7f95700b2be0 2023-10-16T12:26:16.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.788+0000 7f957831c700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95700c5230 con 0x7f957015b860 2023-10-16T12:26:16.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.788+0000 7f957831c700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95700ad1d0 con 0x7f957006aef0 2023-10-16T12:26:16.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.788+0000 7f95758b7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95700b2be0 0x7f95700ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:16.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.788+0000 7f95758b7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95700b2be0 0x7f95700ac8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39110/0 (socket says 172.21.15.16:39110) 2023-10-16T12:26:16.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.788+0000 7f95758b7700 1 -- 172.21.15.16:0/3665631077 learned_addr learned my addr 172.21.15.16:0/3665631077 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:16.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.788+0000 7f95768b9700 1 --2- 172.21.15.16:0/3665631077 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f957015b860 0x7f95700b4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:16.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.788+0000 7f95760b8700 1 --2- 172.21.15.16:0/3665631077 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f957006aef0 0x7f95701509a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:16.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.788+0000 7f95758b7700 1 -- 172.21.15.16:0/3665631077 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f957006aef0 msgr2=0x7f95701509a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:16.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.788+0000 7f95758b7700 1 --2- 172.21.15.16:0/3665631077 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f957006aef0 0x7f95701509a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:16.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.788+0000 7f95758b7700 1 -- 172.21.15.16:0/3665631077 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f957015b860 msgr2=0x7f95700b4390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:16.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.788+0000 7f95758b7700 1 --2- 172.21.15.16:0/3665631077 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f957015b860 0x7f95700b4390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:16.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.788+0000 7f95758b7700 1 -- 172.21.15.16:0/3665631077 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95600094a0 con 0x7f95700b2be0 2023-10-16T12:26:16.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.788+0000 7f95768b9700 1 --2- 172.21.15.16:0/3665631077 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f957015b860 0x7f95700b4390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:26:16.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.788+0000 7f95758b7700 1 --2- 172.21.15.16:0/3665631077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95700b2be0 0x7f95700ac8d0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f95600090b0 tx=0x7f9560004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:16.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.789+0000 7f9566ffd700 1 -- 172.21.15.16:0/3665631077 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9560004590 con 0x7f95700b2be0 2023-10-16T12:26:16.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.789+0000 7f9566ffd700 1 -- 172.21.15.16:0/3665631077 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f956001f070 con 0x7f95700b2be0 2023-10-16T12:26:16.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.789+0000 7f957831c700 1 -- 172.21.15.16:0/3665631077 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95700ad4b0 con 0x7f95700b2be0 2023-10-16T12:26:16.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.789+0000 7f9566ffd700 1 -- 172.21.15.16:0/3665631077 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9560011870 con 0x7f95700b2be0 2023-10-16T12:26:16.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.789+0000 7f957831c700 1 -- 172.21.15.16:0/3665631077 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95700ada70 con 0x7f95700b2be0 2023-10-16T12:26:16.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.789+0000 7f957831c700 1 -- 172.21.15.16:0/3665631077 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95700c5430 con 0x7f95700b2be0 2023-10-16T12:26:16.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.790+0000 7f9566ffd700 1 -- 172.21.15.16:0/3665631077 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 31) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f95600119d0 con 0x7f95700b2be0 2023-10-16T12:26:16.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.790+0000 7f9566ffd700 1 --2- 172.21.15.16:0/3665631077 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f955406b020 0x7f955406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:16.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.791+0000 7f9566ffd700 1 -- 172.21.15.16:0/3665631077 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5979+0+0 (secure 0 0 0) 0x7f9560021070 con 0x7f95700b2be0 2023-10-16T12:26:16.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.791+0000 7f95760b8700 1 --2- 172.21.15.16:0/3665631077 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f955406b020 0x7f955406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:16.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.791+0000 7f95760b8700 1 --2- 172.21.15.16:0/3665631077 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f955406b020 0x7f955406d4e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f956c001650 tx=0x7f956c009040 comp rx=0 tx=0).ready entity=mgr.6396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:16.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:16.795+0000 7f9566ffd700 1 -- 172.21.15.16:0/3665631077 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f95700c5430 con 0x7f95700b2be0 2023-10-16T12:26:17.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:17.084+0000 7f957831c700 1 -- 172.21.15.16:0/3665631077 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]} v 0) v1 -- 0x7f95700c5430 con 0x7f95700b2be0 2023-10-16T12:26:17.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:17.929+0000 7f9566ffd700 1 -- 172.21.15.16:0/3665631077 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]}]=0 v569) v1 ==== 159+0+0 (secure 0 0 0) 0x7f95700c5430 con 0x7f95700b2be0 2023-10-16T12:26:17.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:17.932+0000 7f957831c700 1 -- 172.21.15.16:0/3665631077 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f955406b020 msgr2=0x7f955406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:17.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:17.932+0000 7f957831c700 1 --2- 172.21.15.16:0/3665631077 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f955406b020 0x7f955406d4e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f956c001650 tx=0x7f956c009040 comp rx=0 tx=0).stop 2023-10-16T12:26:17.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:17.932+0000 7f957831c700 1 -- 172.21.15.16:0/3665631077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95700b2be0 msgr2=0x7f95700ac8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:17.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:17.932+0000 7f957831c700 1 --2- 172.21.15.16:0/3665631077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95700b2be0 0x7f95700ac8d0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f95600090b0 tx=0x7f9560004750 comp rx=0 tx=0).stop 2023-10-16T12:26:17.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:17.932+0000 7f957831c700 1 -- 172.21.15.16:0/3665631077 shutdown_connections 2023-10-16T12:26:17.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:17.932+0000 7f957831c700 1 --2- 172.21.15.16:0/3665631077 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f955406b020 0x7f955406d4e0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:17.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:17.932+0000 7f957831c700 1 --2- 172.21.15.16:0/3665631077 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f957015b860 0x7f95700b4390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:17.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:17.932+0000 7f957831c700 1 --2- 172.21.15.16:0/3665631077 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f957006aef0 0x7f95701509a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:17.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:17.932+0000 7f957831c700 1 --2- 172.21.15.16:0/3665631077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95700b2be0 0x7f95700ac8d0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:17.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:17.932+0000 7f957831c700 1 -- 172.21.15.16:0/3665631077 >> 172.21.15.16:0/3665631077 conn(0x7f95700b1fd0 msgr2=0x7f95700718e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:17.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:17.932+0000 7f957831c700 1 -- 172.21.15.16:0/3665631077 shutdown_connections 2023-10-16T12:26:17.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:17.932+0000 7f957831c700 1 -- 172.21.15.16:0/3665631077 wait complete. 2023-10-16T12:26:17.949 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-16T12:26:18.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.082+0000 7f5f5a9f1700 1 Processor -- start 2023-10-16T12:26:18.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.082+0000 7f5f5a9f1700 1 -- start start 2023-10-16T12:26:18.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.083+0000 7f5f5a9f1700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f5f5406beb0 0x7f5f540ce890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:18.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.083+0000 7f5f5a9f1700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f5406aa10 con 0x7f5f540af210 2023-10-16T12:26:18.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.083+0000 7f5f5a9f1700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f540ceed0 con 0x7f5f54160ee0 2023-10-16T12:26:18.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.083+0000 7f5f5a9f1700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f540cf770 con 0x7f5f5406beb0 2023-10-16T12:26:18.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.083+0000 7f5f537fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f5f540af210 0x7f5f540ce180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:56624/0 (socket says 172.21.15.16:56624) 2023-10-16T12:26:18.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.083+0000 7f5f537fe700 1 -- 172.21.15.16:0/2787880141 learned_addr learned my addr 172.21.15.16:0/2787880141 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:26:18.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.083+0000 7f5f58f8e700 1 --2- 172.21.15.16:0/2787880141 >> v2:172.21.15.49:3300/0 conn(0x7f5f5406beb0 0x7f5f540ce890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:18.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.083+0000 7f5f52ffd700 1 -- 172.21.15.16:0/2787880141 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1056197442 0 0) 0x7f5f5406aa10 con 0x7f5f540af210 2023-10-16T12:26:18.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.083+0000 7f5f52ffd700 1 -- 172.21.15.16:0/2787880141 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f3c0035b0 con 0x7f5f540af210 2023-10-16T12:26:18.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.083+0000 7f5f52ffd700 1 -- 172.21.15.16:0/2787880141 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4258600672 0 0) 0x7f5f3c0035b0 con 0x7f5f540af210 2023-10-16T12:26:18.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.083+0000 7f5f52ffd700 1 -- 172.21.15.16:0/2787880141 >> v2:172.21.15.49:3300/0 conn(0x7f5f5406beb0 msgr2=0x7f5f540ce890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:18.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.083+0000 7f5f52ffd700 1 --2- 172.21.15.16:0/2787880141 >> v2:172.21.15.49:3300/0 conn(0x7f5f5406beb0 0x7f5f540ce890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:18.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.083+0000 7f5f52ffd700 1 -- 172.21.15.16:0/2787880141 >> v1:172.21.15.49:6789/0 conn(0x7f5f54160ee0 legacy=0x7f5f540aeb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:18.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.083+0000 7f5f52ffd700 1 -- 172.21.15.16:0/2787880141 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f540cf1b0 con 0x7f5f540af210 2023-10-16T12:26:18.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.083+0000 7f5f52ffd700 1 -- 172.21.15.16:0/2787880141 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f5f48001670 con 0x7f5f540af210 2023-10-16T12:26:18.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.083+0000 7f5f52ffd700 1 -- 172.21.15.16:0/2787880141 >> v1:172.21.15.16:6789/0 conn(0x7f5f540af210 legacy=0x7f5f540ce180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:18.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.083+0000 7f5f52ffd700 1 --2- 172.21.15.16:0/2787880141 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5f3c003a00 0x7f5f3c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:18.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.083+0000 7f5f52ffd700 1 --2- 172.21.15.16:0/2787880141 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f3c004400 0x7f5f3c0048d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:18.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.083+0000 7f5f52ffd700 1 --2- 172.21.15.16:0/2787880141 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5f3c0050d0 0x7f5f3c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:18.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.083+0000 7f5f52ffd700 1 -- 172.21.15.16:0/2787880141 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f540cf770 con 0x7f5f3c004400 2023-10-16T12:26:18.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.083+0000 7f5f52ffd700 1 -- 172.21.15.16:0/2787880141 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f3c00c030 con 0x7f5f3c0050d0 2023-10-16T12:26:18.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.083+0000 7f5f52ffd700 1 -- 172.21.15.16:0/2787880141 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f3c00c1f0 con 0x7f5f3c003a00 2023-10-16T12:26:18.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.084+0000 7f5f537fe700 1 --2- 172.21.15.16:0/2787880141 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f3c004400 0x7f5f3c0048d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:18.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.084+0000 7f5f53fff700 1 --2- 172.21.15.16:0/2787880141 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5f3c003a00 0x7f5f3c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:18.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.084+0000 7f5f58f8e700 1 --2- 172.21.15.16:0/2787880141 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5f3c0050d0 0x7f5f3c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:18.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.084+0000 7f5f537fe700 1 -- 172.21.15.16:0/2787880141 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5f3c003a00 msgr2=0x7f5f3c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:18.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.084+0000 7f5f537fe700 1 --2- 172.21.15.16:0/2787880141 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5f3c003a00 0x7f5f3c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:18.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.084+0000 7f5f537fe700 1 -- 172.21.15.16:0/2787880141 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5f3c0050d0 msgr2=0x7f5f3c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:18.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.084+0000 7f5f537fe700 1 --2- 172.21.15.16:0/2787880141 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5f3c0050d0 0x7f5f3c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:18.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.084+0000 7f5f58f8e700 1 -- 172.21.15.16:0/2787880141 reap_dead start 2023-10-16T12:26:18.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.084+0000 7f5f537fe700 1 -- 172.21.15.16:0/2787880141 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f3c0035b0 con 0x7f5f3c004400 2023-10-16T12:26:18.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.084+0000 7f5f537fe700 1 --2- 172.21.15.16:0/2787880141 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f3c004400 0x7f5f3c0048d0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f5f4800c2f0 tx=0x7f5f3c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4553fffc86478233 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:18.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.084+0000 7f5f52ffd700 1 -- 172.21.15.16:0/2787880141 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f4800cdb0 con 0x7f5f3c004400 2023-10-16T12:26:18.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.084+0000 7f5f52ffd700 1 -- 172.21.15.16:0/2787880141 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f48005960 con 0x7f5f3c004400 2023-10-16T12:26:18.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.084+0000 7f5f52ffd700 1 -- 172.21.15.16:0/2787880141 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f48005d10 con 0x7f5f3c004400 2023-10-16T12:26:18.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.085+0000 7f5f5a9f1700 1 -- 172.21.15.16:0/2787880141 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f3c004400 msgr2=0x7f5f3c0048d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:18.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.085+0000 7f5f5a9f1700 1 --2- 172.21.15.16:0/2787880141 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f3c004400 0x7f5f3c0048d0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f5f4800c2f0 tx=0x7f5f3c005880 comp rx=0 tx=0).stop 2023-10-16T12:26:18.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.085+0000 7f5f5a9f1700 1 -- 172.21.15.16:0/2787880141 shutdown_connections 2023-10-16T12:26:18.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.085+0000 7f5f5a9f1700 1 --2- 172.21.15.16:0/2787880141 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f3c004400 0x7f5f3c0048d0 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:18.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.085+0000 7f5f5a9f1700 1 -- 172.21.15.16:0/2787880141 >> 172.21.15.16:0/2787880141 conn(0x7f5f540cc5e0 msgr2=0x7f5f540cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:18.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.085+0000 7f5f5a9f1700 1 -- 172.21.15.16:0/2787880141 shutdown_connections 2023-10-16T12:26:18.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.085+0000 7f5f5a9f1700 1 -- 172.21.15.16:0/2787880141 wait complete. 2023-10-16T12:26:18.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.086+0000 7f5f5a9f1700 1 Processor -- start 2023-10-16T12:26:18.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.086+0000 7f5f5a9f1700 1 -- start start 2023-10-16T12:26:18.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.087+0000 7f5f5a9f1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f3c004400 0x7f5f540c3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:18.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.087+0000 7f5f5a9f1700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5f540c8150 0x7f5f540c5a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:18.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.087+0000 7f5f5a9f1700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5f540c6c30 0x7f5f540c70a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:18.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.087+0000 7f5f5a9f1700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f540c3f90 con 0x7f5f3c004400 2023-10-16T12:26:18.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.087+0000 7f5f5a9f1700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f540c4270 con 0x7f5f540c6c30 2023-10-16T12:26:18.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.087+0000 7f5f5a9f1700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f540c4550 con 0x7f5f540c8150 2023-10-16T12:26:18.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.087+0000 7f5f53fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f3c004400 0x7f5f540c3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:18.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.087+0000 7f5f53fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f3c004400 0x7f5f540c3690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39128/0 (socket says 172.21.15.16:39128) 2023-10-16T12:26:18.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.087+0000 7f5f53fff700 1 -- 172.21.15.16:0/3442341462 learned_addr learned my addr 172.21.15.16:0/3442341462 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:18.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.087+0000 7f5f537fe700 1 --2- 172.21.15.16:0/3442341462 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5f540c8150 0x7f5f540c5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:18.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.087+0000 7f5f58f8e700 1 --2- 172.21.15.16:0/3442341462 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5f540c6c30 0x7f5f540c70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:18.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.087+0000 7f5f53fff700 1 -- 172.21.15.16:0/3442341462 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5f540c8150 msgr2=0x7f5f540c5a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:18.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.087+0000 7f5f53fff700 1 --2- 172.21.15.16:0/3442341462 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5f540c8150 0x7f5f540c5a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:18.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.087+0000 7f5f53fff700 1 -- 172.21.15.16:0/3442341462 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5f540c6c30 msgr2=0x7f5f540c70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:18.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.087+0000 7f5f53fff700 1 --2- 172.21.15.16:0/3442341462 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5f540c6c30 0x7f5f540c70a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:18.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.087+0000 7f5f53fff700 1 -- 172.21.15.16:0/3442341462 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f540c4830 con 0x7f5f3c004400 2023-10-16T12:26:18.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.087+0000 7f5f58f8e700 1 --2- 172.21.15.16:0/3442341462 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5f540c6c30 0x7f5f540c70a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:26:18.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.087+0000 7f5f53fff700 1 --2- 172.21.15.16:0/3442341462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f3c004400 0x7f5f540c3690 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f5f4400f780 tx=0x7f5f4400fa90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:18.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.088+0000 7f5f50ff9700 1 -- 172.21.15.16:0/3442341462 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f440052e0 con 0x7f5f3c004400 2023-10-16T12:26:18.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.088+0000 7f5f50ff9700 1 -- 172.21.15.16:0/3442341462 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f44005460 con 0x7f5f3c004400 2023-10-16T12:26:18.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.088+0000 7f5f50ff9700 1 -- 172.21.15.16:0/3442341462 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f440038c0 con 0x7f5f3c004400 2023-10-16T12:26:18.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.088+0000 7f5f5a9f1700 1 -- 172.21.15.16:0/3442341462 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f540c4b10 con 0x7f5f3c004400 2023-10-16T12:26:18.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.088+0000 7f5f5a9f1700 1 -- 172.21.15.16:0/3442341462 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f540c3bd0 con 0x7f5f3c004400 2023-10-16T12:26:18.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.089+0000 7f5f5a9f1700 1 -- 172.21.15.16:0/3442341462 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f540c69e0 con 0x7f5f3c004400 2023-10-16T12:26:18.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.089+0000 7f5f50ff9700 1 -- 172.21.15.16:0/3442341462 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 31) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5f44003a20 con 0x7f5f3c004400 2023-10-16T12:26:18.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.089+0000 7f5f50ff9700 1 --2- 172.21.15.16:0/3442341462 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f5f4006b020 0x7f5f4006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:18.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.090+0000 7f5f50ff9700 1 -- 172.21.15.16:0/3442341462 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5979+0+0 (secure 0 0 0) 0x7f5f440895e0 con 0x7f5f3c004400 2023-10-16T12:26:18.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.090+0000 7f5f537fe700 1 --2- 172.21.15.16:0/3442341462 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f5f4006b020 0x7f5f4006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:18.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.090+0000 7f5f537fe700 1 --2- 172.21.15.16:0/3442341462 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f5f4006b020 0x7f5f4006d4e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5f48008fd0 tx=0x7f5f48007000 comp rx=0 tx=0).ready entity=mgr.6396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:18.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.094+0000 7f5f50ff9700 1 -- 172.21.15.16:0/3442341462 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5f44054500 con 0x7f5f3c004400 2023-10-16T12:26:18.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.298+0000 7f5f5a9f1700 1 -- 172.21.15.16:0/3442341462 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f5f5404ebb0 con 0x7f5f3c004400 2023-10-16T12:26:18.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.301+0000 7f5f50ff9700 1 -- 172.21.15.16:0/3442341462 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v22)=0 v22) v1 ==== 122+0+0 (secure 0 0 0) 0x7f5f44057b20 con 0x7f5f3c004400 2023-10-16T12:26:18.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.303+0000 7f5f5a9f1700 1 -- 172.21.15.16:0/3442341462 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f5f4006b020 msgr2=0x7f5f4006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:18.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.303+0000 7f5f5a9f1700 1 --2- 172.21.15.16:0/3442341462 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f5f4006b020 0x7f5f4006d4e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5f48008fd0 tx=0x7f5f48007000 comp rx=0 tx=0).stop 2023-10-16T12:26:18.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.303+0000 7f5f5a9f1700 1 -- 172.21.15.16:0/3442341462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f3c004400 msgr2=0x7f5f540c3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:18.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.303+0000 7f5f5a9f1700 1 --2- 172.21.15.16:0/3442341462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f3c004400 0x7f5f540c3690 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f5f4400f780 tx=0x7f5f4400fa90 comp rx=0 tx=0).stop 2023-10-16T12:26:18.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.303+0000 7f5f5a9f1700 1 -- 172.21.15.16:0/3442341462 shutdown_connections 2023-10-16T12:26:18.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.303+0000 7f5f5a9f1700 1 --2- 172.21.15.16:0/3442341462 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f5f4006b020 0x7f5f4006d4e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:18.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.303+0000 7f5f5a9f1700 1 --2- 172.21.15.16:0/3442341462 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5f540c6c30 0x7f5f540c70a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:18.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.303+0000 7f5f5a9f1700 1 --2- 172.21.15.16:0/3442341462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f3c004400 0x7f5f540c3690 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:18.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.303+0000 7f5f5a9f1700 1 --2- 172.21.15.16:0/3442341462 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5f540c8150 0x7f5f540c5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:18.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.303+0000 7f5f5a9f1700 1 -- 172.21.15.16:0/3442341462 >> 172.21.15.16:0/3442341462 conn(0x7f5f540cc5e0 msgr2=0x7f5f540ce110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:18.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.304+0000 7f5f5a9f1700 1 -- 172.21.15.16:0/3442341462 shutdown_connections 2023-10-16T12:26:18.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.304+0000 7f5f5a9f1700 1 -- 172.21.15.16:0/3442341462 wait complete. 2023-10-16T12:26:18.317 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T12:26:18.318 DEBUG:teuthology.orchestra.run.smithi016:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T12:26:18.356 DEBUG:teuthology.orchestra.run.smithi049:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T12:26:18.404 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T12:26:18.404 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:26:18.405 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:26:18.422 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16287364k,nr_inodes=4071841,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20251 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3261172k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T12:26:18.423 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T12:26:18.423 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:26:18.443 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284204k,nr_inodes=4071051,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=753 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260540k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T12:26:18.443 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:26:18.444 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:26:18.480 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16287364k,nr_inodes=4071841,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20251 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3261172k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T12:26:18.480 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T12:26:18.480 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:26:18.502 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284204k,nr_inodes=4071051,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=753 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260540k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T12:26:18.502 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-16T12:26:18.502 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:26:18.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.643+0000 7fe872917700 1 Processor -- start 2023-10-16T12:26:18.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.643+0000 7fe872917700 1 -- start start 2023-10-16T12:26:18.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.644+0000 7fe872917700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe86c06aef0 0x7fe86c0cb980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:18.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.644+0000 7fe872917700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe86c0ce160 0x7fe86c0ce610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:18.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.644+0000 7fe872917700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe86c0cedb0 con 0x7fe86c15db40 2023-10-16T12:26:18.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.644+0000 7fe872917700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe86c15df20 con 0x7fe86c06aef0 2023-10-16T12:26:18.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.644+0000 7fe872917700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe86c0d12c0 con 0x7fe86c0ce160 2023-10-16T12:26:18.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.644+0000 7fe86b7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe86c06aef0 0x7fe86c0cb980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:18.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.644+0000 7fe86b7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe86c06aef0 0x7fe86c0cb980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39138/0 (socket says 172.21.15.16:39138) 2023-10-16T12:26:18.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.644+0000 7fe86b7fe700 1 -- 172.21.15.16:0/3048926739 learned_addr learned my addr 172.21.15.16:0/3048926739 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:18.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.644+0000 7fe870eb4700 1 --2- 172.21.15.16:0/3048926739 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe86c0ce160 0x7fe86c0ce610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:18.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.644+0000 7fe86b7fe700 1 -- 172.21.15.16:0/3048926739 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe86c0ce160 msgr2=0x7fe86c0ce610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:18.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.644+0000 7fe86b7fe700 1 --2- 172.21.15.16:0/3048926739 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe86c0ce160 0x7fe86c0ce610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:18.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.644+0000 7fe86b7fe700 1 -- 172.21.15.16:0/3048926739 >> v1:172.21.15.49:6789/0 conn(0x7fe86c15db40 legacy=0x7fe86c06a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:18.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.644+0000 7fe86b7fe700 1 -- 172.21.15.16:0/3048926739 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe86c0ceb50 con 0x7fe86c06aef0 2023-10-16T12:26:18.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.644+0000 7fe86b7fe700 1 --2- 172.21.15.16:0/3048926739 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe86c06aef0 0x7fe86c0cb980 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fe8600098f0 tx=0x7fe860009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=634199984290dbfe server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:18.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.644+0000 7fe86affd700 1 -- 172.21.15.16:0/3048926739 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe860011830 con 0x7fe86c06aef0 2023-10-16T12:26:18.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.644+0000 7fe86affd700 1 -- 172.21.15.16:0/3048926739 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe860011990 con 0x7fe86c06aef0 2023-10-16T12:26:18.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.644+0000 7fe86affd700 1 -- 172.21.15.16:0/3048926739 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe860011d20 con 0x7fe86c06aef0 2023-10-16T12:26:18.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.645+0000 7fe872917700 1 -- 172.21.15.16:0/3048926739 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe86c06aef0 msgr2=0x7fe86c0cb980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:18.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.645+0000 7fe872917700 1 --2- 172.21.15.16:0/3048926739 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe86c06aef0 0x7fe86c0cb980 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fe8600098f0 tx=0x7fe860009c00 comp rx=0 tx=0).stop 2023-10-16T12:26:18.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.645+0000 7fe872917700 1 -- 172.21.15.16:0/3048926739 shutdown_connections 2023-10-16T12:26:18.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.645+0000 7fe872917700 1 --2- 172.21.15.16:0/3048926739 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe86c0ce160 0x7fe86c0ce610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:18.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.645+0000 7fe872917700 1 --2- 172.21.15.16:0/3048926739 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe86c06aef0 0x7fe86c0cb980 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:18.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.645+0000 7fe872917700 1 -- 172.21.15.16:0/3048926739 >> 172.21.15.16:0/3048926739 conn(0x7fe86c0b1fd0 msgr2=0x7fe86c0b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:18.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.645+0000 7fe872917700 1 -- 172.21.15.16:0/3048926739 shutdown_connections 2023-10-16T12:26:18.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.645+0000 7fe872917700 1 -- 172.21.15.16:0/3048926739 wait complete. 2023-10-16T12:26:18.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.646+0000 7fe872917700 1 Processor -- start 2023-10-16T12:26:18.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.646+0000 7fe872917700 1 -- start start 2023-10-16T12:26:18.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.647+0000 7fe872917700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe86c06aef0 0x7fe86c0ac960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:18.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.647+0000 7fe872917700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe86c0ce160 0x7fe86c0acea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:18.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.647+0000 7fe872917700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe86c15db40 0x7fe86c1f63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:18.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.647+0000 7fe872917700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe86c0ae330 con 0x7fe86c06aef0 2023-10-16T12:26:18.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.647+0000 7fe872917700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe86c0ad720 con 0x7fe86c15db40 2023-10-16T12:26:18.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.647+0000 7fe872917700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe86c0ada00 con 0x7fe86c0ce160 2023-10-16T12:26:18.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.647+0000 7fe86bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe86c06aef0 0x7fe86c0ac960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:18.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.647+0000 7fe86bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe86c06aef0 0x7fe86c0ac960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39142/0 (socket says 172.21.15.16:39142) 2023-10-16T12:26:18.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.647+0000 7fe86bfff700 1 -- 172.21.15.16:0/939556420 learned_addr learned my addr 172.21.15.16:0/939556420 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:18.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.648+0000 7fe86bfff700 1 -- 172.21.15.16:0/939556420 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe86c0ce160 msgr2=0x7fe86c0acea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:18.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.648+0000 7fe86b7fe700 1 --2- 172.21.15.16:0/939556420 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe86c0ce160 0x7fe86c0acea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:18.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.648+0000 7fe870eb4700 1 --2- 172.21.15.16:0/939556420 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe86c15db40 0x7fe86c1f63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:18.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.648+0000 7fe86bfff700 1 --2- 172.21.15.16:0/939556420 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe86c0ce160 0x7fe86c0acea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:18.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.648+0000 7fe86bfff700 1 -- 172.21.15.16:0/939556420 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe86c15db40 msgr2=0x7fe86c1f63e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:18.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.648+0000 7fe86bfff700 1 --2- 172.21.15.16:0/939556420 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe86c15db40 0x7fe86c1f63e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:18.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.648+0000 7fe86bfff700 1 -- 172.21.15.16:0/939556420 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe860009550 con 0x7fe86c06aef0 2023-10-16T12:26:18.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.648+0000 7fe86bfff700 1 --2- 172.21.15.16:0/939556420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe86c06aef0 0x7fe86c0ac960 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fe85c00c4e0 tx=0x7fe85c00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:18.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.648+0000 7fe868ff9700 1 -- 172.21.15.16:0/939556420 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe85c0052d0 con 0x7fe86c06aef0 2023-10-16T12:26:18.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.648+0000 7fe868ff9700 1 -- 172.21.15.16:0/939556420 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe85c005430 con 0x7fe86c06aef0 2023-10-16T12:26:18.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.648+0000 7fe868ff9700 1 -- 172.21.15.16:0/939556420 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe85c0038c0 con 0x7fe86c06aef0 2023-10-16T12:26:18.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.648+0000 7fe872917700 1 -- 172.21.15.16:0/939556420 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe86c0adce0 con 0x7fe86c06aef0 2023-10-16T12:26:18.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.648+0000 7fe872917700 1 -- 172.21.15.16:0/939556420 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe86c0ad3e0 con 0x7fe86c06aef0 2023-10-16T12:26:18.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.649+0000 7fe872917700 1 -- 172.21.15.16:0/939556420 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe86c04ebb0 con 0x7fe86c06aef0 2023-10-16T12:26:18.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.650+0000 7fe868ff9700 1 -- 172.21.15.16:0/939556420 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 31) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe85c003a20 con 0x7fe86c06aef0 2023-10-16T12:26:18.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.650+0000 7fe868ff9700 1 --2- 172.21.15.16:0/939556420 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7fe85806b020 0x7fe85806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:18.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.650+0000 7fe868ff9700 1 -- 172.21.15.16:0/939556420 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5979+0+0 (secure 0 0 0) 0x7fe85c088cb0 con 0x7fe86c06aef0 2023-10-16T12:26:18.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.650+0000 7fe86b7fe700 1 --2- 172.21.15.16:0/939556420 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7fe85806b020 0x7fe85806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:18.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.650+0000 7fe86b7fe700 1 --2- 172.21.15.16:0/939556420 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7fe85806b020 0x7fe85806d4e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe86c15aa60 tx=0x7fe860008030 comp rx=0 tx=0).ready entity=mgr.6396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:18.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.655+0000 7fe868ff9700 1 -- 172.21.15.16:0/939556420 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe85c053bd0 con 0x7fe86c06aef0 2023-10-16T12:26:18.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.904+0000 7fe872917700 1 -- 172.21.15.16:0/939556420 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe86c0cb880 con 0x7fe86c06aef0 2023-10-16T12:26:18.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.904+0000 7fe868ff9700 1 -- 172.21.15.16:0/939556420 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+3699 (secure 0 0 0) 0x7fe85c017090 con 0x7fe86c06aef0 2023-10-16T12:26:18.907 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:26:18.907 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":29,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5947,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/1103417847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":1103417847},{"type":"v1","addr":"172.21.15.49:6835","nonce":1103417847}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":6096,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/1380792179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1380792179},{"type":"v1","addr":"172.21.15.16:6836","nonce":1380792179}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":6099,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2238585443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":2238585443},{"type":"v1","addr":"172.21.15.16:6837","nonce":2238585443}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":6292,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/3292702790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":3292702790},{"type":"v1","addr":"172.21.15.49:6834","nonce":3292702790}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[]} 2023-10-16T12:26:18.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.906+0000 7fe872917700 1 -- 172.21.15.16:0/939556420 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7fe85806b020 msgr2=0x7fe85806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:18.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.906+0000 7fe872917700 1 --2- 172.21.15.16:0/939556420 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7fe85806b020 0x7fe85806d4e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe86c15aa60 tx=0x7fe860008030 comp rx=0 tx=0).stop 2023-10-16T12:26:18.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.906+0000 7fe872917700 1 -- 172.21.15.16:0/939556420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe86c06aef0 msgr2=0x7fe86c0ac960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:18.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.906+0000 7fe872917700 1 --2- 172.21.15.16:0/939556420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe86c06aef0 0x7fe86c0ac960 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fe85c00c4e0 tx=0x7fe85c00c8a0 comp rx=0 tx=0).stop 2023-10-16T12:26:18.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.906+0000 7fe872917700 1 -- 172.21.15.16:0/939556420 shutdown_connections 2023-10-16T12:26:18.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.906+0000 7fe872917700 1 --2- 172.21.15.16:0/939556420 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7fe85806b020 0x7fe85806d4e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:18.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.906+0000 7fe872917700 1 --2- 172.21.15.16:0/939556420 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe86c15db40 0x7fe86c1f63e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:18.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.906+0000 7fe872917700 1 --2- 172.21.15.16:0/939556420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe86c06aef0 0x7fe86c0ac960 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:18.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.906+0000 7fe872917700 1 --2- 172.21.15.16:0/939556420 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe86c0ce160 0x7fe86c0acea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:18.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.906+0000 7fe872917700 1 -- 172.21.15.16:0/939556420 >> 172.21.15.16:0/939556420 conn(0x7fe86c0b1fd0 msgr2=0x7fe86c0cca80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:18.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.906+0000 7fe872917700 1 -- 172.21.15.16:0/939556420 shutdown_connections 2023-10-16T12:26:18.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:18.906+0000 7fe872917700 1 -- 172.21.15.16:0/939556420 wait complete. 2023-10-16T12:26:18.911 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 29 2023-10-16T12:26:18.921 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-16T12:26:18.921 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-16T12:26:18.921 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-16T12:26:18.922 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T12:26:18.922 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-16T12:26:18.922 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-16T12:26:18.923 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-16T12:26:18.923 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T12:26:18.923 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-16T12:26:18.923 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-16T12:26:18.924 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-16T12:26:18.924 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T12:26:18.924 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-16T12:26:18.925 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-16T12:26:18.925 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-16T12:26:18.925 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T12:26:18.931 INFO:tasks.ceph.mds.a:Stopped 2023-10-16T12:26:18.931 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-16T12:26:18.932 INFO:tasks.ceph.mds.c:Stopped 2023-10-16T12:26:18.932 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-16T12:26:18.936 INFO:tasks.ceph.mds.b:Stopped 2023-10-16T12:26:18.936 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-16T12:26:18.937 INFO:tasks.ceph.mds.d:Stopped 2023-10-16T12:26:18.937 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-16T12:26:18.974 INFO:tasks.ceph.mds.a:Started 2023-10-16T12:26:18.975 DEBUG:teuthology.parallel:result is None 2023-10-16T12:26:18.975 INFO:tasks.ceph.mds.c:Started 2023-10-16T12:26:18.976 DEBUG:teuthology.parallel:result is None 2023-10-16T12:26:18.979 INFO:tasks.ceph.mds.b:Started 2023-10-16T12:26:18.979 DEBUG:teuthology.parallel:result is None 2023-10-16T12:26:18.980 INFO:tasks.ceph.mds.d:Started 2023-10-16T12:26:18.980 DEBUG:teuthology.parallel:result is None 2023-10-16T12:26:18.981 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-16T12:26:19.056 INFO:tasks.ceph.mds.c.smithi016.stdout:starting mds.c at 2023-10-16T12:26:19.057 INFO:tasks.ceph.mds.a.smithi016.stdout:starting mds.a at 2023-10-16T12:26:19.067 INFO:tasks.ceph.mds.d.smithi049.stdout:starting mds.d at 2023-10-16T12:26:19.068 INFO:tasks.ceph.mds.b.smithi049.stdout:starting mds.b at 2023-10-16T12:26:19.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.110+0000 7f3c5b687700 1 Processor -- start 2023-10-16T12:26:19.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.110+0000 7f3c5b687700 1 -- start start 2023-10-16T12:26:19.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.111+0000 7f3c5b687700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3c5415b630 0x7f3c540d0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:19.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.111+0000 7f3c5b687700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3c5415ba10 0x7f3c540cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:19.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.111+0000 7f3c5b687700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f3c540cdde0 0x7f3c540ce230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:19.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.111+0000 7f3c5b687700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c540ce770 con 0x7f3c5415ba10 2023-10-16T12:26:19.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.111+0000 7f3c5b687700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c540b0970 con 0x7f3c540cdde0 2023-10-16T12:26:19.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.111+0000 7f3c5b687700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c540cd690 con 0x7f3c5415b630 2023-10-16T12:26:19.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.111+0000 7f3c58c22700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3c5415ba10 0x7f3c540cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:19.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.111+0000 7f3c58c22700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3c5415ba10 0x7f3c540cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39178/0 (socket says 172.21.15.16:39178) 2023-10-16T12:26:19.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.111+0000 7f3c58c22700 1 -- 172.21.15.16:0/253598905 learned_addr learned my addr 172.21.15.16:0/253598905 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:19.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.111+0000 7f3c59c24700 1 --2- 172.21.15.16:0/253598905 >> v2:172.21.15.49:3300/0 conn(0x7f3c540cdde0 0x7f3c540ce230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:19.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.111+0000 7f3c59423700 1 --2- 172.21.15.16:0/253598905 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3c5415b630 0x7f3c540d0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:19.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.111+0000 7f3c58c22700 1 -- 172.21.15.16:0/253598905 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3c5415b630 msgr2=0x7f3c540d0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:19.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.111+0000 7f3c58c22700 1 --2- 172.21.15.16:0/253598905 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3c5415b630 0x7f3c540d0380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:19.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.111+0000 7f3c58c22700 1 -- 172.21.15.16:0/253598905 >> v2:172.21.15.49:3300/0 conn(0x7f3c540cdde0 msgr2=0x7f3c540ce230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:19.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.111+0000 7f3c58c22700 1 --2- 172.21.15.16:0/253598905 >> v2:172.21.15.49:3300/0 conn(0x7f3c540cdde0 0x7f3c540ce230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:19.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.111+0000 7f3c58c22700 1 -- 172.21.15.16:0/253598905 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c540cd970 con 0x7f3c5415ba10 2023-10-16T12:26:19.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.111+0000 7f3c58c22700 1 --2- 172.21.15.16:0/253598905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c5415ba10 0x7f3c540cf290 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f3c540cef90 tx=0x7f3c44009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55af7b6b90e30e10 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:19.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.112+0000 7f3c4bfff700 1 -- 172.21.15.16:0/253598905 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c4400b870 con 0x7f3c5415ba10 2023-10-16T12:26:19.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.112+0000 7f3c4bfff700 1 -- 172.21.15.16:0/253598905 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c4400b9d0 con 0x7f3c5415ba10 2023-10-16T12:26:19.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.112+0000 7f3c4bfff700 1 -- 172.21.15.16:0/253598905 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c4400bd60 con 0x7f3c5415ba10 2023-10-16T12:26:19.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.112+0000 7f3c5b687700 1 -- 172.21.15.16:0/253598905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c5415ba10 msgr2=0x7f3c540cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:19.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.112+0000 7f3c5b687700 1 --2- 172.21.15.16:0/253598905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c5415ba10 0x7f3c540cf290 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f3c540cef90 tx=0x7f3c44009bd0 comp rx=0 tx=0).stop 2023-10-16T12:26:19.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.112+0000 7f3c5b687700 1 -- 172.21.15.16:0/253598905 shutdown_connections 2023-10-16T12:26:19.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.112+0000 7f3c5b687700 1 --2- 172.21.15.16:0/253598905 >> v2:172.21.15.49:3300/0 conn(0x7f3c540cdde0 0x7f3c540ce230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:19.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.112+0000 7f3c5b687700 1 --2- 172.21.15.16:0/253598905 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3c5415b630 0x7f3c540d0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:19.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.112+0000 7f3c5b687700 1 --2- 172.21.15.16:0/253598905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c5415ba10 0x7f3c540cf290 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:19.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.112+0000 7f3c5b687700 1 -- 172.21.15.16:0/253598905 >> 172.21.15.16:0/253598905 conn(0x7f3c540d14b0 msgr2=0x7f3c540d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:19.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.112+0000 7f3c5b687700 1 -- 172.21.15.16:0/253598905 shutdown_connections 2023-10-16T12:26:19.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.112+0000 7f3c5b687700 1 -- 172.21.15.16:0/253598905 wait complete. 2023-10-16T12:26:19.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.113+0000 7f3c5b687700 1 Processor -- start 2023-10-16T12:26:19.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.113+0000 7f3c5b687700 1 -- start start 2023-10-16T12:26:19.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.114+0000 7f3c5b687700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3c540cdde0 0x7f3c541f40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:19.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.114+0000 7f3c5b687700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c5415b630 0x7f3c541ed510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:19.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.114+0000 7f3c5b687700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3c5415ba10 0x7f3c540c5c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:19.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.114+0000 7f3c5b687700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c541f4620 con 0x7f3c5415b630 2023-10-16T12:26:19.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.114+0000 7f3c5b687700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c54158260 con 0x7f3c540cdde0 2023-10-16T12:26:19.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.114+0000 7f3c5b687700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c540b5350 con 0x7f3c5415ba10 2023-10-16T12:26:19.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.114+0000 7f3c58c22700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c5415b630 0x7f3c541ed510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:19.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.114+0000 7f3c58c22700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c5415b630 0x7f3c541ed510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39184/0 (socket says 172.21.15.16:39184) 2023-10-16T12:26:19.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.114+0000 7f3c58c22700 1 -- 172.21.15.16:0/1165667036 learned_addr learned my addr 172.21.15.16:0/1165667036 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:19.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.114+0000 7f3c59c24700 1 --2- 172.21.15.16:0/1165667036 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3c5415ba10 0x7f3c540c5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:19.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.114+0000 7f3c59423700 1 --2- 172.21.15.16:0/1165667036 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3c540cdde0 0x7f3c541f40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:19.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.115+0000 7f3c58c22700 1 -- 172.21.15.16:0/1165667036 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3c5415ba10 msgr2=0x7f3c540c5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:19.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.115+0000 7f3c58c22700 1 --2- 172.21.15.16:0/1165667036 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3c5415ba10 0x7f3c540c5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:19.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.115+0000 7f3c58c22700 1 -- 172.21.15.16:0/1165667036 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3c540cdde0 msgr2=0x7f3c541f40e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:19.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.115+0000 7f3c58c22700 1 --2- 172.21.15.16:0/1165667036 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3c540cdde0 0x7f3c541f40e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:19.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.115+0000 7f3c58c22700 1 -- 172.21.15.16:0/1165667036 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c44009550 con 0x7f3c5415b630 2023-10-16T12:26:19.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.115+0000 7f3c58c22700 1 --2- 172.21.15.16:0/1165667036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c5415b630 0x7f3c541ed510 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f3c44000c00 tx=0x7f3c44004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:19.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.115+0000 7f3c49ffb700 1 -- 172.21.15.16:0/1165667036 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c44004480 con 0x7f3c5415b630 2023-10-16T12:26:19.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.115+0000 7f3c49ffb700 1 -- 172.21.15.16:0/1165667036 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c440045e0 con 0x7f3c5415b630 2023-10-16T12:26:19.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.115+0000 7f3c5b687700 1 -- 172.21.15.16:0/1165667036 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c540b5910 con 0x7f3c5415b630 2023-10-16T12:26:19.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.115+0000 7f3c49ffb700 1 -- 172.21.15.16:0/1165667036 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c440038e0 con 0x7f3c5415b630 2023-10-16T12:26:19.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.115+0000 7f3c5b687700 1 -- 172.21.15.16:0/1165667036 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c540cb350 con 0x7f3c5415b630 2023-10-16T12:26:19.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.116+0000 7f3c5b687700 1 -- 172.21.15.16:0/1165667036 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c540b5aa0 con 0x7f3c5415b630 2023-10-16T12:26:19.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.117+0000 7f3c49ffb700 1 -- 172.21.15.16:0/1165667036 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 31) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3c44021070 con 0x7f3c5415b630 2023-10-16T12:26:19.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.117+0000 7f3c49ffb700 1 --2- 172.21.15.16:0/1165667036 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f3c3806b020 0x7f3c3806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:19.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.117+0000 7f3c49ffb700 1 -- 172.21.15.16:0/1165667036 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5979+0+0 (secure 0 0 0) 0x7f3c4408ad40 con 0x7f3c5415b630 2023-10-16T12:26:19.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.117+0000 7f3c59423700 1 --2- 172.21.15.16:0/1165667036 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f3c3806b020 0x7f3c3806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:19.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.117+0000 7f3c59423700 1 --2- 172.21.15.16:0/1165667036 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f3c3806b020 0x7f3c3806d4e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f3c540b22d0 tx=0x7f3c50008040 comp rx=0 tx=0).ready entity=mgr.6396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:19.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.122+0000 7f3c49ffb700 1 -- 172.21.15.16:0/1165667036 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3c44055d80 con 0x7f3c5415b630 2023-10-16T12:26:19.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.344+0000 7f3c5b687700 1 -- 172.21.15.16:0/1165667036 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f3c5404ebb0 con 0x7f3c5415b630 2023-10-16T12:26:19.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.941+0000 7f3c49ffb700 1 -- 172.21.15.16:0/1165667036 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v79) v1 ==== 97+0+0 (secure 0 0 0) 0x7f3c44015020 con 0x7f3c5415b630 2023-10-16T12:26:19.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.945+0000 7f3c5b687700 1 -- 172.21.15.16:0/1165667036 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f3c3806b020 msgr2=0x7f3c3806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:19.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.945+0000 7f3c5b687700 1 --2- 172.21.15.16:0/1165667036 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f3c3806b020 0x7f3c3806d4e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f3c540b22d0 tx=0x7f3c50008040 comp rx=0 tx=0).stop 2023-10-16T12:26:19.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.945+0000 7f3c5b687700 1 -- 172.21.15.16:0/1165667036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c5415b630 msgr2=0x7f3c541ed510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:19.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.945+0000 7f3c5b687700 1 --2- 172.21.15.16:0/1165667036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c5415b630 0x7f3c541ed510 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f3c44000c00 tx=0x7f3c44004750 comp rx=0 tx=0).stop 2023-10-16T12:26:19.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.945+0000 7f3c5b687700 1 -- 172.21.15.16:0/1165667036 shutdown_connections 2023-10-16T12:26:19.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.945+0000 7f3c5b687700 1 --2- 172.21.15.16:0/1165667036 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f3c3806b020 0x7f3c3806d4e0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:19.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.945+0000 7f3c5b687700 1 --2- 172.21.15.16:0/1165667036 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3c5415ba10 0x7f3c540c5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:19.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.945+0000 7f3c5b687700 1 --2- 172.21.15.16:0/1165667036 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3c540cdde0 0x7f3c541f40e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:19.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.945+0000 7f3c5b687700 1 --2- 172.21.15.16:0/1165667036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c5415b630 0x7f3c541ed510 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:19.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.945+0000 7f3c5b687700 1 -- 172.21.15.16:0/1165667036 >> 172.21.15.16:0/1165667036 conn(0x7f3c540d14b0 msgr2=0x7f3c5406fce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:19.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.945+0000 7f3c5b687700 1 -- 172.21.15.16:0/1165667036 shutdown_connections 2023-10-16T12:26:19.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:19.945+0000 7f3c5b687700 1 -- 172.21.15.16:0/1165667036 wait complete. 2023-10-16T12:26:19.950 INFO:teuthology.orchestra.run.smithi016.stderr: removed all blocklist entries 2023-10-16T12:26:19.964 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-16T12:26:20.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.093+0000 7fb815c14700 1 Processor -- start 2023-10-16T12:26:20.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.093+0000 7fb815c14700 1 -- start start 2023-10-16T12:26:20.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.094+0000 7fb815c14700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb8100ce4f0 0x7fb8100cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:20.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.094+0000 7fb815c14700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8100cc2a0 con 0x7fb81015b630 2023-10-16T12:26:20.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.094+0000 7fb815c14700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8100cc540 con 0x7fb81015ba10 2023-10-16T12:26:20.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.094+0000 7fb815c14700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8100cdb20 con 0x7fb8100ce4f0 2023-10-16T12:26:20.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.094+0000 7fb80f7fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb81015b630 0x7fb8100cdde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:56698/0 (socket says 172.21.15.16:56698) 2023-10-16T12:26:20.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.094+0000 7fb80f7fe700 1 -- 172.21.15.16:0/3353487470 learned_addr learned my addr 172.21.15.16:0/3353487470 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:26:20.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.094+0000 7fb80e7fc700 1 -- 172.21.15.16:0/3353487470 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3082817997 0 0) 0x7fb8100cc2a0 con 0x7fb81015b630 2023-10-16T12:26:20.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.094+0000 7fb80effd700 1 --2- 172.21.15.16:0/3353487470 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb8100ce4f0 0x7fb8100cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:20.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.094+0000 7fb80e7fc700 1 -- 172.21.15.16:0/3353487470 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7f80035b0 con 0x7fb81015b630 2023-10-16T12:26:20.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.094+0000 7fb80e7fc700 1 -- 172.21.15.16:0/3353487470 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2075969719 0 0) 0x7fb7f80035b0 con 0x7fb81015b630 2023-10-16T12:26:20.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.095+0000 7fb80e7fc700 1 -- 172.21.15.16:0/3353487470 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb8100ce4f0 msgr2=0x7fb8100cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:20.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.095+0000 7fb80e7fc700 1 --2- 172.21.15.16:0/3353487470 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb8100ce4f0 0x7fb8100cd5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:20.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.095+0000 7fb80e7fc700 1 -- 172.21.15.16:0/3353487470 >> v1:172.21.15.49:6789/0 conn(0x7fb81015ba10 legacy=0x7fb8100cfca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:20.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.095+0000 7fb80e7fc700 1 -- 172.21.15.16:0/3353487470 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8100cc820 con 0x7fb81015b630 2023-10-16T12:26:20.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.095+0000 7fb80e7fc700 1 -- 172.21.15.16:0/3353487470 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7fb8000017f0 con 0x7fb81015b630 2023-10-16T12:26:20.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.095+0000 7fb80e7fc700 1 -- 172.21.15.16:0/3353487470 >> v1:172.21.15.16:6789/0 conn(0x7fb81015b630 legacy=0x7fb8100cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:20.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.095+0000 7fb80e7fc700 1 --2- 172.21.15.16:0/3353487470 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb8100ce4f0 0x7fb7f8003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:20.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.095+0000 7fb80e7fc700 1 --2- 172.21.15.16:0/3353487470 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7f8003fb0 0x7fb7f8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:20.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.095+0000 7fb80e7fc700 1 --2- 172.21.15.16:0/3353487470 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb7f8004d90 0x7fb7f8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:20.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.095+0000 7fb80e7fc700 1 -- 172.21.15.16:0/3353487470 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f800bba0 con 0x7fb7f8003fb0 2023-10-16T12:26:20.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.095+0000 7fb80e7fc700 1 -- 172.21.15.16:0/3353487470 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f800bce0 con 0x7fb7f8004d90 2023-10-16T12:26:20.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.095+0000 7fb80e7fc700 1 -- 172.21.15.16:0/3353487470 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f800bea0 con 0x7fb8100ce4f0 2023-10-16T12:26:20.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.095+0000 7fb80effd700 1 --2- 172.21.15.16:0/3353487470 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7f8003fb0 0x7fb7f8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:20.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.095+0000 7fb80f7fe700 1 --2- 172.21.15.16:0/3353487470 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb8100ce4f0 0x7fb7f8003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:20.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.095+0000 7fb80ffff700 1 --2- 172.21.15.16:0/3353487470 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb7f8004d90 0x7fb7f8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:20.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.095+0000 7fb80effd700 1 -- 172.21.15.16:0/3353487470 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb8100ce4f0 msgr2=0x7fb7f8003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:20.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.095+0000 7fb80effd700 1 --2- 172.21.15.16:0/3353487470 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb8100ce4f0 0x7fb7f8003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:20.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.095+0000 7fb80effd700 1 -- 172.21.15.16:0/3353487470 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb7f8004d90 msgr2=0x7fb7f8005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:20.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.095+0000 7fb80effd700 1 --2- 172.21.15.16:0/3353487470 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb7f8004d90 0x7fb7f8005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:20.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.095+0000 7fb80effd700 1 -- 172.21.15.16:0/3353487470 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7f80035b0 con 0x7fb7f8003fb0 2023-10-16T12:26:20.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.095+0000 7fb80ffff700 1 --2- 172.21.15.16:0/3353487470 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb7f8004d90 0x7fb7f8005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:26:20.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.095+0000 7fb80f7fe700 1 --2- 172.21.15.16:0/3353487470 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb8100ce4f0 0x7fb7f8003a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:26:20.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.095+0000 7fb80effd700 1 --2- 172.21.15.16:0/3353487470 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7f8003fb0 0x7fb7f8004850 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fb804009b60 tx=0x7fb7f8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcafdd658923be93 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:20.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.095+0000 7fb80e7fc700 1 -- 172.21.15.16:0/3353487470 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb80400e6c0 con 0x7fb7f8003fb0 2023-10-16T12:26:20.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.095+0000 7fb80e7fc700 1 -- 172.21.15.16:0/3353487470 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb80400e820 con 0x7fb7f8003fb0 2023-10-16T12:26:20.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.095+0000 7fb80e7fc700 1 -- 172.21.15.16:0/3353487470 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb80400eb90 con 0x7fb7f8003fb0 2023-10-16T12:26:20.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.096+0000 7fb815c14700 1 -- 172.21.15.16:0/3353487470 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7f8003fb0 msgr2=0x7fb7f8004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:20.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.096+0000 7fb815c14700 1 --2- 172.21.15.16:0/3353487470 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7f8003fb0 0x7fb7f8004850 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fb804009b60 tx=0x7fb7f8005880 comp rx=0 tx=0).stop 2023-10-16T12:26:20.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.096+0000 7fb80ffff700 1 -- 172.21.15.16:0/3353487470 reap_dead start 2023-10-16T12:26:20.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.096+0000 7fb815c14700 1 -- 172.21.15.16:0/3353487470 shutdown_connections 2023-10-16T12:26:20.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.096+0000 7fb815c14700 1 -- 172.21.15.16:0/3353487470 >> 172.21.15.16:0/3353487470 conn(0x7fb8100d14b0 msgr2=0x7fb8100d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:20.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.096+0000 7fb815c14700 1 -- 172.21.15.16:0/3353487470 shutdown_connections 2023-10-16T12:26:20.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.096+0000 7fb815c14700 1 -- 172.21.15.16:0/3353487470 wait complete. 2023-10-16T12:26:20.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.097+0000 7fb815c14700 1 Processor -- start 2023-10-16T12:26:20.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.097+0000 7fb815c14700 1 -- start start 2023-10-16T12:26:20.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.098+0000 7fb815c14700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb7f8003fb0 0x7fb81006b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:20.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.098+0000 7fb815c14700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb8100c9910 0x7fb8100c4080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:20.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.098+0000 7fb815c14700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8101f65a0 0x7fb8101f6a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:20.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.098+0000 7fb815c14700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb81006be30 con 0x7fb8101f65a0 2023-10-16T12:26:20.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.098+0000 7fb815c14700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8101f6f60 con 0x7fb7f8003fb0 2023-10-16T12:26:20.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.098+0000 7fb815c14700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8100c45c0 con 0x7fb8100c9910 2023-10-16T12:26:20.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.098+0000 7fb80ffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8101f65a0 0x7fb8101f6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:20.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.098+0000 7fb80ffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8101f65a0 0x7fb8101f6a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39208/0 (socket says 172.21.15.16:39208) 2023-10-16T12:26:20.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.098+0000 7fb80ffff700 1 -- 172.21.15.16:0/480897905 learned_addr learned my addr 172.21.15.16:0/480897905 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:20.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.098+0000 7fb80effd700 1 --2- 172.21.15.16:0/480897905 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb8100c9910 0x7fb8100c4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:20.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.098+0000 7fb80f7fe700 1 --2- 172.21.15.16:0/480897905 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb7f8003fb0 0x7fb81006b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:20.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.098+0000 7fb80ffff700 1 -- 172.21.15.16:0/480897905 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb8100c9910 msgr2=0x7fb8100c4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:20.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.098+0000 7fb80ffff700 1 --2- 172.21.15.16:0/480897905 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb8100c9910 0x7fb8100c4080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:20.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.098+0000 7fb80ffff700 1 -- 172.21.15.16:0/480897905 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb7f8003fb0 msgr2=0x7fb81006b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:20.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.098+0000 7fb80ffff700 1 --2- 172.21.15.16:0/480897905 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb7f8003fb0 0x7fb81006b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:20.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.099+0000 7fb80ffff700 1 -- 172.21.15.16:0/480897905 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb81006c040 con 0x7fb8101f65a0 2023-10-16T12:26:20.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.099+0000 7fb80f7fe700 1 --2- 172.21.15.16:0/480897905 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb7f8003fb0 0x7fb81006b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:26:20.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.099+0000 7fb80ffff700 1 --2- 172.21.15.16:0/480897905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8101f65a0 0x7fb8101f6a20 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fb81015bdf0 tx=0x7fb80800ddc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:20.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.099+0000 7fb7f7fff700 1 -- 172.21.15.16:0/480897905 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb808003e10 con 0x7fb8101f65a0 2023-10-16T12:26:20.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.099+0000 7fb7f7fff700 1 -- 172.21.15.16:0/480897905 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb808005070 con 0x7fb8101f65a0 2023-10-16T12:26:20.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.099+0000 7fb815c14700 1 -- 172.21.15.16:0/480897905 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb81006c320 con 0x7fb8101f65a0 2023-10-16T12:26:20.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.099+0000 7fb7f7fff700 1 -- 172.21.15.16:0/480897905 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb808005420 con 0x7fb8101f65a0 2023-10-16T12:26:20.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.099+0000 7fb815c14700 1 -- 172.21.15.16:0/480897905 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb81006c8e0 con 0x7fb8101f65a0 2023-10-16T12:26:20.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.100+0000 7fb815c14700 1 -- 172.21.15.16:0/480897905 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8100cb8b0 con 0x7fb8101f65a0 2023-10-16T12:26:20.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.100+0000 7fb7f7fff700 1 -- 172.21.15.16:0/480897905 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 31) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb808005580 con 0x7fb8101f65a0 2023-10-16T12:26:20.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.101+0000 7fb7f7fff700 1 --2- 172.21.15.16:0/480897905 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7fb7fc06b050 0x7fb7fc06d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:20.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.101+0000 7fb7f7fff700 1 -- 172.21.15.16:0/480897905 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5205+0+0 (secure 0 0 0) 0x7fb8080890c0 con 0x7fb8101f65a0 2023-10-16T12:26:20.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.101+0000 7fb80f7fe700 1 --2- 172.21.15.16:0/480897905 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7fb7fc06b050 0x7fb7fc06d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:20.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.101+0000 7fb80f7fe700 1 --2- 172.21.15.16:0/480897905 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7fb7fc06b050 0x7fb7fc06d510 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb800001170 tx=0x7fb800009040 comp rx=0 tx=0).ready entity=mgr.6396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:20.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.105+0000 7fb7f7fff700 1 -- 172.21.15.16:0/480897905 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb8080541f0 con 0x7fb8101f65a0 2023-10-16T12:26:20.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.389+0000 7fb815c14700 1 -- 172.21.15.16:0/480897905 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fb81004ebb0 con 0x7fb8101f65a0 2023-10-16T12:26:20.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.390+0000 7fb7f7fff700 1 -- 172.21.15.16:0/480897905 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v18) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fb808057810 con 0x7fb8101f65a0 2023-10-16T12:26:20.392 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:26:20.392 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-10-16T12:26:20.393 INFO:teuthology.orchestra.run.smithi016.stdout: "auth_dump": [ 2023-10-16T12:26:20.393 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:26:20.393 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.a", 2023-10-16T12:26:20.393 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOKC1l/rBtOhAA/ob5WUvpz8Glp4E+xPZOYA==", 2023-10-16T12:26:20.393 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:26:20.394 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T12:26:20.394 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-10-16T12:26:20.394 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-10-16T12:26:20.394 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:26:20.395 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:26:20.395 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:26:20.395 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:26:20.395 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.b", 2023-10-16T12:26:20.395 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1lKAVsCxAAvhXuLY6i1+Iz1yq5BVPMNQ==", 2023-10-16T12:26:20.396 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:26:20.396 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T12:26:20.396 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-10-16T12:26:20.396 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-10-16T12:26:20.396 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:26:20.397 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:26:20.397 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:26:20.397 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:26:20.397 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.c", 2023-10-16T12:26:20.397 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1lvBowBhAA3waJNYhdgs8PGCYqYNYUog==", 2023-10-16T12:26:20.398 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:26:20.398 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T12:26:20.398 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-10-16T12:26:20.398 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-10-16T12:26:20.398 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:26:20.399 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:26:20.399 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:26:20.399 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:26:20.399 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.d", 2023-10-16T12:26:20.399 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1lS/fnEhAAfNYgQ5BW86JMjYm/bBfnaQ==", 2023-10-16T12:26:20.400 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:26:20.400 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T12:26:20.400 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-10-16T12:26:20.400 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-10-16T12:26:20.400 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:26:20.401 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:26:20.401 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:26:20.401 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:26:20.401 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.0", 2023-10-16T12:26:20.401 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAQKC1lbxllOhAAZeLdp2GMl6HqQPaQjKlNqw==", 2023-10-16T12:26:20.402 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:26:20.402 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T12:26:20.402 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T12:26:20.402 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:26:20.402 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:26:20.403 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:26:20.403 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:26:20.403 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.1", 2023-10-16T12:26:20.403 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQATKC1lnU6KOhAAHxvlbmzBptf1keGgL+Rbkg==", 2023-10-16T12:26:20.403 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:26:20.404 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T12:26:20.404 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T12:26:20.404 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:26:20.404 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:26:20.404 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:26:20.405 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:26:20.405 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.2", 2023-10-16T12:26:20.405 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAXKC1l9jFyARAAr/NAwtWhBMut9sAAoDEGWg==", 2023-10-16T12:26:20.405 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:26:20.405 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T12:26:20.405 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T12:26:20.406 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:26:20.406 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:26:20.406 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:26:20.406 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:26:20.407 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.3", 2023-10-16T12:26:20.407 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAaKC1lbw/lARAAGJ/9DNb5/7kLFSq7vrLtXg==", 2023-10-16T12:26:20.407 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:26:20.407 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T12:26:20.407 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T12:26:20.407 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:26:20.408 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:26:20.408 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:26:20.408 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:26:20.408 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.4", 2023-10-16T12:26:20.408 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAeKC1lZb7jERAAYpvb1xcSli9CTq7L8ChLOA==", 2023-10-16T12:26:20.409 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:26:20.409 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T12:26:20.409 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T12:26:20.409 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:26:20.410 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:26:20.410 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:26:20.410 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:26:20.410 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.5", 2023-10-16T12:26:20.410 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAhKC1l/4HuEBAAqfnRqjknij9KES/fqiyIBQ==", 2023-10-16T12:26:20.411 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:26:20.411 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T12:26:20.411 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T12:26:20.411 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:26:20.412 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:26:20.412 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:26:20.412 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:26:20.412 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.6", 2023-10-16T12:26:20.412 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAkKC1lzoUkEBAA+Q82OiBMNag8+R316WQtUw==", 2023-10-16T12:26:20.412 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:26:20.413 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T12:26:20.413 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T12:26:20.413 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:26:20.413 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:26:20.413 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:26:20.413 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:26:20.414 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.7", 2023-10-16T12:26:20.414 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAnKC1ljUu4DxAAc+G4anJANC2GhLJ8wb6gYg==", 2023-10-16T12:26:20.414 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:26:20.414 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T12:26:20.414 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T12:26:20.415 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:26:20.415 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:26:20.415 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:26:20.415 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:26:20.416 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.0", 2023-10-16T12:26:20.416 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1l2l6PFxAA9QyXtBAHdGAXpwH0JfknuQ==", 2023-10-16T12:26:20.416 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:26:20.416 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T12:26:20.416 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow r", 2023-10-16T12:26:20.416 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-16T12:26:20.417 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:26:20.417 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:26:20.417 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:26:20.417 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.1", 2023-10-16T12:26:20.417 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1lAcHFIxAAcNt2bmv2R6j5L3DYkdw7eQ==", 2023-10-16T12:26:20.418 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:26:20.418 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T12:26:20.418 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-10-16T12:26:20.418 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-10-16T12:26:20.418 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-10-16T12:26:20.419 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:26:20.419 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:26:20.419 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:26:20.419 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.2", 2023-10-16T12:26:20.419 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1lA6wnGxAAdtaqYhtIJu8qENMrD8xvVA==", 2023-10-16T12:26:20.420 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:26:20.420 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T12:26:20.420 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-10-16T12:26:20.420 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-10-16T12:26:20.420 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-10-16T12:26:20.421 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:26:20.421 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:26:20.421 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:26:20.421 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.3", 2023-10-16T12:26:20.421 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1l/ddAJxAA3fNKKn9XkXas1BlDN+ggjw==", 2023-10-16T12:26:20.422 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:26:20.422 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T12:26:20.422 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-10-16T12:26:20.422 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-10-16T12:26:20.422 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-10-16T12:26:20.423 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:26:20.423 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:26:20.423 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:26:20.423 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.admin", 2023-10-16T12:26:20.423 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOKC1ld5jUHBAAc6esnCce3dSEZn3wVjhIlQ==", 2023-10-16T12:26:20.424 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:26:20.424 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2023-10-16T12:26:20.424 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-10-16T12:26:20.424 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-10-16T12:26:20.424 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:26:20.425 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:26:20.425 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:26:20.425 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:26:20.425 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mgr.x", 2023-10-16T12:26:20.425 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOKC1l6b3vNhAAmj5xbfid2sZM2KfHmAf1Wg==", 2023-10-16T12:26:20.426 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:26:20.426 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2023-10-16T12:26:20.426 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile mgr", 2023-10-16T12:26:20.426 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:26:20.426 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:26:20.427 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:26:20.427 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:26:20.427 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mgr.y", 2023-10-16T12:26:20.427 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOKC1lLz2nMhAA8NqFYZ5nDKkzPBMrkwsVuQ==", 2023-10-16T12:26:20.427 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:26:20.428 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2023-10-16T12:26:20.428 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile mgr", 2023-10-16T12:26:20.428 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:26:20.429 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:26:20.429 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:26:20.429 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-10-16T12:26:20.429 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-10-16T12:26:20.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.392+0000 7fb815c14700 1 -- 172.21.15.16:0/480897905 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7fb7fc06b050 msgr2=0x7fb7fc06d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:20.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.392+0000 7fb815c14700 1 --2- 172.21.15.16:0/480897905 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7fb7fc06b050 0x7fb7fc06d510 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb800001170 tx=0x7fb800009040 comp rx=0 tx=0).stop 2023-10-16T12:26:20.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.392+0000 7fb815c14700 1 -- 172.21.15.16:0/480897905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8101f65a0 msgr2=0x7fb8101f6a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:20.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.392+0000 7fb815c14700 1 --2- 172.21.15.16:0/480897905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8101f65a0 0x7fb8101f6a20 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fb81015bdf0 tx=0x7fb80800ddc0 comp rx=0 tx=0).stop 2023-10-16T12:26:20.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.392+0000 7fb815c14700 1 -- 172.21.15.16:0/480897905 shutdown_connections 2023-10-16T12:26:20.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.392+0000 7fb815c14700 1 --2- 172.21.15.16:0/480897905 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7fb7fc06b050 0x7fb7fc06d510 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:20.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.392+0000 7fb815c14700 1 --2- 172.21.15.16:0/480897905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8101f65a0 0x7fb8101f6a20 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:20.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.392+0000 7fb815c14700 1 --2- 172.21.15.16:0/480897905 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb7f8003fb0 0x7fb81006b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:20.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.392+0000 7fb815c14700 1 --2- 172.21.15.16:0/480897905 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb8100c9910 0x7fb8100c4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:20.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.392+0000 7fb815c14700 1 -- 172.21.15.16:0/480897905 >> 172.21.15.16:0/480897905 conn(0x7fb8100d14b0 msgr2=0x7fb81015a970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:20.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.392+0000 7fb815c14700 1 -- 172.21.15.16:0/480897905 shutdown_connections 2023-10-16T12:26:20.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.392+0000 7fb815c14700 1 -- 172.21.15.16:0/480897905 wait complete. 2023-10-16T12:26:20.434 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:26:20.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.557+0000 7f7e8e4de700 1 Processor -- start 2023-10-16T12:26:20.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.557+0000 7f7e8e4de700 1 -- start start 2023-10-16T12:26:20.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.558+0000 7f7e8e4de700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7e8815b630 0x7f7e880cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:20.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.558+0000 7f7e8e4de700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7e880ce4f0 0x7f7e880cfca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:20.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.558+0000 7f7e8e4de700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e880cc2a0 con 0x7f7e8815ba10 2023-10-16T12:26:20.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.558+0000 7f7e8e4de700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e880cdb20 con 0x7f7e8815b630 2023-10-16T12:26:20.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.558+0000 7f7e8e4de700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e880cc540 con 0x7f7e880ce4f0 2023-10-16T12:26:20.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.558+0000 7f7e87fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7e8815b630 0x7f7e880cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:20.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.558+0000 7f7e87fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7e8815b630 0x7f7e880cd5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39212/0 (socket says 172.21.15.16:39212) 2023-10-16T12:26:20.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.558+0000 7f7e87fff700 1 -- 172.21.15.16:0/1341203295 learned_addr learned my addr 172.21.15.16:0/1341203295 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:20.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.558+0000 7f7e8ca7b700 1 --2- 172.21.15.16:0/1341203295 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7e880ce4f0 0x7f7e880cfca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:20.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.558+0000 7f7e87fff700 1 -- 172.21.15.16:0/1341203295 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7e880ce4f0 msgr2=0x7f7e880cfca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:20.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.558+0000 7f7e87fff700 1 --2- 172.21.15.16:0/1341203295 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7e880ce4f0 0x7f7e880cfca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:20.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.558+0000 7f7e87fff700 1 -- 172.21.15.16:0/1341203295 >> v1:172.21.15.49:6789/0 conn(0x7f7e8815ba10 legacy=0x7f7e880cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:20.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.558+0000 7f7e87fff700 1 -- 172.21.15.16:0/1341203295 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e880cc820 con 0x7f7e8815b630 2023-10-16T12:26:20.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.558+0000 7f7e87fff700 1 --2- 172.21.15.16:0/1341203295 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e8815b630 0x7f7e880cd5e0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f7e7c006fd0 tx=0x7f7e7c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f463b491aa5387c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:20.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.558+0000 7f7e86ffd700 1 -- 172.21.15.16:0/1341203295 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e7c005e30 con 0x7f7e8815b630 2023-10-16T12:26:20.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.558+0000 7f7e86ffd700 1 -- 172.21.15.16:0/1341203295 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e7c004030 con 0x7f7e8815b630 2023-10-16T12:26:20.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.558+0000 7f7e86ffd700 1 -- 172.21.15.16:0/1341203295 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e7c0043e0 con 0x7f7e8815b630 2023-10-16T12:26:20.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.559+0000 7f7e8e4de700 1 -- 172.21.15.16:0/1341203295 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e8815b630 msgr2=0x7f7e880cd5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:20.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.559+0000 7f7e8e4de700 1 --2- 172.21.15.16:0/1341203295 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e8815b630 0x7f7e880cd5e0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f7e7c006fd0 tx=0x7f7e7c009bd0 comp rx=0 tx=0).stop 2023-10-16T12:26:20.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.559+0000 7f7e8e4de700 1 -- 172.21.15.16:0/1341203295 shutdown_connections 2023-10-16T12:26:20.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.559+0000 7f7e8e4de700 1 --2- 172.21.15.16:0/1341203295 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7e880ce4f0 0x7f7e880cfca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:20.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.559+0000 7f7e8e4de700 1 --2- 172.21.15.16:0/1341203295 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e8815b630 0x7f7e880cd5e0 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:20.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.559+0000 7f7e8e4de700 1 -- 172.21.15.16:0/1341203295 >> 172.21.15.16:0/1341203295 conn(0x7f7e880d14b0 msgr2=0x7f7e880d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:20.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.559+0000 7f7e8e4de700 1 -- 172.21.15.16:0/1341203295 shutdown_connections 2023-10-16T12:26:20.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.559+0000 7f7e8e4de700 1 -- 172.21.15.16:0/1341203295 wait complete. 2023-10-16T12:26:20.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.560+0000 7f7e8e4de700 1 Processor -- start 2023-10-16T12:26:20.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.560+0000 7f7e8e4de700 1 -- start start 2023-10-16T12:26:20.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.561+0000 7f7e8e4de700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7e880ce4f0 0x7f7e88159ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:20.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.561+0000 7f7e8e4de700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e8815b630 0x7f7e880ca710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:20.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.561+0000 7f7e8e4de700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7e8815ba10 0x7f7e8806b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:20.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.561+0000 7f7e8e4de700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e8815a010 con 0x7f7e8815b630 2023-10-16T12:26:20.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.561+0000 7f7e8e4de700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e880c9670 con 0x7f7e8815ba10 2023-10-16T12:26:20.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.561+0000 7f7e8e4de700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e880c9950 con 0x7f7e880ce4f0 2023-10-16T12:26:20.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.561+0000 7f7e877fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e8815b630 0x7f7e880ca710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:20.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.561+0000 7f7e877fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e8815b630 0x7f7e880ca710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39220/0 (socket says 172.21.15.16:39220) 2023-10-16T12:26:20.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.561+0000 7f7e877fe700 1 -- 172.21.15.16:0/2140206959 learned_addr learned my addr 172.21.15.16:0/2140206959 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:20.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.561+0000 7f7e8ca7b700 1 --2- 172.21.15.16:0/2140206959 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7e8815ba10 0x7f7e8806b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:20.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.561+0000 7f7e87fff700 1 --2- 172.21.15.16:0/2140206959 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7e880ce4f0 0x7f7e88159ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:20.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.562+0000 7f7e877fe700 1 -- 172.21.15.16:0/2140206959 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7e880ce4f0 msgr2=0x7f7e88159ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:20.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.562+0000 7f7e877fe700 1 --2- 172.21.15.16:0/2140206959 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7e880ce4f0 0x7f7e88159ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:20.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.562+0000 7f7e877fe700 1 -- 172.21.15.16:0/2140206959 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7e8815ba10 msgr2=0x7f7e8806b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:20.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.562+0000 7f7e877fe700 1 --2- 172.21.15.16:0/2140206959 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7e8815ba10 0x7f7e8806b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:20.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.562+0000 7f7e877fe700 1 -- 172.21.15.16:0/2140206959 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e7c009500 con 0x7f7e8815b630 2023-10-16T12:26:20.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.562+0000 7f7e877fe700 1 --2- 172.21.15.16:0/2140206959 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e8815b630 0x7f7e880ca710 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f7e7800c4e0 tx=0x7f7e7800c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:20.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.562+0000 7f7e84ff9700 1 -- 172.21.15.16:0/2140206959 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e780052d0 con 0x7f7e8815b630 2023-10-16T12:26:20.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.562+0000 7f7e84ff9700 1 -- 172.21.15.16:0/2140206959 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e78005430 con 0x7f7e8815b630 2023-10-16T12:26:20.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.562+0000 7f7e8e4de700 1 -- 172.21.15.16:0/2140206959 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e880c9c30 con 0x7f7e8815b630 2023-10-16T12:26:20.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.562+0000 7f7e84ff9700 1 -- 172.21.15.16:0/2140206959 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e780038c0 con 0x7f7e8815b630 2023-10-16T12:26:20.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.562+0000 7f7e8e4de700 1 -- 172.21.15.16:0/2140206959 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e880ca1f0 con 0x7f7e8815b630 2023-10-16T12:26:20.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.563+0000 7f7e8e4de700 1 -- 172.21.15.16:0/2140206959 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e8804ebb0 con 0x7f7e8815b630 2023-10-16T12:26:20.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.563+0000 7f7e84ff9700 1 -- 172.21.15.16:0/2140206959 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 31) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7e78003a20 con 0x7f7e8815b630 2023-10-16T12:26:20.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.564+0000 7f7e84ff9700 1 --2- 172.21.15.16:0/2140206959 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f7e7406b020 0x7f7e7406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:20.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.564+0000 7f7e84ff9700 1 -- 172.21.15.16:0/2140206959 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f7e78088990 con 0x7f7e8815b630 2023-10-16T12:26:20.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.564+0000 7f7e87fff700 1 --2- 172.21.15.16:0/2140206959 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f7e7406b020 0x7f7e7406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:20.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.564+0000 7f7e87fff700 1 --2- 172.21.15.16:0/2140206959 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f7e7406b020 0x7f7e7406d4e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7e7c009c10 tx=0x7f7e7c008070 comp rx=0 tx=0).ready entity=mgr.6396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:20.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.568+0000 7f7e84ff9700 1 -- 172.21.15.16:0/2140206959 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7e78053a80 con 0x7f7e8815b630 2023-10-16T12:26:20.617 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:26:20.620 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:26:20.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.777+0000 7f7e8e4de700 1 -- 172.21.15.16:0/2140206959 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7e880b4b90 con 0x7f7e8815b630 2023-10-16T12:26:20.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.777+0000 7f7e84ff9700 1 -- 172.21.15.16:0/2140206959 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+12997 (secure 0 0 0) 0x7f7e78017090 con 0x7f7e8815b630 2023-10-16T12:26:20.780 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:26:20.780 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":81,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:26:19.949412+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:26:20.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.780+0000 7f7e8e4de700 1 -- 172.21.15.16:0/2140206959 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f7e7406b020 msgr2=0x7f7e7406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:20.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.780+0000 7f7e8e4de700 1 --2- 172.21.15.16:0/2140206959 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f7e7406b020 0x7f7e7406d4e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7e7c009c10 tx=0x7f7e7c008070 comp rx=0 tx=0).stop 2023-10-16T12:26:20.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.780+0000 7f7e8e4de700 1 -- 172.21.15.16:0/2140206959 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e8815b630 msgr2=0x7f7e880ca710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:20.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.780+0000 7f7e8e4de700 1 --2- 172.21.15.16:0/2140206959 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e8815b630 0x7f7e880ca710 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f7e7800c4e0 tx=0x7f7e7800c7f0 comp rx=0 tx=0).stop 2023-10-16T12:26:20.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.780+0000 7f7e8e4de700 1 -- 172.21.15.16:0/2140206959 shutdown_connections 2023-10-16T12:26:20.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.780+0000 7f7e8e4de700 1 --2- 172.21.15.16:0/2140206959 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f7e7406b020 0x7f7e7406d4e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:20.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.780+0000 7f7e8e4de700 1 --2- 172.21.15.16:0/2140206959 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7e8815ba10 0x7f7e8806b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:20.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.780+0000 7f7e8e4de700 1 --2- 172.21.15.16:0/2140206959 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7e880ce4f0 0x7f7e88159ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:20.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.780+0000 7f7e8e4de700 1 --2- 172.21.15.16:0/2140206959 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e8815b630 0x7f7e880ca710 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:20.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.780+0000 7f7e8e4de700 1 -- 172.21.15.16:0/2140206959 >> 172.21.15.16:0/2140206959 conn(0x7f7e880d14b0 msgr2=0x7f7e880b61d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:20.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.780+0000 7f7e8e4de700 1 -- 172.21.15.16:0/2140206959 shutdown_connections 2023-10-16T12:26:20.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.780+0000 7f7e8e4de700 1 -- 172.21.15.16:0/2140206959 wait complete. 2023-10-16T12:26:20.794 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T12:26:20.795 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:26:20.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.922+0000 7f8011810700 1 Processor -- start 2023-10-16T12:26:20.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.923+0000 7f8011810700 1 -- start start 2023-10-16T12:26:20.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.923+0000 7f8011810700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f800c15ec80 0x7f800c0cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:20.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.923+0000 7f8011810700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f800c0b2440 con 0x7f800c0d2110 2023-10-16T12:26:20.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.923+0000 7f8011810700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f800c0d00e0 con 0x7f800c15f060 2023-10-16T12:26:20.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.923+0000 7f8011810700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f800c0b2c50 con 0x7f800c15ec80 2023-10-16T12:26:20.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.923+0000 7f800b7fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f800c0d2110 0x7f800c0b4b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:56736/0 (socket says 172.21.15.16:56736) 2023-10-16T12:26:20.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.924+0000 7f800b7fe700 1 -- 172.21.15.16:0/2426580274 learned_addr learned my addr 172.21.15.16:0/2426580274 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:26:20.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.924+0000 7f800affd700 1 --2- 172.21.15.16:0/2426580274 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f800c15ec80 0x7f800c0cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:20.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.924+0000 7f8009ffb700 1 -- 172.21.15.16:0/2426580274 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3901498576 0 0) 0x7f800c0b2440 con 0x7f800c0d2110 2023-10-16T12:26:20.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.924+0000 7f8009ffb700 1 -- 172.21.15.16:0/2426580274 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ff40035b0 con 0x7f800c0d2110 2023-10-16T12:26:20.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.924+0000 7f8009ffb700 1 -- 172.21.15.16:0/2426580274 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2491302077 0 0) 0x7f7ff40035b0 con 0x7f800c0d2110 2023-10-16T12:26:20.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.924+0000 7f8009ffb700 1 -- 172.21.15.16:0/2426580274 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f800c15ec80 msgr2=0x7f800c0cdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:20.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.924+0000 7f8009ffb700 1 --2- 172.21.15.16:0/2426580274 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f800c15ec80 0x7f800c0cdec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:20.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.924+0000 7f8009ffb700 1 -- 172.21.15.16:0/2426580274 >> v1:172.21.15.49:6789/0 conn(0x7f800c15f060 legacy=0x7f800c0b4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:20.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.924+0000 7f8009ffb700 1 -- 172.21.15.16:0/2426580274 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f800c0cef60 con 0x7f800c0d2110 2023-10-16T12:26:20.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.924+0000 7f8009ffb700 1 -- 172.21.15.16:0/2426580274 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f80040019f0 con 0x7f800c0d2110 2023-10-16T12:26:20.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.924+0000 7f8009ffb700 1 -- 172.21.15.16:0/2426580274 >> v1:172.21.15.16:6789/0 conn(0x7f800c0d2110 legacy=0x7f800c0b4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:20.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.924+0000 7f8009ffb700 1 --2- 172.21.15.16:0/2426580274 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ff4003a30 0x7f7ff4003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:20.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.924+0000 7f8009ffb700 1 --2- 172.21.15.16:0/2426580274 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f800c15ec80 0x7f7ff4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:20.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.924+0000 7f8009ffb700 1 --2- 172.21.15.16:0/2426580274 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7ff4004d90 0x7f7ff4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:20.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.924+0000 7f8009ffb700 1 -- 172.21.15.16:0/2426580274 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f800c0b2c50 con 0x7f7ff4003a30 2023-10-16T12:26:20.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.924+0000 7f8009ffb700 1 -- 172.21.15.16:0/2426580274 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff400bac0 con 0x7f7ff4004d90 2023-10-16T12:26:20.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.924+0000 7f8009ffb700 1 -- 172.21.15.16:0/2426580274 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff400bc80 con 0x7f800c15ec80 2023-10-16T12:26:20.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.924+0000 7f800affd700 1 --2- 172.21.15.16:0/2426580274 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ff4003a30 0x7f7ff4003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:20.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.924+0000 7f800a7fc700 1 --2- 172.21.15.16:0/2426580274 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f800c15ec80 0x7f7ff4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:20.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.924+0000 7f800b7fe700 1 --2- 172.21.15.16:0/2426580274 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7ff4004d90 0x7f7ff4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:20.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.925+0000 7f800affd700 1 -- 172.21.15.16:0/2426580274 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f800c15ec80 msgr2=0x7f7ff4004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:20.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.925+0000 7f800affd700 1 --2- 172.21.15.16:0/2426580274 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f800c15ec80 0x7f7ff4004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:20.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.925+0000 7f800affd700 1 -- 172.21.15.16:0/2426580274 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7ff4004d90 msgr2=0x7f7ff4005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:20.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.925+0000 7f800affd700 1 --2- 172.21.15.16:0/2426580274 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7ff4004d90 0x7f7ff4005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:20.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.925+0000 7f800affd700 1 -- 172.21.15.16:0/2426580274 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ff40035b0 con 0x7f7ff4003a30 2023-10-16T12:26:20.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.925+0000 7f800b7fe700 1 --2- 172.21.15.16:0/2426580274 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7ff4004d90 0x7f7ff4005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:26:20.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.925+0000 7f800affd700 1 --2- 172.21.15.16:0/2426580274 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ff4003a30 0x7f7ff4003e50 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc009b30 tx=0x7f7ff4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bee5533a87eabbf server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:20.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.925+0000 7f8009ffb700 1 -- 172.21.15.16:0/2426580274 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ffc00e6c0 con 0x7f7ff4003a30 2023-10-16T12:26:20.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.925+0000 7f8009ffb700 1 -- 172.21.15.16:0/2426580274 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ffc00e820 con 0x7f7ff4003a30 2023-10-16T12:26:20.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.925+0000 7f8009ffb700 1 -- 172.21.15.16:0/2426580274 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ffc00eb90 con 0x7f7ff4003a30 2023-10-16T12:26:20.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.925+0000 7f8011810700 1 -- 172.21.15.16:0/2426580274 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ff4003a30 msgr2=0x7f7ff4003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:20.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.925+0000 7f8011810700 1 --2- 172.21.15.16:0/2426580274 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ff4003a30 0x7f7ff4003e50 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc009b30 tx=0x7f7ff4005880 comp rx=0 tx=0).stop 2023-10-16T12:26:20.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.925+0000 7f800b7fe700 1 -- 172.21.15.16:0/2426580274 reap_dead start 2023-10-16T12:26:20.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.925+0000 7f8011810700 1 -- 172.21.15.16:0/2426580274 shutdown_connections 2023-10-16T12:26:20.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.925+0000 7f8011810700 1 -- 172.21.15.16:0/2426580274 >> 172.21.15.16:0/2426580274 conn(0x7f800c0d15f0 msgr2=0x7f800c0d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:20.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.925+0000 7f8011810700 1 -- 172.21.15.16:0/2426580274 shutdown_connections 2023-10-16T12:26:20.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.925+0000 7f8011810700 1 -- 172.21.15.16:0/2426580274 wait complete. 2023-10-16T12:26:20.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.927+0000 7f8011810700 1 Processor -- start 2023-10-16T12:26:20.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.927+0000 7f8011810700 1 -- start start 2023-10-16T12:26:20.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.927+0000 7f8011810700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7ff4003a30 0x7f800c150630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:20.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.927+0000 7f8011810700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f800c0caa60 0x7f800c0b5e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:20.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.928+0000 7f8011810700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f800c0b6350 0x7f800c1e55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:20.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.928+0000 7f8011810700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f800c0ce830 con 0x7f800c0caa60 2023-10-16T12:26:20.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.928+0000 7f8011810700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f800c0b67e0 con 0x7f7ff4003a30 2023-10-16T12:26:20.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.928+0000 7f8011810700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f800c0c9790 con 0x7f800c0b6350 2023-10-16T12:26:20.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.928+0000 7f800a7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f800c0caa60 0x7f800c0b5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:20.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.928+0000 7f800a7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f800c0caa60 0x7f800c0b5e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39240/0 (socket says 172.21.15.16:39240) 2023-10-16T12:26:20.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.928+0000 7f800a7fc700 1 -- 172.21.15.16:0/1028866535 learned_addr learned my addr 172.21.15.16:0/1028866535 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:20.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.928+0000 7f800b7fe700 1 --2- 172.21.15.16:0/1028866535 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f800c0b6350 0x7f800c1e55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:20.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.928+0000 7f800affd700 1 --2- 172.21.15.16:0/1028866535 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7ff4003a30 0x7f800c150630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:20.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.928+0000 7f800a7fc700 1 -- 172.21.15.16:0/1028866535 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f800c0b6350 msgr2=0x7f800c1e55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:20.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.928+0000 7f800a7fc700 1 --2- 172.21.15.16:0/1028866535 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f800c0b6350 0x7f800c1e55f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:20.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.928+0000 7f800a7fc700 1 -- 172.21.15.16:0/1028866535 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7ff4003a30 msgr2=0x7f800c150630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:20.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.928+0000 7f800a7fc700 1 --2- 172.21.15.16:0/1028866535 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7ff4003a30 0x7f800c150630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:20.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.928+0000 7f800a7fc700 1 -- 172.21.15.16:0/1028866535 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f800c0ca5d0 con 0x7f800c0caa60 2023-10-16T12:26:20.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.928+0000 7f800a7fc700 1 --2- 172.21.15.16:0/1028866535 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f800c0caa60 0x7f800c0b5e10 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f800000f850 tx=0x7f800000fb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:20.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.928+0000 7f7ff3fff700 1 -- 172.21.15.16:0/1028866535 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8000005320 con 0x7f800c0caa60 2023-10-16T12:26:20.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.929+0000 7f7ff3fff700 1 -- 172.21.15.16:0/1028866535 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8000005480 con 0x7f800c0caa60 2023-10-16T12:26:20.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.929+0000 7f7ff3fff700 1 -- 172.21.15.16:0/1028866535 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8000006aa0 con 0x7f800c0caa60 2023-10-16T12:26:20.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.929+0000 7f8011810700 1 -- 172.21.15.16:0/1028866535 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f800c0c99c0 con 0x7f800c0caa60 2023-10-16T12:26:20.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.929+0000 7f8011810700 1 -- 172.21.15.16:0/1028866535 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f800c0c9f80 con 0x7f800c0caa60 2023-10-16T12:26:20.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.930+0000 7f8011810700 1 -- 172.21.15.16:0/1028866535 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f800c0b2970 con 0x7f800c0caa60 2023-10-16T12:26:20.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.931+0000 7f7ff3fff700 1 -- 172.21.15.16:0/1028866535 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 31) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8000003680 con 0x7f800c0caa60 2023-10-16T12:26:20.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.931+0000 7f7ff3fff700 1 --2- 172.21.15.16:0/1028866535 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f7ff806b070 0x7f7ff806d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:20.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.932+0000 7f7ff3fff700 1 -- 172.21.15.16:0/1028866535 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f8000088cd0 con 0x7f800c0caa60 2023-10-16T12:26:20.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.932+0000 7f800affd700 1 --2- 172.21.15.16:0/1028866535 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f7ff806b070 0x7f7ff806d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:20.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.932+0000 7f800affd700 1 --2- 172.21.15.16:0/1028866535 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f7ff806b070 0x7f7ff806d530 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc0090b0 tx=0x7f7ffc01f040 comp rx=0 tx=0).ready entity=mgr.6396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:20.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:20.935+0000 7f7ff3fff700 1 -- 172.21.15.16:0/1028866535 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8000053f40 con 0x7f800c0caa60 2023-10-16T12:26:21.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.140+0000 7f8011810700 1 -- 172.21.15.16:0/1028866535 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f800c0b2970 con 0x7f800c0caa60 2023-10-16T12:26:21.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.141+0000 7f7ff3fff700 1 -- 172.21.15.16:0/1028866535 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) v1 ==== 93+0+10 (secure 0 0 0) 0x7f800c0b2970 con 0x7f800c0caa60 2023-10-16T12:26:21.143 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T12:26:21.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.143+0000 7f8011810700 1 -- 172.21.15.16:0/1028866535 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f7ff806b070 msgr2=0x7f7ff806d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:21.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.143+0000 7f8011810700 1 --2- 172.21.15.16:0/1028866535 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f7ff806b070 0x7f7ff806d530 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc0090b0 tx=0x7f7ffc01f040 comp rx=0 tx=0).stop 2023-10-16T12:26:21.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.143+0000 7f8011810700 1 -- 172.21.15.16:0/1028866535 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f800c0caa60 msgr2=0x7f800c0b5e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:21.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.143+0000 7f8011810700 1 --2- 172.21.15.16:0/1028866535 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f800c0caa60 0x7f800c0b5e10 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f800000f850 tx=0x7f800000fb60 comp rx=0 tx=0).stop 2023-10-16T12:26:21.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.143+0000 7f8011810700 1 -- 172.21.15.16:0/1028866535 shutdown_connections 2023-10-16T12:26:21.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.143+0000 7f8011810700 1 --2- 172.21.15.16:0/1028866535 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f7ff806b070 0x7f7ff806d530 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:21.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.143+0000 7f8011810700 1 --2- 172.21.15.16:0/1028866535 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f800c0b6350 0x7f800c1e55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:21.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.143+0000 7f8011810700 1 --2- 172.21.15.16:0/1028866535 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7ff4003a30 0x7f800c150630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:21.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.143+0000 7f8011810700 1 --2- 172.21.15.16:0/1028866535 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f800c0caa60 0x7f800c0b5e10 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:21.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.143+0000 7f8011810700 1 -- 172.21.15.16:0/1028866535 >> 172.21.15.16:0/1028866535 conn(0x7f800c0d15f0 msgr2=0x7f800c0b4170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:21.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.143+0000 7f8011810700 1 -- 172.21.15.16:0/1028866535 shutdown_connections 2023-10-16T12:26:21.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.143+0000 7f8011810700 1 -- 172.21.15.16:0/1028866535 wait complete. 2023-10-16T12:26:21.157 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-16T12:26:21.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.281+0000 7f5a413a0700 1 Processor -- start 2023-10-16T12:26:21.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.281+0000 7f5a413a0700 1 -- start start 2023-10-16T12:26:21.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.282+0000 7f5a413a0700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5a3c15b570 0x7f5a3c072080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:21.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.282+0000 7f5a413a0700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f5a3c06fdc0 0x7f5a3c06a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:21.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.282+0000 7f5a413a0700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a3c0cc680 con 0x7f5a3c0ca390 2023-10-16T12:26:21.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.282+0000 7f5a413a0700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a3c0cd7a0 con 0x7f5a3c15b570 2023-10-16T12:26:21.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.282+0000 7f5a413a0700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a3c071bf0 con 0x7f5a3c06fdc0 2023-10-16T12:26:21.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.282+0000 7f5a3bfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5a3c15b570 0x7f5a3c072080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:21.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.282+0000 7f5a3bfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5a3c15b570 0x7f5a3c072080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39248/0 (socket says 172.21.15.16:39248) 2023-10-16T12:26:21.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.282+0000 7f5a3bfff700 1 -- 172.21.15.16:0/606792230 learned_addr learned my addr 172.21.15.16:0/606792230 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:21.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.282+0000 7f5a3b7fe700 1 --2- 172.21.15.16:0/606792230 >> v2:172.21.15.49:3300/0 conn(0x7f5a3c06fdc0 0x7f5a3c06a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:21.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.282+0000 7f5a3bfff700 1 -- 172.21.15.16:0/606792230 >> v2:172.21.15.49:3300/0 conn(0x7f5a3c06fdc0 msgr2=0x7f5a3c06a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:21.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.282+0000 7f5a3bfff700 1 --2- 172.21.15.16:0/606792230 >> v2:172.21.15.49:3300/0 conn(0x7f5a3c06fdc0 0x7f5a3c06a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:21.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.282+0000 7f5a3bfff700 1 -- 172.21.15.16:0/606792230 >> v1:172.21.15.16:6789/0 conn(0x7f5a3c0ca390 legacy=0x7f5a3c15f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:21.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.283+0000 7f5a3bfff700 1 -- 172.21.15.16:0/606792230 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a3c15b950 con 0x7f5a3c15b570 2023-10-16T12:26:21.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.283+0000 7f5a3bfff700 1 --2- 172.21.15.16:0/606792230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a3c15b570 0x7f5a3c072080 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c0675e0 tx=0x7f5a2c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1152174f03a87d41 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:21.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.283+0000 7f5a3affd700 1 -- 172.21.15.16:0/606792230 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a2c00b870 con 0x7f5a3c15b570 2023-10-16T12:26:21.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.283+0000 7f5a3affd700 1 -- 172.21.15.16:0/606792230 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a2c00b9d0 con 0x7f5a3c15b570 2023-10-16T12:26:21.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.283+0000 7f5a3affd700 1 -- 172.21.15.16:0/606792230 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a2c00bd60 con 0x7f5a3c15b570 2023-10-16T12:26:21.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.283+0000 7f5a413a0700 1 -- 172.21.15.16:0/606792230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a3c15b570 msgr2=0x7f5a3c072080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:21.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.283+0000 7f5a413a0700 1 --2- 172.21.15.16:0/606792230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a3c15b570 0x7f5a3c072080 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c0675e0 tx=0x7f5a2c009bd0 comp rx=0 tx=0).stop 2023-10-16T12:26:21.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.283+0000 7f5a413a0700 1 -- 172.21.15.16:0/606792230 shutdown_connections 2023-10-16T12:26:21.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.283+0000 7f5a413a0700 1 --2- 172.21.15.16:0/606792230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a3c15b570 0x7f5a3c072080 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:21.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.283+0000 7f5a413a0700 1 --2- 172.21.15.16:0/606792230 >> v2:172.21.15.49:3300/0 conn(0x7f5a3c06fdc0 0x7f5a3c06a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:21.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.283+0000 7f5a413a0700 1 -- 172.21.15.16:0/606792230 >> 172.21.15.16:0/606792230 conn(0x7f5a3c0ccc80 msgr2=0x7f5a3c0cd090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:21.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.283+0000 7f5a413a0700 1 -- 172.21.15.16:0/606792230 shutdown_connections 2023-10-16T12:26:21.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.284+0000 7f5a413a0700 1 -- 172.21.15.16:0/606792230 wait complete. 2023-10-16T12:26:21.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.285+0000 7f5a413a0700 1 Processor -- start 2023-10-16T12:26:21.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.285+0000 7f5a413a0700 1 -- start start 2023-10-16T12:26:21.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.286+0000 7f5a413a0700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5a3c06fdc0 0x7f5a3c1e39f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:21.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.286+0000 7f5a413a0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a3c0ca390 0x7f5a3c1e3370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:21.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.286+0000 7f5a413a0700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5a3c15b570 0x7f5a3c143640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:21.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.286+0000 7f5a413a0700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a3c150f50 con 0x7f5a3c0ca390 2023-10-16T12:26:21.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.286+0000 7f5a413a0700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a3c151780 con 0x7f5a3c06fdc0 2023-10-16T12:26:21.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.286+0000 7f5a413a0700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a3c1540c0 con 0x7f5a3c15b570 2023-10-16T12:26:21.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.286+0000 7f5a3b7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a3c0ca390 0x7f5a3c1e3370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:21.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.286+0000 7f5a3b7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a3c0ca390 0x7f5a3c1e3370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39254/0 (socket says 172.21.15.16:39254) 2023-10-16T12:26:21.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.286+0000 7f5a3b7fe700 1 -- 172.21.15.16:0/2025897559 learned_addr learned my addr 172.21.15.16:0/2025897559 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:21.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.286+0000 7f5a3bfff700 1 --2- 172.21.15.16:0/2025897559 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5a3c06fdc0 0x7f5a3c1e39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:21.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.286+0000 7f5a40b9f700 1 --2- 172.21.15.16:0/2025897559 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5a3c15b570 0x7f5a3c143640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:21.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.286+0000 7f5a3b7fe700 1 -- 172.21.15.16:0/2025897559 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5a3c15b570 msgr2=0x7f5a3c143640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:21.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.286+0000 7f5a3b7fe700 1 --2- 172.21.15.16:0/2025897559 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5a3c15b570 0x7f5a3c143640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:21.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.286+0000 7f5a3b7fe700 1 -- 172.21.15.16:0/2025897559 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5a3c06fdc0 msgr2=0x7f5a3c1e39f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:21.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.286+0000 7f5a3b7fe700 1 --2- 172.21.15.16:0/2025897559 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5a3c06fdc0 0x7f5a3c1e39f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:21.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.286+0000 7f5a3b7fe700 1 -- 172.21.15.16:0/2025897559 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a2c009550 con 0x7f5a3c0ca390 2023-10-16T12:26:21.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.286+0000 7f5a3bfff700 1 --2- 172.21.15.16:0/2025897559 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5a3c06fdc0 0x7f5a3c1e39f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:26:21.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.286+0000 7f5a3b7fe700 1 --2- 172.21.15.16:0/2025897559 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a3c0ca390 0x7f5a3c1e3370 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f5a3000d800 tx=0x7f5a3000dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:21.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.286+0000 7f5a38ff9700 1 -- 172.21.15.16:0/2025897559 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a300125f0 con 0x7f5a3c0ca390 2023-10-16T12:26:21.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.286+0000 7f5a38ff9700 1 -- 172.21.15.16:0/2025897559 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a30012750 con 0x7f5a3c0ca390 2023-10-16T12:26:21.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.286+0000 7f5a413a0700 1 -- 172.21.15.16:0/2025897559 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a3c151130 con 0x7f5a3c0ca390 2023-10-16T12:26:21.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.286+0000 7f5a38ff9700 1 -- 172.21.15.16:0/2025897559 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a30012ae0 con 0x7f5a3c0ca390 2023-10-16T12:26:21.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.286+0000 7f5a413a0700 1 -- 172.21.15.16:0/2025897559 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a3c143b80 con 0x7f5a3c0ca390 2023-10-16T12:26:21.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.287+0000 7f5a413a0700 1 -- 172.21.15.16:0/2025897559 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a3c0b42d0 con 0x7f5a3c0ca390 2023-10-16T12:26:21.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.288+0000 7f5a38ff9700 1 -- 172.21.15.16:0/2025897559 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 31) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5a30012c40 con 0x7f5a3c0ca390 2023-10-16T12:26:21.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.288+0000 7f5a38ff9700 1 --2- 172.21.15.16:0/2025897559 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f5a2806b020 0x7f5a2806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:21.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.288+0000 7f5a38ff9700 1 -- 172.21.15.16:0/2025897559 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f5a300898c0 con 0x7f5a3c0ca390 2023-10-16T12:26:21.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.288+0000 7f5a3bfff700 1 --2- 172.21.15.16:0/2025897559 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f5a2806b020 0x7f5a2806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:21.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.289+0000 7f5a3bfff700 1 --2- 172.21.15.16:0/2025897559 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f5a2806b020 0x7f5a2806d4e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5a2c000c00 tx=0x7f5a2c008030 comp rx=0 tx=0).ready entity=mgr.6396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:21.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.293+0000 7f5a38ff9700 1 -- 172.21.15.16:0/2025897559 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5a30054a40 con 0x7f5a3c0ca390 2023-10-16T12:26:21.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.499+0000 7f5a413a0700 1 -- 172.21.15.16:0/2025897559 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f5a3c04ebb0 con 0x7f5a3c0ca390 2023-10-16T12:26:21.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.500+0000 7f5a38ff9700 1 -- 172.21.15.16:0/2025897559 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v81) v1 ==== 84+0+65 (secure 0 0 0) 0x7f5a30058060 con 0x7f5a3c0ca390 2023-10-16T12:26:21.502 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:26:21.503 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2023-10-16T12:26:21.503 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:26:21.503 INFO:teuthology.orchestra.run.smithi016.stdout: "poolnum": 1, 2023-10-16T12:26:21.503 INFO:teuthology.orchestra.run.smithi016.stdout: "poolname": ".mgr" 2023-10-16T12:26:21.503 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:26:21.504 INFO:teuthology.orchestra.run.smithi016.stdout:] 2023-10-16T12:26:21.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.502+0000 7f5a413a0700 1 -- 172.21.15.16:0/2025897559 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f5a2806b020 msgr2=0x7f5a2806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:21.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.502+0000 7f5a413a0700 1 --2- 172.21.15.16:0/2025897559 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f5a2806b020 0x7f5a2806d4e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5a2c000c00 tx=0x7f5a2c008030 comp rx=0 tx=0).stop 2023-10-16T12:26:21.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.502+0000 7f5a413a0700 1 -- 172.21.15.16:0/2025897559 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a3c0ca390 msgr2=0x7f5a3c1e3370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:21.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.502+0000 7f5a413a0700 1 --2- 172.21.15.16:0/2025897559 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a3c0ca390 0x7f5a3c1e3370 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f5a3000d800 tx=0x7f5a3000dbc0 comp rx=0 tx=0).stop 2023-10-16T12:26:21.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.502+0000 7f5a413a0700 1 -- 172.21.15.16:0/2025897559 shutdown_connections 2023-10-16T12:26:21.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.502+0000 7f5a413a0700 1 --2- 172.21.15.16:0/2025897559 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f5a2806b020 0x7f5a2806d4e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:21.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.502+0000 7f5a413a0700 1 --2- 172.21.15.16:0/2025897559 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5a3c15b570 0x7f5a3c143640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:21.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.502+0000 7f5a413a0700 1 --2- 172.21.15.16:0/2025897559 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5a3c06fdc0 0x7f5a3c1e39f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:21.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.502+0000 7f5a413a0700 1 --2- 172.21.15.16:0/2025897559 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a3c0ca390 0x7f5a3c1e3370 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:21.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.502+0000 7f5a413a0700 1 -- 172.21.15.16:0/2025897559 >> 172.21.15.16:0/2025897559 conn(0x7f5a3c0ccc80 msgr2=0x7f5a3c06d660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:21.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.502+0000 7f5a413a0700 1 -- 172.21.15.16:0/2025897559 shutdown_connections 2023-10-16T12:26:21.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.502+0000 7f5a413a0700 1 -- 172.21.15.16:0/2025897559 wait complete. 2023-10-16T12:26:21.516 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-16T12:26:21.517 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-16T12:26:21.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.644+0000 7f9c4d1a5700 1 Processor -- start 2023-10-16T12:26:21.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.644+0000 7f9c4d1a5700 1 -- start start 2023-10-16T12:26:21.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.645+0000 7f9c4d1a5700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f9c4815b500 0x7f9c4815fab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:21.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.645+0000 7f9c4d1a5700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9c4806e590 0x7f9c480ca930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:21.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.645+0000 7f9c4d1a5700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9c480cae70 0x7f9c48071970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:21.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.645+0000 7f9c4d1a5700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c48160090 con 0x7f9c480cae70 2023-10-16T12:26:21.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.645+0000 7f9c4d1a5700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c480cb320 con 0x7f9c4815b500 2023-10-16T12:26:21.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.645+0000 7f9c4d1a5700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c480cc800 con 0x7f9c4806e590 2023-10-16T12:26:21.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.645+0000 7f9c4759e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9c480cae70 0x7f9c48071970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:21.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.645+0000 7f9c4759e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9c480cae70 0x7f9c48071970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39262/0 (socket says 172.21.15.16:39262) 2023-10-16T12:26:21.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.645+0000 7f9c4759e700 1 -- 172.21.15.16:0/1140054608 learned_addr learned my addr 172.21.15.16:0/1140054608 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:21.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.645+0000 7f9c46d9d700 1 --2- 172.21.15.16:0/1140054608 >> v2:172.21.15.49:3300/0 conn(0x7f9c4815b500 0x7f9c4815fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:21.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.645+0000 7f9c4659c700 1 --2- 172.21.15.16:0/1140054608 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9c4806e590 0x7f9c480ca930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:21.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.646+0000 7f9c4759e700 1 -- 172.21.15.16:0/1140054608 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9c4806e590 msgr2=0x7f9c480ca930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:21.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.646+0000 7f9c4759e700 1 --2- 172.21.15.16:0/1140054608 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9c4806e590 0x7f9c480ca930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:21.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.646+0000 7f9c4759e700 1 -- 172.21.15.16:0/1140054608 >> v2:172.21.15.49:3300/0 conn(0x7f9c4815b500 msgr2=0x7f9c4815fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:21.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.646+0000 7f9c4759e700 1 --2- 172.21.15.16:0/1140054608 >> v2:172.21.15.49:3300/0 conn(0x7f9c4815b500 0x7f9c4815fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:21.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.646+0000 7f9c4759e700 1 -- 172.21.15.16:0/1140054608 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c4806b220 con 0x7f9c480cae70 2023-10-16T12:26:21.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.646+0000 7f9c4759e700 1 --2- 172.21.15.16:0/1140054608 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c480cae70 0x7f9c48071970 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f9c48073080 tx=0x7f9c4000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8842a9b98022dd6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:21.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.646+0000 7f9c45d9b700 1 -- 172.21.15.16:0/1140054608 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c4000e070 con 0x7f9c480cae70 2023-10-16T12:26:21.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.646+0000 7f9c45d9b700 1 -- 172.21.15.16:0/1140054608 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c4000b910 con 0x7f9c480cae70 2023-10-16T12:26:21.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.646+0000 7f9c45d9b700 1 -- 172.21.15.16:0/1140054608 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c4000bcc0 con 0x7f9c480cae70 2023-10-16T12:26:21.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.646+0000 7f9c4d1a5700 1 -- 172.21.15.16:0/1140054608 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c480cae70 msgr2=0x7f9c48071970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:21.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.646+0000 7f9c4d1a5700 1 --2- 172.21.15.16:0/1140054608 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c480cae70 0x7f9c48071970 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f9c48073080 tx=0x7f9c4000b330 comp rx=0 tx=0).stop 2023-10-16T12:26:21.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.646+0000 7f9c4d1a5700 1 -- 172.21.15.16:0/1140054608 shutdown_connections 2023-10-16T12:26:21.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.646+0000 7f9c4d1a5700 1 --2- 172.21.15.16:0/1140054608 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c480cae70 0x7f9c48071970 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:21.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.646+0000 7f9c4d1a5700 1 --2- 172.21.15.16:0/1140054608 >> v2:172.21.15.49:3300/0 conn(0x7f9c4815b500 0x7f9c4815fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:21.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.646+0000 7f9c4d1a5700 1 --2- 172.21.15.16:0/1140054608 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9c4806e590 0x7f9c480ca930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:21.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.646+0000 7f9c4d1a5700 1 -- 172.21.15.16:0/1140054608 >> 172.21.15.16:0/1140054608 conn(0x7f9c4806f430 msgr2=0x7f9c4806f810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:21.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.646+0000 7f9c4d1a5700 1 -- 172.21.15.16:0/1140054608 shutdown_connections 2023-10-16T12:26:21.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.646+0000 7f9c4d1a5700 1 -- 172.21.15.16:0/1140054608 wait complete. 2023-10-16T12:26:21.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.648+0000 7f9c4d1a5700 1 Processor -- start 2023-10-16T12:26:21.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.648+0000 7f9c4d1a5700 1 -- start start 2023-10-16T12:26:21.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.649+0000 7f9c4d1a5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c4806e590 0x7f9c4814e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:21.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.649+0000 7f9c4d1a5700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9c480cae70 0x7f9c4815ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:21.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.649+0000 7f9c4d1a5700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9c4815b500 0x7f9c480a8dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:21.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.649+0000 7f9c4d1a5700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c4815d960 con 0x7f9c4806e590 2023-10-16T12:26:21.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.649+0000 7f9c4d1a5700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c4815dc40 con 0x7f9c480cae70 2023-10-16T12:26:21.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.649+0000 7f9c4d1a5700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c4815df20 con 0x7f9c4815b500 2023-10-16T12:26:21.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.649+0000 7f9c46d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c4806e590 0x7f9c4814e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:21.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.649+0000 7f9c46d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c4806e590 0x7f9c4814e310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39266/0 (socket says 172.21.15.16:39266) 2023-10-16T12:26:21.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.649+0000 7f9c46d9d700 1 -- 172.21.15.16:0/1855727319 learned_addr learned my addr 172.21.15.16:0/1855727319 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:21.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.649+0000 7f9c4659c700 1 --2- 172.21.15.16:0/1855727319 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9c480cae70 0x7f9c4815ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:21.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.649+0000 7f9c4759e700 1 --2- 172.21.15.16:0/1855727319 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9c4815b500 0x7f9c480a8dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:21.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.649+0000 7f9c46d9d700 1 -- 172.21.15.16:0/1855727319 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9c4815b500 msgr2=0x7f9c480a8dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:21.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.649+0000 7f9c46d9d700 1 --2- 172.21.15.16:0/1855727319 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9c4815b500 0x7f9c480a8dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:21.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.649+0000 7f9c46d9d700 1 -- 172.21.15.16:0/1855727319 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9c480cae70 msgr2=0x7f9c4815ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:21.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.649+0000 7f9c46d9d700 1 --2- 172.21.15.16:0/1855727319 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9c480cae70 0x7f9c4815ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:21.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.649+0000 7f9c46d9d700 1 -- 172.21.15.16:0/1855727319 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c40009b30 con 0x7f9c4806e590 2023-10-16T12:26:21.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.649+0000 7f9c4659c700 1 --2- 172.21.15.16:0/1855727319 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9c480cae70 0x7f9c4815ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:26:21.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.649+0000 7f9c4759e700 1 --2- 172.21.15.16:0/1855727319 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9c4815b500 0x7f9c480a8dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:26:21.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.649+0000 7f9c46d9d700 1 --2- 172.21.15.16:0/1855727319 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c4806e590 0x7f9c4814e310 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f9c3c00e870 tx=0x7f9c3c00ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:21.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.649+0000 7f9c2f7fe700 1 -- 172.21.15.16:0/1855727319 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c3c009590 con 0x7f9c4806e590 2023-10-16T12:26:21.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.649+0000 7f9c2f7fe700 1 -- 172.21.15.16:0/1855727319 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c3c00ee70 con 0x7f9c4806e590 2023-10-16T12:26:21.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.650+0000 7f9c4d1a5700 1 -- 172.21.15.16:0/1855727319 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c4815e200 con 0x7f9c4806e590 2023-10-16T12:26:21.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.650+0000 7f9c2f7fe700 1 -- 172.21.15.16:0/1855727319 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c3c00ca50 con 0x7f9c4806e590 2023-10-16T12:26:21.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.650+0000 7f9c4d1a5700 1 -- 172.21.15.16:0/1855727319 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c4815d580 con 0x7f9c4806e590 2023-10-16T12:26:21.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.651+0000 7f9c4d1a5700 1 -- 172.21.15.16:0/1855727319 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c4815e850 con 0x7f9c4806e590 2023-10-16T12:26:21.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.651+0000 7f9c2f7fe700 1 -- 172.21.15.16:0/1855727319 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 31) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9c3c00cbb0 con 0x7f9c4806e590 2023-10-16T12:26:21.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.651+0000 7f9c2f7fe700 1 --2- 172.21.15.16:0/1855727319 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f9c3406b020 0x7f9c3406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:21.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.652+0000 7f9c2f7fe700 1 -- 172.21.15.16:0/1855727319 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f9c3c0894c0 con 0x7f9c4806e590 2023-10-16T12:26:21.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.652+0000 7f9c4659c700 1 --2- 172.21.15.16:0/1855727319 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f9c3406b020 0x7f9c3406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:21.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.652+0000 7f9c4659c700 1 --2- 172.21.15.16:0/1855727319 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f9c3406b020 0x7f9c3406d4e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9c4815a2c0 tx=0x7f9c38008040 comp rx=0 tx=0).ready entity=mgr.6396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:21.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.656+0000 7f9c2f7fe700 1 -- 172.21.15.16:0/1855727319 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9c3c054640 con 0x7f9c4806e590 2023-10-16T12:26:21.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:21.861+0000 7f9c4d1a5700 1 -- 172.21.15.16:0/1855727319 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f9c4804ec20 con 0x7f9c4806e590 2023-10-16T12:26:22.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:22.240+0000 7f9c2f7fe700 1 -- 172.21.15.16:0/1855727319 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v82) v1 ==== 138+0+0 (secure 0 0 0) 0x7f9c3c00ce60 con 0x7f9c4806e590 2023-10-16T12:26:22.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:22.243+0000 7f9c4d1a5700 1 -- 172.21.15.16:0/1855727319 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f9c3406b020 msgr2=0x7f9c3406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:22.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:22.243+0000 7f9c4d1a5700 1 --2- 172.21.15.16:0/1855727319 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f9c3406b020 0x7f9c3406d4e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9c4815a2c0 tx=0x7f9c38008040 comp rx=0 tx=0).stop 2023-10-16T12:26:22.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:22.243+0000 7f9c4d1a5700 1 -- 172.21.15.16:0/1855727319 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c4806e590 msgr2=0x7f9c4814e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:22.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:22.243+0000 7f9c4d1a5700 1 --2- 172.21.15.16:0/1855727319 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c4806e590 0x7f9c4814e310 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f9c3c00e870 tx=0x7f9c3c00ec30 comp rx=0 tx=0).stop 2023-10-16T12:26:22.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:22.244+0000 7f9c4d1a5700 1 -- 172.21.15.16:0/1855727319 shutdown_connections 2023-10-16T12:26:22.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:22.244+0000 7f9c4d1a5700 1 --2- 172.21.15.16:0/1855727319 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f9c3406b020 0x7f9c3406d4e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:22.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:22.244+0000 7f9c4d1a5700 1 --2- 172.21.15.16:0/1855727319 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9c4815b500 0x7f9c480a8dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:22.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:22.244+0000 7f9c4d1a5700 1 --2- 172.21.15.16:0/1855727319 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c4806e590 0x7f9c4814e310 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:22.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:22.244+0000 7f9c4d1a5700 1 --2- 172.21.15.16:0/1855727319 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9c480cae70 0x7f9c4815ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:22.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:22.244+0000 7f9c4d1a5700 1 -- 172.21.15.16:0/1855727319 >> 172.21.15.16:0/1855727319 conn(0x7f9c4806f430 msgr2=0x7f9c480afda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:22.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:22.244+0000 7f9c4d1a5700 1 -- 172.21.15.16:0/1855727319 shutdown_connections 2023-10-16T12:26:22.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:22.244+0000 7f9c4d1a5700 1 -- 172.21.15.16:0/1855727319 wait complete. 2023-10-16T12:26:22.249 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' created 2023-10-16T12:26:22.263 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-16T12:26:22.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:22.397+0000 7f70d5a7c700 1 Processor -- start 2023-10-16T12:26:22.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:22.397+0000 7f70d5a7c700 1 -- start start 2023-10-16T12:26:22.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:22.398+0000 7f70d5a7c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f70d015a640 0x7f70d00cb710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:22.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:22.398+0000 7f70d5a7c700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f70d015e140 0x7f70d00cbc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:22.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:22.398+0000 7f70d5a7c700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70d015aa20 con 0x7f70d00c6770 2023-10-16T12:26:22.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:22.398+0000 7f70d5a7c700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70d006eb40 con 0x7f70d015a640 2023-10-16T12:26:22.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:22.398+0000 7f70d5a7c700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70d006ee20 con 0x7f70d015e140 2023-10-16T12:26:22.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:22.398+0000 7f70cf7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f70d015a640 0x7f70d00cb710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:22.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:22.398+0000 7f70ceffd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f70d00c6770 0x7f70d00ca1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:56782/0 (socket says 172.21.15.16:56782) 2023-10-16T12:26:22.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:22.398+0000 7f70cf7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f70d015a640 0x7f70d00cb710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39274/0 (socket says 172.21.15.16:39274) 2023-10-16T12:26:22.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:22.398+0000 7f70ceffd700 1 -- 172.21.15.16:0/3014445564 learned_addr learned my addr 172.21.15.16:0/3014445564 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:26:22.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:22.398+0000 7f70cffff700 1 --2- 172.21.15.16:0/3014445564 >> v2:172.21.15.49:3300/0 conn(0x7f70d015e140 0x7f70d00cbc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:22.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:22.398+0000 7f70cf7fe700 1 -- 172.21.15.16:0/3014445564 >> v2:172.21.15.49:3300/0 conn(0x7f70d015e140 msgr2=0x7f70d00cbc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:22.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:22.398+0000 7f70cf7fe700 1 --2- 172.21.15.16:0/3014445564 >> v2:172.21.15.49:3300/0 conn(0x7f70d015e140 0x7f70d00cbc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:22.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:22.398+0000 7f70cf7fe700 1 -- 172.21.15.16:0/3014445564 >> v1:172.21.15.16:6789/0 conn(0x7f70d00c6770 legacy=0x7f70d00ca1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:22.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:22.398+0000 7f70cf7fe700 1 -- 172.21.15.16:0/3014445564 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70d006f100 con 0x7f70d015a640 2023-10-16T12:26:22.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:22.398+0000 7f70cffff700 1 --2- 172.21.15.16:0/3014445564 >> v2:172.21.15.49:3300/0 conn(0x7f70d015e140 0x7f70d00cbc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:26:22.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:22.399+0000 7f70cf7fe700 1 --2- 172.21.15.16:0/3014445564 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70d015a640 0x7f70d00cb710 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f70d0071180 tx=0x7f70c8009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2899e9a5ff0e28ee server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:22.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:22.399+0000 7f70ce7fc700 1 -- 172.21.15.16:0/3014445564 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70c8011830 con 0x7f70d015a640 2023-10-16T12:26:22.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:22.399+0000 7f70ce7fc700 1 -- 172.21.15.16:0/3014445564 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70c8011990 con 0x7f70d015a640 2023-10-16T12:26:22.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:22.399+0000 7f70ce7fc700 1 -- 172.21.15.16:0/3014445564 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70c8011d20 con 0x7f70d015a640 2023-10-16T12:26:22.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:22.399+0000 7f70d5a7c700 1 -- 172.21.15.16:0/3014445564 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70d015a640 msgr2=0x7f70d00cb710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:22.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:22.399+0000 7f70d5a7c700 1 --2- 172.21.15.16:0/3014445564 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70d015a640 0x7f70d00cb710 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f70d0071180 tx=0x7f70c8009b20 comp rx=0 tx=0).stop 2023-10-16T12:26:22.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:22.399+0000 7f70d5a7c700 1 -- 172.21.15.16:0/3014445564 shutdown_connections 2023-10-16T12:26:22.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:22.399+0000 7f70d5a7c700 1 --2- 172.21.15.16:0/3014445564 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70d015a640 0x7f70d00cb710 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:22.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:22.399+0000 7f70d5a7c700 1 --2- 172.21.15.16:0/3014445564 >> v2:172.21.15.49:3300/0 conn(0x7f70d015e140 0x7f70d00cbc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:22.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:22.399+0000 7f70d5a7c700 1 -- 172.21.15.16:0/3014445564 >> 172.21.15.16:0/3014445564 conn(0x7f70d00cc5e0 msgr2=0x7f70d00cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:22.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:22.399+0000 7f70d5a7c700 1 -- 172.21.15.16:0/3014445564 shutdown_connections 2023-10-16T12:26:22.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:22.399+0000 7f70d5a7c700 1 -- 172.21.15.16:0/3014445564 wait complete. 2023-10-16T12:26:22.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:22.401+0000 7f70d5a7c700 1 Processor -- start 2023-10-16T12:26:22.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:22.401+0000 7f70d5a7c700 1 -- start start 2023-10-16T12:26:22.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:22.401+0000 7f70d5a7c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f70d00c6770 0x7f70d00b3c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:22.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:22.401+0000 7f70d5a7c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70d015a640 0x7f70d01f19d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:22.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:22.402+0000 7f70d5a7c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f70d015e140 0x7f70d01f1f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:22.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:22.402+0000 7f70d5a7c700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70d00b2b60 con 0x7f70d015a640 2023-10-16T12:26:22.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:22.402+0000 7f70d5a7c700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70d00b2e40 con 0x7f70d015e140 2023-10-16T12:26:22.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:22.402+0000 7f70d5a7c700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70d00b3120 con 0x7f70d00c6770 2023-10-16T12:26:22.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:22.402+0000 7f70ceffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70d015a640 0x7f70d01f19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:22.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:22.402+0000 7f70ceffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70d015a640 0x7f70d01f19d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39282/0 (socket says 172.21.15.16:39282) 2023-10-16T12:26:22.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:22.402+0000 7f70ceffd700 1 -- 172.21.15.16:0/2296852816 learned_addr learned my addr 172.21.15.16:0/2296852816 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:22.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:22.402+0000 7f70cffff700 1 --2- 172.21.15.16:0/2296852816 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f70d015e140 0x7f70d01f1f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:22.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:22.402+0000 7f70cf7fe700 1 --2- 172.21.15.16:0/2296852816 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f70d00c6770 0x7f70d00b3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:22.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:22.402+0000 7f70ceffd700 1 -- 172.21.15.16:0/2296852816 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f70d00c6770 msgr2=0x7f70d00b3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:22.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:22.402+0000 7f70ceffd700 1 --2- 172.21.15.16:0/2296852816 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f70d00c6770 0x7f70d00b3c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:22.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:22.402+0000 7f70ceffd700 1 -- 172.21.15.16:0/2296852816 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f70d015e140 msgr2=0x7f70d01f1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:22.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:22.402+0000 7f70ceffd700 1 --2- 172.21.15.16:0/2296852816 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f70d015e140 0x7f70d01f1f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:22.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:22.402+0000 7f70ceffd700 1 -- 172.21.15.16:0/2296852816 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70c80094a0 con 0x7f70d015a640 2023-10-16T12:26:22.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:22.402+0000 7f70cf7fe700 1 --2- 172.21.15.16:0/2296852816 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f70d00c6770 0x7f70d00b3c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:26:22.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:22.402+0000 7f70ceffd700 1 --2- 172.21.15.16:0/2296852816 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70d015a640 0x7f70d01f19d0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f70b800c500 tx=0x7f70b800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:22.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:22.402+0000 7f70c7fff700 1 -- 172.21.15.16:0/2296852816 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70b8005d70 con 0x7f70d015a640 2023-10-16T12:26:22.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:22.402+0000 7f70c7fff700 1 -- 172.21.15.16:0/2296852816 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70b80070d0 con 0x7f70d015a640 2023-10-16T12:26:22.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:22.402+0000 7f70d5a7c700 1 -- 172.21.15.16:0/2296852816 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70d00b3400 con 0x7f70d015a640 2023-10-16T12:26:22.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:22.402+0000 7f70c7fff700 1 -- 172.21.15.16:0/2296852816 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70b8007440 con 0x7f70d015a640 2023-10-16T12:26:22.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:22.402+0000 7f70d5a7c700 1 -- 172.21.15.16:0/2296852816 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70d01562f0 con 0x7f70d015a640 2023-10-16T12:26:22.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:22.403+0000 7f70d5a7c700 1 -- 172.21.15.16:0/2296852816 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70d00b3920 con 0x7f70d015a640 2023-10-16T12:26:22.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:22.404+0000 7f70c7fff700 1 -- 172.21.15.16:0/2296852816 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 31) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f70b80075a0 con 0x7f70d015a640 2023-10-16T12:26:22.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:22.404+0000 7f70c7fff700 1 --2- 172.21.15.16:0/2296852816 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f70b006b020 0x7f70b006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:22.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:22.404+0000 7f70c7fff700 1 -- 172.21.15.16:0/2296852816 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5575+0+0 (secure 0 0 0) 0x7f70b8088860 con 0x7f70d015a640 2023-10-16T12:26:22.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:22.404+0000 7f70cf7fe700 1 --2- 172.21.15.16:0/2296852816 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f70b006b020 0x7f70b006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:22.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:22.405+0000 7f70cf7fe700 1 --2- 172.21.15.16:0/2296852816 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f70b006b020 0x7f70b006d4e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f70c80090b0 tx=0x7f70c8008030 comp rx=0 tx=0).ready entity=mgr.6396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:22.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:22.409+0000 7f70c7fff700 1 -- 172.21.15.16:0/2296852816 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f70d00b3920 con 0x7f70d015a640 2023-10-16T12:26:22.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:22.611+0000 7f70d5a7c700 1 -- 172.21.15.16:0/2296852816 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f70d015aa20 con 0x7f70d015a640 2023-10-16T12:26:23.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:23.243+0000 7f70c7fff700 1 -- 172.21.15.16:0/2296852816 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v83) v1 ==== 170+0+0 (secure 0 0 0) 0x7f70d015aa20 con 0x7f70d015a640 2023-10-16T12:26:23.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:23.247+0000 7f70d5a7c700 1 -- 172.21.15.16:0/2296852816 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f70b006b020 msgr2=0x7f70b006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:23.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:23.247+0000 7f70d5a7c700 1 --2- 172.21.15.16:0/2296852816 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f70b006b020 0x7f70b006d4e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f70c80090b0 tx=0x7f70c8008030 comp rx=0 tx=0).stop 2023-10-16T12:26:23.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:23.247+0000 7f70d5a7c700 1 -- 172.21.15.16:0/2296852816 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70d015a640 msgr2=0x7f70d01f19d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:23.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:23.247+0000 7f70d5a7c700 1 --2- 172.21.15.16:0/2296852816 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70d015a640 0x7f70d01f19d0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f70b800c500 tx=0x7f70b800c8c0 comp rx=0 tx=0).stop 2023-10-16T12:26:23.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:23.247+0000 7f70d5a7c700 1 -- 172.21.15.16:0/2296852816 shutdown_connections 2023-10-16T12:26:23.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:23.247+0000 7f70d5a7c700 1 --2- 172.21.15.16:0/2296852816 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f70b006b020 0x7f70b006d4e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:23.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:23.247+0000 7f70d5a7c700 1 --2- 172.21.15.16:0/2296852816 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f70d015e140 0x7f70d01f1f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:23.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:23.247+0000 7f70d5a7c700 1 --2- 172.21.15.16:0/2296852816 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f70d00c6770 0x7f70d00b3c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:23.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:23.247+0000 7f70d5a7c700 1 --2- 172.21.15.16:0/2296852816 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70d015a640 0x7f70d01f19d0 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:23.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:23.247+0000 7f70d5a7c700 1 -- 172.21.15.16:0/2296852816 >> 172.21.15.16:0/2296852816 conn(0x7f70d00cc5e0 msgr2=0x7f70d00c3c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:23.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:23.248+0000 7f70d5a7c700 1 -- 172.21.15.16:0/2296852816 shutdown_connections 2023-10-16T12:26:23.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:23.248+0000 7f70d5a7c700 1 -- 172.21.15.16:0/2296852816 wait complete. 2023-10-16T12:26:23.252 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' created 2023-10-16T12:26:23.274 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-16T12:26:23.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:23.406+0000 7fefe157e700 1 Processor -- start 2023-10-16T12:26:23.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:23.406+0000 7fefe157e700 1 -- start start 2023-10-16T12:26:23.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:23.406+0000 7fefe157e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fefdc0b36b0 0x7fefdc06be40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:23.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:23.406+0000 7fefe157e700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fefdc162390 0x7fefdc06c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:23.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:23.406+0000 7fefe157e700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefdc161f00 con 0x7fefdc15d8c0 2023-10-16T12:26:23.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:23.406+0000 7fefe157e700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefdc15f180 con 0x7fefdc0b36b0 2023-10-16T12:26:23.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:23.406+0000 7fefe157e700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefdc15ecb0 con 0x7fefdc162390 2023-10-16T12:26:23.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:23.407+0000 7fefda7fc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fefdc0b36b0 0x7fefdc06be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:23.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:23.407+0000 7fefda7fc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fefdc0b36b0 0x7fefdc06be40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39290/0 (socket says 172.21.15.16:39290) 2023-10-16T12:26:23.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:23.407+0000 7fefda7fc700 1 -- 172.21.15.16:0/1179487492 learned_addr learned my addr 172.21.15.16:0/1179487492 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:23.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:23.407+0000 7fefdb7fe700 1 --2- 172.21.15.16:0/1179487492 >> v2:172.21.15.49:3300/0 conn(0x7fefdc162390 0x7fefdc06c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:23.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:23.407+0000 7fefda7fc700 1 -- 172.21.15.16:0/1179487492 >> v2:172.21.15.49:3300/0 conn(0x7fefdc162390 msgr2=0x7fefdc06c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:23.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:23.407+0000 7fefda7fc700 1 --2- 172.21.15.16:0/1179487492 >> v2:172.21.15.49:3300/0 conn(0x7fefdc162390 0x7fefdc06c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:23.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:23.407+0000 7fefda7fc700 1 -- 172.21.15.16:0/1179487492 >> v1:172.21.15.16:6789/0 conn(0x7fefdc15d8c0 legacy=0x7fefdc0b2fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:23.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:23.407+0000 7fefd9ffb700 1 -- 172.21.15.16:0/1179487492 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2176073868 0 0) 0x7fefdc161f00 con 0x7fefdc15d8c0 2023-10-16T12:26:23.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:23.407+0000 7fefda7fc700 1 -- 172.21.15.16:0/1179487492 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefdc15eea0 con 0x7fefdc0b36b0 2023-10-16T12:26:23.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:23.407+0000 7fefdb7fe700 1 --2- 172.21.15.16:0/1179487492 >> v2:172.21.15.49:3300/0 conn(0x7fefdc162390 0x7fefdc06c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:26:23.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:23.407+0000 7fefda7fc700 1 --2- 172.21.15.16:0/1179487492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fefdc0b36b0 0x7fefdc06be40 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fefdc161670 tx=0x7fefd0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de2ec28d271619cc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:23.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:23.407+0000 7fefd9ffb700 1 -- 172.21.15.16:0/1179487492 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefd000b860 con 0x7fefdc0b36b0 2023-10-16T12:26:23.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:23.407+0000 7fefd9ffb700 1 -- 172.21.15.16:0/1179487492 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefd000b9c0 con 0x7fefdc0b36b0 2023-10-16T12:26:23.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:23.407+0000 7fefd9ffb700 1 -- 172.21.15.16:0/1179487492 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefd000bd50 con 0x7fefdc0b36b0 2023-10-16T12:26:23.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:23.408+0000 7fefe157e700 1 -- 172.21.15.16:0/1179487492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fefdc0b36b0 msgr2=0x7fefdc06be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:23.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:23.408+0000 7fefe157e700 1 --2- 172.21.15.16:0/1179487492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fefdc0b36b0 0x7fefdc06be40 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fefdc161670 tx=0x7fefd0009c00 comp rx=0 tx=0).stop 2023-10-16T12:26:23.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:23.408+0000 7fefe157e700 1 -- 172.21.15.16:0/1179487492 shutdown_connections 2023-10-16T12:26:23.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:23.408+0000 7fefe157e700 1 --2- 172.21.15.16:0/1179487492 >> v2:172.21.15.49:3300/0 conn(0x7fefdc162390 0x7fefdc06c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:23.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:23.408+0000 7fefe157e700 1 --2- 172.21.15.16:0/1179487492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fefdc0b36b0 0x7fefdc06be40 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:23.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:23.408+0000 7fefe157e700 1 -- 172.21.15.16:0/1179487492 >> 172.21.15.16:0/1179487492 conn(0x7fefdc0cdfa0 msgr2=0x7fefdc0ce3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:23.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:23.408+0000 7fefe157e700 1 -- 172.21.15.16:0/1179487492 shutdown_connections 2023-10-16T12:26:23.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:23.408+0000 7fefe157e700 1 -- 172.21.15.16:0/1179487492 wait complete. 2023-10-16T12:26:23.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:23.409+0000 7fefe157e700 1 Processor -- start 2023-10-16T12:26:23.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:23.409+0000 7fefe157e700 1 -- start start 2023-10-16T12:26:23.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:23.410+0000 7fefe157e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fefdc0b36b0 0x7fefdc0cf7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:23.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:23.410+0000 7fefe157e700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fefdc15d8c0 0x7fefdc1589c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:23.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:23.410+0000 7fefe157e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fefdc162390 0x7fefdc0ce380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:23.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:23.410+0000 7fefe157e700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefdc1f3dc0 con 0x7fefdc162390 2023-10-16T12:26:23.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:23.410+0000 7fefe157e700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefdc1f3140 con 0x7fefdc15d8c0 2023-10-16T12:26:23.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:23.410+0000 7fefe157e700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefdc1f3420 con 0x7fefdc0b36b0 2023-10-16T12:26:23.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:23.410+0000 7fefdb7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fefdc162390 0x7fefdc0ce380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:23.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:23.410+0000 7fefdb7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fefdc162390 0x7fefdc0ce380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39296/0 (socket says 172.21.15.16:39296) 2023-10-16T12:26:23.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:23.410+0000 7fefdb7fe700 1 -- 172.21.15.16:0/227661481 learned_addr learned my addr 172.21.15.16:0/227661481 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:23.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:23.410+0000 7fefda7fc700 1 --2- 172.21.15.16:0/227661481 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fefdc15d8c0 0x7fefdc1589c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:23.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:23.410+0000 7fefdaffd700 1 --2- 172.21.15.16:0/227661481 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fefdc0b36b0 0x7fefdc0cf7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:23.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:23.410+0000 7fefdb7fe700 1 -- 172.21.15.16:0/227661481 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fefdc0b36b0 msgr2=0x7fefdc0cf7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:23.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:23.410+0000 7fefdb7fe700 1 --2- 172.21.15.16:0/227661481 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fefdc0b36b0 0x7fefdc0cf7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:23.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:23.410+0000 7fefdb7fe700 1 -- 172.21.15.16:0/227661481 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fefdc15d8c0 msgr2=0x7fefdc1589c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:23.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:23.410+0000 7fefdb7fe700 1 --2- 172.21.15.16:0/227661481 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fefdc15d8c0 0x7fefdc1589c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:23.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:23.410+0000 7fefdb7fe700 1 -- 172.21.15.16:0/227661481 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefd0009550 con 0x7fefdc162390 2023-10-16T12:26:23.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:23.410+0000 7fefdaffd700 1 --2- 172.21.15.16:0/227661481 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fefdc0b36b0 0x7fefdc0cf7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:26:23.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:23.411+0000 7fefdb7fe700 1 --2- 172.21.15.16:0/227661481 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fefdc162390 0x7fefdc0ce380 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fefd400cd50 tx=0x7fefd400a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:23.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:23.411+0000 7fefc37fe700 1 -- 172.21.15.16:0/227661481 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefd400cea0 con 0x7fefdc162390 2023-10-16T12:26:23.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:23.411+0000 7fefc37fe700 1 -- 172.21.15.16:0/227661481 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefd400f480 con 0x7fefdc162390 2023-10-16T12:26:23.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:23.411+0000 7fefc37fe700 1 -- 172.21.15.16:0/227661481 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefd400f810 con 0x7fefdc162390 2023-10-16T12:26:23.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:23.411+0000 7fefe157e700 1 -- 172.21.15.16:0/227661481 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefdc1f3700 con 0x7fefdc162390 2023-10-16T12:26:23.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:23.411+0000 7fefe157e700 1 -- 172.21.15.16:0/227661481 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefdc0cfd00 con 0x7fefdc162390 2023-10-16T12:26:23.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:23.412+0000 7fefe157e700 1 -- 172.21.15.16:0/227661481 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefdc1f3f00 con 0x7fefdc162390 2023-10-16T12:26:23.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:23.413+0000 7fefc37fe700 1 -- 172.21.15.16:0/227661481 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 31) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fefd400b040 con 0x7fefdc162390 2023-10-16T12:26:23.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:23.413+0000 7fefc37fe700 1 --2- 172.21.15.16:0/227661481 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7fefc806b020 0x7fefc806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:23.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:23.413+0000 7fefc37fe700 1 -- 172.21.15.16:0/227661481 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5957+0+0 (secure 0 0 0) 0x7fefd408a020 con 0x7fefdc162390 2023-10-16T12:26:23.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:23.413+0000 7fefdaffd700 1 --2- 172.21.15.16:0/227661481 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7fefc806b020 0x7fefc806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:23.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:23.414+0000 7fefdaffd700 1 --2- 172.21.15.16:0/227661481 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7fefc806b020 0x7fefc806d4e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fefcc000f60 tx=0x7fefcc009040 comp rx=0 tx=0).ready entity=mgr.6396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:23.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:23.418+0000 7fefc37fe700 1 -- 172.21.15.16:0/227661481 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fefd4054e90 con 0x7fefdc162390 2023-10-16T12:26:23.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:23.669+0000 7fefe157e700 1 -- 172.21.15.16:0/227661481 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fefdc1f3f00 con 0x7fefdc162390 2023-10-16T12:26:24.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.285+0000 7fefc37fe700 1 -- 172.21.15.16:0/227661481 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 9 and data pool 10 v32) v1 ==== 173+0+0 (secure 0 0 0) 0x7fefdc1f3f00 con 0x7fefdc162390 2023-10-16T12:26:24.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.293+0000 7fefe157e700 1 -- 172.21.15.16:0/227661481 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7fefc806b020 msgr2=0x7fefc806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:24.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.293+0000 7fefe157e700 1 --2- 172.21.15.16:0/227661481 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7fefc806b020 0x7fefc806d4e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fefcc000f60 tx=0x7fefcc009040 comp rx=0 tx=0).stop 2023-10-16T12:26:24.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.293+0000 7fefe157e700 1 -- 172.21.15.16:0/227661481 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fefdc162390 msgr2=0x7fefdc0ce380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:24.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.293+0000 7fefe157e700 1 --2- 172.21.15.16:0/227661481 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fefdc162390 0x7fefdc0ce380 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fefd400cd50 tx=0x7fefd400a560 comp rx=0 tx=0).stop 2023-10-16T12:26:24.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.294+0000 7fefe157e700 1 -- 172.21.15.16:0/227661481 shutdown_connections 2023-10-16T12:26:24.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.294+0000 7fefe157e700 1 --2- 172.21.15.16:0/227661481 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7fefc806b020 0x7fefc806d4e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:24.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.294+0000 7fefe157e700 1 --2- 172.21.15.16:0/227661481 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fefdc162390 0x7fefdc0ce380 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:24.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.294+0000 7fefe157e700 1 --2- 172.21.15.16:0/227661481 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fefdc0b36b0 0x7fefdc0cf7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:24.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.294+0000 7fefe157e700 1 --2- 172.21.15.16:0/227661481 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fefdc15d8c0 0x7fefdc1589c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:24.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.294+0000 7fefe157e700 1 -- 172.21.15.16:0/227661481 >> 172.21.15.16:0/227661481 conn(0x7fefdc0cdfa0 msgr2=0x7fefdc15cac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:24.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.296+0000 7fefe157e700 1 -- 172.21.15.16:0/227661481 shutdown_connections 2023-10-16T12:26:24.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.299+0000 7fefe157e700 1 -- 172.21.15.16:0/227661481 wait complete. 2023-10-16T12:26:24.306 INFO:teuthology.orchestra.run.smithi016.stderr:new fs with metadata pool 9 and data pool 10 2023-10-16T12:26:24.322 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:26:24.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.450+0000 7fb3b6cfc700 1 Processor -- start 2023-10-16T12:26:24.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.450+0000 7fb3b6cfc700 1 -- start start 2023-10-16T12:26:24.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.451+0000 7fb3b6cfc700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fb3b0160eb0 0x7fb3b006d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:24.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.451+0000 7fb3b6cfc700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb3b006b900 0x7fb3b00caff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:24.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.451+0000 7fb3b6cfc700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3b006c150 con 0x7fb3b00cab70 2023-10-16T12:26:24.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.451+0000 7fb3b6cfc700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3b00af370 con 0x7fb3b0160eb0 2023-10-16T12:26:24.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.451+0000 7fb3b6cfc700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b00cb530 con 0x7fb3b006b900 2023-10-16T12:26:24.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.451+0000 7fb3affff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb3b00cab70 0x7fb3b006b1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:56820/0 (socket says 172.21.15.16:56820) 2023-10-16T12:26:24.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.451+0000 7fb3affff700 1 -- 172.21.15.16:0/3993444007 learned_addr learned my addr 172.21.15.16:0/3993444007 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:26:24.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.451+0000 7fb3b4a98700 1 --2- 172.21.15.16:0/3993444007 >> v2:172.21.15.49:3300/0 conn(0x7fb3b0160eb0 0x7fb3b006d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:24.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.451+0000 7fb3b5299700 1 --2- 172.21.15.16:0/3993444007 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb3b006b900 0x7fb3b00caff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:24.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.451+0000 7fb3af7fe700 1 -- 172.21.15.16:0/3993444007 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2452186636 0 0) 0x7fb3b006c150 con 0x7fb3b00cab70 2023-10-16T12:26:24.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.451+0000 7fb3af7fe700 1 -- 172.21.15.16:0/3993444007 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3a00035b0 con 0x7fb3b00cab70 2023-10-16T12:26:24.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.451+0000 7fb3b5299700 1 -- 172.21.15.16:0/3993444007 >> v2:172.21.15.49:3300/0 conn(0x7fb3b0160eb0 msgr2=0x7fb3b006d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:24.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.451+0000 7fb3b5299700 1 --2- 172.21.15.16:0/3993444007 >> v2:172.21.15.49:3300/0 conn(0x7fb3b0160eb0 0x7fb3b006d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:24.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.451+0000 7fb3b5299700 1 -- 172.21.15.16:0/3993444007 >> v1:172.21.15.16:6789/0 conn(0x7fb3b00cab70 legacy=0x7fb3b006b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:24.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.451+0000 7fb3b5299700 1 -- 172.21.15.16:0/3993444007 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3b006be70 con 0x7fb3b006b900 2023-10-16T12:26:24.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.452+0000 7fb3b5299700 1 --2- 172.21.15.16:0/3993444007 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb3b006b900 0x7fb3b00caff0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb3a8009d90 tx=0x7fb3a800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a965247eb8ec0303 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:24.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.452+0000 7fb3af7fe700 1 -- 172.21.15.16:0/3993444007 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3a800e070 con 0x7fb3b006b900 2023-10-16T12:26:24.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.452+0000 7fb3af7fe700 1 -- 172.21.15.16:0/3993444007 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3a800bb20 con 0x7fb3b006b900 2023-10-16T12:26:24.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.452+0000 7fb3af7fe700 1 -- 172.21.15.16:0/3993444007 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3a800be90 con 0x7fb3b006b900 2023-10-16T12:26:24.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.452+0000 7fb3b6cfc700 1 -- 172.21.15.16:0/3993444007 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb3b006b900 msgr2=0x7fb3b00caff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:24.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.452+0000 7fb3b6cfc700 1 --2- 172.21.15.16:0/3993444007 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb3b006b900 0x7fb3b00caff0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb3a8009d90 tx=0x7fb3a800b330 comp rx=0 tx=0).stop 2023-10-16T12:26:24.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.452+0000 7fb3b6cfc700 1 -- 172.21.15.16:0/3993444007 shutdown_connections 2023-10-16T12:26:24.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.452+0000 7fb3b6cfc700 1 --2- 172.21.15.16:0/3993444007 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb3b006b900 0x7fb3b00caff0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:24.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.452+0000 7fb3b6cfc700 1 --2- 172.21.15.16:0/3993444007 >> v2:172.21.15.49:3300/0 conn(0x7fb3b0160eb0 0x7fb3b006d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:24.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.452+0000 7fb3b6cfc700 1 -- 172.21.15.16:0/3993444007 >> 172.21.15.16:0/3993444007 conn(0x7fb3b00cc5b0 msgr2=0x7fb3b00cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:24.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.452+0000 7fb3b6cfc700 1 -- 172.21.15.16:0/3993444007 shutdown_connections 2023-10-16T12:26:24.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.452+0000 7fb3b6cfc700 1 -- 172.21.15.16:0/3993444007 wait complete. 2023-10-16T12:26:24.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.454+0000 7fb3b6cfc700 1 Processor -- start 2023-10-16T12:26:24.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.454+0000 7fb3b6cfc700 1 -- start start 2023-10-16T12:26:24.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.455+0000 7fb3b6cfc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3b006b900 0x7fb3b0155800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:24.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.455+0000 7fb3b6cfc700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb3b00cab70 0x7fb3b0157080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:24.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.455+0000 7fb3b6cfc700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb3b0160eb0 0x7fb3b00b43c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:24.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.455+0000 7fb3b6cfc700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b00c6d30 con 0x7fb3b006b900 2023-10-16T12:26:24.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.455+0000 7fb3b6cfc700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b00c7010 con 0x7fb3b0160eb0 2023-10-16T12:26:24.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.455+0000 7fb3b6cfc700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b00c72f0 con 0x7fb3b00cab70 2023-10-16T12:26:24.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.455+0000 7fb3b4a98700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3b006b900 0x7fb3b0155800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:24.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.455+0000 7fb3b4a98700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3b006b900 0x7fb3b0155800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39320/0 (socket says 172.21.15.16:39320) 2023-10-16T12:26:24.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.455+0000 7fb3b4a98700 1 -- 172.21.15.16:0/141409399 learned_addr learned my addr 172.21.15.16:0/141409399 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:24.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.455+0000 7fb3b4a98700 1 -- 172.21.15.16:0/141409399 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb3b00cab70 msgr2=0x7fb3b0157080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:24.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.455+0000 7fb3b4a98700 1 --2- 172.21.15.16:0/141409399 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb3b00cab70 0x7fb3b0157080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:24.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.455+0000 7fb3b4a98700 1 -- 172.21.15.16:0/141409399 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb3b0160eb0 msgr2=0x7fb3b00b43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:24.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.455+0000 7fb3b4a98700 1 --2- 172.21.15.16:0/141409399 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb3b0160eb0 0x7fb3b00b43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:24.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.455+0000 7fb3b4a98700 1 -- 172.21.15.16:0/141409399 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3a8009a70 con 0x7fb3b006b900 2023-10-16T12:26:24.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.455+0000 7fb3b4a98700 1 --2- 172.21.15.16:0/141409399 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3b006b900 0x7fb3b0155800 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fb3a400e9b0 tx=0x7fb3a400ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:24.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.455+0000 7fb3ad7fa700 1 -- 172.21.15.16:0/141409399 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3a4009710 con 0x7fb3b006b900 2023-10-16T12:26:24.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.455+0000 7fb3ad7fa700 1 -- 172.21.15.16:0/141409399 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3a400c850 con 0x7fb3b006b900 2023-10-16T12:26:24.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.455+0000 7fb3ad7fa700 1 -- 172.21.15.16:0/141409399 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3a400cbc0 con 0x7fb3b006b900 2023-10-16T12:26:24.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.455+0000 7fb3b6cfc700 1 -- 172.21.15.16:0/141409399 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3b00c75d0 con 0x7fb3b006b900 2023-10-16T12:26:24.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.455+0000 7fb3b6cfc700 1 -- 172.21.15.16:0/141409399 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3b0156020 con 0x7fb3b006b900 2023-10-16T12:26:24.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.456+0000 7fb3b6cfc700 1 -- 172.21.15.16:0/141409399 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3b004ebb0 con 0x7fb3b006b900 2023-10-16T12:26:24.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.457+0000 7fb3ad7fa700 1 -- 172.21.15.16:0/141409399 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 31) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb3a400cd20 con 0x7fb3b006b900 2023-10-16T12:26:24.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.457+0000 7fb3ad7fa700 1 --2- 172.21.15.16:0/141409399 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7fb39406b050 0x7fb39406d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:24.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.457+0000 7fb3ad7fa700 1 -- 172.21.15.16:0/141409399 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fb3a4089960 con 0x7fb3b006b900 2023-10-16T12:26:24.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.457+0000 7fb3affff700 1 --2- 172.21.15.16:0/141409399 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7fb39406b050 0x7fb39406d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:24.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.458+0000 7fb3affff700 1 --2- 172.21.15.16:0/141409399 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7fb39406b050 0x7fb39406d510 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb39c007fd0 tx=0x7fb39c007980 comp rx=0 tx=0).ready entity=mgr.6396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:24.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.462+0000 7fb3ad7fa700 1 -- 172.21.15.16:0/141409399 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb3a40547a0 con 0x7fb3b006b900 2023-10-16T12:26:24.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.665+0000 7fb3b6cfc700 1 -- 172.21.15.16:0/141409399 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb3b00c6e70 con 0x7fb3b006b900 2023-10-16T12:26:24.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.666+0000 7fb3ad7fa700 1 -- 172.21.15.16:0/141409399 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+16664 (secure 0 0 0) 0x7fb3b00c6e70 con 0x7fb3b006b900 2023-10-16T12:26:24.668 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:26:24.668 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":84,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:26:23.670284+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:26:21.893734+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2023-10-16T12:26:22.642815+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:26:24.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.668+0000 7fb3b6cfc700 1 -- 172.21.15.16:0/141409399 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7fb39406b050 msgr2=0x7fb39406d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:24.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.668+0000 7fb3b6cfc700 1 --2- 172.21.15.16:0/141409399 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7fb39406b050 0x7fb39406d510 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb39c007fd0 tx=0x7fb39c007980 comp rx=0 tx=0).stop 2023-10-16T12:26:24.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.668+0000 7fb3b6cfc700 1 -- 172.21.15.16:0/141409399 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3b006b900 msgr2=0x7fb3b0155800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:24.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.668+0000 7fb3b6cfc700 1 --2- 172.21.15.16:0/141409399 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3b006b900 0x7fb3b0155800 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fb3a400e9b0 tx=0x7fb3a400ed70 comp rx=0 tx=0).stop 2023-10-16T12:26:24.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.668+0000 7fb3b6cfc700 1 -- 172.21.15.16:0/141409399 shutdown_connections 2023-10-16T12:26:24.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.668+0000 7fb3b6cfc700 1 --2- 172.21.15.16:0/141409399 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7fb39406b050 0x7fb39406d510 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:24.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.668+0000 7fb3b6cfc700 1 --2- 172.21.15.16:0/141409399 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb3b0160eb0 0x7fb3b00b43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:24.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.668+0000 7fb3b6cfc700 1 --2- 172.21.15.16:0/141409399 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3b006b900 0x7fb3b0155800 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:24.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.668+0000 7fb3b6cfc700 1 --2- 172.21.15.16:0/141409399 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb3b00cab70 0x7fb3b0157080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:24.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.668+0000 7fb3b6cfc700 1 -- 172.21.15.16:0/141409399 >> 172.21.15.16:0/141409399 conn(0x7fb3b00cc5b0 msgr2=0x7fb3b015fec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:24.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.668+0000 7fb3b6cfc700 1 -- 172.21.15.16:0/141409399 shutdown_connections 2023-10-16T12:26:24.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.668+0000 7fb3b6cfc700 1 -- 172.21.15.16:0/141409399 wait complete. 2023-10-16T12:26:24.683 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:26:24.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.805+0000 7f663802b700 1 Processor -- start 2023-10-16T12:26:24.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.805+0000 7f663802b700 1 -- start start 2023-10-16T12:26:24.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.806+0000 7f663802b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f663015db30 0x7f66300cf920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:24.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.806+0000 7f663802b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6630161490 0x7f66301618c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:24.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.806+0000 7f663802b700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6630161e00 con 0x7f663015d750 2023-10-16T12:26:24.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.806+0000 7f663802b700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66300cfe60 con 0x7f663015db30 2023-10-16T12:26:24.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.806+0000 7f663802b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66300d0070 con 0x7f6630161490 2023-10-16T12:26:24.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.806+0000 7f66355c6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f663015db30 0x7f66300cf920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:24.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.806+0000 7f66355c6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f663015db30 0x7f66300cf920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39330/0 (socket says 172.21.15.16:39330) 2023-10-16T12:26:24.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.806+0000 7f66355c6700 1 -- 172.21.15.16:0/3542566703 learned_addr learned my addr 172.21.15.16:0/3542566703 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:24.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.806+0000 7f6635dc7700 1 --1- 172.21.15.16:0/3542566703 >> v1:172.21.15.16:6789/0 conn(0x7f663015d750 0x7f663006c960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:56834/0 (socket says 172.21.15.16:56834) 2023-10-16T12:26:24.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.807+0000 7f66365c8700 1 --2- 172.21.15.16:0/3542566703 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6630161490 0x7f66301618c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:24.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.807+0000 7f66355c6700 1 -- 172.21.15.16:0/3542566703 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6630161490 msgr2=0x7f66301618c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:24.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.807+0000 7f66355c6700 1 --2- 172.21.15.16:0/3542566703 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6630161490 0x7f66301618c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:24.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.807+0000 7f66355c6700 1 -- 172.21.15.16:0/3542566703 >> v1:172.21.15.16:6789/0 conn(0x7f663015d750 legacy=0x7f663006c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:24.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.807+0000 7f66355c6700 1 -- 172.21.15.16:0/3542566703 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66300d0350 con 0x7f663015db30 2023-10-16T12:26:24.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.807+0000 7f6634dc5700 1 -- 172.21.15.16:0/3542566703 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 960837366 0 0) 0x7f6630161e00 con 0x7f663015d750 2023-10-16T12:26:24.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.807+0000 7f66365c8700 1 --2- 172.21.15.16:0/3542566703 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6630161490 0x7f66301618c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:26:24.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.807+0000 7f66355c6700 1 --2- 172.21.15.16:0/3542566703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f663015db30 0x7f66300cf920 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f662c0098f0 tx=0x7f662c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80b0160f4e352f1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:24.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.807+0000 7f6634dc5700 1 -- 172.21.15.16:0/3542566703 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f662c00b8a0 con 0x7f663015db30 2023-10-16T12:26:24.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.807+0000 7f6634dc5700 1 -- 172.21.15.16:0/3542566703 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f662c00ba00 con 0x7f663015db30 2023-10-16T12:26:24.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.807+0000 7f6634dc5700 1 -- 172.21.15.16:0/3542566703 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f662c00bd90 con 0x7f663015db30 2023-10-16T12:26:24.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.808+0000 7f663802b700 1 -- 172.21.15.16:0/3542566703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f663015db30 msgr2=0x7f66300cf920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:24.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.808+0000 7f663802b700 1 --2- 172.21.15.16:0/3542566703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f663015db30 0x7f66300cf920 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f662c0098f0 tx=0x7f662c009c00 comp rx=0 tx=0).stop 2023-10-16T12:26:24.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.808+0000 7f663802b700 1 -- 172.21.15.16:0/3542566703 shutdown_connections 2023-10-16T12:26:24.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.808+0000 7f663802b700 1 --2- 172.21.15.16:0/3542566703 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6630161490 0x7f66301618c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:24.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.808+0000 7f663802b700 1 --2- 172.21.15.16:0/3542566703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f663015db30 0x7f66300cf920 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:24.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.808+0000 7f663802b700 1 -- 172.21.15.16:0/3542566703 >> 172.21.15.16:0/3542566703 conn(0x7f66300d14b0 msgr2=0x7f66300d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:24.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.808+0000 7f663802b700 1 -- 172.21.15.16:0/3542566703 shutdown_connections 2023-10-16T12:26:24.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.808+0000 7f663802b700 1 -- 172.21.15.16:0/3542566703 wait complete. 2023-10-16T12:26:24.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.809+0000 7f663802b700 1 Processor -- start 2023-10-16T12:26:24.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.809+0000 7f663802b700 1 -- start start 2023-10-16T12:26:24.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.810+0000 7f663802b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f663015d750 0x7f6630152a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:24.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.810+0000 7f663802b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f663015db30 0x7f66301f6a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:24.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.810+0000 7f663802b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6630161490 0x7f6630158190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:24.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.810+0000 7f663802b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66300b4300 con 0x7f663015d750 2023-10-16T12:26:24.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.810+0000 7f663802b700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66301586d0 con 0x7f663015db30 2023-10-16T12:26:24.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.810+0000 7f663802b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6630158950 con 0x7f6630161490 2023-10-16T12:26:24.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.810+0000 7f6635dc7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f663015d750 0x7f6630152a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:24.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.810+0000 7f6635dc7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f663015d750 0x7f6630152a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39334/0 (socket says 172.21.15.16:39334) 2023-10-16T12:26:24.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.810+0000 7f6635dc7700 1 -- 172.21.15.16:0/3663721004 learned_addr learned my addr 172.21.15.16:0/3663721004 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:24.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.811+0000 7f6635dc7700 1 -- 172.21.15.16:0/3663721004 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6630161490 msgr2=0x7f6630158190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:24.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.811+0000 7f6635dc7700 1 --2- 172.21.15.16:0/3663721004 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6630161490 0x7f6630158190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:24.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.811+0000 7f6635dc7700 1 -- 172.21.15.16:0/3663721004 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f663015db30 msgr2=0x7f66301f6a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:24.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.811+0000 7f6635dc7700 1 --2- 172.21.15.16:0/3663721004 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f663015db30 0x7f66301f6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:24.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.811+0000 7f6635dc7700 1 -- 172.21.15.16:0/3663721004 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f662c009550 con 0x7f663015d750 2023-10-16T12:26:24.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.811+0000 7f6635dc7700 1 --2- 172.21.15.16:0/3663721004 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f663015d750 0x7f6630152a60 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f662000c4a0 tx=0x7f662000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:24.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.811+0000 7f66267fc700 1 -- 172.21.15.16:0/3663721004 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66200074e0 con 0x7f663015d750 2023-10-16T12:26:24.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.811+0000 7f66267fc700 1 -- 172.21.15.16:0/3663721004 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6620007640 con 0x7f663015d750 2023-10-16T12:26:24.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.811+0000 7f663802b700 1 -- 172.21.15.16:0/3663721004 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6630158c30 con 0x7f663015d750 2023-10-16T12:26:24.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.811+0000 7f66267fc700 1 -- 172.21.15.16:0/3663721004 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6620005a50 con 0x7f663015d750 2023-10-16T12:26:24.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.811+0000 7f663802b700 1 -- 172.21.15.16:0/3663721004 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66301591f0 con 0x7f663015d750 2023-10-16T12:26:24.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.812+0000 7f663802b700 1 -- 172.21.15.16:0/3663721004 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f663004ebb0 con 0x7f663015d750 2023-10-16T12:26:24.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.813+0000 7f66267fc700 1 -- 172.21.15.16:0/3663721004 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 31) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6620005bb0 con 0x7f663015d750 2023-10-16T12:26:24.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.813+0000 7f66267fc700 1 --2- 172.21.15.16:0/3663721004 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f661406b020 0x7f661406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:24.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.813+0000 7f66267fc700 1 -- 172.21.15.16:0/3663721004 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f6620088e20 con 0x7f663015d750 2023-10-16T12:26:24.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.813+0000 7f66355c6700 1 --2- 172.21.15.16:0/3663721004 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f661406b020 0x7f661406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:24.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.814+0000 7f66355c6700 1 --2- 172.21.15.16:0/3663721004 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f661406b020 0x7f661406d4e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f663015df60 tx=0x7f662c010000 comp rx=0 tx=0).ready entity=mgr.6396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:24.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:24.818+0000 7f66267fc700 1 -- 172.21.15.16:0/3663721004 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6620053c60 con 0x7f663015d750 2023-10-16T12:26:25.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:25.024+0000 7f663802b700 1 -- 172.21.15.16:0/3663721004 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f66300d0600 con 0x7f663015d750 2023-10-16T12:26:25.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:25.025+0000 7f66267fc700 1 -- 172.21.15.16:0/3663721004 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+16664 (secure 0 0 0) 0x7f6620017020 con 0x7f663015d750 2023-10-16T12:26:25.027 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:26:25.028 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":84,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:26:23.670284+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:26:21.893734+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2023-10-16T12:26:22.642815+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:26:25.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:25.027+0000 7f663802b700 1 -- 172.21.15.16:0/3663721004 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f661406b020 msgr2=0x7f661406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:25.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:25.027+0000 7f663802b700 1 --2- 172.21.15.16:0/3663721004 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f661406b020 0x7f661406d4e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f663015df60 tx=0x7f662c010000 comp rx=0 tx=0).stop 2023-10-16T12:26:25.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:25.027+0000 7f663802b700 1 -- 172.21.15.16:0/3663721004 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f663015d750 msgr2=0x7f6630152a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:25.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:25.027+0000 7f663802b700 1 --2- 172.21.15.16:0/3663721004 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f663015d750 0x7f6630152a60 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f662000c4a0 tx=0x7f662000c860 comp rx=0 tx=0).stop 2023-10-16T12:26:25.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:25.027+0000 7f663802b700 1 -- 172.21.15.16:0/3663721004 shutdown_connections 2023-10-16T12:26:25.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:25.027+0000 7f663802b700 1 --2- 172.21.15.16:0/3663721004 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f661406b020 0x7f661406d4e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:25.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:25.027+0000 7f663802b700 1 --2- 172.21.15.16:0/3663721004 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6630161490 0x7f6630158190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:25.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:25.027+0000 7f663802b700 1 --2- 172.21.15.16:0/3663721004 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f663015d750 0x7f6630152a60 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:25.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:25.027+0000 7f663802b700 1 --2- 172.21.15.16:0/3663721004 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f663015db30 0x7f66301f6a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:25.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:25.027+0000 7f663802b700 1 -- 172.21.15.16:0/3663721004 >> 172.21.15.16:0/3663721004 conn(0x7f66300d14b0 msgr2=0x7f663015df90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:25.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:25.027+0000 7f663802b700 1 -- 172.21.15.16:0/3663721004 shutdown_connections 2023-10-16T12:26:25.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:25.027+0000 7f663802b700 1 -- 172.21.15.16:0/3663721004 wait complete. 2023-10-16T12:26:25.042 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-16T12:26:25.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:25.167+0000 7fdfb9e93700 1 Processor -- start 2023-10-16T12:26:25.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:25.167+0000 7fdfb9e93700 1 -- start start 2023-10-16T12:26:25.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:25.168+0000 7fdfb9e93700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdfb40cb880 0x7fdfb40cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:25.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:25.168+0000 7fdfb9e93700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fdfb406da60 0x7fdfb406def0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:25.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:25.168+0000 7fdfb9e93700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfb40cef70 con 0x7fdfb415b070 2023-10-16T12:26:25.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:25.168+0000 7fdfb9e93700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfb406e430 con 0x7fdfb406da60 2023-10-16T12:26:25.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:25.168+0000 7fdfb9e93700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb40ce360 con 0x7fdfb40cb880 2023-10-16T12:26:25.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:25.169+0000 7fdfb2ffd700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdfb40cb880 0x7fdfb40cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:25.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:25.169+0000 7fdfb3fff700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fdfb406da60 0x7fdfb406def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:25.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:25.169+0000 7fdfb37fe700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7fdfb415b070 0x7fdfb415b450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:35798/0 (socket says 172.21.15.16:35798) 2023-10-16T12:26:25.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:25.169+0000 7fdfb37fe700 1 -- 172.21.15.16:0/3818391818 learned_addr learned my addr 172.21.15.16:0/3818391818 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:26:25.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:25.169+0000 7fdfb2ffd700 1 -- 172.21.15.16:0/3818391818 >> v2:172.21.15.49:3300/0 conn(0x7fdfb406da60 msgr2=0x7fdfb406def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:25.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:25.169+0000 7fdfb2ffd700 1 --2- 172.21.15.16:0/3818391818 >> v2:172.21.15.49:3300/0 conn(0x7fdfb406da60 0x7fdfb406def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:25.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:25.169+0000 7fdfb2ffd700 1 -- 172.21.15.16:0/3818391818 >> v1:172.21.15.49:6789/0 conn(0x7fdfb415b070 legacy=0x7fdfb415b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:25.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:25.169+0000 7fdfb2ffd700 1 -- 172.21.15.16:0/3818391818 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfb40ce640 con 0x7fdfb40cb880 2023-10-16T12:26:25.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:25.169+0000 7fdfb3fff700 1 --2- 172.21.15.16:0/3818391818 >> v2:172.21.15.49:3300/0 conn(0x7fdfb406da60 0x7fdfb406def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:26:25.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:25.169+0000 7fdfb2ffd700 1 --2- 172.21.15.16:0/3818391818 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdfb40cb880 0x7fdfb40cbd00 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c009840 tx=0x7fdf9c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=872032c0094941a6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:25.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:25.169+0000 7fdfb27fc700 1 -- 172.21.15.16:0/3818391818 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf9c011830 con 0x7fdfb40cb880 2023-10-16T12:26:25.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:25.170+0000 7fdfb27fc700 1 -- 172.21.15.16:0/3818391818 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf9c011990 con 0x7fdfb40cb880 2023-10-16T12:26:25.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:25.170+0000 7fdfb27fc700 1 -- 172.21.15.16:0/3818391818 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf9c011d20 con 0x7fdfb40cb880 2023-10-16T12:26:25.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:25.170+0000 7fdfb9e93700 1 -- 172.21.15.16:0/3818391818 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdfb40cb880 msgr2=0x7fdfb40cbd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:25.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:25.170+0000 7fdfb9e93700 1 --2- 172.21.15.16:0/3818391818 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdfb40cb880 0x7fdfb40cbd00 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c009840 tx=0x7fdf9c009b50 comp rx=0 tx=0).stop 2023-10-16T12:26:25.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:25.170+0000 7fdfb9e93700 1 -- 172.21.15.16:0/3818391818 shutdown_connections 2023-10-16T12:26:25.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:25.170+0000 7fdfb9e93700 1 --2- 172.21.15.16:0/3818391818 >> v2:172.21.15.49:3300/0 conn(0x7fdfb406da60 0x7fdfb406def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:25.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:25.170+0000 7fdfb9e93700 1 --2- 172.21.15.16:0/3818391818 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdfb40cb880 0x7fdfb40cbd00 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:25.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:25.170+0000 7fdfb9e93700 1 -- 172.21.15.16:0/3818391818 >> 172.21.15.16:0/3818391818 conn(0x7fdfb40b1f40 msgr2=0x7fdfb40b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:25.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:25.170+0000 7fdfb9e93700 1 -- 172.21.15.16:0/3818391818 shutdown_connections 2023-10-16T12:26:25.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:25.170+0000 7fdfb9e93700 1 -- 172.21.15.16:0/3818391818 wait complete. 2023-10-16T12:26:25.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:25.172+0000 7fdfb9e93700 1 Processor -- start 2023-10-16T12:26:25.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:25.172+0000 7fdfb9e93700 1 -- start start 2023-10-16T12:26:25.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:25.173+0000 7fdfb9e93700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdfb406da60 0x7fdfb41f1a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:25.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:25.173+0000 7fdfb9e93700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdfb40cb880 0x7fdfb40c9940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:25.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:25.173+0000 7fdfb9e93700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfb415b070 0x7fdfb41534b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:25.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:25.173+0000 7fdfb9e93700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb41f5300 con 0x7fdfb415b070 2023-10-16T12:26:25.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:25.173+0000 7fdfb9e93700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb41f1f80 con 0x7fdfb40cb880 2023-10-16T12:26:25.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:25.173+0000 7fdfb9e93700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb41554e0 con 0x7fdfb406da60 2023-10-16T12:26:25.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:25.173+0000 7fdfb3fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfb415b070 0x7fdfb41534b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:25.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:25.173+0000 7fdfb3fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfb415b070 0x7fdfb41534b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39352/0 (socket says 172.21.15.16:39352) 2023-10-16T12:26:25.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:25.173+0000 7fdfb3fff700 1 -- 172.21.15.16:0/1363198437 learned_addr learned my addr 172.21.15.16:0/1363198437 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:25.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:25.173+0000 7fdfb37fe700 1 --2- 172.21.15.16:0/1363198437 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdfb406da60 0x7fdfb41f1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:25.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:25.173+0000 7fdfb2ffd700 1 --2- 172.21.15.16:0/1363198437 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdfb40cb880 0x7fdfb40c9940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:25.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:25.173+0000 7fdfb3fff700 1 -- 172.21.15.16:0/1363198437 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdfb406da60 msgr2=0x7fdfb41f1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:25.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:25.173+0000 7fdfb3fff700 1 --2- 172.21.15.16:0/1363198437 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdfb406da60 0x7fdfb41f1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:25.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:25.173+0000 7fdfb3fff700 1 -- 172.21.15.16:0/1363198437 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdfb40cb880 msgr2=0x7fdfb40c9940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:25.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:25.173+0000 7fdfb3fff700 1 --2- 172.21.15.16:0/1363198437 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdfb40cb880 0x7fdfb40c9940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:25.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:25.173+0000 7fdfb3fff700 1 -- 172.21.15.16:0/1363198437 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfac009a70 con 0x7fdfb415b070 2023-10-16T12:26:25.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:25.174+0000 7fdfb3fff700 1 --2- 172.21.15.16:0/1363198437 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfb415b070 0x7fdfb41534b0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fdfac00efc0 tx=0x7fdfac00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:25.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:25.174+0000 7fdfabfff700 1 -- 172.21.15.16:0/1363198437 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfac00ca90 con 0x7fdfb415b070 2023-10-16T12:26:25.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:25.174+0000 7fdfabfff700 1 -- 172.21.15.16:0/1363198437 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfac00f040 con 0x7fdfb415b070 2023-10-16T12:26:25.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:25.174+0000 7fdfb9e93700 1 -- 172.21.15.16:0/1363198437 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf9c0094a0 con 0x7fdfb415b070 2023-10-16T12:26:25.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:25.174+0000 7fdfabfff700 1 -- 172.21.15.16:0/1363198437 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfac00ce10 con 0x7fdfb415b070 2023-10-16T12:26:25.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:25.174+0000 7fdfb9e93700 1 -- 172.21.15.16:0/1363198437 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfb41f4960 con 0x7fdfb415b070 2023-10-16T12:26:25.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:25.175+0000 7fdfb9e93700 1 -- 172.21.15.16:0/1363198437 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfb41f5440 con 0x7fdfb415b070 2023-10-16T12:26:25.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:25.175+0000 7fdfabfff700 1 -- 172.21.15.16:0/1363198437 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 31) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fdfac014480 con 0x7fdfb415b070 2023-10-16T12:26:25.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:25.175+0000 7fdfabfff700 1 --2- 172.21.15.16:0/1363198437 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7fdf9406b020 0x7fdf9406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:25.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:25.176+0000 7fdfabfff700 1 -- 172.21.15.16:0/1363198437 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fdfac089f80 con 0x7fdfb415b070 2023-10-16T12:26:25.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:25.176+0000 7fdfb37fe700 1 --2- 172.21.15.16:0/1363198437 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7fdf9406b020 0x7fdf9406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:25.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:25.176+0000 7fdfb37fe700 1 --2- 172.21.15.16:0/1363198437 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7fdf9406b020 0x7fdf9406d4e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fdfb41eeb50 tx=0x7fdfa4007f10 comp rx=0 tx=0).ready entity=mgr.6396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:25.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:25.180+0000 7fdfabfff700 1 -- 172.21.15.16:0/1363198437 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdfac0bcfe0 con 0x7fdfb415b070 2023-10-16T12:26:25.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:25.438+0000 7fdfb9e93700 1 -- 172.21.15.16:0/1363198437 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fdfb404ebb0 con 0x7fdfb415b070 2023-10-16T12:26:25.948 INFO:tasks.ceph.mds.a.smithi016.stderr:2023-10-16T12:26:25.946+0000 7f4c06141700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T12:26:26.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:26.288+0000 7fdfabfff700 1 -- 172.21.15.16:0/1363198437 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v35) v1 ==== 118+0+0 (secure 0 0 0) 0x7fdfac054d40 con 0x7fdfb415b070 2023-10-16T12:26:26.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:26.291+0000 7fdfb9e93700 1 -- 172.21.15.16:0/1363198437 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7fdf9406b020 msgr2=0x7fdf9406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:26.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:26.291+0000 7fdfb9e93700 1 --2- 172.21.15.16:0/1363198437 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7fdf9406b020 0x7fdf9406d4e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fdfb41eeb50 tx=0x7fdfa4007f10 comp rx=0 tx=0).stop 2023-10-16T12:26:26.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:26.292+0000 7fdfb9e93700 1 -- 172.21.15.16:0/1363198437 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfb415b070 msgr2=0x7fdfb41534b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:26.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:26.292+0000 7fdfb9e93700 1 --2- 172.21.15.16:0/1363198437 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfb415b070 0x7fdfb41534b0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fdfac00efc0 tx=0x7fdfac00c3a0 comp rx=0 tx=0).stop 2023-10-16T12:26:26.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:26.292+0000 7fdfb9e93700 1 -- 172.21.15.16:0/1363198437 shutdown_connections 2023-10-16T12:26:26.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:26.292+0000 7fdfb9e93700 1 --2- 172.21.15.16:0/1363198437 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7fdf9406b020 0x7fdf9406d4e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:26.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:26.292+0000 7fdfb9e93700 1 --2- 172.21.15.16:0/1363198437 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfb415b070 0x7fdfb41534b0 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:26.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:26.292+0000 7fdfb9e93700 1 --2- 172.21.15.16:0/1363198437 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdfb406da60 0x7fdfb41f1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:26.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:26.292+0000 7fdfb9e93700 1 --2- 172.21.15.16:0/1363198437 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdfb40cb880 0x7fdfb40c9940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:26.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:26.292+0000 7fdfb9e93700 1 -- 172.21.15.16:0/1363198437 >> 172.21.15.16:0/1363198437 conn(0x7fdfb40b1f40 msgr2=0x7fdfb415d460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:26.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:26.292+0000 7fdfb9e93700 1 -- 172.21.15.16:0/1363198437 shutdown_connections 2023-10-16T12:26:26.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:26.292+0000 7fdfb9e93700 1 -- 172.21.15.16:0/1363198437 wait complete. 2023-10-16T12:26:26.310 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-16T12:26:26.310 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-16T12:26:26.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:26.442+0000 7f605bca4700 1 Processor -- start 2023-10-16T12:26:26.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:26.442+0000 7f605bca4700 1 -- start start 2023-10-16T12:26:26.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:26.443+0000 7f605bca4700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f605415d8d0 0x7f60540b2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:26.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:26.443+0000 7f605bca4700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f60540ce350 0x7f60540ce780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:26.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:26.443+0000 7f605bca4700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60540cecc0 con 0x7f60540b8fa0 2023-10-16T12:26:26.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:26.443+0000 7f605bca4700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60540d0bb0 con 0x7f605415d8d0 2023-10-16T12:26:26.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:26.443+0000 7f605bca4700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f605415dcb0 con 0x7f60540ce350 2023-10-16T12:26:26.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:26.443+0000 7f6059a40700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f605415d8d0 0x7f60540b2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:26.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:26.443+0000 7f6059a40700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f605415d8d0 0x7f60540b2830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39356/0 (socket says 172.21.15.16:39356) 2023-10-16T12:26:26.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:26.443+0000 7f6059a40700 1 -- 172.21.15.16:0/710583786 learned_addr learned my addr 172.21.15.16:0/710583786 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:26.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:26.443+0000 7f605a241700 1 --2- 172.21.15.16:0/710583786 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f60540ce350 0x7f60540ce780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:26.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:26.443+0000 7f6059a40700 1 -- 172.21.15.16:0/710583786 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f60540ce350 msgr2=0x7f60540ce780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:26.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:26.443+0000 7f6059a40700 1 --2- 172.21.15.16:0/710583786 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f60540ce350 0x7f60540ce780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:26.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:26.443+0000 7f6059a40700 1 -- 172.21.15.16:0/710583786 >> v1:172.21.15.16:6789/0 conn(0x7f60540b8fa0 legacy=0x7f60540b6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:26.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:26.443+0000 7f6059a40700 1 -- 172.21.15.16:0/710583786 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60540b2f30 con 0x7f605415d8d0 2023-10-16T12:26:26.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:26.444+0000 7f6059a40700 1 --2- 172.21.15.16:0/710583786 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f605415d8d0 0x7f60540b2830 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f60440098f0 tx=0x7f6044009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f53f27d635ba8bf server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:26.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:26.444+0000 7f6058a3e700 1 -- 172.21.15.16:0/710583786 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f604400b8a0 con 0x7f605415d8d0 2023-10-16T12:26:26.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:26.444+0000 7f6058a3e700 1 -- 172.21.15.16:0/710583786 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f604400ba00 con 0x7f605415d8d0 2023-10-16T12:26:26.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:26.444+0000 7f6058a3e700 1 -- 172.21.15.16:0/710583786 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f604400bd90 con 0x7f605415d8d0 2023-10-16T12:26:26.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:26.444+0000 7f605bca4700 1 -- 172.21.15.16:0/710583786 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f605415d8d0 msgr2=0x7f60540b2830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:26.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:26.444+0000 7f605bca4700 1 --2- 172.21.15.16:0/710583786 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f605415d8d0 0x7f60540b2830 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f60440098f0 tx=0x7f6044009c00 comp rx=0 tx=0).stop 2023-10-16T12:26:26.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:26.444+0000 7f605bca4700 1 -- 172.21.15.16:0/710583786 shutdown_connections 2023-10-16T12:26:26.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:26.444+0000 7f605bca4700 1 --2- 172.21.15.16:0/710583786 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f60540ce350 0x7f60540ce780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:26.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:26.444+0000 7f605bca4700 1 --2- 172.21.15.16:0/710583786 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f605415d8d0 0x7f60540b2830 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:26.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:26.444+0000 7f605bca4700 1 -- 172.21.15.16:0/710583786 >> 172.21.15.16:0/710583786 conn(0x7f6054072330 msgr2=0x7f6054072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:26.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:26.444+0000 7f605bca4700 1 -- 172.21.15.16:0/710583786 shutdown_connections 2023-10-16T12:26:26.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:26.444+0000 7f605bca4700 1 -- 172.21.15.16:0/710583786 wait complete. 2023-10-16T12:26:26.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:26.446+0000 7f605bca4700 1 Processor -- start 2023-10-16T12:26:26.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:26.446+0000 7f605bca4700 1 -- start start 2023-10-16T12:26:26.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:26.446+0000 7f605bca4700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f60540b8fa0 0x7f605414f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:26.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:26.446+0000 7f605bca4700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f60540ce350 0x7f605414eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:26.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:26.446+0000 7f605bca4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f605415d8d0 0x7f60540cbe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:26.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:26.446+0000 7f605bca4700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60540cb830 con 0x7f605415d8d0 2023-10-16T12:26:26.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:26.446+0000 7f605bca4700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f605414f050 con 0x7f60540b8fa0 2023-10-16T12:26:26.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:26.446+0000 7f605bca4700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60540cb550 con 0x7f60540ce350 2023-10-16T12:26:26.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:26.446+0000 7f605a241700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f605415d8d0 0x7f60540cbe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:26.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:26.446+0000 7f605a241700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f605415d8d0 0x7f60540cbe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39364/0 (socket says 172.21.15.16:39364) 2023-10-16T12:26:26.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:26.446+0000 7f605a241700 1 -- 172.21.15.16:0/2769024856 learned_addr learned my addr 172.21.15.16:0/2769024856 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:26.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:26.447+0000 7f6059a40700 1 --2- 172.21.15.16:0/2769024856 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f60540b8fa0 0x7f605414f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:26.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:26.447+0000 7f605923f700 1 --2- 172.21.15.16:0/2769024856 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f60540ce350 0x7f605414eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:26.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:26.447+0000 7f605a241700 1 -- 172.21.15.16:0/2769024856 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f60540ce350 msgr2=0x7f605414eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:26.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:26.447+0000 7f605a241700 1 --2- 172.21.15.16:0/2769024856 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f60540ce350 0x7f605414eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:26.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:26.447+0000 7f605a241700 1 -- 172.21.15.16:0/2769024856 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f60540b8fa0 msgr2=0x7f605414f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:26.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:26.447+0000 7f605a241700 1 --2- 172.21.15.16:0/2769024856 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f60540b8fa0 0x7f605414f330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:26.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:26.447+0000 7f605a241700 1 -- 172.21.15.16:0/2769024856 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6044009550 con 0x7f605415d8d0 2023-10-16T12:26:26.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:26.447+0000 7f605a241700 1 --2- 172.21.15.16:0/2769024856 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f605415d8d0 0x7f60540cbe70 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f604c007cf0 tx=0x7f604c00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:26.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:26.447+0000 7f604a7fc700 1 -- 172.21.15.16:0/2769024856 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f604c00ede0 con 0x7f605415d8d0 2023-10-16T12:26:26.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:26.447+0000 7f604a7fc700 1 -- 172.21.15.16:0/2769024856 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f604c014420 con 0x7f605415d8d0 2023-10-16T12:26:26.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:26.447+0000 7f605bca4700 1 -- 172.21.15.16:0/2769024856 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60540c71e0 con 0x7f605415d8d0 2023-10-16T12:26:26.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:26.447+0000 7f604a7fc700 1 -- 172.21.15.16:0/2769024856 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f604c0147d0 con 0x7f605415d8d0 2023-10-16T12:26:26.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:26.447+0000 7f605bca4700 1 -- 172.21.15.16:0/2769024856 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60540c77a0 con 0x7f605415d8d0 2023-10-16T12:26:26.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:26.448+0000 7f605bca4700 1 -- 172.21.15.16:0/2769024856 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60540c7040 con 0x7f605415d8d0 2023-10-16T12:26:26.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:26.449+0000 7f604a7fc700 1 -- 172.21.15.16:0/2769024856 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 31) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f604c014930 con 0x7f605415d8d0 2023-10-16T12:26:26.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:26.450+0000 7f604a7fc700 1 --2- 172.21.15.16:0/2769024856 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f603806b020 0x7f603806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:26.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:26.450+0000 7f604a7fc700 1 -- 172.21.15.16:0/2769024856 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f604c08ac50 con 0x7f605415d8d0 2023-10-16T12:26:26.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:26.450+0000 7f6059a40700 1 --2- 172.21.15.16:0/2769024856 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f603806b020 0x7f603806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:26.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:26.450+0000 7f6059a40700 1 --2- 172.21.15.16:0/2769024856 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f603806b020 0x7f603806d4e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6044014040 tx=0x7f6044008030 comp rx=0 tx=0).ready entity=mgr.6396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:26.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:26.453+0000 7f604a7fc700 1 -- 172.21.15.16:0/2769024856 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f604c055b00 con 0x7f605415d8d0 2023-10-16T12:26:26.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:26.707+0000 7f605bca4700 1 -- 172.21.15.16:0/2769024856 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f60540c7040 con 0x7f605415d8d0 2023-10-16T12:26:27.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.293+0000 7f604a7fc700 1 -- 172.21.15.16:0/2769024856 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v36) v1 ==== 122+0+0 (secure 0 0 0) 0x7f60540c7040 con 0x7f605415d8d0 2023-10-16T12:26:27.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.296+0000 7f605bca4700 1 -- 172.21.15.16:0/2769024856 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f603806b020 msgr2=0x7f603806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:27.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.296+0000 7f605bca4700 1 --2- 172.21.15.16:0/2769024856 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f603806b020 0x7f603806d4e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6044014040 tx=0x7f6044008030 comp rx=0 tx=0).stop 2023-10-16T12:26:27.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.296+0000 7f605bca4700 1 -- 172.21.15.16:0/2769024856 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f605415d8d0 msgr2=0x7f60540cbe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:27.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.296+0000 7f605bca4700 1 --2- 172.21.15.16:0/2769024856 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f605415d8d0 0x7f60540cbe70 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f604c007cf0 tx=0x7f604c00e3c0 comp rx=0 tx=0).stop 2023-10-16T12:26:27.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.296+0000 7f605bca4700 1 -- 172.21.15.16:0/2769024856 shutdown_connections 2023-10-16T12:26:27.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.296+0000 7f605bca4700 1 --2- 172.21.15.16:0/2769024856 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f603806b020 0x7f603806d4e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:27.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.296+0000 7f605bca4700 1 --2- 172.21.15.16:0/2769024856 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f605415d8d0 0x7f60540cbe70 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:27.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.296+0000 7f605bca4700 1 --2- 172.21.15.16:0/2769024856 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f60540b8fa0 0x7f605414f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:27.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.296+0000 7f605bca4700 1 --2- 172.21.15.16:0/2769024856 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f60540ce350 0x7f605414eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:27.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.296+0000 7f605bca4700 1 -- 172.21.15.16:0/2769024856 >> 172.21.15.16:0/2769024856 conn(0x7f6054072330 msgr2=0x7f60540d2860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:27.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.296+0000 7f605bca4700 1 -- 172.21.15.16:0/2769024856 shutdown_connections 2023-10-16T12:26:27.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.296+0000 7f605bca4700 1 -- 172.21.15.16:0/2769024856 wait complete. 2023-10-16T12:26:27.314 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-16T12:26:27.315 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:26:27.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.444+0000 7ff6883f4700 1 Processor -- start 2023-10-16T12:26:27.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.444+0000 7ff6883f4700 1 -- start start 2023-10-16T12:26:27.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.445+0000 7ff6883f4700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7ff68015b630 0x7ff6800cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:27.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.445+0000 7ff6883f4700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff68015ba10 0x7ff6800cdde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:27.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.445+0000 7ff6883f4700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6800ce770 con 0x7ff6800ce320 2023-10-16T12:26:27.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.445+0000 7ff6883f4700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6800cdb20 con 0x7ff68015b630 2023-10-16T12:26:27.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.445+0000 7ff6883f4700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6800cc2a0 con 0x7ff68015ba10 2023-10-16T12:26:27.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.445+0000 7ff686991700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff6800ce320 0x7ff6800cfca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:56878/0 (socket says 172.21.15.16:56878) 2023-10-16T12:26:27.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.445+0000 7ff686991700 1 -- 172.21.15.16:0/1522622410 learned_addr learned my addr 172.21.15.16:0/1522622410 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:26:27.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.445+0000 7ff68598f700 1 --2- 172.21.15.16:0/1522622410 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff68015ba10 0x7ff6800cdde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:27.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.445+0000 7ff686190700 1 --2- 172.21.15.16:0/1522622410 >> v2:172.21.15.49:3300/0 conn(0x7ff68015b630 0x7ff6800cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:27.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.445+0000 7ff68518e700 1 -- 172.21.15.16:0/1522622410 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2038911713 0 0) 0x7ff678001490 con 0x7ff6800ce320 2023-10-16T12:26:27.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.445+0000 7ff68518e700 1 -- 172.21.15.16:0/1522622410 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff66c0035b0 con 0x7ff6800ce320 2023-10-16T12:26:27.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.445+0000 7ff68518e700 1 -- 172.21.15.16:0/1522622410 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 8334533 0 0) 0x7ff66c0035b0 con 0x7ff6800ce320 2023-10-16T12:26:27.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.445+0000 7ff68518e700 1 -- 172.21.15.16:0/1522622410 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff68015ba10 msgr2=0x7ff6800cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:27.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.445+0000 7ff68518e700 1 --2- 172.21.15.16:0/1522622410 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff68015ba10 0x7ff6800cdde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:27.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.445+0000 7ff68518e700 1 -- 172.21.15.16:0/1522622410 >> v2:172.21.15.49:3300/0 conn(0x7ff68015b630 msgr2=0x7ff6800cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:27.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.445+0000 7ff68518e700 1 --2- 172.21.15.16:0/1522622410 >> v2:172.21.15.49:3300/0 conn(0x7ff68015b630 0x7ff6800cd5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:27.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.445+0000 7ff68518e700 1 -- 172.21.15.16:0/1522622410 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6800cc540 con 0x7ff6800ce320 2023-10-16T12:26:27.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.445+0000 7ff68518e700 1 -- 172.21.15.16:0/1522622410 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7ff678001b80 con 0x7ff6800ce320 2023-10-16T12:26:27.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.445+0000 7ff68518e700 1 -- 172.21.15.16:0/1522622410 >> v1:172.21.15.16:6789/0 conn(0x7ff6800ce320 legacy=0x7ff6800cfca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:27.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.445+0000 7ff68518e700 1 --2- 172.21.15.16:0/1522622410 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff68015ba10 0x7ff66c003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:27.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.445+0000 7ff68518e700 1 --2- 172.21.15.16:0/1522622410 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff66c003f80 0x7ff66c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:27.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.445+0000 7ff68518e700 1 --2- 172.21.15.16:0/1522622410 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff66c004d90 0x7ff66c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:27.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.445+0000 7ff68518e700 1 -- 172.21.15.16:0/1522622410 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6800cc2a0 con 0x7ff66c003f80 2023-10-16T12:26:27.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.445+0000 7ff68518e700 1 -- 172.21.15.16:0/1522622410 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff66c00ba70 con 0x7ff66c004d90 2023-10-16T12:26:27.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.445+0000 7ff68518e700 1 -- 172.21.15.16:0/1522622410 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff66c00bc30 con 0x7ff68015ba10 2023-10-16T12:26:27.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.446+0000 7ff68598f700 1 --2- 172.21.15.16:0/1522622410 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff66c003f80 0x7ff66c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:27.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.446+0000 7ff686190700 1 --2- 172.21.15.16:0/1522622410 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff68015ba10 0x7ff66c003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:27.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.446+0000 7ff686991700 1 --2- 172.21.15.16:0/1522622410 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff66c004d90 0x7ff66c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:27.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.446+0000 7ff68598f700 1 -- 172.21.15.16:0/1522622410 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff68015ba10 msgr2=0x7ff66c003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:27.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.446+0000 7ff68598f700 1 --2- 172.21.15.16:0/1522622410 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff68015ba10 0x7ff66c003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:27.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.446+0000 7ff68598f700 1 -- 172.21.15.16:0/1522622410 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff66c004d90 msgr2=0x7ff66c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:27.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.446+0000 7ff68598f700 1 --2- 172.21.15.16:0/1522622410 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff66c004d90 0x7ff66c005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:27.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.446+0000 7ff68598f700 1 -- 172.21.15.16:0/1522622410 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff66c0035b0 con 0x7ff66c003f80 2023-10-16T12:26:27.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.446+0000 7ff686991700 1 --2- 172.21.15.16:0/1522622410 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff66c004d90 0x7ff66c005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:26:27.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.446+0000 7ff68598f700 1 --2- 172.21.15.16:0/1522622410 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff66c003f80 0x7ff66c004850 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7ff670009b60 tx=0x7ff66c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=896f4a9c2d5bbd9c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:27.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.446+0000 7ff68518e700 1 -- 172.21.15.16:0/1522622410 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff67000e6c0 con 0x7ff66c003f80 2023-10-16T12:26:27.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.446+0000 7ff68518e700 1 -- 172.21.15.16:0/1522622410 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff67000e820 con 0x7ff66c003f80 2023-10-16T12:26:27.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.446+0000 7ff68518e700 1 -- 172.21.15.16:0/1522622410 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff67000eb90 con 0x7ff66c003f80 2023-10-16T12:26:27.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.447+0000 7ff6883f4700 1 -- 172.21.15.16:0/1522622410 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff66c003f80 msgr2=0x7ff66c004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:27.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.447+0000 7ff6883f4700 1 --2- 172.21.15.16:0/1522622410 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff66c003f80 0x7ff66c004850 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7ff670009b60 tx=0x7ff66c005880 comp rx=0 tx=0).stop 2023-10-16T12:26:27.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.447+0000 7ff686991700 1 -- 172.21.15.16:0/1522622410 reap_dead start 2023-10-16T12:26:27.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.447+0000 7ff6883f4700 1 -- 172.21.15.16:0/1522622410 shutdown_connections 2023-10-16T12:26:27.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.447+0000 7ff6883f4700 1 -- 172.21.15.16:0/1522622410 >> 172.21.15.16:0/1522622410 conn(0x7ff6800d14b0 msgr2=0x7ff6800d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:27.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.447+0000 7ff6883f4700 1 -- 172.21.15.16:0/1522622410 shutdown_connections 2023-10-16T12:26:27.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.447+0000 7ff6883f4700 1 -- 172.21.15.16:0/1522622410 wait complete. 2023-10-16T12:26:27.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.448+0000 7ff6883f4700 1 Processor -- start 2023-10-16T12:26:27.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.448+0000 7ff6883f4700 1 -- start start 2023-10-16T12:26:27.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.449+0000 7ff6883f4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff66c003f80 0x7ff6801f6790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:27.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.449+0000 7ff6883f4700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff68006d600 0x7ff6801f6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:27.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.449+0000 7ff6883f4700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff68006c1c0 0x7ff68006c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:27.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.449+0000 7ff6883f4700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6801f42b0 con 0x7ff66c003f80 2023-10-16T12:26:27.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.449+0000 7ff6883f4700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff68015cf50 con 0x7ff68006c1c0 2023-10-16T12:26:27.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.449+0000 7ff6883f4700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6800c6430 con 0x7ff68006d600 2023-10-16T12:26:27.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.449+0000 7ff686190700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff66c003f80 0x7ff6801f6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:27.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.449+0000 7ff686190700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff66c003f80 0x7ff6801f6790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39380/0 (socket says 172.21.15.16:39380) 2023-10-16T12:26:27.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.449+0000 7ff686190700 1 -- 172.21.15.16:0/334569482 learned_addr learned my addr 172.21.15.16:0/334569482 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:27.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.449+0000 7ff686991700 1 --2- 172.21.15.16:0/334569482 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff68006c1c0 0x7ff68006c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:27.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.449+0000 7ff68598f700 1 --2- 172.21.15.16:0/334569482 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff68006d600 0x7ff6801f6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:27.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.449+0000 7ff686190700 1 -- 172.21.15.16:0/334569482 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff68006d600 msgr2=0x7ff6801f6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:27.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.449+0000 7ff686190700 1 --2- 172.21.15.16:0/334569482 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff68006d600 0x7ff6801f6cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:27.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.449+0000 7ff686190700 1 -- 172.21.15.16:0/334569482 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff68006c1c0 msgr2=0x7ff68006c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:27.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.449+0000 7ff686190700 1 --2- 172.21.15.16:0/334569482 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff68006c1c0 0x7ff68006c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:27.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.449+0000 7ff686190700 1 -- 172.21.15.16:0/334569482 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6800c6710 con 0x7ff66c003f80 2023-10-16T12:26:27.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.449+0000 7ff686190700 1 --2- 172.21.15.16:0/334569482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff66c003f80 0x7ff6801f6790 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7ff67c011a70 tx=0x7ff67c011e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:27.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.449+0000 7ff676ffd700 1 -- 172.21.15.16:0/334569482 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff67c005a20 con 0x7ff66c003f80 2023-10-16T12:26:27.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.449+0000 7ff676ffd700 1 -- 172.21.15.16:0/334569482 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff67c005b80 con 0x7ff66c003f80 2023-10-16T12:26:27.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.449+0000 7ff6883f4700 1 -- 172.21.15.16:0/334569482 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6800c69f0 con 0x7ff66c003f80 2023-10-16T12:26:27.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.449+0000 7ff676ffd700 1 -- 172.21.15.16:0/334569482 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff67c00fa00 con 0x7ff66c003f80 2023-10-16T12:26:27.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.449+0000 7ff6883f4700 1 -- 172.21.15.16:0/334569482 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff68006cb70 con 0x7ff66c003f80 2023-10-16T12:26:27.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.450+0000 7ff6883f4700 1 -- 172.21.15.16:0/334569482 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff68004ebb0 con 0x7ff66c003f80 2023-10-16T12:26:27.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.451+0000 7ff676ffd700 1 -- 172.21.15.16:0/334569482 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 31) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff67c00fb60 con 0x7ff66c003f80 2023-10-16T12:26:27.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.451+0000 7ff676ffd700 1 --2- 172.21.15.16:0/334569482 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7ff66406b020 0x7ff66406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:27.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.451+0000 7ff676ffd700 1 -- 172.21.15.16:0/334569482 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6057+0+0 (secure 0 0 0) 0x7ff67c089dd0 con 0x7ff66c003f80 2023-10-16T12:26:27.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.451+0000 7ff68598f700 1 --2- 172.21.15.16:0/334569482 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7ff66406b020 0x7ff66406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:27.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.452+0000 7ff68598f700 1 --2- 172.21.15.16:0/334569482 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7ff66406b020 0x7ff66406d4e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff6700090b0 tx=0x7ff670008030 comp rx=0 tx=0).ready entity=mgr.6396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:27.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.456+0000 7ff676ffd700 1 -- 172.21.15.16:0/334569482 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff67c054c10 con 0x7ff66c003f80 2023-10-16T12:26:27.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.702+0000 7ff6883f4700 1 -- 172.21.15.16:0/334569482 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff68006b5d0 con 0x7ff66c003f80 2023-10-16T12:26:27.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.703+0000 7ff676ffd700 1 -- 172.21.15.16:0/334569482 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+4911 (secure 0 0 0) 0x7ff67c08b090 con 0x7ff66c003f80 2023-10-16T12:26:27.706 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:26:27.707 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":36,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5861,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/1236108916","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":1236108916},{"type":"v1","addr":"172.21.15.49:6834","nonce":1236108916}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31},{"gid":5864,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/2762335434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":2762335434},{"type":"v1","addr":"172.21.15.49:6835","nonce":2762335434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31},{"gid":6379,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/1502256436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1502256436},{"type":"v1","addr":"172.21.15.16:6836","nonce":1502256436}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:26:23.670609+0000","modified":"2023-10-16T12:26:26.710098+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6627},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6627":{"gid":6627,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/962243511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":962243511},{"type":"v1","addr":"172.21.15.16:6837","nonce":962243511}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-10-16T12:26:27.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.705+0000 7ff6883f4700 1 -- 172.21.15.16:0/334569482 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7ff66406b020 msgr2=0x7ff66406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:27.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.705+0000 7ff6883f4700 1 --2- 172.21.15.16:0/334569482 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7ff66406b020 0x7ff66406d4e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff6700090b0 tx=0x7ff670008030 comp rx=0 tx=0).stop 2023-10-16T12:26:27.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.705+0000 7ff6883f4700 1 -- 172.21.15.16:0/334569482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff66c003f80 msgr2=0x7ff6801f6790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:27.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.705+0000 7ff6883f4700 1 --2- 172.21.15.16:0/334569482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff66c003f80 0x7ff6801f6790 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7ff67c011a70 tx=0x7ff67c011e30 comp rx=0 tx=0).stop 2023-10-16T12:26:27.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.705+0000 7ff6883f4700 1 -- 172.21.15.16:0/334569482 shutdown_connections 2023-10-16T12:26:27.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.705+0000 7ff6883f4700 1 --2- 172.21.15.16:0/334569482 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7ff66406b020 0x7ff66406d4e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:27.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.705+0000 7ff6883f4700 1 --2- 172.21.15.16:0/334569482 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff68006c1c0 0x7ff68006c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:27.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.705+0000 7ff6883f4700 1 --2- 172.21.15.16:0/334569482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff66c003f80 0x7ff6801f6790 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:27.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.705+0000 7ff6883f4700 1 --2- 172.21.15.16:0/334569482 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff68006d600 0x7ff6801f6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:27.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.705+0000 7ff6883f4700 1 -- 172.21.15.16:0/334569482 >> 172.21.15.16:0/334569482 conn(0x7ff6800d14b0 msgr2=0x7ff68015aa50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:27.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.705+0000 7ff6883f4700 1 -- 172.21.15.16:0/334569482 shutdown_connections 2023-10-16T12:26:27.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.705+0000 7ff6883f4700 1 -- 172.21.15.16:0/334569482 wait complete. 2023-10-16T12:26:27.711 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 36 2023-10-16T12:26:27.720 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:26:27.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.842+0000 7fbb072df700 1 Processor -- start 2023-10-16T12:26:27.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.842+0000 7fbb072df700 1 -- start start 2023-10-16T12:26:27.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.843+0000 7fbb072df700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fbb0006d350 0x7fbb0006eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:27.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.843+0000 7fbb072df700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb0006f0a0 con 0x7fbb0015a870 2023-10-16T12:26:27.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.843+0000 7fbb072df700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb0006f380 con 0x7fbb000cbed0 2023-10-16T12:26:27.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.843+0000 7fbb072df700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb0015ac50 con 0x7fbb0006d350 2023-10-16T12:26:27.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.843+0000 7fbb0507b700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fbb0015a870 0x7fbb000cb7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:56894/0 (socket says 172.21.15.16:56894) 2023-10-16T12:26:27.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.843+0000 7fbb0507b700 1 -- 172.21.15.16:0/1209258304 learned_addr learned my addr 172.21.15.16:0/1209258304 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:26:27.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.843+0000 7fbb0487a700 1 --2- 172.21.15.16:0/1209258304 >> v2:172.21.15.49:3300/0 conn(0x7fbb0006d350 0x7fbb0006eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:27.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.843+0000 7fbaf7fff700 1 -- 172.21.15.16:0/1209258304 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 458822839 0 0) 0x7fbb0006f0a0 con 0x7fbb0015a870 2023-10-16T12:26:27.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.843+0000 7fbaf7fff700 1 -- 172.21.15.16:0/1209258304 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbaec0035b0 con 0x7fbb0015a870 2023-10-16T12:26:27.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.844+0000 7fbaf7fff700 1 -- 172.21.15.16:0/1209258304 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3361597802 0 0) 0x7fbaec0035b0 con 0x7fbb0015a870 2023-10-16T12:26:27.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.844+0000 7fbaf7fff700 1 -- 172.21.15.16:0/1209258304 >> v2:172.21.15.49:3300/0 conn(0x7fbb0006d350 msgr2=0x7fbb0006eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:27.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.844+0000 7fbaf7fff700 1 --2- 172.21.15.16:0/1209258304 >> v2:172.21.15.49:3300/0 conn(0x7fbb0006d350 0x7fbb0006eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:27.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.844+0000 7fbaf7fff700 1 -- 172.21.15.16:0/1209258304 >> v1:172.21.15.49:6789/0 conn(0x7fbb000cbed0 legacy=0x7fbb000cde10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:27.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.844+0000 7fbaf7fff700 1 -- 172.21.15.16:0/1209258304 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb0006f660 con 0x7fbb0015a870 2023-10-16T12:26:27.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.844+0000 7fbaf7fff700 1 -- 172.21.15.16:0/1209258304 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7fbaf0001670 con 0x7fbb0015a870 2023-10-16T12:26:27.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.844+0000 7fbaf7fff700 1 -- 172.21.15.16:0/1209258304 >> v1:172.21.15.16:6789/0 conn(0x7fbb0015a870 legacy=0x7fbb000cb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:27.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.844+0000 7fbaf7fff700 1 --2- 172.21.15.16:0/1209258304 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbaec003a00 0x7fbaec003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:27.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.844+0000 7fbaf7fff700 1 --2- 172.21.15.16:0/1209258304 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbaec004850 0x7fbaec004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:27.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.844+0000 7fbaf7fff700 1 --2- 172.21.15.16:0/1209258304 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbaec005210 0x7fbaec00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:27.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.844+0000 7fbaf7fff700 1 -- 172.21.15.16:0/1209258304 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb0015ac50 con 0x7fbaec005210 2023-10-16T12:26:27.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.844+0000 7fbaf7fff700 1 -- 172.21.15.16:0/1209258304 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaec00c0b0 con 0x7fbaec004850 2023-10-16T12:26:27.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.844+0000 7fbaf7fff700 1 -- 172.21.15.16:0/1209258304 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbaec00c270 con 0x7fbaec003a00 2023-10-16T12:26:27.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.844+0000 7fbb0587c700 1 --2- 172.21.15.16:0/1209258304 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbaec005210 0x7fbaec00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:27.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.844+0000 7fbb0507b700 1 --2- 172.21.15.16:0/1209258304 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbaec003a00 0x7fbaec003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:27.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.844+0000 7fbb0487a700 1 --2- 172.21.15.16:0/1209258304 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbaec004850 0x7fbaec004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:27.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.844+0000 7fbb0587c700 1 -- 172.21.15.16:0/1209258304 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbaec003a00 msgr2=0x7fbaec003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:27.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.844+0000 7fbb0587c700 1 --2- 172.21.15.16:0/1209258304 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbaec003a00 0x7fbaec003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:27.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.844+0000 7fbb0587c700 1 -- 172.21.15.16:0/1209258304 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbaec004850 msgr2=0x7fbaec004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:27.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.844+0000 7fbb0587c700 1 --2- 172.21.15.16:0/1209258304 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbaec004850 0x7fbaec004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:27.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.844+0000 7fbb0587c700 1 -- 172.21.15.16:0/1209258304 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaec0035b0 con 0x7fbaec005210 2023-10-16T12:26:27.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.844+0000 7fbb0487a700 1 --2- 172.21.15.16:0/1209258304 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbaec004850 0x7fbaec004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:26:27.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.844+0000 7fbb0507b700 1 --2- 172.21.15.16:0/1209258304 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbaec003a00 0x7fbaec003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:26:27.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.844+0000 7fbb0587c700 1 -- 172.21.15.16:0/1209258304 reap_dead start 2023-10-16T12:26:27.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.844+0000 7fbb0587c700 1 --2- 172.21.15.16:0/1209258304 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbaec005210 0x7fbaec00ba10 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fbaf800c190 tx=0x7fbaec005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5a5c845d76a7dc0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:27.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.845+0000 7fbaf7fff700 1 -- 172.21.15.16:0/1209258304 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf8001db0 con 0x7fbaec005210 2023-10-16T12:26:27.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.845+0000 7fbaf7fff700 1 -- 172.21.15.16:0/1209258304 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaf800ccf0 con 0x7fbaec005210 2023-10-16T12:26:27.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.845+0000 7fbaf7fff700 1 -- 172.21.15.16:0/1209258304 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf8003c40 con 0x7fbaec005210 2023-10-16T12:26:27.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.845+0000 7fbb072df700 1 -- 172.21.15.16:0/1209258304 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbaec005210 msgr2=0x7fbaec00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:27.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.845+0000 7fbb072df700 1 --2- 172.21.15.16:0/1209258304 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbaec005210 0x7fbaec00ba10 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fbaf800c190 tx=0x7fbaec005880 comp rx=0 tx=0).stop 2023-10-16T12:26:27.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.845+0000 7fbb072df700 1 -- 172.21.15.16:0/1209258304 shutdown_connections 2023-10-16T12:26:27.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.845+0000 7fbb072df700 1 --2- 172.21.15.16:0/1209258304 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbaec005210 0x7fbaec00ba10 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:27.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.845+0000 7fbb072df700 1 -- 172.21.15.16:0/1209258304 >> 172.21.15.16:0/1209258304 conn(0x7fbb000cc5b0 msgr2=0x7fbb000cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:27.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.845+0000 7fbb072df700 1 -- 172.21.15.16:0/1209258304 shutdown_connections 2023-10-16T12:26:27.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.845+0000 7fbb072df700 1 -- 172.21.15.16:0/1209258304 wait complete. 2023-10-16T12:26:27.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.846+0000 7fbb072df700 1 Processor -- start 2023-10-16T12:26:27.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.846+0000 7fbb072df700 1 -- start start 2023-10-16T12:26:27.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.847+0000 7fbb072df700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbaec005210 0x7fbb00155240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:27.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.847+0000 7fbb072df700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb000cb0e0 0x7fbb000cb560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:27.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.847+0000 7fbb072df700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbb0006a9b0 0x7fbb0006ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:27.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.847+0000 7fbb072df700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb001f5060 con 0x7fbb000cb0e0 2023-10-16T12:26:27.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.847+0000 7fbb072df700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb000cbaa0 con 0x7fbb0006a9b0 2023-10-16T12:26:27.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.847+0000 7fbb072df700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb001f4470 con 0x7fbaec005210 2023-10-16T12:26:27.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.847+0000 7fbb0487a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb000cb0e0 0x7fbb000cb560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:27.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.847+0000 7fbb0487a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb000cb0e0 0x7fbb000cb560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39402/0 (socket says 172.21.15.16:39402) 2023-10-16T12:26:27.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.847+0000 7fbb0487a700 1 -- 172.21.15.16:0/3506653599 learned_addr learned my addr 172.21.15.16:0/3506653599 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:27.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.848+0000 7fbb0507b700 1 --2- 172.21.15.16:0/3506653599 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbaec005210 0x7fbb00155240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:27.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.848+0000 7fbb0487a700 1 -- 172.21.15.16:0/3506653599 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbaec005210 msgr2=0x7fbb00155240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:27.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.848+0000 7fbb0487a700 1 --2- 172.21.15.16:0/3506653599 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbaec005210 0x7fbb00155240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:27.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.848+0000 7fbb0487a700 1 -- 172.21.15.16:0/3506653599 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbb0006a9b0 msgr2=0x7fbb0006ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:27.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.848+0000 7fbb0587c700 1 --2- 172.21.15.16:0/3506653599 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbb0006a9b0 0x7fbb0006ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:27.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.848+0000 7fbb0487a700 1 --2- 172.21.15.16:0/3506653599 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbb0006a9b0 0x7fbb0006ae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:27.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.848+0000 7fbb0487a700 1 -- 172.21.15.16:0/3506653599 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb001f46c0 con 0x7fbb000cb0e0 2023-10-16T12:26:27.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.848+0000 7fbb0487a700 1 --2- 172.21.15.16:0/3506653599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb000cb0e0 0x7fbb000cb560 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fbafc010850 tx=0x7fbafc010c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:27.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.848+0000 7fbaf5ffb700 1 -- 172.21.15.16:0/3506653599 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbafc010e50 con 0x7fbb000cb0e0 2023-10-16T12:26:27.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.848+0000 7fbaf5ffb700 1 -- 172.21.15.16:0/3506653599 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbafc004020 con 0x7fbb000cb0e0 2023-10-16T12:26:27.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.848+0000 7fbb072df700 1 -- 172.21.15.16:0/3506653599 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb001f49a0 con 0x7fbb000cb0e0 2023-10-16T12:26:27.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.848+0000 7fbaf5ffb700 1 -- 172.21.15.16:0/3506653599 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbafc0043b0 con 0x7fbb000cb0e0 2023-10-16T12:26:27.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.848+0000 7fbb072df700 1 -- 172.21.15.16:0/3506653599 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb0006b370 con 0x7fbb000cb0e0 2023-10-16T12:26:27.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.849+0000 7fbb072df700 1 -- 172.21.15.16:0/3506653599 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb0004ebb0 con 0x7fbb000cb0e0 2023-10-16T12:26:27.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.850+0000 7fbaf5ffb700 1 -- 172.21.15.16:0/3506653599 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 31) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fbafc00e870 con 0x7fbb000cb0e0 2023-10-16T12:26:27.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.850+0000 7fbaf5ffb700 1 --2- 172.21.15.16:0/3506653599 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7fbae406b020 0x7fbae406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:27.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.850+0000 7fbaf5ffb700 1 -- 172.21.15.16:0/3506653599 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fbafc089ae0 con 0x7fbb000cb0e0 2023-10-16T12:26:27.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.850+0000 7fbb0507b700 1 --2- 172.21.15.16:0/3506653599 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7fbae406b020 0x7fbae406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:27.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.851+0000 7fbb0507b700 1 --2- 172.21.15.16:0/3506653599 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7fbae406b020 0x7fbae406d4e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbaf0000f60 tx=0x7fbaf0001110 comp rx=0 tx=0).ready entity=mgr.6396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:27.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:27.855+0000 7fbaf5ffb700 1 -- 172.21.15.16:0/3506653599 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbafc0549b0 con 0x7fbb000cb0e0 2023-10-16T12:26:28.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.061+0000 7fbb072df700 1 -- 172.21.15.16:0/3506653599 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbb00159f00 con 0x7fbb000cb0e0 2023-10-16T12:26:28.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.063+0000 7fbaf5ffb700 1 -- 172.21.15.16:0/3506653599 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+16651 (secure 0 0 0) 0x7fbafc057fd0 con 0x7fbb000cb0e0 2023-10-16T12:26:28.066 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:26:28.066 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":85,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:26:25.280576+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:26:21.893734+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2023-10-16T12:26:22.642815+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:26:28.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.065+0000 7fbb072df700 1 -- 172.21.15.16:0/3506653599 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7fbae406b020 msgr2=0x7fbae406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:28.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.065+0000 7fbb072df700 1 --2- 172.21.15.16:0/3506653599 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7fbae406b020 0x7fbae406d4e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbaf0000f60 tx=0x7fbaf0001110 comp rx=0 tx=0).stop 2023-10-16T12:26:28.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.065+0000 7fbb072df700 1 -- 172.21.15.16:0/3506653599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb000cb0e0 msgr2=0x7fbb000cb560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:28.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.065+0000 7fbb072df700 1 --2- 172.21.15.16:0/3506653599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb000cb0e0 0x7fbb000cb560 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fbafc010850 tx=0x7fbafc010c10 comp rx=0 tx=0).stop 2023-10-16T12:26:28.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.065+0000 7fbb072df700 1 -- 172.21.15.16:0/3506653599 shutdown_connections 2023-10-16T12:26:28.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.065+0000 7fbb072df700 1 --2- 172.21.15.16:0/3506653599 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7fbae406b020 0x7fbae406d4e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:28.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.065+0000 7fbb072df700 1 --2- 172.21.15.16:0/3506653599 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbb0006a9b0 0x7fbb0006ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:28.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.065+0000 7fbb072df700 1 --2- 172.21.15.16:0/3506653599 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbaec005210 0x7fbb00155240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:28.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.065+0000 7fbb072df700 1 --2- 172.21.15.16:0/3506653599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb000cb0e0 0x7fbb000cb560 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:28.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.065+0000 7fbb072df700 1 -- 172.21.15.16:0/3506653599 >> 172.21.15.16:0/3506653599 conn(0x7fbb000cc5b0 msgr2=0x7fbb00158ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:28.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.065+0000 7fbb072df700 1 -- 172.21.15.16:0/3506653599 shutdown_connections 2023-10-16T12:26:28.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.065+0000 7fbb072df700 1 -- 172.21.15.16:0/3506653599 wait complete. 2023-10-16T12:26:28.080 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-16T12:26:28.081 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T12:26:28.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.203+0000 7fe81b4bf700 1 Processor -- start 2023-10-16T12:26:28.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.203+0000 7fe81b4bf700 1 -- start start 2023-10-16T12:26:28.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.204+0000 7fe81b4bf700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe81415dc90 0x7fe81406dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:28.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.204+0000 7fe81b4bf700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe81415e070 0x7fe8140caed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:28.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.204+0000 7fe81b4bf700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8140cb860 con 0x7fe8140cb410 2023-10-16T12:26:28.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.204+0000 7fe81b4bf700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8140cfa00 con 0x7fe81415e070 2023-10-16T12:26:28.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.204+0000 7fe81b4bf700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8140af1e0 con 0x7fe81415dc90 2023-10-16T12:26:28.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.204+0000 7fe818a5a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe81415e070 0x7fe8140caed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:28.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.204+0000 7fe818a5a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe81415e070 0x7fe8140caed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39408/0 (socket says 172.21.15.16:39408) 2023-10-16T12:26:28.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.204+0000 7fe818a5a700 1 -- 172.21.15.16:0/2631007642 learned_addr learned my addr 172.21.15.16:0/2631007642 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:28.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.204+0000 7fe81925b700 1 --2- 172.21.15.16:0/2631007642 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe81415dc90 0x7fe81406dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:28.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.204+0000 7fe818a5a700 1 -- 172.21.15.16:0/2631007642 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe81415dc90 msgr2=0x7fe81406dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:28.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.204+0000 7fe818a5a700 1 --2- 172.21.15.16:0/2631007642 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe81415dc90 0x7fe81406dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:28.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.204+0000 7fe818a5a700 1 -- 172.21.15.16:0/2631007642 >> v1:172.21.15.49:6789/0 conn(0x7fe8140cb410 legacy=0x7fe8140ceb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:28.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.204+0000 7fe818a5a700 1 -- 172.21.15.16:0/2631007642 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe81406e1a0 con 0x7fe81415e070 2023-10-16T12:26:28.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.205+0000 7fe818a5a700 1 --2- 172.21.15.16:0/2631007642 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe81415e070 0x7fe8140caed0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fe8040098f0 tx=0x7fe804009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d98e4a003303154b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:28.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.205+0000 7fe80bfff700 1 -- 172.21.15.16:0/2631007642 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe80400b8a0 con 0x7fe81415e070 2023-10-16T12:26:28.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.205+0000 7fe80bfff700 1 -- 172.21.15.16:0/2631007642 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe80400ba00 con 0x7fe81415e070 2023-10-16T12:26:28.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.205+0000 7fe80bfff700 1 -- 172.21.15.16:0/2631007642 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe80400bd90 con 0x7fe81415e070 2023-10-16T12:26:28.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.205+0000 7fe81b4bf700 1 -- 172.21.15.16:0/2631007642 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe81415e070 msgr2=0x7fe8140caed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:28.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.205+0000 7fe81b4bf700 1 --2- 172.21.15.16:0/2631007642 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe81415e070 0x7fe8140caed0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fe8040098f0 tx=0x7fe804009c00 comp rx=0 tx=0).stop 2023-10-16T12:26:28.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.205+0000 7fe81b4bf700 1 -- 172.21.15.16:0/2631007642 shutdown_connections 2023-10-16T12:26:28.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.205+0000 7fe81b4bf700 1 --2- 172.21.15.16:0/2631007642 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe81415dc90 0x7fe81406dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:28.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.205+0000 7fe81b4bf700 1 --2- 172.21.15.16:0/2631007642 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe81415e070 0x7fe8140caed0 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:28.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.205+0000 7fe81b4bf700 1 -- 172.21.15.16:0/2631007642 >> 172.21.15.16:0/2631007642 conn(0x7fe8140cc5e0 msgr2=0x7fe8140cc9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:28.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.205+0000 7fe81b4bf700 1 -- 172.21.15.16:0/2631007642 shutdown_connections 2023-10-16T12:26:28.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.205+0000 7fe81b4bf700 1 -- 172.21.15.16:0/2631007642 wait complete. 2023-10-16T12:26:28.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.207+0000 7fe81b4bf700 1 Processor -- start 2023-10-16T12:26:28.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.207+0000 7fe81b4bf700 1 -- start start 2023-10-16T12:26:28.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.207+0000 7fe81b4bf700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8140cb410 0x7fe8140c95c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:28.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.207+0000 7fe81b4bf700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe81415dc90 0x7fe814152740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:28.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.207+0000 7fe81b4bf700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe81415e070 0x7fe8140c6c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:28.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.207+0000 7fe81b4bf700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8140c7a80 con 0x7fe8140cb410 2023-10-16T12:26:28.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.207+0000 7fe81b4bf700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8140c77a0 con 0x7fe81415dc90 2023-10-16T12:26:28.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.207+0000 7fe81b4bf700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8140c8ae0 con 0x7fe81415e070 2023-10-16T12:26:28.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.207+0000 7fe81925b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8140cb410 0x7fe8140c95c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:28.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.207+0000 7fe81925b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8140cb410 0x7fe8140c95c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39412/0 (socket says 172.21.15.16:39412) 2023-10-16T12:26:28.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.207+0000 7fe81925b700 1 -- 172.21.15.16:0/1019950398 learned_addr learned my addr 172.21.15.16:0/1019950398 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:28.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.208+0000 7fe818a5a700 1 --2- 172.21.15.16:0/1019950398 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe81415dc90 0x7fe814152740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:28.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.208+0000 7fe819a5c700 1 --2- 172.21.15.16:0/1019950398 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe81415e070 0x7fe8140c6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:28.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.208+0000 7fe81925b700 1 -- 172.21.15.16:0/1019950398 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe81415e070 msgr2=0x7fe8140c6c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:28.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.208+0000 7fe81925b700 1 --2- 172.21.15.16:0/1019950398 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe81415e070 0x7fe8140c6c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:28.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.208+0000 7fe81925b700 1 -- 172.21.15.16:0/1019950398 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe81415dc90 msgr2=0x7fe814152740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:28.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.208+0000 7fe81925b700 1 --2- 172.21.15.16:0/1019950398 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe81415dc90 0x7fe814152740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:28.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.208+0000 7fe81925b700 1 -- 172.21.15.16:0/1019950398 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe804009550 con 0x7fe8140cb410 2023-10-16T12:26:28.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.208+0000 7fe818a5a700 1 --2- 172.21.15.16:0/1019950398 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe81415dc90 0x7fe814152740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:26:28.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.208+0000 7fe81925b700 1 --2- 172.21.15.16:0/1019950398 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8140cb410 0x7fe8140c95c0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fe81000e810 tx=0x7fe81000ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:28.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.208+0000 7fe809ffb700 1 -- 172.21.15.16:0/1019950398 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe810013610 con 0x7fe8140cb410 2023-10-16T12:26:28.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.208+0000 7fe809ffb700 1 -- 172.21.15.16:0/1019950398 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe810013770 con 0x7fe8140cb410 2023-10-16T12:26:28.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.208+0000 7fe81b4bf700 1 -- 172.21.15.16:0/1019950398 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8140c8dc0 con 0x7fe8140cb410 2023-10-16T12:26:28.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.208+0000 7fe809ffb700 1 -- 172.21.15.16:0/1019950398 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe810013b20 con 0x7fe8140cb410 2023-10-16T12:26:28.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.208+0000 7fe81b4bf700 1 -- 172.21.15.16:0/1019950398 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe814153200 con 0x7fe8140cb410 2023-10-16T12:26:28.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.209+0000 7fe81b4bf700 1 -- 172.21.15.16:0/1019950398 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8140b0a80 con 0x7fe8140cb410 2023-10-16T12:26:28.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.210+0000 7fe809ffb700 1 -- 172.21.15.16:0/1019950398 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 31) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe810013c80 con 0x7fe8140cb410 2023-10-16T12:26:28.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.210+0000 7fe809ffb700 1 --2- 172.21.15.16:0/1019950398 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7fe7f806b020 0x7fe7f806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:28.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.210+0000 7fe809ffb700 1 -- 172.21.15.16:0/1019950398 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fe8100896a0 con 0x7fe8140cb410 2023-10-16T12:26:28.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.210+0000 7fe818a5a700 1 --2- 172.21.15.16:0/1019950398 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7fe7f806b020 0x7fe7f806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:28.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.211+0000 7fe818a5a700 1 --2- 172.21.15.16:0/1019950398 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7fe7f806b020 0x7fe7f806d4e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe81415e450 tx=0x7fe804008030 comp rx=0 tx=0).ready entity=mgr.6396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:28.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.215+0000 7fe809ffb700 1 -- 172.21.15.16:0/1019950398 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe810054460 con 0x7fe8140cb410 2023-10-16T12:26:28.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.415+0000 7fe81b4bf700 1 -- 172.21.15.16:0/1019950398 --> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe81406a980 con 0x7fe7f806b020 2023-10-16T12:26:28.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.424+0000 7fe809ffb700 1 -- 172.21.15.16:0/1019950398 <== mgr.6396 v2:172.21.15.16:6832/3415303009 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289419 (secure 0 0 0) 0x7fe80400bb70 con 0x7fe7f806b020 2023-10-16T12:26:28.430 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:26:28.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.427+0000 7fe81b4bf700 1 -- 172.21.15.16:0/1019950398 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7fe7f806b020 msgr2=0x7fe7f806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:28.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.427+0000 7fe81b4bf700 1 --2- 172.21.15.16:0/1019950398 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7fe7f806b020 0x7fe7f806d4e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe81415e450 tx=0x7fe804008030 comp rx=0 tx=0).stop 2023-10-16T12:26:28.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.427+0000 7fe81b4bf700 1 -- 172.21.15.16:0/1019950398 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8140cb410 msgr2=0x7fe8140c95c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:28.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.427+0000 7fe81b4bf700 1 --2- 172.21.15.16:0/1019950398 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8140cb410 0x7fe8140c95c0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fe81000e810 tx=0x7fe81000ebd0 comp rx=0 tx=0).stop 2023-10-16T12:26:28.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.427+0000 7fe81b4bf700 1 -- 172.21.15.16:0/1019950398 shutdown_connections 2023-10-16T12:26:28.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.427+0000 7fe81b4bf700 1 --2- 172.21.15.16:0/1019950398 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7fe7f806b020 0x7fe7f806d4e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:28.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.427+0000 7fe81b4bf700 1 --2- 172.21.15.16:0/1019950398 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe81415e070 0x7fe8140c6c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:28.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.427+0000 7fe81b4bf700 1 --2- 172.21.15.16:0/1019950398 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8140cb410 0x7fe8140c95c0 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:28.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.427+0000 7fe81b4bf700 1 --2- 172.21.15.16:0/1019950398 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe81415dc90 0x7fe814152740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:28.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.427+0000 7fe81b4bf700 1 -- 172.21.15.16:0/1019950398 >> 172.21.15.16:0/1019950398 conn(0x7fe8140cc5e0 msgr2=0x7fe8140b0370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:28.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.427+0000 7fe81b4bf700 1 -- 172.21.15.16:0/1019950398 shutdown_connections 2023-10-16T12:26:28.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.427+0000 7fe81b4bf700 1 -- 172.21.15.16:0/1019950398 wait complete. 2023-10-16T12:26:28.435 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-10-16T12:26:28.443 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2023-10-16T12:26:27.837053+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":283,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":199,"ondisk_log_size":199,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":128,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":341948,"kb_used_data":9228,"kb_used_omap":12,"kb_used_meta":332595,"kb_avail":754632772,"statfs":{"total":773094113280,"available":772743958528,"internally_reserved":0,"allocated":9449472,"data_stored":5263356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12740,"internal_metadata":340577852},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.595300"},"pg_stats":[{"pgid":"10.3f","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.303934+0000","last_change":"2023-10-16T12:26:24.291482+0000","last_active":"2023-10-16T12:26:24.303934+0000","last_peered":"2023-10-16T12:26:24.303934+0000","last_clean":"2023-10-16T12:26:24.303934+0000","last_became_active":"2023-10-16T12:26:24.291214+0000","last_became_peered":"2023-10-16T12:26:24.291214+0000","last_unstale":"2023-10-16T12:26:24.303934+0000","last_undegraded":"2023-10-16T12:26:24.303934+0000","last_fullsized":"2023-10-16T12:26:24.303934+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:30:37.750430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.304100+0000","last_change":"2023-10-16T12:26:24.293356+0000","last_active":"2023-10-16T12:26:24.304100+0000","last_peered":"2023-10-16T12:26:24.304100+0000","last_clean":"2023-10-16T12:26:24.304100+0000","last_became_active":"2023-10-16T12:26:24.293064+0000","last_became_peered":"2023-10-16T12:26:24.293064+0000","last_unstale":"2023-10-16T12:26:24.304100+0000","last_undegraded":"2023-10-16T12:26:24.304100+0000","last_fullsized":"2023-10-16T12:26:24.304100+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:29:54.433098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3d","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.568895+0000","last_change":"2023-10-16T12:26:24.307746+0000","last_active":"2023-10-16T12:26:25.568895+0000","last_peered":"2023-10-16T12:26:25.568895+0000","last_clean":"2023-10-16T12:26:25.568895+0000","last_became_active":"2023-10-16T12:26:24.307102+0000","last_became_peered":"2023-10-16T12:26:24.307102+0000","last_unstale":"2023-10-16T12:26:25.568895+0000","last_undegraded":"2023-10-16T12:26:25.568895+0000","last_fullsized":"2023-10-16T12:26:25.568895+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:38:53.910319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.595730+0000","last_change":"2023-10-16T12:26:24.296652+0000","last_active":"2023-10-16T12:26:25.595730+0000","last_peered":"2023-10-16T12:26:25.595730+0000","last_clean":"2023-10-16T12:26:25.595730+0000","last_became_active":"2023-10-16T12:26:24.296197+0000","last_became_peered":"2023-10-16T12:26:24.296197+0000","last_unstale":"2023-10-16T12:26:25.595730+0000","last_undegraded":"2023-10-16T12:26:25.595730+0000","last_fullsized":"2023-10-16T12:26:25.595730+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:25:13.048790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.298521+0000","last_change":"2023-10-16T12:26:24.294961+0000","last_active":"2023-10-16T12:26:24.298521+0000","last_peered":"2023-10-16T12:26:24.298521+0000","last_clean":"2023-10-16T12:26:24.298521+0000","last_became_active":"2023-10-16T12:26:24.294651+0000","last_became_peered":"2023-10-16T12:26:24.294651+0000","last_unstale":"2023-10-16T12:26:24.298521+0000","last_undegraded":"2023-10-16T12:26:24.298521+0000","last_fullsized":"2023-10-16T12:26:24.298521+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:15:21.085868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.288337+0000","last_change":"2023-10-16T12:26:24.297705+0000","last_active":"2023-10-16T12:26:25.288337+0000","last_peered":"2023-10-16T12:26:25.288337+0000","last_clean":"2023-10-16T12:26:25.288337+0000","last_became_active":"2023-10-16T12:26:24.293985+0000","last_became_peered":"2023-10-16T12:26:24.293985+0000","last_unstale":"2023-10-16T12:26:25.288337+0000","last_undegraded":"2023-10-16T12:26:25.288337+0000","last_fullsized":"2023-10-16T12:26:25.288337+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:36:36.653796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.302030+0000","last_change":"2023-10-16T12:26:24.292411+0000","last_active":"2023-10-16T12:26:24.302030+0000","last_peered":"2023-10-16T12:26:24.302030+0000","last_clean":"2023-10-16T12:26:24.302030+0000","last_became_active":"2023-10-16T12:26:24.291569+0000","last_became_peered":"2023-10-16T12:26:24.291569+0000","last_unstale":"2023-10-16T12:26:24.302030+0000","last_undegraded":"2023-10-16T12:26:24.302030+0000","last_fullsized":"2023-10-16T12:26:24.302030+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:15:12.996970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.294807+0000","last_change":"2023-10-16T12:26:24.288129+0000","last_active":"2023-10-16T12:26:24.294807+0000","last_peered":"2023-10-16T12:26:24.294807+0000","last_clean":"2023-10-16T12:26:24.294807+0000","last_became_active":"2023-10-16T12:26:24.287853+0000","last_became_peered":"2023-10-16T12:26:24.287853+0000","last_unstale":"2023-10-16T12:26:24.294807+0000","last_undegraded":"2023-10-16T12:26:24.294807+0000","last_fullsized":"2023-10-16T12:26:24.294807+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:47:00.719878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.569207+0000","last_change":"2023-10-16T12:26:24.306698+0000","last_active":"2023-10-16T12:26:25.569207+0000","last_peered":"2023-10-16T12:26:25.569207+0000","last_clean":"2023-10-16T12:26:25.569207+0000","last_became_active":"2023-10-16T12:26:24.305524+0000","last_became_peered":"2023-10-16T12:26:24.305524+0000","last_unstale":"2023-10-16T12:26:25.569207+0000","last_undegraded":"2023-10-16T12:26:25.569207+0000","last_fullsized":"2023-10-16T12:26:25.569207+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:34:24.998126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.569057+0000","last_change":"2023-10-16T12:26:24.299860+0000","last_active":"2023-10-16T12:26:25.569057+0000","last_peered":"2023-10-16T12:26:25.569057+0000","last_clean":"2023-10-16T12:26:25.569057+0000","last_became_active":"2023-10-16T12:26:24.299595+0000","last_became_peered":"2023-10-16T12:26:24.299595+0000","last_unstale":"2023-10-16T12:26:25.569057+0000","last_undegraded":"2023-10-16T12:26:25.569057+0000","last_fullsized":"2023-10-16T12:26:25.569057+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:24:05.500326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.595810+0000","last_change":"2023-10-16T12:26:24.299719+0000","last_active":"2023-10-16T12:26:25.595810+0000","last_peered":"2023-10-16T12:26:25.595810+0000","last_clean":"2023-10-16T12:26:25.595810+0000","last_became_active":"2023-10-16T12:26:24.299265+0000","last_became_peered":"2023-10-16T12:26:24.299265+0000","last_unstale":"2023-10-16T12:26:25.595810+0000","last_undegraded":"2023-10-16T12:26:25.595810+0000","last_fullsized":"2023-10-16T12:26:25.595810+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:21:20.665768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.294976+0000","last_change":"2023-10-16T12:26:24.292201+0000","last_active":"2023-10-16T12:26:24.294976+0000","last_peered":"2023-10-16T12:26:24.294976+0000","last_clean":"2023-10-16T12:26:24.294976+0000","last_became_active":"2023-10-16T12:26:24.291873+0000","last_became_peered":"2023-10-16T12:26:24.291873+0000","last_unstale":"2023-10-16T12:26:24.294976+0000","last_undegraded":"2023-10-16T12:26:24.294976+0000","last_fullsized":"2023-10-16T12:26:24.294976+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:24:21.635364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.295802+0000","last_change":"2023-10-16T12:26:24.294587+0000","last_active":"2023-10-16T12:26:24.295802+0000","last_peered":"2023-10-16T12:26:24.295802+0000","last_clean":"2023-10-16T12:26:24.295802+0000","last_became_active":"2023-10-16T12:26:24.294238+0000","last_became_peered":"2023-10-16T12:26:24.294238+0000","last_unstale":"2023-10-16T12:26:24.295802+0000","last_undegraded":"2023-10-16T12:26:24.295802+0000","last_fullsized":"2023-10-16T12:26:24.295802+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:58:55.626587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.569213+0000","last_change":"2023-10-16T12:26:24.305849+0000","last_active":"2023-10-16T12:26:25.569213+0000","last_peered":"2023-10-16T12:26:25.569213+0000","last_clean":"2023-10-16T12:26:25.569213+0000","last_became_active":"2023-10-16T12:26:24.305271+0000","last_became_peered":"2023-10-16T12:26:24.305271+0000","last_unstale":"2023-10-16T12:26:25.569213+0000","last_undegraded":"2023-10-16T12:26:25.569213+0000","last_fullsized":"2023-10-16T12:26:25.569213+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:40:05.795607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.569065+0000","last_change":"2023-10-16T12:26:24.292049+0000","last_active":"2023-10-16T12:26:25.569065+0000","last_peered":"2023-10-16T12:26:25.569065+0000","last_clean":"2023-10-16T12:26:25.569065+0000","last_became_active":"2023-10-16T12:26:24.291685+0000","last_became_peered":"2023-10-16T12:26:24.291685+0000","last_unstale":"2023-10-16T12:26:25.569065+0000","last_undegraded":"2023-10-16T12:26:25.569065+0000","last_fullsized":"2023-10-16T12:26:25.569065+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:00:37.235614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.302766+0000","last_change":"2023-10-16T12:26:24.296582+0000","last_active":"2023-10-16T12:26:24.302766+0000","last_peered":"2023-10-16T12:26:24.302766+0000","last_clean":"2023-10-16T12:26:24.302766+0000","last_became_active":"2023-10-16T12:26:24.292026+0000","last_became_peered":"2023-10-16T12:26:24.292026+0000","last_unstale":"2023-10-16T12:26:24.302766+0000","last_undegraded":"2023-10-16T12:26:24.302766+0000","last_fullsized":"2023-10-16T12:26:24.302766+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:03:25.282953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.304785+0000","last_change":"2023-10-16T12:26:24.298939+0000","last_active":"2023-10-16T12:26:24.304785+0000","last_peered":"2023-10-16T12:26:24.304785+0000","last_clean":"2023-10-16T12:26:24.304785+0000","last_became_active":"2023-10-16T12:26:24.298658+0000","last_became_peered":"2023-10-16T12:26:24.298658+0000","last_unstale":"2023-10-16T12:26:24.304785+0000","last_undegraded":"2023-10-16T12:26:24.304785+0000","last_fullsized":"2023-10-16T12:26:24.304785+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:33:21.708694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.595581+0000","last_change":"2023-10-16T12:26:24.294938+0000","last_active":"2023-10-16T12:26:25.595581+0000","last_peered":"2023-10-16T12:26:25.595581+0000","last_clean":"2023-10-16T12:26:25.595581+0000","last_became_active":"2023-10-16T12:26:24.294373+0000","last_became_peered":"2023-10-16T12:26:24.294373+0000","last_unstale":"2023-10-16T12:26:25.595581+0000","last_undegraded":"2023-10-16T12:26:25.595581+0000","last_fullsized":"2023-10-16T12:26:25.595581+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:23:12.559260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.295044+0000","last_change":"2023-10-16T12:26:24.292910+0000","last_active":"2023-10-16T12:26:24.295044+0000","last_peered":"2023-10-16T12:26:24.295044+0000","last_clean":"2023-10-16T12:26:24.295044+0000","last_became_active":"2023-10-16T12:26:24.292438+0000","last_became_peered":"2023-10-16T12:26:24.292438+0000","last_unstale":"2023-10-16T12:26:24.295044+0000","last_undegraded":"2023-10-16T12:26:24.295044+0000","last_fullsized":"2023-10-16T12:26:24.295044+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:20:02.702684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.288327+0000","last_change":"2023-10-16T12:26:24.290041+0000","last_active":"2023-10-16T12:26:25.288327+0000","last_peered":"2023-10-16T12:26:25.288327+0000","last_clean":"2023-10-16T12:26:25.288327+0000","last_became_active":"2023-10-16T12:26:24.289796+0000","last_became_peered":"2023-10-16T12:26:24.289796+0000","last_unstale":"2023-10-16T12:26:25.288327+0000","last_undegraded":"2023-10-16T12:26:25.288327+0000","last_fullsized":"2023-10-16T12:26:25.288327+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:14:41.508247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.297822+0000","last_change":"2023-10-16T12:26:24.295721+0000","last_active":"2023-10-16T12:26:24.297822+0000","last_peered":"2023-10-16T12:26:24.297822+0000","last_clean":"2023-10-16T12:26:24.297822+0000","last_became_active":"2023-10-16T12:26:24.295354+0000","last_became_peered":"2023-10-16T12:26:24.295354+0000","last_unstale":"2023-10-16T12:26:24.297822+0000","last_undegraded":"2023-10-16T12:26:24.297822+0000","last_fullsized":"2023-10-16T12:26:24.297822+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:13:49.992850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.287970+0000","last_change":"2023-10-16T12:26:24.293166+0000","last_active":"2023-10-16T12:26:25.287970+0000","last_peered":"2023-10-16T12:26:25.287970+0000","last_clean":"2023-10-16T12:26:25.287970+0000","last_became_active":"2023-10-16T12:26:24.292885+0000","last_became_peered":"2023-10-16T12:26:24.292885+0000","last_unstale":"2023-10-16T12:26:25.287970+0000","last_undegraded":"2023-10-16T12:26:25.287970+0000","last_fullsized":"2023-10-16T12:26:25.287970+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:23:51.982187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.287816+0000","last_change":"2023-10-16T12:26:24.293639+0000","last_active":"2023-10-16T12:26:25.287816+0000","last_peered":"2023-10-16T12:26:25.287816+0000","last_clean":"2023-10-16T12:26:25.287816+0000","last_became_active":"2023-10-16T12:26:24.293414+0000","last_became_peered":"2023-10-16T12:26:24.293414+0000","last_unstale":"2023-10-16T12:26:25.287816+0000","last_undegraded":"2023-10-16T12:26:25.287816+0000","last_fullsized":"2023-10-16T12:26:25.287816+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:16:12.888429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.568704+0000","last_change":"2023-10-16T12:26:24.306975+0000","last_active":"2023-10-16T12:26:25.568704+0000","last_peered":"2023-10-16T12:26:25.568704+0000","last_clean":"2023-10-16T12:26:25.568704+0000","last_became_active":"2023-10-16T12:26:24.306548+0000","last_became_peered":"2023-10-16T12:26:24.306548+0000","last_unstale":"2023-10-16T12:26:25.568704+0000","last_undegraded":"2023-10-16T12:26:25.568704+0000","last_fullsized":"2023-10-16T12:26:25.568704+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:21:46.897480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.294499+0000","last_change":"2023-10-16T12:26:24.287586+0000","last_active":"2023-10-16T12:26:24.294499+0000","last_peered":"2023-10-16T12:26:24.294499+0000","last_clean":"2023-10-16T12:26:24.294499+0000","last_became_active":"2023-10-16T12:26:24.287184+0000","last_became_peered":"2023-10-16T12:26:24.287184+0000","last_unstale":"2023-10-16T12:26:24.294499+0000","last_undegraded":"2023-10-16T12:26:24.294499+0000","last_fullsized":"2023-10-16T12:26:24.294499+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:05:59.476003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.294452+0000","last_change":"2023-10-16T12:26:24.287423+0000","last_active":"2023-10-16T12:26:24.294452+0000","last_peered":"2023-10-16T12:26:24.294452+0000","last_clean":"2023-10-16T12:26:24.294452+0000","last_became_active":"2023-10-16T12:26:24.287102+0000","last_became_peered":"2023-10-16T12:26:24.287102+0000","last_unstale":"2023-10-16T12:26:24.294452+0000","last_undegraded":"2023-10-16T12:26:24.294452+0000","last_fullsized":"2023-10-16T12:26:24.294452+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:49:20.689145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.595932+0000","last_change":"2023-10-16T12:26:24.296525+0000","last_active":"2023-10-16T12:26:25.595932+0000","last_peered":"2023-10-16T12:26:25.595932+0000","last_clean":"2023-10-16T12:26:25.595932+0000","last_became_active":"2023-10-16T12:26:24.294698+0000","last_became_peered":"2023-10-16T12:26:24.294698+0000","last_unstale":"2023-10-16T12:26:25.595932+0000","last_undegraded":"2023-10-16T12:26:25.595932+0000","last_fullsized":"2023-10-16T12:26:25.595932+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:36:54.572808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.296559+0000","last_change":"2023-10-16T12:26:24.285157+0000","last_active":"2023-10-16T12:26:24.296559+0000","last_peered":"2023-10-16T12:26:24.296559+0000","last_clean":"2023-10-16T12:26:24.296559+0000","last_became_active":"2023-10-16T12:26:24.284854+0000","last_became_peered":"2023-10-16T12:26:24.284854+0000","last_unstale":"2023-10-16T12:26:24.296559+0000","last_undegraded":"2023-10-16T12:26:24.296559+0000","last_fullsized":"2023-10-16T12:26:24.296559+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:00:55.517863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.569122+0000","last_change":"2023-10-16T12:26:24.302840+0000","last_active":"2023-10-16T12:26:25.569122+0000","last_peered":"2023-10-16T12:26:25.569122+0000","last_clean":"2023-10-16T12:26:25.569122+0000","last_became_active":"2023-10-16T12:26:24.302545+0000","last_became_peered":"2023-10-16T12:26:24.302545+0000","last_unstale":"2023-10-16T12:26:25.569122+0000","last_undegraded":"2023-10-16T12:26:25.569122+0000","last_fullsized":"2023-10-16T12:26:25.569122+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:00:11.640752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.295524+0000","last_change":"2023-10-16T12:26:24.293502+0000","last_active":"2023-10-16T12:26:24.295524+0000","last_peered":"2023-10-16T12:26:24.295524+0000","last_clean":"2023-10-16T12:26:24.295524+0000","last_became_active":"2023-10-16T12:26:24.293207+0000","last_became_peered":"2023-10-16T12:26:24.293207+0000","last_unstale":"2023-10-16T12:26:24.295524+0000","last_undegraded":"2023-10-16T12:26:24.295524+0000","last_fullsized":"2023-10-16T12:26:24.295524+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:46:33.594590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.595432+0000","last_change":"2023-10-16T12:26:24.299413+0000","last_active":"2023-10-16T12:26:25.595432+0000","last_peered":"2023-10-16T12:26:25.595432+0000","last_clean":"2023-10-16T12:26:25.595432+0000","last_became_active":"2023-10-16T12:26:24.296113+0000","last_became_peered":"2023-10-16T12:26:24.296113+0000","last_unstale":"2023-10-16T12:26:25.595432+0000","last_undegraded":"2023-10-16T12:26:25.595432+0000","last_fullsized":"2023-10-16T12:26:25.595432+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:30:37.750430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.288025+0000","last_change":"2023-10-16T12:26:24.293780+0000","last_active":"2023-10-16T12:26:25.288025+0000","last_peered":"2023-10-16T12:26:25.288025+0000","last_clean":"2023-10-16T12:26:25.288025+0000","last_became_active":"2023-10-16T12:26:24.293403+0000","last_became_peered":"2023-10-16T12:26:24.293403+0000","last_unstale":"2023-10-16T12:26:25.288025+0000","last_undegraded":"2023-10-16T12:26:25.288025+0000","last_fullsized":"2023-10-16T12:26:25.288025+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:03:52.845551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.294876+0000","last_change":"2023-10-16T12:26:24.292760+0000","last_active":"2023-10-16T12:26:24.294876+0000","last_peered":"2023-10-16T12:26:24.294876+0000","last_clean":"2023-10-16T12:26:24.294876+0000","last_became_active":"2023-10-16T12:26:24.292372+0000","last_became_peered":"2023-10-16T12:26:24.292372+0000","last_unstale":"2023-10-16T12:26:24.294876+0000","last_undegraded":"2023-10-16T12:26:24.294876+0000","last_fullsized":"2023-10-16T12:26:24.294876+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:38:04.076463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"84'4","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.307739+0000","last_change":"2023-10-16T12:26:23.265538+0000","last_active":"2023-10-16T12:26:24.307739+0000","last_peered":"2023-10-16T12:26:24.307739+0000","last_clean":"2023-10-16T12:26:24.307739+0000","last_became_active":"2023-10-16T12:26:23.265073+0000","last_became_peered":"2023-10-16T12:26:23.265073+0000","last_unstale":"2023-10-16T12:26:24.307739+0000","last_undegraded":"2023-10-16T12:26:24.307739+0000","last_fullsized":"2023-10-16T12:26:24.307739+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_clean_scrub_stamp":"2023-10-16T12:26:22.237327+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:28:46.749334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.595971+0000","last_change":"2023-10-16T12:26:24.294447+0000","last_active":"2023-10-16T12:26:25.595971+0000","last_peered":"2023-10-16T12:26:25.595971+0000","last_clean":"2023-10-16T12:26:25.595971+0000","last_became_active":"2023-10-16T12:26:24.294111+0000","last_became_peered":"2023-10-16T12:26:24.294111+0000","last_unstale":"2023-10-16T12:26:25.595971+0000","last_undegraded":"2023-10-16T12:26:25.595971+0000","last_fullsized":"2023-10-16T12:26:25.595971+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:52:58.368751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"84'1","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.288826+0000","last_change":"2023-10-16T12:26:23.253068+0000","last_active":"2023-10-16T12:26:25.288826+0000","last_peered":"2023-10-16T12:26:25.288826+0000","last_clean":"2023-10-16T12:26:25.288826+0000","last_became_active":"2023-10-16T12:26:23.252558+0000","last_became_peered":"2023-10-16T12:26:23.252558+0000","last_unstale":"2023-10-16T12:26:25.288826+0000","last_undegraded":"2023-10-16T12:26:25.288826+0000","last_fullsized":"2023-10-16T12:26:25.288826+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_clean_scrub_stamp":"2023-10-16T12:26:22.237327+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:38:37.255759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.288883+0000","last_change":"2023-10-16T12:26:24.297676+0000","last_active":"2023-10-16T12:26:25.288883+0000","last_peered":"2023-10-16T12:26:25.288883+0000","last_clean":"2023-10-16T12:26:25.288883+0000","last_became_active":"2023-10-16T12:26:24.293650+0000","last_became_peered":"2023-10-16T12:26:24.293650+0000","last_unstale":"2023-10-16T12:26:25.288883+0000","last_undegraded":"2023-10-16T12:26:25.288883+0000","last_fullsized":"2023-10-16T12:26:25.288883+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:41:43.972573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"84'2","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.288791+0000","last_change":"2023-10-16T12:26:23.267505+0000","last_active":"2023-10-16T12:26:25.288791+0000","last_peered":"2023-10-16T12:26:25.288791+0000","last_clean":"2023-10-16T12:26:25.288791+0000","last_became_active":"2023-10-16T12:26:23.267141+0000","last_became_peered":"2023-10-16T12:26:23.267141+0000","last_unstale":"2023-10-16T12:26:25.288791+0000","last_undegraded":"2023-10-16T12:26:25.288791+0000","last_fullsized":"2023-10-16T12:26:25.288791+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_clean_scrub_stamp":"2023-10-16T12:26:22.237327+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:00:10.636690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.569282+0000","last_change":"2023-10-16T12:26:24.305977+0000","last_active":"2023-10-16T12:26:25.569282+0000","last_peered":"2023-10-16T12:26:25.569282+0000","last_clean":"2023-10-16T12:26:25.569282+0000","last_became_active":"2023-10-16T12:26:24.305647+0000","last_became_peered":"2023-10-16T12:26:24.305647+0000","last_unstale":"2023-10-16T12:26:25.569282+0000","last_undegraded":"2023-10-16T12:26:25.569282+0000","last_fullsized":"2023-10-16T12:26:25.569282+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:02:17.655193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"84'2","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.288568+0000","last_change":"2023-10-16T12:26:23.269237+0000","last_active":"2023-10-16T12:26:25.288568+0000","last_peered":"2023-10-16T12:26:25.288568+0000","last_clean":"2023-10-16T12:26:25.288568+0000","last_became_active":"2023-10-16T12:26:23.268929+0000","last_became_peered":"2023-10-16T12:26:23.268929+0000","last_unstale":"2023-10-16T12:26:25.288568+0000","last_undegraded":"2023-10-16T12:26:25.288568+0000","last_fullsized":"2023-10-16T12:26:25.288568+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_clean_scrub_stamp":"2023-10-16T12:26:22.237327+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:02:16.651131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.302793+0000","last_change":"2023-10-16T12:26:24.296806+0000","last_active":"2023-10-16T12:26:24.302793+0000","last_peered":"2023-10-16T12:26:24.302793+0000","last_clean":"2023-10-16T12:26:24.302793+0000","last_became_active":"2023-10-16T12:26:24.291631+0000","last_became_peered":"2023-10-16T12:26:24.291631+0000","last_unstale":"2023-10-16T12:26:24.302793+0000","last_undegraded":"2023-10-16T12:26:24.302793+0000","last_fullsized":"2023-10-16T12:26:24.302793+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:28:46.365589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"84'1","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.288200+0000","last_change":"2023-10-16T12:26:23.263406+0000","last_active":"2023-10-16T12:26:25.288200+0000","last_peered":"2023-10-16T12:26:25.288200+0000","last_clean":"2023-10-16T12:26:25.288200+0000","last_became_active":"2023-10-16T12:26:23.263106+0000","last_became_peered":"2023-10-16T12:26:23.263106+0000","last_unstale":"2023-10-16T12:26:25.288200+0000","last_undegraded":"2023-10-16T12:26:25.288200+0000","last_fullsized":"2023-10-16T12:26:25.288200+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_clean_scrub_stamp":"2023-10-16T12:26:22.237327+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:34:23.994064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.304643+0000","last_change":"2023-10-16T12:26:24.295804+0000","last_active":"2023-10-16T12:26:24.304643+0000","last_peered":"2023-10-16T12:26:24.304643+0000","last_clean":"2023-10-16T12:26:24.304643+0000","last_became_active":"2023-10-16T12:26:24.295466+0000","last_became_peered":"2023-10-16T12:26:24.295466+0000","last_unstale":"2023-10-16T12:26:24.304643+0000","last_undegraded":"2023-10-16T12:26:24.304643+0000","last_fullsized":"2023-10-16T12:26:24.304643+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:34:11.928461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"84'1","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.287879+0000","last_change":"2023-10-16T12:26:23.265444+0000","last_active":"2023-10-16T12:26:25.287879+0000","last_peered":"2023-10-16T12:26:25.287879+0000","last_clean":"2023-10-16T12:26:25.287879+0000","last_became_active":"2023-10-16T12:26:23.265117+0000","last_became_peered":"2023-10-16T12:26:23.265117+0000","last_unstale":"2023-10-16T12:26:25.287879+0000","last_undegraded":"2023-10-16T12:26:25.287879+0000","last_fullsized":"2023-10-16T12:26:25.287879+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_clean_scrub_stamp":"2023-10-16T12:26:22.237327+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:02:16.651131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.278004+0000","last_change":"2023-10-16T12:26:24.277477+0000","last_active":"2023-10-16T12:26:24.278004+0000","last_peered":"2023-10-16T12:26:24.278004+0000","last_clean":"2023-10-16T12:26:24.278004+0000","last_became_active":"2023-10-16T12:26:24.277203+0000","last_became_peered":"2023-10-16T12:26:24.277203+0000","last_unstale":"2023-10-16T12:26:24.278004+0000","last_undegraded":"2023-10-16T12:26:24.278004+0000","last_fullsized":"2023-10-16T12:26:24.278004+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:41:16.602571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.287967+0000","last_change":"2023-10-16T12:26:23.266252+0000","last_active":"2023-10-16T12:26:25.287967+0000","last_peered":"2023-10-16T12:26:25.287967+0000","last_clean":"2023-10-16T12:26:25.287967+0000","last_became_active":"2023-10-16T12:26:23.265945+0000","last_became_peered":"2023-10-16T12:26:23.265945+0000","last_unstale":"2023-10-16T12:26:25.287967+0000","last_undegraded":"2023-10-16T12:26:25.287967+0000","last_fullsized":"2023-10-16T12:26:25.287967+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_clean_scrub_stamp":"2023-10-16T12:26:22.237327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:51:18.592928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.288106+0000","last_change":"2023-10-16T12:26:24.301407+0000","last_active":"2023-10-16T12:26:25.288106+0000","last_peered":"2023-10-16T12:26:25.288106+0000","last_clean":"2023-10-16T12:26:25.288106+0000","last_became_active":"2023-10-16T12:26:24.301089+0000","last_became_peered":"2023-10-16T12:26:24.301089+0000","last_unstale":"2023-10-16T12:26:25.288106+0000","last_undegraded":"2023-10-16T12:26:25.288106+0000","last_fullsized":"2023-10-16T12:26:25.288106+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:47:40.000592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.277000+0000","last_change":"2023-10-16T12:26:23.259422+0000","last_active":"2023-10-16T12:26:24.277000+0000","last_peered":"2023-10-16T12:26:24.277000+0000","last_clean":"2023-10-16T12:26:24.277000+0000","last_became_active":"2023-10-16T12:26:23.259137+0000","last_became_peered":"2023-10-16T12:26:23.259137+0000","last_unstale":"2023-10-16T12:26:24.277000+0000","last_undegraded":"2023-10-16T12:26:24.277000+0000","last_fullsized":"2023-10-16T12:26:24.277000+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_clean_scrub_stamp":"2023-10-16T12:26:22.237327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:31:48.776291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.304026+0000","last_change":"2023-10-16T12:26:24.293242+0000","last_active":"2023-10-16T12:26:24.304026+0000","last_peered":"2023-10-16T12:26:24.304026+0000","last_clean":"2023-10-16T12:26:24.304026+0000","last_became_active":"2023-10-16T12:26:24.292959+0000","last_became_peered":"2023-10-16T12:26:24.292959+0000","last_unstale":"2023-10-16T12:26:24.304026+0000","last_undegraded":"2023-10-16T12:26:24.304026+0000","last_fullsized":"2023-10-16T12:26:24.304026+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:37:23.623750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.569303+0000","last_change":"2023-10-16T12:26:23.267689+0000","last_active":"2023-10-16T12:26:25.569303+0000","last_peered":"2023-10-16T12:26:25.569303+0000","last_clean":"2023-10-16T12:26:25.569303+0000","last_became_active":"2023-10-16T12:26:23.266807+0000","last_became_peered":"2023-10-16T12:26:23.266807+0000","last_unstale":"2023-10-16T12:26:25.569303+0000","last_undegraded":"2023-10-16T12:26:25.569303+0000","last_fullsized":"2023-10-16T12:26:25.569303+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_clean_scrub_stamp":"2023-10-16T12:26:22.237327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:09:36.660402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.288477+0000","last_change":"2023-10-16T12:26:24.295479+0000","last_active":"2023-10-16T12:26:25.288477+0000","last_peered":"2023-10-16T12:26:25.288477+0000","last_clean":"2023-10-16T12:26:25.288477+0000","last_became_active":"2023-10-16T12:26:24.295101+0000","last_became_peered":"2023-10-16T12:26:24.295101+0000","last_unstale":"2023-10-16T12:26:25.288477+0000","last_undegraded":"2023-10-16T12:26:25.288477+0000","last_fullsized":"2023-10-16T12:26:25.288477+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:48:27.755542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.595891+0000","last_change":"2023-10-16T12:26:23.260738+0000","last_active":"2023-10-16T12:26:25.595891+0000","last_peered":"2023-10-16T12:26:25.595891+0000","last_clean":"2023-10-16T12:26:25.595891+0000","last_became_active":"2023-10-16T12:26:23.260449+0000","last_became_peered":"2023-10-16T12:26:23.260449+0000","last_unstale":"2023-10-16T12:26:25.595891+0000","last_undegraded":"2023-10-16T12:26:25.595891+0000","last_fullsized":"2023-10-16T12:26:25.595891+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_clean_scrub_stamp":"2023-10-16T12:26:22.237327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:53:21.641122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.289086+0000","last_change":"2023-10-16T12:26:24.285035+0000","last_active":"2023-10-16T12:26:24.289086+0000","last_peered":"2023-10-16T12:26:24.289086+0000","last_clean":"2023-10-16T12:26:24.289086+0000","last_became_active":"2023-10-16T12:26:24.284743+0000","last_became_peered":"2023-10-16T12:26:24.284743+0000","last_unstale":"2023-10-16T12:26:24.289086+0000","last_undegraded":"2023-10-16T12:26:24.289086+0000","last_fullsized":"2023-10-16T12:26:24.289086+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:22:48.536783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.15","version":"84'3","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.568785+0000","last_change":"2023-10-16T12:26:23.267138+0000","last_active":"2023-10-16T12:26:25.568785+0000","last_peered":"2023-10-16T12:26:25.568785+0000","last_clean":"2023-10-16T12:26:25.568785+0000","last_became_active":"2023-10-16T12:26:23.266839+0000","last_became_peered":"2023-10-16T12:26:23.266839+0000","last_unstale":"2023-10-16T12:26:25.568785+0000","last_undegraded":"2023-10-16T12:26:25.568785+0000","last_fullsized":"2023-10-16T12:26:25.568785+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_clean_scrub_stamp":"2023-10-16T12:26:22.237327+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:58:54.622524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.595934+0000","last_change":"2023-10-16T12:26:24.290640+0000","last_active":"2023-10-16T12:26:25.595934+0000","last_peered":"2023-10-16T12:26:25.595934+0000","last_clean":"2023-10-16T12:26:25.595934+0000","last_became_active":"2023-10-16T12:26:24.290395+0000","last_became_peered":"2023-10-16T12:26:24.290395+0000","last_unstale":"2023-10-16T12:26:25.595934+0000","last_undegraded":"2023-10-16T12:26:25.595934+0000","last_fullsized":"2023-10-16T12:26:25.595934+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:57:11.946684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"84'6","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.309885+0000","last_change":"2023-10-16T12:26:23.268885+0000","last_active":"2023-10-16T12:26:24.309885+0000","last_peered":"2023-10-16T12:26:24.309885+0000","last_clean":"2023-10-16T12:26:24.309885+0000","last_became_active":"2023-10-16T12:26:23.268605+0000","last_became_peered":"2023-10-16T12:26:23.268605+0000","last_unstale":"2023-10-16T12:26:24.309885+0000","last_undegraded":"2023-10-16T12:26:24.309885+0000","last_fullsized":"2023-10-16T12:26:24.309885+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_clean_scrub_stamp":"2023-10-16T12:26:22.237327+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:05:49.004866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.595417+0000","last_change":"2023-10-16T12:26:24.295985+0000","last_active":"2023-10-16T12:26:25.595417+0000","last_peered":"2023-10-16T12:26:25.595417+0000","last_clean":"2023-10-16T12:26:25.595417+0000","last_became_active":"2023-10-16T12:26:24.295703+0000","last_became_peered":"2023-10-16T12:26:24.295703+0000","last_unstale":"2023-10-16T12:26:25.595417+0000","last_undegraded":"2023-10-16T12:26:25.595417+0000","last_fullsized":"2023-10-16T12:26:25.595417+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:53:28.283879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.13","version":"84'2","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.569138+0000","last_change":"2023-10-16T12:26:23.259511+0000","last_active":"2023-10-16T12:26:25.569138+0000","last_peered":"2023-10-16T12:26:25.569138+0000","last_clean":"2023-10-16T12:26:25.569138+0000","last_became_active":"2023-10-16T12:26:23.259189+0000","last_became_peered":"2023-10-16T12:26:23.259189+0000","last_unstale":"2023-10-16T12:26:25.569138+0000","last_undegraded":"2023-10-16T12:26:25.569138+0000","last_fullsized":"2023-10-16T12:26:25.569138+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_clean_scrub_stamp":"2023-10-16T12:26:22.237327+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:24:50.857165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.294984+0000","last_change":"2023-10-16T12:26:24.287755+0000","last_active":"2023-10-16T12:26:24.294984+0000","last_peered":"2023-10-16T12:26:24.294984+0000","last_clean":"2023-10-16T12:26:24.294984+0000","last_became_active":"2023-10-16T12:26:24.287450+0000","last_became_peered":"2023-10-16T12:26:24.287450+0000","last_unstale":"2023-10-16T12:26:24.294984+0000","last_undegraded":"2023-10-16T12:26:24.294984+0000","last_fullsized":"2023-10-16T12:26:24.294984+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:12:41.847476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.281035+0000","last_change":"2023-10-16T12:26:23.267748+0000","last_active":"2023-10-16T12:26:24.281035+0000","last_peered":"2023-10-16T12:26:24.281035+0000","last_clean":"2023-10-16T12:26:24.281035+0000","last_became_active":"2023-10-16T12:26:23.267464+0000","last_became_peered":"2023-10-16T12:26:23.267464+0000","last_unstale":"2023-10-16T12:26:24.281035+0000","last_undegraded":"2023-10-16T12:26:24.281035+0000","last_fullsized":"2023-10-16T12:26:24.281035+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_clean_scrub_stamp":"2023-10-16T12:26:22.237327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:42:08.818035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.289312+0000","last_change":"2023-10-16T12:26:24.281566+0000","last_active":"2023-10-16T12:26:24.289312+0000","last_peered":"2023-10-16T12:26:24.289312+0000","last_clean":"2023-10-16T12:26:24.289312+0000","last_became_active":"2023-10-16T12:26:24.281274+0000","last_became_peered":"2023-10-16T12:26:24.281274+0000","last_unstale":"2023-10-16T12:26:24.289312+0000","last_undegraded":"2023-10-16T12:26:24.289312+0000","last_fullsized":"2023-10-16T12:26:24.289312+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:24:31.347918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.569223+0000","last_change":"2023-10-16T12:26:23.264717+0000","last_active":"2023-10-16T12:26:25.569223+0000","last_peered":"2023-10-16T12:26:25.569223+0000","last_clean":"2023-10-16T12:26:25.569223+0000","last_became_active":"2023-10-16T12:26:23.264412+0000","last_became_peered":"2023-10-16T12:26:23.264412+0000","last_unstale":"2023-10-16T12:26:25.569223+0000","last_undegraded":"2023-10-16T12:26:25.569223+0000","last_fullsized":"2023-10-16T12:26:25.569223+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_clean_scrub_stamp":"2023-10-16T12:26:22.237327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:05:58.471940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.303949+0000","last_change":"2023-10-16T12:26:24.292328+0000","last_active":"2023-10-16T12:26:24.303949+0000","last_peered":"2023-10-16T12:26:24.303949+0000","last_clean":"2023-10-16T12:26:24.303949+0000","last_became_active":"2023-10-16T12:26:24.292043+0000","last_became_peered":"2023-10-16T12:26:24.292043+0000","last_unstale":"2023-10-16T12:26:24.303949+0000","last_undegraded":"2023-10-16T12:26:24.303949+0000","last_fullsized":"2023-10-16T12:26:24.303949+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:23:12.559260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"84'8","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.595337+0000","last_change":"2023-10-16T12:26:23.262544+0000","last_active":"2023-10-16T12:26:25.595337+0000","last_peered":"2023-10-16T12:26:25.595337+0000","last_clean":"2023-10-16T12:26:25.595337+0000","last_became_active":"2023-10-16T12:26:23.262262+0000","last_became_peered":"2023-10-16T12:26:23.262262+0000","last_unstale":"2023-10-16T12:26:25.595337+0000","last_undegraded":"2023-10-16T12:26:25.595337+0000","last_fullsized":"2023-10-16T12:26:25.595337+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_clean_scrub_stamp":"2023-10-16T12:26:22.237327+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:05:49.004866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.568992+0000","last_change":"2023-10-16T12:26:24.298699+0000","last_active":"2023-10-16T12:26:25.568992+0000","last_peered":"2023-10-16T12:26:25.568992+0000","last_clean":"2023-10-16T12:26:25.568992+0000","last_became_active":"2023-10-16T12:26:24.298273+0000","last_became_peered":"2023-10-16T12:26:24.298273+0000","last_unstale":"2023-10-16T12:26:25.568992+0000","last_undegraded":"2023-10-16T12:26:25.568992+0000","last_fullsized":"2023-10-16T12:26:25.568992+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:08:36.957709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"84'1","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.306220+0000","last_change":"2023-10-16T12:26:23.267470+0000","last_active":"2023-10-16T12:26:24.306220+0000","last_peered":"2023-10-16T12:26:24.306220+0000","last_clean":"2023-10-16T12:26:24.306220+0000","last_became_active":"2023-10-16T12:26:23.267162+0000","last_became_peered":"2023-10-16T12:26:23.267162+0000","last_unstale":"2023-10-16T12:26:24.306220+0000","last_undegraded":"2023-10-16T12:26:24.306220+0000","last_fullsized":"2023-10-16T12:26:24.306220+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_clean_scrub_stamp":"2023-10-16T12:26:22.237327+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:33:19.198155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.301636+0000","last_change":"2023-10-16T12:26:24.300661+0000","last_active":"2023-10-16T12:26:24.301636+0000","last_peered":"2023-10-16T12:26:24.301636+0000","last_clean":"2023-10-16T12:26:24.301636+0000","last_became_active":"2023-10-16T12:26:24.300286+0000","last_became_peered":"2023-10-16T12:26:24.300286+0000","last_unstale":"2023-10-16T12:26:24.301636+0000","last_undegraded":"2023-10-16T12:26:24.301636+0000","last_fullsized":"2023-10-16T12:26:24.301636+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:58:00.944373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"84'2","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.569005+0000","last_change":"2023-10-16T12:26:23.258970+0000","last_active":"2023-10-16T12:26:25.569005+0000","last_peered":"2023-10-16T12:26:25.569005+0000","last_clean":"2023-10-16T12:26:25.569005+0000","last_became_active":"2023-10-16T12:26:23.258519+0000","last_became_peered":"2023-10-16T12:26:23.258519+0000","last_unstale":"2023-10-16T12:26:25.569005+0000","last_undegraded":"2023-10-16T12:26:25.569005+0000","last_fullsized":"2023-10-16T12:26:25.569005+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_clean_scrub_stamp":"2023-10-16T12:26:22.237327+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:44:34.923567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.595664+0000","last_change":"2023-10-16T12:26:24.291315+0000","last_active":"2023-10-16T12:26:25.595664+0000","last_peered":"2023-10-16T12:26:25.595664+0000","last_clean":"2023-10-16T12:26:25.595664+0000","last_became_active":"2023-10-16T12:26:24.290993+0000","last_became_peered":"2023-10-16T12:26:24.290993+0000","last_unstale":"2023-10-16T12:26:25.595664+0000","last_undegraded":"2023-10-16T12:26:25.595664+0000","last_fullsized":"2023-10-16T12:26:25.595664+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:42:18.585503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"84'1","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.568994+0000","last_change":"2023-10-16T12:26:23.261915+0000","last_active":"2023-10-16T12:26:25.568994+0000","last_peered":"2023-10-16T12:26:25.568994+0000","last_clean":"2023-10-16T12:26:25.568994+0000","last_became_active":"2023-10-16T12:26:23.261529+0000","last_became_peered":"2023-10-16T12:26:23.261529+0000","last_unstale":"2023-10-16T12:26:25.568994+0000","last_undegraded":"2023-10-16T12:26:25.568994+0000","last_fullsized":"2023-10-16T12:26:25.568994+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_clean_scrub_stamp":"2023-10-16T12:26:22.237327+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:42:08.818035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.294823+0000","last_change":"2023-10-16T12:26:24.289953+0000","last_active":"2023-10-16T12:26:24.294823+0000","last_peered":"2023-10-16T12:26:24.294823+0000","last_clean":"2023-10-16T12:26:24.294823+0000","last_became_active":"2023-10-16T12:26:24.289677+0000","last_became_peered":"2023-10-16T12:26:24.289677+0000","last_unstale":"2023-10-16T12:26:24.294823+0000","last_undegraded":"2023-10-16T12:26:24.294823+0000","last_fullsized":"2023-10-16T12:26:24.294823+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:38:06.705069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.288407+0000","last_change":"2023-10-16T12:26:23.262042+0000","last_active":"2023-10-16T12:26:25.288407+0000","last_peered":"2023-10-16T12:26:25.288407+0000","last_clean":"2023-10-16T12:26:25.288407+0000","last_became_active":"2023-10-16T12:26:23.261710+0000","last_became_peered":"2023-10-16T12:26:23.261710+0000","last_unstale":"2023-10-16T12:26:25.288407+0000","last_undegraded":"2023-10-16T12:26:25.288407+0000","last_fullsized":"2023-10-16T12:26:25.288407+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_clean_scrub_stamp":"2023-10-16T12:26:22.237327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:55:24.362404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.288134+0000","last_change":"2023-10-16T12:26:24.285641+0000","last_active":"2023-10-16T12:26:25.288134+0000","last_peered":"2023-10-16T12:26:25.288134+0000","last_clean":"2023-10-16T12:26:25.288134+0000","last_became_active":"2023-10-16T12:26:24.285341+0000","last_became_peered":"2023-10-16T12:26:24.285341+0000","last_unstale":"2023-10-16T12:26:25.288134+0000","last_undegraded":"2023-10-16T12:26:25.288134+0000","last_fullsized":"2023-10-16T12:26:25.288134+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:55:25.366467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"84'2","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.288703+0000","last_change":"2023-10-16T12:26:23.258309+0000","last_active":"2023-10-16T12:26:25.288703+0000","last_peered":"2023-10-16T12:26:25.288703+0000","last_clean":"2023-10-16T12:26:25.288703+0000","last_became_active":"2023-10-16T12:26:23.258013+0000","last_became_peered":"2023-10-16T12:26:23.258013+0000","last_unstale":"2023-10-16T12:26:25.288703+0000","last_undegraded":"2023-10-16T12:26:25.288703+0000","last_fullsized":"2023-10-16T12:26:25.288703+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_clean_scrub_stamp":"2023-10-16T12:26:22.237327+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:00:36.231552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.289658+0000","last_change":"2023-10-16T12:26:24.286194+0000","last_active":"2023-10-16T12:26:24.289658+0000","last_peered":"2023-10-16T12:26:24.289658+0000","last_clean":"2023-10-16T12:26:24.289658+0000","last_became_active":"2023-10-16T12:26:24.285745+0000","last_became_peered":"2023-10-16T12:26:24.285745+0000","last_unstale":"2023-10-16T12:26:24.289658+0000","last_undegraded":"2023-10-16T12:26:24.289658+0000","last_fullsized":"2023-10-16T12:26:24.289658+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:32:19.269409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.281199+0000","last_change":"2023-10-16T12:26:23.267368+0000","last_active":"2023-10-16T12:26:24.281199+0000","last_peered":"2023-10-16T12:26:24.281199+0000","last_clean":"2023-10-16T12:26:24.281199+0000","last_became_active":"2023-10-16T12:26:23.267067+0000","last_became_peered":"2023-10-16T12:26:23.267067+0000","last_unstale":"2023-10-16T12:26:24.281199+0000","last_undegraded":"2023-10-16T12:26:24.281199+0000","last_fullsized":"2023-10-16T12:26:24.281199+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_clean_scrub_stamp":"2023-10-16T12:26:22.237327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:24:48.895339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.277754+0000","last_change":"2023-10-16T12:26:24.276796+0000","last_active":"2023-10-16T12:26:24.277754+0000","last_peered":"2023-10-16T12:26:24.277754+0000","last_clean":"2023-10-16T12:26:24.277754+0000","last_became_active":"2023-10-16T12:26:24.276445+0000","last_became_peered":"2023-10-16T12:26:24.276445+0000","last_unstale":"2023-10-16T12:26:24.277754+0000","last_undegraded":"2023-10-16T12:26:24.277754+0000","last_fullsized":"2023-10-16T12:26:24.277754+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:56:07.516751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.287973+0000","last_change":"2023-10-16T12:26:23.252571+0000","last_active":"2023-10-16T12:26:25.287973+0000","last_peered":"2023-10-16T12:26:25.287973+0000","last_clean":"2023-10-16T12:26:25.287973+0000","last_became_active":"2023-10-16T12:26:23.252277+0000","last_became_peered":"2023-10-16T12:26:23.252277+0000","last_unstale":"2023-10-16T12:26:25.287973+0000","last_undegraded":"2023-10-16T12:26:25.287973+0000","last_fullsized":"2023-10-16T12:26:25.287973+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_clean_scrub_stamp":"2023-10-16T12:26:22.237327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:21:45.893418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.288623+0000","last_change":"2023-10-16T12:26:24.297532+0000","last_active":"2023-10-16T12:26:25.288623+0000","last_peered":"2023-10-16T12:26:25.288623+0000","last_clean":"2023-10-16T12:26:25.288623+0000","last_became_active":"2023-10-16T12:26:24.294140+0000","last_became_peered":"2023-10-16T12:26:24.294140+0000","last_unstale":"2023-10-16T12:26:25.288623+0000","last_undegraded":"2023-10-16T12:26:25.288623+0000","last_fullsized":"2023-10-16T12:26:25.288623+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:51:19.596991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"84'2","reported_seq":13,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.308957+0000","last_change":"2023-10-16T12:26:23.267218+0000","last_active":"2023-10-16T12:26:24.308957+0000","last_peered":"2023-10-16T12:26:24.308957+0000","last_clean":"2023-10-16T12:26:24.308957+0000","last_became_active":"2023-10-16T12:26:23.266912+0000","last_became_peered":"2023-10-16T12:26:23.266912+0000","last_unstale":"2023-10-16T12:26:24.308957+0000","last_undegraded":"2023-10-16T12:26:24.308957+0000","last_fullsized":"2023-10-16T12:26:24.308957+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_clean_scrub_stamp":"2023-10-16T12:26:22.237327+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:24:50.857165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.302789+0000","last_change":"2023-10-16T12:26:24.297628+0000","last_active":"2023-10-16T12:26:24.302789+0000","last_peered":"2023-10-16T12:26:24.302789+0000","last_clean":"2023-10-16T12:26:24.302789+0000","last_became_active":"2023-10-16T12:26:24.297296+0000","last_became_peered":"2023-10-16T12:26:24.297296+0000","last_unstale":"2023-10-16T12:26:24.302789+0000","last_undegraded":"2023-10-16T12:26:24.302789+0000","last_fullsized":"2023-10-16T12:26:24.302789+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:59:42.505808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.281373+0000","last_change":"2023-10-16T12:26:23.266323+0000","last_active":"2023-10-16T12:26:24.281373+0000","last_peered":"2023-10-16T12:26:24.281373+0000","last_clean":"2023-10-16T12:26:24.281373+0000","last_became_active":"2023-10-16T12:26:23.265989+0000","last_became_peered":"2023-10-16T12:26:23.265989+0000","last_unstale":"2023-10-16T12:26:24.281373+0000","last_undegraded":"2023-10-16T12:26:24.281373+0000","last_fullsized":"2023-10-16T12:26:24.281373+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_clean_scrub_stamp":"2023-10-16T12:26:22.237327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:44:34.923567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.295094+0000","last_change":"2023-10-16T12:26:24.289242+0000","last_active":"2023-10-16T12:26:24.295094+0000","last_peered":"2023-10-16T12:26:24.295094+0000","last_clean":"2023-10-16T12:26:24.295094+0000","last_became_active":"2023-10-16T12:26:24.288955+0000","last_became_peered":"2023-10-16T12:26:24.288955+0000","last_unstale":"2023-10-16T12:26:24.295094+0000","last_undegraded":"2023-10-16T12:26:24.295094+0000","last_fullsized":"2023-10-16T12:26:24.295094+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:50:59.014744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.288675+0000","last_change":"2023-10-16T12:26:23.267396+0000","last_active":"2023-10-16T12:26:25.288675+0000","last_peered":"2023-10-16T12:26:25.288675+0000","last_clean":"2023-10-16T12:26:25.288675+0000","last_became_active":"2023-10-16T12:26:23.267099+0000","last_became_peered":"2023-10-16T12:26:23.267099+0000","last_unstale":"2023-10-16T12:26:25.288675+0000","last_undegraded":"2023-10-16T12:26:25.288675+0000","last_fullsized":"2023-10-16T12:26:25.288675+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_clean_scrub_stamp":"2023-10-16T12:26:22.237327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:40:04.791545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.294849+0000","last_change":"2023-10-16T12:26:24.290108+0000","last_active":"2023-10-16T12:26:24.294849+0000","last_peered":"2023-10-16T12:26:24.294849+0000","last_clean":"2023-10-16T12:26:24.294849+0000","last_became_active":"2023-10-16T12:26:24.289811+0000","last_became_peered":"2023-10-16T12:26:24.289811+0000","last_unstale":"2023-10-16T12:26:24.294849+0000","last_undegraded":"2023-10-16T12:26:24.294849+0000","last_fullsized":"2023-10-16T12:26:24.294849+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:43:17.002196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.279814+0000","last_change":"2023-10-16T12:26:23.268364+0000","last_active":"2023-10-16T12:26:24.279814+0000","last_peered":"2023-10-16T12:26:24.279814+0000","last_clean":"2023-10-16T12:26:24.279814+0000","last_became_active":"2023-10-16T12:26:23.265514+0000","last_became_peered":"2023-10-16T12:26:23.265514+0000","last_unstale":"2023-10-16T12:26:24.279814+0000","last_undegraded":"2023-10-16T12:26:24.279814+0000","last_fullsized":"2023-10-16T12:26:24.279814+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_clean_scrub_stamp":"2023-10-16T12:26:22.237327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:53:21.641122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.289743+0000","last_change":"2023-10-16T12:26:24.283228+0000","last_active":"2023-10-16T12:26:24.289743+0000","last_peered":"2023-10-16T12:26:24.289743+0000","last_clean":"2023-10-16T12:26:24.289743+0000","last_became_active":"2023-10-16T12:26:24.282921+0000","last_became_peered":"2023-10-16T12:26:24.282921+0000","last_unstale":"2023-10-16T12:26:24.289743+0000","last_undegraded":"2023-10-16T12:26:24.289743+0000","last_fullsized":"2023-10-16T12:26:24.289743+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:12:45.626596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.569267+0000","last_change":"2023-10-16T12:26:23.259640+0000","last_active":"2023-10-16T12:26:25.569267+0000","last_peered":"2023-10-16T12:26:25.569267+0000","last_clean":"2023-10-16T12:26:25.569267+0000","last_became_active":"2023-10-16T12:26:23.259351+0000","last_became_peered":"2023-10-16T12:26:23.259351+0000","last_unstale":"2023-10-16T12:26:25.569267+0000","last_undegraded":"2023-10-16T12:26:25.569267+0000","last_fullsized":"2023-10-16T12:26:25.569267+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_clean_scrub_stamp":"2023-10-16T12:26:22.237327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:33:19.198155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.595509+0000","last_change":"2023-10-16T12:26:24.297109+0000","last_active":"2023-10-16T12:26:25.595509+0000","last_peered":"2023-10-16T12:26:25.595509+0000","last_clean":"2023-10-16T12:26:25.595509+0000","last_became_active":"2023-10-16T12:26:24.296614+0000","last_became_peered":"2023-10-16T12:26:24.296614+0000","last_unstale":"2023-10-16T12:26:25.595509+0000","last_undegraded":"2023-10-16T12:26:25.595509+0000","last_fullsized":"2023-10-16T12:26:25.595509+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:28:47.753397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.288018+0000","last_change":"2023-10-16T12:26:23.267165+0000","last_active":"2023-10-16T12:26:25.288018+0000","last_peered":"2023-10-16T12:26:25.288018+0000","last_clean":"2023-10-16T12:26:25.288018+0000","last_became_active":"2023-10-16T12:26:23.266875+0000","last_became_peered":"2023-10-16T12:26:23.266875+0000","last_unstale":"2023-10-16T12:26:25.288018+0000","last_undegraded":"2023-10-16T12:26:25.288018+0000","last_fullsized":"2023-10-16T12:26:25.288018+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_clean_scrub_stamp":"2023-10-16T12:26:22.237327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:34:46.883916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.304828+0000","last_change":"2023-10-16T12:26:24.299736+0000","last_active":"2023-10-16T12:26:24.304828+0000","last_peered":"2023-10-16T12:26:24.304828+0000","last_clean":"2023-10-16T12:26:24.304828+0000","last_became_active":"2023-10-16T12:26:24.299445+0000","last_became_peered":"2023-10-16T12:26:24.299445+0000","last_unstale":"2023-10-16T12:26:24.304828+0000","last_undegraded":"2023-10-16T12:26:24.304828+0000","last_fullsized":"2023-10-16T12:26:24.304828+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:53:28.283879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.288065+0000","last_change":"2023-10-16T12:26:23.263113+0000","last_active":"2023-10-16T12:26:25.288065+0000","last_peered":"2023-10-16T12:26:25.288065+0000","last_clean":"2023-10-16T12:26:25.288065+0000","last_became_active":"2023-10-16T12:26:23.262795+0000","last_became_peered":"2023-10-16T12:26:23.262795+0000","last_unstale":"2023-10-16T12:26:25.288065+0000","last_undegraded":"2023-10-16T12:26:25.288065+0000","last_fullsized":"2023-10-16T12:26:25.288065+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_clean_scrub_stamp":"2023-10-16T12:26:22.237327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:41:42.968510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.292380+0000","last_change":"2023-10-16T12:26:24.290731+0000","last_active":"2023-10-16T12:26:24.292380+0000","last_peered":"2023-10-16T12:26:24.292380+0000","last_clean":"2023-10-16T12:26:24.292380+0000","last_became_active":"2023-10-16T12:26:24.290417+0000","last_became_peered":"2023-10-16T12:26:24.290417+0000","last_unstale":"2023-10-16T12:26:24.292380+0000","last_undegraded":"2023-10-16T12:26:24.292380+0000","last_fullsized":"2023-10-16T12:26:24.292380+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:05:21.201046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.278612+0000","last_change":"2023-10-16T12:26:23.262045+0000","last_active":"2023-10-16T12:26:24.278612+0000","last_peered":"2023-10-16T12:26:24.278612+0000","last_clean":"2023-10-16T12:26:24.278612+0000","last_became_active":"2023-10-16T12:26:23.261702+0000","last_became_peered":"2023-10-16T12:26:23.261702+0000","last_unstale":"2023-10-16T12:26:24.278612+0000","last_undegraded":"2023-10-16T12:26:24.278612+0000","last_fullsized":"2023-10-16T12:26:24.278612+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_clean_scrub_stamp":"2023-10-16T12:26:22.237327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:28:49.990521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.295053+0000","last_change":"2023-10-16T12:26:24.292589+0000","last_active":"2023-10-16T12:26:24.295053+0000","last_peered":"2023-10-16T12:26:24.295053+0000","last_clean":"2023-10-16T12:26:24.295053+0000","last_became_active":"2023-10-16T12:26:24.292235+0000","last_became_peered":"2023-10-16T12:26:24.292235+0000","last_unstale":"2023-10-16T12:26:24.295053+0000","last_undegraded":"2023-10-16T12:26:24.295053+0000","last_fullsized":"2023-10-16T12:26:24.295053+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:38:04.765408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.287747+0000","last_change":"2023-10-16T12:26:23.265387+0000","last_active":"2023-10-16T12:26:25.287747+0000","last_peered":"2023-10-16T12:26:25.287747+0000","last_clean":"2023-10-16T12:26:25.287747+0000","last_became_active":"2023-10-16T12:26:23.265077+0000","last_became_peered":"2023-10-16T12:26:23.265077+0000","last_unstale":"2023-10-16T12:26:25.287747+0000","last_undegraded":"2023-10-16T12:26:25.287747+0000","last_fullsized":"2023-10-16T12:26:25.287747+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_clean_scrub_stamp":"2023-10-16T12:26:22.237327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:24:04.496264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"75'161","reported_seq":385,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.283779+0000","last_change":"2023-10-16T12:10:34.746820+0000","last_active":"2023-10-16T12:26:24.283779+0000","last_peered":"2023-10-16T12:26:24.283779+0000","last_clean":"2023-10-16T12:26:24.283779+0000","last_became_active":"2023-10-16T12:10:34.745574+0000","last_became_peered":"2023-10-16T12:10:34.745574+0000","last_unstale":"2023-10-16T12:26:24.283779+0000","last_undegraded":"2023-10-16T12:26:24.283779+0000","last_fullsized":"2023-10-16T12:26:24.283779+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_clean_scrub_stamp":"2023-10-16T12:10:33.577111+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:14.461986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575041,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59156,"kb_used_data":1604,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312684,"statfs":{"total":96636764160,"available":96576188416,"internally_reserved":0,"allocated":1642496,"data_stored":1105024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58915275},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:26:05 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.59699999999999998,"15min":0.58499999999999996},"min":{"1min":0.437,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.59399999999999997,"15min":0.58199999999999996},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.312},"max":{"1min":0.85299999999999998,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.78100000000000003}]},{"osd":1,"last update":"Mon Oct 16 12:26:05 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.60399999999999998,"15min":0.59999999999999998},"min":{"1min":0.36099999999999999,"5min":0.35999999999999999,"15min":0.247},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.62,"15min":0.60099999999999998},"min":{"1min":0.45600000000000002,"5min":0.33200000000000002,"15min":0.32400000000000001},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.64100000000000001}]},{"osd":2,"last update":"Mon Oct 16 12:26:05 2023","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.67600000000000005,"15min":0.65600000000000003},"min":{"1min":0.53700000000000003,"5min":0.40699999999999997,"15min":0.36099999999999999},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.68799999999999994,"15min":0.66600000000000004},"min":{"1min":0.495,"5min":0.376,"15min":0.36699999999999999},"max":{"1min":0.94599999999999995,"5min":0.94799999999999995,"15min":0.96699999999999997},"last":0.53800000000000003}]},{"osd":3,"last update":"Mon Oct 16 12:26:05 2023","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.63,"15min":0.61899999999999999},"min":{"1min":0.43099999999999999,"5min":0.35499999999999998,"15min":0.34899999999999998},"max":{"1min":0.91200000000000003,"5min":0.92200000000000004,"15min":1.1000000000000001},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.65700000000000003,"15min":0.63900000000000001},"min":{"1min":0.52200000000000002,"5min":0.376,"15min":0.376},"max":{"1min":0.996,"5min":1.2509999999999999,"15min":1.2509999999999999},"last":0.72299999999999998}]},{"osd":4,"last update":"Mon Oct 16 12:26:05 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53800000000000003,"15min":0.53100000000000003},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.28499999999999998},"max":{"1min":0.72699999999999998,"5min":1.383,"15min":1.383},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.625,"15min":0.60699999999999998},"min":{"1min":0.40799999999999997,"5min":0.36299999999999999,"15min":0.307},"max":{"1min":0.86599999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.59199999999999997}]},{"osd":5,"last update":"Mon Oct 16 12:26:05 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.55800000000000005,"15min":0.54300000000000004},"min":{"1min":0.40000000000000002,"5min":0.314,"15min":0.29799999999999999},"max":{"1min":0.84299999999999997,"5min":1.3109999999999999,"15min":1.3109999999999999},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.59399999999999997,"15min":0.58699999999999997},"min":{"1min":0.41799999999999998,"5min":0.33500000000000002,"15min":0.32400000000000001},"max":{"1min":0.83899999999999997,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":0.50900000000000001}]},{"osd":6,"last update":"Mon Oct 16 12:26:05 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.63500000000000001,"15min":0.621},"min":{"1min":0.44700000000000001,"5min":0.39300000000000002,"15min":0.36399999999999999},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.65600000000000003,"15min":0.63100000000000001},"min":{"1min":0.443,"5min":0.34599999999999997,"15min":0.32300000000000001},"max":{"1min":0.88300000000000001,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.66100000000000003}]}]},{"osd":6,"up_from":13,"seq":55834575039,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32180,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339660,"statfs":{"total":96636764160,"available":96603811840,"internally_reserved":0,"allocated":1024000,"data_stored":507266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:25:58 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.60099999999999998,"15min":0.628},"min":{"1min":0.34200000000000003,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.88200000000000001,"5min":1.196,"15min":5.9349999999999996},"last":0.628},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60499999999999998,"15min":0.64000000000000001},"min":{"1min":0.44900000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.81799999999999995,"5min":0.997,"15min":5.9489999999999998},"last":0.59199999999999997}]},{"osd":1,"last update":"Mon Oct 16 12:25:58 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.55500000000000005,"15min":0.57499999999999996},"min":{"1min":0.40200000000000002,"5min":0.32800000000000001,"15min":0.29399999999999998},"max":{"1min":0.71199999999999997,"5min":0.83899999999999997,"15min":0.875},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57899999999999996,"15min":0.59199999999999997},"min":{"1min":0.44500000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.83099999999999996,"5min":1.091,"15min":1.091},"last":0.48299999999999998}]},{"osd":2,"last update":"Mon Oct 16 12:25:58 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.57799999999999996,"15min":0.61799999999999999},"min":{"1min":0.41699999999999998,"5min":0.36199999999999999,"15min":0.34499999999999997},"max":{"1min":0.69199999999999995,"5min":0.91200000000000003,"15min":8.0489999999999995},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.57699999999999996,"15min":0.60899999999999999},"min":{"1min":0.41099999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.76300000000000001,"5min":1.1539999999999999,"15min":2.855},"last":0.44900000000000001}]},{"osd":3,"last update":"Mon Oct 16 12:25:58 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.60599999999999998,"15min":0.629},"min":{"1min":0.45500000000000002,"5min":0.34200000000000003,"15min":0.307},"max":{"1min":0.85399999999999998,"5min":0.93999999999999995,"15min":1.083},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67400000000000004,"15min":0.67900000000000005},"min":{"1min":0.48099999999999998,"5min":0.40899999999999997,"15min":0.36599999999999999},"max":{"1min":0.94499999999999995,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.55400000000000005}]},{"osd":4,"last update":"Mon Oct 16 12:25:58 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.52800000000000002,"15min":0.52000000000000002},"min":{"1min":0.33900000000000002,"5min":0.308,"15min":0.29899999999999999},"max":{"1min":0.83399999999999996,"5min":0.85599999999999998,"15min":0.88200000000000001},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.60299999999999998,"15min":0.59399999999999997},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.30199999999999999},"max":{"1min":0.78500000000000003,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.65000000000000002}]},{"osd":5,"last update":"Mon Oct 16 12:25:58 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59099999999999997,"15min":0.57799999999999996},"min":{"1min":0.39900000000000002,"5min":0.34799999999999998,"15min":0.30099999999999999},"max":{"1min":0.80100000000000005,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63400000000000001,"15min":0.624},"min":{"1min":0.39400000000000002,"5min":0.35099999999999998,"15min":0.311},"max":{"1min":0.81599999999999995,"5min":1.3540000000000001,"15min":1.3540000000000001},"last":0.70099999999999996}]},{"osd":7,"last update":"Mon Oct 16 12:25:58 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59499999999999997,"15min":0.57499999999999996},"min":{"1min":0.36499999999999999,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.86599999999999999,"5min":1.29,"15min":1.29},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.62,"5min":0.625,"15min":0.61799999999999999},"min":{"1min":0.38800000000000001,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.84399999999999997,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.61299999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575039,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32204,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339636,"statfs":{"total":96636764160,"available":96603787264,"internally_reserved":0,"allocated":1048576,"data_stored":509608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":31914441},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:26:00 2023","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.373,"15min":0.378},"min":{"1min":0.28199999999999997,"5min":0.21299999999999999,"15min":0.186},"max":{"1min":0.52700000000000002,"5min":0.63600000000000001,"15min":0.66000000000000003},"last":0.314},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.45900000000000002,"15min":0.46000000000000002},"min":{"1min":0.33000000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.65700000000000003,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.28399999999999997}]},{"osd":2,"last update":"Mon Oct 16 12:26:00 2023","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.41399999999999998,"15min":0.42899999999999999},"min":{"1min":0.28699999999999998,"5min":0.25,"15min":0.215},"max":{"1min":0.61499999999999999,"5min":0.69199999999999995,"15min":0.93000000000000005},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.46300000000000002,"15min":0.46400000000000002},"min":{"1min":0.32300000000000001,"5min":0.28199999999999997,"15min":0.24399999999999999},"max":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.98099999999999998},"last":0.372}]},{"osd":3,"last update":"Mon Oct 16 12:26:00 2023","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.40200000000000002,"15min":0.40400000000000003},"min":{"1min":0.32500000000000001,"5min":0.25900000000000001,"15min":0.23599999999999999},"max":{"1min":0.69599999999999995,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.245},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.44,"15min":0.45700000000000002},"min":{"1min":0.33800000000000002,"5min":0.252,"15min":0.22900000000000001},"max":{"1min":0.60199999999999998,"5min":0.91200000000000003,"15min":1.127},"last":0.26100000000000001}]},{"osd":4,"last update":"Mon Oct 16 12:26:00 2023","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.61799999999999999,"15min":0.63100000000000001},"min":{"1min":0.44900000000000001,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.76600000000000001,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.749},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.628,"15min":0.63900000000000001},"min":{"1min":0.46999999999999997,"5min":0.39100000000000001,"15min":0.34699999999999998},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":1.089},"last":0.65000000000000002}]},{"osd":5,"last update":"Mon Oct 16 12:26:00 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.63700000000000001,"15min":0.66600000000000004},"min":{"1min":0.47799999999999998,"5min":0.38300000000000001,"15min":0.28499999999999998},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.68100000000000005,"15min":0.68600000000000005},"min":{"1min":0.47399999999999998,"5min":0.38900000000000001,"15min":0.32500000000000001},"max":{"1min":0.98999999999999999,"5min":1.0620000000000001,"15min":1.0649999999999999},"last":0.71399999999999997}]},{"osd":6,"last update":"Mon Oct 16 12:26:00 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.63600000000000001,"15min":0.65300000000000002},"min":{"1min":0.502,"5min":0.378,"15min":0.378},"max":{"1min":0.876,"5min":0.94999999999999996,"15min":1.0129999999999999},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.65800000000000003,"15min":0.66200000000000003},"min":{"1min":0.51500000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":1.0609999999999999},"last":0.63700000000000001}]},{"osd":7,"last update":"Mon Oct 16 12:26:00 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.68899999999999995,"15min":0.68799999999999994},"min":{"1min":0.49299999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.0469999999999999,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.68300000000000005,"15min":0.70299999999999996},"min":{"1min":0.60699999999999998,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":1.167},"last":0.68500000000000005}]}]},{"osd":0,"up_from":13,"seq":55834575040,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36936,"kb_used_data":1588,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334904,"statfs":{"total":96636764160,"available":96598941696,"internally_reserved":0,"allocated":1626112,"data_stored":1104248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 12:25:54 2023","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.38900000000000001,"15min":0.38600000000000001},"min":{"1min":0.307,"5min":0.223,"15min":0.223},"max":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.73699999999999999},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.41699999999999998,"15min":0.41799999999999998},"min":{"1min":0.27300000000000002,"5min":0.24299999999999999,"15min":0.23200000000000001},"max":{"1min":0.64700000000000002,"5min":0.71799999999999997,"15min":0.73099999999999998},"last":0.34100000000000003}]},{"osd":2,"last update":"Mon Oct 16 12:25:54 2023","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.42799999999999999,"15min":0.42499999999999999},"min":{"1min":0.32600000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.628,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.33300000000000002,"5min":0.29599999999999999,"15min":0.27900000000000003},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.40899999999999997}]},{"osd":3,"last update":"Mon Oct 16 12:25:54 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.432,"15min":0.42099999999999999},"min":{"1min":0.372,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.57299999999999995,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.38},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.48199999999999998,"15min":0.48499999999999999},"min":{"1min":0.37,"5min":0.29099999999999998,"15min":0.27600000000000002},"max":{"1min":0.66400000000000003,"5min":0.68500000000000005,"15min":0.878},"last":0.32500000000000001}]},{"osd":4,"last update":"Mon Oct 16 12:25:54 2023","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.65300000000000002,"15min":0.64800000000000002},"min":{"1min":0.45100000000000001,"5min":0.377,"15min":0.36699999999999999},"max":{"1min":0.85999999999999999,"5min":0.90700000000000003,"15min":1.0960000000000001},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64500000000000002,"15min":0.65500000000000003},"min":{"1min":0.52600000000000002,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.81499999999999995,"5min":0.94999999999999996,"15min":1.1719999999999999},"last":0.59499999999999997}]},{"osd":5,"last update":"Mon Oct 16 12:25:54 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.65100000000000002,"15min":0.65000000000000002},"min":{"1min":0.60199999999999998,"5min":0.374,"15min":0.34399999999999997},"max":{"1min":0.84499999999999997,"5min":1,"15min":1.127},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.67000000000000004,"15min":0.67900000000000005},"min":{"1min":0.54500000000000004,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.874,"5min":0.92900000000000005,"15min":1.1160000000000001},"last":0.58099999999999996}]},{"osd":6,"last update":"Mon Oct 16 12:25:54 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.67400000000000004,"15min":0.68100000000000005},"min":{"1min":0.56699999999999995,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.91400000000000003,"5min":0.98599999999999999,"15min":1.079},"last":0.627},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.67400000000000004,"15min":0.68500000000000005},"min":{"1min":0.56299999999999994,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.92100000000000004,"5min":0.95899999999999996,"15min":1.1499999999999999},"last":0.48999999999999999}]},{"osd":7,"last update":"Mon Oct 16 12:25:54 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.68400000000000005,"15min":0.69599999999999995},"min":{"1min":0.61399999999999999,"5min":0.41499999999999998,"15min":0.40699999999999997},"max":{"1min":0.88800000000000001,"5min":0.999,"15min":1.0349999999999999},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.67300000000000004,"15min":0.71199999999999997},"min":{"1min":0.56000000000000005,"5min":0.41899999999999998,"15min":0.41299999999999998},"max":{"1min":0.84899999999999998,"5min":1.0329999999999999,"15min":6.7300000000000004},"last":0.50800000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575040,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58544,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313296,"statfs":{"total":96636764160,"available":96576815104,"internally_reserved":0,"allocated":1019904,"data_stored":507232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:25:55 2023","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.38,"15min":0.38200000000000001},"min":{"1min":0.29699999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.48299999999999998,"5min":0.59499999999999997,"15min":0.73799999999999999},"last":0.378},{"interface":"front","average":{"1min":0.498,"5min":0.45800000000000002,"15min":0.44900000000000001},"min":{"1min":0.379,"5min":0.29299999999999998,"15min":0.26400000000000001},"max":{"1min":0.66500000000000004,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.31900000000000001}]},{"osd":1,"last update":"Mon Oct 16 12:25:55 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.46899999999999997,"15min":0.46600000000000003},"min":{"1min":0.39000000000000001,"5min":0.25600000000000001,"15min":0.254},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.439,"15min":0.436},"min":{"1min":0.34599999999999997,"5min":0.27700000000000002,"15min":0.246},"max":{"1min":0.59199999999999997,"5min":0.69699999999999995,"15min":0.76500000000000001},"last":0.442}]},{"osd":3,"last update":"Mon Oct 16 12:25:55 2023","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.41899999999999998,"15min":0.41499999999999998},"min":{"1min":0.31900000000000001,"5min":0.27700000000000002,"15min":0.26300000000000001},"max":{"1min":0.63600000000000001,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.51800000000000002,"15min":0.51600000000000001},"min":{"1min":0.44400000000000001,"5min":0.318,"15min":0.246},"max":{"1min":0.78000000000000003,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.36599999999999999}]},{"osd":4,"last update":"Mon Oct 16 12:25:55 2023","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.68999999999999995,"15min":0.68600000000000005},"min":{"1min":0.56499999999999995,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.91100000000000003,"5min":1.3839999999999999,"15min":1.3839999999999999},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.69699999999999995,"15min":0.68600000000000005},"min":{"1min":0.54900000000000004,"5min":0.46400000000000002,"15min":0.36699999999999999},"max":{"1min":0.97599999999999998,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.51800000000000002}]},{"osd":5,"last update":"Mon Oct 16 12:25:55 2023","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.67700000000000005,"15min":0.66600000000000004},"min":{"1min":0.56499999999999995,"5min":0.40200000000000002,"15min":0.312},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":0.504},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.67800000000000005,"15min":0.66700000000000004},"min":{"1min":0.51800000000000002,"5min":0.41299999999999998,"15min":0.317},"max":{"1min":1.0089999999999999,"5min":1.331,"15min":1.331},"last":0.60599999999999998}]},{"osd":6,"last update":"Mon Oct 16 12:25:55 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.65900000000000003,"15min":0.64400000000000002},"min":{"1min":0.497,"5min":0.439,"15min":0.34699999999999998},"max":{"1min":0.99299999999999999,"5min":1.036,"15min":1.046},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.70299999999999996,"15min":0.68999999999999995},"min":{"1min":0.63,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.032,"5min":1.032,"15min":1.0389999999999999},"last":0.54600000000000004}]},{"osd":7,"last update":"Mon Oct 16 12:25:55 2023","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.68500000000000005,"15min":0.68200000000000005},"min":{"1min":0.53300000000000003,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.96399999999999997,"5min":1.016,"15min":1.026},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.754,"5min":0.69999999999999996,"15min":0.69399999999999995},"min":{"1min":0.53800000000000003,"5min":0.442,"15min":0.40600000000000003},"max":{"1min":1.145,"5min":1.145,"15min":1.145},"last":0.59099999999999997}]}]},{"osd":3,"up_from":13,"seq":55834575039,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36340,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335500,"statfs":{"total":96636764160,"available":96599552000,"internally_reserved":0,"allocated":1019904,"data_stored":507232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:26:02 2023","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.40100000000000002,"15min":0.39400000000000002},"min":{"1min":0.29499999999999998,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.435,"5min":0.41199999999999998,"15min":0.40500000000000003},"min":{"1min":0.28999999999999998,"5min":0.255,"15min":0.22600000000000001},"max":{"1min":0.61899999999999999,"5min":0.63100000000000001,"15min":0.73099999999999998},"last":0.42799999999999999}]},{"osd":1,"last update":"Mon Oct 16 12:26:02 2023","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.41999999999999998,"15min":0.40999999999999998},"min":{"1min":0.29499999999999998,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.64100000000000001,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.374},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.46300000000000002,"15min":0.45800000000000002},"min":{"1min":0.38900000000000001,"5min":0.25800000000000001,"15min":0.25700000000000001},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.747},"last":0.45600000000000002}]},{"osd":2,"last update":"Mon Oct 16 12:26:02 2023","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.47399999999999998,"15min":0.46800000000000003},"min":{"1min":0.39600000000000002,"5min":0.307,"15min":0.26900000000000002},"max":{"1min":0.63700000000000001,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.503,"15min":0.49199999999999999},"min":{"1min":0.42199999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.60299999999999998,"5min":0.752,"15min":0.752},"last":0.54200000000000004}]},{"osd":4,"last update":"Mon Oct 16 12:26:02 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.63200000000000001,"15min":0.64500000000000002},"min":{"1min":0.49399999999999999,"5min":0.39100000000000001,"15min":0.377},"max":{"1min":0.81699999999999995,"5min":0.92700000000000005,"15min":1.1539999999999999},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.65400000000000003,"15min":0.66800000000000004},"min":{"1min":0.55700000000000005,"5min":0.44900000000000001,"15min":0.40200000000000002},"max":{"1min":0.90600000000000003,"5min":1.0409999999999999,"15min":1.234},"last":0.53100000000000003}]},{"osd":5,"last update":"Mon Oct 16 12:26:02 2023","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.68600000000000005,"15min":0.68300000000000005},"min":{"1min":0.48899999999999999,"5min":0.443,"15min":0.41199999999999998},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.194},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.65600000000000003,"15min":0.66700000000000004},"min":{"1min":0.56000000000000005,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":1.202},"last":0.63}]},{"osd":6,"last update":"Mon Oct 16 12:26:02 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.67500000000000004,"15min":0.67400000000000004},"min":{"1min":0.53300000000000003,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.94199999999999995,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.75,"5min":0.70599999999999996,"15min":0.70699999999999996},"min":{"1min":0.60099999999999998,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.1559999999999999},"last":0.67700000000000005}]},{"osd":7,"last update":"Mon Oct 16 12:26:02 2023","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.68000000000000005,"15min":0.68899999999999995},"min":{"1min":0.499,"5min":0.435,"15min":0.38},"max":{"1min":0.92300000000000004,"5min":1.0189999999999999,"15min":1.2430000000000001},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.70999999999999996,"15min":0.70699999999999996},"min":{"1min":0.48899999999999999,"5min":0.45200000000000001,"15min":0.42699999999999999},"max":{"1min":0.99099999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.65500000000000003}]}]},{"osd":4,"up_from":13,"seq":55834575038,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32200,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339640,"statfs":{"total":96636764160,"available":96603791360,"internally_reserved":0,"allocated":1044480,"data_stored":515492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:26:00 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.59799999999999998,"15min":0.59199999999999997},"min":{"1min":0.30399999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.78300000000000003,"5min":0.85099999999999998,"15min":1.171},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.62,"5min":0.58699999999999997,"15min":0.60199999999999998},"min":{"1min":0.42399999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.76400000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.57999999999999996}]},{"osd":1,"last update":"Mon Oct 16 12:26:00 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.59299999999999997,"15min":0.60999999999999999},"min":{"1min":0.433,"5min":0.35499999999999998,"15min":0.28499999999999998},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.92600000000000005},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.622,"5min":0.61499999999999999,"15min":0.621},"min":{"1min":0.374,"5min":0.374,"15min":0.33600000000000002},"max":{"1min":0.73899999999999999,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.66900000000000004}]},{"osd":2,"last update":"Mon Oct 16 12:26:00 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.61199999999999999,"15min":0.625},"min":{"1min":0.39900000000000002,"5min":0.35399999999999998,"15min":0.32000000000000001},"max":{"1min":0.77000000000000002,"5min":0.93000000000000005,"15min":1.012},"last":0.434},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61299999999999999,"15min":0.627},"min":{"1min":0.47899999999999998,"5min":0.38800000000000001,"15min":0.34100000000000003},"max":{"1min":0.74099999999999999,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.71899999999999997}]},{"osd":3,"last update":"Mon Oct 16 12:26:00 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.59599999999999997,"15min":0.61299999999999999},"min":{"1min":0.46300000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.70699999999999996,"5min":0.79500000000000004,"15min":1.3779999999999999},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63,"15min":0.64100000000000001},"min":{"1min":0.41199999999999998,"5min":0.38800000000000001,"15min":0.37},"max":{"1min":0.77800000000000002,"5min":0.89800000000000002,"15min":0.97199999999999998},"last":0.63800000000000001}]},{"osd":5,"last update":"Mon Oct 16 12:26:00 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48699999999999999,"15min":0.5},"min":{"1min":0.34799999999999998,"5min":0.29799999999999999,"15min":0.27600000000000002},"max":{"1min":0.68100000000000005,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.504,"5min":0.51400000000000001,"15min":0.52500000000000002},"min":{"1min":0.371,"5min":0.316,"15min":0.29599999999999999},"max":{"1min":0.72599999999999998,"5min":0.754,"15min":0.94199999999999995},"last":0.40600000000000003}]},{"osd":6,"last update":"Mon Oct 16 12:26:00 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.54600000000000004,"15min":0.55900000000000005},"min":{"1min":0.39800000000000002,"5min":0.34300000000000003,"15min":0.28699999999999998},"max":{"1min":0.68700000000000006,"5min":0.88400000000000001,"15min":0.89800000000000002},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.63700000000000001,"15min":0.63600000000000001},"min":{"1min":0.45000000000000001,"5min":0.34799999999999998,"15min":0.32000000000000001},"max":{"1min":0.84099999999999997,"5min":0.99299999999999999,"15min":1.1180000000000001},"last":0.62}]},{"osd":7,"last update":"Mon Oct 16 12:26:00 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57599999999999996,"15min":0.58099999999999996},"min":{"1min":0.42099999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.76200000000000001,"5min":0.80900000000000005,"15min":0.89500000000000002},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.628,"15min":0.626},"min":{"1min":0.42599999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.87,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.68300000000000005}]}]},{"osd":5,"up_from":13,"seq":55834575039,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54388,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317452,"statfs":{"total":96636764160,"available":96581070848,"internally_reserved":0,"allocated":1024000,"data_stored":507254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:25:58 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.59599999999999997,"15min":0.59399999999999997},"min":{"1min":0.49399999999999999,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.84499999999999997,"5min":0.873,"15min":0.873},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.61099999999999999,"15min":0.622},"min":{"1min":0.432,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.88800000000000001},"last":0.505}]},{"osd":1,"last update":"Mon Oct 16 12:25:58 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.61499999999999999,"15min":0.61699999999999999},"min":{"1min":0.47899999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.89700000000000002,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.63500000000000001,"15min":0.626},"min":{"1min":0.52600000000000002,"5min":0.32800000000000001,"15min":0.28799999999999998},"max":{"1min":0.84799999999999998,"5min":0.89000000000000001,"15min":0.95199999999999996},"last":0.60699999999999998}]},{"osd":2,"last update":"Mon Oct 16 12:25:58 2023","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.63300000000000001,"15min":0.629},"min":{"1min":0.49299999999999999,"5min":0.29099999999999998,"15min":0.26400000000000001},"max":{"1min":0.86199999999999999,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.61599999999999999,"15min":0.61899999999999999},"min":{"1min":0.48699999999999999,"5min":0.378,"15min":0.36499999999999999},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.89700000000000002},"last":0.42199999999999999}]},{"osd":3,"last update":"Mon Oct 16 12:25:58 2023","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.63700000000000001,"15min":0.64700000000000002},"min":{"1min":0.53700000000000003,"5min":0.371,"15min":0.36099999999999999},"max":{"1min":0.81899999999999995,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.65400000000000003,"15min":0.65800000000000003},"min":{"1min":0.47299999999999998,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.88700000000000001,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.44600000000000001}]},{"osd":4,"last update":"Mon Oct 16 12:25:58 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.51000000000000001,"15min":0.51900000000000002},"min":{"1min":0.39200000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.77200000000000002,"5min":0.85699999999999998,"15min":1.0089999999999999},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.57299999999999995,"15min":0.57799999999999996},"min":{"1min":0.41299999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.755,"5min":1.347,"15min":1.347},"last":0.67500000000000004}]},{"osd":6,"last update":"Mon Oct 16 12:25:58 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.55200000000000005,"15min":0.56100000000000005},"min":{"1min":0.40400000000000003,"5min":0.314,"15min":0.314},"max":{"1min":0.89400000000000002,"5min":0.93500000000000005,"15min":0.98699999999999999},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.60799999999999998,"15min":0.60699999999999998},"min":{"1min":0.371,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.81699999999999995,"5min":0.92700000000000005,"15min":0.95999999999999996},"last":0.58699999999999997}]},{"osd":7,"last update":"Mon Oct 16 12:25:58 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.621,"15min":0.626},"min":{"1min":0.42099999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.84699999999999998,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.626},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.66600000000000004,"15min":0.66500000000000004},"min":{"1min":0.50800000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.91600000000000004,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.65800000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T12:26:28.450 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T12:26:28.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.605+0000 7f4172b25700 1 Processor -- start 2023-10-16T12:26:28.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.605+0000 7f4172b25700 1 -- start start 2023-10-16T12:26:28.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.605+0000 7f4172b25700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f416c0ce4f0 0x7f416c0d0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:28.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.605+0000 7f4172b25700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f416c15ba10 0x7f416c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:28.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.605+0000 7f4172b25700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f416c0b0970 con 0x7f416c15b630 2023-10-16T12:26:28.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.605+0000 7f4172b25700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f416c0cd690 con 0x7f416c15ba10 2023-10-16T12:26:28.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.605+0000 7f4172b25700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f416c0cd970 con 0x7f416c0ce4f0 2023-10-16T12:26:28.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.606+0000 7f41710c2700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f416c15ba10 0x7f416c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:28.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.606+0000 7f41710c2700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f416c15ba10 0x7f416c0cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39424/0 (socket says 172.21.15.16:39424) 2023-10-16T12:26:28.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.606+0000 7f41710c2700 1 -- 172.21.15.16:0/772449909 learned_addr learned my addr 172.21.15.16:0/772449909 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:28.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.606+0000 7f416bfff700 1 --2- 172.21.15.16:0/772449909 >> v2:172.21.15.49:3300/0 conn(0x7f416c0ce4f0 0x7f416c0d0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:28.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.606+0000 7f41710c2700 1 -- 172.21.15.16:0/772449909 >> v2:172.21.15.49:3300/0 conn(0x7f416c0ce4f0 msgr2=0x7f416c0d0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:28.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.606+0000 7f41710c2700 1 --2- 172.21.15.16:0/772449909 >> v2:172.21.15.49:3300/0 conn(0x7f416c0ce4f0 0x7f416c0d0380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:28.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.606+0000 7f41710c2700 1 -- 172.21.15.16:0/772449909 >> v1:172.21.15.49:6789/0 conn(0x7f416c15b630 legacy=0x7f416c0cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:28.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.606+0000 7f41710c2700 1 -- 172.21.15.16:0/772449909 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f416c0b0c30 con 0x7f416c15ba10 2023-10-16T12:26:28.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.606+0000 7f41710c2700 1 --2- 172.21.15.16:0/772449909 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f416c15ba10 0x7f416c0cf290 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f416c0d2460 tx=0x7f416400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=254634cbb3500edb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:28.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.606+0000 7f416b7fe700 1 -- 172.21.15.16:0/772449909 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f416400e070 con 0x7f416c15ba10 2023-10-16T12:26:28.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.606+0000 7f416b7fe700 1 -- 172.21.15.16:0/772449909 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f416400b9c0 con 0x7f416c15ba10 2023-10-16T12:26:28.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.606+0000 7f416b7fe700 1 -- 172.21.15.16:0/772449909 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f416400bd70 con 0x7f416c15ba10 2023-10-16T12:26:28.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.607+0000 7f4172b25700 1 -- 172.21.15.16:0/772449909 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f416c15ba10 msgr2=0x7f416c0cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:28.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.607+0000 7f4172b25700 1 --2- 172.21.15.16:0/772449909 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f416c15ba10 0x7f416c0cf290 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f416c0d2460 tx=0x7f416400b330 comp rx=0 tx=0).stop 2023-10-16T12:26:28.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.607+0000 7f4172b25700 1 -- 172.21.15.16:0/772449909 shutdown_connections 2023-10-16T12:26:28.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.607+0000 7f4172b25700 1 --2- 172.21.15.16:0/772449909 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f416c15ba10 0x7f416c0cf290 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:28.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.607+0000 7f4172b25700 1 --2- 172.21.15.16:0/772449909 >> v2:172.21.15.49:3300/0 conn(0x7f416c0ce4f0 0x7f416c0d0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:28.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.607+0000 7f4172b25700 1 -- 172.21.15.16:0/772449909 >> 172.21.15.16:0/772449909 conn(0x7f416c0d14b0 msgr2=0x7f416c0d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:28.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.607+0000 7f4172b25700 1 -- 172.21.15.16:0/772449909 shutdown_connections 2023-10-16T12:26:28.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.607+0000 7f4172b25700 1 -- 172.21.15.16:0/772449909 wait complete. 2023-10-16T12:26:28.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.608+0000 7f4172b25700 1 Processor -- start 2023-10-16T12:26:28.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.608+0000 7f4172b25700 1 -- start start 2023-10-16T12:26:28.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.609+0000 7f4172b25700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f416c0ce4f0 0x7f416c0ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:28.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.609+0000 7f4172b25700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f416c15b630 0x7f416c0b56e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:28.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.609+0000 7f4172b25700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f416c15ba10 0x7f416c06b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:28.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.609+0000 7f4172b25700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f416c0c86b0 con 0x7f416c15ba10 2023-10-16T12:26:28.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.609+0000 7f4172b25700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f416c06be30 con 0x7f416c15b630 2023-10-16T12:26:28.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.609+0000 7f4172b25700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f416c06c950 con 0x7f416c0ce4f0 2023-10-16T12:26:28.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.609+0000 7f41710c2700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f416c15ba10 0x7f416c06b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:28.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.609+0000 7f41710c2700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f416c15ba10 0x7f416c06b8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39430/0 (socket says 172.21.15.16:39430) 2023-10-16T12:26:28.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.609+0000 7f41710c2700 1 -- 172.21.15.16:0/1270113072 learned_addr learned my addr 172.21.15.16:0/1270113072 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:28.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.609+0000 7f416bfff700 1 --2- 172.21.15.16:0/1270113072 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f416c15b630 0x7f416c0b56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:28.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.609+0000 7f41708c1700 1 --2- 172.21.15.16:0/1270113072 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f416c0ce4f0 0x7f416c0ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:28.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.609+0000 7f41710c2700 1 -- 172.21.15.16:0/1270113072 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f416c0ce4f0 msgr2=0x7f416c0ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:28.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.609+0000 7f41710c2700 1 --2- 172.21.15.16:0/1270113072 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f416c0ce4f0 0x7f416c0ca3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:28.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.609+0000 7f41710c2700 1 -- 172.21.15.16:0/1270113072 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f416c15b630 msgr2=0x7f416c0b56e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:28.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.609+0000 7f41710c2700 1 --2- 172.21.15.16:0/1270113072 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f416c15b630 0x7f416c0b56e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:28.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.609+0000 7f41710c2700 1 -- 172.21.15.16:0/1270113072 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4164009b30 con 0x7f416c15ba10 2023-10-16T12:26:28.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.610+0000 7f41710c2700 1 --2- 172.21.15.16:0/1270113072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f416c15ba10 0x7f416c06b8f0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f416c072680 tx=0x7f4164004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:28.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.610+0000 7f41697fa700 1 -- 172.21.15.16:0/1270113072 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41640048c0 con 0x7f416c15ba10 2023-10-16T12:26:28.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.610+0000 7f41697fa700 1 -- 172.21.15.16:0/1270113072 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41640099b0 con 0x7f416c15ba10 2023-10-16T12:26:28.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.610+0000 7f4172b25700 1 -- 172.21.15.16:0/1270113072 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f416c06c0b0 con 0x7f416c15ba10 2023-10-16T12:26:28.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.610+0000 7f41697fa700 1 -- 172.21.15.16:0/1270113072 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f416401b670 con 0x7f416c15ba10 2023-10-16T12:26:28.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.610+0000 7f4172b25700 1 -- 172.21.15.16:0/1270113072 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f416c06c670 con 0x7f416c15ba10 2023-10-16T12:26:28.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.611+0000 7f4172b25700 1 -- 172.21.15.16:0/1270113072 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f416c0c8860 con 0x7f416c15ba10 2023-10-16T12:26:28.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.611+0000 7f41697fa700 1 -- 172.21.15.16:0/1270113072 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 31) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f416401b7d0 con 0x7f416c15ba10 2023-10-16T12:26:28.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.612+0000 7f41697fa700 1 --2- 172.21.15.16:0/1270113072 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f414c06b020 0x7f414c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:28.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.612+0000 7f41697fa700 1 -- 172.21.15.16:0/1270113072 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f416408b130 con 0x7f416c15ba10 2023-10-16T12:26:28.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.612+0000 7f41708c1700 1 --2- 172.21.15.16:0/1270113072 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f414c06b020 0x7f414c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:28.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.612+0000 7f41708c1700 1 --2- 172.21.15.16:0/1270113072 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f414c06b020 0x7f414c06d4e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f41540018a0 tx=0x7f4154009040 comp rx=0 tx=0).ready entity=mgr.6396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:28.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.616+0000 7f41697fa700 1 -- 172.21.15.16:0/1270113072 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f416c0c8860 con 0x7f416c15ba10 2023-10-16T12:26:28.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.815+0000 7f4172b25700 1 -- 172.21.15.16:0/1270113072 --> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f416c0b0f10 con 0x7f414c06b020 2023-10-16T12:26:28.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.823+0000 7f41697fa700 1 -- 172.21.15.16:0/1270113072 <== mgr.6396 v2:172.21.15.16:6832/3415303009 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289419 (secure 0 0 0) 0x7f4154001540 con 0x7f414c06b020 2023-10-16T12:26:28.829 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:26:28.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.826+0000 7f4172b25700 1 -- 172.21.15.16:0/1270113072 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f414c06b020 msgr2=0x7f414c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:28.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.826+0000 7f4172b25700 1 --2- 172.21.15.16:0/1270113072 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f414c06b020 0x7f414c06d4e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f41540018a0 tx=0x7f4154009040 comp rx=0 tx=0).stop 2023-10-16T12:26:28.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.826+0000 7f4172b25700 1 -- 172.21.15.16:0/1270113072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f416c15ba10 msgr2=0x7f416c06b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:28.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.826+0000 7f4172b25700 1 --2- 172.21.15.16:0/1270113072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f416c15ba10 0x7f416c06b8f0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f416c072680 tx=0x7f4164004ad0 comp rx=0 tx=0).stop 2023-10-16T12:26:28.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.826+0000 7f4172b25700 1 -- 172.21.15.16:0/1270113072 shutdown_connections 2023-10-16T12:26:28.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.826+0000 7f4172b25700 1 --2- 172.21.15.16:0/1270113072 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f414c06b020 0x7f414c06d4e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:28.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.826+0000 7f4172b25700 1 --2- 172.21.15.16:0/1270113072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f416c15ba10 0x7f416c06b8f0 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:28.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.826+0000 7f4172b25700 1 --2- 172.21.15.16:0/1270113072 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f416c0ce4f0 0x7f416c0ca3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:28.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.826+0000 7f4172b25700 1 --2- 172.21.15.16:0/1270113072 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f416c15b630 0x7f416c0b56e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:28.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.826+0000 7f4172b25700 1 -- 172.21.15.16:0/1270113072 >> 172.21.15.16:0/1270113072 conn(0x7f416c0d14b0 msgr2=0x7f416c0b1330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:28.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.826+0000 7f4172b25700 1 -- 172.21.15.16:0/1270113072 shutdown_connections 2023-10-16T12:26:28.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:28.826+0000 7f4172b25700 1 -- 172.21.15.16:0/1270113072 wait complete. 2023-10-16T12:26:28.834 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-10-16T12:26:28.841 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2023-10-16T12:26:27.837053+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":283,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":199,"ondisk_log_size":199,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":128,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":341948,"kb_used_data":9228,"kb_used_omap":12,"kb_used_meta":332595,"kb_avail":754632772,"statfs":{"total":773094113280,"available":772743958528,"internally_reserved":0,"allocated":9449472,"data_stored":5263356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12740,"internal_metadata":340577852},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.595300"},"pg_stats":[{"pgid":"10.3f","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.303934+0000","last_change":"2023-10-16T12:26:24.291482+0000","last_active":"2023-10-16T12:26:24.303934+0000","last_peered":"2023-10-16T12:26:24.303934+0000","last_clean":"2023-10-16T12:26:24.303934+0000","last_became_active":"2023-10-16T12:26:24.291214+0000","last_became_peered":"2023-10-16T12:26:24.291214+0000","last_unstale":"2023-10-16T12:26:24.303934+0000","last_undegraded":"2023-10-16T12:26:24.303934+0000","last_fullsized":"2023-10-16T12:26:24.303934+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:30:37.750430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.304100+0000","last_change":"2023-10-16T12:26:24.293356+0000","last_active":"2023-10-16T12:26:24.304100+0000","last_peered":"2023-10-16T12:26:24.304100+0000","last_clean":"2023-10-16T12:26:24.304100+0000","last_became_active":"2023-10-16T12:26:24.293064+0000","last_became_peered":"2023-10-16T12:26:24.293064+0000","last_unstale":"2023-10-16T12:26:24.304100+0000","last_undegraded":"2023-10-16T12:26:24.304100+0000","last_fullsized":"2023-10-16T12:26:24.304100+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:29:54.433098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3d","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.568895+0000","last_change":"2023-10-16T12:26:24.307746+0000","last_active":"2023-10-16T12:26:25.568895+0000","last_peered":"2023-10-16T12:26:25.568895+0000","last_clean":"2023-10-16T12:26:25.568895+0000","last_became_active":"2023-10-16T12:26:24.307102+0000","last_became_peered":"2023-10-16T12:26:24.307102+0000","last_unstale":"2023-10-16T12:26:25.568895+0000","last_undegraded":"2023-10-16T12:26:25.568895+0000","last_fullsized":"2023-10-16T12:26:25.568895+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:38:53.910319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.595730+0000","last_change":"2023-10-16T12:26:24.296652+0000","last_active":"2023-10-16T12:26:25.595730+0000","last_peered":"2023-10-16T12:26:25.595730+0000","last_clean":"2023-10-16T12:26:25.595730+0000","last_became_active":"2023-10-16T12:26:24.296197+0000","last_became_peered":"2023-10-16T12:26:24.296197+0000","last_unstale":"2023-10-16T12:26:25.595730+0000","last_undegraded":"2023-10-16T12:26:25.595730+0000","last_fullsized":"2023-10-16T12:26:25.595730+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:25:13.048790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.298521+0000","last_change":"2023-10-16T12:26:24.294961+0000","last_active":"2023-10-16T12:26:24.298521+0000","last_peered":"2023-10-16T12:26:24.298521+0000","last_clean":"2023-10-16T12:26:24.298521+0000","last_became_active":"2023-10-16T12:26:24.294651+0000","last_became_peered":"2023-10-16T12:26:24.294651+0000","last_unstale":"2023-10-16T12:26:24.298521+0000","last_undegraded":"2023-10-16T12:26:24.298521+0000","last_fullsized":"2023-10-16T12:26:24.298521+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:15:21.085868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.288337+0000","last_change":"2023-10-16T12:26:24.297705+0000","last_active":"2023-10-16T12:26:25.288337+0000","last_peered":"2023-10-16T12:26:25.288337+0000","last_clean":"2023-10-16T12:26:25.288337+0000","last_became_active":"2023-10-16T12:26:24.293985+0000","last_became_peered":"2023-10-16T12:26:24.293985+0000","last_unstale":"2023-10-16T12:26:25.288337+0000","last_undegraded":"2023-10-16T12:26:25.288337+0000","last_fullsized":"2023-10-16T12:26:25.288337+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:36:36.653796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.302030+0000","last_change":"2023-10-16T12:26:24.292411+0000","last_active":"2023-10-16T12:26:24.302030+0000","last_peered":"2023-10-16T12:26:24.302030+0000","last_clean":"2023-10-16T12:26:24.302030+0000","last_became_active":"2023-10-16T12:26:24.291569+0000","last_became_peered":"2023-10-16T12:26:24.291569+0000","last_unstale":"2023-10-16T12:26:24.302030+0000","last_undegraded":"2023-10-16T12:26:24.302030+0000","last_fullsized":"2023-10-16T12:26:24.302030+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:15:12.996970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.294807+0000","last_change":"2023-10-16T12:26:24.288129+0000","last_active":"2023-10-16T12:26:24.294807+0000","last_peered":"2023-10-16T12:26:24.294807+0000","last_clean":"2023-10-16T12:26:24.294807+0000","last_became_active":"2023-10-16T12:26:24.287853+0000","last_became_peered":"2023-10-16T12:26:24.287853+0000","last_unstale":"2023-10-16T12:26:24.294807+0000","last_undegraded":"2023-10-16T12:26:24.294807+0000","last_fullsized":"2023-10-16T12:26:24.294807+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:47:00.719878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.569207+0000","last_change":"2023-10-16T12:26:24.306698+0000","last_active":"2023-10-16T12:26:25.569207+0000","last_peered":"2023-10-16T12:26:25.569207+0000","last_clean":"2023-10-16T12:26:25.569207+0000","last_became_active":"2023-10-16T12:26:24.305524+0000","last_became_peered":"2023-10-16T12:26:24.305524+0000","last_unstale":"2023-10-16T12:26:25.569207+0000","last_undegraded":"2023-10-16T12:26:25.569207+0000","last_fullsized":"2023-10-16T12:26:25.569207+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:34:24.998126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.569057+0000","last_change":"2023-10-16T12:26:24.299860+0000","last_active":"2023-10-16T12:26:25.569057+0000","last_peered":"2023-10-16T12:26:25.569057+0000","last_clean":"2023-10-16T12:26:25.569057+0000","last_became_active":"2023-10-16T12:26:24.299595+0000","last_became_peered":"2023-10-16T12:26:24.299595+0000","last_unstale":"2023-10-16T12:26:25.569057+0000","last_undegraded":"2023-10-16T12:26:25.569057+0000","last_fullsized":"2023-10-16T12:26:25.569057+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:24:05.500326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.595810+0000","last_change":"2023-10-16T12:26:24.299719+0000","last_active":"2023-10-16T12:26:25.595810+0000","last_peered":"2023-10-16T12:26:25.595810+0000","last_clean":"2023-10-16T12:26:25.595810+0000","last_became_active":"2023-10-16T12:26:24.299265+0000","last_became_peered":"2023-10-16T12:26:24.299265+0000","last_unstale":"2023-10-16T12:26:25.595810+0000","last_undegraded":"2023-10-16T12:26:25.595810+0000","last_fullsized":"2023-10-16T12:26:25.595810+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:21:20.665768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.294976+0000","last_change":"2023-10-16T12:26:24.292201+0000","last_active":"2023-10-16T12:26:24.294976+0000","last_peered":"2023-10-16T12:26:24.294976+0000","last_clean":"2023-10-16T12:26:24.294976+0000","last_became_active":"2023-10-16T12:26:24.291873+0000","last_became_peered":"2023-10-16T12:26:24.291873+0000","last_unstale":"2023-10-16T12:26:24.294976+0000","last_undegraded":"2023-10-16T12:26:24.294976+0000","last_fullsized":"2023-10-16T12:26:24.294976+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:24:21.635364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.295802+0000","last_change":"2023-10-16T12:26:24.294587+0000","last_active":"2023-10-16T12:26:24.295802+0000","last_peered":"2023-10-16T12:26:24.295802+0000","last_clean":"2023-10-16T12:26:24.295802+0000","last_became_active":"2023-10-16T12:26:24.294238+0000","last_became_peered":"2023-10-16T12:26:24.294238+0000","last_unstale":"2023-10-16T12:26:24.295802+0000","last_undegraded":"2023-10-16T12:26:24.295802+0000","last_fullsized":"2023-10-16T12:26:24.295802+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:58:55.626587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.569213+0000","last_change":"2023-10-16T12:26:24.305849+0000","last_active":"2023-10-16T12:26:25.569213+0000","last_peered":"2023-10-16T12:26:25.569213+0000","last_clean":"2023-10-16T12:26:25.569213+0000","last_became_active":"2023-10-16T12:26:24.305271+0000","last_became_peered":"2023-10-16T12:26:24.305271+0000","last_unstale":"2023-10-16T12:26:25.569213+0000","last_undegraded":"2023-10-16T12:26:25.569213+0000","last_fullsized":"2023-10-16T12:26:25.569213+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:40:05.795607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.569065+0000","last_change":"2023-10-16T12:26:24.292049+0000","last_active":"2023-10-16T12:26:25.569065+0000","last_peered":"2023-10-16T12:26:25.569065+0000","last_clean":"2023-10-16T12:26:25.569065+0000","last_became_active":"2023-10-16T12:26:24.291685+0000","last_became_peered":"2023-10-16T12:26:24.291685+0000","last_unstale":"2023-10-16T12:26:25.569065+0000","last_undegraded":"2023-10-16T12:26:25.569065+0000","last_fullsized":"2023-10-16T12:26:25.569065+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:00:37.235614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.302766+0000","last_change":"2023-10-16T12:26:24.296582+0000","last_active":"2023-10-16T12:26:24.302766+0000","last_peered":"2023-10-16T12:26:24.302766+0000","last_clean":"2023-10-16T12:26:24.302766+0000","last_became_active":"2023-10-16T12:26:24.292026+0000","last_became_peered":"2023-10-16T12:26:24.292026+0000","last_unstale":"2023-10-16T12:26:24.302766+0000","last_undegraded":"2023-10-16T12:26:24.302766+0000","last_fullsized":"2023-10-16T12:26:24.302766+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:03:25.282953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.304785+0000","last_change":"2023-10-16T12:26:24.298939+0000","last_active":"2023-10-16T12:26:24.304785+0000","last_peered":"2023-10-16T12:26:24.304785+0000","last_clean":"2023-10-16T12:26:24.304785+0000","last_became_active":"2023-10-16T12:26:24.298658+0000","last_became_peered":"2023-10-16T12:26:24.298658+0000","last_unstale":"2023-10-16T12:26:24.304785+0000","last_undegraded":"2023-10-16T12:26:24.304785+0000","last_fullsized":"2023-10-16T12:26:24.304785+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:33:21.708694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.595581+0000","last_change":"2023-10-16T12:26:24.294938+0000","last_active":"2023-10-16T12:26:25.595581+0000","last_peered":"2023-10-16T12:26:25.595581+0000","last_clean":"2023-10-16T12:26:25.595581+0000","last_became_active":"2023-10-16T12:26:24.294373+0000","last_became_peered":"2023-10-16T12:26:24.294373+0000","last_unstale":"2023-10-16T12:26:25.595581+0000","last_undegraded":"2023-10-16T12:26:25.595581+0000","last_fullsized":"2023-10-16T12:26:25.595581+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:23:12.559260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.295044+0000","last_change":"2023-10-16T12:26:24.292910+0000","last_active":"2023-10-16T12:26:24.295044+0000","last_peered":"2023-10-16T12:26:24.295044+0000","last_clean":"2023-10-16T12:26:24.295044+0000","last_became_active":"2023-10-16T12:26:24.292438+0000","last_became_peered":"2023-10-16T12:26:24.292438+0000","last_unstale":"2023-10-16T12:26:24.295044+0000","last_undegraded":"2023-10-16T12:26:24.295044+0000","last_fullsized":"2023-10-16T12:26:24.295044+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:20:02.702684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.288327+0000","last_change":"2023-10-16T12:26:24.290041+0000","last_active":"2023-10-16T12:26:25.288327+0000","last_peered":"2023-10-16T12:26:25.288327+0000","last_clean":"2023-10-16T12:26:25.288327+0000","last_became_active":"2023-10-16T12:26:24.289796+0000","last_became_peered":"2023-10-16T12:26:24.289796+0000","last_unstale":"2023-10-16T12:26:25.288327+0000","last_undegraded":"2023-10-16T12:26:25.288327+0000","last_fullsized":"2023-10-16T12:26:25.288327+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:14:41.508247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.297822+0000","last_change":"2023-10-16T12:26:24.295721+0000","last_active":"2023-10-16T12:26:24.297822+0000","last_peered":"2023-10-16T12:26:24.297822+0000","last_clean":"2023-10-16T12:26:24.297822+0000","last_became_active":"2023-10-16T12:26:24.295354+0000","last_became_peered":"2023-10-16T12:26:24.295354+0000","last_unstale":"2023-10-16T12:26:24.297822+0000","last_undegraded":"2023-10-16T12:26:24.297822+0000","last_fullsized":"2023-10-16T12:26:24.297822+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:13:49.992850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.287970+0000","last_change":"2023-10-16T12:26:24.293166+0000","last_active":"2023-10-16T12:26:25.287970+0000","last_peered":"2023-10-16T12:26:25.287970+0000","last_clean":"2023-10-16T12:26:25.287970+0000","last_became_active":"2023-10-16T12:26:24.292885+0000","last_became_peered":"2023-10-16T12:26:24.292885+0000","last_unstale":"2023-10-16T12:26:25.287970+0000","last_undegraded":"2023-10-16T12:26:25.287970+0000","last_fullsized":"2023-10-16T12:26:25.287970+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:23:51.982187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.287816+0000","last_change":"2023-10-16T12:26:24.293639+0000","last_active":"2023-10-16T12:26:25.287816+0000","last_peered":"2023-10-16T12:26:25.287816+0000","last_clean":"2023-10-16T12:26:25.287816+0000","last_became_active":"2023-10-16T12:26:24.293414+0000","last_became_peered":"2023-10-16T12:26:24.293414+0000","last_unstale":"2023-10-16T12:26:25.287816+0000","last_undegraded":"2023-10-16T12:26:25.287816+0000","last_fullsized":"2023-10-16T12:26:25.287816+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:16:12.888429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.568704+0000","last_change":"2023-10-16T12:26:24.306975+0000","last_active":"2023-10-16T12:26:25.568704+0000","last_peered":"2023-10-16T12:26:25.568704+0000","last_clean":"2023-10-16T12:26:25.568704+0000","last_became_active":"2023-10-16T12:26:24.306548+0000","last_became_peered":"2023-10-16T12:26:24.306548+0000","last_unstale":"2023-10-16T12:26:25.568704+0000","last_undegraded":"2023-10-16T12:26:25.568704+0000","last_fullsized":"2023-10-16T12:26:25.568704+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:21:46.897480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.294499+0000","last_change":"2023-10-16T12:26:24.287586+0000","last_active":"2023-10-16T12:26:24.294499+0000","last_peered":"2023-10-16T12:26:24.294499+0000","last_clean":"2023-10-16T12:26:24.294499+0000","last_became_active":"2023-10-16T12:26:24.287184+0000","last_became_peered":"2023-10-16T12:26:24.287184+0000","last_unstale":"2023-10-16T12:26:24.294499+0000","last_undegraded":"2023-10-16T12:26:24.294499+0000","last_fullsized":"2023-10-16T12:26:24.294499+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:05:59.476003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.294452+0000","last_change":"2023-10-16T12:26:24.287423+0000","last_active":"2023-10-16T12:26:24.294452+0000","last_peered":"2023-10-16T12:26:24.294452+0000","last_clean":"2023-10-16T12:26:24.294452+0000","last_became_active":"2023-10-16T12:26:24.287102+0000","last_became_peered":"2023-10-16T12:26:24.287102+0000","last_unstale":"2023-10-16T12:26:24.294452+0000","last_undegraded":"2023-10-16T12:26:24.294452+0000","last_fullsized":"2023-10-16T12:26:24.294452+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:49:20.689145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.595932+0000","last_change":"2023-10-16T12:26:24.296525+0000","last_active":"2023-10-16T12:26:25.595932+0000","last_peered":"2023-10-16T12:26:25.595932+0000","last_clean":"2023-10-16T12:26:25.595932+0000","last_became_active":"2023-10-16T12:26:24.294698+0000","last_became_peered":"2023-10-16T12:26:24.294698+0000","last_unstale":"2023-10-16T12:26:25.595932+0000","last_undegraded":"2023-10-16T12:26:25.595932+0000","last_fullsized":"2023-10-16T12:26:25.595932+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:36:54.572808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.296559+0000","last_change":"2023-10-16T12:26:24.285157+0000","last_active":"2023-10-16T12:26:24.296559+0000","last_peered":"2023-10-16T12:26:24.296559+0000","last_clean":"2023-10-16T12:26:24.296559+0000","last_became_active":"2023-10-16T12:26:24.284854+0000","last_became_peered":"2023-10-16T12:26:24.284854+0000","last_unstale":"2023-10-16T12:26:24.296559+0000","last_undegraded":"2023-10-16T12:26:24.296559+0000","last_fullsized":"2023-10-16T12:26:24.296559+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:00:55.517863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.569122+0000","last_change":"2023-10-16T12:26:24.302840+0000","last_active":"2023-10-16T12:26:25.569122+0000","last_peered":"2023-10-16T12:26:25.569122+0000","last_clean":"2023-10-16T12:26:25.569122+0000","last_became_active":"2023-10-16T12:26:24.302545+0000","last_became_peered":"2023-10-16T12:26:24.302545+0000","last_unstale":"2023-10-16T12:26:25.569122+0000","last_undegraded":"2023-10-16T12:26:25.569122+0000","last_fullsized":"2023-10-16T12:26:25.569122+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:00:11.640752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.295524+0000","last_change":"2023-10-16T12:26:24.293502+0000","last_active":"2023-10-16T12:26:24.295524+0000","last_peered":"2023-10-16T12:26:24.295524+0000","last_clean":"2023-10-16T12:26:24.295524+0000","last_became_active":"2023-10-16T12:26:24.293207+0000","last_became_peered":"2023-10-16T12:26:24.293207+0000","last_unstale":"2023-10-16T12:26:24.295524+0000","last_undegraded":"2023-10-16T12:26:24.295524+0000","last_fullsized":"2023-10-16T12:26:24.295524+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:46:33.594590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.595432+0000","last_change":"2023-10-16T12:26:24.299413+0000","last_active":"2023-10-16T12:26:25.595432+0000","last_peered":"2023-10-16T12:26:25.595432+0000","last_clean":"2023-10-16T12:26:25.595432+0000","last_became_active":"2023-10-16T12:26:24.296113+0000","last_became_peered":"2023-10-16T12:26:24.296113+0000","last_unstale":"2023-10-16T12:26:25.595432+0000","last_undegraded":"2023-10-16T12:26:25.595432+0000","last_fullsized":"2023-10-16T12:26:25.595432+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:30:37.750430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.288025+0000","last_change":"2023-10-16T12:26:24.293780+0000","last_active":"2023-10-16T12:26:25.288025+0000","last_peered":"2023-10-16T12:26:25.288025+0000","last_clean":"2023-10-16T12:26:25.288025+0000","last_became_active":"2023-10-16T12:26:24.293403+0000","last_became_peered":"2023-10-16T12:26:24.293403+0000","last_unstale":"2023-10-16T12:26:25.288025+0000","last_undegraded":"2023-10-16T12:26:25.288025+0000","last_fullsized":"2023-10-16T12:26:25.288025+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:03:52.845551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.294876+0000","last_change":"2023-10-16T12:26:24.292760+0000","last_active":"2023-10-16T12:26:24.294876+0000","last_peered":"2023-10-16T12:26:24.294876+0000","last_clean":"2023-10-16T12:26:24.294876+0000","last_became_active":"2023-10-16T12:26:24.292372+0000","last_became_peered":"2023-10-16T12:26:24.292372+0000","last_unstale":"2023-10-16T12:26:24.294876+0000","last_undegraded":"2023-10-16T12:26:24.294876+0000","last_fullsized":"2023-10-16T12:26:24.294876+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:38:04.076463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"84'4","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.307739+0000","last_change":"2023-10-16T12:26:23.265538+0000","last_active":"2023-10-16T12:26:24.307739+0000","last_peered":"2023-10-16T12:26:24.307739+0000","last_clean":"2023-10-16T12:26:24.307739+0000","last_became_active":"2023-10-16T12:26:23.265073+0000","last_became_peered":"2023-10-16T12:26:23.265073+0000","last_unstale":"2023-10-16T12:26:24.307739+0000","last_undegraded":"2023-10-16T12:26:24.307739+0000","last_fullsized":"2023-10-16T12:26:24.307739+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_clean_scrub_stamp":"2023-10-16T12:26:22.237327+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:28:46.749334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.595971+0000","last_change":"2023-10-16T12:26:24.294447+0000","last_active":"2023-10-16T12:26:25.595971+0000","last_peered":"2023-10-16T12:26:25.595971+0000","last_clean":"2023-10-16T12:26:25.595971+0000","last_became_active":"2023-10-16T12:26:24.294111+0000","last_became_peered":"2023-10-16T12:26:24.294111+0000","last_unstale":"2023-10-16T12:26:25.595971+0000","last_undegraded":"2023-10-16T12:26:25.595971+0000","last_fullsized":"2023-10-16T12:26:25.595971+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:52:58.368751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"84'1","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.288826+0000","last_change":"2023-10-16T12:26:23.253068+0000","last_active":"2023-10-16T12:26:25.288826+0000","last_peered":"2023-10-16T12:26:25.288826+0000","last_clean":"2023-10-16T12:26:25.288826+0000","last_became_active":"2023-10-16T12:26:23.252558+0000","last_became_peered":"2023-10-16T12:26:23.252558+0000","last_unstale":"2023-10-16T12:26:25.288826+0000","last_undegraded":"2023-10-16T12:26:25.288826+0000","last_fullsized":"2023-10-16T12:26:25.288826+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_clean_scrub_stamp":"2023-10-16T12:26:22.237327+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:38:37.255759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.288883+0000","last_change":"2023-10-16T12:26:24.297676+0000","last_active":"2023-10-16T12:26:25.288883+0000","last_peered":"2023-10-16T12:26:25.288883+0000","last_clean":"2023-10-16T12:26:25.288883+0000","last_became_active":"2023-10-16T12:26:24.293650+0000","last_became_peered":"2023-10-16T12:26:24.293650+0000","last_unstale":"2023-10-16T12:26:25.288883+0000","last_undegraded":"2023-10-16T12:26:25.288883+0000","last_fullsized":"2023-10-16T12:26:25.288883+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:41:43.972573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"84'2","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.288791+0000","last_change":"2023-10-16T12:26:23.267505+0000","last_active":"2023-10-16T12:26:25.288791+0000","last_peered":"2023-10-16T12:26:25.288791+0000","last_clean":"2023-10-16T12:26:25.288791+0000","last_became_active":"2023-10-16T12:26:23.267141+0000","last_became_peered":"2023-10-16T12:26:23.267141+0000","last_unstale":"2023-10-16T12:26:25.288791+0000","last_undegraded":"2023-10-16T12:26:25.288791+0000","last_fullsized":"2023-10-16T12:26:25.288791+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_clean_scrub_stamp":"2023-10-16T12:26:22.237327+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:00:10.636690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.569282+0000","last_change":"2023-10-16T12:26:24.305977+0000","last_active":"2023-10-16T12:26:25.569282+0000","last_peered":"2023-10-16T12:26:25.569282+0000","last_clean":"2023-10-16T12:26:25.569282+0000","last_became_active":"2023-10-16T12:26:24.305647+0000","last_became_peered":"2023-10-16T12:26:24.305647+0000","last_unstale":"2023-10-16T12:26:25.569282+0000","last_undegraded":"2023-10-16T12:26:25.569282+0000","last_fullsized":"2023-10-16T12:26:25.569282+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:02:17.655193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"84'2","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.288568+0000","last_change":"2023-10-16T12:26:23.269237+0000","last_active":"2023-10-16T12:26:25.288568+0000","last_peered":"2023-10-16T12:26:25.288568+0000","last_clean":"2023-10-16T12:26:25.288568+0000","last_became_active":"2023-10-16T12:26:23.268929+0000","last_became_peered":"2023-10-16T12:26:23.268929+0000","last_unstale":"2023-10-16T12:26:25.288568+0000","last_undegraded":"2023-10-16T12:26:25.288568+0000","last_fullsized":"2023-10-16T12:26:25.288568+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_clean_scrub_stamp":"2023-10-16T12:26:22.237327+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:02:16.651131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.302793+0000","last_change":"2023-10-16T12:26:24.296806+0000","last_active":"2023-10-16T12:26:24.302793+0000","last_peered":"2023-10-16T12:26:24.302793+0000","last_clean":"2023-10-16T12:26:24.302793+0000","last_became_active":"2023-10-16T12:26:24.291631+0000","last_became_peered":"2023-10-16T12:26:24.291631+0000","last_unstale":"2023-10-16T12:26:24.302793+0000","last_undegraded":"2023-10-16T12:26:24.302793+0000","last_fullsized":"2023-10-16T12:26:24.302793+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:28:46.365589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"84'1","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.288200+0000","last_change":"2023-10-16T12:26:23.263406+0000","last_active":"2023-10-16T12:26:25.288200+0000","last_peered":"2023-10-16T12:26:25.288200+0000","last_clean":"2023-10-16T12:26:25.288200+0000","last_became_active":"2023-10-16T12:26:23.263106+0000","last_became_peered":"2023-10-16T12:26:23.263106+0000","last_unstale":"2023-10-16T12:26:25.288200+0000","last_undegraded":"2023-10-16T12:26:25.288200+0000","last_fullsized":"2023-10-16T12:26:25.288200+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_clean_scrub_stamp":"2023-10-16T12:26:22.237327+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:34:23.994064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.304643+0000","last_change":"2023-10-16T12:26:24.295804+0000","last_active":"2023-10-16T12:26:24.304643+0000","last_peered":"2023-10-16T12:26:24.304643+0000","last_clean":"2023-10-16T12:26:24.304643+0000","last_became_active":"2023-10-16T12:26:24.295466+0000","last_became_peered":"2023-10-16T12:26:24.295466+0000","last_unstale":"2023-10-16T12:26:24.304643+0000","last_undegraded":"2023-10-16T12:26:24.304643+0000","last_fullsized":"2023-10-16T12:26:24.304643+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:34:11.928461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"84'1","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.287879+0000","last_change":"2023-10-16T12:26:23.265444+0000","last_active":"2023-10-16T12:26:25.287879+0000","last_peered":"2023-10-16T12:26:25.287879+0000","last_clean":"2023-10-16T12:26:25.287879+0000","last_became_active":"2023-10-16T12:26:23.265117+0000","last_became_peered":"2023-10-16T12:26:23.265117+0000","last_unstale":"2023-10-16T12:26:25.287879+0000","last_undegraded":"2023-10-16T12:26:25.287879+0000","last_fullsized":"2023-10-16T12:26:25.287879+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_clean_scrub_stamp":"2023-10-16T12:26:22.237327+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:02:16.651131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.278004+0000","last_change":"2023-10-16T12:26:24.277477+0000","last_active":"2023-10-16T12:26:24.278004+0000","last_peered":"2023-10-16T12:26:24.278004+0000","last_clean":"2023-10-16T12:26:24.278004+0000","last_became_active":"2023-10-16T12:26:24.277203+0000","last_became_peered":"2023-10-16T12:26:24.277203+0000","last_unstale":"2023-10-16T12:26:24.278004+0000","last_undegraded":"2023-10-16T12:26:24.278004+0000","last_fullsized":"2023-10-16T12:26:24.278004+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:41:16.602571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.287967+0000","last_change":"2023-10-16T12:26:23.266252+0000","last_active":"2023-10-16T12:26:25.287967+0000","last_peered":"2023-10-16T12:26:25.287967+0000","last_clean":"2023-10-16T12:26:25.287967+0000","last_became_active":"2023-10-16T12:26:23.265945+0000","last_became_peered":"2023-10-16T12:26:23.265945+0000","last_unstale":"2023-10-16T12:26:25.287967+0000","last_undegraded":"2023-10-16T12:26:25.287967+0000","last_fullsized":"2023-10-16T12:26:25.287967+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_clean_scrub_stamp":"2023-10-16T12:26:22.237327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:51:18.592928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.288106+0000","last_change":"2023-10-16T12:26:24.301407+0000","last_active":"2023-10-16T12:26:25.288106+0000","last_peered":"2023-10-16T12:26:25.288106+0000","last_clean":"2023-10-16T12:26:25.288106+0000","last_became_active":"2023-10-16T12:26:24.301089+0000","last_became_peered":"2023-10-16T12:26:24.301089+0000","last_unstale":"2023-10-16T12:26:25.288106+0000","last_undegraded":"2023-10-16T12:26:25.288106+0000","last_fullsized":"2023-10-16T12:26:25.288106+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:47:40.000592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.277000+0000","last_change":"2023-10-16T12:26:23.259422+0000","last_active":"2023-10-16T12:26:24.277000+0000","last_peered":"2023-10-16T12:26:24.277000+0000","last_clean":"2023-10-16T12:26:24.277000+0000","last_became_active":"2023-10-16T12:26:23.259137+0000","last_became_peered":"2023-10-16T12:26:23.259137+0000","last_unstale":"2023-10-16T12:26:24.277000+0000","last_undegraded":"2023-10-16T12:26:24.277000+0000","last_fullsized":"2023-10-16T12:26:24.277000+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_clean_scrub_stamp":"2023-10-16T12:26:22.237327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:31:48.776291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.304026+0000","last_change":"2023-10-16T12:26:24.293242+0000","last_active":"2023-10-16T12:26:24.304026+0000","last_peered":"2023-10-16T12:26:24.304026+0000","last_clean":"2023-10-16T12:26:24.304026+0000","last_became_active":"2023-10-16T12:26:24.292959+0000","last_became_peered":"2023-10-16T12:26:24.292959+0000","last_unstale":"2023-10-16T12:26:24.304026+0000","last_undegraded":"2023-10-16T12:26:24.304026+0000","last_fullsized":"2023-10-16T12:26:24.304026+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:37:23.623750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.569303+0000","last_change":"2023-10-16T12:26:23.267689+0000","last_active":"2023-10-16T12:26:25.569303+0000","last_peered":"2023-10-16T12:26:25.569303+0000","last_clean":"2023-10-16T12:26:25.569303+0000","last_became_active":"2023-10-16T12:26:23.266807+0000","last_became_peered":"2023-10-16T12:26:23.266807+0000","last_unstale":"2023-10-16T12:26:25.569303+0000","last_undegraded":"2023-10-16T12:26:25.569303+0000","last_fullsized":"2023-10-16T12:26:25.569303+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_clean_scrub_stamp":"2023-10-16T12:26:22.237327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:09:36.660402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.288477+0000","last_change":"2023-10-16T12:26:24.295479+0000","last_active":"2023-10-16T12:26:25.288477+0000","last_peered":"2023-10-16T12:26:25.288477+0000","last_clean":"2023-10-16T12:26:25.288477+0000","last_became_active":"2023-10-16T12:26:24.295101+0000","last_became_peered":"2023-10-16T12:26:24.295101+0000","last_unstale":"2023-10-16T12:26:25.288477+0000","last_undegraded":"2023-10-16T12:26:25.288477+0000","last_fullsized":"2023-10-16T12:26:25.288477+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:48:27.755542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.595891+0000","last_change":"2023-10-16T12:26:23.260738+0000","last_active":"2023-10-16T12:26:25.595891+0000","last_peered":"2023-10-16T12:26:25.595891+0000","last_clean":"2023-10-16T12:26:25.595891+0000","last_became_active":"2023-10-16T12:26:23.260449+0000","last_became_peered":"2023-10-16T12:26:23.260449+0000","last_unstale":"2023-10-16T12:26:25.595891+0000","last_undegraded":"2023-10-16T12:26:25.595891+0000","last_fullsized":"2023-10-16T12:26:25.595891+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_clean_scrub_stamp":"2023-10-16T12:26:22.237327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:53:21.641122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.289086+0000","last_change":"2023-10-16T12:26:24.285035+0000","last_active":"2023-10-16T12:26:24.289086+0000","last_peered":"2023-10-16T12:26:24.289086+0000","last_clean":"2023-10-16T12:26:24.289086+0000","last_became_active":"2023-10-16T12:26:24.284743+0000","last_became_peered":"2023-10-16T12:26:24.284743+0000","last_unstale":"2023-10-16T12:26:24.289086+0000","last_undegraded":"2023-10-16T12:26:24.289086+0000","last_fullsized":"2023-10-16T12:26:24.289086+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:22:48.536783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.15","version":"84'3","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.568785+0000","last_change":"2023-10-16T12:26:23.267138+0000","last_active":"2023-10-16T12:26:25.568785+0000","last_peered":"2023-10-16T12:26:25.568785+0000","last_clean":"2023-10-16T12:26:25.568785+0000","last_became_active":"2023-10-16T12:26:23.266839+0000","last_became_peered":"2023-10-16T12:26:23.266839+0000","last_unstale":"2023-10-16T12:26:25.568785+0000","last_undegraded":"2023-10-16T12:26:25.568785+0000","last_fullsized":"2023-10-16T12:26:25.568785+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_clean_scrub_stamp":"2023-10-16T12:26:22.237327+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:58:54.622524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.595934+0000","last_change":"2023-10-16T12:26:24.290640+0000","last_active":"2023-10-16T12:26:25.595934+0000","last_peered":"2023-10-16T12:26:25.595934+0000","last_clean":"2023-10-16T12:26:25.595934+0000","last_became_active":"2023-10-16T12:26:24.290395+0000","last_became_peered":"2023-10-16T12:26:24.290395+0000","last_unstale":"2023-10-16T12:26:25.595934+0000","last_undegraded":"2023-10-16T12:26:25.595934+0000","last_fullsized":"2023-10-16T12:26:25.595934+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:57:11.946684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"84'6","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.309885+0000","last_change":"2023-10-16T12:26:23.268885+0000","last_active":"2023-10-16T12:26:24.309885+0000","last_peered":"2023-10-16T12:26:24.309885+0000","last_clean":"2023-10-16T12:26:24.309885+0000","last_became_active":"2023-10-16T12:26:23.268605+0000","last_became_peered":"2023-10-16T12:26:23.268605+0000","last_unstale":"2023-10-16T12:26:24.309885+0000","last_undegraded":"2023-10-16T12:26:24.309885+0000","last_fullsized":"2023-10-16T12:26:24.309885+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_clean_scrub_stamp":"2023-10-16T12:26:22.237327+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:05:49.004866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.595417+0000","last_change":"2023-10-16T12:26:24.295985+0000","last_active":"2023-10-16T12:26:25.595417+0000","last_peered":"2023-10-16T12:26:25.595417+0000","last_clean":"2023-10-16T12:26:25.595417+0000","last_became_active":"2023-10-16T12:26:24.295703+0000","last_became_peered":"2023-10-16T12:26:24.295703+0000","last_unstale":"2023-10-16T12:26:25.595417+0000","last_undegraded":"2023-10-16T12:26:25.595417+0000","last_fullsized":"2023-10-16T12:26:25.595417+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:53:28.283879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.13","version":"84'2","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.569138+0000","last_change":"2023-10-16T12:26:23.259511+0000","last_active":"2023-10-16T12:26:25.569138+0000","last_peered":"2023-10-16T12:26:25.569138+0000","last_clean":"2023-10-16T12:26:25.569138+0000","last_became_active":"2023-10-16T12:26:23.259189+0000","last_became_peered":"2023-10-16T12:26:23.259189+0000","last_unstale":"2023-10-16T12:26:25.569138+0000","last_undegraded":"2023-10-16T12:26:25.569138+0000","last_fullsized":"2023-10-16T12:26:25.569138+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_clean_scrub_stamp":"2023-10-16T12:26:22.237327+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:24:50.857165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.294984+0000","last_change":"2023-10-16T12:26:24.287755+0000","last_active":"2023-10-16T12:26:24.294984+0000","last_peered":"2023-10-16T12:26:24.294984+0000","last_clean":"2023-10-16T12:26:24.294984+0000","last_became_active":"2023-10-16T12:26:24.287450+0000","last_became_peered":"2023-10-16T12:26:24.287450+0000","last_unstale":"2023-10-16T12:26:24.294984+0000","last_undegraded":"2023-10-16T12:26:24.294984+0000","last_fullsized":"2023-10-16T12:26:24.294984+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:12:41.847476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.281035+0000","last_change":"2023-10-16T12:26:23.267748+0000","last_active":"2023-10-16T12:26:24.281035+0000","last_peered":"2023-10-16T12:26:24.281035+0000","last_clean":"2023-10-16T12:26:24.281035+0000","last_became_active":"2023-10-16T12:26:23.267464+0000","last_became_peered":"2023-10-16T12:26:23.267464+0000","last_unstale":"2023-10-16T12:26:24.281035+0000","last_undegraded":"2023-10-16T12:26:24.281035+0000","last_fullsized":"2023-10-16T12:26:24.281035+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_clean_scrub_stamp":"2023-10-16T12:26:22.237327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:42:08.818035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.289312+0000","last_change":"2023-10-16T12:26:24.281566+0000","last_active":"2023-10-16T12:26:24.289312+0000","last_peered":"2023-10-16T12:26:24.289312+0000","last_clean":"2023-10-16T12:26:24.289312+0000","last_became_active":"2023-10-16T12:26:24.281274+0000","last_became_peered":"2023-10-16T12:26:24.281274+0000","last_unstale":"2023-10-16T12:26:24.289312+0000","last_undegraded":"2023-10-16T12:26:24.289312+0000","last_fullsized":"2023-10-16T12:26:24.289312+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:24:31.347918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.569223+0000","last_change":"2023-10-16T12:26:23.264717+0000","last_active":"2023-10-16T12:26:25.569223+0000","last_peered":"2023-10-16T12:26:25.569223+0000","last_clean":"2023-10-16T12:26:25.569223+0000","last_became_active":"2023-10-16T12:26:23.264412+0000","last_became_peered":"2023-10-16T12:26:23.264412+0000","last_unstale":"2023-10-16T12:26:25.569223+0000","last_undegraded":"2023-10-16T12:26:25.569223+0000","last_fullsized":"2023-10-16T12:26:25.569223+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_clean_scrub_stamp":"2023-10-16T12:26:22.237327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:05:58.471940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.303949+0000","last_change":"2023-10-16T12:26:24.292328+0000","last_active":"2023-10-16T12:26:24.303949+0000","last_peered":"2023-10-16T12:26:24.303949+0000","last_clean":"2023-10-16T12:26:24.303949+0000","last_became_active":"2023-10-16T12:26:24.292043+0000","last_became_peered":"2023-10-16T12:26:24.292043+0000","last_unstale":"2023-10-16T12:26:24.303949+0000","last_undegraded":"2023-10-16T12:26:24.303949+0000","last_fullsized":"2023-10-16T12:26:24.303949+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:23:12.559260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"84'8","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.595337+0000","last_change":"2023-10-16T12:26:23.262544+0000","last_active":"2023-10-16T12:26:25.595337+0000","last_peered":"2023-10-16T12:26:25.595337+0000","last_clean":"2023-10-16T12:26:25.595337+0000","last_became_active":"2023-10-16T12:26:23.262262+0000","last_became_peered":"2023-10-16T12:26:23.262262+0000","last_unstale":"2023-10-16T12:26:25.595337+0000","last_undegraded":"2023-10-16T12:26:25.595337+0000","last_fullsized":"2023-10-16T12:26:25.595337+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_clean_scrub_stamp":"2023-10-16T12:26:22.237327+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:05:49.004866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.568992+0000","last_change":"2023-10-16T12:26:24.298699+0000","last_active":"2023-10-16T12:26:25.568992+0000","last_peered":"2023-10-16T12:26:25.568992+0000","last_clean":"2023-10-16T12:26:25.568992+0000","last_became_active":"2023-10-16T12:26:24.298273+0000","last_became_peered":"2023-10-16T12:26:24.298273+0000","last_unstale":"2023-10-16T12:26:25.568992+0000","last_undegraded":"2023-10-16T12:26:25.568992+0000","last_fullsized":"2023-10-16T12:26:25.568992+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:08:36.957709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"84'1","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.306220+0000","last_change":"2023-10-16T12:26:23.267470+0000","last_active":"2023-10-16T12:26:24.306220+0000","last_peered":"2023-10-16T12:26:24.306220+0000","last_clean":"2023-10-16T12:26:24.306220+0000","last_became_active":"2023-10-16T12:26:23.267162+0000","last_became_peered":"2023-10-16T12:26:23.267162+0000","last_unstale":"2023-10-16T12:26:24.306220+0000","last_undegraded":"2023-10-16T12:26:24.306220+0000","last_fullsized":"2023-10-16T12:26:24.306220+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_clean_scrub_stamp":"2023-10-16T12:26:22.237327+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:33:19.198155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.301636+0000","last_change":"2023-10-16T12:26:24.300661+0000","last_active":"2023-10-16T12:26:24.301636+0000","last_peered":"2023-10-16T12:26:24.301636+0000","last_clean":"2023-10-16T12:26:24.301636+0000","last_became_active":"2023-10-16T12:26:24.300286+0000","last_became_peered":"2023-10-16T12:26:24.300286+0000","last_unstale":"2023-10-16T12:26:24.301636+0000","last_undegraded":"2023-10-16T12:26:24.301636+0000","last_fullsized":"2023-10-16T12:26:24.301636+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:58:00.944373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"84'2","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.569005+0000","last_change":"2023-10-16T12:26:23.258970+0000","last_active":"2023-10-16T12:26:25.569005+0000","last_peered":"2023-10-16T12:26:25.569005+0000","last_clean":"2023-10-16T12:26:25.569005+0000","last_became_active":"2023-10-16T12:26:23.258519+0000","last_became_peered":"2023-10-16T12:26:23.258519+0000","last_unstale":"2023-10-16T12:26:25.569005+0000","last_undegraded":"2023-10-16T12:26:25.569005+0000","last_fullsized":"2023-10-16T12:26:25.569005+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_clean_scrub_stamp":"2023-10-16T12:26:22.237327+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:44:34.923567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.595664+0000","last_change":"2023-10-16T12:26:24.291315+0000","last_active":"2023-10-16T12:26:25.595664+0000","last_peered":"2023-10-16T12:26:25.595664+0000","last_clean":"2023-10-16T12:26:25.595664+0000","last_became_active":"2023-10-16T12:26:24.290993+0000","last_became_peered":"2023-10-16T12:26:24.290993+0000","last_unstale":"2023-10-16T12:26:25.595664+0000","last_undegraded":"2023-10-16T12:26:25.595664+0000","last_fullsized":"2023-10-16T12:26:25.595664+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:42:18.585503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"84'1","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.568994+0000","last_change":"2023-10-16T12:26:23.261915+0000","last_active":"2023-10-16T12:26:25.568994+0000","last_peered":"2023-10-16T12:26:25.568994+0000","last_clean":"2023-10-16T12:26:25.568994+0000","last_became_active":"2023-10-16T12:26:23.261529+0000","last_became_peered":"2023-10-16T12:26:23.261529+0000","last_unstale":"2023-10-16T12:26:25.568994+0000","last_undegraded":"2023-10-16T12:26:25.568994+0000","last_fullsized":"2023-10-16T12:26:25.568994+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_clean_scrub_stamp":"2023-10-16T12:26:22.237327+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:42:08.818035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.294823+0000","last_change":"2023-10-16T12:26:24.289953+0000","last_active":"2023-10-16T12:26:24.294823+0000","last_peered":"2023-10-16T12:26:24.294823+0000","last_clean":"2023-10-16T12:26:24.294823+0000","last_became_active":"2023-10-16T12:26:24.289677+0000","last_became_peered":"2023-10-16T12:26:24.289677+0000","last_unstale":"2023-10-16T12:26:24.294823+0000","last_undegraded":"2023-10-16T12:26:24.294823+0000","last_fullsized":"2023-10-16T12:26:24.294823+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:38:06.705069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.288407+0000","last_change":"2023-10-16T12:26:23.262042+0000","last_active":"2023-10-16T12:26:25.288407+0000","last_peered":"2023-10-16T12:26:25.288407+0000","last_clean":"2023-10-16T12:26:25.288407+0000","last_became_active":"2023-10-16T12:26:23.261710+0000","last_became_peered":"2023-10-16T12:26:23.261710+0000","last_unstale":"2023-10-16T12:26:25.288407+0000","last_undegraded":"2023-10-16T12:26:25.288407+0000","last_fullsized":"2023-10-16T12:26:25.288407+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_clean_scrub_stamp":"2023-10-16T12:26:22.237327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:55:24.362404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.288134+0000","last_change":"2023-10-16T12:26:24.285641+0000","last_active":"2023-10-16T12:26:25.288134+0000","last_peered":"2023-10-16T12:26:25.288134+0000","last_clean":"2023-10-16T12:26:25.288134+0000","last_became_active":"2023-10-16T12:26:24.285341+0000","last_became_peered":"2023-10-16T12:26:24.285341+0000","last_unstale":"2023-10-16T12:26:25.288134+0000","last_undegraded":"2023-10-16T12:26:25.288134+0000","last_fullsized":"2023-10-16T12:26:25.288134+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:55:25.366467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"84'2","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.288703+0000","last_change":"2023-10-16T12:26:23.258309+0000","last_active":"2023-10-16T12:26:25.288703+0000","last_peered":"2023-10-16T12:26:25.288703+0000","last_clean":"2023-10-16T12:26:25.288703+0000","last_became_active":"2023-10-16T12:26:23.258013+0000","last_became_peered":"2023-10-16T12:26:23.258013+0000","last_unstale":"2023-10-16T12:26:25.288703+0000","last_undegraded":"2023-10-16T12:26:25.288703+0000","last_fullsized":"2023-10-16T12:26:25.288703+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_clean_scrub_stamp":"2023-10-16T12:26:22.237327+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:00:36.231552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.289658+0000","last_change":"2023-10-16T12:26:24.286194+0000","last_active":"2023-10-16T12:26:24.289658+0000","last_peered":"2023-10-16T12:26:24.289658+0000","last_clean":"2023-10-16T12:26:24.289658+0000","last_became_active":"2023-10-16T12:26:24.285745+0000","last_became_peered":"2023-10-16T12:26:24.285745+0000","last_unstale":"2023-10-16T12:26:24.289658+0000","last_undegraded":"2023-10-16T12:26:24.289658+0000","last_fullsized":"2023-10-16T12:26:24.289658+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:32:19.269409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.281199+0000","last_change":"2023-10-16T12:26:23.267368+0000","last_active":"2023-10-16T12:26:24.281199+0000","last_peered":"2023-10-16T12:26:24.281199+0000","last_clean":"2023-10-16T12:26:24.281199+0000","last_became_active":"2023-10-16T12:26:23.267067+0000","last_became_peered":"2023-10-16T12:26:23.267067+0000","last_unstale":"2023-10-16T12:26:24.281199+0000","last_undegraded":"2023-10-16T12:26:24.281199+0000","last_fullsized":"2023-10-16T12:26:24.281199+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_clean_scrub_stamp":"2023-10-16T12:26:22.237327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:24:48.895339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.277754+0000","last_change":"2023-10-16T12:26:24.276796+0000","last_active":"2023-10-16T12:26:24.277754+0000","last_peered":"2023-10-16T12:26:24.277754+0000","last_clean":"2023-10-16T12:26:24.277754+0000","last_became_active":"2023-10-16T12:26:24.276445+0000","last_became_peered":"2023-10-16T12:26:24.276445+0000","last_unstale":"2023-10-16T12:26:24.277754+0000","last_undegraded":"2023-10-16T12:26:24.277754+0000","last_fullsized":"2023-10-16T12:26:24.277754+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:56:07.516751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.287973+0000","last_change":"2023-10-16T12:26:23.252571+0000","last_active":"2023-10-16T12:26:25.287973+0000","last_peered":"2023-10-16T12:26:25.287973+0000","last_clean":"2023-10-16T12:26:25.287973+0000","last_became_active":"2023-10-16T12:26:23.252277+0000","last_became_peered":"2023-10-16T12:26:23.252277+0000","last_unstale":"2023-10-16T12:26:25.287973+0000","last_undegraded":"2023-10-16T12:26:25.287973+0000","last_fullsized":"2023-10-16T12:26:25.287973+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_clean_scrub_stamp":"2023-10-16T12:26:22.237327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:21:45.893418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.288623+0000","last_change":"2023-10-16T12:26:24.297532+0000","last_active":"2023-10-16T12:26:25.288623+0000","last_peered":"2023-10-16T12:26:25.288623+0000","last_clean":"2023-10-16T12:26:25.288623+0000","last_became_active":"2023-10-16T12:26:24.294140+0000","last_became_peered":"2023-10-16T12:26:24.294140+0000","last_unstale":"2023-10-16T12:26:25.288623+0000","last_undegraded":"2023-10-16T12:26:25.288623+0000","last_fullsized":"2023-10-16T12:26:25.288623+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:51:19.596991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"84'2","reported_seq":13,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.308957+0000","last_change":"2023-10-16T12:26:23.267218+0000","last_active":"2023-10-16T12:26:24.308957+0000","last_peered":"2023-10-16T12:26:24.308957+0000","last_clean":"2023-10-16T12:26:24.308957+0000","last_became_active":"2023-10-16T12:26:23.266912+0000","last_became_peered":"2023-10-16T12:26:23.266912+0000","last_unstale":"2023-10-16T12:26:24.308957+0000","last_undegraded":"2023-10-16T12:26:24.308957+0000","last_fullsized":"2023-10-16T12:26:24.308957+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_clean_scrub_stamp":"2023-10-16T12:26:22.237327+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:24:50.857165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.302789+0000","last_change":"2023-10-16T12:26:24.297628+0000","last_active":"2023-10-16T12:26:24.302789+0000","last_peered":"2023-10-16T12:26:24.302789+0000","last_clean":"2023-10-16T12:26:24.302789+0000","last_became_active":"2023-10-16T12:26:24.297296+0000","last_became_peered":"2023-10-16T12:26:24.297296+0000","last_unstale":"2023-10-16T12:26:24.302789+0000","last_undegraded":"2023-10-16T12:26:24.302789+0000","last_fullsized":"2023-10-16T12:26:24.302789+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:59:42.505808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.281373+0000","last_change":"2023-10-16T12:26:23.266323+0000","last_active":"2023-10-16T12:26:24.281373+0000","last_peered":"2023-10-16T12:26:24.281373+0000","last_clean":"2023-10-16T12:26:24.281373+0000","last_became_active":"2023-10-16T12:26:23.265989+0000","last_became_peered":"2023-10-16T12:26:23.265989+0000","last_unstale":"2023-10-16T12:26:24.281373+0000","last_undegraded":"2023-10-16T12:26:24.281373+0000","last_fullsized":"2023-10-16T12:26:24.281373+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_clean_scrub_stamp":"2023-10-16T12:26:22.237327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:44:34.923567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.295094+0000","last_change":"2023-10-16T12:26:24.289242+0000","last_active":"2023-10-16T12:26:24.295094+0000","last_peered":"2023-10-16T12:26:24.295094+0000","last_clean":"2023-10-16T12:26:24.295094+0000","last_became_active":"2023-10-16T12:26:24.288955+0000","last_became_peered":"2023-10-16T12:26:24.288955+0000","last_unstale":"2023-10-16T12:26:24.295094+0000","last_undegraded":"2023-10-16T12:26:24.295094+0000","last_fullsized":"2023-10-16T12:26:24.295094+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:50:59.014744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.288675+0000","last_change":"2023-10-16T12:26:23.267396+0000","last_active":"2023-10-16T12:26:25.288675+0000","last_peered":"2023-10-16T12:26:25.288675+0000","last_clean":"2023-10-16T12:26:25.288675+0000","last_became_active":"2023-10-16T12:26:23.267099+0000","last_became_peered":"2023-10-16T12:26:23.267099+0000","last_unstale":"2023-10-16T12:26:25.288675+0000","last_undegraded":"2023-10-16T12:26:25.288675+0000","last_fullsized":"2023-10-16T12:26:25.288675+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_clean_scrub_stamp":"2023-10-16T12:26:22.237327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:40:04.791545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.294849+0000","last_change":"2023-10-16T12:26:24.290108+0000","last_active":"2023-10-16T12:26:24.294849+0000","last_peered":"2023-10-16T12:26:24.294849+0000","last_clean":"2023-10-16T12:26:24.294849+0000","last_became_active":"2023-10-16T12:26:24.289811+0000","last_became_peered":"2023-10-16T12:26:24.289811+0000","last_unstale":"2023-10-16T12:26:24.294849+0000","last_undegraded":"2023-10-16T12:26:24.294849+0000","last_fullsized":"2023-10-16T12:26:24.294849+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:43:17.002196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.279814+0000","last_change":"2023-10-16T12:26:23.268364+0000","last_active":"2023-10-16T12:26:24.279814+0000","last_peered":"2023-10-16T12:26:24.279814+0000","last_clean":"2023-10-16T12:26:24.279814+0000","last_became_active":"2023-10-16T12:26:23.265514+0000","last_became_peered":"2023-10-16T12:26:23.265514+0000","last_unstale":"2023-10-16T12:26:24.279814+0000","last_undegraded":"2023-10-16T12:26:24.279814+0000","last_fullsized":"2023-10-16T12:26:24.279814+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_clean_scrub_stamp":"2023-10-16T12:26:22.237327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:53:21.641122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.289743+0000","last_change":"2023-10-16T12:26:24.283228+0000","last_active":"2023-10-16T12:26:24.289743+0000","last_peered":"2023-10-16T12:26:24.289743+0000","last_clean":"2023-10-16T12:26:24.289743+0000","last_became_active":"2023-10-16T12:26:24.282921+0000","last_became_peered":"2023-10-16T12:26:24.282921+0000","last_unstale":"2023-10-16T12:26:24.289743+0000","last_undegraded":"2023-10-16T12:26:24.289743+0000","last_fullsized":"2023-10-16T12:26:24.289743+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:12:45.626596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.569267+0000","last_change":"2023-10-16T12:26:23.259640+0000","last_active":"2023-10-16T12:26:25.569267+0000","last_peered":"2023-10-16T12:26:25.569267+0000","last_clean":"2023-10-16T12:26:25.569267+0000","last_became_active":"2023-10-16T12:26:23.259351+0000","last_became_peered":"2023-10-16T12:26:23.259351+0000","last_unstale":"2023-10-16T12:26:25.569267+0000","last_undegraded":"2023-10-16T12:26:25.569267+0000","last_fullsized":"2023-10-16T12:26:25.569267+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_clean_scrub_stamp":"2023-10-16T12:26:22.237327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:33:19.198155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.595509+0000","last_change":"2023-10-16T12:26:24.297109+0000","last_active":"2023-10-16T12:26:25.595509+0000","last_peered":"2023-10-16T12:26:25.595509+0000","last_clean":"2023-10-16T12:26:25.595509+0000","last_became_active":"2023-10-16T12:26:24.296614+0000","last_became_peered":"2023-10-16T12:26:24.296614+0000","last_unstale":"2023-10-16T12:26:25.595509+0000","last_undegraded":"2023-10-16T12:26:25.595509+0000","last_fullsized":"2023-10-16T12:26:25.595509+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:28:47.753397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.288018+0000","last_change":"2023-10-16T12:26:23.267165+0000","last_active":"2023-10-16T12:26:25.288018+0000","last_peered":"2023-10-16T12:26:25.288018+0000","last_clean":"2023-10-16T12:26:25.288018+0000","last_became_active":"2023-10-16T12:26:23.266875+0000","last_became_peered":"2023-10-16T12:26:23.266875+0000","last_unstale":"2023-10-16T12:26:25.288018+0000","last_undegraded":"2023-10-16T12:26:25.288018+0000","last_fullsized":"2023-10-16T12:26:25.288018+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_clean_scrub_stamp":"2023-10-16T12:26:22.237327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:34:46.883916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.304828+0000","last_change":"2023-10-16T12:26:24.299736+0000","last_active":"2023-10-16T12:26:24.304828+0000","last_peered":"2023-10-16T12:26:24.304828+0000","last_clean":"2023-10-16T12:26:24.304828+0000","last_became_active":"2023-10-16T12:26:24.299445+0000","last_became_peered":"2023-10-16T12:26:24.299445+0000","last_unstale":"2023-10-16T12:26:24.304828+0000","last_undegraded":"2023-10-16T12:26:24.304828+0000","last_fullsized":"2023-10-16T12:26:24.304828+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:53:28.283879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.288065+0000","last_change":"2023-10-16T12:26:23.263113+0000","last_active":"2023-10-16T12:26:25.288065+0000","last_peered":"2023-10-16T12:26:25.288065+0000","last_clean":"2023-10-16T12:26:25.288065+0000","last_became_active":"2023-10-16T12:26:23.262795+0000","last_became_peered":"2023-10-16T12:26:23.262795+0000","last_unstale":"2023-10-16T12:26:25.288065+0000","last_undegraded":"2023-10-16T12:26:25.288065+0000","last_fullsized":"2023-10-16T12:26:25.288065+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_clean_scrub_stamp":"2023-10-16T12:26:22.237327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:41:42.968510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.292380+0000","last_change":"2023-10-16T12:26:24.290731+0000","last_active":"2023-10-16T12:26:24.292380+0000","last_peered":"2023-10-16T12:26:24.292380+0000","last_clean":"2023-10-16T12:26:24.292380+0000","last_became_active":"2023-10-16T12:26:24.290417+0000","last_became_peered":"2023-10-16T12:26:24.290417+0000","last_unstale":"2023-10-16T12:26:24.292380+0000","last_undegraded":"2023-10-16T12:26:24.292380+0000","last_fullsized":"2023-10-16T12:26:24.292380+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:05:21.201046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.278612+0000","last_change":"2023-10-16T12:26:23.262045+0000","last_active":"2023-10-16T12:26:24.278612+0000","last_peered":"2023-10-16T12:26:24.278612+0000","last_clean":"2023-10-16T12:26:24.278612+0000","last_became_active":"2023-10-16T12:26:23.261702+0000","last_became_peered":"2023-10-16T12:26:23.261702+0000","last_unstale":"2023-10-16T12:26:24.278612+0000","last_undegraded":"2023-10-16T12:26:24.278612+0000","last_fullsized":"2023-10-16T12:26:24.278612+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_clean_scrub_stamp":"2023-10-16T12:26:22.237327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:28:49.990521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.295053+0000","last_change":"2023-10-16T12:26:24.292589+0000","last_active":"2023-10-16T12:26:24.295053+0000","last_peered":"2023-10-16T12:26:24.295053+0000","last_clean":"2023-10-16T12:26:24.295053+0000","last_became_active":"2023-10-16T12:26:24.292235+0000","last_became_peered":"2023-10-16T12:26:24.292235+0000","last_unstale":"2023-10-16T12:26:24.295053+0000","last_undegraded":"2023-10-16T12:26:24.295053+0000","last_fullsized":"2023-10-16T12:26:24.295053+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:23.241390+0000","last_clean_scrub_stamp":"2023-10-16T12:26:23.241390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:38:04.765408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-16T12:26:25.287747+0000","last_change":"2023-10-16T12:26:23.265387+0000","last_active":"2023-10-16T12:26:25.287747+0000","last_peered":"2023-10-16T12:26:25.287747+0000","last_clean":"2023-10-16T12:26:25.287747+0000","last_became_active":"2023-10-16T12:26:23.265077+0000","last_became_peered":"2023-10-16T12:26:23.265077+0000","last_unstale":"2023-10-16T12:26:25.287747+0000","last_undegraded":"2023-10-16T12:26:25.287747+0000","last_fullsized":"2023-10-16T12:26:25.287747+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:22.237327+0000","last_clean_scrub_stamp":"2023-10-16T12:26:22.237327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:24:04.496264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"75'161","reported_seq":385,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-16T12:26:24.283779+0000","last_change":"2023-10-16T12:10:34.746820+0000","last_active":"2023-10-16T12:26:24.283779+0000","last_peered":"2023-10-16T12:26:24.283779+0000","last_clean":"2023-10-16T12:26:24.283779+0000","last_became_active":"2023-10-16T12:10:34.745574+0000","last_became_peered":"2023-10-16T12:10:34.745574+0000","last_unstale":"2023-10-16T12:26:24.283779+0000","last_undegraded":"2023-10-16T12:26:24.283779+0000","last_fullsized":"2023-10-16T12:26:24.283779+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_clean_scrub_stamp":"2023-10-16T12:10:33.577111+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:14.461986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575041,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59156,"kb_used_data":1604,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312684,"statfs":{"total":96636764160,"available":96576188416,"internally_reserved":0,"allocated":1642496,"data_stored":1105024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58915275},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:26:05 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.59699999999999998,"15min":0.58499999999999996},"min":{"1min":0.437,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.59399999999999997,"15min":0.58199999999999996},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.312},"max":{"1min":0.85299999999999998,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.78100000000000003}]},{"osd":1,"last update":"Mon Oct 16 12:26:05 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.60399999999999998,"15min":0.59999999999999998},"min":{"1min":0.36099999999999999,"5min":0.35999999999999999,"15min":0.247},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.62,"15min":0.60099999999999998},"min":{"1min":0.45600000000000002,"5min":0.33200000000000002,"15min":0.32400000000000001},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.64100000000000001}]},{"osd":2,"last update":"Mon Oct 16 12:26:05 2023","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.67600000000000005,"15min":0.65600000000000003},"min":{"1min":0.53700000000000003,"5min":0.40699999999999997,"15min":0.36099999999999999},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.68799999999999994,"15min":0.66600000000000004},"min":{"1min":0.495,"5min":0.376,"15min":0.36699999999999999},"max":{"1min":0.94599999999999995,"5min":0.94799999999999995,"15min":0.96699999999999997},"last":0.53800000000000003}]},{"osd":3,"last update":"Mon Oct 16 12:26:05 2023","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.63,"15min":0.61899999999999999},"min":{"1min":0.43099999999999999,"5min":0.35499999999999998,"15min":0.34899999999999998},"max":{"1min":0.91200000000000003,"5min":0.92200000000000004,"15min":1.1000000000000001},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.65700000000000003,"15min":0.63900000000000001},"min":{"1min":0.52200000000000002,"5min":0.376,"15min":0.376},"max":{"1min":0.996,"5min":1.2509999999999999,"15min":1.2509999999999999},"last":0.72299999999999998}]},{"osd":4,"last update":"Mon Oct 16 12:26:05 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53800000000000003,"15min":0.53100000000000003},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.28499999999999998},"max":{"1min":0.72699999999999998,"5min":1.383,"15min":1.383},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.625,"15min":0.60699999999999998},"min":{"1min":0.40799999999999997,"5min":0.36299999999999999,"15min":0.307},"max":{"1min":0.86599999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.59199999999999997}]},{"osd":5,"last update":"Mon Oct 16 12:26:05 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.55800000000000005,"15min":0.54300000000000004},"min":{"1min":0.40000000000000002,"5min":0.314,"15min":0.29799999999999999},"max":{"1min":0.84299999999999997,"5min":1.3109999999999999,"15min":1.3109999999999999},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.59399999999999997,"15min":0.58699999999999997},"min":{"1min":0.41799999999999998,"5min":0.33500000000000002,"15min":0.32400000000000001},"max":{"1min":0.83899999999999997,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":0.50900000000000001}]},{"osd":6,"last update":"Mon Oct 16 12:26:05 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.63500000000000001,"15min":0.621},"min":{"1min":0.44700000000000001,"5min":0.39300000000000002,"15min":0.36399999999999999},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.65600000000000003,"15min":0.63100000000000001},"min":{"1min":0.443,"5min":0.34599999999999997,"15min":0.32300000000000001},"max":{"1min":0.88300000000000001,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.66100000000000003}]}]},{"osd":6,"up_from":13,"seq":55834575039,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32180,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339660,"statfs":{"total":96636764160,"available":96603811840,"internally_reserved":0,"allocated":1024000,"data_stored":507266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:25:58 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.60099999999999998,"15min":0.628},"min":{"1min":0.34200000000000003,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.88200000000000001,"5min":1.196,"15min":5.9349999999999996},"last":0.628},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60499999999999998,"15min":0.64000000000000001},"min":{"1min":0.44900000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.81799999999999995,"5min":0.997,"15min":5.9489999999999998},"last":0.59199999999999997}]},{"osd":1,"last update":"Mon Oct 16 12:25:58 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.55500000000000005,"15min":0.57499999999999996},"min":{"1min":0.40200000000000002,"5min":0.32800000000000001,"15min":0.29399999999999998},"max":{"1min":0.71199999999999997,"5min":0.83899999999999997,"15min":0.875},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57899999999999996,"15min":0.59199999999999997},"min":{"1min":0.44500000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.83099999999999996,"5min":1.091,"15min":1.091},"last":0.48299999999999998}]},{"osd":2,"last update":"Mon Oct 16 12:25:58 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.57799999999999996,"15min":0.61799999999999999},"min":{"1min":0.41699999999999998,"5min":0.36199999999999999,"15min":0.34499999999999997},"max":{"1min":0.69199999999999995,"5min":0.91200000000000003,"15min":8.0489999999999995},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.57699999999999996,"15min":0.60899999999999999},"min":{"1min":0.41099999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.76300000000000001,"5min":1.1539999999999999,"15min":2.855},"last":0.44900000000000001}]},{"osd":3,"last update":"Mon Oct 16 12:25:58 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.60599999999999998,"15min":0.629},"min":{"1min":0.45500000000000002,"5min":0.34200000000000003,"15min":0.307},"max":{"1min":0.85399999999999998,"5min":0.93999999999999995,"15min":1.083},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67400000000000004,"15min":0.67900000000000005},"min":{"1min":0.48099999999999998,"5min":0.40899999999999997,"15min":0.36599999999999999},"max":{"1min":0.94499999999999995,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.55400000000000005}]},{"osd":4,"last update":"Mon Oct 16 12:25:58 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.52800000000000002,"15min":0.52000000000000002},"min":{"1min":0.33900000000000002,"5min":0.308,"15min":0.29899999999999999},"max":{"1min":0.83399999999999996,"5min":0.85599999999999998,"15min":0.88200000000000001},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.60299999999999998,"15min":0.59399999999999997},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.30199999999999999},"max":{"1min":0.78500000000000003,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.65000000000000002}]},{"osd":5,"last update":"Mon Oct 16 12:25:58 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59099999999999997,"15min":0.57799999999999996},"min":{"1min":0.39900000000000002,"5min":0.34799999999999998,"15min":0.30099999999999999},"max":{"1min":0.80100000000000005,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63400000000000001,"15min":0.624},"min":{"1min":0.39400000000000002,"5min":0.35099999999999998,"15min":0.311},"max":{"1min":0.81599999999999995,"5min":1.3540000000000001,"15min":1.3540000000000001},"last":0.70099999999999996}]},{"osd":7,"last update":"Mon Oct 16 12:25:58 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59499999999999997,"15min":0.57499999999999996},"min":{"1min":0.36499999999999999,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.86599999999999999,"5min":1.29,"15min":1.29},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.62,"5min":0.625,"15min":0.61799999999999999},"min":{"1min":0.38800000000000001,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.84399999999999997,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.61299999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575039,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32204,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339636,"statfs":{"total":96636764160,"available":96603787264,"internally_reserved":0,"allocated":1048576,"data_stored":509608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":31914441},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:26:00 2023","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.373,"15min":0.378},"min":{"1min":0.28199999999999997,"5min":0.21299999999999999,"15min":0.186},"max":{"1min":0.52700000000000002,"5min":0.63600000000000001,"15min":0.66000000000000003},"last":0.314},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.45900000000000002,"15min":0.46000000000000002},"min":{"1min":0.33000000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.65700000000000003,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.28399999999999997}]},{"osd":2,"last update":"Mon Oct 16 12:26:00 2023","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.41399999999999998,"15min":0.42899999999999999},"min":{"1min":0.28699999999999998,"5min":0.25,"15min":0.215},"max":{"1min":0.61499999999999999,"5min":0.69199999999999995,"15min":0.93000000000000005},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.46300000000000002,"15min":0.46400000000000002},"min":{"1min":0.32300000000000001,"5min":0.28199999999999997,"15min":0.24399999999999999},"max":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.98099999999999998},"last":0.372}]},{"osd":3,"last update":"Mon Oct 16 12:26:00 2023","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.40200000000000002,"15min":0.40400000000000003},"min":{"1min":0.32500000000000001,"5min":0.25900000000000001,"15min":0.23599999999999999},"max":{"1min":0.69599999999999995,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.245},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.44,"15min":0.45700000000000002},"min":{"1min":0.33800000000000002,"5min":0.252,"15min":0.22900000000000001},"max":{"1min":0.60199999999999998,"5min":0.91200000000000003,"15min":1.127},"last":0.26100000000000001}]},{"osd":4,"last update":"Mon Oct 16 12:26:00 2023","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.61799999999999999,"15min":0.63100000000000001},"min":{"1min":0.44900000000000001,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.76600000000000001,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.749},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.628,"15min":0.63900000000000001},"min":{"1min":0.46999999999999997,"5min":0.39100000000000001,"15min":0.34699999999999998},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":1.089},"last":0.65000000000000002}]},{"osd":5,"last update":"Mon Oct 16 12:26:00 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.63700000000000001,"15min":0.66600000000000004},"min":{"1min":0.47799999999999998,"5min":0.38300000000000001,"15min":0.28499999999999998},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.68100000000000005,"15min":0.68600000000000005},"min":{"1min":0.47399999999999998,"5min":0.38900000000000001,"15min":0.32500000000000001},"max":{"1min":0.98999999999999999,"5min":1.0620000000000001,"15min":1.0649999999999999},"last":0.71399999999999997}]},{"osd":6,"last update":"Mon Oct 16 12:26:00 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.63600000000000001,"15min":0.65300000000000002},"min":{"1min":0.502,"5min":0.378,"15min":0.378},"max":{"1min":0.876,"5min":0.94999999999999996,"15min":1.0129999999999999},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.65800000000000003,"15min":0.66200000000000003},"min":{"1min":0.51500000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":1.0609999999999999},"last":0.63700000000000001}]},{"osd":7,"last update":"Mon Oct 16 12:26:00 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.68899999999999995,"15min":0.68799999999999994},"min":{"1min":0.49299999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.0469999999999999,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.68300000000000005,"15min":0.70299999999999996},"min":{"1min":0.60699999999999998,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":1.167},"last":0.68500000000000005}]}]},{"osd":0,"up_from":13,"seq":55834575040,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36936,"kb_used_data":1588,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334904,"statfs":{"total":96636764160,"available":96598941696,"internally_reserved":0,"allocated":1626112,"data_stored":1104248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 12:25:54 2023","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.38900000000000001,"15min":0.38600000000000001},"min":{"1min":0.307,"5min":0.223,"15min":0.223},"max":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.73699999999999999},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.41699999999999998,"15min":0.41799999999999998},"min":{"1min":0.27300000000000002,"5min":0.24299999999999999,"15min":0.23200000000000001},"max":{"1min":0.64700000000000002,"5min":0.71799999999999997,"15min":0.73099999999999998},"last":0.34100000000000003}]},{"osd":2,"last update":"Mon Oct 16 12:25:54 2023","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.42799999999999999,"15min":0.42499999999999999},"min":{"1min":0.32600000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.628,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.33300000000000002,"5min":0.29599999999999999,"15min":0.27900000000000003},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.40899999999999997}]},{"osd":3,"last update":"Mon Oct 16 12:25:54 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.432,"15min":0.42099999999999999},"min":{"1min":0.372,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.57299999999999995,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.38},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.48199999999999998,"15min":0.48499999999999999},"min":{"1min":0.37,"5min":0.29099999999999998,"15min":0.27600000000000002},"max":{"1min":0.66400000000000003,"5min":0.68500000000000005,"15min":0.878},"last":0.32500000000000001}]},{"osd":4,"last update":"Mon Oct 16 12:25:54 2023","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.65300000000000002,"15min":0.64800000000000002},"min":{"1min":0.45100000000000001,"5min":0.377,"15min":0.36699999999999999},"max":{"1min":0.85999999999999999,"5min":0.90700000000000003,"15min":1.0960000000000001},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64500000000000002,"15min":0.65500000000000003},"min":{"1min":0.52600000000000002,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.81499999999999995,"5min":0.94999999999999996,"15min":1.1719999999999999},"last":0.59499999999999997}]},{"osd":5,"last update":"Mon Oct 16 12:25:54 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.65100000000000002,"15min":0.65000000000000002},"min":{"1min":0.60199999999999998,"5min":0.374,"15min":0.34399999999999997},"max":{"1min":0.84499999999999997,"5min":1,"15min":1.127},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.67000000000000004,"15min":0.67900000000000005},"min":{"1min":0.54500000000000004,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.874,"5min":0.92900000000000005,"15min":1.1160000000000001},"last":0.58099999999999996}]},{"osd":6,"last update":"Mon Oct 16 12:25:54 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.67400000000000004,"15min":0.68100000000000005},"min":{"1min":0.56699999999999995,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.91400000000000003,"5min":0.98599999999999999,"15min":1.079},"last":0.627},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.67400000000000004,"15min":0.68500000000000005},"min":{"1min":0.56299999999999994,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.92100000000000004,"5min":0.95899999999999996,"15min":1.1499999999999999},"last":0.48999999999999999}]},{"osd":7,"last update":"Mon Oct 16 12:25:54 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.68400000000000005,"15min":0.69599999999999995},"min":{"1min":0.61399999999999999,"5min":0.41499999999999998,"15min":0.40699999999999997},"max":{"1min":0.88800000000000001,"5min":0.999,"15min":1.0349999999999999},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.67300000000000004,"15min":0.71199999999999997},"min":{"1min":0.56000000000000005,"5min":0.41899999999999998,"15min":0.41299999999999998},"max":{"1min":0.84899999999999998,"5min":1.0329999999999999,"15min":6.7300000000000004},"last":0.50800000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575040,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58544,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313296,"statfs":{"total":96636764160,"available":96576815104,"internally_reserved":0,"allocated":1019904,"data_stored":507232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:25:55 2023","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.38,"15min":0.38200000000000001},"min":{"1min":0.29699999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.48299999999999998,"5min":0.59499999999999997,"15min":0.73799999999999999},"last":0.378},{"interface":"front","average":{"1min":0.498,"5min":0.45800000000000002,"15min":0.44900000000000001},"min":{"1min":0.379,"5min":0.29299999999999998,"15min":0.26400000000000001},"max":{"1min":0.66500000000000004,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.31900000000000001}]},{"osd":1,"last update":"Mon Oct 16 12:25:55 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.46899999999999997,"15min":0.46600000000000003},"min":{"1min":0.39000000000000001,"5min":0.25600000000000001,"15min":0.254},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.439,"15min":0.436},"min":{"1min":0.34599999999999997,"5min":0.27700000000000002,"15min":0.246},"max":{"1min":0.59199999999999997,"5min":0.69699999999999995,"15min":0.76500000000000001},"last":0.442}]},{"osd":3,"last update":"Mon Oct 16 12:25:55 2023","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.41899999999999998,"15min":0.41499999999999998},"min":{"1min":0.31900000000000001,"5min":0.27700000000000002,"15min":0.26300000000000001},"max":{"1min":0.63600000000000001,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.51800000000000002,"15min":0.51600000000000001},"min":{"1min":0.44400000000000001,"5min":0.318,"15min":0.246},"max":{"1min":0.78000000000000003,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.36599999999999999}]},{"osd":4,"last update":"Mon Oct 16 12:25:55 2023","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.68999999999999995,"15min":0.68600000000000005},"min":{"1min":0.56499999999999995,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.91100000000000003,"5min":1.3839999999999999,"15min":1.3839999999999999},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.69699999999999995,"15min":0.68600000000000005},"min":{"1min":0.54900000000000004,"5min":0.46400000000000002,"15min":0.36699999999999999},"max":{"1min":0.97599999999999998,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.51800000000000002}]},{"osd":5,"last update":"Mon Oct 16 12:25:55 2023","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.67700000000000005,"15min":0.66600000000000004},"min":{"1min":0.56499999999999995,"5min":0.40200000000000002,"15min":0.312},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":0.504},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.67800000000000005,"15min":0.66700000000000004},"min":{"1min":0.51800000000000002,"5min":0.41299999999999998,"15min":0.317},"max":{"1min":1.0089999999999999,"5min":1.331,"15min":1.331},"last":0.60599999999999998}]},{"osd":6,"last update":"Mon Oct 16 12:25:55 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.65900000000000003,"15min":0.64400000000000002},"min":{"1min":0.497,"5min":0.439,"15min":0.34699999999999998},"max":{"1min":0.99299999999999999,"5min":1.036,"15min":1.046},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.70299999999999996,"15min":0.68999999999999995},"min":{"1min":0.63,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.032,"5min":1.032,"15min":1.0389999999999999},"last":0.54600000000000004}]},{"osd":7,"last update":"Mon Oct 16 12:25:55 2023","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.68500000000000005,"15min":0.68200000000000005},"min":{"1min":0.53300000000000003,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.96399999999999997,"5min":1.016,"15min":1.026},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.754,"5min":0.69999999999999996,"15min":0.69399999999999995},"min":{"1min":0.53800000000000003,"5min":0.442,"15min":0.40600000000000003},"max":{"1min":1.145,"5min":1.145,"15min":1.145},"last":0.59099999999999997}]}]},{"osd":3,"up_from":13,"seq":55834575039,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36340,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335500,"statfs":{"total":96636764160,"available":96599552000,"internally_reserved":0,"allocated":1019904,"data_stored":507232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:26:02 2023","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.40100000000000002,"15min":0.39400000000000002},"min":{"1min":0.29499999999999998,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.435,"5min":0.41199999999999998,"15min":0.40500000000000003},"min":{"1min":0.28999999999999998,"5min":0.255,"15min":0.22600000000000001},"max":{"1min":0.61899999999999999,"5min":0.63100000000000001,"15min":0.73099999999999998},"last":0.42799999999999999}]},{"osd":1,"last update":"Mon Oct 16 12:26:02 2023","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.41999999999999998,"15min":0.40999999999999998},"min":{"1min":0.29499999999999998,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.64100000000000001,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.374},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.46300000000000002,"15min":0.45800000000000002},"min":{"1min":0.38900000000000001,"5min":0.25800000000000001,"15min":0.25700000000000001},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.747},"last":0.45600000000000002}]},{"osd":2,"last update":"Mon Oct 16 12:26:02 2023","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.47399999999999998,"15min":0.46800000000000003},"min":{"1min":0.39600000000000002,"5min":0.307,"15min":0.26900000000000002},"max":{"1min":0.63700000000000001,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.503,"15min":0.49199999999999999},"min":{"1min":0.42199999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.60299999999999998,"5min":0.752,"15min":0.752},"last":0.54200000000000004}]},{"osd":4,"last update":"Mon Oct 16 12:26:02 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.63200000000000001,"15min":0.64500000000000002},"min":{"1min":0.49399999999999999,"5min":0.39100000000000001,"15min":0.377},"max":{"1min":0.81699999999999995,"5min":0.92700000000000005,"15min":1.1539999999999999},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.65400000000000003,"15min":0.66800000000000004},"min":{"1min":0.55700000000000005,"5min":0.44900000000000001,"15min":0.40200000000000002},"max":{"1min":0.90600000000000003,"5min":1.0409999999999999,"15min":1.234},"last":0.53100000000000003}]},{"osd":5,"last update":"Mon Oct 16 12:26:02 2023","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.68600000000000005,"15min":0.68300000000000005},"min":{"1min":0.48899999999999999,"5min":0.443,"15min":0.41199999999999998},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.194},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.65600000000000003,"15min":0.66700000000000004},"min":{"1min":0.56000000000000005,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":1.202},"last":0.63}]},{"osd":6,"last update":"Mon Oct 16 12:26:02 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.67500000000000004,"15min":0.67400000000000004},"min":{"1min":0.53300000000000003,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.94199999999999995,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.75,"5min":0.70599999999999996,"15min":0.70699999999999996},"min":{"1min":0.60099999999999998,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.1559999999999999},"last":0.67700000000000005}]},{"osd":7,"last update":"Mon Oct 16 12:26:02 2023","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.68000000000000005,"15min":0.68899999999999995},"min":{"1min":0.499,"5min":0.435,"15min":0.38},"max":{"1min":0.92300000000000004,"5min":1.0189999999999999,"15min":1.2430000000000001},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.70999999999999996,"15min":0.70699999999999996},"min":{"1min":0.48899999999999999,"5min":0.45200000000000001,"15min":0.42699999999999999},"max":{"1min":0.99099999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.65500000000000003}]}]},{"osd":4,"up_from":13,"seq":55834575038,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32200,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339640,"statfs":{"total":96636764160,"available":96603791360,"internally_reserved":0,"allocated":1044480,"data_stored":515492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:26:00 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.59799999999999998,"15min":0.59199999999999997},"min":{"1min":0.30399999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.78300000000000003,"5min":0.85099999999999998,"15min":1.171},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.62,"5min":0.58699999999999997,"15min":0.60199999999999998},"min":{"1min":0.42399999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.76400000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.57999999999999996}]},{"osd":1,"last update":"Mon Oct 16 12:26:00 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.59299999999999997,"15min":0.60999999999999999},"min":{"1min":0.433,"5min":0.35499999999999998,"15min":0.28499999999999998},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.92600000000000005},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.622,"5min":0.61499999999999999,"15min":0.621},"min":{"1min":0.374,"5min":0.374,"15min":0.33600000000000002},"max":{"1min":0.73899999999999999,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.66900000000000004}]},{"osd":2,"last update":"Mon Oct 16 12:26:00 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.61199999999999999,"15min":0.625},"min":{"1min":0.39900000000000002,"5min":0.35399999999999998,"15min":0.32000000000000001},"max":{"1min":0.77000000000000002,"5min":0.93000000000000005,"15min":1.012},"last":0.434},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61299999999999999,"15min":0.627},"min":{"1min":0.47899999999999998,"5min":0.38800000000000001,"15min":0.34100000000000003},"max":{"1min":0.74099999999999999,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.71899999999999997}]},{"osd":3,"last update":"Mon Oct 16 12:26:00 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.59599999999999997,"15min":0.61299999999999999},"min":{"1min":0.46300000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.70699999999999996,"5min":0.79500000000000004,"15min":1.3779999999999999},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63,"15min":0.64100000000000001},"min":{"1min":0.41199999999999998,"5min":0.38800000000000001,"15min":0.37},"max":{"1min":0.77800000000000002,"5min":0.89800000000000002,"15min":0.97199999999999998},"last":0.63800000000000001}]},{"osd":5,"last update":"Mon Oct 16 12:26:00 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48699999999999999,"15min":0.5},"min":{"1min":0.34799999999999998,"5min":0.29799999999999999,"15min":0.27600000000000002},"max":{"1min":0.68100000000000005,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.504,"5min":0.51400000000000001,"15min":0.52500000000000002},"min":{"1min":0.371,"5min":0.316,"15min":0.29599999999999999},"max":{"1min":0.72599999999999998,"5min":0.754,"15min":0.94199999999999995},"last":0.40600000000000003}]},{"osd":6,"last update":"Mon Oct 16 12:26:00 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.54600000000000004,"15min":0.55900000000000005},"min":{"1min":0.39800000000000002,"5min":0.34300000000000003,"15min":0.28699999999999998},"max":{"1min":0.68700000000000006,"5min":0.88400000000000001,"15min":0.89800000000000002},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.63700000000000001,"15min":0.63600000000000001},"min":{"1min":0.45000000000000001,"5min":0.34799999999999998,"15min":0.32000000000000001},"max":{"1min":0.84099999999999997,"5min":0.99299999999999999,"15min":1.1180000000000001},"last":0.62}]},{"osd":7,"last update":"Mon Oct 16 12:26:00 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57599999999999996,"15min":0.58099999999999996},"min":{"1min":0.42099999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.76200000000000001,"5min":0.80900000000000005,"15min":0.89500000000000002},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.628,"15min":0.626},"min":{"1min":0.42599999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.87,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.68300000000000005}]}]},{"osd":5,"up_from":13,"seq":55834575039,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54388,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317452,"statfs":{"total":96636764160,"available":96581070848,"internally_reserved":0,"allocated":1024000,"data_stored":507254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:25:58 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.59599999999999997,"15min":0.59399999999999997},"min":{"1min":0.49399999999999999,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.84499999999999997,"5min":0.873,"15min":0.873},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.61099999999999999,"15min":0.622},"min":{"1min":0.432,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.88800000000000001},"last":0.505}]},{"osd":1,"last update":"Mon Oct 16 12:25:58 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.61499999999999999,"15min":0.61699999999999999},"min":{"1min":0.47899999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.89700000000000002,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.63500000000000001,"15min":0.626},"min":{"1min":0.52600000000000002,"5min":0.32800000000000001,"15min":0.28799999999999998},"max":{"1min":0.84799999999999998,"5min":0.89000000000000001,"15min":0.95199999999999996},"last":0.60699999999999998}]},{"osd":2,"last update":"Mon Oct 16 12:25:58 2023","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.63300000000000001,"15min":0.629},"min":{"1min":0.49299999999999999,"5min":0.29099999999999998,"15min":0.26400000000000001},"max":{"1min":0.86199999999999999,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.61599999999999999,"15min":0.61899999999999999},"min":{"1min":0.48699999999999999,"5min":0.378,"15min":0.36499999999999999},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.89700000000000002},"last":0.42199999999999999}]},{"osd":3,"last update":"Mon Oct 16 12:25:58 2023","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.63700000000000001,"15min":0.64700000000000002},"min":{"1min":0.53700000000000003,"5min":0.371,"15min":0.36099999999999999},"max":{"1min":0.81899999999999995,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.65400000000000003,"15min":0.65800000000000003},"min":{"1min":0.47299999999999998,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.88700000000000001,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.44600000000000001}]},{"osd":4,"last update":"Mon Oct 16 12:25:58 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.51000000000000001,"15min":0.51900000000000002},"min":{"1min":0.39200000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.77200000000000002,"5min":0.85699999999999998,"15min":1.0089999999999999},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.57299999999999995,"15min":0.57799999999999996},"min":{"1min":0.41299999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.755,"5min":1.347,"15min":1.347},"last":0.67500000000000004}]},{"osd":6,"last update":"Mon Oct 16 12:25:58 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.55200000000000005,"15min":0.56100000000000005},"min":{"1min":0.40400000000000003,"5min":0.314,"15min":0.314},"max":{"1min":0.89400000000000002,"5min":0.93500000000000005,"15min":0.98699999999999999},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.60799999999999998,"15min":0.60699999999999998},"min":{"1min":0.371,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.81699999999999995,"5min":0.92700000000000005,"15min":0.95999999999999996},"last":0.58699999999999997}]},{"osd":7,"last update":"Mon Oct 16 12:25:58 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.621,"15min":0.626},"min":{"1min":0.42099999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.84699999999999998,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.626},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.66600000000000004,"15min":0.66500000000000004},"min":{"1min":0.50800000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.91600000000000004,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.65800000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T12:26:28.848 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-16T12:26:28.848 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-16T12:26:29.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.011+0000 7f34fef85700 1 Processor -- start 2023-10-16T12:26:29.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.011+0000 7f34fef85700 1 -- start start 2023-10-16T12:26:29.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.012+0000 7f34fef85700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f34f815b2c0 0x7f34f8161340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:29.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.012+0000 7f34fef85700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34f815ecb0 con 0x7f34f806a7e0 2023-10-16T12:26:29.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.012+0000 7f34fef85700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34f8161880 con 0x7f34f806e570 2023-10-16T12:26:29.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.012+0000 7f34fef85700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34f815b6a0 con 0x7f34f815b2c0 2023-10-16T12:26:29.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.012+0000 7f34fcd21700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f34f815b2c0 0x7f34f8161340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:29.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.012+0000 7f34fcd21700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f34f815b2c0 0x7f34f8161340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39434/0 (socket says 172.21.15.16:39434) 2023-10-16T12:26:29.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.012+0000 7f34fcd21700 1 -- 172.21.15.16:0/2398729834 learned_addr learned my addr 172.21.15.16:0/2398729834 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:29.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.012+0000 7f34fcd21700 1 -- 172.21.15.16:0/2398729834 >> v1:172.21.15.49:6789/0 conn(0x7f34f806e570 legacy=0x7f34f815e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:29.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.012+0000 7f34fcd21700 1 -- 172.21.15.16:0/2398729834 >> v1:172.21.15.16:6789/0 conn(0x7f34f806a7e0 legacy=0x7f34f8071a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:29.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.012+0000 7f34fcd21700 1 -- 172.21.15.16:0/2398729834 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34f8161b20 con 0x7f34f815b2c0 2023-10-16T12:26:29.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.012+0000 7f34fcd21700 1 --2- 172.21.15.16:0/2398729834 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34f815b2c0 0x7f34f8161340 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f34ec0098f0 tx=0x7f34ec009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b304b3222708f2a5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:29.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.012+0000 7f34f77fe700 1 -- 172.21.15.16:0/2398729834 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34ec00b8a0 con 0x7f34f815b2c0 2023-10-16T12:26:29.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.013+0000 7f34f77fe700 1 -- 172.21.15.16:0/2398729834 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34ec00ba00 con 0x7f34f815b2c0 2023-10-16T12:26:29.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.013+0000 7f34f77fe700 1 -- 172.21.15.16:0/2398729834 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34ec00bd90 con 0x7f34f815b2c0 2023-10-16T12:26:29.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.013+0000 7f34fef85700 1 -- 172.21.15.16:0/2398729834 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34f815b2c0 msgr2=0x7f34f8161340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:29.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.013+0000 7f34fef85700 1 --2- 172.21.15.16:0/2398729834 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34f815b2c0 0x7f34f8161340 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f34ec0098f0 tx=0x7f34ec009c00 comp rx=0 tx=0).stop 2023-10-16T12:26:29.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.013+0000 7f34fef85700 1 -- 172.21.15.16:0/2398729834 shutdown_connections 2023-10-16T12:26:29.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.013+0000 7f34fef85700 1 --2- 172.21.15.16:0/2398729834 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34f815b2c0 0x7f34f8161340 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:29.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.013+0000 7f34fef85700 1 -- 172.21.15.16:0/2398729834 >> 172.21.15.16:0/2398729834 conn(0x7f34f80b1f40 msgr2=0x7f34f80b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:29.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.013+0000 7f34fef85700 1 -- 172.21.15.16:0/2398729834 shutdown_connections 2023-10-16T12:26:29.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.013+0000 7f34fef85700 1 -- 172.21.15.16:0/2398729834 wait complete. 2023-10-16T12:26:29.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.014+0000 7f34fef85700 1 Processor -- start 2023-10-16T12:26:29.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.014+0000 7f34fef85700 1 -- start start 2023-10-16T12:26:29.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.015+0000 7f34fef85700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34f806a7e0 0x7f34f80b5d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:29.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.015+0000 7f34fef85700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f34f806e570 0x7f34f80b3920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:29.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.015+0000 7f34fef85700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f34f815b2c0 0x7f34f814d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:29.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.015+0000 7f34fef85700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34f80b3e60 con 0x7f34f806a7e0 2023-10-16T12:26:29.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.015+0000 7f34fef85700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34f80b4ce0 con 0x7f34f806e570 2023-10-16T12:26:29.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.015+0000 7f34fef85700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34f80b4fc0 con 0x7f34f815b2c0 2023-10-16T12:26:29.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.015+0000 7f34fcd21700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34f806a7e0 0x7f34f80b5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:29.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.015+0000 7f34fcd21700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34f806a7e0 0x7f34f80b5d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39440/0 (socket says 172.21.15.16:39440) 2023-10-16T12:26:29.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.015+0000 7f34fcd21700 1 -- 172.21.15.16:0/3269233215 learned_addr learned my addr 172.21.15.16:0/3269233215 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:29.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.016+0000 7f34fd522700 1 --2- 172.21.15.16:0/3269233215 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f34f815b2c0 0x7f34f814d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:29.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.016+0000 7f34f7fff700 1 --2- 172.21.15.16:0/3269233215 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f34f806e570 0x7f34f80b3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:29.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.016+0000 7f34fcd21700 1 -- 172.21.15.16:0/3269233215 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f34f815b2c0 msgr2=0x7f34f814d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:29.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.016+0000 7f34fcd21700 1 --2- 172.21.15.16:0/3269233215 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f34f815b2c0 0x7f34f814d7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:29.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.016+0000 7f34fcd21700 1 -- 172.21.15.16:0/3269233215 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f34f806e570 msgr2=0x7f34f80b3920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:29.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.016+0000 7f34fcd21700 1 --2- 172.21.15.16:0/3269233215 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f34f806e570 0x7f34f80b3920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:29.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.016+0000 7f34fcd21700 1 -- 172.21.15.16:0/3269233215 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34ec009550 con 0x7f34f806a7e0 2023-10-16T12:26:29.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.016+0000 7f34fcd21700 1 --2- 172.21.15.16:0/3269233215 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34f806a7e0 0x7f34f80b5d60 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f34ec009130 tx=0x7f34ec004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:29.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.016+0000 7f34f57fa700 1 -- 172.21.15.16:0/3269233215 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34ec0036a0 con 0x7f34f806a7e0 2023-10-16T12:26:29.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.016+0000 7f34f57fa700 1 -- 172.21.15.16:0/3269233215 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34ec01e070 con 0x7f34f806a7e0 2023-10-16T12:26:29.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.016+0000 7f34fef85700 1 -- 172.21.15.16:0/3269233215 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34f80b52a0 con 0x7f34f806a7e0 2023-10-16T12:26:29.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.016+0000 7f34f57fa700 1 -- 172.21.15.16:0/3269233215 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34ec003a20 con 0x7f34f806a7e0 2023-10-16T12:26:29.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.016+0000 7f34fef85700 1 -- 172.21.15.16:0/3269233215 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34f80b5860 con 0x7f34f806a7e0 2023-10-16T12:26:29.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.017+0000 7f34fef85700 1 -- 172.21.15.16:0/3269233215 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34f80b59f0 con 0x7f34f806a7e0 2023-10-16T12:26:29.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.017+0000 7f34f57fa700 1 -- 172.21.15.16:0/3269233215 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 31) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f34ec003b80 con 0x7f34f806a7e0 2023-10-16T12:26:29.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.018+0000 7f34f57fa700 1 --2- 172.21.15.16:0/3269233215 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f34dc06b020 0x7f34dc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:29.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.018+0000 7f34f57fa700 1 -- 172.21.15.16:0/3269233215 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f34ec020070 con 0x7f34f806a7e0 2023-10-16T12:26:29.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.018+0000 7f34f7fff700 1 --2- 172.21.15.16:0/3269233215 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f34dc06b020 0x7f34dc06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:29.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.018+0000 7f34f7fff700 1 --2- 172.21.15.16:0/3269233215 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f34dc06b020 0x7f34dc06d4e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f34e4000f60 tx=0x7f34e4009040 comp rx=0 tx=0).ready entity=mgr.6396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:29.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.022+0000 7f34f57fa700 1 -- 172.21.15.16:0/3269233215 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f34ec055d50 con 0x7f34f806a7e0 2023-10-16T12:26:29.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.308+0000 7f34fef85700 1 -- 172.21.15.16:0/3269233215 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f34f804ebb0 con 0x7f34f806a7e0 2023-10-16T12:26:29.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.310+0000 7f34f57fa700 1 -- 172.21.15.16:0/3269233215 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v20) v1 ==== 190+0+147 (secure 0 0 0) 0x7f34ec015020 con 0x7f34f806a7e0 2023-10-16T12:26:29.313 INFO:teuthology.orchestra.run.smithi016.stdout:[client.0] 2023-10-16T12:26:29.313 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAPKC1l2l6PFxAA9QyXtBAHdGAXpwH0JfknuQ== 2023-10-16T12:26:29.314 INFO:teuthology.orchestra.run.smithi016.stdout: caps mds = "allow" 2023-10-16T12:26:29.314 INFO:teuthology.orchestra.run.smithi016.stdout: caps mon = "allow r" 2023-10-16T12:26:29.314 INFO:teuthology.orchestra.run.smithi016.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-16T12:26:29.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.312+0000 7f34fef85700 1 -- 172.21.15.16:0/3269233215 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f34dc06b020 msgr2=0x7f34dc06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:29.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.312+0000 7f34fef85700 1 --2- 172.21.15.16:0/3269233215 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f34dc06b020 0x7f34dc06d4e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f34e4000f60 tx=0x7f34e4009040 comp rx=0 tx=0).stop 2023-10-16T12:26:29.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.312+0000 7f34fef85700 1 -- 172.21.15.16:0/3269233215 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34f806a7e0 msgr2=0x7f34f80b5d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:29.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.312+0000 7f34fef85700 1 --2- 172.21.15.16:0/3269233215 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34f806a7e0 0x7f34f80b5d60 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f34ec009130 tx=0x7f34ec004750 comp rx=0 tx=0).stop 2023-10-16T12:26:29.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.313+0000 7f34fef85700 1 -- 172.21.15.16:0/3269233215 shutdown_connections 2023-10-16T12:26:29.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.313+0000 7f34fef85700 1 --2- 172.21.15.16:0/3269233215 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f34dc06b020 0x7f34dc06d4e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:29.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.313+0000 7f34fef85700 1 --2- 172.21.15.16:0/3269233215 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f34f815b2c0 0x7f34f814d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:29.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.313+0000 7f34fef85700 1 --2- 172.21.15.16:0/3269233215 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34f806a7e0 0x7f34f80b5d60 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:29.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.313+0000 7f34fef85700 1 --2- 172.21.15.16:0/3269233215 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f34f806e570 0x7f34f80b3920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:29.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.313+0000 7f34fef85700 1 -- 172.21.15.16:0/3269233215 >> 172.21.15.16:0/3269233215 conn(0x7f34f80b1f40 msgr2=0x7f34f80cc190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:29.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.313+0000 7f34fef85700 1 -- 172.21.15.16:0/3269233215 shutdown_connections 2023-10-16T12:26:29.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.313+0000 7f34fef85700 1 -- 172.21.15.16:0/3269233215 wait complete. 2023-10-16T12:26:29.318 INFO:teuthology.orchestra.run.smithi016.stderr:updated caps for client.0 2023-10-16T12:26:29.328 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:26:29.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.450+0000 7f5098aed700 1 Processor -- start 2023-10-16T12:26:29.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.450+0000 7f5098aed700 1 -- start start 2023-10-16T12:26:29.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.451+0000 7f5098aed700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f509415b630 0x7f50940cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:29.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.451+0000 7f5098aed700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f50940ce4f0 0x7f50940cfca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:29.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.451+0000 7f5098aed700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50940cc2a0 con 0x7f509415ba10 2023-10-16T12:26:29.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.451+0000 7f5098aed700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50940cdb20 con 0x7f509415b630 2023-10-16T12:26:29.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.451+0000 7f5098aed700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50940cc540 con 0x7f50940ce4f0 2023-10-16T12:26:29.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.451+0000 7f509259c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f509415b630 0x7f50940cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:29.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.451+0000 7f509259c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f509415b630 0x7f50940cd5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39448/0 (socket says 172.21.15.16:39448) 2023-10-16T12:26:29.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.451+0000 7f509259c700 1 -- 172.21.15.16:0/1149078197 learned_addr learned my addr 172.21.15.16:0/1149078197 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:29.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.451+0000 7f509259c700 1 -- 172.21.15.16:0/1149078197 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f50940ce4f0 msgr2=0x7f50940cfca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:29.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.451+0000 7f5092d9d700 1 --2- 172.21.15.16:0/1149078197 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f50940ce4f0 0x7f50940cfca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:29.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.451+0000 7f509259c700 1 --2- 172.21.15.16:0/1149078197 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f50940ce4f0 0x7f50940cfca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:29.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.451+0000 7f509259c700 1 -- 172.21.15.16:0/1149078197 >> v1:172.21.15.16:6789/0 conn(0x7f509415ba10 legacy=0x7f50940cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:29.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.451+0000 7f509259c700 1 -- 172.21.15.16:0/1149078197 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50940cc820 con 0x7f509415b630 2023-10-16T12:26:29.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.451+0000 7f5092d9d700 1 --2- 172.21.15.16:0/1149078197 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f50940ce4f0 0x7f50940cfca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:26:29.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.451+0000 7f509259c700 1 --2- 172.21.15.16:0/1149078197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f509415b630 0x7f50940cd5e0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f5084006fd0 tx=0x7f5084009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=299ada234463a517 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:29.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.452+0000 7f509159a700 1 -- 172.21.15.16:0/1149078197 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5084005e30 con 0x7f509415b630 2023-10-16T12:26:29.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.452+0000 7f509159a700 1 -- 172.21.15.16:0/1149078197 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5084004030 con 0x7f509415b630 2023-10-16T12:26:29.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.452+0000 7f509159a700 1 -- 172.21.15.16:0/1149078197 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50840043e0 con 0x7f509415b630 2023-10-16T12:26:29.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.452+0000 7f5098aed700 1 -- 172.21.15.16:0/1149078197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f509415b630 msgr2=0x7f50940cd5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:29.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.452+0000 7f5098aed700 1 --2- 172.21.15.16:0/1149078197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f509415b630 0x7f50940cd5e0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f5084006fd0 tx=0x7f5084009bd0 comp rx=0 tx=0).stop 2023-10-16T12:26:29.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.452+0000 7f5098aed700 1 -- 172.21.15.16:0/1149078197 shutdown_connections 2023-10-16T12:26:29.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.452+0000 7f5098aed700 1 --2- 172.21.15.16:0/1149078197 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f50940ce4f0 0x7f50940cfca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:29.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.452+0000 7f5098aed700 1 --2- 172.21.15.16:0/1149078197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f509415b630 0x7f50940cd5e0 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:29.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.452+0000 7f5098aed700 1 -- 172.21.15.16:0/1149078197 >> 172.21.15.16:0/1149078197 conn(0x7f50940d14b0 msgr2=0x7f50940d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:29.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.452+0000 7f5098aed700 1 -- 172.21.15.16:0/1149078197 shutdown_connections 2023-10-16T12:26:29.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.452+0000 7f5098aed700 1 -- 172.21.15.16:0/1149078197 wait complete. 2023-10-16T12:26:29.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.454+0000 7f5098aed700 1 Processor -- start 2023-10-16T12:26:29.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.454+0000 7f5098aed700 1 -- start start 2023-10-16T12:26:29.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.454+0000 7f5098aed700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f50940ce4f0 0x7f509415cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:29.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.454+0000 7f5098aed700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f509415b630 0x7f50940c4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:29.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.454+0000 7f5098aed700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f509415ba10 0x7f50940c5520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:29.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.454+0000 7f5098aed700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f509415e140 con 0x7f509415ba10 2023-10-16T12:26:29.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.454+0000 7f5098aed700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f509415d530 con 0x7f509415b630 2023-10-16T12:26:29.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.454+0000 7f5098aed700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f509415d810 con 0x7f50940ce4f0 2023-10-16T12:26:29.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.454+0000 7f5092d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f509415ba10 0x7f50940c5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:29.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.455+0000 7f5092d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f509415ba10 0x7f50940c5520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39458/0 (socket says 172.21.15.16:39458) 2023-10-16T12:26:29.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.455+0000 7f5092d9d700 1 -- 172.21.15.16:0/1488202632 learned_addr learned my addr 172.21.15.16:0/1488202632 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:29.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.455+0000 7f509259c700 1 --2- 172.21.15.16:0/1488202632 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f50940ce4f0 0x7f509415cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:29.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.455+0000 7f5091d9b700 1 --2- 172.21.15.16:0/1488202632 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f509415b630 0x7f50940c4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:29.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.455+0000 7f5092d9d700 1 -- 172.21.15.16:0/1488202632 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f50940ce4f0 msgr2=0x7f509415cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:29.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.455+0000 7f5092d9d700 1 --2- 172.21.15.16:0/1488202632 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f50940ce4f0 0x7f509415cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:29.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.455+0000 7f5092d9d700 1 -- 172.21.15.16:0/1488202632 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f509415b630 msgr2=0x7f50940c4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:29.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.455+0000 7f5092d9d700 1 --2- 172.21.15.16:0/1488202632 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f509415b630 0x7f50940c4fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:29.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.455+0000 7f5092d9d700 1 -- 172.21.15.16:0/1488202632 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5084009500 con 0x7f509415ba10 2023-10-16T12:26:29.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.455+0000 7f5092d9d700 1 --2- 172.21.15.16:0/1488202632 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f509415ba10 0x7f50940c5520 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f508c00cd80 tx=0x7f508c00a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:29.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.455+0000 7f5082ffd700 1 -- 172.21.15.16:0/1488202632 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f508c003a20 con 0x7f509415ba10 2023-10-16T12:26:29.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.455+0000 7f5082ffd700 1 -- 172.21.15.16:0/1488202632 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f508c003ba0 con 0x7f509415ba10 2023-10-16T12:26:29.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.455+0000 7f5098aed700 1 -- 172.21.15.16:0/1488202632 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f509415daf0 con 0x7f509415ba10 2023-10-16T12:26:29.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.455+0000 7f5082ffd700 1 -- 172.21.15.16:0/1488202632 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f508c004020 con 0x7f509415ba10 2023-10-16T12:26:29.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.455+0000 7f5098aed700 1 -- 172.21.15.16:0/1488202632 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f509415e5d0 con 0x7f509415ba10 2023-10-16T12:26:29.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.456+0000 7f5098aed700 1 -- 172.21.15.16:0/1488202632 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f509415e850 con 0x7f509415ba10 2023-10-16T12:26:29.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.457+0000 7f5082ffd700 1 -- 172.21.15.16:0/1488202632 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 31) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f508c00b040 con 0x7f509415ba10 2023-10-16T12:26:29.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.457+0000 7f5082ffd700 1 --2- 172.21.15.16:0/1488202632 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f507406b020 0x7f507406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:29.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.457+0000 7f5082ffd700 1 -- 172.21.15.16:0/1488202632 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f508c089ef0 con 0x7f509415ba10 2023-10-16T12:26:29.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.457+0000 7f509259c700 1 --2- 172.21.15.16:0/1488202632 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f507406b020 0x7f507406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:29.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.458+0000 7f509259c700 1 --2- 172.21.15.16:0/1488202632 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f507406b020 0x7f507406d4e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5084009c10 tx=0x7f5084008070 comp rx=0 tx=0).ready entity=mgr.6396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:29.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.462+0000 7f5082ffd700 1 -- 172.21.15.16:0/1488202632 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f508c054c30 con 0x7f509415ba10 2023-10-16T12:26:29.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.715+0000 7f5098aed700 1 -- 172.21.15.16:0/1488202632 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f50940ccad0 con 0x7f509415ba10 2023-10-16T12:26:29.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.715+0000 7f5082ffd700 1 -- 172.21.15.16:0/1488202632 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+4911 (secure 0 0 0) 0x7f508c058250 con 0x7f509415ba10 2023-10-16T12:26:29.719 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:26:29.719 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":36,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5861,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/1236108916","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":1236108916},{"type":"v1","addr":"172.21.15.49:6834","nonce":1236108916}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31},{"gid":5864,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/2762335434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":2762335434},{"type":"v1","addr":"172.21.15.49:6835","nonce":2762335434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31},{"gid":6379,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/1502256436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1502256436},{"type":"v1","addr":"172.21.15.16:6836","nonce":1502256436}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:26:23.670609+0000","modified":"2023-10-16T12:26:26.710098+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6627},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6627":{"gid":6627,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/962243511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":962243511},{"type":"v1","addr":"172.21.15.16:6837","nonce":962243511}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-10-16T12:26:29.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.717+0000 7f5098aed700 1 -- 172.21.15.16:0/1488202632 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f507406b020 msgr2=0x7f507406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:29.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.717+0000 7f5098aed700 1 --2- 172.21.15.16:0/1488202632 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f507406b020 0x7f507406d4e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5084009c10 tx=0x7f5084008070 comp rx=0 tx=0).stop 2023-10-16T12:26:29.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.717+0000 7f5098aed700 1 -- 172.21.15.16:0/1488202632 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f509415ba10 msgr2=0x7f50940c5520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:29.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.717+0000 7f5098aed700 1 --2- 172.21.15.16:0/1488202632 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f509415ba10 0x7f50940c5520 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f508c00cd80 tx=0x7f508c00a560 comp rx=0 tx=0).stop 2023-10-16T12:26:29.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.717+0000 7f5098aed700 1 -- 172.21.15.16:0/1488202632 shutdown_connections 2023-10-16T12:26:29.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.717+0000 7f5098aed700 1 --2- 172.21.15.16:0/1488202632 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f507406b020 0x7f507406d4e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:29.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.717+0000 7f5098aed700 1 --2- 172.21.15.16:0/1488202632 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f509415ba10 0x7f50940c5520 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:29.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.717+0000 7f5098aed700 1 --2- 172.21.15.16:0/1488202632 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f50940ce4f0 0x7f509415cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:29.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.717+0000 7f5098aed700 1 --2- 172.21.15.16:0/1488202632 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f509415b630 0x7f50940c4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:29.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.717+0000 7f5098aed700 1 -- 172.21.15.16:0/1488202632 >> 172.21.15.16:0/1488202632 conn(0x7f50940d14b0 msgr2=0x7f50940b55e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:29.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.717+0000 7f5098aed700 1 -- 172.21.15.16:0/1488202632 shutdown_connections 2023-10-16T12:26:29.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:29.717+0000 7f5098aed700 1 -- 172.21.15.16:0/1488202632 wait complete. 2023-10-16T12:26:29.723 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 36 2023-10-16T12:26:30.135 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 36, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T12:26:23.670609+0000', 'modified': '2023-10-16T12:26:26.710098+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6627}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6627': {'gid': 6627, 'name': 'a', 'rank': 0, 'incarnation': 32, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.16:6837/962243511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6835', 'nonce': 962243511}, {'type': 'v1', 'addr': '172.21.15.16:6837', 'nonce': 962243511}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [10], 'metadata_pool': 9, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-16T12:26:30.136 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-16T12:26:30.136 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-10-16T12:26:30.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.268+0000 7fdd6007a700 1 Processor -- start 2023-10-16T12:26:30.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.268+0000 7fdd6007a700 1 -- start start 2023-10-16T12:26:30.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.269+0000 7fdd6007a700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fdd5815a5c0 0x7fdd580cef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:30.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.269+0000 7fdd6007a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdd580d0ae0 0x7fdd580cc1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:30.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.269+0000 7fdd6007a700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd58072a50 con 0x7fdd580ce010 2023-10-16T12:26:30.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.269+0000 7fdd6007a700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd580cf440 con 0x7fdd5815a5c0 2023-10-16T12:26:30.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.269+0000 7fdd6007a700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd580cba30 con 0x7fdd580d0ae0 2023-10-16T12:26:30.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.269+0000 7fdd5e617700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdd580d0ae0 0x7fdd580cc1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:30.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.269+0000 7fdd5d615700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7fdd580ce010 0x7fdd580d03d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:35920/0 (socket says 172.21.15.16:35920) 2023-10-16T12:26:30.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.269+0000 7fdd5de16700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fdd5815a5c0 0x7fdd580cef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:30.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.269+0000 7fdd5d615700 1 -- 172.21.15.16:0/456692421 learned_addr learned my addr 172.21.15.16:0/456692421 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:26:30.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.270+0000 7fdd5ce14700 1 -- 172.21.15.16:0/456692421 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2387081058 0 0) 0x7fdd58072a50 con 0x7fdd580ce010 2023-10-16T12:26:30.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.270+0000 7fdd5ce14700 1 -- 172.21.15.16:0/456692421 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd440035b0 con 0x7fdd580ce010 2023-10-16T12:26:30.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.270+0000 7fdd5e617700 1 -- 172.21.15.16:0/456692421 >> v2:172.21.15.49:3300/0 conn(0x7fdd5815a5c0 msgr2=0x7fdd580cef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:30.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.270+0000 7fdd5e617700 1 --2- 172.21.15.16:0/456692421 >> v2:172.21.15.49:3300/0 conn(0x7fdd5815a5c0 0x7fdd580cef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:30.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.270+0000 7fdd5e617700 1 -- 172.21.15.16:0/456692421 >> v1:172.21.15.49:6789/0 conn(0x7fdd580ce010 legacy=0x7fdd580d03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:30.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.270+0000 7fdd5e617700 1 -- 172.21.15.16:0/456692421 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd580cf630 con 0x7fdd580d0ae0 2023-10-16T12:26:30.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.270+0000 7fdd5e617700 1 --2- 172.21.15.16:0/456692421 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdd580d0ae0 0x7fdd580cc1f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fdd50009d90 tx=0x7fdd5000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa4edcdf95d741e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:30.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.270+0000 7fdd5ce14700 1 -- 172.21.15.16:0/456692421 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd5000e070 con 0x7fdd580d0ae0 2023-10-16T12:26:30.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.270+0000 7fdd5ce14700 1 -- 172.21.15.16:0/456692421 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd5000bb20 con 0x7fdd580d0ae0 2023-10-16T12:26:30.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.270+0000 7fdd5ce14700 1 -- 172.21.15.16:0/456692421 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd5000be90 con 0x7fdd580d0ae0 2023-10-16T12:26:30.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.271+0000 7fdd6007a700 1 -- 172.21.15.16:0/456692421 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdd580d0ae0 msgr2=0x7fdd580cc1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:30.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.271+0000 7fdd6007a700 1 --2- 172.21.15.16:0/456692421 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdd580d0ae0 0x7fdd580cc1f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fdd50009d90 tx=0x7fdd5000b330 comp rx=0 tx=0).stop 2023-10-16T12:26:30.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.271+0000 7fdd6007a700 1 -- 172.21.15.16:0/456692421 shutdown_connections 2023-10-16T12:26:30.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.271+0000 7fdd6007a700 1 --2- 172.21.15.16:0/456692421 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdd580d0ae0 0x7fdd580cc1f0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:30.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.271+0000 7fdd6007a700 1 --2- 172.21.15.16:0/456692421 >> v2:172.21.15.49:3300/0 conn(0x7fdd5815a5c0 0x7fdd580cef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:30.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.271+0000 7fdd6007a700 1 -- 172.21.15.16:0/456692421 >> 172.21.15.16:0/456692421 conn(0x7fdd580b1fd0 msgr2=0x7fdd580b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:30.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.271+0000 7fdd6007a700 1 -- 172.21.15.16:0/456692421 shutdown_connections 2023-10-16T12:26:30.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.271+0000 7fdd6007a700 1 -- 172.21.15.16:0/456692421 wait complete. 2023-10-16T12:26:30.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.272+0000 7fdd6007a700 1 Processor -- start 2023-10-16T12:26:30.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.272+0000 7fdd6007a700 1 -- start start 2023-10-16T12:26:30.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.273+0000 7fdd6007a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd580ce010 0x7fdd58147400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:30.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.273+0000 7fdd6007a700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdd580d0ae0 0x7fdd58149b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:30.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.273+0000 7fdd6007a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdd5815a5c0 0x7fdd580c9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:30.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.273+0000 7fdd6007a700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd580c81d0 con 0x7fdd580ce010 2023-10-16T12:26:30.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.273+0000 7fdd6007a700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd580c95b0 con 0x7fdd580d0ae0 2023-10-16T12:26:30.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.273+0000 7fdd6007a700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd5814aac0 con 0x7fdd5815a5c0 2023-10-16T12:26:30.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.273+0000 7fdd5de16700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd580ce010 0x7fdd58147400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:30.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.273+0000 7fdd5de16700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd580ce010 0x7fdd58147400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39468/0 (socket says 172.21.15.16:39468) 2023-10-16T12:26:30.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.273+0000 7fdd5de16700 1 -- 172.21.15.16:0/2867449549 learned_addr learned my addr 172.21.15.16:0/2867449549 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:30.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.273+0000 7fdd5d615700 1 --2- 172.21.15.16:0/2867449549 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdd580d0ae0 0x7fdd58149b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:30.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.273+0000 7fdd5e617700 1 --2- 172.21.15.16:0/2867449549 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdd5815a5c0 0x7fdd580c9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:30.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.273+0000 7fdd5de16700 1 -- 172.21.15.16:0/2867449549 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdd5815a5c0 msgr2=0x7fdd580c9fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:30.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.273+0000 7fdd5de16700 1 --2- 172.21.15.16:0/2867449549 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdd5815a5c0 0x7fdd580c9fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:30.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.273+0000 7fdd5de16700 1 -- 172.21.15.16:0/2867449549 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdd580d0ae0 msgr2=0x7fdd58149b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:30.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.273+0000 7fdd5de16700 1 --2- 172.21.15.16:0/2867449549 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdd580d0ae0 0x7fdd58149b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:30.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.273+0000 7fdd5de16700 1 -- 172.21.15.16:0/2867449549 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd50009a70 con 0x7fdd580ce010 2023-10-16T12:26:30.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.273+0000 7fdd5de16700 1 --2- 172.21.15.16:0/2867449549 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd580ce010 0x7fdd58147400 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fdd4800e9e0 tx=0x7fdd4800eda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:30.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.273+0000 7fdd4e7fc700 1 -- 172.21.15.16:0/2867449549 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd48009710 con 0x7fdd580ce010 2023-10-16T12:26:30.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.273+0000 7fdd4e7fc700 1 -- 172.21.15.16:0/2867449549 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd4800c850 con 0x7fdd580ce010 2023-10-16T12:26:30.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.273+0000 7fdd6007a700 1 -- 172.21.15.16:0/2867449549 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd580c9b50 con 0x7fdd580ce010 2023-10-16T12:26:30.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.274+0000 7fdd4e7fc700 1 -- 172.21.15.16:0/2867449549 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd4800cbc0 con 0x7fdd580ce010 2023-10-16T12:26:30.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.274+0000 7fdd6007a700 1 -- 172.21.15.16:0/2867449549 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd5814a7e0 con 0x7fdd580ce010 2023-10-16T12:26:30.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.275+0000 7fdd4e7fc700 1 -- 172.21.15.16:0/2867449549 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 31) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fdd4800cd20 con 0x7fdd580ce010 2023-10-16T12:26:30.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.275+0000 7fdd4e7fc700 1 --2- 172.21.15.16:0/2867449549 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7fdd3c06b020 0x7fdd3c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:30.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.276+0000 7fdd4e7fc700 1 -- 172.21.15.16:0/2867449549 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fdd48089920 con 0x7fdd580ce010 2023-10-16T12:26:30.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.276+0000 7fdd5d615700 1 --2- 172.21.15.16:0/2867449549 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7fdd3c06b020 0x7fdd3c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:30.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.276+0000 7fdd5d615700 1 --2- 172.21.15.16:0/2867449549 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7fdd3c06b020 0x7fdd3c06d4e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fdd580ccce0 tx=0x7fdd54009040 comp rx=0 tx=0).ready entity=mgr.6396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:30.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.278+0000 7fdd6c264b80 1 Processor -- start 2023-10-16T12:26:30.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.278+0000 7fdd6c264b80 1 -- start start 2023-10-16T12:26:30.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.278+0000 7fdd6c264b80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55f1b1d39220 0x55f1b1d40a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:30.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.278+0000 7fdd6c264b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1b1d3e630 0x55f1b1d40fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:30.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.278+0000 7fdd6c264b80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f1b1d14010 0x55f1b1d146c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:30.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.278+0000 7fdd6c264b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1b1c8ac20 con 0x55f1b1d3e630 2023-10-16T12:26:30.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.278+0000 7fdd6c264b80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1b1c6fc10 con 0x55f1b1d14010 2023-10-16T12:26:30.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.278+0000 7fdd6c264b80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f1b1c6d530 con 0x55f1b1d39220 2023-10-16T12:26:30.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.278+0000 7fdd5d615700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1b1d3e630 0x55f1b1d40fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:30.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.279+0000 7fdd5d615700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1b1d3e630 0x55f1b1d40fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39478/0 (socket says 172.21.15.16:39478) 2023-10-16T12:26:30.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.279+0000 7fdd5d615700 1 -- 172.21.15.16:0/2388688930 learned_addr learned my addr 172.21.15.16:0/2388688930 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:30.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.279+0000 7fdd5d615700 1 -- 172.21.15.16:0/2388688930 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55f1b1d39220 msgr2=0x55f1b1d40a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:30.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.279+0000 7fdd5d615700 1 --2- 172.21.15.16:0/2388688930 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55f1b1d39220 0x55f1b1d40a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:30.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.279+0000 7fdd5d615700 1 -- 172.21.15.16:0/2388688930 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f1b1d14010 msgr2=0x55f1b1d146c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:30.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.279+0000 7fdd5d615700 1 --2- 172.21.15.16:0/2388688930 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f1b1d14010 0x55f1b1d146c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:30.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.279+0000 7fdd5d615700 1 -- 172.21.15.16:0/2388688930 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f1b1c263a0 con 0x55f1b1d3e630 2023-10-16T12:26:30.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.279+0000 7fdd5d615700 1 --2- 172.21.15.16:0/2388688930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1b1d3e630 0x55f1b1d40fb0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fdd54005b70 tx=0x7fdd54010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39199abc1b95ac36 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:30.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.279+0000 7fdd3b7fe700 1 -- 172.21.15.16:0/2388688930 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd54020450 con 0x55f1b1d3e630 2023-10-16T12:26:30.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.279+0000 7fdd3b7fe700 1 -- 172.21.15.16:0/2388688930 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd540205b0 con 0x55f1b1d3e630 2023-10-16T12:26:30.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.279+0000 7fdd3b7fe700 1 -- 172.21.15.16:0/2388688930 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd54020920 con 0x55f1b1d3e630 2023-10-16T12:26:30.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.280+0000 7fdd6c264b80 1 -- 172.21.15.16:0/2388688930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1b1d3e630 msgr2=0x55f1b1d40fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:30.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.280+0000 7fdd6c264b80 1 --2- 172.21.15.16:0/2388688930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1b1d3e630 0x55f1b1d40fb0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fdd54005b70 tx=0x7fdd54010000 comp rx=0 tx=0).stop 2023-10-16T12:26:30.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.280+0000 7fdd6c264b80 1 -- 172.21.15.16:0/2388688930 shutdown_connections 2023-10-16T12:26:30.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.280+0000 7fdd6c264b80 1 --2- 172.21.15.16:0/2388688930 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f1b1d14010 0x55f1b1d146c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:30.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.280+0000 7fdd6c264b80 1 --2- 172.21.15.16:0/2388688930 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55f1b1d39220 0x55f1b1d40a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:30.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.280+0000 7fdd6c264b80 1 --2- 172.21.15.16:0/2388688930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1b1d3e630 0x55f1b1d40fb0 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:30.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.280+0000 7fdd6c264b80 1 -- 172.21.15.16:0/2388688930 >> 172.21.15.16:0/2388688930 conn(0x55f1b1c43d80 msgr2=0x55f1b1d5eff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:30.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.280+0000 7fdd6c264b80 1 -- 172.21.15.16:0/2388688930 shutdown_connections 2023-10-16T12:26:30.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.280+0000 7fdd6c264b80 1 -- 172.21.15.16:0/2388688930 wait complete. 2023-10-16T12:26:30.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.280+0000 7fdd6c264b80 1 -- start start 2023-10-16T12:26:30.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.280+0000 7fdd6c264b80 1 Processor -- start 2023-10-16T12:26:30.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.281+0000 7fdd6c264b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1b1d14010 0x55f1b1d16680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:30.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.281+0000 7fdd6c264b80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55f1b1d39220 0x55f1b1d39670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:30.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.281+0000 7fdd6c264b80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f1b1d3e630 0x55f1b1d37e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:30.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.281+0000 7fdd6c264b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1b1c68e20 con 0x55f1b1d14010 2023-10-16T12:26:30.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.281+0000 7fdd6c264b80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1b1c056c0 con 0x55f1b1d3e630 2023-10-16T12:26:30.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.281+0000 7fdd6c264b80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f1b1c05360 con 0x55f1b1d39220 2023-10-16T12:26:30.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.281+0000 7fdd6c264b80 10 client.0 _ll_register_callbacks cb 0x55f1b1a627b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T12:26:30.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.281+0000 7fdd5de16700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1b1d14010 0x55f1b1d16680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:30.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.282+0000 7fdd5de16700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1b1d14010 0x55f1b1d16680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39482/0 (socket says 172.21.15.16:39482) 2023-10-16T12:26:30.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.282+0000 7fdd5de16700 1 -- 172.21.15.16:0/520290930 learned_addr learned my addr 172.21.15.16:0/520290930 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:30.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.282+0000 7fdd5de16700 1 -- 172.21.15.16:0/520290930 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55f1b1d39220 msgr2=0x55f1b1d39670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:30.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.282+0000 7fdd5de16700 1 --2- 172.21.15.16:0/520290930 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55f1b1d39220 0x55f1b1d39670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:30.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.282+0000 7fdd5de16700 1 -- 172.21.15.16:0/520290930 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f1b1d3e630 msgr2=0x55f1b1d37e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:30.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.282+0000 7fdd5de16700 1 --2- 172.21.15.16:0/520290930 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f1b1d3e630 0x55f1b1d37e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:30.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.282+0000 7fdd5de16700 1 -- 172.21.15.16:0/520290930 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd440035b0 con 0x55f1b1d14010 2023-10-16T12:26:30.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.282+0000 7fdd5de16700 1 --2- 172.21.15.16:0/520290930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1b1d14010 0x55f1b1d16680 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7fdd48058440 tx=0x7fdd480585e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bca6b84cd8837bf5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:30.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.282+0000 7fdd38ff9700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2023-10-16T12:26:30.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.282+0000 7fdd38ff9700 1 -- 172.21.15.16:0/520290930 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd480579a0 con 0x55f1b1d14010 2023-10-16T12:26:30.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.282+0000 7fdd38ff9700 1 -- 172.21.15.16:0/520290930 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(85..85 src has 1..85) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fdd48057b00 con 0x55f1b1d14010 2023-10-16T12:26:30.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.282+0000 7fdd6c264b80 1 -- 172.21.15.16:0/520290930 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f1b1d095b0 con 0x55f1b1d14010 2023-10-16T12:26:30.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.282+0000 7fdd38ff9700 1 -- 172.21.15.16:0/520290930 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdd4805c170 con 0x55f1b1d14010 2023-10-16T12:26:30.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.282+0000 7fdd6c264b80 10 client.6771 fetch_fsmap learned FSMap version 36 2023-10-16T12:26:30.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.282+0000 7fdd6c264b80 1 -- 172.21.15.16:0/520290930 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=2+}) v3 -- 0x55f1b1a93d90 con 0x55f1b1d14010 2023-10-16T12:26:30.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.282+0000 7fdd38ff9700 1 -- 172.21.15.16:0/520290930 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55f1b1d095b0 con 0x55f1b1d14010 2023-10-16T12:26:30.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.282+0000 7fdd38ff9700 1 -- 172.21.15.16:0/520290930 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 36) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fdd48060320 con 0x55f1b1d14010 2023-10-16T12:26:30.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.283+0000 7fdd6c264b80 10 client.6771 fetch_fsmap finished waiting for FSMap version 36 2023-10-16T12:26:30.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.283+0000 7fdd6c264b80 10 client.6771 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-16T12:26:30.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.283+0000 7fdd6c264b80 20 client.6771 populate_metadata read hostname 'smithi016' 2023-10-16T12:26:30.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.283+0000 7fdd6c264b80 1 --2- 172.21.15.16:0/520290930 >> [v2:172.21.15.16:6835/962243511,v1:172.21.15.16:6837/962243511] conn(0x55f1b1d3c940 0x55f1b1d3cd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:30.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.283+0000 7fdd6c264b80 4 client.6771 mds_command: new command op to 6627 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T12:26:30.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.283+0000 7fdd6c264b80 1 -- 172.21.15.16:0/520290930 --> [v2:172.21.15.16:6835/962243511,v1:172.21.15.16:6837/962243511] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f1b1c61e00 con 0x55f1b1d3c940 2023-10-16T12:26:30.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.283+0000 7fdd5d615700 1 --2- 172.21.15.16:0/520290930 >> [v2:172.21.15.16:6835/962243511,v1:172.21.15.16:6837/962243511] conn(0x55f1b1d3c940 0x55f1b1d3cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:30.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.283+0000 7fdd38ff9700 10 client.6771 ms_handle_connect on v2:172.21.15.16:6835/962243511 2023-10-16T12:26:30.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.283+0000 7fdd5d615700 1 --2- 172.21.15.16:0/520290930 >> [v2:172.21.15.16:6835/962243511,v1:172.21.15.16:6837/962243511] conn(0x55f1b1d3c940 0x55f1b1d3cd20 crc :-1 s=READY pgs=10 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd3676d2908f2145 server_cookie=ab6cee9acb98c530 in_seq=0 out_seq=0 2023-10-16T12:26:30.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.284+0000 7fdd38ff9700 1 -- 172.21.15.16:0/520290930 <== mds.0 v2:172.21.15.16:6835/962243511 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55f1b1c61e00 con 0x55f1b1d3c940 2023-10-16T12:26:30.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.284+0000 7fdd38ff9700 10 client.6771 handle_command_reply: tid=0 2023-10-16T12:26:30.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.284+0000 7fdd6c264b80 1 client.6771 shutdown 2023-10-16T12:26:30.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.284+0000 7fdd6c264b80 1 -- 172.21.15.16:0/520290930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1b1d14010 msgr2=0x55f1b1d16680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:30.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.284+0000 7fdd6c264b80 1 --2- 172.21.15.16:0/520290930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1b1d14010 0x55f1b1d16680 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7fdd48058440 tx=0x7fdd480585e0 comp rx=0 tx=0).stop 2023-10-16T12:26:30.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.284+0000 7fdd6c264b80 1 -- 172.21.15.16:0/520290930 shutdown_connections 2023-10-16T12:26:30.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.284+0000 7fdd6c264b80 1 --2- 172.21.15.16:0/520290930 >> [v2:172.21.15.16:6835/962243511,v1:172.21.15.16:6837/962243511] conn(0x55f1b1d3c940 0x55f1b1d3cd20 crc :-1 s=READY pgs=10 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:30.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.284+0000 7fdd6c264b80 1 --2- 172.21.15.16:0/520290930 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f1b1d3e630 0x55f1b1d37e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:30.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.284+0000 7fdd6c264b80 1 --2- 172.21.15.16:0/520290930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1b1d14010 0x55f1b1d16680 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:30.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.284+0000 7fdd38ff9700 0 client.6771 ms_handle_reset on v2:172.21.15.16:6835/962243511 2023-10-16T12:26:30.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.284+0000 7fdd6c264b80 1 --2- 172.21.15.16:0/520290930 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55f1b1d39220 0x55f1b1d39670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:30.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.284+0000 7fdd6c264b80 1 -- 172.21.15.16:0/520290930 >> 172.21.15.16:0/520290930 conn(0x55f1b1c43d80 msgr2=0x55f1b1d0b640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:30.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.284+0000 7fdd6c264b80 1 -- 172.21.15.16:0/520290930 shutdown_connections 2023-10-16T12:26:30.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.284+0000 7fdd6c264b80 1 -- 172.21.15.16:0/520290930 wait complete. 2023-10-16T12:26:30.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.284+0000 7fdd6c264b80 20 client.6771 trim_cache size 0 max 16384 2023-10-16T12:26:30.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.302+0000 7fdd6c264b80 1 Processor -- start 2023-10-16T12:26:30.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.302+0000 7fdd6c264b80 1 -- start start 2023-10-16T12:26:30.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.303+0000 7fdd6c264b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1b1d14010 0x55f1b1d3a220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:30.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.303+0000 7fdd6c264b80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55f1b1d39220 0x55f1b1d3cd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:30.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.303+0000 7fdd6c264b80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f1b1d3e630 0x55f1b1d3d260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:30.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.303+0000 7fdd6c264b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1b1ae98d0 con 0x55f1b1d14010 2023-10-16T12:26:30.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.303+0000 7fdd6c264b80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1b1cec3c0 con 0x55f1b1d3e630 2023-10-16T12:26:30.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.303+0000 7fdd6c264b80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f1b1d173a0 con 0x55f1b1d39220 2023-10-16T12:26:30.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.303+0000 7fdd5de16700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1b1d14010 0x55f1b1d3a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:30.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.303+0000 7fdd5de16700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1b1d14010 0x55f1b1d3a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39490/0 (socket says 172.21.15.16:39490) 2023-10-16T12:26:30.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.303+0000 7fdd5de16700 1 -- 172.21.15.16:0/2618489035 learned_addr learned my addr 172.21.15.16:0/2618489035 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:30.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.303+0000 7fdd5d615700 1 --2- 172.21.15.16:0/2618489035 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55f1b1d39220 0x55f1b1d3cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:30.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.303+0000 7fdd5e617700 1 --2- 172.21.15.16:0/2618489035 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f1b1d3e630 0x55f1b1d3d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:30.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.304+0000 7fdd5de16700 1 -- 172.21.15.16:0/2618489035 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55f1b1d39220 msgr2=0x55f1b1d3cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:30.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.304+0000 7fdd5de16700 1 --2- 172.21.15.16:0/2618489035 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55f1b1d39220 0x55f1b1d3cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:30.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.304+0000 7fdd5de16700 1 -- 172.21.15.16:0/2618489035 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f1b1d3e630 msgr2=0x55f1b1d3d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:30.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.304+0000 7fdd5de16700 1 --2- 172.21.15.16:0/2618489035 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f1b1d3e630 0x55f1b1d3d260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:30.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.304+0000 7fdd5de16700 1 -- 172.21.15.16:0/2618489035 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f1b1d12f20 con 0x55f1b1d14010 2023-10-16T12:26:30.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.304+0000 7fdd5de16700 1 --2- 172.21.15.16:0/2618489035 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1b1d14010 0x55f1b1d3a220 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fdd480581b0 tx=0x7fdd4805c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2698bd61514b8a6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:30.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.304+0000 7fdd3bfff700 1 -- 172.21.15.16:0/2618489035 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd4805c170 con 0x55f1b1d14010 2023-10-16T12:26:30.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.304+0000 7fdd3bfff700 1 -- 172.21.15.16:0/2618489035 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd48058920 con 0x55f1b1d14010 2023-10-16T12:26:30.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.304+0000 7fdd3bfff700 1 -- 172.21.15.16:0/2618489035 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd48060320 con 0x55f1b1d14010 2023-10-16T12:26:30.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.305+0000 7fdd6c264b80 1 -- 172.21.15.16:0/2618489035 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1b1d14010 msgr2=0x55f1b1d3a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:30.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.305+0000 7fdd6c264b80 1 --2- 172.21.15.16:0/2618489035 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1b1d14010 0x55f1b1d3a220 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fdd480581b0 tx=0x7fdd4805c000 comp rx=0 tx=0).stop 2023-10-16T12:26:30.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.305+0000 7fdd6c264b80 1 -- 172.21.15.16:0/2618489035 shutdown_connections 2023-10-16T12:26:30.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.305+0000 7fdd6c264b80 1 --2- 172.21.15.16:0/2618489035 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f1b1d3e630 0x55f1b1d3d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:30.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.305+0000 7fdd6c264b80 1 --2- 172.21.15.16:0/2618489035 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1b1d14010 0x55f1b1d3a220 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:30.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.305+0000 7fdd6c264b80 1 --2- 172.21.15.16:0/2618489035 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55f1b1d39220 0x55f1b1d3cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:30.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.305+0000 7fdd6c264b80 1 -- 172.21.15.16:0/2618489035 >> 172.21.15.16:0/2618489035 conn(0x55f1b1c43d80 msgr2=0x55f1b1d7d800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:30.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.305+0000 7fdd6c264b80 1 -- 172.21.15.16:0/2618489035 shutdown_connections 2023-10-16T12:26:30.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.305+0000 7fdd6c264b80 1 -- 172.21.15.16:0/2618489035 wait complete. 2023-10-16T12:26:30.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.305+0000 7fdd6c264b80 1 -- start start 2023-10-16T12:26:30.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.305+0000 7fdd6c264b80 1 Processor -- start 2023-10-16T12:26:30.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.306+0000 7fdd6c264b80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f1b1d14010 0x55f1b1d436e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:30.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.306+0000 7fdd6c264b80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55f1b1d39220 0x55f1b1d43c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:30.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.306+0000 7fdd6c264b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1b1d3e630 0x55f1b1d37330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:30.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.306+0000 7fdd6c264b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1b1dc3930 con 0x55f1b1d3e630 2023-10-16T12:26:30.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.306+0000 7fdd6c264b80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1b1d73f20 con 0x55f1b1d14010 2023-10-16T12:26:30.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.306+0000 7fdd6c264b80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f1b1d36bc0 con 0x55f1b1d39220 2023-10-16T12:26:30.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.306+0000 7fdd6c264b80 10 client.0 _ll_register_callbacks cb 0x7fdd48057ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T12:26:30.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.306+0000 7fdd5e617700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1b1d3e630 0x55f1b1d37330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:30.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.306+0000 7fdd5e617700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1b1d3e630 0x55f1b1d37330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39500/0 (socket says 172.21.15.16:39500) 2023-10-16T12:26:30.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.306+0000 7fdd5e617700 1 -- 172.21.15.16:0/3481622489 learned_addr learned my addr 172.21.15.16:0/3481622489 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:30.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.306+0000 7fdd5de16700 1 --2- 172.21.15.16:0/3481622489 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f1b1d14010 0x55f1b1d436e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:30.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.306+0000 7fdd5d615700 1 --2- 172.21.15.16:0/3481622489 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55f1b1d39220 0x55f1b1d43c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:30.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.307+0000 7fdd5e617700 1 -- 172.21.15.16:0/3481622489 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55f1b1d39220 msgr2=0x55f1b1d43c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:30.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.307+0000 7fdd5e617700 1 --2- 172.21.15.16:0/3481622489 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55f1b1d39220 0x55f1b1d43c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:30.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.307+0000 7fdd5e617700 1 -- 172.21.15.16:0/3481622489 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f1b1d14010 msgr2=0x55f1b1d436e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:30.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.307+0000 7fdd5e617700 1 --2- 172.21.15.16:0/3481622489 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f1b1d14010 0x55f1b1d436e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:30.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.307+0000 7fdd5e617700 1 -- 172.21.15.16:0/3481622489 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f1b1a93d90 con 0x55f1b1d3e630 2023-10-16T12:26:30.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.307+0000 7fdd5e617700 1 --2- 172.21.15.16:0/3481622489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1b1d3e630 0x55f1b1d37330 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fdd50013000 tx=0x7fdd500099b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee87483ead16a45f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:30.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.307+0000 7fdd38ff9700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2023-10-16T12:26:30.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.307+0000 7fdd38ff9700 1 -- 172.21.15.16:0/3481622489 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd50004350 con 0x55f1b1d3e630 2023-10-16T12:26:30.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.307+0000 7fdd6c264b80 1 -- 172.21.15.16:0/3481622489 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdd540205b0 con 0x55f1b1d3e630 2023-10-16T12:26:30.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.307+0000 7fdd38ff9700 1 -- 172.21.15.16:0/3481622489 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(85..85 src has 1..85) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fdd500044b0 con 0x55f1b1d3e630 2023-10-16T12:26:30.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.307+0000 7fdd38ff9700 1 -- 172.21.15.16:0/3481622489 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdd50004670 con 0x55f1b1d3e630 2023-10-16T12:26:30.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.307+0000 7fdd6c264b80 10 client.6777 fetch_fsmap learned FSMap version 36 2023-10-16T12:26:30.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.307+0000 7fdd6c264b80 1 -- 172.21.15.16:0/3481622489 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=2+}) v3 -- 0x55f1b1a7cc10 con 0x55f1b1d3e630 2023-10-16T12:26:30.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.307+0000 7fdd38ff9700 1 -- 172.21.15.16:0/3481622489 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd540205b0 con 0x55f1b1d3e630 2023-10-16T12:26:30.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.307+0000 7fdd38ff9700 1 -- 172.21.15.16:0/3481622489 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 36) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fdd500047d0 con 0x55f1b1d3e630 2023-10-16T12:26:30.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.307+0000 7fdd6c264b80 10 client.6777 fetch_fsmap finished waiting for FSMap version 36 2023-10-16T12:26:30.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.307+0000 7fdd6c264b80 10 client.6777 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-16T12:26:30.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.307+0000 7fdd6c264b80 20 client.6777 populate_metadata read hostname 'smithi016' 2023-10-16T12:26:30.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.307+0000 7fdd6c264b80 1 --2- 172.21.15.16:0/3481622489 >> [v2:172.21.15.16:6835/962243511,v1:172.21.15.16:6837/962243511] conn(0x55f1b1d37d40 0x55f1b1dc1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:30.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.308+0000 7fdd6c264b80 4 client.6777 mds_command: new command op to 6627 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T12:26:30.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.308+0000 7fdd6c264b80 1 -- 172.21.15.16:0/3481622489 --> [v2:172.21.15.16:6835/962243511,v1:172.21.15.16:6837/962243511] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55f1b1c61b20 con 0x55f1b1d37d40 2023-10-16T12:26:30.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.308+0000 7fdd5de16700 1 --2- 172.21.15.16:0/3481622489 >> [v2:172.21.15.16:6835/962243511,v1:172.21.15.16:6837/962243511] conn(0x55f1b1d37d40 0x55f1b1dc1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:30.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.308+0000 7fdd38ff9700 10 client.6777 ms_handle_connect on v2:172.21.15.16:6835/962243511 2023-10-16T12:26:30.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.308+0000 7fdd5de16700 1 --2- 172.21.15.16:0/3481622489 >> [v2:172.21.15.16:6835/962243511,v1:172.21.15.16:6837/962243511] conn(0x55f1b1d37d40 0x55f1b1dc1cc0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=81c74058f9887564 server_cookie=47df28c3f8606ba in_seq=0 out_seq=0 2023-10-16T12:26:30.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.308+0000 7fdd38ff9700 1 -- 172.21.15.16:0/3481622489 <== mds.0 v2:172.21.15.16:6835/962243511 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x55f1b1c61b20 con 0x55f1b1d37d40 2023-10-16T12:26:30.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.308+0000 7fdd38ff9700 10 client.6777 handle_command_reply: tid=0 2023-10-16T12:26:30.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.308+0000 7fdd6c264b80 1 client.6777 shutdown 2023-10-16T12:26:30.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.308+0000 7fdd6c264b80 1 -- 172.21.15.16:0/3481622489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1b1d3e630 msgr2=0x55f1b1d37330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:30.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.308+0000 7fdd6c264b80 1 --2- 172.21.15.16:0/3481622489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1b1d3e630 0x55f1b1d37330 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fdd50013000 tx=0x7fdd500099b0 comp rx=0 tx=0).stop 2023-10-16T12:26:30.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.308+0000 7fdd6c264b80 1 -- 172.21.15.16:0/3481622489 shutdown_connections 2023-10-16T12:26:30.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.308+0000 7fdd6c264b80 1 --2- 172.21.15.16:0/3481622489 >> [v2:172.21.15.16:6835/962243511,v1:172.21.15.16:6837/962243511] conn(0x55f1b1d37d40 0x55f1b1dc1cc0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:30.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.308+0000 7fdd6c264b80 1 --2- 172.21.15.16:0/3481622489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f1b1d3e630 0x55f1b1d37330 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:30.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.308+0000 7fdd6c264b80 1 --2- 172.21.15.16:0/3481622489 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f1b1d14010 0x55f1b1d436e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:30.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.308+0000 7fdd38ff9700 0 client.6777 ms_handle_reset on v2:172.21.15.16:6835/962243511 2023-10-16T12:26:30.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.308+0000 7fdd6c264b80 1 --2- 172.21.15.16:0/3481622489 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55f1b1d39220 0x55f1b1d43c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:30.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.308+0000 7fdd6c264b80 1 -- 172.21.15.16:0/3481622489 >> 172.21.15.16:0/3481622489 conn(0x55f1b1c43d80 msgr2=0x55f1b1d3c220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:30.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.309+0000 7fdd6c264b80 1 -- 172.21.15.16:0/3481622489 shutdown_connections 2023-10-16T12:26:30.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.309+0000 7fdd6c264b80 1 -- 172.21.15.16:0/3481622489 wait complete. 2023-10-16T12:26:30.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.309+0000 7fdd6c264b80 20 client.6777 trim_cache size 0 max 16384 2023-10-16T12:26:30.328 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-10-16T12:26:30.328 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "5b28dfa1-b21f-474a-9b2e-9fb77af58254", 2023-10-16T12:26:30.328 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 0, 2023-10-16T12:26:30.329 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 6627, 2023-10-16T12:26:30.329 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2023-10-16T12:26:30.329 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2023-10-16T12:26:30.329 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2023-10-16T12:26:30.329 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 6.028656937, 2023-10-16T12:26:30.329 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 36, 2023-10-16T12:26:30.330 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 85, 2023-10-16T12:26:30.330 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 85, 2023-10-16T12:26:30.330 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 11.252934250999999 2023-10-16T12:26:30.330 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-10-16T12:26:30.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.309+0000 7fdd6007a700 1 -- 172.21.15.16:0/2867449549 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7fdd3c06b020 msgr2=0x7fdd3c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:30.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.309+0000 7fdd6007a700 1 --2- 172.21.15.16:0/2867449549 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7fdd3c06b020 0x7fdd3c06d4e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fdd580ccce0 tx=0x7fdd54009040 comp rx=0 tx=0).stop 2023-10-16T12:26:30.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.309+0000 7fdd6007a700 1 -- 172.21.15.16:0/2867449549 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd580ce010 msgr2=0x7fdd58147400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:30.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.309+0000 7fdd6007a700 1 --2- 172.21.15.16:0/2867449549 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd580ce010 0x7fdd58147400 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fdd4800e9e0 tx=0x7fdd4800eda0 comp rx=0 tx=0).stop 2023-10-16T12:26:30.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.309+0000 7fdd6007a700 1 -- 172.21.15.16:0/2867449549 shutdown_connections 2023-10-16T12:26:30.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.309+0000 7fdd6007a700 1 --2- 172.21.15.16:0/2867449549 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7fdd3c06b020 0x7fdd3c06d4e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:30.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.309+0000 7fdd6007a700 1 --2- 172.21.15.16:0/2867449549 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdd5815a5c0 0x7fdd580c9fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:30.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.309+0000 7fdd6007a700 1 --2- 172.21.15.16:0/2867449549 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd580ce010 0x7fdd58147400 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:30.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.309+0000 7fdd6007a700 1 --2- 172.21.15.16:0/2867449549 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdd580d0ae0 0x7fdd58149b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:30.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.309+0000 7fdd6007a700 1 -- 172.21.15.16:0/2867449549 >> 172.21.15.16:0/2867449549 conn(0x7fdd580b1fd0 msgr2=0x7fdd580b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:30.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.309+0000 7fdd6007a700 1 -- 172.21.15.16:0/2867449549 shutdown_connections 2023-10-16T12:26:30.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.309+0000 7fdd6007a700 1 -- 172.21.15.16:0/2867449549 wait complete. 2023-10-16T12:26:30.334 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T12:26:30.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.458+0000 7ffb36540700 1 Processor -- start 2023-10-16T12:26:30.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.458+0000 7ffb36540700 1 -- start start 2023-10-16T12:26:30.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.459+0000 7ffb36540700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7ffb300ce4f0 0x7ffb300cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:30.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.459+0000 7ffb36540700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ffb3015ba10 0x7ffb300cfca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:30.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.459+0000 7ffb36540700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb300cc2a0 con 0x7ffb3015b630 2023-10-16T12:26:30.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.459+0000 7ffb36540700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb300cdb20 con 0x7ffb3015ba10 2023-10-16T12:26:30.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.459+0000 7ffb36540700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb300cc540 con 0x7ffb300ce4f0 2023-10-16T12:26:30.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.459+0000 7ffb34add700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ffb3015ba10 0x7ffb300cfca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:30.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.459+0000 7ffb34add700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ffb3015ba10 0x7ffb300cfca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39508/0 (socket says 172.21.15.16:39508) 2023-10-16T12:26:30.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.459+0000 7ffb34add700 1 -- 172.21.15.16:0/2244348543 learned_addr learned my addr 172.21.15.16:0/2244348543 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:30.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.459+0000 7ffb2f7fe700 1 --2- 172.21.15.16:0/2244348543 >> v2:172.21.15.49:3300/0 conn(0x7ffb300ce4f0 0x7ffb300cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:30.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.459+0000 7ffb34add700 1 -- 172.21.15.16:0/2244348543 >> v2:172.21.15.49:3300/0 conn(0x7ffb300ce4f0 msgr2=0x7ffb300cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:30.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.459+0000 7ffb34add700 1 --2- 172.21.15.16:0/2244348543 >> v2:172.21.15.49:3300/0 conn(0x7ffb300ce4f0 0x7ffb300cd5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:30.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.459+0000 7ffb34add700 1 -- 172.21.15.16:0/2244348543 >> v1:172.21.15.16:6789/0 conn(0x7ffb3015b630 legacy=0x7ffb300cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:30.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.459+0000 7ffb34add700 1 -- 172.21.15.16:0/2244348543 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb300cc820 con 0x7ffb3015ba10 2023-10-16T12:26:30.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.459+0000 7ffb2f7fe700 1 --2- 172.21.15.16:0/2244348543 >> v2:172.21.15.49:3300/0 conn(0x7ffb300ce4f0 0x7ffb300cd5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:26:30.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.459+0000 7ffb34add700 1 --2- 172.21.15.16:0/2244348543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb3015ba10 0x7ffb300cfca0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7ffb300c4f90 tx=0x7ffb2800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98b2eb4732200158 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:30.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.459+0000 7ffb2effd700 1 -- 172.21.15.16:0/2244348543 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb2800e070 con 0x7ffb3015ba10 2023-10-16T12:26:30.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.460+0000 7ffb2effd700 1 -- 172.21.15.16:0/2244348543 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb2800b9c0 con 0x7ffb3015ba10 2023-10-16T12:26:30.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.460+0000 7ffb2effd700 1 -- 172.21.15.16:0/2244348543 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb2800bd70 con 0x7ffb3015ba10 2023-10-16T12:26:30.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.460+0000 7ffb36540700 1 -- 172.21.15.16:0/2244348543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb3015ba10 msgr2=0x7ffb300cfca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:30.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.460+0000 7ffb36540700 1 --2- 172.21.15.16:0/2244348543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb3015ba10 0x7ffb300cfca0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7ffb300c4f90 tx=0x7ffb2800b330 comp rx=0 tx=0).stop 2023-10-16T12:26:30.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.460+0000 7ffb36540700 1 -- 172.21.15.16:0/2244348543 shutdown_connections 2023-10-16T12:26:30.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.460+0000 7ffb36540700 1 --2- 172.21.15.16:0/2244348543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb3015ba10 0x7ffb300cfca0 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:30.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.460+0000 7ffb36540700 1 --2- 172.21.15.16:0/2244348543 >> v2:172.21.15.49:3300/0 conn(0x7ffb300ce4f0 0x7ffb300cd5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:30.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.460+0000 7ffb36540700 1 -- 172.21.15.16:0/2244348543 >> 172.21.15.16:0/2244348543 conn(0x7ffb300d14b0 msgr2=0x7ffb300d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:30.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.460+0000 7ffb36540700 1 -- 172.21.15.16:0/2244348543 shutdown_connections 2023-10-16T12:26:30.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.460+0000 7ffb36540700 1 -- 172.21.15.16:0/2244348543 wait complete. 2023-10-16T12:26:30.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.461+0000 7ffb36540700 1 Processor -- start 2023-10-16T12:26:30.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.461+0000 7ffb36540700 1 -- start start 2023-10-16T12:26:30.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.462+0000 7ffb36540700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ffb300ce4f0 0x7ffb3015e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:30.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.462+0000 7ffb36540700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ffb3015b630 0x7ffb300c6010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:30.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.462+0000 7ffb36540700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb3015ba10 0x7ffb301f6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:30.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.462+0000 7ffb36540700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb3015eda0 con 0x7ffb3015ba10 2023-10-16T12:26:30.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.462+0000 7ffb36540700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb3015eac0 con 0x7ffb300ce4f0 2023-10-16T12:26:30.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.462+0000 7ffb36540700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb3006ba90 con 0x7ffb3015b630 2023-10-16T12:26:30.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.462+0000 7ffb34add700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb3015ba10 0x7ffb301f6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:30.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.462+0000 7ffb34add700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb3015ba10 0x7ffb301f6320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39514/0 (socket says 172.21.15.16:39514) 2023-10-16T12:26:30.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.462+0000 7ffb34add700 1 -- 172.21.15.16:0/1422130524 learned_addr learned my addr 172.21.15.16:0/1422130524 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:30.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.462+0000 7ffb2f7fe700 1 --2- 172.21.15.16:0/1422130524 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ffb3015b630 0x7ffb300c6010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:30.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.462+0000 7ffb2ffff700 1 --2- 172.21.15.16:0/1422130524 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ffb300ce4f0 0x7ffb3015e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:30.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.463+0000 7ffb34add700 1 -- 172.21.15.16:0/1422130524 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ffb3015b630 msgr2=0x7ffb300c6010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:30.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.463+0000 7ffb34add700 1 --2- 172.21.15.16:0/1422130524 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ffb3015b630 0x7ffb300c6010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:30.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.463+0000 7ffb34add700 1 -- 172.21.15.16:0/1422130524 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ffb300ce4f0 msgr2=0x7ffb3015e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:30.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.463+0000 7ffb34add700 1 --2- 172.21.15.16:0/1422130524 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ffb300ce4f0 0x7ffb3015e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:30.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.463+0000 7ffb34add700 1 -- 172.21.15.16:0/1422130524 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb28009b30 con 0x7ffb3015ba10 2023-10-16T12:26:30.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.463+0000 7ffb34add700 1 --2- 172.21.15.16:0/1422130524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb3015ba10 0x7ffb301f6320 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7ffb2800b730 tx=0x7ffb28004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:30.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.463+0000 7ffb2cff9700 1 -- 172.21.15.16:0/1422130524 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb28004870 con 0x7ffb3015ba10 2023-10-16T12:26:30.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.463+0000 7ffb2cff9700 1 -- 172.21.15.16:0/1422130524 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb280099b0 con 0x7ffb3015ba10 2023-10-16T12:26:30.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.463+0000 7ffb36540700 1 -- 172.21.15.16:0/1422130524 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb3006bd70 con 0x7ffb3015ba10 2023-10-16T12:26:30.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.463+0000 7ffb2cff9700 1 -- 172.21.15.16:0/1422130524 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb2801b670 con 0x7ffb3015ba10 2023-10-16T12:26:30.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.463+0000 7ffb36540700 1 -- 172.21.15.16:0/1422130524 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb300c6550 con 0x7ffb3015ba10 2023-10-16T12:26:30.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.464+0000 7ffb36540700 1 -- 172.21.15.16:0/1422130524 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb3006b8f0 con 0x7ffb3015ba10 2023-10-16T12:26:30.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.464+0000 7ffb2cff9700 1 -- 172.21.15.16:0/1422130524 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 31) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ffb2801b7d0 con 0x7ffb3015ba10 2023-10-16T12:26:30.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.465+0000 7ffb2cff9700 1 --2- 172.21.15.16:0/1422130524 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7ffb1c06b020 0x7ffb1c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:30.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.465+0000 7ffb2cff9700 1 -- 172.21.15.16:0/1422130524 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6057+0+0 (secure 0 0 0) 0x7ffb2808af40 con 0x7ffb3015ba10 2023-10-16T12:26:30.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.465+0000 7ffb2ffff700 1 --2- 172.21.15.16:0/1422130524 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7ffb1c06b020 0x7ffb1c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:30.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.465+0000 7ffb2ffff700 1 --2- 172.21.15.16:0/1422130524 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7ffb1c06b020 0x7ffb1c06d4e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ffb20007fd0 tx=0x7ffb20009040 comp rx=0 tx=0).ready entity=mgr.6396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:30.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.469+0000 7ffb2cff9700 1 -- 172.21.15.16:0/1422130524 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ffb3006b8f0 con 0x7ffb3015ba10 2023-10-16T12:26:30.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.721+0000 7ffb36540700 1 -- 172.21.15.16:0/1422130524 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7ffb3006b8f0 con 0x7ffb3015ba10 2023-10-16T12:26:30.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.721+0000 7ffb2cff9700 1 -- 172.21.15.16:0/1422130524 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v36) v1 ==== 53+0+73 (secure 0 0 0) 0x7ffb2801f060 con 0x7ffb3015ba10 2023-10-16T12:26:30.724 INFO:teuthology.orchestra.run.smithi016.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-16T12:26:30.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.723+0000 7ffb36540700 1 -- 172.21.15.16:0/1422130524 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7ffb1c06b020 msgr2=0x7ffb1c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:30.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.723+0000 7ffb36540700 1 --2- 172.21.15.16:0/1422130524 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7ffb1c06b020 0x7ffb1c06d4e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ffb20007fd0 tx=0x7ffb20009040 comp rx=0 tx=0).stop 2023-10-16T12:26:30.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.723+0000 7ffb36540700 1 -- 172.21.15.16:0/1422130524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb3015ba10 msgr2=0x7ffb301f6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:30.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.723+0000 7ffb36540700 1 --2- 172.21.15.16:0/1422130524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb3015ba10 0x7ffb301f6320 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7ffb2800b730 tx=0x7ffb28004ad0 comp rx=0 tx=0).stop 2023-10-16T12:26:30.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.723+0000 7ffb36540700 1 -- 172.21.15.16:0/1422130524 shutdown_connections 2023-10-16T12:26:30.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.723+0000 7ffb36540700 1 --2- 172.21.15.16:0/1422130524 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7ffb1c06b020 0x7ffb1c06d4e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:30.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.723+0000 7ffb36540700 1 --2- 172.21.15.16:0/1422130524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb3015ba10 0x7ffb301f6320 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:30.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.723+0000 7ffb36540700 1 --2- 172.21.15.16:0/1422130524 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ffb300ce4f0 0x7ffb3015e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:30.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.723+0000 7ffb36540700 1 --2- 172.21.15.16:0/1422130524 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ffb3015b630 0x7ffb300c6010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:30.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.723+0000 7ffb36540700 1 -- 172.21.15.16:0/1422130524 >> 172.21.15.16:0/1422130524 conn(0x7ffb300d14b0 msgr2=0x7ffb300d1890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:30.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.723+0000 7ffb36540700 1 -- 172.21.15.16:0/1422130524 shutdown_connections 2023-10-16T12:26:30.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:30.723+0000 7ffb36540700 1 -- 172.21.15.16:0/1422130524 wait complete. 2023-10-16T12:26:30.738 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-16T12:26:30.739 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T12:26:30.739 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi016.front.sepia.ceph.com 2023-10-16T12:26:30.739 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-16T12:26:30.740 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-16T12:26:30.740 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-16T12:26:30.740 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-16T12:26:30.741 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T12:26:30.741 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T12:26:30.741 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:26:30.742 DEBUG:teuthology.orchestra.run.smithi016:> ip addr 2023-10-16T12:26:30.759 INFO:teuthology.orchestra.run.smithi016.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T12:26:30.760 INFO:teuthology.orchestra.run.smithi016.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T12:26:30.760 INFO:teuthology.orchestra.run.smithi016.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T12:26:30.760 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:26:30.760 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 ::1/128 scope host 2023-10-16T12:26:30.761 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:26:30.761 INFO:teuthology.orchestra.run.smithi016.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T12:26:30.761 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e0 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:26:30.761 INFO:teuthology.orchestra.run.smithi016.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T12:26:30.761 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e1 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:26:30.762 INFO:teuthology.orchestra.run.smithi016.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T12:26:30.762 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:64 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:26:30.762 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.21.15.16/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-16T12:26:30.762 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft 41345sec preferred_lft 41345sec 2023-10-16T12:26:30.762 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::ec4:7aff:febd:1564/64 scope link 2023-10-16T12:26:30.763 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:26:30.763 INFO:teuthology.orchestra.run.smithi016.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T12:26:30.763 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:65 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:26:30.763 INFO:teuthology.orchestra.run.smithi016.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T12:26:30.763 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 36:cf:ea:25:db:f7 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:26:30.763 INFO:teuthology.orchestra.run.smithi016.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-16T12:26:30.764 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:26:30.764 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::cfc:c5ff:fe52:a786/64 scope link 2023-10-16T12:26:30.764 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:26:30.764 INFO:teuthology.orchestra.run.smithi016.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T12:26:30.764 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 6e:62:9d:a4:43:23 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T12:26:30.764 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::6c62:9dff:fea4:4323/64 scope link 2023-10-16T12:26:30.765 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:26:30.765 INFO:teuthology.orchestra.run.smithi016.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T12:26:30.765 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 36:cf:ea:25:db:f7 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-16T12:26:30.765 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::34cf:eaff:fe25:dbf7/64 scope link 2023-10-16T12:26:30.765 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:26:30.766 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:26:30.766 DEBUG:teuthology.orchestra.run.smithi016:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T12:26:30.825 INFO:teuthology.orchestra.run.smithi016.stdout:1 2023-10-16T12:26:30.826 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:26:30.827 DEBUG:teuthology.orchestra.run.smithi016:> route 2023-10-16T12:26:30.885 INFO:teuthology.orchestra.run.smithi016.stdout:Kernel IP routing table 2023-10-16T12:26:30.885 INFO:teuthology.orchestra.run.smithi016.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T12:26:30.885 INFO:teuthology.orchestra.run.smithi016.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-16T12:26:30.885 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-16T12:26:30.886 INFO:teuthology.orchestra.run.smithi016.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T12:26:30.887 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:26:30.887 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec bash -c ' 2023-10-16T12:26:30.888 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2023-10-16T12:26:30.888 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-16T12:26:30.888 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-16T12:26:30.889 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-16T12:26:30.889 DEBUG:teuthology.orchestra.run.smithi016:> ') 2023-10-16T12:26:30.993 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:26:30.994 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2023-10-16T12:26:31.011 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-16T12:26:31.011 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-16T12:26:31.011 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T12:26:31.012 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:26:31.012 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set brx.0 up 2023-10-16T12:26:31.078 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T12:26:31.078 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-16T12:26:31.136 INFO:teuthology.orchestra.run.smithi016.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-16T12:26:31.136 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T12:26:31.136 DEBUG:teuthology.orchestra.run.smithi016:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-16T12:26:31.192 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-16T12:26:31.193 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:26:31.193 DEBUG:teuthology.orchestra.run.smithi016:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T12:26:31.280 INFO:teuthology.orchestra.run.smithi016.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T12:26:31.280 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T12:26:31.281 INFO:teuthology.orchestra.run.smithi016.stdout:invalid new device string format 2023-10-16T12:26:31.289 INFO:teuthology.orchestra.run.smithi016.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T12:26:31.289 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T12:26:31.290 INFO:teuthology.orchestra.run.smithi016.stdout:invalid new device string format 2023-10-16T12:26:31.290 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: resolved to: "172.21.15.16:3300,172.21.15.49:3300,172.21.15.49:3301" 2023-10-16T12:26:31.290 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T12:26:31.362 INFO:teuthology.orchestra.run.smithi016.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T12:26:31.363 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T12:26:31.363 INFO:teuthology.orchestra.run.smithi016.stdout:invalid new device string format 2023-10-16T12:26:31.363 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: resolved to: "172.21.15.16:3300,172.21.15.49:3300,172.21.15.49:3301" 2023-10-16T12:26:31.363 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T12:26:31.364 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: resolved to: "172.21.15.16:6789,172.21.15.49:6789,172.21.15.49:6790" 2023-10-16T12:26:31.364 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.16:6789,172.21.15.49:6789,172.21.15.49:6790:/ 2023-10-16T12:26:31.365 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=5b28dfa1-b21f-474a-9b2e-9fb77af58254" will pass to kernel 2023-10-16T12:26:31.366 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-16T12:26:31.367 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:26:31.367 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-16T12:26:31.395 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:26:31.395 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T12:26:31.484 INFO:teuthology.orchestra.run.smithi016.stdout:client6496 2023-10-16T12:26:31.485 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:26:31.485 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:26:31.502 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-16T12:26:31.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.665+0000 7f8fcb629700 1 Processor -- start 2023-10-16T12:26:31.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.665+0000 7f8fcb629700 1 -- start start 2023-10-16T12:26:31.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.666+0000 7f8fcb629700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8fc40cf880 0x7f8fc40b52f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:31.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.666+0000 7f8fcb629700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f8fc40b5830 0x7f8fc415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:31.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.666+0000 7f8fcb629700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fc415bc80 con 0x7f8fc40cf4a0 2023-10-16T12:26:31.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.666+0000 7f8fcb629700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fc415b070 con 0x7f8fc40cf880 2023-10-16T12:26:31.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.666+0000 7f8fcb629700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fc415b350 con 0x7f8fc40b5830 2023-10-16T12:26:31.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.666+0000 7f8fc8bc4700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8fc40cf880 0x7f8fc40b52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:31.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.666+0000 7f8fc8bc4700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8fc40cf880 0x7f8fc40b52f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39524/0 (socket says 172.21.15.16:39524) 2023-10-16T12:26:31.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.666+0000 7f8fc8bc4700 1 -- 172.21.15.16:0/1614749125 learned_addr learned my addr 172.21.15.16:0/1614749125 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:31.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.666+0000 7f8fc9bc6700 1 --2- 172.21.15.16:0/1614749125 >> v2:172.21.15.49:3300/0 conn(0x7f8fc40b5830 0x7f8fc415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:31.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.666+0000 7f8fc8bc4700 1 -- 172.21.15.16:0/1614749125 >> v2:172.21.15.49:3300/0 conn(0x7f8fc40b5830 msgr2=0x7f8fc415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:31.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.666+0000 7f8fc8bc4700 1 --2- 172.21.15.16:0/1614749125 >> v2:172.21.15.49:3300/0 conn(0x7f8fc40b5830 0x7f8fc415a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:31.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.666+0000 7f8fc8bc4700 1 -- 172.21.15.16:0/1614749125 >> v1:172.21.15.16:6789/0 conn(0x7f8fc40cf4a0 legacy=0x7f8fc40726e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:31.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.666+0000 7f8fbbfff700 1 -- 172.21.15.16:0/1614749125 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3253562157 0 0) 0x7f8fc415bc80 con 0x7f8fc40cf4a0 2023-10-16T12:26:31.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.666+0000 7f8fc8bc4700 1 -- 172.21.15.16:0/1614749125 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fc415b630 con 0x7f8fc40cf880 2023-10-16T12:26:31.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.666+0000 7f8fc9bc6700 1 --2- 172.21.15.16:0/1614749125 >> v2:172.21.15.49:3300/0 conn(0x7f8fc40b5830 0x7f8fc415a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:26:31.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.666+0000 7f8fc8bc4700 1 --2- 172.21.15.16:0/1614749125 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fc40cf880 0x7f8fc40b52f0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f8fc4072df0 tx=0x7f8fb4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2772d7d12bce162 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:31.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.667+0000 7f8fbbfff700 1 -- 172.21.15.16:0/1614749125 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fb400b870 con 0x7f8fc40cf880 2023-10-16T12:26:31.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.667+0000 7f8fbbfff700 1 -- 172.21.15.16:0/1614749125 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fb400b9d0 con 0x7f8fc40cf880 2023-10-16T12:26:31.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.667+0000 7f8fbbfff700 1 -- 172.21.15.16:0/1614749125 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fb400bd60 con 0x7f8fc40cf880 2023-10-16T12:26:31.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.667+0000 7f8fcb629700 1 -- 172.21.15.16:0/1614749125 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fc40cf880 msgr2=0x7f8fc40b52f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:31.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.667+0000 7f8fcb629700 1 --2- 172.21.15.16:0/1614749125 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fc40cf880 0x7f8fc40b52f0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f8fc4072df0 tx=0x7f8fb4009bd0 comp rx=0 tx=0).stop 2023-10-16T12:26:31.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.667+0000 7f8fcb629700 1 -- 172.21.15.16:0/1614749125 shutdown_connections 2023-10-16T12:26:31.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.667+0000 7f8fcb629700 1 --2- 172.21.15.16:0/1614749125 >> v2:172.21.15.49:3300/0 conn(0x7f8fc40b5830 0x7f8fc415a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:31.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.667+0000 7f8fcb629700 1 --2- 172.21.15.16:0/1614749125 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fc40cf880 0x7f8fc40b52f0 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:31.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.667+0000 7f8fcb629700 1 -- 172.21.15.16:0/1614749125 >> 172.21.15.16:0/1614749125 conn(0x7f8fc40b1fd0 msgr2=0x7f8fc40b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:31.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.667+0000 7f8fcb629700 1 -- 172.21.15.16:0/1614749125 shutdown_connections 2023-10-16T12:26:31.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.667+0000 7f8fcb629700 1 -- 172.21.15.16:0/1614749125 wait complete. 2023-10-16T12:26:31.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.668+0000 7f8fcb629700 1 Processor -- start 2023-10-16T12:26:31.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.668+0000 7f8fcb629700 1 -- start start 2023-10-16T12:26:31.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.669+0000 7f8fcb629700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fc40b5830 0x7f8fc41496b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:31.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.669+0000 7f8fcb629700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8fc40cf4a0 0x7f8fc4156350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:31.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.669+0000 7f8fcb629700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8fc40cf880 0x7f8fc4156890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:31.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.669+0000 7f8fcb629700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fc41596c0 con 0x7f8fc40b5830 2023-10-16T12:26:31.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.669+0000 7f8fcb629700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fc40cbbc0 con 0x7f8fc40cf880 2023-10-16T12:26:31.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.669+0000 7f8fcb629700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fc4159100 con 0x7f8fc40cf4a0 2023-10-16T12:26:31.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.669+0000 7f8fc93c5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fc40b5830 0x7f8fc41496b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:31.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.669+0000 7f8fc93c5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fc40b5830 0x7f8fc41496b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39528/0 (socket says 172.21.15.16:39528) 2023-10-16T12:26:31.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.669+0000 7f8fc93c5700 1 -- 172.21.15.16:0/1128917452 learned_addr learned my addr 172.21.15.16:0/1128917452 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:31.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.670+0000 7f8fc8bc4700 1 --2- 172.21.15.16:0/1128917452 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8fc40cf4a0 0x7f8fc4156350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:31.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.670+0000 7f8fc9bc6700 1 --2- 172.21.15.16:0/1128917452 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8fc40cf880 0x7f8fc4156890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:31.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.670+0000 7f8fc93c5700 1 -- 172.21.15.16:0/1128917452 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8fc40cf4a0 msgr2=0x7f8fc4156350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:31.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.670+0000 7f8fc93c5700 1 --2- 172.21.15.16:0/1128917452 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8fc40cf4a0 0x7f8fc4156350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:31.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.670+0000 7f8fc93c5700 1 -- 172.21.15.16:0/1128917452 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8fc40cf880 msgr2=0x7f8fc4156890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:31.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.670+0000 7f8fc93c5700 1 --2- 172.21.15.16:0/1128917452 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8fc40cf880 0x7f8fc4156890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:31.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.670+0000 7f8fc93c5700 1 -- 172.21.15.16:0/1128917452 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fb4009550 con 0x7f8fc40b5830 2023-10-16T12:26:31.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.670+0000 7f8fc9bc6700 1 --2- 172.21.15.16:0/1128917452 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8fc40cf880 0x7f8fc4156890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:26:31.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.670+0000 7f8fc8bc4700 1 --2- 172.21.15.16:0/1128917452 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8fc40cf4a0 0x7f8fc4156350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:26:31.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.670+0000 7f8fc93c5700 1 --2- 172.21.15.16:0/1128917452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fc40b5830 0x7f8fc41496b0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f8fc00011e0 tx=0x7f8fc000cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:31.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.670+0000 7f8fb9ffb700 1 -- 172.21.15.16:0/1128917452 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc00054e0 con 0x7f8fc40b5830 2023-10-16T12:26:31.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.670+0000 7f8fb9ffb700 1 -- 172.21.15.16:0/1128917452 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fc0003680 con 0x7f8fc40b5830 2023-10-16T12:26:31.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.670+0000 7f8fcb629700 1 -- 172.21.15.16:0/1128917452 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fc41593e0 con 0x7f8fc40b5830 2023-10-16T12:26:31.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.670+0000 7f8fb9ffb700 1 -- 172.21.15.16:0/1128917452 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc0003a30 con 0x7f8fc40b5830 2023-10-16T12:26:31.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.670+0000 7f8fcb629700 1 -- 172.21.15.16:0/1128917452 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fc41f32c0 con 0x7f8fc40b5830 2023-10-16T12:26:31.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.671+0000 7f8fcb629700 1 -- 172.21.15.16:0/1128917452 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fc404ebb0 con 0x7f8fc40b5830 2023-10-16T12:26:31.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.671+0000 7f8fb9ffb700 1 -- 172.21.15.16:0/1128917452 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 31) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8fc0003b90 con 0x7f8fc40b5830 2023-10-16T12:26:31.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.672+0000 7f8fb9ffb700 1 --2- 172.21.15.16:0/1128917452 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f8fa806b020 0x7f8fa806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:31.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.672+0000 7f8fb9ffb700 1 -- 172.21.15.16:0/1128917452 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f8fc0088d40 con 0x7f8fc40b5830 2023-10-16T12:26:31.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.672+0000 7f8fc8bc4700 1 --2- 172.21.15.16:0/1128917452 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f8fa806b020 0x7f8fa806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:31.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.672+0000 7f8fc8bc4700 1 --2- 172.21.15.16:0/1128917452 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f8fa806b020 0x7f8fa806d4e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8fc41e7e90 tx=0x7f8fb4008030 comp rx=0 tx=0).ready entity=mgr.6396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:31.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.676+0000 7f8fb9ffb700 1 -- 172.21.15.16:0/1128917452 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8fc0053c10 con 0x7f8fc40b5830 2023-10-16T12:26:31.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.911+0000 7f8fcb629700 1 -- 172.21.15.16:0/1128917452 --> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f8fc415b910 con 0x7f8fa806b020 2023-10-16T12:26:31.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.912+0000 7f8fb9ffb700 1 -- 172.21.15.16:0/1128917452 <== mgr.6396 v2:172.21.15.16:6832/3415303009 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f8fb400bb40 con 0x7f8fa806b020 2023-10-16T12:26:31.915 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2023-10-16T12:26:31.915 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:26:31.916 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cephfs" 2023-10-16T12:26:31.916 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:26:31.916 INFO:teuthology.orchestra.run.smithi016.stdout:] 2023-10-16T12:26:31.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.914+0000 7f8fcb629700 1 -- 172.21.15.16:0/1128917452 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f8fa806b020 msgr2=0x7f8fa806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:31.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.914+0000 7f8fcb629700 1 --2- 172.21.15.16:0/1128917452 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f8fa806b020 0x7f8fa806d4e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8fc41e7e90 tx=0x7f8fb4008030 comp rx=0 tx=0).stop 2023-10-16T12:26:31.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.914+0000 7f8fcb629700 1 -- 172.21.15.16:0/1128917452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fc40b5830 msgr2=0x7f8fc41496b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:31.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.914+0000 7f8fcb629700 1 --2- 172.21.15.16:0/1128917452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fc40b5830 0x7f8fc41496b0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f8fc00011e0 tx=0x7f8fc000cae0 comp rx=0 tx=0).stop 2023-10-16T12:26:31.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.914+0000 7f8fcb629700 1 -- 172.21.15.16:0/1128917452 shutdown_connections 2023-10-16T12:26:31.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.914+0000 7f8fcb629700 1 --2- 172.21.15.16:0/1128917452 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f8fa806b020 0x7f8fa806d4e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:31.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.914+0000 7f8fcb629700 1 --2- 172.21.15.16:0/1128917452 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8fc40cf880 0x7f8fc4156890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:31.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.914+0000 7f8fcb629700 1 --2- 172.21.15.16:0/1128917452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fc40b5830 0x7f8fc41496b0 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:31.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.914+0000 7f8fcb629700 1 --2- 172.21.15.16:0/1128917452 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8fc40cf4a0 0x7f8fc4156350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:31.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.914+0000 7f8fcb629700 1 -- 172.21.15.16:0/1128917452 >> 172.21.15.16:0/1128917452 conn(0x7f8fc40b1fd0 msgr2=0x7f8fc40b1550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:31.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.915+0000 7f8fcb629700 1 -- 172.21.15.16:0/1128917452 shutdown_connections 2023-10-16T12:26:31.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:31.915+0000 7f8fcb629700 1 -- 172.21.15.16:0/1128917452 wait complete. 2023-10-16T12:26:31.930 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-16T12:26:32.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:32.055+0000 7fb43e292700 1 Processor -- start 2023-10-16T12:26:32.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:32.055+0000 7fb43e292700 1 -- start start 2023-10-16T12:26:32.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:32.055+0000 7fb43e292700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb438160fe0 0x7fb4380ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:32.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:32.055+0000 7fb43e292700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fb43806aef0 0x7fb4380adf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:32.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:32.055+0000 7fb43e292700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4381613c0 con 0x7fb43806da30 2023-10-16T12:26:32.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:32.055+0000 7fb43e292700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4380cf680 con 0x7fb43806aef0 2023-10-16T12:26:32.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:32.055+0000 7fb43e292700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4380adb00 con 0x7fb438160fe0 2023-10-16T12:26:32.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:32.056+0000 7fb437fff700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb438160fe0 0x7fb4380ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:32.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:32.056+0000 7fb437fff700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb438160fe0 0x7fb4380ac8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:56026/0 (socket says 172.21.15.16:56026) 2023-10-16T12:26:32.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:32.056+0000 7fb437fff700 1 -- 172.21.15.16:0/1215804629 learned_addr learned my addr 172.21.15.16:0/1215804629 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:32.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:32.056+0000 7fb43c82f700 1 --2- 172.21.15.16:0/1215804629 >> v2:172.21.15.49:3300/0 conn(0x7fb43806aef0 0x7fb4380adf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:32.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:32.056+0000 7fb436ffd700 1 -- 172.21.15.16:0/1215804629 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1387985699 0 0) 0x7fb42c001110 con 0x7fb43806da30 2023-10-16T12:26:32.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:32.056+0000 7fb436ffd700 1 -- 172.21.15.16:0/1215804629 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4200035b0 con 0x7fb43806da30 2023-10-16T12:26:32.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:32.056+0000 7fb437fff700 1 -- 172.21.15.16:0/1215804629 >> v2:172.21.15.49:3300/0 conn(0x7fb43806aef0 msgr2=0x7fb4380adf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:32.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:32.056+0000 7fb437fff700 1 --2- 172.21.15.16:0/1215804629 >> v2:172.21.15.49:3300/0 conn(0x7fb43806aef0 0x7fb4380adf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:32.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:32.056+0000 7fb437fff700 1 -- 172.21.15.16:0/1215804629 >> v1:172.21.15.49:6789/0 conn(0x7fb43806da30 legacy=0x7fb43806a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:32.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:32.056+0000 7fb437fff700 1 -- 172.21.15.16:0/1215804629 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4380acef0 con 0x7fb438160fe0 2023-10-16T12:26:32.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:32.057+0000 7fb437fff700 1 --2- 172.21.15.16:0/1215804629 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb438160fe0 0x7fb4380ac8d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fb428006fd0 tx=0x7fb428009bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a7f8d659e3a0bd81 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:32.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:32.057+0000 7fb436ffd700 1 -- 172.21.15.16:0/1215804629 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb428005e30 con 0x7fb438160fe0 2023-10-16T12:26:32.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:32.057+0000 7fb436ffd700 1 -- 172.21.15.16:0/1215804629 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb428004030 con 0x7fb438160fe0 2023-10-16T12:26:32.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:32.057+0000 7fb436ffd700 1 -- 172.21.15.16:0/1215804629 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4280043e0 con 0x7fb438160fe0 2023-10-16T12:26:32.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:32.057+0000 7fb43e292700 1 -- 172.21.15.16:0/1215804629 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb438160fe0 msgr2=0x7fb4380ac8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:32.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:32.057+0000 7fb43e292700 1 --2- 172.21.15.16:0/1215804629 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb438160fe0 0x7fb4380ac8d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fb428006fd0 tx=0x7fb428009bd0 comp rx=0 tx=0).stop 2023-10-16T12:26:32.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:32.057+0000 7fb43e292700 1 -- 172.21.15.16:0/1215804629 shutdown_connections 2023-10-16T12:26:32.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:32.057+0000 7fb43e292700 1 --2- 172.21.15.16:0/1215804629 >> v2:172.21.15.49:3300/0 conn(0x7fb43806aef0 0x7fb4380adf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:32.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:32.057+0000 7fb43e292700 1 --2- 172.21.15.16:0/1215804629 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb438160fe0 0x7fb4380ac8d0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:32.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:32.057+0000 7fb43e292700 1 -- 172.21.15.16:0/1215804629 >> 172.21.15.16:0/1215804629 conn(0x7fb4380b1ed0 msgr2=0x7fb4380b22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:32.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:32.057+0000 7fb43e292700 1 -- 172.21.15.16:0/1215804629 shutdown_connections 2023-10-16T12:26:32.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:32.057+0000 7fb43e292700 1 -- 172.21.15.16:0/1215804629 wait complete. 2023-10-16T12:26:32.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:32.059+0000 7fb43e292700 1 Processor -- start 2023-10-16T12:26:32.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:32.059+0000 7fb43e292700 1 -- start start 2023-10-16T12:26:32.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:32.059+0000 7fb43e292700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb43806aef0 0x7fb4380b2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:32.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:32.059+0000 7fb43e292700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb43806da30 0x7fb438153700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:32.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:32.059+0000 7fb43e292700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb438160fe0 0x7fb4380c5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:32.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:32.060+0000 7fb43e292700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4380b4220 con 0x7fb43806aef0 2023-10-16T12:26:32.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:32.060+0000 7fb43e292700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4380c5980 con 0x7fb438160fe0 2023-10-16T12:26:32.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:32.060+0000 7fb43e292700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4380b35a0 con 0x7fb43806da30 2023-10-16T12:26:32.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:32.060+0000 7fb437fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb43806aef0 0x7fb4380b2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:32.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:32.060+0000 7fb437fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb43806aef0 0x7fb4380b2fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39542/0 (socket says 172.21.15.16:39542) 2023-10-16T12:26:32.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:32.060+0000 7fb437fff700 1 -- 172.21.15.16:0/1286103584 learned_addr learned my addr 172.21.15.16:0/1286103584 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:32.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:32.060+0000 7fb4377fe700 1 --2- 172.21.15.16:0/1286103584 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb43806da30 0x7fb438153700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:32.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:32.060+0000 7fb43c82f700 1 --2- 172.21.15.16:0/1286103584 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb438160fe0 0x7fb4380c5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:32.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:32.060+0000 7fb437fff700 1 -- 172.21.15.16:0/1286103584 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb43806da30 msgr2=0x7fb438153700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:32.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:32.060+0000 7fb437fff700 1 --2- 172.21.15.16:0/1286103584 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb43806da30 0x7fb438153700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:32.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:32.060+0000 7fb437fff700 1 -- 172.21.15.16:0/1286103584 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb438160fe0 msgr2=0x7fb4380c5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:32.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:32.060+0000 7fb437fff700 1 --2- 172.21.15.16:0/1286103584 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb438160fe0 0x7fb4380c5440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:32.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:32.060+0000 7fb437fff700 1 -- 172.21.15.16:0/1286103584 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb428009500 con 0x7fb43806aef0 2023-10-16T12:26:32.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:32.060+0000 7fb437fff700 1 --2- 172.21.15.16:0/1286103584 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb43806aef0 0x7fb4380b2fc0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fb428009c10 tx=0x7fb428004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:32.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:32.060+0000 7fb434ff9700 1 -- 172.21.15.16:0/1286103584 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb428003ce0 con 0x7fb43806aef0 2023-10-16T12:26:32.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:32.060+0000 7fb434ff9700 1 -- 172.21.15.16:0/1286103584 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb428003e60 con 0x7fb43806aef0 2023-10-16T12:26:32.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:32.060+0000 7fb43e292700 1 -- 172.21.15.16:0/1286103584 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4380b3880 con 0x7fb43806aef0 2023-10-16T12:26:32.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:32.060+0000 7fb434ff9700 1 -- 172.21.15.16:0/1286103584 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb42801d670 con 0x7fb43806aef0 2023-10-16T12:26:32.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:32.060+0000 7fb43e292700 1 -- 172.21.15.16:0/1286103584 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4380b3e40 con 0x7fb43806aef0 2023-10-16T12:26:32.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:32.061+0000 7fb43e292700 1 -- 172.21.15.16:0/1286103584 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4380b4360 con 0x7fb43806aef0 2023-10-16T12:26:32.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:32.062+0000 7fb434ff9700 1 -- 172.21.15.16:0/1286103584 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 31) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb42801d7d0 con 0x7fb43806aef0 2023-10-16T12:26:32.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:32.062+0000 7fb434ff9700 1 --2- 172.21.15.16:0/1286103584 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7fb42406b020 0x7fb42406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:32.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:32.062+0000 7fb434ff9700 1 -- 172.21.15.16:0/1286103584 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fb42808abf0 con 0x7fb43806aef0 2023-10-16T12:26:32.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:32.062+0000 7fb4377fe700 1 --2- 172.21.15.16:0/1286103584 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7fb42406b020 0x7fb42406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:32.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:32.063+0000 7fb4377fe700 1 --2- 172.21.15.16:0/1286103584 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7fb42406b020 0x7fb42406d4e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb42c007fd0 tx=0x7fb42c009040 comp rx=0 tx=0).ready entity=mgr.6396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:32.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:32.067+0000 7fb434ff9700 1 -- 172.21.15.16:0/1286103584 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb4380b4360 con 0x7fb43806aef0 2023-10-16T12:26:32.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:32.335+0000 7fb43e292700 1 -- 172.21.15.16:0/1286103584 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7fb4380b4360 con 0x7fb43806aef0 2023-10-16T12:26:33.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.319+0000 7fb434ff9700 1 -- 172.21.15.16:0/1286103584 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v32) v1 ==== 92+0+0 (secure 0 0 0) 0x7fb4380b4360 con 0x7fb43806aef0 2023-10-16T12:26:33.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.323+0000 7fb43e292700 1 -- 172.21.15.16:0/1286103584 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7fb42406b020 msgr2=0x7fb42406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:33.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.323+0000 7fb43e292700 1 --2- 172.21.15.16:0/1286103584 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7fb42406b020 0x7fb42406d4e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb42c007fd0 tx=0x7fb42c009040 comp rx=0 tx=0).stop 2023-10-16T12:26:33.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.323+0000 7fb43e292700 1 -- 172.21.15.16:0/1286103584 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb43806aef0 msgr2=0x7fb4380b2fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:33.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.323+0000 7fb43e292700 1 --2- 172.21.15.16:0/1286103584 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb43806aef0 0x7fb4380b2fc0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fb428009c10 tx=0x7fb428004750 comp rx=0 tx=0).stop 2023-10-16T12:26:33.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.323+0000 7fb43e292700 1 -- 172.21.15.16:0/1286103584 shutdown_connections 2023-10-16T12:26:33.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.323+0000 7fb43e292700 1 --2- 172.21.15.16:0/1286103584 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7fb42406b020 0x7fb42406d4e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:33.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.323+0000 7fb43e292700 1 --2- 172.21.15.16:0/1286103584 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb438160fe0 0x7fb4380c5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:33.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.323+0000 7fb43e292700 1 --2- 172.21.15.16:0/1286103584 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb43806aef0 0x7fb4380b2fc0 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:33.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.323+0000 7fb43e292700 1 --2- 172.21.15.16:0/1286103584 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb43806da30 0x7fb438153700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:33.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.323+0000 7fb43e292700 1 -- 172.21.15.16:0/1286103584 >> 172.21.15.16:0/1286103584 conn(0x7fb4380b1ed0 msgr2=0x7fb4380afd40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:33.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.325+0000 7fb43e292700 1 -- 172.21.15.16:0/1286103584 shutdown_connections 2023-10-16T12:26:33.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.325+0000 7fb43e292700 1 -- 172.21.15.16:0/1286103584 wait complete. 2023-10-16T12:26:33.344 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-16T12:26:33.475 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:26:33.473+0000 7fd9b8dab200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T12:26:33.483 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:33.480+0000 7fbdb19b2200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T12:26:33.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.481+0000 7f1f51378700 1 Processor -- start 2023-10-16T12:26:33.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.481+0000 7f1f51378700 1 -- start start 2023-10-16T12:26:33.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.482+0000 7f1f51378700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1f4c15b790 0x7f1f4c0cda60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:33.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.482+0000 7f1f51378700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1f4c15bb70 0x7f1f4c06d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:33.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.482+0000 7f1f51378700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f4c0ce0e0 con 0x7f1f4c06dea0 2023-10-16T12:26:33.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.482+0000 7f1f51378700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f4c06e310 con 0x7f1f4c15bb70 2023-10-16T12:26:33.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.482+0000 7f1f51378700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f4c06a7e0 con 0x7f1f4c15b790 2023-10-16T12:26:33.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.482+0000 7f1f4a7fc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1f4c15bb70 0x7f1f4c06d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:33.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.482+0000 7f1f4a7fc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1f4c15bb70 0x7f1f4c06d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39578/0 (socket says 172.21.15.16:39578) 2023-10-16T12:26:33.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.482+0000 7f1f4a7fc700 1 -- 172.21.15.16:0/1771604466 learned_addr learned my addr 172.21.15.16:0/1771604466 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:33.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.482+0000 7f1f4affd700 1 --2- 172.21.15.16:0/1771604466 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1f4c15b790 0x7f1f4c0cda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:33.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.483+0000 7f1f4a7fc700 1 -- 172.21.15.16:0/1771604466 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1f4c15b790 msgr2=0x7f1f4c0cda60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:33.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.483+0000 7f1f4a7fc700 1 --2- 172.21.15.16:0/1771604466 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1f4c15b790 0x7f1f4c0cda60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:33.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.483+0000 7f1f4a7fc700 1 -- 172.21.15.16:0/1771604466 >> v1:172.21.15.49:6789/0 conn(0x7f1f4c06dea0 legacy=0x7f1f4c15f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:33.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.483+0000 7f1f4a7fc700 1 -- 172.21.15.16:0/1771604466 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f4c15c410 con 0x7f1f4c15bb70 2023-10-16T12:26:33.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.483+0000 7f1f4a7fc700 1 --2- 172.21.15.16:0/1771604466 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f4c15bb70 0x7f1f4c06d960 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f1f3c0098f0 tx=0x7f1f3c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f964361fdeadd733 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:33.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.483+0000 7f1f49ffb700 1 -- 172.21.15.16:0/1771604466 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f3c011830 con 0x7f1f4c15bb70 2023-10-16T12:26:33.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.483+0000 7f1f49ffb700 1 -- 172.21.15.16:0/1771604466 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f3c011990 con 0x7f1f4c15bb70 2023-10-16T12:26:33.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.483+0000 7f1f49ffb700 1 -- 172.21.15.16:0/1771604466 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f3c011d20 con 0x7f1f4c15bb70 2023-10-16T12:26:33.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.484+0000 7f1f51378700 1 -- 172.21.15.16:0/1771604466 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f4c15bb70 msgr2=0x7f1f4c06d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:33.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.484+0000 7f1f51378700 1 --2- 172.21.15.16:0/1771604466 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f4c15bb70 0x7f1f4c06d960 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f1f3c0098f0 tx=0x7f1f3c009c00 comp rx=0 tx=0).stop 2023-10-16T12:26:33.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.484+0000 7f1f51378700 1 -- 172.21.15.16:0/1771604466 shutdown_connections 2023-10-16T12:26:33.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.484+0000 7f1f51378700 1 --2- 172.21.15.16:0/1771604466 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1f4c15b790 0x7f1f4c0cda60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:33.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.484+0000 7f1f51378700 1 --2- 172.21.15.16:0/1771604466 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f4c15bb70 0x7f1f4c06d960 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:33.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.484+0000 7f1f51378700 1 -- 172.21.15.16:0/1771604466 >> 172.21.15.16:0/1771604466 conn(0x7f1f4c0cc5e0 msgr2=0x7f1f4c0cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:33.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.484+0000 7f1f51378700 1 -- 172.21.15.16:0/1771604466 shutdown_connections 2023-10-16T12:26:33.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.484+0000 7f1f51378700 1 -- 172.21.15.16:0/1771604466 wait complete. 2023-10-16T12:26:33.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.485+0000 7f1f51378700 1 Processor -- start 2023-10-16T12:26:33.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.486+0000 7f1f51378700 1 -- start start 2023-10-16T12:26:33.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.486+0000 7f1f51378700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f4c06dea0 0x7f1f4c15f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:33.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.486+0000 7f1f51378700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1f4c15b790 0x7f1f4c0ca960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:33.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.486+0000 7f1f51378700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1f4c15bb70 0x7f1f4c1572b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:33.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.486+0000 7f1f51378700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f4c0cbf50 con 0x7f1f4c06dea0 2023-10-16T12:26:33.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.486+0000 7f1f51378700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f4c0cb2d0 con 0x7f1f4c15b790 2023-10-16T12:26:33.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.486+0000 7f1f51378700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f4c0cb5b0 con 0x7f1f4c15bb70 2023-10-16T12:26:33.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.486+0000 7f1f4affd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f4c06dea0 0x7f1f4c15f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:33.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.486+0000 7f1f4affd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f4c06dea0 0x7f1f4c15f6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39582/0 (socket says 172.21.15.16:39582) 2023-10-16T12:26:33.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.486+0000 7f1f4affd700 1 -- 172.21.15.16:0/2291953085 learned_addr learned my addr 172.21.15.16:0/2291953085 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:33.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.487+0000 7f1f4a7fc700 1 --2- 172.21.15.16:0/2291953085 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1f4c15b790 0x7f1f4c0ca960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:33.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.487+0000 7f1f4b7fe700 1 --2- 172.21.15.16:0/2291953085 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1f4c15bb70 0x7f1f4c1572b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:33.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.487+0000 7f1f4affd700 1 -- 172.21.15.16:0/2291953085 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1f4c15bb70 msgr2=0x7f1f4c1572b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:33.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.487+0000 7f1f4affd700 1 --2- 172.21.15.16:0/2291953085 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1f4c15bb70 0x7f1f4c1572b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:33.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.487+0000 7f1f4affd700 1 -- 172.21.15.16:0/2291953085 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1f4c15b790 msgr2=0x7f1f4c0ca960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:33.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.487+0000 7f1f4affd700 1 --2- 172.21.15.16:0/2291953085 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1f4c15b790 0x7f1f4c0ca960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:33.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.487+0000 7f1f4affd700 1 -- 172.21.15.16:0/2291953085 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f3c009550 con 0x7f1f4c06dea0 2023-10-16T12:26:33.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.487+0000 7f1f4affd700 1 --2- 172.21.15.16:0/2291953085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f4c06dea0 0x7f1f4c15f6b0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f1f4000e9b0 tx=0x7f1f4000ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:33.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.487+0000 7f1f337fe700 1 -- 172.21.15.16:0/2291953085 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f40009710 con 0x7f1f4c06dea0 2023-10-16T12:26:33.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.487+0000 7f1f337fe700 1 -- 172.21.15.16:0/2291953085 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f4000ee50 con 0x7f1f4c06dea0 2023-10-16T12:26:33.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.487+0000 7f1f51378700 1 -- 172.21.15.16:0/2291953085 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f4c0cb890 con 0x7f1f4c06dea0 2023-10-16T12:26:33.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.487+0000 7f1f337fe700 1 -- 172.21.15.16:0/2291953085 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f4000ca50 con 0x7f1f4c06dea0 2023-10-16T12:26:33.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.487+0000 7f1f51378700 1 -- 172.21.15.16:0/2291953085 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f4c15fbf0 con 0x7f1f4c06dea0 2023-10-16T12:26:33.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.488+0000 7f1f51378700 1 -- 172.21.15.16:0/2291953085 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f4c15bff0 con 0x7f1f4c06dea0 2023-10-16T12:26:33.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.489+0000 7f1f337fe700 1 -- 172.21.15.16:0/2291953085 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 32) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f1f4000cbb0 con 0x7f1f4c06dea0 2023-10-16T12:26:33.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.489+0000 7f1f337fe700 1 --2- 172.21.15.16:0/2291953085 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f1f3806b070 0x7f1f3806d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:33.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.489+0000 7f1f4a7fc700 1 -- 172.21.15.16:0/2291953085 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f1f3806b070 msgr2=0x7f1f3806d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3415303009 2023-10-16T12:26:33.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.489+0000 7f1f337fe700 1 -- 172.21.15.16:0/2291953085 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f1f400897f0 con 0x7f1f4c06dea0 2023-10-16T12:26:33.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.489+0000 7f1f4a7fc700 1 --2- 172.21.15.16:0/2291953085 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f1f3806b070 0x7f1f3806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:26:33.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.494+0000 7f1f337fe700 1 -- 172.21.15.16:0/2291953085 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1f400546c0 con 0x7f1f4c06dea0 2023-10-16T12:26:33.544 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:26:33.542+0000 7fd9b8dab200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T12:26:33.554 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:33.551+0000 7fbdb19b2200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T12:26:33.618 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:26:33.616+0000 7fd9b8dab200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T12:26:33.629 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:33.627+0000 7fbdb19b2200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T12:26:33.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.690+0000 7f1f4a7fc700 1 -- 172.21.15.16:0/2291953085 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f1f3806b070 msgr2=0x7f1f3806d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3415303009 2023-10-16T12:26:33.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.690+0000 7f1f4a7fc700 1 --2- 172.21.15.16:0/2291953085 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f1f3806b070 0x7f1f3806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:26:33.700 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:26:33.698+0000 7fd9b8dab200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T12:26:33.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.699+0000 7f1f51378700 1 -- 172.21.15.16:0/2291953085 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f1f4c04ebb0 con 0x7f1f4c06dea0 2023-10-16T12:26:33.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.702+0000 7f1f337fe700 1 -- 172.21.15.16:0/2291953085 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v23)=0 v23) v1 ==== 138+0+0 (secure 0 0 0) 0x7f1f4000ce60 con 0x7f1f4c06dea0 2023-10-16T12:26:33.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.704+0000 7f1f51378700 1 -- 172.21.15.16:0/2291953085 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f1f3806b070 msgr2=0x7f1f3806d530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:26:33.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.704+0000 7f1f51378700 1 --2- 172.21.15.16:0/2291953085 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f1f3806b070 0x7f1f3806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:33.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.704+0000 7f1f51378700 1 -- 172.21.15.16:0/2291953085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f4c06dea0 msgr2=0x7f1f4c15f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:33.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.704+0000 7f1f51378700 1 --2- 172.21.15.16:0/2291953085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f4c06dea0 0x7f1f4c15f6b0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f1f4000e9b0 tx=0x7f1f4000ecc0 comp rx=0 tx=0).stop 2023-10-16T12:26:33.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.704+0000 7f1f51378700 1 -- 172.21.15.16:0/2291953085 shutdown_connections 2023-10-16T12:26:33.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.704+0000 7f1f51378700 1 --2- 172.21.15.16:0/2291953085 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f1f3806b070 0x7f1f3806d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:33.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.704+0000 7f1f51378700 1 --2- 172.21.15.16:0/2291953085 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1f4c15bb70 0x7f1f4c1572b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:33.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.704+0000 7f1f51378700 1 --2- 172.21.15.16:0/2291953085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f4c06dea0 0x7f1f4c15f6b0 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:33.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.704+0000 7f1f51378700 1 --2- 172.21.15.16:0/2291953085 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1f4c15b790 0x7f1f4c0ca960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:33.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.704+0000 7f1f51378700 1 -- 172.21.15.16:0/2291953085 >> 172.21.15.16:0/2291953085 conn(0x7f1f4c0cc5e0 msgr2=0x7f1f4c06d790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:33.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.704+0000 7f1f51378700 1 -- 172.21.15.16:0/2291953085 shutdown_connections 2023-10-16T12:26:33.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.704+0000 7f1f51378700 1 -- 172.21.15.16:0/2291953085 wait complete. 2023-10-16T12:26:33.712 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:33.709+0000 7fbdb19b2200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T12:26:33.720 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-16T12:26:33.849 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:26:33.846+0000 7fd9b8dab200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T12:26:33.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.847+0000 7fc94ca7a700 1 Processor -- start 2023-10-16T12:26:33.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.847+0000 7fc94ca7a700 1 -- start start 2023-10-16T12:26:33.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.847+0000 7fc94ca7a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc94815da10 0x7fc9480d1520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:33.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.848+0000 7fc94ca7a700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fc9480d4060 0x7fc9480cf2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:33.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.848+0000 7fc94ca7a700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9480d4540 con 0x7fc9480d1d20 2023-10-16T12:26:33.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.848+0000 7fc94ca7a700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9480ce7e0 con 0x7fc9480d4060 2023-10-16T12:26:33.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.848+0000 7fc94ca7a700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9480ceac0 con 0x7fc94815da10 2023-10-16T12:26:33.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.848+0000 7fc946d9d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc9480d1d20 0x7fc9480d2130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57146/0 (socket says 172.21.15.16:57146) 2023-10-16T12:26:33.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.848+0000 7fc946d9d700 1 -- 172.21.15.16:0/340462729 learned_addr learned my addr 172.21.15.16:0/340462729 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:26:33.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.848+0000 7fc94659c700 1 --2- 172.21.15.16:0/340462729 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc94815da10 0x7fc9480d1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:33.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.848+0000 7fc945d9b700 1 --2- 172.21.15.16:0/340462729 >> v2:172.21.15.49:3300/0 conn(0x7fc9480d4060 0x7fc9480cf2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:33.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.848+0000 7fc94559a700 1 -- 172.21.15.16:0/340462729 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2320517623 0 0) 0x7fc9480d4540 con 0x7fc9480d1d20 2023-10-16T12:26:33.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.848+0000 7fc94559a700 1 -- 172.21.15.16:0/340462729 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9300035b0 con 0x7fc9480d1d20 2023-10-16T12:26:33.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.848+0000 7fc945d9b700 1 -- 172.21.15.16:0/340462729 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc94815da10 msgr2=0x7fc9480d1520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:33.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.848+0000 7fc945d9b700 1 --2- 172.21.15.16:0/340462729 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc94815da10 0x7fc9480d1520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:33.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.848+0000 7fc945d9b700 1 -- 172.21.15.16:0/340462729 >> v1:172.21.15.16:6789/0 conn(0x7fc9480d1d20 legacy=0x7fc9480d2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:33.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.848+0000 7fc945d9b700 1 -- 172.21.15.16:0/340462729 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9480ceda0 con 0x7fc9480d4060 2023-10-16T12:26:33.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.848+0000 7fc94659c700 1 --2- 172.21.15.16:0/340462729 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc94815da10 0x7fc9480d1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:26:33.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.848+0000 7fc945d9b700 1 --2- 172.21.15.16:0/340462729 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc9480d4060 0x7fc9480cf2a0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fc93c009840 tx=0x7fc93c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0f78eea95b4b61f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:33.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.849+0000 7fc94559a700 1 -- 172.21.15.16:0/340462729 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc93c011830 con 0x7fc9480d4060 2023-10-16T12:26:33.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.849+0000 7fc94559a700 1 -- 172.21.15.16:0/340462729 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc93c011990 con 0x7fc9480d4060 2023-10-16T12:26:33.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.849+0000 7fc94559a700 1 -- 172.21.15.16:0/340462729 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc93c011d20 con 0x7fc9480d4060 2023-10-16T12:26:33.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.849+0000 7fc94ca7a700 1 -- 172.21.15.16:0/340462729 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc9480d4060 msgr2=0x7fc9480cf2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:33.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.849+0000 7fc94ca7a700 1 --2- 172.21.15.16:0/340462729 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc9480d4060 0x7fc9480cf2a0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fc93c009840 tx=0x7fc93c009b50 comp rx=0 tx=0).stop 2023-10-16T12:26:33.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.849+0000 7fc94ca7a700 1 -- 172.21.15.16:0/340462729 shutdown_connections 2023-10-16T12:26:33.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.849+0000 7fc94ca7a700 1 --2- 172.21.15.16:0/340462729 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc94815da10 0x7fc9480d1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:33.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.849+0000 7fc94ca7a700 1 --2- 172.21.15.16:0/340462729 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc9480d4060 0x7fc9480cf2a0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:33.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.849+0000 7fc94ca7a700 1 -- 172.21.15.16:0/340462729 >> 172.21.15.16:0/340462729 conn(0x7fc94809c9f0 msgr2=0x7fc9480d3740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:33.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.849+0000 7fc94ca7a700 1 -- 172.21.15.16:0/340462729 shutdown_connections 2023-10-16T12:26:33.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.849+0000 7fc94ca7a700 1 -- 172.21.15.16:0/340462729 wait complete. 2023-10-16T12:26:33.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.851+0000 7fc94ca7a700 1 Processor -- start 2023-10-16T12:26:33.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.851+0000 7fc94ca7a700 1 -- start start 2023-10-16T12:26:33.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.851+0000 7fc94ca7a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9480d1d20 0x7fc9480cb4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:33.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.851+0000 7fc94ca7a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc9480d4060 0x7fc9480b3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:33.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.851+0000 7fc94ca7a700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc94815da10 0x7fc94815b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:33.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.851+0000 7fc94ca7a700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9480ca810 con 0x7fc9480d1d20 2023-10-16T12:26:33.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.851+0000 7fc94ca7a700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9480cb0b0 con 0x7fc94815da10 2023-10-16T12:26:33.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.851+0000 7fc94ca7a700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9480b4a90 con 0x7fc9480d4060 2023-10-16T12:26:33.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.852+0000 7fc94659c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9480d1d20 0x7fc9480cb4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:33.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.852+0000 7fc94659c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9480d1d20 0x7fc9480cb4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39642/0 (socket says 172.21.15.16:39642) 2023-10-16T12:26:33.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.852+0000 7fc94659c700 1 -- 172.21.15.16:0/2713170943 learned_addr learned my addr 172.21.15.16:0/2713170943 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:33.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.852+0000 7fc945d9b700 1 --2- 172.21.15.16:0/2713170943 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc9480d4060 0x7fc9480b3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:33.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.852+0000 7fc946d9d700 1 --2- 172.21.15.16:0/2713170943 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc94815da10 0x7fc94815b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:33.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.852+0000 7fc94659c700 1 -- 172.21.15.16:0/2713170943 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc9480d4060 msgr2=0x7fc9480b3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:33.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.852+0000 7fc94659c700 1 --2- 172.21.15.16:0/2713170943 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc9480d4060 0x7fc9480b3ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:33.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.852+0000 7fc94659c700 1 -- 172.21.15.16:0/2713170943 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc94815da10 msgr2=0x7fc94815b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:33.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.852+0000 7fc94659c700 1 --2- 172.21.15.16:0/2713170943 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc94815da10 0x7fc94815b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:33.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.852+0000 7fc94659c700 1 -- 172.21.15.16:0/2713170943 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc938009430 con 0x7fc9480d1d20 2023-10-16T12:26:33.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.852+0000 7fc94659c700 1 --2- 172.21.15.16:0/2713170943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9480d1d20 0x7fc9480cb4d0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fc93800ec10 tx=0x7fc93800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:33.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.852+0000 7fc936ffd700 1 -- 172.21.15.16:0/2713170943 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc938009740 con 0x7fc9480d1d20 2023-10-16T12:26:33.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.852+0000 7fc936ffd700 1 -- 172.21.15.16:0/2713170943 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc938005a20 con 0x7fc9480d1d20 2023-10-16T12:26:33.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.852+0000 7fc94ca7a700 1 -- 172.21.15.16:0/2713170943 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc93c0094a0 con 0x7fc9480d1d20 2023-10-16T12:26:33.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.852+0000 7fc936ffd700 1 -- 172.21.15.16:0/2713170943 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc938005d90 con 0x7fc9480d1d20 2023-10-16T12:26:33.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.852+0000 7fc94ca7a700 1 -- 172.21.15.16:0/2713170943 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9480ca9f0 con 0x7fc9480d1d20 2023-10-16T12:26:33.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.854+0000 7fc936ffd700 1 -- 172.21.15.16:0/2713170943 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 32) v1 ==== 89348+0+0 (secure 0 0 0) 0x7fc93800c850 con 0x7fc9480d1d20 2023-10-16T12:26:33.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.854+0000 7fc94ca7a700 1 -- 172.21.15.16:0/2713170943 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc94804ebb0 con 0x7fc9480d1d20 2023-10-16T12:26:33.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.854+0000 7fc936ffd700 1 --2- 172.21.15.16:0/2713170943 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7fc92806b070 0x7fc92806d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:33.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.854+0000 7fc945d9b700 1 -- 172.21.15.16:0/2713170943 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7fc92806b070 msgr2=0x7fc92806d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3415303009 2023-10-16T12:26:33.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.854+0000 7fc936ffd700 1 -- 172.21.15.16:0/2713170943 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fc9380899e0 con 0x7fc9480d1d20 2023-10-16T12:26:33.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.854+0000 7fc945d9b700 1 --2- 172.21.15.16:0/2713170943 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7fc92806b070 0x7fc92806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:26:33.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:33.863+0000 7fc936ffd700 1 -- 172.21.15.16:0/2713170943 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc9380548b0 con 0x7fc9480d1d20 2023-10-16T12:26:33.868 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:33.865+0000 7fbdb19b2200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T12:26:34.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.055+0000 7fc945d9b700 1 -- 172.21.15.16:0/2713170943 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7fc92806b070 msgr2=0x7fc92806d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3415303009 2023-10-16T12:26:34.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.055+0000 7fc945d9b700 1 --2- 172.21.15.16:0/2713170943 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7fc92806b070 0x7fc92806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:26:34.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.070+0000 7fc94ca7a700 1 -- 172.21.15.16:0/2713170943 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fc9480d1b00 con 0x7fc9480d1d20 2023-10-16T12:26:34.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.073+0000 7fc936ffd700 1 -- 172.21.15.16:0/2713170943 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v24)=0 v24) v1 ==== 133+0+0 (secure 0 0 0) 0x7fc938020070 con 0x7fc9480d1d20 2023-10-16T12:26:34.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.076+0000 7fc94ca7a700 1 -- 172.21.15.16:0/2713170943 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7fc92806b070 msgr2=0x7fc92806d530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:26:34.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.076+0000 7fc94ca7a700 1 --2- 172.21.15.16:0/2713170943 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7fc92806b070 0x7fc92806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:34.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.076+0000 7fc94ca7a700 1 -- 172.21.15.16:0/2713170943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9480d1d20 msgr2=0x7fc9480cb4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:34.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.076+0000 7fc94ca7a700 1 --2- 172.21.15.16:0/2713170943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9480d1d20 0x7fc9480cb4d0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fc93800ec10 tx=0x7fc93800ef20 comp rx=0 tx=0).stop 2023-10-16T12:26:34.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.076+0000 7fc94ca7a700 1 -- 172.21.15.16:0/2713170943 shutdown_connections 2023-10-16T12:26:34.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.076+0000 7fc94ca7a700 1 --2- 172.21.15.16:0/2713170943 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7fc92806b070 0x7fc92806d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:34.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.076+0000 7fc94ca7a700 1 --2- 172.21.15.16:0/2713170943 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc94815da10 0x7fc94815b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:34.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.076+0000 7fc94ca7a700 1 --2- 172.21.15.16:0/2713170943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9480d1d20 0x7fc9480cb4d0 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:34.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.076+0000 7fc94ca7a700 1 --2- 172.21.15.16:0/2713170943 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc9480d4060 0x7fc9480b3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:34.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.076+0000 7fc94ca7a700 1 -- 172.21.15.16:0/2713170943 >> 172.21.15.16:0/2713170943 conn(0x7fc94809c9f0 msgr2=0x7fc94815fe30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:34.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.076+0000 7fc94ca7a700 1 -- 172.21.15.16:0/2713170943 shutdown_connections 2023-10-16T12:26:34.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.076+0000 7fc94ca7a700 1 -- 172.21.15.16:0/2713170943 wait complete. 2023-10-16T12:26:34.091 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:26:34.092 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-10-16T12:26:34.112 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list snap_test_dir1 2023-10-16T12:26:34.192 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:26:34.190+0000 7fd9b8dab200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T12:26:34.219 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:34.216+0000 7fbdb19b2200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T12:26:34.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.277+0000 7f9e45145700 1 Processor -- start 2023-10-16T12:26:34.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.277+0000 7f9e45145700 1 -- start start 2023-10-16T12:26:34.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.278+0000 7f9e45145700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f9e4015f060 0x7f9e400cc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:34.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.278+0000 7f9e45145700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e400b2440 con 0x7f9e400ce5d0 2023-10-16T12:26:34.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.278+0000 7f9e45145700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e400d00e0 con 0x7f9e4015ec80 2023-10-16T12:26:34.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.278+0000 7f9e45145700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e400b2c50 con 0x7f9e4015f060 2023-10-16T12:26:34.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.278+0000 7f9e3e59c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f9e400ce5d0 0x7f9e400cc190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57204/0 (socket says 172.21.15.16:57204) 2023-10-16T12:26:34.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.278+0000 7f9e3e59c700 1 -- 172.21.15.16:0/4245341142 learned_addr learned my addr 172.21.15.16:0/4245341142 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:26:34.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.279+0000 7f9e3f59e700 1 --2- 172.21.15.16:0/4245341142 >> v2:172.21.15.49:3300/0 conn(0x7f9e4015f060 0x7f9e400cc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:34.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.279+0000 7f9e3dd9b700 1 -- 172.21.15.16:0/4245341142 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2923892197 0 0) 0x7f9e400b2440 con 0x7f9e400ce5d0 2023-10-16T12:26:34.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.279+0000 7f9e3dd9b700 1 -- 172.21.15.16:0/4245341142 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e2c0035b0 con 0x7f9e400ce5d0 2023-10-16T12:26:34.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.279+0000 7f9e3dd9b700 1 -- 172.21.15.16:0/4245341142 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 731861089 0 0) 0x7f9e2c0035b0 con 0x7f9e400ce5d0 2023-10-16T12:26:34.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.279+0000 7f9e3dd9b700 1 -- 172.21.15.16:0/4245341142 >> v2:172.21.15.49:3300/0 conn(0x7f9e4015f060 msgr2=0x7f9e400cc8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:34.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.279+0000 7f9e3dd9b700 1 --2- 172.21.15.16:0/4245341142 >> v2:172.21.15.49:3300/0 conn(0x7f9e4015f060 0x7f9e400cc8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:34.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.279+0000 7f9e3dd9b700 1 -- 172.21.15.16:0/4245341142 >> v1:172.21.15.49:6789/0 conn(0x7f9e4015ec80 legacy=0x7f9e400cdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:34.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.279+0000 7f9e3dd9b700 1 -- 172.21.15.16:0/4245341142 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e400cef60 con 0x7f9e400ce5d0 2023-10-16T12:26:34.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.279+0000 7f9e3dd9b700 1 -- 172.21.15.16:0/4245341142 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f9e28001670 con 0x7f9e400ce5d0 2023-10-16T12:26:34.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.279+0000 7f9e3dd9b700 1 -- 172.21.15.16:0/4245341142 >> v1:172.21.15.16:6789/0 conn(0x7f9e400ce5d0 legacy=0x7f9e400cc190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:34.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.279+0000 7f9e3dd9b700 1 --2- 172.21.15.16:0/4245341142 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9e2c003a30 0x7f9e2c003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:34.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.279+0000 7f9e3dd9b700 1 --2- 172.21.15.16:0/4245341142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e2c004850 0x7f9e2c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:34.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.279+0000 7f9e3dd9b700 1 --2- 172.21.15.16:0/4245341142 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9e2c005210 0x7f9e2c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:34.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.279+0000 7f9e3dd9b700 1 -- 172.21.15.16:0/4245341142 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e400b2c50 con 0x7f9e2c004850 2023-10-16T12:26:34.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.279+0000 7f9e3dd9b700 1 -- 172.21.15.16:0/4245341142 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e2c00c0b0 con 0x7f9e2c003a30 2023-10-16T12:26:34.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.279+0000 7f9e3dd9b700 1 -- 172.21.15.16:0/4245341142 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e2c00c270 con 0x7f9e2c005210 2023-10-16T12:26:34.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.279+0000 7f9e3f59e700 1 --2- 172.21.15.16:0/4245341142 >> v2:172.21.15.49:3300/0 conn(0x7f9e4015f060 0x7f9e400cc8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:26:34.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.279+0000 7f9e3e59c700 1 --2- 172.21.15.16:0/4245341142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e2c004850 0x7f9e2c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:34.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.279+0000 7f9e3ed9d700 1 --2- 172.21.15.16:0/4245341142 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9e2c003a30 0x7f9e2c003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:34.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.279+0000 7f9e3f59e700 1 --2- 172.21.15.16:0/4245341142 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9e2c005210 0x7f9e2c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:34.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.279+0000 7f9e3e59c700 1 -- 172.21.15.16:0/4245341142 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9e2c005210 msgr2=0x7f9e2c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:34.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.279+0000 7f9e3e59c700 1 --2- 172.21.15.16:0/4245341142 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9e2c005210 0x7f9e2c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:34.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.279+0000 7f9e3e59c700 1 -- 172.21.15.16:0/4245341142 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9e2c003a30 msgr2=0x7f9e2c003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:34.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.279+0000 7f9e3e59c700 1 --2- 172.21.15.16:0/4245341142 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9e2c003a30 0x7f9e2c003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:34.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.279+0000 7f9e3e59c700 1 -- 172.21.15.16:0/4245341142 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e2c0035b0 con 0x7f9e2c004850 2023-10-16T12:26:34.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.280+0000 7f9e3f59e700 1 --2- 172.21.15.16:0/4245341142 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9e2c005210 0x7f9e2c00ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:26:34.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.280+0000 7f9e3f59e700 1 -- 172.21.15.16:0/4245341142 reap_dead start 2023-10-16T12:26:34.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.280+0000 7f9e3e59c700 1 --2- 172.21.15.16:0/4245341142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e2c004850 0x7f9e2c004cd0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f9e2800c240 tx=0x7f9e2c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ebe857de162719c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:34.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.280+0000 7f9e3dd9b700 1 -- 172.21.15.16:0/4245341142 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e2800cd80 con 0x7f9e2c004850 2023-10-16T12:26:34.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.280+0000 7f9e3dd9b700 1 -- 172.21.15.16:0/4245341142 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e28007ba0 con 0x7f9e2c004850 2023-10-16T12:26:34.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.280+0000 7f9e3dd9b700 1 -- 172.21.15.16:0/4245341142 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e28005ba0 con 0x7f9e2c004850 2023-10-16T12:26:34.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.280+0000 7f9e45145700 1 -- 172.21.15.16:0/4245341142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e2c004850 msgr2=0x7f9e2c004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:34.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.280+0000 7f9e45145700 1 --2- 172.21.15.16:0/4245341142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e2c004850 0x7f9e2c004cd0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f9e2800c240 tx=0x7f9e2c005880 comp rx=0 tx=0).stop 2023-10-16T12:26:34.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.280+0000 7f9e45145700 1 -- 172.21.15.16:0/4245341142 shutdown_connections 2023-10-16T12:26:34.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.280+0000 7f9e45145700 1 --2- 172.21.15.16:0/4245341142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e2c004850 0x7f9e2c004cd0 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:34.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.280+0000 7f9e45145700 1 -- 172.21.15.16:0/4245341142 >> 172.21.15.16:0/4245341142 conn(0x7f9e400d15f0 msgr2=0x7f9e400d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:34.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.281+0000 7f9e45145700 1 -- 172.21.15.16:0/4245341142 shutdown_connections 2023-10-16T12:26:34.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.281+0000 7f9e45145700 1 -- 172.21.15.16:0/4245341142 wait complete. 2023-10-16T12:26:34.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.282+0000 7f9e45145700 1 Processor -- start 2023-10-16T12:26:34.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.282+0000 7f9e45145700 1 -- start start 2023-10-16T12:26:34.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.283+0000 7f9e45145700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9e2c004850 0x7f9e4015a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:34.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.283+0000 7f9e45145700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e4015a930 0x7f9e400c5830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:34.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.283+0000 7f9e45145700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9e400c5d70 0x7f9e40156b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:34.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.283+0000 7f9e45145700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e400c50c0 con 0x7f9e4015a930 2023-10-16T12:26:34.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.283+0000 7f9e45145700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e400c53a0 con 0x7f9e2c004850 2023-10-16T12:26:34.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.283+0000 7f9e45145700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e40154ec0 con 0x7f9e400c5d70 2023-10-16T12:26:34.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.283+0000 7f9e3e59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e4015a930 0x7f9e400c5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:34.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.283+0000 7f9e3e59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e4015a930 0x7f9e400c5830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39708/0 (socket says 172.21.15.16:39708) 2023-10-16T12:26:34.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.283+0000 7f9e3e59c700 1 -- 172.21.15.16:0/1508299655 learned_addr learned my addr 172.21.15.16:0/1508299655 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:34.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.283+0000 7f9e3ed9d700 1 --2- 172.21.15.16:0/1508299655 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9e2c004850 0x7f9e4015a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:34.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.283+0000 7f9e3f59e700 1 --2- 172.21.15.16:0/1508299655 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9e400c5d70 0x7f9e40156b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:34.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.283+0000 7f9e3e59c700 1 -- 172.21.15.16:0/1508299655 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9e400c5d70 msgr2=0x7f9e40156b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:34.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.283+0000 7f9e3e59c700 1 --2- 172.21.15.16:0/1508299655 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9e400c5d70 0x7f9e40156b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:34.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.283+0000 7f9e3e59c700 1 -- 172.21.15.16:0/1508299655 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9e2c004850 msgr2=0x7f9e4015a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:34.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.283+0000 7f9e3e59c700 1 --2- 172.21.15.16:0/1508299655 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9e2c004850 0x7f9e4015a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:34.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.283+0000 7f9e3e59c700 1 -- 172.21.15.16:0/1508299655 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e401551a0 con 0x7f9e4015a930 2023-10-16T12:26:34.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.284+0000 7f9e3e59c700 1 --2- 172.21.15.16:0/1508299655 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e4015a930 0x7f9e400c5830 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f9e2800d000 tx=0x7f9e400d1030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:34.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.284+0000 7f9e377fe700 1 -- 172.21.15.16:0/1508299655 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e28004d40 con 0x7f9e4015a930 2023-10-16T12:26:34.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.284+0000 7f9e377fe700 1 -- 172.21.15.16:0/1508299655 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e280070d0 con 0x7f9e4015a930 2023-10-16T12:26:34.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.284+0000 7f9e45145700 1 -- 172.21.15.16:0/1508299655 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e40155480 con 0x7f9e4015a930 2023-10-16T12:26:34.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.284+0000 7f9e377fe700 1 -- 172.21.15.16:0/1508299655 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e28004ea0 con 0x7f9e4015a930 2023-10-16T12:26:34.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.284+0000 7f9e45145700 1 -- 172.21.15.16:0/1508299655 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e40154c30 con 0x7f9e4015a930 2023-10-16T12:26:34.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.285+0000 7f9e377fe700 1 -- 172.21.15.16:0/1508299655 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 32) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f9e28007480 con 0x7f9e4015a930 2023-10-16T12:26:34.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.285+0000 7f9e45145700 1 -- 172.21.15.16:0/1508299655 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e400c5530 con 0x7f9e4015a930 2023-10-16T12:26:34.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.286+0000 7f9e377fe700 1 --2- 172.21.15.16:0/1508299655 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f9e2006b070 0x7f9e2006d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:34.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.286+0000 7f9e3ed9d700 1 -- 172.21.15.16:0/1508299655 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f9e2006b070 msgr2=0x7f9e2006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3415303009 2023-10-16T12:26:34.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.286+0000 7f9e377fe700 1 -- 172.21.15.16:0/1508299655 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f9e2808a1c0 con 0x7f9e4015a930 2023-10-16T12:26:34.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.286+0000 7f9e3ed9d700 1 --2- 172.21.15.16:0/1508299655 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f9e2006b070 0x7f9e2006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:26:34.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.290+0000 7f9e377fe700 1 -- 172.21.15.16:0/1508299655 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9e28055090 con 0x7f9e4015a930 2023-10-16T12:26:34.312 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:26:34.310+0000 7fd9b8dab200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T12:26:34.354 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:34.351+0000 7fbdb19b2200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T12:26:34.390 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:26:34.388+0000 7fd9b8dab200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T12:26:34.432 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:34.429+0000 7fbdb19b2200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T12:26:34.474 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:26:34.472+0000 7fd9b8dab200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T12:26:34.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.487+0000 7f9e3ed9d700 1 -- 172.21.15.16:0/1508299655 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f9e2006b070 msgr2=0x7f9e2006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3415303009 2023-10-16T12:26:34.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.487+0000 7f9e3ed9d700 1 --2- 172.21.15.16:0/1508299655 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f9e2006b070 0x7f9e2006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:26:34.516 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:34.513+0000 7fbdb19b2200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T12:26:34.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.534+0000 7f9e45145700 1 -- 172.21.15.16:0/1508299655 --> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f9e400b2970 con 0x7f9e2006b070 2023-10-16T12:26:34.861 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:26:34.859+0000 7fd9b8dab200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T12:26:34.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.888+0000 7f9e3ed9d700 1 -- 172.21.15.16:0/1508299655 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f9e2006b070 msgr2=0x7f9e2006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3415303009 2023-10-16T12:26:34.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:34.888+0000 7f9e3ed9d700 1 --2- 172.21.15.16:0/1508299655 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f9e2006b070 0x7f9e2006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-16T12:26:34.903 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:34.900+0000 7fbdb19b2200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T12:26:35.467 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:26:35.464+0000 7fd9b8dab200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T12:26:35.510 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:35.507+0000 7fbdb19b2200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T12:26:35.539 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:26:35.537+0000 7fd9b8dab200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T12:26:35.583 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:35.580+0000 7fbdb19b2200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T12:26:35.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:35.689+0000 7f9e3ed9d700 1 -- 172.21.15.16:0/1508299655 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f9e2006b070 msgr2=0x7f9e2006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3415303009 2023-10-16T12:26:35.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:35.689+0000 7f9e3ed9d700 1 --2- 172.21.15.16:0/1508299655 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f9e2006b070 0x7f9e2006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-16T12:26:36.200 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:26:36.198+0000 7fd9b8dab200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:26:36.247 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:36.245+0000 7fbdb19b2200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:26:36.282 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:26:36.280+0000 7fd9b8dab200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T12:26:36.329 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:36.327+0000 7fbdb19b2200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T12:26:36.432 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:26:36.430+0000 7fd9b8dab200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:26:36.479 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:36.477+0000 7fbdb19b2200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:26:36.667 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:26:36.665+0000 7fd9b8dab200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T12:26:36.718 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:36.716+0000 7fbdb19b2200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T12:26:36.744 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:26:36.741+0000 7fd9b8dab200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T12:26:36.795 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:36.793+0000 7fbdb19b2200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T12:26:37.017 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:26:37.015+0000 7fd9b8dab200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T12:26:37.071 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:37.070+0000 7fbdb19b2200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T12:26:37.246 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:26:37.244+0000 7fd9b8dab200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T12:26:37.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:37.291+0000 7f9e3ed9d700 1 -- 172.21.15.16:0/1508299655 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f9e2006b070 msgr2=0x7f9e2006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3415303009 2023-10-16T12:26:37.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:37.291+0000 7f9e3ed9d700 1 --2- 172.21.15.16:0/1508299655 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f9e2006b070 0x7f9e2006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-10-16T12:26:37.303 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:37.301+0000 7fbdb19b2200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T12:26:37.400 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:26:37.398+0000 7fd9b8dab200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T12:26:37.459 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:37.457+0000 7fbdb19b2200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T12:26:37.521 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:26:37.517+0000 7fd9b8dab200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T12:26:37.579 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:37.577+0000 7fbdb19b2200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T12:26:37.679 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:26:37.677+0000 7fd9b8dab200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T12:26:37.741 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:37.739+0000 7fbdb19b2200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T12:26:37.950 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:26:37.948+0000 7fd9b8dab200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T12:26:38.014 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:38.012+0000 7fbdb19b2200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T12:26:38.115 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:26:38.113+0000 7fd9b8dab200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T12:26:38.182 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:38.180+0000 7fbdb19b2200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T12:26:38.665 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:26:38.663+0000 7fd9b8dab200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T12:26:38.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:38.734+0000 7f9e377fe700 1 -- 172.21.15.16:0/1508299655 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 33) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f9e28007730 con 0x7f9e4015a930 2023-10-16T12:26:38.738 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:38.736+0000 7fbdb19b2200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T12:26:38.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:38.755+0000 7f9e377fe700 1 -- 172.21.15.16:0/1508299655 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mgrmap(e 34) v1 ==== 88957+0+0 (secure 0 0 0) 0x7f9e28055460 con 0x7f9e4015a930 2023-10-16T12:26:38.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:38.756+0000 7f9e377fe700 1 -- 172.21.15.16:0/1508299655 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f9e2006b070 msgr2=0x7f9e2006d530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:26:38.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:38.756+0000 7f9e377fe700 1 --2- 172.21.15.16:0/1508299655 >> [v2:172.21.15.16:6832/3415303009,v1:172.21.15.16:6833/3415303009] conn(0x7f9e2006b070 0x7f9e2006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:38.901 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:38.900+0000 7fbd2498d700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:26:38.902 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:38.900+0000 7fbd2498d700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:26:38.902 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:38.900+0000 7fbd2498d700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:26:38.902 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:38.900+0000 7fbd2498d700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:26:38.903 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:38.900+0000 7fbd2498d700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:26:38.903 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:38.900+0000 7fbd27993700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:26:38.904 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:38.900+0000 7fbd27993700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:26:38.904 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:38.900+0000 7fbd27993700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:26:38.904 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:38.900+0000 7fbd27993700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:26:38.904 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:38.900+0000 7fbd27993700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:26:39.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:39.759+0000 7f9e377fe700 1 -- 172.21.15.16:0/1508299655 <== mon.0 v2:172.21.15.16:3300/0 9 ==== mgrmap(e 35) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f9e280711a0 con 0x7f9e4015a930 2023-10-16T12:26:39.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:39.759+0000 7f9e377fe700 1 --2- 172.21.15.16:0/1508299655 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f9e2001d250 0x7f9e200754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:39.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:39.759+0000 7f9e377fe700 1 -- 172.21.15.16:0/1508299655 --> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f9e2001d630 con 0x7f9e2001d250 2023-10-16T12:26:39.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:39.760+0000 7f9e3ed9d700 1 --2- 172.21.15.16:0/1508299655 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f9e2001d250 0x7f9e200754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:39.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:39.762+0000 7f9e3ed9d700 1 --2- 172.21.15.16:0/1508299655 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f9e2001d250 0x7f9e200754a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f9e400b74c0 tx=0x7f9e30007d50 comp rx=0 tx=0).ready entity=mgr.6813 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:39.773 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:39.771+0000 7fbd2e9e1700 -1 mgr.server reply reply (2) No such file or directory SnapSchedule for snap_test_dir1 not found 2023-10-16T12:26:39.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:39.771+0000 7f9e377fe700 1 -- 172.21.15.16:0/1508299655 <== mgr.6813 v2:172.21.15.16:6832/2230262559 1 ==== mgr_command_reply(tid 0: -2 SnapSchedule for snap_test_dir1 not found) v1 ==== 49+0+0 (secure 0 0 0) 0x7f9e2001d630 con 0x7f9e2001d250 2023-10-16T12:26:39.775 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENOENT: SnapSchedule for snap_test_dir1 not found 2023-10-16T12:26:39.779 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-10-16T12:26:39.780 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:26:39.780 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2023-10-16T12:26:39.801 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-16T12:26:39.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:39.971+0000 7f8fb31c3700 1 Processor -- start 2023-10-16T12:26:39.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:39.972+0000 7f8fb31c3700 1 -- start start 2023-10-16T12:26:39.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:39.972+0000 7f8fb31c3700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f8fac15b2c0 0x7f8fac161340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:39.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:39.972+0000 7f8fb31c3700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8fac06e570 0x7f8fac15e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:39.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:39.972+0000 7f8fb31c3700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8fac06a7e0 0x7f8fac06ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:39.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:39.972+0000 7f8fb31c3700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fac06b1b0 con 0x7f8fac06a7e0 2023-10-16T12:26:39.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:39.972+0000 7f8fb31c3700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fac161880 con 0x7f8fac15b2c0 2023-10-16T12:26:39.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:39.972+0000 7f8fb31c3700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fac15b6a0 con 0x7f8fac06e570 2023-10-16T12:26:39.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:39.973+0000 7f8fb1760700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8fac06a7e0 0x7f8fac06ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:39.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:39.973+0000 7f8fb1760700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8fac06a7e0 0x7f8fac06ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39906/0 (socket says 172.21.15.16:39906) 2023-10-16T12:26:39.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:39.973+0000 7f8fb1760700 1 -- 172.21.15.16:0/1362975581 learned_addr learned my addr 172.21.15.16:0/1362975581 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:39.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:39.973+0000 7f8fb0f5f700 1 --2- 172.21.15.16:0/1362975581 >> v2:172.21.15.49:3300/0 conn(0x7f8fac15b2c0 0x7f8fac161340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:39.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:39.973+0000 7f8fabfff700 1 --2- 172.21.15.16:0/1362975581 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8fac06e570 0x7f8fac15e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:39.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:39.973+0000 7f8fb1760700 1 -- 172.21.15.16:0/1362975581 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8fac06e570 msgr2=0x7f8fac15e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:39.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:39.973+0000 7f8fb1760700 1 --2- 172.21.15.16:0/1362975581 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8fac06e570 0x7f8fac15e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:39.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:39.973+0000 7f8fb1760700 1 -- 172.21.15.16:0/1362975581 >> v2:172.21.15.49:3300/0 conn(0x7f8fac15b2c0 msgr2=0x7f8fac161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:39.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:39.973+0000 7f8fb1760700 1 --2- 172.21.15.16:0/1362975581 >> v2:172.21.15.49:3300/0 conn(0x7f8fac15b2c0 0x7f8fac161340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:39.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:39.973+0000 7f8fb1760700 1 -- 172.21.15.16:0/1362975581 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fac161b20 con 0x7f8fac06a7e0 2023-10-16T12:26:39.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:39.973+0000 7f8fabfff700 1 --2- 172.21.15.16:0/1362975581 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8fac06e570 0x7f8fac15e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:26:39.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:39.973+0000 7f8fb1760700 1 --2- 172.21.15.16:0/1362975581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fac06a7e0 0x7f8fac06ac70 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f8fac06b4a0 tx=0x7f8fa400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8530016d909dc650 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:39.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:39.973+0000 7f8fab7fe700 1 -- 172.21.15.16:0/1362975581 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fa400e070 con 0x7f8fac06a7e0 2023-10-16T12:26:39.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:39.973+0000 7f8fab7fe700 1 -- 172.21.15.16:0/1362975581 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fa400b9c0 con 0x7f8fac06a7e0 2023-10-16T12:26:39.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:39.973+0000 7f8fab7fe700 1 -- 172.21.15.16:0/1362975581 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fa400bd70 con 0x7f8fac06a7e0 2023-10-16T12:26:39.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:39.974+0000 7f8fb31c3700 1 -- 172.21.15.16:0/1362975581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fac06a7e0 msgr2=0x7f8fac06ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:39.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:39.974+0000 7f8fb31c3700 1 --2- 172.21.15.16:0/1362975581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fac06a7e0 0x7f8fac06ac70 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f8fac06b4a0 tx=0x7f8fa400b330 comp rx=0 tx=0).stop 2023-10-16T12:26:39.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:39.974+0000 7f8fb31c3700 1 -- 172.21.15.16:0/1362975581 shutdown_connections 2023-10-16T12:26:39.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:39.974+0000 7f8fb31c3700 1 --2- 172.21.15.16:0/1362975581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fac06a7e0 0x7f8fac06ac70 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:39.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:39.974+0000 7f8fb31c3700 1 --2- 172.21.15.16:0/1362975581 >> v2:172.21.15.49:3300/0 conn(0x7f8fac15b2c0 0x7f8fac161340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:39.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:39.974+0000 7f8fb31c3700 1 --2- 172.21.15.16:0/1362975581 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8fac06e570 0x7f8fac15e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:39.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:39.974+0000 7f8fb31c3700 1 -- 172.21.15.16:0/1362975581 >> 172.21.15.16:0/1362975581 conn(0x7f8fac0b1f40 msgr2=0x7f8fac0b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:39.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:39.974+0000 7f8fb31c3700 1 -- 172.21.15.16:0/1362975581 shutdown_connections 2023-10-16T12:26:39.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:39.974+0000 7f8fb31c3700 1 -- 172.21.15.16:0/1362975581 wait complete. 2023-10-16T12:26:39.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:39.975+0000 7f8fb31c3700 1 Processor -- start 2023-10-16T12:26:39.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:39.975+0000 7f8fb31c3700 1 -- start start 2023-10-16T12:26:39.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:39.976+0000 7f8fb31c3700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8fac06a7e0 0x7f8fac0c74c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:39.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:39.976+0000 7f8fb31c3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fac06e570 0x7f8fac1f09e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:39.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:39.976+0000 7f8fb31c3700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8fac15b2c0 0x7f8fac0b3730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:39.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:39.976+0000 7f8fb31c3700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fac0ca660 con 0x7f8fac06e570 2023-10-16T12:26:39.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:39.976+0000 7f8fb31c3700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fac1f0f20 con 0x7f8fac06a7e0 2023-10-16T12:26:39.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:39.976+0000 7f8fb31c3700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fac0b3c70 con 0x7f8fac15b2c0 2023-10-16T12:26:39.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:39.976+0000 7f8fabfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fac06e570 0x7f8fac1f09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:39.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:39.976+0000 7f8fabfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fac06e570 0x7f8fac1f09e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39910/0 (socket says 172.21.15.16:39910) 2023-10-16T12:26:39.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:39.976+0000 7f8fabfff700 1 -- 172.21.15.16:0/959910382 learned_addr learned my addr 172.21.15.16:0/959910382 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:39.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:39.976+0000 7f8fb0f5f700 1 --2- 172.21.15.16:0/959910382 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8fac06a7e0 0x7f8fac0c74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:39.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:39.976+0000 7f8fb1760700 1 --2- 172.21.15.16:0/959910382 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8fac15b2c0 0x7f8fac0b3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:39.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:39.976+0000 7f8fabfff700 1 -- 172.21.15.16:0/959910382 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8fac15b2c0 msgr2=0x7f8fac0b3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:39.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:39.976+0000 7f8fabfff700 1 --2- 172.21.15.16:0/959910382 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8fac15b2c0 0x7f8fac0b3730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:39.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:39.976+0000 7f8fabfff700 1 -- 172.21.15.16:0/959910382 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8fac06a7e0 msgr2=0x7f8fac0c74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:39.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:39.976+0000 7f8fabfff700 1 --2- 172.21.15.16:0/959910382 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8fac06a7e0 0x7f8fac0c74c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:39.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:39.976+0000 7f8fabfff700 1 -- 172.21.15.16:0/959910382 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fa4009b30 con 0x7f8fac06e570 2023-10-16T12:26:39.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:39.976+0000 7f8fb0f5f700 1 --2- 172.21.15.16:0/959910382 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8fac06a7e0 0x7f8fac0c74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:26:39.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:39.976+0000 7f8fb1760700 1 --2- 172.21.15.16:0/959910382 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8fac15b2c0 0x7f8fac0b3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:26:39.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:39.977+0000 7f8fabfff700 1 --2- 172.21.15.16:0/959910382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fac06e570 0x7f8fac1f09e0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f8f9800e760 tx=0x7f8f9800eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:39.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:39.977+0000 7f8fa97fa700 1 -- 172.21.15.16:0/959910382 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f98013610 con 0x7f8fac06e570 2023-10-16T12:26:39.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:39.977+0000 7f8fa97fa700 1 -- 172.21.15.16:0/959910382 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f98013770 con 0x7f8fac06e570 2023-10-16T12:26:39.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:39.977+0000 7f8fb31c3700 1 -- 172.21.15.16:0/959910382 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fac0c9cc0 con 0x7f8fac06e570 2023-10-16T12:26:39.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:39.977+0000 7f8fa97fa700 1 -- 172.21.15.16:0/959910382 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f98013b20 con 0x7f8fac06e570 2023-10-16T12:26:39.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:39.977+0000 7f8fb31c3700 1 -- 172.21.15.16:0/959910382 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fac0ca280 con 0x7f8fac06e570 2023-10-16T12:26:39.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:39.978+0000 7f8fb31c3700 1 -- 172.21.15.16:0/959910382 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fac04ebb0 con 0x7f8fac06e570 2023-10-16T12:26:39.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:39.978+0000 7f8fa97fa700 1 -- 172.21.15.16:0/959910382 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 35) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f8f98013c80 con 0x7f8fac06e570 2023-10-16T12:26:39.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:39.978+0000 7f8fa97fa700 1 --2- 172.21.15.16:0/959910382 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f8f9006ada0 0x7f8f9006d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:39.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:39.979+0000 7f8fa97fa700 1 -- 172.21.15.16:0/959910382 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f8f98089150 con 0x7f8fac06e570 2023-10-16T12:26:39.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:39.979+0000 7f8fb0f5f700 1 --2- 172.21.15.16:0/959910382 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f8f9006ada0 0x7f8f9006d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:39.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:39.979+0000 7f8fb0f5f700 1 --2- 172.21.15.16:0/959910382 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f8f9006ada0 0x7f8f9006d260 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8fa0009970 tx=0x7f8fa0008040 comp rx=0 tx=0).ready entity=mgr.6813 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:39.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:39.983+0000 7f8fa97fa700 1 -- 172.21.15.16:0/959910382 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8f98054280 con 0x7f8fac06e570 2023-10-16T12:26:40.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:40.246+0000 7f8fb31c3700 1 -- 172.21.15.16:0/959910382 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f8fac0b6370 con 0x7f8fac06e570 2023-10-16T12:26:40.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:40.761+0000 7f8fa97fa700 1 -- 172.21.15.16:0/959910382 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v36) v1 ==== 93+0+0 (secure 0 0 0) 0x7f8f980578a0 con 0x7f8fac06e570 2023-10-16T12:26:40.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:40.765+0000 7f8fb31c3700 1 -- 172.21.15.16:0/959910382 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f8f9006ada0 msgr2=0x7f8f9006d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:40.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:40.765+0000 7f8fb31c3700 1 --2- 172.21.15.16:0/959910382 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f8f9006ada0 0x7f8f9006d260 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8fa0009970 tx=0x7f8fa0008040 comp rx=0 tx=0).stop 2023-10-16T12:26:40.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:40.765+0000 7f8fb31c3700 1 -- 172.21.15.16:0/959910382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fac06e570 msgr2=0x7f8fac1f09e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:40.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:40.765+0000 7f8fb31c3700 1 --2- 172.21.15.16:0/959910382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fac06e570 0x7f8fac1f09e0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f8f9800e760 tx=0x7f8f9800eb20 comp rx=0 tx=0).stop 2023-10-16T12:26:40.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:40.767+0000 7f8fb31c3700 1 -- 172.21.15.16:0/959910382 shutdown_connections 2023-10-16T12:26:40.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:40.767+0000 7f8fb31c3700 1 --2- 172.21.15.16:0/959910382 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f8f9006ada0 0x7f8f9006d260 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:40.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:40.767+0000 7f8fb31c3700 1 --2- 172.21.15.16:0/959910382 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8fac15b2c0 0x7f8fac0b3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:40.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:40.767+0000 7f8fb31c3700 1 --2- 172.21.15.16:0/959910382 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8fac06a7e0 0x7f8fac0c74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:40.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:40.767+0000 7f8fb31c3700 1 --2- 172.21.15.16:0/959910382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fac06e570 0x7f8fac1f09e0 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:40.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:40.767+0000 7f8fb31c3700 1 -- 172.21.15.16:0/959910382 >> 172.21.15.16:0/959910382 conn(0x7f8fac0b1f40 msgr2=0x7f8fac0cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:40.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:40.767+0000 7f8fb31c3700 1 -- 172.21.15.16:0/959910382 shutdown_connections 2023-10-16T12:26:40.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:40.767+0000 7f8fb31c3700 1 -- 172.21.15.16:0/959910382 wait complete. 2023-10-16T12:26:40.785 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T12:26:40.786 DEBUG:teuthology.orchestra.run.smithi016:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T12:26:40.828 DEBUG:teuthology.orchestra.run.smithi049:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T12:26:40.870 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T12:26:40.871 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:26:40.871 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:26:40.890 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:26:40.890 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:26:40.921 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:26:40.920+0000 7f50d29c5200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T12:26:40.922 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:40.920+0000 7f11cbf9c200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T12:26:40.939 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:26:40.990 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:26:40.989+0000 7f50d29c5200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T12:26:40.994 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:40.992+0000 7f11cbf9c200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T12:26:41.065 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:26:41.064+0000 7f50d29c5200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T12:26:41.071 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:41.069+0000 7f11cbf9c200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T12:26:41.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.106+0000 7f90482b2700 1 Processor -- start 2023-10-16T12:26:41.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.106+0000 7f90482b2700 1 -- start start 2023-10-16T12:26:41.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.107+0000 7f90482b2700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f904015d900 0x7f90400d2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:41.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.107+0000 7f90482b2700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f90400ce9f0 0x7f90400cdc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:41.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.107+0000 7f90482b2700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90400ce1a0 con 0x7f90400d2860 2023-10-16T12:26:41.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.107+0000 7f90482b2700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90400cee70 con 0x7f904015d900 2023-10-16T12:26:41.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.107+0000 7f90482b2700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90400ce400 con 0x7f90400ce9f0 2023-10-16T12:26:41.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.108+0000 7f904604e700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f904015d900 0x7f90400d2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:41.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.108+0000 7f904584d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f90400ce9f0 0x7f90400cdc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:41.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.108+0000 7f904684f700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7f90400d2860 0x7f90400b5120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:36446/0 (socket says 172.21.15.16:36446) 2023-10-16T12:26:41.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.108+0000 7f904684f700 1 -- 172.21.15.16:0/1791412436 learned_addr learned my addr 172.21.15.16:0/1791412436 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:26:41.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.108+0000 7f904604e700 1 -- 172.21.15.16:0/1791412436 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f90400ce9f0 msgr2=0x7f90400cdc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:41.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.108+0000 7f904604e700 1 --2- 172.21.15.16:0/1791412436 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f90400ce9f0 0x7f90400cdc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:41.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.108+0000 7f904604e700 1 -- 172.21.15.16:0/1791412436 >> v1:172.21.15.49:6789/0 conn(0x7f90400d2860 legacy=0x7f90400b5120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:41.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.108+0000 7f904604e700 1 -- 172.21.15.16:0/1791412436 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f904015dce0 con 0x7f904015d900 2023-10-16T12:26:41.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.108+0000 7f904504c700 1 -- 172.21.15.16:0/1791412436 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 132991851 0 0) 0x7f90400ce1a0 con 0x7f90400d2860 2023-10-16T12:26:41.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.108+0000 7f904604e700 1 --2- 172.21.15.16:0/1791412436 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f904015d900 0x7f90400d2040 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f903c009840 tx=0x7f903c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7aa51bf3710974ef server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:41.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.108+0000 7f904504c700 1 -- 172.21.15.16:0/1791412436 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f903c011830 con 0x7f904015d900 2023-10-16T12:26:41.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.108+0000 7f904504c700 1 -- 172.21.15.16:0/1791412436 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f903c011990 con 0x7f904015d900 2023-10-16T12:26:41.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.108+0000 7f904504c700 1 -- 172.21.15.16:0/1791412436 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f903c011d20 con 0x7f904015d900 2023-10-16T12:26:41.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.109+0000 7f90482b2700 1 -- 172.21.15.16:0/1791412436 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f904015d900 msgr2=0x7f90400d2040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:41.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.109+0000 7f90482b2700 1 --2- 172.21.15.16:0/1791412436 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f904015d900 0x7f90400d2040 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f903c009840 tx=0x7f903c009b50 comp rx=0 tx=0).stop 2023-10-16T12:26:41.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.109+0000 7f90482b2700 1 -- 172.21.15.16:0/1791412436 shutdown_connections 2023-10-16T12:26:41.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.109+0000 7f90482b2700 1 --2- 172.21.15.16:0/1791412436 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f904015d900 0x7f90400d2040 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:41.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.109+0000 7f90482b2700 1 --2- 172.21.15.16:0/1791412436 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f90400ce9f0 0x7f90400cdc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:41.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.109+0000 7f90482b2700 1 -- 172.21.15.16:0/1791412436 >> 172.21.15.16:0/1791412436 conn(0x7f9040072330 msgr2=0x7f9040072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:41.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.109+0000 7f90482b2700 1 -- 172.21.15.16:0/1791412436 shutdown_connections 2023-10-16T12:26:41.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.109+0000 7f90482b2700 1 -- 172.21.15.16:0/1791412436 wait complete. 2023-10-16T12:26:41.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.110+0000 7f90482b2700 1 Processor -- start 2023-10-16T12:26:41.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.110+0000 7f90482b2700 1 -- start start 2023-10-16T12:26:41.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.111+0000 7f90482b2700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90400ce9f0 0x7f90401f2670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:41.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.111+0000 7f90482b2700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f90400d2860 0x7f904014f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:41.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.111+0000 7f90482b2700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f904015d900 0x7f904014fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:41.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.111+0000 7f90482b2700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90401511b0 con 0x7f90400ce9f0 2023-10-16T12:26:41.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.111+0000 7f90482b2700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90401f2bb0 con 0x7f904015d900 2023-10-16T12:26:41.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.111+0000 7f90482b2700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9040151400 con 0x7f90400d2860 2023-10-16T12:26:41.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.111+0000 7f904604e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90400ce9f0 0x7f90401f2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:41.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.111+0000 7f904604e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90400ce9f0 0x7f90401f2670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39992/0 (socket says 172.21.15.16:39992) 2023-10-16T12:26:41.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.111+0000 7f904604e700 1 -- 172.21.15.16:0/2999864960 learned_addr learned my addr 172.21.15.16:0/2999864960 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:41.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.111+0000 7f904584d700 1 --2- 172.21.15.16:0/2999864960 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f90400d2860 0x7f904014f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:41.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.111+0000 7f904684f700 1 --2- 172.21.15.16:0/2999864960 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f904015d900 0x7f904014fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:41.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.111+0000 7f904604e700 1 -- 172.21.15.16:0/2999864960 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f90400d2860 msgr2=0x7f904014f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:41.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.112+0000 7f904604e700 1 --2- 172.21.15.16:0/2999864960 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f90400d2860 0x7f904014f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:41.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.112+0000 7f904604e700 1 -- 172.21.15.16:0/2999864960 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f904015d900 msgr2=0x7f904014fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:41.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.112+0000 7f904604e700 1 --2- 172.21.15.16:0/2999864960 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f904015d900 0x7f904014fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:41.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.112+0000 7f904604e700 1 -- 172.21.15.16:0/2999864960 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f903c0094a0 con 0x7f90400ce9f0 2023-10-16T12:26:41.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.112+0000 7f904604e700 1 --2- 172.21.15.16:0/2999864960 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90400ce9f0 0x7f90401f2670 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f903c014040 tx=0x7f903c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:41.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.112+0000 7f9036ffd700 1 -- 172.21.15.16:0/2999864960 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f903c0036a0 con 0x7f90400ce9f0 2023-10-16T12:26:41.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.112+0000 7f9036ffd700 1 -- 172.21.15.16:0/2999864960 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f903c01e070 con 0x7f90400ce9f0 2023-10-16T12:26:41.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.112+0000 7f90482b2700 1 -- 172.21.15.16:0/2999864960 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90401516e0 con 0x7f90400ce9f0 2023-10-16T12:26:41.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.112+0000 7f9036ffd700 1 -- 172.21.15.16:0/2999864960 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f903c003a40 con 0x7f90400ce9f0 2023-10-16T12:26:41.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.112+0000 7f90482b2700 1 -- 172.21.15.16:0/2999864960 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9040151ca0 con 0x7f90400ce9f0 2023-10-16T12:26:41.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.113+0000 7f90482b2700 1 -- 172.21.15.16:0/2999864960 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f904004ebb0 con 0x7f90400ce9f0 2023-10-16T12:26:41.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.114+0000 7f9036ffd700 1 -- 172.21.15.16:0/2999864960 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 36) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f903c003ba0 con 0x7f90400ce9f0 2023-10-16T12:26:41.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.114+0000 7f9036ffd700 1 --2- 172.21.15.16:0/2999864960 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f902406ad50 0x7f902406d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:41.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.114+0000 7f904584d700 1 -- 172.21.15.16:0/2999864960 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f902406ad50 msgr2=0x7f902406d210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2230262559 2023-10-16T12:26:41.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.114+0000 7f9036ffd700 1 -- 172.21.15.16:0/2999864960 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f903c08a830 con 0x7f90400ce9f0 2023-10-16T12:26:41.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.114+0000 7f904584d700 1 --2- 172.21.15.16:0/2999864960 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f902406ad50 0x7f902406d210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:26:41.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.118+0000 7f9036ffd700 1 -- 172.21.15.16:0/2999864960 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f903c055960 con 0x7f90400ce9f0 2023-10-16T12:26:41.146 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:26:41.145+0000 7f50d29c5200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T12:26:41.184 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:41.155+0000 7f11cbf9c200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T12:26:41.295 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:26:41.294+0000 7f50d29c5200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T12:26:41.306 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:41.305+0000 7f11cbf9c200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T12:26:41.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.315+0000 7f904584d700 1 -- 172.21.15.16:0/2999864960 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f902406ad50 msgr2=0x7f902406d210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2230262559 2023-10-16T12:26:41.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.315+0000 7f904584d700 1 --2- 172.21.15.16:0/2999864960 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f902406ad50 0x7f902406d210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:26:41.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.322+0000 7f90482b2700 1 -- 172.21.15.16:0/2999864960 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f90400c7140 con 0x7f90400ce9f0 2023-10-16T12:26:41.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.323+0000 7f9036ffd700 1 -- 172.21.15.16:0/2999864960 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+17151 (secure 0 0 0) 0x7f903c058f80 con 0x7f90400ce9f0 2023-10-16T12:26:41.326 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:26:41.326 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":86,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:26:38.750010+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:26:21.893734+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2023-10-16T12:26:22.642815+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/2700763633":"2023-10-17T12:26:38.749960+0000","172.21.15.16:0/2069262541":"2023-10-17T12:26:38.749960+0000","172.21.15.16:0/1444840585":"2023-10-17T12:26:38.749960+0000","172.21.15.16:6833/3415303009":"2023-10-17T12:26:38.749960+0000","172.21.15.16:6832/3415303009":"2023-10-17T12:26:38.749960+0000","172.21.15.16:0/779547724":"2023-10-17T12:26:38.749960+0000","172.21.15.16:0/4285863251":"2023-10-17T12:26:38.749960+0000","172.21.15.16:0/4057240133":"2023-10-17T12:26:38.749960+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:26:41.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.326+0000 7f90482b2700 1 -- 172.21.15.16:0/2999864960 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f902406ad50 msgr2=0x7f902406d210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:26:41.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.326+0000 7f90482b2700 1 --2- 172.21.15.16:0/2999864960 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f902406ad50 0x7f902406d210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:41.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.326+0000 7f90482b2700 1 -- 172.21.15.16:0/2999864960 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90400ce9f0 msgr2=0x7f90401f2670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:41.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.326+0000 7f90482b2700 1 --2- 172.21.15.16:0/2999864960 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90400ce9f0 0x7f90401f2670 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f903c014040 tx=0x7f903c004750 comp rx=0 tx=0).stop 2023-10-16T12:26:41.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.326+0000 7f90482b2700 1 -- 172.21.15.16:0/2999864960 shutdown_connections 2023-10-16T12:26:41.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.326+0000 7f90482b2700 1 --2- 172.21.15.16:0/2999864960 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f902406ad50 0x7f902406d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:41.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.326+0000 7f90482b2700 1 --2- 172.21.15.16:0/2999864960 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f904015d900 0x7f904014fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:41.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.326+0000 7f90482b2700 1 --2- 172.21.15.16:0/2999864960 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90400ce9f0 0x7f90401f2670 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:41.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.326+0000 7f90482b2700 1 --2- 172.21.15.16:0/2999864960 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f90400d2860 0x7f904014f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:41.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.326+0000 7f90482b2700 1 -- 172.21.15.16:0/2999864960 >> 172.21.15.16:0/2999864960 conn(0x7f9040072330 msgr2=0x7f90400ca6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:41.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.326+0000 7f90482b2700 1 -- 172.21.15.16:0/2999864960 shutdown_connections 2023-10-16T12:26:41.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.326+0000 7f90482b2700 1 -- 172.21.15.16:0/2999864960 wait complete. 2023-10-16T12:26:41.341 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T12:26:21.893734+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T12:26:22.642815+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T12:26:41.342 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:26:41.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.509+0000 7f0938168700 1 Processor -- start 2023-10-16T12:26:41.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.509+0000 7f0938168700 1 -- start start 2023-10-16T12:26:41.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.510+0000 7f0938168700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f093015ec80 0x7f09300cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:41.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.510+0000 7f0938168700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f093015f060 0x7f09300ce400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:41.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.510+0000 7f0938168700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09300b2440 con 0x7f09300d2110 2023-10-16T12:26:41.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.510+0000 7f0938168700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09300b2c50 con 0x7f093015f060 2023-10-16T12:26:41.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.510+0000 7f0938168700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09300d00e0 con 0x7f093015ec80 2023-10-16T12:26:41.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.510+0000 7f0935703700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f093015f060 0x7f09300ce400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:41.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.510+0000 7f0935f04700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f093015ec80 0x7f09300cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:41.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.510+0000 7f0936705700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7f09300d2110 0x7f09300b4440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:36506/0 (socket says 172.21.15.16:36506) 2023-10-16T12:26:41.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.510+0000 7f0935f04700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f093015ec80 0x7f09300cdec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:56536/0 (socket says 172.21.15.16:56536) 2023-10-16T12:26:41.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.510+0000 7f0935703700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f093015f060 0x7f09300ce400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:39110/0 (socket says 172.21.15.16:39110) 2023-10-16T12:26:41.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.510+0000 7f0936705700 1 -- 172.21.15.16:0/2159045951 learned_addr learned my addr 172.21.15.16:0/2159045951 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:26:41.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.511+0000 7f0935f04700 1 -- 172.21.15.16:0/2159045951 >> v2:172.21.15.49:3300/0 conn(0x7f093015f060 msgr2=0x7f09300ce400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:41.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.511+0000 7f0935f04700 1 --2- 172.21.15.16:0/2159045951 >> v2:172.21.15.49:3300/0 conn(0x7f093015f060 0x7f09300ce400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:41.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.511+0000 7f0935f04700 1 -- 172.21.15.16:0/2159045951 >> v1:172.21.15.49:6789/0 conn(0x7f09300d2110 legacy=0x7f09300b4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:41.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.511+0000 7f0935f04700 1 -- 172.21.15.16:0/2159045951 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09300cef60 con 0x7f093015ec80 2023-10-16T12:26:41.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.511+0000 7f0935703700 1 --2- 172.21.15.16:0/2159045951 >> v2:172.21.15.49:3300/0 conn(0x7f093015f060 0x7f09300ce400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:26:41.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.511+0000 7f0935f04700 1 --2- 172.21.15.16:0/2159045951 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f093015ec80 0x7f09300cdec0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f092c009840 tx=0x7f092c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77ca780f3bf8661a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:41.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.511+0000 7f0934f02700 1 -- 172.21.15.16:0/2159045951 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f092c011830 con 0x7f093015ec80 2023-10-16T12:26:41.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.511+0000 7f0934f02700 1 -- 172.21.15.16:0/2159045951 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f092c011990 con 0x7f093015ec80 2023-10-16T12:26:41.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.511+0000 7f0934f02700 1 -- 172.21.15.16:0/2159045951 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f092c011d20 con 0x7f093015ec80 2023-10-16T12:26:41.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.512+0000 7f0938168700 1 -- 172.21.15.16:0/2159045951 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f093015ec80 msgr2=0x7f09300cdec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:41.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.512+0000 7f0938168700 1 --2- 172.21.15.16:0/2159045951 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f093015ec80 0x7f09300cdec0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f092c009840 tx=0x7f092c009b50 comp rx=0 tx=0).stop 2023-10-16T12:26:41.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.512+0000 7f0938168700 1 -- 172.21.15.16:0/2159045951 shutdown_connections 2023-10-16T12:26:41.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.512+0000 7f0938168700 1 --2- 172.21.15.16:0/2159045951 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f093015ec80 0x7f09300cdec0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:41.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.512+0000 7f0938168700 1 --2- 172.21.15.16:0/2159045951 >> v2:172.21.15.49:3300/0 conn(0x7f093015f060 0x7f09300ce400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:41.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.512+0000 7f0938168700 1 -- 172.21.15.16:0/2159045951 >> 172.21.15.16:0/2159045951 conn(0x7f09300d15f0 msgr2=0x7f09300d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:41.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.512+0000 7f0938168700 1 -- 172.21.15.16:0/2159045951 shutdown_connections 2023-10-16T12:26:41.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.512+0000 7f0938168700 1 -- 172.21.15.16:0/2159045951 wait complete. 2023-10-16T12:26:41.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.514+0000 7f0938168700 1 Processor -- start 2023-10-16T12:26:41.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.514+0000 7f0938168700 1 -- start start 2023-10-16T12:26:41.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.515+0000 7f0938168700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f09300d2110 0x7f0930151640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:41.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.515+0000 7f0938168700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f093015ec80 0x7f09301e4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:41.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.515+0000 7f0938168700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f093015f060 0x7f09301f4380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:41.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.515+0000 7f0938168700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09300c8a30 con 0x7f093015f060 2023-10-16T12:26:41.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.515+0000 7f0938168700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f093015c270 con 0x7f093015ec80 2023-10-16T12:26:41.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.515+0000 7f0938168700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f093015c550 con 0x7f09300d2110 2023-10-16T12:26:41.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.516+0000 7f0936705700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f093015f060 0x7f09301f4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:41.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.516+0000 7f0935703700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f093015ec80 0x7f09301e4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:41.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.516+0000 7f0936705700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f093015f060 0x7f09301f4380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40056/0 (socket says 172.21.15.16:40056) 2023-10-16T12:26:41.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.516+0000 7f0936705700 1 -- 172.21.15.16:0/1848473207 learned_addr learned my addr 172.21.15.16:0/1848473207 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:41.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.516+0000 7f0935f04700 1 --2- 172.21.15.16:0/1848473207 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f09300d2110 0x7f0930151640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:41.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.516+0000 7f0936705700 1 -- 172.21.15.16:0/1848473207 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f09300d2110 msgr2=0x7f0930151640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:41.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.516+0000 7f0936705700 1 --2- 172.21.15.16:0/1848473207 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f09300d2110 0x7f0930151640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:41.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.516+0000 7f0936705700 1 -- 172.21.15.16:0/1848473207 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f093015ec80 msgr2=0x7f09301e4b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:41.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.516+0000 7f0936705700 1 --2- 172.21.15.16:0/1848473207 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f093015ec80 0x7f09301e4b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:41.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.516+0000 7f0936705700 1 -- 172.21.15.16:0/1848473207 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09200094a0 con 0x7f093015f060 2023-10-16T12:26:41.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.516+0000 7f0936705700 1 --2- 172.21.15.16:0/1848473207 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f093015f060 0x7f09301f4380 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f092800cbc0 tx=0x7f092800ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:41.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.516+0000 7f09267fc700 1 -- 172.21.15.16:0/1848473207 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f092800a9e0 con 0x7f093015f060 2023-10-16T12:26:41.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.516+0000 7f09267fc700 1 -- 172.21.15.16:0/1848473207 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f092800ab40 con 0x7f093015f060 2023-10-16T12:26:41.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.516+0000 7f0938168700 1 -- 172.21.15.16:0/1848473207 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f092c0094a0 con 0x7f093015f060 2023-10-16T12:26:41.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.516+0000 7f09267fc700 1 -- 172.21.15.16:0/1848473207 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0928012400 con 0x7f093015f060 2023-10-16T12:26:41.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.516+0000 7f0938168700 1 -- 172.21.15.16:0/1848473207 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09300c86c0 con 0x7f093015f060 2023-10-16T12:26:41.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.517+0000 7f0938168700 1 -- 172.21.15.16:0/1848473207 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09300c8ba0 con 0x7f093015f060 2023-10-16T12:26:41.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.518+0000 7f09267fc700 1 -- 172.21.15.16:0/1848473207 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 36) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f092801b3b0 con 0x7f093015f060 2023-10-16T12:26:41.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.518+0000 7f09267fc700 1 --2- 172.21.15.16:0/1848473207 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f091406ad50 0x7f091406d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:41.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.518+0000 7f0935f04700 1 -- 172.21.15.16:0/1848473207 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f091406ad50 msgr2=0x7f091406d210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2230262559 2023-10-16T12:26:41.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.518+0000 7f09267fc700 1 -- 172.21.15.16:0/1848473207 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f0928088f00 con 0x7f093015f060 2023-10-16T12:26:41.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.519+0000 7f0935f04700 1 --2- 172.21.15.16:0/1848473207 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f091406ad50 0x7f091406d210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:26:41.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.523+0000 7f09267fc700 1 -- 172.21.15.16:0/1848473207 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f09280540e0 con 0x7f093015f060 2023-10-16T12:26:41.634 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:26:41.633+0000 7f50d29c5200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T12:26:41.656 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:41.654+0000 7f11cbf9c200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T12:26:41.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.720+0000 7f0935f04700 1 -- 172.21.15.16:0/1848473207 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f091406ad50 msgr2=0x7f091406d210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2230262559 2023-10-16T12:26:41.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.720+0000 7f0935f04700 1 --2- 172.21.15.16:0/1848473207 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f091406ad50 0x7f091406d210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:26:41.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.725+0000 7f0938168700 1 -- 172.21.15.16:0/1848473207 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f093004ebb0 con 0x7f093015f060 2023-10-16T12:26:41.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.725+0000 7f09267fc700 1 -- 172.21.15.16:0/1848473207 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v86) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0928017020 con 0x7f093015f060 2023-10-16T12:26:41.728 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T12:26:41.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.728+0000 7f0938168700 1 -- 172.21.15.16:0/1848473207 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f091406ad50 msgr2=0x7f091406d210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:26:41.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.728+0000 7f0938168700 1 --2- 172.21.15.16:0/1848473207 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f091406ad50 0x7f091406d210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:41.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.728+0000 7f0938168700 1 -- 172.21.15.16:0/1848473207 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f093015f060 msgr2=0x7f09301f4380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:41.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.728+0000 7f0938168700 1 --2- 172.21.15.16:0/1848473207 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f093015f060 0x7f09301f4380 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f092800cbc0 tx=0x7f092800ced0 comp rx=0 tx=0).stop 2023-10-16T12:26:41.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.728+0000 7f0938168700 1 -- 172.21.15.16:0/1848473207 shutdown_connections 2023-10-16T12:26:41.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.728+0000 7f0938168700 1 --2- 172.21.15.16:0/1848473207 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f091406ad50 0x7f091406d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:41.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.728+0000 7f0938168700 1 --2- 172.21.15.16:0/1848473207 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f093015f060 0x7f09301f4380 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:41.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.728+0000 7f0938168700 1 --2- 172.21.15.16:0/1848473207 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f09300d2110 0x7f0930151640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:41.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.728+0000 7f0938168700 1 --2- 172.21.15.16:0/1848473207 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f093015ec80 0x7f09301e4b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:41.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.728+0000 7f0938168700 1 -- 172.21.15.16:0/1848473207 >> 172.21.15.16:0/1848473207 conn(0x7f09300d15f0 msgr2=0x7f09300d19d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:41.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.728+0000 7f0938168700 1 -- 172.21.15.16:0/1848473207 shutdown_connections 2023-10-16T12:26:41.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.728+0000 7f0938168700 1 -- 172.21.15.16:0/1848473207 wait complete. 2023-10-16T12:26:41.743 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T12:26:41.756 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:26:41.755+0000 7f50d29c5200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T12:26:41.782 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:41.780+0000 7f11cbf9c200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T12:26:41.835 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:26:41.834+0000 7f50d29c5200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T12:26:41.860 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:41.858+0000 7f11cbf9c200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T12:26:41.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.877+0000 7fd13f0fc700 1 Processor -- start 2023-10-16T12:26:41.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.877+0000 7fd13f0fc700 1 -- start start 2023-10-16T12:26:41.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.878+0000 7fd13f0fc700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd13815bb40 0x7fd138071a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:41.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.878+0000 7fd13f0fc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd1380caa40 0x7fd1380cae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:41.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.878+0000 7fd13f0fc700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1380cb3b0 con 0x7fd138072b30 2023-10-16T12:26:41.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.878+0000 7fd13f0fc700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd13806ea20 con 0x7fd1380caa40 2023-10-16T12:26:41.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.878+0000 7fd13f0fc700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd13815c040 con 0x7fd13815bb40 2023-10-16T12:26:41.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.878+0000 7fd13d699700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd1380caa40 0x7fd1380cae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:41.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.878+0000 7fd13ce98700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd138072b30 0x7fd13806e310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57568/0 (socket says 172.21.15.16:57568) 2023-10-16T12:26:41.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.878+0000 7fd13ce98700 1 -- 172.21.15.16:0/2437751707 learned_addr learned my addr 172.21.15.16:0/2437751707 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:26:41.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.878+0000 7fd13d699700 1 --2- 172.21.15.16:0/2437751707 >> v2:172.21.15.16:3300/0 conn(0x7fd1380caa40 0x7fd1380cae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40066/0 (socket says 172.21.15.16:40066) 2023-10-16T12:26:41.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.878+0000 7fd137fff700 1 --2- 172.21.15.16:0/2437751707 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd13815bb40 0x7fd138071a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:41.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.879+0000 7fd13d699700 1 -- 172.21.15.16:0/2437751707 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd13815bb40 msgr2=0x7fd138071a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:41.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.879+0000 7fd13d699700 1 --2- 172.21.15.16:0/2437751707 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd13815bb40 0x7fd138071a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:41.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.879+0000 7fd13d699700 1 -- 172.21.15.16:0/2437751707 >> v1:172.21.15.16:6789/0 conn(0x7fd138072b30 legacy=0x7fd13806e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:41.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.879+0000 7fd13d699700 1 -- 172.21.15.16:0/2437751707 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd13815c320 con 0x7fd1380caa40 2023-10-16T12:26:41.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.879+0000 7fd1377fe700 1 -- 172.21.15.16:0/2437751707 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1931874432 0 0) 0x7fd1380cb3b0 con 0x7fd138072b30 2023-10-16T12:26:41.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.879+0000 7fd13d699700 1 --2- 172.21.15.16:0/2437751707 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1380caa40 0x7fd1380cae70 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fd130009d90 tx=0x7fd13000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdbf1e0a25a1e4f4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:41.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.879+0000 7fd1377fe700 1 -- 172.21.15.16:0/2437751707 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd13000e070 con 0x7fd1380caa40 2023-10-16T12:26:41.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.879+0000 7fd1377fe700 1 -- 172.21.15.16:0/2437751707 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd13000bb20 con 0x7fd1380caa40 2023-10-16T12:26:41.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.879+0000 7fd1377fe700 1 -- 172.21.15.16:0/2437751707 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd130012430 con 0x7fd1380caa40 2023-10-16T12:26:41.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.880+0000 7fd13f0fc700 1 -- 172.21.15.16:0/2437751707 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1380caa40 msgr2=0x7fd1380cae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:41.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.880+0000 7fd13f0fc700 1 --2- 172.21.15.16:0/2437751707 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1380caa40 0x7fd1380cae70 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fd130009d90 tx=0x7fd13000b330 comp rx=0 tx=0).stop 2023-10-16T12:26:41.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.880+0000 7fd13f0fc700 1 -- 172.21.15.16:0/2437751707 shutdown_connections 2023-10-16T12:26:41.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.880+0000 7fd13f0fc700 1 --2- 172.21.15.16:0/2437751707 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1380caa40 0x7fd1380cae70 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:41.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.880+0000 7fd13f0fc700 1 --2- 172.21.15.16:0/2437751707 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd13815bb40 0x7fd138071a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:41.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.880+0000 7fd13f0fc700 1 -- 172.21.15.16:0/2437751707 >> 172.21.15.16:0/2437751707 conn(0x7fd1380b1f40 msgr2=0x7fd1380b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:41.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.880+0000 7fd13f0fc700 1 -- 172.21.15.16:0/2437751707 shutdown_connections 2023-10-16T12:26:41.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.880+0000 7fd13f0fc700 1 -- 172.21.15.16:0/2437751707 wait complete. 2023-10-16T12:26:41.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.882+0000 7fd13f0fc700 1 Processor -- start 2023-10-16T12:26:41.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.882+0000 7fd13f0fc700 1 -- start start 2023-10-16T12:26:41.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.883+0000 7fd13f0fc700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd138072b30 0x7fd138150d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:41.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.883+0000 7fd13f0fc700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd1380caa40 0x7fd1380ae5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:41.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.883+0000 7fd13f0fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd13815bb40 0x7fd1380aeaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:41.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.883+0000 7fd13f0fc700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1380cdfc0 con 0x7fd13815bb40 2023-10-16T12:26:41.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.883+0000 7fd13f0fc700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1380ce280 con 0x7fd1380caa40 2023-10-16T12:26:41.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.883+0000 7fd13f0fc700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1380ce560 con 0x7fd138072b30 2023-10-16T12:26:41.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.883+0000 7fd13d699700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd13815bb40 0x7fd1380aeaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:41.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.883+0000 7fd13ce98700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd138072b30 0x7fd138150d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:41.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.883+0000 7fd13d699700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd13815bb40 0x7fd1380aeaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40072/0 (socket says 172.21.15.16:40072) 2023-10-16T12:26:41.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.883+0000 7fd13d699700 1 -- 172.21.15.16:0/2001376111 learned_addr learned my addr 172.21.15.16:0/2001376111 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:41.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.883+0000 7fd137fff700 1 --2- 172.21.15.16:0/2001376111 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd1380caa40 0x7fd1380ae5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:41.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.884+0000 7fd13d699700 1 -- 172.21.15.16:0/2001376111 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd138072b30 msgr2=0x7fd138150d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:41.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.884+0000 7fd13d699700 1 --2- 172.21.15.16:0/2001376111 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd138072b30 0x7fd138150d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:41.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.884+0000 7fd13d699700 1 -- 172.21.15.16:0/2001376111 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd1380caa40 msgr2=0x7fd1380ae5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:41.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.884+0000 7fd13d699700 1 --2- 172.21.15.16:0/2001376111 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd1380caa40 0x7fd1380ae5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:41.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.884+0000 7fd13d699700 1 -- 172.21.15.16:0/2001376111 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd130009a70 con 0x7fd13815bb40 2023-10-16T12:26:41.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.884+0000 7fd137fff700 1 --2- 172.21.15.16:0/2001376111 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd1380caa40 0x7fd1380ae5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:26:41.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.884+0000 7fd13ce98700 1 --2- 172.21.15.16:0/2001376111 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd138072b30 0x7fd138150d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:26:41.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.884+0000 7fd13d699700 1 --2- 172.21.15.16:0/2001376111 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd13815bb40 0x7fd1380aeaf0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7fd1380705c0 tx=0x7fd130004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:41.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.884+0000 7fd1357fa700 1 -- 172.21.15.16:0/2001376111 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd130012430 con 0x7fd13815bb40 2023-10-16T12:26:41.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.884+0000 7fd1357fa700 1 -- 172.21.15.16:0/2001376111 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd130020070 con 0x7fd13815bb40 2023-10-16T12:26:41.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.884+0000 7fd13f0fc700 1 -- 172.21.15.16:0/2001376111 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1380ce840 con 0x7fd13815bb40 2023-10-16T12:26:41.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.884+0000 7fd1357fa700 1 -- 172.21.15.16:0/2001376111 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd130004280 con 0x7fd13815bb40 2023-10-16T12:26:41.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.884+0000 7fd13f0fc700 1 -- 172.21.15.16:0/2001376111 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1380cee00 con 0x7fd13815bb40 2023-10-16T12:26:41.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.885+0000 7fd13f0fc700 1 -- 172.21.15.16:0/2001376111 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd138151570 con 0x7fd13815bb40 2023-10-16T12:26:41.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.886+0000 7fd1357fa700 1 -- 172.21.15.16:0/2001376111 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 37) v1 ==== 89332+0+0 (secure 0 0 0) 0x7fd1300043e0 con 0x7fd13815bb40 2023-10-16T12:26:41.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.886+0000 7fd1357fa700 1 --2- 172.21.15.16:0/2001376111 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7fd11c06b020 0x7fd11c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:41.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.886+0000 7fd13ce98700 1 -- 172.21.15.16:0/2001376111 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7fd11c06b020 msgr2=0x7fd11c06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2230262559 2023-10-16T12:26:41.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.886+0000 7fd1357fa700 1 -- 172.21.15.16:0/2001376111 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fd130022070 con 0x7fd13815bb40 2023-10-16T12:26:41.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.886+0000 7fd13ce98700 1 --2- 172.21.15.16:0/2001376111 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7fd11c06b020 0x7fd11c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:26:41.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:41.891+0000 7fd1357fa700 1 -- 172.21.15.16:0/2001376111 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd138151570 con 0x7fd13815bb40 2023-10-16T12:26:41.918 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:26:41.917+0000 7f50d29c5200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T12:26:41.949 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:41.947+0000 7f11cbf9c200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T12:26:42.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.087+0000 7fd13ce98700 1 -- 172.21.15.16:0/2001376111 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7fd11c06b020 msgr2=0x7fd11c06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2230262559 2023-10-16T12:26:42.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.087+0000 7fd13ce98700 1 --2- 172.21.15.16:0/2001376111 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7fd11c06b020 0x7fd11c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:26:42.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.096+0000 7fd13f0fc700 1 -- 172.21.15.16:0/2001376111 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd138151570 con 0x7fd13815bb40 2023-10-16T12:26:42.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.097+0000 7fd1357fa700 1 -- 172.21.15.16:0/2001376111 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v86) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd138151570 con 0x7fd13815bb40 2023-10-16T12:26:42.099 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-10-16T12:26:42.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.099+0000 7fd13f0fc700 1 -- 172.21.15.16:0/2001376111 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7fd11c06b020 msgr2=0x7fd11c06d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:26:42.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.099+0000 7fd13f0fc700 1 --2- 172.21.15.16:0/2001376111 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7fd11c06b020 0x7fd11c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:42.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.099+0000 7fd13f0fc700 1 -- 172.21.15.16:0/2001376111 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd13815bb40 msgr2=0x7fd1380aeaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:42.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.099+0000 7fd13f0fc700 1 --2- 172.21.15.16:0/2001376111 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd13815bb40 0x7fd1380aeaf0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7fd1380705c0 tx=0x7fd130004ad0 comp rx=0 tx=0).stop 2023-10-16T12:26:42.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.099+0000 7fd13f0fc700 1 -- 172.21.15.16:0/2001376111 shutdown_connections 2023-10-16T12:26:42.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.099+0000 7fd13f0fc700 1 --2- 172.21.15.16:0/2001376111 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7fd11c06b020 0x7fd11c06d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:42.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.099+0000 7fd13f0fc700 1 --2- 172.21.15.16:0/2001376111 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd13815bb40 0x7fd1380aeaf0 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:42.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.099+0000 7fd13f0fc700 1 --2- 172.21.15.16:0/2001376111 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd138072b30 0x7fd138150d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:42.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.099+0000 7fd13f0fc700 1 --2- 172.21.15.16:0/2001376111 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd1380caa40 0x7fd1380ae5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:42.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.099+0000 7fd13f0fc700 1 -- 172.21.15.16:0/2001376111 >> 172.21.15.16:0/2001376111 conn(0x7fd1380b1f40 msgr2=0x7fd1380b1670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:42.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.099+0000 7fd13f0fc700 1 -- 172.21.15.16:0/2001376111 shutdown_connections 2023-10-16T12:26:42.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.099+0000 7fd13f0fc700 1 -- 172.21.15.16:0/2001376111 wait complete. 2023-10-16T12:26:42.114 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T12:26:42.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.281+0000 7fd9a7494700 1 Processor -- start 2023-10-16T12:26:42.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.282+0000 7fd9a7494700 1 -- start start 2023-10-16T12:26:42.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.283+0000 7fd9a7494700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd9a015ec80 0x7fd9a00cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:42.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.283+0000 7fd9a7494700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9a00b2440 con 0x7fd9a015f060 2023-10-16T12:26:42.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.283+0000 7fd9a7494700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9a00d00e0 con 0x7fd9a00d2110 2023-10-16T12:26:42.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.283+0000 7fd9a7494700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9a00b2c50 con 0x7fd9a015ec80 2023-10-16T12:26:42.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.283+0000 7fd9a5230700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd9a015ec80 0x7fd9a00cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:42.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.283+0000 7fd9a5230700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd9a015ec80 0x7fd9a00cdec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40120/0 (socket says 172.21.15.16:40120) 2023-10-16T12:26:42.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.283+0000 7fd9a4a2f700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd9a015f060 0x7fd9a00b4440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57626/0 (socket says 172.21.15.16:57626) 2023-10-16T12:26:42.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.283+0000 7fd9a5230700 1 -- 172.21.15.16:0/2619353143 learned_addr learned my addr 172.21.15.16:0/2619353143 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:42.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.283+0000 7fd9a5230700 1 -- 172.21.15.16:0/2619353143 >> v1:172.21.15.49:6789/0 conn(0x7fd9a00d2110 legacy=0x7fd9a00b4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:42.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.283+0000 7fd9a5230700 1 -- 172.21.15.16:0/2619353143 >> v1:172.21.15.16:6789/0 conn(0x7fd9a015f060 legacy=0x7fd9a00b4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:42.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.283+0000 7fd9a5230700 1 -- 172.21.15.16:0/2619353143 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9a00cef60 con 0x7fd9a015ec80 2023-10-16T12:26:42.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.284+0000 7fd9a5230700 1 --2- 172.21.15.16:0/2619353143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9a015ec80 0x7fd9a00cdec0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fd99c0098f0 tx=0x7fd99c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b81707687f7567ba server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:42.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.284+0000 7fd997fff700 1 -- 172.21.15.16:0/2619353143 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd99c00b8a0 con 0x7fd9a015ec80 2023-10-16T12:26:42.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.284+0000 7fd997fff700 1 -- 172.21.15.16:0/2619353143 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd99c00ba00 con 0x7fd9a015ec80 2023-10-16T12:26:42.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.284+0000 7fd997fff700 1 -- 172.21.15.16:0/2619353143 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd99c00bd90 con 0x7fd9a015ec80 2023-10-16T12:26:42.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.285+0000 7fd9a7494700 1 -- 172.21.15.16:0/2619353143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9a015ec80 msgr2=0x7fd9a00cdec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:42.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.285+0000 7fd9a7494700 1 --2- 172.21.15.16:0/2619353143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9a015ec80 0x7fd9a00cdec0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fd99c0098f0 tx=0x7fd99c009c00 comp rx=0 tx=0).stop 2023-10-16T12:26:42.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.285+0000 7fd9a7494700 1 -- 172.21.15.16:0/2619353143 shutdown_connections 2023-10-16T12:26:42.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.285+0000 7fd9a7494700 1 --2- 172.21.15.16:0/2619353143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9a015ec80 0x7fd9a00cdec0 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:42.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.285+0000 7fd9a7494700 1 -- 172.21.15.16:0/2619353143 >> 172.21.15.16:0/2619353143 conn(0x7fd9a00d15f0 msgr2=0x7fd9a00d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:42.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.285+0000 7fd9a7494700 1 -- 172.21.15.16:0/2619353143 shutdown_connections 2023-10-16T12:26:42.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.285+0000 7fd9a7494700 1 -- 172.21.15.16:0/2619353143 wait complete. 2023-10-16T12:26:42.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.287+0000 7fd9a7494700 1 Processor -- start 2023-10-16T12:26:42.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.287+0000 7fd9a7494700 1 -- start start 2023-10-16T12:26:42.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.288+0000 7fd9a7494700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd9a00d2110 0x7fd9a00701a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:42.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.288+0000 7fd9a7494700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd9a015ec80 0x7fd9a0156280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:42.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.288+0000 7fd9a7494700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9a015f060 0x7fd9a00ca3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:42.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.288+0000 7fd9a7494700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9a01f4ae0 con 0x7fd9a015f060 2023-10-16T12:26:42.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.288+0000 7fd9a7494700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9a01f4dc0 con 0x7fd9a00d2110 2023-10-16T12:26:42.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.288+0000 7fd9a7494700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9a01f50a0 con 0x7fd9a015ec80 2023-10-16T12:26:42.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.288+0000 7fd9a5a31700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9a015f060 0x7fd9a00ca3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:42.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.288+0000 7fd9a5a31700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9a015f060 0x7fd9a00ca3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40130/0 (socket says 172.21.15.16:40130) 2023-10-16T12:26:42.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.288+0000 7fd9a5a31700 1 -- 172.21.15.16:0/3251817163 learned_addr learned my addr 172.21.15.16:0/3251817163 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:42.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.288+0000 7fd9a5230700 1 --2- 172.21.15.16:0/3251817163 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd9a00d2110 0x7fd9a00701a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:42.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.288+0000 7fd9a4a2f700 1 --2- 172.21.15.16:0/3251817163 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd9a015ec80 0x7fd9a0156280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:42.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.288+0000 7fd9a5a31700 1 -- 172.21.15.16:0/3251817163 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd9a015ec80 msgr2=0x7fd9a0156280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:42.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.288+0000 7fd9a5a31700 1 --2- 172.21.15.16:0/3251817163 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd9a015ec80 0x7fd9a0156280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:42.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.289+0000 7fd9a5a31700 1 -- 172.21.15.16:0/3251817163 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd9a00d2110 msgr2=0x7fd9a00701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:42.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.289+0000 7fd9a5a31700 1 --2- 172.21.15.16:0/3251817163 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd9a00d2110 0x7fd9a00701a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:42.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.289+0000 7fd9a5a31700 1 -- 172.21.15.16:0/3251817163 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd99c009550 con 0x7fd9a015f060 2023-10-16T12:26:42.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.289+0000 7fd9a5a31700 1 --2- 172.21.15.16:0/3251817163 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9a015f060 0x7fd9a00ca3a0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7fd99800cc50 tx=0x7fd99800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:42.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.289+0000 7fd995ffb700 1 -- 172.21.15.16:0/3251817163 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd998003c00 con 0x7fd9a015f060 2023-10-16T12:26:42.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.289+0000 7fd995ffb700 1 -- 172.21.15.16:0/3251817163 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd998003d60 con 0x7fd9a015f060 2023-10-16T12:26:42.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.289+0000 7fd9a7494700 1 -- 172.21.15.16:0/3251817163 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9a01f5380 con 0x7fd9a015f060 2023-10-16T12:26:42.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.289+0000 7fd995ffb700 1 -- 172.21.15.16:0/3251817163 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd99800aab0 con 0x7fd9a015f060 2023-10-16T12:26:42.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.289+0000 7fd9a7494700 1 -- 172.21.15.16:0/3251817163 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9a00706e0 con 0x7fd9a015f060 2023-10-16T12:26:42.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.290+0000 7fd9a7494700 1 -- 172.21.15.16:0/3251817163 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9a00d0550 con 0x7fd9a015f060 2023-10-16T12:26:42.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.291+0000 7fd995ffb700 1 -- 172.21.15.16:0/3251817163 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 37) v1 ==== 89332+0+0 (secure 0 0 0) 0x7fd99800ac10 con 0x7fd9a015f060 2023-10-16T12:26:42.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.291+0000 7fd995ffb700 1 --2- 172.21.15.16:0/3251817163 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7fd98406b020 0x7fd98406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:42.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.291+0000 7fd9a5230700 1 -- 172.21.15.16:0/3251817163 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7fd98406b020 msgr2=0x7fd98406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2230262559 2023-10-16T12:26:42.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.291+0000 7fd995ffb700 1 -- 172.21.15.16:0/3251817163 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fd998089960 con 0x7fd9a015f060 2023-10-16T12:26:42.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.291+0000 7fd9a5230700 1 --2- 172.21.15.16:0/3251817163 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7fd98406b020 0x7fd98406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:26:42.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.296+0000 7fd995ffb700 1 -- 172.21.15.16:0/3251817163 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd9980546d0 con 0x7fd9a015f060 2023-10-16T12:26:42.302 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:26:42.302+0000 7f50d29c5200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T12:26:42.348 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:42.346+0000 7f11cbf9c200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T12:26:42.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.492+0000 7fd9a5230700 1 -- 172.21.15.16:0/3251817163 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7fd98406b020 msgr2=0x7fd98406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2230262559 2023-10-16T12:26:42.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.492+0000 7fd9a5230700 1 --2- 172.21.15.16:0/3251817163 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7fd98406b020 0x7fd98406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:26:42.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.499+0000 7fd9a7494700 1 -- 172.21.15.16:0/3251817163 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd9a004ebb0 con 0x7fd9a015f060 2023-10-16T12:26:42.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.499+0000 7fd995ffb700 1 -- 172.21.15.16:0/3251817163 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v86) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd998057cf0 con 0x7fd9a015f060 2023-10-16T12:26:42.502 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-10-16T12:26:42.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.502+0000 7fd9a7494700 1 -- 172.21.15.16:0/3251817163 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7fd98406b020 msgr2=0x7fd98406d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:26:42.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.502+0000 7fd9a7494700 1 --2- 172.21.15.16:0/3251817163 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7fd98406b020 0x7fd98406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:42.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.502+0000 7fd9a7494700 1 -- 172.21.15.16:0/3251817163 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9a015f060 msgr2=0x7fd9a00ca3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:42.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.502+0000 7fd9a7494700 1 --2- 172.21.15.16:0/3251817163 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9a015f060 0x7fd9a00ca3a0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7fd99800cc50 tx=0x7fd99800cf60 comp rx=0 tx=0).stop 2023-10-16T12:26:42.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.502+0000 7fd9a7494700 1 -- 172.21.15.16:0/3251817163 shutdown_connections 2023-10-16T12:26:42.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.502+0000 7fd9a7494700 1 --2- 172.21.15.16:0/3251817163 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7fd98406b020 0x7fd98406d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:42.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.502+0000 7fd9a7494700 1 --2- 172.21.15.16:0/3251817163 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9a015f060 0x7fd9a00ca3a0 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:42.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.502+0000 7fd9a7494700 1 --2- 172.21.15.16:0/3251817163 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd9a00d2110 0x7fd9a00701a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:42.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.502+0000 7fd9a7494700 1 --2- 172.21.15.16:0/3251817163 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd9a015ec80 0x7fd9a0156280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:42.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.502+0000 7fd9a7494700 1 -- 172.21.15.16:0/3251817163 >> 172.21.15.16:0/3251817163 conn(0x7fd9a00d15f0 msgr2=0x7fd9a00b4240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:42.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.502+0000 7fd9a7494700 1 -- 172.21.15.16:0/3251817163 shutdown_connections 2023-10-16T12:26:42.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.502+0000 7fd9a7494700 1 -- 172.21.15.16:0/3251817163 wait complete. 2023-10-16T12:26:42.517 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T12:26:42.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.642+0000 7fdd4a903700 1 Processor -- start 2023-10-16T12:26:42.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.642+0000 7fdd4a903700 1 -- start start 2023-10-16T12:26:42.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.643+0000 7fdd4a903700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdd4415db40 0x7fdd440cb980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:42.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.643+0000 7fdd4a903700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdd440ce870 0x7fdd4406ac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:42.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.643+0000 7fdd4a903700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd440cedb0 con 0x7fdd4406a7e0 2023-10-16T12:26:42.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.643+0000 7fdd4a903700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd4406b1a0 con 0x7fdd4415db40 2023-10-16T12:26:42.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.643+0000 7fdd4a903700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd4415df20 con 0x7fdd440ce870 2023-10-16T12:26:42.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.643+0000 7fdd43fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdd4415db40 0x7fdd440cb980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:42.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.643+0000 7fdd43fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdd4415db40 0x7fdd440cb980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40136/0 (socket says 172.21.15.16:40136) 2023-10-16T12:26:42.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.643+0000 7fdd43fff700 1 -- 172.21.15.16:0/1000232424 learned_addr learned my addr 172.21.15.16:0/1000232424 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:42.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.643+0000 7fdd437fe700 1 --1- 172.21.15.16:0/1000232424 >> v1:172.21.15.16:6789/0 conn(0x7fdd4406a7e0 0x7fdd440ce160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57642/0 (socket says 172.21.15.16:57642) 2023-10-16T12:26:42.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.643+0000 7fdd48ea0700 1 --2- 172.21.15.16:0/1000232424 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdd440ce870 0x7fdd4406ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:42.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.643+0000 7fdd43fff700 1 -- 172.21.15.16:0/1000232424 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdd440ce870 msgr2=0x7fdd4406ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:42.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.643+0000 7fdd43fff700 1 --2- 172.21.15.16:0/1000232424 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdd440ce870 0x7fdd4406ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:42.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.643+0000 7fdd43fff700 1 -- 172.21.15.16:0/1000232424 >> v1:172.21.15.16:6789/0 conn(0x7fdd4406a7e0 legacy=0x7fdd440ce160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:42.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.643+0000 7fdd42ffd700 1 -- 172.21.15.16:0/1000232424 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2809483553 0 0) 0x7fdd440cedb0 con 0x7fdd4406a7e0 2023-10-16T12:26:42.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.643+0000 7fdd43fff700 1 -- 172.21.15.16:0/1000232424 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd440d12c0 con 0x7fdd4415db40 2023-10-16T12:26:42.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.643+0000 7fdd43fff700 1 --2- 172.21.15.16:0/1000232424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd4415db40 0x7fdd440cb980 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7fdd340098f0 tx=0x7fdd34009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de80730a0c031542 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:42.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.644+0000 7fdd42ffd700 1 -- 172.21.15.16:0/1000232424 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd34011830 con 0x7fdd4415db40 2023-10-16T12:26:42.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.644+0000 7fdd42ffd700 1 -- 172.21.15.16:0/1000232424 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd34011990 con 0x7fdd4415db40 2023-10-16T12:26:42.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.644+0000 7fdd42ffd700 1 -- 172.21.15.16:0/1000232424 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd34011d20 con 0x7fdd4415db40 2023-10-16T12:26:42.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.644+0000 7fdd4a903700 1 -- 172.21.15.16:0/1000232424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd4415db40 msgr2=0x7fdd440cb980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:42.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.644+0000 7fdd4a903700 1 --2- 172.21.15.16:0/1000232424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd4415db40 0x7fdd440cb980 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7fdd340098f0 tx=0x7fdd34009c00 comp rx=0 tx=0).stop 2023-10-16T12:26:42.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.644+0000 7fdd4a903700 1 -- 172.21.15.16:0/1000232424 shutdown_connections 2023-10-16T12:26:42.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.644+0000 7fdd4a903700 1 --2- 172.21.15.16:0/1000232424 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdd440ce870 0x7fdd4406ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:42.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.644+0000 7fdd4a903700 1 --2- 172.21.15.16:0/1000232424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd4415db40 0x7fdd440cb980 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:42.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.644+0000 7fdd4a903700 1 -- 172.21.15.16:0/1000232424 >> 172.21.15.16:0/1000232424 conn(0x7fdd440b1fd0 msgr2=0x7fdd440b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:42.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.644+0000 7fdd4a903700 1 -- 172.21.15.16:0/1000232424 shutdown_connections 2023-10-16T12:26:42.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.644+0000 7fdd4a903700 1 -- 172.21.15.16:0/1000232424 wait complete. 2023-10-16T12:26:42.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.646+0000 7fdd4a903700 1 Processor -- start 2023-10-16T12:26:42.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.646+0000 7fdd4a903700 1 -- start start 2023-10-16T12:26:42.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.647+0000 7fdd4a903700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdd4406a7e0 0x7fdd441f6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:42.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.647+0000 7fdd4a903700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd440ce870 0x7fdd44160840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:42.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.647+0000 7fdd4a903700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdd4415db40 0x7fdd440c6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:42.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.647+0000 7fdd4a903700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd4414ec80 con 0x7fdd440ce870 2023-10-16T12:26:42.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.647+0000 7fdd4a903700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd440c88d0 con 0x7fdd4415db40 2023-10-16T12:26:42.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.647+0000 7fdd4a903700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd4415e190 con 0x7fdd4406a7e0 2023-10-16T12:26:42.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.647+0000 7fdd437fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd440ce870 0x7fdd44160840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:42.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.647+0000 7fdd437fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd440ce870 0x7fdd44160840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40142/0 (socket says 172.21.15.16:40142) 2023-10-16T12:26:42.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.647+0000 7fdd437fe700 1 -- 172.21.15.16:0/1443880513 learned_addr learned my addr 172.21.15.16:0/1443880513 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:42.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.647+0000 7fdd43fff700 1 --2- 172.21.15.16:0/1443880513 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdd4406a7e0 0x7fdd441f6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:42.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.647+0000 7fdd48ea0700 1 --2- 172.21.15.16:0/1443880513 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdd4415db40 0x7fdd440c6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:42.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.647+0000 7fdd437fe700 1 -- 172.21.15.16:0/1443880513 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdd4406a7e0 msgr2=0x7fdd441f6260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:42.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.647+0000 7fdd437fe700 1 --2- 172.21.15.16:0/1443880513 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdd4406a7e0 0x7fdd441f6260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:42.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.647+0000 7fdd437fe700 1 -- 172.21.15.16:0/1443880513 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdd4415db40 msgr2=0x7fdd440c6360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:42.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.647+0000 7fdd437fe700 1 --2- 172.21.15.16:0/1443880513 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdd4415db40 0x7fdd440c6360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:42.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.647+0000 7fdd437fe700 1 -- 172.21.15.16:0/1443880513 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd34009550 con 0x7fdd440ce870 2023-10-16T12:26:42.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.647+0000 7fdd437fe700 1 --2- 172.21.15.16:0/1443880513 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd440ce870 0x7fdd44160840 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7fdd3800c550 tx=0x7fdd3800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:42.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.647+0000 7fdd40ff9700 1 -- 172.21.15.16:0/1443880513 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd38004c80 con 0x7fdd440ce870 2023-10-16T12:26:42.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.647+0000 7fdd40ff9700 1 -- 172.21.15.16:0/1443880513 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd38004e00 con 0x7fdd440ce870 2023-10-16T12:26:42.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.647+0000 7fdd4a903700 1 -- 172.21.15.16:0/1443880513 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd4415e470 con 0x7fdd440ce870 2023-10-16T12:26:42.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.647+0000 7fdd40ff9700 1 -- 172.21.15.16:0/1443880513 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd38007310 con 0x7fdd440ce870 2023-10-16T12:26:42.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.648+0000 7fdd4a903700 1 -- 172.21.15.16:0/1443880513 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd440c8560 con 0x7fdd440ce870 2023-10-16T12:26:42.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.649+0000 7fdd4a903700 1 -- 172.21.15.16:0/1443880513 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd4404ebb0 con 0x7fdd440ce870 2023-10-16T12:26:42.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.649+0000 7fdd40ff9700 1 -- 172.21.15.16:0/1443880513 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 37) v1 ==== 89332+0+0 (secure 0 0 0) 0x7fdd38007470 con 0x7fdd440ce870 2023-10-16T12:26:42.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.649+0000 7fdd40ff9700 1 --2- 172.21.15.16:0/1443880513 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7fdd3006b020 0x7fdd3006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:42.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.650+0000 7fdd43fff700 1 -- 172.21.15.16:0/1443880513 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7fdd3006b020 msgr2=0x7fdd3006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2230262559 2023-10-16T12:26:42.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.650+0000 7fdd40ff9700 1 -- 172.21.15.16:0/1443880513 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fdd38089530 con 0x7fdd440ce870 2023-10-16T12:26:42.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.650+0000 7fdd43fff700 1 --2- 172.21.15.16:0/1443880513 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7fdd3006b020 0x7fdd3006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:26:42.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.654+0000 7fdd40ff9700 1 -- 172.21.15.16:0/1443880513 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdd380542a0 con 0x7fdd440ce870 2023-10-16T12:26:42.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.851+0000 7fdd43fff700 1 -- 172.21.15.16:0/1443880513 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7fdd3006b020 msgr2=0x7fdd3006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2230262559 2023-10-16T12:26:42.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.851+0000 7fdd43fff700 1 --2- 172.21.15.16:0/1443880513 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7fdd3006b020 0x7fdd3006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:26:42.907 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:26:42.905+0000 7f50d29c5200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T12:26:42.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.908+0000 7fdd4a903700 1 -- 172.21.15.16:0/1443880513 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fdd440cae70 con 0x7fdd440ce870 2023-10-16T12:26:42.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.909+0000 7fdd40ff9700 1 -- 172.21.15.16:0/1443880513 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v86) v1 ==== 80+0+469 (secure 0 0 0) 0x7fdd380578c0 con 0x7fdd440ce870 2023-10-16T12:26:42.911 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2700763633 2023-10-17T12:26:38.749960+0000 2023-10-16T12:26:42.912 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2069262541 2023-10-17T12:26:38.749960+0000 2023-10-16T12:26:42.912 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/1444840585 2023-10-17T12:26:38.749960+0000 2023-10-16T12:26:42.912 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6833/3415303009 2023-10-17T12:26:38.749960+0000 2023-10-16T12:26:42.912 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6832/3415303009 2023-10-17T12:26:38.749960+0000 2023-10-16T12:26:42.913 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/779547724 2023-10-17T12:26:38.749960+0000 2023-10-16T12:26:42.913 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/4285863251 2023-10-17T12:26:38.749960+0000 2023-10-16T12:26:42.913 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/4057240133 2023-10-17T12:26:38.749960+0000 2023-10-16T12:26:42.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.911+0000 7fdd4a903700 1 -- 172.21.15.16:0/1443880513 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7fdd3006b020 msgr2=0x7fdd3006d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:26:42.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.911+0000 7fdd4a903700 1 --2- 172.21.15.16:0/1443880513 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7fdd3006b020 0x7fdd3006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:42.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.911+0000 7fdd4a903700 1 -- 172.21.15.16:0/1443880513 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd440ce870 msgr2=0x7fdd44160840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:42.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.911+0000 7fdd4a903700 1 --2- 172.21.15.16:0/1443880513 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd440ce870 0x7fdd44160840 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7fdd3800c550 tx=0x7fdd3800c860 comp rx=0 tx=0).stop 2023-10-16T12:26:42.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.911+0000 7fdd4a903700 1 -- 172.21.15.16:0/1443880513 shutdown_connections 2023-10-16T12:26:42.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.911+0000 7fdd4a903700 1 --2- 172.21.15.16:0/1443880513 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7fdd3006b020 0x7fdd3006d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:42.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.911+0000 7fdd4a903700 1 --2- 172.21.15.16:0/1443880513 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdd4415db40 0x7fdd440c6360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:42.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.911+0000 7fdd4a903700 1 --2- 172.21.15.16:0/1443880513 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdd4406a7e0 0x7fdd441f6260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:42.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.911+0000 7fdd4a903700 1 --2- 172.21.15.16:0/1443880513 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd440ce870 0x7fdd44160840 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:42.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.911+0000 7fdd4a903700 1 -- 172.21.15.16:0/1443880513 >> 172.21.15.16:0/1443880513 conn(0x7fdd440b1fd0 msgr2=0x7fdd440cd5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:42.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.911+0000 7fdd4a903700 1 -- 172.21.15.16:0/1443880513 shutdown_connections 2023-10-16T12:26:42.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:42.911+0000 7fdd4a903700 1 -- 172.21.15.16:0/1443880513 wait complete. 2023-10-16T12:26:42.916 INFO:teuthology.orchestra.run.smithi016.stderr:listed 8 entries 2023-10-16T12:26:42.927 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-16T12:26:42.928 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:26:42.928 DEBUG:teuthology.orchestra.run.smithi016:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-16T12:26:42.967 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:42.965+0000 7f11cbf9c200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T12:26:42.975 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2023-10-16T12:26:42.975 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:26:42.976 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-16T12:26:42.979 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:26:42.978+0000 7f50d29c5200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T12:26:43.033 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T12:26:43.034 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:26:43.043 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:43.042+0000 7f11cbf9c200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T12:26:43.051 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284204k,nr_inodes=4071051,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=753 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260540k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T12:26:43.052 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:26:43.052 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:26:43.070 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16287364k,nr_inodes=4071841,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20251 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3261172k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T12:26:43.071 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T12:26:43.071 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:26:43.109 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284204k,nr_inodes=4071051,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=753 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260540k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T12:26:43.110 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:26:43.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.235+0000 7f0e685a8700 1 Processor -- start 2023-10-16T12:26:43.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.235+0000 7f0e685a8700 1 -- start start 2023-10-16T12:26:43.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.236+0000 7f0e685a8700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0e600d2040 0x7f0e600ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:43.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.236+0000 7f0e685a8700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e600ce400 con 0x7f0e600d2f70 2023-10-16T12:26:43.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.236+0000 7f0e685a8700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e6015dce0 con 0x7f0e6015d900 2023-10-16T12:26:43.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.236+0000 7f0e685a8700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e600d24d0 con 0x7f0e600d2040 2023-10-16T12:26:43.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.236+0000 7f0e65b43700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0e600d2f70 0x7f0e600b6790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57656/0 (socket says 172.21.15.16:57656) 2023-10-16T12:26:43.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.236+0000 7f0e65b43700 1 -- 172.21.15.16:0/1466668834 learned_addr learned my addr 172.21.15.16:0/1466668834 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:26:43.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.236+0000 7f0e66b45700 1 --2- 172.21.15.16:0/1466668834 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0e600d2040 0x7f0e600ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:43.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.237+0000 7f0e66b45700 1 -- 172.21.15.16:0/1466668834 >> v1:172.21.15.49:6789/0 conn(0x7f0e6015d900 legacy=0x7f0e600d2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:43.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.237+0000 7f0e66b45700 1 -- 172.21.15.16:0/1466668834 >> v1:172.21.15.16:6789/0 conn(0x7f0e600d2f70 legacy=0x7f0e600b6790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:43.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.237+0000 7f0e65342700 1 -- 172.21.15.16:0/1466668834 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1158065644 0 0) 0x7f0e6015dce0 con 0x7f0e6015d900 2023-10-16T12:26:43.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.237+0000 7f0e66b45700 1 -- 172.21.15.16:0/1466668834 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e600d1280 con 0x7f0e600d2040 2023-10-16T12:26:43.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.237+0000 7f0e66b45700 1 --2- 172.21.15.16:0/1466668834 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0e600d2040 0x7f0e600ce9f0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f0e58009dc0 tx=0x7f0e5800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=326bc5f439740d32 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:43.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.237+0000 7f0e65342700 1 -- 172.21.15.16:0/1466668834 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e5800e070 con 0x7f0e600d2040 2023-10-16T12:26:43.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.237+0000 7f0e65342700 1 -- 172.21.15.16:0/1466668834 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e5800bad0 con 0x7f0e600d2040 2023-10-16T12:26:43.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.237+0000 7f0e65342700 1 -- 172.21.15.16:0/1466668834 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e5800be80 con 0x7f0e600d2040 2023-10-16T12:26:43.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.238+0000 7f0e685a8700 1 -- 172.21.15.16:0/1466668834 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0e600d2040 msgr2=0x7f0e600ce9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:43.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.238+0000 7f0e685a8700 1 --2- 172.21.15.16:0/1466668834 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0e600d2040 0x7f0e600ce9f0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f0e58009dc0 tx=0x7f0e5800b330 comp rx=0 tx=0).stop 2023-10-16T12:26:43.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.238+0000 7f0e685a8700 1 -- 172.21.15.16:0/1466668834 shutdown_connections 2023-10-16T12:26:43.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.238+0000 7f0e685a8700 1 --2- 172.21.15.16:0/1466668834 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0e600d2040 0x7f0e600ce9f0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:43.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.238+0000 7f0e685a8700 1 -- 172.21.15.16:0/1466668834 >> 172.21.15.16:0/1466668834 conn(0x7f0e60072330 msgr2=0x7f0e60072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:43.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.238+0000 7f0e685a8700 1 -- 172.21.15.16:0/1466668834 shutdown_connections 2023-10-16T12:26:43.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.238+0000 7f0e685a8700 1 -- 172.21.15.16:0/1466668834 wait complete. 2023-10-16T12:26:43.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.240+0000 7f0e685a8700 1 Processor -- start 2023-10-16T12:26:43.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.240+0000 7f0e685a8700 1 -- start start 2023-10-16T12:26:43.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.241+0000 7f0e685a8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e600d2040 0x7f0e601f3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:43.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.241+0000 7f0e685a8700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0e600d2f70 0x7f0e601534c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:43.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.241+0000 7f0e685a8700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0e6015d900 0x7f0e600b7d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:43.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.241+0000 7f0e685a8700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e601f4300 con 0x7f0e600d2040 2023-10-16T12:26:43.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.241+0000 7f0e685a8700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e601f3a00 con 0x7f0e600d2f70 2023-10-16T12:26:43.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.241+0000 7f0e685a8700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e601f3ce0 con 0x7f0e6015d900 2023-10-16T12:26:43.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.241+0000 7f0e66344700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e600d2040 0x7f0e601f3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:43.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.241+0000 7f0e66344700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e600d2040 0x7f0e601f3140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40156/0 (socket says 172.21.15.16:40156) 2023-10-16T12:26:43.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.241+0000 7f0e66344700 1 -- 172.21.15.16:0/1194699192 learned_addr learned my addr 172.21.15.16:0/1194699192 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:43.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.241+0000 7f0e65b43700 1 --2- 172.21.15.16:0/1194699192 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0e600d2f70 0x7f0e601534c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:43.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.241+0000 7f0e66b45700 1 --2- 172.21.15.16:0/1194699192 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0e6015d900 0x7f0e600b7d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:43.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.241+0000 7f0e66344700 1 -- 172.21.15.16:0/1194699192 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0e6015d900 msgr2=0x7f0e600b7d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:43.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.241+0000 7f0e66344700 1 --2- 172.21.15.16:0/1194699192 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0e6015d900 0x7f0e600b7d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:43.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.241+0000 7f0e66344700 1 -- 172.21.15.16:0/1194699192 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0e600d2f70 msgr2=0x7f0e601534c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:43.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.241+0000 7f0e66344700 1 --2- 172.21.15.16:0/1194699192 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0e600d2f70 0x7f0e601534c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:43.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.241+0000 7f0e66344700 1 -- 172.21.15.16:0/1194699192 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e58009a70 con 0x7f0e600d2040 2023-10-16T12:26:43.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.241+0000 7f0e66344700 1 --2- 172.21.15.16:0/1194699192 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e600d2040 0x7f0e601f3140 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f0e5c00c780 tx=0x7f0e5c00ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:43.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.241+0000 7f0e56ffd700 1 -- 172.21.15.16:0/1194699192 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e5c0054e0 con 0x7f0e600d2040 2023-10-16T12:26:43.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.241+0000 7f0e56ffd700 1 -- 172.21.15.16:0/1194699192 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e5c003680 con 0x7f0e600d2040 2023-10-16T12:26:43.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.241+0000 7f0e685a8700 1 -- 172.21.15.16:0/1194699192 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e601f3fc0 con 0x7f0e600d2040 2023-10-16T12:26:43.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.241+0000 7f0e56ffd700 1 -- 172.21.15.16:0/1194699192 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e5c003a30 con 0x7f0e600d2040 2023-10-16T12:26:43.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.242+0000 7f0e685a8700 1 -- 172.21.15.16:0/1194699192 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e601f3680 con 0x7f0e600d2040 2023-10-16T12:26:43.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.242+0000 7f0e685a8700 1 -- 172.21.15.16:0/1194699192 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e601f47e0 con 0x7f0e600d2040 2023-10-16T12:26:43.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.243+0000 7f0e56ffd700 1 -- 172.21.15.16:0/1194699192 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 37) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f0e5c003b90 con 0x7f0e600d2040 2023-10-16T12:26:43.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.243+0000 7f0e56ffd700 1 --2- 172.21.15.16:0/1194699192 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f0e4406b020 0x7f0e4406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:43.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.243+0000 7f0e65b43700 1 -- 172.21.15.16:0/1194699192 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f0e4406b020 msgr2=0x7f0e4406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2230262559 2023-10-16T12:26:43.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.244+0000 7f0e56ffd700 1 -- 172.21.15.16:0/1194699192 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f0e5c088e70 con 0x7f0e600d2040 2023-10-16T12:26:43.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.244+0000 7f0e65b43700 1 --2- 172.21.15.16:0/1194699192 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f0e4406b020 0x7f0e4406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:26:43.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.248+0000 7f0e56ffd700 1 -- 172.21.15.16:0/1194699192 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0e5c053b50 con 0x7f0e600d2040 2023-10-16T12:26:43.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.445+0000 7f0e65b43700 1 -- 172.21.15.16:0/1194699192 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f0e4406b020 msgr2=0x7f0e4406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2230262559 2023-10-16T12:26:43.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.445+0000 7f0e65b43700 1 --2- 172.21.15.16:0/1194699192 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f0e4406b020 0x7f0e4406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:26:43.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.499+0000 7f0e685a8700 1 -- 172.21.15.16:0/1194699192 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0e600d1810 con 0x7f0e600d2040 2023-10-16T12:26:43.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.500+0000 7f0e56ffd700 1 -- 172.21.15.16:0/1194699192 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+4911 (secure 0 0 0) 0x7f0e5c017070 con 0x7f0e600d2040 2023-10-16T12:26:43.502 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:26:43.502 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":36,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5861,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/1236108916","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":1236108916},{"type":"v1","addr":"172.21.15.49:6834","nonce":1236108916}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31},{"gid":5864,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/2762335434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":2762335434},{"type":"v1","addr":"172.21.15.49:6835","nonce":2762335434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31},{"gid":6379,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/1502256436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1502256436},{"type":"v1","addr":"172.21.15.16:6836","nonce":1502256436}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:26:23.670609+0000","modified":"2023-10-16T12:26:26.710098+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6627},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6627":{"gid":6627,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/962243511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":962243511},{"type":"v1","addr":"172.21.15.16:6837","nonce":962243511}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-10-16T12:26:43.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.502+0000 7f0e685a8700 1 -- 172.21.15.16:0/1194699192 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f0e4406b020 msgr2=0x7f0e4406d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:26:43.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.502+0000 7f0e685a8700 1 --2- 172.21.15.16:0/1194699192 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f0e4406b020 0x7f0e4406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:43.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.502+0000 7f0e685a8700 1 -- 172.21.15.16:0/1194699192 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e600d2040 msgr2=0x7f0e601f3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:43.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.502+0000 7f0e685a8700 1 --2- 172.21.15.16:0/1194699192 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e600d2040 0x7f0e601f3140 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f0e5c00c780 tx=0x7f0e5c00ca90 comp rx=0 tx=0).stop 2023-10-16T12:26:43.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.502+0000 7f0e685a8700 1 -- 172.21.15.16:0/1194699192 shutdown_connections 2023-10-16T12:26:43.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.502+0000 7f0e685a8700 1 --2- 172.21.15.16:0/1194699192 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f0e4406b020 0x7f0e4406d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:43.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.502+0000 7f0e685a8700 1 --2- 172.21.15.16:0/1194699192 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0e6015d900 0x7f0e600b7d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:43.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.502+0000 7f0e685a8700 1 --2- 172.21.15.16:0/1194699192 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e600d2040 0x7f0e601f3140 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:43.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.502+0000 7f0e685a8700 1 --2- 172.21.15.16:0/1194699192 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0e600d2f70 0x7f0e601534c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:43.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.502+0000 7f0e685a8700 1 -- 172.21.15.16:0/1194699192 >> 172.21.15.16:0/1194699192 conn(0x7f0e60072330 msgr2=0x7f0e600cdc40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:43.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.502+0000 7f0e685a8700 1 -- 172.21.15.16:0/1194699192 shutdown_connections 2023-10-16T12:26:43.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.502+0000 7f0e685a8700 1 -- 172.21.15.16:0/1194699192 wait complete. 2023-10-16T12:26:43.507 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 36 2023-10-16T12:26:43.517 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:26:43.644 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:26:43.643+0000 7f50d29c5200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:26:43.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.645+0000 7f04f10f1700 1 Processor -- start 2023-10-16T12:26:43.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.645+0000 7f04f10f1700 1 -- start start 2023-10-16T12:26:43.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.645+0000 7f04f10f1700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f04ec15d900 0x7f04ec0d2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:43.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.645+0000 7f04f10f1700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f04ec0ce9f0 0x7f04ec0cdc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:43.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.646+0000 7f04f10f1700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04ec0ce1a0 con 0x7f04ec0d2860 2023-10-16T12:26:43.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.646+0000 7f04f10f1700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04ec0cee70 con 0x7f04ec15d900 2023-10-16T12:26:43.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.646+0000 7f04f10f1700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04ec0ce400 con 0x7f04ec0ce9f0 2023-10-16T12:26:43.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.646+0000 7f04ead9d700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f04ec15d900 0x7f04ec0d2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:43.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.646+0000 7f04ea59c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f04ec0ce9f0 0x7f04ec0cdc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:43.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.646+0000 7f04ead9d700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f04ec15d900 0x7f04ec0d2040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:39228/0 (socket says 172.21.15.16:39228) 2023-10-16T12:26:43.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.646+0000 7f04ead9d700 1 -- 172.21.15.16:0/2987833717 learned_addr learned my addr 172.21.15.16:0/2987833717 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:43.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.646+0000 7f04ead9d700 1 -- 172.21.15.16:0/2987833717 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f04ec0ce9f0 msgr2=0x7f04ec0cdc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:43.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.646+0000 7f04ead9d700 1 --2- 172.21.15.16:0/2987833717 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f04ec0ce9f0 0x7f04ec0cdc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:43.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.646+0000 7f04ead9d700 1 -- 172.21.15.16:0/2987833717 >> v1:172.21.15.49:6789/0 conn(0x7f04ec0d2860 legacy=0x7f04ec0b5120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:43.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.646+0000 7f04ead9d700 1 -- 172.21.15.16:0/2987833717 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04ec15dce0 con 0x7f04ec15d900 2023-10-16T12:26:43.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.646+0000 7f04ea59c700 1 --2- 172.21.15.16:0/2987833717 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f04ec0ce9f0 0x7f04ec0cdc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:26:43.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.646+0000 7f04ead9d700 1 --2- 172.21.15.16:0/2987833717 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f04ec15d900 0x7f04ec0d2040 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f04dc0098f0 tx=0x7f04dc009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5eb39885c6f81079 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:43.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.646+0000 7f04e9d9b700 1 -- 172.21.15.16:0/2987833717 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04dc011830 con 0x7f04ec15d900 2023-10-16T12:26:43.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.647+0000 7f04e9d9b700 1 -- 172.21.15.16:0/2987833717 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04dc011990 con 0x7f04ec15d900 2023-10-16T12:26:43.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.647+0000 7f04e9d9b700 1 -- 172.21.15.16:0/2987833717 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04dc011d20 con 0x7f04ec15d900 2023-10-16T12:26:43.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.647+0000 7f04f10f1700 1 -- 172.21.15.16:0/2987833717 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f04ec15d900 msgr2=0x7f04ec0d2040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:43.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.647+0000 7f04f10f1700 1 --2- 172.21.15.16:0/2987833717 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f04ec15d900 0x7f04ec0d2040 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f04dc0098f0 tx=0x7f04dc009c00 comp rx=0 tx=0).stop 2023-10-16T12:26:43.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.647+0000 7f04f10f1700 1 -- 172.21.15.16:0/2987833717 shutdown_connections 2023-10-16T12:26:43.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.647+0000 7f04f10f1700 1 --2- 172.21.15.16:0/2987833717 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f04ec15d900 0x7f04ec0d2040 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:43.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.647+0000 7f04f10f1700 1 --2- 172.21.15.16:0/2987833717 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f04ec0ce9f0 0x7f04ec0cdc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:43.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.647+0000 7f04f10f1700 1 -- 172.21.15.16:0/2987833717 >> 172.21.15.16:0/2987833717 conn(0x7f04ec072330 msgr2=0x7f04ec072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:43.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.647+0000 7f04f10f1700 1 -- 172.21.15.16:0/2987833717 shutdown_connections 2023-10-16T12:26:43.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.647+0000 7f04f10f1700 1 -- 172.21.15.16:0/2987833717 wait complete. 2023-10-16T12:26:43.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.649+0000 7f04f10f1700 1 Processor -- start 2023-10-16T12:26:43.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.649+0000 7f04f10f1700 1 -- start start 2023-10-16T12:26:43.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.651+0000 7f04f10f1700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f04ec0ce9f0 0x7f04ec160860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:43.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.651+0000 7f04f10f1700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f04ec0d2860 0x7f04ec14f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:43.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.651+0000 7f04f10f1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04ec15d900 0x7f04ec1f6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:43.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.651+0000 7f04f10f1700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04ec160da0 con 0x7f04ec15d900 2023-10-16T12:26:43.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.651+0000 7f04f10f1700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04ec14fd90 con 0x7f04ec0d2860 2023-10-16T12:26:43.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.651+0000 7f04f10f1700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04ec150880 con 0x7f04ec0ce9f0 2023-10-16T12:26:43.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.651+0000 7f04eb59e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04ec15d900 0x7f04ec1f6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:43.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.651+0000 7f04ead9d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f04ec0ce9f0 0x7f04ec160860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:43.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.651+0000 7f04eb59e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04ec15d900 0x7f04ec1f6ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40176/0 (socket says 172.21.15.16:40176) 2023-10-16T12:26:43.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.651+0000 7f04ea59c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f04ec0d2860 0x7f04ec14f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:43.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.651+0000 7f04eb59e700 1 -- 172.21.15.16:0/2334980849 learned_addr learned my addr 172.21.15.16:0/2334980849 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:43.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.651+0000 7f04ead9d700 1 -- 172.21.15.16:0/2334980849 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f04ec0d2860 msgr2=0x7f04ec14f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:43.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.651+0000 7f04ead9d700 1 --2- 172.21.15.16:0/2334980849 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f04ec0d2860 0x7f04ec14f3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:43.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.651+0000 7f04ead9d700 1 -- 172.21.15.16:0/2334980849 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04ec15d900 msgr2=0x7f04ec1f6ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:43.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.651+0000 7f04ead9d700 1 --2- 172.21.15.16:0/2334980849 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04ec15d900 0x7f04ec1f6ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:43.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.651+0000 7f04ead9d700 1 -- 172.21.15.16:0/2334980849 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04e00094a0 con 0x7f04ec0ce9f0 2023-10-16T12:26:43.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.651+0000 7f04eb59e700 1 --2- 172.21.15.16:0/2334980849 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04ec15d900 0x7f04ec1f6ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:26:43.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.651+0000 7f04ea59c700 1 --2- 172.21.15.16:0/2334980849 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f04ec0d2860 0x7f04ec14f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:26:43.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.651+0000 7f04ead9d700 1 --2- 172.21.15.16:0/2334980849 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f04ec0ce9f0 0x7f04ec160860 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f04dc016000 tx=0x7f04dc004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:43.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.652+0000 7f04d37fe700 1 -- 172.21.15.16:0/2334980849 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04dc004590 con 0x7f04ec0ce9f0 2023-10-16T12:26:43.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.652+0000 7f04d37fe700 1 -- 172.21.15.16:0/2334980849 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04dc01e070 con 0x7f04ec0ce9f0 2023-10-16T12:26:43.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.652+0000 7f04f10f1700 1 -- 172.21.15.16:0/2334980849 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04dc009550 con 0x7f04ec0ce9f0 2023-10-16T12:26:43.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.652+0000 7f04d37fe700 1 -- 172.21.15.16:0/2334980849 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04dc0038a0 con 0x7f04ec0ce9f0 2023-10-16T12:26:43.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.652+0000 7f04f10f1700 1 -- 172.21.15.16:0/2334980849 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04ec1502c0 con 0x7f04ec0ce9f0 2023-10-16T12:26:43.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.653+0000 7f04f10f1700 1 -- 172.21.15.16:0/2334980849 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04ec154970 con 0x7f04ec0ce9f0 2023-10-16T12:26:43.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.653+0000 7f04d37fe700 1 -- 172.21.15.16:0/2334980849 <== mon.2 v2:172.21.15.49:3301/0 4 ==== mgrmap(e 37) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f04dc003a00 con 0x7f04ec0ce9f0 2023-10-16T12:26:43.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.653+0000 7f04d37fe700 1 --2- 172.21.15.16:0/2334980849 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f04d806b020 0x7f04d806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:43.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.653+0000 7f04ea59c700 1 -- 172.21.15.16:0/2334980849 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f04d806b020 msgr2=0x7f04d806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2230262559 2023-10-16T12:26:43.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.654+0000 7f04d37fe700 1 -- 172.21.15.16:0/2334980849 <== mon.2 v2:172.21.15.49:3301/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f04dc08a9f0 con 0x7f04ec0ce9f0 2023-10-16T12:26:43.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.654+0000 7f04ea59c700 1 --2- 172.21.15.16:0/2334980849 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f04d806b020 0x7f04d806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:26:43.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.658+0000 7f04d37fe700 1 -- 172.21.15.16:0/2334980849 <== mon.2 v2:172.21.15.49:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f04dc0556d0 con 0x7f04ec0ce9f0 2023-10-16T12:26:43.726 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:26:43.725+0000 7f50d29c5200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T12:26:43.726 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:43.725+0000 7f11cbf9c200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:26:43.810 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:43.808+0000 7f11cbf9c200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T12:26:43.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.855+0000 7f04ea59c700 1 -- 172.21.15.16:0/2334980849 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f04d806b020 msgr2=0x7f04d806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2230262559 2023-10-16T12:26:43.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.855+0000 7f04ea59c700 1 --2- 172.21.15.16:0/2334980849 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f04d806b020 0x7f04d806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:26:43.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.865+0000 7f04f10f1700 1 -- 172.21.15.16:0/2334980849 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f04ec04ebb0 con 0x7f04ec0ce9f0 2023-10-16T12:26:43.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.867+0000 7f04d37fe700 1 -- 172.21.15.16:0/2334980849 <== mon.2 v2:172.21.15.49:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+17151 (secure 0 0 0) 0x7f04dc020020 con 0x7f04ec0ce9f0 2023-10-16T12:26:43.869 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:26:43.869 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":86,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:26:38.750010+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:26:21.893734+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2023-10-16T12:26:22.642815+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/2700763633":"2023-10-17T12:26:38.749960+0000","172.21.15.16:0/2069262541":"2023-10-17T12:26:38.749960+0000","172.21.15.16:0/1444840585":"2023-10-17T12:26:38.749960+0000","172.21.15.16:6833/3415303009":"2023-10-17T12:26:38.749960+0000","172.21.15.16:6832/3415303009":"2023-10-17T12:26:38.749960+0000","172.21.15.16:0/779547724":"2023-10-17T12:26:38.749960+0000","172.21.15.16:0/4285863251":"2023-10-17T12:26:38.749960+0000","172.21.15.16:0/4057240133":"2023-10-17T12:26:38.749960+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:26:43.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.869+0000 7f04f10f1700 1 -- 172.21.15.16:0/2334980849 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f04d806b020 msgr2=0x7f04d806d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:26:43.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.869+0000 7f04f10f1700 1 --2- 172.21.15.16:0/2334980849 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f04d806b020 0x7f04d806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:43.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.869+0000 7f04f10f1700 1 -- 172.21.15.16:0/2334980849 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f04ec0ce9f0 msgr2=0x7f04ec160860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:43.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.869+0000 7f04f10f1700 1 --2- 172.21.15.16:0/2334980849 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f04ec0ce9f0 0x7f04ec160860 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f04dc016000 tx=0x7f04dc004750 comp rx=0 tx=0).stop 2023-10-16T12:26:43.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.869+0000 7f04f10f1700 1 -- 172.21.15.16:0/2334980849 shutdown_connections 2023-10-16T12:26:43.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.869+0000 7f04f10f1700 1 --2- 172.21.15.16:0/2334980849 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f04d806b020 0x7f04d806d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:43.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.869+0000 7f04f10f1700 1 --2- 172.21.15.16:0/2334980849 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04ec15d900 0x7f04ec1f6ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:43.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.869+0000 7f04f10f1700 1 --2- 172.21.15.16:0/2334980849 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f04ec0ce9f0 0x7f04ec160860 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:43.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.869+0000 7f04f10f1700 1 --2- 172.21.15.16:0/2334980849 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f04ec0d2860 0x7f04ec14f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:43.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.869+0000 7f04f10f1700 1 -- 172.21.15.16:0/2334980849 >> 172.21.15.16:0/2334980849 conn(0x7f04ec072330 msgr2=0x7f04ec0cab60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:43.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.869+0000 7f04f10f1700 1 -- 172.21.15.16:0/2334980849 shutdown_connections 2023-10-16T12:26:43.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:43.869+0000 7f04f10f1700 1 -- 172.21.15.16:0/2334980849 wait complete. 2023-10-16T12:26:43.875 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:26:43.874+0000 7f50d29c5200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:26:43.883 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T12:26:21.893734+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T12:26:22.642815+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T12:26:43.884 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:26:43.965 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:43.963+0000 7f11cbf9c200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:26:44.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.011+0000 7f7e57f72700 1 Processor -- start 2023-10-16T12:26:44.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.011+0000 7f7e57f72700 1 -- start start 2023-10-16T12:26:44.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.012+0000 7f7e57f72700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f7e5015ec80 0x7f7e500cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:44.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.012+0000 7f7e57f72700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7e5015f060 0x7f7e500ce400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:44.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.012+0000 7f7e57f72700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e500b2440 con 0x7f7e500d2110 2023-10-16T12:26:44.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.012+0000 7f7e57f72700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e500b2c50 con 0x7f7e5015ec80 2023-10-16T12:26:44.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.012+0000 7f7e57f72700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e500d00e0 con 0x7f7e5015f060 2023-10-16T12:26:44.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.012+0000 7f7e5550d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7e5015f060 0x7f7e500ce400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:44.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.012+0000 7f7e55d0e700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f7e5015ec80 0x7f7e500cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:44.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.012+0000 7f7e55d0e700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f7e5015ec80 0x7f7e500cdec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:39288/0 (socket says 172.21.15.16:39288) 2023-10-16T12:26:44.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.012+0000 7f7e55d0e700 1 -- 172.21.15.16:0/1174645880 learned_addr learned my addr 172.21.15.16:0/1174645880 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:44.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.012+0000 7f7e5550d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7e5015f060 0x7f7e500ce400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:56716/0 (socket says 172.21.15.16:56716) 2023-10-16T12:26:44.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.012+0000 7f7e55d0e700 1 -- 172.21.15.16:0/1174645880 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7e5015f060 msgr2=0x7f7e500ce400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:44.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.012+0000 7f7e55d0e700 1 --2- 172.21.15.16:0/1174645880 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7e5015f060 0x7f7e500ce400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:44.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.012+0000 7f7e55d0e700 1 -- 172.21.15.16:0/1174645880 >> v1:172.21.15.49:6789/0 conn(0x7f7e500d2110 legacy=0x7f7e500b4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:44.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.012+0000 7f7e54d0c700 1 -- 172.21.15.16:0/1174645880 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 138056032 0 0) 0x7f7e500b2440 con 0x7f7e500d2110 2023-10-16T12:26:44.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.012+0000 7f7e55d0e700 1 -- 172.21.15.16:0/1174645880 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e500cef60 con 0x7f7e5015ec80 2023-10-16T12:26:44.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.013+0000 7f7e5550d700 1 --2- 172.21.15.16:0/1174645880 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7e5015f060 0x7f7e500ce400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:26:44.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.013+0000 7f7e55d0e700 1 --2- 172.21.15.16:0/1174645880 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7e5015ec80 0x7f7e500cdec0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c0098f0 tx=0x7f7e4c009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2a28a88f7e02d686 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:44.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.013+0000 7f7e54d0c700 1 -- 172.21.15.16:0/1174645880 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e4c011830 con 0x7f7e5015ec80 2023-10-16T12:26:44.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.013+0000 7f7e54d0c700 1 -- 172.21.15.16:0/1174645880 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e4c011990 con 0x7f7e5015ec80 2023-10-16T12:26:44.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.013+0000 7f7e54d0c700 1 -- 172.21.15.16:0/1174645880 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e4c011d20 con 0x7f7e5015ec80 2023-10-16T12:26:44.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.013+0000 7f7e57f72700 1 -- 172.21.15.16:0/1174645880 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7e5015ec80 msgr2=0x7f7e500cdec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:44.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.013+0000 7f7e57f72700 1 --2- 172.21.15.16:0/1174645880 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7e5015ec80 0x7f7e500cdec0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c0098f0 tx=0x7f7e4c009c00 comp rx=0 tx=0).stop 2023-10-16T12:26:44.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.014+0000 7f7e57f72700 1 -- 172.21.15.16:0/1174645880 shutdown_connections 2023-10-16T12:26:44.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.014+0000 7f7e57f72700 1 --2- 172.21.15.16:0/1174645880 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7e5015ec80 0x7f7e500cdec0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:44.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.014+0000 7f7e57f72700 1 --2- 172.21.15.16:0/1174645880 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7e5015f060 0x7f7e500ce400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:44.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.014+0000 7f7e57f72700 1 -- 172.21.15.16:0/1174645880 >> 172.21.15.16:0/1174645880 conn(0x7f7e500d15f0 msgr2=0x7f7e500d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:44.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.014+0000 7f7e57f72700 1 -- 172.21.15.16:0/1174645880 shutdown_connections 2023-10-16T12:26:44.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.014+0000 7f7e57f72700 1 -- 172.21.15.16:0/1174645880 wait complete. 2023-10-16T12:26:44.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.015+0000 7f7e57f72700 1 Processor -- start 2023-10-16T12:26:44.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.015+0000 7f7e57f72700 1 -- start start 2023-10-16T12:26:44.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.016+0000 7f7e57f72700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7e500d2110 0x7f7e501f6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:44.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.016+0000 7f7e57f72700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7e5015ec80 0x7f7e5015c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:44.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.016+0000 7f7e57f72700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e5015f060 0x7f7e501521b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:44.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.016+0000 7f7e57f72700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e500c6040 con 0x7f7e5015f060 2023-10-16T12:26:44.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.016+0000 7f7e57f72700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e501f7100 con 0x7f7e5015ec80 2023-10-16T12:26:44.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.016+0000 7f7e57f72700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e500c5430 con 0x7f7e500d2110 2023-10-16T12:26:44.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.016+0000 7f7e5650f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e5015f060 0x7f7e501521b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:44.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.016+0000 7f7e5550d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7e5015ec80 0x7f7e5015c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:44.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.016+0000 7f7e55d0e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7e500d2110 0x7f7e501f6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:44.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.016+0000 7f7e5650f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e5015f060 0x7f7e501521b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40236/0 (socket says 172.21.15.16:40236) 2023-10-16T12:26:44.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.016+0000 7f7e5650f700 1 -- 172.21.15.16:0/667458962 learned_addr learned my addr 172.21.15.16:0/667458962 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:44.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.016+0000 7f7e5650f700 1 -- 172.21.15.16:0/667458962 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7e500d2110 msgr2=0x7f7e501f6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:44.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.016+0000 7f7e5650f700 1 --2- 172.21.15.16:0/667458962 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7e500d2110 0x7f7e501f6bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:44.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.016+0000 7f7e5650f700 1 -- 172.21.15.16:0/667458962 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7e5015ec80 msgr2=0x7f7e5015c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:44.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.016+0000 7f7e5650f700 1 --2- 172.21.15.16:0/667458962 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7e5015ec80 0x7f7e5015c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:44.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.016+0000 7f7e5650f700 1 -- 172.21.15.16:0/667458962 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e400094a0 con 0x7f7e5015f060 2023-10-16T12:26:44.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.016+0000 7f7e5650f700 1 --2- 172.21.15.16:0/667458962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e5015f060 0x7f7e501521b0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f7e4800c810 tx=0x7f7e4800cb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:44.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.016+0000 7f7e467fc700 1 -- 172.21.15.16:0/667458962 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e48003e30 con 0x7f7e5015f060 2023-10-16T12:26:44.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.016+0000 7f7e467fc700 1 -- 172.21.15.16:0/667458962 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e48005070 con 0x7f7e5015f060 2023-10-16T12:26:44.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.016+0000 7f7e467fc700 1 -- 172.21.15.16:0/667458962 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e48005420 con 0x7f7e5015f060 2023-10-16T12:26:44.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.016+0000 7f7e57f72700 1 -- 172.21.15.16:0/667458962 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e4c009550 con 0x7f7e5015f060 2023-10-16T12:26:44.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.017+0000 7f7e57f72700 1 -- 172.21.15.16:0/667458962 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e500c59f0 con 0x7f7e5015f060 2023-10-16T12:26:44.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.018+0000 7f7e57f72700 1 -- 172.21.15.16:0/667458962 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e500c61f0 con 0x7f7e5015f060 2023-10-16T12:26:44.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.018+0000 7f7e467fc700 1 -- 172.21.15.16:0/667458962 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 37) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f7e48005580 con 0x7f7e5015f060 2023-10-16T12:26:44.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.019+0000 7f7e467fc700 1 --2- 172.21.15.16:0/667458962 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f7e3406b020 0x7f7e3406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:44.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.019+0000 7f7e55d0e700 1 -- 172.21.15.16:0/667458962 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f7e3406b020 msgr2=0x7f7e3406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2230262559 2023-10-16T12:26:44.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.019+0000 7f7e467fc700 1 -- 172.21.15.16:0/667458962 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f7e48089600 con 0x7f7e5015f060 2023-10-16T12:26:44.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.019+0000 7f7e55d0e700 1 --2- 172.21.15.16:0/667458962 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f7e3406b020 0x7f7e3406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:26:44.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.023+0000 7f7e467fc700 1 -- 172.21.15.16:0/667458962 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7e500c61f0 con 0x7f7e5015f060 2023-10-16T12:26:44.111 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:26:44.110+0000 7f50d29c5200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T12:26:44.187 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:26:44.186+0000 7f50d29c5200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T12:26:44.209 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:44.208+0000 7f11cbf9c200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T12:26:44.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.220+0000 7f7e55d0e700 1 -- 172.21.15.16:0/667458962 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f7e3406b020 msgr2=0x7f7e3406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2230262559 2023-10-16T12:26:44.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.220+0000 7f7e55d0e700 1 --2- 172.21.15.16:0/667458962 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f7e3406b020 0x7f7e3406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:26:44.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.226+0000 7f7e57f72700 1 -- 172.21.15.16:0/667458962 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7e5004ebb0 con 0x7f7e5015f060 2023-10-16T12:26:44.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.226+0000 7f7e467fc700 1 -- 172.21.15.16:0/667458962 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v86) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7e480542e0 con 0x7f7e5015f060 2023-10-16T12:26:44.229 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T12:26:44.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.229+0000 7f7e57f72700 1 -- 172.21.15.16:0/667458962 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f7e3406b020 msgr2=0x7f7e3406d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:26:44.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.229+0000 7f7e57f72700 1 --2- 172.21.15.16:0/667458962 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f7e3406b020 0x7f7e3406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:44.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.229+0000 7f7e57f72700 1 -- 172.21.15.16:0/667458962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e5015f060 msgr2=0x7f7e501521b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:44.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.229+0000 7f7e57f72700 1 --2- 172.21.15.16:0/667458962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e5015f060 0x7f7e501521b0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f7e4800c810 tx=0x7f7e4800cb20 comp rx=0 tx=0).stop 2023-10-16T12:26:44.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.229+0000 7f7e57f72700 1 -- 172.21.15.16:0/667458962 shutdown_connections 2023-10-16T12:26:44.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.229+0000 7f7e57f72700 1 --2- 172.21.15.16:0/667458962 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f7e3406b020 0x7f7e3406d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:44.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.229+0000 7f7e57f72700 1 --2- 172.21.15.16:0/667458962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e5015f060 0x7f7e501521b0 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:44.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.229+0000 7f7e57f72700 1 --2- 172.21.15.16:0/667458962 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7e500d2110 0x7f7e501f6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:44.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.229+0000 7f7e57f72700 1 --2- 172.21.15.16:0/667458962 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7e5015ec80 0x7f7e5015c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:44.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.229+0000 7f7e57f72700 1 -- 172.21.15.16:0/667458962 >> 172.21.15.16:0/667458962 conn(0x7f7e500d15f0 msgr2=0x7f7e500b6c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:44.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.229+0000 7f7e57f72700 1 -- 172.21.15.16:0/667458962 shutdown_connections 2023-10-16T12:26:44.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.229+0000 7f7e57f72700 1 -- 172.21.15.16:0/667458962 wait complete. 2023-10-16T12:26:44.243 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T12:26:44.291 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:44.289+0000 7f11cbf9c200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T12:26:44.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.373+0000 7f1de4d10700 1 Processor -- start 2023-10-16T12:26:44.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.373+0000 7f1de4d10700 1 -- start start 2023-10-16T12:26:44.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.374+0000 7f1de4d10700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f1de015d8d0 0x7f1de00b2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:44.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.374+0000 7f1de4d10700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1de00ce350 0x7f1de00ce780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:44.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.374+0000 7f1de4d10700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1de00cecc0 con 0x7f1de00b8fa0 2023-10-16T12:26:44.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.374+0000 7f1de4d10700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1de00d0bb0 con 0x7f1de015d8d0 2023-10-16T12:26:44.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.374+0000 7f1de4d10700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1de015dcb0 con 0x7f1de00ce350 2023-10-16T12:26:44.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.374+0000 7f1dde59c700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f1de015d8d0 0x7f1de00b2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:44.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.374+0000 7f1dded9d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1de00ce350 0x7f1de00ce780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:44.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.375+0000 7f1dde59c700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f1de015d8d0 0x7f1de00b2830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:39304/0 (socket says 172.21.15.16:39304) 2023-10-16T12:26:44.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.375+0000 7f1dde59c700 1 -- 172.21.15.16:0/680944964 learned_addr learned my addr 172.21.15.16:0/680944964 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:44.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.375+0000 7f1dde59c700 1 -- 172.21.15.16:0/680944964 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1de00ce350 msgr2=0x7f1de00ce780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:44.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.375+0000 7f1dde59c700 1 --2- 172.21.15.16:0/680944964 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1de00ce350 0x7f1de00ce780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:44.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.375+0000 7f1dde59c700 1 -- 172.21.15.16:0/680944964 >> v1:172.21.15.49:6789/0 conn(0x7f1de00b8fa0 legacy=0x7f1de00b6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:44.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.375+0000 7f1dde59c700 1 -- 172.21.15.16:0/680944964 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1de00b2f30 con 0x7f1de015d8d0 2023-10-16T12:26:44.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.375+0000 7f1dded9d700 1 --2- 172.21.15.16:0/680944964 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1de00ce350 0x7f1de00ce780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:26:44.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.375+0000 7f1ddd59a700 1 -- 172.21.15.16:0/680944964 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1827006196 0 0) 0x7f1de00cecc0 con 0x7f1de00b8fa0 2023-10-16T12:26:44.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.375+0000 7f1dde59c700 1 --2- 172.21.15.16:0/680944964 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1de015d8d0 0x7f1de00b2830 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f1dd00098f0 tx=0x7f1dd0009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c4eee3e02bacd93e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:44.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.375+0000 7f1ddd59a700 1 -- 172.21.15.16:0/680944964 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd0011830 con 0x7f1de015d8d0 2023-10-16T12:26:44.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.375+0000 7f1ddd59a700 1 -- 172.21.15.16:0/680944964 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dd0011990 con 0x7f1de015d8d0 2023-10-16T12:26:44.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.375+0000 7f1ddd59a700 1 -- 172.21.15.16:0/680944964 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd0011d20 con 0x7f1de015d8d0 2023-10-16T12:26:44.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.376+0000 7f1de4d10700 1 -- 172.21.15.16:0/680944964 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1de015d8d0 msgr2=0x7f1de00b2830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:44.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.376+0000 7f1de4d10700 1 --2- 172.21.15.16:0/680944964 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1de015d8d0 0x7f1de00b2830 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f1dd00098f0 tx=0x7f1dd0009c00 comp rx=0 tx=0).stop 2023-10-16T12:26:44.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.376+0000 7f1de4d10700 1 -- 172.21.15.16:0/680944964 shutdown_connections 2023-10-16T12:26:44.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.376+0000 7f1de4d10700 1 --2- 172.21.15.16:0/680944964 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1de00ce350 0x7f1de00ce780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:44.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.376+0000 7f1de4d10700 1 --2- 172.21.15.16:0/680944964 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1de015d8d0 0x7f1de00b2830 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:44.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.376+0000 7f1de4d10700 1 -- 172.21.15.16:0/680944964 >> 172.21.15.16:0/680944964 conn(0x7f1de0072330 msgr2=0x7f1de0072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:44.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.376+0000 7f1de4d10700 1 -- 172.21.15.16:0/680944964 shutdown_connections 2023-10-16T12:26:44.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.376+0000 7f1de4d10700 1 -- 172.21.15.16:0/680944964 wait complete. 2023-10-16T12:26:44.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.378+0000 7f1de4d10700 1 Processor -- start 2023-10-16T12:26:44.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.378+0000 7f1de4d10700 1 -- start start 2023-10-16T12:26:44.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.378+0000 7f1de4d10700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1de00b8fa0 0x7f1de015ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:44.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.378+0000 7f1de4d10700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1de00ce350 0x7f1de00c7040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:44.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.378+0000 7f1de4d10700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1de015d8d0 0x7f1de00c7580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:44.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.378+0000 7f1de4d10700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1de00c8830 con 0x7f1de015d8d0 2023-10-16T12:26:44.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.378+0000 7f1de4d10700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1de00d3210 con 0x7f1de00ce350 2023-10-16T12:26:44.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.378+0000 7f1de4d10700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1de00c7bb0 con 0x7f1de00b8fa0 2023-10-16T12:26:44.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.379+0000 7f1dde59c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1de00b8fa0 0x7f1de015ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:44.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.379+0000 7f1dded9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1de015d8d0 0x7f1de00c7580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:44.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.379+0000 7f1dddd9b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1de00ce350 0x7f1de00c7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:44.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.379+0000 7f1dded9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1de015d8d0 0x7f1de00c7580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40250/0 (socket says 172.21.15.16:40250) 2023-10-16T12:26:44.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.379+0000 7f1dde59c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1de00b8fa0 0x7f1de015ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:56736/0 (socket says 172.21.15.16:56736) 2023-10-16T12:26:44.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.379+0000 7f1dded9d700 1 -- 172.21.15.16:0/830374742 learned_addr learned my addr 172.21.15.16:0/830374742 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:44.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.379+0000 7f1dddd9b700 1 -- 172.21.15.16:0/830374742 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1de00b8fa0 msgr2=0x7f1de015ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:44.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.379+0000 7f1dddd9b700 1 --2- 172.21.15.16:0/830374742 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1de00b8fa0 0x7f1de015ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:44.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.379+0000 7f1dddd9b700 1 -- 172.21.15.16:0/830374742 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1de015d8d0 msgr2=0x7f1de00c7580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:44.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.379+0000 7f1dddd9b700 1 --2- 172.21.15.16:0/830374742 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1de015d8d0 0x7f1de00c7580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:44.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.379+0000 7f1dddd9b700 1 -- 172.21.15.16:0/830374742 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dd8009a70 con 0x7f1de00ce350 2023-10-16T12:26:44.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.379+0000 7f1dde59c700 1 --2- 172.21.15.16:0/830374742 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1de00b8fa0 0x7f1de015ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:26:44.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.379+0000 7f1dded9d700 1 --2- 172.21.15.16:0/830374742 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1de015d8d0 0x7f1de00c7580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:26:44.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.379+0000 7f1dddd9b700 1 --2- 172.21.15.16:0/830374742 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1de00ce350 0x7f1de00c7040 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f1dd400c3e0 tx=0x7f1dd400c6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:44.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.379+0000 7f1dceffd700 1 -- 172.21.15.16:0/830374742 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd40052d0 con 0x7f1de00ce350 2023-10-16T12:26:44.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.379+0000 7f1dceffd700 1 -- 172.21.15.16:0/830374742 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dd4005430 con 0x7f1de00ce350 2023-10-16T12:26:44.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.379+0000 7f1de4d10700 1 -- 172.21.15.16:0/830374742 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dd0009550 con 0x7f1de00ce350 2023-10-16T12:26:44.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.379+0000 7f1dceffd700 1 -- 172.21.15.16:0/830374742 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd40038a0 con 0x7f1de00ce350 2023-10-16T12:26:44.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.379+0000 7f1de4d10700 1 -- 172.21.15.16:0/830374742 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1de00c8170 con 0x7f1de00ce350 2023-10-16T12:26:44.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.380+0000 7f1de4d10700 1 -- 172.21.15.16:0/830374742 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1de00c8970 con 0x7f1de00ce350 2023-10-16T12:26:44.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.381+0000 7f1dceffd700 1 -- 172.21.15.16:0/830374742 <== mon.1 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 37) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f1dd4003a00 con 0x7f1de00ce350 2023-10-16T12:26:44.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.381+0000 7f1dceffd700 1 --2- 172.21.15.16:0/830374742 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f1dc006b020 0x7f1dc006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:44.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.381+0000 7f1dde59c700 1 -- 172.21.15.16:0/830374742 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f1dc006b020 msgr2=0x7f1dc006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2230262559 2023-10-16T12:26:44.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.381+0000 7f1dceffd700 1 -- 172.21.15.16:0/830374742 <== mon.1 v2:172.21.15.49:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f1dd4088d00 con 0x7f1de00ce350 2023-10-16T12:26:44.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.381+0000 7f1dde59c700 1 --2- 172.21.15.16:0/830374742 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f1dc006b020 0x7f1dc006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:26:44.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.386+0000 7f1dceffd700 1 -- 172.21.15.16:0/830374742 <== mon.1 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1de00c8970 con 0x7f1de00ce350 2023-10-16T12:26:44.463 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:26:44.461+0000 7f50d29c5200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T12:26:44.572 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:44.570+0000 7f11cbf9c200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T12:26:44.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.582+0000 7f1dde59c700 1 -- 172.21.15.16:0/830374742 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f1dc006b020 msgr2=0x7f1dc006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2230262559 2023-10-16T12:26:44.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.582+0000 7f1dde59c700 1 --2- 172.21.15.16:0/830374742 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f1dc006b020 0x7f1dc006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:26:44.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.589+0000 7f1de4d10700 1 -- 172.21.15.16:0/830374742 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1de00c8970 con 0x7f1de00ce350 2023-10-16T12:26:44.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.590+0000 7f1dceffd700 1 -- 172.21.15.16:0/830374742 <== mon.1 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v86) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1de00c8970 con 0x7f1de00ce350 2023-10-16T12:26:44.592 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-10-16T12:26:44.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.592+0000 7f1de4d10700 1 -- 172.21.15.16:0/830374742 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f1dc006b020 msgr2=0x7f1dc006d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:26:44.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.592+0000 7f1de4d10700 1 --2- 172.21.15.16:0/830374742 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f1dc006b020 0x7f1dc006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:44.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.592+0000 7f1de4d10700 1 -- 172.21.15.16:0/830374742 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1de00ce350 msgr2=0x7f1de00c7040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:44.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.592+0000 7f1de4d10700 1 --2- 172.21.15.16:0/830374742 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1de00ce350 0x7f1de00c7040 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f1dd400c3e0 tx=0x7f1dd400c6f0 comp rx=0 tx=0).stop 2023-10-16T12:26:44.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.592+0000 7f1de4d10700 1 -- 172.21.15.16:0/830374742 shutdown_connections 2023-10-16T12:26:44.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.592+0000 7f1de4d10700 1 --2- 172.21.15.16:0/830374742 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f1dc006b020 0x7f1dc006d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:44.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.592+0000 7f1de4d10700 1 --2- 172.21.15.16:0/830374742 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1de015d8d0 0x7f1de00c7580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:44.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.592+0000 7f1de4d10700 1 --2- 172.21.15.16:0/830374742 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1de00b8fa0 0x7f1de015ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:44.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.592+0000 7f1de4d10700 1 --2- 172.21.15.16:0/830374742 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1de00ce350 0x7f1de00c7040 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:44.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.592+0000 7f1de4d10700 1 -- 172.21.15.16:0/830374742 >> 172.21.15.16:0/830374742 conn(0x7f1de0072330 msgr2=0x7f1de015e1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:44.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.592+0000 7f1de4d10700 1 -- 172.21.15.16:0/830374742 shutdown_connections 2023-10-16T12:26:44.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.592+0000 7f1de4d10700 1 -- 172.21.15.16:0/830374742 wait complete. 2023-10-16T12:26:44.606 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T12:26:44.692 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:26:44.691+0000 7f50d29c5200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T12:26:44.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.733+0000 7f03020c5700 1 Processor -- start 2023-10-16T12:26:44.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.733+0000 7f03020c5700 1 -- start start 2023-10-16T12:26:44.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.734+0000 7f03020c5700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f02fc0ce6a0 0x7f02fc06c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:44.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.734+0000 7f03020c5700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f02fc0d2110 0x7f02fc06c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:44.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.734+0000 7f03020c5700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02fc0b20f0 con 0x7f02fc0b2620 2023-10-16T12:26:44.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.734+0000 7f03020c5700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02fc0af1f0 con 0x7f02fc0d2110 2023-10-16T12:26:44.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.734+0000 7f03020c5700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02fc0d03d0 con 0x7f02fc0ce6a0 2023-10-16T12:26:44.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.734+0000 7f02fbfff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f02fc0b2620 0x7f02fc0b0f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57766/0 (socket says 172.21.15.16:57766) 2023-10-16T12:26:44.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.734+0000 7f02fbfff700 1 -- 172.21.15.16:0/4025938845 learned_addr learned my addr 172.21.15.16:0/4025938845 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:26:44.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.734+0000 7f02fb7fe700 1 --2- 172.21.15.16:0/4025938845 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f02fc0ce6a0 0x7f02fc06c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:44.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.734+0000 7f02faffd700 1 --2- 172.21.15.16:0/4025938845 >> v2:172.21.15.49:3300/0 conn(0x7f02fc0d2110 0x7f02fc06c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:44.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.734+0000 7f02fa7fc700 1 -- 172.21.15.16:0/4025938845 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4269206700 0 0) 0x7f02fc0b20f0 con 0x7f02fc0b2620 2023-10-16T12:26:44.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.734+0000 7f02fa7fc700 1 -- 172.21.15.16:0/4025938845 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02e80035b0 con 0x7f02fc0b2620 2023-10-16T12:26:44.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.734+0000 7f02fb7fe700 1 -- 172.21.15.16:0/4025938845 >> v2:172.21.15.49:3300/0 conn(0x7f02fc0d2110 msgr2=0x7f02fc06c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:44.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.734+0000 7f02fb7fe700 1 --2- 172.21.15.16:0/4025938845 >> v2:172.21.15.49:3300/0 conn(0x7f02fc0d2110 0x7f02fc06c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:44.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.734+0000 7f02fb7fe700 1 -- 172.21.15.16:0/4025938845 >> v1:172.21.15.16:6789/0 conn(0x7f02fc0b2620 legacy=0x7f02fc0b0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:44.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.734+0000 7f02fa7fc700 1 -- 172.21.15.16:0/4025938845 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 183371513 0 0) 0x7f02e80035b0 con 0x7f02fc0b2620 2023-10-16T12:26:44.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.734+0000 7f02fb7fe700 1 -- 172.21.15.16:0/4025938845 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02fc0b2340 con 0x7f02fc0ce6a0 2023-10-16T12:26:44.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.734+0000 7f02faffd700 1 --2- 172.21.15.16:0/4025938845 >> v2:172.21.15.49:3300/0 conn(0x7f02fc0d2110 0x7f02fc06c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:26:44.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.735+0000 7f02fb7fe700 1 --2- 172.21.15.16:0/4025938845 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f02fc0ce6a0 0x7f02fc06c300 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f02ec009d90 tx=0x7f02ec00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cabdc1cc3508c96b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:44.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.735+0000 7f02fa7fc700 1 -- 172.21.15.16:0/4025938845 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02ec00e070 con 0x7f02fc0ce6a0 2023-10-16T12:26:44.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.735+0000 7f02fa7fc700 1 -- 172.21.15.16:0/4025938845 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02ec003a20 con 0x7f02fc0ce6a0 2023-10-16T12:26:44.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.735+0000 7f02fa7fc700 1 -- 172.21.15.16:0/4025938845 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02f4001670 con 0x7f02fc0ce6a0 2023-10-16T12:26:44.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.735+0000 7f03020c5700 1 -- 172.21.15.16:0/4025938845 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f02fc0ce6a0 msgr2=0x7f02fc06c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:44.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.735+0000 7f03020c5700 1 --2- 172.21.15.16:0/4025938845 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f02fc0ce6a0 0x7f02fc06c300 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f02ec009d90 tx=0x7f02ec00b330 comp rx=0 tx=0).stop 2023-10-16T12:26:44.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.735+0000 7f03020c5700 1 -- 172.21.15.16:0/4025938845 shutdown_connections 2023-10-16T12:26:44.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.735+0000 7f03020c5700 1 --2- 172.21.15.16:0/4025938845 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f02fc0ce6a0 0x7f02fc06c300 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:44.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.735+0000 7f03020c5700 1 --2- 172.21.15.16:0/4025938845 >> v2:172.21.15.49:3300/0 conn(0x7f02fc0d2110 0x7f02fc06c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:44.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.735+0000 7f03020c5700 1 -- 172.21.15.16:0/4025938845 >> 172.21.15.16:0/4025938845 conn(0x7f02fc0d15f0 msgr2=0x7f02fc0d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:44.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.736+0000 7f03020c5700 1 -- 172.21.15.16:0/4025938845 shutdown_connections 2023-10-16T12:26:44.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.736+0000 7f03020c5700 1 -- 172.21.15.16:0/4025938845 wait complete. 2023-10-16T12:26:44.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.737+0000 7f03020c5700 1 Processor -- start 2023-10-16T12:26:44.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.737+0000 7f03020c5700 1 -- start start 2023-10-16T12:26:44.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.738+0000 7f03020c5700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f02fc0b2620 0x7f02fc15dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:44.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.738+0000 7f03020c5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02fc0ce6a0 0x7f02fc09ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:44.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.738+0000 7f03020c5700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f02fc0d2110 0x7f02fc0b0380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:44.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.738+0000 7f03020c5700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02fc09dc80 con 0x7f02fc0ce6a0 2023-10-16T12:26:44.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.738+0000 7f03020c5700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02fc09df60 con 0x7f02fc0d2110 2023-10-16T12:26:44.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.738+0000 7f03020c5700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02fc09e240 con 0x7f02fc0b2620 2023-10-16T12:26:44.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.738+0000 7f02faffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02fc0ce6a0 0x7f02fc09ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:44.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.738+0000 7f02fb7fe700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f02fc0b2620 0x7f02fc15dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:44.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.738+0000 7f02fb7fe700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f02fc0b2620 0x7f02fc15dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:56752/0 (socket says 172.21.15.16:56752) 2023-10-16T12:26:44.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.739+0000 7f02fb7fe700 1 -- 172.21.15.16:0/1722424465 learned_addr learned my addr 172.21.15.16:0/1722424465 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:44.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.738+0000 7f02faffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02fc0ce6a0 0x7f02fc09ed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40264/0 (socket says 172.21.15.16:40264) 2023-10-16T12:26:44.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.739+0000 7f02fbfff700 1 --2- 172.21.15.16:0/1722424465 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f02fc0d2110 0x7f02fc0b0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:44.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.739+0000 7f02faffd700 1 -- 172.21.15.16:0/1722424465 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f02fc0b2620 msgr2=0x7f02fc15dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:44.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.739+0000 7f02faffd700 1 --2- 172.21.15.16:0/1722424465 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f02fc0b2620 0x7f02fc15dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:44.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.739+0000 7f02faffd700 1 -- 172.21.15.16:0/1722424465 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f02fc0d2110 msgr2=0x7f02fc0b0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:44.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.739+0000 7f02faffd700 1 --2- 172.21.15.16:0/1722424465 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f02fc0d2110 0x7f02fc0b0380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:44.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.739+0000 7f02faffd700 1 -- 172.21.15.16:0/1722424465 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02e40094a0 con 0x7f02fc0ce6a0 2023-10-16T12:26:44.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.739+0000 7f02fb7fe700 1 --2- 172.21.15.16:0/1722424465 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f02fc0b2620 0x7f02fc15dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:26:44.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.739+0000 7f02faffd700 1 --2- 172.21.15.16:0/1722424465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02fc0ce6a0 0x7f02fc09ed20 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f02e400e900 tx=0x7f02e400ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:44.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.739+0000 7f02f3fff700 1 -- 172.21.15.16:0/1722424465 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02e4013680 con 0x7f02fc0ce6a0 2023-10-16T12:26:44.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.739+0000 7f02f3fff700 1 -- 172.21.15.16:0/1722424465 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02e40137e0 con 0x7f02fc0ce6a0 2023-10-16T12:26:44.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.739+0000 7f02f3fff700 1 -- 172.21.15.16:0/1722424465 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02e4013b70 con 0x7f02fc0ce6a0 2023-10-16T12:26:44.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.739+0000 7f03020c5700 1 -- 172.21.15.16:0/1722424465 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02ec009a70 con 0x7f02fc0ce6a0 2023-10-16T12:26:44.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.739+0000 7f03020c5700 1 -- 172.21.15.16:0/1722424465 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02fc09e800 con 0x7f02fc0ce6a0 2023-10-16T12:26:44.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.740+0000 7f03020c5700 1 -- 172.21.15.16:0/1722424465 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02fc09ea20 con 0x7f02fc0ce6a0 2023-10-16T12:26:44.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.741+0000 7f02f3fff700 1 -- 172.21.15.16:0/1722424465 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 37) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f02e4013cd0 con 0x7f02fc0ce6a0 2023-10-16T12:26:44.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.741+0000 7f02f3fff700 1 --2- 172.21.15.16:0/1722424465 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f02dc06b020 0x7f02dc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:44.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.741+0000 7f02fb7fe700 1 -- 172.21.15.16:0/1722424465 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f02dc06b020 msgr2=0x7f02dc06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2230262559 2023-10-16T12:26:44.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.741+0000 7f02f3fff700 1 -- 172.21.15.16:0/1722424465 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f02e4089890 con 0x7f02fc0ce6a0 2023-10-16T12:26:44.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.741+0000 7f02fb7fe700 1 --2- 172.21.15.16:0/1722424465 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f02dc06b020 0x7f02dc06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:26:44.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.746+0000 7f02f3fff700 1 -- 172.21.15.16:0/1722424465 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f02e4054570 con 0x7f02fc0ce6a0 2023-10-16T12:26:44.813 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:44.811+0000 7f11cbf9c200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T12:26:44.848 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:26:44.847+0000 7f50d29c5200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T12:26:44.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.942+0000 7f02fb7fe700 1 -- 172.21.15.16:0/1722424465 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f02dc06b020 msgr2=0x7f02dc06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2230262559 2023-10-16T12:26:44.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.942+0000 7f02fb7fe700 1 --2- 172.21.15.16:0/1722424465 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f02dc06b020 0x7f02dc06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:26:44.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.951+0000 7f03020c5700 1 -- 172.21.15.16:0/1722424465 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f02fc04ebb0 con 0x7f02fc0ce6a0 2023-10-16T12:26:44.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.951+0000 7f02f3fff700 1 -- 172.21.15.16:0/1722424465 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v86) v1 ==== 100+0+11 (secure 0 0 0) 0x7f02e40210a0 con 0x7f02fc0ce6a0 2023-10-16T12:26:44.954 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-10-16T12:26:44.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.953+0000 7f03020c5700 1 -- 172.21.15.16:0/1722424465 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f02dc06b020 msgr2=0x7f02dc06d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:26:44.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.953+0000 7f03020c5700 1 --2- 172.21.15.16:0/1722424465 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f02dc06b020 0x7f02dc06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:44.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.953+0000 7f03020c5700 1 -- 172.21.15.16:0/1722424465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02fc0ce6a0 msgr2=0x7f02fc09ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:44.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.953+0000 7f03020c5700 1 --2- 172.21.15.16:0/1722424465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02fc0ce6a0 0x7f02fc09ed20 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f02e400e900 tx=0x7f02e400ec10 comp rx=0 tx=0).stop 2023-10-16T12:26:44.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.953+0000 7f03020c5700 1 -- 172.21.15.16:0/1722424465 shutdown_connections 2023-10-16T12:26:44.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.953+0000 7f03020c5700 1 --2- 172.21.15.16:0/1722424465 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f02dc06b020 0x7f02dc06d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:44.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.953+0000 7f03020c5700 1 --2- 172.21.15.16:0/1722424465 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f02fc0d2110 0x7f02fc0b0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:44.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.953+0000 7f03020c5700 1 --2- 172.21.15.16:0/1722424465 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f02fc0b2620 0x7f02fc15dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:44.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.953+0000 7f03020c5700 1 --2- 172.21.15.16:0/1722424465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02fc0ce6a0 0x7f02fc09ed20 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:44.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.953+0000 7f03020c5700 1 -- 172.21.15.16:0/1722424465 >> 172.21.15.16:0/1722424465 conn(0x7f02fc0d15f0 msgr2=0x7f02fc0ae1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:44.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.954+0000 7f03020c5700 1 -- 172.21.15.16:0/1722424465 shutdown_connections 2023-10-16T12:26:44.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:44.954+0000 7f03020c5700 1 -- 172.21.15.16:0/1722424465 wait complete. 2023-10-16T12:26:44.966 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:26:44.965+0000 7f50d29c5200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T12:26:44.968 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:26:44.972 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:44.971+0000 7f11cbf9c200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T12:26:45.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.093+0000 7fa0e0357700 1 Processor -- start 2023-10-16T12:26:45.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.093+0000 7fa0e0357700 1 -- start start 2023-10-16T12:26:45.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.094+0000 7fa0e0357700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fa0d815d900 0x7fa0d80d2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:45.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.094+0000 7fa0e0357700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0d80cee70 con 0x7fa0d80d2f70 2023-10-16T12:26:45.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.094+0000 7fa0e0357700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0d80ce400 con 0x7fa0d80ce9f0 2023-10-16T12:26:45.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.094+0000 7fa0e0357700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0d815dce0 con 0x7fa0d815d900 2023-10-16T12:26:45.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.094+0000 7fa0de8f4700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa0d80d2f70 0x7fa0d80b6790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57782/0 (socket says 172.21.15.16:57782) 2023-10-16T12:26:45.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.094+0000 7fa0de8f4700 1 -- 172.21.15.16:0/3630728710 learned_addr learned my addr 172.21.15.16:0/3630728710 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:26:45.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.094+0000 7fa0de0f3700 1 --2- 172.21.15.16:0/3630728710 >> v2:172.21.15.49:3300/0 conn(0x7fa0d815d900 0x7fa0d80d2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:45.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.094+0000 7fa0dd0f1700 1 -- 172.21.15.16:0/3630728710 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1286656442 0 0) 0x7fa0d80cee70 con 0x7fa0d80d2f70 2023-10-16T12:26:45.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.094+0000 7fa0dd0f1700 1 -- 172.21.15.16:0/3630728710 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0c40035b0 con 0x7fa0d80d2f70 2023-10-16T12:26:45.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.094+0000 7fa0de0f3700 1 -- 172.21.15.16:0/3630728710 >> v1:172.21.15.49:6789/0 conn(0x7fa0d80ce9f0 legacy=0x7fa0d80d2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:45.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.094+0000 7fa0de0f3700 1 -- 172.21.15.16:0/3630728710 >> v1:172.21.15.16:6789/0 conn(0x7fa0d80d2f70 legacy=0x7fa0d80b6790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:45.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.094+0000 7fa0de0f3700 1 -- 172.21.15.16:0/3630728710 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0d80d1280 con 0x7fa0d815d900 2023-10-16T12:26:45.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.095+0000 7fa0de0f3700 1 --2- 172.21.15.16:0/3630728710 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa0d815d900 0x7fa0d80d2040 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fa0d4009840 tx=0x7fa0d4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e5de00bc8c9069e9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:45.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.095+0000 7fa0dd0f1700 1 -- 172.21.15.16:0/3630728710 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0d4011830 con 0x7fa0d815d900 2023-10-16T12:26:45.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.095+0000 7fa0dd0f1700 1 -- 172.21.15.16:0/3630728710 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0d4011990 con 0x7fa0d815d900 2023-10-16T12:26:45.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.095+0000 7fa0dd0f1700 1 -- 172.21.15.16:0/3630728710 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0d4011d20 con 0x7fa0d815d900 2023-10-16T12:26:45.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.095+0000 7fa0e0357700 1 -- 172.21.15.16:0/3630728710 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa0d815d900 msgr2=0x7fa0d80d2040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:45.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.095+0000 7fa0e0357700 1 --2- 172.21.15.16:0/3630728710 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa0d815d900 0x7fa0d80d2040 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fa0d4009840 tx=0x7fa0d4009b50 comp rx=0 tx=0).stop 2023-10-16T12:26:45.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.096+0000 7fa0e0357700 1 -- 172.21.15.16:0/3630728710 shutdown_connections 2023-10-16T12:26:45.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.096+0000 7fa0e0357700 1 --2- 172.21.15.16:0/3630728710 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa0d815d900 0x7fa0d80d2040 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:45.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.096+0000 7fa0e0357700 1 -- 172.21.15.16:0/3630728710 >> 172.21.15.16:0/3630728710 conn(0x7fa0d8072330 msgr2=0x7fa0d8072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:45.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.096+0000 7fa0e0357700 1 -- 172.21.15.16:0/3630728710 shutdown_connections 2023-10-16T12:26:45.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.096+0000 7fa0e0357700 1 -- 172.21.15.16:0/3630728710 wait complete. 2023-10-16T12:26:45.103 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:45.096+0000 7f11cbf9c200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T12:26:45.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.097+0000 7fa0e0357700 1 Processor -- start 2023-10-16T12:26:45.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.097+0000 7fa0e0357700 1 -- start start 2023-10-16T12:26:45.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.098+0000 7fa0e0357700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa0d80ce9f0 0x7fa0d80cbce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:45.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.098+0000 7fa0e0357700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa0d80d2f70 0x7fa0d81565f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:45.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.098+0000 7fa0e0357700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0d815d900 0x7fa0d80c8d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:45.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.098+0000 7fa0e0357700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0d80c9270 con 0x7fa0d815d900 2023-10-16T12:26:45.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.098+0000 7fa0e0357700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0d80c9510 con 0x7fa0d80ce9f0 2023-10-16T12:26:45.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.098+0000 7fa0e0357700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0d80c97f0 con 0x7fa0d80d2f70 2023-10-16T12:26:45.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.098+0000 7fa0de8f4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0d815d900 0x7fa0d80c8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:45.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.098+0000 7fa0de8f4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0d815d900 0x7fa0d80c8d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40282/0 (socket says 172.21.15.16:40282) 2023-10-16T12:26:45.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.098+0000 7fa0de8f4700 1 -- 172.21.15.16:0/3029345261 learned_addr learned my addr 172.21.15.16:0/3029345261 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:45.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.098+0000 7fa0de0f3700 1 --2- 172.21.15.16:0/3029345261 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa0d80ce9f0 0x7fa0d80cbce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:45.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.098+0000 7fa0dd8f2700 1 --2- 172.21.15.16:0/3029345261 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa0d80d2f70 0x7fa0d81565f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:45.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.098+0000 7fa0de8f4700 1 -- 172.21.15.16:0/3029345261 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa0d80d2f70 msgr2=0x7fa0d81565f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:45.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.098+0000 7fa0de8f4700 1 --2- 172.21.15.16:0/3029345261 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa0d80d2f70 0x7fa0d81565f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:45.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.098+0000 7fa0de8f4700 1 -- 172.21.15.16:0/3029345261 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa0d80ce9f0 msgr2=0x7fa0d80cbce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:45.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.098+0000 7fa0de8f4700 1 --2- 172.21.15.16:0/3029345261 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa0d80ce9f0 0x7fa0d80cbce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:45.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.098+0000 7fa0de8f4700 1 -- 172.21.15.16:0/3029345261 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0d40094a0 con 0x7fa0d815d900 2023-10-16T12:26:45.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.098+0000 7fa0de8f4700 1 --2- 172.21.15.16:0/3029345261 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0d815d900 0x7fa0d80c8d30 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7fa0d000cbb0 tx=0x7fa0d000cec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:45.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.098+0000 7fa0ceffd700 1 -- 172.21.15.16:0/3029345261 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0d000a9e0 con 0x7fa0d815d900 2023-10-16T12:26:45.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.098+0000 7fa0ceffd700 1 -- 172.21.15.16:0/3029345261 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0d000ab40 con 0x7fa0d815d900 2023-10-16T12:26:45.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.098+0000 7fa0e0357700 1 -- 172.21.15.16:0/3029345261 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0d80c9ad0 con 0x7fa0d815d900 2023-10-16T12:26:45.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.098+0000 7fa0ceffd700 1 -- 172.21.15.16:0/3029345261 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0d0012400 con 0x7fa0d815d900 2023-10-16T12:26:45.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.099+0000 7fa0e0357700 1 -- 172.21.15.16:0/3029345261 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0d80ca090 con 0x7fa0d815d900 2023-10-16T12:26:45.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.100+0000 7fa0e0357700 1 -- 172.21.15.16:0/3029345261 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0d81f4220 con 0x7fa0d815d900 2023-10-16T12:26:45.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.100+0000 7fa0ceffd700 1 -- 172.21.15.16:0/3029345261 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 37) v1 ==== 89332+0+0 (secure 0 0 0) 0x7fa0d0012560 con 0x7fa0d815d900 2023-10-16T12:26:45.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.100+0000 7fa0ceffd700 1 --2- 172.21.15.16:0/3029345261 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7fa0bc06b020 0x7fa0bc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:45.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.100+0000 7fa0de0f3700 1 -- 172.21.15.16:0/3029345261 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7fa0bc06b020 msgr2=0x7fa0bc06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2230262559 2023-10-16T12:26:45.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.100+0000 7fa0ceffd700 1 -- 172.21.15.16:0/3029345261 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fa0d0089610 con 0x7fa0d815d900 2023-10-16T12:26:45.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.100+0000 7fa0de0f3700 1 --2- 172.21.15.16:0/3029345261 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7fa0bc06b020 0x7fa0bc06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:26:45.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.105+0000 7fa0ceffd700 1 -- 172.21.15.16:0/3029345261 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa0d81f4220 con 0x7fa0d815d900 2023-10-16T12:26:45.125 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:26:45.125+0000 7f50d29c5200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T12:26:45.264 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:45.262+0000 7f11cbf9c200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T12:26:45.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.301+0000 7fa0de0f3700 1 -- 172.21.15.16:0/3029345261 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7fa0bc06b020 msgr2=0x7fa0bc06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2230262559 2023-10-16T12:26:45.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.301+0000 7fa0de0f3700 1 --2- 172.21.15.16:0/3029345261 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7fa0bc06b020 0x7fa0bc06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:26:45.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.356+0000 7fa0e0357700 1 -- 172.21.15.16:0/3029345261 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa0d81f4220 con 0x7fa0d815d900 2023-10-16T12:26:45.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.357+0000 7fa0ceffd700 1 -- 172.21.15.16:0/3029345261 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+4911 (secure 0 0 0) 0x7fa0d81f4220 con 0x7fa0d815d900 2023-10-16T12:26:45.359 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:26:45.359 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":36,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5861,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/1236108916","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":1236108916},{"type":"v1","addr":"172.21.15.49:6834","nonce":1236108916}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31},{"gid":5864,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/2762335434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":2762335434},{"type":"v1","addr":"172.21.15.49:6835","nonce":2762335434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31},{"gid":6379,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/1502256436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1502256436},{"type":"v1","addr":"172.21.15.16:6836","nonce":1502256436}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:26:23.670609+0000","modified":"2023-10-16T12:26:26.710098+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6627},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6627":{"gid":6627,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/962243511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":962243511},{"type":"v1","addr":"172.21.15.16:6837","nonce":962243511}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-10-16T12:26:45.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.359+0000 7fa0e0357700 1 -- 172.21.15.16:0/3029345261 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7fa0bc06b020 msgr2=0x7fa0bc06d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:26:45.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.359+0000 7fa0e0357700 1 --2- 172.21.15.16:0/3029345261 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7fa0bc06b020 0x7fa0bc06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:45.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.359+0000 7fa0e0357700 1 -- 172.21.15.16:0/3029345261 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0d815d900 msgr2=0x7fa0d80c8d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:45.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.359+0000 7fa0e0357700 1 --2- 172.21.15.16:0/3029345261 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0d815d900 0x7fa0d80c8d30 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7fa0d000cbb0 tx=0x7fa0d000cec0 comp rx=0 tx=0).stop 2023-10-16T12:26:45.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.359+0000 7fa0e0357700 1 -- 172.21.15.16:0/3029345261 shutdown_connections 2023-10-16T12:26:45.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.359+0000 7fa0e0357700 1 --2- 172.21.15.16:0/3029345261 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7fa0bc06b020 0x7fa0bc06d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:45.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.359+0000 7fa0e0357700 1 --2- 172.21.15.16:0/3029345261 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0d815d900 0x7fa0d80c8d30 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:45.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.359+0000 7fa0e0357700 1 --2- 172.21.15.16:0/3029345261 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa0d80ce9f0 0x7fa0d80cbce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:45.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.359+0000 7fa0e0357700 1 --2- 172.21.15.16:0/3029345261 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa0d80d2f70 0x7fa0d81565f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:45.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.359+0000 7fa0e0357700 1 -- 172.21.15.16:0/3029345261 >> 172.21.15.16:0/3029345261 conn(0x7fa0d8072330 msgr2=0x7fa0d8154240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:45.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.359+0000 7fa0e0357700 1 -- 172.21.15.16:0/3029345261 shutdown_connections 2023-10-16T12:26:45.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.359+0000 7fa0e0357700 1 -- 172.21.15.16:0/3029345261 wait complete. 2023-10-16T12:26:45.364 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 36 2023-10-16T12:26:45.375 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:26:45.396 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:26:45.395+0000 7f50d29c5200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T12:26:45.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.539+0000 7f12526f7700 1 Processor -- start 2023-10-16T12:26:45.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.539+0000 7f12526f7700 1 -- start start 2023-10-16T12:26:45.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.540+0000 7f12526f7700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f124c15b630 0x7f124c0d0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:45.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.540+0000 7f12526f7700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f124c0b0970 con 0x7f124c0ce4f0 2023-10-16T12:26:45.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.540+0000 7f12526f7700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f124c0cd690 con 0x7f124c15ba10 2023-10-16T12:26:45.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.540+0000 7f12526f7700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f124c0cd970 con 0x7f124c15b630 2023-10-16T12:26:45.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.540+0000 7f1250c94700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f124c0ce4f0 0x7f124c0b1430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57796/0 (socket says 172.21.15.16:57796) 2023-10-16T12:26:45.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.540+0000 7f1250c94700 1 -- 172.21.15.16:0/1343552053 learned_addr learned my addr 172.21.15.16:0/1343552053 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:26:45.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.541+0000 7f124bfff700 1 --2- 172.21.15.16:0/1343552053 >> v2:172.21.15.49:3300/0 conn(0x7f124c15b630 0x7f124c0d0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:45.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.541+0000 7f124affd700 1 -- 172.21.15.16:0/1343552053 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3148814228 0 0) 0x7f124c0b0970 con 0x7f124c0ce4f0 2023-10-16T12:26:45.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.541+0000 7f124affd700 1 -- 172.21.15.16:0/1343552053 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12340035b0 con 0x7f124c0ce4f0 2023-10-16T12:26:45.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.541+0000 7f124bfff700 1 -- 172.21.15.16:0/1343552053 >> v1:172.21.15.49:6789/0 conn(0x7f124c15ba10 legacy=0x7f124c0cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:45.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.541+0000 7f124bfff700 1 -- 172.21.15.16:0/1343552053 >> v1:172.21.15.16:6789/0 conn(0x7f124c0ce4f0 legacy=0x7f124c0b1430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:45.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.541+0000 7f124bfff700 1 -- 172.21.15.16:0/1343552053 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f124c0b0c30 con 0x7f124c15b630 2023-10-16T12:26:45.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.541+0000 7f124bfff700 1 --2- 172.21.15.16:0/1343552053 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f124c15b630 0x7f124c0d0380 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f123c009840 tx=0x7f123c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=89936c4835087d79 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:45.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.541+0000 7f124affd700 1 -- 172.21.15.16:0/1343552053 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f123c011830 con 0x7f124c15b630 2023-10-16T12:26:45.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.541+0000 7f124affd700 1 -- 172.21.15.16:0/1343552053 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f123c011990 con 0x7f124c15b630 2023-10-16T12:26:45.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.541+0000 7f124affd700 1 -- 172.21.15.16:0/1343552053 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f123c011d00 con 0x7f124c15b630 2023-10-16T12:26:45.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.542+0000 7f12526f7700 1 -- 172.21.15.16:0/1343552053 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f124c15b630 msgr2=0x7f124c0d0380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:45.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.542+0000 7f12526f7700 1 --2- 172.21.15.16:0/1343552053 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f124c15b630 0x7f124c0d0380 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f123c009840 tx=0x7f123c009b50 comp rx=0 tx=0).stop 2023-10-16T12:26:45.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.542+0000 7f12526f7700 1 -- 172.21.15.16:0/1343552053 shutdown_connections 2023-10-16T12:26:45.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.542+0000 7f12526f7700 1 --2- 172.21.15.16:0/1343552053 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f124c15b630 0x7f124c0d0380 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:45.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.542+0000 7f12526f7700 1 -- 172.21.15.16:0/1343552053 >> 172.21.15.16:0/1343552053 conn(0x7f124c0d14b0 msgr2=0x7f124c0d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:45.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.542+0000 7f12526f7700 1 -- 172.21.15.16:0/1343552053 shutdown_connections 2023-10-16T12:26:45.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.542+0000 7f12526f7700 1 -- 172.21.15.16:0/1343552053 wait complete. 2023-10-16T12:26:45.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.544+0000 7f12526f7700 1 Processor -- start 2023-10-16T12:26:45.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.544+0000 7f12526f7700 1 -- start start 2023-10-16T12:26:45.549 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:45.545+0000 7f11cbf9c200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T12:26:45.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.546+0000 7f12526f7700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f124c0ce4f0 0x7f124c1f3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:45.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.546+0000 7f12526f7700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f124c15b630 0x7f124c15e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:45.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.546+0000 7f12526f7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f124c15ba10 0x7f124c0c6c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:45.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.546+0000 7f12526f7700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f124c1f6540 con 0x7f124c15ba10 2023-10-16T12:26:45.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.546+0000 7f12526f7700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f124c1f6820 con 0x7f124c15b630 2023-10-16T12:26:45.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.546+0000 7f12526f7700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f124c1f6b00 con 0x7f124c0ce4f0 2023-10-16T12:26:45.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.546+0000 7f1250c94700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f124c15ba10 0x7f124c0c6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:45.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.546+0000 7f1250c94700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f124c15ba10 0x7f124c0c6c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40296/0 (socket says 172.21.15.16:40296) 2023-10-16T12:26:45.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.546+0000 7f1250c94700 1 -- 172.21.15.16:0/675144912 learned_addr learned my addr 172.21.15.16:0/675144912 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:45.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.546+0000 7f124b7fe700 1 --2- 172.21.15.16:0/675144912 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f124c15b630 0x7f124c15e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:45.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.546+0000 7f124bfff700 1 --2- 172.21.15.16:0/675144912 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f124c0ce4f0 0x7f124c1f3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:45.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.546+0000 7f124b7fe700 1 -- 172.21.15.16:0/675144912 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f124c0ce4f0 msgr2=0x7f124c1f3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:45.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.546+0000 7f124b7fe700 1 --2- 172.21.15.16:0/675144912 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f124c0ce4f0 0x7f124c1f3600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:45.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.546+0000 7f124b7fe700 1 -- 172.21.15.16:0/675144912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f124c15ba10 msgr2=0x7f124c0c6c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:45.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.546+0000 7f124b7fe700 1 --2- 172.21.15.16:0/675144912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f124c15ba10 0x7f124c0c6c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:45.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.546+0000 7f124b7fe700 1 -- 172.21.15.16:0/675144912 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f123c0094a0 con 0x7f124c15b630 2023-10-16T12:26:45.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.546+0000 7f1250c94700 1 --2- 172.21.15.16:0/675144912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f124c15ba10 0x7f124c0c6c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:26:45.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.546+0000 7f124b7fe700 1 --2- 172.21.15.16:0/675144912 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f124c15b630 0x7f124c15e190 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f124000c440 tx=0x7f124000c750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:45.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.547+0000 7f1248ff9700 1 -- 172.21.15.16:0/675144912 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12400052d0 con 0x7f124c15b630 2023-10-16T12:26:45.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.547+0000 7f1248ff9700 1 -- 172.21.15.16:0/675144912 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1240005430 con 0x7f124c15b630 2023-10-16T12:26:45.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.547+0000 7f1248ff9700 1 -- 172.21.15.16:0/675144912 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1240006aa0 con 0x7f124c15b630 2023-10-16T12:26:45.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.547+0000 7f12526f7700 1 -- 172.21.15.16:0/675144912 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f124c1f6de0 con 0x7f124c15b630 2023-10-16T12:26:45.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.547+0000 7f12526f7700 1 -- 172.21.15.16:0/675144912 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f124c1f3b40 con 0x7f124c15b630 2023-10-16T12:26:45.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.548+0000 7f12526f7700 1 -- 172.21.15.16:0/675144912 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f124c1f7300 con 0x7f124c15b630 2023-10-16T12:26:45.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.549+0000 7f1248ff9700 1 -- 172.21.15.16:0/675144912 <== mon.1 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 37) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f1240003680 con 0x7f124c15b630 2023-10-16T12:26:45.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.549+0000 7f1248ff9700 1 --2- 172.21.15.16:0/675144912 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f123806b020 0x7f123806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:45.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.549+0000 7f124bfff700 1 -- 172.21.15.16:0/675144912 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f123806b020 msgr2=0x7f123806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2230262559 2023-10-16T12:26:45.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.549+0000 7f1248ff9700 1 -- 172.21.15.16:0/675144912 <== mon.1 v2:172.21.15.49:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f12400889b0 con 0x7f124c15b630 2023-10-16T12:26:45.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.549+0000 7f124bfff700 1 --2- 172.21.15.16:0/675144912 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f123806b020 0x7f123806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:26:45.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.554+0000 7f1248ff9700 1 -- 172.21.15.16:0/675144912 <== mon.1 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f124c1f7300 con 0x7f124c15b630 2023-10-16T12:26:45.562 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:26:45.560+0000 7f50d29c5200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T12:26:45.721 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:45.719+0000 7f11cbf9c200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T12:26:45.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.750+0000 7f124bfff700 1 -- 172.21.15.16:0/675144912 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f123806b020 msgr2=0x7f123806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2230262559 2023-10-16T12:26:45.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.750+0000 7f124bfff700 1 --2- 172.21.15.16:0/675144912 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f123806b020 0x7f123806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:26:45.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.759+0000 7f12526f7700 1 -- 172.21.15.16:0/675144912 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f124c1f7300 con 0x7f124c15b630 2023-10-16T12:26:45.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.760+0000 7f1248ff9700 1 -- 172.21.15.16:0/675144912 <== mon.1 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+17151 (secure 0 0 0) 0x7f124c1f7300 con 0x7f124c15b630 2023-10-16T12:26:45.763 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:26:45.763 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":86,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:26:38.750010+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:26:21.893734+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2023-10-16T12:26:22.642815+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/2700763633":"2023-10-17T12:26:38.749960+0000","172.21.15.16:0/2069262541":"2023-10-17T12:26:38.749960+0000","172.21.15.16:0/1444840585":"2023-10-17T12:26:38.749960+0000","172.21.15.16:6833/3415303009":"2023-10-17T12:26:38.749960+0000","172.21.15.16:6832/3415303009":"2023-10-17T12:26:38.749960+0000","172.21.15.16:0/779547724":"2023-10-17T12:26:38.749960+0000","172.21.15.16:0/4285863251":"2023-10-17T12:26:38.749960+0000","172.21.15.16:0/4057240133":"2023-10-17T12:26:38.749960+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:26:45.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.762+0000 7f12526f7700 1 -- 172.21.15.16:0/675144912 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f123806b020 msgr2=0x7f123806d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:26:45.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.762+0000 7f12526f7700 1 --2- 172.21.15.16:0/675144912 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f123806b020 0x7f123806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:45.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.762+0000 7f12526f7700 1 -- 172.21.15.16:0/675144912 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f124c15b630 msgr2=0x7f124c15e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:45.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.762+0000 7f12526f7700 1 --2- 172.21.15.16:0/675144912 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f124c15b630 0x7f124c15e190 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f124000c440 tx=0x7f124000c750 comp rx=0 tx=0).stop 2023-10-16T12:26:45.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.762+0000 7f12526f7700 1 -- 172.21.15.16:0/675144912 shutdown_connections 2023-10-16T12:26:45.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.762+0000 7f12526f7700 1 --2- 172.21.15.16:0/675144912 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f123806b020 0x7f123806d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:45.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.762+0000 7f12526f7700 1 --2- 172.21.15.16:0/675144912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f124c15ba10 0x7f124c0c6c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:45.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.762+0000 7f12526f7700 1 --2- 172.21.15.16:0/675144912 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f124c0ce4f0 0x7f124c1f3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:45.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.762+0000 7f12526f7700 1 --2- 172.21.15.16:0/675144912 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f124c15b630 0x7f124c15e190 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:45.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.762+0000 7f12526f7700 1 -- 172.21.15.16:0/675144912 >> 172.21.15.16:0/675144912 conn(0x7f124c0d14b0 msgr2=0x7f124c06f820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:45.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.763+0000 7f12526f7700 1 -- 172.21.15.16:0/675144912 shutdown_connections 2023-10-16T12:26:45.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.763+0000 7f12526f7700 1 -- 172.21.15.16:0/675144912 wait complete. 2023-10-16T12:26:45.778 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-16T12:26:45.778 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:26:45.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.903+0000 7f73683b6700 1 Processor -- start 2023-10-16T12:26:45.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.903+0000 7f73683b6700 1 -- start start 2023-10-16T12:26:45.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.905+0000 7f73683b6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f73600ce4f0 0x7f73600d0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:45.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.905+0000 7f73683b6700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f736015ba10 0x7f73600cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:45.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.905+0000 7f73683b6700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73600b0970 con 0x7f736015b630 2023-10-16T12:26:45.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.905+0000 7f73683b6700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73600cd690 con 0x7f73600ce4f0 2023-10-16T12:26:45.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.905+0000 7f73683b6700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73600cd970 con 0x7f736015ba10 2023-10-16T12:26:45.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.905+0000 7f7365951700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f73600ce4f0 0x7f73600d0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:45.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.905+0000 7f7365951700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f73600ce4f0 0x7f73600d0380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40304/0 (socket says 172.21.15.16:40304) 2023-10-16T12:26:45.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.905+0000 7f7365951700 1 -- 172.21.15.16:0/2015064747 learned_addr learned my addr 172.21.15.16:0/2015064747 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:45.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.905+0000 7f7366152700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f736015b630 0x7f73600cdde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57808/0 (socket says 172.21.15.16:57808) 2023-10-16T12:26:45.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.905+0000 7f7366953700 1 --2- 172.21.15.16:0/2015064747 >> v2:172.21.15.49:3300/0 conn(0x7f736015ba10 0x7f73600cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:45.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.905+0000 7f7365951700 1 -- 172.21.15.16:0/2015064747 >> v2:172.21.15.49:3300/0 conn(0x7f736015ba10 msgr2=0x7f73600cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:45.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.905+0000 7f7365951700 1 --2- 172.21.15.16:0/2015064747 >> v2:172.21.15.49:3300/0 conn(0x7f736015ba10 0x7f73600cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:45.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.905+0000 7f7365951700 1 -- 172.21.15.16:0/2015064747 >> v1:172.21.15.16:6789/0 conn(0x7f736015b630 legacy=0x7f73600cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:45.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.905+0000 7f7365951700 1 -- 172.21.15.16:0/2015064747 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73600b0c30 con 0x7f73600ce4f0 2023-10-16T12:26:45.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.905+0000 7f7365150700 1 -- 172.21.15.16:0/2015064747 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 521489000 0 0) 0x7f73600b0970 con 0x7f736015b630 2023-10-16T12:26:45.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.905+0000 7f7365951700 1 --2- 172.21.15.16:0/2015064747 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73600ce4f0 0x7f73600d0380 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f73600cbb00 tx=0x7f7350009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bbea48af3500579 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:45.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.905+0000 7f7365150700 1 -- 172.21.15.16:0/2015064747 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7350011830 con 0x7f73600ce4f0 2023-10-16T12:26:45.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.906+0000 7f7365150700 1 -- 172.21.15.16:0/2015064747 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7350011990 con 0x7f73600ce4f0 2023-10-16T12:26:45.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.906+0000 7f7365150700 1 -- 172.21.15.16:0/2015064747 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7350011d20 con 0x7f73600ce4f0 2023-10-16T12:26:45.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.906+0000 7f73683b6700 1 -- 172.21.15.16:0/2015064747 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73600ce4f0 msgr2=0x7f73600d0380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:45.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.906+0000 7f73683b6700 1 --2- 172.21.15.16:0/2015064747 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73600ce4f0 0x7f73600d0380 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f73600cbb00 tx=0x7f7350009bd0 comp rx=0 tx=0).stop 2023-10-16T12:26:45.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.906+0000 7f73683b6700 1 -- 172.21.15.16:0/2015064747 shutdown_connections 2023-10-16T12:26:45.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.906+0000 7f73683b6700 1 --2- 172.21.15.16:0/2015064747 >> v2:172.21.15.49:3300/0 conn(0x7f736015ba10 0x7f73600cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:45.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.906+0000 7f73683b6700 1 --2- 172.21.15.16:0/2015064747 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73600ce4f0 0x7f73600d0380 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:45.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.906+0000 7f73683b6700 1 -- 172.21.15.16:0/2015064747 >> 172.21.15.16:0/2015064747 conn(0x7f73600d14b0 msgr2=0x7f73600d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:45.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.906+0000 7f73683b6700 1 -- 172.21.15.16:0/2015064747 shutdown_connections 2023-10-16T12:26:45.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.906+0000 7f73683b6700 1 -- 172.21.15.16:0/2015064747 wait complete. 2023-10-16T12:26:45.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.907+0000 7f73683b6700 1 Processor -- start 2023-10-16T12:26:45.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.907+0000 7f73683b6700 1 -- start start 2023-10-16T12:26:45.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.908+0000 7f73683b6700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f73600ce4f0 0x7f736006d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:45.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.908+0000 7f73683b6700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f736015b630 0x7f73601f2580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:45.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.908+0000 7f73683b6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f736015ba10 0x7f736015e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:45.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.908+0000 7f73683b6700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f736015ead0 con 0x7f736015ba10 2023-10-16T12:26:45.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.908+0000 7f73683b6700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f736015e8a0 con 0x7f73600ce4f0 2023-10-16T12:26:45.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.908+0000 7f73683b6700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f736015bf00 con 0x7f736015b630 2023-10-16T12:26:45.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.908+0000 7f7366953700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f736015ba10 0x7f736015e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:45.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.908+0000 7f7366152700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f73600ce4f0 0x7f736006d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:45.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.908+0000 7f7366953700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f736015ba10 0x7f736015e360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40312/0 (socket says 172.21.15.16:40312) 2023-10-16T12:26:45.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.908+0000 7f7366953700 1 -- 172.21.15.16:0/1765905028 learned_addr learned my addr 172.21.15.16:0/1765905028 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:45.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.908+0000 7f7365951700 1 --2- 172.21.15.16:0/1765905028 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f736015b630 0x7f73601f2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:45.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.909+0000 7f7366152700 1 -- 172.21.15.16:0/1765905028 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f736015b630 msgr2=0x7f73601f2580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:45.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.909+0000 7f7366152700 1 --2- 172.21.15.16:0/1765905028 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f736015b630 0x7f73601f2580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:45.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.909+0000 7f7366152700 1 -- 172.21.15.16:0/1765905028 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f736015ba10 msgr2=0x7f736015e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:45.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.909+0000 7f7366152700 1 --2- 172.21.15.16:0/1765905028 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f736015ba10 0x7f736015e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:45.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.909+0000 7f7366152700 1 -- 172.21.15.16:0/1765905028 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7350009550 con 0x7f73600ce4f0 2023-10-16T12:26:45.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.909+0000 7f7366152700 1 --2- 172.21.15.16:0/1765905028 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f73600ce4f0 0x7f736006d510 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f735c00c4b0 tx=0x7f735c00c7c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:45.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.909+0000 7f7356ffd700 1 -- 172.21.15.16:0/1765905028 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f735c004c80 con 0x7f73600ce4f0 2023-10-16T12:26:45.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.909+0000 7f7356ffd700 1 -- 172.21.15.16:0/1765905028 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f735c004e00 con 0x7f73600ce4f0 2023-10-16T12:26:45.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.909+0000 7f73683b6700 1 -- 172.21.15.16:0/1765905028 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f736015c150 con 0x7f73600ce4f0 2023-10-16T12:26:45.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.909+0000 7f7356ffd700 1 -- 172.21.15.16:0/1765905028 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f735c007310 con 0x7f73600ce4f0 2023-10-16T12:26:45.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.909+0000 7f73683b6700 1 -- 172.21.15.16:0/1765905028 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f736015c710 con 0x7f73600ce4f0 2023-10-16T12:26:45.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.910+0000 7f73683b6700 1 -- 172.21.15.16:0/1765905028 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f736004ebb0 con 0x7f73600ce4f0 2023-10-16T12:26:45.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.911+0000 7f7356ffd700 1 -- 172.21.15.16:0/1765905028 <== mon.1 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 37) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f735c007470 con 0x7f73600ce4f0 2023-10-16T12:26:45.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.911+0000 7f7356ffd700 1 --2- 172.21.15.16:0/1765905028 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f734406b020 0x7f734406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:45.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.911+0000 7f7365951700 1 -- 172.21.15.16:0/1765905028 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f734406b020 msgr2=0x7f734406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2230262559 2023-10-16T12:26:45.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.911+0000 7f7356ffd700 1 -- 172.21.15.16:0/1765905028 <== mon.1 v2:172.21.15.49:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f735c089580 con 0x7f73600ce4f0 2023-10-16T12:26:45.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.911+0000 7f7365951700 1 --2- 172.21.15.16:0/1765905028 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f734406b020 0x7f734406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:26:45.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:45.916+0000 7f7356ffd700 1 -- 172.21.15.16:0/1765905028 <== mon.1 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f735c0543b0 con 0x7f73600ce4f0 2023-10-16T12:26:46.111 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:26:46.109+0000 7f50d29c5200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T12:26:46.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.112+0000 7f7365951700 1 -- 172.21.15.16:0/1765905028 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f734406b020 msgr2=0x7f734406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2230262559 2023-10-16T12:26:46.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.112+0000 7f7365951700 1 --2- 172.21.15.16:0/1765905028 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f734406b020 0x7f734406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:26:46.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.166+0000 7f73683b6700 1 -- 172.21.15.16:0/1765905028 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f736006b5d0 con 0x7f73600ce4f0 2023-10-16T12:26:46.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.167+0000 7f7356ffd700 1 -- 172.21.15.16:0/1765905028 <== mon.1 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+4911 (secure 0 0 0) 0x7f735c0579d0 con 0x7f73600ce4f0 2023-10-16T12:26:46.169 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:26:46.169 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":36,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5861,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/1236108916","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":1236108916},{"type":"v1","addr":"172.21.15.49:6834","nonce":1236108916}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31},{"gid":5864,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/2762335434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":2762335434},{"type":"v1","addr":"172.21.15.49:6835","nonce":2762335434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31},{"gid":6379,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/1502256436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1502256436},{"type":"v1","addr":"172.21.15.16:6836","nonce":1502256436}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:26:23.670609+0000","modified":"2023-10-16T12:26:26.710098+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6627},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6627":{"gid":6627,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/962243511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":962243511},{"type":"v1","addr":"172.21.15.16:6837","nonce":962243511}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-10-16T12:26:46.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.169+0000 7f73683b6700 1 -- 172.21.15.16:0/1765905028 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f734406b020 msgr2=0x7f734406d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:26:46.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.169+0000 7f73683b6700 1 --2- 172.21.15.16:0/1765905028 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f734406b020 0x7f734406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:46.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.169+0000 7f73683b6700 1 -- 172.21.15.16:0/1765905028 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f73600ce4f0 msgr2=0x7f736006d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:46.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.169+0000 7f73683b6700 1 --2- 172.21.15.16:0/1765905028 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f73600ce4f0 0x7f736006d510 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f735c00c4b0 tx=0x7f735c00c7c0 comp rx=0 tx=0).stop 2023-10-16T12:26:46.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.169+0000 7f73683b6700 1 -- 172.21.15.16:0/1765905028 shutdown_connections 2023-10-16T12:26:46.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.169+0000 7f73683b6700 1 --2- 172.21.15.16:0/1765905028 >> [v2:172.21.15.16:6832/2230262559,v1:172.21.15.16:6833/2230262559] conn(0x7f734406b020 0x7f734406d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:46.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.169+0000 7f73683b6700 1 --2- 172.21.15.16:0/1765905028 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f736015ba10 0x7f736015e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:46.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.169+0000 7f73683b6700 1 --2- 172.21.15.16:0/1765905028 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f73600ce4f0 0x7f736006d510 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:46.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.169+0000 7f73683b6700 1 --2- 172.21.15.16:0/1765905028 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f736015b630 0x7f73601f2580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:46.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.169+0000 7f73683b6700 1 -- 172.21.15.16:0/1765905028 >> 172.21.15.16:0/1765905028 conn(0x7f73600d14b0 msgr2=0x7f736015a920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:46.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.169+0000 7f73683b6700 1 -- 172.21.15.16:0/1765905028 shutdown_connections 2023-10-16T12:26:46.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.169+0000 7f73683b6700 1 -- 172.21.15.16:0/1765905028 wait complete. 2023-10-16T12:26:46.174 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 36 2023-10-16T12:26:46.183 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:26:46.296 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:46.295+0000 7f11cbf9c200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T12:26:46.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.307+0000 7f730286f700 1 Processor -- start 2023-10-16T12:26:46.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.307+0000 7f730286f700 1 -- start start 2023-10-16T12:26:46.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.308+0000 7f730286f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f72fc15a5c0 0x7f72fc0cef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:46.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.308+0000 7f730286f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f72fc0d0ae0 0x7f72fc0cc1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:46.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.308+0000 7f730286f700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72fc072a50 con 0x7f72fc0ce010 2023-10-16T12:26:46.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.308+0000 7f730286f700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72fc0cf440 con 0x7f72fc0d0ae0 2023-10-16T12:26:46.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.308+0000 7f730286f700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72fc0cba30 con 0x7f72fc15a5c0 2023-10-16T12:26:46.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.309+0000 7f7300e0c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f72fc0d0ae0 0x7f72fc0cc1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:46.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.309+0000 7f72fbfff700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f72fc15a5c0 0x7f72fc0cef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:46.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.309+0000 7f7300e0c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f72fc0d0ae0 0x7f72fc0cc1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40324/0 (socket says 172.21.15.16:40324) 2023-10-16T12:26:46.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.309+0000 7f7300e0c700 1 -- 172.21.15.16:0/3631730460 learned_addr learned my addr 172.21.15.16:0/3631730460 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:46.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.309+0000 7f72fb7fe700 1 --1- 172.21.15.16:0/3631730460 >> v1:172.21.15.49:6789/0 conn(0x7f72fc0ce010 0x7f72fc0d03d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:36778/0 (socket says 172.21.15.16:36778) 2023-10-16T12:26:46.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.309+0000 7f7300e0c700 1 -- 172.21.15.16:0/3631730460 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f72fc15a5c0 msgr2=0x7f72fc0cef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:46.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.309+0000 7f7300e0c700 1 --2- 172.21.15.16:0/3631730460 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f72fc15a5c0 0x7f72fc0cef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:46.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.309+0000 7f7300e0c700 1 -- 172.21.15.16:0/3631730460 >> v1:172.21.15.49:6789/0 conn(0x7f72fc0ce010 legacy=0x7f72fc0d03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:46.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.309+0000 7f7300e0c700 1 -- 172.21.15.16:0/3631730460 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72fc0cf630 con 0x7f72fc0d0ae0 2023-10-16T12:26:46.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.309+0000 7f7300e0c700 1 --2- 172.21.15.16:0/3631730460 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72fc0d0ae0 0x7f72fc0cc1f0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f72f4009e50 tx=0x7f72f400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b3eb393d606ff28 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:46.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.310+0000 7f72faffd700 1 -- 172.21.15.16:0/3631730460 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f400bb10 con 0x7f72fc0d0ae0 2023-10-16T12:26:46.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.310+0000 7f72faffd700 1 -- 172.21.15.16:0/3631730460 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72f400bc70 con 0x7f72fc0d0ae0 2023-10-16T12:26:46.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.310+0000 7f72faffd700 1 -- 172.21.15.16:0/3631730460 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f4011460 con 0x7f72fc0d0ae0 2023-10-16T12:26:46.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.310+0000 7f730286f700 1 -- 172.21.15.16:0/3631730460 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72fc0d0ae0 msgr2=0x7f72fc0cc1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:46.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.310+0000 7f730286f700 1 --2- 172.21.15.16:0/3631730460 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72fc0d0ae0 0x7f72fc0cc1f0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f72f4009e50 tx=0x7f72f400b330 comp rx=0 tx=0).stop 2023-10-16T12:26:46.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.311+0000 7f730286f700 1 -- 172.21.15.16:0/3631730460 shutdown_connections 2023-10-16T12:26:46.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.311+0000 7f730286f700 1 --2- 172.21.15.16:0/3631730460 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72fc0d0ae0 0x7f72fc0cc1f0 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:46.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.311+0000 7f730286f700 1 --2- 172.21.15.16:0/3631730460 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f72fc15a5c0 0x7f72fc0cef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:46.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.311+0000 7f730286f700 1 -- 172.21.15.16:0/3631730460 >> 172.21.15.16:0/3631730460 conn(0x7f72fc0b1fd0 msgr2=0x7f72fc0b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:46.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.311+0000 7f730286f700 1 -- 172.21.15.16:0/3631730460 shutdown_connections 2023-10-16T12:26:46.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.311+0000 7f730286f700 1 -- 172.21.15.16:0/3631730460 wait complete. 2023-10-16T12:26:46.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.312+0000 7f730286f700 1 Processor -- start 2023-10-16T12:26:46.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.313+0000 7f730286f700 1 -- start start 2023-10-16T12:26:46.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.314+0000 7f730286f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f72fc0ce010 0x7f72fc0b36b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:46.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.314+0000 7f730286f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72fc0d0ae0 0x7f72fc0c8070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:46.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.314+0000 7f730286f700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f72fc15a5c0 0x7f72fc0b0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:46.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.314+0000 7f730286f700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72fc0c85b0 con 0x7f72fc0d0ae0 2023-10-16T12:26:46.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.314+0000 7f730286f700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72fc0b4880 con 0x7f72fc15a5c0 2023-10-16T12:26:46.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.314+0000 7f730286f700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72fc0b3d00 con 0x7f72fc0ce010 2023-10-16T12:26:46.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.316+0000 7f7300e0c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f72fc15a5c0 0x7f72fc0b0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:46.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.316+0000 7f7300e0c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f72fc15a5c0 0x7f72fc0b0dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:39388/0 (socket says 172.21.15.16:39388) 2023-10-16T12:26:46.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.316+0000 7f7300e0c700 1 -- 172.21.15.16:0/615366528 learned_addr learned my addr 172.21.15.16:0/615366528 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:46.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.316+0000 7f72fb7fe700 1 --2- 172.21.15.16:0/615366528 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72fc0d0ae0 0x7f72fc0c8070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:46.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.316+0000 7f7300e0c700 1 -- 172.21.15.16:0/615366528 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f72fc0ce010 msgr2=0x7f72fc0b36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:46.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.316+0000 7f7300e0c700 1 --2- 172.21.15.16:0/615366528 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f72fc0ce010 0x7f72fc0b36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:46.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.316+0000 7f7300e0c700 1 -- 172.21.15.16:0/615366528 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72fc0d0ae0 msgr2=0x7f72fc0c8070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:46.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.316+0000 7f7300e0c700 1 --2- 172.21.15.16:0/615366528 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72fc0d0ae0 0x7f72fc0c8070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:46.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.316+0000 7f7300e0c700 1 -- 172.21.15.16:0/615366528 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72f4009b30 con 0x7f72fc15a5c0 2023-10-16T12:26:46.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.317+0000 7f7300e0c700 1 --2- 172.21.15.16:0/615366528 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f72fc15a5c0 0x7f72fc0b0dc0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f72f4014000 tx=0x7f72f4004ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:46.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.317+0000 7f72f8ff9700 1 -- 172.21.15.16:0/615366528 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f4020070 con 0x7f72fc15a5c0 2023-10-16T12:26:46.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.317+0000 7f72f8ff9700 1 -- 172.21.15.16:0/615366528 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72f40099b0 con 0x7f72fc15a5c0 2023-10-16T12:26:46.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.317+0000 7f72f8ff9700 1 -- 172.21.15.16:0/615366528 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f4011a60 con 0x7f72fc15a5c0 2023-10-16T12:26:46.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.317+0000 7f730286f700 1 -- 172.21.15.16:0/615366528 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72fc0b3fe0 con 0x7f72fc15a5c0 2023-10-16T12:26:46.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.317+0000 7f730286f700 1 -- 172.21.15.16:0/615366528 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72fc0b45a0 con 0x7f72fc15a5c0 2023-10-16T12:26:46.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.318+0000 7f730286f700 1 -- 172.21.15.16:0/615366528 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72fc04ebb0 con 0x7f72fc15a5c0 2023-10-16T12:26:46.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.319+0000 7f72f8ff9700 1 -- 172.21.15.16:0/615366528 <== mon.1 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 38) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f72f4011bc0 con 0x7f72fc15a5c0 2023-10-16T12:26:46.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.320+0000 7f72f8ff9700 1 -- 172.21.15.16:0/615366528 <== mon.1 v2:172.21.15.49:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6745+0+0 (secure 0 0 0) 0x7f72f408c820 con 0x7f72fc15a5c0 2023-10-16T12:26:46.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.325+0000 7f72f8ff9700 1 -- 172.21.15.16:0/615366528 <== mon.1 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f72f4057830 con 0x7f72fc15a5c0 2023-10-16T12:26:46.405 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:46.403+0000 7f11447c2700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:26:46.405 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:46.403+0000 7f11447c2700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:26:46.406 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:46.403+0000 7f11447c2700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:26:46.406 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:46.403+0000 7f11447c2700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:26:46.406 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:46.403+0000 7f11447c2700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:26:46.406 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:46.403+0000 7f11477c8700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:26:46.406 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:46.403+0000 7f11477c8700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:26:46.407 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:46.403+0000 7f11477c8700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:26:46.407 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:46.403+0000 7f11477c8700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:26:46.407 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:26:46.403+0000 7f11477c8700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:26:46.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.588+0000 7f730286f700 1 -- 172.21.15.16:0/615366528 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f72fc15df20 con 0x7f72fc15a5c0 2023-10-16T12:26:46.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.589+0000 7f72f8ff9700 1 -- 172.21.15.16:0/615366528 <== mon.1 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+4911 (secure 0 0 0) 0x7f72f405ae50 con 0x7f72fc15a5c0 2023-10-16T12:26:46.591 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:26:46.592 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":36,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5861,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/1236108916","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":1236108916},{"type":"v1","addr":"172.21.15.49:6834","nonce":1236108916}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31},{"gid":5864,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/2762335434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":2762335434},{"type":"v1","addr":"172.21.15.49:6835","nonce":2762335434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31},{"gid":6379,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/1502256436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1502256436},{"type":"v1","addr":"172.21.15.16:6836","nonce":1502256436}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:26:23.670609+0000","modified":"2023-10-16T12:26:26.710098+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6627},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6627":{"gid":6627,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/962243511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":962243511},{"type":"v1","addr":"172.21.15.16:6837","nonce":962243511}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-10-16T12:26:46.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.591+0000 7f730286f700 1 -- 172.21.15.16:0/615366528 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f72fc15a5c0 msgr2=0x7f72fc0b0dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:46.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.591+0000 7f730286f700 1 --2- 172.21.15.16:0/615366528 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f72fc15a5c0 0x7f72fc0b0dc0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f72f4014000 tx=0x7f72f4004ad0 comp rx=0 tx=0).stop 2023-10-16T12:26:46.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.591+0000 7f730286f700 1 -- 172.21.15.16:0/615366528 shutdown_connections 2023-10-16T12:26:46.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.591+0000 7f730286f700 1 --2- 172.21.15.16:0/615366528 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f72fc15a5c0 0x7f72fc0b0dc0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:46.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.591+0000 7f730286f700 1 --2- 172.21.15.16:0/615366528 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f72fc0ce010 0x7f72fc0b36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:46.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.591+0000 7f730286f700 1 --2- 172.21.15.16:0/615366528 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72fc0d0ae0 0x7f72fc0c8070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:46.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.591+0000 7f730286f700 1 -- 172.21.15.16:0/615366528 >> 172.21.15.16:0/615366528 conn(0x7f72fc0b1fd0 msgr2=0x7f72fc0ccea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:46.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.592+0000 7f730286f700 1 -- 172.21.15.16:0/615366528 shutdown_connections 2023-10-16T12:26:46.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.592+0000 7f730286f700 1 -- 172.21.15.16:0/615366528 wait complete. 2023-10-16T12:26:46.596 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 36 2023-10-16T12:26:46.606 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:26:46.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.728+0000 7fe348869700 1 Processor -- start 2023-10-16T12:26:46.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.728+0000 7fe348869700 1 -- start start 2023-10-16T12:26:46.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.729+0000 7fe348869700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe3440ce4f0 0x7fe3440cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:46.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.729+0000 7fe348869700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fe34415ba10 0x7fe3440cfca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:46.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.729+0000 7fe348869700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3440cc2a0 con 0x7fe34415b630 2023-10-16T12:26:46.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.729+0000 7fe348869700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3440cdb20 con 0x7fe3440ce4f0 2023-10-16T12:26:46.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.729+0000 7fe348869700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3440cc540 con 0x7fe34415ba10 2023-10-16T12:26:46.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.729+0000 7fe341d9b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe3440ce4f0 0x7fe3440cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:46.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.729+0000 7fe34259c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe34415b630 0x7fe3440cdde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57908/0 (socket says 172.21.15.16:57908) 2023-10-16T12:26:46.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.729+0000 7fe34259c700 1 -- 172.21.15.16:0/1827886913 learned_addr learned my addr 172.21.15.16:0/1827886913 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:26:46.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.730+0000 7fe342d9d700 1 --2- 172.21.15.16:0/1827886913 >> v2:172.21.15.49:3300/0 conn(0x7fe34415ba10 0x7fe3440cfca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:46.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.730+0000 7fe341d9b700 1 -- 172.21.15.16:0/1827886913 >> v2:172.21.15.49:3300/0 conn(0x7fe34415ba10 msgr2=0x7fe3440cfca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:46.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.730+0000 7fe341d9b700 1 --2- 172.21.15.16:0/1827886913 >> v2:172.21.15.49:3300/0 conn(0x7fe34415ba10 0x7fe3440cfca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:46.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.730+0000 7fe341d9b700 1 -- 172.21.15.16:0/1827886913 >> v1:172.21.15.16:6789/0 conn(0x7fe34415b630 legacy=0x7fe3440cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:46.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.730+0000 7fe341d9b700 1 -- 172.21.15.16:0/1827886913 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3440cc820 con 0x7fe3440ce4f0 2023-10-16T12:26:46.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.730+0000 7fe34159a700 1 -- 172.21.15.16:0/1827886913 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2534216272 0 0) 0x7fe3440cc2a0 con 0x7fe34415b630 2023-10-16T12:26:46.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.730+0000 7fe341d9b700 1 --2- 172.21.15.16:0/1827886913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3440ce4f0 0x7fe3440cd5e0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7fe34408bcb0 tx=0x7fe338009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dde72f2471f6c228 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:46.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.730+0000 7fe34159a700 1 -- 172.21.15.16:0/1827886913 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe338011830 con 0x7fe3440ce4f0 2023-10-16T12:26:46.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.730+0000 7fe34159a700 1 -- 172.21.15.16:0/1827886913 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe338011990 con 0x7fe3440ce4f0 2023-10-16T12:26:46.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.730+0000 7fe34159a700 1 -- 172.21.15.16:0/1827886913 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe338011d20 con 0x7fe3440ce4f0 2023-10-16T12:26:46.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.731+0000 7fe348869700 1 -- 172.21.15.16:0/1827886913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3440ce4f0 msgr2=0x7fe3440cd5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:46.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.731+0000 7fe348869700 1 --2- 172.21.15.16:0/1827886913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3440ce4f0 0x7fe3440cd5e0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7fe34408bcb0 tx=0x7fe338009b20 comp rx=0 tx=0).stop 2023-10-16T12:26:46.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.731+0000 7fe348869700 1 -- 172.21.15.16:0/1827886913 shutdown_connections 2023-10-16T12:26:46.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.731+0000 7fe348869700 1 --2- 172.21.15.16:0/1827886913 >> v2:172.21.15.49:3300/0 conn(0x7fe34415ba10 0x7fe3440cfca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:46.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.731+0000 7fe348869700 1 --2- 172.21.15.16:0/1827886913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3440ce4f0 0x7fe3440cd5e0 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:46.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.731+0000 7fe348869700 1 -- 172.21.15.16:0/1827886913 >> 172.21.15.16:0/1827886913 conn(0x7fe3440d14b0 msgr2=0x7fe3440d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:46.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.731+0000 7fe348869700 1 -- 172.21.15.16:0/1827886913 shutdown_connections 2023-10-16T12:26:46.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.731+0000 7fe348869700 1 -- 172.21.15.16:0/1827886913 wait complete. 2023-10-16T12:26:46.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.732+0000 7fe348869700 1 Processor -- start 2023-10-16T12:26:46.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.732+0000 7fe348869700 1 -- start start 2023-10-16T12:26:46.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.733+0000 7fe348869700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe3440ce4f0 0x7fe34406cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:46.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.733+0000 7fe348869700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe34415b630 0x7fe3441f4500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:46.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.733+0000 7fe348869700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe34415ba10 0x7fe34406bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:46.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.733+0000 7fe348869700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe34406d6a0 con 0x7fe34415b630 2023-10-16T12:26:46.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.733+0000 7fe348869700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe34406c0a0 con 0x7fe34415ba10 2023-10-16T12:26:46.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.733+0000 7fe348869700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe34406d3c0 con 0x7fe3440ce4f0 2023-10-16T12:26:46.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.733+0000 7fe341d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe34415b630 0x7fe3441f4500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:46.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.733+0000 7fe341d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe34415b630 0x7fe3441f4500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40410/0 (socket says 172.21.15.16:40410) 2023-10-16T12:26:46.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.733+0000 7fe341d9b700 1 -- 172.21.15.16:0/2695038638 learned_addr learned my addr 172.21.15.16:0/2695038638 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:46.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.733+0000 7fe34259c700 1 --2- 172.21.15.16:0/2695038638 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe3440ce4f0 0x7fe34406cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:46.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.733+0000 7fe342d9d700 1 --2- 172.21.15.16:0/2695038638 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe34415ba10 0x7fe34406bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:46.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.734+0000 7fe341d9b700 1 -- 172.21.15.16:0/2695038638 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe3440ce4f0 msgr2=0x7fe34406cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:46.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.734+0000 7fe341d9b700 1 --2- 172.21.15.16:0/2695038638 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe3440ce4f0 0x7fe34406cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:46.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.734+0000 7fe341d9b700 1 -- 172.21.15.16:0/2695038638 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe34415ba10 msgr2=0x7fe34406bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:46.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.734+0000 7fe341d9b700 1 --2- 172.21.15.16:0/2695038638 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe34415ba10 0x7fe34406bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:46.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.734+0000 7fe341d9b700 1 -- 172.21.15.16:0/2695038638 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3380094a0 con 0x7fe34415b630 2023-10-16T12:26:46.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.734+0000 7fe34259c700 1 --2- 172.21.15.16:0/2695038638 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe3440ce4f0 0x7fe34406cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:26:46.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.734+0000 7fe342d9d700 1 --2- 172.21.15.16:0/2695038638 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe34415ba10 0x7fe34406bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:26:46.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.734+0000 7fe341d9b700 1 --2- 172.21.15.16:0/2695038638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe34415b630 0x7fe3441f4500 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fe338000c00 tx=0x7fe338004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:46.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.734+0000 7fe332ffd700 1 -- 172.21.15.16:0/2695038638 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3380036a0 con 0x7fe34415b630 2023-10-16T12:26:46.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.734+0000 7fe332ffd700 1 -- 172.21.15.16:0/2695038638 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe33801f070 con 0x7fe34415b630 2023-10-16T12:26:46.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.734+0000 7fe348869700 1 -- 172.21.15.16:0/2695038638 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3440c6160 con 0x7fe34415b630 2023-10-16T12:26:46.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.734+0000 7fe332ffd700 1 -- 172.21.15.16:0/2695038638 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe338003a20 con 0x7fe34415b630 2023-10-16T12:26:46.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.734+0000 7fe348869700 1 -- 172.21.15.16:0/2695038638 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3440c6720 con 0x7fe34415b630 2023-10-16T12:26:46.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.735+0000 7fe348869700 1 -- 172.21.15.16:0/2695038638 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe34404ebb0 con 0x7fe34415b630 2023-10-16T12:26:46.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.735+0000 7fe332ffd700 1 -- 172.21.15.16:0/2695038638 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fe338003b80 con 0x7fe34415b630 2023-10-16T12:26:46.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.736+0000 7fe332ffd700 1 -- 172.21.15.16:0/2695038638 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6745+0+0 (secure 0 0 0) 0x7fe338021070 con 0x7fe34415b630 2023-10-16T12:26:46.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.740+0000 7fe332ffd700 1 -- 172.21.15.16:0/2695038638 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe338055eb0 con 0x7fe34415b630 2023-10-16T12:26:46.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.942+0000 7fe348869700 1 -- 172.21.15.16:0/2695038638 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe34406b5d0 con 0x7fe34415b630 2023-10-16T12:26:46.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.943+0000 7fe332ffd700 1 -- 172.21.15.16:0/2695038638 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+17652 (secure 0 0 0) 0x7fe33802b090 con 0x7fe34415b630 2023-10-16T12:26:46.945 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:26:46.946 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":87,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:26:46.309812+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:26:21.893734+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2023-10-16T12:26:22.642815+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/2710394622":"2023-10-17T12:26:46.309755+0000","172.21.15.16:0/3754876159":"2023-10-17T12:26:46.309755+0000","172.21.15.16:0/1938940392":"2023-10-17T12:26:46.309755+0000","172.21.15.16:0/3196640464":"2023-10-17T12:26:46.309755+0000","172.21.15.16:0/636468305":"2023-10-17T12:26:46.309755+0000","172.21.15.16:6833/2230262559":"2023-10-17T12:26:46.309755+0000","172.21.15.16:0/2700763633":"2023-10-17T12:26:38.749960+0000","172.21.15.16:0/2069262541":"2023-10-17T12:26:38.749960+0000","172.21.15.16:0/1444840585":"2023-10-17T12:26:38.749960+0000","172.21.15.16:6833/3415303009":"2023-10-17T12:26:38.749960+0000","172.21.15.16:6832/3415303009":"2023-10-17T12:26:38.749960+0000","172.21.15.16:6832/2230262559":"2023-10-17T12:26:46.309755+0000","172.21.15.16:0/779547724":"2023-10-17T12:26:38.749960+0000","172.21.15.16:0/1047057927":"2023-10-17T12:26:46.309755+0000","172.21.15.16:0/4285863251":"2023-10-17T12:26:38.749960+0000","172.21.15.16:0/4057240133":"2023-10-17T12:26:38.749960+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:26:46.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.945+0000 7fe348869700 1 -- 172.21.15.16:0/2695038638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe34415b630 msgr2=0x7fe3441f4500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:46.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.945+0000 7fe348869700 1 --2- 172.21.15.16:0/2695038638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe34415b630 0x7fe3441f4500 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fe338000c00 tx=0x7fe338004750 comp rx=0 tx=0).stop 2023-10-16T12:26:46.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.945+0000 7fe348869700 1 -- 172.21.15.16:0/2695038638 shutdown_connections 2023-10-16T12:26:46.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.945+0000 7fe348869700 1 --2- 172.21.15.16:0/2695038638 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe34415ba10 0x7fe34406bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:46.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.945+0000 7fe348869700 1 --2- 172.21.15.16:0/2695038638 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe3440ce4f0 0x7fe34406cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:46.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.945+0000 7fe348869700 1 --2- 172.21.15.16:0/2695038638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe34415b630 0x7fe3441f4500 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:46.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.945+0000 7fe348869700 1 -- 172.21.15.16:0/2695038638 >> 172.21.15.16:0/2695038638 conn(0x7fe3440d14b0 msgr2=0x7fe34415aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:46.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.945+0000 7fe348869700 1 -- 172.21.15.16:0/2695038638 shutdown_connections 2023-10-16T12:26:46.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:46.945+0000 7fe348869700 1 -- 172.21.15.16:0/2695038638 wait complete. 2023-10-16T12:26:46.960 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-16T12:26:47.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.084+0000 7f6dba126700 1 Processor -- start 2023-10-16T12:26:47.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.084+0000 7f6dba126700 1 -- start start 2023-10-16T12:26:47.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.085+0000 7f6dba126700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6db415b280 0x7f6db41613e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:47.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.085+0000 7f6dba126700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6db406a7e0 0x7f6db406ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:47.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.085+0000 7f6dba126700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6db406b170 con 0x7f6db40cc8a0 2023-10-16T12:26:47.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.085+0000 7f6dba126700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6db4161c30 con 0x7f6db406a7e0 2023-10-16T12:26:47.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.085+0000 7f6dba126700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6db4161f10 con 0x7f6db415b280 2023-10-16T12:26:47.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.085+0000 7f6db3fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6db406a7e0 0x7f6db406ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:47.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.085+0000 7f6db3fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6db406a7e0 0x7f6db406ac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40418/0 (socket says 172.21.15.16:40418) 2023-10-16T12:26:47.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.085+0000 7f6db3fff700 1 -- 172.21.15.16:0/1794763939 learned_addr learned my addr 172.21.15.16:0/1794763939 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:47.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.085+0000 7f6db37fe700 1 --2- 172.21.15.16:0/1794763939 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6db415b280 0x7f6db41613e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:47.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.085+0000 7f6db3fff700 1 -- 172.21.15.16:0/1794763939 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6db415b280 msgr2=0x7f6db41613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:47.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.085+0000 7f6db3fff700 1 --2- 172.21.15.16:0/1794763939 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6db415b280 0x7f6db41613e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:47.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.085+0000 7f6db3fff700 1 -- 172.21.15.16:0/1794763939 >> v1:172.21.15.49:6789/0 conn(0x7f6db40cc8a0 legacy=0x7f6db415e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:47.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.085+0000 7f6db3fff700 1 -- 172.21.15.16:0/1794763939 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6db41621f0 con 0x7f6db406a7e0 2023-10-16T12:26:47.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.085+0000 7f6db3fff700 1 --2- 172.21.15.16:0/1794763939 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6db406a7e0 0x7f6db406ac30 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f6dac009e50 tx=0x7f6dac00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83fdf005193c4361 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:47.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.085+0000 7f6db27fc700 1 -- 172.21.15.16:0/1794763939 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dac00e070 con 0x7f6db406a7e0 2023-10-16T12:26:47.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.085+0000 7f6db27fc700 1 -- 172.21.15.16:0/1794763939 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dac00b9c0 con 0x7f6db406a7e0 2023-10-16T12:26:47.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.085+0000 7f6db27fc700 1 -- 172.21.15.16:0/1794763939 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dac00bd70 con 0x7f6db406a7e0 2023-10-16T12:26:47.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.086+0000 7f6dba126700 1 -- 172.21.15.16:0/1794763939 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6db406a7e0 msgr2=0x7f6db406ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:47.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.086+0000 7f6dba126700 1 --2- 172.21.15.16:0/1794763939 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6db406a7e0 0x7f6db406ac30 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f6dac009e50 tx=0x7f6dac00b330 comp rx=0 tx=0).stop 2023-10-16T12:26:47.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.086+0000 7f6dba126700 1 -- 172.21.15.16:0/1794763939 shutdown_connections 2023-10-16T12:26:47.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.086+0000 7f6dba126700 1 --2- 172.21.15.16:0/1794763939 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6db406a7e0 0x7f6db406ac30 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:47.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.086+0000 7f6dba126700 1 --2- 172.21.15.16:0/1794763939 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6db415b280 0x7f6db41613e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:47.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.086+0000 7f6dba126700 1 -- 172.21.15.16:0/1794763939 >> 172.21.15.16:0/1794763939 conn(0x7f6db40b1fd0 msgr2=0x7f6db40b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:47.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.086+0000 7f6dba126700 1 -- 172.21.15.16:0/1794763939 shutdown_connections 2023-10-16T12:26:47.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.086+0000 7f6dba126700 1 -- 172.21.15.16:0/1794763939 wait complete. 2023-10-16T12:26:47.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.087+0000 7f6dba126700 1 Processor -- start 2023-10-16T12:26:47.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.087+0000 7f6dba126700 1 -- start start 2023-10-16T12:26:47.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.088+0000 7f6dba126700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6db406a7e0 0x7f6db41562c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:47.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.088+0000 7f6dba126700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6db40cc8a0 0x7f6db40c6640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:47.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.088+0000 7f6dba126700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6db415b280 0x7f6db41f6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:47.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.088+0000 7f6dba126700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6db41f5e60 con 0x7f6db40cc8a0 2023-10-16T12:26:47.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.088+0000 7f6dba126700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6db4156800 con 0x7f6db406a7e0 2023-10-16T12:26:47.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.088+0000 7f6dba126700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6db40c6e00 con 0x7f6db415b280 2023-10-16T12:26:47.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.088+0000 7f6db2ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6db40cc8a0 0x7f6db40c6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:47.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.088+0000 7f6db2ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6db40cc8a0 0x7f6db40c6640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40422/0 (socket says 172.21.15.16:40422) 2023-10-16T12:26:47.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.088+0000 7f6db2ffd700 1 -- 172.21.15.16:0/124644523 learned_addr learned my addr 172.21.15.16:0/124644523 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:47.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.088+0000 7f6db3fff700 1 --2- 172.21.15.16:0/124644523 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6db415b280 0x7f6db41f6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:47.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.088+0000 7f6db37fe700 1 --2- 172.21.15.16:0/124644523 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6db406a7e0 0x7f6db41562c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:47.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.088+0000 7f6db2ffd700 1 -- 172.21.15.16:0/124644523 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6db415b280 msgr2=0x7f6db41f6070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:47.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.088+0000 7f6db2ffd700 1 --2- 172.21.15.16:0/124644523 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6db415b280 0x7f6db41f6070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:47.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.089+0000 7f6db2ffd700 1 -- 172.21.15.16:0/124644523 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6db406a7e0 msgr2=0x7f6db41562c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:47.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.089+0000 7f6db2ffd700 1 --2- 172.21.15.16:0/124644523 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6db406a7e0 0x7f6db41562c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:47.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.089+0000 7f6db2ffd700 1 -- 172.21.15.16:0/124644523 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dac009b30 con 0x7f6db40cc8a0 2023-10-16T12:26:47.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.089+0000 7f6db3fff700 1 --2- 172.21.15.16:0/124644523 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6db415b280 0x7f6db41f6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:26:47.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.089+0000 7f6db37fe700 1 --2- 172.21.15.16:0/124644523 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6db406a7e0 0x7f6db41562c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:26:47.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.089+0000 7f6db2ffd700 1 --2- 172.21.15.16:0/124644523 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6db40cc8a0 0x7f6db40c6640 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f6da800c4e0 tx=0x7f6da800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:47.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.089+0000 7f6d9bfff700 1 -- 172.21.15.16:0/124644523 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6da80052d0 con 0x7f6db40cc8a0 2023-10-16T12:26:47.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.089+0000 7f6d9bfff700 1 -- 172.21.15.16:0/124644523 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6da8005430 con 0x7f6db40cc8a0 2023-10-16T12:26:47.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.089+0000 7f6dba126700 1 -- 172.21.15.16:0/124644523 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6db40c6b80 con 0x7f6db40cc8a0 2023-10-16T12:26:47.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.089+0000 7f6d9bfff700 1 -- 172.21.15.16:0/124644523 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6da80038c0 con 0x7f6db40cc8a0 2023-10-16T12:26:47.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.089+0000 7f6dba126700 1 -- 172.21.15.16:0/124644523 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6db415c210 con 0x7f6db40cc8a0 2023-10-16T12:26:47.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.090+0000 7f6dba126700 1 -- 172.21.15.16:0/124644523 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6db404ebb0 con 0x7f6db40cc8a0 2023-10-16T12:26:47.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.090+0000 7f6d9bfff700 1 -- 172.21.15.16:0/124644523 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 38) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f6da8003a20 con 0x7f6db40cc8a0 2023-10-16T12:26:47.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.091+0000 7f6d9bfff700 1 -- 172.21.15.16:0/124644523 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6745+0+0 (secure 0 0 0) 0x7f6da8088800 con 0x7f6db40cc8a0 2023-10-16T12:26:47.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.095+0000 7f6d9bfff700 1 -- 172.21.15.16:0/124644523 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6da80538f0 con 0x7f6db40cc8a0 2023-10-16T12:26:47.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.318+0000 7f6d9bfff700 1 -- 172.21.15.16:0/124644523 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 39) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f6da80533a0 con 0x7f6db40cc8a0 2023-10-16T12:26:47.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.319+0000 7f6d9bfff700 1 --2- 172.21.15.16:0/124644523 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f6da0070bb0 0x7f6da0072fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:47.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.319+0000 7f6db37fe700 1 --2- 172.21.15.16:0/124644523 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f6da0070bb0 0x7f6da0072fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:47.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.320+0000 7f6db37fe700 1 --2- 172.21.15.16:0/124644523 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f6da0070bb0 0x7f6da0072fa0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f6da4007a50 tx=0x7f6da4008040 comp rx=0 tx=0).ready entity=mgr.6020 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:47.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.354+0000 7f6dba126700 1 -- 172.21.15.16:0/124644523 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f6db406b170 con 0x7f6db40cc8a0 2023-10-16T12:26:47.356 INFO:tasks.ceph.mon.a.smithi016.stderr:2023-10-16T12:26:47.355+0000 7f9fc8535700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-16T12:26:47.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.356+0000 7f6d9bfff700 1 -- 172.21.15.16:0/124644523 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v37) v1 ==== 161+0+0 (secure 0 0 0) 0x7f6db406b170 con 0x7f6db40cc8a0 2023-10-16T12:26:47.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.360+0000 7f6dba126700 1 -- 172.21.15.16:0/124644523 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f6da0070bb0 msgr2=0x7f6da0072fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:47.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.360+0000 7f6dba126700 1 --2- 172.21.15.16:0/124644523 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f6da0070bb0 0x7f6da0072fa0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f6da4007a50 tx=0x7f6da4008040 comp rx=0 tx=0).stop 2023-10-16T12:26:47.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.360+0000 7f6dba126700 1 -- 172.21.15.16:0/124644523 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6db40cc8a0 msgr2=0x7f6db40c6640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:47.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.360+0000 7f6dba126700 1 --2- 172.21.15.16:0/124644523 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6db40cc8a0 0x7f6db40c6640 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f6da800c4e0 tx=0x7f6da800c8a0 comp rx=0 tx=0).stop 2023-10-16T12:26:47.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.361+0000 7f6dba126700 1 -- 172.21.15.16:0/124644523 shutdown_connections 2023-10-16T12:26:47.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.361+0000 7f6dba126700 1 --2- 172.21.15.16:0/124644523 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f6da0070bb0 0x7f6da0072fa0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:47.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.361+0000 7f6dba126700 1 --2- 172.21.15.16:0/124644523 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6db415b280 0x7f6db41f6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:47.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.361+0000 7f6dba126700 1 --2- 172.21.15.16:0/124644523 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6db406a7e0 0x7f6db41562c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:47.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.361+0000 7f6dba126700 1 --2- 172.21.15.16:0/124644523 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6db40cc8a0 0x7f6db40c6640 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:47.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.361+0000 7f6dba126700 1 -- 172.21.15.16:0/124644523 >> 172.21.15.16:0/124644523 conn(0x7f6db40b1fd0 msgr2=0x7f6db415a470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:47.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.361+0000 7f6dba126700 1 -- 172.21.15.16:0/124644523 shutdown_connections 2023-10-16T12:26:47.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.361+0000 7f6dba126700 1 -- 172.21.15.16:0/124644523 wait complete. 2023-10-16T12:26:47.365 INFO:teuthology.orchestra.run.smithi016.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-16T12:26:47.365 INFO:tasks.ceph.mds.a.smithi016.stderr: -4837> 2023-10-16T12:26:25.946+0000 7f4c06141700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T12:26:47.376 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-16T12:26:47.393 INFO:tasks.ceph.mds.a.smithi016.stdout:starting mds.a at 2023-10-16T12:26:47.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.502+0000 7ff3ac1f8700 1 Processor -- start 2023-10-16T12:26:47.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.502+0000 7ff3ac1f8700 1 -- start start 2023-10-16T12:26:47.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.503+0000 7ff3ac1f8700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff3a415b790 0x7ff3a40cda60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:47.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.503+0000 7ff3ac1f8700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff3a415bb70 0x7ff3a406d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:47.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.503+0000 7ff3ac1f8700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7ff3a406dea0 0x7ff3a415f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:47.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.503+0000 7ff3ac1f8700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3a406e310 con 0x7ff3a415bb70 2023-10-16T12:26:47.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.503+0000 7ff3ac1f8700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3a40ce0e0 con 0x7ff3a406dea0 2023-10-16T12:26:47.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.503+0000 7ff3ac1f8700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3a406a7e0 con 0x7ff3a415b790 2023-10-16T12:26:47.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.503+0000 7ff3a9793700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff3a415bb70 0x7ff3a406d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:47.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.503+0000 7ff3a9793700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff3a415bb70 0x7ff3a406d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40462/0 (socket says 172.21.15.16:40462) 2023-10-16T12:26:47.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.503+0000 7ff3a9793700 1 -- 172.21.15.16:0/3630166264 learned_addr learned my addr 172.21.15.16:0/3630166264 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:47.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.503+0000 7ff3a9f94700 1 --2- 172.21.15.16:0/3630166264 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff3a415b790 0x7ff3a40cda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:47.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.503+0000 7ff3aa795700 1 --2- 172.21.15.16:0/3630166264 >> v2:172.21.15.49:3300/0 conn(0x7ff3a406dea0 0x7ff3a415f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:47.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.503+0000 7ff3a9793700 1 -- 172.21.15.16:0/3630166264 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff3a415b790 msgr2=0x7ff3a40cda60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:47.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.503+0000 7ff3a9793700 1 --2- 172.21.15.16:0/3630166264 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff3a415b790 0x7ff3a40cda60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:47.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.503+0000 7ff3a9793700 1 -- 172.21.15.16:0/3630166264 >> v2:172.21.15.49:3300/0 conn(0x7ff3a406dea0 msgr2=0x7ff3a415f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:47.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.503+0000 7ff3a9793700 1 --2- 172.21.15.16:0/3630166264 >> v2:172.21.15.49:3300/0 conn(0x7ff3a406dea0 0x7ff3a415f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:47.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.503+0000 7ff3a9793700 1 -- 172.21.15.16:0/3630166264 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3a415c410 con 0x7ff3a415bb70 2023-10-16T12:26:47.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.503+0000 7ff3a9793700 1 --2- 172.21.15.16:0/3630166264 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3a415bb70 0x7ff3a406d960 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7ff3a4070fb0 tx=0x7ff3a0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=670abd804817cb2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:47.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.504+0000 7ff3a8f92700 1 -- 172.21.15.16:0/3630166264 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a0011830 con 0x7ff3a415bb70 2023-10-16T12:26:47.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.504+0000 7ff3a8f92700 1 -- 172.21.15.16:0/3630166264 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3a0011990 con 0x7ff3a415bb70 2023-10-16T12:26:47.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.504+0000 7ff3a8f92700 1 -- 172.21.15.16:0/3630166264 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a0011d20 con 0x7ff3a415bb70 2023-10-16T12:26:47.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.504+0000 7ff3ac1f8700 1 -- 172.21.15.16:0/3630166264 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3a415bb70 msgr2=0x7ff3a406d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:47.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.504+0000 7ff3ac1f8700 1 --2- 172.21.15.16:0/3630166264 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3a415bb70 0x7ff3a406d960 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7ff3a4070fb0 tx=0x7ff3a0009bd0 comp rx=0 tx=0).stop 2023-10-16T12:26:47.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.504+0000 7ff3ac1f8700 1 -- 172.21.15.16:0/3630166264 shutdown_connections 2023-10-16T12:26:47.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.504+0000 7ff3ac1f8700 1 --2- 172.21.15.16:0/3630166264 >> v2:172.21.15.49:3300/0 conn(0x7ff3a406dea0 0x7ff3a415f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:47.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.504+0000 7ff3ac1f8700 1 --2- 172.21.15.16:0/3630166264 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff3a415b790 0x7ff3a40cda60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:47.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.504+0000 7ff3ac1f8700 1 --2- 172.21.15.16:0/3630166264 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3a415bb70 0x7ff3a406d960 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:47.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.504+0000 7ff3ac1f8700 1 -- 172.21.15.16:0/3630166264 >> 172.21.15.16:0/3630166264 conn(0x7ff3a40cc5e0 msgr2=0x7ff3a40cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:47.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.504+0000 7ff3ac1f8700 1 -- 172.21.15.16:0/3630166264 shutdown_connections 2023-10-16T12:26:47.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.504+0000 7ff3ac1f8700 1 -- 172.21.15.16:0/3630166264 wait complete. 2023-10-16T12:26:47.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.506+0000 7ff3ac1f8700 1 Processor -- start 2023-10-16T12:26:47.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.506+0000 7ff3ac1f8700 1 -- start start 2023-10-16T12:26:47.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.507+0000 7ff3ac1f8700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff3a406dea0 0x7ff3a4152340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:47.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.507+0000 7ff3ac1f8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3a415b790 0x7ff3a40c9540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:47.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.507+0000 7ff3ac1f8700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff3a415bb70 0x7ff3a406b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:47.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.507+0000 7ff3ac1f8700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3a40c7440 con 0x7ff3a415b790 2023-10-16T12:26:47.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.507+0000 7ff3ac1f8700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3a4152880 con 0x7ff3a406dea0 2023-10-16T12:26:47.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.507+0000 7ff3ac1f8700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3a40c5ed0 con 0x7ff3a415bb70 2023-10-16T12:26:47.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.507+0000 7ff3a9793700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3a415b790 0x7ff3a40c9540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:47.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.507+0000 7ff3a9793700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3a415b790 0x7ff3a40c9540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40468/0 (socket says 172.21.15.16:40468) 2023-10-16T12:26:47.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.507+0000 7ff3a9793700 1 -- 172.21.15.16:0/2506465920 learned_addr learned my addr 172.21.15.16:0/2506465920 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:47.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.507+0000 7ff3a9f94700 1 --2- 172.21.15.16:0/2506465920 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff3a406dea0 0x7ff3a4152340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:47.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.507+0000 7ff3aa795700 1 --2- 172.21.15.16:0/2506465920 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff3a415bb70 0x7ff3a406b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:47.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.507+0000 7ff3a9793700 1 -- 172.21.15.16:0/2506465920 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff3a415bb70 msgr2=0x7ff3a406b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:47.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.507+0000 7ff3a9793700 1 --2- 172.21.15.16:0/2506465920 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff3a415bb70 0x7ff3a406b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:47.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.507+0000 7ff3a9793700 1 -- 172.21.15.16:0/2506465920 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff3a406dea0 msgr2=0x7ff3a4152340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:47.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.507+0000 7ff3a9793700 1 --2- 172.21.15.16:0/2506465920 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff3a406dea0 0x7ff3a4152340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:47.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.507+0000 7ff3a9793700 1 -- 172.21.15.16:0/2506465920 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3a0009550 con 0x7ff3a415b790 2023-10-16T12:26:47.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.508+0000 7ff3a9793700 1 --2- 172.21.15.16:0/2506465920 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3a415b790 0x7ff3a40c9540 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7ff3a0000c00 tx=0x7ff3a0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:47.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.508+0000 7ff39a7fc700 1 -- 172.21.15.16:0/2506465920 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a0004300 con 0x7ff3a415b790 2023-10-16T12:26:47.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.508+0000 7ff39a7fc700 1 -- 172.21.15.16:0/2506465920 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3a001f070 con 0x7ff3a415b790 2023-10-16T12:26:47.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.508+0000 7ff3ac1f8700 1 -- 172.21.15.16:0/2506465920 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3a40c61b0 con 0x7ff3a415b790 2023-10-16T12:26:47.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.508+0000 7ff39a7fc700 1 -- 172.21.15.16:0/2506465920 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a00036a0 con 0x7ff3a415b790 2023-10-16T12:26:47.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.508+0000 7ff3ac1f8700 1 -- 172.21.15.16:0/2506465920 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3a40c70d0 con 0x7ff3a415b790 2023-10-16T12:26:47.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.509+0000 7ff3ac1f8700 1 -- 172.21.15.16:0/2506465920 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3a404ebb0 con 0x7ff3a415b790 2023-10-16T12:26:47.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.509+0000 7ff39a7fc700 1 -- 172.21.15.16:0/2506465920 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 39) v1 ==== 89067+0+0 (secure 0 0 0) 0x7ff3a0021030 con 0x7ff3a415b790 2023-10-16T12:26:47.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.510+0000 7ff39a7fc700 1 --2- 172.21.15.16:0/2506465920 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7ff38806ad50 0x7ff38806d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:47.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.510+0000 7ff39a7fc700 1 -- 172.21.15.16:0/2506465920 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6831+0+0 (secure 0 0 0) 0x7ff3a008a880 con 0x7ff3a415b790 2023-10-16T12:26:47.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.510+0000 7ff3a9f94700 1 --2- 172.21.15.16:0/2506465920 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7ff38806ad50 0x7ff38806d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:47.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.510+0000 7ff3a9f94700 1 --2- 172.21.15.16:0/2506465920 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7ff38806ad50 0x7ff38806d210 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff394009ac0 tx=0x7ff394009210 comp rx=0 tx=0).ready entity=mgr.6020 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:47.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.514+0000 7ff39a7fc700 1 -- 172.21.15.16:0/2506465920 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff3a0055930 con 0x7ff3a415b790 2023-10-16T12:26:47.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:47.761+0000 7ff3ac1f8700 1 -- 172.21.15.16:0/2506465920 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7ff3a40cedb0 con 0x7ff3a415b790 2023-10-16T12:26:48.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.358+0000 7ff39a7fc700 1 -- 172.21.15.16:0/2506465920 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v38) v1 ==== 104+0+0 (secure 0 0 0) 0x7ff3a0058f50 con 0x7ff3a415b790 2023-10-16T12:26:48.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.361+0000 7ff3ac1f8700 1 -- 172.21.15.16:0/2506465920 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7ff38806ad50 msgr2=0x7ff38806d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:48.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.361+0000 7ff3ac1f8700 1 --2- 172.21.15.16:0/2506465920 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7ff38806ad50 0x7ff38806d210 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff394009ac0 tx=0x7ff394009210 comp rx=0 tx=0).stop 2023-10-16T12:26:48.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.361+0000 7ff3ac1f8700 1 -- 172.21.15.16:0/2506465920 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3a415b790 msgr2=0x7ff3a40c9540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:48.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.361+0000 7ff3ac1f8700 1 --2- 172.21.15.16:0/2506465920 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3a415b790 0x7ff3a40c9540 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7ff3a0000c00 tx=0x7ff3a0004750 comp rx=0 tx=0).stop 2023-10-16T12:26:48.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.361+0000 7ff3ac1f8700 1 -- 172.21.15.16:0/2506465920 shutdown_connections 2023-10-16T12:26:48.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.361+0000 7ff3ac1f8700 1 --2- 172.21.15.16:0/2506465920 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7ff38806ad50 0x7ff38806d210 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:48.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.361+0000 7ff3ac1f8700 1 --2- 172.21.15.16:0/2506465920 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff3a415bb70 0x7ff3a406b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:48.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.361+0000 7ff3ac1f8700 1 --2- 172.21.15.16:0/2506465920 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff3a406dea0 0x7ff3a4152340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:48.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.361+0000 7ff3ac1f8700 1 --2- 172.21.15.16:0/2506465920 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3a415b790 0x7ff3a40c9540 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:48.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.361+0000 7ff3ac1f8700 1 -- 172.21.15.16:0/2506465920 >> 172.21.15.16:0/2506465920 conn(0x7ff3a40cc5e0 msgr2=0x7ff3a415a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:48.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.361+0000 7ff3ac1f8700 1 -- 172.21.15.16:0/2506465920 shutdown_connections 2023-10-16T12:26:48.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.361+0000 7ff3ac1f8700 1 -- 172.21.15.16:0/2506465920 wait complete. 2023-10-16T12:26:48.380 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-16T12:26:48.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.550+0000 7fb5dbc77700 1 Processor -- start 2023-10-16T12:26:48.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.550+0000 7fb5dbc77700 1 -- start start 2023-10-16T12:26:48.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.551+0000 7fb5dbc77700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb5d4155a60 0x7fb5d415fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:48.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.551+0000 7fb5dbc77700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb5d4155e40 0x7fb5d415bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:48.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.551+0000 7fb5dbc77700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fb5d415c190 0x7fb5d40cba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:48.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.551+0000 7fb5dbc77700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5d415f240 con 0x7fb5d4155a60 2023-10-16T12:26:48.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.551+0000 7fb5dbc77700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5d4160190 con 0x7fb5d415c190 2023-10-16T12:26:48.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.551+0000 7fb5dbc77700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5d415c780 con 0x7fb5d4155e40 2023-10-16T12:26:48.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.551+0000 7fb5d9a13700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb5d4155a60 0x7fb5d415fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:48.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.552+0000 7fb5d9a13700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb5d4155a60 0x7fb5d415fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40494/0 (socket says 172.21.15.16:40494) 2023-10-16T12:26:48.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.552+0000 7fb5d9a13700 1 -- 172.21.15.16:0/1137841519 learned_addr learned my addr 172.21.15.16:0/1137841519 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:48.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.552+0000 7fb5d9212700 1 --2- 172.21.15.16:0/1137841519 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb5d4155e40 0x7fb5d415bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:48.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.552+0000 7fb5da214700 1 --2- 172.21.15.16:0/1137841519 >> v2:172.21.15.49:3300/0 conn(0x7fb5d415c190 0x7fb5d40cba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:48.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.552+0000 7fb5d9a13700 1 -- 172.21.15.16:0/1137841519 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb5d4155e40 msgr2=0x7fb5d415bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:48.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.552+0000 7fb5d9a13700 1 --2- 172.21.15.16:0/1137841519 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb5d4155e40 0x7fb5d415bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:48.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.552+0000 7fb5d9a13700 1 -- 172.21.15.16:0/1137841519 >> v2:172.21.15.49:3300/0 conn(0x7fb5d415c190 msgr2=0x7fb5d40cba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:48.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.552+0000 7fb5d9a13700 1 --2- 172.21.15.16:0/1137841519 >> v2:172.21.15.49:3300/0 conn(0x7fb5d415c190 0x7fb5d40cba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:48.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.552+0000 7fb5d9a13700 1 -- 172.21.15.16:0/1137841519 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5d415ca60 con 0x7fb5d4155a60 2023-10-16T12:26:48.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.552+0000 7fb5d9a13700 1 --2- 172.21.15.16:0/1137841519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5d4155a60 0x7fb5d415fc50 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7fb5d40c8450 tx=0x7fb5c4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7c9b4ea77e5a8b5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:48.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.552+0000 7fb5d8a11700 1 -- 172.21.15.16:0/1137841519 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5c4011830 con 0x7fb5d4155a60 2023-10-16T12:26:48.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.552+0000 7fb5d8a11700 1 -- 172.21.15.16:0/1137841519 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5c4011990 con 0x7fb5d4155a60 2023-10-16T12:26:48.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.552+0000 7fb5d8a11700 1 -- 172.21.15.16:0/1137841519 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5c4011d20 con 0x7fb5d4155a60 2023-10-16T12:26:48.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.553+0000 7fb5dbc77700 1 -- 172.21.15.16:0/1137841519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5d4155a60 msgr2=0x7fb5d415fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:48.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.553+0000 7fb5dbc77700 1 --2- 172.21.15.16:0/1137841519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5d4155a60 0x7fb5d415fc50 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7fb5d40c8450 tx=0x7fb5c4009bd0 comp rx=0 tx=0).stop 2023-10-16T12:26:48.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.553+0000 7fb5dbc77700 1 -- 172.21.15.16:0/1137841519 shutdown_connections 2023-10-16T12:26:48.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.553+0000 7fb5dbc77700 1 --2- 172.21.15.16:0/1137841519 >> v2:172.21.15.49:3300/0 conn(0x7fb5d415c190 0x7fb5d40cba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:48.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.553+0000 7fb5dbc77700 1 --2- 172.21.15.16:0/1137841519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5d4155a60 0x7fb5d415fc50 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:48.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.553+0000 7fb5dbc77700 1 --2- 172.21.15.16:0/1137841519 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb5d4155e40 0x7fb5d415bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:48.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.553+0000 7fb5dbc77700 1 -- 172.21.15.16:0/1137841519 >> 172.21.15.16:0/1137841519 conn(0x7fb5d406a7e0 msgr2=0x7fb5d406abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:48.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.553+0000 7fb5dbc77700 1 -- 172.21.15.16:0/1137841519 shutdown_connections 2023-10-16T12:26:48.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.553+0000 7fb5dbc77700 1 -- 172.21.15.16:0/1137841519 wait complete. 2023-10-16T12:26:48.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.554+0000 7fb5dbc77700 1 Processor -- start 2023-10-16T12:26:48.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.554+0000 7fb5dbc77700 1 -- start start 2023-10-16T12:26:48.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.555+0000 7fb5dbc77700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5d4155a60 0x7fb5d40b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:48.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.555+0000 7fb5dbc77700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb5d4155e40 0x7fb5d4157b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:48.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.555+0000 7fb5dbc77700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb5d415c190 0x7fb5d4157190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:48.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.555+0000 7fb5dbc77700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5d4158290 con 0x7fb5d4155a60 2023-10-16T12:26:48.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.555+0000 7fb5dbc77700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5d41563d0 con 0x7fb5d4155e40 2023-10-16T12:26:48.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.555+0000 7fb5dbc77700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5d41566b0 con 0x7fb5d415c190 2023-10-16T12:26:48.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.555+0000 7fb5d9a13700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5d4155a60 0x7fb5d40b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:48.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.555+0000 7fb5d9a13700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5d4155a60 0x7fb5d40b6a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40500/0 (socket says 172.21.15.16:40500) 2023-10-16T12:26:48.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.555+0000 7fb5d9a13700 1 -- 172.21.15.16:0/3753826829 learned_addr learned my addr 172.21.15.16:0/3753826829 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:48.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.555+0000 7fb5da214700 1 --2- 172.21.15.16:0/3753826829 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb5d415c190 0x7fb5d4157190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:48.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.555+0000 7fb5d9212700 1 --2- 172.21.15.16:0/3753826829 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb5d4155e40 0x7fb5d4157b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:48.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.555+0000 7fb5d9a13700 1 -- 172.21.15.16:0/3753826829 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb5d415c190 msgr2=0x7fb5d4157190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:48.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.555+0000 7fb5d9a13700 1 --2- 172.21.15.16:0/3753826829 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb5d415c190 0x7fb5d4157190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:48.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.555+0000 7fb5d9a13700 1 -- 172.21.15.16:0/3753826829 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb5d4155e40 msgr2=0x7fb5d4157b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:48.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.555+0000 7fb5d9a13700 1 --2- 172.21.15.16:0/3753826829 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb5d4155e40 0x7fb5d4157b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:48.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.555+0000 7fb5d9a13700 1 -- 172.21.15.16:0/3753826829 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5c4009550 con 0x7fb5d4155a60 2023-10-16T12:26:48.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.555+0000 7fb5d9a13700 1 --2- 172.21.15.16:0/3753826829 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5d4155a60 0x7fb5d40b6a90 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fb5c4000c00 tx=0x7fb5c4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:48.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.555+0000 7fb5ca7fc700 1 -- 172.21.15.16:0/3753826829 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5c4004240 con 0x7fb5d4155a60 2023-10-16T12:26:48.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.556+0000 7fb5ca7fc700 1 -- 172.21.15.16:0/3753826829 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5c401f070 con 0x7fb5d4155a60 2023-10-16T12:26:48.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.556+0000 7fb5ca7fc700 1 -- 172.21.15.16:0/3753826829 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5c40045e0 con 0x7fb5d4155a60 2023-10-16T12:26:48.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.556+0000 7fb5dbc77700 1 -- 172.21.15.16:0/3753826829 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5d4156990 con 0x7fb5d4155a60 2023-10-16T12:26:48.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.556+0000 7fb5dbc77700 1 -- 172.21.15.16:0/3753826829 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5d41576d0 con 0x7fb5d4155a60 2023-10-16T12:26:48.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.557+0000 7fb5dbc77700 1 -- 172.21.15.16:0/3753826829 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5d404ec20 con 0x7fb5d4155a60 2023-10-16T12:26:48.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.557+0000 7fb5ca7fc700 1 -- 172.21.15.16:0/3753826829 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 39) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fb5c4021030 con 0x7fb5d4155a60 2023-10-16T12:26:48.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.557+0000 7fb5ca7fc700 1 --2- 172.21.15.16:0/3753826829 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7fb5b806ad50 0x7fb5b806d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:48.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.557+0000 7fb5ca7fc700 1 -- 172.21.15.16:0/3753826829 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6831+0+0 (secure 0 0 0) 0x7fb5c408a7d0 con 0x7fb5d4155a60 2023-10-16T12:26:48.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.557+0000 7fb5d9212700 1 --2- 172.21.15.16:0/3753826829 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7fb5b806ad50 0x7fb5b806d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:48.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.558+0000 7fb5d9212700 1 --2- 172.21.15.16:0/3753826829 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7fb5b806ad50 0x7fb5b806d210 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fb5d41f3fb0 tx=0x7fb5d0009210 comp rx=0 tx=0).ready entity=mgr.6020 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:48.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.562+0000 7fb5ca7fc700 1 -- 172.21.15.16:0/3753826829 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb5c4055690 con 0x7fb5d4155a60 2023-10-16T12:26:48.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.768+0000 7fb5dbc77700 1 -- 172.21.15.16:0/3753826829 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb5d40c4500 con 0x7fb5d4155a60 2023-10-16T12:26:48.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:48.782+0000 7fb5ca7fc700 1 -- 172.21.15.16:0/3753826829 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 40) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fb5c4055a60 con 0x7fb5d4155a60 2023-10-16T12:26:49.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:49.359+0000 7fb5ca7fc700 1 -- 172.21.15.16:0/3753826829 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v89) v1 ==== 181+0+0 (secure 0 0 0) 0x7fb5c406f770 con 0x7fb5d4155a60 2023-10-16T12:26:49.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:49.363+0000 7fb5dbc77700 1 -- 172.21.15.16:0/3753826829 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7fb5b806ad50 msgr2=0x7fb5b806d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:49.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:49.363+0000 7fb5dbc77700 1 --2- 172.21.15.16:0/3753826829 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7fb5b806ad50 0x7fb5b806d210 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fb5d41f3fb0 tx=0x7fb5d0009210 comp rx=0 tx=0).stop 2023-10-16T12:26:49.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:49.363+0000 7fb5dbc77700 1 -- 172.21.15.16:0/3753826829 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5d4155a60 msgr2=0x7fb5d40b6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:49.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:49.363+0000 7fb5dbc77700 1 --2- 172.21.15.16:0/3753826829 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5d4155a60 0x7fb5d40b6a90 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fb5c4000c00 tx=0x7fb5c4004750 comp rx=0 tx=0).stop 2023-10-16T12:26:49.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:49.363+0000 7fb5dbc77700 1 -- 172.21.15.16:0/3753826829 shutdown_connections 2023-10-16T12:26:49.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:49.363+0000 7fb5dbc77700 1 --2- 172.21.15.16:0/3753826829 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7fb5b806ad50 0x7fb5b806d210 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:49.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:49.363+0000 7fb5dbc77700 1 --2- 172.21.15.16:0/3753826829 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb5d415c190 0x7fb5d4157190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:49.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:49.363+0000 7fb5dbc77700 1 --2- 172.21.15.16:0/3753826829 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5d4155a60 0x7fb5d40b6a90 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:49.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:49.363+0000 7fb5dbc77700 1 --2- 172.21.15.16:0/3753826829 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb5d4155e40 0x7fb5d4157b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:49.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:49.363+0000 7fb5dbc77700 1 -- 172.21.15.16:0/3753826829 >> 172.21.15.16:0/3753826829 conn(0x7fb5d406a7e0 msgr2=0x7fb5d4159870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:49.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:49.363+0000 7fb5dbc77700 1 -- 172.21.15.16:0/3753826829 shutdown_connections 2023-10-16T12:26:49.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:49.363+0000 7fb5dbc77700 1 -- 172.21.15.16:0/3753826829 wait complete. 2023-10-16T12:26:49.368 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' removed 2023-10-16T12:26:49.380 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-16T12:26:49.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:49.515+0000 7f3980810700 1 Processor -- start 2023-10-16T12:26:49.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:49.515+0000 7f3980810700 1 -- start start 2023-10-16T12:26:49.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:49.516+0000 7f3980810700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f397c161ed0 0x7f397c0cde70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:49.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:49.516+0000 7f3980810700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f397c06a7e0 0x7f397c06abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:49.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:49.516+0000 7f3980810700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f397c158290 con 0x7f397c15e300 2023-10-16T12:26:49.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:49.516+0000 7f3980810700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f397c162360 con 0x7f397c161ed0 2023-10-16T12:26:49.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:49.516+0000 7f3980810700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f397c0ce3b0 con 0x7f397c06a7e0 2023-10-16T12:26:49.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:49.516+0000 7f3979d9b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f397c161ed0 0x7f397c0cde70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:49.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:49.516+0000 7f3979d9b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f397c161ed0 0x7f397c0cde70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40508/0 (socket says 172.21.15.16:40508) 2023-10-16T12:26:49.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:49.516+0000 7f3979d9b700 1 -- 172.21.15.16:0/3125563696 learned_addr learned my addr 172.21.15.16:0/3125563696 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:49.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:49.516+0000 7f3979d9b700 1 -- 172.21.15.16:0/3125563696 >> v2:172.21.15.49:3300/0 conn(0x7f397c06a7e0 msgr2=0x7f397c06abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:49.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:49.516+0000 7f397ad9d700 1 --2- 172.21.15.16:0/3125563696 >> v2:172.21.15.49:3300/0 conn(0x7f397c06a7e0 0x7f397c06abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:49.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:49.516+0000 7f3979d9b700 1 --2- 172.21.15.16:0/3125563696 >> v2:172.21.15.49:3300/0 conn(0x7f397c06a7e0 0x7f397c06abf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:49.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:49.516+0000 7f3979d9b700 1 -- 172.21.15.16:0/3125563696 >> v1:172.21.15.49:6789/0 conn(0x7f397c15e300 legacy=0x7f397c15e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:49.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:49.516+0000 7f3979d9b700 1 -- 172.21.15.16:0/3125563696 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f397c06b130 con 0x7f397c161ed0 2023-10-16T12:26:49.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:49.517+0000 7f3979d9b700 1 --2- 172.21.15.16:0/3125563696 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f397c161ed0 0x7f397c0cde70 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f397c15b930 tx=0x7f3970009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=952ed58996d17ad0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:49.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:49.517+0000 7f397959a700 1 -- 172.21.15.16:0/3125563696 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3970004230 con 0x7f397c161ed0 2023-10-16T12:26:49.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:49.517+0000 7f397959a700 1 -- 172.21.15.16:0/3125563696 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3970004390 con 0x7f397c161ed0 2023-10-16T12:26:49.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:49.517+0000 7f397959a700 1 -- 172.21.15.16:0/3125563696 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39700036a0 con 0x7f397c161ed0 2023-10-16T12:26:49.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:49.517+0000 7f3980810700 1 -- 172.21.15.16:0/3125563696 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f397c161ed0 msgr2=0x7f397c0cde70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:49.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:49.517+0000 7f3980810700 1 --2- 172.21.15.16:0/3125563696 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f397c161ed0 0x7f397c0cde70 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f397c15b930 tx=0x7f3970009bd0 comp rx=0 tx=0).stop 2023-10-16T12:26:49.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:49.517+0000 7f3980810700 1 -- 172.21.15.16:0/3125563696 shutdown_connections 2023-10-16T12:26:49.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:49.517+0000 7f3980810700 1 --2- 172.21.15.16:0/3125563696 >> v2:172.21.15.49:3300/0 conn(0x7f397c06a7e0 0x7f397c06abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:49.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:49.517+0000 7f3980810700 1 --2- 172.21.15.16:0/3125563696 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f397c161ed0 0x7f397c0cde70 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:49.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:49.517+0000 7f3980810700 1 -- 172.21.15.16:0/3125563696 >> 172.21.15.16:0/3125563696 conn(0x7f397c0d02a0 msgr2=0x7f397c0d06b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:49.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:49.517+0000 7f3980810700 1 -- 172.21.15.16:0/3125563696 shutdown_connections 2023-10-16T12:26:49.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:49.517+0000 7f3980810700 1 -- 172.21.15.16:0/3125563696 wait complete. 2023-10-16T12:26:49.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:49.519+0000 7f3980810700 1 Processor -- start 2023-10-16T12:26:49.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:49.519+0000 7f3980810700 1 -- start start 2023-10-16T12:26:49.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:49.519+0000 7f3980810700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f397c06a7e0 0x7f397c0b5a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:49.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:49.519+0000 7f3980810700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f397c15e300 0x7f397c155790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:49.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:49.519+0000 7f3980810700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f397c161ed0 0x7f397c0b2420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:49.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:49.519+0000 7f3980810700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f397c0b1f90 con 0x7f397c15e300 2023-10-16T12:26:49.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:49.519+0000 7f3980810700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f397c0af530 con 0x7f397c161ed0 2023-10-16T12:26:49.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:49.519+0000 7f3980810700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f397c0b1940 con 0x7f397c06a7e0 2023-10-16T12:26:49.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:49.520+0000 7f3979d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f397c15e300 0x7f397c155790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:49.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:49.520+0000 7f3979d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f397c15e300 0x7f397c155790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40514/0 (socket says 172.21.15.16:40514) 2023-10-16T12:26:49.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:49.520+0000 7f3979d9b700 1 -- 172.21.15.16:0/3893923638 learned_addr learned my addr 172.21.15.16:0/3893923638 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:49.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:49.520+0000 7f397a59c700 1 --2- 172.21.15.16:0/3893923638 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f397c06a7e0 0x7f397c0b5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:49.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:49.520+0000 7f397ad9d700 1 --2- 172.21.15.16:0/3893923638 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f397c161ed0 0x7f397c0b2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:49.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:49.520+0000 7f3979d9b700 1 -- 172.21.15.16:0/3893923638 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f397c06a7e0 msgr2=0x7f397c0b5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:49.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:49.520+0000 7f3979d9b700 1 --2- 172.21.15.16:0/3893923638 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f397c06a7e0 0x7f397c0b5a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:49.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:49.520+0000 7f3979d9b700 1 -- 172.21.15.16:0/3893923638 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f397c161ed0 msgr2=0x7f397c0b2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:49.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:49.520+0000 7f3979d9b700 1 --2- 172.21.15.16:0/3893923638 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f397c161ed0 0x7f397c0b2420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:49.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:49.520+0000 7f3979d9b700 1 -- 172.21.15.16:0/3893923638 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3970009550 con 0x7f397c15e300 2023-10-16T12:26:49.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:49.520+0000 7f397ad9d700 1 --2- 172.21.15.16:0/3893923638 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f397c161ed0 0x7f397c0b2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:26:49.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:49.520+0000 7f3979d9b700 1 --2- 172.21.15.16:0/3893923638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f397c15e300 0x7f397c155790 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f3970009130 tx=0x7f3970004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:49.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:49.520+0000 7f396affd700 1 -- 172.21.15.16:0/3893923638 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3970011630 con 0x7f397c15e300 2023-10-16T12:26:49.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:49.520+0000 7f396affd700 1 -- 172.21.15.16:0/3893923638 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3970011790 con 0x7f397c15e300 2023-10-16T12:26:49.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:49.520+0000 7f3980810700 1 -- 172.21.15.16:0/3893923638 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f397c0b1c20 con 0x7f397c15e300 2023-10-16T12:26:49.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:49.520+0000 7f396affd700 1 -- 172.21.15.16:0/3893923638 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3970011b40 con 0x7f397c15e300 2023-10-16T12:26:49.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:49.520+0000 7f3980810700 1 -- 172.21.15.16:0/3893923638 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f397c0b15f0 con 0x7f397c15e300 2023-10-16T12:26:49.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:49.521+0000 7f3980810700 1 -- 172.21.15.16:0/3893923638 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f397c04ec20 con 0x7f397c15e300 2023-10-16T12:26:49.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:49.522+0000 7f396affd700 1 -- 172.21.15.16:0/3893923638 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 40) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f3970011ca0 con 0x7f397c15e300 2023-10-16T12:26:49.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:49.522+0000 7f396affd700 1 --2- 172.21.15.16:0/3893923638 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f395c06af90 0x7f395c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:49.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:49.522+0000 7f396affd700 1 -- 172.21.15.16:0/3893923638 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6393+0+0 (secure 0 0 0) 0x7f397008aee0 con 0x7f397c15e300 2023-10-16T12:26:49.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:49.522+0000 7f397a59c700 1 --2- 172.21.15.16:0/3893923638 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f395c06af90 0x7f395c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:49.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:49.523+0000 7f397a59c700 1 --2- 172.21.15.16:0/3893923638 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f395c06af90 0x7f395c06d450 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f397c0cca70 tx=0x7f396c009040 comp rx=0 tx=0).ready entity=mgr.6020 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:49.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:49.527+0000 7f396affd700 1 -- 172.21.15.16:0/3893923638 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3970055d80 con 0x7f397c15e300 2023-10-16T12:26:49.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:49.731+0000 7f3980810700 1 -- 172.21.15.16:0/3893923638 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f397c0b6220 con 0x7f397c15e300 2023-10-16T12:26:50.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.361+0000 7f396affd700 1 -- 172.21.15.16:0/3893923638 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v90) v1 ==== 169+0+0 (secure 0 0 0) 0x7f39700593a0 con 0x7f397c15e300 2023-10-16T12:26:50.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.367+0000 7f3980810700 1 -- 172.21.15.16:0/3893923638 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f395c06af90 msgr2=0x7f395c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:50.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.367+0000 7f3980810700 1 --2- 172.21.15.16:0/3893923638 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f395c06af90 0x7f395c06d450 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f397c0cca70 tx=0x7f396c009040 comp rx=0 tx=0).stop 2023-10-16T12:26:50.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.367+0000 7f3980810700 1 -- 172.21.15.16:0/3893923638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f397c15e300 msgr2=0x7f397c155790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:50.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.367+0000 7f3980810700 1 --2- 172.21.15.16:0/3893923638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f397c15e300 0x7f397c155790 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f3970009130 tx=0x7f3970004750 comp rx=0 tx=0).stop 2023-10-16T12:26:50.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.368+0000 7f3980810700 1 -- 172.21.15.16:0/3893923638 shutdown_connections 2023-10-16T12:26:50.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.368+0000 7f3980810700 1 --2- 172.21.15.16:0/3893923638 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f395c06af90 0x7f395c06d450 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:50.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.368+0000 7f3980810700 1 --2- 172.21.15.16:0/3893923638 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f397c161ed0 0x7f397c0b2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:50.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.368+0000 7f3980810700 1 --2- 172.21.15.16:0/3893923638 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f397c06a7e0 0x7f397c0b5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:50.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.368+0000 7f3980810700 1 --2- 172.21.15.16:0/3893923638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f397c15e300 0x7f397c155790 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:50.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.368+0000 7f3980810700 1 -- 172.21.15.16:0/3893923638 >> 172.21.15.16:0/3893923638 conn(0x7f397c0d02a0 msgr2=0x7f397c1606f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:50.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.368+0000 7f3980810700 1 -- 172.21.15.16:0/3893923638 shutdown_connections 2023-10-16T12:26:50.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.369+0000 7f3980810700 1 -- 172.21.15.16:0/3893923638 wait complete. 2023-10-16T12:26:50.376 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' removed 2023-10-16T12:26:50.389 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-16T12:26:50.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.521+0000 7f6cfc0e7700 1 Processor -- start 2023-10-16T12:26:50.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.522+0000 7f6cfc0e7700 1 -- start start 2023-10-16T12:26:50.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.523+0000 7f6cfc0e7700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6cf40cf4c0 0x7f6cf40ca1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:50.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.523+0000 7f6cfc0e7700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cf415dda0 con 0x7f6cf4159de0 2023-10-16T12:26:50.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.523+0000 7f6cfc0e7700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cf415e080 con 0x7f6cf406bad0 2023-10-16T12:26:50.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.523+0000 7f6cfc0e7700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf415ea20 con 0x7f6cf40cf4c0 2023-10-16T12:26:50.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.523+0000 7f6cf9e83700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f6cf4159de0 0x7f6cf415a1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:58028/0 (socket says 172.21.15.16:58028) 2023-10-16T12:26:50.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.523+0000 7f6cf9e83700 1 -- 172.21.15.16:0/357011924 learned_addr learned my addr 172.21.15.16:0/357011924 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:26:50.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.523+0000 7f6cf9682700 1 --2- 172.21.15.16:0/357011924 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6cf40cf4c0 0x7f6cf40ca1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:50.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.523+0000 7f6cf8e81700 1 -- 172.21.15.16:0/357011924 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 802643428 0 0) 0x7f6cf415dda0 con 0x7f6cf4159de0 2023-10-16T12:26:50.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.523+0000 7f6cf8e81700 1 -- 172.21.15.16:0/357011924 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ce00035b0 con 0x7f6cf4159de0 2023-10-16T12:26:50.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.523+0000 7f6cf8e81700 1 -- 172.21.15.16:0/357011924 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1340981021 0 0) 0x7f6ce00035b0 con 0x7f6cf4159de0 2023-10-16T12:26:50.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.523+0000 7f6cf8e81700 1 -- 172.21.15.16:0/357011924 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6cf40cf4c0 msgr2=0x7f6cf40ca1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:50.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.523+0000 7f6cf8e81700 1 --2- 172.21.15.16:0/357011924 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6cf40cf4c0 0x7f6cf40ca1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:50.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.524+0000 7f6cf8e81700 1 -- 172.21.15.16:0/357011924 >> v1:172.21.15.49:6789/0 conn(0x7f6cf406bad0 legacy=0x7f6cf40ca6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:50.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.524+0000 7f6cf8e81700 1 -- 172.21.15.16:0/357011924 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cf415e360 con 0x7f6cf4159de0 2023-10-16T12:26:50.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.524+0000 7f6cf8e81700 1 -- 172.21.15.16:0/357011924 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f6ce4001670 con 0x7f6cf4159de0 2023-10-16T12:26:50.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.524+0000 7f6cf8e81700 1 -- 172.21.15.16:0/357011924 >> v1:172.21.15.16:6789/0 conn(0x7f6cf4159de0 legacy=0x7f6cf415a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:50.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.524+0000 7f6cf8e81700 1 --2- 172.21.15.16:0/357011924 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6ce0003a30 0x7f6ce0003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:50.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.524+0000 7f6cf8e81700 1 --2- 172.21.15.16:0/357011924 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6cf40cf4c0 0x7f6ce0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:50.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.524+0000 7f6cf8e81700 1 --2- 172.21.15.16:0/357011924 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ce0004d90 0x7f6ce0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:50.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.524+0000 7f6cf8e81700 1 -- 172.21.15.16:0/357011924 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf415ea20 con 0x7f6ce0004d90 2023-10-16T12:26:50.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.524+0000 7f6cf8e81700 1 -- 172.21.15.16:0/357011924 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce000cc50 con 0x7f6ce0003a30 2023-10-16T12:26:50.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.524+0000 7f6cf8e81700 1 -- 172.21.15.16:0/357011924 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce000ce10 con 0x7f6cf40cf4c0 2023-10-16T12:26:50.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.524+0000 7f6cfa684700 1 --2- 172.21.15.16:0/357011924 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ce0004d90 0x7f6ce0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:50.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.524+0000 7f6cf9e83700 1 --2- 172.21.15.16:0/357011924 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6ce0003a30 0x7f6ce0003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:50.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.524+0000 7f6cf9682700 1 --2- 172.21.15.16:0/357011924 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6cf40cf4c0 0x7f6ce0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:50.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.524+0000 7f6cfa684700 1 -- 172.21.15.16:0/357011924 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6cf40cf4c0 msgr2=0x7f6ce0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:50.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.524+0000 7f6cfa684700 1 --2- 172.21.15.16:0/357011924 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6cf40cf4c0 0x7f6ce0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:50.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.524+0000 7f6cfa684700 1 -- 172.21.15.16:0/357011924 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6ce0003a30 msgr2=0x7f6ce0003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:50.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.524+0000 7f6cfa684700 1 --2- 172.21.15.16:0/357011924 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6ce0003a30 0x7f6ce0003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:50.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.524+0000 7f6cfa684700 1 -- 172.21.15.16:0/357011924 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ce00035b0 con 0x7f6ce0004d90 2023-10-16T12:26:50.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.524+0000 7f6cfa684700 1 --2- 172.21.15.16:0/357011924 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ce0004d90 0x7f6ce0005210 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f6cec00c160 tx=0x7f6ce0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6598fff7293712db server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:50.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.524+0000 7f6cf8e81700 1 -- 172.21.15.16:0/357011924 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cec001db0 con 0x7f6ce0004d90 2023-10-16T12:26:50.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.524+0000 7f6cf8e81700 1 -- 172.21.15.16:0/357011924 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cec00cca0 con 0x7f6ce0004d90 2023-10-16T12:26:50.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.524+0000 7f6cf8e81700 1 -- 172.21.15.16:0/357011924 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cec003c40 con 0x7f6ce0004d90 2023-10-16T12:26:50.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.525+0000 7f6cfc0e7700 1 -- 172.21.15.16:0/357011924 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ce0004d90 msgr2=0x7f6ce0005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:50.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.525+0000 7f6cfc0e7700 1 --2- 172.21.15.16:0/357011924 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ce0004d90 0x7f6ce0005210 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f6cec00c160 tx=0x7f6ce0005880 comp rx=0 tx=0).stop 2023-10-16T12:26:50.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.525+0000 7f6cfa684700 1 -- 172.21.15.16:0/357011924 reap_dead start 2023-10-16T12:26:50.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.525+0000 7f6cfc0e7700 1 -- 172.21.15.16:0/357011924 shutdown_connections 2023-10-16T12:26:50.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.525+0000 7f6cfc0e7700 1 -- 172.21.15.16:0/357011924 >> 172.21.15.16:0/357011924 conn(0x7f6cf40cc5e0 msgr2=0x7f6cf40cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:50.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.525+0000 7f6cfc0e7700 1 -- 172.21.15.16:0/357011924 shutdown_connections 2023-10-16T12:26:50.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.525+0000 7f6cfc0e7700 1 -- 172.21.15.16:0/357011924 wait complete. 2023-10-16T12:26:50.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.526+0000 7f6cfc0e7700 1 Processor -- start 2023-10-16T12:26:50.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.526+0000 7f6cfc0e7700 1 -- start start 2023-10-16T12:26:50.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.527+0000 7f6cfc0e7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cf4146f70 0x7f6cf415d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:50.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.527+0000 7f6cfc0e7700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6cf4153f20 0x7f6cf40ca5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:50.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.527+0000 7f6cfc0e7700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6cf40caae0 0x7f6cf415b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:50.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.527+0000 7f6cfc0e7700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf41e4a90 con 0x7f6cf4146f70 2023-10-16T12:26:50.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.527+0000 7f6cfc0e7700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf40caf60 con 0x7f6cf40caae0 2023-10-16T12:26:50.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.527+0000 7f6cfc0e7700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf41e4280 con 0x7f6cf4153f20 2023-10-16T12:26:50.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.527+0000 7f6cf9e83700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cf4146f70 0x7f6cf415d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:50.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.527+0000 7f6cf9e83700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cf4146f70 0x7f6cf415d0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40534/0 (socket says 172.21.15.16:40534) 2023-10-16T12:26:50.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.527+0000 7f6cf9e83700 1 -- 172.21.15.16:0/82934809 learned_addr learned my addr 172.21.15.16:0/82934809 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:50.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.527+0000 7f6cf9682700 1 --2- 172.21.15.16:0/82934809 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6cf4153f20 0x7f6cf40ca5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:50.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.527+0000 7f6cfa684700 1 --2- 172.21.15.16:0/82934809 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6cf40caae0 0x7f6cf415b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:50.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.527+0000 7f6cf9e83700 1 -- 172.21.15.16:0/82934809 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6cf4153f20 msgr2=0x7f6cf40ca5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:50.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.527+0000 7f6cf9e83700 1 --2- 172.21.15.16:0/82934809 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6cf4153f20 0x7f6cf40ca5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:50.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.527+0000 7f6cf9e83700 1 -- 172.21.15.16:0/82934809 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6cf40caae0 msgr2=0x7f6cf415b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:50.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.527+0000 7f6cf9e83700 1 --2- 172.21.15.16:0/82934809 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6cf40caae0 0x7f6cf415b470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:50.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.527+0000 7f6cf9e83700 1 -- 172.21.15.16:0/82934809 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cf40c4de0 con 0x7f6cf4146f70 2023-10-16T12:26:50.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.527+0000 7f6cfa684700 1 --2- 172.21.15.16:0/82934809 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6cf40caae0 0x7f6cf415b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:26:50.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.528+0000 7f6cf9e83700 1 --2- 172.21.15.16:0/82934809 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cf4146f70 0x7f6cf415d0c0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f6ce4007290 tx=0x7f6ce4007650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:50.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.528+0000 7f6cea7fc700 1 -- 172.21.15.16:0/82934809 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce400f6c0 con 0x7f6cf4146f70 2023-10-16T12:26:50.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.528+0000 7f6cea7fc700 1 -- 172.21.15.16:0/82934809 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ce400f820 con 0x7f6cf4146f70 2023-10-16T12:26:50.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.528+0000 7f6cfc0e7700 1 -- 172.21.15.16:0/82934809 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cf415d600 con 0x7f6cf4146f70 2023-10-16T12:26:50.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.528+0000 7f6cea7fc700 1 -- 172.21.15.16:0/82934809 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce400fbb0 con 0x7f6cf4146f70 2023-10-16T12:26:50.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.528+0000 7f6cfc0e7700 1 -- 172.21.15.16:0/82934809 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cf41e47b0 con 0x7f6cf4146f70 2023-10-16T12:26:50.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.529+0000 7f6cfc0e7700 1 -- 172.21.15.16:0/82934809 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cf404ebb0 con 0x7f6cf4146f70 2023-10-16T12:26:50.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.529+0000 7f6cea7fc700 1 -- 172.21.15.16:0/82934809 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 40) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f6ce400fd10 con 0x7f6cf4146f70 2023-10-16T12:26:50.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.530+0000 7f6cea7fc700 1 --2- 172.21.15.16:0/82934809 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f6cd806af90 0x7f6cd806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:50.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.530+0000 7f6cea7fc700 1 -- 172.21.15.16:0/82934809 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5979+0+0 (secure 0 0 0) 0x7f6ce4089790 con 0x7f6cf4146f70 2023-10-16T12:26:50.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.530+0000 7f6cf9682700 1 --2- 172.21.15.16:0/82934809 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f6cd806af90 0x7f6cd806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:50.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.530+0000 7f6cf9682700 1 --2- 172.21.15.16:0/82934809 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f6cd806af90 0x7f6cd806d450 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6cf41f21e0 tx=0x7f6cf0009210 comp rx=0 tx=0).ready entity=mgr.6020 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:50.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.534+0000 7f6cea7fc700 1 -- 172.21.15.16:0/82934809 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6ce4054740 con 0x7f6cf4146f70 2023-10-16T12:26:50.681 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:26:50.684 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:26:50.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.738+0000 7f6cfc0e7700 1 -- 172.21.15.16:0/82934809 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f6cf40cbfc0 con 0x7f6cf4146f70 2023-10-16T12:26:50.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.741+0000 7f6cea7fc700 1 -- 172.21.15.16:0/82934809 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v25) v1 ==== 115+0+0 (secure 0 0 0) 0x7f6ce4057d60 con 0x7f6cf4146f70 2023-10-16T12:26:50.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.743+0000 7f6cfc0e7700 1 -- 172.21.15.16:0/82934809 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f6cd806af90 msgr2=0x7f6cd806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:50.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.743+0000 7f6cfc0e7700 1 --2- 172.21.15.16:0/82934809 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f6cd806af90 0x7f6cd806d450 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6cf41f21e0 tx=0x7f6cf0009210 comp rx=0 tx=0).stop 2023-10-16T12:26:50.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.743+0000 7f6cfc0e7700 1 -- 172.21.15.16:0/82934809 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cf4146f70 msgr2=0x7f6cf415d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:50.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.743+0000 7f6cfc0e7700 1 --2- 172.21.15.16:0/82934809 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cf4146f70 0x7f6cf415d0c0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f6ce4007290 tx=0x7f6ce4007650 comp rx=0 tx=0).stop 2023-10-16T12:26:50.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.743+0000 7f6cfc0e7700 1 -- 172.21.15.16:0/82934809 shutdown_connections 2023-10-16T12:26:50.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.743+0000 7f6cfc0e7700 1 --2- 172.21.15.16:0/82934809 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f6cd806af90 0x7f6cd806d450 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:50.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.743+0000 7f6cfc0e7700 1 --2- 172.21.15.16:0/82934809 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6cf40caae0 0x7f6cf415b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:50.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.743+0000 7f6cfc0e7700 1 --2- 172.21.15.16:0/82934809 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cf4146f70 0x7f6cf415d0c0 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:50.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.743+0000 7f6cfc0e7700 1 --2- 172.21.15.16:0/82934809 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6cf4153f20 0x7f6cf40ca5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:50.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.743+0000 7f6cfc0e7700 1 -- 172.21.15.16:0/82934809 >> 172.21.15.16:0/82934809 conn(0x7f6cf40cc5e0 msgr2=0x7f6cf4156f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:50.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.743+0000 7f6cfc0e7700 1 -- 172.21.15.16:0/82934809 shutdown_connections 2023-10-16T12:26:50.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.743+0000 7f6cfc0e7700 1 -- 172.21.15.16:0/82934809 wait complete. 2023-10-16T12:26:50.758 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-16T12:26:50.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.929+0000 7ff95dadc700 1 Processor -- start 2023-10-16T12:26:50.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.929+0000 7ff95dadc700 1 -- start start 2023-10-16T12:26:50.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.930+0000 7ff95dadc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff95815b2c0 0x7ff958161340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:50.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.930+0000 7ff95dadc700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff95806e570 0x7ff95815e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:50.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.930+0000 7ff95dadc700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7ff95806a7e0 0x7ff95806ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:50.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.930+0000 7ff95dadc700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff95806b1b0 con 0x7ff95815b2c0 2023-10-16T12:26:50.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.930+0000 7ff95dadc700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff958161880 con 0x7ff95806a7e0 2023-10-16T12:26:50.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.930+0000 7ff95dadc700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff95815b6a0 con 0x7ff95806e570 2023-10-16T12:26:50.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.930+0000 7ff9577fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff95815b2c0 0x7ff958161340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:50.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.930+0000 7ff9577fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff95815b2c0 0x7ff958161340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40542/0 (socket says 172.21.15.16:40542) 2023-10-16T12:26:50.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.930+0000 7ff9577fe700 1 -- 172.21.15.16:0/3798661444 learned_addr learned my addr 172.21.15.16:0/3798661444 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:50.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.930+0000 7ff956ffd700 1 --2- 172.21.15.16:0/3798661444 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff95806e570 0x7ff95815e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:50.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.930+0000 7ff957fff700 1 --2- 172.21.15.16:0/3798661444 >> v2:172.21.15.49:3300/0 conn(0x7ff95806a7e0 0x7ff95806ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:50.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.930+0000 7ff9577fe700 1 -- 172.21.15.16:0/3798661444 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff95806e570 msgr2=0x7ff95815e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:50.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.930+0000 7ff9577fe700 1 --2- 172.21.15.16:0/3798661444 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff95806e570 0x7ff95815e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:50.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.930+0000 7ff9577fe700 1 -- 172.21.15.16:0/3798661444 >> v2:172.21.15.49:3300/0 conn(0x7ff95806a7e0 msgr2=0x7ff95806ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:50.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.930+0000 7ff9577fe700 1 --2- 172.21.15.16:0/3798661444 >> v2:172.21.15.49:3300/0 conn(0x7ff95806a7e0 0x7ff95806ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:50.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.930+0000 7ff9577fe700 1 -- 172.21.15.16:0/3798661444 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff958161b20 con 0x7ff95815b2c0 2023-10-16T12:26:50.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.930+0000 7ff9577fe700 1 --2- 172.21.15.16:0/3798661444 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff95815b2c0 0x7ff958161340 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7ff9580ca970 tx=0x7ff948009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c954bee4103138e9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:50.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.931+0000 7ff9567fc700 1 -- 172.21.15.16:0/3798661444 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff94800b870 con 0x7ff95815b2c0 2023-10-16T12:26:50.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.931+0000 7ff9567fc700 1 -- 172.21.15.16:0/3798661444 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff94800b9d0 con 0x7ff95815b2c0 2023-10-16T12:26:50.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.931+0000 7ff9567fc700 1 -- 172.21.15.16:0/3798661444 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff94800bd60 con 0x7ff95815b2c0 2023-10-16T12:26:50.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.931+0000 7ff95dadc700 1 -- 172.21.15.16:0/3798661444 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff95815b2c0 msgr2=0x7ff958161340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:50.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.931+0000 7ff95dadc700 1 --2- 172.21.15.16:0/3798661444 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff95815b2c0 0x7ff958161340 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7ff9580ca970 tx=0x7ff948009bd0 comp rx=0 tx=0).stop 2023-10-16T12:26:50.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.931+0000 7ff95dadc700 1 -- 172.21.15.16:0/3798661444 shutdown_connections 2023-10-16T12:26:50.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.931+0000 7ff95dadc700 1 --2- 172.21.15.16:0/3798661444 >> v2:172.21.15.49:3300/0 conn(0x7ff95806a7e0 0x7ff95806ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:50.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.931+0000 7ff95dadc700 1 --2- 172.21.15.16:0/3798661444 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff95815b2c0 0x7ff958161340 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:50.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.931+0000 7ff95dadc700 1 --2- 172.21.15.16:0/3798661444 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff95806e570 0x7ff95815e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:50.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.931+0000 7ff95dadc700 1 -- 172.21.15.16:0/3798661444 >> 172.21.15.16:0/3798661444 conn(0x7ff9580b1f40 msgr2=0x7ff9580b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:50.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.931+0000 7ff95dadc700 1 -- 172.21.15.16:0/3798661444 shutdown_connections 2023-10-16T12:26:50.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.931+0000 7ff95dadc700 1 -- 172.21.15.16:0/3798661444 wait complete. 2023-10-16T12:26:50.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.932+0000 7ff95dadc700 1 Processor -- start 2023-10-16T12:26:50.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.933+0000 7ff95dadc700 1 -- start start 2023-10-16T12:26:50.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.933+0000 7ff95dadc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff95806a7e0 0x7ff9580b5b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:50.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.933+0000 7ff95dadc700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff95806e570 0x7ff9580c6d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:50.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.933+0000 7ff95dadc700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff95815b2c0 0x7ff9580b3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:50.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.933+0000 7ff95dadc700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9580c7280 con 0x7ff95806a7e0 2023-10-16T12:26:50.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.933+0000 7ff95dadc700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9580b6150 con 0x7ff95806e570 2023-10-16T12:26:50.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.933+0000 7ff95dadc700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9580c8510 con 0x7ff95815b2c0 2023-10-16T12:26:50.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.934+0000 7ff9577fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff95806a7e0 0x7ff9580b5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:50.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.934+0000 7ff9577fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff95806a7e0 0x7ff9580b5b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40548/0 (socket says 172.21.15.16:40548) 2023-10-16T12:26:50.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.934+0000 7ff9577fe700 1 -- 172.21.15.16:0/4277382903 learned_addr learned my addr 172.21.15.16:0/4277382903 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:50.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.934+0000 7ff956ffd700 1 --2- 172.21.15.16:0/4277382903 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff95806e570 0x7ff9580c6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:50.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.934+0000 7ff957fff700 1 --2- 172.21.15.16:0/4277382903 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff95815b2c0 0x7ff9580b3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:50.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.934+0000 7ff9577fe700 1 -- 172.21.15.16:0/4277382903 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff95815b2c0 msgr2=0x7ff9580b3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:50.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.934+0000 7ff9577fe700 1 --2- 172.21.15.16:0/4277382903 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff95815b2c0 0x7ff9580b3660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:50.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.934+0000 7ff9577fe700 1 -- 172.21.15.16:0/4277382903 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff95806e570 msgr2=0x7ff9580c6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:50.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.934+0000 7ff9577fe700 1 --2- 172.21.15.16:0/4277382903 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff95806e570 0x7ff9580c6d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:50.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.934+0000 7ff9577fe700 1 -- 172.21.15.16:0/4277382903 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff948009550 con 0x7ff95806a7e0 2023-10-16T12:26:50.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.934+0000 7ff956ffd700 1 --2- 172.21.15.16:0/4277382903 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff95806e570 0x7ff9580c6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:26:50.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.934+0000 7ff9577fe700 1 --2- 172.21.15.16:0/4277382903 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff95806a7e0 0x7ff9580b5b20 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7ff948000c00 tx=0x7ff948004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:50.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.934+0000 7ff94ffff700 1 -- 172.21.15.16:0/4277382903 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff948004540 con 0x7ff95806a7e0 2023-10-16T12:26:50.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.934+0000 7ff94ffff700 1 -- 172.21.15.16:0/4277382903 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9480036a0 con 0x7ff95806a7e0 2023-10-16T12:26:50.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.934+0000 7ff95dadc700 1 -- 172.21.15.16:0/4277382903 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff958152e60 con 0x7ff95806a7e0 2023-10-16T12:26:50.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.934+0000 7ff94ffff700 1 -- 172.21.15.16:0/4277382903 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff948003a50 con 0x7ff95806a7e0 2023-10-16T12:26:50.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.934+0000 7ff95dadc700 1 -- 172.21.15.16:0/4277382903 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff958153420 con 0x7ff95806a7e0 2023-10-16T12:26:50.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.935+0000 7ff95dadc700 1 -- 172.21.15.16:0/4277382903 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9580c8730 con 0x7ff95806a7e0 2023-10-16T12:26:50.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.936+0000 7ff94ffff700 1 -- 172.21.15.16:0/4277382903 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 41) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff948021070 con 0x7ff95806a7e0 2023-10-16T12:26:50.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.936+0000 7ff94ffff700 1 --2- 172.21.15.16:0/4277382903 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7ff93806b020 0x7ff93806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:50.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.936+0000 7ff94ffff700 1 -- 172.21.15.16:0/4277382903 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5979+0+0 (secure 0 0 0) 0x7ff94808aef0 con 0x7ff95806a7e0 2023-10-16T12:26:50.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.936+0000 7ff956ffd700 1 --2- 172.21.15.16:0/4277382903 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7ff93806b020 0x7ff93806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:50.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.937+0000 7ff956ffd700 1 --2- 172.21.15.16:0/4277382903 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7ff93806b020 0x7ff93806d4e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff9581f6180 tx=0x7ff940008040 comp rx=0 tx=0).ready entity=mgr.6020 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:50.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:50.941+0000 7ff94ffff700 1 -- 172.21.15.16:0/4277382903 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff9580c8730 con 0x7ff95806a7e0 2023-10-16T12:26:51.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.145+0000 7ff95dadc700 1 -- 172.21.15.16:0/4277382903 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7ff9580c8730 con 0x7ff95806a7e0 2023-10-16T12:26:51.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.148+0000 7ff94ffff700 1 -- 172.21.15.16:0/4277382903 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v26) v1 ==== 104+0+0 (secure 0 0 0) 0x7ff9580c8730 con 0x7ff95806a7e0 2023-10-16T12:26:51.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.150+0000 7ff95dadc700 1 -- 172.21.15.16:0/4277382903 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7ff93806b020 msgr2=0x7ff93806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:51.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.150+0000 7ff95dadc700 1 --2- 172.21.15.16:0/4277382903 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7ff93806b020 0x7ff93806d4e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff9581f6180 tx=0x7ff940008040 comp rx=0 tx=0).stop 2023-10-16T12:26:51.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.150+0000 7ff95dadc700 1 -- 172.21.15.16:0/4277382903 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff95806a7e0 msgr2=0x7ff9580b5b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:51.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.150+0000 7ff95dadc700 1 --2- 172.21.15.16:0/4277382903 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff95806a7e0 0x7ff9580b5b20 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7ff948000c00 tx=0x7ff948004750 comp rx=0 tx=0).stop 2023-10-16T12:26:51.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.150+0000 7ff95dadc700 1 -- 172.21.15.16:0/4277382903 shutdown_connections 2023-10-16T12:26:51.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.150+0000 7ff95dadc700 1 --2- 172.21.15.16:0/4277382903 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7ff93806b020 0x7ff93806d4e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:51.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.150+0000 7ff95dadc700 1 --2- 172.21.15.16:0/4277382903 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff95815b2c0 0x7ff9580b3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:51.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.150+0000 7ff95dadc700 1 --2- 172.21.15.16:0/4277382903 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff95806a7e0 0x7ff9580b5b20 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:51.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.150+0000 7ff95dadc700 1 --2- 172.21.15.16:0/4277382903 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff95806e570 0x7ff9580c6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:51.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.150+0000 7ff95dadc700 1 -- 172.21.15.16:0/4277382903 >> 172.21.15.16:0/4277382903 conn(0x7ff9580b1f40 msgr2=0x7ff958161120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:51.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.150+0000 7ff95dadc700 1 -- 172.21.15.16:0/4277382903 shutdown_connections 2023-10-16T12:26:51.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.150+0000 7ff95dadc700 1 -- 172.21.15.16:0/4277382903 wait complete. 2023-10-16T12:26:51.164 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-16T12:26:51.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.290+0000 7ff8f54c7700 1 Processor -- start 2023-10-16T12:26:51.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.291+0000 7ff8f54c7700 1 -- start start 2023-10-16T12:26:51.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.291+0000 7ff8f54c7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff8f00ce5d0 0x7ff8f00cc190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:51.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.291+0000 7ff8f54c7700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff8f015f060 0x7ff8f00cc6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:51.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.291+0000 7ff8f54c7700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8f00b2440 con 0x7ff8f015ec80 2023-10-16T12:26:51.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.291+0000 7ff8f54c7700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8f00b2c50 con 0x7ff8f00ce5d0 2023-10-16T12:26:51.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.291+0000 7ff8f54c7700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f00d00e0 con 0x7ff8f015f060 2023-10-16T12:26:51.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.291+0000 7ff8ee7fc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff8f00ce5d0 0x7ff8f00cc190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:51.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.291+0000 7ff8ee7fc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff8f00ce5d0 0x7ff8f00cc190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40558/0 (socket says 172.21.15.16:40558) 2023-10-16T12:26:51.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.291+0000 7ff8ee7fc700 1 -- 172.21.15.16:0/3783500474 learned_addr learned my addr 172.21.15.16:0/3783500474 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:51.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.292+0000 7ff8ef7fe700 1 --2- 172.21.15.16:0/3783500474 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff8f015f060 0x7ff8f00cc6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:51.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.292+0000 7ff8ee7fc700 1 -- 172.21.15.16:0/3783500474 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff8f015f060 msgr2=0x7ff8f00cc6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:51.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.292+0000 7ff8ee7fc700 1 --2- 172.21.15.16:0/3783500474 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff8f015f060 0x7ff8f00cc6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:51.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.292+0000 7ff8ee7fc700 1 -- 172.21.15.16:0/3783500474 >> v1:172.21.15.49:6789/0 conn(0x7ff8f015ec80 legacy=0x7ff8f00cdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:51.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.292+0000 7ff8ee7fc700 1 -- 172.21.15.16:0/3783500474 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8f00cef60 con 0x7ff8f00ce5d0 2023-10-16T12:26:51.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.292+0000 7ff8ee7fc700 1 --2- 172.21.15.16:0/3783500474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8f00ce5d0 0x7ff8f00cc190 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7ff8e00098f0 tx=0x7ff8e0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e6fa615ae4407b4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:51.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.292+0000 7ff8edffb700 1 -- 172.21.15.16:0/3783500474 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8e0011830 con 0x7ff8f00ce5d0 2023-10-16T12:26:51.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.292+0000 7ff8edffb700 1 -- 172.21.15.16:0/3783500474 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8e0011990 con 0x7ff8f00ce5d0 2023-10-16T12:26:51.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.292+0000 7ff8edffb700 1 -- 172.21.15.16:0/3783500474 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8e0011d20 con 0x7ff8f00ce5d0 2023-10-16T12:26:51.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.293+0000 7ff8f54c7700 1 -- 172.21.15.16:0/3783500474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8f00ce5d0 msgr2=0x7ff8f00cc190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:51.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.293+0000 7ff8f54c7700 1 --2- 172.21.15.16:0/3783500474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8f00ce5d0 0x7ff8f00cc190 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7ff8e00098f0 tx=0x7ff8e0009c00 comp rx=0 tx=0).stop 2023-10-16T12:26:51.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.293+0000 7ff8f54c7700 1 -- 172.21.15.16:0/3783500474 shutdown_connections 2023-10-16T12:26:51.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.293+0000 7ff8f54c7700 1 --2- 172.21.15.16:0/3783500474 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff8f015f060 0x7ff8f00cc6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:51.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.293+0000 7ff8f54c7700 1 --2- 172.21.15.16:0/3783500474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8f00ce5d0 0x7ff8f00cc190 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:51.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.293+0000 7ff8f54c7700 1 -- 172.21.15.16:0/3783500474 >> 172.21.15.16:0/3783500474 conn(0x7ff8f00d15f0 msgr2=0x7ff8f00d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:51.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.293+0000 7ff8f54c7700 1 -- 172.21.15.16:0/3783500474 shutdown_connections 2023-10-16T12:26:51.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.293+0000 7ff8f54c7700 1 -- 172.21.15.16:0/3783500474 wait complete. 2023-10-16T12:26:51.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.294+0000 7ff8f54c7700 1 Processor -- start 2023-10-16T12:26:51.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.294+0000 7ff8f54c7700 1 -- start start 2023-10-16T12:26:51.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.295+0000 7ff8f54c7700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff8f00ce5d0 0x7ff8f006c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:51.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.295+0000 7ff8f54c7700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff8f015ec80 0x7ff8f00c86c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:51.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.295+0000 7ff8f54c7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8f015f060 0x7ff8f0151a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:51.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.295+0000 7ff8f54c7700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f00c8c00 con 0x7ff8f015f060 2023-10-16T12:26:51.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.295+0000 7ff8f54c7700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f01e5cd0 con 0x7ff8f00ce5d0 2023-10-16T12:26:51.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.295+0000 7ff8f54c7700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f01e61c0 con 0x7ff8f015ec80 2023-10-16T12:26:51.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.295+0000 7ff8ef7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8f015f060 0x7ff8f0151a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:51.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.295+0000 7ff8ef7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8f015f060 0x7ff8f0151a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40566/0 (socket says 172.21.15.16:40566) 2023-10-16T12:26:51.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.295+0000 7ff8ef7fe700 1 -- 172.21.15.16:0/2465738595 learned_addr learned my addr 172.21.15.16:0/2465738595 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:51.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.295+0000 7ff8ee7fc700 1 --2- 172.21.15.16:0/2465738595 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff8f015ec80 0x7ff8f00c86c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:51.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.295+0000 7ff8eeffd700 1 --2- 172.21.15.16:0/2465738595 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff8f00ce5d0 0x7ff8f006c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:51.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.295+0000 7ff8ef7fe700 1 -- 172.21.15.16:0/2465738595 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff8f015ec80 msgr2=0x7ff8f00c86c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:51.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.295+0000 7ff8ef7fe700 1 --2- 172.21.15.16:0/2465738595 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff8f015ec80 0x7ff8f00c86c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:51.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.295+0000 7ff8ef7fe700 1 -- 172.21.15.16:0/2465738595 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff8f00ce5d0 msgr2=0x7ff8f006c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:51.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.295+0000 7ff8ef7fe700 1 --2- 172.21.15.16:0/2465738595 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff8f00ce5d0 0x7ff8f006c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:51.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.295+0000 7ff8ef7fe700 1 -- 172.21.15.16:0/2465738595 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8e0009550 con 0x7ff8f015f060 2023-10-16T12:26:51.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.295+0000 7ff8ef7fe700 1 --2- 172.21.15.16:0/2465738595 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8f015f060 0x7ff8f0151a90 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7ff8e800ef90 tx=0x7ff8e8009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:51.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.296+0000 7ff8d77fe700 1 -- 172.21.15.16:0/2465738595 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8e8009b20 con 0x7ff8f015f060 2023-10-16T12:26:51.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.296+0000 7ff8d77fe700 1 -- 172.21.15.16:0/2465738595 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8e800f040 con 0x7ff8f015f060 2023-10-16T12:26:51.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.296+0000 7ff8f54c7700 1 -- 172.21.15.16:0/2465738595 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8f01e5ee0 con 0x7ff8f015f060 2023-10-16T12:26:51.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.296+0000 7ff8d77fe700 1 -- 172.21.15.16:0/2465738595 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8e800cd20 con 0x7ff8f015f060 2023-10-16T12:26:51.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.296+0000 7ff8f54c7700 1 -- 172.21.15.16:0/2465738595 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8f006bd00 con 0x7ff8f015f060 2023-10-16T12:26:51.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.297+0000 7ff8f54c7700 1 -- 172.21.15.16:0/2465738595 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8f004ebb0 con 0x7ff8f015f060 2023-10-16T12:26:51.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.297+0000 7ff8d77fe700 1 -- 172.21.15.16:0/2465738595 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 41) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff8e8014480 con 0x7ff8f015f060 2023-10-16T12:26:51.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.297+0000 7ff8d77fe700 1 --2- 172.21.15.16:0/2465738595 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7ff8dc06b020 0x7ff8dc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:51.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.298+0000 7ff8d77fe700 1 -- 172.21.15.16:0/2465738595 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5979+0+0 (secure 0 0 0) 0x7ff8e808a360 con 0x7ff8f015f060 2023-10-16T12:26:51.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.298+0000 7ff8eeffd700 1 --2- 172.21.15.16:0/2465738595 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7ff8dc06b020 0x7ff8dc06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:51.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.298+0000 7ff8eeffd700 1 --2- 172.21.15.16:0/2465738595 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7ff8dc06b020 0x7ff8dc06d4e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff8e4000f60 tx=0x7ff8e4009040 comp rx=0 tx=0).ready entity=mgr.6020 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:51.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.302+0000 7ff8d77fe700 1 -- 172.21.15.16:0/2465738595 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff8e8055170 con 0x7ff8f015f060 2023-10-16T12:26:51.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.511+0000 7ff8f54c7700 1 -- 172.21.15.16:0/2465738595 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7ff8f00b2220 con 0x7ff8f015f060 2023-10-16T12:26:51.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.514+0000 7ff8d77fe700 1 -- 172.21.15.16:0/2465738595 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v27) v1 ==== 120+0+0 (secure 0 0 0) 0x7ff8e8058790 con 0x7ff8f015f060 2023-10-16T12:26:51.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.516+0000 7ff8f54c7700 1 -- 172.21.15.16:0/2465738595 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7ff8dc06b020 msgr2=0x7ff8dc06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:51.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.516+0000 7ff8f54c7700 1 --2- 172.21.15.16:0/2465738595 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7ff8dc06b020 0x7ff8dc06d4e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff8e4000f60 tx=0x7ff8e4009040 comp rx=0 tx=0).stop 2023-10-16T12:26:51.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.517+0000 7ff8f54c7700 1 -- 172.21.15.16:0/2465738595 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8f015f060 msgr2=0x7ff8f0151a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:51.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.517+0000 7ff8f54c7700 1 --2- 172.21.15.16:0/2465738595 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8f015f060 0x7ff8f0151a90 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7ff8e800ef90 tx=0x7ff8e8009f60 comp rx=0 tx=0).stop 2023-10-16T12:26:51.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.517+0000 7ff8f54c7700 1 -- 172.21.15.16:0/2465738595 shutdown_connections 2023-10-16T12:26:51.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.517+0000 7ff8f54c7700 1 --2- 172.21.15.16:0/2465738595 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7ff8dc06b020 0x7ff8dc06d4e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:51.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.517+0000 7ff8f54c7700 1 --2- 172.21.15.16:0/2465738595 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8f015f060 0x7ff8f0151a90 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:51.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.517+0000 7ff8f54c7700 1 --2- 172.21.15.16:0/2465738595 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff8f00ce5d0 0x7ff8f006c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:51.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.517+0000 7ff8f54c7700 1 --2- 172.21.15.16:0/2465738595 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff8f015ec80 0x7ff8f00c86c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:51.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.517+0000 7ff8f54c7700 1 -- 172.21.15.16:0/2465738595 >> 172.21.15.16:0/2465738595 conn(0x7ff8f00d15f0 msgr2=0x7ff8f015dab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:51.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.517+0000 7ff8f54c7700 1 -- 172.21.15.16:0/2465738595 shutdown_connections 2023-10-16T12:26:51.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.517+0000 7ff8f54c7700 1 -- 172.21.15.16:0/2465738595 wait complete. 2023-10-16T12:26:51.532 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule' 2023-10-16T12:26:51.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.655+0000 7f6b4c0a7700 1 Processor -- start 2023-10-16T12:26:51.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.655+0000 7f6b4c0a7700 1 -- start start 2023-10-16T12:26:51.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.656+0000 7f6b4c0a7700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6b4406a7e0 0x7f6b4406ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:51.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.656+0000 7f6b4c0a7700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b44161c30 con 0x7f6b440cc8a0 2023-10-16T12:26:51.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.656+0000 7f6b4c0a7700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b44161f10 con 0x7f6b4415b280 2023-10-16T12:26:51.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.656+0000 7f6b4c0a7700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b4406b190 con 0x7f6b4406a7e0 2023-10-16T12:26:51.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.656+0000 7f6b49642700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f6b440cc8a0 0x7f6b4415e700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:58078/0 (socket says 172.21.15.16:58078) 2023-10-16T12:26:51.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.656+0000 7f6b49642700 1 -- 172.21.15.16:0/2618178551 learned_addr learned my addr 172.21.15.16:0/2618178551 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:26:51.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.656+0000 7f6b4a644700 1 --2- 172.21.15.16:0/2618178551 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6b4406a7e0 0x7f6b4406ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:51.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.656+0000 7f6b48e41700 1 -- 172.21.15.16:0/2618178551 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2318185746 0 0) 0x7f6b44161c30 con 0x7f6b440cc8a0 2023-10-16T12:26:51.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.656+0000 7f6b48e41700 1 -- 172.21.15.16:0/2618178551 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b300035b0 con 0x7f6b440cc8a0 2023-10-16T12:26:51.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.656+0000 7f6b48e41700 1 -- 172.21.15.16:0/2618178551 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4270677198 0 0) 0x7f6b44161f10 con 0x7f6b4415b280 2023-10-16T12:26:51.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.656+0000 7f6b48e41700 1 -- 172.21.15.16:0/2618178551 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b44161c30 con 0x7f6b4415b280 2023-10-16T12:26:51.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.656+0000 7f6b48e41700 1 -- 172.21.15.16:0/2618178551 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 160394141 0 0) 0x7f6b300035b0 con 0x7f6b440cc8a0 2023-10-16T12:26:51.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.656+0000 7f6b4a644700 1 -- 172.21.15.16:0/2618178551 >> v1:172.21.15.49:6789/0 conn(0x7f6b4415b280 legacy=0x7f6b441613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:51.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.656+0000 7f6b4a644700 1 -- 172.21.15.16:0/2618178551 >> v1:172.21.15.16:6789/0 conn(0x7f6b440cc8a0 legacy=0x7f6b4415e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:51.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.656+0000 7f6b4a644700 1 -- 172.21.15.16:0/2618178551 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b441621f0 con 0x7f6b4406a7e0 2023-10-16T12:26:51.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.657+0000 7f6b4a644700 1 --2- 172.21.15.16:0/2618178551 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6b4406a7e0 0x7f6b4406ac50 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f6b3c009dc0 tx=0x7f6b3c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5edd6d923b150bc1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:51.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.657+0000 7f6b48e41700 1 -- 172.21.15.16:0/2618178551 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b3c00e070 con 0x7f6b4406a7e0 2023-10-16T12:26:51.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.657+0000 7f6b48e41700 1 -- 172.21.15.16:0/2618178551 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b3c003a20 con 0x7f6b4406a7e0 2023-10-16T12:26:51.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.657+0000 7f6b48e41700 1 -- 172.21.15.16:0/2618178551 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b34001670 con 0x7f6b4406a7e0 2023-10-16T12:26:51.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.657+0000 7f6b4c0a7700 1 -- 172.21.15.16:0/2618178551 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6b4406a7e0 msgr2=0x7f6b4406ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:51.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.657+0000 7f6b4c0a7700 1 --2- 172.21.15.16:0/2618178551 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6b4406a7e0 0x7f6b4406ac50 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f6b3c009dc0 tx=0x7f6b3c00b330 comp rx=0 tx=0).stop 2023-10-16T12:26:51.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.657+0000 7f6b4c0a7700 1 -- 172.21.15.16:0/2618178551 shutdown_connections 2023-10-16T12:26:51.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.657+0000 7f6b4c0a7700 1 --2- 172.21.15.16:0/2618178551 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6b4406a7e0 0x7f6b4406ac50 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:51.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.657+0000 7f6b4c0a7700 1 -- 172.21.15.16:0/2618178551 >> 172.21.15.16:0/2618178551 conn(0x7f6b440b1fd0 msgr2=0x7f6b440b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:51.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.658+0000 7f6b4c0a7700 1 -- 172.21.15.16:0/2618178551 shutdown_connections 2023-10-16T12:26:51.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.658+0000 7f6b4c0a7700 1 -- 172.21.15.16:0/2618178551 wait complete. 2023-10-16T12:26:51.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.659+0000 7f6b4c0a7700 1 Processor -- start 2023-10-16T12:26:51.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.659+0000 7f6b4c0a7700 1 -- start start 2023-10-16T12:26:51.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.659+0000 7f6b4c0a7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b4406a7e0 0x7f6b440c3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:51.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.659+0000 7f6b4c0a7700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6b440cc8a0 0x7f6b440ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:51.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.659+0000 7f6b4c0a7700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6b4415b280 0x7f6b440726c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:51.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.659+0000 7f6b4c0a7700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b44072c00 con 0x7f6b4406a7e0 2023-10-16T12:26:51.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.659+0000 7f6b4c0a7700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b440c4600 con 0x7f6b440cc8a0 2023-10-16T12:26:51.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.659+0000 7f6b4c0a7700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b440c56b0 con 0x7f6b4415b280 2023-10-16T12:26:51.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.660+0000 7f6b49e43700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b4406a7e0 0x7f6b440c3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:51.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.660+0000 7f6b49e43700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b4406a7e0 0x7f6b440c3f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40578/0 (socket says 172.21.15.16:40578) 2023-10-16T12:26:51.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.660+0000 7f6b49e43700 1 -- 172.21.15.16:0/3817655489 learned_addr learned my addr 172.21.15.16:0/3817655489 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:51.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.660+0000 7f6b4a644700 1 --2- 172.21.15.16:0/3817655489 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6b4415b280 0x7f6b440726c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:51.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.660+0000 7f6b49642700 1 --2- 172.21.15.16:0/3817655489 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6b440cc8a0 0x7f6b440ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:51.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.660+0000 7f6b49e43700 1 -- 172.21.15.16:0/3817655489 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6b4415b280 msgr2=0x7f6b440726c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:51.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.660+0000 7f6b49e43700 1 --2- 172.21.15.16:0/3817655489 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6b4415b280 0x7f6b440726c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:51.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.660+0000 7f6b49e43700 1 -- 172.21.15.16:0/3817655489 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6b440cc8a0 msgr2=0x7f6b440ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:51.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.660+0000 7f6b49e43700 1 --2- 172.21.15.16:0/3817655489 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6b440cc8a0 0x7f6b440ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:51.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.660+0000 7f6b49e43700 1 -- 172.21.15.16:0/3817655489 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b3c009a70 con 0x7f6b4406a7e0 2023-10-16T12:26:51.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.660+0000 7f6b49e43700 1 --2- 172.21.15.16:0/3817655489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b4406a7e0 0x7f6b440c3f70 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f6b4000c530 tx=0x7f6b4000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:51.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.660+0000 7f6b3a7fc700 1 -- 172.21.15.16:0/3817655489 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b400052d0 con 0x7f6b4406a7e0 2023-10-16T12:26:51.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.660+0000 7f6b3a7fc700 1 -- 172.21.15.16:0/3817655489 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b40005450 con 0x7f6b4406a7e0 2023-10-16T12:26:51.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.660+0000 7f6b4c0a7700 1 -- 172.21.15.16:0/3817655489 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b440c4dc0 con 0x7f6b4406a7e0 2023-10-16T12:26:51.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.660+0000 7f6b3a7fc700 1 -- 172.21.15.16:0/3817655489 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b400038c0 con 0x7f6b4406a7e0 2023-10-16T12:26:51.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.660+0000 7f6b4c0a7700 1 -- 172.21.15.16:0/3817655489 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b440c5340 con 0x7f6b4406a7e0 2023-10-16T12:26:51.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.662+0000 7f6b4c0a7700 1 -- 172.21.15.16:0/3817655489 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b4404ebb0 con 0x7f6b4406a7e0 2023-10-16T12:26:51.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.662+0000 7f6b3a7fc700 1 -- 172.21.15.16:0/3817655489 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 41) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6b40003a20 con 0x7f6b4406a7e0 2023-10-16T12:26:51.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.662+0000 7f6b3a7fc700 1 --2- 172.21.15.16:0/3817655489 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f6b2806b020 0x7f6b2806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:51.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.662+0000 7f6b3a7fc700 1 -- 172.21.15.16:0/3817655489 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5979+0+0 (secure 0 0 0) 0x7f6b40088cf0 con 0x7f6b4406a7e0 2023-10-16T12:26:51.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.662+0000 7f6b49642700 1 --2- 172.21.15.16:0/3817655489 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f6b2806b020 0x7f6b2806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:51.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.663+0000 7f6b49642700 1 --2- 172.21.15.16:0/3817655489 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f6b2806b020 0x7f6b2806d4e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6b44072650 tx=0x7f6b34001c20 comp rx=0 tx=0).ready entity=mgr.6020 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:51.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.667+0000 7f6b3a7fc700 1 -- 172.21.15.16:0/3817655489 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6b4008d070 con 0x7f6b4406a7e0 2023-10-16T12:26:51.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:51.962+0000 7f6b4c0a7700 1 -- 172.21.15.16:0/3817655489 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]} v 0) v1 -- 0x7f6b4415ded0 con 0x7f6b4406a7e0 2023-10-16T12:26:52.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.363+0000 7f6b3a7fc700 1 -- 172.21.15.16:0/3817655489 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]}]=0 v598) v1 ==== 156+0+0 (secure 0 0 0) 0x7f6b400170e0 con 0x7f6b4406a7e0 2023-10-16T12:26:52.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.366+0000 7f6b4c0a7700 1 -- 172.21.15.16:0/3817655489 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f6b2806b020 msgr2=0x7f6b2806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:52.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.366+0000 7f6b4c0a7700 1 --2- 172.21.15.16:0/3817655489 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f6b2806b020 0x7f6b2806d4e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6b44072650 tx=0x7f6b34001c20 comp rx=0 tx=0).stop 2023-10-16T12:26:52.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.366+0000 7f6b4c0a7700 1 -- 172.21.15.16:0/3817655489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b4406a7e0 msgr2=0x7f6b440c3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:52.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.366+0000 7f6b4c0a7700 1 --2- 172.21.15.16:0/3817655489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b4406a7e0 0x7f6b440c3f70 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f6b4000c530 tx=0x7f6b4000c8f0 comp rx=0 tx=0).stop 2023-10-16T12:26:52.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.366+0000 7f6b4c0a7700 1 -- 172.21.15.16:0/3817655489 shutdown_connections 2023-10-16T12:26:52.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.366+0000 7f6b4c0a7700 1 --2- 172.21.15.16:0/3817655489 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f6b2806b020 0x7f6b2806d4e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:52.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.366+0000 7f6b4c0a7700 1 --2- 172.21.15.16:0/3817655489 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6b4415b280 0x7f6b440726c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:52.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.366+0000 7f6b4c0a7700 1 --2- 172.21.15.16:0/3817655489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b4406a7e0 0x7f6b440c3f70 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:52.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.366+0000 7f6b4c0a7700 1 --2- 172.21.15.16:0/3817655489 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6b440cc8a0 0x7f6b440ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:52.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.366+0000 7f6b4c0a7700 1 -- 172.21.15.16:0/3817655489 >> 172.21.15.16:0/3817655489 conn(0x7f6b440b1fd0 msgr2=0x7f6b441588e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:52.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.366+0000 7f6b4c0a7700 1 -- 172.21.15.16:0/3817655489 shutdown_connections 2023-10-16T12:26:52.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.366+0000 7f6b4c0a7700 1 -- 172.21.15.16:0/3817655489 wait complete. 2023-10-16T12:26:52.383 INFO:tasks.cephfs_test_runner:Test listing non-existing snap schedules failure ... ok 2023-10-16T12:26:52.383 INFO:tasks.cephfs_test_runner:Starting test: test_non_existent_snap_schedule_list (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test listing snap schedules on a non-existing filesystem path failure 2023-10-16T12:26:52.384 INFO:tasks.cephfs_test_runner:test_non_existent_snap_schedule_list (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-16T12:26:52.384 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:26:52.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.514+0000 7f59788d9700 1 Processor -- start 2023-10-16T12:26:52.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.514+0000 7f59788d9700 1 -- start start 2023-10-16T12:26:52.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.515+0000 7f59788d9700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f59740cb280 0x7f597406d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:52.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.515+0000 7f59788d9700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f597406b1f0 0x7f597406b640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:52.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.515+0000 7f59788d9700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f597406c150 con 0x7f5974160eb0 2023-10-16T12:26:52.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.515+0000 7f59788d9700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59740af370 con 0x7f59740cb280 2023-10-16T12:26:52.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.515+0000 7f59788d9700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f597406be70 con 0x7f597406b1f0 2023-10-16T12:26:52.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.515+0000 7f5971d9b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f59740cb280 0x7f597406d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:52.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.515+0000 7f597259c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f5974160eb0 0x7f59740cab70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:58090/0 (socket says 172.21.15.16:58090) 2023-10-16T12:26:52.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.516+0000 7f597259c700 1 -- 172.21.15.16:0/3244331578 learned_addr learned my addr 172.21.15.16:0/3244331578 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:26:52.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.516+0000 7f5972d9d700 1 --2- 172.21.15.16:0/3244331578 >> v2:172.21.15.49:3300/0 conn(0x7f597406b1f0 0x7f597406b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:52.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.516+0000 7f5971d9b700 1 -- 172.21.15.16:0/3244331578 >> v2:172.21.15.49:3300/0 conn(0x7f597406b1f0 msgr2=0x7f597406b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:52.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.516+0000 7f5971d9b700 1 --2- 172.21.15.16:0/3244331578 >> v2:172.21.15.49:3300/0 conn(0x7f597406b1f0 0x7f597406b640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:52.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.516+0000 7f5971d9b700 1 -- 172.21.15.16:0/3244331578 >> v1:172.21.15.16:6789/0 conn(0x7f5974160eb0 legacy=0x7f59740cab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:52.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.516+0000 7f5971d9b700 1 -- 172.21.15.16:0/3244331578 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59740cdf10 con 0x7f59740cb280 2023-10-16T12:26:52.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.516+0000 7f5972d9d700 1 --2- 172.21.15.16:0/3244331578 >> v2:172.21.15.49:3300/0 conn(0x7f597406b1f0 0x7f597406b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:26:52.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.516+0000 7f5971d9b700 1 --2- 172.21.15.16:0/3244331578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59740cb280 0x7f597406d8f0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f59740aef40 tx=0x7f5964009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ebe4b6d839ca067 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:52.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.516+0000 7f597159a700 1 -- 172.21.15.16:0/3244331578 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5964011830 con 0x7f59740cb280 2023-10-16T12:26:52.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.516+0000 7f597159a700 1 -- 172.21.15.16:0/3244331578 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5964011990 con 0x7f59740cb280 2023-10-16T12:26:52.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.516+0000 7f597159a700 1 -- 172.21.15.16:0/3244331578 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5964011d20 con 0x7f59740cb280 2023-10-16T12:26:52.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.517+0000 7f59788d9700 1 -- 172.21.15.16:0/3244331578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59740cb280 msgr2=0x7f597406d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:52.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.517+0000 7f59788d9700 1 --2- 172.21.15.16:0/3244331578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59740cb280 0x7f597406d8f0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f59740aef40 tx=0x7f5964009b20 comp rx=0 tx=0).stop 2023-10-16T12:26:52.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.517+0000 7f59788d9700 1 -- 172.21.15.16:0/3244331578 shutdown_connections 2023-10-16T12:26:52.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.517+0000 7f59788d9700 1 --2- 172.21.15.16:0/3244331578 >> v2:172.21.15.49:3300/0 conn(0x7f597406b1f0 0x7f597406b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:52.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.517+0000 7f59788d9700 1 --2- 172.21.15.16:0/3244331578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59740cb280 0x7f597406d8f0 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:52.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.517+0000 7f59788d9700 1 -- 172.21.15.16:0/3244331578 >> 172.21.15.16:0/3244331578 conn(0x7f59740cc5b0 msgr2=0x7f59740cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:52.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.517+0000 7f59788d9700 1 -- 172.21.15.16:0/3244331578 shutdown_connections 2023-10-16T12:26:52.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.517+0000 7f59788d9700 1 -- 172.21.15.16:0/3244331578 wait complete. 2023-10-16T12:26:52.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.518+0000 7f59788d9700 1 Processor -- start 2023-10-16T12:26:52.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.518+0000 7f59788d9700 1 -- start start 2023-10-16T12:26:52.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.519+0000 7f59788d9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f597406b1f0 0x7f5974155d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:52.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.519+0000 7f59788d9700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f59740cb280 0x7f5974156d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:52.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.519+0000 7f59788d9700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5974160eb0 0x7f5974159100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:52.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.519+0000 7f59788d9700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59741583c0 con 0x7f597406b1f0 2023-10-16T12:26:52.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.519+0000 7f59788d9700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59741577b0 con 0x7f5974160eb0 2023-10-16T12:26:52.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.519+0000 7f59788d9700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5974157a90 con 0x7f59740cb280 2023-10-16T12:26:52.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.519+0000 7f597259c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f597406b1f0 0x7f5974155d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:52.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.519+0000 7f597259c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f597406b1f0 0x7f5974155d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40590/0 (socket says 172.21.15.16:40590) 2023-10-16T12:26:52.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.519+0000 7f597259c700 1 -- 172.21.15.16:0/1650510076 learned_addr learned my addr 172.21.15.16:0/1650510076 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:52.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.519+0000 7f5972d9d700 1 --2- 172.21.15.16:0/1650510076 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5974160eb0 0x7f5974159100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:52.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.519+0000 7f5971d9b700 1 --2- 172.21.15.16:0/1650510076 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f59740cb280 0x7f5974156d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:52.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.519+0000 7f597259c700 1 -- 172.21.15.16:0/1650510076 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f59740cb280 msgr2=0x7f5974156d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:52.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.519+0000 7f597259c700 1 --2- 172.21.15.16:0/1650510076 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f59740cb280 0x7f5974156d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:52.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.519+0000 7f597259c700 1 -- 172.21.15.16:0/1650510076 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5974160eb0 msgr2=0x7f5974159100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:52.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.519+0000 7f597259c700 1 --2- 172.21.15.16:0/1650510076 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5974160eb0 0x7f5974159100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:52.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.519+0000 7f597259c700 1 -- 172.21.15.16:0/1650510076 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59640094a0 con 0x7f597406b1f0 2023-10-16T12:26:52.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.520+0000 7f597259c700 1 --2- 172.21.15.16:0/1650510076 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f597406b1f0 0x7f5974155d70 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f596800c520 tx=0x7f596800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:52.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.520+0000 7f5962ffd700 1 -- 172.21.15.16:0/1650510076 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5968005d70 con 0x7f597406b1f0 2023-10-16T12:26:52.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.520+0000 7f5962ffd700 1 -- 172.21.15.16:0/1650510076 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59680070d0 con 0x7f597406b1f0 2023-10-16T12:26:52.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.520+0000 7f59788d9700 1 -- 172.21.15.16:0/1650510076 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5974157d70 con 0x7f597406b1f0 2023-10-16T12:26:52.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.520+0000 7f5962ffd700 1 -- 172.21.15.16:0/1650510076 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5968007440 con 0x7f597406b1f0 2023-10-16T12:26:52.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.520+0000 7f59788d9700 1 -- 172.21.15.16:0/1650510076 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59741562b0 con 0x7f597406b1f0 2023-10-16T12:26:52.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.521+0000 7f59788d9700 1 -- 172.21.15.16:0/1650510076 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f597404ebb0 con 0x7f597406b1f0 2023-10-16T12:26:52.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.521+0000 7f5962ffd700 1 -- 172.21.15.16:0/1650510076 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 41) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f59680075a0 con 0x7f597406b1f0 2023-10-16T12:26:52.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.522+0000 7f5962ffd700 1 --2- 172.21.15.16:0/1650510076 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f595406b020 0x7f595406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:52.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.522+0000 7f5971d9b700 1 --2- 172.21.15.16:0/1650510076 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f595406b020 0x7f595406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:52.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.522+0000 7f5962ffd700 1 -- 172.21.15.16:0/1650510076 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5979+0+0 (secure 0 0 0) 0x7f5968088d10 con 0x7f597406b1f0 2023-10-16T12:26:52.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.522+0000 7f5971d9b700 1 --2- 172.21.15.16:0/1650510076 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f595406b020 0x7f595406d4e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f59640090b0 tx=0x7f5964008030 comp rx=0 tx=0).ready entity=mgr.6020 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:52.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.526+0000 7f5962ffd700 1 -- 172.21.15.16:0/1650510076 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f596808e020 con 0x7f597406b1f0 2023-10-16T12:26:52.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.731+0000 7f59788d9700 1 -- 172.21.15.16:0/1650510076 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f59740cdbe0 con 0x7f597406b1f0 2023-10-16T12:26:52.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.732+0000 7f5962ffd700 1 -- 172.21.15.16:0/1650510076 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+14127 (secure 0 0 0) 0x7f5968017090 con 0x7f597406b1f0 2023-10-16T12:26:52.734 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:26:52.734 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":90,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:26:50.360364+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6835/962243511":"2023-10-17T12:26:47.355824+0000","172.21.15.16:0/2710394622":"2023-10-17T12:26:46.309755+0000","172.21.15.16:0/3754876159":"2023-10-17T12:26:46.309755+0000","172.21.15.16:0/1938940392":"2023-10-17T12:26:46.309755+0000","172.21.15.16:0/3196640464":"2023-10-17T12:26:46.309755+0000","172.21.15.16:0/636468305":"2023-10-17T12:26:46.309755+0000","172.21.15.16:6833/2230262559":"2023-10-17T12:26:46.309755+0000","172.21.15.16:0/2700763633":"2023-10-17T12:26:38.749960+0000","172.21.15.16:0/2069262541":"2023-10-17T12:26:38.749960+0000","172.21.15.16:6837/962243511":"2023-10-17T12:26:47.355824+0000","172.21.15.16:0/1444840585":"2023-10-17T12:26:38.749960+0000","172.21.15.16:6833/3415303009":"2023-10-17T12:26:38.749960+0000","172.21.15.16:6832/3415303009":"2023-10-17T12:26:38.749960+0000","172.21.15.16:6832/2230262559":"2023-10-17T12:26:46.309755+0000","172.21.15.16:0/779547724":"2023-10-17T12:26:38.749960+0000","172.21.15.16:0/1047057927":"2023-10-17T12:26:46.309755+0000","172.21.15.16:0/4285863251":"2023-10-17T12:26:38.749960+0000","172.21.15.16:0/4057240133":"2023-10-17T12:26:38.749960+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:26:52.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.734+0000 7f59788d9700 1 -- 172.21.15.16:0/1650510076 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f595406b020 msgr2=0x7f595406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:52.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.734+0000 7f59788d9700 1 --2- 172.21.15.16:0/1650510076 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f595406b020 0x7f595406d4e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f59640090b0 tx=0x7f5964008030 comp rx=0 tx=0).stop 2023-10-16T12:26:52.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.734+0000 7f59788d9700 1 -- 172.21.15.16:0/1650510076 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f597406b1f0 msgr2=0x7f5974155d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:52.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.734+0000 7f59788d9700 1 --2- 172.21.15.16:0/1650510076 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f597406b1f0 0x7f5974155d70 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f596800c520 tx=0x7f596800c8e0 comp rx=0 tx=0).stop 2023-10-16T12:26:52.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.734+0000 7f59788d9700 1 -- 172.21.15.16:0/1650510076 shutdown_connections 2023-10-16T12:26:52.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.734+0000 7f59788d9700 1 --2- 172.21.15.16:0/1650510076 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f595406b020 0x7f595406d4e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:52.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.734+0000 7f59788d9700 1 --2- 172.21.15.16:0/1650510076 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5974160eb0 0x7f5974159100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:52.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.734+0000 7f59788d9700 1 --2- 172.21.15.16:0/1650510076 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f597406b1f0 0x7f5974155d70 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:52.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.734+0000 7f59788d9700 1 --2- 172.21.15.16:0/1650510076 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f59740cb280 0x7f5974156d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:52.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.734+0000 7f59788d9700 1 -- 172.21.15.16:0/1650510076 >> 172.21.15.16:0/1650510076 conn(0x7f59740cc5b0 msgr2=0x7f59740b06a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:52.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.734+0000 7f59788d9700 1 -- 172.21.15.16:0/1650510076 shutdown_connections 2023-10-16T12:26:52.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.734+0000 7f59788d9700 1 -- 172.21.15.16:0/1650510076 wait complete. 2023-10-16T12:26:52.748 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T12:26:52.749 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:26:52.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.875+0000 7f6a777a3700 1 Processor -- start 2023-10-16T12:26:52.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.875+0000 7f6a777a3700 1 -- start start 2023-10-16T12:26:52.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.876+0000 7f6a777a3700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f6a700d2110 0x7f6a700ae1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:52.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.876+0000 7f6a777a3700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6a7006ca10 0x7f6a700ae720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:52.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.876+0000 7f6a777a3700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a700b20f0 con 0x7f6a700ce6a0 2023-10-16T12:26:52.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.876+0000 7f6a777a3700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a700d03d0 con 0x7f6a700d2110 2023-10-16T12:26:52.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.876+0000 7f6a777a3700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a700b2340 con 0x7f6a7006ca10 2023-10-16T12:26:52.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.876+0000 7f6a75d40700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6a7006ca10 0x7f6a700ae720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:52.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.876+0000 7f6a74d3e700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f6a700d2110 0x7f6a700ae1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:52.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.876+0000 7f6a75d40700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6a7006ca10 0x7f6a700ae720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:57092/0 (socket says 172.21.15.16:57092) 2023-10-16T12:26:52.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.876+0000 7f6a75d40700 1 -- 172.21.15.16:0/178660461 learned_addr learned my addr 172.21.15.16:0/178660461 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:52.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.876+0000 7f6a74d3e700 1 -- 172.21.15.16:0/178660461 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6a7006ca10 msgr2=0x7f6a700ae720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:52.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.876+0000 7f6a74d3e700 1 --2- 172.21.15.16:0/178660461 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6a7006ca10 0x7f6a700ae720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:52.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.876+0000 7f6a74d3e700 1 -- 172.21.15.16:0/178660461 >> v1:172.21.15.49:6789/0 conn(0x7f6a700ce6a0 legacy=0x7f6a7006c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:52.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.876+0000 7f6a74d3e700 1 -- 172.21.15.16:0/178660461 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a7006bb00 con 0x7f6a700d2110 2023-10-16T12:26:52.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.876+0000 7f6a75d40700 1 --2- 172.21.15.16:0/178660461 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6a7006ca10 0x7f6a700ae720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:26:52.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.877+0000 7f6a74d3e700 1 --2- 172.21.15.16:0/178660461 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6a700d2110 0x7f6a700ae1e0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f6a60009840 tx=0x7f6a60009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ff416c79436f5f96 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:52.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.877+0000 7f6a67fff700 1 -- 172.21.15.16:0/178660461 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a60011830 con 0x7f6a700d2110 2023-10-16T12:26:52.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.877+0000 7f6a67fff700 1 -- 172.21.15.16:0/178660461 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a60011990 con 0x7f6a700d2110 2023-10-16T12:26:52.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.877+0000 7f6a67fff700 1 -- 172.21.15.16:0/178660461 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a60011d20 con 0x7f6a700d2110 2023-10-16T12:26:52.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.877+0000 7f6a777a3700 1 -- 172.21.15.16:0/178660461 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6a700d2110 msgr2=0x7f6a700ae1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:52.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.877+0000 7f6a777a3700 1 --2- 172.21.15.16:0/178660461 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6a700d2110 0x7f6a700ae1e0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f6a60009840 tx=0x7f6a60009b50 comp rx=0 tx=0).stop 2023-10-16T12:26:52.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.877+0000 7f6a777a3700 1 -- 172.21.15.16:0/178660461 shutdown_connections 2023-10-16T12:26:52.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.877+0000 7f6a777a3700 1 --2- 172.21.15.16:0/178660461 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6a7006ca10 0x7f6a700ae720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:52.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.877+0000 7f6a777a3700 1 --2- 172.21.15.16:0/178660461 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6a700d2110 0x7f6a700ae1e0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:52.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.877+0000 7f6a777a3700 1 -- 172.21.15.16:0/178660461 >> 172.21.15.16:0/178660461 conn(0x7f6a700d15f0 msgr2=0x7f6a700d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:52.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.878+0000 7f6a777a3700 1 -- 172.21.15.16:0/178660461 shutdown_connections 2023-10-16T12:26:52.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.878+0000 7f6a777a3700 1 -- 172.21.15.16:0/178660461 wait complete. 2023-10-16T12:26:52.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.879+0000 7f6a777a3700 1 Processor -- start 2023-10-16T12:26:52.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.879+0000 7f6a777a3700 1 -- start start 2023-10-16T12:26:52.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.880+0000 7f6a777a3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a7006ca10 0x7f6a701f0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:52.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.880+0000 7f6a777a3700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6a700ce6a0 0x7f6a700b4b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:52.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.880+0000 7f6a777a3700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6a700d2110 0x7f6a7015f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:52.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.880+0000 7f6a777a3700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a701f32b0 con 0x7f6a7006ca10 2023-10-16T12:26:52.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.880+0000 7f6a777a3700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a701f0dd0 con 0x7f6a700d2110 2023-10-16T12:26:52.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.880+0000 7f6a777a3700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a700b5550 con 0x7f6a700ce6a0 2023-10-16T12:26:52.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.880+0000 7f6a7553f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a7006ca10 0x7f6a701f0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:52.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.880+0000 7f6a7553f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a7006ca10 0x7f6a701f0890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40604/0 (socket says 172.21.15.16:40604) 2023-10-16T12:26:52.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.880+0000 7f6a7553f700 1 -- 172.21.15.16:0/2951821471 learned_addr learned my addr 172.21.15.16:0/2951821471 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:52.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.880+0000 7f6a75d40700 1 --2- 172.21.15.16:0/2951821471 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6a700d2110 0x7f6a7015f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:52.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.880+0000 7f6a74d3e700 1 --2- 172.21.15.16:0/2951821471 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6a700ce6a0 0x7f6a700b4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:52.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.880+0000 7f6a7553f700 1 -- 172.21.15.16:0/2951821471 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6a700ce6a0 msgr2=0x7f6a700b4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:52.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.880+0000 7f6a7553f700 1 --2- 172.21.15.16:0/2951821471 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6a700ce6a0 0x7f6a700b4b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:52.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.880+0000 7f6a7553f700 1 -- 172.21.15.16:0/2951821471 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6a700d2110 msgr2=0x7f6a7015f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:52.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.880+0000 7f6a7553f700 1 --2- 172.21.15.16:0/2951821471 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6a700d2110 0x7f6a7015f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:52.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.880+0000 7f6a7553f700 1 -- 172.21.15.16:0/2951821471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a68009b30 con 0x7f6a7006ca10 2023-10-16T12:26:52.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.880+0000 7f6a75d40700 1 --2- 172.21.15.16:0/2951821471 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6a700d2110 0x7f6a7015f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:26:52.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.880+0000 7f6a74d3e700 1 --2- 172.21.15.16:0/2951821471 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6a700ce6a0 0x7f6a700b4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:26:52.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.880+0000 7f6a7553f700 1 --2- 172.21.15.16:0/2951821471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a7006ca10 0x7f6a701f0890 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f6a6c00c4e0 tx=0x7f6a6c00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:52.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.880+0000 7f6a65ffb700 1 -- 172.21.15.16:0/2951821471 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a6c0052d0 con 0x7f6a7006ca10 2023-10-16T12:26:52.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.880+0000 7f6a65ffb700 1 -- 172.21.15.16:0/2951821471 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a6c005430 con 0x7f6a7006ca10 2023-10-16T12:26:52.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.880+0000 7f6a777a3700 1 -- 172.21.15.16:0/2951821471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a600094a0 con 0x7f6a7006ca10 2023-10-16T12:26:52.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.880+0000 7f6a65ffb700 1 -- 172.21.15.16:0/2951821471 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a6c0038c0 con 0x7f6a7006ca10 2023-10-16T12:26:52.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.881+0000 7f6a777a3700 1 -- 172.21.15.16:0/2951821471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a701f3790 con 0x7f6a7006ca10 2023-10-16T12:26:52.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.882+0000 7f6a777a3700 1 -- 172.21.15.16:0/2951821471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a7004ebb0 con 0x7f6a7006ca10 2023-10-16T12:26:52.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.882+0000 7f6a65ffb700 1 -- 172.21.15.16:0/2951821471 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 41) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6a6c003a20 con 0x7f6a7006ca10 2023-10-16T12:26:52.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.882+0000 7f6a65ffb700 1 --2- 172.21.15.16:0/2951821471 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f6a5406aff0 0x7f6a5406d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:52.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.883+0000 7f6a65ffb700 1 -- 172.21.15.16:0/2951821471 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5979+0+0 (secure 0 0 0) 0x7f6a6c088eb0 con 0x7f6a7006ca10 2023-10-16T12:26:52.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.883+0000 7f6a74d3e700 1 --2- 172.21.15.16:0/2951821471 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f6a5406aff0 0x7f6a5406d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:52.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.883+0000 7f6a74d3e700 1 --2- 172.21.15.16:0/2951821471 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f6a5406aff0 0x7f6a5406d4b0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6a60009840 tx=0x7f6a6001c040 comp rx=0 tx=0).ready entity=mgr.6020 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:52.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:52.887+0000 7f6a65ffb700 1 -- 172.21.15.16:0/2951821471 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6a6c053ca0 con 0x7f6a7006ca10 2023-10-16T12:26:53.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:53.089+0000 7f6a777a3700 1 -- 172.21.15.16:0/2951821471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6a7006bdb0 con 0x7f6a7006ca10 2023-10-16T12:26:53.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:53.089+0000 7f6a65ffb700 1 -- 172.21.15.16:0/2951821471 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6a6c0170a0 con 0x7f6a7006ca10 2023-10-16T12:26:53.092 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T12:26:53.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:53.092+0000 7f6a777a3700 1 -- 172.21.15.16:0/2951821471 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f6a5406aff0 msgr2=0x7f6a5406d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:53.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:53.092+0000 7f6a777a3700 1 --2- 172.21.15.16:0/2951821471 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f6a5406aff0 0x7f6a5406d4b0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6a60009840 tx=0x7f6a6001c040 comp rx=0 tx=0).stop 2023-10-16T12:26:53.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:53.092+0000 7f6a777a3700 1 -- 172.21.15.16:0/2951821471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a7006ca10 msgr2=0x7f6a701f0890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:53.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:53.092+0000 7f6a777a3700 1 --2- 172.21.15.16:0/2951821471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a7006ca10 0x7f6a701f0890 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f6a6c00c4e0 tx=0x7f6a6c00c8a0 comp rx=0 tx=0).stop 2023-10-16T12:26:53.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:53.092+0000 7f6a777a3700 1 -- 172.21.15.16:0/2951821471 shutdown_connections 2023-10-16T12:26:53.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:53.092+0000 7f6a777a3700 1 --2- 172.21.15.16:0/2951821471 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f6a5406aff0 0x7f6a5406d4b0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:53.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:53.092+0000 7f6a777a3700 1 --2- 172.21.15.16:0/2951821471 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6a700d2110 0x7f6a7015f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:53.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:53.092+0000 7f6a777a3700 1 --2- 172.21.15.16:0/2951821471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a7006ca10 0x7f6a701f0890 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:53.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:53.092+0000 7f6a777a3700 1 --2- 172.21.15.16:0/2951821471 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6a700ce6a0 0x7f6a700b4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:53.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:53.092+0000 7f6a777a3700 1 -- 172.21.15.16:0/2951821471 >> 172.21.15.16:0/2951821471 conn(0x7f6a700d15f0 msgr2=0x7f6a700d19d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:53.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:53.092+0000 7f6a777a3700 1 -- 172.21.15.16:0/2951821471 shutdown_connections 2023-10-16T12:26:53.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:53.092+0000 7f6a777a3700 1 -- 172.21.15.16:0/2951821471 wait complete. 2023-10-16T12:26:53.106 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list' 2023-10-16T12:26:53.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:53.230+0000 7fab4d59c700 1 Processor -- start 2023-10-16T12:26:53.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:53.230+0000 7fab4d59c700 1 -- start start 2023-10-16T12:26:53.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:53.231+0000 7fab4d59c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fab4806aef0 0x7fab480b35b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:53.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:53.231+0000 7fab4d59c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fab480b2be0 0x7fab480b3af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:53.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:53.231+0000 7fab4d59c700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab480d03b0 con 0x7fab4815b760 2023-10-16T12:26:53.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:53.231+0000 7fab4d59c700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab4815bb40 con 0x7fab480b2be0 2023-10-16T12:26:53.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:53.231+0000 7fab4d59c700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab480cb760 con 0x7fab4806aef0 2023-10-16T12:26:53.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:53.231+0000 7fab477fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fab480b2be0 0x7fab480b3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:53.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:53.231+0000 7fab46ffd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fab4815b760 0x7fab4806a7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:58118/0 (socket says 172.21.15.16:58118) 2023-10-16T12:26:53.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:53.231+0000 7fab46ffd700 1 -- 172.21.15.16:0/1972051793 learned_addr learned my addr 172.21.15.16:0/1972051793 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:26:53.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:53.231+0000 7fab467fc700 1 --2- 172.21.15.16:0/1972051793 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fab4806aef0 0x7fab480b35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:53.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:53.231+0000 7fab477fe700 1 -- 172.21.15.16:0/1972051793 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fab4806aef0 msgr2=0x7fab480b35b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:53.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:53.231+0000 7fab477fe700 1 --2- 172.21.15.16:0/1972051793 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fab4806aef0 0x7fab480b35b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:53.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:53.231+0000 7fab477fe700 1 -- 172.21.15.16:0/1972051793 >> v1:172.21.15.16:6789/0 conn(0x7fab4815b760 legacy=0x7fab4806a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:53.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:53.231+0000 7fab45ffb700 1 -- 172.21.15.16:0/1972051793 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4247272667 0 0) 0x7fab480d03b0 con 0x7fab4815b760 2023-10-16T12:26:53.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:53.231+0000 7fab477fe700 1 -- 172.21.15.16:0/1972051793 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab480d0600 con 0x7fab480b2be0 2023-10-16T12:26:53.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:53.231+0000 7fab477fe700 1 --2- 172.21.15.16:0/1972051793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab480b2be0 0x7fab480b3af0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7fab40009d90 tx=0x7fab4000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad58fe9b2605a8ce server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:53.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:53.231+0000 7fab45ffb700 1 -- 172.21.15.16:0/1972051793 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab4000e070 con 0x7fab480b2be0 2023-10-16T12:26:53.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:53.232+0000 7fab45ffb700 1 -- 172.21.15.16:0/1972051793 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab4000bbd0 con 0x7fab480b2be0 2023-10-16T12:26:53.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:53.232+0000 7fab45ffb700 1 -- 172.21.15.16:0/1972051793 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab40012430 con 0x7fab480b2be0 2023-10-16T12:26:53.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:53.232+0000 7fab4d59c700 1 -- 172.21.15.16:0/1972051793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab480b2be0 msgr2=0x7fab480b3af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:53.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:53.232+0000 7fab4d59c700 1 --2- 172.21.15.16:0/1972051793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab480b2be0 0x7fab480b3af0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7fab40009d90 tx=0x7fab4000b330 comp rx=0 tx=0).stop 2023-10-16T12:26:53.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:53.232+0000 7fab4d59c700 1 -- 172.21.15.16:0/1972051793 shutdown_connections 2023-10-16T12:26:53.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:53.232+0000 7fab4d59c700 1 --2- 172.21.15.16:0/1972051793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab480b2be0 0x7fab480b3af0 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:53.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:53.232+0000 7fab4d59c700 1 --2- 172.21.15.16:0/1972051793 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fab4806aef0 0x7fab480b35b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:53.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:53.232+0000 7fab4d59c700 1 -- 172.21.15.16:0/1972051793 >> 172.21.15.16:0/1972051793 conn(0x7fab480b1fd0 msgr2=0x7fab480b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:53.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:53.232+0000 7fab4d59c700 1 -- 172.21.15.16:0/1972051793 shutdown_connections 2023-10-16T12:26:53.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:53.232+0000 7fab4d59c700 1 -- 172.21.15.16:0/1972051793 wait complete. 2023-10-16T12:26:53.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:53.234+0000 7fab4d59c700 1 Processor -- start 2023-10-16T12:26:53.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:53.234+0000 7fab4d59c700 1 -- start start 2023-10-16T12:26:53.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:53.235+0000 7fab4d59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab4806aef0 0x7fab480af180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:53.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:53.235+0000 7fab4d59c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fab480b2be0 0x7fab48158380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:53.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:53.235+0000 7fab4d59c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fab4815b760 0x7fab481564a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:53.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:53.235+0000 7fab4d59c700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab480af7d0 con 0x7fab4806aef0 2023-10-16T12:26:53.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:53.235+0000 7fab4d59c700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab481569e0 con 0x7fab480b2be0 2023-10-16T12:26:53.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:53.235+0000 7fab4d59c700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab480c7640 con 0x7fab4815b760 2023-10-16T12:26:53.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:53.235+0000 7fab46ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab4806aef0 0x7fab480af180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:53.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:53.235+0000 7fab46ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab4806aef0 0x7fab480af180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40618/0 (socket says 172.21.15.16:40618) 2023-10-16T12:26:53.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:53.235+0000 7fab46ffd700 1 -- 172.21.15.16:0/4083281326 learned_addr learned my addr 172.21.15.16:0/4083281326 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:53.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:53.235+0000 7fab477fe700 1 --2- 172.21.15.16:0/4083281326 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fab4815b760 0x7fab481564a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:53.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:53.235+0000 7fab467fc700 1 --2- 172.21.15.16:0/4083281326 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fab480b2be0 0x7fab48158380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:53.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:53.235+0000 7fab46ffd700 1 -- 172.21.15.16:0/4083281326 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fab4815b760 msgr2=0x7fab481564a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:53.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:53.235+0000 7fab46ffd700 1 --2- 172.21.15.16:0/4083281326 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fab4815b760 0x7fab481564a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:53.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:53.235+0000 7fab46ffd700 1 -- 172.21.15.16:0/4083281326 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fab480b2be0 msgr2=0x7fab48158380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:53.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:53.235+0000 7fab46ffd700 1 --2- 172.21.15.16:0/4083281326 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fab480b2be0 0x7fab48158380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:53.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:53.235+0000 7fab46ffd700 1 -- 172.21.15.16:0/4083281326 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab40009a70 con 0x7fab4806aef0 2023-10-16T12:26:53.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:53.235+0000 7fab46ffd700 1 --2- 172.21.15.16:0/4083281326 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab4806aef0 0x7fab480af180 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7fab3000c580 tx=0x7fab3000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:53.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:53.236+0000 7fab3f7fe700 1 -- 172.21.15.16:0/4083281326 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab30004c80 con 0x7fab4806aef0 2023-10-16T12:26:53.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:53.236+0000 7fab3f7fe700 1 -- 172.21.15.16:0/4083281326 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab30004e00 con 0x7fab4806aef0 2023-10-16T12:26:53.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:53.236+0000 7fab4d59c700 1 -- 172.21.15.16:0/4083281326 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab480c7920 con 0x7fab4806aef0 2023-10-16T12:26:53.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:53.236+0000 7fab3f7fe700 1 -- 172.21.15.16:0/4083281326 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab30007310 con 0x7fab4806aef0 2023-10-16T12:26:53.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:53.236+0000 7fab4d59c700 1 -- 172.21.15.16:0/4083281326 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab480c7ee0 con 0x7fab4806aef0 2023-10-16T12:26:53.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:53.237+0000 7fab4d59c700 1 -- 172.21.15.16:0/4083281326 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab4804ebb0 con 0x7fab4806aef0 2023-10-16T12:26:53.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:53.237+0000 7fab3f7fe700 1 -- 172.21.15.16:0/4083281326 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 41) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fab30007470 con 0x7fab4806aef0 2023-10-16T12:26:53.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:53.237+0000 7fab3f7fe700 1 --2- 172.21.15.16:0/4083281326 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7fab2806b050 0x7fab2806d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:53.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:53.237+0000 7fab3f7fe700 1 -- 172.21.15.16:0/4083281326 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5979+0+0 (secure 0 0 0) 0x7fab300894c0 con 0x7fab4806aef0 2023-10-16T12:26:53.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:53.238+0000 7fab467fc700 1 --2- 172.21.15.16:0/4083281326 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7fab2806b050 0x7fab2806d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:53.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:53.238+0000 7fab467fc700 1 --2- 172.21.15.16:0/4083281326 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7fab2806b050 0x7fab2806d510 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fab480b3cf0 tx=0x7fab38008040 comp rx=0 tx=0).ready entity=mgr.6020 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:53.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:53.242+0000 7fab3f7fe700 1 -- 172.21.15.16:0/4083281326 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fab300543e0 con 0x7fab4806aef0 2023-10-16T12:26:53.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:53.532+0000 7fab4d59c700 1 -- 172.21.15.16:0/4083281326 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]} v 0) v1 -- 0x7fab480b1ab0 con 0x7fab4806aef0 2023-10-16T12:26:54.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.369+0000 7fab3f7fe700 1 -- 172.21.15.16:0/4083281326 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]}]=0 v600) v1 ==== 169+0+0 (secure 0 0 0) 0x7fab30057a00 con 0x7fab4806aef0 2023-10-16T12:26:54.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.372+0000 7fab4d59c700 1 -- 172.21.15.16:0/4083281326 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7fab2806b050 msgr2=0x7fab2806d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:54.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.372+0000 7fab4d59c700 1 --2- 172.21.15.16:0/4083281326 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7fab2806b050 0x7fab2806d510 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fab480b3cf0 tx=0x7fab38008040 comp rx=0 tx=0).stop 2023-10-16T12:26:54.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.372+0000 7fab4d59c700 1 -- 172.21.15.16:0/4083281326 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab4806aef0 msgr2=0x7fab480af180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:54.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.372+0000 7fab4d59c700 1 --2- 172.21.15.16:0/4083281326 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab4806aef0 0x7fab480af180 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7fab3000c580 tx=0x7fab3000c940 comp rx=0 tx=0).stop 2023-10-16T12:26:54.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.372+0000 7fab4d59c700 1 -- 172.21.15.16:0/4083281326 shutdown_connections 2023-10-16T12:26:54.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.372+0000 7fab4d59c700 1 --2- 172.21.15.16:0/4083281326 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7fab2806b050 0x7fab2806d510 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:54.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.372+0000 7fab4d59c700 1 --2- 172.21.15.16:0/4083281326 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fab4815b760 0x7fab481564a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:54.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.372+0000 7fab4d59c700 1 --2- 172.21.15.16:0/4083281326 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab4806aef0 0x7fab480af180 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:54.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.372+0000 7fab4d59c700 1 --2- 172.21.15.16:0/4083281326 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fab480b2be0 0x7fab48158380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:54.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.372+0000 7fab4d59c700 1 -- 172.21.15.16:0/4083281326 >> 172.21.15.16:0/4083281326 conn(0x7fab480b1fd0 msgr2=0x7fab480b4f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:54.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.372+0000 7fab4d59c700 1 -- 172.21.15.16:0/4083281326 shutdown_connections 2023-10-16T12:26:54.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.372+0000 7fab4d59c700 1 -- 172.21.15.16:0/4083281326 wait complete. 2023-10-16T12:26:54.389 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-16T12:26:54.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.523+0000 7f1245333700 1 Processor -- start 2023-10-16T12:26:54.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.523+0000 7f1245333700 1 -- start start 2023-10-16T12:26:54.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.523+0000 7f1245333700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f124015d8d0 0x7f12400b2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:54.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.523+0000 7f1245333700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12400b2f30 con 0x7f12400ce350 2023-10-16T12:26:54.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.523+0000 7f1245333700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12400d1c40 con 0x7f12400b8fa0 2023-10-16T12:26:54.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.523+0000 7f1245333700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12400d0bb0 con 0x7f124015d8d0 2023-10-16T12:26:54.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.524+0000 7f123f7fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f12400ce350 0x7f12400d29c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:58136/0 (socket says 172.21.15.16:58136) 2023-10-16T12:26:54.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.524+0000 7f123f7fe700 1 -- 172.21.15.16:0/3490134888 learned_addr learned my addr 172.21.15.16:0/3490134888 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:26:54.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.524+0000 7f123effd700 1 --2- 172.21.15.16:0/3490134888 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f124015d8d0 0x7f12400b2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:54.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.524+0000 7f123dffb700 1 -- 172.21.15.16:0/3490134888 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2880341803 0 0) 0x7f12400b2f30 con 0x7f12400ce350 2023-10-16T12:26:54.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.524+0000 7f123dffb700 1 -- 172.21.15.16:0/3490134888 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f122c0035b0 con 0x7f12400ce350 2023-10-16T12:26:54.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.524+0000 7f123dffb700 1 -- 172.21.15.16:0/3490134888 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1415471954 0 0) 0x7f12400d1c40 con 0x7f12400b8fa0 2023-10-16T12:26:54.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.524+0000 7f123dffb700 1 -- 172.21.15.16:0/3490134888 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12400b2f30 con 0x7f12400b8fa0 2023-10-16T12:26:54.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.524+0000 7f123dffb700 1 -- 172.21.15.16:0/3490134888 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2035384093 0 0) 0x7f122c0035b0 con 0x7f12400ce350 2023-10-16T12:26:54.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.524+0000 7f123dffb700 1 -- 172.21.15.16:0/3490134888 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f124015d8d0 msgr2=0x7f12400b2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:54.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.524+0000 7f123dffb700 1 --2- 172.21.15.16:0/3490134888 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f124015d8d0 0x7f12400b2830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:54.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.524+0000 7f123dffb700 1 -- 172.21.15.16:0/3490134888 >> v1:172.21.15.49:6789/0 conn(0x7f12400b8fa0 legacy=0x7f12400b6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:54.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.524+0000 7f123dffb700 1 -- 172.21.15.16:0/3490134888 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12400d1ee0 con 0x7f12400ce350 2023-10-16T12:26:54.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.524+0000 7f123dffb700 1 -- 172.21.15.16:0/3490134888 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f12380019f0 con 0x7f12400ce350 2023-10-16T12:26:54.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.524+0000 7f123dffb700 1 -- 172.21.15.16:0/3490134888 >> v1:172.21.15.16:6789/0 conn(0x7f12400ce350 legacy=0x7f12400d29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:54.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.524+0000 7f123dffb700 1 --2- 172.21.15.16:0/3490134888 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f122c003bc0 0x7f122c003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:54.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.524+0000 7f123dffb700 1 --2- 172.21.15.16:0/3490134888 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f122c004850 0x7f122c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:54.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.524+0000 7f123dffb700 1 --2- 172.21.15.16:0/3490134888 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f122c005210 0x7f122c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:54.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.524+0000 7f123dffb700 1 -- 172.21.15.16:0/3490134888 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12400d0bb0 con 0x7f122c003bc0 2023-10-16T12:26:54.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.524+0000 7f123dffb700 1 -- 172.21.15.16:0/3490134888 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f122c00c140 con 0x7f122c005210 2023-10-16T12:26:54.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.524+0000 7f123dffb700 1 -- 172.21.15.16:0/3490134888 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f122c00c300 con 0x7f122c004850 2023-10-16T12:26:54.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.524+0000 7f123effd700 1 --2- 172.21.15.16:0/3490134888 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f124015d8d0 0x7f12400b2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:26:54.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.525+0000 7f123f7fe700 1 --2- 172.21.15.16:0/3490134888 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f122c005210 0x7f122c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:54.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.525+0000 7f123effd700 1 --2- 172.21.15.16:0/3490134888 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f122c003bc0 0x7f122c003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:54.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.525+0000 7f123e7fc700 1 --2- 172.21.15.16:0/3490134888 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f122c004850 0x7f122c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:54.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.525+0000 7f123effd700 1 -- 172.21.15.16:0/3490134888 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f122c004850 msgr2=0x7f122c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:54.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.525+0000 7f123effd700 1 --2- 172.21.15.16:0/3490134888 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f122c004850 0x7f122c004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:54.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.525+0000 7f123effd700 1 -- 172.21.15.16:0/3490134888 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f122c005210 msgr2=0x7f122c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:54.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.525+0000 7f123effd700 1 --2- 172.21.15.16:0/3490134888 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f122c005210 0x7f122c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:54.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.525+0000 7f123effd700 1 -- 172.21.15.16:0/3490134888 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f122c0035b0 con 0x7f122c003bc0 2023-10-16T12:26:54.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.525+0000 7f123effd700 1 --2- 172.21.15.16:0/3490134888 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f122c003bc0 0x7f122c003fc0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f12400d4250 tx=0x7f122c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c5b20a5f40d12c8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:54.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.525+0000 7f123dffb700 1 -- 172.21.15.16:0/3490134888 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1230004030 con 0x7f122c003bc0 2023-10-16T12:26:54.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.525+0000 7f123dffb700 1 -- 172.21.15.16:0/3490134888 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1230004190 con 0x7f122c003bc0 2023-10-16T12:26:54.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.525+0000 7f123dffb700 1 -- 172.21.15.16:0/3490134888 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1230004540 con 0x7f122c003bc0 2023-10-16T12:26:54.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.526+0000 7f1245333700 1 -- 172.21.15.16:0/3490134888 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f122c003bc0 msgr2=0x7f122c003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:54.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.526+0000 7f1245333700 1 --2- 172.21.15.16:0/3490134888 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f122c003bc0 0x7f122c003fc0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f12400d4250 tx=0x7f122c005880 comp rx=0 tx=0).stop 2023-10-16T12:26:54.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.526+0000 7f123f7fe700 1 -- 172.21.15.16:0/3490134888 reap_dead start 2023-10-16T12:26:54.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.526+0000 7f1245333700 1 -- 172.21.15.16:0/3490134888 shutdown_connections 2023-10-16T12:26:54.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.526+0000 7f1245333700 1 -- 172.21.15.16:0/3490134888 >> 172.21.15.16:0/3490134888 conn(0x7f1240072330 msgr2=0x7f1240072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:54.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.526+0000 7f1245333700 1 -- 172.21.15.16:0/3490134888 shutdown_connections 2023-10-16T12:26:54.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.526+0000 7f1245333700 1 -- 172.21.15.16:0/3490134888 wait complete. 2023-10-16T12:26:54.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.527+0000 7f1245333700 1 Processor -- start 2023-10-16T12:26:54.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.527+0000 7f1245333700 1 -- start start 2023-10-16T12:26:54.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.528+0000 7f1245333700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f122c003bc0 0x7f12401f2f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:54.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.528+0000 7f1245333700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f124015dcb0 0x7f1240160810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:54.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.528+0000 7f1245333700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f124015f890 0x7f12400b1560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:54.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.528+0000 7f1245333700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1240160d50 con 0x7f124015dcb0 2023-10-16T12:26:54.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.528+0000 7f1245333700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12401f3ef0 con 0x7f124015f890 2023-10-16T12:26:54.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.528+0000 7f1245333700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12401f3650 con 0x7f122c003bc0 2023-10-16T12:26:54.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.528+0000 7f123e7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f124015dcb0 0x7f1240160810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:54.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.528+0000 7f123e7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f124015dcb0 0x7f1240160810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40640/0 (socket says 172.21.15.16:40640) 2023-10-16T12:26:54.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.528+0000 7f123e7fc700 1 -- 172.21.15.16:0/2233717070 learned_addr learned my addr 172.21.15.16:0/2233717070 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:54.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.528+0000 7f123effd700 1 --2- 172.21.15.16:0/2233717070 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f122c003bc0 0x7f12401f2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:54.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.528+0000 7f123f7fe700 1 --2- 172.21.15.16:0/2233717070 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f124015f890 0x7f12400b1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:54.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.528+0000 7f123e7fc700 1 -- 172.21.15.16:0/2233717070 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f122c003bc0 msgr2=0x7f12401f2f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:54.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.528+0000 7f123e7fc700 1 --2- 172.21.15.16:0/2233717070 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f122c003bc0 0x7f12401f2f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:54.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.528+0000 7f123e7fc700 1 -- 172.21.15.16:0/2233717070 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f124015f890 msgr2=0x7f12400b1560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:54.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.528+0000 7f123e7fc700 1 --2- 172.21.15.16:0/2233717070 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f124015f890 0x7f12400b1560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:54.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.528+0000 7f123e7fc700 1 -- 172.21.15.16:0/2233717070 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12401f3930 con 0x7f124015dcb0 2023-10-16T12:26:54.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.528+0000 7f123effd700 1 --2- 172.21.15.16:0/2233717070 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f122c003bc0 0x7f12401f2f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:26:54.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.529+0000 7f123e7fc700 1 --2- 172.21.15.16:0/2233717070 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f124015dcb0 0x7f1240160810 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f122800f820 tx=0x7f122800fb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:54.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.529+0000 7f12377fe700 1 -- 172.21.15.16:0/2233717070 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12280052e0 con 0x7f124015dcb0 2023-10-16T12:26:54.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.529+0000 7f12377fe700 1 -- 172.21.15.16:0/2233717070 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1228005460 con 0x7f124015dcb0 2023-10-16T12:26:54.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.529+0000 7f12377fe700 1 -- 172.21.15.16:0/2233717070 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12280038c0 con 0x7f124015dcb0 2023-10-16T12:26:54.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.529+0000 7f1245333700 1 -- 172.21.15.16:0/2233717070 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12401f3c10 con 0x7f124015dcb0 2023-10-16T12:26:54.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.529+0000 7f1245333700 1 -- 172.21.15.16:0/2233717070 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f124015fd20 con 0x7f124015dcb0 2023-10-16T12:26:54.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.530+0000 7f1245333700 1 -- 172.21.15.16:0/2233717070 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12400aa2b0 con 0x7f124015dcb0 2023-10-16T12:26:54.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.530+0000 7f12377fe700 1 -- 172.21.15.16:0/2233717070 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 41) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1228003a20 con 0x7f124015dcb0 2023-10-16T12:26:54.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.531+0000 7f12377fe700 1 --2- 172.21.15.16:0/2233717070 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f122006b020 0x7f122006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:54.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.531+0000 7f12377fe700 1 -- 172.21.15.16:0/2233717070 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5979+0+0 (secure 0 0 0) 0x7f1228089520 con 0x7f124015dcb0 2023-10-16T12:26:54.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.531+0000 7f123effd700 1 --2- 172.21.15.16:0/2233717070 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f122006b020 0x7f122006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:54.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.531+0000 7f123effd700 1 --2- 172.21.15.16:0/2233717070 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f122006b020 0x7f122006d4e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1240160240 tx=0x7f1230020040 comp rx=0 tx=0).ready entity=mgr.6020 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:54.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.535+0000 7f12377fe700 1 -- 172.21.15.16:0/2233717070 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f12400aa2b0 con 0x7f124015dcb0 2023-10-16T12:26:54.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.741+0000 7f1245333700 1 -- 172.21.15.16:0/2233717070 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f12400aa2b0 con 0x7f124015dcb0 2023-10-16T12:26:54.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.744+0000 7f12377fe700 1 -- 172.21.15.16:0/2233717070 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v28)=0 v28) v1 ==== 122+0+0 (secure 0 0 0) 0x7f12400aa2b0 con 0x7f124015dcb0 2023-10-16T12:26:54.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.747+0000 7f1245333700 1 -- 172.21.15.16:0/2233717070 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f122006b020 msgr2=0x7f122006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:54.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.747+0000 7f1245333700 1 --2- 172.21.15.16:0/2233717070 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f122006b020 0x7f122006d4e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1240160240 tx=0x7f1230020040 comp rx=0 tx=0).stop 2023-10-16T12:26:54.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.747+0000 7f1245333700 1 -- 172.21.15.16:0/2233717070 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f124015dcb0 msgr2=0x7f1240160810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:54.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.747+0000 7f1245333700 1 --2- 172.21.15.16:0/2233717070 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f124015dcb0 0x7f1240160810 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f122800f820 tx=0x7f122800fb30 comp rx=0 tx=0).stop 2023-10-16T12:26:54.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.747+0000 7f1245333700 1 -- 172.21.15.16:0/2233717070 shutdown_connections 2023-10-16T12:26:54.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.747+0000 7f1245333700 1 --2- 172.21.15.16:0/2233717070 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f122006b020 0x7f122006d4e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:54.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.747+0000 7f1245333700 1 --2- 172.21.15.16:0/2233717070 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f124015f890 0x7f12400b1560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:54.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.747+0000 7f1245333700 1 --2- 172.21.15.16:0/2233717070 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f122c003bc0 0x7f12401f2f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:54.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.747+0000 7f1245333700 1 --2- 172.21.15.16:0/2233717070 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f124015dcb0 0x7f1240160810 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:54.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.747+0000 7f1245333700 1 -- 172.21.15.16:0/2233717070 >> 172.21.15.16:0/2233717070 conn(0x7f1240072330 msgr2=0x7f124015cca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:54.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.747+0000 7f1245333700 1 -- 172.21.15.16:0/2233717070 shutdown_connections 2023-10-16T12:26:54.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:54.747+0000 7f1245333700 1 -- 172.21.15.16:0/2233717070 wait complete. 2023-10-16T12:26:54.761 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T12:26:54.761 DEBUG:teuthology.orchestra.run.smithi016:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T12:26:54.801 DEBUG:teuthology.orchestra.run.smithi049:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T12:26:54.854 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T12:26:54.854 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:26:54.855 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:26:54.872 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16287364k,nr_inodes=4071841,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20251 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3261172k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T12:26:54.872 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T12:26:54.873 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:26:54.892 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284204k,nr_inodes=4071051,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=753 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260540k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T12:26:54.893 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:26:54.893 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:26:54.930 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16287364k,nr_inodes=4071841,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20251 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3261172k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T12:26:54.930 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T12:26:54.931 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:26:54.952 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284204k,nr_inodes=4071051,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=753 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260540k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T12:26:54.952 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-16T12:26:54.953 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:26:55.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.092+0000 7f9f956ff700 1 Processor -- start 2023-10-16T12:26:55.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.092+0000 7f9f956ff700 1 -- start start 2023-10-16T12:26:55.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.093+0000 7f9f956ff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9f900cc8a0 0x7f9f9015e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:55.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.093+0000 7f9f956ff700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f9f9006a7e0 0x7f9f9006ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:55.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.093+0000 7f9f956ff700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f9006b150 con 0x7f9f9015b280 2023-10-16T12:26:55.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.093+0000 7f9f956ff700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f90161c30 con 0x7f9f900cc8a0 2023-10-16T12:26:55.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.093+0000 7f9f956ff700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f90161f10 con 0x7f9f9006a7e0 2023-10-16T12:26:55.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.093+0000 7f9f8e7fc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9f900cc8a0 0x7f9f9015e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:55.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.093+0000 7f9f8e7fc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9f900cc8a0 0x7f9f9015e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40648/0 (socket says 172.21.15.16:40648) 2023-10-16T12:26:55.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.093+0000 7f9f8e7fc700 1 -- 172.21.15.16:0/1619660381 learned_addr learned my addr 172.21.15.16:0/1619660381 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:55.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.093+0000 7f9f8f7fe700 1 --2- 172.21.15.16:0/1619660381 >> v2:172.21.15.49:3300/0 conn(0x7f9f9006a7e0 0x7f9f9006ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:55.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.094+0000 7f9f8e7fc700 1 -- 172.21.15.16:0/1619660381 >> v2:172.21.15.49:3300/0 conn(0x7f9f9006a7e0 msgr2=0x7f9f9006ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:55.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.094+0000 7f9f8e7fc700 1 --2- 172.21.15.16:0/1619660381 >> v2:172.21.15.49:3300/0 conn(0x7f9f9006a7e0 0x7f9f9006ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:55.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.094+0000 7f9f8e7fc700 1 -- 172.21.15.16:0/1619660381 >> v1:172.21.15.16:6789/0 conn(0x7f9f9015b280 legacy=0x7f9f901613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:55.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.094+0000 7f9f8dffb700 1 -- 172.21.15.16:0/1619660381 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 875934571 0 0) 0x7f9f9006b150 con 0x7f9f9015b280 2023-10-16T12:26:55.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.094+0000 7f9f8e7fc700 1 -- 172.21.15.16:0/1619660381 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f901621f0 con 0x7f9f900cc8a0 2023-10-16T12:26:55.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.094+0000 7f9f8e7fc700 1 --2- 172.21.15.16:0/1619660381 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f900cc8a0 0x7f9f9015e700 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f9f900cbed0 tx=0x7f9f80009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41e636605a6021f1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:55.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.094+0000 7f9f8dffb700 1 -- 172.21.15.16:0/1619660381 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f80011830 con 0x7f9f900cc8a0 2023-10-16T12:26:55.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.094+0000 7f9f8dffb700 1 -- 172.21.15.16:0/1619660381 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f80011990 con 0x7f9f900cc8a0 2023-10-16T12:26:55.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.094+0000 7f9f8dffb700 1 -- 172.21.15.16:0/1619660381 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f80011d20 con 0x7f9f900cc8a0 2023-10-16T12:26:55.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.094+0000 7f9f956ff700 1 -- 172.21.15.16:0/1619660381 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f900cc8a0 msgr2=0x7f9f9015e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:55.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.094+0000 7f9f956ff700 1 --2- 172.21.15.16:0/1619660381 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f900cc8a0 0x7f9f9015e700 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f9f900cbed0 tx=0x7f9f80009bd0 comp rx=0 tx=0).stop 2023-10-16T12:26:55.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.095+0000 7f9f956ff700 1 -- 172.21.15.16:0/1619660381 shutdown_connections 2023-10-16T12:26:55.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.095+0000 7f9f956ff700 1 --2- 172.21.15.16:0/1619660381 >> v2:172.21.15.49:3300/0 conn(0x7f9f9006a7e0 0x7f9f9006ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:55.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.095+0000 7f9f956ff700 1 --2- 172.21.15.16:0/1619660381 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f900cc8a0 0x7f9f9015e700 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:55.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.095+0000 7f9f956ff700 1 -- 172.21.15.16:0/1619660381 >> 172.21.15.16:0/1619660381 conn(0x7f9f900b1fd0 msgr2=0x7f9f900b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:55.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.095+0000 7f9f956ff700 1 -- 172.21.15.16:0/1619660381 shutdown_connections 2023-10-16T12:26:55.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.095+0000 7f9f956ff700 1 -- 172.21.15.16:0/1619660381 wait complete. 2023-10-16T12:26:55.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.096+0000 7f9f956ff700 1 Processor -- start 2023-10-16T12:26:55.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.096+0000 7f9f956ff700 1 -- start start 2023-10-16T12:26:55.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.097+0000 7f9f956ff700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9f9006a7e0 0x7f9f901f2820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:55.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.097+0000 7f9f956ff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f900cc8a0 0x7f9f900c5960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:55.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.097+0000 7f9f956ff700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9f9015b280 0x7f9f900ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:55.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.097+0000 7f9f956ff700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f900c62d0 con 0x7f9f900cc8a0 2023-10-16T12:26:55.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.097+0000 7f9f956ff700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f900c65b0 con 0x7f9f9015b280 2023-10-16T12:26:55.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.097+0000 7f9f956ff700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f900c6890 con 0x7f9f9006a7e0 2023-10-16T12:26:55.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.097+0000 7f9f8e7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f900cc8a0 0x7f9f900c5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:55.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.097+0000 7f9f8e7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f900cc8a0 0x7f9f900c5960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40654/0 (socket says 172.21.15.16:40654) 2023-10-16T12:26:55.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.097+0000 7f9f8e7fc700 1 -- 172.21.15.16:0/1022163677 learned_addr learned my addr 172.21.15.16:0/1022163677 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:55.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.098+0000 7f9f8f7fe700 1 --2- 172.21.15.16:0/1022163677 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9f9015b280 0x7f9f900ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:55.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.098+0000 7f9f8effd700 1 --2- 172.21.15.16:0/1022163677 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9f9006a7e0 0x7f9f901f2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:55.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.098+0000 7f9f8e7fc700 1 -- 172.21.15.16:0/1022163677 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9f9006a7e0 msgr2=0x7f9f901f2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:55.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.098+0000 7f9f8e7fc700 1 --2- 172.21.15.16:0/1022163677 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9f9006a7e0 0x7f9f901f2820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:55.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.098+0000 7f9f8e7fc700 1 -- 172.21.15.16:0/1022163677 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9f9015b280 msgr2=0x7f9f900ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:55.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.098+0000 7f9f8e7fc700 1 --2- 172.21.15.16:0/1022163677 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9f9015b280 0x7f9f900ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:55.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.098+0000 7f9f8e7fc700 1 -- 172.21.15.16:0/1022163677 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f80009550 con 0x7f9f900cc8a0 2023-10-16T12:26:55.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.098+0000 7f9f8f7fe700 1 --2- 172.21.15.16:0/1022163677 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9f9015b280 0x7f9f900ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:26:55.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.098+0000 7f9f8e7fc700 1 --2- 172.21.15.16:0/1022163677 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f900cc8a0 0x7f9f900c5960 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f9f80009c10 tx=0x7f9f80004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:55.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.098+0000 7f9f777fe700 1 -- 172.21.15.16:0/1022163677 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f800036a0 con 0x7f9f900cc8a0 2023-10-16T12:26:55.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.098+0000 7f9f777fe700 1 -- 172.21.15.16:0/1022163677 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f80003800 con 0x7f9f900cc8a0 2023-10-16T12:26:55.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.098+0000 7f9f956ff700 1 -- 172.21.15.16:0/1022163677 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f900c4010 con 0x7f9f900cc8a0 2023-10-16T12:26:55.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.098+0000 7f9f777fe700 1 -- 172.21.15.16:0/1022163677 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f80003b70 con 0x7f9f900cc8a0 2023-10-16T12:26:55.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.098+0000 7f9f956ff700 1 -- 172.21.15.16:0/1022163677 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f901f2d60 con 0x7f9f900cc8a0 2023-10-16T12:26:55.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.099+0000 7f9f956ff700 1 -- 172.21.15.16:0/1022163677 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f900c4530 con 0x7f9f900cc8a0 2023-10-16T12:26:55.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.100+0000 7f9f777fe700 1 -- 172.21.15.16:0/1022163677 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 41) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9f80003cd0 con 0x7f9f900cc8a0 2023-10-16T12:26:55.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.100+0000 7f9f777fe700 1 --2- 172.21.15.16:0/1022163677 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f9f7c06b020 0x7f9f7c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:55.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.100+0000 7f9f777fe700 1 -- 172.21.15.16:0/1022163677 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5979+0+0 (secure 0 0 0) 0x7f9f80021030 con 0x7f9f900cc8a0 2023-10-16T12:26:55.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.100+0000 7f9f8effd700 1 --2- 172.21.15.16:0/1022163677 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f9f7c06b020 0x7f9f7c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:55.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.101+0000 7f9f8effd700 1 --2- 172.21.15.16:0/1022163677 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f9f7c06b020 0x7f9f7c06d4e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9f84007fd0 tx=0x7f9f84009040 comp rx=0 tx=0).ready entity=mgr.6020 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:55.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.105+0000 7f9f777fe700 1 -- 172.21.15.16:0/1022163677 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9f900c4530 con 0x7f9f900cc8a0 2023-10-16T12:26:55.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.352+0000 7f9f956ff700 1 -- 172.21.15.16:0/1022163677 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9f900c4530 con 0x7f9f900cc8a0 2023-10-16T12:26:55.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.353+0000 7f9f777fe700 1 -- 172.21.15.16:0/1022163677 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+3699 (secure 0 0 0) 0x7f9f900c4530 con 0x7f9f900cc8a0 2023-10-16T12:26:55.356 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:26:55.356 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":38,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5861,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/1236108916","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":1236108916},{"type":"v1","addr":"172.21.15.49:6834","nonce":1236108916}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31},{"gid":5864,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/2762335434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":2762335434},{"type":"v1","addr":"172.21.15.49:6835","nonce":2762335434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31},{"gid":6379,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/1502256436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1502256436},{"type":"v1","addr":"172.21.15.16:6836","nonce":1502256436}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":7017,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3335751877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":3335751877},{"type":"v1","addr":"172.21.15.16:6837","nonce":3335751877}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38}],"filesystems":[]} 2023-10-16T12:26:55.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.355+0000 7f9f956ff700 1 -- 172.21.15.16:0/1022163677 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f9f7c06b020 msgr2=0x7f9f7c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:55.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.355+0000 7f9f956ff700 1 --2- 172.21.15.16:0/1022163677 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f9f7c06b020 0x7f9f7c06d4e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9f84007fd0 tx=0x7f9f84009040 comp rx=0 tx=0).stop 2023-10-16T12:26:55.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.355+0000 7f9f956ff700 1 -- 172.21.15.16:0/1022163677 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f900cc8a0 msgr2=0x7f9f900c5960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:55.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.355+0000 7f9f956ff700 1 --2- 172.21.15.16:0/1022163677 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f900cc8a0 0x7f9f900c5960 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f9f80009c10 tx=0x7f9f80004750 comp rx=0 tx=0).stop 2023-10-16T12:26:55.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.355+0000 7f9f956ff700 1 -- 172.21.15.16:0/1022163677 shutdown_connections 2023-10-16T12:26:55.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.355+0000 7f9f956ff700 1 --2- 172.21.15.16:0/1022163677 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f9f7c06b020 0x7f9f7c06d4e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:55.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.355+0000 7f9f956ff700 1 --2- 172.21.15.16:0/1022163677 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9f9015b280 0x7f9f900ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:55.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.355+0000 7f9f956ff700 1 --2- 172.21.15.16:0/1022163677 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9f9006a7e0 0x7f9f901f2820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:55.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.355+0000 7f9f956ff700 1 --2- 172.21.15.16:0/1022163677 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f900cc8a0 0x7f9f900c5960 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:55.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.355+0000 7f9f956ff700 1 -- 172.21.15.16:0/1022163677 >> 172.21.15.16:0/1022163677 conn(0x7f9f900b1fd0 msgr2=0x7f9f90072600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:55.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.355+0000 7f9f956ff700 1 -- 172.21.15.16:0/1022163677 shutdown_connections 2023-10-16T12:26:55.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.355+0000 7f9f956ff700 1 -- 172.21.15.16:0/1022163677 wait complete. 2023-10-16T12:26:55.360 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 38 2023-10-16T12:26:55.370 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-16T12:26:55.370 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-16T12:26:55.371 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-16T12:26:55.371 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T12:26:55.371 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-16T12:26:55.371 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-16T12:26:55.372 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-16T12:26:55.372 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T12:26:55.372 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-16T12:26:55.372 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-16T12:26:55.372 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-16T12:26:55.373 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T12:26:55.373 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-16T12:26:55.373 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-16T12:26:55.373 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-16T12:26:55.374 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T12:26:55.380 INFO:tasks.ceph.mds.a:Stopped 2023-10-16T12:26:55.380 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-16T12:26:55.381 INFO:tasks.ceph.mds.c:Stopped 2023-10-16T12:26:55.381 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-16T12:26:55.384 INFO:tasks.ceph.mds.b:Stopped 2023-10-16T12:26:55.384 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-16T12:26:55.385 INFO:tasks.ceph.mds.d:Stopped 2023-10-16T12:26:55.385 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-16T12:26:55.423 INFO:tasks.ceph.mds.a:Started 2023-10-16T12:26:55.423 DEBUG:teuthology.parallel:result is None 2023-10-16T12:26:55.424 INFO:tasks.ceph.mds.c:Started 2023-10-16T12:26:55.424 DEBUG:teuthology.parallel:result is None 2023-10-16T12:26:55.426 INFO:tasks.ceph.mds.b:Started 2023-10-16T12:26:55.427 DEBUG:teuthology.parallel:result is None 2023-10-16T12:26:55.427 INFO:tasks.ceph.mds.d:Started 2023-10-16T12:26:55.428 DEBUG:teuthology.parallel:result is None 2023-10-16T12:26:55.428 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-16T12:26:55.505 INFO:tasks.ceph.mds.a.smithi016.stdout:starting mds.a at 2023-10-16T12:26:55.507 INFO:tasks.ceph.mds.c.smithi016.stdout:starting mds.c at 2023-10-16T12:26:55.507 INFO:tasks.ceph.mds.c.smithi016.stderr:2023-10-16T12:26:55.504+0000 7f06ab64b700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6834/0: (98) Address already in use 2023-10-16T12:26:55.519 INFO:tasks.ceph.mds.b.smithi049.stdout:starting mds.b at 2023-10-16T12:26:55.519 INFO:tasks.ceph.mds.d.smithi049.stdout:starting mds.d at 2023-10-16T12:26:55.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.559+0000 7fd77490b700 1 Processor -- start 2023-10-16T12:26:55.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.559+0000 7fd77490b700 1 -- start start 2023-10-16T12:26:55.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.560+0000 7fd77490b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd7700cc8a0 0x7fd77015e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:55.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.560+0000 7fd77490b700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd770161c30 con 0x7fd77006a7e0 2023-10-16T12:26:55.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.560+0000 7fd77490b700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd770161f10 con 0x7fd77015b280 2023-10-16T12:26:55.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.560+0000 7fd77490b700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7701621f0 con 0x7fd7700cc8a0 2023-10-16T12:26:55.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.560+0000 7fd76dd9b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd7700cc8a0 0x7fd77015e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:55.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.560+0000 7fd76ed9d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd77006a7e0 0x7fd7700d02b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:58194/0 (socket says 172.21.15.16:58194) 2023-10-16T12:26:55.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.560+0000 7fd76ed9d700 1 -- 172.21.15.16:0/2359398440 learned_addr learned my addr 172.21.15.16:0/2359398440 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:26:55.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.560+0000 7fd76d59a700 1 -- 172.21.15.16:0/2359398440 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2047139538 0 0) 0x7fd770161c30 con 0x7fd77006a7e0 2023-10-16T12:26:55.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.560+0000 7fd76d59a700 1 -- 172.21.15.16:0/2359398440 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7580035b0 con 0x7fd77006a7e0 2023-10-16T12:26:55.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.560+0000 7fd76dd9b700 1 -- 172.21.15.16:0/2359398440 >> v1:172.21.15.49:6789/0 conn(0x7fd77015b280 legacy=0x7fd7701613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:55.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.560+0000 7fd76dd9b700 1 -- 172.21.15.16:0/2359398440 >> v1:172.21.15.16:6789/0 conn(0x7fd77006a7e0 legacy=0x7fd7700d02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:55.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.560+0000 7fd76dd9b700 1 -- 172.21.15.16:0/2359398440 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd77015df00 con 0x7fd7700cc8a0 2023-10-16T12:26:55.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.560+0000 7fd76dd9b700 1 --2- 172.21.15.16:0/2359398440 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7700cc8a0 0x7fd77015e700 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fd764009840 tx=0x7fd764009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4a603c9abf101d4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:55.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.561+0000 7fd76d59a700 1 -- 172.21.15.16:0/2359398440 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd764011830 con 0x7fd7700cc8a0 2023-10-16T12:26:55.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.561+0000 7fd76d59a700 1 -- 172.21.15.16:0/2359398440 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd764011990 con 0x7fd7700cc8a0 2023-10-16T12:26:55.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.561+0000 7fd76d59a700 1 -- 172.21.15.16:0/2359398440 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd764011d00 con 0x7fd7700cc8a0 2023-10-16T12:26:55.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.561+0000 7fd77490b700 1 -- 172.21.15.16:0/2359398440 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7700cc8a0 msgr2=0x7fd77015e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:55.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.561+0000 7fd77490b700 1 --2- 172.21.15.16:0/2359398440 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7700cc8a0 0x7fd77015e700 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fd764009840 tx=0x7fd764009b50 comp rx=0 tx=0).stop 2023-10-16T12:26:55.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.561+0000 7fd77490b700 1 -- 172.21.15.16:0/2359398440 shutdown_connections 2023-10-16T12:26:55.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.561+0000 7fd77490b700 1 --2- 172.21.15.16:0/2359398440 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7700cc8a0 0x7fd77015e700 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:55.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.561+0000 7fd77490b700 1 -- 172.21.15.16:0/2359398440 >> 172.21.15.16:0/2359398440 conn(0x7fd7700b1fd0 msgr2=0x7fd7700b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:55.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.561+0000 7fd77490b700 1 -- 172.21.15.16:0/2359398440 shutdown_connections 2023-10-16T12:26:55.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.561+0000 7fd77490b700 1 -- 172.21.15.16:0/2359398440 wait complete. 2023-10-16T12:26:55.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.563+0000 7fd77490b700 1 Processor -- start 2023-10-16T12:26:55.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.563+0000 7fd77490b700 1 -- start start 2023-10-16T12:26:55.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.563+0000 7fd77490b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd77006a7e0 0x7fd7701f2590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:55.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.563+0000 7fd77490b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd7700cc8a0 0x7fd770158200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:55.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.563+0000 7fd77490b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd77015b280 0x7fd7701f6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:55.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.563+0000 7fd77490b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd77015f4b0 con 0x7fd77006a7e0 2023-10-16T12:26:55.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.563+0000 7fd77490b700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd770158c50 con 0x7fd7700cc8a0 2023-10-16T12:26:55.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.563+0000 7fd77490b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd770158f30 con 0x7fd77015b280 2023-10-16T12:26:55.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.563+0000 7fd76e59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd77006a7e0 0x7fd7701f2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:55.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.563+0000 7fd76e59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd77006a7e0 0x7fd7701f2590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40690/0 (socket says 172.21.15.16:40690) 2023-10-16T12:26:55.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.564+0000 7fd76e59c700 1 -- 172.21.15.16:0/281320860 learned_addr learned my addr 172.21.15.16:0/281320860 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:55.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.564+0000 7fd76dd9b700 1 --2- 172.21.15.16:0/281320860 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd7700cc8a0 0x7fd770158200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:55.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.564+0000 7fd76ed9d700 1 --2- 172.21.15.16:0/281320860 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd77015b280 0x7fd7701f6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:55.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.564+0000 7fd76e59c700 1 -- 172.21.15.16:0/281320860 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd77015b280 msgr2=0x7fd7701f6380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:55.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.564+0000 7fd76e59c700 1 --2- 172.21.15.16:0/281320860 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd77015b280 0x7fd7701f6380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:55.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.564+0000 7fd76e59c700 1 -- 172.21.15.16:0/281320860 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd7700cc8a0 msgr2=0x7fd770158200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:55.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.564+0000 7fd76e59c700 1 --2- 172.21.15.16:0/281320860 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd7700cc8a0 0x7fd770158200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:55.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.564+0000 7fd76e59c700 1 -- 172.21.15.16:0/281320860 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7640094a0 con 0x7fd77006a7e0 2023-10-16T12:26:55.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.564+0000 7fd76e59c700 1 --2- 172.21.15.16:0/281320860 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd77006a7e0 0x7fd7701f2590 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7fd76000c430 tx=0x7fd76000c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:55.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.564+0000 7fd75effd700 1 -- 172.21.15.16:0/281320860 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7600050d0 con 0x7fd77006a7e0 2023-10-16T12:26:55.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.564+0000 7fd75effd700 1 -- 172.21.15.16:0/281320860 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd760005250 con 0x7fd77006a7e0 2023-10-16T12:26:55.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.564+0000 7fd77490b700 1 -- 172.21.15.16:0/281320860 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd77015ee60 con 0x7fd77006a7e0 2023-10-16T12:26:55.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.564+0000 7fd75effd700 1 -- 172.21.15.16:0/281320860 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd760006860 con 0x7fd77006a7e0 2023-10-16T12:26:55.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.564+0000 7fd77490b700 1 -- 172.21.15.16:0/281320860 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7700c5730 con 0x7fd77006a7e0 2023-10-16T12:26:55.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.565+0000 7fd77490b700 1 -- 172.21.15.16:0/281320860 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7700c58c0 con 0x7fd77006a7e0 2023-10-16T12:26:55.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.566+0000 7fd75effd700 1 -- 172.21.15.16:0/281320860 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 41) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd760003680 con 0x7fd77006a7e0 2023-10-16T12:26:55.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.566+0000 7fd75effd700 1 --2- 172.21.15.16:0/281320860 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7fd75006b020 0x7fd75006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:55.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.566+0000 7fd75effd700 1 -- 172.21.15.16:0/281320860 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5979+0+0 (secure 0 0 0) 0x7fd760088940 con 0x7fd77006a7e0 2023-10-16T12:26:55.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.566+0000 7fd76dd9b700 1 --2- 172.21.15.16:0/281320860 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7fd75006b020 0x7fd75006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:55.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.567+0000 7fd76dd9b700 1 --2- 172.21.15.16:0/281320860 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7fd75006b020 0x7fd75006d4e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd764009840 tx=0x7fd764008030 comp rx=0 tx=0).ready entity=mgr.6020 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:55.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.571+0000 7fd75effd700 1 -- 172.21.15.16:0/281320860 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd7600537d0 con 0x7fd77006a7e0 2023-10-16T12:26:55.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:55.801+0000 7fd77490b700 1 -- 172.21.15.16:0/281320860 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fd77004ebb0 con 0x7fd77006a7e0 2023-10-16T12:26:57.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.259+0000 7fd75effd700 1 -- 172.21.15.16:0/281320860 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v95) v1 ==== 97+0+0 (secure 0 0 0) 0x7fd760056df0 con 0x7fd77006a7e0 2023-10-16T12:26:57.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.262+0000 7fd77490b700 1 -- 172.21.15.16:0/281320860 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7fd75006b020 msgr2=0x7fd75006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:57.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.262+0000 7fd77490b700 1 --2- 172.21.15.16:0/281320860 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7fd75006b020 0x7fd75006d4e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd764009840 tx=0x7fd764008030 comp rx=0 tx=0).stop 2023-10-16T12:26:57.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.262+0000 7fd77490b700 1 -- 172.21.15.16:0/281320860 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd77006a7e0 msgr2=0x7fd7701f2590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:57.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.262+0000 7fd77490b700 1 --2- 172.21.15.16:0/281320860 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd77006a7e0 0x7fd7701f2590 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7fd76000c430 tx=0x7fd76000c740 comp rx=0 tx=0).stop 2023-10-16T12:26:57.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.262+0000 7fd77490b700 1 -- 172.21.15.16:0/281320860 shutdown_connections 2023-10-16T12:26:57.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.262+0000 7fd77490b700 1 --2- 172.21.15.16:0/281320860 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7fd75006b020 0x7fd75006d4e0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:57.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.262+0000 7fd77490b700 1 --2- 172.21.15.16:0/281320860 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd77015b280 0x7fd7701f6380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:57.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.262+0000 7fd77490b700 1 --2- 172.21.15.16:0/281320860 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd77006a7e0 0x7fd7701f2590 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:57.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.262+0000 7fd77490b700 1 --2- 172.21.15.16:0/281320860 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd7700cc8a0 0x7fd770158200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:57.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.262+0000 7fd77490b700 1 -- 172.21.15.16:0/281320860 >> 172.21.15.16:0/281320860 conn(0x7fd7700b1fd0 msgr2=0x7fd77015a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:57.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.262+0000 7fd77490b700 1 -- 172.21.15.16:0/281320860 shutdown_connections 2023-10-16T12:26:57.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.263+0000 7fd77490b700 1 -- 172.21.15.16:0/281320860 wait complete. 2023-10-16T12:26:57.267 INFO:teuthology.orchestra.run.smithi016.stderr: removed all blocklist entries 2023-10-16T12:26:57.279 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-16T12:26:57.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.411+0000 7fb91eb4d700 1 Processor -- start 2023-10-16T12:26:57.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.411+0000 7fb91eb4d700 1 -- start start 2023-10-16T12:26:57.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.411+0000 7fb91eb4d700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fb91815b860 0x7fb91806a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:57.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.411+0000 7fb91eb4d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb9180b2be0 0x7fb91806ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:57.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.411+0000 7fb91eb4d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb9180cd770 0x7fb9180719e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:57.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.411+0000 7fb91eb4d700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9180d03d0 con 0x7fb9180b2be0 2023-10-16T12:26:57.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.411+0000 7fb91eb4d700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9180d11c0 con 0x7fb91815b860 2023-10-16T12:26:57.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.411+0000 7fb91eb4d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9180cd360 con 0x7fb9180cd770 2023-10-16T12:26:57.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.412+0000 7fb917fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb9180b2be0 0x7fb91806ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:57.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.412+0000 7fb917fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb9180b2be0 0x7fb91806ad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40702/0 (socket says 172.21.15.16:40702) 2023-10-16T12:26:57.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.412+0000 7fb917fff700 1 -- 172.21.15.16:0/3435683055 learned_addr learned my addr 172.21.15.16:0/3435683055 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:57.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.412+0000 7fb91d0ea700 1 --2- 172.21.15.16:0/3435683055 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb9180cd770 0x7fb9180719e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:57.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.412+0000 7fb91c8e9700 1 --2- 172.21.15.16:0/3435683055 >> v2:172.21.15.49:3300/0 conn(0x7fb91815b860 0x7fb91806a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:57.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.412+0000 7fb917fff700 1 -- 172.21.15.16:0/3435683055 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb9180cd770 msgr2=0x7fb9180719e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:57.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.412+0000 7fb917fff700 1 --2- 172.21.15.16:0/3435683055 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb9180cd770 0x7fb9180719e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:57.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.412+0000 7fb917fff700 1 -- 172.21.15.16:0/3435683055 >> v2:172.21.15.49:3300/0 conn(0x7fb91815b860 msgr2=0x7fb91806a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:57.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.412+0000 7fb917fff700 1 --2- 172.21.15.16:0/3435683055 >> v2:172.21.15.49:3300/0 conn(0x7fb91815b860 0x7fb91806a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:57.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.412+0000 7fb917fff700 1 -- 172.21.15.16:0/3435683055 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9180d05b0 con 0x7fb9180b2be0 2023-10-16T12:26:57.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.412+0000 7fb91c8e9700 1 --2- 172.21.15.16:0/3435683055 >> v2:172.21.15.49:3300/0 conn(0x7fb91815b860 0x7fb91806a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:26:57.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.412+0000 7fb917fff700 1 --2- 172.21.15.16:0/3435683055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9180b2be0 0x7fb91806ad20 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fb918072e20 tx=0x7fb904009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41033e5b73d9e096 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:57.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.412+0000 7fb9177fe700 1 -- 172.21.15.16:0/3435683055 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb904011830 con 0x7fb9180b2be0 2023-10-16T12:26:57.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.412+0000 7fb9177fe700 1 -- 172.21.15.16:0/3435683055 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb904011990 con 0x7fb9180b2be0 2023-10-16T12:26:57.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.412+0000 7fb9177fe700 1 -- 172.21.15.16:0/3435683055 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb904011d20 con 0x7fb9180b2be0 2023-10-16T12:26:57.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.413+0000 7fb91eb4d700 1 -- 172.21.15.16:0/3435683055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9180b2be0 msgr2=0x7fb91806ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:57.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.413+0000 7fb91eb4d700 1 --2- 172.21.15.16:0/3435683055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9180b2be0 0x7fb91806ad20 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fb918072e20 tx=0x7fb904009bd0 comp rx=0 tx=0).stop 2023-10-16T12:26:57.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.413+0000 7fb91eb4d700 1 -- 172.21.15.16:0/3435683055 shutdown_connections 2023-10-16T12:26:57.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.413+0000 7fb91eb4d700 1 --2- 172.21.15.16:0/3435683055 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb9180cd770 0x7fb9180719e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:57.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.413+0000 7fb91eb4d700 1 --2- 172.21.15.16:0/3435683055 >> v2:172.21.15.49:3300/0 conn(0x7fb91815b860 0x7fb91806a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:57.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.413+0000 7fb91eb4d700 1 --2- 172.21.15.16:0/3435683055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9180b2be0 0x7fb91806ad20 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:57.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.413+0000 7fb91eb4d700 1 -- 172.21.15.16:0/3435683055 >> 172.21.15.16:0/3435683055 conn(0x7fb9180b1fd0 msgr2=0x7fb9180b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:57.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.413+0000 7fb91eb4d700 1 -- 172.21.15.16:0/3435683055 shutdown_connections 2023-10-16T12:26:57.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.413+0000 7fb91eb4d700 1 -- 172.21.15.16:0/3435683055 wait complete. 2023-10-16T12:26:57.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.414+0000 7fb91eb4d700 1 Processor -- start 2023-10-16T12:26:57.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.415+0000 7fb91eb4d700 1 -- start start 2023-10-16T12:26:57.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.416+0000 7fb91eb4d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9180b2be0 0x7fb9180ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:57.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.416+0000 7fb91eb4d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb9180cd770 0x7fb9181f28a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:57.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.416+0000 7fb91eb4d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb91815b860 0x7fb9181e4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:57.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.416+0000 7fb91eb4d700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9180ce010 con 0x7fb9180b2be0 2023-10-16T12:26:57.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.416+0000 7fb91eb4d700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9180ceb90 con 0x7fb91815b860 2023-10-16T12:26:57.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.416+0000 7fb91eb4d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9180ce260 con 0x7fb9180cd770 2023-10-16T12:26:57.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.416+0000 7fb91c8e9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9180b2be0 0x7fb9180ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:57.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.416+0000 7fb91c8e9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9180b2be0 0x7fb9180ac8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40708/0 (socket says 172.21.15.16:40708) 2023-10-16T12:26:57.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.416+0000 7fb91c8e9700 1 -- 172.21.15.16:0/953598293 learned_addr learned my addr 172.21.15.16:0/953598293 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:57.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.416+0000 7fb917fff700 1 --2- 172.21.15.16:0/953598293 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb9180cd770 0x7fb9181f28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:57.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.416+0000 7fb91d0ea700 1 --2- 172.21.15.16:0/953598293 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb91815b860 0x7fb9181e4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:57.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.416+0000 7fb91c8e9700 1 -- 172.21.15.16:0/953598293 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb9180cd770 msgr2=0x7fb9181f28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:57.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.416+0000 7fb91c8e9700 1 --2- 172.21.15.16:0/953598293 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb9180cd770 0x7fb9181f28a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:57.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.416+0000 7fb91c8e9700 1 -- 172.21.15.16:0/953598293 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb91815b860 msgr2=0x7fb9181e4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:57.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.416+0000 7fb91c8e9700 1 --2- 172.21.15.16:0/953598293 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb91815b860 0x7fb9181e4e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:57.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.416+0000 7fb91c8e9700 1 -- 172.21.15.16:0/953598293 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb904009550 con 0x7fb9180b2be0 2023-10-16T12:26:57.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.416+0000 7fb917fff700 1 --2- 172.21.15.16:0/953598293 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb9180cd770 0x7fb9181f28a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:26:57.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.416+0000 7fb91c8e9700 1 --2- 172.21.15.16:0/953598293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9180b2be0 0x7fb9180ac8d0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7fb90c00e840 tx=0x7fb90c00ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:57.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.416+0000 7fb9157fa700 1 -- 172.21.15.16:0/953598293 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb90c004230 con 0x7fb9180b2be0 2023-10-16T12:26:57.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.416+0000 7fb9157fa700 1 -- 172.21.15.16:0/953598293 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb90c004390 con 0x7fb9180b2be0 2023-10-16T12:26:57.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.416+0000 7fb91eb4d700 1 -- 172.21.15.16:0/953598293 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9180ce540 con 0x7fb9180b2be0 2023-10-16T12:26:57.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.417+0000 7fb9157fa700 1 -- 172.21.15.16:0/953598293 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb90c0036a0 con 0x7fb9180b2be0 2023-10-16T12:26:57.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.417+0000 7fb91eb4d700 1 -- 172.21.15.16:0/953598293 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9180cf020 con 0x7fb9180b2be0 2023-10-16T12:26:57.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.418+0000 7fb91eb4d700 1 -- 172.21.15.16:0/953598293 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9180cee70 con 0x7fb9180b2be0 2023-10-16T12:26:57.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.418+0000 7fb9157fa700 1 -- 172.21.15.16:0/953598293 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 41) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb90c003800 con 0x7fb9180b2be0 2023-10-16T12:26:57.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.418+0000 7fb9157fa700 1 --2- 172.21.15.16:0/953598293 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7fb8fc06b020 0x7fb8fc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:57.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.419+0000 7fb9157fa700 1 -- 172.21.15.16:0/953598293 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5205+0+0 (secure 0 0 0) 0x7fb90c0891e0 con 0x7fb9180b2be0 2023-10-16T12:26:57.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.419+0000 7fb917fff700 1 --2- 172.21.15.16:0/953598293 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7fb8fc06b020 0x7fb8fc06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:57.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.419+0000 7fb917fff700 1 --2- 172.21.15.16:0/953598293 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7fb8fc06b020 0x7fb8fc06d4e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb9181e5ce0 tx=0x7fb904008030 comp rx=0 tx=0).ready entity=mgr.6020 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:57.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.423+0000 7fb9157fa700 1 -- 172.21.15.16:0/953598293 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb90c054260 con 0x7fb9180b2be0 2023-10-16T12:26:57.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.711+0000 7fb91eb4d700 1 -- 172.21.15.16:0/953598293 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fb91804ebb0 con 0x7fb9180b2be0 2023-10-16T12:26:57.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.711+0000 7fb9157fa700 1 -- 172.21.15.16:0/953598293 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v20) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fb90c0094c0 con 0x7fb9180b2be0 2023-10-16T12:26:57.714 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:26:57.714 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-10-16T12:26:57.714 INFO:teuthology.orchestra.run.smithi016.stdout: "auth_dump": [ 2023-10-16T12:26:57.715 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:26:57.715 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.a", 2023-10-16T12:26:57.715 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOKC1l/rBtOhAA/ob5WUvpz8Glp4E+xPZOYA==", 2023-10-16T12:26:57.715 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:26:57.715 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T12:26:57.716 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-10-16T12:26:57.716 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-10-16T12:26:57.716 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:26:57.716 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:26:57.716 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:26:57.717 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:26:57.717 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.b", 2023-10-16T12:26:57.717 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1lKAVsCxAAvhXuLY6i1+Iz1yq5BVPMNQ==", 2023-10-16T12:26:57.717 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:26:57.717 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T12:26:57.718 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-10-16T12:26:57.718 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-10-16T12:26:57.718 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:26:57.718 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:26:57.718 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:26:57.718 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:26:57.719 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.c", 2023-10-16T12:26:57.719 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1lvBowBhAA3waJNYhdgs8PGCYqYNYUog==", 2023-10-16T12:26:57.719 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:26:57.719 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T12:26:57.719 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-10-16T12:26:57.720 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-10-16T12:26:57.720 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:26:57.720 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:26:57.720 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:26:57.720 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:26:57.720 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.d", 2023-10-16T12:26:57.721 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1lS/fnEhAAfNYgQ5BW86JMjYm/bBfnaQ==", 2023-10-16T12:26:57.721 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:26:57.721 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T12:26:57.721 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-10-16T12:26:57.721 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-10-16T12:26:57.722 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:26:57.722 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:26:57.722 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:26:57.722 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:26:57.722 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.0", 2023-10-16T12:26:57.723 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAQKC1lbxllOhAAZeLdp2GMl6HqQPaQjKlNqw==", 2023-10-16T12:26:57.723 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:26:57.723 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T12:26:57.723 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T12:26:57.723 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:26:57.723 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:26:57.724 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:26:57.724 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:26:57.724 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.1", 2023-10-16T12:26:57.724 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQATKC1lnU6KOhAAHxvlbmzBptf1keGgL+Rbkg==", 2023-10-16T12:26:57.724 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:26:57.725 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T12:26:57.725 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T12:26:57.725 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:26:57.725 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:26:57.725 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:26:57.725 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:26:57.726 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.2", 2023-10-16T12:26:57.726 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAXKC1l9jFyARAAr/NAwtWhBMut9sAAoDEGWg==", 2023-10-16T12:26:57.726 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:26:57.726 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T12:26:57.726 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T12:26:57.727 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:26:57.727 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:26:57.727 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:26:57.727 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:26:57.727 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.3", 2023-10-16T12:26:57.727 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAaKC1lbw/lARAAGJ/9DNb5/7kLFSq7vrLtXg==", 2023-10-16T12:26:57.728 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:26:57.728 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T12:26:57.728 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T12:26:57.728 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:26:57.728 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:26:57.729 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:26:57.729 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:26:57.729 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.4", 2023-10-16T12:26:57.729 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAeKC1lZb7jERAAYpvb1xcSli9CTq7L8ChLOA==", 2023-10-16T12:26:57.729 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:26:57.729 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T12:26:57.730 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T12:26:57.730 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:26:57.730 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:26:57.730 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:26:57.730 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:26:57.731 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.5", 2023-10-16T12:26:57.731 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAhKC1l/4HuEBAAqfnRqjknij9KES/fqiyIBQ==", 2023-10-16T12:26:57.731 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:26:57.731 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T12:26:57.731 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T12:26:57.732 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:26:57.732 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:26:57.732 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:26:57.732 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:26:57.732 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.6", 2023-10-16T12:26:57.733 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAkKC1lzoUkEBAA+Q82OiBMNag8+R316WQtUw==", 2023-10-16T12:26:57.733 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:26:57.733 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T12:26:57.733 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T12:26:57.733 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:26:57.734 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:26:57.734 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:26:57.734 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:26:57.734 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.7", 2023-10-16T12:26:57.734 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAnKC1ljUu4DxAAc+G4anJANC2GhLJ8wb6gYg==", 2023-10-16T12:26:57.734 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:26:57.735 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T12:26:57.735 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T12:26:57.735 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:26:57.735 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:26:57.735 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:26:57.736 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:26:57.736 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.0", 2023-10-16T12:26:57.736 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1l2l6PFxAA9QyXtBAHdGAXpwH0JfknuQ==", 2023-10-16T12:26:57.736 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:26:57.736 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T12:26:57.736 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow r", 2023-10-16T12:26:57.737 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-16T12:26:57.737 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:26:57.737 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:26:57.737 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:26:57.737 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.1", 2023-10-16T12:26:57.738 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1lAcHFIxAAcNt2bmv2R6j5L3DYkdw7eQ==", 2023-10-16T12:26:57.738 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:26:57.738 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T12:26:57.738 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-10-16T12:26:57.738 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-10-16T12:26:57.739 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-10-16T12:26:57.739 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:26:57.739 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:26:57.739 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:26:57.739 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.2", 2023-10-16T12:26:57.739 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1lA6wnGxAAdtaqYhtIJu8qENMrD8xvVA==", 2023-10-16T12:26:57.740 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:26:57.740 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T12:26:57.740 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-10-16T12:26:57.740 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-10-16T12:26:57.740 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-10-16T12:26:57.741 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:26:57.741 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:26:57.741 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:26:57.741 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.3", 2023-10-16T12:26:57.741 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1l/ddAJxAA3fNKKn9XkXas1BlDN+ggjw==", 2023-10-16T12:26:57.742 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:26:57.742 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T12:26:57.742 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-10-16T12:26:57.742 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-10-16T12:26:57.742 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-10-16T12:26:57.743 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:26:57.743 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:26:57.743 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:26:57.743 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.admin", 2023-10-16T12:26:57.743 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOKC1ld5jUHBAAc6esnCce3dSEZn3wVjhIlQ==", 2023-10-16T12:26:57.744 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:26:57.744 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2023-10-16T12:26:57.744 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-10-16T12:26:57.744 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-10-16T12:26:57.744 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:26:57.745 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:26:57.745 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:26:57.745 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:26:57.745 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mgr.x", 2023-10-16T12:26:57.745 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOKC1l6b3vNhAAmj5xbfid2sZM2KfHmAf1Wg==", 2023-10-16T12:26:57.746 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:26:57.746 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2023-10-16T12:26:57.746 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile mgr", 2023-10-16T12:26:57.746 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:26:57.746 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:26:57.747 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:26:57.747 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:26:57.747 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mgr.y", 2023-10-16T12:26:57.747 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOKC1lLz2nMhAA8NqFYZ5nDKkzPBMrkwsVuQ==", 2023-10-16T12:26:57.747 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:26:57.747 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2023-10-16T12:26:57.748 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile mgr", 2023-10-16T12:26:57.748 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:26:57.748 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:26:57.748 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:26:57.748 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-10-16T12:26:57.749 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-10-16T12:26:57.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.713+0000 7fb91eb4d700 1 -- 172.21.15.16:0/953598293 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7fb8fc06b020 msgr2=0x7fb8fc06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:57.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.713+0000 7fb91eb4d700 1 --2- 172.21.15.16:0/953598293 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7fb8fc06b020 0x7fb8fc06d4e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb9181e5ce0 tx=0x7fb904008030 comp rx=0 tx=0).stop 2023-10-16T12:26:57.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.713+0000 7fb91eb4d700 1 -- 172.21.15.16:0/953598293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9180b2be0 msgr2=0x7fb9180ac8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:57.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.713+0000 7fb91eb4d700 1 --2- 172.21.15.16:0/953598293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9180b2be0 0x7fb9180ac8d0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7fb90c00e840 tx=0x7fb90c00ec00 comp rx=0 tx=0).stop 2023-10-16T12:26:57.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.714+0000 7fb91eb4d700 1 -- 172.21.15.16:0/953598293 shutdown_connections 2023-10-16T12:26:57.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.714+0000 7fb91eb4d700 1 --2- 172.21.15.16:0/953598293 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7fb8fc06b020 0x7fb8fc06d4e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:57.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.714+0000 7fb91eb4d700 1 --2- 172.21.15.16:0/953598293 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb91815b860 0x7fb9181e4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:57.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.714+0000 7fb91eb4d700 1 --2- 172.21.15.16:0/953598293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9180b2be0 0x7fb9180ac8d0 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:57.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.714+0000 7fb91eb4d700 1 --2- 172.21.15.16:0/953598293 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb9180cd770 0x7fb9181f28a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:57.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.714+0000 7fb91eb4d700 1 -- 172.21.15.16:0/953598293 >> 172.21.15.16:0/953598293 conn(0x7fb9180b1fd0 msgr2=0x7fb91806a7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:57.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.714+0000 7fb91eb4d700 1 -- 172.21.15.16:0/953598293 shutdown_connections 2023-10-16T12:26:57.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.714+0000 7fb91eb4d700 1 -- 172.21.15.16:0/953598293 wait complete. 2023-10-16T12:26:57.753 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:26:57.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.876+0000 7fa6d07a5700 1 Processor -- start 2023-10-16T12:26:57.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.876+0000 7fa6d07a5700 1 -- start start 2023-10-16T12:26:57.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.877+0000 7fa6d07a5700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa6c815b630 0x7fa6c80d0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:57.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.877+0000 7fa6d07a5700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6c80b0970 con 0x7fa6c80ce4f0 2023-10-16T12:26:57.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.877+0000 7fa6d07a5700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6c80cd690 con 0x7fa6c815ba10 2023-10-16T12:26:57.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.877+0000 7fa6d07a5700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6c80cd970 con 0x7fa6c815b630 2023-10-16T12:26:57.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.877+0000 7fa6ce541700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa6c815b630 0x7fa6c80d0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:57.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.877+0000 7fa6ced42700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa6c80ce4f0 0x7fa6c80b1430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:58226/0 (socket says 172.21.15.16:58226) 2023-10-16T12:26:57.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.877+0000 7fa6ce541700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa6c815b630 0x7fa6c80d0380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40716/0 (socket says 172.21.15.16:40716) 2023-10-16T12:26:57.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.877+0000 7fa6ced42700 1 -- 172.21.15.16:0/3494007883 learned_addr learned my addr 172.21.15.16:0/3494007883 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:26:57.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.878+0000 7fa6ce541700 1 -- 172.21.15.16:0/3494007883 >> v1:172.21.15.49:6789/0 conn(0x7fa6c815ba10 legacy=0x7fa6c80cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:57.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.878+0000 7fa6ce541700 1 -- 172.21.15.16:0/3494007883 >> v1:172.21.15.16:6789/0 conn(0x7fa6c80ce4f0 legacy=0x7fa6c80b1430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:57.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.878+0000 7fa6cd53f700 1 -- 172.21.15.16:0/3494007883 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1206914020 0 0) 0x7fa6c80b0970 con 0x7fa6c80ce4f0 2023-10-16T12:26:57.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.878+0000 7fa6ce541700 1 -- 172.21.15.16:0/3494007883 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6c80b0c30 con 0x7fa6c815b630 2023-10-16T12:26:57.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.878+0000 7fa6ce541700 1 --2- 172.21.15.16:0/3494007883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6c815b630 0x7fa6c80d0380 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7fa6b8009840 tx=0x7fa6b8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7eafa725015009e3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:57.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.878+0000 7fa6cd53f700 1 -- 172.21.15.16:0/3494007883 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b8004230 con 0x7fa6c815b630 2023-10-16T12:26:57.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.878+0000 7fa6cd53f700 1 -- 172.21.15.16:0/3494007883 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6b8004390 con 0x7fa6c815b630 2023-10-16T12:26:57.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.878+0000 7fa6cd53f700 1 -- 172.21.15.16:0/3494007883 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b80036a0 con 0x7fa6c815b630 2023-10-16T12:26:57.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.878+0000 7fa6d07a5700 1 -- 172.21.15.16:0/3494007883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6c815b630 msgr2=0x7fa6c80d0380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:57.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.878+0000 7fa6d07a5700 1 --2- 172.21.15.16:0/3494007883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6c815b630 0x7fa6c80d0380 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7fa6b8009840 tx=0x7fa6b8009b50 comp rx=0 tx=0).stop 2023-10-16T12:26:57.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.879+0000 7fa6d07a5700 1 -- 172.21.15.16:0/3494007883 shutdown_connections 2023-10-16T12:26:57.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.879+0000 7fa6d07a5700 1 --2- 172.21.15.16:0/3494007883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6c815b630 0x7fa6c80d0380 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:57.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.879+0000 7fa6d07a5700 1 -- 172.21.15.16:0/3494007883 >> 172.21.15.16:0/3494007883 conn(0x7fa6c80d14b0 msgr2=0x7fa6c80d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:57.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.879+0000 7fa6d07a5700 1 -- 172.21.15.16:0/3494007883 shutdown_connections 2023-10-16T12:26:57.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.879+0000 7fa6d07a5700 1 -- 172.21.15.16:0/3494007883 wait complete. 2023-10-16T12:26:57.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.880+0000 7fa6d07a5700 1 Processor -- start 2023-10-16T12:26:57.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.880+0000 7fa6d07a5700 1 -- start start 2023-10-16T12:26:57.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.881+0000 7fa6d07a5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6c80ce4f0 0x7fa6c80b6b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:57.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.881+0000 7fa6d07a5700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa6c815b630 0x7fa6c81508b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:57.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.881+0000 7fa6d07a5700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa6c815ba10 0x7fa6c806bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:57.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.881+0000 7fa6d07a5700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c815e420 con 0x7fa6c80ce4f0 2023-10-16T12:26:57.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.881+0000 7fa6d07a5700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c815e700 con 0x7fa6c815ba10 2023-10-16T12:26:57.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.881+0000 7fa6d07a5700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c815e9e0 con 0x7fa6c815b630 2023-10-16T12:26:57.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.881+0000 7fa6ce541700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6c80ce4f0 0x7fa6c80b6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:57.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.881+0000 7fa6ce541700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6c80ce4f0 0x7fa6c80b6b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40722/0 (socket says 172.21.15.16:40722) 2023-10-16T12:26:57.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.881+0000 7fa6ce541700 1 -- 172.21.15.16:0/2354749768 learned_addr learned my addr 172.21.15.16:0/2354749768 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:57.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.881+0000 7fa6ced42700 1 --2- 172.21.15.16:0/2354749768 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa6c815ba10 0x7fa6c806bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:57.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.881+0000 7fa6cdd40700 1 --2- 172.21.15.16:0/2354749768 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa6c815b630 0x7fa6c81508b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:57.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.881+0000 7fa6ce541700 1 -- 172.21.15.16:0/2354749768 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa6c815b630 msgr2=0x7fa6c81508b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:57.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.881+0000 7fa6ce541700 1 --2- 172.21.15.16:0/2354749768 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa6c815b630 0x7fa6c81508b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:57.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.881+0000 7fa6ce541700 1 -- 172.21.15.16:0/2354749768 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa6c815ba10 msgr2=0x7fa6c806bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:57.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.881+0000 7fa6ce541700 1 --2- 172.21.15.16:0/2354749768 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa6c815ba10 0x7fa6c806bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:57.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.881+0000 7fa6ce541700 1 -- 172.21.15.16:0/2354749768 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6b80094a0 con 0x7fa6c80ce4f0 2023-10-16T12:26:57.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.881+0000 7fa6ced42700 1 --2- 172.21.15.16:0/2354749768 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa6c815ba10 0x7fa6c806bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:26:57.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.881+0000 7fa6ce541700 1 --2- 172.21.15.16:0/2354749768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6c80ce4f0 0x7fa6c80b6b30 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7fa6b80090b0 tx=0x7fa6b8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:57.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.881+0000 7fa6beffd700 1 -- 172.21.15.16:0/2354749768 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b8011830 con 0x7fa6c80ce4f0 2023-10-16T12:26:57.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.881+0000 7fa6beffd700 1 -- 172.21.15.16:0/2354749768 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6b801e070 con 0x7fa6c80ce4f0 2023-10-16T12:26:57.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.881+0000 7fa6d07a5700 1 -- 172.21.15.16:0/2354749768 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6c815ecc0 con 0x7fa6c80ce4f0 2023-10-16T12:26:57.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.881+0000 7fa6beffd700 1 -- 172.21.15.16:0/2354749768 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b8011bb0 con 0x7fa6c80ce4f0 2023-10-16T12:26:57.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.881+0000 7fa6d07a5700 1 -- 172.21.15.16:0/2354749768 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6c80b7070 con 0x7fa6c80ce4f0 2023-10-16T12:26:57.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.882+0000 7fa6d07a5700 1 -- 172.21.15.16:0/2354749768 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6c81f6840 con 0x7fa6c80ce4f0 2023-10-16T12:26:57.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.883+0000 7fa6beffd700 1 -- 172.21.15.16:0/2354749768 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 41) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa6b8011d10 con 0x7fa6c80ce4f0 2023-10-16T12:26:57.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.883+0000 7fa6beffd700 1 --2- 172.21.15.16:0/2354749768 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7fa6ac06b020 0x7fa6ac06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:57.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.883+0000 7fa6beffd700 1 -- 172.21.15.16:0/2354749768 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5205+0+0 (secure 0 0 0) 0x7fa6b808a8d0 con 0x7fa6c80ce4f0 2023-10-16T12:26:57.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.883+0000 7fa6cdd40700 1 --2- 172.21.15.16:0/2354749768 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7fa6ac06b020 0x7fa6ac06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:57.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.884+0000 7fa6cdd40700 1 --2- 172.21.15.16:0/2354749768 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7fa6ac06b020 0x7fa6ac06d4e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa6c80c4f60 tx=0x7fa6c4008040 comp rx=0 tx=0).ready entity=mgr.6020 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:57.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:57.888+0000 7fa6beffd700 1 -- 172.21.15.16:0/2354749768 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa6b8055af0 con 0x7fa6c80ce4f0 2023-10-16T12:26:58.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.092+0000 7fa6d07a5700 1 -- 172.21.15.16:0/2354749768 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa6c804ebb0 con 0x7fa6c80ce4f0 2023-10-16T12:26:58.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.093+0000 7fa6beffd700 1 -- 172.21.15.16:0/2354749768 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) v1 ==== 74+0+12998 (secure 0 0 0) 0x7fa6b8020020 con 0x7fa6c80ce4f0 2023-10-16T12:26:58.096 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:26:58.096 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":95,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:26:57.255359+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:26:58.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.095+0000 7fa6d07a5700 1 -- 172.21.15.16:0/2354749768 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7fa6ac06b020 msgr2=0x7fa6ac06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:58.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.095+0000 7fa6d07a5700 1 --2- 172.21.15.16:0/2354749768 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7fa6ac06b020 0x7fa6ac06d4e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa6c80c4f60 tx=0x7fa6c4008040 comp rx=0 tx=0).stop 2023-10-16T12:26:58.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.095+0000 7fa6d07a5700 1 -- 172.21.15.16:0/2354749768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6c80ce4f0 msgr2=0x7fa6c80b6b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:58.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.095+0000 7fa6d07a5700 1 --2- 172.21.15.16:0/2354749768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6c80ce4f0 0x7fa6c80b6b30 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7fa6b80090b0 tx=0x7fa6b8004750 comp rx=0 tx=0).stop 2023-10-16T12:26:58.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.095+0000 7fa6d07a5700 1 -- 172.21.15.16:0/2354749768 shutdown_connections 2023-10-16T12:26:58.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.095+0000 7fa6d07a5700 1 --2- 172.21.15.16:0/2354749768 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7fa6ac06b020 0x7fa6ac06d4e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:58.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.095+0000 7fa6d07a5700 1 --2- 172.21.15.16:0/2354749768 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa6c815ba10 0x7fa6c806bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:58.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.095+0000 7fa6d07a5700 1 --2- 172.21.15.16:0/2354749768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6c80ce4f0 0x7fa6c80b6b30 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:58.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.095+0000 7fa6d07a5700 1 --2- 172.21.15.16:0/2354749768 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa6c815b630 0x7fa6c81508b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:58.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.095+0000 7fa6d07a5700 1 -- 172.21.15.16:0/2354749768 >> 172.21.15.16:0/2354749768 conn(0x7fa6c80d14b0 msgr2=0x7fa6c80cf4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:58.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.096+0000 7fa6d07a5700 1 -- 172.21.15.16:0/2354749768 shutdown_connections 2023-10-16T12:26:58.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.096+0000 7fa6d07a5700 1 -- 172.21.15.16:0/2354749768 wait complete. 2023-10-16T12:26:58.110 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T12:26:58.111 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:26:58.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.237+0000 7efdbc70f700 1 Processor -- start 2023-10-16T12:26:58.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.237+0000 7efdbc70f700 1 -- start start 2023-10-16T12:26:58.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.238+0000 7efdbc70f700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7efdb415b2c0 0x7efdb4161340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:58.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.238+0000 7efdbc70f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7efdb406a7e0 0x7efdb406ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:58.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.238+0000 7efdbc70f700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdb415ecb0 con 0x7efdb406e570 2023-10-16T12:26:58.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.238+0000 7efdbc70f700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdb406b190 con 0x7efdb406a7e0 2023-10-16T12:26:58.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.238+0000 7efdbc70f700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdb4161880 con 0x7efdb415b2c0 2023-10-16T12:26:58.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.238+0000 7efdbacac700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7efdb406a7e0 0x7efdb406ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:58.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.238+0000 7efdbacac700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7efdb406a7e0 0x7efdb406ac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40734/0 (socket says 172.21.15.16:40734) 2023-10-16T12:26:58.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.238+0000 7efdbacac700 1 -- 172.21.15.16:0/2124280253 learned_addr learned my addr 172.21.15.16:0/2124280253 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:58.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.238+0000 7efdba4ab700 1 --2- 172.21.15.16:0/2124280253 >> v2:172.21.15.49:3300/0 conn(0x7efdb415b2c0 0x7efdb4161340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:58.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.238+0000 7efdbacac700 1 -- 172.21.15.16:0/2124280253 >> v2:172.21.15.49:3300/0 conn(0x7efdb415b2c0 msgr2=0x7efdb4161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:58.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.238+0000 7efdbacac700 1 --2- 172.21.15.16:0/2124280253 >> v2:172.21.15.49:3300/0 conn(0x7efdb415b2c0 0x7efdb4161340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:58.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.238+0000 7efdbacac700 1 -- 172.21.15.16:0/2124280253 >> v1:172.21.15.49:6789/0 conn(0x7efdb406e570 legacy=0x7efdb415e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:58.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.238+0000 7efdbacac700 1 -- 172.21.15.16:0/2124280253 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdb415b6a0 con 0x7efdb406a7e0 2023-10-16T12:26:58.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.238+0000 7efdbacac700 1 --2- 172.21.15.16:0/2124280253 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efdb406a7e0 0x7efdb406ac50 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7efdb406b4a0 tx=0x7efdac00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34d2180e470b192c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:58.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.238+0000 7efdb94a9700 1 -- 172.21.15.16:0/2124280253 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdac00e070 con 0x7efdb406a7e0 2023-10-16T12:26:58.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.238+0000 7efdb94a9700 1 -- 172.21.15.16:0/2124280253 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdac003a20 con 0x7efdb406a7e0 2023-10-16T12:26:58.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.238+0000 7efdb94a9700 1 -- 172.21.15.16:0/2124280253 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdac003dd0 con 0x7efdb406a7e0 2023-10-16T12:26:58.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.239+0000 7efdbc70f700 1 -- 172.21.15.16:0/2124280253 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efdb406a7e0 msgr2=0x7efdb406ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:58.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.239+0000 7efdbc70f700 1 --2- 172.21.15.16:0/2124280253 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efdb406a7e0 0x7efdb406ac50 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7efdb406b4a0 tx=0x7efdac00b330 comp rx=0 tx=0).stop 2023-10-16T12:26:58.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.239+0000 7efdbc70f700 1 -- 172.21.15.16:0/2124280253 shutdown_connections 2023-10-16T12:26:58.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.239+0000 7efdbc70f700 1 --2- 172.21.15.16:0/2124280253 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efdb406a7e0 0x7efdb406ac50 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:58.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.239+0000 7efdbc70f700 1 --2- 172.21.15.16:0/2124280253 >> v2:172.21.15.49:3300/0 conn(0x7efdb415b2c0 0x7efdb4161340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:58.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.239+0000 7efdbc70f700 1 -- 172.21.15.16:0/2124280253 >> 172.21.15.16:0/2124280253 conn(0x7efdb40b1f40 msgr2=0x7efdb40b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:58.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.239+0000 7efdbc70f700 1 -- 172.21.15.16:0/2124280253 shutdown_connections 2023-10-16T12:26:58.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.239+0000 7efdbc70f700 1 -- 172.21.15.16:0/2124280253 wait complete. 2023-10-16T12:26:58.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.240+0000 7efdbc70f700 1 Processor -- start 2023-10-16T12:26:58.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.240+0000 7efdbc70f700 1 -- start start 2023-10-16T12:26:58.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.241+0000 7efdbc70f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efdb406a7e0 0x7efdb41601a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:58.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.241+0000 7efdbc70f700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efdb406e570 0x7efdb41f34d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:58.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.241+0000 7efdbc70f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efdb415b2c0 0x7efdb415a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:58.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.241+0000 7efdbc70f700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdb40c7100 con 0x7efdb415b2c0 2023-10-16T12:26:58.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.241+0000 7efdbc70f700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdb40c68e0 con 0x7efdb406e570 2023-10-16T12:26:58.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.241+0000 7efdbc70f700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdb40c6b40 con 0x7efdb406a7e0 2023-10-16T12:26:58.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.241+0000 7efdbacac700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efdb415b2c0 0x7efdb415a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:58.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.241+0000 7efdbacac700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efdb415b2c0 0x7efdb415a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40740/0 (socket says 172.21.15.16:40740) 2023-10-16T12:26:58.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.241+0000 7efdbacac700 1 -- 172.21.15.16:0/3198464413 learned_addr learned my addr 172.21.15.16:0/3198464413 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:58.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.241+0000 7efdba4ab700 1 --2- 172.21.15.16:0/3198464413 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efdb406a7e0 0x7efdb41601a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:58.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.241+0000 7efdb9caa700 1 --2- 172.21.15.16:0/3198464413 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efdb406e570 0x7efdb41f34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:58.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.241+0000 7efdbacac700 1 -- 172.21.15.16:0/3198464413 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efdb406a7e0 msgr2=0x7efdb41601a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:58.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.241+0000 7efdbacac700 1 --2- 172.21.15.16:0/3198464413 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efdb406a7e0 0x7efdb41601a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:58.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.241+0000 7efdbacac700 1 -- 172.21.15.16:0/3198464413 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efdb406e570 msgr2=0x7efdb41f34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:58.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.241+0000 7efdbacac700 1 --2- 172.21.15.16:0/3198464413 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efdb406e570 0x7efdb41f34d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:58.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.241+0000 7efdbacac700 1 -- 172.21.15.16:0/3198464413 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdac009b30 con 0x7efdb415b2c0 2023-10-16T12:26:58.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.242+0000 7efdbacac700 1 --2- 172.21.15.16:0/3198464413 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efdb415b2c0 0x7efdb415a500 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7efdac015040 tx=0x7efdac004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:58.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.242+0000 7efdaaffd700 1 -- 172.21.15.16:0/3198464413 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdac004440 con 0x7efdb415b2c0 2023-10-16T12:26:58.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.242+0000 7efdaaffd700 1 -- 172.21.15.16:0/3198464413 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdac0045c0 con 0x7efdb415b2c0 2023-10-16T12:26:58.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.242+0000 7efdaaffd700 1 -- 172.21.15.16:0/3198464413 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdac004970 con 0x7efdb415b2c0 2023-10-16T12:26:58.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.242+0000 7efdbc70f700 1 -- 172.21.15.16:0/3198464413 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdb40c6e20 con 0x7efdb415b2c0 2023-10-16T12:26:58.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.242+0000 7efdbc70f700 1 -- 172.21.15.16:0/3198464413 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdb41f4780 con 0x7efdb415b2c0 2023-10-16T12:26:58.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.243+0000 7efdbc70f700 1 -- 172.21.15.16:0/3198464413 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdb404ebb0 con 0x7efdb415b2c0 2023-10-16T12:26:58.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.243+0000 7efdaaffd700 1 -- 172.21.15.16:0/3198464413 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 41) v1 ==== 89331+0+0 (secure 0 0 0) 0x7efdac012430 con 0x7efdb415b2c0 2023-10-16T12:26:58.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.244+0000 7efdaaffd700 1 --2- 172.21.15.16:0/3198464413 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7efd9806b020 0x7efd9806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:58.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.244+0000 7efdaaffd700 1 -- 172.21.15.16:0/3198464413 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5205+0+0 (secure 0 0 0) 0x7efdac08ab60 con 0x7efdb415b2c0 2023-10-16T12:26:58.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.244+0000 7efdba4ab700 1 --2- 172.21.15.16:0/3198464413 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7efd9806b020 0x7efd9806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:58.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.244+0000 7efdba4ab700 1 --2- 172.21.15.16:0/3198464413 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7efd9806b020 0x7efd9806d4e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7efdb000b9b0 tx=0x7efdb0008040 comp rx=0 tx=0).ready entity=mgr.6020 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:58.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.249+0000 7efdaaffd700 1 -- 172.21.15.16:0/3198464413 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7efdac055bd0 con 0x7efdb415b2c0 2023-10-16T12:26:58.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.453+0000 7efdbc70f700 1 -- 172.21.15.16:0/3198464413 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7efdb40ca720 con 0x7efdb415b2c0 2023-10-16T12:26:58.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.453+0000 7efdaaffd700 1 -- 172.21.15.16:0/3198464413 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v95) v1 ==== 93+0+10 (secure 0 0 0) 0x7efdac0170a0 con 0x7efdb415b2c0 2023-10-16T12:26:58.455 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T12:26:58.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.455+0000 7efdbc70f700 1 -- 172.21.15.16:0/3198464413 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7efd9806b020 msgr2=0x7efd9806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:58.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.455+0000 7efdbc70f700 1 --2- 172.21.15.16:0/3198464413 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7efd9806b020 0x7efd9806d4e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7efdb000b9b0 tx=0x7efdb0008040 comp rx=0 tx=0).stop 2023-10-16T12:26:58.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.455+0000 7efdbc70f700 1 -- 172.21.15.16:0/3198464413 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efdb415b2c0 msgr2=0x7efdb415a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:58.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.455+0000 7efdbc70f700 1 --2- 172.21.15.16:0/3198464413 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efdb415b2c0 0x7efdb415a500 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7efdac015040 tx=0x7efdac004ad0 comp rx=0 tx=0).stop 2023-10-16T12:26:58.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.455+0000 7efdbc70f700 1 -- 172.21.15.16:0/3198464413 shutdown_connections 2023-10-16T12:26:58.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.455+0000 7efdbc70f700 1 --2- 172.21.15.16:0/3198464413 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7efd9806b020 0x7efd9806d4e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:58.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.455+0000 7efdbc70f700 1 --2- 172.21.15.16:0/3198464413 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efdb415b2c0 0x7efdb415a500 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:58.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.455+0000 7efdbc70f700 1 --2- 172.21.15.16:0/3198464413 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efdb406a7e0 0x7efdb41601a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:58.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.455+0000 7efdbc70f700 1 --2- 172.21.15.16:0/3198464413 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efdb406e570 0x7efdb41f34d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:58.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.455+0000 7efdbc70f700 1 -- 172.21.15.16:0/3198464413 >> 172.21.15.16:0/3198464413 conn(0x7efdb40b1f40 msgr2=0x7efdb40cad40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:58.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.455+0000 7efdbc70f700 1 -- 172.21.15.16:0/3198464413 shutdown_connections 2023-10-16T12:26:58.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.455+0000 7efdbc70f700 1 -- 172.21.15.16:0/3198464413 wait complete. 2023-10-16T12:26:58.469 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-16T12:26:58.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.592+0000 7f307c322700 1 Processor -- start 2023-10-16T12:26:58.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.593+0000 7f307c322700 1 -- start start 2023-10-16T12:26:58.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.594+0000 7f307c322700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3074159c90 0x7f307415d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:58.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.594+0000 7f307c322700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f307406b1f0 con 0x7f307415a070 2023-10-16T12:26:58.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.594+0000 7f307c322700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f307406b4b0 con 0x7f30740cb190 2023-10-16T12:26:58.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.594+0000 7f307c322700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30740af340 con 0x7f3074159c90 2023-10-16T12:26:58.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.594+0000 7f30798bd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f307415a070 0x7f30740ae520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:58252/0 (socket says 172.21.15.16:58252) 2023-10-16T12:26:58.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.594+0000 7f30798bd700 1 -- 172.21.15.16:0/1906267893 learned_addr learned my addr 172.21.15.16:0/1906267893 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:26:58.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.594+0000 7f30790bc700 1 -- 172.21.15.16:0/1906267893 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1328433053 0 0) 0x7f307406b1f0 con 0x7f307415a070 2023-10-16T12:26:58.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.594+0000 7f307a0be700 1 --2- 172.21.15.16:0/1906267893 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3074159c90 0x7f307415d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:58.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.594+0000 7f30790bc700 1 -- 172.21.15.16:0/1906267893 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30600035b0 con 0x7f307415a070 2023-10-16T12:26:58.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.595+0000 7f30790bc700 1 -- 172.21.15.16:0/1906267893 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 291242894 0 0) 0x7f30600035b0 con 0x7f307415a070 2023-10-16T12:26:58.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.595+0000 7f30790bc700 1 -- 172.21.15.16:0/1906267893 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3074159c90 msgr2=0x7f307415d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:58.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.595+0000 7f30790bc700 1 --2- 172.21.15.16:0/1906267893 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3074159c90 0x7f307415d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:58.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.595+0000 7f30790bc700 1 -- 172.21.15.16:0/1906267893 >> v1:172.21.15.49:6789/0 conn(0x7f30740cb190 legacy=0x7f30740aec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:58.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.595+0000 7f30790bc700 1 -- 172.21.15.16:0/1906267893 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f307406b790 con 0x7f307415a070 2023-10-16T12:26:58.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.595+0000 7f30790bc700 1 -- 172.21.15.16:0/1906267893 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f3064001670 con 0x7f307415a070 2023-10-16T12:26:58.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.595+0000 7f30790bc700 1 -- 172.21.15.16:0/1906267893 >> v1:172.21.15.16:6789/0 conn(0x7f307415a070 legacy=0x7f30740ae520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:58.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.595+0000 7f30790bc700 1 --2- 172.21.15.16:0/1906267893 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3074159c90 0x7f3060003c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:58.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.595+0000 7f30790bc700 1 --2- 172.21.15.16:0/1906267893 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3060004180 0x7f306000a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:58.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.595+0000 7f30790bc700 1 --2- 172.21.15.16:0/1906267893 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f306000b030 0x7f306000d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:58.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.595+0000 7f30790bc700 1 -- 172.21.15.16:0/1906267893 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30740af340 con 0x7f3060004180 2023-10-16T12:26:58.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.595+0000 7f30790bc700 1 -- 172.21.15.16:0/1906267893 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f306000dda0 con 0x7f306000b030 2023-10-16T12:26:58.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.595+0000 7f30790bc700 1 -- 172.21.15.16:0/1906267893 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f306000df60 con 0x7f3074159c90 2023-10-16T12:26:58.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.595+0000 7f30798bd700 1 --2- 172.21.15.16:0/1906267893 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3060004180 0x7f306000a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:58.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.595+0000 7f307a0be700 1 --2- 172.21.15.16:0/1906267893 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3074159c90 0x7f3060003c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:58.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.595+0000 7f307a8bf700 1 --2- 172.21.15.16:0/1906267893 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f306000b030 0x7f306000d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:58.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.595+0000 7f30798bd700 1 -- 172.21.15.16:0/1906267893 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3074159c90 msgr2=0x7f3060003c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:58.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.595+0000 7f30798bd700 1 --2- 172.21.15.16:0/1906267893 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3074159c90 0x7f3060003c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:58.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.595+0000 7f30798bd700 1 -- 172.21.15.16:0/1906267893 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f306000b030 msgr2=0x7f306000d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:58.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.595+0000 7f30798bd700 1 --2- 172.21.15.16:0/1906267893 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f306000b030 0x7f306000d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:58.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.595+0000 7f30798bd700 1 -- 172.21.15.16:0/1906267893 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30600035b0 con 0x7f3060004180 2023-10-16T12:26:58.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.595+0000 7f307a8bf700 1 --2- 172.21.15.16:0/1906267893 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f306000b030 0x7f306000d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:26:58.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.595+0000 7f30798bd700 1 --2- 172.21.15.16:0/1906267893 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3060004180 0x7f306000a830 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f306400c210 tx=0x7f3060003860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f9dbde4fbc6cdee server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:58.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.595+0000 7f30790bc700 1 -- 172.21.15.16:0/1906267893 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3064007c10 con 0x7f3060004180 2023-10-16T12:26:58.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.595+0000 7f30790bc700 1 -- 172.21.15.16:0/1906267893 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f306400cdd0 con 0x7f3060004180 2023-10-16T12:26:58.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.595+0000 7f30790bc700 1 -- 172.21.15.16:0/1906267893 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3064005ba0 con 0x7f3060004180 2023-10-16T12:26:58.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.596+0000 7f307c322700 1 -- 172.21.15.16:0/1906267893 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3060004180 msgr2=0x7f306000a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:58.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.596+0000 7f307c322700 1 --2- 172.21.15.16:0/1906267893 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3060004180 0x7f306000a830 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f306400c210 tx=0x7f3060003860 comp rx=0 tx=0).stop 2023-10-16T12:26:58.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.596+0000 7f307a8bf700 1 -- 172.21.15.16:0/1906267893 reap_dead start 2023-10-16T12:26:58.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.596+0000 7f307c322700 1 -- 172.21.15.16:0/1906267893 shutdown_connections 2023-10-16T12:26:58.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.596+0000 7f307c322700 1 -- 172.21.15.16:0/1906267893 >> 172.21.15.16:0/1906267893 conn(0x7f30740cc5b0 msgr2=0x7f30740cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:58.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.596+0000 7f307c322700 1 -- 172.21.15.16:0/1906267893 shutdown_connections 2023-10-16T12:26:58.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.596+0000 7f307c322700 1 -- 172.21.15.16:0/1906267893 wait complete. 2023-10-16T12:26:58.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.597+0000 7f307c322700 1 Processor -- start 2023-10-16T12:26:58.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.597+0000 7f307c322700 1 -- start start 2023-10-16T12:26:58.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.598+0000 7f307c322700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3060004180 0x7f307414d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:58.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.598+0000 7f307c322700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3074155ff0 0x7f3074156470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:58.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.598+0000 7f307c322700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f30740c3800 0x7f30740c3c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:58.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.598+0000 7f307c322700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30740b36f0 con 0x7f3074155ff0 2023-10-16T12:26:58.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.598+0000 7f307c322700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f307414de70 con 0x7f30740c3800 2023-10-16T12:26:58.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.598+0000 7f307c322700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30740b2790 con 0x7f3060004180 2023-10-16T12:26:58.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.598+0000 7f30798bd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3074155ff0 0x7f3074156470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:58.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.598+0000 7f30798bd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3074155ff0 0x7f3074156470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40758/0 (socket says 172.21.15.16:40758) 2023-10-16T12:26:58.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.598+0000 7f30798bd700 1 -- 172.21.15.16:0/4132314542 learned_addr learned my addr 172.21.15.16:0/4132314542 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:58.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.598+0000 7f307a0be700 1 --2- 172.21.15.16:0/4132314542 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3060004180 0x7f307414d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:58.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.598+0000 7f307a8bf700 1 --2- 172.21.15.16:0/4132314542 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f30740c3800 0x7f30740c3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:58.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.598+0000 7f30798bd700 1 -- 172.21.15.16:0/4132314542 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3060004180 msgr2=0x7f307414d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:58.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.598+0000 7f30798bd700 1 --2- 172.21.15.16:0/4132314542 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3060004180 0x7f307414d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:58.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.598+0000 7f30798bd700 1 -- 172.21.15.16:0/4132314542 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f30740c3800 msgr2=0x7f30740c3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:58.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.598+0000 7f30798bd700 1 --2- 172.21.15.16:0/4132314542 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f30740c3800 0x7f30740c3c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:58.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.598+0000 7f30798bd700 1 -- 172.21.15.16:0/4132314542 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30740b2a70 con 0x7f3074155ff0 2023-10-16T12:26:58.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.598+0000 7f307a8bf700 1 --2- 172.21.15.16:0/4132314542 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f30740c3800 0x7f30740c3c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:26:58.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.599+0000 7f30798bd700 1 --2- 172.21.15.16:0/4132314542 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3074155ff0 0x7f3074156470 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f306400c800 tx=0x7f3064001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:58.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.599+0000 7f306affd700 1 -- 172.21.15.16:0/4132314542 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3064001630 con 0x7f3074155ff0 2023-10-16T12:26:58.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.599+0000 7f306affd700 1 -- 172.21.15.16:0/4132314542 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30640070d0 con 0x7f3074155ff0 2023-10-16T12:26:58.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.599+0000 7f307c322700 1 -- 172.21.15.16:0/4132314542 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30740b2d50 con 0x7f3074155ff0 2023-10-16T12:26:58.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.599+0000 7f306affd700 1 -- 172.21.15.16:0/4132314542 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3064007480 con 0x7f3074155ff0 2023-10-16T12:26:58.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.599+0000 7f307c322700 1 -- 172.21.15.16:0/4132314542 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30740b3310 con 0x7f3074155ff0 2023-10-16T12:26:58.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.600+0000 7f307c322700 1 -- 172.21.15.16:0/4132314542 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30740b3960 con 0x7f3074155ff0 2023-10-16T12:26:58.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.600+0000 7f306affd700 1 -- 172.21.15.16:0/4132314542 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 41) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3064019030 con 0x7f3074155ff0 2023-10-16T12:26:58.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.601+0000 7f306affd700 1 --2- 172.21.15.16:0/4132314542 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f305806b020 0x7f305806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:58.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.601+0000 7f306affd700 1 -- 172.21.15.16:0/4132314542 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f306408aa40 con 0x7f3074155ff0 2023-10-16T12:26:58.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.601+0000 7f307a0be700 1 --2- 172.21.15.16:0/4132314542 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f305806b020 0x7f305806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:58.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.601+0000 7f307a0be700 1 --2- 172.21.15.16:0/4132314542 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f305806b020 0x7f305806d4e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3070009c60 tx=0x7f3070009210 comp rx=0 tx=0).ready entity=mgr.6020 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:58.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.605+0000 7f306affd700 1 -- 172.21.15.16:0/4132314542 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f306408ce60 con 0x7f3074155ff0 2023-10-16T12:26:58.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.814+0000 7f307c322700 1 -- 172.21.15.16:0/4132314542 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f307404ebb0 con 0x7f3074155ff0 2023-10-16T12:26:58.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.814+0000 7f306affd700 1 -- 172.21.15.16:0/4132314542 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v95) v1 ==== 84+0+65 (secure 0 0 0) 0x7f3064017020 con 0x7f3074155ff0 2023-10-16T12:26:58.816 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:26:58.817 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2023-10-16T12:26:58.817 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:26:58.817 INFO:teuthology.orchestra.run.smithi016.stdout: "poolnum": 1, 2023-10-16T12:26:58.817 INFO:teuthology.orchestra.run.smithi016.stdout: "poolname": ".mgr" 2023-10-16T12:26:58.817 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:26:58.818 INFO:teuthology.orchestra.run.smithi016.stdout:] 2023-10-16T12:26:58.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.816+0000 7f307c322700 1 -- 172.21.15.16:0/4132314542 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f305806b020 msgr2=0x7f305806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:58.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.816+0000 7f307c322700 1 --2- 172.21.15.16:0/4132314542 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f305806b020 0x7f305806d4e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3070009c60 tx=0x7f3070009210 comp rx=0 tx=0).stop 2023-10-16T12:26:58.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.816+0000 7f307c322700 1 -- 172.21.15.16:0/4132314542 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3074155ff0 msgr2=0x7f3074156470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:58.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.816+0000 7f307c322700 1 --2- 172.21.15.16:0/4132314542 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3074155ff0 0x7f3074156470 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f306400c800 tx=0x7f3064001a00 comp rx=0 tx=0).stop 2023-10-16T12:26:58.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.816+0000 7f307c322700 1 -- 172.21.15.16:0/4132314542 shutdown_connections 2023-10-16T12:26:58.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.816+0000 7f307c322700 1 --2- 172.21.15.16:0/4132314542 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f305806b020 0x7f305806d4e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:58.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.816+0000 7f307c322700 1 --2- 172.21.15.16:0/4132314542 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f30740c3800 0x7f30740c3c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:58.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.816+0000 7f307c322700 1 --2- 172.21.15.16:0/4132314542 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3060004180 0x7f307414d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:58.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.816+0000 7f307c322700 1 --2- 172.21.15.16:0/4132314542 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3074155ff0 0x7f3074156470 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:58.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.816+0000 7f307c322700 1 -- 172.21.15.16:0/4132314542 >> 172.21.15.16:0/4132314542 conn(0x7f30740cc5b0 msgr2=0x7f3074158f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:58.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.816+0000 7f307c322700 1 -- 172.21.15.16:0/4132314542 shutdown_connections 2023-10-16T12:26:58.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.816+0000 7f307c322700 1 -- 172.21.15.16:0/4132314542 wait complete. 2023-10-16T12:26:58.830 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-16T12:26:58.831 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-16T12:26:58.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.958+0000 7f56c18d3700 1 Processor -- start 2023-10-16T12:26:58.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.958+0000 7f56c18d3700 1 -- start start 2023-10-16T12:26:58.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.959+0000 7f56c18d3700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f56bc15d7a0 0x7f56bc0b7240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:58.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.959+0000 7f56c18d3700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56bc0b91b0 con 0x7f56bc0b7780 2023-10-16T12:26:58.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.959+0000 7f56c18d3700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56bc0b7d00 con 0x7f56bc0ce970 2023-10-16T12:26:58.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.959+0000 7f56c18d3700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56bc0b8980 con 0x7f56bc15d7a0 2023-10-16T12:26:58.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.959+0000 7f56ba7fc700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f56bc0b7780 0x7f56bc0ce260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:58270/0 (socket says 172.21.15.16:58270) 2023-10-16T12:26:58.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.959+0000 7f56ba7fc700 1 -- 172.21.15.16:0/92295303 learned_addr learned my addr 172.21.15.16:0/92295303 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:26:58.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.959+0000 7f56baffd700 1 --2- 172.21.15.16:0/92295303 >> v2:172.21.15.49:3300/0 conn(0x7f56bc15d7a0 0x7f56bc0b7240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:58.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.959+0000 7f56b9ffb700 1 -- 172.21.15.16:0/92295303 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1167329980 0 0) 0x7f56bc0b91b0 con 0x7f56bc0b7780 2023-10-16T12:26:58.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.959+0000 7f56b9ffb700 1 -- 172.21.15.16:0/92295303 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56a40035b0 con 0x7f56bc0b7780 2023-10-16T12:26:58.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.960+0000 7f56baffd700 1 -- 172.21.15.16:0/92295303 >> v1:172.21.15.49:6789/0 conn(0x7f56bc0ce970 legacy=0x7f56bc0d1740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:58.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.960+0000 7f56baffd700 1 -- 172.21.15.16:0/92295303 >> v1:172.21.15.16:6789/0 conn(0x7f56bc0b7780 legacy=0x7f56bc0ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:58.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.960+0000 7f56baffd700 1 -- 172.21.15.16:0/92295303 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56bc0b7fe0 con 0x7f56bc15d7a0 2023-10-16T12:26:58.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.960+0000 7f56baffd700 1 --2- 172.21.15.16:0/92295303 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f56bc15d7a0 0x7f56bc0b7240 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f56b0009840 tx=0x7f56b0009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d16a2d03ddc83bb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:58.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.960+0000 7f56b9ffb700 1 -- 172.21.15.16:0/92295303 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b0011830 con 0x7f56bc15d7a0 2023-10-16T12:26:58.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.960+0000 7f56b9ffb700 1 -- 172.21.15.16:0/92295303 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56b0011990 con 0x7f56bc15d7a0 2023-10-16T12:26:58.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.960+0000 7f56b9ffb700 1 -- 172.21.15.16:0/92295303 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b0011d20 con 0x7f56bc15d7a0 2023-10-16T12:26:58.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.960+0000 7f56c18d3700 1 -- 172.21.15.16:0/92295303 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f56bc15d7a0 msgr2=0x7f56bc0b7240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:58.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.960+0000 7f56c18d3700 1 --2- 172.21.15.16:0/92295303 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f56bc15d7a0 0x7f56bc0b7240 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f56b0009840 tx=0x7f56b0009b50 comp rx=0 tx=0).stop 2023-10-16T12:26:58.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.961+0000 7f56c18d3700 1 -- 172.21.15.16:0/92295303 shutdown_connections 2023-10-16T12:26:58.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.961+0000 7f56c18d3700 1 --2- 172.21.15.16:0/92295303 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f56bc15d7a0 0x7f56bc0b7240 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:58.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.961+0000 7f56c18d3700 1 -- 172.21.15.16:0/92295303 >> 172.21.15.16:0/92295303 conn(0x7f56bc072330 msgr2=0x7f56bc072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:58.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.961+0000 7f56c18d3700 1 -- 172.21.15.16:0/92295303 shutdown_connections 2023-10-16T12:26:58.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.961+0000 7f56c18d3700 1 -- 172.21.15.16:0/92295303 wait complete. 2023-10-16T12:26:58.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.962+0000 7f56c18d3700 1 Processor -- start 2023-10-16T12:26:58.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.962+0000 7f56c18d3700 1 -- start start 2023-10-16T12:26:58.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.963+0000 7f56c18d3700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f56bc0b7780 0x7f56bc157380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:58.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.963+0000 7f56c18d3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56bc0ce970 0x7f56bc1578c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:58.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.963+0000 7f56c18d3700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f56bc15d7a0 0x7f56bc1594c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:58.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.963+0000 7f56c18d3700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56bc0b3cc0 con 0x7f56bc0ce970 2023-10-16T12:26:58.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.963+0000 7f56c18d3700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56bc0b4eb0 con 0x7f56bc0b7780 2023-10-16T12:26:58.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.963+0000 7f56c18d3700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56bc0b5190 con 0x7f56bc15d7a0 2023-10-16T12:26:58.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.963+0000 7f56ba7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56bc0ce970 0x7f56bc1578c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:58.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.963+0000 7f56ba7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56bc0ce970 0x7f56bc1578c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40772/0 (socket says 172.21.15.16:40772) 2023-10-16T12:26:58.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.964+0000 7f56ba7fc700 1 -- 172.21.15.16:0/1369210982 learned_addr learned my addr 172.21.15.16:0/1369210982 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:58.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.964+0000 7f56bb7fe700 1 --2- 172.21.15.16:0/1369210982 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f56bc15d7a0 0x7f56bc1594c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:58.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.964+0000 7f56baffd700 1 --2- 172.21.15.16:0/1369210982 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f56bc0b7780 0x7f56bc157380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:58.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.964+0000 7f56ba7fc700 1 -- 172.21.15.16:0/1369210982 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f56bc15d7a0 msgr2=0x7f56bc1594c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:58.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.964+0000 7f56ba7fc700 1 --2- 172.21.15.16:0/1369210982 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f56bc15d7a0 0x7f56bc1594c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:58.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.964+0000 7f56ba7fc700 1 -- 172.21.15.16:0/1369210982 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f56bc0b7780 msgr2=0x7f56bc157380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:58.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.964+0000 7f56ba7fc700 1 --2- 172.21.15.16:0/1369210982 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f56bc0b7780 0x7f56bc157380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:58.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.964+0000 7f56ba7fc700 1 -- 172.21.15.16:0/1369210982 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56b00094a0 con 0x7f56bc0ce970 2023-10-16T12:26:58.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.964+0000 7f56ba7fc700 1 --2- 172.21.15.16:0/1369210982 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56bc0ce970 0x7f56bc1578c0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f56ac00c700 tx=0x7f56ac00ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:58.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.964+0000 7f56a3fff700 1 -- 172.21.15.16:0/1369210982 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56ac005d70 con 0x7f56bc0ce970 2023-10-16T12:26:58.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.964+0000 7f56a3fff700 1 -- 172.21.15.16:0/1369210982 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56ac0070d0 con 0x7f56bc0ce970 2023-10-16T12:26:58.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.964+0000 7f56c18d3700 1 -- 172.21.15.16:0/1369210982 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56bc0b5470 con 0x7f56bc0ce970 2023-10-16T12:26:58.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.964+0000 7f56a3fff700 1 -- 172.21.15.16:0/1369210982 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56ac007460 con 0x7f56bc0ce970 2023-10-16T12:26:58.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.964+0000 7f56c18d3700 1 -- 172.21.15.16:0/1369210982 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56bc0c95f0 con 0x7f56bc0ce970 2023-10-16T12:26:58.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.965+0000 7f56c18d3700 1 -- 172.21.15.16:0/1369210982 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56bc0b5690 con 0x7f56bc0ce970 2023-10-16T12:26:58.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.966+0000 7f56a3fff700 1 -- 172.21.15.16:0/1369210982 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 41) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f56ac0075c0 con 0x7f56bc0ce970 2023-10-16T12:26:58.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.966+0000 7f56a3fff700 1 --2- 172.21.15.16:0/1369210982 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f56a806b020 0x7f56a806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:58.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.966+0000 7f56a3fff700 1 -- 172.21.15.16:0/1369210982 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f56ac0889a0 con 0x7f56bc0ce970 2023-10-16T12:26:58.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.966+0000 7f56baffd700 1 --2- 172.21.15.16:0/1369210982 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f56a806b020 0x7f56a806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:58.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.967+0000 7f56baffd700 1 --2- 172.21.15.16:0/1369210982 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f56a806b020 0x7f56a806d4e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f56b00090b0 tx=0x7f56b001c040 comp rx=0 tx=0).ready entity=mgr.6020 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:58.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:58.971+0000 7f56a3fff700 1 -- 172.21.15.16:0/1369210982 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f56ac053b30 con 0x7f56bc0ce970 2023-10-16T12:26:59.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.174+0000 7f56c18d3700 1 -- 172.21.15.16:0/1369210982 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f56bc04ec20 con 0x7f56bc0ce970 2023-10-16T12:26:59.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.386+0000 7f56a3fff700 1 -- 172.21.15.16:0/1369210982 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v96) v1 ==== 138+0+0 (secure 0 0 0) 0x7f56ac017020 con 0x7f56bc0ce970 2023-10-16T12:26:59.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.389+0000 7f56c18d3700 1 -- 172.21.15.16:0/1369210982 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f56a806b020 msgr2=0x7f56a806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:59.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.389+0000 7f56c18d3700 1 --2- 172.21.15.16:0/1369210982 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f56a806b020 0x7f56a806d4e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f56b00090b0 tx=0x7f56b001c040 comp rx=0 tx=0).stop 2023-10-16T12:26:59.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.389+0000 7f56c18d3700 1 -- 172.21.15.16:0/1369210982 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56bc0ce970 msgr2=0x7f56bc1578c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:59.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.389+0000 7f56c18d3700 1 --2- 172.21.15.16:0/1369210982 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56bc0ce970 0x7f56bc1578c0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f56ac00c700 tx=0x7f56ac00ca10 comp rx=0 tx=0).stop 2023-10-16T12:26:59.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.390+0000 7f56c18d3700 1 -- 172.21.15.16:0/1369210982 shutdown_connections 2023-10-16T12:26:59.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.390+0000 7f56c18d3700 1 --2- 172.21.15.16:0/1369210982 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f56a806b020 0x7f56a806d4e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:59.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.390+0000 7f56c18d3700 1 --2- 172.21.15.16:0/1369210982 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f56bc15d7a0 0x7f56bc1594c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:59.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.390+0000 7f56c18d3700 1 --2- 172.21.15.16:0/1369210982 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f56bc0b7780 0x7f56bc157380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:59.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.390+0000 7f56c18d3700 1 --2- 172.21.15.16:0/1369210982 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56bc0ce970 0x7f56bc1578c0 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:59.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.390+0000 7f56c18d3700 1 -- 172.21.15.16:0/1369210982 >> 172.21.15.16:0/1369210982 conn(0x7f56bc072330 msgr2=0x7f56bc0b21f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:59.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.391+0000 7f56c18d3700 1 -- 172.21.15.16:0/1369210982 shutdown_connections 2023-10-16T12:26:59.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.391+0000 7f56c18d3700 1 -- 172.21.15.16:0/1369210982 wait complete. 2023-10-16T12:26:59.395 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' created 2023-10-16T12:26:59.408 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-16T12:26:59.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.538+0000 7f2638539700 1 Processor -- start 2023-10-16T12:26:59.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.538+0000 7f2638539700 1 -- start start 2023-10-16T12:26:59.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.539+0000 7f2638539700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f263015ec80 0x7f26300cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:59.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.539+0000 7f2638539700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26300b2440 con 0x7f26300d2110 2023-10-16T12:26:59.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.539+0000 7f2638539700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26300d00e0 con 0x7f263015f060 2023-10-16T12:26:59.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.539+0000 7f2638539700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26300b2c50 con 0x7f263015ec80 2023-10-16T12:26:59.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.540+0000 7f2636ad6700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f26300d2110 0x7f26300b4b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:58288/0 (socket says 172.21.15.16:58288) 2023-10-16T12:26:59.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.540+0000 7f2636ad6700 1 -- 172.21.15.16:0/334801478 learned_addr learned my addr 172.21.15.16:0/334801478 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:26:59.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.540+0000 7f26352d3700 1 -- 172.21.15.16:0/334801478 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 329264155 0 0) 0x7f26300b2440 con 0x7f26300d2110 2023-10-16T12:26:59.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.540+0000 7f26362d5700 1 --2- 172.21.15.16:0/334801478 >> v2:172.21.15.49:3300/0 conn(0x7f263015ec80 0x7f26300cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:59.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.540+0000 7f26352d3700 1 -- 172.21.15.16:0/334801478 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f261c0035b0 con 0x7f26300d2110 2023-10-16T12:26:59.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.540+0000 7f26352d3700 1 -- 172.21.15.16:0/334801478 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2938565530 0 0) 0x7f261c0035b0 con 0x7f26300d2110 2023-10-16T12:26:59.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.540+0000 7f26352d3700 1 -- 172.21.15.16:0/334801478 >> v2:172.21.15.49:3300/0 conn(0x7f263015ec80 msgr2=0x7f26300cdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:59.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.540+0000 7f26352d3700 1 --2- 172.21.15.16:0/334801478 >> v2:172.21.15.49:3300/0 conn(0x7f263015ec80 0x7f26300cdec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:59.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.540+0000 7f26352d3700 1 -- 172.21.15.16:0/334801478 >> v1:172.21.15.49:6789/0 conn(0x7f263015f060 legacy=0x7f26300b4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:59.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.540+0000 7f26352d3700 1 -- 172.21.15.16:0/334801478 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26300cef60 con 0x7f26300d2110 2023-10-16T12:26:59.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.540+0000 7f26352d3700 1 -- 172.21.15.16:0/334801478 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f26280019f0 con 0x7f26300d2110 2023-10-16T12:26:59.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.540+0000 7f26352d3700 1 -- 172.21.15.16:0/334801478 >> v1:172.21.15.16:6789/0 conn(0x7f26300d2110 legacy=0x7f26300b4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:59.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.540+0000 7f26352d3700 1 --2- 172.21.15.16:0/334801478 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f261c003a00 0x7f261c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:59.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.540+0000 7f26352d3700 1 --2- 172.21.15.16:0/334801478 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f261c004850 0x7f261c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:59.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.540+0000 7f26352d3700 1 --2- 172.21.15.16:0/334801478 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f261c005210 0x7f261c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:59.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.540+0000 7f26352d3700 1 -- 172.21.15.16:0/334801478 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26300b2c50 con 0x7f261c003a00 2023-10-16T12:26:59.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.540+0000 7f26352d3700 1 -- 172.21.15.16:0/334801478 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f261c00c0b0 con 0x7f261c004850 2023-10-16T12:26:59.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.540+0000 7f26352d3700 1 -- 172.21.15.16:0/334801478 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f261c00c270 con 0x7f261c005210 2023-10-16T12:26:59.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.540+0000 7f26362d5700 1 --2- 172.21.15.16:0/334801478 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f261c003a00 0x7f261c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:59.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.540+0000 7f2635ad4700 1 --2- 172.21.15.16:0/334801478 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f261c004850 0x7f261c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:59.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.541+0000 7f2636ad6700 1 --2- 172.21.15.16:0/334801478 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f261c005210 0x7f261c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:59.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.541+0000 7f26362d5700 1 -- 172.21.15.16:0/334801478 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f261c005210 msgr2=0x7f261c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:59.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.541+0000 7f26362d5700 1 --2- 172.21.15.16:0/334801478 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f261c005210 0x7f261c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:59.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.541+0000 7f26362d5700 1 -- 172.21.15.16:0/334801478 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f261c004850 msgr2=0x7f261c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:26:59.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.541+0000 7f26362d5700 1 --2- 172.21.15.16:0/334801478 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f261c004850 0x7f261c004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:59.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.541+0000 7f26362d5700 1 -- 172.21.15.16:0/334801478 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f261c0035b0 con 0x7f261c003a00 2023-10-16T12:26:59.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.541+0000 7f2636ad6700 1 -- 172.21.15.16:0/334801478 reap_dead start 2023-10-16T12:26:59.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.541+0000 7f26362d5700 1 --2- 172.21.15.16:0/334801478 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f261c003a00 0x7f261c003e20 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f2620009b60 tx=0x7f261c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11c7758634cbde9f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:59.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.541+0000 7f26352d3700 1 -- 172.21.15.16:0/334801478 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f262000e640 con 0x7f261c003a00 2023-10-16T12:26:59.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.541+0000 7f26352d3700 1 -- 172.21.15.16:0/334801478 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f262000e7a0 con 0x7f261c003a00 2023-10-16T12:26:59.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.541+0000 7f26352d3700 1 -- 172.21.15.16:0/334801478 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f262000eb10 con 0x7f261c003a00 2023-10-16T12:26:59.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.542+0000 7f2638539700 1 -- 172.21.15.16:0/334801478 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f261c003a00 msgr2=0x7f261c003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:59.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.542+0000 7f2638539700 1 --2- 172.21.15.16:0/334801478 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f261c003a00 0x7f261c003e20 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f2620009b60 tx=0x7f261c005880 comp rx=0 tx=0).stop 2023-10-16T12:26:59.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.542+0000 7f2638539700 1 -- 172.21.15.16:0/334801478 shutdown_connections 2023-10-16T12:26:59.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.542+0000 7f2638539700 1 --2- 172.21.15.16:0/334801478 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f261c003a00 0x7f261c003e20 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:59.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.542+0000 7f2638539700 1 -- 172.21.15.16:0/334801478 >> 172.21.15.16:0/334801478 conn(0x7f26300d15f0 msgr2=0x7f26300d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:26:59.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.542+0000 7f2638539700 1 -- 172.21.15.16:0/334801478 shutdown_connections 2023-10-16T12:26:59.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.542+0000 7f2638539700 1 -- 172.21.15.16:0/334801478 wait complete. 2023-10-16T12:26:59.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.543+0000 7f2638539700 1 Processor -- start 2023-10-16T12:26:59.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.543+0000 7f2638539700 1 -- start start 2023-10-16T12:26:59.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.544+0000 7f2638539700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f261c003a00 0x7f26300c8c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:59.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.544+0000 7f2638539700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2630151f20 0x7f26301e5bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:59.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.544+0000 7f2638539700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2630151380 0x7f26300cd320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:59.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.544+0000 7f2638539700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2630155a50 con 0x7f2630151f20 2023-10-16T12:26:59.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.544+0000 7f2638539700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2630151800 con 0x7f261c003a00 2023-10-16T12:26:59.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.544+0000 7f2638539700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2630155ca0 con 0x7f2630151380 2023-10-16T12:26:59.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.544+0000 7f2635ad4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2630151f20 0x7f26301e5bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:59.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.544+0000 7f2635ad4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2630151f20 0x7f26301e5bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40792/0 (socket says 172.21.15.16:40792) 2023-10-16T12:26:59.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.544+0000 7f2635ad4700 1 -- 172.21.15.16:0/1829086911 learned_addr learned my addr 172.21.15.16:0/1829086911 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:26:59.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.544+0000 7f26362d5700 1 --2- 172.21.15.16:0/1829086911 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f261c003a00 0x7f26300c8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:59.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.544+0000 7f2636ad6700 1 --2- 172.21.15.16:0/1829086911 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2630151380 0x7f26300cd320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:59.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.544+0000 7f2635ad4700 1 -- 172.21.15.16:0/1829086911 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2630151380 msgr2=0x7f26300cd320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:59.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.544+0000 7f2635ad4700 1 --2- 172.21.15.16:0/1829086911 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2630151380 0x7f26300cd320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:59.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.544+0000 7f2635ad4700 1 -- 172.21.15.16:0/1829086911 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f261c003a00 msgr2=0x7f26300c8c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:26:59.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.544+0000 7f2635ad4700 1 --2- 172.21.15.16:0/1829086911 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f261c003a00 0x7f26300c8c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:26:59.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.544+0000 7f2635ad4700 1 -- 172.21.15.16:0/1829086911 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2630155f80 con 0x7f2630151f20 2023-10-16T12:26:59.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.544+0000 7f2635ad4700 1 --2- 172.21.15.16:0/1829086911 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2630151f20 0x7f26301e5bc0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f262c00f7a0 tx=0x7f262c00fab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:59.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.545+0000 7f2626ffd700 1 -- 172.21.15.16:0/1829086911 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f262c005320 con 0x7f2630151f20 2023-10-16T12:26:59.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.545+0000 7f2626ffd700 1 -- 172.21.15.16:0/1829086911 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f262c005480 con 0x7f2630151f20 2023-10-16T12:26:59.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.545+0000 7f2638539700 1 -- 172.21.15.16:0/1829086911 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2630156260 con 0x7f2630151f20 2023-10-16T12:26:59.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.545+0000 7f2626ffd700 1 -- 172.21.15.16:0/1829086911 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f262c006a80 con 0x7f2630151f20 2023-10-16T12:26:59.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.545+0000 7f2638539700 1 -- 172.21.15.16:0/1829086911 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2630156820 con 0x7f2630151f20 2023-10-16T12:26:59.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.545+0000 7f2638539700 1 -- 172.21.15.16:0/1829086911 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26301569e0 con 0x7f2630151f20 2023-10-16T12:26:59.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.547+0000 7f2626ffd700 1 -- 172.21.15.16:0/1829086911 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 41) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f262c006be0 con 0x7f2630151f20 2023-10-16T12:26:59.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.547+0000 7f2626ffd700 1 --2- 172.21.15.16:0/1829086911 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f261406b020 0x7f261406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:26:59.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.547+0000 7f2626ffd700 1 -- 172.21.15.16:0/1829086911 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5575+0+0 (secure 0 0 0) 0x7f262c0890a0 con 0x7f2630151f20 2023-10-16T12:26:59.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.547+0000 7f26362d5700 1 --2- 172.21.15.16:0/1829086911 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f261406b020 0x7f261406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:26:59.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.548+0000 7f26362d5700 1 --2- 172.21.15.16:0/1829086911 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f261406b020 0x7f261406d4e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f26200090b0 tx=0x7f262001e040 comp rx=0 tx=0).ready entity=mgr.6020 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:26:59.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.551+0000 7f2626ffd700 1 -- 172.21.15.16:0/1829086911 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f262c0541a0 con 0x7f2630151f20 2023-10-16T12:26:59.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:26:59.752+0000 7f2638539700 1 -- 172.21.15.16:0/1829086911 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f2630156b90 con 0x7f2630151f20 2023-10-16T12:27:00.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.392+0000 7f2626ffd700 1 -- 172.21.15.16:0/1829086911 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v97) v1 ==== 170+0+0 (secure 0 0 0) 0x7f262c0577c0 con 0x7f2630151f20 2023-10-16T12:27:00.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.397+0000 7f2638539700 1 -- 172.21.15.16:0/1829086911 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f261406b020 msgr2=0x7f261406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:00.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.397+0000 7f2638539700 1 --2- 172.21.15.16:0/1829086911 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f261406b020 0x7f261406d4e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f26200090b0 tx=0x7f262001e040 comp rx=0 tx=0).stop 2023-10-16T12:27:00.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.397+0000 7f2638539700 1 -- 172.21.15.16:0/1829086911 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2630151f20 msgr2=0x7f26301e5bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:00.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.397+0000 7f2638539700 1 --2- 172.21.15.16:0/1829086911 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2630151f20 0x7f26301e5bc0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f262c00f7a0 tx=0x7f262c00fab0 comp rx=0 tx=0).stop 2023-10-16T12:27:00.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.398+0000 7f2638539700 1 -- 172.21.15.16:0/1829086911 shutdown_connections 2023-10-16T12:27:00.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.398+0000 7f2638539700 1 --2- 172.21.15.16:0/1829086911 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f261406b020 0x7f261406d4e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:00.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.398+0000 7f2638539700 1 --2- 172.21.15.16:0/1829086911 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2630151380 0x7f26300cd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:00.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.398+0000 7f2638539700 1 --2- 172.21.15.16:0/1829086911 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f261c003a00 0x7f26300c8c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:00.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.398+0000 7f2638539700 1 --2- 172.21.15.16:0/1829086911 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2630151f20 0x7f26301e5bc0 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:00.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.398+0000 7f2638539700 1 -- 172.21.15.16:0/1829086911 >> 172.21.15.16:0/1829086911 conn(0x7f26300d15f0 msgr2=0x7f263015d320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:00.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.399+0000 7f2638539700 1 -- 172.21.15.16:0/1829086911 shutdown_connections 2023-10-16T12:27:00.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.399+0000 7f2638539700 1 -- 172.21.15.16:0/1829086911 wait complete. 2023-10-16T12:27:00.413 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' created 2023-10-16T12:27:00.429 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-16T12:27:00.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.561+0000 7fed212e1700 1 Processor -- start 2023-10-16T12:27:00.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.561+0000 7fed212e1700 1 -- start start 2023-10-16T12:27:00.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.561+0000 7fed212e1700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fed1c15d8d0 0x7fed1c0b2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:00.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.561+0000 7fed212e1700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed1c0b2f30 con 0x7fed1c0ce350 2023-10-16T12:27:00.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.561+0000 7fed212e1700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed1c0d1c40 con 0x7fed1c0b8fa0 2023-10-16T12:27:00.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.561+0000 7fed212e1700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed1c0d0bb0 con 0x7fed1c15d8d0 2023-10-16T12:27:00.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.562+0000 7fed1b7fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fed1c0ce350 0x7fed1c0d29c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:58308/0 (socket says 172.21.15.16:58308) 2023-10-16T12:27:00.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.562+0000 7fed1b7fe700 1 -- 172.21.15.16:0/2585526908 learned_addr learned my addr 172.21.15.16:0/2585526908 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:27:00.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.562+0000 7fed1affd700 1 --2- 172.21.15.16:0/2585526908 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fed1c15d8d0 0x7fed1c0b2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:00.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.562+0000 7fed19ffb700 1 -- 172.21.15.16:0/2585526908 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1426706499 0 0) 0x7fed1c0b2f30 con 0x7fed1c0ce350 2023-10-16T12:27:00.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.562+0000 7fed19ffb700 1 -- 172.21.15.16:0/2585526908 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed080035b0 con 0x7fed1c0ce350 2023-10-16T12:27:00.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.562+0000 7fed19ffb700 1 -- 172.21.15.16:0/2585526908 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 414903530 0 0) 0x7fed080035b0 con 0x7fed1c0ce350 2023-10-16T12:27:00.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.562+0000 7fed19ffb700 1 -- 172.21.15.16:0/2585526908 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fed1c15d8d0 msgr2=0x7fed1c0b2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:00.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.562+0000 7fed19ffb700 1 --2- 172.21.15.16:0/2585526908 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fed1c15d8d0 0x7fed1c0b2830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:00.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.562+0000 7fed19ffb700 1 -- 172.21.15.16:0/2585526908 >> v1:172.21.15.49:6789/0 conn(0x7fed1c0b8fa0 legacy=0x7fed1c0b6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:00.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.562+0000 7fed19ffb700 1 -- 172.21.15.16:0/2585526908 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed1c0d1ee0 con 0x7fed1c0ce350 2023-10-16T12:27:00.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.562+0000 7fed19ffb700 1 -- 172.21.15.16:0/2585526908 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7fed140019f0 con 0x7fed1c0ce350 2023-10-16T12:27:00.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.562+0000 7fed19ffb700 1 -- 172.21.15.16:0/2585526908 >> v1:172.21.15.16:6789/0 conn(0x7fed1c0ce350 legacy=0x7fed1c0d29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:00.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.562+0000 7fed19ffb700 1 --2- 172.21.15.16:0/2585526908 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed08003a30 0x7fed08003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:00.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.562+0000 7fed19ffb700 1 --2- 172.21.15.16:0/2585526908 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fed1c15d8d0 0x7fed08004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:00.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.562+0000 7fed19ffb700 1 --2- 172.21.15.16:0/2585526908 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fed08004d90 0x7fed08005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:00.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.562+0000 7fed19ffb700 1 -- 172.21.15.16:0/2585526908 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed1c0d0bb0 con 0x7fed08003a30 2023-10-16T12:27:00.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.562+0000 7fed19ffb700 1 -- 172.21.15.16:0/2585526908 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed0800bba0 con 0x7fed08004d90 2023-10-16T12:27:00.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.562+0000 7fed19ffb700 1 -- 172.21.15.16:0/2585526908 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed0800bd30 con 0x7fed1c15d8d0 2023-10-16T12:27:00.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.562+0000 7fed1affd700 1 --2- 172.21.15.16:0/2585526908 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed08003a30 0x7fed08003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:00.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.562+0000 7fed1a7fc700 1 --2- 172.21.15.16:0/2585526908 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fed1c15d8d0 0x7fed08004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:00.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.563+0000 7fed1b7fe700 1 --2- 172.21.15.16:0/2585526908 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fed08004d90 0x7fed08005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:00.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.563+0000 7fed1affd700 1 -- 172.21.15.16:0/2585526908 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fed1c15d8d0 msgr2=0x7fed08004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:00.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.563+0000 7fed1affd700 1 --2- 172.21.15.16:0/2585526908 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fed1c15d8d0 0x7fed08004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:00.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.563+0000 7fed1affd700 1 -- 172.21.15.16:0/2585526908 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fed08004d90 msgr2=0x7fed08005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:00.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.563+0000 7fed1affd700 1 --2- 172.21.15.16:0/2585526908 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fed08004d90 0x7fed08005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:00.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.563+0000 7fed1affd700 1 -- 172.21.15.16:0/2585526908 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed080035b0 con 0x7fed08003a30 2023-10-16T12:27:00.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.563+0000 7fed1affd700 1 --2- 172.21.15.16:0/2585526908 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed08003a30 0x7fed08003e10 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7fed0c009b90 tx=0x7fed08005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b6ed0ba02d8d7bd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:00.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.563+0000 7fed19ffb700 1 -- 172.21.15.16:0/2585526908 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed0c00e6c0 con 0x7fed08003a30 2023-10-16T12:27:00.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.563+0000 7fed19ffb700 1 -- 172.21.15.16:0/2585526908 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed0c00e820 con 0x7fed08003a30 2023-10-16T12:27:00.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.563+0000 7fed19ffb700 1 -- 172.21.15.16:0/2585526908 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed0c00eb90 con 0x7fed08003a30 2023-10-16T12:27:00.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.564+0000 7fed212e1700 1 -- 172.21.15.16:0/2585526908 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed08003a30 msgr2=0x7fed08003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:00.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.564+0000 7fed212e1700 1 --2- 172.21.15.16:0/2585526908 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed08003a30 0x7fed08003e10 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7fed0c009b90 tx=0x7fed08005880 comp rx=0 tx=0).stop 2023-10-16T12:27:00.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.564+0000 7fed1b7fe700 1 -- 172.21.15.16:0/2585526908 reap_dead start 2023-10-16T12:27:00.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.564+0000 7fed212e1700 1 -- 172.21.15.16:0/2585526908 shutdown_connections 2023-10-16T12:27:00.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.564+0000 7fed212e1700 1 -- 172.21.15.16:0/2585526908 >> 172.21.15.16:0/2585526908 conn(0x7fed1c072330 msgr2=0x7fed1c072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:00.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.564+0000 7fed212e1700 1 -- 172.21.15.16:0/2585526908 shutdown_connections 2023-10-16T12:27:00.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.564+0000 7fed212e1700 1 -- 172.21.15.16:0/2585526908 wait complete. 2023-10-16T12:27:00.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.565+0000 7fed212e1700 1 Processor -- start 2023-10-16T12:27:00.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.565+0000 7fed212e1700 1 -- start start 2023-10-16T12:27:00.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.566+0000 7fed212e1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed08003a30 0x7fed1c1f2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:00.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.566+0000 7fed212e1700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fed1c0b5c40 0x7fed1c0cadd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:00.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.566+0000 7fed212e1700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fed1c0c6c20 0x7fed1c0b4870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:00.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.566+0000 7fed212e1700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed1c0b60a0 con 0x7fed08003a30 2023-10-16T12:27:00.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.566+0000 7fed212e1700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed1c0cc9e0 con 0x7fed1c0c6c20 2023-10-16T12:27:00.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.566+0000 7fed212e1700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed1c1f3470 con 0x7fed1c0b5c40 2023-10-16T12:27:00.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.566+0000 7fed1affd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed08003a30 0x7fed1c1f2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:00.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.566+0000 7fed1affd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed08003a30 0x7fed1c1f2e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40810/0 (socket says 172.21.15.16:40810) 2023-10-16T12:27:00.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.566+0000 7fed1affd700 1 -- 172.21.15.16:0/3977262966 learned_addr learned my addr 172.21.15.16:0/3977262966 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:00.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.566+0000 7fed1b7fe700 1 --2- 172.21.15.16:0/3977262966 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fed1c0c6c20 0x7fed1c0b4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:00.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.566+0000 7fed1a7fc700 1 --2- 172.21.15.16:0/3977262966 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fed1c0b5c40 0x7fed1c0cadd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:00.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.566+0000 7fed1affd700 1 -- 172.21.15.16:0/3977262966 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fed1c0b5c40 msgr2=0x7fed1c0cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:00.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.566+0000 7fed1affd700 1 --2- 172.21.15.16:0/3977262966 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fed1c0b5c40 0x7fed1c0cadd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:00.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.566+0000 7fed1affd700 1 -- 172.21.15.16:0/3977262966 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fed1c0c6c20 msgr2=0x7fed1c0b4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:00.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.566+0000 7fed1affd700 1 --2- 172.21.15.16:0/3977262966 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fed1c0c6c20 0x7fed1c0b4870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:00.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.566+0000 7fed1affd700 1 -- 172.21.15.16:0/3977262966 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed1c1f3750 con 0x7fed08003a30 2023-10-16T12:27:00.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.566+0000 7fed1affd700 1 --2- 172.21.15.16:0/3977262966 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed08003a30 0x7fed1c1f2e20 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7fed080057f0 tx=0x7fed0c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:00.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.567+0000 7fed137fe700 1 -- 172.21.15.16:0/3977262966 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed0c021070 con 0x7fed08003a30 2023-10-16T12:27:00.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.567+0000 7fed137fe700 1 -- 172.21.15.16:0/3977262966 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed0c0043b0 con 0x7fed08003a30 2023-10-16T12:27:00.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.567+0000 7fed212e1700 1 -- 172.21.15.16:0/3977262966 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed1c1f3a30 con 0x7fed08003a30 2023-10-16T12:27:00.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.567+0000 7fed137fe700 1 -- 172.21.15.16:0/3977262966 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed0c00e6c0 con 0x7fed08003a30 2023-10-16T12:27:00.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.567+0000 7fed212e1700 1 -- 172.21.15.16:0/3977262966 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed1c1f3d10 con 0x7fed08003a30 2023-10-16T12:27:00.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.568+0000 7fed212e1700 1 -- 172.21.15.16:0/3977262966 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed1c04ebb0 con 0x7fed08003a30 2023-10-16T12:27:00.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.568+0000 7fed137fe700 1 -- 172.21.15.16:0/3977262966 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 41) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fed0c025070 con 0x7fed08003a30 2023-10-16T12:27:00.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.569+0000 7fed137fe700 1 --2- 172.21.15.16:0/3977262966 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7fecfc06b050 0x7fecfc06d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:00.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.569+0000 7fed137fe700 1 -- 172.21.15.16:0/3977262966 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5957+0+0 (secure 0 0 0) 0x7fed0c08a960 con 0x7fed08003a30 2023-10-16T12:27:00.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.569+0000 7fed1a7fc700 1 --2- 172.21.15.16:0/3977262966 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7fecfc06b050 0x7fecfc06d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:00.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.569+0000 7fed1a7fc700 1 --2- 172.21.15.16:0/3977262966 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7fecfc06b050 0x7fecfc06d510 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fed04007fd0 tx=0x7fed04009040 comp rx=0 tx=0).ready entity=mgr.6020 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:00.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.573+0000 7fed137fe700 1 -- 172.21.15.16:0/3977262966 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fed0c0559b0 con 0x7fed08003a30 2023-10-16T12:27:00.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:00.824+0000 7fed212e1700 1 -- 172.21.15.16:0/3977262966 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fed1c0ccb20 con 0x7fed08003a30 2023-10-16T12:27:01.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.394+0000 7fed137fe700 1 -- 172.21.15.16:0/3977262966 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 11 and data pool 12 v40) v1 ==== 174+0+0 (secure 0 0 0) 0x7fed1c0ccb20 con 0x7fed08003a30 2023-10-16T12:27:01.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.399+0000 7fed212e1700 1 -- 172.21.15.16:0/3977262966 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7fecfc06b050 msgr2=0x7fecfc06d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:01.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.399+0000 7fed212e1700 1 --2- 172.21.15.16:0/3977262966 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7fecfc06b050 0x7fecfc06d510 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fed04007fd0 tx=0x7fed04009040 comp rx=0 tx=0).stop 2023-10-16T12:27:01.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.399+0000 7fed212e1700 1 -- 172.21.15.16:0/3977262966 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed08003a30 msgr2=0x7fed1c1f2e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:01.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.399+0000 7fed212e1700 1 --2- 172.21.15.16:0/3977262966 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed08003a30 0x7fed1c1f2e20 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7fed080057f0 tx=0x7fed0c0047c0 comp rx=0 tx=0).stop 2023-10-16T12:27:01.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.399+0000 7fed212e1700 1 -- 172.21.15.16:0/3977262966 shutdown_connections 2023-10-16T12:27:01.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.399+0000 7fed212e1700 1 --2- 172.21.15.16:0/3977262966 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7fecfc06b050 0x7fecfc06d510 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:01.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.399+0000 7fed212e1700 1 --2- 172.21.15.16:0/3977262966 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fed1c0c6c20 0x7fed1c0b4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:01.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.399+0000 7fed212e1700 1 --2- 172.21.15.16:0/3977262966 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed08003a30 0x7fed1c1f2e20 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:01.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.399+0000 7fed212e1700 1 --2- 172.21.15.16:0/3977262966 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fed1c0b5c40 0x7fed1c0cadd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:01.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.399+0000 7fed212e1700 1 -- 172.21.15.16:0/3977262966 >> 172.21.15.16:0/3977262966 conn(0x7fed1c072330 msgr2=0x7fed1c0686c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:01.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.402+0000 7fed212e1700 1 -- 172.21.15.16:0/3977262966 shutdown_connections 2023-10-16T12:27:01.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.404+0000 7fed212e1700 1 -- 172.21.15.16:0/3977262966 wait complete. 2023-10-16T12:27:01.409 INFO:teuthology.orchestra.run.smithi016.stderr:new fs with metadata pool 11 and data pool 12 2023-10-16T12:27:01.422 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:27:01.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.552+0000 7fe33cea5700 1 Processor -- start 2023-10-16T12:27:01.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.552+0000 7fe33cea5700 1 -- start start 2023-10-16T12:27:01.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.553+0000 7fe33cea5700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fe33815b760 0x7fe33806a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:01.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.553+0000 7fe33cea5700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe3380b2be0 0x7fe33806ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:01.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.553+0000 7fe33cea5700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3380d03b0 con 0x7fe3380cd6e0 2023-10-16T12:27:01.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.553+0000 7fe33cea5700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe33815bb40 con 0x7fe3380b2be0 2023-10-16T12:27:01.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.553+0000 7fe33cea5700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3380cb760 con 0x7fe33815b760 2023-10-16T12:27:01.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.553+0000 7fe335d9b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe3380b2be0 0x7fe33806ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:01.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.553+0000 7fe335d9b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe3380b2be0 0x7fe33806ad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40832/0 (socket says 172.21.15.16:40832) 2023-10-16T12:27:01.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.553+0000 7fe335d9b700 1 -- 172.21.15.16:0/1914758713 learned_addr learned my addr 172.21.15.16:0/1914758713 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:01.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.554+0000 7fe33659c700 1 --2- 172.21.15.16:0/1914758713 >> v2:172.21.15.49:3300/0 conn(0x7fe33815b760 0x7fe33806a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:01.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.554+0000 7fe335d9b700 1 -- 172.21.15.16:0/1914758713 >> v2:172.21.15.49:3300/0 conn(0x7fe33815b760 msgr2=0x7fe33806a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:01.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.554+0000 7fe335d9b700 1 --2- 172.21.15.16:0/1914758713 >> v2:172.21.15.49:3300/0 conn(0x7fe33815b760 0x7fe33806a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:01.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.554+0000 7fe335d9b700 1 -- 172.21.15.16:0/1914758713 >> v1:172.21.15.49:6789/0 conn(0x7fe3380cd6e0 legacy=0x7fe3380719e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:01.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.554+0000 7fe335d9b700 1 -- 172.21.15.16:0/1914758713 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3380d0600 con 0x7fe3380b2be0 2023-10-16T12:27:01.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.554+0000 7fe335d9b700 1 --2- 172.21.15.16:0/1914758713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3380b2be0 0x7fe33806ad20 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7fe338073910 tx=0x7fe32c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61aa0b9abf191695 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:01.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.554+0000 7fe33559a700 1 -- 172.21.15.16:0/1914758713 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe32c00b870 con 0x7fe3380b2be0 2023-10-16T12:27:01.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.554+0000 7fe33559a700 1 -- 172.21.15.16:0/1914758713 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe32c00b9d0 con 0x7fe3380b2be0 2023-10-16T12:27:01.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.554+0000 7fe33559a700 1 -- 172.21.15.16:0/1914758713 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe32c00bd60 con 0x7fe3380b2be0 2023-10-16T12:27:01.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.555+0000 7fe33cea5700 1 -- 172.21.15.16:0/1914758713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3380b2be0 msgr2=0x7fe33806ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:01.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.555+0000 7fe33cea5700 1 --2- 172.21.15.16:0/1914758713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3380b2be0 0x7fe33806ad20 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7fe338073910 tx=0x7fe32c009bd0 comp rx=0 tx=0).stop 2023-10-16T12:27:01.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.555+0000 7fe33cea5700 1 -- 172.21.15.16:0/1914758713 shutdown_connections 2023-10-16T12:27:01.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.555+0000 7fe33cea5700 1 --2- 172.21.15.16:0/1914758713 >> v2:172.21.15.49:3300/0 conn(0x7fe33815b760 0x7fe33806a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:01.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.555+0000 7fe33cea5700 1 --2- 172.21.15.16:0/1914758713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3380b2be0 0x7fe33806ad20 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:01.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.555+0000 7fe33cea5700 1 -- 172.21.15.16:0/1914758713 >> 172.21.15.16:0/1914758713 conn(0x7fe3380b1fd0 msgr2=0x7fe3380b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:01.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.555+0000 7fe33cea5700 1 -- 172.21.15.16:0/1914758713 shutdown_connections 2023-10-16T12:27:01.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.555+0000 7fe33cea5700 1 -- 172.21.15.16:0/1914758713 wait complete. 2023-10-16T12:27:01.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.556+0000 7fe33cea5700 1 Processor -- start 2023-10-16T12:27:01.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.556+0000 7fe33cea5700 1 -- start start 2023-10-16T12:27:01.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.557+0000 7fe33cea5700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe3380b2be0 0x7fe3380c7510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:01.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.557+0000 7fe33cea5700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe3380cd6e0 0x7fe3380a84c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:01.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.557+0000 7fe33cea5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe33815b760 0x7fe3380a99a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:01.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.557+0000 7fe33cea5700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3380c93b0 con 0x7fe33815b760 2023-10-16T12:27:01.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.557+0000 7fe33cea5700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3380a8a00 con 0x7fe3380cd6e0 2023-10-16T12:27:01.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.557+0000 7fe33cea5700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3380a9550 con 0x7fe3380b2be0 2023-10-16T12:27:01.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.558+0000 7fe336d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe33815b760 0x7fe3380a99a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:01.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.558+0000 7fe336d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe33815b760 0x7fe3380a99a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40842/0 (socket says 172.21.15.16:40842) 2023-10-16T12:27:01.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.558+0000 7fe336d9d700 1 -- 172.21.15.16:0/1468100676 learned_addr learned my addr 172.21.15.16:0/1468100676 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:01.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.558+0000 7fe335d9b700 1 --2- 172.21.15.16:0/1468100676 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe3380cd6e0 0x7fe3380a84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:01.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.558+0000 7fe33659c700 1 --2- 172.21.15.16:0/1468100676 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe3380b2be0 0x7fe3380c7510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:01.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.558+0000 7fe336d9d700 1 -- 172.21.15.16:0/1468100676 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe3380b2be0 msgr2=0x7fe3380c7510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:01.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.558+0000 7fe336d9d700 1 --2- 172.21.15.16:0/1468100676 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe3380b2be0 0x7fe3380c7510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:01.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.558+0000 7fe336d9d700 1 -- 172.21.15.16:0/1468100676 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe3380cd6e0 msgr2=0x7fe3380a84c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:01.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.558+0000 7fe336d9d700 1 --2- 172.21.15.16:0/1468100676 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe3380cd6e0 0x7fe3380a84c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:01.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.558+0000 7fe336d9d700 1 -- 172.21.15.16:0/1468100676 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe32c009550 con 0x7fe33815b760 2023-10-16T12:27:01.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.558+0000 7fe335d9b700 1 --2- 172.21.15.16:0/1468100676 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe3380cd6e0 0x7fe3380a84c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:27:01.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.558+0000 7fe33659c700 1 --2- 172.21.15.16:0/1468100676 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe3380b2be0 0x7fe3380c7510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:27:01.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.558+0000 7fe336d9d700 1 --2- 172.21.15.16:0/1468100676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe33815b760 0x7fe3380a99a0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7fe33000cb40 tx=0x7fe33000cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:01.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.558+0000 7fe326ffd700 1 -- 172.21.15.16:0/1468100676 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe330003c00 con 0x7fe33815b760 2023-10-16T12:27:01.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.558+0000 7fe326ffd700 1 -- 172.21.15.16:0/1468100676 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe330003d60 con 0x7fe33815b760 2023-10-16T12:27:01.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.558+0000 7fe326ffd700 1 -- 172.21.15.16:0/1468100676 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe33000ab90 con 0x7fe33815b760 2023-10-16T12:27:01.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.558+0000 7fe33cea5700 1 -- 172.21.15.16:0/1468100676 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3380a8cb0 con 0x7fe33815b760 2023-10-16T12:27:01.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.558+0000 7fe33cea5700 1 -- 172.21.15.16:0/1468100676 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3380a9270 con 0x7fe33815b760 2023-10-16T12:27:01.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.559+0000 7fe33cea5700 1 -- 172.21.15.16:0/1468100676 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3380c9520 con 0x7fe33815b760 2023-10-16T12:27:01.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.560+0000 7fe326ffd700 1 -- 172.21.15.16:0/1468100676 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 41) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe33000acf0 con 0x7fe33815b760 2023-10-16T12:27:01.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.560+0000 7fe326ffd700 1 --2- 172.21.15.16:0/1468100676 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7fe31806b020 0x7fe31806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:01.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.560+0000 7fe326ffd700 1 -- 172.21.15.16:0/1468100676 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fe3300899d0 con 0x7fe33815b760 2023-10-16T12:27:01.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.560+0000 7fe33659c700 1 --2- 172.21.15.16:0/1468100676 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7fe31806b020 0x7fe31806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:01.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.561+0000 7fe33659c700 1 --2- 172.21.15.16:0/1468100676 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7fe31806b020 0x7fe31806d4e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe328009820 tx=0x7fe328008040 comp rx=0 tx=0).ready entity=mgr.6020 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:01.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.565+0000 7fe326ffd700 1 -- 172.21.15.16:0/1468100676 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe330054770 con 0x7fe33815b760 2023-10-16T12:27:01.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.770+0000 7fe33cea5700 1 -- 172.21.15.16:0/1468100676 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe33804ebb0 con 0x7fe33815b760 2023-10-16T12:27:01.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.771+0000 7fe326ffd700 1 -- 172.21.15.16:0/1468100676 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) v1 ==== 74+0+16725 (secure 0 0 0) 0x7fe330057d90 con 0x7fe33815b760 2023-10-16T12:27:01.773 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:27:01.773 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":98,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:27:00.825612+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:26:59.206630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2023-10-16T12:26:59.784585+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:27:01.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.773+0000 7fe33cea5700 1 -- 172.21.15.16:0/1468100676 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7fe31806b020 msgr2=0x7fe31806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:01.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.773+0000 7fe33cea5700 1 --2- 172.21.15.16:0/1468100676 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7fe31806b020 0x7fe31806d4e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe328009820 tx=0x7fe328008040 comp rx=0 tx=0).stop 2023-10-16T12:27:01.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.773+0000 7fe33cea5700 1 -- 172.21.15.16:0/1468100676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe33815b760 msgr2=0x7fe3380a99a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:01.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.773+0000 7fe33cea5700 1 --2- 172.21.15.16:0/1468100676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe33815b760 0x7fe3380a99a0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7fe33000cb40 tx=0x7fe33000cf00 comp rx=0 tx=0).stop 2023-10-16T12:27:01.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.773+0000 7fe33cea5700 1 -- 172.21.15.16:0/1468100676 shutdown_connections 2023-10-16T12:27:01.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.773+0000 7fe33cea5700 1 --2- 172.21.15.16:0/1468100676 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7fe31806b020 0x7fe31806d4e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:01.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.773+0000 7fe33cea5700 1 --2- 172.21.15.16:0/1468100676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe33815b760 0x7fe3380a99a0 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:01.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.773+0000 7fe33cea5700 1 --2- 172.21.15.16:0/1468100676 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe3380b2be0 0x7fe3380c7510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:01.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.773+0000 7fe33cea5700 1 --2- 172.21.15.16:0/1468100676 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe3380cd6e0 0x7fe3380a84c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:01.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.773+0000 7fe33cea5700 1 -- 172.21.15.16:0/1468100676 >> 172.21.15.16:0/1468100676 conn(0x7fe3380b1fd0 msgr2=0x7fe3380719c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:01.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.773+0000 7fe33cea5700 1 -- 172.21.15.16:0/1468100676 shutdown_connections 2023-10-16T12:27:01.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.773+0000 7fe33cea5700 1 -- 172.21.15.16:0/1468100676 wait complete. 2023-10-16T12:27:01.788 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:27:01.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.910+0000 7f0edc2bb700 1 Processor -- start 2023-10-16T12:27:01.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.910+0000 7f0edc2bb700 1 -- start start 2023-10-16T12:27:01.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.911+0000 7f0edc2bb700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f0ed415b9f0 0x7f0ed40b29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:01.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.911+0000 7f0edc2bb700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0ed40cf280 0x7f0ed406ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:01.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.911+0000 7f0edc2bb700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ed40c3490 con 0x7f0ed406d940 2023-10-16T12:27:01.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.911+0000 7f0edc2bb700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ed406e300 con 0x7f0ed40cf280 2023-10-16T12:27:01.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.911+0000 7f0edc2bb700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ed40cf730 con 0x7f0ed415b9f0 2023-10-16T12:27:01.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.911+0000 7f0eda858700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0ed40cf280 0x7f0ed406ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:01.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.911+0000 7f0eda858700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0ed40cf280 0x7f0ed406ddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40848/0 (socket says 172.21.15.16:40848) 2023-10-16T12:27:01.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.911+0000 7f0ed9856700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0ed406d940 0x7f0ed40ceb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:58352/0 (socket says 172.21.15.16:58352) 2023-10-16T12:27:01.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.911+0000 7f0eda858700 1 -- 172.21.15.16:0/2811226612 learned_addr learned my addr 172.21.15.16:0/2811226612 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:01.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.911+0000 7f0eda057700 1 --2- 172.21.15.16:0/2811226612 >> v2:172.21.15.49:3300/0 conn(0x7f0ed415b9f0 0x7f0ed40b29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:01.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.912+0000 7f0eda858700 1 -- 172.21.15.16:0/2811226612 >> v2:172.21.15.49:3300/0 conn(0x7f0ed415b9f0 msgr2=0x7f0ed40b29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:01.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.912+0000 7f0eda858700 1 --2- 172.21.15.16:0/2811226612 >> v2:172.21.15.49:3300/0 conn(0x7f0ed415b9f0 0x7f0ed40b29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:01.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.912+0000 7f0eda858700 1 -- 172.21.15.16:0/2811226612 >> v1:172.21.15.16:6789/0 conn(0x7f0ed406d940 legacy=0x7f0ed40ceb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:01.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.912+0000 7f0ed9055700 1 -- 172.21.15.16:0/2811226612 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 706557154 0 0) 0x7f0ed40c3490 con 0x7f0ed406d940 2023-10-16T12:27:01.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.912+0000 7f0eda858700 1 -- 172.21.15.16:0/2811226612 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ed40cdad0 con 0x7f0ed40cf280 2023-10-16T12:27:01.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.912+0000 7f0eda858700 1 --2- 172.21.15.16:0/2811226612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ed40cf280 0x7f0ed406ddc0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f0ed40af5e0 tx=0x7f0ecc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e29cab3a2cebe759 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:01.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.912+0000 7f0ed9055700 1 -- 172.21.15.16:0/2811226612 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ecc00e070 con 0x7f0ed40cf280 2023-10-16T12:27:01.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.912+0000 7f0ed9055700 1 -- 172.21.15.16:0/2811226612 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ecc00b910 con 0x7f0ed40cf280 2023-10-16T12:27:01.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.912+0000 7f0ed9055700 1 -- 172.21.15.16:0/2811226612 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ecc00bcc0 con 0x7f0ed40cf280 2023-10-16T12:27:01.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.912+0000 7f0edc2bb700 1 -- 172.21.15.16:0/2811226612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ed40cf280 msgr2=0x7f0ed406ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:01.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.913+0000 7f0edc2bb700 1 --2- 172.21.15.16:0/2811226612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ed40cf280 0x7f0ed406ddc0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f0ed40af5e0 tx=0x7f0ecc00b330 comp rx=0 tx=0).stop 2023-10-16T12:27:01.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.913+0000 7f0edc2bb700 1 -- 172.21.15.16:0/2811226612 shutdown_connections 2023-10-16T12:27:01.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.913+0000 7f0edc2bb700 1 --2- 172.21.15.16:0/2811226612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ed40cf280 0x7f0ed406ddc0 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:01.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.913+0000 7f0edc2bb700 1 --2- 172.21.15.16:0/2811226612 >> v2:172.21.15.49:3300/0 conn(0x7f0ed415b9f0 0x7f0ed40b29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:01.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.913+0000 7f0edc2bb700 1 -- 172.21.15.16:0/2811226612 >> 172.21.15.16:0/2811226612 conn(0x7f0ed40cc5b0 msgr2=0x7f0ed40cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:01.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.913+0000 7f0edc2bb700 1 -- 172.21.15.16:0/2811226612 shutdown_connections 2023-10-16T12:27:01.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.913+0000 7f0edc2bb700 1 -- 172.21.15.16:0/2811226612 wait complete. 2023-10-16T12:27:01.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.914+0000 7f0edc2bb700 1 Processor -- start 2023-10-16T12:27:01.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.914+0000 7f0edc2bb700 1 -- start start 2023-10-16T12:27:01.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.915+0000 7f0edc2bb700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0ed406d940 0x7f0ed414c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:01.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.915+0000 7f0edc2bb700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ed40cf280 0x7f0ed40c5d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:01.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.915+0000 7f0edc2bb700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0ed40c62b0 0x7f0ed40c3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:01.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.915+0000 7f0edc2bb700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed40cb920 con 0x7f0ed40cf280 2023-10-16T12:27:01.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.915+0000 7f0edc2bb700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed40cad10 con 0x7f0ed40c62b0 2023-10-16T12:27:01.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.915+0000 7f0edc2bb700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed40caff0 con 0x7f0ed406d940 2023-10-16T12:27:01.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.915+0000 7f0ed9856700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ed40cf280 0x7f0ed40c5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:01.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.915+0000 7f0ed9856700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ed40cf280 0x7f0ed40c5d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40852/0 (socket says 172.21.15.16:40852) 2023-10-16T12:27:01.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.915+0000 7f0ed9856700 1 -- 172.21.15.16:0/3915514600 learned_addr learned my addr 172.21.15.16:0/3915514600 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:01.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.915+0000 7f0eda858700 1 --2- 172.21.15.16:0/3915514600 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0ed40c62b0 0x7f0ed40c3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:01.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.915+0000 7f0eda057700 1 --2- 172.21.15.16:0/3915514600 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0ed406d940 0x7f0ed414c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:01.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.915+0000 7f0ed9856700 1 -- 172.21.15.16:0/3915514600 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0ed406d940 msgr2=0x7f0ed414c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:01.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.915+0000 7f0ed9856700 1 --2- 172.21.15.16:0/3915514600 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0ed406d940 0x7f0ed414c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:01.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.915+0000 7f0ed9856700 1 -- 172.21.15.16:0/3915514600 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0ed40c62b0 msgr2=0x7f0ed40c3dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:01.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.915+0000 7f0ed9856700 1 --2- 172.21.15.16:0/3915514600 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0ed40c62b0 0x7f0ed40c3dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:01.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.915+0000 7f0ed9856700 1 -- 172.21.15.16:0/3915514600 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ecc009b30 con 0x7f0ed40cf280 2023-10-16T12:27:01.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.915+0000 7f0ed9856700 1 --2- 172.21.15.16:0/3915514600 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ed40cf280 0x7f0ed40c5d70 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f0ed000c520 tx=0x7f0ed000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:01.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.915+0000 7f0ecaffd700 1 -- 172.21.15.16:0/3915514600 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ed0004c80 con 0x7f0ed40cf280 2023-10-16T12:27:01.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.915+0000 7f0ecaffd700 1 -- 172.21.15.16:0/3915514600 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ed0004de0 con 0x7f0ed40cf280 2023-10-16T12:27:01.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.915+0000 7f0edc2bb700 1 -- 172.21.15.16:0/3915514600 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ed40cb2d0 con 0x7f0ed40cf280 2023-10-16T12:27:01.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.915+0000 7f0ecaffd700 1 -- 172.21.15.16:0/3915514600 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ed00072d0 con 0x7f0ed40cf280 2023-10-16T12:27:01.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.916+0000 7f0edc2bb700 1 -- 172.21.15.16:0/3915514600 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ed414c6f0 con 0x7f0ed40cf280 2023-10-16T12:27:01.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.916+0000 7f0edc2bb700 1 -- 172.21.15.16:0/3915514600 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ed40cbad0 con 0x7f0ed40cf280 2023-10-16T12:27:01.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.917+0000 7f0ecaffd700 1 -- 172.21.15.16:0/3915514600 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 41) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0ed0007430 con 0x7f0ed40cf280 2023-10-16T12:27:01.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.918+0000 7f0ecaffd700 1 --2- 172.21.15.16:0/3915514600 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f0eb806b020 0x7f0eb806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:01.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.918+0000 7f0eda057700 1 --2- 172.21.15.16:0/3915514600 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f0eb806b020 0x7f0eb806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:01.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.918+0000 7f0ecaffd700 1 -- 172.21.15.16:0/3915514600 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f0ed0089550 con 0x7f0ed40cf280 2023-10-16T12:27:01.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.918+0000 7f0eda057700 1 --2- 172.21.15.16:0/3915514600 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f0eb806b020 0x7f0eb806d4e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0ec4009b70 tx=0x7f0ec4008040 comp rx=0 tx=0).ready entity=mgr.6020 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:01.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:01.922+0000 7f0ecaffd700 1 -- 172.21.15.16:0/3915514600 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0ed40cbad0 con 0x7f0ed40cf280 2023-10-16T12:27:02.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:02.125+0000 7f0edc2bb700 1 -- 172.21.15.16:0/3915514600 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0ed404ebb0 con 0x7f0ed40cf280 2023-10-16T12:27:02.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:02.126+0000 7f0ecaffd700 1 -- 172.21.15.16:0/3915514600 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) v1 ==== 74+0+16725 (secure 0 0 0) 0x7f0ed0054420 con 0x7f0ed40cf280 2023-10-16T12:27:02.128 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:27:02.128 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":98,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:27:00.825612+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:26:59.206630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2023-10-16T12:26:59.784585+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:27:02.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:02.128+0000 7f0edc2bb700 1 -- 172.21.15.16:0/3915514600 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f0eb806b020 msgr2=0x7f0eb806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:02.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:02.128+0000 7f0edc2bb700 1 --2- 172.21.15.16:0/3915514600 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f0eb806b020 0x7f0eb806d4e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0ec4009b70 tx=0x7f0ec4008040 comp rx=0 tx=0).stop 2023-10-16T12:27:02.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:02.128+0000 7f0edc2bb700 1 -- 172.21.15.16:0/3915514600 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ed40cf280 msgr2=0x7f0ed40c5d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:02.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:02.128+0000 7f0edc2bb700 1 --2- 172.21.15.16:0/3915514600 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ed40cf280 0x7f0ed40c5d70 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f0ed000c520 tx=0x7f0ed000c8e0 comp rx=0 tx=0).stop 2023-10-16T12:27:02.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:02.128+0000 7f0edc2bb700 1 -- 172.21.15.16:0/3915514600 shutdown_connections 2023-10-16T12:27:02.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:02.128+0000 7f0edc2bb700 1 --2- 172.21.15.16:0/3915514600 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f0eb806b020 0x7f0eb806d4e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:02.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:02.128+0000 7f0edc2bb700 1 --2- 172.21.15.16:0/3915514600 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0ed40c62b0 0x7f0ed40c3dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:02.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:02.128+0000 7f0edc2bb700 1 --2- 172.21.15.16:0/3915514600 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0ed406d940 0x7f0ed414c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:02.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:02.128+0000 7f0edc2bb700 1 --2- 172.21.15.16:0/3915514600 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ed40cf280 0x7f0ed40c5d70 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:02.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:02.128+0000 7f0edc2bb700 1 -- 172.21.15.16:0/3915514600 >> 172.21.15.16:0/3915514600 conn(0x7f0ed40cc5b0 msgr2=0x7f0ed40711b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:02.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:02.128+0000 7f0edc2bb700 1 -- 172.21.15.16:0/3915514600 shutdown_connections 2023-10-16T12:27:02.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:02.128+0000 7f0edc2bb700 1 -- 172.21.15.16:0/3915514600 wait complete. 2023-10-16T12:27:02.143 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-16T12:27:02.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:02.271+0000 7f27fac2b700 1 Processor -- start 2023-10-16T12:27:02.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:02.271+0000 7f27fac2b700 1 -- start start 2023-10-16T12:27:02.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:02.272+0000 7f27fac2b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f27f415d8d0 0x7f27f40b2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:02.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:02.272+0000 7f27fac2b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f27f40ce350 0x7f27f40ce780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:02.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:02.272+0000 7f27fac2b700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27f40cecc0 con 0x7f27f40b8fa0 2023-10-16T12:27:02.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:02.272+0000 7f27fac2b700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27f40d0bb0 con 0x7f27f415d8d0 2023-10-16T12:27:02.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:02.272+0000 7f27fac2b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27f415dcb0 con 0x7f27f40ce350 2023-10-16T12:27:02.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:02.272+0000 7f27f89c7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f27f415d8d0 0x7f27f40b2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:02.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:02.272+0000 7f27f3fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f27f40b8fa0 0x7f27f40b6720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:58366/0 (socket says 172.21.15.16:58366) 2023-10-16T12:27:02.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:02.272+0000 7f27f3fff700 1 -- 172.21.15.16:0/3068613934 learned_addr learned my addr 172.21.15.16:0/3068613934 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:27:02.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:02.272+0000 7f27f89c7700 1 -- 172.21.15.16:0/3068613934 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f27f40ce350 msgr2=0x7f27f40ce780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:02.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:02.272+0000 7f27f89c7700 1 --2- 172.21.15.16:0/3068613934 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f27f40ce350 0x7f27f40ce780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:02.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:02.272+0000 7f27f89c7700 1 -- 172.21.15.16:0/3068613934 >> v1:172.21.15.16:6789/0 conn(0x7f27f40b8fa0 legacy=0x7f27f40b6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:02.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:02.272+0000 7f27f37fe700 1 -- 172.21.15.16:0/3068613934 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1427931058 0 0) 0x7f27f40cecc0 con 0x7f27f40b8fa0 2023-10-16T12:27:02.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:02.272+0000 7f27f89c7700 1 -- 172.21.15.16:0/3068613934 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27f40b2f30 con 0x7f27f415d8d0 2023-10-16T12:27:02.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:02.273+0000 7f27f89c7700 1 --2- 172.21.15.16:0/3068613934 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27f415d8d0 0x7f27f40b2830 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f27dc009840 tx=0x7f27dc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f87718a6c411387 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:02.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:02.273+0000 7f27f37fe700 1 -- 172.21.15.16:0/3068613934 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27dc011830 con 0x7f27f415d8d0 2023-10-16T12:27:02.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:02.273+0000 7f27f37fe700 1 -- 172.21.15.16:0/3068613934 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27dc011990 con 0x7f27f415d8d0 2023-10-16T12:27:02.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:02.273+0000 7f27f37fe700 1 -- 172.21.15.16:0/3068613934 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27dc011d20 con 0x7f27f415d8d0 2023-10-16T12:27:02.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:02.273+0000 7f27fac2b700 1 -- 172.21.15.16:0/3068613934 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27f415d8d0 msgr2=0x7f27f40b2830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:02.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:02.273+0000 7f27fac2b700 1 --2- 172.21.15.16:0/3068613934 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27f415d8d0 0x7f27f40b2830 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f27dc009840 tx=0x7f27dc009b50 comp rx=0 tx=0).stop 2023-10-16T12:27:02.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:02.273+0000 7f27fac2b700 1 -- 172.21.15.16:0/3068613934 shutdown_connections 2023-10-16T12:27:02.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:02.273+0000 7f27fac2b700 1 --2- 172.21.15.16:0/3068613934 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f27f40ce350 0x7f27f40ce780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:02.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:02.273+0000 7f27fac2b700 1 --2- 172.21.15.16:0/3068613934 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27f415d8d0 0x7f27f40b2830 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:02.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:02.273+0000 7f27fac2b700 1 -- 172.21.15.16:0/3068613934 >> 172.21.15.16:0/3068613934 conn(0x7f27f4072330 msgr2=0x7f27f4072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:02.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:02.273+0000 7f27fac2b700 1 -- 172.21.15.16:0/3068613934 shutdown_connections 2023-10-16T12:27:02.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:02.273+0000 7f27fac2b700 1 -- 172.21.15.16:0/3068613934 wait complete. 2023-10-16T12:27:02.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:02.275+0000 7f27fac2b700 1 Processor -- start 2023-10-16T12:27:02.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:02.275+0000 7f27fac2b700 1 -- start start 2023-10-16T12:27:02.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:02.275+0000 7f27fac2b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f27f40b8fa0 0x7f27f40cc8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:02.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:02.275+0000 7f27fac2b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27f40ce350 0x7f27f40b3120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:02.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:02.275+0000 7f27fac2b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f27f415d8d0 0x7f27f40b3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:02.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:02.275+0000 7f27fac2b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27f415a1a0 con 0x7f27f40ce350 2023-10-16T12:27:02.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:02.275+0000 7f27fac2b700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27f41f6c60 con 0x7f27f415d8d0 2023-10-16T12:27:02.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:02.275+0000 7f27fac2b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27f40b3d30 con 0x7f27f40b8fa0 2023-10-16T12:27:02.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:02.276+0000 7f27f3fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27f40ce350 0x7f27f40b3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:02.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:02.276+0000 7f27f3fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27f40ce350 0x7f27f40b3120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40866/0 (socket says 172.21.15.16:40866) 2023-10-16T12:27:02.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:02.276+0000 7f27f3fff700 1 -- 172.21.15.16:0/366359911 learned_addr learned my addr 172.21.15.16:0/366359911 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:02.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:02.276+0000 7f27f89c7700 1 --2- 172.21.15.16:0/366359911 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f27f40b8fa0 0x7f27f40cc8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:02.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:02.276+0000 7f27f91c8700 1 --2- 172.21.15.16:0/366359911 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f27f415d8d0 0x7f27f40b3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:02.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:02.276+0000 7f27f3fff700 1 -- 172.21.15.16:0/366359911 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f27f40b8fa0 msgr2=0x7f27f40cc8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:02.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:02.276+0000 7f27f3fff700 1 --2- 172.21.15.16:0/366359911 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f27f40b8fa0 0x7f27f40cc8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:02.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:02.276+0000 7f27f3fff700 1 -- 172.21.15.16:0/366359911 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f27f415d8d0 msgr2=0x7f27f40b3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:02.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:02.276+0000 7f27f3fff700 1 --2- 172.21.15.16:0/366359911 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f27f415d8d0 0x7f27f40b3660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:02.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:02.276+0000 7f27f3fff700 1 -- 172.21.15.16:0/366359911 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27dc0094a0 con 0x7f27f40ce350 2023-10-16T12:27:02.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:02.276+0000 7f27f91c8700 1 --2- 172.21.15.16:0/366359911 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f27f415d8d0 0x7f27f40b3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:27:02.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:02.276+0000 7f27f3fff700 1 --2- 172.21.15.16:0/366359911 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27f40ce350 0x7f27f40b3120 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f27e400c550 tx=0x7f27e400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:02.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:02.276+0000 7f27f17fa700 1 -- 172.21.15.16:0/366359911 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27e4004c80 con 0x7f27f40ce350 2023-10-16T12:27:02.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:02.276+0000 7f27f17fa700 1 -- 172.21.15.16:0/366359911 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27e4004e00 con 0x7f27f40ce350 2023-10-16T12:27:02.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:02.276+0000 7f27fac2b700 1 -- 172.21.15.16:0/366359911 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27f41f6f20 con 0x7f27f40ce350 2023-10-16T12:27:02.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:02.276+0000 7f27f17fa700 1 -- 172.21.15.16:0/366359911 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27e4007310 con 0x7f27f40ce350 2023-10-16T12:27:02.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:02.276+0000 7f27fac2b700 1 -- 172.21.15.16:0/366359911 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27f40cce30 con 0x7f27f40ce350 2023-10-16T12:27:02.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:02.277+0000 7f27fac2b700 1 -- 172.21.15.16:0/366359911 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27f41f7440 con 0x7f27f40ce350 2023-10-16T12:27:02.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:02.278+0000 7f27f17fa700 1 -- 172.21.15.16:0/366359911 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 41) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f27e4007470 con 0x7f27f40ce350 2023-10-16T12:27:02.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:02.278+0000 7f27f17fa700 1 --2- 172.21.15.16:0/366359911 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f27d406b020 0x7f27d406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:02.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:02.279+0000 7f27f17fa700 1 -- 172.21.15.16:0/366359911 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f27e40899b0 con 0x7f27f40ce350 2023-10-16T12:27:02.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:02.279+0000 7f27f89c7700 1 --2- 172.21.15.16:0/366359911 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f27d406b020 0x7f27d406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:02.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:02.279+0000 7f27f89c7700 1 --2- 172.21.15.16:0/366359911 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f27d406b020 0x7f27d406d4e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f27dc000c00 tx=0x7f27dc01c040 comp rx=0 tx=0).ready entity=mgr.6020 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:02.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:02.283+0000 7f27f17fa700 1 -- 172.21.15.16:0/366359911 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f27e4054860 con 0x7f27f40ce350 2023-10-16T12:27:02.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:02.538+0000 7f27fac2b700 1 -- 172.21.15.16:0/366359911 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f27f41f7440 con 0x7f27f40ce350 2023-10-16T12:27:03.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:03.409+0000 7f27f17fa700 1 -- 172.21.15.16:0/366359911 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v43) v1 ==== 118+0+0 (secure 0 0 0) 0x7f27f41f7440 con 0x7f27f40ce350 2023-10-16T12:27:03.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:03.412+0000 7f27fac2b700 1 -- 172.21.15.16:0/366359911 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f27d406b020 msgr2=0x7f27d406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:03.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:03.412+0000 7f27fac2b700 1 --2- 172.21.15.16:0/366359911 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f27d406b020 0x7f27d406d4e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f27dc000c00 tx=0x7f27dc01c040 comp rx=0 tx=0).stop 2023-10-16T12:27:03.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:03.412+0000 7f27fac2b700 1 -- 172.21.15.16:0/366359911 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27f40ce350 msgr2=0x7f27f40b3120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:03.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:03.412+0000 7f27fac2b700 1 --2- 172.21.15.16:0/366359911 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27f40ce350 0x7f27f40b3120 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f27e400c550 tx=0x7f27e400c860 comp rx=0 tx=0).stop 2023-10-16T12:27:03.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:03.412+0000 7f27fac2b700 1 -- 172.21.15.16:0/366359911 shutdown_connections 2023-10-16T12:27:03.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:03.412+0000 7f27fac2b700 1 --2- 172.21.15.16:0/366359911 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f27d406b020 0x7f27d406d4e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:03.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:03.412+0000 7f27fac2b700 1 --2- 172.21.15.16:0/366359911 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f27f415d8d0 0x7f27f40b3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:03.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:03.412+0000 7f27fac2b700 1 --2- 172.21.15.16:0/366359911 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f27f40b8fa0 0x7f27f40cc8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:03.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:03.412+0000 7f27fac2b700 1 --2- 172.21.15.16:0/366359911 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27f40ce350 0x7f27f40b3120 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:03.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:03.412+0000 7f27fac2b700 1 -- 172.21.15.16:0/366359911 >> 172.21.15.16:0/366359911 conn(0x7f27f4072330 msgr2=0x7f27f40cff00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:03.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:03.412+0000 7f27fac2b700 1 -- 172.21.15.16:0/366359911 shutdown_connections 2023-10-16T12:27:03.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:03.412+0000 7f27fac2b700 1 -- 172.21.15.16:0/366359911 wait complete. 2023-10-16T12:27:03.429 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-16T12:27:03.430 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-16T12:27:03.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:03.563+0000 7fac583c5700 1 Processor -- start 2023-10-16T12:27:03.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:03.563+0000 7fac583c5700 1 -- start start 2023-10-16T12:27:03.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:03.564+0000 7fac583c5700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fac5006e170 0x7fac500cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:03.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:03.564+0000 7fac583c5700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac500cef70 con 0x7fac500cb880 2023-10-16T12:27:03.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:03.564+0000 7fac583c5700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac500ce360 con 0x7fac5015b070 2023-10-16T12:27:03.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:03.564+0000 7fac583c5700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac500ce640 con 0x7fac5006e170 2023-10-16T12:27:03.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:03.564+0000 7fac56962700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fac5006e170 0x7fac500cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:03.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:03.564+0000 7fac55960700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fac500cb880 0x7fac5006da60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:58380/0 (socket says 172.21.15.16:58380) 2023-10-16T12:27:03.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:03.564+0000 7fac55960700 1 -- 172.21.15.16:0/4101355599 learned_addr learned my addr 172.21.15.16:0/4101355599 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:27:03.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:03.565+0000 7fac56962700 1 -- 172.21.15.16:0/4101355599 >> v1:172.21.15.49:6789/0 conn(0x7fac5015b070 legacy=0x7fac5015b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:03.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:03.565+0000 7fac56962700 1 -- 172.21.15.16:0/4101355599 >> v1:172.21.15.16:6789/0 conn(0x7fac500cb880 legacy=0x7fac5006da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:03.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:03.565+0000 7fac56962700 1 -- 172.21.15.16:0/4101355599 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac500ce920 con 0x7fac5006e170 2023-10-16T12:27:03.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:03.565+0000 7fac5515f700 1 -- 172.21.15.16:0/4101355599 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3338286795 0 0) 0x7fac500cef70 con 0x7fac500cb880 2023-10-16T12:27:03.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:03.565+0000 7fac56962700 1 --2- 172.21.15.16:0/4101355599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac5006e170 0x7fac500cbd00 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fac48009d90 tx=0x7fac4800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45b913b9dc9b3d2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:03.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:03.565+0000 7fac5515f700 1 -- 172.21.15.16:0/4101355599 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac4800e070 con 0x7fac5006e170 2023-10-16T12:27:03.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:03.565+0000 7fac5515f700 1 -- 172.21.15.16:0/4101355599 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac4800bb90 con 0x7fac5006e170 2023-10-16T12:27:03.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:03.565+0000 7fac5515f700 1 -- 172.21.15.16:0/4101355599 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac48003a20 con 0x7fac5006e170 2023-10-16T12:27:03.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:03.565+0000 7fac583c5700 1 -- 172.21.15.16:0/4101355599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac5006e170 msgr2=0x7fac500cbd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:03.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:03.566+0000 7fac583c5700 1 --2- 172.21.15.16:0/4101355599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac5006e170 0x7fac500cbd00 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fac48009d90 tx=0x7fac4800b330 comp rx=0 tx=0).stop 2023-10-16T12:27:03.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:03.566+0000 7fac583c5700 1 -- 172.21.15.16:0/4101355599 shutdown_connections 2023-10-16T12:27:03.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:03.566+0000 7fac583c5700 1 --2- 172.21.15.16:0/4101355599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac5006e170 0x7fac500cbd00 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:03.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:03.566+0000 7fac583c5700 1 -- 172.21.15.16:0/4101355599 >> 172.21.15.16:0/4101355599 conn(0x7fac500b1f40 msgr2=0x7fac500b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:03.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:03.566+0000 7fac583c5700 1 -- 172.21.15.16:0/4101355599 shutdown_connections 2023-10-16T12:27:03.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:03.566+0000 7fac583c5700 1 -- 172.21.15.16:0/4101355599 wait complete. 2023-10-16T12:27:03.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:03.567+0000 7fac583c5700 1 Processor -- start 2023-10-16T12:27:03.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:03.567+0000 7fac583c5700 1 -- start start 2023-10-16T12:27:03.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:03.568+0000 7fac583c5700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fac5006e170 0x7fac501f2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:03.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:03.568+0000 7fac583c5700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fac500cb880 0x7fac501f5be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:03.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:03.568+0000 7fac583c5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac5015b070 0x7fac500b4080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:03.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:03.568+0000 7fac583c5700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac501543a0 con 0x7fac5015b070 2023-10-16T12:27:03.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:03.568+0000 7fac583c5700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac500b45c0 con 0x7fac500cb880 2023-10-16T12:27:03.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:03.568+0000 7fac583c5700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac500b4860 con 0x7fac5006e170 2023-10-16T12:27:03.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:03.568+0000 7fac56962700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac5015b070 0x7fac500b4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:03.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:03.568+0000 7fac56962700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac5015b070 0x7fac500b4080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40882/0 (socket says 172.21.15.16:40882) 2023-10-16T12:27:03.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:03.568+0000 7fac56962700 1 -- 172.21.15.16:0/3284707759 learned_addr learned my addr 172.21.15.16:0/3284707759 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:03.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:03.568+0000 7fac55960700 1 --2- 172.21.15.16:0/3284707759 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fac500cb880 0x7fac501f5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:03.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:03.568+0000 7fac56161700 1 --2- 172.21.15.16:0/3284707759 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fac5006e170 0x7fac501f2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:03.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:03.568+0000 7fac56962700 1 -- 172.21.15.16:0/3284707759 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fac5006e170 msgr2=0x7fac501f2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:03.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:03.568+0000 7fac56962700 1 --2- 172.21.15.16:0/3284707759 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fac5006e170 0x7fac501f2c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:03.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:03.568+0000 7fac56962700 1 -- 172.21.15.16:0/3284707759 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fac500cb880 msgr2=0x7fac501f5be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:03.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:03.568+0000 7fac56962700 1 --2- 172.21.15.16:0/3284707759 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fac500cb880 0x7fac501f5be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:03.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:03.568+0000 7fac56962700 1 -- 172.21.15.16:0/3284707759 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac48009a70 con 0x7fac5015b070 2023-10-16T12:27:03.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:03.568+0000 7fac56962700 1 --2- 172.21.15.16:0/3284707759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac5015b070 0x7fac500b4080 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7fac50070620 tx=0x7fac48004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:03.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:03.568+0000 7fac46ffd700 1 -- 172.21.15.16:0/3284707759 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac48003a20 con 0x7fac5015b070 2023-10-16T12:27:03.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:03.568+0000 7fac46ffd700 1 -- 172.21.15.16:0/3284707759 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac48012860 con 0x7fac5015b070 2023-10-16T12:27:03.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:03.568+0000 7fac46ffd700 1 -- 172.21.15.16:0/3284707759 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac48012c10 con 0x7fac5015b070 2023-10-16T12:27:03.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:03.568+0000 7fac583c5700 1 -- 172.21.15.16:0/3284707759 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac500b4b40 con 0x7fac5015b070 2023-10-16T12:27:03.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:03.569+0000 7fac583c5700 1 -- 172.21.15.16:0/3284707759 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac500b5100 con 0x7fac5015b070 2023-10-16T12:27:03.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:03.569+0000 7fac583c5700 1 -- 172.21.15.16:0/3284707759 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac5004ebb0 con 0x7fac5015b070 2023-10-16T12:27:03.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:03.570+0000 7fac46ffd700 1 -- 172.21.15.16:0/3284707759 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 41) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fac4801d420 con 0x7fac5015b070 2023-10-16T12:27:03.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:03.570+0000 7fac46ffd700 1 --2- 172.21.15.16:0/3284707759 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7fac3406b020 0x7fac3406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:03.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:03.570+0000 7fac46ffd700 1 -- 172.21.15.16:0/3284707759 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fac4808bc50 con 0x7fac5015b070 2023-10-16T12:27:03.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:03.570+0000 7fac56161700 1 --2- 172.21.15.16:0/3284707759 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7fac3406b020 0x7fac3406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:03.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:03.571+0000 7fac56161700 1 --2- 172.21.15.16:0/3284707759 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7fac3406b020 0x7fac3406d4e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fac40009870 tx=0x7fac40008040 comp rx=0 tx=0).ready entity=mgr.6020 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:03.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:03.575+0000 7fac46ffd700 1 -- 172.21.15.16:0/3284707759 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fac48091020 con 0x7fac5015b070 2023-10-16T12:27:03.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:03.830+0000 7fac583c5700 1 -- 172.21.15.16:0/3284707759 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fac500d1300 con 0x7fac5015b070 2023-10-16T12:27:04.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.410+0000 7fac46ffd700 1 -- 172.21.15.16:0/3284707759 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v44) v1 ==== 122+0+0 (secure 0 0 0) 0x7fac48056a10 con 0x7fac5015b070 2023-10-16T12:27:04.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.413+0000 7fac583c5700 1 -- 172.21.15.16:0/3284707759 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7fac3406b020 msgr2=0x7fac3406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:04.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.413+0000 7fac583c5700 1 --2- 172.21.15.16:0/3284707759 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7fac3406b020 0x7fac3406d4e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fac40009870 tx=0x7fac40008040 comp rx=0 tx=0).stop 2023-10-16T12:27:04.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.413+0000 7fac583c5700 1 -- 172.21.15.16:0/3284707759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac5015b070 msgr2=0x7fac500b4080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:04.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.413+0000 7fac583c5700 1 --2- 172.21.15.16:0/3284707759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac5015b070 0x7fac500b4080 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7fac50070620 tx=0x7fac48004ad0 comp rx=0 tx=0).stop 2023-10-16T12:27:04.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.413+0000 7fac583c5700 1 -- 172.21.15.16:0/3284707759 shutdown_connections 2023-10-16T12:27:04.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.413+0000 7fac583c5700 1 --2- 172.21.15.16:0/3284707759 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7fac3406b020 0x7fac3406d4e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:04.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.413+0000 7fac583c5700 1 --2- 172.21.15.16:0/3284707759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac5015b070 0x7fac500b4080 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:04.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.413+0000 7fac583c5700 1 --2- 172.21.15.16:0/3284707759 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fac5006e170 0x7fac501f2c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:04.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.413+0000 7fac583c5700 1 --2- 172.21.15.16:0/3284707759 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fac500cb880 0x7fac501f5be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:04.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.413+0000 7fac583c5700 1 -- 172.21.15.16:0/3284707759 >> 172.21.15.16:0/3284707759 conn(0x7fac500b1f40 msgr2=0x7fac500cbd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:04.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.413+0000 7fac583c5700 1 -- 172.21.15.16:0/3284707759 shutdown_connections 2023-10-16T12:27:04.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.413+0000 7fac583c5700 1 -- 172.21.15.16:0/3284707759 wait complete. 2023-10-16T12:27:04.431 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-16T12:27:04.431 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:27:04.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.563+0000 7f0df15f6700 1 Processor -- start 2023-10-16T12:27:04.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.563+0000 7f0df15f6700 1 -- start start 2023-10-16T12:27:04.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.563+0000 7f0df15f6700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f0dec0ce930 0x7f0dec0caa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:04.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.563+0000 7f0df15f6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0dec1471f0 0x7f0dec06dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:04.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.563+0000 7f0df15f6700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dec0ced10 con 0x7f0dec146c10 2023-10-16T12:27:04.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.563+0000 7f0df15f6700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0dec15dd50 con 0x7f0dec1471f0 2023-10-16T12:27:04.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.563+0000 7f0df15f6700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0dec0ccdf0 con 0x7f0dec0ce930 2023-10-16T12:27:04.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.564+0000 7f0dea7fc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0dec1471f0 0x7f0dec06dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:04.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.564+0000 7f0dea7fc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0dec1471f0 0x7f0dec06dbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40890/0 (socket says 172.21.15.16:40890) 2023-10-16T12:27:04.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.564+0000 7f0dea7fc700 1 -- 172.21.15.16:0/3957153190 learned_addr learned my addr 172.21.15.16:0/3957153190 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:04.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.564+0000 7f0deaffd700 1 --2- 172.21.15.16:0/3957153190 >> v2:172.21.15.49:3300/0 conn(0x7f0dec0ce930 0x7f0dec0caa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:04.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.564+0000 7f0dea7fc700 1 -- 172.21.15.16:0/3957153190 >> v2:172.21.15.49:3300/0 conn(0x7f0dec0ce930 msgr2=0x7f0dec0caa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:04.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.564+0000 7f0dea7fc700 1 --2- 172.21.15.16:0/3957153190 >> v2:172.21.15.49:3300/0 conn(0x7f0dec0ce930 0x7f0dec0caa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:04.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.564+0000 7f0dea7fc700 1 -- 172.21.15.16:0/3957153190 >> v1:172.21.15.49:6789/0 conn(0x7f0dec146c10 legacy=0x7f0dec0cd2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:04.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.564+0000 7f0dea7fc700 1 -- 172.21.15.16:0/3957153190 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dec06ac50 con 0x7f0dec1471f0 2023-10-16T12:27:04.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.564+0000 7f0dea7fc700 1 --2- 172.21.15.16:0/3957153190 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0dec1471f0 0x7f0dec06dbe0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f0dec0ccdc0 tx=0x7f0dd4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=703cceb00f598a3f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:04.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.564+0000 7f0de9ffb700 1 -- 172.21.15.16:0/3957153190 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dd400b870 con 0x7f0dec1471f0 2023-10-16T12:27:04.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.564+0000 7f0de9ffb700 1 -- 172.21.15.16:0/3957153190 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dd400b9d0 con 0x7f0dec1471f0 2023-10-16T12:27:04.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.564+0000 7f0de9ffb700 1 -- 172.21.15.16:0/3957153190 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dd400bd60 con 0x7f0dec1471f0 2023-10-16T12:27:04.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.565+0000 7f0df15f6700 1 -- 172.21.15.16:0/3957153190 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0dec1471f0 msgr2=0x7f0dec06dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:04.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.565+0000 7f0df15f6700 1 --2- 172.21.15.16:0/3957153190 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0dec1471f0 0x7f0dec06dbe0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f0dec0ccdc0 tx=0x7f0dd4009bd0 comp rx=0 tx=0).stop 2023-10-16T12:27:04.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.565+0000 7f0df15f6700 1 -- 172.21.15.16:0/3957153190 shutdown_connections 2023-10-16T12:27:04.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.565+0000 7f0df15f6700 1 --2- 172.21.15.16:0/3957153190 >> v2:172.21.15.49:3300/0 conn(0x7f0dec0ce930 0x7f0dec0caa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:04.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.565+0000 7f0df15f6700 1 --2- 172.21.15.16:0/3957153190 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0dec1471f0 0x7f0dec06dbe0 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:04.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.565+0000 7f0df15f6700 1 -- 172.21.15.16:0/3957153190 >> 172.21.15.16:0/3957153190 conn(0x7f0dec06bad0 msgr2=0x7f0dec06bee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:04.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.565+0000 7f0df15f6700 1 -- 172.21.15.16:0/3957153190 shutdown_connections 2023-10-16T12:27:04.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.565+0000 7f0df15f6700 1 -- 172.21.15.16:0/3957153190 wait complete. 2023-10-16T12:27:04.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.566+0000 7f0df15f6700 1 Processor -- start 2023-10-16T12:27:04.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.566+0000 7f0df15f6700 1 -- start start 2023-10-16T12:27:04.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.567+0000 7f0df15f6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0dec0ce930 0x7f0dec0acda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:04.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.567+0000 7f0df15f6700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0dec146c10 0x7f0dec0ad2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:04.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.567+0000 7f0df15f6700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0dec1471f0 0x7f0dec0ad820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:04.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.567+0000 7f0df15f6700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dec0add60 con 0x7f0dec0ce930 2023-10-16T12:27:04.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.567+0000 7f0df15f6700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dec0c7ec0 con 0x7f0dec146c10 2023-10-16T12:27:04.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.567+0000 7f0df15f6700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dec0c2ed0 con 0x7f0dec1471f0 2023-10-16T12:27:04.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.567+0000 7f0deaffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0dec0ce930 0x7f0dec0acda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:04.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.567+0000 7f0deaffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0dec0ce930 0x7f0dec0acda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40892/0 (socket says 172.21.15.16:40892) 2023-10-16T12:27:04.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.567+0000 7f0deaffd700 1 -- 172.21.15.16:0/81152705 learned_addr learned my addr 172.21.15.16:0/81152705 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:04.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.567+0000 7f0dea7fc700 1 --2- 172.21.15.16:0/81152705 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0dec146c10 0x7f0dec0ad2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:04.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.567+0000 7f0deb7fe700 1 --2- 172.21.15.16:0/81152705 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0dec1471f0 0x7f0dec0ad820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:04.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.567+0000 7f0deaffd700 1 -- 172.21.15.16:0/81152705 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0dec1471f0 msgr2=0x7f0dec0ad820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:04.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.567+0000 7f0deaffd700 1 --2- 172.21.15.16:0/81152705 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0dec1471f0 0x7f0dec0ad820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:04.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.567+0000 7f0deaffd700 1 -- 172.21.15.16:0/81152705 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0dec146c10 msgr2=0x7f0dec0ad2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:04.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.567+0000 7f0deaffd700 1 --2- 172.21.15.16:0/81152705 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0dec146c10 0x7f0dec0ad2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:04.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.567+0000 7f0deaffd700 1 -- 172.21.15.16:0/81152705 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dd4009550 con 0x7f0dec0ce930 2023-10-16T12:27:04.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.568+0000 7f0deaffd700 1 --2- 172.21.15.16:0/81152705 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0dec0ce930 0x7f0dec0acda0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f0ddc00e840 tx=0x7f0ddc00eb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:04.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.568+0000 7f0de37fe700 1 -- 172.21.15.16:0/81152705 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ddc013610 con 0x7f0dec0ce930 2023-10-16T12:27:04.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.568+0000 7f0de37fe700 1 -- 172.21.15.16:0/81152705 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ddc013770 con 0x7f0dec0ce930 2023-10-16T12:27:04.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.568+0000 7f0df15f6700 1 -- 172.21.15.16:0/81152705 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dec0c6580 con 0x7f0dec0ce930 2023-10-16T12:27:04.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.568+0000 7f0de37fe700 1 -- 172.21.15.16:0/81152705 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ddc013b20 con 0x7f0dec0ce930 2023-10-16T12:27:04.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.568+0000 7f0df15f6700 1 -- 172.21.15.16:0/81152705 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dec0c83f0 con 0x7f0dec0ce930 2023-10-16T12:27:04.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.569+0000 7f0df15f6700 1 -- 172.21.15.16:0/81152705 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dec0c8a40 con 0x7f0dec0ce930 2023-10-16T12:27:04.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.569+0000 7f0de37fe700 1 -- 172.21.15.16:0/81152705 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 41) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0ddc013c80 con 0x7f0dec0ce930 2023-10-16T12:27:04.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.570+0000 7f0de37fe700 1 --2- 172.21.15.16:0/81152705 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f0dcc06b020 0x7f0dcc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:04.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.570+0000 7f0de37fe700 1 -- 172.21.15.16:0/81152705 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f0ddc089550 con 0x7f0dec0ce930 2023-10-16T12:27:04.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.570+0000 7f0dea7fc700 1 --2- 172.21.15.16:0/81152705 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f0dcc06b020 0x7f0dcc06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:04.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.570+0000 7f0dea7fc700 1 --2- 172.21.15.16:0/81152705 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f0dcc06b020 0x7f0dcc06d4e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0dd4014040 tx=0x7f0dd4008030 comp rx=0 tx=0).ready entity=mgr.6020 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:04.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.574+0000 7f0de37fe700 1 -- 172.21.15.16:0/81152705 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0ddc054290 con 0x7f0dec0ce930 2023-10-16T12:27:04.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.823+0000 7f0df15f6700 1 -- 172.21.15.16:0/81152705 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0dec04ebb0 con 0x7f0dec0ce930 2023-10-16T12:27:04.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.824+0000 7f0de37fe700 1 -- 172.21.15.16:0/81152705 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+4915 (secure 0 0 0) 0x7f0ddc0578b0 con 0x7f0dec0ce930 2023-10-16T12:27:04.826 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:27:04.826 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":44,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6820,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/1256919442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":1256919442},{"type":"v1","addr":"172.21.15.49:6834","nonce":1256919442}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":6823,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/3811815346","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":3811815346},{"type":"v1","addr":"172.21.15.49:6835","nonce":3811815346}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":7110,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/1226064755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":1226064755},{"type":"v1","addr":"172.21.15.16:6836","nonce":1226064755}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:27:00.825934+0000","modified":"2023-10-16T12:27:03.831266+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7113},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7113":{"gid":7113,"name":"a","rank":0,"incarnation":40,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/1843321005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1843321005},{"type":"v1","addr":"172.21.15.16:6837","nonce":1843321005}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-10-16T12:27:04.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.826+0000 7f0df15f6700 1 -- 172.21.15.16:0/81152705 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f0dcc06b020 msgr2=0x7f0dcc06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:04.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.826+0000 7f0df15f6700 1 --2- 172.21.15.16:0/81152705 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f0dcc06b020 0x7f0dcc06d4e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0dd4014040 tx=0x7f0dd4008030 comp rx=0 tx=0).stop 2023-10-16T12:27:04.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.826+0000 7f0df15f6700 1 -- 172.21.15.16:0/81152705 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0dec0ce930 msgr2=0x7f0dec0acda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:04.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.826+0000 7f0df15f6700 1 --2- 172.21.15.16:0/81152705 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0dec0ce930 0x7f0dec0acda0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f0ddc00e840 tx=0x7f0ddc00eb50 comp rx=0 tx=0).stop 2023-10-16T12:27:04.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.826+0000 7f0df15f6700 1 -- 172.21.15.16:0/81152705 shutdown_connections 2023-10-16T12:27:04.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.826+0000 7f0df15f6700 1 --2- 172.21.15.16:0/81152705 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f0dcc06b020 0x7f0dcc06d4e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:04.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.826+0000 7f0df15f6700 1 --2- 172.21.15.16:0/81152705 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0dec1471f0 0x7f0dec0ad820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:04.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.826+0000 7f0df15f6700 1 --2- 172.21.15.16:0/81152705 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0dec0ce930 0x7f0dec0acda0 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:04.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.826+0000 7f0df15f6700 1 --2- 172.21.15.16:0/81152705 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0dec146c10 0x7f0dec0ad2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:04.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.826+0000 7f0df15f6700 1 -- 172.21.15.16:0/81152705 >> 172.21.15.16:0/81152705 conn(0x7f0dec06bad0 msgr2=0x7f0dec06dbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:04.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.826+0000 7f0df15f6700 1 -- 172.21.15.16:0/81152705 shutdown_connections 2023-10-16T12:27:04.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.826+0000 7f0df15f6700 1 -- 172.21.15.16:0/81152705 wait complete. 2023-10-16T12:27:04.831 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 44 2023-10-16T12:27:04.841 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:27:04.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.963+0000 7fb30479a700 1 Processor -- start 2023-10-16T12:27:04.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.963+0000 7fb30479a700 1 -- start start 2023-10-16T12:27:04.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.964+0000 7fb30479a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb2fc06aef0 0x7fb2fc0b35b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:04.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.964+0000 7fb30479a700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2fc0d03b0 con 0x7fb2fc0b2be0 2023-10-16T12:27:04.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.964+0000 7fb30479a700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2fc15bb40 con 0x7fb2fc15b760 2023-10-16T12:27:04.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.964+0000 7fb30479a700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2fc0cb760 con 0x7fb2fc06aef0 2023-10-16T12:27:04.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.964+0000 7fb301d35700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb2fc06aef0 0x7fb2fc0b35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:04.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.964+0000 7fb301d35700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb2fc06aef0 0x7fb2fc0b35b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40902/0 (socket says 172.21.15.16:40902) 2023-10-16T12:27:04.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.964+0000 7fb301d35700 1 -- 172.21.15.16:0/3150623832 learned_addr learned my addr 172.21.15.16:0/3150623832 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:04.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.964+0000 7fb302d37700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb2fc0b2be0 0x7fb2fc0b3af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:58410/0 (socket says 172.21.15.16:58410) 2023-10-16T12:27:04.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.964+0000 7fb301d35700 1 -- 172.21.15.16:0/3150623832 >> v1:172.21.15.49:6789/0 conn(0x7fb2fc15b760 legacy=0x7fb2fc06a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:04.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.964+0000 7fb301d35700 1 -- 172.21.15.16:0/3150623832 >> v1:172.21.15.16:6789/0 conn(0x7fb2fc0b2be0 legacy=0x7fb2fc0b3af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:04.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.964+0000 7fb301d35700 1 -- 172.21.15.16:0/3150623832 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2fc0d0600 con 0x7fb2fc06aef0 2023-10-16T12:27:04.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.964+0000 7fb301d35700 1 --2- 172.21.15.16:0/3150623832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2fc06aef0 0x7fb2fc0b35b0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7fb2ec0098f0 tx=0x7fb2ec009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9df77d24ce968946 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:04.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.964+0000 7fb301534700 1 -- 172.21.15.16:0/3150623832 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2ec00b8a0 con 0x7fb2fc06aef0 2023-10-16T12:27:04.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.965+0000 7fb301534700 1 -- 172.21.15.16:0/3150623832 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2ec00ba00 con 0x7fb2fc06aef0 2023-10-16T12:27:04.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.965+0000 7fb301534700 1 -- 172.21.15.16:0/3150623832 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2ec00bd90 con 0x7fb2fc06aef0 2023-10-16T12:27:04.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.965+0000 7fb30479a700 1 -- 172.21.15.16:0/3150623832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2fc06aef0 msgr2=0x7fb2fc0b35b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:04.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.965+0000 7fb30479a700 1 --2- 172.21.15.16:0/3150623832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2fc06aef0 0x7fb2fc0b35b0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7fb2ec0098f0 tx=0x7fb2ec009c00 comp rx=0 tx=0).stop 2023-10-16T12:27:04.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.965+0000 7fb30479a700 1 -- 172.21.15.16:0/3150623832 shutdown_connections 2023-10-16T12:27:04.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.965+0000 7fb30479a700 1 --2- 172.21.15.16:0/3150623832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2fc06aef0 0x7fb2fc0b35b0 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:04.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.965+0000 7fb30479a700 1 -- 172.21.15.16:0/3150623832 >> 172.21.15.16:0/3150623832 conn(0x7fb2fc0b1fd0 msgr2=0x7fb2fc0b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:04.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.965+0000 7fb30479a700 1 -- 172.21.15.16:0/3150623832 shutdown_connections 2023-10-16T12:27:04.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.965+0000 7fb30479a700 1 -- 172.21.15.16:0/3150623832 wait complete. 2023-10-16T12:27:04.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.967+0000 7fb30479a700 1 Processor -- start 2023-10-16T12:27:04.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.967+0000 7fb30479a700 1 -- start start 2023-10-16T12:27:04.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.968+0000 7fb30479a700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb2fc06aef0 0x7fb2fc14d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:04.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.968+0000 7fb30479a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb2fc0b2be0 0x7fb2fc14b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:04.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.968+0000 7fb30479a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2fc15b760 0x7fb2fc149f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:04.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.968+0000 7fb30479a700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2fc14b780 con 0x7fb2fc15b760 2023-10-16T12:27:04.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.968+0000 7fb30479a700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2fc14b9b0 con 0x7fb2fc06aef0 2023-10-16T12:27:04.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.968+0000 7fb30479a700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2fc14bc90 con 0x7fb2fc0b2be0 2023-10-16T12:27:04.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.968+0000 7fb302d37700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2fc15b760 0x7fb2fc149f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:04.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.968+0000 7fb302d37700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2fc15b760 0x7fb2fc149f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40910/0 (socket says 172.21.15.16:40910) 2023-10-16T12:27:04.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.968+0000 7fb302d37700 1 -- 172.21.15.16:0/820449770 learned_addr learned my addr 172.21.15.16:0/820449770 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:04.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.968+0000 7fb301d35700 1 --2- 172.21.15.16:0/820449770 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb2fc0b2be0 0x7fb2fc14b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:04.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.968+0000 7fb302536700 1 --2- 172.21.15.16:0/820449770 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb2fc06aef0 0x7fb2fc14d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:04.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.968+0000 7fb302d37700 1 -- 172.21.15.16:0/820449770 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb2fc0b2be0 msgr2=0x7fb2fc14b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:04.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.968+0000 7fb302d37700 1 --2- 172.21.15.16:0/820449770 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb2fc0b2be0 0x7fb2fc14b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:04.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.968+0000 7fb302d37700 1 -- 172.21.15.16:0/820449770 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb2fc06aef0 msgr2=0x7fb2fc14d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:04.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.968+0000 7fb302d37700 1 --2- 172.21.15.16:0/820449770 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb2fc06aef0 0x7fb2fc14d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:04.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.968+0000 7fb302d37700 1 -- 172.21.15.16:0/820449770 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2ec009550 con 0x7fb2fc15b760 2023-10-16T12:27:04.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.968+0000 7fb302d37700 1 --2- 172.21.15.16:0/820449770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2fc15b760 0x7fb2fc149f90 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7fb2f400cbc0 tx=0x7fb2f400cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:04.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.968+0000 7fb2f2ffd700 1 -- 172.21.15.16:0/820449770 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2f400ab50 con 0x7fb2fc15b760 2023-10-16T12:27:04.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.969+0000 7fb2f2ffd700 1 -- 172.21.15.16:0/820449770 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2f400acb0 con 0x7fb2fc15b760 2023-10-16T12:27:04.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.969+0000 7fb2f2ffd700 1 -- 172.21.15.16:0/820449770 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2f4007270 con 0x7fb2fc15b760 2023-10-16T12:27:04.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.969+0000 7fb30479a700 1 -- 172.21.15.16:0/820449770 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2fc14bf70 con 0x7fb2fc15b760 2023-10-16T12:27:04.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.969+0000 7fb30479a700 1 -- 172.21.15.16:0/820449770 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2fc14c530 con 0x7fb2fc15b760 2023-10-16T12:27:04.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.969+0000 7fb30479a700 1 -- 172.21.15.16:0/820449770 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2fc04ebb0 con 0x7fb2fc15b760 2023-10-16T12:27:04.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.970+0000 7fb2f2ffd700 1 -- 172.21.15.16:0/820449770 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 41) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb2f40073d0 con 0x7fb2fc15b760 2023-10-16T12:27:04.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.970+0000 7fb2f2ffd700 1 --2- 172.21.15.16:0/820449770 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7fb2e006b020 0x7fb2e006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:04.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.970+0000 7fb2f2ffd700 1 -- 172.21.15.16:0/820449770 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fb2f4089560 con 0x7fb2fc15b760 2023-10-16T12:27:04.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.970+0000 7fb302536700 1 --2- 172.21.15.16:0/820449770 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7fb2e006b020 0x7fb2e006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:04.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.971+0000 7fb302536700 1 --2- 172.21.15.16:0/820449770 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7fb2e006b020 0x7fb2e006d4e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb2f8000f60 tx=0x7fb2f8009040 comp rx=0 tx=0).ready entity=mgr.6020 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:04.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:04.975+0000 7fb2f2ffd700 1 -- 172.21.15.16:0/820449770 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb2f4054450 con 0x7fb2fc15b760 2023-10-16T12:27:05.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.180+0000 7fb30479a700 1 -- 172.21.15.16:0/820449770 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb2fc0caee0 con 0x7fb2fc15b760 2023-10-16T12:27:05.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.182+0000 7fb2f2ffd700 1 -- 172.21.15.16:0/820449770 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) v1 ==== 74+0+16712 (secure 0 0 0) 0x7fb2f4057a70 con 0x7fb2fc15b760 2023-10-16T12:27:05.184 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:27:05.184 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":99,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:27:02.246140+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:26:59.206630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2023-10-16T12:26:59.784585+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:27:05.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.184+0000 7fb30479a700 1 -- 172.21.15.16:0/820449770 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7fb2e006b020 msgr2=0x7fb2e006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:05.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.184+0000 7fb30479a700 1 --2- 172.21.15.16:0/820449770 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7fb2e006b020 0x7fb2e006d4e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb2f8000f60 tx=0x7fb2f8009040 comp rx=0 tx=0).stop 2023-10-16T12:27:05.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.184+0000 7fb30479a700 1 -- 172.21.15.16:0/820449770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2fc15b760 msgr2=0x7fb2fc149f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:05.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.184+0000 7fb30479a700 1 --2- 172.21.15.16:0/820449770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2fc15b760 0x7fb2fc149f90 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7fb2f400cbc0 tx=0x7fb2f400cf80 comp rx=0 tx=0).stop 2023-10-16T12:27:05.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.184+0000 7fb30479a700 1 -- 172.21.15.16:0/820449770 shutdown_connections 2023-10-16T12:27:05.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.184+0000 7fb30479a700 1 --2- 172.21.15.16:0/820449770 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7fb2e006b020 0x7fb2e006d4e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:05.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.184+0000 7fb30479a700 1 --2- 172.21.15.16:0/820449770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2fc15b760 0x7fb2fc149f90 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:05.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.184+0000 7fb30479a700 1 --2- 172.21.15.16:0/820449770 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb2fc06aef0 0x7fb2fc14d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:05.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.184+0000 7fb30479a700 1 --2- 172.21.15.16:0/820449770 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb2fc0b2be0 0x7fb2fc14b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:05.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.184+0000 7fb30479a700 1 -- 172.21.15.16:0/820449770 >> 172.21.15.16:0/820449770 conn(0x7fb2fc0b1fd0 msgr2=0x7fb2fc15e6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:05.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.184+0000 7fb30479a700 1 -- 172.21.15.16:0/820449770 shutdown_connections 2023-10-16T12:27:05.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.184+0000 7fb30479a700 1 -- 172.21.15.16:0/820449770 wait complete. 2023-10-16T12:27:05.198 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-16T12:27:05.199 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T12:27:05.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.322+0000 7f5f47b6a700 1 Processor -- start 2023-10-16T12:27:05.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.322+0000 7f5f47b6a700 1 -- start start 2023-10-16T12:27:05.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.323+0000 7f5f47b6a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5f400ce870 0x7f5f400cb980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:05.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.323+0000 7f5f47b6a700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f400cedb0 con 0x7f5f4006aef0 2023-10-16T12:27:05.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.323+0000 7f5f47b6a700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f400d12c0 con 0x7f5f4015db40 2023-10-16T12:27:05.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.323+0000 7f5f47b6a700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f4015df20 con 0x7f5f400ce870 2023-10-16T12:27:05.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.323+0000 7f5f45105700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f5f4006aef0 0x7f5f400ce160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:58422/0 (socket says 172.21.15.16:58422) 2023-10-16T12:27:05.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.323+0000 7f5f45105700 1 -- 172.21.15.16:0/2539236666 learned_addr learned my addr 172.21.15.16:0/2539236666 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:27:05.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.323+0000 7f5f46107700 1 --2- 172.21.15.16:0/2539236666 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5f400ce870 0x7f5f400cb980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:05.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.323+0000 7f5f44904700 1 -- 172.21.15.16:0/2539236666 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2681458520 0 0) 0x7f5f400cedb0 con 0x7f5f4006aef0 2023-10-16T12:27:05.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.323+0000 7f5f44904700 1 -- 172.21.15.16:0/2539236666 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f2c0035b0 con 0x7f5f4006aef0 2023-10-16T12:27:05.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.324+0000 7f5f44904700 1 -- 172.21.15.16:0/2539236666 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1804959215 0 0) 0x7f5f400d12c0 con 0x7f5f4015db40 2023-10-16T12:27:05.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.324+0000 7f5f44904700 1 -- 172.21.15.16:0/2539236666 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f400cedb0 con 0x7f5f4015db40 2023-10-16T12:27:05.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.324+0000 7f5f44904700 1 -- 172.21.15.16:0/2539236666 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3814085450 0 0) 0x7f5f2c0035b0 con 0x7f5f4006aef0 2023-10-16T12:27:05.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.324+0000 7f5f46107700 1 -- 172.21.15.16:0/2539236666 >> v1:172.21.15.49:6789/0 conn(0x7f5f4015db40 legacy=0x7f5f4006a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:05.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.324+0000 7f5f46107700 1 -- 172.21.15.16:0/2539236666 >> v1:172.21.15.16:6789/0 conn(0x7f5f4006aef0 legacy=0x7f5f400ce160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:05.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.324+0000 7f5f46107700 1 -- 172.21.15.16:0/2539236666 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f400cabc0 con 0x7f5f400ce870 2023-10-16T12:27:05.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.324+0000 7f5f46107700 1 --2- 172.21.15.16:0/2539236666 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5f400ce870 0x7f5f400cb980 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f5f38009dc0 tx=0x7f5f3800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1bc9977e30cbe439 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:05.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.324+0000 7f5f44904700 1 -- 172.21.15.16:0/2539236666 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f3800e070 con 0x7f5f400ce870 2023-10-16T12:27:05.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.324+0000 7f5f44904700 1 -- 172.21.15.16:0/2539236666 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f38003a20 con 0x7f5f400ce870 2023-10-16T12:27:05.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.324+0000 7f5f44904700 1 -- 172.21.15.16:0/2539236666 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f3c001670 con 0x7f5f400ce870 2023-10-16T12:27:05.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.325+0000 7f5f47b6a700 1 -- 172.21.15.16:0/2539236666 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5f400ce870 msgr2=0x7f5f400cb980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:05.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.325+0000 7f5f47b6a700 1 --2- 172.21.15.16:0/2539236666 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5f400ce870 0x7f5f400cb980 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f5f38009dc0 tx=0x7f5f3800b330 comp rx=0 tx=0).stop 2023-10-16T12:27:05.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.325+0000 7f5f47b6a700 1 -- 172.21.15.16:0/2539236666 shutdown_connections 2023-10-16T12:27:05.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.325+0000 7f5f47b6a700 1 --2- 172.21.15.16:0/2539236666 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5f400ce870 0x7f5f400cb980 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:05.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.325+0000 7f5f47b6a700 1 -- 172.21.15.16:0/2539236666 >> 172.21.15.16:0/2539236666 conn(0x7f5f400b1fd0 msgr2=0x7f5f400b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:05.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.325+0000 7f5f47b6a700 1 -- 172.21.15.16:0/2539236666 shutdown_connections 2023-10-16T12:27:05.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.325+0000 7f5f47b6a700 1 -- 172.21.15.16:0/2539236666 wait complete. 2023-10-16T12:27:05.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.326+0000 7f5f47b6a700 1 Processor -- start 2023-10-16T12:27:05.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.326+0000 7f5f47b6a700 1 -- start start 2023-10-16T12:27:05.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.327+0000 7f5f47b6a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f4006aef0 0x7f5f400c63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:05.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.327+0000 7f5f47b6a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5f400ce870 0x7f5f40156e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:05.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.327+0000 7f5f47b6a700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5f4015db40 0x7f5f400c8560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:05.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.327+0000 7f5f47b6a700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f400c4410 con 0x7f5f4006aef0 2023-10-16T12:27:05.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.327+0000 7f5f47b6a700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f400c9500 con 0x7f5f4015db40 2023-10-16T12:27:05.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.327+0000 7f5f47b6a700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f400c8f40 con 0x7f5f400ce870 2023-10-16T12:27:05.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.327+0000 7f5f45906700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f4006aef0 0x7f5f400c63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:05.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.327+0000 7f5f45906700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f4006aef0 0x7f5f400c63a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40920/0 (socket says 172.21.15.16:40920) 2023-10-16T12:27:05.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.327+0000 7f5f45906700 1 -- 172.21.15.16:0/3358444234 learned_addr learned my addr 172.21.15.16:0/3358444234 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:05.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.327+0000 7f5f45105700 1 --2- 172.21.15.16:0/3358444234 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5f400ce870 0x7f5f40156e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:05.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.327+0000 7f5f46107700 1 --2- 172.21.15.16:0/3358444234 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5f4015db40 0x7f5f400c8560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:05.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.327+0000 7f5f45906700 1 -- 172.21.15.16:0/3358444234 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5f400ce870 msgr2=0x7f5f40156e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:05.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.327+0000 7f5f45906700 1 --2- 172.21.15.16:0/3358444234 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5f400ce870 0x7f5f40156e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:05.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.327+0000 7f5f45906700 1 -- 172.21.15.16:0/3358444234 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5f4015db40 msgr2=0x7f5f400c8560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:05.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.327+0000 7f5f45906700 1 --2- 172.21.15.16:0/3358444234 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5f4015db40 0x7f5f400c8560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:05.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.327+0000 7f5f45906700 1 -- 172.21.15.16:0/3358444234 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f38009a70 con 0x7f5f4006aef0 2023-10-16T12:27:05.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.327+0000 7f5f45906700 1 --2- 172.21.15.16:0/3358444234 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f4006aef0 0x7f5f400c63a0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f5f3000c520 tx=0x7f5f3000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:05.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.327+0000 7f5f367fc700 1 -- 172.21.15.16:0/3358444234 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f300052d0 con 0x7f5f4006aef0 2023-10-16T12:27:05.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.327+0000 7f5f367fc700 1 -- 172.21.15.16:0/3358444234 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f30005430 con 0x7f5f4006aef0 2023-10-16T12:27:05.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.327+0000 7f5f47b6a700 1 -- 172.21.15.16:0/3358444234 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f400c9220 con 0x7f5f4006aef0 2023-10-16T12:27:05.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.327+0000 7f5f367fc700 1 -- 172.21.15.16:0/3358444234 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f300038c0 con 0x7f5f4006aef0 2023-10-16T12:27:05.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.327+0000 7f5f47b6a700 1 -- 172.21.15.16:0/3358444234 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f400c68e0 con 0x7f5f4006aef0 2023-10-16T12:27:05.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.328+0000 7f5f47b6a700 1 -- 172.21.15.16:0/3358444234 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f4004ebb0 con 0x7f5f4006aef0 2023-10-16T12:27:05.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.329+0000 7f5f367fc700 1 -- 172.21.15.16:0/3358444234 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 41) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5f30003a20 con 0x7f5f4006aef0 2023-10-16T12:27:05.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.329+0000 7f5f367fc700 1 --2- 172.21.15.16:0/3358444234 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f5f2406b020 0x7f5f2406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:05.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.329+0000 7f5f367fc700 1 -- 172.21.15.16:0/3358444234 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f5f30088cf0 con 0x7f5f4006aef0 2023-10-16T12:27:05.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.329+0000 7f5f45105700 1 --2- 172.21.15.16:0/3358444234 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f5f2406b020 0x7f5f2406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:05.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.330+0000 7f5f45105700 1 --2- 172.21.15.16:0/3358444234 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f5f2406b020 0x7f5f2406d4e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5f3c007e40 tx=0x7f5f3c007dd0 comp rx=0 tx=0).ready entity=mgr.6020 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:05.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.334+0000 7f5f367fc700 1 -- 172.21.15.16:0/3358444234 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5f3008d070 con 0x7f5f4006aef0 2023-10-16T12:27:05.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.534+0000 7f5f47b6a700 1 -- 172.21.15.16:0/3358444234 --> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5f400cb180 con 0x7f5f2406b020 2023-10-16T12:27:05.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.543+0000 7f5f367fc700 1 -- 172.21.15.16:0/3358444234 <== mgr.6020 v2:172.21.15.16:6832/292891612 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289547 (secure 0 0 0) 0x7f5f3c004af0 con 0x7f5f2406b020 2023-10-16T12:27:05.549 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:27:05.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.547+0000 7f5f47b6a700 1 -- 172.21.15.16:0/3358444234 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f5f2406b020 msgr2=0x7f5f2406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:05.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.547+0000 7f5f47b6a700 1 --2- 172.21.15.16:0/3358444234 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f5f2406b020 0x7f5f2406d4e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5f3c007e40 tx=0x7f5f3c007dd0 comp rx=0 tx=0).stop 2023-10-16T12:27:05.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.547+0000 7f5f47b6a700 1 -- 172.21.15.16:0/3358444234 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f4006aef0 msgr2=0x7f5f400c63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:05.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.547+0000 7f5f47b6a700 1 --2- 172.21.15.16:0/3358444234 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f4006aef0 0x7f5f400c63a0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f5f3000c520 tx=0x7f5f3000c830 comp rx=0 tx=0).stop 2023-10-16T12:27:05.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.547+0000 7f5f47b6a700 1 -- 172.21.15.16:0/3358444234 shutdown_connections 2023-10-16T12:27:05.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.547+0000 7f5f47b6a700 1 --2- 172.21.15.16:0/3358444234 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f5f2406b020 0x7f5f2406d4e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:05.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.547+0000 7f5f47b6a700 1 --2- 172.21.15.16:0/3358444234 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5f4015db40 0x7f5f400c8560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:05.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.547+0000 7f5f47b6a700 1 --2- 172.21.15.16:0/3358444234 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f4006aef0 0x7f5f400c63a0 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:05.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.547+0000 7f5f47b6a700 1 --2- 172.21.15.16:0/3358444234 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5f400ce870 0x7f5f40156e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:05.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.547+0000 7f5f47b6a700 1 -- 172.21.15.16:0/3358444234 >> 172.21.15.16:0/3358444234 conn(0x7f5f400b1fd0 msgr2=0x7f5f4006a7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:05.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.547+0000 7f5f47b6a700 1 -- 172.21.15.16:0/3358444234 shutdown_connections 2023-10-16T12:27:05.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.547+0000 7f5f47b6a700 1 -- 172.21.15.16:0/3358444234 wait complete. 2023-10-16T12:27:05.554 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-10-16T12:27:05.561 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-10-16T12:27:04.329426+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":596,"num_read_kb":1130,"num_write":303,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":210,"ondisk_log_size":210,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":70,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":343336,"kb_used_data":10600,"kb_used_omap":12,"kb_used_meta":332595,"kb_avail":754631384,"statfs":{"total":773094113280,"available":772742537216,"internally_reserved":0,"allocated":10854400,"data_stored":5981820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12740,"internal_metadata":340577852},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.941673"},"pg_stats":[{"pgid":"12.3f","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.271907+0000","last_change":"2023-10-16T12:27:01.271233+0000","last_active":"2023-10-16T12:27:01.271907+0000","last_peered":"2023-10-16T12:27:01.271907+0000","last_clean":"2023-10-16T12:27:01.271907+0000","last_became_active":"2023-10-16T12:27:01.270778+0000","last_became_peered":"2023-10-16T12:27:01.270778+0000","last_unstale":"2023-10-16T12:27:01.271907+0000","last_undegraded":"2023-10-16T12:27:01.271907+0000","last_fullsized":"2023-10-16T12:27:01.271907+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:23:13.033596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.272194+0000","last_change":"2023-10-16T12:27:01.270267+0000","last_active":"2023-10-16T12:27:01.272194+0000","last_peered":"2023-10-16T12:27:01.272194+0000","last_clean":"2023-10-16T12:27:01.272194+0000","last_became_active":"2023-10-16T12:27:01.269957+0000","last_became_peered":"2023-10-16T12:27:01.269957+0000","last_unstale":"2023-10-16T12:27:01.272194+0000","last_undegraded":"2023-10-16T12:27:01.272194+0000","last_fullsized":"2023-10-16T12:27:01.272194+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:57:06.277568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.267221+0000","last_change":"2023-10-16T12:27:01.263515+0000","last_active":"2023-10-16T12:27:01.267221+0000","last_peered":"2023-10-16T12:27:01.267221+0000","last_clean":"2023-10-16T12:27:01.267221+0000","last_became_active":"2023-10-16T12:27:01.263147+0000","last_became_peered":"2023-10-16T12:27:01.263147+0000","last_unstale":"2023-10-16T12:27:01.267221+0000","last_undegraded":"2023-10-16T12:27:01.267221+0000","last_fullsized":"2023-10-16T12:27:01.267221+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:02:29.598278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.267875+0000","last_change":"2023-10-16T12:27:01.264999+0000","last_active":"2023-10-16T12:27:01.267875+0000","last_peered":"2023-10-16T12:27:01.267875+0000","last_clean":"2023-10-16T12:27:01.267875+0000","last_became_active":"2023-10-16T12:27:01.264258+0000","last_became_peered":"2023-10-16T12:27:01.264258+0000","last_unstale":"2023-10-16T12:27:01.267875+0000","last_undegraded":"2023-10-16T12:27:01.267875+0000","last_fullsized":"2023-10-16T12:27:01.267875+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:36:42.760644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:27:02.249323+0000","last_change":"2023-10-16T12:27:01.268966+0000","last_active":"2023-10-16T12:27:02.249323+0000","last_peered":"2023-10-16T12:27:02.249323+0000","last_clean":"2023-10-16T12:27:02.249323+0000","last_became_active":"2023-10-16T12:27:01.268630+0000","last_became_peered":"2023-10-16T12:27:01.268630+0000","last_unstale":"2023-10-16T12:27:02.249323+0000","last_undegraded":"2023-10-16T12:27:02.249323+0000","last_fullsized":"2023-10-16T12:27:02.249323+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:23:25.683414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:27:02.250838+0000","last_change":"2023-10-16T12:27:01.265337+0000","last_active":"2023-10-16T12:27:02.250838+0000","last_peered":"2023-10-16T12:27:02.250838+0000","last_clean":"2023-10-16T12:27:02.250838+0000","last_became_active":"2023-10-16T12:27:01.264574+0000","last_became_peered":"2023-10-16T12:27:01.264574+0000","last_unstale":"2023-10-16T12:27:02.250838+0000","last_undegraded":"2023-10-16T12:27:02.250838+0000","last_fullsized":"2023-10-16T12:27:02.250838+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:35:14.191443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.272228+0000","last_change":"2023-10-16T12:27:01.270411+0000","last_active":"2023-10-16T12:27:01.272228+0000","last_peered":"2023-10-16T12:27:01.272228+0000","last_clean":"2023-10-16T12:27:01.272228+0000","last_became_active":"2023-10-16T12:27:01.270119+0000","last_became_peered":"2023-10-16T12:27:01.270119+0000","last_unstale":"2023-10-16T12:27:01.272228+0000","last_undegraded":"2023-10-16T12:27:01.272228+0000","last_fullsized":"2023-10-16T12:27:01.272228+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:12:06.021778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:27:02.249153+0000","last_change":"2023-10-16T12:27:01.260934+0000","last_active":"2023-10-16T12:27:02.249153+0000","last_peered":"2023-10-16T12:27:02.249153+0000","last_clean":"2023-10-16T12:27:02.249153+0000","last_became_active":"2023-10-16T12:27:01.260649+0000","last_became_peered":"2023-10-16T12:27:01.260649+0000","last_unstale":"2023-10-16T12:27:02.249153+0000","last_undegraded":"2023-10-16T12:27:02.249153+0000","last_fullsized":"2023-10-16T12:27:02.249153+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:56:44.663382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.270533+0000","last_change":"2023-10-16T12:27:01.261518+0000","last_active":"2023-10-16T12:27:01.270533+0000","last_peered":"2023-10-16T12:27:01.270533+0000","last_clean":"2023-10-16T12:27:01.270533+0000","last_became_active":"2023-10-16T12:27:01.261019+0000","last_became_peered":"2023-10-16T12:27:01.261019+0000","last_unstale":"2023-10-16T12:27:01.270533+0000","last_undegraded":"2023-10-16T12:27:01.270533+0000","last_fullsized":"2023-10-16T12:27:01.270533+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:02:29.598278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.265919+0000","last_change":"2023-10-16T12:27:01.263781+0000","last_active":"2023-10-16T12:27:01.265919+0000","last_peered":"2023-10-16T12:27:01.265919+0000","last_clean":"2023-10-16T12:27:01.265919+0000","last_became_active":"2023-10-16T12:27:01.263479+0000","last_became_peered":"2023-10-16T12:27:01.263479+0000","last_unstale":"2023-10-16T12:27:01.265919+0000","last_undegraded":"2023-10-16T12:27:01.265919+0000","last_fullsized":"2023-10-16T12:27:01.265919+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:38:16.102632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.272706+0000","last_change":"2023-10-16T12:27:01.271026+0000","last_active":"2023-10-16T12:27:01.272706+0000","last_peered":"2023-10-16T12:27:01.272706+0000","last_clean":"2023-10-16T12:27:01.272706+0000","last_became_active":"2023-10-16T12:27:01.270554+0000","last_became_peered":"2023-10-16T12:27:01.270554+0000","last_unstale":"2023-10-16T12:27:01.272706+0000","last_undegraded":"2023-10-16T12:27:01.272706+0000","last_fullsized":"2023-10-16T12:27:01.272706+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:36:09.450050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.264875+0000","last_change":"2023-10-16T12:27:01.261928+0000","last_active":"2023-10-16T12:27:01.264875+0000","last_peered":"2023-10-16T12:27:01.264875+0000","last_clean":"2023-10-16T12:27:01.264875+0000","last_became_active":"2023-10-16T12:27:01.261505+0000","last_became_peered":"2023-10-16T12:27:01.261505+0000","last_unstale":"2023-10-16T12:27:01.264875+0000","last_undegraded":"2023-10-16T12:27:01.264875+0000","last_fullsized":"2023-10-16T12:27:01.264875+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:09:15.255373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:27:02.249446+0000","last_change":"2023-10-16T12:27:01.266636+0000","last_active":"2023-10-16T12:27:02.249446+0000","last_peered":"2023-10-16T12:27:02.249446+0000","last_clean":"2023-10-16T12:27:02.249446+0000","last_became_active":"2023-10-16T12:27:01.264686+0000","last_became_peered":"2023-10-16T12:27:01.264686+0000","last_unstale":"2023-10-16T12:27:02.249446+0000","last_undegraded":"2023-10-16T12:27:02.249446+0000","last_fullsized":"2023-10-16T12:27:02.249446+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:41:53.749202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:27:02.250700+0000","last_change":"2023-10-16T12:27:01.264860+0000","last_active":"2023-10-16T12:27:02.250700+0000","last_peered":"2023-10-16T12:27:02.250700+0000","last_clean":"2023-10-16T12:27:02.250700+0000","last_became_active":"2023-10-16T12:27:01.263874+0000","last_became_peered":"2023-10-16T12:27:01.263874+0000","last_unstale":"2023-10-16T12:27:02.250700+0000","last_undegraded":"2023-10-16T12:27:02.250700+0000","last_fullsized":"2023-10-16T12:27:02.250700+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:51:29.236436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.269053+0000","last_change":"2023-10-16T12:27:01.265757+0000","last_active":"2023-10-16T12:27:01.269053+0000","last_peered":"2023-10-16T12:27:01.269053+0000","last_clean":"2023-10-16T12:27:01.269053+0000","last_became_active":"2023-10-16T12:27:01.265432+0000","last_became_peered":"2023-10-16T12:27:01.265432+0000","last_unstale":"2023-10-16T12:27:01.269053+0000","last_undegraded":"2023-10-16T12:27:01.269053+0000","last_fullsized":"2023-10-16T12:27:01.269053+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:59:38.006434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.266198+0000","last_change":"2023-10-16T12:27:01.261829+0000","last_active":"2023-10-16T12:27:01.266198+0000","last_peered":"2023-10-16T12:27:01.266198+0000","last_clean":"2023-10-16T12:27:01.266198+0000","last_became_active":"2023-10-16T12:27:01.261535+0000","last_became_peered":"2023-10-16T12:27:01.261535+0000","last_unstale":"2023-10-16T12:27:01.266198+0000","last_undegraded":"2023-10-16T12:27:01.266198+0000","last_fullsized":"2023-10-16T12:27:01.266198+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:19:43.046725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:27:02.249714+0000","last_change":"2023-10-16T12:27:01.269571+0000","last_active":"2023-10-16T12:27:02.249714+0000","last_peered":"2023-10-16T12:27:02.249714+0000","last_clean":"2023-10-16T12:27:02.249714+0000","last_became_active":"2023-10-16T12:27:01.269289+0000","last_became_peered":"2023-10-16T12:27:01.269289+0000","last_unstale":"2023-10-16T12:27:02.249714+0000","last_undegraded":"2023-10-16T12:27:02.249714+0000","last_fullsized":"2023-10-16T12:27:02.249714+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:38:43.851700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.267227+0000","last_change":"2023-10-16T12:27:01.264429+0000","last_active":"2023-10-16T12:27:01.267227+0000","last_peered":"2023-10-16T12:27:01.267227+0000","last_clean":"2023-10-16T12:27:01.267227+0000","last_became_active":"2023-10-16T12:27:01.264105+0000","last_became_peered":"2023-10-16T12:27:01.264105+0000","last_unstale":"2023-10-16T12:27:01.267227+0000","last_undegraded":"2023-10-16T12:27:01.267227+0000","last_fullsized":"2023-10-16T12:27:01.267227+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:41:56.365816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:27:02.250977+0000","last_change":"2023-10-16T12:27:01.269257+0000","last_active":"2023-10-16T12:27:02.250977+0000","last_peered":"2023-10-16T12:27:02.250977+0000","last_clean":"2023-10-16T12:27:02.250977+0000","last_became_active":"2023-10-16T12:27:01.268958+0000","last_became_peered":"2023-10-16T12:27:01.268958+0000","last_unstale":"2023-10-16T12:27:02.250977+0000","last_undegraded":"2023-10-16T12:27:02.250977+0000","last_fullsized":"2023-10-16T12:27:02.250977+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:13:33.750568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:27:02.249634+0000","last_change":"2023-10-16T12:27:01.267116+0000","last_active":"2023-10-16T12:27:02.249634+0000","last_peered":"2023-10-16T12:27:02.249634+0000","last_clean":"2023-10-16T12:27:02.249634+0000","last_became_active":"2023-10-16T12:27:01.265971+0000","last_became_peered":"2023-10-16T12:27:01.265971+0000","last_unstale":"2023-10-16T12:27:02.249634+0000","last_undegraded":"2023-10-16T12:27:02.249634+0000","last_fullsized":"2023-10-16T12:27:02.249634+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:58:38.091004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.270725+0000","last_change":"2023-10-16T12:27:01.266804+0000","last_active":"2023-10-16T12:27:01.270725+0000","last_peered":"2023-10-16T12:27:01.270725+0000","last_clean":"2023-10-16T12:27:01.270725+0000","last_became_active":"2023-10-16T12:27:01.266533+0000","last_became_peered":"2023-10-16T12:27:01.266533+0000","last_unstale":"2023-10-16T12:27:01.270725+0000","last_undegraded":"2023-10-16T12:27:01.270725+0000","last_fullsized":"2023-10-16T12:27:01.270725+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:41:56.365816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:27:02.249495+0000","last_change":"2023-10-16T12:27:01.272204+0000","last_active":"2023-10-16T12:27:02.249495+0000","last_peered":"2023-10-16T12:27:02.249495+0000","last_clean":"2023-10-16T12:27:02.249495+0000","last_became_active":"2023-10-16T12:27:01.271941+0000","last_became_peered":"2023-10-16T12:27:01.271941+0000","last_unstale":"2023-10-16T12:27:02.249495+0000","last_undegraded":"2023-10-16T12:27:02.249495+0000","last_fullsized":"2023-10-16T12:27:02.249495+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:38:41.912039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.269229+0000","last_change":"2023-10-16T12:27:01.261162+0000","last_active":"2023-10-16T12:27:01.269229+0000","last_peered":"2023-10-16T12:27:01.269229+0000","last_clean":"2023-10-16T12:27:01.269229+0000","last_became_active":"2023-10-16T12:27:01.260843+0000","last_became_peered":"2023-10-16T12:27:01.260843+0000","last_unstale":"2023-10-16T12:27:01.269229+0000","last_undegraded":"2023-10-16T12:27:01.269229+0000","last_fullsized":"2023-10-16T12:27:01.269229+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:28:01.320829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.263240+0000","last_change":"2023-10-16T12:27:01.260912+0000","last_active":"2023-10-16T12:27:01.263240+0000","last_peered":"2023-10-16T12:27:01.263240+0000","last_clean":"2023-10-16T12:27:01.263240+0000","last_became_active":"2023-10-16T12:27:01.260625+0000","last_became_peered":"2023-10-16T12:27:01.260625+0000","last_unstale":"2023-10-16T12:27:01.263240+0000","last_undegraded":"2023-10-16T12:27:01.263240+0000","last_fullsized":"2023-10-16T12:27:01.263240+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:45:29.052520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.262437+0000","last_change":"2023-10-16T12:27:01.260078+0000","last_active":"2023-10-16T12:27:01.262437+0000","last_peered":"2023-10-16T12:27:01.262437+0000","last_clean":"2023-10-16T12:27:01.262437+0000","last_became_active":"2023-10-16T12:27:01.259751+0000","last_became_peered":"2023-10-16T12:27:01.259751+0000","last_unstale":"2023-10-16T12:27:01.262437+0000","last_undegraded":"2023-10-16T12:27:01.262437+0000","last_fullsized":"2023-10-16T12:27:01.262437+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:51:34.535878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.261115+0000","last_change":"2023-10-16T12:27:01.259396+0000","last_active":"2023-10-16T12:27:01.261115+0000","last_peered":"2023-10-16T12:27:01.261115+0000","last_clean":"2023-10-16T12:27:01.261115+0000","last_became_active":"2023-10-16T12:27:01.258993+0000","last_became_peered":"2023-10-16T12:27:01.258993+0000","last_unstale":"2023-10-16T12:27:01.261115+0000","last_undegraded":"2023-10-16T12:27:01.261115+0000","last_fullsized":"2023-10-16T12:27:01.261115+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:55:14.108031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.263102+0000","last_change":"2023-10-16T12:27:01.261039+0000","last_active":"2023-10-16T12:27:01.263102+0000","last_peered":"2023-10-16T12:27:01.263102+0000","last_clean":"2023-10-16T12:27:01.263102+0000","last_became_active":"2023-10-16T12:27:01.260755+0000","last_became_peered":"2023-10-16T12:27:01.260755+0000","last_unstale":"2023-10-16T12:27:01.263102+0000","last_undegraded":"2023-10-16T12:27:01.263102+0000","last_fullsized":"2023-10-16T12:27:01.263102+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:50:05.396430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:27:02.249756+0000","last_change":"2023-10-16T12:27:01.265675+0000","last_active":"2023-10-16T12:27:02.249756+0000","last_peered":"2023-10-16T12:27:02.249756+0000","last_clean":"2023-10-16T12:27:02.249756+0000","last_became_active":"2023-10-16T12:27:01.265354+0000","last_became_peered":"2023-10-16T12:27:01.265354+0000","last_unstale":"2023-10-16T12:27:02.249756+0000","last_undegraded":"2023-10-16T12:27:02.249756+0000","last_fullsized":"2023-10-16T12:27:02.249756+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:15:58.232499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.268428+0000","last_change":"2023-10-16T12:27:01.263522+0000","last_active":"2023-10-16T12:27:01.268428+0000","last_peered":"2023-10-16T12:27:01.268428+0000","last_clean":"2023-10-16T12:27:01.268428+0000","last_became_active":"2023-10-16T12:27:01.263175+0000","last_became_peered":"2023-10-16T12:27:01.263175+0000","last_unstale":"2023-10-16T12:27:01.268428+0000","last_undegraded":"2023-10-16T12:27:01.268428+0000","last_fullsized":"2023-10-16T12:27:01.268428+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:34:37.846730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.265597+0000","last_change":"2023-10-16T12:27:01.263563+0000","last_active":"2023-10-16T12:27:01.265597+0000","last_peered":"2023-10-16T12:27:01.265597+0000","last_clean":"2023-10-16T12:27:01.265597+0000","last_became_active":"2023-10-16T12:27:01.263280+0000","last_became_peered":"2023-10-16T12:27:01.263280+0000","last_unstale":"2023-10-16T12:27:01.265597+0000","last_undegraded":"2023-10-16T12:27:01.265597+0000","last_fullsized":"2023-10-16T12:27:01.265597+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:27:27.941593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.272572+0000","last_change":"2023-10-16T12:27:01.271183+0000","last_active":"2023-10-16T12:27:01.272572+0000","last_peered":"2023-10-16T12:27:01.272572+0000","last_clean":"2023-10-16T12:27:01.272572+0000","last_became_active":"2023-10-16T12:27:01.270895+0000","last_became_peered":"2023-10-16T12:27:01.270895+0000","last_unstale":"2023-10-16T12:27:01.272572+0000","last_undegraded":"2023-10-16T12:27:01.272572+0000","last_fullsized":"2023-10-16T12:27:01.272572+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:22:34.967887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.266219+0000","last_change":"2023-10-16T12:27:01.262809+0000","last_active":"2023-10-16T12:27:01.266219+0000","last_peered":"2023-10-16T12:27:01.266219+0000","last_clean":"2023-10-16T12:27:01.266219+0000","last_became_active":"2023-10-16T12:27:01.262457+0000","last_became_peered":"2023-10-16T12:27:01.262457+0000","last_unstale":"2023-10-16T12:27:01.266219+0000","last_undegraded":"2023-10-16T12:27:01.266219+0000","last_fullsized":"2023-10-16T12:27:01.266219+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:20:55.292512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.267462+0000","last_change":"2023-10-16T12:27:01.264439+0000","last_active":"2023-10-16T12:27:01.267462+0000","last_peered":"2023-10-16T12:27:01.267462+0000","last_clean":"2023-10-16T12:27:01.267462+0000","last_became_active":"2023-10-16T12:27:01.264127+0000","last_became_peered":"2023-10-16T12:27:01.264127+0000","last_unstale":"2023-10-16T12:27:01.267462+0000","last_undegraded":"2023-10-16T12:27:01.267462+0000","last_fullsized":"2023-10-16T12:27:01.267462+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:52:02.124855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"98'4","reported_seq":16,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:27:02.249570+0000","last_change":"2023-10-16T12:27:00.406941+0000","last_active":"2023-10-16T12:27:02.249570+0000","last_peered":"2023-10-16T12:27:02.249570+0000","last_clean":"2023-10-16T12:27:02.249570+0000","last_became_active":"2023-10-16T12:27:00.406664+0000","last_became_peered":"2023-10-16T12:27:00.406664+0000","last_unstale":"2023-10-16T12:27:02.249570+0000","last_undegraded":"2023-10-16T12:27:02.249570+0000","last_fullsized":"2023-10-16T12:27:02.249570+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_clean_scrub_stamp":"2023-10-16T12:26:59.382546+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:53:42.971928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.266165+0000","last_change":"2023-10-16T12:27:01.259191+0000","last_active":"2023-10-16T12:27:01.266165+0000","last_peered":"2023-10-16T12:27:01.266165+0000","last_clean":"2023-10-16T12:27:01.266165+0000","last_became_active":"2023-10-16T12:27:01.258896+0000","last_became_peered":"2023-10-16T12:27:01.258896+0000","last_unstale":"2023-10-16T12:27:01.266165+0000","last_undegraded":"2023-10-16T12:27:01.266165+0000","last_fullsized":"2023-10-16T12:27:01.266165+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:27:44.803461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"98'1","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.405359+0000","last_change":"2023-10-16T12:27:00.419326+0000","last_active":"2023-10-16T12:27:01.405359+0000","last_peered":"2023-10-16T12:27:01.405359+0000","last_clean":"2023-10-16T12:27:01.405359+0000","last_became_active":"2023-10-16T12:27:00.418899+0000","last_became_peered":"2023-10-16T12:27:00.418899+0000","last_unstale":"2023-10-16T12:27:01.405359+0000","last_undegraded":"2023-10-16T12:27:01.405359+0000","last_fullsized":"2023-10-16T12:27:01.405359+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_clean_scrub_stamp":"2023-10-16T12:26:59.382546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:10:41.359173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.271195+0000","last_change":"2023-10-16T12:27:01.268591+0000","last_active":"2023-10-16T12:27:01.271195+0000","last_peered":"2023-10-16T12:27:01.271195+0000","last_clean":"2023-10-16T12:27:01.271195+0000","last_became_active":"2023-10-16T12:27:01.267918+0000","last_became_peered":"2023-10-16T12:27:01.267918+0000","last_unstale":"2023-10-16T12:27:01.271195+0000","last_undegraded":"2023-10-16T12:27:01.271195+0000","last_fullsized":"2023-10-16T12:27:01.271195+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:35:50.007900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"98'2","reported_seq":13,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.404599+0000","last_change":"2023-10-16T12:27:00.411306+0000","last_active":"2023-10-16T12:27:01.404599+0000","last_peered":"2023-10-16T12:27:01.404599+0000","last_clean":"2023-10-16T12:27:01.404599+0000","last_became_active":"2023-10-16T12:27:00.411013+0000","last_became_peered":"2023-10-16T12:27:00.411013+0000","last_unstale":"2023-10-16T12:27:01.404599+0000","last_undegraded":"2023-10-16T12:27:01.404599+0000","last_fullsized":"2023-10-16T12:27:01.404599+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_clean_scrub_stamp":"2023-10-16T12:26:59.382546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:34:36.841255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.270703+0000","last_change":"2023-10-16T12:27:01.266535+0000","last_active":"2023-10-16T12:27:01.270703+0000","last_peered":"2023-10-16T12:27:01.270703+0000","last_clean":"2023-10-16T12:27:01.270703+0000","last_became_active":"2023-10-16T12:27:01.266233+0000","last_became_peered":"2023-10-16T12:27:01.266233+0000","last_unstale":"2023-10-16T12:27:01.270703+0000","last_undegraded":"2023-10-16T12:27:01.270703+0000","last_fullsized":"2023-10-16T12:27:01.270703+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:10:22.284373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"98'2","reported_seq":13,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.406134+0000","last_change":"2023-10-16T12:27:00.416224+0000","last_active":"2023-10-16T12:27:01.406134+0000","last_peered":"2023-10-16T12:27:01.406134+0000","last_clean":"2023-10-16T12:27:01.406134+0000","last_became_active":"2023-10-16T12:27:00.415718+0000","last_became_peered":"2023-10-16T12:27:00.415718+0000","last_unstale":"2023-10-16T12:27:01.406134+0000","last_undegraded":"2023-10-16T12:27:01.406134+0000","last_fullsized":"2023-10-16T12:27:01.406134+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_clean_scrub_stamp":"2023-10-16T12:26:59.382546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:35:49.002426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.272561+0000","last_change":"2023-10-16T12:27:01.270883+0000","last_active":"2023-10-16T12:27:01.272561+0000","last_peered":"2023-10-16T12:27:01.272561+0000","last_clean":"2023-10-16T12:27:01.272561+0000","last_became_active":"2023-10-16T12:27:01.270584+0000","last_became_peered":"2023-10-16T12:27:01.270584+0000","last_unstale":"2023-10-16T12:27:01.272561+0000","last_undegraded":"2023-10-16T12:27:01.272561+0000","last_fullsized":"2023-10-16T12:27:01.272561+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:15:40.122075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"98'1","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.406899+0000","last_change":"2023-10-16T12:27:00.410815+0000","last_active":"2023-10-16T12:27:01.406899+0000","last_peered":"2023-10-16T12:27:01.406899+0000","last_clean":"2023-10-16T12:27:01.406899+0000","last_became_active":"2023-10-16T12:27:00.410518+0000","last_became_peered":"2023-10-16T12:27:00.410518+0000","last_unstale":"2023-10-16T12:27:01.406899+0000","last_undegraded":"2023-10-16T12:27:01.406899+0000","last_fullsized":"2023-10-16T12:27:01.406899+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_clean_scrub_stamp":"2023-10-16T12:26:59.382546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:52:01.119381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.272686+0000","last_change":"2023-10-16T12:27:01.270848+0000","last_active":"2023-10-16T12:27:01.272686+0000","last_peered":"2023-10-16T12:27:01.272686+0000","last_clean":"2023-10-16T12:27:01.272686+0000","last_became_active":"2023-10-16T12:27:01.270061+0000","last_became_peered":"2023-10-16T12:27:01.270061+0000","last_unstale":"2023-10-16T12:27:01.272686+0000","last_undegraded":"2023-10-16T12:27:01.272686+0000","last_fullsized":"2023-10-16T12:27:01.272686+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:05:11.575127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"98'1","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.407682+0000","last_change":"2023-10-16T12:27:00.412976+0000","last_active":"2023-10-16T12:27:01.407682+0000","last_peered":"2023-10-16T12:27:01.407682+0000","last_clean":"2023-10-16T12:27:01.407682+0000","last_became_active":"2023-10-16T12:27:00.412553+0000","last_became_peered":"2023-10-16T12:27:00.412553+0000","last_unstale":"2023-10-16T12:27:01.407682+0000","last_undegraded":"2023-10-16T12:27:01.407682+0000","last_fullsized":"2023-10-16T12:27:01.407682+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_clean_scrub_stamp":"2023-10-16T12:26:59.382546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:10:59.683263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.268702+0000","last_change":"2023-10-16T12:27:01.259677+0000","last_active":"2023-10-16T12:27:01.268702+0000","last_peered":"2023-10-16T12:27:01.268702+0000","last_clean":"2023-10-16T12:27:01.268702+0000","last_became_active":"2023-10-16T12:27:01.259405+0000","last_became_peered":"2023-10-16T12:27:01.259405+0000","last_unstale":"2023-10-16T12:27:01.268702+0000","last_undegraded":"2023-10-16T12:27:01.268702+0000","last_fullsized":"2023-10-16T12:27:01.268702+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:51:29.215938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.248646+0000","last_change":"2023-10-16T12:27:00.407890+0000","last_active":"2023-10-16T12:27:01.248646+0000","last_peered":"2023-10-16T12:27:01.248646+0000","last_clean":"2023-10-16T12:27:01.248646+0000","last_became_active":"2023-10-16T12:27:00.407608+0000","last_became_peered":"2023-10-16T12:27:00.407608+0000","last_unstale":"2023-10-16T12:27:01.248646+0000","last_undegraded":"2023-10-16T12:27:01.248646+0000","last_fullsized":"2023-10-16T12:27:01.248646+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_clean_scrub_stamp":"2023-10-16T12:26:59.382546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:41:50.717258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.274937+0000","last_change":"2023-10-16T12:27:01.274537+0000","last_active":"2023-10-16T12:27:01.274937+0000","last_peered":"2023-10-16T12:27:01.274937+0000","last_clean":"2023-10-16T12:27:01.274937+0000","last_became_active":"2023-10-16T12:27:01.274290+0000","last_became_peered":"2023-10-16T12:27:01.274290+0000","last_unstale":"2023-10-16T12:27:01.274937+0000","last_undegraded":"2023-10-16T12:27:01.274937+0000","last_fullsized":"2023-10-16T12:27:01.274937+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:51:29.215938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:27:02.249196+0000","last_change":"2023-10-16T12:27:00.411391+0000","last_active":"2023-10-16T12:27:02.249196+0000","last_peered":"2023-10-16T12:27:02.249196+0000","last_clean":"2023-10-16T12:27:02.249196+0000","last_became_active":"2023-10-16T12:27:00.411135+0000","last_became_peered":"2023-10-16T12:27:00.411135+0000","last_unstale":"2023-10-16T12:27:02.249196+0000","last_undegraded":"2023-10-16T12:27:02.249196+0000","last_fullsized":"2023-10-16T12:27:02.249196+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_clean_scrub_stamp":"2023-10-16T12:26:59.382546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:55:25.767744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.263260+0000","last_change":"2023-10-16T12:27:01.262067+0000","last_active":"2023-10-16T12:27:01.263260+0000","last_peered":"2023-10-16T12:27:01.263260+0000","last_clean":"2023-10-16T12:27:01.263260+0000","last_became_active":"2023-10-16T12:27:01.260866+0000","last_became_peered":"2023-10-16T12:27:01.260866+0000","last_unstale":"2023-10-16T12:27:01.263260+0000","last_undegraded":"2023-10-16T12:27:01.263260+0000","last_fullsized":"2023-10-16T12:27:01.263260+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:23:13.033596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.255245+0000","last_change":"2023-10-16T12:27:00.417116+0000","last_active":"2023-10-16T12:27:01.255245+0000","last_peered":"2023-10-16T12:27:01.255245+0000","last_clean":"2023-10-16T12:27:01.255245+0000","last_became_active":"2023-10-16T12:27:00.415892+0000","last_became_peered":"2023-10-16T12:27:00.415892+0000","last_unstale":"2023-10-16T12:27:01.255245+0000","last_undegraded":"2023-10-16T12:27:01.255245+0000","last_fullsized":"2023-10-16T12:27:01.255245+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_clean_scrub_stamp":"2023-10-16T12:26:59.382546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:15:39.116600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.270165+0000","last_change":"2023-10-16T12:27:01.269361+0000","last_active":"2023-10-16T12:27:01.270165+0000","last_peered":"2023-10-16T12:27:01.270165+0000","last_clean":"2023-10-16T12:27:01.270165+0000","last_became_active":"2023-10-16T12:27:01.268996+0000","last_became_peered":"2023-10-16T12:27:01.268996+0000","last_unstale":"2023-10-16T12:27:01.270165+0000","last_undegraded":"2023-10-16T12:27:01.270165+0000","last_fullsized":"2023-10-16T12:27:01.270165+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:15:18.654877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.255044+0000","last_change":"2023-10-16T12:27:00.406244+0000","last_active":"2023-10-16T12:27:01.255044+0000","last_peered":"2023-10-16T12:27:01.255044+0000","last_clean":"2023-10-16T12:27:01.255044+0000","last_became_active":"2023-10-16T12:27:00.405950+0000","last_became_peered":"2023-10-16T12:27:00.405950+0000","last_unstale":"2023-10-16T12:27:01.255044+0000","last_undegraded":"2023-10-16T12:27:01.255044+0000","last_fullsized":"2023-10-16T12:27:01.255044+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_clean_scrub_stamp":"2023-10-16T12:26:59.382546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:10:08.160283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:27:02.250743+0000","last_change":"2023-10-16T12:27:01.265182+0000","last_active":"2023-10-16T12:27:02.250743+0000","last_peered":"2023-10-16T12:27:02.250743+0000","last_clean":"2023-10-16T12:27:02.250743+0000","last_became_active":"2023-10-16T12:27:01.264203+0000","last_became_peered":"2023-10-16T12:27:01.264203+0000","last_unstale":"2023-10-16T12:27:02.250743+0000","last_undegraded":"2023-10-16T12:27:02.250743+0000","last_fullsized":"2023-10-16T12:27:02.250743+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:41:49.065472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"98'3","reported_seq":15,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:27:02.249641+0000","last_change":"2023-10-16T12:27:00.416524+0000","last_active":"2023-10-16T12:27:02.249641+0000","last_peered":"2023-10-16T12:27:02.249641+0000","last_clean":"2023-10-16T12:27:02.249641+0000","last_became_active":"2023-10-16T12:27:00.416242+0000","last_became_peered":"2023-10-16T12:27:00.416242+0000","last_unstale":"2023-10-16T12:27:02.249641+0000","last_undegraded":"2023-10-16T12:27:02.249641+0000","last_fullsized":"2023-10-16T12:27:02.249641+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_clean_scrub_stamp":"2023-10-16T12:26:59.382546+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:04:14.487791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.267682+0000","last_change":"2023-10-16T12:27:01.265279+0000","last_active":"2023-10-16T12:27:01.267682+0000","last_peered":"2023-10-16T12:27:01.267682+0000","last_clean":"2023-10-16T12:27:01.267682+0000","last_became_active":"2023-10-16T12:27:01.264870+0000","last_became_peered":"2023-10-16T12:27:01.264870+0000","last_unstale":"2023-10-16T12:27:01.267682+0000","last_undegraded":"2023-10-16T12:27:01.267682+0000","last_fullsized":"2023-10-16T12:27:01.267682+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:10:22.284373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"98'6","reported_seq":17,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.410422+0000","last_change":"2023-10-16T12:27:00.415413+0000","last_active":"2023-10-16T12:27:01.410422+0000","last_peered":"2023-10-16T12:27:01.410422+0000","last_clean":"2023-10-16T12:27:01.410422+0000","last_became_active":"2023-10-16T12:27:00.413893+0000","last_became_peered":"2023-10-16T12:27:00.413893+0000","last_unstale":"2023-10-16T12:27:01.410422+0000","last_undegraded":"2023-10-16T12:27:01.410422+0000","last_fullsized":"2023-10-16T12:27:01.410422+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_clean_scrub_stamp":"2023-10-16T12:26:59.382546+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:02:28.592803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.267895+0000","last_change":"2023-10-16T12:27:01.265199+0000","last_active":"2023-10-16T12:27:01.267895+0000","last_peered":"2023-10-16T12:27:01.267895+0000","last_clean":"2023-10-16T12:27:01.267895+0000","last_became_active":"2023-10-16T12:27:01.264698+0000","last_became_peered":"2023-10-16T12:27:01.264698+0000","last_unstale":"2023-10-16T12:27:01.267895+0000","last_undegraded":"2023-10-16T12:27:01.267895+0000","last_fullsized":"2023-10-16T12:27:01.267895+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:49:04.902173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"98'2","reported_seq":13,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.409827+0000","last_change":"2023-10-16T12:27:00.411236+0000","last_active":"2023-10-16T12:27:01.409827+0000","last_peered":"2023-10-16T12:27:01.409827+0000","last_clean":"2023-10-16T12:27:01.409827+0000","last_became_active":"2023-10-16T12:27:00.410932+0000","last_became_peered":"2023-10-16T12:27:00.410932+0000","last_unstale":"2023-10-16T12:27:01.409827+0000","last_undegraded":"2023-10-16T12:27:01.409827+0000","last_fullsized":"2023-10-16T12:27:01.409827+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_clean_scrub_stamp":"2023-10-16T12:26:59.382546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:50:04.390955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.271789+0000","last_change":"2023-10-16T12:27:01.269536+0000","last_active":"2023-10-16T12:27:01.271789+0000","last_peered":"2023-10-16T12:27:01.271789+0000","last_clean":"2023-10-16T12:27:01.271789+0000","last_became_active":"2023-10-16T12:27:01.269161+0000","last_became_peered":"2023-10-16T12:27:01.269161+0000","last_unstale":"2023-10-16T12:27:01.271789+0000","last_undegraded":"2023-10-16T12:27:01.271789+0000","last_fullsized":"2023-10-16T12:27:01.271789+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:08:38.228680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.252902+0000","last_change":"2023-10-16T12:27:00.414135+0000","last_active":"2023-10-16T12:27:01.252902+0000","last_peered":"2023-10-16T12:27:01.252902+0000","last_clean":"2023-10-16T12:27:01.252902+0000","last_became_active":"2023-10-16T12:27:00.413815+0000","last_became_peered":"2023-10-16T12:27:00.413815+0000","last_unstale":"2023-10-16T12:27:01.252902+0000","last_undegraded":"2023-10-16T12:27:01.252902+0000","last_fullsized":"2023-10-16T12:27:01.252902+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_clean_scrub_stamp":"2023-10-16T12:26:59.382546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:41:50.717258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.266635+0000","last_change":"2023-10-16T12:27:01.263997+0000","last_active":"2023-10-16T12:27:01.266635+0000","last_peered":"2023-10-16T12:27:01.266635+0000","last_clean":"2023-10-16T12:27:01.266635+0000","last_became_active":"2023-10-16T12:27:01.263703+0000","last_became_peered":"2023-10-16T12:27:01.263703+0000","last_unstale":"2023-10-16T12:27:01.266635+0000","last_undegraded":"2023-10-16T12:27:01.266635+0000","last_fullsized":"2023-10-16T12:27:01.266635+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:52:41.515619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:27:02.249340+0000","last_change":"2023-10-16T12:27:00.409189+0000","last_active":"2023-10-16T12:27:02.249340+0000","last_peered":"2023-10-16T12:27:02.249340+0000","last_clean":"2023-10-16T12:27:02.249340+0000","last_became_active":"2023-10-16T12:27:00.408912+0000","last_became_peered":"2023-10-16T12:27:00.408912+0000","last_unstale":"2023-10-16T12:27:02.249340+0000","last_undegraded":"2023-10-16T12:27:02.249340+0000","last_fullsized":"2023-10-16T12:27:02.249340+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_clean_scrub_stamp":"2023-10-16T12:26:59.382546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:51:31.033168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.269773+0000","last_change":"2023-10-16T12:27:01.265110+0000","last_active":"2023-10-16T12:27:01.269773+0000","last_peered":"2023-10-16T12:27:01.269773+0000","last_clean":"2023-10-16T12:27:01.269773+0000","last_became_active":"2023-10-16T12:27:01.264576+0000","last_became_peered":"2023-10-16T12:27:01.264576+0000","last_unstale":"2023-10-16T12:27:01.269773+0000","last_undegraded":"2023-10-16T12:27:01.269773+0000","last_fullsized":"2023-10-16T12:27:01.269773+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:45:52.773246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"98'8","reported_seq":19,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.408703+0000","last_change":"2023-10-16T12:27:00.419332+0000","last_active":"2023-10-16T12:27:01.408703+0000","last_peered":"2023-10-16T12:27:01.408703+0000","last_clean":"2023-10-16T12:27:01.408703+0000","last_became_active":"2023-10-16T12:27:00.419020+0000","last_became_peered":"2023-10-16T12:27:00.419020+0000","last_unstale":"2023-10-16T12:27:01.408703+0000","last_undegraded":"2023-10-16T12:27:01.408703+0000","last_fullsized":"2023-10-16T12:27:01.408703+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_clean_scrub_stamp":"2023-10-16T12:26:59.382546+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:20:54.287038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:27:02.250428+0000","last_change":"2023-10-16T12:27:01.261823+0000","last_active":"2023-10-16T12:27:02.250428+0000","last_peered":"2023-10-16T12:27:02.250428+0000","last_clean":"2023-10-16T12:27:02.250428+0000","last_became_active":"2023-10-16T12:27:01.261469+0000","last_became_peered":"2023-10-16T12:27:01.261469+0000","last_unstale":"2023-10-16T12:27:02.250428+0000","last_undegraded":"2023-10-16T12:27:02.250428+0000","last_fullsized":"2023-10-16T12:27:02.250428+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:20:32.306289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"98'1","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.404619+0000","last_change":"2023-10-16T12:27:00.416749+0000","last_active":"2023-10-16T12:27:01.404619+0000","last_peered":"2023-10-16T12:27:01.404619+0000","last_clean":"2023-10-16T12:27:01.404619+0000","last_became_active":"2023-10-16T12:27:00.416458+0000","last_became_peered":"2023-10-16T12:27:00.416458+0000","last_unstale":"2023-10-16T12:27:01.404619+0000","last_undegraded":"2023-10-16T12:27:01.404619+0000","last_fullsized":"2023-10-16T12:27:01.404619+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_clean_scrub_stamp":"2023-10-16T12:26:59.382546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:27:43.797986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:27:02.249251+0000","last_change":"2023-10-16T12:27:01.269914+0000","last_active":"2023-10-16T12:27:02.249251+0000","last_peered":"2023-10-16T12:27:02.249251+0000","last_clean":"2023-10-16T12:27:02.249251+0000","last_became_active":"2023-10-16T12:27:01.269570+0000","last_became_peered":"2023-10-16T12:27:01.269570+0000","last_unstale":"2023-10-16T12:27:02.249251+0000","last_undegraded":"2023-10-16T12:27:02.249251+0000","last_fullsized":"2023-10-16T12:27:02.249251+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:38:41.223094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"98'2","reported_seq":13,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.406545+0000","last_change":"2023-10-16T12:27:00.413690+0000","last_active":"2023-10-16T12:27:01.406545+0000","last_peered":"2023-10-16T12:27:01.406545+0000","last_clean":"2023-10-16T12:27:01.406545+0000","last_became_active":"2023-10-16T12:27:00.413411+0000","last_became_peered":"2023-10-16T12:27:00.413411+0000","last_unstale":"2023-10-16T12:27:01.406545+0000","last_undegraded":"2023-10-16T12:27:01.406545+0000","last_fullsized":"2023-10-16T12:27:01.406545+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_clean_scrub_stamp":"2023-10-16T12:26:59.382546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:25:01.715586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:27:02.249353+0000","last_change":"2023-10-16T12:27:01.260408+0000","last_active":"2023-10-16T12:27:02.249353+0000","last_peered":"2023-10-16T12:27:02.249353+0000","last_clean":"2023-10-16T12:27:02.249353+0000","last_became_active":"2023-10-16T12:27:01.260131+0000","last_became_peered":"2023-10-16T12:27:01.260131+0000","last_unstale":"2023-10-16T12:27:02.249353+0000","last_undegraded":"2023-10-16T12:27:02.249353+0000","last_fullsized":"2023-10-16T12:27:02.249353+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:25:08.494549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"98'1","reported_seq":13,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:27:02.249674+0000","last_change":"2023-10-16T12:27:00.411188+0000","last_active":"2023-10-16T12:27:02.249674+0000","last_peered":"2023-10-16T12:27:02.249674+0000","last_clean":"2023-10-16T12:27:02.249674+0000","last_became_active":"2023-10-16T12:27:00.410747+0000","last_became_peered":"2023-10-16T12:27:00.410747+0000","last_unstale":"2023-10-16T12:27:02.249674+0000","last_undegraded":"2023-10-16T12:27:02.249674+0000","last_fullsized":"2023-10-16T12:27:02.249674+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_clean_scrub_stamp":"2023-10-16T12:26:59.382546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:22:25.427269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:27:02.250564+0000","last_change":"2023-10-16T12:27:01.262106+0000","last_active":"2023-10-16T12:27:02.250564+0000","last_peered":"2023-10-16T12:27:02.250564+0000","last_clean":"2023-10-16T12:27:02.250564+0000","last_became_active":"2023-10-16T12:27:01.261598+0000","last_became_peered":"2023-10-16T12:27:01.261598+0000","last_unstale":"2023-10-16T12:27:02.250564+0000","last_undegraded":"2023-10-16T12:27:02.250564+0000","last_fullsized":"2023-10-16T12:27:02.250564+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:46:55.924586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.252747+0000","last_change":"2023-10-16T12:27:00.416466+0000","last_active":"2023-10-16T12:27:01.252747+0000","last_peered":"2023-10-16T12:27:01.252747+0000","last_clean":"2023-10-16T12:27:01.252747+0000","last_became_active":"2023-10-16T12:27:00.416179+0000","last_became_peered":"2023-10-16T12:27:00.416179+0000","last_unstale":"2023-10-16T12:27:01.252747+0000","last_undegraded":"2023-10-16T12:27:01.252747+0000","last_fullsized":"2023-10-16T12:27:01.252747+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_clean_scrub_stamp":"2023-10-16T12:26:59.382546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:35:49.002426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.269359+0000","last_change":"2023-10-16T12:27:01.266417+0000","last_active":"2023-10-16T12:27:01.269359+0000","last_peered":"2023-10-16T12:27:01.269359+0000","last_clean":"2023-10-16T12:27:01.269359+0000","last_became_active":"2023-10-16T12:27:01.265856+0000","last_became_peered":"2023-10-16T12:27:01.265856+0000","last_unstale":"2023-10-16T12:27:01.269359+0000","last_undegraded":"2023-10-16T12:27:01.269359+0000","last_fullsized":"2023-10-16T12:27:01.269359+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:54:00.495639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"98'2","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:27:02.250554+0000","last_change":"2023-10-16T12:27:00.414010+0000","last_active":"2023-10-16T12:27:02.250554+0000","last_peered":"2023-10-16T12:27:02.250554+0000","last_clean":"2023-10-16T12:27:02.250554+0000","last_became_active":"2023-10-16T12:27:00.413709+0000","last_became_peered":"2023-10-16T12:27:00.413709+0000","last_unstale":"2023-10-16T12:27:02.250554+0000","last_undegraded":"2023-10-16T12:27:02.250554+0000","last_fullsized":"2023-10-16T12:27:02.250554+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_clean_scrub_stamp":"2023-10-16T12:26:59.382546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:57:05.272093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.270668+0000","last_change":"2023-10-16T12:27:01.267765+0000","last_active":"2023-10-16T12:27:01.270668+0000","last_peered":"2023-10-16T12:27:01.270668+0000","last_clean":"2023-10-16T12:27:01.270668+0000","last_became_active":"2023-10-16T12:27:01.267456+0000","last_became_peered":"2023-10-16T12:27:01.267456+0000","last_unstale":"2023-10-16T12:27:01.270668+0000","last_undegraded":"2023-10-16T12:27:01.270668+0000","last_fullsized":"2023-10-16T12:27:01.270668+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:46:52.865162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:27:02.250302+0000","last_change":"2023-10-16T12:27:00.420678+0000","last_active":"2023-10-16T12:27:02.250302+0000","last_peered":"2023-10-16T12:27:02.250302+0000","last_clean":"2023-10-16T12:27:02.250302+0000","last_became_active":"2023-10-16T12:27:00.420342+0000","last_became_peered":"2023-10-16T12:27:00.420342+0000","last_unstale":"2023-10-16T12:27:02.250302+0000","last_undegraded":"2023-10-16T12:27:02.250302+0000","last_fullsized":"2023-10-16T12:27:02.250302+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_clean_scrub_stamp":"2023-10-16T12:26:59.382546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:29:03.134564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.269751+0000","last_change":"2023-10-16T12:27:01.264901+0000","last_active":"2023-10-16T12:27:01.269751+0000","last_peered":"2023-10-16T12:27:01.269751+0000","last_clean":"2023-10-16T12:27:01.269751+0000","last_became_active":"2023-10-16T12:27:01.263217+0000","last_became_peered":"2023-10-16T12:27:01.263217+0000","last_unstale":"2023-10-16T12:27:01.269751+0000","last_undegraded":"2023-10-16T12:27:01.269751+0000","last_fullsized":"2023-10-16T12:27:01.269751+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:19:03.776179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:27:02.250516+0000","last_change":"2023-10-16T12:27:00.419553+0000","last_active":"2023-10-16T12:27:02.250516+0000","last_peered":"2023-10-16T12:27:02.250516+0000","last_clean":"2023-10-16T12:27:02.250516+0000","last_became_active":"2023-10-16T12:27:00.419169+0000","last_became_peered":"2023-10-16T12:27:00.419169+0000","last_unstale":"2023-10-16T12:27:02.250516+0000","last_undegraded":"2023-10-16T12:27:02.250516+0000","last_fullsized":"2023-10-16T12:27:02.250516+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_clean_scrub_stamp":"2023-10-16T12:26:59.382546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:53:59.490165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.270603+0000","last_change":"2023-10-16T12:27:01.263865+0000","last_active":"2023-10-16T12:27:01.270603+0000","last_peered":"2023-10-16T12:27:01.270603+0000","last_clean":"2023-10-16T12:27:01.270603+0000","last_became_active":"2023-10-16T12:27:01.263521+0000","last_became_peered":"2023-10-16T12:27:01.263521+0000","last_unstale":"2023-10-16T12:27:01.270603+0000","last_undegraded":"2023-10-16T12:27:01.270603+0000","last_fullsized":"2023-10-16T12:27:01.270603+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:19:03.776179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"98'2","reported_seq":13,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.408834+0000","last_change":"2023-10-16T12:27:00.416130+0000","last_active":"2023-10-16T12:27:01.408834+0000","last_peered":"2023-10-16T12:27:01.408834+0000","last_clean":"2023-10-16T12:27:01.408834+0000","last_became_active":"2023-10-16T12:27:00.415843+0000","last_became_peered":"2023-10-16T12:27:00.415843+0000","last_unstale":"2023-10-16T12:27:01.408834+0000","last_undegraded":"2023-10-16T12:27:01.408834+0000","last_fullsized":"2023-10-16T12:27:01.408834+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_clean_scrub_stamp":"2023-10-16T12:26:59.382546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:27:26.936118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.266694+0000","last_change":"2023-10-16T12:27:01.263778+0000","last_active":"2023-10-16T12:27:01.266694+0000","last_peered":"2023-10-16T12:27:01.266694+0000","last_clean":"2023-10-16T12:27:01.266694+0000","last_became_active":"2023-10-16T12:27:01.263486+0000","last_became_peered":"2023-10-16T12:27:01.263486+0000","last_unstale":"2023-10-16T12:27:01.266694+0000","last_undegraded":"2023-10-16T12:27:01.266694+0000","last_fullsized":"2023-10-16T12:27:01.266694+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:12:51.949103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.249803+0000","last_change":"2023-10-16T12:27:00.409609+0000","last_active":"2023-10-16T12:27:01.249803+0000","last_peered":"2023-10-16T12:27:01.249803+0000","last_clean":"2023-10-16T12:27:01.249803+0000","last_became_active":"2023-10-16T12:27:00.409170+0000","last_became_peered":"2023-10-16T12:27:00.409170+0000","last_unstale":"2023-10-16T12:27:01.249803+0000","last_undegraded":"2023-10-16T12:27:01.249803+0000","last_fullsized":"2023-10-16T12:27:01.249803+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_clean_scrub_stamp":"2023-10-16T12:26:59.382546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:29:03.134564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.271174+0000","last_change":"2023-10-16T12:27:01.268373+0000","last_active":"2023-10-16T12:27:01.271174+0000","last_peered":"2023-10-16T12:27:01.271174+0000","last_clean":"2023-10-16T12:27:01.271174+0000","last_became_active":"2023-10-16T12:27:01.267507+0000","last_became_peered":"2023-10-16T12:27:01.267507+0000","last_unstale":"2023-10-16T12:27:01.271174+0000","last_undegraded":"2023-10-16T12:27:01.271174+0000","last_fullsized":"2023-10-16T12:27:01.271174+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:22:34.967887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.252621+0000","last_change":"2023-10-16T12:27:00.410724+0000","last_active":"2023-10-16T12:27:01.252621+0000","last_peered":"2023-10-16T12:27:01.252621+0000","last_clean":"2023-10-16T12:27:01.252621+0000","last_became_active":"2023-10-16T12:27:00.410401+0000","last_became_peered":"2023-10-16T12:27:00.410401+0000","last_unstale":"2023-10-16T12:27:01.252621+0000","last_undegraded":"2023-10-16T12:27:01.252621+0000","last_fullsized":"2023-10-16T12:27:01.252621+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_clean_scrub_stamp":"2023-10-16T12:26:59.382546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:45:28.047046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.263350+0000","last_change":"2023-10-16T12:27:01.261039+0000","last_active":"2023-10-16T12:27:01.263350+0000","last_peered":"2023-10-16T12:27:01.263350+0000","last_clean":"2023-10-16T12:27:01.263350+0000","last_became_active":"2023-10-16T12:27:01.260229+0000","last_became_peered":"2023-10-16T12:27:01.260229+0000","last_unstale":"2023-10-16T12:27:01.263350+0000","last_undegraded":"2023-10-16T12:27:01.263350+0000","last_fullsized":"2023-10-16T12:27:01.263350+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:19:43.046725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.251987+0000","last_change":"2023-10-16T12:27:00.419567+0000","last_active":"2023-10-16T12:27:01.251987+0000","last_peered":"2023-10-16T12:27:01.251987+0000","last_clean":"2023-10-16T12:27:01.251987+0000","last_became_active":"2023-10-16T12:27:00.418977+0000","last_became_peered":"2023-10-16T12:27:00.418977+0000","last_unstale":"2023-10-16T12:27:01.251987+0000","last_undegraded":"2023-10-16T12:27:01.251987+0000","last_fullsized":"2023-10-16T12:27:01.251987+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_clean_scrub_stamp":"2023-10-16T12:26:59.382546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:10:08.160283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.269734+0000","last_change":"2023-10-16T12:27:01.264955+0000","last_active":"2023-10-16T12:27:01.269734+0000","last_peered":"2023-10-16T12:27:01.269734+0000","last_clean":"2023-10-16T12:27:01.269734+0000","last_became_active":"2023-10-16T12:27:01.264200+0000","last_became_peered":"2023-10-16T12:27:01.264200+0000","last_unstale":"2023-10-16T12:27:01.269734+0000","last_undegraded":"2023-10-16T12:27:01.269734+0000","last_fullsized":"2023-10-16T12:27:01.269734+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:28:01.320829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.259535+0000","last_change":"2023-10-16T12:27:00.416980+0000","last_active":"2023-10-16T12:27:01.259535+0000","last_peered":"2023-10-16T12:27:01.259535+0000","last_clean":"2023-10-16T12:27:01.259535+0000","last_became_active":"2023-10-16T12:27:00.416674+0000","last_became_peered":"2023-10-16T12:27:00.416674+0000","last_unstale":"2023-10-16T12:27:01.259535+0000","last_undegraded":"2023-10-16T12:27:01.259535+0000","last_fullsized":"2023-10-16T12:27:01.259535+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_clean_scrub_stamp":"2023-10-16T12:26:59.382546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:22:33.962412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.275091+0000","last_change":"2023-10-16T12:27:01.274531+0000","last_active":"2023-10-16T12:27:01.275091+0000","last_peered":"2023-10-16T12:27:01.275091+0000","last_clean":"2023-10-16T12:27:01.275091+0000","last_became_active":"2023-10-16T12:27:01.274232+0000","last_became_peered":"2023-10-16T12:27:01.274232+0000","last_unstale":"2023-10-16T12:27:01.275091+0000","last_undegraded":"2023-10-16T12:27:01.275091+0000","last_fullsized":"2023-10-16T12:27:01.275091+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:46:37.809954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:27:02.249451+0000","last_change":"2023-10-16T12:27:00.405523+0000","last_active":"2023-10-16T12:27:02.249451+0000","last_peered":"2023-10-16T12:27:02.249451+0000","last_clean":"2023-10-16T12:27:02.249451+0000","last_became_active":"2023-10-16T12:27:00.405055+0000","last_became_peered":"2023-10-16T12:27:00.405055+0000","last_unstale":"2023-10-16T12:27:02.249451+0000","last_undegraded":"2023-10-16T12:27:02.249451+0000","last_fullsized":"2023-10-16T12:27:02.249451+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_clean_scrub_stamp":"2023-10-16T12:26:59.382546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:59:25.573397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.267195+0000","last_change":"2023-10-16T12:27:01.263608+0000","last_active":"2023-10-16T12:27:01.267195+0000","last_peered":"2023-10-16T12:27:01.267195+0000","last_clean":"2023-10-16T12:27:01.267195+0000","last_became_active":"2023-10-16T12:27:01.263334+0000","last_became_peered":"2023-10-16T12:27:01.263334+0000","last_unstale":"2023-10-16T12:27:01.267195+0000","last_undegraded":"2023-10-16T12:27:01.267195+0000","last_fullsized":"2023-10-16T12:27:01.267195+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:09:01.408305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.259744+0000","last_change":"2023-10-16T12:27:00.415906+0000","last_active":"2023-10-16T12:27:01.259744+0000","last_peered":"2023-10-16T12:27:01.259744+0000","last_clean":"2023-10-16T12:27:01.259744+0000","last_became_active":"2023-10-16T12:27:00.415600+0000","last_became_peered":"2023-10-16T12:27:00.415600+0000","last_unstale":"2023-10-16T12:27:01.259744+0000","last_undegraded":"2023-10-16T12:27:01.259744+0000","last_fullsized":"2023-10-16T12:27:01.259744+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_clean_scrub_stamp":"2023-10-16T12:26:59.382546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:25:01.715586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.267105+0000","last_change":"2023-10-16T12:27:01.263040+0000","last_active":"2023-10-16T12:27:01.267105+0000","last_peered":"2023-10-16T12:27:01.267105+0000","last_clean":"2023-10-16T12:27:01.267105+0000","last_became_active":"2023-10-16T12:27:01.262735+0000","last_became_peered":"2023-10-16T12:27:01.262735+0000","last_unstale":"2023-10-16T12:27:01.267105+0000","last_undegraded":"2023-10-16T12:27:01.267105+0000","last_fullsized":"2023-10-16T12:27:01.267105+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:25:02.721060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.254772+0000","last_change":"2023-10-16T12:27:00.413071+0000","last_active":"2023-10-16T12:27:01.254772+0000","last_peered":"2023-10-16T12:27:01.254772+0000","last_clean":"2023-10-16T12:27:01.254772+0000","last_became_active":"2023-10-16T12:27:00.412744+0000","last_became_peered":"2023-10-16T12:27:00.412744+0000","last_unstale":"2023-10-16T12:27:01.254772+0000","last_undegraded":"2023-10-16T12:27:01.254772+0000","last_fullsized":"2023-10-16T12:27:01.254772+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_clean_scrub_stamp":"2023-10-16T12:26:59.382546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:17:19.746182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.268562+0000","last_change":"2023-10-16T12:27:01.265090+0000","last_active":"2023-10-16T12:27:01.268562+0000","last_peered":"2023-10-16T12:27:01.268562+0000","last_clean":"2023-10-16T12:27:01.268562+0000","last_became_active":"2023-10-16T12:27:01.264757+0000","last_became_peered":"2023-10-16T12:27:01.264757+0000","last_unstale":"2023-10-16T12:27:01.268562+0000","last_undegraded":"2023-10-16T12:27:01.268562+0000","last_fullsized":"2023-10-16T12:27:01.268562+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:59:38.006434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:27:02.250482+0000","last_change":"2023-10-16T12:27:00.419932+0000","last_active":"2023-10-16T12:27:02.250482+0000","last_peered":"2023-10-16T12:27:02.250482+0000","last_clean":"2023-10-16T12:27:02.250482+0000","last_became_active":"2023-10-16T12:27:00.419643+0000","last_became_peered":"2023-10-16T12:27:00.419643+0000","last_unstale":"2023-10-16T12:27:02.250482+0000","last_undegraded":"2023-10-16T12:27:02.250482+0000","last_fullsized":"2023-10-16T12:27:02.250482+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_clean_scrub_stamp":"2023-10-16T12:26:59.382546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:05:10.569652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"88'172","reported_seq":436,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T12:27:00.402330+0000","last_change":"2023-10-16T12:10:34.746820+0000","last_active":"2023-10-16T12:27:00.402330+0000","last_peered":"2023-10-16T12:27:00.402330+0000","last_clean":"2023-10-16T12:27:00.402330+0000","last_became_active":"2023-10-16T12:10:34.745574+0000","last_became_peered":"2023-10-16T12:10:34.745574+0000","last_unstale":"2023-10-16T12:27:00.402330+0000","last_undegraded":"2023-10-16T12:27:00.402330+0000","last_fullsized":"2023-10-16T12:27:00.402330+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_clean_scrub_stamp":"2023-10-16T12:10:33.577111+0000","objects_scrubbed":0,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:14.461986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":596,"num_read_kb":1130,"num_write":259,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":596,"num_read_kb":1130,"num_write":259,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":172,"ondisk_log_size":172,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575047,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59308,"kb_used_data":1756,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312532,"statfs":{"total":96636764160,"available":96576032768,"internally_reserved":0,"allocated":1798144,"data_stored":1189923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58915275},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:26:05 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.59699999999999998,"15min":0.58499999999999996},"min":{"1min":0.437,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.59399999999999997,"15min":0.58199999999999996},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.312},"max":{"1min":0.85299999999999998,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.50600000000000001}]},{"osd":1,"last update":"Mon Oct 16 12:26:05 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.60399999999999998,"15min":0.59999999999999998},"min":{"1min":0.36099999999999999,"5min":0.35999999999999999,"15min":0.247},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.62,"15min":0.60099999999999998},"min":{"1min":0.45600000000000002,"5min":0.33200000000000002,"15min":0.32400000000000001},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.76700000000000002}]},{"osd":2,"last update":"Mon Oct 16 12:26:05 2023","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.67600000000000005,"15min":0.65600000000000003},"min":{"1min":0.53700000000000003,"5min":0.40699999999999997,"15min":0.36099999999999999},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.68799999999999994,"15min":0.66600000000000004},"min":{"1min":0.495,"5min":0.376,"15min":0.36699999999999999},"max":{"1min":0.94599999999999995,"5min":0.94799999999999995,"15min":0.96699999999999997},"last":0.88}]},{"osd":3,"last update":"Mon Oct 16 12:26:05 2023","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.63,"15min":0.61899999999999999},"min":{"1min":0.43099999999999999,"5min":0.35499999999999998,"15min":0.34899999999999998},"max":{"1min":0.91200000000000003,"5min":0.92200000000000004,"15min":1.1000000000000001},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.65700000000000003,"15min":0.63900000000000001},"min":{"1min":0.52200000000000002,"5min":0.376,"15min":0.376},"max":{"1min":0.996,"5min":1.2509999999999999,"15min":1.2509999999999999},"last":0.84999999999999998}]},{"osd":4,"last update":"Mon Oct 16 12:26:05 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53800000000000003,"15min":0.53100000000000003},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.28499999999999998},"max":{"1min":0.72699999999999998,"5min":1.383,"15min":1.383},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.625,"15min":0.60699999999999998},"min":{"1min":0.40799999999999997,"5min":0.36299999999999999,"15min":0.307},"max":{"1min":0.86599999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.93600000000000005}]},{"osd":5,"last update":"Mon Oct 16 12:26:05 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.55800000000000005,"15min":0.54300000000000004},"min":{"1min":0.40000000000000002,"5min":0.314,"15min":0.29799999999999999},"max":{"1min":0.84299999999999997,"5min":1.3109999999999999,"15min":1.3109999999999999},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.59399999999999997,"15min":0.58699999999999997},"min":{"1min":0.41799999999999998,"5min":0.33500000000000002,"15min":0.32400000000000001},"max":{"1min":0.83899999999999997,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":0.72699999999999998}]},{"osd":6,"last update":"Mon Oct 16 12:26:05 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.63500000000000001,"15min":0.621},"min":{"1min":0.44700000000000001,"5min":0.39300000000000002,"15min":0.36399999999999999},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.65600000000000003,"15min":0.63100000000000001},"min":{"1min":0.443,"5min":0.34599999999999997,"15min":0.32300000000000001},"max":{"1min":0.88300000000000001,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.97199999999999998}]}]},{"osd":6,"up_from":13,"seq":55834575045,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32368,"kb_used_data":1184,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339472,"statfs":{"total":96636764160,"available":96603619328,"internally_reserved":0,"allocated":1212416,"data_stored":601145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:25:58 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.60099999999999998,"15min":0.628},"min":{"1min":0.34200000000000003,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.88200000000000001,"5min":1.196,"15min":5.9349999999999996},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60499999999999998,"15min":0.64000000000000001},"min":{"1min":0.44900000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.81799999999999995,"5min":0.997,"15min":5.9489999999999998},"last":0.46800000000000003}]},{"osd":1,"last update":"Mon Oct 16 12:25:58 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.55500000000000005,"15min":0.57499999999999996},"min":{"1min":0.40200000000000002,"5min":0.32800000000000001,"15min":0.29399999999999998},"max":{"1min":0.71199999999999997,"5min":0.83899999999999997,"15min":0.875},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57899999999999996,"15min":0.59199999999999997},"min":{"1min":0.44500000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.83099999999999996,"5min":1.091,"15min":1.091},"last":0.55700000000000005}]},{"osd":2,"last update":"Mon Oct 16 12:25:58 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.57799999999999996,"15min":0.61799999999999999},"min":{"1min":0.41699999999999998,"5min":0.36199999999999999,"15min":0.34499999999999997},"max":{"1min":0.69199999999999995,"5min":0.91200000000000003,"15min":8.0489999999999995},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.57699999999999996,"15min":0.60899999999999999},"min":{"1min":0.41099999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.76300000000000001,"5min":1.1539999999999999,"15min":2.855},"last":0.623}]},{"osd":3,"last update":"Mon Oct 16 12:25:58 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.60599999999999998,"15min":0.629},"min":{"1min":0.45500000000000002,"5min":0.34200000000000003,"15min":0.307},"max":{"1min":0.85399999999999998,"5min":0.93999999999999995,"15min":1.083},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67400000000000004,"15min":0.67900000000000005},"min":{"1min":0.48099999999999998,"5min":0.40899999999999997,"15min":0.36599999999999999},"max":{"1min":0.94499999999999995,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.755}]},{"osd":4,"last update":"Mon Oct 16 12:25:58 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.52800000000000002,"15min":0.52000000000000002},"min":{"1min":0.33900000000000002,"5min":0.308,"15min":0.29899999999999999},"max":{"1min":0.83399999999999996,"5min":0.85599999999999998,"15min":0.88200000000000001},"last":0.379},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.60299999999999998,"15min":0.59399999999999997},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.30199999999999999},"max":{"1min":0.78500000000000003,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.69499999999999995}]},{"osd":5,"last update":"Mon Oct 16 12:25:58 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59099999999999997,"15min":0.57799999999999996},"min":{"1min":0.39900000000000002,"5min":0.34799999999999998,"15min":0.30099999999999999},"max":{"1min":0.80100000000000005,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63400000000000001,"15min":0.624},"min":{"1min":0.39400000000000002,"5min":0.35099999999999998,"15min":0.311},"max":{"1min":0.81599999999999995,"5min":1.3540000000000001,"15min":1.3540000000000001},"last":0.78000000000000003}]},{"osd":7,"last update":"Mon Oct 16 12:25:58 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59499999999999997,"15min":0.57499999999999996},"min":{"1min":0.36499999999999999,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.86599999999999999,"5min":1.29,"15min":1.29},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.62,"5min":0.625,"15min":0.61799999999999999},"min":{"1min":0.38800000000000001,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.84399999999999997,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.59899999999999998}]}]},{"osd":1,"up_from":13,"seq":55834575045,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32360,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339480,"statfs":{"total":96636764160,"available":96603627520,"internally_reserved":0,"allocated":1208320,"data_stored":599645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":31914441},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:27:00 2023","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.36899999999999999,"15min":0.376},"min":{"1min":0.246,"5min":0.21299999999999999,"15min":0.186},"max":{"1min":0.53300000000000003,"5min":0.63600000000000001,"15min":0.66000000000000003},"last":0.246},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.44500000000000001,"15min":0.45800000000000002},"min":{"1min":0.28399999999999997,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.502,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.31900000000000001}]},{"osd":2,"last update":"Mon Oct 16 12:27:00 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.40500000000000003,"15min":0.42599999999999999},"min":{"1min":0.251,"5min":0.25,"15min":0.215},"max":{"1min":0.45000000000000001,"5min":0.69199999999999995,"15min":0.93000000000000005},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.45300000000000001,"15min":0.46400000000000002},"min":{"1min":0.32400000000000001,"5min":0.28199999999999997,"15min":0.24399999999999999},"max":{"1min":0.54500000000000004,"5min":0.69199999999999995,"15min":0.98099999999999998},"last":0.436}]},{"osd":3,"last update":"Mon Oct 16 12:27:00 2023","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.38500000000000001,"15min":0.40100000000000002},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.57999999999999996,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.42799999999999999,"15min":0.45500000000000002},"min":{"1min":0.26100000000000001,"5min":0.252,"15min":0.22900000000000001},"max":{"1min":0.58699999999999997,"5min":0.91200000000000003,"15min":1.127},"last":0.45800000000000002}]},{"osd":4,"last update":"Mon Oct 16 12:27:00 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61099999999999999,"15min":0.63300000000000001},"min":{"1min":0.41799999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.79800000000000004,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.626},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.62,"15min":0.63900000000000001},"min":{"1min":0.41899999999999998,"5min":0.39100000000000001,"15min":0.34699999999999998},"max":{"1min":0.89600000000000002,"5min":0.92000000000000004,"15min":1.089},"last":0.55600000000000005}]},{"osd":5,"last update":"Mon Oct 16 12:27:00 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.63600000000000001,"15min":0.66500000000000004},"min":{"1min":0.496,"5min":0.38300000000000001,"15min":0.28499999999999998},"max":{"1min":0.76700000000000002,"5min":1.069,"15min":1.069},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67300000000000004,"15min":0.68799999999999994},"min":{"1min":0.439,"5min":0.38900000000000001,"15min":0.32500000000000001},"max":{"1min":0.94499999999999995,"5min":1.0620000000000001,"15min":1.0649999999999999},"last":0.67300000000000004}]},{"osd":6,"last update":"Mon Oct 16 12:27:00 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.63,"15min":0.65400000000000003},"min":{"1min":0.48699999999999999,"5min":0.378,"15min":0.378},"max":{"1min":0.77800000000000002,"5min":0.94999999999999996,"15min":1.0129999999999999},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.629,"5min":0.65300000000000002,"15min":0.66200000000000003},"min":{"1min":0.55100000000000005,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.83299999999999996,"5min":0.96899999999999997,"15min":1.048},"last":0.56799999999999995}]},{"osd":7,"last update":"Mon Oct 16 12:27:00 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.67200000000000004,"15min":0.68700000000000006},"min":{"1min":0.42099999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.98899999999999999,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.67600000000000005,"15min":0.70299999999999996},"min":{"1min":0.45700000000000002,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.876,"5min":0.94899999999999995,"15min":1.167},"last":0.58699999999999997}]}]},{"osd":0,"up_from":13,"seq":55834575046,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37100,"kb_used_data":1752,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334740,"statfs":{"total":96636764160,"available":96598773760,"internally_reserved":0,"allocated":1794048,"data_stored":1189901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 12:26:55 2023","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"min":{"1min":0.22500000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.40999999999999998,"15min":0.41799999999999998},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23200000000000001},"max":{"1min":0.64500000000000002,"5min":0.71799999999999997,"15min":0.73099999999999998},"last":0.35099999999999998}]},{"osd":2,"last update":"Mon Oct 16 12:26:55 2023","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.42099999999999999,"15min":0.42499999999999999},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.44500000000000001,"15min":0.45100000000000001},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.46600000000000003}]},{"osd":3,"last update":"Mon Oct 16 12:26:55 2023","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.42699999999999999,"15min":0.42199999999999999},"min":{"1min":0.26300000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.62,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.47799999999999998,"15min":0.48699999999999999},"min":{"1min":0.318,"5min":0.29099999999999998,"15min":0.27600000000000002},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.878},"last":0.44}]},{"osd":4,"last update":"Mon Oct 16 12:26:55 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.64300000000000002,"15min":0.64800000000000002},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.89200000000000002,"5min":0.90400000000000003,"15min":1.0960000000000001},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.626,"15min":0.65500000000000003},"min":{"1min":0.374,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.84199999999999997,"5min":0.93799999999999994,"15min":1.1719999999999999},"last":0.71999999999999997}]},{"osd":5,"last update":"Mon Oct 16 12:26:55 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.63500000000000001,"15min":0.65000000000000002},"min":{"1min":0.373,"5min":0.373,"15min":0.34399999999999997},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":1.127},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.66500000000000004,"15min":0.67700000000000005},"min":{"1min":0.39400000000000002,"5min":0.39200000000000002,"15min":0.39000000000000001},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":1.1160000000000001},"last":0.77500000000000002}]},{"osd":6,"last update":"Mon Oct 16 12:26:55 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.66600000000000004,"15min":0.68000000000000005},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":1.079},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.65900000000000003,"15min":0.68300000000000005},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":1.1499999999999999},"last":0.80300000000000005}]},{"osd":7,"last update":"Mon Oct 16 12:26:55 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.67800000000000005,"15min":0.69899999999999995},"min":{"1min":0.436,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.995,"5min":0.995,"15min":1.0349999999999999},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.65700000000000003,"15min":0.68799999999999994},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":1.0569999999999999},"last":0.66300000000000003}]}]},{"osd":2,"up_from":13,"seq":55834575046,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58736,"kb_used_data":1184,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313104,"statfs":{"total":96636764160,"available":96576618496,"internally_reserved":0,"allocated":1212416,"data_stored":600275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:26:56 2023","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.36799999999999999,"15min":0.38400000000000001},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.62,"5min":0.62,"15min":0.73799999999999999},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.44400000000000001,"15min":0.45000000000000001},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.63400000000000001,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.60699999999999998}]},{"osd":1,"last update":"Mon Oct 16 12:26:56 2023","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.45100000000000001,"15min":0.46800000000000003},"min":{"1min":0.31,"5min":0.25600000000000001,"15min":0.254},"max":{"1min":0.57999999999999996,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.42499999999999999,"15min":0.435},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.246},"max":{"1min":0.54400000000000004,"5min":0.69699999999999995,"15min":0.76500000000000001},"last":0.32300000000000001}]},{"osd":3,"last update":"Mon Oct 16 12:26:56 2023","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.40200000000000002,"15min":0.41599999999999998},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.60199999999999998,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.51000000000000001,"15min":0.51900000000000002},"min":{"1min":0.36599999999999999,"5min":0.318,"15min":0.246},"max":{"1min":0.67600000000000005,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.47399999999999998}]},{"osd":4,"last update":"Mon Oct 16 12:26:56 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.66500000000000004,"15min":0.68100000000000005},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.81200000000000006,"5min":1.3839999999999999,"15min":1.3839999999999999},"last":0.93799999999999994},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.66200000000000003,"15min":0.67900000000000005},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.84899999999999998,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.91200000000000003}]},{"osd":5,"last update":"Mon Oct 16 12:26:56 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.65800000000000003,"15min":0.66600000000000004},"min":{"1min":0.378,"5min":0.378,"15min":0.312},"max":{"1min":0.78700000000000003,"5min":1.093,"15min":1.093},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.64600000000000002,"15min":0.66400000000000003},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.317},"max":{"1min":0.879,"5min":1.331,"15min":1.331},"last":0.97399999999999998}]},{"osd":6,"last update":"Mon Oct 16 12:26:56 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.63200000000000001,"15min":0.63800000000000001},"min":{"1min":0.37,"5min":0.37,"15min":0.34699999999999998},"max":{"1min":0.91900000000000004,"5min":0.99299999999999999,"15min":1.046},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.68400000000000005,"15min":0.69199999999999995},"min":{"1min":0.40799999999999997,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.82599999999999996,"5min":1.032,"15min":1.0389999999999999},"last":0.84999999999999998}]},{"osd":7,"last update":"Mon Oct 16 12:26:56 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.66500000000000004,"15min":0.68000000000000005},"min":{"1min":0.40300000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.879,"5min":0.97499999999999998,"15min":1.026},"last":0.88600000000000001},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.68100000000000005,"15min":0.69299999999999995},"min":{"1min":0.46700000000000003,"5min":0.442,"15min":0.40600000000000003},"max":{"1min":0.89600000000000002,"5min":1.145,"15min":1.145},"last":0.67100000000000004}]}]},{"osd":3,"up_from":13,"seq":55834575045,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36524,"kb_used_data":1176,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335316,"statfs":{"total":96636764160,"available":96599363584,"internally_reserved":0,"allocated":1204224,"data_stored":600115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:26:02 2023","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.40100000000000002,"15min":0.39400000000000002},"min":{"1min":0.29499999999999998,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.435,"5min":0.41199999999999998,"15min":0.40500000000000003},"min":{"1min":0.28999999999999998,"5min":0.255,"15min":0.22600000000000001},"max":{"1min":0.61899999999999999,"5min":0.63100000000000001,"15min":0.73099999999999998},"last":0.317}]},{"osd":1,"last update":"Mon Oct 16 12:26:02 2023","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.41999999999999998,"15min":0.40999999999999998},"min":{"1min":0.29499999999999998,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.64100000000000001,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.46300000000000002,"15min":0.45800000000000002},"min":{"1min":0.38900000000000001,"5min":0.25800000000000001,"15min":0.25700000000000001},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.747},"last":0.27000000000000002}]},{"osd":2,"last update":"Mon Oct 16 12:26:02 2023","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.47399999999999998,"15min":0.46800000000000003},"min":{"1min":0.39600000000000002,"5min":0.307,"15min":0.26900000000000002},"max":{"1min":0.63700000000000001,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.503,"15min":0.49199999999999999},"min":{"1min":0.42199999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.60299999999999998,"5min":0.752,"15min":0.752},"last":0.377}]},{"osd":4,"last update":"Mon Oct 16 12:26:02 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.63200000000000001,"15min":0.64500000000000002},"min":{"1min":0.49399999999999999,"5min":0.39100000000000001,"15min":0.377},"max":{"1min":0.81699999999999995,"5min":0.92700000000000005,"15min":1.1539999999999999},"last":0.501},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.65400000000000003,"15min":0.66800000000000004},"min":{"1min":0.55700000000000005,"5min":0.44900000000000001,"15min":0.40200000000000002},"max":{"1min":0.90600000000000003,"5min":1.0409999999999999,"15min":1.234},"last":0.39500000000000002}]},{"osd":5,"last update":"Mon Oct 16 12:26:02 2023","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.68600000000000005,"15min":0.68300000000000005},"min":{"1min":0.48899999999999999,"5min":0.443,"15min":0.41199999999999998},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.194},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.65600000000000003,"15min":0.66700000000000004},"min":{"1min":0.56000000000000005,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":1.202},"last":0.52600000000000002}]},{"osd":6,"last update":"Mon Oct 16 12:26:02 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.67500000000000004,"15min":0.67400000000000004},"min":{"1min":0.53300000000000003,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.94199999999999995,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.75,"5min":0.70599999999999996,"15min":0.70699999999999996},"min":{"1min":0.60099999999999998,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.1559999999999999},"last":0.45100000000000001}]},{"osd":7,"last update":"Mon Oct 16 12:26:02 2023","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.68000000000000005,"15min":0.68899999999999995},"min":{"1min":0.499,"5min":0.435,"15min":0.38},"max":{"1min":0.92300000000000004,"5min":1.0189999999999999,"15min":1.2430000000000001},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.70999999999999996,"15min":0.70699999999999996},"min":{"1min":0.48899999999999999,"5min":0.45200000000000001,"15min":0.42699999999999999},"max":{"1min":0.99099999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.44}]}]},{"osd":4,"up_from":13,"seq":55834575044,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32372,"kb_used_data":1192,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339468,"statfs":{"total":96636764160,"available":96603615232,"internally_reserved":0,"allocated":1220608,"data_stored":601249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:26:00 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.59799999999999998,"15min":0.59199999999999997},"min":{"1min":0.30399999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.78300000000000003,"5min":0.85099999999999998,"15min":1.171},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.62,"5min":0.58699999999999997,"15min":0.60199999999999998},"min":{"1min":0.42399999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.76400000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.49299999999999999}]},{"osd":1,"last update":"Mon Oct 16 12:26:00 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.59299999999999997,"15min":0.60999999999999999},"min":{"1min":0.433,"5min":0.35499999999999998,"15min":0.28499999999999998},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.92600000000000005},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.622,"5min":0.61499999999999999,"15min":0.621},"min":{"1min":0.374,"5min":0.374,"15min":0.33600000000000002},"max":{"1min":0.73899999999999999,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.58299999999999996}]},{"osd":2,"last update":"Mon Oct 16 12:26:00 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.61199999999999999,"15min":0.625},"min":{"1min":0.39900000000000002,"5min":0.35399999999999998,"15min":0.32000000000000001},"max":{"1min":0.77000000000000002,"5min":0.93000000000000005,"15min":1.012},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61299999999999999,"15min":0.627},"min":{"1min":0.47899999999999998,"5min":0.38800000000000001,"15min":0.34100000000000003},"max":{"1min":0.74099999999999999,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.61799999999999999}]},{"osd":3,"last update":"Mon Oct 16 12:26:00 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.59599999999999997,"15min":0.61299999999999999},"min":{"1min":0.46300000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.70699999999999996,"5min":0.79500000000000004,"15min":1.3779999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63,"15min":0.64100000000000001},"min":{"1min":0.41199999999999998,"5min":0.38800000000000001,"15min":0.37},"max":{"1min":0.77800000000000002,"5min":0.89800000000000002,"15min":0.97199999999999998},"last":0.65000000000000002}]},{"osd":5,"last update":"Mon Oct 16 12:26:00 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48699999999999999,"15min":0.5},"min":{"1min":0.34799999999999998,"5min":0.29799999999999999,"15min":0.27600000000000002},"max":{"1min":0.68100000000000005,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.504,"5min":0.51400000000000001,"15min":0.52500000000000002},"min":{"1min":0.371,"5min":0.316,"15min":0.29599999999999999},"max":{"1min":0.72599999999999998,"5min":0.754,"15min":0.94199999999999995},"last":0.56299999999999994}]},{"osd":6,"last update":"Mon Oct 16 12:26:00 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.54600000000000004,"15min":0.55900000000000005},"min":{"1min":0.39800000000000002,"5min":0.34300000000000003,"15min":0.28699999999999998},"max":{"1min":0.68700000000000006,"5min":0.88400000000000001,"15min":0.89800000000000002},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.63700000000000001,"15min":0.63600000000000001},"min":{"1min":0.45000000000000001,"5min":0.34799999999999998,"15min":0.32000000000000001},"max":{"1min":0.84099999999999997,"5min":0.99299999999999999,"15min":1.1180000000000001},"last":0.70499999999999996}]},{"osd":7,"last update":"Mon Oct 16 12:26:00 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57599999999999996,"15min":0.58099999999999996},"min":{"1min":0.42099999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.76200000000000001,"5min":0.80900000000000005,"15min":0.89500000000000002},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.628,"15min":0.626},"min":{"1min":0.42599999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.87,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.60199999999999998}]}]},{"osd":5,"up_from":13,"seq":55834575045,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54568,"kb_used_data":1176,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317272,"statfs":{"total":96636764160,"available":96580886528,"internally_reserved":0,"allocated":1204224,"data_stored":599567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:26:58 2023","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.56899999999999995,"15min":0.59399999999999997},"min":{"1min":0.32500000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.64600000000000002,"5min":0.84499999999999997,"15min":0.873},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.59399999999999997,"15min":0.62},"min":{"1min":0.36299999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.73499999999999999,"5min":0.85899999999999999,"15min":0.88800000000000001},"last":0.63300000000000001}]},{"osd":1,"last update":"Mon Oct 16 12:26:58 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.60499999999999998,"15min":0.61499999999999999},"min":{"1min":0.39500000000000002,"5min":0.32600000000000001,"15min":0.28999999999999998},"max":{"1min":0.76600000000000001,"5min":0.89700000000000002,"15min":0.91300000000000003},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.503,"5min":0.60699999999999998,"15min":0.62},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.78900000000000003,"5min":0.85999999999999999,"15min":0.89000000000000001},"last":0.61099999999999999}]},{"osd":2,"last update":"Mon Oct 16 12:26:58 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.60299999999999998,"15min":0.623},"min":{"1min":0.40000000000000002,"5min":0.29099999999999998,"15min":0.26400000000000001},"max":{"1min":0.71299999999999997,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.59599999999999997,"15min":0.61399999999999999},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.66900000000000004,"5min":0.88600000000000001,"15min":0.89700000000000002},"last":0.45900000000000002}]},{"osd":3,"last update":"Mon Oct 16 12:26:58 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.61599999999999999,"15min":0.64100000000000001},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.72099999999999997,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.64000000000000001,"15min":0.65000000000000002},"min":{"1min":0.40400000000000003,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.71799999999999997,"5min":0.90800000000000003,"15min":0.93200000000000005},"last":0.56799999999999995}]},{"osd":4,"last update":"Mon Oct 16 12:26:58 2023","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.504,"15min":0.51000000000000001},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.25800000000000001},"max":{"1min":0.77100000000000002,"5min":0.85699999999999998,"15min":0.86399999999999999},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.58499999999999996,"15min":0.57099999999999995},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33500000000000002},"max":{"1min":0.78500000000000003,"5min":1.347,"15min":1.347},"last":0.41099999999999998}]},{"osd":6,"last update":"Mon Oct 16 12:26:58 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.54400000000000004,"15min":0.55200000000000005},"min":{"1min":0.33100000000000002,"5min":0.314,"15min":0.314},"max":{"1min":0.83999999999999997,"5min":0.89400000000000002,"15min":0.93500000000000005},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.59999999999999998,"15min":0.59799999999999998},"min":{"1min":0.38300000000000001,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.82899999999999996,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.65400000000000003}]},{"osd":7,"last update":"Mon Oct 16 12:26:58 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.61699999999999999,"15min":0.61899999999999999},"min":{"1min":0.41499999999999998,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.79400000000000004,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.65300000000000002,"15min":0.65500000000000003},"min":{"1min":0.45600000000000002,"5min":0.379,"15min":0.379},"max":{"1min":0.88900000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.67000000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T12:27:05.570 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T12:27:05.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.725+0000 7f83e108e700 1 Processor -- start 2023-10-16T12:27:05.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.725+0000 7f83e108e700 1 -- start start 2023-10-16T12:27:05.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.725+0000 7f83e108e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f83dc06b7f0 0x7f83dc06bc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:05.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.725+0000 7f83e108e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f83dc06d940 0x7f83dc06ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:05.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.725+0000 7f83e108e700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83dc0c3490 con 0x7f83dc15b9f0 2023-10-16T12:27:05.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.725+0000 7f83e108e700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83dc06c1b0 con 0x7f83dc06b7f0 2023-10-16T12:27:05.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.725+0000 7f83e108e700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83dc06e330 con 0x7f83dc06d940 2023-10-16T12:27:05.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.726+0000 7f83dad9d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f83dc15b9f0 0x7f83dc0b29a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:58434/0 (socket says 172.21.15.16:58434) 2023-10-16T12:27:05.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.726+0000 7f83da59c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f83dc06b7f0 0x7f83dc06bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:05.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.726+0000 7f83dad9d700 1 -- 172.21.15.16:0/3983985925 learned_addr learned my addr 172.21.15.16:0/3983985925 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:27:05.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.726+0000 7f83db59e700 1 --2- 172.21.15.16:0/3983985925 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f83dc06d940 0x7f83dc06ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:05.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.726+0000 7f83da59c700 1 -- 172.21.15.16:0/3983985925 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f83dc06d940 msgr2=0x7f83dc06ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:05.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.726+0000 7f83da59c700 1 --2- 172.21.15.16:0/3983985925 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f83dc06d940 0x7f83dc06ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:05.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.726+0000 7f83da59c700 1 -- 172.21.15.16:0/3983985925 >> v1:172.21.15.16:6789/0 conn(0x7f83dc15b9f0 legacy=0x7f83dc0b29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:05.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.726+0000 7f83d9d9b700 1 -- 172.21.15.16:0/3983985925 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3710365019 0 0) 0x7f83dc0c3490 con 0x7f83dc15b9f0 2023-10-16T12:27:05.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.726+0000 7f83da59c700 1 -- 172.21.15.16:0/3983985925 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83dc0cdad0 con 0x7f83dc06b7f0 2023-10-16T12:27:05.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.726+0000 7f83db59e700 1 --2- 172.21.15.16:0/3983985925 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f83dc06d940 0x7f83dc06ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:27:05.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.726+0000 7f83da59c700 1 --2- 172.21.15.16:0/3983985925 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83dc06b7f0 0x7f83dc06bc70 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f83d0009870 tx=0x7f83d0009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0247a2d151319e0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:05.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.726+0000 7f83d9d9b700 1 -- 172.21.15.16:0/3983985925 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d0011830 con 0x7f83dc06b7f0 2023-10-16T12:27:05.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.726+0000 7f83d9d9b700 1 -- 172.21.15.16:0/3983985925 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83d0011990 con 0x7f83dc06b7f0 2023-10-16T12:27:05.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.726+0000 7f83d9d9b700 1 -- 172.21.15.16:0/3983985925 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d0011d20 con 0x7f83dc06b7f0 2023-10-16T12:27:05.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.727+0000 7f83e108e700 1 -- 172.21.15.16:0/3983985925 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83dc06b7f0 msgr2=0x7f83dc06bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:05.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.727+0000 7f83e108e700 1 --2- 172.21.15.16:0/3983985925 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83dc06b7f0 0x7f83dc06bc70 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f83d0009870 tx=0x7f83d0009b80 comp rx=0 tx=0).stop 2023-10-16T12:27:05.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.727+0000 7f83e108e700 1 -- 172.21.15.16:0/3983985925 shutdown_connections 2023-10-16T12:27:05.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.727+0000 7f83e108e700 1 --2- 172.21.15.16:0/3983985925 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f83dc06d940 0x7f83dc06ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:05.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.727+0000 7f83e108e700 1 --2- 172.21.15.16:0/3983985925 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83dc06b7f0 0x7f83dc06bc70 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:05.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.727+0000 7f83e108e700 1 -- 172.21.15.16:0/3983985925 >> 172.21.15.16:0/3983985925 conn(0x7f83dc0cc5b0 msgr2=0x7f83dc0cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:05.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.727+0000 7f83e108e700 1 -- 172.21.15.16:0/3983985925 shutdown_connections 2023-10-16T12:27:05.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.727+0000 7f83e108e700 1 -- 172.21.15.16:0/3983985925 wait complete. 2023-10-16T12:27:05.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.729+0000 7f83e108e700 1 Processor -- start 2023-10-16T12:27:05.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.729+0000 7f83e108e700 1 -- start start 2023-10-16T12:27:05.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.730+0000 7f83e108e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f83dc06b7f0 0x7f83dc1ee600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:05.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.730+0000 7f83e108e700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f83dc06d940 0x7f83dc15e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:05.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.730+0000 7f83e108e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83dc15e8e0 0x7f83dc0c8f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:05.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.730+0000 7f83e108e700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83dc1f1090 con 0x7f83dc15e8e0 2023-10-16T12:27:05.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.730+0000 7f83e108e700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83dc1eeb40 con 0x7f83dc06d940 2023-10-16T12:27:05.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.730+0000 7f83e108e700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83dc1f12a0 con 0x7f83dc06b7f0 2023-10-16T12:27:05.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.730+0000 7f83db59e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83dc15e8e0 0x7f83dc0c8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:05.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.730+0000 7f83db59e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83dc15e8e0 0x7f83dc0c8f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40938/0 (socket says 172.21.15.16:40938) 2023-10-16T12:27:05.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.730+0000 7f83db59e700 1 -- 172.21.15.16:0/3034047838 learned_addr learned my addr 172.21.15.16:0/3034047838 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:05.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.730+0000 7f83dad9d700 1 --2- 172.21.15.16:0/3034047838 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f83dc06b7f0 0x7f83dc1ee600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:05.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.730+0000 7f83da59c700 1 --2- 172.21.15.16:0/3034047838 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f83dc06d940 0x7f83dc15e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:05.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.730+0000 7f83db59e700 1 -- 172.21.15.16:0/3034047838 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f83dc06b7f0 msgr2=0x7f83dc1ee600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:05.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.730+0000 7f83db59e700 1 --2- 172.21.15.16:0/3034047838 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f83dc06b7f0 0x7f83dc1ee600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:05.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.730+0000 7f83db59e700 1 -- 172.21.15.16:0/3034047838 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f83dc06d940 msgr2=0x7f83dc15e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:05.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.730+0000 7f83db59e700 1 --2- 172.21.15.16:0/3034047838 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f83dc06d940 0x7f83dc15e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:05.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.730+0000 7f83db59e700 1 -- 172.21.15.16:0/3034047838 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83d00094a0 con 0x7f83dc15e8e0 2023-10-16T12:27:05.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.730+0000 7f83db59e700 1 --2- 172.21.15.16:0/3034047838 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83dc15e8e0 0x7f83dc0c8f00 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f83d400cef0 tx=0x7f83d400cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:05.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.731+0000 7f83c37fe700 1 -- 172.21.15.16:0/3034047838 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d400aca0 con 0x7f83dc15e8e0 2023-10-16T12:27:05.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.731+0000 7f83c37fe700 1 -- 172.21.15.16:0/3034047838 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83d400ae00 con 0x7f83dc15e8e0 2023-10-16T12:27:05.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.731+0000 7f83c37fe700 1 -- 172.21.15.16:0/3034047838 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d4015680 con 0x7f83dc15e8e0 2023-10-16T12:27:05.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.731+0000 7f83e108e700 1 -- 172.21.15.16:0/3034047838 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83dc1f1580 con 0x7f83dc15e8e0 2023-10-16T12:27:05.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.731+0000 7f83e108e700 1 -- 172.21.15.16:0/3034047838 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83dc1f1b40 con 0x7f83dc15e8e0 2023-10-16T12:27:05.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.732+0000 7f83e108e700 1 -- 172.21.15.16:0/3034047838 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83dc0ce070 con 0x7f83dc15e8e0 2023-10-16T12:27:05.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.732+0000 7f83c37fe700 1 -- 172.21.15.16:0/3034047838 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 41) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f83d400d430 con 0x7f83dc15e8e0 2023-10-16T12:27:05.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.733+0000 7f83c37fe700 1 --2- 172.21.15.16:0/3034047838 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f83c806b020 0x7f83c806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:05.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.733+0000 7f83c37fe700 1 -- 172.21.15.16:0/3034047838 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f83d4089dd0 con 0x7f83dc15e8e0 2023-10-16T12:27:05.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.733+0000 7f83dad9d700 1 --2- 172.21.15.16:0/3034047838 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f83c806b020 0x7f83c806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:05.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.733+0000 7f83dad9d700 1 --2- 172.21.15.16:0/3034047838 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f83c806b020 0x7f83c806d4e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f83cc007fd0 tx=0x7f83cc007de0 comp rx=0 tx=0).ready entity=mgr.6020 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:05.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.737+0000 7f83c37fe700 1 -- 172.21.15.16:0/3034047838 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f83dc0ce070 con 0x7f83dc15e8e0 2023-10-16T12:27:05.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.937+0000 7f83e108e700 1 -- 172.21.15.16:0/3034047838 --> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f83dc0cddb0 con 0x7f83c806b020 2023-10-16T12:27:05.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.945+0000 7f83c37fe700 1 -- 172.21.15.16:0/3034047838 <== mgr.6020 v2:172.21.15.16:6832/292891612 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289547 (secure 0 0 0) 0x7f83cc001370 con 0x7f83c806b020 2023-10-16T12:27:05.950 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:27:05.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.949+0000 7f83e108e700 1 -- 172.21.15.16:0/3034047838 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f83c806b020 msgr2=0x7f83c806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:05.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.949+0000 7f83e108e700 1 --2- 172.21.15.16:0/3034047838 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f83c806b020 0x7f83c806d4e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f83cc007fd0 tx=0x7f83cc007de0 comp rx=0 tx=0).stop 2023-10-16T12:27:05.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.949+0000 7f83e108e700 1 -- 172.21.15.16:0/3034047838 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83dc15e8e0 msgr2=0x7f83dc0c8f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:05.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.949+0000 7f83e108e700 1 --2- 172.21.15.16:0/3034047838 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83dc15e8e0 0x7f83dc0c8f00 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f83d400cef0 tx=0x7f83d400cf20 comp rx=0 tx=0).stop 2023-10-16T12:27:05.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.949+0000 7f83e108e700 1 -- 172.21.15.16:0/3034047838 shutdown_connections 2023-10-16T12:27:05.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.949+0000 7f83e108e700 1 --2- 172.21.15.16:0/3034047838 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f83c806b020 0x7f83c806d4e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:05.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.949+0000 7f83e108e700 1 --2- 172.21.15.16:0/3034047838 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83dc15e8e0 0x7f83dc0c8f00 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:05.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.949+0000 7f83e108e700 1 --2- 172.21.15.16:0/3034047838 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f83dc06b7f0 0x7f83dc1ee600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:05.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.949+0000 7f83e108e700 1 --2- 172.21.15.16:0/3034047838 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f83dc06d940 0x7f83dc15e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:05.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.949+0000 7f83e108e700 1 -- 172.21.15.16:0/3034047838 >> 172.21.15.16:0/3034047838 conn(0x7f83dc0cc5b0 msgr2=0x7f83dc0b5320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:05.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.949+0000 7f83e108e700 1 -- 172.21.15.16:0/3034047838 shutdown_connections 2023-10-16T12:27:05.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:05.949+0000 7f83e108e700 1 -- 172.21.15.16:0/3034047838 wait complete. 2023-10-16T12:27:05.955 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-10-16T12:27:05.963 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-10-16T12:27:04.329426+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":596,"num_read_kb":1130,"num_write":303,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":210,"ondisk_log_size":210,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":70,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":343336,"kb_used_data":10600,"kb_used_omap":12,"kb_used_meta":332595,"kb_avail":754631384,"statfs":{"total":773094113280,"available":772742537216,"internally_reserved":0,"allocated":10854400,"data_stored":5981820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12740,"internal_metadata":340577852},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.941673"},"pg_stats":[{"pgid":"12.3f","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.271907+0000","last_change":"2023-10-16T12:27:01.271233+0000","last_active":"2023-10-16T12:27:01.271907+0000","last_peered":"2023-10-16T12:27:01.271907+0000","last_clean":"2023-10-16T12:27:01.271907+0000","last_became_active":"2023-10-16T12:27:01.270778+0000","last_became_peered":"2023-10-16T12:27:01.270778+0000","last_unstale":"2023-10-16T12:27:01.271907+0000","last_undegraded":"2023-10-16T12:27:01.271907+0000","last_fullsized":"2023-10-16T12:27:01.271907+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:23:13.033596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.272194+0000","last_change":"2023-10-16T12:27:01.270267+0000","last_active":"2023-10-16T12:27:01.272194+0000","last_peered":"2023-10-16T12:27:01.272194+0000","last_clean":"2023-10-16T12:27:01.272194+0000","last_became_active":"2023-10-16T12:27:01.269957+0000","last_became_peered":"2023-10-16T12:27:01.269957+0000","last_unstale":"2023-10-16T12:27:01.272194+0000","last_undegraded":"2023-10-16T12:27:01.272194+0000","last_fullsized":"2023-10-16T12:27:01.272194+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:57:06.277568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.267221+0000","last_change":"2023-10-16T12:27:01.263515+0000","last_active":"2023-10-16T12:27:01.267221+0000","last_peered":"2023-10-16T12:27:01.267221+0000","last_clean":"2023-10-16T12:27:01.267221+0000","last_became_active":"2023-10-16T12:27:01.263147+0000","last_became_peered":"2023-10-16T12:27:01.263147+0000","last_unstale":"2023-10-16T12:27:01.267221+0000","last_undegraded":"2023-10-16T12:27:01.267221+0000","last_fullsized":"2023-10-16T12:27:01.267221+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:02:29.598278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.267875+0000","last_change":"2023-10-16T12:27:01.264999+0000","last_active":"2023-10-16T12:27:01.267875+0000","last_peered":"2023-10-16T12:27:01.267875+0000","last_clean":"2023-10-16T12:27:01.267875+0000","last_became_active":"2023-10-16T12:27:01.264258+0000","last_became_peered":"2023-10-16T12:27:01.264258+0000","last_unstale":"2023-10-16T12:27:01.267875+0000","last_undegraded":"2023-10-16T12:27:01.267875+0000","last_fullsized":"2023-10-16T12:27:01.267875+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:36:42.760644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:27:02.249323+0000","last_change":"2023-10-16T12:27:01.268966+0000","last_active":"2023-10-16T12:27:02.249323+0000","last_peered":"2023-10-16T12:27:02.249323+0000","last_clean":"2023-10-16T12:27:02.249323+0000","last_became_active":"2023-10-16T12:27:01.268630+0000","last_became_peered":"2023-10-16T12:27:01.268630+0000","last_unstale":"2023-10-16T12:27:02.249323+0000","last_undegraded":"2023-10-16T12:27:02.249323+0000","last_fullsized":"2023-10-16T12:27:02.249323+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:23:25.683414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:27:02.250838+0000","last_change":"2023-10-16T12:27:01.265337+0000","last_active":"2023-10-16T12:27:02.250838+0000","last_peered":"2023-10-16T12:27:02.250838+0000","last_clean":"2023-10-16T12:27:02.250838+0000","last_became_active":"2023-10-16T12:27:01.264574+0000","last_became_peered":"2023-10-16T12:27:01.264574+0000","last_unstale":"2023-10-16T12:27:02.250838+0000","last_undegraded":"2023-10-16T12:27:02.250838+0000","last_fullsized":"2023-10-16T12:27:02.250838+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:35:14.191443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.272228+0000","last_change":"2023-10-16T12:27:01.270411+0000","last_active":"2023-10-16T12:27:01.272228+0000","last_peered":"2023-10-16T12:27:01.272228+0000","last_clean":"2023-10-16T12:27:01.272228+0000","last_became_active":"2023-10-16T12:27:01.270119+0000","last_became_peered":"2023-10-16T12:27:01.270119+0000","last_unstale":"2023-10-16T12:27:01.272228+0000","last_undegraded":"2023-10-16T12:27:01.272228+0000","last_fullsized":"2023-10-16T12:27:01.272228+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:12:06.021778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:27:02.249153+0000","last_change":"2023-10-16T12:27:01.260934+0000","last_active":"2023-10-16T12:27:02.249153+0000","last_peered":"2023-10-16T12:27:02.249153+0000","last_clean":"2023-10-16T12:27:02.249153+0000","last_became_active":"2023-10-16T12:27:01.260649+0000","last_became_peered":"2023-10-16T12:27:01.260649+0000","last_unstale":"2023-10-16T12:27:02.249153+0000","last_undegraded":"2023-10-16T12:27:02.249153+0000","last_fullsized":"2023-10-16T12:27:02.249153+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:56:44.663382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.270533+0000","last_change":"2023-10-16T12:27:01.261518+0000","last_active":"2023-10-16T12:27:01.270533+0000","last_peered":"2023-10-16T12:27:01.270533+0000","last_clean":"2023-10-16T12:27:01.270533+0000","last_became_active":"2023-10-16T12:27:01.261019+0000","last_became_peered":"2023-10-16T12:27:01.261019+0000","last_unstale":"2023-10-16T12:27:01.270533+0000","last_undegraded":"2023-10-16T12:27:01.270533+0000","last_fullsized":"2023-10-16T12:27:01.270533+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:02:29.598278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.265919+0000","last_change":"2023-10-16T12:27:01.263781+0000","last_active":"2023-10-16T12:27:01.265919+0000","last_peered":"2023-10-16T12:27:01.265919+0000","last_clean":"2023-10-16T12:27:01.265919+0000","last_became_active":"2023-10-16T12:27:01.263479+0000","last_became_peered":"2023-10-16T12:27:01.263479+0000","last_unstale":"2023-10-16T12:27:01.265919+0000","last_undegraded":"2023-10-16T12:27:01.265919+0000","last_fullsized":"2023-10-16T12:27:01.265919+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:38:16.102632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.272706+0000","last_change":"2023-10-16T12:27:01.271026+0000","last_active":"2023-10-16T12:27:01.272706+0000","last_peered":"2023-10-16T12:27:01.272706+0000","last_clean":"2023-10-16T12:27:01.272706+0000","last_became_active":"2023-10-16T12:27:01.270554+0000","last_became_peered":"2023-10-16T12:27:01.270554+0000","last_unstale":"2023-10-16T12:27:01.272706+0000","last_undegraded":"2023-10-16T12:27:01.272706+0000","last_fullsized":"2023-10-16T12:27:01.272706+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:36:09.450050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.264875+0000","last_change":"2023-10-16T12:27:01.261928+0000","last_active":"2023-10-16T12:27:01.264875+0000","last_peered":"2023-10-16T12:27:01.264875+0000","last_clean":"2023-10-16T12:27:01.264875+0000","last_became_active":"2023-10-16T12:27:01.261505+0000","last_became_peered":"2023-10-16T12:27:01.261505+0000","last_unstale":"2023-10-16T12:27:01.264875+0000","last_undegraded":"2023-10-16T12:27:01.264875+0000","last_fullsized":"2023-10-16T12:27:01.264875+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:09:15.255373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:27:02.249446+0000","last_change":"2023-10-16T12:27:01.266636+0000","last_active":"2023-10-16T12:27:02.249446+0000","last_peered":"2023-10-16T12:27:02.249446+0000","last_clean":"2023-10-16T12:27:02.249446+0000","last_became_active":"2023-10-16T12:27:01.264686+0000","last_became_peered":"2023-10-16T12:27:01.264686+0000","last_unstale":"2023-10-16T12:27:02.249446+0000","last_undegraded":"2023-10-16T12:27:02.249446+0000","last_fullsized":"2023-10-16T12:27:02.249446+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:41:53.749202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:27:02.250700+0000","last_change":"2023-10-16T12:27:01.264860+0000","last_active":"2023-10-16T12:27:02.250700+0000","last_peered":"2023-10-16T12:27:02.250700+0000","last_clean":"2023-10-16T12:27:02.250700+0000","last_became_active":"2023-10-16T12:27:01.263874+0000","last_became_peered":"2023-10-16T12:27:01.263874+0000","last_unstale":"2023-10-16T12:27:02.250700+0000","last_undegraded":"2023-10-16T12:27:02.250700+0000","last_fullsized":"2023-10-16T12:27:02.250700+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:51:29.236436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.269053+0000","last_change":"2023-10-16T12:27:01.265757+0000","last_active":"2023-10-16T12:27:01.269053+0000","last_peered":"2023-10-16T12:27:01.269053+0000","last_clean":"2023-10-16T12:27:01.269053+0000","last_became_active":"2023-10-16T12:27:01.265432+0000","last_became_peered":"2023-10-16T12:27:01.265432+0000","last_unstale":"2023-10-16T12:27:01.269053+0000","last_undegraded":"2023-10-16T12:27:01.269053+0000","last_fullsized":"2023-10-16T12:27:01.269053+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:59:38.006434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.266198+0000","last_change":"2023-10-16T12:27:01.261829+0000","last_active":"2023-10-16T12:27:01.266198+0000","last_peered":"2023-10-16T12:27:01.266198+0000","last_clean":"2023-10-16T12:27:01.266198+0000","last_became_active":"2023-10-16T12:27:01.261535+0000","last_became_peered":"2023-10-16T12:27:01.261535+0000","last_unstale":"2023-10-16T12:27:01.266198+0000","last_undegraded":"2023-10-16T12:27:01.266198+0000","last_fullsized":"2023-10-16T12:27:01.266198+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:19:43.046725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:27:02.249714+0000","last_change":"2023-10-16T12:27:01.269571+0000","last_active":"2023-10-16T12:27:02.249714+0000","last_peered":"2023-10-16T12:27:02.249714+0000","last_clean":"2023-10-16T12:27:02.249714+0000","last_became_active":"2023-10-16T12:27:01.269289+0000","last_became_peered":"2023-10-16T12:27:01.269289+0000","last_unstale":"2023-10-16T12:27:02.249714+0000","last_undegraded":"2023-10-16T12:27:02.249714+0000","last_fullsized":"2023-10-16T12:27:02.249714+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:38:43.851700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.267227+0000","last_change":"2023-10-16T12:27:01.264429+0000","last_active":"2023-10-16T12:27:01.267227+0000","last_peered":"2023-10-16T12:27:01.267227+0000","last_clean":"2023-10-16T12:27:01.267227+0000","last_became_active":"2023-10-16T12:27:01.264105+0000","last_became_peered":"2023-10-16T12:27:01.264105+0000","last_unstale":"2023-10-16T12:27:01.267227+0000","last_undegraded":"2023-10-16T12:27:01.267227+0000","last_fullsized":"2023-10-16T12:27:01.267227+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:41:56.365816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:27:02.250977+0000","last_change":"2023-10-16T12:27:01.269257+0000","last_active":"2023-10-16T12:27:02.250977+0000","last_peered":"2023-10-16T12:27:02.250977+0000","last_clean":"2023-10-16T12:27:02.250977+0000","last_became_active":"2023-10-16T12:27:01.268958+0000","last_became_peered":"2023-10-16T12:27:01.268958+0000","last_unstale":"2023-10-16T12:27:02.250977+0000","last_undegraded":"2023-10-16T12:27:02.250977+0000","last_fullsized":"2023-10-16T12:27:02.250977+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:13:33.750568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:27:02.249634+0000","last_change":"2023-10-16T12:27:01.267116+0000","last_active":"2023-10-16T12:27:02.249634+0000","last_peered":"2023-10-16T12:27:02.249634+0000","last_clean":"2023-10-16T12:27:02.249634+0000","last_became_active":"2023-10-16T12:27:01.265971+0000","last_became_peered":"2023-10-16T12:27:01.265971+0000","last_unstale":"2023-10-16T12:27:02.249634+0000","last_undegraded":"2023-10-16T12:27:02.249634+0000","last_fullsized":"2023-10-16T12:27:02.249634+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:58:38.091004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.270725+0000","last_change":"2023-10-16T12:27:01.266804+0000","last_active":"2023-10-16T12:27:01.270725+0000","last_peered":"2023-10-16T12:27:01.270725+0000","last_clean":"2023-10-16T12:27:01.270725+0000","last_became_active":"2023-10-16T12:27:01.266533+0000","last_became_peered":"2023-10-16T12:27:01.266533+0000","last_unstale":"2023-10-16T12:27:01.270725+0000","last_undegraded":"2023-10-16T12:27:01.270725+0000","last_fullsized":"2023-10-16T12:27:01.270725+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:41:56.365816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:27:02.249495+0000","last_change":"2023-10-16T12:27:01.272204+0000","last_active":"2023-10-16T12:27:02.249495+0000","last_peered":"2023-10-16T12:27:02.249495+0000","last_clean":"2023-10-16T12:27:02.249495+0000","last_became_active":"2023-10-16T12:27:01.271941+0000","last_became_peered":"2023-10-16T12:27:01.271941+0000","last_unstale":"2023-10-16T12:27:02.249495+0000","last_undegraded":"2023-10-16T12:27:02.249495+0000","last_fullsized":"2023-10-16T12:27:02.249495+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:38:41.912039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.269229+0000","last_change":"2023-10-16T12:27:01.261162+0000","last_active":"2023-10-16T12:27:01.269229+0000","last_peered":"2023-10-16T12:27:01.269229+0000","last_clean":"2023-10-16T12:27:01.269229+0000","last_became_active":"2023-10-16T12:27:01.260843+0000","last_became_peered":"2023-10-16T12:27:01.260843+0000","last_unstale":"2023-10-16T12:27:01.269229+0000","last_undegraded":"2023-10-16T12:27:01.269229+0000","last_fullsized":"2023-10-16T12:27:01.269229+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:28:01.320829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.263240+0000","last_change":"2023-10-16T12:27:01.260912+0000","last_active":"2023-10-16T12:27:01.263240+0000","last_peered":"2023-10-16T12:27:01.263240+0000","last_clean":"2023-10-16T12:27:01.263240+0000","last_became_active":"2023-10-16T12:27:01.260625+0000","last_became_peered":"2023-10-16T12:27:01.260625+0000","last_unstale":"2023-10-16T12:27:01.263240+0000","last_undegraded":"2023-10-16T12:27:01.263240+0000","last_fullsized":"2023-10-16T12:27:01.263240+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:45:29.052520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.262437+0000","last_change":"2023-10-16T12:27:01.260078+0000","last_active":"2023-10-16T12:27:01.262437+0000","last_peered":"2023-10-16T12:27:01.262437+0000","last_clean":"2023-10-16T12:27:01.262437+0000","last_became_active":"2023-10-16T12:27:01.259751+0000","last_became_peered":"2023-10-16T12:27:01.259751+0000","last_unstale":"2023-10-16T12:27:01.262437+0000","last_undegraded":"2023-10-16T12:27:01.262437+0000","last_fullsized":"2023-10-16T12:27:01.262437+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:51:34.535878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.261115+0000","last_change":"2023-10-16T12:27:01.259396+0000","last_active":"2023-10-16T12:27:01.261115+0000","last_peered":"2023-10-16T12:27:01.261115+0000","last_clean":"2023-10-16T12:27:01.261115+0000","last_became_active":"2023-10-16T12:27:01.258993+0000","last_became_peered":"2023-10-16T12:27:01.258993+0000","last_unstale":"2023-10-16T12:27:01.261115+0000","last_undegraded":"2023-10-16T12:27:01.261115+0000","last_fullsized":"2023-10-16T12:27:01.261115+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:55:14.108031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.263102+0000","last_change":"2023-10-16T12:27:01.261039+0000","last_active":"2023-10-16T12:27:01.263102+0000","last_peered":"2023-10-16T12:27:01.263102+0000","last_clean":"2023-10-16T12:27:01.263102+0000","last_became_active":"2023-10-16T12:27:01.260755+0000","last_became_peered":"2023-10-16T12:27:01.260755+0000","last_unstale":"2023-10-16T12:27:01.263102+0000","last_undegraded":"2023-10-16T12:27:01.263102+0000","last_fullsized":"2023-10-16T12:27:01.263102+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:50:05.396430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:27:02.249756+0000","last_change":"2023-10-16T12:27:01.265675+0000","last_active":"2023-10-16T12:27:02.249756+0000","last_peered":"2023-10-16T12:27:02.249756+0000","last_clean":"2023-10-16T12:27:02.249756+0000","last_became_active":"2023-10-16T12:27:01.265354+0000","last_became_peered":"2023-10-16T12:27:01.265354+0000","last_unstale":"2023-10-16T12:27:02.249756+0000","last_undegraded":"2023-10-16T12:27:02.249756+0000","last_fullsized":"2023-10-16T12:27:02.249756+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:15:58.232499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.268428+0000","last_change":"2023-10-16T12:27:01.263522+0000","last_active":"2023-10-16T12:27:01.268428+0000","last_peered":"2023-10-16T12:27:01.268428+0000","last_clean":"2023-10-16T12:27:01.268428+0000","last_became_active":"2023-10-16T12:27:01.263175+0000","last_became_peered":"2023-10-16T12:27:01.263175+0000","last_unstale":"2023-10-16T12:27:01.268428+0000","last_undegraded":"2023-10-16T12:27:01.268428+0000","last_fullsized":"2023-10-16T12:27:01.268428+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:34:37.846730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.265597+0000","last_change":"2023-10-16T12:27:01.263563+0000","last_active":"2023-10-16T12:27:01.265597+0000","last_peered":"2023-10-16T12:27:01.265597+0000","last_clean":"2023-10-16T12:27:01.265597+0000","last_became_active":"2023-10-16T12:27:01.263280+0000","last_became_peered":"2023-10-16T12:27:01.263280+0000","last_unstale":"2023-10-16T12:27:01.265597+0000","last_undegraded":"2023-10-16T12:27:01.265597+0000","last_fullsized":"2023-10-16T12:27:01.265597+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:27:27.941593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.272572+0000","last_change":"2023-10-16T12:27:01.271183+0000","last_active":"2023-10-16T12:27:01.272572+0000","last_peered":"2023-10-16T12:27:01.272572+0000","last_clean":"2023-10-16T12:27:01.272572+0000","last_became_active":"2023-10-16T12:27:01.270895+0000","last_became_peered":"2023-10-16T12:27:01.270895+0000","last_unstale":"2023-10-16T12:27:01.272572+0000","last_undegraded":"2023-10-16T12:27:01.272572+0000","last_fullsized":"2023-10-16T12:27:01.272572+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:22:34.967887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.266219+0000","last_change":"2023-10-16T12:27:01.262809+0000","last_active":"2023-10-16T12:27:01.266219+0000","last_peered":"2023-10-16T12:27:01.266219+0000","last_clean":"2023-10-16T12:27:01.266219+0000","last_became_active":"2023-10-16T12:27:01.262457+0000","last_became_peered":"2023-10-16T12:27:01.262457+0000","last_unstale":"2023-10-16T12:27:01.266219+0000","last_undegraded":"2023-10-16T12:27:01.266219+0000","last_fullsized":"2023-10-16T12:27:01.266219+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:20:55.292512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.267462+0000","last_change":"2023-10-16T12:27:01.264439+0000","last_active":"2023-10-16T12:27:01.267462+0000","last_peered":"2023-10-16T12:27:01.267462+0000","last_clean":"2023-10-16T12:27:01.267462+0000","last_became_active":"2023-10-16T12:27:01.264127+0000","last_became_peered":"2023-10-16T12:27:01.264127+0000","last_unstale":"2023-10-16T12:27:01.267462+0000","last_undegraded":"2023-10-16T12:27:01.267462+0000","last_fullsized":"2023-10-16T12:27:01.267462+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:52:02.124855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"98'4","reported_seq":16,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:27:02.249570+0000","last_change":"2023-10-16T12:27:00.406941+0000","last_active":"2023-10-16T12:27:02.249570+0000","last_peered":"2023-10-16T12:27:02.249570+0000","last_clean":"2023-10-16T12:27:02.249570+0000","last_became_active":"2023-10-16T12:27:00.406664+0000","last_became_peered":"2023-10-16T12:27:00.406664+0000","last_unstale":"2023-10-16T12:27:02.249570+0000","last_undegraded":"2023-10-16T12:27:02.249570+0000","last_fullsized":"2023-10-16T12:27:02.249570+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_clean_scrub_stamp":"2023-10-16T12:26:59.382546+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:53:42.971928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.266165+0000","last_change":"2023-10-16T12:27:01.259191+0000","last_active":"2023-10-16T12:27:01.266165+0000","last_peered":"2023-10-16T12:27:01.266165+0000","last_clean":"2023-10-16T12:27:01.266165+0000","last_became_active":"2023-10-16T12:27:01.258896+0000","last_became_peered":"2023-10-16T12:27:01.258896+0000","last_unstale":"2023-10-16T12:27:01.266165+0000","last_undegraded":"2023-10-16T12:27:01.266165+0000","last_fullsized":"2023-10-16T12:27:01.266165+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:27:44.803461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"98'1","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.405359+0000","last_change":"2023-10-16T12:27:00.419326+0000","last_active":"2023-10-16T12:27:01.405359+0000","last_peered":"2023-10-16T12:27:01.405359+0000","last_clean":"2023-10-16T12:27:01.405359+0000","last_became_active":"2023-10-16T12:27:00.418899+0000","last_became_peered":"2023-10-16T12:27:00.418899+0000","last_unstale":"2023-10-16T12:27:01.405359+0000","last_undegraded":"2023-10-16T12:27:01.405359+0000","last_fullsized":"2023-10-16T12:27:01.405359+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_clean_scrub_stamp":"2023-10-16T12:26:59.382546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:10:41.359173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.271195+0000","last_change":"2023-10-16T12:27:01.268591+0000","last_active":"2023-10-16T12:27:01.271195+0000","last_peered":"2023-10-16T12:27:01.271195+0000","last_clean":"2023-10-16T12:27:01.271195+0000","last_became_active":"2023-10-16T12:27:01.267918+0000","last_became_peered":"2023-10-16T12:27:01.267918+0000","last_unstale":"2023-10-16T12:27:01.271195+0000","last_undegraded":"2023-10-16T12:27:01.271195+0000","last_fullsized":"2023-10-16T12:27:01.271195+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:35:50.007900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"98'2","reported_seq":13,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.404599+0000","last_change":"2023-10-16T12:27:00.411306+0000","last_active":"2023-10-16T12:27:01.404599+0000","last_peered":"2023-10-16T12:27:01.404599+0000","last_clean":"2023-10-16T12:27:01.404599+0000","last_became_active":"2023-10-16T12:27:00.411013+0000","last_became_peered":"2023-10-16T12:27:00.411013+0000","last_unstale":"2023-10-16T12:27:01.404599+0000","last_undegraded":"2023-10-16T12:27:01.404599+0000","last_fullsized":"2023-10-16T12:27:01.404599+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_clean_scrub_stamp":"2023-10-16T12:26:59.382546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:34:36.841255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.270703+0000","last_change":"2023-10-16T12:27:01.266535+0000","last_active":"2023-10-16T12:27:01.270703+0000","last_peered":"2023-10-16T12:27:01.270703+0000","last_clean":"2023-10-16T12:27:01.270703+0000","last_became_active":"2023-10-16T12:27:01.266233+0000","last_became_peered":"2023-10-16T12:27:01.266233+0000","last_unstale":"2023-10-16T12:27:01.270703+0000","last_undegraded":"2023-10-16T12:27:01.270703+0000","last_fullsized":"2023-10-16T12:27:01.270703+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:10:22.284373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"98'2","reported_seq":13,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.406134+0000","last_change":"2023-10-16T12:27:00.416224+0000","last_active":"2023-10-16T12:27:01.406134+0000","last_peered":"2023-10-16T12:27:01.406134+0000","last_clean":"2023-10-16T12:27:01.406134+0000","last_became_active":"2023-10-16T12:27:00.415718+0000","last_became_peered":"2023-10-16T12:27:00.415718+0000","last_unstale":"2023-10-16T12:27:01.406134+0000","last_undegraded":"2023-10-16T12:27:01.406134+0000","last_fullsized":"2023-10-16T12:27:01.406134+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_clean_scrub_stamp":"2023-10-16T12:26:59.382546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:35:49.002426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.272561+0000","last_change":"2023-10-16T12:27:01.270883+0000","last_active":"2023-10-16T12:27:01.272561+0000","last_peered":"2023-10-16T12:27:01.272561+0000","last_clean":"2023-10-16T12:27:01.272561+0000","last_became_active":"2023-10-16T12:27:01.270584+0000","last_became_peered":"2023-10-16T12:27:01.270584+0000","last_unstale":"2023-10-16T12:27:01.272561+0000","last_undegraded":"2023-10-16T12:27:01.272561+0000","last_fullsized":"2023-10-16T12:27:01.272561+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:15:40.122075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"98'1","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.406899+0000","last_change":"2023-10-16T12:27:00.410815+0000","last_active":"2023-10-16T12:27:01.406899+0000","last_peered":"2023-10-16T12:27:01.406899+0000","last_clean":"2023-10-16T12:27:01.406899+0000","last_became_active":"2023-10-16T12:27:00.410518+0000","last_became_peered":"2023-10-16T12:27:00.410518+0000","last_unstale":"2023-10-16T12:27:01.406899+0000","last_undegraded":"2023-10-16T12:27:01.406899+0000","last_fullsized":"2023-10-16T12:27:01.406899+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_clean_scrub_stamp":"2023-10-16T12:26:59.382546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:52:01.119381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.272686+0000","last_change":"2023-10-16T12:27:01.270848+0000","last_active":"2023-10-16T12:27:01.272686+0000","last_peered":"2023-10-16T12:27:01.272686+0000","last_clean":"2023-10-16T12:27:01.272686+0000","last_became_active":"2023-10-16T12:27:01.270061+0000","last_became_peered":"2023-10-16T12:27:01.270061+0000","last_unstale":"2023-10-16T12:27:01.272686+0000","last_undegraded":"2023-10-16T12:27:01.272686+0000","last_fullsized":"2023-10-16T12:27:01.272686+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:05:11.575127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"98'1","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.407682+0000","last_change":"2023-10-16T12:27:00.412976+0000","last_active":"2023-10-16T12:27:01.407682+0000","last_peered":"2023-10-16T12:27:01.407682+0000","last_clean":"2023-10-16T12:27:01.407682+0000","last_became_active":"2023-10-16T12:27:00.412553+0000","last_became_peered":"2023-10-16T12:27:00.412553+0000","last_unstale":"2023-10-16T12:27:01.407682+0000","last_undegraded":"2023-10-16T12:27:01.407682+0000","last_fullsized":"2023-10-16T12:27:01.407682+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_clean_scrub_stamp":"2023-10-16T12:26:59.382546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:10:59.683263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.268702+0000","last_change":"2023-10-16T12:27:01.259677+0000","last_active":"2023-10-16T12:27:01.268702+0000","last_peered":"2023-10-16T12:27:01.268702+0000","last_clean":"2023-10-16T12:27:01.268702+0000","last_became_active":"2023-10-16T12:27:01.259405+0000","last_became_peered":"2023-10-16T12:27:01.259405+0000","last_unstale":"2023-10-16T12:27:01.268702+0000","last_undegraded":"2023-10-16T12:27:01.268702+0000","last_fullsized":"2023-10-16T12:27:01.268702+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:51:29.215938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.248646+0000","last_change":"2023-10-16T12:27:00.407890+0000","last_active":"2023-10-16T12:27:01.248646+0000","last_peered":"2023-10-16T12:27:01.248646+0000","last_clean":"2023-10-16T12:27:01.248646+0000","last_became_active":"2023-10-16T12:27:00.407608+0000","last_became_peered":"2023-10-16T12:27:00.407608+0000","last_unstale":"2023-10-16T12:27:01.248646+0000","last_undegraded":"2023-10-16T12:27:01.248646+0000","last_fullsized":"2023-10-16T12:27:01.248646+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_clean_scrub_stamp":"2023-10-16T12:26:59.382546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:41:50.717258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.274937+0000","last_change":"2023-10-16T12:27:01.274537+0000","last_active":"2023-10-16T12:27:01.274937+0000","last_peered":"2023-10-16T12:27:01.274937+0000","last_clean":"2023-10-16T12:27:01.274937+0000","last_became_active":"2023-10-16T12:27:01.274290+0000","last_became_peered":"2023-10-16T12:27:01.274290+0000","last_unstale":"2023-10-16T12:27:01.274937+0000","last_undegraded":"2023-10-16T12:27:01.274937+0000","last_fullsized":"2023-10-16T12:27:01.274937+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:51:29.215938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:27:02.249196+0000","last_change":"2023-10-16T12:27:00.411391+0000","last_active":"2023-10-16T12:27:02.249196+0000","last_peered":"2023-10-16T12:27:02.249196+0000","last_clean":"2023-10-16T12:27:02.249196+0000","last_became_active":"2023-10-16T12:27:00.411135+0000","last_became_peered":"2023-10-16T12:27:00.411135+0000","last_unstale":"2023-10-16T12:27:02.249196+0000","last_undegraded":"2023-10-16T12:27:02.249196+0000","last_fullsized":"2023-10-16T12:27:02.249196+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_clean_scrub_stamp":"2023-10-16T12:26:59.382546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:55:25.767744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.263260+0000","last_change":"2023-10-16T12:27:01.262067+0000","last_active":"2023-10-16T12:27:01.263260+0000","last_peered":"2023-10-16T12:27:01.263260+0000","last_clean":"2023-10-16T12:27:01.263260+0000","last_became_active":"2023-10-16T12:27:01.260866+0000","last_became_peered":"2023-10-16T12:27:01.260866+0000","last_unstale":"2023-10-16T12:27:01.263260+0000","last_undegraded":"2023-10-16T12:27:01.263260+0000","last_fullsized":"2023-10-16T12:27:01.263260+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:23:13.033596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.255245+0000","last_change":"2023-10-16T12:27:00.417116+0000","last_active":"2023-10-16T12:27:01.255245+0000","last_peered":"2023-10-16T12:27:01.255245+0000","last_clean":"2023-10-16T12:27:01.255245+0000","last_became_active":"2023-10-16T12:27:00.415892+0000","last_became_peered":"2023-10-16T12:27:00.415892+0000","last_unstale":"2023-10-16T12:27:01.255245+0000","last_undegraded":"2023-10-16T12:27:01.255245+0000","last_fullsized":"2023-10-16T12:27:01.255245+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_clean_scrub_stamp":"2023-10-16T12:26:59.382546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:15:39.116600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.270165+0000","last_change":"2023-10-16T12:27:01.269361+0000","last_active":"2023-10-16T12:27:01.270165+0000","last_peered":"2023-10-16T12:27:01.270165+0000","last_clean":"2023-10-16T12:27:01.270165+0000","last_became_active":"2023-10-16T12:27:01.268996+0000","last_became_peered":"2023-10-16T12:27:01.268996+0000","last_unstale":"2023-10-16T12:27:01.270165+0000","last_undegraded":"2023-10-16T12:27:01.270165+0000","last_fullsized":"2023-10-16T12:27:01.270165+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:15:18.654877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.255044+0000","last_change":"2023-10-16T12:27:00.406244+0000","last_active":"2023-10-16T12:27:01.255044+0000","last_peered":"2023-10-16T12:27:01.255044+0000","last_clean":"2023-10-16T12:27:01.255044+0000","last_became_active":"2023-10-16T12:27:00.405950+0000","last_became_peered":"2023-10-16T12:27:00.405950+0000","last_unstale":"2023-10-16T12:27:01.255044+0000","last_undegraded":"2023-10-16T12:27:01.255044+0000","last_fullsized":"2023-10-16T12:27:01.255044+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_clean_scrub_stamp":"2023-10-16T12:26:59.382546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:10:08.160283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:27:02.250743+0000","last_change":"2023-10-16T12:27:01.265182+0000","last_active":"2023-10-16T12:27:02.250743+0000","last_peered":"2023-10-16T12:27:02.250743+0000","last_clean":"2023-10-16T12:27:02.250743+0000","last_became_active":"2023-10-16T12:27:01.264203+0000","last_became_peered":"2023-10-16T12:27:01.264203+0000","last_unstale":"2023-10-16T12:27:02.250743+0000","last_undegraded":"2023-10-16T12:27:02.250743+0000","last_fullsized":"2023-10-16T12:27:02.250743+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:41:49.065472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"98'3","reported_seq":15,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:27:02.249641+0000","last_change":"2023-10-16T12:27:00.416524+0000","last_active":"2023-10-16T12:27:02.249641+0000","last_peered":"2023-10-16T12:27:02.249641+0000","last_clean":"2023-10-16T12:27:02.249641+0000","last_became_active":"2023-10-16T12:27:00.416242+0000","last_became_peered":"2023-10-16T12:27:00.416242+0000","last_unstale":"2023-10-16T12:27:02.249641+0000","last_undegraded":"2023-10-16T12:27:02.249641+0000","last_fullsized":"2023-10-16T12:27:02.249641+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_clean_scrub_stamp":"2023-10-16T12:26:59.382546+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:04:14.487791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.267682+0000","last_change":"2023-10-16T12:27:01.265279+0000","last_active":"2023-10-16T12:27:01.267682+0000","last_peered":"2023-10-16T12:27:01.267682+0000","last_clean":"2023-10-16T12:27:01.267682+0000","last_became_active":"2023-10-16T12:27:01.264870+0000","last_became_peered":"2023-10-16T12:27:01.264870+0000","last_unstale":"2023-10-16T12:27:01.267682+0000","last_undegraded":"2023-10-16T12:27:01.267682+0000","last_fullsized":"2023-10-16T12:27:01.267682+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:10:22.284373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"98'6","reported_seq":17,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.410422+0000","last_change":"2023-10-16T12:27:00.415413+0000","last_active":"2023-10-16T12:27:01.410422+0000","last_peered":"2023-10-16T12:27:01.410422+0000","last_clean":"2023-10-16T12:27:01.410422+0000","last_became_active":"2023-10-16T12:27:00.413893+0000","last_became_peered":"2023-10-16T12:27:00.413893+0000","last_unstale":"2023-10-16T12:27:01.410422+0000","last_undegraded":"2023-10-16T12:27:01.410422+0000","last_fullsized":"2023-10-16T12:27:01.410422+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_clean_scrub_stamp":"2023-10-16T12:26:59.382546+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:02:28.592803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.267895+0000","last_change":"2023-10-16T12:27:01.265199+0000","last_active":"2023-10-16T12:27:01.267895+0000","last_peered":"2023-10-16T12:27:01.267895+0000","last_clean":"2023-10-16T12:27:01.267895+0000","last_became_active":"2023-10-16T12:27:01.264698+0000","last_became_peered":"2023-10-16T12:27:01.264698+0000","last_unstale":"2023-10-16T12:27:01.267895+0000","last_undegraded":"2023-10-16T12:27:01.267895+0000","last_fullsized":"2023-10-16T12:27:01.267895+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:49:04.902173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"98'2","reported_seq":13,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.409827+0000","last_change":"2023-10-16T12:27:00.411236+0000","last_active":"2023-10-16T12:27:01.409827+0000","last_peered":"2023-10-16T12:27:01.409827+0000","last_clean":"2023-10-16T12:27:01.409827+0000","last_became_active":"2023-10-16T12:27:00.410932+0000","last_became_peered":"2023-10-16T12:27:00.410932+0000","last_unstale":"2023-10-16T12:27:01.409827+0000","last_undegraded":"2023-10-16T12:27:01.409827+0000","last_fullsized":"2023-10-16T12:27:01.409827+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_clean_scrub_stamp":"2023-10-16T12:26:59.382546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:50:04.390955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.271789+0000","last_change":"2023-10-16T12:27:01.269536+0000","last_active":"2023-10-16T12:27:01.271789+0000","last_peered":"2023-10-16T12:27:01.271789+0000","last_clean":"2023-10-16T12:27:01.271789+0000","last_became_active":"2023-10-16T12:27:01.269161+0000","last_became_peered":"2023-10-16T12:27:01.269161+0000","last_unstale":"2023-10-16T12:27:01.271789+0000","last_undegraded":"2023-10-16T12:27:01.271789+0000","last_fullsized":"2023-10-16T12:27:01.271789+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:08:38.228680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.252902+0000","last_change":"2023-10-16T12:27:00.414135+0000","last_active":"2023-10-16T12:27:01.252902+0000","last_peered":"2023-10-16T12:27:01.252902+0000","last_clean":"2023-10-16T12:27:01.252902+0000","last_became_active":"2023-10-16T12:27:00.413815+0000","last_became_peered":"2023-10-16T12:27:00.413815+0000","last_unstale":"2023-10-16T12:27:01.252902+0000","last_undegraded":"2023-10-16T12:27:01.252902+0000","last_fullsized":"2023-10-16T12:27:01.252902+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_clean_scrub_stamp":"2023-10-16T12:26:59.382546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:41:50.717258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.266635+0000","last_change":"2023-10-16T12:27:01.263997+0000","last_active":"2023-10-16T12:27:01.266635+0000","last_peered":"2023-10-16T12:27:01.266635+0000","last_clean":"2023-10-16T12:27:01.266635+0000","last_became_active":"2023-10-16T12:27:01.263703+0000","last_became_peered":"2023-10-16T12:27:01.263703+0000","last_unstale":"2023-10-16T12:27:01.266635+0000","last_undegraded":"2023-10-16T12:27:01.266635+0000","last_fullsized":"2023-10-16T12:27:01.266635+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:52:41.515619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:27:02.249340+0000","last_change":"2023-10-16T12:27:00.409189+0000","last_active":"2023-10-16T12:27:02.249340+0000","last_peered":"2023-10-16T12:27:02.249340+0000","last_clean":"2023-10-16T12:27:02.249340+0000","last_became_active":"2023-10-16T12:27:00.408912+0000","last_became_peered":"2023-10-16T12:27:00.408912+0000","last_unstale":"2023-10-16T12:27:02.249340+0000","last_undegraded":"2023-10-16T12:27:02.249340+0000","last_fullsized":"2023-10-16T12:27:02.249340+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_clean_scrub_stamp":"2023-10-16T12:26:59.382546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:51:31.033168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.269773+0000","last_change":"2023-10-16T12:27:01.265110+0000","last_active":"2023-10-16T12:27:01.269773+0000","last_peered":"2023-10-16T12:27:01.269773+0000","last_clean":"2023-10-16T12:27:01.269773+0000","last_became_active":"2023-10-16T12:27:01.264576+0000","last_became_peered":"2023-10-16T12:27:01.264576+0000","last_unstale":"2023-10-16T12:27:01.269773+0000","last_undegraded":"2023-10-16T12:27:01.269773+0000","last_fullsized":"2023-10-16T12:27:01.269773+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:45:52.773246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"98'8","reported_seq":19,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.408703+0000","last_change":"2023-10-16T12:27:00.419332+0000","last_active":"2023-10-16T12:27:01.408703+0000","last_peered":"2023-10-16T12:27:01.408703+0000","last_clean":"2023-10-16T12:27:01.408703+0000","last_became_active":"2023-10-16T12:27:00.419020+0000","last_became_peered":"2023-10-16T12:27:00.419020+0000","last_unstale":"2023-10-16T12:27:01.408703+0000","last_undegraded":"2023-10-16T12:27:01.408703+0000","last_fullsized":"2023-10-16T12:27:01.408703+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_clean_scrub_stamp":"2023-10-16T12:26:59.382546+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:20:54.287038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:27:02.250428+0000","last_change":"2023-10-16T12:27:01.261823+0000","last_active":"2023-10-16T12:27:02.250428+0000","last_peered":"2023-10-16T12:27:02.250428+0000","last_clean":"2023-10-16T12:27:02.250428+0000","last_became_active":"2023-10-16T12:27:01.261469+0000","last_became_peered":"2023-10-16T12:27:01.261469+0000","last_unstale":"2023-10-16T12:27:02.250428+0000","last_undegraded":"2023-10-16T12:27:02.250428+0000","last_fullsized":"2023-10-16T12:27:02.250428+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:20:32.306289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"98'1","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.404619+0000","last_change":"2023-10-16T12:27:00.416749+0000","last_active":"2023-10-16T12:27:01.404619+0000","last_peered":"2023-10-16T12:27:01.404619+0000","last_clean":"2023-10-16T12:27:01.404619+0000","last_became_active":"2023-10-16T12:27:00.416458+0000","last_became_peered":"2023-10-16T12:27:00.416458+0000","last_unstale":"2023-10-16T12:27:01.404619+0000","last_undegraded":"2023-10-16T12:27:01.404619+0000","last_fullsized":"2023-10-16T12:27:01.404619+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_clean_scrub_stamp":"2023-10-16T12:26:59.382546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:27:43.797986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:27:02.249251+0000","last_change":"2023-10-16T12:27:01.269914+0000","last_active":"2023-10-16T12:27:02.249251+0000","last_peered":"2023-10-16T12:27:02.249251+0000","last_clean":"2023-10-16T12:27:02.249251+0000","last_became_active":"2023-10-16T12:27:01.269570+0000","last_became_peered":"2023-10-16T12:27:01.269570+0000","last_unstale":"2023-10-16T12:27:02.249251+0000","last_undegraded":"2023-10-16T12:27:02.249251+0000","last_fullsized":"2023-10-16T12:27:02.249251+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:38:41.223094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"98'2","reported_seq":13,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.406545+0000","last_change":"2023-10-16T12:27:00.413690+0000","last_active":"2023-10-16T12:27:01.406545+0000","last_peered":"2023-10-16T12:27:01.406545+0000","last_clean":"2023-10-16T12:27:01.406545+0000","last_became_active":"2023-10-16T12:27:00.413411+0000","last_became_peered":"2023-10-16T12:27:00.413411+0000","last_unstale":"2023-10-16T12:27:01.406545+0000","last_undegraded":"2023-10-16T12:27:01.406545+0000","last_fullsized":"2023-10-16T12:27:01.406545+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_clean_scrub_stamp":"2023-10-16T12:26:59.382546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:25:01.715586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:27:02.249353+0000","last_change":"2023-10-16T12:27:01.260408+0000","last_active":"2023-10-16T12:27:02.249353+0000","last_peered":"2023-10-16T12:27:02.249353+0000","last_clean":"2023-10-16T12:27:02.249353+0000","last_became_active":"2023-10-16T12:27:01.260131+0000","last_became_peered":"2023-10-16T12:27:01.260131+0000","last_unstale":"2023-10-16T12:27:02.249353+0000","last_undegraded":"2023-10-16T12:27:02.249353+0000","last_fullsized":"2023-10-16T12:27:02.249353+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:25:08.494549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"98'1","reported_seq":13,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:27:02.249674+0000","last_change":"2023-10-16T12:27:00.411188+0000","last_active":"2023-10-16T12:27:02.249674+0000","last_peered":"2023-10-16T12:27:02.249674+0000","last_clean":"2023-10-16T12:27:02.249674+0000","last_became_active":"2023-10-16T12:27:00.410747+0000","last_became_peered":"2023-10-16T12:27:00.410747+0000","last_unstale":"2023-10-16T12:27:02.249674+0000","last_undegraded":"2023-10-16T12:27:02.249674+0000","last_fullsized":"2023-10-16T12:27:02.249674+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_clean_scrub_stamp":"2023-10-16T12:26:59.382546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:22:25.427269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:27:02.250564+0000","last_change":"2023-10-16T12:27:01.262106+0000","last_active":"2023-10-16T12:27:02.250564+0000","last_peered":"2023-10-16T12:27:02.250564+0000","last_clean":"2023-10-16T12:27:02.250564+0000","last_became_active":"2023-10-16T12:27:01.261598+0000","last_became_peered":"2023-10-16T12:27:01.261598+0000","last_unstale":"2023-10-16T12:27:02.250564+0000","last_undegraded":"2023-10-16T12:27:02.250564+0000","last_fullsized":"2023-10-16T12:27:02.250564+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:46:55.924586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.252747+0000","last_change":"2023-10-16T12:27:00.416466+0000","last_active":"2023-10-16T12:27:01.252747+0000","last_peered":"2023-10-16T12:27:01.252747+0000","last_clean":"2023-10-16T12:27:01.252747+0000","last_became_active":"2023-10-16T12:27:00.416179+0000","last_became_peered":"2023-10-16T12:27:00.416179+0000","last_unstale":"2023-10-16T12:27:01.252747+0000","last_undegraded":"2023-10-16T12:27:01.252747+0000","last_fullsized":"2023-10-16T12:27:01.252747+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_clean_scrub_stamp":"2023-10-16T12:26:59.382546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:35:49.002426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.269359+0000","last_change":"2023-10-16T12:27:01.266417+0000","last_active":"2023-10-16T12:27:01.269359+0000","last_peered":"2023-10-16T12:27:01.269359+0000","last_clean":"2023-10-16T12:27:01.269359+0000","last_became_active":"2023-10-16T12:27:01.265856+0000","last_became_peered":"2023-10-16T12:27:01.265856+0000","last_unstale":"2023-10-16T12:27:01.269359+0000","last_undegraded":"2023-10-16T12:27:01.269359+0000","last_fullsized":"2023-10-16T12:27:01.269359+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:54:00.495639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"98'2","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:27:02.250554+0000","last_change":"2023-10-16T12:27:00.414010+0000","last_active":"2023-10-16T12:27:02.250554+0000","last_peered":"2023-10-16T12:27:02.250554+0000","last_clean":"2023-10-16T12:27:02.250554+0000","last_became_active":"2023-10-16T12:27:00.413709+0000","last_became_peered":"2023-10-16T12:27:00.413709+0000","last_unstale":"2023-10-16T12:27:02.250554+0000","last_undegraded":"2023-10-16T12:27:02.250554+0000","last_fullsized":"2023-10-16T12:27:02.250554+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_clean_scrub_stamp":"2023-10-16T12:26:59.382546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:57:05.272093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.270668+0000","last_change":"2023-10-16T12:27:01.267765+0000","last_active":"2023-10-16T12:27:01.270668+0000","last_peered":"2023-10-16T12:27:01.270668+0000","last_clean":"2023-10-16T12:27:01.270668+0000","last_became_active":"2023-10-16T12:27:01.267456+0000","last_became_peered":"2023-10-16T12:27:01.267456+0000","last_unstale":"2023-10-16T12:27:01.270668+0000","last_undegraded":"2023-10-16T12:27:01.270668+0000","last_fullsized":"2023-10-16T12:27:01.270668+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:46:52.865162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:27:02.250302+0000","last_change":"2023-10-16T12:27:00.420678+0000","last_active":"2023-10-16T12:27:02.250302+0000","last_peered":"2023-10-16T12:27:02.250302+0000","last_clean":"2023-10-16T12:27:02.250302+0000","last_became_active":"2023-10-16T12:27:00.420342+0000","last_became_peered":"2023-10-16T12:27:00.420342+0000","last_unstale":"2023-10-16T12:27:02.250302+0000","last_undegraded":"2023-10-16T12:27:02.250302+0000","last_fullsized":"2023-10-16T12:27:02.250302+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_clean_scrub_stamp":"2023-10-16T12:26:59.382546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:29:03.134564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.269751+0000","last_change":"2023-10-16T12:27:01.264901+0000","last_active":"2023-10-16T12:27:01.269751+0000","last_peered":"2023-10-16T12:27:01.269751+0000","last_clean":"2023-10-16T12:27:01.269751+0000","last_became_active":"2023-10-16T12:27:01.263217+0000","last_became_peered":"2023-10-16T12:27:01.263217+0000","last_unstale":"2023-10-16T12:27:01.269751+0000","last_undegraded":"2023-10-16T12:27:01.269751+0000","last_fullsized":"2023-10-16T12:27:01.269751+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:19:03.776179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:27:02.250516+0000","last_change":"2023-10-16T12:27:00.419553+0000","last_active":"2023-10-16T12:27:02.250516+0000","last_peered":"2023-10-16T12:27:02.250516+0000","last_clean":"2023-10-16T12:27:02.250516+0000","last_became_active":"2023-10-16T12:27:00.419169+0000","last_became_peered":"2023-10-16T12:27:00.419169+0000","last_unstale":"2023-10-16T12:27:02.250516+0000","last_undegraded":"2023-10-16T12:27:02.250516+0000","last_fullsized":"2023-10-16T12:27:02.250516+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_clean_scrub_stamp":"2023-10-16T12:26:59.382546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:53:59.490165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.270603+0000","last_change":"2023-10-16T12:27:01.263865+0000","last_active":"2023-10-16T12:27:01.270603+0000","last_peered":"2023-10-16T12:27:01.270603+0000","last_clean":"2023-10-16T12:27:01.270603+0000","last_became_active":"2023-10-16T12:27:01.263521+0000","last_became_peered":"2023-10-16T12:27:01.263521+0000","last_unstale":"2023-10-16T12:27:01.270603+0000","last_undegraded":"2023-10-16T12:27:01.270603+0000","last_fullsized":"2023-10-16T12:27:01.270603+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:19:03.776179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"98'2","reported_seq":13,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.408834+0000","last_change":"2023-10-16T12:27:00.416130+0000","last_active":"2023-10-16T12:27:01.408834+0000","last_peered":"2023-10-16T12:27:01.408834+0000","last_clean":"2023-10-16T12:27:01.408834+0000","last_became_active":"2023-10-16T12:27:00.415843+0000","last_became_peered":"2023-10-16T12:27:00.415843+0000","last_unstale":"2023-10-16T12:27:01.408834+0000","last_undegraded":"2023-10-16T12:27:01.408834+0000","last_fullsized":"2023-10-16T12:27:01.408834+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_clean_scrub_stamp":"2023-10-16T12:26:59.382546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:27:26.936118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.266694+0000","last_change":"2023-10-16T12:27:01.263778+0000","last_active":"2023-10-16T12:27:01.266694+0000","last_peered":"2023-10-16T12:27:01.266694+0000","last_clean":"2023-10-16T12:27:01.266694+0000","last_became_active":"2023-10-16T12:27:01.263486+0000","last_became_peered":"2023-10-16T12:27:01.263486+0000","last_unstale":"2023-10-16T12:27:01.266694+0000","last_undegraded":"2023-10-16T12:27:01.266694+0000","last_fullsized":"2023-10-16T12:27:01.266694+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:12:51.949103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.249803+0000","last_change":"2023-10-16T12:27:00.409609+0000","last_active":"2023-10-16T12:27:01.249803+0000","last_peered":"2023-10-16T12:27:01.249803+0000","last_clean":"2023-10-16T12:27:01.249803+0000","last_became_active":"2023-10-16T12:27:00.409170+0000","last_became_peered":"2023-10-16T12:27:00.409170+0000","last_unstale":"2023-10-16T12:27:01.249803+0000","last_undegraded":"2023-10-16T12:27:01.249803+0000","last_fullsized":"2023-10-16T12:27:01.249803+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_clean_scrub_stamp":"2023-10-16T12:26:59.382546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:29:03.134564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.271174+0000","last_change":"2023-10-16T12:27:01.268373+0000","last_active":"2023-10-16T12:27:01.271174+0000","last_peered":"2023-10-16T12:27:01.271174+0000","last_clean":"2023-10-16T12:27:01.271174+0000","last_became_active":"2023-10-16T12:27:01.267507+0000","last_became_peered":"2023-10-16T12:27:01.267507+0000","last_unstale":"2023-10-16T12:27:01.271174+0000","last_undegraded":"2023-10-16T12:27:01.271174+0000","last_fullsized":"2023-10-16T12:27:01.271174+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:22:34.967887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.252621+0000","last_change":"2023-10-16T12:27:00.410724+0000","last_active":"2023-10-16T12:27:01.252621+0000","last_peered":"2023-10-16T12:27:01.252621+0000","last_clean":"2023-10-16T12:27:01.252621+0000","last_became_active":"2023-10-16T12:27:00.410401+0000","last_became_peered":"2023-10-16T12:27:00.410401+0000","last_unstale":"2023-10-16T12:27:01.252621+0000","last_undegraded":"2023-10-16T12:27:01.252621+0000","last_fullsized":"2023-10-16T12:27:01.252621+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_clean_scrub_stamp":"2023-10-16T12:26:59.382546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:45:28.047046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.263350+0000","last_change":"2023-10-16T12:27:01.261039+0000","last_active":"2023-10-16T12:27:01.263350+0000","last_peered":"2023-10-16T12:27:01.263350+0000","last_clean":"2023-10-16T12:27:01.263350+0000","last_became_active":"2023-10-16T12:27:01.260229+0000","last_became_peered":"2023-10-16T12:27:01.260229+0000","last_unstale":"2023-10-16T12:27:01.263350+0000","last_undegraded":"2023-10-16T12:27:01.263350+0000","last_fullsized":"2023-10-16T12:27:01.263350+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:19:43.046725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.251987+0000","last_change":"2023-10-16T12:27:00.419567+0000","last_active":"2023-10-16T12:27:01.251987+0000","last_peered":"2023-10-16T12:27:01.251987+0000","last_clean":"2023-10-16T12:27:01.251987+0000","last_became_active":"2023-10-16T12:27:00.418977+0000","last_became_peered":"2023-10-16T12:27:00.418977+0000","last_unstale":"2023-10-16T12:27:01.251987+0000","last_undegraded":"2023-10-16T12:27:01.251987+0000","last_fullsized":"2023-10-16T12:27:01.251987+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_clean_scrub_stamp":"2023-10-16T12:26:59.382546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:10:08.160283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.269734+0000","last_change":"2023-10-16T12:27:01.264955+0000","last_active":"2023-10-16T12:27:01.269734+0000","last_peered":"2023-10-16T12:27:01.269734+0000","last_clean":"2023-10-16T12:27:01.269734+0000","last_became_active":"2023-10-16T12:27:01.264200+0000","last_became_peered":"2023-10-16T12:27:01.264200+0000","last_unstale":"2023-10-16T12:27:01.269734+0000","last_undegraded":"2023-10-16T12:27:01.269734+0000","last_fullsized":"2023-10-16T12:27:01.269734+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:28:01.320829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.259535+0000","last_change":"2023-10-16T12:27:00.416980+0000","last_active":"2023-10-16T12:27:01.259535+0000","last_peered":"2023-10-16T12:27:01.259535+0000","last_clean":"2023-10-16T12:27:01.259535+0000","last_became_active":"2023-10-16T12:27:00.416674+0000","last_became_peered":"2023-10-16T12:27:00.416674+0000","last_unstale":"2023-10-16T12:27:01.259535+0000","last_undegraded":"2023-10-16T12:27:01.259535+0000","last_fullsized":"2023-10-16T12:27:01.259535+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_clean_scrub_stamp":"2023-10-16T12:26:59.382546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:22:33.962412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.275091+0000","last_change":"2023-10-16T12:27:01.274531+0000","last_active":"2023-10-16T12:27:01.275091+0000","last_peered":"2023-10-16T12:27:01.275091+0000","last_clean":"2023-10-16T12:27:01.275091+0000","last_became_active":"2023-10-16T12:27:01.274232+0000","last_became_peered":"2023-10-16T12:27:01.274232+0000","last_unstale":"2023-10-16T12:27:01.275091+0000","last_undegraded":"2023-10-16T12:27:01.275091+0000","last_fullsized":"2023-10-16T12:27:01.275091+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:46:37.809954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:27:02.249451+0000","last_change":"2023-10-16T12:27:00.405523+0000","last_active":"2023-10-16T12:27:02.249451+0000","last_peered":"2023-10-16T12:27:02.249451+0000","last_clean":"2023-10-16T12:27:02.249451+0000","last_became_active":"2023-10-16T12:27:00.405055+0000","last_became_peered":"2023-10-16T12:27:00.405055+0000","last_unstale":"2023-10-16T12:27:02.249451+0000","last_undegraded":"2023-10-16T12:27:02.249451+0000","last_fullsized":"2023-10-16T12:27:02.249451+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_clean_scrub_stamp":"2023-10-16T12:26:59.382546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:59:25.573397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.267195+0000","last_change":"2023-10-16T12:27:01.263608+0000","last_active":"2023-10-16T12:27:01.267195+0000","last_peered":"2023-10-16T12:27:01.267195+0000","last_clean":"2023-10-16T12:27:01.267195+0000","last_became_active":"2023-10-16T12:27:01.263334+0000","last_became_peered":"2023-10-16T12:27:01.263334+0000","last_unstale":"2023-10-16T12:27:01.267195+0000","last_undegraded":"2023-10-16T12:27:01.267195+0000","last_fullsized":"2023-10-16T12:27:01.267195+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:09:01.408305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.259744+0000","last_change":"2023-10-16T12:27:00.415906+0000","last_active":"2023-10-16T12:27:01.259744+0000","last_peered":"2023-10-16T12:27:01.259744+0000","last_clean":"2023-10-16T12:27:01.259744+0000","last_became_active":"2023-10-16T12:27:00.415600+0000","last_became_peered":"2023-10-16T12:27:00.415600+0000","last_unstale":"2023-10-16T12:27:01.259744+0000","last_undegraded":"2023-10-16T12:27:01.259744+0000","last_fullsized":"2023-10-16T12:27:01.259744+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_clean_scrub_stamp":"2023-10-16T12:26:59.382546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:25:01.715586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.267105+0000","last_change":"2023-10-16T12:27:01.263040+0000","last_active":"2023-10-16T12:27:01.267105+0000","last_peered":"2023-10-16T12:27:01.267105+0000","last_clean":"2023-10-16T12:27:01.267105+0000","last_became_active":"2023-10-16T12:27:01.262735+0000","last_became_peered":"2023-10-16T12:27:01.262735+0000","last_unstale":"2023-10-16T12:27:01.267105+0000","last_undegraded":"2023-10-16T12:27:01.267105+0000","last_fullsized":"2023-10-16T12:27:01.267105+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:25:02.721060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.254772+0000","last_change":"2023-10-16T12:27:00.413071+0000","last_active":"2023-10-16T12:27:01.254772+0000","last_peered":"2023-10-16T12:27:01.254772+0000","last_clean":"2023-10-16T12:27:01.254772+0000","last_became_active":"2023-10-16T12:27:00.412744+0000","last_became_peered":"2023-10-16T12:27:00.412744+0000","last_unstale":"2023-10-16T12:27:01.254772+0000","last_undegraded":"2023-10-16T12:27:01.254772+0000","last_fullsized":"2023-10-16T12:27:01.254772+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_clean_scrub_stamp":"2023-10-16T12:26:59.382546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:17:19.746182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":9,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-16T12:27:01.268562+0000","last_change":"2023-10-16T12:27:01.265090+0000","last_active":"2023-10-16T12:27:01.268562+0000","last_peered":"2023-10-16T12:27:01.268562+0000","last_clean":"2023-10-16T12:27:01.268562+0000","last_became_active":"2023-10-16T12:27:01.264757+0000","last_became_peered":"2023-10-16T12:27:01.264757+0000","last_unstale":"2023-10-16T12:27:01.268562+0000","last_undegraded":"2023-10-16T12:27:01.268562+0000","last_fullsized":"2023-10-16T12:27:01.268562+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:00.388021+0000","last_clean_scrub_stamp":"2023-10-16T12:27:00.388021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:59:38.006434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-16T12:27:02.250482+0000","last_change":"2023-10-16T12:27:00.419932+0000","last_active":"2023-10-16T12:27:02.250482+0000","last_peered":"2023-10-16T12:27:02.250482+0000","last_clean":"2023-10-16T12:27:02.250482+0000","last_became_active":"2023-10-16T12:27:00.419643+0000","last_became_peered":"2023-10-16T12:27:00.419643+0000","last_unstale":"2023-10-16T12:27:02.250482+0000","last_undegraded":"2023-10-16T12:27:02.250482+0000","last_fullsized":"2023-10-16T12:27:02.250482+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:26:59.382546+0000","last_clean_scrub_stamp":"2023-10-16T12:26:59.382546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:05:10.569652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"88'172","reported_seq":436,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-16T12:27:00.402330+0000","last_change":"2023-10-16T12:10:34.746820+0000","last_active":"2023-10-16T12:27:00.402330+0000","last_peered":"2023-10-16T12:27:00.402330+0000","last_clean":"2023-10-16T12:27:00.402330+0000","last_became_active":"2023-10-16T12:10:34.745574+0000","last_became_peered":"2023-10-16T12:10:34.745574+0000","last_unstale":"2023-10-16T12:27:00.402330+0000","last_undegraded":"2023-10-16T12:27:00.402330+0000","last_fullsized":"2023-10-16T12:27:00.402330+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_clean_scrub_stamp":"2023-10-16T12:10:33.577111+0000","objects_scrubbed":0,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:14.461986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":596,"num_read_kb":1130,"num_write":259,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":596,"num_read_kb":1130,"num_write":259,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":172,"ondisk_log_size":172,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575047,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59308,"kb_used_data":1756,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312532,"statfs":{"total":96636764160,"available":96576032768,"internally_reserved":0,"allocated":1798144,"data_stored":1189923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58915275},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:26:05 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.59699999999999998,"15min":0.58499999999999996},"min":{"1min":0.437,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.59399999999999997,"15min":0.58199999999999996},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.312},"max":{"1min":0.85299999999999998,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.50600000000000001}]},{"osd":1,"last update":"Mon Oct 16 12:26:05 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.60399999999999998,"15min":0.59999999999999998},"min":{"1min":0.36099999999999999,"5min":0.35999999999999999,"15min":0.247},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.62,"15min":0.60099999999999998},"min":{"1min":0.45600000000000002,"5min":0.33200000000000002,"15min":0.32400000000000001},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.76700000000000002}]},{"osd":2,"last update":"Mon Oct 16 12:26:05 2023","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.67600000000000005,"15min":0.65600000000000003},"min":{"1min":0.53700000000000003,"5min":0.40699999999999997,"15min":0.36099999999999999},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.68799999999999994,"15min":0.66600000000000004},"min":{"1min":0.495,"5min":0.376,"15min":0.36699999999999999},"max":{"1min":0.94599999999999995,"5min":0.94799999999999995,"15min":0.96699999999999997},"last":0.88}]},{"osd":3,"last update":"Mon Oct 16 12:26:05 2023","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.63,"15min":0.61899999999999999},"min":{"1min":0.43099999999999999,"5min":0.35499999999999998,"15min":0.34899999999999998},"max":{"1min":0.91200000000000003,"5min":0.92200000000000004,"15min":1.1000000000000001},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.65700000000000003,"15min":0.63900000000000001},"min":{"1min":0.52200000000000002,"5min":0.376,"15min":0.376},"max":{"1min":0.996,"5min":1.2509999999999999,"15min":1.2509999999999999},"last":0.84999999999999998}]},{"osd":4,"last update":"Mon Oct 16 12:26:05 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53800000000000003,"15min":0.53100000000000003},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.28499999999999998},"max":{"1min":0.72699999999999998,"5min":1.383,"15min":1.383},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.625,"15min":0.60699999999999998},"min":{"1min":0.40799999999999997,"5min":0.36299999999999999,"15min":0.307},"max":{"1min":0.86599999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.93600000000000005}]},{"osd":5,"last update":"Mon Oct 16 12:26:05 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.55800000000000005,"15min":0.54300000000000004},"min":{"1min":0.40000000000000002,"5min":0.314,"15min":0.29799999999999999},"max":{"1min":0.84299999999999997,"5min":1.3109999999999999,"15min":1.3109999999999999},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.59399999999999997,"15min":0.58699999999999997},"min":{"1min":0.41799999999999998,"5min":0.33500000000000002,"15min":0.32400000000000001},"max":{"1min":0.83899999999999997,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":0.72699999999999998}]},{"osd":6,"last update":"Mon Oct 16 12:26:05 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.63500000000000001,"15min":0.621},"min":{"1min":0.44700000000000001,"5min":0.39300000000000002,"15min":0.36399999999999999},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.65600000000000003,"15min":0.63100000000000001},"min":{"1min":0.443,"5min":0.34599999999999997,"15min":0.32300000000000001},"max":{"1min":0.88300000000000001,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.97199999999999998}]}]},{"osd":6,"up_from":13,"seq":55834575045,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32368,"kb_used_data":1184,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339472,"statfs":{"total":96636764160,"available":96603619328,"internally_reserved":0,"allocated":1212416,"data_stored":601145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:25:58 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.60099999999999998,"15min":0.628},"min":{"1min":0.34200000000000003,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.88200000000000001,"5min":1.196,"15min":5.9349999999999996},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60499999999999998,"15min":0.64000000000000001},"min":{"1min":0.44900000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.81799999999999995,"5min":0.997,"15min":5.9489999999999998},"last":0.46800000000000003}]},{"osd":1,"last update":"Mon Oct 16 12:25:58 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.55500000000000005,"15min":0.57499999999999996},"min":{"1min":0.40200000000000002,"5min":0.32800000000000001,"15min":0.29399999999999998},"max":{"1min":0.71199999999999997,"5min":0.83899999999999997,"15min":0.875},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57899999999999996,"15min":0.59199999999999997},"min":{"1min":0.44500000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.83099999999999996,"5min":1.091,"15min":1.091},"last":0.55700000000000005}]},{"osd":2,"last update":"Mon Oct 16 12:25:58 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.57799999999999996,"15min":0.61799999999999999},"min":{"1min":0.41699999999999998,"5min":0.36199999999999999,"15min":0.34499999999999997},"max":{"1min":0.69199999999999995,"5min":0.91200000000000003,"15min":8.0489999999999995},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.57699999999999996,"15min":0.60899999999999999},"min":{"1min":0.41099999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.76300000000000001,"5min":1.1539999999999999,"15min":2.855},"last":0.623}]},{"osd":3,"last update":"Mon Oct 16 12:25:58 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.60599999999999998,"15min":0.629},"min":{"1min":0.45500000000000002,"5min":0.34200000000000003,"15min":0.307},"max":{"1min":0.85399999999999998,"5min":0.93999999999999995,"15min":1.083},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67400000000000004,"15min":0.67900000000000005},"min":{"1min":0.48099999999999998,"5min":0.40899999999999997,"15min":0.36599999999999999},"max":{"1min":0.94499999999999995,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.755}]},{"osd":4,"last update":"Mon Oct 16 12:25:58 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.52800000000000002,"15min":0.52000000000000002},"min":{"1min":0.33900000000000002,"5min":0.308,"15min":0.29899999999999999},"max":{"1min":0.83399999999999996,"5min":0.85599999999999998,"15min":0.88200000000000001},"last":0.379},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.60299999999999998,"15min":0.59399999999999997},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.30199999999999999},"max":{"1min":0.78500000000000003,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.69499999999999995}]},{"osd":5,"last update":"Mon Oct 16 12:25:58 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59099999999999997,"15min":0.57799999999999996},"min":{"1min":0.39900000000000002,"5min":0.34799999999999998,"15min":0.30099999999999999},"max":{"1min":0.80100000000000005,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63400000000000001,"15min":0.624},"min":{"1min":0.39400000000000002,"5min":0.35099999999999998,"15min":0.311},"max":{"1min":0.81599999999999995,"5min":1.3540000000000001,"15min":1.3540000000000001},"last":0.78000000000000003}]},{"osd":7,"last update":"Mon Oct 16 12:25:58 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59499999999999997,"15min":0.57499999999999996},"min":{"1min":0.36499999999999999,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.86599999999999999,"5min":1.29,"15min":1.29},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.62,"5min":0.625,"15min":0.61799999999999999},"min":{"1min":0.38800000000000001,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.84399999999999997,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.59899999999999998}]}]},{"osd":1,"up_from":13,"seq":55834575045,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32360,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339480,"statfs":{"total":96636764160,"available":96603627520,"internally_reserved":0,"allocated":1208320,"data_stored":599645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":31914441},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:27:00 2023","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.36899999999999999,"15min":0.376},"min":{"1min":0.246,"5min":0.21299999999999999,"15min":0.186},"max":{"1min":0.53300000000000003,"5min":0.63600000000000001,"15min":0.66000000000000003},"last":0.246},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.44500000000000001,"15min":0.45800000000000002},"min":{"1min":0.28399999999999997,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.502,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.31900000000000001}]},{"osd":2,"last update":"Mon Oct 16 12:27:00 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.40500000000000003,"15min":0.42599999999999999},"min":{"1min":0.251,"5min":0.25,"15min":0.215},"max":{"1min":0.45000000000000001,"5min":0.69199999999999995,"15min":0.93000000000000005},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.45300000000000001,"15min":0.46400000000000002},"min":{"1min":0.32400000000000001,"5min":0.28199999999999997,"15min":0.24399999999999999},"max":{"1min":0.54500000000000004,"5min":0.69199999999999995,"15min":0.98099999999999998},"last":0.436}]},{"osd":3,"last update":"Mon Oct 16 12:27:00 2023","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.38500000000000001,"15min":0.40100000000000002},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.57999999999999996,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.42799999999999999,"15min":0.45500000000000002},"min":{"1min":0.26100000000000001,"5min":0.252,"15min":0.22900000000000001},"max":{"1min":0.58699999999999997,"5min":0.91200000000000003,"15min":1.127},"last":0.45800000000000002}]},{"osd":4,"last update":"Mon Oct 16 12:27:00 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61099999999999999,"15min":0.63300000000000001},"min":{"1min":0.41799999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.79800000000000004,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.626},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.62,"15min":0.63900000000000001},"min":{"1min":0.41899999999999998,"5min":0.39100000000000001,"15min":0.34699999999999998},"max":{"1min":0.89600000000000002,"5min":0.92000000000000004,"15min":1.089},"last":0.55600000000000005}]},{"osd":5,"last update":"Mon Oct 16 12:27:00 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.63600000000000001,"15min":0.66500000000000004},"min":{"1min":0.496,"5min":0.38300000000000001,"15min":0.28499999999999998},"max":{"1min":0.76700000000000002,"5min":1.069,"15min":1.069},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67300000000000004,"15min":0.68799999999999994},"min":{"1min":0.439,"5min":0.38900000000000001,"15min":0.32500000000000001},"max":{"1min":0.94499999999999995,"5min":1.0620000000000001,"15min":1.0649999999999999},"last":0.67300000000000004}]},{"osd":6,"last update":"Mon Oct 16 12:27:00 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.63,"15min":0.65400000000000003},"min":{"1min":0.48699999999999999,"5min":0.378,"15min":0.378},"max":{"1min":0.77800000000000002,"5min":0.94999999999999996,"15min":1.0129999999999999},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.629,"5min":0.65300000000000002,"15min":0.66200000000000003},"min":{"1min":0.55100000000000005,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.83299999999999996,"5min":0.96899999999999997,"15min":1.048},"last":0.56799999999999995}]},{"osd":7,"last update":"Mon Oct 16 12:27:00 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.67200000000000004,"15min":0.68700000000000006},"min":{"1min":0.42099999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.98899999999999999,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.67600000000000005,"15min":0.70299999999999996},"min":{"1min":0.45700000000000002,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.876,"5min":0.94899999999999995,"15min":1.167},"last":0.58699999999999997}]}]},{"osd":0,"up_from":13,"seq":55834575046,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37100,"kb_used_data":1752,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334740,"statfs":{"total":96636764160,"available":96598773760,"internally_reserved":0,"allocated":1794048,"data_stored":1189901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 12:26:55 2023","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"min":{"1min":0.22500000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.40999999999999998,"15min":0.41799999999999998},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23200000000000001},"max":{"1min":0.64500000000000002,"5min":0.71799999999999997,"15min":0.73099999999999998},"last":0.35099999999999998}]},{"osd":2,"last update":"Mon Oct 16 12:26:55 2023","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.42099999999999999,"15min":0.42499999999999999},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.44500000000000001,"15min":0.45100000000000001},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.46600000000000003}]},{"osd":3,"last update":"Mon Oct 16 12:26:55 2023","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.42699999999999999,"15min":0.42199999999999999},"min":{"1min":0.26300000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.62,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.47799999999999998,"15min":0.48699999999999999},"min":{"1min":0.318,"5min":0.29099999999999998,"15min":0.27600000000000002},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.878},"last":0.44}]},{"osd":4,"last update":"Mon Oct 16 12:26:55 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.64300000000000002,"15min":0.64800000000000002},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.89200000000000002,"5min":0.90400000000000003,"15min":1.0960000000000001},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.626,"15min":0.65500000000000003},"min":{"1min":0.374,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.84199999999999997,"5min":0.93799999999999994,"15min":1.1719999999999999},"last":0.71999999999999997}]},{"osd":5,"last update":"Mon Oct 16 12:26:55 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.63500000000000001,"15min":0.65000000000000002},"min":{"1min":0.373,"5min":0.373,"15min":0.34399999999999997},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":1.127},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.66500000000000004,"15min":0.67700000000000005},"min":{"1min":0.39400000000000002,"5min":0.39200000000000002,"15min":0.39000000000000001},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":1.1160000000000001},"last":0.77500000000000002}]},{"osd":6,"last update":"Mon Oct 16 12:26:55 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.66600000000000004,"15min":0.68000000000000005},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":1.079},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.65900000000000003,"15min":0.68300000000000005},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":1.1499999999999999},"last":0.80300000000000005}]},{"osd":7,"last update":"Mon Oct 16 12:26:55 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.67800000000000005,"15min":0.69899999999999995},"min":{"1min":0.436,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.995,"5min":0.995,"15min":1.0349999999999999},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.65700000000000003,"15min":0.68799999999999994},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":1.0569999999999999},"last":0.66300000000000003}]}]},{"osd":2,"up_from":13,"seq":55834575046,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58736,"kb_used_data":1184,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313104,"statfs":{"total":96636764160,"available":96576618496,"internally_reserved":0,"allocated":1212416,"data_stored":600275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:26:56 2023","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.36799999999999999,"15min":0.38400000000000001},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.62,"5min":0.62,"15min":0.73799999999999999},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.44400000000000001,"15min":0.45000000000000001},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.63400000000000001,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.60699999999999998}]},{"osd":1,"last update":"Mon Oct 16 12:26:56 2023","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.45100000000000001,"15min":0.46800000000000003},"min":{"1min":0.31,"5min":0.25600000000000001,"15min":0.254},"max":{"1min":0.57999999999999996,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.42499999999999999,"15min":0.435},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.246},"max":{"1min":0.54400000000000004,"5min":0.69699999999999995,"15min":0.76500000000000001},"last":0.32300000000000001}]},{"osd":3,"last update":"Mon Oct 16 12:26:56 2023","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.40200000000000002,"15min":0.41599999999999998},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.60199999999999998,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.51000000000000001,"15min":0.51900000000000002},"min":{"1min":0.36599999999999999,"5min":0.318,"15min":0.246},"max":{"1min":0.67600000000000005,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.47399999999999998}]},{"osd":4,"last update":"Mon Oct 16 12:26:56 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.66500000000000004,"15min":0.68100000000000005},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.81200000000000006,"5min":1.3839999999999999,"15min":1.3839999999999999},"last":0.93799999999999994},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.66200000000000003,"15min":0.67900000000000005},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.84899999999999998,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.91200000000000003}]},{"osd":5,"last update":"Mon Oct 16 12:26:56 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.65800000000000003,"15min":0.66600000000000004},"min":{"1min":0.378,"5min":0.378,"15min":0.312},"max":{"1min":0.78700000000000003,"5min":1.093,"15min":1.093},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.64600000000000002,"15min":0.66400000000000003},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.317},"max":{"1min":0.879,"5min":1.331,"15min":1.331},"last":0.97399999999999998}]},{"osd":6,"last update":"Mon Oct 16 12:26:56 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.63200000000000001,"15min":0.63800000000000001},"min":{"1min":0.37,"5min":0.37,"15min":0.34699999999999998},"max":{"1min":0.91900000000000004,"5min":0.99299999999999999,"15min":1.046},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.68400000000000005,"15min":0.69199999999999995},"min":{"1min":0.40799999999999997,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.82599999999999996,"5min":1.032,"15min":1.0389999999999999},"last":0.84999999999999998}]},{"osd":7,"last update":"Mon Oct 16 12:26:56 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.66500000000000004,"15min":0.68000000000000005},"min":{"1min":0.40300000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.879,"5min":0.97499999999999998,"15min":1.026},"last":0.88600000000000001},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.68100000000000005,"15min":0.69299999999999995},"min":{"1min":0.46700000000000003,"5min":0.442,"15min":0.40600000000000003},"max":{"1min":0.89600000000000002,"5min":1.145,"15min":1.145},"last":0.67100000000000004}]}]},{"osd":3,"up_from":13,"seq":55834575045,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36524,"kb_used_data":1176,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335316,"statfs":{"total":96636764160,"available":96599363584,"internally_reserved":0,"allocated":1204224,"data_stored":600115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:26:02 2023","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.40100000000000002,"15min":0.39400000000000002},"min":{"1min":0.29499999999999998,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.435,"5min":0.41199999999999998,"15min":0.40500000000000003},"min":{"1min":0.28999999999999998,"5min":0.255,"15min":0.22600000000000001},"max":{"1min":0.61899999999999999,"5min":0.63100000000000001,"15min":0.73099999999999998},"last":0.317}]},{"osd":1,"last update":"Mon Oct 16 12:26:02 2023","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.41999999999999998,"15min":0.40999999999999998},"min":{"1min":0.29499999999999998,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.64100000000000001,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.46300000000000002,"15min":0.45800000000000002},"min":{"1min":0.38900000000000001,"5min":0.25800000000000001,"15min":0.25700000000000001},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.747},"last":0.27000000000000002}]},{"osd":2,"last update":"Mon Oct 16 12:26:02 2023","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.47399999999999998,"15min":0.46800000000000003},"min":{"1min":0.39600000000000002,"5min":0.307,"15min":0.26900000000000002},"max":{"1min":0.63700000000000001,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.503,"15min":0.49199999999999999},"min":{"1min":0.42199999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.60299999999999998,"5min":0.752,"15min":0.752},"last":0.377}]},{"osd":4,"last update":"Mon Oct 16 12:26:02 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.63200000000000001,"15min":0.64500000000000002},"min":{"1min":0.49399999999999999,"5min":0.39100000000000001,"15min":0.377},"max":{"1min":0.81699999999999995,"5min":0.92700000000000005,"15min":1.1539999999999999},"last":0.501},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.65400000000000003,"15min":0.66800000000000004},"min":{"1min":0.55700000000000005,"5min":0.44900000000000001,"15min":0.40200000000000002},"max":{"1min":0.90600000000000003,"5min":1.0409999999999999,"15min":1.234},"last":0.39500000000000002}]},{"osd":5,"last update":"Mon Oct 16 12:26:02 2023","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.68600000000000005,"15min":0.68300000000000005},"min":{"1min":0.48899999999999999,"5min":0.443,"15min":0.41199999999999998},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.194},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.65600000000000003,"15min":0.66700000000000004},"min":{"1min":0.56000000000000005,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":1.202},"last":0.52600000000000002}]},{"osd":6,"last update":"Mon Oct 16 12:26:02 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.67500000000000004,"15min":0.67400000000000004},"min":{"1min":0.53300000000000003,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.94199999999999995,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.75,"5min":0.70599999999999996,"15min":0.70699999999999996},"min":{"1min":0.60099999999999998,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.1559999999999999},"last":0.45100000000000001}]},{"osd":7,"last update":"Mon Oct 16 12:26:02 2023","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.68000000000000005,"15min":0.68899999999999995},"min":{"1min":0.499,"5min":0.435,"15min":0.38},"max":{"1min":0.92300000000000004,"5min":1.0189999999999999,"15min":1.2430000000000001},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.70999999999999996,"15min":0.70699999999999996},"min":{"1min":0.48899999999999999,"5min":0.45200000000000001,"15min":0.42699999999999999},"max":{"1min":0.99099999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.44}]}]},{"osd":4,"up_from":13,"seq":55834575044,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32372,"kb_used_data":1192,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339468,"statfs":{"total":96636764160,"available":96603615232,"internally_reserved":0,"allocated":1220608,"data_stored":601249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:26:00 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.59799999999999998,"15min":0.59199999999999997},"min":{"1min":0.30399999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.78300000000000003,"5min":0.85099999999999998,"15min":1.171},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.62,"5min":0.58699999999999997,"15min":0.60199999999999998},"min":{"1min":0.42399999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.76400000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.49299999999999999}]},{"osd":1,"last update":"Mon Oct 16 12:26:00 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.59299999999999997,"15min":0.60999999999999999},"min":{"1min":0.433,"5min":0.35499999999999998,"15min":0.28499999999999998},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.92600000000000005},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.622,"5min":0.61499999999999999,"15min":0.621},"min":{"1min":0.374,"5min":0.374,"15min":0.33600000000000002},"max":{"1min":0.73899999999999999,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.58299999999999996}]},{"osd":2,"last update":"Mon Oct 16 12:26:00 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.61199999999999999,"15min":0.625},"min":{"1min":0.39900000000000002,"5min":0.35399999999999998,"15min":0.32000000000000001},"max":{"1min":0.77000000000000002,"5min":0.93000000000000005,"15min":1.012},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61299999999999999,"15min":0.627},"min":{"1min":0.47899999999999998,"5min":0.38800000000000001,"15min":0.34100000000000003},"max":{"1min":0.74099999999999999,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.61799999999999999}]},{"osd":3,"last update":"Mon Oct 16 12:26:00 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.59599999999999997,"15min":0.61299999999999999},"min":{"1min":0.46300000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.70699999999999996,"5min":0.79500000000000004,"15min":1.3779999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63,"15min":0.64100000000000001},"min":{"1min":0.41199999999999998,"5min":0.38800000000000001,"15min":0.37},"max":{"1min":0.77800000000000002,"5min":0.89800000000000002,"15min":0.97199999999999998},"last":0.65000000000000002}]},{"osd":5,"last update":"Mon Oct 16 12:26:00 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48699999999999999,"15min":0.5},"min":{"1min":0.34799999999999998,"5min":0.29799999999999999,"15min":0.27600000000000002},"max":{"1min":0.68100000000000005,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.504,"5min":0.51400000000000001,"15min":0.52500000000000002},"min":{"1min":0.371,"5min":0.316,"15min":0.29599999999999999},"max":{"1min":0.72599999999999998,"5min":0.754,"15min":0.94199999999999995},"last":0.56299999999999994}]},{"osd":6,"last update":"Mon Oct 16 12:26:00 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.54600000000000004,"15min":0.55900000000000005},"min":{"1min":0.39800000000000002,"5min":0.34300000000000003,"15min":0.28699999999999998},"max":{"1min":0.68700000000000006,"5min":0.88400000000000001,"15min":0.89800000000000002},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.63700000000000001,"15min":0.63600000000000001},"min":{"1min":0.45000000000000001,"5min":0.34799999999999998,"15min":0.32000000000000001},"max":{"1min":0.84099999999999997,"5min":0.99299999999999999,"15min":1.1180000000000001},"last":0.70499999999999996}]},{"osd":7,"last update":"Mon Oct 16 12:26:00 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57599999999999996,"15min":0.58099999999999996},"min":{"1min":0.42099999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.76200000000000001,"5min":0.80900000000000005,"15min":0.89500000000000002},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.628,"15min":0.626},"min":{"1min":0.42599999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.87,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.60199999999999998}]}]},{"osd":5,"up_from":13,"seq":55834575045,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54568,"kb_used_data":1176,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317272,"statfs":{"total":96636764160,"available":96580886528,"internally_reserved":0,"allocated":1204224,"data_stored":599567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:26:58 2023","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.56899999999999995,"15min":0.59399999999999997},"min":{"1min":0.32500000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.64600000000000002,"5min":0.84499999999999997,"15min":0.873},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.59399999999999997,"15min":0.62},"min":{"1min":0.36299999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.73499999999999999,"5min":0.85899999999999999,"15min":0.88800000000000001},"last":0.63300000000000001}]},{"osd":1,"last update":"Mon Oct 16 12:26:58 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.60499999999999998,"15min":0.61499999999999999},"min":{"1min":0.39500000000000002,"5min":0.32600000000000001,"15min":0.28999999999999998},"max":{"1min":0.76600000000000001,"5min":0.89700000000000002,"15min":0.91300000000000003},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.503,"5min":0.60699999999999998,"15min":0.62},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.78900000000000003,"5min":0.85999999999999999,"15min":0.89000000000000001},"last":0.61099999999999999}]},{"osd":2,"last update":"Mon Oct 16 12:26:58 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.60299999999999998,"15min":0.623},"min":{"1min":0.40000000000000002,"5min":0.29099999999999998,"15min":0.26400000000000001},"max":{"1min":0.71299999999999997,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.59599999999999997,"15min":0.61399999999999999},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.66900000000000004,"5min":0.88600000000000001,"15min":0.89700000000000002},"last":0.45900000000000002}]},{"osd":3,"last update":"Mon Oct 16 12:26:58 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.61599999999999999,"15min":0.64100000000000001},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.72099999999999997,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.64000000000000001,"15min":0.65000000000000002},"min":{"1min":0.40400000000000003,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.71799999999999997,"5min":0.90800000000000003,"15min":0.93200000000000005},"last":0.56799999999999995}]},{"osd":4,"last update":"Mon Oct 16 12:26:58 2023","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.504,"15min":0.51000000000000001},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.25800000000000001},"max":{"1min":0.77100000000000002,"5min":0.85699999999999998,"15min":0.86399999999999999},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.58499999999999996,"15min":0.57099999999999995},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33500000000000002},"max":{"1min":0.78500000000000003,"5min":1.347,"15min":1.347},"last":0.41099999999999998}]},{"osd":6,"last update":"Mon Oct 16 12:26:58 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.54400000000000004,"15min":0.55200000000000005},"min":{"1min":0.33100000000000002,"5min":0.314,"15min":0.314},"max":{"1min":0.83999999999999997,"5min":0.89400000000000002,"15min":0.93500000000000005},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.59999999999999998,"15min":0.59799999999999998},"min":{"1min":0.38300000000000001,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.82899999999999996,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.65400000000000003}]},{"osd":7,"last update":"Mon Oct 16 12:26:58 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.61699999999999999,"15min":0.61899999999999999},"min":{"1min":0.41499999999999998,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.79400000000000004,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.65300000000000002,"15min":0.65500000000000003},"min":{"1min":0.45600000000000002,"5min":0.379,"15min":0.379},"max":{"1min":0.88900000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.67000000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T12:27:05.969 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-16T12:27:05.970 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-16T12:27:06.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.134+0000 7fcf9858d700 1 Processor -- start 2023-10-16T12:27:06.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.134+0000 7fcf9858d700 1 -- start start 2023-10-16T12:27:06.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.135+0000 7fcf9858d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcf9015d8d0 0x7fcf900b2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:06.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.135+0000 7fcf9858d700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf900b2f30 con 0x7fcf900b8fa0 2023-10-16T12:27:06.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.135+0000 7fcf9858d700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf900d1c40 con 0x7fcf900ce350 2023-10-16T12:27:06.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.135+0000 7fcf9858d700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf900d0bb0 con 0x7fcf9015d8d0 2023-10-16T12:27:06.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.136+0000 7fcf96329700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcf9015d8d0 0x7fcf900b2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:06.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.136+0000 7fcf95b28700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fcf900b8fa0 0x7fcf900b6720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:58448/0 (socket says 172.21.15.16:58448) 2023-10-16T12:27:06.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.136+0000 7fcf95b28700 1 -- 172.21.15.16:0/3506411152 learned_addr learned my addr 172.21.15.16:0/3506411152 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:27:06.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.136+0000 7fcf96329700 1 -- 172.21.15.16:0/3506411152 >> v1:172.21.15.49:6789/0 conn(0x7fcf900ce350 legacy=0x7fcf900d29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:06.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.136+0000 7fcf96329700 1 -- 172.21.15.16:0/3506411152 >> v1:172.21.15.16:6789/0 conn(0x7fcf900b8fa0 legacy=0x7fcf900b6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:06.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.136+0000 7fcf96329700 1 -- 172.21.15.16:0/3506411152 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf900d1ee0 con 0x7fcf9015d8d0 2023-10-16T12:27:06.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.136+0000 7fcf96329700 1 --2- 172.21.15.16:0/3506411152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf9015d8d0 0x7fcf900b2830 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7fcf80009840 tx=0x7fcf80009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfaf826e387d1141 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:06.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.136+0000 7fcf95327700 1 -- 172.21.15.16:0/3506411152 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf80004230 con 0x7fcf9015d8d0 2023-10-16T12:27:06.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.136+0000 7fcf95327700 1 -- 172.21.15.16:0/3506411152 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf80004390 con 0x7fcf9015d8d0 2023-10-16T12:27:06.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.136+0000 7fcf95327700 1 -- 172.21.15.16:0/3506411152 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf800036a0 con 0x7fcf9015d8d0 2023-10-16T12:27:06.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.137+0000 7fcf9858d700 1 -- 172.21.15.16:0/3506411152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf9015d8d0 msgr2=0x7fcf900b2830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:06.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.137+0000 7fcf9858d700 1 --2- 172.21.15.16:0/3506411152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf9015d8d0 0x7fcf900b2830 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7fcf80009840 tx=0x7fcf80009b50 comp rx=0 tx=0).stop 2023-10-16T12:27:06.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.137+0000 7fcf9858d700 1 -- 172.21.15.16:0/3506411152 shutdown_connections 2023-10-16T12:27:06.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.137+0000 7fcf9858d700 1 --2- 172.21.15.16:0/3506411152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf9015d8d0 0x7fcf900b2830 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:06.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.137+0000 7fcf9858d700 1 -- 172.21.15.16:0/3506411152 >> 172.21.15.16:0/3506411152 conn(0x7fcf90072330 msgr2=0x7fcf90072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:06.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.137+0000 7fcf9858d700 1 -- 172.21.15.16:0/3506411152 shutdown_connections 2023-10-16T12:27:06.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.137+0000 7fcf9858d700 1 -- 172.21.15.16:0/3506411152 wait complete. 2023-10-16T12:27:06.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.138+0000 7fcf9858d700 1 Processor -- start 2023-10-16T12:27:06.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.138+0000 7fcf9858d700 1 -- start start 2023-10-16T12:27:06.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.139+0000 7fcf9858d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcf900b8fa0 0x7fcf90157ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:06.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.139+0000 7fcf9858d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcf900ce350 0x7fcf900cd350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:06.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.139+0000 7fcf9858d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf9015d8d0 0x7fcf900b7040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:06.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.139+0000 7fcf9858d700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf900cbd40 con 0x7fcf9015d8d0 2023-10-16T12:27:06.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.139+0000 7fcf9858d700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf900cc020 con 0x7fcf900ce350 2023-10-16T12:27:06.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.139+0000 7fcf9858d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf900cc300 con 0x7fcf900b8fa0 2023-10-16T12:27:06.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.139+0000 7fcf96b2a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf9015d8d0 0x7fcf900b7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:06.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.139+0000 7fcf96b2a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf9015d8d0 0x7fcf900b7040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40950/0 (socket says 172.21.15.16:40950) 2023-10-16T12:27:06.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.139+0000 7fcf96b2a700 1 -- 172.21.15.16:0/3049692813 learned_addr learned my addr 172.21.15.16:0/3049692813 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:06.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.139+0000 7fcf96329700 1 --2- 172.21.15.16:0/3049692813 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcf900b8fa0 0x7fcf90157ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:06.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.139+0000 7fcf95b28700 1 --2- 172.21.15.16:0/3049692813 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcf900ce350 0x7fcf900cd350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:06.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.139+0000 7fcf96b2a700 1 -- 172.21.15.16:0/3049692813 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcf900b8fa0 msgr2=0x7fcf90157ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:06.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.139+0000 7fcf96b2a700 1 --2- 172.21.15.16:0/3049692813 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcf900b8fa0 0x7fcf90157ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:06.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.139+0000 7fcf96b2a700 1 -- 172.21.15.16:0/3049692813 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcf900ce350 msgr2=0x7fcf900cd350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:06.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.139+0000 7fcf96b2a700 1 --2- 172.21.15.16:0/3049692813 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcf900ce350 0x7fcf900cd350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:06.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.139+0000 7fcf96b2a700 1 -- 172.21.15.16:0/3049692813 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf800094a0 con 0x7fcf9015d8d0 2023-10-16T12:27:06.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.139+0000 7fcf95b28700 1 --2- 172.21.15.16:0/3049692813 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcf900ce350 0x7fcf900cd350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:27:06.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.140+0000 7fcf96b2a700 1 --2- 172.21.15.16:0/3049692813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf9015d8d0 0x7fcf900b7040 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7fcf8800b9e0 tx=0x7fcf8800bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:06.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.140+0000 7fcf86ffd700 1 -- 172.21.15.16:0/3049692813 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf8800c9b0 con 0x7fcf9015d8d0 2023-10-16T12:27:06.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.140+0000 7fcf86ffd700 1 -- 172.21.15.16:0/3049692813 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf8800cb10 con 0x7fcf9015d8d0 2023-10-16T12:27:06.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.140+0000 7fcf9858d700 1 -- 172.21.15.16:0/3049692813 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf900cc5e0 con 0x7fcf9015d8d0 2023-10-16T12:27:06.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.140+0000 7fcf86ffd700 1 -- 172.21.15.16:0/3049692813 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf88014440 con 0x7fcf9015d8d0 2023-10-16T12:27:06.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.140+0000 7fcf9858d700 1 -- 172.21.15.16:0/3049692813 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf90158400 con 0x7fcf9015d8d0 2023-10-16T12:27:06.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.141+0000 7fcf9858d700 1 -- 172.21.15.16:0/3049692813 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf900ccb00 con 0x7fcf9015d8d0 2023-10-16T12:27:06.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.141+0000 7fcf86ffd700 1 -- 172.21.15.16:0/3049692813 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 41) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fcf88006d50 con 0x7fcf9015d8d0 2023-10-16T12:27:06.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.142+0000 7fcf86ffd700 1 --2- 172.21.15.16:0/3049692813 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7fcf7406b020 0x7fcf7406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:06.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.142+0000 7fcf96329700 1 --2- 172.21.15.16:0/3049692813 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7fcf7406b020 0x7fcf7406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:06.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.142+0000 7fcf86ffd700 1 -- 172.21.15.16:0/3049692813 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fcf880892a0 con 0x7fcf9015d8d0 2023-10-16T12:27:06.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.142+0000 7fcf96329700 1 --2- 172.21.15.16:0/3049692813 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7fcf7406b020 0x7fcf7406d4e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fcf800090b0 tx=0x7fcf80008030 comp rx=0 tx=0).ready entity=mgr.6020 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:06.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.146+0000 7fcf86ffd700 1 -- 172.21.15.16:0/3049692813 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcf900ccb00 con 0x7fcf9015d8d0 2023-10-16T12:27:06.386 INFO:tasks.ceph.mds.a.smithi016.stderr:2023-10-16T12:27:06.382+0000 7f8d3737c700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T12:27:06.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.431+0000 7fcf9858d700 1 -- 172.21.15.16:0/3049692813 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fcf900ccb00 con 0x7fcf9015d8d0 2023-10-16T12:27:06.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.433+0000 7fcf86ffd700 1 -- 172.21.15.16:0/3049692813 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v21) v1 ==== 190+0+147 (secure 0 0 0) 0x7fcf900ccb00 con 0x7fcf9015d8d0 2023-10-16T12:27:06.435 INFO:teuthology.orchestra.run.smithi016.stdout:[client.0] 2023-10-16T12:27:06.435 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAPKC1l2l6PFxAA9QyXtBAHdGAXpwH0JfknuQ== 2023-10-16T12:27:06.436 INFO:teuthology.orchestra.run.smithi016.stdout: caps mds = "allow" 2023-10-16T12:27:06.436 INFO:teuthology.orchestra.run.smithi016.stdout: caps mon = "allow r" 2023-10-16T12:27:06.436 INFO:teuthology.orchestra.run.smithi016.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-16T12:27:06.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.435+0000 7fcf9858d700 1 -- 172.21.15.16:0/3049692813 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7fcf7406b020 msgr2=0x7fcf7406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:06.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.435+0000 7fcf9858d700 1 --2- 172.21.15.16:0/3049692813 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7fcf7406b020 0x7fcf7406d4e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fcf800090b0 tx=0x7fcf80008030 comp rx=0 tx=0).stop 2023-10-16T12:27:06.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.435+0000 7fcf9858d700 1 -- 172.21.15.16:0/3049692813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf9015d8d0 msgr2=0x7fcf900b7040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:06.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.435+0000 7fcf9858d700 1 --2- 172.21.15.16:0/3049692813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf9015d8d0 0x7fcf900b7040 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7fcf8800b9e0 tx=0x7fcf8800bda0 comp rx=0 tx=0).stop 2023-10-16T12:27:06.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.435+0000 7fcf9858d700 1 -- 172.21.15.16:0/3049692813 shutdown_connections 2023-10-16T12:27:06.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.435+0000 7fcf9858d700 1 --2- 172.21.15.16:0/3049692813 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7fcf7406b020 0x7fcf7406d4e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:06.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.435+0000 7fcf9858d700 1 --2- 172.21.15.16:0/3049692813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf9015d8d0 0x7fcf900b7040 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:06.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.435+0000 7fcf9858d700 1 --2- 172.21.15.16:0/3049692813 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcf900b8fa0 0x7fcf90157ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:06.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.435+0000 7fcf9858d700 1 --2- 172.21.15.16:0/3049692813 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcf900ce350 0x7fcf900cd350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:06.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.435+0000 7fcf9858d700 1 -- 172.21.15.16:0/3049692813 >> 172.21.15.16:0/3049692813 conn(0x7fcf90072330 msgr2=0x7fcf900ce730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:06.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.435+0000 7fcf9858d700 1 -- 172.21.15.16:0/3049692813 shutdown_connections 2023-10-16T12:27:06.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.435+0000 7fcf9858d700 1 -- 172.21.15.16:0/3049692813 wait complete. 2023-10-16T12:27:06.440 INFO:teuthology.orchestra.run.smithi016.stderr:updated caps for client.0 2023-10-16T12:27:06.449 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:27:06.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.572+0000 7f8f279ad700 1 Processor -- start 2023-10-16T12:27:06.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.572+0000 7f8f279ad700 1 -- start start 2023-10-16T12:27:06.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.573+0000 7f8f279ad700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8f200cf4a0 0x7f8f200726e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:06.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.573+0000 7f8f279ad700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8f200b5a00 0x7f8f2015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:06.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.573+0000 7f8f279ad700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f2015bc80 con 0x7f8f200cf880 2023-10-16T12:27:06.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.573+0000 7f8f279ad700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f200b5e60 con 0x7f8f200b5a00 2023-10-16T12:27:06.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.573+0000 7f8f279ad700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f20072c20 con 0x7f8f200cf4a0 2023-10-16T12:27:06.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.573+0000 7f8f25f4a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8f200b5a00 0x7f8f2015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:06.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.573+0000 7f8f25f4a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8f200b5a00 0x7f8f2015a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40958/0 (socket says 172.21.15.16:40958) 2023-10-16T12:27:06.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.573+0000 7f8f25f4a700 1 -- 172.21.15.16:0/875113482 learned_addr learned my addr 172.21.15.16:0/875113482 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:06.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.573+0000 7f8f25749700 1 --2- 172.21.15.16:0/875113482 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8f200cf4a0 0x7f8f200726e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:06.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.574+0000 7f8f25f4a700 1 -- 172.21.15.16:0/875113482 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8f200cf4a0 msgr2=0x7f8f200726e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:06.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.574+0000 7f8f25f4a700 1 --2- 172.21.15.16:0/875113482 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8f200cf4a0 0x7f8f200726e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:06.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.574+0000 7f8f25f4a700 1 -- 172.21.15.16:0/875113482 >> v1:172.21.15.49:6789/0 conn(0x7f8f200cf880 legacy=0x7f8f200b52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:06.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.574+0000 7f8f25f4a700 1 -- 172.21.15.16:0/875113482 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f2015b070 con 0x7f8f200b5a00 2023-10-16T12:27:06.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.574+0000 7f8f25f4a700 1 --2- 172.21.15.16:0/875113482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f200b5a00 0x7f8f2015a090 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f8f18009e50 tx=0x7f8f1800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0e33667468720fe server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:06.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.574+0000 7f8f17fff700 1 -- 172.21.15.16:0/875113482 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f1800e070 con 0x7f8f200b5a00 2023-10-16T12:27:06.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.574+0000 7f8f17fff700 1 -- 172.21.15.16:0/875113482 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f1800b9c0 con 0x7f8f200b5a00 2023-10-16T12:27:06.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.574+0000 7f8f17fff700 1 -- 172.21.15.16:0/875113482 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f1800bd70 con 0x7f8f200b5a00 2023-10-16T12:27:06.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.574+0000 7f8f279ad700 1 -- 172.21.15.16:0/875113482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f200b5a00 msgr2=0x7f8f2015a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:06.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.574+0000 7f8f279ad700 1 --2- 172.21.15.16:0/875113482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f200b5a00 0x7f8f2015a090 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f8f18009e50 tx=0x7f8f1800b330 comp rx=0 tx=0).stop 2023-10-16T12:27:06.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.575+0000 7f8f279ad700 1 -- 172.21.15.16:0/875113482 shutdown_connections 2023-10-16T12:27:06.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.575+0000 7f8f279ad700 1 --2- 172.21.15.16:0/875113482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f200b5a00 0x7f8f2015a090 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:06.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.575+0000 7f8f279ad700 1 --2- 172.21.15.16:0/875113482 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8f200cf4a0 0x7f8f200726e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:06.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.575+0000 7f8f279ad700 1 -- 172.21.15.16:0/875113482 >> 172.21.15.16:0/875113482 conn(0x7f8f200b1fd0 msgr2=0x7f8f200b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:06.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.575+0000 7f8f279ad700 1 -- 172.21.15.16:0/875113482 shutdown_connections 2023-10-16T12:27:06.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.575+0000 7f8f279ad700 1 -- 172.21.15.16:0/875113482 wait complete. 2023-10-16T12:27:06.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.576+0000 7f8f279ad700 1 Processor -- start 2023-10-16T12:27:06.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.576+0000 7f8f279ad700 1 -- start start 2023-10-16T12:27:06.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.577+0000 7f8f279ad700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f200b5a00 0x7f8f200c3f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:06.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.577+0000 7f8f279ad700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8f200cf4a0 0x7f8f201f69e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:06.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.577+0000 7f8f279ad700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8f200cf880 0x7f8f200c7db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:06.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.577+0000 7f8f279ad700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f201f6f20 con 0x7f8f200b5a00 2023-10-16T12:27:06.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.577+0000 7f8f279ad700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f20157bf0 con 0x7f8f200cf880 2023-10-16T12:27:06.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.577+0000 7f8f279ad700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f20157ed0 con 0x7f8f200cf4a0 2023-10-16T12:27:06.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.577+0000 7f8f25749700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f200b5a00 0x7f8f200c3f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:06.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.577+0000 7f8f25749700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f200b5a00 0x7f8f200c3f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40960/0 (socket says 172.21.15.16:40960) 2023-10-16T12:27:06.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.577+0000 7f8f25749700 1 -- 172.21.15.16:0/4177613552 learned_addr learned my addr 172.21.15.16:0/4177613552 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:06.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.577+0000 7f8f24f48700 1 --2- 172.21.15.16:0/4177613552 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8f200cf4a0 0x7f8f201f69e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:06.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.577+0000 7f8f25f4a700 1 --2- 172.21.15.16:0/4177613552 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8f200cf880 0x7f8f200c7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:06.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.577+0000 7f8f25749700 1 -- 172.21.15.16:0/4177613552 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8f200cf4a0 msgr2=0x7f8f201f69e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:06.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.577+0000 7f8f25749700 1 --2- 172.21.15.16:0/4177613552 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8f200cf4a0 0x7f8f201f69e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:06.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.577+0000 7f8f25749700 1 -- 172.21.15.16:0/4177613552 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8f200cf880 msgr2=0x7f8f200c7db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:06.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.577+0000 7f8f25749700 1 --2- 172.21.15.16:0/4177613552 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8f200cf880 0x7f8f200c7db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:06.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.577+0000 7f8f25749700 1 -- 172.21.15.16:0/4177613552 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f18009b30 con 0x7f8f200b5a00 2023-10-16T12:27:06.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.577+0000 7f8f25f4a700 1 --2- 172.21.15.16:0/4177613552 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8f200cf880 0x7f8f200c7db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:27:06.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.577+0000 7f8f24f48700 1 --2- 172.21.15.16:0/4177613552 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8f200cf4a0 0x7f8f201f69e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:27:06.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.577+0000 7f8f25749700 1 --2- 172.21.15.16:0/4177613552 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f200b5a00 0x7f8f200c3f40 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f8f1000e810 tx=0x7f8f1000ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:06.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.577+0000 7f8f15ffb700 1 -- 172.21.15.16:0/4177613552 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f10013610 con 0x7f8f200b5a00 2023-10-16T12:27:06.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.577+0000 7f8f15ffb700 1 -- 172.21.15.16:0/4177613552 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f10013770 con 0x7f8f200b5a00 2023-10-16T12:27:06.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.577+0000 7f8f279ad700 1 -- 172.21.15.16:0/4177613552 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f201581b0 con 0x7f8f200b5a00 2023-10-16T12:27:06.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.577+0000 7f8f15ffb700 1 -- 172.21.15.16:0/4177613552 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f10013b20 con 0x7f8f200b5a00 2023-10-16T12:27:06.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.578+0000 7f8f279ad700 1 -- 172.21.15.16:0/4177613552 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f20158770 con 0x7f8f200b5a00 2023-10-16T12:27:06.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.579+0000 7f8f279ad700 1 -- 172.21.15.16:0/4177613552 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f2004ebb0 con 0x7f8f200b5a00 2023-10-16T12:27:06.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.579+0000 7f8f15ffb700 1 -- 172.21.15.16:0/4177613552 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 41) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8f10013c80 con 0x7f8f200b5a00 2023-10-16T12:27:06.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.579+0000 7f8f15ffb700 1 --2- 172.21.15.16:0/4177613552 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f8f0406b020 0x7f8f0406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:06.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.580+0000 7f8f15ffb700 1 -- 172.21.15.16:0/4177613552 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f8f10089730 con 0x7f8f200b5a00 2023-10-16T12:27:06.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.580+0000 7f8f24f48700 1 --2- 172.21.15.16:0/4177613552 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f8f0406b020 0x7f8f0406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:06.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.580+0000 7f8f24f48700 1 --2- 172.21.15.16:0/4177613552 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f8f0406b020 0x7f8f0406d4e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8f200b51e0 tx=0x7f8f1c009040 comp rx=0 tx=0).ready entity=mgr.6020 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:06.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.584+0000 7f8f15ffb700 1 -- 172.21.15.16:0/4177613552 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8f10054470 con 0x7f8f200b5a00 2023-10-16T12:27:06.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.832+0000 7f8f279ad700 1 -- 172.21.15.16:0/4177613552 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8f2015da30 con 0x7f8f200b5a00 2023-10-16T12:27:06.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.832+0000 7f8f15ffb700 1 -- 172.21.15.16:0/4177613552 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+4915 (secure 0 0 0) 0x7f8f10057a90 con 0x7f8f200b5a00 2023-10-16T12:27:06.835 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:27:06.835 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":44,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6820,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/1256919442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":1256919442},{"type":"v1","addr":"172.21.15.49:6834","nonce":1256919442}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":6823,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/3811815346","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":3811815346},{"type":"v1","addr":"172.21.15.49:6835","nonce":3811815346}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":7110,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/1226064755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":1226064755},{"type":"v1","addr":"172.21.15.16:6836","nonce":1226064755}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:27:00.825934+0000","modified":"2023-10-16T12:27:03.831266+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7113},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7113":{"gid":7113,"name":"a","rank":0,"incarnation":40,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/1843321005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1843321005},{"type":"v1","addr":"172.21.15.16:6837","nonce":1843321005}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-10-16T12:27:06.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.835+0000 7f8f279ad700 1 -- 172.21.15.16:0/4177613552 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f8f0406b020 msgr2=0x7f8f0406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:06.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.835+0000 7f8f279ad700 1 --2- 172.21.15.16:0/4177613552 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f8f0406b020 0x7f8f0406d4e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8f200b51e0 tx=0x7f8f1c009040 comp rx=0 tx=0).stop 2023-10-16T12:27:06.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.835+0000 7f8f279ad700 1 -- 172.21.15.16:0/4177613552 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f200b5a00 msgr2=0x7f8f200c3f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:06.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.835+0000 7f8f279ad700 1 --2- 172.21.15.16:0/4177613552 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f200b5a00 0x7f8f200c3f40 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f8f1000e810 tx=0x7f8f1000ebd0 comp rx=0 tx=0).stop 2023-10-16T12:27:06.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.835+0000 7f8f279ad700 1 -- 172.21.15.16:0/4177613552 shutdown_connections 2023-10-16T12:27:06.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.835+0000 7f8f279ad700 1 --2- 172.21.15.16:0/4177613552 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f8f0406b020 0x7f8f0406d4e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:06.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.835+0000 7f8f279ad700 1 --2- 172.21.15.16:0/4177613552 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8f200cf880 0x7f8f200c7db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:06.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.835+0000 7f8f279ad700 1 --2- 172.21.15.16:0/4177613552 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f200b5a00 0x7f8f200c3f40 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:06.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.835+0000 7f8f279ad700 1 --2- 172.21.15.16:0/4177613552 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8f200cf4a0 0x7f8f201f69e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:06.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.835+0000 7f8f279ad700 1 -- 172.21.15.16:0/4177613552 >> 172.21.15.16:0/4177613552 conn(0x7f8f200b1fd0 msgr2=0x7f8f200b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:06.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.835+0000 7f8f279ad700 1 -- 172.21.15.16:0/4177613552 shutdown_connections 2023-10-16T12:27:06.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:06.835+0000 7f8f279ad700 1 -- 172.21.15.16:0/4177613552 wait complete. 2023-10-16T12:27:06.840 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 44 2023-10-16T12:27:07.251 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 44, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T12:27:00.825934+0000', 'modified': '2023-10-16T12:27:03.831266+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7113}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7113': {'gid': 7113, 'name': 'a', 'rank': 0, 'incarnation': 40, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.16:6837/1843321005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6834', 'nonce': 1843321005}, {'type': 'v1', 'addr': '172.21.15.16:6837', 'nonce': 1843321005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [12], 'metadata_pool': 11, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-16T12:27:07.252 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-16T12:27:07.252 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-10-16T12:27:07.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.385+0000 7f6412d35700 1 Processor -- start 2023-10-16T12:27:07.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.385+0000 7f6412d35700 1 -- start start 2023-10-16T12:27:07.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.386+0000 7f6412d35700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f640c0cf880 0x7f640c0b52f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:07.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.386+0000 7f6412d35700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f640c15bc80 con 0x7f640c0b5830 2023-10-16T12:27:07.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.386+0000 7f6412d35700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f640c15b070 con 0x7f640c0cf4a0 2023-10-16T12:27:07.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.386+0000 7f6412d35700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f640c15b350 con 0x7f640c0cf880 2023-10-16T12:27:07.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.386+0000 7f64112d2700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f640c0b5830 0x7f640c15a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:58476/0 (socket says 172.21.15.16:58476) 2023-10-16T12:27:07.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.386+0000 7f64112d2700 1 -- 172.21.15.16:0/3246206871 learned_addr learned my addr 172.21.15.16:0/3246206871 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:27:07.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.386+0000 7f640bfff700 1 --2- 172.21.15.16:0/3246206871 >> v2:172.21.15.49:3300/0 conn(0x7f640c0cf880 0x7f640c0b52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:07.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.386+0000 7f640b7fe700 1 -- 172.21.15.16:0/3246206871 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2606812919 0 0) 0x7f640c15bc80 con 0x7f640c0b5830 2023-10-16T12:27:07.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.386+0000 7f640b7fe700 1 -- 172.21.15.16:0/3246206871 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63fc0057c0 con 0x7f640c0b5830 2023-10-16T12:27:07.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.387+0000 7f640b7fe700 1 -- 172.21.15.16:0/3246206871 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2312464963 0 0) 0x7f63fc0057c0 con 0x7f640c0b5830 2023-10-16T12:27:07.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.387+0000 7f640b7fe700 1 -- 172.21.15.16:0/3246206871 >> v2:172.21.15.49:3300/0 conn(0x7f640c0cf880 msgr2=0x7f640c0b52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:07.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.387+0000 7f640b7fe700 1 --2- 172.21.15.16:0/3246206871 >> v2:172.21.15.49:3300/0 conn(0x7f640c0cf880 0x7f640c0b52f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:07.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.387+0000 7f640b7fe700 1 -- 172.21.15.16:0/3246206871 >> v1:172.21.15.49:6789/0 conn(0x7f640c0cf4a0 legacy=0x7f640c0726e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:07.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.387+0000 7f640b7fe700 1 -- 172.21.15.16:0/3246206871 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f640c15b630 con 0x7f640c0b5830 2023-10-16T12:27:07.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.387+0000 7f640b7fe700 1 -- 172.21.15.16:0/3246206871 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f64040019f0 con 0x7f640c0b5830 2023-10-16T12:27:07.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.387+0000 7f640b7fe700 1 -- 172.21.15.16:0/3246206871 >> v1:172.21.15.16:6789/0 conn(0x7f640c0b5830 legacy=0x7f640c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:07.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.387+0000 7f640b7fe700 1 --2- 172.21.15.16:0/3246206871 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f63fc005c10 0x7f63fc006030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:07.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.387+0000 7f640b7fe700 1 --2- 172.21.15.16:0/3246206871 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f63fc006a60 0x7f63fc006ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:07.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.387+0000 7f640b7fe700 1 --2- 172.21.15.16:0/3246206871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f63fc007420 0x7f63fc00dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:07.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.387+0000 7f640b7fe700 1 -- 172.21.15.16:0/3246206871 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f640c15b350 con 0x7f63fc007420 2023-10-16T12:27:07.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.387+0000 7f640b7fe700 1 -- 172.21.15.16:0/3246206871 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63fc00e2c0 con 0x7f63fc005c10 2023-10-16T12:27:07.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.387+0000 7f640b7fe700 1 -- 172.21.15.16:0/3246206871 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63fc00e480 con 0x7f63fc006a60 2023-10-16T12:27:07.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.387+0000 7f64112d2700 1 --2- 172.21.15.16:0/3246206871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f63fc007420 0x7f63fc00dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:07.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.387+0000 7f6410ad1700 1 --2- 172.21.15.16:0/3246206871 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f63fc005c10 0x7f63fc006030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:07.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.387+0000 7f640bfff700 1 --2- 172.21.15.16:0/3246206871 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f63fc006a60 0x7f63fc006ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:07.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.387+0000 7f64112d2700 1 -- 172.21.15.16:0/3246206871 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f63fc006a60 msgr2=0x7f63fc006ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:07.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.387+0000 7f64112d2700 1 --2- 172.21.15.16:0/3246206871 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f63fc006a60 0x7f63fc006ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:07.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.387+0000 7f64112d2700 1 -- 172.21.15.16:0/3246206871 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f63fc005c10 msgr2=0x7f63fc006030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:07.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.387+0000 7f64112d2700 1 --2- 172.21.15.16:0/3246206871 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f63fc005c10 0x7f63fc006030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:07.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.387+0000 7f64112d2700 1 -- 172.21.15.16:0/3246206871 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63fc0057c0 con 0x7f63fc007420 2023-10-16T12:27:07.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.387+0000 7f64112d2700 1 -- 172.21.15.16:0/3246206871 reap_dead start 2023-10-16T12:27:07.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.388+0000 7f64112d2700 1 --2- 172.21.15.16:0/3246206871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f63fc007420 0x7f63fc00dc20 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f640400a830 tx=0x7f63fc007a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b585c697ed4df753 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:07.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.388+0000 7f640b7fe700 1 -- 172.21.15.16:0/3246206871 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f640400a2f0 con 0x7f63fc007420 2023-10-16T12:27:07.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.388+0000 7f640b7fe700 1 -- 172.21.15.16:0/3246206871 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6404012600 con 0x7f63fc007420 2023-10-16T12:27:07.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.388+0000 7f640b7fe700 1 -- 172.21.15.16:0/3246206871 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64040129b0 con 0x7f63fc007420 2023-10-16T12:27:07.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.388+0000 7f6412d35700 1 -- 172.21.15.16:0/3246206871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f63fc007420 msgr2=0x7f63fc00dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:07.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.388+0000 7f6412d35700 1 --2- 172.21.15.16:0/3246206871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f63fc007420 0x7f63fc00dc20 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f640400a830 tx=0x7f63fc007a90 comp rx=0 tx=0).stop 2023-10-16T12:27:07.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.388+0000 7f6412d35700 1 -- 172.21.15.16:0/3246206871 shutdown_connections 2023-10-16T12:27:07.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.388+0000 7f6412d35700 1 --2- 172.21.15.16:0/3246206871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f63fc007420 0x7f63fc00dc20 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:07.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.388+0000 7f6412d35700 1 -- 172.21.15.16:0/3246206871 >> 172.21.15.16:0/3246206871 conn(0x7f640c0b1fd0 msgr2=0x7f640c0b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:07.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.388+0000 7f6412d35700 1 -- 172.21.15.16:0/3246206871 shutdown_connections 2023-10-16T12:27:07.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.388+0000 7f6412d35700 1 -- 172.21.15.16:0/3246206871 wait complete. 2023-10-16T12:27:07.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.390+0000 7f6412d35700 1 Processor -- start 2023-10-16T12:27:07.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.390+0000 7f6412d35700 1 -- start start 2023-10-16T12:27:07.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.391+0000 7f6412d35700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f63fc007420 0x7f640c0c6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:07.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.391+0000 7f6412d35700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f640c14f780 0x7f640c0ccae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:07.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.391+0000 7f6412d35700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f640c0cd020 0x7f640c0c3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:07.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.391+0000 7f6412d35700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f640c150f50 con 0x7f63fc007420 2023-10-16T12:27:07.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.391+0000 7f6412d35700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f640c0cd4a0 con 0x7f640c14f780 2023-10-16T12:27:07.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.391+0000 7f6412d35700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f640c157ec0 con 0x7f640c0cd020 2023-10-16T12:27:07.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.391+0000 7f6410ad1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f63fc007420 0x7f640c0c6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:07.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.391+0000 7f6410ad1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f63fc007420 0x7f640c0c6280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40982/0 (socket says 172.21.15.16:40982) 2023-10-16T12:27:07.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.391+0000 7f6410ad1700 1 -- 172.21.15.16:0/3576714569 learned_addr learned my addr 172.21.15.16:0/3576714569 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:07.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.391+0000 7f6410ad1700 1 -- 172.21.15.16:0/3576714569 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f640c0cd020 msgr2=0x7f640c0c3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:07.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.391+0000 7f640bfff700 1 --2- 172.21.15.16:0/3576714569 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f640c14f780 0x7f640c0ccae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:07.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.391+0000 7f64112d2700 1 --2- 172.21.15.16:0/3576714569 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f640c0cd020 0x7f640c0c3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:07.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.391+0000 7f6410ad1700 1 --2- 172.21.15.16:0/3576714569 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f640c0cd020 0x7f640c0c3f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:07.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.391+0000 7f6410ad1700 1 -- 172.21.15.16:0/3576714569 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f640c14f780 msgr2=0x7f640c0ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:07.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.391+0000 7f6410ad1700 1 --2- 172.21.15.16:0/3576714569 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f640c14f780 0x7f640c0ccae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:07.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.392+0000 7f6410ad1700 1 -- 172.21.15.16:0/3576714569 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f640c157340 con 0x7f63fc007420 2023-10-16T12:27:07.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.392+0000 7f64112d2700 1 --2- 172.21.15.16:0/3576714569 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f640c0cd020 0x7f640c0c3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:27:07.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.392+0000 7f640bfff700 1 --2- 172.21.15.16:0/3576714569 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f640c14f780 0x7f640c0ccae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:27:07.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.392+0000 7f6410ad1700 1 --2- 172.21.15.16:0/3576714569 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f63fc007420 0x7f640c0c6280 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f640000f820 tx=0x7f640000fbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:07.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.392+0000 7f64097fa700 1 -- 172.21.15.16:0/3576714569 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64000052e0 con 0x7f63fc007420 2023-10-16T12:27:07.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.392+0000 7f64097fa700 1 -- 172.21.15.16:0/3576714569 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6400005460 con 0x7f63fc007420 2023-10-16T12:27:07.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.392+0000 7f6412d35700 1 -- 172.21.15.16:0/3576714569 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f640c157620 con 0x7f63fc007420 2023-10-16T12:27:07.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.392+0000 7f64097fa700 1 -- 172.21.15.16:0/3576714569 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64000038c0 con 0x7f63fc007420 2023-10-16T12:27:07.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.392+0000 7f6412d35700 1 -- 172.21.15.16:0/3576714569 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f640c157be0 con 0x7f63fc007420 2023-10-16T12:27:07.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.394+0000 7f64097fa700 1 -- 172.21.15.16:0/3576714569 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 41) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6400003a20 con 0x7f63fc007420 2023-10-16T12:27:07.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.394+0000 7f64097fa700 1 --2- 172.21.15.16:0/3576714569 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f63f006b050 0x7f63f006d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:07.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.394+0000 7f64097fa700 1 -- 172.21.15.16:0/3576714569 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f6400089680 con 0x7f63fc007420 2023-10-16T12:27:07.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.394+0000 7f640bfff700 1 --2- 172.21.15.16:0/3576714569 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f63f006b050 0x7f63f006d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:07.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.395+0000 7f640bfff700 1 --2- 172.21.15.16:0/3576714569 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f63f006b050 0x7f63f006d510 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f63f8009470 tx=0x7f63f8008040 comp rx=0 tx=0).ready entity=mgr.6020 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:07.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.396+0000 7f641ef1fb80 1 Processor -- start 2023-10-16T12:27:07.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.396+0000 7f641ef1fb80 1 -- start start 2023-10-16T12:27:07.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.397+0000 7f641ef1fb80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55fd372ec220 0x55fd372f3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:07.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.397+0000 7f641ef1fb80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fd372f1630 0x55fd372f3fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:07.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.397+0000 7f641ef1fb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fd372c7010 0x55fd372c76c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:07.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.397+0000 7f641ef1fb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fd3723dc20 con 0x55fd372c7010 2023-10-16T12:27:07.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.397+0000 7f641ef1fb80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fd37222c10 con 0x55fd372f1630 2023-10-16T12:27:07.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.397+0000 7f641ef1fb80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fd37220530 con 0x55fd372ec220 2023-10-16T12:27:07.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.397+0000 7f64112d2700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fd372c7010 0x55fd372c76c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:07.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.397+0000 7f64112d2700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fd372c7010 0x55fd372c76c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40992/0 (socket says 172.21.15.16:40992) 2023-10-16T12:27:07.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.397+0000 7f64112d2700 1 -- 172.21.15.16:0/1932727416 learned_addr learned my addr 172.21.15.16:0/1932727416 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:07.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.397+0000 7f6410ad1700 1 --2- 172.21.15.16:0/1932727416 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55fd372ec220 0x55fd372f3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:07.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.397+0000 7f640bfff700 1 --2- 172.21.15.16:0/1932727416 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fd372f1630 0x55fd372f3fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:07.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.397+0000 7f64112d2700 1 -- 172.21.15.16:0/1932727416 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55fd372ec220 msgr2=0x55fd372f3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:07.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.397+0000 7f64112d2700 1 --2- 172.21.15.16:0/1932727416 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55fd372ec220 0x55fd372f3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:07.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.397+0000 7f64112d2700 1 -- 172.21.15.16:0/1932727416 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fd372f1630 msgr2=0x55fd372f3fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:07.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.397+0000 7f64112d2700 1 --2- 172.21.15.16:0/1932727416 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fd372f1630 0x55fd372f3fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:07.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.397+0000 7f64112d2700 1 -- 172.21.15.16:0/1932727416 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fd371d93a0 con 0x55fd372c7010 2023-10-16T12:27:07.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.397+0000 7f64112d2700 1 --2- 172.21.15.16:0/1932727416 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fd372c7010 0x55fd372c76c0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f6404000f80 tx=0x7f6404010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b0afdd671148e15 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:07.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.397+0000 7f63f67fc700 1 -- 172.21.15.16:0/1932727416 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6404001250 con 0x55fd372c7010 2023-10-16T12:27:07.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.398+0000 7f63f67fc700 1 -- 172.21.15.16:0/1932727416 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f640400a210 con 0x55fd372c7010 2023-10-16T12:27:07.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.398+0000 7f63f67fc700 1 -- 172.21.15.16:0/1932727416 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6404012770 con 0x55fd372c7010 2023-10-16T12:27:07.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.398+0000 7f641ef1fb80 1 -- 172.21.15.16:0/1932727416 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fd372c7010 msgr2=0x55fd372c76c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:07.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.398+0000 7f641ef1fb80 1 --2- 172.21.15.16:0/1932727416 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fd372c7010 0x55fd372c76c0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f6404000f80 tx=0x7f6404010000 comp rx=0 tx=0).stop 2023-10-16T12:27:07.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.398+0000 7f641ef1fb80 1 -- 172.21.15.16:0/1932727416 shutdown_connections 2023-10-16T12:27:07.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.398+0000 7f641ef1fb80 1 --2- 172.21.15.16:0/1932727416 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fd372c7010 0x55fd372c76c0 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:07.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.398+0000 7f641ef1fb80 1 --2- 172.21.15.16:0/1932727416 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55fd372ec220 0x55fd372f3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:07.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.398+0000 7f641ef1fb80 1 --2- 172.21.15.16:0/1932727416 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fd372f1630 0x55fd372f3fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:07.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.398+0000 7f641ef1fb80 1 -- 172.21.15.16:0/1932727416 >> 172.21.15.16:0/1932727416 conn(0x55fd371f6d80 msgr2=0x55fd37311ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:07.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.398+0000 7f641ef1fb80 1 -- 172.21.15.16:0/1932727416 shutdown_connections 2023-10-16T12:27:07.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.398+0000 7f641ef1fb80 1 -- 172.21.15.16:0/1932727416 wait complete. 2023-10-16T12:27:07.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.399+0000 7f641ef1fb80 1 -- start start 2023-10-16T12:27:07.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.399+0000 7f641ef1fb80 1 Processor -- start 2023-10-16T12:27:07.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.400+0000 7f641ef1fb80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55fd372c7010 0x55fd372f7120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:07.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.400+0000 7f641ef1fb80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fd372ec220 0x55fd372ecaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:07.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.400+0000 7f641ef1fb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fd372f1630 0x55fd372ecfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:07.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.400+0000 7f641ef1fb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fd3721be20 con 0x55fd372f1630 2023-10-16T12:27:07.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.400+0000 7f641ef1fb80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fd371b86c0 con 0x55fd372ec220 2023-10-16T12:27:07.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.400+0000 7f641ef1fb80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fd371b8360 con 0x55fd372c7010 2023-10-16T12:27:07.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.400+0000 7f641ef1fb80 10 client.0 _ll_register_callbacks cb 0x55fd370157b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T12:27:07.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.400+0000 7f64112d2700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fd372f1630 0x55fd372ecfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:07.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.400+0000 7f64112d2700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fd372f1630 0x55fd372ecfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40998/0 (socket says 172.21.15.16:40998) 2023-10-16T12:27:07.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.400+0000 7f64112d2700 1 -- 172.21.15.16:0/699651882 learned_addr learned my addr 172.21.15.16:0/699651882 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:07.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.400+0000 7f6410ad1700 1 --2- 172.21.15.16:0/699651882 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55fd372c7010 0x55fd372f7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:07.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.400+0000 7f640bfff700 1 --2- 172.21.15.16:0/699651882 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fd372ec220 0x55fd372ecaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:07.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.400+0000 7f64112d2700 1 -- 172.21.15.16:0/699651882 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55fd372c7010 msgr2=0x55fd372f7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:07.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.400+0000 7f64112d2700 1 --2- 172.21.15.16:0/699651882 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55fd372c7010 0x55fd372f7120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:07.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.400+0000 7f64112d2700 1 -- 172.21.15.16:0/699651882 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fd372ec220 msgr2=0x55fd372ecaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:07.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.400+0000 7f64112d2700 1 --2- 172.21.15.16:0/699651882 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fd372ec220 0x55fd372ecaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:07.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.400+0000 7f64112d2700 1 -- 172.21.15.16:0/699651882 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63fc0057c0 con 0x55fd372f1630 2023-10-16T12:27:07.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.400+0000 7f64112d2700 1 --2- 172.21.15.16:0/699651882 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fd372f1630 0x55fd372ecfe0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f640c06ea20 tx=0x7f6404009050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9d0ab555e020930 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:07.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.400+0000 7f63eb7fe700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2023-10-16T12:27:07.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.401+0000 7f63eb7fe700 1 -- 172.21.15.16:0/699651882 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6404001250 con 0x55fd372f1630 2023-10-16T12:27:07.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.401+0000 7f63eb7fe700 1 -- 172.21.15.16:0/699651882 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f6404012db0 con 0x55fd372f1630 2023-10-16T12:27:07.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.401+0000 7f641ef1fb80 1 -- 172.21.15.16:0/699651882 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55fd372bc5b0 con 0x55fd372f1630 2023-10-16T12:27:07.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.401+0000 7f63eb7fe700 1 -- 172.21.15.16:0/699651882 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6404027450 con 0x55fd372f1630 2023-10-16T12:27:07.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.401+0000 7f641ef1fb80 10 client.7266 fetch_fsmap learned FSMap version 44 2023-10-16T12:27:07.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.401+0000 7f641ef1fb80 1 -- 172.21.15.16:0/699651882 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=2+}) v3 -- 0x55fd37046d90 con 0x55fd372f1630 2023-10-16T12:27:07.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.401+0000 7f63eb7fe700 1 -- 172.21.15.16:0/699651882 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55fd372bc5b0 con 0x55fd372f1630 2023-10-16T12:27:07.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.401+0000 7f63eb7fe700 1 -- 172.21.15.16:0/699651882 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 44) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f6404031030 con 0x55fd372f1630 2023-10-16T12:27:07.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.401+0000 7f641ef1fb80 10 client.7266 fetch_fsmap finished waiting for FSMap version 44 2023-10-16T12:27:07.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.401+0000 7f641ef1fb80 10 client.7266 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-16T12:27:07.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.401+0000 7f641ef1fb80 20 client.7266 populate_metadata read hostname 'smithi016' 2023-10-16T12:27:07.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.401+0000 7f641ef1fb80 1 --2- 172.21.15.16:0/699651882 >> [v2:172.21.15.16:6834/1843321005,v1:172.21.15.16:6837/1843321005] conn(0x55fd372f5950 0x55fd372f5d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:07.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.401+0000 7f641ef1fb80 4 client.7266 mds_command: new command op to 7113 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T12:27:07.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.401+0000 7f641ef1fb80 1 -- 172.21.15.16:0/699651882 --> [v2:172.21.15.16:6834/1843321005,v1:172.21.15.16:6837/1843321005] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fd37214e00 con 0x55fd372f5950 2023-10-16T12:27:07.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.401+0000 7f6410ad1700 1 --2- 172.21.15.16:0/699651882 >> [v2:172.21.15.16:6834/1843321005,v1:172.21.15.16:6837/1843321005] conn(0x55fd372f5950 0x55fd372f5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:07.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.401+0000 7f63eb7fe700 10 client.7266 ms_handle_connect on v2:172.21.15.16:6834/1843321005 2023-10-16T12:27:07.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.402+0000 7f6410ad1700 1 --2- 172.21.15.16:0/699651882 >> [v2:172.21.15.16:6834/1843321005,v1:172.21.15.16:6837/1843321005] conn(0x55fd372f5950 0x55fd372f5d30 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d0f19d87097704d6 server_cookie=c38d6ed8669deb83 in_seq=0 out_seq=0 2023-10-16T12:27:07.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.402+0000 7f63eb7fe700 1 -- 172.21.15.16:0/699651882 <== mds.0 v2:172.21.15.16:6834/1843321005 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55fd37214e00 con 0x55fd372f5950 2023-10-16T12:27:07.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.402+0000 7f63eb7fe700 10 client.7266 handle_command_reply: tid=0 2023-10-16T12:27:07.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.402+0000 7f641ef1fb80 1 client.7266 shutdown 2023-10-16T12:27:07.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.402+0000 7f641ef1fb80 1 -- 172.21.15.16:0/699651882 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fd372f1630 msgr2=0x55fd372ecfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:07.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.402+0000 7f641ef1fb80 1 --2- 172.21.15.16:0/699651882 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fd372f1630 0x55fd372ecfe0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f640c06ea20 tx=0x7f6404009050 comp rx=0 tx=0).stop 2023-10-16T12:27:07.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.402+0000 7f641ef1fb80 1 -- 172.21.15.16:0/699651882 shutdown_connections 2023-10-16T12:27:07.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.402+0000 7f641ef1fb80 1 --2- 172.21.15.16:0/699651882 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fd372f1630 0x55fd372ecfe0 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:07.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.402+0000 7f641ef1fb80 1 --2- 172.21.15.16:0/699651882 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55fd372c7010 0x55fd372f7120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:07.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.402+0000 7f641ef1fb80 1 --2- 172.21.15.16:0/699651882 >> [v2:172.21.15.16:6834/1843321005,v1:172.21.15.16:6837/1843321005] conn(0x55fd372f5950 0x55fd372f5d30 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:07.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.402+0000 7f641ef1fb80 1 --2- 172.21.15.16:0/699651882 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fd372ec220 0x55fd372ecaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:07.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.402+0000 7f63eb7fe700 0 client.7266 ms_handle_reset on v2:172.21.15.16:6834/1843321005 2023-10-16T12:27:07.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.402+0000 7f641ef1fb80 1 -- 172.21.15.16:0/699651882 >> 172.21.15.16:0/699651882 conn(0x55fd371f6d80 msgr2=0x55fd37308730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:07.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.402+0000 7f641ef1fb80 1 -- 172.21.15.16:0/699651882 shutdown_connections 2023-10-16T12:27:07.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.403+0000 7f641ef1fb80 1 -- 172.21.15.16:0/699651882 wait complete. 2023-10-16T12:27:07.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.403+0000 7f641ef1fb80 20 client.7266 trim_cache size 0 max 16384 2023-10-16T12:27:07.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.421+0000 7f641ef1fb80 1 Processor -- start 2023-10-16T12:27:07.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.421+0000 7f641ef1fb80 1 -- start start 2023-10-16T12:27:07.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.422+0000 7f641ef1fb80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fd372c7010 0x55fd372f72c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:07.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.422+0000 7f641ef1fb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fd372ec220 0x55fd372f7800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:07.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.422+0000 7f641ef1fb80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55fd372f1630 0x55fd372f0500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:07.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.422+0000 7f641ef1fb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fd3709c8d0 con 0x55fd372ec220 2023-10-16T12:27:07.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.422+0000 7f641ef1fb80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fd372c68a0 con 0x55fd372c7010 2023-10-16T12:27:07.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.422+0000 7f641ef1fb80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fd37311a70 con 0x55fd372f1630 2023-10-16T12:27:07.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.422+0000 7f640bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fd372ec220 0x55fd372f7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:07.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.422+0000 7f640bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fd372ec220 0x55fd372f7800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41004/0 (socket says 172.21.15.16:41004) 2023-10-16T12:27:07.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.422+0000 7f640bfff700 1 -- 172.21.15.16:0/2701170501 learned_addr learned my addr 172.21.15.16:0/2701170501 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:07.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.422+0000 7f6410ad1700 1 --2- 172.21.15.16:0/2701170501 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fd372c7010 0x55fd372f72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:07.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.422+0000 7f64112d2700 1 --2- 172.21.15.16:0/2701170501 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55fd372f1630 0x55fd372f0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:07.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.422+0000 7f640bfff700 1 -- 172.21.15.16:0/2701170501 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55fd372f1630 msgr2=0x55fd372f0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:07.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.422+0000 7f640bfff700 1 --2- 172.21.15.16:0/2701170501 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55fd372f1630 0x55fd372f0500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:07.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.422+0000 7f640bfff700 1 -- 172.21.15.16:0/2701170501 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fd372c7010 msgr2=0x55fd372f72c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:07.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.422+0000 7f640bfff700 1 --2- 172.21.15.16:0/2701170501 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fd372c7010 0x55fd372f72c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:07.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.422+0000 7f640bfff700 1 -- 172.21.15.16:0/2701170501 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fd3702fc10 con 0x55fd372ec220 2023-10-16T12:27:07.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.422+0000 7f6410ad1700 1 --2- 172.21.15.16:0/2701170501 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fd372c7010 0x55fd372f72c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:27:07.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.422+0000 7f64112d2700 1 --2- 172.21.15.16:0/2701170501 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55fd372f1630 0x55fd372f0500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:27:07.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.422+0000 7f640bfff700 1 --2- 172.21.15.16:0/2701170501 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fd372ec220 0x55fd372f7800 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x55fd372c56e0 tx=0x7f63f8016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fef9b5757c0d6c24 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:07.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.422+0000 7f63f6ffd700 1 -- 172.21.15.16:0/2701170501 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f8016b90 con 0x55fd372ec220 2023-10-16T12:27:07.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.422+0000 7f63f6ffd700 1 -- 172.21.15.16:0/2701170501 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f8016cf0 con 0x55fd372ec220 2023-10-16T12:27:07.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.422+0000 7f63f6ffd700 1 -- 172.21.15.16:0/2701170501 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f802e670 con 0x55fd372ec220 2023-10-16T12:27:07.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.423+0000 7f641ef1fb80 1 -- 172.21.15.16:0/2701170501 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fd372ec220 msgr2=0x55fd372f7800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:07.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.423+0000 7f641ef1fb80 1 --2- 172.21.15.16:0/2701170501 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fd372ec220 0x55fd372f7800 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x55fd372c56e0 tx=0x7f63f8016000 comp rx=0 tx=0).stop 2023-10-16T12:27:07.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.423+0000 7f641ef1fb80 1 -- 172.21.15.16:0/2701170501 shutdown_connections 2023-10-16T12:27:07.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.423+0000 7f641ef1fb80 1 --2- 172.21.15.16:0/2701170501 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55fd372f1630 0x55fd372f0500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:07.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.423+0000 7f641ef1fb80 1 --2- 172.21.15.16:0/2701170501 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fd372c7010 0x55fd372f72c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:07.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.423+0000 7f641ef1fb80 1 --2- 172.21.15.16:0/2701170501 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fd372ec220 0x55fd372f7800 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:07.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.423+0000 7f641ef1fb80 1 -- 172.21.15.16:0/2701170501 >> 172.21.15.16:0/2701170501 conn(0x55fd371f6d80 msgr2=0x55fd37308730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:07.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.423+0000 7f641ef1fb80 1 -- 172.21.15.16:0/2701170501 shutdown_connections 2023-10-16T12:27:07.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.423+0000 7f641ef1fb80 1 -- 172.21.15.16:0/2701170501 wait complete. 2023-10-16T12:27:07.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.424+0000 7f641ef1fb80 1 -- start start 2023-10-16T12:27:07.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.424+0000 7f641ef1fb80 1 Processor -- start 2023-10-16T12:27:07.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.425+0000 7f641ef1fb80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55fd372c7010 0x55fd3730f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:07.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.425+0000 7f641ef1fb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fd372ec220 0x55fd372f7720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:07.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.425+0000 7f641ef1fb80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fd372f1630 0x55fd3735b470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:07.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.425+0000 7f641ef1fb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fd372ed180 con 0x55fd372ec220 2023-10-16T12:27:07.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.425+0000 7f641ef1fb80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fd372f02c0 con 0x55fd372f1630 2023-10-16T12:27:07.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.425+0000 7f641ef1fb80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fd3735afe0 con 0x55fd372c7010 2023-10-16T12:27:07.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.425+0000 7f641ef1fb80 10 client.0 _ll_register_callbacks cb 0x7f63fc005950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T12:27:07.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.425+0000 7f640bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fd372ec220 0x55fd372f7720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:07.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.425+0000 7f640bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fd372ec220 0x55fd372f7720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41010/0 (socket says 172.21.15.16:41010) 2023-10-16T12:27:07.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.425+0000 7f640bfff700 1 -- 172.21.15.16:0/4010735856 learned_addr learned my addr 172.21.15.16:0/4010735856 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:07.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.425+0000 7f6410ad1700 1 --2- 172.21.15.16:0/4010735856 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55fd372c7010 0x55fd3730f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:07.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.425+0000 7f64112d2700 1 --2- 172.21.15.16:0/4010735856 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fd372f1630 0x55fd3735b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:07.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.425+0000 7f640bfff700 1 -- 172.21.15.16:0/4010735856 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55fd372c7010 msgr2=0x55fd3730f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:07.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.425+0000 7f640bfff700 1 --2- 172.21.15.16:0/4010735856 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55fd372c7010 0x55fd3730f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:07.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.425+0000 7f640bfff700 1 -- 172.21.15.16:0/4010735856 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fd372f1630 msgr2=0x55fd3735b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:07.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.425+0000 7f640bfff700 1 --2- 172.21.15.16:0/4010735856 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fd372f1630 0x55fd3735b470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:07.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.425+0000 7f640bfff700 1 -- 172.21.15.16:0/4010735856 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63f8013920 con 0x55fd372ec220 2023-10-16T12:27:07.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.425+0000 7f640bfff700 1 --2- 172.21.15.16:0/4010735856 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fd372ec220 0x55fd372f7720 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f63f8014010 tx=0x7f63f80047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c456cd492c270bfc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:07.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.425+0000 7f63eb7fe700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2023-10-16T12:27:07.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.425+0000 7f63eb7fe700 1 -- 172.21.15.16:0/4010735856 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f8016b90 con 0x55fd372ec220 2023-10-16T12:27:07.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.426+0000 7f63eb7fe700 1 -- 172.21.15.16:0/4010735856 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f63f8009280 con 0x55fd372ec220 2023-10-16T12:27:07.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.426+0000 7f641ef1fb80 1 -- 172.21.15.16:0/4010735856 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f640400a210 con 0x55fd372ec220 2023-10-16T12:27:07.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.426+0000 7f63eb7fe700 1 -- 172.21.15.16:0/4010735856 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) v2 ==== 24+0+0 (secure 0 0 0) 0x7f63f8016e60 con 0x55fd372ec220 2023-10-16T12:27:07.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.426+0000 7f641ef1fb80 10 client.7272 fetch_fsmap learned FSMap version 44 2023-10-16T12:27:07.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.426+0000 7f641ef1fb80 1 -- 172.21.15.16:0/4010735856 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=2+}) v3 -- 0x55fd3706d200 con 0x55fd372ec220 2023-10-16T12:27:07.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.426+0000 7f63eb7fe700 1 -- 172.21.15.16:0/4010735856 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f640400a210 con 0x55fd372ec220 2023-10-16T12:27:07.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.426+0000 7f63eb7fe700 1 -- 172.21.15.16:0/4010735856 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 44) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f63f802eb40 con 0x55fd372ec220 2023-10-16T12:27:07.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.426+0000 7f641ef1fb80 10 client.7272 fetch_fsmap finished waiting for FSMap version 44 2023-10-16T12:27:07.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.426+0000 7f641ef1fb80 10 client.7272 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-16T12:27:07.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.426+0000 7f641ef1fb80 20 client.7272 populate_metadata read hostname 'smithi016' 2023-10-16T12:27:07.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.426+0000 7f641ef1fb80 1 --2- 172.21.15.16:0/4010735856 >> [v2:172.21.15.16:6834/1843321005,v1:172.21.15.16:6837/1843321005] conn(0x55fd372ec600 0x55fd372ec9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:07.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.426+0000 7f641ef1fb80 4 client.7272 mds_command: new command op to 7113 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T12:27:07.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.426+0000 7f641ef1fb80 1 -- 172.21.15.16:0/4010735856 --> [v2:172.21.15.16:6834/1843321005,v1:172.21.15.16:6837/1843321005] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55fd37214b20 con 0x55fd372ec600 2023-10-16T12:27:07.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.426+0000 7f6410ad1700 1 --2- 172.21.15.16:0/4010735856 >> [v2:172.21.15.16:6834/1843321005,v1:172.21.15.16:6837/1843321005] conn(0x55fd372ec600 0x55fd372ec9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:07.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.426+0000 7f63eb7fe700 10 client.7272 ms_handle_connect on v2:172.21.15.16:6834/1843321005 2023-10-16T12:27:07.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.426+0000 7f6410ad1700 1 --2- 172.21.15.16:0/4010735856 >> [v2:172.21.15.16:6834/1843321005,v1:172.21.15.16:6837/1843321005] conn(0x55fd372ec600 0x55fd372ec9e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e41ef025c5d85778 server_cookie=3c8e5e9ddb92f516 in_seq=0 out_seq=0 2023-10-16T12:27:07.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.427+0000 7f63eb7fe700 1 -- 172.21.15.16:0/4010735856 <== mds.0 v2:172.21.15.16:6834/1843321005 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55fd37214b20 con 0x55fd372ec600 2023-10-16T12:27:07.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.427+0000 7f63eb7fe700 10 client.7272 handle_command_reply: tid=0 2023-10-16T12:27:07.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.427+0000 7f641ef1fb80 1 client.7272 shutdown 2023-10-16T12:27:07.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.427+0000 7f641ef1fb80 1 -- 172.21.15.16:0/4010735856 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fd372ec220 msgr2=0x55fd372f7720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:07.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.427+0000 7f641ef1fb80 1 --2- 172.21.15.16:0/4010735856 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fd372ec220 0x55fd372f7720 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f63f8014010 tx=0x7f63f80047c0 comp rx=0 tx=0).stop 2023-10-16T12:27:07.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.427+0000 7f641ef1fb80 1 -- 172.21.15.16:0/4010735856 shutdown_connections 2023-10-16T12:27:07.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.427+0000 7f641ef1fb80 1 --2- 172.21.15.16:0/4010735856 >> [v2:172.21.15.16:6834/1843321005,v1:172.21.15.16:6837/1843321005] conn(0x55fd372ec600 0x55fd372ec9e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:07.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.427+0000 7f641ef1fb80 1 --2- 172.21.15.16:0/4010735856 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fd372f1630 0x55fd3735b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:07.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.427+0000 7f641ef1fb80 1 --2- 172.21.15.16:0/4010735856 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55fd372c7010 0x55fd3730f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:07.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.427+0000 7f63eb7fe700 0 client.7272 ms_handle_reset on v2:172.21.15.16:6834/1843321005 2023-10-16T12:27:07.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.427+0000 7f641ef1fb80 1 --2- 172.21.15.16:0/4010735856 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fd372ec220 0x55fd372f7720 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:07.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.427+0000 7f641ef1fb80 1 -- 172.21.15.16:0/4010735856 >> 172.21.15.16:0/4010735856 conn(0x55fd371f6d80 msgr2=0x55fd372edd40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:07.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.427+0000 7f641ef1fb80 1 -- 172.21.15.16:0/4010735856 shutdown_connections 2023-10-16T12:27:07.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.427+0000 7f641ef1fb80 1 -- 172.21.15.16:0/4010735856 wait complete. 2023-10-16T12:27:07.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.427+0000 7f641ef1fb80 20 client.7272 trim_cache size 0 max 16384 2023-10-16T12:27:07.455 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-10-16T12:27:07.455 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "5b28dfa1-b21f-474a-9b2e-9fb77af58254", 2023-10-16T12:27:07.455 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 0, 2023-10-16T12:27:07.456 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 7113, 2023-10-16T12:27:07.456 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2023-10-16T12:27:07.456 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2023-10-16T12:27:07.456 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2023-10-16T12:27:07.456 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 6.0320736310000003, 2023-10-16T12:27:07.456 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 44, 2023-10-16T12:27:07.457 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 99, 2023-10-16T12:27:07.457 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 99, 2023-10-16T12:27:07.457 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 11.921732390000001 2023-10-16T12:27:07.457 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-10-16T12:27:07.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.427+0000 7f6412d35700 1 -- 172.21.15.16:0/3576714569 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f63f006b050 msgr2=0x7f63f006d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:07.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.427+0000 7f6412d35700 1 --2- 172.21.15.16:0/3576714569 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f63f006b050 0x7f63f006d510 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f63f8009470 tx=0x7f63f8008040 comp rx=0 tx=0).stop 2023-10-16T12:27:07.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.427+0000 7f6412d35700 1 -- 172.21.15.16:0/3576714569 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f63fc007420 msgr2=0x7f640c0c6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:07.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.427+0000 7f6412d35700 1 --2- 172.21.15.16:0/3576714569 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f63fc007420 0x7f640c0c6280 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f640000f820 tx=0x7f640000fbe0 comp rx=0 tx=0).stop 2023-10-16T12:27:07.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.428+0000 7f6412d35700 1 -- 172.21.15.16:0/3576714569 shutdown_connections 2023-10-16T12:27:07.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.428+0000 7f6412d35700 1 --2- 172.21.15.16:0/3576714569 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f63f006b050 0x7f63f006d510 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:07.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.428+0000 7f6412d35700 1 --2- 172.21.15.16:0/3576714569 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f640c0cd020 0x7f640c0c3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:07.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.428+0000 7f6412d35700 1 --2- 172.21.15.16:0/3576714569 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f63fc007420 0x7f640c0c6280 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:07.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.428+0000 7f6412d35700 1 --2- 172.21.15.16:0/3576714569 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f640c14f780 0x7f640c0ccae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:07.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.428+0000 7f6412d35700 1 -- 172.21.15.16:0/3576714569 >> 172.21.15.16:0/3576714569 conn(0x7f640c0b1fd0 msgr2=0x7f640c0afe40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:07.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.428+0000 7f6412d35700 1 -- 172.21.15.16:0/3576714569 shutdown_connections 2023-10-16T12:27:07.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.428+0000 7f6412d35700 1 -- 172.21.15.16:0/3576714569 wait complete. 2023-10-16T12:27:07.461 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T12:27:07.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.586+0000 7f2d76b9d700 1 Processor -- start 2023-10-16T12:27:07.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.587+0000 7f2d76b9d700 1 -- start start 2023-10-16T12:27:07.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.587+0000 7f2d76b9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2d700cc8a0 0x7f2d7015e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:07.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.587+0000 7f2d76b9d700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d70161c30 con 0x7f2d7015b280 2023-10-16T12:27:07.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.587+0000 7f2d76b9d700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d70161f10 con 0x7f2d7006a7e0 2023-10-16T12:27:07.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.587+0000 7f2d76b9d700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d701621f0 con 0x7f2d700cc8a0 2023-10-16T12:27:07.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.587+0000 7f2d6ffff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2d700cc8a0 0x7f2d7015e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:07.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.587+0000 7f2d74939700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2d7015b280 0x7f2d701613e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:58522/0 (socket says 172.21.15.16:58522) 2023-10-16T12:27:07.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.587+0000 7f2d74939700 1 -- 172.21.15.16:0/1051043916 learned_addr learned my addr 172.21.15.16:0/1051043916 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:27:07.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.588+0000 7f2d6ffff700 1 -- 172.21.15.16:0/1051043916 >> v1:172.21.15.49:6789/0 conn(0x7f2d7006a7e0 legacy=0x7f2d700d02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:07.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.588+0000 7f2d6ffff700 1 -- 172.21.15.16:0/1051043916 >> v1:172.21.15.16:6789/0 conn(0x7f2d7015b280 legacy=0x7f2d701613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:07.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.588+0000 7f2d6ffff700 1 -- 172.21.15.16:0/1051043916 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d7015df00 con 0x7f2d700cc8a0 2023-10-16T12:27:07.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.588+0000 7f2d6f7fe700 1 -- 172.21.15.16:0/1051043916 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2416201498 0 0) 0x7f2d70161c30 con 0x7f2d7015b280 2023-10-16T12:27:07.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.588+0000 7f2d6ffff700 1 --2- 172.21.15.16:0/1051043916 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d700cc8a0 0x7f2d7015e700 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f2d5c009840 tx=0x7f2d5c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a755fbe95ee8222 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:07.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.588+0000 7f2d6f7fe700 1 -- 172.21.15.16:0/1051043916 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d5c004230 con 0x7f2d700cc8a0 2023-10-16T12:27:07.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.588+0000 7f2d6f7fe700 1 -- 172.21.15.16:0/1051043916 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d5c004390 con 0x7f2d700cc8a0 2023-10-16T12:27:07.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.588+0000 7f2d6f7fe700 1 -- 172.21.15.16:0/1051043916 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d5c0036a0 con 0x7f2d700cc8a0 2023-10-16T12:27:07.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.589+0000 7f2d76b9d700 1 -- 172.21.15.16:0/1051043916 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d700cc8a0 msgr2=0x7f2d7015e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:07.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.589+0000 7f2d76b9d700 1 --2- 172.21.15.16:0/1051043916 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d700cc8a0 0x7f2d7015e700 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f2d5c009840 tx=0x7f2d5c009b50 comp rx=0 tx=0).stop 2023-10-16T12:27:07.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.589+0000 7f2d76b9d700 1 -- 172.21.15.16:0/1051043916 shutdown_connections 2023-10-16T12:27:07.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.589+0000 7f2d76b9d700 1 --2- 172.21.15.16:0/1051043916 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d700cc8a0 0x7f2d7015e700 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:07.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.589+0000 7f2d76b9d700 1 -- 172.21.15.16:0/1051043916 >> 172.21.15.16:0/1051043916 conn(0x7f2d700b1fd0 msgr2=0x7f2d700b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:07.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.589+0000 7f2d76b9d700 1 -- 172.21.15.16:0/1051043916 shutdown_connections 2023-10-16T12:27:07.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.589+0000 7f2d76b9d700 1 -- 172.21.15.16:0/1051043916 wait complete. 2023-10-16T12:27:07.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.590+0000 7f2d76b9d700 1 Processor -- start 2023-10-16T12:27:07.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.590+0000 7f2d76b9d700 1 -- start start 2023-10-16T12:27:07.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.591+0000 7f2d76b9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d7006a7e0 0x7f2d70158880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:07.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.591+0000 7f2d76b9d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2d700cc8a0 0x7f2d700c3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:07.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.591+0000 7f2d76b9d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2d7015b280 0x7f2d701f3be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:07.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.591+0000 7f2d76b9d700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d700af9e0 con 0x7f2d7006a7e0 2023-10-16T12:27:07.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.591+0000 7f2d76b9d700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d70158dc0 con 0x7f2d700cc8a0 2023-10-16T12:27:07.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.591+0000 7f2d76b9d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d700c44b0 con 0x7f2d7015b280 2023-10-16T12:27:07.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.591+0000 7f2d74939700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d7006a7e0 0x7f2d70158880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:07.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.591+0000 7f2d74939700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d7006a7e0 0x7f2d70158880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41022/0 (socket says 172.21.15.16:41022) 2023-10-16T12:27:07.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.591+0000 7f2d74939700 1 -- 172.21.15.16:0/2650545053 learned_addr learned my addr 172.21.15.16:0/2650545053 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:07.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.591+0000 7f2d6ffff700 1 --2- 172.21.15.16:0/2650545053 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2d700cc8a0 0x7f2d700c3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:07.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.591+0000 7f2d7513a700 1 --2- 172.21.15.16:0/2650545053 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2d7015b280 0x7f2d701f3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:07.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.591+0000 7f2d74939700 1 -- 172.21.15.16:0/2650545053 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2d7015b280 msgr2=0x7f2d701f3be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:07.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.591+0000 7f2d74939700 1 --2- 172.21.15.16:0/2650545053 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2d7015b280 0x7f2d701f3be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:07.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.591+0000 7f2d74939700 1 -- 172.21.15.16:0/2650545053 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2d700cc8a0 msgr2=0x7f2d700c3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:07.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.591+0000 7f2d74939700 1 --2- 172.21.15.16:0/2650545053 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2d700cc8a0 0x7f2d700c3f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:07.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.591+0000 7f2d74939700 1 -- 172.21.15.16:0/2650545053 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d5c0094a0 con 0x7f2d7006a7e0 2023-10-16T12:27:07.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.591+0000 7f2d74939700 1 --2- 172.21.15.16:0/2650545053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d7006a7e0 0x7f2d70158880 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f2d6400c580 tx=0x7f2d6400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:07.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.592+0000 7f2d6d7fa700 1 -- 172.21.15.16:0/2650545053 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d64004c80 con 0x7f2d7006a7e0 2023-10-16T12:27:07.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.592+0000 7f2d6d7fa700 1 -- 172.21.15.16:0/2650545053 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d64004e00 con 0x7f2d7006a7e0 2023-10-16T12:27:07.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.592+0000 7f2d6d7fa700 1 -- 172.21.15.16:0/2650545053 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d64007310 con 0x7f2d7006a7e0 2023-10-16T12:27:07.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.592+0000 7f2d76b9d700 1 -- 172.21.15.16:0/2650545053 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d701f4120 con 0x7f2d7006a7e0 2023-10-16T12:27:07.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.592+0000 7f2d76b9d700 1 -- 172.21.15.16:0/2650545053 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d700c46e0 con 0x7f2d7006a7e0 2023-10-16T12:27:07.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.593+0000 7f2d76b9d700 1 -- 172.21.15.16:0/2650545053 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d7004ebb0 con 0x7f2d7006a7e0 2023-10-16T12:27:07.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.593+0000 7f2d6d7fa700 1 -- 172.21.15.16:0/2650545053 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 41) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2d64007470 con 0x7f2d7006a7e0 2023-10-16T12:27:07.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.594+0000 7f2d6d7fa700 1 --2- 172.21.15.16:0/2650545053 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f2d5406b020 0x7f2d5406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:07.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.594+0000 7f2d6d7fa700 1 -- 172.21.15.16:0/2650545053 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f2d64089240 con 0x7f2d7006a7e0 2023-10-16T12:27:07.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.594+0000 7f2d6ffff700 1 --2- 172.21.15.16:0/2650545053 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f2d5406b020 0x7f2d5406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:07.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.594+0000 7f2d6ffff700 1 --2- 172.21.15.16:0/2650545053 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f2d5406b020 0x7f2d5406d4e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2d5c009840 tx=0x7f2d5c01c040 comp rx=0 tx=0).ready entity=mgr.6020 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:07.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.598+0000 7f2d6d7fa700 1 -- 172.21.15.16:0/2650545053 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2d64054110 con 0x7f2d7006a7e0 2023-10-16T12:27:07.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.847+0000 7f2d76b9d700 1 -- 172.21.15.16:0/2650545053 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f2d701f4120 con 0x7f2d7006a7e0 2023-10-16T12:27:07.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.847+0000 7f2d6d7fa700 1 -- 172.21.15.16:0/2650545053 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v44) v1 ==== 53+0+73 (secure 0 0 0) 0x7f2d701f4120 con 0x7f2d7006a7e0 2023-10-16T12:27:07.849 INFO:teuthology.orchestra.run.smithi016.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-16T12:27:07.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.849+0000 7f2d76b9d700 1 -- 172.21.15.16:0/2650545053 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f2d5406b020 msgr2=0x7f2d5406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:07.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.849+0000 7f2d76b9d700 1 --2- 172.21.15.16:0/2650545053 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f2d5406b020 0x7f2d5406d4e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2d5c009840 tx=0x7f2d5c01c040 comp rx=0 tx=0).stop 2023-10-16T12:27:07.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.849+0000 7f2d76b9d700 1 -- 172.21.15.16:0/2650545053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d7006a7e0 msgr2=0x7f2d70158880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:07.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.849+0000 7f2d76b9d700 1 --2- 172.21.15.16:0/2650545053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d7006a7e0 0x7f2d70158880 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f2d6400c580 tx=0x7f2d6400c890 comp rx=0 tx=0).stop 2023-10-16T12:27:07.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.849+0000 7f2d76b9d700 1 -- 172.21.15.16:0/2650545053 shutdown_connections 2023-10-16T12:27:07.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.849+0000 7f2d76b9d700 1 --2- 172.21.15.16:0/2650545053 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f2d5406b020 0x7f2d5406d4e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:07.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.849+0000 7f2d76b9d700 1 --2- 172.21.15.16:0/2650545053 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2d7015b280 0x7f2d701f3be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:07.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.849+0000 7f2d76b9d700 1 --2- 172.21.15.16:0/2650545053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d7006a7e0 0x7f2d70158880 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:07.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.849+0000 7f2d76b9d700 1 --2- 172.21.15.16:0/2650545053 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2d700cc8a0 0x7f2d700c3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:07.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.849+0000 7f2d76b9d700 1 -- 172.21.15.16:0/2650545053 >> 172.21.15.16:0/2650545053 conn(0x7f2d700b1fd0 msgr2=0x7f2d7006abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:07.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.849+0000 7f2d76b9d700 1 -- 172.21.15.16:0/2650545053 shutdown_connections 2023-10-16T12:27:07.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:07.849+0000 7f2d76b9d700 1 -- 172.21.15.16:0/2650545053 wait complete. 2023-10-16T12:27:07.863 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-16T12:27:07.864 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T12:27:07.864 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi016.front.sepia.ceph.com 2023-10-16T12:27:07.864 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-16T12:27:07.865 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-16T12:27:07.865 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-16T12:27:07.865 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-16T12:27:07.866 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T12:27:07.866 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T12:27:07.866 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:27:07.867 DEBUG:teuthology.orchestra.run.smithi016:> ip addr 2023-10-16T12:27:07.885 INFO:teuthology.orchestra.run.smithi016.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T12:27:07.886 INFO:teuthology.orchestra.run.smithi016.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T12:27:07.886 INFO:teuthology.orchestra.run.smithi016.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T12:27:07.886 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:27:07.886 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 ::1/128 scope host 2023-10-16T12:27:07.886 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:27:07.887 INFO:teuthology.orchestra.run.smithi016.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T12:27:07.887 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e0 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:27:07.887 INFO:teuthology.orchestra.run.smithi016.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T12:27:07.887 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e1 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:27:07.887 INFO:teuthology.orchestra.run.smithi016.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T12:27:07.888 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:64 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:27:07.888 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.21.15.16/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-16T12:27:07.888 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft 41308sec preferred_lft 41308sec 2023-10-16T12:27:07.888 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::ec4:7aff:febd:1564/64 scope link 2023-10-16T12:27:07.888 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:27:07.889 INFO:teuthology.orchestra.run.smithi016.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T12:27:07.889 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:65 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:27:07.889 INFO:teuthology.orchestra.run.smithi016.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T12:27:07.889 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 36:cf:ea:25:db:f7 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:27:07.889 INFO:teuthology.orchestra.run.smithi016.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-16T12:27:07.890 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:27:07.890 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::cfc:c5ff:fe52:a786/64 scope link 2023-10-16T12:27:07.890 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:27:07.890 INFO:teuthology.orchestra.run.smithi016.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T12:27:07.891 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 6e:62:9d:a4:43:23 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T12:27:07.891 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::6c62:9dff:fea4:4323/64 scope link 2023-10-16T12:27:07.891 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:27:07.891 INFO:teuthology.orchestra.run.smithi016.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T12:27:07.891 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 36:cf:ea:25:db:f7 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-16T12:27:07.892 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::34cf:eaff:fe25:dbf7/64 scope link 2023-10-16T12:27:07.892 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:27:07.893 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:27:07.893 DEBUG:teuthology.orchestra.run.smithi016:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T12:27:07.951 INFO:teuthology.orchestra.run.smithi016.stdout:1 2023-10-16T12:27:07.953 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:27:07.953 DEBUG:teuthology.orchestra.run.smithi016:> route 2023-10-16T12:27:08.012 INFO:teuthology.orchestra.run.smithi016.stdout:Kernel IP routing table 2023-10-16T12:27:08.012 INFO:teuthology.orchestra.run.smithi016.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T12:27:08.012 INFO:teuthology.orchestra.run.smithi016.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-16T12:27:08.012 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-16T12:27:08.012 INFO:teuthology.orchestra.run.smithi016.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T12:27:08.013 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:27:08.013 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec bash -c ' 2023-10-16T12:27:08.014 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2023-10-16T12:27:08.014 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-16T12:27:08.015 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-16T12:27:08.015 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-16T12:27:08.016 DEBUG:teuthology.orchestra.run.smithi016:> ') 2023-10-16T12:27:08.120 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:27:08.120 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2023-10-16T12:27:08.138 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-16T12:27:08.138 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-16T12:27:08.139 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T12:27:08.139 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:27:08.139 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set brx.0 up 2023-10-16T12:27:08.206 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T12:27:08.207 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-16T12:27:08.263 INFO:teuthology.orchestra.run.smithi016.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-16T12:27:08.264 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T12:27:08.264 DEBUG:teuthology.orchestra.run.smithi016:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-16T12:27:08.320 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-16T12:27:08.320 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:27:08.321 DEBUG:teuthology.orchestra.run.smithi016:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T12:27:08.407 INFO:teuthology.orchestra.run.smithi016.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T12:27:08.408 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T12:27:08.408 INFO:teuthology.orchestra.run.smithi016.stdout:invalid new device string format 2023-10-16T12:27:08.417 INFO:teuthology.orchestra.run.smithi016.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T12:27:08.417 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T12:27:08.417 INFO:teuthology.orchestra.run.smithi016.stdout:invalid new device string format 2023-10-16T12:27:08.418 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: resolved to: "172.21.15.16:3300,172.21.15.49:3300,172.21.15.49:3301" 2023-10-16T12:27:08.418 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T12:27:08.490 INFO:teuthology.orchestra.run.smithi016.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T12:27:08.490 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T12:27:08.491 INFO:teuthology.orchestra.run.smithi016.stdout:invalid new device string format 2023-10-16T12:27:08.491 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: resolved to: "172.21.15.16:3300,172.21.15.49:3300,172.21.15.49:3301" 2023-10-16T12:27:08.491 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T12:27:08.491 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: resolved to: "172.21.15.16:6789,172.21.15.49:6789,172.21.15.49:6790" 2023-10-16T12:27:08.491 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.16:6789,172.21.15.49:6789,172.21.15.49:6790:/ 2023-10-16T12:27:08.492 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=5b28dfa1-b21f-474a-9b2e-9fb77af58254" will pass to kernel 2023-10-16T12:27:08.493 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-16T12:27:08.493 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:27:08.494 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-16T12:27:08.521 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:27:08.522 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T12:27:08.603 INFO:teuthology.orchestra.run.smithi016.stdout:client6290 2023-10-16T12:27:08.604 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:27:08.604 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:27:08.662 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-16T12:27:08.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:08.826+0000 7f0b1ddf1700 1 Processor -- start 2023-10-16T12:27:08.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:08.826+0000 7f0b1ddf1700 1 -- start start 2023-10-16T12:27:08.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:08.827+0000 7f0b1ddf1700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0b180cf4a0 0x7f0b180726e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:08.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:08.827+0000 7f0b1ddf1700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f0b180cf880 0x7f0b180b52f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:08.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:08.827+0000 7f0b1ddf1700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0b180b5830 0x7f0b1815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:08.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:08.827+0000 7f0b1ddf1700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b1815bc80 con 0x7f0b180b5830 2023-10-16T12:27:08.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:08.827+0000 7f0b1ddf1700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b18072c20 con 0x7f0b180cf880 2023-10-16T12:27:08.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:08.827+0000 7f0b1ddf1700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b1815b070 con 0x7f0b180cf4a0 2023-10-16T12:27:08.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:08.827+0000 7f0b17fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0b180b5830 0x7f0b1815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:08.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:08.827+0000 7f0b17fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0b180b5830 0x7f0b1815a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41036/0 (socket says 172.21.15.16:41036) 2023-10-16T12:27:08.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:08.827+0000 7f0b17fff700 1 -- 172.21.15.16:0/2781699643 learned_addr learned my addr 172.21.15.16:0/2781699643 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:08.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:08.827+0000 7f0b177fe700 1 --2- 172.21.15.16:0/2781699643 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0b180cf4a0 0x7f0b180726e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:08.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:08.827+0000 7f0b16ffd700 1 --2- 172.21.15.16:0/2781699643 >> v2:172.21.15.49:3300/0 conn(0x7f0b180cf880 0x7f0b180b52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:08.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:08.827+0000 7f0b17fff700 1 -- 172.21.15.16:0/2781699643 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0b180cf4a0 msgr2=0x7f0b180726e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:08.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:08.827+0000 7f0b17fff700 1 --2- 172.21.15.16:0/2781699643 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0b180cf4a0 0x7f0b180726e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:08.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:08.827+0000 7f0b17fff700 1 -- 172.21.15.16:0/2781699643 >> v2:172.21.15.49:3300/0 conn(0x7f0b180cf880 msgr2=0x7f0b180b52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:08.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:08.827+0000 7f0b17fff700 1 --2- 172.21.15.16:0/2781699643 >> v2:172.21.15.49:3300/0 conn(0x7f0b180cf880 0x7f0b180b52f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:08.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:08.827+0000 7f0b17fff700 1 -- 172.21.15.16:0/2781699643 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b1815b350 con 0x7f0b180b5830 2023-10-16T12:27:08.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:08.827+0000 7f0b177fe700 1 --2- 172.21.15.16:0/2781699643 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0b180cf4a0 0x7f0b180726e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:27:08.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:08.827+0000 7f0b17fff700 1 --2- 172.21.15.16:0/2781699643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b180b5830 0x7f0b1815a090 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f0b1806ead0 tx=0x7f0b1000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bce0c0f19bbd9a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:08.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:08.827+0000 7f0b167fc700 1 -- 172.21.15.16:0/2781699643 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b1000e070 con 0x7f0b180b5830 2023-10-16T12:27:08.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:08.828+0000 7f0b167fc700 1 -- 172.21.15.16:0/2781699643 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b1000b9c0 con 0x7f0b180b5830 2023-10-16T12:27:08.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:08.828+0000 7f0b167fc700 1 -- 172.21.15.16:0/2781699643 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b1000bd70 con 0x7f0b180b5830 2023-10-16T12:27:08.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:08.828+0000 7f0b1ddf1700 1 -- 172.21.15.16:0/2781699643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b180b5830 msgr2=0x7f0b1815a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:08.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:08.828+0000 7f0b1ddf1700 1 --2- 172.21.15.16:0/2781699643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b180b5830 0x7f0b1815a090 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f0b1806ead0 tx=0x7f0b1000b330 comp rx=0 tx=0).stop 2023-10-16T12:27:08.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:08.828+0000 7f0b1ddf1700 1 -- 172.21.15.16:0/2781699643 shutdown_connections 2023-10-16T12:27:08.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:08.828+0000 7f0b1ddf1700 1 --2- 172.21.15.16:0/2781699643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b180b5830 0x7f0b1815a090 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:08.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:08.828+0000 7f0b1ddf1700 1 --2- 172.21.15.16:0/2781699643 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0b180cf4a0 0x7f0b180726e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:08.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:08.828+0000 7f0b1ddf1700 1 --2- 172.21.15.16:0/2781699643 >> v2:172.21.15.49:3300/0 conn(0x7f0b180cf880 0x7f0b180b52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:08.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:08.828+0000 7f0b1ddf1700 1 -- 172.21.15.16:0/2781699643 >> 172.21.15.16:0/2781699643 conn(0x7f0b180b1fd0 msgr2=0x7f0b180b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:08.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:08.828+0000 7f0b1ddf1700 1 -- 172.21.15.16:0/2781699643 shutdown_connections 2023-10-16T12:27:08.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:08.828+0000 7f0b1ddf1700 1 -- 172.21.15.16:0/2781699643 wait complete. 2023-10-16T12:27:08.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:08.829+0000 7f0b1ddf1700 1 Processor -- start 2023-10-16T12:27:08.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:08.830+0000 7f0b1ddf1700 1 -- start start 2023-10-16T12:27:08.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:08.830+0000 7f0b1ddf1700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0b180b5830 0x7f0b18152420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:08.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:08.830+0000 7f0b1ddf1700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0b180cf4a0 0x7f0b180ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:08.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:08.830+0000 7f0b1ddf1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b180cf880 0x7f0b1815c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:08.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:08.830+0000 7f0b1ddf1700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b180ae960 con 0x7f0b180cf880 2023-10-16T12:27:08.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:08.830+0000 7f0b1ddf1700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b180c3f70 con 0x7f0b180b5830 2023-10-16T12:27:08.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:08.830+0000 7f0b1ddf1700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b18152960 con 0x7f0b180cf4a0 2023-10-16T12:27:08.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:08.830+0000 7f0b17fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b180cf880 0x7f0b1815c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:08.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:08.830+0000 7f0b17fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b180cf880 0x7f0b1815c770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41042/0 (socket says 172.21.15.16:41042) 2023-10-16T12:27:08.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:08.830+0000 7f0b17fff700 1 -- 172.21.15.16:0/4244292300 learned_addr learned my addr 172.21.15.16:0/4244292300 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:08.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:08.830+0000 7f0b177fe700 1 --2- 172.21.15.16:0/4244292300 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0b180b5830 0x7f0b18152420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:08.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:08.831+0000 7f0b16ffd700 1 --2- 172.21.15.16:0/4244292300 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0b180cf4a0 0x7f0b180ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:08.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:08.831+0000 7f0b17fff700 1 -- 172.21.15.16:0/4244292300 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0b180cf4a0 msgr2=0x7f0b180ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:08.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:08.831+0000 7f0b17fff700 1 --2- 172.21.15.16:0/4244292300 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0b180cf4a0 0x7f0b180ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:08.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:08.831+0000 7f0b17fff700 1 -- 172.21.15.16:0/4244292300 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0b180b5830 msgr2=0x7f0b18152420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:08.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:08.831+0000 7f0b17fff700 1 --2- 172.21.15.16:0/4244292300 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0b180b5830 0x7f0b18152420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:08.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:08.831+0000 7f0b17fff700 1 -- 172.21.15.16:0/4244292300 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b10009b30 con 0x7f0b180cf880 2023-10-16T12:27:08.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:08.831+0000 7f0b177fe700 1 --2- 172.21.15.16:0/4244292300 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0b180b5830 0x7f0b18152420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:27:08.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:08.831+0000 7f0b17fff700 1 --2- 172.21.15.16:0/4244292300 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b180cf880 0x7f0b1815c770 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f0b180cdfc0 tx=0x7f0b10004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:08.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:08.831+0000 7f0afffff700 1 -- 172.21.15.16:0/4244292300 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b10004250 con 0x7f0b180cf880 2023-10-16T12:27:08.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:08.831+0000 7f0b1ddf1700 1 -- 172.21.15.16:0/4244292300 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b180aec00 con 0x7f0b180cf880 2023-10-16T12:27:08.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:08.831+0000 7f0afffff700 1 -- 172.21.15.16:0/4244292300 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b100043d0 con 0x7f0b180cf880 2023-10-16T12:27:08.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:08.831+0000 7f0afffff700 1 -- 172.21.15.16:0/4244292300 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b10004780 con 0x7f0b180cf880 2023-10-16T12:27:08.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:08.831+0000 7f0b1ddf1700 1 -- 172.21.15.16:0/4244292300 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b180af1c0 con 0x7f0b180cf880 2023-10-16T12:27:08.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:08.832+0000 7f0b1ddf1700 1 -- 172.21.15.16:0/4244292300 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b1804ebb0 con 0x7f0b180cf880 2023-10-16T12:27:08.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:08.832+0000 7f0afffff700 1 -- 172.21.15.16:0/4244292300 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 41) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0b1001c440 con 0x7f0b180cf880 2023-10-16T12:27:08.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:08.833+0000 7f0afffff700 1 --2- 172.21.15.16:0/4244292300 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f0b0406b020 0x7f0b0406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:08.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:08.833+0000 7f0afffff700 1 -- 172.21.15.16:0/4244292300 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f0b1008af80 con 0x7f0b180cf880 2023-10-16T12:27:08.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:08.833+0000 7f0b177fe700 1 --2- 172.21.15.16:0/4244292300 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f0b0406b020 0x7f0b0406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:08.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:08.833+0000 7f0b177fe700 1 --2- 172.21.15.16:0/4244292300 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f0b0406b020 0x7f0b0406d4e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0b08006fd0 tx=0x7f0b08008040 comp rx=0 tx=0).ready entity=mgr.6020 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:08.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:08.837+0000 7f0afffff700 1 -- 172.21.15.16:0/4244292300 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0b100bdfe0 con 0x7f0b180cf880 2023-10-16T12:27:09.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.068+0000 7f0b1ddf1700 1 -- 172.21.15.16:0/4244292300 --> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f0b1815b910 con 0x7f0b0406b020 2023-10-16T12:27:09.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.069+0000 7f0afffff700 1 -- 172.21.15.16:0/4244292300 <== mgr.6020 v2:172.21.15.16:6832/292891612 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f0b0800c050 con 0x7f0b0406b020 2023-10-16T12:27:09.071 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2023-10-16T12:27:09.071 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:27:09.072 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cephfs" 2023-10-16T12:27:09.072 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:27:09.072 INFO:teuthology.orchestra.run.smithi016.stdout:] 2023-10-16T12:27:09.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.071+0000 7f0b1ddf1700 1 -- 172.21.15.16:0/4244292300 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f0b0406b020 msgr2=0x7f0b0406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:09.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.071+0000 7f0b1ddf1700 1 --2- 172.21.15.16:0/4244292300 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f0b0406b020 0x7f0b0406d4e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0b08006fd0 tx=0x7f0b08008040 comp rx=0 tx=0).stop 2023-10-16T12:27:09.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.071+0000 7f0b1ddf1700 1 -- 172.21.15.16:0/4244292300 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b180cf880 msgr2=0x7f0b1815c770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:09.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.071+0000 7f0b1ddf1700 1 --2- 172.21.15.16:0/4244292300 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b180cf880 0x7f0b1815c770 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f0b180cdfc0 tx=0x7f0b10004ad0 comp rx=0 tx=0).stop 2023-10-16T12:27:09.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.071+0000 7f0b1ddf1700 1 -- 172.21.15.16:0/4244292300 shutdown_connections 2023-10-16T12:27:09.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.071+0000 7f0b1ddf1700 1 --2- 172.21.15.16:0/4244292300 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f0b0406b020 0x7f0b0406d4e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:09.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.071+0000 7f0b1ddf1700 1 --2- 172.21.15.16:0/4244292300 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b180cf880 0x7f0b1815c770 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:09.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.071+0000 7f0b1ddf1700 1 --2- 172.21.15.16:0/4244292300 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0b180b5830 0x7f0b18152420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:09.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.071+0000 7f0b1ddf1700 1 --2- 172.21.15.16:0/4244292300 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0b180cf4a0 0x7f0b180ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:09.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.071+0000 7f0b1ddf1700 1 -- 172.21.15.16:0/4244292300 >> 172.21.15.16:0/4244292300 conn(0x7f0b180b1fd0 msgr2=0x7f0b180afed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:09.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.071+0000 7f0b1ddf1700 1 -- 172.21.15.16:0/4244292300 shutdown_connections 2023-10-16T12:27:09.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.071+0000 7f0b1ddf1700 1 -- 172.21.15.16:0/4244292300 wait complete. 2023-10-16T12:27:09.086 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-16T12:27:09.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.210+0000 7f54d95f4700 1 Processor -- start 2023-10-16T12:27:09.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.210+0000 7f54d95f4700 1 -- start start 2023-10-16T12:27:09.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.211+0000 7f54d95f4700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f54d415ec80 0x7f54d40cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:09.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.211+0000 7f54d95f4700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f54d415f060 0x7f54d40ce400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:09.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.211+0000 7f54d95f4700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f54d40d2110 0x7f54d40b4440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:09.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.211+0000 7f54d95f4700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54d40b2c50 con 0x7f54d40d2110 2023-10-16T12:27:09.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.211+0000 7f54d95f4700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54d40b2440 con 0x7f54d415ec80 2023-10-16T12:27:09.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.211+0000 7f54d95f4700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54d40d00e0 con 0x7f54d415f060 2023-10-16T12:27:09.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.211+0000 7f54d37fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f54d40d2110 0x7f54d40b4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:09.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.211+0000 7f54d37fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f54d40d2110 0x7f54d40b4440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41050/0 (socket says 172.21.15.16:41050) 2023-10-16T12:27:09.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.211+0000 7f54d37fe700 1 -- 172.21.15.16:0/1852109084 learned_addr learned my addr 172.21.15.16:0/1852109084 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:09.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.212+0000 7f54d2ffd700 1 --2- 172.21.15.16:0/1852109084 >> v2:172.21.15.49:3300/0 conn(0x7f54d415ec80 0x7f54d40cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:09.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.212+0000 7f54d27fc700 1 --2- 172.21.15.16:0/1852109084 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f54d415f060 0x7f54d40ce400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:09.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.212+0000 7f54d37fe700 1 -- 172.21.15.16:0/1852109084 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f54d415f060 msgr2=0x7f54d40ce400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:09.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.212+0000 7f54d37fe700 1 --2- 172.21.15.16:0/1852109084 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f54d415f060 0x7f54d40ce400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:09.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.212+0000 7f54d37fe700 1 -- 172.21.15.16:0/1852109084 >> v2:172.21.15.49:3300/0 conn(0x7f54d415ec80 msgr2=0x7f54d40cdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:09.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.212+0000 7f54d37fe700 1 --2- 172.21.15.16:0/1852109084 >> v2:172.21.15.49:3300/0 conn(0x7f54d415ec80 0x7f54d40cdec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:09.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.212+0000 7f54d37fe700 1 -- 172.21.15.16:0/1852109084 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54d40cef60 con 0x7f54d40d2110 2023-10-16T12:27:09.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.212+0000 7f54d2ffd700 1 --2- 172.21.15.16:0/1852109084 >> v2:172.21.15.49:3300/0 conn(0x7f54d415ec80 0x7f54d40cdec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:27:09.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.212+0000 7f54d27fc700 1 --2- 172.21.15.16:0/1852109084 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f54d415f060 0x7f54d40ce400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:27:09.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.212+0000 7f54d37fe700 1 --2- 172.21.15.16:0/1852109084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54d40d2110 0x7f54d40b4440 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f54d4067b20 tx=0x7f54cc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf8ecb4811500034 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:09.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.212+0000 7f54d1ffb700 1 -- 172.21.15.16:0/1852109084 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54cc00e070 con 0x7f54d40d2110 2023-10-16T12:27:09.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.212+0000 7f54d1ffb700 1 -- 172.21.15.16:0/1852109084 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54cc00b9c0 con 0x7f54d40d2110 2023-10-16T12:27:09.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.212+0000 7f54d1ffb700 1 -- 172.21.15.16:0/1852109084 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54cc00bd70 con 0x7f54d40d2110 2023-10-16T12:27:09.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.213+0000 7f54d95f4700 1 -- 172.21.15.16:0/1852109084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54d40d2110 msgr2=0x7f54d40b4440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:09.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.213+0000 7f54d95f4700 1 --2- 172.21.15.16:0/1852109084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54d40d2110 0x7f54d40b4440 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f54d4067b20 tx=0x7f54cc00b330 comp rx=0 tx=0).stop 2023-10-16T12:27:09.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.213+0000 7f54d95f4700 1 -- 172.21.15.16:0/1852109084 shutdown_connections 2023-10-16T12:27:09.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.213+0000 7f54d95f4700 1 --2- 172.21.15.16:0/1852109084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54d40d2110 0x7f54d40b4440 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:09.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.213+0000 7f54d95f4700 1 --2- 172.21.15.16:0/1852109084 >> v2:172.21.15.49:3300/0 conn(0x7f54d415ec80 0x7f54d40cdec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:09.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.213+0000 7f54d95f4700 1 --2- 172.21.15.16:0/1852109084 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f54d415f060 0x7f54d40ce400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:09.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.213+0000 7f54d95f4700 1 -- 172.21.15.16:0/1852109084 >> 172.21.15.16:0/1852109084 conn(0x7f54d40d15f0 msgr2=0x7f54d40d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:09.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.213+0000 7f54d95f4700 1 -- 172.21.15.16:0/1852109084 shutdown_connections 2023-10-16T12:27:09.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.213+0000 7f54d95f4700 1 -- 172.21.15.16:0/1852109084 wait complete. 2023-10-16T12:27:09.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.214+0000 7f54d95f4700 1 Processor -- start 2023-10-16T12:27:09.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.214+0000 7f54d95f4700 1 -- start start 2023-10-16T12:27:09.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.215+0000 7f54d95f4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54d40d2110 0x7f54d40c7e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:09.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.215+0000 7f54d95f4700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f54d415ec80 0x7f54d4157970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:09.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.215+0000 7f54d95f4700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f54d415f060 0x7f54d4152320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:09.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.215+0000 7f54d95f4700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54d40c8370 con 0x7f54d40d2110 2023-10-16T12:27:09.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.215+0000 7f54d95f4700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54d4158310 con 0x7f54d415f060 2023-10-16T12:27:09.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.215+0000 7f54d95f4700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54d41585f0 con 0x7f54d415ec80 2023-10-16T12:27:09.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.215+0000 7f54d2ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54d40d2110 0x7f54d40c7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:09.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.215+0000 7f54d2ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54d40d2110 0x7f54d40c7e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41052/0 (socket says 172.21.15.16:41052) 2023-10-16T12:27:09.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.215+0000 7f54d2ffd700 1 -- 172.21.15.16:0/1813569951 learned_addr learned my addr 172.21.15.16:0/1813569951 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:09.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.215+0000 7f54d37fe700 1 --2- 172.21.15.16:0/1813569951 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f54d415f060 0x7f54d4152320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:09.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.215+0000 7f54d27fc700 1 --2- 172.21.15.16:0/1813569951 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f54d415ec80 0x7f54d4157970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:09.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.215+0000 7f54d2ffd700 1 -- 172.21.15.16:0/1813569951 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f54d415ec80 msgr2=0x7f54d4157970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:09.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.215+0000 7f54d2ffd700 1 --2- 172.21.15.16:0/1813569951 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f54d415ec80 0x7f54d4157970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:09.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.215+0000 7f54d2ffd700 1 -- 172.21.15.16:0/1813569951 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f54d415f060 msgr2=0x7f54d4152320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:09.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.215+0000 7f54d2ffd700 1 --2- 172.21.15.16:0/1813569951 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f54d415f060 0x7f54d4152320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:09.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.215+0000 7f54d2ffd700 1 -- 172.21.15.16:0/1813569951 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54cc009b30 con 0x7f54d40d2110 2023-10-16T12:27:09.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.215+0000 7f54d2ffd700 1 --2- 172.21.15.16:0/1813569951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54d40d2110 0x7f54d40c7e30 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f54c800d800 tx=0x7f54c800dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:09.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.216+0000 7f54bb7fe700 1 -- 172.21.15.16:0/1813569951 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54c8004030 con 0x7f54d40d2110 2023-10-16T12:27:09.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.216+0000 7f54bb7fe700 1 -- 172.21.15.16:0/1813569951 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54c8004190 con 0x7f54d40d2110 2023-10-16T12:27:09.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.216+0000 7f54bb7fe700 1 -- 172.21.15.16:0/1813569951 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54c8004520 con 0x7f54d40d2110 2023-10-16T12:27:09.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.216+0000 7f54d95f4700 1 -- 172.21.15.16:0/1813569951 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54d41588d0 con 0x7f54d40d2110 2023-10-16T12:27:09.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.216+0000 7f54d95f4700 1 -- 172.21.15.16:0/1813569951 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54d4157eb0 con 0x7f54d40d2110 2023-10-16T12:27:09.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.217+0000 7f54d95f4700 1 -- 172.21.15.16:0/1813569951 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54d41580d0 con 0x7f54d40d2110 2023-10-16T12:27:09.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.217+0000 7f54bb7fe700 1 -- 172.21.15.16:0/1813569951 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 41) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f54c80036a0 con 0x7f54d40d2110 2023-10-16T12:27:09.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.218+0000 7f54bb7fe700 1 --2- 172.21.15.16:0/1813569951 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f54c006b020 0x7f54c006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:09.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.218+0000 7f54d27fc700 1 --2- 172.21.15.16:0/1813569951 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f54c006b020 0x7f54c006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:09.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.218+0000 7f54bb7fe700 1 -- 172.21.15.16:0/1813569951 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f54c8089ff0 con 0x7f54d40d2110 2023-10-16T12:27:09.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.218+0000 7f54d27fc700 1 --2- 172.21.15.16:0/1813569951 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f54c006b020 0x7f54c006d4e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f54d40cbb40 tx=0x7f54c4008040 comp rx=0 tx=0).ready entity=mgr.6020 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:09.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.222+0000 7f54bb7fe700 1 -- 172.21.15.16:0/1813569951 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f54c8054f40 con 0x7f54d40d2110 2023-10-16T12:27:09.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:09.495+0000 7f54d95f4700 1 -- 172.21.15.16:0/1813569951 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f54d41580d0 con 0x7f54d40d2110 2023-10-16T12:27:10.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.429+0000 7f54bb7fe700 1 -- 172.21.15.16:0/1813569951 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v42) v1 ==== 92+0+0 (secure 0 0 0) 0x7f54d41580d0 con 0x7f54d40d2110 2023-10-16T12:27:10.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.434+0000 7f54d95f4700 1 -- 172.21.15.16:0/1813569951 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f54c006b020 msgr2=0x7f54c006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:10.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.434+0000 7f54d95f4700 1 --2- 172.21.15.16:0/1813569951 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f54c006b020 0x7f54c006d4e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f54d40cbb40 tx=0x7f54c4008040 comp rx=0 tx=0).stop 2023-10-16T12:27:10.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.434+0000 7f54d95f4700 1 -- 172.21.15.16:0/1813569951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54d40d2110 msgr2=0x7f54d40c7e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:10.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.434+0000 7f54d95f4700 1 --2- 172.21.15.16:0/1813569951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54d40d2110 0x7f54d40c7e30 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f54c800d800 tx=0x7f54c800dbc0 comp rx=0 tx=0).stop 2023-10-16T12:27:10.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.435+0000 7f54d95f4700 1 -- 172.21.15.16:0/1813569951 shutdown_connections 2023-10-16T12:27:10.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.435+0000 7f54d95f4700 1 --2- 172.21.15.16:0/1813569951 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f54c006b020 0x7f54c006d4e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:10.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.435+0000 7f54d95f4700 1 --2- 172.21.15.16:0/1813569951 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f54d415f060 0x7f54d4152320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:10.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.435+0000 7f54d95f4700 1 --2- 172.21.15.16:0/1813569951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54d40d2110 0x7f54d40c7e30 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:10.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.435+0000 7f54d95f4700 1 --2- 172.21.15.16:0/1813569951 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f54d415ec80 0x7f54d4157970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:10.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.435+0000 7f54d95f4700 1 -- 172.21.15.16:0/1813569951 >> 172.21.15.16:0/1813569951 conn(0x7f54d40d15f0 msgr2=0x7f54d4153af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:10.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.435+0000 7f54d95f4700 1 -- 172.21.15.16:0/1813569951 shutdown_connections 2023-10-16T12:27:10.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.435+0000 7f54d95f4700 1 -- 172.21.15.16:0/1813569951 wait complete. 2023-10-16T12:27:10.458 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-16T12:27:10.582 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:27:10.580+0000 7ff6f8ef0200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T12:27:10.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.595+0000 7f5b4adac700 1 Processor -- start 2023-10-16T12:27:10.597 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:10.595+0000 7f10178a2200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T12:27:10.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.595+0000 7f5b4adac700 1 -- start start 2023-10-16T12:27:10.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.595+0000 7f5b4adac700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5b440cde90 0x7f5b4415ea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:10.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.595+0000 7f5b4adac700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f5b440cb6f0 0x7f5b4415efa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:10.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.596+0000 7f5b4adac700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5b440ce270 0x7f5b440cf2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:10.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.596+0000 7f5b4adac700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b440cee50 con 0x7f5b440ce270 2023-10-16T12:27:10.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.596+0000 7f5b4adac700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b440c5020 con 0x7f5b440cb6f0 2023-10-16T12:27:10.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.596+0000 7f5b4adac700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b440d2340 con 0x7f5b440cde90 2023-10-16T12:27:10.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.596+0000 7f5b49349700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5b440ce270 0x7f5b440cf2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:10.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.596+0000 7f5b49349700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5b440ce270 0x7f5b440cf2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41090/0 (socket says 172.21.15.16:41090) 2023-10-16T12:27:10.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.596+0000 7f5b49349700 1 -- 172.21.15.16:0/2586131943 learned_addr learned my addr 172.21.15.16:0/2586131943 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:10.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.596+0000 7f5b48b48700 1 --2- 172.21.15.16:0/2586131943 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5b440cde90 0x7f5b4415ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:10.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.596+0000 7f5b43fff700 1 --2- 172.21.15.16:0/2586131943 >> v2:172.21.15.49:3300/0 conn(0x7f5b440cb6f0 0x7f5b4415efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:10.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.596+0000 7f5b49349700 1 -- 172.21.15.16:0/2586131943 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5b440cde90 msgr2=0x7f5b4415ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:10.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.596+0000 7f5b49349700 1 --2- 172.21.15.16:0/2586131943 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5b440cde90 0x7f5b4415ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:10.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.596+0000 7f5b49349700 1 -- 172.21.15.16:0/2586131943 >> v2:172.21.15.49:3300/0 conn(0x7f5b440cb6f0 msgr2=0x7f5b4415efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:10.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.596+0000 7f5b49349700 1 --2- 172.21.15.16:0/2586131943 >> v2:172.21.15.49:3300/0 conn(0x7f5b440cb6f0 0x7f5b4415efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:10.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.596+0000 7f5b49349700 1 -- 172.21.15.16:0/2586131943 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b440ccdd0 con 0x7f5b440ce270 2023-10-16T12:27:10.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.596+0000 7f5b49349700 1 --2- 172.21.15.16:0/2586131943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b440ce270 0x7f5b440cf2b0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f5b4408bd00 tx=0x7f5b3c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b104208d47dc942b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:10.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.596+0000 7f5b437fe700 1 -- 172.21.15.16:0/2586131943 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b3c00e070 con 0x7f5b440ce270 2023-10-16T12:27:10.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.596+0000 7f5b437fe700 1 -- 172.21.15.16:0/2586131943 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b3c00b910 con 0x7f5b440ce270 2023-10-16T12:27:10.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.596+0000 7f5b437fe700 1 -- 172.21.15.16:0/2586131943 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b3c00bcc0 con 0x7f5b440ce270 2023-10-16T12:27:10.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.597+0000 7f5b4adac700 1 -- 172.21.15.16:0/2586131943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b440ce270 msgr2=0x7f5b440cf2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:10.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.597+0000 7f5b4adac700 1 --2- 172.21.15.16:0/2586131943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b440ce270 0x7f5b440cf2b0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f5b4408bd00 tx=0x7f5b3c00b330 comp rx=0 tx=0).stop 2023-10-16T12:27:10.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.597+0000 7f5b4adac700 1 -- 172.21.15.16:0/2586131943 shutdown_connections 2023-10-16T12:27:10.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.597+0000 7f5b4adac700 1 --2- 172.21.15.16:0/2586131943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b440ce270 0x7f5b440cf2b0 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:10.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.597+0000 7f5b4adac700 1 --2- 172.21.15.16:0/2586131943 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5b440cde90 0x7f5b4415ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:10.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.597+0000 7f5b4adac700 1 --2- 172.21.15.16:0/2586131943 >> v2:172.21.15.49:3300/0 conn(0x7f5b440cb6f0 0x7f5b4415efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:10.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.597+0000 7f5b4adac700 1 -- 172.21.15.16:0/2586131943 >> 172.21.15.16:0/2586131943 conn(0x7f5b440d14d0 msgr2=0x7f5b440d18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:10.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.597+0000 7f5b4adac700 1 -- 172.21.15.16:0/2586131943 shutdown_connections 2023-10-16T12:27:10.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.597+0000 7f5b4adac700 1 -- 172.21.15.16:0/2586131943 wait complete. 2023-10-16T12:27:10.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.598+0000 7f5b4adac700 1 Processor -- start 2023-10-16T12:27:10.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.598+0000 7f5b4adac700 1 -- start start 2023-10-16T12:27:10.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.599+0000 7f5b4adac700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5b440cb6f0 0x7f5b44158950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:10.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.599+0000 7f5b4adac700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5b440cde90 0x7f5b440b40f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:10.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.599+0000 7f5b4adac700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b440ce270 0x7f5b4409e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:10.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.599+0000 7f5b4adac700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b440b4bb0 con 0x7f5b440ce270 2023-10-16T12:27:10.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.599+0000 7f5b4adac700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b440b4e90 con 0x7f5b440cde90 2023-10-16T12:27:10.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.599+0000 7f5b4adac700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b440b5170 con 0x7f5b440cb6f0 2023-10-16T12:27:10.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.599+0000 7f5b49349700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b440ce270 0x7f5b4409e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:10.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.599+0000 7f5b49349700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b440ce270 0x7f5b4409e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41096/0 (socket says 172.21.15.16:41096) 2023-10-16T12:27:10.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.599+0000 7f5b49349700 1 -- 172.21.15.16:0/3640634347 learned_addr learned my addr 172.21.15.16:0/3640634347 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:10.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.599+0000 7f5b48b48700 1 --2- 172.21.15.16:0/3640634347 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5b440cb6f0 0x7f5b44158950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:10.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.599+0000 7f5b43fff700 1 --2- 172.21.15.16:0/3640634347 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5b440cde90 0x7f5b440b40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:10.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.600+0000 7f5b49349700 1 -- 172.21.15.16:0/3640634347 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5b440cb6f0 msgr2=0x7f5b44158950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:10.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.600+0000 7f5b49349700 1 --2- 172.21.15.16:0/3640634347 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5b440cb6f0 0x7f5b44158950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:10.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.600+0000 7f5b49349700 1 -- 172.21.15.16:0/3640634347 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5b440cde90 msgr2=0x7f5b440b40f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:10.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.600+0000 7f5b49349700 1 --2- 172.21.15.16:0/3640634347 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5b440cde90 0x7f5b440b40f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:10.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.600+0000 7f5b49349700 1 -- 172.21.15.16:0/3640634347 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b3c009b30 con 0x7f5b440ce270 2023-10-16T12:27:10.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.600+0000 7f5b48b48700 1 --2- 172.21.15.16:0/3640634347 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5b440cb6f0 0x7f5b44158950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:27:10.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.600+0000 7f5b49349700 1 --2- 172.21.15.16:0/3640634347 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b440ce270 0x7f5b4409e600 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f5b4408b830 tx=0x7f5b3c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:10.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.600+0000 7f5b417fa700 1 -- 172.21.15.16:0/3640634347 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b3c0099b0 con 0x7f5b440ce270 2023-10-16T12:27:10.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.600+0000 7f5b417fa700 1 -- 172.21.15.16:0/3640634347 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b3c00ce80 con 0x7f5b440ce270 2023-10-16T12:27:10.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.600+0000 7f5b4adac700 1 -- 172.21.15.16:0/3640634347 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b440b5450 con 0x7f5b440ce270 2023-10-16T12:27:10.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.600+0000 7f5b417fa700 1 -- 172.21.15.16:0/3640634347 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b3c004490 con 0x7f5b440ce270 2023-10-16T12:27:10.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.600+0000 7f5b4adac700 1 -- 172.21.15.16:0/3640634347 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b440cb120 con 0x7f5b440ce270 2023-10-16T12:27:10.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.601+0000 7f5b4adac700 1 -- 172.21.15.16:0/3640634347 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b440c8880 con 0x7f5b440ce270 2023-10-16T12:27:10.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.602+0000 7f5b417fa700 1 -- 172.21.15.16:0/3640634347 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 42) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f5b3c0045f0 con 0x7f5b440ce270 2023-10-16T12:27:10.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.602+0000 7f5b417fa700 1 --2- 172.21.15.16:0/3640634347 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f5b2806b070 0x7f5b2806d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:10.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.602+0000 7f5b48b48700 1 -- 172.21.15.16:0/3640634347 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f5b2806b070 msgr2=0x7f5b2806d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/292891612 2023-10-16T12:27:10.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.602+0000 7f5b417fa700 1 -- 172.21.15.16:0/3640634347 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f5b3c08ac30 con 0x7f5b440ce270 2023-10-16T12:27:10.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.602+0000 7f5b48b48700 1 --2- 172.21.15.16:0/3640634347 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f5b2806b070 0x7f5b2806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:27:10.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.607+0000 7f5b417fa700 1 -- 172.21.15.16:0/3640634347 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5b440c8880 con 0x7f5b440ce270 2023-10-16T12:27:10.651 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:27:10.649+0000 7ff6f8ef0200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T12:27:10.668 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:10.666+0000 7f10178a2200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T12:27:10.725 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:27:10.723+0000 7ff6f8ef0200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T12:27:10.744 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:10.742+0000 7f10178a2200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T12:27:10.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.803+0000 7f5b48b48700 1 -- 172.21.15.16:0/3640634347 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f5b2806b070 msgr2=0x7f5b2806d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/292891612 2023-10-16T12:27:10.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.803+0000 7f5b48b48700 1 --2- 172.21.15.16:0/3640634347 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f5b2806b070 0x7f5b2806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:27:10.806 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:27:10.804+0000 7ff6f8ef0200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T12:27:10.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.811+0000 7f5b4adac700 1 -- 172.21.15.16:0/3640634347 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f5b440c8880 con 0x7f5b440ce270 2023-10-16T12:27:10.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.814+0000 7f5b417fa700 1 -- 172.21.15.16:0/3640634347 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v29)=0 v29) v1 ==== 138+0+0 (secure 0 0 0) 0x7f5b440c8880 con 0x7f5b440ce270 2023-10-16T12:27:10.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.817+0000 7f5b4adac700 1 -- 172.21.15.16:0/3640634347 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f5b2806b070 msgr2=0x7f5b2806d530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:27:10.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.817+0000 7f5b4adac700 1 --2- 172.21.15.16:0/3640634347 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f5b2806b070 0x7f5b2806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:10.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.817+0000 7f5b4adac700 1 -- 172.21.15.16:0/3640634347 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b440ce270 msgr2=0x7f5b4409e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:10.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.817+0000 7f5b4adac700 1 --2- 172.21.15.16:0/3640634347 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b440ce270 0x7f5b4409e600 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f5b4408b830 tx=0x7f5b3c004ad0 comp rx=0 tx=0).stop 2023-10-16T12:27:10.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.817+0000 7f5b4adac700 1 -- 172.21.15.16:0/3640634347 shutdown_connections 2023-10-16T12:27:10.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.817+0000 7f5b4adac700 1 --2- 172.21.15.16:0/3640634347 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f5b2806b070 0x7f5b2806d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:10.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.817+0000 7f5b4adac700 1 --2- 172.21.15.16:0/3640634347 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b440ce270 0x7f5b4409e600 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:10.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.817+0000 7f5b4adac700 1 --2- 172.21.15.16:0/3640634347 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5b440cb6f0 0x7f5b44158950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:10.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.817+0000 7f5b4adac700 1 --2- 172.21.15.16:0/3640634347 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5b440cde90 0x7f5b440b40f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:10.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.817+0000 7f5b4adac700 1 -- 172.21.15.16:0/3640634347 >> 172.21.15.16:0/3640634347 conn(0x7f5b440d14d0 msgr2=0x7f5b440cfd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:10.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.817+0000 7f5b4adac700 1 -- 172.21.15.16:0/3640634347 shutdown_connections 2023-10-16T12:27:10.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.817+0000 7f5b4adac700 1 -- 172.21.15.16:0/3640634347 wait complete. 2023-10-16T12:27:10.827 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:10.825+0000 7f10178a2200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T12:27:10.831 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-16T12:27:10.953 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:27:10.951+0000 7ff6f8ef0200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T12:27:10.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.960+0000 7f081a0f9700 1 Processor -- start 2023-10-16T12:27:10.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.960+0000 7f081a0f9700 1 -- start start 2023-10-16T12:27:10.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.961+0000 7f081a0f9700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f081406e570 0x7f081415e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:10.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.961+0000 7f081a0f9700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f081415b6a0 con 0x7f081415b2c0 2023-10-16T12:27:10.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.961+0000 7f081a0f9700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0814161b20 con 0x7f081406a7e0 2023-10-16T12:27:10.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.961+0000 7f081a0f9700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0814161e00 con 0x7f081406e570 2023-10-16T12:27:10.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.961+0000 7f08137fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f081415b2c0 0x7f0814161340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:58636/0 (socket says 172.21.15.16:58636) 2023-10-16T12:27:10.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.961+0000 7f08137fe700 1 -- 172.21.15.16:0/3483527569 learned_addr learned my addr 172.21.15.16:0/3483527569 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:27:10.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.961+0000 7f0812ffd700 1 --2- 172.21.15.16:0/3483527569 >> v2:172.21.15.49:3300/0 conn(0x7f081406e570 0x7f081415e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:10.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.961+0000 7f08127fc700 1 -- 172.21.15.16:0/3483527569 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2858874589 0 0) 0x7f081415b6a0 con 0x7f081415b2c0 2023-10-16T12:27:10.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.961+0000 7f08127fc700 1 -- 172.21.15.16:0/3483527569 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07fc0035b0 con 0x7f081415b2c0 2023-10-16T12:27:10.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.962+0000 7f08127fc700 1 -- 172.21.15.16:0/3483527569 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2248963290 0 0) 0x7f0814161b20 con 0x7f081406a7e0 2023-10-16T12:27:10.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.962+0000 7f08127fc700 1 -- 172.21.15.16:0/3483527569 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f081415b6a0 con 0x7f081406a7e0 2023-10-16T12:27:10.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.962+0000 7f0812ffd700 1 -- 172.21.15.16:0/3483527569 >> v1:172.21.15.49:6789/0 conn(0x7f081406a7e0 legacy=0x7f0814071a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:10.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.962+0000 7f0812ffd700 1 -- 172.21.15.16:0/3483527569 >> v1:172.21.15.16:6789/0 conn(0x7f081415b2c0 legacy=0x7f0814161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:10.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.962+0000 7f0812ffd700 1 -- 172.21.15.16:0/3483527569 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08141620e0 con 0x7f081406e570 2023-10-16T12:27:10.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.962+0000 7f0812ffd700 1 --2- 172.21.15.16:0/3483527569 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f081406e570 0x7f081415e5a0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f0804009840 tx=0x7f0804009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e1d4dee1b5998b77 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:10.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.962+0000 7f08127fc700 1 -- 172.21.15.16:0/3483527569 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0804011830 con 0x7f081406e570 2023-10-16T12:27:10.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.962+0000 7f08127fc700 1 -- 172.21.15.16:0/3483527569 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0804011990 con 0x7f081406e570 2023-10-16T12:27:10.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.962+0000 7f08127fc700 1 -- 172.21.15.16:0/3483527569 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0804011d20 con 0x7f081406e570 2023-10-16T12:27:10.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.963+0000 7f081a0f9700 1 -- 172.21.15.16:0/3483527569 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f081406e570 msgr2=0x7f081415e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:10.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.963+0000 7f081a0f9700 1 --2- 172.21.15.16:0/3483527569 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f081406e570 0x7f081415e5a0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f0804009840 tx=0x7f0804009b50 comp rx=0 tx=0).stop 2023-10-16T12:27:10.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.963+0000 7f081a0f9700 1 -- 172.21.15.16:0/3483527569 shutdown_connections 2023-10-16T12:27:10.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.963+0000 7f081a0f9700 1 --2- 172.21.15.16:0/3483527569 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f081406e570 0x7f081415e5a0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:10.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.963+0000 7f081a0f9700 1 -- 172.21.15.16:0/3483527569 >> 172.21.15.16:0/3483527569 conn(0x7f08140b1f40 msgr2=0x7f08140b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:10.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.963+0000 7f081a0f9700 1 -- 172.21.15.16:0/3483527569 shutdown_connections 2023-10-16T12:27:10.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.963+0000 7f081a0f9700 1 -- 172.21.15.16:0/3483527569 wait complete. 2023-10-16T12:27:10.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.964+0000 7f081a0f9700 1 Processor -- start 2023-10-16T12:27:10.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.964+0000 7f081a0f9700 1 -- start start 2023-10-16T12:27:10.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.965+0000 7f081a0f9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f081406a7e0 0x7f0814146760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:10.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.965+0000 7f081a0f9700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f081406e570 0x7f08141499b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:10.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.965+0000 7f081a0f9700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f081415b2c0 0x7f08140cbe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:10.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.965+0000 7f081a0f9700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0814146ca0 con 0x7f081406a7e0 2023-10-16T12:27:10.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.965+0000 7f081a0f9700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08140ca490 con 0x7f081406e570 2023-10-16T12:27:10.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.965+0000 7f081a0f9700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0814149ef0 con 0x7f081415b2c0 2023-10-16T12:27:10.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.965+0000 7f08137fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f081406a7e0 0x7f0814146760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:10.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.965+0000 7f08137fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f081406a7e0 0x7f0814146760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41136/0 (socket says 172.21.15.16:41136) 2023-10-16T12:27:10.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.965+0000 7f08137fe700 1 -- 172.21.15.16:0/4043088672 learned_addr learned my addr 172.21.15.16:0/4043088672 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:10.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.965+0000 7f0812ffd700 1 --2- 172.21.15.16:0/4043088672 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f081406e570 0x7f08141499b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:10.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.965+0000 7f0813fff700 1 --2- 172.21.15.16:0/4043088672 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f081415b2c0 0x7f08140cbe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:10.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.965+0000 7f08137fe700 1 -- 172.21.15.16:0/4043088672 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f081415b2c0 msgr2=0x7f08140cbe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:10.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.965+0000 7f08137fe700 1 --2- 172.21.15.16:0/4043088672 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f081415b2c0 0x7f08140cbe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:10.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.965+0000 7f08137fe700 1 -- 172.21.15.16:0/4043088672 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f081406e570 msgr2=0x7f08141499b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:10.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.965+0000 7f08137fe700 1 --2- 172.21.15.16:0/4043088672 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f081406e570 0x7f08141499b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:10.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.965+0000 7f08137fe700 1 -- 172.21.15.16:0/4043088672 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08040094a0 con 0x7f081406a7e0 2023-10-16T12:27:10.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.965+0000 7f0812ffd700 1 --2- 172.21.15.16:0/4043088672 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f081406e570 0x7f08141499b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:27:10.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.966+0000 7f08137fe700 1 --2- 172.21.15.16:0/4043088672 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f081406a7e0 0x7f0814146760 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f080800c4d0 tx=0x7f080800c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:10.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.966+0000 7f07fbfff700 1 -- 172.21.15.16:0/4043088672 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f080800cea0 con 0x7f081406a7e0 2023-10-16T12:27:10.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.966+0000 7f07fbfff700 1 -- 172.21.15.16:0/4043088672 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0808005d70 con 0x7f081406a7e0 2023-10-16T12:27:10.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.966+0000 7f081a0f9700 1 -- 172.21.15.16:0/4043088672 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08140c9880 con 0x7f081406a7e0 2023-10-16T12:27:10.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.966+0000 7f07fbfff700 1 -- 172.21.15.16:0/4043088672 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08080072f0 con 0x7f081406a7e0 2023-10-16T12:27:10.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.966+0000 7f081a0f9700 1 -- 172.21.15.16:0/4043088672 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08140c9e40 con 0x7f081406a7e0 2023-10-16T12:27:10.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.967+0000 7f081a0f9700 1 -- 172.21.15.16:0/4043088672 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08140ca770 con 0x7f081406a7e0 2023-10-16T12:27:10.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.967+0000 7f07fbfff700 1 -- 172.21.15.16:0/4043088672 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 42) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f0808007450 con 0x7f081406a7e0 2023-10-16T12:27:10.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.968+0000 7f07fbfff700 1 --2- 172.21.15.16:0/4043088672 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f080006b040 0x7f080006d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:10.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.968+0000 7f0812ffd700 1 -- 172.21.15.16:0/4043088672 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f080006b040 msgr2=0x7f080006d500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/292891612 2023-10-16T12:27:10.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.968+0000 7f07fbfff700 1 -- 172.21.15.16:0/4043088672 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f0808088d70 con 0x7f081406a7e0 2023-10-16T12:27:10.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.968+0000 7f0812ffd700 1 --2- 172.21.15.16:0/4043088672 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f080006b040 0x7f080006d500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:27:10.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:10.972+0000 7f07fbfff700 1 -- 172.21.15.16:0/4043088672 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0808053b30 con 0x7f081406a7e0 2023-10-16T12:27:10.986 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:10.984+0000 7f10178a2200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T12:27:11.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.169+0000 7f0812ffd700 1 -- 172.21.15.16:0/4043088672 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f080006b040 msgr2=0x7f080006d500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/292891612 2023-10-16T12:27:11.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.169+0000 7f0812ffd700 1 --2- 172.21.15.16:0/4043088672 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f080006b040 0x7f080006d500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:27:11.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.178+0000 7f081a0f9700 1 -- 172.21.15.16:0/4043088672 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f081404ebb0 con 0x7f081406a7e0 2023-10-16T12:27:11.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.181+0000 7f07fbfff700 1 -- 172.21.15.16:0/4043088672 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v30)=0 v30) v1 ==== 133+0+0 (secure 0 0 0) 0x7f0808057150 con 0x7f081406a7e0 2023-10-16T12:27:11.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.184+0000 7f081a0f9700 1 -- 172.21.15.16:0/4043088672 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f080006b040 msgr2=0x7f080006d500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:27:11.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.184+0000 7f081a0f9700 1 --2- 172.21.15.16:0/4043088672 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f080006b040 0x7f080006d500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:11.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.184+0000 7f081a0f9700 1 -- 172.21.15.16:0/4043088672 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f081406a7e0 msgr2=0x7f0814146760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:11.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.184+0000 7f081a0f9700 1 --2- 172.21.15.16:0/4043088672 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f081406a7e0 0x7f0814146760 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f080800c4d0 tx=0x7f080800c7e0 comp rx=0 tx=0).stop 2023-10-16T12:27:11.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.184+0000 7f081a0f9700 1 -- 172.21.15.16:0/4043088672 shutdown_connections 2023-10-16T12:27:11.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.184+0000 7f081a0f9700 1 --2- 172.21.15.16:0/4043088672 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7f080006b040 0x7f080006d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:11.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.184+0000 7f081a0f9700 1 --2- 172.21.15.16:0/4043088672 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f081415b2c0 0x7f08140cbe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:11.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.184+0000 7f081a0f9700 1 --2- 172.21.15.16:0/4043088672 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f081406a7e0 0x7f0814146760 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:11.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.184+0000 7f081a0f9700 1 --2- 172.21.15.16:0/4043088672 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f081406e570 0x7f08141499b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:11.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.184+0000 7f081a0f9700 1 -- 172.21.15.16:0/4043088672 >> 172.21.15.16:0/4043088672 conn(0x7f08140b1f40 msgr2=0x7f08140cce00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:11.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.184+0000 7f081a0f9700 1 -- 172.21.15.16:0/4043088672 shutdown_connections 2023-10-16T12:27:11.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.184+0000 7f081a0f9700 1 -- 172.21.15.16:0/4043088672 wait complete. 2023-10-16T12:27:11.199 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list snap_test_dir1 2023-10-16T12:27:11.296 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:27:11.294+0000 7ff6f8ef0200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T12:27:11.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.325+0000 7fe521bf6700 1 Processor -- start 2023-10-16T12:27:11.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.325+0000 7fe521bf6700 1 -- start start 2023-10-16T12:27:11.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.326+0000 7fe521bf6700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe51c15da10 0x7fe51c0d1520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:11.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.326+0000 7fe521bf6700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fe51c0d4060 0x7fe51c0cf2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:11.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.326+0000 7fe521bf6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe51c0d1d20 0x7fe51c0d2130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:11.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.326+0000 7fe521bf6700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe51c0d2670 con 0x7fe51c0d1d20 2023-10-16T12:27:11.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.326+0000 7fe521bf6700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe51c0d4540 con 0x7fe51c0d4060 2023-10-16T12:27:11.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.326+0000 7fe521bf6700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe51c0ce7e0 con 0x7fe51c15da10 2023-10-16T12:27:11.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.326+0000 7fe51bfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe51c0d1d20 0x7fe51c0d2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:11.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.326+0000 7fe51bfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe51c0d1d20 0x7fe51c0d2130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41178/0 (socket says 172.21.15.16:41178) 2023-10-16T12:27:11.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.326+0000 7fe51bfff700 1 -- 172.21.15.16:0/3431378352 learned_addr learned my addr 172.21.15.16:0/3431378352 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:11.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.326+0000 7fe51b7fe700 1 --2- 172.21.15.16:0/3431378352 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe51c15da10 0x7fe51c0d1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:11.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.326+0000 7fe51affd700 1 --2- 172.21.15.16:0/3431378352 >> v2:172.21.15.49:3300/0 conn(0x7fe51c0d4060 0x7fe51c0cf2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:11.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.327+0000 7fe51bfff700 1 -- 172.21.15.16:0/3431378352 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe51c15da10 msgr2=0x7fe51c0d1520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:11.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.327+0000 7fe51bfff700 1 --2- 172.21.15.16:0/3431378352 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe51c15da10 0x7fe51c0d1520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:11.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.327+0000 7fe51bfff700 1 -- 172.21.15.16:0/3431378352 >> v2:172.21.15.49:3300/0 conn(0x7fe51c0d4060 msgr2=0x7fe51c0cf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:11.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.327+0000 7fe51bfff700 1 --2- 172.21.15.16:0/3431378352 >> v2:172.21.15.49:3300/0 conn(0x7fe51c0d4060 0x7fe51c0cf2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:11.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.327+0000 7fe51bfff700 1 -- 172.21.15.16:0/3431378352 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe51c0ceac0 con 0x7fe51c0d1d20 2023-10-16T12:27:11.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.327+0000 7fe51bfff700 1 --2- 172.21.15.16:0/3431378352 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe51c0d1d20 0x7fe51c0d2130 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7fe51c08d990 tx=0x7fe51400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23d871649e97dc9e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:11.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.327+0000 7fe51a7fc700 1 -- 172.21.15.16:0/3431378352 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe51400e070 con 0x7fe51c0d1d20 2023-10-16T12:27:11.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.327+0000 7fe51a7fc700 1 -- 172.21.15.16:0/3431378352 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe51400b910 con 0x7fe51c0d1d20 2023-10-16T12:27:11.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.327+0000 7fe51a7fc700 1 -- 172.21.15.16:0/3431378352 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe51400bcc0 con 0x7fe51c0d1d20 2023-10-16T12:27:11.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.327+0000 7fe521bf6700 1 -- 172.21.15.16:0/3431378352 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe51c0d1d20 msgr2=0x7fe51c0d2130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:11.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.327+0000 7fe521bf6700 1 --2- 172.21.15.16:0/3431378352 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe51c0d1d20 0x7fe51c0d2130 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7fe51c08d990 tx=0x7fe51400b330 comp rx=0 tx=0).stop 2023-10-16T12:27:11.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.328+0000 7fe521bf6700 1 -- 172.21.15.16:0/3431378352 shutdown_connections 2023-10-16T12:27:11.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.328+0000 7fe521bf6700 1 --2- 172.21.15.16:0/3431378352 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe51c0d1d20 0x7fe51c0d2130 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:11.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.328+0000 7fe521bf6700 1 --2- 172.21.15.16:0/3431378352 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe51c15da10 0x7fe51c0d1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:11.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.328+0000 7fe521bf6700 1 --2- 172.21.15.16:0/3431378352 >> v2:172.21.15.49:3300/0 conn(0x7fe51c0d4060 0x7fe51c0cf2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:11.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.328+0000 7fe521bf6700 1 -- 172.21.15.16:0/3431378352 >> 172.21.15.16:0/3431378352 conn(0x7fe51c09c9f0 msgr2=0x7fe51c0d3740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:11.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.328+0000 7fe521bf6700 1 -- 172.21.15.16:0/3431378352 shutdown_connections 2023-10-16T12:27:11.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.328+0000 7fe521bf6700 1 -- 172.21.15.16:0/3431378352 wait complete. 2023-10-16T12:27:11.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.329+0000 7fe521bf6700 1 Processor -- start 2023-10-16T12:27:11.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.329+0000 7fe521bf6700 1 -- start start 2023-10-16T12:27:11.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.330+0000 7fe521bf6700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe51c0d1d20 0x7fe51c1607b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:11.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.330+0000 7fe521bf6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe51c0d4060 0x7fe51c0b6d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:11.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.330+0000 7fe521bf6700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe51c15da10 0x7fe51c14f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:11.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.330+0000 7fe521bf6700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe51c0ca810 con 0x7fe51c0d4060 2023-10-16T12:27:11.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.330+0000 7fe521bf6700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe51c0b8380 con 0x7fe51c15da10 2023-10-16T12:27:11.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.330+0000 7fe521bf6700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe51c0caa60 con 0x7fe51c0d1d20 2023-10-16T12:27:11.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.330+0000 7fe51affd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe51c0d4060 0x7fe51c0b6d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:11.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.330+0000 7fe51affd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe51c0d4060 0x7fe51c0b6d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41182/0 (socket says 172.21.15.16:41182) 2023-10-16T12:27:11.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.330+0000 7fe51b7fe700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe51c0d1d20 0x7fe51c1607b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:11.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.330+0000 7fe51affd700 1 -- 172.21.15.16:0/2685476318 learned_addr learned my addr 172.21.15.16:0/2685476318 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:11.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.330+0000 7fe51bfff700 1 --2- 172.21.15.16:0/2685476318 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe51c15da10 0x7fe51c14f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:11.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.330+0000 7fe51affd700 1 -- 172.21.15.16:0/2685476318 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe51c0d1d20 msgr2=0x7fe51c1607b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:11.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.330+0000 7fe51affd700 1 --2- 172.21.15.16:0/2685476318 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe51c0d1d20 0x7fe51c1607b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:11.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.330+0000 7fe51affd700 1 -- 172.21.15.16:0/2685476318 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe51c15da10 msgr2=0x7fe51c14f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:11.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.330+0000 7fe51affd700 1 --2- 172.21.15.16:0/2685476318 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe51c15da10 0x7fe51c14f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:11.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.330+0000 7fe51affd700 1 -- 172.21.15.16:0/2685476318 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe514009b30 con 0x7fe51c0d4060 2023-10-16T12:27:11.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.330+0000 7fe51b7fe700 1 --2- 172.21.15.16:0/2685476318 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe51c0d1d20 0x7fe51c1607b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:27:11.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.330+0000 7fe51affd700 1 --2- 172.21.15.16:0/2685476318 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe51c0d4060 0x7fe51c0b6d10 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7fe51000e930 tx=0x7fe51000ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:11.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.331+0000 7fe503fff700 1 -- 172.21.15.16:0/2685476318 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe510013680 con 0x7fe51c0d4060 2023-10-16T12:27:11.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.331+0000 7fe503fff700 1 -- 172.21.15.16:0/2685476318 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5100137e0 con 0x7fe51c0d4060 2023-10-16T12:27:11.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.331+0000 7fe521bf6700 1 -- 172.21.15.16:0/2685476318 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe51c0cad40 con 0x7fe51c0d4060 2023-10-16T12:27:11.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.331+0000 7fe503fff700 1 -- 172.21.15.16:0/2685476318 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe510013b70 con 0x7fe51c0d4060 2023-10-16T12:27:11.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.331+0000 7fe521bf6700 1 -- 172.21.15.16:0/2685476318 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe51c0cb300 con 0x7fe51c0d4060 2023-10-16T12:27:11.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.332+0000 7fe521bf6700 1 -- 172.21.15.16:0/2685476318 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe51c04ebb0 con 0x7fe51c0d4060 2023-10-16T12:27:11.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.332+0000 7fe503fff700 1 -- 172.21.15.16:0/2685476318 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 42) v1 ==== 89348+0+0 (secure 0 0 0) 0x7fe510013cd0 con 0x7fe51c0d4060 2023-10-16T12:27:11.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.333+0000 7fe503fff700 1 --2- 172.21.15.16:0/2685476318 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7fe50806b070 0x7fe50806d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:11.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.333+0000 7fe51b7fe700 1 -- 172.21.15.16:0/2685476318 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7fe50806b070 msgr2=0x7fe50806d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/292891612 2023-10-16T12:27:11.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.333+0000 7fe503fff700 1 -- 172.21.15.16:0/2685476318 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fe510089820 con 0x7fe51c0d4060 2023-10-16T12:27:11.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.333+0000 7fe51b7fe700 1 --2- 172.21.15.16:0/2685476318 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7fe50806b070 0x7fe50806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:27:11.345 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:11.335+0000 7f10178a2200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T12:27:11.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.337+0000 7fe503fff700 1 -- 172.21.15.16:0/2685476318 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe5100546f0 con 0x7fe51c0d4060 2023-10-16T12:27:11.415 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:27:11.413+0000 7ff6f8ef0200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T12:27:11.460 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:11.459+0000 7f10178a2200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T12:27:11.491 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:27:11.489+0000 7ff6f8ef0200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T12:27:11.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.534+0000 7fe51b7fe700 1 -- 172.21.15.16:0/2685476318 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7fe50806b070 msgr2=0x7fe50806d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/292891612 2023-10-16T12:27:11.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.534+0000 7fe51b7fe700 1 --2- 172.21.15.16:0/2685476318 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7fe50806b070 0x7fe50806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:27:11.538 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:11.536+0000 7f10178a2200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T12:27:11.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.569+0000 7fe521bf6700 1 -- 172.21.15.16:0/2685476318 --> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7fe51c0b2d30 con 0x7fe50806b070 2023-10-16T12:27:11.575 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:27:11.573+0000 7ff6f8ef0200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T12:27:11.623 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:11.621+0000 7f10178a2200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T12:27:11.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.935+0000 7fe51b7fe700 1 -- 172.21.15.16:0/2685476318 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7fe50806b070 msgr2=0x7fe50806d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/292891612 2023-10-16T12:27:11.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:11.935+0000 7fe51b7fe700 1 --2- 172.21.15.16:0/2685476318 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7fe50806b070 0x7fe50806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-16T12:27:11.960 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:27:11.957+0000 7ff6f8ef0200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T12:27:12.013 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:12.012+0000 7f10178a2200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T12:27:12.566 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:27:12.564+0000 7ff6f8ef0200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T12:27:12.627 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:12.625+0000 7f10178a2200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T12:27:12.638 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:27:12.636+0000 7ff6f8ef0200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T12:27:12.700 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:12.698+0000 7f10178a2200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T12:27:12.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:12.736+0000 7fe51b7fe700 1 -- 172.21.15.16:0/2685476318 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7fe50806b070 msgr2=0x7fe50806d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/292891612 2023-10-16T12:27:12.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:12.736+0000 7fe51b7fe700 1 --2- 172.21.15.16:0/2685476318 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7fe50806b070 0x7fe50806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-16T12:27:13.297 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:27:13.295+0000 7ff6f8ef0200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:27:13.369 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:13.367+0000 7f10178a2200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:27:13.377 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:27:13.376+0000 7ff6f8ef0200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T12:27:13.451 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:13.449+0000 7f10178a2200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T12:27:13.528 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:27:13.525+0000 7ff6f8ef0200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:27:13.603 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:13.601+0000 7f10178a2200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:27:13.761 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:27:13.759+0000 7ff6f8ef0200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T12:27:13.840 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:27:13.837+0000 7ff6f8ef0200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T12:27:13.842 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:13.841+0000 7f10178a2200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T12:27:13.921 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:13.919+0000 7f10178a2200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T12:27:14.134 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:27:14.132+0000 7ff6f8ef0200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T12:27:14.198 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:14.196+0000 7f10178a2200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T12:27:14.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:14.337+0000 7fe51b7fe700 1 -- 172.21.15.16:0/2685476318 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7fe50806b070 msgr2=0x7fe50806d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/292891612 2023-10-16T12:27:14.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:14.337+0000 7fe51b7fe700 1 --2- 172.21.15.16:0/2685476318 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7fe50806b070 0x7fe50806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-10-16T12:27:14.365 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:27:14.363+0000 7ff6f8ef0200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T12:27:14.432 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:14.431+0000 7f10178a2200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T12:27:14.521 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:27:14.519+0000 7ff6f8ef0200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T12:27:14.591 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:14.589+0000 7f10178a2200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T12:27:14.639 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:27:14.637+0000 7ff6f8ef0200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T12:27:14.711 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:14.709+0000 7f10178a2200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T12:27:14.800 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:27:14.798+0000 7ff6f8ef0200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T12:27:14.876 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:14.874+0000 7f10178a2200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T12:27:15.078 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:27:15.077+0000 7ff6f8ef0200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T12:27:15.150 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:15.148+0000 7f10178a2200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T12:27:15.245 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:27:15.244+0000 7ff6f8ef0200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T12:27:15.321 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:15.319+0000 7f10178a2200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T12:27:15.797 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:27:15.795+0000 7ff6f8ef0200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T12:27:15.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:15.865+0000 7fe503fff700 1 -- 172.21.15.16:0/2685476318 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 43) v1 ==== 89348+0+0 (secure 0 0 0) 0x7fe510054ac0 con 0x7fe51c0d4060 2023-10-16T12:27:15.887 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:15.885+0000 7f10178a2200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T12:27:15.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:15.906+0000 7fe503fff700 1 -- 172.21.15.16:0/2685476318 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mgrmap(e 44) v1 ==== 88957+0+0 (secure 0 0 0) 0x7fe510087880 con 0x7fe51c0d4060 2023-10-16T12:27:15.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:15.906+0000 7fe503fff700 1 -- 172.21.15.16:0/2685476318 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7fe50806b070 msgr2=0x7fe50806d530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:27:15.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:15.906+0000 7fe503fff700 1 --2- 172.21.15.16:0/2685476318 >> [v2:172.21.15.16:6832/292891612,v1:172.21.15.16:6833/292891612] conn(0x7fe50806b070 0x7fe50806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:16.048 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:16.046+0000 7f0f8d082700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:27:16.049 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:16.046+0000 7f0f8d082700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:27:16.049 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:16.046+0000 7f0f8d082700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:27:16.049 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:16.046+0000 7f0f8d082700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:27:16.049 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:16.046+0000 7f0f8d082700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:27:16.050 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:16.047+0000 7f0f8a87d700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:27:16.050 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:16.047+0000 7f0f8a87d700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:27:16.050 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:16.047+0000 7f0f8a87d700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:27:16.050 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:16.047+0000 7f0f8a87d700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:27:16.051 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:16.047+0000 7f0f8a87d700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:27:16.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:16.909+0000 7fe503fff700 1 -- 172.21.15.16:0/2685476318 <== mon.0 v2:172.21.15.16:3300/0 9 ==== mgrmap(e 45) v1 ==== 89084+0+0 (secure 0 0 0) 0x7fe51001ed70 con 0x7fe51c0d4060 2023-10-16T12:27:16.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:16.910+0000 7fe503fff700 1 --2- 172.21.15.16:0/2685476318 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7fe50801d250 0x7fe5080754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:16.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:16.910+0000 7fe503fff700 1 -- 172.21.15.16:0/2685476318 --> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7fe50801d630 con 0x7fe50801d250 2023-10-16T12:27:16.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:16.912+0000 7fe51b7fe700 1 --2- 172.21.15.16:0/2685476318 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7fe50801d250 0x7fe5080754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:16.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:16.915+0000 7fe51b7fe700 1 --2- 172.21.15.16:0/2685476318 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7fe50801d250 0x7fe5080754a0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fe51c0cd810 tx=0x7fe50c0091c0 comp rx=0 tx=0).ready entity=mgr.7305 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:16.925 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:16.924+0000 7f0f948d1700 -1 mgr.server reply reply (2) No such file or directory SnapSchedule for snap_test_dir1 not found 2023-10-16T12:27:16.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:16.924+0000 7fe503fff700 1 -- 172.21.15.16:0/2685476318 <== mgr.7305 v2:172.21.15.16:6832/444334639 1 ==== mgr_command_reply(tid 0: -2 SnapSchedule for snap_test_dir1 not found) v1 ==== 49+0+0 (secure 0 0 0) 0x7fe50801d630 con 0x7fe50801d250 2023-10-16T12:27:16.927 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENOENT: SnapSchedule for snap_test_dir1 not found 2023-10-16T12:27:16.931 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-10-16T12:27:16.932 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-16T12:27:17.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:17.065+0000 7f6ff1d58700 1 Processor -- start 2023-10-16T12:27:17.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:17.065+0000 7f6ff1d58700 1 -- start start 2023-10-16T12:27:17.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:17.065+0000 7f6ff1d58700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6fec15b2c0 0x7f6fec161340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:17.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:17.066+0000 7f6ff1d58700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fec15ecb0 con 0x7f6fec06a7e0 2023-10-16T12:27:17.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:17.066+0000 7f6ff1d58700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fec161880 con 0x7f6fec06e570 2023-10-16T12:27:17.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:17.066+0000 7f6ff1d58700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fec15b6a0 con 0x7f6fec15b2c0 2023-10-16T12:27:17.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:17.066+0000 7f6feb7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6fec15b2c0 0x7f6fec161340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:17.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:17.066+0000 7f6febfff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f6fec06a7e0 0x7f6fec071a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:58884/0 (socket says 172.21.15.16:58884) 2023-10-16T12:27:17.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:17.066+0000 7f6febfff700 1 -- 172.21.15.16:0/2771165934 learned_addr learned my addr 172.21.15.16:0/2771165934 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:27:17.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:17.066+0000 7f6feb7fe700 1 -- 172.21.15.16:0/2771165934 >> v1:172.21.15.49:6789/0 conn(0x7f6fec06e570 legacy=0x7f6fec15e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:17.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:17.066+0000 7f6feb7fe700 1 -- 172.21.15.16:0/2771165934 >> v1:172.21.15.16:6789/0 conn(0x7f6fec06a7e0 legacy=0x7f6fec071a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:17.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:17.066+0000 7f6fea7fc700 1 -- 172.21.15.16:0/2771165934 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1611582038 0 0) 0x7f6fec15ecb0 con 0x7f6fec06a7e0 2023-10-16T12:27:17.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:17.066+0000 7f6feb7fe700 1 -- 172.21.15.16:0/2771165934 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fec161b20 con 0x7f6fec15b2c0 2023-10-16T12:27:17.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:17.066+0000 7f6feb7fe700 1 --2- 172.21.15.16:0/2771165934 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fec15b2c0 0x7f6fec161340 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc009840 tx=0x7f6fdc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=212fa16e77609922 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:17.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:17.066+0000 7f6fea7fc700 1 -- 172.21.15.16:0/2771165934 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fdc004230 con 0x7f6fec15b2c0 2023-10-16T12:27:17.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:17.066+0000 7f6fea7fc700 1 -- 172.21.15.16:0/2771165934 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fdc004390 con 0x7f6fec15b2c0 2023-10-16T12:27:17.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:17.066+0000 7f6fea7fc700 1 -- 172.21.15.16:0/2771165934 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fdc0036a0 con 0x7f6fec15b2c0 2023-10-16T12:27:17.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:17.067+0000 7f6ff1d58700 1 -- 172.21.15.16:0/2771165934 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fec15b2c0 msgr2=0x7f6fec161340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:17.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:17.067+0000 7f6ff1d58700 1 --2- 172.21.15.16:0/2771165934 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fec15b2c0 0x7f6fec161340 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc009840 tx=0x7f6fdc009b50 comp rx=0 tx=0).stop 2023-10-16T12:27:17.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:17.067+0000 7f6ff1d58700 1 -- 172.21.15.16:0/2771165934 shutdown_connections 2023-10-16T12:27:17.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:17.067+0000 7f6ff1d58700 1 --2- 172.21.15.16:0/2771165934 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fec15b2c0 0x7f6fec161340 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:17.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:17.067+0000 7f6ff1d58700 1 -- 172.21.15.16:0/2771165934 >> 172.21.15.16:0/2771165934 conn(0x7f6fec0b1f40 msgr2=0x7f6fec0b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:17.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:17.067+0000 7f6ff1d58700 1 -- 172.21.15.16:0/2771165934 shutdown_connections 2023-10-16T12:27:17.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:17.067+0000 7f6ff1d58700 1 -- 172.21.15.16:0/2771165934 wait complete. 2023-10-16T12:27:17.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:17.068+0000 7f6ff1d58700 1 Processor -- start 2023-10-16T12:27:17.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:17.068+0000 7f6ff1d58700 1 -- start start 2023-10-16T12:27:17.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:17.069+0000 7f6ff1d58700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6fec06a7e0 0x7f6fec1597f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:17.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:17.069+0000 7f6ff1d58700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6fec06e570 0x7f6fec0cd3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:17.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:17.069+0000 7f6ff1d58700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fec15b2c0 0x7f6fec0cb520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:17.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:17.069+0000 7f6ff1d58700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fec159d30 con 0x7f6fec15b2c0 2023-10-16T12:27:17.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:17.069+0000 7f6ff1d58700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fec15a1e0 con 0x7f6fec06a7e0 2023-10-16T12:27:17.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:17.069+0000 7f6ff1d58700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fec0b2f10 con 0x7f6fec06e570 2023-10-16T12:27:17.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:17.069+0000 7f6febfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fec15b2c0 0x7f6fec0cb520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:17.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:17.069+0000 7f6febfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fec15b2c0 0x7f6fec0cb520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41384/0 (socket says 172.21.15.16:41384) 2023-10-16T12:27:17.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:17.069+0000 7f6febfff700 1 -- 172.21.15.16:0/748841126 learned_addr learned my addr 172.21.15.16:0/748841126 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:17.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:17.069+0000 7f6feb7fe700 1 --2- 172.21.15.16:0/748841126 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6fec06a7e0 0x7f6fec1597f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:17.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:17.069+0000 7f6feaffd700 1 --2- 172.21.15.16:0/748841126 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6fec06e570 0x7f6fec0cd3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:17.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:17.070+0000 7f6febfff700 1 -- 172.21.15.16:0/748841126 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6fec06e570 msgr2=0x7f6fec0cd3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:17.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:17.070+0000 7f6febfff700 1 --2- 172.21.15.16:0/748841126 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6fec06e570 0x7f6fec0cd3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:17.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:17.070+0000 7f6febfff700 1 -- 172.21.15.16:0/748841126 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6fec06a7e0 msgr2=0x7f6fec1597f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:17.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:17.070+0000 7f6febfff700 1 --2- 172.21.15.16:0/748841126 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6fec06a7e0 0x7f6fec1597f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:17.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:17.070+0000 7f6febfff700 1 -- 172.21.15.16:0/748841126 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fdc0094a0 con 0x7f6fec15b2c0 2023-10-16T12:27:17.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:17.070+0000 7f6feaffd700 1 --2- 172.21.15.16:0/748841126 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6fec06e570 0x7f6fec0cd3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:27:17.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:17.070+0000 7f6febfff700 1 --2- 172.21.15.16:0/748841126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fec15b2c0 0x7f6fec0cb520 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f6fe400cb90 tx=0x7f6fe400cea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:17.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:17.070+0000 7f6fd3fff700 1 -- 172.21.15.16:0/748841126 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe400aa80 con 0x7f6fec15b2c0 2023-10-16T12:27:17.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:17.070+0000 7f6fd3fff700 1 -- 172.21.15.16:0/748841126 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fe400abe0 con 0x7f6fec15b2c0 2023-10-16T12:27:17.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:17.070+0000 7f6ff1d58700 1 -- 172.21.15.16:0/748841126 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fec0cd8f0 con 0x7f6fec15b2c0 2023-10-16T12:27:17.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:17.070+0000 7f6fd3fff700 1 -- 172.21.15.16:0/748841126 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe40070d0 con 0x7f6fec15b2c0 2023-10-16T12:27:17.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:17.070+0000 7f6ff1d58700 1 -- 172.21.15.16:0/748841126 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fec0b3140 con 0x7f6fec15b2c0 2023-10-16T12:27:17.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:17.071+0000 7f6ff1d58700 1 -- 172.21.15.16:0/748841126 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fec04ebb0 con 0x7f6fec15b2c0 2023-10-16T12:27:17.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:17.071+0000 7f6fd3fff700 1 -- 172.21.15.16:0/748841126 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 45) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f6fe4007230 con 0x7f6fec15b2c0 2023-10-16T12:27:17.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:17.072+0000 7f6fd3fff700 1 --2- 172.21.15.16:0/748841126 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7f6fd806ada0 0x7f6fd806d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:17.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:17.072+0000 7f6fd3fff700 1 -- 172.21.15.16:0/748841126 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f6fe4089090 con 0x7f6fec15b2c0 2023-10-16T12:27:17.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:17.072+0000 7f6feb7fe700 1 --2- 172.21.15.16:0/748841126 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7f6fd806ada0 0x7f6fd806d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:17.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:17.072+0000 7f6feb7fe700 1 --2- 172.21.15.16:0/748841126 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7f6fd806ada0 0x7f6fd806d260 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc0090b0 tx=0x7f6fdc01c040 comp rx=0 tx=0).ready entity=mgr.7305 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:17.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:17.077+0000 7f6fd3fff700 1 -- 172.21.15.16:0/748841126 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6fe40541c0 con 0x7f6fec15b2c0 2023-10-16T12:27:17.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:17.342+0000 7f6ff1d58700 1 -- 172.21.15.16:0/748841126 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f6fec0cd8f0 con 0x7f6fec15b2c0 2023-10-16T12:27:17.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:17.912+0000 7f6fd3fff700 1 -- 172.21.15.16:0/748841126 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v46) v1 ==== 93+0+0 (secure 0 0 0) 0x7f6fec0cd8f0 con 0x7f6fec15b2c0 2023-10-16T12:27:17.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:17.915+0000 7f6ff1d58700 1 -- 172.21.15.16:0/748841126 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7f6fd806ada0 msgr2=0x7f6fd806d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:17.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:17.915+0000 7f6ff1d58700 1 --2- 172.21.15.16:0/748841126 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7f6fd806ada0 0x7f6fd806d260 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc0090b0 tx=0x7f6fdc01c040 comp rx=0 tx=0).stop 2023-10-16T12:27:17.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:17.915+0000 7f6ff1d58700 1 -- 172.21.15.16:0/748841126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fec15b2c0 msgr2=0x7f6fec0cb520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:17.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:17.915+0000 7f6ff1d58700 1 --2- 172.21.15.16:0/748841126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fec15b2c0 0x7f6fec0cb520 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f6fe400cb90 tx=0x7f6fe400cea0 comp rx=0 tx=0).stop 2023-10-16T12:27:17.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:17.916+0000 7f6ff1d58700 1 -- 172.21.15.16:0/748841126 shutdown_connections 2023-10-16T12:27:17.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:17.916+0000 7f6ff1d58700 1 --2- 172.21.15.16:0/748841126 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7f6fd806ada0 0x7f6fd806d260 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:17.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:17.916+0000 7f6ff1d58700 1 --2- 172.21.15.16:0/748841126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fec15b2c0 0x7f6fec0cb520 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:17.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:17.916+0000 7f6ff1d58700 1 --2- 172.21.15.16:0/748841126 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6fec06a7e0 0x7f6fec1597f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:17.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:17.916+0000 7f6ff1d58700 1 --2- 172.21.15.16:0/748841126 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6fec06e570 0x7f6fec0cd3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:17.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:17.916+0000 7f6ff1d58700 1 -- 172.21.15.16:0/748841126 >> 172.21.15.16:0/748841126 conn(0x7f6fec0b1f40 msgr2=0x7f6fec0cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:17.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:17.916+0000 7f6ff1d58700 1 -- 172.21.15.16:0/748841126 shutdown_connections 2023-10-16T12:27:17.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:17.917+0000 7f6ff1d58700 1 -- 172.21.15.16:0/748841126 wait complete. 2023-10-16T12:27:17.936 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T12:27:17.937 DEBUG:teuthology.orchestra.run.smithi016:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T12:27:17.989 DEBUG:teuthology.orchestra.run.smithi049:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T12:27:18.033 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T12:27:18.034 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:27:18.034 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:27:18.053 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:27:18.053 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:27:18.066 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:27:18.065+0000 7f1533820200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T12:27:18.073 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:18.071+0000 7f1434b9d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T12:27:18.090 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:27:18.135 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:27:18.134+0000 7f1533820200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T12:27:18.143 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:18.141+0000 7f1434b9d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T12:27:18.210 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:27:18.209+0000 7f1533820200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T12:27:18.221 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:18.219+0000 7f1434b9d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T12:27:18.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.255+0000 7faa7e156700 1 Processor -- start 2023-10-16T12:27:18.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.255+0000 7faa7e156700 1 -- start start 2023-10-16T12:27:18.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.256+0000 7faa7e156700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7faa7815d8d0 0x7faa780d1c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:18.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.256+0000 7faa7e156700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faa780d41c0 0x7faa780d0040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:18.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.256+0000 7faa7e156700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa7806ba00 con 0x7faa780d0580 2023-10-16T12:27:18.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.256+0000 7faa7e156700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa780d0a10 con 0x7faa7815d8d0 2023-10-16T12:27:18.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.256+0000 7faa7e156700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa780d2150 con 0x7faa780d41c0 2023-10-16T12:27:18.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.256+0000 7faa777fe700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7faa7815d8d0 0x7faa780d1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:18.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.256+0000 7faa76ffd700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faa780d41c0 0x7faa780d0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:18.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.256+0000 7faa777fe700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7faa7815d8d0 0x7faa780d1c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:40520/0 (socket says 172.21.15.16:40520) 2023-10-16T12:27:18.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.256+0000 7faa777fe700 1 -- 172.21.15.16:0/3934094139 learned_addr learned my addr 172.21.15.16:0/3934094139 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:18.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.256+0000 7faa76ffd700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faa780d41c0 0x7faa780d0040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:57950/0 (socket says 172.21.15.16:57950) 2023-10-16T12:27:18.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.256+0000 7faa77fff700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7faa780d0580 0x7faa780ce240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:37918/0 (socket says 172.21.15.16:37918) 2023-10-16T12:27:18.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.256+0000 7faa777fe700 1 -- 172.21.15.16:0/3934094139 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faa780d41c0 msgr2=0x7faa780d0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:18.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.256+0000 7faa777fe700 1 --2- 172.21.15.16:0/3934094139 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faa780d41c0 0x7faa780d0040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:18.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.256+0000 7faa777fe700 1 -- 172.21.15.16:0/3934094139 >> v1:172.21.15.49:6789/0 conn(0x7faa780d0580 legacy=0x7faa780ce240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:18.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.256+0000 7faa777fe700 1 -- 172.21.15.16:0/3934094139 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa780d26d0 con 0x7faa7815d8d0 2023-10-16T12:27:18.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.256+0000 7faa767fc700 1 -- 172.21.15.16:0/3934094139 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4187437 0 0) 0x7faa7806ba00 con 0x7faa780d0580 2023-10-16T12:27:18.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.256+0000 7faa76ffd700 1 --2- 172.21.15.16:0/3934094139 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faa780d41c0 0x7faa780d0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:27:18.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.257+0000 7faa777fe700 1 --2- 172.21.15.16:0/3934094139 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faa7815d8d0 0x7faa780d1c10 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7faa680098f0 tx=0x7faa68009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fe07233b151b7868 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:18.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.257+0000 7faa767fc700 1 -- 172.21.15.16:0/3934094139 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa68011830 con 0x7faa7815d8d0 2023-10-16T12:27:18.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.257+0000 7faa767fc700 1 -- 172.21.15.16:0/3934094139 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa68011990 con 0x7faa7815d8d0 2023-10-16T12:27:18.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.257+0000 7faa767fc700 1 -- 172.21.15.16:0/3934094139 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa68011d20 con 0x7faa7815d8d0 2023-10-16T12:27:18.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.257+0000 7faa7e156700 1 -- 172.21.15.16:0/3934094139 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faa7815d8d0 msgr2=0x7faa780d1c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:18.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.257+0000 7faa7e156700 1 --2- 172.21.15.16:0/3934094139 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faa7815d8d0 0x7faa780d1c10 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7faa680098f0 tx=0x7faa68009c00 comp rx=0 tx=0).stop 2023-10-16T12:27:18.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.257+0000 7faa7e156700 1 -- 172.21.15.16:0/3934094139 shutdown_connections 2023-10-16T12:27:18.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.257+0000 7faa7e156700 1 --2- 172.21.15.16:0/3934094139 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faa7815d8d0 0x7faa780d1c10 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:18.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.257+0000 7faa7e156700 1 --2- 172.21.15.16:0/3934094139 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faa780d41c0 0x7faa780d0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:18.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.257+0000 7faa7e156700 1 -- 172.21.15.16:0/3934094139 >> 172.21.15.16:0/3934094139 conn(0x7faa780d3630 msgr2=0x7faa780d3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:18.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.257+0000 7faa7e156700 1 -- 172.21.15.16:0/3934094139 shutdown_connections 2023-10-16T12:27:18.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.257+0000 7faa7e156700 1 -- 172.21.15.16:0/3934094139 wait complete. 2023-10-16T12:27:18.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.259+0000 7faa7e156700 1 Processor -- start 2023-10-16T12:27:18.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.259+0000 7faa7e156700 1 -- start start 2023-10-16T12:27:18.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.259+0000 7faa7e156700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa780d0580 0x7faa7814bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:18.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.260+0000 7faa7e156700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faa780d41c0 0x7faa781e5120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:18.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.260+0000 7faa7e156700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faa7815d8d0 0x7faa780b1d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:18.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.260+0000 7faa7e156700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa780cb760 con 0x7faa780d0580 2023-10-16T12:27:18.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.260+0000 7faa7e156700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa7814c090 con 0x7faa7815d8d0 2023-10-16T12:27:18.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.260+0000 7faa7e156700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa780cab50 con 0x7faa780d41c0 2023-10-16T12:27:18.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.260+0000 7faa777fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa780d0580 0x7faa7814bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:18.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.260+0000 7faa777fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa780d0580 0x7faa7814bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41464/0 (socket says 172.21.15.16:41464) 2023-10-16T12:27:18.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.260+0000 7faa777fe700 1 -- 172.21.15.16:0/1860176810 learned_addr learned my addr 172.21.15.16:0/1860176810 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:18.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.260+0000 7faa76ffd700 1 --2- 172.21.15.16:0/1860176810 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faa780d41c0 0x7faa781e5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:18.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.260+0000 7faa77fff700 1 --2- 172.21.15.16:0/1860176810 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faa7815d8d0 0x7faa780b1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:18.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.260+0000 7faa777fe700 1 -- 172.21.15.16:0/1860176810 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faa780d41c0 msgr2=0x7faa781e5120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:18.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.260+0000 7faa777fe700 1 --2- 172.21.15.16:0/1860176810 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faa780d41c0 0x7faa781e5120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:18.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.260+0000 7faa777fe700 1 -- 172.21.15.16:0/1860176810 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faa7815d8d0 msgr2=0x7faa780b1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:18.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.260+0000 7faa777fe700 1 --2- 172.21.15.16:0/1860176810 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faa7815d8d0 0x7faa780b1d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:18.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.260+0000 7faa777fe700 1 -- 172.21.15.16:0/1860176810 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa6c0094a0 con 0x7faa780d0580 2023-10-16T12:27:18.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.260+0000 7faa777fe700 1 --2- 172.21.15.16:0/1860176810 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa780d0580 0x7faa7814bb50 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7faa68009c40 tx=0x7faa68004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:18.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.260+0000 7faa5ffff700 1 -- 172.21.15.16:0/1860176810 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa68004450 con 0x7faa780d0580 2023-10-16T12:27:18.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.260+0000 7faa5ffff700 1 -- 172.21.15.16:0/1860176810 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa6801e070 con 0x7faa780d0580 2023-10-16T12:27:18.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.260+0000 7faa5ffff700 1 -- 172.21.15.16:0/1860176810 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa680038e0 con 0x7faa780d0580 2023-10-16T12:27:18.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.260+0000 7faa7e156700 1 -- 172.21.15.16:0/1860176810 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa68009550 con 0x7faa780d0580 2023-10-16T12:27:18.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.261+0000 7faa7e156700 1 -- 172.21.15.16:0/1860176810 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa780cb110 con 0x7faa780d0580 2023-10-16T12:27:18.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.262+0000 7faa7e156700 1 -- 172.21.15.16:0/1860176810 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa7804ebb0 con 0x7faa780d0580 2023-10-16T12:27:18.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.262+0000 7faa5ffff700 1 -- 172.21.15.16:0/1860176810 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 46) v1 ==== 89067+0+0 (secure 0 0 0) 0x7faa68003a40 con 0x7faa780d0580 2023-10-16T12:27:18.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.263+0000 7faa5ffff700 1 --2- 172.21.15.16:0/1860176810 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7faa6406ad50 0x7faa6406d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:18.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.263+0000 7faa76ffd700 1 -- 172.21.15.16:0/1860176810 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7faa6406ad50 msgr2=0x7faa6406d210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/444334639 2023-10-16T12:27:18.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.263+0000 7faa5ffff700 1 -- 172.21.15.16:0/1860176810 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6401+0+0 (secure 0 0 0) 0x7faa6808a3b0 con 0x7faa780d0580 2023-10-16T12:27:18.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.263+0000 7faa76ffd700 1 --2- 172.21.15.16:0/1860176810 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7faa6406ad50 0x7faa6406d210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:27:18.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.268+0000 7faa5ffff700 1 -- 172.21.15.16:0/1860176810 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7faa68055460 con 0x7faa780d0580 2023-10-16T12:27:18.291 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:27:18.290+0000 7f1533820200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T12:27:18.307 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:18.306+0000 7f1434b9d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T12:27:18.439 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:27:18.438+0000 7f1533820200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T12:27:18.457 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:18.455+0000 7f1434b9d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T12:27:18.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.464+0000 7faa76ffd700 1 -- 172.21.15.16:0/1860176810 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7faa6406ad50 msgr2=0x7faa6406d210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/444334639 2023-10-16T12:27:18.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.464+0000 7faa76ffd700 1 --2- 172.21.15.16:0/1860176810 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7faa6406ad50 0x7faa6406d210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:27:18.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.474+0000 7faa7e156700 1 -- 172.21.15.16:0/1860176810 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faa780a1360 con 0x7faa780d0580 2023-10-16T12:27:18.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.475+0000 7faa5ffff700 1 -- 172.21.15.16:0/1860176810 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) v1 ==== 74+0+17211 (secure 0 0 0) 0x7faa680045b0 con 0x7faa780d0580 2023-10-16T12:27:18.478 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:27:18.479 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":100,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:27:15.899863+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:26:59.206630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2023-10-16T12:26:59.784585+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6833/292891612":"2023-10-17T12:27:15.899818+0000","172.21.15.16:6832/292891612":"2023-10-17T12:27:15.899818+0000","172.21.15.16:0/1020789692":"2023-10-17T12:27:15.899818+0000","172.21.15.16:0/478878390":"2023-10-17T12:27:15.899818+0000","172.21.15.16:0/2293822331":"2023-10-17T12:27:15.899818+0000","172.21.15.16:0/1305507650":"2023-10-17T12:27:15.899818+0000","172.21.15.16:0/3205233425":"2023-10-17T12:27:15.899818+0000","172.21.15.16:0/2477042442":"2023-10-17T12:27:15.899818+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:27:18.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.477+0000 7faa7e156700 1 -- 172.21.15.16:0/1860176810 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7faa6406ad50 msgr2=0x7faa6406d210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:27:18.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.477+0000 7faa7e156700 1 --2- 172.21.15.16:0/1860176810 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7faa6406ad50 0x7faa6406d210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:18.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.477+0000 7faa7e156700 1 -- 172.21.15.16:0/1860176810 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa780d0580 msgr2=0x7faa7814bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:18.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.477+0000 7faa7e156700 1 --2- 172.21.15.16:0/1860176810 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa780d0580 0x7faa7814bb50 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7faa68009c40 tx=0x7faa68004750 comp rx=0 tx=0).stop 2023-10-16T12:27:18.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.477+0000 7faa7e156700 1 -- 172.21.15.16:0/1860176810 shutdown_connections 2023-10-16T12:27:18.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.478+0000 7faa7e156700 1 --2- 172.21.15.16:0/1860176810 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faa7815d8d0 0x7faa780b1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:18.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.478+0000 7faa7e156700 1 --2- 172.21.15.16:0/1860176810 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7faa6406ad50 0x7faa6406d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:18.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.478+0000 7faa7e156700 1 --2- 172.21.15.16:0/1860176810 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa780d0580 0x7faa7814bb50 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:18.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.478+0000 7faa7e156700 1 --2- 172.21.15.16:0/1860176810 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faa780d41c0 0x7faa781e5120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:18.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.478+0000 7faa7e156700 1 -- 172.21.15.16:0/1860176810 >> 172.21.15.16:0/1860176810 conn(0x7faa780d3630 msgr2=0x7faa7815cae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:18.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.478+0000 7faa7e156700 1 -- 172.21.15.16:0/1860176810 shutdown_connections 2023-10-16T12:27:18.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.478+0000 7faa7e156700 1 -- 172.21.15.16:0/1860176810 wait complete. 2023-10-16T12:27:18.493 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T12:26:59.206630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '98', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T12:26:59.784585+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '99', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T12:27:18.494 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:27:18.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.660+0000 7fbda8797700 1 Processor -- start 2023-10-16T12:27:18.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.660+0000 7fbda8797700 1 -- start start 2023-10-16T12:27:18.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.661+0000 7fbda8797700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbda0158050 0x7fbda00cdf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:18.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.661+0000 7fbda8797700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbda00ce470 0x7fbda006da30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:18.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.661+0000 7fbda8797700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbda006eab0 con 0x7fbda006df70 2023-10-16T12:27:18.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.661+0000 7fbda8797700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbda006e4f0 con 0x7fbda0158050 2023-10-16T12:27:18.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.661+0000 7fbda8797700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbda006e7d0 con 0x7fbda00ce470 2023-10-16T12:27:18.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.661+0000 7fbda6533700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbda0158050 0x7fbda00cdf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:18.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.661+0000 7fbda6d34700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fbda006df70 0x7fbda015e1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59028/0 (socket says 172.21.15.16:59028) 2023-10-16T12:27:18.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.661+0000 7fbda6d34700 1 -- 172.21.15.16:0/2091849138 learned_addr learned my addr 172.21.15.16:0/2091849138 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:27:18.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.661+0000 7fbda5d32700 1 --2- 172.21.15.16:0/2091849138 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbda00ce470 0x7fbda006da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:18.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.662+0000 7fbda6533700 1 -- 172.21.15.16:0/2091849138 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbda00ce470 msgr2=0x7fbda006da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:18.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.662+0000 7fbda6533700 1 --2- 172.21.15.16:0/2091849138 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbda00ce470 0x7fbda006da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:18.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.662+0000 7fbda6533700 1 -- 172.21.15.16:0/2091849138 >> v1:172.21.15.16:6789/0 conn(0x7fbda006df70 legacy=0x7fbda015e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:18.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.662+0000 7fbda6533700 1 -- 172.21.15.16:0/2091849138 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbda00cb9d0 con 0x7fbda0158050 2023-10-16T12:27:18.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.662+0000 7fbda5531700 1 -- 172.21.15.16:0/2091849138 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2634300688 0 0) 0x7fbda006eab0 con 0x7fbda006df70 2023-10-16T12:27:18.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.662+0000 7fbda6533700 1 --2- 172.21.15.16:0/2091849138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbda0158050 0x7fbda00cdf30 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7fbd9c009820 tx=0x7fbd9c009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b91e93ec28a1e26 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:18.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.662+0000 7fbda5531700 1 -- 172.21.15.16:0/2091849138 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd9c005e30 con 0x7fbda0158050 2023-10-16T12:27:18.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.662+0000 7fbda5531700 1 -- 172.21.15.16:0/2091849138 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd9c006030 con 0x7fbda0158050 2023-10-16T12:27:18.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.662+0000 7fbda5531700 1 -- 172.21.15.16:0/2091849138 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd9c0063c0 con 0x7fbda0158050 2023-10-16T12:27:18.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.663+0000 7fbda8797700 1 -- 172.21.15.16:0/2091849138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbda0158050 msgr2=0x7fbda00cdf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:18.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.663+0000 7fbda8797700 1 --2- 172.21.15.16:0/2091849138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbda0158050 0x7fbda00cdf30 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7fbd9c009820 tx=0x7fbd9c009b30 comp rx=0 tx=0).stop 2023-10-16T12:27:18.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.663+0000 7fbda8797700 1 -- 172.21.15.16:0/2091849138 shutdown_connections 2023-10-16T12:27:18.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.663+0000 7fbda8797700 1 --2- 172.21.15.16:0/2091849138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbda0158050 0x7fbda00cdf30 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:18.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.663+0000 7fbda8797700 1 --2- 172.21.15.16:0/2091849138 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbda00ce470 0x7fbda006da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:18.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.663+0000 7fbda8797700 1 -- 172.21.15.16:0/2091849138 >> 172.21.15.16:0/2091849138 conn(0x7fbda00b1ed0 msgr2=0x7fbda00b22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:18.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.663+0000 7fbda8797700 1 -- 172.21.15.16:0/2091849138 shutdown_connections 2023-10-16T12:27:18.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.663+0000 7fbda8797700 1 -- 172.21.15.16:0/2091849138 wait complete. 2023-10-16T12:27:18.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.664+0000 7fbda8797700 1 Processor -- start 2023-10-16T12:27:18.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.664+0000 7fbda8797700 1 -- start start 2023-10-16T12:27:18.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.665+0000 7fbda8797700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbda006df70 0x7fbda00c98e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:18.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.665+0000 7fbda8797700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbda00ce470 0x7fbda01f69c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:18.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.665+0000 7fbda8797700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbda0158050 0x7fbda00b0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:18.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.665+0000 7fbda8797700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbda00c8f50 con 0x7fbda006df70 2023-10-16T12:27:18.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.665+0000 7fbda8797700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbda015bce0 con 0x7fbda0158050 2023-10-16T12:27:18.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.665+0000 7fbda8797700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbda00c91a0 con 0x7fbda00ce470 2023-10-16T12:27:18.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.666+0000 7fbda6533700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbda006df70 0x7fbda00c98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:18.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.666+0000 7fbda6533700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbda006df70 0x7fbda00c98e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41524/0 (socket says 172.21.15.16:41524) 2023-10-16T12:27:18.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.666+0000 7fbda6533700 1 -- 172.21.15.16:0/1363816529 learned_addr learned my addr 172.21.15.16:0/1363816529 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:18.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.666+0000 7fbda6d34700 1 --2- 172.21.15.16:0/1363816529 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbda0158050 0x7fbda00b0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:18.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.666+0000 7fbda5d32700 1 --2- 172.21.15.16:0/1363816529 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbda00ce470 0x7fbda01f69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:18.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.666+0000 7fbda6533700 1 -- 172.21.15.16:0/1363816529 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbda00ce470 msgr2=0x7fbda01f69c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:18.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.666+0000 7fbda6533700 1 --2- 172.21.15.16:0/1363816529 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbda00ce470 0x7fbda01f69c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:18.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.666+0000 7fbda6533700 1 -- 172.21.15.16:0/1363816529 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbda0158050 msgr2=0x7fbda00b0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:18.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.666+0000 7fbda6533700 1 --2- 172.21.15.16:0/1363816529 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbda0158050 0x7fbda00b0c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:18.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.666+0000 7fbda6533700 1 -- 172.21.15.16:0/1363816529 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd9c009430 con 0x7fbda006df70 2023-10-16T12:27:18.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.666+0000 7fbda6d34700 1 --2- 172.21.15.16:0/1363816529 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbda0158050 0x7fbda00b0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:27:18.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.666+0000 7fbda6533700 1 --2- 172.21.15.16:0/1363816529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbda006df70 0x7fbda00c98e0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7fbd9c000c00 tx=0x7fbd9c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:18.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.666+0000 7fbd96ffd700 1 -- 172.21.15.16:0/1363816529 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd9c0038a0 con 0x7fbda006df70 2023-10-16T12:27:18.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.666+0000 7fbd96ffd700 1 -- 172.21.15.16:0/1363816529 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd9c003a00 con 0x7fbda006df70 2023-10-16T12:27:18.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.666+0000 7fbd96ffd700 1 -- 172.21.15.16:0/1363816529 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd9c003db0 con 0x7fbda006df70 2023-10-16T12:27:18.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.666+0000 7fbda8797700 1 -- 172.21.15.16:0/1363816529 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbda00c9480 con 0x7fbda006df70 2023-10-16T12:27:18.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.666+0000 7fbda8797700 1 -- 172.21.15.16:0/1363816529 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbda00b11c0 con 0x7fbda006df70 2023-10-16T12:27:18.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.668+0000 7fbda8797700 1 -- 172.21.15.16:0/1363816529 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbda00cbc90 con 0x7fbda006df70 2023-10-16T12:27:18.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.668+0000 7fbd96ffd700 1 -- 172.21.15.16:0/1363816529 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 46) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fbd9c022070 con 0x7fbda006df70 2023-10-16T12:27:18.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.668+0000 7fbd96ffd700 1 --2- 172.21.15.16:0/1363816529 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7fbd8406ad50 0x7fbd8406d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:18.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.668+0000 7fbda5d32700 1 -- 172.21.15.16:0/1363816529 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7fbd8406ad50 msgr2=0x7fbd8406d210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/444334639 2023-10-16T12:27:18.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.668+0000 7fbd96ffd700 1 -- 172.21.15.16:0/1363816529 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fbd9c08ad00 con 0x7fbda006df70 2023-10-16T12:27:18.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.668+0000 7fbda5d32700 1 --2- 172.21.15.16:0/1363816529 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7fbd8406ad50 0x7fbd8406d210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:27:18.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.673+0000 7fbd96ffd700 1 -- 172.21.15.16:0/1363816529 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbda00cbc90 con 0x7fbda006df70 2023-10-16T12:27:18.778 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:27:18.777+0000 7f1533820200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T12:27:18.815 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:18.814+0000 7f1434b9d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T12:27:18.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.869+0000 7fbda5d32700 1 -- 172.21.15.16:0/1363816529 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7fbd8406ad50 msgr2=0x7fbd8406d210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/444334639 2023-10-16T12:27:18.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.869+0000 7fbda5d32700 1 --2- 172.21.15.16:0/1363816529 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7fbd8406ad50 0x7fbd8406d210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:27:18.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.888+0000 7fbda8797700 1 -- 172.21.15.16:0/1363816529 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbda00cbc90 con 0x7fbda006df70 2023-10-16T12:27:18.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.889+0000 7fbd96ffd700 1 -- 172.21.15.16:0/1363816529 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v100) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbda00cbc90 con 0x7fbda006df70 2023-10-16T12:27:18.892 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T12:27:18.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.891+0000 7fbda8797700 1 -- 172.21.15.16:0/1363816529 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7fbd8406ad50 msgr2=0x7fbd8406d210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:27:18.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.891+0000 7fbda8797700 1 --2- 172.21.15.16:0/1363816529 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7fbd8406ad50 0x7fbd8406d210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:18.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.891+0000 7fbda8797700 1 -- 172.21.15.16:0/1363816529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbda006df70 msgr2=0x7fbda00c98e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:18.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.891+0000 7fbda8797700 1 --2- 172.21.15.16:0/1363816529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbda006df70 0x7fbda00c98e0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7fbd9c000c00 tx=0x7fbd9c004750 comp rx=0 tx=0).stop 2023-10-16T12:27:18.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.891+0000 7fbda8797700 1 -- 172.21.15.16:0/1363816529 shutdown_connections 2023-10-16T12:27:18.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.891+0000 7fbda8797700 1 --2- 172.21.15.16:0/1363816529 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbda0158050 0x7fbda00b0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:18.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.891+0000 7fbda8797700 1 --2- 172.21.15.16:0/1363816529 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7fbd8406ad50 0x7fbd8406d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:18.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.891+0000 7fbda8797700 1 --2- 172.21.15.16:0/1363816529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbda006df70 0x7fbda00c98e0 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:18.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.891+0000 7fbda8797700 1 --2- 172.21.15.16:0/1363816529 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbda00ce470 0x7fbda01f69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:18.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.891+0000 7fbda8797700 1 -- 172.21.15.16:0/1363816529 >> 172.21.15.16:0/1363816529 conn(0x7fbda00b1ed0 msgr2=0x7fbda00cce90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:18.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.891+0000 7fbda8797700 1 -- 172.21.15.16:0/1363816529 shutdown_connections 2023-10-16T12:27:18.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:18.891+0000 7fbda8797700 1 -- 172.21.15.16:0/1363816529 wait complete. 2023-10-16T12:27:18.898 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:27:18.897+0000 7f1533820200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T12:27:18.907 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T12:27:18.942 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:18.940+0000 7f1434b9d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T12:27:18.977 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:27:18.975+0000 7f1533820200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T12:27:19.020 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:19.019+0000 7f1434b9d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T12:27:19.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.038+0000 7f210efa8700 1 Processor -- start 2023-10-16T12:27:19.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.038+0000 7f210efa8700 1 -- start start 2023-10-16T12:27:19.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.039+0000 7f210efa8700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f21080d4060 0x7f21080d1520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:19.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.039+0000 7f210efa8700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21080d4540 con 0x7f210815da10 2023-10-16T12:27:19.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.039+0000 7f210efa8700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21080ce7e0 con 0x7f21080d1d20 2023-10-16T12:27:19.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.039+0000 7f210efa8700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21080cf9b0 con 0x7f21080d4060 2023-10-16T12:27:19.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.039+0000 7f210cd44700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f210815da10 0x7f21080cf2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59040/0 (socket says 172.21.15.16:59040) 2023-10-16T12:27:19.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.039+0000 7f210cd44700 1 -- 172.21.15.16:0/1212184089 learned_addr learned my addr 172.21.15.16:0/1212184089 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:27:19.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.040+0000 7f2107fff700 1 --2- 172.21.15.16:0/1212184089 >> v2:172.21.15.49:3300/0 conn(0x7f21080d4060 0x7f21080d1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:19.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.040+0000 7f21077fe700 1 -- 172.21.15.16:0/1212184089 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 74409371 0 0) 0x7f21080d4540 con 0x7f210815da10 2023-10-16T12:27:19.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.040+0000 7f21077fe700 1 -- 172.21.15.16:0/1212184089 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20f80035b0 con 0x7f210815da10 2023-10-16T12:27:19.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.040+0000 7f21077fe700 1 -- 172.21.15.16:0/1212184089 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1624526671 0 0) 0x7f21080ce7e0 con 0x7f21080d1d20 2023-10-16T12:27:19.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.040+0000 7f21077fe700 1 -- 172.21.15.16:0/1212184089 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21080d4540 con 0x7f21080d1d20 2023-10-16T12:27:19.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.040+0000 7f2107fff700 1 -- 172.21.15.16:0/1212184089 >> v1:172.21.15.49:6789/0 conn(0x7f21080d1d20 legacy=0x7f210806b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:19.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.040+0000 7f2107fff700 1 -- 172.21.15.16:0/1212184089 >> v1:172.21.15.16:6789/0 conn(0x7f210815da10 legacy=0x7f21080cf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:19.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.040+0000 7f2107fff700 1 -- 172.21.15.16:0/1212184089 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21080ceac0 con 0x7f21080d4060 2023-10-16T12:27:19.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.040+0000 7f2107fff700 1 --2- 172.21.15.16:0/1212184089 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f21080d4060 0x7f21080d1520 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f20f4009840 tx=0x7f20f4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e33b96fb2d7fd0c6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:19.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.040+0000 7f21077fe700 1 -- 172.21.15.16:0/1212184089 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20f4011830 con 0x7f21080d4060 2023-10-16T12:27:19.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.040+0000 7f21077fe700 1 -- 172.21.15.16:0/1212184089 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20f4011990 con 0x7f21080d4060 2023-10-16T12:27:19.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.040+0000 7f21077fe700 1 -- 172.21.15.16:0/1212184089 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20f4011d20 con 0x7f21080d4060 2023-10-16T12:27:19.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.041+0000 7f210efa8700 1 -- 172.21.15.16:0/1212184089 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f21080d4060 msgr2=0x7f21080d1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:19.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.041+0000 7f210efa8700 1 --2- 172.21.15.16:0/1212184089 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f21080d4060 0x7f21080d1520 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f20f4009840 tx=0x7f20f4009b50 comp rx=0 tx=0).stop 2023-10-16T12:27:19.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.041+0000 7f210efa8700 1 -- 172.21.15.16:0/1212184089 shutdown_connections 2023-10-16T12:27:19.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.041+0000 7f210efa8700 1 --2- 172.21.15.16:0/1212184089 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f21080d4060 0x7f21080d1520 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:19.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.041+0000 7f210efa8700 1 -- 172.21.15.16:0/1212184089 >> 172.21.15.16:0/1212184089 conn(0x7f210809c9f0 msgr2=0x7f21080d3740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:19.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.041+0000 7f210efa8700 1 -- 172.21.15.16:0/1212184089 shutdown_connections 2023-10-16T12:27:19.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.041+0000 7f210efa8700 1 -- 172.21.15.16:0/1212184089 wait complete. 2023-10-16T12:27:19.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.042+0000 7f210efa8700 1 Processor -- start 2023-10-16T12:27:19.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.042+0000 7f210efa8700 1 -- start start 2023-10-16T12:27:19.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.043+0000 7f210efa8700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f21080d1d20 0x7f210806ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:19.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.043+0000 7f210efa8700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f21080d4060 0x7f21080ca810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:19.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.043+0000 7f210efa8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f210815da10 0x7f2108157860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:19.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.043+0000 7f210efa8700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21080cb180 con 0x7f210815da10 2023-10-16T12:27:19.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.043+0000 7f210efa8700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21080cb460 con 0x7f21080d1d20 2023-10-16T12:27:19.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.043+0000 7f210efa8700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21080cb740 con 0x7f21080d4060 2023-10-16T12:27:19.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.043+0000 7f210d545700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f210815da10 0x7f2108157860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:19.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.043+0000 7f210d545700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f210815da10 0x7f2108157860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41544/0 (socket says 172.21.15.16:41544) 2023-10-16T12:27:19.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.043+0000 7f210d545700 1 -- 172.21.15.16:0/1277550693 learned_addr learned my addr 172.21.15.16:0/1277550693 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:19.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.043+0000 7f210cd44700 1 --2- 172.21.15.16:0/1277550693 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f21080d1d20 0x7f210806ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:19.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.043+0000 7f2107fff700 1 --2- 172.21.15.16:0/1277550693 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f21080d4060 0x7f21080ca810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:19.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.044+0000 7f210cd44700 1 -- 172.21.15.16:0/1277550693 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f21080d4060 msgr2=0x7f21080ca810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:19.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.044+0000 7f210cd44700 1 --2- 172.21.15.16:0/1277550693 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f21080d4060 0x7f21080ca810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:19.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.044+0000 7f210cd44700 1 -- 172.21.15.16:0/1277550693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f210815da10 msgr2=0x7f2108157860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:19.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.044+0000 7f210cd44700 1 --2- 172.21.15.16:0/1277550693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f210815da10 0x7f2108157860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:19.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.044+0000 7f210cd44700 1 -- 172.21.15.16:0/1277550693 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20f40094a0 con 0x7f21080d1d20 2023-10-16T12:27:19.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.044+0000 7f210d545700 1 --2- 172.21.15.16:0/1277550693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f210815da10 0x7f2108157860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:27:19.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.044+0000 7f210cd44700 1 --2- 172.21.15.16:0/1277550693 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f21080d1d20 0x7f210806ba80 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f20fc001360 tx=0x7f20fc00c7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:19.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.044+0000 7f21057fa700 1 -- 172.21.15.16:0/1277550693 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20fc0074e0 con 0x7f21080d1d20 2023-10-16T12:27:19.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.044+0000 7f21057fa700 1 -- 172.21.15.16:0/1277550693 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20fc007640 con 0x7f21080d1d20 2023-10-16T12:27:19.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.044+0000 7f21057fa700 1 -- 172.21.15.16:0/1277550693 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20fc005a70 con 0x7f21080d1d20 2023-10-16T12:27:19.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.044+0000 7f210efa8700 1 -- 172.21.15.16:0/1277550693 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21080cba20 con 0x7f21080d1d20 2023-10-16T12:27:19.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.044+0000 7f210efa8700 1 -- 172.21.15.16:0/1277550693 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f210806bfc0 con 0x7f21080d1d20 2023-10-16T12:27:19.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.045+0000 7f210efa8700 1 -- 172.21.15.16:0/1277550693 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f210804ebb0 con 0x7f21080d1d20 2023-10-16T12:27:19.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.046+0000 7f21057fa700 1 -- 172.21.15.16:0/1277550693 <== mon.1 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 47) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f20fc005bd0 con 0x7f21080d1d20 2023-10-16T12:27:19.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.046+0000 7f21057fa700 1 --2- 172.21.15.16:0/1277550693 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7f20ec06b020 0x7f20ec06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:19.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.046+0000 7f2107fff700 1 -- 172.21.15.16:0/1277550693 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7f20ec06b020 msgr2=0x7f20ec06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/444334639 2023-10-16T12:27:19.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.046+0000 7f21057fa700 1 -- 172.21.15.16:0/1277550693 <== mon.1 v2:172.21.15.49:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f20fc088ce0 con 0x7f21080d1d20 2023-10-16T12:27:19.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.046+0000 7f2107fff700 1 --2- 172.21.15.16:0/1277550693 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7f20ec06b020 0x7f20ec06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:27:19.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.051+0000 7f21057fa700 1 -- 172.21.15.16:0/1277550693 <== mon.1 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f20fc053a50 con 0x7f21080d1d20 2023-10-16T12:27:19.063 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:27:19.060+0000 7f1533820200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T12:27:19.108 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:19.106+0000 7f1434b9d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T12:27:19.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.247+0000 7f2107fff700 1 -- 172.21.15.16:0/1277550693 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7f20ec06b020 msgr2=0x7f20ec06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/444334639 2023-10-16T12:27:19.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.247+0000 7f2107fff700 1 --2- 172.21.15.16:0/1277550693 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7f20ec06b020 0x7f20ec06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:27:19.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.256+0000 7f210efa8700 1 -- 172.21.15.16:0/1277550693 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f21080d1b00 con 0x7f21080d1d20 2023-10-16T12:27:19.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.256+0000 7f21057fa700 1 -- 172.21.15.16:0/1277550693 <== mon.1 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v100) v1 ==== 104+0+11 (secure 0 0 0) 0x7f20fc0170a0 con 0x7f21080d1d20 2023-10-16T12:27:19.258 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-10-16T12:27:19.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.258+0000 7f210efa8700 1 -- 172.21.15.16:0/1277550693 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7f20ec06b020 msgr2=0x7f20ec06d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:27:19.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.258+0000 7f210efa8700 1 --2- 172.21.15.16:0/1277550693 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7f20ec06b020 0x7f20ec06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:19.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.258+0000 7f210efa8700 1 -- 172.21.15.16:0/1277550693 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f21080d1d20 msgr2=0x7f210806ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:19.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.258+0000 7f210efa8700 1 --2- 172.21.15.16:0/1277550693 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f21080d1d20 0x7f210806ba80 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f20fc001360 tx=0x7f20fc00c7b0 comp rx=0 tx=0).stop 2023-10-16T12:27:19.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.258+0000 7f210efa8700 1 -- 172.21.15.16:0/1277550693 shutdown_connections 2023-10-16T12:27:19.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.258+0000 7f210efa8700 1 --2- 172.21.15.16:0/1277550693 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7f20ec06b020 0x7f20ec06d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:19.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.258+0000 7f210efa8700 1 --2- 172.21.15.16:0/1277550693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f210815da10 0x7f2108157860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:19.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.258+0000 7f210efa8700 1 --2- 172.21.15.16:0/1277550693 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f21080d1d20 0x7f210806ba80 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:19.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.258+0000 7f210efa8700 1 --2- 172.21.15.16:0/1277550693 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f21080d4060 0x7f21080ca810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:19.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.258+0000 7f210efa8700 1 -- 172.21.15.16:0/1277550693 >> 172.21.15.16:0/1277550693 conn(0x7f210809c9f0 msgr2=0x7f21080cf1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:19.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.259+0000 7f210efa8700 1 -- 172.21.15.16:0/1277550693 shutdown_connections 2023-10-16T12:27:19.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.259+0000 7f210efa8700 1 -- 172.21.15.16:0/1277550693 wait complete. 2023-10-16T12:27:19.273 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T12:27:19.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.445+0000 7f6521967700 1 Processor -- start 2023-10-16T12:27:19.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.445+0000 7f6521967700 1 -- start start 2023-10-16T12:27:19.448 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:27:19.446+0000 7f1533820200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T12:27:19.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.445+0000 7f6521967700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f651c0b8fa0 0x7f651c0b2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:19.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.445+0000 7f6521967700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f651c0b2f30 con 0x7f651c0ce350 2023-10-16T12:27:19.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.445+0000 7f6521967700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f651c0d1c40 con 0x7f651c15d8d0 2023-10-16T12:27:19.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.445+0000 7f6521967700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f651c0d0bb0 con 0x7f651c0b8fa0 2023-10-16T12:27:19.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.446+0000 7f651b7fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f651c0ce350 0x7f651c0d29c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59096/0 (socket says 172.21.15.16:59096) 2023-10-16T12:27:19.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.446+0000 7f651b7fe700 1 -- 172.21.15.16:0/309724927 learned_addr learned my addr 172.21.15.16:0/309724927 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:27:19.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.446+0000 7f651a7fc700 1 --2- 172.21.15.16:0/309724927 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f651c0b8fa0 0x7f651c0b2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:19.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.446+0000 7f6519ffb700 1 -- 172.21.15.16:0/309724927 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2071797732 0 0) 0x7f651c0b2f30 con 0x7f651c0ce350 2023-10-16T12:27:19.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.446+0000 7f6519ffb700 1 -- 172.21.15.16:0/309724927 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65040035b0 con 0x7f651c0ce350 2023-10-16T12:27:19.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.446+0000 7f6519ffb700 1 -- 172.21.15.16:0/309724927 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1438686283 0 0) 0x7f651c0d1c40 con 0x7f651c15d8d0 2023-10-16T12:27:19.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.446+0000 7f651a7fc700 1 -- 172.21.15.16:0/309724927 >> v1:172.21.15.49:6789/0 conn(0x7f651c15d8d0 legacy=0x7f651c0b6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:19.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.446+0000 7f651a7fc700 1 -- 172.21.15.16:0/309724927 >> v1:172.21.15.16:6789/0 conn(0x7f651c0ce350 legacy=0x7f651c0d29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:19.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.446+0000 7f651a7fc700 1 -- 172.21.15.16:0/309724927 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f651c0d1ee0 con 0x7f651c0b8fa0 2023-10-16T12:27:19.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.446+0000 7f651a7fc700 1 --2- 172.21.15.16:0/309724927 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f651c0b8fa0 0x7f651c0b2830 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f650c009870 tx=0x7f650c009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db21702674118fe4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:19.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.446+0000 7f6519ffb700 1 -- 172.21.15.16:0/309724927 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f650c011830 con 0x7f651c0b8fa0 2023-10-16T12:27:19.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.446+0000 7f6519ffb700 1 -- 172.21.15.16:0/309724927 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f650c011990 con 0x7f651c0b8fa0 2023-10-16T12:27:19.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.446+0000 7f6519ffb700 1 -- 172.21.15.16:0/309724927 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f650c011d20 con 0x7f651c0b8fa0 2023-10-16T12:27:19.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.447+0000 7f6521967700 1 -- 172.21.15.16:0/309724927 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f651c0b8fa0 msgr2=0x7f651c0b2830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:19.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.447+0000 7f6521967700 1 --2- 172.21.15.16:0/309724927 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f651c0b8fa0 0x7f651c0b2830 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f650c009870 tx=0x7f650c009b80 comp rx=0 tx=0).stop 2023-10-16T12:27:19.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.447+0000 7f6521967700 1 -- 172.21.15.16:0/309724927 shutdown_connections 2023-10-16T12:27:19.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.447+0000 7f6521967700 1 --2- 172.21.15.16:0/309724927 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f651c0b8fa0 0x7f651c0b2830 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:19.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.447+0000 7f6521967700 1 -- 172.21.15.16:0/309724927 >> 172.21.15.16:0/309724927 conn(0x7f651c072330 msgr2=0x7f651c072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:19.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.447+0000 7f6521967700 1 -- 172.21.15.16:0/309724927 shutdown_connections 2023-10-16T12:27:19.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.447+0000 7f6521967700 1 -- 172.21.15.16:0/309724927 wait complete. 2023-10-16T12:27:19.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.448+0000 7f6521967700 1 Processor -- start 2023-10-16T12:27:19.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.448+0000 7f6521967700 1 -- start start 2023-10-16T12:27:19.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.449+0000 7f6521967700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f651c0b8fa0 0x7f651c0abcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:19.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.449+0000 7f6521967700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f651c0ce350 0x7f651c160810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:19.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.449+0000 7f6521967700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f651c15d8d0 0x7f651c158b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:19.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.449+0000 7f6521967700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f651c15f1d0 con 0x7f651c0ce350 2023-10-16T12:27:19.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.449+0000 7f6521967700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f651c160d50 con 0x7f651c15d8d0 2023-10-16T12:27:19.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.449+0000 7f6521967700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f651c15ee30 con 0x7f651c0b8fa0 2023-10-16T12:27:19.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.449+0000 7f651a7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f651c0ce350 0x7f651c160810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:19.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.449+0000 7f651a7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f651c0ce350 0x7f651c160810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41594/0 (socket says 172.21.15.16:41594) 2023-10-16T12:27:19.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.449+0000 7f651a7fc700 1 -- 172.21.15.16:0/3750488929 learned_addr learned my addr 172.21.15.16:0/3750488929 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:19.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.449+0000 7f651affd700 1 --2- 172.21.15.16:0/3750488929 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f651c0b8fa0 0x7f651c0abcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:19.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.449+0000 7f651b7fe700 1 --2- 172.21.15.16:0/3750488929 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f651c15d8d0 0x7f651c158b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:19.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.450+0000 7f651a7fc700 1 -- 172.21.15.16:0/3750488929 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f651c0b8fa0 msgr2=0x7f651c0abcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:19.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.450+0000 7f651a7fc700 1 --2- 172.21.15.16:0/3750488929 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f651c0b8fa0 0x7f651c0abcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:19.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.450+0000 7f651a7fc700 1 -- 172.21.15.16:0/3750488929 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f651c15d8d0 msgr2=0x7f651c158b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:19.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.450+0000 7f651a7fc700 1 --2- 172.21.15.16:0/3750488929 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f651c15d8d0 0x7f651c158b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:19.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.450+0000 7f651a7fc700 1 -- 172.21.15.16:0/3750488929 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f650c0094a0 con 0x7f651c0ce350 2023-10-16T12:27:19.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.450+0000 7f651affd700 1 --2- 172.21.15.16:0/3750488929 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f651c0b8fa0 0x7f651c0abcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:27:19.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.450+0000 7f651b7fe700 1 --2- 172.21.15.16:0/3750488929 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f651c15d8d0 0x7f651c158b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:27:19.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.450+0000 7f651a7fc700 1 --2- 172.21.15.16:0/3750488929 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f651c0ce350 0x7f651c160810 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f650c014040 tx=0x7f650c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:19.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.450+0000 7f6503fff700 1 -- 172.21.15.16:0/3750488929 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f650c01a9f0 con 0x7f651c0ce350 2023-10-16T12:27:19.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.450+0000 7f6503fff700 1 -- 172.21.15.16:0/3750488929 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f650c01ab50 con 0x7f651c0ce350 2023-10-16T12:27:19.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.450+0000 7f6521967700 1 -- 172.21.15.16:0/3750488929 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f651c0ac360 con 0x7f651c0ce350 2023-10-16T12:27:19.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.450+0000 7f6503fff700 1 -- 172.21.15.16:0/3750488929 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f650c0243f0 con 0x7f651c0ce350 2023-10-16T12:27:19.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.450+0000 7f6521967700 1 -- 172.21.15.16:0/3750488929 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f651c0ac920 con 0x7f651c0ce350 2023-10-16T12:27:19.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.452+0000 7f6521967700 1 -- 172.21.15.16:0/3750488929 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f651c15f310 con 0x7f651c0ce350 2023-10-16T12:27:19.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.452+0000 7f6503fff700 1 -- 172.21.15.16:0/3750488929 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 47) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f650c02d450 con 0x7f651c0ce350 2023-10-16T12:27:19.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.452+0000 7f6503fff700 1 --2- 172.21.15.16:0/3750488929 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7f650806b020 0x7f650806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:19.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.452+0000 7f651affd700 1 -- 172.21.15.16:0/3750488929 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7f650806b020 msgr2=0x7f650806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/444334639 2023-10-16T12:27:19.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.452+0000 7f6503fff700 1 -- 172.21.15.16:0/3750488929 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f650c08ab90 con 0x7f651c0ce350 2023-10-16T12:27:19.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.452+0000 7f651affd700 1 --2- 172.21.15.16:0/3750488929 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7f650806b020 0x7f650806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:27:19.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.457+0000 7f6503fff700 1 -- 172.21.15.16:0/3750488929 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f651c15f310 con 0x7f651c0ce350 2023-10-16T12:27:19.519 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:19.518+0000 7f1434b9d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T12:27:19.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.653+0000 7f651affd700 1 -- 172.21.15.16:0/3750488929 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7f650806b020 msgr2=0x7f650806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/444334639 2023-10-16T12:27:19.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.653+0000 7f651affd700 1 --2- 172.21.15.16:0/3750488929 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7f650806b020 0x7f650806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:27:19.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.671+0000 7f6521967700 1 -- 172.21.15.16:0/3750488929 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f651c15f310 con 0x7f651c0ce350 2023-10-16T12:27:19.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.672+0000 7f6503fff700 1 -- 172.21.15.16:0/3750488929 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v100) v1 ==== 100+0+11 (secure 0 0 0) 0x7f651c15f310 con 0x7f651c0ce350 2023-10-16T12:27:19.674 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-10-16T12:27:19.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.674+0000 7f6521967700 1 -- 172.21.15.16:0/3750488929 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7f650806b020 msgr2=0x7f650806d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:27:19.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.674+0000 7f6521967700 1 --2- 172.21.15.16:0/3750488929 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7f650806b020 0x7f650806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:19.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.674+0000 7f6521967700 1 -- 172.21.15.16:0/3750488929 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f651c0ce350 msgr2=0x7f651c160810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:19.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.674+0000 7f6521967700 1 --2- 172.21.15.16:0/3750488929 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f651c0ce350 0x7f651c160810 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f650c014040 tx=0x7f650c004750 comp rx=0 tx=0).stop 2023-10-16T12:27:19.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.674+0000 7f6521967700 1 -- 172.21.15.16:0/3750488929 shutdown_connections 2023-10-16T12:27:19.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.674+0000 7f6521967700 1 --2- 172.21.15.16:0/3750488929 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f651c15d8d0 0x7f651c158b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:19.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.674+0000 7f6521967700 1 --2- 172.21.15.16:0/3750488929 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f651c0b8fa0 0x7f651c0abcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:19.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.674+0000 7f6521967700 1 --2- 172.21.15.16:0/3750488929 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7f650806b020 0x7f650806d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:19.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.674+0000 7f6521967700 1 --2- 172.21.15.16:0/3750488929 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f651c0ce350 0x7f651c160810 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:19.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.674+0000 7f6521967700 1 -- 172.21.15.16:0/3750488929 >> 172.21.15.16:0/3750488929 conn(0x7f651c072330 msgr2=0x7f651c0b3980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:19.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.674+0000 7f6521967700 1 -- 172.21.15.16:0/3750488929 shutdown_connections 2023-10-16T12:27:19.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.674+0000 7f6521967700 1 -- 172.21.15.16:0/3750488929 wait complete. 2023-10-16T12:27:19.689 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T12:27:19.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.814+0000 7feab90ed700 1 Processor -- start 2023-10-16T12:27:19.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.814+0000 7feab90ed700 1 -- start start 2023-10-16T12:27:19.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.815+0000 7feab90ed700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7feab415b630 0x7feab40d0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:19.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.815+0000 7feab90ed700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7feab415ba10 0x7feab40cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:19.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.815+0000 7feab90ed700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feab40b0970 con 0x7feab40cdde0 2023-10-16T12:27:19.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.815+0000 7feab90ed700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feab40cd690 con 0x7feab415b630 2023-10-16T12:27:19.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.815+0000 7feab90ed700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feab40cd970 con 0x7feab415ba10 2023-10-16T12:27:19.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.815+0000 7feab2d9d700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7feab415b630 0x7feab40d0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:19.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.815+0000 7feab259c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7feab415ba10 0x7feab40cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:19.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.815+0000 7feab259c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7feab415ba10 0x7feab40cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:58094/0 (socket says 172.21.15.16:58094) 2023-10-16T12:27:19.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.815+0000 7feab259c700 1 -- 172.21.15.16:0/2456622644 learned_addr learned my addr 172.21.15.16:0/2456622644 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:19.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.815+0000 7feab2d9d700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7feab415b630 0x7feab40d0380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:40664/0 (socket says 172.21.15.16:40664) 2023-10-16T12:27:19.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.815+0000 7feab359e700 1 --1- 172.21.15.16:0/2456622644 >> v1:172.21.15.49:6789/0 conn(0x7feab40cdde0 0x7feab40cc2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:38062/0 (socket says 172.21.15.16:38062) 2023-10-16T12:27:19.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.816+0000 7feab2d9d700 1 -- 172.21.15.16:0/2456622644 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7feab415ba10 msgr2=0x7feab40cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:19.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.816+0000 7feab2d9d700 1 --2- 172.21.15.16:0/2456622644 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7feab415ba10 0x7feab40cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:19.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.816+0000 7feab2d9d700 1 -- 172.21.15.16:0/2456622644 >> v1:172.21.15.49:6789/0 conn(0x7feab40cdde0 legacy=0x7feab40cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:19.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.816+0000 7feab2d9d700 1 -- 172.21.15.16:0/2456622644 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feab40b0c30 con 0x7feab415b630 2023-10-16T12:27:19.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.816+0000 7feab259c700 1 --2- 172.21.15.16:0/2456622644 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7feab415ba10 0x7feab40cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:27:19.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.816+0000 7feab2d9d700 1 --2- 172.21.15.16:0/2456622644 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feab415b630 0x7feab40d0380 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7feaa4009840 tx=0x7feaa4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=159c6c364a978313 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:19.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.816+0000 7feab1d9b700 1 -- 172.21.15.16:0/2456622644 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaa4011830 con 0x7feab415b630 2023-10-16T12:27:19.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.816+0000 7feab1d9b700 1 -- 172.21.15.16:0/2456622644 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaa4011990 con 0x7feab415b630 2023-10-16T12:27:19.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.816+0000 7feab1d9b700 1 -- 172.21.15.16:0/2456622644 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaa4011d20 con 0x7feab415b630 2023-10-16T12:27:19.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.817+0000 7feab90ed700 1 -- 172.21.15.16:0/2456622644 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feab415b630 msgr2=0x7feab40d0380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:19.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.817+0000 7feab90ed700 1 --2- 172.21.15.16:0/2456622644 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feab415b630 0x7feab40d0380 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7feaa4009840 tx=0x7feaa4009b50 comp rx=0 tx=0).stop 2023-10-16T12:27:19.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.817+0000 7feab90ed700 1 -- 172.21.15.16:0/2456622644 shutdown_connections 2023-10-16T12:27:19.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.817+0000 7feab90ed700 1 --2- 172.21.15.16:0/2456622644 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feab415b630 0x7feab40d0380 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:19.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.817+0000 7feab90ed700 1 --2- 172.21.15.16:0/2456622644 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7feab415ba10 0x7feab40cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:19.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.817+0000 7feab90ed700 1 -- 172.21.15.16:0/2456622644 >> 172.21.15.16:0/2456622644 conn(0x7feab40d14b0 msgr2=0x7feab40d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:19.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.817+0000 7feab90ed700 1 -- 172.21.15.16:0/2456622644 shutdown_connections 2023-10-16T12:27:19.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.817+0000 7feab90ed700 1 -- 172.21.15.16:0/2456622644 wait complete. 2023-10-16T12:27:19.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.818+0000 7feab90ed700 1 Processor -- start 2023-10-16T12:27:19.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.818+0000 7feab90ed700 1 -- start start 2023-10-16T12:27:19.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.819+0000 7feab90ed700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7feab40cdde0 0x7feab4150940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:19.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.819+0000 7feab90ed700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feab415b630 0x7feab415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:19.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.819+0000 7feab90ed700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feab415ba10 0x7feab415d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:19.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.819+0000 7feab90ed700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feab415e3d0 con 0x7feab415ba10 2023-10-16T12:27:19.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.819+0000 7feab90ed700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feab415e6b0 con 0x7feab415b630 2023-10-16T12:27:19.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.819+0000 7feab90ed700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feab415e990 con 0x7feab40cdde0 2023-10-16T12:27:19.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.819+0000 7feab359e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feab415ba10 0x7feab415d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:19.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.819+0000 7feab259c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feab415b630 0x7feab415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:19.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.819+0000 7feab359e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feab415ba10 0x7feab415d910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41612/0 (socket says 172.21.15.16:41612) 2023-10-16T12:27:19.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.819+0000 7feab2d9d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7feab40cdde0 0x7feab4150940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:19.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.819+0000 7feab359e700 1 -- 172.21.15.16:0/3323487053 learned_addr learned my addr 172.21.15.16:0/3323487053 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:19.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.820+0000 7feab259c700 1 -- 172.21.15.16:0/3323487053 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7feab40cdde0 msgr2=0x7feab4150940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:19.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.820+0000 7feab259c700 1 --2- 172.21.15.16:0/3323487053 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7feab40cdde0 0x7feab4150940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:19.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.820+0000 7feab259c700 1 -- 172.21.15.16:0/3323487053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feab415ba10 msgr2=0x7feab415d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:19.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.820+0000 7feab259c700 1 --2- 172.21.15.16:0/3323487053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feab415ba10 0x7feab415d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:19.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.820+0000 7feab259c700 1 -- 172.21.15.16:0/3323487053 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaa8009550 con 0x7feab415b630 2023-10-16T12:27:19.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.820+0000 7feab359e700 1 --2- 172.21.15.16:0/3323487053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feab415ba10 0x7feab415d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:27:19.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.820+0000 7feab259c700 1 --2- 172.21.15.16:0/3323487053 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feab415b630 0x7feab415bee0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7feaa800e980 tx=0x7feaa800ec90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:19.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.820+0000 7fea9b7fe700 1 -- 172.21.15.16:0/3323487053 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaa8004030 con 0x7feab415b630 2023-10-16T12:27:19.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.820+0000 7fea9b7fe700 1 -- 172.21.15.16:0/3323487053 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaa8004190 con 0x7feab415b630 2023-10-16T12:27:19.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.820+0000 7feab90ed700 1 -- 172.21.15.16:0/3323487053 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feaa40094a0 con 0x7feab415b630 2023-10-16T12:27:19.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.820+0000 7fea9b7fe700 1 -- 172.21.15.16:0/3323487053 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaa8004520 con 0x7feab415b630 2023-10-16T12:27:19.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.820+0000 7feab90ed700 1 -- 172.21.15.16:0/3323487053 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feab40c9950 con 0x7feab415b630 2023-10-16T12:27:19.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.821+0000 7feab90ed700 1 -- 172.21.15.16:0/3323487053 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feab40c9b10 con 0x7feab415b630 2023-10-16T12:27:19.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.822+0000 7fea9b7fe700 1 -- 172.21.15.16:0/3323487053 <== mon.1 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 47) v1 ==== 89332+0+0 (secure 0 0 0) 0x7feaa80036a0 con 0x7feab415b630 2023-10-16T12:27:19.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.822+0000 7fea9b7fe700 1 --2- 172.21.15.16:0/3323487053 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7feaa006b020 0x7feaa006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:19.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.823+0000 7feab2d9d700 1 -- 172.21.15.16:0/3323487053 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7feaa006b020 msgr2=0x7feaa006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/444334639 2023-10-16T12:27:19.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.823+0000 7fea9b7fe700 1 -- 172.21.15.16:0/3323487053 <== mon.1 v2:172.21.15.49:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6401+0+0 (secure 0 0 0) 0x7feaa80897d0 con 0x7feab415b630 2023-10-16T12:27:19.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.823+0000 7feab2d9d700 1 --2- 172.21.15.16:0/3323487053 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7feaa006b020 0x7feaa006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:27:19.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:19.831+0000 7fea9b7fe700 1 -- 172.21.15.16:0/3323487053 <== mon.1 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7feaa8054410 con 0x7feab415b630 2023-10-16T12:27:20.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.024+0000 7feab2d9d700 1 -- 172.21.15.16:0/3323487053 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7feaa006b020 msgr2=0x7feaa006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/444334639 2023-10-16T12:27:20.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.024+0000 7feab2d9d700 1 --2- 172.21.15.16:0/3323487053 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7feaa006b020 0x7feaa006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:27:20.058 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:27:20.057+0000 7f1533820200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T12:27:20.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.104+0000 7feab90ed700 1 -- 172.21.15.16:0/3323487053 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7feab415e7f0 con 0x7feab415b630 2023-10-16T12:27:20.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.105+0000 7fea9b7fe700 1 -- 172.21.15.16:0/3323487053 <== mon.1 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v100) v1 ==== 80+0+467 (secure 0 0 0) 0x7feab415e7f0 con 0x7feab415b630 2023-10-16T12:27:20.110 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6833/292891612 2023-10-17T12:27:15.899818+0000 2023-10-16T12:27:20.110 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6832/292891612 2023-10-17T12:27:15.899818+0000 2023-10-16T12:27:20.110 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/1020789692 2023-10-17T12:27:15.899818+0000 2023-10-16T12:27:20.110 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/478878390 2023-10-17T12:27:15.899818+0000 2023-10-16T12:27:20.111 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2293822331 2023-10-17T12:27:15.899818+0000 2023-10-16T12:27:20.111 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/1305507650 2023-10-17T12:27:15.899818+0000 2023-10-16T12:27:20.111 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3205233425 2023-10-17T12:27:15.899818+0000 2023-10-16T12:27:20.111 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2477042442 2023-10-17T12:27:15.899818+0000 2023-10-16T12:27:20.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.108+0000 7feab90ed700 1 -- 172.21.15.16:0/3323487053 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7feaa006b020 msgr2=0x7feaa006d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:27:20.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.108+0000 7feab90ed700 1 --2- 172.21.15.16:0/3323487053 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7feaa006b020 0x7feaa006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:20.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.108+0000 7feab90ed700 1 -- 172.21.15.16:0/3323487053 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feab415b630 msgr2=0x7feab415bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:20.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.108+0000 7feab90ed700 1 --2- 172.21.15.16:0/3323487053 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feab415b630 0x7feab415bee0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7feaa800e980 tx=0x7feaa800ec90 comp rx=0 tx=0).stop 2023-10-16T12:27:20.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.108+0000 7feab90ed700 1 -- 172.21.15.16:0/3323487053 shutdown_connections 2023-10-16T12:27:20.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.108+0000 7feab90ed700 1 --2- 172.21.15.16:0/3323487053 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7feaa006b020 0x7feaa006d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:20.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.108+0000 7feab90ed700 1 --2- 172.21.15.16:0/3323487053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feab415ba10 0x7feab415d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:20.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.108+0000 7feab90ed700 1 --2- 172.21.15.16:0/3323487053 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7feab40cdde0 0x7feab4150940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:20.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.108+0000 7feab90ed700 1 --2- 172.21.15.16:0/3323487053 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feab415b630 0x7feab415bee0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:20.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.108+0000 7feab90ed700 1 -- 172.21.15.16:0/3323487053 >> 172.21.15.16:0/3323487053 conn(0x7feab40d14b0 msgr2=0x7feab406eba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:20.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.108+0000 7feab90ed700 1 -- 172.21.15.16:0/3323487053 shutdown_connections 2023-10-16T12:27:20.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.108+0000 7feab90ed700 1 -- 172.21.15.16:0/3323487053 wait complete. 2023-10-16T12:27:20.115 INFO:teuthology.orchestra.run.smithi016.stderr:listed 8 entries 2023-10-16T12:27:20.123 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-16T12:27:20.123 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:27:20.124 DEBUG:teuthology.orchestra.run.smithi016:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-16T12:27:20.131 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:27:20.130+0000 7f1533820200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T12:27:20.139 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:20.137+0000 7f1434b9d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T12:27:20.166 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2023-10-16T12:27:20.167 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:27:20.167 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-16T12:27:20.214 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:20.213+0000 7f1434b9d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T12:27:20.224 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T12:27:20.225 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:27:20.242 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284204k,nr_inodes=4071051,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=753 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260540k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T12:27:20.242 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:27:20.243 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:27:20.285 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16287364k,nr_inodes=4071841,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20251 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3261172k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T12:27:20.285 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T12:27:20.286 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:27:20.304 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284204k,nr_inodes=4071051,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=753 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260540k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T12:27:20.304 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:27:20.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.449+0000 7f2c9906c700 1 Processor -- start 2023-10-16T12:27:20.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.449+0000 7f2c9906c700 1 -- start start 2023-10-16T12:27:20.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.450+0000 7f2c9906c700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f2c940ce870 0x7f2c940cb980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:20.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.450+0000 7f2c9906c700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c940cedb0 con 0x7f2c9415db40 2023-10-16T12:27:20.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.450+0000 7f2c9906c700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c940d12c0 con 0x7f2c9406aef0 2023-10-16T12:27:20.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.450+0000 7f2c9906c700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c9415df20 con 0x7f2c940ce870 2023-10-16T12:27:20.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.450+0000 7f2c92d9d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2c9415db40 0x7f2c9406a7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59124/0 (socket says 172.21.15.16:59124) 2023-10-16T12:27:20.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.450+0000 7f2c92d9d700 1 -- 172.21.15.16:0/2168265355 learned_addr learned my addr 172.21.15.16:0/2168265355 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:27:20.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.451+0000 7f2c9359e700 1 --2- 172.21.15.16:0/2168265355 >> v2:172.21.15.49:3300/0 conn(0x7f2c940ce870 0x7f2c940cb980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:20.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.451+0000 7f2c91d9b700 1 -- 172.21.15.16:0/2168265355 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1865925588 0 0) 0x7f2c940cedb0 con 0x7f2c9415db40 2023-10-16T12:27:20.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.451+0000 7f2c91d9b700 1 -- 172.21.15.16:0/2168265355 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c7c0035b0 con 0x7f2c9415db40 2023-10-16T12:27:20.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.451+0000 7f2c91d9b700 1 -- 172.21.15.16:0/2168265355 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2900192758 0 0) 0x7f2c940d12c0 con 0x7f2c9406aef0 2023-10-16T12:27:20.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.451+0000 7f2c91d9b700 1 -- 172.21.15.16:0/2168265355 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c940cedb0 con 0x7f2c9406aef0 2023-10-16T12:27:20.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.451+0000 7f2c9359e700 1 -- 172.21.15.16:0/2168265355 >> v1:172.21.15.49:6789/0 conn(0x7f2c9406aef0 legacy=0x7f2c940ce160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:20.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.451+0000 7f2c9359e700 1 -- 172.21.15.16:0/2168265355 >> v1:172.21.15.16:6789/0 conn(0x7f2c9415db40 legacy=0x7f2c9406a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:20.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.451+0000 7f2c9359e700 1 -- 172.21.15.16:0/2168265355 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c940cabc0 con 0x7f2c940ce870 2023-10-16T12:27:20.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.451+0000 7f2c9359e700 1 --2- 172.21.15.16:0/2168265355 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2c940ce870 0x7f2c940cb980 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c009d90 tx=0x7f2c8c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8d635c98371ce882 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:20.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.451+0000 7f2c91d9b700 1 -- 172.21.15.16:0/2168265355 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c8c00e070 con 0x7f2c940ce870 2023-10-16T12:27:20.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.451+0000 7f2c91d9b700 1 -- 172.21.15.16:0/2168265355 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c8c00bb20 con 0x7f2c940ce870 2023-10-16T12:27:20.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.451+0000 7f2c91d9b700 1 -- 172.21.15.16:0/2168265355 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c8c00be90 con 0x7f2c940ce870 2023-10-16T12:27:20.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.452+0000 7f2c9906c700 1 -- 172.21.15.16:0/2168265355 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2c940ce870 msgr2=0x7f2c940cb980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:20.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.452+0000 7f2c9906c700 1 --2- 172.21.15.16:0/2168265355 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2c940ce870 0x7f2c940cb980 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c009d90 tx=0x7f2c8c00b330 comp rx=0 tx=0).stop 2023-10-16T12:27:20.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.452+0000 7f2c9906c700 1 -- 172.21.15.16:0/2168265355 shutdown_connections 2023-10-16T12:27:20.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.452+0000 7f2c9906c700 1 --2- 172.21.15.16:0/2168265355 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2c940ce870 0x7f2c940cb980 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:20.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.452+0000 7f2c9906c700 1 -- 172.21.15.16:0/2168265355 >> 172.21.15.16:0/2168265355 conn(0x7f2c940b1fd0 msgr2=0x7f2c940b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:20.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.452+0000 7f2c9906c700 1 -- 172.21.15.16:0/2168265355 shutdown_connections 2023-10-16T12:27:20.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.452+0000 7f2c9906c700 1 -- 172.21.15.16:0/2168265355 wait complete. 2023-10-16T12:27:20.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.454+0000 7f2c9906c700 1 Processor -- start 2023-10-16T12:27:20.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.454+0000 7f2c9906c700 1 -- start start 2023-10-16T12:27:20.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.454+0000 7f2c9906c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2c9406aef0 0x7f2c9414e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:20.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.454+0000 7f2c9906c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2c940ce870 0x7f2c94151300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:20.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.454+0000 7f2c9906c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c9415db40 0x7f2c941f6780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:20.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.454+0000 7f2c9906c700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c940c6910 con 0x7f2c9415db40 2023-10-16T12:27:20.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.454+0000 7f2c9906c700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c94160c40 con 0x7f2c9406aef0 2023-10-16T12:27:20.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.454+0000 7f2c9906c700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c940c6b80 con 0x7f2c940ce870 2023-10-16T12:27:20.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.454+0000 7f2c9359e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c9415db40 0x7f2c941f6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:20.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.455+0000 7f2c9359e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c9415db40 0x7f2c941f6780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41628/0 (socket says 172.21.15.16:41628) 2023-10-16T12:27:20.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.455+0000 7f2c9259c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2c940ce870 0x7f2c94151300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:20.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.455+0000 7f2c9359e700 1 -- 172.21.15.16:0/3415848891 learned_addr learned my addr 172.21.15.16:0/3415848891 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:20.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.455+0000 7f2c92d9d700 1 --2- 172.21.15.16:0/3415848891 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2c9406aef0 0x7f2c9414e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:20.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.455+0000 7f2c9359e700 1 -- 172.21.15.16:0/3415848891 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2c940ce870 msgr2=0x7f2c94151300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:20.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.455+0000 7f2c9359e700 1 --2- 172.21.15.16:0/3415848891 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2c940ce870 0x7f2c94151300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:20.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.455+0000 7f2c9359e700 1 -- 172.21.15.16:0/3415848891 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2c9406aef0 msgr2=0x7f2c9414e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:20.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.455+0000 7f2c9359e700 1 --2- 172.21.15.16:0/3415848891 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2c9406aef0 0x7f2c9414e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:20.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.455+0000 7f2c9359e700 1 -- 172.21.15.16:0/3415848891 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c8c009a70 con 0x7f2c9415db40 2023-10-16T12:27:20.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.455+0000 7f2c9259c700 1 --2- 172.21.15.16:0/3415848891 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2c940ce870 0x7f2c94151300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:27:20.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.455+0000 7f2c9359e700 1 --2- 172.21.15.16:0/3415848891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c9415db40 0x7f2c941f6780 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c013000 tx=0x7f2c8c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:20.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.455+0000 7f2c7b7fe700 1 -- 172.21.15.16:0/3415848891 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c8c004510 con 0x7f2c9415db40 2023-10-16T12:27:20.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.455+0000 7f2c7b7fe700 1 -- 172.21.15.16:0/3415848891 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c8c020070 con 0x7f2c9415db40 2023-10-16T12:27:20.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.455+0000 7f2c9906c700 1 -- 172.21.15.16:0/3415848891 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c94160f20 con 0x7f2c9415db40 2023-10-16T12:27:20.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.455+0000 7f2c7b7fe700 1 -- 172.21.15.16:0/3415848891 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c8c0048b0 con 0x7f2c9415db40 2023-10-16T12:27:20.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.455+0000 7f2c9906c700 1 -- 172.21.15.16:0/3415848891 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c941f6ea0 con 0x7f2c9415db40 2023-10-16T12:27:20.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.457+0000 7f2c9906c700 1 -- 172.21.15.16:0/3415848891 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c9404ebb0 con 0x7f2c9415db40 2023-10-16T12:27:20.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.457+0000 7f2c7b7fe700 1 -- 172.21.15.16:0/3415848891 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 47) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f2c8c012430 con 0x7f2c9415db40 2023-10-16T12:27:20.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.457+0000 7f2c7b7fe700 1 --2- 172.21.15.16:0/3415848891 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7f2c8006b020 0x7f2c8006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:20.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.457+0000 7f2c92d9d700 1 -- 172.21.15.16:0/3415848891 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7f2c8006b020 msgr2=0x7f2c8006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/444334639 2023-10-16T12:27:20.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.457+0000 7f2c7b7fe700 1 -- 172.21.15.16:0/3415848891 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f2c8c022070 con 0x7f2c9415db40 2023-10-16T12:27:20.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.457+0000 7f2c92d9d700 1 --2- 172.21.15.16:0/3415848891 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7f2c8006b020 0x7f2c8006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:27:20.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.462+0000 7f2c7b7fe700 1 -- 172.21.15.16:0/3415848891 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2c8c0560e0 con 0x7f2c9415db40 2023-10-16T12:27:20.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.658+0000 7f2c92d9d700 1 -- 172.21.15.16:0/3415848891 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7f2c8006b020 msgr2=0x7f2c8006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/444334639 2023-10-16T12:27:20.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.658+0000 7f2c92d9d700 1 --2- 172.21.15.16:0/3415848891 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7f2c8006b020 0x7f2c8006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:27:20.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.712+0000 7f2c9906c700 1 -- 172.21.15.16:0/3415848891 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2c941f6ea0 con 0x7f2c9415db40 2023-10-16T12:27:20.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.712+0000 7f2c7b7fe700 1 -- 172.21.15.16:0/3415848891 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+4915 (secure 0 0 0) 0x7f2c941f6ea0 con 0x7f2c9415db40 2023-10-16T12:27:20.715 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:27:20.716 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":44,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6820,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/1256919442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":1256919442},{"type":"v1","addr":"172.21.15.49:6834","nonce":1256919442}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":6823,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/3811815346","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":3811815346},{"type":"v1","addr":"172.21.15.49:6835","nonce":3811815346}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":7110,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/1226064755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":1226064755},{"type":"v1","addr":"172.21.15.16:6836","nonce":1226064755}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:27:00.825934+0000","modified":"2023-10-16T12:27:03.831266+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7113},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7113":{"gid":7113,"name":"a","rank":0,"incarnation":40,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/1843321005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1843321005},{"type":"v1","addr":"172.21.15.16:6837","nonce":1843321005}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-10-16T12:27:20.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.714+0000 7f2c9906c700 1 -- 172.21.15.16:0/3415848891 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7f2c8006b020 msgr2=0x7f2c8006d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:27:20.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.714+0000 7f2c9906c700 1 --2- 172.21.15.16:0/3415848891 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7f2c8006b020 0x7f2c8006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:20.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.714+0000 7f2c9906c700 1 -- 172.21.15.16:0/3415848891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c9415db40 msgr2=0x7f2c941f6780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:20.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.714+0000 7f2c9906c700 1 --2- 172.21.15.16:0/3415848891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c9415db40 0x7f2c941f6780 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c013000 tx=0x7f2c8c004ad0 comp rx=0 tx=0).stop 2023-10-16T12:27:20.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.714+0000 7f2c9906c700 1 -- 172.21.15.16:0/3415848891 shutdown_connections 2023-10-16T12:27:20.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.714+0000 7f2c9906c700 1 --2- 172.21.15.16:0/3415848891 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7f2c8006b020 0x7f2c8006d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:20.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.714+0000 7f2c9906c700 1 --2- 172.21.15.16:0/3415848891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c9415db40 0x7f2c941f6780 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:20.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.714+0000 7f2c9906c700 1 --2- 172.21.15.16:0/3415848891 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2c9406aef0 0x7f2c9414e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:20.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.714+0000 7f2c9906c700 1 --2- 172.21.15.16:0/3415848891 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2c940ce870 0x7f2c94151300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:20.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.714+0000 7f2c9906c700 1 -- 172.21.15.16:0/3415848891 >> 172.21.15.16:0/3415848891 conn(0x7f2c940b1fd0 msgr2=0x7f2c940ce130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:20.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.714+0000 7f2c9906c700 1 -- 172.21.15.16:0/3415848891 shutdown_connections 2023-10-16T12:27:20.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.715+0000 7f2c9906c700 1 -- 172.21.15.16:0/3415848891 wait complete. 2023-10-16T12:27:20.723 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 44 2023-10-16T12:27:20.729 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:27:20.753 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:27:20.775 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:27:20.799 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:27:20.797+0000 7f1533820200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:27:20.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.857+0000 7fec7c1a8700 1 Processor -- start 2023-10-16T12:27:20.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.857+0000 7fec7c1a8700 1 -- start start 2023-10-16T12:27:20.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.858+0000 7fec7c1a8700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fec740ce4f0 0x7fec740d0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:20.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.858+0000 7fec7c1a8700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec740b0970 con 0x7fec7415ba10 2023-10-16T12:27:20.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.858+0000 7fec7c1a8700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec740cd690 con 0x7fec7415b630 2023-10-16T12:27:20.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.858+0000 7fec7c1a8700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec740cd970 con 0x7fec740ce4f0 2023-10-16T12:27:20.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.858+0000 7fec7a745700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fec7415ba10 0x7fec740b1430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59144/0 (socket says 172.21.15.16:59144) 2023-10-16T12:27:20.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.858+0000 7fec7a745700 1 -- 172.21.15.16:0/3284022075 learned_addr learned my addr 172.21.15.16:0/3284022075 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:27:20.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.859+0000 7fec79743700 1 --2- 172.21.15.16:0/3284022075 >> v2:172.21.15.49:3300/0 conn(0x7fec740ce4f0 0x7fec740d0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:20.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.859+0000 7fec78f42700 1 -- 172.21.15.16:0/3284022075 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3442837825 0 0) 0x7fec740b0970 con 0x7fec7415ba10 2023-10-16T12:27:20.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.859+0000 7fec78f42700 1 -- 172.21.15.16:0/3284022075 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec600035b0 con 0x7fec7415ba10 2023-10-16T12:27:20.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.859+0000 7fec79743700 1 -- 172.21.15.16:0/3284022075 >> v1:172.21.15.49:6789/0 conn(0x7fec7415b630 legacy=0x7fec740cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:20.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.859+0000 7fec79743700 1 -- 172.21.15.16:0/3284022075 >> v1:172.21.15.16:6789/0 conn(0x7fec7415ba10 legacy=0x7fec740b1430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:20.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.859+0000 7fec78f42700 1 -- 172.21.15.16:0/3284022075 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 285088694 0 0) 0x7fec740cd690 con 0x7fec7415b630 2023-10-16T12:27:20.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.859+0000 7fec79743700 1 -- 172.21.15.16:0/3284022075 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec740b0c30 con 0x7fec740ce4f0 2023-10-16T12:27:20.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.859+0000 7fec79743700 1 --2- 172.21.15.16:0/3284022075 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fec740ce4f0 0x7fec740d0380 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fec64009840 tx=0x7fec64009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5de65e9386fe380d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:20.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.859+0000 7fec78f42700 1 -- 172.21.15.16:0/3284022075 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec64011830 con 0x7fec740ce4f0 2023-10-16T12:27:20.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.859+0000 7fec78f42700 1 -- 172.21.15.16:0/3284022075 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec64011990 con 0x7fec740ce4f0 2023-10-16T12:27:20.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.859+0000 7fec78f42700 1 -- 172.21.15.16:0/3284022075 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec64011d20 con 0x7fec740ce4f0 2023-10-16T12:27:20.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.860+0000 7fec7c1a8700 1 -- 172.21.15.16:0/3284022075 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fec740ce4f0 msgr2=0x7fec740d0380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:20.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.860+0000 7fec7c1a8700 1 --2- 172.21.15.16:0/3284022075 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fec740ce4f0 0x7fec740d0380 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fec64009840 tx=0x7fec64009b50 comp rx=0 tx=0).stop 2023-10-16T12:27:20.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.860+0000 7fec7c1a8700 1 -- 172.21.15.16:0/3284022075 shutdown_connections 2023-10-16T12:27:20.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.860+0000 7fec7c1a8700 1 --2- 172.21.15.16:0/3284022075 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fec740ce4f0 0x7fec740d0380 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:20.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.860+0000 7fec7c1a8700 1 -- 172.21.15.16:0/3284022075 >> 172.21.15.16:0/3284022075 conn(0x7fec740d14b0 msgr2=0x7fec740d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:20.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.860+0000 7fec7c1a8700 1 -- 172.21.15.16:0/3284022075 shutdown_connections 2023-10-16T12:27:20.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.860+0000 7fec7c1a8700 1 -- 172.21.15.16:0/3284022075 wait complete. 2023-10-16T12:27:20.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.861+0000 7fec7c1a8700 1 Processor -- start 2023-10-16T12:27:20.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.861+0000 7fec7c1a8700 1 -- start start 2023-10-16T12:27:20.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.862+0000 7fec7c1a8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec740ce4f0 0x7fec7406b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:20.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.862+0000 7fec7c1a8700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fec7415b630 0x7fec740b68b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:20.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.862+0000 7fec7c1a8700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fec7415ba10 0x7fec740b43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:20.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.862+0000 7fec7c1a8700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec7415c010 con 0x7fec740ce4f0 2023-10-16T12:27:20.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.862+0000 7fec7c1a8700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec740b6f90 con 0x7fec7415b630 2023-10-16T12:27:20.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.862+0000 7fec7c1a8700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec740b7270 con 0x7fec7415ba10 2023-10-16T12:27:20.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.862+0000 7fec79f44700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec740ce4f0 0x7fec7406b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:20.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.862+0000 7fec79f44700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec740ce4f0 0x7fec7406b8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41640/0 (socket says 172.21.15.16:41640) 2023-10-16T12:27:20.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.862+0000 7fec79743700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fec7415b630 0x7fec740b68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:20.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.862+0000 7fec79f44700 1 -- 172.21.15.16:0/1164676248 learned_addr learned my addr 172.21.15.16:0/1164676248 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:20.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.862+0000 7fec7a745700 1 --2- 172.21.15.16:0/1164676248 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fec7415ba10 0x7fec740b43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:20.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.862+0000 7fec79f44700 1 -- 172.21.15.16:0/1164676248 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fec7415ba10 msgr2=0x7fec740b43e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:20.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.863+0000 7fec79f44700 1 --2- 172.21.15.16:0/1164676248 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fec7415ba10 0x7fec740b43e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:20.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.863+0000 7fec79f44700 1 -- 172.21.15.16:0/1164676248 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fec7415b630 msgr2=0x7fec740b68b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:20.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.863+0000 7fec79f44700 1 --2- 172.21.15.16:0/1164676248 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fec7415b630 0x7fec740b68b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:20.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.863+0000 7fec79f44700 1 -- 172.21.15.16:0/1164676248 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec640094a0 con 0x7fec740ce4f0 2023-10-16T12:27:20.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.863+0000 7fec79743700 1 --2- 172.21.15.16:0/1164676248 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fec7415b630 0x7fec740b68b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:27:20.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.863+0000 7fec79f44700 1 --2- 172.21.15.16:0/1164676248 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec740ce4f0 0x7fec7406b8f0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7fec700011e0 tx=0x7fec7000ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:20.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.863+0000 7fec6a7fc700 1 -- 172.21.15.16:0/1164676248 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec700054e0 con 0x7fec740ce4f0 2023-10-16T12:27:20.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.863+0000 7fec6a7fc700 1 -- 172.21.15.16:0/1164676248 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec70003680 con 0x7fec740ce4f0 2023-10-16T12:27:20.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.863+0000 7fec7c1a8700 1 -- 172.21.15.16:0/1164676248 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec7415c2b0 con 0x7fec740ce4f0 2023-10-16T12:27:20.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.863+0000 7fec6a7fc700 1 -- 172.21.15.16:0/1164676248 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec70003a30 con 0x7fec740ce4f0 2023-10-16T12:27:20.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.863+0000 7fec7c1a8700 1 -- 172.21.15.16:0/1164676248 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec740b4920 con 0x7fec740ce4f0 2023-10-16T12:27:20.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.864+0000 7fec7c1a8700 1 -- 172.21.15.16:0/1164676248 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec7415c7d0 con 0x7fec740ce4f0 2023-10-16T12:27:20.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.864+0000 7fec6a7fc700 1 -- 172.21.15.16:0/1164676248 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 47) v1 ==== 89332+0+0 (secure 0 0 0) 0x7fec70003b90 con 0x7fec740ce4f0 2023-10-16T12:27:20.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.865+0000 7fec6a7fc700 1 --2- 172.21.15.16:0/1164676248 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7fec5806b020 0x7fec5806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:20.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.865+0000 7fec79743700 1 -- 172.21.15.16:0/1164676248 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7fec5806b020 msgr2=0x7fec5806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/444334639 2023-10-16T12:27:20.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.865+0000 7fec6a7fc700 1 -- 172.21.15.16:0/1164676248 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fec70088d90 con 0x7fec740ce4f0 2023-10-16T12:27:20.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.865+0000 7fec79743700 1 --2- 172.21.15.16:0/1164676248 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7fec5806b020 0x7fec5806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:27:20.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:20.870+0000 7fec6a7fc700 1 -- 172.21.15.16:0/1164676248 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fec7415c7d0 con 0x7fec740ce4f0 2023-10-16T12:27:20.904 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:20.903+0000 7f1434b9d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:27:20.914 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:27:20.879+0000 7f1533820200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T12:27:20.987 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:20.986+0000 7f1434b9d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T12:27:21.034 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:27:21.032+0000 7f1533820200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:27:21.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.066+0000 7fec79743700 1 -- 172.21.15.16:0/1164676248 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7fec5806b020 msgr2=0x7fec5806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/444334639 2023-10-16T12:27:21.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.066+0000 7fec79743700 1 --2- 172.21.15.16:0/1164676248 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7fec5806b020 0x7fec5806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:27:21.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.084+0000 7fec7c1a8700 1 -- 172.21.15.16:0/1164676248 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fec7415c7d0 con 0x7fec740ce4f0 2023-10-16T12:27:21.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.085+0000 7fec6a7fc700 1 -- 172.21.15.16:0/1164676248 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) v1 ==== 74+0+17211 (secure 0 0 0) 0x7fec7415c7d0 con 0x7fec740ce4f0 2023-10-16T12:27:21.087 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:27:21.087 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":100,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:27:15.899863+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:26:59.206630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2023-10-16T12:26:59.784585+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6833/292891612":"2023-10-17T12:27:15.899818+0000","172.21.15.16:6832/292891612":"2023-10-17T12:27:15.899818+0000","172.21.15.16:0/1020789692":"2023-10-17T12:27:15.899818+0000","172.21.15.16:0/478878390":"2023-10-17T12:27:15.899818+0000","172.21.15.16:0/2293822331":"2023-10-17T12:27:15.899818+0000","172.21.15.16:0/1305507650":"2023-10-17T12:27:15.899818+0000","172.21.15.16:0/3205233425":"2023-10-17T12:27:15.899818+0000","172.21.15.16:0/2477042442":"2023-10-17T12:27:15.899818+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:27:21.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.087+0000 7fec7c1a8700 1 -- 172.21.15.16:0/1164676248 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7fec5806b020 msgr2=0x7fec5806d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:27:21.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.087+0000 7fec7c1a8700 1 --2- 172.21.15.16:0/1164676248 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7fec5806b020 0x7fec5806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:21.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.087+0000 7fec7c1a8700 1 -- 172.21.15.16:0/1164676248 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec740ce4f0 msgr2=0x7fec7406b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:21.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.087+0000 7fec7c1a8700 1 --2- 172.21.15.16:0/1164676248 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec740ce4f0 0x7fec7406b8f0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7fec700011e0 tx=0x7fec7000ca30 comp rx=0 tx=0).stop 2023-10-16T12:27:21.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.087+0000 7fec7c1a8700 1 -- 172.21.15.16:0/1164676248 shutdown_connections 2023-10-16T12:27:21.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.087+0000 7fec7c1a8700 1 --2- 172.21.15.16:0/1164676248 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fec7415ba10 0x7fec740b43e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:21.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.087+0000 7fec7c1a8700 1 --2- 172.21.15.16:0/1164676248 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7fec5806b020 0x7fec5806d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:21.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.087+0000 7fec7c1a8700 1 --2- 172.21.15.16:0/1164676248 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec740ce4f0 0x7fec7406b8f0 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:21.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.087+0000 7fec7c1a8700 1 --2- 172.21.15.16:0/1164676248 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fec7415b630 0x7fec740b68b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:21.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.087+0000 7fec7c1a8700 1 -- 172.21.15.16:0/1164676248 >> 172.21.15.16:0/1164676248 conn(0x7fec740d14b0 msgr2=0x7fec740704d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:21.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.087+0000 7fec7c1a8700 1 -- 172.21.15.16:0/1164676248 shutdown_connections 2023-10-16T12:27:21.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.087+0000 7fec7c1a8700 1 -- 172.21.15.16:0/1164676248 wait complete. 2023-10-16T12:27:21.102 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T12:26:59.206630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '98', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T12:26:59.784585+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '99', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T12:27:21.103 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:27:21.141 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:21.139+0000 7f1434b9d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:27:21.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.269+0000 7fe537514700 1 Processor -- start 2023-10-16T12:27:21.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.269+0000 7fe537514700 1 -- start start 2023-10-16T12:27:21.272 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:27:21.269+0000 7f1533820200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T12:27:21.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.270+0000 7fe537514700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fe53015ec80 0x7fe5300cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:21.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.270+0000 7fe537514700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5300b2440 con 0x7fe53015f060 2023-10-16T12:27:21.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.270+0000 7fe537514700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5300d00e0 con 0x7fe5300d2110 2023-10-16T12:27:21.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.270+0000 7fe537514700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5300b2c50 con 0x7fe53015ec80 2023-10-16T12:27:21.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.270+0000 7fe534aaf700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe53015f060 0x7fe5300b4440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59184/0 (socket says 172.21.15.16:59184) 2023-10-16T12:27:21.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.270+0000 7fe534aaf700 1 -- 172.21.15.16:0/2408245421 learned_addr learned my addr 172.21.15.16:0/2408245421 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:27:21.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.270+0000 7fe5352b0700 1 --2- 172.21.15.16:0/2408245421 >> v2:172.21.15.49:3300/0 conn(0x7fe53015ec80 0x7fe5300cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:21.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.270+0000 7fe527fff700 1 -- 172.21.15.16:0/2408245421 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4114577578 0 0) 0x7fe5300b2440 con 0x7fe53015f060 2023-10-16T12:27:21.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.270+0000 7fe527fff700 1 -- 172.21.15.16:0/2408245421 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe51c0035b0 con 0x7fe53015f060 2023-10-16T12:27:21.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.270+0000 7fe527fff700 1 -- 172.21.15.16:0/2408245421 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3152665933 0 0) 0x7fe5300d00e0 con 0x7fe5300d2110 2023-10-16T12:27:21.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.270+0000 7fe5352b0700 1 -- 172.21.15.16:0/2408245421 >> v1:172.21.15.49:6789/0 conn(0x7fe5300d2110 legacy=0x7fe5300b4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:21.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.270+0000 7fe5352b0700 1 -- 172.21.15.16:0/2408245421 >> v1:172.21.15.16:6789/0 conn(0x7fe53015f060 legacy=0x7fe5300b4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:21.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.270+0000 7fe5352b0700 1 -- 172.21.15.16:0/2408245421 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5300cef60 con 0x7fe53015ec80 2023-10-16T12:27:21.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.270+0000 7fe5352b0700 1 --2- 172.21.15.16:0/2408245421 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe53015ec80 0x7fe5300cdec0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fe52c009840 tx=0x7fe52c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b302fbe845dacb3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:21.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.271+0000 7fe527fff700 1 -- 172.21.15.16:0/2408245421 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe52c011830 con 0x7fe53015ec80 2023-10-16T12:27:21.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.271+0000 7fe527fff700 1 -- 172.21.15.16:0/2408245421 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe52c011990 con 0x7fe53015ec80 2023-10-16T12:27:21.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.271+0000 7fe527fff700 1 -- 172.21.15.16:0/2408245421 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe52c011d20 con 0x7fe53015ec80 2023-10-16T12:27:21.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.271+0000 7fe537514700 1 -- 172.21.15.16:0/2408245421 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe53015ec80 msgr2=0x7fe5300cdec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:21.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.271+0000 7fe537514700 1 --2- 172.21.15.16:0/2408245421 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe53015ec80 0x7fe5300cdec0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fe52c009840 tx=0x7fe52c009b50 comp rx=0 tx=0).stop 2023-10-16T12:27:21.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.271+0000 7fe537514700 1 -- 172.21.15.16:0/2408245421 shutdown_connections 2023-10-16T12:27:21.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.271+0000 7fe537514700 1 --2- 172.21.15.16:0/2408245421 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe53015ec80 0x7fe5300cdec0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:21.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.271+0000 7fe537514700 1 -- 172.21.15.16:0/2408245421 >> 172.21.15.16:0/2408245421 conn(0x7fe5300d15f0 msgr2=0x7fe5300d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:21.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.271+0000 7fe537514700 1 -- 172.21.15.16:0/2408245421 shutdown_connections 2023-10-16T12:27:21.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.271+0000 7fe537514700 1 -- 172.21.15.16:0/2408245421 wait complete. 2023-10-16T12:27:21.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.273+0000 7fe537514700 1 Processor -- start 2023-10-16T12:27:21.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.273+0000 7fe537514700 1 -- start start 2023-10-16T12:27:21.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.273+0000 7fe537514700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5300d2110 0x7fe5301f4c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:21.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.273+0000 7fe537514700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe53015ec80 0x7fe530153760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:21.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.273+0000 7fe537514700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe53015f060 0x7fe5300b6820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:21.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.273+0000 7fe537514700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5301e61d0 con 0x7fe5300d2110 2023-10-16T12:27:21.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.273+0000 7fe537514700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5301e69f0 con 0x7fe53015f060 2023-10-16T12:27:21.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.273+0000 7fe537514700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5301e6cd0 con 0x7fe53015ec80 2023-10-16T12:27:21.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.273+0000 7fe5352b0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5300d2110 0x7fe5301f4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:21.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.274+0000 7fe5352b0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5300d2110 0x7fe5301f4c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41682/0 (socket says 172.21.15.16:41682) 2023-10-16T12:27:21.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.274+0000 7fe5352b0700 1 -- 172.21.15.16:0/3706098309 learned_addr learned my addr 172.21.15.16:0/3706098309 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:21.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.274+0000 7fe534aaf700 1 --2- 172.21.15.16:0/3706098309 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe53015ec80 0x7fe530153760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:21.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.274+0000 7fe535ab1700 1 --2- 172.21.15.16:0/3706098309 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe53015f060 0x7fe5300b6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:21.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.274+0000 7fe5352b0700 1 -- 172.21.15.16:0/3706098309 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe53015ec80 msgr2=0x7fe530153760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:21.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.274+0000 7fe5352b0700 1 --2- 172.21.15.16:0/3706098309 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe53015ec80 0x7fe530153760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:21.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.274+0000 7fe5352b0700 1 -- 172.21.15.16:0/3706098309 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe53015f060 msgr2=0x7fe5300b6820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:21.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.274+0000 7fe5352b0700 1 --2- 172.21.15.16:0/3706098309 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe53015f060 0x7fe5300b6820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:21.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.274+0000 7fe5352b0700 1 -- 172.21.15.16:0/3706098309 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe52c0094a0 con 0x7fe5300d2110 2023-10-16T12:27:21.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.274+0000 7fe535ab1700 1 --2- 172.21.15.16:0/3706098309 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe53015f060 0x7fe5300b6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:27:21.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.274+0000 7fe5352b0700 1 --2- 172.21.15.16:0/3706098309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5300d2110 0x7fe5301f4c20 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fe52c0090b0 tx=0x7fe52c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:21.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.274+0000 7fe525ffb700 1 -- 172.21.15.16:0/3706098309 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe52c01a860 con 0x7fe5300d2110 2023-10-16T12:27:21.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.274+0000 7fe525ffb700 1 -- 172.21.15.16:0/3706098309 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe52c01a9c0 con 0x7fe5300d2110 2023-10-16T12:27:21.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.274+0000 7fe537514700 1 -- 172.21.15.16:0/3706098309 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5300c9d80 con 0x7fe5300d2110 2023-10-16T12:27:21.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.274+0000 7fe525ffb700 1 -- 172.21.15.16:0/3706098309 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe52c01ad30 con 0x7fe5300d2110 2023-10-16T12:27:21.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.274+0000 7fe537514700 1 -- 172.21.15.16:0/3706098309 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe530153ca0 con 0x7fe5300d2110 2023-10-16T12:27:21.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.275+0000 7fe537514700 1 -- 172.21.15.16:0/3706098309 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe53004ebb0 con 0x7fe5300d2110 2023-10-16T12:27:21.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.276+0000 7fe525ffb700 1 -- 172.21.15.16:0/3706098309 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 47) v1 ==== 89332+0+0 (secure 0 0 0) 0x7fe52c022070 con 0x7fe5300d2110 2023-10-16T12:27:21.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.276+0000 7fe525ffb700 1 --2- 172.21.15.16:0/3706098309 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7fe51406b020 0x7fe51406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:21.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.277+0000 7fe534aaf700 1 -- 172.21.15.16:0/3706098309 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7fe51406b020 msgr2=0x7fe51406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/444334639 2023-10-16T12:27:21.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.277+0000 7fe525ffb700 1 -- 172.21.15.16:0/3706098309 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fe52c08abf0 con 0x7fe5300d2110 2023-10-16T12:27:21.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.277+0000 7fe534aaf700 1 --2- 172.21.15.16:0/3706098309 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7fe51406b020 0x7fe51406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:27:21.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.281+0000 7fe525ffb700 1 -- 172.21.15.16:0/3706098309 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe52c055970 con 0x7fe5300d2110 2023-10-16T12:27:21.348 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:27:21.346+0000 7f1533820200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T12:27:21.387 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:21.385+0000 7f1434b9d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T12:27:21.466 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:21.464+0000 7f1434b9d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T12:27:21.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.478+0000 7fe534aaf700 1 -- 172.21.15.16:0/3706098309 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7fe51406b020 msgr2=0x7fe51406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/444334639 2023-10-16T12:27:21.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.478+0000 7fe534aaf700 1 --2- 172.21.15.16:0/3706098309 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7fe51406b020 0x7fe51406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:27:21.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.486+0000 7fe537514700 1 -- 172.21.15.16:0/3706098309 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe530067b20 con 0x7fe5300d2110 2023-10-16T12:27:21.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.486+0000 7fe525ffb700 1 -- 172.21.15.16:0/3706098309 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v100) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe52c0290a0 con 0x7fe5300d2110 2023-10-16T12:27:21.488 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T12:27:21.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.488+0000 7fe537514700 1 -- 172.21.15.16:0/3706098309 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7fe51406b020 msgr2=0x7fe51406d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:27:21.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.488+0000 7fe537514700 1 --2- 172.21.15.16:0/3706098309 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7fe51406b020 0x7fe51406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:21.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.488+0000 7fe537514700 1 -- 172.21.15.16:0/3706098309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5300d2110 msgr2=0x7fe5301f4c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:21.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.488+0000 7fe537514700 1 --2- 172.21.15.16:0/3706098309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5300d2110 0x7fe5301f4c20 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fe52c0090b0 tx=0x7fe52c004750 comp rx=0 tx=0).stop 2023-10-16T12:27:21.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.488+0000 7fe537514700 1 -- 172.21.15.16:0/3706098309 shutdown_connections 2023-10-16T12:27:21.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.488+0000 7fe537514700 1 --2- 172.21.15.16:0/3706098309 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe53015f060 0x7fe5300b6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:21.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.488+0000 7fe537514700 1 --2- 172.21.15.16:0/3706098309 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7fe51406b020 0x7fe51406d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:21.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.488+0000 7fe537514700 1 --2- 172.21.15.16:0/3706098309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5300d2110 0x7fe5301f4c20 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:21.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.488+0000 7fe537514700 1 --2- 172.21.15.16:0/3706098309 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe53015ec80 0x7fe530153760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:21.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.488+0000 7fe537514700 1 -- 172.21.15.16:0/3706098309 >> 172.21.15.16:0/3706098309 conn(0x7fe5300d15f0 msgr2=0x7fe5300b4320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:21.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.488+0000 7fe537514700 1 -- 172.21.15.16:0/3706098309 shutdown_connections 2023-10-16T12:27:21.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.488+0000 7fe537514700 1 -- 172.21.15.16:0/3706098309 wait complete. 2023-10-16T12:27:21.503 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T12:27:21.623 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:27:21.621+0000 7f1533820200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T12:27:21.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.634+0000 7f19f6884700 1 Processor -- start 2023-10-16T12:27:21.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.635+0000 7f19f6884700 1 -- start start 2023-10-16T12:27:21.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.635+0000 7f19f6884700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f19f0159de0 0x7f19f015a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:21.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.635+0000 7f19f6884700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f19f00cb1c0 0x7f19f00cb670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:21.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.635+0000 7f19f6884700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19f015dda0 con 0x7f19f00cf4c0 2023-10-16T12:27:21.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.635+0000 7f19f6884700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19f015ea20 con 0x7f19f00cb1c0 2023-10-16T12:27:21.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.635+0000 7f19f6884700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19f00cbbb0 con 0x7f19f0159de0 2023-10-16T12:27:21.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.636+0000 7f19effff700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f19f0159de0 0x7f19f015a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:21.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.636+0000 7f19f4e21700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f19f00cb1c0 0x7f19f00cb670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:21.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.636+0000 7f19effff700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f19f0159de0 0x7f19f015a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:58182/0 (socket says 172.21.15.16:58182) 2023-10-16T12:27:21.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.636+0000 7f19ef7fe700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7f19f00cf4c0 0x7f19f006aac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:38148/0 (socket says 172.21.15.16:38148) 2023-10-16T12:27:21.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.636+0000 7f19effff700 1 -- 172.21.15.16:0/3269629462 learned_addr learned my addr 172.21.15.16:0/3269629462 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:21.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.636+0000 7f19eeffd700 1 -- 172.21.15.16:0/3269629462 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2864551698 0 0) 0x7f19f015dda0 con 0x7f19f00cf4c0 2023-10-16T12:27:21.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.636+0000 7f19eeffd700 1 -- 172.21.15.16:0/3269629462 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19d80035b0 con 0x7f19f00cf4c0 2023-10-16T12:27:21.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.636+0000 7f19effff700 1 -- 172.21.15.16:0/3269629462 >> v2:172.21.15.49:3300/0 conn(0x7f19f00cb1c0 msgr2=0x7f19f00cb670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:21.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.636+0000 7f19effff700 1 --2- 172.21.15.16:0/3269629462 >> v2:172.21.15.49:3300/0 conn(0x7f19f00cb1c0 0x7f19f00cb670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:21.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.636+0000 7f19effff700 1 -- 172.21.15.16:0/3269629462 >> v1:172.21.15.49:6789/0 conn(0x7f19f00cf4c0 legacy=0x7f19f006aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:21.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.636+0000 7f19effff700 1 -- 172.21.15.16:0/3269629462 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19f015e080 con 0x7f19f0159de0 2023-10-16T12:27:21.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.636+0000 7f19f4e21700 1 --2- 172.21.15.16:0/3269629462 >> v2:172.21.15.49:3300/0 conn(0x7f19f00cb1c0 0x7f19f00cb670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:27:21.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.636+0000 7f19effff700 1 --2- 172.21.15.16:0/3269629462 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f19f0159de0 0x7f19f015a1c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f19e00098f0 tx=0x7f19e0009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c43fc7f4ddc543f7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:21.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.636+0000 7f19eeffd700 1 -- 172.21.15.16:0/3269629462 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e0011830 con 0x7f19f0159de0 2023-10-16T12:27:21.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.636+0000 7f19eeffd700 1 -- 172.21.15.16:0/3269629462 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19e0011990 con 0x7f19f0159de0 2023-10-16T12:27:21.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.636+0000 7f19eeffd700 1 -- 172.21.15.16:0/3269629462 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e0011d00 con 0x7f19f0159de0 2023-10-16T12:27:21.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.637+0000 7f19f6884700 1 -- 172.21.15.16:0/3269629462 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f19f0159de0 msgr2=0x7f19f015a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:21.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.637+0000 7f19f6884700 1 --2- 172.21.15.16:0/3269629462 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f19f0159de0 0x7f19f015a1c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f19e00098f0 tx=0x7f19e0009c00 comp rx=0 tx=0).stop 2023-10-16T12:27:21.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.637+0000 7f19f6884700 1 -- 172.21.15.16:0/3269629462 shutdown_connections 2023-10-16T12:27:21.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.637+0000 7f19f6884700 1 --2- 172.21.15.16:0/3269629462 >> v2:172.21.15.49:3300/0 conn(0x7f19f00cb1c0 0x7f19f00cb670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:21.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.637+0000 7f19f6884700 1 --2- 172.21.15.16:0/3269629462 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f19f0159de0 0x7f19f015a1c0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:21.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.637+0000 7f19f6884700 1 -- 172.21.15.16:0/3269629462 >> 172.21.15.16:0/3269629462 conn(0x7f19f00cc5e0 msgr2=0x7f19f00cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:21.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.637+0000 7f19f6884700 1 -- 172.21.15.16:0/3269629462 shutdown_connections 2023-10-16T12:27:21.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.637+0000 7f19f6884700 1 -- 172.21.15.16:0/3269629462 wait complete. 2023-10-16T12:27:21.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.639+0000 7f19f6884700 1 Processor -- start 2023-10-16T12:27:21.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.640+0000 7f19f6884700 1 -- start start 2023-10-16T12:27:21.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.641+0000 7f19f6884700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19f00cb1c0 0x7f19f015c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:21.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.641+0000 7f19f6884700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f19f00cf4c0 0x7f19f0151db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:21.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.641+0000 7f19f6884700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f19f0159de0 0x7f19f01522f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:21.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.641+0000 7f19f6884700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19f0146630 con 0x7f19f00cb1c0 2023-10-16T12:27:21.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.641+0000 7f19f6884700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19f015bd70 con 0x7f19f0159de0 2023-10-16T12:27:21.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.641+0000 7f19f6884700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19f015b7b0 con 0x7f19f00cf4c0 2023-10-16T12:27:21.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.641+0000 7f19effff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19f00cb1c0 0x7f19f015c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:21.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.641+0000 7f19effff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19f00cb1c0 0x7f19f015c1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41698/0 (socket says 172.21.15.16:41698) 2023-10-16T12:27:21.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.641+0000 7f19effff700 1 -- 172.21.15.16:0/1152833500 learned_addr learned my addr 172.21.15.16:0/1152833500 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:21.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.641+0000 7f19ef7fe700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f19f00cf4c0 0x7f19f0151db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:21.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.641+0000 7f19f4e21700 1 --2- 172.21.15.16:0/1152833500 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f19f0159de0 0x7f19f01522f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:21.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.641+0000 7f19f4e21700 1 -- 172.21.15.16:0/1152833500 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f19f00cf4c0 msgr2=0x7f19f0151db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:21.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.641+0000 7f19f4e21700 1 --2- 172.21.15.16:0/1152833500 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f19f00cf4c0 0x7f19f0151db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:21.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.641+0000 7f19f4e21700 1 -- 172.21.15.16:0/1152833500 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19f00cb1c0 msgr2=0x7f19f015c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:21.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.641+0000 7f19f4e21700 1 --2- 172.21.15.16:0/1152833500 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19f00cb1c0 0x7f19f015c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:21.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.641+0000 7f19f4e21700 1 -- 172.21.15.16:0/1152833500 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19e8009a70 con 0x7f19f0159de0 2023-10-16T12:27:21.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.641+0000 7f19effff700 1 --2- 172.21.15.16:0/1152833500 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19f00cb1c0 0x7f19f015c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:27:21.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.641+0000 7f19ef7fe700 1 --2- 172.21.15.16:0/1152833500 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f19f00cf4c0 0x7f19f0151db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:27:21.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.642+0000 7f19f4e21700 1 --2- 172.21.15.16:0/1152833500 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f19f0159de0 0x7f19f01522f0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f19e800efc0 tx=0x7f19e800c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:21.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.642+0000 7f19ecff9700 1 -- 172.21.15.16:0/1152833500 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e8009e80 con 0x7f19f0159de0 2023-10-16T12:27:21.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.642+0000 7f19ecff9700 1 -- 172.21.15.16:0/1152833500 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19e800f040 con 0x7f19f0159de0 2023-10-16T12:27:21.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.642+0000 7f19f6884700 1 -- 172.21.15.16:0/1152833500 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19e0009550 con 0x7f19f0159de0 2023-10-16T12:27:21.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.642+0000 7f19ecff9700 1 -- 172.21.15.16:0/1152833500 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e800ccd0 con 0x7f19f0159de0 2023-10-16T12:27:21.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.642+0000 7f19f6884700 1 -- 172.21.15.16:0/1152833500 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19f0151890 con 0x7f19f0159de0 2023-10-16T12:27:21.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.643+0000 7f19f6884700 1 -- 172.21.15.16:0/1152833500 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19f0066f10 con 0x7f19f0159de0 2023-10-16T12:27:21.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.643+0000 7f19ecff9700 1 -- 172.21.15.16:0/1152833500 <== mon.1 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 47) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f19e8003a20 con 0x7f19f0159de0 2023-10-16T12:27:21.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.644+0000 7f19ecff9700 1 --2- 172.21.15.16:0/1152833500 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7f19dc06b020 0x7f19dc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:21.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.644+0000 7f19effff700 1 -- 172.21.15.16:0/1152833500 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7f19dc06b020 msgr2=0x7f19dc06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/444334639 2023-10-16T12:27:21.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.644+0000 7f19ecff9700 1 -- 172.21.15.16:0/1152833500 <== mon.1 v2:172.21.15.49:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f19e808a3b0 con 0x7f19f0159de0 2023-10-16T12:27:21.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.644+0000 7f19effff700 1 --2- 172.21.15.16:0/1152833500 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7f19dc06b020 0x7f19dc06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:27:21.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.648+0000 7f19ecff9700 1 -- 172.21.15.16:0/1152833500 <== mon.1 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f19e8055060 con 0x7f19f0159de0 2023-10-16T12:27:21.753 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:21.751+0000 7f1434b9d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T12:27:21.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.845+0000 7f19effff700 1 -- 172.21.15.16:0/1152833500 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7f19dc06b020 msgr2=0x7f19dc06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/444334639 2023-10-16T12:27:21.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.845+0000 7f19effff700 1 --2- 172.21.15.16:0/1152833500 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7f19dc06b020 0x7f19dc06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:27:21.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.852+0000 7f19f6884700 1 -- 172.21.15.16:0/1152833500 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f19f0151ab0 con 0x7f19f0159de0 2023-10-16T12:27:21.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.852+0000 7f19ecff9700 1 -- 172.21.15.16:0/1152833500 <== mon.1 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v100) v1 ==== 104+0+11 (secure 0 0 0) 0x7f19e8058680 con 0x7f19f0159de0 2023-10-16T12:27:21.855 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-10-16T12:27:21.855 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:27:21.853+0000 7f1533820200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T12:27:21.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.855+0000 7f19f6884700 1 -- 172.21.15.16:0/1152833500 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7f19dc06b020 msgr2=0x7f19dc06d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:27:21.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.855+0000 7f19f6884700 1 --2- 172.21.15.16:0/1152833500 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7f19dc06b020 0x7f19dc06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:21.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.855+0000 7f19f6884700 1 -- 172.21.15.16:0/1152833500 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f19f0159de0 msgr2=0x7f19f01522f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:21.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.855+0000 7f19f6884700 1 --2- 172.21.15.16:0/1152833500 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f19f0159de0 0x7f19f01522f0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f19e800efc0 tx=0x7f19e800c3a0 comp rx=0 tx=0).stop 2023-10-16T12:27:21.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.855+0000 7f19f6884700 1 -- 172.21.15.16:0/1152833500 shutdown_connections 2023-10-16T12:27:21.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.855+0000 7f19f6884700 1 --2- 172.21.15.16:0/1152833500 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f19f0159de0 0x7f19f01522f0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:21.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.855+0000 7f19f6884700 1 --2- 172.21.15.16:0/1152833500 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7f19dc06b020 0x7f19dc06d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:21.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.855+0000 7f19f6884700 1 --2- 172.21.15.16:0/1152833500 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19f00cb1c0 0x7f19f015c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:21.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.855+0000 7f19f6884700 1 --2- 172.21.15.16:0/1152833500 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f19f00cf4c0 0x7f19f0151db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:21.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.855+0000 7f19f6884700 1 -- 172.21.15.16:0/1152833500 >> 172.21.15.16:0/1152833500 conn(0x7f19f00cc5e0 msgr2=0x7f19f006e3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:21.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.855+0000 7f19f6884700 1 -- 172.21.15.16:0/1152833500 shutdown_connections 2023-10-16T12:27:21.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.855+0000 7f19f6884700 1 -- 172.21.15.16:0/1152833500 wait complete. 2023-10-16T12:27:21.869 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T12:27:21.994 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:21.992+0000 7f1434b9d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T12:27:22.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.999+0000 7f887f5d5700 1 Processor -- start 2023-10-16T12:27:22.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:21.999+0000 7f887f5d5700 1 -- start start 2023-10-16T12:27:22.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.000+0000 7f887f5d5700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f887815f060 0x7f88780cc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:22.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.000+0000 7f887f5d5700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88780b2440 con 0x7f887815ec80 2023-10-16T12:27:22.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.000+0000 7f887f5d5700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88780d00e0 con 0x7f88780ce5d0 2023-10-16T12:27:22.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.000+0000 7f887f5d5700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88780b2c50 con 0x7f887815f060 2023-10-16T12:27:22.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.000+0000 7f887d371700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f887815ec80 0x7f88780cdec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59214/0 (socket says 172.21.15.16:59214) 2023-10-16T12:27:22.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.000+0000 7f887d371700 1 -- 172.21.15.16:0/3451498733 learned_addr learned my addr 172.21.15.16:0/3451498733 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:27:22.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.000+0000 7f887db72700 1 --2- 172.21.15.16:0/3451498733 >> v2:172.21.15.49:3300/0 conn(0x7f887815f060 0x7f88780cc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:22.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.001+0000 7f886ffff700 1 -- 172.21.15.16:0/3451498733 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1791749527 0 0) 0x7f88780b2440 con 0x7f887815ec80 2023-10-16T12:27:22.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.001+0000 7f886ffff700 1 -- 172.21.15.16:0/3451498733 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88640035b0 con 0x7f887815ec80 2023-10-16T12:27:22.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.001+0000 7f887db72700 1 -- 172.21.15.16:0/3451498733 >> v1:172.21.15.49:6789/0 conn(0x7f88780ce5d0 legacy=0x7f88780cc190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:22.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.001+0000 7f887db72700 1 -- 172.21.15.16:0/3451498733 >> v1:172.21.15.16:6789/0 conn(0x7f887815ec80 legacy=0x7f88780cdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:22.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.001+0000 7f887db72700 1 -- 172.21.15.16:0/3451498733 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88780cef60 con 0x7f887815f060 2023-10-16T12:27:22.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.001+0000 7f887db72700 1 --2- 172.21.15.16:0/3451498733 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f887815f060 0x7f88780cc8a0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f8870009dc0 tx=0x7f887000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=af1930a5012f3f7e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:22.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.001+0000 7f886ffff700 1 -- 172.21.15.16:0/3451498733 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f887000e070 con 0x7f887815f060 2023-10-16T12:27:22.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.001+0000 7f886ffff700 1 -- 172.21.15.16:0/3451498733 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f887000bae0 con 0x7f887815f060 2023-10-16T12:27:22.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.001+0000 7f886ffff700 1 -- 172.21.15.16:0/3451498733 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f887000be90 con 0x7f887815f060 2023-10-16T12:27:22.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.002+0000 7f887f5d5700 1 -- 172.21.15.16:0/3451498733 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f887815f060 msgr2=0x7f88780cc8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:22.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.002+0000 7f887f5d5700 1 --2- 172.21.15.16:0/3451498733 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f887815f060 0x7f88780cc8a0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f8870009dc0 tx=0x7f887000b330 comp rx=0 tx=0).stop 2023-10-16T12:27:22.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.002+0000 7f887f5d5700 1 -- 172.21.15.16:0/3451498733 shutdown_connections 2023-10-16T12:27:22.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.002+0000 7f887f5d5700 1 --2- 172.21.15.16:0/3451498733 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f887815f060 0x7f88780cc8a0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:22.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.002+0000 7f887f5d5700 1 -- 172.21.15.16:0/3451498733 >> 172.21.15.16:0/3451498733 conn(0x7f88780d15f0 msgr2=0x7f88780d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:22.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.002+0000 7f887f5d5700 1 -- 172.21.15.16:0/3451498733 shutdown_connections 2023-10-16T12:27:22.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.002+0000 7f887f5d5700 1 -- 172.21.15.16:0/3451498733 wait complete. 2023-10-16T12:27:22.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.003+0000 7f887f5d5700 1 Processor -- start 2023-10-16T12:27:22.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.003+0000 7f887f5d5700 1 -- start start 2023-10-16T12:27:22.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.004+0000 7f887f5d5700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f88780ce5d0 0x7f88780c7b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:22.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.004+0000 7f887f5d5700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f887815ec80 0x7f887806cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:22.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.004+0000 7f887f5d5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f887815f060 0x7f88780c6950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:22.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.004+0000 7f887f5d5700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f887806ba90 con 0x7f887815f060 2023-10-16T12:27:22.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.004+0000 7f887f5d5700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88780c8060 con 0x7f88780ce5d0 2023-10-16T12:27:22.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.004+0000 7f887f5d5700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88781f6dc0 con 0x7f887815ec80 2023-10-16T12:27:22.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.004+0000 7f887db72700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f887815f060 0x7f88780c6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:22.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.004+0000 7f887d371700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f88780ce5d0 0x7f88780c7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:22.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.004+0000 7f887db72700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f887815f060 0x7f88780c6950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41718/0 (socket says 172.21.15.16:41718) 2023-10-16T12:27:22.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.004+0000 7f887db72700 1 -- 172.21.15.16:0/392219862 learned_addr learned my addr 172.21.15.16:0/392219862 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:22.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.004+0000 7f887d371700 1 --2- 172.21.15.16:0/392219862 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f88780ce5d0 0x7f88780c7b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:40776/0 (socket says 172.21.15.16:40776) 2023-10-16T12:27:22.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.004+0000 7f887cb70700 1 --2- 172.21.15.16:0/392219862 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f887815ec80 0x7f887806cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:22.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.004+0000 7f887d371700 1 -- 172.21.15.16:0/392219862 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f887815ec80 msgr2=0x7f887806cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:22.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.004+0000 7f887d371700 1 --2- 172.21.15.16:0/392219862 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f887815ec80 0x7f887806cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:22.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.004+0000 7f887d371700 1 -- 172.21.15.16:0/392219862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f887815f060 msgr2=0x7f88780c6950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:22.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.004+0000 7f887d371700 1 --2- 172.21.15.16:0/392219862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f887815f060 0x7f88780c6950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:22.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.004+0000 7f887d371700 1 -- 172.21.15.16:0/392219862 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8870009a70 con 0x7f88780ce5d0 2023-10-16T12:27:22.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.004+0000 7f887db72700 1 --2- 172.21.15.16:0/392219862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f887815f060 0x7f88780c6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:27:22.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.005+0000 7f887d371700 1 --2- 172.21.15.16:0/392219862 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f88780ce5d0 0x7f88780c7b20 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f887400c490 tx=0x7f887400c7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:22.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.005+0000 7f886dffb700 1 -- 172.21.15.16:0/392219862 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8874005d70 con 0x7f88780ce5d0 2023-10-16T12:27:22.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.005+0000 7f886dffb700 1 -- 172.21.15.16:0/392219862 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88740070d0 con 0x7f88780ce5d0 2023-10-16T12:27:22.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.005+0000 7f886dffb700 1 -- 172.21.15.16:0/392219862 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8874007440 con 0x7f88780ce5d0 2023-10-16T12:27:22.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.005+0000 7f887f5d5700 1 -- 172.21.15.16:0/392219862 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88781f70a0 con 0x7f88780ce5d0 2023-10-16T12:27:22.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.005+0000 7f887f5d5700 1 -- 172.21.15.16:0/392219862 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f887806d3e0 con 0x7f88780ce5d0 2023-10-16T12:27:22.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.006+0000 7f887f5d5700 1 -- 172.21.15.16:0/392219862 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f887804ebb0 con 0x7f88780ce5d0 2023-10-16T12:27:22.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.006+0000 7f886dffb700 1 -- 172.21.15.16:0/392219862 <== mon.1 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 47) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f88740075a0 con 0x7f88780ce5d0 2023-10-16T12:27:22.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.007+0000 7f886dffb700 1 --2- 172.21.15.16:0/392219862 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7f885c06b020 0x7f885c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:22.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.007+0000 7f887cb70700 1 -- 172.21.15.16:0/392219862 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7f885c06b020 msgr2=0x7f885c06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/444334639 2023-10-16T12:27:22.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.007+0000 7f886dffb700 1 -- 172.21.15.16:0/392219862 <== mon.1 v2:172.21.15.49:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f8874088c70 con 0x7f88780ce5d0 2023-10-16T12:27:22.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.007+0000 7f887cb70700 1 --2- 172.21.15.16:0/392219862 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7f885c06b020 0x7f885c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:27:22.018 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:27:22.013+0000 7f1533820200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T12:27:22.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.016+0000 7f886dffb700 1 -- 172.21.15.16:0/392219862 <== mon.1 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8874053a00 con 0x7f88780ce5d0 2023-10-16T12:27:22.135 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:27:22.133+0000 7f1533820200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T12:27:22.165 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:22.163+0000 7f1434b9d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T12:27:22.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.208+0000 7f887cb70700 1 -- 172.21.15.16:0/392219862 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7f885c06b020 msgr2=0x7f885c06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/444334639 2023-10-16T12:27:22.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.208+0000 7f887cb70700 1 --2- 172.21.15.16:0/392219862 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7f885c06b020 0x7f885c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:27:22.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.230+0000 7f887f5d5700 1 -- 172.21.15.16:0/392219862 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f887806fd10 con 0x7f88780ce5d0 2023-10-16T12:27:22.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.230+0000 7f886dffb700 1 -- 172.21.15.16:0/392219862 <== mon.1 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v100) v1 ==== 100+0+11 (secure 0 0 0) 0x7f88740170a0 con 0x7f88780ce5d0 2023-10-16T12:27:22.233 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-10-16T12:27:22.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.233+0000 7f887f5d5700 1 -- 172.21.15.16:0/392219862 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7f885c06b020 msgr2=0x7f885c06d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:27:22.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.233+0000 7f887f5d5700 1 --2- 172.21.15.16:0/392219862 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7f885c06b020 0x7f885c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:22.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.233+0000 7f887f5d5700 1 -- 172.21.15.16:0/392219862 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f88780ce5d0 msgr2=0x7f88780c7b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:22.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.233+0000 7f887f5d5700 1 --2- 172.21.15.16:0/392219862 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f88780ce5d0 0x7f88780c7b20 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f887400c490 tx=0x7f887400c7a0 comp rx=0 tx=0).stop 2023-10-16T12:27:22.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.233+0000 7f887f5d5700 1 -- 172.21.15.16:0/392219862 shutdown_connections 2023-10-16T12:27:22.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.233+0000 7f887f5d5700 1 --2- 172.21.15.16:0/392219862 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7f885c06b020 0x7f885c06d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:22.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.233+0000 7f887f5d5700 1 --2- 172.21.15.16:0/392219862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f887815f060 0x7f88780c6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:22.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.233+0000 7f887f5d5700 1 --2- 172.21.15.16:0/392219862 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f88780ce5d0 0x7f88780c7b20 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:22.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.233+0000 7f887f5d5700 1 --2- 172.21.15.16:0/392219862 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f887815ec80 0x7f887806cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:22.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.233+0000 7f887f5d5700 1 -- 172.21.15.16:0/392219862 >> 172.21.15.16:0/392219862 conn(0x7f88780d15f0 msgr2=0x7f88780cd3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:22.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.233+0000 7f887f5d5700 1 -- 172.21.15.16:0/392219862 shutdown_connections 2023-10-16T12:27:22.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.233+0000 7f887f5d5700 1 -- 172.21.15.16:0/392219862 wait complete. 2023-10-16T12:27:22.248 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:27:22.289 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:22.287+0000 7f1434b9d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T12:27:22.298 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:27:22.296+0000 7f1533820200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T12:27:22.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.372+0000 7fb9891d1700 1 Processor -- start 2023-10-16T12:27:22.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.372+0000 7fb9891d1700 1 -- start start 2023-10-16T12:27:22.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.373+0000 7fb9891d1700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb9840b2b50 0x7fb9840d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:22.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.373+0000 7fb9891d1700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb9840ce380 0x7fb9840d09e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:22.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.373+0000 7fb9891d1700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9840d1210 con 0x7fb9840cdfa0 2023-10-16T12:27:22.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.373+0000 7fb9891d1700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9840cbc30 con 0x7fb9840ce380 2023-10-16T12:27:22.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.373+0000 7fb9891d1700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9840cbf10 con 0x7fb9840b2b50 2023-10-16T12:27:22.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.373+0000 7fb98359e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb9840ce380 0x7fb9840d09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:22.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.373+0000 7fb98359e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb9840ce380 0x7fb9840d09e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41728/0 (socket says 172.21.15.16:41728) 2023-10-16T12:27:22.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.373+0000 7fb98359e700 1 -- 172.21.15.16:0/4256820178 learned_addr learned my addr 172.21.15.16:0/4256820178 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:22.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.373+0000 7fb98259c700 1 --2- 172.21.15.16:0/4256820178 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb9840b2b50 0x7fb9840d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:22.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.373+0000 7fb98359e700 1 -- 172.21.15.16:0/4256820178 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb9840b2b50 msgr2=0x7fb9840d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:22.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.373+0000 7fb98359e700 1 --2- 172.21.15.16:0/4256820178 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb9840b2b50 0x7fb9840d04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:22.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.373+0000 7fb98359e700 1 -- 172.21.15.16:0/4256820178 >> v1:172.21.15.49:6789/0 conn(0x7fb9840cdfa0 legacy=0x7fb9840b2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:22.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.373+0000 7fb98359e700 1 -- 172.21.15.16:0/4256820178 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9840cc1f0 con 0x7fb9840ce380 2023-10-16T12:27:22.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.373+0000 7fb98259c700 1 --2- 172.21.15.16:0/4256820178 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb9840b2b50 0x7fb9840d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:27:22.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.373+0000 7fb98359e700 1 --2- 172.21.15.16:0/4256820178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9840ce380 0x7fb9840d09e0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7fb97c009e50 tx=0x7fb97c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce35ae6693c64123 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:22.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.373+0000 7fb981d9b700 1 -- 172.21.15.16:0/4256820178 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb97c00e070 con 0x7fb9840ce380 2023-10-16T12:27:22.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.374+0000 7fb981d9b700 1 -- 172.21.15.16:0/4256820178 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb97c00b9c0 con 0x7fb9840ce380 2023-10-16T12:27:22.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.374+0000 7fb981d9b700 1 -- 172.21.15.16:0/4256820178 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb97c00bd70 con 0x7fb9840ce380 2023-10-16T12:27:22.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.374+0000 7fb9891d1700 1 -- 172.21.15.16:0/4256820178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9840ce380 msgr2=0x7fb9840d09e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:22.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.374+0000 7fb9891d1700 1 --2- 172.21.15.16:0/4256820178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9840ce380 0x7fb9840d09e0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7fb97c009e50 tx=0x7fb97c00b330 comp rx=0 tx=0).stop 2023-10-16T12:27:22.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.374+0000 7fb9891d1700 1 -- 172.21.15.16:0/4256820178 shutdown_connections 2023-10-16T12:27:22.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.374+0000 7fb9891d1700 1 --2- 172.21.15.16:0/4256820178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9840ce380 0x7fb9840d09e0 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:22.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.374+0000 7fb9891d1700 1 --2- 172.21.15.16:0/4256820178 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb9840b2b50 0x7fb9840d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:22.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.374+0000 7fb9891d1700 1 -- 172.21.15.16:0/4256820178 >> 172.21.15.16:0/4256820178 conn(0x7fb9840d14b0 msgr2=0x7fb9840d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:22.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.374+0000 7fb9891d1700 1 -- 172.21.15.16:0/4256820178 shutdown_connections 2023-10-16T12:27:22.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.374+0000 7fb9891d1700 1 -- 172.21.15.16:0/4256820178 wait complete. 2023-10-16T12:27:22.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.376+0000 7fb9891d1700 1 Processor -- start 2023-10-16T12:27:22.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.376+0000 7fb9891d1700 1 -- start start 2023-10-16T12:27:22.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.377+0000 7fb9891d1700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb9840b2b50 0x7fb9841f2af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:22.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.377+0000 7fb9891d1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9840cdfa0 0x7fb9840c6680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:22.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.377+0000 7fb9891d1700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb9840ce380 0x7fb98406b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:22.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.377+0000 7fb9891d1700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9840c55e0 con 0x7fb9840cdfa0 2023-10-16T12:27:22.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.377+0000 7fb9891d1700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9840c58c0 con 0x7fb9840ce380 2023-10-16T12:27:22.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.377+0000 7fb9891d1700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9840c5ba0 con 0x7fb9840b2b50 2023-10-16T12:27:22.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.377+0000 7fb98259c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9840cdfa0 0x7fb9840c6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:22.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.377+0000 7fb98259c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9840cdfa0 0x7fb9840c6680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41732/0 (socket says 172.21.15.16:41732) 2023-10-16T12:27:22.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.377+0000 7fb98259c700 1 -- 172.21.15.16:0/2269342244 learned_addr learned my addr 172.21.15.16:0/2269342244 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:22.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.377+0000 7fb98359e700 1 --2- 172.21.15.16:0/2269342244 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb9840ce380 0x7fb98406b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:22.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.377+0000 7fb982d9d700 1 --2- 172.21.15.16:0/2269342244 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb9840b2b50 0x7fb9841f2af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:22.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.377+0000 7fb98259c700 1 -- 172.21.15.16:0/2269342244 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb9840b2b50 msgr2=0x7fb9841f2af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:22.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.377+0000 7fb98259c700 1 --2- 172.21.15.16:0/2269342244 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb9840b2b50 0x7fb9841f2af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:22.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.377+0000 7fb98259c700 1 -- 172.21.15.16:0/2269342244 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb9840ce380 msgr2=0x7fb98406b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:22.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.377+0000 7fb98259c700 1 --2- 172.21.15.16:0/2269342244 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb9840ce380 0x7fb98406b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:22.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.377+0000 7fb98259c700 1 -- 172.21.15.16:0/2269342244 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb97c009b30 con 0x7fb9840cdfa0 2023-10-16T12:27:22.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.378+0000 7fb98259c700 1 --2- 172.21.15.16:0/2269342244 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9840cdfa0 0x7fb9840c6680 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7fb97800e7e0 tx=0x7fb97800eaf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:22.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.378+0000 7fb96b7fe700 1 -- 172.21.15.16:0/2269342244 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb97800c070 con 0x7fb9840cdfa0 2023-10-16T12:27:22.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.378+0000 7fb96b7fe700 1 -- 172.21.15.16:0/2269342244 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb978004230 con 0x7fb9840cdfa0 2023-10-16T12:27:22.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.378+0000 7fb9891d1700 1 -- 172.21.15.16:0/2269342244 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9840c5e80 con 0x7fb9840cdfa0 2023-10-16T12:27:22.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.378+0000 7fb96b7fe700 1 -- 172.21.15.16:0/2269342244 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9780045a0 con 0x7fb9840cdfa0 2023-10-16T12:27:22.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.378+0000 7fb9891d1700 1 -- 172.21.15.16:0/2269342244 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9840c6bc0 con 0x7fb9840cdfa0 2023-10-16T12:27:22.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.379+0000 7fb9891d1700 1 -- 172.21.15.16:0/2269342244 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb98404ebb0 con 0x7fb9840cdfa0 2023-10-16T12:27:22.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.379+0000 7fb96b7fe700 1 -- 172.21.15.16:0/2269342244 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 47) v1 ==== 89332+0+0 (secure 0 0 0) 0x7fb9780036a0 con 0x7fb9840cdfa0 2023-10-16T12:27:22.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.380+0000 7fb96b7fe700 1 --2- 172.21.15.16:0/2269342244 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7fb97006b020 0x7fb97006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:22.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.380+0000 7fb982d9d700 1 -- 172.21.15.16:0/2269342244 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7fb97006b020 msgr2=0x7fb97006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/444334639 2023-10-16T12:27:22.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.380+0000 7fb96b7fe700 1 -- 172.21.15.16:0/2269342244 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fb9780897d0 con 0x7fb9840cdfa0 2023-10-16T12:27:22.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.380+0000 7fb982d9d700 1 --2- 172.21.15.16:0/2269342244 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7fb97006b020 0x7fb97006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:27:22.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.384+0000 7fb96b7fe700 1 -- 172.21.15.16:0/2269342244 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb9780543b0 con 0x7fb9840cdfa0 2023-10-16T12:27:22.460 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:22.458+0000 7f1434b9d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T12:27:22.570 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:27:22.567+0000 7f1533820200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T12:27:22.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.581+0000 7fb982d9d700 1 -- 172.21.15.16:0/2269342244 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7fb97006b020 msgr2=0x7fb97006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/444334639 2023-10-16T12:27:22.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.581+0000 7fb982d9d700 1 --2- 172.21.15.16:0/2269342244 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7fb97006b020 0x7fb97006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:27:22.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.635+0000 7fb9891d1700 1 -- 172.21.15.16:0/2269342244 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb9840c5290 con 0x7fb9840cdfa0 2023-10-16T12:27:22.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.636+0000 7fb96b7fe700 1 -- 172.21.15.16:0/2269342244 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+4915 (secure 0 0 0) 0x7fb978087e60 con 0x7fb9840cdfa0 2023-10-16T12:27:22.638 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:27:22.638 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":44,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6820,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/1256919442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":1256919442},{"type":"v1","addr":"172.21.15.49:6834","nonce":1256919442}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":6823,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/3811815346","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":3811815346},{"type":"v1","addr":"172.21.15.49:6835","nonce":3811815346}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":7110,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/1226064755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":1226064755},{"type":"v1","addr":"172.21.15.16:6836","nonce":1226064755}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:27:00.825934+0000","modified":"2023-10-16T12:27:03.831266+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7113},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7113":{"gid":7113,"name":"a","rank":0,"incarnation":40,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/1843321005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1843321005},{"type":"v1","addr":"172.21.15.16:6837","nonce":1843321005}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-10-16T12:27:22.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.638+0000 7fb9891d1700 1 -- 172.21.15.16:0/2269342244 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7fb97006b020 msgr2=0x7fb97006d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:27:22.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.638+0000 7fb9891d1700 1 --2- 172.21.15.16:0/2269342244 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7fb97006b020 0x7fb97006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:22.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.638+0000 7fb9891d1700 1 -- 172.21.15.16:0/2269342244 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9840cdfa0 msgr2=0x7fb9840c6680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:22.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.638+0000 7fb9891d1700 1 --2- 172.21.15.16:0/2269342244 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9840cdfa0 0x7fb9840c6680 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7fb97800e7e0 tx=0x7fb97800eaf0 comp rx=0 tx=0).stop 2023-10-16T12:27:22.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.638+0000 7fb9891d1700 1 -- 172.21.15.16:0/2269342244 shutdown_connections 2023-10-16T12:27:22.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.638+0000 7fb9891d1700 1 --2- 172.21.15.16:0/2269342244 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb9840ce380 0x7fb98406b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:22.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.638+0000 7fb9891d1700 1 --2- 172.21.15.16:0/2269342244 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb9840b2b50 0x7fb9841f2af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:22.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.638+0000 7fb9891d1700 1 --2- 172.21.15.16:0/2269342244 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7fb97006b020 0x7fb97006d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:22.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.638+0000 7fb9891d1700 1 --2- 172.21.15.16:0/2269342244 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9840cdfa0 0x7fb9840c6680 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:22.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.638+0000 7fb9891d1700 1 -- 172.21.15.16:0/2269342244 >> 172.21.15.16:0/2269342244 conn(0x7fb9840d14b0 msgr2=0x7fb9840d1890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:22.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.638+0000 7fb9891d1700 1 -- 172.21.15.16:0/2269342244 shutdown_connections 2023-10-16T12:27:22.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.638+0000 7fb9891d1700 1 -- 172.21.15.16:0/2269342244 wait complete. 2023-10-16T12:27:22.643 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 44 2023-10-16T12:27:22.653 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:27:22.735 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:27:22.734+0000 7f1533820200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T12:27:22.738 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:22.736+0000 7f1434b9d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T12:27:22.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.819+0000 7f77a4a10700 1 Processor -- start 2023-10-16T12:27:22.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.819+0000 7f77a4a10700 1 -- start start 2023-10-16T12:27:22.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.820+0000 7f77a4a10700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f77a015b630 0x7f77a00d0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:22.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.821+0000 7f77a4a10700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f77a00ce4f0 0x7f77a00cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:22.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.821+0000 7f77a4a10700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77a00b0970 con 0x7f77a015ba10 2023-10-16T12:27:22.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.821+0000 7f77a4a10700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77a00cd690 con 0x7f77a00ce4f0 2023-10-16T12:27:22.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.821+0000 7f77a4a10700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77a00cd970 con 0x7f77a015b630 2023-10-16T12:27:22.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.821+0000 7f779ed9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f77a00ce4f0 0x7f77a00cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:22.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.821+0000 7f779ed9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f77a00ce4f0 0x7f77a00cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41744/0 (socket says 172.21.15.16:41744) 2023-10-16T12:27:22.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.821+0000 7f779ed9d700 1 -- 172.21.15.16:0/3623909328 learned_addr learned my addr 172.21.15.16:0/3623909328 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:22.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.821+0000 7f779e59c700 1 --2- 172.21.15.16:0/3623909328 >> v2:172.21.15.49:3300/0 conn(0x7f77a015b630 0x7f77a00d0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:22.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.821+0000 7f779ed9d700 1 -- 172.21.15.16:0/3623909328 >> v2:172.21.15.49:3300/0 conn(0x7f77a015b630 msgr2=0x7f77a00d0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:22.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.821+0000 7f779ed9d700 1 --2- 172.21.15.16:0/3623909328 >> v2:172.21.15.49:3300/0 conn(0x7f77a015b630 0x7f77a00d0380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:22.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.821+0000 7f779ed9d700 1 -- 172.21.15.16:0/3623909328 >> v1:172.21.15.49:6789/0 conn(0x7f77a015ba10 legacy=0x7f77a00cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:22.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.821+0000 7f779ed9d700 1 -- 172.21.15.16:0/3623909328 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77a00b0c30 con 0x7f77a00ce4f0 2023-10-16T12:27:22.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.821+0000 7f779ed9d700 1 --2- 172.21.15.16:0/3623909328 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77a00ce4f0 0x7f77a00cf290 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f77a0067350 tx=0x7f779800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0afde55119e02c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:22.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.821+0000 7f779d59a700 1 -- 172.21.15.16:0/3623909328 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f779800e070 con 0x7f77a00ce4f0 2023-10-16T12:27:22.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.821+0000 7f779d59a700 1 -- 172.21.15.16:0/3623909328 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f779800b910 con 0x7f77a00ce4f0 2023-10-16T12:27:22.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.821+0000 7f779d59a700 1 -- 172.21.15.16:0/3623909328 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f779800bcc0 con 0x7f77a00ce4f0 2023-10-16T12:27:22.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.822+0000 7f77a4a10700 1 -- 172.21.15.16:0/3623909328 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77a00ce4f0 msgr2=0x7f77a00cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:22.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.822+0000 7f77a4a10700 1 --2- 172.21.15.16:0/3623909328 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77a00ce4f0 0x7f77a00cf290 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f77a0067350 tx=0x7f779800b330 comp rx=0 tx=0).stop 2023-10-16T12:27:22.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.822+0000 7f77a4a10700 1 -- 172.21.15.16:0/3623909328 shutdown_connections 2023-10-16T12:27:22.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.822+0000 7f77a4a10700 1 --2- 172.21.15.16:0/3623909328 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77a00ce4f0 0x7f77a00cf290 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:22.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.822+0000 7f77a4a10700 1 --2- 172.21.15.16:0/3623909328 >> v2:172.21.15.49:3300/0 conn(0x7f77a015b630 0x7f77a00d0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:22.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.822+0000 7f77a4a10700 1 -- 172.21.15.16:0/3623909328 >> 172.21.15.16:0/3623909328 conn(0x7f77a00d14b0 msgr2=0x7f77a00d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:22.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.822+0000 7f77a4a10700 1 -- 172.21.15.16:0/3623909328 shutdown_connections 2023-10-16T12:27:22.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.822+0000 7f77a4a10700 1 -- 172.21.15.16:0/3623909328 wait complete. 2023-10-16T12:27:22.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.824+0000 7f77a4a10700 1 Processor -- start 2023-10-16T12:27:22.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.824+0000 7f77a4a10700 1 -- start start 2023-10-16T12:27:22.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.825+0000 7f77a4a10700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f77a00ce4f0 0x7f77a01f4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:22.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.825+0000 7f77a4a10700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77a015b630 0x7f77a01f1540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:22.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.825+0000 7f77a4a10700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f77a015ba10 0x7f77a0150810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:22.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.825+0000 7f77a4a10700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77a01f1a80 con 0x7f77a015b630 2023-10-16T12:27:22.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.825+0000 7f77a4a10700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77a01f40b0 con 0x7f77a00ce4f0 2023-10-16T12:27:22.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.825+0000 7f77a4a10700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77a01f4670 con 0x7f77a015ba10 2023-10-16T12:27:22.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.825+0000 7f779dd9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77a015b630 0x7f77a01f1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:22.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.825+0000 7f779dd9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77a015b630 0x7f77a01f1540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41748/0 (socket says 172.21.15.16:41748) 2023-10-16T12:27:22.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.825+0000 7f779dd9b700 1 -- 172.21.15.16:0/2265480403 learned_addr learned my addr 172.21.15.16:0/2265480403 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:22.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.825+0000 7f779e59c700 1 --2- 172.21.15.16:0/2265480403 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f77a00ce4f0 0x7f77a01f4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:22.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.825+0000 7f779ed9d700 1 --2- 172.21.15.16:0/2265480403 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f77a015ba10 0x7f77a0150810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:22.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.825+0000 7f779dd9b700 1 -- 172.21.15.16:0/2265480403 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f77a015ba10 msgr2=0x7f77a0150810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:22.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.825+0000 7f779dd9b700 1 --2- 172.21.15.16:0/2265480403 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f77a015ba10 0x7f77a0150810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:22.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.825+0000 7f779dd9b700 1 -- 172.21.15.16:0/2265480403 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f77a00ce4f0 msgr2=0x7f77a01f4b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:22.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.825+0000 7f779dd9b700 1 --2- 172.21.15.16:0/2265480403 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f77a00ce4f0 0x7f77a01f4b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:22.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.825+0000 7f779dd9b700 1 -- 172.21.15.16:0/2265480403 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7798009b30 con 0x7f77a015b630 2023-10-16T12:27:22.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.826+0000 7f779dd9b700 1 --2- 172.21.15.16:0/2265480403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77a015b630 0x7f77a01f1540 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f779400c450 tx=0x7f779400c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:22.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.826+0000 7f778effd700 1 -- 172.21.15.16:0/2265480403 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77940050d0 con 0x7f77a015b630 2023-10-16T12:27:22.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.826+0000 7f778effd700 1 -- 172.21.15.16:0/2265480403 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7794005230 con 0x7f77a015b630 2023-10-16T12:27:22.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.826+0000 7f77a4a10700 1 -- 172.21.15.16:0/2265480403 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77a0150d50 con 0x7f77a015b630 2023-10-16T12:27:22.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.826+0000 7f778effd700 1 -- 172.21.15.16:0/2265480403 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7794003680 con 0x7f77a015b630 2023-10-16T12:27:22.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.826+0000 7f77a4a10700 1 -- 172.21.15.16:0/2265480403 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77a0159d70 con 0x7f77a015b630 2023-10-16T12:27:22.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.827+0000 7f77a4a10700 1 -- 172.21.15.16:0/2265480403 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77a004ebb0 con 0x7f77a015b630 2023-10-16T12:27:22.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.827+0000 7f778effd700 1 -- 172.21.15.16:0/2265480403 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 47) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f77940037e0 con 0x7f77a015b630 2023-10-16T12:27:22.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.828+0000 7f778effd700 1 --2- 172.21.15.16:0/2265480403 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7f778006b020 0x7f778006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:22.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.828+0000 7f779e59c700 1 -- 172.21.15.16:0/2265480403 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7f778006b020 msgr2=0x7f778006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/444334639 2023-10-16T12:27:22.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.828+0000 7f778effd700 1 -- 172.21.15.16:0/2265480403 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f7794088c30 con 0x7f77a015b630 2023-10-16T12:27:22.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.828+0000 7f779e59c700 1 --2- 172.21.15.16:0/2265480403 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7f778006b020 0x7f778006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:27:22.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:22.832+0000 7f778effd700 1 -- 172.21.15.16:0/2265480403 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f77940539a0 con 0x7f77a015b630 2023-10-16T12:27:22.915 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:22.913+0000 7f1434b9d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T12:27:23.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.029+0000 7f779e59c700 1 -- 172.21.15.16:0/2265480403 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7f778006b020 msgr2=0x7f778006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/444334639 2023-10-16T12:27:23.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.029+0000 7f779e59c700 1 --2- 172.21.15.16:0/2265480403 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7f778006b020 0x7f778006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:27:23.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.037+0000 7f77a4a10700 1 -- 172.21.15.16:0/2265480403 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f77a006b5d0 con 0x7f77a015b630 2023-10-16T12:27:23.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.038+0000 7f778effd700 1 -- 172.21.15.16:0/2265480403 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) v1 ==== 74+0+17211 (secure 0 0 0) 0x7f7794017090 con 0x7f77a015b630 2023-10-16T12:27:23.041 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:27:23.041 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":100,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:27:15.899863+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:26:59.206630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2023-10-16T12:26:59.784585+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6833/292891612":"2023-10-17T12:27:15.899818+0000","172.21.15.16:6832/292891612":"2023-10-17T12:27:15.899818+0000","172.21.15.16:0/1020789692":"2023-10-17T12:27:15.899818+0000","172.21.15.16:0/478878390":"2023-10-17T12:27:15.899818+0000","172.21.15.16:0/2293822331":"2023-10-17T12:27:15.899818+0000","172.21.15.16:0/1305507650":"2023-10-17T12:27:15.899818+0000","172.21.15.16:0/3205233425":"2023-10-17T12:27:15.899818+0000","172.21.15.16:0/2477042442":"2023-10-17T12:27:15.899818+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:27:23.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.040+0000 7f77a4a10700 1 -- 172.21.15.16:0/2265480403 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7f778006b020 msgr2=0x7f778006d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:27:23.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.040+0000 7f77a4a10700 1 --2- 172.21.15.16:0/2265480403 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7f778006b020 0x7f778006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:23.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.041+0000 7f77a4a10700 1 -- 172.21.15.16:0/2265480403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77a015b630 msgr2=0x7f77a01f1540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:23.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.041+0000 7f77a4a10700 1 --2- 172.21.15.16:0/2265480403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77a015b630 0x7f77a01f1540 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f779400c450 tx=0x7f779400c760 comp rx=0 tx=0).stop 2023-10-16T12:27:23.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.041+0000 7f77a4a10700 1 -- 172.21.15.16:0/2265480403 shutdown_connections 2023-10-16T12:27:23.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.041+0000 7f77a4a10700 1 --2- 172.21.15.16:0/2265480403 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f77a015ba10 0x7f77a0150810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:23.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.041+0000 7f77a4a10700 1 --2- 172.21.15.16:0/2265480403 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f77a00ce4f0 0x7f77a01f4b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:23.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.041+0000 7f77a4a10700 1 --2- 172.21.15.16:0/2265480403 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7f778006b020 0x7f778006d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:23.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.041+0000 7f77a4a10700 1 --2- 172.21.15.16:0/2265480403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77a015b630 0x7f77a01f1540 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:23.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.041+0000 7f77a4a10700 1 -- 172.21.15.16:0/2265480403 >> 172.21.15.16:0/2265480403 conn(0x7f77a00d14b0 msgr2=0x7f77a00d1890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:23.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.041+0000 7f77a4a10700 1 -- 172.21.15.16:0/2265480403 shutdown_connections 2023-10-16T12:27:23.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.041+0000 7f77a4a10700 1 -- 172.21.15.16:0/2265480403 wait complete. 2023-10-16T12:27:23.056 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-16T12:27:23.056 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:27:23.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.180+0000 7ff0bb749700 1 Processor -- start 2023-10-16T12:27:23.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.180+0000 7ff0bb749700 1 -- start start 2023-10-16T12:27:23.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.181+0000 7ff0bb749700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7ff0b415d900 0x7ff0b40d2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:23.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.181+0000 7ff0bb749700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff0b40ce9f0 0x7ff0b40cdc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:23.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.181+0000 7ff0bb749700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0b40ce1a0 con 0x7ff0b40d2860 2023-10-16T12:27:23.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.181+0000 7ff0bb749700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0b40cee70 con 0x7ff0b415d900 2023-10-16T12:27:23.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.181+0000 7ff0bb749700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b40ce400 con 0x7ff0b40ce9f0 2023-10-16T12:27:23.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.181+0000 7ff0b9ce6700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff0b40d2860 0x7ff0b40b5120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59266/0 (socket says 172.21.15.16:59266) 2023-10-16T12:27:23.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.181+0000 7ff0b9ce6700 1 -- 172.21.15.16:0/219623546 learned_addr learned my addr 172.21.15.16:0/219623546 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:27:23.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.181+0000 7ff0b94e5700 1 --2- 172.21.15.16:0/219623546 >> v2:172.21.15.49:3300/0 conn(0x7ff0b415d900 0x7ff0b40d2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:23.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.181+0000 7ff0b8ce4700 1 --2- 172.21.15.16:0/219623546 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff0b40ce9f0 0x7ff0b40cdc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:23.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.182+0000 7ff0abfff700 1 -- 172.21.15.16:0/219623546 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2784528303 0 0) 0x7ff0b40ce1a0 con 0x7ff0b40d2860 2023-10-16T12:27:23.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.182+0000 7ff0abfff700 1 -- 172.21.15.16:0/219623546 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0a00035b0 con 0x7ff0b40d2860 2023-10-16T12:27:23.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.182+0000 7ff0b94e5700 1 -- 172.21.15.16:0/219623546 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff0b40ce9f0 msgr2=0x7ff0b40cdc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:23.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.182+0000 7ff0b94e5700 1 --2- 172.21.15.16:0/219623546 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff0b40ce9f0 0x7ff0b40cdc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:23.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.182+0000 7ff0b94e5700 1 -- 172.21.15.16:0/219623546 >> v1:172.21.15.16:6789/0 conn(0x7ff0b40d2860 legacy=0x7ff0b40b5120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:23.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.182+0000 7ff0b94e5700 1 -- 172.21.15.16:0/219623546 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0b415dce0 con 0x7ff0b415d900 2023-10-16T12:27:23.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.182+0000 7ff0b8ce4700 1 --2- 172.21.15.16:0/219623546 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff0b40ce9f0 0x7ff0b40cdc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:27:23.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.182+0000 7ff0b94e5700 1 --2- 172.21.15.16:0/219623546 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff0b415d900 0x7ff0b40d2040 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7ff0b0009840 tx=0x7ff0b0009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ab19ccdd294d0b94 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:23.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.182+0000 7ff0abfff700 1 -- 172.21.15.16:0/219623546 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0b0011830 con 0x7ff0b415d900 2023-10-16T12:27:23.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.182+0000 7ff0abfff700 1 -- 172.21.15.16:0/219623546 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0b0011990 con 0x7ff0b415d900 2023-10-16T12:27:23.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.182+0000 7ff0abfff700 1 -- 172.21.15.16:0/219623546 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0b0011d20 con 0x7ff0b415d900 2023-10-16T12:27:23.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.183+0000 7ff0bb749700 1 -- 172.21.15.16:0/219623546 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff0b415d900 msgr2=0x7ff0b40d2040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:23.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.183+0000 7ff0bb749700 1 --2- 172.21.15.16:0/219623546 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff0b415d900 0x7ff0b40d2040 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7ff0b0009840 tx=0x7ff0b0009b50 comp rx=0 tx=0).stop 2023-10-16T12:27:23.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.183+0000 7ff0bb749700 1 -- 172.21.15.16:0/219623546 shutdown_connections 2023-10-16T12:27:23.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.183+0000 7ff0bb749700 1 --2- 172.21.15.16:0/219623546 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff0b415d900 0x7ff0b40d2040 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:23.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.183+0000 7ff0bb749700 1 --2- 172.21.15.16:0/219623546 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff0b40ce9f0 0x7ff0b40cdc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:23.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.183+0000 7ff0bb749700 1 -- 172.21.15.16:0/219623546 >> 172.21.15.16:0/219623546 conn(0x7ff0b4072330 msgr2=0x7ff0b4072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:23.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.183+0000 7ff0bb749700 1 -- 172.21.15.16:0/219623546 shutdown_connections 2023-10-16T12:27:23.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.183+0000 7ff0bb749700 1 -- 172.21.15.16:0/219623546 wait complete. 2023-10-16T12:27:23.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.184+0000 7ff0bb749700 1 Processor -- start 2023-10-16T12:27:23.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.184+0000 7ff0bb749700 1 -- start start 2023-10-16T12:27:23.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.185+0000 7ff0bb749700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff0b40ce9f0 0x7ff0b41f5c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:23.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.185+0000 7ff0bb749700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff0b40d2860 0x7ff0b415ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:23.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.185+0000 7ff0bb749700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0b415d900 0x7ff0b414df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:23.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.185+0000 7ff0bb749700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b415e2d0 con 0x7ff0b415d900 2023-10-16T12:27:23.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.185+0000 7ff0bb749700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b414ced0 con 0x7ff0b40d2860 2023-10-16T12:27:23.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.185+0000 7ff0bb749700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b414d1b0 con 0x7ff0b40ce9f0 2023-10-16T12:27:23.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.185+0000 7ff0b9ce6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0b415d900 0x7ff0b414df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:23.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.185+0000 7ff0b9ce6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0b415d900 0x7ff0b414df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41766/0 (socket says 172.21.15.16:41766) 2023-10-16T12:27:23.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.185+0000 7ff0b9ce6700 1 -- 172.21.15.16:0/1414056116 learned_addr learned my addr 172.21.15.16:0/1414056116 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:23.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.185+0000 7ff0b8ce4700 1 --2- 172.21.15.16:0/1414056116 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff0b40d2860 0x7ff0b415ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:23.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.185+0000 7ff0b94e5700 1 --2- 172.21.15.16:0/1414056116 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff0b40ce9f0 0x7ff0b41f5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:23.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.185+0000 7ff0b9ce6700 1 -- 172.21.15.16:0/1414056116 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff0b40ce9f0 msgr2=0x7ff0b41f5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:23.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.185+0000 7ff0b9ce6700 1 --2- 172.21.15.16:0/1414056116 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff0b40ce9f0 0x7ff0b41f5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:23.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.185+0000 7ff0b9ce6700 1 -- 172.21.15.16:0/1414056116 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff0b40d2860 msgr2=0x7ff0b415ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:23.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.185+0000 7ff0b9ce6700 1 --2- 172.21.15.16:0/1414056116 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff0b40d2860 0x7ff0b415ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:23.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.185+0000 7ff0b9ce6700 1 -- 172.21.15.16:0/1414056116 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0a40094a0 con 0x7ff0b415d900 2023-10-16T12:27:23.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.185+0000 7ff0b9ce6700 1 --2- 172.21.15.16:0/1414056116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0b415d900 0x7ff0b414df70 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7ff0ac00cb40 tx=0x7ff0ac00ce50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:23.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.185+0000 7ff0a9ffb700 1 -- 172.21.15.16:0/1414056116 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0ac00a9e0 con 0x7ff0b415d900 2023-10-16T12:27:23.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.186+0000 7ff0a9ffb700 1 -- 172.21.15.16:0/1414056116 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0ac00ab40 con 0x7ff0b415d900 2023-10-16T12:27:23.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.186+0000 7ff0bb749700 1 -- 172.21.15.16:0/1414056116 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0b00094a0 con 0x7ff0b415d900 2023-10-16T12:27:23.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.186+0000 7ff0a9ffb700 1 -- 172.21.15.16:0/1414056116 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0ac012400 con 0x7ff0b415d900 2023-10-16T12:27:23.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.186+0000 7ff0bb749700 1 -- 172.21.15.16:0/1414056116 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0b414d770 con 0x7ff0b415d900 2023-10-16T12:27:23.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.187+0000 7ff0bb749700 1 -- 172.21.15.16:0/1414056116 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0b404ebb0 con 0x7ff0b415d900 2023-10-16T12:27:23.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.187+0000 7ff0a9ffb700 1 -- 172.21.15.16:0/1414056116 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 47) v1 ==== 89332+0+0 (secure 0 0 0) 0x7ff0ac01b3b0 con 0x7ff0b415d900 2023-10-16T12:27:23.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.188+0000 7ff0a9ffb700 1 --2- 172.21.15.16:0/1414056116 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7ff09806b020 0x7ff09806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:23.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.188+0000 7ff0b94e5700 1 -- 172.21.15.16:0/1414056116 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7ff09806b020 msgr2=0x7ff09806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/444334639 2023-10-16T12:27:23.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.188+0000 7ff0a9ffb700 1 -- 172.21.15.16:0/1414056116 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6401+0+0 (secure 0 0 0) 0x7ff0ac0895b0 con 0x7ff0b415d900 2023-10-16T12:27:23.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.188+0000 7ff0b94e5700 1 --2- 172.21.15.16:0/1414056116 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7ff09806b020 0x7ff09806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:27:23.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.192+0000 7ff0a9ffb700 1 -- 172.21.15.16:0/1414056116 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff0ac054340 con 0x7ff0b415d900 2023-10-16T12:27:23.285 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:27:23.283+0000 7f1533820200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T12:27:23.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.389+0000 7ff0b94e5700 1 -- 172.21.15.16:0/1414056116 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7ff09806b020 msgr2=0x7ff09806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/444334639 2023-10-16T12:27:23.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.389+0000 7ff0b94e5700 1 --2- 172.21.15.16:0/1414056116 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7ff09806b020 0x7ff09806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:27:23.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.447+0000 7ff0bb749700 1 -- 172.21.15.16:0/1414056116 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff0b415cd10 con 0x7ff0b415d900 2023-10-16T12:27:23.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.448+0000 7ff0a9ffb700 1 -- 172.21.15.16:0/1414056116 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+4915 (secure 0 0 0) 0x7ff0ac057960 con 0x7ff0b415d900 2023-10-16T12:27:23.450 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:27:23.451 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":44,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6820,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/1256919442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":1256919442},{"type":"v1","addr":"172.21.15.49:6834","nonce":1256919442}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":6823,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/3811815346","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":3811815346},{"type":"v1","addr":"172.21.15.49:6835","nonce":3811815346}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":7110,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/1226064755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":1226064755},{"type":"v1","addr":"172.21.15.16:6836","nonce":1226064755}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:27:00.825934+0000","modified":"2023-10-16T12:27:03.831266+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7113},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7113":{"gid":7113,"name":"a","rank":0,"incarnation":40,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/1843321005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1843321005},{"type":"v1","addr":"172.21.15.16:6837","nonce":1843321005}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-10-16T12:27:23.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.450+0000 7ff0bb749700 1 -- 172.21.15.16:0/1414056116 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7ff09806b020 msgr2=0x7ff09806d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:27:23.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.450+0000 7ff0bb749700 1 --2- 172.21.15.16:0/1414056116 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7ff09806b020 0x7ff09806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:23.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.450+0000 7ff0bb749700 1 -- 172.21.15.16:0/1414056116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0b415d900 msgr2=0x7ff0b414df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:23.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.450+0000 7ff0bb749700 1 --2- 172.21.15.16:0/1414056116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0b415d900 0x7ff0b414df70 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7ff0ac00cb40 tx=0x7ff0ac00ce50 comp rx=0 tx=0).stop 2023-10-16T12:27:23.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.450+0000 7ff0bb749700 1 -- 172.21.15.16:0/1414056116 shutdown_connections 2023-10-16T12:27:23.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.450+0000 7ff0bb749700 1 --2- 172.21.15.16:0/1414056116 >> [v2:172.21.15.16:6832/444334639,v1:172.21.15.16:6833/444334639] conn(0x7ff09806b020 0x7ff09806d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:23.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.450+0000 7ff0bb749700 1 --2- 172.21.15.16:0/1414056116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0b415d900 0x7ff0b414df70 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:23.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.450+0000 7ff0bb749700 1 --2- 172.21.15.16:0/1414056116 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff0b40ce9f0 0x7ff0b41f5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:23.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.450+0000 7ff0bb749700 1 --2- 172.21.15.16:0/1414056116 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff0b40d2860 0x7ff0b415ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:23.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.450+0000 7ff0bb749700 1 -- 172.21.15.16:0/1414056116 >> 172.21.15.16:0/1414056116 conn(0x7ff0b4072330 msgr2=0x7ff0b406bb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:23.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.450+0000 7ff0bb749700 1 -- 172.21.15.16:0/1414056116 shutdown_connections 2023-10-16T12:27:23.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.450+0000 7ff0bb749700 1 -- 172.21.15.16:0/1414056116 wait complete. 2023-10-16T12:27:23.455 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 44 2023-10-16T12:27:23.466 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:27:23.492 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:23.490+0000 7f1434b9d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T12:27:23.574 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:23.572+0000 7f13ad383700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:27:23.574 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:23.572+0000 7f13ad383700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:27:23.574 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:23.572+0000 7f13ad383700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:27:23.575 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:23.572+0000 7f13ad383700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:27:23.575 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:23.572+0000 7f13ad383700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:27:23.575 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:23.572+0000 7f13b0389700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:27:23.575 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:23.572+0000 7f13b0389700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:27:23.575 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:23.572+0000 7f13b0389700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:27:23.576 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:23.572+0000 7f13b0389700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:27:23.576 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:23.572+0000 7f13b0389700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:27:23.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.602+0000 7fc45bdaf700 1 Processor -- start 2023-10-16T12:27:23.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.602+0000 7fc45bdaf700 1 -- start start 2023-10-16T12:27:23.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.603+0000 7fc45bdaf700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fc45415d8d0 0x7fc4540d1c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:23.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.603+0000 7fc45bdaf700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc4540d41c0 0x7fc4540d0040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:23.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.603+0000 7fc45bdaf700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc4540d0580 0x7fc4540ce240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:23.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.603+0000 7fc45bdaf700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4540d0a10 con 0x7fc4540d0580 2023-10-16T12:27:23.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.603+0000 7fc45bdaf700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc45406ba00 con 0x7fc45415d8d0 2023-10-16T12:27:23.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.603+0000 7fc45bdaf700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4540d2150 con 0x7fc4540d41c0 2023-10-16T12:27:23.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.603+0000 7fc45a34c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc4540d0580 0x7fc4540ce240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:23.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.603+0000 7fc45a34c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc4540d0580 0x7fc4540ce240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41848/0 (socket says 172.21.15.16:41848) 2023-10-16T12:27:23.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.603+0000 7fc45a34c700 1 -- 172.21.15.16:0/3092399535 learned_addr learned my addr 172.21.15.16:0/3092399535 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:23.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.603+0000 7fc459b4b700 1 --2- 172.21.15.16:0/3092399535 >> v2:172.21.15.49:3300/0 conn(0x7fc45415d8d0 0x7fc4540d1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:23.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.603+0000 7fc45934a700 1 --2- 172.21.15.16:0/3092399535 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc4540d41c0 0x7fc4540d0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:23.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.603+0000 7fc45a34c700 1 -- 172.21.15.16:0/3092399535 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc4540d41c0 msgr2=0x7fc4540d0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:23.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.603+0000 7fc45a34c700 1 --2- 172.21.15.16:0/3092399535 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc4540d41c0 0x7fc4540d0040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:23.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.603+0000 7fc45a34c700 1 -- 172.21.15.16:0/3092399535 >> v2:172.21.15.49:3300/0 conn(0x7fc45415d8d0 msgr2=0x7fc4540d1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:23.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.603+0000 7fc45a34c700 1 --2- 172.21.15.16:0/3092399535 >> v2:172.21.15.49:3300/0 conn(0x7fc45415d8d0 0x7fc4540d1c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:23.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.603+0000 7fc45a34c700 1 -- 172.21.15.16:0/3092399535 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4540d26d0 con 0x7fc4540d0580 2023-10-16T12:27:23.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.603+0000 7fc45a34c700 1 --2- 172.21.15.16:0/3092399535 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4540d0580 0x7fc4540ce240 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7fc45408b460 tx=0x7fc44c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc91e41bc9e964c6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:23.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.604+0000 7fc458b49700 1 -- 172.21.15.16:0/3092399535 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc44c00e070 con 0x7fc4540d0580 2023-10-16T12:27:23.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.604+0000 7fc458b49700 1 -- 172.21.15.16:0/3092399535 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc44c00b9c0 con 0x7fc4540d0580 2023-10-16T12:27:23.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.604+0000 7fc458b49700 1 -- 172.21.15.16:0/3092399535 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc44c00bd70 con 0x7fc4540d0580 2023-10-16T12:27:23.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.604+0000 7fc45bdaf700 1 -- 172.21.15.16:0/3092399535 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4540d0580 msgr2=0x7fc4540ce240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:23.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.604+0000 7fc45bdaf700 1 --2- 172.21.15.16:0/3092399535 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4540d0580 0x7fc4540ce240 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7fc45408b460 tx=0x7fc44c00b330 comp rx=0 tx=0).stop 2023-10-16T12:27:23.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.604+0000 7fc45bdaf700 1 -- 172.21.15.16:0/3092399535 shutdown_connections 2023-10-16T12:27:23.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.604+0000 7fc45bdaf700 1 --2- 172.21.15.16:0/3092399535 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4540d0580 0x7fc4540ce240 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:23.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.604+0000 7fc45bdaf700 1 --2- 172.21.15.16:0/3092399535 >> v2:172.21.15.49:3300/0 conn(0x7fc45415d8d0 0x7fc4540d1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:23.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.604+0000 7fc45bdaf700 1 --2- 172.21.15.16:0/3092399535 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc4540d41c0 0x7fc4540d0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:23.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.604+0000 7fc45bdaf700 1 -- 172.21.15.16:0/3092399535 >> 172.21.15.16:0/3092399535 conn(0x7fc4540d3630 msgr2=0x7fc4540d3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:23.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.604+0000 7fc45bdaf700 1 -- 172.21.15.16:0/3092399535 shutdown_connections 2023-10-16T12:27:23.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.604+0000 7fc45bdaf700 1 -- 172.21.15.16:0/3092399535 wait complete. 2023-10-16T12:27:23.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.606+0000 7fc45bdaf700 1 Processor -- start 2023-10-16T12:27:23.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.606+0000 7fc45bdaf700 1 -- start start 2023-10-16T12:27:23.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.607+0000 7fc45bdaf700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4540d0580 0x7fc4541f2520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:23.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.607+0000 7fc45bdaf700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc4540d41c0 0x7fc4540c9640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:23.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.607+0000 7fc45bdaf700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc45415d8d0 0x7fc4540c9b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:23.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.607+0000 7fc45bdaf700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4540ccc30 con 0x7fc4540d0580 2023-10-16T12:27:23.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.607+0000 7fc45bdaf700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4541f2a60 con 0x7fc4540d41c0 2023-10-16T12:27:23.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.607+0000 7fc45bdaf700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4541509c0 con 0x7fc45415d8d0 2023-10-16T12:27:23.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.607+0000 7fc459b4b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4540d0580 0x7fc4541f2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:23.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.607+0000 7fc459b4b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4540d0580 0x7fc4541f2520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41850/0 (socket says 172.21.15.16:41850) 2023-10-16T12:27:23.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.607+0000 7fc459b4b700 1 -- 172.21.15.16:0/2098144452 learned_addr learned my addr 172.21.15.16:0/2098144452 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:23.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.607+0000 7fc45934a700 1 --2- 172.21.15.16:0/2098144452 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc4540d41c0 0x7fc4540c9640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:23.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.607+0000 7fc45a34c700 1 --2- 172.21.15.16:0/2098144452 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc45415d8d0 0x7fc4540c9b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:23.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.607+0000 7fc459b4b700 1 -- 172.21.15.16:0/2098144452 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc45415d8d0 msgr2=0x7fc4540c9b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:23.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.607+0000 7fc459b4b700 1 --2- 172.21.15.16:0/2098144452 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc45415d8d0 0x7fc4540c9b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:23.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.607+0000 7fc459b4b700 1 -- 172.21.15.16:0/2098144452 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc4540d41c0 msgr2=0x7fc4540c9640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:23.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.607+0000 7fc459b4b700 1 --2- 172.21.15.16:0/2098144452 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc4540d41c0 0x7fc4540c9640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:23.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.607+0000 7fc459b4b700 1 -- 172.21.15.16:0/2098144452 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc44c009b30 con 0x7fc4540d0580 2023-10-16T12:27:23.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.607+0000 7fc45a34c700 1 --2- 172.21.15.16:0/2098144452 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc45415d8d0 0x7fc4540c9b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:27:23.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.607+0000 7fc45934a700 1 --2- 172.21.15.16:0/2098144452 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc4540d41c0 0x7fc4540c9640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:27:23.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.608+0000 7fc459b4b700 1 --2- 172.21.15.16:0/2098144452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4540d0580 0x7fc4541f2520 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7fc45000e840 tx=0x7fc45000ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:23.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.608+0000 7fc44a7fc700 1 -- 172.21.15.16:0/2098144452 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc450013610 con 0x7fc4540d0580 2023-10-16T12:27:23.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.608+0000 7fc44a7fc700 1 -- 172.21.15.16:0/2098144452 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc450013770 con 0x7fc4540d0580 2023-10-16T12:27:23.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.608+0000 7fc45bdaf700 1 -- 172.21.15.16:0/2098144452 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc45414dca0 con 0x7fc4540d0580 2023-10-16T12:27:23.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.608+0000 7fc44a7fc700 1 -- 172.21.15.16:0/2098144452 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc450013b20 con 0x7fc4540d0580 2023-10-16T12:27:23.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.608+0000 7fc45bdaf700 1 -- 172.21.15.16:0/2098144452 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc45414d650 con 0x7fc4540d0580 2023-10-16T12:27:23.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.609+0000 7fc45bdaf700 1 -- 172.21.15.16:0/2098144452 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc45414d4b0 con 0x7fc4540d0580 2023-10-16T12:27:23.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.609+0000 7fc44a7fc700 1 -- 172.21.15.16:0/2098144452 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 48) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fc450013c80 con 0x7fc4540d0580 2023-10-16T12:27:23.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.610+0000 7fc44a7fc700 1 -- 172.21.15.16:0/2098144452 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6745+0+0 (secure 0 0 0) 0x7fc4500890d0 con 0x7fc4540d0580 2023-10-16T12:27:23.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.614+0000 7fc44a7fc700 1 -- 172.21.15.16:0/2098144452 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc45414d4b0 con 0x7fc4540d0580 2023-10-16T12:27:23.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.865+0000 7fc45bdaf700 1 -- 172.21.15.16:0/2098144452 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc45414d4b0 con 0x7fc4540d0580 2023-10-16T12:27:23.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.865+0000 7fc44a7fc700 1 -- 172.21.15.16:0/2098144452 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+4915 (secure 0 0 0) 0x7fc45414d4b0 con 0x7fc4540d0580 2023-10-16T12:27:23.868 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:27:23.868 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":44,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6820,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/1256919442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":1256919442},{"type":"v1","addr":"172.21.15.49:6834","nonce":1256919442}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":6823,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/3811815346","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":3811815346},{"type":"v1","addr":"172.21.15.49:6835","nonce":3811815346}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":7110,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/1226064755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":1226064755},{"type":"v1","addr":"172.21.15.16:6836","nonce":1226064755}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:27:00.825934+0000","modified":"2023-10-16T12:27:03.831266+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7113},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7113":{"gid":7113,"name":"a","rank":0,"incarnation":40,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/1843321005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1843321005},{"type":"v1","addr":"172.21.15.16:6837","nonce":1843321005}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-10-16T12:27:23.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.867+0000 7fc45bdaf700 1 -- 172.21.15.16:0/2098144452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4540d0580 msgr2=0x7fc4541f2520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:23.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.867+0000 7fc45bdaf700 1 --2- 172.21.15.16:0/2098144452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4540d0580 0x7fc4541f2520 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7fc45000e840 tx=0x7fc45000ec00 comp rx=0 tx=0).stop 2023-10-16T12:27:23.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.867+0000 7fc45bdaf700 1 -- 172.21.15.16:0/2098144452 shutdown_connections 2023-10-16T12:27:23.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.867+0000 7fc45bdaf700 1 --2- 172.21.15.16:0/2098144452 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc45415d8d0 0x7fc4540c9b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:23.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.867+0000 7fc45bdaf700 1 --2- 172.21.15.16:0/2098144452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4540d0580 0x7fc4541f2520 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:23.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.867+0000 7fc45bdaf700 1 --2- 172.21.15.16:0/2098144452 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc4540d41c0 0x7fc4540c9640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:23.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.867+0000 7fc45bdaf700 1 -- 172.21.15.16:0/2098144452 >> 172.21.15.16:0/2098144452 conn(0x7fc4540d3630 msgr2=0x7fc4540ca790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:23.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.867+0000 7fc45bdaf700 1 -- 172.21.15.16:0/2098144452 shutdown_connections 2023-10-16T12:27:23.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:23.867+0000 7fc45bdaf700 1 -- 172.21.15.16:0/2098144452 wait complete. 2023-10-16T12:27:23.871 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 44 2023-10-16T12:27:23.882 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:27:24.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.046+0000 7fe36f8db700 1 Processor -- start 2023-10-16T12:27:24.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.047+0000 7fe36f8db700 1 -- start start 2023-10-16T12:27:24.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.047+0000 7fe36f8db700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe36806aef0 0x7fe3680cb980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:24.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.047+0000 7fe36f8db700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe3680ce160 0x7fe3680ce610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:24.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.047+0000 7fe36f8db700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3680cedb0 con 0x7fe36815db40 2023-10-16T12:27:24.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.047+0000 7fe36f8db700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe36815df20 con 0x7fe3680ce160 2023-10-16T12:27:24.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.047+0000 7fe36f8db700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3680d12c0 con 0x7fe36806aef0 2023-10-16T12:27:24.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.047+0000 7fe36de78700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe3680ce160 0x7fe3680ce610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:24.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.048+0000 7fe36de78700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe3680ce160 0x7fe3680ce610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41860/0 (socket says 172.21.15.16:41860) 2023-10-16T12:27:24.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.048+0000 7fe36de78700 1 -- 172.21.15.16:0/4153051573 learned_addr learned my addr 172.21.15.16:0/4153051573 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:24.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.048+0000 7fe36ce76700 1 --2- 172.21.15.16:0/4153051573 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe36806aef0 0x7fe3680cb980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:24.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.048+0000 7fe36de78700 1 -- 172.21.15.16:0/4153051573 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe36806aef0 msgr2=0x7fe3680cb980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:24.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.048+0000 7fe36de78700 1 --2- 172.21.15.16:0/4153051573 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe36806aef0 0x7fe3680cb980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:24.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.048+0000 7fe36de78700 1 -- 172.21.15.16:0/4153051573 >> v1:172.21.15.49:6789/0 conn(0x7fe36815db40 legacy=0x7fe36806a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:24.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.048+0000 7fe36de78700 1 -- 172.21.15.16:0/4153051573 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3680ceb50 con 0x7fe3680ce160 2023-10-16T12:27:24.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.048+0000 7fe36de78700 1 --2- 172.21.15.16:0/4153051573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3680ce160 0x7fe3680ce610 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7fe360009e50 tx=0x7fe36000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfbe8de82fb674b4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:24.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.048+0000 7fe35ffff700 1 -- 172.21.15.16:0/4153051573 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe36000e070 con 0x7fe3680ce160 2023-10-16T12:27:24.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.048+0000 7fe35ffff700 1 -- 172.21.15.16:0/4153051573 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe360003a20 con 0x7fe3680ce160 2023-10-16T12:27:24.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.048+0000 7fe35ffff700 1 -- 172.21.15.16:0/4153051573 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe360003dd0 con 0x7fe3680ce160 2023-10-16T12:27:24.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.049+0000 7fe36f8db700 1 -- 172.21.15.16:0/4153051573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3680ce160 msgr2=0x7fe3680ce610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:24.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.049+0000 7fe36f8db700 1 --2- 172.21.15.16:0/4153051573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3680ce160 0x7fe3680ce610 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7fe360009e50 tx=0x7fe36000b330 comp rx=0 tx=0).stop 2023-10-16T12:27:24.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.049+0000 7fe36f8db700 1 -- 172.21.15.16:0/4153051573 shutdown_connections 2023-10-16T12:27:24.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.049+0000 7fe36f8db700 1 --2- 172.21.15.16:0/4153051573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3680ce160 0x7fe3680ce610 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:24.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.049+0000 7fe36f8db700 1 --2- 172.21.15.16:0/4153051573 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe36806aef0 0x7fe3680cb980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:24.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.049+0000 7fe36f8db700 1 -- 172.21.15.16:0/4153051573 >> 172.21.15.16:0/4153051573 conn(0x7fe3680b1fd0 msgr2=0x7fe3680b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:24.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.049+0000 7fe36f8db700 1 -- 172.21.15.16:0/4153051573 shutdown_connections 2023-10-16T12:27:24.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.049+0000 7fe36f8db700 1 -- 172.21.15.16:0/4153051573 wait complete. 2023-10-16T12:27:24.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.050+0000 7fe36f8db700 1 Processor -- start 2023-10-16T12:27:24.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.050+0000 7fe36f8db700 1 -- start start 2023-10-16T12:27:24.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.051+0000 7fe36f8db700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe36806aef0 0x7fe3680b1200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:24.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.051+0000 7fe36f8db700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe3680ce160 0x7fe368160250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:24.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.051+0000 7fe36f8db700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe36815db40 0x7fe368160790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:24.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.051+0000 7fe36f8db700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3680c6a50 con 0x7fe36806aef0 2023-10-16T12:27:24.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.051+0000 7fe36f8db700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3680c5ed0 con 0x7fe36815db40 2023-10-16T12:27:24.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.051+0000 7fe36f8db700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3680c61b0 con 0x7fe3680ce160 2023-10-16T12:27:24.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.051+0000 7fe36d677700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe36806aef0 0x7fe3680b1200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:24.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.051+0000 7fe36d677700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe36806aef0 0x7fe3680b1200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41862/0 (socket says 172.21.15.16:41862) 2023-10-16T12:27:24.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.051+0000 7fe36d677700 1 -- 172.21.15.16:0/1235178316 learned_addr learned my addr 172.21.15.16:0/1235178316 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:24.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.051+0000 7fe36de78700 1 --2- 172.21.15.16:0/1235178316 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe36815db40 0x7fe368160790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:24.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.051+0000 7fe36ce76700 1 --2- 172.21.15.16:0/1235178316 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe3680ce160 0x7fe368160250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:24.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.051+0000 7fe36d677700 1 -- 172.21.15.16:0/1235178316 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe3680ce160 msgr2=0x7fe368160250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:24.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.051+0000 7fe36d677700 1 --2- 172.21.15.16:0/1235178316 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe3680ce160 0x7fe368160250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:24.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.051+0000 7fe36d677700 1 -- 172.21.15.16:0/1235178316 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe36815db40 msgr2=0x7fe368160790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:24.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.051+0000 7fe36d677700 1 --2- 172.21.15.16:0/1235178316 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe36815db40 0x7fe368160790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:24.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.051+0000 7fe36d677700 1 -- 172.21.15.16:0/1235178316 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe360009b30 con 0x7fe36806aef0 2023-10-16T12:27:24.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.051+0000 7fe36ce76700 1 --2- 172.21.15.16:0/1235178316 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe3680ce160 0x7fe368160250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:27:24.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.051+0000 7fe36de78700 1 --2- 172.21.15.16:0/1235178316 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe36815db40 0x7fe368160790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:27:24.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.051+0000 7fe36d677700 1 --2- 172.21.15.16:0/1235178316 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe36806aef0 0x7fe3680b1200 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7fe35800c4d0 tx=0x7fe35800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:24.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.051+0000 7fe35dffb700 1 -- 172.21.15.16:0/1235178316 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe358011790 con 0x7fe36806aef0 2023-10-16T12:27:24.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.051+0000 7fe35dffb700 1 -- 172.21.15.16:0/1235178316 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3580118f0 con 0x7fe36806aef0 2023-10-16T12:27:24.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.052+0000 7fe36f8db700 1 -- 172.21.15.16:0/1235178316 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3680c6490 con 0x7fe36806aef0 2023-10-16T12:27:24.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.052+0000 7fe35dffb700 1 -- 172.21.15.16:0/1235178316 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe358011ca0 con 0x7fe36806aef0 2023-10-16T12:27:24.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.052+0000 7fe36f8db700 1 -- 172.21.15.16:0/1235178316 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3680b1740 con 0x7fe36806aef0 2023-10-16T12:27:24.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.053+0000 7fe36f8db700 1 -- 172.21.15.16:0/1235178316 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3680b2c80 con 0x7fe36806aef0 2023-10-16T12:27:24.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.053+0000 7fe35dffb700 1 -- 172.21.15.16:0/1235178316 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 48) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fe35801c460 con 0x7fe36806aef0 2023-10-16T12:27:24.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.054+0000 7fe35dffb700 1 -- 172.21.15.16:0/1235178316 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6745+0+0 (secure 0 0 0) 0x7fe3580892f0 con 0x7fe36806aef0 2023-10-16T12:27:24.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.058+0000 7fe35dffb700 1 -- 172.21.15.16:0/1235178316 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe3580543e0 con 0x7fe36806aef0 2023-10-16T12:27:24.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.263+0000 7fe36f8db700 1 -- 172.21.15.16:0/1235178316 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe36804ebb0 con 0x7fe36806aef0 2023-10-16T12:27:24.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.265+0000 7fe35dffb700 1 -- 172.21.15.16:0/1235178316 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v101) v1 ==== 74+0+17710 (secure 0 0 0) 0x7fe358057a00 con 0x7fe36806aef0 2023-10-16T12:27:24.267 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:27:24.267 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":101,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:27:23.504749+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:26:59.206630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2023-10-16T12:26:59.784585+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/389861626":"2023-10-17T12:27:23.504689+0000","172.21.15.16:0/2994287568":"2023-10-17T12:27:23.504689+0000","172.21.15.16:0/2177553605":"2023-10-17T12:27:23.504689+0000","172.21.15.16:0/2685123653":"2023-10-17T12:27:23.504689+0000","172.21.15.16:6833/444334639":"2023-10-17T12:27:23.504689+0000","172.21.15.16:6833/292891612":"2023-10-17T12:27:15.899818+0000","172.21.15.16:6832/444334639":"2023-10-17T12:27:23.504689+0000","172.21.15.16:6832/292891612":"2023-10-17T12:27:15.899818+0000","172.21.15.16:0/1020789692":"2023-10-17T12:27:15.899818+0000","172.21.15.16:0/478878390":"2023-10-17T12:27:15.899818+0000","172.21.15.16:0/2293822331":"2023-10-17T12:27:15.899818+0000","172.21.15.16:0/2440484844":"2023-10-17T12:27:23.504689+0000","172.21.15.16:0/1305507650":"2023-10-17T12:27:15.899818+0000","172.21.15.16:0/3205233425":"2023-10-17T12:27:15.899818+0000","172.21.15.16:0/3310973712":"2023-10-17T12:27:23.504689+0000","172.21.15.16:0/2477042442":"2023-10-17T12:27:15.899818+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:27:24.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.267+0000 7fe36f8db700 1 -- 172.21.15.16:0/1235178316 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe36806aef0 msgr2=0x7fe3680b1200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:24.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.267+0000 7fe36f8db700 1 --2- 172.21.15.16:0/1235178316 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe36806aef0 0x7fe3680b1200 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7fe35800c4d0 tx=0x7fe35800c890 comp rx=0 tx=0).stop 2023-10-16T12:27:24.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.267+0000 7fe36f8db700 1 -- 172.21.15.16:0/1235178316 shutdown_connections 2023-10-16T12:27:24.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.267+0000 7fe36f8db700 1 --2- 172.21.15.16:0/1235178316 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe36815db40 0x7fe368160790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:24.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.267+0000 7fe36f8db700 1 --2- 172.21.15.16:0/1235178316 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe36806aef0 0x7fe3680b1200 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:24.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.267+0000 7fe36f8db700 1 --2- 172.21.15.16:0/1235178316 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe3680ce160 0x7fe368160250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:24.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.267+0000 7fe36f8db700 1 -- 172.21.15.16:0/1235178316 >> 172.21.15.16:0/1235178316 conn(0x7fe3680b1fd0 msgr2=0x7fe3680ccf00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:24.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.267+0000 7fe36f8db700 1 -- 172.21.15.16:0/1235178316 shutdown_connections 2023-10-16T12:27:24.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.267+0000 7fe36f8db700 1 -- 172.21.15.16:0/1235178316 wait complete. 2023-10-16T12:27:24.281 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-16T12:27:24.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.405+0000 7f3371cb8700 1 Processor -- start 2023-10-16T12:27:24.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.405+0000 7f3371cb8700 1 -- start start 2023-10-16T12:27:24.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.406+0000 7f3371cb8700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f336c0cc8a0 0x7f336c15e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:24.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.406+0000 7f3371cb8700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f336c06a7e0 0x7f336c06ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:24.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.406+0000 7f3371cb8700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f336c06b150 con 0x7f336c15b280 2023-10-16T12:27:24.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.406+0000 7f3371cb8700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f336c161c30 con 0x7f336c0cc8a0 2023-10-16T12:27:24.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.406+0000 7f3371cb8700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f336c161f10 con 0x7f336c06a7e0 2023-10-16T12:27:24.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.406+0000 7f336affd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f336c0cc8a0 0x7f336c15e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:24.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.406+0000 7f336b7fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f336c15b280 0x7f336c1613e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59374/0 (socket says 172.21.15.16:59374) 2023-10-16T12:27:24.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.406+0000 7f336b7fe700 1 -- 172.21.15.16:0/805510830 learned_addr learned my addr 172.21.15.16:0/805510830 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:27:24.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.406+0000 7f336bfff700 1 --2- 172.21.15.16:0/805510830 >> v2:172.21.15.49:3300/0 conn(0x7f336c06a7e0 0x7f336c06ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:24.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.407+0000 7f336affd700 1 -- 172.21.15.16:0/805510830 >> v2:172.21.15.49:3300/0 conn(0x7f336c06a7e0 msgr2=0x7f336c06ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:24.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.407+0000 7f336affd700 1 --2- 172.21.15.16:0/805510830 >> v2:172.21.15.49:3300/0 conn(0x7f336c06a7e0 0x7f336c06ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:24.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.407+0000 7f336affd700 1 -- 172.21.15.16:0/805510830 >> v1:172.21.15.16:6789/0 conn(0x7f336c15b280 legacy=0x7f336c1613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:24.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.407+0000 7f336affd700 1 -- 172.21.15.16:0/805510830 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f336c1621f0 con 0x7f336c0cc8a0 2023-10-16T12:27:24.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.407+0000 7f336a7fc700 1 -- 172.21.15.16:0/805510830 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2470725493 0 0) 0x7f336c06b150 con 0x7f336c15b280 2023-10-16T12:27:24.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.407+0000 7f336affd700 1 --2- 172.21.15.16:0/805510830 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f336c0cc8a0 0x7f336c15e700 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f336c0cbed0 tx=0x7f3360009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b9eef54d06c18be server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:24.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.407+0000 7f336a7fc700 1 -- 172.21.15.16:0/805510830 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3360011830 con 0x7f336c0cc8a0 2023-10-16T12:27:24.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.407+0000 7f336a7fc700 1 -- 172.21.15.16:0/805510830 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3360011990 con 0x7f336c0cc8a0 2023-10-16T12:27:24.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.407+0000 7f336a7fc700 1 -- 172.21.15.16:0/805510830 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3360011d20 con 0x7f336c0cc8a0 2023-10-16T12:27:24.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.408+0000 7f3371cb8700 1 -- 172.21.15.16:0/805510830 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f336c0cc8a0 msgr2=0x7f336c15e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:24.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.408+0000 7f3371cb8700 1 --2- 172.21.15.16:0/805510830 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f336c0cc8a0 0x7f336c15e700 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f336c0cbed0 tx=0x7f3360009b20 comp rx=0 tx=0).stop 2023-10-16T12:27:24.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.408+0000 7f3371cb8700 1 -- 172.21.15.16:0/805510830 shutdown_connections 2023-10-16T12:27:24.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.408+0000 7f3371cb8700 1 --2- 172.21.15.16:0/805510830 >> v2:172.21.15.49:3300/0 conn(0x7f336c06a7e0 0x7f336c06ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:24.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.408+0000 7f3371cb8700 1 --2- 172.21.15.16:0/805510830 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f336c0cc8a0 0x7f336c15e700 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:24.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.408+0000 7f3371cb8700 1 -- 172.21.15.16:0/805510830 >> 172.21.15.16:0/805510830 conn(0x7f336c0b1fd0 msgr2=0x7f336c0b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:24.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.408+0000 7f3371cb8700 1 -- 172.21.15.16:0/805510830 shutdown_connections 2023-10-16T12:27:24.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.408+0000 7f3371cb8700 1 -- 172.21.15.16:0/805510830 wait complete. 2023-10-16T12:27:24.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.409+0000 7f3371cb8700 1 Processor -- start 2023-10-16T12:27:24.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.409+0000 7f3371cb8700 1 -- start start 2023-10-16T12:27:24.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.410+0000 7f3371cb8700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f336c06a7e0 0x7f336c1f2820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:24.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.410+0000 7f3371cb8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f336c0cc8a0 0x7f336c0c5960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:24.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.410+0000 7f3371cb8700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f336c15b280 0x7f336c0ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:24.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.410+0000 7f3371cb8700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f336c0c62d0 con 0x7f336c0cc8a0 2023-10-16T12:27:24.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.410+0000 7f3371cb8700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f336c0c65b0 con 0x7f336c06a7e0 2023-10-16T12:27:24.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.410+0000 7f3371cb8700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f336c0c6890 con 0x7f336c15b280 2023-10-16T12:27:24.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.410+0000 7f336affd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f336c0cc8a0 0x7f336c0c5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:24.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.410+0000 7f336affd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f336c0cc8a0 0x7f336c0c5960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41876/0 (socket says 172.21.15.16:41876) 2023-10-16T12:27:24.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.410+0000 7f336affd700 1 -- 172.21.15.16:0/4079486228 learned_addr learned my addr 172.21.15.16:0/4079486228 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:24.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.410+0000 7f336bfff700 1 --2- 172.21.15.16:0/4079486228 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f336c15b280 0x7f336c0ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:24.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.410+0000 7f336b7fe700 1 --2- 172.21.15.16:0/4079486228 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f336c06a7e0 0x7f336c1f2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:24.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.411+0000 7f336affd700 1 -- 172.21.15.16:0/4079486228 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f336c15b280 msgr2=0x7f336c0ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:24.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.411+0000 7f336affd700 1 --2- 172.21.15.16:0/4079486228 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f336c15b280 0x7f336c0ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:24.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.411+0000 7f336affd700 1 -- 172.21.15.16:0/4079486228 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f336c06a7e0 msgr2=0x7f336c1f2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:24.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.411+0000 7f336affd700 1 --2- 172.21.15.16:0/4079486228 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f336c06a7e0 0x7f336c1f2820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:24.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.411+0000 7f336affd700 1 -- 172.21.15.16:0/4079486228 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33600094a0 con 0x7f336c0cc8a0 2023-10-16T12:27:24.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.411+0000 7f336b7fe700 1 --2- 172.21.15.16:0/4079486228 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f336c06a7e0 0x7f336c1f2820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:27:24.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.411+0000 7f336bfff700 1 --2- 172.21.15.16:0/4079486228 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f336c15b280 0x7f336c0ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:27:24.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.411+0000 7f336affd700 1 --2- 172.21.15.16:0/4079486228 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f336c0cc8a0 0x7f336c0c5960 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f33600090b0 tx=0x7f3360004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:24.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.411+0000 7f3353fff700 1 -- 172.21.15.16:0/4079486228 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33600038a0 con 0x7f336c0cc8a0 2023-10-16T12:27:24.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.411+0000 7f3353fff700 1 -- 172.21.15.16:0/4079486228 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f336001f070 con 0x7f336c0cc8a0 2023-10-16T12:27:24.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.411+0000 7f3371cb8700 1 -- 172.21.15.16:0/4079486228 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f336c0c4010 con 0x7f336c0cc8a0 2023-10-16T12:27:24.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.411+0000 7f3353fff700 1 -- 172.21.15.16:0/4079486228 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3360003c20 con 0x7f336c0cc8a0 2023-10-16T12:27:24.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.411+0000 7f3371cb8700 1 -- 172.21.15.16:0/4079486228 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f336c1f2d60 con 0x7f336c0cc8a0 2023-10-16T12:27:24.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.412+0000 7f3371cb8700 1 -- 172.21.15.16:0/4079486228 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f336c0c66f0 con 0x7f336c0cc8a0 2023-10-16T12:27:24.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.413+0000 7f3353fff700 1 -- 172.21.15.16:0/4079486228 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 48) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f336001a430 con 0x7f336c0cc8a0 2023-10-16T12:27:24.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.413+0000 7f3353fff700 1 -- 172.21.15.16:0/4079486228 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6745+0+0 (secure 0 0 0) 0x7f3360021070 con 0x7f336c0cc8a0 2023-10-16T12:27:24.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.418+0000 7f3353fff700 1 -- 172.21.15.16:0/4079486228 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f336c0c66f0 con 0x7f336c0cc8a0 2023-10-16T12:27:24.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.515+0000 7f3353fff700 1 -- 172.21.15.16:0/4079486228 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 49) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f3360055940 con 0x7f336c0cc8a0 2023-10-16T12:27:24.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.515+0000 7f3353fff700 1 --2- 172.21.15.16:0/4079486228 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f3358070bb0 0x7f3358072fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:24.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.516+0000 7f336b7fe700 1 --2- 172.21.15.16:0/4079486228 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f3358070bb0 0x7f3358072fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:24.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.517+0000 7f336b7fe700 1 --2- 172.21.15.16:0/4079486228 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f3358070bb0 0x7f3358072fa0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f335c000f60 tx=0x7f335c007d30 comp rx=0 tx=0).ready entity=mgr.6365 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:24.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.673+0000 7f3371cb8700 1 -- 172.21.15.16:0/4079486228 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f336c0c4530 con 0x7f336c0cc8a0 2023-10-16T12:27:24.676 INFO:tasks.ceph.mon.a.smithi016.stderr:2023-10-16T12:27:24.674+0000 7f9fc8535700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-16T12:27:24.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.675+0000 7f3353fff700 1 -- 172.21.15.16:0/4079486228 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v45) v1 ==== 161+0+0 (secure 0 0 0) 0x7f336c0c4530 con 0x7f336c0cc8a0 2023-10-16T12:27:24.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.678+0000 7f3371cb8700 1 -- 172.21.15.16:0/4079486228 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f3358070bb0 msgr2=0x7f3358072fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:24.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.678+0000 7f3371cb8700 1 --2- 172.21.15.16:0/4079486228 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f3358070bb0 0x7f3358072fa0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f335c000f60 tx=0x7f335c007d30 comp rx=0 tx=0).stop 2023-10-16T12:27:24.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.678+0000 7f3371cb8700 1 -- 172.21.15.16:0/4079486228 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f336c0cc8a0 msgr2=0x7f336c0c5960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:24.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.678+0000 7f3371cb8700 1 --2- 172.21.15.16:0/4079486228 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f336c0cc8a0 0x7f336c0c5960 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f33600090b0 tx=0x7f3360004750 comp rx=0 tx=0).stop 2023-10-16T12:27:24.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.678+0000 7f3371cb8700 1 -- 172.21.15.16:0/4079486228 shutdown_connections 2023-10-16T12:27:24.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.678+0000 7f3371cb8700 1 --2- 172.21.15.16:0/4079486228 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f336c15b280 0x7f336c0ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:24.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.678+0000 7f3371cb8700 1 --2- 172.21.15.16:0/4079486228 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f3358070bb0 0x7f3358072fa0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:24.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.678+0000 7f3371cb8700 1 --2- 172.21.15.16:0/4079486228 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f336c06a7e0 0x7f336c1f2820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:24.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.678+0000 7f3371cb8700 1 --2- 172.21.15.16:0/4079486228 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f336c0cc8a0 0x7f336c0c5960 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:24.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.678+0000 7f3371cb8700 1 -- 172.21.15.16:0/4079486228 >> 172.21.15.16:0/4079486228 conn(0x7f336c0b1fd0 msgr2=0x7f336c072600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:24.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.678+0000 7f3371cb8700 1 -- 172.21.15.16:0/4079486228 shutdown_connections 2023-10-16T12:27:24.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.678+0000 7f3371cb8700 1 -- 172.21.15.16:0/4079486228 wait complete. 2023-10-16T12:27:24.683 INFO:teuthology.orchestra.run.smithi016.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-16T12:27:24.683 INFO:tasks.ceph.mds.a.smithi016.stderr: -3791> 2023-10-16T12:27:06.382+0000 7f8d3737c700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T12:27:24.693 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-16T12:27:24.712 INFO:tasks.ceph.mds.a.smithi016.stdout:starting mds.a at 2023-10-16T12:27:24.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.822+0000 7fe95f33f700 1 Processor -- start 2023-10-16T12:27:24.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.822+0000 7fe95f33f700 1 -- start start 2023-10-16T12:27:24.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.823+0000 7fe95f33f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe95815d8c0 0x7fe9580b2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:24.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.823+0000 7fe95f33f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe958162390 0x7fe9580b34e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:24.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.823+0000 7fe95f33f700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe958161f00 con 0x7fe95806be40 2023-10-16T12:27:24.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.823+0000 7fe95f33f700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe95815f180 con 0x7fe958162390 2023-10-16T12:27:24.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.823+0000 7fe95f33f700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe95815ecb0 con 0x7fe95815d8c0 2023-10-16T12:27:24.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.823+0000 7fe95c8da700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe958162390 0x7fe9580b34e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:24.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.823+0000 7fe95c8da700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe958162390 0x7fe9580b34e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41920/0 (socket says 172.21.15.16:41920) 2023-10-16T12:27:24.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.823+0000 7fe95c8da700 1 -- 172.21.15.16:0/2692030623 learned_addr learned my addr 172.21.15.16:0/2692030623 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:24.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.823+0000 7fe95d0db700 1 --2- 172.21.15.16:0/2692030623 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe95815d8c0 0x7fe9580b2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:24.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.823+0000 7fe95c8da700 1 -- 172.21.15.16:0/2692030623 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe95815d8c0 msgr2=0x7fe9580b2fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:24.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.823+0000 7fe95c8da700 1 --2- 172.21.15.16:0/2692030623 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe95815d8c0 0x7fe9580b2fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:24.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.823+0000 7fe95c8da700 1 -- 172.21.15.16:0/2692030623 >> v1:172.21.15.49:6789/0 conn(0x7fe95806be40 legacy=0x7fe95806c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:24.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.824+0000 7fe95c8da700 1 -- 172.21.15.16:0/2692030623 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe95815eea0 con 0x7fe958162390 2023-10-16T12:27:24.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.824+0000 7fe95c8da700 1 --2- 172.21.15.16:0/2692030623 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe958162390 0x7fe9580b34e0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7fe954009920 tx=0x7fe954009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7edee676bc8aa46 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:24.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.824+0000 7fe94ffff700 1 -- 172.21.15.16:0/2692030623 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe954011830 con 0x7fe958162390 2023-10-16T12:27:24.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.824+0000 7fe94ffff700 1 -- 172.21.15.16:0/2692030623 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe954011990 con 0x7fe958162390 2023-10-16T12:27:24.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.824+0000 7fe94ffff700 1 -- 172.21.15.16:0/2692030623 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe954011d20 con 0x7fe958162390 2023-10-16T12:27:24.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.824+0000 7fe95f33f700 1 -- 172.21.15.16:0/2692030623 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe958162390 msgr2=0x7fe9580b34e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:24.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.824+0000 7fe95f33f700 1 --2- 172.21.15.16:0/2692030623 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe958162390 0x7fe9580b34e0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7fe954009920 tx=0x7fe954009c30 comp rx=0 tx=0).stop 2023-10-16T12:27:24.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.824+0000 7fe95f33f700 1 -- 172.21.15.16:0/2692030623 shutdown_connections 2023-10-16T12:27:24.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.824+0000 7fe95f33f700 1 --2- 172.21.15.16:0/2692030623 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe95815d8c0 0x7fe9580b2fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:24.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.824+0000 7fe95f33f700 1 --2- 172.21.15.16:0/2692030623 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe958162390 0x7fe9580b34e0 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:24.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.824+0000 7fe95f33f700 1 -- 172.21.15.16:0/2692030623 >> 172.21.15.16:0/2692030623 conn(0x7fe9580cdfa0 msgr2=0x7fe9580ce3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:24.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.825+0000 7fe95f33f700 1 -- 172.21.15.16:0/2692030623 shutdown_connections 2023-10-16T12:27:24.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.825+0000 7fe95f33f700 1 -- 172.21.15.16:0/2692030623 wait complete. 2023-10-16T12:27:24.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.826+0000 7fe95f33f700 1 Processor -- start 2023-10-16T12:27:24.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.826+0000 7fe95f33f700 1 -- start start 2023-10-16T12:27:24.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.827+0000 7fe95f33f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe95806be40 0x7fe958148040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:24.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.827+0000 7fe95f33f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe95815d8c0 0x7fe95815c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:24.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.827+0000 7fe95f33f700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe958162390 0x7fe95815c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:24.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.827+0000 7fe95f33f700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9580c64b0 con 0x7fe95815d8c0 2023-10-16T12:27:24.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.827+0000 7fe95f33f700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe95814b940 con 0x7fe958162390 2023-10-16T12:27:24.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.827+0000 7fe95f33f700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe95814b660 con 0x7fe95806be40 2023-10-16T12:27:24.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.827+0000 7fe95c8da700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe95815d8c0 0x7fe95815c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:24.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.827+0000 7fe95c8da700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe95815d8c0 0x7fe95815c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41924/0 (socket says 172.21.15.16:41924) 2023-10-16T12:27:24.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.827+0000 7fe95c8da700 1 -- 172.21.15.16:0/490463228 learned_addr learned my addr 172.21.15.16:0/490463228 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:24.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.827+0000 7fe95d8dc700 1 --2- 172.21.15.16:0/490463228 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe958162390 0x7fe95815c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:24.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.827+0000 7fe95d0db700 1 --2- 172.21.15.16:0/490463228 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe95806be40 0x7fe958148040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:24.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.827+0000 7fe95c8da700 1 -- 172.21.15.16:0/490463228 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe95806be40 msgr2=0x7fe958148040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:24.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.827+0000 7fe95c8da700 1 --2- 172.21.15.16:0/490463228 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe95806be40 0x7fe958148040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:24.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.827+0000 7fe95c8da700 1 -- 172.21.15.16:0/490463228 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe958162390 msgr2=0x7fe95815c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:24.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.827+0000 7fe95c8da700 1 --2- 172.21.15.16:0/490463228 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe958162390 0x7fe95815c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:24.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.827+0000 7fe95c8da700 1 -- 172.21.15.16:0/490463228 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe954009550 con 0x7fe95815d8c0 2023-10-16T12:27:24.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.827+0000 7fe95c8da700 1 --2- 172.21.15.16:0/490463228 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe95815d8c0 0x7fe95815c340 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7fe954009c70 tx=0x7fe954004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:24.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.827+0000 7fe94dffb700 1 -- 172.21.15.16:0/490463228 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe954004400 con 0x7fe95815d8c0 2023-10-16T12:27:24.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.827+0000 7fe94dffb700 1 -- 172.21.15.16:0/490463228 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe95401f070 con 0x7fe95815d8c0 2023-10-16T12:27:24.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.827+0000 7fe95f33f700 1 -- 172.21.15.16:0/490463228 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9580c66c0 con 0x7fe95815d8c0 2023-10-16T12:27:24.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.827+0000 7fe94dffb700 1 -- 172.21.15.16:0/490463228 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9540038e0 con 0x7fe95815d8c0 2023-10-16T12:27:24.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.828+0000 7fe95f33f700 1 -- 172.21.15.16:0/490463228 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9580c6c80 con 0x7fe95815d8c0 2023-10-16T12:27:24.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.828+0000 7fe95f33f700 1 -- 172.21.15.16:0/490463228 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9580c6e70 con 0x7fe95815d8c0 2023-10-16T12:27:24.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.830+0000 7fe94dffb700 1 -- 172.21.15.16:0/490463228 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 49) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fe954021030 con 0x7fe95815d8c0 2023-10-16T12:27:24.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.831+0000 7fe94dffb700 1 --2- 172.21.15.16:0/490463228 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fe93c06ad50 0x7fe93c06d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:24.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.831+0000 7fe94dffb700 1 -- 172.21.15.16:0/490463228 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 6831+0+0 (secure 0 0 0) 0x7fe95408b0f0 con 0x7fe95815d8c0 2023-10-16T12:27:24.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.831+0000 7fe95d0db700 1 --2- 172.21.15.16:0/490463228 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fe93c06ad50 0x7fe93c06d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:24.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.831+0000 7fe95d0db700 1 --2- 172.21.15.16:0/490463228 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fe93c06ad50 0x7fe93c06d210 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe9580b39f0 tx=0x7fe948008040 comp rx=0 tx=0).ready entity=mgr.6365 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:24.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:24.834+0000 7fe94dffb700 1 -- 172.21.15.16:0/490463228 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe954056140 con 0x7fe95815d8c0 2023-10-16T12:27:25.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:25.084+0000 7fe95f33f700 1 -- 172.21.15.16:0/490463228 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fe95804ebb0 con 0x7fe95815d8c0 2023-10-16T12:27:25.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:25.676+0000 7fe94dffb700 1 -- 172.21.15.16:0/490463228 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v46) v1 ==== 104+0+0 (secure 0 0 0) 0x7fe954059760 con 0x7fe95815d8c0 2023-10-16T12:27:25.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:25.680+0000 7fe95f33f700 1 -- 172.21.15.16:0/490463228 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fe93c06ad50 msgr2=0x7fe93c06d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:25.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:25.680+0000 7fe95f33f700 1 --2- 172.21.15.16:0/490463228 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fe93c06ad50 0x7fe93c06d210 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe9580b39f0 tx=0x7fe948008040 comp rx=0 tx=0).stop 2023-10-16T12:27:25.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:25.680+0000 7fe95f33f700 1 -- 172.21.15.16:0/490463228 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe95815d8c0 msgr2=0x7fe95815c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:25.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:25.680+0000 7fe95f33f700 1 --2- 172.21.15.16:0/490463228 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe95815d8c0 0x7fe95815c340 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7fe954009c70 tx=0x7fe954004750 comp rx=0 tx=0).stop 2023-10-16T12:27:25.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:25.680+0000 7fe95f33f700 1 -- 172.21.15.16:0/490463228 shutdown_connections 2023-10-16T12:27:25.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:25.680+0000 7fe95f33f700 1 --2- 172.21.15.16:0/490463228 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fe93c06ad50 0x7fe93c06d210 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:25.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:25.680+0000 7fe95f33f700 1 --2- 172.21.15.16:0/490463228 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe958162390 0x7fe95815c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:25.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:25.680+0000 7fe95f33f700 1 --2- 172.21.15.16:0/490463228 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe95806be40 0x7fe958148040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:25.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:25.680+0000 7fe95f33f700 1 --2- 172.21.15.16:0/490463228 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe95815d8c0 0x7fe95815c340 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:25.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:25.680+0000 7fe95f33f700 1 -- 172.21.15.16:0/490463228 >> 172.21.15.16:0/490463228 conn(0x7fe9580cdfa0 msgr2=0x7fe958070530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:25.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:25.680+0000 7fe95f33f700 1 -- 172.21.15.16:0/490463228 shutdown_connections 2023-10-16T12:27:25.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:25.680+0000 7fe95f33f700 1 -- 172.21.15.16:0/490463228 wait complete. 2023-10-16T12:27:25.696 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-16T12:27:25.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:25.869+0000 7efe24fc8700 1 Processor -- start 2023-10-16T12:27:25.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:25.869+0000 7efe24fc8700 1 -- start start 2023-10-16T12:27:25.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:25.870+0000 7efe24fc8700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efe2015a3d0 0x7efe2006d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:25.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:25.870+0000 7efe24fc8700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7efe200cf600 0x7efe200cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:25.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:25.870+0000 7efe24fc8700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe200cb870 con 0x7efe2015e0b0 2023-10-16T12:27:25.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:25.870+0000 7efe24fc8700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe2015f0c0 con 0x7efe200cf600 2023-10-16T12:27:25.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:25.870+0000 7efe24fc8700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe20072990 con 0x7efe2015a3d0 2023-10-16T12:27:25.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:25.870+0000 7efe1dd9b700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7efe2015e0b0 0x7efe200ceef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59456/0 (socket says 172.21.15.16:59456) 2023-10-16T12:27:25.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:25.870+0000 7efe1dd9b700 1 -- 172.21.15.16:0/4210730728 learned_addr learned my addr 172.21.15.16:0/4210730728 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:27:25.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:25.870+0000 7efe1e59c700 1 --2- 172.21.15.16:0/4210730728 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efe2015a3d0 0x7efe2006d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:25.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:25.870+0000 7efe1ed9d700 1 --2- 172.21.15.16:0/4210730728 >> v2:172.21.15.49:3300/0 conn(0x7efe200cf600 0x7efe200cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:25.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:25.870+0000 7efe1d59a700 1 -- 172.21.15.16:0/4210730728 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3831498758 0 0) 0x7efe200cb870 con 0x7efe2015e0b0 2023-10-16T12:27:25.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:25.870+0000 7efe1d59a700 1 -- 172.21.15.16:0/4210730728 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe0c0035b0 con 0x7efe2015e0b0 2023-10-16T12:27:25.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:25.870+0000 7efe1ed9d700 1 -- 172.21.15.16:0/4210730728 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efe2015a3d0 msgr2=0x7efe2006d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:25.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:25.870+0000 7efe1ed9d700 1 --2- 172.21.15.16:0/4210730728 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efe2015a3d0 0x7efe2006d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:25.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:25.870+0000 7efe1ed9d700 1 -- 172.21.15.16:0/4210730728 >> v1:172.21.15.16:6789/0 conn(0x7efe2015e0b0 legacy=0x7efe200ceef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:25.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:25.870+0000 7efe1ed9d700 1 -- 172.21.15.16:0/4210730728 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe2006e040 con 0x7efe200cf600 2023-10-16T12:27:25.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:25.870+0000 7efe1e59c700 1 --2- 172.21.15.16:0/4210730728 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efe2015a3d0 0x7efe2006d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:27:25.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:25.870+0000 7efe1ed9d700 1 --2- 172.21.15.16:0/4210730728 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efe200cf600 0x7efe200cbd00 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7efe18009dc0 tx=0x7efe1800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=38df8f7cd70d2ec5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:25.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:25.871+0000 7efe1d59a700 1 -- 172.21.15.16:0/4210730728 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe1800e070 con 0x7efe200cf600 2023-10-16T12:27:25.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:25.871+0000 7efe1d59a700 1 -- 172.21.15.16:0/4210730728 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe1800bae0 con 0x7efe200cf600 2023-10-16T12:27:25.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:25.871+0000 7efe1d59a700 1 -- 172.21.15.16:0/4210730728 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe1800be90 con 0x7efe200cf600 2023-10-16T12:27:25.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:25.871+0000 7efe24fc8700 1 -- 172.21.15.16:0/4210730728 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efe200cf600 msgr2=0x7efe200cbd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:25.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:25.871+0000 7efe24fc8700 1 --2- 172.21.15.16:0/4210730728 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efe200cf600 0x7efe200cbd00 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7efe18009dc0 tx=0x7efe1800b330 comp rx=0 tx=0).stop 2023-10-16T12:27:25.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:25.871+0000 7efe24fc8700 1 -- 172.21.15.16:0/4210730728 shutdown_connections 2023-10-16T12:27:25.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:25.871+0000 7efe24fc8700 1 --2- 172.21.15.16:0/4210730728 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efe2015a3d0 0x7efe2006d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:25.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:25.871+0000 7efe24fc8700 1 --2- 172.21.15.16:0/4210730728 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efe200cf600 0x7efe200cbd00 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:25.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:25.871+0000 7efe24fc8700 1 -- 172.21.15.16:0/4210730728 >> 172.21.15.16:0/4210730728 conn(0x7efe2006f430 msgr2=0x7efe2006f810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:25.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:25.871+0000 7efe24fc8700 1 -- 172.21.15.16:0/4210730728 shutdown_connections 2023-10-16T12:27:25.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:25.871+0000 7efe24fc8700 1 -- 172.21.15.16:0/4210730728 wait complete. 2023-10-16T12:27:25.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:25.873+0000 7efe24fc8700 1 Processor -- start 2023-10-16T12:27:25.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:25.873+0000 7efe24fc8700 1 -- start start 2023-10-16T12:27:25.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:25.873+0000 7efe24fc8700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efe200cf600 0x7efe2014e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:25.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:25.873+0000 7efe24fc8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe2015a3d0 0x7efe2014f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:25.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:25.873+0000 7efe24fc8700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efe2015e0b0 0x7efe2015bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:25.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:25.873+0000 7efe24fc8700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe200b4ad0 con 0x7efe2015a3d0 2023-10-16T12:27:25.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:25.873+0000 7efe24fc8700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe2014ee30 con 0x7efe2015e0b0 2023-10-16T12:27:25.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:25.873+0000 7efe24fc8700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe200b4510 con 0x7efe200cf600 2023-10-16T12:27:25.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:25.873+0000 7efe1dd9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe2015a3d0 0x7efe2014f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:25.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:25.874+0000 7efe1dd9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe2015a3d0 0x7efe2014f2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41958/0 (socket says 172.21.15.16:41958) 2023-10-16T12:27:25.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:25.874+0000 7efe1dd9b700 1 -- 172.21.15.16:0/825805240 learned_addr learned my addr 172.21.15.16:0/825805240 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:25.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:25.874+0000 7efe1ed9d700 1 --2- 172.21.15.16:0/825805240 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efe2015e0b0 0x7efe2015bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:25.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:25.874+0000 7efe1e59c700 1 --2- 172.21.15.16:0/825805240 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efe200cf600 0x7efe2014e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:25.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:25.874+0000 7efe1dd9b700 1 -- 172.21.15.16:0/825805240 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efe200cf600 msgr2=0x7efe2014e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:25.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:25.874+0000 7efe1dd9b700 1 --2- 172.21.15.16:0/825805240 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efe200cf600 0x7efe2014e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:25.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:25.874+0000 7efe1dd9b700 1 -- 172.21.15.16:0/825805240 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efe2015e0b0 msgr2=0x7efe2015bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:25.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:25.874+0000 7efe1dd9b700 1 --2- 172.21.15.16:0/825805240 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efe2015e0b0 0x7efe2015bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:25.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:25.874+0000 7efe1dd9b700 1 -- 172.21.15.16:0/825805240 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe100094a0 con 0x7efe2015a3d0 2023-10-16T12:27:25.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:25.874+0000 7efe1dd9b700 1 --2- 172.21.15.16:0/825805240 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe2015a3d0 0x7efe2014f2c0 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7efe08001eb0 tx=0x7efe0800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:25.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:25.874+0000 7efe16ffd700 1 -- 172.21.15.16:0/825805240 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe08005d70 con 0x7efe2015a3d0 2023-10-16T12:27:25.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:25.874+0000 7efe16ffd700 1 -- 172.21.15.16:0/825805240 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe080070d0 con 0x7efe2015a3d0 2023-10-16T12:27:25.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:25.874+0000 7efe24fc8700 1 -- 172.21.15.16:0/825805240 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe18009a70 con 0x7efe2015a3d0 2023-10-16T12:27:25.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:25.874+0000 7efe16ffd700 1 -- 172.21.15.16:0/825805240 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe08007460 con 0x7efe2015a3d0 2023-10-16T12:27:25.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:25.874+0000 7efe24fc8700 1 -- 172.21.15.16:0/825805240 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe200c7550 con 0x7efe2015a3d0 2023-10-16T12:27:25.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:25.875+0000 7efe24fc8700 1 -- 172.21.15.16:0/825805240 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe200c7390 con 0x7efe2015a3d0 2023-10-16T12:27:25.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:25.876+0000 7efe16ffd700 1 -- 172.21.15.16:0/825805240 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 49) v1 ==== 89067+0+0 (secure 0 0 0) 0x7efe080075c0 con 0x7efe2015a3d0 2023-10-16T12:27:25.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:25.876+0000 7efe16ffd700 1 --2- 172.21.15.16:0/825805240 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7efe0006ad50 0x7efe0006d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:25.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:25.876+0000 7efe16ffd700 1 -- 172.21.15.16:0/825805240 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 6831+0+0 (secure 0 0 0) 0x7efe08088b60 con 0x7efe2015a3d0 2023-10-16T12:27:25.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:25.876+0000 7efe1e59c700 1 --2- 172.21.15.16:0/825805240 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7efe0006ad50 0x7efe0006d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:25.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:25.877+0000 7efe1e59c700 1 --2- 172.21.15.16:0/825805240 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7efe0006ad50 0x7efe0006d210 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7efe10009d00 tx=0x7efe10008040 comp rx=0 tx=0).ready entity=mgr.6365 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:25.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:25.881+0000 7efe16ffd700 1 -- 172.21.15.16:0/825805240 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7efe08053af0 con 0x7efe2015a3d0 2023-10-16T12:27:26.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:26.086+0000 7efe24fc8700 1 -- 172.21.15.16:0/825805240 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7efe2004ec20 con 0x7efe2015a3d0 2023-10-16T12:27:26.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:26.519+0000 7efe16ffd700 1 -- 172.21.15.16:0/825805240 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 50) v1 ==== 89275+0+0 (secure 0 0 0) 0x7efe08053ec0 con 0x7efe2015a3d0 2023-10-16T12:27:26.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:26.937+0000 7efe16ffd700 1 -- 172.21.15.16:0/825805240 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v103) v1 ==== 181+0+0 (secure 0 0 0) 0x7efe08073190 con 0x7efe2015a3d0 2023-10-16T12:27:26.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:26.941+0000 7efe24fc8700 1 -- 172.21.15.16:0/825805240 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7efe0006ad50 msgr2=0x7efe0006d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:26.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:26.941+0000 7efe24fc8700 1 --2- 172.21.15.16:0/825805240 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7efe0006ad50 0x7efe0006d210 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7efe10009d00 tx=0x7efe10008040 comp rx=0 tx=0).stop 2023-10-16T12:27:26.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:26.941+0000 7efe24fc8700 1 -- 172.21.15.16:0/825805240 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe2015a3d0 msgr2=0x7efe2014f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:26.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:26.941+0000 7efe24fc8700 1 --2- 172.21.15.16:0/825805240 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe2015a3d0 0x7efe2014f2c0 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7efe08001eb0 tx=0x7efe0800c860 comp rx=0 tx=0).stop 2023-10-16T12:27:26.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:26.941+0000 7efe24fc8700 1 -- 172.21.15.16:0/825805240 shutdown_connections 2023-10-16T12:27:26.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:26.941+0000 7efe24fc8700 1 --2- 172.21.15.16:0/825805240 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7efe0006ad50 0x7efe0006d210 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:26.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:26.941+0000 7efe24fc8700 1 --2- 172.21.15.16:0/825805240 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efe2015e0b0 0x7efe2015bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:26.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:26.941+0000 7efe24fc8700 1 --2- 172.21.15.16:0/825805240 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efe200cf600 0x7efe2014e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:26.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:26.941+0000 7efe24fc8700 1 --2- 172.21.15.16:0/825805240 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe2015a3d0 0x7efe2014f2c0 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:26.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:26.941+0000 7efe24fc8700 1 -- 172.21.15.16:0/825805240 >> 172.21.15.16:0/825805240 conn(0x7efe2006f430 msgr2=0x7efe200cde90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:26.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:26.942+0000 7efe24fc8700 1 -- 172.21.15.16:0/825805240 shutdown_connections 2023-10-16T12:27:26.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:26.942+0000 7efe24fc8700 1 -- 172.21.15.16:0/825805240 wait complete. 2023-10-16T12:27:26.945 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' removed 2023-10-16T12:27:26.958 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-16T12:27:27.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:27.093+0000 7f93dd75b700 1 Processor -- start 2023-10-16T12:27:27.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:27.093+0000 7f93dd75b700 1 -- start start 2023-10-16T12:27:27.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:27.094+0000 7f93dd75b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f93d806c5c0 0x7f93d806ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:27.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:27.094+0000 7f93dd75b700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93d80cdd60 con 0x7f93d80ce610 2023-10-16T12:27:27.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:27.094+0000 7f93dd75b700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93d80ca340 con 0x7f93d8165610 2023-10-16T12:27:27.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:27.094+0000 7f93dd75b700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93d806cfb0 con 0x7f93d806c5c0 2023-10-16T12:27:27.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:27.095+0000 7f93d67fc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f93d806c5c0 0x7f93d806ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:27.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:27.095+0000 7f93d6ffd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f93d80ce610 0x7f93d80c97d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59470/0 (socket says 172.21.15.16:59470) 2023-10-16T12:27:27.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:27.095+0000 7f93d6ffd700 1 -- 172.21.15.16:0/952414140 learned_addr learned my addr 172.21.15.16:0/952414140 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:27:27.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:27.095+0000 7f93d67fc700 1 -- 172.21.15.16:0/952414140 >> v1:172.21.15.49:6789/0 conn(0x7f93d8165610 legacy=0x7f93d8165a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:27.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:27.095+0000 7f93d67fc700 1 -- 172.21.15.16:0/952414140 >> v1:172.21.15.16:6789/0 conn(0x7f93d80ce610 legacy=0x7f93d80c97d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:27.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:27.095+0000 7f93d67fc700 1 -- 172.21.15.16:0/952414140 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93d8162370 con 0x7f93d806c5c0 2023-10-16T12:27:27.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:27.095+0000 7f93d67fc700 1 --2- 172.21.15.16:0/952414140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93d806c5c0 0x7f93d806ca70 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f93c8009840 tx=0x7f93c8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a32a1af6b5faa34f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:27.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:27.095+0000 7f93d5ffb700 1 -- 172.21.15.16:0/952414140 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93c8011830 con 0x7f93d806c5c0 2023-10-16T12:27:27.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:27.095+0000 7f93d5ffb700 1 -- 172.21.15.16:0/952414140 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93c8011990 con 0x7f93d806c5c0 2023-10-16T12:27:27.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:27.095+0000 7f93d5ffb700 1 -- 172.21.15.16:0/952414140 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93c8011d20 con 0x7f93d806c5c0 2023-10-16T12:27:27.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:27.096+0000 7f93dd75b700 1 -- 172.21.15.16:0/952414140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93d806c5c0 msgr2=0x7f93d806ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:27.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:27.096+0000 7f93dd75b700 1 --2- 172.21.15.16:0/952414140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93d806c5c0 0x7f93d806ca70 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f93c8009840 tx=0x7f93c8009b50 comp rx=0 tx=0).stop 2023-10-16T12:27:27.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:27.096+0000 7f93dd75b700 1 -- 172.21.15.16:0/952414140 shutdown_connections 2023-10-16T12:27:27.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:27.096+0000 7f93dd75b700 1 --2- 172.21.15.16:0/952414140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93d806c5c0 0x7f93d806ca70 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:27.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:27.096+0000 7f93dd75b700 1 -- 172.21.15.16:0/952414140 >> 172.21.15.16:0/952414140 conn(0x7f93d80cf180 msgr2=0x7f93d80cf560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:27.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:27.096+0000 7f93dd75b700 1 -- 172.21.15.16:0/952414140 shutdown_connections 2023-10-16T12:27:27.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:27.096+0000 7f93dd75b700 1 -- 172.21.15.16:0/952414140 wait complete. 2023-10-16T12:27:27.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:27.098+0000 7f93dd75b700 1 Processor -- start 2023-10-16T12:27:27.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:27.098+0000 7f93dd75b700 1 -- start start 2023-10-16T12:27:27.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:27.098+0000 7f93dd75b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f93d806c5c0 0x7f93d81fb8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:27.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:27.098+0000 7f93dd75b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93d80ce610 0x7f93d815b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:27.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:27.098+0000 7f93dd75b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f93d8165610 0x7f93d80c5190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:27.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:27.098+0000 7f93dd75b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93d81fb4a0 con 0x7f93d80ce610 2023-10-16T12:27:27.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:27.098+0000 7f93dd75b700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93d80c6030 con 0x7f93d8165610 2023-10-16T12:27:27.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:27.098+0000 7f93dd75b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93d80c5d50 con 0x7f93d806c5c0 2023-10-16T12:27:27.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:27.099+0000 7f93d67fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93d80ce610 0x7f93d815b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:27.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:27.099+0000 7f93d67fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93d80ce610 0x7f93d815b810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41972/0 (socket says 172.21.15.16:41972) 2023-10-16T12:27:27.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:27.099+0000 7f93d67fc700 1 -- 172.21.15.16:0/633370885 learned_addr learned my addr 172.21.15.16:0/633370885 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:27.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:27.099+0000 7f93d77fe700 1 --2- 172.21.15.16:0/633370885 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f93d8165610 0x7f93d80c5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:27.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:27.099+0000 7f93d6ffd700 1 --2- 172.21.15.16:0/633370885 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f93d806c5c0 0x7f93d81fb8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:27.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:27.099+0000 7f93d67fc700 1 -- 172.21.15.16:0/633370885 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f93d806c5c0 msgr2=0x7f93d81fb8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:27.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:27.099+0000 7f93d67fc700 1 --2- 172.21.15.16:0/633370885 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f93d806c5c0 0x7f93d81fb8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:27.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:27.099+0000 7f93d67fc700 1 -- 172.21.15.16:0/633370885 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f93d8165610 msgr2=0x7f93d80c5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:27.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:27.099+0000 7f93d67fc700 1 --2- 172.21.15.16:0/633370885 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f93d8165610 0x7f93d80c5190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:27.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:27.099+0000 7f93d67fc700 1 -- 172.21.15.16:0/633370885 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93c80094a0 con 0x7f93d80ce610 2023-10-16T12:27:27.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:27.099+0000 7f93d6ffd700 1 --2- 172.21.15.16:0/633370885 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f93d806c5c0 0x7f93d81fb8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:27:27.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:27.099+0000 7f93d77fe700 1 --2- 172.21.15.16:0/633370885 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f93d8165610 0x7f93d80c5190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:27:27.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:27.099+0000 7f93d67fc700 1 --2- 172.21.15.16:0/633370885 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93d80ce610 0x7f93d815b810 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f93c80090b0 tx=0x7f93c8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:27.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:27.099+0000 7f93bf7fe700 1 -- 172.21.15.16:0/633370885 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93c80038a0 con 0x7f93d80ce610 2023-10-16T12:27:27.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:27.099+0000 7f93bf7fe700 1 -- 172.21.15.16:0/633370885 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93c801f070 con 0x7f93d80ce610 2023-10-16T12:27:27.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:27.099+0000 7f93bf7fe700 1 -- 172.21.15.16:0/633370885 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93c8003c60 con 0x7f93d80ce610 2023-10-16T12:27:27.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:27.099+0000 7f93dd75b700 1 -- 172.21.15.16:0/633370885 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93d81fade0 con 0x7f93d80ce610 2023-10-16T12:27:27.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:27.099+0000 7f93dd75b700 1 -- 172.21.15.16:0/633370885 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93d80c2a90 con 0x7f93d80ce610 2023-10-16T12:27:27.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:27.100+0000 7f93dd75b700 1 -- 172.21.15.16:0/633370885 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93d81fb5e0 con 0x7f93d80ce610 2023-10-16T12:27:27.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:27.101+0000 7f93bf7fe700 1 -- 172.21.15.16:0/633370885 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 50) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f93c801a430 con 0x7f93d80ce610 2023-10-16T12:27:27.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:27.101+0000 7f93bf7fe700 1 --2- 172.21.15.16:0/633370885 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f93c406af90 0x7f93c406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:27.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:27.101+0000 7f93bf7fe700 1 -- 172.21.15.16:0/633370885 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 6393+0+0 (secure 0 0 0) 0x7f93c8021070 con 0x7f93d80ce610 2023-10-16T12:27:27.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:27.101+0000 7f93d6ffd700 1 --2- 172.21.15.16:0/633370885 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f93c406af90 0x7f93c406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:27.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:27.102+0000 7f93d6ffd700 1 --2- 172.21.15.16:0/633370885 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f93c406af90 0x7f93c406d450 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f93cc007fd0 tx=0x7f93cc009040 comp rx=0 tx=0).ready entity=mgr.6365 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:27.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:27.106+0000 7f93bf7fe700 1 -- 172.21.15.16:0/633370885 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f93d81fb5e0 con 0x7f93d80ce610 2023-10-16T12:27:27.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:27.310+0000 7f93dd75b700 1 -- 172.21.15.16:0/633370885 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f93d81fb5e0 con 0x7f93d80ce610 2023-10-16T12:27:27.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:27.947+0000 7f93bf7fe700 1 -- 172.21.15.16:0/633370885 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v104) v1 ==== 169+0+0 (secure 0 0 0) 0x7f93d81fb5e0 con 0x7f93d80ce610 2023-10-16T12:27:27.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:27.955+0000 7f93bf7fe700 1 -- 172.21.15.16:0/633370885 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mgrmap(e 51) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f93c8056530 con 0x7f93d80ce610 2023-10-16T12:27:27.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:27.957+0000 7f93dd75b700 1 -- 172.21.15.16:0/633370885 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f93c406af90 msgr2=0x7f93c406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:27.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:27.957+0000 7f93dd75b700 1 --2- 172.21.15.16:0/633370885 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f93c406af90 0x7f93c406d450 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f93cc007fd0 tx=0x7f93cc009040 comp rx=0 tx=0).stop 2023-10-16T12:27:27.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:27.957+0000 7f93dd75b700 1 -- 172.21.15.16:0/633370885 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93d80ce610 msgr2=0x7f93d815b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:27.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:27.958+0000 7f93dd75b700 1 --2- 172.21.15.16:0/633370885 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93d80ce610 0x7f93d815b810 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f93c80090b0 tx=0x7f93c8004750 comp rx=0 tx=0).stop 2023-10-16T12:27:27.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:27.958+0000 7f93dd75b700 1 -- 172.21.15.16:0/633370885 shutdown_connections 2023-10-16T12:27:27.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:27.958+0000 7f93dd75b700 1 --2- 172.21.15.16:0/633370885 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f93c406af90 0x7f93c406d450 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:27.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:27.958+0000 7f93dd75b700 1 --2- 172.21.15.16:0/633370885 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f93d8165610 0x7f93d80c5190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:27.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:27.958+0000 7f93dd75b700 1 --2- 172.21.15.16:0/633370885 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f93d806c5c0 0x7f93d81fb8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:27.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:27.958+0000 7f93dd75b700 1 --2- 172.21.15.16:0/633370885 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93d80ce610 0x7f93d815b810 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:27.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:27.958+0000 7f93dd75b700 1 -- 172.21.15.16:0/633370885 >> 172.21.15.16:0/633370885 conn(0x7f93d80cf180 msgr2=0x7f93d80cc3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:27.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:27.960+0000 7f93dd75b700 1 -- 172.21.15.16:0/633370885 shutdown_connections 2023-10-16T12:27:27.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:27.960+0000 7f93dd75b700 1 -- 172.21.15.16:0/633370885 wait complete. 2023-10-16T12:27:27.963 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' removed 2023-10-16T12:27:27.977 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-16T12:27:28.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.109+0000 7f9e7198a700 1 Processor -- start 2023-10-16T12:27:28.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.109+0000 7f9e7198a700 1 -- start start 2023-10-16T12:27:28.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.110+0000 7f9e7198a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9e6c0cf8d0 0x7f9e6c0710a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:28.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.110+0000 7f9e7198a700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f9e6c0715e0 0x7f9e6c0b2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:28.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.110+0000 7f9e7198a700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e6c072350 con 0x7f9e6c1577a0 2023-10-16T12:27:28.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.110+0000 7f9e7198a700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e6c071d90 con 0x7f9e6c0cf8d0 2023-10-16T12:27:28.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.110+0000 7f9e7198a700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e6c072070 con 0x7f9e6c0715e0 2023-10-16T12:27:28.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.110+0000 7f9e6a7fc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9e6c0cf8d0 0x7f9e6c0710a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:28.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.110+0000 7f9e6a7fc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9e6c0cf8d0 0x7f9e6c0710a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41980/0 (socket says 172.21.15.16:41980) 2023-10-16T12:27:28.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.110+0000 7f9e6a7fc700 1 -- 172.21.15.16:0/3643773489 learned_addr learned my addr 172.21.15.16:0/3643773489 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:28.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.110+0000 7f9e6b7fe700 1 --2- 172.21.15.16:0/3643773489 >> v2:172.21.15.49:3300/0 conn(0x7f9e6c0715e0 0x7f9e6c0b2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:28.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.110+0000 7f9e6a7fc700 1 -- 172.21.15.16:0/3643773489 >> v2:172.21.15.49:3300/0 conn(0x7f9e6c0715e0 msgr2=0x7f9e6c0b2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:28.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.110+0000 7f9e6a7fc700 1 --2- 172.21.15.16:0/3643773489 >> v2:172.21.15.49:3300/0 conn(0x7f9e6c0715e0 0x7f9e6c0b2ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:28.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.110+0000 7f9e6a7fc700 1 -- 172.21.15.16:0/3643773489 >> v1:172.21.15.49:6789/0 conn(0x7f9e6c1577a0 legacy=0x7f9e6c0cf1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:28.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.110+0000 7f9e6a7fc700 1 -- 172.21.15.16:0/3643773489 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e6c0b22c0 con 0x7f9e6c0cf8d0 2023-10-16T12:27:28.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.110+0000 7f9e6a7fc700 1 --2- 172.21.15.16:0/3643773489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e6c0cf8d0 0x7f9e6c0710a0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c072a50 tx=0x7f9e5c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3390bc11781409c0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:28.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.111+0000 7f9e69ffb700 1 -- 172.21.15.16:0/3643773489 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e5c00b870 con 0x7f9e6c0cf8d0 2023-10-16T12:27:28.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.111+0000 7f9e69ffb700 1 -- 172.21.15.16:0/3643773489 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e5c00b9d0 con 0x7f9e6c0cf8d0 2023-10-16T12:27:28.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.111+0000 7f9e69ffb700 1 -- 172.21.15.16:0/3643773489 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e5c00bd60 con 0x7f9e6c0cf8d0 2023-10-16T12:27:28.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.111+0000 7f9e7198a700 1 -- 172.21.15.16:0/3643773489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e6c0cf8d0 msgr2=0x7f9e6c0710a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:28.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.111+0000 7f9e7198a700 1 --2- 172.21.15.16:0/3643773489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e6c0cf8d0 0x7f9e6c0710a0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c072a50 tx=0x7f9e5c009bd0 comp rx=0 tx=0).stop 2023-10-16T12:27:28.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.111+0000 7f9e7198a700 1 -- 172.21.15.16:0/3643773489 shutdown_connections 2023-10-16T12:27:28.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.111+0000 7f9e7198a700 1 --2- 172.21.15.16:0/3643773489 >> v2:172.21.15.49:3300/0 conn(0x7f9e6c0715e0 0x7f9e6c0b2ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:28.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.111+0000 7f9e7198a700 1 --2- 172.21.15.16:0/3643773489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e6c0cf8d0 0x7f9e6c0710a0 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:28.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.111+0000 7f9e7198a700 1 -- 172.21.15.16:0/3643773489 >> 172.21.15.16:0/3643773489 conn(0x7f9e6c06a7e0 msgr2=0x7f9e6c06abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:28.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.111+0000 7f9e7198a700 1 -- 172.21.15.16:0/3643773489 shutdown_connections 2023-10-16T12:27:28.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.111+0000 7f9e7198a700 1 -- 172.21.15.16:0/3643773489 wait complete. 2023-10-16T12:27:28.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.112+0000 7f9e7198a700 1 Processor -- start 2023-10-16T12:27:28.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.113+0000 7f9e7198a700 1 -- start start 2023-10-16T12:27:28.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.113+0000 7f9e7198a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9e6c0715e0 0x7f9e6c154e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:28.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.113+0000 7f9e7198a700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9e6c0cf8d0 0x7f9e6c155890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:28.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.113+0000 7f9e7198a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e6c1577a0 0x7f9e6c1497d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:28.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.113+0000 7f9e7198a700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e6c14aa40 con 0x7f9e6c1577a0 2023-10-16T12:27:28.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.113+0000 7f9e7198a700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e6c14e3a0 con 0x7f9e6c0cf8d0 2023-10-16T12:27:28.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.113+0000 7f9e7198a700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e6c155f00 con 0x7f9e6c0715e0 2023-10-16T12:27:28.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.113+0000 7f9e6b7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e6c1577a0 0x7f9e6c1497d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:28.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.113+0000 7f9e6b7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e6c1577a0 0x7f9e6c1497d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41988/0 (socket says 172.21.15.16:41988) 2023-10-16T12:27:28.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.113+0000 7f9e6b7fe700 1 -- 172.21.15.16:0/1921459706 learned_addr learned my addr 172.21.15.16:0/1921459706 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:28.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.114+0000 7f9e6a7fc700 1 --2- 172.21.15.16:0/1921459706 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9e6c0cf8d0 0x7f9e6c155890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:28.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.114+0000 7f9e6b7fe700 1 -- 172.21.15.16:0/1921459706 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9e6c0715e0 msgr2=0x7f9e6c154e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:28.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.114+0000 7f9e6b7fe700 1 --2- 172.21.15.16:0/1921459706 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9e6c0715e0 0x7f9e6c154e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:28.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.114+0000 7f9e6b7fe700 1 -- 172.21.15.16:0/1921459706 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9e6c0cf8d0 msgr2=0x7f9e6c155890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:28.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.114+0000 7f9e6b7fe700 1 --2- 172.21.15.16:0/1921459706 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9e6c0cf8d0 0x7f9e6c155890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:28.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.114+0000 7f9e6b7fe700 1 -- 172.21.15.16:0/1921459706 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e5c009550 con 0x7f9e6c1577a0 2023-10-16T12:27:28.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.114+0000 7f9e6b7fe700 1 --2- 172.21.15.16:0/1921459706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e6c1577a0 0x7f9e6c1497d0 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f9e64007e40 tx=0x7f9e6400e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:28.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.114+0000 7f9e53fff700 1 -- 172.21.15.16:0/1921459706 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e6400eea0 con 0x7f9e6c1577a0 2023-10-16T12:27:28.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.114+0000 7f9e53fff700 1 -- 172.21.15.16:0/1921459706 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e64014420 con 0x7f9e6c1577a0 2023-10-16T12:27:28.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.114+0000 7f9e7198a700 1 -- 172.21.15.16:0/1921459706 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e6c150f50 con 0x7f9e6c1577a0 2023-10-16T12:27:28.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.114+0000 7f9e53fff700 1 -- 172.21.15.16:0/1921459706 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e640147b0 con 0x7f9e6c1577a0 2023-10-16T12:27:28.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.114+0000 7f9e7198a700 1 -- 172.21.15.16:0/1921459706 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e6c151510 con 0x7f9e6c1577a0 2023-10-16T12:27:28.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.115+0000 7f9e7198a700 1 -- 172.21.15.16:0/1921459706 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e6c04ebb0 con 0x7f9e6c1577a0 2023-10-16T12:27:28.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.115+0000 7f9e53fff700 1 -- 172.21.15.16:0/1921459706 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 51) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9e64014910 con 0x7f9e6c1577a0 2023-10-16T12:27:28.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.116+0000 7f9e53fff700 1 --2- 172.21.15.16:0/1921459706 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f9e5806b020 0x7f9e5806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:28.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.116+0000 7f9e53fff700 1 -- 172.21.15.16:0/1921459706 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5979+0+0 (secure 0 0 0) 0x7f9e6408a030 con 0x7f9e6c1577a0 2023-10-16T12:27:28.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.116+0000 7f9e6affd700 1 --2- 172.21.15.16:0/1921459706 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f9e5806b020 0x7f9e5806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:28.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.116+0000 7f9e6affd700 1 --2- 172.21.15.16:0/1921459706 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f9e5806b020 0x7f9e5806d4e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9e60000f60 tx=0x7f9e60007a40 comp rx=0 tx=0).ready entity=mgr.6365 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:28.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.120+0000 7f9e53fff700 1 -- 172.21.15.16:0/1921459706 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9e64054ec0 con 0x7f9e6c1577a0 2023-10-16T12:27:28.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.324+0000 7f9e7198a700 1 -- 172.21.15.16:0/1921459706 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f9e6c0b3d70 con 0x7f9e6c1577a0 2023-10-16T12:27:28.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.327+0000 7f9e53fff700 1 -- 172.21.15.16:0/1921459706 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v31) v1 ==== 115+0+0 (secure 0 0 0) 0x7f9e640584e0 con 0x7f9e6c1577a0 2023-10-16T12:27:28.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.329+0000 7f9e7198a700 1 -- 172.21.15.16:0/1921459706 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f9e5806b020 msgr2=0x7f9e5806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:28.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.329+0000 7f9e7198a700 1 --2- 172.21.15.16:0/1921459706 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f9e5806b020 0x7f9e5806d4e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9e60000f60 tx=0x7f9e60007a40 comp rx=0 tx=0).stop 2023-10-16T12:27:28.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.329+0000 7f9e7198a700 1 -- 172.21.15.16:0/1921459706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e6c1577a0 msgr2=0x7f9e6c1497d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:28.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.329+0000 7f9e7198a700 1 --2- 172.21.15.16:0/1921459706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e6c1577a0 0x7f9e6c1497d0 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f9e64007e40 tx=0x7f9e6400e3c0 comp rx=0 tx=0).stop 2023-10-16T12:27:28.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.329+0000 7f9e7198a700 1 -- 172.21.15.16:0/1921459706 shutdown_connections 2023-10-16T12:27:28.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.329+0000 7f9e7198a700 1 --2- 172.21.15.16:0/1921459706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e6c1577a0 0x7f9e6c1497d0 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:28.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.329+0000 7f9e7198a700 1 --2- 172.21.15.16:0/1921459706 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9e6c0715e0 0x7f9e6c154e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:28.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.329+0000 7f9e7198a700 1 --2- 172.21.15.16:0/1921459706 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f9e5806b020 0x7f9e5806d4e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:28.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.329+0000 7f9e7198a700 1 --2- 172.21.15.16:0/1921459706 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9e6c0cf8d0 0x7f9e6c155890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:28.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.329+0000 7f9e7198a700 1 -- 172.21.15.16:0/1921459706 >> 172.21.15.16:0/1921459706 conn(0x7f9e6c06a7e0 msgr2=0x7f9e6c0b3250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:28.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.330+0000 7f9e7198a700 1 -- 172.21.15.16:0/1921459706 shutdown_connections 2023-10-16T12:27:28.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.330+0000 7f9e7198a700 1 -- 172.21.15.16:0/1921459706 wait complete. 2023-10-16T12:27:28.344 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-16T12:27:28.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.469+0000 7f39c2412700 1 Processor -- start 2023-10-16T12:27:28.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.469+0000 7f39c2412700 1 -- start start 2023-10-16T12:27:28.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.470+0000 7f39c2412700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f39bc15b070 0x7f39bc15b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:28.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.470+0000 7f39c2412700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f39bc06e170 0x7f39bc0cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:28.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.470+0000 7f39c2412700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39bc0cef70 con 0x7f39bc0cb880 2023-10-16T12:27:28.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.470+0000 7f39c2412700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39bc0ce360 con 0x7f39bc15b070 2023-10-16T12:27:28.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.470+0000 7f39c2412700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39bc0ce640 con 0x7f39bc06e170 2023-10-16T12:27:28.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.470+0000 7f39bbfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f39bc15b070 0x7f39bc15b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:28.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.470+0000 7f39bb7fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f39bc0cb880 0x7f39bc06da60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59498/0 (socket says 172.21.15.16:59498) 2023-10-16T12:27:28.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.470+0000 7f39bb7fe700 1 -- 172.21.15.16:0/2115211012 learned_addr learned my addr 172.21.15.16:0/2115211012 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:27:28.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.470+0000 7f39c09af700 1 --2- 172.21.15.16:0/2115211012 >> v2:172.21.15.49:3300/0 conn(0x7f39bc06e170 0x7f39bc0cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:28.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.470+0000 7f39bbfff700 1 -- 172.21.15.16:0/2115211012 >> v2:172.21.15.49:3300/0 conn(0x7f39bc06e170 msgr2=0x7f39bc0cbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:28.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.470+0000 7f39bbfff700 1 --2- 172.21.15.16:0/2115211012 >> v2:172.21.15.49:3300/0 conn(0x7f39bc06e170 0x7f39bc0cbd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:28.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.470+0000 7f39bbfff700 1 -- 172.21.15.16:0/2115211012 >> v1:172.21.15.16:6789/0 conn(0x7f39bc0cb880 legacy=0x7f39bc06da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:28.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.470+0000 7f39baffd700 1 -- 172.21.15.16:0/2115211012 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2094470262 0 0) 0x7f39bc0cef70 con 0x7f39bc0cb880 2023-10-16T12:27:28.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.470+0000 7f39bbfff700 1 -- 172.21.15.16:0/2115211012 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39bc0ce920 con 0x7f39bc15b070 2023-10-16T12:27:28.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.470+0000 7f39bbfff700 1 --2- 172.21.15.16:0/2115211012 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39bc15b070 0x7f39bc15b450 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f39bc15aad0 tx=0x7f39ac00bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60affe1d01e9f6ce server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:28.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.471+0000 7f39baffd700 1 -- 172.21.15.16:0/2115211012 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39ac00dc70 con 0x7f39bc15b070 2023-10-16T12:27:28.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.471+0000 7f39baffd700 1 -- 172.21.15.16:0/2115211012 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39ac00ddd0 con 0x7f39bc15b070 2023-10-16T12:27:28.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.471+0000 7f39baffd700 1 -- 172.21.15.16:0/2115211012 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39ac010690 con 0x7f39bc15b070 2023-10-16T12:27:28.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.471+0000 7f39c2412700 1 -- 172.21.15.16:0/2115211012 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39bc15b070 msgr2=0x7f39bc15b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:28.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.471+0000 7f39c2412700 1 --2- 172.21.15.16:0/2115211012 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39bc15b070 0x7f39bc15b450 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f39bc15aad0 tx=0x7f39ac00bdc0 comp rx=0 tx=0).stop 2023-10-16T12:27:28.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.471+0000 7f39c2412700 1 -- 172.21.15.16:0/2115211012 shutdown_connections 2023-10-16T12:27:28.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.471+0000 7f39c2412700 1 --2- 172.21.15.16:0/2115211012 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39bc15b070 0x7f39bc15b450 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:28.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.471+0000 7f39c2412700 1 --2- 172.21.15.16:0/2115211012 >> v2:172.21.15.49:3300/0 conn(0x7f39bc06e170 0x7f39bc0cbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:28.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.471+0000 7f39c2412700 1 -- 172.21.15.16:0/2115211012 >> 172.21.15.16:0/2115211012 conn(0x7f39bc0b1f40 msgr2=0x7f39bc0b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:28.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.471+0000 7f39c2412700 1 -- 172.21.15.16:0/2115211012 shutdown_connections 2023-10-16T12:27:28.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.471+0000 7f39c2412700 1 -- 172.21.15.16:0/2115211012 wait complete. 2023-10-16T12:27:28.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.472+0000 7f39c2412700 1 Processor -- start 2023-10-16T12:27:28.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.473+0000 7f39c2412700 1 -- start start 2023-10-16T12:27:28.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.473+0000 7f39c2412700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39bc06e170 0x7f39bc0c74c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:28.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.473+0000 7f39c2412700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f39bc0cb880 0x7f39bc1595a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:28.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.473+0000 7f39c2412700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f39bc15b070 0x7f39bc159ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:28.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.473+0000 7f39c2412700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39bc0c98e0 con 0x7f39bc06e170 2023-10-16T12:27:28.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.473+0000 7f39c2412700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39bc0c8450 con 0x7f39bc15b070 2023-10-16T12:27:28.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.473+0000 7f39c2412700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39bc0c7a00 con 0x7f39bc0cb880 2023-10-16T12:27:28.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.473+0000 7f39bbfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39bc06e170 0x7f39bc0c74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:28.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.473+0000 7f39bbfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39bc06e170 0x7f39bc0c74c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41998/0 (socket says 172.21.15.16:41998) 2023-10-16T12:27:28.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.473+0000 7f39bbfff700 1 -- 172.21.15.16:0/1549477424 learned_addr learned my addr 172.21.15.16:0/1549477424 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:28.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.474+0000 7f39c09af700 1 --2- 172.21.15.16:0/1549477424 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f39bc15b070 0x7f39bc159ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:28.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.474+0000 7f39bbfff700 1 -- 172.21.15.16:0/1549477424 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f39bc0cb880 msgr2=0x7f39bc1595a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:28.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.474+0000 7f39bb7fe700 1 --2- 172.21.15.16:0/1549477424 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f39bc0cb880 0x7f39bc1595a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:28.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.474+0000 7f39bbfff700 1 --2- 172.21.15.16:0/1549477424 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f39bc0cb880 0x7f39bc1595a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:28.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.474+0000 7f39bbfff700 1 -- 172.21.15.16:0/1549477424 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f39bc15b070 msgr2=0x7f39bc159ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:28.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.474+0000 7f39bbfff700 1 --2- 172.21.15.16:0/1549477424 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f39bc15b070 0x7f39bc159ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:28.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.474+0000 7f39bbfff700 1 -- 172.21.15.16:0/1549477424 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39ac00b740 con 0x7f39bc06e170 2023-10-16T12:27:28.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.474+0000 7f39bb7fe700 1 --2- 172.21.15.16:0/1549477424 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f39bc0cb880 0x7f39bc1595a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:27:28.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.474+0000 7f39bbfff700 1 --2- 172.21.15.16:0/1549477424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39bc06e170 0x7f39bc0c74c0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f39bc070620 tx=0x7f39ac004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:28.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.474+0000 7f39b8ff9700 1 -- 172.21.15.16:0/1549477424 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39ac003cf0 con 0x7f39bc06e170 2023-10-16T12:27:28.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.474+0000 7f39b8ff9700 1 -- 172.21.15.16:0/1549477424 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39ac003e50 con 0x7f39bc06e170 2023-10-16T12:27:28.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.474+0000 7f39b8ff9700 1 -- 172.21.15.16:0/1549477424 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39ac0106b0 con 0x7f39bc06e170 2023-10-16T12:27:28.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.474+0000 7f39c2412700 1 -- 172.21.15.16:0/1549477424 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39bc0c7e90 con 0x7f39bc06e170 2023-10-16T12:27:28.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.474+0000 7f39c2412700 1 -- 172.21.15.16:0/1549477424 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39bc0b3c50 con 0x7f39bc06e170 2023-10-16T12:27:28.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.475+0000 7f39c2412700 1 -- 172.21.15.16:0/1549477424 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39bc0b3ab0 con 0x7f39bc06e170 2023-10-16T12:27:28.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.476+0000 7f39b8ff9700 1 -- 172.21.15.16:0/1549477424 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 51) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f39ac019400 con 0x7f39bc06e170 2023-10-16T12:27:28.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.476+0000 7f39b8ff9700 1 --2- 172.21.15.16:0/1549477424 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f39a806b020 0x7f39a806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:28.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.476+0000 7f39b8ff9700 1 -- 172.21.15.16:0/1549477424 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5979+0+0 (secure 0 0 0) 0x7f39ac08b4e0 con 0x7f39bc06e170 2023-10-16T12:27:28.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.476+0000 7f39bb7fe700 1 --2- 172.21.15.16:0/1549477424 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f39a806b020 0x7f39a806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:28.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.477+0000 7f39bb7fe700 1 --2- 172.21.15.16:0/1549477424 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f39a806b020 0x7f39a806d4e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f39bc0b3950 tx=0x7f39b0009040 comp rx=0 tx=0).ready entity=mgr.6365 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:28.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.481+0000 7f39b8ff9700 1 -- 172.21.15.16:0/1549477424 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f39bc0b3ab0 con 0x7f39bc06e170 2023-10-16T12:27:28.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.694+0000 7f39c2412700 1 -- 172.21.15.16:0/1549477424 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f39bc0b3ab0 con 0x7f39bc06e170 2023-10-16T12:27:28.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.697+0000 7f39b8ff9700 1 -- 172.21.15.16:0/1549477424 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v32) v1 ==== 104+0+0 (secure 0 0 0) 0x7f39bc0b3ab0 con 0x7f39bc06e170 2023-10-16T12:27:28.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.700+0000 7f39c2412700 1 -- 172.21.15.16:0/1549477424 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f39a806b020 msgr2=0x7f39a806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:28.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.700+0000 7f39c2412700 1 --2- 172.21.15.16:0/1549477424 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f39a806b020 0x7f39a806d4e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f39bc0b3950 tx=0x7f39b0009040 comp rx=0 tx=0).stop 2023-10-16T12:27:28.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.700+0000 7f39c2412700 1 -- 172.21.15.16:0/1549477424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39bc06e170 msgr2=0x7f39bc0c74c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:28.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.700+0000 7f39c2412700 1 --2- 172.21.15.16:0/1549477424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39bc06e170 0x7f39bc0c74c0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f39bc070620 tx=0x7f39ac004750 comp rx=0 tx=0).stop 2023-10-16T12:27:28.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.700+0000 7f39c2412700 1 -- 172.21.15.16:0/1549477424 shutdown_connections 2023-10-16T12:27:28.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.700+0000 7f39c2412700 1 --2- 172.21.15.16:0/1549477424 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f39a806b020 0x7f39a806d4e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:28.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.700+0000 7f39c2412700 1 --2- 172.21.15.16:0/1549477424 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f39bc15b070 0x7f39bc159ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:28.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.700+0000 7f39c2412700 1 --2- 172.21.15.16:0/1549477424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39bc06e170 0x7f39bc0c74c0 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:28.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.700+0000 7f39c2412700 1 --2- 172.21.15.16:0/1549477424 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f39bc0cb880 0x7f39bc1595a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:28.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.700+0000 7f39c2412700 1 -- 172.21.15.16:0/1549477424 >> 172.21.15.16:0/1549477424 conn(0x7f39bc0b1f40 msgr2=0x7f39bc1548f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:28.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.700+0000 7f39c2412700 1 -- 172.21.15.16:0/1549477424 shutdown_connections 2023-10-16T12:27:28.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.700+0000 7f39c2412700 1 -- 172.21.15.16:0/1549477424 wait complete. 2023-10-16T12:27:28.714 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-16T12:27:28.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.839+0000 7f1c26929700 1 Processor -- start 2023-10-16T12:27:28.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.839+0000 7f1c26929700 1 -- start start 2023-10-16T12:27:28.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.840+0000 7f1c26929700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f1c2006beb0 0x7f1c200ce890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:28.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.840+0000 7f1c26929700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c2006aa10 con 0x7f1c200af210 2023-10-16T12:27:28.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.840+0000 7f1c26929700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c200ceed0 con 0x7f1c20160ee0 2023-10-16T12:27:28.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.840+0000 7f1c26929700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c200cf770 con 0x7f1c2006beb0 2023-10-16T12:27:28.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.840+0000 7f1c1f7fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f1c200af210 0x7f1c200ce180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59512/0 (socket says 172.21.15.16:59512) 2023-10-16T12:27:28.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.840+0000 7f1c1f7fe700 1 -- 172.21.15.16:0/1201268579 learned_addr learned my addr 172.21.15.16:0/1201268579 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:27:28.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.841+0000 7f1c24ec6700 1 --2- 172.21.15.16:0/1201268579 >> v2:172.21.15.49:3300/0 conn(0x7f1c2006beb0 0x7f1c200ce890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:28.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.841+0000 7f1c1effd700 1 -- 172.21.15.16:0/1201268579 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2866282217 0 0) 0x7f1c2006aa10 con 0x7f1c200af210 2023-10-16T12:27:28.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.841+0000 7f1c1effd700 1 -- 172.21.15.16:0/1201268579 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c0c0035b0 con 0x7f1c200af210 2023-10-16T12:27:28.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.841+0000 7f1c1effd700 1 -- 172.21.15.16:0/1201268579 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 547643572 0 0) 0x7f1c0c0035b0 con 0x7f1c200af210 2023-10-16T12:27:28.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.841+0000 7f1c1effd700 1 -- 172.21.15.16:0/1201268579 >> v2:172.21.15.49:3300/0 conn(0x7f1c2006beb0 msgr2=0x7f1c200ce890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:28.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.841+0000 7f1c1effd700 1 --2- 172.21.15.16:0/1201268579 >> v2:172.21.15.49:3300/0 conn(0x7f1c2006beb0 0x7f1c200ce890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:28.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.841+0000 7f1c1effd700 1 -- 172.21.15.16:0/1201268579 >> v1:172.21.15.49:6789/0 conn(0x7f1c20160ee0 legacy=0x7f1c200aeb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:28.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.841+0000 7f1c1effd700 1 -- 172.21.15.16:0/1201268579 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c200cf1b0 con 0x7f1c200af210 2023-10-16T12:27:28.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.841+0000 7f1c24ec6700 1 --2- 172.21.15.16:0/1201268579 >> v2:172.21.15.49:3300/0 conn(0x7f1c2006beb0 0x7f1c200ce890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:27:28.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.841+0000 7f1c1effd700 1 -- 172.21.15.16:0/1201268579 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f1c10001670 con 0x7f1c200af210 2023-10-16T12:27:28.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.841+0000 7f1c1effd700 1 -- 172.21.15.16:0/1201268579 >> v1:172.21.15.16:6789/0 conn(0x7f1c200af210 legacy=0x7f1c200ce180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:28.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.841+0000 7f1c1effd700 1 --2- 172.21.15.16:0/1201268579 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1c0c003a00 0x7f1c0c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:28.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.841+0000 7f1c1effd700 1 --2- 172.21.15.16:0/1201268579 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c0c004850 0x7f1c0c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:28.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.841+0000 7f1c1effd700 1 --2- 172.21.15.16:0/1201268579 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1c0c005210 0x7f1c0c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:28.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.841+0000 7f1c1effd700 1 -- 172.21.15.16:0/1201268579 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c200cf770 con 0x7f1c0c004850 2023-10-16T12:27:28.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.841+0000 7f1c1effd700 1 -- 172.21.15.16:0/1201268579 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c0c00c0b0 con 0x7f1c0c005210 2023-10-16T12:27:28.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.841+0000 7f1c1effd700 1 -- 172.21.15.16:0/1201268579 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c0c00c270 con 0x7f1c0c003a00 2023-10-16T12:27:28.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.841+0000 7f1c1f7fe700 1 --2- 172.21.15.16:0/1201268579 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c0c004850 0x7f1c0c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:28.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.841+0000 7f1c1ffff700 1 --2- 172.21.15.16:0/1201268579 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1c0c003a00 0x7f1c0c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:28.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.841+0000 7f1c24ec6700 1 --2- 172.21.15.16:0/1201268579 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1c0c005210 0x7f1c0c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:28.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.841+0000 7f1c1f7fe700 1 -- 172.21.15.16:0/1201268579 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1c0c003a00 msgr2=0x7f1c0c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:28.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.841+0000 7f1c1f7fe700 1 --2- 172.21.15.16:0/1201268579 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1c0c003a00 0x7f1c0c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:28.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.841+0000 7f1c1f7fe700 1 -- 172.21.15.16:0/1201268579 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1c0c005210 msgr2=0x7f1c0c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:28.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.841+0000 7f1c1f7fe700 1 --2- 172.21.15.16:0/1201268579 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1c0c005210 0x7f1c0c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:28.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.841+0000 7f1c1f7fe700 1 -- 172.21.15.16:0/1201268579 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c0c0035b0 con 0x7f1c0c004850 2023-10-16T12:27:28.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.841+0000 7f1c24ec6700 1 -- 172.21.15.16:0/1201268579 reap_dead start 2023-10-16T12:27:28.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.842+0000 7f1c1f7fe700 1 --2- 172.21.15.16:0/1201268579 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c0c004850 0x7f1c0c004cd0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f1c1000c240 tx=0x7f1c0c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69cb8fe6bf7c40e4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:28.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.842+0000 7f1c1effd700 1 -- 172.21.15.16:0/1201268579 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c10007c10 con 0x7f1c0c004850 2023-10-16T12:27:28.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.842+0000 7f1c1effd700 1 -- 172.21.15.16:0/1201268579 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c1000cd80 con 0x7f1c0c004850 2023-10-16T12:27:28.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.842+0000 7f1c1effd700 1 -- 172.21.15.16:0/1201268579 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c10005ba0 con 0x7f1c0c004850 2023-10-16T12:27:28.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.842+0000 7f1c26929700 1 -- 172.21.15.16:0/1201268579 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c0c004850 msgr2=0x7f1c0c004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:28.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.842+0000 7f1c26929700 1 --2- 172.21.15.16:0/1201268579 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c0c004850 0x7f1c0c004cd0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f1c1000c240 tx=0x7f1c0c005880 comp rx=0 tx=0).stop 2023-10-16T12:27:28.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.842+0000 7f1c26929700 1 -- 172.21.15.16:0/1201268579 shutdown_connections 2023-10-16T12:27:28.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.842+0000 7f1c26929700 1 --2- 172.21.15.16:0/1201268579 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c0c004850 0x7f1c0c004cd0 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:28.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.842+0000 7f1c26929700 1 -- 172.21.15.16:0/1201268579 >> 172.21.15.16:0/1201268579 conn(0x7f1c200cc5e0 msgr2=0x7f1c200cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:28.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.842+0000 7f1c26929700 1 -- 172.21.15.16:0/1201268579 shutdown_connections 2023-10-16T12:27:28.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.842+0000 7f1c26929700 1 -- 172.21.15.16:0/1201268579 wait complete. 2023-10-16T12:27:28.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.844+0000 7f1c26929700 1 Processor -- start 2023-10-16T12:27:28.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.844+0000 7f1c26929700 1 -- start start 2023-10-16T12:27:28.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.845+0000 7f1c26929700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1c0c004850 0x7f1c200c5d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:28.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.845+0000 7f1c26929700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c200c75f0 0x7f1c201560b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:28.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.845+0000 7f1c26929700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1c201572b0 0x7f1c20157760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:28.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.845+0000 7f1c26929700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c200c7d90 con 0x7f1c200c75f0 2023-10-16T12:27:28.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.845+0000 7f1c26929700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c200c8070 con 0x7f1c0c004850 2023-10-16T12:27:28.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.845+0000 7f1c26929700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c200c8350 con 0x7f1c201572b0 2023-10-16T12:27:28.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.845+0000 7f1c1f7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c200c75f0 0x7f1c201560b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:28.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.845+0000 7f1c1f7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c200c75f0 0x7f1c201560b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42020/0 (socket says 172.21.15.16:42020) 2023-10-16T12:27:28.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.845+0000 7f1c1f7fe700 1 -- 172.21.15.16:0/2457650709 learned_addr learned my addr 172.21.15.16:0/2457650709 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:28.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.845+0000 7f1c1ffff700 1 --2- 172.21.15.16:0/2457650709 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1c0c004850 0x7f1c200c5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:28.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.845+0000 7f1c1f7fe700 1 -- 172.21.15.16:0/2457650709 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1c201572b0 msgr2=0x7f1c20157760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:28.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.845+0000 7f1c24ec6700 1 --2- 172.21.15.16:0/2457650709 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1c201572b0 0x7f1c20157760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:28.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.845+0000 7f1c1f7fe700 1 --2- 172.21.15.16:0/2457650709 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1c201572b0 0x7f1c20157760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:28.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.845+0000 7f1c1f7fe700 1 -- 172.21.15.16:0/2457650709 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1c0c004850 msgr2=0x7f1c200c5d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:28.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.845+0000 7f1c1f7fe700 1 --2- 172.21.15.16:0/2457650709 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1c0c004850 0x7f1c200c5d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:28.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.845+0000 7f1c1f7fe700 1 -- 172.21.15.16:0/2457650709 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c200c8630 con 0x7f1c200c75f0 2023-10-16T12:27:28.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.845+0000 7f1c1ffff700 1 --2- 172.21.15.16:0/2457650709 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1c0c004850 0x7f1c200c5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:27:28.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.845+0000 7f1c1f7fe700 1 --2- 172.21.15.16:0/2457650709 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c200c75f0 0x7f1c201560b0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f1c1000c830 tx=0x7f1c10001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:28.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.845+0000 7f1c1cff9700 1 -- 172.21.15.16:0/2457650709 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c10001630 con 0x7f1c200c75f0 2023-10-16T12:27:28.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.845+0000 7f1c1cff9700 1 -- 172.21.15.16:0/2457650709 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c100070d0 con 0x7f1c200c75f0 2023-10-16T12:27:28.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.845+0000 7f1c26929700 1 -- 172.21.15.16:0/2457650709 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c200c8910 con 0x7f1c200c75f0 2023-10-16T12:27:28.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.845+0000 7f1c1cff9700 1 -- 172.21.15.16:0/2457650709 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c10007480 con 0x7f1c200c75f0 2023-10-16T12:27:28.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.845+0000 7f1c26929700 1 -- 172.21.15.16:0/2457650709 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c201565f0 con 0x7f1c200c75f0 2023-10-16T12:27:28.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.846+0000 7f1c26929700 1 -- 172.21.15.16:0/2457650709 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c200c8c80 con 0x7f1c200c75f0 2023-10-16T12:27:28.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.847+0000 7f1c1cff9700 1 -- 172.21.15.16:0/2457650709 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 51) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1c100075e0 con 0x7f1c200c75f0 2023-10-16T12:27:28.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.847+0000 7f1c1cff9700 1 --2- 172.21.15.16:0/2457650709 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f1c0006b020 0x7f1c0006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:28.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.847+0000 7f1c1cff9700 1 -- 172.21.15.16:0/2457650709 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5979+0+0 (secure 0 0 0) 0x7f1c1008af30 con 0x7f1c200c75f0 2023-10-16T12:27:28.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.847+0000 7f1c1ffff700 1 --2- 172.21.15.16:0/2457650709 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f1c0006b020 0x7f1c0006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:28.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.848+0000 7f1c1ffff700 1 --2- 172.21.15.16:0/2457650709 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f1c0006b020 0x7f1c0006d4e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f1c08000c00 tx=0x7f1c08009040 comp rx=0 tx=0).ready entity=mgr.6365 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:28.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:28.852+0000 7f1c1cff9700 1 -- 172.21.15.16:0/2457650709 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1c10055ef0 con 0x7f1c200c75f0 2023-10-16T12:27:29.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.055+0000 7f1c26929700 1 -- 172.21.15.16:0/2457650709 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f1c2004ebb0 con 0x7f1c200c75f0 2023-10-16T12:27:29.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.058+0000 7f1c1cff9700 1 -- 172.21.15.16:0/2457650709 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v33) v1 ==== 120+0+0 (secure 0 0 0) 0x7f1c10059510 con 0x7f1c200c75f0 2023-10-16T12:27:29.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.060+0000 7f1c26929700 1 -- 172.21.15.16:0/2457650709 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f1c0006b020 msgr2=0x7f1c0006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:29.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.060+0000 7f1c26929700 1 --2- 172.21.15.16:0/2457650709 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f1c0006b020 0x7f1c0006d4e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f1c08000c00 tx=0x7f1c08009040 comp rx=0 tx=0).stop 2023-10-16T12:27:29.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.060+0000 7f1c26929700 1 -- 172.21.15.16:0/2457650709 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c200c75f0 msgr2=0x7f1c201560b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:29.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.060+0000 7f1c26929700 1 --2- 172.21.15.16:0/2457650709 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c200c75f0 0x7f1c201560b0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f1c1000c830 tx=0x7f1c10001a00 comp rx=0 tx=0).stop 2023-10-16T12:27:29.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.060+0000 7f1c26929700 1 -- 172.21.15.16:0/2457650709 shutdown_connections 2023-10-16T12:27:29.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.060+0000 7f1c26929700 1 --2- 172.21.15.16:0/2457650709 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1c201572b0 0x7f1c20157760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:29.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.060+0000 7f1c26929700 1 --2- 172.21.15.16:0/2457650709 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f1c0006b020 0x7f1c0006d4e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:29.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.060+0000 7f1c26929700 1 --2- 172.21.15.16:0/2457650709 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1c0c004850 0x7f1c200c5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:29.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.060+0000 7f1c26929700 1 --2- 172.21.15.16:0/2457650709 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c200c75f0 0x7f1c201560b0 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:29.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.060+0000 7f1c26929700 1 -- 172.21.15.16:0/2457650709 >> 172.21.15.16:0/2457650709 conn(0x7f1c200cc5e0 msgr2=0x7f1c200710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:29.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.060+0000 7f1c26929700 1 -- 172.21.15.16:0/2457650709 shutdown_connections 2023-10-16T12:27:29.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.060+0000 7f1c26929700 1 -- 172.21.15.16:0/2457650709 wait complete. 2023-10-16T12:27:29.075 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list' 2023-10-16T12:27:29.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.197+0000 7fdebfbfc700 1 Processor -- start 2023-10-16T12:27:29.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.197+0000 7fdebfbfc700 1 -- start start 2023-10-16T12:27:29.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.198+0000 7fdebfbfc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdeb80cf880 0x7fdeb80b52f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:29.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.198+0000 7fdebfbfc700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdeb815bc80 con 0x7fdeb80b5830 2023-10-16T12:27:29.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.198+0000 7fdebfbfc700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdeb815b070 con 0x7fdeb80cf4a0 2023-10-16T12:27:29.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.198+0000 7fdebfbfc700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdeb815b350 con 0x7fdeb80cf880 2023-10-16T12:27:29.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.198+0000 7fdebd197700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdeb80cf880 0x7fdeb80b52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:29.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.198+0000 7fdebe199700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fdeb80b5830 0x7fdeb815a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59536/0 (socket says 172.21.15.16:59536) 2023-10-16T12:27:29.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.198+0000 7fdebe199700 1 -- 172.21.15.16:0/1720895515 learned_addr learned my addr 172.21.15.16:0/1720895515 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:27:29.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.199+0000 7fdebd197700 1 -- 172.21.15.16:0/1720895515 >> v1:172.21.15.49:6789/0 conn(0x7fdeb80cf4a0 legacy=0x7fdeb80726e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:29.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.199+0000 7fdebd197700 1 -- 172.21.15.16:0/1720895515 >> v1:172.21.15.16:6789/0 conn(0x7fdeb80b5830 legacy=0x7fdeb815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:29.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.199+0000 7fdebd197700 1 -- 172.21.15.16:0/1720895515 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdeb815b630 con 0x7fdeb80cf880 2023-10-16T12:27:29.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.199+0000 7fdebd197700 1 --2- 172.21.15.16:0/1720895515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdeb80cf880 0x7fdeb80b52f0 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7fdea8009840 tx=0x7fdea8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=537771993fa9bf16 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:29.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.199+0000 7fdebc996700 1 -- 172.21.15.16:0/1720895515 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea8011830 con 0x7fdeb80cf880 2023-10-16T12:27:29.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.199+0000 7fdebc996700 1 -- 172.21.15.16:0/1720895515 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdea8011990 con 0x7fdeb80cf880 2023-10-16T12:27:29.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.199+0000 7fdebc996700 1 -- 172.21.15.16:0/1720895515 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea8011d20 con 0x7fdeb80cf880 2023-10-16T12:27:29.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.200+0000 7fdebfbfc700 1 -- 172.21.15.16:0/1720895515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdeb80cf880 msgr2=0x7fdeb80b52f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:29.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.200+0000 7fdebfbfc700 1 --2- 172.21.15.16:0/1720895515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdeb80cf880 0x7fdeb80b52f0 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7fdea8009840 tx=0x7fdea8009b50 comp rx=0 tx=0).stop 2023-10-16T12:27:29.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.200+0000 7fdebfbfc700 1 -- 172.21.15.16:0/1720895515 shutdown_connections 2023-10-16T12:27:29.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.200+0000 7fdebfbfc700 1 --2- 172.21.15.16:0/1720895515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdeb80cf880 0x7fdeb80b52f0 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:29.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.200+0000 7fdebfbfc700 1 -- 172.21.15.16:0/1720895515 >> 172.21.15.16:0/1720895515 conn(0x7fdeb80b1fd0 msgr2=0x7fdeb80b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:29.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.200+0000 7fdebfbfc700 1 -- 172.21.15.16:0/1720895515 shutdown_connections 2023-10-16T12:27:29.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.200+0000 7fdebfbfc700 1 -- 172.21.15.16:0/1720895515 wait complete. 2023-10-16T12:27:29.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.201+0000 7fdebfbfc700 1 Processor -- start 2023-10-16T12:27:29.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.201+0000 7fdebfbfc700 1 -- start start 2023-10-16T12:27:29.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.202+0000 7fdebfbfc700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdeb80b5830 0x7fdeb80ae010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:29.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.202+0000 7fdebfbfc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdeb80cf4a0 0x7fdeb80ae770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:29.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.202+0000 7fdebfbfc700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdeb80cf880 0x7fdeb8158e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:29.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.202+0000 7fdebfbfc700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb80ad2e0 con 0x7fdeb80cf4a0 2023-10-16T12:27:29.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.202+0000 7fdebfbfc700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb8159f60 con 0x7fdeb80cf880 2023-10-16T12:27:29.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.202+0000 7fdebfbfc700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb80c5610 con 0x7fdeb80b5830 2023-10-16T12:27:29.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.202+0000 7fdebd197700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdeb80cf4a0 0x7fdeb80ae770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:29.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.202+0000 7fdebd197700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdeb80cf4a0 0x7fdeb80ae770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42034/0 (socket says 172.21.15.16:42034) 2023-10-16T12:27:29.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.202+0000 7fdebd197700 1 -- 172.21.15.16:0/721571140 learned_addr learned my addr 172.21.15.16:0/721571140 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:29.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.202+0000 7fdebe199700 1 --2- 172.21.15.16:0/721571140 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdeb80cf880 0x7fdeb8158e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:29.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.202+0000 7fdebd998700 1 --2- 172.21.15.16:0/721571140 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdeb80b5830 0x7fdeb80ae010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:29.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.202+0000 7fdebd197700 1 -- 172.21.15.16:0/721571140 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdeb80b5830 msgr2=0x7fdeb80ae010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:29.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.202+0000 7fdebd197700 1 --2- 172.21.15.16:0/721571140 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdeb80b5830 0x7fdeb80ae010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:29.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.202+0000 7fdebd197700 1 -- 172.21.15.16:0/721571140 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdeb80cf880 msgr2=0x7fdeb8158e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:29.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.202+0000 7fdebd197700 1 --2- 172.21.15.16:0/721571140 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdeb80cf880 0x7fdeb8158e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:29.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.202+0000 7fdebd197700 1 -- 172.21.15.16:0/721571140 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdea80094a0 con 0x7fdeb80cf4a0 2023-10-16T12:27:29.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.202+0000 7fdebd197700 1 --2- 172.21.15.16:0/721571140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdeb80cf4a0 0x7fdeb80ae770 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7fdea80090b0 tx=0x7fdea8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:29.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.203+0000 7fdeae7fc700 1 -- 172.21.15.16:0/721571140 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea80036a0 con 0x7fdeb80cf4a0 2023-10-16T12:27:29.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.203+0000 7fdeae7fc700 1 -- 172.21.15.16:0/721571140 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdea801e070 con 0x7fdeb80cf4a0 2023-10-16T12:27:29.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.203+0000 7fdebfbfc700 1 -- 172.21.15.16:0/721571140 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdeb80ad530 con 0x7fdeb80cf4a0 2023-10-16T12:27:29.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.203+0000 7fdeae7fc700 1 -- 172.21.15.16:0/721571140 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea8003a60 con 0x7fdeb80cf4a0 2023-10-16T12:27:29.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.203+0000 7fdebfbfc700 1 -- 172.21.15.16:0/721571140 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdeb80adaf0 con 0x7fdeb80cf4a0 2023-10-16T12:27:29.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.203+0000 7fdebfbfc700 1 -- 172.21.15.16:0/721571140 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdeb8067b10 con 0x7fdeb80cf4a0 2023-10-16T12:27:29.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.204+0000 7fdeae7fc700 1 -- 172.21.15.16:0/721571140 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 51) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fdea8003bc0 con 0x7fdeb80cf4a0 2023-10-16T12:27:29.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.205+0000 7fdeae7fc700 1 --2- 172.21.15.16:0/721571140 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fde9c06b050 0x7fde9c06d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:29.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.205+0000 7fdebd998700 1 --2- 172.21.15.16:0/721571140 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fde9c06b050 0x7fde9c06d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:29.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.205+0000 7fdeae7fc700 1 -- 172.21.15.16:0/721571140 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5979+0+0 (secure 0 0 0) 0x7fdea808a8c0 con 0x7fdeb80cf4a0 2023-10-16T12:27:29.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.205+0000 7fdebd998700 1 --2- 172.21.15.16:0/721571140 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fde9c06b050 0x7fde9c06d510 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fdeb4007fd0 tx=0x7fdeb4009040 comp rx=0 tx=0).ready entity=mgr.6365 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:29.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.209+0000 7fdeae7fc700 1 -- 172.21.15.16:0/721571140 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdea8055750 con 0x7fdeb80cf4a0 2023-10-16T12:27:29.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.498+0000 7fdebfbfc700 1 -- 172.21.15.16:0/721571140 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]} v 0) v1 -- 0x7fdeb804ebb0 con 0x7fdeb80cf4a0 2023-10-16T12:27:29.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.949+0000 7fdeae7fc700 1 -- 172.21.15.16:0/721571140 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]}]=0 v632) v1 ==== 166+0+0 (secure 0 0 0) 0x7fdea80150e0 con 0x7fdeb80cf4a0 2023-10-16T12:27:29.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.953+0000 7fdebfbfc700 1 -- 172.21.15.16:0/721571140 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fde9c06b050 msgr2=0x7fde9c06d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:29.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.953+0000 7fdebfbfc700 1 --2- 172.21.15.16:0/721571140 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fde9c06b050 0x7fde9c06d510 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fdeb4007fd0 tx=0x7fdeb4009040 comp rx=0 tx=0).stop 2023-10-16T12:27:29.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.953+0000 7fdebfbfc700 1 -- 172.21.15.16:0/721571140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdeb80cf4a0 msgr2=0x7fdeb80ae770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:29.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.953+0000 7fdebfbfc700 1 --2- 172.21.15.16:0/721571140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdeb80cf4a0 0x7fdeb80ae770 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7fdea80090b0 tx=0x7fdea8004750 comp rx=0 tx=0).stop 2023-10-16T12:27:29.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.953+0000 7fdebfbfc700 1 -- 172.21.15.16:0/721571140 shutdown_connections 2023-10-16T12:27:29.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.953+0000 7fdebfbfc700 1 --2- 172.21.15.16:0/721571140 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fde9c06b050 0x7fde9c06d510 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:29.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.953+0000 7fdebfbfc700 1 --2- 172.21.15.16:0/721571140 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdeb80cf880 0x7fdeb8158e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:29.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.953+0000 7fdebfbfc700 1 --2- 172.21.15.16:0/721571140 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdeb80b5830 0x7fdeb80ae010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:29.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.953+0000 7fdebfbfc700 1 --2- 172.21.15.16:0/721571140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdeb80cf4a0 0x7fdeb80ae770 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:29.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.953+0000 7fdebfbfc700 1 -- 172.21.15.16:0/721571140 >> 172.21.15.16:0/721571140 conn(0x7fdeb80b1fd0 msgr2=0x7fdeb80b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:29.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.953+0000 7fdebfbfc700 1 -- 172.21.15.16:0/721571140 shutdown_connections 2023-10-16T12:27:29.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:29.953+0000 7fdebfbfc700 1 -- 172.21.15.16:0/721571140 wait complete. 2023-10-16T12:27:29.971 INFO:tasks.cephfs_test_runner:Test listing snap schedules on a non-existing filesystem path failure ... ok 2023-10-16T12:27:29.971 INFO:tasks.cephfs_test_runner:Starting test: test_schedule_auto_deactivation_for_non_existent_path (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test that a non-existent path leads to schedule deactivation after a few retries. 2023-10-16T12:27:29.972 INFO:tasks.cephfs_test_runner:test_schedule_auto_deactivation_for_non_existent_path (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-16T12:27:29.972 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:27:30.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.102+0000 7f3e6af9a700 1 Processor -- start 2023-10-16T12:27:30.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.102+0000 7f3e6af9a700 1 -- start start 2023-10-16T12:27:30.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.103+0000 7f3e6af9a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3e640ce4f0 0x7f3e640cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:30.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.103+0000 7f3e6af9a700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e640cc2a0 con 0x7f3e6415ba10 2023-10-16T12:27:30.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.103+0000 7f3e6af9a700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e640cc540 con 0x7f3e6415b630 2023-10-16T12:27:30.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.103+0000 7f3e6af9a700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e640cdb20 con 0x7f3e640ce4f0 2023-10-16T12:27:30.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.103+0000 7f3e63fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3e640ce4f0 0x7f3e640cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:30.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.103+0000 7f3e69537700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f3e6415ba10 0x7f3e640cfca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59550/0 (socket says 172.21.15.16:59550) 2023-10-16T12:27:30.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.103+0000 7f3e69537700 1 -- 172.21.15.16:0/2950778808 learned_addr learned my addr 172.21.15.16:0/2950778808 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:27:30.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.104+0000 7f3e63fff700 1 -- 172.21.15.16:0/2950778808 >> v1:172.21.15.49:6789/0 conn(0x7f3e6415b630 legacy=0x7f3e640cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:30.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.104+0000 7f3e63fff700 1 -- 172.21.15.16:0/2950778808 >> v1:172.21.15.16:6789/0 conn(0x7f3e6415ba10 legacy=0x7f3e640cfca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:30.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.104+0000 7f3e637fe700 1 -- 172.21.15.16:0/2950778808 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 742705632 0 0) 0x7f3e640cc2a0 con 0x7f3e6415ba10 2023-10-16T12:27:30.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.104+0000 7f3e63fff700 1 -- 172.21.15.16:0/2950778808 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e640cc820 con 0x7f3e640ce4f0 2023-10-16T12:27:30.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.104+0000 7f3e63fff700 1 --2- 172.21.15.16:0/2950778808 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e640ce4f0 0x7f3e640cd5e0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f3e50009840 tx=0x7f3e50009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=358036ab930a4a40 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:30.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.104+0000 7f3e637fe700 1 -- 172.21.15.16:0/2950778808 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e50011830 con 0x7f3e640ce4f0 2023-10-16T12:27:30.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.104+0000 7f3e637fe700 1 -- 172.21.15.16:0/2950778808 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e50011990 con 0x7f3e640ce4f0 2023-10-16T12:27:30.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.104+0000 7f3e637fe700 1 -- 172.21.15.16:0/2950778808 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e50011d20 con 0x7f3e640ce4f0 2023-10-16T12:27:30.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.104+0000 7f3e6af9a700 1 -- 172.21.15.16:0/2950778808 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e640ce4f0 msgr2=0x7f3e640cd5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:30.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.104+0000 7f3e6af9a700 1 --2- 172.21.15.16:0/2950778808 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e640ce4f0 0x7f3e640cd5e0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f3e50009840 tx=0x7f3e50009b50 comp rx=0 tx=0).stop 2023-10-16T12:27:30.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.105+0000 7f3e6af9a700 1 -- 172.21.15.16:0/2950778808 shutdown_connections 2023-10-16T12:27:30.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.105+0000 7f3e6af9a700 1 --2- 172.21.15.16:0/2950778808 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e640ce4f0 0x7f3e640cd5e0 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:30.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.105+0000 7f3e6af9a700 1 -- 172.21.15.16:0/2950778808 >> 172.21.15.16:0/2950778808 conn(0x7f3e640d14b0 msgr2=0x7f3e640d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:30.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.105+0000 7f3e6af9a700 1 -- 172.21.15.16:0/2950778808 shutdown_connections 2023-10-16T12:27:30.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.105+0000 7f3e6af9a700 1 -- 172.21.15.16:0/2950778808 wait complete. 2023-10-16T12:27:30.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.106+0000 7f3e6af9a700 1 Processor -- start 2023-10-16T12:27:30.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.106+0000 7f3e6af9a700 1 -- start start 2023-10-16T12:27:30.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.107+0000 7f3e6af9a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e640ce4f0 0x7f3e6415ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:30.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.107+0000 7f3e6af9a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3e6415b630 0x7f3e641f04d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:30.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.107+0000 7f3e6af9a700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3e6415ba10 0x7f3e640c9510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:30.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.107+0000 7f3e6af9a700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e64153580 con 0x7f3e640ce4f0 2023-10-16T12:27:30.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.107+0000 7f3e6af9a700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e640c9e60 con 0x7f3e6415ba10 2023-10-16T12:27:30.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.107+0000 7f3e6af9a700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e640ca140 con 0x7f3e6415b630 2023-10-16T12:27:30.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.107+0000 7f3e68d36700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e640ce4f0 0x7f3e6415ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:30.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.107+0000 7f3e68d36700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e640ce4f0 0x7f3e6415ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42046/0 (socket says 172.21.15.16:42046) 2023-10-16T12:27:30.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.107+0000 7f3e68d36700 1 -- 172.21.15.16:0/1634289049 learned_addr learned my addr 172.21.15.16:0/1634289049 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:30.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.107+0000 7f3e69537700 1 --2- 172.21.15.16:0/1634289049 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3e6415ba10 0x7f3e640c9510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:30.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.107+0000 7f3e63fff700 1 --2- 172.21.15.16:0/1634289049 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3e6415b630 0x7f3e641f04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:30.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.107+0000 7f3e68d36700 1 -- 172.21.15.16:0/1634289049 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3e6415b630 msgr2=0x7f3e641f04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:30.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.107+0000 7f3e68d36700 1 --2- 172.21.15.16:0/1634289049 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3e6415b630 0x7f3e641f04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:30.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.107+0000 7f3e68d36700 1 -- 172.21.15.16:0/1634289049 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3e6415ba10 msgr2=0x7f3e640c9510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:30.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.107+0000 7f3e68d36700 1 --2- 172.21.15.16:0/1634289049 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3e6415ba10 0x7f3e640c9510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:30.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.107+0000 7f3e68d36700 1 -- 172.21.15.16:0/1634289049 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e500094a0 con 0x7f3e640ce4f0 2023-10-16T12:27:30.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.107+0000 7f3e63fff700 1 --2- 172.21.15.16:0/1634289049 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3e6415b630 0x7f3e641f04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:27:30.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.107+0000 7f3e68d36700 1 --2- 172.21.15.16:0/1634289049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e640ce4f0 0x7f3e6415ce60 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f3e5800c4d0 tx=0x7f3e5800c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:30.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.108+0000 7f3e617fa700 1 -- 172.21.15.16:0/1634289049 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e580052d0 con 0x7f3e640ce4f0 2023-10-16T12:27:30.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.108+0000 7f3e617fa700 1 -- 172.21.15.16:0/1634289049 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e58005430 con 0x7f3e640ce4f0 2023-10-16T12:27:30.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.108+0000 7f3e6af9a700 1 -- 172.21.15.16:0/1634289049 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e64153210 con 0x7f3e640ce4f0 2023-10-16T12:27:30.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.108+0000 7f3e617fa700 1 -- 172.21.15.16:0/1634289049 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e580038f0 con 0x7f3e640ce4f0 2023-10-16T12:27:30.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.108+0000 7f3e6af9a700 1 -- 172.21.15.16:0/1634289049 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e64152ec0 con 0x7f3e640ce4f0 2023-10-16T12:27:30.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.109+0000 7f3e6af9a700 1 -- 172.21.15.16:0/1634289049 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e6404ebb0 con 0x7f3e640ce4f0 2023-10-16T12:27:30.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.109+0000 7f3e617fa700 1 -- 172.21.15.16:0/1634289049 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 51) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3e58003a50 con 0x7f3e640ce4f0 2023-10-16T12:27:30.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.110+0000 7f3e617fa700 1 --2- 172.21.15.16:0/1634289049 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f3e4806b020 0x7f3e4806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:30.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.110+0000 7f3e617fa700 1 -- 172.21.15.16:0/1634289049 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5979+0+0 (secure 0 0 0) 0x7f3e58089120 con 0x7f3e640ce4f0 2023-10-16T12:27:30.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.110+0000 7f3e63fff700 1 --2- 172.21.15.16:0/1634289049 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f3e4806b020 0x7f3e4806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:30.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.110+0000 7f3e63fff700 1 --2- 172.21.15.16:0/1634289049 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f3e4806b020 0x7f3e4806d4e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3e640b25b0 tx=0x7f3e5001c040 comp rx=0 tx=0).ready entity=mgr.6365 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:30.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.114+0000 7f3e617fa700 1 -- 172.21.15.16:0/1634289049 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3e58054020 con 0x7f3e640ce4f0 2023-10-16T12:27:30.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.319+0000 7f3e6af9a700 1 -- 172.21.15.16:0/1634289049 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3e6406b5d0 con 0x7f3e640ce4f0 2023-10-16T12:27:30.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.320+0000 7f3e617fa700 1 -- 172.21.15.16:0/1634289049 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v104) v1 ==== 74+0+14126 (secure 0 0 0) 0x7f3e58057640 con 0x7f3e640ce4f0 2023-10-16T12:27:30.325 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:27:30.326 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":104,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:27:27.943656+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6837/1843321005":"2023-10-17T12:27:24.674977+0000","172.21.15.16:6834/1843321005":"2023-10-17T12:27:24.674977+0000","172.21.15.16:0/389861626":"2023-10-17T12:27:23.504689+0000","172.21.15.16:0/2994287568":"2023-10-17T12:27:23.504689+0000","172.21.15.16:0/2177553605":"2023-10-17T12:27:23.504689+0000","172.21.15.16:0/2685123653":"2023-10-17T12:27:23.504689+0000","172.21.15.16:6833/444334639":"2023-10-17T12:27:23.504689+0000","172.21.15.16:6833/292891612":"2023-10-17T12:27:15.899818+0000","172.21.15.16:6832/444334639":"2023-10-17T12:27:23.504689+0000","172.21.15.16:6832/292891612":"2023-10-17T12:27:15.899818+0000","172.21.15.16:0/1020789692":"2023-10-17T12:27:15.899818+0000","172.21.15.16:0/478878390":"2023-10-17T12:27:15.899818+0000","172.21.15.16:0/2293822331":"2023-10-17T12:27:15.899818+0000","172.21.15.16:0/2440484844":"2023-10-17T12:27:23.504689+0000","172.21.15.16:0/1305507650":"2023-10-17T12:27:15.899818+0000","172.21.15.16:0/3205233425":"2023-10-17T12:27:15.899818+0000","172.21.15.16:0/3310973712":"2023-10-17T12:27:23.504689+0000","172.21.15.16:0/2477042442":"2023-10-17T12:27:15.899818+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:27:30.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.322+0000 7f3e6af9a700 1 -- 172.21.15.16:0/1634289049 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f3e4806b020 msgr2=0x7f3e4806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:30.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.322+0000 7f3e6af9a700 1 --2- 172.21.15.16:0/1634289049 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f3e4806b020 0x7f3e4806d4e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3e640b25b0 tx=0x7f3e5001c040 comp rx=0 tx=0).stop 2023-10-16T12:27:30.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.322+0000 7f3e6af9a700 1 -- 172.21.15.16:0/1634289049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e640ce4f0 msgr2=0x7f3e6415ce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:30.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.322+0000 7f3e6af9a700 1 --2- 172.21.15.16:0/1634289049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e640ce4f0 0x7f3e6415ce60 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f3e5800c4d0 tx=0x7f3e5800c7e0 comp rx=0 tx=0).stop 2023-10-16T12:27:30.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.322+0000 7f3e6af9a700 1 -- 172.21.15.16:0/1634289049 shutdown_connections 2023-10-16T12:27:30.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.322+0000 7f3e6af9a700 1 --2- 172.21.15.16:0/1634289049 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f3e4806b020 0x7f3e4806d4e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:30.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.322+0000 7f3e6af9a700 1 --2- 172.21.15.16:0/1634289049 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3e6415ba10 0x7f3e640c9510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:30.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.322+0000 7f3e6af9a700 1 --2- 172.21.15.16:0/1634289049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e640ce4f0 0x7f3e6415ce60 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:30.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.322+0000 7f3e6af9a700 1 --2- 172.21.15.16:0/1634289049 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3e6415b630 0x7f3e641f04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:30.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.322+0000 7f3e6af9a700 1 -- 172.21.15.16:0/1634289049 >> 172.21.15.16:0/1634289049 conn(0x7f3e640d14b0 msgr2=0x7f3e640cf770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:30.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.322+0000 7f3e6af9a700 1 -- 172.21.15.16:0/1634289049 shutdown_connections 2023-10-16T12:27:30.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.322+0000 7f3e6af9a700 1 -- 172.21.15.16:0/1634289049 wait complete. 2023-10-16T12:27:30.337 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T12:27:30.338 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:27:30.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.464+0000 7fb1fd66a700 1 Processor -- start 2023-10-16T12:27:30.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.464+0000 7fb1fd66a700 1 -- start start 2023-10-16T12:27:30.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.465+0000 7fb1fd66a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb1f815d8d0 0x7fb1f80b2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:30.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.465+0000 7fb1fd66a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb1f80b8fa0 0x7fb1f80b6720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:30.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.465+0000 7fb1fd66a700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fb1f80b6c60 0x7fb1f80ce350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:30.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.465+0000 7fb1fd66a700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1f80d0bb0 con 0x7fb1f815d8d0 2023-10-16T12:27:30.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.465+0000 7fb1fd66a700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1f815dcb0 con 0x7fb1f80b6c60 2023-10-16T12:27:30.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.465+0000 7fb1fd66a700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f80b2f30 con 0x7fb1f80b8fa0 2023-10-16T12:27:30.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.465+0000 7fb1f6ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb1f815d8d0 0x7fb1f80b2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:30.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.465+0000 7fb1f6ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb1f815d8d0 0x7fb1f80b2830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42054/0 (socket says 172.21.15.16:42054) 2023-10-16T12:27:30.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.465+0000 7fb1f6ffd700 1 -- 172.21.15.16:0/3578249021 learned_addr learned my addr 172.21.15.16:0/3578249021 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:30.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.465+0000 7fb1f67fc700 1 --2- 172.21.15.16:0/3578249021 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb1f80b8fa0 0x7fb1f80b6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:30.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.465+0000 7fb1f77fe700 1 --2- 172.21.15.16:0/3578249021 >> v2:172.21.15.49:3300/0 conn(0x7fb1f80b6c60 0x7fb1f80ce350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:30.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.465+0000 7fb1f6ffd700 1 -- 172.21.15.16:0/3578249021 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb1f80b8fa0 msgr2=0x7fb1f80b6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:30.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.465+0000 7fb1f6ffd700 1 --2- 172.21.15.16:0/3578249021 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb1f80b8fa0 0x7fb1f80b6720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:30.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.465+0000 7fb1f6ffd700 1 -- 172.21.15.16:0/3578249021 >> v2:172.21.15.49:3300/0 conn(0x7fb1f80b6c60 msgr2=0x7fb1f80ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:30.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.465+0000 7fb1f6ffd700 1 --2- 172.21.15.16:0/3578249021 >> v2:172.21.15.49:3300/0 conn(0x7fb1f80b6c60 0x7fb1f80ce350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:30.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.465+0000 7fb1f6ffd700 1 -- 172.21.15.16:0/3578249021 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1f80d1c40 con 0x7fb1f815d8d0 2023-10-16T12:27:30.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.465+0000 7fb1f6ffd700 1 --2- 172.21.15.16:0/3578249021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1f815d8d0 0x7fb1f80b2830 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7fb1f80cdfd0 tx=0x7fb1ec009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c8519ef7abfbeb2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:30.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.465+0000 7fb1f5ffb700 1 -- 172.21.15.16:0/3578249021 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1ec00b870 con 0x7fb1f815d8d0 2023-10-16T12:27:30.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.466+0000 7fb1f5ffb700 1 -- 172.21.15.16:0/3578249021 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1ec00b9d0 con 0x7fb1f815d8d0 2023-10-16T12:27:30.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.466+0000 7fb1f5ffb700 1 -- 172.21.15.16:0/3578249021 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1ec00bd60 con 0x7fb1f815d8d0 2023-10-16T12:27:30.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.466+0000 7fb1fd66a700 1 -- 172.21.15.16:0/3578249021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1f815d8d0 msgr2=0x7fb1f80b2830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:30.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.466+0000 7fb1fd66a700 1 --2- 172.21.15.16:0/3578249021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1f815d8d0 0x7fb1f80b2830 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7fb1f80cdfd0 tx=0x7fb1ec009bd0 comp rx=0 tx=0).stop 2023-10-16T12:27:30.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.466+0000 7fb1fd66a700 1 -- 172.21.15.16:0/3578249021 shutdown_connections 2023-10-16T12:27:30.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.466+0000 7fb1fd66a700 1 --2- 172.21.15.16:0/3578249021 >> v2:172.21.15.49:3300/0 conn(0x7fb1f80b6c60 0x7fb1f80ce350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:30.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.466+0000 7fb1fd66a700 1 --2- 172.21.15.16:0/3578249021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1f815d8d0 0x7fb1f80b2830 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:30.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.466+0000 7fb1fd66a700 1 --2- 172.21.15.16:0/3578249021 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb1f80b8fa0 0x7fb1f80b6720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:30.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.466+0000 7fb1fd66a700 1 -- 172.21.15.16:0/3578249021 >> 172.21.15.16:0/3578249021 conn(0x7fb1f8072330 msgr2=0x7fb1f8072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:30.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.466+0000 7fb1fd66a700 1 -- 172.21.15.16:0/3578249021 shutdown_connections 2023-10-16T12:27:30.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.466+0000 7fb1fd66a700 1 -- 172.21.15.16:0/3578249021 wait complete. 2023-10-16T12:27:30.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.467+0000 7fb1fd66a700 1 Processor -- start 2023-10-16T12:27:30.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.468+0000 7fb1fd66a700 1 -- start start 2023-10-16T12:27:30.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.468+0000 7fb1fd66a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb1f80b6c60 0x7fb1f81f29b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:30.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.468+0000 7fb1fd66a700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb1f80b8fa0 0x7fb1f80ca9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:30.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.468+0000 7fb1fd66a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1f815d8d0 0x7fb1f80caf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:30.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.468+0000 7fb1fd66a700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f80b5b40 con 0x7fb1f815d8d0 2023-10-16T12:27:30.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.468+0000 7fb1fd66a700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f81f2ef0 con 0x7fb1f80b8fa0 2023-10-16T12:27:30.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.468+0000 7fb1fd66a700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f80b4fc0 con 0x7fb1f80b6c60 2023-10-16T12:27:30.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.468+0000 7fb1f77fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1f815d8d0 0x7fb1f80caf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:30.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.468+0000 7fb1f77fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1f815d8d0 0x7fb1f80caf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42064/0 (socket says 172.21.15.16:42064) 2023-10-16T12:27:30.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.468+0000 7fb1f77fe700 1 -- 172.21.15.16:0/2876561546 learned_addr learned my addr 172.21.15.16:0/2876561546 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:30.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.468+0000 7fb1f67fc700 1 --2- 172.21.15.16:0/2876561546 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb1f80b8fa0 0x7fb1f80ca9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:30.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.468+0000 7fb1f6ffd700 1 --2- 172.21.15.16:0/2876561546 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb1f80b6c60 0x7fb1f81f29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:30.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.469+0000 7fb1f77fe700 1 -- 172.21.15.16:0/2876561546 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb1f80b6c60 msgr2=0x7fb1f81f29b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:30.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.469+0000 7fb1f77fe700 1 --2- 172.21.15.16:0/2876561546 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb1f80b6c60 0x7fb1f81f29b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:30.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.469+0000 7fb1f77fe700 1 -- 172.21.15.16:0/2876561546 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb1f80b8fa0 msgr2=0x7fb1f80ca9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:30.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.469+0000 7fb1f77fe700 1 --2- 172.21.15.16:0/2876561546 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb1f80b8fa0 0x7fb1f80ca9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:30.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.469+0000 7fb1f77fe700 1 -- 172.21.15.16:0/2876561546 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1ec009550 con 0x7fb1f815d8d0 2023-10-16T12:27:30.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.469+0000 7fb1f77fe700 1 --2- 172.21.15.16:0/2876561546 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1f815d8d0 0x7fb1f80caf20 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7fb1f0007d40 tx=0x7fb1f000e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:30.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.469+0000 7fb1df7fe700 1 -- 172.21.15.16:0/2876561546 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f000ed30 con 0x7fb1f815d8d0 2023-10-16T12:27:30.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.469+0000 7fb1df7fe700 1 -- 172.21.15.16:0/2876561546 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1f000ee90 con 0x7fb1f815d8d0 2023-10-16T12:27:30.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.469+0000 7fb1df7fe700 1 -- 172.21.15.16:0/2876561546 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f0014660 con 0x7fb1f815d8d0 2023-10-16T12:27:30.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.469+0000 7fb1fd66a700 1 -- 172.21.15.16:0/2876561546 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1f80b52a0 con 0x7fb1f815d8d0 2023-10-16T12:27:30.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.469+0000 7fb1fd66a700 1 -- 172.21.15.16:0/2876561546 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1f80b5860 con 0x7fb1f815d8d0 2023-10-16T12:27:30.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.470+0000 7fb1fd66a700 1 -- 172.21.15.16:0/2876561546 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1f804ebb0 con 0x7fb1f815d8d0 2023-10-16T12:27:30.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.470+0000 7fb1df7fe700 1 -- 172.21.15.16:0/2876561546 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 51) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb1f00147c0 con 0x7fb1f815d8d0 2023-10-16T12:27:30.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.471+0000 7fb1df7fe700 1 --2- 172.21.15.16:0/2876561546 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fb1e406b020 0x7fb1e406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:30.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.471+0000 7fb1df7fe700 1 -- 172.21.15.16:0/2876561546 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5979+0+0 (secure 0 0 0) 0x7fb1f0089c70 con 0x7fb1f815d8d0 2023-10-16T12:27:30.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.471+0000 7fb1f6ffd700 1 --2- 172.21.15.16:0/2876561546 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fb1e406b020 0x7fb1e406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:30.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.471+0000 7fb1f6ffd700 1 --2- 172.21.15.16:0/2876561546 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fb1e406b020 0x7fb1e406d4e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb1ec014040 tx=0x7fb1ec008030 comp rx=0 tx=0).ready entity=mgr.6365 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:30.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.475+0000 7fb1df7fe700 1 -- 172.21.15.16:0/2876561546 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb1f0054b90 con 0x7fb1f815d8d0 2023-10-16T12:27:30.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.676+0000 7fb1fd66a700 1 -- 172.21.15.16:0/2876561546 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb1f80b62c0 con 0x7fb1f815d8d0 2023-10-16T12:27:30.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.677+0000 7fb1df7fe700 1 -- 172.21.15.16:0/2876561546 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v104) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb1f00581b0 con 0x7fb1f815d8d0 2023-10-16T12:27:30.680 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T12:27:30.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.679+0000 7fb1fd66a700 1 -- 172.21.15.16:0/2876561546 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fb1e406b020 msgr2=0x7fb1e406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:30.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.679+0000 7fb1fd66a700 1 --2- 172.21.15.16:0/2876561546 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fb1e406b020 0x7fb1e406d4e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb1ec014040 tx=0x7fb1ec008030 comp rx=0 tx=0).stop 2023-10-16T12:27:30.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.679+0000 7fb1fd66a700 1 -- 172.21.15.16:0/2876561546 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1f815d8d0 msgr2=0x7fb1f80caf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:30.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.679+0000 7fb1fd66a700 1 --2- 172.21.15.16:0/2876561546 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1f815d8d0 0x7fb1f80caf20 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7fb1f0007d40 tx=0x7fb1f000e3c0 comp rx=0 tx=0).stop 2023-10-16T12:27:30.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.679+0000 7fb1fd66a700 1 -- 172.21.15.16:0/2876561546 shutdown_connections 2023-10-16T12:27:30.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.679+0000 7fb1fd66a700 1 --2- 172.21.15.16:0/2876561546 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1f815d8d0 0x7fb1f80caf20 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:30.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.679+0000 7fb1fd66a700 1 --2- 172.21.15.16:0/2876561546 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb1f80b6c60 0x7fb1f81f29b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:30.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.679+0000 7fb1fd66a700 1 --2- 172.21.15.16:0/2876561546 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fb1e406b020 0x7fb1e406d4e0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:30.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.679+0000 7fb1fd66a700 1 --2- 172.21.15.16:0/2876561546 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb1f80b8fa0 0x7fb1f80ca9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:30.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.679+0000 7fb1fd66a700 1 -- 172.21.15.16:0/2876561546 >> 172.21.15.16:0/2876561546 conn(0x7fb1f8072330 msgr2=0x7fb1f815e9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:30.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.679+0000 7fb1fd66a700 1 -- 172.21.15.16:0/2876561546 shutdown_connections 2023-10-16T12:27:30.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.679+0000 7fb1fd66a700 1 -- 172.21.15.16:0/2876561546 wait complete. 2023-10-16T12:27:30.694 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path' 2023-10-16T12:27:30.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.816+0000 7fae97866700 1 Processor -- start 2023-10-16T12:27:30.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.816+0000 7fae97866700 1 -- start start 2023-10-16T12:27:30.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.817+0000 7fae97866700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fae9015b860 0x7fae9006a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:30.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.817+0000 7fae97866700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fae900b2be0 0x7fae9006ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:30.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.817+0000 7fae97866700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae900d11c0 con 0x7fae900cd770 2023-10-16T12:27:30.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.817+0000 7fae97866700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae900d03d0 con 0x7fae900b2be0 2023-10-16T12:27:30.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.817+0000 7fae97866700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae900cd360 con 0x7fae9015b860 2023-10-16T12:27:30.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.817+0000 7fae95e03700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fae900cd770 0x7fae900719e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59578/0 (socket says 172.21.15.16:59578) 2023-10-16T12:27:30.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.817+0000 7fae95e03700 1 -- 172.21.15.16:0/2479383013 learned_addr learned my addr 172.21.15.16:0/2479383013 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:27:30.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.817+0000 7fae94e01700 1 --2- 172.21.15.16:0/2479383013 >> v2:172.21.15.49:3300/0 conn(0x7fae900b2be0 0x7fae9006ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:30.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.817+0000 7fae95602700 1 --2- 172.21.15.16:0/2479383013 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fae9015b860 0x7fae9006a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:30.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.817+0000 7fae87fff700 1 -- 172.21.15.16:0/2479383013 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 564858425 0 0) 0x7fae900d11c0 con 0x7fae900cd770 2023-10-16T12:27:30.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.817+0000 7fae87fff700 1 -- 172.21.15.16:0/2479383013 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae7c0035b0 con 0x7fae900cd770 2023-10-16T12:27:30.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.817+0000 7fae94e01700 1 -- 172.21.15.16:0/2479383013 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fae9015b860 msgr2=0x7fae9006a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:30.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.817+0000 7fae94e01700 1 --2- 172.21.15.16:0/2479383013 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fae9015b860 0x7fae9006a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:30.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.817+0000 7fae94e01700 1 -- 172.21.15.16:0/2479383013 >> v1:172.21.15.16:6789/0 conn(0x7fae900cd770 legacy=0x7fae900719e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:30.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.817+0000 7fae94e01700 1 -- 172.21.15.16:0/2479383013 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae900d05b0 con 0x7fae900b2be0 2023-10-16T12:27:30.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.818+0000 7fae94e01700 1 --2- 172.21.15.16:0/2479383013 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fae900b2be0 0x7fae9006ad20 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fae8c009840 tx=0x7fae8c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e48ba850078353b1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:30.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.818+0000 7fae87fff700 1 -- 172.21.15.16:0/2479383013 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae8c011830 con 0x7fae900b2be0 2023-10-16T12:27:30.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.818+0000 7fae87fff700 1 -- 172.21.15.16:0/2479383013 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae8c011990 con 0x7fae900b2be0 2023-10-16T12:27:30.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.818+0000 7fae87fff700 1 -- 172.21.15.16:0/2479383013 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae8c011d20 con 0x7fae900b2be0 2023-10-16T12:27:30.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.818+0000 7fae97866700 1 -- 172.21.15.16:0/2479383013 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fae900b2be0 msgr2=0x7fae9006ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:30.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.818+0000 7fae97866700 1 --2- 172.21.15.16:0/2479383013 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fae900b2be0 0x7fae9006ad20 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fae8c009840 tx=0x7fae8c009b50 comp rx=0 tx=0).stop 2023-10-16T12:27:30.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.819+0000 7fae97866700 1 -- 172.21.15.16:0/2479383013 shutdown_connections 2023-10-16T12:27:30.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.819+0000 7fae97866700 1 --2- 172.21.15.16:0/2479383013 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fae9015b860 0x7fae9006a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:30.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.819+0000 7fae97866700 1 --2- 172.21.15.16:0/2479383013 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fae900b2be0 0x7fae9006ad20 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:30.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.819+0000 7fae97866700 1 -- 172.21.15.16:0/2479383013 >> 172.21.15.16:0/2479383013 conn(0x7fae900b1fd0 msgr2=0x7fae900b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:30.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.819+0000 7fae97866700 1 -- 172.21.15.16:0/2479383013 shutdown_connections 2023-10-16T12:27:30.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.819+0000 7fae97866700 1 -- 172.21.15.16:0/2479383013 wait complete. 2023-10-16T12:27:30.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.820+0000 7fae97866700 1 Processor -- start 2023-10-16T12:27:30.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.820+0000 7fae97866700 1 -- start start 2023-10-16T12:27:30.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.821+0000 7fae97866700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae900b2be0 0x7fae9015d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:30.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.821+0000 7fae97866700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fae900cd770 0x7fae9015c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:30.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.821+0000 7fae97866700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fae9015b860 0x7fae90157500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:30.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.821+0000 7fae97866700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae9015cc70 con 0x7fae900b2be0 2023-10-16T12:27:30.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.821+0000 7fae97866700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae900c7510 con 0x7fae900cd770 2023-10-16T12:27:30.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.821+0000 7fae97866700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae90157a40 con 0x7fae9015b860 2023-10-16T12:27:30.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.821+0000 7fae95602700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae900b2be0 0x7fae9015d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:30.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.821+0000 7fae95602700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae900b2be0 0x7fae9015d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42074/0 (socket says 172.21.15.16:42074) 2023-10-16T12:27:30.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.821+0000 7fae95602700 1 -- 172.21.15.16:0/200832678 learned_addr learned my addr 172.21.15.16:0/200832678 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:30.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.821+0000 7fae95e03700 1 --2- 172.21.15.16:0/200832678 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fae9015b860 0x7fae90157500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:30.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.821+0000 7fae94e01700 1 --2- 172.21.15.16:0/200832678 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fae900cd770 0x7fae9015c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:30.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.821+0000 7fae95602700 1 -- 172.21.15.16:0/200832678 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fae9015b860 msgr2=0x7fae90157500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:30.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.821+0000 7fae95602700 1 --2- 172.21.15.16:0/200832678 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fae9015b860 0x7fae90157500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:30.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.821+0000 7fae95602700 1 -- 172.21.15.16:0/200832678 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fae900cd770 msgr2=0x7fae9015c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:30.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.821+0000 7fae95602700 1 --2- 172.21.15.16:0/200832678 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fae900cd770 0x7fae9015c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:30.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.821+0000 7fae95602700 1 -- 172.21.15.16:0/200832678 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae8c0094a0 con 0x7fae900b2be0 2023-10-16T12:27:30.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.821+0000 7fae95602700 1 --2- 172.21.15.16:0/200832678 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae900b2be0 0x7fae9015d960 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7fae8000e9b0 tx=0x7fae8000ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:30.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.822+0000 7fae85ffb700 1 -- 172.21.15.16:0/200832678 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae80009710 con 0x7fae900b2be0 2023-10-16T12:27:30.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.822+0000 7fae85ffb700 1 -- 172.21.15.16:0/200832678 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae8000ee50 con 0x7fae900b2be0 2023-10-16T12:27:30.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.822+0000 7fae97866700 1 -- 172.21.15.16:0/200832678 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae9015e060 con 0x7fae900b2be0 2023-10-16T12:27:30.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.822+0000 7fae85ffb700 1 -- 172.21.15.16:0/200832678 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae8000ca50 con 0x7fae900b2be0 2023-10-16T12:27:30.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.822+0000 7fae97866700 1 -- 172.21.15.16:0/200832678 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae900c7a40 con 0x7fae900b2be0 2023-10-16T12:27:30.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.823+0000 7fae97866700 1 -- 172.21.15.16:0/200832678 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae900c8090 con 0x7fae900b2be0 2023-10-16T12:27:30.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.824+0000 7fae85ffb700 1 -- 172.21.15.16:0/200832678 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 51) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fae8000cbb0 con 0x7fae900b2be0 2023-10-16T12:27:30.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.824+0000 7fae85ffb700 1 --2- 172.21.15.16:0/200832678 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fae7406b020 0x7fae7406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:30.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.824+0000 7fae85ffb700 1 -- 172.21.15.16:0/200832678 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5979+0+0 (secure 0 0 0) 0x7fae80089fb0 con 0x7fae900b2be0 2023-10-16T12:27:30.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.824+0000 7fae94e01700 1 --2- 172.21.15.16:0/200832678 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fae7406b020 0x7fae7406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:30.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.824+0000 7fae94e01700 1 --2- 172.21.15.16:0/200832678 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fae7406b020 0x7fae7406d4e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fae8c000c00 tx=0x7fae8c01c040 comp rx=0 tx=0).ready entity=mgr.6365 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:30.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:30.828+0000 7fae85ffb700 1 -- 172.21.15.16:0/200832678 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fae80054f20 con 0x7fae900b2be0 2023-10-16T12:27:31.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:31.122+0000 7fae97866700 1 -- 172.21.15.16:0/200832678 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]} v 0) v1 -- 0x7fae9004ebb0 con 0x7fae900b2be0 2023-10-16T12:27:31.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:31.952+0000 7fae85ffb700 1 -- 172.21.15.16:0/200832678 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]}]=0 v634) v1 ==== 186+0+0 (secure 0 0 0) 0x7fae80021100 con 0x7fae900b2be0 2023-10-16T12:27:31.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:31.955+0000 7fae97866700 1 -- 172.21.15.16:0/200832678 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fae7406b020 msgr2=0x7fae7406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:31.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:31.955+0000 7fae97866700 1 --2- 172.21.15.16:0/200832678 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fae7406b020 0x7fae7406d4e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fae8c000c00 tx=0x7fae8c01c040 comp rx=0 tx=0).stop 2023-10-16T12:27:31.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:31.955+0000 7fae97866700 1 -- 172.21.15.16:0/200832678 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae900b2be0 msgr2=0x7fae9015d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:31.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:31.955+0000 7fae97866700 1 --2- 172.21.15.16:0/200832678 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae900b2be0 0x7fae9015d960 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7fae8000e9b0 tx=0x7fae8000ecc0 comp rx=0 tx=0).stop 2023-10-16T12:27:31.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:31.955+0000 7fae97866700 1 -- 172.21.15.16:0/200832678 shutdown_connections 2023-10-16T12:27:31.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:31.955+0000 7fae97866700 1 --2- 172.21.15.16:0/200832678 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fae9015b860 0x7fae90157500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:31.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:31.955+0000 7fae97866700 1 --2- 172.21.15.16:0/200832678 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae900b2be0 0x7fae9015d960 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:31.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:31.955+0000 7fae97866700 1 --2- 172.21.15.16:0/200832678 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fae7406b020 0x7fae7406d4e0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:31.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:31.955+0000 7fae97866700 1 --2- 172.21.15.16:0/200832678 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fae900cd770 0x7fae9015c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:31.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:31.955+0000 7fae97866700 1 -- 172.21.15.16:0/200832678 >> 172.21.15.16:0/200832678 conn(0x7fae900b1fd0 msgr2=0x7fae90159860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:31.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:31.955+0000 7fae97866700 1 -- 172.21.15.16:0/200832678 shutdown_connections 2023-10-16T12:27:31.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:31.955+0000 7fae97866700 1 -- 172.21.15.16:0/200832678 wait complete. 2023-10-16T12:27:31.973 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-16T12:27:32.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.106+0000 7f069ffae700 1 Processor -- start 2023-10-16T12:27:32.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.106+0000 7f069ffae700 1 -- start start 2023-10-16T12:27:32.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.107+0000 7f069ffae700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f06980ce6a0 0x7f069806c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:32.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.107+0000 7f069ffae700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06980b20f0 con 0x7f06980d2110 2023-10-16T12:27:32.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.107+0000 7f069ffae700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06980af1f0 con 0x7f06980b2620 2023-10-16T12:27:32.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.107+0000 7f069ffae700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06980b1d70 con 0x7f06980ce6a0 2023-10-16T12:27:32.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.107+0000 7f069dd4a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f06980ce6a0 0x7f069806c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:32.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.107+0000 7f069d549700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f06980d2110 0x7f06980b0f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59588/0 (socket says 172.21.15.16:59588) 2023-10-16T12:27:32.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.107+0000 7f069d549700 1 -- 172.21.15.16:0/2411828539 learned_addr learned my addr 172.21.15.16:0/2411828539 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:27:32.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.108+0000 7f069dd4a700 1 -- 172.21.15.16:0/2411828539 >> v1:172.21.15.49:6789/0 conn(0x7f06980b2620 legacy=0x7f06980b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:32.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.108+0000 7f069dd4a700 1 -- 172.21.15.16:0/2411828539 >> v1:172.21.15.16:6789/0 conn(0x7f06980d2110 legacy=0x7f06980b0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:32.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.108+0000 7f069dd4a700 1 -- 172.21.15.16:0/2411828539 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06980d03d0 con 0x7f06980ce6a0 2023-10-16T12:27:32.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.108+0000 7f069dd4a700 1 --2- 172.21.15.16:0/2411828539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06980ce6a0 0x7f069806c300 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f0688009840 tx=0x7f0688009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20bc18bfb709cabb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:32.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.108+0000 7f069cd48700 1 -- 172.21.15.16:0/2411828539 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0688011830 con 0x7f06980ce6a0 2023-10-16T12:27:32.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.108+0000 7f069cd48700 1 -- 172.21.15.16:0/2411828539 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0688011990 con 0x7f06980ce6a0 2023-10-16T12:27:32.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.108+0000 7f069cd48700 1 -- 172.21.15.16:0/2411828539 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0688011d20 con 0x7f06980ce6a0 2023-10-16T12:27:32.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.108+0000 7f069ffae700 1 -- 172.21.15.16:0/2411828539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06980ce6a0 msgr2=0x7f069806c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:32.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.109+0000 7f069ffae700 1 --2- 172.21.15.16:0/2411828539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06980ce6a0 0x7f069806c300 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f0688009840 tx=0x7f0688009b50 comp rx=0 tx=0).stop 2023-10-16T12:27:32.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.109+0000 7f069ffae700 1 -- 172.21.15.16:0/2411828539 shutdown_connections 2023-10-16T12:27:32.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.109+0000 7f069ffae700 1 --2- 172.21.15.16:0/2411828539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06980ce6a0 0x7f069806c300 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:32.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.109+0000 7f069ffae700 1 -- 172.21.15.16:0/2411828539 >> 172.21.15.16:0/2411828539 conn(0x7f06980d15f0 msgr2=0x7f06980d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:32.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.109+0000 7f069ffae700 1 -- 172.21.15.16:0/2411828539 shutdown_connections 2023-10-16T12:27:32.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.109+0000 7f069ffae700 1 -- 172.21.15.16:0/2411828539 wait complete. 2023-10-16T12:27:32.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.110+0000 7f069ffae700 1 Processor -- start 2023-10-16T12:27:32.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.110+0000 7f069ffae700 1 -- start start 2023-10-16T12:27:32.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.111+0000 7f069ffae700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f06980b2620 0x7f06980c70e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:32.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.111+0000 7f069ffae700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06980ce6a0 0x7f06981561f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:32.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.111+0000 7f069ffae700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f06980d2110 0x7f0698156730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:32.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.111+0000 7f069ffae700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06980b4600 con 0x7f06980ce6a0 2023-10-16T12:27:32.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.111+0000 7f069ffae700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06980b48e0 con 0x7f06980b2620 2023-10-16T12:27:32.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.111+0000 7f069ffae700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06980b4bc0 con 0x7f06980d2110 2023-10-16T12:27:32.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.111+0000 7f069d549700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06980ce6a0 0x7f06981561f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:32.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.111+0000 7f069d549700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06980ce6a0 0x7f06981561f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42090/0 (socket says 172.21.15.16:42090) 2023-10-16T12:27:32.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.111+0000 7f069d549700 1 -- 172.21.15.16:0/2101554809 learned_addr learned my addr 172.21.15.16:0/2101554809 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:32.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.111+0000 7f069e54b700 1 --2- 172.21.15.16:0/2101554809 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f06980d2110 0x7f0698156730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:32.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.111+0000 7f069d549700 1 -- 172.21.15.16:0/2101554809 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f06980d2110 msgr2=0x7f0698156730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:32.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.111+0000 7f069dd4a700 1 --2- 172.21.15.16:0/2101554809 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f06980b2620 0x7f06980c70e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:32.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.111+0000 7f069d549700 1 --2- 172.21.15.16:0/2101554809 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f06980d2110 0x7f0698156730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:32.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.111+0000 7f069d549700 1 -- 172.21.15.16:0/2101554809 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f06980b2620 msgr2=0x7f06980c70e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:32.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.111+0000 7f069d549700 1 --2- 172.21.15.16:0/2101554809 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f06980b2620 0x7f06980c70e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:32.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.111+0000 7f069d549700 1 -- 172.21.15.16:0/2101554809 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06880094a0 con 0x7f06980ce6a0 2023-10-16T12:27:32.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.111+0000 7f069e54b700 1 --2- 172.21.15.16:0/2101554809 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f06980d2110 0x7f0698156730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:27:32.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.111+0000 7f069dd4a700 1 --2- 172.21.15.16:0/2101554809 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f06980b2620 0x7f06980c70e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:27:32.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.111+0000 7f069d549700 1 --2- 172.21.15.16:0/2101554809 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06980ce6a0 0x7f06981561f0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f069400c420 tx=0x7f069400c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:32.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.111+0000 7f068e7fc700 1 -- 172.21.15.16:0/2101554809 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0694005d70 con 0x7f06980ce6a0 2023-10-16T12:27:32.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.111+0000 7f068e7fc700 1 -- 172.21.15.16:0/2101554809 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06940070d0 con 0x7f06980ce6a0 2023-10-16T12:27:32.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.111+0000 7f069ffae700 1 -- 172.21.15.16:0/2101554809 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06980b4ea0 con 0x7f06980ce6a0 2023-10-16T12:27:32.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.111+0000 7f068e7fc700 1 -- 172.21.15.16:0/2101554809 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0694007440 con 0x7f06980ce6a0 2023-10-16T12:27:32.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.112+0000 7f069ffae700 1 -- 172.21.15.16:0/2101554809 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06980c7620 con 0x7f06980ce6a0 2023-10-16T12:27:32.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.112+0000 7f069ffae700 1 -- 172.21.15.16:0/2101554809 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f069804ebb0 con 0x7f06980ce6a0 2023-10-16T12:27:32.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.113+0000 7f068e7fc700 1 -- 172.21.15.16:0/2101554809 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 51) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f06940075a0 con 0x7f06980ce6a0 2023-10-16T12:27:32.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.113+0000 7f068e7fc700 1 --2- 172.21.15.16:0/2101554809 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f067c06b020 0x7f067c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:32.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.113+0000 7f068e7fc700 1 -- 172.21.15.16:0/2101554809 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5979+0+0 (secure 0 0 0) 0x7f0694088c00 con 0x7f06980ce6a0 2023-10-16T12:27:32.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.113+0000 7f069dd4a700 1 --2- 172.21.15.16:0/2101554809 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f067c06b020 0x7f067c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:32.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.114+0000 7f069dd4a700 1 --2- 172.21.15.16:0/2101554809 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f067c06b020 0x7f067c06d4e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f06880090b0 tx=0x7f0688008030 comp rx=0 tx=0).ready entity=mgr.6365 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:32.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.118+0000 7f068e7fc700 1 -- 172.21.15.16:0/2101554809 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0694053aa0 con 0x7f06980ce6a0 2023-10-16T12:27:32.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.323+0000 7f069ffae700 1 -- 172.21.15.16:0/2101554809 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f0698160a50 con 0x7f06980ce6a0 2023-10-16T12:27:32.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.326+0000 7f068e7fc700 1 -- 172.21.15.16:0/2101554809 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v34)=0 v34) v1 ==== 122+0+0 (secure 0 0 0) 0x7f06940170c0 con 0x7f06980ce6a0 2023-10-16T12:27:32.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.328+0000 7f069ffae700 1 -- 172.21.15.16:0/2101554809 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f067c06b020 msgr2=0x7f067c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:32.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.328+0000 7f069ffae700 1 --2- 172.21.15.16:0/2101554809 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f067c06b020 0x7f067c06d4e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f06880090b0 tx=0x7f0688008030 comp rx=0 tx=0).stop 2023-10-16T12:27:32.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.328+0000 7f069ffae700 1 -- 172.21.15.16:0/2101554809 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06980ce6a0 msgr2=0x7f06981561f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:32.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.328+0000 7f069ffae700 1 --2- 172.21.15.16:0/2101554809 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06980ce6a0 0x7f06981561f0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f069400c420 tx=0x7f069400c7e0 comp rx=0 tx=0).stop 2023-10-16T12:27:32.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.328+0000 7f069ffae700 1 -- 172.21.15.16:0/2101554809 shutdown_connections 2023-10-16T12:27:32.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.328+0000 7f069ffae700 1 --2- 172.21.15.16:0/2101554809 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f06980d2110 0x7f0698156730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:32.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.328+0000 7f069ffae700 1 --2- 172.21.15.16:0/2101554809 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f067c06b020 0x7f067c06d4e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:32.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.328+0000 7f069ffae700 1 --2- 172.21.15.16:0/2101554809 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f06980b2620 0x7f06980c70e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:32.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.328+0000 7f069ffae700 1 --2- 172.21.15.16:0/2101554809 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06980ce6a0 0x7f06981561f0 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:32.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.328+0000 7f069ffae700 1 -- 172.21.15.16:0/2101554809 >> 172.21.15.16:0/2101554809 conn(0x7f06980d15f0 msgr2=0x7f06980cbc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:32.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.328+0000 7f069ffae700 1 -- 172.21.15.16:0/2101554809 shutdown_connections 2023-10-16T12:27:32.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.328+0000 7f069ffae700 1 -- 172.21.15.16:0/2101554809 wait complete. 2023-10-16T12:27:32.343 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T12:27:32.343 DEBUG:teuthology.orchestra.run.smithi016:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T12:27:32.383 DEBUG:teuthology.orchestra.run.smithi049:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T12:27:32.453 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T12:27:32.454 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:27:32.454 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:27:32.471 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16287364k,nr_inodes=4071841,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20251 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3261172k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T12:27:32.472 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T12:27:32.472 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:27:32.493 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284204k,nr_inodes=4071051,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=753 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260540k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T12:27:32.493 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:27:32.493 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:27:32.528 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16287364k,nr_inodes=4071841,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20251 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3261172k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T12:27:32.529 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T12:27:32.529 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:27:32.552 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284204k,nr_inodes=4071051,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=753 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260540k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T12:27:32.552 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-16T12:27:32.553 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:27:32.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.690+0000 7fbb6a16e700 1 Processor -- start 2023-10-16T12:27:32.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.690+0000 7fbb6a16e700 1 -- start start 2023-10-16T12:27:32.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.691+0000 7fbb6a16e700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fbb6415b630 0x7fbb640cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:32.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.691+0000 7fbb6a16e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbb6415ba10 0x7fbb640cdde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:32.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.691+0000 7fbb6a16e700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb640ce730 con 0x7fbb640ce320 2023-10-16T12:27:32.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.691+0000 7fbb6a16e700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb640cdb20 con 0x7fbb6415ba10 2023-10-16T12:27:32.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.691+0000 7fbb6a16e700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb640cc2a0 con 0x7fbb6415b630 2023-10-16T12:27:32.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.691+0000 7fbb62ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbb6415ba10 0x7fbb640cdde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:32.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.691+0000 7fbb62ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbb6415ba10 0x7fbb640cdde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42098/0 (socket says 172.21.15.16:42098) 2023-10-16T12:27:32.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.691+0000 7fbb62ffd700 1 -- 172.21.15.16:0/861086293 learned_addr learned my addr 172.21.15.16:0/861086293 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:32.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.692+0000 7fbb637fe700 1 --2- 172.21.15.16:0/861086293 >> v2:172.21.15.49:3300/0 conn(0x7fbb6415b630 0x7fbb640cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:32.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.692+0000 7fbb62ffd700 1 -- 172.21.15.16:0/861086293 >> v2:172.21.15.49:3300/0 conn(0x7fbb6415b630 msgr2=0x7fbb640cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:32.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.692+0000 7fbb62ffd700 1 --2- 172.21.15.16:0/861086293 >> v2:172.21.15.49:3300/0 conn(0x7fbb6415b630 0x7fbb640cd5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:32.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.692+0000 7fbb62ffd700 1 -- 172.21.15.16:0/861086293 >> v1:172.21.15.49:6789/0 conn(0x7fbb640ce320 legacy=0x7fbb640cfca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:32.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.692+0000 7fbb62ffd700 1 -- 172.21.15.16:0/861086293 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb640cc540 con 0x7fbb6415ba10 2023-10-16T12:27:32.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.692+0000 7fbb637fe700 1 --2- 172.21.15.16:0/861086293 >> v2:172.21.15.49:3300/0 conn(0x7fbb6415b630 0x7fbb640cd5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:27:32.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.692+0000 7fbb62ffd700 1 --2- 172.21.15.16:0/861086293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb6415ba10 0x7fbb640cdde0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7fbb640cefb0 tx=0x7fbb54009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=155410e82ef4343d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:32.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.692+0000 7fbb627fc700 1 -- 172.21.15.16:0/861086293 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb54004230 con 0x7fbb6415ba10 2023-10-16T12:27:32.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.692+0000 7fbb627fc700 1 -- 172.21.15.16:0/861086293 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb54004390 con 0x7fbb6415ba10 2023-10-16T12:27:32.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.692+0000 7fbb627fc700 1 -- 172.21.15.16:0/861086293 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb540036a0 con 0x7fbb6415ba10 2023-10-16T12:27:32.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.692+0000 7fbb6a16e700 1 -- 172.21.15.16:0/861086293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb6415ba10 msgr2=0x7fbb640cdde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:32.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.692+0000 7fbb6a16e700 1 --2- 172.21.15.16:0/861086293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb6415ba10 0x7fbb640cdde0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7fbb640cefb0 tx=0x7fbb54009bd0 comp rx=0 tx=0).stop 2023-10-16T12:27:32.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.693+0000 7fbb6a16e700 1 -- 172.21.15.16:0/861086293 shutdown_connections 2023-10-16T12:27:32.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.693+0000 7fbb6a16e700 1 --2- 172.21.15.16:0/861086293 >> v2:172.21.15.49:3300/0 conn(0x7fbb6415b630 0x7fbb640cd5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:32.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.693+0000 7fbb6a16e700 1 --2- 172.21.15.16:0/861086293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb6415ba10 0x7fbb640cdde0 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:32.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.693+0000 7fbb6a16e700 1 -- 172.21.15.16:0/861086293 >> 172.21.15.16:0/861086293 conn(0x7fbb640d14b0 msgr2=0x7fbb640d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:32.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.693+0000 7fbb6a16e700 1 -- 172.21.15.16:0/861086293 shutdown_connections 2023-10-16T12:27:32.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.693+0000 7fbb6a16e700 1 -- 172.21.15.16:0/861086293 wait complete. 2023-10-16T12:27:32.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.694+0000 7fbb6a16e700 1 Processor -- start 2023-10-16T12:27:32.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.694+0000 7fbb6a16e700 1 -- start start 2023-10-16T12:27:32.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.695+0000 7fbb6a16e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbb640ce320 0x7fbb641f0570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:32.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.695+0000 7fbb6a16e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb6415b630 0x7fbb640c69f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:32.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.695+0000 7fbb6a16e700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbb6415ba10 0x7fbb640c6f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:32.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.695+0000 7fbb6a16e700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb640c9f60 con 0x7fbb6415b630 2023-10-16T12:27:32.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.695+0000 7fbb6a16e700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb640c92e0 con 0x7fbb6415ba10 2023-10-16T12:27:32.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.695+0000 7fbb6a16e700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb640c95c0 con 0x7fbb640ce320 2023-10-16T12:27:32.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.695+0000 7fbb62ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb6415b630 0x7fbb640c69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:32.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.696+0000 7fbb62ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb6415b630 0x7fbb640c69f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42104/0 (socket says 172.21.15.16:42104) 2023-10-16T12:27:32.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.696+0000 7fbb62ffd700 1 -- 172.21.15.16:0/604553752 learned_addr learned my addr 172.21.15.16:0/604553752 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:32.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.696+0000 7fbb63fff700 1 --2- 172.21.15.16:0/604553752 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbb6415ba10 0x7fbb640c6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:32.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.696+0000 7fbb637fe700 1 --2- 172.21.15.16:0/604553752 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbb640ce320 0x7fbb641f0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:32.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.696+0000 7fbb62ffd700 1 -- 172.21.15.16:0/604553752 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbb640ce320 msgr2=0x7fbb641f0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:32.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.696+0000 7fbb62ffd700 1 --2- 172.21.15.16:0/604553752 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbb640ce320 0x7fbb641f0570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:32.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.696+0000 7fbb62ffd700 1 -- 172.21.15.16:0/604553752 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbb6415ba10 msgr2=0x7fbb640c6f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:32.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.696+0000 7fbb62ffd700 1 --2- 172.21.15.16:0/604553752 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbb6415ba10 0x7fbb640c6f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:32.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.696+0000 7fbb62ffd700 1 -- 172.21.15.16:0/604553752 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb54009550 con 0x7fbb6415b630 2023-10-16T12:27:32.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.696+0000 7fbb63fff700 1 --2- 172.21.15.16:0/604553752 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbb6415ba10 0x7fbb640c6f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:27:32.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.696+0000 7fbb62ffd700 1 --2- 172.21.15.16:0/604553752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb6415b630 0x7fbb640c69f0 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7fbb54009130 tx=0x7fbb54004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:32.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.696+0000 7fbb5bfff700 1 -- 172.21.15.16:0/604553752 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb54011630 con 0x7fbb6415b630 2023-10-16T12:27:32.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.696+0000 7fbb5bfff700 1 -- 172.21.15.16:0/604553752 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb54011790 con 0x7fbb6415b630 2023-10-16T12:27:32.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.696+0000 7fbb5bfff700 1 -- 172.21.15.16:0/604553752 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb54011b00 con 0x7fbb6415b630 2023-10-16T12:27:32.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.696+0000 7fbb6a16e700 1 -- 172.21.15.16:0/604553752 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb640c98a0 con 0x7fbb6415b630 2023-10-16T12:27:32.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.696+0000 7fbb6a16e700 1 -- 172.21.15.16:0/604553752 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb640c8f90 con 0x7fbb6415b630 2023-10-16T12:27:32.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.697+0000 7fbb6a16e700 1 -- 172.21.15.16:0/604553752 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb6404ebb0 con 0x7fbb6415b630 2023-10-16T12:27:32.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.698+0000 7fbb5bfff700 1 -- 172.21.15.16:0/604553752 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 51) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fbb54021070 con 0x7fbb6415b630 2023-10-16T12:27:32.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.698+0000 7fbb5bfff700 1 --2- 172.21.15.16:0/604553752 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fbb4406b020 0x7fbb4406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:32.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.698+0000 7fbb5bfff700 1 -- 172.21.15.16:0/604553752 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5979+0+0 (secure 0 0 0) 0x7fbb5408ae90 con 0x7fbb6415b630 2023-10-16T12:27:32.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.698+0000 7fbb637fe700 1 --2- 172.21.15.16:0/604553752 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fbb4406b020 0x7fbb4406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:32.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.698+0000 7fbb637fe700 1 --2- 172.21.15.16:0/604553752 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fbb4406b020 0x7fbb4406d4e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fbb4c00aa50 tx=0x7fbb4c008040 comp rx=0 tx=0).ready entity=mgr.6365 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:32.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.702+0000 7fbb5bfff700 1 -- 172.21.15.16:0/604553752 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbb54055db0 con 0x7fbb6415b630 2023-10-16T12:27:32.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.954+0000 7fbb6a16e700 1 -- 172.21.15.16:0/604553752 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbb640707f0 con 0x7fbb6415b630 2023-10-16T12:27:32.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.955+0000 7fbb5bfff700 1 -- 172.21.15.16:0/604553752 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+3696 (secure 0 0 0) 0x7fbb54028e80 con 0x7fbb6415b630 2023-10-16T12:27:32.958 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:27:32.958 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":46,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6820,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/1256919442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":1256919442},{"type":"v1","addr":"172.21.15.49:6834","nonce":1256919442}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":6823,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/3811815346","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":3811815346},{"type":"v1","addr":"172.21.15.49:6835","nonce":3811815346}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":7110,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/1226064755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":1226064755},{"type":"v1","addr":"172.21.15.16:6836","nonce":1226064755}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":7512,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/784234220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":784234220},{"type":"v1","addr":"172.21.15.16:6837","nonce":784234220}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":46}],"filesystems":[]} 2023-10-16T12:27:32.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.957+0000 7fbb6a16e700 1 -- 172.21.15.16:0/604553752 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fbb4406b020 msgr2=0x7fbb4406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:32.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.957+0000 7fbb6a16e700 1 --2- 172.21.15.16:0/604553752 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fbb4406b020 0x7fbb4406d4e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fbb4c00aa50 tx=0x7fbb4c008040 comp rx=0 tx=0).stop 2023-10-16T12:27:32.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.957+0000 7fbb6a16e700 1 -- 172.21.15.16:0/604553752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb6415b630 msgr2=0x7fbb640c69f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:32.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.957+0000 7fbb6a16e700 1 --2- 172.21.15.16:0/604553752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb6415b630 0x7fbb640c69f0 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7fbb54009130 tx=0x7fbb54004750 comp rx=0 tx=0).stop 2023-10-16T12:27:32.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.957+0000 7fbb6a16e700 1 -- 172.21.15.16:0/604553752 shutdown_connections 2023-10-16T12:27:32.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.957+0000 7fbb6a16e700 1 --2- 172.21.15.16:0/604553752 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fbb4406b020 0x7fbb4406d4e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:32.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.957+0000 7fbb6a16e700 1 --2- 172.21.15.16:0/604553752 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbb6415ba10 0x7fbb640c6f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:32.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.957+0000 7fbb6a16e700 1 --2- 172.21.15.16:0/604553752 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbb640ce320 0x7fbb641f0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:32.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.957+0000 7fbb6a16e700 1 --2- 172.21.15.16:0/604553752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb6415b630 0x7fbb640c69f0 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:32.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.957+0000 7fbb6a16e700 1 -- 172.21.15.16:0/604553752 >> 172.21.15.16:0/604553752 conn(0x7fbb640d14b0 msgr2=0x7fbb640b4a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:32.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.957+0000 7fbb6a16e700 1 -- 172.21.15.16:0/604553752 shutdown_connections 2023-10-16T12:27:32.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:32.957+0000 7fbb6a16e700 1 -- 172.21.15.16:0/604553752 wait complete. 2023-10-16T12:27:32.963 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 46 2023-10-16T12:27:32.973 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-16T12:27:32.973 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-16T12:27:32.973 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-16T12:27:32.974 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T12:27:32.974 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-16T12:27:32.974 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-16T12:27:32.975 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-16T12:27:32.975 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T12:27:32.975 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-16T12:27:32.975 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-16T12:27:32.976 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-16T12:27:32.976 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T12:27:32.976 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-16T12:27:32.976 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-16T12:27:32.977 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-16T12:27:32.977 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T12:27:32.982 INFO:tasks.ceph.mds.a:Stopped 2023-10-16T12:27:32.983 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-16T12:27:32.983 INFO:tasks.ceph.mds.c:Stopped 2023-10-16T12:27:32.984 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-16T12:27:32.987 INFO:tasks.ceph.mds.b:Stopped 2023-10-16T12:27:32.987 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-16T12:27:32.989 INFO:tasks.ceph.mds.d:Stopped 2023-10-16T12:27:32.989 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-16T12:27:33.025 INFO:tasks.ceph.mds.a:Started 2023-10-16T12:27:33.025 DEBUG:teuthology.parallel:result is None 2023-10-16T12:27:33.026 INFO:tasks.ceph.mds.c:Started 2023-10-16T12:27:33.026 DEBUG:teuthology.parallel:result is None 2023-10-16T12:27:33.031 INFO:tasks.ceph.mds.b:Started 2023-10-16T12:27:33.031 DEBUG:teuthology.parallel:result is None 2023-10-16T12:27:33.032 INFO:tasks.ceph.mds.d:Started 2023-10-16T12:27:33.033 DEBUG:teuthology.parallel:result is None 2023-10-16T12:27:33.033 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-16T12:27:33.106 INFO:tasks.ceph.mds.a.smithi016.stdout:starting mds.a at 2023-10-16T12:27:33.107 INFO:tasks.ceph.mds.c.smithi016.stdout:starting mds.c at 2023-10-16T12:27:33.124 INFO:tasks.ceph.mds.b.smithi049.stdout:starting mds.b at 2023-10-16T12:27:33.124 INFO:tasks.ceph.mds.d.smithi049.stdout:starting mds.d at 2023-10-16T12:27:33.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.162+0000 7f4d15fe3700 1 Processor -- start 2023-10-16T12:27:33.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.163+0000 7f4d15fe3700 1 -- start start 2023-10-16T12:27:33.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.163+0000 7f4d15fe3700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4d100d2040 0x7f4d100ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:33.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.163+0000 7f4d15fe3700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d100ce400 con 0x7f4d1015d900 2023-10-16T12:27:33.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.163+0000 7f4d15fe3700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d1015dce0 con 0x7f4d100d2f70 2023-10-16T12:27:33.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.163+0000 7f4d15fe3700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d100d24d0 con 0x7f4d100d2040 2023-10-16T12:27:33.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.163+0000 7f4d0f7fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f4d1015d900 0x7f4d100d2860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59640/0 (socket says 172.21.15.16:59640) 2023-10-16T12:27:33.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.163+0000 7f4d0f7fe700 1 -- 172.21.15.16:0/3912791784 learned_addr learned my addr 172.21.15.16:0/3912791784 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:27:33.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.164+0000 7f4d0ffff700 1 --2- 172.21.15.16:0/3912791784 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4d100d2040 0x7f4d100ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:33.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.164+0000 7f4d0e7fc700 1 -- 172.21.15.16:0/3912791784 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3243495519 0 0) 0x7f4d100ce400 con 0x7f4d1015d900 2023-10-16T12:27:33.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.164+0000 7f4d0e7fc700 1 -- 172.21.15.16:0/3912791784 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4cf80035b0 con 0x7f4d1015d900 2023-10-16T12:27:33.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.164+0000 7f4d0e7fc700 1 -- 172.21.15.16:0/3912791784 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 628621359 0 0) 0x7f4cf80035b0 con 0x7f4d1015d900 2023-10-16T12:27:33.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.164+0000 7f4d0e7fc700 1 -- 172.21.15.16:0/3912791784 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4d100d2040 msgr2=0x7f4d100ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:33.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.164+0000 7f4d0e7fc700 1 --2- 172.21.15.16:0/3912791784 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4d100d2040 0x7f4d100ce9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:33.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.164+0000 7f4d0e7fc700 1 -- 172.21.15.16:0/3912791784 >> v1:172.21.15.49:6789/0 conn(0x7f4d100d2f70 legacy=0x7f4d100b6790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:33.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.164+0000 7f4d0e7fc700 1 -- 172.21.15.16:0/3912791784 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d100d1280 con 0x7f4d1015d900 2023-10-16T12:27:33.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.164+0000 7f4d0e7fc700 1 -- 172.21.15.16:0/3912791784 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f4d00001670 con 0x7f4d1015d900 2023-10-16T12:27:33.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.164+0000 7f4d0ffff700 1 --2- 172.21.15.16:0/3912791784 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4d100d2040 0x7f4d100ce9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:27:33.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.164+0000 7f4d0e7fc700 1 -- 172.21.15.16:0/3912791784 >> v1:172.21.15.16:6789/0 conn(0x7f4d1015d900 legacy=0x7f4d100d2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:33.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.164+0000 7f4d0e7fc700 1 --2- 172.21.15.16:0/3912791784 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4d100d2040 0x7f4cf8003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:33.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.164+0000 7f4d0e7fc700 1 --2- 172.21.15.16:0/3912791784 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4cf8003f70 0x7f4cf8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:33.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.164+0000 7f4d0e7fc700 1 --2- 172.21.15.16:0/3912791784 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cf8004d90 0x7f4cf8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:33.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.164+0000 7f4d0e7fc700 1 -- 172.21.15.16:0/3912791784 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d100d24d0 con 0x7f4cf8004d90 2023-10-16T12:27:33.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.164+0000 7f4d0e7fc700 1 -- 172.21.15.16:0/3912791784 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf800ba70 con 0x7f4cf8003f70 2023-10-16T12:27:33.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.164+0000 7f4d0e7fc700 1 -- 172.21.15.16:0/3912791784 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf800bc30 con 0x7f4d100d2040 2023-10-16T12:27:33.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.164+0000 7f4d0ffff700 1 --2- 172.21.15.16:0/3912791784 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cf8004d90 0x7f4cf8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:33.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.164+0000 7f4d0f7fe700 1 --2- 172.21.15.16:0/3912791784 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4d100d2040 0x7f4cf8003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:33.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.164+0000 7f4d0effd700 1 --2- 172.21.15.16:0/3912791784 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4cf8003f70 0x7f4cf8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:33.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.164+0000 7f4d0ffff700 1 -- 172.21.15.16:0/3912791784 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4d100d2040 msgr2=0x7f4cf8003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:33.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.164+0000 7f4d0ffff700 1 --2- 172.21.15.16:0/3912791784 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4d100d2040 0x7f4cf8003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:33.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.164+0000 7f4d0ffff700 1 -- 172.21.15.16:0/3912791784 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4cf8003f70 msgr2=0x7f4cf8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:33.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.165+0000 7f4d0ffff700 1 --2- 172.21.15.16:0/3912791784 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4cf8003f70 0x7f4cf8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:33.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.165+0000 7f4d0ffff700 1 -- 172.21.15.16:0/3912791784 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d08009a70 con 0x7f4cf8004d90 2023-10-16T12:27:33.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.165+0000 7f4d0effd700 1 --2- 172.21.15.16:0/3912791784 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4cf8003f70 0x7f4cf8004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:27:33.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.165+0000 7f4d0ffff700 1 --2- 172.21.15.16:0/3912791784 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cf8004d90 0x7f4cf8005210 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f4d0800e350 tx=0x7f4cf8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4636d2bdd9b131e5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:33.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.165+0000 7f4d0e7fc700 1 -- 172.21.15.16:0/3912791784 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d0800ee10 con 0x7f4cf8004d90 2023-10-16T12:27:33.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.165+0000 7f4d0e7fc700 1 -- 172.21.15.16:0/3912791784 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d0800f040 con 0x7f4cf8004d90 2023-10-16T12:27:33.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.165+0000 7f4d0e7fc700 1 -- 172.21.15.16:0/3912791784 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d08003c40 con 0x7f4cf8004d90 2023-10-16T12:27:33.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.165+0000 7f4d15fe3700 1 -- 172.21.15.16:0/3912791784 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cf8004d90 msgr2=0x7f4cf8005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:33.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.165+0000 7f4d15fe3700 1 --2- 172.21.15.16:0/3912791784 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cf8004d90 0x7f4cf8005210 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f4d0800e350 tx=0x7f4cf8005880 comp rx=0 tx=0).stop 2023-10-16T12:27:33.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.165+0000 7f4d0ffff700 1 -- 172.21.15.16:0/3912791784 reap_dead start 2023-10-16T12:27:33.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.165+0000 7f4d15fe3700 1 -- 172.21.15.16:0/3912791784 shutdown_connections 2023-10-16T12:27:33.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.165+0000 7f4d15fe3700 1 -- 172.21.15.16:0/3912791784 >> 172.21.15.16:0/3912791784 conn(0x7f4d10072330 msgr2=0x7f4d10072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:33.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.165+0000 7f4d15fe3700 1 -- 172.21.15.16:0/3912791784 shutdown_connections 2023-10-16T12:27:33.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.165+0000 7f4d15fe3700 1 -- 172.21.15.16:0/3912791784 wait complete. 2023-10-16T12:27:33.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.167+0000 7f4d15fe3700 1 Processor -- start 2023-10-16T12:27:33.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.167+0000 7f4d15fe3700 1 -- start start 2023-10-16T12:27:33.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.168+0000 7f4d15fe3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d101f23c0 0x7f4d100b4150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:33.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.168+0000 7f4d15fe3700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4d100b5cf0 0x7f4d1006b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:33.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.168+0000 7f4d15fe3700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4d101f27a0 0x7f4d1006bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:33.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.168+0000 7f4d15fe3700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d100b5660 con 0x7f4d101f23c0 2023-10-16T12:27:33.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.168+0000 7f4d15fe3700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d100b4a50 con 0x7f4d101f27a0 2023-10-16T12:27:33.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.168+0000 7f4d15fe3700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d100b4d30 con 0x7f4d100b5cf0 2023-10-16T12:27:33.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.168+0000 7f4d0f7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d101f23c0 0x7f4d100b4150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:33.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.168+0000 7f4d0f7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d101f23c0 0x7f4d100b4150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42146/0 (socket says 172.21.15.16:42146) 2023-10-16T12:27:33.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.168+0000 7f4d0f7fe700 1 -- 172.21.15.16:0/1211470674 learned_addr learned my addr 172.21.15.16:0/1211470674 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:33.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.168+0000 7f4d0ffff700 1 --2- 172.21.15.16:0/1211470674 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4d101f27a0 0x7f4d1006bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:33.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.168+0000 7f4d0effd700 1 --2- 172.21.15.16:0/1211470674 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4d100b5cf0 0x7f4d1006b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:33.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.168+0000 7f4d0f7fe700 1 -- 172.21.15.16:0/1211470674 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4d100b5cf0 msgr2=0x7f4d1006b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:33.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.168+0000 7f4d0f7fe700 1 --2- 172.21.15.16:0/1211470674 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4d100b5cf0 0x7f4d1006b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:33.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.168+0000 7f4d0f7fe700 1 -- 172.21.15.16:0/1211470674 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4d101f27a0 msgr2=0x7f4d1006bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:33.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.168+0000 7f4d0f7fe700 1 --2- 172.21.15.16:0/1211470674 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4d101f27a0 0x7f4d1006bd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:33.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.168+0000 7f4d0f7fe700 1 -- 172.21.15.16:0/1211470674 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d100b5010 con 0x7f4d101f23c0 2023-10-16T12:27:33.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.168+0000 7f4d0effd700 1 --2- 172.21.15.16:0/1211470674 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4d100b5cf0 0x7f4d1006b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:27:33.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.168+0000 7f4d0f7fe700 1 --2- 172.21.15.16:0/1211470674 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d101f23c0 0x7f4d100b4150 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f4d00007290 tx=0x7f4d000075a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:33.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.168+0000 7f4cf7fff700 1 -- 172.21.15.16:0/1211470674 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d00007780 con 0x7f4d101f23c0 2023-10-16T12:27:33.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.168+0000 7f4cf7fff700 1 -- 172.21.15.16:0/1211470674 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d0000f8d0 con 0x7f4d101f23c0 2023-10-16T12:27:33.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.168+0000 7f4d15fe3700 1 -- 172.21.15.16:0/1211470674 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d100b52f0 con 0x7f4d101f23c0 2023-10-16T12:27:33.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.168+0000 7f4cf7fff700 1 -- 172.21.15.16:0/1211470674 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d0000fc80 con 0x7f4d101f23c0 2023-10-16T12:27:33.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.168+0000 7f4d15fe3700 1 -- 172.21.15.16:0/1211470674 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d1014a940 con 0x7f4d101f23c0 2023-10-16T12:27:33.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.169+0000 7f4d15fe3700 1 -- 172.21.15.16:0/1211470674 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d1014ab30 con 0x7f4d101f23c0 2023-10-16T12:27:33.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.170+0000 7f4cf7fff700 1 -- 172.21.15.16:0/1211470674 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 51) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4d0001d460 con 0x7f4d101f23c0 2023-10-16T12:27:33.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.170+0000 7f4cf7fff700 1 --2- 172.21.15.16:0/1211470674 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f4cfc06b050 0x7f4cfc06d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:33.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.170+0000 7f4cf7fff700 1 -- 172.21.15.16:0/1211470674 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5979+0+0 (secure 0 0 0) 0x7f4d000899c0 con 0x7f4d101f23c0 2023-10-16T12:27:33.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.170+0000 7f4d0effd700 1 --2- 172.21.15.16:0/1211470674 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f4cfc06b050 0x7f4cfc06d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:33.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.171+0000 7f4d0effd700 1 --2- 172.21.15.16:0/1211470674 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f4cfc06b050 0x7f4cfc06d510 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4d04007bd0 tx=0x7f4d04009040 comp rx=0 tx=0).ready entity=mgr.6365 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:33.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.175+0000 7f4cf7fff700 1 -- 172.21.15.16:0/1211470674 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4d1014ab30 con 0x7f4d101f23c0 2023-10-16T12:27:33.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.397+0000 7f4d15fe3700 1 -- 172.21.15.16:0/1211470674 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f4d1014ab30 con 0x7f4d101f23c0 2023-10-16T12:27:33.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.962+0000 7f4cf7fff700 1 -- 172.21.15.16:0/1211470674 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v106) v1 ==== 97+0+0 (secure 0 0 0) 0x7f4d1014ab30 con 0x7f4d101f23c0 2023-10-16T12:27:33.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.966+0000 7f4d15fe3700 1 -- 172.21.15.16:0/1211470674 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f4cfc06b050 msgr2=0x7f4cfc06d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:33.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.966+0000 7f4d15fe3700 1 --2- 172.21.15.16:0/1211470674 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f4cfc06b050 0x7f4cfc06d510 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4d04007bd0 tx=0x7f4d04009040 comp rx=0 tx=0).stop 2023-10-16T12:27:33.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.966+0000 7f4d15fe3700 1 -- 172.21.15.16:0/1211470674 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d101f23c0 msgr2=0x7f4d100b4150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:33.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.966+0000 7f4d15fe3700 1 --2- 172.21.15.16:0/1211470674 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d101f23c0 0x7f4d100b4150 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f4d00007290 tx=0x7f4d000075a0 comp rx=0 tx=0).stop 2023-10-16T12:27:33.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.966+0000 7f4d15fe3700 1 -- 172.21.15.16:0/1211470674 shutdown_connections 2023-10-16T12:27:33.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.966+0000 7f4d15fe3700 1 --2- 172.21.15.16:0/1211470674 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f4cfc06b050 0x7f4cfc06d510 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:33.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.966+0000 7f4d15fe3700 1 --2- 172.21.15.16:0/1211470674 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4d101f27a0 0x7f4d1006bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:33.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.966+0000 7f4d15fe3700 1 --2- 172.21.15.16:0/1211470674 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d101f23c0 0x7f4d100b4150 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:33.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.966+0000 7f4d15fe3700 1 --2- 172.21.15.16:0/1211470674 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4d100b5cf0 0x7f4d1006b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:33.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.966+0000 7f4d15fe3700 1 -- 172.21.15.16:0/1211470674 >> 172.21.15.16:0/1211470674 conn(0x7f4d10072330 msgr2=0x7f4d100d2710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:33.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.966+0000 7f4d15fe3700 1 -- 172.21.15.16:0/1211470674 shutdown_connections 2023-10-16T12:27:33.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:33.966+0000 7f4d15fe3700 1 -- 172.21.15.16:0/1211470674 wait complete. 2023-10-16T12:27:33.971 INFO:teuthology.orchestra.run.smithi016.stderr: removed all blocklist entries 2023-10-16T12:27:33.987 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-16T12:27:34.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.116+0000 7ff581f2a700 1 Processor -- start 2023-10-16T12:27:34.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.116+0000 7ff581f2a700 1 -- start start 2023-10-16T12:27:34.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.117+0000 7ff581f2a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff57c15b630 0x7ff57c0d0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:34.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.117+0000 7ff581f2a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff57c15ba10 0x7ff57c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:34.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.117+0000 7ff581f2a700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7ff57c0cdde0 0x7ff57c0ce230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:34.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.117+0000 7ff581f2a700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff57c0ce770 con 0x7ff57c15b630 2023-10-16T12:27:34.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.117+0000 7ff581f2a700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff57c0b0970 con 0x7ff57c0cdde0 2023-10-16T12:27:34.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.117+0000 7ff581f2a700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff57c0cd690 con 0x7ff57c15ba10 2023-10-16T12:27:34.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.117+0000 7ff57b7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff57c15b630 0x7ff57c0d0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:34.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.117+0000 7ff57b7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff57c15b630 0x7ff57c0d0380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42156/0 (socket says 172.21.15.16:42156) 2023-10-16T12:27:34.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.117+0000 7ff57b7fe700 1 -- 172.21.15.16:0/1430022889 learned_addr learned my addr 172.21.15.16:0/1430022889 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:34.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.117+0000 7ff57bfff700 1 --2- 172.21.15.16:0/1430022889 >> v2:172.21.15.49:3300/0 conn(0x7ff57c0cdde0 0x7ff57c0ce230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:34.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.117+0000 7ff57affd700 1 --2- 172.21.15.16:0/1430022889 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff57c15ba10 0x7ff57c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:34.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.117+0000 7ff57b7fe700 1 -- 172.21.15.16:0/1430022889 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff57c15ba10 msgr2=0x7ff57c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:34.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.117+0000 7ff57b7fe700 1 --2- 172.21.15.16:0/1430022889 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff57c15ba10 0x7ff57c0cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:34.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.117+0000 7ff57b7fe700 1 -- 172.21.15.16:0/1430022889 >> v2:172.21.15.49:3300/0 conn(0x7ff57c0cdde0 msgr2=0x7ff57c0ce230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:34.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.117+0000 7ff57b7fe700 1 --2- 172.21.15.16:0/1430022889 >> v2:172.21.15.49:3300/0 conn(0x7ff57c0cdde0 0x7ff57c0ce230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:34.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.117+0000 7ff57b7fe700 1 -- 172.21.15.16:0/1430022889 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff57c0cd970 con 0x7ff57c15b630 2023-10-16T12:27:34.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.117+0000 7ff57b7fe700 1 --2- 172.21.15.16:0/1430022889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff57c15b630 0x7ff57c0d0380 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7ff57c0cef90 tx=0x7ff56c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d33bd9b4afd7c2f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:34.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.117+0000 7ff57a7fc700 1 -- 172.21.15.16:0/1430022889 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff56c00b870 con 0x7ff57c15b630 2023-10-16T12:27:34.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.117+0000 7ff57a7fc700 1 -- 172.21.15.16:0/1430022889 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff56c00b9d0 con 0x7ff57c15b630 2023-10-16T12:27:34.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.117+0000 7ff57a7fc700 1 -- 172.21.15.16:0/1430022889 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff56c00bd60 con 0x7ff57c15b630 2023-10-16T12:27:34.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.118+0000 7ff581f2a700 1 -- 172.21.15.16:0/1430022889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff57c15b630 msgr2=0x7ff57c0d0380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:34.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.118+0000 7ff581f2a700 1 --2- 172.21.15.16:0/1430022889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff57c15b630 0x7ff57c0d0380 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7ff57c0cef90 tx=0x7ff56c009bd0 comp rx=0 tx=0).stop 2023-10-16T12:27:34.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.118+0000 7ff581f2a700 1 -- 172.21.15.16:0/1430022889 shutdown_connections 2023-10-16T12:27:34.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.118+0000 7ff581f2a700 1 --2- 172.21.15.16:0/1430022889 >> v2:172.21.15.49:3300/0 conn(0x7ff57c0cdde0 0x7ff57c0ce230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:34.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.118+0000 7ff581f2a700 1 --2- 172.21.15.16:0/1430022889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff57c15b630 0x7ff57c0d0380 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:34.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.118+0000 7ff581f2a700 1 --2- 172.21.15.16:0/1430022889 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff57c15ba10 0x7ff57c0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:34.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.118+0000 7ff581f2a700 1 -- 172.21.15.16:0/1430022889 >> 172.21.15.16:0/1430022889 conn(0x7ff57c0d14b0 msgr2=0x7ff57c0d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:34.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.118+0000 7ff581f2a700 1 -- 172.21.15.16:0/1430022889 shutdown_connections 2023-10-16T12:27:34.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.118+0000 7ff581f2a700 1 -- 172.21.15.16:0/1430022889 wait complete. 2023-10-16T12:27:34.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.119+0000 7ff581f2a700 1 Processor -- start 2023-10-16T12:27:34.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.119+0000 7ff581f2a700 1 -- start start 2023-10-16T12:27:34.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.120+0000 7ff581f2a700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff57c0cdde0 0x7ff57c1489c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:34.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.120+0000 7ff581f2a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff57c15b630 0x7ff57c06c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:34.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.120+0000 7ff581f2a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff57c15ba10 0x7ff57c06c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:34.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.120+0000 7ff581f2a700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff57c0c5fb0 con 0x7ff57c15ba10 2023-10-16T12:27:34.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.120+0000 7ff581f2a700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff57c0c7df0 con 0x7ff57c0cdde0 2023-10-16T12:27:34.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.120+0000 7ff581f2a700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff57c0c7270 con 0x7ff57c15b630 2023-10-16T12:27:34.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.120+0000 7ff57bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff57c15ba10 0x7ff57c06c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:34.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.120+0000 7ff57bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff57c15ba10 0x7ff57c06c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42164/0 (socket says 172.21.15.16:42164) 2023-10-16T12:27:34.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.120+0000 7ff57bfff700 1 -- 172.21.15.16:0/500501171 learned_addr learned my addr 172.21.15.16:0/500501171 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:34.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.120+0000 7ff57b7fe700 1 --2- 172.21.15.16:0/500501171 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff57c0cdde0 0x7ff57c1489c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:34.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.120+0000 7ff57affd700 1 --2- 172.21.15.16:0/500501171 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff57c15b630 0x7ff57c06c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:34.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.121+0000 7ff57bfff700 1 -- 172.21.15.16:0/500501171 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff57c15b630 msgr2=0x7ff57c06c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:34.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.121+0000 7ff57bfff700 1 --2- 172.21.15.16:0/500501171 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff57c15b630 0x7ff57c06c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:34.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.121+0000 7ff57bfff700 1 -- 172.21.15.16:0/500501171 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff57c0cdde0 msgr2=0x7ff57c1489c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:34.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.121+0000 7ff57bfff700 1 --2- 172.21.15.16:0/500501171 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff57c0cdde0 0x7ff57c1489c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:34.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.121+0000 7ff57bfff700 1 -- 172.21.15.16:0/500501171 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff56c009550 con 0x7ff57c15ba10 2023-10-16T12:27:34.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.121+0000 7ff57bfff700 1 --2- 172.21.15.16:0/500501171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff57c15ba10 0x7ff57c06c840 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7ff574007e60 tx=0x7ff57400e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:34.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.121+0000 7ff563fff700 1 -- 172.21.15.16:0/500501171 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff57400ed30 con 0x7ff57c15ba10 2023-10-16T12:27:34.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.121+0000 7ff563fff700 1 -- 172.21.15.16:0/500501171 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff574014420 con 0x7ff57c15ba10 2023-10-16T12:27:34.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.121+0000 7ff581f2a700 1 -- 172.21.15.16:0/500501171 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff57c0c7550 con 0x7ff57c15ba10 2023-10-16T12:27:34.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.121+0000 7ff563fff700 1 -- 172.21.15.16:0/500501171 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5740147d0 con 0x7ff57c15ba10 2023-10-16T12:27:34.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.121+0000 7ff581f2a700 1 -- 172.21.15.16:0/500501171 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff57c0c7b10 con 0x7ff57c15ba10 2023-10-16T12:27:34.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.122+0000 7ff581f2a700 1 -- 172.21.15.16:0/500501171 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff57c0d12b0 con 0x7ff57c15ba10 2023-10-16T12:27:34.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.122+0000 7ff563fff700 1 -- 172.21.15.16:0/500501171 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 51) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff574014930 con 0x7ff57c15ba10 2023-10-16T12:27:34.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.123+0000 7ff563fff700 1 --2- 172.21.15.16:0/500501171 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7ff56806b020 0x7ff56806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:34.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.123+0000 7ff563fff700 1 -- 172.21.15.16:0/500501171 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5205+0+0 (secure 0 0 0) 0x7ff574089a70 con 0x7ff57c15ba10 2023-10-16T12:27:34.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.123+0000 7ff57b7fe700 1 --2- 172.21.15.16:0/500501171 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7ff56806b020 0x7ff56806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:34.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.123+0000 7ff57b7fe700 1 --2- 172.21.15.16:0/500501171 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7ff56806b020 0x7ff56806d4e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff56c000c00 tx=0x7ff56c008030 comp rx=0 tx=0).ready entity=mgr.6365 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:34.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.128+0000 7ff563fff700 1 -- 172.21.15.16:0/500501171 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff574054c90 con 0x7ff57c15ba10 2023-10-16T12:27:34.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.411+0000 7ff581f2a700 1 -- 172.21.15.16:0/500501171 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7ff57c15bee0 con 0x7ff57c15ba10 2023-10-16T12:27:34.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.412+0000 7ff563fff700 1 -- 172.21.15.16:0/500501171 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v21) v1 ==== 80+0+5304 (secure 0 0 0) 0x7ff57c15bee0 con 0x7ff57c15ba10 2023-10-16T12:27:34.415 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:27:34.415 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-10-16T12:27:34.416 INFO:teuthology.orchestra.run.smithi016.stdout: "auth_dump": [ 2023-10-16T12:27:34.416 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:27:34.416 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.a", 2023-10-16T12:27:34.416 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOKC1l/rBtOhAA/ob5WUvpz8Glp4E+xPZOYA==", 2023-10-16T12:27:34.417 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:27:34.417 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T12:27:34.417 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-10-16T12:27:34.417 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-10-16T12:27:34.417 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:27:34.417 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:27:34.418 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:27:34.418 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:27:34.418 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.b", 2023-10-16T12:27:34.418 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1lKAVsCxAAvhXuLY6i1+Iz1yq5BVPMNQ==", 2023-10-16T12:27:34.418 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:27:34.419 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T12:27:34.419 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-10-16T12:27:34.419 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-10-16T12:27:34.419 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:27:34.419 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:27:34.419 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:27:34.420 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:27:34.420 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.c", 2023-10-16T12:27:34.420 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1lvBowBhAA3waJNYhdgs8PGCYqYNYUog==", 2023-10-16T12:27:34.420 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:27:34.420 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T12:27:34.421 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-10-16T12:27:34.421 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-10-16T12:27:34.421 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:27:34.421 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:27:34.421 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:27:34.422 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:27:34.422 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.d", 2023-10-16T12:27:34.422 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1lS/fnEhAAfNYgQ5BW86JMjYm/bBfnaQ==", 2023-10-16T12:27:34.422 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:27:34.422 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T12:27:34.422 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-10-16T12:27:34.423 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-10-16T12:27:34.423 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:27:34.423 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:27:34.423 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:27:34.423 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:27:34.424 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.0", 2023-10-16T12:27:34.424 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAQKC1lbxllOhAAZeLdp2GMl6HqQPaQjKlNqw==", 2023-10-16T12:27:34.424 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:27:34.424 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T12:27:34.424 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T12:27:34.424 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:27:34.425 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:27:34.425 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:27:34.425 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:27:34.425 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.1", 2023-10-16T12:27:34.425 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQATKC1lnU6KOhAAHxvlbmzBptf1keGgL+Rbkg==", 2023-10-16T12:27:34.426 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:27:34.426 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T12:27:34.426 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T12:27:34.426 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:27:34.426 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:27:34.427 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:27:34.427 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:27:34.427 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.2", 2023-10-16T12:27:34.427 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAXKC1l9jFyARAAr/NAwtWhBMut9sAAoDEGWg==", 2023-10-16T12:27:34.427 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:27:34.427 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T12:27:34.428 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T12:27:34.428 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:27:34.428 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:27:34.428 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:27:34.428 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:27:34.429 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.3", 2023-10-16T12:27:34.429 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAaKC1lbw/lARAAGJ/9DNb5/7kLFSq7vrLtXg==", 2023-10-16T12:27:34.429 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:27:34.429 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T12:27:34.430 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T12:27:34.430 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:27:34.430 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:27:34.430 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:27:34.430 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:27:34.431 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.4", 2023-10-16T12:27:34.431 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAeKC1lZb7jERAAYpvb1xcSli9CTq7L8ChLOA==", 2023-10-16T12:27:34.431 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:27:34.431 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T12:27:34.431 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T12:27:34.432 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:27:34.432 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:27:34.432 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:27:34.432 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:27:34.432 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.5", 2023-10-16T12:27:34.433 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAhKC1l/4HuEBAAqfnRqjknij9KES/fqiyIBQ==", 2023-10-16T12:27:34.433 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:27:34.433 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T12:27:34.433 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T12:27:34.433 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:27:34.433 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:27:34.434 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:27:34.434 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:27:34.434 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.6", 2023-10-16T12:27:34.434 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAkKC1lzoUkEBAA+Q82OiBMNag8+R316WQtUw==", 2023-10-16T12:27:34.434 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:27:34.435 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T12:27:34.435 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T12:27:34.435 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:27:34.435 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:27:34.435 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:27:34.435 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:27:34.436 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.7", 2023-10-16T12:27:34.436 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAnKC1ljUu4DxAAc+G4anJANC2GhLJ8wb6gYg==", 2023-10-16T12:27:34.436 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:27:34.436 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T12:27:34.436 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T12:27:34.437 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:27:34.437 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:27:34.437 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:27:34.437 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:27:34.437 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.0", 2023-10-16T12:27:34.438 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1l2l6PFxAA9QyXtBAHdGAXpwH0JfknuQ==", 2023-10-16T12:27:34.438 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:27:34.438 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T12:27:34.438 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow r", 2023-10-16T12:27:34.438 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-16T12:27:34.438 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:27:34.439 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:27:34.439 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:27:34.439 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.1", 2023-10-16T12:27:34.439 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1lAcHFIxAAcNt2bmv2R6j5L3DYkdw7eQ==", 2023-10-16T12:27:34.439 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:27:34.440 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T12:27:34.440 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-10-16T12:27:34.440 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-10-16T12:27:34.441 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-10-16T12:27:34.441 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:27:34.442 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:27:34.442 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:27:34.442 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.2", 2023-10-16T12:27:34.442 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1lA6wnGxAAdtaqYhtIJu8qENMrD8xvVA==", 2023-10-16T12:27:34.442 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:27:34.443 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T12:27:34.443 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-10-16T12:27:34.443 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-10-16T12:27:34.443 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-10-16T12:27:34.443 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:27:34.444 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:27:34.444 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:27:34.444 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.3", 2023-10-16T12:27:34.444 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1l/ddAJxAA3fNKKn9XkXas1BlDN+ggjw==", 2023-10-16T12:27:34.444 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:27:34.444 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T12:27:34.445 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-10-16T12:27:34.445 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-10-16T12:27:34.445 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-10-16T12:27:34.445 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:27:34.445 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:27:34.446 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:27:34.446 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.admin", 2023-10-16T12:27:34.446 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOKC1ld5jUHBAAc6esnCce3dSEZn3wVjhIlQ==", 2023-10-16T12:27:34.446 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:27:34.446 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2023-10-16T12:27:34.447 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-10-16T12:27:34.447 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-10-16T12:27:34.447 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:27:34.447 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:27:34.447 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:27:34.448 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:27:34.448 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mgr.x", 2023-10-16T12:27:34.448 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOKC1l6b3vNhAAmj5xbfid2sZM2KfHmAf1Wg==", 2023-10-16T12:27:34.448 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:27:34.448 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2023-10-16T12:27:34.449 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile mgr", 2023-10-16T12:27:34.449 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:27:34.449 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:27:34.449 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:27:34.449 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:27:34.450 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mgr.y", 2023-10-16T12:27:34.450 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOKC1lLz2nMhAA8NqFYZ5nDKkzPBMrkwsVuQ==", 2023-10-16T12:27:34.450 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:27:34.450 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2023-10-16T12:27:34.450 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile mgr", 2023-10-16T12:27:34.451 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:27:34.451 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:27:34.451 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:27:34.451 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-10-16T12:27:34.451 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-10-16T12:27:34.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.414+0000 7ff581f2a700 1 -- 172.21.15.16:0/500501171 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7ff56806b020 msgr2=0x7ff56806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:34.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.414+0000 7ff581f2a700 1 --2- 172.21.15.16:0/500501171 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7ff56806b020 0x7ff56806d4e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff56c000c00 tx=0x7ff56c008030 comp rx=0 tx=0).stop 2023-10-16T12:27:34.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.414+0000 7ff581f2a700 1 -- 172.21.15.16:0/500501171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff57c15ba10 msgr2=0x7ff57c06c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:34.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.414+0000 7ff581f2a700 1 --2- 172.21.15.16:0/500501171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff57c15ba10 0x7ff57c06c840 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7ff574007e60 tx=0x7ff57400e3c0 comp rx=0 tx=0).stop 2023-10-16T12:27:34.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.414+0000 7ff581f2a700 1 -- 172.21.15.16:0/500501171 shutdown_connections 2023-10-16T12:27:34.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.414+0000 7ff581f2a700 1 --2- 172.21.15.16:0/500501171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff57c15ba10 0x7ff57c06c840 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:34.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.414+0000 7ff581f2a700 1 --2- 172.21.15.16:0/500501171 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7ff56806b020 0x7ff56806d4e0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:34.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.414+0000 7ff581f2a700 1 --2- 172.21.15.16:0/500501171 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff57c0cdde0 0x7ff57c1489c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:34.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.414+0000 7ff581f2a700 1 --2- 172.21.15.16:0/500501171 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff57c15b630 0x7ff57c06c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:34.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.414+0000 7ff581f2a700 1 -- 172.21.15.16:0/500501171 >> 172.21.15.16:0/500501171 conn(0x7ff57c0d14b0 msgr2=0x7ff57c06d270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:34.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.414+0000 7ff581f2a700 1 -- 172.21.15.16:0/500501171 shutdown_connections 2023-10-16T12:27:34.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.414+0000 7ff581f2a700 1 -- 172.21.15.16:0/500501171 wait complete. 2023-10-16T12:27:34.456 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:27:34.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.579+0000 7f6b3e3a5700 1 Processor -- start 2023-10-16T12:27:34.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.579+0000 7f6b3e3a5700 1 -- start start 2023-10-16T12:27:34.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.580+0000 7f6b3e3a5700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6b3815b280 0x7f6b381613e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:34.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.580+0000 7f6b3e3a5700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6b380cc8a0 0x7f6b3815e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:34.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.580+0000 7f6b3e3a5700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b38161c30 con 0x7f6b3806a7e0 2023-10-16T12:27:34.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.580+0000 7f6b3e3a5700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b38161f10 con 0x7f6b380cc8a0 2023-10-16T12:27:34.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.580+0000 7f6b3e3a5700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b381621f0 con 0x7f6b3815b280 2023-10-16T12:27:34.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.580+0000 7f6b377fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6b380cc8a0 0x7f6b3815e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:34.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.580+0000 7f6b377fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6b380cc8a0 0x7f6b3815e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42168/0 (socket says 172.21.15.16:42168) 2023-10-16T12:27:34.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.580+0000 7f6b377fe700 1 -- 172.21.15.16:0/4257433949 learned_addr learned my addr 172.21.15.16:0/4257433949 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:34.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.580+0000 7f6b37fff700 1 --2- 172.21.15.16:0/4257433949 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6b3815b280 0x7f6b381613e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:34.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.580+0000 7f6b377fe700 1 -- 172.21.15.16:0/4257433949 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6b3815b280 msgr2=0x7f6b381613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:34.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.580+0000 7f6b377fe700 1 --2- 172.21.15.16:0/4257433949 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6b3815b280 0x7f6b381613e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:34.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.580+0000 7f6b377fe700 1 -- 172.21.15.16:0/4257433949 >> v1:172.21.15.49:6789/0 conn(0x7f6b3806a7e0 legacy=0x7f6b380d02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:34.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.580+0000 7f6b377fe700 1 -- 172.21.15.16:0/4257433949 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b3815df00 con 0x7f6b380cc8a0 2023-10-16T12:27:34.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.580+0000 7f6b377fe700 1 --2- 172.21.15.16:0/4257433949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b380cc8a0 0x7f6b3815e700 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f6b200098f0 tx=0x7f6b20009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c162168717e802b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:34.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.580+0000 7f6b36ffd700 1 -- 172.21.15.16:0/4257433949 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b2000b8a0 con 0x7f6b380cc8a0 2023-10-16T12:27:34.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.581+0000 7f6b36ffd700 1 -- 172.21.15.16:0/4257433949 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b2000ba00 con 0x7f6b380cc8a0 2023-10-16T12:27:34.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.581+0000 7f6b36ffd700 1 -- 172.21.15.16:0/4257433949 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b2000bd90 con 0x7f6b380cc8a0 2023-10-16T12:27:34.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.581+0000 7f6b3e3a5700 1 -- 172.21.15.16:0/4257433949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b380cc8a0 msgr2=0x7f6b3815e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:34.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.581+0000 7f6b3e3a5700 1 --2- 172.21.15.16:0/4257433949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b380cc8a0 0x7f6b3815e700 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f6b200098f0 tx=0x7f6b20009c00 comp rx=0 tx=0).stop 2023-10-16T12:27:34.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.581+0000 7f6b3e3a5700 1 -- 172.21.15.16:0/4257433949 shutdown_connections 2023-10-16T12:27:34.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.581+0000 7f6b3e3a5700 1 --2- 172.21.15.16:0/4257433949 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6b3815b280 0x7f6b381613e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:34.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.581+0000 7f6b3e3a5700 1 --2- 172.21.15.16:0/4257433949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b380cc8a0 0x7f6b3815e700 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:34.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.581+0000 7f6b3e3a5700 1 -- 172.21.15.16:0/4257433949 >> 172.21.15.16:0/4257433949 conn(0x7f6b380b1fd0 msgr2=0x7f6b380b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:34.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.581+0000 7f6b3e3a5700 1 -- 172.21.15.16:0/4257433949 shutdown_connections 2023-10-16T12:27:34.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.581+0000 7f6b3e3a5700 1 -- 172.21.15.16:0/4257433949 wait complete. 2023-10-16T12:27:34.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.582+0000 7f6b3e3a5700 1 Processor -- start 2023-10-16T12:27:34.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.583+0000 7f6b3e3a5700 1 -- start start 2023-10-16T12:27:34.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.583+0000 7f6b3e3a5700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6b3806a7e0 0x7f6b380c5be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:34.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.583+0000 7f6b3e3a5700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6b380cc8a0 0x7f6b381589d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:34.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.583+0000 7f6b3e3a5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b3815b280 0x7f6b3815f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:34.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.583+0000 7f6b3e3a5700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b380c6c50 con 0x7f6b3815b280 2023-10-16T12:27:34.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.583+0000 7f6b3e3a5700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b380c5750 con 0x7f6b380cc8a0 2023-10-16T12:27:34.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.583+0000 7f6b3e3a5700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b38158f10 con 0x7f6b3806a7e0 2023-10-16T12:27:34.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.583+0000 7f6b3c942700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b3815b280 0x7f6b3815f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:34.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.583+0000 7f6b3c942700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b3815b280 0x7f6b3815f480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42178/0 (socket says 172.21.15.16:42178) 2023-10-16T12:27:34.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.583+0000 7f6b3c942700 1 -- 172.21.15.16:0/3202760757 learned_addr learned my addr 172.21.15.16:0/3202760757 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:34.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.584+0000 7f6b37fff700 1 --2- 172.21.15.16:0/3202760757 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6b3806a7e0 0x7f6b380c5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:34.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.584+0000 7f6b377fe700 1 --2- 172.21.15.16:0/3202760757 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6b380cc8a0 0x7f6b381589d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:34.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.584+0000 7f6b3c942700 1 -- 172.21.15.16:0/3202760757 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6b3806a7e0 msgr2=0x7f6b380c5be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:34.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.584+0000 7f6b3c942700 1 --2- 172.21.15.16:0/3202760757 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6b3806a7e0 0x7f6b380c5be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:34.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.584+0000 7f6b3c942700 1 -- 172.21.15.16:0/3202760757 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6b380cc8a0 msgr2=0x7f6b381589d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:34.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.584+0000 7f6b3c942700 1 --2- 172.21.15.16:0/3202760757 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6b380cc8a0 0x7f6b381589d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:34.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.584+0000 7f6b3c942700 1 -- 172.21.15.16:0/3202760757 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b20009550 con 0x7f6b3815b280 2023-10-16T12:27:34.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.584+0000 7f6b3c942700 1 --2- 172.21.15.16:0/3202760757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b3815b280 0x7f6b3815f480 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f6b3000cc50 tx=0x7f6b3000a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:34.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.584+0000 7f6b34ff9700 1 -- 172.21.15.16:0/3202760757 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b3000ad80 con 0x7f6b3815b280 2023-10-16T12:27:34.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.584+0000 7f6b34ff9700 1 -- 172.21.15.16:0/3202760757 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b3000ce50 con 0x7f6b3815b280 2023-10-16T12:27:34.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.584+0000 7f6b34ff9700 1 -- 172.21.15.16:0/3202760757 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b30003c40 con 0x7f6b3815b280 2023-10-16T12:27:34.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.584+0000 7f6b3e3a5700 1 -- 172.21.15.16:0/3202760757 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b380c4b40 con 0x7f6b3815b280 2023-10-16T12:27:34.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.584+0000 7f6b3e3a5700 1 -- 172.21.15.16:0/3202760757 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b380c5100 con 0x7f6b3815b280 2023-10-16T12:27:34.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.585+0000 7f6b3e3a5700 1 -- 172.21.15.16:0/3202760757 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b380c58c0 con 0x7f6b3815b280 2023-10-16T12:27:34.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.586+0000 7f6b34ff9700 1 -- 172.21.15.16:0/3202760757 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 51) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6b300050d0 con 0x7f6b3815b280 2023-10-16T12:27:34.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.586+0000 7f6b34ff9700 1 --2- 172.21.15.16:0/3202760757 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f6b1806b020 0x7f6b1806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:34.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.586+0000 7f6b34ff9700 1 -- 172.21.15.16:0/3202760757 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f6b30089680 con 0x7f6b3815b280 2023-10-16T12:27:34.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.586+0000 7f6b37fff700 1 --2- 172.21.15.16:0/3202760757 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f6b1806b020 0x7f6b1806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:34.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.586+0000 7f6b37fff700 1 --2- 172.21.15.16:0/3202760757 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f6b1806b020 0x7f6b1806d4e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6b28009770 tx=0x7f6b28008040 comp rx=0 tx=0).ready entity=mgr.6365 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:34.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.590+0000 7f6b34ff9700 1 -- 172.21.15.16:0/3202760757 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6b30054810 con 0x7f6b3815b280 2023-10-16T12:27:34.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.794+0000 7f6b3e3a5700 1 -- 172.21.15.16:0/3202760757 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6b3804ebb0 con 0x7f6b3815b280 2023-10-16T12:27:34.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.795+0000 7f6b34ff9700 1 -- 172.21.15.16:0/3202760757 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v108) v1 ==== 74+0+12999 (secure 0 0 0) 0x7f6b30057e30 con 0x7f6b3815b280 2023-10-16T12:27:34.798 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:27:34.798 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":108,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:27:33.971459+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:27:34.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.797+0000 7f6b3e3a5700 1 -- 172.21.15.16:0/3202760757 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f6b1806b020 msgr2=0x7f6b1806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:34.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.797+0000 7f6b3e3a5700 1 --2- 172.21.15.16:0/3202760757 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f6b1806b020 0x7f6b1806d4e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6b28009770 tx=0x7f6b28008040 comp rx=0 tx=0).stop 2023-10-16T12:27:34.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.797+0000 7f6b3e3a5700 1 -- 172.21.15.16:0/3202760757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b3815b280 msgr2=0x7f6b3815f480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:34.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.797+0000 7f6b3e3a5700 1 --2- 172.21.15.16:0/3202760757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b3815b280 0x7f6b3815f480 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f6b3000cc50 tx=0x7f6b3000a340 comp rx=0 tx=0).stop 2023-10-16T12:27:34.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.797+0000 7f6b3e3a5700 1 -- 172.21.15.16:0/3202760757 shutdown_connections 2023-10-16T12:27:34.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.797+0000 7f6b3e3a5700 1 --2- 172.21.15.16:0/3202760757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b3815b280 0x7f6b3815f480 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:34.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.797+0000 7f6b3e3a5700 1 --2- 172.21.15.16:0/3202760757 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6b3806a7e0 0x7f6b380c5be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:34.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.797+0000 7f6b3e3a5700 1 --2- 172.21.15.16:0/3202760757 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f6b1806b020 0x7f6b1806d4e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:34.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.797+0000 7f6b3e3a5700 1 --2- 172.21.15.16:0/3202760757 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6b380cc8a0 0x7f6b381589d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:34.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.797+0000 7f6b3e3a5700 1 -- 172.21.15.16:0/3202760757 >> 172.21.15.16:0/3202760757 conn(0x7f6b380b1fd0 msgr2=0x7f6b380cd850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:34.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.797+0000 7f6b3e3a5700 1 -- 172.21.15.16:0/3202760757 shutdown_connections 2023-10-16T12:27:34.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.797+0000 7f6b3e3a5700 1 -- 172.21.15.16:0/3202760757 wait complete. 2023-10-16T12:27:34.812 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T12:27:34.813 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:27:34.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.937+0000 7f0fba2f0700 1 Processor -- start 2023-10-16T12:27:34.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.937+0000 7f0fba2f0700 1 -- start start 2023-10-16T12:27:34.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.938+0000 7f0fba2f0700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f0fb415bb40 0x7f0fb4071a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:34.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.938+0000 7f0fba2f0700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0fb40caa40 0x7f0fb40cae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:34.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.938+0000 7f0fba2f0700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fb40cb3d0 con 0x7f0fb4072b30 2023-10-16T12:27:34.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.938+0000 7f0fba2f0700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fb406ea20 con 0x7f0fb415bb40 2023-10-16T12:27:34.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.938+0000 7f0fba2f0700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb415c040 con 0x7f0fb40caa40 2023-10-16T12:27:34.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.938+0000 7f0fb37fe700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f0fb415bb40 0x7f0fb4071a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:34.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.938+0000 7f0fb888d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0fb40caa40 0x7f0fb40cae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:34.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.938+0000 7f0fb37fe700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f0fb415bb40 0x7f0fb4071a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:41244/0 (socket says 172.21.15.16:41244) 2023-10-16T12:27:34.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.938+0000 7f0fb888d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0fb40caa40 0x7f0fb40cae90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:58676/0 (socket says 172.21.15.16:58676) 2023-10-16T12:27:34.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.938+0000 7f0fb37fe700 1 -- 172.21.15.16:0/3955096377 learned_addr learned my addr 172.21.15.16:0/3955096377 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:34.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.939+0000 7f0fb888d700 1 -- 172.21.15.16:0/3955096377 >> v2:172.21.15.49:3300/0 conn(0x7f0fb415bb40 msgr2=0x7f0fb4071a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:34.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.939+0000 7f0fb888d700 1 --2- 172.21.15.16:0/3955096377 >> v2:172.21.15.49:3300/0 conn(0x7f0fb415bb40 0x7f0fb4071a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:34.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.939+0000 7f0fb888d700 1 -- 172.21.15.16:0/3955096377 >> v1:172.21.15.49:6789/0 conn(0x7f0fb4072b30 legacy=0x7f0fb406e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:34.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.939+0000 7f0fb888d700 1 -- 172.21.15.16:0/3955096377 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fb415c320 con 0x7f0fb40caa40 2023-10-16T12:27:34.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.939+0000 7f0fb37fe700 1 --2- 172.21.15.16:0/3955096377 >> v2:172.21.15.49:3300/0 conn(0x7f0fb415bb40 0x7f0fb4071a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:27:34.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.939+0000 7f0fb888d700 1 --2- 172.21.15.16:0/3955096377 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0fb40caa40 0x7f0fb40cae90 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f0fac009d90 tx=0x7f0fac00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b683cc6b33b6e1fb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:34.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.939+0000 7f0fb2ffd700 1 -- 172.21.15.16:0/3955096377 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fac00e070 con 0x7f0fb40caa40 2023-10-16T12:27:34.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.939+0000 7f0fb2ffd700 1 -- 172.21.15.16:0/3955096377 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fac00bb20 con 0x7f0fb40caa40 2023-10-16T12:27:34.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.939+0000 7f0fb2ffd700 1 -- 172.21.15.16:0/3955096377 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fac012430 con 0x7f0fb40caa40 2023-10-16T12:27:34.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.940+0000 7f0fba2f0700 1 -- 172.21.15.16:0/3955096377 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0fb40caa40 msgr2=0x7f0fb40cae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:34.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.940+0000 7f0fba2f0700 1 --2- 172.21.15.16:0/3955096377 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0fb40caa40 0x7f0fb40cae90 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f0fac009d90 tx=0x7f0fac00b330 comp rx=0 tx=0).stop 2023-10-16T12:27:34.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.940+0000 7f0fba2f0700 1 -- 172.21.15.16:0/3955096377 shutdown_connections 2023-10-16T12:27:34.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.940+0000 7f0fba2f0700 1 --2- 172.21.15.16:0/3955096377 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0fb40caa40 0x7f0fb40cae90 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:34.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.940+0000 7f0fba2f0700 1 --2- 172.21.15.16:0/3955096377 >> v2:172.21.15.49:3300/0 conn(0x7f0fb415bb40 0x7f0fb4071a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:34.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.940+0000 7f0fba2f0700 1 -- 172.21.15.16:0/3955096377 >> 172.21.15.16:0/3955096377 conn(0x7f0fb40b1f40 msgr2=0x7f0fb40b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:34.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.940+0000 7f0fba2f0700 1 -- 172.21.15.16:0/3955096377 shutdown_connections 2023-10-16T12:27:34.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.940+0000 7f0fba2f0700 1 -- 172.21.15.16:0/3955096377 wait complete. 2023-10-16T12:27:34.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.941+0000 7f0fba2f0700 1 Processor -- start 2023-10-16T12:27:34.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.941+0000 7f0fba2f0700 1 -- start start 2023-10-16T12:27:34.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.942+0000 7f0fba2f0700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0fb4072b30 0x7f0fb4155240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:34.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.942+0000 7f0fba2f0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fb40caa40 0x7f0fb40ca130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:34.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.942+0000 7f0fba2f0700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0fb415bb40 0x7f0fb40b5b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:34.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.942+0000 7f0fba2f0700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb4155780 con 0x7f0fb40caa40 2023-10-16T12:27:34.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.942+0000 7f0fba2f0700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb40b4af0 con 0x7f0fb4072b30 2023-10-16T12:27:34.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.942+0000 7f0fba2f0700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb40b4dd0 con 0x7f0fb415bb40 2023-10-16T12:27:34.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.942+0000 7f0fb37fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fb40caa40 0x7f0fb40ca130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:34.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.942+0000 7f0fb37fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fb40caa40 0x7f0fb40ca130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42188/0 (socket says 172.21.15.16:42188) 2023-10-16T12:27:34.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.942+0000 7f0fb37fe700 1 -- 172.21.15.16:0/2551703935 learned_addr learned my addr 172.21.15.16:0/2551703935 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:34.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.942+0000 7f0fb888d700 1 --2- 172.21.15.16:0/2551703935 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0fb415bb40 0x7f0fb40b5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:34.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.942+0000 7f0fb3fff700 1 --2- 172.21.15.16:0/2551703935 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0fb4072b30 0x7f0fb4155240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:34.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.943+0000 7f0fb37fe700 1 -- 172.21.15.16:0/2551703935 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0fb415bb40 msgr2=0x7f0fb40b5b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:34.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.943+0000 7f0fb37fe700 1 --2- 172.21.15.16:0/2551703935 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0fb415bb40 0x7f0fb40b5b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:34.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.943+0000 7f0fb37fe700 1 -- 172.21.15.16:0/2551703935 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0fb4072b30 msgr2=0x7f0fb4155240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:34.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.943+0000 7f0fb37fe700 1 --2- 172.21.15.16:0/2551703935 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0fb4072b30 0x7f0fb4155240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:34.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.943+0000 7f0fb37fe700 1 -- 172.21.15.16:0/2551703935 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fa4009550 con 0x7f0fb40caa40 2023-10-16T12:27:34.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.943+0000 7f0fb37fe700 1 --2- 172.21.15.16:0/2551703935 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fb40caa40 0x7f0fb40ca130 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f0fa400ebb0 tx=0x7f0fa400eec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:34.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.943+0000 7f0fb0ff9700 1 -- 172.21.15.16:0/2551703935 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fa400c850 con 0x7f0fb40caa40 2023-10-16T12:27:34.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.943+0000 7f0fb0ff9700 1 -- 172.21.15.16:0/2551703935 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fa400c9b0 con 0x7f0fb40caa40 2023-10-16T12:27:34.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.943+0000 7f0fba2f0700 1 -- 172.21.15.16:0/2551703935 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fac009a70 con 0x7f0fb40caa40 2023-10-16T12:27:34.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.943+0000 7f0fb0ff9700 1 -- 172.21.15.16:0/2551703935 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fa400cd40 con 0x7f0fb40caa40 2023-10-16T12:27:34.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.943+0000 7f0fba2f0700 1 -- 172.21.15.16:0/2551703935 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fb40b5390 con 0x7f0fb40caa40 2023-10-16T12:27:34.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.944+0000 7f0fba2f0700 1 -- 172.21.15.16:0/2551703935 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fb404ebb0 con 0x7f0fb40caa40 2023-10-16T12:27:34.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.944+0000 7f0fb0ff9700 1 -- 172.21.15.16:0/2551703935 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 51) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0fa4013470 con 0x7f0fb40caa40 2023-10-16T12:27:34.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.945+0000 7f0fb0ff9700 1 --2- 172.21.15.16:0/2551703935 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f0f9406b020 0x7f0f9406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:34.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.945+0000 7f0fb0ff9700 1 -- 172.21.15.16:0/2551703935 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f0fa4089680 con 0x7f0fb40caa40 2023-10-16T12:27:34.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.945+0000 7f0fb3fff700 1 --2- 172.21.15.16:0/2551703935 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f0f9406b020 0x7f0f9406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:34.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.945+0000 7f0fb3fff700 1 --2- 172.21.15.16:0/2551703935 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f0f9406b020 0x7f0f9406d4e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0fb40b4060 tx=0x7f0f9c009040 comp rx=0 tx=0).ready entity=mgr.6365 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:34.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:34.950+0000 7f0fb0ff9700 1 -- 172.21.15.16:0/2551703935 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0fa4054810 con 0x7f0fb40caa40 2023-10-16T12:27:35.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.155+0000 7f0fba2f0700 1 -- 172.21.15.16:0/2551703935 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0fb415c8b0 con 0x7f0fb40caa40 2023-10-16T12:27:35.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.155+0000 7f0fb0ff9700 1 -- 172.21.15.16:0/2551703935 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v108) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0fa4057e30 con 0x7f0fb40caa40 2023-10-16T12:27:35.159 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T12:27:35.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.157+0000 7f0fba2f0700 1 -- 172.21.15.16:0/2551703935 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f0f9406b020 msgr2=0x7f0f9406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:35.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.157+0000 7f0fba2f0700 1 --2- 172.21.15.16:0/2551703935 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f0f9406b020 0x7f0f9406d4e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0fb40b4060 tx=0x7f0f9c009040 comp rx=0 tx=0).stop 2023-10-16T12:27:35.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.157+0000 7f0fba2f0700 1 -- 172.21.15.16:0/2551703935 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fb40caa40 msgr2=0x7f0fb40ca130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:35.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.157+0000 7f0fba2f0700 1 --2- 172.21.15.16:0/2551703935 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fb40caa40 0x7f0fb40ca130 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f0fa400ebb0 tx=0x7f0fa400eec0 comp rx=0 tx=0).stop 2023-10-16T12:27:35.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.157+0000 7f0fba2f0700 1 -- 172.21.15.16:0/2551703935 shutdown_connections 2023-10-16T12:27:35.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.157+0000 7f0fba2f0700 1 --2- 172.21.15.16:0/2551703935 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0fb415bb40 0x7f0fb40b5b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:35.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.157+0000 7f0fba2f0700 1 --2- 172.21.15.16:0/2551703935 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f0f9406b020 0x7f0f9406d4e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:35.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.157+0000 7f0fba2f0700 1 --2- 172.21.15.16:0/2551703935 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0fb4072b30 0x7f0fb4155240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:35.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.157+0000 7f0fba2f0700 1 --2- 172.21.15.16:0/2551703935 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fb40caa40 0x7f0fb40ca130 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:35.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.157+0000 7f0fba2f0700 1 -- 172.21.15.16:0/2551703935 >> 172.21.15.16:0/2551703935 conn(0x7f0fb40b1f40 msgr2=0x7f0fb41596b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:35.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.157+0000 7f0fba2f0700 1 -- 172.21.15.16:0/2551703935 shutdown_connections 2023-10-16T12:27:35.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.157+0000 7f0fba2f0700 1 -- 172.21.15.16:0/2551703935 wait complete. 2023-10-16T12:27:35.173 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-16T12:27:35.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.295+0000 7fc5624c0700 1 Processor -- start 2023-10-16T12:27:35.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.295+0000 7fc5624c0700 1 -- start start 2023-10-16T12:27:35.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.295+0000 7fc5624c0700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fc55c15a5c0 0x7fc55c0cef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:35.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.295+0000 7fc5624c0700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc55c0d0ae0 0x7fc55c0cc1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:35.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.295+0000 7fc5624c0700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc55c072a50 con 0x7fc55c0ce010 2023-10-16T12:27:35.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.295+0000 7fc5624c0700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc55c0cf440 con 0x7fc55c0d0ae0 2023-10-16T12:27:35.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.295+0000 7fc5624c0700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc55c0cba30 con 0x7fc55c15a5c0 2023-10-16T12:27:35.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.296+0000 7fc560a5d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc55c0d0ae0 0x7fc55c0cc1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:35.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.296+0000 7fc560a5d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc55c0d0ae0 0x7fc55c0cc1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42200/0 (socket says 172.21.15.16:42200) 2023-10-16T12:27:35.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.296+0000 7fc560a5d700 1 -- 172.21.15.16:0/2570449268 learned_addr learned my addr 172.21.15.16:0/2570449268 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:35.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.296+0000 7fc55bfff700 1 --2- 172.21.15.16:0/2570449268 >> v2:172.21.15.49:3300/0 conn(0x7fc55c15a5c0 0x7fc55c0cef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:35.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.296+0000 7fc560a5d700 1 -- 172.21.15.16:0/2570449268 >> v2:172.21.15.49:3300/0 conn(0x7fc55c15a5c0 msgr2=0x7fc55c0cef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:35.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.296+0000 7fc560a5d700 1 --2- 172.21.15.16:0/2570449268 >> v2:172.21.15.49:3300/0 conn(0x7fc55c15a5c0 0x7fc55c0cef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:35.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.296+0000 7fc560a5d700 1 -- 172.21.15.16:0/2570449268 >> v1:172.21.15.49:6789/0 conn(0x7fc55c0ce010 legacy=0x7fc55c0d03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:35.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.296+0000 7fc560a5d700 1 -- 172.21.15.16:0/2570449268 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc55c0cf630 con 0x7fc55c0d0ae0 2023-10-16T12:27:35.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.296+0000 7fc55bfff700 1 --2- 172.21.15.16:0/2570449268 >> v2:172.21.15.49:3300/0 conn(0x7fc55c15a5c0 0x7fc55c0cef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:27:35.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.296+0000 7fc560a5d700 1 --2- 172.21.15.16:0/2570449268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc55c0d0ae0 0x7fc55c0cc1f0 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7fc55c0b2ce0 tx=0x7fc55400d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=806f05926ff3a17c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:35.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.296+0000 7fc55affd700 1 -- 172.21.15.16:0/2570449268 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc554010070 con 0x7fc55c0d0ae0 2023-10-16T12:27:35.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.296+0000 7fc55affd700 1 -- 172.21.15.16:0/2570449268 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc55400dd50 con 0x7fc55c0d0ae0 2023-10-16T12:27:35.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.296+0000 7fc55affd700 1 -- 172.21.15.16:0/2570449268 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc55400b310 con 0x7fc55c0d0ae0 2023-10-16T12:27:35.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.297+0000 7fc5624c0700 1 -- 172.21.15.16:0/2570449268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc55c0d0ae0 msgr2=0x7fc55c0cc1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:35.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.297+0000 7fc5624c0700 1 --2- 172.21.15.16:0/2570449268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc55c0d0ae0 0x7fc55c0cc1f0 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7fc55c0b2ce0 tx=0x7fc55400d430 comp rx=0 tx=0).stop 2023-10-16T12:27:35.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.297+0000 7fc5624c0700 1 -- 172.21.15.16:0/2570449268 shutdown_connections 2023-10-16T12:27:35.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.297+0000 7fc5624c0700 1 --2- 172.21.15.16:0/2570449268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc55c0d0ae0 0x7fc55c0cc1f0 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:35.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.297+0000 7fc5624c0700 1 --2- 172.21.15.16:0/2570449268 >> v2:172.21.15.49:3300/0 conn(0x7fc55c15a5c0 0x7fc55c0cef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:35.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.297+0000 7fc5624c0700 1 -- 172.21.15.16:0/2570449268 >> 172.21.15.16:0/2570449268 conn(0x7fc55c0b1fd0 msgr2=0x7fc55c0b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:35.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.297+0000 7fc5624c0700 1 -- 172.21.15.16:0/2570449268 shutdown_connections 2023-10-16T12:27:35.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.297+0000 7fc5624c0700 1 -- 172.21.15.16:0/2570449268 wait complete. 2023-10-16T12:27:35.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.298+0000 7fc5624c0700 1 Processor -- start 2023-10-16T12:27:35.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.298+0000 7fc5624c0700 1 -- start start 2023-10-16T12:27:35.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.299+0000 7fc5624c0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc55c0ce010 0x7fc55c15d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:35.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.299+0000 7fc5624c0700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc55c0d0ae0 0x7fc55c0c7cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:35.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.299+0000 7fc5624c0700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc55c15a5c0 0x7fc55c0b30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:35.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.299+0000 7fc5624c0700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc55c0c7510 con 0x7fc55c0ce010 2023-10-16T12:27:35.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.299+0000 7fc5624c0700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc55c1f37e0 con 0x7fc55c15a5c0 2023-10-16T12:27:35.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.299+0000 7fc5624c0700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc55c1f3ac0 con 0x7fc55c0d0ae0 2023-10-16T12:27:35.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.299+0000 7fc55bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc55c0ce010 0x7fc55c15d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:35.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.299+0000 7fc55bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc55c0ce010 0x7fc55c15d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42202/0 (socket says 172.21.15.16:42202) 2023-10-16T12:27:35.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.299+0000 7fc55bfff700 1 -- 172.21.15.16:0/3768088213 learned_addr learned my addr 172.21.15.16:0/3768088213 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:35.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.299+0000 7fc560a5d700 1 --2- 172.21.15.16:0/3768088213 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc55c15a5c0 0x7fc55c0b30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:35.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.299+0000 7fc55b7fe700 1 --2- 172.21.15.16:0/3768088213 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc55c0d0ae0 0x7fc55c0c7cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:35.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.299+0000 7fc55bfff700 1 -- 172.21.15.16:0/3768088213 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc55c0d0ae0 msgr2=0x7fc55c0c7cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:35.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.299+0000 7fc55bfff700 1 --2- 172.21.15.16:0/3768088213 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc55c0d0ae0 0x7fc55c0c7cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:35.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.299+0000 7fc55bfff700 1 -- 172.21.15.16:0/3768088213 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc55c15a5c0 msgr2=0x7fc55c0b30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:35.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.299+0000 7fc55bfff700 1 --2- 172.21.15.16:0/3768088213 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc55c15a5c0 0x7fc55c0b30c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:35.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.299+0000 7fc55bfff700 1 -- 172.21.15.16:0/3768088213 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc55400bd40 con 0x7fc55c0ce010 2023-10-16T12:27:35.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.300+0000 7fc55bfff700 1 --2- 172.21.15.16:0/3768088213 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc55c0ce010 0x7fc55c15d450 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7fc54c00c7f0 tx=0x7fc54c00cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:35.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.300+0000 7fc558ff9700 1 -- 172.21.15.16:0/3768088213 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc54c011800 con 0x7fc55c0ce010 2023-10-16T12:27:35.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.300+0000 7fc558ff9700 1 -- 172.21.15.16:0/3768088213 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc54c011960 con 0x7fc55c0ce010 2023-10-16T12:27:35.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.300+0000 7fc5624c0700 1 -- 172.21.15.16:0/3768088213 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc55c1f3da0 con 0x7fc55c0ce010 2023-10-16T12:27:35.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.300+0000 7fc558ff9700 1 -- 172.21.15.16:0/3768088213 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc54c011cf0 con 0x7fc55c0ce010 2023-10-16T12:27:35.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.300+0000 7fc5624c0700 1 -- 172.21.15.16:0/3768088213 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc55c0c8230 con 0x7fc55c0ce010 2023-10-16T12:27:35.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.301+0000 7fc5624c0700 1 -- 172.21.15.16:0/3768088213 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc55c0c7a10 con 0x7fc55c0ce010 2023-10-16T12:27:35.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.301+0000 7fc558ff9700 1 -- 172.21.15.16:0/3768088213 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 51) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc54c018360 con 0x7fc55c0ce010 2023-10-16T12:27:35.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.302+0000 7fc558ff9700 1 --2- 172.21.15.16:0/3768088213 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fc54806b020 0x7fc54806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:35.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.302+0000 7fc558ff9700 1 -- 172.21.15.16:0/3768088213 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5205+0+0 (secure 0 0 0) 0x7fc54c0894e0 con 0x7fc55c0ce010 2023-10-16T12:27:35.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.302+0000 7fc55b7fe700 1 --2- 172.21.15.16:0/3768088213 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fc54806b020 0x7fc54806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:35.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.302+0000 7fc55b7fe700 1 --2- 172.21.15.16:0/3768088213 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fc54806b020 0x7fc54806d4e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc55c06b6e0 tx=0x7fc550007ec0 comp rx=0 tx=0).ready entity=mgr.6365 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:35.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.307+0000 7fc558ff9700 1 -- 172.21.15.16:0/3768088213 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc55c0c7a10 con 0x7fc55c0ce010 2023-10-16T12:27:35.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.512+0000 7fc5624c0700 1 -- 172.21.15.16:0/3768088213 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fc55c0c7a10 con 0x7fc55c0ce010 2023-10-16T12:27:35.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.513+0000 7fc558ff9700 1 -- 172.21.15.16:0/3768088213 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v108) v1 ==== 84+0+65 (secure 0 0 0) 0x7fc55c0c7a10 con 0x7fc55c0ce010 2023-10-16T12:27:35.516 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:27:35.516 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2023-10-16T12:27:35.517 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:27:35.517 INFO:teuthology.orchestra.run.smithi016.stdout: "poolnum": 1, 2023-10-16T12:27:35.517 INFO:teuthology.orchestra.run.smithi016.stdout: "poolname": ".mgr" 2023-10-16T12:27:35.517 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:27:35.517 INFO:teuthology.orchestra.run.smithi016.stdout:] 2023-10-16T12:27:35.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.515+0000 7fc5624c0700 1 -- 172.21.15.16:0/3768088213 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fc54806b020 msgr2=0x7fc54806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:35.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.515+0000 7fc5624c0700 1 --2- 172.21.15.16:0/3768088213 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fc54806b020 0x7fc54806d4e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc55c06b6e0 tx=0x7fc550007ec0 comp rx=0 tx=0).stop 2023-10-16T12:27:35.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.515+0000 7fc5624c0700 1 -- 172.21.15.16:0/3768088213 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc55c0ce010 msgr2=0x7fc55c15d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:35.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.515+0000 7fc5624c0700 1 --2- 172.21.15.16:0/3768088213 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc55c0ce010 0x7fc55c15d450 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7fc54c00c7f0 tx=0x7fc54c00cb00 comp rx=0 tx=0).stop 2023-10-16T12:27:35.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.515+0000 7fc5624c0700 1 -- 172.21.15.16:0/3768088213 shutdown_connections 2023-10-16T12:27:35.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.515+0000 7fc5624c0700 1 --2- 172.21.15.16:0/3768088213 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fc54806b020 0x7fc54806d4e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:35.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.515+0000 7fc5624c0700 1 --2- 172.21.15.16:0/3768088213 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc55c15a5c0 0x7fc55c0b30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:35.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.515+0000 7fc5624c0700 1 --2- 172.21.15.16:0/3768088213 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc55c0ce010 0x7fc55c15d450 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:35.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.515+0000 7fc5624c0700 1 --2- 172.21.15.16:0/3768088213 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc55c0d0ae0 0x7fc55c0c7cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:35.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.515+0000 7fc5624c0700 1 -- 172.21.15.16:0/3768088213 >> 172.21.15.16:0/3768088213 conn(0x7fc55c0b1fd0 msgr2=0x7fc55c158ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:35.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.515+0000 7fc5624c0700 1 -- 172.21.15.16:0/3768088213 shutdown_connections 2023-10-16T12:27:35.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.515+0000 7fc5624c0700 1 -- 172.21.15.16:0/3768088213 wait complete. 2023-10-16T12:27:35.530 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-16T12:27:35.531 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-16T12:27:35.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.657+0000 7f617afd3700 1 Processor -- start 2023-10-16T12:27:35.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.657+0000 7f617afd3700 1 -- start start 2023-10-16T12:27:35.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.658+0000 7f617afd3700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f61740cf640 0x7f617415eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:35.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.658+0000 7f617afd3700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6174162260 con 0x7f6174160ef0 2023-10-16T12:27:35.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.658+0000 7f617afd3700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61741616e0 con 0x7f61740cf260 2023-10-16T12:27:35.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.658+0000 7f617afd3700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61741619c0 con 0x7f61740cf640 2023-10-16T12:27:35.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.658+0000 7f6173fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f6174160ef0 0x7f617415e7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59716/0 (socket says 172.21.15.16:59716) 2023-10-16T12:27:35.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.658+0000 7f6173fff700 1 -- 172.21.15.16:0/285080273 learned_addr learned my addr 172.21.15.16:0/285080273 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:27:35.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.659+0000 7f6179570700 1 --2- 172.21.15.16:0/285080273 >> v2:172.21.15.49:3300/0 conn(0x7f61740cf640 0x7f617415eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:35.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.659+0000 7f61737fe700 1 -- 172.21.15.16:0/285080273 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2678648465 0 0) 0x7f6174162260 con 0x7f6174160ef0 2023-10-16T12:27:35.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.659+0000 7f61737fe700 1 -- 172.21.15.16:0/285080273 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61640035b0 con 0x7f6174160ef0 2023-10-16T12:27:35.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.659+0000 7f61737fe700 1 -- 172.21.15.16:0/285080273 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1461685902 0 0) 0x7f61640035b0 con 0x7f6174160ef0 2023-10-16T12:27:35.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.659+0000 7f6179570700 1 -- 172.21.15.16:0/285080273 >> v1:172.21.15.49:6789/0 conn(0x7f61740cf260 legacy=0x7f61741607e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:35.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.659+0000 7f6179570700 1 -- 172.21.15.16:0/285080273 >> v1:172.21.15.16:6789/0 conn(0x7f6174160ef0 legacy=0x7f617415e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:35.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.659+0000 7f6179570700 1 -- 172.21.15.16:0/285080273 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6174161ca0 con 0x7f61740cf640 2023-10-16T12:27:35.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.659+0000 7f6179570700 1 --2- 172.21.15.16:0/285080273 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f61740cf640 0x7f617415eef0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f616c009d90 tx=0x7f616c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f7796c74ab9d08ce server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:35.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.659+0000 7f61737fe700 1 -- 172.21.15.16:0/285080273 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f616c00e070 con 0x7f61740cf640 2023-10-16T12:27:35.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.659+0000 7f61737fe700 1 -- 172.21.15.16:0/285080273 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f616c003a20 con 0x7f61740cf640 2023-10-16T12:27:35.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.659+0000 7f61737fe700 1 -- 172.21.15.16:0/285080273 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6160001670 con 0x7f61740cf640 2023-10-16T12:27:35.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.660+0000 7f617afd3700 1 -- 172.21.15.16:0/285080273 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f61740cf640 msgr2=0x7f617415eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:35.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.660+0000 7f617afd3700 1 --2- 172.21.15.16:0/285080273 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f61740cf640 0x7f617415eef0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f616c009d90 tx=0x7f616c00b330 comp rx=0 tx=0).stop 2023-10-16T12:27:35.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.660+0000 7f617afd3700 1 -- 172.21.15.16:0/285080273 shutdown_connections 2023-10-16T12:27:35.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.660+0000 7f617afd3700 1 --2- 172.21.15.16:0/285080273 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f61740cf640 0x7f617415eef0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:35.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.660+0000 7f617afd3700 1 -- 172.21.15.16:0/285080273 >> 172.21.15.16:0/285080273 conn(0x7f617406ec70 msgr2=0x7f617406f050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:35.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.660+0000 7f617afd3700 1 -- 172.21.15.16:0/285080273 shutdown_connections 2023-10-16T12:27:35.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.660+0000 7f617afd3700 1 -- 172.21.15.16:0/285080273 wait complete. 2023-10-16T12:27:35.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.661+0000 7f617afd3700 1 Processor -- start 2023-10-16T12:27:35.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.661+0000 7f617afd3700 1 -- start start 2023-10-16T12:27:35.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.662+0000 7f617afd3700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f61740cf260 0x7f61740caeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:35.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.662+0000 7f617afd3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61740cf640 0x7f61740c7570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:35.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.662+0000 7f617afd3700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6174160ef0 0x7f61740b5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:35.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.662+0000 7f617afd3700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61741561b0 con 0x7f61740cf640 2023-10-16T12:27:35.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.662+0000 7f617afd3700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61740c7ab0 con 0x7f61740cf260 2023-10-16T12:27:35.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.662+0000 7f617afd3700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6174156ae0 con 0x7f6174160ef0 2023-10-16T12:27:35.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.662+0000 7f6173fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61740cf640 0x7f61740c7570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:35.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.662+0000 7f6173fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61740cf640 0x7f61740c7570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42216/0 (socket says 172.21.15.16:42216) 2023-10-16T12:27:35.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.662+0000 7f6173fff700 1 -- 172.21.15.16:0/2049021046 learned_addr learned my addr 172.21.15.16:0/2049021046 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:35.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.662+0000 7f6178d6f700 1 --2- 172.21.15.16:0/2049021046 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f61740cf260 0x7f61740caeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:35.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.662+0000 7f6179570700 1 --2- 172.21.15.16:0/2049021046 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6174160ef0 0x7f61740b5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:35.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.662+0000 7f6173fff700 1 -- 172.21.15.16:0/2049021046 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6174160ef0 msgr2=0x7f61740b5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:35.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.662+0000 7f6173fff700 1 --2- 172.21.15.16:0/2049021046 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6174160ef0 0x7f61740b5ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:35.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.662+0000 7f6173fff700 1 -- 172.21.15.16:0/2049021046 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f61740cf260 msgr2=0x7f61740caeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:35.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.662+0000 7f6173fff700 1 --2- 172.21.15.16:0/2049021046 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f61740cf260 0x7f61740caeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:35.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.662+0000 7f6173fff700 1 -- 172.21.15.16:0/2049021046 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f616c009a70 con 0x7f61740cf640 2023-10-16T12:27:35.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.663+0000 7f6173fff700 1 --2- 172.21.15.16:0/2049021046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61740cf640 0x7f61740c7570 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f616000c750 tx=0x7f616000ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:35.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.663+0000 7f61717fa700 1 -- 172.21.15.16:0/2049021046 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6160004c80 con 0x7f61740cf640 2023-10-16T12:27:35.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.663+0000 7f61717fa700 1 -- 172.21.15.16:0/2049021046 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6160004de0 con 0x7f61740cf640 2023-10-16T12:27:35.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.663+0000 7f617afd3700 1 -- 172.21.15.16:0/2049021046 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6174156dc0 con 0x7f61740cf640 2023-10-16T12:27:35.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.663+0000 7f61717fa700 1 -- 172.21.15.16:0/2049021046 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61600072f0 con 0x7f61740cf640 2023-10-16T12:27:35.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.663+0000 7f617afd3700 1 -- 172.21.15.16:0/2049021046 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6174157380 con 0x7f61740cf640 2023-10-16T12:27:35.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.664+0000 7f617afd3700 1 -- 172.21.15.16:0/2049021046 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f617404ec20 con 0x7f61740cf640 2023-10-16T12:27:35.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.664+0000 7f61717fa700 1 -- 172.21.15.16:0/2049021046 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 51) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6160007450 con 0x7f61740cf640 2023-10-16T12:27:35.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.665+0000 7f61717fa700 1 --2- 172.21.15.16:0/2049021046 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f615806b020 0x7f615806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:35.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.665+0000 7f61717fa700 1 -- 172.21.15.16:0/2049021046 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f61600892b0 con 0x7f61740cf640 2023-10-16T12:27:35.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.665+0000 7f6178d6f700 1 --2- 172.21.15.16:0/2049021046 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f615806b020 0x7f615806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:35.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.665+0000 7f6178d6f700 1 --2- 172.21.15.16:0/2049021046 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f615806b020 0x7f615806d4e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f6168007fd0 tx=0x7f6168009040 comp rx=0 tx=0).ready entity=mgr.6365 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:35.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.669+0000 7f61717fa700 1 -- 172.21.15.16:0/2049021046 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f61600543a0 con 0x7f61740cf640 2023-10-16T12:27:35.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.874+0000 7f617afd3700 1 -- 172.21.15.16:0/2049021046 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f617415b150 con 0x7f61740cf640 2023-10-16T12:27:35.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.964+0000 7f61717fa700 1 -- 172.21.15.16:0/2049021046 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v109) v1 ==== 138+0+0 (secure 0 0 0) 0x7f6160001340 con 0x7f61740cf640 2023-10-16T12:27:35.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.967+0000 7f617afd3700 1 -- 172.21.15.16:0/2049021046 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f615806b020 msgr2=0x7f615806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:35.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.967+0000 7f617afd3700 1 --2- 172.21.15.16:0/2049021046 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f615806b020 0x7f615806d4e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f6168007fd0 tx=0x7f6168009040 comp rx=0 tx=0).stop 2023-10-16T12:27:35.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.967+0000 7f617afd3700 1 -- 172.21.15.16:0/2049021046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61740cf640 msgr2=0x7f61740c7570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:35.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.967+0000 7f617afd3700 1 --2- 172.21.15.16:0/2049021046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61740cf640 0x7f61740c7570 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f616000c750 tx=0x7f616000ca60 comp rx=0 tx=0).stop 2023-10-16T12:27:35.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.967+0000 7f617afd3700 1 -- 172.21.15.16:0/2049021046 shutdown_connections 2023-10-16T12:27:35.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.967+0000 7f617afd3700 1 --2- 172.21.15.16:0/2049021046 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6174160ef0 0x7f61740b5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:35.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.967+0000 7f617afd3700 1 --2- 172.21.15.16:0/2049021046 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f615806b020 0x7f615806d4e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:35.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.967+0000 7f617afd3700 1 --2- 172.21.15.16:0/2049021046 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f61740cf260 0x7f61740caeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:35.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.967+0000 7f617afd3700 1 --2- 172.21.15.16:0/2049021046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61740cf640 0x7f61740c7570 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:35.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.967+0000 7f617afd3700 1 -- 172.21.15.16:0/2049021046 >> 172.21.15.16:0/2049021046 conn(0x7f617406ec70 msgr2=0x7f617406f050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:35.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.967+0000 7f617afd3700 1 -- 172.21.15.16:0/2049021046 shutdown_connections 2023-10-16T12:27:35.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:35.967+0000 7f617afd3700 1 -- 172.21.15.16:0/2049021046 wait complete. 2023-10-16T12:27:35.974 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' created 2023-10-16T12:27:35.985 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-16T12:27:36.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:36.116+0000 7fd84d912700 1 Processor -- start 2023-10-16T12:27:36.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:36.116+0000 7fd84d912700 1 -- start start 2023-10-16T12:27:36.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:36.117+0000 7fd84d912700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd84815d8d0 0x7fd8480b2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:36.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:36.117+0000 7fd84d912700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd8480ce350 0x7fd8480ce780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:36.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:36.117+0000 7fd84d912700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8480cecc0 con 0x7fd8480b8fa0 2023-10-16T12:27:36.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:36.117+0000 7fd84d912700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8480d0bb0 con 0x7fd84815d8d0 2023-10-16T12:27:36.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:36.117+0000 7fd84d912700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd84815dcb0 con 0x7fd8480ce350 2023-10-16T12:27:36.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:36.117+0000 7fd846ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd84815d8d0 0x7fd8480b2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:36.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:36.117+0000 7fd846ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd84815d8d0 0x7fd8480b2830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42224/0 (socket says 172.21.15.16:42224) 2023-10-16T12:27:36.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:36.117+0000 7fd846ffd700 1 -- 172.21.15.16:0/387075322 learned_addr learned my addr 172.21.15.16:0/387075322 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:36.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:36.117+0000 7fd846ffd700 1 -- 172.21.15.16:0/387075322 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd8480ce350 msgr2=0x7fd8480ce780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:36.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:36.117+0000 7fd8477fe700 1 --2- 172.21.15.16:0/387075322 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd8480ce350 0x7fd8480ce780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:36.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:36.117+0000 7fd846ffd700 1 --2- 172.21.15.16:0/387075322 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd8480ce350 0x7fd8480ce780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:36.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:36.117+0000 7fd846ffd700 1 -- 172.21.15.16:0/387075322 >> v1:172.21.15.16:6789/0 conn(0x7fd8480b8fa0 legacy=0x7fd8480b6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:36.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:36.117+0000 7fd845ffb700 1 -- 172.21.15.16:0/387075322 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2762123677 0 0) 0x7fd8480cecc0 con 0x7fd8480b8fa0 2023-10-16T12:27:36.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:36.117+0000 7fd846ffd700 1 -- 172.21.15.16:0/387075322 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8480b2f30 con 0x7fd84815d8d0 2023-10-16T12:27:36.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:36.117+0000 7fd8477fe700 1 --2- 172.21.15.16:0/387075322 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd8480ce350 0x7fd8480ce780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:27:36.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:36.117+0000 7fd846ffd700 1 --2- 172.21.15.16:0/387075322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd84815d8d0 0x7fd8480b2830 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7fd8380098f0 tx=0x7fd838009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2de94a912698734e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:36.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:36.118+0000 7fd845ffb700 1 -- 172.21.15.16:0/387075322 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd83800b8a0 con 0x7fd84815d8d0 2023-10-16T12:27:36.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:36.118+0000 7fd845ffb700 1 -- 172.21.15.16:0/387075322 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd83800ba00 con 0x7fd84815d8d0 2023-10-16T12:27:36.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:36.118+0000 7fd845ffb700 1 -- 172.21.15.16:0/387075322 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd83800bd90 con 0x7fd84815d8d0 2023-10-16T12:27:36.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:36.118+0000 7fd84d912700 1 -- 172.21.15.16:0/387075322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd84815d8d0 msgr2=0x7fd8480b2830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:36.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:36.118+0000 7fd84d912700 1 --2- 172.21.15.16:0/387075322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd84815d8d0 0x7fd8480b2830 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7fd8380098f0 tx=0x7fd838009c00 comp rx=0 tx=0).stop 2023-10-16T12:27:36.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:36.118+0000 7fd84d912700 1 -- 172.21.15.16:0/387075322 shutdown_connections 2023-10-16T12:27:36.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:36.118+0000 7fd84d912700 1 --2- 172.21.15.16:0/387075322 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd8480ce350 0x7fd8480ce780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:36.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:36.118+0000 7fd84d912700 1 --2- 172.21.15.16:0/387075322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd84815d8d0 0x7fd8480b2830 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:36.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:36.118+0000 7fd84d912700 1 -- 172.21.15.16:0/387075322 >> 172.21.15.16:0/387075322 conn(0x7fd848072330 msgr2=0x7fd848072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:36.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:36.118+0000 7fd84d912700 1 -- 172.21.15.16:0/387075322 shutdown_connections 2023-10-16T12:27:36.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:36.118+0000 7fd84d912700 1 -- 172.21.15.16:0/387075322 wait complete. 2023-10-16T12:27:36.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:36.119+0000 7fd84d912700 1 Processor -- start 2023-10-16T12:27:36.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:36.120+0000 7fd84d912700 1 -- start start 2023-10-16T12:27:36.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:36.120+0000 7fd84d912700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8480b8fa0 0x7fd848151df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:36.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:36.120+0000 7fd84d912700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd8480ce350 0x7fd8481f4380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:36.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:36.120+0000 7fd84d912700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd84815d8d0 0x7fd8480c96a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:36.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:36.120+0000 7fd84d912700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8480c7040 con 0x7fd8480b8fa0 2023-10-16T12:27:36.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:36.120+0000 7fd84d912700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8480c7320 con 0x7fd8480ce350 2023-10-16T12:27:36.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:36.120+0000 7fd84d912700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8480c7600 con 0x7fd84815d8d0 2023-10-16T12:27:36.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:36.120+0000 7fd846ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8480b8fa0 0x7fd848151df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:36.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:36.120+0000 7fd846ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8480b8fa0 0x7fd848151df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42230/0 (socket says 172.21.15.16:42230) 2023-10-16T12:27:36.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:36.120+0000 7fd846ffd700 1 -- 172.21.15.16:0/3946007672 learned_addr learned my addr 172.21.15.16:0/3946007672 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:36.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:36.121+0000 7fd846ffd700 1 -- 172.21.15.16:0/3946007672 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd84815d8d0 msgr2=0x7fd8480c96a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:36.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:36.121+0000 7fd846ffd700 1 --2- 172.21.15.16:0/3946007672 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd84815d8d0 0x7fd8480c96a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:36.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:36.121+0000 7fd846ffd700 1 -- 172.21.15.16:0/3946007672 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd8480ce350 msgr2=0x7fd8481f4380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:36.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:36.121+0000 7fd846ffd700 1 --2- 172.21.15.16:0/3946007672 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd8480ce350 0x7fd8481f4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:36.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:36.121+0000 7fd846ffd700 1 -- 172.21.15.16:0/3946007672 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd838009550 con 0x7fd8480b8fa0 2023-10-16T12:27:36.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:36.121+0000 7fd846ffd700 1 --2- 172.21.15.16:0/3946007672 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8480b8fa0 0x7fd848151df0 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7fd838000c00 tx=0x7fd838004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:36.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:36.121+0000 7fd83ffff700 1 -- 172.21.15.16:0/3946007672 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8380036a0 con 0x7fd8480b8fa0 2023-10-16T12:27:36.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:36.121+0000 7fd83ffff700 1 -- 172.21.15.16:0/3946007672 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd83801f070 con 0x7fd8480b8fa0 2023-10-16T12:27:36.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:36.121+0000 7fd84d912700 1 -- 172.21.15.16:0/3946007672 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8480c78e0 con 0x7fd8480b8fa0 2023-10-16T12:27:36.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:36.121+0000 7fd83ffff700 1 -- 172.21.15.16:0/3946007672 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd838003a20 con 0x7fd8480b8fa0 2023-10-16T12:27:36.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:36.121+0000 7fd84d912700 1 -- 172.21.15.16:0/3946007672 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8481f48c0 con 0x7fd8480b8fa0 2023-10-16T12:27:36.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:36.122+0000 7fd84d912700 1 -- 172.21.15.16:0/3946007672 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8480c7e00 con 0x7fd8480b8fa0 2023-10-16T12:27:36.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:36.123+0000 7fd83ffff700 1 -- 172.21.15.16:0/3946007672 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 51) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd838003b80 con 0x7fd8480b8fa0 2023-10-16T12:27:36.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:36.124+0000 7fd83ffff700 1 --2- 172.21.15.16:0/3946007672 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fd82806b020 0x7fd82806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:36.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:36.124+0000 7fd83ffff700 1 -- 172.21.15.16:0/3946007672 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5575+0+0 (secure 0 0 0) 0x7fd838021070 con 0x7fd8480b8fa0 2023-10-16T12:27:36.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:36.124+0000 7fd8467fc700 1 --2- 172.21.15.16:0/3946007672 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fd82806b020 0x7fd82806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:36.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:36.124+0000 7fd8467fc700 1 --2- 172.21.15.16:0/3946007672 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fd82806b020 0x7fd82806d4e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd8480b8300 tx=0x7fd8300079c0 comp rx=0 tx=0).ready entity=mgr.6365 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:36.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:36.128+0000 7fd83ffff700 1 -- 172.21.15.16:0/3946007672 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd838056160 con 0x7fd8480b8fa0 2023-10-16T12:27:36.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:36.329+0000 7fd84d912700 1 -- 172.21.15.16:0/3946007672 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fd8480c7e00 con 0x7fd8480b8fa0 2023-10-16T12:27:36.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:36.967+0000 7fd83ffff700 1 -- 172.21.15.16:0/3946007672 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v110) v1 ==== 170+0+0 (secure 0 0 0) 0x7fd8480c7e00 con 0x7fd8480b8fa0 2023-10-16T12:27:36.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:36.971+0000 7fd84d912700 1 -- 172.21.15.16:0/3946007672 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fd82806b020 msgr2=0x7fd82806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:36.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:36.971+0000 7fd84d912700 1 --2- 172.21.15.16:0/3946007672 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fd82806b020 0x7fd82806d4e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd8480b8300 tx=0x7fd8300079c0 comp rx=0 tx=0).stop 2023-10-16T12:27:36.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:36.971+0000 7fd84d912700 1 -- 172.21.15.16:0/3946007672 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8480b8fa0 msgr2=0x7fd848151df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:36.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:36.971+0000 7fd84d912700 1 --2- 172.21.15.16:0/3946007672 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8480b8fa0 0x7fd848151df0 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7fd838000c00 tx=0x7fd838004750 comp rx=0 tx=0).stop 2023-10-16T12:27:36.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:36.971+0000 7fd84d912700 1 -- 172.21.15.16:0/3946007672 shutdown_connections 2023-10-16T12:27:36.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:36.971+0000 7fd84d912700 1 --2- 172.21.15.16:0/3946007672 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd84815d8d0 0x7fd8480c96a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:36.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:36.971+0000 7fd84d912700 1 --2- 172.21.15.16:0/3946007672 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8480b8fa0 0x7fd848151df0 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:36.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:36.971+0000 7fd84d912700 1 --2- 172.21.15.16:0/3946007672 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fd82806b020 0x7fd82806d4e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:36.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:36.971+0000 7fd84d912700 1 --2- 172.21.15.16:0/3946007672 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd8480ce350 0x7fd8481f4380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:36.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:36.971+0000 7fd84d912700 1 -- 172.21.15.16:0/3946007672 >> 172.21.15.16:0/3946007672 conn(0x7fd848072330 msgr2=0x7fd8480b8490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:36.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:36.971+0000 7fd84d912700 1 -- 172.21.15.16:0/3946007672 shutdown_connections 2023-10-16T12:27:36.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:36.971+0000 7fd84d912700 1 -- 172.21.15.16:0/3946007672 wait complete. 2023-10-16T12:27:36.985 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' created 2023-10-16T12:27:37.003 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-16T12:27:37.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:37.134+0000 7fec6ef09700 1 Processor -- start 2023-10-16T12:27:37.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:37.134+0000 7fec6ef09700 1 -- start start 2023-10-16T12:27:37.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:37.135+0000 7fec6ef09700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fec6815b070 0x7fec6815b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:37.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:37.135+0000 7fec6ef09700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fec680cb880 0x7fec680cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:37.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:37.135+0000 7fec6ef09700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec680cef70 con 0x7fec6806da60 2023-10-16T12:27:37.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:37.135+0000 7fec6ef09700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec680ce360 con 0x7fec6815b070 2023-10-16T12:27:37.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:37.135+0000 7fec6ef09700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec680ce640 con 0x7fec680cb880 2023-10-16T12:27:37.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:37.135+0000 7fec6cca5700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fec6815b070 0x7fec6815b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:37.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:37.135+0000 7fec6cca5700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fec6815b070 0x7fec6815b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42238/0 (socket says 172.21.15.16:42238) 2023-10-16T12:27:37.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:37.135+0000 7fec6cca5700 1 -- 172.21.15.16:0/2716548340 learned_addr learned my addr 172.21.15.16:0/2716548340 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:37.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:37.135+0000 7fec67fff700 1 --2- 172.21.15.16:0/2716548340 >> v2:172.21.15.49:3300/0 conn(0x7fec680cb880 0x7fec680cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:37.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:37.135+0000 7fec6cca5700 1 -- 172.21.15.16:0/2716548340 >> v2:172.21.15.49:3300/0 conn(0x7fec680cb880 msgr2=0x7fec680cbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:37.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:37.135+0000 7fec6cca5700 1 --2- 172.21.15.16:0/2716548340 >> v2:172.21.15.49:3300/0 conn(0x7fec680cb880 0x7fec680cbd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:37.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:37.135+0000 7fec6cca5700 1 -- 172.21.15.16:0/2716548340 >> v1:172.21.15.49:6789/0 conn(0x7fec6806da60 legacy=0x7fec680cf400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:37.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:37.135+0000 7fec6cca5700 1 -- 172.21.15.16:0/2716548340 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec680ce920 con 0x7fec6815b070 2023-10-16T12:27:37.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:37.136+0000 7fec6cca5700 1 --2- 172.21.15.16:0/2716548340 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec6815b070 0x7fec6815b450 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7fec68070500 tx=0x7fec5c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9f53e1c9fb04bdb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:37.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:37.136+0000 7fec677fe700 1 -- 172.21.15.16:0/2716548340 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec5c004230 con 0x7fec6815b070 2023-10-16T12:27:37.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:37.136+0000 7fec677fe700 1 -- 172.21.15.16:0/2716548340 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec5c004390 con 0x7fec6815b070 2023-10-16T12:27:37.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:37.136+0000 7fec677fe700 1 -- 172.21.15.16:0/2716548340 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec5c0036a0 con 0x7fec6815b070 2023-10-16T12:27:37.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:37.136+0000 7fec6ef09700 1 -- 172.21.15.16:0/2716548340 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec6815b070 msgr2=0x7fec6815b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:37.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:37.136+0000 7fec6ef09700 1 --2- 172.21.15.16:0/2716548340 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec6815b070 0x7fec6815b450 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7fec68070500 tx=0x7fec5c009bd0 comp rx=0 tx=0).stop 2023-10-16T12:27:37.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:37.136+0000 7fec6ef09700 1 -- 172.21.15.16:0/2716548340 shutdown_connections 2023-10-16T12:27:37.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:37.136+0000 7fec6ef09700 1 --2- 172.21.15.16:0/2716548340 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec6815b070 0x7fec6815b450 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:37.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:37.136+0000 7fec6ef09700 1 --2- 172.21.15.16:0/2716548340 >> v2:172.21.15.49:3300/0 conn(0x7fec680cb880 0x7fec680cbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:37.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:37.136+0000 7fec6ef09700 1 -- 172.21.15.16:0/2716548340 >> 172.21.15.16:0/2716548340 conn(0x7fec680b1f40 msgr2=0x7fec680b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:37.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:37.136+0000 7fec6ef09700 1 -- 172.21.15.16:0/2716548340 shutdown_connections 2023-10-16T12:27:37.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:37.136+0000 7fec6ef09700 1 -- 172.21.15.16:0/2716548340 wait complete. 2023-10-16T12:27:37.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:37.137+0000 7fec6ef09700 1 Processor -- start 2023-10-16T12:27:37.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:37.138+0000 7fec6ef09700 1 -- start start 2023-10-16T12:27:37.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:37.138+0000 7fec6ef09700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec6806da60 0x7fec681518a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:37.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:37.138+0000 7fec6ef09700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fec680cb880 0x7fec680cc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:37.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:37.138+0000 7fec6ef09700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fec6815b070 0x7fec680cc900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:37.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:37.138+0000 7fec6ef09700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec6815d100 con 0x7fec6806da60 2023-10-16T12:27:37.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:37.138+0000 7fec6ef09700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec68156570 con 0x7fec6815b070 2023-10-16T12:27:37.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:37.138+0000 7fec6ef09700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec681f3d80 con 0x7fec680cb880 2023-10-16T12:27:37.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:37.139+0000 7fec6cca5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec6806da60 0x7fec681518a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:37.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:37.139+0000 7fec6cca5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec6806da60 0x7fec681518a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42244/0 (socket says 172.21.15.16:42244) 2023-10-16T12:27:37.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:37.139+0000 7fec6cca5700 1 -- 172.21.15.16:0/3038357405 learned_addr learned my addr 172.21.15.16:0/3038357405 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:37.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:37.139+0000 7fec6d4a6700 1 --2- 172.21.15.16:0/3038357405 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fec6815b070 0x7fec680cc900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:37.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:37.139+0000 7fec67fff700 1 --2- 172.21.15.16:0/3038357405 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fec680cb880 0x7fec680cc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:37.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:37.139+0000 7fec6cca5700 1 -- 172.21.15.16:0/3038357405 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fec680cb880 msgr2=0x7fec680cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:37.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:37.139+0000 7fec6cca5700 1 --2- 172.21.15.16:0/3038357405 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fec680cb880 0x7fec680cc3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:37.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:37.139+0000 7fec6cca5700 1 -- 172.21.15.16:0/3038357405 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fec6815b070 msgr2=0x7fec680cc900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:37.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:37.139+0000 7fec6cca5700 1 --2- 172.21.15.16:0/3038357405 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fec6815b070 0x7fec680cc900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:37.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:37.139+0000 7fec6cca5700 1 -- 172.21.15.16:0/3038357405 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec5c009550 con 0x7fec6806da60 2023-10-16T12:27:37.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:37.139+0000 7fec67fff700 1 --2- 172.21.15.16:0/3038357405 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fec680cb880 0x7fec680cc3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:27:37.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:37.139+0000 7fec6cca5700 1 --2- 172.21.15.16:0/3038357405 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec6806da60 0x7fec681518a0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7fec5c009130 tx=0x7fec5c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:37.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:37.139+0000 7fec657fa700 1 -- 172.21.15.16:0/3038357405 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec5c003e20 con 0x7fec6806da60 2023-10-16T12:27:37.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:37.139+0000 7fec657fa700 1 -- 172.21.15.16:0/3038357405 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec5c011630 con 0x7fec6806da60 2023-10-16T12:27:37.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:37.139+0000 7fec6ef09700 1 -- 172.21.15.16:0/3038357405 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec681f4060 con 0x7fec6806da60 2023-10-16T12:27:37.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:37.139+0000 7fec657fa700 1 -- 172.21.15.16:0/3038357405 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec5c0119e0 con 0x7fec6806da60 2023-10-16T12:27:37.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:37.139+0000 7fec6ef09700 1 -- 172.21.15.16:0/3038357405 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec68156200 con 0x7fec6806da60 2023-10-16T12:27:37.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:37.140+0000 7fec6ef09700 1 -- 172.21.15.16:0/3038357405 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec68155d80 con 0x7fec6806da60 2023-10-16T12:27:37.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:37.141+0000 7fec657fa700 1 -- 172.21.15.16:0/3038357405 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 51) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fec5c011b40 con 0x7fec6806da60 2023-10-16T12:27:37.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:37.141+0000 7fec657fa700 1 --2- 172.21.15.16:0/3038357405 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fec4c06b050 0x7fec4c06d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:37.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:37.141+0000 7fec657fa700 1 -- 172.21.15.16:0/3038357405 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5957+0+0 (secure 0 0 0) 0x7fec5c08ae80 con 0x7fec6806da60 2023-10-16T12:27:37.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:37.141+0000 7fec67fff700 1 --2- 172.21.15.16:0/3038357405 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fec4c06b050 0x7fec4c06d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:37.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:37.142+0000 7fec67fff700 1 --2- 172.21.15.16:0/3038357405 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fec4c06b050 0x7fec4c06d510 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fec681f42c0 tx=0x7fec54008040 comp rx=0 tx=0).ready entity=mgr.6365 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:37.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:37.146+0000 7fec657fa700 1 -- 172.21.15.16:0/3038357405 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fec5c055ed0 con 0x7fec6806da60 2023-10-16T12:27:37.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:37.395+0000 7fec6ef09700 1 -- 172.21.15.16:0/3038357405 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fec68155d80 con 0x7fec6806da60 2023-10-16T12:27:38.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:37.998+0000 7fec657fa700 1 -- 172.21.15.16:0/3038357405 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 13 and data pool 14 v49) v1 ==== 174+0+0 (secure 0 0 0) 0x7fec68155d80 con 0x7fec6806da60 2023-10-16T12:27:38.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.009+0000 7fec6ef09700 1 -- 172.21.15.16:0/3038357405 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fec4c06b050 msgr2=0x7fec4c06d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:38.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.009+0000 7fec6ef09700 1 --2- 172.21.15.16:0/3038357405 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fec4c06b050 0x7fec4c06d510 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fec681f42c0 tx=0x7fec54008040 comp rx=0 tx=0).stop 2023-10-16T12:27:38.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.009+0000 7fec6ef09700 1 -- 172.21.15.16:0/3038357405 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec6806da60 msgr2=0x7fec681518a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:38.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.009+0000 7fec6ef09700 1 --2- 172.21.15.16:0/3038357405 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec6806da60 0x7fec681518a0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7fec5c009130 tx=0x7fec5c004750 comp rx=0 tx=0).stop 2023-10-16T12:27:38.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.010+0000 7fec6ef09700 1 -- 172.21.15.16:0/3038357405 shutdown_connections 2023-10-16T12:27:38.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.010+0000 7fec6ef09700 1 --2- 172.21.15.16:0/3038357405 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fec4c06b050 0x7fec4c06d510 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:38.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.010+0000 7fec6ef09700 1 --2- 172.21.15.16:0/3038357405 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fec6815b070 0x7fec680cc900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:38.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.010+0000 7fec6ef09700 1 --2- 172.21.15.16:0/3038357405 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec6806da60 0x7fec681518a0 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:38.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.010+0000 7fec6ef09700 1 --2- 172.21.15.16:0/3038357405 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fec680cb880 0x7fec680cc3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:38.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.010+0000 7fec6ef09700 1 -- 172.21.15.16:0/3038357405 >> 172.21.15.16:0/3038357405 conn(0x7fec680b1f40 msgr2=0x7fec68159980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:38.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.011+0000 7fec6ef09700 1 -- 172.21.15.16:0/3038357405 shutdown_connections 2023-10-16T12:27:38.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.011+0000 7fec6ef09700 1 -- 172.21.15.16:0/3038357405 wait complete. 2023-10-16T12:27:38.021 INFO:teuthology.orchestra.run.smithi016.stderr:new fs with metadata pool 13 and data pool 14 2023-10-16T12:27:38.041 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:27:38.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.167+0000 7fdf03ce5700 1 Processor -- start 2023-10-16T12:27:38.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.168+0000 7fdf03ce5700 1 -- start start 2023-10-16T12:27:38.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.168+0000 7fdf03ce5700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fdefc15b280 0x7fdefc1613e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:38.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.168+0000 7fdf03ce5700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdefc0cc8a0 0x7fdefc15e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:38.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.168+0000 7fdf03ce5700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdefc161c30 con 0x7fdefc06a7e0 2023-10-16T12:27:38.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.168+0000 7fdf03ce5700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdefc161f10 con 0x7fdefc15b280 2023-10-16T12:27:38.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.168+0000 7fdf03ce5700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdefc1621f0 con 0x7fdefc0cc8a0 2023-10-16T12:27:38.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.168+0000 7fdf02282700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fdefc06a7e0 0x7fdefc0d02b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59760/0 (socket says 172.21.15.16:59760) 2023-10-16T12:27:38.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.169+0000 7fdf02282700 1 -- 172.21.15.16:0/4134896378 learned_addr learned my addr 172.21.15.16:0/4134896378 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:27:38.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.169+0000 7fdf01280700 1 --2- 172.21.15.16:0/4134896378 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdefc0cc8a0 0x7fdefc15e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:38.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.169+0000 7fdf01a81700 1 --2- 172.21.15.16:0/4134896378 >> v2:172.21.15.49:3300/0 conn(0x7fdefc15b280 0x7fdefc1613e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:38.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.169+0000 7fdf00a7f700 1 -- 172.21.15.16:0/4134896378 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2297078746 0 0) 0x7fdefc161c30 con 0x7fdefc06a7e0 2023-10-16T12:27:38.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.169+0000 7fdf00a7f700 1 -- 172.21.15.16:0/4134896378 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdee80035b0 con 0x7fdefc06a7e0 2023-10-16T12:27:38.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.169+0000 7fdf00a7f700 1 -- 172.21.15.16:0/4134896378 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1105635015 0 0) 0x7fdee80035b0 con 0x7fdefc06a7e0 2023-10-16T12:27:38.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.169+0000 7fdf00a7f700 1 -- 172.21.15.16:0/4134896378 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdefc0cc8a0 msgr2=0x7fdefc15e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:38.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.169+0000 7fdf00a7f700 1 --2- 172.21.15.16:0/4134896378 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdefc0cc8a0 0x7fdefc15e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:38.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.169+0000 7fdf00a7f700 1 -- 172.21.15.16:0/4134896378 >> v2:172.21.15.49:3300/0 conn(0x7fdefc15b280 msgr2=0x7fdefc1613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:38.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.169+0000 7fdf00a7f700 1 --2- 172.21.15.16:0/4134896378 >> v2:172.21.15.49:3300/0 conn(0x7fdefc15b280 0x7fdefc1613e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:38.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.169+0000 7fdf00a7f700 1 -- 172.21.15.16:0/4134896378 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdefc15df00 con 0x7fdefc06a7e0 2023-10-16T12:27:38.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.169+0000 7fdf00a7f700 1 -- 172.21.15.16:0/4134896378 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7fdef40019f0 con 0x7fdefc06a7e0 2023-10-16T12:27:38.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.169+0000 7fdf00a7f700 1 -- 172.21.15.16:0/4134896378 >> v1:172.21.15.16:6789/0 conn(0x7fdefc06a7e0 legacy=0x7fdefc0d02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:38.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.169+0000 7fdf00a7f700 1 --2- 172.21.15.16:0/4134896378 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdee8003a00 0x7fdee8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:38.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.169+0000 7fdf00a7f700 1 --2- 172.21.15.16:0/4134896378 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdefc0cc8a0 0x7fdee8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:38.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.169+0000 7fdf00a7f700 1 --2- 172.21.15.16:0/4134896378 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdee8004d90 0x7fdee8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:38.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.169+0000 7fdf00a7f700 1 -- 172.21.15.16:0/4134896378 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdefc161f10 con 0x7fdee8003a00 2023-10-16T12:27:38.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.169+0000 7fdf00a7f700 1 -- 172.21.15.16:0/4134896378 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdefc1621f0 con 0x7fdee8004d90 2023-10-16T12:27:38.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.169+0000 7fdf00a7f700 1 -- 172.21.15.16:0/4134896378 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdee800ba10 con 0x7fdefc0cc8a0 2023-10-16T12:27:38.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.169+0000 7fdf01a81700 1 --2- 172.21.15.16:0/4134896378 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdee8003a00 0x7fdee8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:38.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.169+0000 7fdf02282700 1 --2- 172.21.15.16:0/4134896378 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdee8004d90 0x7fdee8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:38.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.169+0000 7fdf01280700 1 --2- 172.21.15.16:0/4134896378 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdefc0cc8a0 0x7fdee8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:38.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.170+0000 7fdf01a81700 1 -- 172.21.15.16:0/4134896378 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdefc0cc8a0 msgr2=0x7fdee8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:38.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.170+0000 7fdf01a81700 1 --2- 172.21.15.16:0/4134896378 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdefc0cc8a0 0x7fdee8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:38.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.170+0000 7fdf01a81700 1 -- 172.21.15.16:0/4134896378 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdee8004d90 msgr2=0x7fdee8005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:38.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.170+0000 7fdf01a81700 1 --2- 172.21.15.16:0/4134896378 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdee8004d90 0x7fdee8005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:38.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.170+0000 7fdf01a81700 1 -- 172.21.15.16:0/4134896378 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdee80035b0 con 0x7fdee8003a00 2023-10-16T12:27:38.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.170+0000 7fdf01280700 1 --2- 172.21.15.16:0/4134896378 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdefc0cc8a0 0x7fdee8004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:27:38.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.170+0000 7fdf01a81700 1 --2- 172.21.15.16:0/4134896378 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdee8003a00 0x7fdee8003e20 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7fdef0009b90 tx=0x7fdee8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e532296e6bb742ad server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:38.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.170+0000 7fdf00a7f700 1 -- 172.21.15.16:0/4134896378 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdef000e6c0 con 0x7fdee8003a00 2023-10-16T12:27:38.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.170+0000 7fdf00a7f700 1 -- 172.21.15.16:0/4134896378 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdef000e820 con 0x7fdee8003a00 2023-10-16T12:27:38.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.170+0000 7fdf00a7f700 1 -- 172.21.15.16:0/4134896378 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdef000eb90 con 0x7fdee8003a00 2023-10-16T12:27:38.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.170+0000 7fdf03ce5700 1 -- 172.21.15.16:0/4134896378 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdee8003a00 msgr2=0x7fdee8003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:38.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.170+0000 7fdf03ce5700 1 --2- 172.21.15.16:0/4134896378 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdee8003a00 0x7fdee8003e20 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7fdef0009b90 tx=0x7fdee8005880 comp rx=0 tx=0).stop 2023-10-16T12:27:38.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.170+0000 7fdf02282700 1 -- 172.21.15.16:0/4134896378 reap_dead start 2023-10-16T12:27:38.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.170+0000 7fdf03ce5700 1 -- 172.21.15.16:0/4134896378 shutdown_connections 2023-10-16T12:27:38.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.170+0000 7fdf03ce5700 1 -- 172.21.15.16:0/4134896378 >> 172.21.15.16:0/4134896378 conn(0x7fdefc0b1fd0 msgr2=0x7fdefc0b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:38.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.171+0000 7fdf03ce5700 1 -- 172.21.15.16:0/4134896378 shutdown_connections 2023-10-16T12:27:38.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.171+0000 7fdf03ce5700 1 -- 172.21.15.16:0/4134896378 wait complete. 2023-10-16T12:27:38.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.172+0000 7fdf03ce5700 1 Processor -- start 2023-10-16T12:27:38.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.172+0000 7fdf03ce5700 1 -- start start 2023-10-16T12:27:38.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.172+0000 7fdf03ce5700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdee8003a00 0x7fdefc0c4380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:38.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.172+0000 7fdf03ce5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdefc1488f0 0x7fdefc15ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:38.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.172+0000 7fdf03ce5700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdefc0cb260 0x7fdefc0cb6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:38.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.173+0000 7fdf03ce5700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdefc0cbc20 con 0x7fdefc1488f0 2023-10-16T12:27:38.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.173+0000 7fdf03ce5700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdefc0c7e10 con 0x7fdee8003a00 2023-10-16T12:27:38.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.173+0000 7fdf03ce5700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdefc148d70 con 0x7fdefc0cb260 2023-10-16T12:27:38.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.173+0000 7fdf01280700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdefc1488f0 0x7fdefc15ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:38.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.173+0000 7fdf01280700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdefc1488f0 0x7fdefc15ffb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42264/0 (socket says 172.21.15.16:42264) 2023-10-16T12:27:38.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.173+0000 7fdf01280700 1 -- 172.21.15.16:0/1974277530 learned_addr learned my addr 172.21.15.16:0/1974277530 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:38.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.173+0000 7fdf01a81700 1 --2- 172.21.15.16:0/1974277530 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdee8003a00 0x7fdefc0c4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:38.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.173+0000 7fdf02282700 1 --2- 172.21.15.16:0/1974277530 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdefc0cb260 0x7fdefc0cb6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:38.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.173+0000 7fdf01280700 1 -- 172.21.15.16:0/1974277530 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdefc0cb260 msgr2=0x7fdefc0cb6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:38.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.173+0000 7fdf01280700 1 --2- 172.21.15.16:0/1974277530 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdefc0cb260 0x7fdefc0cb6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:38.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.173+0000 7fdf01280700 1 -- 172.21.15.16:0/1974277530 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdee8003a00 msgr2=0x7fdefc0c4380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:38.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.173+0000 7fdf01280700 1 --2- 172.21.15.16:0/1974277530 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdee8003a00 0x7fdefc0c4380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:38.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.173+0000 7fdf01280700 1 -- 172.21.15.16:0/1974277530 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdefc0c8060 con 0x7fdefc1488f0 2023-10-16T12:27:38.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.173+0000 7fdf01280700 1 --2- 172.21.15.16:0/1974277530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdefc1488f0 0x7fdefc15ffb0 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7fdef80118d0 tx=0x7fdef8011c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:38.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.173+0000 7fdeee7fc700 1 -- 172.21.15.16:0/1974277530 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdef8004220 con 0x7fdefc1488f0 2023-10-16T12:27:38.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.173+0000 7fdeee7fc700 1 -- 172.21.15.16:0/1974277530 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdef8004380 con 0x7fdefc1488f0 2023-10-16T12:27:38.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.173+0000 7fdeee7fc700 1 -- 172.21.15.16:0/1974277530 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdef80036a0 con 0x7fdefc1488f0 2023-10-16T12:27:38.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.173+0000 7fdf03ce5700 1 -- 172.21.15.16:0/1974277530 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdefc0c8340 con 0x7fdefc1488f0 2023-10-16T12:27:38.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.173+0000 7fdf03ce5700 1 -- 172.21.15.16:0/1974277530 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdefc0c8900 con 0x7fdefc1488f0 2023-10-16T12:27:38.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.174+0000 7fdf03ce5700 1 -- 172.21.15.16:0/1974277530 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdefc04ebb0 con 0x7fdefc1488f0 2023-10-16T12:27:38.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.175+0000 7fdeee7fc700 1 -- 172.21.15.16:0/1974277530 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 51) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fdef8003800 con 0x7fdefc1488f0 2023-10-16T12:27:38.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.175+0000 7fdeee7fc700 1 --2- 172.21.15.16:0/1974277530 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fdee006b020 0x7fdee006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:38.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.175+0000 7fdeee7fc700 1 -- 172.21.15.16:0/1974277530 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fdef8089ca0 con 0x7fdefc1488f0 2023-10-16T12:27:38.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.175+0000 7fdf01a81700 1 --2- 172.21.15.16:0/1974277530 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fdee006b020 0x7fdee006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:38.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.176+0000 7fdf01a81700 1 --2- 172.21.15.16:0/1974277530 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fdee006b020 0x7fdee006d4e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fdef00090b0 tx=0x7fdef0008030 comp rx=0 tx=0).ready entity=mgr.6365 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:38.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.180+0000 7fdeee7fc700 1 -- 172.21.15.16:0/1974277530 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdef8054b70 con 0x7fdefc1488f0 2023-10-16T12:27:38.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.385+0000 7fdf03ce5700 1 -- 172.21.15.16:0/1974277530 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdefc0720b0 con 0x7fdefc1488f0 2023-10-16T12:27:38.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.386+0000 7fdeee7fc700 1 -- 172.21.15.16:0/1974277530 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v111) v1 ==== 74+0+16676 (secure 0 0 0) 0x7fdef8022020 con 0x7fdefc1488f0 2023-10-16T12:27:38.389 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:27:38.390 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":111,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:27:37.397478+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:27:35.907633+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2023-10-16T12:27:36.362861+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:27:38.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.388+0000 7fdf03ce5700 1 -- 172.21.15.16:0/1974277530 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fdee006b020 msgr2=0x7fdee006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:38.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.388+0000 7fdf03ce5700 1 --2- 172.21.15.16:0/1974277530 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fdee006b020 0x7fdee006d4e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fdef00090b0 tx=0x7fdef0008030 comp rx=0 tx=0).stop 2023-10-16T12:27:38.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.388+0000 7fdf03ce5700 1 -- 172.21.15.16:0/1974277530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdefc1488f0 msgr2=0x7fdefc15ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:38.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.388+0000 7fdf03ce5700 1 --2- 172.21.15.16:0/1974277530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdefc1488f0 0x7fdefc15ffb0 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7fdef80118d0 tx=0x7fdef8011c90 comp rx=0 tx=0).stop 2023-10-16T12:27:38.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.388+0000 7fdf03ce5700 1 -- 172.21.15.16:0/1974277530 shutdown_connections 2023-10-16T12:27:38.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.388+0000 7fdf03ce5700 1 --2- 172.21.15.16:0/1974277530 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdefc0cb260 0x7fdefc0cb6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:38.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.388+0000 7fdf03ce5700 1 --2- 172.21.15.16:0/1974277530 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fdee006b020 0x7fdee006d4e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:38.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.388+0000 7fdf03ce5700 1 --2- 172.21.15.16:0/1974277530 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdee8003a00 0x7fdefc0c4380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:38.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.388+0000 7fdf03ce5700 1 --2- 172.21.15.16:0/1974277530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdefc1488f0 0x7fdefc15ffb0 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:38.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.388+0000 7fdf03ce5700 1 -- 172.21.15.16:0/1974277530 >> 172.21.15.16:0/1974277530 conn(0x7fdefc0b1fd0 msgr2=0x7fdefc15a930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:38.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.388+0000 7fdf03ce5700 1 -- 172.21.15.16:0/1974277530 shutdown_connections 2023-10-16T12:27:38.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.389+0000 7fdf03ce5700 1 -- 172.21.15.16:0/1974277530 wait complete. 2023-10-16T12:27:38.404 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:27:38.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.525+0000 7fed79960700 1 Processor -- start 2023-10-16T12:27:38.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.525+0000 7fed79960700 1 -- start start 2023-10-16T12:27:38.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.526+0000 7fed79960700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fed740cf880 0x7fed740b52f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:38.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.526+0000 7fed79960700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed7415bc80 con 0x7fed740b5830 2023-10-16T12:27:38.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.526+0000 7fed79960700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed7415b070 con 0x7fed740cf4a0 2023-10-16T12:27:38.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.526+0000 7fed79960700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed7415b350 con 0x7fed740cf880 2023-10-16T12:27:38.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.526+0000 7fed737fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fed740b5830 0x7fed7415a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59780/0 (socket says 172.21.15.16:59780) 2023-10-16T12:27:38.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.526+0000 7fed737fe700 1 -- 172.21.15.16:0/3052874857 learned_addr learned my addr 172.21.15.16:0/3052874857 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:27:38.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.526+0000 7fed727fc700 1 --2- 172.21.15.16:0/3052874857 >> v2:172.21.15.49:3300/0 conn(0x7fed740cf880 0x7fed740b52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:38.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.527+0000 7fed71ffb700 1 -- 172.21.15.16:0/3052874857 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2760648224 0 0) 0x7fed7415bc80 con 0x7fed740b5830 2023-10-16T12:27:38.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.527+0000 7fed71ffb700 1 -- 172.21.15.16:0/3052874857 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed5c0035b0 con 0x7fed740b5830 2023-10-16T12:27:38.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.527+0000 7fed71ffb700 1 -- 172.21.15.16:0/3052874857 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 142515970 0 0) 0x7fed5c0035b0 con 0x7fed740b5830 2023-10-16T12:27:38.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.527+0000 7fed727fc700 1 -- 172.21.15.16:0/3052874857 >> v1:172.21.15.49:6789/0 conn(0x7fed740cf4a0 legacy=0x7fed740726e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:38.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.527+0000 7fed727fc700 1 -- 172.21.15.16:0/3052874857 >> v1:172.21.15.16:6789/0 conn(0x7fed740b5830 legacy=0x7fed7415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:38.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.527+0000 7fed727fc700 1 -- 172.21.15.16:0/3052874857 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed7415b630 con 0x7fed740cf880 2023-10-16T12:27:38.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.527+0000 7fed727fc700 1 --2- 172.21.15.16:0/3052874857 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fed740cf880 0x7fed740b52f0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fed68009840 tx=0x7fed68009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f1962d6e2eac35bd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:38.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.527+0000 7fed71ffb700 1 -- 172.21.15.16:0/3052874857 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed6c0019f0 con 0x7fed740cf880 2023-10-16T12:27:38.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.527+0000 7fed71ffb700 1 -- 172.21.15.16:0/3052874857 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed68004030 con 0x7fed740cf880 2023-10-16T12:27:38.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.527+0000 7fed71ffb700 1 -- 172.21.15.16:0/3052874857 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed680043c0 con 0x7fed740cf880 2023-10-16T12:27:38.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.528+0000 7fed79960700 1 -- 172.21.15.16:0/3052874857 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fed740cf880 msgr2=0x7fed740b52f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:38.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.528+0000 7fed79960700 1 --2- 172.21.15.16:0/3052874857 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fed740cf880 0x7fed740b52f0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fed68009840 tx=0x7fed68009b50 comp rx=0 tx=0).stop 2023-10-16T12:27:38.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.528+0000 7fed79960700 1 -- 172.21.15.16:0/3052874857 shutdown_connections 2023-10-16T12:27:38.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.528+0000 7fed79960700 1 --2- 172.21.15.16:0/3052874857 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fed740cf880 0x7fed740b52f0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:38.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.528+0000 7fed79960700 1 -- 172.21.15.16:0/3052874857 >> 172.21.15.16:0/3052874857 conn(0x7fed740b1fd0 msgr2=0x7fed740b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:38.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.528+0000 7fed79960700 1 -- 172.21.15.16:0/3052874857 shutdown_connections 2023-10-16T12:27:38.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.528+0000 7fed79960700 1 -- 172.21.15.16:0/3052874857 wait complete. 2023-10-16T12:27:38.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.529+0000 7fed79960700 1 Processor -- start 2023-10-16T12:27:38.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.529+0000 7fed79960700 1 -- start start 2023-10-16T12:27:38.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.530+0000 7fed79960700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed740b5830 0x7fed74154bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:38.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.530+0000 7fed79960700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fed740cf4a0 0x7fed740c67f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:38.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.530+0000 7fed79960700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fed740cf880 0x7fed7415c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:38.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.530+0000 7fed79960700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed741f6fe0 con 0x7fed740b5830 2023-10-16T12:27:38.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.530+0000 7fed79960700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed741f7e40 con 0x7fed740cf4a0 2023-10-16T12:27:38.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.530+0000 7fed79960700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed741f71c0 con 0x7fed740cf880 2023-10-16T12:27:38.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.530+0000 7fed72ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed740b5830 0x7fed74154bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:38.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.530+0000 7fed72ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed740b5830 0x7fed74154bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42276/0 (socket says 172.21.15.16:42276) 2023-10-16T12:27:38.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.530+0000 7fed72ffd700 1 -- 172.21.15.16:0/1407382916 learned_addr learned my addr 172.21.15.16:0/1407382916 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:38.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.530+0000 7fed727fc700 1 --2- 172.21.15.16:0/1407382916 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fed740cf4a0 0x7fed740c67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:38.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.530+0000 7fed737fe700 1 --2- 172.21.15.16:0/1407382916 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fed740cf880 0x7fed7415c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:38.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.530+0000 7fed72ffd700 1 -- 172.21.15.16:0/1407382916 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fed740cf880 msgr2=0x7fed7415c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:38.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.530+0000 7fed72ffd700 1 --2- 172.21.15.16:0/1407382916 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fed740cf880 0x7fed7415c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:38.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.530+0000 7fed72ffd700 1 -- 172.21.15.16:0/1407382916 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fed740cf4a0 msgr2=0x7fed740c67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:38.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.530+0000 7fed72ffd700 1 --2- 172.21.15.16:0/1407382916 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fed740cf4a0 0x7fed740c67f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:38.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.530+0000 7fed72ffd700 1 -- 172.21.15.16:0/1407382916 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed680094a0 con 0x7fed740b5830 2023-10-16T12:27:38.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.531+0000 7fed72ffd700 1 --2- 172.21.15.16:0/1407382916 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed740b5830 0x7fed74154bb0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7fed6400c490 tx=0x7fed6400c7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:38.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.531+0000 7fed5bfff700 1 -- 172.21.15.16:0/1407382916 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed6400cea0 con 0x7fed740b5830 2023-10-16T12:27:38.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.531+0000 7fed5bfff700 1 -- 172.21.15.16:0/1407382916 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed640052f0 con 0x7fed740b5830 2023-10-16T12:27:38.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.531+0000 7fed79960700 1 -- 172.21.15.16:0/1407382916 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed741f74a0 con 0x7fed740b5830 2023-10-16T12:27:38.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.531+0000 7fed5bfff700 1 -- 172.21.15.16:0/1407382916 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed64006860 con 0x7fed740b5830 2023-10-16T12:27:38.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.531+0000 7fed79960700 1 -- 172.21.15.16:0/1407382916 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed741f7a60 con 0x7fed740b5830 2023-10-16T12:27:38.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.532+0000 7fed79960700 1 -- 172.21.15.16:0/1407382916 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed741f7bf0 con 0x7fed740b5830 2023-10-16T12:27:38.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.532+0000 7fed5bfff700 1 -- 172.21.15.16:0/1407382916 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 51) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fed64003680 con 0x7fed740b5830 2023-10-16T12:27:38.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.533+0000 7fed5bfff700 1 --2- 172.21.15.16:0/1407382916 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fed6006b020 0x7fed6006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:38.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.533+0000 7fed5bfff700 1 -- 172.21.15.16:0/1407382916 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fed64088a80 con 0x7fed740b5830 2023-10-16T12:27:38.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.533+0000 7fed727fc700 1 --2- 172.21.15.16:0/1407382916 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fed6006b020 0x7fed6006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:38.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.533+0000 7fed727fc700 1 --2- 172.21.15.16:0/1407382916 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fed6006b020 0x7fed6006d4e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fed680090b0 tx=0x7fed68008030 comp rx=0 tx=0).ready entity=mgr.6365 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:38.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.537+0000 7fed5bfff700 1 -- 172.21.15.16:0/1407382916 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fed640538c0 con 0x7fed740b5830 2023-10-16T12:27:38.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.741+0000 7fed79960700 1 -- 172.21.15.16:0/1407382916 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fed7404ebb0 con 0x7fed740b5830 2023-10-16T12:27:38.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.742+0000 7fed5bfff700 1 -- 172.21.15.16:0/1407382916 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v111) v1 ==== 74+0+16676 (secure 0 0 0) 0x7fed64056ee0 con 0x7fed740b5830 2023-10-16T12:27:38.746 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:27:38.746 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":111,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:27:37.397478+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:27:35.907633+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2023-10-16T12:27:36.362861+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:27:38.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.744+0000 7fed79960700 1 -- 172.21.15.16:0/1407382916 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fed6006b020 msgr2=0x7fed6006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:38.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.744+0000 7fed79960700 1 --2- 172.21.15.16:0/1407382916 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fed6006b020 0x7fed6006d4e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fed680090b0 tx=0x7fed68008030 comp rx=0 tx=0).stop 2023-10-16T12:27:38.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.744+0000 7fed79960700 1 -- 172.21.15.16:0/1407382916 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed740b5830 msgr2=0x7fed74154bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:38.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.744+0000 7fed79960700 1 --2- 172.21.15.16:0/1407382916 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed740b5830 0x7fed74154bb0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7fed6400c490 tx=0x7fed6400c7a0 comp rx=0 tx=0).stop 2023-10-16T12:27:38.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.744+0000 7fed79960700 1 -- 172.21.15.16:0/1407382916 shutdown_connections 2023-10-16T12:27:38.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.744+0000 7fed79960700 1 --2- 172.21.15.16:0/1407382916 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fed740cf880 0x7fed7415c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:38.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.744+0000 7fed79960700 1 --2- 172.21.15.16:0/1407382916 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed740b5830 0x7fed74154bb0 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:38.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.744+0000 7fed79960700 1 --2- 172.21.15.16:0/1407382916 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fed6006b020 0x7fed6006d4e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:38.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.744+0000 7fed79960700 1 --2- 172.21.15.16:0/1407382916 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fed740cf4a0 0x7fed740c67f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:38.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.744+0000 7fed79960700 1 -- 172.21.15.16:0/1407382916 >> 172.21.15.16:0/1407382916 conn(0x7fed740b1fd0 msgr2=0x7fed740b1670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:38.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.745+0000 7fed79960700 1 -- 172.21.15.16:0/1407382916 shutdown_connections 2023-10-16T12:27:38.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.745+0000 7fed79960700 1 -- 172.21.15.16:0/1407382916 wait complete. 2023-10-16T12:27:38.760 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-16T12:27:38.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.885+0000 7f0c75da7700 1 Processor -- start 2023-10-16T12:27:38.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.885+0000 7f0c75da7700 1 -- start start 2023-10-16T12:27:38.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.886+0000 7f0c75da7700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f0c7006e570 0x7f0c7015e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:38.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.886+0000 7f0c75da7700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0c7006a7e0 0x7f0c7006ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:38.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.886+0000 7f0c75da7700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c7015b6a0 con 0x7f0c7015b2c0 2023-10-16T12:27:38.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.886+0000 7f0c75da7700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c7006b1b0 con 0x7f0c7006e570 2023-10-16T12:27:38.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.886+0000 7f0c75da7700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c70161b20 con 0x7f0c7006a7e0 2023-10-16T12:27:38.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.886+0000 7f0c6effd700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f0c7006e570 0x7f0c7015e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:38.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.886+0000 7f0c6ffff700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0c7006a7e0 0x7f0c7006ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:38.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.886+0000 7f0c6effd700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f0c7006e570 0x7f0c7015e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:41346/0 (socket says 172.21.15.16:41346) 2023-10-16T12:27:38.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.886+0000 7f0c6effd700 1 -- 172.21.15.16:0/3738746123 learned_addr learned my addr 172.21.15.16:0/3738746123 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:38.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.887+0000 7f0c6effd700 1 -- 172.21.15.16:0/3738746123 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0c7006a7e0 msgr2=0x7f0c7006ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:38.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.887+0000 7f0c6effd700 1 --2- 172.21.15.16:0/3738746123 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0c7006a7e0 0x7f0c7006ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:38.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.887+0000 7f0c6effd700 1 -- 172.21.15.16:0/3738746123 >> v1:172.21.15.49:6789/0 conn(0x7f0c7015b2c0 legacy=0x7f0c70161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:38.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.887+0000 7f0c6effd700 1 -- 172.21.15.16:0/3738746123 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c70161e00 con 0x7f0c7006e570 2023-10-16T12:27:38.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.887+0000 7f0c6effd700 1 --2- 172.21.15.16:0/3738746123 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0c7006e570 0x7f0c7015e5a0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f0c600098f0 tx=0x7f0c60009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b51e671be98fe911 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:38.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.887+0000 7f0c6e7fc700 1 -- 172.21.15.16:0/3738746123 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c60011830 con 0x7f0c7006e570 2023-10-16T12:27:38.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.887+0000 7f0c6e7fc700 1 -- 172.21.15.16:0/3738746123 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c60011990 con 0x7f0c7006e570 2023-10-16T12:27:38.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.887+0000 7f0c6e7fc700 1 -- 172.21.15.16:0/3738746123 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c60011d20 con 0x7f0c7006e570 2023-10-16T12:27:38.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.888+0000 7f0c75da7700 1 -- 172.21.15.16:0/3738746123 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0c7006e570 msgr2=0x7f0c7015e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:38.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.888+0000 7f0c75da7700 1 --2- 172.21.15.16:0/3738746123 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0c7006e570 0x7f0c7015e5a0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f0c600098f0 tx=0x7f0c60009c00 comp rx=0 tx=0).stop 2023-10-16T12:27:38.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.888+0000 7f0c75da7700 1 -- 172.21.15.16:0/3738746123 shutdown_connections 2023-10-16T12:27:38.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.888+0000 7f0c75da7700 1 --2- 172.21.15.16:0/3738746123 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0c7006a7e0 0x7f0c7006ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:38.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.888+0000 7f0c75da7700 1 --2- 172.21.15.16:0/3738746123 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0c7006e570 0x7f0c7015e5a0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:38.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.888+0000 7f0c75da7700 1 -- 172.21.15.16:0/3738746123 >> 172.21.15.16:0/3738746123 conn(0x7f0c700b1f40 msgr2=0x7f0c700b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:38.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.888+0000 7f0c75da7700 1 -- 172.21.15.16:0/3738746123 shutdown_connections 2023-10-16T12:27:38.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.888+0000 7f0c75da7700 1 -- 172.21.15.16:0/3738746123 wait complete. 2023-10-16T12:27:38.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.889+0000 7f0c75da7700 1 Processor -- start 2023-10-16T12:27:38.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.889+0000 7f0c75da7700 1 -- start start 2023-10-16T12:27:38.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.890+0000 7f0c75da7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c7006a7e0 0x7f0c700ca180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:38.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.890+0000 7f0c75da7700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0c7006e570 0x7f0c7014e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:38.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.890+0000 7f0c75da7700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0c7015b2c0 0x7f0c700c5e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:38.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.890+0000 7f0c75da7700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c700ca6c0 con 0x7f0c7006a7e0 2023-10-16T12:27:38.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.890+0000 7f0c75da7700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c700c7830 con 0x7f0c7015b2c0 2023-10-16T12:27:38.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.890+0000 7f0c75da7700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c7014ec80 con 0x7f0c7006e570 2023-10-16T12:27:38.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.890+0000 7f0c6f7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c7006a7e0 0x7f0c700ca180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:38.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.890+0000 7f0c6f7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c7006a7e0 0x7f0c700ca180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42288/0 (socket says 172.21.15.16:42288) 2023-10-16T12:27:38.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.890+0000 7f0c6f7fe700 1 -- 172.21.15.16:0/3833604430 learned_addr learned my addr 172.21.15.16:0/3833604430 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:38.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.890+0000 7f0c6effd700 1 --2- 172.21.15.16:0/3833604430 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0c7006e570 0x7f0c7014e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:38.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.890+0000 7f0c6ffff700 1 --2- 172.21.15.16:0/3833604430 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0c7015b2c0 0x7f0c700c5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:38.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.890+0000 7f0c6f7fe700 1 -- 172.21.15.16:0/3833604430 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0c7006e570 msgr2=0x7f0c7014e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:38.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.890+0000 7f0c6f7fe700 1 --2- 172.21.15.16:0/3833604430 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0c7006e570 0x7f0c7014e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:38.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.890+0000 7f0c6f7fe700 1 -- 172.21.15.16:0/3833604430 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0c7015b2c0 msgr2=0x7f0c700c5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:38.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.890+0000 7f0c6f7fe700 1 --2- 172.21.15.16:0/3833604430 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0c7015b2c0 0x7f0c700c5e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:38.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.890+0000 7f0c6f7fe700 1 -- 172.21.15.16:0/3833604430 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c60009550 con 0x7f0c7006a7e0 2023-10-16T12:27:38.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.890+0000 7f0c6f7fe700 1 --2- 172.21.15.16:0/3833604430 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c7006a7e0 0x7f0c700ca180 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f0c5800c4e0 tx=0x7f0c5800c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:38.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.891+0000 7f0c67fff700 1 -- 172.21.15.16:0/3833604430 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c580052d0 con 0x7f0c7006a7e0 2023-10-16T12:27:38.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.891+0000 7f0c67fff700 1 -- 172.21.15.16:0/3833604430 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c5800ce90 con 0x7f0c7006a7e0 2023-10-16T12:27:38.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.891+0000 7f0c75da7700 1 -- 172.21.15.16:0/3833604430 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c700c4b70 con 0x7f0c7006a7e0 2023-10-16T12:27:38.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.891+0000 7f0c67fff700 1 -- 172.21.15.16:0/3833604430 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c58003680 con 0x7f0c7006a7e0 2023-10-16T12:27:38.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.891+0000 7f0c75da7700 1 -- 172.21.15.16:0/3833604430 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c700c5130 con 0x7f0c7006a7e0 2023-10-16T12:27:38.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.892+0000 7f0c75da7700 1 -- 172.21.15.16:0/3833604430 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c7004ebb0 con 0x7f0c7006a7e0 2023-10-16T12:27:38.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.892+0000 7f0c67fff700 1 -- 172.21.15.16:0/3833604430 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 51) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0c580037e0 con 0x7f0c7006a7e0 2023-10-16T12:27:38.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.892+0000 7f0c67fff700 1 --2- 172.21.15.16:0/3833604430 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f0c5006b020 0x7f0c5006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:38.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.893+0000 7f0c67fff700 1 -- 172.21.15.16:0/3833604430 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f0c58088c90 con 0x7f0c7006a7e0 2023-10-16T12:27:38.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.893+0000 7f0c6effd700 1 --2- 172.21.15.16:0/3833604430 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f0c5006b020 0x7f0c5006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:38.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.893+0000 7f0c6effd700 1 --2- 172.21.15.16:0/3833604430 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f0c5006b020 0x7f0c5006d4e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0c60009c40 tx=0x7f0c60008030 comp rx=0 tx=0).ready entity=mgr.6365 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:38.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:38.898+0000 7f0c67fff700 1 -- 172.21.15.16:0/3833604430 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0c58053a50 con 0x7f0c7006a7e0 2023-10-16T12:27:39.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:39.153+0000 7f0c75da7700 1 -- 172.21.15.16:0/3833604430 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f0c7015dd20 con 0x7f0c7006a7e0 2023-10-16T12:27:39.970 INFO:tasks.ceph.mds.b.smithi049.stderr:2023-10-16T12:27:39.968+0000 7f6ea4143700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T12:27:40.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:40.007+0000 7f0c67fff700 1 -- 172.21.15.16:0/3833604430 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v52) v1 ==== 118+0+0 (secure 0 0 0) 0x7f0c58057070 con 0x7f0c7006a7e0 2023-10-16T12:27:40.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:40.010+0000 7f0c75da7700 1 -- 172.21.15.16:0/3833604430 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f0c5006b020 msgr2=0x7f0c5006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:40.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:40.010+0000 7f0c75da7700 1 --2- 172.21.15.16:0/3833604430 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f0c5006b020 0x7f0c5006d4e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0c60009c40 tx=0x7f0c60008030 comp rx=0 tx=0).stop 2023-10-16T12:27:40.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:40.010+0000 7f0c75da7700 1 -- 172.21.15.16:0/3833604430 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c7006a7e0 msgr2=0x7f0c700ca180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:40.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:40.010+0000 7f0c75da7700 1 --2- 172.21.15.16:0/3833604430 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c7006a7e0 0x7f0c700ca180 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f0c5800c4e0 tx=0x7f0c5800c7f0 comp rx=0 tx=0).stop 2023-10-16T12:27:40.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:40.010+0000 7f0c75da7700 1 -- 172.21.15.16:0/3833604430 shutdown_connections 2023-10-16T12:27:40.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:40.010+0000 7f0c75da7700 1 --2- 172.21.15.16:0/3833604430 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f0c5006b020 0x7f0c5006d4e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:40.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:40.010+0000 7f0c75da7700 1 --2- 172.21.15.16:0/3833604430 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0c7015b2c0 0x7f0c700c5e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:40.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:40.010+0000 7f0c75da7700 1 --2- 172.21.15.16:0/3833604430 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c7006a7e0 0x7f0c700ca180 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:40.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:40.010+0000 7f0c75da7700 1 --2- 172.21.15.16:0/3833604430 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0c7006e570 0x7f0c7014e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:40.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:40.010+0000 7f0c75da7700 1 -- 172.21.15.16:0/3833604430 >> 172.21.15.16:0/3833604430 conn(0x7f0c700b1f40 msgr2=0x7f0c7015cf70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:40.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:40.010+0000 7f0c75da7700 1 -- 172.21.15.16:0/3833604430 shutdown_connections 2023-10-16T12:27:40.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:40.010+0000 7f0c75da7700 1 -- 172.21.15.16:0/3833604430 wait complete. 2023-10-16T12:27:40.027 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-16T12:27:40.028 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-16T12:27:40.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:40.161+0000 7f6e09630700 1 Processor -- start 2023-10-16T12:27:40.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:40.162+0000 7f6e09630700 1 -- start start 2023-10-16T12:27:40.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:40.162+0000 7f6e09630700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6e0415d8d0 0x7f6e040b2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:40.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:40.162+0000 7f6e09630700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f6e040b8fa0 0x7f6e040b6720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:40.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:40.162+0000 7f6e09630700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e040b2f30 con 0x7f6e040b6c60 2023-10-16T12:27:40.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:40.162+0000 7f6e09630700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e040d0bb0 con 0x7f6e040b8fa0 2023-10-16T12:27:40.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:40.162+0000 7f6e09630700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e0415dcb0 con 0x7f6e0415d8d0 2023-10-16T12:27:40.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:40.163+0000 7f6e027fc700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f6e040b8fa0 0x7f6e040b6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:40.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:40.163+0000 7f6e02ffd700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6e0415d8d0 0x7f6e040b2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:40.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:40.163+0000 7f6e027fc700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f6e040b8fa0 0x7f6e040b6720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:41358/0 (socket says 172.21.15.16:41358) 2023-10-16T12:27:40.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:40.163+0000 7f6e027fc700 1 -- 172.21.15.16:0/1424038672 learned_addr learned my addr 172.21.15.16:0/1424038672 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:40.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:40.163+0000 7f6e037fe700 1 --1- 172.21.15.16:0/1424038672 >> v1:172.21.15.49:6789/0 conn(0x7f6e040b6c60 0x7f6e040ce350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:38756/0 (socket says 172.21.15.16:38756) 2023-10-16T12:27:40.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:40.163+0000 7f6e02ffd700 1 -- 172.21.15.16:0/1424038672 >> v2:172.21.15.49:3300/0 conn(0x7f6e040b8fa0 msgr2=0x7f6e040b6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:40.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:40.163+0000 7f6e02ffd700 1 --2- 172.21.15.16:0/1424038672 >> v2:172.21.15.49:3300/0 conn(0x7f6e040b8fa0 0x7f6e040b6720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:40.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:40.163+0000 7f6e02ffd700 1 -- 172.21.15.16:0/1424038672 >> v1:172.21.15.49:6789/0 conn(0x7f6e040b6c60 legacy=0x7f6e040ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:40.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:40.163+0000 7f6e02ffd700 1 -- 172.21.15.16:0/1424038672 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e040d1c40 con 0x7f6e0415d8d0 2023-10-16T12:27:40.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:40.163+0000 7f6e027fc700 1 --2- 172.21.15.16:0/1424038672 >> v2:172.21.15.49:3300/0 conn(0x7f6e040b8fa0 0x7f6e040b6720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:27:40.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:40.163+0000 7f6e02ffd700 1 --2- 172.21.15.16:0/1424038672 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6e0415d8d0 0x7f6e040b2830 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f6df8009840 tx=0x7f6df8009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb3b3561eb72682a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:40.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:40.163+0000 7f6e01ffb700 1 -- 172.21.15.16:0/1424038672 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6df8011830 con 0x7f6e0415d8d0 2023-10-16T12:27:40.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:40.164+0000 7f6e01ffb700 1 -- 172.21.15.16:0/1424038672 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6df8011990 con 0x7f6e0415d8d0 2023-10-16T12:27:40.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:40.164+0000 7f6e01ffb700 1 -- 172.21.15.16:0/1424038672 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6df8011d20 con 0x7f6e0415d8d0 2023-10-16T12:27:40.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:40.164+0000 7f6e09630700 1 -- 172.21.15.16:0/1424038672 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6e0415d8d0 msgr2=0x7f6e040b2830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:40.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:40.164+0000 7f6e09630700 1 --2- 172.21.15.16:0/1424038672 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6e0415d8d0 0x7f6e040b2830 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f6df8009840 tx=0x7f6df8009b50 comp rx=0 tx=0).stop 2023-10-16T12:27:40.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:40.164+0000 7f6e09630700 1 -- 172.21.15.16:0/1424038672 shutdown_connections 2023-10-16T12:27:40.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:40.164+0000 7f6e09630700 1 --2- 172.21.15.16:0/1424038672 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6e0415d8d0 0x7f6e040b2830 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:40.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:40.164+0000 7f6e09630700 1 --2- 172.21.15.16:0/1424038672 >> v2:172.21.15.49:3300/0 conn(0x7f6e040b8fa0 0x7f6e040b6720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:40.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:40.164+0000 7f6e09630700 1 -- 172.21.15.16:0/1424038672 >> 172.21.15.16:0/1424038672 conn(0x7f6e04072330 msgr2=0x7f6e04072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:40.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:40.164+0000 7f6e09630700 1 -- 172.21.15.16:0/1424038672 shutdown_connections 2023-10-16T12:27:40.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:40.164+0000 7f6e09630700 1 -- 172.21.15.16:0/1424038672 wait complete. 2023-10-16T12:27:40.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:40.165+0000 7f6e09630700 1 Processor -- start 2023-10-16T12:27:40.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:40.165+0000 7f6e09630700 1 -- start start 2023-10-16T12:27:40.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:40.166+0000 7f6e09630700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e040b6c60 0x7f6e040b46c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:40.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:40.166+0000 7f6e09630700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6e040b8fa0 0x7f6e040cc6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:40.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:40.166+0000 7f6e09630700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6e0415d8d0 0x7f6e04155660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:40.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:40.166+0000 7f6e09630700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e040ccc00 con 0x7f6e040b6c60 2023-10-16T12:27:40.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:40.166+0000 7f6e09630700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e04155ba0 con 0x7f6e0415d8d0 2023-10-16T12:27:40.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:40.166+0000 7f6e09630700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e041566e0 con 0x7f6e040b8fa0 2023-10-16T12:27:40.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:40.166+0000 7f6e02ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e040b6c60 0x7f6e040b46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:40.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:40.166+0000 7f6e02ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e040b6c60 0x7f6e040b46c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42302/0 (socket says 172.21.15.16:42302) 2023-10-16T12:27:40.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:40.166+0000 7f6e02ffd700 1 -- 172.21.15.16:0/1710944539 learned_addr learned my addr 172.21.15.16:0/1710944539 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:40.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:40.166+0000 7f6e037fe700 1 --2- 172.21.15.16:0/1710944539 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6e0415d8d0 0x7f6e04155660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:40.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:40.166+0000 7f6e027fc700 1 --2- 172.21.15.16:0/1710944539 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6e040b8fa0 0x7f6e040cc6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:40.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:40.167+0000 7f6e02ffd700 1 -- 172.21.15.16:0/1710944539 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6e040b8fa0 msgr2=0x7f6e040cc6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:40.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:40.167+0000 7f6e02ffd700 1 --2- 172.21.15.16:0/1710944539 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6e040b8fa0 0x7f6e040cc6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:40.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:40.167+0000 7f6e02ffd700 1 -- 172.21.15.16:0/1710944539 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6e0415d8d0 msgr2=0x7f6e04155660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:40.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:40.167+0000 7f6e02ffd700 1 --2- 172.21.15.16:0/1710944539 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6e0415d8d0 0x7f6e04155660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:40.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:40.167+0000 7f6e02ffd700 1 -- 172.21.15.16:0/1710944539 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6df4009550 con 0x7f6e040b6c60 2023-10-16T12:27:40.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:40.167+0000 7f6e02ffd700 1 --2- 172.21.15.16:0/1710944539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e040b6c60 0x7f6e040b46c0 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f6df8009b90 tx=0x7f6df8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:40.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:40.167+0000 7f6deb7fe700 1 -- 172.21.15.16:0/1710944539 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6df8004590 con 0x7f6e040b6c60 2023-10-16T12:27:40.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:40.167+0000 7f6deb7fe700 1 -- 172.21.15.16:0/1710944539 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6df80036a0 con 0x7f6e040b6c60 2023-10-16T12:27:40.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:40.167+0000 7f6deb7fe700 1 -- 172.21.15.16:0/1710944539 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6df8003a30 con 0x7f6e040b6c60 2023-10-16T12:27:40.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:40.167+0000 7f6e09630700 1 -- 172.21.15.16:0/1710944539 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6df80094a0 con 0x7f6e040b6c60 2023-10-16T12:27:40.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:40.167+0000 7f6e09630700 1 -- 172.21.15.16:0/1710944539 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e04156120 con 0x7f6e040b6c60 2023-10-16T12:27:40.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:40.168+0000 7f6e09630700 1 -- 172.21.15.16:0/1710944539 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e0404ebb0 con 0x7f6e040b6c60 2023-10-16T12:27:40.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:40.168+0000 7f6deb7fe700 1 -- 172.21.15.16:0/1710944539 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 51) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6df8003b90 con 0x7f6e040b6c60 2023-10-16T12:27:40.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:40.169+0000 7f6deb7fe700 1 --2- 172.21.15.16:0/1710944539 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f6df006b020 0x7f6df006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:40.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:40.169+0000 7f6e027fc700 1 --2- 172.21.15.16:0/1710944539 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f6df006b020 0x7f6df006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:40.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:40.169+0000 7f6deb7fe700 1 -- 172.21.15.16:0/1710944539 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f6df808abd0 con 0x7f6e040b6c60 2023-10-16T12:27:40.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:40.169+0000 7f6e027fc700 1 --2- 172.21.15.16:0/1710944539 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f6df006b020 0x7f6df006d4e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6e04153060 tx=0x7f6df40092c0 comp rx=0 tx=0).ready entity=mgr.6365 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:40.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:40.174+0000 7f6deb7fe700 1 -- 172.21.15.16:0/1710944539 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6df8055a20 con 0x7f6e040b6c60 2023-10-16T12:27:40.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:40.427+0000 7f6e09630700 1 -- 172.21.15.16:0/1710944539 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f6e0406a7e0 con 0x7f6e040b6c60 2023-10-16T12:27:41.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.012+0000 7f6deb7fe700 1 -- 172.21.15.16:0/1710944539 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v53) v1 ==== 122+0+0 (secure 0 0 0) 0x7f6df8059040 con 0x7f6e040b6c60 2023-10-16T12:27:41.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.015+0000 7f6e09630700 1 -- 172.21.15.16:0/1710944539 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f6df006b020 msgr2=0x7f6df006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:41.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.015+0000 7f6e09630700 1 --2- 172.21.15.16:0/1710944539 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f6df006b020 0x7f6df006d4e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6e04153060 tx=0x7f6df40092c0 comp rx=0 tx=0).stop 2023-10-16T12:27:41.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.015+0000 7f6e09630700 1 -- 172.21.15.16:0/1710944539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e040b6c60 msgr2=0x7f6e040b46c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:41.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.015+0000 7f6e09630700 1 --2- 172.21.15.16:0/1710944539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e040b6c60 0x7f6e040b46c0 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f6df8009b90 tx=0x7f6df8004750 comp rx=0 tx=0).stop 2023-10-16T12:27:41.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.015+0000 7f6e09630700 1 -- 172.21.15.16:0/1710944539 shutdown_connections 2023-10-16T12:27:41.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.015+0000 7f6e09630700 1 --2- 172.21.15.16:0/1710944539 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f6df006b020 0x7f6df006d4e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:41.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.015+0000 7f6e09630700 1 --2- 172.21.15.16:0/1710944539 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6e0415d8d0 0x7f6e04155660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:41.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.015+0000 7f6e09630700 1 --2- 172.21.15.16:0/1710944539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e040b6c60 0x7f6e040b46c0 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:41.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.015+0000 7f6e09630700 1 --2- 172.21.15.16:0/1710944539 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6e040b8fa0 0x7f6e040cc6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:41.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.015+0000 7f6e09630700 1 -- 172.21.15.16:0/1710944539 >> 172.21.15.16:0/1710944539 conn(0x7f6e04072330 msgr2=0x7f6e040d28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:41.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.015+0000 7f6e09630700 1 -- 172.21.15.16:0/1710944539 shutdown_connections 2023-10-16T12:27:41.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.015+0000 7f6e09630700 1 -- 172.21.15.16:0/1710944539 wait complete. 2023-10-16T12:27:41.032 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-16T12:27:41.033 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:27:41.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.163+0000 7fc7a32c8700 1 Processor -- start 2023-10-16T12:27:41.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.163+0000 7fc7a32c8700 1 -- start start 2023-10-16T12:27:41.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.164+0000 7fc7a32c8700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fc79c06a7e0 0x7fc79c06ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:41.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.164+0000 7fc7a32c8700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc79c06b150 con 0x7fc79c0cc8a0 2023-10-16T12:27:41.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.164+0000 7fc7a32c8700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc79c161c30 con 0x7fc79c15b280 2023-10-16T12:27:41.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.164+0000 7fc7a32c8700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc79c161f10 con 0x7fc79c06a7e0 2023-10-16T12:27:41.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.164+0000 7fc7a0863700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc79c0cc8a0 0x7fc79c15e700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59816/0 (socket says 172.21.15.16:59816) 2023-10-16T12:27:41.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.164+0000 7fc7a0863700 1 -- 172.21.15.16:0/1715288720 learned_addr learned my addr 172.21.15.16:0/1715288720 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:27:41.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.164+0000 7fc7a1865700 1 --2- 172.21.15.16:0/1715288720 >> v2:172.21.15.49:3300/0 conn(0x7fc79c06a7e0 0x7fc79c06ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:41.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.164+0000 7fc793fff700 1 -- 172.21.15.16:0/1715288720 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2765820094 0 0) 0x7fc79c06b150 con 0x7fc79c0cc8a0 2023-10-16T12:27:41.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.164+0000 7fc793fff700 1 -- 172.21.15.16:0/1715288720 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7880035b0 con 0x7fc79c0cc8a0 2023-10-16T12:27:41.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.164+0000 7fc793fff700 1 -- 172.21.15.16:0/1715288720 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 697903996 0 0) 0x7fc7880035b0 con 0x7fc79c0cc8a0 2023-10-16T12:27:41.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.164+0000 7fc793fff700 1 -- 172.21.15.16:0/1715288720 >> v2:172.21.15.49:3300/0 conn(0x7fc79c06a7e0 msgr2=0x7fc79c06ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:41.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.164+0000 7fc793fff700 1 --2- 172.21.15.16:0/1715288720 >> v2:172.21.15.49:3300/0 conn(0x7fc79c06a7e0 0x7fc79c06ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:41.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.164+0000 7fc793fff700 1 -- 172.21.15.16:0/1715288720 >> v1:172.21.15.49:6789/0 conn(0x7fc79c15b280 legacy=0x7fc79c1613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:41.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.164+0000 7fc793fff700 1 -- 172.21.15.16:0/1715288720 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc79c1621f0 con 0x7fc79c0cc8a0 2023-10-16T12:27:41.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.164+0000 7fc793fff700 1 -- 172.21.15.16:0/1715288720 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7fc78c001670 con 0x7fc79c0cc8a0 2023-10-16T12:27:41.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.164+0000 7fc793fff700 1 -- 172.21.15.16:0/1715288720 >> v1:172.21.15.16:6789/0 conn(0x7fc79c0cc8a0 legacy=0x7fc79c15e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:41.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.164+0000 7fc793fff700 1 --2- 172.21.15.16:0/1715288720 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc788003a00 0x7fc788003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:41.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.164+0000 7fc793fff700 1 --2- 172.21.15.16:0/1715288720 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc788004850 0x7fc788004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:41.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.164+0000 7fc793fff700 1 --2- 172.21.15.16:0/1715288720 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc788005210 0x7fc78800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:41.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.164+0000 7fc793fff700 1 -- 172.21.15.16:0/1715288720 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc79c161f10 con 0x7fc788005210 2023-10-16T12:27:41.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.164+0000 7fc793fff700 1 -- 172.21.15.16:0/1715288720 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc78800c0b0 con 0x7fc788004850 2023-10-16T12:27:41.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.164+0000 7fc793fff700 1 -- 172.21.15.16:0/1715288720 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc78800c270 con 0x7fc788003a00 2023-10-16T12:27:41.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.165+0000 7fc7a1865700 1 --2- 172.21.15.16:0/1715288720 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc788005210 0x7fc78800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:41.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.165+0000 7fc7a0863700 1 --2- 172.21.15.16:0/1715288720 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc788004850 0x7fc788004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:41.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.165+0000 7fc7a1064700 1 --2- 172.21.15.16:0/1715288720 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc788003a00 0x7fc788003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:41.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.165+0000 7fc7a1865700 1 -- 172.21.15.16:0/1715288720 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc788003a00 msgr2=0x7fc788003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:41.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.165+0000 7fc7a1865700 1 --2- 172.21.15.16:0/1715288720 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc788003a00 0x7fc788003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:41.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.165+0000 7fc7a1865700 1 -- 172.21.15.16:0/1715288720 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc788004850 msgr2=0x7fc788004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:41.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.165+0000 7fc7a1865700 1 --2- 172.21.15.16:0/1715288720 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc788004850 0x7fc788004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:41.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.165+0000 7fc7a1865700 1 -- 172.21.15.16:0/1715288720 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7880035b0 con 0x7fc788005210 2023-10-16T12:27:41.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.165+0000 7fc7a1865700 1 -- 172.21.15.16:0/1715288720 reap_dead start 2023-10-16T12:27:41.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.165+0000 7fc7a1865700 1 --2- 172.21.15.16:0/1715288720 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc788005210 0x7fc78800ba10 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7fc794009fc0 tx=0x7fc788005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=218efbd9de35bf17 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:41.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.165+0000 7fc793fff700 1 -- 172.21.15.16:0/1715288720 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc794003a20 con 0x7fc788005210 2023-10-16T12:27:41.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.165+0000 7fc793fff700 1 -- 172.21.15.16:0/1715288720 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc79400f040 con 0x7fc788005210 2023-10-16T12:27:41.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.165+0000 7fc793fff700 1 -- 172.21.15.16:0/1715288720 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc794003de0 con 0x7fc788005210 2023-10-16T12:27:41.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.166+0000 7fc7a32c8700 1 -- 172.21.15.16:0/1715288720 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc788005210 msgr2=0x7fc78800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:41.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.166+0000 7fc7a32c8700 1 --2- 172.21.15.16:0/1715288720 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc788005210 0x7fc78800ba10 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7fc794009fc0 tx=0x7fc788005880 comp rx=0 tx=0).stop 2023-10-16T12:27:41.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.166+0000 7fc7a32c8700 1 -- 172.21.15.16:0/1715288720 shutdown_connections 2023-10-16T12:27:41.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.166+0000 7fc7a32c8700 1 --2- 172.21.15.16:0/1715288720 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc788005210 0x7fc78800ba10 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:41.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.166+0000 7fc7a32c8700 1 -- 172.21.15.16:0/1715288720 >> 172.21.15.16:0/1715288720 conn(0x7fc79c0b1fd0 msgr2=0x7fc79c0b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:41.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.166+0000 7fc7a32c8700 1 -- 172.21.15.16:0/1715288720 shutdown_connections 2023-10-16T12:27:41.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.166+0000 7fc7a32c8700 1 -- 172.21.15.16:0/1715288720 wait complete. 2023-10-16T12:27:41.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.167+0000 7fc7a32c8700 1 Processor -- start 2023-10-16T12:27:41.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.167+0000 7fc7a32c8700 1 -- start start 2023-10-16T12:27:41.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.168+0000 7fc7a32c8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc788005210 0x7fc79c150a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:41.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.168+0000 7fc7a32c8700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc79c0c3f70 0x7fc79c0c9c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:41.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.168+0000 7fc7a32c8700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc79c0ca160 0x7fc79c0c7d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:41.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.168+0000 7fc7a32c8700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc79c0c43f0 con 0x7fc788005210 2023-10-16T12:27:41.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.168+0000 7fc7a32c8700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc79c1571a0 con 0x7fc79c0ca160 2023-10-16T12:27:41.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.168+0000 7fc7a32c8700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc79c157920 con 0x7fc79c0c3f70 2023-10-16T12:27:41.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.168+0000 7fc7a1064700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc788005210 0x7fc79c150a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:41.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.168+0000 7fc7a1064700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc788005210 0x7fc79c150a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42322/0 (socket says 172.21.15.16:42322) 2023-10-16T12:27:41.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.168+0000 7fc7a1064700 1 -- 172.21.15.16:0/906013675 learned_addr learned my addr 172.21.15.16:0/906013675 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:41.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.168+0000 7fc7a1865700 1 --2- 172.21.15.16:0/906013675 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc79c0ca160 0x7fc79c0c7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:41.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.168+0000 7fc7a0863700 1 --2- 172.21.15.16:0/906013675 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc79c0c3f70 0x7fc79c0c9c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:41.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.168+0000 7fc7a1064700 1 -- 172.21.15.16:0/906013675 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc79c0c3f70 msgr2=0x7fc79c0c9c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:41.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.168+0000 7fc7a1064700 1 --2- 172.21.15.16:0/906013675 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc79c0c3f70 0x7fc79c0c9c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:41.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.168+0000 7fc7a1064700 1 -- 172.21.15.16:0/906013675 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc79c0ca160 msgr2=0x7fc79c0c7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:41.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.168+0000 7fc7a1064700 1 --2- 172.21.15.16:0/906013675 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc79c0ca160 0x7fc79c0c7d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:41.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.168+0000 7fc7a1064700 1 -- 172.21.15.16:0/906013675 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc79c157c00 con 0x7fc788005210 2023-10-16T12:27:41.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.168+0000 7fc7a1865700 1 --2- 172.21.15.16:0/906013675 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc79c0ca160 0x7fc79c0c7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:27:41.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.168+0000 7fc7a1064700 1 --2- 172.21.15.16:0/906013675 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc788005210 0x7fc79c150a70 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7fc79800f820 tx=0x7fc79800fb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:41.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.169+0000 7fc791ffb700 1 -- 172.21.15.16:0/906013675 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7980052e0 con 0x7fc788005210 2023-10-16T12:27:41.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.169+0000 7fc791ffb700 1 -- 172.21.15.16:0/906013675 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc798005440 con 0x7fc788005210 2023-10-16T12:27:41.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.169+0000 7fc791ffb700 1 -- 172.21.15.16:0/906013675 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7980038c0 con 0x7fc788005210 2023-10-16T12:27:41.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.169+0000 7fc7a32c8700 1 -- 172.21.15.16:0/906013675 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc79c157ee0 con 0x7fc788005210 2023-10-16T12:27:41.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.169+0000 7fc7a32c8700 1 -- 172.21.15.16:0/906013675 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc79c1584a0 con 0x7fc788005210 2023-10-16T12:27:41.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.170+0000 7fc7a32c8700 1 -- 172.21.15.16:0/906013675 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc79c158810 con 0x7fc788005210 2023-10-16T12:27:41.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.170+0000 7fc791ffb700 1 -- 172.21.15.16:0/906013675 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 51) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc798003a20 con 0x7fc788005210 2023-10-16T12:27:41.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.171+0000 7fc791ffb700 1 --2- 172.21.15.16:0/906013675 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fc78006b020 0x7fc78006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:41.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.171+0000 7fc791ffb700 1 -- 172.21.15.16:0/906013675 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fc798089710 con 0x7fc788005210 2023-10-16T12:27:41.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.171+0000 7fc7a0863700 1 --2- 172.21.15.16:0/906013675 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fc78006b020 0x7fc78006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:41.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.171+0000 7fc7a0863700 1 --2- 172.21.15.16:0/906013675 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fc78006b020 0x7fc78006d4e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc79c159e00 tx=0x7fc78c000ed0 comp rx=0 tx=0).ready entity=mgr.6365 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:41.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.175+0000 7fc791ffb700 1 -- 172.21.15.16:0/906013675 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc798054420 con 0x7fc788005210 2023-10-16T12:27:41.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.424+0000 7fc7a32c8700 1 -- 172.21.15.16:0/906013675 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc79c158630 con 0x7fc788005210 2023-10-16T12:27:41.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.425+0000 7fc791ffb700 1 -- 172.21.15.16:0/906013675 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+4903 (secure 0 0 0) 0x7fc798087e60 con 0x7fc788005210 2023-10-16T12:27:41.427 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:27:41.427 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":53,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6524,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/854923476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":854923476},{"type":"v1","addr":"172.21.15.49:6834","nonce":854923476}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":7249,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/1186699359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1186699359},{"type":"v1","addr":"172.21.15.16:6836","nonce":1186699359}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":7252,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/22289021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":22289021},{"type":"v1","addr":"172.21.15.16:6837","nonce":22289021}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:27:37.397811+0000","modified":"2023-10-16T12:27:40.428748+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7258},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7258":{"gid":7258,"name":"b","rank":0,"incarnation":49,"state":"up:active","state_seq":3,"addr":"172.21.15.49:6835/729289774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":729289774},{"type":"v1","addr":"172.21.15.49:6835","nonce":729289774}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-10-16T12:27:41.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.427+0000 7fc7a32c8700 1 -- 172.21.15.16:0/906013675 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fc78006b020 msgr2=0x7fc78006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:41.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.427+0000 7fc7a32c8700 1 --2- 172.21.15.16:0/906013675 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fc78006b020 0x7fc78006d4e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc79c159e00 tx=0x7fc78c000ed0 comp rx=0 tx=0).stop 2023-10-16T12:27:41.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.427+0000 7fc7a32c8700 1 -- 172.21.15.16:0/906013675 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc788005210 msgr2=0x7fc79c150a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:41.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.427+0000 7fc7a32c8700 1 --2- 172.21.15.16:0/906013675 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc788005210 0x7fc79c150a70 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7fc79800f820 tx=0x7fc79800fb30 comp rx=0 tx=0).stop 2023-10-16T12:27:41.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.427+0000 7fc7a32c8700 1 -- 172.21.15.16:0/906013675 shutdown_connections 2023-10-16T12:27:41.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.427+0000 7fc7a32c8700 1 --2- 172.21.15.16:0/906013675 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fc78006b020 0x7fc78006d4e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:41.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.427+0000 7fc7a32c8700 1 --2- 172.21.15.16:0/906013675 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc79c0ca160 0x7fc79c0c7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:41.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.427+0000 7fc7a32c8700 1 --2- 172.21.15.16:0/906013675 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc788005210 0x7fc79c150a70 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:41.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.427+0000 7fc7a32c8700 1 --2- 172.21.15.16:0/906013675 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc79c0c3f70 0x7fc79c0c9c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:41.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.427+0000 7fc7a32c8700 1 -- 172.21.15.16:0/906013675 >> 172.21.15.16:0/906013675 conn(0x7fc79c0b1fd0 msgr2=0x7fc79c0721f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:41.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.427+0000 7fc7a32c8700 1 -- 172.21.15.16:0/906013675 shutdown_connections 2023-10-16T12:27:41.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.427+0000 7fc7a32c8700 1 -- 172.21.15.16:0/906013675 wait complete. 2023-10-16T12:27:41.432 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 53 2023-10-16T12:27:41.441 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:27:41.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.564+0000 7fc420e5d700 1 Processor -- start 2023-10-16T12:27:41.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.564+0000 7fc420e5d700 1 -- start start 2023-10-16T12:27:41.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.565+0000 7fc420e5d700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fc41c0cf4a0 0x7fc41c0726e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:41.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.565+0000 7fc420e5d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc41c0b5a00 0x7fc41c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:41.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.565+0000 7fc420e5d700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc41c15bc80 con 0x7fc41c0cf880 2023-10-16T12:27:41.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.565+0000 7fc420e5d700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc41c0b5e40 con 0x7fc41c0cf4a0 2023-10-16T12:27:41.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.565+0000 7fc420e5d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc41c072c20 con 0x7fc41c0b5a00 2023-10-16T12:27:41.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.566+0000 7fc41a59c700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fc41c0cf4a0 0x7fc41c0726e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:41.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.566+0000 7fc41ad9d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc41c0b5a00 0x7fc41c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:41.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.566+0000 7fc41a59c700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fc41c0cf4a0 0x7fc41c0726e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:41392/0 (socket says 172.21.15.16:41392) 2023-10-16T12:27:41.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.566+0000 7fc41ad9d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc41c0b5a00 0x7fc41c15a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:58824/0 (socket says 172.21.15.16:58824) 2023-10-16T12:27:41.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.566+0000 7fc41a59c700 1 -- 172.21.15.16:0/332142150 learned_addr learned my addr 172.21.15.16:0/332142150 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:41.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.566+0000 7fc419d9b700 1 --1- 172.21.15.16:0/332142150 >> v1:172.21.15.49:6789/0 conn(0x7fc41c0cf880 0x7fc41c0b52f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:38786/0 (socket says 172.21.15.16:38786) 2023-10-16T12:27:41.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.566+0000 7fc41ad9d700 1 -- 172.21.15.16:0/332142150 >> v2:172.21.15.49:3300/0 conn(0x7fc41c0cf4a0 msgr2=0x7fc41c0726e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:41.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.566+0000 7fc41ad9d700 1 --2- 172.21.15.16:0/332142150 >> v2:172.21.15.49:3300/0 conn(0x7fc41c0cf4a0 0x7fc41c0726e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:41.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.566+0000 7fc41ad9d700 1 -- 172.21.15.16:0/332142150 >> v1:172.21.15.49:6789/0 conn(0x7fc41c0cf880 legacy=0x7fc41c0b52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:41.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.566+0000 7fc41ad9d700 1 -- 172.21.15.16:0/332142150 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc41c15b070 con 0x7fc41c0b5a00 2023-10-16T12:27:41.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.566+0000 7fc41a59c700 1 --2- 172.21.15.16:0/332142150 >> v2:172.21.15.49:3300/0 conn(0x7fc41c0cf4a0 0x7fc41c0726e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:27:41.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.566+0000 7fc41ad9d700 1 --2- 172.21.15.16:0/332142150 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc41c0b5a00 0x7fc41c15a090 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fc414009d90 tx=0x7fc41400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac4d7c8f75262de9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:41.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.566+0000 7fc41959a700 1 -- 172.21.15.16:0/332142150 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc41400e070 con 0x7fc41c0b5a00 2023-10-16T12:27:41.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.566+0000 7fc41959a700 1 -- 172.21.15.16:0/332142150 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc41400bb20 con 0x7fc41c0b5a00 2023-10-16T12:27:41.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.566+0000 7fc41959a700 1 -- 172.21.15.16:0/332142150 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc414012430 con 0x7fc41c0b5a00 2023-10-16T12:27:41.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.567+0000 7fc420e5d700 1 -- 172.21.15.16:0/332142150 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc41c0b5a00 msgr2=0x7fc41c15a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:41.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.567+0000 7fc420e5d700 1 --2- 172.21.15.16:0/332142150 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc41c0b5a00 0x7fc41c15a090 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fc414009d90 tx=0x7fc41400b330 comp rx=0 tx=0).stop 2023-10-16T12:27:41.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.567+0000 7fc420e5d700 1 -- 172.21.15.16:0/332142150 shutdown_connections 2023-10-16T12:27:41.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.567+0000 7fc420e5d700 1 --2- 172.21.15.16:0/332142150 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc41c0b5a00 0x7fc41c15a090 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:41.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.567+0000 7fc420e5d700 1 --2- 172.21.15.16:0/332142150 >> v2:172.21.15.49:3300/0 conn(0x7fc41c0cf4a0 0x7fc41c0726e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:41.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.567+0000 7fc420e5d700 1 -- 172.21.15.16:0/332142150 >> 172.21.15.16:0/332142150 conn(0x7fc41c0b1fd0 msgr2=0x7fc41c0b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:41.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.567+0000 7fc420e5d700 1 -- 172.21.15.16:0/332142150 shutdown_connections 2023-10-16T12:27:41.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.567+0000 7fc420e5d700 1 -- 172.21.15.16:0/332142150 wait complete. 2023-10-16T12:27:41.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.568+0000 7fc420e5d700 1 Processor -- start 2023-10-16T12:27:41.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.568+0000 7fc420e5d700 1 -- start start 2023-10-16T12:27:41.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.569+0000 7fc420e5d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc41c0b5a00 0x7fc41c0cc730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:41.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.569+0000 7fc420e5d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc41c0cf4a0 0x7fc41c0c9b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:41.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.569+0000 7fc420e5d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc41c0cf880 0x7fc41c14d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:41.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.569+0000 7fc420e5d700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc41c0c49f0 con 0x7fc41c0cf4a0 2023-10-16T12:27:41.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.569+0000 7fc420e5d700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc41c0ca620 con 0x7fc41c0cf880 2023-10-16T12:27:41.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.569+0000 7fc420e5d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc41c0c43a0 con 0x7fc41c0b5a00 2023-10-16T12:27:41.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.569+0000 7fc419d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc41c0cf4a0 0x7fc41c0c9b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:41.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.569+0000 7fc419d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc41c0cf4a0 0x7fc41c0c9b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42336/0 (socket says 172.21.15.16:42336) 2023-10-16T12:27:41.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.569+0000 7fc419d9b700 1 -- 172.21.15.16:0/482962626 learned_addr learned my addr 172.21.15.16:0/482962626 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:41.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.569+0000 7fc41ad9d700 1 --2- 172.21.15.16:0/482962626 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc41c0cf880 0x7fc41c14d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:41.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.569+0000 7fc41a59c700 1 --2- 172.21.15.16:0/482962626 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc41c0b5a00 0x7fc41c0cc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:41.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.570+0000 7fc419d9b700 1 -- 172.21.15.16:0/482962626 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc41c0b5a00 msgr2=0x7fc41c0cc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:41.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.570+0000 7fc419d9b700 1 --2- 172.21.15.16:0/482962626 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc41c0b5a00 0x7fc41c0cc730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:41.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.570+0000 7fc419d9b700 1 -- 172.21.15.16:0/482962626 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc41c0cf880 msgr2=0x7fc41c14d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:41.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.570+0000 7fc419d9b700 1 --2- 172.21.15.16:0/482962626 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc41c0cf880 0x7fc41c14d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:41.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.570+0000 7fc419d9b700 1 -- 172.21.15.16:0/482962626 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc410009550 con 0x7fc41c0cf4a0 2023-10-16T12:27:41.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.570+0000 7fc419d9b700 1 --2- 172.21.15.16:0/482962626 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc41c0cf4a0 0x7fc41c0c9b80 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7fc40c00e940 tx=0x7fc40c00ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:41.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.570+0000 7fc40affd700 1 -- 172.21.15.16:0/482962626 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc40c005cb0 con 0x7fc41c0cf4a0 2023-10-16T12:27:41.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.570+0000 7fc40affd700 1 -- 172.21.15.16:0/482962626 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc40c005e10 con 0x7fc41c0cf4a0 2023-10-16T12:27:41.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.570+0000 7fc420e5d700 1 -- 172.21.15.16:0/482962626 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc414009a70 con 0x7fc41c0cf4a0 2023-10-16T12:27:41.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.570+0000 7fc40affd700 1 -- 172.21.15.16:0/482962626 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc40c007270 con 0x7fc41c0cf4a0 2023-10-16T12:27:41.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.570+0000 7fc420e5d700 1 -- 172.21.15.16:0/482962626 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc41c0c6e10 con 0x7fc41c0cf4a0 2023-10-16T12:27:41.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.571+0000 7fc420e5d700 1 -- 172.21.15.16:0/482962626 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc41c04ebb0 con 0x7fc41c0cf4a0 2023-10-16T12:27:41.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.571+0000 7fc40affd700 1 -- 172.21.15.16:0/482962626 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 51) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc40c0073d0 con 0x7fc41c0cf4a0 2023-10-16T12:27:41.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.572+0000 7fc40affd700 1 --2- 172.21.15.16:0/482962626 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fc3fc06b020 0x7fc3fc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:41.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.572+0000 7fc40affd700 1 -- 172.21.15.16:0/482962626 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fc40c08bad0 con 0x7fc41c0cf4a0 2023-10-16T12:27:41.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.572+0000 7fc41a59c700 1 --2- 172.21.15.16:0/482962626 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fc3fc06b020 0x7fc3fc06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:41.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.572+0000 7fc41a59c700 1 --2- 172.21.15.16:0/482962626 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fc3fc06b020 0x7fc3fc06d4e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc41c0c4160 tx=0x7fc4100092c0 comp rx=0 tx=0).ready entity=mgr.6365 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:41.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.577+0000 7fc40affd700 1 -- 172.21.15.16:0/482962626 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc40c056800 con 0x7fc41c0cf4a0 2023-10-16T12:27:41.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.781+0000 7fc420e5d700 1 -- 172.21.15.16:0/482962626 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc41c0c4b30 con 0x7fc41c0cf4a0 2023-10-16T12:27:41.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.782+0000 7fc40affd700 1 -- 172.21.15.16:0/482962626 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v112) v1 ==== 74+0+16663 (secure 0 0 0) 0x7fc40c01f020 con 0x7fc41c0cf4a0 2023-10-16T12:27:41.784 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:27:41.784 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":112,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:27:38.998769+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:27:35.907633+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2023-10-16T12:27:36.362861+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:27:41.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.784+0000 7fc420e5d700 1 -- 172.21.15.16:0/482962626 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fc3fc06b020 msgr2=0x7fc3fc06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:41.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.784+0000 7fc420e5d700 1 --2- 172.21.15.16:0/482962626 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fc3fc06b020 0x7fc3fc06d4e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc41c0c4160 tx=0x7fc4100092c0 comp rx=0 tx=0).stop 2023-10-16T12:27:41.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.784+0000 7fc420e5d700 1 -- 172.21.15.16:0/482962626 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc41c0cf4a0 msgr2=0x7fc41c0c9b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:41.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.784+0000 7fc420e5d700 1 --2- 172.21.15.16:0/482962626 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc41c0cf4a0 0x7fc41c0c9b80 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7fc40c00e940 tx=0x7fc40c00ec50 comp rx=0 tx=0).stop 2023-10-16T12:27:41.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.784+0000 7fc420e5d700 1 -- 172.21.15.16:0/482962626 shutdown_connections 2023-10-16T12:27:41.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.784+0000 7fc420e5d700 1 --2- 172.21.15.16:0/482962626 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fc3fc06b020 0x7fc3fc06d4e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:41.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.784+0000 7fc420e5d700 1 --2- 172.21.15.16:0/482962626 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc41c0cf880 0x7fc41c14d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:41.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.784+0000 7fc420e5d700 1 --2- 172.21.15.16:0/482962626 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc41c0b5a00 0x7fc41c0cc730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:41.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.784+0000 7fc420e5d700 1 --2- 172.21.15.16:0/482962626 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc41c0cf4a0 0x7fc41c0c9b80 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:41.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.784+0000 7fc420e5d700 1 -- 172.21.15.16:0/482962626 >> 172.21.15.16:0/482962626 conn(0x7fc41c0b1fd0 msgr2=0x7fc41c159c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:41.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.784+0000 7fc420e5d700 1 -- 172.21.15.16:0/482962626 shutdown_connections 2023-10-16T12:27:41.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.784+0000 7fc420e5d700 1 -- 172.21.15.16:0/482962626 wait complete. 2023-10-16T12:27:41.799 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-16T12:27:41.799 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T12:27:41.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.922+0000 7f9ce26c4700 1 Processor -- start 2023-10-16T12:27:41.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.922+0000 7f9ce26c4700 1 -- start start 2023-10-16T12:27:41.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.924+0000 7f9ce26c4700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f9cdc15b630 0x7f9cdc0cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:41.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.924+0000 7f9ce26c4700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9cdc15ba10 0x7f9cdc0cdde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:41.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.924+0000 7f9ce26c4700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cdc0ce770 con 0x7f9cdc0ce320 2023-10-16T12:27:41.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.924+0000 7f9ce26c4700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cdc0cdb20 con 0x7f9cdc15b630 2023-10-16T12:27:41.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.924+0000 7f9ce26c4700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cdc0cc2a0 con 0x7f9cdc15ba10 2023-10-16T12:27:41.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.924+0000 7f9ce0c61700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f9cdc0ce320 0x7f9cdc0cfca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59854/0 (socket says 172.21.15.16:59854) 2023-10-16T12:27:41.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.924+0000 7f9ce0c61700 1 -- 172.21.15.16:0/1772946293 learned_addr learned my addr 172.21.15.16:0/1772946293 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:27:41.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.924+0000 7f9cdb7fe700 1 --2- 172.21.15.16:0/1772946293 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9cdc15ba10 0x7f9cdc0cdde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:41.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.924+0000 7f9cdbfff700 1 --2- 172.21.15.16:0/1772946293 >> v2:172.21.15.49:3300/0 conn(0x7f9cdc15b630 0x7f9cdc0cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:41.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.924+0000 7f9cdaffd700 1 -- 172.21.15.16:0/1772946293 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2236051077 0 0) 0x7f9cd4001490 con 0x7f9cdc0ce320 2023-10-16T12:27:41.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.924+0000 7f9cdaffd700 1 -- 172.21.15.16:0/1772946293 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9cc80035b0 con 0x7f9cdc0ce320 2023-10-16T12:27:41.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.924+0000 7f9cdaffd700 1 -- 172.21.15.16:0/1772946293 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3311834015 0 0) 0x7f9cc80035b0 con 0x7f9cdc0ce320 2023-10-16T12:27:41.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.924+0000 7f9cdaffd700 1 -- 172.21.15.16:0/1772946293 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9cdc15ba10 msgr2=0x7f9cdc0cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:41.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.924+0000 7f9cdaffd700 1 --2- 172.21.15.16:0/1772946293 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9cdc15ba10 0x7f9cdc0cdde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:41.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.924+0000 7f9cdaffd700 1 -- 172.21.15.16:0/1772946293 >> v2:172.21.15.49:3300/0 conn(0x7f9cdc15b630 msgr2=0x7f9cdc0cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:41.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.924+0000 7f9cdaffd700 1 --2- 172.21.15.16:0/1772946293 >> v2:172.21.15.49:3300/0 conn(0x7f9cdc15b630 0x7f9cdc0cd5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:41.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.924+0000 7f9cdaffd700 1 -- 172.21.15.16:0/1772946293 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cdc0cc540 con 0x7f9cdc0ce320 2023-10-16T12:27:41.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.924+0000 7f9cdaffd700 1 -- 172.21.15.16:0/1772946293 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f9cd4001b80 con 0x7f9cdc0ce320 2023-10-16T12:27:41.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.924+0000 7f9cdaffd700 1 -- 172.21.15.16:0/1772946293 >> v1:172.21.15.16:6789/0 conn(0x7f9cdc0ce320 legacy=0x7f9cdc0cfca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:41.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.924+0000 7f9cdaffd700 1 --2- 172.21.15.16:0/1772946293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9cc8003a00 0x7f9cc8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:41.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.924+0000 7f9cdaffd700 1 --2- 172.21.15.16:0/1772946293 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9cc8004850 0x7f9cc8004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:41.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.924+0000 7f9cdaffd700 1 --2- 172.21.15.16:0/1772946293 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9cdc15ba10 0x7f9cc8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:41.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.924+0000 7f9cdaffd700 1 -- 172.21.15.16:0/1772946293 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cdc0cc2a0 con 0x7f9cc8003a00 2023-10-16T12:27:41.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.924+0000 7f9cdaffd700 1 -- 172.21.15.16:0/1772946293 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc800ba40 con 0x7f9cc8004850 2023-10-16T12:27:41.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.924+0000 7f9cdaffd700 1 -- 172.21.15.16:0/1772946293 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc800bc00 con 0x7f9cdc15ba10 2023-10-16T12:27:41.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.924+0000 7f9cdbfff700 1 --2- 172.21.15.16:0/1772946293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9cc8003a00 0x7f9cc8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:41.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.925+0000 7f9cdb7fe700 1 --2- 172.21.15.16:0/1772946293 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9cc8004850 0x7f9cc8004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:41.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.925+0000 7f9ce0c61700 1 --2- 172.21.15.16:0/1772946293 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9cdc15ba10 0x7f9cc8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:41.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.925+0000 7f9cdbfff700 1 -- 172.21.15.16:0/1772946293 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9cdc15ba10 msgr2=0x7f9cc8005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:41.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.925+0000 7f9cdbfff700 1 --2- 172.21.15.16:0/1772946293 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9cdc15ba10 0x7f9cc8005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:41.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.925+0000 7f9cdbfff700 1 -- 172.21.15.16:0/1772946293 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9cc8004850 msgr2=0x7f9cc8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:41.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.925+0000 7f9cdbfff700 1 --2- 172.21.15.16:0/1772946293 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9cc8004850 0x7f9cc8004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:41.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.925+0000 7f9cdbfff700 1 -- 172.21.15.16:0/1772946293 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cc80035b0 con 0x7f9cc8003a00 2023-10-16T12:27:41.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.925+0000 7f9cdbfff700 1 --2- 172.21.15.16:0/1772946293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9cc8003a00 0x7f9cc8003e20 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc009b40 tx=0x7f9cc8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ebc923461558f0b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:41.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.925+0000 7f9cdaffd700 1 -- 172.21.15.16:0/1772946293 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ccc005a20 con 0x7f9cc8003a00 2023-10-16T12:27:41.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.925+0000 7f9cdaffd700 1 -- 172.21.15.16:0/1772946293 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ccc005b80 con 0x7f9cc8003a00 2023-10-16T12:27:41.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.925+0000 7f9cdaffd700 1 -- 172.21.15.16:0/1772946293 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ccc00e6c0 con 0x7f9cc8003a00 2023-10-16T12:27:41.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.926+0000 7f9ce26c4700 1 -- 172.21.15.16:0/1772946293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9cc8003a00 msgr2=0x7f9cc8003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:41.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.926+0000 7f9ce26c4700 1 --2- 172.21.15.16:0/1772946293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9cc8003a00 0x7f9cc8003e20 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc009b40 tx=0x7f9cc8005880 comp rx=0 tx=0).stop 2023-10-16T12:27:41.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.926+0000 7f9ce0c61700 1 -- 172.21.15.16:0/1772946293 reap_dead start 2023-10-16T12:27:41.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.926+0000 7f9ce26c4700 1 -- 172.21.15.16:0/1772946293 shutdown_connections 2023-10-16T12:27:41.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.926+0000 7f9ce26c4700 1 -- 172.21.15.16:0/1772946293 >> 172.21.15.16:0/1772946293 conn(0x7f9cdc0d14b0 msgr2=0x7f9cdc0d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:41.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.926+0000 7f9ce26c4700 1 -- 172.21.15.16:0/1772946293 shutdown_connections 2023-10-16T12:27:41.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.926+0000 7f9ce26c4700 1 -- 172.21.15.16:0/1772946293 wait complete. 2023-10-16T12:27:41.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.927+0000 7f9ce26c4700 1 Processor -- start 2023-10-16T12:27:41.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.927+0000 7f9ce26c4700 1 -- start start 2023-10-16T12:27:41.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.928+0000 7f9ce26c4700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9cc8003a00 0x7f9cdc1f2340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:41.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.928+0000 7f9ce26c4700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9cdc155d80 0x7f9cdc06d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:41.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.928+0000 7f9ce26c4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9cdc155020 0x7f9cdc1554a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:41.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.928+0000 7f9ce26c4700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cdc1559e0 con 0x7f9cdc155020 2023-10-16T12:27:41.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.928+0000 7f9ce26c4700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cdc06ce40 con 0x7f9cc8003a00 2023-10-16T12:27:41.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.928+0000 7f9ce26c4700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cdc157bd0 con 0x7f9cdc155d80 2023-10-16T12:27:41.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.928+0000 7f9ce0c61700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9cdc155020 0x7f9cdc1554a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:41.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.928+0000 7f9ce0c61700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9cdc155020 0x7f9cdc1554a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42360/0 (socket says 172.21.15.16:42360) 2023-10-16T12:27:41.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.928+0000 7f9ce0c61700 1 -- 172.21.15.16:0/3484667932 learned_addr learned my addr 172.21.15.16:0/3484667932 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:41.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.928+0000 7f9cdb7fe700 1 --2- 172.21.15.16:0/3484667932 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9cdc155d80 0x7f9cdc06d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:41.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.928+0000 7f9cdbfff700 1 --2- 172.21.15.16:0/3484667932 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9cc8003a00 0x7f9cdc1f2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:41.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.928+0000 7f9ce0c61700 1 -- 172.21.15.16:0/3484667932 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9cdc155d80 msgr2=0x7f9cdc06d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:41.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.928+0000 7f9ce0c61700 1 --2- 172.21.15.16:0/3484667932 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9cdc155d80 0x7f9cdc06d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:41.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.928+0000 7f9ce0c61700 1 -- 172.21.15.16:0/3484667932 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9cc8003a00 msgr2=0x7f9cdc1f2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:41.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.928+0000 7f9ce0c61700 1 --2- 172.21.15.16:0/3484667932 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9cc8003a00 0x7f9cdc1f2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:41.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.928+0000 7f9ce0c61700 1 -- 172.21.15.16:0/3484667932 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cdc157610 con 0x7f9cdc155020 2023-10-16T12:27:41.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.928+0000 7f9cdbfff700 1 --2- 172.21.15.16:0/3484667932 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9cc8003a00 0x7f9cdc1f2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:27:41.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.929+0000 7f9ce0c61700 1 --2- 172.21.15.16:0/3484667932 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9cdc155020 0x7f9cdc1554a0 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc0672d0 tx=0x7f9cd4007340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:41.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.929+0000 7f9cd8ff9700 1 -- 172.21.15.16:0/3484667932 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cd4007ab0 con 0x7f9cdc155020 2023-10-16T12:27:41.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.929+0000 7f9cd8ff9700 1 -- 172.21.15.16:0/3484667932 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cd40133f0 con 0x7f9cdc155020 2023-10-16T12:27:41.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.929+0000 7f9ce26c4700 1 -- 172.21.15.16:0/3484667932 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cdc1578f0 con 0x7f9cdc155020 2023-10-16T12:27:41.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.929+0000 7f9cd8ff9700 1 -- 172.21.15.16:0/3484667932 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cd40137a0 con 0x7f9cdc155020 2023-10-16T12:27:41.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.929+0000 7f9ce26c4700 1 -- 172.21.15.16:0/3484667932 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cdc06c920 con 0x7f9cdc155020 2023-10-16T12:27:41.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.930+0000 7f9ce26c4700 1 -- 172.21.15.16:0/3484667932 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cdc04ebb0 con 0x7f9cdc155020 2023-10-16T12:27:41.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.930+0000 7f9cd8ff9700 1 -- 172.21.15.16:0/3484667932 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 51) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9cd4013900 con 0x7f9cdc155020 2023-10-16T12:27:41.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.931+0000 7f9cd8ff9700 1 --2- 172.21.15.16:0/3484667932 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f9cbc06b020 0x7f9cbc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:41.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.931+0000 7f9cd8ff9700 1 -- 172.21.15.16:0/3484667932 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f9cd408a0c0 con 0x7f9cdc155020 2023-10-16T12:27:41.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.931+0000 7f9cdbfff700 1 --2- 172.21.15.16:0/3484667932 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f9cbc06b020 0x7f9cbc06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:41.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.931+0000 7f9cdbfff700 1 --2- 172.21.15.16:0/3484667932 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f9cbc06b020 0x7f9cbc06d4e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc00a010 tx=0x7f9ccc008030 comp rx=0 tx=0).ready entity=mgr.6365 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:41.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:41.935+0000 7f9cd8ff9700 1 -- 172.21.15.16:0/3484667932 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9cd4054e60 con 0x7f9cdc155020 2023-10-16T12:27:42.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.135+0000 7f9ce26c4700 1 -- 172.21.15.16:0/3484667932 --> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9cdc0ccb00 con 0x7f9cbc06b020 2023-10-16T12:27:42.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.144+0000 7f9cd8ff9700 1 -- 172.21.15.16:0/3484667932 <== mgr.6365 v2:172.21.15.16:6832/4089191642 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289981 (secure 0 0 0) 0x7f9ccc005cf0 con 0x7f9cbc06b020 2023-10-16T12:27:42.149 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:27:42.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.147+0000 7f9ce26c4700 1 -- 172.21.15.16:0/3484667932 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f9cbc06b020 msgr2=0x7f9cbc06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:42.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.147+0000 7f9ce26c4700 1 --2- 172.21.15.16:0/3484667932 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f9cbc06b020 0x7f9cbc06d4e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc00a010 tx=0x7f9ccc008030 comp rx=0 tx=0).stop 2023-10-16T12:27:42.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.147+0000 7f9ce26c4700 1 -- 172.21.15.16:0/3484667932 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9cdc155020 msgr2=0x7f9cdc1554a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:42.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.147+0000 7f9ce26c4700 1 --2- 172.21.15.16:0/3484667932 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9cdc155020 0x7f9cdc1554a0 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc0672d0 tx=0x7f9cd4007340 comp rx=0 tx=0).stop 2023-10-16T12:27:42.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.148+0000 7f9ce26c4700 1 -- 172.21.15.16:0/3484667932 shutdown_connections 2023-10-16T12:27:42.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.148+0000 7f9ce26c4700 1 --2- 172.21.15.16:0/3484667932 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9cdc155020 0x7f9cdc1554a0 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:42.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.148+0000 7f9ce26c4700 1 --2- 172.21.15.16:0/3484667932 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f9cbc06b020 0x7f9cbc06d4e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:42.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.148+0000 7f9ce26c4700 1 --2- 172.21.15.16:0/3484667932 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9cc8003a00 0x7f9cdc1f2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:42.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.148+0000 7f9ce26c4700 1 --2- 172.21.15.16:0/3484667932 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9cdc155d80 0x7f9cdc06d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:42.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.148+0000 7f9ce26c4700 1 -- 172.21.15.16:0/3484667932 >> 172.21.15.16:0/3484667932 conn(0x7f9cdc0d14b0 msgr2=0x7f9cdc06f5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:42.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.148+0000 7f9ce26c4700 1 -- 172.21.15.16:0/3484667932 shutdown_connections 2023-10-16T12:27:42.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.148+0000 7f9ce26c4700 1 -- 172.21.15.16:0/3484667932 wait complete. 2023-10-16T12:27:42.154 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-10-16T12:27:42.162 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2023-10-16T12:27:41.522961+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":692,"num_read_kb":1336,"num_write":323,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":221,"ondisk_log_size":221,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":156,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":344632,"kb_used_data":11912,"kb_used_omap":12,"kb_used_meta":332595,"kb_avail":754630088,"statfs":{"total":773094113280,"available":772741210112,"internally_reserved":0,"allocated":12197888,"data_stored":6658412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12740,"internal_metadata":340577852},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.555387"},"pg_stats":[{"pgid":"14.3f","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.006282+0000","last_change":"2023-10-16T12:27:38.005483+0000","last_active":"2023-10-16T12:27:39.006282+0000","last_peered":"2023-10-16T12:27:39.006282+0000","last_clean":"2023-10-16T12:27:39.006282+0000","last_became_active":"2023-10-16T12:27:38.005224+0000","last_became_peered":"2023-10-16T12:27:38.005224+0000","last_unstale":"2023-10-16T12:27:39.006282+0000","last_undegraded":"2023-10-16T12:27:39.006282+0000","last_fullsized":"2023-10-16T12:27:39.006282+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:12:41.771844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3e","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.067841+0000","last_change":"2023-10-16T12:27:38.020825+0000","last_active":"2023-10-16T12:27:39.067841+0000","last_peered":"2023-10-16T12:27:39.067841+0000","last_clean":"2023-10-16T12:27:39.067841+0000","last_became_active":"2023-10-16T12:27:38.020526+0000","last_became_peered":"2023-10-16T12:27:38.020526+0000","last_unstale":"2023-10-16T12:27:39.067841+0000","last_undegraded":"2023-10-16T12:27:39.067841+0000","last_fullsized":"2023-10-16T12:27:39.067841+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:25:17.395792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.3d","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.006234+0000","last_change":"2023-10-16T12:27:38.027615+0000","last_active":"2023-10-16T12:27:39.006234+0000","last_peered":"2023-10-16T12:27:39.006234+0000","last_clean":"2023-10-16T12:27:39.006234+0000","last_became_active":"2023-10-16T12:27:38.027101+0000","last_became_peered":"2023-10-16T12:27:38.027101+0000","last_unstale":"2023-10-16T12:27:39.006234+0000","last_undegraded":"2023-10-16T12:27:39.006234+0000","last_fullsized":"2023-10-16T12:27:39.006234+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:07:35.137319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3c","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.246720+0000","last_change":"2023-10-16T12:27:38.021342+0000","last_active":"2023-10-16T12:27:39.246720+0000","last_peered":"2023-10-16T12:27:39.246720+0000","last_clean":"2023-10-16T12:27:39.246720+0000","last_became_active":"2023-10-16T12:27:38.016660+0000","last_became_peered":"2023-10-16T12:27:38.016660+0000","last_unstale":"2023-10-16T12:27:39.246720+0000","last_undegraded":"2023-10-16T12:27:39.246720+0000","last_fullsized":"2023-10-16T12:27:39.246720+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:42:57.697956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.3b","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.023841+0000","last_change":"2023-10-16T12:27:38.022837+0000","last_active":"2023-10-16T12:27:38.023841+0000","last_peered":"2023-10-16T12:27:38.023841+0000","last_clean":"2023-10-16T12:27:38.023841+0000","last_became_active":"2023-10-16T12:27:38.018004+0000","last_became_peered":"2023-10-16T12:27:38.018004+0000","last_unstale":"2023-10-16T12:27:38.023841+0000","last_undegraded":"2023-10-16T12:27:38.023841+0000","last_fullsized":"2023-10-16T12:27:38.023841+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:35:38.723509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.3a","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.005298+0000","last_change":"2023-10-16T12:27:38.022739+0000","last_active":"2023-10-16T12:27:39.005298+0000","last_peered":"2023-10-16T12:27:39.005298+0000","last_clean":"2023-10-16T12:27:39.005298+0000","last_became_active":"2023-10-16T12:27:38.022469+0000","last_became_peered":"2023-10-16T12:27:38.022469+0000","last_unstale":"2023-10-16T12:27:39.005298+0000","last_undegraded":"2023-10-16T12:27:39.005298+0000","last_fullsized":"2023-10-16T12:27:39.005298+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:55:52.173496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.39","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.017899+0000","last_change":"2023-10-16T12:27:38.015269+0000","last_active":"2023-10-16T12:27:38.017899+0000","last_peered":"2023-10-16T12:27:38.017899+0000","last_clean":"2023-10-16T12:27:38.017899+0000","last_became_active":"2023-10-16T12:27:38.014957+0000","last_became_peered":"2023-10-16T12:27:38.014957+0000","last_unstale":"2023-10-16T12:27:38.017899+0000","last_undegraded":"2023-10-16T12:27:38.017899+0000","last_fullsized":"2023-10-16T12:27:38.017899+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:31:05.189021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.38","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.246191+0000","last_change":"2023-10-16T12:27:38.023995+0000","last_active":"2023-10-16T12:27:39.246191+0000","last_peered":"2023-10-16T12:27:39.246191+0000","last_clean":"2023-10-16T12:27:39.246191+0000","last_became_active":"2023-10-16T12:27:38.023354+0000","last_became_peered":"2023-10-16T12:27:38.023354+0000","last_unstale":"2023-10-16T12:27:39.246191+0000","last_undegraded":"2023-10-16T12:27:39.246191+0000","last_fullsized":"2023-10-16T12:27:39.246191+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:35:38.723509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.37","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.022551+0000","last_change":"2023-10-16T12:27:38.016632+0000","last_active":"2023-10-16T12:27:38.022551+0000","last_peered":"2023-10-16T12:27:38.022551+0000","last_clean":"2023-10-16T12:27:38.022551+0000","last_became_active":"2023-10-16T12:27:38.016298+0000","last_became_peered":"2023-10-16T12:27:38.016298+0000","last_unstale":"2023-10-16T12:27:38.022551+0000","last_undegraded":"2023-10-16T12:27:38.022551+0000","last_fullsized":"2023-10-16T12:27:38.022551+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:17:05.807429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.36","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.019566+0000","last_change":"2023-10-16T12:27:38.017919+0000","last_active":"2023-10-16T12:27:38.019566+0000","last_peered":"2023-10-16T12:27:38.019566+0000","last_clean":"2023-10-16T12:27:38.019566+0000","last_became_active":"2023-10-16T12:27:38.017617+0000","last_became_peered":"2023-10-16T12:27:38.017617+0000","last_unstale":"2023-10-16T12:27:38.019566+0000","last_undegraded":"2023-10-16T12:27:38.019566+0000","last_fullsized":"2023-10-16T12:27:38.019566+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:24:26.284643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.35","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.026458+0000","last_change":"2023-10-16T12:27:38.025552+0000","last_active":"2023-10-16T12:27:38.026458+0000","last_peered":"2023-10-16T12:27:38.026458+0000","last_clean":"2023-10-16T12:27:38.026458+0000","last_became_active":"2023-10-16T12:27:38.024341+0000","last_became_peered":"2023-10-16T12:27:38.024341+0000","last_unstale":"2023-10-16T12:27:38.026458+0000","last_undegraded":"2023-10-16T12:27:38.026458+0000","last_fullsized":"2023-10-16T12:27:38.026458+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:10:51.389847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.34","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:37.999427+0000","last_change":"2023-10-16T12:27:37.997869+0000","last_active":"2023-10-16T12:27:37.999427+0000","last_peered":"2023-10-16T12:27:37.999427+0000","last_clean":"2023-10-16T12:27:37.999427+0000","last_became_active":"2023-10-16T12:27:37.997549+0000","last_became_peered":"2023-10-16T12:27:37.997549+0000","last_unstale":"2023-10-16T12:27:37.999427+0000","last_undegraded":"2023-10-16T12:27:37.999427+0000","last_fullsized":"2023-10-16T12:27:37.999427+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:34:35.434077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.33","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.246443+0000","last_change":"2023-10-16T12:27:38.016063+0000","last_active":"2023-10-16T12:27:39.246443+0000","last_peered":"2023-10-16T12:27:39.246443+0000","last_clean":"2023-10-16T12:27:39.246443+0000","last_became_active":"2023-10-16T12:27:38.014322+0000","last_became_peered":"2023-10-16T12:27:38.014322+0000","last_unstale":"2023-10-16T12:27:39.246443+0000","last_undegraded":"2023-10-16T12:27:39.246443+0000","last_fullsized":"2023-10-16T12:27:39.246443+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:05:06.570934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.32","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.025675+0000","last_change":"2023-10-16T12:27:38.024608+0000","last_active":"2023-10-16T12:27:38.025675+0000","last_peered":"2023-10-16T12:27:38.025675+0000","last_clean":"2023-10-16T12:27:38.025675+0000","last_became_active":"2023-10-16T12:27:38.024326+0000","last_became_peered":"2023-10-16T12:27:38.024326+0000","last_unstale":"2023-10-16T12:27:38.025675+0000","last_undegraded":"2023-10-16T12:27:38.025675+0000","last_fullsized":"2023-10-16T12:27:38.025675+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:26:45.443813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.31","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.246535+0000","last_change":"2023-10-16T12:27:38.017128+0000","last_active":"2023-10-16T12:27:39.246535+0000","last_peered":"2023-10-16T12:27:39.246535+0000","last_clean":"2023-10-16T12:27:39.246535+0000","last_became_active":"2023-10-16T12:27:38.016473+0000","last_became_peered":"2023-10-16T12:27:38.016473+0000","last_unstale":"2023-10-16T12:27:39.246535+0000","last_undegraded":"2023-10-16T12:27:39.246535+0000","last_fullsized":"2023-10-16T12:27:39.246535+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:56:39.091850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.30","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.005206+0000","last_change":"2023-10-16T12:27:38.019437+0000","last_active":"2023-10-16T12:27:39.005206+0000","last_peered":"2023-10-16T12:27:39.005206+0000","last_clean":"2023-10-16T12:27:39.005206+0000","last_became_active":"2023-10-16T12:27:38.016139+0000","last_became_peered":"2023-10-16T12:27:38.016139+0000","last_unstale":"2023-10-16T12:27:39.005206+0000","last_undegraded":"2023-10-16T12:27:39.005206+0000","last_fullsized":"2023-10-16T12:27:39.005206+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:36.707208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2f","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.005391+0000","last_change":"2023-10-16T12:27:38.020885+0000","last_active":"2023-10-16T12:27:39.005391+0000","last_peered":"2023-10-16T12:27:39.005391+0000","last_clean":"2023-10-16T12:27:39.005391+0000","last_became_active":"2023-10-16T12:27:38.019259+0000","last_became_peered":"2023-10-16T12:27:38.019259+0000","last_unstale":"2023-10-16T12:27:39.005391+0000","last_undegraded":"2023-10-16T12:27:39.005391+0000","last_fullsized":"2023-10-16T12:27:39.005391+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:25:31.018138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2e","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.006301+0000","last_change":"2023-10-16T12:27:38.012752+0000","last_active":"2023-10-16T12:27:39.006301+0000","last_peered":"2023-10-16T12:27:39.006301+0000","last_clean":"2023-10-16T12:27:39.006301+0000","last_became_active":"2023-10-16T12:27:38.012512+0000","last_became_peered":"2023-10-16T12:27:38.012512+0000","last_unstale":"2023-10-16T12:27:39.006301+0000","last_undegraded":"2023-10-16T12:27:39.006301+0000","last_fullsized":"2023-10-16T12:27:39.006301+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:19:09.853556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2d","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.027539+0000","last_change":"2023-10-16T12:27:38.025778+0000","last_active":"2023-10-16T12:27:38.027539+0000","last_peered":"2023-10-16T12:27:38.027539+0000","last_clean":"2023-10-16T12:27:38.027539+0000","last_became_active":"2023-10-16T12:27:38.025477+0000","last_became_peered":"2023-10-16T12:27:38.025477+0000","last_unstale":"2023-10-16T12:27:38.027539+0000","last_undegraded":"2023-10-16T12:27:38.027539+0000","last_fullsized":"2023-10-16T12:27:38.027539+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:40:07.635702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2c","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.246437+0000","last_change":"2023-10-16T12:27:38.021327+0000","last_active":"2023-10-16T12:27:39.246437+0000","last_peered":"2023-10-16T12:27:39.246437+0000","last_clean":"2023-10-16T12:27:39.246437+0000","last_became_active":"2023-10-16T12:27:38.016635+0000","last_became_peered":"2023-10-16T12:27:38.016635+0000","last_unstale":"2023-10-16T12:27:39.246437+0000","last_undegraded":"2023-10-16T12:27:39.246437+0000","last_fullsized":"2023-10-16T12:27:39.246437+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:39:51.985205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.2b","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.023462+0000","last_change":"2023-10-16T12:27:38.017752+0000","last_active":"2023-10-16T12:27:38.023462+0000","last_peered":"2023-10-16T12:27:38.023462+0000","last_clean":"2023-10-16T12:27:38.023462+0000","last_became_active":"2023-10-16T12:27:38.017492+0000","last_became_peered":"2023-10-16T12:27:38.017492+0000","last_unstale":"2023-10-16T12:27:38.023462+0000","last_undegraded":"2023-10-16T12:27:38.023462+0000","last_fullsized":"2023-10-16T12:27:38.023462+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:40:59.602579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2a","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.025625+0000","last_change":"2023-10-16T12:27:38.024306+0000","last_active":"2023-10-16T12:27:38.025625+0000","last_peered":"2023-10-16T12:27:38.025625+0000","last_clean":"2023-10-16T12:27:38.025625+0000","last_became_active":"2023-10-16T12:27:38.023892+0000","last_became_peered":"2023-10-16T12:27:38.023892+0000","last_unstale":"2023-10-16T12:27:38.025625+0000","last_undegraded":"2023-10-16T12:27:38.025625+0000","last_fullsized":"2023-10-16T12:27:38.025625+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:45:49.653013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.29","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.020867+0000","last_change":"2023-10-16T12:27:38.019317+0000","last_active":"2023-10-16T12:27:38.020867+0000","last_peered":"2023-10-16T12:27:38.020867+0000","last_clean":"2023-10-16T12:27:38.020867+0000","last_became_active":"2023-10-16T12:27:38.019054+0000","last_became_peered":"2023-10-16T12:27:38.019054+0000","last_unstale":"2023-10-16T12:27:38.020867+0000","last_undegraded":"2023-10-16T12:27:38.020867+0000","last_fullsized":"2023-10-16T12:27:38.020867+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:50:34.414528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.28","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.005986+0000","last_change":"2023-10-16T12:27:38.004330+0000","last_active":"2023-10-16T12:27:39.005986+0000","last_peered":"2023-10-16T12:27:39.005986+0000","last_clean":"2023-10-16T12:27:39.005986+0000","last_became_active":"2023-10-16T12:27:38.004054+0000","last_became_peered":"2023-10-16T12:27:38.004054+0000","last_unstale":"2023-10-16T12:27:39.005986+0000","last_undegraded":"2023-10-16T12:27:39.005986+0000","last_fullsized":"2023-10-16T12:27:39.005986+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:40:35.029721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.27","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.067884+0000","last_change":"2023-10-16T12:27:38.005381+0000","last_active":"2023-10-16T12:27:39.067884+0000","last_peered":"2023-10-16T12:27:39.067884+0000","last_clean":"2023-10-16T12:27:39.067884+0000","last_became_active":"2023-10-16T12:27:38.005081+0000","last_became_peered":"2023-10-16T12:27:38.005081+0000","last_unstale":"2023-10-16T12:27:39.067884+0000","last_undegraded":"2023-10-16T12:27:39.067884+0000","last_fullsized":"2023-10-16T12:27:39.067884+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:31:05.189021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.26","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.023846+0000","last_change":"2023-10-16T12:27:38.022828+0000","last_active":"2023-10-16T12:27:38.023846+0000","last_peered":"2023-10-16T12:27:38.023846+0000","last_clean":"2023-10-16T12:27:38.023846+0000","last_became_active":"2023-10-16T12:27:38.022464+0000","last_became_peered":"2023-10-16T12:27:38.022464+0000","last_unstale":"2023-10-16T12:27:38.023846+0000","last_undegraded":"2023-10-16T12:27:38.023846+0000","last_fullsized":"2023-10-16T12:27:38.023846+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:01:50.960997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.25","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.009839+0000","last_change":"2023-10-16T12:27:38.005554+0000","last_active":"2023-10-16T12:27:38.009839+0000","last_peered":"2023-10-16T12:27:38.009839+0000","last_clean":"2023-10-16T12:27:38.009839+0000","last_became_active":"2023-10-16T12:27:38.005251+0000","last_became_peered":"2023-10-16T12:27:38.005251+0000","last_unstale":"2023-10-16T12:27:38.009839+0000","last_undegraded":"2023-10-16T12:27:38.009839+0000","last_fullsized":"2023-10-16T12:27:38.009839+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:30:01.478780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.24","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.246521+0000","last_change":"2023-10-16T12:27:38.024543+0000","last_active":"2023-10-16T12:27:39.246521+0000","last_peered":"2023-10-16T12:27:39.246521+0000","last_clean":"2023-10-16T12:27:39.246521+0000","last_became_active":"2023-10-16T12:27:38.024200+0000","last_became_peered":"2023-10-16T12:27:38.024200+0000","last_unstale":"2023-10-16T12:27:39.246521+0000","last_undegraded":"2023-10-16T12:27:39.246521+0000","last_fullsized":"2023-10-16T12:27:39.246521+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:37:50.379179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.23","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.011782+0000","last_change":"2023-10-16T12:27:38.008834+0000","last_active":"2023-10-16T12:27:38.011782+0000","last_peered":"2023-10-16T12:27:38.011782+0000","last_clean":"2023-10-16T12:27:38.011782+0000","last_became_active":"2023-10-16T12:27:38.008529+0000","last_became_peered":"2023-10-16T12:27:38.008529+0000","last_unstale":"2023-10-16T12:27:38.011782+0000","last_undegraded":"2023-10-16T12:27:38.011782+0000","last_fullsized":"2023-10-16T12:27:38.011782+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:38:19.735626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.22","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.023568+0000","last_change":"2023-10-16T12:27:38.014992+0000","last_active":"2023-10-16T12:27:38.023568+0000","last_peered":"2023-10-16T12:27:38.023568+0000","last_clean":"2023-10-16T12:27:38.023568+0000","last_became_active":"2023-10-16T12:27:38.014683+0000","last_became_peered":"2023-10-16T12:27:38.014683+0000","last_unstale":"2023-10-16T12:27:38.023568+0000","last_undegraded":"2023-10-16T12:27:38.023568+0000","last_fullsized":"2023-10-16T12:27:38.023568+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:26:03.624784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.21","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.067565+0000","last_change":"2023-10-16T12:27:38.016478+0000","last_active":"2023-10-16T12:27:39.067565+0000","last_peered":"2023-10-16T12:27:39.067565+0000","last_clean":"2023-10-16T12:27:39.067565+0000","last_became_active":"2023-10-16T12:27:38.016068+0000","last_became_peered":"2023-10-16T12:27:38.016068+0000","last_unstale":"2023-10-16T12:27:39.067565+0000","last_undegraded":"2023-10-16T12:27:39.067565+0000","last_fullsized":"2023-10-16T12:27:39.067565+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:10:26.061080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.20","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.004842+0000","last_change":"2023-10-16T12:27:38.019579+0000","last_active":"2023-10-16T12:27:39.004842+0000","last_peered":"2023-10-16T12:27:39.004842+0000","last_clean":"2023-10-16T12:27:39.004842+0000","last_became_active":"2023-10-16T12:27:38.016194+0000","last_became_peered":"2023-10-16T12:27:38.016194+0000","last_unstale":"2023-10-16T12:27:39.004842+0000","last_undegraded":"2023-10-16T12:27:39.004842+0000","last_fullsized":"2023-10-16T12:27:39.004842+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:30:36.115116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.005192+0000","last_change":"2023-10-16T12:27:38.019739+0000","last_active":"2023-10-16T12:27:39.005192+0000","last_peered":"2023-10-16T12:27:39.005192+0000","last_clean":"2023-10-16T12:27:39.005192+0000","last_became_active":"2023-10-16T12:27:38.019452+0000","last_became_peered":"2023-10-16T12:27:38.019452+0000","last_unstale":"2023-10-16T12:27:39.005192+0000","last_undegraded":"2023-10-16T12:27:39.005192+0000","last_fullsized":"2023-10-16T12:27:39.005192+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:41:54.837529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1f","version":"111'4","reported_seq":15,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.026827+0000","last_change":"2023-10-16T12:27:36.976544+0000","last_active":"2023-10-16T12:27:38.026827+0000","last_peered":"2023-10-16T12:27:38.026827+0000","last_clean":"2023-10-16T12:27:38.026827+0000","last_became_active":"2023-10-16T12:27:36.976186+0000","last_became_peered":"2023-10-16T12:27:36.976186+0000","last_unstale":"2023-10-16T12:27:38.026827+0000","last_undegraded":"2023-10-16T12:27:38.026827+0000","last_fullsized":"2023-10-16T12:27:38.026827+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_clean_scrub_stamp":"2023-10-16T12:27:35.962232+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:10:50.385306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.005574+0000","last_change":"2023-10-16T12:27:38.012429+0000","last_active":"2023-10-16T12:27:39.005574+0000","last_peered":"2023-10-16T12:27:39.005574+0000","last_clean":"2023-10-16T12:27:39.005574+0000","last_became_active":"2023-10-16T12:27:38.011990+0000","last_became_peered":"2023-10-16T12:27:38.011990+0000","last_unstale":"2023-10-16T12:27:39.005574+0000","last_undegraded":"2023-10-16T12:27:39.005574+0000","last_fullsized":"2023-10-16T12:27:39.005574+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:38:29.081086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1e","version":"111'1","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.023706+0000","last_change":"2023-10-16T12:27:36.985203+0000","last_active":"2023-10-16T12:27:38.023706+0000","last_peered":"2023-10-16T12:27:38.023706+0000","last_clean":"2023-10-16T12:27:38.023706+0000","last_became_active":"2023-10-16T12:27:36.984995+0000","last_became_peered":"2023-10-16T12:27:36.984995+0000","last_unstale":"2023-10-16T12:27:38.023706+0000","last_undegraded":"2023-10-16T12:27:38.023706+0000","last_fullsized":"2023-10-16T12:27:38.023706+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_clean_scrub_stamp":"2023-10-16T12:27:35.962232+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:36:00.608821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.019214+0000","last_change":"2023-10-16T12:27:38.018002+0000","last_active":"2023-10-16T12:27:38.019214+0000","last_peered":"2023-10-16T12:27:38.019214+0000","last_clean":"2023-10-16T12:27:38.019214+0000","last_became_active":"2023-10-16T12:27:38.017694+0000","last_became_peered":"2023-10-16T12:27:38.017694+0000","last_unstale":"2023-10-16T12:27:38.019214+0000","last_undegraded":"2023-10-16T12:27:38.019214+0000","last_fullsized":"2023-10-16T12:27:38.019214+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:30:42.276938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.1d","version":"111'2","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.019546+0000","last_change":"2023-10-16T12:27:36.988403+0000","last_active":"2023-10-16T12:27:38.019546+0000","last_peered":"2023-10-16T12:27:38.019546+0000","last_clean":"2023-10-16T12:27:38.019546+0000","last_became_active":"2023-10-16T12:27:36.988084+0000","last_became_peered":"2023-10-16T12:27:36.988084+0000","last_unstale":"2023-10-16T12:27:38.019546+0000","last_undegraded":"2023-10-16T12:27:38.019546+0000","last_fullsized":"2023-10-16T12:27:38.019546+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_clean_scrub_stamp":"2023-10-16T12:27:35.962232+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:40:57.661929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.005514+0000","last_change":"2023-10-16T12:27:38.011634+0000","last_active":"2023-10-16T12:27:39.005514+0000","last_peered":"2023-10-16T12:27:39.005514+0000","last_clean":"2023-10-16T12:27:39.005514+0000","last_became_active":"2023-10-16T12:27:38.011327+0000","last_became_peered":"2023-10-16T12:27:38.011327+0000","last_unstale":"2023-10-16T12:27:39.005514+0000","last_undegraded":"2023-10-16T12:27:39.005514+0000","last_fullsized":"2023-10-16T12:27:39.005514+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:23:12.852377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1c","version":"111'2","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.028839+0000","last_change":"2023-10-16T12:27:36.988560+0000","last_active":"2023-10-16T12:27:38.028839+0000","last_peered":"2023-10-16T12:27:38.028839+0000","last_clean":"2023-10-16T12:27:38.028839+0000","last_became_active":"2023-10-16T12:27:36.988191+0000","last_became_peered":"2023-10-16T12:27:36.988191+0000","last_unstale":"2023-10-16T12:27:38.028839+0000","last_undegraded":"2023-10-16T12:27:38.028839+0000","last_fullsized":"2023-10-16T12:27:38.028839+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_clean_scrub_stamp":"2023-10-16T12:27:35.962232+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:40:58.598038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.246215+0000","last_change":"2023-10-16T12:27:38.023815+0000","last_active":"2023-10-16T12:27:39.246215+0000","last_peered":"2023-10-16T12:27:39.246215+0000","last_clean":"2023-10-16T12:27:39.246215+0000","last_became_active":"2023-10-16T12:27:38.023028+0000","last_became_peered":"2023-10-16T12:27:38.023028+0000","last_unstale":"2023-10-16T12:27:39.246215+0000","last_undegraded":"2023-10-16T12:27:39.246215+0000","last_fullsized":"2023-10-16T12:27:39.246215+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:41:19.520990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1b","version":"111'1","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.023249+0000","last_change":"2023-10-16T12:27:36.985795+0000","last_active":"2023-10-16T12:27:38.023249+0000","last_peered":"2023-10-16T12:27:38.023249+0000","last_clean":"2023-10-16T12:27:38.023249+0000","last_became_active":"2023-10-16T12:27:36.984740+0000","last_became_peered":"2023-10-16T12:27:36.984740+0000","last_unstale":"2023-10-16T12:27:38.023249+0000","last_undegraded":"2023-10-16T12:27:38.023249+0000","last_fullsized":"2023-10-16T12:27:38.023249+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_clean_scrub_stamp":"2023-10-16T12:27:35.962232+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:41:18.516450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.011113+0000","last_change":"2023-10-16T12:27:38.009370+0000","last_active":"2023-10-16T12:27:38.011113+0000","last_peered":"2023-10-16T12:27:38.011113+0000","last_clean":"2023-10-16T12:27:38.011113+0000","last_became_active":"2023-10-16T12:27:38.009049+0000","last_became_peered":"2023-10-16T12:27:38.009049+0000","last_unstale":"2023-10-16T12:27:38.011113+0000","last_undegraded":"2023-10-16T12:27:38.011113+0000","last_fullsized":"2023-10-16T12:27:38.011113+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:34:33.927601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1a","version":"111'1","reported_seq":13,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.246725+0000","last_change":"2023-10-16T12:27:36.992130+0000","last_active":"2023-10-16T12:27:39.246725+0000","last_peered":"2023-10-16T12:27:39.246725+0000","last_clean":"2023-10-16T12:27:39.246725+0000","last_became_active":"2023-10-16T12:27:36.991795+0000","last_became_peered":"2023-10-16T12:27:36.991795+0000","last_unstale":"2023-10-16T12:27:39.246725+0000","last_undegraded":"2023-10-16T12:27:39.246725+0000","last_fullsized":"2023-10-16T12:27:39.246725+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_clean_scrub_stamp":"2023-10-16T12:27:35.962232+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:01:49.956456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.004947+0000","last_change":"2023-10-16T12:27:38.023001+0000","last_active":"2023-10-16T12:27:39.004947+0000","last_peered":"2023-10-16T12:27:39.004947+0000","last_clean":"2023-10-16T12:27:39.004947+0000","last_became_active":"2023-10-16T12:27:38.022696+0000","last_became_peered":"2023-10-16T12:27:38.022696+0000","last_unstale":"2023-10-16T12:27:39.004947+0000","last_undegraded":"2023-10-16T12:27:39.004947+0000","last_fullsized":"2023-10-16T12:27:39.004947+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:48:11.313600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.006218+0000","last_change":"2023-10-16T12:27:36.989874+0000","last_active":"2023-10-16T12:27:39.006218+0000","last_peered":"2023-10-16T12:27:39.006218+0000","last_clean":"2023-10-16T12:27:39.006218+0000","last_became_active":"2023-10-16T12:27:36.989572+0000","last_became_peered":"2023-10-16T12:27:36.989572+0000","last_unstale":"2023-10-16T12:27:39.006218+0000","last_undegraded":"2023-10-16T12:27:39.006218+0000","last_fullsized":"2023-10-16T12:27:39.006218+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_clean_scrub_stamp":"2023-10-16T12:27:35.962232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:31:48.599920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.246432+0000","last_change":"2023-10-16T12:27:38.013437+0000","last_active":"2023-10-16T12:27:39.246432+0000","last_peered":"2023-10-16T12:27:39.246432+0000","last_clean":"2023-10-16T12:27:39.246432+0000","last_became_active":"2023-10-16T12:27:38.013138+0000","last_became_peered":"2023-10-16T12:27:38.013138+0000","last_unstale":"2023-10-16T12:27:39.246432+0000","last_undegraded":"2023-10-16T12:27:39.246432+0000","last_fullsized":"2023-10-16T12:27:39.246432+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:36:01.613362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:37.996866+0000","last_change":"2023-10-16T12:27:36.990706+0000","last_active":"2023-10-16T12:27:37.996866+0000","last_peered":"2023-10-16T12:27:37.996866+0000","last_clean":"2023-10-16T12:27:37.996866+0000","last_became_active":"2023-10-16T12:27:36.990444+0000","last_became_peered":"2023-10-16T12:27:36.990444+0000","last_unstale":"2023-10-16T12:27:37.996866+0000","last_undegraded":"2023-10-16T12:27:37.996866+0000","last_fullsized":"2023-10-16T12:27:37.996866+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_clean_scrub_stamp":"2023-10-16T12:27:35.962232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:38:21.525776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.022316+0000","last_change":"2023-10-16T12:27:38.015555+0000","last_active":"2023-10-16T12:27:38.022316+0000","last_peered":"2023-10-16T12:27:38.022316+0000","last_clean":"2023-10-16T12:27:38.022316+0000","last_became_active":"2023-10-16T12:27:38.015262+0000","last_became_peered":"2023-10-16T12:27:38.015262+0000","last_unstale":"2023-10-16T12:27:38.022316+0000","last_undegraded":"2023-10-16T12:27:38.022316+0000","last_fullsized":"2023-10-16T12:27:38.022316+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:23:00.622863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.067945+0000","last_change":"2023-10-16T12:27:36.981921+0000","last_active":"2023-10-16T12:27:39.067945+0000","last_peered":"2023-10-16T12:27:39.067945+0000","last_clean":"2023-10-16T12:27:39.067945+0000","last_became_active":"2023-10-16T12:27:36.981574+0000","last_became_peered":"2023-10-16T12:27:36.981574+0000","last_unstale":"2023-10-16T12:27:39.067945+0000","last_undegraded":"2023-10-16T12:27:39.067945+0000","last_fullsized":"2023-10-16T12:27:39.067945+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_clean_scrub_stamp":"2023-10-16T12:27:35.962232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:37:47.438147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.026737+0000","last_change":"2023-10-16T12:27:38.026059+0000","last_active":"2023-10-16T12:27:38.026737+0000","last_peered":"2023-10-16T12:27:38.026737+0000","last_clean":"2023-10-16T12:27:38.026737+0000","last_became_active":"2023-10-16T12:27:38.025831+0000","last_became_peered":"2023-10-16T12:27:38.025831+0000","last_unstale":"2023-10-16T12:27:38.026737+0000","last_undegraded":"2023-10-16T12:27:38.026737+0000","last_fullsized":"2023-10-16T12:27:38.026737+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:38:31.277703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.006597+0000","last_change":"2023-10-16T12:27:36.994850+0000","last_active":"2023-10-16T12:27:39.006597+0000","last_peered":"2023-10-16T12:27:39.006597+0000","last_clean":"2023-10-16T12:27:39.006597+0000","last_became_active":"2023-10-16T12:27:36.994478+0000","last_became_peered":"2023-10-16T12:27:36.994478+0000","last_unstale":"2023-10-16T12:27:39.006597+0000","last_undegraded":"2023-10-16T12:27:39.006597+0000","last_fullsized":"2023-10-16T12:27:39.006597+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_clean_scrub_stamp":"2023-10-16T12:27:35.962232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:58:11.208246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.006438+0000","last_change":"2023-10-16T12:27:38.009003+0000","last_active":"2023-10-16T12:27:39.006438+0000","last_peered":"2023-10-16T12:27:39.006438+0000","last_clean":"2023-10-16T12:27:39.006438+0000","last_became_active":"2023-10-16T12:27:38.008692+0000","last_became_peered":"2023-10-16T12:27:38.008692+0000","last_unstale":"2023-10-16T12:27:39.006438+0000","last_undegraded":"2023-10-16T12:27:39.006438+0000","last_fullsized":"2023-10-16T12:27:39.006438+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:48:13.638880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.15","version":"111'3","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.029182+0000","last_change":"2023-10-16T12:27:36.990285+0000","last_active":"2023-10-16T12:27:38.029182+0000","last_peered":"2023-10-16T12:27:38.029182+0000","last_clean":"2023-10-16T12:27:38.029182+0000","last_became_active":"2023-10-16T12:27:36.989921+0000","last_became_peered":"2023-10-16T12:27:36.989921+0000","last_unstale":"2023-10-16T12:27:38.029182+0000","last_undegraded":"2023-10-16T12:27:38.029182+0000","last_fullsized":"2023-10-16T12:27:38.029182+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_clean_scrub_stamp":"2023-10-16T12:27:35.962232+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:25:18.221168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.246616+0000","last_change":"2023-10-16T12:27:38.016179+0000","last_active":"2023-10-16T12:27:39.246616+0000","last_peered":"2023-10-16T12:27:39.246616+0000","last_clean":"2023-10-16T12:27:39.246616+0000","last_became_active":"2023-10-16T12:27:38.014743+0000","last_became_peered":"2023-10-16T12:27:38.014743+0000","last_unstale":"2023-10-16T12:27:39.246616+0000","last_undegraded":"2023-10-16T12:27:39.246616+0000","last_fullsized":"2023-10-16T12:27:39.246616+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:52:33.322374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.14","version":"111'6","reported_seq":18,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.005301+0000","last_change":"2023-10-16T12:27:36.981047+0000","last_active":"2023-10-16T12:27:39.005301+0000","last_peered":"2023-10-16T12:27:39.005301+0000","last_clean":"2023-10-16T12:27:39.005301+0000","last_became_active":"2023-10-16T12:27:36.980732+0000","last_became_peered":"2023-10-16T12:27:36.980732+0000","last_unstale":"2023-10-16T12:27:39.005301+0000","last_undegraded":"2023-10-16T12:27:39.005301+0000","last_fullsized":"2023-10-16T12:27:39.005301+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_clean_scrub_stamp":"2023-10-16T12:27:35.962232+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:29:38.117150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.246658+0000","last_change":"2023-10-16T12:27:38.023643+0000","last_active":"2023-10-16T12:27:39.246658+0000","last_peered":"2023-10-16T12:27:39.246658+0000","last_clean":"2023-10-16T12:27:39.246658+0000","last_became_active":"2023-10-16T12:27:38.022731+0000","last_became_peered":"2023-10-16T12:27:38.022731+0000","last_unstale":"2023-10-16T12:27:39.246658+0000","last_undegraded":"2023-10-16T12:27:39.246658+0000","last_fullsized":"2023-10-16T12:27:39.246658+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:03:31.380576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"111'2","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.024610+0000","last_change":"2023-10-16T12:27:36.990608+0000","last_active":"2023-10-16T12:27:38.024610+0000","last_peered":"2023-10-16T12:27:38.024610+0000","last_clean":"2023-10-16T12:27:38.024610+0000","last_became_active":"2023-10-16T12:27:36.990310+0000","last_became_peered":"2023-10-16T12:27:36.990310+0000","last_unstale":"2023-10-16T12:27:38.024610+0000","last_undegraded":"2023-10-16T12:27:38.024610+0000","last_fullsized":"2023-10-16T12:27:38.024610+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_clean_scrub_stamp":"2023-10-16T12:27:35.962232+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:58:11.208246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.023667+0000","last_change":"2023-10-16T12:27:38.018003+0000","last_active":"2023-10-16T12:27:38.023667+0000","last_peered":"2023-10-16T12:27:38.023667+0000","last_clean":"2023-10-16T12:27:38.023667+0000","last_became_active":"2023-10-16T12:27:38.015244+0000","last_became_peered":"2023-10-16T12:27:38.015244+0000","last_unstale":"2023-10-16T12:27:38.023667+0000","last_undegraded":"2023-10-16T12:27:38.023667+0000","last_fullsized":"2023-10-16T12:27:38.023667+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:43:23.547480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.005040+0000","last_change":"2023-10-16T12:27:36.984442+0000","last_active":"2023-10-16T12:27:39.005040+0000","last_peered":"2023-10-16T12:27:39.005040+0000","last_clean":"2023-10-16T12:27:39.005040+0000","last_became_active":"2023-10-16T12:27:36.984169+0000","last_became_peered":"2023-10-16T12:27:36.984169+0000","last_unstale":"2023-10-16T12:27:39.005040+0000","last_undegraded":"2023-10-16T12:27:39.005040+0000","last_fullsized":"2023-10-16T12:27:39.005040+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_clean_scrub_stamp":"2023-10-16T12:27:35.962232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:16:40.357072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.016405+0000","last_change":"2023-10-16T12:27:38.013871+0000","last_active":"2023-10-16T12:27:38.016405+0000","last_peered":"2023-10-16T12:27:38.016405+0000","last_clean":"2023-10-16T12:27:38.016405+0000","last_became_active":"2023-10-16T12:27:38.013586+0000","last_became_peered":"2023-10-16T12:27:38.013586+0000","last_unstale":"2023-10-16T12:27:38.016405+0000","last_undegraded":"2023-10-16T12:27:38.016405+0000","last_fullsized":"2023-10-16T12:27:38.016405+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:54:42.009262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.246374+0000","last_change":"2023-10-16T12:27:36.993517+0000","last_active":"2023-10-16T12:27:39.246374+0000","last_peered":"2023-10-16T12:27:39.246374+0000","last_clean":"2023-10-16T12:27:39.246374+0000","last_became_active":"2023-10-16T12:27:36.993193+0000","last_became_peered":"2023-10-16T12:27:36.993193+0000","last_unstale":"2023-10-16T12:27:39.246374+0000","last_undegraded":"2023-10-16T12:27:39.246374+0000","last_fullsized":"2023-10-16T12:27:39.246374+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_clean_scrub_stamp":"2023-10-16T12:27:35.962232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:25:18.221168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.029964+0000","last_change":"2023-10-16T12:27:38.018760+0000","last_active":"2023-10-16T12:27:38.029964+0000","last_peered":"2023-10-16T12:27:38.029964+0000","last_clean":"2023-10-16T12:27:38.029964+0000","last_became_active":"2023-10-16T12:27:38.018513+0000","last_became_peered":"2023-10-16T12:27:38.018513+0000","last_unstale":"2023-10-16T12:27:38.029964+0000","last_undegraded":"2023-10-16T12:27:38.029964+0000","last_fullsized":"2023-10-16T12:27:38.029964+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:54:36.370568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.10","version":"111'8","reported_seq":20,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.246738+0000","last_change":"2023-10-16T12:27:36.992178+0000","last_active":"2023-10-16T12:27:39.246738+0000","last_peered":"2023-10-16T12:27:39.246738+0000","last_clean":"2023-10-16T12:27:39.246738+0000","last_became_active":"2023-10-16T12:27:36.991857+0000","last_became_peered":"2023-10-16T12:27:36.991857+0000","last_unstale":"2023-10-16T12:27:39.246738+0000","last_undegraded":"2023-10-16T12:27:39.246738+0000","last_fullsized":"2023-10-16T12:27:39.246738+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_clean_scrub_stamp":"2023-10-16T12:27:35.962232+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:22:59.618322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.004949+0000","last_change":"2023-10-16T12:27:38.025560+0000","last_active":"2023-10-16T12:27:39.004949+0000","last_peered":"2023-10-16T12:27:39.004949+0000","last_clean":"2023-10-16T12:27:39.004949+0000","last_became_active":"2023-10-16T12:27:38.025179+0000","last_became_peered":"2023-10-16T12:27:38.025179+0000","last_unstale":"2023-10-16T12:27:39.004949+0000","last_undegraded":"2023-10-16T12:27:39.004949+0000","last_fullsized":"2023-10-16T12:27:39.004949+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:38:49.527365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.f","version":"111'1","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.019763+0000","last_change":"2023-10-16T12:27:36.981258+0000","last_active":"2023-10-16T12:27:38.019763+0000","last_peered":"2023-10-16T12:27:38.019763+0000","last_clean":"2023-10-16T12:27:38.019763+0000","last_became_active":"2023-10-16T12:27:36.981002+0000","last_became_peered":"2023-10-16T12:27:36.981002+0000","last_unstale":"2023-10-16T12:27:38.019763+0000","last_undegraded":"2023-10-16T12:27:38.019763+0000","last_fullsized":"2023-10-16T12:27:38.019763+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_clean_scrub_stamp":"2023-10-16T12:27:35.962232+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:45:48.648472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.010760+0000","last_change":"2023-10-16T12:27:38.004689+0000","last_active":"2023-10-16T12:27:38.010760+0000","last_peered":"2023-10-16T12:27:38.010760+0000","last_clean":"2023-10-16T12:27:38.010760+0000","last_became_active":"2023-10-16T12:27:38.003921+0000","last_became_peered":"2023-10-16T12:27:38.003921+0000","last_unstale":"2023-10-16T12:27:38.010760+0000","last_undegraded":"2023-10-16T12:27:38.010760+0000","last_fullsized":"2023-10-16T12:27:38.010760+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:47:47.319973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.e","version":"111'2","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.021067+0000","last_change":"2023-10-16T12:27:36.985355+0000","last_active":"2023-10-16T12:27:38.021067+0000","last_peered":"2023-10-16T12:27:38.021067+0000","last_clean":"2023-10-16T12:27:38.021067+0000","last_became_active":"2023-10-16T12:27:36.985049+0000","last_became_peered":"2023-10-16T12:27:36.985049+0000","last_unstale":"2023-10-16T12:27:38.021067+0000","last_undegraded":"2023-10-16T12:27:38.021067+0000","last_fullsized":"2023-10-16T12:27:38.021067+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_clean_scrub_stamp":"2023-10-16T12:27:35.962232+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:36:46.367885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.024143+0000","last_change":"2023-10-16T12:27:38.023358+0000","last_active":"2023-10-16T12:27:38.024143+0000","last_peered":"2023-10-16T12:27:38.024143+0000","last_clean":"2023-10-16T12:27:38.024143+0000","last_became_active":"2023-10-16T12:27:38.023064+0000","last_became_peered":"2023-10-16T12:27:38.023064+0000","last_unstale":"2023-10-16T12:27:38.024143+0000","last_undegraded":"2023-10-16T12:27:38.024143+0000","last_fullsized":"2023-10-16T12:27:38.024143+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:03:31.380576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.d","version":"111'1","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.029925+0000","last_change":"2023-10-16T12:27:36.987805+0000","last_active":"2023-10-16T12:27:38.029925+0000","last_peered":"2023-10-16T12:27:38.029925+0000","last_clean":"2023-10-16T12:27:38.029925+0000","last_became_active":"2023-10-16T12:27:36.987428+0000","last_became_peered":"2023-10-16T12:27:36.987428+0000","last_unstale":"2023-10-16T12:27:38.029925+0000","last_undegraded":"2023-10-16T12:27:38.029925+0000","last_fullsized":"2023-10-16T12:27:38.029925+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_clean_scrub_stamp":"2023-10-16T12:27:35.962232+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:09:49.678551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.004989+0000","last_change":"2023-10-16T12:27:38.027468+0000","last_active":"2023-10-16T12:27:39.004989+0000","last_peered":"2023-10-16T12:27:39.004989+0000","last_clean":"2023-10-16T12:27:39.004989+0000","last_became_active":"2023-10-16T12:27:38.027198+0000","last_became_peered":"2023-10-16T12:27:38.027198+0000","last_unstale":"2023-10-16T12:27:39.004989+0000","last_undegraded":"2023-10-16T12:27:39.004989+0000","last_fullsized":"2023-10-16T12:27:39.004989+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:28:16.544155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.b","version":"111'2","reported_seq":14,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.067529+0000","last_change":"2023-10-16T12:27:36.997826+0000","last_active":"2023-10-16T12:27:39.067529+0000","last_peered":"2023-10-16T12:27:39.067529+0000","last_clean":"2023-10-16T12:27:39.067529+0000","last_became_active":"2023-10-16T12:27:36.997563+0000","last_became_peered":"2023-10-16T12:27:36.997563+0000","last_unstale":"2023-10-16T12:27:39.067529+0000","last_undegraded":"2023-10-16T12:27:39.067529+0000","last_fullsized":"2023-10-16T12:27:39.067529+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_clean_scrub_stamp":"2023-10-16T12:27:35.962232+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:14:09.324779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.067489+0000","last_change":"2023-10-16T12:27:38.021150+0000","last_active":"2023-10-16T12:27:39.067489+0000","last_peered":"2023-10-16T12:27:39.067489+0000","last_clean":"2023-10-16T12:27:39.067489+0000","last_became_active":"2023-10-16T12:27:38.020516+0000","last_became_peered":"2023-10-16T12:27:38.020516+0000","last_unstale":"2023-10-16T12:27:39.067489+0000","last_undegraded":"2023-10-16T12:27:39.067489+0000","last_fullsized":"2023-10-16T12:27:39.067489+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:38:22.530317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:37.992948+0000","last_change":"2023-10-16T12:27:36.995162+0000","last_active":"2023-10-16T12:27:37.992948+0000","last_peered":"2023-10-16T12:27:37.992948+0000","last_clean":"2023-10-16T12:27:37.992948+0000","last_became_active":"2023-10-16T12:27:36.994741+0000","last_became_peered":"2023-10-16T12:27:36.994741+0000","last_unstale":"2023-10-16T12:27:37.992948+0000","last_undegraded":"2023-10-16T12:27:37.992948+0000","last_fullsized":"2023-10-16T12:27:37.992948+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_clean_scrub_stamp":"2023-10-16T12:27:35.962232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:31:50.471272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.067587+0000","last_change":"2023-10-16T12:27:38.020395+0000","last_active":"2023-10-16T12:27:39.067587+0000","last_peered":"2023-10-16T12:27:39.067587+0000","last_clean":"2023-10-16T12:27:39.067587+0000","last_became_active":"2023-10-16T12:27:38.019999+0000","last_became_peered":"2023-10-16T12:27:38.019999+0000","last_unstale":"2023-10-16T12:27:39.067587+0000","last_undegraded":"2023-10-16T12:27:39.067587+0000","last_fullsized":"2023-10-16T12:27:39.067587+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:36:47.372426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.067608+0000","last_change":"2023-10-16T12:27:36.987144+0000","last_active":"2023-10-16T12:27:39.067608+0000","last_peered":"2023-10-16T12:27:39.067608+0000","last_clean":"2023-10-16T12:27:39.067608+0000","last_became_active":"2023-10-16T12:27:36.986846+0000","last_became_peered":"2023-10-16T12:27:36.986846+0000","last_unstale":"2023-10-16T12:27:39.067608+0000","last_undegraded":"2023-10-16T12:27:39.067608+0000","last_fullsized":"2023-10-16T12:27:39.067608+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_clean_scrub_stamp":"2023-10-16T12:27:35.962232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:56:24.192112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.005359+0000","last_change":"2023-10-16T12:27:38.009266+0000","last_active":"2023-10-16T12:27:39.005359+0000","last_peered":"2023-10-16T12:27:39.005359+0000","last_clean":"2023-10-16T12:27:39.005359+0000","last_became_active":"2023-10-16T12:27:38.008764+0000","last_became_peered":"2023-10-16T12:27:38.008764+0000","last_unstale":"2023-10-16T12:27:39.005359+0000","last_undegraded":"2023-10-16T12:27:39.005359+0000","last_fullsized":"2023-10-16T12:27:39.005359+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:52:37.606383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:37.992616+0000","last_change":"2023-10-16T12:27:36.986348+0000","last_active":"2023-10-16T12:27:37.992616+0000","last_peered":"2023-10-16T12:27:37.992616+0000","last_clean":"2023-10-16T12:27:37.992616+0000","last_became_active":"2023-10-16T12:27:36.984643+0000","last_became_peered":"2023-10-16T12:27:36.984643+0000","last_unstale":"2023-10-16T12:27:37.992616+0000","last_undegraded":"2023-10-16T12:27:37.992616+0000","last_fullsized":"2023-10-16T12:27:37.992616+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_clean_scrub_stamp":"2023-10-16T12:27:35.962232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:07:02.729770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.011340+0000","last_change":"2023-10-16T12:27:38.009764+0000","last_active":"2023-10-16T12:27:38.011340+0000","last_peered":"2023-10-16T12:27:38.011340+0000","last_clean":"2023-10-16T12:27:38.011340+0000","last_became_active":"2023-10-16T12:27:38.009487+0000","last_became_peered":"2023-10-16T12:27:38.009487+0000","last_unstale":"2023-10-16T12:27:38.011340+0000","last_undegraded":"2023-10-16T12:27:38.011340+0000","last_fullsized":"2023-10-16T12:27:38.011340+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:26:03.624784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.7","version":"111'2","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.019638+0000","last_change":"2023-10-16T12:27:36.985562+0000","last_active":"2023-10-16T12:27:38.019638+0000","last_peered":"2023-10-16T12:27:38.019638+0000","last_clean":"2023-10-16T12:27:38.019638+0000","last_became_active":"2023-10-16T12:27:36.985277+0000","last_became_peered":"2023-10-16T12:27:36.985277+0000","last_unstale":"2023-10-16T12:27:38.019638+0000","last_undegraded":"2023-10-16T12:27:38.019638+0000","last_fullsized":"2023-10-16T12:27:38.019638+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_clean_scrub_stamp":"2023-10-16T12:27:35.962232+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:31:48.599920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.005705+0000","last_change":"2023-10-16T12:27:38.026401+0000","last_active":"2023-10-16T12:27:39.005705+0000","last_peered":"2023-10-16T12:27:39.005705+0000","last_clean":"2023-10-16T12:27:39.005705+0000","last_became_active":"2023-10-16T12:27:38.026112+0000","last_became_peered":"2023-10-16T12:27:38.026112+0000","last_unstale":"2023-10-16T12:27:39.005705+0000","last_undegraded":"2023-10-16T12:27:39.005705+0000","last_fullsized":"2023-10-16T12:27:39.005705+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:00:46.759996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.015690+0000","last_change":"2023-10-16T12:27:36.985411+0000","last_active":"2023-10-16T12:27:38.015690+0000","last_peered":"2023-10-16T12:27:38.015690+0000","last_clean":"2023-10-16T12:27:38.015690+0000","last_became_active":"2023-10-16T12:27:36.985143+0000","last_became_peered":"2023-10-16T12:27:36.985143+0000","last_unstale":"2023-10-16T12:27:38.015690+0000","last_undegraded":"2023-10-16T12:27:38.015690+0000","last_fullsized":"2023-10-16T12:27:38.015690+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_clean_scrub_stamp":"2023-10-16T12:27:35.962232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:39:50.980664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.005406+0000","last_change":"2023-10-16T12:27:38.008382+0000","last_active":"2023-10-16T12:27:39.005406+0000","last_peered":"2023-10-16T12:27:39.005406+0000","last_clean":"2023-10-16T12:27:39.005406+0000","last_became_active":"2023-10-16T12:27:38.008039+0000","last_became_peered":"2023-10-16T12:27:38.008039+0000","last_unstale":"2023-10-16T12:27:39.005406+0000","last_undegraded":"2023-10-16T12:27:39.005406+0000","last_fullsized":"2023-10-16T12:27:39.005406+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:09:13.873193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.007171+0000","last_change":"2023-10-16T12:27:36.990124+0000","last_active":"2023-10-16T12:27:38.007171+0000","last_peered":"2023-10-16T12:27:38.007171+0000","last_clean":"2023-10-16T12:27:38.007171+0000","last_became_active":"2023-10-16T12:27:36.989815+0000","last_became_peered":"2023-10-16T12:27:36.989815+0000","last_unstale":"2023-10-16T12:27:38.007171+0000","last_undegraded":"2023-10-16T12:27:38.007171+0000","last_fullsized":"2023-10-16T12:27:38.007171+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_clean_scrub_stamp":"2023-10-16T12:27:35.962232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:01:24.361594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.067971+0000","last_change":"2023-10-16T12:27:38.021334+0000","last_active":"2023-10-16T12:27:39.067971+0000","last_peered":"2023-10-16T12:27:39.067971+0000","last_clean":"2023-10-16T12:27:39.067971+0000","last_became_active":"2023-10-16T12:27:38.020967+0000","last_became_peered":"2023-10-16T12:27:38.020967+0000","last_unstale":"2023-10-16T12:27:39.067971+0000","last_undegraded":"2023-10-16T12:27:39.067971+0000","last_fullsized":"2023-10-16T12:27:39.067971+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:30:42.276938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:37.999743+0000","last_change":"2023-10-16T12:27:36.980048+0000","last_active":"2023-10-16T12:27:37.999743+0000","last_peered":"2023-10-16T12:27:37.999743+0000","last_clean":"2023-10-16T12:27:37.999743+0000","last_became_active":"2023-10-16T12:27:36.979754+0000","last_became_peered":"2023-10-16T12:27:36.979754+0000","last_unstale":"2023-10-16T12:27:37.999743+0000","last_undegraded":"2023-10-16T12:27:37.999743+0000","last_fullsized":"2023-10-16T12:27:37.999743+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_clean_scrub_stamp":"2023-10-16T12:27:35.962232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:26:44.439272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.007806+0000","last_change":"2023-10-16T12:27:38.005483+0000","last_active":"2023-10-16T12:27:38.007806+0000","last_peered":"2023-10-16T12:27:38.007806+0000","last_clean":"2023-10-16T12:27:38.007806+0000","last_became_active":"2023-10-16T12:27:38.005175+0000","last_became_peered":"2023-10-16T12:27:38.005175+0000","last_unstale":"2023-10-16T12:27:38.007806+0000","last_undegraded":"2023-10-16T12:27:38.007806+0000","last_fullsized":"2023-10-16T12:27:38.007806+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:02:09.243246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.005164+0000","last_change":"2023-10-16T12:27:36.989102+0000","last_active":"2023-10-16T12:27:39.005164+0000","last_peered":"2023-10-16T12:27:39.005164+0000","last_clean":"2023-10-16T12:27:39.005164+0000","last_became_active":"2023-10-16T12:27:36.988556+0000","last_became_peered":"2023-10-16T12:27:36.988556+0000","last_unstale":"2023-10-16T12:27:39.005164+0000","last_undegraded":"2023-10-16T12:27:39.005164+0000","last_fullsized":"2023-10-16T12:27:39.005164+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_clean_scrub_stamp":"2023-10-16T12:27:35.962232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:37:24.745515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.006110+0000","last_change":"2023-10-16T12:27:38.016951+0000","last_active":"2023-10-16T12:27:39.006110+0000","last_peered":"2023-10-16T12:27:39.006110+0000","last_clean":"2023-10-16T12:27:39.006110+0000","last_became_active":"2023-10-16T12:27:38.016244+0000","last_became_peered":"2023-10-16T12:27:38.016244+0000","last_unstale":"2023-10-16T12:27:39.006110+0000","last_undegraded":"2023-10-16T12:27:39.006110+0000","last_fullsized":"2023-10-16T12:27:39.006110+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:28:18.102314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.067581+0000","last_change":"2023-10-16T12:27:36.998503+0000","last_active":"2023-10-16T12:27:39.067581+0000","last_peered":"2023-10-16T12:27:39.067581+0000","last_clean":"2023-10-16T12:27:39.067581+0000","last_became_active":"2023-10-16T12:27:36.998252+0000","last_became_peered":"2023-10-16T12:27:36.998252+0000","last_unstale":"2023-10-16T12:27:39.067581+0000","last_undegraded":"2023-10-16T12:27:39.067581+0000","last_fullsized":"2023-10-16T12:27:39.067581+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_clean_scrub_stamp":"2023-10-16T12:27:35.962232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:47:05.480286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.023761+0000","last_change":"2023-10-16T12:27:38.022432+0000","last_active":"2023-10-16T12:27:38.023761+0000","last_peered":"2023-10-16T12:27:38.023761+0000","last_clean":"2023-10-16T12:27:38.023761+0000","last_became_active":"2023-10-16T12:27:38.022143+0000","last_became_peered":"2023-10-16T12:27:38.022143+0000","last_unstale":"2023-10-16T12:27:38.023761+0000","last_undegraded":"2023-10-16T12:27:38.023761+0000","last_fullsized":"2023-10-16T12:27:38.023761+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:43:23.547480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.002832+0000","last_change":"2023-10-16T12:27:36.980107+0000","last_active":"2023-10-16T12:27:38.002832+0000","last_peered":"2023-10-16T12:27:38.002832+0000","last_clean":"2023-10-16T12:27:38.002832+0000","last_became_active":"2023-10-16T12:27:36.979805+0000","last_became_peered":"2023-10-16T12:27:36.979805+0000","last_unstale":"2023-10-16T12:27:38.002832+0000","last_undegraded":"2023-10-16T12:27:38.002832+0000","last_fullsized":"2023-10-16T12:27:38.002832+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_clean_scrub_stamp":"2023-10-16T12:27:35.962232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:17:04.802888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.005083+0000","last_change":"2023-10-16T12:27:38.020021+0000","last_active":"2023-10-16T12:27:39.005083+0000","last_peered":"2023-10-16T12:27:39.005083+0000","last_clean":"2023-10-16T12:27:39.005083+0000","last_became_active":"2023-10-16T12:27:38.019699+0000","last_became_peered":"2023-10-16T12:27:38.019699+0000","last_unstale":"2023-10-16T12:27:39.005083+0000","last_undegraded":"2023-10-16T12:27:39.005083+0000","last_fullsized":"2023-10-16T12:27:39.005083+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:51:39.535828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:37.997226+0000","last_change":"2023-10-16T12:27:36.993811+0000","last_active":"2023-10-16T12:27:37.997226+0000","last_peered":"2023-10-16T12:27:37.997226+0000","last_clean":"2023-10-16T12:27:37.997226+0000","last_became_active":"2023-10-16T12:27:36.993494+0000","last_became_peered":"2023-10-16T12:27:36.993494+0000","last_unstale":"2023-10-16T12:27:37.997226+0000","last_undegraded":"2023-10-16T12:27:37.997226+0000","last_fullsized":"2023-10-16T12:27:37.997226+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_clean_scrub_stamp":"2023-10-16T12:27:35.962232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:13:56.125305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.017955+0000","last_change":"2023-10-16T12:27:38.016474+0000","last_active":"2023-10-16T12:27:38.017955+0000","last_peered":"2023-10-16T12:27:38.017955+0000","last_clean":"2023-10-16T12:27:38.017955+0000","last_became_active":"2023-10-16T12:27:38.015428+0000","last_became_peered":"2023-10-16T12:27:38.015428+0000","last_unstale":"2023-10-16T12:27:38.017955+0000","last_undegraded":"2023-10-16T12:27:38.017955+0000","last_fullsized":"2023-10-16T12:27:38.017955+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:25:17.395792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.246727+0000","last_change":"2023-10-16T12:27:36.987949+0000","last_active":"2023-10-16T12:27:39.246727+0000","last_peered":"2023-10-16T12:27:39.246727+0000","last_clean":"2023-10-16T12:27:39.246727+0000","last_became_active":"2023-10-16T12:27:36.987631+0000","last_became_peered":"2023-10-16T12:27:36.987631+0000","last_unstale":"2023-10-16T12:27:39.246727+0000","last_undegraded":"2023-10-16T12:27:39.246727+0000","last_fullsized":"2023-10-16T12:27:39.246727+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_clean_scrub_stamp":"2023-10-16T12:27:35.962232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:40:06.631161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"102'183","reported_seq":484,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-16T12:27:36.980888+0000","last_change":"2023-10-16T12:10:34.746820+0000","last_active":"2023-10-16T12:27:36.980888+0000","last_peered":"2023-10-16T12:27:36.980888+0000","last_clean":"2023-10-16T12:27:36.980888+0000","last_became_active":"2023-10-16T12:10:34.745574+0000","last_became_peered":"2023-10-16T12:10:34.745574+0000","last_unstale":"2023-10-16T12:27:36.980888+0000","last_undegraded":"2023-10-16T12:27:36.980888+0000","last_fullsized":"2023-10-16T12:27:36.980888+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_clean_scrub_stamp":"2023-10-16T12:10:33.577111+0000","objects_scrubbed":0,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:14.461986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":692,"num_read_kb":1336,"num_write":279,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":14,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":692,"num_read_kb":1336,"num_write":279,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":183,"ondisk_log_size":183,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575053,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59476,"kb_used_data":1924,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312364,"statfs":{"total":96636764160,"available":96575860736,"internally_reserved":0,"allocated":1970176,"data_stored":1274531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58915275},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:27:08 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.57099999999999995,"15min":0.58699999999999997},"min":{"1min":0.31,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.76300000000000001,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.56699999999999995,"15min":0.58099999999999996},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.78100000000000003,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.51900000000000002}]},{"osd":1,"last update":"Mon Oct 16 12:27:08 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.57499999999999996,"15min":0.60199999999999998},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.247},"max":{"1min":0.88100000000000001,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.59699999999999998,"15min":0.60099999999999998},"min":{"1min":0.377,"5min":0.33200000000000002,"15min":0.32400000000000001},"max":{"1min":0.76700000000000002,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.59299999999999997}]},{"osd":2,"last update":"Mon Oct 16 12:27:08 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.65200000000000002,"15min":0.65900000000000003},"min":{"1min":0.46999999999999997,"5min":0.40699999999999997,"15min":0.36099999999999999},"max":{"1min":0.79900000000000004,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.65600000000000003,"15min":0.66600000000000004},"min":{"1min":0.42899999999999999,"5min":0.376,"15min":0.36699999999999999},"max":{"1min":0.88,"5min":0.94599999999999995,"15min":0.96699999999999997},"last":0.54700000000000004}]},{"osd":3,"last update":"Mon Oct 16 12:27:08 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.60499999999999998,"15min":0.61699999999999999},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.82899999999999996,"5min":0.92200000000000004,"15min":1.1000000000000001},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.64900000000000002,"15min":0.63800000000000001},"min":{"1min":0.42199999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.84999999999999998,"5min":1.2509999999999999,"15min":1.2509999999999999},"last":0.67200000000000004}]},{"osd":4,"last update":"Mon Oct 16 12:27:08 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.55300000000000005,"15min":0.52900000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.28499999999999998},"max":{"1min":0.78800000000000003,"5min":1.383,"15min":1.383},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.628,"15min":0.60599999999999998},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.307},"max":{"1min":0.93600000000000005,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.70199999999999996}]},{"osd":5,"last update":"Mon Oct 16 12:27:08 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.57099999999999995,"15min":0.54000000000000004},"min":{"1min":0.33000000000000002,"5min":0.314,"15min":0.29799999999999999},"max":{"1min":0.90800000000000003,"5min":1.3109999999999999,"15min":1.3109999999999999},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.60099999999999998,"15min":0.58499999999999996},"min":{"1min":0.34699999999999998,"5min":0.33500000000000002,"15min":0.32400000000000001},"max":{"1min":0.93600000000000005,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":0.68899999999999995}]},{"osd":6,"last update":"Mon Oct 16 12:27:08 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.64500000000000002,"15min":0.62},"min":{"1min":0.41299999999999998,"5min":0.39300000000000002,"15min":0.36399999999999999},"max":{"1min":0.92900000000000005,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.65300000000000002,"15min":0.627},"min":{"1min":0.36799999999999999,"5min":0.34599999999999997,"15min":0.32300000000000001},"max":{"1min":0.97199999999999998,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.629}]}]},{"osd":6,"up_from":13,"seq":55834575051,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32540,"kb_used_data":1356,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339300,"statfs":{"total":96636764160,"available":96603443200,"internally_reserved":0,"allocated":1388544,"data_stored":686323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:27:02 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.58399999999999996,"15min":0.61099999999999999},"min":{"1min":0.29299999999999998,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.85199999999999998,"5min":1.196,"15min":1.196},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.58299999999999996,"15min":0.622},"min":{"1min":0.38500000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.80800000000000005,"5min":0.997,"15min":0.997},"last":0.83399999999999996}]},{"osd":1,"last update":"Mon Oct 16 12:27:02 2023","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.53600000000000003,"15min":0.57099999999999995},"min":{"1min":0.34399999999999997,"5min":0.32800000000000001,"15min":0.29399999999999998},"max":{"1min":0.66500000000000004,"5min":0.83899999999999997,"15min":0.875},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.55900000000000005,"15min":0.58799999999999997},"min":{"1min":0.32800000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.70599999999999996,"5min":1.091,"15min":1.091},"last":0.70699999999999996}]},{"osd":2,"last update":"Mon Oct 16 12:27:02 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.56899999999999995,"15min":0.59199999999999997},"min":{"1min":0.36799999999999999,"5min":0.36199999999999999,"15min":0.34499999999999997},"max":{"1min":0.76100000000000001,"5min":0.91200000000000003,"15min":0.97099999999999997},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.55600000000000005,"15min":0.59999999999999998},"min":{"1min":0.35499999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.85999999999999999,"5min":1.1539999999999999,"15min":2.3879999999999999},"last":0.748}]},{"osd":3,"last update":"Mon Oct 16 12:27:02 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.59399999999999997,"15min":0.625},"min":{"1min":0.37,"5min":0.34200000000000003,"15min":0.33500000000000002},"max":{"1min":0.88,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.629,"5min":0.66300000000000003,"15min":0.67900000000000005},"min":{"1min":0.45200000000000001,"5min":0.40899999999999997,"15min":0.40100000000000002},"max":{"1min":0.90600000000000003,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.88500000000000001}]},{"osd":4,"last update":"Mon Oct 16 12:27:02 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.53100000000000003,"15min":0.52100000000000002},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.29899999999999999},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.435},{"interface":"front","average":{"1min":0.63,"5min":0.60199999999999998,"15min":0.59699999999999998},"min":{"1min":0.49199999999999999,"5min":0.32300000000000001,"15min":0.30299999999999999},"max":{"1min":0.82299999999999995,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.68000000000000005}]},{"osd":5,"last update":"Mon Oct 16 12:27:02 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.58799999999999997,"15min":0.57599999999999996},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.30099999999999999},"max":{"1min":0.98799999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.63,"15min":0.625},"min":{"1min":0.47899999999999998,"5min":0.35099999999999998,"15min":0.311},"max":{"1min":1.0029999999999999,"5min":1.3540000000000001,"15min":1.3540000000000001},"last":0.60599999999999998}]},{"osd":7,"last update":"Mon Oct 16 12:27:02 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.58399999999999996,"15min":0.57299999999999995},"min":{"1min":0.435,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.92700000000000005,"5min":1.29,"15min":1.29},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.622,"15min":0.61799999999999999},"min":{"1min":0.46999999999999997,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.57399999999999995}]}]},{"osd":1,"up_from":13,"seq":55834575051,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32524,"kb_used_data":1340,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339316,"statfs":{"total":96636764160,"available":96603459584,"internally_reserved":0,"allocated":1372160,"data_stored":684177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":31914441},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:27:00 2023","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.36899999999999999,"15min":0.376},"min":{"1min":0.246,"5min":0.21299999999999999,"15min":0.186},"max":{"1min":0.53300000000000003,"5min":0.63600000000000001,"15min":0.66000000000000003},"last":0.372},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.44500000000000001,"15min":0.45800000000000002},"min":{"1min":0.28399999999999997,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.502,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.46300000000000002}]},{"osd":2,"last update":"Mon Oct 16 12:27:00 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.40500000000000003,"15min":0.42599999999999999},"min":{"1min":0.251,"5min":0.25,"15min":0.215},"max":{"1min":0.45000000000000001,"5min":0.69199999999999995,"15min":0.93000000000000005},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.45300000000000001,"15min":0.46400000000000002},"min":{"1min":0.32400000000000001,"5min":0.28199999999999997,"15min":0.24399999999999999},"max":{"1min":0.54500000000000004,"5min":0.69199999999999995,"15min":0.98099999999999998},"last":0.47799999999999998}]},{"osd":3,"last update":"Mon Oct 16 12:27:00 2023","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.38500000000000001,"15min":0.40100000000000002},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.57999999999999996,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.42799999999999999,"15min":0.45500000000000002},"min":{"1min":0.26100000000000001,"5min":0.252,"15min":0.22900000000000001},"max":{"1min":0.58699999999999997,"5min":0.91200000000000003,"15min":1.127},"last":0.498}]},{"osd":4,"last update":"Mon Oct 16 12:27:00 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61099999999999999,"15min":0.63300000000000001},"min":{"1min":0.41799999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.79800000000000004,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.752},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.62,"15min":0.63900000000000001},"min":{"1min":0.41899999999999998,"5min":0.39100000000000001,"15min":0.34699999999999998},"max":{"1min":0.89600000000000002,"5min":0.92000000000000004,"15min":1.089},"last":0.63800000000000001}]},{"osd":5,"last update":"Mon Oct 16 12:27:00 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.63600000000000001,"15min":0.66500000000000004},"min":{"1min":0.496,"5min":0.38300000000000001,"15min":0.28499999999999998},"max":{"1min":0.76700000000000002,"5min":1.069,"15min":1.069},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67300000000000004,"15min":0.68799999999999994},"min":{"1min":0.439,"5min":0.38900000000000001,"15min":0.32500000000000001},"max":{"1min":0.94499999999999995,"5min":1.0620000000000001,"15min":1.0649999999999999},"last":0.81100000000000005}]},{"osd":6,"last update":"Mon Oct 16 12:27:00 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.63,"15min":0.65400000000000003},"min":{"1min":0.48699999999999999,"5min":0.378,"15min":0.378},"max":{"1min":0.77800000000000002,"5min":0.94999999999999996,"15min":1.0129999999999999},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.629,"5min":0.65300000000000002,"15min":0.66200000000000003},"min":{"1min":0.55100000000000005,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.83299999999999996,"5min":0.96899999999999997,"15min":1.048},"last":0.68999999999999995}]},{"osd":7,"last update":"Mon Oct 16 12:27:00 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.67200000000000004,"15min":0.68700000000000006},"min":{"1min":0.42099999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.98899999999999999,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.67600000000000005,"15min":0.70299999999999996},"min":{"1min":0.45700000000000002,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.876,"5min":0.94899999999999995,"15min":1.167},"last":0.78100000000000003}]}]},{"osd":0,"up_from":13,"seq":55834575052,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37280,"kb_used_data":1940,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334560,"statfs":{"total":96636764160,"available":96598589440,"internally_reserved":0,"allocated":1986560,"data_stored":1276793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 12:26:55 2023","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"min":{"1min":0.22500000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.40999999999999998,"15min":0.41799999999999998},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23200000000000001},"max":{"1min":0.64500000000000002,"5min":0.71799999999999997,"15min":0.73099999999999998},"last":0.27900000000000003}]},{"osd":2,"last update":"Mon Oct 16 12:26:55 2023","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.42099999999999999,"15min":0.42499999999999999},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.44500000000000001,"15min":0.45100000000000001},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.317}]},{"osd":3,"last update":"Mon Oct 16 12:26:55 2023","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.42699999999999999,"15min":0.42199999999999999},"min":{"1min":0.26300000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.62,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.47799999999999998,"15min":0.48699999999999999},"min":{"1min":0.318,"5min":0.29099999999999998,"15min":0.27600000000000002},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.878},"last":0.41999999999999998}]},{"osd":4,"last update":"Mon Oct 16 12:26:55 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.64300000000000002,"15min":0.64800000000000002},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.89200000000000002,"5min":0.90400000000000003,"15min":1.0960000000000001},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.626,"15min":0.65500000000000003},"min":{"1min":0.374,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.84199999999999997,"5min":0.93799999999999994,"15min":1.1719999999999999},"last":0.47399999999999998}]},{"osd":5,"last update":"Mon Oct 16 12:26:55 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.63500000000000001,"15min":0.65000000000000002},"min":{"1min":0.373,"5min":0.373,"15min":0.34399999999999997},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":1.127},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.66500000000000004,"15min":0.67700000000000005},"min":{"1min":0.39400000000000002,"5min":0.39200000000000002,"15min":0.39000000000000001},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":1.1160000000000001},"last":0.55900000000000005}]},{"osd":6,"last update":"Mon Oct 16 12:26:55 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.66600000000000004,"15min":0.68000000000000005},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":1.079},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.65900000000000003,"15min":0.68300000000000005},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":1.1499999999999999},"last":0.46400000000000002}]},{"osd":7,"last update":"Mon Oct 16 12:26:55 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.67800000000000005,"15min":0.69899999999999995},"min":{"1min":0.436,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.995,"5min":0.995,"15min":1.0349999999999999},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.65700000000000003,"15min":0.68799999999999994},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":1.0569999999999999},"last":0.44500000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575052,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58884,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312956,"statfs":{"total":96636764160,"available":96576466944,"internally_reserved":0,"allocated":1368064,"data_stored":684107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:26:56 2023","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.36799999999999999,"15min":0.38400000000000001},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.62,"5min":0.62,"15min":0.73799999999999999},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.44400000000000001,"15min":0.45000000000000001},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.63400000000000001,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.36399999999999999}]},{"osd":1,"last update":"Mon Oct 16 12:26:56 2023","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.45100000000000001,"15min":0.46800000000000003},"min":{"1min":0.31,"5min":0.25600000000000001,"15min":0.254},"max":{"1min":0.57999999999999996,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.42499999999999999,"15min":0.435},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.246},"max":{"1min":0.54400000000000004,"5min":0.69699999999999995,"15min":0.76500000000000001},"last":0.379}]},{"osd":3,"last update":"Mon Oct 16 12:26:56 2023","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.40200000000000002,"15min":0.41599999999999998},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.60199999999999998,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.51000000000000001,"15min":0.51900000000000002},"min":{"1min":0.36599999999999999,"5min":0.318,"15min":0.246},"max":{"1min":0.67600000000000005,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.39000000000000001}]},{"osd":4,"last update":"Mon Oct 16 12:26:56 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.66500000000000004,"15min":0.68100000000000005},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.81200000000000006,"5min":1.3839999999999999,"15min":1.3839999999999999},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.66200000000000003,"15min":0.67900000000000005},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.84899999999999998,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.54400000000000004}]},{"osd":5,"last update":"Mon Oct 16 12:26:56 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.65800000000000003,"15min":0.66600000000000004},"min":{"1min":0.378,"5min":0.378,"15min":0.312},"max":{"1min":0.78700000000000003,"5min":1.093,"15min":1.093},"last":0.437},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.64600000000000002,"15min":0.66400000000000003},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.317},"max":{"1min":0.879,"5min":1.331,"15min":1.331},"last":0.55700000000000005}]},{"osd":6,"last update":"Mon Oct 16 12:26:56 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.63200000000000001,"15min":0.63800000000000001},"min":{"1min":0.37,"5min":0.37,"15min":0.34699999999999998},"max":{"1min":0.91900000000000004,"5min":0.99299999999999999,"15min":1.046},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.68400000000000005,"15min":0.69199999999999995},"min":{"1min":0.40799999999999997,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.82599999999999996,"5min":1.032,"15min":1.0389999999999999},"last":0.47099999999999997}]},{"osd":7,"last update":"Mon Oct 16 12:26:56 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.66500000000000004,"15min":0.68000000000000005},"min":{"1min":0.40300000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.879,"5min":0.97499999999999998,"15min":1.026},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.68100000000000005,"15min":0.69299999999999995},"min":{"1min":0.46700000000000003,"5min":0.442,"15min":0.40600000000000003},"max":{"1min":0.89600000000000002,"5min":1.145,"15min":1.145},"last":0.41599999999999998}]}]},{"osd":3,"up_from":13,"seq":55834575051,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36676,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335164,"statfs":{"total":96636764160,"available":96599207936,"internally_reserved":0,"allocated":1368064,"data_stored":684107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:27:04 2023","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.39300000000000002,"15min":0.39500000000000002},"min":{"1min":0.216,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.60699999999999998,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.40200000000000002,"15min":0.40400000000000003},"min":{"1min":0.26600000000000001,"5min":0.25900000000000001,"15min":0.249},"max":{"1min":0.47299999999999998,"5min":0.63100000000000001,"15min":0.73099999999999998},"last":0.32100000000000001}]},{"osd":1,"last update":"Mon Oct 16 12:27:04 2023","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.39700000000000002,"15min":0.40899999999999997},"min":{"1min":0.245,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.48299999999999998,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.375,"5min":0.44700000000000001,"15min":0.45800000000000002},"min":{"1min":0.27000000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.51400000000000001,"5min":0.67300000000000004,"15min":0.747},"last":0.41799999999999998}]},{"osd":2,"last update":"Mon Oct 16 12:27:04 2023","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.46400000000000002,"15min":0.46899999999999997},"min":{"1min":0.32200000000000001,"5min":0.307,"15min":0.26900000000000002},"max":{"1min":0.54800000000000004,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.44,"5min":0.48799999999999999,"15min":0.495},"min":{"1min":0.30599999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.73299999999999998,"5min":0.752,"15min":0.752},"last":0.35999999999999999}]},{"osd":4,"last update":"Mon Oct 16 12:27:04 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.622,"15min":0.64700000000000002},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.85399999999999998,"5min":0.92700000000000005,"15min":1.1539999999999999},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.64200000000000002,"15min":0.66700000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.78400000000000003,"5min":1.0409999999999999,"15min":1.234},"last":0.66100000000000003}]},{"osd":5,"last update":"Mon Oct 16 12:27:04 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.67000000000000004,"15min":0.68400000000000005},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.92800000000000005,"5min":1.0009999999999999,"15min":1.194},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.64500000000000002,"15min":0.66600000000000004},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.86499999999999999,"5min":0.97299999999999998,"15min":1.202},"last":0.46700000000000003}]},{"osd":6,"last update":"Mon Oct 16 12:27:04 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.65900000000000003,"15min":0.67400000000000004},"min":{"1min":0.42999999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.71099999999999997,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.626},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.68500000000000005,"15min":0.70499999999999996},"min":{"1min":0.45100000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.89600000000000002,"5min":1.0349999999999999,"15min":1.1559999999999999},"last":0.53200000000000003}]},{"osd":7,"last update":"Mon Oct 16 12:27:04 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.66200000000000003,"15min":0.68700000000000006},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.38},"max":{"1min":0.78400000000000003,"5min":1.0189999999999999,"15min":1.2430000000000001},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.69999999999999996,"15min":0.70599999999999996},"min":{"1min":0.44,"5min":0.44,"15min":0.42699999999999999},"max":{"1min":0.89700000000000002,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.57399999999999995}]}]},{"osd":4,"up_from":13,"seq":55834575050,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32524,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339316,"statfs":{"total":96636764160,"available":96603459584,"internally_reserved":0,"allocated":1376256,"data_stored":684267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:27:02 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.56699999999999995,"15min":0.58799999999999997},"min":{"1min":0.32000000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.78000000000000003,"5min":0.85099999999999998,"15min":0.94599999999999995},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.57099999999999995,"15min":0.60499999999999998},"min":{"1min":0.32800000000000001,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.83199999999999996,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.50600000000000001}]},{"osd":1,"last update":"Mon Oct 16 12:27:02 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.57599999999999996,"15min":0.60899999999999999},"min":{"1min":0.36399999999999999,"5min":0.35499999999999998,"15min":0.28499999999999998},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.92600000000000005},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.59999999999999998,"15min":0.621},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.33600000000000002},"max":{"1min":0.80300000000000005,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.53600000000000003}]},{"osd":2,"last update":"Mon Oct 16 12:27:02 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.60199999999999998,"15min":0.626},"min":{"1min":0.39800000000000002,"5min":0.35399999999999998,"15min":0.32000000000000001},"max":{"1min":0.83399999999999996,"5min":0.93000000000000005,"15min":1.012},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.59399999999999997,"15min":0.626},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.34100000000000003},"max":{"1min":0.751,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.373}]},{"osd":3,"last update":"Mon Oct 16 12:27:02 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.58099999999999996,"15min":0.60699999999999998},"min":{"1min":0.40899999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.71999999999999997,"5min":0.79500000000000004,"15min":0.93200000000000005},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.63,"5min":0.627,"15min":0.64100000000000001},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.97199999999999998},"last":0.56299999999999994}]},{"osd":5,"last update":"Mon Oct 16 12:27:02 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.499,"15min":0.497},"min":{"1min":0.377,"5min":0.34000000000000002,"15min":0.27600000000000002},"max":{"1min":0.79700000000000004,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.34200000000000003,"5min":0.316,"15min":0.29599999999999999},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.442}]},{"osd":6,"last update":"Mon Oct 16 12:27:02 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.55100000000000005,"15min":0.55400000000000005},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.28699999999999998},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.63},{"interface":"front","average":{"1min":0.628,"5min":0.63300000000000001,"15min":0.63},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.32000000000000001},"max":{"1min":0.879,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.59099999999999997}]},{"osd":7,"last update":"Mon Oct 16 12:27:02 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.58499999999999996,"15min":0.57799999999999996},"min":{"1min":0.45200000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.64100000000000001,"15min":0.62},"min":{"1min":0.38300000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.66400000000000003}]}]},{"osd":5,"up_from":13,"seq":55834575051,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54728,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317112,"statfs":{"total":96636764160,"available":96580722688,"internally_reserved":0,"allocated":1368064,"data_stored":684107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:26:58 2023","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.56899999999999995,"15min":0.59399999999999997},"min":{"1min":0.32500000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.64600000000000002,"5min":0.84499999999999997,"15min":0.873},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.59399999999999997,"15min":0.62},"min":{"1min":0.36299999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.73499999999999999,"5min":0.85899999999999999,"15min":0.88800000000000001},"last":0.54400000000000004}]},{"osd":1,"last update":"Mon Oct 16 12:26:58 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.60499999999999998,"15min":0.61499999999999999},"min":{"1min":0.39500000000000002,"5min":0.32600000000000001,"15min":0.28999999999999998},"max":{"1min":0.76600000000000001,"5min":0.89700000000000002,"15min":0.91300000000000003},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.503,"5min":0.60699999999999998,"15min":0.62},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.78900000000000003,"5min":0.85999999999999999,"15min":0.89000000000000001},"last":0.65500000000000003}]},{"osd":2,"last update":"Mon Oct 16 12:26:58 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.60299999999999998,"15min":0.623},"min":{"1min":0.40000000000000002,"5min":0.29099999999999998,"15min":0.26400000000000001},"max":{"1min":0.71299999999999997,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.59599999999999997,"15min":0.61399999999999999},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.66900000000000004,"5min":0.88600000000000001,"15min":0.89700000000000002},"last":0.61199999999999999}]},{"osd":3,"last update":"Mon Oct 16 12:26:58 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.61599999999999999,"15min":0.64100000000000001},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.72099999999999997,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.752},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.64000000000000001,"15min":0.65000000000000002},"min":{"1min":0.40400000000000003,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.71799999999999997,"5min":0.90800000000000003,"15min":0.93200000000000005},"last":0.72899999999999998}]},{"osd":4,"last update":"Mon Oct 16 12:26:58 2023","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.504,"15min":0.51000000000000001},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.25800000000000001},"max":{"1min":0.77100000000000002,"5min":0.85699999999999998,"15min":0.86399999999999999},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.58499999999999996,"15min":0.57099999999999995},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33500000000000002},"max":{"1min":0.78500000000000003,"5min":1.347,"15min":1.347},"last":0.876}]},{"osd":6,"last update":"Mon Oct 16 12:26:58 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.54400000000000004,"15min":0.55200000000000005},"min":{"1min":0.33100000000000002,"5min":0.314,"15min":0.314},"max":{"1min":0.83999999999999997,"5min":0.89400000000000002,"15min":0.93500000000000005},"last":0.91800000000000004},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.59999999999999998,"15min":0.59799999999999998},"min":{"1min":0.38300000000000001,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.82899999999999996,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.79500000000000004}]},{"osd":7,"last update":"Mon Oct 16 12:26:58 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.61699999999999999,"15min":0.61899999999999999},"min":{"1min":0.41499999999999998,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.79400000000000004,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.65300000000000002,"15min":0.65500000000000003},"min":{"1min":0.45600000000000002,"5min":0.379,"15min":0.379},"max":{"1min":0.88900000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.81899999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T12:27:42.168 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T12:27:42.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.326+0000 7f72ef062700 1 Processor -- start 2023-10-16T12:27:42.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.326+0000 7f72ef062700 1 -- start start 2023-10-16T12:27:42.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.327+0000 7f72ef062700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f72e80cf880 0x7f72e80b52f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:42.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.327+0000 7f72ef062700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f72e80b5830 0x7f72e815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:42.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.327+0000 7f72ef062700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72e815bc80 con 0x7f72e80cf4a0 2023-10-16T12:27:42.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.327+0000 7f72ef062700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72e815b070 con 0x7f72e80b5830 2023-10-16T12:27:42.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.327+0000 7f72ef062700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72e815b350 con 0x7f72e80cf880 2023-10-16T12:27:42.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.327+0000 7f72ed5ff700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f72e80b5830 0x7f72e815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:42.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.327+0000 7f72e7fff700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f72e80cf880 0x7f72e80b52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:42.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.327+0000 7f72ecdfe700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7f72e80cf4a0 0x7f72e80726e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:38820/0 (socket says 172.21.15.16:38820) 2023-10-16T12:27:42.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.327+0000 7f72ecdfe700 1 -- 172.21.15.16:0/1238112034 learned_addr learned my addr 172.21.15.16:0/1238112034 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:27:42.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.327+0000 7f72e7fff700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f72e80cf880 0x7f72e80b52f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:58856/0 (socket says 172.21.15.16:58856) 2023-10-16T12:27:42.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.327+0000 7f72e7fff700 1 -- 172.21.15.16:0/1238112034 >> v2:172.21.15.49:3300/0 conn(0x7f72e80b5830 msgr2=0x7f72e815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:42.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.327+0000 7f72e7fff700 1 --2- 172.21.15.16:0/1238112034 >> v2:172.21.15.49:3300/0 conn(0x7f72e80b5830 0x7f72e815a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:42.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.327+0000 7f72e7fff700 1 -- 172.21.15.16:0/1238112034 >> v1:172.21.15.49:6789/0 conn(0x7f72e80cf4a0 legacy=0x7f72e80726e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:42.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.327+0000 7f72e7fff700 1 -- 172.21.15.16:0/1238112034 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72e815b630 con 0x7f72e80cf880 2023-10-16T12:27:42.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.327+0000 7f72ed5ff700 1 --2- 172.21.15.16:0/1238112034 >> v2:172.21.15.49:3300/0 conn(0x7f72e80b5830 0x7f72e815a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:27:42.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.328+0000 7f72e7fff700 1 --2- 172.21.15.16:0/1238112034 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f72e80cf880 0x7f72e80b52f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f72d0009840 tx=0x7f72d0009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c982ba462528c81 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:42.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.328+0000 7f72e77fe700 1 -- 172.21.15.16:0/1238112034 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72d0011830 con 0x7f72e80cf880 2023-10-16T12:27:42.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.328+0000 7f72e77fe700 1 -- 172.21.15.16:0/1238112034 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72d0011990 con 0x7f72e80cf880 2023-10-16T12:27:42.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.328+0000 7f72e77fe700 1 -- 172.21.15.16:0/1238112034 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72d0011d20 con 0x7f72e80cf880 2023-10-16T12:27:42.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.328+0000 7f72ef062700 1 -- 172.21.15.16:0/1238112034 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f72e80cf880 msgr2=0x7f72e80b52f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:42.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.328+0000 7f72ef062700 1 --2- 172.21.15.16:0/1238112034 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f72e80cf880 0x7f72e80b52f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f72d0009840 tx=0x7f72d0009b50 comp rx=0 tx=0).stop 2023-10-16T12:27:42.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.328+0000 7f72ef062700 1 -- 172.21.15.16:0/1238112034 shutdown_connections 2023-10-16T12:27:42.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.328+0000 7f72ef062700 1 --2- 172.21.15.16:0/1238112034 >> v2:172.21.15.49:3300/0 conn(0x7f72e80b5830 0x7f72e815a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:42.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.328+0000 7f72ef062700 1 --2- 172.21.15.16:0/1238112034 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f72e80cf880 0x7f72e80b52f0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:42.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.328+0000 7f72ef062700 1 -- 172.21.15.16:0/1238112034 >> 172.21.15.16:0/1238112034 conn(0x7f72e80b1fd0 msgr2=0x7f72e80b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:42.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.328+0000 7f72ef062700 1 -- 172.21.15.16:0/1238112034 shutdown_connections 2023-10-16T12:27:42.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.328+0000 7f72ef062700 1 -- 172.21.15.16:0/1238112034 wait complete. 2023-10-16T12:27:42.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.330+0000 7f72ef062700 1 Processor -- start 2023-10-16T12:27:42.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.330+0000 7f72ef062700 1 -- start start 2023-10-16T12:27:42.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.330+0000 7f72ef062700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f72e80b5830 0x7f72e815a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:42.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.331+0000 7f72ef062700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f72e80cf4a0 0x7f72e80a84c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:42.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.331+0000 7f72ef062700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72e80cf880 0x7f72e80cbb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:42.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.331+0000 7f72ef062700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72e814ada0 con 0x7f72e80cf880 2023-10-16T12:27:42.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.331+0000 7f72ef062700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72e815a8a0 con 0x7f72e80cf4a0 2023-10-16T12:27:42.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.331+0000 7f72ef062700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72e814af90 con 0x7f72e80b5830 2023-10-16T12:27:42.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.331+0000 7f72ed5ff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72e80cf880 0x7f72e80cbb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:42.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.331+0000 7f72ed5ff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72e80cf880 0x7f72e80cbb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42374/0 (socket says 172.21.15.16:42374) 2023-10-16T12:27:42.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.331+0000 7f72ed5ff700 1 -- 172.21.15.16:0/3454254532 learned_addr learned my addr 172.21.15.16:0/3454254532 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:42.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.331+0000 7f72ecdfe700 1 --2- 172.21.15.16:0/3454254532 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f72e80b5830 0x7f72e815a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:42.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.331+0000 7f72e7fff700 1 --2- 172.21.15.16:0/3454254532 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f72e80cf4a0 0x7f72e80a84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:42.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.331+0000 7f72ed5ff700 1 -- 172.21.15.16:0/3454254532 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f72e80b5830 msgr2=0x7f72e815a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:42.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.331+0000 7f72ed5ff700 1 --2- 172.21.15.16:0/3454254532 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f72e80b5830 0x7f72e815a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:42.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.331+0000 7f72ed5ff700 1 -- 172.21.15.16:0/3454254532 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f72e80cf4a0 msgr2=0x7f72e80a84c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:42.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.331+0000 7f72ed5ff700 1 --2- 172.21.15.16:0/3454254532 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f72e80cf4a0 0x7f72e80a84c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:42.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.331+0000 7f72ed5ff700 1 -- 172.21.15.16:0/3454254532 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72e0009a70 con 0x7f72e80cf880 2023-10-16T12:27:42.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.331+0000 7f72ecdfe700 1 --2- 172.21.15.16:0/3454254532 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f72e80b5830 0x7f72e815a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:27:42.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.331+0000 7f72ed5ff700 1 --2- 172.21.15.16:0/3454254532 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72e80cf880 0x7f72e80cbb60 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f72e000efc0 tx=0x7f72e000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:42.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.331+0000 7f72e57fa700 1 -- 172.21.15.16:0/3454254532 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e0009d70 con 0x7f72e80cf880 2023-10-16T12:27:42.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.331+0000 7f72e57fa700 1 -- 172.21.15.16:0/3454254532 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72e000f040 con 0x7f72e80cf880 2023-10-16T12:27:42.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.331+0000 7f72ef062700 1 -- 172.21.15.16:0/3454254532 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72d00094a0 con 0x7f72e80cf880 2023-10-16T12:27:42.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.331+0000 7f72e57fa700 1 -- 172.21.15.16:0/3454254532 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e000ce00 con 0x7f72e80cf880 2023-10-16T12:27:42.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.331+0000 7f72ef062700 1 -- 172.21.15.16:0/3454254532 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72e814b550 con 0x7f72e80cf880 2023-10-16T12:27:42.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.332+0000 7f72ef062700 1 -- 172.21.15.16:0/3454254532 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72e804ebb0 con 0x7f72e80cf880 2023-10-16T12:27:42.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.333+0000 7f72e57fa700 1 -- 172.21.15.16:0/3454254532 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 51) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f72e0014480 con 0x7f72e80cf880 2023-10-16T12:27:42.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.333+0000 7f72e57fa700 1 --2- 172.21.15.16:0/3454254532 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f72c806b020 0x7f72c806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:42.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.333+0000 7f72e57fa700 1 -- 172.21.15.16:0/3454254532 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f72e008a150 con 0x7f72e80cf880 2023-10-16T12:27:42.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.333+0000 7f72ecdfe700 1 --2- 172.21.15.16:0/3454254532 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f72c806b020 0x7f72c806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:42.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.334+0000 7f72ecdfe700 1 --2- 172.21.15.16:0/3454254532 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f72c806b020 0x7f72c806d4e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f72d8007fd0 tx=0x7f72d8009040 comp rx=0 tx=0).ready entity=mgr.6365 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:42.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.338+0000 7f72e57fa700 1 -- 172.21.15.16:0/3454254532 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f72e0054e80 con 0x7f72e80cf880 2023-10-16T12:27:42.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.538+0000 7f72ef062700 1 -- 172.21.15.16:0/3454254532 --> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f72e815b910 con 0x7f72c806b020 2023-10-16T12:27:42.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.546+0000 7f72e57fa700 1 -- 172.21.15.16:0/3454254532 <== mgr.6365 v2:172.21.15.16:6832/4089191642 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289981 (secure 0 0 0) 0x7f72d80014c0 con 0x7f72c806b020 2023-10-16T12:27:42.552 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:27:42.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.550+0000 7f72ef062700 1 -- 172.21.15.16:0/3454254532 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f72c806b020 msgr2=0x7f72c806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:42.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.550+0000 7f72ef062700 1 --2- 172.21.15.16:0/3454254532 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f72c806b020 0x7f72c806d4e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f72d8007fd0 tx=0x7f72d8009040 comp rx=0 tx=0).stop 2023-10-16T12:27:42.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.550+0000 7f72ef062700 1 -- 172.21.15.16:0/3454254532 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72e80cf880 msgr2=0x7f72e80cbb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:42.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.550+0000 7f72ef062700 1 --2- 172.21.15.16:0/3454254532 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72e80cf880 0x7f72e80cbb60 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f72e000efc0 tx=0x7f72e000c3a0 comp rx=0 tx=0).stop 2023-10-16T12:27:42.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.550+0000 7f72ef062700 1 -- 172.21.15.16:0/3454254532 shutdown_connections 2023-10-16T12:27:42.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.550+0000 7f72ef062700 1 --2- 172.21.15.16:0/3454254532 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72e80cf880 0x7f72e80cbb60 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:42.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.550+0000 7f72ef062700 1 --2- 172.21.15.16:0/3454254532 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f72e80b5830 0x7f72e815a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:42.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.550+0000 7f72ef062700 1 --2- 172.21.15.16:0/3454254532 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f72c806b020 0x7f72c806d4e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:42.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.550+0000 7f72ef062700 1 --2- 172.21.15.16:0/3454254532 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f72e80cf4a0 0x7f72e80a84c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:42.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.550+0000 7f72ef062700 1 -- 172.21.15.16:0/3454254532 >> 172.21.15.16:0/3454254532 conn(0x7f72e80b1fd0 msgr2=0x7f72e80b0590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:42.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.550+0000 7f72ef062700 1 -- 172.21.15.16:0/3454254532 shutdown_connections 2023-10-16T12:27:42.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.550+0000 7f72ef062700 1 -- 172.21.15.16:0/3454254532 wait complete. 2023-10-16T12:27:42.556 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-10-16T12:27:42.564 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2023-10-16T12:27:41.522961+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":692,"num_read_kb":1336,"num_write":323,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":221,"ondisk_log_size":221,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":156,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":344632,"kb_used_data":11912,"kb_used_omap":12,"kb_used_meta":332595,"kb_avail":754630088,"statfs":{"total":773094113280,"available":772741210112,"internally_reserved":0,"allocated":12197888,"data_stored":6658412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12740,"internal_metadata":340577852},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.555387"},"pg_stats":[{"pgid":"14.3f","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.006282+0000","last_change":"2023-10-16T12:27:38.005483+0000","last_active":"2023-10-16T12:27:39.006282+0000","last_peered":"2023-10-16T12:27:39.006282+0000","last_clean":"2023-10-16T12:27:39.006282+0000","last_became_active":"2023-10-16T12:27:38.005224+0000","last_became_peered":"2023-10-16T12:27:38.005224+0000","last_unstale":"2023-10-16T12:27:39.006282+0000","last_undegraded":"2023-10-16T12:27:39.006282+0000","last_fullsized":"2023-10-16T12:27:39.006282+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:12:41.771844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3e","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.067841+0000","last_change":"2023-10-16T12:27:38.020825+0000","last_active":"2023-10-16T12:27:39.067841+0000","last_peered":"2023-10-16T12:27:39.067841+0000","last_clean":"2023-10-16T12:27:39.067841+0000","last_became_active":"2023-10-16T12:27:38.020526+0000","last_became_peered":"2023-10-16T12:27:38.020526+0000","last_unstale":"2023-10-16T12:27:39.067841+0000","last_undegraded":"2023-10-16T12:27:39.067841+0000","last_fullsized":"2023-10-16T12:27:39.067841+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:25:17.395792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.3d","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.006234+0000","last_change":"2023-10-16T12:27:38.027615+0000","last_active":"2023-10-16T12:27:39.006234+0000","last_peered":"2023-10-16T12:27:39.006234+0000","last_clean":"2023-10-16T12:27:39.006234+0000","last_became_active":"2023-10-16T12:27:38.027101+0000","last_became_peered":"2023-10-16T12:27:38.027101+0000","last_unstale":"2023-10-16T12:27:39.006234+0000","last_undegraded":"2023-10-16T12:27:39.006234+0000","last_fullsized":"2023-10-16T12:27:39.006234+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:07:35.137319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3c","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.246720+0000","last_change":"2023-10-16T12:27:38.021342+0000","last_active":"2023-10-16T12:27:39.246720+0000","last_peered":"2023-10-16T12:27:39.246720+0000","last_clean":"2023-10-16T12:27:39.246720+0000","last_became_active":"2023-10-16T12:27:38.016660+0000","last_became_peered":"2023-10-16T12:27:38.016660+0000","last_unstale":"2023-10-16T12:27:39.246720+0000","last_undegraded":"2023-10-16T12:27:39.246720+0000","last_fullsized":"2023-10-16T12:27:39.246720+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:42:57.697956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.3b","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.023841+0000","last_change":"2023-10-16T12:27:38.022837+0000","last_active":"2023-10-16T12:27:38.023841+0000","last_peered":"2023-10-16T12:27:38.023841+0000","last_clean":"2023-10-16T12:27:38.023841+0000","last_became_active":"2023-10-16T12:27:38.018004+0000","last_became_peered":"2023-10-16T12:27:38.018004+0000","last_unstale":"2023-10-16T12:27:38.023841+0000","last_undegraded":"2023-10-16T12:27:38.023841+0000","last_fullsized":"2023-10-16T12:27:38.023841+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:35:38.723509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.3a","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.005298+0000","last_change":"2023-10-16T12:27:38.022739+0000","last_active":"2023-10-16T12:27:39.005298+0000","last_peered":"2023-10-16T12:27:39.005298+0000","last_clean":"2023-10-16T12:27:39.005298+0000","last_became_active":"2023-10-16T12:27:38.022469+0000","last_became_peered":"2023-10-16T12:27:38.022469+0000","last_unstale":"2023-10-16T12:27:39.005298+0000","last_undegraded":"2023-10-16T12:27:39.005298+0000","last_fullsized":"2023-10-16T12:27:39.005298+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:55:52.173496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.39","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.017899+0000","last_change":"2023-10-16T12:27:38.015269+0000","last_active":"2023-10-16T12:27:38.017899+0000","last_peered":"2023-10-16T12:27:38.017899+0000","last_clean":"2023-10-16T12:27:38.017899+0000","last_became_active":"2023-10-16T12:27:38.014957+0000","last_became_peered":"2023-10-16T12:27:38.014957+0000","last_unstale":"2023-10-16T12:27:38.017899+0000","last_undegraded":"2023-10-16T12:27:38.017899+0000","last_fullsized":"2023-10-16T12:27:38.017899+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:31:05.189021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.38","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.246191+0000","last_change":"2023-10-16T12:27:38.023995+0000","last_active":"2023-10-16T12:27:39.246191+0000","last_peered":"2023-10-16T12:27:39.246191+0000","last_clean":"2023-10-16T12:27:39.246191+0000","last_became_active":"2023-10-16T12:27:38.023354+0000","last_became_peered":"2023-10-16T12:27:38.023354+0000","last_unstale":"2023-10-16T12:27:39.246191+0000","last_undegraded":"2023-10-16T12:27:39.246191+0000","last_fullsized":"2023-10-16T12:27:39.246191+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:35:38.723509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.37","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.022551+0000","last_change":"2023-10-16T12:27:38.016632+0000","last_active":"2023-10-16T12:27:38.022551+0000","last_peered":"2023-10-16T12:27:38.022551+0000","last_clean":"2023-10-16T12:27:38.022551+0000","last_became_active":"2023-10-16T12:27:38.016298+0000","last_became_peered":"2023-10-16T12:27:38.016298+0000","last_unstale":"2023-10-16T12:27:38.022551+0000","last_undegraded":"2023-10-16T12:27:38.022551+0000","last_fullsized":"2023-10-16T12:27:38.022551+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:17:05.807429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.36","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.019566+0000","last_change":"2023-10-16T12:27:38.017919+0000","last_active":"2023-10-16T12:27:38.019566+0000","last_peered":"2023-10-16T12:27:38.019566+0000","last_clean":"2023-10-16T12:27:38.019566+0000","last_became_active":"2023-10-16T12:27:38.017617+0000","last_became_peered":"2023-10-16T12:27:38.017617+0000","last_unstale":"2023-10-16T12:27:38.019566+0000","last_undegraded":"2023-10-16T12:27:38.019566+0000","last_fullsized":"2023-10-16T12:27:38.019566+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:24:26.284643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.35","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.026458+0000","last_change":"2023-10-16T12:27:38.025552+0000","last_active":"2023-10-16T12:27:38.026458+0000","last_peered":"2023-10-16T12:27:38.026458+0000","last_clean":"2023-10-16T12:27:38.026458+0000","last_became_active":"2023-10-16T12:27:38.024341+0000","last_became_peered":"2023-10-16T12:27:38.024341+0000","last_unstale":"2023-10-16T12:27:38.026458+0000","last_undegraded":"2023-10-16T12:27:38.026458+0000","last_fullsized":"2023-10-16T12:27:38.026458+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:10:51.389847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.34","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:37.999427+0000","last_change":"2023-10-16T12:27:37.997869+0000","last_active":"2023-10-16T12:27:37.999427+0000","last_peered":"2023-10-16T12:27:37.999427+0000","last_clean":"2023-10-16T12:27:37.999427+0000","last_became_active":"2023-10-16T12:27:37.997549+0000","last_became_peered":"2023-10-16T12:27:37.997549+0000","last_unstale":"2023-10-16T12:27:37.999427+0000","last_undegraded":"2023-10-16T12:27:37.999427+0000","last_fullsized":"2023-10-16T12:27:37.999427+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:34:35.434077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.33","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.246443+0000","last_change":"2023-10-16T12:27:38.016063+0000","last_active":"2023-10-16T12:27:39.246443+0000","last_peered":"2023-10-16T12:27:39.246443+0000","last_clean":"2023-10-16T12:27:39.246443+0000","last_became_active":"2023-10-16T12:27:38.014322+0000","last_became_peered":"2023-10-16T12:27:38.014322+0000","last_unstale":"2023-10-16T12:27:39.246443+0000","last_undegraded":"2023-10-16T12:27:39.246443+0000","last_fullsized":"2023-10-16T12:27:39.246443+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:05:06.570934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.32","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.025675+0000","last_change":"2023-10-16T12:27:38.024608+0000","last_active":"2023-10-16T12:27:38.025675+0000","last_peered":"2023-10-16T12:27:38.025675+0000","last_clean":"2023-10-16T12:27:38.025675+0000","last_became_active":"2023-10-16T12:27:38.024326+0000","last_became_peered":"2023-10-16T12:27:38.024326+0000","last_unstale":"2023-10-16T12:27:38.025675+0000","last_undegraded":"2023-10-16T12:27:38.025675+0000","last_fullsized":"2023-10-16T12:27:38.025675+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:26:45.443813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.31","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.246535+0000","last_change":"2023-10-16T12:27:38.017128+0000","last_active":"2023-10-16T12:27:39.246535+0000","last_peered":"2023-10-16T12:27:39.246535+0000","last_clean":"2023-10-16T12:27:39.246535+0000","last_became_active":"2023-10-16T12:27:38.016473+0000","last_became_peered":"2023-10-16T12:27:38.016473+0000","last_unstale":"2023-10-16T12:27:39.246535+0000","last_undegraded":"2023-10-16T12:27:39.246535+0000","last_fullsized":"2023-10-16T12:27:39.246535+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:56:39.091850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.30","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.005206+0000","last_change":"2023-10-16T12:27:38.019437+0000","last_active":"2023-10-16T12:27:39.005206+0000","last_peered":"2023-10-16T12:27:39.005206+0000","last_clean":"2023-10-16T12:27:39.005206+0000","last_became_active":"2023-10-16T12:27:38.016139+0000","last_became_peered":"2023-10-16T12:27:38.016139+0000","last_unstale":"2023-10-16T12:27:39.005206+0000","last_undegraded":"2023-10-16T12:27:39.005206+0000","last_fullsized":"2023-10-16T12:27:39.005206+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:36.707208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2f","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.005391+0000","last_change":"2023-10-16T12:27:38.020885+0000","last_active":"2023-10-16T12:27:39.005391+0000","last_peered":"2023-10-16T12:27:39.005391+0000","last_clean":"2023-10-16T12:27:39.005391+0000","last_became_active":"2023-10-16T12:27:38.019259+0000","last_became_peered":"2023-10-16T12:27:38.019259+0000","last_unstale":"2023-10-16T12:27:39.005391+0000","last_undegraded":"2023-10-16T12:27:39.005391+0000","last_fullsized":"2023-10-16T12:27:39.005391+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:25:31.018138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2e","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.006301+0000","last_change":"2023-10-16T12:27:38.012752+0000","last_active":"2023-10-16T12:27:39.006301+0000","last_peered":"2023-10-16T12:27:39.006301+0000","last_clean":"2023-10-16T12:27:39.006301+0000","last_became_active":"2023-10-16T12:27:38.012512+0000","last_became_peered":"2023-10-16T12:27:38.012512+0000","last_unstale":"2023-10-16T12:27:39.006301+0000","last_undegraded":"2023-10-16T12:27:39.006301+0000","last_fullsized":"2023-10-16T12:27:39.006301+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:19:09.853556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2d","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.027539+0000","last_change":"2023-10-16T12:27:38.025778+0000","last_active":"2023-10-16T12:27:38.027539+0000","last_peered":"2023-10-16T12:27:38.027539+0000","last_clean":"2023-10-16T12:27:38.027539+0000","last_became_active":"2023-10-16T12:27:38.025477+0000","last_became_peered":"2023-10-16T12:27:38.025477+0000","last_unstale":"2023-10-16T12:27:38.027539+0000","last_undegraded":"2023-10-16T12:27:38.027539+0000","last_fullsized":"2023-10-16T12:27:38.027539+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:40:07.635702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2c","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.246437+0000","last_change":"2023-10-16T12:27:38.021327+0000","last_active":"2023-10-16T12:27:39.246437+0000","last_peered":"2023-10-16T12:27:39.246437+0000","last_clean":"2023-10-16T12:27:39.246437+0000","last_became_active":"2023-10-16T12:27:38.016635+0000","last_became_peered":"2023-10-16T12:27:38.016635+0000","last_unstale":"2023-10-16T12:27:39.246437+0000","last_undegraded":"2023-10-16T12:27:39.246437+0000","last_fullsized":"2023-10-16T12:27:39.246437+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:39:51.985205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.2b","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.023462+0000","last_change":"2023-10-16T12:27:38.017752+0000","last_active":"2023-10-16T12:27:38.023462+0000","last_peered":"2023-10-16T12:27:38.023462+0000","last_clean":"2023-10-16T12:27:38.023462+0000","last_became_active":"2023-10-16T12:27:38.017492+0000","last_became_peered":"2023-10-16T12:27:38.017492+0000","last_unstale":"2023-10-16T12:27:38.023462+0000","last_undegraded":"2023-10-16T12:27:38.023462+0000","last_fullsized":"2023-10-16T12:27:38.023462+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:40:59.602579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2a","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.025625+0000","last_change":"2023-10-16T12:27:38.024306+0000","last_active":"2023-10-16T12:27:38.025625+0000","last_peered":"2023-10-16T12:27:38.025625+0000","last_clean":"2023-10-16T12:27:38.025625+0000","last_became_active":"2023-10-16T12:27:38.023892+0000","last_became_peered":"2023-10-16T12:27:38.023892+0000","last_unstale":"2023-10-16T12:27:38.025625+0000","last_undegraded":"2023-10-16T12:27:38.025625+0000","last_fullsized":"2023-10-16T12:27:38.025625+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:45:49.653013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.29","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.020867+0000","last_change":"2023-10-16T12:27:38.019317+0000","last_active":"2023-10-16T12:27:38.020867+0000","last_peered":"2023-10-16T12:27:38.020867+0000","last_clean":"2023-10-16T12:27:38.020867+0000","last_became_active":"2023-10-16T12:27:38.019054+0000","last_became_peered":"2023-10-16T12:27:38.019054+0000","last_unstale":"2023-10-16T12:27:38.020867+0000","last_undegraded":"2023-10-16T12:27:38.020867+0000","last_fullsized":"2023-10-16T12:27:38.020867+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:50:34.414528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.28","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.005986+0000","last_change":"2023-10-16T12:27:38.004330+0000","last_active":"2023-10-16T12:27:39.005986+0000","last_peered":"2023-10-16T12:27:39.005986+0000","last_clean":"2023-10-16T12:27:39.005986+0000","last_became_active":"2023-10-16T12:27:38.004054+0000","last_became_peered":"2023-10-16T12:27:38.004054+0000","last_unstale":"2023-10-16T12:27:39.005986+0000","last_undegraded":"2023-10-16T12:27:39.005986+0000","last_fullsized":"2023-10-16T12:27:39.005986+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:40:35.029721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.27","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.067884+0000","last_change":"2023-10-16T12:27:38.005381+0000","last_active":"2023-10-16T12:27:39.067884+0000","last_peered":"2023-10-16T12:27:39.067884+0000","last_clean":"2023-10-16T12:27:39.067884+0000","last_became_active":"2023-10-16T12:27:38.005081+0000","last_became_peered":"2023-10-16T12:27:38.005081+0000","last_unstale":"2023-10-16T12:27:39.067884+0000","last_undegraded":"2023-10-16T12:27:39.067884+0000","last_fullsized":"2023-10-16T12:27:39.067884+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:31:05.189021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.26","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.023846+0000","last_change":"2023-10-16T12:27:38.022828+0000","last_active":"2023-10-16T12:27:38.023846+0000","last_peered":"2023-10-16T12:27:38.023846+0000","last_clean":"2023-10-16T12:27:38.023846+0000","last_became_active":"2023-10-16T12:27:38.022464+0000","last_became_peered":"2023-10-16T12:27:38.022464+0000","last_unstale":"2023-10-16T12:27:38.023846+0000","last_undegraded":"2023-10-16T12:27:38.023846+0000","last_fullsized":"2023-10-16T12:27:38.023846+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:01:50.960997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.25","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.009839+0000","last_change":"2023-10-16T12:27:38.005554+0000","last_active":"2023-10-16T12:27:38.009839+0000","last_peered":"2023-10-16T12:27:38.009839+0000","last_clean":"2023-10-16T12:27:38.009839+0000","last_became_active":"2023-10-16T12:27:38.005251+0000","last_became_peered":"2023-10-16T12:27:38.005251+0000","last_unstale":"2023-10-16T12:27:38.009839+0000","last_undegraded":"2023-10-16T12:27:38.009839+0000","last_fullsized":"2023-10-16T12:27:38.009839+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:30:01.478780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.24","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.246521+0000","last_change":"2023-10-16T12:27:38.024543+0000","last_active":"2023-10-16T12:27:39.246521+0000","last_peered":"2023-10-16T12:27:39.246521+0000","last_clean":"2023-10-16T12:27:39.246521+0000","last_became_active":"2023-10-16T12:27:38.024200+0000","last_became_peered":"2023-10-16T12:27:38.024200+0000","last_unstale":"2023-10-16T12:27:39.246521+0000","last_undegraded":"2023-10-16T12:27:39.246521+0000","last_fullsized":"2023-10-16T12:27:39.246521+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:37:50.379179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.23","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.011782+0000","last_change":"2023-10-16T12:27:38.008834+0000","last_active":"2023-10-16T12:27:38.011782+0000","last_peered":"2023-10-16T12:27:38.011782+0000","last_clean":"2023-10-16T12:27:38.011782+0000","last_became_active":"2023-10-16T12:27:38.008529+0000","last_became_peered":"2023-10-16T12:27:38.008529+0000","last_unstale":"2023-10-16T12:27:38.011782+0000","last_undegraded":"2023-10-16T12:27:38.011782+0000","last_fullsized":"2023-10-16T12:27:38.011782+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:38:19.735626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.22","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.023568+0000","last_change":"2023-10-16T12:27:38.014992+0000","last_active":"2023-10-16T12:27:38.023568+0000","last_peered":"2023-10-16T12:27:38.023568+0000","last_clean":"2023-10-16T12:27:38.023568+0000","last_became_active":"2023-10-16T12:27:38.014683+0000","last_became_peered":"2023-10-16T12:27:38.014683+0000","last_unstale":"2023-10-16T12:27:38.023568+0000","last_undegraded":"2023-10-16T12:27:38.023568+0000","last_fullsized":"2023-10-16T12:27:38.023568+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:26:03.624784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.21","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.067565+0000","last_change":"2023-10-16T12:27:38.016478+0000","last_active":"2023-10-16T12:27:39.067565+0000","last_peered":"2023-10-16T12:27:39.067565+0000","last_clean":"2023-10-16T12:27:39.067565+0000","last_became_active":"2023-10-16T12:27:38.016068+0000","last_became_peered":"2023-10-16T12:27:38.016068+0000","last_unstale":"2023-10-16T12:27:39.067565+0000","last_undegraded":"2023-10-16T12:27:39.067565+0000","last_fullsized":"2023-10-16T12:27:39.067565+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:10:26.061080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.20","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.004842+0000","last_change":"2023-10-16T12:27:38.019579+0000","last_active":"2023-10-16T12:27:39.004842+0000","last_peered":"2023-10-16T12:27:39.004842+0000","last_clean":"2023-10-16T12:27:39.004842+0000","last_became_active":"2023-10-16T12:27:38.016194+0000","last_became_peered":"2023-10-16T12:27:38.016194+0000","last_unstale":"2023-10-16T12:27:39.004842+0000","last_undegraded":"2023-10-16T12:27:39.004842+0000","last_fullsized":"2023-10-16T12:27:39.004842+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:30:36.115116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.005192+0000","last_change":"2023-10-16T12:27:38.019739+0000","last_active":"2023-10-16T12:27:39.005192+0000","last_peered":"2023-10-16T12:27:39.005192+0000","last_clean":"2023-10-16T12:27:39.005192+0000","last_became_active":"2023-10-16T12:27:38.019452+0000","last_became_peered":"2023-10-16T12:27:38.019452+0000","last_unstale":"2023-10-16T12:27:39.005192+0000","last_undegraded":"2023-10-16T12:27:39.005192+0000","last_fullsized":"2023-10-16T12:27:39.005192+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:41:54.837529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1f","version":"111'4","reported_seq":15,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.026827+0000","last_change":"2023-10-16T12:27:36.976544+0000","last_active":"2023-10-16T12:27:38.026827+0000","last_peered":"2023-10-16T12:27:38.026827+0000","last_clean":"2023-10-16T12:27:38.026827+0000","last_became_active":"2023-10-16T12:27:36.976186+0000","last_became_peered":"2023-10-16T12:27:36.976186+0000","last_unstale":"2023-10-16T12:27:38.026827+0000","last_undegraded":"2023-10-16T12:27:38.026827+0000","last_fullsized":"2023-10-16T12:27:38.026827+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_clean_scrub_stamp":"2023-10-16T12:27:35.962232+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:10:50.385306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.005574+0000","last_change":"2023-10-16T12:27:38.012429+0000","last_active":"2023-10-16T12:27:39.005574+0000","last_peered":"2023-10-16T12:27:39.005574+0000","last_clean":"2023-10-16T12:27:39.005574+0000","last_became_active":"2023-10-16T12:27:38.011990+0000","last_became_peered":"2023-10-16T12:27:38.011990+0000","last_unstale":"2023-10-16T12:27:39.005574+0000","last_undegraded":"2023-10-16T12:27:39.005574+0000","last_fullsized":"2023-10-16T12:27:39.005574+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:38:29.081086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1e","version":"111'1","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.023706+0000","last_change":"2023-10-16T12:27:36.985203+0000","last_active":"2023-10-16T12:27:38.023706+0000","last_peered":"2023-10-16T12:27:38.023706+0000","last_clean":"2023-10-16T12:27:38.023706+0000","last_became_active":"2023-10-16T12:27:36.984995+0000","last_became_peered":"2023-10-16T12:27:36.984995+0000","last_unstale":"2023-10-16T12:27:38.023706+0000","last_undegraded":"2023-10-16T12:27:38.023706+0000","last_fullsized":"2023-10-16T12:27:38.023706+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_clean_scrub_stamp":"2023-10-16T12:27:35.962232+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:36:00.608821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.019214+0000","last_change":"2023-10-16T12:27:38.018002+0000","last_active":"2023-10-16T12:27:38.019214+0000","last_peered":"2023-10-16T12:27:38.019214+0000","last_clean":"2023-10-16T12:27:38.019214+0000","last_became_active":"2023-10-16T12:27:38.017694+0000","last_became_peered":"2023-10-16T12:27:38.017694+0000","last_unstale":"2023-10-16T12:27:38.019214+0000","last_undegraded":"2023-10-16T12:27:38.019214+0000","last_fullsized":"2023-10-16T12:27:38.019214+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:30:42.276938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.1d","version":"111'2","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.019546+0000","last_change":"2023-10-16T12:27:36.988403+0000","last_active":"2023-10-16T12:27:38.019546+0000","last_peered":"2023-10-16T12:27:38.019546+0000","last_clean":"2023-10-16T12:27:38.019546+0000","last_became_active":"2023-10-16T12:27:36.988084+0000","last_became_peered":"2023-10-16T12:27:36.988084+0000","last_unstale":"2023-10-16T12:27:38.019546+0000","last_undegraded":"2023-10-16T12:27:38.019546+0000","last_fullsized":"2023-10-16T12:27:38.019546+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_clean_scrub_stamp":"2023-10-16T12:27:35.962232+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:40:57.661929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.005514+0000","last_change":"2023-10-16T12:27:38.011634+0000","last_active":"2023-10-16T12:27:39.005514+0000","last_peered":"2023-10-16T12:27:39.005514+0000","last_clean":"2023-10-16T12:27:39.005514+0000","last_became_active":"2023-10-16T12:27:38.011327+0000","last_became_peered":"2023-10-16T12:27:38.011327+0000","last_unstale":"2023-10-16T12:27:39.005514+0000","last_undegraded":"2023-10-16T12:27:39.005514+0000","last_fullsized":"2023-10-16T12:27:39.005514+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:23:12.852377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1c","version":"111'2","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.028839+0000","last_change":"2023-10-16T12:27:36.988560+0000","last_active":"2023-10-16T12:27:38.028839+0000","last_peered":"2023-10-16T12:27:38.028839+0000","last_clean":"2023-10-16T12:27:38.028839+0000","last_became_active":"2023-10-16T12:27:36.988191+0000","last_became_peered":"2023-10-16T12:27:36.988191+0000","last_unstale":"2023-10-16T12:27:38.028839+0000","last_undegraded":"2023-10-16T12:27:38.028839+0000","last_fullsized":"2023-10-16T12:27:38.028839+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_clean_scrub_stamp":"2023-10-16T12:27:35.962232+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:40:58.598038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.246215+0000","last_change":"2023-10-16T12:27:38.023815+0000","last_active":"2023-10-16T12:27:39.246215+0000","last_peered":"2023-10-16T12:27:39.246215+0000","last_clean":"2023-10-16T12:27:39.246215+0000","last_became_active":"2023-10-16T12:27:38.023028+0000","last_became_peered":"2023-10-16T12:27:38.023028+0000","last_unstale":"2023-10-16T12:27:39.246215+0000","last_undegraded":"2023-10-16T12:27:39.246215+0000","last_fullsized":"2023-10-16T12:27:39.246215+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:41:19.520990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1b","version":"111'1","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.023249+0000","last_change":"2023-10-16T12:27:36.985795+0000","last_active":"2023-10-16T12:27:38.023249+0000","last_peered":"2023-10-16T12:27:38.023249+0000","last_clean":"2023-10-16T12:27:38.023249+0000","last_became_active":"2023-10-16T12:27:36.984740+0000","last_became_peered":"2023-10-16T12:27:36.984740+0000","last_unstale":"2023-10-16T12:27:38.023249+0000","last_undegraded":"2023-10-16T12:27:38.023249+0000","last_fullsized":"2023-10-16T12:27:38.023249+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_clean_scrub_stamp":"2023-10-16T12:27:35.962232+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:41:18.516450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.011113+0000","last_change":"2023-10-16T12:27:38.009370+0000","last_active":"2023-10-16T12:27:38.011113+0000","last_peered":"2023-10-16T12:27:38.011113+0000","last_clean":"2023-10-16T12:27:38.011113+0000","last_became_active":"2023-10-16T12:27:38.009049+0000","last_became_peered":"2023-10-16T12:27:38.009049+0000","last_unstale":"2023-10-16T12:27:38.011113+0000","last_undegraded":"2023-10-16T12:27:38.011113+0000","last_fullsized":"2023-10-16T12:27:38.011113+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:34:33.927601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1a","version":"111'1","reported_seq":13,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.246725+0000","last_change":"2023-10-16T12:27:36.992130+0000","last_active":"2023-10-16T12:27:39.246725+0000","last_peered":"2023-10-16T12:27:39.246725+0000","last_clean":"2023-10-16T12:27:39.246725+0000","last_became_active":"2023-10-16T12:27:36.991795+0000","last_became_peered":"2023-10-16T12:27:36.991795+0000","last_unstale":"2023-10-16T12:27:39.246725+0000","last_undegraded":"2023-10-16T12:27:39.246725+0000","last_fullsized":"2023-10-16T12:27:39.246725+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_clean_scrub_stamp":"2023-10-16T12:27:35.962232+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:01:49.956456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.004947+0000","last_change":"2023-10-16T12:27:38.023001+0000","last_active":"2023-10-16T12:27:39.004947+0000","last_peered":"2023-10-16T12:27:39.004947+0000","last_clean":"2023-10-16T12:27:39.004947+0000","last_became_active":"2023-10-16T12:27:38.022696+0000","last_became_peered":"2023-10-16T12:27:38.022696+0000","last_unstale":"2023-10-16T12:27:39.004947+0000","last_undegraded":"2023-10-16T12:27:39.004947+0000","last_fullsized":"2023-10-16T12:27:39.004947+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:48:11.313600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.006218+0000","last_change":"2023-10-16T12:27:36.989874+0000","last_active":"2023-10-16T12:27:39.006218+0000","last_peered":"2023-10-16T12:27:39.006218+0000","last_clean":"2023-10-16T12:27:39.006218+0000","last_became_active":"2023-10-16T12:27:36.989572+0000","last_became_peered":"2023-10-16T12:27:36.989572+0000","last_unstale":"2023-10-16T12:27:39.006218+0000","last_undegraded":"2023-10-16T12:27:39.006218+0000","last_fullsized":"2023-10-16T12:27:39.006218+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_clean_scrub_stamp":"2023-10-16T12:27:35.962232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:31:48.599920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.246432+0000","last_change":"2023-10-16T12:27:38.013437+0000","last_active":"2023-10-16T12:27:39.246432+0000","last_peered":"2023-10-16T12:27:39.246432+0000","last_clean":"2023-10-16T12:27:39.246432+0000","last_became_active":"2023-10-16T12:27:38.013138+0000","last_became_peered":"2023-10-16T12:27:38.013138+0000","last_unstale":"2023-10-16T12:27:39.246432+0000","last_undegraded":"2023-10-16T12:27:39.246432+0000","last_fullsized":"2023-10-16T12:27:39.246432+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:36:01.613362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:37.996866+0000","last_change":"2023-10-16T12:27:36.990706+0000","last_active":"2023-10-16T12:27:37.996866+0000","last_peered":"2023-10-16T12:27:37.996866+0000","last_clean":"2023-10-16T12:27:37.996866+0000","last_became_active":"2023-10-16T12:27:36.990444+0000","last_became_peered":"2023-10-16T12:27:36.990444+0000","last_unstale":"2023-10-16T12:27:37.996866+0000","last_undegraded":"2023-10-16T12:27:37.996866+0000","last_fullsized":"2023-10-16T12:27:37.996866+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_clean_scrub_stamp":"2023-10-16T12:27:35.962232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:38:21.525776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.022316+0000","last_change":"2023-10-16T12:27:38.015555+0000","last_active":"2023-10-16T12:27:38.022316+0000","last_peered":"2023-10-16T12:27:38.022316+0000","last_clean":"2023-10-16T12:27:38.022316+0000","last_became_active":"2023-10-16T12:27:38.015262+0000","last_became_peered":"2023-10-16T12:27:38.015262+0000","last_unstale":"2023-10-16T12:27:38.022316+0000","last_undegraded":"2023-10-16T12:27:38.022316+0000","last_fullsized":"2023-10-16T12:27:38.022316+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:23:00.622863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.067945+0000","last_change":"2023-10-16T12:27:36.981921+0000","last_active":"2023-10-16T12:27:39.067945+0000","last_peered":"2023-10-16T12:27:39.067945+0000","last_clean":"2023-10-16T12:27:39.067945+0000","last_became_active":"2023-10-16T12:27:36.981574+0000","last_became_peered":"2023-10-16T12:27:36.981574+0000","last_unstale":"2023-10-16T12:27:39.067945+0000","last_undegraded":"2023-10-16T12:27:39.067945+0000","last_fullsized":"2023-10-16T12:27:39.067945+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_clean_scrub_stamp":"2023-10-16T12:27:35.962232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:37:47.438147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.026737+0000","last_change":"2023-10-16T12:27:38.026059+0000","last_active":"2023-10-16T12:27:38.026737+0000","last_peered":"2023-10-16T12:27:38.026737+0000","last_clean":"2023-10-16T12:27:38.026737+0000","last_became_active":"2023-10-16T12:27:38.025831+0000","last_became_peered":"2023-10-16T12:27:38.025831+0000","last_unstale":"2023-10-16T12:27:38.026737+0000","last_undegraded":"2023-10-16T12:27:38.026737+0000","last_fullsized":"2023-10-16T12:27:38.026737+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:38:31.277703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.006597+0000","last_change":"2023-10-16T12:27:36.994850+0000","last_active":"2023-10-16T12:27:39.006597+0000","last_peered":"2023-10-16T12:27:39.006597+0000","last_clean":"2023-10-16T12:27:39.006597+0000","last_became_active":"2023-10-16T12:27:36.994478+0000","last_became_peered":"2023-10-16T12:27:36.994478+0000","last_unstale":"2023-10-16T12:27:39.006597+0000","last_undegraded":"2023-10-16T12:27:39.006597+0000","last_fullsized":"2023-10-16T12:27:39.006597+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_clean_scrub_stamp":"2023-10-16T12:27:35.962232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:58:11.208246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.006438+0000","last_change":"2023-10-16T12:27:38.009003+0000","last_active":"2023-10-16T12:27:39.006438+0000","last_peered":"2023-10-16T12:27:39.006438+0000","last_clean":"2023-10-16T12:27:39.006438+0000","last_became_active":"2023-10-16T12:27:38.008692+0000","last_became_peered":"2023-10-16T12:27:38.008692+0000","last_unstale":"2023-10-16T12:27:39.006438+0000","last_undegraded":"2023-10-16T12:27:39.006438+0000","last_fullsized":"2023-10-16T12:27:39.006438+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:48:13.638880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.15","version":"111'3","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.029182+0000","last_change":"2023-10-16T12:27:36.990285+0000","last_active":"2023-10-16T12:27:38.029182+0000","last_peered":"2023-10-16T12:27:38.029182+0000","last_clean":"2023-10-16T12:27:38.029182+0000","last_became_active":"2023-10-16T12:27:36.989921+0000","last_became_peered":"2023-10-16T12:27:36.989921+0000","last_unstale":"2023-10-16T12:27:38.029182+0000","last_undegraded":"2023-10-16T12:27:38.029182+0000","last_fullsized":"2023-10-16T12:27:38.029182+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_clean_scrub_stamp":"2023-10-16T12:27:35.962232+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:25:18.221168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.246616+0000","last_change":"2023-10-16T12:27:38.016179+0000","last_active":"2023-10-16T12:27:39.246616+0000","last_peered":"2023-10-16T12:27:39.246616+0000","last_clean":"2023-10-16T12:27:39.246616+0000","last_became_active":"2023-10-16T12:27:38.014743+0000","last_became_peered":"2023-10-16T12:27:38.014743+0000","last_unstale":"2023-10-16T12:27:39.246616+0000","last_undegraded":"2023-10-16T12:27:39.246616+0000","last_fullsized":"2023-10-16T12:27:39.246616+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:52:33.322374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.14","version":"111'6","reported_seq":18,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.005301+0000","last_change":"2023-10-16T12:27:36.981047+0000","last_active":"2023-10-16T12:27:39.005301+0000","last_peered":"2023-10-16T12:27:39.005301+0000","last_clean":"2023-10-16T12:27:39.005301+0000","last_became_active":"2023-10-16T12:27:36.980732+0000","last_became_peered":"2023-10-16T12:27:36.980732+0000","last_unstale":"2023-10-16T12:27:39.005301+0000","last_undegraded":"2023-10-16T12:27:39.005301+0000","last_fullsized":"2023-10-16T12:27:39.005301+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_clean_scrub_stamp":"2023-10-16T12:27:35.962232+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:29:38.117150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.246658+0000","last_change":"2023-10-16T12:27:38.023643+0000","last_active":"2023-10-16T12:27:39.246658+0000","last_peered":"2023-10-16T12:27:39.246658+0000","last_clean":"2023-10-16T12:27:39.246658+0000","last_became_active":"2023-10-16T12:27:38.022731+0000","last_became_peered":"2023-10-16T12:27:38.022731+0000","last_unstale":"2023-10-16T12:27:39.246658+0000","last_undegraded":"2023-10-16T12:27:39.246658+0000","last_fullsized":"2023-10-16T12:27:39.246658+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:03:31.380576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"111'2","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.024610+0000","last_change":"2023-10-16T12:27:36.990608+0000","last_active":"2023-10-16T12:27:38.024610+0000","last_peered":"2023-10-16T12:27:38.024610+0000","last_clean":"2023-10-16T12:27:38.024610+0000","last_became_active":"2023-10-16T12:27:36.990310+0000","last_became_peered":"2023-10-16T12:27:36.990310+0000","last_unstale":"2023-10-16T12:27:38.024610+0000","last_undegraded":"2023-10-16T12:27:38.024610+0000","last_fullsized":"2023-10-16T12:27:38.024610+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_clean_scrub_stamp":"2023-10-16T12:27:35.962232+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:58:11.208246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.023667+0000","last_change":"2023-10-16T12:27:38.018003+0000","last_active":"2023-10-16T12:27:38.023667+0000","last_peered":"2023-10-16T12:27:38.023667+0000","last_clean":"2023-10-16T12:27:38.023667+0000","last_became_active":"2023-10-16T12:27:38.015244+0000","last_became_peered":"2023-10-16T12:27:38.015244+0000","last_unstale":"2023-10-16T12:27:38.023667+0000","last_undegraded":"2023-10-16T12:27:38.023667+0000","last_fullsized":"2023-10-16T12:27:38.023667+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:43:23.547480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.005040+0000","last_change":"2023-10-16T12:27:36.984442+0000","last_active":"2023-10-16T12:27:39.005040+0000","last_peered":"2023-10-16T12:27:39.005040+0000","last_clean":"2023-10-16T12:27:39.005040+0000","last_became_active":"2023-10-16T12:27:36.984169+0000","last_became_peered":"2023-10-16T12:27:36.984169+0000","last_unstale":"2023-10-16T12:27:39.005040+0000","last_undegraded":"2023-10-16T12:27:39.005040+0000","last_fullsized":"2023-10-16T12:27:39.005040+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_clean_scrub_stamp":"2023-10-16T12:27:35.962232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:16:40.357072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.016405+0000","last_change":"2023-10-16T12:27:38.013871+0000","last_active":"2023-10-16T12:27:38.016405+0000","last_peered":"2023-10-16T12:27:38.016405+0000","last_clean":"2023-10-16T12:27:38.016405+0000","last_became_active":"2023-10-16T12:27:38.013586+0000","last_became_peered":"2023-10-16T12:27:38.013586+0000","last_unstale":"2023-10-16T12:27:38.016405+0000","last_undegraded":"2023-10-16T12:27:38.016405+0000","last_fullsized":"2023-10-16T12:27:38.016405+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:54:42.009262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.246374+0000","last_change":"2023-10-16T12:27:36.993517+0000","last_active":"2023-10-16T12:27:39.246374+0000","last_peered":"2023-10-16T12:27:39.246374+0000","last_clean":"2023-10-16T12:27:39.246374+0000","last_became_active":"2023-10-16T12:27:36.993193+0000","last_became_peered":"2023-10-16T12:27:36.993193+0000","last_unstale":"2023-10-16T12:27:39.246374+0000","last_undegraded":"2023-10-16T12:27:39.246374+0000","last_fullsized":"2023-10-16T12:27:39.246374+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_clean_scrub_stamp":"2023-10-16T12:27:35.962232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:25:18.221168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.029964+0000","last_change":"2023-10-16T12:27:38.018760+0000","last_active":"2023-10-16T12:27:38.029964+0000","last_peered":"2023-10-16T12:27:38.029964+0000","last_clean":"2023-10-16T12:27:38.029964+0000","last_became_active":"2023-10-16T12:27:38.018513+0000","last_became_peered":"2023-10-16T12:27:38.018513+0000","last_unstale":"2023-10-16T12:27:38.029964+0000","last_undegraded":"2023-10-16T12:27:38.029964+0000","last_fullsized":"2023-10-16T12:27:38.029964+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:54:36.370568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.10","version":"111'8","reported_seq":20,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.246738+0000","last_change":"2023-10-16T12:27:36.992178+0000","last_active":"2023-10-16T12:27:39.246738+0000","last_peered":"2023-10-16T12:27:39.246738+0000","last_clean":"2023-10-16T12:27:39.246738+0000","last_became_active":"2023-10-16T12:27:36.991857+0000","last_became_peered":"2023-10-16T12:27:36.991857+0000","last_unstale":"2023-10-16T12:27:39.246738+0000","last_undegraded":"2023-10-16T12:27:39.246738+0000","last_fullsized":"2023-10-16T12:27:39.246738+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_clean_scrub_stamp":"2023-10-16T12:27:35.962232+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:22:59.618322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.004949+0000","last_change":"2023-10-16T12:27:38.025560+0000","last_active":"2023-10-16T12:27:39.004949+0000","last_peered":"2023-10-16T12:27:39.004949+0000","last_clean":"2023-10-16T12:27:39.004949+0000","last_became_active":"2023-10-16T12:27:38.025179+0000","last_became_peered":"2023-10-16T12:27:38.025179+0000","last_unstale":"2023-10-16T12:27:39.004949+0000","last_undegraded":"2023-10-16T12:27:39.004949+0000","last_fullsized":"2023-10-16T12:27:39.004949+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:38:49.527365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.f","version":"111'1","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.019763+0000","last_change":"2023-10-16T12:27:36.981258+0000","last_active":"2023-10-16T12:27:38.019763+0000","last_peered":"2023-10-16T12:27:38.019763+0000","last_clean":"2023-10-16T12:27:38.019763+0000","last_became_active":"2023-10-16T12:27:36.981002+0000","last_became_peered":"2023-10-16T12:27:36.981002+0000","last_unstale":"2023-10-16T12:27:38.019763+0000","last_undegraded":"2023-10-16T12:27:38.019763+0000","last_fullsized":"2023-10-16T12:27:38.019763+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_clean_scrub_stamp":"2023-10-16T12:27:35.962232+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:45:48.648472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.010760+0000","last_change":"2023-10-16T12:27:38.004689+0000","last_active":"2023-10-16T12:27:38.010760+0000","last_peered":"2023-10-16T12:27:38.010760+0000","last_clean":"2023-10-16T12:27:38.010760+0000","last_became_active":"2023-10-16T12:27:38.003921+0000","last_became_peered":"2023-10-16T12:27:38.003921+0000","last_unstale":"2023-10-16T12:27:38.010760+0000","last_undegraded":"2023-10-16T12:27:38.010760+0000","last_fullsized":"2023-10-16T12:27:38.010760+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:47:47.319973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.e","version":"111'2","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.021067+0000","last_change":"2023-10-16T12:27:36.985355+0000","last_active":"2023-10-16T12:27:38.021067+0000","last_peered":"2023-10-16T12:27:38.021067+0000","last_clean":"2023-10-16T12:27:38.021067+0000","last_became_active":"2023-10-16T12:27:36.985049+0000","last_became_peered":"2023-10-16T12:27:36.985049+0000","last_unstale":"2023-10-16T12:27:38.021067+0000","last_undegraded":"2023-10-16T12:27:38.021067+0000","last_fullsized":"2023-10-16T12:27:38.021067+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_clean_scrub_stamp":"2023-10-16T12:27:35.962232+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:36:46.367885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.024143+0000","last_change":"2023-10-16T12:27:38.023358+0000","last_active":"2023-10-16T12:27:38.024143+0000","last_peered":"2023-10-16T12:27:38.024143+0000","last_clean":"2023-10-16T12:27:38.024143+0000","last_became_active":"2023-10-16T12:27:38.023064+0000","last_became_peered":"2023-10-16T12:27:38.023064+0000","last_unstale":"2023-10-16T12:27:38.024143+0000","last_undegraded":"2023-10-16T12:27:38.024143+0000","last_fullsized":"2023-10-16T12:27:38.024143+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:03:31.380576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.d","version":"111'1","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.029925+0000","last_change":"2023-10-16T12:27:36.987805+0000","last_active":"2023-10-16T12:27:38.029925+0000","last_peered":"2023-10-16T12:27:38.029925+0000","last_clean":"2023-10-16T12:27:38.029925+0000","last_became_active":"2023-10-16T12:27:36.987428+0000","last_became_peered":"2023-10-16T12:27:36.987428+0000","last_unstale":"2023-10-16T12:27:38.029925+0000","last_undegraded":"2023-10-16T12:27:38.029925+0000","last_fullsized":"2023-10-16T12:27:38.029925+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_clean_scrub_stamp":"2023-10-16T12:27:35.962232+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:09:49.678551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.004989+0000","last_change":"2023-10-16T12:27:38.027468+0000","last_active":"2023-10-16T12:27:39.004989+0000","last_peered":"2023-10-16T12:27:39.004989+0000","last_clean":"2023-10-16T12:27:39.004989+0000","last_became_active":"2023-10-16T12:27:38.027198+0000","last_became_peered":"2023-10-16T12:27:38.027198+0000","last_unstale":"2023-10-16T12:27:39.004989+0000","last_undegraded":"2023-10-16T12:27:39.004989+0000","last_fullsized":"2023-10-16T12:27:39.004989+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:28:16.544155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.b","version":"111'2","reported_seq":14,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.067529+0000","last_change":"2023-10-16T12:27:36.997826+0000","last_active":"2023-10-16T12:27:39.067529+0000","last_peered":"2023-10-16T12:27:39.067529+0000","last_clean":"2023-10-16T12:27:39.067529+0000","last_became_active":"2023-10-16T12:27:36.997563+0000","last_became_peered":"2023-10-16T12:27:36.997563+0000","last_unstale":"2023-10-16T12:27:39.067529+0000","last_undegraded":"2023-10-16T12:27:39.067529+0000","last_fullsized":"2023-10-16T12:27:39.067529+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_clean_scrub_stamp":"2023-10-16T12:27:35.962232+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:14:09.324779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.067489+0000","last_change":"2023-10-16T12:27:38.021150+0000","last_active":"2023-10-16T12:27:39.067489+0000","last_peered":"2023-10-16T12:27:39.067489+0000","last_clean":"2023-10-16T12:27:39.067489+0000","last_became_active":"2023-10-16T12:27:38.020516+0000","last_became_peered":"2023-10-16T12:27:38.020516+0000","last_unstale":"2023-10-16T12:27:39.067489+0000","last_undegraded":"2023-10-16T12:27:39.067489+0000","last_fullsized":"2023-10-16T12:27:39.067489+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:38:22.530317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:37.992948+0000","last_change":"2023-10-16T12:27:36.995162+0000","last_active":"2023-10-16T12:27:37.992948+0000","last_peered":"2023-10-16T12:27:37.992948+0000","last_clean":"2023-10-16T12:27:37.992948+0000","last_became_active":"2023-10-16T12:27:36.994741+0000","last_became_peered":"2023-10-16T12:27:36.994741+0000","last_unstale":"2023-10-16T12:27:37.992948+0000","last_undegraded":"2023-10-16T12:27:37.992948+0000","last_fullsized":"2023-10-16T12:27:37.992948+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_clean_scrub_stamp":"2023-10-16T12:27:35.962232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:31:50.471272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.067587+0000","last_change":"2023-10-16T12:27:38.020395+0000","last_active":"2023-10-16T12:27:39.067587+0000","last_peered":"2023-10-16T12:27:39.067587+0000","last_clean":"2023-10-16T12:27:39.067587+0000","last_became_active":"2023-10-16T12:27:38.019999+0000","last_became_peered":"2023-10-16T12:27:38.019999+0000","last_unstale":"2023-10-16T12:27:39.067587+0000","last_undegraded":"2023-10-16T12:27:39.067587+0000","last_fullsized":"2023-10-16T12:27:39.067587+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:36:47.372426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.067608+0000","last_change":"2023-10-16T12:27:36.987144+0000","last_active":"2023-10-16T12:27:39.067608+0000","last_peered":"2023-10-16T12:27:39.067608+0000","last_clean":"2023-10-16T12:27:39.067608+0000","last_became_active":"2023-10-16T12:27:36.986846+0000","last_became_peered":"2023-10-16T12:27:36.986846+0000","last_unstale":"2023-10-16T12:27:39.067608+0000","last_undegraded":"2023-10-16T12:27:39.067608+0000","last_fullsized":"2023-10-16T12:27:39.067608+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_clean_scrub_stamp":"2023-10-16T12:27:35.962232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:56:24.192112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.005359+0000","last_change":"2023-10-16T12:27:38.009266+0000","last_active":"2023-10-16T12:27:39.005359+0000","last_peered":"2023-10-16T12:27:39.005359+0000","last_clean":"2023-10-16T12:27:39.005359+0000","last_became_active":"2023-10-16T12:27:38.008764+0000","last_became_peered":"2023-10-16T12:27:38.008764+0000","last_unstale":"2023-10-16T12:27:39.005359+0000","last_undegraded":"2023-10-16T12:27:39.005359+0000","last_fullsized":"2023-10-16T12:27:39.005359+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:52:37.606383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:37.992616+0000","last_change":"2023-10-16T12:27:36.986348+0000","last_active":"2023-10-16T12:27:37.992616+0000","last_peered":"2023-10-16T12:27:37.992616+0000","last_clean":"2023-10-16T12:27:37.992616+0000","last_became_active":"2023-10-16T12:27:36.984643+0000","last_became_peered":"2023-10-16T12:27:36.984643+0000","last_unstale":"2023-10-16T12:27:37.992616+0000","last_undegraded":"2023-10-16T12:27:37.992616+0000","last_fullsized":"2023-10-16T12:27:37.992616+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_clean_scrub_stamp":"2023-10-16T12:27:35.962232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:07:02.729770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.011340+0000","last_change":"2023-10-16T12:27:38.009764+0000","last_active":"2023-10-16T12:27:38.011340+0000","last_peered":"2023-10-16T12:27:38.011340+0000","last_clean":"2023-10-16T12:27:38.011340+0000","last_became_active":"2023-10-16T12:27:38.009487+0000","last_became_peered":"2023-10-16T12:27:38.009487+0000","last_unstale":"2023-10-16T12:27:38.011340+0000","last_undegraded":"2023-10-16T12:27:38.011340+0000","last_fullsized":"2023-10-16T12:27:38.011340+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:26:03.624784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.7","version":"111'2","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.019638+0000","last_change":"2023-10-16T12:27:36.985562+0000","last_active":"2023-10-16T12:27:38.019638+0000","last_peered":"2023-10-16T12:27:38.019638+0000","last_clean":"2023-10-16T12:27:38.019638+0000","last_became_active":"2023-10-16T12:27:36.985277+0000","last_became_peered":"2023-10-16T12:27:36.985277+0000","last_unstale":"2023-10-16T12:27:38.019638+0000","last_undegraded":"2023-10-16T12:27:38.019638+0000","last_fullsized":"2023-10-16T12:27:38.019638+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_clean_scrub_stamp":"2023-10-16T12:27:35.962232+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:31:48.599920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.005705+0000","last_change":"2023-10-16T12:27:38.026401+0000","last_active":"2023-10-16T12:27:39.005705+0000","last_peered":"2023-10-16T12:27:39.005705+0000","last_clean":"2023-10-16T12:27:39.005705+0000","last_became_active":"2023-10-16T12:27:38.026112+0000","last_became_peered":"2023-10-16T12:27:38.026112+0000","last_unstale":"2023-10-16T12:27:39.005705+0000","last_undegraded":"2023-10-16T12:27:39.005705+0000","last_fullsized":"2023-10-16T12:27:39.005705+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:00:46.759996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.015690+0000","last_change":"2023-10-16T12:27:36.985411+0000","last_active":"2023-10-16T12:27:38.015690+0000","last_peered":"2023-10-16T12:27:38.015690+0000","last_clean":"2023-10-16T12:27:38.015690+0000","last_became_active":"2023-10-16T12:27:36.985143+0000","last_became_peered":"2023-10-16T12:27:36.985143+0000","last_unstale":"2023-10-16T12:27:38.015690+0000","last_undegraded":"2023-10-16T12:27:38.015690+0000","last_fullsized":"2023-10-16T12:27:38.015690+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_clean_scrub_stamp":"2023-10-16T12:27:35.962232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:39:50.980664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.005406+0000","last_change":"2023-10-16T12:27:38.008382+0000","last_active":"2023-10-16T12:27:39.005406+0000","last_peered":"2023-10-16T12:27:39.005406+0000","last_clean":"2023-10-16T12:27:39.005406+0000","last_became_active":"2023-10-16T12:27:38.008039+0000","last_became_peered":"2023-10-16T12:27:38.008039+0000","last_unstale":"2023-10-16T12:27:39.005406+0000","last_undegraded":"2023-10-16T12:27:39.005406+0000","last_fullsized":"2023-10-16T12:27:39.005406+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:09:13.873193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.007171+0000","last_change":"2023-10-16T12:27:36.990124+0000","last_active":"2023-10-16T12:27:38.007171+0000","last_peered":"2023-10-16T12:27:38.007171+0000","last_clean":"2023-10-16T12:27:38.007171+0000","last_became_active":"2023-10-16T12:27:36.989815+0000","last_became_peered":"2023-10-16T12:27:36.989815+0000","last_unstale":"2023-10-16T12:27:38.007171+0000","last_undegraded":"2023-10-16T12:27:38.007171+0000","last_fullsized":"2023-10-16T12:27:38.007171+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_clean_scrub_stamp":"2023-10-16T12:27:35.962232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:01:24.361594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.067971+0000","last_change":"2023-10-16T12:27:38.021334+0000","last_active":"2023-10-16T12:27:39.067971+0000","last_peered":"2023-10-16T12:27:39.067971+0000","last_clean":"2023-10-16T12:27:39.067971+0000","last_became_active":"2023-10-16T12:27:38.020967+0000","last_became_peered":"2023-10-16T12:27:38.020967+0000","last_unstale":"2023-10-16T12:27:39.067971+0000","last_undegraded":"2023-10-16T12:27:39.067971+0000","last_fullsized":"2023-10-16T12:27:39.067971+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:30:42.276938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:37.999743+0000","last_change":"2023-10-16T12:27:36.980048+0000","last_active":"2023-10-16T12:27:37.999743+0000","last_peered":"2023-10-16T12:27:37.999743+0000","last_clean":"2023-10-16T12:27:37.999743+0000","last_became_active":"2023-10-16T12:27:36.979754+0000","last_became_peered":"2023-10-16T12:27:36.979754+0000","last_unstale":"2023-10-16T12:27:37.999743+0000","last_undegraded":"2023-10-16T12:27:37.999743+0000","last_fullsized":"2023-10-16T12:27:37.999743+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_clean_scrub_stamp":"2023-10-16T12:27:35.962232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:26:44.439272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.007806+0000","last_change":"2023-10-16T12:27:38.005483+0000","last_active":"2023-10-16T12:27:38.007806+0000","last_peered":"2023-10-16T12:27:38.007806+0000","last_clean":"2023-10-16T12:27:38.007806+0000","last_became_active":"2023-10-16T12:27:38.005175+0000","last_became_peered":"2023-10-16T12:27:38.005175+0000","last_unstale":"2023-10-16T12:27:38.007806+0000","last_undegraded":"2023-10-16T12:27:38.007806+0000","last_fullsized":"2023-10-16T12:27:38.007806+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:02:09.243246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.005164+0000","last_change":"2023-10-16T12:27:36.989102+0000","last_active":"2023-10-16T12:27:39.005164+0000","last_peered":"2023-10-16T12:27:39.005164+0000","last_clean":"2023-10-16T12:27:39.005164+0000","last_became_active":"2023-10-16T12:27:36.988556+0000","last_became_peered":"2023-10-16T12:27:36.988556+0000","last_unstale":"2023-10-16T12:27:39.005164+0000","last_undegraded":"2023-10-16T12:27:39.005164+0000","last_fullsized":"2023-10-16T12:27:39.005164+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_clean_scrub_stamp":"2023-10-16T12:27:35.962232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:37:24.745515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.006110+0000","last_change":"2023-10-16T12:27:38.016951+0000","last_active":"2023-10-16T12:27:39.006110+0000","last_peered":"2023-10-16T12:27:39.006110+0000","last_clean":"2023-10-16T12:27:39.006110+0000","last_became_active":"2023-10-16T12:27:38.016244+0000","last_became_peered":"2023-10-16T12:27:38.016244+0000","last_unstale":"2023-10-16T12:27:39.006110+0000","last_undegraded":"2023-10-16T12:27:39.006110+0000","last_fullsized":"2023-10-16T12:27:39.006110+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:28:18.102314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.067581+0000","last_change":"2023-10-16T12:27:36.998503+0000","last_active":"2023-10-16T12:27:39.067581+0000","last_peered":"2023-10-16T12:27:39.067581+0000","last_clean":"2023-10-16T12:27:39.067581+0000","last_became_active":"2023-10-16T12:27:36.998252+0000","last_became_peered":"2023-10-16T12:27:36.998252+0000","last_unstale":"2023-10-16T12:27:39.067581+0000","last_undegraded":"2023-10-16T12:27:39.067581+0000","last_fullsized":"2023-10-16T12:27:39.067581+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_clean_scrub_stamp":"2023-10-16T12:27:35.962232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:47:05.480286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.023761+0000","last_change":"2023-10-16T12:27:38.022432+0000","last_active":"2023-10-16T12:27:38.023761+0000","last_peered":"2023-10-16T12:27:38.023761+0000","last_clean":"2023-10-16T12:27:38.023761+0000","last_became_active":"2023-10-16T12:27:38.022143+0000","last_became_peered":"2023-10-16T12:27:38.022143+0000","last_unstale":"2023-10-16T12:27:38.023761+0000","last_undegraded":"2023-10-16T12:27:38.023761+0000","last_fullsized":"2023-10-16T12:27:38.023761+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:43:23.547480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.002832+0000","last_change":"2023-10-16T12:27:36.980107+0000","last_active":"2023-10-16T12:27:38.002832+0000","last_peered":"2023-10-16T12:27:38.002832+0000","last_clean":"2023-10-16T12:27:38.002832+0000","last_became_active":"2023-10-16T12:27:36.979805+0000","last_became_peered":"2023-10-16T12:27:36.979805+0000","last_unstale":"2023-10-16T12:27:38.002832+0000","last_undegraded":"2023-10-16T12:27:38.002832+0000","last_fullsized":"2023-10-16T12:27:38.002832+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_clean_scrub_stamp":"2023-10-16T12:27:35.962232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:17:04.802888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.005083+0000","last_change":"2023-10-16T12:27:38.020021+0000","last_active":"2023-10-16T12:27:39.005083+0000","last_peered":"2023-10-16T12:27:39.005083+0000","last_clean":"2023-10-16T12:27:39.005083+0000","last_became_active":"2023-10-16T12:27:38.019699+0000","last_became_peered":"2023-10-16T12:27:38.019699+0000","last_unstale":"2023-10-16T12:27:39.005083+0000","last_undegraded":"2023-10-16T12:27:39.005083+0000","last_fullsized":"2023-10-16T12:27:39.005083+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:51:39.535828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:37.997226+0000","last_change":"2023-10-16T12:27:36.993811+0000","last_active":"2023-10-16T12:27:37.997226+0000","last_peered":"2023-10-16T12:27:37.997226+0000","last_clean":"2023-10-16T12:27:37.997226+0000","last_became_active":"2023-10-16T12:27:36.993494+0000","last_became_peered":"2023-10-16T12:27:36.993494+0000","last_unstale":"2023-10-16T12:27:37.997226+0000","last_undegraded":"2023-10-16T12:27:37.997226+0000","last_fullsized":"2023-10-16T12:27:37.997226+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_clean_scrub_stamp":"2023-10-16T12:27:35.962232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:13:56.125305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":9,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-16T12:27:38.017955+0000","last_change":"2023-10-16T12:27:38.016474+0000","last_active":"2023-10-16T12:27:38.017955+0000","last_peered":"2023-10-16T12:27:38.017955+0000","last_clean":"2023-10-16T12:27:38.017955+0000","last_became_active":"2023-10-16T12:27:38.015428+0000","last_became_peered":"2023-10-16T12:27:38.015428+0000","last_unstale":"2023-10-16T12:27:38.017955+0000","last_undegraded":"2023-10-16T12:27:38.017955+0000","last_fullsized":"2023-10-16T12:27:38.017955+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:36.966773+0000","last_clean_scrub_stamp":"2023-10-16T12:27:36.966773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:25:17.395792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-16T12:27:39.246727+0000","last_change":"2023-10-16T12:27:36.987949+0000","last_active":"2023-10-16T12:27:39.246727+0000","last_peered":"2023-10-16T12:27:39.246727+0000","last_clean":"2023-10-16T12:27:39.246727+0000","last_became_active":"2023-10-16T12:27:36.987631+0000","last_became_peered":"2023-10-16T12:27:36.987631+0000","last_unstale":"2023-10-16T12:27:39.246727+0000","last_undegraded":"2023-10-16T12:27:39.246727+0000","last_fullsized":"2023-10-16T12:27:39.246727+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:27:35.962232+0000","last_clean_scrub_stamp":"2023-10-16T12:27:35.962232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:40:06.631161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"102'183","reported_seq":484,"reported_epoch":110,"state":"active+clean","last_fresh":"2023-10-16T12:27:36.980888+0000","last_change":"2023-10-16T12:10:34.746820+0000","last_active":"2023-10-16T12:27:36.980888+0000","last_peered":"2023-10-16T12:27:36.980888+0000","last_clean":"2023-10-16T12:27:36.980888+0000","last_became_active":"2023-10-16T12:10:34.745574+0000","last_became_peered":"2023-10-16T12:10:34.745574+0000","last_unstale":"2023-10-16T12:27:36.980888+0000","last_undegraded":"2023-10-16T12:27:36.980888+0000","last_fullsized":"2023-10-16T12:27:36.980888+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_clean_scrub_stamp":"2023-10-16T12:10:33.577111+0000","objects_scrubbed":0,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:14.461986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":692,"num_read_kb":1336,"num_write":279,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":14,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":692,"num_read_kb":1336,"num_write":279,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":183,"ondisk_log_size":183,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575053,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59476,"kb_used_data":1924,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312364,"statfs":{"total":96636764160,"available":96575860736,"internally_reserved":0,"allocated":1970176,"data_stored":1274531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58915275},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:27:08 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.57099999999999995,"15min":0.58699999999999997},"min":{"1min":0.31,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.76300000000000001,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.56699999999999995,"15min":0.58099999999999996},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.78100000000000003,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.51900000000000002}]},{"osd":1,"last update":"Mon Oct 16 12:27:08 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.57499999999999996,"15min":0.60199999999999998},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.247},"max":{"1min":0.88100000000000001,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.59699999999999998,"15min":0.60099999999999998},"min":{"1min":0.377,"5min":0.33200000000000002,"15min":0.32400000000000001},"max":{"1min":0.76700000000000002,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.59299999999999997}]},{"osd":2,"last update":"Mon Oct 16 12:27:08 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.65200000000000002,"15min":0.65900000000000003},"min":{"1min":0.46999999999999997,"5min":0.40699999999999997,"15min":0.36099999999999999},"max":{"1min":0.79900000000000004,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.65600000000000003,"15min":0.66600000000000004},"min":{"1min":0.42899999999999999,"5min":0.376,"15min":0.36699999999999999},"max":{"1min":0.88,"5min":0.94599999999999995,"15min":0.96699999999999997},"last":0.54700000000000004}]},{"osd":3,"last update":"Mon Oct 16 12:27:08 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.60499999999999998,"15min":0.61699999999999999},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.82899999999999996,"5min":0.92200000000000004,"15min":1.1000000000000001},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.64900000000000002,"15min":0.63800000000000001},"min":{"1min":0.42199999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.84999999999999998,"5min":1.2509999999999999,"15min":1.2509999999999999},"last":0.67200000000000004}]},{"osd":4,"last update":"Mon Oct 16 12:27:08 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.55300000000000005,"15min":0.52900000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.28499999999999998},"max":{"1min":0.78800000000000003,"5min":1.383,"15min":1.383},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.628,"15min":0.60599999999999998},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.307},"max":{"1min":0.93600000000000005,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.70199999999999996}]},{"osd":5,"last update":"Mon Oct 16 12:27:08 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.57099999999999995,"15min":0.54000000000000004},"min":{"1min":0.33000000000000002,"5min":0.314,"15min":0.29799999999999999},"max":{"1min":0.90800000000000003,"5min":1.3109999999999999,"15min":1.3109999999999999},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.60099999999999998,"15min":0.58499999999999996},"min":{"1min":0.34699999999999998,"5min":0.33500000000000002,"15min":0.32400000000000001},"max":{"1min":0.93600000000000005,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":0.68899999999999995}]},{"osd":6,"last update":"Mon Oct 16 12:27:08 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.64500000000000002,"15min":0.62},"min":{"1min":0.41299999999999998,"5min":0.39300000000000002,"15min":0.36399999999999999},"max":{"1min":0.92900000000000005,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.65300000000000002,"15min":0.627},"min":{"1min":0.36799999999999999,"5min":0.34599999999999997,"15min":0.32300000000000001},"max":{"1min":0.97199999999999998,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.629}]}]},{"osd":6,"up_from":13,"seq":55834575051,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32540,"kb_used_data":1356,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339300,"statfs":{"total":96636764160,"available":96603443200,"internally_reserved":0,"allocated":1388544,"data_stored":686323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:27:02 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.58399999999999996,"15min":0.61099999999999999},"min":{"1min":0.29299999999999998,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.85199999999999998,"5min":1.196,"15min":1.196},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.58299999999999996,"15min":0.622},"min":{"1min":0.38500000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.80800000000000005,"5min":0.997,"15min":0.997},"last":0.83399999999999996}]},{"osd":1,"last update":"Mon Oct 16 12:27:02 2023","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.53600000000000003,"15min":0.57099999999999995},"min":{"1min":0.34399999999999997,"5min":0.32800000000000001,"15min":0.29399999999999998},"max":{"1min":0.66500000000000004,"5min":0.83899999999999997,"15min":0.875},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.55900000000000005,"15min":0.58799999999999997},"min":{"1min":0.32800000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.70599999999999996,"5min":1.091,"15min":1.091},"last":0.70699999999999996}]},{"osd":2,"last update":"Mon Oct 16 12:27:02 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.56899999999999995,"15min":0.59199999999999997},"min":{"1min":0.36799999999999999,"5min":0.36199999999999999,"15min":0.34499999999999997},"max":{"1min":0.76100000000000001,"5min":0.91200000000000003,"15min":0.97099999999999997},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.55600000000000005,"15min":0.59999999999999998},"min":{"1min":0.35499999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.85999999999999999,"5min":1.1539999999999999,"15min":2.3879999999999999},"last":0.748}]},{"osd":3,"last update":"Mon Oct 16 12:27:02 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.59399999999999997,"15min":0.625},"min":{"1min":0.37,"5min":0.34200000000000003,"15min":0.33500000000000002},"max":{"1min":0.88,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.629,"5min":0.66300000000000003,"15min":0.67900000000000005},"min":{"1min":0.45200000000000001,"5min":0.40899999999999997,"15min":0.40100000000000002},"max":{"1min":0.90600000000000003,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.88500000000000001}]},{"osd":4,"last update":"Mon Oct 16 12:27:02 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.53100000000000003,"15min":0.52100000000000002},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.29899999999999999},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.435},{"interface":"front","average":{"1min":0.63,"5min":0.60199999999999998,"15min":0.59699999999999998},"min":{"1min":0.49199999999999999,"5min":0.32300000000000001,"15min":0.30299999999999999},"max":{"1min":0.82299999999999995,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.68000000000000005}]},{"osd":5,"last update":"Mon Oct 16 12:27:02 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.58799999999999997,"15min":0.57599999999999996},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.30099999999999999},"max":{"1min":0.98799999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.63,"15min":0.625},"min":{"1min":0.47899999999999998,"5min":0.35099999999999998,"15min":0.311},"max":{"1min":1.0029999999999999,"5min":1.3540000000000001,"15min":1.3540000000000001},"last":0.60599999999999998}]},{"osd":7,"last update":"Mon Oct 16 12:27:02 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.58399999999999996,"15min":0.57299999999999995},"min":{"1min":0.435,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.92700000000000005,"5min":1.29,"15min":1.29},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.622,"15min":0.61799999999999999},"min":{"1min":0.46999999999999997,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.57399999999999995}]}]},{"osd":1,"up_from":13,"seq":55834575051,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32524,"kb_used_data":1340,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339316,"statfs":{"total":96636764160,"available":96603459584,"internally_reserved":0,"allocated":1372160,"data_stored":684177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":31914441},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:27:00 2023","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.36899999999999999,"15min":0.376},"min":{"1min":0.246,"5min":0.21299999999999999,"15min":0.186},"max":{"1min":0.53300000000000003,"5min":0.63600000000000001,"15min":0.66000000000000003},"last":0.372},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.44500000000000001,"15min":0.45800000000000002},"min":{"1min":0.28399999999999997,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.502,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.46300000000000002}]},{"osd":2,"last update":"Mon Oct 16 12:27:00 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.40500000000000003,"15min":0.42599999999999999},"min":{"1min":0.251,"5min":0.25,"15min":0.215},"max":{"1min":0.45000000000000001,"5min":0.69199999999999995,"15min":0.93000000000000005},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.45300000000000001,"15min":0.46400000000000002},"min":{"1min":0.32400000000000001,"5min":0.28199999999999997,"15min":0.24399999999999999},"max":{"1min":0.54500000000000004,"5min":0.69199999999999995,"15min":0.98099999999999998},"last":0.47799999999999998}]},{"osd":3,"last update":"Mon Oct 16 12:27:00 2023","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.38500000000000001,"15min":0.40100000000000002},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.57999999999999996,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.42799999999999999,"15min":0.45500000000000002},"min":{"1min":0.26100000000000001,"5min":0.252,"15min":0.22900000000000001},"max":{"1min":0.58699999999999997,"5min":0.91200000000000003,"15min":1.127},"last":0.498}]},{"osd":4,"last update":"Mon Oct 16 12:27:00 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61099999999999999,"15min":0.63300000000000001},"min":{"1min":0.41799999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.79800000000000004,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.752},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.62,"15min":0.63900000000000001},"min":{"1min":0.41899999999999998,"5min":0.39100000000000001,"15min":0.34699999999999998},"max":{"1min":0.89600000000000002,"5min":0.92000000000000004,"15min":1.089},"last":0.63800000000000001}]},{"osd":5,"last update":"Mon Oct 16 12:27:00 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.63600000000000001,"15min":0.66500000000000004},"min":{"1min":0.496,"5min":0.38300000000000001,"15min":0.28499999999999998},"max":{"1min":0.76700000000000002,"5min":1.069,"15min":1.069},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67300000000000004,"15min":0.68799999999999994},"min":{"1min":0.439,"5min":0.38900000000000001,"15min":0.32500000000000001},"max":{"1min":0.94499999999999995,"5min":1.0620000000000001,"15min":1.0649999999999999},"last":0.81100000000000005}]},{"osd":6,"last update":"Mon Oct 16 12:27:00 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.63,"15min":0.65400000000000003},"min":{"1min":0.48699999999999999,"5min":0.378,"15min":0.378},"max":{"1min":0.77800000000000002,"5min":0.94999999999999996,"15min":1.0129999999999999},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.629,"5min":0.65300000000000002,"15min":0.66200000000000003},"min":{"1min":0.55100000000000005,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.83299999999999996,"5min":0.96899999999999997,"15min":1.048},"last":0.68999999999999995}]},{"osd":7,"last update":"Mon Oct 16 12:27:00 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.67200000000000004,"15min":0.68700000000000006},"min":{"1min":0.42099999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.98899999999999999,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.67600000000000005,"15min":0.70299999999999996},"min":{"1min":0.45700000000000002,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.876,"5min":0.94899999999999995,"15min":1.167},"last":0.78100000000000003}]}]},{"osd":0,"up_from":13,"seq":55834575052,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37280,"kb_used_data":1940,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334560,"statfs":{"total":96636764160,"available":96598589440,"internally_reserved":0,"allocated":1986560,"data_stored":1276793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 12:26:55 2023","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"min":{"1min":0.22500000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.40999999999999998,"15min":0.41799999999999998},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23200000000000001},"max":{"1min":0.64500000000000002,"5min":0.71799999999999997,"15min":0.73099999999999998},"last":0.27900000000000003}]},{"osd":2,"last update":"Mon Oct 16 12:26:55 2023","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.42099999999999999,"15min":0.42499999999999999},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.44500000000000001,"15min":0.45100000000000001},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.317}]},{"osd":3,"last update":"Mon Oct 16 12:26:55 2023","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.42699999999999999,"15min":0.42199999999999999},"min":{"1min":0.26300000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.62,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.47799999999999998,"15min":0.48699999999999999},"min":{"1min":0.318,"5min":0.29099999999999998,"15min":0.27600000000000002},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.878},"last":0.41999999999999998}]},{"osd":4,"last update":"Mon Oct 16 12:26:55 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.64300000000000002,"15min":0.64800000000000002},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.89200000000000002,"5min":0.90400000000000003,"15min":1.0960000000000001},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.626,"15min":0.65500000000000003},"min":{"1min":0.374,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.84199999999999997,"5min":0.93799999999999994,"15min":1.1719999999999999},"last":0.47399999999999998}]},{"osd":5,"last update":"Mon Oct 16 12:26:55 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.63500000000000001,"15min":0.65000000000000002},"min":{"1min":0.373,"5min":0.373,"15min":0.34399999999999997},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":1.127},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.66500000000000004,"15min":0.67700000000000005},"min":{"1min":0.39400000000000002,"5min":0.39200000000000002,"15min":0.39000000000000001},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":1.1160000000000001},"last":0.55900000000000005}]},{"osd":6,"last update":"Mon Oct 16 12:26:55 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.66600000000000004,"15min":0.68000000000000005},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":1.079},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.65900000000000003,"15min":0.68300000000000005},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":1.1499999999999999},"last":0.46400000000000002}]},{"osd":7,"last update":"Mon Oct 16 12:26:55 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.67800000000000005,"15min":0.69899999999999995},"min":{"1min":0.436,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.995,"5min":0.995,"15min":1.0349999999999999},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.65700000000000003,"15min":0.68799999999999994},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":1.0569999999999999},"last":0.44500000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575052,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58884,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312956,"statfs":{"total":96636764160,"available":96576466944,"internally_reserved":0,"allocated":1368064,"data_stored":684107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:26:56 2023","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.36799999999999999,"15min":0.38400000000000001},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.62,"5min":0.62,"15min":0.73799999999999999},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.44400000000000001,"15min":0.45000000000000001},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.63400000000000001,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.36399999999999999}]},{"osd":1,"last update":"Mon Oct 16 12:26:56 2023","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.45100000000000001,"15min":0.46800000000000003},"min":{"1min":0.31,"5min":0.25600000000000001,"15min":0.254},"max":{"1min":0.57999999999999996,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.42499999999999999,"15min":0.435},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.246},"max":{"1min":0.54400000000000004,"5min":0.69699999999999995,"15min":0.76500000000000001},"last":0.379}]},{"osd":3,"last update":"Mon Oct 16 12:26:56 2023","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.40200000000000002,"15min":0.41599999999999998},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.60199999999999998,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.51000000000000001,"15min":0.51900000000000002},"min":{"1min":0.36599999999999999,"5min":0.318,"15min":0.246},"max":{"1min":0.67600000000000005,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.39000000000000001}]},{"osd":4,"last update":"Mon Oct 16 12:26:56 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.66500000000000004,"15min":0.68100000000000005},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.81200000000000006,"5min":1.3839999999999999,"15min":1.3839999999999999},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.66200000000000003,"15min":0.67900000000000005},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.84899999999999998,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.54400000000000004}]},{"osd":5,"last update":"Mon Oct 16 12:26:56 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.65800000000000003,"15min":0.66600000000000004},"min":{"1min":0.378,"5min":0.378,"15min":0.312},"max":{"1min":0.78700000000000003,"5min":1.093,"15min":1.093},"last":0.437},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.64600000000000002,"15min":0.66400000000000003},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.317},"max":{"1min":0.879,"5min":1.331,"15min":1.331},"last":0.55700000000000005}]},{"osd":6,"last update":"Mon Oct 16 12:26:56 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.63200000000000001,"15min":0.63800000000000001},"min":{"1min":0.37,"5min":0.37,"15min":0.34699999999999998},"max":{"1min":0.91900000000000004,"5min":0.99299999999999999,"15min":1.046},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.68400000000000005,"15min":0.69199999999999995},"min":{"1min":0.40799999999999997,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.82599999999999996,"5min":1.032,"15min":1.0389999999999999},"last":0.47099999999999997}]},{"osd":7,"last update":"Mon Oct 16 12:26:56 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.66500000000000004,"15min":0.68000000000000005},"min":{"1min":0.40300000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.879,"5min":0.97499999999999998,"15min":1.026},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.68100000000000005,"15min":0.69299999999999995},"min":{"1min":0.46700000000000003,"5min":0.442,"15min":0.40600000000000003},"max":{"1min":0.89600000000000002,"5min":1.145,"15min":1.145},"last":0.41599999999999998}]}]},{"osd":3,"up_from":13,"seq":55834575051,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36676,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335164,"statfs":{"total":96636764160,"available":96599207936,"internally_reserved":0,"allocated":1368064,"data_stored":684107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:27:04 2023","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.39300000000000002,"15min":0.39500000000000002},"min":{"1min":0.216,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.60699999999999998,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.40200000000000002,"15min":0.40400000000000003},"min":{"1min":0.26600000000000001,"5min":0.25900000000000001,"15min":0.249},"max":{"1min":0.47299999999999998,"5min":0.63100000000000001,"15min":0.73099999999999998},"last":0.32100000000000001}]},{"osd":1,"last update":"Mon Oct 16 12:27:04 2023","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.39700000000000002,"15min":0.40899999999999997},"min":{"1min":0.245,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.48299999999999998,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.375,"5min":0.44700000000000001,"15min":0.45800000000000002},"min":{"1min":0.27000000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.51400000000000001,"5min":0.67300000000000004,"15min":0.747},"last":0.41799999999999998}]},{"osd":2,"last update":"Mon Oct 16 12:27:04 2023","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.46400000000000002,"15min":0.46899999999999997},"min":{"1min":0.32200000000000001,"5min":0.307,"15min":0.26900000000000002},"max":{"1min":0.54800000000000004,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.44,"5min":0.48799999999999999,"15min":0.495},"min":{"1min":0.30599999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.73299999999999998,"5min":0.752,"15min":0.752},"last":0.35999999999999999}]},{"osd":4,"last update":"Mon Oct 16 12:27:04 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.622,"15min":0.64700000000000002},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.85399999999999998,"5min":0.92700000000000005,"15min":1.1539999999999999},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.64200000000000002,"15min":0.66700000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.78400000000000003,"5min":1.0409999999999999,"15min":1.234},"last":0.66100000000000003}]},{"osd":5,"last update":"Mon Oct 16 12:27:04 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.67000000000000004,"15min":0.68400000000000005},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.92800000000000005,"5min":1.0009999999999999,"15min":1.194},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.64500000000000002,"15min":0.66600000000000004},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.86499999999999999,"5min":0.97299999999999998,"15min":1.202},"last":0.46700000000000003}]},{"osd":6,"last update":"Mon Oct 16 12:27:04 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.65900000000000003,"15min":0.67400000000000004},"min":{"1min":0.42999999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.71099999999999997,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.626},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.68500000000000005,"15min":0.70499999999999996},"min":{"1min":0.45100000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.89600000000000002,"5min":1.0349999999999999,"15min":1.1559999999999999},"last":0.53200000000000003}]},{"osd":7,"last update":"Mon Oct 16 12:27:04 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.66200000000000003,"15min":0.68700000000000006},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.38},"max":{"1min":0.78400000000000003,"5min":1.0189999999999999,"15min":1.2430000000000001},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.69999999999999996,"15min":0.70599999999999996},"min":{"1min":0.44,"5min":0.44,"15min":0.42699999999999999},"max":{"1min":0.89700000000000002,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.57399999999999995}]}]},{"osd":4,"up_from":13,"seq":55834575050,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32524,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339316,"statfs":{"total":96636764160,"available":96603459584,"internally_reserved":0,"allocated":1376256,"data_stored":684267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:27:02 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.56699999999999995,"15min":0.58799999999999997},"min":{"1min":0.32000000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.78000000000000003,"5min":0.85099999999999998,"15min":0.94599999999999995},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.57099999999999995,"15min":0.60499999999999998},"min":{"1min":0.32800000000000001,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.83199999999999996,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.50600000000000001}]},{"osd":1,"last update":"Mon Oct 16 12:27:02 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.57599999999999996,"15min":0.60899999999999999},"min":{"1min":0.36399999999999999,"5min":0.35499999999999998,"15min":0.28499999999999998},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.92600000000000005},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.59999999999999998,"15min":0.621},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.33600000000000002},"max":{"1min":0.80300000000000005,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.53600000000000003}]},{"osd":2,"last update":"Mon Oct 16 12:27:02 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.60199999999999998,"15min":0.626},"min":{"1min":0.39800000000000002,"5min":0.35399999999999998,"15min":0.32000000000000001},"max":{"1min":0.83399999999999996,"5min":0.93000000000000005,"15min":1.012},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.59399999999999997,"15min":0.626},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.34100000000000003},"max":{"1min":0.751,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.373}]},{"osd":3,"last update":"Mon Oct 16 12:27:02 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.58099999999999996,"15min":0.60699999999999998},"min":{"1min":0.40899999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.71999999999999997,"5min":0.79500000000000004,"15min":0.93200000000000005},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.63,"5min":0.627,"15min":0.64100000000000001},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.97199999999999998},"last":0.56299999999999994}]},{"osd":5,"last update":"Mon Oct 16 12:27:02 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.499,"15min":0.497},"min":{"1min":0.377,"5min":0.34000000000000002,"15min":0.27600000000000002},"max":{"1min":0.79700000000000004,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.34200000000000003,"5min":0.316,"15min":0.29599999999999999},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.442}]},{"osd":6,"last update":"Mon Oct 16 12:27:02 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.55100000000000005,"15min":0.55400000000000005},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.28699999999999998},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.63},{"interface":"front","average":{"1min":0.628,"5min":0.63300000000000001,"15min":0.63},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.32000000000000001},"max":{"1min":0.879,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.59099999999999997}]},{"osd":7,"last update":"Mon Oct 16 12:27:02 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.58499999999999996,"15min":0.57799999999999996},"min":{"1min":0.45200000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.64100000000000001,"15min":0.62},"min":{"1min":0.38300000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.66400000000000003}]}]},{"osd":5,"up_from":13,"seq":55834575051,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54728,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317112,"statfs":{"total":96636764160,"available":96580722688,"internally_reserved":0,"allocated":1368064,"data_stored":684107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:26:58 2023","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.56899999999999995,"15min":0.59399999999999997},"min":{"1min":0.32500000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.64600000000000002,"5min":0.84499999999999997,"15min":0.873},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.59399999999999997,"15min":0.62},"min":{"1min":0.36299999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.73499999999999999,"5min":0.85899999999999999,"15min":0.88800000000000001},"last":0.54400000000000004}]},{"osd":1,"last update":"Mon Oct 16 12:26:58 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.60499999999999998,"15min":0.61499999999999999},"min":{"1min":0.39500000000000002,"5min":0.32600000000000001,"15min":0.28999999999999998},"max":{"1min":0.76600000000000001,"5min":0.89700000000000002,"15min":0.91300000000000003},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.503,"5min":0.60699999999999998,"15min":0.62},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.78900000000000003,"5min":0.85999999999999999,"15min":0.89000000000000001},"last":0.65500000000000003}]},{"osd":2,"last update":"Mon Oct 16 12:26:58 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.60299999999999998,"15min":0.623},"min":{"1min":0.40000000000000002,"5min":0.29099999999999998,"15min":0.26400000000000001},"max":{"1min":0.71299999999999997,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.59599999999999997,"15min":0.61399999999999999},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.66900000000000004,"5min":0.88600000000000001,"15min":0.89700000000000002},"last":0.61199999999999999}]},{"osd":3,"last update":"Mon Oct 16 12:26:58 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.61599999999999999,"15min":0.64100000000000001},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.72099999999999997,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.752},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.64000000000000001,"15min":0.65000000000000002},"min":{"1min":0.40400000000000003,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.71799999999999997,"5min":0.90800000000000003,"15min":0.93200000000000005},"last":0.72899999999999998}]},{"osd":4,"last update":"Mon Oct 16 12:26:58 2023","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.504,"15min":0.51000000000000001},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.25800000000000001},"max":{"1min":0.77100000000000002,"5min":0.85699999999999998,"15min":0.86399999999999999},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.58499999999999996,"15min":0.57099999999999995},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33500000000000002},"max":{"1min":0.78500000000000003,"5min":1.347,"15min":1.347},"last":0.876}]},{"osd":6,"last update":"Mon Oct 16 12:26:58 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.54400000000000004,"15min":0.55200000000000005},"min":{"1min":0.33100000000000002,"5min":0.314,"15min":0.314},"max":{"1min":0.83999999999999997,"5min":0.89400000000000002,"15min":0.93500000000000005},"last":0.91800000000000004},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.59999999999999998,"15min":0.59799999999999998},"min":{"1min":0.38300000000000001,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.82899999999999996,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.79500000000000004}]},{"osd":7,"last update":"Mon Oct 16 12:26:58 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.61699999999999999,"15min":0.61899999999999999},"min":{"1min":0.41499999999999998,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.79400000000000004,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.65300000000000002,"15min":0.65500000000000003},"min":{"1min":0.45600000000000002,"5min":0.379,"15min":0.379},"max":{"1min":0.88900000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.81899999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T12:27:42.570 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-16T12:27:42.571 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-16T12:27:42.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.735+0000 7f69d6419700 1 Processor -- start 2023-10-16T12:27:42.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.735+0000 7f69d6419700 1 -- start start 2023-10-16T12:27:42.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.736+0000 7f69d6419700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f69d015b070 0x7f69d015b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:42.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.736+0000 7f69d6419700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f69d00cb880 0x7f69d00cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:42.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.736+0000 7f69d6419700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69d00cef70 con 0x7f69d006da60 2023-10-16T12:27:42.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.736+0000 7f69d6419700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69d00ce360 con 0x7f69d00cb880 2023-10-16T12:27:42.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.736+0000 7f69d6419700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69d00ce640 con 0x7f69d015b070 2023-10-16T12:27:42.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.736+0000 7f69d49b6700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f69d006da60 0x7f69d00cf400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59888/0 (socket says 172.21.15.16:59888) 2023-10-16T12:27:42.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.736+0000 7f69d49b6700 1 -- 172.21.15.16:0/2795855165 learned_addr learned my addr 172.21.15.16:0/2795855165 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:27:42.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.736+0000 7f69cf7fe700 1 --2- 172.21.15.16:0/2795855165 >> v2:172.21.15.49:3300/0 conn(0x7f69d00cb880 0x7f69d00cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:42.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.736+0000 7f69cffff700 1 --2- 172.21.15.16:0/2795855165 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f69d015b070 0x7f69d015b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:42.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.736+0000 7f69ceffd700 1 -- 172.21.15.16:0/2795855165 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2136011548 0 0) 0x7f69d00cef70 con 0x7f69d006da60 2023-10-16T12:27:42.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.736+0000 7f69ceffd700 1 -- 172.21.15.16:0/2795855165 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69bc0035b0 con 0x7f69d006da60 2023-10-16T12:27:42.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.736+0000 7f69ceffd700 1 -- 172.21.15.16:0/2795855165 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2560451060 0 0) 0x7f69bc0035b0 con 0x7f69d006da60 2023-10-16T12:27:42.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.736+0000 7f69ceffd700 1 -- 172.21.15.16:0/2795855165 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f69d015b070 msgr2=0x7f69d015b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:42.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.736+0000 7f69ceffd700 1 --2- 172.21.15.16:0/2795855165 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f69d015b070 0x7f69d015b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:42.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.736+0000 7f69ceffd700 1 -- 172.21.15.16:0/2795855165 >> v2:172.21.15.49:3300/0 conn(0x7f69d00cb880 msgr2=0x7f69d00cbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:42.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.736+0000 7f69ceffd700 1 --2- 172.21.15.16:0/2795855165 >> v2:172.21.15.49:3300/0 conn(0x7f69d00cb880 0x7f69d00cbd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:42.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.736+0000 7f69ceffd700 1 -- 172.21.15.16:0/2795855165 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69d00ce920 con 0x7f69d006da60 2023-10-16T12:27:42.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.736+0000 7f69ceffd700 1 -- 172.21.15.16:0/2795855165 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f69c80019f0 con 0x7f69d006da60 2023-10-16T12:27:42.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.736+0000 7f69ceffd700 1 -- 172.21.15.16:0/2795855165 >> v1:172.21.15.16:6789/0 conn(0x7f69d006da60 legacy=0x7f69d00cf400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:42.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.736+0000 7f69ceffd700 1 --2- 172.21.15.16:0/2795855165 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69bc003a00 0x7f69bc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:42.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.736+0000 7f69ceffd700 1 --2- 172.21.15.16:0/2795855165 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f69d015b070 0x7f69bc004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:42.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.737+0000 7f69ceffd700 1 --2- 172.21.15.16:0/2795855165 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f69bc004d90 0x7f69bc005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:42.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.737+0000 7f69ceffd700 1 -- 172.21.15.16:0/2795855165 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69d00ce360 con 0x7f69bc003a00 2023-10-16T12:27:42.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.737+0000 7f69ceffd700 1 -- 172.21.15.16:0/2795855165 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69d00ce640 con 0x7f69bc004d90 2023-10-16T12:27:42.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.737+0000 7f69ceffd700 1 -- 172.21.15.16:0/2795855165 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69bc00ba10 con 0x7f69d015b070 2023-10-16T12:27:42.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.737+0000 7f69cffff700 1 --2- 172.21.15.16:0/2795855165 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69bc003a00 0x7f69bc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:42.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.737+0000 7f69cf7fe700 1 --2- 172.21.15.16:0/2795855165 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f69d015b070 0x7f69bc004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:42.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.737+0000 7f69d49b6700 1 --2- 172.21.15.16:0/2795855165 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f69bc004d90 0x7f69bc005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:42.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.737+0000 7f69cffff700 1 -- 172.21.15.16:0/2795855165 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f69d015b070 msgr2=0x7f69bc004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:42.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.737+0000 7f69cffff700 1 --2- 172.21.15.16:0/2795855165 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f69d015b070 0x7f69bc004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:42.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.737+0000 7f69cffff700 1 -- 172.21.15.16:0/2795855165 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f69bc004d90 msgr2=0x7f69bc005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:42.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.737+0000 7f69cffff700 1 --2- 172.21.15.16:0/2795855165 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f69bc004d90 0x7f69bc005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:42.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.737+0000 7f69cffff700 1 -- 172.21.15.16:0/2795855165 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69bc0035b0 con 0x7f69bc003a00 2023-10-16T12:27:42.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.737+0000 7f69cffff700 1 --2- 172.21.15.16:0/2795855165 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69bc003a00 0x7f69bc003e20 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f69c0009b60 tx=0x7f69bc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a86234f4eda4d87 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:42.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.737+0000 7f69ceffd700 1 -- 172.21.15.16:0/2795855165 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69c000e6c0 con 0x7f69bc003a00 2023-10-16T12:27:42.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.737+0000 7f69ceffd700 1 -- 172.21.15.16:0/2795855165 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69c000e820 con 0x7f69bc003a00 2023-10-16T12:27:42.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.737+0000 7f69ceffd700 1 -- 172.21.15.16:0/2795855165 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69c000eb90 con 0x7f69bc003a00 2023-10-16T12:27:42.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.738+0000 7f69d6419700 1 -- 172.21.15.16:0/2795855165 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69bc003a00 msgr2=0x7f69bc003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:42.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.738+0000 7f69d6419700 1 --2- 172.21.15.16:0/2795855165 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69bc003a00 0x7f69bc003e20 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f69c0009b60 tx=0x7f69bc005880 comp rx=0 tx=0).stop 2023-10-16T12:27:42.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.738+0000 7f69d49b6700 1 -- 172.21.15.16:0/2795855165 reap_dead start 2023-10-16T12:27:42.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.738+0000 7f69d6419700 1 -- 172.21.15.16:0/2795855165 shutdown_connections 2023-10-16T12:27:42.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.738+0000 7f69d6419700 1 -- 172.21.15.16:0/2795855165 >> 172.21.15.16:0/2795855165 conn(0x7f69d00b1f40 msgr2=0x7f69d00b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:42.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.738+0000 7f69d6419700 1 -- 172.21.15.16:0/2795855165 shutdown_connections 2023-10-16T12:27:42.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.738+0000 7f69d6419700 1 -- 172.21.15.16:0/2795855165 wait complete. 2023-10-16T12:27:42.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.739+0000 7f69d6419700 1 Processor -- start 2023-10-16T12:27:42.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.739+0000 7f69d6419700 1 -- start start 2023-10-16T12:27:42.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.740+0000 7f69d6419700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f69bc003a00 0x7f69d00c6050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:42.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.740+0000 7f69d6419700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69d01f0fd0 0x7f69d01474b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:42.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.740+0000 7f69d6419700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f69d0158eb0 0x7f69d0159360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:42.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.740+0000 7f69d6419700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69d01598a0 con 0x7f69d01f0fd0 2023-10-16T12:27:42.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.740+0000 7f69d6419700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69d00c9400 con 0x7f69d0158eb0 2023-10-16T12:27:42.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.740+0000 7f69d6419700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69d01479f0 con 0x7f69bc003a00 2023-10-16T12:27:42.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.740+0000 7f69cf7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69d01f0fd0 0x7f69d01474b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:42.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.740+0000 7f69cf7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69d01f0fd0 0x7f69d01474b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42392/0 (socket says 172.21.15.16:42392) 2023-10-16T12:27:42.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.740+0000 7f69cf7fe700 1 -- 172.21.15.16:0/3207603595 learned_addr learned my addr 172.21.15.16:0/3207603595 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:42.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.740+0000 7f69cffff700 1 --2- 172.21.15.16:0/3207603595 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f69bc003a00 0x7f69d00c6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:42.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.740+0000 7f69d49b6700 1 --2- 172.21.15.16:0/3207603595 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f69d0158eb0 0x7f69d0159360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:42.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.740+0000 7f69cf7fe700 1 -- 172.21.15.16:0/3207603595 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f69bc003a00 msgr2=0x7f69d00c6050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:42.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.740+0000 7f69cf7fe700 1 --2- 172.21.15.16:0/3207603595 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f69bc003a00 0x7f69d00c6050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:42.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.740+0000 7f69cf7fe700 1 -- 172.21.15.16:0/3207603595 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f69d0158eb0 msgr2=0x7f69d0159360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:42.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.740+0000 7f69cf7fe700 1 --2- 172.21.15.16:0/3207603595 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f69d0158eb0 0x7f69d0159360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:42.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.740+0000 7f69cf7fe700 1 -- 172.21.15.16:0/3207603595 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69d01f1460 con 0x7f69d01f0fd0 2023-10-16T12:27:42.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.740+0000 7f69cf7fe700 1 --2- 172.21.15.16:0/3207603595 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69d01f0fd0 0x7f69d01474b0 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f69b80116f0 tx=0x7f69b8011a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:42.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.741+0000 7f69ccff9700 1 -- 172.21.15.16:0/3207603595 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b8015870 con 0x7f69d01f0fd0 2023-10-16T12:27:42.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.741+0000 7f69ccff9700 1 -- 172.21.15.16:0/3207603595 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69b80159d0 con 0x7f69d01f0fd0 2023-10-16T12:27:42.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.741+0000 7f69ccff9700 1 -- 172.21.15.16:0/3207603595 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b8015d40 con 0x7f69d01f0fd0 2023-10-16T12:27:42.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.741+0000 7f69d6419700 1 -- 172.21.15.16:0/3207603595 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69d01f1740 con 0x7f69d01f0fd0 2023-10-16T12:27:42.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.741+0000 7f69d6419700 1 -- 172.21.15.16:0/3207603595 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69d01f1d00 con 0x7f69d01f0fd0 2023-10-16T12:27:42.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.742+0000 7f69d6419700 1 -- 172.21.15.16:0/3207603595 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69d01f1ec0 con 0x7f69d01f0fd0 2023-10-16T12:27:42.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.743+0000 7f69ccff9700 1 -- 172.21.15.16:0/3207603595 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 51) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f69b8013400 con 0x7f69d01f0fd0 2023-10-16T12:27:42.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.743+0000 7f69ccff9700 1 --2- 172.21.15.16:0/3207603595 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f69b006b020 0x7f69b006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:42.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.743+0000 7f69ccff9700 1 -- 172.21.15.16:0/3207603595 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f69b808a2b0 con 0x7f69d01f0fd0 2023-10-16T12:27:42.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.743+0000 7f69cffff700 1 --2- 172.21.15.16:0/3207603595 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f69b006b020 0x7f69b006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:42.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.744+0000 7f69cffff700 1 --2- 172.21.15.16:0/3207603595 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f69b006b020 0x7f69b006d4e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f69c00090b0 tx=0x7f69c001f040 comp rx=0 tx=0).ready entity=mgr.6365 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:42.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:42.747+0000 7f69ccff9700 1 -- 172.21.15.16:0/3207603595 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f69b80551d0 con 0x7f69d01f0fd0 2023-10-16T12:27:43.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.033+0000 7f69d6419700 1 -- 172.21.15.16:0/3207603595 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f69d00c9590 con 0x7f69d01f0fd0 2023-10-16T12:27:43.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.035+0000 7f69ccff9700 1 -- 172.21.15.16:0/3207603595 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v22) v1 ==== 190+0+147 (secure 0 0 0) 0x7f69b80587f0 con 0x7f69d01f0fd0 2023-10-16T12:27:43.037 INFO:teuthology.orchestra.run.smithi016.stdout:[client.0] 2023-10-16T12:27:43.037 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAPKC1l2l6PFxAA9QyXtBAHdGAXpwH0JfknuQ== 2023-10-16T12:27:43.038 INFO:teuthology.orchestra.run.smithi016.stdout: caps mds = "allow" 2023-10-16T12:27:43.038 INFO:teuthology.orchestra.run.smithi016.stdout: caps mon = "allow r" 2023-10-16T12:27:43.038 INFO:teuthology.orchestra.run.smithi016.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-16T12:27:43.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.037+0000 7f69d6419700 1 -- 172.21.15.16:0/3207603595 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f69b006b020 msgr2=0x7f69b006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:43.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.037+0000 7f69d6419700 1 --2- 172.21.15.16:0/3207603595 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f69b006b020 0x7f69b006d4e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f69c00090b0 tx=0x7f69c001f040 comp rx=0 tx=0).stop 2023-10-16T12:27:43.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.037+0000 7f69d6419700 1 -- 172.21.15.16:0/3207603595 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69d01f0fd0 msgr2=0x7f69d01474b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:43.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.037+0000 7f69d6419700 1 --2- 172.21.15.16:0/3207603595 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69d01f0fd0 0x7f69d01474b0 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f69b80116f0 tx=0x7f69b8011a00 comp rx=0 tx=0).stop 2023-10-16T12:27:43.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.037+0000 7f69d6419700 1 -- 172.21.15.16:0/3207603595 shutdown_connections 2023-10-16T12:27:43.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.037+0000 7f69d6419700 1 --2- 172.21.15.16:0/3207603595 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f69b006b020 0x7f69b006d4e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:43.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.037+0000 7f69d6419700 1 --2- 172.21.15.16:0/3207603595 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f69d0158eb0 0x7f69d0159360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:43.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.037+0000 7f69d6419700 1 --2- 172.21.15.16:0/3207603595 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f69bc003a00 0x7f69d00c6050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:43.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.037+0000 7f69d6419700 1 --2- 172.21.15.16:0/3207603595 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69d01f0fd0 0x7f69d01474b0 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:43.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.037+0000 7f69d6419700 1 -- 172.21.15.16:0/3207603595 >> 172.21.15.16:0/3207603595 conn(0x7f69d00b1f40 msgr2=0x7f69d00ccc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:43.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.038+0000 7f69d6419700 1 -- 172.21.15.16:0/3207603595 shutdown_connections 2023-10-16T12:27:43.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.038+0000 7f69d6419700 1 -- 172.21.15.16:0/3207603595 wait complete. 2023-10-16T12:27:43.042 INFO:teuthology.orchestra.run.smithi016.stderr:updated caps for client.0 2023-10-16T12:27:43.052 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:27:43.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.176+0000 7fd8fdb75700 1 Processor -- start 2023-10-16T12:27:43.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.176+0000 7fd8fdb75700 1 -- start start 2023-10-16T12:27:43.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.177+0000 7fd8fdb75700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd8f80cb280 0x7fd8f806d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:43.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.177+0000 7fd8fdb75700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd8f806b1f0 0x7fd8f806b680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:43.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.177+0000 7fd8fdb75700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8f806c150 con 0x7fd8f8160eb0 2023-10-16T12:27:43.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.177+0000 7fd8fdb75700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8f80af370 con 0x7fd8f80cb280 2023-10-16T12:27:43.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.177+0000 7fd8fdb75700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f806bbc0 con 0x7fd8f806b1f0 2023-10-16T12:27:43.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.177+0000 7fd8effff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd8f80cb280 0x7fd8f806d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:43.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.177+0000 7fd8effff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd8f80cb280 0x7fd8f806d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42400/0 (socket says 172.21.15.16:42400) 2023-10-16T12:27:43.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.177+0000 7fd8effff700 1 -- 172.21.15.16:0/901869762 learned_addr learned my addr 172.21.15.16:0/901869762 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:43.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.177+0000 7fd8f7fff700 1 --2- 172.21.15.16:0/901869762 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd8f806b1f0 0x7fd8f806b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:43.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.177+0000 7fd8effff700 1 -- 172.21.15.16:0/901869762 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd8f806b1f0 msgr2=0x7fd8f806b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:43.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.177+0000 7fd8effff700 1 --2- 172.21.15.16:0/901869762 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd8f806b1f0 0x7fd8f806b680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:43.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.177+0000 7fd8effff700 1 -- 172.21.15.16:0/901869762 >> v1:172.21.15.49:6789/0 conn(0x7fd8f8160eb0 legacy=0x7fd8f80cab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:43.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.177+0000 7fd8effff700 1 -- 172.21.15.16:0/901869762 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8f806be70 con 0x7fd8f80cb280 2023-10-16T12:27:43.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.177+0000 7fd8effff700 1 --2- 172.21.15.16:0/901869762 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8f80cb280 0x7fd8f806d8f0 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7fd8e00098f0 tx=0x7fd8e0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7823a46df7739546 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:43.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.177+0000 7fd8f6ffd700 1 -- 172.21.15.16:0/901869762 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8e0004230 con 0x7fd8f80cb280 2023-10-16T12:27:43.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.177+0000 7fd8f6ffd700 1 -- 172.21.15.16:0/901869762 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8e0004390 con 0x7fd8f80cb280 2023-10-16T12:27:43.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.177+0000 7fd8f6ffd700 1 -- 172.21.15.16:0/901869762 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8e00036a0 con 0x7fd8f80cb280 2023-10-16T12:27:43.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.178+0000 7fd8fdb75700 1 -- 172.21.15.16:0/901869762 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8f80cb280 msgr2=0x7fd8f806d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:43.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.178+0000 7fd8fdb75700 1 --2- 172.21.15.16:0/901869762 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8f80cb280 0x7fd8f806d8f0 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7fd8e00098f0 tx=0x7fd8e0009c00 comp rx=0 tx=0).stop 2023-10-16T12:27:43.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.178+0000 7fd8fdb75700 1 -- 172.21.15.16:0/901869762 shutdown_connections 2023-10-16T12:27:43.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.178+0000 7fd8fdb75700 1 --2- 172.21.15.16:0/901869762 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd8f806b1f0 0x7fd8f806b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:43.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.178+0000 7fd8fdb75700 1 --2- 172.21.15.16:0/901869762 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8f80cb280 0x7fd8f806d8f0 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:43.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.178+0000 7fd8fdb75700 1 -- 172.21.15.16:0/901869762 >> 172.21.15.16:0/901869762 conn(0x7fd8f80cc5b0 msgr2=0x7fd8f80cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:43.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.178+0000 7fd8fdb75700 1 -- 172.21.15.16:0/901869762 shutdown_connections 2023-10-16T12:27:43.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.178+0000 7fd8fdb75700 1 -- 172.21.15.16:0/901869762 wait complete. 2023-10-16T12:27:43.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.179+0000 7fd8fdb75700 1 Processor -- start 2023-10-16T12:27:43.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.179+0000 7fd8fdb75700 1 -- start start 2023-10-16T12:27:43.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.180+0000 7fd8fdb75700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd8f806b1f0 0x7fd8f8156110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:43.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.180+0000 7fd8fdb75700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8f80cb280 0x7fd8f80c5e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:43.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.180+0000 7fd8fdb75700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd8f8160eb0 0x7fd8f80c87c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:43.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.180+0000 7fd8fdb75700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f8156650 con 0x7fd8f80cb280 2023-10-16T12:27:43.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.180+0000 7fd8fdb75700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f81577b0 con 0x7fd8f8160eb0 2023-10-16T12:27:43.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.180+0000 7fd8fdb75700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f80c45e0 con 0x7fd8f806b1f0 2023-10-16T12:27:43.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.180+0000 7fd8effff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8f80cb280 0x7fd8f80c5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:43.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.180+0000 7fd8effff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8f80cb280 0x7fd8f80c5e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42406/0 (socket says 172.21.15.16:42406) 2023-10-16T12:27:43.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.180+0000 7fd8effff700 1 -- 172.21.15.16:0/831205827 learned_addr learned my addr 172.21.15.16:0/831205827 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:43.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.180+0000 7fd8f7fff700 1 --2- 172.21.15.16:0/831205827 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd8f8160eb0 0x7fd8f80c87c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:43.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.180+0000 7fd8f77fe700 1 --2- 172.21.15.16:0/831205827 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd8f806b1f0 0x7fd8f8156110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:43.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.181+0000 7fd8effff700 1 -- 172.21.15.16:0/831205827 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd8f806b1f0 msgr2=0x7fd8f8156110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:43.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.181+0000 7fd8effff700 1 --2- 172.21.15.16:0/831205827 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd8f806b1f0 0x7fd8f8156110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:43.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.181+0000 7fd8effff700 1 -- 172.21.15.16:0/831205827 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd8f8160eb0 msgr2=0x7fd8f80c87c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:43.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.181+0000 7fd8effff700 1 --2- 172.21.15.16:0/831205827 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd8f8160eb0 0x7fd8f80c87c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:43.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.181+0000 7fd8effff700 1 -- 172.21.15.16:0/831205827 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8e0009550 con 0x7fd8f80cb280 2023-10-16T12:27:43.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.181+0000 7fd8f7fff700 1 --2- 172.21.15.16:0/831205827 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd8f8160eb0 0x7fd8f80c87c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:27:43.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.181+0000 7fd8effff700 1 --2- 172.21.15.16:0/831205827 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8f80cb280 0x7fd8f80c5e30 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7fd8e0009130 tx=0x7fd8e0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:43.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.181+0000 7fd8f4ff9700 1 -- 172.21.15.16:0/831205827 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8e000b860 con 0x7fd8f80cb280 2023-10-16T12:27:43.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.181+0000 7fd8f4ff9700 1 -- 172.21.15.16:0/831205827 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8e000b9c0 con 0x7fd8f80cb280 2023-10-16T12:27:43.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.181+0000 7fd8fdb75700 1 -- 172.21.15.16:0/831205827 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8f80c48c0 con 0x7fd8f80cb280 2023-10-16T12:27:43.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.181+0000 7fd8f4ff9700 1 -- 172.21.15.16:0/831205827 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8e000bd70 con 0x7fd8f80cb280 2023-10-16T12:27:43.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.181+0000 7fd8fdb75700 1 -- 172.21.15.16:0/831205827 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8f80c4e80 con 0x7fd8f80cb280 2023-10-16T12:27:43.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.182+0000 7fd8fdb75700 1 -- 172.21.15.16:0/831205827 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8f804ebb0 con 0x7fd8f80cb280 2023-10-16T12:27:43.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.182+0000 7fd8f4ff9700 1 -- 172.21.15.16:0/831205827 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 51) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd8e001b440 con 0x7fd8f80cb280 2023-10-16T12:27:43.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.183+0000 7fd8f4ff9700 1 --2- 172.21.15.16:0/831205827 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fd8d806b020 0x7fd8d806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:43.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.183+0000 7fd8f4ff9700 1 -- 172.21.15.16:0/831205827 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fd8e008b2f0 con 0x7fd8f80cb280 2023-10-16T12:27:43.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.183+0000 7fd8f77fe700 1 --2- 172.21.15.16:0/831205827 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fd8d806b020 0x7fd8d806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:43.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.183+0000 7fd8f77fe700 1 --2- 172.21.15.16:0/831205827 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fd8d806b020 0x7fd8d806d4e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd8e8004f30 tx=0x7fd8e8009040 comp rx=0 tx=0).ready entity=mgr.6365 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:43.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.187+0000 7fd8f4ff9700 1 -- 172.21.15.16:0/831205827 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd8e0056120 con 0x7fd8f80cb280 2023-10-16T12:27:43.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.438+0000 7fd8fdb75700 1 -- 172.21.15.16:0/831205827 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd8f80aeb00 con 0x7fd8f80cb280 2023-10-16T12:27:43.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.438+0000 7fd8f4ff9700 1 -- 172.21.15.16:0/831205827 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+4903 (secure 0 0 0) 0x7fd8e0059740 con 0x7fd8f80cb280 2023-10-16T12:27:43.440 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:27:43.441 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":53,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6524,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/854923476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":854923476},{"type":"v1","addr":"172.21.15.49:6834","nonce":854923476}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":7249,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/1186699359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1186699359},{"type":"v1","addr":"172.21.15.16:6836","nonce":1186699359}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":7252,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/22289021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":22289021},{"type":"v1","addr":"172.21.15.16:6837","nonce":22289021}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:27:37.397811+0000","modified":"2023-10-16T12:27:40.428748+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7258},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7258":{"gid":7258,"name":"b","rank":0,"incarnation":49,"state":"up:active","state_seq":3,"addr":"172.21.15.49:6835/729289774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":729289774},{"type":"v1","addr":"172.21.15.49:6835","nonce":729289774}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-10-16T12:27:43.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.440+0000 7fd8fdb75700 1 -- 172.21.15.16:0/831205827 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fd8d806b020 msgr2=0x7fd8d806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:43.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.440+0000 7fd8fdb75700 1 --2- 172.21.15.16:0/831205827 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fd8d806b020 0x7fd8d806d4e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd8e8004f30 tx=0x7fd8e8009040 comp rx=0 tx=0).stop 2023-10-16T12:27:43.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.440+0000 7fd8fdb75700 1 -- 172.21.15.16:0/831205827 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8f80cb280 msgr2=0x7fd8f80c5e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:43.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.440+0000 7fd8fdb75700 1 --2- 172.21.15.16:0/831205827 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8f80cb280 0x7fd8f80c5e30 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7fd8e0009130 tx=0x7fd8e0004750 comp rx=0 tx=0).stop 2023-10-16T12:27:43.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.440+0000 7fd8fdb75700 1 -- 172.21.15.16:0/831205827 shutdown_connections 2023-10-16T12:27:43.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.440+0000 7fd8fdb75700 1 --2- 172.21.15.16:0/831205827 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fd8d806b020 0x7fd8d806d4e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:43.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.440+0000 7fd8fdb75700 1 --2- 172.21.15.16:0/831205827 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd8f8160eb0 0x7fd8f80c87c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:43.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.440+0000 7fd8fdb75700 1 --2- 172.21.15.16:0/831205827 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd8f806b1f0 0x7fd8f8156110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:43.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.440+0000 7fd8fdb75700 1 --2- 172.21.15.16:0/831205827 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8f80cb280 0x7fd8f80c5e30 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:43.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.441+0000 7fd8fdb75700 1 -- 172.21.15.16:0/831205827 >> 172.21.15.16:0/831205827 conn(0x7fd8f80cc5b0 msgr2=0x7fd8f8071960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:43.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.441+0000 7fd8fdb75700 1 -- 172.21.15.16:0/831205827 shutdown_connections 2023-10-16T12:27:43.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.441+0000 7fd8fdb75700 1 -- 172.21.15.16:0/831205827 wait complete. 2023-10-16T12:27:43.445 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 53 2023-10-16T12:27:43.857 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 53, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T12:27:37.397811+0000', 'modified': '2023-10-16T12:27:40.428748+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7258}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7258': {'gid': 7258, 'name': 'b', 'rank': 0, 'incarnation': 49, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.49:6835/729289774', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6833', 'nonce': 729289774}, {'type': 'v1', 'addr': '172.21.15.49:6835', 'nonce': 729289774}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [14], 'metadata_pool': 13, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-16T12:27:43.857 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-16T12:27:43.858 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-10-16T12:27:43.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.992+0000 7f163e26c700 1 Processor -- start 2023-10-16T12:27:43.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.992+0000 7f163e26c700 1 -- start start 2023-10-16T12:27:43.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.993+0000 7f163e26c700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f16380cf4a0 0x7f16380726e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:43.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.993+0000 7f163e26c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f16380cf880 0x7f16380b52f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:43.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.993+0000 7f163e26c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f16380b5830 0x7f163815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:43.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.993+0000 7f163e26c700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f163815bc80 con 0x7f16380cf880 2023-10-16T12:27:43.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.993+0000 7f163e26c700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1638072c20 con 0x7f16380cf4a0 2023-10-16T12:27:43.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.993+0000 7f163e26c700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f163815b070 con 0x7f16380b5830 2023-10-16T12:27:43.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.993+0000 7f16377fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f16380cf880 0x7f16380b52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:43.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.993+0000 7f16377fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f16380cf880 0x7f16380b52f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42412/0 (socket says 172.21.15.16:42412) 2023-10-16T12:27:43.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.993+0000 7f16377fe700 1 -- 172.21.15.16:0/40484666 learned_addr learned my addr 172.21.15.16:0/40484666 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:44.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.993+0000 7f163c809700 1 --2- 172.21.15.16:0/40484666 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f16380b5830 0x7f163815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:44.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.993+0000 7f1637fff700 1 --2- 172.21.15.16:0/40484666 >> v2:172.21.15.49:3300/0 conn(0x7f16380cf4a0 0x7f16380726e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:44.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.994+0000 7f16377fe700 1 -- 172.21.15.16:0/40484666 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f16380b5830 msgr2=0x7f163815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:44.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.994+0000 7f16377fe700 1 --2- 172.21.15.16:0/40484666 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f16380b5830 0x7f163815a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:44.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.994+0000 7f16377fe700 1 -- 172.21.15.16:0/40484666 >> v2:172.21.15.49:3300/0 conn(0x7f16380cf4a0 msgr2=0x7f16380726e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:44.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.994+0000 7f16377fe700 1 --2- 172.21.15.16:0/40484666 >> v2:172.21.15.49:3300/0 conn(0x7f16380cf4a0 0x7f16380726e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:44.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.994+0000 7f16377fe700 1 -- 172.21.15.16:0/40484666 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f163815b350 con 0x7f16380cf880 2023-10-16T12:27:44.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.994+0000 7f1637fff700 1 --2- 172.21.15.16:0/40484666 >> v2:172.21.15.49:3300/0 conn(0x7f16380cf4a0 0x7f16380726e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:27:44.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.994+0000 7f16377fe700 1 --2- 172.21.15.16:0/40484666 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16380cf880 0x7f16380b52f0 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f16380b2d20 tx=0x7f1620009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eede28113abc0f11 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:44.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.994+0000 7f1636ffd700 1 -- 172.21.15.16:0/40484666 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1620011830 con 0x7f16380cf880 2023-10-16T12:27:44.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.994+0000 7f1636ffd700 1 -- 172.21.15.16:0/40484666 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1620011990 con 0x7f16380cf880 2023-10-16T12:27:44.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.994+0000 7f1636ffd700 1 -- 172.21.15.16:0/40484666 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1620011d20 con 0x7f16380cf880 2023-10-16T12:27:44.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.994+0000 7f163e26c700 1 -- 172.21.15.16:0/40484666 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16380cf880 msgr2=0x7f16380b52f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:44.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.994+0000 7f163e26c700 1 --2- 172.21.15.16:0/40484666 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16380cf880 0x7f16380b52f0 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f16380b2d20 tx=0x7f1620009bd0 comp rx=0 tx=0).stop 2023-10-16T12:27:44.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.994+0000 7f163e26c700 1 -- 172.21.15.16:0/40484666 shutdown_connections 2023-10-16T12:27:44.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.994+0000 7f163e26c700 1 --2- 172.21.15.16:0/40484666 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f16380b5830 0x7f163815a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:44.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.994+0000 7f163e26c700 1 --2- 172.21.15.16:0/40484666 >> v2:172.21.15.49:3300/0 conn(0x7f16380cf4a0 0x7f16380726e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:44.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.994+0000 7f163e26c700 1 --2- 172.21.15.16:0/40484666 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16380cf880 0x7f16380b52f0 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:44.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.995+0000 7f163e26c700 1 -- 172.21.15.16:0/40484666 >> 172.21.15.16:0/40484666 conn(0x7f16380b1fd0 msgr2=0x7f16380b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:44.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.995+0000 7f163e26c700 1 -- 172.21.15.16:0/40484666 shutdown_connections 2023-10-16T12:27:44.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.995+0000 7f163e26c700 1 -- 172.21.15.16:0/40484666 wait complete. 2023-10-16T12:27:44.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.996+0000 7f163e26c700 1 Processor -- start 2023-10-16T12:27:44.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.996+0000 7f163e26c700 1 -- start start 2023-10-16T12:27:44.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.997+0000 7f163e26c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f16380b5830 0x7f16380aec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:44.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.997+0000 7f163e26c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16380cf4a0 0x7f163815c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:44.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.997+0000 7f163e26c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f16380cf880 0x7f163815c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:44.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.997+0000 7f163e26c700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16381f1c10 con 0x7f16380cf4a0 2023-10-16T12:27:44.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.997+0000 7f163e26c700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16381f4ae0 con 0x7f16380b5830 2023-10-16T12:27:44.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.997+0000 7f163e26c700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16380ca040 con 0x7f16380cf880 2023-10-16T12:27:44.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.997+0000 7f16377fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16380cf4a0 0x7f163815c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:44.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.997+0000 7f16377fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16380cf4a0 0x7f163815c220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42420/0 (socket says 172.21.15.16:42420) 2023-10-16T12:27:44.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.997+0000 7f16377fe700 1 -- 172.21.15.16:0/1000235511 learned_addr learned my addr 172.21.15.16:0/1000235511 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:44.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.998+0000 7f1637fff700 1 --2- 172.21.15.16:0/1000235511 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f16380b5830 0x7f16380aec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:44.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.998+0000 7f16377fe700 1 -- 172.21.15.16:0/1000235511 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f16380cf880 msgr2=0x7f163815c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:44.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.998+0000 7f163c809700 1 --2- 172.21.15.16:0/1000235511 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f16380cf880 0x7f163815c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:44.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.998+0000 7f16377fe700 1 --2- 172.21.15.16:0/1000235511 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f16380cf880 0x7f163815c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:44.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.998+0000 7f16377fe700 1 -- 172.21.15.16:0/1000235511 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f16380b5830 msgr2=0x7f16380aec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:44.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.998+0000 7f16377fe700 1 --2- 172.21.15.16:0/1000235511 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f16380b5830 0x7f16380aec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:44.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.998+0000 7f16377fe700 1 -- 172.21.15.16:0/1000235511 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1620009550 con 0x7f16380cf4a0 2023-10-16T12:27:44.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.998+0000 7f1637fff700 1 --2- 172.21.15.16:0/1000235511 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f16380b5830 0x7f16380aec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:27:44.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.998+0000 7f163c809700 1 --2- 172.21.15.16:0/1000235511 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f16380cf880 0x7f163815c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:27:44.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.998+0000 7f16377fe700 1 --2- 172.21.15.16:0/1000235511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16380cf4a0 0x7f163815c220 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f1620009c10 tx=0x7f1620004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:44.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.998+0000 7f1634ff9700 1 -- 172.21.15.16:0/1000235511 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16200036a0 con 0x7f16380cf4a0 2023-10-16T12:27:44.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.998+0000 7f1634ff9700 1 -- 172.21.15.16:0/1000235511 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f162001e070 con 0x7f16380cf4a0 2023-10-16T12:27:44.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.998+0000 7f163e26c700 1 -- 172.21.15.16:0/1000235511 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16381f1930 con 0x7f16380cf4a0 2023-10-16T12:27:44.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.998+0000 7f1634ff9700 1 -- 172.21.15.16:0/1000235511 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1620003a40 con 0x7f16380cf4a0 2023-10-16T12:27:44.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:43.998+0000 7f163e26c700 1 -- 172.21.15.16:0/1000235511 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16381f4700 con 0x7f16380cf4a0 2023-10-16T12:27:44.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.000+0000 7f1634ff9700 1 -- 172.21.15.16:0/1000235511 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 51) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1620003ba0 con 0x7f16380cf4a0 2023-10-16T12:27:44.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.000+0000 7f1634ff9700 1 --2- 172.21.15.16:0/1000235511 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f161806b020 0x7f161806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:44.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.000+0000 7f1634ff9700 1 -- 172.21.15.16:0/1000235511 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f162008a820 con 0x7f16380cf4a0 2023-10-16T12:27:44.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.000+0000 7f1637fff700 1 --2- 172.21.15.16:0/1000235511 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f161806b020 0x7f161806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:44.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.000+0000 7f1637fff700 1 --2- 172.21.15.16:0/1000235511 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f161806b020 0x7f161806d4e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f16380c6fe0 tx=0x7f1628008040 comp rx=0 tx=0).ready entity=mgr.6365 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:44.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.002+0000 7f164a456b80 1 Processor -- start 2023-10-16T12:27:44.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.002+0000 7f164a456b80 1 -- start start 2023-10-16T12:27:44.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.003+0000 7f164a456b80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x5641db791b40 0x5641db791f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:44.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.003+0000 7f164a456b80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5641db7b8f30 0x5641db791280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:44.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.003+0000 7f164a456b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5641db78cf00 0x5641db7b8300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:44.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.003+0000 7f164a456b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641db705c20 con 0x5641db78cf00 2023-10-16T12:27:44.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.003+0000 7f164a456b80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641db6eac10 con 0x5641db7b8f30 2023-10-16T12:27:44.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.003+0000 7f164a456b80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x5641db6e8530 con 0x5641db791b40 2023-10-16T12:27:44.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.003+0000 7f163c809700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5641db78cf00 0x5641db7b8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:44.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.003+0000 7f163c809700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5641db78cf00 0x5641db7b8300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42430/0 (socket says 172.21.15.16:42430) 2023-10-16T12:27:44.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.003+0000 7f163c809700 1 -- 172.21.15.16:0/794488379 learned_addr learned my addr 172.21.15.16:0/794488379 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:44.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.003+0000 7f1637fff700 1 --2- 172.21.15.16:0/794488379 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x5641db791b40 0x5641db791f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:44.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.003+0000 7f16377fe700 1 --2- 172.21.15.16:0/794488379 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5641db7b8f30 0x5641db791280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:44.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.003+0000 7f163c809700 1 -- 172.21.15.16:0/794488379 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x5641db791b40 msgr2=0x5641db791f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:44.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.003+0000 7f163c809700 1 --2- 172.21.15.16:0/794488379 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x5641db791b40 0x5641db791f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:44.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.003+0000 7f163c809700 1 -- 172.21.15.16:0/794488379 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5641db7b8f30 msgr2=0x5641db791280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:44.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.003+0000 7f163c809700 1 --2- 172.21.15.16:0/794488379 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5641db7b8f30 0x5641db791280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:44.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.003+0000 7f163c809700 1 -- 172.21.15.16:0/794488379 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5641db6a13a0 con 0x5641db78cf00 2023-10-16T12:27:44.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.003+0000 7f1637fff700 1 --2- 172.21.15.16:0/794488379 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x5641db791b40 0x5641db791f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:27:44.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.003+0000 7f163c809700 1 --2- 172.21.15.16:0/794488379 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5641db78cf00 0x5641db7b8300 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f1630007680 tx=0x7f1630009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37915222ed0a636a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:44.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.003+0000 7f162dffb700 1 -- 172.21.15.16:0/794488379 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16300155f0 con 0x5641db78cf00 2023-10-16T12:27:44.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.003+0000 7f162dffb700 1 -- 172.21.15.16:0/794488379 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1630011040 con 0x5641db78cf00 2023-10-16T12:27:44.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.003+0000 7f162dffb700 1 -- 172.21.15.16:0/794488379 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1630015950 con 0x5641db78cf00 2023-10-16T12:27:44.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.004+0000 7f164a456b80 1 -- 172.21.15.16:0/794488379 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5641db78cf00 msgr2=0x5641db7b8300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:44.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.004+0000 7f164a456b80 1 --2- 172.21.15.16:0/794488379 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5641db78cf00 0x5641db7b8300 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f1630007680 tx=0x7f1630009750 comp rx=0 tx=0).stop 2023-10-16T12:27:44.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.004+0000 7f164a456b80 1 -- 172.21.15.16:0/794488379 shutdown_connections 2023-10-16T12:27:44.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.004+0000 7f164a456b80 1 --2- 172.21.15.16:0/794488379 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5641db78cf00 0x5641db7b8300 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:44.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.004+0000 7f164a456b80 1 --2- 172.21.15.16:0/794488379 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x5641db791b40 0x5641db791f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:44.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.004+0000 7f164a456b80 1 --2- 172.21.15.16:0/794488379 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5641db7b8f30 0x5641db791280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:44.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.004+0000 7f164a456b80 1 -- 172.21.15.16:0/794488379 >> 172.21.15.16:0/794488379 conn(0x5641db6bed80 msgr2=0x5641db7d9ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:44.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.004+0000 7f164a456b80 1 -- 172.21.15.16:0/794488379 shutdown_connections 2023-10-16T12:27:44.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.004+0000 7f164a456b80 1 -- 172.21.15.16:0/794488379 wait complete. 2023-10-16T12:27:44.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.005+0000 7f164a456b80 1 -- start start 2023-10-16T12:27:44.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.005+0000 7f164a456b80 1 Processor -- start 2023-10-16T12:27:44.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.006+0000 7f164a456b80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5641db78cf00 0x5641db7b3c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:44.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.006+0000 7f164a456b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5641db791b40 0x5641db7b5b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:44.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.006+0000 7f164a456b80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x5641db7b8f30 0x5641db7b80d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:44.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.006+0000 7f164a456b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641db6e3e20 con 0x5641db791b40 2023-10-16T12:27:44.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.006+0000 7f164a456b80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641db6806c0 con 0x5641db78cf00 2023-10-16T12:27:44.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.006+0000 7f164a456b80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x5641db680360 con 0x5641db7b8f30 2023-10-16T12:27:44.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.006+0000 7f164a456b80 10 client.0 _ll_register_callbacks cb 0x5641db4dd7b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T12:27:44.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.006+0000 7f16377fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5641db791b40 0x5641db7b5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:44.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.006+0000 7f16377fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5641db791b40 0x5641db7b5b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42434/0 (socket says 172.21.15.16:42434) 2023-10-16T12:27:44.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.006+0000 7f16377fe700 1 -- 172.21.15.16:0/2235524772 learned_addr learned my addr 172.21.15.16:0/2235524772 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:44.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.006+0000 7f1637fff700 1 --2- 172.21.15.16:0/2235524772 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5641db78cf00 0x5641db7b3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:44.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.006+0000 7f163c809700 1 --2- 172.21.15.16:0/2235524772 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x5641db7b8f30 0x5641db7b80d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:44.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.006+0000 7f16377fe700 1 -- 172.21.15.16:0/2235524772 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x5641db7b8f30 msgr2=0x5641db7b80d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:44.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.006+0000 7f16377fe700 1 --2- 172.21.15.16:0/2235524772 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x5641db7b8f30 0x5641db7b80d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:44.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.006+0000 7f16377fe700 1 -- 172.21.15.16:0/2235524772 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5641db78cf00 msgr2=0x5641db7b3c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:44.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.006+0000 7f16377fe700 1 --2- 172.21.15.16:0/2235524772 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5641db78cf00 0x5641db7b3c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:44.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.006+0000 7f16377fe700 1 -- 172.21.15.16:0/2235524772 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1630007330 con 0x5641db791b40 2023-10-16T12:27:44.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.006+0000 7f16377fe700 1 --2- 172.21.15.16:0/2235524772 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5641db791b40 0x5641db7b5b60 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f16200592a0 tx=0x7f1620059b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bceb1ec46cb84146 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:44.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.006+0000 7f1616ffd700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2023-10-16T12:27:44.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.006+0000 7f1616ffd700 1 -- 172.21.15.16:0/2235524772 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1620061610 con 0x5641db791b40 2023-10-16T12:27:44.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.007+0000 7f1616ffd700 1 -- 172.21.15.16:0/2235524772 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(112..112 src has 1..112) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f162005d020 con 0x5641db791b40 2023-10-16T12:27:44.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.007+0000 7f164a456b80 1 -- 172.21.15.16:0/2235524772 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5641db7845b0 con 0x5641db791b40 2023-10-16T12:27:44.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.007+0000 7f1616ffd700 1 -- 172.21.15.16:0/2235524772 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) v2 ==== 24+0+0 (secure 0 0 0) 0x7f162005d1e0 con 0x5641db791b40 2023-10-16T12:27:44.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.007+0000 7f164a456b80 10 client.7734 fetch_fsmap learned FSMap version 53 2023-10-16T12:27:44.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.007+0000 7f164a456b80 1 -- 172.21.15.16:0/2235524772 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=2+}) v3 -- 0x5641db50ed90 con 0x5641db791b40 2023-10-16T12:27:44.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.007+0000 7f1616ffd700 1 -- 172.21.15.16:0/2235524772 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5641db7845b0 con 0x5641db791b40 2023-10-16T12:27:44.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.007+0000 7f1616ffd700 1 -- 172.21.15.16:0/2235524772 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 53) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f1620059570 con 0x5641db791b40 2023-10-16T12:27:44.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.007+0000 7f164a456b80 10 client.7734 fetch_fsmap finished waiting for FSMap version 53 2023-10-16T12:27:44.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.007+0000 7f164a456b80 10 client.7734 resolve_mds: resolved name 'b' to daemon mds.b 2023-10-16T12:27:44.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.007+0000 7f164a456b80 20 client.7734 populate_metadata read hostname 'smithi016' 2023-10-16T12:27:44.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.007+0000 7f164a456b80 1 --2- 172.21.15.16:0/2235524772 >> [v2:172.21.15.49:6833/729289774,v1:172.21.15.49:6835/729289774] conn(0x5641db7bda50 0x5641db7bde30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:44.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.007+0000 7f164a456b80 4 client.7734 mds_command: new command op to 7258 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T12:27:44.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.007+0000 7f164a456b80 1 -- 172.21.15.16:0/2235524772 --> [v2:172.21.15.49:6833/729289774,v1:172.21.15.49:6835/729289774] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5641db6dce00 con 0x5641db7bda50 2023-10-16T12:27:44.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.007+0000 7f1637fff700 1 --2- 172.21.15.16:0/2235524772 >> [v2:172.21.15.49:6833/729289774,v1:172.21.15.49:6835/729289774] conn(0x5641db7bda50 0x5641db7bde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:44.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.008+0000 7f1616ffd700 10 client.7734 ms_handle_connect on v2:172.21.15.49:6833/729289774 2023-10-16T12:27:44.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.008+0000 7f1637fff700 1 --2- 172.21.15.16:0/2235524772 >> [v2:172.21.15.49:6833/729289774,v1:172.21.15.49:6835/729289774] conn(0x5641db7bda50 0x5641db7bde30 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1e54adf699800f2a server_cookie=d7bb5a2bbb50027f in_seq=0 out_seq=0 2023-10-16T12:27:44.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.008+0000 7f1616ffd700 1 -- 172.21.15.16:0/2235524772 <== mds.0 v2:172.21.15.49:6833/729289774 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5641db6dce00 con 0x5641db7bda50 2023-10-16T12:27:44.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.008+0000 7f1616ffd700 10 client.7734 handle_command_reply: tid=0 2023-10-16T12:27:44.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.008+0000 7f164a456b80 1 client.7734 shutdown 2023-10-16T12:27:44.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.009+0000 7f164a456b80 1 -- 172.21.15.16:0/2235524772 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5641db791b40 msgr2=0x5641db7b5b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:44.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.009+0000 7f164a456b80 1 --2- 172.21.15.16:0/2235524772 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5641db791b40 0x5641db7b5b60 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f16200592a0 tx=0x7f1620059b00 comp rx=0 tx=0).stop 2023-10-16T12:27:44.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.009+0000 7f164a456b80 1 -- 172.21.15.16:0/2235524772 shutdown_connections 2023-10-16T12:27:44.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.009+0000 7f164a456b80 1 --2- 172.21.15.16:0/2235524772 >> [v2:172.21.15.49:6833/729289774,v1:172.21.15.49:6835/729289774] conn(0x5641db7bda50 0x5641db7bde30 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:44.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.009+0000 7f164a456b80 1 --2- 172.21.15.16:0/2235524772 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x5641db7b8f30 0x5641db7b80d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:44.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.009+0000 7f164a456b80 1 --2- 172.21.15.16:0/2235524772 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5641db78cf00 0x5641db7b3c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:44.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.009+0000 7f1616ffd700 0 client.7734 ms_handle_reset on v2:172.21.15.49:6833/729289774 2023-10-16T12:27:44.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.009+0000 7f164a456b80 1 --2- 172.21.15.16:0/2235524772 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5641db791b40 0x5641db7b5b60 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:44.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.009+0000 7f164a456b80 1 -- 172.21.15.16:0/2235524772 >> 172.21.15.16:0/2235524772 conn(0x5641db6bed80 msgr2=0x5641db7911f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:44.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.009+0000 7f164a456b80 1 -- 172.21.15.16:0/2235524772 shutdown_connections 2023-10-16T12:27:44.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.009+0000 7f164a456b80 1 -- 172.21.15.16:0/2235524772 wait complete. 2023-10-16T12:27:44.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.009+0000 7f164a456b80 20 client.7734 trim_cache size 0 max 16384 2023-10-16T12:27:44.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.026+0000 7f164a456b80 1 Processor -- start 2023-10-16T12:27:44.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.027+0000 7f164a456b80 1 -- start start 2023-10-16T12:27:44.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.027+0000 7f164a456b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5641db78cf00 0x5641db7869d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:44.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.027+0000 7f164a456b80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5641db791b40 0x5641db78c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:44.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.027+0000 7f164a456b80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x5641db7b8f30 0x5641db7da8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:44.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.027+0000 7f164a456b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641db5648d0 con 0x5641db78cf00 2023-10-16T12:27:44.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.027+0000 7f164a456b80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641db7b60f0 con 0x5641db791b40 2023-10-16T12:27:44.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.027+0000 7f164a456b80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x5641db78d850 con 0x5641db7b8f30 2023-10-16T12:27:44.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.028+0000 7f1637fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5641db78cf00 0x5641db7869d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:44.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.028+0000 7f1637fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5641db78cf00 0x5641db7869d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42440/0 (socket says 172.21.15.16:42440) 2023-10-16T12:27:44.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.028+0000 7f1637fff700 1 -- 172.21.15.16:0/2910317584 learned_addr learned my addr 172.21.15.16:0/2910317584 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:44.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.028+0000 7f163c809700 1 --2- 172.21.15.16:0/2910317584 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x5641db7b8f30 0x5641db7da8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:44.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.028+0000 7f16377fe700 1 --2- 172.21.15.16:0/2910317584 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5641db791b40 0x5641db78c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:44.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.028+0000 7f1637fff700 1 -- 172.21.15.16:0/2910317584 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x5641db7b8f30 msgr2=0x5641db7da8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:44.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.028+0000 7f1637fff700 1 --2- 172.21.15.16:0/2910317584 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x5641db7b8f30 0x5641db7da8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:44.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.028+0000 7f1637fff700 1 -- 172.21.15.16:0/2910317584 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5641db791b40 msgr2=0x5641db78c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:44.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.028+0000 7f1637fff700 1 --2- 172.21.15.16:0/2910317584 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5641db791b40 0x5641db78c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:44.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.028+0000 7f1637fff700 1 -- 172.21.15.16:0/2910317584 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5641db4f7c10 con 0x5641db78cf00 2023-10-16T12:27:44.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.028+0000 7f16377fe700 1 --2- 172.21.15.16:0/2910317584 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5641db791b40 0x5641db78c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:27:44.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.028+0000 7f1637fff700 1 --2- 172.21.15.16:0/2910317584 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5641db78cf00 0x5641db7869d0 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f1628009860 tx=0x7f1628014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3abdd2381d726d63 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:44.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.028+0000 7f162e7fc700 1 -- 172.21.15.16:0/2910317584 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1628027070 con 0x5641db78cf00 2023-10-16T12:27:44.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.028+0000 7f162e7fc700 1 -- 172.21.15.16:0/2910317584 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f162800c050 con 0x5641db78cf00 2023-10-16T12:27:44.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.028+0000 7f162e7fc700 1 -- 172.21.15.16:0/2910317584 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f162800e770 con 0x5641db78cf00 2023-10-16T12:27:44.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.029+0000 7f164a456b80 1 -- 172.21.15.16:0/2910317584 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5641db78cf00 msgr2=0x5641db7869d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:44.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.029+0000 7f164a456b80 1 --2- 172.21.15.16:0/2910317584 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5641db78cf00 0x5641db7869d0 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f1628009860 tx=0x7f1628014000 comp rx=0 tx=0).stop 2023-10-16T12:27:44.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.029+0000 7f164a456b80 1 -- 172.21.15.16:0/2910317584 shutdown_connections 2023-10-16T12:27:44.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.029+0000 7f164a456b80 1 --2- 172.21.15.16:0/2910317584 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x5641db7b8f30 0x5641db7da8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:44.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.029+0000 7f164a456b80 1 --2- 172.21.15.16:0/2910317584 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5641db78cf00 0x5641db7869d0 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:44.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.029+0000 7f164a456b80 1 --2- 172.21.15.16:0/2910317584 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5641db791b40 0x5641db78c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:44.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.029+0000 7f164a456b80 1 -- 172.21.15.16:0/2910317584 >> 172.21.15.16:0/2910317584 conn(0x5641db6bed80 msgr2=0x5641db7bde30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:44.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.029+0000 7f164a456b80 1 -- 172.21.15.16:0/2910317584 shutdown_connections 2023-10-16T12:27:44.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.029+0000 7f164a456b80 1 -- 172.21.15.16:0/2910317584 wait complete. 2023-10-16T12:27:44.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.029+0000 7f164a456b80 1 -- start start 2023-10-16T12:27:44.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.029+0000 7f164a456b80 1 Processor -- start 2023-10-16T12:27:44.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.030+0000 7f164a456b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5641db78cf00 0x5641db7cfa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:44.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.030+0000 7f164a456b80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5641db791b40 0x5641db7cffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:44.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.030+0000 7f164a456b80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x5641db7b8f30 0x5641db7bac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:44.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.030+0000 7f164a456b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641db7b5bf0 con 0x5641db78cf00 2023-10-16T12:27:44.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.030+0000 7f164a456b80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641db7b6970 con 0x5641db791b40 2023-10-16T12:27:44.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.030+0000 7f164a456b80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x5641db7b6360 con 0x5641db7b8f30 2023-10-16T12:27:44.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.030+0000 7f164a456b80 10 client.0 _ll_register_callbacks cb 0x5641db4dd7b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T12:27:44.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.030+0000 7f1637fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5641db78cf00 0x5641db7cfa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:44.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.030+0000 7f1637fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5641db78cf00 0x5641db7cfa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42446/0 (socket says 172.21.15.16:42446) 2023-10-16T12:27:44.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.030+0000 7f1637fff700 1 -- 172.21.15.16:0/3084929270 learned_addr learned my addr 172.21.15.16:0/3084929270 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:44.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.030+0000 7f16377fe700 1 --2- 172.21.15.16:0/3084929270 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5641db791b40 0x5641db7cffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:44.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.030+0000 7f163c809700 1 --2- 172.21.15.16:0/3084929270 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x5641db7b8f30 0x5641db7bac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:44.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.031+0000 7f1637fff700 1 -- 172.21.15.16:0/3084929270 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x5641db7b8f30 msgr2=0x5641db7bac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:44.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.031+0000 7f1637fff700 1 --2- 172.21.15.16:0/3084929270 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x5641db7b8f30 0x5641db7bac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:44.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.031+0000 7f1637fff700 1 -- 172.21.15.16:0/3084929270 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5641db791b40 msgr2=0x5641db7cffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:44.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.031+0000 7f1637fff700 1 --2- 172.21.15.16:0/3084929270 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5641db791b40 0x5641db7cffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:44.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.031+0000 7f1637fff700 1 -- 172.21.15.16:0/3084929270 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f162800eb80 con 0x5641db78cf00 2023-10-16T12:27:44.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.031+0000 7f1637fff700 1 --2- 172.21.15.16:0/3084929270 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5641db78cf00 0x5641db7cfa60 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f162800e4b0 tx=0x7f16280047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd86df2eed34f3e5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:44.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.031+0000 7f1616ffd700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2023-10-16T12:27:44.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.031+0000 7f1616ffd700 1 -- 172.21.15.16:0/3084929270 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1628027040 con 0x5641db78cf00 2023-10-16T12:27:44.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.031+0000 7f1616ffd700 1 -- 172.21.15.16:0/3084929270 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(112..112 src has 1..112) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f1628013d20 con 0x5641db78cf00 2023-10-16T12:27:44.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.031+0000 7f164a456b80 1 -- 172.21.15.16:0/3084929270 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1630011040 con 0x5641db78cf00 2023-10-16T12:27:44.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.031+0000 7f1616ffd700 1 -- 172.21.15.16:0/3084929270 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1628013ee0 con 0x5641db78cf00 2023-10-16T12:27:44.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.031+0000 7f164a456b80 10 client.7740 fetch_fsmap learned FSMap version 53 2023-10-16T12:27:44.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.031+0000 7f164a456b80 1 -- 172.21.15.16:0/3084929270 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=2+}) v3 -- 0x5641db535200 con 0x5641db78cf00 2023-10-16T12:27:44.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.031+0000 7f1616ffd700 1 -- 172.21.15.16:0/3084929270 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1630011040 con 0x5641db78cf00 2023-10-16T12:27:44.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.031+0000 7f1616ffd700 1 -- 172.21.15.16:0/3084929270 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 53) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f1628038060 con 0x5641db78cf00 2023-10-16T12:27:44.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.031+0000 7f164a456b80 10 client.7740 fetch_fsmap finished waiting for FSMap version 53 2023-10-16T12:27:44.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.031+0000 7f164a456b80 10 client.7740 resolve_mds: resolved name 'b' to daemon mds.b 2023-10-16T12:27:44.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.031+0000 7f164a456b80 20 client.7740 populate_metadata read hostname 'smithi016' 2023-10-16T12:27:44.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.031+0000 7f164a456b80 1 --2- 172.21.15.16:0/3084929270 >> [v2:172.21.15.49:6833/729289774,v1:172.21.15.49:6835/729289774] conn(0x5641db7bb1e0 0x5641db7bb5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:44.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.031+0000 7f164a456b80 4 client.7740 mds_command: new command op to 7258 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T12:27:44.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.031+0000 7f164a456b80 1 -- 172.21.15.16:0/3084929270 --> [v2:172.21.15.49:6833/729289774,v1:172.21.15.49:6835/729289774] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5641db6dcb20 con 0x5641db7bb1e0 2023-10-16T12:27:44.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.032+0000 7f16377fe700 1 --2- 172.21.15.16:0/3084929270 >> [v2:172.21.15.49:6833/729289774,v1:172.21.15.49:6835/729289774] conn(0x5641db7bb1e0 0x5641db7bb5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:44.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.032+0000 7f1616ffd700 10 client.7740 ms_handle_connect on v2:172.21.15.49:6833/729289774 2023-10-16T12:27:44.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.032+0000 7f16377fe700 1 --2- 172.21.15.16:0/3084929270 >> [v2:172.21.15.49:6833/729289774,v1:172.21.15.49:6835/729289774] conn(0x5641db7bb1e0 0x5641db7bb5c0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d9f81b7c77cc178f server_cookie=ba717455f3566f4b in_seq=0 out_seq=0 2023-10-16T12:27:44.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.032+0000 7f1616ffd700 1 -- 172.21.15.16:0/3084929270 <== mds.0 v2:172.21.15.49:6833/729289774 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x5641db6dcb20 con 0x5641db7bb1e0 2023-10-16T12:27:44.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.032+0000 7f1616ffd700 10 client.7740 handle_command_reply: tid=0 2023-10-16T12:27:44.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.032+0000 7f164a456b80 1 client.7740 shutdown 2023-10-16T12:27:44.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.032+0000 7f164a456b80 1 -- 172.21.15.16:0/3084929270 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5641db78cf00 msgr2=0x5641db7cfa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:44.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.032+0000 7f164a456b80 1 --2- 172.21.15.16:0/3084929270 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5641db78cf00 0x5641db7cfa60 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f162800e4b0 tx=0x7f16280047c0 comp rx=0 tx=0).stop 2023-10-16T12:27:44.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.032+0000 7f164a456b80 1 -- 172.21.15.16:0/3084929270 shutdown_connections 2023-10-16T12:27:44.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.032+0000 7f164a456b80 1 --2- 172.21.15.16:0/3084929270 >> [v2:172.21.15.49:6833/729289774,v1:172.21.15.49:6835/729289774] conn(0x5641db7bb1e0 0x5641db7bb5c0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:44.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.032+0000 7f164a456b80 1 --2- 172.21.15.16:0/3084929270 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x5641db7b8f30 0x5641db7bac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:44.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.032+0000 7f164a456b80 1 --2- 172.21.15.16:0/3084929270 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5641db78cf00 0x5641db7cfa60 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:44.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.032+0000 7f164a456b80 1 --2- 172.21.15.16:0/3084929270 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5641db791b40 0x5641db7cffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:44.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.032+0000 7f1616ffd700 0 client.7740 ms_handle_reset on v2:172.21.15.49:6833/729289774 2023-10-16T12:27:44.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.032+0000 7f164a456b80 1 -- 172.21.15.16:0/3084929270 >> 172.21.15.16:0/3084929270 conn(0x5641db6bed80 msgr2=0x5641db7b76f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:44.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.033+0000 7f164a456b80 1 -- 172.21.15.16:0/3084929270 shutdown_connections 2023-10-16T12:27:44.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.033+0000 7f164a456b80 1 -- 172.21.15.16:0/3084929270 wait complete. 2023-10-16T12:27:44.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.033+0000 7f164a456b80 20 client.7740 trim_cache size 0 max 16384 2023-10-16T12:27:44.060 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-10-16T12:27:44.060 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "5b28dfa1-b21f-474a-9b2e-9fb77af58254", 2023-10-16T12:27:44.061 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 0, 2023-10-16T12:27:44.061 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 7258, 2023-10-16T12:27:44.061 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2023-10-16T12:27:44.061 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2023-10-16T12:27:44.061 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2023-10-16T12:27:44.062 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 6.0311211870000001, 2023-10-16T12:27:44.062 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 53, 2023-10-16T12:27:44.062 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 112, 2023-10-16T12:27:44.062 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 112, 2023-10-16T12:27:44.062 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 10.907959452 2023-10-16T12:27:44.063 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-10-16T12:27:44.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.033+0000 7f163e26c700 1 -- 172.21.15.16:0/1000235511 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f161806b020 msgr2=0x7f161806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:44.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.033+0000 7f163e26c700 1 --2- 172.21.15.16:0/1000235511 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f161806b020 0x7f161806d4e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f16380c6fe0 tx=0x7f1628008040 comp rx=0 tx=0).stop 2023-10-16T12:27:44.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.033+0000 7f163e26c700 1 -- 172.21.15.16:0/1000235511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16380cf4a0 msgr2=0x7f163815c220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:44.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.033+0000 7f163e26c700 1 --2- 172.21.15.16:0/1000235511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16380cf4a0 0x7f163815c220 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f1620009c10 tx=0x7f1620004750 comp rx=0 tx=0).stop 2023-10-16T12:27:44.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.033+0000 7f163e26c700 1 -- 172.21.15.16:0/1000235511 shutdown_connections 2023-10-16T12:27:44.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.033+0000 7f163e26c700 1 --2- 172.21.15.16:0/1000235511 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f16380cf880 0x7f163815c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:44.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.033+0000 7f163e26c700 1 --2- 172.21.15.16:0/1000235511 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f161806b020 0x7f161806d4e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:44.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.033+0000 7f163e26c700 1 --2- 172.21.15.16:0/1000235511 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f16380b5830 0x7f16380aec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:44.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.033+0000 7f163e26c700 1 --2- 172.21.15.16:0/1000235511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16380cf4a0 0x7f163815c220 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:44.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.033+0000 7f163e26c700 1 -- 172.21.15.16:0/1000235511 >> 172.21.15.16:0/1000235511 conn(0x7f16380b1fd0 msgr2=0x7f16380b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:44.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.033+0000 7f163e26c700 1 -- 172.21.15.16:0/1000235511 shutdown_connections 2023-10-16T12:27:44.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.033+0000 7f163e26c700 1 -- 172.21.15.16:0/1000235511 wait complete. 2023-10-16T12:27:44.067 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T12:27:44.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.191+0000 7f37898f6700 1 Processor -- start 2023-10-16T12:27:44.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.191+0000 7f37898f6700 1 -- start start 2023-10-16T12:27:44.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.192+0000 7f37898f6700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f37840d41c0 0x7f37840d0040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:44.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.192+0000 7f37898f6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f37840d0580 0x7f378406c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:44.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.192+0000 7f37898f6700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f378406ba00 con 0x7f378415d8d0 2023-10-16T12:27:44.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.192+0000 7f37898f6700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37840d0a10 con 0x7f37840d0580 2023-10-16T12:27:44.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.192+0000 7f37898f6700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37840d26d0 con 0x7f37840d41c0 2023-10-16T12:27:44.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.192+0000 7f37837fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f37840d0580 0x7f378406c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:44.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.192+0000 7f37837fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f37840d0580 0x7f378406c4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42458/0 (socket says 172.21.15.16:42458) 2023-10-16T12:27:44.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.192+0000 7f37837fe700 1 -- 172.21.15.16:0/578625850 learned_addr learned my addr 172.21.15.16:0/578625850 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:44.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.192+0000 7f37827fc700 1 --2- 172.21.15.16:0/578625850 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f37840d41c0 0x7f37840d0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:44.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.192+0000 7f37837fe700 1 -- 172.21.15.16:0/578625850 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f37840d41c0 msgr2=0x7f37840d0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:44.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.192+0000 7f37837fe700 1 --2- 172.21.15.16:0/578625850 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f37840d41c0 0x7f37840d0040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:44.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.192+0000 7f37837fe700 1 -- 172.21.15.16:0/578625850 >> v1:172.21.15.49:6789/0 conn(0x7f378415d8d0 legacy=0x7f37840d1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:44.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.192+0000 7f37837fe700 1 -- 172.21.15.16:0/578625850 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37840d23f0 con 0x7f37840d0580 2023-10-16T12:27:44.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.192+0000 7f37837fe700 1 --2- 172.21.15.16:0/578625850 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37840d0580 0x7f378406c4a0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f377c009e80 tx=0x7f377c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26c73876277b5f2f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:44.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.193+0000 7f3781ffb700 1 -- 172.21.15.16:0/578625850 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f377c00e070 con 0x7f37840d0580 2023-10-16T12:27:44.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.193+0000 7f3781ffb700 1 -- 172.21.15.16:0/578625850 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f377c00b980 con 0x7f37840d0580 2023-10-16T12:27:44.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.193+0000 7f3781ffb700 1 -- 172.21.15.16:0/578625850 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f377c00bd30 con 0x7f37840d0580 2023-10-16T12:27:44.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.193+0000 7f37898f6700 1 -- 172.21.15.16:0/578625850 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37840d0580 msgr2=0x7f378406c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:44.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.193+0000 7f37898f6700 1 --2- 172.21.15.16:0/578625850 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37840d0580 0x7f378406c4a0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f377c009e80 tx=0x7f377c00b330 comp rx=0 tx=0).stop 2023-10-16T12:27:44.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.193+0000 7f37898f6700 1 -- 172.21.15.16:0/578625850 shutdown_connections 2023-10-16T12:27:44.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.193+0000 7f37898f6700 1 --2- 172.21.15.16:0/578625850 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37840d0580 0x7f378406c4a0 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:44.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.193+0000 7f37898f6700 1 --2- 172.21.15.16:0/578625850 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f37840d41c0 0x7f37840d0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:44.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.193+0000 7f37898f6700 1 -- 172.21.15.16:0/578625850 >> 172.21.15.16:0/578625850 conn(0x7f37840d3630 msgr2=0x7f37840d3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:44.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.193+0000 7f37898f6700 1 -- 172.21.15.16:0/578625850 shutdown_connections 2023-10-16T12:27:44.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.193+0000 7f37898f6700 1 -- 172.21.15.16:0/578625850 wait complete. 2023-10-16T12:27:44.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.194+0000 7f37898f6700 1 Processor -- start 2023-10-16T12:27:44.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.195+0000 7f37898f6700 1 -- start start 2023-10-16T12:27:44.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.195+0000 7f37898f6700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f37840d0580 0x7f37841f2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:44.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.195+0000 7f37898f6700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f37840d41c0 0x7f378415aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:44.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.195+0000 7f37898f6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f378415d8d0 0x7f378415afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:44.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.195+0000 7f37898f6700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37840c95b0 con 0x7f378415d8d0 2023-10-16T12:27:44.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.195+0000 7f37898f6700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37841f3210 con 0x7f37840d0580 2023-10-16T12:27:44.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.195+0000 7f37898f6700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f378409d4b0 con 0x7f37840d41c0 2023-10-16T12:27:44.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.196+0000 7f37837fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f378415d8d0 0x7f378415afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:44.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.196+0000 7f37837fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f378415d8d0 0x7f378415afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42462/0 (socket says 172.21.15.16:42462) 2023-10-16T12:27:44.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.196+0000 7f37837fe700 1 -- 172.21.15.16:0/4288000393 learned_addr learned my addr 172.21.15.16:0/4288000393 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:44.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.196+0000 7f37837fe700 1 -- 172.21.15.16:0/4288000393 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f37840d41c0 msgr2=0x7f378415aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:44.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.196+0000 7f37827fc700 1 --2- 172.21.15.16:0/4288000393 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f37840d41c0 0x7f378415aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:44.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.196+0000 7f37837fe700 1 --2- 172.21.15.16:0/4288000393 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f37840d41c0 0x7f378415aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:44.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.196+0000 7f37837fe700 1 -- 172.21.15.16:0/4288000393 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f37840d0580 msgr2=0x7f37841f2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:44.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.196+0000 7f37837fe700 1 --2- 172.21.15.16:0/4288000393 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f37840d0580 0x7f37841f2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:44.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.196+0000 7f37837fe700 1 -- 172.21.15.16:0/4288000393 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f377c009b30 con 0x7f378415d8d0 2023-10-16T12:27:44.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.196+0000 7f37837fe700 1 --2- 172.21.15.16:0/4288000393 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f378415d8d0 0x7f378415afd0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f377c00b730 tx=0x7f377c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:44.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.196+0000 7f376bfff700 1 -- 172.21.15.16:0/4288000393 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f377c0099b0 con 0x7f378415d8d0 2023-10-16T12:27:44.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.196+0000 7f376bfff700 1 -- 172.21.15.16:0/4288000393 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f377c0092e0 con 0x7f378415d8d0 2023-10-16T12:27:44.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.196+0000 7f37898f6700 1 -- 172.21.15.16:0/4288000393 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37840c8ff0 con 0x7f378415d8d0 2023-10-16T12:27:44.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.196+0000 7f376bfff700 1 -- 172.21.15.16:0/4288000393 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f377c01c640 con 0x7f378415d8d0 2023-10-16T12:27:44.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.196+0000 7f37898f6700 1 -- 172.21.15.16:0/4288000393 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f378409d750 con 0x7f378415d8d0 2023-10-16T12:27:44.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.197+0000 7f37898f6700 1 -- 172.21.15.16:0/4288000393 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f378409d310 con 0x7f378415d8d0 2023-10-16T12:27:44.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.198+0000 7f376bfff700 1 -- 172.21.15.16:0/4288000393 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 51) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f377c01c7a0 con 0x7f378415d8d0 2023-10-16T12:27:44.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.198+0000 7f376bfff700 1 --2- 172.21.15.16:0/4288000393 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f377006b020 0x7f377006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:44.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.198+0000 7f376bfff700 1 -- 172.21.15.16:0/4288000393 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f377c08b260 con 0x7f378415d8d0 2023-10-16T12:27:44.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.198+0000 7f3782ffd700 1 --2- 172.21.15.16:0/4288000393 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f377006b020 0x7f377006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:44.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.199+0000 7f3782ffd700 1 --2- 172.21.15.16:0/4288000393 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f377006b020 0x7f377006d4e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3778000f60 tx=0x7f3778007a40 comp rx=0 tx=0).ready entity=mgr.6365 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:44.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.203+0000 7f376bfff700 1 -- 172.21.15.16:0/4288000393 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f378409d310 con 0x7f378415d8d0 2023-10-16T12:27:44.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.455+0000 7f37898f6700 1 -- 172.21.15.16:0/4288000393 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f378409d310 con 0x7f378415d8d0 2023-10-16T12:27:44.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.455+0000 7f376bfff700 1 -- 172.21.15.16:0/4288000393 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v53) v1 ==== 53+0+73 (secure 0 0 0) 0x7f378409d310 con 0x7f378415d8d0 2023-10-16T12:27:44.457 INFO:teuthology.orchestra.run.smithi016.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-16T12:27:44.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.457+0000 7f37898f6700 1 -- 172.21.15.16:0/4288000393 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f377006b020 msgr2=0x7f377006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:44.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.457+0000 7f37898f6700 1 --2- 172.21.15.16:0/4288000393 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f377006b020 0x7f377006d4e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3778000f60 tx=0x7f3778007a40 comp rx=0 tx=0).stop 2023-10-16T12:27:44.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.457+0000 7f37898f6700 1 -- 172.21.15.16:0/4288000393 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f378415d8d0 msgr2=0x7f378415afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:44.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.457+0000 7f37898f6700 1 --2- 172.21.15.16:0/4288000393 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f378415d8d0 0x7f378415afd0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f377c00b730 tx=0x7f377c004ad0 comp rx=0 tx=0).stop 2023-10-16T12:27:44.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.457+0000 7f37898f6700 1 -- 172.21.15.16:0/4288000393 shutdown_connections 2023-10-16T12:27:44.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.457+0000 7f37898f6700 1 --2- 172.21.15.16:0/4288000393 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f378415d8d0 0x7f378415afd0 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:44.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.457+0000 7f37898f6700 1 --2- 172.21.15.16:0/4288000393 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f377006b020 0x7f377006d4e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:44.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.457+0000 7f37898f6700 1 --2- 172.21.15.16:0/4288000393 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f37840d0580 0x7f37841f2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:44.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.457+0000 7f37898f6700 1 --2- 172.21.15.16:0/4288000393 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f37840d41c0 0x7f378415aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:44.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.457+0000 7f37898f6700 1 -- 172.21.15.16:0/4288000393 >> 172.21.15.16:0/4288000393 conn(0x7f37840d3630 msgr2=0x7f378415cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:44.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.457+0000 7f37898f6700 1 -- 172.21.15.16:0/4288000393 shutdown_connections 2023-10-16T12:27:44.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:44.457+0000 7f37898f6700 1 -- 172.21.15.16:0/4288000393 wait complete. 2023-10-16T12:27:44.471 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-16T12:27:44.472 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T12:27:44.472 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi016.front.sepia.ceph.com 2023-10-16T12:27:44.473 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-16T12:27:44.473 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-16T12:27:44.473 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-16T12:27:44.474 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-16T12:27:44.474 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T12:27:44.474 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T12:27:44.475 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:27:44.475 DEBUG:teuthology.orchestra.run.smithi016:> ip addr 2023-10-16T12:27:44.493 INFO:teuthology.orchestra.run.smithi016.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T12:27:44.493 INFO:teuthology.orchestra.run.smithi016.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T12:27:44.494 INFO:teuthology.orchestra.run.smithi016.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T12:27:44.494 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:27:44.494 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 ::1/128 scope host 2023-10-16T12:27:44.494 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:27:44.494 INFO:teuthology.orchestra.run.smithi016.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T12:27:44.495 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e0 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:27:44.495 INFO:teuthology.orchestra.run.smithi016.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T12:27:44.495 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e1 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:27:44.495 INFO:teuthology.orchestra.run.smithi016.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T12:27:44.495 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:64 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:27:44.496 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.21.15.16/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-16T12:27:44.496 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft 41272sec preferred_lft 41272sec 2023-10-16T12:27:44.496 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::ec4:7aff:febd:1564/64 scope link 2023-10-16T12:27:44.496 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:27:44.496 INFO:teuthology.orchestra.run.smithi016.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T12:27:44.497 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:65 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:27:44.497 INFO:teuthology.orchestra.run.smithi016.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T12:27:44.497 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 36:cf:ea:25:db:f7 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:27:44.497 INFO:teuthology.orchestra.run.smithi016.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-16T12:27:44.497 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:27:44.497 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::cfc:c5ff:fe52:a786/64 scope link 2023-10-16T12:27:44.498 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:27:44.498 INFO:teuthology.orchestra.run.smithi016.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T12:27:44.498 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 6e:62:9d:a4:43:23 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T12:27:44.498 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::6c62:9dff:fea4:4323/64 scope link 2023-10-16T12:27:44.498 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:27:44.499 INFO:teuthology.orchestra.run.smithi016.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T12:27:44.499 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 36:cf:ea:25:db:f7 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-16T12:27:44.499 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::34cf:eaff:fe25:dbf7/64 scope link 2023-10-16T12:27:44.499 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:27:44.499 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:27:44.500 DEBUG:teuthology.orchestra.run.smithi016:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T12:27:44.559 INFO:teuthology.orchestra.run.smithi016.stdout:1 2023-10-16T12:27:44.561 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:27:44.561 DEBUG:teuthology.orchestra.run.smithi016:> route 2023-10-16T12:27:44.619 INFO:teuthology.orchestra.run.smithi016.stdout:Kernel IP routing table 2023-10-16T12:27:44.619 INFO:teuthology.orchestra.run.smithi016.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T12:27:44.619 INFO:teuthology.orchestra.run.smithi016.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-16T12:27:44.619 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-16T12:27:44.620 INFO:teuthology.orchestra.run.smithi016.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T12:27:44.621 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:27:44.621 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec bash -c ' 2023-10-16T12:27:44.621 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2023-10-16T12:27:44.622 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-16T12:27:44.622 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-16T12:27:44.623 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-16T12:27:44.623 DEBUG:teuthology.orchestra.run.smithi016:> ') 2023-10-16T12:27:44.727 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:27:44.727 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2023-10-16T12:27:44.744 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-16T12:27:44.744 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-16T12:27:44.745 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T12:27:44.745 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:27:44.746 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set brx.0 up 2023-10-16T12:27:44.812 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T12:27:44.813 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-16T12:27:44.870 INFO:teuthology.orchestra.run.smithi016.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-16T12:27:44.870 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T12:27:44.871 DEBUG:teuthology.orchestra.run.smithi016:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-16T12:27:44.926 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-16T12:27:44.927 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:27:44.927 DEBUG:teuthology.orchestra.run.smithi016:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T12:27:45.014 INFO:teuthology.orchestra.run.smithi016.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T12:27:45.014 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T12:27:45.015 INFO:teuthology.orchestra.run.smithi016.stdout:invalid new device string format 2023-10-16T12:27:45.023 INFO:teuthology.orchestra.run.smithi016.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T12:27:45.023 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T12:27:45.023 INFO:teuthology.orchestra.run.smithi016.stdout:invalid new device string format 2023-10-16T12:27:45.023 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: resolved to: "172.21.15.16:3300,172.21.15.49:3300,172.21.15.49:3301" 2023-10-16T12:27:45.024 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T12:27:45.096 INFO:teuthology.orchestra.run.smithi016.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T12:27:45.096 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T12:27:45.096 INFO:teuthology.orchestra.run.smithi016.stdout:invalid new device string format 2023-10-16T12:27:45.097 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: resolved to: "172.21.15.16:3300,172.21.15.49:3300,172.21.15.49:3301" 2023-10-16T12:27:45.097 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T12:27:45.097 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: resolved to: "172.21.15.16:6789,172.21.15.49:6789,172.21.15.49:6790" 2023-10-16T12:27:45.097 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.16:6789,172.21.15.49:6789,172.21.15.49:6790:/ 2023-10-16T12:27:45.097 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=5b28dfa1-b21f-474a-9b2e-9fb77af58254" will pass to kernel 2023-10-16T12:27:45.099 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-16T12:27:45.099 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:27:45.099 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-16T12:27:45.127 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:27:45.128 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T12:27:45.217 INFO:teuthology.orchestra.run.smithi016.stdout:client6629 2023-10-16T12:27:45.217 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:27:45.218 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:27:45.235 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-16T12:27:45.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.399+0000 7fd3dfa6a700 1 Processor -- start 2023-10-16T12:27:45.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.400+0000 7fd3dfa6a700 1 -- start start 2023-10-16T12:27:45.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.400+0000 7fd3dfa6a700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fd3d80d2f70 0x7fd3d80d2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:45.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.400+0000 7fd3dfa6a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd3d80ce9f0 0x7fd3d80cdc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:45.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.400+0000 7fd3dfa6a700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3d80ce1a0 con 0x7fd3d815d900 2023-10-16T12:27:45.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.400+0000 7fd3dfa6a700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3d80ce400 con 0x7fd3d80ce9f0 2023-10-16T12:27:45.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.400+0000 7fd3dfa6a700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3d80cee40 con 0x7fd3d80d2f70 2023-10-16T12:27:45.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.400+0000 7fd3de007700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd3d80ce9f0 0x7fd3d80cdc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:45.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.400+0000 7fd3de007700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd3d80ce9f0 0x7fd3d80cdc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42474/0 (socket says 172.21.15.16:42474) 2023-10-16T12:27:45.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.400+0000 7fd3de007700 1 -- 172.21.15.16:0/2754132335 learned_addr learned my addr 172.21.15.16:0/2754132335 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:45.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.401+0000 7fd3dd005700 1 --2- 172.21.15.16:0/2754132335 >> v2:172.21.15.49:3300/0 conn(0x7fd3d80d2f70 0x7fd3d80d2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:45.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.401+0000 7fd3de007700 1 -- 172.21.15.16:0/2754132335 >> v2:172.21.15.49:3300/0 conn(0x7fd3d80d2f70 msgr2=0x7fd3d80d2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:45.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.401+0000 7fd3de007700 1 --2- 172.21.15.16:0/2754132335 >> v2:172.21.15.49:3300/0 conn(0x7fd3d80d2f70 0x7fd3d80d2040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:45.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.401+0000 7fd3de007700 1 -- 172.21.15.16:0/2754132335 >> v1:172.21.15.49:6789/0 conn(0x7fd3d815d900 legacy=0x7fd3d80d2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:45.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.401+0000 7fd3de007700 1 -- 172.21.15.16:0/2754132335 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3d815dce0 con 0x7fd3d80ce9f0 2023-10-16T12:27:45.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.401+0000 7fd3de007700 1 --2- 172.21.15.16:0/2754132335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3d80ce9f0 0x7fd3d80cdc60 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7fd3d80b9810 tx=0x7fd3d000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eed2db3f6f28a5f7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:45.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.401+0000 7fd3dc804700 1 -- 172.21.15.16:0/2754132335 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3d000e070 con 0x7fd3d80ce9f0 2023-10-16T12:27:45.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.401+0000 7fd3dc804700 1 -- 172.21.15.16:0/2754132335 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3d0003a20 con 0x7fd3d80ce9f0 2023-10-16T12:27:45.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.401+0000 7fd3dc804700 1 -- 172.21.15.16:0/2754132335 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3d0003dd0 con 0x7fd3d80ce9f0 2023-10-16T12:27:45.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.402+0000 7fd3dfa6a700 1 -- 172.21.15.16:0/2754132335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3d80ce9f0 msgr2=0x7fd3d80cdc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:45.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.402+0000 7fd3dfa6a700 1 --2- 172.21.15.16:0/2754132335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3d80ce9f0 0x7fd3d80cdc60 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7fd3d80b9810 tx=0x7fd3d000b330 comp rx=0 tx=0).stop 2023-10-16T12:27:45.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.402+0000 7fd3dfa6a700 1 -- 172.21.15.16:0/2754132335 shutdown_connections 2023-10-16T12:27:45.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.402+0000 7fd3dfa6a700 1 --2- 172.21.15.16:0/2754132335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3d80ce9f0 0x7fd3d80cdc60 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:45.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.402+0000 7fd3dfa6a700 1 --2- 172.21.15.16:0/2754132335 >> v2:172.21.15.49:3300/0 conn(0x7fd3d80d2f70 0x7fd3d80d2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:45.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.402+0000 7fd3dfa6a700 1 -- 172.21.15.16:0/2754132335 >> 172.21.15.16:0/2754132335 conn(0x7fd3d8072330 msgr2=0x7fd3d8072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:45.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.402+0000 7fd3dfa6a700 1 -- 172.21.15.16:0/2754132335 shutdown_connections 2023-10-16T12:27:45.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.402+0000 7fd3dfa6a700 1 -- 172.21.15.16:0/2754132335 wait complete. 2023-10-16T12:27:45.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.403+0000 7fd3dfa6a700 1 Processor -- start 2023-10-16T12:27:45.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.403+0000 7fd3dfa6a700 1 -- start start 2023-10-16T12:27:45.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.404+0000 7fd3dfa6a700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd3d80ce9f0 0x7fd3d81577e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:45.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.404+0000 7fd3dfa6a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd3d80d2f70 0x7fd3d8157d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:45.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.404+0000 7fd3dfa6a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3d815d900 0x7fd3d815f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:45.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.404+0000 7fd3dfa6a700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d80b3e30 con 0x7fd3d815d900 2023-10-16T12:27:45.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.404+0000 7fd3dfa6a700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d80b5f40 con 0x7fd3d80ce9f0 2023-10-16T12:27:45.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.404+0000 7fd3dfa6a700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d80b53c0 con 0x7fd3d80d2f70 2023-10-16T12:27:45.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.404+0000 7fd3de007700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3d815d900 0x7fd3d815f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:45.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.404+0000 7fd3de007700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3d815d900 0x7fd3d815f220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42480/0 (socket says 172.21.15.16:42480) 2023-10-16T12:27:45.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.404+0000 7fd3de007700 1 -- 172.21.15.16:0/2193341780 learned_addr learned my addr 172.21.15.16:0/2193341780 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:45.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.404+0000 7fd3dd806700 1 --2- 172.21.15.16:0/2193341780 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd3d80ce9f0 0x7fd3d81577e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:45.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.404+0000 7fd3dd005700 1 --2- 172.21.15.16:0/2193341780 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd3d80d2f70 0x7fd3d8157d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:45.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.404+0000 7fd3de007700 1 -- 172.21.15.16:0/2193341780 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd3d80d2f70 msgr2=0x7fd3d8157d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:45.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.404+0000 7fd3de007700 1 --2- 172.21.15.16:0/2193341780 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd3d80d2f70 0x7fd3d8157d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:45.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.404+0000 7fd3de007700 1 -- 172.21.15.16:0/2193341780 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd3d80ce9f0 msgr2=0x7fd3d81577e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:45.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.404+0000 7fd3de007700 1 --2- 172.21.15.16:0/2193341780 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd3d80ce9f0 0x7fd3d81577e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:45.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.404+0000 7fd3de007700 1 -- 172.21.15.16:0/2193341780 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3d0009b30 con 0x7fd3d815d900 2023-10-16T12:27:45.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.404+0000 7fd3de007700 1 --2- 172.21.15.16:0/2193341780 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3d815d900 0x7fd3d815f220 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7fd3d0015040 tx=0x7fd3d0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:45.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.404+0000 7fd3ce7fc700 1 -- 172.21.15.16:0/2193341780 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3d0004440 con 0x7fd3d815d900 2023-10-16T12:27:45.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.405+0000 7fd3ce7fc700 1 -- 172.21.15.16:0/2193341780 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3d00092e0 con 0x7fd3d815d900 2023-10-16T12:27:45.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.405+0000 7fd3ce7fc700 1 -- 172.21.15.16:0/2193341780 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3d00047e0 con 0x7fd3d815d900 2023-10-16T12:27:45.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.405+0000 7fd3dfa6a700 1 -- 172.21.15.16:0/2193341780 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3d80b56a0 con 0x7fd3d815d900 2023-10-16T12:27:45.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.405+0000 7fd3dfa6a700 1 -- 172.21.15.16:0/2193341780 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3d80b5c60 con 0x7fd3d815d900 2023-10-16T12:27:45.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.405+0000 7fd3dfa6a700 1 -- 172.21.15.16:0/2193341780 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3d815ef40 con 0x7fd3d815d900 2023-10-16T12:27:45.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.406+0000 7fd3ce7fc700 1 -- 172.21.15.16:0/2193341780 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 51) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd3d0012430 con 0x7fd3d815d900 2023-10-16T12:27:45.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.406+0000 7fd3ce7fc700 1 --2- 172.21.15.16:0/2193341780 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fd3bc06b020 0x7fd3bc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:45.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.406+0000 7fd3ce7fc700 1 -- 172.21.15.16:0/2193341780 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fd3d008ae30 con 0x7fd3d815d900 2023-10-16T12:27:45.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.406+0000 7fd3dd806700 1 --2- 172.21.15.16:0/2193341780 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fd3bc06b020 0x7fd3bc06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:45.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.407+0000 7fd3dd806700 1 --2- 172.21.15.16:0/2193341780 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fd3bc06b020 0x7fd3bc06d4e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd3d4009960 tx=0x7fd3d40090e0 comp rx=0 tx=0).ready entity=mgr.6365 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:45.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.411+0000 7fd3ce7fc700 1 -- 172.21.15.16:0/2193341780 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd3d815ef40 con 0x7fd3d815d900 2023-10-16T12:27:45.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.644+0000 7fd3dfa6a700 1 -- 172.21.15.16:0/2193341780 --> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fd3d80d1560 con 0x7fd3bc06b020 2023-10-16T12:27:45.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.646+0000 7fd3ce7fc700 1 -- 172.21.15.16:0/2193341780 <== mgr.6365 v2:172.21.15.16:6832/4089191642 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fd3d4009e30 con 0x7fd3bc06b020 2023-10-16T12:27:45.648 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2023-10-16T12:27:45.648 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:27:45.648 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cephfs" 2023-10-16T12:27:45.649 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:27:45.649 INFO:teuthology.orchestra.run.smithi016.stdout:] 2023-10-16T12:27:45.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.648+0000 7fd3dfa6a700 1 -- 172.21.15.16:0/2193341780 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fd3bc06b020 msgr2=0x7fd3bc06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:45.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.648+0000 7fd3dfa6a700 1 --2- 172.21.15.16:0/2193341780 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fd3bc06b020 0x7fd3bc06d4e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd3d4009960 tx=0x7fd3d40090e0 comp rx=0 tx=0).stop 2023-10-16T12:27:45.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.648+0000 7fd3dfa6a700 1 -- 172.21.15.16:0/2193341780 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3d815d900 msgr2=0x7fd3d815f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:45.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.648+0000 7fd3dfa6a700 1 --2- 172.21.15.16:0/2193341780 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3d815d900 0x7fd3d815f220 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7fd3d0015040 tx=0x7fd3d0004ad0 comp rx=0 tx=0).stop 2023-10-16T12:27:45.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.648+0000 7fd3dfa6a700 1 -- 172.21.15.16:0/2193341780 shutdown_connections 2023-10-16T12:27:45.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.648+0000 7fd3dfa6a700 1 --2- 172.21.15.16:0/2193341780 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3d815d900 0x7fd3d815f220 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:45.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.648+0000 7fd3dfa6a700 1 --2- 172.21.15.16:0/2193341780 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fd3bc06b020 0x7fd3bc06d4e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:45.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.648+0000 7fd3dfa6a700 1 --2- 172.21.15.16:0/2193341780 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd3d80ce9f0 0x7fd3d81577e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:45.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.648+0000 7fd3dfa6a700 1 --2- 172.21.15.16:0/2193341780 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd3d80d2f70 0x7fd3d8157d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:45.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.648+0000 7fd3dfa6a700 1 -- 172.21.15.16:0/2193341780 >> 172.21.15.16:0/2193341780 conn(0x7fd3d8072330 msgr2=0x7fd3d8158980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:45.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.648+0000 7fd3dfa6a700 1 -- 172.21.15.16:0/2193341780 shutdown_connections 2023-10-16T12:27:45.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.648+0000 7fd3dfa6a700 1 -- 172.21.15.16:0/2193341780 wait complete. 2023-10-16T12:27:45.663 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-16T12:27:45.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.788+0000 7f9c789d7700 1 Processor -- start 2023-10-16T12:27:45.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.788+0000 7f9c789d7700 1 -- start start 2023-10-16T12:27:45.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.789+0000 7f9c789d7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9c740ce6a0 0x7f9c7406c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:45.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.789+0000 7f9c789d7700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9c740d2110 0x7f9c7406c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:45.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.789+0000 7f9c789d7700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c740b20f0 con 0x7f9c740b2620 2023-10-16T12:27:45.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.789+0000 7f9c789d7700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c740af1f0 con 0x7f9c740ce6a0 2023-10-16T12:27:45.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.789+0000 7f9c789d7700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c740d03d0 con 0x7f9c740d2110 2023-10-16T12:27:45.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.789+0000 7f9c7259c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9c740ce6a0 0x7f9c7406c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:45.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.789+0000 7f9c7259c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9c740ce6a0 0x7f9c7406c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42484/0 (socket says 172.21.15.16:42484) 2023-10-16T12:27:45.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.789+0000 7f9c7259c700 1 -- 172.21.15.16:0/1741096647 learned_addr learned my addr 172.21.15.16:0/1741096647 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:45.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.789+0000 7f9c7259c700 1 -- 172.21.15.16:0/1741096647 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9c740d2110 msgr2=0x7f9c7406c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:45.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.789+0000 7f9c71d9b700 1 --2- 172.21.15.16:0/1741096647 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9c740d2110 0x7f9c7406c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:45.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.789+0000 7f9c7259c700 1 --2- 172.21.15.16:0/1741096647 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9c740d2110 0x7f9c7406c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:45.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.789+0000 7f9c7259c700 1 -- 172.21.15.16:0/1741096647 >> v1:172.21.15.49:6789/0 conn(0x7f9c740b2620 legacy=0x7f9c740b0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:45.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.789+0000 7f9c7259c700 1 -- 172.21.15.16:0/1741096647 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c740b2340 con 0x7f9c740ce6a0 2023-10-16T12:27:45.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.789+0000 7f9c71d9b700 1 --2- 172.21.15.16:0/1741096647 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9c740d2110 0x7f9c7406c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:27:45.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.789+0000 7f9c7259c700 1 --2- 172.21.15.16:0/1741096647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c740ce6a0 0x7f9c7406c300 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f9c640098f0 tx=0x7f9c64009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b49fa4ff679736d2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:45.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.789+0000 7f9c7159a700 1 -- 172.21.15.16:0/1741096647 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c64004230 con 0x7f9c740ce6a0 2023-10-16T12:27:45.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.789+0000 7f9c7159a700 1 -- 172.21.15.16:0/1741096647 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c64004390 con 0x7f9c740ce6a0 2023-10-16T12:27:45.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.789+0000 7f9c7159a700 1 -- 172.21.15.16:0/1741096647 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c640036a0 con 0x7f9c740ce6a0 2023-10-16T12:27:45.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.790+0000 7f9c789d7700 1 -- 172.21.15.16:0/1741096647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c740ce6a0 msgr2=0x7f9c7406c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:45.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.790+0000 7f9c789d7700 1 --2- 172.21.15.16:0/1741096647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c740ce6a0 0x7f9c7406c300 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f9c640098f0 tx=0x7f9c64009c00 comp rx=0 tx=0).stop 2023-10-16T12:27:45.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.790+0000 7f9c789d7700 1 -- 172.21.15.16:0/1741096647 shutdown_connections 2023-10-16T12:27:45.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.790+0000 7f9c789d7700 1 --2- 172.21.15.16:0/1741096647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c740ce6a0 0x7f9c7406c300 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:45.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.790+0000 7f9c789d7700 1 --2- 172.21.15.16:0/1741096647 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9c740d2110 0x7f9c7406c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:45.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.790+0000 7f9c789d7700 1 -- 172.21.15.16:0/1741096647 >> 172.21.15.16:0/1741096647 conn(0x7f9c740d15f0 msgr2=0x7f9c740d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:45.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.790+0000 7f9c789d7700 1 -- 172.21.15.16:0/1741096647 shutdown_connections 2023-10-16T12:27:45.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.790+0000 7f9c789d7700 1 -- 172.21.15.16:0/1741096647 wait complete. 2023-10-16T12:27:45.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.791+0000 7f9c789d7700 1 Processor -- start 2023-10-16T12:27:45.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.792+0000 7f9c789d7700 1 -- start start 2023-10-16T12:27:45.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.792+0000 7f9c789d7700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9c740b2620 0x7f9c7415e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:45.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.792+0000 7f9c789d7700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9c740ce6a0 0x7f9c74159490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:45.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.792+0000 7f9c789d7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c740d2110 0x7f9c7415f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:45.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.792+0000 7f9c789d7700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c741599d0 con 0x7f9c740d2110 2023-10-16T12:27:45.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.792+0000 7f9c789d7700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c74159c70 con 0x7f9c740ce6a0 2023-10-16T12:27:45.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.792+0000 7f9c789d7700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c74159f50 con 0x7f9c740b2620 2023-10-16T12:27:45.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.792+0000 7f9c72d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c740d2110 0x7f9c7415f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:45.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.792+0000 7f9c72d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c740d2110 0x7f9c7415f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42494/0 (socket says 172.21.15.16:42494) 2023-10-16T12:27:45.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.792+0000 7f9c72d9d700 1 -- 172.21.15.16:0/897468613 learned_addr learned my addr 172.21.15.16:0/897468613 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:45.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.792+0000 7f9c71d9b700 1 --2- 172.21.15.16:0/897468613 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9c740ce6a0 0x7f9c74159490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:45.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.792+0000 7f9c7259c700 1 --2- 172.21.15.16:0/897468613 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9c740b2620 0x7f9c7415e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:45.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.793+0000 7f9c72d9d700 1 -- 172.21.15.16:0/897468613 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9c740b2620 msgr2=0x7f9c7415e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:45.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.793+0000 7f9c72d9d700 1 --2- 172.21.15.16:0/897468613 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9c740b2620 0x7f9c7415e2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:45.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.793+0000 7f9c72d9d700 1 -- 172.21.15.16:0/897468613 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9c740ce6a0 msgr2=0x7f9c74159490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:45.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.793+0000 7f9c72d9d700 1 --2- 172.21.15.16:0/897468613 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9c740ce6a0 0x7f9c74159490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:45.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.793+0000 7f9c72d9d700 1 -- 172.21.15.16:0/897468613 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c64009550 con 0x7f9c740d2110 2023-10-16T12:27:45.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.793+0000 7f9c72d9d700 1 --2- 172.21.15.16:0/897468613 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c740d2110 0x7f9c7415f060 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f9c6c00cc10 tx=0x7f9c6c00cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:45.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.793+0000 7f9c62ffd700 1 -- 172.21.15.16:0/897468613 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c6c00aa60 con 0x7f9c740d2110 2023-10-16T12:27:45.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.793+0000 7f9c62ffd700 1 -- 172.21.15.16:0/897468613 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c6c00abc0 con 0x7f9c740d2110 2023-10-16T12:27:45.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.793+0000 7f9c789d7700 1 -- 172.21.15.16:0/897468613 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c7415a230 con 0x7f9c740d2110 2023-10-16T12:27:45.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.793+0000 7f9c62ffd700 1 -- 172.21.15.16:0/897468613 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c6c012420 con 0x7f9c740d2110 2023-10-16T12:27:45.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.793+0000 7f9c789d7700 1 -- 172.21.15.16:0/897468613 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c7415a7f0 con 0x7f9c740d2110 2023-10-16T12:27:45.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.794+0000 7f9c789d7700 1 -- 172.21.15.16:0/897468613 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c7415a9e0 con 0x7f9c740d2110 2023-10-16T12:27:45.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.795+0000 7f9c62ffd700 1 -- 172.21.15.16:0/897468613 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 51) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9c6c012580 con 0x7f9c740d2110 2023-10-16T12:27:45.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.795+0000 7f9c62ffd700 1 --2- 172.21.15.16:0/897468613 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f9c5406b020 0x7f9c5406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:45.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.795+0000 7f9c7259c700 1 --2- 172.21.15.16:0/897468613 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f9c5406b020 0x7f9c5406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:45.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.795+0000 7f9c62ffd700 1 -- 172.21.15.16:0/897468613 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f9c6c089b40 con 0x7f9c740d2110 2023-10-16T12:27:45.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.796+0000 7f9c7259c700 1 --2- 172.21.15.16:0/897468613 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f9c5406b020 0x7f9c5406d4e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9c64009130 tx=0x7f9c64008030 comp rx=0 tx=0).ready entity=mgr.6365 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:45.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:45.800+0000 7f9c62ffd700 1 -- 172.21.15.16:0/897468613 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9c6c054980 con 0x7f9c740d2110 2023-10-16T12:27:46.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.070+0000 7f9c789d7700 1 -- 172.21.15.16:0/897468613 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f9c7404ebb0 con 0x7f9c740d2110 2023-10-16T12:27:46.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.264+0000 7f9c62ffd700 1 -- 172.21.15.16:0/897468613 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v52) v1 ==== 92+0+0 (secure 0 0 0) 0x7f9c6c057fa0 con 0x7f9c740d2110 2023-10-16T12:27:46.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.268+0000 7f9c62ffd700 1 -- 172.21.15.16:0/897468613 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mgrmap(e 52) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f9c6c087f00 con 0x7f9c740d2110 2023-10-16T12:27:46.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.269+0000 7f9c789d7700 1 -- 172.21.15.16:0/897468613 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f9c5406b020 msgr2=0x7f9c5406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:46.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.269+0000 7f9c789d7700 1 --2- 172.21.15.16:0/897468613 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f9c5406b020 0x7f9c5406d4e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9c64009130 tx=0x7f9c64008030 comp rx=0 tx=0).stop 2023-10-16T12:27:46.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.269+0000 7f9c789d7700 1 -- 172.21.15.16:0/897468613 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c740d2110 msgr2=0x7f9c7415f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:46.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.269+0000 7f9c789d7700 1 --2- 172.21.15.16:0/897468613 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c740d2110 0x7f9c7415f060 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f9c6c00cc10 tx=0x7f9c6c00cf20 comp rx=0 tx=0).stop 2023-10-16T12:27:46.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.269+0000 7f9c789d7700 1 -- 172.21.15.16:0/897468613 shutdown_connections 2023-10-16T12:27:46.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.269+0000 7f9c789d7700 1 --2- 172.21.15.16:0/897468613 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c740d2110 0x7f9c7415f060 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:46.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.269+0000 7f9c789d7700 1 --2- 172.21.15.16:0/897468613 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9c740b2620 0x7f9c7415e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:46.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.269+0000 7f9c789d7700 1 --2- 172.21.15.16:0/897468613 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f9c5406b020 0x7f9c5406d4e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:46.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.269+0000 7f9c789d7700 1 --2- 172.21.15.16:0/897468613 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9c740ce6a0 0x7f9c74159490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:46.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.269+0000 7f9c789d7700 1 -- 172.21.15.16:0/897468613 >> 172.21.15.16:0/897468613 conn(0x7f9c740d15f0 msgr2=0x7f9c740cf910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:46.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.269+0000 7f9c789d7700 1 -- 172.21.15.16:0/897468613 shutdown_connections 2023-10-16T12:27:46.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.269+0000 7f9c789d7700 1 -- 172.21.15.16:0/897468613 wait complete. 2023-10-16T12:27:46.285 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-16T12:27:46.419 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:27:46.417+0000 7fb4b7a15200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T12:27:46.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.423+0000 7fceacef0700 1 Processor -- start 2023-10-16T12:27:46.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.423+0000 7fceacef0700 1 -- start start 2023-10-16T12:27:46.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.424+0000 7fceacef0700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcea815ec80 0x7fcea80cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:46.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.424+0000 7fceacef0700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcea80b2440 con 0x7fcea80d2110 2023-10-16T12:27:46.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.424+0000 7fceacef0700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcea80d00e0 con 0x7fcea815f060 2023-10-16T12:27:46.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.424+0000 7fceacef0700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcea80b2c50 con 0x7fcea815ec80 2023-10-16T12:27:46.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.425+0000 7fcea6d9d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fcea80d2110 0x7fcea80b4b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:60034/0 (socket says 172.21.15.16:60034) 2023-10-16T12:27:46.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.425+0000 7fcea6d9d700 1 -- 172.21.15.16:0/3659543218 learned_addr learned my addr 172.21.15.16:0/3659543218 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:27:46.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.425+0000 7fcea659c700 1 --2- 172.21.15.16:0/3659543218 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcea815ec80 0x7fcea80cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:46.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.425+0000 7fcea559a700 1 -- 172.21.15.16:0/3659543218 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4265561773 0 0) 0x7fcea80b2440 con 0x7fcea80d2110 2023-10-16T12:27:46.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.425+0000 7fcea559a700 1 -- 172.21.15.16:0/3659543218 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce900035b0 con 0x7fcea80d2110 2023-10-16T12:27:46.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.425+0000 7fcea559a700 1 -- 172.21.15.16:0/3659543218 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4148103451 0 0) 0x7fcea80d00e0 con 0x7fcea815f060 2023-10-16T12:27:46.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.425+0000 7fcea559a700 1 -- 172.21.15.16:0/3659543218 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcea80b2440 con 0x7fcea815f060 2023-10-16T12:27:46.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.425+0000 7fcea659c700 1 -- 172.21.15.16:0/3659543218 >> v1:172.21.15.49:6789/0 conn(0x7fcea815f060 legacy=0x7fcea80b4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:46.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.425+0000 7fcea659c700 1 -- 172.21.15.16:0/3659543218 >> v1:172.21.15.16:6789/0 conn(0x7fcea80d2110 legacy=0x7fcea80b4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:46.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.425+0000 7fcea659c700 1 -- 172.21.15.16:0/3659543218 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcea80cef60 con 0x7fcea815ec80 2023-10-16T12:27:46.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.425+0000 7fcea659c700 1 --2- 172.21.15.16:0/3659543218 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcea815ec80 0x7fcea80cdec0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fce98009870 tx=0x7fce98009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ea2114d29b5c51b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:46.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.425+0000 7fcea559a700 1 -- 172.21.15.16:0/3659543218 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce98011830 con 0x7fcea815ec80 2023-10-16T12:27:46.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.426+0000 7fcea559a700 1 -- 172.21.15.16:0/3659543218 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce98011990 con 0x7fcea815ec80 2023-10-16T12:27:46.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.426+0000 7fcea559a700 1 -- 172.21.15.16:0/3659543218 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce98011d20 con 0x7fcea815ec80 2023-10-16T12:27:46.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.426+0000 7fceacef0700 1 -- 172.21.15.16:0/3659543218 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcea815ec80 msgr2=0x7fcea80cdec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:46.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.426+0000 7fceacef0700 1 --2- 172.21.15.16:0/3659543218 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcea815ec80 0x7fcea80cdec0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fce98009870 tx=0x7fce98009b80 comp rx=0 tx=0).stop 2023-10-16T12:27:46.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.426+0000 7fceacef0700 1 -- 172.21.15.16:0/3659543218 shutdown_connections 2023-10-16T12:27:46.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.426+0000 7fceacef0700 1 --2- 172.21.15.16:0/3659543218 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcea815ec80 0x7fcea80cdec0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:46.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.426+0000 7fceacef0700 1 -- 172.21.15.16:0/3659543218 >> 172.21.15.16:0/3659543218 conn(0x7fcea80d15f0 msgr2=0x7fcea80d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:46.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.426+0000 7fceacef0700 1 -- 172.21.15.16:0/3659543218 shutdown_connections 2023-10-16T12:27:46.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.426+0000 7fceacef0700 1 -- 172.21.15.16:0/3659543218 wait complete. 2023-10-16T12:27:46.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.428+0000 7fceacef0700 1 Processor -- start 2023-10-16T12:27:46.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.428+0000 7fceacef0700 1 -- start start 2023-10-16T12:27:46.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.428+0000 7fceacef0700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcea80d2110 0x7fcea80c90d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:46.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.428+0000 7fceacef0700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcea815ec80 0x7fcea8151b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:46.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.428+0000 7fceacef0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcea815f060 0x7fcea81e5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:46.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.428+0000 7fceacef0700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea80c7680 con 0x7fcea815f060 2023-10-16T12:27:46.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.428+0000 7fceacef0700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea81520d0 con 0x7fcea80d2110 2023-10-16T12:27:46.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.428+0000 7fceacef0700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcea80c6b00 con 0x7fcea815ec80 2023-10-16T12:27:46.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.428+0000 7fcea6d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcea815f060 0x7fcea81e5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:46.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.428+0000 7fcea6d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcea815f060 0x7fcea81e5450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42534/0 (socket says 172.21.15.16:42534) 2023-10-16T12:27:46.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.429+0000 7fcea6d9d700 1 -- 172.21.15.16:0/1792673408 learned_addr learned my addr 172.21.15.16:0/1792673408 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:46.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.428+0000 7fcea5d9b700 1 --2- 172.21.15.16:0/1792673408 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcea815ec80 0x7fcea8151b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:46.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.429+0000 7fcea659c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcea80d2110 0x7fcea80c90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:46.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.429+0000 7fcea6d9d700 1 -- 172.21.15.16:0/1792673408 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcea815ec80 msgr2=0x7fcea8151b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:46.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.429+0000 7fcea6d9d700 1 --2- 172.21.15.16:0/1792673408 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcea815ec80 0x7fcea8151b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:46.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.429+0000 7fcea6d9d700 1 -- 172.21.15.16:0/1792673408 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcea80d2110 msgr2=0x7fcea80c90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:46.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.429+0000 7fcea6d9d700 1 --2- 172.21.15.16:0/1792673408 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcea80d2110 0x7fcea80c90d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:46.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.429+0000 7fcea6d9d700 1 -- 172.21.15.16:0/1792673408 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce980094a0 con 0x7fcea815f060 2023-10-16T12:27:46.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.429+0000 7fcea6d9d700 1 --2- 172.21.15.16:0/1792673408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcea815f060 0x7fcea81e5450 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7fcea000cb40 tx=0x7fcea000ce50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:46.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.429+0000 7fce96ffd700 1 -- 172.21.15.16:0/1792673408 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea000a9e0 con 0x7fcea815f060 2023-10-16T12:27:46.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.429+0000 7fce96ffd700 1 -- 172.21.15.16:0/1792673408 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcea000ab40 con 0x7fcea815f060 2023-10-16T12:27:46.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.429+0000 7fce96ffd700 1 -- 172.21.15.16:0/1792673408 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea0012400 con 0x7fcea815f060 2023-10-16T12:27:46.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.429+0000 7fceacef0700 1 -- 172.21.15.16:0/1792673408 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcea80c6de0 con 0x7fcea815f060 2023-10-16T12:27:46.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.429+0000 7fceacef0700 1 -- 172.21.15.16:0/1792673408 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcea80c73a0 con 0x7fcea815f060 2023-10-16T12:27:46.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.431+0000 7fceacef0700 1 -- 172.21.15.16:0/1792673408 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcea804ebb0 con 0x7fcea815f060 2023-10-16T12:27:46.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.431+0000 7fce96ffd700 1 -- 172.21.15.16:0/1792673408 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 52) v1 ==== 89348+0+0 (secure 0 0 0) 0x7fcea001b3b0 con 0x7fcea815f060 2023-10-16T12:27:46.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.431+0000 7fce96ffd700 1 --2- 172.21.15.16:0/1792673408 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fce8806b070 0x7fce8806d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:46.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.431+0000 7fcea659c700 1 -- 172.21.15.16:0/1792673408 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fce8806b070 msgr2=0x7fce8806d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/4089191642 2023-10-16T12:27:46.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.431+0000 7fce96ffd700 1 -- 172.21.15.16:0/1792673408 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fcea00895d0 con 0x7fcea815f060 2023-10-16T12:27:46.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.431+0000 7fcea659c700 1 --2- 172.21.15.16:0/1792673408 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fce8806b070 0x7fce8806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:27:46.442 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:46.433+0000 7fc827778200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T12:27:46.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.436+0000 7fce96ffd700 1 -- 172.21.15.16:0/1792673408 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcea0054410 con 0x7fcea815f060 2023-10-16T12:27:46.489 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:27:46.487+0000 7fb4b7a15200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T12:27:46.505 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:46.504+0000 7fc827778200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T12:27:46.564 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:27:46.562+0000 7fb4b7a15200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T12:27:46.581 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:46.580+0000 7fc827778200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T12:27:46.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.632+0000 7fcea659c700 1 -- 172.21.15.16:0/1792673408 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fce8806b070 msgr2=0x7fce8806d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/4089191642 2023-10-16T12:27:46.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.632+0000 7fcea659c700 1 --2- 172.21.15.16:0/1792673408 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fce8806b070 0x7fce8806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:27:46.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.641+0000 7fceacef0700 1 -- 172.21.15.16:0/1792673408 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7fcea806fa80 con 0x7fcea815f060 2023-10-16T12:27:46.645 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:27:46.643+0000 7fb4b7a15200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T12:27:46.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.644+0000 7fce96ffd700 1 -- 172.21.15.16:0/1792673408 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v35)=0 v35) v1 ==== 138+0+0 (secure 0 0 0) 0x7fcea0057a30 con 0x7fcea815f060 2023-10-16T12:27:46.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.647+0000 7fceacef0700 1 -- 172.21.15.16:0/1792673408 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fce8806b070 msgr2=0x7fce8806d530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:27:46.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.647+0000 7fceacef0700 1 --2- 172.21.15.16:0/1792673408 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fce8806b070 0x7fce8806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:46.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.647+0000 7fceacef0700 1 -- 172.21.15.16:0/1792673408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcea815f060 msgr2=0x7fcea81e5450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:46.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.647+0000 7fceacef0700 1 --2- 172.21.15.16:0/1792673408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcea815f060 0x7fcea81e5450 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7fcea000cb40 tx=0x7fcea000ce50 comp rx=0 tx=0).stop 2023-10-16T12:27:46.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.647+0000 7fceacef0700 1 -- 172.21.15.16:0/1792673408 shutdown_connections 2023-10-16T12:27:46.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.647+0000 7fceacef0700 1 --2- 172.21.15.16:0/1792673408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcea815f060 0x7fcea81e5450 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:46.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.647+0000 7fceacef0700 1 --2- 172.21.15.16:0/1792673408 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fce8806b070 0x7fce8806d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:46.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.647+0000 7fceacef0700 1 --2- 172.21.15.16:0/1792673408 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcea80d2110 0x7fcea80c90d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:46.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.647+0000 7fceacef0700 1 --2- 172.21.15.16:0/1792673408 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcea815ec80 0x7fcea8151b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:46.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.647+0000 7fceacef0700 1 -- 172.21.15.16:0/1792673408 >> 172.21.15.16:0/1792673408 conn(0x7fcea80d15f0 msgr2=0x7fcea8070280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:46.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.647+0000 7fceacef0700 1 -- 172.21.15.16:0/1792673408 shutdown_connections 2023-10-16T12:27:46.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.647+0000 7fceacef0700 1 -- 172.21.15.16:0/1792673408 wait complete. 2023-10-16T12:27:46.662 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-16T12:27:46.665 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:46.663+0000 7fc827778200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T12:27:46.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.790+0000 7f1e8b61a700 1 Processor -- start 2023-10-16T12:27:46.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.790+0000 7f1e8b61a700 1 -- start start 2023-10-16T12:27:46.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.791+0000 7f1e8b61a700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f1e8415d8d0 0x7f1e840b2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:46.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.791+0000 7f1e8b61a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1e840b8fa0 0x7f1e840b6720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:46.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.791+0000 7f1e8b61a700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e840b2f30 con 0x7f1e840b6c60 2023-10-16T12:27:46.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.791+0000 7f1e8b61a700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e840d0bb0 con 0x7f1e8415d8d0 2023-10-16T12:27:46.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.791+0000 7f1e8b61a700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8415dcb0 con 0x7f1e840b8fa0 2023-10-16T12:27:46.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.791+0000 7f1e89bb7700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f1e840b6c60 0x7f1e840ce350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:60070/0 (socket says 172.21.15.16:60070) 2023-10-16T12:27:46.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.791+0000 7f1e89bb7700 1 -- 172.21.15.16:0/3622943287 learned_addr learned my addr 172.21.15.16:0/3622943287 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:27:46.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.791+0000 7f1e88bb5700 1 --2- 172.21.15.16:0/3622943287 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1e840b8fa0 0x7f1e840b6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:46.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.791+0000 7f1e893b6700 1 --2- 172.21.15.16:0/3622943287 >> v2:172.21.15.49:3300/0 conn(0x7f1e8415d8d0 0x7f1e840b2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:46.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.791+0000 7f1e7bfff700 1 -- 172.21.15.16:0/3622943287 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 848889619 0 0) 0x7f1e840b2f30 con 0x7f1e840b6c60 2023-10-16T12:27:46.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.792+0000 7f1e7bfff700 1 -- 172.21.15.16:0/3622943287 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e700035b0 con 0x7f1e840b6c60 2023-10-16T12:27:46.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.792+0000 7f1e893b6700 1 -- 172.21.15.16:0/3622943287 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1e840b8fa0 msgr2=0x7f1e840b6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:46.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.792+0000 7f1e893b6700 1 --2- 172.21.15.16:0/3622943287 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1e840b8fa0 0x7f1e840b6720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:46.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.792+0000 7f1e893b6700 1 -- 172.21.15.16:0/3622943287 >> v1:172.21.15.16:6789/0 conn(0x7f1e840b6c60 legacy=0x7f1e840ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:46.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.792+0000 7f1e893b6700 1 -- 172.21.15.16:0/3622943287 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e840d1c40 con 0x7f1e8415d8d0 2023-10-16T12:27:46.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.792+0000 7f1e88bb5700 1 --2- 172.21.15.16:0/3622943287 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1e840b8fa0 0x7f1e840b6720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:27:46.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.792+0000 7f1e893b6700 1 --2- 172.21.15.16:0/3622943287 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1e8415d8d0 0x7f1e840b2830 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f1e80009840 tx=0x7f1e80009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e5b0b17bcff9243 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:46.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.792+0000 7f1e7bfff700 1 -- 172.21.15.16:0/3622943287 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e80011830 con 0x7f1e8415d8d0 2023-10-16T12:27:46.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.792+0000 7f1e7bfff700 1 -- 172.21.15.16:0/3622943287 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e80011990 con 0x7f1e8415d8d0 2023-10-16T12:27:46.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.792+0000 7f1e7bfff700 1 -- 172.21.15.16:0/3622943287 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e80011d00 con 0x7f1e8415d8d0 2023-10-16T12:27:46.798 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:27:46.792+0000 7fb4b7a15200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T12:27:46.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.793+0000 7f1e8b61a700 1 -- 172.21.15.16:0/3622943287 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1e8415d8d0 msgr2=0x7f1e840b2830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:46.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.793+0000 7f1e8b61a700 1 --2- 172.21.15.16:0/3622943287 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1e8415d8d0 0x7f1e840b2830 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f1e80009840 tx=0x7f1e80009b50 comp rx=0 tx=0).stop 2023-10-16T12:27:46.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.793+0000 7f1e8b61a700 1 -- 172.21.15.16:0/3622943287 shutdown_connections 2023-10-16T12:27:46.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.793+0000 7f1e8b61a700 1 --2- 172.21.15.16:0/3622943287 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1e8415d8d0 0x7f1e840b2830 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:46.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.793+0000 7f1e8b61a700 1 --2- 172.21.15.16:0/3622943287 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1e840b8fa0 0x7f1e840b6720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:46.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.793+0000 7f1e8b61a700 1 -- 172.21.15.16:0/3622943287 >> 172.21.15.16:0/3622943287 conn(0x7f1e84072330 msgr2=0x7f1e84072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:46.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.793+0000 7f1e8b61a700 1 -- 172.21.15.16:0/3622943287 shutdown_connections 2023-10-16T12:27:46.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.793+0000 7f1e8b61a700 1 -- 172.21.15.16:0/3622943287 wait complete. 2023-10-16T12:27:46.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.794+0000 7f1e8b61a700 1 Processor -- start 2023-10-16T12:27:46.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.794+0000 7f1e8b61a700 1 -- start start 2023-10-16T12:27:46.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.795+0000 7f1e8b61a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e840b6c60 0x7f1e841f71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:46.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.795+0000 7f1e8b61a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1e840b8fa0 0x7f1e841f2920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:46.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.795+0000 7f1e8b61a700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1e8415d8d0 0x7f1e840b8090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:46.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.795+0000 7f1e8b61a700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8414d540 con 0x7f1e840b6c60 2023-10-16T12:27:46.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.795+0000 7f1e8b61a700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8414d260 con 0x7f1e8415d8d0 2023-10-16T12:27:46.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.795+0000 7f1e8b61a700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e841f66d0 con 0x7f1e840b8fa0 2023-10-16T12:27:46.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.795+0000 7f1e893b6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e840b6c60 0x7f1e841f71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:46.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.795+0000 7f1e893b6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e840b6c60 0x7f1e841f71b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42566/0 (socket says 172.21.15.16:42566) 2023-10-16T12:27:46.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.795+0000 7f1e893b6700 1 -- 172.21.15.16:0/2387818242 learned_addr learned my addr 172.21.15.16:0/2387818242 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:46.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.795+0000 7f1e89bb7700 1 --2- 172.21.15.16:0/2387818242 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1e8415d8d0 0x7f1e840b8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:46.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.795+0000 7f1e88bb5700 1 --2- 172.21.15.16:0/2387818242 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1e840b8fa0 0x7f1e841f2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:46.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.795+0000 7f1e893b6700 1 -- 172.21.15.16:0/2387818242 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1e840b8fa0 msgr2=0x7f1e841f2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:46.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.795+0000 7f1e893b6700 1 --2- 172.21.15.16:0/2387818242 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1e840b8fa0 0x7f1e841f2920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:46.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.795+0000 7f1e893b6700 1 -- 172.21.15.16:0/2387818242 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1e8415d8d0 msgr2=0x7f1e840b8090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:46.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.795+0000 7f1e893b6700 1 --2- 172.21.15.16:0/2387818242 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1e8415d8d0 0x7f1e840b8090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:46.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.795+0000 7f1e893b6700 1 -- 172.21.15.16:0/2387818242 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e740094a0 con 0x7f1e840b6c60 2023-10-16T12:27:46.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.796+0000 7f1e893b6700 1 --2- 172.21.15.16:0/2387818242 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e840b6c60 0x7f1e841f71b0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f1e80009b90 tx=0x7f1e80004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:46.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.796+0000 7f1e79ffb700 1 -- 172.21.15.16:0/2387818242 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e80004550 con 0x7f1e840b6c60 2023-10-16T12:27:46.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.796+0000 7f1e79ffb700 1 -- 172.21.15.16:0/2387818242 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e8001f070 con 0x7f1e840b6c60 2023-10-16T12:27:46.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.796+0000 7f1e8b61a700 1 -- 172.21.15.16:0/2387818242 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e800094a0 con 0x7f1e840b6c60 2023-10-16T12:27:46.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.796+0000 7f1e79ffb700 1 -- 172.21.15.16:0/2387818242 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e800038e0 con 0x7f1e840b6c60 2023-10-16T12:27:46.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.796+0000 7f1e8b61a700 1 -- 172.21.15.16:0/2387818242 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e841f6c90 con 0x7f1e840b6c60 2023-10-16T12:27:46.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.797+0000 7f1e8b61a700 1 -- 172.21.15.16:0/2387818242 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e8404ebb0 con 0x7f1e840b6c60 2023-10-16T12:27:46.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.797+0000 7f1e79ffb700 1 -- 172.21.15.16:0/2387818242 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 52) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f1e80003a40 con 0x7f1e840b6c60 2023-10-16T12:27:46.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.798+0000 7f1e79ffb700 1 --2- 172.21.15.16:0/2387818242 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f1e6806b070 0x7f1e6806d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:46.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.798+0000 7f1e88bb5700 1 -- 172.21.15.16:0/2387818242 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f1e6806b070 msgr2=0x7f1e6806d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/4089191642 2023-10-16T12:27:46.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.798+0000 7f1e79ffb700 1 -- 172.21.15.16:0/2387818242 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f1e80021070 con 0x7f1e840b6c60 2023-10-16T12:27:46.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.798+0000 7f1e88bb5700 1 --2- 172.21.15.16:0/2387818242 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f1e6806b070 0x7f1e6806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:27:46.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.802+0000 7f1e79ffb700 1 -- 172.21.15.16:0/2387818242 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1e80055f00 con 0x7f1e840b6c60 2023-10-16T12:27:46.820 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:46.818+0000 7fc827778200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T12:27:47.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.999+0000 7f1e88bb5700 1 -- 172.21.15.16:0/2387818242 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f1e6806b070 msgr2=0x7f1e6806d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/4089191642 2023-10-16T12:27:47.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:46.999+0000 7f1e88bb5700 1 --2- 172.21.15.16:0/2387818242 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f1e6806b070 0x7f1e6806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:27:47.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.009+0000 7f1e8b61a700 1 -- 172.21.15.16:0/2387818242 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f1e84160790 con 0x7f1e840b6c60 2023-10-16T12:27:47.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.012+0000 7f1e79ffb700 1 -- 172.21.15.16:0/2387818242 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v36)=0 v36) v1 ==== 133+0+0 (secure 0 0 0) 0x7f1e8002b0c0 con 0x7f1e840b6c60 2023-10-16T12:27:47.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.016+0000 7f1e8b61a700 1 -- 172.21.15.16:0/2387818242 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f1e6806b070 msgr2=0x7f1e6806d530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:27:47.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.016+0000 7f1e8b61a700 1 --2- 172.21.15.16:0/2387818242 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f1e6806b070 0x7f1e6806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:47.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.016+0000 7f1e8b61a700 1 -- 172.21.15.16:0/2387818242 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e840b6c60 msgr2=0x7f1e841f71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:47.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.016+0000 7f1e8b61a700 1 --2- 172.21.15.16:0/2387818242 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e840b6c60 0x7f1e841f71b0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f1e80009b90 tx=0x7f1e80004750 comp rx=0 tx=0).stop 2023-10-16T12:27:47.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.016+0000 7f1e8b61a700 1 -- 172.21.15.16:0/2387818242 shutdown_connections 2023-10-16T12:27:47.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.016+0000 7f1e8b61a700 1 --2- 172.21.15.16:0/2387818242 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7f1e6806b070 0x7f1e6806d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:47.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.016+0000 7f1e8b61a700 1 --2- 172.21.15.16:0/2387818242 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1e8415d8d0 0x7f1e840b8090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:47.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.016+0000 7f1e8b61a700 1 --2- 172.21.15.16:0/2387818242 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e840b6c60 0x7f1e841f71b0 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:47.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.016+0000 7f1e8b61a700 1 --2- 172.21.15.16:0/2387818242 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1e840b8fa0 0x7f1e841f2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:47.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.016+0000 7f1e8b61a700 1 -- 172.21.15.16:0/2387818242 >> 172.21.15.16:0/2387818242 conn(0x7f1e84072330 msgr2=0x7f1e8415ddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:47.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.016+0000 7f1e8b61a700 1 -- 172.21.15.16:0/2387818242 shutdown_connections 2023-10-16T12:27:47.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.016+0000 7f1e8b61a700 1 -- 172.21.15.16:0/2387818242 wait complete. 2023-10-16T12:27:47.035 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /bad-path 1m 2023-10-16T12:27:47.138 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:27:47.136+0000 7fb4b7a15200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T12:27:47.176 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:47.174+0000 7fc827778200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T12:27:47.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.202+0000 7fdc03208700 1 Processor -- start 2023-10-16T12:27:47.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.202+0000 7fdc03208700 1 -- start start 2023-10-16T12:27:47.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.204+0000 7fdc03208700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdbfc15ec80 0x7fdbfc0cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:47.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.204+0000 7fdc03208700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdbfc0d2110 0x7fdbfc0cd1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:47.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.204+0000 7fdc03208700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbfc0b2440 con 0x7fdbfc15f060 2023-10-16T12:27:47.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.204+0000 7fdc03208700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbfc0b2c50 con 0x7fdbfc15ec80 2023-10-16T12:27:47.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.204+0000 7fdc03208700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbfc0d00e0 con 0x7fdbfc0d2110 2023-10-16T12:27:47.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.204+0000 7fdc00fa4700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdbfc15ec80 0x7fdbfc0cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:47.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.204+0000 7fdc00fa4700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdbfc15ec80 0x7fdbfc0cdec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42610/0 (socket says 172.21.15.16:42610) 2023-10-16T12:27:47.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.204+0000 7fdc00fa4700 1 -- 172.21.15.16:0/3371078674 learned_addr learned my addr 172.21.15.16:0/3371078674 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:47.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.204+0000 7fdc017a5700 1 --2- 172.21.15.16:0/3371078674 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdbfc0d2110 0x7fdbfc0cd1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:47.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.204+0000 7fdc00fa4700 1 -- 172.21.15.16:0/3371078674 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdbfc0d2110 msgr2=0x7fdbfc0cd1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:47.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.204+0000 7fdc00fa4700 1 --2- 172.21.15.16:0/3371078674 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdbfc0d2110 0x7fdbfc0cd1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:47.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.204+0000 7fdc00fa4700 1 -- 172.21.15.16:0/3371078674 >> v1:172.21.15.49:6789/0 conn(0x7fdbfc15f060 legacy=0x7fdbfc0b4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:47.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.204+0000 7fdc00fa4700 1 -- 172.21.15.16:0/3371078674 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbfc0cef60 con 0x7fdbfc15ec80 2023-10-16T12:27:47.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.205+0000 7fdc00fa4700 1 --2- 172.21.15.16:0/3371078674 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbfc15ec80 0x7fdbfc0cdec0 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7fdbf00098f0 tx=0x7fdbf0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=baf4a757bc5d3466 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:47.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.205+0000 7fdbfb7fe700 1 -- 172.21.15.16:0/3371078674 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf0011830 con 0x7fdbfc15ec80 2023-10-16T12:27:47.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.205+0000 7fdbfb7fe700 1 -- 172.21.15.16:0/3371078674 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbf0011990 con 0x7fdbfc15ec80 2023-10-16T12:27:47.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.205+0000 7fdbfb7fe700 1 -- 172.21.15.16:0/3371078674 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf0011d20 con 0x7fdbfc15ec80 2023-10-16T12:27:47.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.205+0000 7fdc03208700 1 -- 172.21.15.16:0/3371078674 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbfc15ec80 msgr2=0x7fdbfc0cdec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:47.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.205+0000 7fdc03208700 1 --2- 172.21.15.16:0/3371078674 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbfc15ec80 0x7fdbfc0cdec0 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7fdbf00098f0 tx=0x7fdbf0009c00 comp rx=0 tx=0).stop 2023-10-16T12:27:47.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.205+0000 7fdc03208700 1 -- 172.21.15.16:0/3371078674 shutdown_connections 2023-10-16T12:27:47.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.205+0000 7fdc03208700 1 --2- 172.21.15.16:0/3371078674 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdbfc0d2110 0x7fdbfc0cd1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:47.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.205+0000 7fdc03208700 1 --2- 172.21.15.16:0/3371078674 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbfc15ec80 0x7fdbfc0cdec0 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:47.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.205+0000 7fdc03208700 1 -- 172.21.15.16:0/3371078674 >> 172.21.15.16:0/3371078674 conn(0x7fdbfc0d15f0 msgr2=0x7fdbfc0d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:47.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.205+0000 7fdc03208700 1 -- 172.21.15.16:0/3371078674 shutdown_connections 2023-10-16T12:27:47.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.205+0000 7fdc03208700 1 -- 172.21.15.16:0/3371078674 wait complete. 2023-10-16T12:27:47.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.207+0000 7fdc03208700 1 Processor -- start 2023-10-16T12:27:47.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.207+0000 7fdc03208700 1 -- start start 2023-10-16T12:27:47.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.209+0000 7fdc03208700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbfc0d2110 0x7fdbfc0b69f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:47.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.209+0000 7fdc03208700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdbfc15ec80 0x7fdbfc1e5a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:47.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.209+0000 7fdc03208700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdbfc15f060 0x7fdbfc1511a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:47.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.209+0000 7fdc03208700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbfc1e5190 con 0x7fdbfc0d2110 2023-10-16T12:27:47.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.209+0000 7fdc03208700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbfc1e4510 con 0x7fdbfc15ec80 2023-10-16T12:27:47.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.209+0000 7fdc03208700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbfc1e47f0 con 0x7fdbfc15f060 2023-10-16T12:27:47.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.209+0000 7fdc00fa4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbfc0d2110 0x7fdbfc0b69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:47.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.209+0000 7fdc00fa4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbfc0d2110 0x7fdbfc0b69f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42616/0 (socket says 172.21.15.16:42616) 2023-10-16T12:27:47.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.209+0000 7fdc00fa4700 1 -- 172.21.15.16:0/2365162763 learned_addr learned my addr 172.21.15.16:0/2365162763 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:47.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.209+0000 7fdc017a5700 1 --2- 172.21.15.16:0/2365162763 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdbfc15f060 0x7fdbfc1511a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:47.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.209+0000 7fdbfbfff700 1 --2- 172.21.15.16:0/2365162763 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdbfc15ec80 0x7fdbfc1e5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:47.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.209+0000 7fdc00fa4700 1 -- 172.21.15.16:0/2365162763 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdbfc15f060 msgr2=0x7fdbfc1511a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:47.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.209+0000 7fdc00fa4700 1 --2- 172.21.15.16:0/2365162763 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdbfc15f060 0x7fdbfc1511a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:47.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.209+0000 7fdc00fa4700 1 -- 172.21.15.16:0/2365162763 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdbfc15ec80 msgr2=0x7fdbfc1e5a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:47.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.209+0000 7fdc00fa4700 1 --2- 172.21.15.16:0/2365162763 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdbfc15ec80 0x7fdbfc1e5a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:47.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.209+0000 7fdc00fa4700 1 -- 172.21.15.16:0/2365162763 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbf0009550 con 0x7fdbfc0d2110 2023-10-16T12:27:47.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.209+0000 7fdc00fa4700 1 --2- 172.21.15.16:0/2365162763 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbfc0d2110 0x7fdbfc0b69f0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7fdbf0000c00 tx=0x7fdbf0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:47.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.209+0000 7fdbf97fa700 1 -- 172.21.15.16:0/2365162763 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf0004400 con 0x7fdbfc0d2110 2023-10-16T12:27:47.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.210+0000 7fdbf97fa700 1 -- 172.21.15.16:0/2365162763 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbf001f070 con 0x7fdbfc0d2110 2023-10-16T12:27:47.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.210+0000 7fdbf97fa700 1 -- 172.21.15.16:0/2365162763 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf00038e0 con 0x7fdbfc0d2110 2023-10-16T12:27:47.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.210+0000 7fdc03208700 1 -- 172.21.15.16:0/2365162763 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbfc1e4ad0 con 0x7fdbfc0d2110 2023-10-16T12:27:47.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.210+0000 7fdc03208700 1 -- 172.21.15.16:0/2365162763 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbfc1e41a0 con 0x7fdbfc0d2110 2023-10-16T12:27:47.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.210+0000 7fdc03208700 1 -- 172.21.15.16:0/2365162763 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbfc04ebb0 con 0x7fdbfc0d2110 2023-10-16T12:27:47.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.211+0000 7fdbf97fa700 1 -- 172.21.15.16:0/2365162763 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 52) v1 ==== 89348+0+0 (secure 0 0 0) 0x7fdbf0021030 con 0x7fdbfc0d2110 2023-10-16T12:27:47.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.212+0000 7fdbf97fa700 1 --2- 172.21.15.16:0/2365162763 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fdbe006b070 0x7fdbe006d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:47.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.212+0000 7fdbfbfff700 1 -- 172.21.15.16:0/2365162763 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fdbe006b070 msgr2=0x7fdbe006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/4089191642 2023-10-16T12:27:47.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.212+0000 7fdbf97fa700 1 -- 172.21.15.16:0/2365162763 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fdbf008ac80 con 0x7fdbfc0d2110 2023-10-16T12:27:47.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.212+0000 7fdbfbfff700 1 --2- 172.21.15.16:0/2365162763 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fdbe006b070 0x7fdbe006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:27:47.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.216+0000 7fdbf97fa700 1 -- 172.21.15.16:0/2365162763 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdbf0055b80 con 0x7fdbfc0d2110 2023-10-16T12:27:47.258 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:27:47.256+0000 7fb4b7a15200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T12:27:47.303 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:47.301+0000 7fc827778200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T12:27:47.335 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:27:47.333+0000 7fb4b7a15200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T12:27:47.380 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:47.378+0000 7fc827778200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T12:27:47.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.413+0000 7fdbfbfff700 1 -- 172.21.15.16:0/2365162763 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fdbe006b070 msgr2=0x7fdbe006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/4089191642 2023-10-16T12:27:47.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.413+0000 7fdbfbfff700 1 --2- 172.21.15.16:0/2365162763 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fdbe006b070 0x7fdbe006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:27:47.419 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:27:47.417+0000 7fb4b7a15200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T12:27:47.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.453+0000 7fdc03208700 1 -- 172.21.15.16:0/2365162763 --> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/bad-path", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fdbfc0b2970 con 0x7fdbe006b070 2023-10-16T12:27:47.465 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:47.464+0000 7fc827778200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T12:27:47.807 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:27:47.805+0000 7fb4b7a15200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T12:27:47.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.814+0000 7fdbfbfff700 1 -- 172.21.15.16:0/2365162763 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fdbe006b070 msgr2=0x7fdbe006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/4089191642 2023-10-16T12:27:47.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:47.814+0000 7fdbfbfff700 1 --2- 172.21.15.16:0/2365162763 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fdbe006b070 0x7fdbe006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-16T12:27:47.856 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:47.854+0000 7fc827778200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T12:27:48.415 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:27:48.413+0000 7fb4b7a15200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T12:27:48.466 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:48.464+0000 7fc827778200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T12:27:48.488 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:27:48.486+0000 7fb4b7a15200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T12:27:48.540 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:48.538+0000 7fc827778200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T12:27:48.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:48.615+0000 7fdbfbfff700 1 -- 172.21.15.16:0/2365162763 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fdbe006b070 msgr2=0x7fdbe006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/4089191642 2023-10-16T12:27:48.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:48.615+0000 7fdbfbfff700 1 --2- 172.21.15.16:0/2365162763 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fdbe006b070 0x7fdbe006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-16T12:27:49.148 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:27:49.146+0000 7fb4b7a15200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:27:49.210 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:49.208+0000 7fc827778200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:27:49.229 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:27:49.227+0000 7fb4b7a15200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T12:27:49.292 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:49.291+0000 7fc827778200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T12:27:49.379 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:27:49.377+0000 7fb4b7a15200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:27:49.444 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:49.443+0000 7fc827778200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:27:49.614 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:27:49.612+0000 7fb4b7a15200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T12:27:49.685 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:49.684+0000 7fc827778200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T12:27:49.690 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:27:49.688+0000 7fb4b7a15200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T12:27:49.764 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:49.762+0000 7fc827778200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T12:27:49.965 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:27:49.963+0000 7fb4b7a15200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T12:27:50.042 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:50.040+0000 7fc827778200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T12:27:50.193 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:27:50.192+0000 7fb4b7a15200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T12:27:50.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:50.216+0000 7fdbfbfff700 1 -- 172.21.15.16:0/2365162763 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fdbe006b070 msgr2=0x7fdbe006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/4089191642 2023-10-16T12:27:50.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:50.216+0000 7fdbfbfff700 1 --2- 172.21.15.16:0/2365162763 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fdbe006b070 0x7fdbe006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-10-16T12:27:50.277 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:50.275+0000 7fc827778200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T12:27:50.348 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:27:50.347+0000 7fb4b7a15200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T12:27:50.435 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:50.433+0000 7fc827778200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T12:27:50.467 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:27:50.466+0000 7fb4b7a15200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T12:27:50.555 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:50.553+0000 7fc827778200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T12:27:50.628 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:27:50.627+0000 7fb4b7a15200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T12:27:50.720 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:50.718+0000 7fc827778200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T12:27:50.899 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:27:50.898+0000 7fb4b7a15200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T12:27:50.942 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:27:50.944 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:27:50.999 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:50.997+0000 7fc827778200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T12:27:51.068 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:27:51.067+0000 7fb4b7a15200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T12:27:51.170 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:51.168+0000 7fc827778200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T12:27:51.622 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:27:51.621+0000 7fb4b7a15200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T12:27:51.734 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:51.732+0000 7fc827778200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T12:27:51.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:51.751+0000 7fdbf97fa700 1 -- 172.21.15.16:0/2365162763 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 53) v1 ==== 88957+0+0 (secure 0 0 0) 0x7fdbf0055f50 con 0x7fdbfc0d2110 2023-10-16T12:27:51.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:51.751+0000 7fdbf97fa700 1 -- 172.21.15.16:0/2365162763 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fdbe006b070 msgr2=0x7fdbe006d530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:27:51.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:51.751+0000 7fdbf97fa700 1 --2- 172.21.15.16:0/2365162763 >> [v2:172.21.15.16:6832/4089191642,v1:172.21.15.16:6833/4089191642] conn(0x7fdbe006b070 0x7fdbe006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:51.891 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:51.890+0000 7fc79cf58700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:27:51.892 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:51.890+0000 7fc79cf58700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:27:51.892 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:51.890+0000 7fc79cf58700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:27:51.892 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:51.890+0000 7fc79cf58700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:27:51.892 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:51.890+0000 7fc79cf58700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:27:51.893 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:51.890+0000 7fc79a753700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:27:51.893 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:51.890+0000 7fc79a753700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:27:51.893 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:51.890+0000 7fc79a753700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:27:51.893 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:51.890+0000 7fc79a753700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:27:51.893 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:27:51.890+0000 7fc79a753700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:27:52.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:52.755+0000 7fdbf97fa700 1 -- 172.21.15.16:0/2365162763 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mgrmap(e 54) v1 ==== 89084+0+0 (secure 0 0 0) 0x7fdbf005d580 con 0x7fdbfc0d2110 2023-10-16T12:27:52.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:52.755+0000 7fdbf97fa700 1 --2- 172.21.15.16:0/2365162763 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7fdbe0035910 0x7fdbe00754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:52.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:52.755+0000 7fdbf97fa700 1 -- 172.21.15.16:0/2365162763 --> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/bad-path", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fdbe0035cf0 con 0x7fdbe0035910 2023-10-16T12:27:52.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:52.757+0000 7fdbfbfff700 1 --2- 172.21.15.16:0/2365162763 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7fdbe0035910 0x7fdbe00754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:52.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:52.760+0000 7fdbfbfff700 1 --2- 172.21.15.16:0/2365162763 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7fdbe0035910 0x7fdbe00754a0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc0cc850 tx=0x7fdbe8007ec0 comp rx=0 tx=0).ready entity=mgr.7773 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:52.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:52.831+0000 7fdbf97fa700 1 -- 172.21.15.16:0/2365162763 <== mgr.7773 v2:172.21.15.16:6832/1458543709 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7fdbe0035cf0 con 0x7fdbe0035910 2023-10-16T12:27:53.053 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule set for path /bad-path 2023-10-16T12:27:53.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:52.834+0000 7fdc03208700 1 -- 172.21.15.16:0/2365162763 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7fdbe0035910 msgr2=0x7fdbe00754a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:53.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:52.834+0000 7fdc03208700 1 --2- 172.21.15.16:0/2365162763 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7fdbe0035910 0x7fdbe00754a0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc0cc850 tx=0x7fdbe8007ec0 comp rx=0 tx=0).stop 2023-10-16T12:27:53.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:52.834+0000 7fdc03208700 1 -- 172.21.15.16:0/2365162763 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbfc0d2110 msgr2=0x7fdbfc0b69f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:53.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:52.834+0000 7fdc03208700 1 --2- 172.21.15.16:0/2365162763 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbfc0d2110 0x7fdbfc0b69f0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7fdbf0000c00 tx=0x7fdbf0004750 comp rx=0 tx=0).stop 2023-10-16T12:27:53.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:52.834+0000 7fdc017a5700 1 -- 172.21.15.16:0/2365162763 reap_dead start 2023-10-16T12:27:53.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:52.834+0000 7fdc03208700 1 -- 172.21.15.16:0/2365162763 shutdown_connections 2023-10-16T12:27:53.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:52.834+0000 7fdc03208700 1 -- 172.21.15.16:0/2365162763 >> 172.21.15.16:0/2365162763 conn(0x7fdbfc0d15f0 msgr2=0x7fdbfc15dd30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:53.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:52.834+0000 7fdc03208700 1 -- 172.21.15.16:0/2365162763 shutdown_connections 2023-10-16T12:27:53.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:52.834+0000 7fdc03208700 1 -- 172.21.15.16:0/2365162763 wait complete. 2023-10-16T12:27:53.057 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /bad-path 2023-10-16T12:27:53.058 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /bad-path 2023-10-16T12:27:53.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:53.234+0000 7fc31a6d1700 1 Processor -- start 2023-10-16T12:27:53.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:53.234+0000 7fc31a6d1700 1 -- start start 2023-10-16T12:27:53.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:53.235+0000 7fc31a6d1700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc31415b070 0x7fc31415b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:53.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:53.235+0000 7fc31a6d1700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3140cef70 con 0x7fc31406e170 2023-10-16T12:27:53.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:53.235+0000 7fc31a6d1700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3140ce360 con 0x7fc3140cb880 2023-10-16T12:27:53.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:53.235+0000 7fc31a6d1700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3140ce640 con 0x7fc31415b070 2023-10-16T12:27:53.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:53.235+0000 7fc313fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc31415b070 0x7fc31415b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:53.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:53.235+0000 7fc313fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc31415b070 0x7fc31415b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42802/0 (socket says 172.21.15.16:42802) 2023-10-16T12:27:53.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:53.235+0000 7fc313fff700 1 -- 172.21.15.16:0/2951401554 learned_addr learned my addr 172.21.15.16:0/2951401554 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:53.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:53.235+0000 7fc313fff700 1 -- 172.21.15.16:0/2951401554 >> v1:172.21.15.49:6789/0 conn(0x7fc3140cb880 legacy=0x7fc31406da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:27:53.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:53.235+0000 7fc313fff700 1 -- 172.21.15.16:0/2951401554 >> v1:172.21.15.16:6789/0 conn(0x7fc31406e170 legacy=0x7fc3140cf400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:53.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:53.235+0000 7fc313fff700 1 -- 172.21.15.16:0/2951401554 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3140ce920 con 0x7fc31415b070 2023-10-16T12:27:53.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:53.235+0000 7fc313fff700 1 --2- 172.21.15.16:0/2951401554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc31415b070 0x7fc31415b450 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7fc3040098f0 tx=0x7fc304009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=860f7cf32d6ccc4c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:53.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:53.236+0000 7fc3137fe700 1 -- 172.21.15.16:0/2951401554 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc304004230 con 0x7fc31415b070 2023-10-16T12:27:53.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:53.236+0000 7fc3137fe700 1 -- 172.21.15.16:0/2951401554 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc304004390 con 0x7fc31415b070 2023-10-16T12:27:53.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:53.236+0000 7fc3137fe700 1 -- 172.21.15.16:0/2951401554 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3040036a0 con 0x7fc31415b070 2023-10-16T12:27:53.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:53.236+0000 7fc31a6d1700 1 -- 172.21.15.16:0/2951401554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc31415b070 msgr2=0x7fc31415b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:53.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:53.236+0000 7fc31a6d1700 1 --2- 172.21.15.16:0/2951401554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc31415b070 0x7fc31415b450 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7fc3040098f0 tx=0x7fc304009c00 comp rx=0 tx=0).stop 2023-10-16T12:27:53.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:53.236+0000 7fc31a6d1700 1 -- 172.21.15.16:0/2951401554 shutdown_connections 2023-10-16T12:27:53.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:53.236+0000 7fc31a6d1700 1 --2- 172.21.15.16:0/2951401554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc31415b070 0x7fc31415b450 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:53.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:53.236+0000 7fc31a6d1700 1 -- 172.21.15.16:0/2951401554 >> 172.21.15.16:0/2951401554 conn(0x7fc3140b1f40 msgr2=0x7fc3140b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:53.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:53.236+0000 7fc31a6d1700 1 -- 172.21.15.16:0/2951401554 shutdown_connections 2023-10-16T12:27:53.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:53.236+0000 7fc31a6d1700 1 -- 172.21.15.16:0/2951401554 wait complete. 2023-10-16T12:27:53.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:53.237+0000 7fc31a6d1700 1 Processor -- start 2023-10-16T12:27:53.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:53.238+0000 7fc31a6d1700 1 -- start start 2023-10-16T12:27:53.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:53.238+0000 7fc31a6d1700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc31406e170 0x7fc314159e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:53.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:53.238+0000 7fc31a6d1700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc3140cb880 0x7fc31414abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:53.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:53.238+0000 7fc31a6d1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc31415b070 0x7fc314168270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:53.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:53.238+0000 7fc31a6d1700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc31414c3c0 con 0x7fc31415b070 2023-10-16T12:27:53.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:53.238+0000 7fc31a6d1700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3141595a0 con 0x7fc3140cb880 2023-10-16T12:27:53.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:53.238+0000 7fc31a6d1700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc314159880 con 0x7fc31406e170 2023-10-16T12:27:53.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:53.238+0000 7fc318c6e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc31415b070 0x7fc314168270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:53.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:53.238+0000 7fc318c6e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc31415b070 0x7fc314168270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42812/0 (socket says 172.21.15.16:42812) 2023-10-16T12:27:53.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:53.238+0000 7fc318c6e700 1 -- 172.21.15.16:0/1920467520 learned_addr learned my addr 172.21.15.16:0/1920467520 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:27:53.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:53.239+0000 7fc30bfff700 1 --2- 172.21.15.16:0/1920467520 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc3140cb880 0x7fc31414abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:53.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:53.239+0000 7fc313fff700 1 --2- 172.21.15.16:0/1920467520 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc31406e170 0x7fc314159e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:53.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:53.239+0000 7fc318c6e700 1 -- 172.21.15.16:0/1920467520 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc31406e170 msgr2=0x7fc314159e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:53.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:53.239+0000 7fc318c6e700 1 --2- 172.21.15.16:0/1920467520 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc31406e170 0x7fc314159e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:53.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:53.239+0000 7fc318c6e700 1 -- 172.21.15.16:0/1920467520 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc3140cb880 msgr2=0x7fc31414abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:53.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:53.239+0000 7fc318c6e700 1 --2- 172.21.15.16:0/1920467520 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc3140cb880 0x7fc31414abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:53.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:53.239+0000 7fc318c6e700 1 -- 172.21.15.16:0/1920467520 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc304009550 con 0x7fc31415b070 2023-10-16T12:27:53.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:53.239+0000 7fc30bfff700 1 --2- 172.21.15.16:0/1920467520 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc3140cb880 0x7fc31414abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:27:53.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:53.239+0000 7fc313fff700 1 --2- 172.21.15.16:0/1920467520 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc31406e170 0x7fc314159e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:27:53.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:53.239+0000 7fc318c6e700 1 --2- 172.21.15.16:0/1920467520 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc31415b070 0x7fc314168270 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7fc30c00cb90 tx=0x7fc30c00cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:53.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:53.239+0000 7fc3117fa700 1 -- 172.21.15.16:0/1920467520 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc30c003c00 con 0x7fc31415b070 2023-10-16T12:27:53.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:53.239+0000 7fc3117fa700 1 -- 172.21.15.16:0/1920467520 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc30c003d60 con 0x7fc31415b070 2023-10-16T12:27:53.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:53.239+0000 7fc31a6d1700 1 -- 172.21.15.16:0/1920467520 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc314169ee0 con 0x7fc31415b070 2023-10-16T12:27:53.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:53.239+0000 7fc3117fa700 1 -- 172.21.15.16:0/1920467520 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc30c00abb0 con 0x7fc31415b070 2023-10-16T12:27:53.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:53.239+0000 7fc31a6d1700 1 -- 172.21.15.16:0/1920467520 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc31414c050 con 0x7fc31415b070 2023-10-16T12:27:53.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:53.240+0000 7fc31a6d1700 1 -- 172.21.15.16:0/1920467520 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc31414c570 con 0x7fc31415b070 2023-10-16T12:27:53.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:53.241+0000 7fc3117fa700 1 -- 172.21.15.16:0/1920467520 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 54) v1 ==== 89084+0+0 (secure 0 0 0) 0x7fc30c00ad10 con 0x7fc31415b070 2023-10-16T12:27:53.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:53.241+0000 7fc3117fa700 1 --2- 172.21.15.16:0/1920467520 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7fc2f406ada0 0x7fc2f406d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:27:53.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:53.241+0000 7fc3117fa700 1 -- 172.21.15.16:0/1920467520 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fc30c0895b0 con 0x7fc31415b070 2023-10-16T12:27:53.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:53.241+0000 7fc313fff700 1 --2- 172.21.15.16:0/1920467520 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7fc2f406ada0 0x7fc2f406d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:27:53.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:53.242+0000 7fc313fff700 1 --2- 172.21.15.16:0/1920467520 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7fc2f406ada0 0x7fc2f406d260 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc3140b1540 tx=0x7fc304008030 comp rx=0 tx=0).ready entity=mgr.7773 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:27:53.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:53.246+0000 7fc3117fa700 1 -- 172.21.15.16:0/1920467520 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc31414c570 con 0x7fc31415b070 2023-10-16T12:27:53.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:53.477+0000 7fc31a6d1700 1 -- 172.21.15.16:0/1920467520 --> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/bad-path", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc3140728d0 con 0x7fc2f406ada0 2023-10-16T12:27:53.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:53.487+0000 7fc3117fa700 1 -- 172.21.15.16:0/1920467520 <== mgr.7773 v2:172.21.15.16:6832/1458543709 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+302 (secure 0 0 0) 0x7fc30401f070 con 0x7fc2f406ada0 2023-10-16T12:27:53.489 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:27:53.490 INFO:teuthology.orchestra.run.smithi016.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T12:27:52", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true}] 2023-10-16T12:27:53.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:53.490+0000 7fc31a6d1700 1 -- 172.21.15.16:0/1920467520 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7fc2f406ada0 msgr2=0x7fc2f406d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:53.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:53.490+0000 7fc31a6d1700 1 --2- 172.21.15.16:0/1920467520 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7fc2f406ada0 0x7fc2f406d260 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc3140b1540 tx=0x7fc304008030 comp rx=0 tx=0).stop 2023-10-16T12:27:53.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:53.490+0000 7fc31a6d1700 1 -- 172.21.15.16:0/1920467520 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc31415b070 msgr2=0x7fc314168270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:27:53.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:53.490+0000 7fc31a6d1700 1 --2- 172.21.15.16:0/1920467520 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc31415b070 0x7fc314168270 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7fc30c00cb90 tx=0x7fc30c00cf50 comp rx=0 tx=0).stop 2023-10-16T12:27:53.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:53.490+0000 7fc31a6d1700 1 -- 172.21.15.16:0/1920467520 shutdown_connections 2023-10-16T12:27:53.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:53.490+0000 7fc31a6d1700 1 --2- 172.21.15.16:0/1920467520 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7fc2f406ada0 0x7fc2f406d260 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:53.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:53.490+0000 7fc31a6d1700 1 --2- 172.21.15.16:0/1920467520 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc31415b070 0x7fc314168270 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:53.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:53.490+0000 7fc31a6d1700 1 --2- 172.21.15.16:0/1920467520 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc31406e170 0x7fc314159e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:53.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:53.490+0000 7fc31a6d1700 1 --2- 172.21.15.16:0/1920467520 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc3140cb880 0x7fc31414abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:27:53.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:53.490+0000 7fc31a6d1700 1 -- 172.21.15.16:0/1920467520 >> 172.21.15.16:0/1920467520 conn(0x7fc3140b1f40 msgr2=0x7fc3140cbd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:27:53.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:53.490+0000 7fc31a6d1700 1 -- 172.21.15.16:0/1920467520 shutdown_connections 2023-10-16T12:27:53.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:27:53.490+0000 7fc31a6d1700 1 -- 172.21.15.16:0/1920467520 wait complete. 2023-10-16T12:27:53.504 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T12:27:52", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true}] 2023-10-16T12:28:00.844 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:28:00.842+0000 7fc78cf38700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:28:00.845 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:28:00.842+0000 7fc78cf38700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:28:00.845 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:28:00.842+0000 7fc78cf38700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:28:00.845 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:28:00.842+0000 7fc78cf38700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:28:00.845 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:28:00.842+0000 7fc78cf38700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:28:20.997 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:28:21.000 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:28:51.069 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:28:51.072 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:28:53.505 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /bad-path 2023-10-16T12:28:53.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:53.645+0000 7fdd84798700 1 Processor -- start 2023-10-16T12:28:53.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:53.645+0000 7fdd84798700 1 -- start start 2023-10-16T12:28:53.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:53.646+0000 7fdd84798700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdd7c0ce6a0 0x7fdd7c06c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:53.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:53.646+0000 7fdd84798700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdd7c0d2110 0x7fdd7c06c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:53.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:53.646+0000 7fdd84798700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd7c0b20f0 con 0x7fdd7c0b2620 2023-10-16T12:28:53.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:53.646+0000 7fdd84798700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd7c0af1f0 con 0x7fdd7c0ce6a0 2023-10-16T12:28:53.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:53.646+0000 7fdd84798700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd7c0d03d0 con 0x7fdd7c0d2110 2023-10-16T12:28:53.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:53.646+0000 7fdd82534700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdd7c0ce6a0 0x7fdd7c06c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:28:53.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:53.646+0000 7fdd82534700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdd7c0ce6a0 0x7fdd7c06c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42848/0 (socket says 172.21.15.16:42848) 2023-10-16T12:28:53.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:53.646+0000 7fdd82534700 1 -- 172.21.15.16:0/3764793494 learned_addr learned my addr 172.21.15.16:0/3764793494 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:28:53.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:53.646+0000 7fdd81d33700 1 --2- 172.21.15.16:0/3764793494 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdd7c0d2110 0x7fdd7c06c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:28:53.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:53.646+0000 7fdd82534700 1 -- 172.21.15.16:0/3764793494 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdd7c0d2110 msgr2=0x7fdd7c06c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:28:53.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:53.646+0000 7fdd82534700 1 --2- 172.21.15.16:0/3764793494 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdd7c0d2110 0x7fdd7c06c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:53.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:53.646+0000 7fdd82534700 1 -- 172.21.15.16:0/3764793494 >> v1:172.21.15.49:6789/0 conn(0x7fdd7c0b2620 legacy=0x7fdd7c0b0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:28:53.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:53.646+0000 7fdd82534700 1 -- 172.21.15.16:0/3764793494 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd7c0b2340 con 0x7fdd7c0ce6a0 2023-10-16T12:28:53.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:53.646+0000 7fdd82534700 1 --2- 172.21.15.16:0/3764793494 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd7c0ce6a0 0x7fdd7c06c300 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7fdd780098f0 tx=0x7fdd78009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1c0d2d06da119d3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:28:53.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:53.647+0000 7fdd81532700 1 -- 172.21.15.16:0/3764793494 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd78004230 con 0x7fdd7c0ce6a0 2023-10-16T12:28:53.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:53.647+0000 7fdd81532700 1 -- 172.21.15.16:0/3764793494 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd78004390 con 0x7fdd7c0ce6a0 2023-10-16T12:28:53.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:53.647+0000 7fdd81532700 1 -- 172.21.15.16:0/3764793494 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd780036a0 con 0x7fdd7c0ce6a0 2023-10-16T12:28:53.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:53.647+0000 7fdd84798700 1 -- 172.21.15.16:0/3764793494 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd7c0ce6a0 msgr2=0x7fdd7c06c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:28:53.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:53.647+0000 7fdd84798700 1 --2- 172.21.15.16:0/3764793494 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd7c0ce6a0 0x7fdd7c06c300 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7fdd780098f0 tx=0x7fdd78009c00 comp rx=0 tx=0).stop 2023-10-16T12:28:53.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:53.647+0000 7fdd84798700 1 -- 172.21.15.16:0/3764793494 shutdown_connections 2023-10-16T12:28:53.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:53.647+0000 7fdd84798700 1 --2- 172.21.15.16:0/3764793494 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd7c0ce6a0 0x7fdd7c06c300 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:53.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:53.647+0000 7fdd84798700 1 --2- 172.21.15.16:0/3764793494 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdd7c0d2110 0x7fdd7c06c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:53.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:53.647+0000 7fdd84798700 1 -- 172.21.15.16:0/3764793494 >> 172.21.15.16:0/3764793494 conn(0x7fdd7c0d15f0 msgr2=0x7fdd7c0d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:28:53.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:53.647+0000 7fdd84798700 1 -- 172.21.15.16:0/3764793494 shutdown_connections 2023-10-16T12:28:53.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:53.647+0000 7fdd84798700 1 -- 172.21.15.16:0/3764793494 wait complete. 2023-10-16T12:28:53.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:53.649+0000 7fdd84798700 1 Processor -- start 2023-10-16T12:28:53.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:53.649+0000 7fdd84798700 1 -- start start 2023-10-16T12:28:53.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:53.649+0000 7fdd84798700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdd7c0b2620 0x7fdd7c15af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:53.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:53.649+0000 7fdd84798700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdd7c0ce6a0 0x7fdd7c0b6500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:53.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:53.649+0000 7fdd84798700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd7c0d2110 0x7fdd7c0b6a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:53.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:53.649+0000 7fdd84798700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd7c15b490 con 0x7fdd7c0d2110 2023-10-16T12:28:53.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:53.649+0000 7fdd84798700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd7c15c180 con 0x7fdd7c0b2620 2023-10-16T12:28:53.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:53.649+0000 7fdd84798700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd7c0b5c90 con 0x7fdd7c0ce6a0 2023-10-16T12:28:53.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:53.649+0000 7fdd82d35700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd7c0d2110 0x7fdd7c0b6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:28:53.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:53.650+0000 7fdd82d35700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd7c0d2110 0x7fdd7c0b6a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42858/0 (socket says 172.21.15.16:42858) 2023-10-16T12:28:53.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:53.650+0000 7fdd82d35700 1 -- 172.21.15.16:0/2167269722 learned_addr learned my addr 172.21.15.16:0/2167269722 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:28:53.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:53.650+0000 7fdd82534700 1 --2- 172.21.15.16:0/2167269722 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdd7c0b2620 0x7fdd7c15af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:28:53.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:53.650+0000 7fdd81d33700 1 --2- 172.21.15.16:0/2167269722 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdd7c0ce6a0 0x7fdd7c0b6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:28:53.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:53.650+0000 7fdd82d35700 1 -- 172.21.15.16:0/2167269722 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdd7c0ce6a0 msgr2=0x7fdd7c0b6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:28:53.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:53.650+0000 7fdd82d35700 1 --2- 172.21.15.16:0/2167269722 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdd7c0ce6a0 0x7fdd7c0b6500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:53.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:53.650+0000 7fdd82d35700 1 -- 172.21.15.16:0/2167269722 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdd7c0b2620 msgr2=0x7fdd7c15af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:28:53.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:53.650+0000 7fdd82d35700 1 --2- 172.21.15.16:0/2167269722 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdd7c0b2620 0x7fdd7c15af50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:53.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:53.650+0000 7fdd82d35700 1 -- 172.21.15.16:0/2167269722 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd78009550 con 0x7fdd7c0d2110 2023-10-16T12:28:53.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:53.650+0000 7fdd82d35700 1 --2- 172.21.15.16:0/2167269722 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd7c0d2110 0x7fdd7c0b6a40 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7fdd7400cc10 tx=0x7fdd7400cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:28:53.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:53.650+0000 7fdd72ffd700 1 -- 172.21.15.16:0/2167269722 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd7400aa60 con 0x7fdd7c0d2110 2023-10-16T12:28:53.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:53.650+0000 7fdd72ffd700 1 -- 172.21.15.16:0/2167269722 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd7400abc0 con 0x7fdd7c0d2110 2023-10-16T12:28:53.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:53.650+0000 7fdd84798700 1 -- 172.21.15.16:0/2167269722 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd7c0b5110 con 0x7fdd7c0d2110 2023-10-16T12:28:53.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:53.650+0000 7fdd72ffd700 1 -- 172.21.15.16:0/2167269722 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd74012420 con 0x7fdd7c0d2110 2023-10-16T12:28:53.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:53.650+0000 7fdd84798700 1 -- 172.21.15.16:0/2167269722 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd7c0b56d0 con 0x7fdd7c0d2110 2023-10-16T12:28:53.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:53.651+0000 7fdd84798700 1 -- 172.21.15.16:0/2167269722 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd7c04ebb0 con 0x7fdd7c0d2110 2023-10-16T12:28:53.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:53.652+0000 7fdd72ffd700 1 -- 172.21.15.16:0/2167269722 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 57) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fdd74012580 con 0x7fdd7c0d2110 2023-10-16T12:28:53.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:53.652+0000 7fdd72ffd700 1 --2- 172.21.15.16:0/2167269722 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7fdd6006b190 0x7fdd6006d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:53.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:53.652+0000 7fdd72ffd700 1 -- 172.21.15.16:0/2167269722 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 6517+0+0 (secure 0 0 0) 0x7fdd7408a0b0 con 0x7fdd7c0d2110 2023-10-16T12:28:53.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:53.652+0000 7fdd82534700 1 --2- 172.21.15.16:0/2167269722 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7fdd6006b190 0x7fdd6006d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:28:53.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:53.653+0000 7fdd82534700 1 --2- 172.21.15.16:0/2167269722 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7fdd6006b190 0x7fdd6006d650 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fdd78009130 tx=0x7fdd78008030 comp rx=0 tx=0).ready entity=mgr.7773 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:28:53.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:53.657+0000 7fdd72ffd700 1 -- 172.21.15.16:0/2167269722 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdd74054ad0 con 0x7fdd7c0d2110 2023-10-16T12:28:53.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:53.888+0000 7fdd84798700 1 -- 172.21.15.16:0/2167269722 --> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/bad-path", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fdd7c06bde0 con 0x7fdd6006b190 2023-10-16T12:28:53.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:53.895+0000 7fdd72ffd700 1 -- 172.21.15.16:0/2167269722 <== mgr.7773 v2:172.21.15.16:6832/1458543709 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+303 (secure 0 0 0) 0x7fdd780036a0 con 0x7fdd6006b190 2023-10-16T12:28:53.897 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:28:53.897 INFO:teuthology.orchestra.run.smithi016.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T12:27:52", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": false}] 2023-10-16T12:28:53.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:53.897+0000 7fdd84798700 1 -- 172.21.15.16:0/2167269722 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7fdd6006b190 msgr2=0x7fdd6006d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:28:53.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:53.897+0000 7fdd84798700 1 --2- 172.21.15.16:0/2167269722 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7fdd6006b190 0x7fdd6006d650 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fdd78009130 tx=0x7fdd78008030 comp rx=0 tx=0).stop 2023-10-16T12:28:53.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:53.897+0000 7fdd84798700 1 -- 172.21.15.16:0/2167269722 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd7c0d2110 msgr2=0x7fdd7c0b6a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:28:53.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:53.897+0000 7fdd84798700 1 --2- 172.21.15.16:0/2167269722 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd7c0d2110 0x7fdd7c0b6a40 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7fdd7400cc10 tx=0x7fdd7400cf20 comp rx=0 tx=0).stop 2023-10-16T12:28:53.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:53.897+0000 7fdd84798700 1 -- 172.21.15.16:0/2167269722 shutdown_connections 2023-10-16T12:28:53.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:53.897+0000 7fdd84798700 1 --2- 172.21.15.16:0/2167269722 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7fdd6006b190 0x7fdd6006d650 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:53.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:53.897+0000 7fdd84798700 1 --2- 172.21.15.16:0/2167269722 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd7c0d2110 0x7fdd7c0b6a40 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:53.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:53.897+0000 7fdd84798700 1 --2- 172.21.15.16:0/2167269722 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdd7c0b2620 0x7fdd7c15af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:53.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:53.897+0000 7fdd84798700 1 --2- 172.21.15.16:0/2167269722 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdd7c0ce6a0 0x7fdd7c0b6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:53.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:53.897+0000 7fdd84798700 1 -- 172.21.15.16:0/2167269722 >> 172.21.15.16:0/2167269722 conn(0x7fdd7c0d15f0 msgr2=0x7fdd7c0afcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:28:53.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:53.897+0000 7fdd84798700 1 -- 172.21.15.16:0/2167269722 shutdown_connections 2023-10-16T12:28:53.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:53.897+0000 7fdd84798700 1 -- 172.21.15.16:0/2167269722 wait complete. 2023-10-16T12:28:53.911 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T12:27:52", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": false}] 2023-10-16T12:28:53.912 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /bad-path 2023-10-16T12:28:54.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.037+0000 7fba8ccb4700 1 Processor -- start 2023-10-16T12:28:54.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.037+0000 7fba8ccb4700 1 -- start start 2023-10-16T12:28:54.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.038+0000 7fba8ccb4700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fba88072b30 0x7fba88071a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:54.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.038+0000 7fba8ccb4700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fba8815bb40 0x7fba8806e310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:54.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.038+0000 7fba8ccb4700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba8815c040 con 0x7fba880caa40 2023-10-16T12:28:54.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.038+0000 7fba8ccb4700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba8815c320 con 0x7fba88072b30 2023-10-16T12:28:54.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.038+0000 7fba8ccb4700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba8815c600 con 0x7fba8815bb40 2023-10-16T12:28:54.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.038+0000 7fba8659c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fba88072b30 0x7fba88071a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:28:54.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.038+0000 7fba86d9d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fba880caa40 0x7fba880ccf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:60368/0 (socket says 172.21.15.16:60368) 2023-10-16T12:28:54.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.038+0000 7fba86d9d700 1 -- 172.21.15.16:0/1107104818 learned_addr learned my addr 172.21.15.16:0/1107104818 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:28:54.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.038+0000 7fba85d9b700 1 --2- 172.21.15.16:0/1107104818 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fba8815bb40 0x7fba8806e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:28:54.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.039+0000 7fba8659c700 1 -- 172.21.15.16:0/1107104818 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fba8815bb40 msgr2=0x7fba8806e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:28:54.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.039+0000 7fba8659c700 1 --2- 172.21.15.16:0/1107104818 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fba8815bb40 0x7fba8806e310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:54.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.039+0000 7fba8659c700 1 -- 172.21.15.16:0/1107104818 >> v1:172.21.15.16:6789/0 conn(0x7fba880caa40 legacy=0x7fba880ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:28:54.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.039+0000 7fba8559a700 1 -- 172.21.15.16:0/1107104818 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1263658427 0 0) 0x7fba8815c040 con 0x7fba880caa40 2023-10-16T12:28:54.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.039+0000 7fba8659c700 1 -- 172.21.15.16:0/1107104818 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba8815c8e0 con 0x7fba88072b30 2023-10-16T12:28:54.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.039+0000 7fba8659c700 1 --2- 172.21.15.16:0/1107104818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba88072b30 0x7fba88071a70 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7fba78009840 tx=0x7fba78009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b32d6e318a87f3ca server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:28:54.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.039+0000 7fba8559a700 1 -- 172.21.15.16:0/1107104818 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba78011830 con 0x7fba88072b30 2023-10-16T12:28:54.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.039+0000 7fba8559a700 1 -- 172.21.15.16:0/1107104818 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba78011990 con 0x7fba88072b30 2023-10-16T12:28:54.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.039+0000 7fba8559a700 1 -- 172.21.15.16:0/1107104818 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba78011d20 con 0x7fba88072b30 2023-10-16T12:28:54.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.039+0000 7fba8ccb4700 1 -- 172.21.15.16:0/1107104818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba88072b30 msgr2=0x7fba88071a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:28:54.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.039+0000 7fba8ccb4700 1 --2- 172.21.15.16:0/1107104818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba88072b30 0x7fba88071a70 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7fba78009840 tx=0x7fba78009b50 comp rx=0 tx=0).stop 2023-10-16T12:28:54.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.040+0000 7fba8ccb4700 1 -- 172.21.15.16:0/1107104818 shutdown_connections 2023-10-16T12:28:54.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.040+0000 7fba8ccb4700 1 --2- 172.21.15.16:0/1107104818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba88072b30 0x7fba88071a70 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:54.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.040+0000 7fba8ccb4700 1 --2- 172.21.15.16:0/1107104818 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fba8815bb40 0x7fba8806e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:54.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.040+0000 7fba8ccb4700 1 -- 172.21.15.16:0/1107104818 >> 172.21.15.16:0/1107104818 conn(0x7fba880b1f40 msgr2=0x7fba880b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:28:54.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.040+0000 7fba8ccb4700 1 -- 172.21.15.16:0/1107104818 shutdown_connections 2023-10-16T12:28:54.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.040+0000 7fba8ccb4700 1 -- 172.21.15.16:0/1107104818 wait complete. 2023-10-16T12:28:54.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.041+0000 7fba8ccb4700 1 Processor -- start 2023-10-16T12:28:54.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.041+0000 7fba8ccb4700 1 -- start start 2023-10-16T12:28:54.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.042+0000 7fba8ccb4700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fba88072b30 0x7fba88156170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:54.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.042+0000 7fba8ccb4700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fba880caa40 0x7fba88152180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:54.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.042+0000 7fba8ccb4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba8815bb40 0x7fba881542b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:54.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.042+0000 7fba8ccb4700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba881566b0 con 0x7fba8815bb40 2023-10-16T12:28:54.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.042+0000 7fba8ccb4700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba881526c0 con 0x7fba880caa40 2023-10-16T12:28:54.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.042+0000 7fba8ccb4700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba88156fb0 con 0x7fba88072b30 2023-10-16T12:28:54.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.042+0000 7fba86d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba8815bb40 0x7fba881542b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:28:54.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.042+0000 7fba86d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba8815bb40 0x7fba881542b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42870/0 (socket says 172.21.15.16:42870) 2023-10-16T12:28:54.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.042+0000 7fba86d9d700 1 -- 172.21.15.16:0/375662949 learned_addr learned my addr 172.21.15.16:0/375662949 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:28:54.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.042+0000 7fba8659c700 1 --2- 172.21.15.16:0/375662949 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fba88072b30 0x7fba88156170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:28:54.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.042+0000 7fba85d9b700 1 --2- 172.21.15.16:0/375662949 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fba880caa40 0x7fba88152180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:28:54.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.042+0000 7fba86d9d700 1 -- 172.21.15.16:0/375662949 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fba88072b30 msgr2=0x7fba88156170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:28:54.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.042+0000 7fba86d9d700 1 --2- 172.21.15.16:0/375662949 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fba88072b30 0x7fba88156170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:54.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.042+0000 7fba86d9d700 1 -- 172.21.15.16:0/375662949 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fba880caa40 msgr2=0x7fba88152180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:28:54.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.042+0000 7fba86d9d700 1 --2- 172.21.15.16:0/375662949 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fba880caa40 0x7fba88152180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:54.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.042+0000 7fba86d9d700 1 -- 172.21.15.16:0/375662949 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba780094a0 con 0x7fba8815bb40 2023-10-16T12:28:54.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.042+0000 7fba8659c700 1 --2- 172.21.15.16:0/375662949 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fba88072b30 0x7fba88156170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:28:54.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.042+0000 7fba86d9d700 1 --2- 172.21.15.16:0/375662949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba8815bb40 0x7fba881542b0 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7fba80002230 tx=0x7fba8000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:28:54.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.042+0000 7fba76ffd700 1 -- 172.21.15.16:0/375662949 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba8000ac20 con 0x7fba8815bb40 2023-10-16T12:28:54.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.042+0000 7fba76ffd700 1 -- 172.21.15.16:0/375662949 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba8000ada0 con 0x7fba8815bb40 2023-10-16T12:28:54.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.042+0000 7fba8ccb4700 1 -- 172.21.15.16:0/375662949 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba88157290 con 0x7fba8815bb40 2023-10-16T12:28:54.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.042+0000 7fba76ffd700 1 -- 172.21.15.16:0/375662949 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba80007310 con 0x7fba8815bb40 2023-10-16T12:28:54.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.042+0000 7fba8ccb4700 1 -- 172.21.15.16:0/375662949 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba880b4860 con 0x7fba8815bb40 2023-10-16T12:28:54.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.043+0000 7fba8ccb4700 1 -- 172.21.15.16:0/375662949 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba880b4a80 con 0x7fba8815bb40 2023-10-16T12:28:54.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.044+0000 7fba76ffd700 1 -- 172.21.15.16:0/375662949 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 57) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fba80007470 con 0x7fba8815bb40 2023-10-16T12:28:54.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.045+0000 7fba76ffd700 1 --2- 172.21.15.16:0/375662949 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7fba6806b190 0x7fba6806d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:54.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.045+0000 7fba76ffd700 1 -- 172.21.15.16:0/375662949 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 6517+0+0 (secure 0 0 0) 0x7fba80089f10 con 0x7fba8815bb40 2023-10-16T12:28:54.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.045+0000 7fba8659c700 1 --2- 172.21.15.16:0/375662949 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7fba6806b190 0x7fba6806d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:28:54.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.045+0000 7fba8659c700 1 --2- 172.21.15.16:0/375662949 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7fba6806b190 0x7fba6806d650 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fba880b0a70 tx=0x7fba78008030 comp rx=0 tx=0).ready entity=mgr.7773 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:28:54.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.049+0000 7fba76ffd700 1 -- 172.21.15.16:0/375662949 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fba80054a10 con 0x7fba8815bb40 2023-10-16T12:28:54.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.280+0000 7fba8ccb4700 1 -- 172.21.15.16:0/375662949 --> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/bad-path", "target": ["mon-mgr", ""]}) v1 -- 0x7fba880cc9f0 con 0x7fba6806b190 2023-10-16T12:28:54.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.350+0000 7fba76ffd700 1 -- 172.21.15.16:0/375662949 <== mgr.7773 v2:172.21.15.16:6832/1458543709 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+36 (secure 0 0 0) 0x7fba78011b00 con 0x7fba6806b190 2023-10-16T12:28:54.352 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule removed for path /bad-path 2023-10-16T12:28:54.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.352+0000 7fba8ccb4700 1 -- 172.21.15.16:0/375662949 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7fba6806b190 msgr2=0x7fba6806d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:28:54.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.352+0000 7fba8ccb4700 1 --2- 172.21.15.16:0/375662949 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7fba6806b190 0x7fba6806d650 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fba880b0a70 tx=0x7fba78008030 comp rx=0 tx=0).stop 2023-10-16T12:28:54.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.352+0000 7fba8ccb4700 1 -- 172.21.15.16:0/375662949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba8815bb40 msgr2=0x7fba881542b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:28:54.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.352+0000 7fba8ccb4700 1 --2- 172.21.15.16:0/375662949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba8815bb40 0x7fba881542b0 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7fba80002230 tx=0x7fba8000a060 comp rx=0 tx=0).stop 2023-10-16T12:28:54.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.352+0000 7fba8ccb4700 1 -- 172.21.15.16:0/375662949 shutdown_connections 2023-10-16T12:28:54.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.352+0000 7fba8ccb4700 1 --2- 172.21.15.16:0/375662949 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7fba6806b190 0x7fba6806d650 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:54.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.352+0000 7fba8ccb4700 1 --2- 172.21.15.16:0/375662949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba8815bb40 0x7fba881542b0 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:54.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.352+0000 7fba8ccb4700 1 --2- 172.21.15.16:0/375662949 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fba88072b30 0x7fba88156170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:54.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.352+0000 7fba8ccb4700 1 --2- 172.21.15.16:0/375662949 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fba880caa40 0x7fba88152180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:54.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.352+0000 7fba8ccb4700 1 -- 172.21.15.16:0/375662949 >> 172.21.15.16:0/375662949 conn(0x7fba880b1f40 msgr2=0x7fba880b14a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:28:54.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.352+0000 7fba8ccb4700 1 -- 172.21.15.16:0/375662949 shutdown_connections 2023-10-16T12:28:54.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.352+0000 7fba8ccb4700 1 -- 172.21.15.16:0/375662949 wait complete. 2023-10-16T12:28:54.366 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /bad-path 2023-10-16T12:28:54.366 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-16T12:28:54.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.492+0000 7f979daf4700 1 Processor -- start 2023-10-16T12:28:54.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.492+0000 7f979daf4700 1 -- start start 2023-10-16T12:28:54.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.493+0000 7f979daf4700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f97980d2110 0x7f97980ae1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:54.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.493+0000 7f979daf4700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97980b20f0 con 0x7f97980ce6a0 2023-10-16T12:28:54.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.493+0000 7f979daf4700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97980d03d0 con 0x7f979806ca10 2023-10-16T12:28:54.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.493+0000 7f979daf4700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97980b2340 con 0x7f97980d2110 2023-10-16T12:28:54.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.493+0000 7f978ffff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f97980d2110 0x7f97980ae1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:28:54.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.493+0000 7f97977fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f97980ce6a0 0x7f979806c300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:60380/0 (socket says 172.21.15.16:60380) 2023-10-16T12:28:54.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.493+0000 7f97977fe700 1 -- 172.21.15.16:0/1926696213 learned_addr learned my addr 172.21.15.16:0/1926696213 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:28:54.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.493+0000 7f978ffff700 1 -- 172.21.15.16:0/1926696213 >> v1:172.21.15.49:6789/0 conn(0x7f979806ca10 legacy=0x7f97980ae720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:28:54.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.493+0000 7f978ffff700 1 -- 172.21.15.16:0/1926696213 >> v1:172.21.15.16:6789/0 conn(0x7f97980ce6a0 legacy=0x7f979806c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:28:54.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.493+0000 7f978ffff700 1 -- 172.21.15.16:0/1926696213 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f979806bb00 con 0x7f97980d2110 2023-10-16T12:28:54.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.493+0000 7f978ffff700 1 --2- 172.21.15.16:0/1926696213 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97980d2110 0x7f97980ae1e0 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f9780009840 tx=0x7f9780009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5b89aec7eaaafe server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:28:54.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.493+0000 7f9796ffd700 1 -- 172.21.15.16:0/1926696213 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9780004230 con 0x7f97980d2110 2023-10-16T12:28:54.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.493+0000 7f9796ffd700 1 -- 172.21.15.16:0/1926696213 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9780004390 con 0x7f97980d2110 2023-10-16T12:28:54.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.493+0000 7f9796ffd700 1 -- 172.21.15.16:0/1926696213 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97800036a0 con 0x7f97980d2110 2023-10-16T12:28:54.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.494+0000 7f979daf4700 1 -- 172.21.15.16:0/1926696213 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97980d2110 msgr2=0x7f97980ae1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:28:54.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.494+0000 7f979daf4700 1 --2- 172.21.15.16:0/1926696213 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97980d2110 0x7f97980ae1e0 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f9780009840 tx=0x7f9780009b50 comp rx=0 tx=0).stop 2023-10-16T12:28:54.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.494+0000 7f979daf4700 1 -- 172.21.15.16:0/1926696213 shutdown_connections 2023-10-16T12:28:54.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.494+0000 7f979daf4700 1 --2- 172.21.15.16:0/1926696213 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97980d2110 0x7f97980ae1e0 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:54.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.494+0000 7f979daf4700 1 -- 172.21.15.16:0/1926696213 >> 172.21.15.16:0/1926696213 conn(0x7f97980d15f0 msgr2=0x7f97980d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:28:54.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.494+0000 7f979daf4700 1 -- 172.21.15.16:0/1926696213 shutdown_connections 2023-10-16T12:28:54.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.494+0000 7f979daf4700 1 -- 172.21.15.16:0/1926696213 wait complete. 2023-10-16T12:28:54.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.496+0000 7f979daf4700 1 Processor -- start 2023-10-16T12:28:54.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.496+0000 7f979daf4700 1 -- start start 2023-10-16T12:28:54.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.497+0000 7f979daf4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f979806ca10 0x7f97980ae480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:54.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.497+0000 7f979daf4700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f97980ce6a0 0x7f97980b4170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:54.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.497+0000 7f979daf4700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f97980d2110 0x7f97980b46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:54.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.497+0000 7f979daf4700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97980ae9c0 con 0x7f979806ca10 2023-10-16T12:28:54.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.497+0000 7f979daf4700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97980aec20 con 0x7f97980d2110 2023-10-16T12:28:54.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.497+0000 7f979daf4700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97980aef00 con 0x7f97980ce6a0 2023-10-16T12:28:54.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.497+0000 7f97977fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f979806ca10 0x7f97980ae480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:28:54.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.497+0000 7f97977fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f979806ca10 0x7f97980ae480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42880/0 (socket says 172.21.15.16:42880) 2023-10-16T12:28:54.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.497+0000 7f97977fe700 1 -- 172.21.15.16:0/2671178017 learned_addr learned my addr 172.21.15.16:0/2671178017 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:28:54.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.497+0000 7f97977fe700 1 -- 172.21.15.16:0/2671178017 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f97980ce6a0 msgr2=0x7f97980b4170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:28:54.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.497+0000 7f97977fe700 1 --2- 172.21.15.16:0/2671178017 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f97980ce6a0 0x7f97980b4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:54.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.497+0000 7f97977fe700 1 -- 172.21.15.16:0/2671178017 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f97980d2110 msgr2=0x7f97980b46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:28:54.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.497+0000 7f9797fff700 1 --2- 172.21.15.16:0/2671178017 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f97980d2110 0x7f97980b46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:28:54.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.497+0000 7f97977fe700 1 --2- 172.21.15.16:0/2671178017 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f97980d2110 0x7f97980b46b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:54.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.497+0000 7f97977fe700 1 -- 172.21.15.16:0/2671178017 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97800094a0 con 0x7f979806ca10 2023-10-16T12:28:54.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.497+0000 7f97977fe700 1 --2- 172.21.15.16:0/2671178017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f979806ca10 0x7f97980ae480 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f978800c490 tx=0x7f978800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:28:54.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.497+0000 7f9794ff9700 1 -- 172.21.15.16:0/2671178017 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9788005d70 con 0x7f979806ca10 2023-10-16T12:28:54.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.497+0000 7f9794ff9700 1 -- 172.21.15.16:0/2671178017 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97880070d0 con 0x7f979806ca10 2023-10-16T12:28:54.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.497+0000 7f979daf4700 1 -- 172.21.15.16:0/2671178017 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97980af1e0 con 0x7f979806ca10 2023-10-16T12:28:54.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.497+0000 7f9794ff9700 1 -- 172.21.15.16:0/2671178017 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9788007460 con 0x7f979806ca10 2023-10-16T12:28:54.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.497+0000 7f979daf4700 1 -- 172.21.15.16:0/2671178017 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97980af7a0 con 0x7f979806ca10 2023-10-16T12:28:54.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.498+0000 7f979daf4700 1 -- 172.21.15.16:0/2671178017 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f979804ebb0 con 0x7f979806ca10 2023-10-16T12:28:54.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.500+0000 7f9794ff9700 1 -- 172.21.15.16:0/2671178017 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 57) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f97880075c0 con 0x7f979806ca10 2023-10-16T12:28:54.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.500+0000 7f9794ff9700 1 --2- 172.21.15.16:0/2671178017 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f977806b160 0x7f977806d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:54.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.500+0000 7f9794ff9700 1 -- 172.21.15.16:0/2671178017 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f9788089630 con 0x7f979806ca10 2023-10-16T12:28:54.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.500+0000 7f978ffff700 1 --2- 172.21.15.16:0/2671178017 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f977806b160 0x7f977806d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:28:54.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.501+0000 7f978ffff700 1 --2- 172.21.15.16:0/2671178017 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f977806b160 0x7f977806d620 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9780009840 tx=0x7f9780013040 comp rx=0 tx=0).ready entity=mgr.7773 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:28:54.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.504+0000 7f9794ff9700 1 -- 172.21.15.16:0/2671178017 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9788054130 con 0x7f979806ca10 2023-10-16T12:28:54.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:54.768+0000 7f979daf4700 1 -- 172.21.15.16:0/2671178017 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f97980b0cb0 con 0x7f979806ca10 2023-10-16T12:28:55.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.296+0000 7f9794ff9700 1 -- 172.21.15.16:0/2671178017 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v58) v1 ==== 93+0+0 (secure 0 0 0) 0x7f9788057750 con 0x7f979806ca10 2023-10-16T12:28:55.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.300+0000 7f979daf4700 1 -- 172.21.15.16:0/2671178017 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f977806b160 msgr2=0x7f977806d620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:28:55.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.300+0000 7f979daf4700 1 --2- 172.21.15.16:0/2671178017 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f977806b160 0x7f977806d620 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9780009840 tx=0x7f9780013040 comp rx=0 tx=0).stop 2023-10-16T12:28:55.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.301+0000 7f979daf4700 1 -- 172.21.15.16:0/2671178017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f979806ca10 msgr2=0x7f97980ae480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:28:55.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.301+0000 7f979daf4700 1 --2- 172.21.15.16:0/2671178017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f979806ca10 0x7f97980ae480 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f978800c490 tx=0x7f978800c850 comp rx=0 tx=0).stop 2023-10-16T12:28:55.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.302+0000 7f979daf4700 1 -- 172.21.15.16:0/2671178017 shutdown_connections 2023-10-16T12:28:55.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.302+0000 7f979daf4700 1 --2- 172.21.15.16:0/2671178017 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f977806b160 0x7f977806d620 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:55.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.302+0000 7f979daf4700 1 --2- 172.21.15.16:0/2671178017 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f97980d2110 0x7f97980b46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:55.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.302+0000 7f979daf4700 1 --2- 172.21.15.16:0/2671178017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f979806ca10 0x7f97980ae480 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:55.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.302+0000 7f979daf4700 1 --2- 172.21.15.16:0/2671178017 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f97980ce6a0 0x7f97980b4170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:55.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.302+0000 7f979daf4700 1 -- 172.21.15.16:0/2671178017 >> 172.21.15.16:0/2671178017 conn(0x7f97980d15f0 msgr2=0x7f97980cf880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:28:55.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.302+0000 7f979daf4700 1 -- 172.21.15.16:0/2671178017 shutdown_connections 2023-10-16T12:28:55.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.302+0000 7f979daf4700 1 -- 172.21.15.16:0/2671178017 wait complete. 2023-10-16T12:28:55.321 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T12:28:55.322 DEBUG:teuthology.orchestra.run.smithi016:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T12:28:55.367 DEBUG:teuthology.orchestra.run.smithi049:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T12:28:55.407 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T12:28:55.407 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:28:55.408 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:28:55.426 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:28:55.426 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:28:55.452 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:28:55.450+0000 7fa4e0be9200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T12:28:55.458 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:28:55.456+0000 7f9831e6b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T12:28:55.476 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:28:55.521 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:28:55.520+0000 7fa4e0be9200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T12:28:55.527 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:28:55.525+0000 7f9831e6b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T12:28:55.597 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:28:55.594+0000 7fa4e0be9200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T12:28:55.605 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:28:55.603+0000 7f9831e6b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T12:28:55.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.642+0000 7ff0a880e700 1 Processor -- start 2023-10-16T12:28:55.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.642+0000 7ff0a880e700 1 -- start start 2023-10-16T12:28:55.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.643+0000 7ff0a880e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff0a40d0750 0x7ff0a4168450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:55.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.643+0000 7ff0a880e700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0a406ba00 con 0x7ff0a40d41c0 2023-10-16T12:28:55.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.643+0000 7ff0a880e700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0a40d26d0 con 0x7ff0a415d8d0 2023-10-16T12:28:55.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.643+0000 7ff0a880e700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0a40d23f0 con 0x7ff0a40d0750 2023-10-16T12:28:55.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.643+0000 7ff0a1d9b700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff0a40d41c0 0x7ff0a40d0040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:60454/0 (socket says 172.21.15.16:60454) 2023-10-16T12:28:55.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.643+0000 7ff0a1d9b700 1 -- 172.21.15.16:0/973618712 learned_addr learned my addr 172.21.15.16:0/973618712 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:28:55.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.643+0000 7ff0a2d9d700 1 --2- 172.21.15.16:0/973618712 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff0a40d0750 0x7ff0a4168450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:28:55.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.643+0000 7ff0a159a700 1 -- 172.21.15.16:0/973618712 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2606828320 0 0) 0x7ff0a406ba00 con 0x7ff0a40d41c0 2023-10-16T12:28:55.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.643+0000 7ff0a159a700 1 -- 172.21.15.16:0/973618712 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff08c0035b0 con 0x7ff0a40d41c0 2023-10-16T12:28:55.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.643+0000 7ff0a159a700 1 -- 172.21.15.16:0/973618712 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 33069205 0 0) 0x7ff0a40d26d0 con 0x7ff0a415d8d0 2023-10-16T12:28:55.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.643+0000 7ff0a159a700 1 -- 172.21.15.16:0/973618712 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0a406ba00 con 0x7ff0a415d8d0 2023-10-16T12:28:55.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.643+0000 7ff0a2d9d700 1 -- 172.21.15.16:0/973618712 >> v1:172.21.15.49:6789/0 conn(0x7ff0a415d8d0 legacy=0x7ff0a40d1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:28:55.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.643+0000 7ff0a2d9d700 1 -- 172.21.15.16:0/973618712 >> v1:172.21.15.16:6789/0 conn(0x7ff0a40d41c0 legacy=0x7ff0a40d0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:28:55.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.643+0000 7ff0a2d9d700 1 -- 172.21.15.16:0/973618712 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0a406bca0 con 0x7ff0a40d0750 2023-10-16T12:28:55.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.644+0000 7ff0a2d9d700 1 --2- 172.21.15.16:0/973618712 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff0a40d0750 0x7ff0a4168450 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ff09c009dc0 tx=0x7ff09c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=454084fa08b3abe6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:28:55.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.644+0000 7ff0a159a700 1 -- 172.21.15.16:0/973618712 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff09c00e070 con 0x7ff0a40d0750 2023-10-16T12:28:55.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.644+0000 7ff0a159a700 1 -- 172.21.15.16:0/973618712 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff09c00bb20 con 0x7ff0a40d0750 2023-10-16T12:28:55.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.644+0000 7ff0a159a700 1 -- 172.21.15.16:0/973618712 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff09c00be90 con 0x7ff0a40d0750 2023-10-16T12:28:55.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.644+0000 7ff0a880e700 1 -- 172.21.15.16:0/973618712 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff0a40d0750 msgr2=0x7ff0a4168450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:28:55.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.644+0000 7ff0a880e700 1 --2- 172.21.15.16:0/973618712 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff0a40d0750 0x7ff0a4168450 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ff09c009dc0 tx=0x7ff09c00b330 comp rx=0 tx=0).stop 2023-10-16T12:28:55.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.645+0000 7ff0a880e700 1 -- 172.21.15.16:0/973618712 shutdown_connections 2023-10-16T12:28:55.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.645+0000 7ff0a880e700 1 --2- 172.21.15.16:0/973618712 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff0a40d0750 0x7ff0a4168450 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:55.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.645+0000 7ff0a880e700 1 -- 172.21.15.16:0/973618712 >> 172.21.15.16:0/973618712 conn(0x7ff0a40d3630 msgr2=0x7ff0a40d3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:28:55.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.645+0000 7ff0a880e700 1 -- 172.21.15.16:0/973618712 shutdown_connections 2023-10-16T12:28:55.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.645+0000 7ff0a880e700 1 -- 172.21.15.16:0/973618712 wait complete. 2023-10-16T12:28:55.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.646+0000 7ff0a880e700 1 Processor -- start 2023-10-16T12:28:55.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.646+0000 7ff0a880e700 1 -- start start 2023-10-16T12:28:55.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.647+0000 7ff0a880e700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff0a40d0750 0x7ff0a40c6de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:55.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.647+0000 7ff0a880e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff0a40d41c0 0x7ff0a41f2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:55.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.647+0000 7ff0a880e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0a415d8d0 0x7ff0a414e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:55.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.647+0000 7ff0a880e700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0a41e6170 con 0x7ff0a415d8d0 2023-10-16T12:28:55.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.647+0000 7ff0a880e700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0a41f3210 con 0x7ff0a40d0750 2023-10-16T12:28:55.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.647+0000 7ff0a880e700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0a41e6350 con 0x7ff0a40d41c0 2023-10-16T12:28:55.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.647+0000 7ff0a2d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0a415d8d0 0x7ff0a414e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:28:55.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.647+0000 7ff0a259c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff0a40d0750 0x7ff0a40c6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:28:55.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.647+0000 7ff0a2d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0a415d8d0 0x7ff0a414e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42958/0 (socket says 172.21.15.16:42958) 2023-10-16T12:28:55.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.647+0000 7ff0a2d9d700 1 -- 172.21.15.16:0/2714880439 learned_addr learned my addr 172.21.15.16:0/2714880439 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:28:55.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.647+0000 7ff0a1d9b700 1 --2- 172.21.15.16:0/2714880439 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff0a40d41c0 0x7ff0a41f2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:28:55.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.647+0000 7ff0a2d9d700 1 -- 172.21.15.16:0/2714880439 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff0a40d41c0 msgr2=0x7ff0a41f2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:28:55.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.647+0000 7ff0a2d9d700 1 --2- 172.21.15.16:0/2714880439 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff0a40d41c0 0x7ff0a41f2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:55.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.647+0000 7ff0a2d9d700 1 -- 172.21.15.16:0/2714880439 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff0a40d0750 msgr2=0x7ff0a40c6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:28:55.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.647+0000 7ff0a2d9d700 1 --2- 172.21.15.16:0/2714880439 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff0a40d0750 0x7ff0a40c6de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:55.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.647+0000 7ff0a2d9d700 1 -- 172.21.15.16:0/2714880439 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff09c009a70 con 0x7ff0a415d8d0 2023-10-16T12:28:55.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.647+0000 7ff0a1d9b700 1 --2- 172.21.15.16:0/2714880439 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff0a40d41c0 0x7ff0a41f2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:28:55.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.647+0000 7ff0a2d9d700 1 --2- 172.21.15.16:0/2714880439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0a415d8d0 0x7ff0a414e400 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7ff09c014040 tx=0x7ff09c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:28:55.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.648+0000 7ff092ffd700 1 -- 172.21.15.16:0/2714880439 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff09c0048c0 con 0x7ff0a415d8d0 2023-10-16T12:28:55.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.648+0000 7ff092ffd700 1 -- 172.21.15.16:0/2714880439 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff09c020070 con 0x7ff0a415d8d0 2023-10-16T12:28:55.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.648+0000 7ff0a880e700 1 -- 172.21.15.16:0/2714880439 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0a41e6630 con 0x7ff0a415d8d0 2023-10-16T12:28:55.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.648+0000 7ff092ffd700 1 -- 172.21.15.16:0/2714880439 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff09c027680 con 0x7ff0a415d8d0 2023-10-16T12:28:55.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.648+0000 7ff0a880e700 1 -- 172.21.15.16:0/2714880439 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0a41e6bf0 con 0x7ff0a415d8d0 2023-10-16T12:28:55.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.649+0000 7ff0a880e700 1 -- 172.21.15.16:0/2714880439 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0a404ebb0 con 0x7ff0a415d8d0 2023-10-16T12:28:55.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.650+0000 7ff092ffd700 1 -- 172.21.15.16:0/2714880439 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 58) v1 ==== 89445+0+0 (secure 0 0 0) 0x7ff09c012430 con 0x7ff0a415d8d0 2023-10-16T12:28:55.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.651+0000 7ff092ffd700 1 --2- 172.21.15.16:0/2714880439 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7ff08406b170 0x7ff08406d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:55.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.651+0000 7ff0a259c700 1 -- 172.21.15.16:0/2714880439 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7ff08406b170 msgr2=0x7ff08406d630 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1458543709 2023-10-16T12:28:55.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.651+0000 7ff092ffd700 1 -- 172.21.15.16:0/2714880439 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 6517+0+0 (secure 0 0 0) 0x7ff09c022070 con 0x7ff0a415d8d0 2023-10-16T12:28:55.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.651+0000 7ff0a259c700 1 --2- 172.21.15.16:0/2714880439 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7ff08406b170 0x7ff08406d630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:28:55.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.654+0000 7ff092ffd700 1 -- 172.21.15.16:0/2714880439 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff09c056c70 con 0x7ff0a415d8d0 2023-10-16T12:28:55.678 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:28:55.676+0000 7fa4e0be9200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T12:28:55.690 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:28:55.689+0000 7f9831e6b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T12:28:55.826 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:28:55.824+0000 7fa4e0be9200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T12:28:55.839 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:28:55.838+0000 7f9831e6b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T12:28:55.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.852+0000 7ff0a259c700 1 -- 172.21.15.16:0/2714880439 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7ff08406b170 msgr2=0x7ff08406d630 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1458543709 2023-10-16T12:28:55.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.852+0000 7ff0a259c700 1 --2- 172.21.15.16:0/2714880439 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7ff08406b170 0x7ff08406d630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:28:55.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.861+0000 7ff0a880e700 1 -- 172.21.15.16:0/2714880439 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff0a40b1740 con 0x7ff0a415d8d0 2023-10-16T12:28:55.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.862+0000 7ff092ffd700 1 -- 172.21.15.16:0/2714880439 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v115) v1 ==== 74+0+17357 (secure 0 0 0) 0x7ff09c05a290 con 0x7ff0a415d8d0 2023-10-16T12:28:55.864 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:28:55.864 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":115,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:28:53.284270+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:27:35.907633+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2023-10-16T12:27:36.362861+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.10","mappings":[{"from":1,"to":5}]},{"pgid":"14.2d","mappings":[{"from":1,"to":7}]},{"pgid":"14.32","mappings":[{"from":1,"to":2}]},{"pgid":"14.3c","mappings":[{"from":1,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6833/4089191642":"2023-10-17T12:27:51.745827+0000","172.21.15.16:0/2709574245":"2023-10-17T12:27:51.745827+0000","172.21.15.16:0/769212771":"2023-10-17T12:27:51.745827+0000","172.21.15.16:6832/4089191642":"2023-10-17T12:27:51.745827+0000","172.21.15.16:0/3579843680":"2023-10-17T12:27:51.745827+0000","172.21.15.16:0/1190603852":"2023-10-17T12:27:51.745827+0000","172.21.15.16:0/3144677187":"2023-10-17T12:27:51.745827+0000","172.21.15.16:0/3478352903":"2023-10-17T12:27:51.745827+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:28:55.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.864+0000 7ff0a880e700 1 -- 172.21.15.16:0/2714880439 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7ff08406b170 msgr2=0x7ff08406d630 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:28:55.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.864+0000 7ff0a880e700 1 --2- 172.21.15.16:0/2714880439 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7ff08406b170 0x7ff08406d630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:55.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.864+0000 7ff0a880e700 1 -- 172.21.15.16:0/2714880439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0a415d8d0 msgr2=0x7ff0a414e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:28:55.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.864+0000 7ff0a880e700 1 --2- 172.21.15.16:0/2714880439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0a415d8d0 0x7ff0a414e400 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7ff09c014040 tx=0x7ff09c004ad0 comp rx=0 tx=0).stop 2023-10-16T12:28:55.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.864+0000 7ff0a880e700 1 -- 172.21.15.16:0/2714880439 shutdown_connections 2023-10-16T12:28:55.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.864+0000 7ff0a880e700 1 --2- 172.21.15.16:0/2714880439 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7ff08406b170 0x7ff08406d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:55.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.864+0000 7ff0a880e700 1 --2- 172.21.15.16:0/2714880439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0a415d8d0 0x7ff0a414e400 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:55.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.864+0000 7ff0a880e700 1 --2- 172.21.15.16:0/2714880439 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff0a40d0750 0x7ff0a40c6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:55.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.864+0000 7ff0a880e700 1 --2- 172.21.15.16:0/2714880439 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff0a40d41c0 0x7ff0a41f2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:55.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.864+0000 7ff0a880e700 1 -- 172.21.15.16:0/2714880439 >> 172.21.15.16:0/2714880439 conn(0x7ff0a40d3630 msgr2=0x7ff0a40b1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:28:55.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.864+0000 7ff0a880e700 1 -- 172.21.15.16:0/2714880439 shutdown_connections 2023-10-16T12:28:55.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:55.864+0000 7ff0a880e700 1 -- 172.21.15.16:0/2714880439 wait complete. 2023-10-16T12:28:55.879 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T12:27:35.907633+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '111', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T12:27:36.362861+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '112', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T12:28:55.880 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:28:56.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.046+0000 7f7ec62f4700 1 Processor -- start 2023-10-16T12:28:56.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.046+0000 7f7ec62f4700 1 -- start start 2023-10-16T12:28:56.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.047+0000 7f7ec62f4700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7ec00cde90 0x7f7ec015ea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:56.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.047+0000 7f7ec62f4700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f7ec00cb6f0 0x7f7ec015efa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:56.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.047+0000 7f7ec62f4700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7ec00ce270 0x7f7ec00cf2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:56.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.047+0000 7f7ec62f4700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ec00cee50 con 0x7f7ec00cde90 2023-10-16T12:28:56.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.047+0000 7f7ec62f4700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ec00c5020 con 0x7f7ec00cb6f0 2023-10-16T12:28:56.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.047+0000 7f7ec62f4700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec00d2340 con 0x7f7ec00ce270 2023-10-16T12:28:56.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.047+0000 7f7ebffff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7ec00cde90 0x7f7ec015ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:28:56.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.047+0000 7f7ebffff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7ec00cde90 0x7f7ec015ea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42998/0 (socket says 172.21.15.16:42998) 2023-10-16T12:28:56.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.047+0000 7f7ebffff700 1 -- 172.21.15.16:0/2516781256 learned_addr learned my addr 172.21.15.16:0/2516781256 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:28:56.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.047+0000 7f7ec4891700 1 --2- 172.21.15.16:0/2516781256 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7ec00ce270 0x7f7ec00cf2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:28:56.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.047+0000 7f7ebf7fe700 1 --2- 172.21.15.16:0/2516781256 >> v2:172.21.15.49:3300/0 conn(0x7f7ec00cb6f0 0x7f7ec015efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:28:56.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.047+0000 7f7ebffff700 1 -- 172.21.15.16:0/2516781256 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7ec00ce270 msgr2=0x7f7ec00cf2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:28:56.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.047+0000 7f7ebffff700 1 --2- 172.21.15.16:0/2516781256 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7ec00ce270 0x7f7ec00cf2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:56.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.047+0000 7f7ebffff700 1 -- 172.21.15.16:0/2516781256 >> v2:172.21.15.49:3300/0 conn(0x7f7ec00cb6f0 msgr2=0x7f7ec015efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:28:56.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.047+0000 7f7ebffff700 1 --2- 172.21.15.16:0/2516781256 >> v2:172.21.15.49:3300/0 conn(0x7f7ec00cb6f0 0x7f7ec015efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:56.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.047+0000 7f7ebffff700 1 -- 172.21.15.16:0/2516781256 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ec00ccdd0 con 0x7f7ec00cde90 2023-10-16T12:28:56.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.047+0000 7f7ec4891700 1 --2- 172.21.15.16:0/2516781256 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7ec00ce270 0x7f7ec00cf2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:28:56.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.048+0000 7f7ebffff700 1 --2- 172.21.15.16:0/2516781256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ec00cde90 0x7f7ec015ea60 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f7ec00b2f50 tx=0x7f7eb0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c16786c4e1e6d92c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:28:56.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.048+0000 7f7ebeffd700 1 -- 172.21.15.16:0/2516781256 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eb0011830 con 0x7f7ec00cde90 2023-10-16T12:28:56.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.048+0000 7f7ebeffd700 1 -- 172.21.15.16:0/2516781256 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7eb0011990 con 0x7f7ec00cde90 2023-10-16T12:28:56.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.048+0000 7f7ebeffd700 1 -- 172.21.15.16:0/2516781256 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eb0011d20 con 0x7f7ec00cde90 2023-10-16T12:28:56.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.048+0000 7f7ec62f4700 1 -- 172.21.15.16:0/2516781256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ec00cde90 msgr2=0x7f7ec015ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:28:56.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.048+0000 7f7ec62f4700 1 --2- 172.21.15.16:0/2516781256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ec00cde90 0x7f7ec015ea60 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f7ec00b2f50 tx=0x7f7eb0009bd0 comp rx=0 tx=0).stop 2023-10-16T12:28:56.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.048+0000 7f7ec62f4700 1 -- 172.21.15.16:0/2516781256 shutdown_connections 2023-10-16T12:28:56.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.048+0000 7f7ec62f4700 1 --2- 172.21.15.16:0/2516781256 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7ec00ce270 0x7f7ec00cf2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:56.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.048+0000 7f7ec62f4700 1 --2- 172.21.15.16:0/2516781256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ec00cde90 0x7f7ec015ea60 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:56.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.048+0000 7f7ec62f4700 1 --2- 172.21.15.16:0/2516781256 >> v2:172.21.15.49:3300/0 conn(0x7f7ec00cb6f0 0x7f7ec015efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:56.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.048+0000 7f7ec62f4700 1 -- 172.21.15.16:0/2516781256 >> 172.21.15.16:0/2516781256 conn(0x7f7ec00d14d0 msgr2=0x7f7ec00d18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:28:56.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.049+0000 7f7ec62f4700 1 -- 172.21.15.16:0/2516781256 shutdown_connections 2023-10-16T12:28:56.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.049+0000 7f7ec62f4700 1 -- 172.21.15.16:0/2516781256 wait complete. 2023-10-16T12:28:56.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.050+0000 7f7ec62f4700 1 Processor -- start 2023-10-16T12:28:56.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.050+0000 7f7ec62f4700 1 -- start start 2023-10-16T12:28:56.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.051+0000 7f7ec62f4700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7ec00cb6f0 0x7f7ec00b6950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:56.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.051+0000 7f7ec62f4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ec00cde90 0x7f7ec00c85a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:56.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.051+0000 7f7ec62f4700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7ec00ce270 0x7f7ec0155820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:56.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.051+0000 7f7ec62f4700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec00b6e90 con 0x7f7ec00cde90 2023-10-16T12:28:56.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.051+0000 7f7ec62f4700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec01542a0 con 0x7f7ec00ce270 2023-10-16T12:28:56.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.051+0000 7f7ec62f4700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec00b51a0 con 0x7f7ec00cb6f0 2023-10-16T12:28:56.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.051+0000 7f7ebf7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ec00cde90 0x7f7ec00c85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:28:56.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.051+0000 7f7ebf7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ec00cde90 0x7f7ec00c85a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43006/0 (socket says 172.21.15.16:43006) 2023-10-16T12:28:56.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.051+0000 7f7ebf7fe700 1 -- 172.21.15.16:0/645680622 learned_addr learned my addr 172.21.15.16:0/645680622 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:28:56.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.051+0000 7f7ebffff700 1 --2- 172.21.15.16:0/645680622 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7ec00cb6f0 0x7f7ec00b6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:28:56.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.051+0000 7f7ec4891700 1 --2- 172.21.15.16:0/645680622 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7ec00ce270 0x7f7ec0155820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:28:56.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.051+0000 7f7ebf7fe700 1 -- 172.21.15.16:0/645680622 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7ec00cb6f0 msgr2=0x7f7ec00b6950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:28:56.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.051+0000 7f7ebf7fe700 1 --2- 172.21.15.16:0/645680622 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7ec00cb6f0 0x7f7ec00b6950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:56.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.051+0000 7f7ebf7fe700 1 -- 172.21.15.16:0/645680622 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7ec00ce270 msgr2=0x7f7ec0155820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:28:56.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.051+0000 7f7ebf7fe700 1 --2- 172.21.15.16:0/645680622 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7ec00ce270 0x7f7ec0155820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:56.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.051+0000 7f7ebf7fe700 1 -- 172.21.15.16:0/645680622 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eb0009550 con 0x7f7ec00cde90 2023-10-16T12:28:56.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.051+0000 7f7ebf7fe700 1 --2- 172.21.15.16:0/645680622 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ec00cde90 0x7f7ec00c85a0 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f7eb400e8a0 tx=0x7f7eb400ebb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:28:56.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.051+0000 7f7ebcff9700 1 -- 172.21.15.16:0/645680622 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eb4013680 con 0x7f7ec00cde90 2023-10-16T12:28:56.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.051+0000 7f7ebcff9700 1 -- 172.21.15.16:0/645680622 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7eb40137e0 con 0x7f7ec00cde90 2023-10-16T12:28:56.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.051+0000 7f7ec62f4700 1 -- 172.21.15.16:0/645680622 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ec0153c50 con 0x7f7ec00cde90 2023-10-16T12:28:56.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.051+0000 7f7ebcff9700 1 -- 172.21.15.16:0/645680622 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eb4013b90 con 0x7f7ec00cde90 2023-10-16T12:28:56.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.051+0000 7f7ec62f4700 1 -- 172.21.15.16:0/645680622 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ec00b5480 con 0x7f7ec00cde90 2023-10-16T12:28:56.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.052+0000 7f7ec62f4700 1 -- 172.21.15.16:0/645680622 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ec004ebb0 con 0x7f7ec00cde90 2023-10-16T12:28:56.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.053+0000 7f7ebcff9700 1 -- 172.21.15.16:0/645680622 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 58) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f7eb4013cf0 con 0x7f7ec00cde90 2023-10-16T12:28:56.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.053+0000 7f7ebcff9700 1 --2- 172.21.15.16:0/645680622 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f7eac06b140 0x7f7eac06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:56.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.054+0000 7f7ebffff700 1 -- 172.21.15.16:0/645680622 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f7eac06b140 msgr2=0x7f7eac06d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1458543709 2023-10-16T12:28:56.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.054+0000 7f7ebcff9700 1 -- 172.21.15.16:0/645680622 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f7eb408a2f0 con 0x7f7ec00cde90 2023-10-16T12:28:56.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.054+0000 7f7ebffff700 1 --2- 172.21.15.16:0/645680622 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f7eac06b140 0x7f7eac06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:28:56.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.058+0000 7f7ebcff9700 1 -- 172.21.15.16:0/645680622 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7eb4054d80 con 0x7f7ec00cde90 2023-10-16T12:28:56.166 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:28:56.164+0000 7fa4e0be9200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T12:28:56.188 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:28:56.186+0000 7f9831e6b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T12:28:56.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.255+0000 7f7ebffff700 1 -- 172.21.15.16:0/645680622 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f7eac06b140 msgr2=0x7f7eac06d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1458543709 2023-10-16T12:28:56.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.255+0000 7f7ebffff700 1 --2- 172.21.15.16:0/645680622 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f7eac06b140 0x7f7eac06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:28:56.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.262+0000 7f7ec62f4700 1 -- 172.21.15.16:0/645680622 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7ec01543e0 con 0x7f7ec00cde90 2023-10-16T12:28:56.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.263+0000 7f7ebcff9700 1 -- 172.21.15.16:0/645680622 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v115) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7eb4017020 con 0x7f7ec00cde90 2023-10-16T12:28:56.265 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T12:28:56.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.265+0000 7f7ec62f4700 1 -- 172.21.15.16:0/645680622 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f7eac06b140 msgr2=0x7f7eac06d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:28:56.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.265+0000 7f7ec62f4700 1 --2- 172.21.15.16:0/645680622 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f7eac06b140 0x7f7eac06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:56.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.265+0000 7f7ec62f4700 1 -- 172.21.15.16:0/645680622 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ec00cde90 msgr2=0x7f7ec00c85a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:28:56.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.265+0000 7f7ec62f4700 1 --2- 172.21.15.16:0/645680622 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ec00cde90 0x7f7ec00c85a0 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f7eb400e8a0 tx=0x7f7eb400ebb0 comp rx=0 tx=0).stop 2023-10-16T12:28:56.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.265+0000 7f7ec62f4700 1 -- 172.21.15.16:0/645680622 shutdown_connections 2023-10-16T12:28:56.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.265+0000 7f7ec62f4700 1 --2- 172.21.15.16:0/645680622 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f7eac06b140 0x7f7eac06d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:56.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.265+0000 7f7ec62f4700 1 --2- 172.21.15.16:0/645680622 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7ec00ce270 0x7f7ec0155820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:56.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.265+0000 7f7ec62f4700 1 --2- 172.21.15.16:0/645680622 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7ec00cb6f0 0x7f7ec00b6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:56.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.265+0000 7f7ec62f4700 1 --2- 172.21.15.16:0/645680622 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ec00cde90 0x7f7ec00c85a0 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:56.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.265+0000 7f7ec62f4700 1 -- 172.21.15.16:0/645680622 >> 172.21.15.16:0/645680622 conn(0x7f7ec00d14d0 msgr2=0x7f7ec006cf60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:28:56.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.265+0000 7f7ec62f4700 1 -- 172.21.15.16:0/645680622 shutdown_connections 2023-10-16T12:28:56.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.265+0000 7f7ec62f4700 1 -- 172.21.15.16:0/645680622 wait complete. 2023-10-16T12:28:56.282 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T12:28:56.288 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:28:56.286+0000 7fa4e0be9200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T12:28:56.312 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:28:56.311+0000 7f9831e6b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T12:28:56.366 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:28:56.364+0000 7fa4e0be9200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T12:28:56.391 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:28:56.389+0000 7f9831e6b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T12:28:56.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.409+0000 7f71206d8700 1 Processor -- start 2023-10-16T12:28:56.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.409+0000 7f71206d8700 1 -- start start 2023-10-16T12:28:56.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.410+0000 7f71206d8700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f711815ec80 0x7f71180cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:56.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.410+0000 7f71206d8700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f711815f060 0x7f71180ce400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:56.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.410+0000 7f71206d8700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71180b2440 con 0x7f71180d2110 2023-10-16T12:28:56.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.410+0000 7f71206d8700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71180b2c50 con 0x7f711815ec80 2023-10-16T12:28:56.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.410+0000 7f71206d8700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71180d00e0 con 0x7f711815f060 2023-10-16T12:28:56.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.411+0000 7f711e474700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f711815ec80 0x7f71180cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:28:56.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.411+0000 7f711e474700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f711815ec80 0x7f71180cdec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43014/0 (socket says 172.21.15.16:43014) 2023-10-16T12:28:56.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.411+0000 7f711e474700 1 -- 172.21.15.16:0/3776839145 learned_addr learned my addr 172.21.15.16:0/3776839145 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:28:56.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.411+0000 7f711dc73700 1 --2- 172.21.15.16:0/3776839145 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f711815f060 0x7f71180ce400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:28:56.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.411+0000 7f711e474700 1 -- 172.21.15.16:0/3776839145 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f711815f060 msgr2=0x7f71180ce400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:28:56.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.411+0000 7f711e474700 1 --2- 172.21.15.16:0/3776839145 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f711815f060 0x7f71180ce400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:56.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.411+0000 7f711e474700 1 -- 172.21.15.16:0/3776839145 >> v1:172.21.15.49:6789/0 conn(0x7f71180d2110 legacy=0x7f71180b4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:28:56.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.411+0000 7f711e474700 1 -- 172.21.15.16:0/3776839145 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71180cef60 con 0x7f711815ec80 2023-10-16T12:28:56.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.411+0000 7f711dc73700 1 --2- 172.21.15.16:0/3776839145 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f711815f060 0x7f71180ce400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:28:56.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.411+0000 7f711e474700 1 --2- 172.21.15.16:0/3776839145 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f711815ec80 0x7f71180cdec0 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f71140098f0 tx=0x7f7114009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb880062ed8626e1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:28:56.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.411+0000 7f711d472700 1 -- 172.21.15.16:0/3776839145 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f711400b8a0 con 0x7f711815ec80 2023-10-16T12:28:56.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.411+0000 7f711d472700 1 -- 172.21.15.16:0/3776839145 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f711400ba00 con 0x7f711815ec80 2023-10-16T12:28:56.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.411+0000 7f711d472700 1 -- 172.21.15.16:0/3776839145 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f711400bd90 con 0x7f711815ec80 2023-10-16T12:28:56.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.412+0000 7f71206d8700 1 -- 172.21.15.16:0/3776839145 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f711815ec80 msgr2=0x7f71180cdec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:28:56.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.412+0000 7f71206d8700 1 --2- 172.21.15.16:0/3776839145 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f711815ec80 0x7f71180cdec0 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f71140098f0 tx=0x7f7114009c00 comp rx=0 tx=0).stop 2023-10-16T12:28:56.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.412+0000 7f71206d8700 1 -- 172.21.15.16:0/3776839145 shutdown_connections 2023-10-16T12:28:56.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.412+0000 7f71206d8700 1 --2- 172.21.15.16:0/3776839145 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f711815ec80 0x7f71180cdec0 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:56.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.412+0000 7f71206d8700 1 --2- 172.21.15.16:0/3776839145 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f711815f060 0x7f71180ce400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:56.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.412+0000 7f71206d8700 1 -- 172.21.15.16:0/3776839145 >> 172.21.15.16:0/3776839145 conn(0x7f71180d15f0 msgr2=0x7f71180d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:28:56.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.412+0000 7f71206d8700 1 -- 172.21.15.16:0/3776839145 shutdown_connections 2023-10-16T12:28:56.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.412+0000 7f71206d8700 1 -- 172.21.15.16:0/3776839145 wait complete. 2023-10-16T12:28:56.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.414+0000 7f71206d8700 1 Processor -- start 2023-10-16T12:28:56.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.414+0000 7f71206d8700 1 -- start start 2023-10-16T12:28:56.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.415+0000 7f71206d8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71180d2110 0x7f71180c86c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:56.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.415+0000 7f71206d8700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f711815ec80 0x7f71180c79a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:56.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.415+0000 7f71206d8700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f711815f060 0x7f71180b6a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:56.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.415+0000 7f71206d8700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71180c9920 con 0x7f71180d2110 2023-10-16T12:28:56.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.415+0000 7f71206d8700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71180c9d40 con 0x7f711815ec80 2023-10-16T12:28:56.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.415+0000 7f71206d8700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71180c8ca0 con 0x7f711815f060 2023-10-16T12:28:56.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.415+0000 7f711e474700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71180d2110 0x7f71180c86c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:28:56.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.415+0000 7f711e474700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71180d2110 0x7f71180c86c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43020/0 (socket says 172.21.15.16:43020) 2023-10-16T12:28:56.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.415+0000 7f711e474700 1 -- 172.21.15.16:0/1556668538 learned_addr learned my addr 172.21.15.16:0/1556668538 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:28:56.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.415+0000 7f711dc73700 1 --2- 172.21.15.16:0/1556668538 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f711815ec80 0x7f71180c79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:28:56.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.415+0000 7f711ec75700 1 --2- 172.21.15.16:0/1556668538 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f711815f060 0x7f71180b6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:28:56.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.415+0000 7f711e474700 1 -- 172.21.15.16:0/1556668538 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f711815f060 msgr2=0x7f71180b6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:28:56.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.415+0000 7f711e474700 1 --2- 172.21.15.16:0/1556668538 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f711815f060 0x7f71180b6a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:56.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.415+0000 7f711e474700 1 -- 172.21.15.16:0/1556668538 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f711815ec80 msgr2=0x7f71180c79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:28:56.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.415+0000 7f711e474700 1 --2- 172.21.15.16:0/1556668538 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f711815ec80 0x7f71180c79a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:56.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.415+0000 7f711e474700 1 -- 172.21.15.16:0/1556668538 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7114009550 con 0x7f71180d2110 2023-10-16T12:28:56.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.415+0000 7f711e474700 1 --2- 172.21.15.16:0/1556668538 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71180d2110 0x7f71180c86c0 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f7114000c00 tx=0x7f7114004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:28:56.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.415+0000 7f710effd700 1 -- 172.21.15.16:0/1556668538 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7114004300 con 0x7f71180d2110 2023-10-16T12:28:56.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.415+0000 7f710effd700 1 -- 172.21.15.16:0/1556668538 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f711401f070 con 0x7f71180d2110 2023-10-16T12:28:56.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.415+0000 7f71206d8700 1 -- 172.21.15.16:0/1556668538 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71180c8f80 con 0x7f71180d2110 2023-10-16T12:28:56.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.415+0000 7f710effd700 1 -- 172.21.15.16:0/1556668538 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71140036a0 con 0x7f71180d2110 2023-10-16T12:28:56.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.416+0000 7f71206d8700 1 -- 172.21.15.16:0/1556668538 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71180c9540 con 0x7f71180d2110 2023-10-16T12:28:56.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.417+0000 7f71206d8700 1 -- 172.21.15.16:0/1556668538 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71180c9a60 con 0x7f71180d2110 2023-10-16T12:28:56.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.417+0000 7f710effd700 1 -- 172.21.15.16:0/1556668538 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 58) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f7114021030 con 0x7f71180d2110 2023-10-16T12:28:56.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.417+0000 7f710effd700 1 --2- 172.21.15.16:0/1556668538 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f70fc06b140 0x7f70fc06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:56.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.417+0000 7f711dc73700 1 -- 172.21.15.16:0/1556668538 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f70fc06b140 msgr2=0x7f70fc06d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1458543709 2023-10-16T12:28:56.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.417+0000 7f710effd700 1 -- 172.21.15.16:0/1556668538 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f711408b000 con 0x7f71180d2110 2023-10-16T12:28:56.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.417+0000 7f711dc73700 1 --2- 172.21.15.16:0/1556668538 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f70fc06b140 0x7f70fc06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:28:56.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.422+0000 7f710effd700 1 -- 172.21.15.16:0/1556668538 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7114055a30 con 0x7f71180d2110 2023-10-16T12:28:56.450 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:28:56.448+0000 7fa4e0be9200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T12:28:56.478 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:28:56.476+0000 7f9831e6b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T12:28:56.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.618+0000 7f711dc73700 1 -- 172.21.15.16:0/1556668538 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f70fc06b140 msgr2=0x7f70fc06d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1458543709 2023-10-16T12:28:56.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.618+0000 7f711dc73700 1 --2- 172.21.15.16:0/1556668538 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f70fc06b140 0x7f70fc06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:28:56.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.627+0000 7f71206d8700 1 -- 172.21.15.16:0/1556668538 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f711804ebb0 con 0x7f71180d2110 2023-10-16T12:28:56.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.628+0000 7f710effd700 1 -- 172.21.15.16:0/1556668538 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v115) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7114059050 con 0x7f71180d2110 2023-10-16T12:28:56.630 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-10-16T12:28:56.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.630+0000 7f71206d8700 1 -- 172.21.15.16:0/1556668538 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f70fc06b140 msgr2=0x7f70fc06d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:28:56.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.630+0000 7f71206d8700 1 --2- 172.21.15.16:0/1556668538 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f70fc06b140 0x7f70fc06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:56.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.630+0000 7f71206d8700 1 -- 172.21.15.16:0/1556668538 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71180d2110 msgr2=0x7f71180c86c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:28:56.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.630+0000 7f71206d8700 1 --2- 172.21.15.16:0/1556668538 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71180d2110 0x7f71180c86c0 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f7114000c00 tx=0x7f7114004750 comp rx=0 tx=0).stop 2023-10-16T12:28:56.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.630+0000 7f71206d8700 1 -- 172.21.15.16:0/1556668538 shutdown_connections 2023-10-16T12:28:56.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.630+0000 7f71206d8700 1 --2- 172.21.15.16:0/1556668538 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f70fc06b140 0x7f70fc06d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:56.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.630+0000 7f71206d8700 1 --2- 172.21.15.16:0/1556668538 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f711815f060 0x7f71180b6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:56.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.630+0000 7f71206d8700 1 --2- 172.21.15.16:0/1556668538 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71180d2110 0x7f71180c86c0 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:56.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.630+0000 7f71206d8700 1 --2- 172.21.15.16:0/1556668538 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f711815ec80 0x7f71180c79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:56.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.630+0000 7f71206d8700 1 -- 172.21.15.16:0/1556668538 >> 172.21.15.16:0/1556668538 conn(0x7f71180d15f0 msgr2=0x7f71180d19d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:28:56.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.630+0000 7f71206d8700 1 -- 172.21.15.16:0/1556668538 shutdown_connections 2023-10-16T12:28:56.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.630+0000 7f71206d8700 1 -- 172.21.15.16:0/1556668538 wait complete. 2023-10-16T12:28:56.649 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T12:28:56.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.816+0000 7ff3fa996700 1 Processor -- start 2023-10-16T12:28:56.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.816+0000 7ff3fa996700 1 -- start start 2023-10-16T12:28:56.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.817+0000 7ff3fa996700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7ff3f415ec80 0x7ff3f40cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:56.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.817+0000 7ff3fa996700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff3f40d2110 0x7ff3f40cd1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:56.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.817+0000 7ff3fa996700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3f40b2440 con 0x7ff3f415f060 2023-10-16T12:28:56.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.817+0000 7ff3fa996700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3f40b2c50 con 0x7ff3f415ec80 2023-10-16T12:28:56.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.817+0000 7ff3fa996700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f40d00e0 con 0x7ff3f40d2110 2023-10-16T12:28:56.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.817+0000 7ff3f37fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff3f415f060 0x7ff3f40b4440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:60558/0 (socket says 172.21.15.16:60558) 2023-10-16T12:28:56.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.817+0000 7ff3f37fe700 1 -- 172.21.15.16:0/3367968050 learned_addr learned my addr 172.21.15.16:0/3367968050 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:28:56.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.817+0000 7ff3f3fff700 1 --2- 172.21.15.16:0/3367968050 >> v2:172.21.15.49:3300/0 conn(0x7ff3f415ec80 0x7ff3f40cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:28:56.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.817+0000 7ff3f8f33700 1 --2- 172.21.15.16:0/3367968050 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff3f40d2110 0x7ff3f40cd1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:28:56.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.817+0000 7ff3f2ffd700 1 -- 172.21.15.16:0/3367968050 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 41540741 0 0) 0x7ff3f40b2440 con 0x7ff3f415f060 2023-10-16T12:28:56.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.817+0000 7ff3f2ffd700 1 -- 172.21.15.16:0/3367968050 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3dc0035b0 con 0x7ff3f415f060 2023-10-16T12:28:56.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.818+0000 7ff3f8f33700 1 -- 172.21.15.16:0/3367968050 >> v2:172.21.15.49:3300/0 conn(0x7ff3f415ec80 msgr2=0x7ff3f40cdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:28:56.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.818+0000 7ff3f8f33700 1 --2- 172.21.15.16:0/3367968050 >> v2:172.21.15.49:3300/0 conn(0x7ff3f415ec80 0x7ff3f40cdec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:56.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.818+0000 7ff3f8f33700 1 -- 172.21.15.16:0/3367968050 >> v1:172.21.15.16:6789/0 conn(0x7ff3f415f060 legacy=0x7ff3f40b4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:28:56.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.818+0000 7ff3f8f33700 1 -- 172.21.15.16:0/3367968050 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3f40cef60 con 0x7ff3f40d2110 2023-10-16T12:28:56.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.818+0000 7ff3f3fff700 1 --2- 172.21.15.16:0/3367968050 >> v2:172.21.15.49:3300/0 conn(0x7ff3f415ec80 0x7ff3f40cdec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:28:56.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.818+0000 7ff3f8f33700 1 --2- 172.21.15.16:0/3367968050 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff3f40d2110 0x7ff3f40cd1a0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ff3ec009d90 tx=0x7ff3ec00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47a78c01f314cead server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:28:56.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.818+0000 7ff3f2ffd700 1 -- 172.21.15.16:0/3367968050 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3ec00e070 con 0x7ff3f40d2110 2023-10-16T12:28:56.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.818+0000 7ff3f2ffd700 1 -- 172.21.15.16:0/3367968050 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3ec00bb20 con 0x7ff3f40d2110 2023-10-16T12:28:56.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.818+0000 7ff3f2ffd700 1 -- 172.21.15.16:0/3367968050 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3ec012430 con 0x7ff3f40d2110 2023-10-16T12:28:56.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.819+0000 7ff3fa996700 1 -- 172.21.15.16:0/3367968050 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff3f40d2110 msgr2=0x7ff3f40cd1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:28:56.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.819+0000 7ff3fa996700 1 --2- 172.21.15.16:0/3367968050 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff3f40d2110 0x7ff3f40cd1a0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ff3ec009d90 tx=0x7ff3ec00b330 comp rx=0 tx=0).stop 2023-10-16T12:28:56.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.819+0000 7ff3fa996700 1 -- 172.21.15.16:0/3367968050 shutdown_connections 2023-10-16T12:28:56.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.819+0000 7ff3fa996700 1 --2- 172.21.15.16:0/3367968050 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff3f40d2110 0x7ff3f40cd1a0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:56.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.819+0000 7ff3fa996700 1 --2- 172.21.15.16:0/3367968050 >> v2:172.21.15.49:3300/0 conn(0x7ff3f415ec80 0x7ff3f40cdec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:56.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.819+0000 7ff3fa996700 1 -- 172.21.15.16:0/3367968050 >> 172.21.15.16:0/3367968050 conn(0x7ff3f40d15f0 msgr2=0x7ff3f40d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:28:56.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.819+0000 7ff3fa996700 1 -- 172.21.15.16:0/3367968050 shutdown_connections 2023-10-16T12:28:56.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.819+0000 7ff3fa996700 1 -- 172.21.15.16:0/3367968050 wait complete. 2023-10-16T12:28:56.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.820+0000 7ff3fa996700 1 Processor -- start 2023-10-16T12:28:56.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.821+0000 7ff3fa996700 1 -- start start 2023-10-16T12:28:56.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.822+0000 7ff3fa996700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff3f40d2110 0x7ff3f40c6e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:56.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.822+0000 7ff3fa996700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff3f415ec80 0x7ff3f414c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:56.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.822+0000 7ff3fa996700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3f415f060 0x7ff3f40c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:56.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.822+0000 7ff3fa996700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f414c900 con 0x7ff3f415f060 2023-10-16T12:28:56.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.822+0000 7ff3fa996700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f40c74a0 con 0x7ff3f40d2110 2023-10-16T12:28:56.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.822+0000 7ff3fa996700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f40c7780 con 0x7ff3f415ec80 2023-10-16T12:28:56.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.822+0000 7ff3f8f33700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3f415f060 0x7ff3f40c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:28:56.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.822+0000 7ff3f3fff700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff3f40d2110 0x7ff3f40c6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:28:56.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.822+0000 7ff3f37fe700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff3f415ec80 0x7ff3f414c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:28:56.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.822+0000 7ff3f8f33700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3f415f060 0x7ff3f40c5c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43060/0 (socket says 172.21.15.16:43060) 2023-10-16T12:28:56.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.822+0000 7ff3f8f33700 1 -- 172.21.15.16:0/635680526 learned_addr learned my addr 172.21.15.16:0/635680526 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:28:56.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.822+0000 7ff3f3fff700 1 -- 172.21.15.16:0/635680526 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff3f415ec80 msgr2=0x7ff3f414c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:28:56.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.822+0000 7ff3f3fff700 1 --2- 172.21.15.16:0/635680526 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff3f415ec80 0x7ff3f414c3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:56.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.822+0000 7ff3f3fff700 1 -- 172.21.15.16:0/635680526 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3f415f060 msgr2=0x7ff3f40c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:28:56.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.822+0000 7ff3f3fff700 1 --2- 172.21.15.16:0/635680526 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3f415f060 0x7ff3f40c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:56.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.822+0000 7ff3f3fff700 1 -- 172.21.15.16:0/635680526 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3e40094a0 con 0x7ff3f40d2110 2023-10-16T12:28:56.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.822+0000 7ff3f8f33700 1 --2- 172.21.15.16:0/635680526 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3f415f060 0x7ff3f40c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:28:56.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.822+0000 7ff3f3fff700 1 --2- 172.21.15.16:0/635680526 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff3f40d2110 0x7ff3f40c6e10 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7ff3e400e9b0 tx=0x7ff3e400ecc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:28:56.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.822+0000 7ff3f0ff9700 1 -- 172.21.15.16:0/635680526 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3e400ee50 con 0x7ff3f40d2110 2023-10-16T12:28:56.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.822+0000 7ff3f0ff9700 1 -- 172.21.15.16:0/635680526 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3e400c850 con 0x7ff3f40d2110 2023-10-16T12:28:56.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.823+0000 7ff3fa996700 1 -- 172.21.15.16:0/635680526 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3ec009a70 con 0x7ff3f40d2110 2023-10-16T12:28:56.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.823+0000 7ff3f0ff9700 1 -- 172.21.15.16:0/635680526 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3e400cc00 con 0x7ff3f40d2110 2023-10-16T12:28:56.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.823+0000 7ff3fa996700 1 -- 172.21.15.16:0/635680526 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3f40c7d40 con 0x7ff3f40d2110 2023-10-16T12:28:56.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.823+0000 7ff3fa996700 1 -- 172.21.15.16:0/635680526 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3f404ebb0 con 0x7ff3f40d2110 2023-10-16T12:28:56.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.824+0000 7ff3f0ff9700 1 -- 172.21.15.16:0/635680526 <== mon.1 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 58) v1 ==== 89445+0+0 (secure 0 0 0) 0x7ff3e4013470 con 0x7ff3f40d2110 2023-10-16T12:28:56.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.824+0000 7ff3f0ff9700 1 --2- 172.21.15.16:0/635680526 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7ff3e006b140 0x7ff3e006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:56.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.824+0000 7ff3f37fe700 1 -- 172.21.15.16:0/635680526 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7ff3e006b140 msgr2=0x7ff3e006d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1458543709 2023-10-16T12:28:56.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.825+0000 7ff3f0ff9700 1 -- 172.21.15.16:0/635680526 <== mon.1 v2:172.21.15.49:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 6517+0+0 (secure 0 0 0) 0x7ff3e4089ce0 con 0x7ff3f40d2110 2023-10-16T12:28:56.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.825+0000 7ff3f37fe700 1 --2- 172.21.15.16:0/635680526 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7ff3e006b140 0x7ff3e006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:28:56.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:56.829+0000 7ff3f0ff9700 1 -- 172.21.15.16:0/635680526 <== mon.1 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff3e4054790 con 0x7ff3f40d2110 2023-10-16T12:28:56.837 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:28:56.835+0000 7fa4e0be9200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T12:28:56.878 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:28:56.876+0000 7f9831e6b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T12:28:57.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.026+0000 7ff3f37fe700 1 -- 172.21.15.16:0/635680526 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7ff3e006b140 msgr2=0x7ff3e006d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1458543709 2023-10-16T12:28:57.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.026+0000 7ff3f37fe700 1 --2- 172.21.15.16:0/635680526 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7ff3e006b140 0x7ff3e006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:28:57.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.035+0000 7ff3fa996700 1 -- 172.21.15.16:0/635680526 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff3f40d1b70 con 0x7ff3f40d2110 2023-10-16T12:28:57.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.035+0000 7ff3f0ff9700 1 -- 172.21.15.16:0/635680526 <== mon.1 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v115) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff3e4021020 con 0x7ff3f40d2110 2023-10-16T12:28:57.037 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-10-16T12:28:57.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.038+0000 7ff3fa996700 1 -- 172.21.15.16:0/635680526 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7ff3e006b140 msgr2=0x7ff3e006d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:28:57.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.038+0000 7ff3fa996700 1 --2- 172.21.15.16:0/635680526 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7ff3e006b140 0x7ff3e006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:57.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.038+0000 7ff3fa996700 1 -- 172.21.15.16:0/635680526 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff3f40d2110 msgr2=0x7ff3f40c6e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:28:57.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.038+0000 7ff3fa996700 1 --2- 172.21.15.16:0/635680526 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff3f40d2110 0x7ff3f40c6e10 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7ff3e400e9b0 tx=0x7ff3e400ecc0 comp rx=0 tx=0).stop 2023-10-16T12:28:57.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.038+0000 7ff3fa996700 1 -- 172.21.15.16:0/635680526 shutdown_connections 2023-10-16T12:28:57.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.038+0000 7ff3fa996700 1 --2- 172.21.15.16:0/635680526 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7ff3e006b140 0x7ff3e006d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:57.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.038+0000 7ff3fa996700 1 --2- 172.21.15.16:0/635680526 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3f415f060 0x7ff3f40c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:57.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.038+0000 7ff3fa996700 1 --2- 172.21.15.16:0/635680526 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff3f40d2110 0x7ff3f40c6e10 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:57.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.038+0000 7ff3fa996700 1 --2- 172.21.15.16:0/635680526 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff3f415ec80 0x7ff3f414c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:57.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.038+0000 7ff3fa996700 1 -- 172.21.15.16:0/635680526 >> 172.21.15.16:0/635680526 conn(0x7ff3f40d15f0 msgr2=0x7ff3f415dc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:28:57.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.038+0000 7ff3fa996700 1 -- 172.21.15.16:0/635680526 shutdown_connections 2023-10-16T12:28:57.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.038+0000 7ff3fa996700 1 -- 172.21.15.16:0/635680526 wait complete. 2023-10-16T12:28:57.053 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T12:28:57.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.178+0000 7f3ae2f0e700 1 Processor -- start 2023-10-16T12:28:57.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.178+0000 7f3ae2f0e700 1 -- start start 2023-10-16T12:28:57.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.179+0000 7f3ae2f0e700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f3adc0b2b50 0x7f3adc0d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:57.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.179+0000 7f3ae2f0e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3adc0ce380 0x7f3adc0d09e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:57.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.179+0000 7f3ae2f0e700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3adc0d1210 con 0x7f3adc0cdfa0 2023-10-16T12:28:57.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.179+0000 7f3ae2f0e700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3adc0cbc30 con 0x7f3adc0ce380 2023-10-16T12:28:57.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.179+0000 7f3ae2f0e700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3adc0cbf10 con 0x7f3adc0b2b50 2023-10-16T12:28:57.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.179+0000 7f3ae14ab700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3adc0ce380 0x7f3adc0d09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:28:57.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.179+0000 7f3ae14ab700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3adc0ce380 0x7f3adc0d09e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43070/0 (socket says 172.21.15.16:43070) 2023-10-16T12:28:57.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.179+0000 7f3ae0caa700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f3adc0cdfa0 0x7f3adc0b2440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:60572/0 (socket says 172.21.15.16:60572) 2023-10-16T12:28:57.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.179+0000 7f3ae14ab700 1 -- 172.21.15.16:0/1767010503 learned_addr learned my addr 172.21.15.16:0/1767010503 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:28:57.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.179+0000 7f3adbfff700 1 --2- 172.21.15.16:0/1767010503 >> v2:172.21.15.49:3300/0 conn(0x7f3adc0b2b50 0x7f3adc0d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:28:57.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.179+0000 7f3ae14ab700 1 -- 172.21.15.16:0/1767010503 >> v2:172.21.15.49:3300/0 conn(0x7f3adc0b2b50 msgr2=0x7f3adc0d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:28:57.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.179+0000 7f3ae14ab700 1 --2- 172.21.15.16:0/1767010503 >> v2:172.21.15.49:3300/0 conn(0x7f3adc0b2b50 0x7f3adc0d04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:57.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.179+0000 7f3ae14ab700 1 -- 172.21.15.16:0/1767010503 >> v1:172.21.15.16:6789/0 conn(0x7f3adc0cdfa0 legacy=0x7f3adc0b2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:28:57.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.179+0000 7f3ae14ab700 1 -- 172.21.15.16:0/1767010503 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3adc0cc1f0 con 0x7f3adc0ce380 2023-10-16T12:28:57.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.180+0000 7f3ae14ab700 1 --2- 172.21.15.16:0/1767010503 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3adc0ce380 0x7f3adc0d09e0 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f3ad4009e50 tx=0x7f3ad400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=102c85a47542214b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:28:57.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.180+0000 7f3adb7fe700 1 -- 172.21.15.16:0/1767010503 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ad400e070 con 0x7f3adc0ce380 2023-10-16T12:28:57.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.180+0000 7f3adb7fe700 1 -- 172.21.15.16:0/1767010503 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ad400b8d0 con 0x7f3adc0ce380 2023-10-16T12:28:57.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.180+0000 7f3adb7fe700 1 -- 172.21.15.16:0/1767010503 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ad400bc80 con 0x7f3adc0ce380 2023-10-16T12:28:57.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.180+0000 7f3ae2f0e700 1 -- 172.21.15.16:0/1767010503 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3adc0ce380 msgr2=0x7f3adc0d09e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:28:57.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.180+0000 7f3ae2f0e700 1 --2- 172.21.15.16:0/1767010503 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3adc0ce380 0x7f3adc0d09e0 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f3ad4009e50 tx=0x7f3ad400b330 comp rx=0 tx=0).stop 2023-10-16T12:28:57.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.181+0000 7f3ae2f0e700 1 -- 172.21.15.16:0/1767010503 shutdown_connections 2023-10-16T12:28:57.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.181+0000 7f3ae2f0e700 1 --2- 172.21.15.16:0/1767010503 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3adc0ce380 0x7f3adc0d09e0 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:57.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.181+0000 7f3ae2f0e700 1 --2- 172.21.15.16:0/1767010503 >> v2:172.21.15.49:3300/0 conn(0x7f3adc0b2b50 0x7f3adc0d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:57.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.181+0000 7f3ae2f0e700 1 -- 172.21.15.16:0/1767010503 >> 172.21.15.16:0/1767010503 conn(0x7f3adc0d14b0 msgr2=0x7f3adc0d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:28:57.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.181+0000 7f3ae2f0e700 1 -- 172.21.15.16:0/1767010503 shutdown_connections 2023-10-16T12:28:57.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.181+0000 7f3ae2f0e700 1 -- 172.21.15.16:0/1767010503 wait complete. 2023-10-16T12:28:57.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.182+0000 7f3ae2f0e700 1 Processor -- start 2023-10-16T12:28:57.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.182+0000 7f3ae2f0e700 1 -- start start 2023-10-16T12:28:57.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.183+0000 7f3ae2f0e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3adc0b2b50 0x7f3adc0cb3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:57.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.183+0000 7f3ae2f0e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3adc0cdfa0 0x7f3adc15eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:57.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.183+0000 7f3ae2f0e700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3adc0ce380 0x7f3adc151a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:57.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.183+0000 7f3ae2f0e700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3adc0ca340 con 0x7f3adc0cdfa0 2023-10-16T12:28:57.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.183+0000 7f3ae2f0e700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3adc0ca620 con 0x7f3adc0ce380 2023-10-16T12:28:57.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.183+0000 7f3ae2f0e700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3adc0ca900 con 0x7f3adc0b2b50 2023-10-16T12:28:57.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.183+0000 7f3adbfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3adc0cdfa0 0x7f3adc15eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:28:57.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.183+0000 7f3adbfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3adc0cdfa0 0x7f3adc15eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43074/0 (socket says 172.21.15.16:43074) 2023-10-16T12:28:57.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.183+0000 7f3adbfff700 1 -- 172.21.15.16:0/1949251143 learned_addr learned my addr 172.21.15.16:0/1949251143 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:28:57.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.183+0000 7f3ae14ab700 1 --2- 172.21.15.16:0/1949251143 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3adc0ce380 0x7f3adc151a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:28:57.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.183+0000 7f3ae0caa700 1 --2- 172.21.15.16:0/1949251143 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3adc0b2b50 0x7f3adc0cb3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:28:57.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.183+0000 7f3adbfff700 1 -- 172.21.15.16:0/1949251143 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3adc0b2b50 msgr2=0x7f3adc0cb3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:28:57.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.183+0000 7f3adbfff700 1 --2- 172.21.15.16:0/1949251143 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3adc0b2b50 0x7f3adc0cb3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:57.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.183+0000 7f3adbfff700 1 -- 172.21.15.16:0/1949251143 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3adc0ce380 msgr2=0x7f3adc151a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:28:57.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.183+0000 7f3adbfff700 1 --2- 172.21.15.16:0/1949251143 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3adc0ce380 0x7f3adc151a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:57.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.183+0000 7f3adbfff700 1 -- 172.21.15.16:0/1949251143 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ad4009b30 con 0x7f3adc0cdfa0 2023-10-16T12:28:57.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.184+0000 7f3adbfff700 1 --2- 172.21.15.16:0/1949251143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3adc0cdfa0 0x7f3adc15eba0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f3ac800e9b0 tx=0x7f3ac800ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:28:57.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.184+0000 7f3ad97fa700 1 -- 172.21.15.16:0/1949251143 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac8009710 con 0x7f3adc0cdfa0 2023-10-16T12:28:57.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.184+0000 7f3ad97fa700 1 -- 172.21.15.16:0/1949251143 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ac800ee50 con 0x7f3adc0cdfa0 2023-10-16T12:28:57.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.184+0000 7f3ad97fa700 1 -- 172.21.15.16:0/1949251143 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac800ca50 con 0x7f3adc0cdfa0 2023-10-16T12:28:57.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.184+0000 7f3ae2f0e700 1 -- 172.21.15.16:0/1949251143 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3adc0cabe0 con 0x7f3adc0cdfa0 2023-10-16T12:28:57.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.184+0000 7f3ae2f0e700 1 -- 172.21.15.16:0/1949251143 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3adc0c7c30 con 0x7f3adc0cdfa0 2023-10-16T12:28:57.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.185+0000 7f3ae2f0e700 1 -- 172.21.15.16:0/1949251143 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3adc04ebb0 con 0x7f3adc0cdfa0 2023-10-16T12:28:57.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.185+0000 7f3ad97fa700 1 -- 172.21.15.16:0/1949251143 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 58) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f3ac800cbb0 con 0x7f3adc0cdfa0 2023-10-16T12:28:57.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.186+0000 7f3ad97fa700 1 --2- 172.21.15.16:0/1949251143 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f3ac006b140 0x7f3ac006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:57.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.186+0000 7f3ae0caa700 1 -- 172.21.15.16:0/1949251143 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f3ac006b140 msgr2=0x7f3ac006d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1458543709 2023-10-16T12:28:57.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.186+0000 7f3ad97fa700 1 -- 172.21.15.16:0/1949251143 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f3ac8089d00 con 0x7f3adc0cdfa0 2023-10-16T12:28:57.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.186+0000 7f3ae0caa700 1 --2- 172.21.15.16:0/1949251143 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f3ac006b140 0x7f3ac006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:28:57.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.190+0000 7f3ad97fa700 1 -- 172.21.15.16:0/1949251143 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3ac8054720 con 0x7f3adc0cdfa0 2023-10-16T12:28:57.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.387+0000 7f3ae0caa700 1 -- 172.21.15.16:0/1949251143 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f3ac006b140 msgr2=0x7f3ac006d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1458543709 2023-10-16T12:28:57.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.387+0000 7f3ae0caa700 1 --2- 172.21.15.16:0/1949251143 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f3ac006b140 0x7f3ac006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:28:57.447 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:28:57.445+0000 7fa4e0be9200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T12:28:57.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.458+0000 7f3ae2f0e700 1 -- 172.21.15.16:0/1949251143 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f3adc0cfca0 con 0x7f3adc0cdfa0 2023-10-16T12:28:57.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.458+0000 7f3ad97fa700 1 -- 172.21.15.16:0/1949251143 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v115) v1 ==== 80+0+469 (secure 0 0 0) 0x7f3ac8021080 con 0x7f3adc0cdfa0 2023-10-16T12:28:57.460 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6833/4089191642 2023-10-17T12:27:51.745827+0000 2023-10-16T12:28:57.461 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2709574245 2023-10-17T12:27:51.745827+0000 2023-10-16T12:28:57.461 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/769212771 2023-10-17T12:27:51.745827+0000 2023-10-16T12:28:57.461 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6832/4089191642 2023-10-17T12:27:51.745827+0000 2023-10-16T12:28:57.461 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3579843680 2023-10-17T12:27:51.745827+0000 2023-10-16T12:28:57.461 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/1190603852 2023-10-17T12:27:51.745827+0000 2023-10-16T12:28:57.462 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3144677187 2023-10-17T12:27:51.745827+0000 2023-10-16T12:28:57.462 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3478352903 2023-10-17T12:27:51.745827+0000 2023-10-16T12:28:57.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.461+0000 7f3ae2f0e700 1 -- 172.21.15.16:0/1949251143 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f3ac006b140 msgr2=0x7f3ac006d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:28:57.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.461+0000 7f3ae2f0e700 1 --2- 172.21.15.16:0/1949251143 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f3ac006b140 0x7f3ac006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:57.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.461+0000 7f3ae2f0e700 1 -- 172.21.15.16:0/1949251143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3adc0cdfa0 msgr2=0x7f3adc15eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:28:57.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.461+0000 7f3ae2f0e700 1 --2- 172.21.15.16:0/1949251143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3adc0cdfa0 0x7f3adc15eba0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f3ac800e9b0 tx=0x7f3ac800ecc0 comp rx=0 tx=0).stop 2023-10-16T12:28:57.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.461+0000 7f3ae2f0e700 1 -- 172.21.15.16:0/1949251143 shutdown_connections 2023-10-16T12:28:57.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.461+0000 7f3ae2f0e700 1 --2- 172.21.15.16:0/1949251143 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f3ac006b140 0x7f3ac006d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:57.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.461+0000 7f3ae2f0e700 1 --2- 172.21.15.16:0/1949251143 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3adc0ce380 0x7f3adc151a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:57.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.461+0000 7f3ae2f0e700 1 --2- 172.21.15.16:0/1949251143 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3adc0b2b50 0x7f3adc0cb3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:57.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.461+0000 7f3ae2f0e700 1 --2- 172.21.15.16:0/1949251143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3adc0cdfa0 0x7f3adc15eba0 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:57.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.461+0000 7f3ae2f0e700 1 -- 172.21.15.16:0/1949251143 >> 172.21.15.16:0/1949251143 conn(0x7f3adc0d14b0 msgr2=0x7f3adc06eba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:28:57.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.461+0000 7f3ae2f0e700 1 -- 172.21.15.16:0/1949251143 shutdown_connections 2023-10-16T12:28:57.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.461+0000 7f3ae2f0e700 1 -- 172.21.15.16:0/1949251143 wait complete. 2023-10-16T12:28:57.465 INFO:teuthology.orchestra.run.smithi016.stderr:listed 8 entries 2023-10-16T12:28:57.476 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-16T12:28:57.477 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:28:57.477 DEBUG:teuthology.orchestra.run.smithi016:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-16T12:28:57.493 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:28:57.491+0000 7f9831e6b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T12:28:57.519 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:28:57.517+0000 7fa4e0be9200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T12:28:57.523 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2023-10-16T12:28:57.524 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:28:57.524 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-16T12:28:57.568 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:28:57.567+0000 7f9831e6b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T12:28:57.580 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T12:28:57.581 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:28:57.600 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284204k,nr_inodes=4071051,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=753 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260540k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T12:28:57.600 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:28:57.601 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:28:57.639 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16287364k,nr_inodes=4071841,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20251 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3261172k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T12:28:57.640 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T12:28:57.640 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:28:57.658 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284204k,nr_inodes=4071051,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=753 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260540k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T12:28:57.659 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:28:57.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.815+0000 7fce0673a700 1 Processor -- start 2023-10-16T12:28:57.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.815+0000 7fce0673a700 1 -- start start 2023-10-16T12:28:57.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.816+0000 7fce0673a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fce0015d8d0 0x7fce000d1c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:57.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.816+0000 7fce0673a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fce000d41c0 0x7fce000d0040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:57.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.816+0000 7fce0673a700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce0006ba00 con 0x7fce000d0580 2023-10-16T12:28:57.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.816+0000 7fce0673a700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce000d0a10 con 0x7fce000d41c0 2023-10-16T12:28:57.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.816+0000 7fce0673a700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce000d2150 con 0x7fce0015d8d0 2023-10-16T12:28:57.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.816+0000 7fcdff7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fce000d41c0 0x7fce000d0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:28:57.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.816+0000 7fcdff7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fce000d41c0 0x7fce000d0040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43084/0 (socket says 172.21.15.16:43084) 2023-10-16T12:28:57.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.816+0000 7fcdff7fe700 1 -- 172.21.15.16:0/3888442910 learned_addr learned my addr 172.21.15.16:0/3888442910 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:28:57.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.816+0000 7fcdfffff700 1 --2- 172.21.15.16:0/3888442910 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fce0015d8d0 0x7fce000d1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:28:57.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.817+0000 7fcdff7fe700 1 -- 172.21.15.16:0/3888442910 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fce0015d8d0 msgr2=0x7fce000d1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:28:57.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.817+0000 7fcdff7fe700 1 --2- 172.21.15.16:0/3888442910 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fce0015d8d0 0x7fce000d1c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:57.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.817+0000 7fcdff7fe700 1 -- 172.21.15.16:0/3888442910 >> v1:172.21.15.49:6789/0 conn(0x7fce000d0580 legacy=0x7fce000ce240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:28:57.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.817+0000 7fcdff7fe700 1 -- 172.21.15.16:0/3888442910 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce000d26d0 con 0x7fce000d41c0 2023-10-16T12:28:57.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.817+0000 7fcdff7fe700 1 --2- 172.21.15.16:0/3888442910 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce000d41c0 0x7fce000d0040 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7fcdf40098f0 tx=0x7fcdf4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cb10ebf85baf2ff server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:28:57.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.817+0000 7fcdfeffd700 1 -- 172.21.15.16:0/3888442910 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf4011830 con 0x7fce000d41c0 2023-10-16T12:28:57.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.817+0000 7fcdfeffd700 1 -- 172.21.15.16:0/3888442910 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdf4011990 con 0x7fce000d41c0 2023-10-16T12:28:57.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.817+0000 7fcdfeffd700 1 -- 172.21.15.16:0/3888442910 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf4011d20 con 0x7fce000d41c0 2023-10-16T12:28:57.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.817+0000 7fce0673a700 1 -- 172.21.15.16:0/3888442910 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce000d41c0 msgr2=0x7fce000d0040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:28:57.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.817+0000 7fce0673a700 1 --2- 172.21.15.16:0/3888442910 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce000d41c0 0x7fce000d0040 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7fcdf40098f0 tx=0x7fcdf4009c00 comp rx=0 tx=0).stop 2023-10-16T12:28:57.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.818+0000 7fce0673a700 1 -- 172.21.15.16:0/3888442910 shutdown_connections 2023-10-16T12:28:57.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.818+0000 7fce0673a700 1 --2- 172.21.15.16:0/3888442910 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fce0015d8d0 0x7fce000d1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:57.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.818+0000 7fce0673a700 1 --2- 172.21.15.16:0/3888442910 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce000d41c0 0x7fce000d0040 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:57.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.818+0000 7fce0673a700 1 -- 172.21.15.16:0/3888442910 >> 172.21.15.16:0/3888442910 conn(0x7fce000d3630 msgr2=0x7fce000d3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:28:57.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.818+0000 7fce0673a700 1 -- 172.21.15.16:0/3888442910 shutdown_connections 2023-10-16T12:28:57.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.818+0000 7fce0673a700 1 -- 172.21.15.16:0/3888442910 wait complete. 2023-10-16T12:28:57.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.820+0000 7fce0673a700 1 Processor -- start 2023-10-16T12:28:57.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.820+0000 7fce0673a700 1 -- start start 2023-10-16T12:28:57.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.821+0000 7fce0673a700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fce000d0580 0x7fce000ca700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:57.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.821+0000 7fce0673a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fce000d41c0 0x7fce0014ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:57.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.821+0000 7fce0673a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce0015d8d0 0x7fce001e4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:57.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.821+0000 7fce0673a700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce000cac40 con 0x7fce0015d8d0 2023-10-16T12:28:57.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.821+0000 7fce0673a700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce0014bf90 con 0x7fce000d0580 2023-10-16T12:28:57.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.821+0000 7fce0673a700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce000b2500 con 0x7fce000d41c0 2023-10-16T12:28:57.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.821+0000 7fce04cd7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce0015d8d0 0x7fce001e4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:28:57.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.821+0000 7fcdfffff700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fce000d0580 0x7fce000ca700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:28:57.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.821+0000 7fce04cd7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce0015d8d0 0x7fce001e4f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43092/0 (socket says 172.21.15.16:43092) 2023-10-16T12:28:57.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.821+0000 7fce04cd7700 1 -- 172.21.15.16:0/3250136524 learned_addr learned my addr 172.21.15.16:0/3250136524 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:28:57.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.821+0000 7fcdff7fe700 1 --2- 172.21.15.16:0/3250136524 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fce000d41c0 0x7fce0014ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:28:57.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.822+0000 7fce04cd7700 1 -- 172.21.15.16:0/3250136524 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fce000d41c0 msgr2=0x7fce0014ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:28:57.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.822+0000 7fce04cd7700 1 --2- 172.21.15.16:0/3250136524 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fce000d41c0 0x7fce0014ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:57.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.822+0000 7fce04cd7700 1 -- 172.21.15.16:0/3250136524 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fce000d0580 msgr2=0x7fce000ca700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:28:57.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.822+0000 7fce04cd7700 1 --2- 172.21.15.16:0/3250136524 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fce000d0580 0x7fce000ca700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:57.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.822+0000 7fce04cd7700 1 -- 172.21.15.16:0/3250136524 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdf4009550 con 0x7fce0015d8d0 2023-10-16T12:28:57.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.822+0000 7fcdff7fe700 1 --2- 172.21.15.16:0/3250136524 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fce000d41c0 0x7fce0014ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:28:57.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.822+0000 7fce04cd7700 1 --2- 172.21.15.16:0/3250136524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce0015d8d0 0x7fce001e4f70 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7fcdf800cc50 tx=0x7fcdf800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:28:57.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.822+0000 7fcdfcff9700 1 -- 172.21.15.16:0/3250136524 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf8003c00 con 0x7fce0015d8d0 2023-10-16T12:28:57.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.822+0000 7fcdfcff9700 1 -- 172.21.15.16:0/3250136524 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdf8003d60 con 0x7fce0015d8d0 2023-10-16T12:28:57.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.822+0000 7fce0673a700 1 -- 172.21.15.16:0/3250136524 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce001f4ae0 con 0x7fce0015d8d0 2023-10-16T12:28:57.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.822+0000 7fcdfcff9700 1 -- 172.21.15.16:0/3250136524 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf800aa50 con 0x7fce0015d8d0 2023-10-16T12:28:57.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.822+0000 7fce0673a700 1 -- 172.21.15.16:0/3250136524 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce001f50a0 con 0x7fce0015d8d0 2023-10-16T12:28:57.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.823+0000 7fcdfcff9700 1 -- 172.21.15.16:0/3250136524 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 58) v1 ==== 89445+0+0 (secure 0 0 0) 0x7fcdf800abb0 con 0x7fce0015d8d0 2023-10-16T12:28:57.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.823+0000 7fce0673a700 1 -- 172.21.15.16:0/3250136524 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce0004ebb0 con 0x7fce0015d8d0 2023-10-16T12:28:57.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.824+0000 7fcdfcff9700 1 --2- 172.21.15.16:0/3250136524 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7fcdec06b170 0x7fcdec06d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:57.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.824+0000 7fcdfffff700 1 -- 172.21.15.16:0/3250136524 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7fcdec06b170 msgr2=0x7fcdec06d630 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1458543709 2023-10-16T12:28:57.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.824+0000 7fcdfcff9700 1 -- 172.21.15.16:0/3250136524 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 6517+0+0 (secure 0 0 0) 0x7fcdf8089c20 con 0x7fce0015d8d0 2023-10-16T12:28:57.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.824+0000 7fcdfffff700 1 --2- 172.21.15.16:0/3250136524 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7fcdec06b170 0x7fcdec06d630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:28:57.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:57.829+0000 7fcdfcff9700 1 -- 172.21.15.16:0/3250136524 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcdf80546d0 con 0x7fce0015d8d0 2023-10-16T12:28:58.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.025+0000 7fcdfffff700 1 -- 172.21.15.16:0/3250136524 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7fcdec06b170 msgr2=0x7fcdec06d630 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1458543709 2023-10-16T12:28:58.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.025+0000 7fcdfffff700 1 --2- 172.21.15.16:0/3250136524 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7fcdec06b170 0x7fcdec06d630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:28:58.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.090+0000 7fce0673a700 1 -- 172.21.15.16:0/3250136524 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fce0006d250 con 0x7fce0015d8d0 2023-10-16T12:28:58.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.090+0000 7fcdfcff9700 1 -- 172.21.15.16:0/3250136524 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+4903 (secure 0 0 0) 0x7fcdf800ae60 con 0x7fce0015d8d0 2023-10-16T12:28:58.093 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:28:58.093 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":53,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6524,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/854923476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":854923476},{"type":"v1","addr":"172.21.15.49:6834","nonce":854923476}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":7249,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/1186699359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1186699359},{"type":"v1","addr":"172.21.15.16:6836","nonce":1186699359}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":7252,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/22289021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":22289021},{"type":"v1","addr":"172.21.15.16:6837","nonce":22289021}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:27:37.397811+0000","modified":"2023-10-16T12:27:40.428748+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7258},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7258":{"gid":7258,"name":"b","rank":0,"incarnation":49,"state":"up:active","state_seq":3,"addr":"172.21.15.49:6835/729289774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":729289774},{"type":"v1","addr":"172.21.15.49:6835","nonce":729289774}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-10-16T12:28:58.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.092+0000 7fce0673a700 1 -- 172.21.15.16:0/3250136524 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7fcdec06b170 msgr2=0x7fcdec06d630 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:28:58.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.092+0000 7fce0673a700 1 --2- 172.21.15.16:0/3250136524 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7fcdec06b170 0x7fcdec06d630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:58.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.092+0000 7fce0673a700 1 -- 172.21.15.16:0/3250136524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce0015d8d0 msgr2=0x7fce001e4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:28:58.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.092+0000 7fce0673a700 1 --2- 172.21.15.16:0/3250136524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce0015d8d0 0x7fce001e4f70 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7fcdf800cc50 tx=0x7fcdf800cf60 comp rx=0 tx=0).stop 2023-10-16T12:28:58.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.093+0000 7fce0673a700 1 -- 172.21.15.16:0/3250136524 shutdown_connections 2023-10-16T12:28:58.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.093+0000 7fce0673a700 1 --2- 172.21.15.16:0/3250136524 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7fcdec06b170 0x7fcdec06d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:58.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.093+0000 7fce0673a700 1 --2- 172.21.15.16:0/3250136524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce0015d8d0 0x7fce001e4f70 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:58.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.093+0000 7fce0673a700 1 --2- 172.21.15.16:0/3250136524 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fce000d0580 0x7fce000ca700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:58.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.093+0000 7fce0673a700 1 --2- 172.21.15.16:0/3250136524 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fce000d41c0 0x7fce0014ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:58.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.093+0000 7fce0673a700 1 -- 172.21.15.16:0/3250136524 >> 172.21.15.16:0/3250136524 conn(0x7fce000d3630 msgr2=0x7fce0015ca50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:28:58.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.093+0000 7fce0673a700 1 -- 172.21.15.16:0/3250136524 shutdown_connections 2023-10-16T12:28:58.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.093+0000 7fce0673a700 1 -- 172.21.15.16:0/3250136524 wait complete. 2023-10-16T12:28:58.097 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 53 2023-10-16T12:28:58.107 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:28:58.187 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:28:58.185+0000 7fa4e0be9200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:28:58.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.235+0000 7f6ae86fb700 1 Processor -- start 2023-10-16T12:28:58.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.235+0000 7f6ae86fb700 1 -- start start 2023-10-16T12:28:58.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.236+0000 7f6ae86fb700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6ae00cdfa0 0x7f6ae00b2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:58.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.236+0000 7f6ae86fb700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ae00d0fe0 con 0x7f6ae00ce380 2023-10-16T12:28:58.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.236+0000 7f6ae86fb700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ae00d1210 con 0x7f6ae00d0bb0 2023-10-16T12:28:58.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.236+0000 7f6ae86fb700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae00cbc30 con 0x7f6ae00cdfa0 2023-10-16T12:28:58.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.236+0000 7f6ae5c96700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f6ae00ce380 0x7f6ae00d04a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:60606/0 (socket says 172.21.15.16:60606) 2023-10-16T12:28:58.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.236+0000 7f6ae5c96700 1 -- 172.21.15.16:0/4248799991 learned_addr learned my addr 172.21.15.16:0/4248799991 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:28:58.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.236+0000 7f6ae6497700 1 --2- 172.21.15.16:0/4248799991 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6ae00cdfa0 0x7f6ae00b2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:28:58.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.237+0000 7f6ae5495700 1 -- 172.21.15.16:0/4248799991 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1145846190 0 0) 0x7f6ae00d0fe0 con 0x7f6ae00ce380 2023-10-16T12:28:58.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.237+0000 7f6ae5495700 1 -- 172.21.15.16:0/4248799991 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6acc0035b0 con 0x7f6ae00ce380 2023-10-16T12:28:58.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.237+0000 7f6ae5495700 1 -- 172.21.15.16:0/4248799991 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 229145457 0 0) 0x7f6ae00d1210 con 0x7f6ae00d0bb0 2023-10-16T12:28:58.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.237+0000 7f6ae5495700 1 -- 172.21.15.16:0/4248799991 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ae00d0fe0 con 0x7f6ae00d0bb0 2023-10-16T12:28:58.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.237+0000 7f6ae6497700 1 -- 172.21.15.16:0/4248799991 >> v1:172.21.15.49:6789/0 conn(0x7f6ae00d0bb0 legacy=0x7f6ae015d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:28:58.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.237+0000 7f6ae6497700 1 -- 172.21.15.16:0/4248799991 >> v1:172.21.15.16:6789/0 conn(0x7f6ae00ce380 legacy=0x7f6ae00d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:28:58.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.237+0000 7f6ae6497700 1 -- 172.21.15.16:0/4248799991 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ae00cbf10 con 0x7f6ae00cdfa0 2023-10-16T12:28:58.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.237+0000 7f6ae6497700 1 --2- 172.21.15.16:0/4248799991 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6ae00cdfa0 0x7f6ae00b2440 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f6adc009870 tx=0x7f6adc009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=642aebbeeb0c4f8e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:28:58.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.237+0000 7f6ae5495700 1 -- 172.21.15.16:0/4248799991 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6adc011830 con 0x7f6ae00cdfa0 2023-10-16T12:28:58.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.237+0000 7f6ae5495700 1 -- 172.21.15.16:0/4248799991 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6adc011990 con 0x7f6ae00cdfa0 2023-10-16T12:28:58.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.237+0000 7f6ae5495700 1 -- 172.21.15.16:0/4248799991 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6adc011d20 con 0x7f6ae00cdfa0 2023-10-16T12:28:58.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.238+0000 7f6ae86fb700 1 -- 172.21.15.16:0/4248799991 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6ae00cdfa0 msgr2=0x7f6ae00b2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:28:58.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.238+0000 7f6ae86fb700 1 --2- 172.21.15.16:0/4248799991 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6ae00cdfa0 0x7f6ae00b2440 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f6adc009870 tx=0x7f6adc009b80 comp rx=0 tx=0).stop 2023-10-16T12:28:58.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.238+0000 7f6ae86fb700 1 -- 172.21.15.16:0/4248799991 shutdown_connections 2023-10-16T12:28:58.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.238+0000 7f6ae86fb700 1 --2- 172.21.15.16:0/4248799991 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6ae00cdfa0 0x7f6ae00b2440 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:58.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.238+0000 7f6ae86fb700 1 -- 172.21.15.16:0/4248799991 >> 172.21.15.16:0/4248799991 conn(0x7f6ae00d14b0 msgr2=0x7f6ae00d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:28:58.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.238+0000 7f6ae86fb700 1 -- 172.21.15.16:0/4248799991 shutdown_connections 2023-10-16T12:28:58.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.238+0000 7f6ae86fb700 1 -- 172.21.15.16:0/4248799991 wait complete. 2023-10-16T12:28:58.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.239+0000 7f6ae86fb700 1 Processor -- start 2023-10-16T12:28:58.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.239+0000 7f6ae86fb700 1 -- start start 2023-10-16T12:28:58.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.240+0000 7f6ae86fb700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6ae00cdfa0 0x7f6ae014f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:58.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.240+0000 7f6ae86fb700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ae00ce380 0x7f6ae00c4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:58.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.240+0000 7f6ae86fb700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6ae00d0bb0 0x7f6ae0150380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:58.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.240+0000 7f6ae86fb700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae00ca550 con 0x7f6ae00ce380 2023-10-16T12:28:58.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.240+0000 7f6ae86fb700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae00c9940 con 0x7f6ae00cdfa0 2023-10-16T12:28:58.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.240+0000 7f6ae86fb700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae00c9c20 con 0x7f6ae00d0bb0 2023-10-16T12:28:58.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.240+0000 7f6ae5c96700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ae00ce380 0x7f6ae00c4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:28:58.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.240+0000 7f6ae5c96700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ae00ce380 0x7f6ae00c4fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43106/0 (socket says 172.21.15.16:43106) 2023-10-16T12:28:58.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.240+0000 7f6ae5c96700 1 -- 172.21.15.16:0/3978368127 learned_addr learned my addr 172.21.15.16:0/3978368127 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:28:58.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.240+0000 7f6ae6497700 1 --2- 172.21.15.16:0/3978368127 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6ae00cdfa0 0x7f6ae014f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:28:58.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.240+0000 7f6ae6c98700 1 --2- 172.21.15.16:0/3978368127 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6ae00d0bb0 0x7f6ae0150380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:28:58.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.241+0000 7f6ae5c96700 1 -- 172.21.15.16:0/3978368127 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6ae00d0bb0 msgr2=0x7f6ae0150380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:28:58.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.241+0000 7f6ae5c96700 1 --2- 172.21.15.16:0/3978368127 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6ae00d0bb0 0x7f6ae0150380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:58.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.241+0000 7f6ae5c96700 1 -- 172.21.15.16:0/3978368127 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6ae00cdfa0 msgr2=0x7f6ae014f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:28:58.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.241+0000 7f6ae5c96700 1 --2- 172.21.15.16:0/3978368127 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6ae00cdfa0 0x7f6ae014f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:58.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.241+0000 7f6ae5c96700 1 -- 172.21.15.16:0/3978368127 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6adc0094a0 con 0x7f6ae00ce380 2023-10-16T12:28:58.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.241+0000 7f6ae5c96700 1 --2- 172.21.15.16:0/3978368127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ae00ce380 0x7f6ae00c4fe0 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f6ad000c700 tx=0x7f6ad000ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:28:58.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.241+0000 7f6ad6ffd700 1 -- 172.21.15.16:0/3978368127 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ad0005d70 con 0x7f6ae00ce380 2023-10-16T12:28:58.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.241+0000 7f6ad6ffd700 1 -- 172.21.15.16:0/3978368127 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ad00070d0 con 0x7f6ae00ce380 2023-10-16T12:28:58.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.241+0000 7f6ad6ffd700 1 -- 172.21.15.16:0/3978368127 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ad0007460 con 0x7f6ae00ce380 2023-10-16T12:28:58.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.241+0000 7f6ae86fb700 1 -- 172.21.15.16:0/3978368127 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ae00c9f00 con 0x7f6ae00ce380 2023-10-16T12:28:58.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.241+0000 7f6ae86fb700 1 -- 172.21.15.16:0/3978368127 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ae014fe00 con 0x7f6ae00ce380 2023-10-16T12:28:58.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.242+0000 7f6ae86fb700 1 -- 172.21.15.16:0/3978368127 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ae00ca700 con 0x7f6ae00ce380 2023-10-16T12:28:58.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.243+0000 7f6ad6ffd700 1 -- 172.21.15.16:0/3978368127 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 58) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f6ad00075c0 con 0x7f6ae00ce380 2023-10-16T12:28:58.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.243+0000 7f6ad6ffd700 1 --2- 172.21.15.16:0/3978368127 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f6ac406b140 0x7f6ac406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:58.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.243+0000 7f6ae6497700 1 -- 172.21.15.16:0/3978368127 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f6ac406b140 msgr2=0x7f6ac406d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1458543709 2023-10-16T12:28:58.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.243+0000 7f6ad6ffd700 1 -- 172.21.15.16:0/3978368127 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f6ad00891a0 con 0x7f6ae00ce380 2023-10-16T12:28:58.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.243+0000 7f6ae6497700 1 --2- 172.21.15.16:0/3978368127 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f6ac406b140 0x7f6ac406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:28:58.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.248+0000 7f6ad6ffd700 1 -- 172.21.15.16:0/3978368127 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6ae00ca700 con 0x7f6ae00ce380 2023-10-16T12:28:58.269 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:28:58.267+0000 7fa4e0be9200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T12:28:58.272 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:28:58.270+0000 7f9831e6b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:28:58.355 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:28:58.354+0000 7f9831e6b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T12:28:58.421 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:28:58.419+0000 7fa4e0be9200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:28:58.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.444+0000 7f6ae6497700 1 -- 172.21.15.16:0/3978368127 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f6ac406b140 msgr2=0x7f6ac406d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1458543709 2023-10-16T12:28:58.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.444+0000 7f6ae6497700 1 --2- 172.21.15.16:0/3978368127 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f6ac406b140 0x7f6ac406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:28:58.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.454+0000 7f6ae86fb700 1 -- 172.21.15.16:0/3978368127 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6ae004ebb0 con 0x7f6ae00ce380 2023-10-16T12:28:58.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.455+0000 7f6ad6ffd700 1 -- 172.21.15.16:0/3978368127 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v115) v1 ==== 74+0+17357 (secure 0 0 0) 0x7f6ad0053bb0 con 0x7f6ae00ce380 2023-10-16T12:28:58.457 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:28:58.458 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":115,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:28:53.284270+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:27:35.907633+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2023-10-16T12:27:36.362861+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.10","mappings":[{"from":1,"to":5}]},{"pgid":"14.2d","mappings":[{"from":1,"to":7}]},{"pgid":"14.32","mappings":[{"from":1,"to":2}]},{"pgid":"14.3c","mappings":[{"from":1,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6833/4089191642":"2023-10-17T12:27:51.745827+0000","172.21.15.16:0/2709574245":"2023-10-17T12:27:51.745827+0000","172.21.15.16:0/769212771":"2023-10-17T12:27:51.745827+0000","172.21.15.16:6832/4089191642":"2023-10-17T12:27:51.745827+0000","172.21.15.16:0/3579843680":"2023-10-17T12:27:51.745827+0000","172.21.15.16:0/1190603852":"2023-10-17T12:27:51.745827+0000","172.21.15.16:0/3144677187":"2023-10-17T12:27:51.745827+0000","172.21.15.16:0/3478352903":"2023-10-17T12:27:51.745827+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:28:58.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.457+0000 7f6ae86fb700 1 -- 172.21.15.16:0/3978368127 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f6ac406b140 msgr2=0x7f6ac406d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:28:58.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.457+0000 7f6ae86fb700 1 --2- 172.21.15.16:0/3978368127 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f6ac406b140 0x7f6ac406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:58.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.457+0000 7f6ae86fb700 1 -- 172.21.15.16:0/3978368127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ae00ce380 msgr2=0x7f6ae00c4fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:28:58.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.457+0000 7f6ae86fb700 1 --2- 172.21.15.16:0/3978368127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ae00ce380 0x7f6ae00c4fe0 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f6ad000c700 tx=0x7f6ad000ca10 comp rx=0 tx=0).stop 2023-10-16T12:28:58.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.457+0000 7f6ae86fb700 1 -- 172.21.15.16:0/3978368127 shutdown_connections 2023-10-16T12:28:58.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.457+0000 7f6ae86fb700 1 --2- 172.21.15.16:0/3978368127 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f6ac406b140 0x7f6ac406d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:58.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.457+0000 7f6ae86fb700 1 --2- 172.21.15.16:0/3978368127 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6ae00d0bb0 0x7f6ae0150380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:58.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.457+0000 7f6ae86fb700 1 --2- 172.21.15.16:0/3978368127 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6ae00cdfa0 0x7f6ae014f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:58.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.457+0000 7f6ae86fb700 1 --2- 172.21.15.16:0/3978368127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ae00ce380 0x7f6ae00c4fe0 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:58.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.457+0000 7f6ae86fb700 1 -- 172.21.15.16:0/3978368127 >> 172.21.15.16:0/3978368127 conn(0x7f6ae00d14b0 msgr2=0x7f6ae006eba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:28:58.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.457+0000 7f6ae86fb700 1 -- 172.21.15.16:0/3978368127 shutdown_connections 2023-10-16T12:28:58.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.457+0000 7f6ae86fb700 1 -- 172.21.15.16:0/3978368127 wait complete. 2023-10-16T12:28:58.472 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T12:27:35.907633+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '111', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T12:27:36.362861+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '112', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T12:28:58.473 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:28:58.511 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:28:58.509+0000 7f9831e6b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:28:58.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.600+0000 7f392500b700 1 Processor -- start 2023-10-16T12:28:58.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.601+0000 7f392500b700 1 -- start start 2023-10-16T12:28:58.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.601+0000 7f392500b700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f39200ce5d0 0x7f39200cc190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:58.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.601+0000 7f392500b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f392015f060 0x7f39200cc6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:58.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.601+0000 7f392500b700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39200b2440 con 0x7f392015ec80 2023-10-16T12:28:58.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.601+0000 7f392500b700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39200b2c50 con 0x7f392015f060 2023-10-16T12:28:58.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.601+0000 7f392500b700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39200d00e0 con 0x7f39200ce5d0 2023-10-16T12:28:58.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.602+0000 7f391f59e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f392015f060 0x7f39200cc6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:28:58.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.602+0000 7f391f59e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f392015f060 0x7f39200cc6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43138/0 (socket says 172.21.15.16:43138) 2023-10-16T12:28:58.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.602+0000 7f391f59e700 1 -- 172.21.15.16:0/1487371229 learned_addr learned my addr 172.21.15.16:0/1487371229 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:28:58.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.602+0000 7f391e59c700 1 --2- 172.21.15.16:0/1487371229 >> v2:172.21.15.49:3300/0 conn(0x7f39200ce5d0 0x7f39200cc190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:28:58.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.602+0000 7f391f59e700 1 -- 172.21.15.16:0/1487371229 >> v2:172.21.15.49:3300/0 conn(0x7f39200ce5d0 msgr2=0x7f39200cc190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:28:58.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.602+0000 7f391f59e700 1 --2- 172.21.15.16:0/1487371229 >> v2:172.21.15.49:3300/0 conn(0x7f39200ce5d0 0x7f39200cc190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:58.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.602+0000 7f391f59e700 1 -- 172.21.15.16:0/1487371229 >> v1:172.21.15.49:6789/0 conn(0x7f392015ec80 legacy=0x7f39200cdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:28:58.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.602+0000 7f391f59e700 1 -- 172.21.15.16:0/1487371229 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39200cef60 con 0x7f392015f060 2023-10-16T12:28:58.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.602+0000 7f391f59e700 1 --2- 172.21.15.16:0/1487371229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f392015f060 0x7f39200cc6d0 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f3918009e50 tx=0x7f391800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32e0ae81ae3f6d6b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:28:58.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.602+0000 7f391dd9b700 1 -- 172.21.15.16:0/1487371229 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f391800e070 con 0x7f392015f060 2023-10-16T12:28:58.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.602+0000 7f391dd9b700 1 -- 172.21.15.16:0/1487371229 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f391800b8d0 con 0x7f392015f060 2023-10-16T12:28:58.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.602+0000 7f391dd9b700 1 -- 172.21.15.16:0/1487371229 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f391800bc80 con 0x7f392015f060 2023-10-16T12:28:58.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.603+0000 7f392500b700 1 -- 172.21.15.16:0/1487371229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f392015f060 msgr2=0x7f39200cc6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:28:58.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.603+0000 7f392500b700 1 --2- 172.21.15.16:0/1487371229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f392015f060 0x7f39200cc6d0 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f3918009e50 tx=0x7f391800b330 comp rx=0 tx=0).stop 2023-10-16T12:28:58.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.603+0000 7f392500b700 1 -- 172.21.15.16:0/1487371229 shutdown_connections 2023-10-16T12:28:58.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.603+0000 7f392500b700 1 --2- 172.21.15.16:0/1487371229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f392015f060 0x7f39200cc6d0 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:58.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.603+0000 7f392500b700 1 --2- 172.21.15.16:0/1487371229 >> v2:172.21.15.49:3300/0 conn(0x7f39200ce5d0 0x7f39200cc190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:58.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.603+0000 7f392500b700 1 -- 172.21.15.16:0/1487371229 >> 172.21.15.16:0/1487371229 conn(0x7f39200d15f0 msgr2=0x7f39200d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:28:58.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.603+0000 7f392500b700 1 -- 172.21.15.16:0/1487371229 shutdown_connections 2023-10-16T12:28:58.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.603+0000 7f392500b700 1 -- 172.21.15.16:0/1487371229 wait complete. 2023-10-16T12:28:58.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.604+0000 7f392500b700 1 Processor -- start 2023-10-16T12:28:58.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.604+0000 7f392500b700 1 -- start start 2023-10-16T12:28:58.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.605+0000 7f392500b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f39200ce5d0 0x7f392015ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:58.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.605+0000 7f392500b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f392015ec80 0x7f39201e53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:58.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.605+0000 7f392500b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f392015f060 0x7f39200c6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:58.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.605+0000 7f392500b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39201e5910 con 0x7f392015ec80 2023-10-16T12:28:58.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.605+0000 7f392500b700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f392015c070 con 0x7f392015f060 2023-10-16T12:28:58.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.605+0000 7f392500b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f392015c350 con 0x7f39200ce5d0 2023-10-16T12:28:58.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.605+0000 7f391e59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f392015ec80 0x7f39201e53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:28:58.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.605+0000 7f391e59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f392015ec80 0x7f39201e53d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43142/0 (socket says 172.21.15.16:43142) 2023-10-16T12:28:58.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.605+0000 7f391e59c700 1 -- 172.21.15.16:0/811937200 learned_addr learned my addr 172.21.15.16:0/811937200 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:28:58.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.605+0000 7f391f59e700 1 --2- 172.21.15.16:0/811937200 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f392015f060 0x7f39200c6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:28:58.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.605+0000 7f391ed9d700 1 --2- 172.21.15.16:0/811937200 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f39200ce5d0 0x7f392015ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:28:58.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.606+0000 7f391e59c700 1 -- 172.21.15.16:0/811937200 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f39200ce5d0 msgr2=0x7f392015ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:28:58.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.606+0000 7f391e59c700 1 --2- 172.21.15.16:0/811937200 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f39200ce5d0 0x7f392015ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:58.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.606+0000 7f391e59c700 1 -- 172.21.15.16:0/811937200 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f392015f060 msgr2=0x7f39200c6ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:28:58.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.606+0000 7f391e59c700 1 --2- 172.21.15.16:0/811937200 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f392015f060 0x7f39200c6ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:58.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.606+0000 7f391e59c700 1 -- 172.21.15.16:0/811937200 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3918009b30 con 0x7f392015ec80 2023-10-16T12:28:58.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.606+0000 7f391f59e700 1 --2- 172.21.15.16:0/811937200 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f392015f060 0x7f39200c6ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:28:58.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.606+0000 7f391e59c700 1 --2- 172.21.15.16:0/811937200 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f392015ec80 0x7f39201e53d0 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f391400e9b0 tx=0x7f391400ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:28:58.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.606+0000 7f39077fe700 1 -- 172.21.15.16:0/811937200 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3914009710 con 0x7f392015ec80 2023-10-16T12:28:58.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.606+0000 7f39077fe700 1 -- 172.21.15.16:0/811937200 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f391400ee50 con 0x7f392015ec80 2023-10-16T12:28:58.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.606+0000 7f392500b700 1 -- 172.21.15.16:0/811937200 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f392015c630 con 0x7f392015ec80 2023-10-16T12:28:58.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.606+0000 7f39077fe700 1 -- 172.21.15.16:0/811937200 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f391400ca50 con 0x7f392015ec80 2023-10-16T12:28:58.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.606+0000 7f392500b700 1 -- 172.21.15.16:0/811937200 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39200c8df0 con 0x7f392015ec80 2023-10-16T12:28:58.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.607+0000 7f392500b700 1 -- 172.21.15.16:0/811937200 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f392004ebb0 con 0x7f392015ec80 2023-10-16T12:28:58.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.608+0000 7f39077fe700 1 -- 172.21.15.16:0/811937200 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 58) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f391400cbb0 con 0x7f392015ec80 2023-10-16T12:28:58.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.608+0000 7f39077fe700 1 --2- 172.21.15.16:0/811937200 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f390c06b140 0x7f390c06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:58.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.608+0000 7f391ed9d700 1 -- 172.21.15.16:0/811937200 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f390c06b140 msgr2=0x7f390c06d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1458543709 2023-10-16T12:28:58.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.608+0000 7f39077fe700 1 -- 172.21.15.16:0/811937200 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f3914089c90 con 0x7f392015ec80 2023-10-16T12:28:58.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.608+0000 7f391ed9d700 1 --2- 172.21.15.16:0/811937200 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f390c06b140 0x7f390c06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:28:58.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.613+0000 7f39077fe700 1 -- 172.21.15.16:0/811937200 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3914054740 con 0x7f392015ec80 2023-10-16T12:28:58.657 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:28:58.655+0000 7fa4e0be9200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T12:28:58.734 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:28:58.732+0000 7fa4e0be9200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T12:28:58.769 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:28:58.767+0000 7f9831e6b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T12:28:58.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.809+0000 7f391ed9d700 1 -- 172.21.15.16:0/811937200 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f390c06b140 msgr2=0x7f390c06d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1458543709 2023-10-16T12:28:58.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.809+0000 7f391ed9d700 1 --2- 172.21.15.16:0/811937200 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f390c06b140 0x7f390c06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:28:58.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.830+0000 7f392500b700 1 -- 172.21.15.16:0/811937200 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f39200b6890 con 0x7f392015ec80 2023-10-16T12:28:58.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.831+0000 7f39077fe700 1 -- 172.21.15.16:0/811937200 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v115) v1 ==== 93+0+10 (secure 0 0 0) 0x7f39140210a0 con 0x7f392015ec80 2023-10-16T12:28:58.835 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T12:28:58.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.835+0000 7f392500b700 1 -- 172.21.15.16:0/811937200 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f390c06b140 msgr2=0x7f390c06d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:28:58.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.835+0000 7f392500b700 1 --2- 172.21.15.16:0/811937200 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f390c06b140 0x7f390c06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:58.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.835+0000 7f392500b700 1 -- 172.21.15.16:0/811937200 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f392015ec80 msgr2=0x7f39201e53d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:28:58.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.835+0000 7f392500b700 1 --2- 172.21.15.16:0/811937200 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f392015ec80 0x7f39201e53d0 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f391400e9b0 tx=0x7f391400ecc0 comp rx=0 tx=0).stop 2023-10-16T12:28:58.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.835+0000 7f392500b700 1 -- 172.21.15.16:0/811937200 shutdown_connections 2023-10-16T12:28:58.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.835+0000 7f392500b700 1 --2- 172.21.15.16:0/811937200 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f390c06b140 0x7f390c06d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:58.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.835+0000 7f392500b700 1 --2- 172.21.15.16:0/811937200 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f392015f060 0x7f39200c6ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:58.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.835+0000 7f392500b700 1 --2- 172.21.15.16:0/811937200 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f39200ce5d0 0x7f392015ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:58.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.835+0000 7f392500b700 1 --2- 172.21.15.16:0/811937200 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f392015ec80 0x7f39201e53d0 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:58.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.835+0000 7f392500b700 1 -- 172.21.15.16:0/811937200 >> 172.21.15.16:0/811937200 conn(0x7f39200d15f0 msgr2=0x7f39200b6db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:28:58.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.835+0000 7f392500b700 1 -- 172.21.15.16:0/811937200 shutdown_connections 2023-10-16T12:28:58.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.835+0000 7f392500b700 1 -- 172.21.15.16:0/811937200 wait complete. 2023-10-16T12:28:58.850 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T12:28:58.851 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:28:58.849+0000 7f9831e6b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T12:28:58.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.982+0000 7f83bd7c6700 1 Processor -- start 2023-10-16T12:28:58.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.982+0000 7f83bd7c6700 1 -- start start 2023-10-16T12:28:58.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.983+0000 7f83bd7c6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f83b80ce6a0 0x7f83b806c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:58.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.983+0000 7f83bd7c6700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f83b80d2110 0x7f83b806c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:58.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.983+0000 7f83bd7c6700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83b80b20f0 con 0x7f83b80b2620 2023-10-16T12:28:58.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.983+0000 7f83bd7c6700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83b80af1f0 con 0x7f83b80ce6a0 2023-10-16T12:28:58.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.983+0000 7f83bd7c6700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83b80d03d0 con 0x7f83b80d2110 2023-10-16T12:28:58.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.983+0000 7f83b6ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f83b80ce6a0 0x7f83b806c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:28:58.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.983+0000 7f83b6ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f83b80ce6a0 0x7f83b806c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43150/0 (socket says 172.21.15.16:43150) 2023-10-16T12:28:58.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.983+0000 7f83b6ffd700 1 -- 172.21.15.16:0/2281085139 learned_addr learned my addr 172.21.15.16:0/2281085139 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:28:58.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.983+0000 7f83b67fc700 1 --2- 172.21.15.16:0/2281085139 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f83b80d2110 0x7f83b806c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:28:58.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.984+0000 7f83b6ffd700 1 -- 172.21.15.16:0/2281085139 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f83b80d2110 msgr2=0x7f83b806c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:28:58.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.984+0000 7f83b6ffd700 1 --2- 172.21.15.16:0/2281085139 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f83b80d2110 0x7f83b806c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:58.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.984+0000 7f83b6ffd700 1 -- 172.21.15.16:0/2281085139 >> v1:172.21.15.49:6789/0 conn(0x7f83b80b2620 legacy=0x7f83b80b0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:28:58.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.984+0000 7f83b6ffd700 1 -- 172.21.15.16:0/2281085139 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83b80b2340 con 0x7f83b80ce6a0 2023-10-16T12:28:58.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.984+0000 7f83b6ffd700 1 --2- 172.21.15.16:0/2281085139 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83b80ce6a0 0x7f83b806c300 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f83a80098f0 tx=0x7f83a8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=332ae79a3eebb9f2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:28:58.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.984+0000 7f83b5ffb700 1 -- 172.21.15.16:0/2281085139 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a8011830 con 0x7f83b80ce6a0 2023-10-16T12:28:58.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.984+0000 7f83b5ffb700 1 -- 172.21.15.16:0/2281085139 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83a8011990 con 0x7f83b80ce6a0 2023-10-16T12:28:58.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.984+0000 7f83b5ffb700 1 -- 172.21.15.16:0/2281085139 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a8011d20 con 0x7f83b80ce6a0 2023-10-16T12:28:58.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.985+0000 7f83bd7c6700 1 -- 172.21.15.16:0/2281085139 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83b80ce6a0 msgr2=0x7f83b806c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:28:58.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.985+0000 7f83bd7c6700 1 --2- 172.21.15.16:0/2281085139 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83b80ce6a0 0x7f83b806c300 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f83a80098f0 tx=0x7f83a8009c00 comp rx=0 tx=0).stop 2023-10-16T12:28:58.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.985+0000 7f83bd7c6700 1 -- 172.21.15.16:0/2281085139 shutdown_connections 2023-10-16T12:28:58.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.985+0000 7f83bd7c6700 1 --2- 172.21.15.16:0/2281085139 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83b80ce6a0 0x7f83b806c300 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:58.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.985+0000 7f83bd7c6700 1 --2- 172.21.15.16:0/2281085139 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f83b80d2110 0x7f83b806c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:58.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.985+0000 7f83bd7c6700 1 -- 172.21.15.16:0/2281085139 >> 172.21.15.16:0/2281085139 conn(0x7f83b80d15f0 msgr2=0x7f83b80d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:28:58.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.985+0000 7f83bd7c6700 1 -- 172.21.15.16:0/2281085139 shutdown_connections 2023-10-16T12:28:58.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.985+0000 7f83bd7c6700 1 -- 172.21.15.16:0/2281085139 wait complete. 2023-10-16T12:28:58.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.986+0000 7f83bd7c6700 1 Processor -- start 2023-10-16T12:28:58.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.986+0000 7f83bd7c6700 1 -- start start 2023-10-16T12:28:58.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.987+0000 7f83bd7c6700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f83b80b2620 0x7f83b80b4b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:58.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.987+0000 7f83bd7c6700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f83b80ce6a0 0x7f83b80b5090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:58.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.987+0000 7f83bd7c6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83b80d2110 0x7f83b81f3160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:58.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.987+0000 7f83bd7c6700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b80b55d0 con 0x7f83b80d2110 2023-10-16T12:28:58.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.987+0000 7f83bd7c6700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b80c8760 con 0x7f83b80b2620 2023-10-16T12:28:58.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.987+0000 7f83bd7c6700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83b80b5800 con 0x7f83b80ce6a0 2023-10-16T12:28:58.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.987+0000 7f83b77fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83b80d2110 0x7f83b81f3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:28:58.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.987+0000 7f83b6ffd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f83b80b2620 0x7f83b80b4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:28:58.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.987+0000 7f83b67fc700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f83b80ce6a0 0x7f83b80b5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:28:58.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.987+0000 7f83b67fc700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f83b80ce6a0 0x7f83b80b5090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:59648/0 (socket says 172.21.15.16:59648) 2023-10-16T12:28:58.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.987+0000 7f83b77fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83b80d2110 0x7f83b81f3160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43160/0 (socket says 172.21.15.16:43160) 2023-10-16T12:28:58.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.987+0000 7f83b67fc700 1 -- 172.21.15.16:0/448141860 learned_addr learned my addr 172.21.15.16:0/448141860 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:28:58.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.987+0000 7f83b67fc700 1 -- 172.21.15.16:0/448141860 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f83b80b2620 msgr2=0x7f83b80b4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:28:58.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.987+0000 7f83b67fc700 1 --2- 172.21.15.16:0/448141860 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f83b80b2620 0x7f83b80b4b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:58.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.987+0000 7f83b67fc700 1 -- 172.21.15.16:0/448141860 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83b80d2110 msgr2=0x7f83b81f3160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:28:58.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.987+0000 7f83b67fc700 1 --2- 172.21.15.16:0/448141860 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83b80d2110 0x7f83b81f3160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:59.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.987+0000 7f83b67fc700 1 -- 172.21.15.16:0/448141860 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83a8009550 con 0x7f83b80ce6a0 2023-10-16T12:28:59.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.988+0000 7f83b67fc700 1 --2- 172.21.15.16:0/448141860 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f83b80ce6a0 0x7f83b80b5090 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f83ac00e870 tx=0x7f83ac00eb80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:28:59.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.988+0000 7f839f7fe700 1 -- 172.21.15.16:0/448141860 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83ac009590 con 0x7f83b80ce6a0 2023-10-16T12:28:59.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.988+0000 7f839f7fe700 1 -- 172.21.15.16:0/448141860 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83ac013680 con 0x7f83b80ce6a0 2023-10-16T12:28:59.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.988+0000 7f83bd7c6700 1 -- 172.21.15.16:0/448141860 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83b80c89d0 con 0x7f83b80ce6a0 2023-10-16T12:28:59.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.988+0000 7f839f7fe700 1 -- 172.21.15.16:0/448141860 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83ac0139f0 con 0x7f83b80ce6a0 2023-10-16T12:28:59.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.988+0000 7f83bd7c6700 1 -- 172.21.15.16:0/448141860 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83b80b5ae0 con 0x7f83b80ce6a0 2023-10-16T12:28:59.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.989+0000 7f83bd7c6700 1 -- 172.21.15.16:0/448141860 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83b80c9020 con 0x7f83b80ce6a0 2023-10-16T12:28:59.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.989+0000 7f839f7fe700 1 -- 172.21.15.16:0/448141860 <== mon.2 v2:172.21.15.49:3301/0 4 ==== mgrmap(e 58) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f83ac013b50 con 0x7f83b80ce6a0 2023-10-16T12:28:59.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.990+0000 7f839f7fe700 1 --2- 172.21.15.16:0/448141860 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f83a406b140 0x7f83a406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:59.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.990+0000 7f83b6ffd700 1 -- 172.21.15.16:0/448141860 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f83a406b140 msgr2=0x7f83a406d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1458543709 2023-10-16T12:28:59.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.990+0000 7f839f7fe700 1 -- 172.21.15.16:0/448141860 <== mon.2 v2:172.21.15.49:3301/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f83ac089ba0 con 0x7f83b80ce6a0 2023-10-16T12:28:59.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.990+0000 7f83b6ffd700 1 --2- 172.21.15.16:0/448141860 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f83a406b140 0x7f83a406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:28:59.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:58.995+0000 7f839f7fe700 1 -- 172.21.15.16:0/448141860 <== mon.2 v2:172.21.15.49:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f83ac0545c0 con 0x7f83b80ce6a0 2023-10-16T12:28:59.010 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:28:59.008+0000 7fa4e0be9200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T12:28:59.135 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:28:59.133+0000 7f9831e6b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T12:28:59.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.191+0000 7f83b6ffd700 1 -- 172.21.15.16:0/448141860 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f83a406b140 msgr2=0x7f83a406d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1458543709 2023-10-16T12:28:59.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.191+0000 7f83b6ffd700 1 --2- 172.21.15.16:0/448141860 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f83a406b140 0x7f83a406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:28:59.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.198+0000 7f83bd7c6700 1 -- 172.21.15.16:0/448141860 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f83b80c8590 con 0x7f83b80ce6a0 2023-10-16T12:28:59.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.198+0000 7f839f7fe700 1 -- 172.21.15.16:0/448141860 <== mon.2 v2:172.21.15.49:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v115) v1 ==== 104+0+11 (secure 0 0 0) 0x7f83ac021020 con 0x7f83b80ce6a0 2023-10-16T12:28:59.200 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-10-16T12:28:59.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.201+0000 7f83bd7c6700 1 -- 172.21.15.16:0/448141860 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f83a406b140 msgr2=0x7f83a406d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:28:59.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.201+0000 7f83bd7c6700 1 --2- 172.21.15.16:0/448141860 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f83a406b140 0x7f83a406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:59.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.201+0000 7f83bd7c6700 1 -- 172.21.15.16:0/448141860 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f83b80ce6a0 msgr2=0x7f83b80b5090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:28:59.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.201+0000 7f83bd7c6700 1 --2- 172.21.15.16:0/448141860 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f83b80ce6a0 0x7f83b80b5090 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f83ac00e870 tx=0x7f83ac00eb80 comp rx=0 tx=0).stop 2023-10-16T12:28:59.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.201+0000 7f83bd7c6700 1 -- 172.21.15.16:0/448141860 shutdown_connections 2023-10-16T12:28:59.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.201+0000 7f83bd7c6700 1 --2- 172.21.15.16:0/448141860 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f83a406b140 0x7f83a406d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:59.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.201+0000 7f83bd7c6700 1 --2- 172.21.15.16:0/448141860 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83b80d2110 0x7f83b81f3160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:59.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.201+0000 7f83bd7c6700 1 --2- 172.21.15.16:0/448141860 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f83b80b2620 0x7f83b80b4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:59.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.201+0000 7f83bd7c6700 1 --2- 172.21.15.16:0/448141860 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f83b80ce6a0 0x7f83b80b5090 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:59.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.201+0000 7f83bd7c6700 1 -- 172.21.15.16:0/448141860 >> 172.21.15.16:0/448141860 conn(0x7f83b80d15f0 msgr2=0x7f83b80cf880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:28:59.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.201+0000 7f83bd7c6700 1 -- 172.21.15.16:0/448141860 shutdown_connections 2023-10-16T12:28:59.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.201+0000 7f83bd7c6700 1 -- 172.21.15.16:0/448141860 wait complete. 2023-10-16T12:28:59.215 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T12:28:59.242 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:28:59.239+0000 7fa4e0be9200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T12:28:59.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.343+0000 7fe429ff8700 1 Processor -- start 2023-10-16T12:28:59.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.343+0000 7fe429ff8700 1 -- start start 2023-10-16T12:28:59.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.344+0000 7fe429ff8700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe424159de0 0x7fe42415a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:59.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.344+0000 7fe429ff8700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe42415dda0 con 0x7fe4240cb1c0 2023-10-16T12:28:59.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.344+0000 7fe429ff8700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe42415e080 con 0x7fe4240cf4c0 2023-10-16T12:28:59.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.344+0000 7fe429ff8700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe42415ea20 con 0x7fe424159de0 2023-10-16T12:28:59.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.344+0000 7fe4237fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe424159de0 0x7fe42415a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:28:59.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.344+0000 7fe4237fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe424159de0 0x7fe42415a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43166/0 (socket says 172.21.15.16:43166) 2023-10-16T12:28:59.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.344+0000 7fe4237fe700 1 -- 172.21.15.16:0/3587441957 learned_addr learned my addr 172.21.15.16:0/3587441957 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:28:59.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.344+0000 7fe423fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe4240cb1c0 0x7fe42406b1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:60676/0 (socket says 172.21.15.16:60676) 2023-10-16T12:28:59.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.344+0000 7fe4237fe700 1 -- 172.21.15.16:0/3587441957 >> v1:172.21.15.49:6789/0 conn(0x7fe4240cf4c0 legacy=0x7fe42406aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:28:59.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.344+0000 7fe4237fe700 1 -- 172.21.15.16:0/3587441957 >> v1:172.21.15.16:6789/0 conn(0x7fe4240cb1c0 legacy=0x7fe42406b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:28:59.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.344+0000 7fe4227fc700 1 -- 172.21.15.16:0/3587441957 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2769423402 0 0) 0x7fe42415e080 con 0x7fe4240cf4c0 2023-10-16T12:28:59.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.344+0000 7fe4237fe700 1 -- 172.21.15.16:0/3587441957 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe42415e360 con 0x7fe424159de0 2023-10-16T12:28:59.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.345+0000 7fe4237fe700 1 --2- 172.21.15.16:0/3587441957 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe424159de0 0x7fe42415a1c0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7fe414009e50 tx=0x7fe41400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe5e0acf26bb2589 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:28:59.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.345+0000 7fe4227fc700 1 -- 172.21.15.16:0/3587441957 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe41400e070 con 0x7fe424159de0 2023-10-16T12:28:59.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.345+0000 7fe4227fc700 1 -- 172.21.15.16:0/3587441957 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe41400b9c0 con 0x7fe424159de0 2023-10-16T12:28:59.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.345+0000 7fe4227fc700 1 -- 172.21.15.16:0/3587441957 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe41400bd70 con 0x7fe424159de0 2023-10-16T12:28:59.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.345+0000 7fe429ff8700 1 -- 172.21.15.16:0/3587441957 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe424159de0 msgr2=0x7fe42415a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:28:59.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.345+0000 7fe429ff8700 1 --2- 172.21.15.16:0/3587441957 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe424159de0 0x7fe42415a1c0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7fe414009e50 tx=0x7fe41400b330 comp rx=0 tx=0).stop 2023-10-16T12:28:59.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.345+0000 7fe429ff8700 1 -- 172.21.15.16:0/3587441957 shutdown_connections 2023-10-16T12:28:59.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.345+0000 7fe429ff8700 1 --2- 172.21.15.16:0/3587441957 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe424159de0 0x7fe42415a1c0 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:59.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.345+0000 7fe429ff8700 1 -- 172.21.15.16:0/3587441957 >> 172.21.15.16:0/3587441957 conn(0x7fe4240cc5e0 msgr2=0x7fe4240cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:28:59.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.346+0000 7fe429ff8700 1 -- 172.21.15.16:0/3587441957 shutdown_connections 2023-10-16T12:28:59.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.346+0000 7fe429ff8700 1 -- 172.21.15.16:0/3587441957 wait complete. 2023-10-16T12:28:59.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.347+0000 7fe429ff8700 1 Processor -- start 2023-10-16T12:28:59.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.347+0000 7fe429ff8700 1 -- start start 2023-10-16T12:28:59.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.348+0000 7fe429ff8700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe4240cb1c0 0x7fe4240c5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:59.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.348+0000 7fe429ff8700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe4240cf4c0 0x7fe4240af660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:59.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.348+0000 7fe429ff8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe424159de0 0x7fe4240afba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:59.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.348+0000 7fe429ff8700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4240c42c0 con 0x7fe424159de0 2023-10-16T12:28:59.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.348+0000 7fe429ff8700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4240c45a0 con 0x7fe4240cb1c0 2023-10-16T12:28:59.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.348+0000 7fe429ff8700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4240c4880 con 0x7fe4240cf4c0 2023-10-16T12:28:59.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.348+0000 7fe423fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe424159de0 0x7fe4240afba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:28:59.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.348+0000 7fe423fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe424159de0 0x7fe4240afba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43176/0 (socket says 172.21.15.16:43176) 2023-10-16T12:28:59.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.348+0000 7fe4237fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe4240cb1c0 0x7fe4240c5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:28:59.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.348+0000 7fe423fff700 1 -- 172.21.15.16:0/470110621 learned_addr learned my addr 172.21.15.16:0/470110621 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:28:59.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.348+0000 7fe422ffd700 1 --2- 172.21.15.16:0/470110621 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe4240cf4c0 0x7fe4240af660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:28:59.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.348+0000 7fe423fff700 1 -- 172.21.15.16:0/470110621 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe4240cf4c0 msgr2=0x7fe4240af660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:28:59.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.348+0000 7fe423fff700 1 --2- 172.21.15.16:0/470110621 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe4240cf4c0 0x7fe4240af660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:59.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.348+0000 7fe423fff700 1 -- 172.21.15.16:0/470110621 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe4240cb1c0 msgr2=0x7fe4240c5920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:28:59.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.348+0000 7fe423fff700 1 --2- 172.21.15.16:0/470110621 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe4240cb1c0 0x7fe4240c5920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:59.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.348+0000 7fe423fff700 1 -- 172.21.15.16:0/470110621 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe414009b30 con 0x7fe424159de0 2023-10-16T12:28:59.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.348+0000 7fe423fff700 1 --2- 172.21.15.16:0/470110621 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe424159de0 0x7fe4240afba0 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7fe41c00c430 tx=0x7fe41c00c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:28:59.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.349+0000 7fe41bfff700 1 -- 172.21.15.16:0/470110621 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe41c005ab0 con 0x7fe424159de0 2023-10-16T12:28:59.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.349+0000 7fe41bfff700 1 -- 172.21.15.16:0/470110621 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe41c005c10 con 0x7fe424159de0 2023-10-16T12:28:59.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.349+0000 7fe41bfff700 1 -- 172.21.15.16:0/470110621 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe41c0070d0 con 0x7fe424159de0 2023-10-16T12:28:59.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.349+0000 7fe429ff8700 1 -- 172.21.15.16:0/470110621 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe424137730 con 0x7fe424159de0 2023-10-16T12:28:59.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.349+0000 7fe429ff8700 1 -- 172.21.15.16:0/470110621 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4240c3670 con 0x7fe424159de0 2023-10-16T12:28:59.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.350+0000 7fe429ff8700 1 -- 172.21.15.16:0/470110621 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4240c4ac0 con 0x7fe424159de0 2023-10-16T12:28:59.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.350+0000 7fe41bfff700 1 -- 172.21.15.16:0/470110621 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 58) v1 ==== 89445+0+0 (secure 0 0 0) 0x7fe41c007230 con 0x7fe424159de0 2023-10-16T12:28:59.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.351+0000 7fe41bfff700 1 --2- 172.21.15.16:0/470110621 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7fe40406b140 0x7fe40406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:59.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.351+0000 7fe4237fe700 1 -- 172.21.15.16:0/470110621 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7fe40406b140 msgr2=0x7fe40406d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1458543709 2023-10-16T12:28:59.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.351+0000 7fe41bfff700 1 -- 172.21.15.16:0/470110621 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 6517+0+0 (secure 0 0 0) 0x7fe41c088f50 con 0x7fe424159de0 2023-10-16T12:28:59.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.351+0000 7fe4237fe700 1 --2- 172.21.15.16:0/470110621 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7fe40406b140 0x7fe40406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:28:59.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.355+0000 7fe41bfff700 1 -- 172.21.15.16:0/470110621 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe4240c4ac0 con 0x7fe424159de0 2023-10-16T12:28:59.374 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:28:59.372+0000 7f9831e6b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T12:28:59.399 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:28:59.396+0000 7fa4e0be9200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T12:28:59.518 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:28:59.516+0000 7fa4e0be9200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T12:28:59.532 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:28:59.531+0000 7f9831e6b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T12:28:59.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.552+0000 7fe4237fe700 1 -- 172.21.15.16:0/470110621 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7fe40406b140 msgr2=0x7fe40406d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1458543709 2023-10-16T12:28:59.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.552+0000 7fe4237fe700 1 --2- 172.21.15.16:0/470110621 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7fe40406b140 0x7fe40406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:28:59.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.559+0000 7fe429ff8700 1 -- 172.21.15.16:0/470110621 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe424137590 con 0x7fe424159de0 2023-10-16T12:28:59.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.560+0000 7fe41bfff700 1 -- 172.21.15.16:0/470110621 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v115) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe424137590 con 0x7fe424159de0 2023-10-16T12:28:59.562 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-10-16T12:28:59.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.562+0000 7fe429ff8700 1 -- 172.21.15.16:0/470110621 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7fe40406b140 msgr2=0x7fe40406d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:28:59.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.562+0000 7fe429ff8700 1 --2- 172.21.15.16:0/470110621 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7fe40406b140 0x7fe40406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:59.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.562+0000 7fe429ff8700 1 -- 172.21.15.16:0/470110621 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe424159de0 msgr2=0x7fe4240afba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:28:59.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.562+0000 7fe429ff8700 1 --2- 172.21.15.16:0/470110621 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe424159de0 0x7fe4240afba0 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7fe41c00c430 tx=0x7fe41c00c740 comp rx=0 tx=0).stop 2023-10-16T12:28:59.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.562+0000 7fe429ff8700 1 -- 172.21.15.16:0/470110621 shutdown_connections 2023-10-16T12:28:59.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.562+0000 7fe429ff8700 1 --2- 172.21.15.16:0/470110621 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7fe40406b140 0x7fe40406d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:59.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.562+0000 7fe429ff8700 1 --2- 172.21.15.16:0/470110621 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe424159de0 0x7fe4240afba0 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:59.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.562+0000 7fe429ff8700 1 --2- 172.21.15.16:0/470110621 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe4240cb1c0 0x7fe4240c5920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:59.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.562+0000 7fe429ff8700 1 --2- 172.21.15.16:0/470110621 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe4240cf4c0 0x7fe4240af660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:59.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.562+0000 7fe429ff8700 1 -- 172.21.15.16:0/470110621 >> 172.21.15.16:0/470110621 conn(0x7fe4240cc5e0 msgr2=0x7fe42406b880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:28:59.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.562+0000 7fe429ff8700 1 -- 172.21.15.16:0/470110621 shutdown_connections 2023-10-16T12:28:59.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.562+0000 7fe429ff8700 1 -- 172.21.15.16:0/470110621 wait complete. 2023-10-16T12:28:59.577 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:28:59.657 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:28:59.656+0000 7f9831e6b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T12:28:59.679 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:28:59.676+0000 7fa4e0be9200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T12:28:59.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.702+0000 7f76813a1700 1 Processor -- start 2023-10-16T12:28:59.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.702+0000 7f76813a1700 1 -- start start 2023-10-16T12:28:59.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.703+0000 7f76813a1700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f767c0b2b50 0x7f767c0d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:59.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.703+0000 7f76813a1700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f767c0d1210 con 0x7f767c0ce380 2023-10-16T12:28:59.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.703+0000 7f76813a1700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f767c0cbc30 con 0x7f767c0cdfa0 2023-10-16T12:28:59.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.703+0000 7f76813a1700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f767c0cbf10 con 0x7f767c0b2b50 2023-10-16T12:28:59.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.703+0000 7f767b7fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f767c0ce380 0x7f767c0d09e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:60692/0 (socket says 172.21.15.16:60692) 2023-10-16T12:28:59.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.703+0000 7f767b7fe700 1 -- 172.21.15.16:0/2835213537 learned_addr learned my addr 172.21.15.16:0/2835213537 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:28:59.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.704+0000 7f767a7fc700 1 --2- 172.21.15.16:0/2835213537 >> v2:172.21.15.49:3300/0 conn(0x7f767c0b2b50 0x7f767c0d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:28:59.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.704+0000 7f7679ffb700 1 -- 172.21.15.16:0/2835213537 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3895460237 0 0) 0x7f767c0d1210 con 0x7f767c0ce380 2023-10-16T12:28:59.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.704+0000 7f7679ffb700 1 -- 172.21.15.16:0/2835213537 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76640035b0 con 0x7f767c0ce380 2023-10-16T12:28:59.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.704+0000 7f767a7fc700 1 -- 172.21.15.16:0/2835213537 >> v1:172.21.15.49:6789/0 conn(0x7f767c0cdfa0 legacy=0x7f767c0b2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:28:59.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.704+0000 7f767a7fc700 1 -- 172.21.15.16:0/2835213537 >> v1:172.21.15.16:6789/0 conn(0x7f767c0ce380 legacy=0x7f767c0d09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:28:59.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.704+0000 7f767a7fc700 1 -- 172.21.15.16:0/2835213537 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f767c0cc1f0 con 0x7f767c0b2b50 2023-10-16T12:28:59.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.704+0000 7f767a7fc700 1 --2- 172.21.15.16:0/2835213537 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f767c0b2b50 0x7f767c0d04a0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f7670009840 tx=0x7f7670009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f8aa4cf706c3143 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:28:59.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.704+0000 7f7679ffb700 1 -- 172.21.15.16:0/2835213537 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7670011830 con 0x7f767c0b2b50 2023-10-16T12:28:59.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.704+0000 7f7679ffb700 1 -- 172.21.15.16:0/2835213537 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7670011990 con 0x7f767c0b2b50 2023-10-16T12:28:59.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.704+0000 7f7679ffb700 1 -- 172.21.15.16:0/2835213537 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7670011d20 con 0x7f767c0b2b50 2023-10-16T12:28:59.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.705+0000 7f76813a1700 1 -- 172.21.15.16:0/2835213537 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f767c0b2b50 msgr2=0x7f767c0d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:28:59.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.705+0000 7f76813a1700 1 --2- 172.21.15.16:0/2835213537 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f767c0b2b50 0x7f767c0d04a0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f7670009840 tx=0x7f7670009b50 comp rx=0 tx=0).stop 2023-10-16T12:28:59.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.705+0000 7f76813a1700 1 -- 172.21.15.16:0/2835213537 shutdown_connections 2023-10-16T12:28:59.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.705+0000 7f76813a1700 1 --2- 172.21.15.16:0/2835213537 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f767c0b2b50 0x7f767c0d04a0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:59.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.705+0000 7f76813a1700 1 -- 172.21.15.16:0/2835213537 >> 172.21.15.16:0/2835213537 conn(0x7f767c0d14b0 msgr2=0x7f767c0d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:28:59.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.705+0000 7f76813a1700 1 -- 172.21.15.16:0/2835213537 shutdown_connections 2023-10-16T12:28:59.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.705+0000 7f76813a1700 1 -- 172.21.15.16:0/2835213537 wait complete. 2023-10-16T12:28:59.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.707+0000 7f76813a1700 1 Processor -- start 2023-10-16T12:28:59.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.707+0000 7f76813a1700 1 -- start start 2023-10-16T12:28:59.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.708+0000 7f76813a1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f767c0b2b50 0x7f767c0c7810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:59.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.708+0000 7f76813a1700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f767c0cdfa0 0x7f767c06fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:59.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.708+0000 7f76813a1700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f767c0ce380 0x7f767c09e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:59.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.708+0000 7f76813a1700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f767c09d350 con 0x7f767c0b2b50 2023-10-16T12:28:59.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.708+0000 7f76813a1700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f767c09d630 con 0x7f767c0ce380 2023-10-16T12:28:59.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.708+0000 7f76813a1700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f767c09d910 con 0x7f767c0cdfa0 2023-10-16T12:28:59.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.708+0000 7f767affd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f767c0b2b50 0x7f767c0c7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:28:59.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.708+0000 7f767affd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f767c0b2b50 0x7f767c0c7810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43188/0 (socket says 172.21.15.16:43188) 2023-10-16T12:28:59.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.708+0000 7f767affd700 1 -- 172.21.15.16:0/3972391187 learned_addr learned my addr 172.21.15.16:0/3972391187 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:28:59.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.708+0000 7f767b7fe700 1 --2- 172.21.15.16:0/3972391187 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f767c0ce380 0x7f767c09e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:28:59.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.708+0000 7f767a7fc700 1 --2- 172.21.15.16:0/3972391187 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f767c0cdfa0 0x7f767c06fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:28:59.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.709+0000 7f767affd700 1 -- 172.21.15.16:0/3972391187 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f767c0cdfa0 msgr2=0x7f767c06fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:28:59.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.709+0000 7f767affd700 1 --2- 172.21.15.16:0/3972391187 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f767c0cdfa0 0x7f767c06fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:59.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.709+0000 7f767affd700 1 -- 172.21.15.16:0/3972391187 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f767c0ce380 msgr2=0x7f767c09e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:28:59.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.709+0000 7f767affd700 1 --2- 172.21.15.16:0/3972391187 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f767c0ce380 0x7f767c09e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:59.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.709+0000 7f767affd700 1 -- 172.21.15.16:0/3972391187 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76700094a0 con 0x7f767c0b2b50 2023-10-16T12:28:59.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.709+0000 7f767affd700 1 --2- 172.21.15.16:0/3972391187 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f767c0b2b50 0x7f767c0c7810 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f766c00c4e0 tx=0x7f766c00c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:28:59.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.709+0000 7f76637fe700 1 -- 172.21.15.16:0/3972391187 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f766c0052d0 con 0x7f767c0b2b50 2023-10-16T12:28:59.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.709+0000 7f76637fe700 1 -- 172.21.15.16:0/3972391187 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f766c005430 con 0x7f767c0b2b50 2023-10-16T12:28:59.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.709+0000 7f76813a1700 1 -- 172.21.15.16:0/3972391187 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f767c09dbf0 con 0x7f767c0b2b50 2023-10-16T12:28:59.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.709+0000 7f76637fe700 1 -- 172.21.15.16:0/3972391187 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f766c0038c0 con 0x7f767c0b2b50 2023-10-16T12:28:59.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.709+0000 7f76813a1700 1 -- 172.21.15.16:0/3972391187 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f767c09e1b0 con 0x7f767c0b2b50 2023-10-16T12:28:59.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.710+0000 7f76813a1700 1 -- 172.21.15.16:0/3972391187 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f767c067870 con 0x7f767c0b2b50 2023-10-16T12:28:59.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.711+0000 7f76637fe700 1 -- 172.21.15.16:0/3972391187 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 58) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f766c003a20 con 0x7f767c0b2b50 2023-10-16T12:28:59.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.711+0000 7f76637fe700 1 --2- 172.21.15.16:0/3972391187 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f766806b140 0x7f766806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:28:59.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.711+0000 7f767a7fc700 1 -- 172.21.15.16:0/3972391187 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f766806b140 msgr2=0x7f766806d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1458543709 2023-10-16T12:28:59.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.711+0000 7f76637fe700 1 -- 172.21.15.16:0/3972391187 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f766c089000 con 0x7f767c0b2b50 2023-10-16T12:28:59.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.711+0000 7f767a7fc700 1 --2- 172.21.15.16:0/3972391187 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f766806b140 0x7f766806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:28:59.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.716+0000 7f76637fe700 1 -- 172.21.15.16:0/3972391187 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f766c053af0 con 0x7f767c0b2b50 2023-10-16T12:28:59.831 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:28:59.829+0000 7f9831e6b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T12:28:59.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.912+0000 7f767a7fc700 1 -- 172.21.15.16:0/3972391187 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f766806b140 msgr2=0x7f766806d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1458543709 2023-10-16T12:28:59.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.912+0000 7f767a7fc700 1 --2- 172.21.15.16:0/3972391187 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f766806b140 0x7f766806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:28:59.949 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:28:59.948+0000 7fa4e0be9200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T12:28:59.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.971+0000 7f76813a1700 1 -- 172.21.15.16:0/3972391187 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f767c06ca70 con 0x7f767c0b2b50 2023-10-16T12:28:59.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.971+0000 7f76637fe700 1 -- 172.21.15.16:0/3972391187 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+4903 (secure 0 0 0) 0x7f766c057110 con 0x7f767c0b2b50 2023-10-16T12:28:59.974 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:28:59.975 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":53,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6524,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/854923476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":854923476},{"type":"v1","addr":"172.21.15.49:6834","nonce":854923476}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":7249,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/1186699359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1186699359},{"type":"v1","addr":"172.21.15.16:6836","nonce":1186699359}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":7252,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/22289021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":22289021},{"type":"v1","addr":"172.21.15.16:6837","nonce":22289021}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:27:37.397811+0000","modified":"2023-10-16T12:27:40.428748+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7258},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7258":{"gid":7258,"name":"b","rank":0,"incarnation":49,"state":"up:active","state_seq":3,"addr":"172.21.15.49:6835/729289774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":729289774},{"type":"v1","addr":"172.21.15.49:6835","nonce":729289774}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-10-16T12:28:59.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.973+0000 7f76813a1700 1 -- 172.21.15.16:0/3972391187 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f766806b140 msgr2=0x7f766806d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:28:59.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.973+0000 7f76813a1700 1 --2- 172.21.15.16:0/3972391187 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f766806b140 0x7f766806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:59.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.973+0000 7f76813a1700 1 -- 172.21.15.16:0/3972391187 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f767c0b2b50 msgr2=0x7f767c0c7810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:28:59.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.973+0000 7f76813a1700 1 --2- 172.21.15.16:0/3972391187 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f767c0b2b50 0x7f767c0c7810 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f766c00c4e0 tx=0x7f766c00c7f0 comp rx=0 tx=0).stop 2023-10-16T12:28:59.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.974+0000 7f76813a1700 1 -- 172.21.15.16:0/3972391187 shutdown_connections 2023-10-16T12:28:59.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.974+0000 7f76813a1700 1 --2- 172.21.15.16:0/3972391187 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f766806b140 0x7f766806d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:59.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.974+0000 7f76813a1700 1 --2- 172.21.15.16:0/3972391187 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f767c0ce380 0x7f767c09e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:59.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.974+0000 7f76813a1700 1 --2- 172.21.15.16:0/3972391187 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f767c0b2b50 0x7f767c0c7810 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:59.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.974+0000 7f76813a1700 1 --2- 172.21.15.16:0/3972391187 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f767c0cdfa0 0x7f767c06fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:28:59.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.974+0000 7f76813a1700 1 -- 172.21.15.16:0/3972391187 >> 172.21.15.16:0/3972391187 conn(0x7f767c0d14b0 msgr2=0x7f767c160ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:28:59.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.974+0000 7f76813a1700 1 -- 172.21.15.16:0/3972391187 shutdown_connections 2023-10-16T12:28:59.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:28:59.974+0000 7f76813a1700 1 -- 172.21.15.16:0/3972391187 wait complete. 2023-10-16T12:28:59.981 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 53 2023-10-16T12:28:59.990 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:29:00.112 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:29:00.110+0000 7f9831e6b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T12:29:00.118 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:29:00.113+0000 7fa4e0be9200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T12:29:00.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.125+0000 7f3057c3b700 1 Processor -- start 2023-10-16T12:29:00.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.126+0000 7f3057c3b700 1 -- start start 2023-10-16T12:29:00.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.127+0000 7f3057c3b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f305015ba10 0x7f30500d0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:00.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.127+0000 7f3057c3b700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30500b0970 con 0x7f305015b630 2023-10-16T12:29:00.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.127+0000 7f3057c3b700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30500cd690 con 0x7f30500ce4f0 2023-10-16T12:29:00.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.127+0000 7f3057c3b700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30500cd970 con 0x7f305015ba10 2023-10-16T12:29:00.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.127+0000 7f30561d8700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f305015ba10 0x7f30500d0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:00.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.127+0000 7f30559d7700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f305015b630 0x7f30500cdde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:60704/0 (socket says 172.21.15.16:60704) 2023-10-16T12:29:00.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.127+0000 7f30559d7700 1 -- 172.21.15.16:0/1201161215 learned_addr learned my addr 172.21.15.16:0/1201161215 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:29:00.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.127+0000 7f30561d8700 1 -- 172.21.15.16:0/1201161215 >> v1:172.21.15.49:6789/0 conn(0x7f30500ce4f0 legacy=0x7f30500b1430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:00.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.127+0000 7f30561d8700 1 -- 172.21.15.16:0/1201161215 >> v1:172.21.15.16:6789/0 conn(0x7f305015b630 legacy=0x7f30500cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:00.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.127+0000 7f30561d8700 1 -- 172.21.15.16:0/1201161215 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30500b0c30 con 0x7f305015ba10 2023-10-16T12:29:00.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.128+0000 7f30561d8700 1 --2- 172.21.15.16:0/1201161215 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f305015ba10 0x7f30500d0380 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f3048009d90 tx=0x7f304800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96394fdac0708bf1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:00.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.128+0000 7f30549d5700 1 -- 172.21.15.16:0/1201161215 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f304800e070 con 0x7f305015ba10 2023-10-16T12:29:00.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.128+0000 7f30549d5700 1 -- 172.21.15.16:0/1201161215 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f304800bbd0 con 0x7f305015ba10 2023-10-16T12:29:00.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.128+0000 7f30549d5700 1 -- 172.21.15.16:0/1201161215 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3048012430 con 0x7f305015ba10 2023-10-16T12:29:00.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.129+0000 7f3057c3b700 1 -- 172.21.15.16:0/1201161215 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f305015ba10 msgr2=0x7f30500d0380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:00.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.129+0000 7f3057c3b700 1 --2- 172.21.15.16:0/1201161215 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f305015ba10 0x7f30500d0380 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f3048009d90 tx=0x7f304800b330 comp rx=0 tx=0).stop 2023-10-16T12:29:00.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.129+0000 7f3057c3b700 1 -- 172.21.15.16:0/1201161215 shutdown_connections 2023-10-16T12:29:00.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.129+0000 7f3057c3b700 1 --2- 172.21.15.16:0/1201161215 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f305015ba10 0x7f30500d0380 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:00.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.129+0000 7f3057c3b700 1 -- 172.21.15.16:0/1201161215 >> 172.21.15.16:0/1201161215 conn(0x7f30500d14b0 msgr2=0x7f30500d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:00.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.129+0000 7f3057c3b700 1 -- 172.21.15.16:0/1201161215 shutdown_connections 2023-10-16T12:29:00.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.129+0000 7f3057c3b700 1 -- 172.21.15.16:0/1201161215 wait complete. 2023-10-16T12:29:00.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.131+0000 7f3057c3b700 1 Processor -- start 2023-10-16T12:29:00.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.131+0000 7f3057c3b700 1 -- start start 2023-10-16T12:29:00.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.132+0000 7f3057c3b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f30500ce4f0 0x7f30500c4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:00.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.132+0000 7f3057c3b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f305015b630 0x7f30500cae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:00.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.132+0000 7f3057c3b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f305015ba10 0x7f305006f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:00.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.132+0000 7f3057c3b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30501f75a0 con 0x7f305015ba10 2023-10-16T12:29:00.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.132+0000 7f3057c3b700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3050152200 con 0x7f30500ce4f0 2023-10-16T12:29:00.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.132+0000 7f3057c3b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30500c8580 con 0x7f305015b630 2023-10-16T12:29:00.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.132+0000 7f30561d8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f305015ba10 0x7f305006f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:00.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.132+0000 7f30561d8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f305015ba10 0x7f305006f870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43208/0 (socket says 172.21.15.16:43208) 2023-10-16T12:29:00.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.132+0000 7f30561d8700 1 -- 172.21.15.16:0/2244943740 learned_addr learned my addr 172.21.15.16:0/2244943740 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:29:00.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.132+0000 7f30559d7700 1 --2- 172.21.15.16:0/2244943740 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f30500ce4f0 0x7f30500c4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:00.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.132+0000 7f30551d6700 1 --2- 172.21.15.16:0/2244943740 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f305015b630 0x7f30500cae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:00.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.133+0000 7f30561d8700 1 -- 172.21.15.16:0/2244943740 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f305015b630 msgr2=0x7f30500cae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:00.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.133+0000 7f30561d8700 1 --2- 172.21.15.16:0/2244943740 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f305015b630 0x7f30500cae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:00.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.133+0000 7f30561d8700 1 -- 172.21.15.16:0/2244943740 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f30500ce4f0 msgr2=0x7f30500c4fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:00.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.133+0000 7f30561d8700 1 --2- 172.21.15.16:0/2244943740 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f30500ce4f0 0x7f30500c4fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:00.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.133+0000 7f30561d8700 1 -- 172.21.15.16:0/2244943740 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3048009a70 con 0x7f305015ba10 2023-10-16T12:29:00.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.133+0000 7f30561d8700 1 --2- 172.21.15.16:0/2244943740 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f305015ba10 0x7f305006f870 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f3048014040 tx=0x7f3048004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:00.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.133+0000 7f30467fc700 1 -- 172.21.15.16:0/2244943740 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3048012d20 con 0x7f305015ba10 2023-10-16T12:29:00.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.133+0000 7f30467fc700 1 -- 172.21.15.16:0/2244943740 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3048020070 con 0x7f305015ba10 2023-10-16T12:29:00.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.133+0000 7f30467fc700 1 -- 172.21.15.16:0/2244943740 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3048012ea0 con 0x7f305015ba10 2023-10-16T12:29:00.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.133+0000 7f3057c3b700 1 -- 172.21.15.16:0/2244943740 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30501f66d0 con 0x7f305015ba10 2023-10-16T12:29:00.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.133+0000 7f3057c3b700 1 -- 172.21.15.16:0/2244943740 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30501f6c00 con 0x7f305015ba10 2023-10-16T12:29:00.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.134+0000 7f3057c3b700 1 -- 172.21.15.16:0/2244943740 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f305004ebb0 con 0x7f305015ba10 2023-10-16T12:29:00.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.135+0000 7f30467fc700 1 -- 172.21.15.16:0/2244943740 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 58) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f3048004280 con 0x7f305015ba10 2023-10-16T12:29:00.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.135+0000 7f30467fc700 1 --2- 172.21.15.16:0/2244943740 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f303406b140 0x7f303406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:00.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.135+0000 7f30559d7700 1 -- 172.21.15.16:0/2244943740 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f303406b140 msgr2=0x7f303406d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1458543709 2023-10-16T12:29:00.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.135+0000 7f30467fc700 1 -- 172.21.15.16:0/2244943740 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f3048022070 con 0x7f305015ba10 2023-10-16T12:29:00.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.135+0000 7f30559d7700 1 --2- 172.21.15.16:0/2244943740 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f303406b140 0x7f303406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:29:00.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.139+0000 7f30467fc700 1 -- 172.21.15.16:0/2244943740 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3048056500 con 0x7f305015ba10 2023-10-16T12:29:00.289 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:29:00.287+0000 7f9831e6b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T12:29:00.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.336+0000 7f30559d7700 1 -- 172.21.15.16:0/2244943740 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f303406b140 msgr2=0x7f303406d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1458543709 2023-10-16T12:29:00.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.336+0000 7f30559d7700 1 --2- 172.21.15.16:0/2244943740 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f303406b140 0x7f303406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:29:00.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.345+0000 7f3057c3b700 1 -- 172.21.15.16:0/2244943740 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f305006b5d0 con 0x7f305015ba10 2023-10-16T12:29:00.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.347+0000 7f30467fc700 1 -- 172.21.15.16:0/2244943740 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v115) v1 ==== 74+0+17357 (secure 0 0 0) 0x7f304802c020 con 0x7f305015ba10 2023-10-16T12:29:00.349 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:29:00.349 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":115,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:28:53.284270+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:27:35.907633+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2023-10-16T12:27:36.362861+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.10","mappings":[{"from":1,"to":5}]},{"pgid":"14.2d","mappings":[{"from":1,"to":7}]},{"pgid":"14.32","mappings":[{"from":1,"to":2}]},{"pgid":"14.3c","mappings":[{"from":1,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6833/4089191642":"2023-10-17T12:27:51.745827+0000","172.21.15.16:0/2709574245":"2023-10-17T12:27:51.745827+0000","172.21.15.16:0/769212771":"2023-10-17T12:27:51.745827+0000","172.21.15.16:6832/4089191642":"2023-10-17T12:27:51.745827+0000","172.21.15.16:0/3579843680":"2023-10-17T12:27:51.745827+0000","172.21.15.16:0/1190603852":"2023-10-17T12:27:51.745827+0000","172.21.15.16:0/3144677187":"2023-10-17T12:27:51.745827+0000","172.21.15.16:0/3478352903":"2023-10-17T12:27:51.745827+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:29:00.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.349+0000 7f3057c3b700 1 -- 172.21.15.16:0/2244943740 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f303406b140 msgr2=0x7f303406d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:29:00.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.349+0000 7f3057c3b700 1 --2- 172.21.15.16:0/2244943740 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f303406b140 0x7f303406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:00.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.349+0000 7f3057c3b700 1 -- 172.21.15.16:0/2244943740 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f305015ba10 msgr2=0x7f305006f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:00.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.349+0000 7f3057c3b700 1 --2- 172.21.15.16:0/2244943740 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f305015ba10 0x7f305006f870 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f3048014040 tx=0x7f3048004ad0 comp rx=0 tx=0).stop 2023-10-16T12:29:00.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.349+0000 7f3057c3b700 1 -- 172.21.15.16:0/2244943740 shutdown_connections 2023-10-16T12:29:00.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.349+0000 7f3057c3b700 1 --2- 172.21.15.16:0/2244943740 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f303406b140 0x7f303406d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:00.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.349+0000 7f3057c3b700 1 --2- 172.21.15.16:0/2244943740 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f305015ba10 0x7f305006f870 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:00.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.349+0000 7f3057c3b700 1 --2- 172.21.15.16:0/2244943740 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f30500ce4f0 0x7f30500c4fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:00.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.349+0000 7f3057c3b700 1 --2- 172.21.15.16:0/2244943740 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f305015b630 0x7f30500cae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:00.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.349+0000 7f3057c3b700 1 -- 172.21.15.16:0/2244943740 >> 172.21.15.16:0/2244943740 conn(0x7f30500d14b0 msgr2=0x7f30500b2400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:00.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.349+0000 7f3057c3b700 1 -- 172.21.15.16:0/2244943740 shutdown_connections 2023-10-16T12:29:00.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.349+0000 7f3057c3b700 1 -- 172.21.15.16:0/2244943740 wait complete. 2023-10-16T12:29:00.364 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-16T12:29:00.364 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:29:00.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.489+0000 7f8daf0bc700 1 Processor -- start 2023-10-16T12:29:00.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.489+0000 7f8daf0bc700 1 -- start start 2023-10-16T12:29:00.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.490+0000 7f8daf0bc700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f8da815d230 0x7f8da80b5a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:00.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.490+0000 7f8daf0bc700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8da815bc80 con 0x7f8da80cf880 2023-10-16T12:29:00.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.490+0000 7f8daf0bc700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8da815b070 con 0x7f8da80cf4a0 2023-10-16T12:29:00.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.490+0000 7f8daf0bc700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8da815b350 con 0x7f8da815d230 2023-10-16T12:29:00.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.490+0000 7f8da7fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f8da80cf880 0x7f8da80b52f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:60720/0 (socket says 172.21.15.16:60720) 2023-10-16T12:29:00.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.490+0000 7f8da7fff700 1 -- 172.21.15.16:0/3525629235 learned_addr learned my addr 172.21.15.16:0/3525629235 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:29:00.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.490+0000 7f8dad659700 1 --2- 172.21.15.16:0/3525629235 >> v2:172.21.15.49:3300/0 conn(0x7f8da815d230 0x7f8da80b5a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:00.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.490+0000 7f8da77fe700 1 -- 172.21.15.16:0/3525629235 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1763342130 0 0) 0x7f8da815bc80 con 0x7f8da80cf880 2023-10-16T12:29:00.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.490+0000 7f8da77fe700 1 -- 172.21.15.16:0/3525629235 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d980035b0 con 0x7f8da80cf880 2023-10-16T12:29:00.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.490+0000 7f8da77fe700 1 -- 172.21.15.16:0/3525629235 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1756677303 0 0) 0x7f8da815b070 con 0x7f8da80cf4a0 2023-10-16T12:29:00.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.490+0000 7f8da77fe700 1 -- 172.21.15.16:0/3525629235 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8da815bc80 con 0x7f8da80cf4a0 2023-10-16T12:29:00.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.490+0000 7f8dad659700 1 -- 172.21.15.16:0/3525629235 >> v1:172.21.15.49:6789/0 conn(0x7f8da80cf4a0 legacy=0x7f8da80726e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:00.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.490+0000 7f8da77fe700 1 -- 172.21.15.16:0/3525629235 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 749725313 0 0) 0x7f8d980035b0 con 0x7f8da80cf880 2023-10-16T12:29:00.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.490+0000 7f8dad659700 1 -- 172.21.15.16:0/3525629235 >> v1:172.21.15.16:6789/0 conn(0x7f8da80cf880 legacy=0x7f8da80b52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:00.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.490+0000 7f8dad659700 1 -- 172.21.15.16:0/3525629235 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8da815b630 con 0x7f8da815d230 2023-10-16T12:29:00.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.491+0000 7f8dad659700 1 --2- 172.21.15.16:0/3525629235 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8da815d230 0x7f8da80b5a00 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f8da0009d90 tx=0x7f8da000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fb21c1d2ddf4d3ba server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:00.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.491+0000 7f8da77fe700 1 -- 172.21.15.16:0/3525629235 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8da000e070 con 0x7f8da815d230 2023-10-16T12:29:00.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.491+0000 7f8da77fe700 1 -- 172.21.15.16:0/3525629235 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8da0003a20 con 0x7f8da815d230 2023-10-16T12:29:00.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.491+0000 7f8da77fe700 1 -- 172.21.15.16:0/3525629235 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d94001670 con 0x7f8da815d230 2023-10-16T12:29:00.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.491+0000 7f8daf0bc700 1 -- 172.21.15.16:0/3525629235 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8da815d230 msgr2=0x7f8da80b5a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:00.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.491+0000 7f8daf0bc700 1 --2- 172.21.15.16:0/3525629235 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8da815d230 0x7f8da80b5a00 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f8da0009d90 tx=0x7f8da000b330 comp rx=0 tx=0).stop 2023-10-16T12:29:00.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.492+0000 7f8daf0bc700 1 -- 172.21.15.16:0/3525629235 shutdown_connections 2023-10-16T12:29:00.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.492+0000 7f8daf0bc700 1 --2- 172.21.15.16:0/3525629235 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8da815d230 0x7f8da80b5a00 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:00.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.492+0000 7f8daf0bc700 1 -- 172.21.15.16:0/3525629235 >> 172.21.15.16:0/3525629235 conn(0x7f8da80b1fd0 msgr2=0x7f8da80b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:00.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.492+0000 7f8daf0bc700 1 -- 172.21.15.16:0/3525629235 shutdown_connections 2023-10-16T12:29:00.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.492+0000 7f8daf0bc700 1 -- 172.21.15.16:0/3525629235 wait complete. 2023-10-16T12:29:00.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.493+0000 7f8daf0bc700 1 Processor -- start 2023-10-16T12:29:00.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.493+0000 7f8daf0bc700 1 -- start start 2023-10-16T12:29:00.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.494+0000 7f8daf0bc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8da80cf4a0 0x7f8da81598f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:00.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.494+0000 7f8daf0bc700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8da80cf880 0x7f8da81591b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:00.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.494+0000 7f8daf0bc700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8da815d230 0x7f8da80af520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:00.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.494+0000 7f8daf0bc700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da80a84c0 con 0x7f8da80cf4a0 2023-10-16T12:29:00.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.494+0000 7f8daf0bc700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da81596f0 con 0x7f8da815d230 2023-10-16T12:29:00.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.494+0000 7f8daf0bc700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8da8159ed0 con 0x7f8da80cf880 2023-10-16T12:29:00.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.494+0000 7f8dace58700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8da80cf4a0 0x7f8da81598f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:00.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.494+0000 7f8dace58700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8da80cf4a0 0x7f8da81598f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43218/0 (socket says 172.21.15.16:43218) 2023-10-16T12:29:00.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.494+0000 7f8dace58700 1 -- 172.21.15.16:0/1017754807 learned_addr learned my addr 172.21.15.16:0/1017754807 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:29:00.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.494+0000 7f8dad659700 1 --2- 172.21.15.16:0/1017754807 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8da815d230 0x7f8da80af520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:00.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.494+0000 7f8da7fff700 1 --2- 172.21.15.16:0/1017754807 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8da80cf880 0x7f8da81591b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:00.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.494+0000 7f8dace58700 1 -- 172.21.15.16:0/1017754807 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8da80cf880 msgr2=0x7f8da81591b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:00.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.494+0000 7f8dace58700 1 --2- 172.21.15.16:0/1017754807 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8da80cf880 0x7f8da81591b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:00.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.494+0000 7f8dace58700 1 -- 172.21.15.16:0/1017754807 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8da815d230 msgr2=0x7f8da80af520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:00.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.494+0000 7f8dace58700 1 --2- 172.21.15.16:0/1017754807 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8da815d230 0x7f8da80af520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:00.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.494+0000 7f8dace58700 1 -- 172.21.15.16:0/1017754807 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8da0009a70 con 0x7f8da80cf4a0 2023-10-16T12:29:00.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.494+0000 7f8dad659700 1 --2- 172.21.15.16:0/1017754807 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8da815d230 0x7f8da80af520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:29:00.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.494+0000 7f8dace58700 1 --2- 172.21.15.16:0/1017754807 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8da80cf4a0 0x7f8da81598f0 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f8d9c00c520 tx=0x7f8d9c00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:00.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.495+0000 7f8da57fa700 1 -- 172.21.15.16:0/1017754807 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d9c0052d0 con 0x7f8da80cf4a0 2023-10-16T12:29:00.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.495+0000 7f8da57fa700 1 -- 172.21.15.16:0/1017754807 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d9c005430 con 0x7f8da80cf4a0 2023-10-16T12:29:00.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.495+0000 7f8daf0bc700 1 -- 172.21.15.16:0/1017754807 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8da80a8710 con 0x7f8da80cf4a0 2023-10-16T12:29:00.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.495+0000 7f8da57fa700 1 -- 172.21.15.16:0/1017754807 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d9c0038c0 con 0x7f8da80cf4a0 2023-10-16T12:29:00.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.495+0000 7f8daf0bc700 1 -- 172.21.15.16:0/1017754807 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8da80a8cd0 con 0x7f8da80cf4a0 2023-10-16T12:29:00.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.496+0000 7f8daf0bc700 1 -- 172.21.15.16:0/1017754807 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8da80a8ef0 con 0x7f8da80cf4a0 2023-10-16T12:29:00.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.496+0000 7f8da57fa700 1 -- 172.21.15.16:0/1017754807 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 58) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f8d9c003a20 con 0x7f8da80cf4a0 2023-10-16T12:29:00.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.496+0000 7f8da57fa700 1 --2- 172.21.15.16:0/1017754807 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f8d8c06b140 0x7f8d8c06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:00.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.497+0000 7f8da7fff700 1 -- 172.21.15.16:0/1017754807 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f8d8c06b140 msgr2=0x7f8d8c06d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1458543709 2023-10-16T12:29:00.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.497+0000 7f8da57fa700 1 -- 172.21.15.16:0/1017754807 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f8d9c08b8d0 con 0x7f8da80cf4a0 2023-10-16T12:29:00.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.497+0000 7f8da7fff700 1 --2- 172.21.15.16:0/1017754807 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f8d8c06b140 0x7f8d8c06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:29:00.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.501+0000 7f8da57fa700 1 -- 172.21.15.16:0/1017754807 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8d9c056450 con 0x7f8da80cf4a0 2023-10-16T12:29:00.669 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:29:00.668+0000 7fa4e0be9200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T12:29:00.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.698+0000 7f8da7fff700 1 -- 172.21.15.16:0/1017754807 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f8d8c06b140 msgr2=0x7f8d8c06d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1458543709 2023-10-16T12:29:00.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.698+0000 7f8da7fff700 1 --2- 172.21.15.16:0/1017754807 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f8d8c06b140 0x7f8d8c06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:29:00.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.780+0000 7f8daf0bc700 1 -- 172.21.15.16:0/1017754807 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8da804ebb0 con 0x7f8da80cf4a0 2023-10-16T12:29:00.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.780+0000 7f8da57fa700 1 -- 172.21.15.16:0/1017754807 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+4903 (secure 0 0 0) 0x7f8d9c059a70 con 0x7f8da80cf4a0 2023-10-16T12:29:00.783 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:29:00.783 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":53,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6524,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/854923476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":854923476},{"type":"v1","addr":"172.21.15.49:6834","nonce":854923476}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":7249,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/1186699359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1186699359},{"type":"v1","addr":"172.21.15.16:6836","nonce":1186699359}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":7252,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/22289021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":22289021},{"type":"v1","addr":"172.21.15.16:6837","nonce":22289021}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:27:37.397811+0000","modified":"2023-10-16T12:27:40.428748+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7258},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7258":{"gid":7258,"name":"b","rank":0,"incarnation":49,"state":"up:active","state_seq":3,"addr":"172.21.15.49:6835/729289774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":729289774},{"type":"v1","addr":"172.21.15.49:6835","nonce":729289774}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-10-16T12:29:00.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.782+0000 7f8daf0bc700 1 -- 172.21.15.16:0/1017754807 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f8d8c06b140 msgr2=0x7f8d8c06d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:29:00.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.782+0000 7f8daf0bc700 1 --2- 172.21.15.16:0/1017754807 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f8d8c06b140 0x7f8d8c06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:00.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.782+0000 7f8daf0bc700 1 -- 172.21.15.16:0/1017754807 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8da80cf4a0 msgr2=0x7f8da81598f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:00.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.782+0000 7f8daf0bc700 1 --2- 172.21.15.16:0/1017754807 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8da80cf4a0 0x7f8da81598f0 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f8d9c00c520 tx=0x7f8d9c00c830 comp rx=0 tx=0).stop 2023-10-16T12:29:00.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.782+0000 7f8daf0bc700 1 -- 172.21.15.16:0/1017754807 shutdown_connections 2023-10-16T12:29:00.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.782+0000 7f8daf0bc700 1 --2- 172.21.15.16:0/1017754807 >> [v2:172.21.15.16:6832/1458543709,v1:172.21.15.16:6833/1458543709] conn(0x7f8d8c06b140 0x7f8d8c06d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:00.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.782+0000 7f8daf0bc700 1 --2- 172.21.15.16:0/1017754807 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8da815d230 0x7f8da80af520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:00.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.782+0000 7f8daf0bc700 1 --2- 172.21.15.16:0/1017754807 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8da80cf4a0 0x7f8da81598f0 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:00.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.782+0000 7f8daf0bc700 1 --2- 172.21.15.16:0/1017754807 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8da80cf880 0x7f8da81591b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:00.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.782+0000 7f8daf0bc700 1 -- 172.21.15.16:0/1017754807 >> 172.21.15.16:0/1017754807 conn(0x7f8da80b1fd0 msgr2=0x7f8da80c3380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:00.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.782+0000 7f8daf0bc700 1 -- 172.21.15.16:0/1017754807 shutdown_connections 2023-10-16T12:29:00.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.782+0000 7f8daf0bc700 1 -- 172.21.15.16:0/1017754807 wait complete. 2023-10-16T12:29:00.789 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 53 2023-10-16T12:29:00.797 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:29:00.882 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:29:00.881+0000 7f9831e6b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T12:29:00.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.932+0000 7f18ae4d6700 1 Processor -- start 2023-10-16T12:29:00.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.932+0000 7f18ae4d6700 1 -- start start 2023-10-16T12:29:00.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.933+0000 7f18ae4d6700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f18a80cbed0 0x7f18a806eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:00.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.933+0000 7f18ae4d6700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18a806f0a0 con 0x7f18a815a870 2023-10-16T12:29:00.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.933+0000 7f18ae4d6700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18a806f380 con 0x7f18a806d350 2023-10-16T12:29:00.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.933+0000 7f18ae4d6700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18a815ac50 con 0x7f18a80cbed0 2023-10-16T12:29:00.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.933+0000 7f18a7fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f18a815a870 0x7f18a80cb7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:60754/0 (socket says 172.21.15.16:60754) 2023-10-16T12:29:00.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.933+0000 7f18a7fff700 1 -- 172.21.15.16:0/2971631558 learned_addr learned my addr 172.21.15.16:0/2971631558 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:29:00.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.934+0000 7f18aca73700 1 --2- 172.21.15.16:0/2971631558 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f18a80cbed0 0x7f18a806eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:00.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.934+0000 7f18a6ffd700 1 -- 172.21.15.16:0/2971631558 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1028935963 0 0) 0x7f18a806f0a0 con 0x7f18a815a870 2023-10-16T12:29:00.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.934+0000 7f18a6ffd700 1 -- 172.21.15.16:0/2971631558 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18900035b0 con 0x7f18a815a870 2023-10-16T12:29:00.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.934+0000 7f18aca73700 1 -- 172.21.15.16:0/2971631558 >> v1:172.21.15.49:6789/0 conn(0x7f18a806d350 legacy=0x7f18a80cde10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:00.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.934+0000 7f18a6ffd700 1 -- 172.21.15.16:0/2971631558 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2884308997 0 0) 0x7f18a806f380 con 0x7f18a806d350 2023-10-16T12:29:00.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.934+0000 7f18aca73700 1 -- 172.21.15.16:0/2971631558 >> v1:172.21.15.16:6789/0 conn(0x7f18a815a870 legacy=0x7f18a80cb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:00.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.934+0000 7f18aca73700 1 -- 172.21.15.16:0/2971631558 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18a806f660 con 0x7f18a80cbed0 2023-10-16T12:29:00.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.934+0000 7f18aca73700 1 --2- 172.21.15.16:0/2971631558 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f18a80cbed0 0x7f18a806eaa0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f18a0009dc0 tx=0x7f18a000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ffaf39fa1355ca51 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:00.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.934+0000 7f18a6ffd700 1 -- 172.21.15.16:0/2971631558 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a000e070 con 0x7f18a80cbed0 2023-10-16T12:29:00.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.934+0000 7f18a6ffd700 1 -- 172.21.15.16:0/2971631558 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18a000bb20 con 0x7f18a80cbed0 2023-10-16T12:29:00.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.934+0000 7f18a6ffd700 1 -- 172.21.15.16:0/2971631558 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a000bed0 con 0x7f18a80cbed0 2023-10-16T12:29:00.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.935+0000 7f18ae4d6700 1 -- 172.21.15.16:0/2971631558 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f18a80cbed0 msgr2=0x7f18a806eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:00.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.935+0000 7f18ae4d6700 1 --2- 172.21.15.16:0/2971631558 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f18a80cbed0 0x7f18a806eaa0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f18a0009dc0 tx=0x7f18a000b330 comp rx=0 tx=0).stop 2023-10-16T12:29:00.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.935+0000 7f18ae4d6700 1 -- 172.21.15.16:0/2971631558 shutdown_connections 2023-10-16T12:29:00.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.935+0000 7f18ae4d6700 1 --2- 172.21.15.16:0/2971631558 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f18a80cbed0 0x7f18a806eaa0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:00.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.935+0000 7f18ae4d6700 1 -- 172.21.15.16:0/2971631558 >> 172.21.15.16:0/2971631558 conn(0x7f18a80cc5b0 msgr2=0x7f18a80cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:00.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.935+0000 7f18ae4d6700 1 -- 172.21.15.16:0/2971631558 shutdown_connections 2023-10-16T12:29:00.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.935+0000 7f18ae4d6700 1 -- 172.21.15.16:0/2971631558 wait complete. 2023-10-16T12:29:00.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.936+0000 7f18ae4d6700 1 Processor -- start 2023-10-16T12:29:00.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.937+0000 7f18ae4d6700 1 -- start start 2023-10-16T12:29:00.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.937+0000 7f18ae4d6700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f18a806d350 0x7f18a814dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:00.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.937+0000 7f18ae4d6700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f18a80cbed0 0x7f18a81e5330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:00.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.937+0000 7f18ae4d6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18a815a870 0x7f18a81e5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:00.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.937+0000 7f18ae4d6700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18a81e4ad0 con 0x7f18a815a870 2023-10-16T12:29:00.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.937+0000 7f18ae4d6700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18a80c6ec0 con 0x7f18a80cbed0 2023-10-16T12:29:00.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.937+0000 7f18ae4d6700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18a81e3c70 con 0x7f18a806d350 2023-10-16T12:29:00.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.938+0000 7f18aca73700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18a815a870 0x7f18a81e5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:00.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.938+0000 7f18aca73700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18a815a870 0x7f18a81e5870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43256/0 (socket says 172.21.15.16:43256) 2023-10-16T12:29:00.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.938+0000 7f18aca73700 1 -- 172.21.15.16:0/582967404 learned_addr learned my addr 172.21.15.16:0/582967404 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:29:00.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.938+0000 7f18a7fff700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f18a806d350 0x7f18a814dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:00.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.938+0000 7f18a77fe700 1 --2- 172.21.15.16:0/582967404 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f18a80cbed0 0x7f18a81e5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:00.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.938+0000 7f18aca73700 1 -- 172.21.15.16:0/582967404 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f18a806d350 msgr2=0x7f18a814dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:00.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.938+0000 7f18aca73700 1 --2- 172.21.15.16:0/582967404 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f18a806d350 0x7f18a814dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:00.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.938+0000 7f18aca73700 1 -- 172.21.15.16:0/582967404 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f18a80cbed0 msgr2=0x7f18a81e5330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:00.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.938+0000 7f18aca73700 1 --2- 172.21.15.16:0/582967404 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f18a80cbed0 0x7f18a81e5330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:00.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.938+0000 7f18aca73700 1 -- 172.21.15.16:0/582967404 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18a0009a70 con 0x7f18a815a870 2023-10-16T12:29:00.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.938+0000 7f18a7fff700 1 --2- 172.21.15.16:0/582967404 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f18a806d350 0x7f18a814dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:29:00.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.938+0000 7f18aca73700 1 --2- 172.21.15.16:0/582967404 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18a815a870 0x7f18a81e5870 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f18a0009f30 tx=0x7f18a0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:00.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.938+0000 7f18a4ff9700 1 -- 172.21.15.16:0/582967404 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a001b810 con 0x7f18a815a870 2023-10-16T12:29:00.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.938+0000 7f18a4ff9700 1 -- 172.21.15.16:0/582967404 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18a0020070 con 0x7f18a815a870 2023-10-16T12:29:00.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.938+0000 7f18ae4d6700 1 -- 172.21.15.16:0/582967404 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18a81e3f50 con 0x7f18a815a870 2023-10-16T12:29:00.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.938+0000 7f18a4ff9700 1 -- 172.21.15.16:0/582967404 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a001bbf0 con 0x7f18a815a870 2023-10-16T12:29:00.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.938+0000 7f18ae4d6700 1 -- 172.21.15.16:0/582967404 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18a81e4510 con 0x7f18a815a870 2023-10-16T12:29:00.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.940+0000 7f18ae4d6700 1 -- 172.21.15.16:0/582967404 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18a8067f80 con 0x7f18a815a870 2023-10-16T12:29:00.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.940+0000 7f18a4ff9700 1 -- 172.21.15.16:0/582967404 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 59) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f18a001bd50 con 0x7f18a815a870 2023-10-16T12:29:00.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.941+0000 7f18a4ff9700 1 -- 172.21.15.16:0/582967404 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6947+0+0 (secure 0 0 0) 0x7f18a0022070 con 0x7f18a815a870 2023-10-16T12:29:00.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:00.946+0000 7f18a4ff9700 1 -- 172.21.15.16:0/582967404 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f18a0056d50 con 0x7f18a815a870 2023-10-16T12:29:00.973 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:29:00.970+0000 7f97aa691700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:29:00.973 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:29:00.970+0000 7f97aa691700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:29:00.973 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:29:00.970+0000 7f97aa691700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:29:00.973 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:29:00.970+0000 7f97aa691700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:29:00.974 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:29:00.970+0000 7f97aa691700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:29:00.974 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:29:00.971+0000 7f97ad697700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:29:00.974 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:29:00.971+0000 7f97ad697700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:29:00.975 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:29:00.971+0000 7f97ad697700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:29:00.975 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:29:00.971+0000 7f97ad697700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:29:00.975 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:29:00.971+0000 7f97ad697700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:29:01.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.204+0000 7f18ae4d6700 1 -- 172.21.15.16:0/582967404 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f18a80ce320 con 0x7f18a815a870 2023-10-16T12:29:01.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.205+0000 7f18a4ff9700 1 -- 172.21.15.16:0/582967404 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+4903 (secure 0 0 0) 0x7f18a005a370 con 0x7f18a815a870 2023-10-16T12:29:01.209 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:29:01.210 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":53,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6524,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/854923476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":854923476},{"type":"v1","addr":"172.21.15.49:6834","nonce":854923476}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":7249,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/1186699359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1186699359},{"type":"v1","addr":"172.21.15.16:6836","nonce":1186699359}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":7252,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/22289021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":22289021},{"type":"v1","addr":"172.21.15.16:6837","nonce":22289021}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:27:37.397811+0000","modified":"2023-10-16T12:27:40.428748+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7258},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7258":{"gid":7258,"name":"b","rank":0,"incarnation":49,"state":"up:active","state_seq":3,"addr":"172.21.15.49:6835/729289774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":729289774},{"type":"v1","addr":"172.21.15.49:6835","nonce":729289774}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-10-16T12:29:01.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.207+0000 7f18ae4d6700 1 -- 172.21.15.16:0/582967404 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18a815a870 msgr2=0x7f18a81e5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:01.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.207+0000 7f18ae4d6700 1 --2- 172.21.15.16:0/582967404 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18a815a870 0x7f18a81e5870 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f18a0009f30 tx=0x7f18a0004ad0 comp rx=0 tx=0).stop 2023-10-16T12:29:01.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.207+0000 7f18ae4d6700 1 -- 172.21.15.16:0/582967404 shutdown_connections 2023-10-16T12:29:01.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.207+0000 7f18ae4d6700 1 --2- 172.21.15.16:0/582967404 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18a815a870 0x7f18a81e5870 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:01.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.207+0000 7f18ae4d6700 1 --2- 172.21.15.16:0/582967404 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f18a806d350 0x7f18a814dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:01.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.207+0000 7f18ae4d6700 1 --2- 172.21.15.16:0/582967404 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f18a80cbed0 0x7f18a81e5330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:01.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.207+0000 7f18ae4d6700 1 -- 172.21.15.16:0/582967404 >> 172.21.15.16:0/582967404 conn(0x7f18a80cc5b0 msgr2=0x7f18a80b3de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:01.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.207+0000 7f18ae4d6700 1 -- 172.21.15.16:0/582967404 shutdown_connections 2023-10-16T12:29:01.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.207+0000 7f18ae4d6700 1 -- 172.21.15.16:0/582967404 wait complete. 2023-10-16T12:29:01.216 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 53 2023-10-16T12:29:01.222 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:29:01.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.349+0000 7f0f46a38700 1 Processor -- start 2023-10-16T12:29:01.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.349+0000 7f0f46a38700 1 -- start start 2023-10-16T12:29:01.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.350+0000 7f0f46a38700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f40159c90 0x7f0f4015d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:01.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.350+0000 7f0f46a38700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f4006b1f0 con 0x7f0f400cb190 2023-10-16T12:29:01.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.350+0000 7f0f46a38700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f4006b4b0 con 0x7f0f4015a070 2023-10-16T12:29:01.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.350+0000 7f0f46a38700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f400af340 con 0x7f0f40159c90 2023-10-16T12:29:01.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.350+0000 7f0f44fd5700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0f400cb190 0x7f0f400aec30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:60820/0 (socket says 172.21.15.16:60820) 2023-10-16T12:29:01.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.350+0000 7f0f44fd5700 1 -- 172.21.15.16:0/3826076725 learned_addr learned my addr 172.21.15.16:0/3826076725 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:29:01.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.350+0000 7f0f3ffff700 1 --2- 172.21.15.16:0/3826076725 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f40159c90 0x7f0f4015d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:01.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.350+0000 7f0f3effd700 1 -- 172.21.15.16:0/3826076725 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2949137548 0 0) 0x7f0f4006b1f0 con 0x7f0f400cb190 2023-10-16T12:29:01.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.350+0000 7f0f3effd700 1 -- 172.21.15.16:0/3826076725 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f280057c0 con 0x7f0f400cb190 2023-10-16T12:29:01.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.350+0000 7f0f3effd700 1 -- 172.21.15.16:0/3826076725 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3309986933 0 0) 0x7f0f280057c0 con 0x7f0f400cb190 2023-10-16T12:29:01.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.350+0000 7f0f3effd700 1 -- 172.21.15.16:0/3826076725 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f40159c90 msgr2=0x7f0f4015d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:29:01.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.350+0000 7f0f3effd700 1 --2- 172.21.15.16:0/3826076725 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f40159c90 0x7f0f4015d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:01.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.350+0000 7f0f3effd700 1 -- 172.21.15.16:0/3826076725 >> v1:172.21.15.49:6789/0 conn(0x7f0f4015a070 legacy=0x7f0f400ae520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:01.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.350+0000 7f0f3effd700 1 -- 172.21.15.16:0/3826076725 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f4006b790 con 0x7f0f400cb190 2023-10-16T12:29:01.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.350+0000 7f0f3effd700 1 -- 172.21.15.16:0/3826076725 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f0f380019f0 con 0x7f0f400cb190 2023-10-16T12:29:01.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.350+0000 7f0f3effd700 1 -- 172.21.15.16:0/3826076725 >> v1:172.21.15.16:6789/0 conn(0x7f0f400cb190 legacy=0x7f0f400aec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:01.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.350+0000 7f0f3effd700 1 --2- 172.21.15.16:0/3826076725 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f28005c40 0x7f0f28006020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:01.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.351+0000 7f0f3effd700 1 --2- 172.21.15.16:0/3826076725 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f280065f0 0x7f0f28006ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:01.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.351+0000 7f0f3effd700 1 --2- 172.21.15.16:0/3826076725 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f40159c90 0x7f0f280073c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:01.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.351+0000 7f0f3effd700 1 -- 172.21.15.16:0/3826076725 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f400af340 con 0x7f0f280065f0 2023-10-16T12:29:01.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.351+0000 7f0f3effd700 1 -- 172.21.15.16:0/3826076725 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f2800dcd0 con 0x7f0f28005c40 2023-10-16T12:29:01.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.351+0000 7f0f3effd700 1 -- 172.21.15.16:0/3826076725 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f2800de90 con 0x7f0f40159c90 2023-10-16T12:29:01.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.351+0000 7f0f3f7fe700 1 --2- 172.21.15.16:0/3826076725 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f280065f0 0x7f0f28006ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:01.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.351+0000 7f0f3ffff700 1 --2- 172.21.15.16:0/3826076725 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f28005c40 0x7f0f28006020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:01.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.351+0000 7f0f44fd5700 1 --2- 172.21.15.16:0/3826076725 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f40159c90 0x7f0f280073c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:01.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.351+0000 7f0f3f7fe700 1 -- 172.21.15.16:0/3826076725 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f40159c90 msgr2=0x7f0f280073c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:29:01.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.351+0000 7f0f3f7fe700 1 --2- 172.21.15.16:0/3826076725 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f40159c90 0x7f0f280073c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:01.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.351+0000 7f0f3f7fe700 1 -- 172.21.15.16:0/3826076725 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f28005c40 msgr2=0x7f0f28006020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:29:01.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.351+0000 7f0f3f7fe700 1 --2- 172.21.15.16:0/3826076725 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f28005c40 0x7f0f28006020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:01.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.351+0000 7f0f3f7fe700 1 -- 172.21.15.16:0/3826076725 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f280057c0 con 0x7f0f280065f0 2023-10-16T12:29:01.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.351+0000 7f0f3ffff700 1 --2- 172.21.15.16:0/3826076725 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f28005c40 0x7f0f28006020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:29:01.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.351+0000 7f0f3f7fe700 1 --2- 172.21.15.16:0/3826076725 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f280065f0 0x7f0f28006ac0 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f0f4015a470 tx=0x7f0f28007a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a25670ed28c89f37 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:01.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.351+0000 7f0f3effd700 1 -- 172.21.15.16:0/3826076725 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f3400c660 con 0x7f0f280065f0 2023-10-16T12:29:01.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.351+0000 7f0f3effd700 1 -- 172.21.15.16:0/3826076725 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f3400c7c0 con 0x7f0f280065f0 2023-10-16T12:29:01.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.351+0000 7f0f3effd700 1 -- 172.21.15.16:0/3826076725 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f3400cb70 con 0x7f0f280065f0 2023-10-16T12:29:01.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.352+0000 7f0f46a38700 1 -- 172.21.15.16:0/3826076725 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f280065f0 msgr2=0x7f0f28006ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:01.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.352+0000 7f0f46a38700 1 --2- 172.21.15.16:0/3826076725 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f280065f0 0x7f0f28006ac0 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f0f4015a470 tx=0x7f0f28007a90 comp rx=0 tx=0).stop 2023-10-16T12:29:01.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.352+0000 7f0f44fd5700 1 -- 172.21.15.16:0/3826076725 reap_dead start 2023-10-16T12:29:01.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.352+0000 7f0f46a38700 1 -- 172.21.15.16:0/3826076725 shutdown_connections 2023-10-16T12:29:01.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.352+0000 7f0f46a38700 1 -- 172.21.15.16:0/3826076725 >> 172.21.15.16:0/3826076725 conn(0x7f0f400cc5b0 msgr2=0x7f0f400cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:01.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.352+0000 7f0f46a38700 1 -- 172.21.15.16:0/3826076725 shutdown_connections 2023-10-16T12:29:01.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.352+0000 7f0f46a38700 1 -- 172.21.15.16:0/3826076725 wait complete. 2023-10-16T12:29:01.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.353+0000 7f0f46a38700 1 Processor -- start 2023-10-16T12:29:01.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.353+0000 7f0f46a38700 1 -- start start 2023-10-16T12:29:01.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.354+0000 7f0f46a38700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f280065f0 0x7f0f400c79c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:01.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.354+0000 7f0f46a38700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f4014e580 0x7f0f4014ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:01.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.354+0000 7f0f46a38700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f4014b090 0x7f0f400c98a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:01.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.354+0000 7f0f46a38700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f4014b540 con 0x7f0f4014e580 2023-10-16T12:29:01.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.354+0000 7f0f46a38700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f4014ca60 con 0x7f0f4014b090 2023-10-16T12:29:01.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.354+0000 7f0f46a38700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f4014c480 con 0x7f0f280065f0 2023-10-16T12:29:01.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.354+0000 7f0f3f7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f4014e580 0x7f0f4014ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:01.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.354+0000 7f0f3f7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f4014e580 0x7f0f4014ab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43324/0 (socket says 172.21.15.16:43324) 2023-10-16T12:29:01.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.354+0000 7f0f3f7fe700 1 -- 172.21.15.16:0/2153734133 learned_addr learned my addr 172.21.15.16:0/2153734133 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:29:01.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.354+0000 7f0f3ffff700 1 --2- 172.21.15.16:0/2153734133 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f280065f0 0x7f0f400c79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:01.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.354+0000 7f0f44fd5700 1 --2- 172.21.15.16:0/2153734133 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f4014b090 0x7f0f400c98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:01.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.355+0000 7f0f3f7fe700 1 -- 172.21.15.16:0/2153734133 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f280065f0 msgr2=0x7f0f400c79c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:01.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.355+0000 7f0f3f7fe700 1 --2- 172.21.15.16:0/2153734133 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f280065f0 0x7f0f400c79c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:01.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.355+0000 7f0f3f7fe700 1 -- 172.21.15.16:0/2153734133 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f4014b090 msgr2=0x7f0f400c98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:01.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.355+0000 7f0f3f7fe700 1 --2- 172.21.15.16:0/2153734133 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f4014b090 0x7f0f400c98a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:01.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.355+0000 7f0f3f7fe700 1 -- 172.21.15.16:0/2153734133 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f400c7050 con 0x7f0f4014e580 2023-10-16T12:29:01.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.355+0000 7f0f3ffff700 1 --2- 172.21.15.16:0/2153734133 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f280065f0 0x7f0f400c79c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:29:01.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.355+0000 7f0f3f7fe700 1 --2- 172.21.15.16:0/2153734133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f4014e580 0x7f0f4014ab50 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f0f340052b0 tx=0x7f0f34001230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:01.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.355+0000 7f0f3cff9700 1 -- 172.21.15.16:0/2153734133 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f34000ed0 con 0x7f0f4014e580 2023-10-16T12:29:01.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.355+0000 7f0f3cff9700 1 -- 172.21.15.16:0/2153734133 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f34021070 con 0x7f0f4014e580 2023-10-16T12:29:01.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.355+0000 7f0f46a38700 1 -- 172.21.15.16:0/2153734133 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f4014c6f0 con 0x7f0f4014e580 2023-10-16T12:29:01.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.355+0000 7f0f3cff9700 1 -- 172.21.15.16:0/2153734133 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f3400cb70 con 0x7f0f4014e580 2023-10-16T12:29:01.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.355+0000 7f0f46a38700 1 -- 172.21.15.16:0/2153734133 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f400c7f00 con 0x7f0f4014e580 2023-10-16T12:29:01.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.356+0000 7f0f46a38700 1 -- 172.21.15.16:0/2153734133 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f400c80f0 con 0x7f0f4014e580 2023-10-16T12:29:01.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.356+0000 7f0f3cff9700 1 -- 172.21.15.16:0/2153734133 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 59) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f0f34023030 con 0x7f0f4014e580 2023-10-16T12:29:01.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.357+0000 7f0f3cff9700 1 -- 172.21.15.16:0/2153734133 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6947+0+0 (secure 0 0 0) 0x7f0f3408a6f0 con 0x7f0f4014e580 2023-10-16T12:29:01.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.361+0000 7f0f3cff9700 1 -- 172.21.15.16:0/2153734133 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0f34055830 con 0x7f0f4014e580 2023-10-16T12:29:01.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.566+0000 7f0f46a38700 1 -- 172.21.15.16:0/2153734133 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0f4014c880 con 0x7f0f4014e580 2023-10-16T12:29:01.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.567+0000 7f0f3cff9700 1 -- 172.21.15.16:0/2153734133 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v116) v1 ==== 74+0+17980 (secure 0 0 0) 0x7f0f34058e50 con 0x7f0f4014e580 2023-10-16T12:29:01.569 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:29:01.570 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":116,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:29:00.894746+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:27:35.907633+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2023-10-16T12:27:36.362861+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.10","mappings":[{"from":1,"to":5}]},{"pgid":"14.2d","mappings":[{"from":1,"to":7}]},{"pgid":"14.32","mappings":[{"from":1,"to":2}]},{"pgid":"14.3c","mappings":[{"from":1,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/1965208057":"2023-10-17T12:29:00.894705+0000","172.21.15.16:0/709605099":"2023-10-17T12:29:00.894705+0000","172.21.15.16:6832/1458543709":"2023-10-17T12:29:00.894705+0000","172.21.15.16:0/4068400710":"2023-10-17T12:29:00.894705+0000","172.21.15.16:0/531582398":"2023-10-17T12:29:00.894705+0000","172.21.15.16:0/650861850":"2023-10-17T12:29:00.894705+0000","172.21.15.16:6833/4089191642":"2023-10-17T12:27:51.745827+0000","172.21.15.16:0/2709574245":"2023-10-17T12:27:51.745827+0000","172.21.15.16:0/2810361636":"2023-10-17T12:29:00.894705+0000","172.21.15.16:0/769212771":"2023-10-17T12:27:51.745827+0000","172.21.15.16:0/1257374801":"2023-10-17T12:29:00.894705+0000","172.21.15.16:6832/4089191642":"2023-10-17T12:27:51.745827+0000","172.21.15.16:0/3579843680":"2023-10-17T12:27:51.745827+0000","172.21.15.16:0/1190603852":"2023-10-17T12:27:51.745827+0000","172.21.15.16:0/3144677187":"2023-10-17T12:27:51.745827+0000","172.21.15.16:0/2624839014":"2023-10-17T12:29:00.894705+0000","172.21.15.16:6833/1458543709":"2023-10-17T12:29:00.894705+0000","172.21.15.16:0/3478352903":"2023-10-17T12:27:51.745827+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:29:01.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.569+0000 7f0f46a38700 1 -- 172.21.15.16:0/2153734133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f4014e580 msgr2=0x7f0f4014ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:01.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.569+0000 7f0f46a38700 1 --2- 172.21.15.16:0/2153734133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f4014e580 0x7f0f4014ab50 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f0f340052b0 tx=0x7f0f34001230 comp rx=0 tx=0).stop 2023-10-16T12:29:01.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.569+0000 7f0f46a38700 1 -- 172.21.15.16:0/2153734133 shutdown_connections 2023-10-16T12:29:01.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.569+0000 7f0f46a38700 1 --2- 172.21.15.16:0/2153734133 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f4014b090 0x7f0f400c98a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:01.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.569+0000 7f0f46a38700 1 --2- 172.21.15.16:0/2153734133 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f280065f0 0x7f0f400c79c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:01.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.569+0000 7f0f46a38700 1 --2- 172.21.15.16:0/2153734133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f4014e580 0x7f0f4014ab50 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:01.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.569+0000 7f0f46a38700 1 -- 172.21.15.16:0/2153734133 >> 172.21.15.16:0/2153734133 conn(0x7f0f400cc5b0 msgr2=0x7f0f4015a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:01.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.569+0000 7f0f46a38700 1 -- 172.21.15.16:0/2153734133 shutdown_connections 2023-10-16T12:29:01.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.569+0000 7f0f46a38700 1 -- 172.21.15.16:0/2153734133 wait complete. 2023-10-16T12:29:01.584 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-16T12:29:01.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.708+0000 7fe10ca68700 1 Processor -- start 2023-10-16T12:29:01.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.708+0000 7fe10ca68700 1 -- start start 2023-10-16T12:29:01.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.709+0000 7fe10ca68700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fe1080d0750 0x7fe108168450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:01.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.709+0000 7fe10ca68700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10806ba00 con 0x7fe1080d41c0 2023-10-16T12:29:01.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.709+0000 7fe10ca68700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1080d26d0 con 0x7fe10815d8d0 2023-10-16T12:29:01.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.709+0000 7fe10ca68700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1080d23f0 con 0x7fe1080d0750 2023-10-16T12:29:01.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.709+0000 7fe105d9b700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe1080d41c0 0x7fe1080d0040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:60836/0 (socket says 172.21.15.16:60836) 2023-10-16T12:29:01.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.709+0000 7fe105d9b700 1 -- 172.21.15.16:0/201695771 learned_addr learned my addr 172.21.15.16:0/201695771 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:29:01.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.710+0000 7fe106d9d700 1 --2- 172.21.15.16:0/201695771 >> v2:172.21.15.49:3300/0 conn(0x7fe1080d0750 0x7fe108168450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:01.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.710+0000 7fe10559a700 1 -- 172.21.15.16:0/201695771 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1441006654 0 0) 0x7fe10806ba00 con 0x7fe1080d41c0 2023-10-16T12:29:01.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.710+0000 7fe10559a700 1 -- 172.21.15.16:0/201695771 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0f00035b0 con 0x7fe1080d41c0 2023-10-16T12:29:01.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.710+0000 7fe10559a700 1 -- 172.21.15.16:0/201695771 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2230737960 0 0) 0x7fe1080d26d0 con 0x7fe10815d8d0 2023-10-16T12:29:01.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.710+0000 7fe10559a700 1 -- 172.21.15.16:0/201695771 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe10806ba00 con 0x7fe10815d8d0 2023-10-16T12:29:01.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.710+0000 7fe10559a700 1 -- 172.21.15.16:0/201695771 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2622757661 0 0) 0x7fe0f00035b0 con 0x7fe1080d41c0 2023-10-16T12:29:01.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.710+0000 7fe10559a700 1 -- 172.21.15.16:0/201695771 >> v2:172.21.15.49:3300/0 conn(0x7fe1080d0750 msgr2=0x7fe108168450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:29:01.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.710+0000 7fe10559a700 1 --2- 172.21.15.16:0/201695771 >> v2:172.21.15.49:3300/0 conn(0x7fe1080d0750 0x7fe108168450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:01.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.710+0000 7fe10559a700 1 -- 172.21.15.16:0/201695771 >> v1:172.21.15.49:6789/0 conn(0x7fe10815d8d0 legacy=0x7fe1080d1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:01.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.710+0000 7fe10559a700 1 -- 172.21.15.16:0/201695771 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe10806bca0 con 0x7fe1080d41c0 2023-10-16T12:29:01.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.710+0000 7fe10559a700 1 -- 172.21.15.16:0/201695771 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7fe0f8001670 con 0x7fe1080d41c0 2023-10-16T12:29:01.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.710+0000 7fe10559a700 1 -- 172.21.15.16:0/201695771 >> v1:172.21.15.16:6789/0 conn(0x7fe1080d41c0 legacy=0x7fe1080d0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:01.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.710+0000 7fe10559a700 1 --2- 172.21.15.16:0/201695771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0f0003b90 0x7fe0f0003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:01.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.710+0000 7fe10559a700 1 --2- 172.21.15.16:0/201695771 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe0f0004850 0x7fe0f0004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:01.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.710+0000 7fe10559a700 1 --2- 172.21.15.16:0/201695771 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe0f0005210 0x7fe0f000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:01.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.710+0000 7fe10559a700 1 -- 172.21.15.16:0/201695771 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1080d23f0 con 0x7fe0f0003b90 2023-10-16T12:29:01.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.710+0000 7fe10559a700 1 -- 172.21.15.16:0/201695771 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f000c090 con 0x7fe0f0005210 2023-10-16T12:29:01.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.710+0000 7fe10559a700 1 -- 172.21.15.16:0/201695771 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f000c250 con 0x7fe0f0004850 2023-10-16T12:29:01.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.710+0000 7fe10659c700 1 --2- 172.21.15.16:0/201695771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0f0003b90 0x7fe0f0003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:01.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.710+0000 7fe105d9b700 1 --2- 172.21.15.16:0/201695771 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe0f0004850 0x7fe0f0004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:01.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.710+0000 7fe106d9d700 1 --2- 172.21.15.16:0/201695771 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe0f0005210 0x7fe0f000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:01.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.710+0000 7fe10659c700 1 -- 172.21.15.16:0/201695771 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe0f0004850 msgr2=0x7fe0f0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:29:01.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.710+0000 7fe10659c700 1 --2- 172.21.15.16:0/201695771 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe0f0004850 0x7fe0f0004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:01.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.710+0000 7fe10659c700 1 -- 172.21.15.16:0/201695771 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe0f0005210 msgr2=0x7fe0f000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:29:01.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.710+0000 7fe10659c700 1 --2- 172.21.15.16:0/201695771 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe0f0005210 0x7fe0f000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:01.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.710+0000 7fe10659c700 1 -- 172.21.15.16:0/201695771 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0f00035b0 con 0x7fe0f0003b90 2023-10-16T12:29:01.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.711+0000 7fe105d9b700 1 --2- 172.21.15.16:0/201695771 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe0f0004850 0x7fe0f0004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:29:01.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.711+0000 7fe106d9d700 1 --2- 172.21.15.16:0/201695771 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe0f0005210 0x7fe0f000ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:29:01.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.711+0000 7fe106d9d700 1 -- 172.21.15.16:0/201695771 reap_dead start 2023-10-16T12:29:01.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.711+0000 7fe10659c700 1 --2- 172.21.15.16:0/201695771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0f0003b90 0x7fe0f0003f90 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc001940 tx=0x7fe0f0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3459ccd995c93c5e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:01.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.711+0000 7fe10559a700 1 -- 172.21.15.16:0/201695771 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0fc001430 con 0x7fe0f0003b90 2023-10-16T12:29:01.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.711+0000 7fe10559a700 1 -- 172.21.15.16:0/201695771 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0fc001d70 con 0x7fe0f0003b90 2023-10-16T12:29:01.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.711+0000 7fe10559a700 1 -- 172.21.15.16:0/201695771 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0fc005510 con 0x7fe0f0003b90 2023-10-16T12:29:01.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.711+0000 7fe10ca68700 1 -- 172.21.15.16:0/201695771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0f0003b90 msgr2=0x7fe0f0003f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:01.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.711+0000 7fe10ca68700 1 --2- 172.21.15.16:0/201695771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0f0003b90 0x7fe0f0003f90 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc001940 tx=0x7fe0f0005880 comp rx=0 tx=0).stop 2023-10-16T12:29:01.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.711+0000 7fe10ca68700 1 -- 172.21.15.16:0/201695771 shutdown_connections 2023-10-16T12:29:01.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.711+0000 7fe10ca68700 1 --2- 172.21.15.16:0/201695771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0f0003b90 0x7fe0f0003f90 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:01.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.711+0000 7fe10ca68700 1 -- 172.21.15.16:0/201695771 >> 172.21.15.16:0/201695771 conn(0x7fe1080d3630 msgr2=0x7fe1080d3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:01.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.711+0000 7fe10ca68700 1 -- 172.21.15.16:0/201695771 shutdown_connections 2023-10-16T12:29:01.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.711+0000 7fe10ca68700 1 -- 172.21.15.16:0/201695771 wait complete. 2023-10-16T12:29:01.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.713+0000 7fe10ca68700 1 Processor -- start 2023-10-16T12:29:01.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.713+0000 7fe10ca68700 1 -- start start 2023-10-16T12:29:01.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.714+0000 7fe10ca68700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe0f0003b90 0x7fe10814d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:01.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.714+0000 7fe10ca68700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe1081514c0 0x7fe1081f28a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:01.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.714+0000 7fe10ca68700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1080ce200 0x7fe1080ce6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:01.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.714+0000 7fe10ca68700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1080cebf0 con 0x7fe1080ce200 2023-10-16T12:29:01.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.714+0000 7fe10ca68700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe10809f500 con 0x7fe0f0003b90 2023-10-16T12:29:01.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.714+0000 7fe10ca68700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe10814e620 con 0x7fe1081514c0 2023-10-16T12:29:01.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.714+0000 7fe106d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1080ce200 0x7fe1080ce6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:01.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.714+0000 7fe106d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1080ce200 0x7fe1080ce6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43344/0 (socket says 172.21.15.16:43344) 2023-10-16T12:29:01.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.714+0000 7fe106d9d700 1 -- 172.21.15.16:0/2644374307 learned_addr learned my addr 172.21.15.16:0/2644374307 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:29:01.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.714+0000 7fe105d9b700 1 --2- 172.21.15.16:0/2644374307 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe1081514c0 0x7fe1081f28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:01.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.714+0000 7fe10659c700 1 --2- 172.21.15.16:0/2644374307 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe0f0003b90 0x7fe10814d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:01.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.714+0000 7fe106d9d700 1 -- 172.21.15.16:0/2644374307 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe1081514c0 msgr2=0x7fe1081f28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:01.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.714+0000 7fe106d9d700 1 --2- 172.21.15.16:0/2644374307 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe1081514c0 0x7fe1081f28a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:01.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.714+0000 7fe106d9d700 1 -- 172.21.15.16:0/2644374307 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe0f0003b90 msgr2=0x7fe10814d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:01.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.714+0000 7fe106d9d700 1 --2- 172.21.15.16:0/2644374307 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe0f0003b90 0x7fe10814d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:01.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.714+0000 7fe106d9d700 1 -- 172.21.15.16:0/2644374307 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe10814dd80 con 0x7fe1080ce200 2023-10-16T12:29:01.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.714+0000 7fe105d9b700 1 --2- 172.21.15.16:0/2644374307 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe1081514c0 0x7fe1081f28a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:29:01.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.714+0000 7fe10659c700 1 --2- 172.21.15.16:0/2644374307 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe0f0003b90 0x7fe10814d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:29:01.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.714+0000 7fe106d9d700 1 --2- 172.21.15.16:0/2644374307 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1080ce200 0x7fe1080ce6b0 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7fe0f00047d0 tx=0x7fe1000104a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:01.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.714+0000 7fe0f6ffd700 1 -- 172.21.15.16:0/2644374307 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe10001b070 con 0x7fe1080ce200 2023-10-16T12:29:01.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.714+0000 7fe0f6ffd700 1 -- 172.21.15.16:0/2644374307 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe10000f040 con 0x7fe1080ce200 2023-10-16T12:29:01.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.714+0000 7fe10ca68700 1 -- 172.21.15.16:0/2644374307 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe10814e060 con 0x7fe1080ce200 2023-10-16T12:29:01.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.714+0000 7fe0f6ffd700 1 -- 172.21.15.16:0/2644374307 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe10000e050 con 0x7fe1080ce200 2023-10-16T12:29:01.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.714+0000 7fe10ca68700 1 -- 172.21.15.16:0/2644374307 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe10809f730 con 0x7fe1080ce200 2023-10-16T12:29:01.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.716+0000 7fe10ca68700 1 -- 172.21.15.16:0/2644374307 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe10804ebb0 con 0x7fe1080ce200 2023-10-16T12:29:01.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.716+0000 7fe0f6ffd700 1 -- 172.21.15.16:0/2644374307 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 59) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fe100015440 con 0x7fe1080ce200 2023-10-16T12:29:01.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.716+0000 7fe0f6ffd700 1 -- 172.21.15.16:0/2644374307 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6947+0+0 (secure 0 0 0) 0x7fe1000898a0 con 0x7fe1080ce200 2023-10-16T12:29:01.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.721+0000 7fe0f6ffd700 1 -- 172.21.15.16:0/2644374307 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe1000548c0 con 0x7fe1080ce200 2023-10-16T12:29:01.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.904+0000 7fe0f6ffd700 1 -- 172.21.15.16:0/2644374307 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 60) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fe100054370 con 0x7fe1080ce200 2023-10-16T12:29:01.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.905+0000 7fe0f6ffd700 1 --2- 172.21.15.16:0/2644374307 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fe0e8070ef0 0x7fe0e80732e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:01.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.906+0000 7fe10659c700 1 --2- 172.21.15.16:0/2644374307 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fe0e8070ef0 0x7fe0e80732e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:01.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.907+0000 7fe10659c700 1 --2- 172.21.15.16:0/2644374307 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fe0e8070ef0 0x7fe0e80732e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc015040 tx=0x7fe0fc00a000 comp rx=0 tx=0).ready entity=mgr.7884 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:01.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.976+0000 7fe10ca68700 1 -- 172.21.15.16:0/2644374307 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fe1080b1260 con 0x7fe1080ce200 2023-10-16T12:29:01.978 INFO:tasks.ceph.mon.a.smithi016.stderr:2023-10-16T12:29:01.976+0000 7f9fc8535700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-16T12:29:01.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.978+0000 7fe0f6ffd700 1 -- 172.21.15.16:0/2644374307 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v54) v1 ==== 161+0+0 (secure 0 0 0) 0x7fe100086470 con 0x7fe1080ce200 2023-10-16T12:29:01.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.981+0000 7fe10ca68700 1 -- 172.21.15.16:0/2644374307 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fe0e8070ef0 msgr2=0x7fe0e80732e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:01.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.981+0000 7fe10ca68700 1 --2- 172.21.15.16:0/2644374307 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fe0e8070ef0 0x7fe0e80732e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc015040 tx=0x7fe0fc00a000 comp rx=0 tx=0).stop 2023-10-16T12:29:01.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.981+0000 7fe10ca68700 1 -- 172.21.15.16:0/2644374307 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1080ce200 msgr2=0x7fe1080ce6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:01.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.981+0000 7fe10ca68700 1 --2- 172.21.15.16:0/2644374307 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1080ce200 0x7fe1080ce6b0 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7fe0f00047d0 tx=0x7fe1000104a0 comp rx=0 tx=0).stop 2023-10-16T12:29:01.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.981+0000 7fe10ca68700 1 -- 172.21.15.16:0/2644374307 shutdown_connections 2023-10-16T12:29:01.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.981+0000 7fe10ca68700 1 --2- 172.21.15.16:0/2644374307 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1080ce200 0x7fe1080ce6b0 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:01.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.981+0000 7fe10ca68700 1 --2- 172.21.15.16:0/2644374307 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe0f0003b90 0x7fe10814d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:01.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.981+0000 7fe10ca68700 1 --2- 172.21.15.16:0/2644374307 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fe0e8070ef0 0x7fe0e80732e0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:01.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.981+0000 7fe10ca68700 1 --2- 172.21.15.16:0/2644374307 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe1081514c0 0x7fe1081f28a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:01.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.981+0000 7fe10ca68700 1 -- 172.21.15.16:0/2644374307 >> 172.21.15.16:0/2644374307 conn(0x7fe1080d3630 msgr2=0x7fe1080b1780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:01.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.981+0000 7fe10ca68700 1 -- 172.21.15.16:0/2644374307 shutdown_connections 2023-10-16T12:29:01.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:01.981+0000 7fe10ca68700 1 -- 172.21.15.16:0/2644374307 wait complete. 2023-10-16T12:29:01.986 INFO:teuthology.orchestra.run.smithi016.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-16T12:29:01.987 INFO:tasks.ceph.mds.b.smithi049.stderr: -8771> 2023-10-16T12:27:39.968+0000 7f6ea4143700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T12:29:01.995 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-16T12:29:02.022 INFO:tasks.ceph.mds.b.smithi049.stdout:starting mds.b at 2023-10-16T12:29:02.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:02.120+0000 7f5ae6698700 1 Processor -- start 2023-10-16T12:29:02.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:02.120+0000 7f5ae6698700 1 -- start start 2023-10-16T12:29:02.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:02.121+0000 7f5ae6698700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5ae015b2c0 0x7f5ae0161340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:02.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:02.121+0000 7f5ae6698700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5ae006a7e0 0x7f5ae006ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:02.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:02.121+0000 7f5ae6698700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ae015ecb0 con 0x7f5ae006e570 2023-10-16T12:29:02.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:02.121+0000 7f5ae6698700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ae006b1b0 con 0x7f5ae006a7e0 2023-10-16T12:29:02.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:02.121+0000 7f5ae6698700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ae0161880 con 0x7f5ae015b2c0 2023-10-16T12:29:02.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:02.121+0000 7f5ae4c35700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5ae006a7e0 0x7f5ae006ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:02.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:02.121+0000 7f5adf7fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f5ae006e570 0x7f5ae015e5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:60876/0 (socket says 172.21.15.16:60876) 2023-10-16T12:29:02.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:02.121+0000 7f5adf7fe700 1 -- 172.21.15.16:0/4237818200 learned_addr learned my addr 172.21.15.16:0/4237818200 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:29:02.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:02.121+0000 7f5adffff700 1 --2- 172.21.15.16:0/4237818200 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5ae015b2c0 0x7f5ae0161340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:02.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:02.122+0000 7f5ae4c35700 1 -- 172.21.15.16:0/4237818200 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5ae015b2c0 msgr2=0x7f5ae0161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:29:02.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:02.122+0000 7f5ae4c35700 1 --2- 172.21.15.16:0/4237818200 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5ae015b2c0 0x7f5ae0161340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:02.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:02.122+0000 7f5ae4c35700 1 -- 172.21.15.16:0/4237818200 >> v1:172.21.15.16:6789/0 conn(0x7f5ae006e570 legacy=0x7f5ae015e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:02.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:02.122+0000 7f5adeffd700 1 -- 172.21.15.16:0/4237818200 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4188190299 0 0) 0x7f5ae015ecb0 con 0x7f5ae006e570 2023-10-16T12:29:02.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:02.122+0000 7f5ae4c35700 1 -- 172.21.15.16:0/4237818200 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ae015b6a0 con 0x7f5ae006a7e0 2023-10-16T12:29:02.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:02.122+0000 7f5adffff700 1 --2- 172.21.15.16:0/4237818200 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5ae015b2c0 0x7f5ae0161340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:29:02.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:02.122+0000 7f5ae4c35700 1 --2- 172.21.15.16:0/4237818200 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ae006a7e0 0x7f5ae006ac70 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f5ad8009d90 tx=0x7f5ad800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8106dfd1d0b0c20 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:02.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:02.122+0000 7f5adeffd700 1 -- 172.21.15.16:0/4237818200 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ad800e070 con 0x7f5ae006a7e0 2023-10-16T12:29:02.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:02.122+0000 7f5adeffd700 1 -- 172.21.15.16:0/4237818200 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ad800bbd0 con 0x7f5ae006a7e0 2023-10-16T12:29:02.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:02.122+0000 7f5adeffd700 1 -- 172.21.15.16:0/4237818200 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ad8012430 con 0x7f5ae006a7e0 2023-10-16T12:29:02.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:02.122+0000 7f5ae6698700 1 -- 172.21.15.16:0/4237818200 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ae006a7e0 msgr2=0x7f5ae006ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:02.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:02.122+0000 7f5ae6698700 1 --2- 172.21.15.16:0/4237818200 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ae006a7e0 0x7f5ae006ac70 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f5ad8009d90 tx=0x7f5ad800b330 comp rx=0 tx=0).stop 2023-10-16T12:29:02.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:02.122+0000 7f5ae6698700 1 -- 172.21.15.16:0/4237818200 shutdown_connections 2023-10-16T12:29:02.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:02.122+0000 7f5ae6698700 1 --2- 172.21.15.16:0/4237818200 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ae006a7e0 0x7f5ae006ac70 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:02.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:02.122+0000 7f5ae6698700 1 --2- 172.21.15.16:0/4237818200 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5ae015b2c0 0x7f5ae0161340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:02.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:02.122+0000 7f5ae6698700 1 -- 172.21.15.16:0/4237818200 >> 172.21.15.16:0/4237818200 conn(0x7f5ae00b1f40 msgr2=0x7f5ae00b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:02.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:02.123+0000 7f5ae6698700 1 -- 172.21.15.16:0/4237818200 shutdown_connections 2023-10-16T12:29:02.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:02.123+0000 7f5ae6698700 1 -- 172.21.15.16:0/4237818200 wait complete. 2023-10-16T12:29:02.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:02.124+0000 7f5ae6698700 1 Processor -- start 2023-10-16T12:29:02.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:02.124+0000 7f5ae6698700 1 -- start start 2023-10-16T12:29:02.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:02.125+0000 7f5ae6698700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ae006a7e0 0x7f5ae00b3920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:02.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:02.125+0000 7f5ae6698700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5ae006e570 0x7f5ae01f2a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:02.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:02.125+0000 7f5ae6698700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5ae015b2c0 0x7f5ae01f2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:02.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:02.125+0000 7f5ae6698700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ae0160c80 con 0x7f5ae006a7e0 2023-10-16T12:29:02.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:02.125+0000 7f5ae6698700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ae0156eb0 con 0x7f5ae015b2c0 2023-10-16T12:29:02.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:02.125+0000 7f5ae6698700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ae0160000 con 0x7f5ae006e570 2023-10-16T12:29:02.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:02.125+0000 7f5adffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ae006a7e0 0x7f5ae00b3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:02.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:02.125+0000 7f5adffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ae006a7e0 0x7f5ae00b3920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43376/0 (socket says 172.21.15.16:43376) 2023-10-16T12:29:02.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:02.125+0000 7f5adffff700 1 -- 172.21.15.16:0/2846039912 learned_addr learned my addr 172.21.15.16:0/2846039912 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:29:02.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:02.125+0000 7f5ae4c35700 1 --2- 172.21.15.16:0/2846039912 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5ae015b2c0 0x7f5ae01f2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:02.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:02.125+0000 7f5adf7fe700 1 --2- 172.21.15.16:0/2846039912 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5ae006e570 0x7f5ae01f2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:02.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:02.125+0000 7f5adffff700 1 -- 172.21.15.16:0/2846039912 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5ae006e570 msgr2=0x7f5ae01f2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:02.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:02.125+0000 7f5adffff700 1 --2- 172.21.15.16:0/2846039912 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5ae006e570 0x7f5ae01f2a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:02.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:02.125+0000 7f5adffff700 1 -- 172.21.15.16:0/2846039912 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5ae015b2c0 msgr2=0x7f5ae01f2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:02.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:02.125+0000 7f5adffff700 1 --2- 172.21.15.16:0/2846039912 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5ae015b2c0 0x7f5ae01f2fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:02.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:02.125+0000 7f5adffff700 1 -- 172.21.15.16:0/2846039912 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ad8009a70 con 0x7f5ae006a7e0 2023-10-16T12:29:02.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:02.125+0000 7f5ae4c35700 1 --2- 172.21.15.16:0/2846039912 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5ae015b2c0 0x7f5ae01f2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:29:02.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:02.125+0000 7f5adffff700 1 --2- 172.21.15.16:0/2846039912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ae006a7e0 0x7f5ae00b3920 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f5ad000c780 tx=0x7f5ad000cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:02.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:02.125+0000 7f5adcff9700 1 -- 172.21.15.16:0/2846039912 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ad0011800 con 0x7f5ae006a7e0 2023-10-16T12:29:02.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:02.125+0000 7f5adcff9700 1 -- 172.21.15.16:0/2846039912 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ad0011960 con 0x7f5ae006a7e0 2023-10-16T12:29:02.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:02.125+0000 7f5ae6698700 1 -- 172.21.15.16:0/2846039912 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ae01602e0 con 0x7f5ae006a7e0 2023-10-16T12:29:02.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:02.125+0000 7f5adcff9700 1 -- 172.21.15.16:0/2846039912 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ad0011cd0 con 0x7f5ae006a7e0 2023-10-16T12:29:02.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:02.126+0000 7f5ae6698700 1 -- 172.21.15.16:0/2846039912 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ae01608a0 con 0x7f5ae006a7e0 2023-10-16T12:29:02.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:02.126+0000 7f5ae6698700 1 -- 172.21.15.16:0/2846039912 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ae004ebb0 con 0x7f5ae006a7e0 2023-10-16T12:29:02.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:02.127+0000 7f5adcff9700 1 -- 172.21.15.16:0/2846039912 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 60) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f5ad0021350 con 0x7f5ae006a7e0 2023-10-16T12:29:02.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:02.127+0000 7f5adcff9700 1 --2- 172.21.15.16:0/2846039912 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f5acc06ad50 0x7f5acc06d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:02.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:02.128+0000 7f5adf7fe700 1 --2- 172.21.15.16:0/2846039912 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f5acc06ad50 0x7f5acc06d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:02.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:02.128+0000 7f5adcff9700 1 -- 172.21.15.16:0/2846039912 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 7033+0+0 (secure 0 0 0) 0x7f5ad0089750 con 0x7f5ae006a7e0 2023-10-16T12:29:02.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:02.128+0000 7f5adf7fe700 1 --2- 172.21.15.16:0/2846039912 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f5acc06ad50 0x7f5acc06d210 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f5ae01f0660 tx=0x7f5ad4009040 comp rx=0 tx=0).ready entity=mgr.7884 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:02.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:02.132+0000 7f5adcff9700 1 -- 172.21.15.16:0/2846039912 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5ad0054590 con 0x7f5ae006a7e0 2023-10-16T12:29:02.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:02.392+0000 7f5ae6698700 1 -- 172.21.15.16:0/2846039912 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f5ae00b6370 con 0x7f5ae006a7e0 2023-10-16T12:29:02.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:02.979+0000 7f5adcff9700 1 -- 172.21.15.16:0/2846039912 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v55) v1 ==== 104+0+0 (secure 0 0 0) 0x7f5ad0057bb0 con 0x7f5ae006a7e0 2023-10-16T12:29:02.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:02.982+0000 7f5ae6698700 1 -- 172.21.15.16:0/2846039912 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f5acc06ad50 msgr2=0x7f5acc06d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:02.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:02.982+0000 7f5ae6698700 1 --2- 172.21.15.16:0/2846039912 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f5acc06ad50 0x7f5acc06d210 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f5ae01f0660 tx=0x7f5ad4009040 comp rx=0 tx=0).stop 2023-10-16T12:29:02.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:02.982+0000 7f5ae6698700 1 -- 172.21.15.16:0/2846039912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ae006a7e0 msgr2=0x7f5ae00b3920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:02.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:02.982+0000 7f5ae6698700 1 --2- 172.21.15.16:0/2846039912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ae006a7e0 0x7f5ae00b3920 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f5ad000c780 tx=0x7f5ad000cb40 comp rx=0 tx=0).stop 2023-10-16T12:29:02.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:02.982+0000 7f5ae6698700 1 -- 172.21.15.16:0/2846039912 shutdown_connections 2023-10-16T12:29:02.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:02.982+0000 7f5ae6698700 1 --2- 172.21.15.16:0/2846039912 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5ae015b2c0 0x7f5ae01f2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:02.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:02.982+0000 7f5ae6698700 1 --2- 172.21.15.16:0/2846039912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ae006a7e0 0x7f5ae00b3920 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:02.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:02.982+0000 7f5ae6698700 1 --2- 172.21.15.16:0/2846039912 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f5acc06ad50 0x7f5acc06d210 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:02.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:02.982+0000 7f5ae6698700 1 --2- 172.21.15.16:0/2846039912 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5ae006e570 0x7f5ae01f2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:02.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:02.982+0000 7f5ae6698700 1 -- 172.21.15.16:0/2846039912 >> 172.21.15.16:0/2846039912 conn(0x7f5ae00b1f40 msgr2=0x7f5ae00cce00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:02.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:02.982+0000 7f5ae6698700 1 -- 172.21.15.16:0/2846039912 shutdown_connections 2023-10-16T12:29:02.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:02.982+0000 7f5ae6698700 1 -- 172.21.15.16:0/2846039912 wait complete. 2023-10-16T12:29:03.000 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-16T12:29:03.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:03.134+0000 7f34e9c60700 1 Processor -- start 2023-10-16T12:29:03.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:03.134+0000 7f34e9c60700 1 -- start start 2023-10-16T12:29:03.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:03.135+0000 7f34e9c60700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f34e40cecf0 0x7f34e40cffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:03.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:03.135+0000 7f34e9c60700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f34e40d04e0 0x7f34e40b6b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:03.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:03.135+0000 7f34e9c60700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34e40b6800 con 0x7f34e415d890 2023-10-16T12:29:03.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:03.135+0000 7f34e9c60700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34e40d0950 con 0x7f34e40cecf0 2023-10-16T12:29:03.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:03.135+0000 7f34e9c60700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34e40d2580 con 0x7f34e40d04e0 2023-10-16T12:29:03.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:03.135+0000 7f34e2ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f34e40cecf0 0x7f34e40cffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:03.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:03.135+0000 7f34e2ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f34e40cecf0 0x7f34e40cffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43404/0 (socket says 172.21.15.16:43404) 2023-10-16T12:29:03.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:03.135+0000 7f34e2ffd700 1 -- 172.21.15.16:0/1997777850 learned_addr learned my addr 172.21.15.16:0/1997777850 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:29:03.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:03.135+0000 7f34e3fff700 1 --2- 172.21.15.16:0/1997777850 >> v2:172.21.15.49:3300/0 conn(0x7f34e40d04e0 0x7f34e40b6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:03.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:03.135+0000 7f34e2ffd700 1 -- 172.21.15.16:0/1997777850 >> v2:172.21.15.49:3300/0 conn(0x7f34e40d04e0 msgr2=0x7f34e40b6b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:29:03.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:03.135+0000 7f34e2ffd700 1 --2- 172.21.15.16:0/1997777850 >> v2:172.21.15.49:3300/0 conn(0x7f34e40d04e0 0x7f34e40b6b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:03.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:03.135+0000 7f34e2ffd700 1 -- 172.21.15.16:0/1997777850 >> v1:172.21.15.16:6789/0 conn(0x7f34e415d890 legacy=0x7f34e40ce5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:03.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:03.135+0000 7f34e2ffd700 1 -- 172.21.15.16:0/1997777850 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34e40d33e0 con 0x7f34e40cecf0 2023-10-16T12:29:03.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:03.135+0000 7f34e2ffd700 1 --2- 172.21.15.16:0/1997777850 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34e40cecf0 0x7f34e40cffa0 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f34e40d0c20 tx=0x7f34d8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73e207e1a9606acb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:03.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:03.135+0000 7f34e27fc700 1 -- 172.21.15.16:0/1997777850 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34d800b870 con 0x7f34e40cecf0 2023-10-16T12:29:03.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:03.136+0000 7f34e27fc700 1 -- 172.21.15.16:0/1997777850 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34d800b9d0 con 0x7f34e40cecf0 2023-10-16T12:29:03.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:03.136+0000 7f34e27fc700 1 -- 172.21.15.16:0/1997777850 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34d800bd60 con 0x7f34e40cecf0 2023-10-16T12:29:03.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:03.136+0000 7f34e9c60700 1 -- 172.21.15.16:0/1997777850 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34e40cecf0 msgr2=0x7f34e40cffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:03.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:03.136+0000 7f34e9c60700 1 --2- 172.21.15.16:0/1997777850 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34e40cecf0 0x7f34e40cffa0 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f34e40d0c20 tx=0x7f34d8009bd0 comp rx=0 tx=0).stop 2023-10-16T12:29:03.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:03.136+0000 7f34e9c60700 1 -- 172.21.15.16:0/1997777850 shutdown_connections 2023-10-16T12:29:03.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:03.136+0000 7f34e9c60700 1 --2- 172.21.15.16:0/1997777850 >> v2:172.21.15.49:3300/0 conn(0x7f34e40d04e0 0x7f34e40b6b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:03.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:03.136+0000 7f34e9c60700 1 --2- 172.21.15.16:0/1997777850 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34e40cecf0 0x7f34e40cffa0 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:03.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:03.136+0000 7f34e9c60700 1 -- 172.21.15.16:0/1997777850 >> 172.21.15.16:0/1997777850 conn(0x7f34e4072360 msgr2=0x7f34e4072770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:03.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:03.136+0000 7f34e9c60700 1 -- 172.21.15.16:0/1997777850 shutdown_connections 2023-10-16T12:29:03.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:03.136+0000 7f34e9c60700 1 -- 172.21.15.16:0/1997777850 wait complete. 2023-10-16T12:29:03.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:03.137+0000 7f34e9c60700 1 Processor -- start 2023-10-16T12:29:03.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:03.138+0000 7f34e9c60700 1 -- start start 2023-10-16T12:29:03.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:03.138+0000 7f34e9c60700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34e40cecf0 0x7f34e4151c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:03.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:03.138+0000 7f34e9c60700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f34e40d04e0 0x7f34e4156740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:03.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:03.138+0000 7f34e9c60700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f34e415d890 0x7f34e4160420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:03.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:03.138+0000 7f34e9c60700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34e40cc4e0 con 0x7f34e40cecf0 2023-10-16T12:29:03.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:03.138+0000 7f34e9c60700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34e40cb8d0 con 0x7f34e40d04e0 2023-10-16T12:29:03.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:03.138+0000 7f34e9c60700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34e40cbbb0 con 0x7f34e415d890 2023-10-16T12:29:03.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:03.138+0000 7f34e37fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34e40cecf0 0x7f34e4151c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:03.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:03.138+0000 7f34e37fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34e40cecf0 0x7f34e4151c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43408/0 (socket says 172.21.15.16:43408) 2023-10-16T12:29:03.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:03.138+0000 7f34e37fe700 1 -- 172.21.15.16:0/1099812536 learned_addr learned my addr 172.21.15.16:0/1099812536 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:29:03.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:03.139+0000 7f34e2ffd700 1 --2- 172.21.15.16:0/1099812536 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f34e40d04e0 0x7f34e4156740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:03.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:03.139+0000 7f34e3fff700 1 --2- 172.21.15.16:0/1099812536 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f34e415d890 0x7f34e4160420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:03.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:03.139+0000 7f34e37fe700 1 -- 172.21.15.16:0/1099812536 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f34e415d890 msgr2=0x7f34e4160420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:03.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:03.139+0000 7f34e37fe700 1 --2- 172.21.15.16:0/1099812536 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f34e415d890 0x7f34e4160420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:03.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:03.139+0000 7f34e37fe700 1 -- 172.21.15.16:0/1099812536 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f34e40d04e0 msgr2=0x7f34e4156740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:03.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:03.139+0000 7f34e37fe700 1 --2- 172.21.15.16:0/1099812536 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f34e40d04e0 0x7f34e4156740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:03.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:03.139+0000 7f34e37fe700 1 -- 172.21.15.16:0/1099812536 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34d8009550 con 0x7f34e40cecf0 2023-10-16T12:29:03.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:03.139+0000 7f34e37fe700 1 --2- 172.21.15.16:0/1099812536 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34e40cecf0 0x7f34e4151c20 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f34d400c450 tx=0x7f34d400c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:03.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:03.139+0000 7f34cbfff700 1 -- 172.21.15.16:0/1099812536 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34d40052d0 con 0x7f34e40cecf0 2023-10-16T12:29:03.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:03.139+0000 7f34cbfff700 1 -- 172.21.15.16:0/1099812536 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34d4005430 con 0x7f34e40cecf0 2023-10-16T12:29:03.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:03.139+0000 7f34e9c60700 1 -- 172.21.15.16:0/1099812536 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34e40cbe90 con 0x7f34e40cecf0 2023-10-16T12:29:03.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:03.139+0000 7f34cbfff700 1 -- 172.21.15.16:0/1099812536 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34d40038c0 con 0x7f34e40cecf0 2023-10-16T12:29:03.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:03.139+0000 7f34e9c60700 1 -- 172.21.15.16:0/1099812536 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34e4152160 con 0x7f34e40cecf0 2023-10-16T12:29:03.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:03.140+0000 7f34e9c60700 1 -- 172.21.15.16:0/1099812536 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34e40d2d70 con 0x7f34e40cecf0 2023-10-16T12:29:03.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:03.141+0000 7f34cbfff700 1 -- 172.21.15.16:0/1099812536 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 60) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f34d4003a20 con 0x7f34e40cecf0 2023-10-16T12:29:03.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:03.141+0000 7f34cbfff700 1 --2- 172.21.15.16:0/1099812536 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f34d006ad50 0x7f34d006d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:03.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:03.141+0000 7f34cbfff700 1 -- 172.21.15.16:0/1099812536 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 7033+0+0 (secure 0 0 0) 0x7f34d4088a80 con 0x7f34e40cecf0 2023-10-16T12:29:03.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:03.141+0000 7f34e2ffd700 1 --2- 172.21.15.16:0/1099812536 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f34d006ad50 0x7f34d006d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:03.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:03.141+0000 7f34e2ffd700 1 --2- 172.21.15.16:0/1099812536 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f34d006ad50 0x7f34d006d210 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f34d8000c00 tx=0x7f34d8008030 comp rx=0 tx=0).ready entity=mgr.7884 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:03.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:03.146+0000 7f34cbfff700 1 -- 172.21.15.16:0/1099812536 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f34e40d2d70 con 0x7f34e40cecf0 2023-10-16T12:29:03.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:03.348+0000 7f34e9c60700 1 -- 172.21.15.16:0/1099812536 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f34e40d2d70 con 0x7f34e40cecf0 2023-10-16T12:29:03.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:03.911+0000 7f34cbfff700 1 -- 172.21.15.16:0/1099812536 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 61) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f34d4053d10 con 0x7f34e40cecf0 2023-10-16T12:29:04.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.326+0000 7f34cbfff700 1 -- 172.21.15.16:0/1099812536 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v118) v1 ==== 181+0+0 (secure 0 0 0) 0x7f34d4074520 con 0x7f34e40cecf0 2023-10-16T12:29:04.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.330+0000 7f34e9c60700 1 -- 172.21.15.16:0/1099812536 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f34d006ad50 msgr2=0x7f34d006d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:04.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.330+0000 7f34e9c60700 1 --2- 172.21.15.16:0/1099812536 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f34d006ad50 0x7f34d006d210 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f34d8000c00 tx=0x7f34d8008030 comp rx=0 tx=0).stop 2023-10-16T12:29:04.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.330+0000 7f34e9c60700 1 -- 172.21.15.16:0/1099812536 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34e40cecf0 msgr2=0x7f34e4151c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:04.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.330+0000 7f34e9c60700 1 --2- 172.21.15.16:0/1099812536 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34e40cecf0 0x7f34e4151c20 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f34d400c450 tx=0x7f34d400c810 comp rx=0 tx=0).stop 2023-10-16T12:29:04.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.331+0000 7f34e9c60700 1 -- 172.21.15.16:0/1099812536 shutdown_connections 2023-10-16T12:29:04.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.331+0000 7f34e9c60700 1 --2- 172.21.15.16:0/1099812536 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f34d006ad50 0x7f34d006d210 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:04.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.331+0000 7f34e9c60700 1 --2- 172.21.15.16:0/1099812536 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f34e415d890 0x7f34e4160420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:04.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.331+0000 7f34e9c60700 1 --2- 172.21.15.16:0/1099812536 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34e40cecf0 0x7f34e4151c20 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:04.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.331+0000 7f34e9c60700 1 --2- 172.21.15.16:0/1099812536 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f34e40d04e0 0x7f34e4156740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:04.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.331+0000 7f34e9c60700 1 -- 172.21.15.16:0/1099812536 >> 172.21.15.16:0/1099812536 conn(0x7f34e4072360 msgr2=0x7f34e415be90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:04.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.332+0000 7f34e9c60700 1 -- 172.21.15.16:0/1099812536 shutdown_connections 2023-10-16T12:29:04.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.332+0000 7f34e9c60700 1 -- 172.21.15.16:0/1099812536 wait complete. 2023-10-16T12:29:04.336 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' removed 2023-10-16T12:29:04.350 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-16T12:29:04.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.484+0000 7fc35964e700 1 Processor -- start 2023-10-16T12:29:04.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.484+0000 7fc35964e700 1 -- start start 2023-10-16T12:29:04.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.485+0000 7fc35964e700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fc35415d7a0 0x7fc3540b7240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:04.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.485+0000 7fc35964e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc3540b7780 0x7fc3540ce260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:04.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.485+0000 7fc35964e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc3540ce7a0 0x7fc3540b5010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:04.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.485+0000 7fc35964e700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3540b8980 con 0x7fc3540ce7a0 2023-10-16T12:29:04.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.485+0000 7fc35964e700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3540cec30 con 0x7fc35415d7a0 2023-10-16T12:29:04.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.485+0000 7fc35964e700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3540b91b0 con 0x7fc3540b7780 2023-10-16T12:29:04.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.485+0000 7fc3537fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc3540ce7a0 0x7fc3540b5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:04.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.485+0000 7fc3537fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc3540ce7a0 0x7fc3540b5010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43420/0 (socket says 172.21.15.16:43420) 2023-10-16T12:29:04.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.485+0000 7fc3537fe700 1 -- 172.21.15.16:0/3220228113 learned_addr learned my addr 172.21.15.16:0/3220228113 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:29:04.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.485+0000 7fc352ffd700 1 --2- 172.21.15.16:0/3220228113 >> v2:172.21.15.49:3300/0 conn(0x7fc35415d7a0 0x7fc3540b7240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:04.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.485+0000 7fc3527fc700 1 --2- 172.21.15.16:0/3220228113 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc3540b7780 0x7fc3540ce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:04.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.485+0000 7fc3537fe700 1 -- 172.21.15.16:0/3220228113 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc3540b7780 msgr2=0x7fc3540ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:29:04.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.485+0000 7fc3537fe700 1 --2- 172.21.15.16:0/3220228113 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc3540b7780 0x7fc3540ce260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:04.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.485+0000 7fc3537fe700 1 -- 172.21.15.16:0/3220228113 >> v2:172.21.15.49:3300/0 conn(0x7fc35415d7a0 msgr2=0x7fc3540b7240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:29:04.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.485+0000 7fc3537fe700 1 --2- 172.21.15.16:0/3220228113 >> v2:172.21.15.49:3300/0 conn(0x7fc35415d7a0 0x7fc3540b7240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:04.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.485+0000 7fc3537fe700 1 -- 172.21.15.16:0/3220228113 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3540b7d00 con 0x7fc3540ce7a0 2023-10-16T12:29:04.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.485+0000 7fc3527fc700 1 --2- 172.21.15.16:0/3220228113 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc3540b7780 0x7fc3540ce260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:29:04.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.486+0000 7fc3537fe700 1 --2- 172.21.15.16:0/3220228113 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3540ce7a0 0x7fc3540b5010 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7fc35406b0c0 tx=0x7fc34c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c11ace03892eae14 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:04.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.486+0000 7fc351ffb700 1 -- 172.21.15.16:0/3220228113 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc34c00e070 con 0x7fc3540ce7a0 2023-10-16T12:29:04.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.486+0000 7fc351ffb700 1 -- 172.21.15.16:0/3220228113 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc34c00b9c0 con 0x7fc3540ce7a0 2023-10-16T12:29:04.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.486+0000 7fc351ffb700 1 -- 172.21.15.16:0/3220228113 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc34c00bd70 con 0x7fc3540ce7a0 2023-10-16T12:29:04.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.486+0000 7fc35964e700 1 -- 172.21.15.16:0/3220228113 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3540ce7a0 msgr2=0x7fc3540b5010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:04.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.486+0000 7fc35964e700 1 --2- 172.21.15.16:0/3220228113 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3540ce7a0 0x7fc3540b5010 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7fc35406b0c0 tx=0x7fc34c00b330 comp rx=0 tx=0).stop 2023-10-16T12:29:04.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.486+0000 7fc35964e700 1 -- 172.21.15.16:0/3220228113 shutdown_connections 2023-10-16T12:29:04.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.486+0000 7fc35964e700 1 --2- 172.21.15.16:0/3220228113 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3540ce7a0 0x7fc3540b5010 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:04.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.486+0000 7fc35964e700 1 --2- 172.21.15.16:0/3220228113 >> v2:172.21.15.49:3300/0 conn(0x7fc35415d7a0 0x7fc3540b7240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:04.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.486+0000 7fc35964e700 1 --2- 172.21.15.16:0/3220228113 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc3540b7780 0x7fc3540ce260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:04.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.486+0000 7fc35964e700 1 -- 172.21.15.16:0/3220228113 >> 172.21.15.16:0/3220228113 conn(0x7fc354072330 msgr2=0x7fc354072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:04.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.486+0000 7fc35964e700 1 -- 172.21.15.16:0/3220228113 shutdown_connections 2023-10-16T12:29:04.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.486+0000 7fc35964e700 1 -- 172.21.15.16:0/3220228113 wait complete. 2023-10-16T12:29:04.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.488+0000 7fc35964e700 1 Processor -- start 2023-10-16T12:29:04.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.488+0000 7fc35964e700 1 -- start start 2023-10-16T12:29:04.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.489+0000 7fc35964e700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc3540b7780 0x7fc354150380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:04.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.489+0000 7fc35964e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3540ce7a0 0x7fc3541f6470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:04.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.489+0000 7fc35964e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc35415d7a0 0x7fc3541f69b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:04.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.489+0000 7fc35964e700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3540c7520 con 0x7fc3540ce7a0 2023-10-16T12:29:04.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.489+0000 7fc35964e700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3540cc920 con 0x7fc3540b7780 2023-10-16T12:29:04.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.489+0000 7fc35964e700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3540b4d90 con 0x7fc35415d7a0 2023-10-16T12:29:04.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.489+0000 7fc3527fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3540ce7a0 0x7fc3541f6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:04.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.489+0000 7fc3527fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3540ce7a0 0x7fc3541f6470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43424/0 (socket says 172.21.15.16:43424) 2023-10-16T12:29:04.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.489+0000 7fc3527fc700 1 -- 172.21.15.16:0/1236159478 learned_addr learned my addr 172.21.15.16:0/1236159478 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:29:04.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.489+0000 7fc352ffd700 1 --2- 172.21.15.16:0/1236159478 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc3540b7780 0x7fc354150380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:04.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.489+0000 7fc3537fe700 1 --2- 172.21.15.16:0/1236159478 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc35415d7a0 0x7fc3541f69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:04.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.489+0000 7fc3527fc700 1 -- 172.21.15.16:0/1236159478 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc35415d7a0 msgr2=0x7fc3541f69b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:04.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.489+0000 7fc3527fc700 1 --2- 172.21.15.16:0/1236159478 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc35415d7a0 0x7fc3541f69b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:04.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.489+0000 7fc3527fc700 1 -- 172.21.15.16:0/1236159478 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc3540b7780 msgr2=0x7fc354150380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:04.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.489+0000 7fc3527fc700 1 --2- 172.21.15.16:0/1236159478 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc3540b7780 0x7fc354150380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:04.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.489+0000 7fc3527fc700 1 -- 172.21.15.16:0/1236159478 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc34c009b30 con 0x7fc3540ce7a0 2023-10-16T12:29:04.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.489+0000 7fc352ffd700 1 --2- 172.21.15.16:0/1236159478 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc3540b7780 0x7fc354150380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:29:04.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.489+0000 7fc3537fe700 1 --2- 172.21.15.16:0/1236159478 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc35415d7a0 0x7fc3541f69b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:29:04.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.490+0000 7fc3527fc700 1 --2- 172.21.15.16:0/1236159478 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3540ce7a0 0x7fc3541f6470 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7fc34400c740 tx=0x7fc34400cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:04.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.490+0000 7fc33b7fe700 1 -- 172.21.15.16:0/1236159478 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc344011800 con 0x7fc3540ce7a0 2023-10-16T12:29:04.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.490+0000 7fc33b7fe700 1 -- 172.21.15.16:0/1236159478 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc344011960 con 0x7fc3540ce7a0 2023-10-16T12:29:04.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.490+0000 7fc33b7fe700 1 -- 172.21.15.16:0/1236159478 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc344011cd0 con 0x7fc3540ce7a0 2023-10-16T12:29:04.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.490+0000 7fc35964e700 1 -- 172.21.15.16:0/1236159478 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3541508c0 con 0x7fc3540ce7a0 2023-10-16T12:29:04.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.490+0000 7fc35964e700 1 -- 172.21.15.16:0/1236159478 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3540cc260 con 0x7fc3540ce7a0 2023-10-16T12:29:04.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.491+0000 7fc35964e700 1 -- 172.21.15.16:0/1236159478 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3540cca60 con 0x7fc3540ce7a0 2023-10-16T12:29:04.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.491+0000 7fc33b7fe700 1 -- 172.21.15.16:0/1236159478 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 61) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fc344021350 con 0x7fc3540ce7a0 2023-10-16T12:29:04.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.492+0000 7fc33b7fe700 1 --2- 172.21.15.16:0/1236159478 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fc34006af90 0x7fc34006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:04.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.492+0000 7fc33b7fe700 1 -- 172.21.15.16:0/1236159478 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6595+0+0 (secure 0 0 0) 0x7fc344089a00 con 0x7fc3540ce7a0 2023-10-16T12:29:04.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.492+0000 7fc352ffd700 1 --2- 172.21.15.16:0/1236159478 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fc34006af90 0x7fc34006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:04.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.492+0000 7fc352ffd700 1 --2- 172.21.15.16:0/1236159478 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fc34006af90 0x7fc34006d450 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc348009820 tx=0x7fc348008040 comp rx=0 tx=0).ready entity=mgr.7884 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:04.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.496+0000 7fc33b7fe700 1 -- 172.21.15.16:0/1236159478 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc3540cca60 con 0x7fc3540ce7a0 2023-10-16T12:29:04.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:04.701+0000 7fc35964e700 1 -- 172.21.15.16:0/1236159478 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc3540cca60 con 0x7fc3540ce7a0 2023-10-16T12:29:05.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.338+0000 7fc33b7fe700 1 -- 172.21.15.16:0/1236159478 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v119) v1 ==== 169+0+0 (secure 0 0 0) 0x7fc3540cca60 con 0x7fc3540ce7a0 2023-10-16T12:29:05.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.343+0000 7fc35964e700 1 -- 172.21.15.16:0/1236159478 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fc34006af90 msgr2=0x7fc34006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:05.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.343+0000 7fc35964e700 1 --2- 172.21.15.16:0/1236159478 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fc34006af90 0x7fc34006d450 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc348009820 tx=0x7fc348008040 comp rx=0 tx=0).stop 2023-10-16T12:29:05.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.343+0000 7fc35964e700 1 -- 172.21.15.16:0/1236159478 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3540ce7a0 msgr2=0x7fc3541f6470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:05.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.343+0000 7fc35964e700 1 --2- 172.21.15.16:0/1236159478 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3540ce7a0 0x7fc3541f6470 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7fc34400c740 tx=0x7fc34400cb00 comp rx=0 tx=0).stop 2023-10-16T12:29:05.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.345+0000 7fc35964e700 1 -- 172.21.15.16:0/1236159478 shutdown_connections 2023-10-16T12:29:05.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.345+0000 7fc35964e700 1 --2- 172.21.15.16:0/1236159478 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fc34006af90 0x7fc34006d450 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:05.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.345+0000 7fc35964e700 1 --2- 172.21.15.16:0/1236159478 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc35415d7a0 0x7fc3541f69b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:05.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.345+0000 7fc35964e700 1 --2- 172.21.15.16:0/1236159478 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc3540b7780 0x7fc354150380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:05.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.345+0000 7fc35964e700 1 --2- 172.21.15.16:0/1236159478 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3540ce7a0 0x7fc3541f6470 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:05.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.345+0000 7fc35964e700 1 -- 172.21.15.16:0/1236159478 >> 172.21.15.16:0/1236159478 conn(0x7fc354072330 msgr2=0x7fc3540cdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:05.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.346+0000 7fc35964e700 1 -- 172.21.15.16:0/1236159478 shutdown_connections 2023-10-16T12:29:05.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.346+0000 7fc35964e700 1 -- 172.21.15.16:0/1236159478 wait complete. 2023-10-16T12:29:05.351 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' removed 2023-10-16T12:29:05.365 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-16T12:29:05.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.503+0000 7faa20896700 1 Processor -- start 2023-10-16T12:29:05.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.503+0000 7faa20896700 1 -- start start 2023-10-16T12:29:05.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.504+0000 7faa20896700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faa1c0ce6a0 0x7faa1c06c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:05.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.504+0000 7faa20896700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7faa1c0d2110 0x7faa1c06c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:05.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.504+0000 7faa20896700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa1c0b20f0 con 0x7faa1c0b2620 2023-10-16T12:29:05.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.504+0000 7faa20896700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa1c0af1f0 con 0x7faa1c0d2110 2023-10-16T12:29:05.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.504+0000 7faa20896700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa1c0d03d0 con 0x7faa1c0ce6a0 2023-10-16T12:29:05.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.505+0000 7faa19d9b700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7faa1c0d2110 0x7faa1c06c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:05.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.505+0000 7faa1a59c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faa1c0ce6a0 0x7faa1c06c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:05.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.505+0000 7faa1ad9d700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7faa1c0b2620 0x7faa1c0b0f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:39888/0 (socket says 172.21.15.16:39888) 2023-10-16T12:29:05.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.505+0000 7faa1ad9d700 1 -- 172.21.15.16:0/2634290918 learned_addr learned my addr 172.21.15.16:0/2634290918 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:29:05.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.505+0000 7faa1959a700 1 -- 172.21.15.16:0/2634290918 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3435543190 0 0) 0x7faa1c0b20f0 con 0x7faa1c0b2620 2023-10-16T12:29:05.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.505+0000 7faa1959a700 1 -- 172.21.15.16:0/2634290918 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa040035b0 con 0x7faa1c0b2620 2023-10-16T12:29:05.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.505+0000 7faa19d9b700 1 -- 172.21.15.16:0/2634290918 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faa1c0ce6a0 msgr2=0x7faa1c06c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:29:05.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.505+0000 7faa19d9b700 1 --2- 172.21.15.16:0/2634290918 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faa1c0ce6a0 0x7faa1c06c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:05.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.505+0000 7faa19d9b700 1 -- 172.21.15.16:0/2634290918 >> v1:172.21.15.49:6789/0 conn(0x7faa1c0b2620 legacy=0x7faa1c0b0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:05.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.505+0000 7faa19d9b700 1 -- 172.21.15.16:0/2634290918 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa1c0b2340 con 0x7faa1c0d2110 2023-10-16T12:29:05.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.505+0000 7faa1a59c700 1 --2- 172.21.15.16:0/2634290918 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faa1c0ce6a0 0x7faa1c06c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:29:05.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.505+0000 7faa19d9b700 1 --2- 172.21.15.16:0/2634290918 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faa1c0d2110 0x7faa1c06c840 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7faa0c009840 tx=0x7faa0c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3960e60d2bbceb9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:05.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.505+0000 7faa1959a700 1 -- 172.21.15.16:0/2634290918 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa0c011830 con 0x7faa1c0d2110 2023-10-16T12:29:05.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.506+0000 7faa1959a700 1 -- 172.21.15.16:0/2634290918 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa0c011990 con 0x7faa1c0d2110 2023-10-16T12:29:05.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.506+0000 7faa1959a700 1 -- 172.21.15.16:0/2634290918 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa0c011d20 con 0x7faa1c0d2110 2023-10-16T12:29:05.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.506+0000 7faa20896700 1 -- 172.21.15.16:0/2634290918 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faa1c0d2110 msgr2=0x7faa1c06c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:05.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.506+0000 7faa20896700 1 --2- 172.21.15.16:0/2634290918 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faa1c0d2110 0x7faa1c06c840 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7faa0c009840 tx=0x7faa0c009b50 comp rx=0 tx=0).stop 2023-10-16T12:29:05.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.506+0000 7faa20896700 1 -- 172.21.15.16:0/2634290918 shutdown_connections 2023-10-16T12:29:05.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.506+0000 7faa20896700 1 --2- 172.21.15.16:0/2634290918 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faa1c0ce6a0 0x7faa1c06c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:05.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.506+0000 7faa20896700 1 --2- 172.21.15.16:0/2634290918 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faa1c0d2110 0x7faa1c06c840 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:05.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.506+0000 7faa20896700 1 -- 172.21.15.16:0/2634290918 >> 172.21.15.16:0/2634290918 conn(0x7faa1c0d15f0 msgr2=0x7faa1c0d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:05.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.506+0000 7faa20896700 1 -- 172.21.15.16:0/2634290918 shutdown_connections 2023-10-16T12:29:05.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.506+0000 7faa20896700 1 -- 172.21.15.16:0/2634290918 wait complete. 2023-10-16T12:29:05.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.507+0000 7faa20896700 1 Processor -- start 2023-10-16T12:29:05.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.508+0000 7faa20896700 1 -- start start 2023-10-16T12:29:05.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.508+0000 7faa20896700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faa1c0b2620 0x7faa1c0c7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:05.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.508+0000 7faa20896700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faa1c0ce6a0 0x7faa1c0cb050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:05.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.508+0000 7faa20896700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa1c0d2110 0x7faa1c0c86c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:05.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.508+0000 7faa20896700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa1c1f3960 con 0x7faa1c0d2110 2023-10-16T12:29:05.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.508+0000 7faa20896700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa1c1f3c40 con 0x7faa1c0ce6a0 2023-10-16T12:29:05.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.508+0000 7faa20896700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa1c0b4310 con 0x7faa1c0b2620 2023-10-16T12:29:05.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.508+0000 7faa1ad9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa1c0d2110 0x7faa1c0c86c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:05.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.508+0000 7faa1ad9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa1c0d2110 0x7faa1c0c86c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43438/0 (socket says 172.21.15.16:43438) 2023-10-16T12:29:05.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.509+0000 7faa1ad9d700 1 -- 172.21.15.16:0/2828682484 learned_addr learned my addr 172.21.15.16:0/2828682484 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:29:05.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.509+0000 7faa19d9b700 1 --2- 172.21.15.16:0/2828682484 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faa1c0ce6a0 0x7faa1c0cb050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:05.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.509+0000 7faa1a59c700 1 --2- 172.21.15.16:0/2828682484 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faa1c0b2620 0x7faa1c0c7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:05.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.509+0000 7faa1ad9d700 1 -- 172.21.15.16:0/2828682484 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faa1c0b2620 msgr2=0x7faa1c0c7ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:05.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.509+0000 7faa1ad9d700 1 --2- 172.21.15.16:0/2828682484 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faa1c0b2620 0x7faa1c0c7ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:05.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.509+0000 7faa1ad9d700 1 -- 172.21.15.16:0/2828682484 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faa1c0ce6a0 msgr2=0x7faa1c0cb050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:05.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.509+0000 7faa1ad9d700 1 --2- 172.21.15.16:0/2828682484 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faa1c0ce6a0 0x7faa1c0cb050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:05.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.509+0000 7faa1ad9d700 1 -- 172.21.15.16:0/2828682484 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa100094a0 con 0x7faa1c0d2110 2023-10-16T12:29:05.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.509+0000 7faa1ad9d700 1 --2- 172.21.15.16:0/2828682484 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa1c0d2110 0x7faa1c0c86c0 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7faa1400cb40 tx=0x7faa1400cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:05.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.509+0000 7faa0affd700 1 -- 172.21.15.16:0/2828682484 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa1400ab50 con 0x7faa1c0d2110 2023-10-16T12:29:05.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.509+0000 7faa0affd700 1 -- 172.21.15.16:0/2828682484 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa1400acb0 con 0x7faa1c0d2110 2023-10-16T12:29:05.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.509+0000 7faa20896700 1 -- 172.21.15.16:0/2828682484 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa0c0094a0 con 0x7faa1c0d2110 2023-10-16T12:29:05.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.509+0000 7faa0affd700 1 -- 172.21.15.16:0/2828682484 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa14012620 con 0x7faa1c0d2110 2023-10-16T12:29:05.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.509+0000 7faa20896700 1 -- 172.21.15.16:0/2828682484 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa1c0b48d0 con 0x7faa1c0d2110 2023-10-16T12:29:05.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.510+0000 7faa20896700 1 -- 172.21.15.16:0/2828682484 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa1c0b4ac0 con 0x7faa1c0d2110 2023-10-16T12:29:05.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.511+0000 7faa0affd700 1 -- 172.21.15.16:0/2828682484 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 62) v1 ==== 89331+0+0 (secure 0 0 0) 0x7faa1401b3b0 con 0x7faa1c0d2110 2023-10-16T12:29:05.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.511+0000 7faa0affd700 1 --2- 172.21.15.16:0/2828682484 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fa9fc06b020 0x7fa9fc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:05.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.511+0000 7faa0affd700 1 -- 172.21.15.16:0/2828682484 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6065+0+0 (secure 0 0 0) 0x7faa14089700 con 0x7faa1c0d2110 2023-10-16T12:29:05.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.511+0000 7faa1a59c700 1 --2- 172.21.15.16:0/2828682484 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fa9fc06b020 0x7fa9fc06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:05.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.511+0000 7faa1a59c700 1 --2- 172.21.15.16:0/2828682484 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fa9fc06b020 0x7fa9fc06d4e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7faa1c0b0c40 tx=0x7faa10008040 comp rx=0 tx=0).ready entity=mgr.7884 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:05.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.516+0000 7faa0affd700 1 -- 172.21.15.16:0/2828682484 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7faa14054530 con 0x7faa1c0d2110 2023-10-16T12:29:05.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.724+0000 7faa20896700 1 -- 172.21.15.16:0/2828682484 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7faa1c04ebb0 con 0x7faa1c0d2110 2023-10-16T12:29:05.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.727+0000 7faa0affd700 1 -- 172.21.15.16:0/2828682484 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v37) v1 ==== 115+0+0 (secure 0 0 0) 0x7faa14057b50 con 0x7faa1c0d2110 2023-10-16T12:29:05.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.730+0000 7faa20896700 1 -- 172.21.15.16:0/2828682484 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fa9fc06b020 msgr2=0x7fa9fc06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:05.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.730+0000 7faa20896700 1 --2- 172.21.15.16:0/2828682484 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fa9fc06b020 0x7fa9fc06d4e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7faa1c0b0c40 tx=0x7faa10008040 comp rx=0 tx=0).stop 2023-10-16T12:29:05.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.730+0000 7faa20896700 1 -- 172.21.15.16:0/2828682484 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa1c0d2110 msgr2=0x7faa1c0c86c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:05.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.730+0000 7faa20896700 1 --2- 172.21.15.16:0/2828682484 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa1c0d2110 0x7faa1c0c86c0 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7faa1400cb40 tx=0x7faa1400cf00 comp rx=0 tx=0).stop 2023-10-16T12:29:05.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.730+0000 7faa20896700 1 -- 172.21.15.16:0/2828682484 shutdown_connections 2023-10-16T12:29:05.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.730+0000 7faa20896700 1 --2- 172.21.15.16:0/2828682484 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa1c0d2110 0x7faa1c0c86c0 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:05.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.730+0000 7faa20896700 1 --2- 172.21.15.16:0/2828682484 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fa9fc06b020 0x7fa9fc06d4e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:05.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.730+0000 7faa20896700 1 --2- 172.21.15.16:0/2828682484 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faa1c0b2620 0x7faa1c0c7ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:05.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.730+0000 7faa20896700 1 --2- 172.21.15.16:0/2828682484 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faa1c0ce6a0 0x7faa1c0cb050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:05.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.730+0000 7faa20896700 1 -- 172.21.15.16:0/2828682484 >> 172.21.15.16:0/2828682484 conn(0x7faa1c0d15f0 msgr2=0x7faa1c160c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:05.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.730+0000 7faa20896700 1 -- 172.21.15.16:0/2828682484 shutdown_connections 2023-10-16T12:29:05.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.730+0000 7faa20896700 1 -- 172.21.15.16:0/2828682484 wait complete. 2023-10-16T12:29:05.746 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-16T12:29:05.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.871+0000 7f0a37e9c700 1 Processor -- start 2023-10-16T12:29:05.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.872+0000 7f0a37e9c700 1 -- start start 2023-10-16T12:29:05.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.872+0000 7f0a37e9c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0a3015b070 0x7f0a3015b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:05.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.872+0000 7f0a37e9c700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f0a300cb880 0x7f0a300cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:05.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.872+0000 7f0a37e9c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0a3006da60 0x7f0a3006def0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:05.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.872+0000 7f0a37e9c700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a3006e430 con 0x7f0a3015b070 2023-10-16T12:29:05.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.872+0000 7f0a37e9c700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a300cef70 con 0x7f0a300cb880 2023-10-16T12:29:05.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.872+0000 7f0a37e9c700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a300ce360 con 0x7f0a3006da60 2023-10-16T12:29:05.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.873+0000 7f0a35c38700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0a3015b070 0x7f0a3015b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:05.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.873+0000 7f0a35c38700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0a3015b070 0x7f0a3015b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43442/0 (socket says 172.21.15.16:43442) 2023-10-16T12:29:05.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.873+0000 7f0a35c38700 1 -- 172.21.15.16:0/2412097979 learned_addr learned my addr 172.21.15.16:0/2412097979 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:29:05.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.873+0000 7f0a35437700 1 --2- 172.21.15.16:0/2412097979 >> v2:172.21.15.49:3300/0 conn(0x7f0a300cb880 0x7f0a300cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:05.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.873+0000 7f0a36439700 1 --2- 172.21.15.16:0/2412097979 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0a3006da60 0x7f0a3006def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:05.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.873+0000 7f0a35c38700 1 -- 172.21.15.16:0/2412097979 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0a3006da60 msgr2=0x7f0a3006def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:29:05.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.873+0000 7f0a35c38700 1 --2- 172.21.15.16:0/2412097979 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0a3006da60 0x7f0a3006def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:05.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.873+0000 7f0a35c38700 1 -- 172.21.15.16:0/2412097979 >> v2:172.21.15.49:3300/0 conn(0x7f0a300cb880 msgr2=0x7f0a300cbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:29:05.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.873+0000 7f0a35c38700 1 --2- 172.21.15.16:0/2412097979 >> v2:172.21.15.49:3300/0 conn(0x7f0a300cb880 0x7f0a300cbd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:05.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.873+0000 7f0a35c38700 1 -- 172.21.15.16:0/2412097979 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a300ce640 con 0x7f0a3015b070 2023-10-16T12:29:05.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.873+0000 7f0a35437700 1 --2- 172.21.15.16:0/2412097979 >> v2:172.21.15.49:3300/0 conn(0x7f0a300cb880 0x7f0a300cbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:29:05.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.873+0000 7f0a35c38700 1 --2- 172.21.15.16:0/2412097979 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a3015b070 0x7f0a3015b450 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f0a30070500 tx=0x7f0a2c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af1e4dd3209838cf server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:05.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.873+0000 7f0a34c36700 1 -- 172.21.15.16:0/2412097979 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a2c00b870 con 0x7f0a3015b070 2023-10-16T12:29:05.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.873+0000 7f0a34c36700 1 -- 172.21.15.16:0/2412097979 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a2c00b9d0 con 0x7f0a3015b070 2023-10-16T12:29:05.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.873+0000 7f0a34c36700 1 -- 172.21.15.16:0/2412097979 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a2c00bd60 con 0x7f0a3015b070 2023-10-16T12:29:05.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.874+0000 7f0a37e9c700 1 -- 172.21.15.16:0/2412097979 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a3015b070 msgr2=0x7f0a3015b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:05.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.874+0000 7f0a37e9c700 1 --2- 172.21.15.16:0/2412097979 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a3015b070 0x7f0a3015b450 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f0a30070500 tx=0x7f0a2c009bd0 comp rx=0 tx=0).stop 2023-10-16T12:29:05.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.874+0000 7f0a37e9c700 1 -- 172.21.15.16:0/2412097979 shutdown_connections 2023-10-16T12:29:05.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.874+0000 7f0a37e9c700 1 --2- 172.21.15.16:0/2412097979 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0a3006da60 0x7f0a3006def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:05.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.874+0000 7f0a37e9c700 1 --2- 172.21.15.16:0/2412097979 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a3015b070 0x7f0a3015b450 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:05.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.874+0000 7f0a37e9c700 1 --2- 172.21.15.16:0/2412097979 >> v2:172.21.15.49:3300/0 conn(0x7f0a300cb880 0x7f0a300cbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:05.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.874+0000 7f0a37e9c700 1 -- 172.21.15.16:0/2412097979 >> 172.21.15.16:0/2412097979 conn(0x7f0a300b1f40 msgr2=0x7f0a300b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:05.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.874+0000 7f0a37e9c700 1 -- 172.21.15.16:0/2412097979 shutdown_connections 2023-10-16T12:29:05.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.874+0000 7f0a37e9c700 1 -- 172.21.15.16:0/2412097979 wait complete. 2023-10-16T12:29:05.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.875+0000 7f0a37e9c700 1 Processor -- start 2023-10-16T12:29:05.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.875+0000 7f0a37e9c700 1 -- start start 2023-10-16T12:29:05.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.876+0000 7f0a37e9c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0a3006da60 0x7f0a3014d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:05.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.876+0000 7f0a37e9c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0a300cb880 0x7f0a301fa480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:05.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.876+0000 7f0a37e9c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a3015b070 0x7f0a300b0da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:05.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.876+0000 7f0a37e9c700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a300c9690 con 0x7f0a3015b070 2023-10-16T12:29:05.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.876+0000 7f0a37e9c700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a301fad60 con 0x7f0a300cb880 2023-10-16T12:29:05.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.876+0000 7f0a37e9c700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a301fb040 con 0x7f0a3006da60 2023-10-16T12:29:05.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.877+0000 7f0a36439700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a3015b070 0x7f0a300b0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:05.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.877+0000 7f0a36439700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a3015b070 0x7f0a300b0da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43452/0 (socket says 172.21.15.16:43452) 2023-10-16T12:29:05.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.877+0000 7f0a36439700 1 -- 172.21.15.16:0/1672604060 learned_addr learned my addr 172.21.15.16:0/1672604060 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:29:05.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.877+0000 7f0a35437700 1 --2- 172.21.15.16:0/1672604060 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0a300cb880 0x7f0a301fa480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:05.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.877+0000 7f0a35c38700 1 --2- 172.21.15.16:0/1672604060 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0a3006da60 0x7f0a3014d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:05.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.877+0000 7f0a36439700 1 -- 172.21.15.16:0/1672604060 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0a3006da60 msgr2=0x7f0a3014d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:05.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.877+0000 7f0a36439700 1 --2- 172.21.15.16:0/1672604060 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0a3006da60 0x7f0a3014d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:05.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.877+0000 7f0a36439700 1 -- 172.21.15.16:0/1672604060 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0a300cb880 msgr2=0x7f0a301fa480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:05.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.877+0000 7f0a36439700 1 --2- 172.21.15.16:0/1672604060 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0a300cb880 0x7f0a301fa480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:05.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.877+0000 7f0a36439700 1 -- 172.21.15.16:0/1672604060 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a2c009550 con 0x7f0a3015b070 2023-10-16T12:29:05.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.877+0000 7f0a36439700 1 --2- 172.21.15.16:0/1672604060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a3015b070 0x7f0a300b0da0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f0a28007e60 tx=0x7f0a2800e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:05.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.877+0000 7f0a267fc700 1 -- 172.21.15.16:0/1672604060 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a2800ed30 con 0x7f0a3015b070 2023-10-16T12:29:05.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.877+0000 7f0a267fc700 1 -- 172.21.15.16:0/1672604060 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a28014420 con 0x7f0a3015b070 2023-10-16T12:29:05.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.877+0000 7f0a37e9c700 1 -- 172.21.15.16:0/1672604060 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a300c9320 con 0x7f0a3015b070 2023-10-16T12:29:05.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.877+0000 7f0a267fc700 1 -- 172.21.15.16:0/1672604060 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a280147d0 con 0x7f0a3015b070 2023-10-16T12:29:05.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.877+0000 7f0a37e9c700 1 -- 172.21.15.16:0/1672604060 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a301fa9c0 con 0x7f0a3015b070 2023-10-16T12:29:05.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.878+0000 7f0a37e9c700 1 -- 172.21.15.16:0/1672604060 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a300c9840 con 0x7f0a3015b070 2023-10-16T12:29:05.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.879+0000 7f0a267fc700 1 -- 172.21.15.16:0/1672604060 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 62) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0a28014930 con 0x7f0a3015b070 2023-10-16T12:29:05.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.879+0000 7f0a267fc700 1 --2- 172.21.15.16:0/1672604060 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f0a1406b020 0x7f0a1406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:05.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.879+0000 7f0a35c38700 1 --2- 172.21.15.16:0/1672604060 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f0a1406b020 0x7f0a1406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:05.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.879+0000 7f0a267fc700 1 -- 172.21.15.16:0/1672604060 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6065+0+0 (secure 0 0 0) 0x7f0a28089ea0 con 0x7f0a3015b070 2023-10-16T12:29:05.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.880+0000 7f0a35c38700 1 --2- 172.21.15.16:0/1672604060 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f0a1406b020 0x7f0a1406d4e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c014040 tx=0x7f0a2c008030 comp rx=0 tx=0).ready entity=mgr.7884 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:05.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:05.884+0000 7f0a267fc700 1 -- 172.21.15.16:0/1672604060 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0a300c9840 con 0x7f0a3015b070 2023-10-16T12:29:06.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.093+0000 7f0a37e9c700 1 -- 172.21.15.16:0/1672604060 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f0a3015ac20 con 0x7f0a3015b070 2023-10-16T12:29:06.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.095+0000 7f0a267fc700 1 -- 172.21.15.16:0/1672604060 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v38) v1 ==== 104+0+0 (secure 0 0 0) 0x7f0a28054d80 con 0x7f0a3015b070 2023-10-16T12:29:06.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.098+0000 7f0a37e9c700 1 -- 172.21.15.16:0/1672604060 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f0a1406b020 msgr2=0x7f0a1406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:06.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.098+0000 7f0a37e9c700 1 --2- 172.21.15.16:0/1672604060 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f0a1406b020 0x7f0a1406d4e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c014040 tx=0x7f0a2c008030 comp rx=0 tx=0).stop 2023-10-16T12:29:06.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.098+0000 7f0a37e9c700 1 -- 172.21.15.16:0/1672604060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a3015b070 msgr2=0x7f0a300b0da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:06.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.098+0000 7f0a37e9c700 1 --2- 172.21.15.16:0/1672604060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a3015b070 0x7f0a300b0da0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f0a28007e60 tx=0x7f0a2800e3c0 comp rx=0 tx=0).stop 2023-10-16T12:29:06.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.098+0000 7f0a37e9c700 1 -- 172.21.15.16:0/1672604060 shutdown_connections 2023-10-16T12:29:06.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.098+0000 7f0a37e9c700 1 --2- 172.21.15.16:0/1672604060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a3015b070 0x7f0a300b0da0 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:06.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.098+0000 7f0a37e9c700 1 --2- 172.21.15.16:0/1672604060 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f0a1406b020 0x7f0a1406d4e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:06.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.098+0000 7f0a37e9c700 1 --2- 172.21.15.16:0/1672604060 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0a3006da60 0x7f0a3014d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:06.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.098+0000 7f0a37e9c700 1 --2- 172.21.15.16:0/1672604060 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0a300cb880 0x7f0a301fa480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:06.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.098+0000 7f0a37e9c700 1 -- 172.21.15.16:0/1672604060 >> 172.21.15.16:0/1672604060 conn(0x7f0a300b1f40 msgr2=0x7f0a3015d460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:06.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.098+0000 7f0a37e9c700 1 -- 172.21.15.16:0/1672604060 shutdown_connections 2023-10-16T12:29:06.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.098+0000 7f0a37e9c700 1 -- 172.21.15.16:0/1672604060 wait complete. 2023-10-16T12:29:06.112 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-16T12:29:06.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.238+0000 7f1f261eb700 1 Processor -- start 2023-10-16T12:29:06.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.238+0000 7f1f261eb700 1 -- start start 2023-10-16T12:29:06.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.238+0000 7f1f261eb700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f1f2006e170 0x7f1f200cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:06.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.238+0000 7f1f261eb700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f200cef70 con 0x7f1f2015b070 2023-10-16T12:29:06.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.239+0000 7f1f261eb700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f200ce360 con 0x7f1f200cb880 2023-10-16T12:29:06.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.239+0000 7f1f261eb700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f200ce640 con 0x7f1f2006e170 2023-10-16T12:29:06.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.239+0000 7f1f1f7fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f1f2015b070 0x7f1f2015b450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:60962/0 (socket says 172.21.15.16:60962) 2023-10-16T12:29:06.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.239+0000 7f1f1f7fe700 1 -- 172.21.15.16:0/2370031325 learned_addr learned my addr 172.21.15.16:0/2370031325 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:29:06.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.239+0000 7f1f1ffff700 1 --2- 172.21.15.16:0/2370031325 >> v2:172.21.15.49:3300/0 conn(0x7f1f2006e170 0x7f1f200cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:06.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.239+0000 7f1f1e7fc700 1 -- 172.21.15.16:0/2370031325 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 18548117 0 0) 0x7f1f200cef70 con 0x7f1f2015b070 2023-10-16T12:29:06.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.239+0000 7f1f1e7fc700 1 -- 172.21.15.16:0/2370031325 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f080035b0 con 0x7f1f2015b070 2023-10-16T12:29:06.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.239+0000 7f1f1e7fc700 1 -- 172.21.15.16:0/2370031325 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2944937735 0 0) 0x7f1f080035b0 con 0x7f1f2015b070 2023-10-16T12:29:06.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.239+0000 7f1f1e7fc700 1 -- 172.21.15.16:0/2370031325 >> v2:172.21.15.49:3300/0 conn(0x7f1f2006e170 msgr2=0x7f1f200cbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:29:06.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.239+0000 7f1f1e7fc700 1 --2- 172.21.15.16:0/2370031325 >> v2:172.21.15.49:3300/0 conn(0x7f1f2006e170 0x7f1f200cbd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:06.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.239+0000 7f1f1e7fc700 1 -- 172.21.15.16:0/2370031325 >> v1:172.21.15.49:6789/0 conn(0x7f1f200cb880 legacy=0x7f1f2006da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:06.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.239+0000 7f1f1e7fc700 1 -- 172.21.15.16:0/2370031325 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f200ce920 con 0x7f1f2015b070 2023-10-16T12:29:06.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.239+0000 7f1f1e7fc700 1 -- 172.21.15.16:0/2370031325 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f1f10001670 con 0x7f1f2015b070 2023-10-16T12:29:06.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.239+0000 7f1f1ffff700 1 --2- 172.21.15.16:0/2370031325 >> v2:172.21.15.49:3300/0 conn(0x7f1f2006e170 0x7f1f200cbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:29:06.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.239+0000 7f1f1e7fc700 1 -- 172.21.15.16:0/2370031325 >> v1:172.21.15.16:6789/0 conn(0x7f1f2015b070 legacy=0x7f1f2015b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:06.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.239+0000 7f1f1e7fc700 1 --2- 172.21.15.16:0/2370031325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f08003a00 0x7f1f08003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:06.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.239+0000 7f1f1e7fc700 1 --2- 172.21.15.16:0/2370031325 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1f08004850 0x7f1f08004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:06.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.239+0000 7f1f1e7fc700 1 --2- 172.21.15.16:0/2370031325 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1f08005210 0x7f1f0800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:06.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.239+0000 7f1f1e7fc700 1 -- 172.21.15.16:0/2370031325 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f200ce640 con 0x7f1f08003a00 2023-10-16T12:29:06.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.239+0000 7f1f1e7fc700 1 -- 172.21.15.16:0/2370031325 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f0800c1b0 con 0x7f1f08004850 2023-10-16T12:29:06.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.239+0000 7f1f1e7fc700 1 -- 172.21.15.16:0/2370031325 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f0800e580 con 0x7f1f08005210 2023-10-16T12:29:06.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.239+0000 7f1f1f7fe700 1 --2- 172.21.15.16:0/2370031325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f08003a00 0x7f1f08003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:06.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.240+0000 7f1f1effd700 1 --2- 172.21.15.16:0/2370031325 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1f08004850 0x7f1f08004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:06.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.240+0000 7f1f1f7fe700 1 -- 172.21.15.16:0/2370031325 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1f08005210 msgr2=0x7f1f0800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:29:06.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.240+0000 7f1f1ffff700 1 --2- 172.21.15.16:0/2370031325 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1f08005210 0x7f1f0800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:06.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.240+0000 7f1f1f7fe700 1 --2- 172.21.15.16:0/2370031325 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1f08005210 0x7f1f0800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:06.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.240+0000 7f1f1f7fe700 1 -- 172.21.15.16:0/2370031325 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1f08004850 msgr2=0x7f1f08004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:29:06.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.240+0000 7f1f1f7fe700 1 --2- 172.21.15.16:0/2370031325 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1f08004850 0x7f1f08004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:06.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.240+0000 7f1f1f7fe700 1 -- 172.21.15.16:0/2370031325 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f080035b0 con 0x7f1f08003a00 2023-10-16T12:29:06.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.240+0000 7f1f1effd700 1 --2- 172.21.15.16:0/2370031325 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1f08004850 0x7f1f08004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:29:06.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.240+0000 7f1f1ffff700 1 --2- 172.21.15.16:0/2370031325 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1f08005210 0x7f1f0800ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:29:06.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.240+0000 7f1f1ffff700 1 -- 172.21.15.16:0/2370031325 reap_dead start 2023-10-16T12:29:06.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.240+0000 7f1f1f7fe700 1 --2- 172.21.15.16:0/2370031325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f08003a00 0x7f1f08003e20 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f1f1000c240 tx=0x7f1f08005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e904a7b5bd2e1c1e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:06.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.240+0000 7f1f1e7fc700 1 -- 172.21.15.16:0/2370031325 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f10007c10 con 0x7f1f08003a00 2023-10-16T12:29:06.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.240+0000 7f1f1e7fc700 1 -- 172.21.15.16:0/2370031325 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f1000ce00 con 0x7f1f08003a00 2023-10-16T12:29:06.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.240+0000 7f1f1e7fc700 1 -- 172.21.15.16:0/2370031325 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f10005ba0 con 0x7f1f08003a00 2023-10-16T12:29:06.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.241+0000 7f1f261eb700 1 -- 172.21.15.16:0/2370031325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f08003a00 msgr2=0x7f1f08003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:06.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.241+0000 7f1f261eb700 1 --2- 172.21.15.16:0/2370031325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f08003a00 0x7f1f08003e20 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f1f1000c240 tx=0x7f1f08005880 comp rx=0 tx=0).stop 2023-10-16T12:29:06.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.241+0000 7f1f261eb700 1 -- 172.21.15.16:0/2370031325 shutdown_connections 2023-10-16T12:29:06.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.241+0000 7f1f261eb700 1 --2- 172.21.15.16:0/2370031325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f08003a00 0x7f1f08003e20 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:06.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.241+0000 7f1f261eb700 1 -- 172.21.15.16:0/2370031325 >> 172.21.15.16:0/2370031325 conn(0x7f1f200b1f40 msgr2=0x7f1f200b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:06.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.241+0000 7f1f261eb700 1 -- 172.21.15.16:0/2370031325 shutdown_connections 2023-10-16T12:29:06.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.241+0000 7f1f261eb700 1 -- 172.21.15.16:0/2370031325 wait complete. 2023-10-16T12:29:06.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.242+0000 7f1f261eb700 1 Processor -- start 2023-10-16T12:29:06.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.242+0000 7f1f261eb700 1 -- start start 2023-10-16T12:29:06.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.243+0000 7f1f261eb700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1f08003a00 0x7f1f20152da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:06.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.243+0000 7f1f261eb700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f200b3030 0x7f1f20152460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:06.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.243+0000 7f1f261eb700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1f200b2320 0x7f1f200b27d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:06.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.243+0000 7f1f261eb700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f200b2d10 con 0x7f1f200b3030 2023-10-16T12:29:06.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.243+0000 7f1f261eb700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f200b4140 con 0x7f1f08003a00 2023-10-16T12:29:06.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.243+0000 7f1f261eb700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f201584c0 con 0x7f1f200b2320 2023-10-16T12:29:06.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.243+0000 7f1f1effd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f200b3030 0x7f1f20152460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:06.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.243+0000 7f1f1effd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f200b3030 0x7f1f20152460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43470/0 (socket says 172.21.15.16:43470) 2023-10-16T12:29:06.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.243+0000 7f1f1effd700 1 -- 172.21.15.16:0/867475095 learned_addr learned my addr 172.21.15.16:0/867475095 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:29:06.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.243+0000 7f1f1ffff700 1 --2- 172.21.15.16:0/867475095 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1f200b2320 0x7f1f200b27d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:06.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.243+0000 7f1f1f7fe700 1 --2- 172.21.15.16:0/867475095 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1f08003a00 0x7f1f20152da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:06.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.243+0000 7f1f1effd700 1 -- 172.21.15.16:0/867475095 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1f200b2320 msgr2=0x7f1f200b27d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:06.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.243+0000 7f1f1effd700 1 --2- 172.21.15.16:0/867475095 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1f200b2320 0x7f1f200b27d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:06.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.243+0000 7f1f1effd700 1 -- 172.21.15.16:0/867475095 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1f08003a00 msgr2=0x7f1f20152da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:06.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.243+0000 7f1f1effd700 1 --2- 172.21.15.16:0/867475095 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1f08003a00 0x7f1f20152da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:06.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.243+0000 7f1f1effd700 1 -- 172.21.15.16:0/867475095 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f200b34c0 con 0x7f1f200b3030 2023-10-16T12:29:06.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.243+0000 7f1f1ffff700 1 --2- 172.21.15.16:0/867475095 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1f200b2320 0x7f1f200b27d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:29:06.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.243+0000 7f1f1effd700 1 --2- 172.21.15.16:0/867475095 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f200b3030 0x7f1f20152460 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f1f1400e550 tx=0x7f1f1400e860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:06.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.243+0000 7f1f07fff700 1 -- 172.21.15.16:0/867475095 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f140052e0 con 0x7f1f200b3030 2023-10-16T12:29:06.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.244+0000 7f1f07fff700 1 -- 172.21.15.16:0/867475095 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f14005440 con 0x7f1f200b3030 2023-10-16T12:29:06.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.244+0000 7f1f261eb700 1 -- 172.21.15.16:0/867475095 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f200b37a0 con 0x7f1f200b3030 2023-10-16T12:29:06.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.244+0000 7f1f07fff700 1 -- 172.21.15.16:0/867475095 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f140038c0 con 0x7f1f200b3030 2023-10-16T12:29:06.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.244+0000 7f1f261eb700 1 -- 172.21.15.16:0/867475095 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f200b3d60 con 0x7f1f200b3030 2023-10-16T12:29:06.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.244+0000 7f1f261eb700 1 -- 172.21.15.16:0/867475095 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f200c8050 con 0x7f1f200b3030 2023-10-16T12:29:06.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.245+0000 7f1f07fff700 1 -- 172.21.15.16:0/867475095 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 62) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1f14003a20 con 0x7f1f200b3030 2023-10-16T12:29:06.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.245+0000 7f1f07fff700 1 --2- 172.21.15.16:0/867475095 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f1f0c06b020 0x7f1f0c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:06.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.245+0000 7f1f07fff700 1 -- 172.21.15.16:0/867475095 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6065+0+0 (secure 0 0 0) 0x7f1f14088e90 con 0x7f1f200b3030 2023-10-16T12:29:06.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.246+0000 7f1f1f7fe700 1 --2- 172.21.15.16:0/867475095 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f1f0c06b020 0x7f1f0c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:06.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.246+0000 7f1f1f7fe700 1 --2- 172.21.15.16:0/867475095 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f1f0c06b020 0x7f1f0c06d4e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f1f10000c00 tx=0x7f1f10007000 comp rx=0 tx=0).ready entity=mgr.7884 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:06.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.250+0000 7f1f07fff700 1 -- 172.21.15.16:0/867475095 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1f200c8050 con 0x7f1f200b3030 2023-10-16T12:29:06.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.452+0000 7f1f261eb700 1 -- 172.21.15.16:0/867475095 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f1f200c8050 con 0x7f1f200b3030 2023-10-16T12:29:06.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.456+0000 7f1f07fff700 1 -- 172.21.15.16:0/867475095 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v39) v1 ==== 120+0+0 (secure 0 0 0) 0x7f1f200c8050 con 0x7f1f200b3030 2023-10-16T12:29:06.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.461+0000 7f1f261eb700 1 -- 172.21.15.16:0/867475095 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f1f0c06b020 msgr2=0x7f1f0c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:06.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.461+0000 7f1f261eb700 1 --2- 172.21.15.16:0/867475095 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f1f0c06b020 0x7f1f0c06d4e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f1f10000c00 tx=0x7f1f10007000 comp rx=0 tx=0).stop 2023-10-16T12:29:06.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.461+0000 7f1f261eb700 1 -- 172.21.15.16:0/867475095 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f200b3030 msgr2=0x7f1f20152460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:06.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.461+0000 7f1f261eb700 1 --2- 172.21.15.16:0/867475095 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f200b3030 0x7f1f20152460 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f1f1400e550 tx=0x7f1f1400e860 comp rx=0 tx=0).stop 2023-10-16T12:29:06.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.462+0000 7f1f261eb700 1 -- 172.21.15.16:0/867475095 shutdown_connections 2023-10-16T12:29:06.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.462+0000 7f1f261eb700 1 --2- 172.21.15.16:0/867475095 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f1f0c06b020 0x7f1f0c06d4e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:06.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.462+0000 7f1f261eb700 1 --2- 172.21.15.16:0/867475095 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1f200b2320 0x7f1f200b27d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:06.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.462+0000 7f1f261eb700 1 --2- 172.21.15.16:0/867475095 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1f08003a00 0x7f1f20152da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:06.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.462+0000 7f1f261eb700 1 --2- 172.21.15.16:0/867475095 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f200b3030 0x7f1f20152460 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:06.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.462+0000 7f1f261eb700 1 -- 172.21.15.16:0/867475095 >> 172.21.15.16:0/867475095 conn(0x7f1f200b1f40 msgr2=0x7f1f200cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:06.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.464+0000 7f1f261eb700 1 -- 172.21.15.16:0/867475095 shutdown_connections 2023-10-16T12:29:06.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.466+0000 7f1f261eb700 1 -- 172.21.15.16:0/867475095 wait complete. 2023-10-16T12:29:06.485 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path' 2023-10-16T12:29:06.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.610+0000 7f76aaf9b700 1 Processor -- start 2023-10-16T12:29:06.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.610+0000 7f76aaf9b700 1 -- start start 2023-10-16T12:29:06.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.611+0000 7f76aaf9b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f76a415a5a0 0x7f76a4071c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:06.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.611+0000 7f76aaf9b700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f76a415a980 0x7f76a406f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:06.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.611+0000 7f76aaf9b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f76a40725d0 0x7f76a406f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:06.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.611+0000 7f76aaf9b700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76a40cf110 con 0x7f76a415a5a0 2023-10-16T12:29:06.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.611+0000 7f76aaf9b700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76a40cfc90 con 0x7f76a415a980 2023-10-16T12:29:06.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.611+0000 7f76aaf9b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76a40cf360 con 0x7f76a40725d0 2023-10-16T12:29:06.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.611+0000 7f76a8d37700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f76a415a5a0 0x7f76a4071c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:06.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.611+0000 7f76a8d37700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f76a415a5a0 0x7f76a4071c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43476/0 (socket says 172.21.15.16:43476) 2023-10-16T12:29:06.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.611+0000 7f76a8d37700 1 -- 172.21.15.16:0/2929390647 learned_addr learned my addr 172.21.15.16:0/2929390647 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:29:06.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.611+0000 7f76a8d37700 1 -- 172.21.15.16:0/2929390647 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f76a40725d0 msgr2=0x7f76a406f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:29:06.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.611+0000 7f76a8d37700 1 --2- 172.21.15.16:0/2929390647 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f76a40725d0 0x7f76a406f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:06.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.611+0000 7f76a8d37700 1 -- 172.21.15.16:0/2929390647 >> v2:172.21.15.49:3300/0 conn(0x7f76a415a980 msgr2=0x7f76a406f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:29:06.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.611+0000 7f76a8d37700 1 --2- 172.21.15.16:0/2929390647 >> v2:172.21.15.49:3300/0 conn(0x7f76a415a980 0x7f76a406f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:06.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.611+0000 7f76a8d37700 1 -- 172.21.15.16:0/2929390647 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76a40cf640 con 0x7f76a415a5a0 2023-10-16T12:29:06.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.612+0000 7f76a8d37700 1 --2- 172.21.15.16:0/2929390647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76a415a5a0 0x7f76a4071c80 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f76a40b3100 tx=0x7f7698009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1128683f2306b276 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:06.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.612+0000 7f76a37fe700 1 -- 172.21.15.16:0/2929390647 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f769800b870 con 0x7f76a415a5a0 2023-10-16T12:29:06.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.612+0000 7f76a37fe700 1 -- 172.21.15.16:0/2929390647 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f769800b9d0 con 0x7f76a415a5a0 2023-10-16T12:29:06.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.612+0000 7f76a37fe700 1 -- 172.21.15.16:0/2929390647 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f769800bd60 con 0x7f76a415a5a0 2023-10-16T12:29:06.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.612+0000 7f76aaf9b700 1 -- 172.21.15.16:0/2929390647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76a415a5a0 msgr2=0x7f76a4071c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:06.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.612+0000 7f76aaf9b700 1 --2- 172.21.15.16:0/2929390647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76a415a5a0 0x7f76a4071c80 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f76a40b3100 tx=0x7f7698009bd0 comp rx=0 tx=0).stop 2023-10-16T12:29:06.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.612+0000 7f76aaf9b700 1 -- 172.21.15.16:0/2929390647 shutdown_connections 2023-10-16T12:29:06.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.612+0000 7f76aaf9b700 1 --2- 172.21.15.16:0/2929390647 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f76a40725d0 0x7f76a406f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:06.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.612+0000 7f76aaf9b700 1 --2- 172.21.15.16:0/2929390647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76a415a5a0 0x7f76a4071c80 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:06.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.612+0000 7f76aaf9b700 1 --2- 172.21.15.16:0/2929390647 >> v2:172.21.15.49:3300/0 conn(0x7f76a415a980 0x7f76a406f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:06.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.612+0000 7f76aaf9b700 1 -- 172.21.15.16:0/2929390647 >> 172.21.15.16:0/2929390647 conn(0x7f76a40b2040 msgr2=0x7f76a40b2420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:06.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.612+0000 7f76aaf9b700 1 -- 172.21.15.16:0/2929390647 shutdown_connections 2023-10-16T12:29:06.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.612+0000 7f76aaf9b700 1 -- 172.21.15.16:0/2929390647 wait complete. 2023-10-16T12:29:06.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.614+0000 7f76aaf9b700 1 Processor -- start 2023-10-16T12:29:06.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.614+0000 7f76aaf9b700 1 -- start start 2023-10-16T12:29:06.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.614+0000 7f76aaf9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76a40725d0 0x7f76a41591a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:06.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.615+0000 7f76aaf9b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f76a415a5a0 0x7f76a40c75a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:06.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.615+0000 7f76aaf9b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f76a415a980 0x7f76a40aebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:06.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.615+0000 7f76aaf9b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76a40b3130 con 0x7f76a40725d0 2023-10-16T12:29:06.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.615+0000 7f76aaf9b700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76a40c7ae0 con 0x7f76a415a980 2023-10-16T12:29:06.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.615+0000 7f76aaf9b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76a40b3380 con 0x7f76a415a5a0 2023-10-16T12:29:06.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.615+0000 7f76a8d37700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76a40725d0 0x7f76a41591a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:06.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.615+0000 7f76a8d37700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76a40725d0 0x7f76a41591a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43484/0 (socket says 172.21.15.16:43484) 2023-10-16T12:29:06.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.615+0000 7f76a8d37700 1 -- 172.21.15.16:0/2907828751 learned_addr learned my addr 172.21.15.16:0/2907828751 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:29:06.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.615+0000 7f76a8d37700 1 -- 172.21.15.16:0/2907828751 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f76a415a5a0 msgr2=0x7f76a40c75a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:06.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.615+0000 7f76a8d37700 1 --2- 172.21.15.16:0/2907828751 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f76a415a5a0 0x7f76a40c75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:06.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.615+0000 7f76a8d37700 1 -- 172.21.15.16:0/2907828751 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f76a415a980 msgr2=0x7f76a40aebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:06.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.615+0000 7f76a8d37700 1 --2- 172.21.15.16:0/2907828751 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f76a415a980 0x7f76a40aebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:06.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.615+0000 7f76a8d37700 1 -- 172.21.15.16:0/2907828751 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7698009550 con 0x7f76a40725d0 2023-10-16T12:29:06.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.615+0000 7f76a8d37700 1 --2- 172.21.15.16:0/2907828751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76a40725d0 0x7f76a41591a0 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f7698009fd0 tx=0x7f7698004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:06.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.615+0000 7f76a17fa700 1 -- 172.21.15.16:0/2907828751 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7698004310 con 0x7f76a40725d0 2023-10-16T12:29:06.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.615+0000 7f76a17fa700 1 -- 172.21.15.16:0/2907828751 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f769801f070 con 0x7f76a40725d0 2023-10-16T12:29:06.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.615+0000 7f76a17fa700 1 -- 172.21.15.16:0/2907828751 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76980036a0 con 0x7f76a40725d0 2023-10-16T12:29:06.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.615+0000 7f76aaf9b700 1 -- 172.21.15.16:0/2907828751 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76a40b3660 con 0x7f76a40725d0 2023-10-16T12:29:06.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.615+0000 7f76aaf9b700 1 -- 172.21.15.16:0/2907828751 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76a40b3c20 con 0x7f76a40725d0 2023-10-16T12:29:06.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.616+0000 7f76aaf9b700 1 -- 172.21.15.16:0/2907828751 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76a40d1620 con 0x7f76a40725d0 2023-10-16T12:29:06.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.617+0000 7f76a17fa700 1 -- 172.21.15.16:0/2907828751 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 62) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7698021030 con 0x7f76a40725d0 2023-10-16T12:29:06.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.617+0000 7f76a17fa700 1 --2- 172.21.15.16:0/2907828751 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f768806b020 0x7f768806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:06.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.617+0000 7f76a17fa700 1 -- 172.21.15.16:0/2907828751 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6065+0+0 (secure 0 0 0) 0x7f769808ad80 con 0x7f76a40725d0 2023-10-16T12:29:06.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.617+0000 7f76a3fff700 1 --2- 172.21.15.16:0/2907828751 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f768806b020 0x7f768806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:06.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.618+0000 7f76a3fff700 1 --2- 172.21.15.16:0/2907828751 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f768806b020 0x7f768806d4e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f76a40c6620 tx=0x7f7690006c80 comp rx=0 tx=0).ready entity=mgr.7884 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:06.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.622+0000 7f76a17fa700 1 -- 172.21.15.16:0/2907828751 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7698055d60 con 0x7f76a40725d0 2023-10-16T12:29:06.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:06.915+0000 7f76aaf9b700 1 -- 172.21.15.16:0/2907828751 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]} v 0) v1 -- 0x7f76a404ebb0 con 0x7f76a40725d0 2023-10-16T12:29:07.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.340+0000 7f76a17fa700 1 -- 172.21.15.16:0/2907828751 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]}]=0 v697) v1 ==== 183+0+0 (secure 0 0 0) 0x7f7698059380 con 0x7f76a40725d0 2023-10-16T12:29:07.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.344+0000 7f76aaf9b700 1 -- 172.21.15.16:0/2907828751 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f768806b020 msgr2=0x7f768806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:07.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.344+0000 7f76aaf9b700 1 --2- 172.21.15.16:0/2907828751 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f768806b020 0x7f768806d4e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f76a40c6620 tx=0x7f7690006c80 comp rx=0 tx=0).stop 2023-10-16T12:29:07.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.344+0000 7f76aaf9b700 1 -- 172.21.15.16:0/2907828751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76a40725d0 msgr2=0x7f76a41591a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:07.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.344+0000 7f76aaf9b700 1 --2- 172.21.15.16:0/2907828751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76a40725d0 0x7f76a41591a0 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f7698009fd0 tx=0x7f7698004750 comp rx=0 tx=0).stop 2023-10-16T12:29:07.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.344+0000 7f76aaf9b700 1 -- 172.21.15.16:0/2907828751 shutdown_connections 2023-10-16T12:29:07.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.344+0000 7f76aaf9b700 1 --2- 172.21.15.16:0/2907828751 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f76a415a980 0x7f76a40aebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:07.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.344+0000 7f76aaf9b700 1 --2- 172.21.15.16:0/2907828751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76a40725d0 0x7f76a41591a0 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:07.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.344+0000 7f76aaf9b700 1 --2- 172.21.15.16:0/2907828751 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f768806b020 0x7f768806d4e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:07.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.344+0000 7f76aaf9b700 1 --2- 172.21.15.16:0/2907828751 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f76a415a5a0 0x7f76a40c75a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:07.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.344+0000 7f76aaf9b700 1 -- 172.21.15.16:0/2907828751 >> 172.21.15.16:0/2907828751 conn(0x7f76a40b2040 msgr2=0x7f76a40ccfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:07.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.344+0000 7f76aaf9b700 1 -- 172.21.15.16:0/2907828751 shutdown_connections 2023-10-16T12:29:07.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.344+0000 7f76aaf9b700 1 -- 172.21.15.16:0/2907828751 wait complete. 2023-10-16T12:29:07.361 INFO:tasks.cephfs_test_runner:Test that a non-existent path leads to schedule deactivation after a few retries. ... ok 2023-10-16T12:29:07.361 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test existence of a scheduled snapshot 2023-10-16T12:29:07.361 INFO:tasks.cephfs_test_runner:test_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-16T12:29:07.362 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:29:07.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.492+0000 7fd294b40700 1 Processor -- start 2023-10-16T12:29:07.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.492+0000 7fd294b40700 1 -- start start 2023-10-16T12:29:07.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.493+0000 7fd294b40700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fd29015b280 0x7fd2901613e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:07.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.493+0000 7fd294b40700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd290161c30 con 0x7fd29006a7e0 2023-10-16T12:29:07.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.493+0000 7fd294b40700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd290161f10 con 0x7fd2900cc8a0 2023-10-16T12:29:07.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.493+0000 7fd294b40700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2901621f0 con 0x7fd29015b280 2023-10-16T12:29:07.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.493+0000 7fd28ed9d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd29006a7e0 0x7fd2900d02b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:32768/0 (socket says 172.21.15.16:32768) 2023-10-16T12:29:07.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.493+0000 7fd28ed9d700 1 -- 172.21.15.16:0/534011485 learned_addr learned my addr 172.21.15.16:0/534011485 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:29:07.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.494+0000 7fd28d59a700 1 -- 172.21.15.16:0/534011485 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1381013457 0 0) 0x7fd290161c30 con 0x7fd29006a7e0 2023-10-16T12:29:07.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.494+0000 7fd28e59c700 1 --2- 172.21.15.16:0/534011485 >> v2:172.21.15.49:3300/0 conn(0x7fd29015b280 0x7fd2901613e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:07.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.494+0000 7fd28d59a700 1 -- 172.21.15.16:0/534011485 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd27c0035b0 con 0x7fd29006a7e0 2023-10-16T12:29:07.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.494+0000 7fd28d59a700 1 -- 172.21.15.16:0/534011485 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 157791008 0 0) 0x7fd27c0035b0 con 0x7fd29006a7e0 2023-10-16T12:29:07.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.494+0000 7fd28d59a700 1 -- 172.21.15.16:0/534011485 >> v2:172.21.15.49:3300/0 conn(0x7fd29015b280 msgr2=0x7fd2901613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:29:07.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.494+0000 7fd28d59a700 1 --2- 172.21.15.16:0/534011485 >> v2:172.21.15.49:3300/0 conn(0x7fd29015b280 0x7fd2901613e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:07.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.494+0000 7fd28d59a700 1 -- 172.21.15.16:0/534011485 >> v1:172.21.15.49:6789/0 conn(0x7fd2900cc8a0 legacy=0x7fd29015e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:07.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.494+0000 7fd28d59a700 1 -- 172.21.15.16:0/534011485 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd29015df00 con 0x7fd29006a7e0 2023-10-16T12:29:07.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.494+0000 7fd28d59a700 1 -- 172.21.15.16:0/534011485 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7fd2880019f0 con 0x7fd29006a7e0 2023-10-16T12:29:07.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.494+0000 7fd28d59a700 1 -- 172.21.15.16:0/534011485 >> v1:172.21.15.16:6789/0 conn(0x7fd29006a7e0 legacy=0x7fd2900d02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:07.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.494+0000 7fd28d59a700 1 --2- 172.21.15.16:0/534011485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd27c003a00 0x7fd27c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:07.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.494+0000 7fd28d59a700 1 --2- 172.21.15.16:0/534011485 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd27c004850 0x7fd27c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:07.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.494+0000 7fd28d59a700 1 --2- 172.21.15.16:0/534011485 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd27c005210 0x7fd27c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:07.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.494+0000 7fd28d59a700 1 -- 172.21.15.16:0/534011485 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2901621f0 con 0x7fd27c003a00 2023-10-16T12:29:07.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.494+0000 7fd28d59a700 1 -- 172.21.15.16:0/534011485 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd27c00c0b0 con 0x7fd27c005210 2023-10-16T12:29:07.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.494+0000 7fd28d59a700 1 -- 172.21.15.16:0/534011485 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd27c00c270 con 0x7fd27c004850 2023-10-16T12:29:07.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.494+0000 7fd28e59c700 1 --2- 172.21.15.16:0/534011485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd27c003a00 0x7fd27c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:07.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.494+0000 7fd28ed9d700 1 --2- 172.21.15.16:0/534011485 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd27c005210 0x7fd27c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:07.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.494+0000 7fd28dd9b700 1 --2- 172.21.15.16:0/534011485 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd27c004850 0x7fd27c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:07.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.495+0000 7fd28e59c700 1 -- 172.21.15.16:0/534011485 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd27c004850 msgr2=0x7fd27c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:29:07.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.495+0000 7fd28e59c700 1 --2- 172.21.15.16:0/534011485 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd27c004850 0x7fd27c004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:07.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.495+0000 7fd28e59c700 1 -- 172.21.15.16:0/534011485 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd27c005210 msgr2=0x7fd27c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:29:07.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.495+0000 7fd28e59c700 1 --2- 172.21.15.16:0/534011485 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd27c005210 0x7fd27c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:07.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.495+0000 7fd28e59c700 1 -- 172.21.15.16:0/534011485 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd27c0035b0 con 0x7fd27c003a00 2023-10-16T12:29:07.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.495+0000 7fd28ed9d700 1 -- 172.21.15.16:0/534011485 reap_dead start 2023-10-16T12:29:07.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.495+0000 7fd28dd9b700 1 --2- 172.21.15.16:0/534011485 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd27c004850 0x7fd27c004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:29:07.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.495+0000 7fd28e59c700 1 --2- 172.21.15.16:0/534011485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd27c003a00 0x7fd27c003e20 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7fd280009b60 tx=0x7fd27c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9030a56412b45a5a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:07.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.495+0000 7fd28d59a700 1 -- 172.21.15.16:0/534011485 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd28000e6c0 con 0x7fd27c003a00 2023-10-16T12:29:07.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.495+0000 7fd28d59a700 1 -- 172.21.15.16:0/534011485 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd28000e820 con 0x7fd27c003a00 2023-10-16T12:29:07.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.495+0000 7fd28d59a700 1 -- 172.21.15.16:0/534011485 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd28000eb90 con 0x7fd27c003a00 2023-10-16T12:29:07.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.495+0000 7fd294b40700 1 -- 172.21.15.16:0/534011485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd27c003a00 msgr2=0x7fd27c003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:07.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.495+0000 7fd294b40700 1 --2- 172.21.15.16:0/534011485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd27c003a00 0x7fd27c003e20 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7fd280009b60 tx=0x7fd27c005880 comp rx=0 tx=0).stop 2023-10-16T12:29:07.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.495+0000 7fd294b40700 1 -- 172.21.15.16:0/534011485 shutdown_connections 2023-10-16T12:29:07.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.495+0000 7fd294b40700 1 --2- 172.21.15.16:0/534011485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd27c003a00 0x7fd27c003e20 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:07.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.495+0000 7fd294b40700 1 -- 172.21.15.16:0/534011485 >> 172.21.15.16:0/534011485 conn(0x7fd2900b1fd0 msgr2=0x7fd2900b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:07.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.495+0000 7fd294b40700 1 -- 172.21.15.16:0/534011485 shutdown_connections 2023-10-16T12:29:07.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.496+0000 7fd294b40700 1 -- 172.21.15.16:0/534011485 wait complete. 2023-10-16T12:29:07.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.497+0000 7fd294b40700 1 Processor -- start 2023-10-16T12:29:07.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.497+0000 7fd294b40700 1 -- start start 2023-10-16T12:29:07.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.498+0000 7fd294b40700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd27c003a00 0x7fd2900af500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:07.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.498+0000 7fd294b40700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd290159350 0x7fd2901597d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:07.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.498+0000 7fd294b40700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2900c42a0 0x7fd2900c4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:07.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.498+0000 7fd294b40700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd290159d10 con 0x7fd2900c42a0 2023-10-16T12:29:07.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.498+0000 7fd294b40700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2900c5c30 con 0x7fd27c003a00 2023-10-16T12:29:07.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.498+0000 7fd294b40700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2900c5020 con 0x7fd290159350 2023-10-16T12:29:07.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.498+0000 7fd28ed9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2900c42a0 0x7fd2900c4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:07.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.498+0000 7fd28ed9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2900c42a0 0x7fd2900c4720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43506/0 (socket says 172.21.15.16:43506) 2023-10-16T12:29:07.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.498+0000 7fd28ed9d700 1 -- 172.21.15.16:0/1338354621 learned_addr learned my addr 172.21.15.16:0/1338354621 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:29:07.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.498+0000 7fd28e59c700 1 --2- 172.21.15.16:0/1338354621 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd27c003a00 0x7fd2900af500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:07.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.498+0000 7fd28dd9b700 1 --2- 172.21.15.16:0/1338354621 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd290159350 0x7fd2901597d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:07.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.498+0000 7fd28ed9d700 1 -- 172.21.15.16:0/1338354621 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd290159350 msgr2=0x7fd2901597d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:07.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.498+0000 7fd28ed9d700 1 --2- 172.21.15.16:0/1338354621 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd290159350 0x7fd2901597d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:07.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.498+0000 7fd28ed9d700 1 -- 172.21.15.16:0/1338354621 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd27c003a00 msgr2=0x7fd2900af500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:07.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.498+0000 7fd28ed9d700 1 --2- 172.21.15.16:0/1338354621 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd27c003a00 0x7fd2900af500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:07.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.498+0000 7fd28ed9d700 1 -- 172.21.15.16:0/1338354621 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2900c5300 con 0x7fd2900c42a0 2023-10-16T12:29:07.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.498+0000 7fd28ed9d700 1 --2- 172.21.15.16:0/1338354621 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2900c42a0 0x7fd2900c4720 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7fd2900ce130 tx=0x7fd28800a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:07.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.498+0000 7fd286ffd700 1 -- 172.21.15.16:0/1338354621 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd28800ac50 con 0x7fd2900c42a0 2023-10-16T12:29:07.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.498+0000 7fd286ffd700 1 -- 172.21.15.16:0/1338354621 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd28800adb0 con 0x7fd2900c42a0 2023-10-16T12:29:07.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.498+0000 7fd294b40700 1 -- 172.21.15.16:0/1338354621 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2900c55e0 con 0x7fd2900c42a0 2023-10-16T12:29:07.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.499+0000 7fd286ffd700 1 -- 172.21.15.16:0/1338354621 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2880135f0 con 0x7fd2900c42a0 2023-10-16T12:29:07.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.499+0000 7fd294b40700 1 -- 172.21.15.16:0/1338354621 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2900afa40 con 0x7fd2900c42a0 2023-10-16T12:29:07.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.499+0000 7fd294b40700 1 -- 172.21.15.16:0/1338354621 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd29004ebb0 con 0x7fd2900c42a0 2023-10-16T12:29:07.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.500+0000 7fd286ffd700 1 -- 172.21.15.16:0/1338354621 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 62) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd288013750 con 0x7fd2900c42a0 2023-10-16T12:29:07.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.500+0000 7fd286ffd700 1 --2- 172.21.15.16:0/1338354621 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fd27006b020 0x7fd27006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:07.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.501+0000 7fd286ffd700 1 -- 172.21.15.16:0/1338354621 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6065+0+0 (secure 0 0 0) 0x7fd28808a0e0 con 0x7fd2900c42a0 2023-10-16T12:29:07.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.501+0000 7fd28e59c700 1 --2- 172.21.15.16:0/1338354621 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fd27006b020 0x7fd27006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:07.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.501+0000 7fd28e59c700 1 --2- 172.21.15.16:0/1338354621 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fd27006b020 0x7fd27006d4e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd2800090b0 tx=0x7fd28001e040 comp rx=0 tx=0).ready entity=mgr.7884 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:07.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.505+0000 7fd286ffd700 1 -- 172.21.15.16:0/1338354621 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd288054f10 con 0x7fd2900c42a0 2023-10-16T12:29:07.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.711+0000 7fd294b40700 1 -- 172.21.15.16:0/1338354621 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd2900cb4f0 con 0x7fd2900c42a0 2023-10-16T12:29:07.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.712+0000 7fd286ffd700 1 -- 172.21.15.16:0/1338354621 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v119) v1 ==== 74+0+14258 (secure 0 0 0) 0x7fd288058530 con 0x7fd2900c42a0 2023-10-16T12:29:07.714 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:29:07.715 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":119,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:29:05.334268+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.49:6835/729289774":"2023-10-17T12:29:01.977400+0000","172.21.15.16:0/1965208057":"2023-10-17T12:29:00.894705+0000","172.21.15.16:0/709605099":"2023-10-17T12:29:00.894705+0000","172.21.15.16:6832/1458543709":"2023-10-17T12:29:00.894705+0000","172.21.15.16:0/4068400710":"2023-10-17T12:29:00.894705+0000","172.21.15.16:0/531582398":"2023-10-17T12:29:00.894705+0000","172.21.15.16:0/650861850":"2023-10-17T12:29:00.894705+0000","172.21.15.16:6833/4089191642":"2023-10-17T12:27:51.745827+0000","172.21.15.16:0/2709574245":"2023-10-17T12:27:51.745827+0000","172.21.15.16:0/2810361636":"2023-10-17T12:29:00.894705+0000","172.21.15.16:0/769212771":"2023-10-17T12:27:51.745827+0000","172.21.15.49:6833/729289774":"2023-10-17T12:29:01.977400+0000","172.21.15.16:0/1257374801":"2023-10-17T12:29:00.894705+0000","172.21.15.16:6832/4089191642":"2023-10-17T12:27:51.745827+0000","172.21.15.16:0/3579843680":"2023-10-17T12:27:51.745827+0000","172.21.15.16:0/1190603852":"2023-10-17T12:27:51.745827+0000","172.21.15.16:0/3144677187":"2023-10-17T12:27:51.745827+0000","172.21.15.16:0/2624839014":"2023-10-17T12:29:00.894705+0000","172.21.15.16:6833/1458543709":"2023-10-17T12:29:00.894705+0000","172.21.15.16:0/3478352903":"2023-10-17T12:27:51.745827+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:29:07.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.714+0000 7fd294b40700 1 -- 172.21.15.16:0/1338354621 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fd27006b020 msgr2=0x7fd27006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:07.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.714+0000 7fd294b40700 1 --2- 172.21.15.16:0/1338354621 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fd27006b020 0x7fd27006d4e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd2800090b0 tx=0x7fd28001e040 comp rx=0 tx=0).stop 2023-10-16T12:29:07.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.714+0000 7fd294b40700 1 -- 172.21.15.16:0/1338354621 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2900c42a0 msgr2=0x7fd2900c4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:07.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.714+0000 7fd294b40700 1 --2- 172.21.15.16:0/1338354621 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2900c42a0 0x7fd2900c4720 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7fd2900ce130 tx=0x7fd28800a5f0 comp rx=0 tx=0).stop 2023-10-16T12:29:07.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.714+0000 7fd294b40700 1 -- 172.21.15.16:0/1338354621 shutdown_connections 2023-10-16T12:29:07.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.714+0000 7fd294b40700 1 --2- 172.21.15.16:0/1338354621 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2900c42a0 0x7fd2900c4720 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:07.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.714+0000 7fd294b40700 1 --2- 172.21.15.16:0/1338354621 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd27c003a00 0x7fd2900af500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:07.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.714+0000 7fd294b40700 1 --2- 172.21.15.16:0/1338354621 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fd27006b020 0x7fd27006d4e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:07.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.714+0000 7fd294b40700 1 --2- 172.21.15.16:0/1338354621 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd290159350 0x7fd2901597d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:07.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.714+0000 7fd294b40700 1 -- 172.21.15.16:0/1338354621 >> 172.21.15.16:0/1338354621 conn(0x7fd2900b1fd0 msgr2=0x7fd2900d0270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:07.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.714+0000 7fd294b40700 1 -- 172.21.15.16:0/1338354621 shutdown_connections 2023-10-16T12:29:07.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.715+0000 7fd294b40700 1 -- 172.21.15.16:0/1338354621 wait complete. 2023-10-16T12:29:07.729 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T12:29:07.729 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:29:07.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.855+0000 7f39e5ab4700 1 Processor -- start 2023-10-16T12:29:07.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.855+0000 7f39e5ab4700 1 -- start start 2023-10-16T12:29:07.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.856+0000 7f39e5ab4700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f39e015b2c0 0x7f39e0161340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:07.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.856+0000 7f39e5ab4700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f39e006a7e0 0x7f39e006ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:07.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.856+0000 7f39e5ab4700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39e015ecb0 con 0x7f39e006e570 2023-10-16T12:29:07.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.856+0000 7f39e5ab4700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39e006b1b0 con 0x7f39e006a7e0 2023-10-16T12:29:07.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.856+0000 7f39e5ab4700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39e0161880 con 0x7f39e015b2c0 2023-10-16T12:29:07.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.856+0000 7f39df7fe700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f39e015b2c0 0x7f39e0161340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:07.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.856+0000 7f39dffff700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f39e006a7e0 0x7f39e006ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:07.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.857+0000 7f39deffd700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7f39e006e570 0x7f39e015e5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:39966/0 (socket says 172.21.15.16:39966) 2023-10-16T12:29:07.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.857+0000 7f39deffd700 1 -- 172.21.15.16:0/3608413542 learned_addr learned my addr 172.21.15.16:0/3608413542 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:29:07.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.857+0000 7f39de7fc700 1 -- 172.21.15.16:0/3608413542 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1650308667 0 0) 0x7f39e015ecb0 con 0x7f39e006e570 2023-10-16T12:29:07.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.857+0000 7f39de7fc700 1 -- 172.21.15.16:0/3608413542 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39cc0035b0 con 0x7f39e006e570 2023-10-16T12:29:07.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.857+0000 7f39df7fe700 1 -- 172.21.15.16:0/3608413542 >> v2:172.21.15.49:3300/0 conn(0x7f39e006a7e0 msgr2=0x7f39e006ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:29:07.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.857+0000 7f39df7fe700 1 --2- 172.21.15.16:0/3608413542 >> v2:172.21.15.49:3300/0 conn(0x7f39e006a7e0 0x7f39e006ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:07.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.857+0000 7f39df7fe700 1 -- 172.21.15.16:0/3608413542 >> v1:172.21.15.49:6789/0 conn(0x7f39e006e570 legacy=0x7f39e015e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:07.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.857+0000 7f39df7fe700 1 -- 172.21.15.16:0/3608413542 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39e015b6a0 con 0x7f39e015b2c0 2023-10-16T12:29:07.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.857+0000 7f39dffff700 1 --2- 172.21.15.16:0/3608413542 >> v2:172.21.15.49:3300/0 conn(0x7f39e006a7e0 0x7f39e006ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:29:07.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.857+0000 7f39df7fe700 1 --2- 172.21.15.16:0/3608413542 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f39e015b2c0 0x7f39e0161340 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f39d0009840 tx=0x7f39d0009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=27ff0e7c1424502b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:07.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.857+0000 7f39de7fc700 1 -- 172.21.15.16:0/3608413542 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39d0011830 con 0x7f39e015b2c0 2023-10-16T12:29:07.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.857+0000 7f39de7fc700 1 -- 172.21.15.16:0/3608413542 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39d0011990 con 0x7f39e015b2c0 2023-10-16T12:29:07.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.857+0000 7f39de7fc700 1 -- 172.21.15.16:0/3608413542 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39d0011d00 con 0x7f39e015b2c0 2023-10-16T12:29:07.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.858+0000 7f39e5ab4700 1 -- 172.21.15.16:0/3608413542 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f39e015b2c0 msgr2=0x7f39e0161340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:07.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.858+0000 7f39e5ab4700 1 --2- 172.21.15.16:0/3608413542 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f39e015b2c0 0x7f39e0161340 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f39d0009840 tx=0x7f39d0009b50 comp rx=0 tx=0).stop 2023-10-16T12:29:07.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.858+0000 7f39e5ab4700 1 -- 172.21.15.16:0/3608413542 shutdown_connections 2023-10-16T12:29:07.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.858+0000 7f39e5ab4700 1 --2- 172.21.15.16:0/3608413542 >> v2:172.21.15.49:3300/0 conn(0x7f39e006a7e0 0x7f39e006ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:07.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.858+0000 7f39e5ab4700 1 --2- 172.21.15.16:0/3608413542 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f39e015b2c0 0x7f39e0161340 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:07.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.858+0000 7f39e5ab4700 1 -- 172.21.15.16:0/3608413542 >> 172.21.15.16:0/3608413542 conn(0x7f39e00b1f40 msgr2=0x7f39e00b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:07.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.858+0000 7f39e5ab4700 1 -- 172.21.15.16:0/3608413542 shutdown_connections 2023-10-16T12:29:07.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.858+0000 7f39e5ab4700 1 -- 172.21.15.16:0/3608413542 wait complete. 2023-10-16T12:29:07.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.859+0000 7f39e5ab4700 1 Processor -- start 2023-10-16T12:29:07.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.860+0000 7f39e5ab4700 1 -- start start 2023-10-16T12:29:07.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.861+0000 7f39e5ab4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39e006a7e0 0x7f39e00b3b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:07.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.861+0000 7f39e5ab4700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f39e006e570 0x7f39e015eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:07.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.861+0000 7f39e5ab4700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f39e015b2c0 0x7f39e00c9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:07.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.861+0000 7f39e5ab4700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39e015f7b0 con 0x7f39e006a7e0 2023-10-16T12:29:07.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.861+0000 7f39e5ab4700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39e00c75f0 con 0x7f39e006e570 2023-10-16T12:29:07.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.861+0000 7f39e5ab4700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39e00c78d0 con 0x7f39e015b2c0 2023-10-16T12:29:07.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.861+0000 7f39df7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39e006a7e0 0x7f39e00b3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:07.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.861+0000 7f39df7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39e006a7e0 0x7f39e00b3b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43514/0 (socket says 172.21.15.16:43514) 2023-10-16T12:29:07.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.861+0000 7f39df7fe700 1 -- 172.21.15.16:0/1464049617 learned_addr learned my addr 172.21.15.16:0/1464049617 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:29:07.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.861+0000 7f39dffff700 1 --2- 172.21.15.16:0/1464049617 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f39e015b2c0 0x7f39e00c9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:07.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.861+0000 7f39deffd700 1 --2- 172.21.15.16:0/1464049617 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f39e006e570 0x7f39e015eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:07.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.861+0000 7f39df7fe700 1 -- 172.21.15.16:0/1464049617 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f39e015b2c0 msgr2=0x7f39e00c9ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:07.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.861+0000 7f39df7fe700 1 --2- 172.21.15.16:0/1464049617 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f39e015b2c0 0x7f39e00c9ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:07.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.861+0000 7f39df7fe700 1 -- 172.21.15.16:0/1464049617 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f39e006e570 msgr2=0x7f39e015eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:07.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.861+0000 7f39df7fe700 1 --2- 172.21.15.16:0/1464049617 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f39e006e570 0x7f39e015eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:07.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.861+0000 7f39df7fe700 1 -- 172.21.15.16:0/1464049617 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39d8009a70 con 0x7f39e006a7e0 2023-10-16T12:29:07.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.861+0000 7f39df7fe700 1 --2- 172.21.15.16:0/1464049617 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39e006a7e0 0x7f39e00b3b00 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f39d0009b90 tx=0x7f39d0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:07.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.861+0000 7f39d7fff700 1 -- 172.21.15.16:0/1464049617 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39d00044d0 con 0x7f39e006a7e0 2023-10-16T12:29:07.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.862+0000 7f39d7fff700 1 -- 172.21.15.16:0/1464049617 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39d00036a0 con 0x7f39e006a7e0 2023-10-16T12:29:07.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.862+0000 7f39e5ab4700 1 -- 172.21.15.16:0/1464049617 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39d00094a0 con 0x7f39e006a7e0 2023-10-16T12:29:07.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.862+0000 7f39d7fff700 1 -- 172.21.15.16:0/1464049617 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39d0003a30 con 0x7f39e006a7e0 2023-10-16T12:29:07.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.862+0000 7f39e5ab4700 1 -- 172.21.15.16:0/1464049617 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39e00c7e90 con 0x7f39e006a7e0 2023-10-16T12:29:07.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.863+0000 7f39e5ab4700 1 -- 172.21.15.16:0/1464049617 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39e004ebb0 con 0x7f39e006a7e0 2023-10-16T12:29:07.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.863+0000 7f39d7fff700 1 -- 172.21.15.16:0/1464049617 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 62) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f39d0003b90 con 0x7f39e006a7e0 2023-10-16T12:29:07.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.863+0000 7f39d7fff700 1 --2- 172.21.15.16:0/1464049617 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f39c006b020 0x7f39c006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:07.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.864+0000 7f39d7fff700 1 -- 172.21.15.16:0/1464049617 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6065+0+0 (secure 0 0 0) 0x7f39d008ab50 con 0x7f39e006a7e0 2023-10-16T12:29:07.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.864+0000 7f39deffd700 1 --2- 172.21.15.16:0/1464049617 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f39c006b020 0x7f39c006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:07.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.864+0000 7f39deffd700 1 --2- 172.21.15.16:0/1464049617 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f39c006b020 0x7f39c006d4e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f39e00c97d0 tx=0x7f39c8009040 comp rx=0 tx=0).ready entity=mgr.7884 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:07.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:07.868+0000 7f39d7fff700 1 -- 172.21.15.16:0/1464049617 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f39d0055b30 con 0x7f39e006a7e0 2023-10-16T12:29:08.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:08.070+0000 7f39e5ab4700 1 -- 172.21.15.16:0/1464049617 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f39e00b6370 con 0x7f39e006a7e0 2023-10-16T12:29:08.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:08.070+0000 7f39d7fff700 1 -- 172.21.15.16:0/1464049617 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v119) v1 ==== 93+0+10 (secure 0 0 0) 0x7f39d0015020 con 0x7f39e006a7e0 2023-10-16T12:29:08.073 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T12:29:08.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:08.072+0000 7f39e5ab4700 1 -- 172.21.15.16:0/1464049617 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f39c006b020 msgr2=0x7f39c006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:08.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:08.072+0000 7f39e5ab4700 1 --2- 172.21.15.16:0/1464049617 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f39c006b020 0x7f39c006d4e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f39e00c97d0 tx=0x7f39c8009040 comp rx=0 tx=0).stop 2023-10-16T12:29:08.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:08.072+0000 7f39e5ab4700 1 -- 172.21.15.16:0/1464049617 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39e006a7e0 msgr2=0x7f39e00b3b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:08.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:08.072+0000 7f39e5ab4700 1 --2- 172.21.15.16:0/1464049617 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39e006a7e0 0x7f39e00b3b00 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f39d0009b90 tx=0x7f39d0004750 comp rx=0 tx=0).stop 2023-10-16T12:29:08.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:08.073+0000 7f39e5ab4700 1 -- 172.21.15.16:0/1464049617 shutdown_connections 2023-10-16T12:29:08.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:08.073+0000 7f39e5ab4700 1 --2- 172.21.15.16:0/1464049617 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f39c006b020 0x7f39c006d4e0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:08.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:08.073+0000 7f39e5ab4700 1 --2- 172.21.15.16:0/1464049617 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f39e015b2c0 0x7f39e00c9ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:08.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:08.073+0000 7f39e5ab4700 1 --2- 172.21.15.16:0/1464049617 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39e006a7e0 0x7f39e00b3b00 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:08.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:08.073+0000 7f39e5ab4700 1 --2- 172.21.15.16:0/1464049617 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f39e006e570 0x7f39e015eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:08.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:08.073+0000 7f39e5ab4700 1 -- 172.21.15.16:0/1464049617 >> 172.21.15.16:0/1464049617 conn(0x7f39e00b1f40 msgr2=0x7f39e00cce00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:08.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:08.073+0000 7f39e5ab4700 1 -- 172.21.15.16:0/1464049617 shutdown_connections 2023-10-16T12:29:08.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:08.073+0000 7f39e5ab4700 1 -- 172.21.15.16:0/1464049617 wait complete. 2023-10-16T12:29:08.087 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule' 2023-10-16T12:29:08.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:08.210+0000 7fd19edef700 1 Processor -- start 2023-10-16T12:29:08.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:08.211+0000 7fd19edef700 1 -- start start 2023-10-16T12:29:08.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:08.211+0000 7fd19edef700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd19815b280 0x7fd1981613e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:08.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:08.211+0000 7fd19edef700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd19806a7e0 0x7fd19806ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:08.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:08.211+0000 7fd19edef700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd19806b170 con 0x7fd1980cc8a0 2023-10-16T12:29:08.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:08.211+0000 7fd19edef700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd198161c30 con 0x7fd19806a7e0 2023-10-16T12:29:08.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:08.211+0000 7fd19edef700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd198161f10 con 0x7fd19815b280 2023-10-16T12:29:08.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:08.212+0000 7fd19d38c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd19806a7e0 0x7fd19806ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:08.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:08.212+0000 7fd19d38c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd19806a7e0 0x7fd19806ac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43526/0 (socket says 172.21.15.16:43526) 2023-10-16T12:29:08.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:08.212+0000 7fd19d38c700 1 -- 172.21.15.16:0/4251493462 learned_addr learned my addr 172.21.15.16:0/4251493462 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:29:08.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:08.212+0000 7fd19cb8b700 1 --2- 172.21.15.16:0/4251493462 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd19815b280 0x7fd1981613e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:08.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:08.212+0000 7fd19d38c700 1 -- 172.21.15.16:0/4251493462 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd19815b280 msgr2=0x7fd1981613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:29:08.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:08.212+0000 7fd19d38c700 1 --2- 172.21.15.16:0/4251493462 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd19815b280 0x7fd1981613e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:08.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:08.212+0000 7fd19d38c700 1 -- 172.21.15.16:0/4251493462 >> v1:172.21.15.16:6789/0 conn(0x7fd1980cc8a0 legacy=0x7fd19815e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:08.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:08.212+0000 7fd19d38c700 1 -- 172.21.15.16:0/4251493462 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1981621f0 con 0x7fd19806a7e0 2023-10-16T12:29:08.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:08.212+0000 7fd19d38c700 1 --2- 172.21.15.16:0/4251493462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd19806a7e0 0x7fd19806ac30 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7fd190009e50 tx=0x7fd19000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21e94331aad066e4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:08.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:08.212+0000 7fd1977fe700 1 -- 172.21.15.16:0/4251493462 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd19000e070 con 0x7fd19806a7e0 2023-10-16T12:29:08.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:08.212+0000 7fd1977fe700 1 -- 172.21.15.16:0/4251493462 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd19000b9c0 con 0x7fd19806a7e0 2023-10-16T12:29:08.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:08.212+0000 7fd1977fe700 1 -- 172.21.15.16:0/4251493462 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd19000bd70 con 0x7fd19806a7e0 2023-10-16T12:29:08.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:08.213+0000 7fd19edef700 1 -- 172.21.15.16:0/4251493462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd19806a7e0 msgr2=0x7fd19806ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:08.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:08.213+0000 7fd19edef700 1 --2- 172.21.15.16:0/4251493462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd19806a7e0 0x7fd19806ac30 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7fd190009e50 tx=0x7fd19000b330 comp rx=0 tx=0).stop 2023-10-16T12:29:08.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:08.213+0000 7fd19edef700 1 -- 172.21.15.16:0/4251493462 shutdown_connections 2023-10-16T12:29:08.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:08.213+0000 7fd19edef700 1 --2- 172.21.15.16:0/4251493462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd19806a7e0 0x7fd19806ac30 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:08.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:08.213+0000 7fd19edef700 1 --2- 172.21.15.16:0/4251493462 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd19815b280 0x7fd1981613e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:08.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:08.213+0000 7fd19edef700 1 -- 172.21.15.16:0/4251493462 >> 172.21.15.16:0/4251493462 conn(0x7fd1980b1fd0 msgr2=0x7fd1980b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:08.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:08.213+0000 7fd19edef700 1 -- 172.21.15.16:0/4251493462 shutdown_connections 2023-10-16T12:29:08.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:08.213+0000 7fd19edef700 1 -- 172.21.15.16:0/4251493462 wait complete. 2023-10-16T12:29:08.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:08.214+0000 7fd19edef700 1 Processor -- start 2023-10-16T12:29:08.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:08.214+0000 7fd19edef700 1 -- start start 2023-10-16T12:29:08.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:08.216+0000 7fd19edef700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd19806a7e0 0x7fd1981f7140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:08.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:08.216+0000 7fd19edef700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd1980cc8a0 0x7fd1980c5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:08.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:08.216+0000 7fd19edef700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd19815b280 0x7fd1980b1070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:08.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:08.216+0000 7fd19edef700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1981f3220 con 0x7fd19815b280 2023-10-16T12:29:08.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:08.216+0000 7fd19edef700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1980ca060 con 0x7fd19806a7e0 2023-10-16T12:29:08.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:08.216+0000 7fd19edef700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1981f7680 con 0x7fd1980cc8a0 2023-10-16T12:29:08.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:08.216+0000 7fd19d38c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd19815b280 0x7fd1980b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:08.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:08.216+0000 7fd19d38c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd19815b280 0x7fd1980b1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43532/0 (socket says 172.21.15.16:43532) 2023-10-16T12:29:08.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:08.216+0000 7fd19d38c700 1 -- 172.21.15.16:0/4027283466 learned_addr learned my addr 172.21.15.16:0/4027283466 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:29:08.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:08.216+0000 7fd19d38c700 1 -- 172.21.15.16:0/4027283466 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd1980cc8a0 msgr2=0x7fd1980c5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:08.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:08.216+0000 7fd19d38c700 1 --2- 172.21.15.16:0/4027283466 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd1980cc8a0 0x7fd1980c5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:08.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:08.216+0000 7fd19d38c700 1 -- 172.21.15.16:0/4027283466 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd19806a7e0 msgr2=0x7fd1981f7140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:08.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:08.216+0000 7fd19d38c700 1 --2- 172.21.15.16:0/4027283466 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd19806a7e0 0x7fd1981f7140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:08.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:08.216+0000 7fd19d38c700 1 -- 172.21.15.16:0/4027283466 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd190009b30 con 0x7fd19815b280 2023-10-16T12:29:08.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:08.216+0000 7fd19d38c700 1 --2- 172.21.15.16:0/4027283466 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd19815b280 0x7fd1980b1070 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7fd1980d1540 tx=0x7fd190004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:08.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:08.216+0000 7fd1957fa700 1 -- 172.21.15.16:0/4027283466 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd190004960 con 0x7fd19815b280 2023-10-16T12:29:08.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:08.216+0000 7fd1957fa700 1 -- 172.21.15.16:0/4027283466 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1900099b0 con 0x7fd19815b280 2023-10-16T12:29:08.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:08.216+0000 7fd19edef700 1 -- 172.21.15.16:0/4027283466 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd19814b010 con 0x7fd19815b280 2023-10-16T12:29:08.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:08.216+0000 7fd1957fa700 1 -- 172.21.15.16:0/4027283466 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd19001c680 con 0x7fd19815b280 2023-10-16T12:29:08.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:08.216+0000 7fd19edef700 1 -- 172.21.15.16:0/4027283466 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd19814b5d0 con 0x7fd19815b280 2023-10-16T12:29:08.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:08.217+0000 7fd19edef700 1 -- 172.21.15.16:0/4027283466 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd19804ebb0 con 0x7fd19815b280 2023-10-16T12:29:08.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:08.218+0000 7fd1957fa700 1 -- 172.21.15.16:0/4027283466 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 62) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd19001c7e0 con 0x7fd19815b280 2023-10-16T12:29:08.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:08.218+0000 7fd1957fa700 1 --2- 172.21.15.16:0/4027283466 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fd17c06b020 0x7fd17c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:08.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:08.218+0000 7fd1957fa700 1 -- 172.21.15.16:0/4027283466 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6065+0+0 (secure 0 0 0) 0x7fd19008b700 con 0x7fd19815b280 2023-10-16T12:29:08.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:08.218+0000 7fd19cb8b700 1 --2- 172.21.15.16:0/4027283466 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fd17c06b020 0x7fd17c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:08.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:08.219+0000 7fd19cb8b700 1 --2- 172.21.15.16:0/4027283466 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fd17c06b020 0x7fd17c06d4e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd18c0098d0 tx=0x7fd18c008040 comp rx=0 tx=0).ready entity=mgr.7884 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:08.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:08.223+0000 7fd1957fa700 1 -- 172.21.15.16:0/4027283466 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd1900564b0 con 0x7fd19815b280 2023-10-16T12:29:08.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:08.511+0000 7fd19edef700 1 -- 172.21.15.16:0/4027283466 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]} v 0) v1 -- 0x7fd1981596e0 con 0x7fd19815b280 2023-10-16T12:29:09.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.346+0000 7fd1957fa700 1 -- 172.21.15.16:0/4027283466 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]}]=0 v699) v1 ==== 151+0+0 (secure 0 0 0) 0x7fd190059ad0 con 0x7fd19815b280 2023-10-16T12:29:09.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.349+0000 7fd19edef700 1 -- 172.21.15.16:0/4027283466 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fd17c06b020 msgr2=0x7fd17c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:09.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.349+0000 7fd19edef700 1 --2- 172.21.15.16:0/4027283466 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fd17c06b020 0x7fd17c06d4e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd18c0098d0 tx=0x7fd18c008040 comp rx=0 tx=0).stop 2023-10-16T12:29:09.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.349+0000 7fd19edef700 1 -- 172.21.15.16:0/4027283466 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd19815b280 msgr2=0x7fd1980b1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:09.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.349+0000 7fd19edef700 1 --2- 172.21.15.16:0/4027283466 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd19815b280 0x7fd1980b1070 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7fd1980d1540 tx=0x7fd190004ad0 comp rx=0 tx=0).stop 2023-10-16T12:29:09.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.350+0000 7fd19edef700 1 -- 172.21.15.16:0/4027283466 shutdown_connections 2023-10-16T12:29:09.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.350+0000 7fd19edef700 1 --2- 172.21.15.16:0/4027283466 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd19815b280 0x7fd1980b1070 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:09.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.350+0000 7fd19edef700 1 --2- 172.21.15.16:0/4027283466 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd19806a7e0 0x7fd1981f7140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:09.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.350+0000 7fd19edef700 1 --2- 172.21.15.16:0/4027283466 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fd17c06b020 0x7fd17c06d4e0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:09.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.350+0000 7fd19edef700 1 --2- 172.21.15.16:0/4027283466 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd1980cc8a0 0x7fd1980c5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:09.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.350+0000 7fd19edef700 1 -- 172.21.15.16:0/4027283466 >> 172.21.15.16:0/4027283466 conn(0x7fd1980b1fd0 msgr2=0x7fd1980d0de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:09.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.350+0000 7fd19edef700 1 -- 172.21.15.16:0/4027283466 shutdown_connections 2023-10-16T12:29:09.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.350+0000 7fd19edef700 1 -- 172.21.15.16:0/4027283466 wait complete. 2023-10-16T12:29:09.367 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-16T12:29:09.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.501+0000 7f50ffc5b700 1 Processor -- start 2023-10-16T12:29:09.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.501+0000 7f50ffc5b700 1 -- start start 2023-10-16T12:29:09.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.502+0000 7f50ffc5b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f50f815da10 0x7f50f80d1520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:09.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.502+0000 7f50ffc5b700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f50f80d4060 0x7f50f80cf2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:09.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.502+0000 7f50ffc5b700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50f80d4540 con 0x7f50f80d1d20 2023-10-16T12:29:09.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.502+0000 7f50ffc5b700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50f80ce7e0 con 0x7f50f80d4060 2023-10-16T12:29:09.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.502+0000 7f50ffc5b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50f80ceac0 con 0x7f50f815da10 2023-10-16T12:29:09.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.503+0000 7f50fd9f7700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f50f815da10 0x7f50f80d1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:09.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.503+0000 7f50fd1f6700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f50f80d4060 0x7f50f80cf2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:09.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.503+0000 7f50fd9f7700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f50f815da10 0x7f50f80d1520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:60026/0 (socket says 172.21.15.16:60026) 2023-10-16T12:29:09.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.503+0000 7f50fd1f6700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f50f80d4060 0x7f50f80cf2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:42598/0 (socket says 172.21.15.16:42598) 2023-10-16T12:29:09.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.503+0000 7f50fd9f7700 1 -- 172.21.15.16:0/2205694744 learned_addr learned my addr 172.21.15.16:0/2205694744 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:29:09.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.503+0000 7f50fc9f5700 1 -- 172.21.15.16:0/2205694744 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3786450252 0 0) 0x7f50f80d4540 con 0x7f50f80d1d20 2023-10-16T12:29:09.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.503+0000 7f50fc9f5700 1 -- 172.21.15.16:0/2205694744 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50e40035b0 con 0x7f50f80d1d20 2023-10-16T12:29:09.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.503+0000 7f50fd1f6700 1 -- 172.21.15.16:0/2205694744 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f50f815da10 msgr2=0x7f50f80d1520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:29:09.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.503+0000 7f50fd1f6700 1 --2- 172.21.15.16:0/2205694744 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f50f815da10 0x7f50f80d1520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:09.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.503+0000 7f50fd1f6700 1 -- 172.21.15.16:0/2205694744 >> v1:172.21.15.49:6789/0 conn(0x7f50f80d1d20 legacy=0x7f50f80d2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:09.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.503+0000 7f50fd1f6700 1 -- 172.21.15.16:0/2205694744 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50f80ceda0 con 0x7f50f80d4060 2023-10-16T12:29:09.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.503+0000 7f50fd9f7700 1 --2- 172.21.15.16:0/2205694744 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f50f815da10 0x7f50f80d1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:29:09.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.503+0000 7f50fd1f6700 1 --2- 172.21.15.16:0/2205694744 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f50f80d4060 0x7f50f80cf2a0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f50f4009840 tx=0x7f50f4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=12beaa0cfc2223a3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:09.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.503+0000 7f50fc9f5700 1 -- 172.21.15.16:0/2205694744 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50f4011830 con 0x7f50f80d4060 2023-10-16T12:29:09.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.504+0000 7f50fc9f5700 1 -- 172.21.15.16:0/2205694744 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50f4011990 con 0x7f50f80d4060 2023-10-16T12:29:09.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.504+0000 7f50fc9f5700 1 -- 172.21.15.16:0/2205694744 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50f4011d20 con 0x7f50f80d4060 2023-10-16T12:29:09.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.504+0000 7f50ffc5b700 1 -- 172.21.15.16:0/2205694744 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f50f80d4060 msgr2=0x7f50f80cf2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:09.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.504+0000 7f50ffc5b700 1 --2- 172.21.15.16:0/2205694744 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f50f80d4060 0x7f50f80cf2a0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f50f4009840 tx=0x7f50f4009b50 comp rx=0 tx=0).stop 2023-10-16T12:29:09.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.504+0000 7f50ffc5b700 1 -- 172.21.15.16:0/2205694744 shutdown_connections 2023-10-16T12:29:09.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.504+0000 7f50ffc5b700 1 --2- 172.21.15.16:0/2205694744 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f50f815da10 0x7f50f80d1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:09.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.504+0000 7f50ffc5b700 1 --2- 172.21.15.16:0/2205694744 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f50f80d4060 0x7f50f80cf2a0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:09.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.504+0000 7f50ffc5b700 1 -- 172.21.15.16:0/2205694744 >> 172.21.15.16:0/2205694744 conn(0x7f50f809c9f0 msgr2=0x7f50f80d3740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:09.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.504+0000 7f50ffc5b700 1 -- 172.21.15.16:0/2205694744 shutdown_connections 2023-10-16T12:29:09.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.504+0000 7f50ffc5b700 1 -- 172.21.15.16:0/2205694744 wait complete. 2023-10-16T12:29:09.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.505+0000 7f50ffc5b700 1 Processor -- start 2023-10-16T12:29:09.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.505+0000 7f50ffc5b700 1 -- start start 2023-10-16T12:29:09.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.506+0000 7f50ffc5b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f50f80d1d20 0x7f50f8152b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:09.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.506+0000 7f50ffc5b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f50f80d4060 0x7f50f80b4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:09.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.506+0000 7f50ffc5b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50f815da10 0x7f50f80b5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:09.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.506+0000 7f50ffc5b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50f81530a0 con 0x7f50f815da10 2023-10-16T12:29:09.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.506+0000 7f50ffc5b700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50f80b6fe0 con 0x7f50f80d1d20 2023-10-16T12:29:09.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.506+0000 7f50ffc5b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50f80b63d0 con 0x7f50f80d4060 2023-10-16T12:29:09.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.506+0000 7f50fe1f8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50f815da10 0x7f50f80b5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:09.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.506+0000 7f50fe1f8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50f815da10 0x7f50f80b5490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43542/0 (socket says 172.21.15.16:43542) 2023-10-16T12:29:09.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.506+0000 7f50fe1f8700 1 -- 172.21.15.16:0/1020016509 learned_addr learned my addr 172.21.15.16:0/1020016509 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:29:09.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.506+0000 7f50fd9f7700 1 --2- 172.21.15.16:0/1020016509 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f50f80d1d20 0x7f50f8152b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:09.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.506+0000 7f50fd1f6700 1 --2- 172.21.15.16:0/1020016509 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f50f80d4060 0x7f50f80b4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:09.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.507+0000 7f50fe1f8700 1 -- 172.21.15.16:0/1020016509 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f50f80d4060 msgr2=0x7f50f80b4f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:09.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.507+0000 7f50fe1f8700 1 --2- 172.21.15.16:0/1020016509 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f50f80d4060 0x7f50f80b4f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:09.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.507+0000 7f50fe1f8700 1 -- 172.21.15.16:0/1020016509 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f50f80d1d20 msgr2=0x7f50f8152b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:09.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.507+0000 7f50fe1f8700 1 --2- 172.21.15.16:0/1020016509 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f50f80d1d20 0x7f50f8152b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:09.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.507+0000 7f50fe1f8700 1 -- 172.21.15.16:0/1020016509 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50e8009500 con 0x7f50f815da10 2023-10-16T12:29:09.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.507+0000 7f50fe1f8700 1 --2- 172.21.15.16:0/1020016509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50f815da10 0x7f50f80b5490 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f50f000cc20 tx=0x7f50f000cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:09.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.507+0000 7f50ee7fc700 1 -- 172.21.15.16:0/1020016509 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50f0003c00 con 0x7f50f815da10 2023-10-16T12:29:09.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.507+0000 7f50ee7fc700 1 -- 172.21.15.16:0/1020016509 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50f0003d60 con 0x7f50f815da10 2023-10-16T12:29:09.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.507+0000 7f50ffc5b700 1 -- 172.21.15.16:0/1020016509 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50f40094a0 con 0x7f50f815da10 2023-10-16T12:29:09.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.507+0000 7f50ee7fc700 1 -- 172.21.15.16:0/1020016509 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50f000aae0 con 0x7f50f815da10 2023-10-16T12:29:09.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.507+0000 7f50ffc5b700 1 -- 172.21.15.16:0/1020016509 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50f80b6990 con 0x7f50f815da10 2023-10-16T12:29:09.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.508+0000 7f50ffc5b700 1 -- 172.21.15.16:0/1020016509 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50f80b7150 con 0x7f50f815da10 2023-10-16T12:29:09.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.509+0000 7f50ee7fc700 1 -- 172.21.15.16:0/1020016509 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 62) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f50f000ac40 con 0x7f50f815da10 2023-10-16T12:29:09.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.509+0000 7f50ee7fc700 1 --2- 172.21.15.16:0/1020016509 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f50dc06b020 0x7f50dc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:09.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.509+0000 7f50ee7fc700 1 -- 172.21.15.16:0/1020016509 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6065+0+0 (secure 0 0 0) 0x7f50f0089860 con 0x7f50f815da10 2023-10-16T12:29:09.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.509+0000 7f50fd9f7700 1 --2- 172.21.15.16:0/1020016509 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f50dc06b020 0x7f50dc06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:09.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.509+0000 7f50fd9f7700 1 --2- 172.21.15.16:0/1020016509 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f50dc06b020 0x7f50dc06d4e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f50e8006fd0 tx=0x7f50e8009290 comp rx=0 tx=0).ready entity=mgr.7884 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:09.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.513+0000 7f50ee7fc700 1 -- 172.21.15.16:0/1020016509 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f50f0054720 con 0x7f50f815da10 2023-10-16T12:29:09.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.718+0000 7f50ffc5b700 1 -- 172.21.15.16:0/1020016509 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f50f804ebb0 con 0x7f50f815da10 2023-10-16T12:29:09.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.721+0000 7f50ee7fc700 1 -- 172.21.15.16:0/1020016509 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v40)=0 v40) v1 ==== 122+0+0 (secure 0 0 0) 0x7f50f0057d40 con 0x7f50f815da10 2023-10-16T12:29:09.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.723+0000 7f50ffc5b700 1 -- 172.21.15.16:0/1020016509 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f50dc06b020 msgr2=0x7f50dc06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:09.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.723+0000 7f50ffc5b700 1 --2- 172.21.15.16:0/1020016509 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f50dc06b020 0x7f50dc06d4e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f50e8006fd0 tx=0x7f50e8009290 comp rx=0 tx=0).stop 2023-10-16T12:29:09.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.723+0000 7f50ffc5b700 1 -- 172.21.15.16:0/1020016509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50f815da10 msgr2=0x7f50f80b5490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:09.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.723+0000 7f50ffc5b700 1 --2- 172.21.15.16:0/1020016509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50f815da10 0x7f50f80b5490 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f50f000cc20 tx=0x7f50f000cf30 comp rx=0 tx=0).stop 2023-10-16T12:29:09.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.723+0000 7f50ffc5b700 1 -- 172.21.15.16:0/1020016509 shutdown_connections 2023-10-16T12:29:09.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.723+0000 7f50ffc5b700 1 --2- 172.21.15.16:0/1020016509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50f815da10 0x7f50f80b5490 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:09.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.723+0000 7f50ffc5b700 1 --2- 172.21.15.16:0/1020016509 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f50f80d1d20 0x7f50f8152b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:09.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.723+0000 7f50ffc5b700 1 --2- 172.21.15.16:0/1020016509 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f50dc06b020 0x7f50dc06d4e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:09.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.723+0000 7f50ffc5b700 1 --2- 172.21.15.16:0/1020016509 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f50f80d4060 0x7f50f80b4f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:09.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.723+0000 7f50ffc5b700 1 -- 172.21.15.16:0/1020016509 >> 172.21.15.16:0/1020016509 conn(0x7f50f809c9f0 msgr2=0x7f50f815e000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:09.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.723+0000 7f50ffc5b700 1 -- 172.21.15.16:0/1020016509 shutdown_connections 2023-10-16T12:29:09.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:09.723+0000 7f50ffc5b700 1 -- 172.21.15.16:0/1020016509 wait complete. 2023-10-16T12:29:09.737 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T12:29:09.738 DEBUG:teuthology.orchestra.run.smithi016:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T12:29:09.775 DEBUG:teuthology.orchestra.run.smithi049:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T12:29:09.828 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T12:29:09.828 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:29:09.829 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:29:09.846 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16287364k,nr_inodes=4071841,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20251 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3261172k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T12:29:09.847 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T12:29:09.847 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:29:09.868 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284204k,nr_inodes=4071051,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=753 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260540k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T12:29:09.868 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:29:09.868 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:29:09.904 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16287364k,nr_inodes=4071841,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20251 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3261172k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T12:29:09.904 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T12:29:09.904 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:29:09.928 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284204k,nr_inodes=4071051,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=753 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260540k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T12:29:09.928 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-16T12:29:09.928 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:29:10.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.067+0000 7fab4d0ea700 1 Processor -- start 2023-10-16T12:29:10.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.067+0000 7fab4d0ea700 1 -- start start 2023-10-16T12:29:10.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.068+0000 7fab4d0ea700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fab4815a870 0x7fab4806eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:10.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.068+0000 7fab4d0ea700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fab480cbed0 0x7fab480cde10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:10.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.068+0000 7fab4d0ea700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab4806f0a0 con 0x7fab4806d350 2023-10-16T12:29:10.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.068+0000 7fab4d0ea700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab4815ac50 con 0x7fab480cbed0 2023-10-16T12:29:10.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.068+0000 7fab4d0ea700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab4806f380 con 0x7fab4815a870 2023-10-16T12:29:10.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.068+0000 7fab46d9d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fab4815a870 0x7fab4806eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:10.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.068+0000 7fab4759e700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fab480cbed0 0x7fab480cde10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:10.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.068+0000 7fab4659c700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7fab4806d350 0x7fab480cb7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:40004/0 (socket says 172.21.15.16:40004) 2023-10-16T12:29:10.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.068+0000 7fab4759e700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fab480cbed0 0x7fab480cde10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:42614/0 (socket says 172.21.15.16:42614) 2023-10-16T12:29:10.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.068+0000 7fab46d9d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fab4815a870 0x7fab4806eaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:60038/0 (socket says 172.21.15.16:60038) 2023-10-16T12:29:10.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.068+0000 7fab4659c700 1 -- 172.21.15.16:0/4139986991 learned_addr learned my addr 172.21.15.16:0/4139986991 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:29:10.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.068+0000 7fab46d9d700 1 -- 172.21.15.16:0/4139986991 >> v2:172.21.15.49:3300/0 conn(0x7fab480cbed0 msgr2=0x7fab480cde10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:29:10.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.068+0000 7fab46d9d700 1 --2- 172.21.15.16:0/4139986991 >> v2:172.21.15.49:3300/0 conn(0x7fab480cbed0 0x7fab480cde10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:10.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.068+0000 7fab46d9d700 1 -- 172.21.15.16:0/4139986991 >> v1:172.21.15.49:6789/0 conn(0x7fab4806d350 legacy=0x7fab480cb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:10.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.068+0000 7fab46d9d700 1 -- 172.21.15.16:0/4139986991 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab4806f660 con 0x7fab4815a870 2023-10-16T12:29:10.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.069+0000 7fab4759e700 1 --2- 172.21.15.16:0/4139986991 >> v2:172.21.15.49:3300/0 conn(0x7fab480cbed0 0x7fab480cde10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:29:10.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.069+0000 7fab46d9d700 1 --2- 172.21.15.16:0/4139986991 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fab4815a870 0x7fab4806eaa0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fab3c009840 tx=0x7fab3c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8be499fb6ae2903d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:10.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.069+0000 7fab45d9b700 1 -- 172.21.15.16:0/4139986991 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab3c011830 con 0x7fab4815a870 2023-10-16T12:29:10.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.069+0000 7fab45d9b700 1 -- 172.21.15.16:0/4139986991 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab3c011990 con 0x7fab4815a870 2023-10-16T12:29:10.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.069+0000 7fab45d9b700 1 -- 172.21.15.16:0/4139986991 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab3c011d20 con 0x7fab4815a870 2023-10-16T12:29:10.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.069+0000 7fab4d0ea700 1 -- 172.21.15.16:0/4139986991 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fab4815a870 msgr2=0x7fab4806eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:10.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.069+0000 7fab4d0ea700 1 --2- 172.21.15.16:0/4139986991 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fab4815a870 0x7fab4806eaa0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fab3c009840 tx=0x7fab3c009b50 comp rx=0 tx=0).stop 2023-10-16T12:29:10.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.069+0000 7fab4d0ea700 1 -- 172.21.15.16:0/4139986991 shutdown_connections 2023-10-16T12:29:10.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.069+0000 7fab4d0ea700 1 --2- 172.21.15.16:0/4139986991 >> v2:172.21.15.49:3300/0 conn(0x7fab480cbed0 0x7fab480cde10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:10.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.069+0000 7fab4d0ea700 1 --2- 172.21.15.16:0/4139986991 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fab4815a870 0x7fab4806eaa0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:10.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.069+0000 7fab4d0ea700 1 -- 172.21.15.16:0/4139986991 >> 172.21.15.16:0/4139986991 conn(0x7fab480cc5b0 msgr2=0x7fab480cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:10.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.070+0000 7fab4d0ea700 1 -- 172.21.15.16:0/4139986991 shutdown_connections 2023-10-16T12:29:10.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.070+0000 7fab4d0ea700 1 -- 172.21.15.16:0/4139986991 wait complete. 2023-10-16T12:29:10.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.071+0000 7fab4d0ea700 1 Processor -- start 2023-10-16T12:29:10.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.071+0000 7fab4d0ea700 1 -- start start 2023-10-16T12:29:10.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.072+0000 7fab4d0ea700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fab4806d350 0x7fab481e48f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:10.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.072+0000 7fab4d0ea700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fab480cbed0 0x7fab480c5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:10.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.072+0000 7fab4d0ea700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab4815a870 0x7fab480ce4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:10.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.072+0000 7fab4d0ea700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab4814fcf0 con 0x7fab4815a870 2023-10-16T12:29:10.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.072+0000 7fab4d0ea700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab481e4e30 con 0x7fab4806d350 2023-10-16T12:29:10.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.072+0000 7fab4d0ea700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab48151220 con 0x7fab480cbed0 2023-10-16T12:29:10.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.072+0000 7fab4759e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab4815a870 0x7fab480ce4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:10.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.072+0000 7fab4759e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab4815a870 0x7fab480ce4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43558/0 (socket says 172.21.15.16:43558) 2023-10-16T12:29:10.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.072+0000 7fab4759e700 1 -- 172.21.15.16:0/1897777030 learned_addr learned my addr 172.21.15.16:0/1897777030 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:29:10.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.072+0000 7fab4659c700 1 --2- 172.21.15.16:0/1897777030 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fab480cbed0 0x7fab480c5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:10.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.072+0000 7fab46d9d700 1 --2- 172.21.15.16:0/1897777030 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fab4806d350 0x7fab481e48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:10.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.072+0000 7fab4759e700 1 -- 172.21.15.16:0/1897777030 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fab480cbed0 msgr2=0x7fab480c5990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:10.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.072+0000 7fab4759e700 1 --2- 172.21.15.16:0/1897777030 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fab480cbed0 0x7fab480c5990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:10.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.072+0000 7fab4759e700 1 -- 172.21.15.16:0/1897777030 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fab4806d350 msgr2=0x7fab481e48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:10.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.072+0000 7fab4759e700 1 --2- 172.21.15.16:0/1897777030 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fab4806d350 0x7fab481e48f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:10.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.072+0000 7fab4759e700 1 -- 172.21.15.16:0/1897777030 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab40009a70 con 0x7fab4815a870 2023-10-16T12:29:10.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.073+0000 7fab4759e700 1 --2- 172.21.15.16:0/1897777030 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab4815a870 0x7fab480ce4d0 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7fab4000efc0 tx=0x7fab4000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:10.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.073+0000 7fab2f7fe700 1 -- 172.21.15.16:0/1897777030 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab4000cc90 con 0x7fab4815a870 2023-10-16T12:29:10.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.073+0000 7fab2f7fe700 1 -- 172.21.15.16:0/1897777030 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab4000f040 con 0x7fab4815a870 2023-10-16T12:29:10.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.073+0000 7fab4d0ea700 1 -- 172.21.15.16:0/1897777030 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab3c0094a0 con 0x7fab4815a870 2023-10-16T12:29:10.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.073+0000 7fab2f7fe700 1 -- 172.21.15.16:0/1897777030 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab400146c0 con 0x7fab4815a870 2023-10-16T12:29:10.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.073+0000 7fab4d0ea700 1 -- 172.21.15.16:0/1897777030 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab48150480 con 0x7fab4815a870 2023-10-16T12:29:10.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.074+0000 7fab4d0ea700 1 -- 172.21.15.16:0/1897777030 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab48151090 con 0x7fab4815a870 2023-10-16T12:29:10.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.074+0000 7fab2f7fe700 1 -- 172.21.15.16:0/1897777030 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 62) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fab40014820 con 0x7fab4815a870 2023-10-16T12:29:10.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.075+0000 7fab2f7fe700 1 --2- 172.21.15.16:0/1897777030 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fab3406b020 0x7fab3406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:10.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.075+0000 7fab2f7fe700 1 -- 172.21.15.16:0/1897777030 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6065+0+0 (secure 0 0 0) 0x7fab4008a670 con 0x7fab4815a870 2023-10-16T12:29:10.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.075+0000 7fab46d9d700 1 --2- 172.21.15.16:0/1897777030 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fab3406b020 0x7fab3406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:10.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.075+0000 7fab46d9d700 1 --2- 172.21.15.16:0/1897777030 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fab3406b020 0x7fab3406d4e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fab3c000c00 tx=0x7fab3c008030 comp rx=0 tx=0).ready entity=mgr.7884 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:10.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.079+0000 7fab2f7fe700 1 -- 172.21.15.16:0/1897777030 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fab40055380 con 0x7fab4815a870 2023-10-16T12:29:10.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.328+0000 7fab4d0ea700 1 -- 172.21.15.16:0/1897777030 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fab4804ebb0 con 0x7fab4815a870 2023-10-16T12:29:10.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.329+0000 7fab2f7fe700 1 -- 172.21.15.16:0/1897777030 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) v1 ==== 94+0+3690 (secure 0 0 0) 0x7fab400589a0 con 0x7fab4815a870 2023-10-16T12:29:10.331 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:29:10.332 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":55,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6524,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/854923476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":854923476},{"type":"v1","addr":"172.21.15.49:6834","nonce":854923476}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":7249,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/1186699359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1186699359},{"type":"v1","addr":"172.21.15.16:6836","nonce":1186699359}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":7252,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/22289021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":22289021},{"type":"v1","addr":"172.21.15.16:6837","nonce":22289021}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":7627,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/3058391995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":3058391995},{"type":"v1","addr":"172.21.15.49:6835","nonce":3058391995}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55}],"filesystems":[]} 2023-10-16T12:29:10.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.331+0000 7fab4d0ea700 1 -- 172.21.15.16:0/1897777030 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fab3406b020 msgr2=0x7fab3406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:10.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.331+0000 7fab4d0ea700 1 --2- 172.21.15.16:0/1897777030 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fab3406b020 0x7fab3406d4e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fab3c000c00 tx=0x7fab3c008030 comp rx=0 tx=0).stop 2023-10-16T12:29:10.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.331+0000 7fab4d0ea700 1 -- 172.21.15.16:0/1897777030 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab4815a870 msgr2=0x7fab480ce4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:10.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.331+0000 7fab4d0ea700 1 --2- 172.21.15.16:0/1897777030 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab4815a870 0x7fab480ce4d0 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7fab4000efc0 tx=0x7fab4000c3a0 comp rx=0 tx=0).stop 2023-10-16T12:29:10.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.331+0000 7fab4d0ea700 1 -- 172.21.15.16:0/1897777030 shutdown_connections 2023-10-16T12:29:10.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.331+0000 7fab4d0ea700 1 --2- 172.21.15.16:0/1897777030 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab4815a870 0x7fab480ce4d0 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:10.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.331+0000 7fab4d0ea700 1 --2- 172.21.15.16:0/1897777030 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fab4806d350 0x7fab481e48f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:10.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.331+0000 7fab4d0ea700 1 --2- 172.21.15.16:0/1897777030 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fab3406b020 0x7fab3406d4e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:10.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.331+0000 7fab4d0ea700 1 --2- 172.21.15.16:0/1897777030 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fab480cbed0 0x7fab480c5990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:10.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.331+0000 7fab4d0ea700 1 -- 172.21.15.16:0/1897777030 >> 172.21.15.16:0/1897777030 conn(0x7fab480cc5b0 msgr2=0x7fab48151990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:10.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.331+0000 7fab4d0ea700 1 -- 172.21.15.16:0/1897777030 shutdown_connections 2023-10-16T12:29:10.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.331+0000 7fab4d0ea700 1 -- 172.21.15.16:0/1897777030 wait complete. 2023-10-16T12:29:10.336 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 55 2023-10-16T12:29:10.345 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-16T12:29:10.345 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-16T12:29:10.346 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-16T12:29:10.346 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T12:29:10.346 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-16T12:29:10.347 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-16T12:29:10.347 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-16T12:29:10.347 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T12:29:10.347 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-16T12:29:10.348 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-16T12:29:10.348 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-16T12:29:10.348 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T12:29:10.349 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-16T12:29:10.349 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-16T12:29:10.349 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-16T12:29:10.349 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T12:29:10.355 INFO:tasks.ceph.mds.a:Stopped 2023-10-16T12:29:10.355 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-16T12:29:10.356 INFO:tasks.ceph.mds.c:Stopped 2023-10-16T12:29:10.357 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-16T12:29:10.360 INFO:tasks.ceph.mds.b:Stopped 2023-10-16T12:29:10.360 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-16T12:29:10.361 INFO:tasks.ceph.mds.d:Stopped 2023-10-16T12:29:10.361 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-16T12:29:10.398 INFO:tasks.ceph.mds.a:Started 2023-10-16T12:29:10.399 DEBUG:teuthology.parallel:result is None 2023-10-16T12:29:10.400 INFO:tasks.ceph.mds.c:Started 2023-10-16T12:29:10.400 DEBUG:teuthology.parallel:result is None 2023-10-16T12:29:10.402 INFO:tasks.ceph.mds.b:Started 2023-10-16T12:29:10.403 DEBUG:teuthology.parallel:result is None 2023-10-16T12:29:10.403 INFO:tasks.ceph.mds.d:Started 2023-10-16T12:29:10.404 DEBUG:teuthology.parallel:result is None 2023-10-16T12:29:10.404 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-16T12:29:10.480 INFO:tasks.ceph.mds.a.smithi016.stdout:starting mds.a at 2023-10-16T12:29:10.481 INFO:tasks.ceph.mds.c.smithi016.stdout:starting mds.c at 2023-10-16T12:29:10.495 INFO:tasks.ceph.mds.b.smithi049.stdout:starting mds.b at 2023-10-16T12:29:10.496 INFO:tasks.ceph.mds.d.smithi049.stdout:starting mds.d at 2023-10-16T12:29:10.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.534+0000 7f74cfb92700 1 Processor -- start 2023-10-16T12:29:10.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.535+0000 7f74cfb92700 1 -- start start 2023-10-16T12:29:10.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.535+0000 7f74cfb92700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f74c80d41c0 0x7f74c80d0040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:10.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.535+0000 7f74cfb92700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74c806ba00 con 0x7f74c80d0580 2023-10-16T12:29:10.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.535+0000 7f74cfb92700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74c80d26d0 con 0x7f74c815d8d0 2023-10-16T12:29:10.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.535+0000 7f74cfb92700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74c80d0a10 con 0x7f74c80d41c0 2023-10-16T12:29:10.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.536+0000 7f74ce12f700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f74c80d0580 0x7f74c806c4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:32864/0 (socket says 172.21.15.16:32864) 2023-10-16T12:29:10.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.536+0000 7f74ce12f700 1 -- 172.21.15.16:0/973176280 learned_addr learned my addr 172.21.15.16:0/973176280 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:29:10.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.536+0000 7f74cd12d700 1 --2- 172.21.15.16:0/973176280 >> v2:172.21.15.49:3300/0 conn(0x7f74c80d41c0 0x7f74c80d0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:10.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.536+0000 7f74cc92c700 1 -- 172.21.15.16:0/973176280 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2620926012 0 0) 0x7f74c806ba00 con 0x7f74c80d0580 2023-10-16T12:29:10.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.536+0000 7f74cc92c700 1 -- 172.21.15.16:0/973176280 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74b40035b0 con 0x7f74c80d0580 2023-10-16T12:29:10.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.536+0000 7f74cd12d700 1 -- 172.21.15.16:0/973176280 >> v1:172.21.15.49:6789/0 conn(0x7f74c815d8d0 legacy=0x7f74c80d1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:10.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.536+0000 7f74cc92c700 1 -- 172.21.15.16:0/973176280 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3761117728 0 0) 0x7f74b40035b0 con 0x7f74c80d0580 2023-10-16T12:29:10.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.536+0000 7f74cd12d700 1 -- 172.21.15.16:0/973176280 >> v1:172.21.15.16:6789/0 conn(0x7f74c80d0580 legacy=0x7f74c806c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:10.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.536+0000 7f74cd12d700 1 -- 172.21.15.16:0/973176280 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74c80d23f0 con 0x7f74c80d41c0 2023-10-16T12:29:10.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.536+0000 7f74cd12d700 1 --2- 172.21.15.16:0/973176280 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f74c80d41c0 0x7f74c80d0040 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f74c4009840 tx=0x7f74c4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3b68e340d89cb407 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:10.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.536+0000 7f74cc92c700 1 -- 172.21.15.16:0/973176280 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74c00019f0 con 0x7f74c80d41c0 2023-10-16T12:29:10.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.536+0000 7f74cc92c700 1 -- 172.21.15.16:0/973176280 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74c4004030 con 0x7f74c80d41c0 2023-10-16T12:29:10.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.537+0000 7f74cc92c700 1 -- 172.21.15.16:0/973176280 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74c40043c0 con 0x7f74c80d41c0 2023-10-16T12:29:10.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.537+0000 7f74cfb92700 1 -- 172.21.15.16:0/973176280 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f74c80d41c0 msgr2=0x7f74c80d0040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:10.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.537+0000 7f74cfb92700 1 --2- 172.21.15.16:0/973176280 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f74c80d41c0 0x7f74c80d0040 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f74c4009840 tx=0x7f74c4009b50 comp rx=0 tx=0).stop 2023-10-16T12:29:10.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.537+0000 7f74cfb92700 1 -- 172.21.15.16:0/973176280 shutdown_connections 2023-10-16T12:29:10.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.537+0000 7f74cfb92700 1 --2- 172.21.15.16:0/973176280 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f74c80d41c0 0x7f74c80d0040 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:10.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.537+0000 7f74cfb92700 1 -- 172.21.15.16:0/973176280 >> 172.21.15.16:0/973176280 conn(0x7f74c80d3630 msgr2=0x7f74c80d3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:10.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.537+0000 7f74cfb92700 1 -- 172.21.15.16:0/973176280 shutdown_connections 2023-10-16T12:29:10.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.537+0000 7f74cfb92700 1 -- 172.21.15.16:0/973176280 wait complete. 2023-10-16T12:29:10.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.538+0000 7f74cfb92700 1 Processor -- start 2023-10-16T12:29:10.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.538+0000 7f74cfb92700 1 -- start start 2023-10-16T12:29:10.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.539+0000 7f74cfb92700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f74c80d0580 0x7f74c815b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:10.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.539+0000 7f74cfb92700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f74c80d41c0 0x7f74c80c7b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:10.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.539+0000 7f74cfb92700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74c815d8d0 0x7f74c80c8920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:10.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.539+0000 7f74cfb92700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74c80a01b0 con 0x7f74c815d8d0 2023-10-16T12:29:10.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.539+0000 7f74cfb92700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74c80c8530 con 0x7f74c80d0580 2023-10-16T12:29:10.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.539+0000 7f74cfb92700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74c809faf0 con 0x7f74c80d41c0 2023-10-16T12:29:10.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.539+0000 7f74ce12f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74c815d8d0 0x7f74c80c8920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:10.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.539+0000 7f74ce12f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74c815d8d0 0x7f74c80c8920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43596/0 (socket says 172.21.15.16:43596) 2023-10-16T12:29:10.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.539+0000 7f74ce12f700 1 -- 172.21.15.16:0/194891070 learned_addr learned my addr 172.21.15.16:0/194891070 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:29:10.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.539+0000 7f74cd12d700 1 --2- 172.21.15.16:0/194891070 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f74c80d41c0 0x7f74c80c7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:10.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.539+0000 7f74cd92e700 1 --2- 172.21.15.16:0/194891070 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f74c80d0580 0x7f74c815b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:10.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.539+0000 7f74ce12f700 1 -- 172.21.15.16:0/194891070 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f74c80d41c0 msgr2=0x7f74c80c7b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:10.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.539+0000 7f74ce12f700 1 --2- 172.21.15.16:0/194891070 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f74c80d41c0 0x7f74c80c7b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:10.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.539+0000 7f74ce12f700 1 -- 172.21.15.16:0/194891070 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f74c80d0580 msgr2=0x7f74c815b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:10.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.539+0000 7f74ce12f700 1 --2- 172.21.15.16:0/194891070 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f74c80d0580 0x7f74c815b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:10.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.539+0000 7f74ce12f700 1 -- 172.21.15.16:0/194891070 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74c40094a0 con 0x7f74c815d8d0 2023-10-16T12:29:10.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.539+0000 7f74cd12d700 1 --2- 172.21.15.16:0/194891070 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f74c80d41c0 0x7f74c80c7b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:29:10.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.539+0000 7f74cd92e700 1 --2- 172.21.15.16:0/194891070 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f74c80d0580 0x7f74c815b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:29:10.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.540+0000 7f74ce12f700 1 --2- 172.21.15.16:0/194891070 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74c815d8d0 0x7f74c80c8920 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f74c0001b20 tx=0x7f74c000a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:10.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.540+0000 7f74be7fc700 1 -- 172.21.15.16:0/194891070 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74c000ade0 con 0x7f74c815d8d0 2023-10-16T12:29:10.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.540+0000 7f74be7fc700 1 -- 172.21.15.16:0/194891070 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74c000ce60 con 0x7f74c815d8d0 2023-10-16T12:29:10.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.540+0000 7f74cfb92700 1 -- 172.21.15.16:0/194891070 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74c809fdd0 con 0x7f74c815d8d0 2023-10-16T12:29:10.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.540+0000 7f74be7fc700 1 -- 172.21.15.16:0/194891070 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74c00072f0 con 0x7f74c815d8d0 2023-10-16T12:29:10.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.540+0000 7f74cfb92700 1 -- 172.21.15.16:0/194891070 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74c809f7a0 con 0x7f74c815d8d0 2023-10-16T12:29:10.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.541+0000 7f74cfb92700 1 -- 172.21.15.16:0/194891070 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74c804ebb0 con 0x7f74c815d8d0 2023-10-16T12:29:10.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.541+0000 7f74be7fc700 1 -- 172.21.15.16:0/194891070 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 62) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f74c0007450 con 0x7f74c815d8d0 2023-10-16T12:29:10.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.541+0000 7f74be7fc700 1 --2- 172.21.15.16:0/194891070 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f74ac06b020 0x7f74ac06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:10.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.542+0000 7f74be7fc700 1 -- 172.21.15.16:0/194891070 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6065+0+0 (secure 0 0 0) 0x7f74c00897b0 con 0x7f74c815d8d0 2023-10-16T12:29:10.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.542+0000 7f74cd92e700 1 --2- 172.21.15.16:0/194891070 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f74ac06b020 0x7f74ac06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:10.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.542+0000 7f74cd92e700 1 --2- 172.21.15.16:0/194891070 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f74ac06b020 0x7f74ac06d4e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f74b80018c0 tx=0x7f74b8009040 comp rx=0 tx=0).ready entity=mgr.7884 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:10.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.546+0000 7f74be7fc700 1 -- 172.21.15.16:0/194891070 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f74c0054670 con 0x7f74c815d8d0 2023-10-16T12:29:10.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:10.764+0000 7f74cfb92700 1 -- 172.21.15.16:0/194891070 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f74c80b13d0 con 0x7f74c815d8d0 2023-10-16T12:29:12.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.286+0000 7f74be7fc700 1 -- 172.21.15.16:0/194891070 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v124) v1 ==== 97+0+0 (secure 0 0 0) 0x7f74c0057c90 con 0x7f74c815d8d0 2023-10-16T12:29:12.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.289+0000 7f74cfb92700 1 -- 172.21.15.16:0/194891070 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f74ac06b020 msgr2=0x7f74ac06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:12.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.289+0000 7f74cfb92700 1 --2- 172.21.15.16:0/194891070 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f74ac06b020 0x7f74ac06d4e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f74b80018c0 tx=0x7f74b8009040 comp rx=0 tx=0).stop 2023-10-16T12:29:12.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.289+0000 7f74cfb92700 1 -- 172.21.15.16:0/194891070 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74c815d8d0 msgr2=0x7f74c80c8920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:12.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.289+0000 7f74cfb92700 1 --2- 172.21.15.16:0/194891070 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74c815d8d0 0x7f74c80c8920 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f74c0001b20 tx=0x7f74c000a340 comp rx=0 tx=0).stop 2023-10-16T12:29:12.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.289+0000 7f74cfb92700 1 -- 172.21.15.16:0/194891070 shutdown_connections 2023-10-16T12:29:12.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.289+0000 7f74cfb92700 1 --2- 172.21.15.16:0/194891070 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74c815d8d0 0x7f74c80c8920 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:12.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.289+0000 7f74cfb92700 1 --2- 172.21.15.16:0/194891070 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f74c80d0580 0x7f74c815b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:12.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.289+0000 7f74cfb92700 1 --2- 172.21.15.16:0/194891070 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f74ac06b020 0x7f74ac06d4e0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:12.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.289+0000 7f74cfb92700 1 --2- 172.21.15.16:0/194891070 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f74c80d41c0 0x7f74c80c7b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:12.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.290+0000 7f74cfb92700 1 -- 172.21.15.16:0/194891070 >> 172.21.15.16:0/194891070 conn(0x7f74c80d3630 msgr2=0x7f74c80d1c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:12.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.290+0000 7f74cfb92700 1 -- 172.21.15.16:0/194891070 shutdown_connections 2023-10-16T12:29:12.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.290+0000 7f74cfb92700 1 -- 172.21.15.16:0/194891070 wait complete. 2023-10-16T12:29:12.295 INFO:teuthology.orchestra.run.smithi016.stderr: removed all blocklist entries 2023-10-16T12:29:12.307 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-16T12:29:12.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.440+0000 7f426d916700 1 Processor -- start 2023-10-16T12:29:12.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.440+0000 7f426d916700 1 -- start start 2023-10-16T12:29:12.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.440+0000 7f426d916700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f426815d8d0 0x7f42680d1c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:12.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.440+0000 7f426d916700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f42680d0750 0x7f426806c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:12.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.440+0000 7f426d916700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f426806ba00 con 0x7f42680d41c0 2023-10-16T12:29:12.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.440+0000 7f426d916700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42680d2150 con 0x7f426815d8d0 2023-10-16T12:29:12.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.440+0000 7f426d916700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42680d26d0 con 0x7f42680d0750 2023-10-16T12:29:12.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.441+0000 7f4266ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f426815d8d0 0x7f42680d1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:12.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.441+0000 7f4266ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f426815d8d0 0x7f42680d1c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43604/0 (socket says 172.21.15.16:43604) 2023-10-16T12:29:12.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.441+0000 7f4266ffd700 1 -- 172.21.15.16:0/698419598 learned_addr learned my addr 172.21.15.16:0/698419598 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:29:12.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.441+0000 7f42677fe700 1 --2- 172.21.15.16:0/698419598 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f42680d0750 0x7f426806c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:12.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.441+0000 7f4266ffd700 1 -- 172.21.15.16:0/698419598 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f42680d0750 msgr2=0x7f426806c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:29:12.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.441+0000 7f4266ffd700 1 --2- 172.21.15.16:0/698419598 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f42680d0750 0x7f426806c4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:12.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.441+0000 7f4266ffd700 1 -- 172.21.15.16:0/698419598 >> v1:172.21.15.49:6789/0 conn(0x7f42680d41c0 legacy=0x7f42680d0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:29:12.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.441+0000 7f4266ffd700 1 -- 172.21.15.16:0/698419598 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42680d23f0 con 0x7f426815d8d0 2023-10-16T12:29:12.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.441+0000 7f4266ffd700 1 --2- 172.21.15.16:0/698419598 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f426815d8d0 0x7f42680d1c10 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f425c0098f0 tx=0x7f425c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1ab949888a5b09b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:12.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.441+0000 7f4265ffb700 1 -- 172.21.15.16:0/698419598 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f425c004230 con 0x7f426815d8d0 2023-10-16T12:29:12.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.441+0000 7f4265ffb700 1 -- 172.21.15.16:0/698419598 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f425c004390 con 0x7f426815d8d0 2023-10-16T12:29:12.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.441+0000 7f4265ffb700 1 -- 172.21.15.16:0/698419598 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f425c0036a0 con 0x7f426815d8d0 2023-10-16T12:29:12.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.442+0000 7f426d916700 1 -- 172.21.15.16:0/698419598 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f426815d8d0 msgr2=0x7f42680d1c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:12.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.442+0000 7f426d916700 1 --2- 172.21.15.16:0/698419598 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f426815d8d0 0x7f42680d1c10 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f425c0098f0 tx=0x7f425c009c00 comp rx=0 tx=0).stop 2023-10-16T12:29:12.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.442+0000 7f426d916700 1 -- 172.21.15.16:0/698419598 shutdown_connections 2023-10-16T12:29:12.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.442+0000 7f426d916700 1 --2- 172.21.15.16:0/698419598 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f42680d0750 0x7f426806c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:12.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.442+0000 7f426d916700 1 --2- 172.21.15.16:0/698419598 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f426815d8d0 0x7f42680d1c10 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:12.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.442+0000 7f426d916700 1 -- 172.21.15.16:0/698419598 >> 172.21.15.16:0/698419598 conn(0x7f42680d3630 msgr2=0x7f42680d3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:12.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.442+0000 7f426d916700 1 -- 172.21.15.16:0/698419598 shutdown_connections 2023-10-16T12:29:12.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.442+0000 7f426d916700 1 -- 172.21.15.16:0/698419598 wait complete. 2023-10-16T12:29:12.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.443+0000 7f426d916700 1 Processor -- start 2023-10-16T12:29:12.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.443+0000 7f426d916700 1 -- start start 2023-10-16T12:29:12.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.444+0000 7f426d916700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f42680d0750 0x7f42680b86d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:12.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.444+0000 7f426d916700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42680d41c0 0x7f4268148760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:12.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.444+0000 7f426d916700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f426815d8d0 0x7f4268148ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:12.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.444+0000 7f426d916700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42680b8c10 con 0x7f42680d41c0 2023-10-16T12:29:12.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.444+0000 7f426d916700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f426815a010 con 0x7f426815d8d0 2023-10-16T12:29:12.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.444+0000 7f426d916700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42680c8e00 con 0x7f42680d0750 2023-10-16T12:29:12.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.444+0000 7f42667fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42680d41c0 0x7f4268148760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:12.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.444+0000 7f42667fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42680d41c0 0x7f4268148760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43612/0 (socket says 172.21.15.16:43612) 2023-10-16T12:29:12.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.444+0000 7f42667fc700 1 -- 172.21.15.16:0/1183792635 learned_addr learned my addr 172.21.15.16:0/1183792635 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:29:12.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.444+0000 7f4266ffd700 1 --2- 172.21.15.16:0/1183792635 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f42680d0750 0x7f42680b86d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:12.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.444+0000 7f42677fe700 1 --2- 172.21.15.16:0/1183792635 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f426815d8d0 0x7f4268148ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:12.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.444+0000 7f42667fc700 1 -- 172.21.15.16:0/1183792635 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f42680d0750 msgr2=0x7f42680b86d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:12.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.444+0000 7f42667fc700 1 --2- 172.21.15.16:0/1183792635 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f42680d0750 0x7f42680b86d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:12.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.444+0000 7f42667fc700 1 -- 172.21.15.16:0/1183792635 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f426815d8d0 msgr2=0x7f4268148ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:12.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.444+0000 7f42667fc700 1 --2- 172.21.15.16:0/1183792635 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f426815d8d0 0x7f4268148ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:12.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.444+0000 7f42667fc700 1 -- 172.21.15.16:0/1183792635 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f425c009550 con 0x7f42680d41c0 2023-10-16T12:29:12.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.445+0000 7f42667fc700 1 --2- 172.21.15.16:0/1183792635 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42680d41c0 0x7f4268148760 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f425800c4d0 tx=0x7f425800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:12.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.445+0000 7f424ffff700 1 -- 172.21.15.16:0/1183792635 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4258011790 con 0x7f42680d41c0 2023-10-16T12:29:12.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.445+0000 7f424ffff700 1 -- 172.21.15.16:0/1183792635 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42580118f0 con 0x7f42680d41c0 2023-10-16T12:29:12.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.445+0000 7f426d916700 1 -- 172.21.15.16:0/1183792635 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42680c90e0 con 0x7f42680d41c0 2023-10-16T12:29:12.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.445+0000 7f424ffff700 1 -- 172.21.15.16:0/1183792635 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4258011ca0 con 0x7f42680d41c0 2023-10-16T12:29:12.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.445+0000 7f426d916700 1 -- 172.21.15.16:0/1183792635 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42680c96a0 con 0x7f42680d41c0 2023-10-16T12:29:12.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.446+0000 7f426d916700 1 -- 172.21.15.16:0/1183792635 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f426804ebb0 con 0x7f42680d41c0 2023-10-16T12:29:12.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.446+0000 7f424ffff700 1 -- 172.21.15.16:0/1183792635 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 62) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f425801c460 con 0x7f42680d41c0 2023-10-16T12:29:12.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.447+0000 7f424ffff700 1 --2- 172.21.15.16:0/1183792635 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f425406b020 0x7f425406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:12.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.447+0000 7f4266ffd700 1 --2- 172.21.15.16:0/1183792635 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f425406b020 0x7f425406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:12.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.447+0000 7f424ffff700 1 -- 172.21.15.16:0/1183792635 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f42580893d0 con 0x7f42680d41c0 2023-10-16T12:29:12.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.447+0000 7f4266ffd700 1 --2- 172.21.15.16:0/1183792635 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f425406b020 0x7f425406d4e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f425c009130 tx=0x7f425c008030 comp rx=0 tx=0).ready entity=mgr.7884 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:12.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.451+0000 7f424ffff700 1 -- 172.21.15.16:0/1183792635 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4258054560 con 0x7f42680d41c0 2023-10-16T12:29:12.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.736+0000 7f426d916700 1 -- 172.21.15.16:0/1183792635 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f426806d400 con 0x7f42680d41c0 2023-10-16T12:29:12.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.736+0000 7f424ffff700 1 -- 172.21.15.16:0/1183792635 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v22) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f4258057b80 con 0x7f42680d41c0 2023-10-16T12:29:12.739 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:29:12.739 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-10-16T12:29:12.739 INFO:teuthology.orchestra.run.smithi016.stdout: "auth_dump": [ 2023-10-16T12:29:12.740 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:29:12.740 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.a", 2023-10-16T12:29:12.740 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOKC1l/rBtOhAA/ob5WUvpz8Glp4E+xPZOYA==", 2023-10-16T12:29:12.741 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:29:12.741 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T12:29:12.741 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-10-16T12:29:12.741 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-10-16T12:29:12.741 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:29:12.741 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:29:12.742 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:29:12.742 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:29:12.742 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.b", 2023-10-16T12:29:12.743 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1lKAVsCxAAvhXuLY6i1+Iz1yq5BVPMNQ==", 2023-10-16T12:29:12.743 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:29:12.743 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T12:29:12.743 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-10-16T12:29:12.743 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-10-16T12:29:12.744 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:29:12.744 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:29:12.744 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:29:12.744 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:29:12.745 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.c", 2023-10-16T12:29:12.745 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1lvBowBhAA3waJNYhdgs8PGCYqYNYUog==", 2023-10-16T12:29:12.745 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:29:12.746 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T12:29:12.746 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-10-16T12:29:12.746 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-10-16T12:29:12.746 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:29:12.747 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:29:12.747 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:29:12.747 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:29:12.747 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.d", 2023-10-16T12:29:12.748 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1lS/fnEhAAfNYgQ5BW86JMjYm/bBfnaQ==", 2023-10-16T12:29:12.748 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:29:12.748 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T12:29:12.748 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-10-16T12:29:12.749 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-10-16T12:29:12.749 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:29:12.749 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:29:12.749 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:29:12.749 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:29:12.749 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.0", 2023-10-16T12:29:12.750 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAQKC1lbxllOhAAZeLdp2GMl6HqQPaQjKlNqw==", 2023-10-16T12:29:12.750 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:29:12.750 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T12:29:12.750 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T12:29:12.750 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:29:12.751 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:29:12.751 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:29:12.751 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:29:12.751 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.1", 2023-10-16T12:29:12.751 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQATKC1lnU6KOhAAHxvlbmzBptf1keGgL+Rbkg==", 2023-10-16T12:29:12.753 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:29:12.753 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T12:29:12.753 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T12:29:12.753 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:29:12.754 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:29:12.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:29:12.754 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:29:12.754 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.2", 2023-10-16T12:29:12.755 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAXKC1l9jFyARAAr/NAwtWhBMut9sAAoDEGWg==", 2023-10-16T12:29:12.755 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:29:12.755 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T12:29:12.756 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T12:29:12.756 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:29:12.756 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:29:12.756 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:29:12.757 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:29:12.757 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.3", 2023-10-16T12:29:12.757 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAaKC1lbw/lARAAGJ/9DNb5/7kLFSq7vrLtXg==", 2023-10-16T12:29:12.757 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:29:12.758 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T12:29:12.758 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T12:29:12.758 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:29:12.758 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:29:12.758 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:29:12.759 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:29:12.759 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.4", 2023-10-16T12:29:12.759 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAeKC1lZb7jERAAYpvb1xcSli9CTq7L8ChLOA==", 2023-10-16T12:29:12.759 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:29:12.759 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T12:29:12.760 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T12:29:12.760 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:29:12.760 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:29:12.760 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:29:12.761 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:29:12.761 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.5", 2023-10-16T12:29:12.761 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAhKC1l/4HuEBAAqfnRqjknij9KES/fqiyIBQ==", 2023-10-16T12:29:12.761 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:29:12.762 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T12:29:12.762 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T12:29:12.762 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:29:12.762 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:29:12.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:29:12.762 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:29:12.763 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.6", 2023-10-16T12:29:12.763 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAkKC1lzoUkEBAA+Q82OiBMNag8+R316WQtUw==", 2023-10-16T12:29:12.763 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:29:12.763 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T12:29:12.763 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T12:29:12.764 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:29:12.764 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:29:12.764 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:29:12.764 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:29:12.765 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.7", 2023-10-16T12:29:12.765 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAnKC1ljUu4DxAAc+G4anJANC2GhLJ8wb6gYg==", 2023-10-16T12:29:12.765 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:29:12.765 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T12:29:12.766 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T12:29:12.766 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:29:12.767 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:29:12.767 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:29:12.767 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:29:12.767 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.0", 2023-10-16T12:29:12.767 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1l2l6PFxAA9QyXtBAHdGAXpwH0JfknuQ==", 2023-10-16T12:29:12.768 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:29:12.768 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T12:29:12.768 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow r", 2023-10-16T12:29:12.768 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-16T12:29:12.768 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:29:12.769 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:29:12.769 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:29:12.769 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.1", 2023-10-16T12:29:12.769 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1lAcHFIxAAcNt2bmv2R6j5L3DYkdw7eQ==", 2023-10-16T12:29:12.769 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:29:12.770 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T12:29:12.770 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-10-16T12:29:12.770 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-10-16T12:29:12.770 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-10-16T12:29:12.770 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:29:12.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:29:12.771 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:29:12.771 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.2", 2023-10-16T12:29:12.771 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1lA6wnGxAAdtaqYhtIJu8qENMrD8xvVA==", 2023-10-16T12:29:12.771 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:29:12.771 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T12:29:12.772 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-10-16T12:29:12.772 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-10-16T12:29:12.772 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-10-16T12:29:12.772 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:29:12.772 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:29:12.773 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:29:12.773 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.3", 2023-10-16T12:29:12.773 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1l/ddAJxAA3fNKKn9XkXas1BlDN+ggjw==", 2023-10-16T12:29:12.773 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:29:12.773 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T12:29:12.774 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-10-16T12:29:12.774 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-10-16T12:29:12.774 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-10-16T12:29:12.774 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:29:12.774 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:29:12.775 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:29:12.775 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.admin", 2023-10-16T12:29:12.775 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOKC1ld5jUHBAAc6esnCce3dSEZn3wVjhIlQ==", 2023-10-16T12:29:12.775 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:29:12.775 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2023-10-16T12:29:12.776 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-10-16T12:29:12.776 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-10-16T12:29:12.776 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:29:12.776 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:29:12.776 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:29:12.777 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:29:12.777 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mgr.x", 2023-10-16T12:29:12.777 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOKC1l6b3vNhAAmj5xbfid2sZM2KfHmAf1Wg==", 2023-10-16T12:29:12.777 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:29:12.777 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2023-10-16T12:29:12.777 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile mgr", 2023-10-16T12:29:12.778 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:29:12.778 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:29:12.778 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:29:12.778 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:29:12.778 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mgr.y", 2023-10-16T12:29:12.779 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOKC1lLz2nMhAA8NqFYZ5nDKkzPBMrkwsVuQ==", 2023-10-16T12:29:12.779 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:29:12.779 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2023-10-16T12:29:12.779 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile mgr", 2023-10-16T12:29:12.779 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:29:12.780 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:29:12.780 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:29:12.780 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-10-16T12:29:12.780 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-10-16T12:29:12.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.738+0000 7f426d916700 1 -- 172.21.15.16:0/1183792635 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f425406b020 msgr2=0x7f425406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:12.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.738+0000 7f426d916700 1 --2- 172.21.15.16:0/1183792635 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f425406b020 0x7f425406d4e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f425c009130 tx=0x7f425c008030 comp rx=0 tx=0).stop 2023-10-16T12:29:12.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.738+0000 7f426d916700 1 -- 172.21.15.16:0/1183792635 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42680d41c0 msgr2=0x7f4268148760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:12.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.738+0000 7f426d916700 1 --2- 172.21.15.16:0/1183792635 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42680d41c0 0x7f4268148760 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f425800c4d0 tx=0x7f425800c890 comp rx=0 tx=0).stop 2023-10-16T12:29:12.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.738+0000 7f426d916700 1 -- 172.21.15.16:0/1183792635 shutdown_connections 2023-10-16T12:29:12.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.738+0000 7f426d916700 1 --2- 172.21.15.16:0/1183792635 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f426815d8d0 0x7f4268148ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:12.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.739+0000 7f426d916700 1 --2- 172.21.15.16:0/1183792635 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f425406b020 0x7f425406d4e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:12.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.739+0000 7f426d916700 1 --2- 172.21.15.16:0/1183792635 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f42680d0750 0x7f42680b86d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:12.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.739+0000 7f426d916700 1 --2- 172.21.15.16:0/1183792635 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42680d41c0 0x7f4268148760 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:12.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.739+0000 7f426d916700 1 -- 172.21.15.16:0/1183792635 >> 172.21.15.16:0/1183792635 conn(0x7f42680d3630 msgr2=0x7f42680caae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:12.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.739+0000 7f426d916700 1 -- 172.21.15.16:0/1183792635 shutdown_connections 2023-10-16T12:29:12.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.739+0000 7f426d916700 1 -- 172.21.15.16:0/1183792635 wait complete. 2023-10-16T12:29:12.785 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:29:12.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.908+0000 7f0237954700 1 Processor -- start 2023-10-16T12:29:12.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.908+0000 7f0237954700 1 -- start start 2023-10-16T12:29:12.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.909+0000 7f0237954700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f023015b280 0x7f02301613e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:12.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.909+0000 7f0237954700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f023006a7e0 0x7f023006ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:12.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.909+0000 7f0237954700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f023006b150 con 0x7f02300cc8a0 2023-10-16T12:29:12.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.909+0000 7f0237954700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0230161c30 con 0x7f023015b280 2023-10-16T12:29:12.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.909+0000 7f0237954700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0230161f10 con 0x7f023006a7e0 2023-10-16T12:29:12.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.909+0000 7f02356f0700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f023015b280 0x7f02301613e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:12.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.909+0000 7f02356f0700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f023015b280 0x7f02301613e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43618/0 (socket says 172.21.15.16:43618) 2023-10-16T12:29:12.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.909+0000 7f02356f0700 1 -- 172.21.15.16:0/315517428 learned_addr learned my addr 172.21.15.16:0/315517428 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:29:12.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.909+0000 7f0235ef1700 1 --2- 172.21.15.16:0/315517428 >> v2:172.21.15.49:3300/0 conn(0x7f023006a7e0 0x7f023006ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:12.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.909+0000 7f02356f0700 1 -- 172.21.15.16:0/315517428 >> v2:172.21.15.49:3300/0 conn(0x7f023006a7e0 msgr2=0x7f023006ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:29:12.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.909+0000 7f02356f0700 1 --2- 172.21.15.16:0/315517428 >> v2:172.21.15.49:3300/0 conn(0x7f023006a7e0 0x7f023006ac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:12.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.909+0000 7f02356f0700 1 -- 172.21.15.16:0/315517428 >> v1:172.21.15.49:6789/0 conn(0x7f02300cc8a0 legacy=0x7f023015e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:29:12.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.909+0000 7f02356f0700 1 -- 172.21.15.16:0/315517428 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02301621f0 con 0x7f023015b280 2023-10-16T12:29:12.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.909+0000 7f02356f0700 1 --2- 172.21.15.16:0/315517428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f023015b280 0x7f02301613e0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f02300cbed0 tx=0x7f022c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30e2c3bc2bb67dc9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:12.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.910+0000 7f0227fff700 1 -- 172.21.15.16:0/315517428 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f022c004230 con 0x7f023015b280 2023-10-16T12:29:12.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.910+0000 7f0227fff700 1 -- 172.21.15.16:0/315517428 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f022c004390 con 0x7f023015b280 2023-10-16T12:29:12.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.910+0000 7f0227fff700 1 -- 172.21.15.16:0/315517428 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f022c0036a0 con 0x7f023015b280 2023-10-16T12:29:12.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.910+0000 7f0237954700 1 -- 172.21.15.16:0/315517428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f023015b280 msgr2=0x7f02301613e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:12.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.910+0000 7f0237954700 1 --2- 172.21.15.16:0/315517428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f023015b280 0x7f02301613e0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f02300cbed0 tx=0x7f022c009bd0 comp rx=0 tx=0).stop 2023-10-16T12:29:12.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.910+0000 7f0237954700 1 -- 172.21.15.16:0/315517428 shutdown_connections 2023-10-16T12:29:12.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.910+0000 7f0237954700 1 --2- 172.21.15.16:0/315517428 >> v2:172.21.15.49:3300/0 conn(0x7f023006a7e0 0x7f023006ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:12.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.910+0000 7f0237954700 1 --2- 172.21.15.16:0/315517428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f023015b280 0x7f02301613e0 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:12.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.910+0000 7f0237954700 1 -- 172.21.15.16:0/315517428 >> 172.21.15.16:0/315517428 conn(0x7f02300b1fd0 msgr2=0x7f02300b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:12.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.910+0000 7f0237954700 1 -- 172.21.15.16:0/315517428 shutdown_connections 2023-10-16T12:29:12.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.910+0000 7f0237954700 1 -- 172.21.15.16:0/315517428 wait complete. 2023-10-16T12:29:12.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.912+0000 7f0237954700 1 Processor -- start 2023-10-16T12:29:12.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.912+0000 7f0237954700 1 -- start start 2023-10-16T12:29:12.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.912+0000 7f0237954700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f023006a7e0 0x7f02300c9c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:12.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.912+0000 7f0237954700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f02300cc8a0 0x7f02301f3130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:12.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.912+0000 7f0237954700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f023015b280 0x7f02301f3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:12.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.912+0000 7f0237954700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02300c7640 con 0x7f023006a7e0 2023-10-16T12:29:12.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.912+0000 7f0237954700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02300c64c0 con 0x7f023015b280 2023-10-16T12:29:12.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.912+0000 7f0237954700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02300c67a0 con 0x7f02300cc8a0 2023-10-16T12:29:12.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.912+0000 7f02356f0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f023006a7e0 0x7f02300c9c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:12.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.912+0000 7f02356f0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f023006a7e0 0x7f02300c9c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43624/0 (socket says 172.21.15.16:43624) 2023-10-16T12:29:12.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.912+0000 7f02356f0700 1 -- 172.21.15.16:0/662404798 learned_addr learned my addr 172.21.15.16:0/662404798 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:29:12.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.913+0000 7f02356f0700 1 -- 172.21.15.16:0/662404798 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f02300cc8a0 msgr2=0x7f02301f3130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:12.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.913+0000 7f0235ef1700 1 --2- 172.21.15.16:0/662404798 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f023015b280 0x7f02301f3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:12.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.913+0000 7f0234eef700 1 --2- 172.21.15.16:0/662404798 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f02300cc8a0 0x7f02301f3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:12.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.913+0000 7f02356f0700 1 --2- 172.21.15.16:0/662404798 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f02300cc8a0 0x7f02301f3130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:12.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.913+0000 7f02356f0700 1 -- 172.21.15.16:0/662404798 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f023015b280 msgr2=0x7f02301f3670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:12.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.913+0000 7f02356f0700 1 --2- 172.21.15.16:0/662404798 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f023015b280 0x7f02301f3670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:12.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.913+0000 7f02356f0700 1 -- 172.21.15.16:0/662404798 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f022c009550 con 0x7f023006a7e0 2023-10-16T12:29:12.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.913+0000 7f0235ef1700 1 --2- 172.21.15.16:0/662404798 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f023015b280 0x7f02301f3670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:29:12.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.913+0000 7f0234eef700 1 --2- 172.21.15.16:0/662404798 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f02300cc8a0 0x7f02301f3130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:29:12.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.913+0000 7f02356f0700 1 --2- 172.21.15.16:0/662404798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f023006a7e0 0x7f02300c9c10 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f022c009130 tx=0x7f022c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:12.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.913+0000 7f0225ffb700 1 -- 172.21.15.16:0/662404798 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f022c003dc0 con 0x7f023006a7e0 2023-10-16T12:29:12.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.913+0000 7f0225ffb700 1 -- 172.21.15.16:0/662404798 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f022c01f070 con 0x7f023006a7e0 2023-10-16T12:29:12.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.913+0000 7f0237954700 1 -- 172.21.15.16:0/662404798 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02300c6a80 con 0x7f023006a7e0 2023-10-16T12:29:12.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.913+0000 7f0225ffb700 1 -- 172.21.15.16:0/662404798 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f022c011870 con 0x7f023006a7e0 2023-10-16T12:29:12.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.913+0000 7f0237954700 1 -- 172.21.15.16:0/662404798 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f023015d400 con 0x7f023006a7e0 2023-10-16T12:29:12.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.914+0000 7f0237954700 1 -- 172.21.15.16:0/662404798 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f023015b660 con 0x7f023006a7e0 2023-10-16T12:29:12.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.915+0000 7f0225ffb700 1 -- 172.21.15.16:0/662404798 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 62) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f022c021030 con 0x7f023006a7e0 2023-10-16T12:29:12.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.915+0000 7f0225ffb700 1 --2- 172.21.15.16:0/662404798 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f021406b020 0x7f021406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:12.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.915+0000 7f0225ffb700 1 -- 172.21.15.16:0/662404798 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f022c08acc0 con 0x7f023006a7e0 2023-10-16T12:29:12.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.915+0000 7f0234eef700 1 --2- 172.21.15.16:0/662404798 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f021406b020 0x7f021406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:12.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.916+0000 7f0234eef700 1 --2- 172.21.15.16:0/662404798 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f021406b020 0x7f021406d4e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f02301f0d20 tx=0x7f0220009040 comp rx=0 tx=0).ready entity=mgr.7884 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:12.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:12.920+0000 7f0225ffb700 1 -- 172.21.15.16:0/662404798 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f022c091080 con 0x7f023006a7e0 2023-10-16T12:29:13.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.127+0000 7f0237954700 1 -- 172.21.15.16:0/662404798 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f023004ebb0 con 0x7f023006a7e0 2023-10-16T12:29:13.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.127+0000 7f0225ffb700 1 -- 172.21.15.16:0/662404798 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v124) v1 ==== 74+0+13007 (secure 0 0 0) 0x7f022c055f60 con 0x7f023006a7e0 2023-10-16T12:29:13.129 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:29:13.130 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":124,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:29:12.285815+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:29:13.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.129+0000 7f0237954700 1 -- 172.21.15.16:0/662404798 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f021406b020 msgr2=0x7f021406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:13.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.129+0000 7f0237954700 1 --2- 172.21.15.16:0/662404798 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f021406b020 0x7f021406d4e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f02301f0d20 tx=0x7f0220009040 comp rx=0 tx=0).stop 2023-10-16T12:29:13.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.129+0000 7f0237954700 1 -- 172.21.15.16:0/662404798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f023006a7e0 msgr2=0x7f02300c9c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:13.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.129+0000 7f0237954700 1 --2- 172.21.15.16:0/662404798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f023006a7e0 0x7f02300c9c10 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f022c009130 tx=0x7f022c004750 comp rx=0 tx=0).stop 2023-10-16T12:29:13.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.130+0000 7f0237954700 1 -- 172.21.15.16:0/662404798 shutdown_connections 2023-10-16T12:29:13.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.130+0000 7f0237954700 1 --2- 172.21.15.16:0/662404798 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f023015b280 0x7f02301f3670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:13.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.130+0000 7f0237954700 1 --2- 172.21.15.16:0/662404798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f023006a7e0 0x7f02300c9c10 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:13.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.130+0000 7f0237954700 1 --2- 172.21.15.16:0/662404798 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f021406b020 0x7f021406d4e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:13.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.130+0000 7f0237954700 1 --2- 172.21.15.16:0/662404798 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f02300cc8a0 0x7f02301f3130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:13.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.130+0000 7f0237954700 1 -- 172.21.15.16:0/662404798 >> 172.21.15.16:0/662404798 conn(0x7f02300b1fd0 msgr2=0x7f02300b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:13.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.130+0000 7f0237954700 1 -- 172.21.15.16:0/662404798 shutdown_connections 2023-10-16T12:29:13.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.130+0000 7f0237954700 1 -- 172.21.15.16:0/662404798 wait complete. 2023-10-16T12:29:13.144 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T12:29:13.144 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:29:13.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.271+0000 7fed6e1ca700 1 Processor -- start 2023-10-16T12:29:13.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.271+0000 7fed6e1ca700 1 -- start start 2023-10-16T12:29:13.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.271+0000 7fed6e1ca700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fed6815ec80 0x7fed680cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:13.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.271+0000 7fed6e1ca700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fed6815f060 0x7fed680ce400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:13.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.271+0000 7fed6e1ca700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed680b2440 con 0x7fed680d2110 2023-10-16T12:29:13.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.271+0000 7fed6e1ca700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed680b2c50 con 0x7fed6815f060 2023-10-16T12:29:13.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.271+0000 7fed6e1ca700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed680d00e0 con 0x7fed6815ec80 2023-10-16T12:29:13.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.271+0000 7fed66ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fed6815f060 0x7fed680ce400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:13.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.272+0000 7fed66ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fed6815f060 0x7fed680ce400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43632/0 (socket says 172.21.15.16:43632) 2023-10-16T12:29:13.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.272+0000 7fed66ffd700 1 -- 172.21.15.16:0/1743066335 learned_addr learned my addr 172.21.15.16:0/1743066335 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:29:13.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.272+0000 7fed677fe700 1 --2- 172.21.15.16:0/1743066335 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fed6815ec80 0x7fed680cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:13.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.272+0000 7fed66ffd700 1 -- 172.21.15.16:0/1743066335 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fed6815ec80 msgr2=0x7fed680cdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:29:13.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.272+0000 7fed66ffd700 1 --2- 172.21.15.16:0/1743066335 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fed6815ec80 0x7fed680cdec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:13.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.272+0000 7fed66ffd700 1 -- 172.21.15.16:0/1743066335 >> v1:172.21.15.16:6789/0 conn(0x7fed680d2110 legacy=0x7fed680b4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:13.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.272+0000 7fed667fc700 1 -- 172.21.15.16:0/1743066335 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 640663923 0 0) 0x7fed680b2440 con 0x7fed680d2110 2023-10-16T12:29:13.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.272+0000 7fed66ffd700 1 -- 172.21.15.16:0/1743066335 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed680cef60 con 0x7fed6815f060 2023-10-16T12:29:13.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.272+0000 7fed677fe700 1 --2- 172.21.15.16:0/1743066335 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fed6815ec80 0x7fed680cdec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:29:13.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.272+0000 7fed66ffd700 1 --2- 172.21.15.16:0/1743066335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed6815f060 0x7fed680ce400 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7fed5c0098f0 tx=0x7fed5c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdb4dd1b636fcc3b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:13.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.272+0000 7fed667fc700 1 -- 172.21.15.16:0/1743066335 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed5c00b8a0 con 0x7fed6815f060 2023-10-16T12:29:13.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.272+0000 7fed667fc700 1 -- 172.21.15.16:0/1743066335 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed5c00ba00 con 0x7fed6815f060 2023-10-16T12:29:13.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.272+0000 7fed667fc700 1 -- 172.21.15.16:0/1743066335 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed5c00bd90 con 0x7fed6815f060 2023-10-16T12:29:13.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.273+0000 7fed6e1ca700 1 -- 172.21.15.16:0/1743066335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed6815f060 msgr2=0x7fed680ce400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:13.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.273+0000 7fed6e1ca700 1 --2- 172.21.15.16:0/1743066335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed6815f060 0x7fed680ce400 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7fed5c0098f0 tx=0x7fed5c009c00 comp rx=0 tx=0).stop 2023-10-16T12:29:13.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.273+0000 7fed6e1ca700 1 -- 172.21.15.16:0/1743066335 shutdown_connections 2023-10-16T12:29:13.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.273+0000 7fed6e1ca700 1 --2- 172.21.15.16:0/1743066335 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fed6815ec80 0x7fed680cdec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:13.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.273+0000 7fed6e1ca700 1 --2- 172.21.15.16:0/1743066335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed6815f060 0x7fed680ce400 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:13.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.273+0000 7fed6e1ca700 1 -- 172.21.15.16:0/1743066335 >> 172.21.15.16:0/1743066335 conn(0x7fed680d15f0 msgr2=0x7fed680d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:13.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.273+0000 7fed6e1ca700 1 -- 172.21.15.16:0/1743066335 shutdown_connections 2023-10-16T12:29:13.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.273+0000 7fed6e1ca700 1 -- 172.21.15.16:0/1743066335 wait complete. 2023-10-16T12:29:13.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.274+0000 7fed6e1ca700 1 Processor -- start 2023-10-16T12:29:13.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.274+0000 7fed6e1ca700 1 -- start start 2023-10-16T12:29:13.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.275+0000 7fed6e1ca700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fed680d2110 0x7fed68154b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:13.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.275+0000 7fed6e1ca700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed6815ec80 0x7fed68153f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:13.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.275+0000 7fed6e1ca700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fed6815f060 0x7fed680c87e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:13.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.275+0000 7fed6e1ca700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed681544d0 con 0x7fed6815ec80 2023-10-16T12:29:13.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.275+0000 7fed6e1ca700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed681e6800 con 0x7fed6815f060 2023-10-16T12:29:13.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.275+0000 7fed6e1ca700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed681e5c80 con 0x7fed680d2110 2023-10-16T12:29:13.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.275+0000 7fed66ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed6815ec80 0x7fed68153f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:13.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.275+0000 7fed66ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed6815ec80 0x7fed68153f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43640/0 (socket says 172.21.15.16:43640) 2023-10-16T12:29:13.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.275+0000 7fed66ffd700 1 -- 172.21.15.16:0/1039807748 learned_addr learned my addr 172.21.15.16:0/1039807748 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:29:13.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.275+0000 7fed677fe700 1 --2- 172.21.15.16:0/1039807748 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fed680d2110 0x7fed68154b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:13.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.275+0000 7fed67fff700 1 --2- 172.21.15.16:0/1039807748 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fed6815f060 0x7fed680c87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:13.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.275+0000 7fed66ffd700 1 -- 172.21.15.16:0/1039807748 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fed680d2110 msgr2=0x7fed68154b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:13.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.275+0000 7fed66ffd700 1 --2- 172.21.15.16:0/1039807748 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fed680d2110 0x7fed68154b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:13.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.275+0000 7fed66ffd700 1 -- 172.21.15.16:0/1039807748 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fed6815f060 msgr2=0x7fed680c87e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:13.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.275+0000 7fed66ffd700 1 --2- 172.21.15.16:0/1039807748 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fed6815f060 0x7fed680c87e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:13.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.275+0000 7fed66ffd700 1 -- 172.21.15.16:0/1039807748 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed5c009550 con 0x7fed6815ec80 2023-10-16T12:29:13.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.276+0000 7fed66ffd700 1 --2- 172.21.15.16:0/1039807748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed6815ec80 0x7fed68153f90 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7fed5c000c00 tx=0x7fed5c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:13.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.276+0000 7fed4ffff700 1 -- 172.21.15.16:0/1039807748 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed5c0038a0 con 0x7fed6815ec80 2023-10-16T12:29:13.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.276+0000 7fed4ffff700 1 -- 172.21.15.16:0/1039807748 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed5c01e070 con 0x7fed6815ec80 2023-10-16T12:29:13.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.276+0000 7fed4ffff700 1 -- 172.21.15.16:0/1039807748 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed5c003c20 con 0x7fed6815ec80 2023-10-16T12:29:13.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.276+0000 7fed6e1ca700 1 -- 172.21.15.16:0/1039807748 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed681e5f60 con 0x7fed6815ec80 2023-10-16T12:29:13.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.276+0000 7fed6e1ca700 1 -- 172.21.15.16:0/1039807748 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed681e6520 con 0x7fed6815ec80 2023-10-16T12:29:13.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.277+0000 7fed6e1ca700 1 -- 172.21.15.16:0/1039807748 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed680c6150 con 0x7fed6815ec80 2023-10-16T12:29:13.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.277+0000 7fed4ffff700 1 -- 172.21.15.16:0/1039807748 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 62) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fed5c011420 con 0x7fed6815ec80 2023-10-16T12:29:13.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.277+0000 7fed4ffff700 1 --2- 172.21.15.16:0/1039807748 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fed5406b020 0x7fed5406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:13.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.278+0000 7fed4ffff700 1 -- 172.21.15.16:0/1039807748 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5205+0+0 (secure 0 0 0) 0x7fed5c020070 con 0x7fed6815ec80 2023-10-16T12:29:13.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.278+0000 7fed677fe700 1 --2- 172.21.15.16:0/1039807748 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fed5406b020 0x7fed5406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:13.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.278+0000 7fed677fe700 1 --2- 172.21.15.16:0/1039807748 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fed5406b020 0x7fed5406d4e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fed58006fd0 tx=0x7fed58008040 comp rx=0 tx=0).ready entity=mgr.7884 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:13.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.282+0000 7fed4ffff700 1 -- 172.21.15.16:0/1039807748 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fed680c6150 con 0x7fed6815ec80 2023-10-16T12:29:13.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.484+0000 7fed6e1ca700 1 -- 172.21.15.16:0/1039807748 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fed680c6150 con 0x7fed6815ec80 2023-10-16T12:29:13.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.484+0000 7fed4ffff700 1 -- 172.21.15.16:0/1039807748 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v124) v1 ==== 93+0+10 (secure 0 0 0) 0x7fed680c6150 con 0x7fed6815ec80 2023-10-16T12:29:13.486 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T12:29:13.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.486+0000 7fed6e1ca700 1 -- 172.21.15.16:0/1039807748 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fed5406b020 msgr2=0x7fed5406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:13.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.486+0000 7fed6e1ca700 1 --2- 172.21.15.16:0/1039807748 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fed5406b020 0x7fed5406d4e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fed58006fd0 tx=0x7fed58008040 comp rx=0 tx=0).stop 2023-10-16T12:29:13.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.486+0000 7fed6e1ca700 1 -- 172.21.15.16:0/1039807748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed6815ec80 msgr2=0x7fed68153f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:13.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.486+0000 7fed6e1ca700 1 --2- 172.21.15.16:0/1039807748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed6815ec80 0x7fed68153f90 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7fed5c000c00 tx=0x7fed5c004750 comp rx=0 tx=0).stop 2023-10-16T12:29:13.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.486+0000 7fed6e1ca700 1 -- 172.21.15.16:0/1039807748 shutdown_connections 2023-10-16T12:29:13.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.486+0000 7fed6e1ca700 1 --2- 172.21.15.16:0/1039807748 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fed6815f060 0x7fed680c87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:13.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.486+0000 7fed6e1ca700 1 --2- 172.21.15.16:0/1039807748 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fed5406b020 0x7fed5406d4e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:13.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.486+0000 7fed6e1ca700 1 --2- 172.21.15.16:0/1039807748 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fed680d2110 0x7fed68154b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:13.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.486+0000 7fed6e1ca700 1 --2- 172.21.15.16:0/1039807748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed6815ec80 0x7fed68153f90 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:13.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.486+0000 7fed6e1ca700 1 -- 172.21.15.16:0/1039807748 >> 172.21.15.16:0/1039807748 conn(0x7fed680d15f0 msgr2=0x7fed680cc190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:13.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.486+0000 7fed6e1ca700 1 -- 172.21.15.16:0/1039807748 shutdown_connections 2023-10-16T12:29:13.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.486+0000 7fed6e1ca700 1 -- 172.21.15.16:0/1039807748 wait complete. 2023-10-16T12:29:13.501 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-16T12:29:13.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.624+0000 7f42cfb99700 1 Processor -- start 2023-10-16T12:29:13.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.624+0000 7f42cfb99700 1 -- start start 2023-10-16T12:29:13.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.625+0000 7f42cfb99700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f42c815b630 0x7f42c80cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:13.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.625+0000 7f42cfb99700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f42c80ce4f0 0x7f42c80cfca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:13.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.625+0000 7f42cfb99700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42c80cc2a0 con 0x7f42c815ba10 2023-10-16T12:29:13.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.625+0000 7f42cfb99700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42c80cdb20 con 0x7f42c815b630 2023-10-16T12:29:13.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.625+0000 7f42cfb99700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42c80cc540 con 0x7f42c80ce4f0 2023-10-16T12:29:13.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.625+0000 7f42cd134700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f42c815ba10 0x7f42c80cdde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:32920/0 (socket says 172.21.15.16:32920) 2023-10-16T12:29:13.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.625+0000 7f42cd134700 1 -- 172.21.15.16:0/1132376783 learned_addr learned my addr 172.21.15.16:0/1132376783 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:29:13.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.625+0000 7f42ce136700 1 --2- 172.21.15.16:0/1132376783 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f42c80ce4f0 0x7f42c80cfca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:13.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.625+0000 7f42cd935700 1 --2- 172.21.15.16:0/1132376783 >> v2:172.21.15.49:3300/0 conn(0x7f42c815b630 0x7f42c80cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:13.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.625+0000 7f42cc933700 1 -- 172.21.15.16:0/1132376783 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 712976076 0 0) 0x7f42c80cc2a0 con 0x7f42c815ba10 2023-10-16T12:29:13.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.625+0000 7f42cc933700 1 -- 172.21.15.16:0/1132376783 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42b40035b0 con 0x7f42c815ba10 2023-10-16T12:29:13.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.625+0000 7f42cc933700 1 -- 172.21.15.16:0/1132376783 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3898884087 0 0) 0x7f42b40035b0 con 0x7f42c815ba10 2023-10-16T12:29:13.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.625+0000 7f42cc933700 1 -- 172.21.15.16:0/1132376783 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f42c80ce4f0 msgr2=0x7f42c80cfca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:29:13.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.625+0000 7f42cc933700 1 --2- 172.21.15.16:0/1132376783 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f42c80ce4f0 0x7f42c80cfca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:13.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.625+0000 7f42cc933700 1 -- 172.21.15.16:0/1132376783 >> v2:172.21.15.49:3300/0 conn(0x7f42c815b630 msgr2=0x7f42c80cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:29:13.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.625+0000 7f42cc933700 1 --2- 172.21.15.16:0/1132376783 >> v2:172.21.15.49:3300/0 conn(0x7f42c815b630 0x7f42c80cd5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:13.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.625+0000 7f42cc933700 1 -- 172.21.15.16:0/1132376783 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42c80cc820 con 0x7f42c815ba10 2023-10-16T12:29:13.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.625+0000 7f42cc933700 1 -- 172.21.15.16:0/1132376783 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f42b8001670 con 0x7f42c815ba10 2023-10-16T12:29:13.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.625+0000 7f42cc933700 1 -- 172.21.15.16:0/1132376783 >> v1:172.21.15.16:6789/0 conn(0x7f42c815ba10 legacy=0x7f42c80cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:13.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.625+0000 7f42cc933700 1 --2- 172.21.15.16:0/1132376783 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f42b4003a00 0x7f42b4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:13.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.625+0000 7f42cc933700 1 --2- 172.21.15.16:0/1132376783 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f42c80ce4f0 0x7f42b4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:13.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.625+0000 7f42cc933700 1 --2- 172.21.15.16:0/1132376783 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42b4004d90 0x7f42b4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:13.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.625+0000 7f42cc933700 1 -- 172.21.15.16:0/1132376783 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42c80cc540 con 0x7f42b4004d90 2023-10-16T12:29:13.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.625+0000 7f42cc933700 1 -- 172.21.15.16:0/1132376783 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42b400ba10 con 0x7f42b4003a00 2023-10-16T12:29:13.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.625+0000 7f42cc933700 1 -- 172.21.15.16:0/1132376783 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42b400bb80 con 0x7f42c80ce4f0 2023-10-16T12:29:13.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.626+0000 7f42ce136700 1 --2- 172.21.15.16:0/1132376783 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42b4004d90 0x7f42b4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:13.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.626+0000 7f42cd935700 1 --2- 172.21.15.16:0/1132376783 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f42b4003a00 0x7f42b4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:13.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.626+0000 7f42cd134700 1 --2- 172.21.15.16:0/1132376783 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f42c80ce4f0 0x7f42b4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:13.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.626+0000 7f42ce136700 1 -- 172.21.15.16:0/1132376783 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f42c80ce4f0 msgr2=0x7f42b4004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:29:13.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.626+0000 7f42ce136700 1 --2- 172.21.15.16:0/1132376783 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f42c80ce4f0 0x7f42b4004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:13.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.626+0000 7f42ce136700 1 -- 172.21.15.16:0/1132376783 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f42b4003a00 msgr2=0x7f42b4003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:29:13.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.626+0000 7f42ce136700 1 --2- 172.21.15.16:0/1132376783 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f42b4003a00 0x7f42b4003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:13.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.626+0000 7f42ce136700 1 -- 172.21.15.16:0/1132376783 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42b40035b0 con 0x7f42b4004d90 2023-10-16T12:29:13.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.626+0000 7f42ce136700 1 --2- 172.21.15.16:0/1132376783 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42b4004d90 0x7f42b4005210 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f42c0009f90 tx=0x7f42b4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d65c01cf0b88df34 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:13.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.626+0000 7f42cc933700 1 -- 172.21.15.16:0/1132376783 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42c000ebd0 con 0x7f42b4004d90 2023-10-16T12:29:13.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.626+0000 7f42cc933700 1 -- 172.21.15.16:0/1132376783 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42c000f040 con 0x7f42b4004d90 2023-10-16T12:29:13.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.626+0000 7f42cc933700 1 -- 172.21.15.16:0/1132376783 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42c0014480 con 0x7f42b4004d90 2023-10-16T12:29:13.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.627+0000 7f42cfb99700 1 -- 172.21.15.16:0/1132376783 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42b4004d90 msgr2=0x7f42b4005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:13.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.627+0000 7f42cfb99700 1 --2- 172.21.15.16:0/1132376783 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42b4004d90 0x7f42b4005210 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f42c0009f90 tx=0x7f42b4005880 comp rx=0 tx=0).stop 2023-10-16T12:29:13.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.627+0000 7f42ce136700 1 -- 172.21.15.16:0/1132376783 reap_dead start 2023-10-16T12:29:13.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.627+0000 7f42cfb99700 1 -- 172.21.15.16:0/1132376783 shutdown_connections 2023-10-16T12:29:13.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.627+0000 7f42cfb99700 1 -- 172.21.15.16:0/1132376783 >> 172.21.15.16:0/1132376783 conn(0x7f42c80d14b0 msgr2=0x7f42c80d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:13.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.627+0000 7f42cfb99700 1 -- 172.21.15.16:0/1132376783 shutdown_connections 2023-10-16T12:29:13.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.627+0000 7f42cfb99700 1 -- 172.21.15.16:0/1132376783 wait complete. 2023-10-16T12:29:13.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.628+0000 7f42cfb99700 1 Processor -- start 2023-10-16T12:29:13.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.628+0000 7f42cfb99700 1 -- start start 2023-10-16T12:29:13.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.629+0000 7f42cfb99700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f42b4004d90 0x7f42c806c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:13.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.629+0000 7f42cfb99700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f42c80caf00 0x7f42c80c6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:13.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.629+0000 7f42cfb99700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42c80c7c80 0x7f42c80b4a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:13.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.629+0000 7f42cfb99700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42c80cb380 con 0x7f42c80c7c80 2023-10-16T12:29:13.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.629+0000 7f42cfb99700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42c80c6be0 con 0x7f42b4004d90 2023-10-16T12:29:13.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.629+0000 7f42cfb99700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42c80c6ec0 con 0x7f42c80caf00 2023-10-16T12:29:13.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.629+0000 7f42ce136700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42c80c7c80 0x7f42c80b4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:13.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.629+0000 7f42ce136700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42c80c7c80 0x7f42c80b4a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43662/0 (socket says 172.21.15.16:43662) 2023-10-16T12:29:13.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.629+0000 7f42ce136700 1 -- 172.21.15.16:0/304987865 learned_addr learned my addr 172.21.15.16:0/304987865 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:29:13.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.629+0000 7f42cd935700 1 --2- 172.21.15.16:0/304987865 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f42b4004d90 0x7f42c806c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:13.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.629+0000 7f42cd134700 1 --2- 172.21.15.16:0/304987865 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f42c80caf00 0x7f42c80c6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:13.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.629+0000 7f42ce136700 1 -- 172.21.15.16:0/304987865 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f42c80caf00 msgr2=0x7f42c80c6300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:13.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.629+0000 7f42ce136700 1 --2- 172.21.15.16:0/304987865 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f42c80caf00 0x7f42c80c6300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:13.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.629+0000 7f42ce136700 1 -- 172.21.15.16:0/304987865 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f42b4004d90 msgr2=0x7f42c806c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:13.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.629+0000 7f42ce136700 1 --2- 172.21.15.16:0/304987865 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f42b4004d90 0x7f42c806c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:13.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.629+0000 7f42ce136700 1 -- 172.21.15.16:0/304987865 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42c80c71a0 con 0x7f42c80c7c80 2023-10-16T12:29:13.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.629+0000 7f42ce136700 1 --2- 172.21.15.16:0/304987865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42c80c7c80 0x7f42c80b4a50 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f42c0016000 tx=0x7f42c0009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:13.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.630+0000 7f42be7fc700 1 -- 172.21.15.16:0/304987865 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42c0007000 con 0x7f42c80c7c80 2023-10-16T12:29:13.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.630+0000 7f42be7fc700 1 -- 172.21.15.16:0/304987865 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42c0015a80 con 0x7f42c80c7c80 2023-10-16T12:29:13.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.630+0000 7f42cfb99700 1 -- 172.21.15.16:0/304987865 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42c80c7480 con 0x7f42c80c7c80 2023-10-16T12:29:13.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.630+0000 7f42be7fc700 1 -- 172.21.15.16:0/304987865 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42c0015e90 con 0x7f42c80c7c80 2023-10-16T12:29:13.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.630+0000 7f42cfb99700 1 -- 172.21.15.16:0/304987865 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42c80c6840 con 0x7f42c80c7c80 2023-10-16T12:29:13.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.630+0000 7f42cfb99700 1 -- 172.21.15.16:0/304987865 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42c80c7940 con 0x7f42c80c7c80 2023-10-16T12:29:13.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.631+0000 7f42be7fc700 1 -- 172.21.15.16:0/304987865 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 62) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f42c001e430 con 0x7f42c80c7c80 2023-10-16T12:29:13.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.631+0000 7f42be7fc700 1 --2- 172.21.15.16:0/304987865 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f42ac06b020 0x7f42ac06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:13.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.632+0000 7f42be7fc700 1 -- 172.21.15.16:0/304987865 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f42c0024030 con 0x7f42c80c7c80 2023-10-16T12:29:13.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.632+0000 7f42cd935700 1 --2- 172.21.15.16:0/304987865 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f42ac06b020 0x7f42ac06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:13.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.632+0000 7f42cd935700 1 --2- 172.21.15.16:0/304987865 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f42ac06b020 0x7f42ac06d4e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f42c4009c40 tx=0x7f42c40091c0 comp rx=0 tx=0).ready entity=mgr.7884 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:13.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.636+0000 7f42be7fc700 1 -- 172.21.15.16:0/304987865 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f42c00562d0 con 0x7f42c80c7c80 2023-10-16T12:29:13.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.839+0000 7f42cfb99700 1 -- 172.21.15.16:0/304987865 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f42c804ebb0 con 0x7f42c80c7c80 2023-10-16T12:29:13.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.839+0000 7f42be7fc700 1 -- 172.21.15.16:0/304987865 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v124) v1 ==== 84+0+65 (secure 0 0 0) 0x7f42c0018060 con 0x7f42c80c7c80 2023-10-16T12:29:13.842 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:29:13.842 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2023-10-16T12:29:13.842 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:29:13.842 INFO:teuthology.orchestra.run.smithi016.stdout: "poolnum": 1, 2023-10-16T12:29:13.842 INFO:teuthology.orchestra.run.smithi016.stdout: "poolname": ".mgr" 2023-10-16T12:29:13.843 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:29:13.843 INFO:teuthology.orchestra.run.smithi016.stdout:] 2023-10-16T12:29:13.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.842+0000 7f42cfb99700 1 -- 172.21.15.16:0/304987865 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f42ac06b020 msgr2=0x7f42ac06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:13.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.842+0000 7f42cfb99700 1 --2- 172.21.15.16:0/304987865 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f42ac06b020 0x7f42ac06d4e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f42c4009c40 tx=0x7f42c40091c0 comp rx=0 tx=0).stop 2023-10-16T12:29:13.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.842+0000 7f42cfb99700 1 -- 172.21.15.16:0/304987865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42c80c7c80 msgr2=0x7f42c80b4a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:13.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.842+0000 7f42cfb99700 1 --2- 172.21.15.16:0/304987865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42c80c7c80 0x7f42c80b4a50 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f42c0016000 tx=0x7f42c0009750 comp rx=0 tx=0).stop 2023-10-16T12:29:13.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.842+0000 7f42cfb99700 1 -- 172.21.15.16:0/304987865 shutdown_connections 2023-10-16T12:29:13.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.842+0000 7f42cfb99700 1 --2- 172.21.15.16:0/304987865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42c80c7c80 0x7f42c80b4a50 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:13.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.842+0000 7f42cfb99700 1 --2- 172.21.15.16:0/304987865 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f42b4004d90 0x7f42c806c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:13.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.842+0000 7f42cfb99700 1 --2- 172.21.15.16:0/304987865 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f42ac06b020 0x7f42ac06d4e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:13.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.842+0000 7f42cfb99700 1 --2- 172.21.15.16:0/304987865 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f42c80caf00 0x7f42c80c6300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:13.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.842+0000 7f42cfb99700 1 -- 172.21.15.16:0/304987865 >> 172.21.15.16:0/304987865 conn(0x7f42c80d14b0 msgr2=0x7f42c80cf060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:13.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.842+0000 7f42cfb99700 1 -- 172.21.15.16:0/304987865 shutdown_connections 2023-10-16T12:29:13.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.842+0000 7f42cfb99700 1 -- 172.21.15.16:0/304987865 wait complete. 2023-10-16T12:29:13.856 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-16T12:29:13.856 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-16T12:29:13.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.984+0000 7f368e8f8700 1 Processor -- start 2023-10-16T12:29:13.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.984+0000 7f368e8f8700 1 -- start start 2023-10-16T12:29:13.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.985+0000 7f368e8f8700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f368815e0b0 0x7f368806d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:13.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.985+0000 7f368e8f8700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36880cb870 con 0x7f368815a3d0 2023-10-16T12:29:13.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.985+0000 7f368e8f8700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f368806e040 con 0x7f36880cf600 2023-10-16T12:29:13.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.985+0000 7f368e8f8700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f368815f0c0 con 0x7f368815e0b0 2023-10-16T12:29:13.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.985+0000 7f3687fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f368815a3d0 0x7f36880ceef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:32940/0 (socket says 172.21.15.16:32940) 2023-10-16T12:29:13.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.985+0000 7f3687fff700 1 -- 172.21.15.16:0/2597060971 learned_addr learned my addr 172.21.15.16:0/2597060971 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:29:13.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.985+0000 7f368ce95700 1 --2- 172.21.15.16:0/2597060971 >> v2:172.21.15.49:3300/0 conn(0x7f368815e0b0 0x7f368806d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:13.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.985+0000 7f3686ffd700 1 -- 172.21.15.16:0/2597060971 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1185165922 0 0) 0x7f36880cb870 con 0x7f368815a3d0 2023-10-16T12:29:13.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.985+0000 7f3686ffd700 1 -- 172.21.15.16:0/2597060971 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36700035b0 con 0x7f368815a3d0 2023-10-16T12:29:13.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.986+0000 7f3686ffd700 1 -- 172.21.15.16:0/2597060971 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3108690487 0 0) 0x7f36700035b0 con 0x7f368815a3d0 2023-10-16T12:29:13.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.986+0000 7f3686ffd700 1 -- 172.21.15.16:0/2597060971 >> v2:172.21.15.49:3300/0 conn(0x7f368815e0b0 msgr2=0x7f368806d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:29:13.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.986+0000 7f3686ffd700 1 --2- 172.21.15.16:0/2597060971 >> v2:172.21.15.49:3300/0 conn(0x7f368815e0b0 0x7f368806d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:13.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.986+0000 7f3686ffd700 1 -- 172.21.15.16:0/2597060971 >> v1:172.21.15.49:6789/0 conn(0x7f36880cf600 legacy=0x7f36880cbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:13.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.986+0000 7f3686ffd700 1 -- 172.21.15.16:0/2597060971 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f368806e320 con 0x7f368815a3d0 2023-10-16T12:29:13.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.986+0000 7f3686ffd700 1 -- 172.21.15.16:0/2597060971 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f3678001670 con 0x7f368815a3d0 2023-10-16T12:29:13.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.986+0000 7f3686ffd700 1 -- 172.21.15.16:0/2597060971 >> v1:172.21.15.16:6789/0 conn(0x7f368815a3d0 legacy=0x7f36880ceef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:13.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.986+0000 7f3686ffd700 1 --2- 172.21.15.16:0/2597060971 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3670003a00 0x7f3670003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:13.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.986+0000 7f3686ffd700 1 --2- 172.21.15.16:0/2597060971 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3670004850 0x7f3670004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:13.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.986+0000 7f3686ffd700 1 --2- 172.21.15.16:0/2597060971 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3670005210 0x7f367000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:13.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.986+0000 7f3686ffd700 1 -- 172.21.15.16:0/2597060971 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f368815f0c0 con 0x7f3670004850 2023-10-16T12:29:13.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.986+0000 7f3686ffd700 1 -- 172.21.15.16:0/2597060971 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f367000c0b0 con 0x7f3670005210 2023-10-16T12:29:13.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.986+0000 7f3686ffd700 1 -- 172.21.15.16:0/2597060971 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f367000c270 con 0x7f3670003a00 2023-10-16T12:29:13.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.986+0000 7f36877fe700 1 --2- 172.21.15.16:0/2597060971 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3670004850 0x7f3670004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:13.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.986+0000 7f3687fff700 1 --2- 172.21.15.16:0/2597060971 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3670003a00 0x7f3670003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:13.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.986+0000 7f368ce95700 1 --2- 172.21.15.16:0/2597060971 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3670005210 0x7f367000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:13.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.986+0000 7f36877fe700 1 -- 172.21.15.16:0/2597060971 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3670003a00 msgr2=0x7f3670003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:29:13.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.986+0000 7f36877fe700 1 --2- 172.21.15.16:0/2597060971 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3670003a00 0x7f3670003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:13.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.986+0000 7f36877fe700 1 -- 172.21.15.16:0/2597060971 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3670005210 msgr2=0x7f367000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:29:13.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.986+0000 7f36877fe700 1 --2- 172.21.15.16:0/2597060971 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3670005210 0x7f367000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:13.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.986+0000 7f36877fe700 1 -- 172.21.15.16:0/2597060971 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36700035b0 con 0x7f3670004850 2023-10-16T12:29:13.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.986+0000 7f368ce95700 1 --2- 172.21.15.16:0/2597060971 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3670005210 0x7f367000ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:29:13.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.986+0000 7f368ce95700 1 -- 172.21.15.16:0/2597060971 reap_dead start 2023-10-16T12:29:13.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.986+0000 7f36877fe700 1 --2- 172.21.15.16:0/2597060971 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3670004850 0x7f3670004cd0 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f367c0017c0 tx=0x7f3670005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6c1002e96e4a197 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:13.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.987+0000 7f3686ffd700 1 -- 172.21.15.16:0/2597060971 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f367c0052d0 con 0x7f3670004850 2023-10-16T12:29:13.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.987+0000 7f3686ffd700 1 -- 172.21.15.16:0/2597060971 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f367c005430 con 0x7f3670004850 2023-10-16T12:29:13.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.987+0000 7f3686ffd700 1 -- 172.21.15.16:0/2597060971 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f367c0038c0 con 0x7f3670004850 2023-10-16T12:29:13.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.987+0000 7f368e8f8700 1 -- 172.21.15.16:0/2597060971 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3670004850 msgr2=0x7f3670004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:13.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.987+0000 7f368e8f8700 1 --2- 172.21.15.16:0/2597060971 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3670004850 0x7f3670004cd0 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f367c0017c0 tx=0x7f3670005880 comp rx=0 tx=0).stop 2023-10-16T12:29:13.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.987+0000 7f368e8f8700 1 -- 172.21.15.16:0/2597060971 shutdown_connections 2023-10-16T12:29:13.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.987+0000 7f368e8f8700 1 --2- 172.21.15.16:0/2597060971 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3670004850 0x7f3670004cd0 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:13.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.987+0000 7f368e8f8700 1 -- 172.21.15.16:0/2597060971 >> 172.21.15.16:0/2597060971 conn(0x7f368806f430 msgr2=0x7f368806f810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:13.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.987+0000 7f368e8f8700 1 -- 172.21.15.16:0/2597060971 shutdown_connections 2023-10-16T12:29:13.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.987+0000 7f368e8f8700 1 -- 172.21.15.16:0/2597060971 wait complete. 2023-10-16T12:29:13.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.988+0000 7f368e8f8700 1 Processor -- start 2023-10-16T12:29:14.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.989+0000 7f368e8f8700 1 -- start start 2023-10-16T12:29:14.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.989+0000 7f368e8f8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3670004850 0x7f36881f2620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:14.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.989+0000 7f368e8f8700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f368814d940 0x7f368814ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:14.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.989+0000 7f368e8f8700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f36881f3120 0x7f36881f35d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:14.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.989+0000 7f368e8f8700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f368814e300 con 0x7f3670004850 2023-10-16T12:29:14.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.989+0000 7f368e8f8700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36881f6330 con 0x7f368814d940 2023-10-16T12:29:14.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.989+0000 7f368e8f8700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36881f65d0 con 0x7f36881f3120 2023-10-16T12:29:14.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.989+0000 7f3687fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3670004850 0x7f36881f2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:14.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.989+0000 7f3687fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3670004850 0x7f36881f2620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43676/0 (socket says 172.21.15.16:43676) 2023-10-16T12:29:14.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.989+0000 7f3687fff700 1 -- 172.21.15.16:0/3190698731 learned_addr learned my addr 172.21.15.16:0/3190698731 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:29:14.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.990+0000 7f368ce95700 1 --2- 172.21.15.16:0/3190698731 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f36881f3120 0x7f36881f35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:14.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.990+0000 7f3687fff700 1 -- 172.21.15.16:0/3190698731 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f36881f3120 msgr2=0x7f36881f35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:14.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.990+0000 7f3687fff700 1 --2- 172.21.15.16:0/3190698731 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f36881f3120 0x7f36881f35d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:14.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.990+0000 7f3687fff700 1 -- 172.21.15.16:0/3190698731 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f368814d940 msgr2=0x7f368814ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:14.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.990+0000 7f3687fff700 1 --2- 172.21.15.16:0/3190698731 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f368814d940 0x7f368814ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:14.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.990+0000 7f3687fff700 1 -- 172.21.15.16:0/3190698731 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36881f68b0 con 0x7f3670004850 2023-10-16T12:29:14.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.990+0000 7f3687fff700 1 --2- 172.21.15.16:0/3190698731 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3670004850 0x7f36881f2620 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7f3678007290 tx=0x7f3678007650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:14.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.990+0000 7f3684ff9700 1 -- 172.21.15.16:0/3190698731 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36780014c0 con 0x7f3670004850 2023-10-16T12:29:14.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.990+0000 7f3684ff9700 1 -- 172.21.15.16:0/3190698731 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f367800f8d0 con 0x7f3670004850 2023-10-16T12:29:14.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.990+0000 7f368e8f8700 1 -- 172.21.15.16:0/3190698731 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36881f6b90 con 0x7f3670004850 2023-10-16T12:29:14.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.990+0000 7f3684ff9700 1 -- 172.21.15.16:0/3190698731 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f367800fc80 con 0x7f3670004850 2023-10-16T12:29:14.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.990+0000 7f368e8f8700 1 -- 172.21.15.16:0/3190698731 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36881f2b60 con 0x7f3670004850 2023-10-16T12:29:14.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.991+0000 7f368e8f8700 1 -- 172.21.15.16:0/3190698731 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f368804ec20 con 0x7f3670004850 2023-10-16T12:29:14.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.992+0000 7f3684ff9700 1 -- 172.21.15.16:0/3190698731 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 62) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f367801d460 con 0x7f3670004850 2023-10-16T12:29:14.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.992+0000 7f3684ff9700 1 --2- 172.21.15.16:0/3190698731 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f367406b020 0x7f367406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:14.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.992+0000 7f36877fe700 1 --2- 172.21.15.16:0/3190698731 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f367406b020 0x7f367406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:14.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.992+0000 7f3684ff9700 1 -- 172.21.15.16:0/3190698731 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f36780896a0 con 0x7f3670004850 2023-10-16T12:29:14.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.993+0000 7f36877fe700 1 --2- 172.21.15.16:0/3190698731 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f367406b020 0x7f367406d4e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f36880b61f0 tx=0x7f367c011000 comp rx=0 tx=0).ready entity=mgr.7884 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:14.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:13.997+0000 7f3684ff9700 1 -- 172.21.15.16:0/3190698731 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3678054830 con 0x7f3670004850 2023-10-16T12:29:14.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:14.200+0000 7f368e8f8700 1 -- 172.21.15.16:0/3190698731 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f36880b6260 con 0x7f3670004850 2023-10-16T12:29:14.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:14.363+0000 7f3684ff9700 1 -- 172.21.15.16:0/3190698731 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v125) v1 ==== 138+0+0 (secure 0 0 0) 0x7f3678057e50 con 0x7f3670004850 2023-10-16T12:29:14.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:14.366+0000 7f368e8f8700 1 -- 172.21.15.16:0/3190698731 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f367406b020 msgr2=0x7f367406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:14.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:14.366+0000 7f368e8f8700 1 --2- 172.21.15.16:0/3190698731 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f367406b020 0x7f367406d4e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f36880b61f0 tx=0x7f367c011000 comp rx=0 tx=0).stop 2023-10-16T12:29:14.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:14.366+0000 7f368e8f8700 1 -- 172.21.15.16:0/3190698731 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3670004850 msgr2=0x7f36881f2620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:14.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:14.366+0000 7f368e8f8700 1 --2- 172.21.15.16:0/3190698731 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3670004850 0x7f36881f2620 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7f3678007290 tx=0x7f3678007650 comp rx=0 tx=0).stop 2023-10-16T12:29:14.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:14.367+0000 7f368e8f8700 1 -- 172.21.15.16:0/3190698731 shutdown_connections 2023-10-16T12:29:14.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:14.367+0000 7f368e8f8700 1 --2- 172.21.15.16:0/3190698731 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f367406b020 0x7f367406d4e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:14.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:14.367+0000 7f368e8f8700 1 --2- 172.21.15.16:0/3190698731 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f36881f3120 0x7f36881f35d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:14.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:14.367+0000 7f368e8f8700 1 --2- 172.21.15.16:0/3190698731 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3670004850 0x7f36881f2620 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:14.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:14.367+0000 7f368e8f8700 1 --2- 172.21.15.16:0/3190698731 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f368814d940 0x7f368814ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:14.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:14.367+0000 7f368e8f8700 1 -- 172.21.15.16:0/3190698731 >> 172.21.15.16:0/3190698731 conn(0x7f368806f430 msgr2=0x7f36880cc980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:14.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:14.368+0000 7f368e8f8700 1 -- 172.21.15.16:0/3190698731 shutdown_connections 2023-10-16T12:29:14.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:14.368+0000 7f368e8f8700 1 -- 172.21.15.16:0/3190698731 wait complete. 2023-10-16T12:29:14.373 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' created 2023-10-16T12:29:14.385 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-16T12:29:14.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:14.515+0000 7fc076dc7700 1 Processor -- start 2023-10-16T12:29:14.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:14.516+0000 7fc076dc7700 1 -- start start 2023-10-16T12:29:14.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:14.516+0000 7fc076dc7700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fc07015f060 0x7fc0700cc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:14.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:14.516+0000 7fc076dc7700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0700b2440 con 0x7fc0700ce5d0 2023-10-16T12:29:14.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:14.516+0000 7fc076dc7700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0700d00e0 con 0x7fc07015ec80 2023-10-16T12:29:14.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:14.516+0000 7fc076dc7700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0700b2c50 con 0x7fc07015f060 2023-10-16T12:29:14.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:14.517+0000 7fc06ffff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc0700ce5d0 0x7fc0700cc190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:32960/0 (socket says 172.21.15.16:32960) 2023-10-16T12:29:14.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:14.517+0000 7fc06ffff700 1 -- 172.21.15.16:0/1921725406 learned_addr learned my addr 172.21.15.16:0/1921725406 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:29:14.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:14.517+0000 7fc075364700 1 --2- 172.21.15.16:0/1921725406 >> v2:172.21.15.49:3300/0 conn(0x7fc07015f060 0x7fc0700cc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:14.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:14.517+0000 7fc06f7fe700 1 -- 172.21.15.16:0/1921725406 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 814401192 0 0) 0x7fc0700b2440 con 0x7fc0700ce5d0 2023-10-16T12:29:14.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:14.517+0000 7fc06f7fe700 1 -- 172.21.15.16:0/1921725406 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0600035b0 con 0x7fc0700ce5d0 2023-10-16T12:29:14.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:14.517+0000 7fc075364700 1 -- 172.21.15.16:0/1921725406 >> v1:172.21.15.49:6789/0 conn(0x7fc07015ec80 legacy=0x7fc0700cdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:14.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:14.517+0000 7fc075364700 1 -- 172.21.15.16:0/1921725406 >> v1:172.21.15.16:6789/0 conn(0x7fc0700ce5d0 legacy=0x7fc0700cc190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:14.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:14.517+0000 7fc075364700 1 -- 172.21.15.16:0/1921725406 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0700cef60 con 0x7fc07015f060 2023-10-16T12:29:14.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:14.517+0000 7fc075364700 1 --2- 172.21.15.16:0/1921725406 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc07015f060 0x7fc0700cc8a0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fc068009dc0 tx=0x7fc06800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=60d66d3937e1690d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:14.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:14.517+0000 7fc06f7fe700 1 -- 172.21.15.16:0/1921725406 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc06800e070 con 0x7fc07015f060 2023-10-16T12:29:14.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:14.517+0000 7fc06f7fe700 1 -- 172.21.15.16:0/1921725406 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc06800bae0 con 0x7fc07015f060 2023-10-16T12:29:14.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:14.517+0000 7fc06f7fe700 1 -- 172.21.15.16:0/1921725406 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc06800be90 con 0x7fc07015f060 2023-10-16T12:29:14.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:14.518+0000 7fc076dc7700 1 -- 172.21.15.16:0/1921725406 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc07015f060 msgr2=0x7fc0700cc8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:14.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:14.518+0000 7fc076dc7700 1 --2- 172.21.15.16:0/1921725406 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc07015f060 0x7fc0700cc8a0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fc068009dc0 tx=0x7fc06800b330 comp rx=0 tx=0).stop 2023-10-16T12:29:14.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:14.518+0000 7fc076dc7700 1 -- 172.21.15.16:0/1921725406 shutdown_connections 2023-10-16T12:29:14.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:14.518+0000 7fc076dc7700 1 --2- 172.21.15.16:0/1921725406 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc07015f060 0x7fc0700cc8a0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:14.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:14.518+0000 7fc076dc7700 1 -- 172.21.15.16:0/1921725406 >> 172.21.15.16:0/1921725406 conn(0x7fc0700d15f0 msgr2=0x7fc0700d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:14.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:14.518+0000 7fc076dc7700 1 -- 172.21.15.16:0/1921725406 shutdown_connections 2023-10-16T12:29:14.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:14.518+0000 7fc076dc7700 1 -- 172.21.15.16:0/1921725406 wait complete. 2023-10-16T12:29:14.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:14.519+0000 7fc076dc7700 1 Processor -- start 2023-10-16T12:29:14.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:14.519+0000 7fc076dc7700 1 -- start start 2023-10-16T12:29:14.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:14.520+0000 7fc076dc7700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc0700ce5d0 0x7fc070154990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:14.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:14.520+0000 7fc076dc7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc07015ec80 0x7fc0700c6f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:14.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:14.520+0000 7fc076dc7700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc07015f060 0x7fc0700cad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:14.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:14.520+0000 7fc076dc7700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc070154ed0 con 0x7fc07015ec80 2023-10-16T12:29:14.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:14.520+0000 7fc076dc7700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0700c9c80 con 0x7fc0700ce5d0 2023-10-16T12:29:14.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:14.520+0000 7fc076dc7700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0700c9f60 con 0x7fc07015f060 2023-10-16T12:29:14.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:14.520+0000 7fc06ffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc07015ec80 0x7fc0700c6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:14.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:14.520+0000 7fc06ffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc07015ec80 0x7fc0700c6f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43692/0 (socket says 172.21.15.16:43692) 2023-10-16T12:29:14.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:14.520+0000 7fc06ffff700 1 -- 172.21.15.16:0/2656289554 learned_addr learned my addr 172.21.15.16:0/2656289554 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:29:14.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:14.520+0000 7fc075364700 1 --2- 172.21.15.16:0/2656289554 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc07015f060 0x7fc0700cad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:14.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:14.520+0000 7fc074b63700 1 --2- 172.21.15.16:0/2656289554 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc0700ce5d0 0x7fc070154990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:14.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:14.521+0000 7fc06ffff700 1 -- 172.21.15.16:0/2656289554 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc07015f060 msgr2=0x7fc0700cad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:14.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:14.521+0000 7fc06ffff700 1 --2- 172.21.15.16:0/2656289554 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc07015f060 0x7fc0700cad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:14.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:14.521+0000 7fc06ffff700 1 -- 172.21.15.16:0/2656289554 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc0700ce5d0 msgr2=0x7fc070154990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:14.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:14.521+0000 7fc06ffff700 1 --2- 172.21.15.16:0/2656289554 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc0700ce5d0 0x7fc070154990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:14.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:14.521+0000 7fc06ffff700 1 -- 172.21.15.16:0/2656289554 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc068009a70 con 0x7fc07015ec80 2023-10-16T12:29:14.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:14.521+0000 7fc06ffff700 1 --2- 172.21.15.16:0/2656289554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc07015ec80 0x7fc0700c6f60 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7fc05c00c700 tx=0x7fc05c00ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:14.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:14.521+0000 7fc06d7fa700 1 -- 172.21.15.16:0/2656289554 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc05c005d70 con 0x7fc07015ec80 2023-10-16T12:29:14.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:14.521+0000 7fc06d7fa700 1 -- 172.21.15.16:0/2656289554 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc05c0070d0 con 0x7fc07015ec80 2023-10-16T12:29:14.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:14.521+0000 7fc076dc7700 1 -- 172.21.15.16:0/2656289554 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0700ca240 con 0x7fc07015ec80 2023-10-16T12:29:14.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:14.521+0000 7fc06d7fa700 1 -- 172.21.15.16:0/2656289554 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc05c007460 con 0x7fc07015ec80 2023-10-16T12:29:14.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:14.521+0000 7fc076dc7700 1 -- 172.21.15.16:0/2656289554 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0700ca800 con 0x7fc07015ec80 2023-10-16T12:29:14.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:14.522+0000 7fc076dc7700 1 -- 172.21.15.16:0/2656289554 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc07004ebb0 con 0x7fc07015ec80 2023-10-16T12:29:14.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:14.523+0000 7fc06d7fa700 1 -- 172.21.15.16:0/2656289554 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 62) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc05c0075c0 con 0x7fc07015ec80 2023-10-16T12:29:14.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:14.523+0000 7fc06d7fa700 1 --2- 172.21.15.16:0/2656289554 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fc05406b020 0x7fc05406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:14.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:14.523+0000 7fc06d7fa700 1 -- 172.21.15.16:0/2656289554 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5575+0+0 (secure 0 0 0) 0x7fc05c088b70 con 0x7fc07015ec80 2023-10-16T12:29:14.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:14.523+0000 7fc074b63700 1 --2- 172.21.15.16:0/2656289554 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fc05406b020 0x7fc05406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:14.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:14.524+0000 7fc074b63700 1 --2- 172.21.15.16:0/2656289554 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fc05406b020 0x7fc05406d4e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc064007fd0 tx=0x7fc064009040 comp rx=0 tx=0).ready entity=mgr.7884 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:14.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:14.528+0000 7fc06d7fa700 1 -- 172.21.15.16:0/2656289554 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc05c053b90 con 0x7fc07015ec80 2023-10-16T12:29:14.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:14.731+0000 7fc076dc7700 1 -- 172.21.15.16:0/2656289554 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fc07015d6c0 con 0x7fc07015ec80 2023-10-16T12:29:15.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:15.368+0000 7fc06d7fa700 1 -- 172.21.15.16:0/2656289554 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v126) v1 ==== 170+0+0 (secure 0 0 0) 0x7fc05c017020 con 0x7fc07015ec80 2023-10-16T12:29:15.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:15.372+0000 7fc076dc7700 1 -- 172.21.15.16:0/2656289554 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fc05406b020 msgr2=0x7fc05406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:15.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:15.372+0000 7fc076dc7700 1 --2- 172.21.15.16:0/2656289554 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fc05406b020 0x7fc05406d4e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc064007fd0 tx=0x7fc064009040 comp rx=0 tx=0).stop 2023-10-16T12:29:15.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:15.372+0000 7fc076dc7700 1 -- 172.21.15.16:0/2656289554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc07015ec80 msgr2=0x7fc0700c6f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:15.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:15.372+0000 7fc076dc7700 1 --2- 172.21.15.16:0/2656289554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc07015ec80 0x7fc0700c6f60 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7fc05c00c700 tx=0x7fc05c00ca10 comp rx=0 tx=0).stop 2023-10-16T12:29:15.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:15.374+0000 7fc076dc7700 1 -- 172.21.15.16:0/2656289554 shutdown_connections 2023-10-16T12:29:15.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:15.374+0000 7fc076dc7700 1 --2- 172.21.15.16:0/2656289554 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fc05406b020 0x7fc05406d4e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:15.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:15.374+0000 7fc076dc7700 1 --2- 172.21.15.16:0/2656289554 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc07015f060 0x7fc0700cad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:15.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:15.374+0000 7fc076dc7700 1 --2- 172.21.15.16:0/2656289554 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc0700ce5d0 0x7fc070154990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:15.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:15.374+0000 7fc076dc7700 1 --2- 172.21.15.16:0/2656289554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc07015ec80 0x7fc0700c6f60 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:15.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:15.374+0000 7fc076dc7700 1 -- 172.21.15.16:0/2656289554 >> 172.21.15.16:0/2656289554 conn(0x7fc0700d15f0 msgr2=0x7fc07015dec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:15.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:15.374+0000 7fc076dc7700 1 -- 172.21.15.16:0/2656289554 shutdown_connections 2023-10-16T12:29:15.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:15.375+0000 7fc076dc7700 1 -- 172.21.15.16:0/2656289554 wait complete. 2023-10-16T12:29:15.389 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' created 2023-10-16T12:29:15.405 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-16T12:29:15.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:15.537+0000 7fa6f5eb4700 1 Processor -- start 2023-10-16T12:29:15.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:15.537+0000 7fa6f5eb4700 1 -- start start 2023-10-16T12:29:15.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:15.538+0000 7fa6f5eb4700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa6f015b2c0 0x7fa6f0161340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:15.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:15.538+0000 7fa6f5eb4700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa6f006e570 0x7fa6f015e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:15.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:15.538+0000 7fa6f5eb4700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6f0161880 con 0x7fa6f006a7e0 2023-10-16T12:29:15.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:15.538+0000 7fa6f5eb4700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6f015b6a0 con 0x7fa6f015b2c0 2023-10-16T12:29:15.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:15.538+0000 7fa6f5eb4700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f0161b20 con 0x7fa6f006e570 2023-10-16T12:29:15.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:15.538+0000 7fa6ef7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa6f015b2c0 0x7fa6f0161340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:15.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:15.538+0000 7fa6ef7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa6f015b2c0 0x7fa6f0161340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43698/0 (socket says 172.21.15.16:43698) 2023-10-16T12:29:15.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:15.538+0000 7fa6ef7fe700 1 -- 172.21.15.16:0/2168676946 learned_addr learned my addr 172.21.15.16:0/2168676946 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:29:15.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:15.538+0000 7fa6effff700 1 --1- 172.21.15.16:0/2168676946 >> v1:172.21.15.16:6789/0 conn(0x7fa6f006a7e0 0x7fa6f0071a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:32976/0 (socket says 172.21.15.16:32976) 2023-10-16T12:29:15.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:15.538+0000 7fa6eeffd700 1 --2- 172.21.15.16:0/2168676946 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa6f006e570 0x7fa6f015e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:15.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:15.539+0000 7fa6ef7fe700 1 -- 172.21.15.16:0/2168676946 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa6f006e570 msgr2=0x7fa6f015e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:29:15.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:15.539+0000 7fa6ef7fe700 1 --2- 172.21.15.16:0/2168676946 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa6f006e570 0x7fa6f015e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:15.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:15.539+0000 7fa6ef7fe700 1 -- 172.21.15.16:0/2168676946 >> v1:172.21.15.16:6789/0 conn(0x7fa6f006a7e0 legacy=0x7fa6f0071a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:15.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:15.539+0000 7fa6ef7fe700 1 -- 172.21.15.16:0/2168676946 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6f0161e00 con 0x7fa6f015b2c0 2023-10-16T12:29:15.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:15.539+0000 7fa6eeffd700 1 --2- 172.21.15.16:0/2168676946 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa6f006e570 0x7fa6f015e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:29:15.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:15.539+0000 7fa6ef7fe700 1 --2- 172.21.15.16:0/2168676946 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6f015b2c0 0x7fa6f0161340 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7fa6e00098f0 tx=0x7fa6e0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fa8d014a9485e34 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:15.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:15.539+0000 7fa6ee7fc700 1 -- 172.21.15.16:0/2168676946 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6e000b8a0 con 0x7fa6f015b2c0 2023-10-16T12:29:15.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:15.539+0000 7fa6ee7fc700 1 -- 172.21.15.16:0/2168676946 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6e000ba00 con 0x7fa6f015b2c0 2023-10-16T12:29:15.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:15.539+0000 7fa6ee7fc700 1 -- 172.21.15.16:0/2168676946 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6e000bd90 con 0x7fa6f015b2c0 2023-10-16T12:29:15.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:15.539+0000 7fa6f5eb4700 1 -- 172.21.15.16:0/2168676946 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6f015b2c0 msgr2=0x7fa6f0161340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:15.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:15.539+0000 7fa6f5eb4700 1 --2- 172.21.15.16:0/2168676946 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6f015b2c0 0x7fa6f0161340 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7fa6e00098f0 tx=0x7fa6e0009c00 comp rx=0 tx=0).stop 2023-10-16T12:29:15.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:15.539+0000 7fa6f5eb4700 1 -- 172.21.15.16:0/2168676946 shutdown_connections 2023-10-16T12:29:15.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:15.539+0000 7fa6f5eb4700 1 --2- 172.21.15.16:0/2168676946 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6f015b2c0 0x7fa6f0161340 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:15.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:15.539+0000 7fa6f5eb4700 1 --2- 172.21.15.16:0/2168676946 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa6f006e570 0x7fa6f015e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:15.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:15.539+0000 7fa6f5eb4700 1 -- 172.21.15.16:0/2168676946 >> 172.21.15.16:0/2168676946 conn(0x7fa6f00b1f40 msgr2=0x7fa6f00b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:15.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:15.540+0000 7fa6f5eb4700 1 -- 172.21.15.16:0/2168676946 shutdown_connections 2023-10-16T12:29:15.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:15.540+0000 7fa6f5eb4700 1 -- 172.21.15.16:0/2168676946 wait complete. 2023-10-16T12:29:15.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:15.541+0000 7fa6f5eb4700 1 Processor -- start 2023-10-16T12:29:15.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:15.541+0000 7fa6f5eb4700 1 -- start start 2023-10-16T12:29:15.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:15.542+0000 7fa6f5eb4700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa6f006a7e0 0x7fa6f01597f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:15.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:15.542+0000 7fa6f5eb4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6f006e570 0x7fa6f00c74c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:15.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:15.542+0000 7fa6f5eb4700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa6f015b2c0 0x7fa6f00cce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:15.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:15.542+0000 7fa6f5eb4700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f00c8ae0 con 0x7fa6f006e570 2023-10-16T12:29:15.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:15.542+0000 7fa6f5eb4700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f0159d30 con 0x7fa6f015b2c0 2023-10-16T12:29:15.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:15.542+0000 7fa6f5eb4700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f00c7f60 con 0x7fa6f006a7e0 2023-10-16T12:29:15.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:15.542+0000 7fa6eeffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6f006e570 0x7fa6f00c74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:15.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:15.542+0000 7fa6eeffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6f006e570 0x7fa6f00c74c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43706/0 (socket says 172.21.15.16:43706) 2023-10-16T12:29:15.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:15.542+0000 7fa6eeffd700 1 -- 172.21.15.16:0/493993009 learned_addr learned my addr 172.21.15.16:0/493993009 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:29:15.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:15.542+0000 7fa6effff700 1 --2- 172.21.15.16:0/493993009 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa6f015b2c0 0x7fa6f00cce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:15.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:15.542+0000 7fa6ef7fe700 1 --2- 172.21.15.16:0/493993009 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa6f006a7e0 0x7fa6f01597f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:15.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:15.542+0000 7fa6eeffd700 1 -- 172.21.15.16:0/493993009 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa6f006a7e0 msgr2=0x7fa6f01597f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:15.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:15.542+0000 7fa6eeffd700 1 --2- 172.21.15.16:0/493993009 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa6f006a7e0 0x7fa6f01597f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:15.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:15.542+0000 7fa6eeffd700 1 -- 172.21.15.16:0/493993009 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa6f015b2c0 msgr2=0x7fa6f00cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:15.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:15.542+0000 7fa6eeffd700 1 --2- 172.21.15.16:0/493993009 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa6f015b2c0 0x7fa6f00cce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:15.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:15.542+0000 7fa6eeffd700 1 -- 172.21.15.16:0/493993009 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6e0009550 con 0x7fa6f006e570 2023-10-16T12:29:15.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:15.542+0000 7fa6effff700 1 --2- 172.21.15.16:0/493993009 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa6f015b2c0 0x7fa6f00cce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:29:15.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:15.542+0000 7fa6eeffd700 1 --2- 172.21.15.16:0/493993009 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6f006e570 0x7fa6f00c74c0 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7fa6d800c750 tx=0x7fa6d800cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:15.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:15.542+0000 7fa6e7fff700 1 -- 172.21.15.16:0/493993009 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6d8011800 con 0x7fa6f006e570 2023-10-16T12:29:15.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:15.542+0000 7fa6e7fff700 1 -- 172.21.15.16:0/493993009 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6d8011960 con 0x7fa6f006e570 2023-10-16T12:29:15.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:15.542+0000 7fa6f5eb4700 1 -- 172.21.15.16:0/493993009 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6f00c8240 con 0x7fa6f006e570 2023-10-16T12:29:15.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:15.542+0000 7fa6e7fff700 1 -- 172.21.15.16:0/493993009 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6d8011cd0 con 0x7fa6f006e570 2023-10-16T12:29:15.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:15.543+0000 7fa6f5eb4700 1 -- 172.21.15.16:0/493993009 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6f00c8800 con 0x7fa6f006e570 2023-10-16T12:29:15.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:15.544+0000 7fa6f5eb4700 1 -- 172.21.15.16:0/493993009 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6f01f3060 con 0x7fa6f006e570 2023-10-16T12:29:15.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:15.544+0000 7fa6e7fff700 1 -- 172.21.15.16:0/493993009 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 62) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa6d8021350 con 0x7fa6f006e570 2023-10-16T12:29:15.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:15.544+0000 7fa6e7fff700 1 --2- 172.21.15.16:0/493993009 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fa6d006b020 0x7fa6d006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:15.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:15.545+0000 7fa6ef7fe700 1 --2- 172.21.15.16:0/493993009 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fa6d006b020 0x7fa6d006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:15.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:15.545+0000 7fa6e7fff700 1 -- 172.21.15.16:0/493993009 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5957+0+0 (secure 0 0 0) 0x7fa6d8089900 con 0x7fa6f006e570 2023-10-16T12:29:15.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:15.545+0000 7fa6ef7fe700 1 --2- 172.21.15.16:0/493993009 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fa6d006b020 0x7fa6d006d4e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa6e0009c40 tx=0x7fa6e0008030 comp rx=0 tx=0).ready entity=mgr.7884 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:15.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:15.549+0000 7fa6e7fff700 1 -- 172.21.15.16:0/493993009 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa6f01f3060 con 0x7fa6f006e570 2023-10-16T12:29:15.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:15.799+0000 7fa6f5eb4700 1 -- 172.21.15.16:0/493993009 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fa6f01f3060 con 0x7fa6f006e570 2023-10-16T12:29:16.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.369+0000 7fa6e7fff700 1 -- 172.21.15.16:0/493993009 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 15 and data pool 16 v57) v1 ==== 174+0+0 (secure 0 0 0) 0x7fa6f01f3060 con 0x7fa6f006e570 2023-10-16T12:29:16.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.373+0000 7fa6f5eb4700 1 -- 172.21.15.16:0/493993009 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fa6d006b020 msgr2=0x7fa6d006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:16.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.373+0000 7fa6f5eb4700 1 --2- 172.21.15.16:0/493993009 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fa6d006b020 0x7fa6d006d4e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa6e0009c40 tx=0x7fa6e0008030 comp rx=0 tx=0).stop 2023-10-16T12:29:16.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.373+0000 7fa6f5eb4700 1 -- 172.21.15.16:0/493993009 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6f006e570 msgr2=0x7fa6f00c74c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:16.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.373+0000 7fa6f5eb4700 1 --2- 172.21.15.16:0/493993009 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6f006e570 0x7fa6f00c74c0 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7fa6d800c750 tx=0x7fa6d800cb10 comp rx=0 tx=0).stop 2023-10-16T12:29:16.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.374+0000 7fa6f5eb4700 1 -- 172.21.15.16:0/493993009 shutdown_connections 2023-10-16T12:29:16.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.374+0000 7fa6f5eb4700 1 --2- 172.21.15.16:0/493993009 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa6f015b2c0 0x7fa6f00cce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:16.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.374+0000 7fa6f5eb4700 1 --2- 172.21.15.16:0/493993009 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fa6d006b020 0x7fa6d006d4e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:16.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.374+0000 7fa6f5eb4700 1 --2- 172.21.15.16:0/493993009 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa6f006a7e0 0x7fa6f01597f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:16.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.374+0000 7fa6f5eb4700 1 --2- 172.21.15.16:0/493993009 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6f006e570 0x7fa6f00c74c0 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:16.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.374+0000 7fa6f5eb4700 1 -- 172.21.15.16:0/493993009 >> 172.21.15.16:0/493993009 conn(0x7fa6f00b1f40 msgr2=0x7fa6f00d0030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:16.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.374+0000 7fa6f5eb4700 1 -- 172.21.15.16:0/493993009 shutdown_connections 2023-10-16T12:29:16.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.374+0000 7fa6f5eb4700 1 -- 172.21.15.16:0/493993009 wait complete. 2023-10-16T12:29:16.379 INFO:teuthology.orchestra.run.smithi016.stderr:new fs with metadata pool 15 and data pool 16 2023-10-16T12:29:16.393 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:29:16.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.519+0000 7f34ca013700 1 Processor -- start 2023-10-16T12:29:16.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.519+0000 7f34ca013700 1 -- start start 2023-10-16T12:29:16.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.520+0000 7f34ca013700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f34c415b280 0x7f34c41613e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:16.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.520+0000 7f34ca013700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f34c40cc8a0 0x7f34c415e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:16.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.520+0000 7f34ca013700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34c4161c30 con 0x7f34c406a7e0 2023-10-16T12:29:16.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.520+0000 7f34ca013700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34c4161f10 con 0x7f34c415b280 2023-10-16T12:29:16.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.520+0000 7f34ca013700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34c41621f0 con 0x7f34c40cc8a0 2023-10-16T12:29:16.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.520+0000 7f34c37fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f34c415b280 0x7f34c41613e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:16.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.520+0000 7f34c37fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f34c415b280 0x7f34c41613e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43726/0 (socket says 172.21.15.16:43726) 2023-10-16T12:29:16.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.520+0000 7f34c37fe700 1 -- 172.21.15.16:0/799945075 learned_addr learned my addr 172.21.15.16:0/799945075 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:29:16.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.520+0000 7f34c37fe700 1 -- 172.21.15.16:0/799945075 >> v2:172.21.15.49:3300/0 conn(0x7f34c40cc8a0 msgr2=0x7f34c415e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:29:16.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.520+0000 7f34c2ffd700 1 --2- 172.21.15.16:0/799945075 >> v2:172.21.15.49:3300/0 conn(0x7f34c40cc8a0 0x7f34c415e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:16.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.520+0000 7f34c37fe700 1 --2- 172.21.15.16:0/799945075 >> v2:172.21.15.49:3300/0 conn(0x7f34c40cc8a0 0x7f34c415e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:16.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.520+0000 7f34c37fe700 1 -- 172.21.15.16:0/799945075 >> v1:172.21.15.49:6789/0 conn(0x7f34c406a7e0 legacy=0x7f34c40d02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:29:16.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.520+0000 7f34c37fe700 1 -- 172.21.15.16:0/799945075 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34c415df00 con 0x7f34c415b280 2023-10-16T12:29:16.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.520+0000 7f34c2ffd700 1 --2- 172.21.15.16:0/799945075 >> v2:172.21.15.49:3300/0 conn(0x7f34c40cc8a0 0x7f34c415e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:29:16.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.520+0000 7f34c37fe700 1 --2- 172.21.15.16:0/799945075 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34c415b280 0x7f34c41613e0 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f34c4071970 tx=0x7f34b8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1797682bcc12fa92 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:16.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.520+0000 7f34c27fc700 1 -- 172.21.15.16:0/799945075 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34b8004230 con 0x7f34c415b280 2023-10-16T12:29:16.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.521+0000 7f34c27fc700 1 -- 172.21.15.16:0/799945075 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34b8004390 con 0x7f34c415b280 2023-10-16T12:29:16.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.521+0000 7f34c27fc700 1 -- 172.21.15.16:0/799945075 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34b80036a0 con 0x7f34c415b280 2023-10-16T12:29:16.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.521+0000 7f34ca013700 1 -- 172.21.15.16:0/799945075 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34c415b280 msgr2=0x7f34c41613e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:16.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.521+0000 7f34ca013700 1 --2- 172.21.15.16:0/799945075 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34c415b280 0x7f34c41613e0 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f34c4071970 tx=0x7f34b8009bd0 comp rx=0 tx=0).stop 2023-10-16T12:29:16.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.521+0000 7f34ca013700 1 -- 172.21.15.16:0/799945075 shutdown_connections 2023-10-16T12:29:16.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.521+0000 7f34ca013700 1 --2- 172.21.15.16:0/799945075 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34c415b280 0x7f34c41613e0 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:16.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.521+0000 7f34ca013700 1 --2- 172.21.15.16:0/799945075 >> v2:172.21.15.49:3300/0 conn(0x7f34c40cc8a0 0x7f34c415e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:16.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.521+0000 7f34ca013700 1 -- 172.21.15.16:0/799945075 >> 172.21.15.16:0/799945075 conn(0x7f34c40b1fd0 msgr2=0x7f34c40b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:16.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.521+0000 7f34ca013700 1 -- 172.21.15.16:0/799945075 shutdown_connections 2023-10-16T12:29:16.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.521+0000 7f34ca013700 1 -- 172.21.15.16:0/799945075 wait complete. 2023-10-16T12:29:16.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.522+0000 7f34ca013700 1 Processor -- start 2023-10-16T12:29:16.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.522+0000 7f34ca013700 1 -- start start 2023-10-16T12:29:16.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.523+0000 7f34ca013700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34c406a7e0 0x7f34c40c65a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:16.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.523+0000 7f34ca013700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f34c40cc8a0 0x7f34c414f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:16.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.523+0000 7f34ca013700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f34c415b280 0x7f34c40c49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:16.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.523+0000 7f34ca013700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34c40c6ae0 con 0x7f34c406a7e0 2023-10-16T12:29:16.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.523+0000 7f34ca013700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34c40c90b0 con 0x7f34c40cc8a0 2023-10-16T12:29:16.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.523+0000 7f34ca013700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34c40c9390 con 0x7f34c415b280 2023-10-16T12:29:16.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.523+0000 7f34c37fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34c406a7e0 0x7f34c40c65a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:16.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.523+0000 7f34c37fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34c406a7e0 0x7f34c40c65a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43730/0 (socket says 172.21.15.16:43730) 2023-10-16T12:29:16.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.523+0000 7f34c37fe700 1 -- 172.21.15.16:0/763813576 learned_addr learned my addr 172.21.15.16:0/763813576 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:29:16.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.523+0000 7f34c2ffd700 1 --2- 172.21.15.16:0/763813576 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f34c40cc8a0 0x7f34c414f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:16.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.524+0000 7f34c3fff700 1 --2- 172.21.15.16:0/763813576 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f34c415b280 0x7f34c40c49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:16.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.524+0000 7f34c37fe700 1 -- 172.21.15.16:0/763813576 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f34c415b280 msgr2=0x7f34c40c49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:16.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.524+0000 7f34c37fe700 1 --2- 172.21.15.16:0/763813576 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f34c415b280 0x7f34c40c49f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:16.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.524+0000 7f34c37fe700 1 -- 172.21.15.16:0/763813576 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f34c40cc8a0 msgr2=0x7f34c414f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:16.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.524+0000 7f34c37fe700 1 --2- 172.21.15.16:0/763813576 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f34c40cc8a0 0x7f34c414f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:16.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.524+0000 7f34c37fe700 1 -- 172.21.15.16:0/763813576 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34b8009550 con 0x7f34c406a7e0 2023-10-16T12:29:16.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.524+0000 7f34c2ffd700 1 --2- 172.21.15.16:0/763813576 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f34c40cc8a0 0x7f34c414f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:29:16.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.524+0000 7f34c3fff700 1 --2- 172.21.15.16:0/763813576 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f34c415b280 0x7f34c40c49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:29:16.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.524+0000 7f34c37fe700 1 --2- 172.21.15.16:0/763813576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34c406a7e0 0x7f34c40c65a0 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f34c40d1540 tx=0x7f34b8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:16.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.524+0000 7f34abfff700 1 -- 172.21.15.16:0/763813576 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34b8003d80 con 0x7f34c406a7e0 2023-10-16T12:29:16.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.524+0000 7f34abfff700 1 -- 172.21.15.16:0/763813576 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34b801f070 con 0x7f34c406a7e0 2023-10-16T12:29:16.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.524+0000 7f34ca013700 1 -- 172.21.15.16:0/763813576 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34c40c9670 con 0x7f34c406a7e0 2023-10-16T12:29:16.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.524+0000 7f34abfff700 1 -- 172.21.15.16:0/763813576 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34b8011870 con 0x7f34c406a7e0 2023-10-16T12:29:16.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.524+0000 7f34ca013700 1 -- 172.21.15.16:0/763813576 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34c414fda0 con 0x7f34c406a7e0 2023-10-16T12:29:16.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.525+0000 7f34ca013700 1 -- 172.21.15.16:0/763813576 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34c415b940 con 0x7f34c406a7e0 2023-10-16T12:29:16.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.526+0000 7f34abfff700 1 -- 172.21.15.16:0/763813576 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 62) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f34b8021030 con 0x7f34c406a7e0 2023-10-16T12:29:16.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.526+0000 7f34abfff700 1 --2- 172.21.15.16:0/763813576 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f34b006b020 0x7f34b006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:16.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.526+0000 7f34abfff700 1 -- 172.21.15.16:0/763813576 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f34b808af00 con 0x7f34c406a7e0 2023-10-16T12:29:16.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.526+0000 7f34c2ffd700 1 --2- 172.21.15.16:0/763813576 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f34b006b020 0x7f34b006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:16.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.526+0000 7f34c2ffd700 1 --2- 172.21.15.16:0/763813576 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f34b006b020 0x7f34b006d4e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f34c4072590 tx=0x7f34b4008040 comp rx=0 tx=0).ready entity=mgr.7884 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:16.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.531+0000 7f34abfff700 1 -- 172.21.15.16:0/763813576 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f34b8055ef0 con 0x7f34c406a7e0 2023-10-16T12:29:16.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.738+0000 7f34ca013700 1 -- 172.21.15.16:0/763813576 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f34c404ebb0 con 0x7f34c406a7e0 2023-10-16T12:29:16.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.739+0000 7f34abfff700 1 -- 172.21.15.16:0/763813576 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v127) v1 ==== 74+0+16664 (secure 0 0 0) 0x7f34b8088130 con 0x7f34c406a7e0 2023-10-16T12:29:16.741 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:29:16.742 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":127,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:29:15.801172+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:29:14.232554+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2023-10-16T12:29:14.764132+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:29:16.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.741+0000 7f34ca013700 1 -- 172.21.15.16:0/763813576 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f34b006b020 msgr2=0x7f34b006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:16.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.741+0000 7f34ca013700 1 --2- 172.21.15.16:0/763813576 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f34b006b020 0x7f34b006d4e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f34c4072590 tx=0x7f34b4008040 comp rx=0 tx=0).stop 2023-10-16T12:29:16.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.741+0000 7f34ca013700 1 -- 172.21.15.16:0/763813576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34c406a7e0 msgr2=0x7f34c40c65a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:16.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.741+0000 7f34ca013700 1 --2- 172.21.15.16:0/763813576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34c406a7e0 0x7f34c40c65a0 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f34c40d1540 tx=0x7f34b8004750 comp rx=0 tx=0).stop 2023-10-16T12:29:16.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.742+0000 7f34ca013700 1 -- 172.21.15.16:0/763813576 shutdown_connections 2023-10-16T12:29:16.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.742+0000 7f34ca013700 1 --2- 172.21.15.16:0/763813576 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f34b006b020 0x7f34b006d4e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:16.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.742+0000 7f34ca013700 1 --2- 172.21.15.16:0/763813576 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f34c415b280 0x7f34c40c49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:16.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.742+0000 7f34ca013700 1 --2- 172.21.15.16:0/763813576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34c406a7e0 0x7f34c40c65a0 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:16.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.742+0000 7f34ca013700 1 --2- 172.21.15.16:0/763813576 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f34c40cc8a0 0x7f34c414f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:16.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.742+0000 7f34ca013700 1 -- 172.21.15.16:0/763813576 >> 172.21.15.16:0/763813576 conn(0x7f34c40b1fd0 msgr2=0x7f34c40cb770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:16.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.742+0000 7f34ca013700 1 -- 172.21.15.16:0/763813576 shutdown_connections 2023-10-16T12:29:16.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.742+0000 7f34ca013700 1 -- 172.21.15.16:0/763813576 wait complete. 2023-10-16T12:29:16.756 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:29:16.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.879+0000 7f5ddb728700 1 Processor -- start 2023-10-16T12:29:16.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.879+0000 7f5ddb728700 1 -- start start 2023-10-16T12:29:16.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.880+0000 7f5ddb728700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f5dd415b630 0x7f5dd40d0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:16.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.880+0000 7f5ddb728700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5dd415ba10 0x7f5dd40cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:16.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.880+0000 7f5ddb728700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dd40b0970 con 0x7f5dd40cdde0 2023-10-16T12:29:16.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.880+0000 7f5ddb728700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dd40cd690 con 0x7f5dd415b630 2023-10-16T12:29:16.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.880+0000 7f5ddb728700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd40cd970 con 0x7f5dd415ba10 2023-10-16T12:29:16.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.880+0000 7f5dd94c4700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f5dd415b630 0x7f5dd40d0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:16.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.880+0000 7f5dd8cc3700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5dd415ba10 0x7f5dd40cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:16.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.880+0000 7f5dd94c4700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f5dd415b630 0x7f5dd40d0380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:42800/0 (socket says 172.21.15.16:42800) 2023-10-16T12:29:16.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.880+0000 7f5dd94c4700 1 -- 172.21.15.16:0/2327019538 learned_addr learned my addr 172.21.15.16:0/2327019538 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:29:16.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.880+0000 7f5dd9cc5700 1 --1- 172.21.15.16:0/2327019538 >> v1:172.21.15.49:6789/0 conn(0x7f5dd40cdde0 0x7f5dd40cc2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:40198/0 (socket says 172.21.15.16:40198) 2023-10-16T12:29:16.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.881+0000 7f5dd94c4700 1 -- 172.21.15.16:0/2327019538 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5dd415ba10 msgr2=0x7f5dd40cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:29:16.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.881+0000 7f5dd94c4700 1 --2- 172.21.15.16:0/2327019538 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5dd415ba10 0x7f5dd40cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:16.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.881+0000 7f5dd94c4700 1 -- 172.21.15.16:0/2327019538 >> v1:172.21.15.49:6789/0 conn(0x7f5dd40cdde0 legacy=0x7f5dd40cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:16.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.881+0000 7f5dd94c4700 1 -- 172.21.15.16:0/2327019538 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dd40b0c30 con 0x7f5dd415b630 2023-10-16T12:29:16.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.881+0000 7f5dd94c4700 1 --2- 172.21.15.16:0/2327019538 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5dd415b630 0x7f5dd40d0380 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f5dc80098f0 tx=0x7f5dc8009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e6666efc046ac750 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:16.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.881+0000 7f5dc7fff700 1 -- 172.21.15.16:0/2327019538 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dc8011830 con 0x7f5dd415b630 2023-10-16T12:29:16.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.881+0000 7f5dc7fff700 1 -- 172.21.15.16:0/2327019538 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dc8011990 con 0x7f5dd415b630 2023-10-16T12:29:16.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.881+0000 7f5dc7fff700 1 -- 172.21.15.16:0/2327019538 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dc8011d20 con 0x7f5dd415b630 2023-10-16T12:29:16.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.882+0000 7f5ddb728700 1 -- 172.21.15.16:0/2327019538 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5dd415b630 msgr2=0x7f5dd40d0380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:16.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.882+0000 7f5ddb728700 1 --2- 172.21.15.16:0/2327019538 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5dd415b630 0x7f5dd40d0380 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f5dc80098f0 tx=0x7f5dc8009c00 comp rx=0 tx=0).stop 2023-10-16T12:29:16.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.882+0000 7f5ddb728700 1 -- 172.21.15.16:0/2327019538 shutdown_connections 2023-10-16T12:29:16.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.882+0000 7f5ddb728700 1 --2- 172.21.15.16:0/2327019538 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5dd415b630 0x7f5dd40d0380 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:16.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.882+0000 7f5ddb728700 1 --2- 172.21.15.16:0/2327019538 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5dd415ba10 0x7f5dd40cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:16.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.882+0000 7f5ddb728700 1 -- 172.21.15.16:0/2327019538 >> 172.21.15.16:0/2327019538 conn(0x7f5dd40d14b0 msgr2=0x7f5dd40d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:16.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.882+0000 7f5ddb728700 1 -- 172.21.15.16:0/2327019538 shutdown_connections 2023-10-16T12:29:16.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.882+0000 7f5ddb728700 1 -- 172.21.15.16:0/2327019538 wait complete. 2023-10-16T12:29:16.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.883+0000 7f5ddb728700 1 Processor -- start 2023-10-16T12:29:16.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.883+0000 7f5ddb728700 1 -- start start 2023-10-16T12:29:16.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.884+0000 7f5ddb728700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5dd40cdde0 0x7f5dd41f06e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:16.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.884+0000 7f5ddb728700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5dd415b630 0x7f5dd40b6c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:16.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.884+0000 7f5ddb728700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5dd415ba10 0x7f5dd40c6da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:16.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.884+0000 7f5ddb728700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd40b61d0 con 0x7f5dd415b630 2023-10-16T12:29:16.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.884+0000 7f5ddb728700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd40b64b0 con 0x7f5dd415ba10 2023-10-16T12:29:16.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.884+0000 7f5ddb728700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd40b6790 con 0x7f5dd40cdde0 2023-10-16T12:29:16.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.884+0000 7f5dd8cc3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5dd415b630 0x7f5dd40b6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:16.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.884+0000 7f5dd8cc3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5dd415b630 0x7f5dd40b6c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43746/0 (socket says 172.21.15.16:43746) 2023-10-16T12:29:16.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.884+0000 7f5dd8cc3700 1 -- 172.21.15.16:0/544208267 learned_addr learned my addr 172.21.15.16:0/544208267 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:29:16.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.884+0000 7f5dd9cc5700 1 --2- 172.21.15.16:0/544208267 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5dd415ba10 0x7f5dd40c6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:16.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.884+0000 7f5dd94c4700 1 --2- 172.21.15.16:0/544208267 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5dd40cdde0 0x7f5dd41f06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:16.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.884+0000 7f5dd8cc3700 1 -- 172.21.15.16:0/544208267 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5dd40cdde0 msgr2=0x7f5dd41f06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:16.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.884+0000 7f5dd8cc3700 1 --2- 172.21.15.16:0/544208267 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5dd40cdde0 0x7f5dd41f06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:16.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.884+0000 7f5dd8cc3700 1 -- 172.21.15.16:0/544208267 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5dd415ba10 msgr2=0x7f5dd40c6da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:16.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.884+0000 7f5dd8cc3700 1 --2- 172.21.15.16:0/544208267 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5dd415ba10 0x7f5dd40c6da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:16.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.884+0000 7f5dd8cc3700 1 -- 172.21.15.16:0/544208267 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dc8009550 con 0x7f5dd415b630 2023-10-16T12:29:16.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.885+0000 7f5dd8cc3700 1 --2- 172.21.15.16:0/544208267 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5dd415b630 0x7f5dd40b6c90 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f5dd000e850 tx=0x7f5dd000eb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:16.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.885+0000 7f5dc5ffb700 1 -- 172.21.15.16:0/544208267 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dd00095b0 con 0x7f5dd415b630 2023-10-16T12:29:16.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.885+0000 7f5dc5ffb700 1 -- 172.21.15.16:0/544208267 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dd0013680 con 0x7f5dd415b630 2023-10-16T12:29:16.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.885+0000 7f5ddb728700 1 -- 172.21.15.16:0/544208267 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dd40c4fe0 con 0x7f5dd415b630 2023-10-16T12:29:16.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.885+0000 7f5dc5ffb700 1 -- 172.21.15.16:0/544208267 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dd0013a10 con 0x7f5dd415b630 2023-10-16T12:29:16.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.885+0000 7f5ddb728700 1 -- 172.21.15.16:0/544208267 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dd41f0c20 con 0x7f5dd415b630 2023-10-16T12:29:16.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.886+0000 7f5ddb728700 1 -- 172.21.15.16:0/544208267 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dd404ebb0 con 0x7f5dd415b630 2023-10-16T12:29:16.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.886+0000 7f5dc5ffb700 1 -- 172.21.15.16:0/544208267 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 62) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5dd0013b70 con 0x7f5dd415b630 2023-10-16T12:29:16.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.887+0000 7f5dc5ffb700 1 --2- 172.21.15.16:0/544208267 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f5db806b020 0x7f5db806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:16.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.887+0000 7f5dc5ffb700 1 -- 172.21.15.16:0/544208267 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f5dd00896f0 con 0x7f5dd415b630 2023-10-16T12:29:16.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.887+0000 7f5dd94c4700 1 --2- 172.21.15.16:0/544208267 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f5db806b020 0x7f5db806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:16.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.887+0000 7f5dd94c4700 1 --2- 172.21.15.16:0/544208267 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f5db806b020 0x7f5db806d4e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5dc8009c40 tx=0x7f5dc8008030 comp rx=0 tx=0).ready entity=mgr.7884 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:16.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:16.891+0000 7f5dc5ffb700 1 -- 172.21.15.16:0/544208267 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5dd0054530 con 0x7f5dd415b630 2023-10-16T12:29:17.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:17.097+0000 7f5ddb728700 1 -- 172.21.15.16:0/544208267 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5dd40b6310 con 0x7f5dd415b630 2023-10-16T12:29:17.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:17.099+0000 7f5dc5ffb700 1 -- 172.21.15.16:0/544208267 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v127) v1 ==== 74+0+16664 (secure 0 0 0) 0x7f5dd40b6310 con 0x7f5dd415b630 2023-10-16T12:29:17.101 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:29:17.101 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":127,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:29:15.801172+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:29:14.232554+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2023-10-16T12:29:14.764132+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:29:17.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:17.101+0000 7f5ddb728700 1 -- 172.21.15.16:0/544208267 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f5db806b020 msgr2=0x7f5db806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:17.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:17.101+0000 7f5ddb728700 1 --2- 172.21.15.16:0/544208267 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f5db806b020 0x7f5db806d4e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5dc8009c40 tx=0x7f5dc8008030 comp rx=0 tx=0).stop 2023-10-16T12:29:17.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:17.101+0000 7f5ddb728700 1 -- 172.21.15.16:0/544208267 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5dd415b630 msgr2=0x7f5dd40b6c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:17.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:17.101+0000 7f5ddb728700 1 --2- 172.21.15.16:0/544208267 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5dd415b630 0x7f5dd40b6c90 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f5dd000e850 tx=0x7f5dd000eb60 comp rx=0 tx=0).stop 2023-10-16T12:29:17.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:17.101+0000 7f5ddb728700 1 -- 172.21.15.16:0/544208267 shutdown_connections 2023-10-16T12:29:17.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:17.101+0000 7f5ddb728700 1 --2- 172.21.15.16:0/544208267 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5dd415ba10 0x7f5dd40c6da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:17.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:17.101+0000 7f5ddb728700 1 --2- 172.21.15.16:0/544208267 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f5db806b020 0x7f5db806d4e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:17.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:17.101+0000 7f5ddb728700 1 --2- 172.21.15.16:0/544208267 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5dd40cdde0 0x7f5dd41f06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:17.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:17.101+0000 7f5ddb728700 1 --2- 172.21.15.16:0/544208267 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5dd415b630 0x7f5dd40b6c90 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:17.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:17.101+0000 7f5ddb728700 1 -- 172.21.15.16:0/544208267 >> 172.21.15.16:0/544208267 conn(0x7f5dd40d14b0 msgr2=0x7f5dd40d1890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:17.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:17.101+0000 7f5ddb728700 1 -- 172.21.15.16:0/544208267 shutdown_connections 2023-10-16T12:29:17.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:17.101+0000 7f5ddb728700 1 -- 172.21.15.16:0/544208267 wait complete. 2023-10-16T12:29:17.115 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-16T12:29:17.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:17.242+0000 7f46d4096700 1 Processor -- start 2023-10-16T12:29:17.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:17.242+0000 7f46d4096700 1 -- start start 2023-10-16T12:29:17.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:17.243+0000 7f46d4096700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f46cc06e570 0x7f46cc15e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:17.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:17.243+0000 7f46d4096700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46cc15b6a0 con 0x7f46cc15b2c0 2023-10-16T12:29:17.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:17.243+0000 7f46d4096700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46cc161b20 con 0x7f46cc06a7e0 2023-10-16T12:29:17.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:17.243+0000 7f46d4096700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46cc161e00 con 0x7f46cc06e570 2023-10-16T12:29:17.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:17.243+0000 7f46d1631700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f46cc06e570 0x7f46cc15e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:17.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:17.243+0000 7f46d1631700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f46cc06e570 0x7f46cc15e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43754/0 (socket says 172.21.15.16:43754) 2023-10-16T12:29:17.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:17.243+0000 7f46d1e32700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f46cc15b2c0 0x7f46cc161340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33026/0 (socket says 172.21.15.16:33026) 2023-10-16T12:29:17.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:17.243+0000 7f46d1631700 1 -- 172.21.15.16:0/1898831247 learned_addr learned my addr 172.21.15.16:0/1898831247 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:29:17.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:17.243+0000 7f46d1631700 1 -- 172.21.15.16:0/1898831247 >> v1:172.21.15.49:6789/0 conn(0x7f46cc06a7e0 legacy=0x7f46cc071a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:29:17.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:17.243+0000 7f46d1631700 1 -- 172.21.15.16:0/1898831247 >> v1:172.21.15.16:6789/0 conn(0x7f46cc15b2c0 legacy=0x7f46cc161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:17.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:17.243+0000 7f46d1631700 1 -- 172.21.15.16:0/1898831247 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46cc1620e0 con 0x7f46cc06e570 2023-10-16T12:29:17.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:17.243+0000 7f46d1631700 1 --2- 172.21.15.16:0/1898831247 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46cc06e570 0x7f46cc15e5a0 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f46c80098f0 tx=0x7f46c8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a810b2b3a0483bf8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:17.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:17.244+0000 7f46d0e30700 1 -- 172.21.15.16:0/1898831247 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c8004230 con 0x7f46cc06e570 2023-10-16T12:29:17.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:17.244+0000 7f46d0e30700 1 -- 172.21.15.16:0/1898831247 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46c8004390 con 0x7f46cc06e570 2023-10-16T12:29:17.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:17.244+0000 7f46d0e30700 1 -- 172.21.15.16:0/1898831247 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c80036a0 con 0x7f46cc06e570 2023-10-16T12:29:17.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:17.244+0000 7f46d4096700 1 -- 172.21.15.16:0/1898831247 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46cc06e570 msgr2=0x7f46cc15e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:17.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:17.244+0000 7f46d4096700 1 --2- 172.21.15.16:0/1898831247 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46cc06e570 0x7f46cc15e5a0 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f46c80098f0 tx=0x7f46c8009c00 comp rx=0 tx=0).stop 2023-10-16T12:29:17.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:17.244+0000 7f46d4096700 1 -- 172.21.15.16:0/1898831247 shutdown_connections 2023-10-16T12:29:17.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:17.244+0000 7f46d4096700 1 --2- 172.21.15.16:0/1898831247 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46cc06e570 0x7f46cc15e5a0 unknown :-1 s=CLOSED pgs=1372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:17.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:17.244+0000 7f46d4096700 1 -- 172.21.15.16:0/1898831247 >> 172.21.15.16:0/1898831247 conn(0x7f46cc0b1f40 msgr2=0x7f46cc0b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:17.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:17.244+0000 7f46d4096700 1 -- 172.21.15.16:0/1898831247 shutdown_connections 2023-10-16T12:29:17.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:17.244+0000 7f46d4096700 1 -- 172.21.15.16:0/1898831247 wait complete. 2023-10-16T12:29:17.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:17.246+0000 7f46d4096700 1 Processor -- start 2023-10-16T12:29:17.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:17.246+0000 7f46d4096700 1 -- start start 2023-10-16T12:29:17.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:17.247+0000 7f46d4096700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f46cc06a7e0 0x7f46cc14c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:17.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:17.247+0000 7f46d4096700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f46cc06e570 0x7f46cc0c74c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:17.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:17.247+0000 7f46d4096700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46cc15b2c0 0x7f46cc0b4500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:17.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:17.247+0000 7f46d4096700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46cc14cd70 con 0x7f46cc15b2c0 2023-10-16T12:29:17.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:17.247+0000 7f46d4096700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46cc0c7f40 con 0x7f46cc06e570 2023-10-16T12:29:17.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:17.247+0000 7f46d4096700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46cc0c8220 con 0x7f46cc06a7e0 2023-10-16T12:29:17.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:17.247+0000 7f46d2633700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46cc15b2c0 0x7f46cc0b4500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:17.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:17.247+0000 7f46d2633700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46cc15b2c0 0x7f46cc0b4500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43762/0 (socket says 172.21.15.16:43762) 2023-10-16T12:29:17.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:17.247+0000 7f46d2633700 1 -- 172.21.15.16:0/4115901000 learned_addr learned my addr 172.21.15.16:0/4115901000 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:29:17.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:17.247+0000 7f46d1e32700 1 --2- 172.21.15.16:0/4115901000 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f46cc06a7e0 0x7f46cc14c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:17.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:17.247+0000 7f46d1631700 1 --2- 172.21.15.16:0/4115901000 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f46cc06e570 0x7f46cc0c74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:17.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:17.247+0000 7f46d2633700 1 -- 172.21.15.16:0/4115901000 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f46cc06a7e0 msgr2=0x7f46cc14c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:17.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:17.247+0000 7f46d2633700 1 --2- 172.21.15.16:0/4115901000 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f46cc06a7e0 0x7f46cc14c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:17.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:17.247+0000 7f46d2633700 1 -- 172.21.15.16:0/4115901000 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f46cc06e570 msgr2=0x7f46cc0c74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:17.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:17.247+0000 7f46d2633700 1 --2- 172.21.15.16:0/4115901000 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f46cc06e570 0x7f46cc0c74c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:17.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:17.247+0000 7f46d2633700 1 -- 172.21.15.16:0/4115901000 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46c8009550 con 0x7f46cc15b2c0 2023-10-16T12:29:17.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:17.247+0000 7f46d1e32700 1 --2- 172.21.15.16:0/4115901000 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f46cc06a7e0 0x7f46cc14c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:29:17.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:17.248+0000 7f46d2633700 1 --2- 172.21.15.16:0/4115901000 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46cc15b2c0 0x7f46cc0b4500 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f46c400b990 tx=0x7f46c400bca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:17.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:17.248+0000 7f46c27fc700 1 -- 172.21.15.16:0/4115901000 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c400c930 con 0x7f46cc15b2c0 2023-10-16T12:29:17.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:17.248+0000 7f46c27fc700 1 -- 172.21.15.16:0/4115901000 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46c400ca90 con 0x7f46cc15b2c0 2023-10-16T12:29:17.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:17.248+0000 7f46d4096700 1 -- 172.21.15.16:0/4115901000 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46cc0c8500 con 0x7f46cc15b2c0 2023-10-16T12:29:17.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:17.248+0000 7f46c27fc700 1 -- 172.21.15.16:0/4115901000 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c400ce40 con 0x7f46cc15b2c0 2023-10-16T12:29:17.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:17.248+0000 7f46d4096700 1 -- 172.21.15.16:0/4115901000 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46cc0c8ac0 con 0x7f46cc15b2c0 2023-10-16T12:29:17.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:17.249+0000 7f46d4096700 1 -- 172.21.15.16:0/4115901000 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46cc0c8cb0 con 0x7f46cc15b2c0 2023-10-16T12:29:17.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:17.249+0000 7f46c27fc700 1 -- 172.21.15.16:0/4115901000 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 62) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f46c4006d50 con 0x7f46cc15b2c0 2023-10-16T12:29:17.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:17.250+0000 7f46c27fc700 1 --2- 172.21.15.16:0/4115901000 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f46b006b020 0x7f46b006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:17.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:17.250+0000 7f46c27fc700 1 -- 172.21.15.16:0/4115901000 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f46c40893b0 con 0x7f46cc15b2c0 2023-10-16T12:29:17.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:17.250+0000 7f46d1e32700 1 --2- 172.21.15.16:0/4115901000 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f46b006b020 0x7f46b006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:17.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:17.250+0000 7f46d1e32700 1 --2- 172.21.15.16:0/4115901000 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f46b006b020 0x7f46b006d4e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f46bc000f60 tx=0x7f46bc007df0 comp rx=0 tx=0).ready entity=mgr.7884 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:17.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:17.254+0000 7f46c27fc700 1 -- 172.21.15.16:0/4115901000 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f46c4054170 con 0x7f46cc15b2c0 2023-10-16T12:29:17.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:17.516+0000 7f46d4096700 1 -- 172.21.15.16:0/4115901000 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f46cc04ebb0 con 0x7f46cc15b2c0 2023-10-16T12:29:18.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:18.385+0000 7f46c27fc700 1 -- 172.21.15.16:0/4115901000 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v60) v1 ==== 118+0+0 (secure 0 0 0) 0x7f46c4057790 con 0x7f46cc15b2c0 2023-10-16T12:29:18.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:18.388+0000 7f46d4096700 1 -- 172.21.15.16:0/4115901000 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f46b006b020 msgr2=0x7f46b006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:18.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:18.388+0000 7f46d4096700 1 --2- 172.21.15.16:0/4115901000 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f46b006b020 0x7f46b006d4e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f46bc000f60 tx=0x7f46bc007df0 comp rx=0 tx=0).stop 2023-10-16T12:29:18.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:18.388+0000 7f46d4096700 1 -- 172.21.15.16:0/4115901000 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46cc15b2c0 msgr2=0x7f46cc0b4500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:18.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:18.388+0000 7f46d4096700 1 --2- 172.21.15.16:0/4115901000 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46cc15b2c0 0x7f46cc0b4500 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f46c400b990 tx=0x7f46c400bca0 comp rx=0 tx=0).stop 2023-10-16T12:29:18.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:18.388+0000 7f46d4096700 1 -- 172.21.15.16:0/4115901000 shutdown_connections 2023-10-16T12:29:18.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:18.388+0000 7f46d4096700 1 --2- 172.21.15.16:0/4115901000 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46cc15b2c0 0x7f46cc0b4500 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:18.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:18.388+0000 7f46d4096700 1 --2- 172.21.15.16:0/4115901000 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f46b006b020 0x7f46b006d4e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:18.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:18.388+0000 7f46d4096700 1 --2- 172.21.15.16:0/4115901000 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f46cc06a7e0 0x7f46cc14c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:18.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:18.388+0000 7f46d4096700 1 --2- 172.21.15.16:0/4115901000 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f46cc06e570 0x7f46cc0c74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:18.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:18.388+0000 7f46d4096700 1 -- 172.21.15.16:0/4115901000 >> 172.21.15.16:0/4115901000 conn(0x7f46cc0b1f40 msgr2=0x7f46cc0cc480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:18.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:18.388+0000 7f46d4096700 1 -- 172.21.15.16:0/4115901000 shutdown_connections 2023-10-16T12:29:18.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:18.388+0000 7f46d4096700 1 -- 172.21.15.16:0/4115901000 wait complete. 2023-10-16T12:29:18.405 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-16T12:29:18.405 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-16T12:29:18.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:18.538+0000 7f6d4dfdf700 1 Processor -- start 2023-10-16T12:29:18.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:18.538+0000 7f6d4dfdf700 1 -- start start 2023-10-16T12:29:18.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:18.539+0000 7f6d4dfdf700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6d480cb880 0x7f6d480cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:18.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:18.539+0000 7f6d4dfdf700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f6d4806da60 0x7f6d4806def0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:18.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:18.539+0000 7f6d4dfdf700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d480cef70 con 0x7f6d4815b070 2023-10-16T12:29:18.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:18.539+0000 7f6d4dfdf700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d4806e430 con 0x7f6d480cb880 2023-10-16T12:29:18.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:18.539+0000 7f6d4dfdf700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d480ce360 con 0x7f6d4806da60 2023-10-16T12:29:18.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:18.539+0000 7f6d46ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6d480cb880 0x7f6d480cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:18.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:18.539+0000 7f6d46ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6d480cb880 0x7f6d480cbd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43768/0 (socket says 172.21.15.16:43768) 2023-10-16T12:29:18.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:18.539+0000 7f6d477fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f6d4815b070 0x7f6d4815b450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33040/0 (socket says 172.21.15.16:33040) 2023-10-16T12:29:18.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:18.539+0000 7f6d46ffd700 1 -- 172.21.15.16:0/2315796444 learned_addr learned my addr 172.21.15.16:0/2315796444 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:29:18.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:18.540+0000 7f6d46ffd700 1 -- 172.21.15.16:0/2315796444 >> v2:172.21.15.49:3300/0 conn(0x7f6d4806da60 msgr2=0x7f6d4806def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:29:18.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:18.540+0000 7f6d47fff700 1 --2- 172.21.15.16:0/2315796444 >> v2:172.21.15.49:3300/0 conn(0x7f6d4806da60 0x7f6d4806def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:18.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:18.540+0000 7f6d46ffd700 1 --2- 172.21.15.16:0/2315796444 >> v2:172.21.15.49:3300/0 conn(0x7f6d4806da60 0x7f6d4806def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:18.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:18.540+0000 7f6d467fc700 1 -- 172.21.15.16:0/2315796444 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3225901061 0 0) 0x7f6d480cef70 con 0x7f6d4815b070 2023-10-16T12:29:18.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:18.540+0000 7f6d46ffd700 1 -- 172.21.15.16:0/2315796444 >> v1:172.21.15.16:6789/0 conn(0x7f6d4815b070 legacy=0x7f6d4815b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:18.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:18.540+0000 7f6d46ffd700 1 -- 172.21.15.16:0/2315796444 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d480ce640 con 0x7f6d480cb880 2023-10-16T12:29:18.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:18.540+0000 7f6d46ffd700 1 --2- 172.21.15.16:0/2315796444 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d480cb880 0x7f6d480cbd00 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f6d4806edc0 tx=0x7f6d30009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea3dcd23d5da392b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:18.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:18.540+0000 7f6d467fc700 1 -- 172.21.15.16:0/2315796444 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d3000b870 con 0x7f6d480cb880 2023-10-16T12:29:18.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:18.540+0000 7f6d467fc700 1 -- 172.21.15.16:0/2315796444 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d3000b9d0 con 0x7f6d480cb880 2023-10-16T12:29:18.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:18.540+0000 7f6d467fc700 1 -- 172.21.15.16:0/2315796444 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d3000bd60 con 0x7f6d480cb880 2023-10-16T12:29:18.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:18.541+0000 7f6d4dfdf700 1 -- 172.21.15.16:0/2315796444 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d480cb880 msgr2=0x7f6d480cbd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:18.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:18.541+0000 7f6d4dfdf700 1 --2- 172.21.15.16:0/2315796444 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d480cb880 0x7f6d480cbd00 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f6d4806edc0 tx=0x7f6d30009bd0 comp rx=0 tx=0).stop 2023-10-16T12:29:18.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:18.541+0000 7f6d4dfdf700 1 -- 172.21.15.16:0/2315796444 shutdown_connections 2023-10-16T12:29:18.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:18.541+0000 7f6d4dfdf700 1 --2- 172.21.15.16:0/2315796444 >> v2:172.21.15.49:3300/0 conn(0x7f6d4806da60 0x7f6d4806def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:18.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:18.541+0000 7f6d4dfdf700 1 --2- 172.21.15.16:0/2315796444 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d480cb880 0x7f6d480cbd00 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:18.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:18.541+0000 7f6d4dfdf700 1 -- 172.21.15.16:0/2315796444 >> 172.21.15.16:0/2315796444 conn(0x7f6d480b1f40 msgr2=0x7f6d480b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:18.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:18.541+0000 7f6d4dfdf700 1 -- 172.21.15.16:0/2315796444 shutdown_connections 2023-10-16T12:29:18.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:18.541+0000 7f6d4dfdf700 1 -- 172.21.15.16:0/2315796444 wait complete. 2023-10-16T12:29:18.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:18.542+0000 7f6d4dfdf700 1 Processor -- start 2023-10-16T12:29:18.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:18.542+0000 7f6d4dfdf700 1 -- start start 2023-10-16T12:29:18.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:18.543+0000 7f6d4dfdf700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d4806da60 0x7f6d48154510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:18.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:18.543+0000 7f6d4dfdf700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6d480cb880 0x7f6d480ad9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:18.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:18.543+0000 7f6d4dfdf700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6d4815b070 0x7f6d4815bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:18.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:18.543+0000 7f6d4dfdf700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d481f67b0 con 0x7f6d4806da60 2023-10-16T12:29:18.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:18.543+0000 7f6d4dfdf700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d48154a50 con 0x7f6d480cb880 2023-10-16T12:29:18.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:18.543+0000 7f6d4dfdf700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d48155650 con 0x7f6d4815b070 2023-10-16T12:29:18.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:18.543+0000 7f6d477fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d4806da60 0x7f6d48154510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:18.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:18.543+0000 7f6d477fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d4806da60 0x7f6d48154510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43774/0 (socket says 172.21.15.16:43774) 2023-10-16T12:29:18.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:18.543+0000 7f6d477fe700 1 -- 172.21.15.16:0/1397322527 learned_addr learned my addr 172.21.15.16:0/1397322527 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:29:18.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:18.543+0000 7f6d46ffd700 1 --2- 172.21.15.16:0/1397322527 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6d480cb880 0x7f6d480ad9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:18.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:18.543+0000 7f6d47fff700 1 --2- 172.21.15.16:0/1397322527 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6d4815b070 0x7f6d4815bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:18.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:18.543+0000 7f6d477fe700 1 -- 172.21.15.16:0/1397322527 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6d4815b070 msgr2=0x7f6d4815bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:18.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:18.543+0000 7f6d477fe700 1 --2- 172.21.15.16:0/1397322527 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6d4815b070 0x7f6d4815bdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:18.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:18.543+0000 7f6d477fe700 1 -- 172.21.15.16:0/1397322527 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6d480cb880 msgr2=0x7f6d480ad9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:18.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:18.543+0000 7f6d477fe700 1 --2- 172.21.15.16:0/1397322527 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6d480cb880 0x7f6d480ad9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:18.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:18.543+0000 7f6d477fe700 1 -- 172.21.15.16:0/1397322527 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d30009550 con 0x7f6d4806da60 2023-10-16T12:29:18.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:18.543+0000 7f6d47fff700 1 --2- 172.21.15.16:0/1397322527 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6d4815b070 0x7f6d4815bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:29:18.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:18.544+0000 7f6d477fe700 1 --2- 172.21.15.16:0/1397322527 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d4806da60 0x7f6d48154510 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f6d380011e0 tx=0x7f6d3800ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:18.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:18.544+0000 7f6d3ffff700 1 -- 172.21.15.16:0/1397322527 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d38004c80 con 0x7f6d4806da60 2023-10-16T12:29:18.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:18.544+0000 7f6d3ffff700 1 -- 172.21.15.16:0/1397322527 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d38004de0 con 0x7f6d4806da60 2023-10-16T12:29:18.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:18.544+0000 7f6d3ffff700 1 -- 172.21.15.16:0/1397322527 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d38007310 con 0x7f6d4806da60 2023-10-16T12:29:18.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:18.544+0000 7f6d4dfdf700 1 -- 172.21.15.16:0/1397322527 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d480ae3b0 con 0x7f6d4806da60 2023-10-16T12:29:18.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:18.544+0000 7f6d4dfdf700 1 -- 172.21.15.16:0/1397322527 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d481f6160 con 0x7f6d4806da60 2023-10-16T12:29:18.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:18.545+0000 7f6d4dfdf700 1 -- 172.21.15.16:0/1397322527 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d4804ebb0 con 0x7f6d4806da60 2023-10-16T12:29:18.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:18.545+0000 7f6d3ffff700 1 -- 172.21.15.16:0/1397322527 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 62) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6d38007470 con 0x7f6d4806da60 2023-10-16T12:29:18.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:18.546+0000 7f6d3ffff700 1 --2- 172.21.15.16:0/1397322527 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f6d2806b020 0x7f6d2806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:18.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:18.546+0000 7f6d3ffff700 1 -- 172.21.15.16:0/1397322527 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f6d380896b0 con 0x7f6d4806da60 2023-10-16T12:29:18.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:18.546+0000 7f6d46ffd700 1 --2- 172.21.15.16:0/1397322527 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f6d2806b020 0x7f6d2806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:18.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:18.546+0000 7f6d46ffd700 1 --2- 172.21.15.16:0/1397322527 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f6d2806b020 0x7f6d2806d4e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6d30000c00 tx=0x7f6d30008030 comp rx=0 tx=0).ready entity=mgr.7884 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:18.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:18.550+0000 7f6d3ffff700 1 -- 172.21.15.16:0/1397322527 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6d38054450 con 0x7f6d4806da60 2023-10-16T12:29:18.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:18.805+0000 7f6d4dfdf700 1 -- 172.21.15.16:0/1397322527 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f6d480cbeb0 con 0x7f6d4806da60 2023-10-16T12:29:19.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.390+0000 7f6d3ffff700 1 -- 172.21.15.16:0/1397322527 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v61) v1 ==== 122+0+0 (secure 0 0 0) 0x7f6d38057a70 con 0x7f6d4806da60 2023-10-16T12:29:19.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.393+0000 7f6d4dfdf700 1 -- 172.21.15.16:0/1397322527 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f6d2806b020 msgr2=0x7f6d2806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:19.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.393+0000 7f6d4dfdf700 1 --2- 172.21.15.16:0/1397322527 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f6d2806b020 0x7f6d2806d4e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6d30000c00 tx=0x7f6d30008030 comp rx=0 tx=0).stop 2023-10-16T12:29:19.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.393+0000 7f6d4dfdf700 1 -- 172.21.15.16:0/1397322527 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d4806da60 msgr2=0x7f6d48154510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:19.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.393+0000 7f6d4dfdf700 1 --2- 172.21.15.16:0/1397322527 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d4806da60 0x7f6d48154510 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f6d380011e0 tx=0x7f6d3800ca30 comp rx=0 tx=0).stop 2023-10-16T12:29:19.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.393+0000 7f6d4dfdf700 1 -- 172.21.15.16:0/1397322527 shutdown_connections 2023-10-16T12:29:19.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.393+0000 7f6d4dfdf700 1 --2- 172.21.15.16:0/1397322527 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f6d2806b020 0x7f6d2806d4e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:19.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.393+0000 7f6d4dfdf700 1 --2- 172.21.15.16:0/1397322527 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6d4815b070 0x7f6d4815bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:19.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.393+0000 7f6d4dfdf700 1 --2- 172.21.15.16:0/1397322527 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d4806da60 0x7f6d48154510 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:19.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.393+0000 7f6d4dfdf700 1 --2- 172.21.15.16:0/1397322527 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6d480cb880 0x7f6d480ad9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:19.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.393+0000 7f6d4dfdf700 1 -- 172.21.15.16:0/1397322527 >> 172.21.15.16:0/1397322527 conn(0x7f6d480b1f40 msgr2=0x7f6d4815e410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:19.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.393+0000 7f6d4dfdf700 1 -- 172.21.15.16:0/1397322527 shutdown_connections 2023-10-16T12:29:19.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.393+0000 7f6d4dfdf700 1 -- 172.21.15.16:0/1397322527 wait complete. 2023-10-16T12:29:19.410 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-16T12:29:19.411 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:29:19.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.541+0000 7f0e5354b700 1 Processor -- start 2023-10-16T12:29:19.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.542+0000 7f0e5354b700 1 -- start start 2023-10-16T12:29:19.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.543+0000 7f0e5354b700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f0e4c0cb190 0x7f0e4c0ae520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:19.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.543+0000 7f0e5354b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0e4c15e0b0 0x7f0e4c0aea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:19.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.543+0000 7f0e5354b700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e4c06b1f0 con 0x7f0e4c159c90 2023-10-16T12:29:19.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.543+0000 7f0e5354b700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e4c06b4b0 con 0x7f0e4c15e0b0 2023-10-16T12:29:19.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.543+0000 7f0e5354b700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e4c06b790 con 0x7f0e4c0cb190 2023-10-16T12:29:19.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.543+0000 7f0e51ae8700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0e4c15e0b0 0x7f0e4c0aea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:19.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.543+0000 7f0e51ae8700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0e4c15e0b0 0x7f0e4c0aea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43784/0 (socket says 172.21.15.16:43784) 2023-10-16T12:29:19.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.543+0000 7f0e51ae8700 1 -- 172.21.15.16:0/3939034338 learned_addr learned my addr 172.21.15.16:0/3939034338 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:29:19.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.543+0000 7f0e51ae8700 1 -- 172.21.15.16:0/3939034338 >> v2:172.21.15.49:3300/0 conn(0x7f0e4c0cb190 msgr2=0x7f0e4c0ae520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:29:19.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.543+0000 7f0e51ae8700 1 --2- 172.21.15.16:0/3939034338 >> v2:172.21.15.49:3300/0 conn(0x7f0e4c0cb190 0x7f0e4c0ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:19.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.543+0000 7f0e51ae8700 1 -- 172.21.15.16:0/3939034338 >> v1:172.21.15.49:6789/0 conn(0x7f0e4c159c90 legacy=0x7f0e4c15d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:29:19.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.543+0000 7f0e51ae8700 1 -- 172.21.15.16:0/3939034338 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e4c06ba70 con 0x7f0e4c15e0b0 2023-10-16T12:29:19.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.543+0000 7f0e51ae8700 1 --2- 172.21.15.16:0/3939034338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e4c15e0b0 0x7f0e4c0aea60 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c0cd620 tx=0x7f0e4400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb7c8f5b6f61170c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:19.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.543+0000 7f0e43fff700 1 -- 172.21.15.16:0/3939034338 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e4400e070 con 0x7f0e4c15e0b0 2023-10-16T12:29:19.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.544+0000 7f0e43fff700 1 -- 172.21.15.16:0/3939034338 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e44003a20 con 0x7f0e4c15e0b0 2023-10-16T12:29:19.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.544+0000 7f0e43fff700 1 -- 172.21.15.16:0/3939034338 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e44003dd0 con 0x7f0e4c15e0b0 2023-10-16T12:29:19.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.544+0000 7f0e5354b700 1 -- 172.21.15.16:0/3939034338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e4c15e0b0 msgr2=0x7f0e4c0aea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:19.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.544+0000 7f0e5354b700 1 --2- 172.21.15.16:0/3939034338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e4c15e0b0 0x7f0e4c0aea60 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c0cd620 tx=0x7f0e4400b330 comp rx=0 tx=0).stop 2023-10-16T12:29:19.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.544+0000 7f0e5354b700 1 -- 172.21.15.16:0/3939034338 shutdown_connections 2023-10-16T12:29:19.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.544+0000 7f0e5354b700 1 --2- 172.21.15.16:0/3939034338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e4c15e0b0 0x7f0e4c0aea60 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:19.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.544+0000 7f0e5354b700 1 --2- 172.21.15.16:0/3939034338 >> v2:172.21.15.49:3300/0 conn(0x7f0e4c0cb190 0x7f0e4c0ae520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:19.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.544+0000 7f0e5354b700 1 -- 172.21.15.16:0/3939034338 >> 172.21.15.16:0/3939034338 conn(0x7f0e4c0cc5b0 msgr2=0x7f0e4c0cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:19.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.544+0000 7f0e5354b700 1 -- 172.21.15.16:0/3939034338 shutdown_connections 2023-10-16T12:29:19.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.544+0000 7f0e5354b700 1 -- 172.21.15.16:0/3939034338 wait complete. 2023-10-16T12:29:19.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.546+0000 7f0e5354b700 1 Processor -- start 2023-10-16T12:29:19.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.546+0000 7f0e5354b700 1 -- start start 2023-10-16T12:29:19.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.547+0000 7f0e5354b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0e4c0cb190 0x7f0e4c0c3620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:19.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.547+0000 7f0e5354b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e4c159c90 0x7f0e4c0c4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:19.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.547+0000 7f0e5354b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0e4c15e0b0 0x7f0e4c157790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:19.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.547+0000 7f0e5354b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e4c0c3c00 con 0x7f0e4c159c90 2023-10-16T12:29:19.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.547+0000 7f0e5354b700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e4c0c3ee0 con 0x7f0e4c15e0b0 2023-10-16T12:29:19.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.547+0000 7f0e5354b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e4c0c41c0 con 0x7f0e4c0cb190 2023-10-16T12:29:19.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.547+0000 7f0e50ae6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e4c159c90 0x7f0e4c0c4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:19.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.547+0000 7f0e50ae6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e4c159c90 0x7f0e4c0c4ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43788/0 (socket says 172.21.15.16:43788) 2023-10-16T12:29:19.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.547+0000 7f0e50ae6700 1 -- 172.21.15.16:0/3010034390 learned_addr learned my addr 172.21.15.16:0/3010034390 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:29:19.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.547+0000 7f0e512e7700 1 --2- 172.21.15.16:0/3010034390 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0e4c0cb190 0x7f0e4c0c3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:19.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.547+0000 7f0e51ae8700 1 --2- 172.21.15.16:0/3010034390 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0e4c15e0b0 0x7f0e4c157790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:19.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.547+0000 7f0e50ae6700 1 -- 172.21.15.16:0/3010034390 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0e4c0cb190 msgr2=0x7f0e4c0c3620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:19.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.547+0000 7f0e50ae6700 1 --2- 172.21.15.16:0/3010034390 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0e4c0cb190 0x7f0e4c0c3620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:19.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.547+0000 7f0e50ae6700 1 -- 172.21.15.16:0/3010034390 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0e4c15e0b0 msgr2=0x7f0e4c157790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:19.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.547+0000 7f0e50ae6700 1 --2- 172.21.15.16:0/3010034390 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0e4c15e0b0 0x7f0e4c157790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:19.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.547+0000 7f0e50ae6700 1 -- 172.21.15.16:0/3010034390 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e44009b30 con 0x7f0e4c159c90 2023-10-16T12:29:19.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.547+0000 7f0e512e7700 1 --2- 172.21.15.16:0/3010034390 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0e4c0cb190 0x7f0e4c0c3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:29:19.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.547+0000 7f0e51ae8700 1 --2- 172.21.15.16:0/3010034390 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0e4c15e0b0 0x7f0e4c157790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:29:19.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.547+0000 7f0e50ae6700 1 --2- 172.21.15.16:0/3010034390 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e4c159c90 0x7f0e4c0c4ca0 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f0e4800b500 tx=0x7f0e4800b8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:19.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.547+0000 7f0e41ffb700 1 -- 172.21.15.16:0/3010034390 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e48011830 con 0x7f0e4c159c90 2023-10-16T12:29:19.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.548+0000 7f0e41ffb700 1 -- 172.21.15.16:0/3010034390 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e480119b0 con 0x7f0e4c159c90 2023-10-16T12:29:19.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.548+0000 7f0e5354b700 1 -- 172.21.15.16:0/3010034390 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e4c0c44a0 con 0x7f0e4c159c90 2023-10-16T12:29:19.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.548+0000 7f0e41ffb700 1 -- 172.21.15.16:0/3010034390 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e48011d60 con 0x7f0e4c159c90 2023-10-16T12:29:19.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.548+0000 7f0e5354b700 1 -- 172.21.15.16:0/3010034390 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e4c0c6150 con 0x7f0e4c159c90 2023-10-16T12:29:19.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.549+0000 7f0e5354b700 1 -- 172.21.15.16:0/3010034390 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e4c066f10 con 0x7f0e4c159c90 2023-10-16T12:29:19.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.549+0000 7f0e41ffb700 1 -- 172.21.15.16:0/3010034390 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 62) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0e4801a400 con 0x7f0e4c159c90 2023-10-16T12:29:19.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.550+0000 7f0e41ffb700 1 --2- 172.21.15.16:0/3010034390 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f0e3006b020 0x7f0e3006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:19.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.550+0000 7f0e41ffb700 1 -- 172.21.15.16:0/3010034390 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f0e48089910 con 0x7f0e4c159c90 2023-10-16T12:29:19.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.550+0000 7f0e512e7700 1 --2- 172.21.15.16:0/3010034390 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f0e3006b020 0x7f0e3006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:19.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.550+0000 7f0e512e7700 1 --2- 172.21.15.16:0/3010034390 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f0e3006b020 0x7f0e3006d4e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0e3c0017d0 tx=0x7f0e3c009040 comp rx=0 tx=0).ready entity=mgr.7884 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:19.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.554+0000 7f0e41ffb700 1 -- 172.21.15.16:0/3010034390 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0e48054740 con 0x7f0e4c159c90 2023-10-16T12:29:19.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.801+0000 7f0e5354b700 1 -- 172.21.15.16:0/3010034390 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0e4c0c49c0 con 0x7f0e4c159c90 2023-10-16T12:29:19.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.802+0000 7f0e41ffb700 1 -- 172.21.15.16:0/3010034390 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+4909 (secure 0 0 0) 0x7f0e4c0c49c0 con 0x7f0e4c159c90 2023-10-16T12:29:19.804 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:29:19.804 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":61,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6893,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/2971585675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":2971585675},{"type":"v1","addr":"172.21.15.49:6835","nonce":2971585675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":7723,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/3151669585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":3151669585},{"type":"v1","addr":"172.21.15.49:6834","nonce":3151669585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":8106,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/52924937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":52924937},{"type":"v1","addr":"172.21.15.16:6836","nonce":52924937}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:29:15.801502+0000","modified":"2023-10-16T12:29:18.806285+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8109},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8109":{"gid":8109,"name":"a","rank":0,"incarnation":57,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/3203451346","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":3203451346},{"type":"v1","addr":"172.21.15.16:6837","nonce":3203451346}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2023-10-16T12:29:19.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.804+0000 7f0e5354b700 1 -- 172.21.15.16:0/3010034390 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f0e3006b020 msgr2=0x7f0e3006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:19.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.804+0000 7f0e5354b700 1 --2- 172.21.15.16:0/3010034390 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f0e3006b020 0x7f0e3006d4e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0e3c0017d0 tx=0x7f0e3c009040 comp rx=0 tx=0).stop 2023-10-16T12:29:19.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.804+0000 7f0e5354b700 1 -- 172.21.15.16:0/3010034390 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e4c159c90 msgr2=0x7f0e4c0c4ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:19.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.804+0000 7f0e5354b700 1 --2- 172.21.15.16:0/3010034390 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e4c159c90 0x7f0e4c0c4ca0 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f0e4800b500 tx=0x7f0e4800b8c0 comp rx=0 tx=0).stop 2023-10-16T12:29:19.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.804+0000 7f0e5354b700 1 -- 172.21.15.16:0/3010034390 shutdown_connections 2023-10-16T12:29:19.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.804+0000 7f0e5354b700 1 --2- 172.21.15.16:0/3010034390 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0e4c15e0b0 0x7f0e4c157790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:19.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.804+0000 7f0e5354b700 1 --2- 172.21.15.16:0/3010034390 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f0e3006b020 0x7f0e3006d4e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:19.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.804+0000 7f0e5354b700 1 --2- 172.21.15.16:0/3010034390 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0e4c0cb190 0x7f0e4c0c3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:19.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.804+0000 7f0e5354b700 1 --2- 172.21.15.16:0/3010034390 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e4c159c90 0x7f0e4c0c4ca0 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:19.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.804+0000 7f0e5354b700 1 -- 172.21.15.16:0/3010034390 >> 172.21.15.16:0/3010034390 conn(0x7f0e4c0cc5b0 msgr2=0x7f0e4c0ced10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:19.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.804+0000 7f0e5354b700 1 -- 172.21.15.16:0/3010034390 shutdown_connections 2023-10-16T12:29:19.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.804+0000 7f0e5354b700 1 -- 172.21.15.16:0/3010034390 wait complete. 2023-10-16T12:29:19.809 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 61 2023-10-16T12:29:19.818 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:29:19.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.943+0000 7f61da4d0700 1 Processor -- start 2023-10-16T12:29:19.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.943+0000 7f61da4d0700 1 -- start start 2023-10-16T12:29:19.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.944+0000 7f61da4d0700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f61d415db30 0x7f61d40cf920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:19.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.944+0000 7f61da4d0700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f61d4161490 0x7f61d4161920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:19.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.944+0000 7f61da4d0700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61d40d0070 con 0x7f61d415d750 2023-10-16T12:29:19.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.944+0000 7f61da4d0700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61d4161e60 con 0x7f61d4161490 2023-10-16T12:29:19.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.944+0000 7f61da4d0700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61d40cfe60 con 0x7f61d415db30 2023-10-16T12:29:19.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.945+0000 7f61d37fe700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f61d415db30 0x7f61d40cf920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:19.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.945+0000 7f61d8a6d700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f61d4161490 0x7f61d4161920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:19.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.945+0000 7f61d37fe700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f61d415db30 0x7f61d40cf920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:60286/0 (socket says 172.21.15.16:60286) 2023-10-16T12:29:19.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.945+0000 7f61d37fe700 1 -- 172.21.15.16:0/3558303239 learned_addr learned my addr 172.21.15.16:0/3558303239 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:29:19.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.945+0000 7f61d3fff700 1 --1- 172.21.15.16:0/3558303239 >> v1:172.21.15.49:6789/0 conn(0x7f61d415d750 0x7f61d406c960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:40250/0 (socket says 172.21.15.16:40250) 2023-10-16T12:29:19.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.945+0000 7f61d37fe700 1 -- 172.21.15.16:0/3558303239 >> v2:172.21.15.49:3300/0 conn(0x7f61d4161490 msgr2=0x7f61d4161920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:29:19.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.945+0000 7f61d37fe700 1 --2- 172.21.15.16:0/3558303239 >> v2:172.21.15.49:3300/0 conn(0x7f61d4161490 0x7f61d4161920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:19.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.945+0000 7f61d37fe700 1 -- 172.21.15.16:0/3558303239 >> v1:172.21.15.49:6789/0 conn(0x7f61d415d750 legacy=0x7f61d406c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:19.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.945+0000 7f61d37fe700 1 -- 172.21.15.16:0/3558303239 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61d40d0350 con 0x7f61d415db30 2023-10-16T12:29:19.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.945+0000 7f61d37fe700 1 --2- 172.21.15.16:0/3558303239 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f61d415db30 0x7f61d40cf920 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f61c80098f0 tx=0x7f61c8009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aebc91b30113772b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:19.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.945+0000 7f61d2ffd700 1 -- 172.21.15.16:0/3558303239 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61c8011830 con 0x7f61d415db30 2023-10-16T12:29:19.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.945+0000 7f61d2ffd700 1 -- 172.21.15.16:0/3558303239 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61c8011990 con 0x7f61d415db30 2023-10-16T12:29:19.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.945+0000 7f61d2ffd700 1 -- 172.21.15.16:0/3558303239 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61c8011d20 con 0x7f61d415db30 2023-10-16T12:29:19.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.946+0000 7f61da4d0700 1 -- 172.21.15.16:0/3558303239 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f61d415db30 msgr2=0x7f61d40cf920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:19.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.946+0000 7f61da4d0700 1 --2- 172.21.15.16:0/3558303239 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f61d415db30 0x7f61d40cf920 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f61c80098f0 tx=0x7f61c8009c00 comp rx=0 tx=0).stop 2023-10-16T12:29:19.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.946+0000 7f61da4d0700 1 -- 172.21.15.16:0/3558303239 shutdown_connections 2023-10-16T12:29:19.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.946+0000 7f61da4d0700 1 --2- 172.21.15.16:0/3558303239 >> v2:172.21.15.49:3300/0 conn(0x7f61d4161490 0x7f61d4161920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:19.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.946+0000 7f61da4d0700 1 --2- 172.21.15.16:0/3558303239 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f61d415db30 0x7f61d40cf920 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:19.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.946+0000 7f61da4d0700 1 -- 172.21.15.16:0/3558303239 >> 172.21.15.16:0/3558303239 conn(0x7f61d40d14b0 msgr2=0x7f61d40d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:19.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.946+0000 7f61da4d0700 1 -- 172.21.15.16:0/3558303239 shutdown_connections 2023-10-16T12:29:19.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.946+0000 7f61da4d0700 1 -- 172.21.15.16:0/3558303239 wait complete. 2023-10-16T12:29:19.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.947+0000 7f61da4d0700 1 Processor -- start 2023-10-16T12:29:19.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.947+0000 7f61da4d0700 1 -- start start 2023-10-16T12:29:19.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.948+0000 7f61da4d0700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f61d415d750 0x7f61d409ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:19.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.948+0000 7f61da4d0700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f61d415db30 0x7f61d409b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:19.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.948+0000 7f61da4d0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61d4161490 0x7f61d40b4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:19.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.948+0000 7f61da4d0700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61d40b6130 con 0x7f61d4161490 2023-10-16T12:29:19.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.948+0000 7f61da4d0700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61d4158490 con 0x7f61d415db30 2023-10-16T12:29:19.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.948+0000 7f61da4d0700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61d40b5de0 con 0x7f61d415d750 2023-10-16T12:29:19.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.948+0000 7f61d8a6d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61d4161490 0x7f61d40b4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:19.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.948+0000 7f61d8a6d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61d4161490 0x7f61d40b4300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43804/0 (socket says 172.21.15.16:43804) 2023-10-16T12:29:19.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.948+0000 7f61d8a6d700 1 -- 172.21.15.16:0/606750403 learned_addr learned my addr 172.21.15.16:0/606750403 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:29:19.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.948+0000 7f61d3fff700 1 --2- 172.21.15.16:0/606750403 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f61d415d750 0x7f61d409ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:19.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.948+0000 7f61d37fe700 1 --2- 172.21.15.16:0/606750403 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f61d415db30 0x7f61d409b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:19.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.949+0000 7f61d8a6d700 1 -- 172.21.15.16:0/606750403 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f61d415d750 msgr2=0x7f61d409ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:19.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.949+0000 7f61d8a6d700 1 --2- 172.21.15.16:0/606750403 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f61d415d750 0x7f61d409ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:19.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.949+0000 7f61d8a6d700 1 -- 172.21.15.16:0/606750403 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f61d415db30 msgr2=0x7f61d409b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:19.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.949+0000 7f61d8a6d700 1 --2- 172.21.15.16:0/606750403 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f61d415db30 0x7f61d409b130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:19.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.949+0000 7f61d8a6d700 1 -- 172.21.15.16:0/606750403 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61c8009550 con 0x7f61d4161490 2023-10-16T12:29:19.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.949+0000 7f61d8a6d700 1 --2- 172.21.15.16:0/606750403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61d4161490 0x7f61d40b4300 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f61cc00efc0 tx=0x7f61cc00c0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:19.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.949+0000 7f61d0ff9700 1 -- 172.21.15.16:0/606750403 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61cc00cad0 con 0x7f61d4161490 2023-10-16T12:29:19.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.949+0000 7f61d0ff9700 1 -- 172.21.15.16:0/606750403 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61cc00f040 con 0x7f61d4161490 2023-10-16T12:29:19.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.949+0000 7f61da4d0700 1 -- 172.21.15.16:0/606750403 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61d409b670 con 0x7f61d4161490 2023-10-16T12:29:19.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.949+0000 7f61d0ff9700 1 -- 172.21.15.16:0/606750403 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61cc00ce70 con 0x7f61d4161490 2023-10-16T12:29:19.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.949+0000 7f61da4d0700 1 -- 172.21.15.16:0/606750403 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61d40b4ae0 con 0x7f61d4161490 2023-10-16T12:29:19.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.950+0000 7f61da4d0700 1 -- 172.21.15.16:0/606750403 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61d40d0610 con 0x7f61d4161490 2023-10-16T12:29:19.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.951+0000 7f61d0ff9700 1 -- 172.21.15.16:0/606750403 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 62) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f61cc014480 con 0x7f61d4161490 2023-10-16T12:29:19.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.951+0000 7f61d0ff9700 1 --2- 172.21.15.16:0/606750403 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f61c006b020 0x7f61c006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:19.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.951+0000 7f61d0ff9700 1 -- 172.21.15.16:0/606750403 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f61cc08a080 con 0x7f61d4161490 2023-10-16T12:29:19.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.951+0000 7f61d3fff700 1 --2- 172.21.15.16:0/606750403 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f61c006b020 0x7f61c006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:19.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.952+0000 7f61d3fff700 1 --2- 172.21.15.16:0/606750403 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f61c006b020 0x7f61c006d4e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f61c4000fb0 tx=0x7f61c4007ec0 comp rx=0 tx=0).ready entity=mgr.7884 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:19.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:19.956+0000 7f61d0ff9700 1 -- 172.21.15.16:0/606750403 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f61d40d0610 con 0x7f61d4161490 2023-10-16T12:29:20.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.159+0000 7f61da4d0700 1 -- 172.21.15.16:0/606750403 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f61d40b6270 con 0x7f61d4161490 2023-10-16T12:29:20.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.160+0000 7f61d0ff9700 1 -- 172.21.15.16:0/606750403 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v128) v1 ==== 74+0+16651 (secure 0 0 0) 0x7f61d40b6270 con 0x7f61d4161490 2023-10-16T12:29:20.163 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:29:20.163 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":128,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:29:17.276813+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:29:14.232554+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2023-10-16T12:29:14.764132+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:29:20.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.162+0000 7f61da4d0700 1 -- 172.21.15.16:0/606750403 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f61c006b020 msgr2=0x7f61c006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:20.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.162+0000 7f61da4d0700 1 --2- 172.21.15.16:0/606750403 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f61c006b020 0x7f61c006d4e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f61c4000fb0 tx=0x7f61c4007ec0 comp rx=0 tx=0).stop 2023-10-16T12:29:20.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.162+0000 7f61da4d0700 1 -- 172.21.15.16:0/606750403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61d4161490 msgr2=0x7f61d40b4300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:20.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.162+0000 7f61da4d0700 1 --2- 172.21.15.16:0/606750403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61d4161490 0x7f61d40b4300 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f61cc00efc0 tx=0x7f61cc00c0c0 comp rx=0 tx=0).stop 2023-10-16T12:29:20.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.163+0000 7f61da4d0700 1 -- 172.21.15.16:0/606750403 shutdown_connections 2023-10-16T12:29:20.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.163+0000 7f61da4d0700 1 --2- 172.21.15.16:0/606750403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61d4161490 0x7f61d40b4300 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:20.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.163+0000 7f61da4d0700 1 --2- 172.21.15.16:0/606750403 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f61c006b020 0x7f61c006d4e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:20.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.163+0000 7f61da4d0700 1 --2- 172.21.15.16:0/606750403 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f61d415d750 0x7f61d409ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:20.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.163+0000 7f61da4d0700 1 --2- 172.21.15.16:0/606750403 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f61d415db30 0x7f61d409b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:20.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.163+0000 7f61da4d0700 1 -- 172.21.15.16:0/606750403 >> 172.21.15.16:0/606750403 conn(0x7f61d40d14b0 msgr2=0x7f61d40ce590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:20.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.163+0000 7f61da4d0700 1 -- 172.21.15.16:0/606750403 shutdown_connections 2023-10-16T12:29:20.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.163+0000 7f61da4d0700 1 -- 172.21.15.16:0/606750403 wait complete. 2023-10-16T12:29:20.177 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-16T12:29:20.177 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T12:29:20.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.300+0000 7fd1c2c89700 1 Processor -- start 2023-10-16T12:29:20.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.300+0000 7fd1c2c89700 1 -- start start 2023-10-16T12:29:20.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.301+0000 7fd1c2c89700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fd1bc0cf4a0 0x7fd1bc0726e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:20.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.301+0000 7fd1c2c89700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd1bc0cf880 0x7fd1bc0b52f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:20.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.301+0000 7fd1c2c89700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd1bc0b5830 0x7fd1bc15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:20.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.301+0000 7fd1c2c89700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1bc15bc80 con 0x7fd1bc0cf880 2023-10-16T12:29:20.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.301+0000 7fd1c2c89700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1bc072c20 con 0x7fd1bc0cf4a0 2023-10-16T12:29:20.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.302+0000 7fd1c2c89700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1bc15b070 con 0x7fd1bc0b5830 2023-10-16T12:29:20.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.302+0000 7fd1bbfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd1bc0cf880 0x7fd1bc0b52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:20.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.302+0000 7fd1bbfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd1bc0cf880 0x7fd1bc0b52f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43808/0 (socket says 172.21.15.16:43808) 2023-10-16T12:29:20.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.302+0000 7fd1bbfff700 1 -- 172.21.15.16:0/1974230800 learned_addr learned my addr 172.21.15.16:0/1974230800 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:29:20.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.302+0000 7fd1c1226700 1 --2- 172.21.15.16:0/1974230800 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd1bc0b5830 0x7fd1bc15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:20.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.302+0000 7fd1c0a25700 1 --2- 172.21.15.16:0/1974230800 >> v2:172.21.15.49:3300/0 conn(0x7fd1bc0cf4a0 0x7fd1bc0726e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:20.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.302+0000 7fd1bbfff700 1 -- 172.21.15.16:0/1974230800 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd1bc0b5830 msgr2=0x7fd1bc15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:29:20.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.302+0000 7fd1bbfff700 1 --2- 172.21.15.16:0/1974230800 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd1bc0b5830 0x7fd1bc15a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:20.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.302+0000 7fd1bbfff700 1 -- 172.21.15.16:0/1974230800 >> v2:172.21.15.49:3300/0 conn(0x7fd1bc0cf4a0 msgr2=0x7fd1bc0726e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:29:20.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.302+0000 7fd1bbfff700 1 --2- 172.21.15.16:0/1974230800 >> v2:172.21.15.49:3300/0 conn(0x7fd1bc0cf4a0 0x7fd1bc0726e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:20.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.302+0000 7fd1bbfff700 1 -- 172.21.15.16:0/1974230800 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1bc15b350 con 0x7fd1bc0cf880 2023-10-16T12:29:20.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.302+0000 7fd1c0a25700 1 --2- 172.21.15.16:0/1974230800 >> v2:172.21.15.49:3300/0 conn(0x7fd1bc0cf4a0 0x7fd1bc0726e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:29:20.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.302+0000 7fd1bbfff700 1 --2- 172.21.15.16:0/1974230800 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1bc0cf880 0x7fd1bc0b52f0 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc0b2d20 tx=0x7fd1a8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8819391c3f862d3d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:20.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.302+0000 7fd1bb7fe700 1 -- 172.21.15.16:0/1974230800 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a8011830 con 0x7fd1bc0cf880 2023-10-16T12:29:20.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.302+0000 7fd1bb7fe700 1 -- 172.21.15.16:0/1974230800 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1a8011990 con 0x7fd1bc0cf880 2023-10-16T12:29:20.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.302+0000 7fd1bb7fe700 1 -- 172.21.15.16:0/1974230800 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a8011d20 con 0x7fd1bc0cf880 2023-10-16T12:29:20.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.303+0000 7fd1c2c89700 1 -- 172.21.15.16:0/1974230800 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1bc0cf880 msgr2=0x7fd1bc0b52f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:20.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.303+0000 7fd1c2c89700 1 --2- 172.21.15.16:0/1974230800 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1bc0cf880 0x7fd1bc0b52f0 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc0b2d20 tx=0x7fd1a8009bd0 comp rx=0 tx=0).stop 2023-10-16T12:29:20.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.303+0000 7fd1c2c89700 1 -- 172.21.15.16:0/1974230800 shutdown_connections 2023-10-16T12:29:20.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.303+0000 7fd1c2c89700 1 --2- 172.21.15.16:0/1974230800 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd1bc0b5830 0x7fd1bc15a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:20.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.303+0000 7fd1c2c89700 1 --2- 172.21.15.16:0/1974230800 >> v2:172.21.15.49:3300/0 conn(0x7fd1bc0cf4a0 0x7fd1bc0726e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:20.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.303+0000 7fd1c2c89700 1 --2- 172.21.15.16:0/1974230800 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1bc0cf880 0x7fd1bc0b52f0 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:20.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.303+0000 7fd1c2c89700 1 -- 172.21.15.16:0/1974230800 >> 172.21.15.16:0/1974230800 conn(0x7fd1bc0b1fd0 msgr2=0x7fd1bc0b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:20.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.303+0000 7fd1c2c89700 1 -- 172.21.15.16:0/1974230800 shutdown_connections 2023-10-16T12:29:20.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.303+0000 7fd1c2c89700 1 -- 172.21.15.16:0/1974230800 wait complete. 2023-10-16T12:29:20.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.304+0000 7fd1c2c89700 1 Processor -- start 2023-10-16T12:29:20.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.304+0000 7fd1c2c89700 1 -- start start 2023-10-16T12:29:20.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.305+0000 7fd1c2c89700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd1bc0b5830 0x7fd1bc0cba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:20.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.305+0000 7fd1c2c89700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd1bc0cf4a0 0x7fd1bc0c6650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:20.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.305+0000 7fd1c2c89700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1bc0cf880 0x7fd1bc0cd380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:20.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.305+0000 7fd1c2c89700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1bc0ccf60 con 0x7fd1bc0cf880 2023-10-16T12:29:20.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.305+0000 7fd1c2c89700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1bc0cc2e0 con 0x7fd1bc0cf4a0 2023-10-16T12:29:20.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.305+0000 7fd1c2c89700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1bc0cc5c0 con 0x7fd1bc0b5830 2023-10-16T12:29:20.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.305+0000 7fd1c1226700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1bc0cf880 0x7fd1bc0cd380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:20.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.305+0000 7fd1c1226700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1bc0cf880 0x7fd1bc0cd380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43818/0 (socket says 172.21.15.16:43818) 2023-10-16T12:29:20.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.305+0000 7fd1c1226700 1 -- 172.21.15.16:0/2454324046 learned_addr learned my addr 172.21.15.16:0/2454324046 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:29:20.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.305+0000 7fd1c0a25700 1 --2- 172.21.15.16:0/2454324046 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd1bc0b5830 0x7fd1bc0cba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:20.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.305+0000 7fd1bbfff700 1 --2- 172.21.15.16:0/2454324046 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd1bc0cf4a0 0x7fd1bc0c6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:20.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.305+0000 7fd1c1226700 1 -- 172.21.15.16:0/2454324046 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd1bc0b5830 msgr2=0x7fd1bc0cba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:20.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.306+0000 7fd1c1226700 1 --2- 172.21.15.16:0/2454324046 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd1bc0b5830 0x7fd1bc0cba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:20.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.306+0000 7fd1c1226700 1 -- 172.21.15.16:0/2454324046 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd1bc0cf4a0 msgr2=0x7fd1bc0c6650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:20.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.306+0000 7fd1c1226700 1 --2- 172.21.15.16:0/2454324046 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd1bc0cf4a0 0x7fd1bc0c6650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:20.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.306+0000 7fd1c1226700 1 -- 172.21.15.16:0/2454324046 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1a8009550 con 0x7fd1bc0cf880 2023-10-16T12:29:20.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.306+0000 7fd1c0a25700 1 --2- 172.21.15.16:0/2454324046 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd1bc0b5830 0x7fd1bc0cba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:29:20.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.306+0000 7fd1bbfff700 1 --2- 172.21.15.16:0/2454324046 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd1bc0cf4a0 0x7fd1bc0c6650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:29:20.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.306+0000 7fd1c1226700 1 --2- 172.21.15.16:0/2454324046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1bc0cf880 0x7fd1bc0cd380 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7fd1b400eed0 tx=0x7fd1b400b390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:20.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.306+0000 7fd1b97fa700 1 -- 172.21.15.16:0/2454324046 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1b4010040 con 0x7fd1bc0cf880 2023-10-16T12:29:20.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.306+0000 7fd1b97fa700 1 -- 172.21.15.16:0/2454324046 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1b400bd30 con 0x7fd1bc0cf880 2023-10-16T12:29:20.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.306+0000 7fd1c2c89700 1 -- 172.21.15.16:0/2454324046 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1bc0cc8a0 con 0x7fd1bc0cf880 2023-10-16T12:29:20.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.306+0000 7fd1b97fa700 1 -- 172.21.15.16:0/2454324046 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1b4009ea0 con 0x7fd1bc0cf880 2023-10-16T12:29:20.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.306+0000 7fd1c2c89700 1 -- 172.21.15.16:0/2454324046 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1bc0cbfd0 con 0x7fd1bc0cf880 2023-10-16T12:29:20.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.307+0000 7fd1c2c89700 1 -- 172.21.15.16:0/2454324046 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1bc0cd0a0 con 0x7fd1bc0cf880 2023-10-16T12:29:20.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.307+0000 7fd1b97fa700 1 -- 172.21.15.16:0/2454324046 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 62) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd1b4014640 con 0x7fd1bc0cf880 2023-10-16T12:29:20.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.308+0000 7fd1b97fa700 1 --2- 172.21.15.16:0/2454324046 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fd1a006b020 0x7fd1a006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:20.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.308+0000 7fd1b97fa700 1 -- 172.21.15.16:0/2454324046 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fd1b4089fd0 con 0x7fd1bc0cf880 2023-10-16T12:29:20.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.308+0000 7fd1c0a25700 1 --2- 172.21.15.16:0/2454324046 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fd1a006b020 0x7fd1a006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:20.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.308+0000 7fd1c0a25700 1 --2- 172.21.15.16:0/2454324046 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fd1a006b020 0x7fd1a006d4e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc1f6280 tx=0x7fd1b0008040 comp rx=0 tx=0).ready entity=mgr.7884 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:20.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.312+0000 7fd1b97fa700 1 -- 172.21.15.16:0/2454324046 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd1bc0cd0a0 con 0x7fd1bc0cf880 2023-10-16T12:29:20.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.512+0000 7fd1c2c89700 1 -- 172.21.15.16:0/2454324046 --> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd1bc15b910 con 0x7fd1a006b020 2023-10-16T12:29:20.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.521+0000 7fd1b97fa700 1 -- 172.21.15.16:0/2454324046 <== mgr.7884 v2:172.21.15.16:6832/2309613638 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+290007 (secure 0 0 0) 0x7fd1b000c050 con 0x7fd1a006b020 2023-10-16T12:29:20.527 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:29:20.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.524+0000 7fd1c2c89700 1 -- 172.21.15.16:0/2454324046 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fd1a006b020 msgr2=0x7fd1a006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:20.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.524+0000 7fd1c2c89700 1 --2- 172.21.15.16:0/2454324046 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fd1a006b020 0x7fd1a006d4e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc1f6280 tx=0x7fd1b0008040 comp rx=0 tx=0).stop 2023-10-16T12:29:20.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.524+0000 7fd1c2c89700 1 -- 172.21.15.16:0/2454324046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1bc0cf880 msgr2=0x7fd1bc0cd380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:20.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.524+0000 7fd1c2c89700 1 --2- 172.21.15.16:0/2454324046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1bc0cf880 0x7fd1bc0cd380 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7fd1b400eed0 tx=0x7fd1b400b390 comp rx=0 tx=0).stop 2023-10-16T12:29:20.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.524+0000 7fd1c2c89700 1 -- 172.21.15.16:0/2454324046 shutdown_connections 2023-10-16T12:29:20.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.524+0000 7fd1c2c89700 1 --2- 172.21.15.16:0/2454324046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1bc0cf880 0x7fd1bc0cd380 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:20.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.524+0000 7fd1c2c89700 1 --2- 172.21.15.16:0/2454324046 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fd1a006b020 0x7fd1a006d4e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:20.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.524+0000 7fd1c2c89700 1 --2- 172.21.15.16:0/2454324046 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd1bc0b5830 0x7fd1bc0cba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:20.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.524+0000 7fd1c2c89700 1 --2- 172.21.15.16:0/2454324046 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd1bc0cf4a0 0x7fd1bc0c6650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:20.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.524+0000 7fd1c2c89700 1 -- 172.21.15.16:0/2454324046 >> 172.21.15.16:0/2454324046 conn(0x7fd1bc0b1fd0 msgr2=0x7fd1bc0b0570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:20.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.524+0000 7fd1c2c89700 1 -- 172.21.15.16:0/2454324046 shutdown_connections 2023-10-16T12:29:20.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.524+0000 7fd1c2c89700 1 -- 172.21.15.16:0/2454324046 wait complete. 2023-10-16T12:29:20.532 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-10-16T12:29:20.538 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-10-16T12:29:18.915190+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":884,"num_read_kb":1748,"num_write":363,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":243,"ondisk_log_size":243,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":121,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":354504,"kb_used_data":13452,"kb_used_omap":12,"kb_used_meta":340915,"kb_avail":754620216,"statfs":{"total":773094113280,"available":772731101184,"internally_reserved":0,"allocated":13774848,"data_stored":7495585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12740,"internal_metadata":349097532},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.551030"},"pg_stats":[{"pgid":"16.3f","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.300047+0000","last_change":"2023-10-16T12:29:16.295442+0000","last_active":"2023-10-16T12:29:16.300047+0000","last_peered":"2023-10-16T12:29:16.300047+0000","last_clean":"2023-10-16T12:29:16.300047+0000","last_became_active":"2023-10-16T12:29:16.293293+0000","last_became_peered":"2023-10-16T12:29:16.293293+0000","last_unstale":"2023-10-16T12:29:16.300047+0000","last_undegraded":"2023-10-16T12:29:16.300047+0000","last_fullsized":"2023-10-16T12:29:16.300047+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:45:34.212638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3e","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.304749+0000","last_change":"2023-10-16T12:29:16.304101+0000","last_active":"2023-10-16T12:29:16.304749+0000","last_peered":"2023-10-16T12:29:16.304749+0000","last_clean":"2023-10-16T12:29:16.304749+0000","last_became_active":"2023-10-16T12:29:16.303818+0000","last_became_peered":"2023-10-16T12:29:16.303818+0000","last_unstale":"2023-10-16T12:29:16.304749+0000","last_undegraded":"2023-10-16T12:29:16.304749+0000","last_fullsized":"2023-10-16T12:29:16.304749+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:37:29.167682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.3d","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.303703+0000","last_change":"2023-10-16T12:29:16.296339+0000","last_active":"2023-10-16T12:29:16.303703+0000","last_peered":"2023-10-16T12:29:16.303703+0000","last_clean":"2023-10-16T12:29:16.303703+0000","last_became_active":"2023-10-16T12:29:16.296083+0000","last_became_peered":"2023-10-16T12:29:16.296083+0000","last_unstale":"2023-10-16T12:29:16.303703+0000","last_undegraded":"2023-10-16T12:29:16.303703+0000","last_fullsized":"2023-10-16T12:29:16.303703+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:33:34.851022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3c","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.300969+0000","last_change":"2023-10-16T12:29:16.297638+0000","last_active":"2023-10-16T12:29:16.300969+0000","last_peered":"2023-10-16T12:29:16.300969+0000","last_clean":"2023-10-16T12:29:16.300969+0000","last_became_active":"2023-10-16T12:29:16.297121+0000","last_became_peered":"2023-10-16T12:29:16.297121+0000","last_unstale":"2023-10-16T12:29:16.300969+0000","last_undegraded":"2023-10-16T12:29:16.300969+0000","last_fullsized":"2023-10-16T12:29:16.300969+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:14:59.533040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.3b","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.299033+0000","last_change":"2023-10-16T12:29:16.297543+0000","last_active":"2023-10-16T12:29:16.299033+0000","last_peered":"2023-10-16T12:29:16.299033+0000","last_clean":"2023-10-16T12:29:16.299033+0000","last_became_active":"2023-10-16T12:29:16.297266+0000","last_became_peered":"2023-10-16T12:29:16.297266+0000","last_unstale":"2023-10-16T12:29:16.299033+0000","last_undegraded":"2023-10-16T12:29:16.299033+0000","last_fullsized":"2023-10-16T12:29:16.299033+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:15:48.726807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.3a","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.293761+0000","last_change":"2023-10-16T12:29:16.291104+0000","last_active":"2023-10-16T12:29:16.293761+0000","last_peered":"2023-10-16T12:29:16.293761+0000","last_clean":"2023-10-16T12:29:16.293761+0000","last_became_active":"2023-10-16T12:29:16.290781+0000","last_became_peered":"2023-10-16T12:29:16.290781+0000","last_unstale":"2023-10-16T12:29:16.293761+0000","last_undegraded":"2023-10-16T12:29:16.293761+0000","last_fullsized":"2023-10-16T12:29:16.293761+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:31:19.116278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.39","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.291331+0000","last_change":"2023-10-16T12:29:16.285561+0000","last_active":"2023-10-16T12:29:16.291331+0000","last_peered":"2023-10-16T12:29:16.291331+0000","last_clean":"2023-10-16T12:29:16.291331+0000","last_became_active":"2023-10-16T12:29:16.285354+0000","last_became_peered":"2023-10-16T12:29:16.285354+0000","last_unstale":"2023-10-16T12:29:16.291331+0000","last_undegraded":"2023-10-16T12:29:16.291331+0000","last_fullsized":"2023-10-16T12:29:16.291331+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:38:24.426289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.38","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.292514+0000","last_change":"2023-10-16T12:29:16.290861+0000","last_active":"2023-10-16T12:29:16.292514+0000","last_peered":"2023-10-16T12:29:16.292514+0000","last_clean":"2023-10-16T12:29:16.292514+0000","last_became_active":"2023-10-16T12:29:16.290578+0000","last_became_peered":"2023-10-16T12:29:16.290578+0000","last_unstale":"2023-10-16T12:29:16.292514+0000","last_undegraded":"2023-10-16T12:29:16.292514+0000","last_fullsized":"2023-10-16T12:29:16.292514+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:53:51.137615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.37","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.304509+0000","last_change":"2023-10-16T12:29:16.303198+0000","last_active":"2023-10-16T12:29:16.304509+0000","last_peered":"2023-10-16T12:29:16.304509+0000","last_clean":"2023-10-16T12:29:16.304509+0000","last_became_active":"2023-10-16T12:29:16.301618+0000","last_became_peered":"2023-10-16T12:29:16.301618+0000","last_unstale":"2023-10-16T12:29:16.304509+0000","last_undegraded":"2023-10-16T12:29:16.304509+0000","last_fullsized":"2023-10-16T12:29:16.304509+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:15:48.726807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.36","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.298649+0000","last_change":"2023-10-16T12:29:16.295295+0000","last_active":"2023-10-16T12:29:16.298649+0000","last_peered":"2023-10-16T12:29:16.298649+0000","last_clean":"2023-10-16T12:29:16.298649+0000","last_became_active":"2023-10-16T12:29:16.294939+0000","last_became_peered":"2023-10-16T12:29:16.294939+0000","last_unstale":"2023-10-16T12:29:16.298649+0000","last_undegraded":"2023-10-16T12:29:16.298649+0000","last_fullsized":"2023-10-16T12:29:16.298649+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:12:17.761721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.35","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.298915+0000","last_change":"2023-10-16T12:29:16.295342+0000","last_active":"2023-10-16T12:29:16.298915+0000","last_peered":"2023-10-16T12:29:16.298915+0000","last_clean":"2023-10-16T12:29:16.298915+0000","last_became_active":"2023-10-16T12:29:16.295021+0000","last_became_peered":"2023-10-16T12:29:16.295021+0000","last_unstale":"2023-10-16T12:29:16.298915+0000","last_undegraded":"2023-10-16T12:29:16.298915+0000","last_fullsized":"2023-10-16T12:29:16.298915+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:10:52.270680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.34","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.301204+0000","last_change":"2023-10-16T12:29:16.297042+0000","last_active":"2023-10-16T12:29:16.301204+0000","last_peered":"2023-10-16T12:29:16.301204+0000","last_clean":"2023-10-16T12:29:16.301204+0000","last_became_active":"2023-10-16T12:29:16.295527+0000","last_became_peered":"2023-10-16T12:29:16.295527+0000","last_unstale":"2023-10-16T12:29:16.301204+0000","last_undegraded":"2023-10-16T12:29:16.301204+0000","last_fullsized":"2023-10-16T12:29:16.301204+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:32:43.586508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.33","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.299862+0000","last_change":"2023-10-16T12:29:16.296802+0000","last_active":"2023-10-16T12:29:16.299862+0000","last_peered":"2023-10-16T12:29:16.299862+0000","last_clean":"2023-10-16T12:29:16.299862+0000","last_became_active":"2023-10-16T12:29:16.296485+0000","last_became_peered":"2023-10-16T12:29:16.296485+0000","last_unstale":"2023-10-16T12:29:16.299862+0000","last_undegraded":"2023-10-16T12:29:16.299862+0000","last_fullsized":"2023-10-16T12:29:16.299862+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:20:51.410821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.32","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.299666+0000","last_change":"2023-10-16T12:29:16.297814+0000","last_active":"2023-10-16T12:29:16.299666+0000","last_peered":"2023-10-16T12:29:16.299666+0000","last_clean":"2023-10-16T12:29:16.299666+0000","last_became_active":"2023-10-16T12:29:16.297483+0000","last_became_peered":"2023-10-16T12:29:16.297483+0000","last_unstale":"2023-10-16T12:29:16.299666+0000","last_undegraded":"2023-10-16T12:29:16.299666+0000","last_fullsized":"2023-10-16T12:29:16.299666+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:03:19.526603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.31","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.299983+0000","last_change":"2023-10-16T12:29:16.296869+0000","last_active":"2023-10-16T12:29:16.299983+0000","last_peered":"2023-10-16T12:29:16.299983+0000","last_clean":"2023-10-16T12:29:16.299983+0000","last_became_active":"2023-10-16T12:29:16.295954+0000","last_became_peered":"2023-10-16T12:29:16.295954+0000","last_unstale":"2023-10-16T12:29:16.299983+0000","last_undegraded":"2023-10-16T12:29:16.299983+0000","last_fullsized":"2023-10-16T12:29:16.299983+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:47:23.445036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.30","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.299089+0000","last_change":"2023-10-16T12:29:16.296579+0000","last_active":"2023-10-16T12:29:16.299089+0000","last_peered":"2023-10-16T12:29:16.299089+0000","last_clean":"2023-10-16T12:29:16.299089+0000","last_became_active":"2023-10-16T12:29:16.296268+0000","last_became_peered":"2023-10-16T12:29:16.296268+0000","last_unstale":"2023-10-16T12:29:16.299089+0000","last_undegraded":"2023-10-16T12:29:16.299089+0000","last_fullsized":"2023-10-16T12:29:16.299089+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:14:59.533040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.2f","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.304106+0000","last_change":"2023-10-16T12:29:16.299703+0000","last_active":"2023-10-16T12:29:16.304106+0000","last_peered":"2023-10-16T12:29:16.304106+0000","last_clean":"2023-10-16T12:29:16.304106+0000","last_became_active":"2023-10-16T12:29:16.297900+0000","last_became_peered":"2023-10-16T12:29:16.297900+0000","last_unstale":"2023-10-16T12:29:16.304106+0000","last_undegraded":"2023-10-16T12:29:16.304106+0000","last_fullsized":"2023-10-16T12:29:16.304106+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:35:45.656103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2e","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.293346+0000","last_change":"2023-10-16T12:29:16.291910+0000","last_active":"2023-10-16T12:29:16.293346+0000","last_peered":"2023-10-16T12:29:16.293346+0000","last_clean":"2023-10-16T12:29:16.293346+0000","last_became_active":"2023-10-16T12:29:16.291575+0000","last_became_peered":"2023-10-16T12:29:16.291575+0000","last_unstale":"2023-10-16T12:29:16.293346+0000","last_undegraded":"2023-10-16T12:29:16.293346+0000","last_fullsized":"2023-10-16T12:29:16.293346+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:44:04.041711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.2d","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.304233+0000","last_change":"2023-10-16T12:29:16.300658+0000","last_active":"2023-10-16T12:29:16.304233+0000","last_peered":"2023-10-16T12:29:16.304233+0000","last_clean":"2023-10-16T12:29:16.304233+0000","last_became_active":"2023-10-16T12:29:16.300186+0000","last_became_peered":"2023-10-16T12:29:16.300186+0000","last_unstale":"2023-10-16T12:29:16.304233+0000","last_undegraded":"2023-10-16T12:29:16.304233+0000","last_fullsized":"2023-10-16T12:29:16.304233+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:37:10.259178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2c","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.301982+0000","last_change":"2023-10-16T12:29:16.299844+0000","last_active":"2023-10-16T12:29:16.301982+0000","last_peered":"2023-10-16T12:29:16.301982+0000","last_clean":"2023-10-16T12:29:16.301982+0000","last_became_active":"2023-10-16T12:29:16.299526+0000","last_became_peered":"2023-10-16T12:29:16.299526+0000","last_unstale":"2023-10-16T12:29:16.301982+0000","last_undegraded":"2023-10-16T12:29:16.301982+0000","last_fullsized":"2023-10-16T12:29:16.301982+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:26:55.793279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2b","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.295832+0000","last_change":"2023-10-16T12:29:16.295444+0000","last_active":"2023-10-16T12:29:16.295832+0000","last_peered":"2023-10-16T12:29:16.295832+0000","last_clean":"2023-10-16T12:29:16.295832+0000","last_became_active":"2023-10-16T12:29:16.295201+0000","last_became_peered":"2023-10-16T12:29:16.295201+0000","last_unstale":"2023-10-16T12:29:16.295832+0000","last_undegraded":"2023-10-16T12:29:16.295832+0000","last_fullsized":"2023-10-16T12:29:16.295832+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:41:46.033189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2a","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.298373+0000","last_change":"2023-10-16T12:29:16.294905+0000","last_active":"2023-10-16T12:29:16.298373+0000","last_peered":"2023-10-16T12:29:16.298373+0000","last_clean":"2023-10-16T12:29:16.298373+0000","last_became_active":"2023-10-16T12:29:16.294617+0000","last_became_peered":"2023-10-16T12:29:16.294617+0000","last_unstale":"2023-10-16T12:29:16.298373+0000","last_undegraded":"2023-10-16T12:29:16.298373+0000","last_fullsized":"2023-10-16T12:29:16.298373+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:37:29.167682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.29","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.303742+0000","last_change":"2023-10-16T12:29:16.296846+0000","last_active":"2023-10-16T12:29:16.303742+0000","last_peered":"2023-10-16T12:29:16.303742+0000","last_clean":"2023-10-16T12:29:16.303742+0000","last_became_active":"2023-10-16T12:29:16.296529+0000","last_became_peered":"2023-10-16T12:29:16.296529+0000","last_unstale":"2023-10-16T12:29:16.303742+0000","last_undegraded":"2023-10-16T12:29:16.303742+0000","last_fullsized":"2023-10-16T12:29:16.303742+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:34:41.695364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.28","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.302020+0000","last_change":"2023-10-16T12:29:16.301568+0000","last_active":"2023-10-16T12:29:16.302020+0000","last_peered":"2023-10-16T12:29:16.302020+0000","last_clean":"2023-10-16T12:29:16.302020+0000","last_became_active":"2023-10-16T12:29:16.301243+0000","last_became_peered":"2023-10-16T12:29:16.301243+0000","last_unstale":"2023-10-16T12:29:16.302020+0000","last_undegraded":"2023-10-16T12:29:16.302020+0000","last_fullsized":"2023-10-16T12:29:16.302020+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:44:04.041711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.27","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.301168+0000","last_change":"2023-10-16T12:29:16.296472+0000","last_active":"2023-10-16T12:29:16.301168+0000","last_peered":"2023-10-16T12:29:16.301168+0000","last_clean":"2023-10-16T12:29:16.301168+0000","last_became_active":"2023-10-16T12:29:16.296142+0000","last_became_peered":"2023-10-16T12:29:16.296142+0000","last_unstale":"2023-10-16T12:29:16.301168+0000","last_undegraded":"2023-10-16T12:29:16.301168+0000","last_fullsized":"2023-10-16T12:29:16.301168+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:40:09.675190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.26","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.295818+0000","last_change":"2023-10-16T12:29:16.292770+0000","last_active":"2023-10-16T12:29:16.295818+0000","last_peered":"2023-10-16T12:29:16.295818+0000","last_clean":"2023-10-16T12:29:16.295818+0000","last_became_active":"2023-10-16T12:29:16.292498+0000","last_became_peered":"2023-10-16T12:29:16.292498+0000","last_unstale":"2023-10-16T12:29:16.295818+0000","last_undegraded":"2023-10-16T12:29:16.295818+0000","last_fullsized":"2023-10-16T12:29:16.295818+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:12:04.458567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.25","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.302311+0000","last_change":"2023-10-16T12:29:16.298811+0000","last_active":"2023-10-16T12:29:16.302311+0000","last_peered":"2023-10-16T12:29:16.302311+0000","last_clean":"2023-10-16T12:29:16.302311+0000","last_became_active":"2023-10-16T12:29:16.298486+0000","last_became_peered":"2023-10-16T12:29:16.298486+0000","last_unstale":"2023-10-16T12:29:16.302311+0000","last_undegraded":"2023-10-16T12:29:16.302311+0000","last_fullsized":"2023-10-16T12:29:16.302311+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:27:09.415625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.24","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.303920+0000","last_change":"2023-10-16T12:29:16.295140+0000","last_active":"2023-10-16T12:29:16.303920+0000","last_peered":"2023-10-16T12:29:16.303920+0000","last_clean":"2023-10-16T12:29:16.303920+0000","last_became_active":"2023-10-16T12:29:16.294860+0000","last_became_peered":"2023-10-16T12:29:16.294860+0000","last_unstale":"2023-10-16T12:29:16.303920+0000","last_undegraded":"2023-10-16T12:29:16.303920+0000","last_fullsized":"2023-10-16T12:29:16.303920+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:48:44.882314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.23","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.297914+0000","last_change":"2023-10-16T12:29:16.297550+0000","last_active":"2023-10-16T12:29:16.297914+0000","last_peered":"2023-10-16T12:29:16.297914+0000","last_clean":"2023-10-16T12:29:16.297914+0000","last_became_active":"2023-10-16T12:29:16.297141+0000","last_became_peered":"2023-10-16T12:29:16.297141+0000","last_unstale":"2023-10-16T12:29:16.297914+0000","last_undegraded":"2023-10-16T12:29:16.297914+0000","last_fullsized":"2023-10-16T12:29:16.297914+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:28:23.841300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.22","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.301579+0000","last_change":"2023-10-16T12:29:16.299334+0000","last_active":"2023-10-16T12:29:16.301579+0000","last_peered":"2023-10-16T12:29:16.301579+0000","last_clean":"2023-10-16T12:29:16.301579+0000","last_became_active":"2023-10-16T12:29:16.298951+0000","last_became_peered":"2023-10-16T12:29:16.298951+0000","last_unstale":"2023-10-16T12:29:16.301579+0000","last_undegraded":"2023-10-16T12:29:16.301579+0000","last_fullsized":"2023-10-16T12:29:16.301579+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:15:35.527333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.21","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.304703+0000","last_change":"2023-10-16T12:29:16.303779+0000","last_active":"2023-10-16T12:29:16.304703+0000","last_peered":"2023-10-16T12:29:16.304703+0000","last_clean":"2023-10-16T12:29:16.304703+0000","last_became_active":"2023-10-16T12:29:16.302770+0000","last_became_peered":"2023-10-16T12:29:16.302770+0000","last_unstale":"2023-10-16T12:29:16.304703+0000","last_undegraded":"2023-10-16T12:29:16.304703+0000","last_fullsized":"2023-10-16T12:29:16.304703+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:56:15.471878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.20","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.297271+0000","last_change":"2023-10-16T12:29:16.296088+0000","last_active":"2023-10-16T12:29:16.297271+0000","last_peered":"2023-10-16T12:29:16.297271+0000","last_clean":"2023-10-16T12:29:16.297271+0000","last_became_active":"2023-10-16T12:29:16.295770+0000","last_became_peered":"2023-10-16T12:29:16.295770+0000","last_unstale":"2023-10-16T12:29:16.297271+0000","last_undegraded":"2023-10-16T12:29:16.297271+0000","last_fullsized":"2023-10-16T12:29:16.297271+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:01:47.749457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.304160+0000","last_change":"2023-10-16T12:29:16.300370+0000","last_active":"2023-10-16T12:29:16.304160+0000","last_peered":"2023-10-16T12:29:16.304160+0000","last_clean":"2023-10-16T12:29:16.304160+0000","last_became_active":"2023-10-16T12:29:16.300002+0000","last_became_peered":"2023-10-16T12:29:16.300002+0000","last_unstale":"2023-10-16T12:29:16.304160+0000","last_undegraded":"2023-10-16T12:29:16.304160+0000","last_fullsized":"2023-10-16T12:29:16.304160+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:16:33.614568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"127'4","reported_seq":15,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.381354+0000","last_change":"2023-10-16T12:29:15.387578+0000","last_active":"2023-10-16T12:29:16.381354+0000","last_peered":"2023-10-16T12:29:16.381354+0000","last_clean":"2023-10-16T12:29:16.381354+0000","last_became_active":"2023-10-16T12:29:15.387246+0000","last_became_peered":"2023-10-16T12:29:15.387246+0000","last_unstale":"2023-10-16T12:29:16.381354+0000","last_undegraded":"2023-10-16T12:29:16.381354+0000","last_fullsized":"2023-10-16T12:29:16.381354+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_clean_scrub_stamp":"2023-10-16T12:29:14.359222+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:35:20.655861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.293148+0000","last_change":"2023-10-16T12:29:16.291871+0000","last_active":"2023-10-16T12:29:16.293148+0000","last_peered":"2023-10-16T12:29:16.293148+0000","last_clean":"2023-10-16T12:29:16.293148+0000","last_became_active":"2023-10-16T12:29:16.290827+0000","last_became_peered":"2023-10-16T12:29:16.290827+0000","last_unstale":"2023-10-16T12:29:16.293148+0000","last_undegraded":"2023-10-16T12:29:16.293148+0000","last_fullsized":"2023-10-16T12:29:16.293148+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:08:51.598873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.1e","version":"127'1","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.379677+0000","last_change":"2023-10-16T12:29:15.378884+0000","last_active":"2023-10-16T12:29:16.379677+0000","last_peered":"2023-10-16T12:29:16.379677+0000","last_clean":"2023-10-16T12:29:16.379677+0000","last_became_active":"2023-10-16T12:29:15.378557+0000","last_became_peered":"2023-10-16T12:29:15.378557+0000","last_unstale":"2023-10-16T12:29:16.379677+0000","last_undegraded":"2023-10-16T12:29:16.379677+0000","last_fullsized":"2023-10-16T12:29:16.379677+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_clean_scrub_stamp":"2023-10-16T12:29:14.359222+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:27:41.017234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.298015+0000","last_change":"2023-10-16T12:29:16.295636+0000","last_active":"2023-10-16T12:29:16.298015+0000","last_peered":"2023-10-16T12:29:16.298015+0000","last_clean":"2023-10-16T12:29:16.298015+0000","last_became_active":"2023-10-16T12:29:16.295348+0000","last_became_peered":"2023-10-16T12:29:16.295348+0000","last_unstale":"2023-10-16T12:29:16.298015+0000","last_undegraded":"2023-10-16T12:29:16.298015+0000","last_fullsized":"2023-10-16T12:29:16.298015+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:32:20.674425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1d","version":"127'2","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.381441+0000","last_change":"2023-10-16T12:29:15.389455+0000","last_active":"2023-10-16T12:29:16.381441+0000","last_peered":"2023-10-16T12:29:16.381441+0000","last_clean":"2023-10-16T12:29:16.381441+0000","last_became_active":"2023-10-16T12:29:15.389146+0000","last_became_peered":"2023-10-16T12:29:15.389146+0000","last_unstale":"2023-10-16T12:29:16.381441+0000","last_undegraded":"2023-10-16T12:29:16.381441+0000","last_fullsized":"2023-10-16T12:29:16.381441+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_clean_scrub_stamp":"2023-10-16T12:29:14.359222+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:39:59.922766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.301445+0000","last_change":"2023-10-16T12:29:16.299426+0000","last_active":"2023-10-16T12:29:16.301445+0000","last_peered":"2023-10-16T12:29:16.301445+0000","last_clean":"2023-10-16T12:29:16.301445+0000","last_became_active":"2023-10-16T12:29:16.299099+0000","last_became_peered":"2023-10-16T12:29:16.299099+0000","last_unstale":"2023-10-16T12:29:16.301445+0000","last_undegraded":"2023-10-16T12:29:16.301445+0000","last_fullsized":"2023-10-16T12:29:16.301445+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:02:25.157483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.1c","version":"127'2","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.382231+0000","last_change":"2023-10-16T12:29:15.389220+0000","last_active":"2023-10-16T12:29:16.382231+0000","last_peered":"2023-10-16T12:29:16.382231+0000","last_clean":"2023-10-16T12:29:16.382231+0000","last_became_active":"2023-10-16T12:29:15.388900+0000","last_became_peered":"2023-10-16T12:29:15.388900+0000","last_unstale":"2023-10-16T12:29:16.382231+0000","last_undegraded":"2023-10-16T12:29:16.382231+0000","last_fullsized":"2023-10-16T12:29:16.382231+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_clean_scrub_stamp":"2023-10-16T12:29:14.359222+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:58:30.142694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.286282+0000","last_change":"2023-10-16T12:29:16.283097+0000","last_active":"2023-10-16T12:29:16.286282+0000","last_peered":"2023-10-16T12:29:16.286282+0000","last_clean":"2023-10-16T12:29:16.286282+0000","last_became_active":"2023-10-16T12:29:16.282597+0000","last_became_peered":"2023-10-16T12:29:16.282597+0000","last_unstale":"2023-10-16T12:29:16.286282+0000","last_undegraded":"2023-10-16T12:29:16.286282+0000","last_fullsized":"2023-10-16T12:29:16.286282+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:15:33.049530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1b","version":"127'1","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.380428+0000","last_change":"2023-10-16T12:29:15.382823+0000","last_active":"2023-10-16T12:29:16.380428+0000","last_peered":"2023-10-16T12:29:16.380428+0000","last_clean":"2023-10-16T12:29:16.380428+0000","last_became_active":"2023-10-16T12:29:15.382516+0000","last_became_peered":"2023-10-16T12:29:15.382516+0000","last_unstale":"2023-10-16T12:29:16.380428+0000","last_undegraded":"2023-10-16T12:29:16.380428+0000","last_fullsized":"2023-10-16T12:29:16.380428+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_clean_scrub_stamp":"2023-10-16T12:29:14.359222+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:35:20.655861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.304185+0000","last_change":"2023-10-16T12:29:16.300472+0000","last_active":"2023-10-16T12:29:16.304185+0000","last_peered":"2023-10-16T12:29:16.304185+0000","last_clean":"2023-10-16T12:29:16.304185+0000","last_became_active":"2023-10-16T12:29:16.299823+0000","last_became_peered":"2023-10-16T12:29:16.299823+0000","last_unstale":"2023-10-16T12:29:16.304185+0000","last_undegraded":"2023-10-16T12:29:16.304185+0000","last_fullsized":"2023-10-16T12:29:16.304185+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:46:00.177635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"127'1","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.381539+0000","last_change":"2023-10-16T12:29:15.391284+0000","last_active":"2023-10-16T12:29:16.381539+0000","last_peered":"2023-10-16T12:29:16.381539+0000","last_clean":"2023-10-16T12:29:16.381539+0000","last_became_active":"2023-10-16T12:29:15.390981+0000","last_became_peered":"2023-10-16T12:29:15.390981+0000","last_unstale":"2023-10-16T12:29:16.381539+0000","last_undegraded":"2023-10-16T12:29:16.381539+0000","last_fullsized":"2023-10-16T12:29:16.381539+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_clean_scrub_stamp":"2023-10-16T12:29:14.359222+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:24:50.244826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.304496+0000","last_change":"2023-10-16T12:29:16.303165+0000","last_active":"2023-10-16T12:29:16.304496+0000","last_peered":"2023-10-16T12:29:16.304496+0000","last_clean":"2023-10-16T12:29:16.304496+0000","last_became_active":"2023-10-16T12:29:16.295443+0000","last_became_peered":"2023-10-16T12:29:16.295443+0000","last_unstale":"2023-10-16T12:29:16.304496+0000","last_undegraded":"2023-10-16T12:29:16.304496+0000","last_fullsized":"2023-10-16T12:29:16.304496+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:53:44.212676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.285610+0000","last_change":"2023-10-16T12:29:15.389676+0000","last_active":"2023-10-16T12:29:16.285610+0000","last_peered":"2023-10-16T12:29:16.285610+0000","last_clean":"2023-10-16T12:29:16.285610+0000","last_became_active":"2023-10-16T12:29:15.389359+0000","last_became_peered":"2023-10-16T12:29:15.389359+0000","last_unstale":"2023-10-16T12:29:16.285610+0000","last_undegraded":"2023-10-16T12:29:16.285610+0000","last_fullsized":"2023-10-16T12:29:16.285610+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_clean_scrub_stamp":"2023-10-16T12:29:14.359222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:39:57.128076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.299939+0000","last_change":"2023-10-16T12:29:16.296899+0000","last_active":"2023-10-16T12:29:16.299939+0000","last_peered":"2023-10-16T12:29:16.299939+0000","last_clean":"2023-10-16T12:29:16.299939+0000","last_became_active":"2023-10-16T12:29:16.296569+0000","last_became_peered":"2023-10-16T12:29:16.296569+0000","last_unstale":"2023-10-16T12:29:16.299939+0000","last_undegraded":"2023-10-16T12:29:16.299939+0000","last_fullsized":"2023-10-16T12:29:16.299939+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:47:23.445036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.280980+0000","last_change":"2023-10-16T12:29:15.389613+0000","last_active":"2023-10-16T12:29:16.280980+0000","last_peered":"2023-10-16T12:29:16.280980+0000","last_clean":"2023-10-16T12:29:16.280980+0000","last_became_active":"2023-10-16T12:29:15.389287+0000","last_became_peered":"2023-10-16T12:29:15.389287+0000","last_unstale":"2023-10-16T12:29:16.280980+0000","last_undegraded":"2023-10-16T12:29:16.280980+0000","last_fullsized":"2023-10-16T12:29:16.280980+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_clean_scrub_stamp":"2023-10-16T12:29:14.359222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:30:18.319981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.304211+0000","last_change":"2023-10-16T12:29:16.300614+0000","last_active":"2023-10-16T12:29:16.304211+0000","last_peered":"2023-10-16T12:29:16.304211+0000","last_clean":"2023-10-16T12:29:16.304211+0000","last_became_active":"2023-10-16T12:29:16.300315+0000","last_became_peered":"2023-10-16T12:29:16.300315+0000","last_unstale":"2023-10-16T12:29:16.304211+0000","last_undegraded":"2023-10-16T12:29:16.304211+0000","last_fullsized":"2023-10-16T12:29:16.304211+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:38:24.159543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.289816+0000","last_change":"2023-10-16T12:29:15.390204+0000","last_active":"2023-10-16T12:29:16.289816+0000","last_peered":"2023-10-16T12:29:16.289816+0000","last_clean":"2023-10-16T12:29:16.289816+0000","last_became_active":"2023-10-16T12:29:15.389919+0000","last_became_peered":"2023-10-16T12:29:15.389919+0000","last_unstale":"2023-10-16T12:29:16.289816+0000","last_undegraded":"2023-10-16T12:29:16.289816+0000","last_fullsized":"2023-10-16T12:29:16.289816+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_clean_scrub_stamp":"2023-10-16T12:29:14.359222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:59:49.605236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.290159+0000","last_change":"2023-10-16T12:29:16.287910+0000","last_active":"2023-10-16T12:29:16.290159+0000","last_peered":"2023-10-16T12:29:16.290159+0000","last_clean":"2023-10-16T12:29:16.290159+0000","last_became_active":"2023-10-16T12:29:16.287626+0000","last_became_peered":"2023-10-16T12:29:16.287626+0000","last_unstale":"2023-10-16T12:29:16.290159+0000","last_undegraded":"2023-10-16T12:29:16.290159+0000","last_fullsized":"2023-10-16T12:29:16.290159+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:56:18.062438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.292896+0000","last_change":"2023-10-16T12:29:15.392631+0000","last_active":"2023-10-16T12:29:16.292896+0000","last_peered":"2023-10-16T12:29:16.292896+0000","last_clean":"2023-10-16T12:29:16.292896+0000","last_became_active":"2023-10-16T12:29:15.392138+0000","last_became_peered":"2023-10-16T12:29:15.392138+0000","last_unstale":"2023-10-16T12:29:16.292896+0000","last_undegraded":"2023-10-16T12:29:16.292896+0000","last_fullsized":"2023-10-16T12:29:16.292896+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_clean_scrub_stamp":"2023-10-16T12:29:14.359222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:38:34.083984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.292564+0000","last_change":"2023-10-16T12:29:16.291543+0000","last_active":"2023-10-16T12:29:16.292564+0000","last_peered":"2023-10-16T12:29:16.292564+0000","last_clean":"2023-10-16T12:29:16.292564+0000","last_became_active":"2023-10-16T12:29:16.291290+0000","last_became_peered":"2023-10-16T12:29:16.291290+0000","last_unstale":"2023-10-16T12:29:16.292564+0000","last_undegraded":"2023-10-16T12:29:16.292564+0000","last_fullsized":"2023-10-16T12:29:16.292564+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:52:12.812015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.15","version":"127'3","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.381581+0000","last_change":"2023-10-16T12:29:15.393973+0000","last_active":"2023-10-16T12:29:16.381581+0000","last_peered":"2023-10-16T12:29:16.381581+0000","last_clean":"2023-10-16T12:29:16.381581+0000","last_became_active":"2023-10-16T12:29:15.393692+0000","last_became_peered":"2023-10-16T12:29:15.393692+0000","last_unstale":"2023-10-16T12:29:16.381581+0000","last_undegraded":"2023-10-16T12:29:16.381581+0000","last_fullsized":"2023-10-16T12:29:16.381581+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_clean_scrub_stamp":"2023-10-16T12:29:14.359222+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:24:31.663023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.292791+0000","last_change":"2023-10-16T12:29:16.291588+0000","last_active":"2023-10-16T12:29:16.292791+0000","last_peered":"2023-10-16T12:29:16.292791+0000","last_clean":"2023-10-16T12:29:16.292791+0000","last_became_active":"2023-10-16T12:29:16.291308+0000","last_became_peered":"2023-10-16T12:29:16.291308+0000","last_unstale":"2023-10-16T12:29:16.292791+0000","last_undegraded":"2023-10-16T12:29:16.292791+0000","last_fullsized":"2023-10-16T12:29:16.292791+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:39:26.840175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.14","version":"127'6","reported_seq":17,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.383321+0000","last_change":"2023-10-16T12:29:15.392511+0000","last_active":"2023-10-16T12:29:16.383321+0000","last_peered":"2023-10-16T12:29:16.383321+0000","last_clean":"2023-10-16T12:29:16.383321+0000","last_became_active":"2023-10-16T12:29:15.391941+0000","last_became_peered":"2023-10-16T12:29:15.391941+0000","last_unstale":"2023-10-16T12:29:16.383321+0000","last_undegraded":"2023-10-16T12:29:16.383321+0000","last_fullsized":"2023-10-16T12:29:16.383321+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_clean_scrub_stamp":"2023-10-16T12:29:14.359222+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:58:02.589102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.301383+0000","last_change":"2023-10-16T12:29:16.298832+0000","last_active":"2023-10-16T12:29:16.301383+0000","last_peered":"2023-10-16T12:29:16.301383+0000","last_clean":"2023-10-16T12:29:16.301383+0000","last_became_active":"2023-10-16T12:29:16.298089+0000","last_became_peered":"2023-10-16T12:29:16.298089+0000","last_unstale":"2023-10-16T12:29:16.301383+0000","last_undegraded":"2023-10-16T12:29:16.301383+0000","last_fullsized":"2023-10-16T12:29:16.301383+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:25:32.385405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.13","version":"127'2","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.382317+0000","last_change":"2023-10-16T12:29:15.384560+0000","last_active":"2023-10-16T12:29:16.382317+0000","last_peered":"2023-10-16T12:29:16.382317+0000","last_clean":"2023-10-16T12:29:16.382317+0000","last_became_active":"2023-10-16T12:29:15.384283+0000","last_became_peered":"2023-10-16T12:29:15.384283+0000","last_unstale":"2023-10-16T12:29:16.382317+0000","last_undegraded":"2023-10-16T12:29:16.382317+0000","last_fullsized":"2023-10-16T12:29:16.382317+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_clean_scrub_stamp":"2023-10-16T12:29:14.359222+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:21:31.258287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.293300+0000","last_change":"2023-10-16T12:29:16.292325+0000","last_active":"2023-10-16T12:29:16.293300+0000","last_peered":"2023-10-16T12:29:16.293300+0000","last_clean":"2023-10-16T12:29:16.293300+0000","last_became_active":"2023-10-16T12:29:16.291480+0000","last_became_peered":"2023-10-16T12:29:16.291480+0000","last_unstale":"2023-10-16T12:29:16.293300+0000","last_undegraded":"2023-10-16T12:29:16.293300+0000","last_fullsized":"2023-10-16T12:29:16.293300+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:07:26.551366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.281410+0000","last_change":"2023-10-16T12:29:15.389559+0000","last_active":"2023-10-16T12:29:16.281410+0000","last_peered":"2023-10-16T12:29:16.281410+0000","last_clean":"2023-10-16T12:29:16.281410+0000","last_became_active":"2023-10-16T12:29:15.389268+0000","last_became_peered":"2023-10-16T12:29:15.389268+0000","last_unstale":"2023-10-16T12:29:16.281410+0000","last_undegraded":"2023-10-16T12:29:16.281410+0000","last_fullsized":"2023-10-16T12:29:16.281410+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_clean_scrub_stamp":"2023-10-16T12:29:14.359222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:40:16.898102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.297098+0000","last_change":"2023-10-16T12:29:16.296508+0000","last_active":"2023-10-16T12:29:16.297098+0000","last_peered":"2023-10-16T12:29:16.297098+0000","last_clean":"2023-10-16T12:29:16.297098+0000","last_became_active":"2023-10-16T12:29:16.296228+0000","last_became_peered":"2023-10-16T12:29:16.296228+0000","last_unstale":"2023-10-16T12:29:16.297098+0000","last_undegraded":"2023-10-16T12:29:16.297098+0000","last_fullsized":"2023-10-16T12:29:16.297098+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:36:12.325088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.284044+0000","last_change":"2023-10-16T12:29:15.387981+0000","last_active":"2023-10-16T12:29:16.284044+0000","last_peered":"2023-10-16T12:29:16.284044+0000","last_clean":"2023-10-16T12:29:16.284044+0000","last_became_active":"2023-10-16T12:29:15.387545+0000","last_became_peered":"2023-10-16T12:29:15.387545+0000","last_unstale":"2023-10-16T12:29:16.284044+0000","last_undegraded":"2023-10-16T12:29:16.284044+0000","last_fullsized":"2023-10-16T12:29:16.284044+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_clean_scrub_stamp":"2023-10-16T12:29:14.359222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:49:24.712423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.303659+0000","last_change":"2023-10-16T12:29:16.295754+0000","last_active":"2023-10-16T12:29:16.303659+0000","last_peered":"2023-10-16T12:29:16.303659+0000","last_clean":"2023-10-16T12:29:16.303659+0000","last_became_active":"2023-10-16T12:29:16.295492+0000","last_became_peered":"2023-10-16T12:29:16.295492+0000","last_unstale":"2023-10-16T12:29:16.303659+0000","last_undegraded":"2023-10-16T12:29:16.303659+0000","last_fullsized":"2023-10-16T12:29:16.303659+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:25:45.881535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.10","version":"127'8","reported_seq":19,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.382446+0000","last_change":"2023-10-16T12:29:15.389747+0000","last_active":"2023-10-16T12:29:16.382446+0000","last_peered":"2023-10-16T12:29:16.382446+0000","last_clean":"2023-10-16T12:29:16.382446+0000","last_became_active":"2023-10-16T12:29:15.389112+0000","last_became_peered":"2023-10-16T12:29:15.389112+0000","last_unstale":"2023-10-16T12:29:16.382446+0000","last_undegraded":"2023-10-16T12:29:16.382446+0000","last_fullsized":"2023-10-16T12:29:16.382446+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_clean_scrub_stamp":"2023-10-16T12:29:14.359222+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:38:24.764875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.301098+0000","last_change":"2023-10-16T12:29:16.300770+0000","last_active":"2023-10-16T12:29:16.301098+0000","last_peered":"2023-10-16T12:29:16.301098+0000","last_clean":"2023-10-16T12:29:16.301098+0000","last_became_active":"2023-10-16T12:29:16.300498+0000","last_became_peered":"2023-10-16T12:29:16.300498+0000","last_unstale":"2023-10-16T12:29:16.301098+0000","last_undegraded":"2023-10-16T12:29:16.301098+0000","last_fullsized":"2023-10-16T12:29:16.301098+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:15:33.970346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.f","version":"127'1","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.380527+0000","last_change":"2023-10-16T12:29:15.386345+0000","last_active":"2023-10-16T12:29:16.380527+0000","last_peered":"2023-10-16T12:29:16.380527+0000","last_clean":"2023-10-16T12:29:16.380527+0000","last_became_active":"2023-10-16T12:29:15.386052+0000","last_became_peered":"2023-10-16T12:29:15.386052+0000","last_unstale":"2023-10-16T12:29:16.380527+0000","last_undegraded":"2023-10-16T12:29:16.380527+0000","last_fullsized":"2023-10-16T12:29:16.380527+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_clean_scrub_stamp":"2023-10-16T12:29:14.359222+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:12:16.756683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.295340+0000","last_change":"2023-10-16T12:29:16.293616+0000","last_active":"2023-10-16T12:29:16.295340+0000","last_peered":"2023-10-16T12:29:16.295340+0000","last_clean":"2023-10-16T12:29:16.295340+0000","last_became_active":"2023-10-16T12:29:16.293202+0000","last_became_peered":"2023-10-16T12:29:16.293202+0000","last_unstale":"2023-10-16T12:29:16.295340+0000","last_undegraded":"2023-10-16T12:29:16.295340+0000","last_fullsized":"2023-10-16T12:29:16.295340+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:49:07.841401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"127'1","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.381657+0000","last_change":"2023-10-16T12:29:15.388508+0000","last_active":"2023-10-16T12:29:16.381657+0000","last_peered":"2023-10-16T12:29:16.381657+0000","last_clean":"2023-10-16T12:29:16.381657+0000","last_became_active":"2023-10-16T12:29:15.388179+0000","last_became_peered":"2023-10-16T12:29:15.388179+0000","last_unstale":"2023-10-16T12:29:16.381657+0000","last_undegraded":"2023-10-16T12:29:16.381657+0000","last_fullsized":"2023-10-16T12:29:16.381657+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_clean_scrub_stamp":"2023-10-16T12:29:14.359222+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:15:32.044492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.299045+0000","last_change":"2023-10-16T12:29:16.298101+0000","last_active":"2023-10-16T12:29:16.299045+0000","last_peered":"2023-10-16T12:29:16.299045+0000","last_clean":"2023-10-16T12:29:16.299045+0000","last_became_active":"2023-10-16T12:29:16.297814+0000","last_became_peered":"2023-10-16T12:29:16.297814+0000","last_unstale":"2023-10-16T12:29:16.299045+0000","last_undegraded":"2023-10-16T12:29:16.299045+0000","last_fullsized":"2023-10-16T12:29:16.299045+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:58:27.254916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.284766+0000","last_change":"2023-10-16T12:29:15.387284+0000","last_active":"2023-10-16T12:29:16.284766+0000","last_peered":"2023-10-16T12:29:16.284766+0000","last_clean":"2023-10-16T12:29:16.284766+0000","last_became_active":"2023-10-16T12:29:15.386957+0000","last_became_peered":"2023-10-16T12:29:15.386957+0000","last_unstale":"2023-10-16T12:29:16.284766+0000","last_undegraded":"2023-10-16T12:29:16.284766+0000","last_fullsized":"2023-10-16T12:29:16.284766+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_clean_scrub_stamp":"2023-10-16T12:29:14.359222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:47:28.957497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.300077+0000","last_change":"2023-10-16T12:29:16.295453+0000","last_active":"2023-10-16T12:29:16.300077+0000","last_peered":"2023-10-16T12:29:16.300077+0000","last_clean":"2023-10-16T12:29:16.300077+0000","last_became_active":"2023-10-16T12:29:16.295177+0000","last_became_peered":"2023-10-16T12:29:16.295177+0000","last_unstale":"2023-10-16T12:29:16.300077+0000","last_undegraded":"2023-10-16T12:29:16.300077+0000","last_fullsized":"2023-10-16T12:29:16.300077+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:32:17.862905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"127'2","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.377640+0000","last_change":"2023-10-16T12:29:15.384862+0000","last_active":"2023-10-16T12:29:16.377640+0000","last_peered":"2023-10-16T12:29:16.377640+0000","last_clean":"2023-10-16T12:29:16.377640+0000","last_became_active":"2023-10-16T12:29:15.384500+0000","last_became_peered":"2023-10-16T12:29:15.384500+0000","last_unstale":"2023-10-16T12:29:16.377640+0000","last_undegraded":"2023-10-16T12:29:16.377640+0000","last_fullsized":"2023-10-16T12:29:16.377640+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_clean_scrub_stamp":"2023-10-16T12:29:14.359222+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:11:28.075541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.303777+0000","last_change":"2023-10-16T12:29:16.297361+0000","last_active":"2023-10-16T12:29:16.303777+0000","last_peered":"2023-10-16T12:29:16.303777+0000","last_clean":"2023-10-16T12:29:16.303777+0000","last_became_active":"2023-10-16T12:29:16.293132+0000","last_became_peered":"2023-10-16T12:29:16.293132+0000","last_unstale":"2023-10-16T12:29:16.303777+0000","last_undegraded":"2023-10-16T12:29:16.303777+0000","last_fullsized":"2023-10-16T12:29:16.303777+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:46:48.194963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.284152+0000","last_change":"2023-10-16T12:29:15.390086+0000","last_active":"2023-10-16T12:29:16.284152+0000","last_peered":"2023-10-16T12:29:16.284152+0000","last_clean":"2023-10-16T12:29:16.284152+0000","last_became_active":"2023-10-16T12:29:15.389832+0000","last_became_peered":"2023-10-16T12:29:15.389832+0000","last_unstale":"2023-10-16T12:29:16.284152+0000","last_undegraded":"2023-10-16T12:29:16.284152+0000","last_fullsized":"2023-10-16T12:29:16.284152+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_clean_scrub_stamp":"2023-10-16T12:29:14.359222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:45:00.939929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.293989+0000","last_change":"2023-10-16T12:29:16.293273+0000","last_active":"2023-10-16T12:29:16.293989+0000","last_peered":"2023-10-16T12:29:16.293989+0000","last_clean":"2023-10-16T12:29:16.293989+0000","last_became_active":"2023-10-16T12:29:16.292979+0000","last_became_peered":"2023-10-16T12:29:16.292979+0000","last_unstale":"2023-10-16T12:29:16.293989+0000","last_undegraded":"2023-10-16T12:29:16.293989+0000","last_fullsized":"2023-10-16T12:29:16.293989+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:24:32.668061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.284025+0000","last_change":"2023-10-16T12:29:15.389602+0000","last_active":"2023-10-16T12:29:16.284025+0000","last_peered":"2023-10-16T12:29:16.284025+0000","last_clean":"2023-10-16T12:29:16.284025+0000","last_became_active":"2023-10-16T12:29:15.389318+0000","last_became_peered":"2023-10-16T12:29:15.389318+0000","last_unstale":"2023-10-16T12:29:16.284025+0000","last_undegraded":"2023-10-16T12:29:16.284025+0000","last_fullsized":"2023-10-16T12:29:16.284025+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_clean_scrub_stamp":"2023-10-16T12:29:14.359222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:22:46.277491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.293186+0000","last_change":"2023-10-16T12:29:16.292152+0000","last_active":"2023-10-16T12:29:16.293186+0000","last_peered":"2023-10-16T12:29:16.293186+0000","last_clean":"2023-10-16T12:29:16.293186+0000","last_became_active":"2023-10-16T12:29:16.290631+0000","last_became_peered":"2023-10-16T12:29:16.290631+0000","last_unstale":"2023-10-16T12:29:16.293186+0000","last_undegraded":"2023-10-16T12:29:16.293186+0000","last_fullsized":"2023-10-16T12:29:16.293186+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:27:43.984098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.284569+0000","last_change":"2023-10-16T12:29:15.390056+0000","last_active":"2023-10-16T12:29:16.284569+0000","last_peered":"2023-10-16T12:29:16.284569+0000","last_clean":"2023-10-16T12:29:16.284569+0000","last_became_active":"2023-10-16T12:29:15.389777+0000","last_became_peered":"2023-10-16T12:29:15.389777+0000","last_unstale":"2023-10-16T12:29:16.284569+0000","last_undegraded":"2023-10-16T12:29:16.284569+0000","last_fullsized":"2023-10-16T12:29:16.284569+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_clean_scrub_stamp":"2023-10-16T12:29:14.359222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:12:28.782296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.304197+0000","last_change":"2023-10-16T12:29:16.300538+0000","last_active":"2023-10-16T12:29:16.304197+0000","last_peered":"2023-10-16T12:29:16.304197+0000","last_clean":"2023-10-16T12:29:16.304197+0000","last_became_active":"2023-10-16T12:29:16.300029+0000","last_became_peered":"2023-10-16T12:29:16.300029+0000","last_unstale":"2023-10-16T12:29:16.304197+0000","last_undegraded":"2023-10-16T12:29:16.304197+0000","last_fullsized":"2023-10-16T12:29:16.304197+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:11:30.050315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.7","version":"127'2","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.382315+0000","last_change":"2023-10-16T12:29:15.385060+0000","last_active":"2023-10-16T12:29:16.382315+0000","last_peered":"2023-10-16T12:29:16.382315+0000","last_clean":"2023-10-16T12:29:16.382315+0000","last_became_active":"2023-10-16T12:29:15.384763+0000","last_became_peered":"2023-10-16T12:29:15.384763+0000","last_unstale":"2023-10-16T12:29:16.382315+0000","last_undegraded":"2023-10-16T12:29:16.382315+0000","last_fullsized":"2023-10-16T12:29:16.382315+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_clean_scrub_stamp":"2023-10-16T12:29:14.359222+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:48:43.877276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.299892+0000","last_change":"2023-10-16T12:29:16.296485+0000","last_active":"2023-10-16T12:29:16.299892+0000","last_peered":"2023-10-16T12:29:16.299892+0000","last_clean":"2023-10-16T12:29:16.299892+0000","last_became_active":"2023-10-16T12:29:16.296178+0000","last_became_peered":"2023-10-16T12:29:16.296178+0000","last_unstale":"2023-10-16T12:29:16.299892+0000","last_undegraded":"2023-10-16T12:29:16.299892+0000","last_fullsized":"2023-10-16T12:29:16.299892+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:33:28.001948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.292863+0000","last_change":"2023-10-16T12:29:15.392423+0000","last_active":"2023-10-16T12:29:16.292863+0000","last_peered":"2023-10-16T12:29:16.292863+0000","last_clean":"2023-10-16T12:29:16.292863+0000","last_became_active":"2023-10-16T12:29:15.391765+0000","last_became_peered":"2023-10-16T12:29:15.391765+0000","last_unstale":"2023-10-16T12:29:16.292863+0000","last_undegraded":"2023-10-16T12:29:16.292863+0000","last_fullsized":"2023-10-16T12:29:16.292863+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_clean_scrub_stamp":"2023-10-16T12:29:14.359222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:17:59.782860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.293944+0000","last_change":"2023-10-16T12:29:16.292605+0000","last_active":"2023-10-16T12:29:16.293944+0000","last_peered":"2023-10-16T12:29:16.293944+0000","last_clean":"2023-10-16T12:29:16.293944+0000","last_became_active":"2023-10-16T12:29:16.292293+0000","last_became_peered":"2023-10-16T12:29:16.292293+0000","last_unstale":"2023-10-16T12:29:16.293944+0000","last_undegraded":"2023-10-16T12:29:16.293944+0000","last_fullsized":"2023-10-16T12:29:16.293944+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:03:19.526603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.283615+0000","last_change":"2023-10-16T12:29:15.390558+0000","last_active":"2023-10-16T12:29:16.283615+0000","last_peered":"2023-10-16T12:29:16.283615+0000","last_clean":"2023-10-16T12:29:16.283615+0000","last_became_active":"2023-10-16T12:29:15.390202+0000","last_became_peered":"2023-10-16T12:29:15.390202+0000","last_unstale":"2023-10-16T12:29:16.283615+0000","last_undegraded":"2023-10-16T12:29:16.283615+0000","last_fullsized":"2023-10-16T12:29:16.283615+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_clean_scrub_stamp":"2023-10-16T12:29:14.359222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:59:20.248769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.295964+0000","last_change":"2023-10-16T12:29:16.294448+0000","last_active":"2023-10-16T12:29:16.295964+0000","last_peered":"2023-10-16T12:29:16.295964+0000","last_clean":"2023-10-16T12:29:16.295964+0000","last_became_active":"2023-10-16T12:29:16.294105+0000","last_became_peered":"2023-10-16T12:29:16.294105+0000","last_unstale":"2023-10-16T12:29:16.295964+0000","last_undegraded":"2023-10-16T12:29:16.295964+0000","last_fullsized":"2023-10-16T12:29:16.295964+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:20:05.649120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.279301+0000","last_change":"2023-10-16T12:29:15.388073+0000","last_active":"2023-10-16T12:29:16.279301+0000","last_peered":"2023-10-16T12:29:16.279301+0000","last_clean":"2023-10-16T12:29:16.279301+0000","last_became_active":"2023-10-16T12:29:15.387780+0000","last_became_peered":"2023-10-16T12:29:15.387780+0000","last_unstale":"2023-10-16T12:29:16.279301+0000","last_undegraded":"2023-10-16T12:29:16.279301+0000","last_fullsized":"2023-10-16T12:29:16.279301+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_clean_scrub_stamp":"2023-10-16T12:29:14.359222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:58:02.589102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.292745+0000","last_change":"2023-10-16T12:29:16.291463+0000","last_active":"2023-10-16T12:29:16.292745+0000","last_peered":"2023-10-16T12:29:16.292745+0000","last_clean":"2023-10-16T12:29:16.292745+0000","last_became_active":"2023-10-16T12:29:16.291203+0000","last_became_peered":"2023-10-16T12:29:16.291203+0000","last_unstale":"2023-10-16T12:29:16.292745+0000","last_undegraded":"2023-10-16T12:29:16.292745+0000","last_fullsized":"2023-10-16T12:29:16.292745+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:18:44.204916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.289137+0000","last_change":"2023-10-16T12:29:15.384127+0000","last_active":"2023-10-16T12:29:16.289137+0000","last_peered":"2023-10-16T12:29:16.289137+0000","last_clean":"2023-10-16T12:29:16.289137+0000","last_became_active":"2023-10-16T12:29:15.383826+0000","last_became_peered":"2023-10-16T12:29:15.383826+0000","last_unstale":"2023-10-16T12:29:16.289137+0000","last_undegraded":"2023-10-16T12:29:16.289137+0000","last_fullsized":"2023-10-16T12:29:16.289137+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_clean_scrub_stamp":"2023-10-16T12:29:14.359222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:42:36.058919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.292855+0000","last_change":"2023-10-16T12:29:16.290924+0000","last_active":"2023-10-16T12:29:16.292855+0000","last_peered":"2023-10-16T12:29:16.292855+0000","last_clean":"2023-10-16T12:29:16.292855+0000","last_became_active":"2023-10-16T12:29:16.290643+0000","last_became_peered":"2023-10-16T12:29:16.290643+0000","last_unstale":"2023-10-16T12:29:16.292855+0000","last_undegraded":"2023-10-16T12:29:16.292855+0000","last_fullsized":"2023-10-16T12:29:16.292855+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:35:11.392279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.284316+0000","last_change":"2023-10-16T12:29:15.391874+0000","last_active":"2023-10-16T12:29:16.284316+0000","last_peered":"2023-10-16T12:29:16.284316+0000","last_clean":"2023-10-16T12:29:16.284316+0000","last_became_active":"2023-10-16T12:29:15.391527+0000","last_became_peered":"2023-10-16T12:29:15.391527+0000","last_unstale":"2023-10-16T12:29:16.284316+0000","last_undegraded":"2023-10-16T12:29:16.284316+0000","last_fullsized":"2023-10-16T12:29:16.284316+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_clean_scrub_stamp":"2023-10-16T12:29:14.359222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:47:27.045462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.299696+0000","last_change":"2023-10-16T12:29:16.297340+0000","last_active":"2023-10-16T12:29:16.299696+0000","last_peered":"2023-10-16T12:29:16.299696+0000","last_clean":"2023-10-16T12:29:16.299696+0000","last_became_active":"2023-10-16T12:29:16.297009+0000","last_became_peered":"2023-10-16T12:29:16.297009+0000","last_unstale":"2023-10-16T12:29:16.299696+0000","last_undegraded":"2023-10-16T12:29:16.299696+0000","last_fullsized":"2023-10-16T12:29:16.299696+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:30:19.325019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.284077+0000","last_change":"2023-10-16T12:29:15.385906+0000","last_active":"2023-10-16T12:29:16.284077+0000","last_peered":"2023-10-16T12:29:16.284077+0000","last_clean":"2023-10-16T12:29:16.284077+0000","last_became_active":"2023-10-16T12:29:15.385603+0000","last_became_peered":"2023-10-16T12:29:15.385603+0000","last_unstale":"2023-10-16T12:29:16.284077+0000","last_undegraded":"2023-10-16T12:29:16.284077+0000","last_fullsized":"2023-10-16T12:29:16.284077+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_clean_scrub_stamp":"2023-10-16T12:29:14.359222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:49:09.895788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.298686+0000","last_change":"2023-10-16T12:29:16.296586+0000","last_active":"2023-10-16T12:29:16.298686+0000","last_peered":"2023-10-16T12:29:16.298686+0000","last_clean":"2023-10-16T12:29:16.298686+0000","last_became_active":"2023-10-16T12:29:16.296300+0000","last_became_peered":"2023-10-16T12:29:16.296300+0000","last_unstale":"2023-10-16T12:29:16.298686+0000","last_undegraded":"2023-10-16T12:29:16.298686+0000","last_fullsized":"2023-10-16T12:29:16.298686+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:40:07.478573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.286316+0000","last_change":"2023-10-16T12:29:15.387886+0000","last_active":"2023-10-16T12:29:16.286316+0000","last_peered":"2023-10-16T12:29:16.286316+0000","last_clean":"2023-10-16T12:29:16.286316+0000","last_became_active":"2023-10-16T12:29:15.387641+0000","last_became_peered":"2023-10-16T12:29:15.387641+0000","last_unstale":"2023-10-16T12:29:16.286316+0000","last_undegraded":"2023-10-16T12:29:16.286316+0000","last_fullsized":"2023-10-16T12:29:16.286316+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_clean_scrub_stamp":"2023-10-16T12:29:14.359222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:24:38.015313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.301217+0000","last_change":"2023-10-16T12:29:16.298167+0000","last_active":"2023-10-16T12:29:16.301217+0000","last_peered":"2023-10-16T12:29:16.301217+0000","last_clean":"2023-10-16T12:29:16.301217+0000","last_became_active":"2023-10-16T12:29:16.297855+0000","last_became_peered":"2023-10-16T12:29:16.297855+0000","last_unstale":"2023-10-16T12:29:16.301217+0000","last_undegraded":"2023-10-16T12:29:16.301217+0000","last_fullsized":"2023-10-16T12:29:16.301217+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:46:09.057491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.e","version":"127'2","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.381173+0000","last_change":"2023-10-16T12:29:15.389554+0000","last_active":"2023-10-16T12:29:16.381173+0000","last_peered":"2023-10-16T12:29:16.381173+0000","last_clean":"2023-10-16T12:29:16.381173+0000","last_became_active":"2023-10-16T12:29:15.388188+0000","last_became_peered":"2023-10-16T12:29:15.388188+0000","last_unstale":"2023-10-16T12:29:16.381173+0000","last_undegraded":"2023-10-16T12:29:16.381173+0000","last_fullsized":"2023-10-16T12:29:16.381173+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_clean_scrub_stamp":"2023-10-16T12:29:14.359222+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:42:36.995029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"117'205","reported_seq":576,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T12:29:15.377559+0000","last_change":"2023-10-16T12:10:34.746820+0000","last_active":"2023-10-16T12:29:15.377559+0000","last_peered":"2023-10-16T12:29:15.377559+0000","last_clean":"2023-10-16T12:29:15.377559+0000","last_became_active":"2023-10-16T12:10:34.745574+0000","last_became_peered":"2023-10-16T12:10:34.745574+0000","last_unstale":"2023-10-16T12:29:15.377559+0000","last_undegraded":"2023-10-16T12:29:15.377559+0000","last_fullsized":"2023-10-16T12:29:15.377559+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_clean_scrub_stamp":"2023-10-16T12:10:33.577111+0000","objects_scrubbed":0,"log_size":205,"log_dups_size":0,"ondisk_log_size":205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:14.461986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":884,"num_read_kb":1748,"num_write":319,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":16,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":15,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":884,"num_read_kb":1748,"num_write":319,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":205,"ondisk_log_size":205,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575071,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63836,"kb_used_data":2124,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94308004,"statfs":{"total":96636764160,"available":96571396096,"internally_reserved":0,"allocated":2174976,"data_stored":1382234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":63175115},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:28:12 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.59199999999999997,"15min":0.60399999999999998},"min":{"1min":0.39400000000000002,"5min":0.31,"15min":0.27400000000000002},"max":{"1min":4.2469999999999999,"5min":4.2469999999999999,"15min":4.2469999999999999},"last":0.5},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.56100000000000005,"15min":0.58299999999999996},"min":{"1min":0.40100000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.746,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.84099999999999997}]},{"osd":1,"last update":"Mon Oct 16 12:28:12 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.56499999999999995,"15min":0.60299999999999998},"min":{"1min":0.38100000000000001,"5min":0.35499999999999998,"15min":0.247},"max":{"1min":0.69299999999999995,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.58899999999999997,"15min":0.60599999999999998},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.32400000000000001},"max":{"1min":0.79100000000000004,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.66800000000000004}]},{"osd":2,"last update":"Mon Oct 16 12:28:12 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.63700000000000001,"15min":0.66000000000000003},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.36099999999999999},"max":{"1min":0.77300000000000002,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.64200000000000002,"15min":0.66700000000000004},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.376},"max":{"1min":0.76600000000000001,"5min":0.94599999999999995,"15min":0.96699999999999997},"last":0.75800000000000001}]},{"osd":3,"last update":"Mon Oct 16 12:28:12 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.59799999999999998,"15min":0.61699999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.80200000000000005,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.64000000000000001,"15min":0.63900000000000001},"min":{"1min":0.42999999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.82899999999999996,"5min":1.2509999999999999,"15min":1.2509999999999999},"last":0.57199999999999995}]},{"osd":4,"last update":"Mon Oct 16 12:28:12 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.55000000000000004,"15min":0.52700000000000002},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.89700000000000002,"5min":1.383,"15min":1.383},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.621,"5min":0.63100000000000001,"15min":0.60699999999999998},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.86699999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.59899999999999998}]},{"osd":5,"last update":"Mon Oct 16 12:28:12 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.57599999999999996,"15min":0.53900000000000003},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.74199999999999999,"5min":1.3109999999999999,"15min":1.3109999999999999},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.60399999999999998,"15min":0.58299999999999996},"min":{"1min":0.35399999999999998,"5min":0.33500000000000002,"15min":0.32400000000000001},"max":{"1min":0.84599999999999997,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":0.55500000000000005}]},{"osd":6,"last update":"Mon Oct 16 12:28:12 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.64500000000000002,"15min":0.62},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.80500000000000005,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.64800000000000002,"15min":0.626},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.83399999999999996,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.38200000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575069,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36864,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334976,"statfs":{"total":96636764160,"available":96599015424,"internally_reserved":0,"allocated":1560576,"data_stored":784583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.56799999999999995,"15min":0.61099999999999999},"min":{"1min":0.33800000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.80700000000000005,"5min":0.88200000000000001,"15min":1.196},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.58199999999999996,"15min":0.623},"min":{"1min":0.40799999999999997,"5min":0.38500000000000001,"15min":0.33600000000000002},"max":{"1min":0.78900000000000003,"5min":0.83399999999999996,"15min":0.997},"last":0.505}]},{"osd":1,"last update":"Mon Oct 16 12:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.53000000000000003,"15min":0.56699999999999995},"min":{"1min":0.40600000000000003,"5min":0.33100000000000002,"15min":0.29399999999999998},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.875},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.623,"5min":0.55500000000000005,"15min":0.58799999999999997},"min":{"1min":0.36699999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":1.091},"last":0.52700000000000002}]},{"osd":2,"last update":"Mon Oct 16 12:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.56499999999999995,"15min":0.59099999999999997},"min":{"1min":0.433,"5min":0.35599999999999998,"15min":0.34499999999999997},"max":{"1min":0.77200000000000002,"5min":0.81499999999999995,"15min":0.97099999999999997},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.55400000000000005,"15min":0.59999999999999998},"min":{"1min":0.41799999999999998,"5min":0.34999999999999998,"15min":0.27100000000000002},"max":{"1min":0.76500000000000001,"5min":0.85999999999999999,"15min":2.3879999999999999},"last":0.44700000000000001}]},{"osd":3,"last update":"Mon Oct 16 12:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.59499999999999997,"15min":0.625},"min":{"1min":0.38600000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.85499999999999998,"5min":0.88,"15min":0.93999999999999995},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.65800000000000003,"15min":0.68100000000000005},"min":{"1min":0.52700000000000002,"5min":0.41499999999999998,"15min":0.40899999999999997},"max":{"1min":0.83899999999999997,"5min":0.94499999999999995,"15min":1.3080000000000001},"last":0.58599999999999997}]},{"osd":4,"last update":"Mon Oct 16 12:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.53100000000000003,"15min":0.51900000000000002},"min":{"1min":0.35299999999999998,"5min":0.30599999999999999,"15min":0.29899999999999999},"max":{"1min":0.79100000000000004,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.623,"5min":0.59399999999999997,"15min":0.59999999999999998},"min":{"1min":0.34000000000000002,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.81599999999999995,"5min":0.82299999999999995,"15min":1.2250000000000001},"last":0.56699999999999995}]},{"osd":5,"last update":"Mon Oct 16 12:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.58899999999999997,"15min":0.57799999999999996},"min":{"1min":0.36899999999999999,"5min":0.30299999999999999,"15min":0.30099999999999999},"max":{"1min":0.86199999999999999,"5min":0.98799999999999999,"15min":1.0589999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.63300000000000001,"15min":0.63},"min":{"1min":0.45000000000000001,"5min":0.32100000000000001,"15min":0.311},"max":{"1min":0.88200000000000001,"5min":1.0029999999999999,"15min":1.3540000000000001},"last":0.63300000000000001}]},{"osd":7,"last update":"Mon Oct 16 12:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.57999999999999996,"15min":0.57699999999999996},"min":{"1min":0.42699999999999999,"5min":0.29699999999999999,"15min":0.26000000000000001},"max":{"1min":0.83299999999999996,"5min":0.92700000000000005,"15min":1.29},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.622,"15min":0.62},"min":{"1min":0.48399999999999999,"5min":0.32900000000000001,"15min":0.29799999999999999},"max":{"1min":0.89900000000000002,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.61399999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575069,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32708,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339132,"statfs":{"total":96636764160,"available":96603271168,"internally_reserved":0,"allocated":1560576,"data_stored":784583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":31914441},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:29:06 2023","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.375,"15min":0.38100000000000001},"min":{"1min":0.29499999999999998,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.66000000000000003},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.442,"15min":0.46100000000000002},"min":{"1min":0.31,"5min":0.28399999999999997,"15min":0.26700000000000002},"max":{"1min":0.66300000000000003,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.379}]},{"osd":2,"last update":"Mon Oct 16 12:29:06 2023","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.39700000000000002,"15min":0.42599999999999999},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.93000000000000005},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.45600000000000002,"15min":0.46700000000000003},"min":{"1min":0.34899999999999998,"5min":0.29099999999999998,"15min":0.28199999999999997},"max":{"1min":0.68300000000000005,"5min":0.69199999999999995,"15min":0.98099999999999998},"last":0.40000000000000002}]},{"osd":3,"last update":"Mon Oct 16 12:29:06 2023","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.38900000000000001,"15min":0.40300000000000002},"min":{"1min":0.308,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.60499999999999998,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.314},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.438,"15min":0.45600000000000002},"min":{"1min":0.36799999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.62,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.47899999999999998}]},{"osd":4,"last update":"Mon Oct 16 12:29:06 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.61099999999999999,"15min":0.63200000000000001},"min":{"1min":0.46600000000000003,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.878,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.432},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.623,"15min":0.64400000000000002},"min":{"1min":0.54500000000000004,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":1.089},"last":0.41599999999999998}]},{"osd":5,"last update":"Mon Oct 16 12:29:06 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.63900000000000001,"15min":0.66800000000000004},"min":{"1min":0.46800000000000003,"5min":0.42599999999999999,"15min":0.38300000000000001},"max":{"1min":1.0209999999999999,"5min":1.069,"15min":1.069},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.67900000000000005,"15min":0.68999999999999995},"min":{"1min":0.47499999999999998,"5min":0.439,"15min":0.32500000000000001},"max":{"1min":1.034,"5min":1.0620000000000001,"15min":1.0649999999999999},"last":0.53500000000000003}]},{"osd":6,"last update":"Mon Oct 16 12:29:06 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.63900000000000001,"15min":0.65800000000000003},"min":{"1min":0.48099999999999998,"5min":0.45600000000000002,"15min":0.378},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.64900000000000002,"15min":0.66500000000000004},"min":{"1min":0.46300000000000002,"5min":0.434,"15min":0.32900000000000001},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.048},"last":0.44800000000000001}]},{"osd":7,"last update":"Mon Oct 16 12:29:06 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.66200000000000003,"15min":0.68600000000000005},"min":{"1min":0.53500000000000003,"5min":0.40200000000000002,"15min":0.33700000000000002},"max":{"1min":0.90400000000000003,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":0.497},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.68300000000000005,"15min":0.70399999999999996},"min":{"1min":0.56999999999999995,"5min":0.45700000000000002,"15min":0.42099999999999999},"max":{"1min":1.006,"5min":1.006,"15min":1.167},"last":0.52300000000000002}]}]},{"osd":0,"up_from":13,"seq":55834575070,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37480,"kb_used_data":2132,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334360,"statfs":{"total":96636764160,"available":96598384640,"internally_reserved":0,"allocated":2183168,"data_stored":1383776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 12:28:58 2023","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.39700000000000002,"15min":0.39200000000000002},"min":{"1min":0.23899999999999999,"5min":0.219,"15min":0.219},"max":{"1min":0.59299999999999997,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.40500000000000003,"15min":0.41999999999999998},"min":{"1min":0.29399999999999998,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.66000000000000003,"5min":0.71799999999999997,"15min":0.73099999999999998},"last":0.58299999999999996}]},{"osd":2,"last update":"Mon Oct 16 12:28:58 2023","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.41899999999999998,"15min":0.42599999999999999},"min":{"1min":0.315,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.63600000000000001,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.44400000000000001,"15min":0.45400000000000001},"min":{"1min":0.32000000000000001,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.65500000000000003,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.54900000000000004}]},{"osd":3,"last update":"Mon Oct 16 12:28:58 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.42699999999999999,"15min":0.42599999999999999},"min":{"1min":0.34899999999999998,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.57499999999999996,"5min":0.62,"15min":0.79400000000000004},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.497,"5min":0.47899999999999998,"15min":0.48999999999999999},"min":{"1min":0.311,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.755,"5min":0.79200000000000004,"15min":0.878},"last":0.378}]},{"osd":4,"last update":"Mon Oct 16 12:28:58 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.63900000000000001,"15min":0.65200000000000002},"min":{"1min":0.378,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.0329999999999999,"5min":1.05,"15min":1.0960000000000001},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.63900000000000001,"15min":0.65800000000000003},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.34100000000000003},"max":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.1719999999999999},"last":0.432}]},{"osd":5,"last update":"Mon Oct 16 12:28:58 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.63900000000000001,"15min":0.65400000000000003},"min":{"1min":0.441,"5min":0.373,"15min":0.373},"max":{"1min":0.92200000000000004,"5min":0.97699999999999998,"15min":1.127},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.67400000000000004,"15min":0.68000000000000005},"min":{"1min":0.48699999999999999,"5min":0.39400000000000002,"15min":0.39000000000000001},"max":{"1min":0.875,"5min":1.0700000000000001,"15min":1.1160000000000001},"last":0.48999999999999999}]},{"osd":6,"last update":"Mon Oct 16 12:28:58 2023","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.67500000000000004,"15min":0.68400000000000005},"min":{"1min":0.51100000000000001,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":1.079},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.66700000000000004,"15min":0.68799999999999994},"min":{"1min":0.42799999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.95099999999999996,"5min":0.96399999999999997,"15min":1.1499999999999999},"last":0.752}]},{"osd":7,"last update":"Mon Oct 16 12:28:58 2023","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.69199999999999995,"15min":0.70499999999999996},"min":{"1min":0.45200000000000001,"5min":0.436,"15min":0.41499999999999998},"max":{"1min":0.98499999999999999,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.65800000000000003,"15min":0.68899999999999995},"min":{"1min":0.47699999999999998,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.91000000000000003,"5min":0.97499999999999998,"15min":1.0569999999999999},"last":0.47599999999999998}]}]},{"osd":2,"up_from":13,"seq":55834575070,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59096,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312744,"statfs":{"total":96636764160,"available":96576249856,"internally_reserved":0,"allocated":1576960,"data_stored":791740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:29:04 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.36899999999999999,"15min":0.38300000000000001},"min":{"1min":0.20799999999999999,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.52100000000000002,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.432,"15min":0.44700000000000001},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.53500000000000003,"5min":0.66500000000000004,"15min":0.82999999999999996},"last":0.30399999999999999}]},{"osd":1,"last update":"Mon Oct 16 12:29:04 2023","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.44500000000000001,"15min":0.46600000000000003},"min":{"1min":0.29399999999999998,"5min":0.27300000000000002,"15min":0.25600000000000001},"max":{"1min":0.64000000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.40899999999999997,"15min":0.432},"min":{"1min":0.28699999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.57599999999999996,"5min":0.59199999999999997,"15min":0.69699999999999995},"last":0.224}]},{"osd":3,"last update":"Mon Oct 16 12:29:04 2023","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.39500000000000002,"15min":0.41099999999999998},"min":{"1min":0.26300000000000001,"5min":0.214,"15min":0.214},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.89300000000000002},"last":0.25},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.498,"15min":0.51500000000000001},"min":{"1min":0.33900000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.74399999999999999,"5min":0.78000000000000003,"15min":0.79600000000000004},"last":0.35099999999999998}]},{"osd":4,"last update":"Mon Oct 16 12:29:04 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.65600000000000003,"15min":0.67300000000000004},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.92300000000000004,"5min":1.3839999999999999,"15min":1.3839999999999999},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.64100000000000001,"15min":0.66900000000000004},"min":{"1min":0.46500000000000002,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.94399999999999995,"5min":0.97599999999999998,"15min":1.0109999999999999},"last":0.435}]},{"osd":5,"last update":"Mon Oct 16 12:29:04 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.63700000000000001,"15min":0.66000000000000003},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.312},"max":{"1min":1.0109999999999999,"5min":1.093,"15min":1.093},"last":0.497},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.63200000000000001,"15min":0.65400000000000003},"min":{"1min":0.39900000000000002,"5min":0.33100000000000002,"15min":0.317},"max":{"1min":0.81899999999999995,"5min":1.331,"15min":1.331},"last":0.41999999999999998}]},{"osd":6,"last update":"Mon Oct 16 12:29:04 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.60499999999999998,"15min":0.628},"min":{"1min":0.436,"5min":0.34899999999999998,"15min":0.34699999999999998},"max":{"1min":0.82399999999999995,"5min":0.99299999999999999,"15min":1.0369999999999999},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.66600000000000004,"15min":0.68200000000000005},"min":{"1min":0.41499999999999998,"5min":0.40799999999999997,"15min":0.40000000000000002},"max":{"1min":0.96299999999999997,"5min":1.032,"15min":1.032},"last":0.47399999999999998}]},{"osd":7,"last update":"Mon Oct 16 12:29:04 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.65900000000000003,"15min":0.67000000000000004},"min":{"1min":0.438,"5min":0.40300000000000002,"15min":0.38400000000000001},"max":{"1min":0.86199999999999999,"5min":0.96399999999999997,"15min":1.016},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.66300000000000003,"15min":0.68200000000000005},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.0409999999999999,"5min":1.145,"15min":1.145},"last":0.44400000000000001}]}]},{"osd":3,"up_from":13,"seq":55834575069,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36888,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334952,"statfs":{"total":96636764160,"available":96598990848,"internally_reserved":0,"allocated":1576960,"data_stored":792288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:29:09 2023","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.39500000000000002,"15min":0.40000000000000002},"min":{"1min":0.25,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.628,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.434,"5min":0.39100000000000001,"15min":0.40600000000000003},"min":{"1min":0.28199999999999997,"5min":0.254,"15min":0.254},"max":{"1min":0.59499999999999997,"5min":0.61899999999999999,"15min":0.73099999999999998},"last":0.311}]},{"osd":1,"last update":"Mon Oct 16 12:29:09 2023","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.39600000000000002,"15min":0.41299999999999998},"min":{"1min":0.25700000000000001,"5min":0.245,"15min":0.24399999999999999},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70799999999999996},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.439,"15min":0.45900000000000002},"min":{"1min":0.28199999999999997,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.747},"last":0.41699999999999998}]},{"osd":2,"last update":"Mon Oct 16 12:29:09 2023","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.45600000000000002,"15min":0.47199999999999998},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29199999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.47899999999999998,"15min":0.498},"min":{"1min":0.35799999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.752},"last":0.36699999999999999}]},{"osd":4,"last update":"Mon Oct 16 12:29:09 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.61199999999999999,"15min":0.64300000000000002},"min":{"1min":0.378,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":1.1539999999999999},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.621,"15min":0.65800000000000003},"min":{"1min":0.39500000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.95399999999999996,"5min":0.96999999999999997,"15min":1.0409999999999999},"last":0.60699999999999998}]},{"osd":5,"last update":"Mon Oct 16 12:29:09 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.65300000000000002,"15min":0.68200000000000005},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.1080000000000001},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.63,"15min":0.66000000000000003},"min":{"1min":0.48299999999999998,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.96799999999999997,"5min":0.97299999999999998,"15min":1.022},"last":0.52900000000000003}]},{"osd":6,"last update":"Mon Oct 16 12:29:09 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.63500000000000001,"15min":0.66800000000000004},"min":{"1min":0.438,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.86599999999999999,"5min":0.94199999999999995,"15min":1.0760000000000001},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.67800000000000005,"15min":0.70499999999999996},"min":{"1min":0.49399999999999999,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.006,"5min":1.0349999999999999,"15min":1.0409999999999999},"last":0.629}]},{"osd":7,"last update":"Mon Oct 16 12:29:09 2023","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.65300000000000002,"15min":0.68400000000000005},"min":{"1min":0.52800000000000002,"5min":0.42399999999999999,"15min":0.38},"max":{"1min":1.0149999999999999,"5min":1.0189999999999999,"15min":1.2430000000000001},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.748,"5min":0.68600000000000005,"15min":0.70399999999999996},"min":{"1min":0.5,"5min":0.44,"15min":0.42699999999999999},"max":{"1min":1.002,"5min":1.03,"15min":1.1339999999999999},"last":0.51700000000000002}]}]},{"osd":4,"up_from":13,"seq":55834575068,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32720,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339120,"statfs":{"total":96636764160,"available":96603258880,"internally_reserved":0,"allocated":1581056,"data_stored":791798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.57099999999999995,"15min":0.58699999999999997},"min":{"1min":0.38900000000000001,"5min":0.30399999999999999,"15min":0.27900000000000003},"max":{"1min":0.78500000000000003,"5min":0.85099999999999998,"15min":0.94599999999999995},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.57299999999999995,"15min":0.59999999999999998},"min":{"1min":0.42699999999999999,"5min":0.29999999999999999,"15min":0.27600000000000002},"max":{"1min":0.76400000000000001,"5min":0.83199999999999996,"15min":0.88700000000000001},"last":0.66800000000000004}]},{"osd":1,"last update":"Mon Oct 16 12:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.58799999999999997,"15min":0.60999999999999999},"min":{"1min":0.39700000000000002,"5min":0.31900000000000001,"15min":0.28499999999999998},"max":{"1min":0.80200000000000005,"5min":0.82599999999999996,"15min":0.92600000000000005},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.61799999999999999},"min":{"1min":0.36899999999999999,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.78300000000000003,"5min":0.91700000000000004,"15min":0.98099999999999998},"last":0.69499999999999995}]},{"osd":2,"last update":"Mon Oct 16 12:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.59499999999999997,"15min":0.623},"min":{"1min":0.432,"5min":0.33800000000000002,"15min":0.32000000000000001},"max":{"1min":0.84299999999999997,"5min":0.85699999999999998,"15min":1.012},"last":0.505},{"interface":"front","average":{"1min":0.63,"5min":0.60599999999999998,"15min":0.625},"min":{"1min":0.42599999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.85599999999999998,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.65400000000000003}]},{"osd":3,"last update":"Mon Oct 16 12:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.57499999999999996,"15min":0.60499999999999998},"min":{"1min":0.46200000000000002,"5min":0.35599999999999998,"15min":0.33700000000000002},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.93200000000000005},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.63600000000000001,"15min":0.64400000000000002},"min":{"1min":0.46600000000000003,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.84999999999999998,"5min":0.90600000000000003,"15min":0.97199999999999998},"last":0.68100000000000005}]},{"osd":5,"last update":"Mon Oct 16 12:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.52000000000000002,"15min":0.503},"min":{"1min":0.35199999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.82599999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.52900000000000003,"15min":0.52500000000000002},"min":{"1min":0.34999999999999998,"5min":0.32100000000000001,"15min":0.308},"max":{"1min":0.86599999999999999,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.45800000000000002}]},{"osd":6,"last update":"Mon Oct 16 12:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.372,"5min":0.33800000000000002,"15min":0.32500000000000001},"max":{"1min":0.79800000000000004,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.64400000000000002,"15min":0.63600000000000001},"min":{"1min":0.46200000000000002,"5min":0.35499999999999998,"15min":0.32000000000000001},"max":{"1min":0.89300000000000002,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.40500000000000003}]},{"osd":7,"last update":"Mon Oct 16 12:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.58999999999999997,"15min":0.58199999999999996},"min":{"1min":0.41699999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.63500000000000001,"15min":0.629},"min":{"1min":0.434,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.90800000000000003,"5min":1.006,"15min":1.006},"last":0.437}]}]},{"osd":5,"up_from":13,"seq":55834575069,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54912,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316928,"statfs":{"total":96636764160,"available":96580534272,"internally_reserved":0,"allocated":1560576,"data_stored":784583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:29:03 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.58799999999999997},"min":{"1min":0.37,"5min":0.32500000000000001,"15min":0.28000000000000003},"max":{"1min":0.76200000000000001,"5min":0.84499999999999997,"15min":0.873},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.59899999999999998,"15min":0.61799999999999999},"min":{"1min":0.51800000000000002,"5min":0.313,"15min":0.27600000000000002},"max":{"1min":0.85799999999999998,"5min":0.85899999999999999,"15min":0.88800000000000001},"last":0.89600000000000002}]},{"osd":1,"last update":"Mon Oct 16 12:29:03 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.58699999999999997,"15min":0.61099999999999999},"min":{"1min":0.37,"5min":0.35999999999999999,"15min":0.28999999999999998},"max":{"1min":0.79600000000000004,"5min":0.89700000000000002,"15min":0.91300000000000003},"last":0.93799999999999994},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.59299999999999997,"15min":0.61799999999999999},"min":{"1min":0.40699999999999997,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.84399999999999997,"5min":0.85999999999999999,"15min":0.89000000000000001},"last":0.71999999999999997}]},{"osd":2,"last update":"Mon Oct 16 12:29:03 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.58799999999999997,"15min":0.621},"min":{"1min":0.371,"5min":0.36899999999999999,"15min":0.29099999999999998},"max":{"1min":0.876,"5min":0.876,"15min":0.94399999999999995},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.58799999999999997,"15min":0.60999999999999999},"min":{"1min":0.35399999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.77100000000000002,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.91400000000000003}]},{"osd":3,"last update":"Mon Oct 16 12:29:03 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.61599999999999999,"15min":0.63900000000000001},"min":{"1min":0.47199999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.97499999999999998},"last":0.755},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.627,"15min":0.64500000000000002},"min":{"1min":0.51500000000000001,"5min":0.40400000000000003,"15min":0.35999999999999999},"max":{"1min":0.83199999999999996,"5min":0.90800000000000003,"15min":0.93200000000000005},"last":0.64600000000000002}]},{"osd":4,"last update":"Mon Oct 16 12:29:03 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.499,"15min":0.50700000000000001},"min":{"1min":0.33000000000000002,"5min":0.28100000000000003,"15min":0.25800000000000001},"max":{"1min":0.81299999999999994,"5min":0.85699999999999998,"15min":0.86399999999999999},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.57799999999999996,"15min":0.57199999999999995},"min":{"1min":0.34999999999999998,"5min":0.33900000000000002,"15min":0.33500000000000002},"max":{"1min":0.86099999999999999,"5min":0.90200000000000002,"15min":1.347},"last":0.83999999999999997}]},{"osd":6,"last update":"Mon Oct 16 12:29:03 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.55800000000000005,"15min":0.55400000000000005},"min":{"1min":0.39400000000000002,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.86699999999999999,"5min":0.91800000000000004,"15min":0.93500000000000005},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.621,"5min":0.59599999999999997,"15min":0.59699999999999998},"min":{"1min":0.39200000000000002,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.88400000000000001,"5min":0.92600000000000005,"15min":0.92700000000000005},"last":0.51300000000000001}]},{"osd":7,"last update":"Mon Oct 16 12:29:03 2023","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.63200000000000001,"15min":0.621},"min":{"1min":0.41699999999999998,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.435,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.93899999999999995,"5min":0.95199999999999996,"15min":1.1180000000000001},"last":0.54100000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T12:29:20.545 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T12:29:20.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.702+0000 7fc736dec700 1 Processor -- start 2023-10-16T12:29:20.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.702+0000 7fc736dec700 1 -- start start 2023-10-16T12:29:20.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.704+0000 7fc736dec700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc7300b2be0 0x7fc7300b3b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:20.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.704+0000 7fc736dec700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7300d11c0 con 0x7fc73006aef0 2023-10-16T12:29:20.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.704+0000 7fc736dec700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7300cd360 con 0x7fc73015b860 2023-10-16T12:29:20.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.704+0000 7fc736dec700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7300d03d0 con 0x7fc7300b2be0 2023-10-16T12:29:20.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.704+0000 7fc735389700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc7300b2be0 0x7fc7300b3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:20.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.704+0000 7fc72ffff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc73006aef0 0x7fc7300b3480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33098/0 (socket says 172.21.15.16:33098) 2023-10-16T12:29:20.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.704+0000 7fc72ffff700 1 -- 172.21.15.16:0/2415489046 learned_addr learned my addr 172.21.15.16:0/2415489046 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:29:20.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.704+0000 7fc735389700 1 -- 172.21.15.16:0/2415489046 >> v1:172.21.15.49:6789/0 conn(0x7fc73015b860 legacy=0x7fc73006a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:29:20.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.704+0000 7fc735389700 1 -- 172.21.15.16:0/2415489046 >> v1:172.21.15.16:6789/0 conn(0x7fc73006aef0 legacy=0x7fc7300b3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:20.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.704+0000 7fc735389700 1 -- 172.21.15.16:0/2415489046 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7300d05b0 con 0x7fc7300b2be0 2023-10-16T12:29:20.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.704+0000 7fc72f7fe700 1 -- 172.21.15.16:0/2415489046 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3538795648 0 0) 0x7fc7300d11c0 con 0x7fc73006aef0 2023-10-16T12:29:20.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.704+0000 7fc735389700 1 --2- 172.21.15.16:0/2415489046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7300b2be0 0x7fc7300b3b90 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7fc728009d90 tx=0x7fc72800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc18d99d968d8fce server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:20.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.704+0000 7fc72f7fe700 1 -- 172.21.15.16:0/2415489046 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc72800e070 con 0x7fc7300b2be0 2023-10-16T12:29:20.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.704+0000 7fc72f7fe700 1 -- 172.21.15.16:0/2415489046 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc72800bb90 con 0x7fc7300b2be0 2023-10-16T12:29:20.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.704+0000 7fc72f7fe700 1 -- 172.21.15.16:0/2415489046 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc728003a20 con 0x7fc7300b2be0 2023-10-16T12:29:20.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.705+0000 7fc736dec700 1 -- 172.21.15.16:0/2415489046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7300b2be0 msgr2=0x7fc7300b3b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:20.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.705+0000 7fc736dec700 1 --2- 172.21.15.16:0/2415489046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7300b2be0 0x7fc7300b3b90 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7fc728009d90 tx=0x7fc72800b330 comp rx=0 tx=0).stop 2023-10-16T12:29:20.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.705+0000 7fc736dec700 1 -- 172.21.15.16:0/2415489046 shutdown_connections 2023-10-16T12:29:20.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.705+0000 7fc736dec700 1 --2- 172.21.15.16:0/2415489046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7300b2be0 0x7fc7300b3b90 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:20.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.705+0000 7fc736dec700 1 -- 172.21.15.16:0/2415489046 >> 172.21.15.16:0/2415489046 conn(0x7fc7300b1fd0 msgr2=0x7fc7300b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:20.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.705+0000 7fc736dec700 1 -- 172.21.15.16:0/2415489046 shutdown_connections 2023-10-16T12:29:20.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.705+0000 7fc736dec700 1 -- 172.21.15.16:0/2415489046 wait complete. 2023-10-16T12:29:20.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.706+0000 7fc736dec700 1 Processor -- start 2023-10-16T12:29:20.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.706+0000 7fc736dec700 1 -- start start 2023-10-16T12:29:20.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.707+0000 7fc736dec700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc73006aef0 0x7fc73015a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:20.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.707+0000 7fc736dec700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7300b2be0 0x7fc7300aeab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:20.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.707+0000 7fc736dec700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc73015b860 0x7fc7300718e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:20.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.707+0000 7fc736dec700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7300ae620 con 0x7fc7300b2be0 2023-10-16T12:29:20.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.707+0000 7fc736dec700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7300add10 con 0x7fc73006aef0 2023-10-16T12:29:20.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.707+0000 7fc736dec700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7300ad800 con 0x7fc73015b860 2023-10-16T12:29:20.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.707+0000 7fc72ffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7300b2be0 0x7fc7300aeab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:20.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.707+0000 7fc72ffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7300b2be0 0x7fc7300aeab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43832/0 (socket says 172.21.15.16:43832) 2023-10-16T12:29:20.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.707+0000 7fc72ffff700 1 -- 172.21.15.16:0/3279902471 learned_addr learned my addr 172.21.15.16:0/3279902471 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:29:20.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.707+0000 7fc735389700 1 --2- 172.21.15.16:0/3279902471 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc73015b860 0x7fc7300718e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:20.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.707+0000 7fc734b88700 1 --2- 172.21.15.16:0/3279902471 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc73006aef0 0x7fc73015a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:20.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.707+0000 7fc72ffff700 1 -- 172.21.15.16:0/3279902471 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc73015b860 msgr2=0x7fc7300718e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:20.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.707+0000 7fc72ffff700 1 --2- 172.21.15.16:0/3279902471 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc73015b860 0x7fc7300718e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:20.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.707+0000 7fc72ffff700 1 -- 172.21.15.16:0/3279902471 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc73006aef0 msgr2=0x7fc73015a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:20.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.707+0000 7fc72ffff700 1 --2- 172.21.15.16:0/3279902471 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc73006aef0 0x7fc73015a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:20.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.707+0000 7fc72ffff700 1 -- 172.21.15.16:0/3279902471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc728009a70 con 0x7fc7300b2be0 2023-10-16T12:29:20.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.707+0000 7fc735389700 1 --2- 172.21.15.16:0/3279902471 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc73015b860 0x7fc7300718e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:29:20.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.707+0000 7fc734b88700 1 --2- 172.21.15.16:0/3279902471 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc73006aef0 0x7fc73015a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:29:20.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.708+0000 7fc72ffff700 1 --2- 172.21.15.16:0/3279902471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7300b2be0 0x7fc7300aeab0 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7fc71c00c550 tx=0x7fc71c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:20.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.708+0000 7fc72d7fa700 1 -- 172.21.15.16:0/3279902471 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc71c004c80 con 0x7fc7300b2be0 2023-10-16T12:29:20.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.708+0000 7fc72d7fa700 1 -- 172.21.15.16:0/3279902471 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc71c004e00 con 0x7fc7300b2be0 2023-10-16T12:29:20.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.708+0000 7fc736dec700 1 -- 172.21.15.16:0/3279902471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7300adfd0 con 0x7fc7300b2be0 2023-10-16T12:29:20.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.708+0000 7fc72d7fa700 1 -- 172.21.15.16:0/3279902471 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc71c007310 con 0x7fc7300b2be0 2023-10-16T12:29:20.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.708+0000 7fc736dec700 1 -- 172.21.15.16:0/3279902471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7300aeff0 con 0x7fc7300b2be0 2023-10-16T12:29:20.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.709+0000 7fc736dec700 1 -- 172.21.15.16:0/3279902471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7300af210 con 0x7fc7300b2be0 2023-10-16T12:29:20.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.709+0000 7fc72d7fa700 1 -- 172.21.15.16:0/3279902471 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 62) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc71c007470 con 0x7fc7300b2be0 2023-10-16T12:29:20.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.710+0000 7fc72d7fa700 1 --2- 172.21.15.16:0/3279902471 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fc71406b020 0x7fc71406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:20.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.710+0000 7fc734b88700 1 --2- 172.21.15.16:0/3279902471 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fc71406b020 0x7fc71406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:20.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.710+0000 7fc72d7fa700 1 -- 172.21.15.16:0/3279902471 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fc71c089670 con 0x7fc7300b2be0 2023-10-16T12:29:20.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.710+0000 7fc734b88700 1 --2- 172.21.15.16:0/3279902471 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fc71406b020 0x7fc71406d4e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc7240017d0 tx=0x7fc724009040 comp rx=0 tx=0).ready entity=mgr.7884 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:20.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.714+0000 7fc72d7fa700 1 -- 172.21.15.16:0/3279902471 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc71c054410 con 0x7fc7300b2be0 2023-10-16T12:29:20.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.912+0000 7fc736dec700 1 -- 172.21.15.16:0/3279902471 --> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc7300d0b70 con 0x7fc71406b020 2023-10-16T12:29:20.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.920+0000 7fc72d7fa700 1 -- 172.21.15.16:0/3279902471 <== mgr.7884 v2:172.21.15.16:6832/2309613638 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+290007 (secure 0 0 0) 0x7fc724001470 con 0x7fc71406b020 2023-10-16T12:29:20.925 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:29:20.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.923+0000 7fc736dec700 1 -- 172.21.15.16:0/3279902471 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fc71406b020 msgr2=0x7fc71406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:20.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.923+0000 7fc736dec700 1 --2- 172.21.15.16:0/3279902471 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fc71406b020 0x7fc71406d4e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc7240017d0 tx=0x7fc724009040 comp rx=0 tx=0).stop 2023-10-16T12:29:20.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.923+0000 7fc736dec700 1 -- 172.21.15.16:0/3279902471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7300b2be0 msgr2=0x7fc7300aeab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:20.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.923+0000 7fc736dec700 1 --2- 172.21.15.16:0/3279902471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7300b2be0 0x7fc7300aeab0 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7fc71c00c550 tx=0x7fc71c00c910 comp rx=0 tx=0).stop 2023-10-16T12:29:20.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.923+0000 7fc736dec700 1 -- 172.21.15.16:0/3279902471 shutdown_connections 2023-10-16T12:29:20.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.923+0000 7fc736dec700 1 --2- 172.21.15.16:0/3279902471 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fc71406b020 0x7fc71406d4e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:20.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.923+0000 7fc736dec700 1 --2- 172.21.15.16:0/3279902471 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc73015b860 0x7fc7300718e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:20.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.923+0000 7fc736dec700 1 --2- 172.21.15.16:0/3279902471 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc73006aef0 0x7fc73015a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:20.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.923+0000 7fc736dec700 1 --2- 172.21.15.16:0/3279902471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7300b2be0 0x7fc7300aeab0 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:20.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.923+0000 7fc736dec700 1 -- 172.21.15.16:0/3279902471 >> 172.21.15.16:0/3279902471 conn(0x7fc7300b1fd0 msgr2=0x7fc7300aff40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:20.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.924+0000 7fc736dec700 1 -- 172.21.15.16:0/3279902471 shutdown_connections 2023-10-16T12:29:20.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:20.924+0000 7fc736dec700 1 -- 172.21.15.16:0/3279902471 wait complete. 2023-10-16T12:29:20.930 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-10-16T12:29:20.938 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-10-16T12:29:18.915190+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":884,"num_read_kb":1748,"num_write":363,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":243,"ondisk_log_size":243,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":121,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":354504,"kb_used_data":13452,"kb_used_omap":12,"kb_used_meta":340915,"kb_avail":754620216,"statfs":{"total":773094113280,"available":772731101184,"internally_reserved":0,"allocated":13774848,"data_stored":7495585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12740,"internal_metadata":349097532},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.551030"},"pg_stats":[{"pgid":"16.3f","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.300047+0000","last_change":"2023-10-16T12:29:16.295442+0000","last_active":"2023-10-16T12:29:16.300047+0000","last_peered":"2023-10-16T12:29:16.300047+0000","last_clean":"2023-10-16T12:29:16.300047+0000","last_became_active":"2023-10-16T12:29:16.293293+0000","last_became_peered":"2023-10-16T12:29:16.293293+0000","last_unstale":"2023-10-16T12:29:16.300047+0000","last_undegraded":"2023-10-16T12:29:16.300047+0000","last_fullsized":"2023-10-16T12:29:16.300047+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:45:34.212638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3e","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.304749+0000","last_change":"2023-10-16T12:29:16.304101+0000","last_active":"2023-10-16T12:29:16.304749+0000","last_peered":"2023-10-16T12:29:16.304749+0000","last_clean":"2023-10-16T12:29:16.304749+0000","last_became_active":"2023-10-16T12:29:16.303818+0000","last_became_peered":"2023-10-16T12:29:16.303818+0000","last_unstale":"2023-10-16T12:29:16.304749+0000","last_undegraded":"2023-10-16T12:29:16.304749+0000","last_fullsized":"2023-10-16T12:29:16.304749+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:37:29.167682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.3d","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.303703+0000","last_change":"2023-10-16T12:29:16.296339+0000","last_active":"2023-10-16T12:29:16.303703+0000","last_peered":"2023-10-16T12:29:16.303703+0000","last_clean":"2023-10-16T12:29:16.303703+0000","last_became_active":"2023-10-16T12:29:16.296083+0000","last_became_peered":"2023-10-16T12:29:16.296083+0000","last_unstale":"2023-10-16T12:29:16.303703+0000","last_undegraded":"2023-10-16T12:29:16.303703+0000","last_fullsized":"2023-10-16T12:29:16.303703+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:33:34.851022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3c","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.300969+0000","last_change":"2023-10-16T12:29:16.297638+0000","last_active":"2023-10-16T12:29:16.300969+0000","last_peered":"2023-10-16T12:29:16.300969+0000","last_clean":"2023-10-16T12:29:16.300969+0000","last_became_active":"2023-10-16T12:29:16.297121+0000","last_became_peered":"2023-10-16T12:29:16.297121+0000","last_unstale":"2023-10-16T12:29:16.300969+0000","last_undegraded":"2023-10-16T12:29:16.300969+0000","last_fullsized":"2023-10-16T12:29:16.300969+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:14:59.533040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.3b","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.299033+0000","last_change":"2023-10-16T12:29:16.297543+0000","last_active":"2023-10-16T12:29:16.299033+0000","last_peered":"2023-10-16T12:29:16.299033+0000","last_clean":"2023-10-16T12:29:16.299033+0000","last_became_active":"2023-10-16T12:29:16.297266+0000","last_became_peered":"2023-10-16T12:29:16.297266+0000","last_unstale":"2023-10-16T12:29:16.299033+0000","last_undegraded":"2023-10-16T12:29:16.299033+0000","last_fullsized":"2023-10-16T12:29:16.299033+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:15:48.726807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.3a","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.293761+0000","last_change":"2023-10-16T12:29:16.291104+0000","last_active":"2023-10-16T12:29:16.293761+0000","last_peered":"2023-10-16T12:29:16.293761+0000","last_clean":"2023-10-16T12:29:16.293761+0000","last_became_active":"2023-10-16T12:29:16.290781+0000","last_became_peered":"2023-10-16T12:29:16.290781+0000","last_unstale":"2023-10-16T12:29:16.293761+0000","last_undegraded":"2023-10-16T12:29:16.293761+0000","last_fullsized":"2023-10-16T12:29:16.293761+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:31:19.116278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.39","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.291331+0000","last_change":"2023-10-16T12:29:16.285561+0000","last_active":"2023-10-16T12:29:16.291331+0000","last_peered":"2023-10-16T12:29:16.291331+0000","last_clean":"2023-10-16T12:29:16.291331+0000","last_became_active":"2023-10-16T12:29:16.285354+0000","last_became_peered":"2023-10-16T12:29:16.285354+0000","last_unstale":"2023-10-16T12:29:16.291331+0000","last_undegraded":"2023-10-16T12:29:16.291331+0000","last_fullsized":"2023-10-16T12:29:16.291331+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:38:24.426289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.38","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.292514+0000","last_change":"2023-10-16T12:29:16.290861+0000","last_active":"2023-10-16T12:29:16.292514+0000","last_peered":"2023-10-16T12:29:16.292514+0000","last_clean":"2023-10-16T12:29:16.292514+0000","last_became_active":"2023-10-16T12:29:16.290578+0000","last_became_peered":"2023-10-16T12:29:16.290578+0000","last_unstale":"2023-10-16T12:29:16.292514+0000","last_undegraded":"2023-10-16T12:29:16.292514+0000","last_fullsized":"2023-10-16T12:29:16.292514+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:53:51.137615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.37","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.304509+0000","last_change":"2023-10-16T12:29:16.303198+0000","last_active":"2023-10-16T12:29:16.304509+0000","last_peered":"2023-10-16T12:29:16.304509+0000","last_clean":"2023-10-16T12:29:16.304509+0000","last_became_active":"2023-10-16T12:29:16.301618+0000","last_became_peered":"2023-10-16T12:29:16.301618+0000","last_unstale":"2023-10-16T12:29:16.304509+0000","last_undegraded":"2023-10-16T12:29:16.304509+0000","last_fullsized":"2023-10-16T12:29:16.304509+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:15:48.726807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.36","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.298649+0000","last_change":"2023-10-16T12:29:16.295295+0000","last_active":"2023-10-16T12:29:16.298649+0000","last_peered":"2023-10-16T12:29:16.298649+0000","last_clean":"2023-10-16T12:29:16.298649+0000","last_became_active":"2023-10-16T12:29:16.294939+0000","last_became_peered":"2023-10-16T12:29:16.294939+0000","last_unstale":"2023-10-16T12:29:16.298649+0000","last_undegraded":"2023-10-16T12:29:16.298649+0000","last_fullsized":"2023-10-16T12:29:16.298649+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:12:17.761721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.35","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.298915+0000","last_change":"2023-10-16T12:29:16.295342+0000","last_active":"2023-10-16T12:29:16.298915+0000","last_peered":"2023-10-16T12:29:16.298915+0000","last_clean":"2023-10-16T12:29:16.298915+0000","last_became_active":"2023-10-16T12:29:16.295021+0000","last_became_peered":"2023-10-16T12:29:16.295021+0000","last_unstale":"2023-10-16T12:29:16.298915+0000","last_undegraded":"2023-10-16T12:29:16.298915+0000","last_fullsized":"2023-10-16T12:29:16.298915+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:10:52.270680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.34","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.301204+0000","last_change":"2023-10-16T12:29:16.297042+0000","last_active":"2023-10-16T12:29:16.301204+0000","last_peered":"2023-10-16T12:29:16.301204+0000","last_clean":"2023-10-16T12:29:16.301204+0000","last_became_active":"2023-10-16T12:29:16.295527+0000","last_became_peered":"2023-10-16T12:29:16.295527+0000","last_unstale":"2023-10-16T12:29:16.301204+0000","last_undegraded":"2023-10-16T12:29:16.301204+0000","last_fullsized":"2023-10-16T12:29:16.301204+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:32:43.586508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.33","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.299862+0000","last_change":"2023-10-16T12:29:16.296802+0000","last_active":"2023-10-16T12:29:16.299862+0000","last_peered":"2023-10-16T12:29:16.299862+0000","last_clean":"2023-10-16T12:29:16.299862+0000","last_became_active":"2023-10-16T12:29:16.296485+0000","last_became_peered":"2023-10-16T12:29:16.296485+0000","last_unstale":"2023-10-16T12:29:16.299862+0000","last_undegraded":"2023-10-16T12:29:16.299862+0000","last_fullsized":"2023-10-16T12:29:16.299862+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:20:51.410821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.32","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.299666+0000","last_change":"2023-10-16T12:29:16.297814+0000","last_active":"2023-10-16T12:29:16.299666+0000","last_peered":"2023-10-16T12:29:16.299666+0000","last_clean":"2023-10-16T12:29:16.299666+0000","last_became_active":"2023-10-16T12:29:16.297483+0000","last_became_peered":"2023-10-16T12:29:16.297483+0000","last_unstale":"2023-10-16T12:29:16.299666+0000","last_undegraded":"2023-10-16T12:29:16.299666+0000","last_fullsized":"2023-10-16T12:29:16.299666+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:03:19.526603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.31","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.299983+0000","last_change":"2023-10-16T12:29:16.296869+0000","last_active":"2023-10-16T12:29:16.299983+0000","last_peered":"2023-10-16T12:29:16.299983+0000","last_clean":"2023-10-16T12:29:16.299983+0000","last_became_active":"2023-10-16T12:29:16.295954+0000","last_became_peered":"2023-10-16T12:29:16.295954+0000","last_unstale":"2023-10-16T12:29:16.299983+0000","last_undegraded":"2023-10-16T12:29:16.299983+0000","last_fullsized":"2023-10-16T12:29:16.299983+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:47:23.445036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.30","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.299089+0000","last_change":"2023-10-16T12:29:16.296579+0000","last_active":"2023-10-16T12:29:16.299089+0000","last_peered":"2023-10-16T12:29:16.299089+0000","last_clean":"2023-10-16T12:29:16.299089+0000","last_became_active":"2023-10-16T12:29:16.296268+0000","last_became_peered":"2023-10-16T12:29:16.296268+0000","last_unstale":"2023-10-16T12:29:16.299089+0000","last_undegraded":"2023-10-16T12:29:16.299089+0000","last_fullsized":"2023-10-16T12:29:16.299089+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:14:59.533040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.2f","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.304106+0000","last_change":"2023-10-16T12:29:16.299703+0000","last_active":"2023-10-16T12:29:16.304106+0000","last_peered":"2023-10-16T12:29:16.304106+0000","last_clean":"2023-10-16T12:29:16.304106+0000","last_became_active":"2023-10-16T12:29:16.297900+0000","last_became_peered":"2023-10-16T12:29:16.297900+0000","last_unstale":"2023-10-16T12:29:16.304106+0000","last_undegraded":"2023-10-16T12:29:16.304106+0000","last_fullsized":"2023-10-16T12:29:16.304106+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:35:45.656103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2e","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.293346+0000","last_change":"2023-10-16T12:29:16.291910+0000","last_active":"2023-10-16T12:29:16.293346+0000","last_peered":"2023-10-16T12:29:16.293346+0000","last_clean":"2023-10-16T12:29:16.293346+0000","last_became_active":"2023-10-16T12:29:16.291575+0000","last_became_peered":"2023-10-16T12:29:16.291575+0000","last_unstale":"2023-10-16T12:29:16.293346+0000","last_undegraded":"2023-10-16T12:29:16.293346+0000","last_fullsized":"2023-10-16T12:29:16.293346+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:44:04.041711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.2d","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.304233+0000","last_change":"2023-10-16T12:29:16.300658+0000","last_active":"2023-10-16T12:29:16.304233+0000","last_peered":"2023-10-16T12:29:16.304233+0000","last_clean":"2023-10-16T12:29:16.304233+0000","last_became_active":"2023-10-16T12:29:16.300186+0000","last_became_peered":"2023-10-16T12:29:16.300186+0000","last_unstale":"2023-10-16T12:29:16.304233+0000","last_undegraded":"2023-10-16T12:29:16.304233+0000","last_fullsized":"2023-10-16T12:29:16.304233+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:37:10.259178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2c","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.301982+0000","last_change":"2023-10-16T12:29:16.299844+0000","last_active":"2023-10-16T12:29:16.301982+0000","last_peered":"2023-10-16T12:29:16.301982+0000","last_clean":"2023-10-16T12:29:16.301982+0000","last_became_active":"2023-10-16T12:29:16.299526+0000","last_became_peered":"2023-10-16T12:29:16.299526+0000","last_unstale":"2023-10-16T12:29:16.301982+0000","last_undegraded":"2023-10-16T12:29:16.301982+0000","last_fullsized":"2023-10-16T12:29:16.301982+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:26:55.793279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2b","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.295832+0000","last_change":"2023-10-16T12:29:16.295444+0000","last_active":"2023-10-16T12:29:16.295832+0000","last_peered":"2023-10-16T12:29:16.295832+0000","last_clean":"2023-10-16T12:29:16.295832+0000","last_became_active":"2023-10-16T12:29:16.295201+0000","last_became_peered":"2023-10-16T12:29:16.295201+0000","last_unstale":"2023-10-16T12:29:16.295832+0000","last_undegraded":"2023-10-16T12:29:16.295832+0000","last_fullsized":"2023-10-16T12:29:16.295832+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:41:46.033189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2a","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.298373+0000","last_change":"2023-10-16T12:29:16.294905+0000","last_active":"2023-10-16T12:29:16.298373+0000","last_peered":"2023-10-16T12:29:16.298373+0000","last_clean":"2023-10-16T12:29:16.298373+0000","last_became_active":"2023-10-16T12:29:16.294617+0000","last_became_peered":"2023-10-16T12:29:16.294617+0000","last_unstale":"2023-10-16T12:29:16.298373+0000","last_undegraded":"2023-10-16T12:29:16.298373+0000","last_fullsized":"2023-10-16T12:29:16.298373+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:37:29.167682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.29","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.303742+0000","last_change":"2023-10-16T12:29:16.296846+0000","last_active":"2023-10-16T12:29:16.303742+0000","last_peered":"2023-10-16T12:29:16.303742+0000","last_clean":"2023-10-16T12:29:16.303742+0000","last_became_active":"2023-10-16T12:29:16.296529+0000","last_became_peered":"2023-10-16T12:29:16.296529+0000","last_unstale":"2023-10-16T12:29:16.303742+0000","last_undegraded":"2023-10-16T12:29:16.303742+0000","last_fullsized":"2023-10-16T12:29:16.303742+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:34:41.695364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.28","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.302020+0000","last_change":"2023-10-16T12:29:16.301568+0000","last_active":"2023-10-16T12:29:16.302020+0000","last_peered":"2023-10-16T12:29:16.302020+0000","last_clean":"2023-10-16T12:29:16.302020+0000","last_became_active":"2023-10-16T12:29:16.301243+0000","last_became_peered":"2023-10-16T12:29:16.301243+0000","last_unstale":"2023-10-16T12:29:16.302020+0000","last_undegraded":"2023-10-16T12:29:16.302020+0000","last_fullsized":"2023-10-16T12:29:16.302020+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:44:04.041711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.27","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.301168+0000","last_change":"2023-10-16T12:29:16.296472+0000","last_active":"2023-10-16T12:29:16.301168+0000","last_peered":"2023-10-16T12:29:16.301168+0000","last_clean":"2023-10-16T12:29:16.301168+0000","last_became_active":"2023-10-16T12:29:16.296142+0000","last_became_peered":"2023-10-16T12:29:16.296142+0000","last_unstale":"2023-10-16T12:29:16.301168+0000","last_undegraded":"2023-10-16T12:29:16.301168+0000","last_fullsized":"2023-10-16T12:29:16.301168+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:40:09.675190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.26","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.295818+0000","last_change":"2023-10-16T12:29:16.292770+0000","last_active":"2023-10-16T12:29:16.295818+0000","last_peered":"2023-10-16T12:29:16.295818+0000","last_clean":"2023-10-16T12:29:16.295818+0000","last_became_active":"2023-10-16T12:29:16.292498+0000","last_became_peered":"2023-10-16T12:29:16.292498+0000","last_unstale":"2023-10-16T12:29:16.295818+0000","last_undegraded":"2023-10-16T12:29:16.295818+0000","last_fullsized":"2023-10-16T12:29:16.295818+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:12:04.458567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.25","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.302311+0000","last_change":"2023-10-16T12:29:16.298811+0000","last_active":"2023-10-16T12:29:16.302311+0000","last_peered":"2023-10-16T12:29:16.302311+0000","last_clean":"2023-10-16T12:29:16.302311+0000","last_became_active":"2023-10-16T12:29:16.298486+0000","last_became_peered":"2023-10-16T12:29:16.298486+0000","last_unstale":"2023-10-16T12:29:16.302311+0000","last_undegraded":"2023-10-16T12:29:16.302311+0000","last_fullsized":"2023-10-16T12:29:16.302311+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:27:09.415625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.24","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.303920+0000","last_change":"2023-10-16T12:29:16.295140+0000","last_active":"2023-10-16T12:29:16.303920+0000","last_peered":"2023-10-16T12:29:16.303920+0000","last_clean":"2023-10-16T12:29:16.303920+0000","last_became_active":"2023-10-16T12:29:16.294860+0000","last_became_peered":"2023-10-16T12:29:16.294860+0000","last_unstale":"2023-10-16T12:29:16.303920+0000","last_undegraded":"2023-10-16T12:29:16.303920+0000","last_fullsized":"2023-10-16T12:29:16.303920+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:48:44.882314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.23","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.297914+0000","last_change":"2023-10-16T12:29:16.297550+0000","last_active":"2023-10-16T12:29:16.297914+0000","last_peered":"2023-10-16T12:29:16.297914+0000","last_clean":"2023-10-16T12:29:16.297914+0000","last_became_active":"2023-10-16T12:29:16.297141+0000","last_became_peered":"2023-10-16T12:29:16.297141+0000","last_unstale":"2023-10-16T12:29:16.297914+0000","last_undegraded":"2023-10-16T12:29:16.297914+0000","last_fullsized":"2023-10-16T12:29:16.297914+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:28:23.841300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.22","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.301579+0000","last_change":"2023-10-16T12:29:16.299334+0000","last_active":"2023-10-16T12:29:16.301579+0000","last_peered":"2023-10-16T12:29:16.301579+0000","last_clean":"2023-10-16T12:29:16.301579+0000","last_became_active":"2023-10-16T12:29:16.298951+0000","last_became_peered":"2023-10-16T12:29:16.298951+0000","last_unstale":"2023-10-16T12:29:16.301579+0000","last_undegraded":"2023-10-16T12:29:16.301579+0000","last_fullsized":"2023-10-16T12:29:16.301579+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:15:35.527333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.21","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.304703+0000","last_change":"2023-10-16T12:29:16.303779+0000","last_active":"2023-10-16T12:29:16.304703+0000","last_peered":"2023-10-16T12:29:16.304703+0000","last_clean":"2023-10-16T12:29:16.304703+0000","last_became_active":"2023-10-16T12:29:16.302770+0000","last_became_peered":"2023-10-16T12:29:16.302770+0000","last_unstale":"2023-10-16T12:29:16.304703+0000","last_undegraded":"2023-10-16T12:29:16.304703+0000","last_fullsized":"2023-10-16T12:29:16.304703+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:56:15.471878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.20","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.297271+0000","last_change":"2023-10-16T12:29:16.296088+0000","last_active":"2023-10-16T12:29:16.297271+0000","last_peered":"2023-10-16T12:29:16.297271+0000","last_clean":"2023-10-16T12:29:16.297271+0000","last_became_active":"2023-10-16T12:29:16.295770+0000","last_became_peered":"2023-10-16T12:29:16.295770+0000","last_unstale":"2023-10-16T12:29:16.297271+0000","last_undegraded":"2023-10-16T12:29:16.297271+0000","last_fullsized":"2023-10-16T12:29:16.297271+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:01:47.749457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.304160+0000","last_change":"2023-10-16T12:29:16.300370+0000","last_active":"2023-10-16T12:29:16.304160+0000","last_peered":"2023-10-16T12:29:16.304160+0000","last_clean":"2023-10-16T12:29:16.304160+0000","last_became_active":"2023-10-16T12:29:16.300002+0000","last_became_peered":"2023-10-16T12:29:16.300002+0000","last_unstale":"2023-10-16T12:29:16.304160+0000","last_undegraded":"2023-10-16T12:29:16.304160+0000","last_fullsized":"2023-10-16T12:29:16.304160+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:16:33.614568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"127'4","reported_seq":15,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.381354+0000","last_change":"2023-10-16T12:29:15.387578+0000","last_active":"2023-10-16T12:29:16.381354+0000","last_peered":"2023-10-16T12:29:16.381354+0000","last_clean":"2023-10-16T12:29:16.381354+0000","last_became_active":"2023-10-16T12:29:15.387246+0000","last_became_peered":"2023-10-16T12:29:15.387246+0000","last_unstale":"2023-10-16T12:29:16.381354+0000","last_undegraded":"2023-10-16T12:29:16.381354+0000","last_fullsized":"2023-10-16T12:29:16.381354+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_clean_scrub_stamp":"2023-10-16T12:29:14.359222+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:35:20.655861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.293148+0000","last_change":"2023-10-16T12:29:16.291871+0000","last_active":"2023-10-16T12:29:16.293148+0000","last_peered":"2023-10-16T12:29:16.293148+0000","last_clean":"2023-10-16T12:29:16.293148+0000","last_became_active":"2023-10-16T12:29:16.290827+0000","last_became_peered":"2023-10-16T12:29:16.290827+0000","last_unstale":"2023-10-16T12:29:16.293148+0000","last_undegraded":"2023-10-16T12:29:16.293148+0000","last_fullsized":"2023-10-16T12:29:16.293148+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:08:51.598873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.1e","version":"127'1","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.379677+0000","last_change":"2023-10-16T12:29:15.378884+0000","last_active":"2023-10-16T12:29:16.379677+0000","last_peered":"2023-10-16T12:29:16.379677+0000","last_clean":"2023-10-16T12:29:16.379677+0000","last_became_active":"2023-10-16T12:29:15.378557+0000","last_became_peered":"2023-10-16T12:29:15.378557+0000","last_unstale":"2023-10-16T12:29:16.379677+0000","last_undegraded":"2023-10-16T12:29:16.379677+0000","last_fullsized":"2023-10-16T12:29:16.379677+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_clean_scrub_stamp":"2023-10-16T12:29:14.359222+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:27:41.017234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.298015+0000","last_change":"2023-10-16T12:29:16.295636+0000","last_active":"2023-10-16T12:29:16.298015+0000","last_peered":"2023-10-16T12:29:16.298015+0000","last_clean":"2023-10-16T12:29:16.298015+0000","last_became_active":"2023-10-16T12:29:16.295348+0000","last_became_peered":"2023-10-16T12:29:16.295348+0000","last_unstale":"2023-10-16T12:29:16.298015+0000","last_undegraded":"2023-10-16T12:29:16.298015+0000","last_fullsized":"2023-10-16T12:29:16.298015+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:32:20.674425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1d","version":"127'2","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.381441+0000","last_change":"2023-10-16T12:29:15.389455+0000","last_active":"2023-10-16T12:29:16.381441+0000","last_peered":"2023-10-16T12:29:16.381441+0000","last_clean":"2023-10-16T12:29:16.381441+0000","last_became_active":"2023-10-16T12:29:15.389146+0000","last_became_peered":"2023-10-16T12:29:15.389146+0000","last_unstale":"2023-10-16T12:29:16.381441+0000","last_undegraded":"2023-10-16T12:29:16.381441+0000","last_fullsized":"2023-10-16T12:29:16.381441+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_clean_scrub_stamp":"2023-10-16T12:29:14.359222+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:39:59.922766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.301445+0000","last_change":"2023-10-16T12:29:16.299426+0000","last_active":"2023-10-16T12:29:16.301445+0000","last_peered":"2023-10-16T12:29:16.301445+0000","last_clean":"2023-10-16T12:29:16.301445+0000","last_became_active":"2023-10-16T12:29:16.299099+0000","last_became_peered":"2023-10-16T12:29:16.299099+0000","last_unstale":"2023-10-16T12:29:16.301445+0000","last_undegraded":"2023-10-16T12:29:16.301445+0000","last_fullsized":"2023-10-16T12:29:16.301445+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:02:25.157483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.1c","version":"127'2","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.382231+0000","last_change":"2023-10-16T12:29:15.389220+0000","last_active":"2023-10-16T12:29:16.382231+0000","last_peered":"2023-10-16T12:29:16.382231+0000","last_clean":"2023-10-16T12:29:16.382231+0000","last_became_active":"2023-10-16T12:29:15.388900+0000","last_became_peered":"2023-10-16T12:29:15.388900+0000","last_unstale":"2023-10-16T12:29:16.382231+0000","last_undegraded":"2023-10-16T12:29:16.382231+0000","last_fullsized":"2023-10-16T12:29:16.382231+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_clean_scrub_stamp":"2023-10-16T12:29:14.359222+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:58:30.142694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.286282+0000","last_change":"2023-10-16T12:29:16.283097+0000","last_active":"2023-10-16T12:29:16.286282+0000","last_peered":"2023-10-16T12:29:16.286282+0000","last_clean":"2023-10-16T12:29:16.286282+0000","last_became_active":"2023-10-16T12:29:16.282597+0000","last_became_peered":"2023-10-16T12:29:16.282597+0000","last_unstale":"2023-10-16T12:29:16.286282+0000","last_undegraded":"2023-10-16T12:29:16.286282+0000","last_fullsized":"2023-10-16T12:29:16.286282+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:15:33.049530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1b","version":"127'1","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.380428+0000","last_change":"2023-10-16T12:29:15.382823+0000","last_active":"2023-10-16T12:29:16.380428+0000","last_peered":"2023-10-16T12:29:16.380428+0000","last_clean":"2023-10-16T12:29:16.380428+0000","last_became_active":"2023-10-16T12:29:15.382516+0000","last_became_peered":"2023-10-16T12:29:15.382516+0000","last_unstale":"2023-10-16T12:29:16.380428+0000","last_undegraded":"2023-10-16T12:29:16.380428+0000","last_fullsized":"2023-10-16T12:29:16.380428+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_clean_scrub_stamp":"2023-10-16T12:29:14.359222+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:35:20.655861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.304185+0000","last_change":"2023-10-16T12:29:16.300472+0000","last_active":"2023-10-16T12:29:16.304185+0000","last_peered":"2023-10-16T12:29:16.304185+0000","last_clean":"2023-10-16T12:29:16.304185+0000","last_became_active":"2023-10-16T12:29:16.299823+0000","last_became_peered":"2023-10-16T12:29:16.299823+0000","last_unstale":"2023-10-16T12:29:16.304185+0000","last_undegraded":"2023-10-16T12:29:16.304185+0000","last_fullsized":"2023-10-16T12:29:16.304185+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:46:00.177635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"127'1","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.381539+0000","last_change":"2023-10-16T12:29:15.391284+0000","last_active":"2023-10-16T12:29:16.381539+0000","last_peered":"2023-10-16T12:29:16.381539+0000","last_clean":"2023-10-16T12:29:16.381539+0000","last_became_active":"2023-10-16T12:29:15.390981+0000","last_became_peered":"2023-10-16T12:29:15.390981+0000","last_unstale":"2023-10-16T12:29:16.381539+0000","last_undegraded":"2023-10-16T12:29:16.381539+0000","last_fullsized":"2023-10-16T12:29:16.381539+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_clean_scrub_stamp":"2023-10-16T12:29:14.359222+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:24:50.244826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.304496+0000","last_change":"2023-10-16T12:29:16.303165+0000","last_active":"2023-10-16T12:29:16.304496+0000","last_peered":"2023-10-16T12:29:16.304496+0000","last_clean":"2023-10-16T12:29:16.304496+0000","last_became_active":"2023-10-16T12:29:16.295443+0000","last_became_peered":"2023-10-16T12:29:16.295443+0000","last_unstale":"2023-10-16T12:29:16.304496+0000","last_undegraded":"2023-10-16T12:29:16.304496+0000","last_fullsized":"2023-10-16T12:29:16.304496+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:53:44.212676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.285610+0000","last_change":"2023-10-16T12:29:15.389676+0000","last_active":"2023-10-16T12:29:16.285610+0000","last_peered":"2023-10-16T12:29:16.285610+0000","last_clean":"2023-10-16T12:29:16.285610+0000","last_became_active":"2023-10-16T12:29:15.389359+0000","last_became_peered":"2023-10-16T12:29:15.389359+0000","last_unstale":"2023-10-16T12:29:16.285610+0000","last_undegraded":"2023-10-16T12:29:16.285610+0000","last_fullsized":"2023-10-16T12:29:16.285610+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_clean_scrub_stamp":"2023-10-16T12:29:14.359222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:39:57.128076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.299939+0000","last_change":"2023-10-16T12:29:16.296899+0000","last_active":"2023-10-16T12:29:16.299939+0000","last_peered":"2023-10-16T12:29:16.299939+0000","last_clean":"2023-10-16T12:29:16.299939+0000","last_became_active":"2023-10-16T12:29:16.296569+0000","last_became_peered":"2023-10-16T12:29:16.296569+0000","last_unstale":"2023-10-16T12:29:16.299939+0000","last_undegraded":"2023-10-16T12:29:16.299939+0000","last_fullsized":"2023-10-16T12:29:16.299939+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:47:23.445036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.280980+0000","last_change":"2023-10-16T12:29:15.389613+0000","last_active":"2023-10-16T12:29:16.280980+0000","last_peered":"2023-10-16T12:29:16.280980+0000","last_clean":"2023-10-16T12:29:16.280980+0000","last_became_active":"2023-10-16T12:29:15.389287+0000","last_became_peered":"2023-10-16T12:29:15.389287+0000","last_unstale":"2023-10-16T12:29:16.280980+0000","last_undegraded":"2023-10-16T12:29:16.280980+0000","last_fullsized":"2023-10-16T12:29:16.280980+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_clean_scrub_stamp":"2023-10-16T12:29:14.359222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:30:18.319981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.304211+0000","last_change":"2023-10-16T12:29:16.300614+0000","last_active":"2023-10-16T12:29:16.304211+0000","last_peered":"2023-10-16T12:29:16.304211+0000","last_clean":"2023-10-16T12:29:16.304211+0000","last_became_active":"2023-10-16T12:29:16.300315+0000","last_became_peered":"2023-10-16T12:29:16.300315+0000","last_unstale":"2023-10-16T12:29:16.304211+0000","last_undegraded":"2023-10-16T12:29:16.304211+0000","last_fullsized":"2023-10-16T12:29:16.304211+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:38:24.159543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.289816+0000","last_change":"2023-10-16T12:29:15.390204+0000","last_active":"2023-10-16T12:29:16.289816+0000","last_peered":"2023-10-16T12:29:16.289816+0000","last_clean":"2023-10-16T12:29:16.289816+0000","last_became_active":"2023-10-16T12:29:15.389919+0000","last_became_peered":"2023-10-16T12:29:15.389919+0000","last_unstale":"2023-10-16T12:29:16.289816+0000","last_undegraded":"2023-10-16T12:29:16.289816+0000","last_fullsized":"2023-10-16T12:29:16.289816+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_clean_scrub_stamp":"2023-10-16T12:29:14.359222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:59:49.605236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.290159+0000","last_change":"2023-10-16T12:29:16.287910+0000","last_active":"2023-10-16T12:29:16.290159+0000","last_peered":"2023-10-16T12:29:16.290159+0000","last_clean":"2023-10-16T12:29:16.290159+0000","last_became_active":"2023-10-16T12:29:16.287626+0000","last_became_peered":"2023-10-16T12:29:16.287626+0000","last_unstale":"2023-10-16T12:29:16.290159+0000","last_undegraded":"2023-10-16T12:29:16.290159+0000","last_fullsized":"2023-10-16T12:29:16.290159+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:56:18.062438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.292896+0000","last_change":"2023-10-16T12:29:15.392631+0000","last_active":"2023-10-16T12:29:16.292896+0000","last_peered":"2023-10-16T12:29:16.292896+0000","last_clean":"2023-10-16T12:29:16.292896+0000","last_became_active":"2023-10-16T12:29:15.392138+0000","last_became_peered":"2023-10-16T12:29:15.392138+0000","last_unstale":"2023-10-16T12:29:16.292896+0000","last_undegraded":"2023-10-16T12:29:16.292896+0000","last_fullsized":"2023-10-16T12:29:16.292896+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_clean_scrub_stamp":"2023-10-16T12:29:14.359222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:38:34.083984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.292564+0000","last_change":"2023-10-16T12:29:16.291543+0000","last_active":"2023-10-16T12:29:16.292564+0000","last_peered":"2023-10-16T12:29:16.292564+0000","last_clean":"2023-10-16T12:29:16.292564+0000","last_became_active":"2023-10-16T12:29:16.291290+0000","last_became_peered":"2023-10-16T12:29:16.291290+0000","last_unstale":"2023-10-16T12:29:16.292564+0000","last_undegraded":"2023-10-16T12:29:16.292564+0000","last_fullsized":"2023-10-16T12:29:16.292564+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:52:12.812015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.15","version":"127'3","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.381581+0000","last_change":"2023-10-16T12:29:15.393973+0000","last_active":"2023-10-16T12:29:16.381581+0000","last_peered":"2023-10-16T12:29:16.381581+0000","last_clean":"2023-10-16T12:29:16.381581+0000","last_became_active":"2023-10-16T12:29:15.393692+0000","last_became_peered":"2023-10-16T12:29:15.393692+0000","last_unstale":"2023-10-16T12:29:16.381581+0000","last_undegraded":"2023-10-16T12:29:16.381581+0000","last_fullsized":"2023-10-16T12:29:16.381581+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_clean_scrub_stamp":"2023-10-16T12:29:14.359222+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:24:31.663023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.292791+0000","last_change":"2023-10-16T12:29:16.291588+0000","last_active":"2023-10-16T12:29:16.292791+0000","last_peered":"2023-10-16T12:29:16.292791+0000","last_clean":"2023-10-16T12:29:16.292791+0000","last_became_active":"2023-10-16T12:29:16.291308+0000","last_became_peered":"2023-10-16T12:29:16.291308+0000","last_unstale":"2023-10-16T12:29:16.292791+0000","last_undegraded":"2023-10-16T12:29:16.292791+0000","last_fullsized":"2023-10-16T12:29:16.292791+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:39:26.840175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.14","version":"127'6","reported_seq":17,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.383321+0000","last_change":"2023-10-16T12:29:15.392511+0000","last_active":"2023-10-16T12:29:16.383321+0000","last_peered":"2023-10-16T12:29:16.383321+0000","last_clean":"2023-10-16T12:29:16.383321+0000","last_became_active":"2023-10-16T12:29:15.391941+0000","last_became_peered":"2023-10-16T12:29:15.391941+0000","last_unstale":"2023-10-16T12:29:16.383321+0000","last_undegraded":"2023-10-16T12:29:16.383321+0000","last_fullsized":"2023-10-16T12:29:16.383321+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_clean_scrub_stamp":"2023-10-16T12:29:14.359222+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:58:02.589102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.301383+0000","last_change":"2023-10-16T12:29:16.298832+0000","last_active":"2023-10-16T12:29:16.301383+0000","last_peered":"2023-10-16T12:29:16.301383+0000","last_clean":"2023-10-16T12:29:16.301383+0000","last_became_active":"2023-10-16T12:29:16.298089+0000","last_became_peered":"2023-10-16T12:29:16.298089+0000","last_unstale":"2023-10-16T12:29:16.301383+0000","last_undegraded":"2023-10-16T12:29:16.301383+0000","last_fullsized":"2023-10-16T12:29:16.301383+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:25:32.385405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.13","version":"127'2","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.382317+0000","last_change":"2023-10-16T12:29:15.384560+0000","last_active":"2023-10-16T12:29:16.382317+0000","last_peered":"2023-10-16T12:29:16.382317+0000","last_clean":"2023-10-16T12:29:16.382317+0000","last_became_active":"2023-10-16T12:29:15.384283+0000","last_became_peered":"2023-10-16T12:29:15.384283+0000","last_unstale":"2023-10-16T12:29:16.382317+0000","last_undegraded":"2023-10-16T12:29:16.382317+0000","last_fullsized":"2023-10-16T12:29:16.382317+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_clean_scrub_stamp":"2023-10-16T12:29:14.359222+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:21:31.258287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.293300+0000","last_change":"2023-10-16T12:29:16.292325+0000","last_active":"2023-10-16T12:29:16.293300+0000","last_peered":"2023-10-16T12:29:16.293300+0000","last_clean":"2023-10-16T12:29:16.293300+0000","last_became_active":"2023-10-16T12:29:16.291480+0000","last_became_peered":"2023-10-16T12:29:16.291480+0000","last_unstale":"2023-10-16T12:29:16.293300+0000","last_undegraded":"2023-10-16T12:29:16.293300+0000","last_fullsized":"2023-10-16T12:29:16.293300+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:07:26.551366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.281410+0000","last_change":"2023-10-16T12:29:15.389559+0000","last_active":"2023-10-16T12:29:16.281410+0000","last_peered":"2023-10-16T12:29:16.281410+0000","last_clean":"2023-10-16T12:29:16.281410+0000","last_became_active":"2023-10-16T12:29:15.389268+0000","last_became_peered":"2023-10-16T12:29:15.389268+0000","last_unstale":"2023-10-16T12:29:16.281410+0000","last_undegraded":"2023-10-16T12:29:16.281410+0000","last_fullsized":"2023-10-16T12:29:16.281410+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_clean_scrub_stamp":"2023-10-16T12:29:14.359222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:40:16.898102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.297098+0000","last_change":"2023-10-16T12:29:16.296508+0000","last_active":"2023-10-16T12:29:16.297098+0000","last_peered":"2023-10-16T12:29:16.297098+0000","last_clean":"2023-10-16T12:29:16.297098+0000","last_became_active":"2023-10-16T12:29:16.296228+0000","last_became_peered":"2023-10-16T12:29:16.296228+0000","last_unstale":"2023-10-16T12:29:16.297098+0000","last_undegraded":"2023-10-16T12:29:16.297098+0000","last_fullsized":"2023-10-16T12:29:16.297098+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:36:12.325088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.284044+0000","last_change":"2023-10-16T12:29:15.387981+0000","last_active":"2023-10-16T12:29:16.284044+0000","last_peered":"2023-10-16T12:29:16.284044+0000","last_clean":"2023-10-16T12:29:16.284044+0000","last_became_active":"2023-10-16T12:29:15.387545+0000","last_became_peered":"2023-10-16T12:29:15.387545+0000","last_unstale":"2023-10-16T12:29:16.284044+0000","last_undegraded":"2023-10-16T12:29:16.284044+0000","last_fullsized":"2023-10-16T12:29:16.284044+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_clean_scrub_stamp":"2023-10-16T12:29:14.359222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:49:24.712423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.303659+0000","last_change":"2023-10-16T12:29:16.295754+0000","last_active":"2023-10-16T12:29:16.303659+0000","last_peered":"2023-10-16T12:29:16.303659+0000","last_clean":"2023-10-16T12:29:16.303659+0000","last_became_active":"2023-10-16T12:29:16.295492+0000","last_became_peered":"2023-10-16T12:29:16.295492+0000","last_unstale":"2023-10-16T12:29:16.303659+0000","last_undegraded":"2023-10-16T12:29:16.303659+0000","last_fullsized":"2023-10-16T12:29:16.303659+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:25:45.881535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.10","version":"127'8","reported_seq":19,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.382446+0000","last_change":"2023-10-16T12:29:15.389747+0000","last_active":"2023-10-16T12:29:16.382446+0000","last_peered":"2023-10-16T12:29:16.382446+0000","last_clean":"2023-10-16T12:29:16.382446+0000","last_became_active":"2023-10-16T12:29:15.389112+0000","last_became_peered":"2023-10-16T12:29:15.389112+0000","last_unstale":"2023-10-16T12:29:16.382446+0000","last_undegraded":"2023-10-16T12:29:16.382446+0000","last_fullsized":"2023-10-16T12:29:16.382446+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_clean_scrub_stamp":"2023-10-16T12:29:14.359222+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:38:24.764875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.301098+0000","last_change":"2023-10-16T12:29:16.300770+0000","last_active":"2023-10-16T12:29:16.301098+0000","last_peered":"2023-10-16T12:29:16.301098+0000","last_clean":"2023-10-16T12:29:16.301098+0000","last_became_active":"2023-10-16T12:29:16.300498+0000","last_became_peered":"2023-10-16T12:29:16.300498+0000","last_unstale":"2023-10-16T12:29:16.301098+0000","last_undegraded":"2023-10-16T12:29:16.301098+0000","last_fullsized":"2023-10-16T12:29:16.301098+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:15:33.970346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.f","version":"127'1","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.380527+0000","last_change":"2023-10-16T12:29:15.386345+0000","last_active":"2023-10-16T12:29:16.380527+0000","last_peered":"2023-10-16T12:29:16.380527+0000","last_clean":"2023-10-16T12:29:16.380527+0000","last_became_active":"2023-10-16T12:29:15.386052+0000","last_became_peered":"2023-10-16T12:29:15.386052+0000","last_unstale":"2023-10-16T12:29:16.380527+0000","last_undegraded":"2023-10-16T12:29:16.380527+0000","last_fullsized":"2023-10-16T12:29:16.380527+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_clean_scrub_stamp":"2023-10-16T12:29:14.359222+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:12:16.756683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.295340+0000","last_change":"2023-10-16T12:29:16.293616+0000","last_active":"2023-10-16T12:29:16.295340+0000","last_peered":"2023-10-16T12:29:16.295340+0000","last_clean":"2023-10-16T12:29:16.295340+0000","last_became_active":"2023-10-16T12:29:16.293202+0000","last_became_peered":"2023-10-16T12:29:16.293202+0000","last_unstale":"2023-10-16T12:29:16.295340+0000","last_undegraded":"2023-10-16T12:29:16.295340+0000","last_fullsized":"2023-10-16T12:29:16.295340+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:49:07.841401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"127'1","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.381657+0000","last_change":"2023-10-16T12:29:15.388508+0000","last_active":"2023-10-16T12:29:16.381657+0000","last_peered":"2023-10-16T12:29:16.381657+0000","last_clean":"2023-10-16T12:29:16.381657+0000","last_became_active":"2023-10-16T12:29:15.388179+0000","last_became_peered":"2023-10-16T12:29:15.388179+0000","last_unstale":"2023-10-16T12:29:16.381657+0000","last_undegraded":"2023-10-16T12:29:16.381657+0000","last_fullsized":"2023-10-16T12:29:16.381657+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_clean_scrub_stamp":"2023-10-16T12:29:14.359222+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:15:32.044492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.299045+0000","last_change":"2023-10-16T12:29:16.298101+0000","last_active":"2023-10-16T12:29:16.299045+0000","last_peered":"2023-10-16T12:29:16.299045+0000","last_clean":"2023-10-16T12:29:16.299045+0000","last_became_active":"2023-10-16T12:29:16.297814+0000","last_became_peered":"2023-10-16T12:29:16.297814+0000","last_unstale":"2023-10-16T12:29:16.299045+0000","last_undegraded":"2023-10-16T12:29:16.299045+0000","last_fullsized":"2023-10-16T12:29:16.299045+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:58:27.254916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.284766+0000","last_change":"2023-10-16T12:29:15.387284+0000","last_active":"2023-10-16T12:29:16.284766+0000","last_peered":"2023-10-16T12:29:16.284766+0000","last_clean":"2023-10-16T12:29:16.284766+0000","last_became_active":"2023-10-16T12:29:15.386957+0000","last_became_peered":"2023-10-16T12:29:15.386957+0000","last_unstale":"2023-10-16T12:29:16.284766+0000","last_undegraded":"2023-10-16T12:29:16.284766+0000","last_fullsized":"2023-10-16T12:29:16.284766+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_clean_scrub_stamp":"2023-10-16T12:29:14.359222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:47:28.957497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.300077+0000","last_change":"2023-10-16T12:29:16.295453+0000","last_active":"2023-10-16T12:29:16.300077+0000","last_peered":"2023-10-16T12:29:16.300077+0000","last_clean":"2023-10-16T12:29:16.300077+0000","last_became_active":"2023-10-16T12:29:16.295177+0000","last_became_peered":"2023-10-16T12:29:16.295177+0000","last_unstale":"2023-10-16T12:29:16.300077+0000","last_undegraded":"2023-10-16T12:29:16.300077+0000","last_fullsized":"2023-10-16T12:29:16.300077+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:32:17.862905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"127'2","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.377640+0000","last_change":"2023-10-16T12:29:15.384862+0000","last_active":"2023-10-16T12:29:16.377640+0000","last_peered":"2023-10-16T12:29:16.377640+0000","last_clean":"2023-10-16T12:29:16.377640+0000","last_became_active":"2023-10-16T12:29:15.384500+0000","last_became_peered":"2023-10-16T12:29:15.384500+0000","last_unstale":"2023-10-16T12:29:16.377640+0000","last_undegraded":"2023-10-16T12:29:16.377640+0000","last_fullsized":"2023-10-16T12:29:16.377640+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_clean_scrub_stamp":"2023-10-16T12:29:14.359222+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:11:28.075541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.303777+0000","last_change":"2023-10-16T12:29:16.297361+0000","last_active":"2023-10-16T12:29:16.303777+0000","last_peered":"2023-10-16T12:29:16.303777+0000","last_clean":"2023-10-16T12:29:16.303777+0000","last_became_active":"2023-10-16T12:29:16.293132+0000","last_became_peered":"2023-10-16T12:29:16.293132+0000","last_unstale":"2023-10-16T12:29:16.303777+0000","last_undegraded":"2023-10-16T12:29:16.303777+0000","last_fullsized":"2023-10-16T12:29:16.303777+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:46:48.194963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.284152+0000","last_change":"2023-10-16T12:29:15.390086+0000","last_active":"2023-10-16T12:29:16.284152+0000","last_peered":"2023-10-16T12:29:16.284152+0000","last_clean":"2023-10-16T12:29:16.284152+0000","last_became_active":"2023-10-16T12:29:15.389832+0000","last_became_peered":"2023-10-16T12:29:15.389832+0000","last_unstale":"2023-10-16T12:29:16.284152+0000","last_undegraded":"2023-10-16T12:29:16.284152+0000","last_fullsized":"2023-10-16T12:29:16.284152+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_clean_scrub_stamp":"2023-10-16T12:29:14.359222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:45:00.939929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.293989+0000","last_change":"2023-10-16T12:29:16.293273+0000","last_active":"2023-10-16T12:29:16.293989+0000","last_peered":"2023-10-16T12:29:16.293989+0000","last_clean":"2023-10-16T12:29:16.293989+0000","last_became_active":"2023-10-16T12:29:16.292979+0000","last_became_peered":"2023-10-16T12:29:16.292979+0000","last_unstale":"2023-10-16T12:29:16.293989+0000","last_undegraded":"2023-10-16T12:29:16.293989+0000","last_fullsized":"2023-10-16T12:29:16.293989+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:24:32.668061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.284025+0000","last_change":"2023-10-16T12:29:15.389602+0000","last_active":"2023-10-16T12:29:16.284025+0000","last_peered":"2023-10-16T12:29:16.284025+0000","last_clean":"2023-10-16T12:29:16.284025+0000","last_became_active":"2023-10-16T12:29:15.389318+0000","last_became_peered":"2023-10-16T12:29:15.389318+0000","last_unstale":"2023-10-16T12:29:16.284025+0000","last_undegraded":"2023-10-16T12:29:16.284025+0000","last_fullsized":"2023-10-16T12:29:16.284025+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_clean_scrub_stamp":"2023-10-16T12:29:14.359222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:22:46.277491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.293186+0000","last_change":"2023-10-16T12:29:16.292152+0000","last_active":"2023-10-16T12:29:16.293186+0000","last_peered":"2023-10-16T12:29:16.293186+0000","last_clean":"2023-10-16T12:29:16.293186+0000","last_became_active":"2023-10-16T12:29:16.290631+0000","last_became_peered":"2023-10-16T12:29:16.290631+0000","last_unstale":"2023-10-16T12:29:16.293186+0000","last_undegraded":"2023-10-16T12:29:16.293186+0000","last_fullsized":"2023-10-16T12:29:16.293186+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:27:43.984098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.284569+0000","last_change":"2023-10-16T12:29:15.390056+0000","last_active":"2023-10-16T12:29:16.284569+0000","last_peered":"2023-10-16T12:29:16.284569+0000","last_clean":"2023-10-16T12:29:16.284569+0000","last_became_active":"2023-10-16T12:29:15.389777+0000","last_became_peered":"2023-10-16T12:29:15.389777+0000","last_unstale":"2023-10-16T12:29:16.284569+0000","last_undegraded":"2023-10-16T12:29:16.284569+0000","last_fullsized":"2023-10-16T12:29:16.284569+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_clean_scrub_stamp":"2023-10-16T12:29:14.359222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:12:28.782296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.304197+0000","last_change":"2023-10-16T12:29:16.300538+0000","last_active":"2023-10-16T12:29:16.304197+0000","last_peered":"2023-10-16T12:29:16.304197+0000","last_clean":"2023-10-16T12:29:16.304197+0000","last_became_active":"2023-10-16T12:29:16.300029+0000","last_became_peered":"2023-10-16T12:29:16.300029+0000","last_unstale":"2023-10-16T12:29:16.304197+0000","last_undegraded":"2023-10-16T12:29:16.304197+0000","last_fullsized":"2023-10-16T12:29:16.304197+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:11:30.050315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.7","version":"127'2","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.382315+0000","last_change":"2023-10-16T12:29:15.385060+0000","last_active":"2023-10-16T12:29:16.382315+0000","last_peered":"2023-10-16T12:29:16.382315+0000","last_clean":"2023-10-16T12:29:16.382315+0000","last_became_active":"2023-10-16T12:29:15.384763+0000","last_became_peered":"2023-10-16T12:29:15.384763+0000","last_unstale":"2023-10-16T12:29:16.382315+0000","last_undegraded":"2023-10-16T12:29:16.382315+0000","last_fullsized":"2023-10-16T12:29:16.382315+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_clean_scrub_stamp":"2023-10-16T12:29:14.359222+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:48:43.877276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.299892+0000","last_change":"2023-10-16T12:29:16.296485+0000","last_active":"2023-10-16T12:29:16.299892+0000","last_peered":"2023-10-16T12:29:16.299892+0000","last_clean":"2023-10-16T12:29:16.299892+0000","last_became_active":"2023-10-16T12:29:16.296178+0000","last_became_peered":"2023-10-16T12:29:16.296178+0000","last_unstale":"2023-10-16T12:29:16.299892+0000","last_undegraded":"2023-10-16T12:29:16.299892+0000","last_fullsized":"2023-10-16T12:29:16.299892+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:33:28.001948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.292863+0000","last_change":"2023-10-16T12:29:15.392423+0000","last_active":"2023-10-16T12:29:16.292863+0000","last_peered":"2023-10-16T12:29:16.292863+0000","last_clean":"2023-10-16T12:29:16.292863+0000","last_became_active":"2023-10-16T12:29:15.391765+0000","last_became_peered":"2023-10-16T12:29:15.391765+0000","last_unstale":"2023-10-16T12:29:16.292863+0000","last_undegraded":"2023-10-16T12:29:16.292863+0000","last_fullsized":"2023-10-16T12:29:16.292863+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_clean_scrub_stamp":"2023-10-16T12:29:14.359222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:17:59.782860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.293944+0000","last_change":"2023-10-16T12:29:16.292605+0000","last_active":"2023-10-16T12:29:16.293944+0000","last_peered":"2023-10-16T12:29:16.293944+0000","last_clean":"2023-10-16T12:29:16.293944+0000","last_became_active":"2023-10-16T12:29:16.292293+0000","last_became_peered":"2023-10-16T12:29:16.292293+0000","last_unstale":"2023-10-16T12:29:16.293944+0000","last_undegraded":"2023-10-16T12:29:16.293944+0000","last_fullsized":"2023-10-16T12:29:16.293944+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:03:19.526603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.283615+0000","last_change":"2023-10-16T12:29:15.390558+0000","last_active":"2023-10-16T12:29:16.283615+0000","last_peered":"2023-10-16T12:29:16.283615+0000","last_clean":"2023-10-16T12:29:16.283615+0000","last_became_active":"2023-10-16T12:29:15.390202+0000","last_became_peered":"2023-10-16T12:29:15.390202+0000","last_unstale":"2023-10-16T12:29:16.283615+0000","last_undegraded":"2023-10-16T12:29:16.283615+0000","last_fullsized":"2023-10-16T12:29:16.283615+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_clean_scrub_stamp":"2023-10-16T12:29:14.359222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:59:20.248769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.295964+0000","last_change":"2023-10-16T12:29:16.294448+0000","last_active":"2023-10-16T12:29:16.295964+0000","last_peered":"2023-10-16T12:29:16.295964+0000","last_clean":"2023-10-16T12:29:16.295964+0000","last_became_active":"2023-10-16T12:29:16.294105+0000","last_became_peered":"2023-10-16T12:29:16.294105+0000","last_unstale":"2023-10-16T12:29:16.295964+0000","last_undegraded":"2023-10-16T12:29:16.295964+0000","last_fullsized":"2023-10-16T12:29:16.295964+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:20:05.649120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.279301+0000","last_change":"2023-10-16T12:29:15.388073+0000","last_active":"2023-10-16T12:29:16.279301+0000","last_peered":"2023-10-16T12:29:16.279301+0000","last_clean":"2023-10-16T12:29:16.279301+0000","last_became_active":"2023-10-16T12:29:15.387780+0000","last_became_peered":"2023-10-16T12:29:15.387780+0000","last_unstale":"2023-10-16T12:29:16.279301+0000","last_undegraded":"2023-10-16T12:29:16.279301+0000","last_fullsized":"2023-10-16T12:29:16.279301+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_clean_scrub_stamp":"2023-10-16T12:29:14.359222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:58:02.589102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.292745+0000","last_change":"2023-10-16T12:29:16.291463+0000","last_active":"2023-10-16T12:29:16.292745+0000","last_peered":"2023-10-16T12:29:16.292745+0000","last_clean":"2023-10-16T12:29:16.292745+0000","last_became_active":"2023-10-16T12:29:16.291203+0000","last_became_peered":"2023-10-16T12:29:16.291203+0000","last_unstale":"2023-10-16T12:29:16.292745+0000","last_undegraded":"2023-10-16T12:29:16.292745+0000","last_fullsized":"2023-10-16T12:29:16.292745+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:18:44.204916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.289137+0000","last_change":"2023-10-16T12:29:15.384127+0000","last_active":"2023-10-16T12:29:16.289137+0000","last_peered":"2023-10-16T12:29:16.289137+0000","last_clean":"2023-10-16T12:29:16.289137+0000","last_became_active":"2023-10-16T12:29:15.383826+0000","last_became_peered":"2023-10-16T12:29:15.383826+0000","last_unstale":"2023-10-16T12:29:16.289137+0000","last_undegraded":"2023-10-16T12:29:16.289137+0000","last_fullsized":"2023-10-16T12:29:16.289137+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_clean_scrub_stamp":"2023-10-16T12:29:14.359222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:42:36.058919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.292855+0000","last_change":"2023-10-16T12:29:16.290924+0000","last_active":"2023-10-16T12:29:16.292855+0000","last_peered":"2023-10-16T12:29:16.292855+0000","last_clean":"2023-10-16T12:29:16.292855+0000","last_became_active":"2023-10-16T12:29:16.290643+0000","last_became_peered":"2023-10-16T12:29:16.290643+0000","last_unstale":"2023-10-16T12:29:16.292855+0000","last_undegraded":"2023-10-16T12:29:16.292855+0000","last_fullsized":"2023-10-16T12:29:16.292855+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:35:11.392279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.284316+0000","last_change":"2023-10-16T12:29:15.391874+0000","last_active":"2023-10-16T12:29:16.284316+0000","last_peered":"2023-10-16T12:29:16.284316+0000","last_clean":"2023-10-16T12:29:16.284316+0000","last_became_active":"2023-10-16T12:29:15.391527+0000","last_became_peered":"2023-10-16T12:29:15.391527+0000","last_unstale":"2023-10-16T12:29:16.284316+0000","last_undegraded":"2023-10-16T12:29:16.284316+0000","last_fullsized":"2023-10-16T12:29:16.284316+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_clean_scrub_stamp":"2023-10-16T12:29:14.359222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:47:27.045462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.299696+0000","last_change":"2023-10-16T12:29:16.297340+0000","last_active":"2023-10-16T12:29:16.299696+0000","last_peered":"2023-10-16T12:29:16.299696+0000","last_clean":"2023-10-16T12:29:16.299696+0000","last_became_active":"2023-10-16T12:29:16.297009+0000","last_became_peered":"2023-10-16T12:29:16.297009+0000","last_unstale":"2023-10-16T12:29:16.299696+0000","last_undegraded":"2023-10-16T12:29:16.299696+0000","last_fullsized":"2023-10-16T12:29:16.299696+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:30:19.325019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.284077+0000","last_change":"2023-10-16T12:29:15.385906+0000","last_active":"2023-10-16T12:29:16.284077+0000","last_peered":"2023-10-16T12:29:16.284077+0000","last_clean":"2023-10-16T12:29:16.284077+0000","last_became_active":"2023-10-16T12:29:15.385603+0000","last_became_peered":"2023-10-16T12:29:15.385603+0000","last_unstale":"2023-10-16T12:29:16.284077+0000","last_undegraded":"2023-10-16T12:29:16.284077+0000","last_fullsized":"2023-10-16T12:29:16.284077+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_clean_scrub_stamp":"2023-10-16T12:29:14.359222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:49:09.895788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.298686+0000","last_change":"2023-10-16T12:29:16.296586+0000","last_active":"2023-10-16T12:29:16.298686+0000","last_peered":"2023-10-16T12:29:16.298686+0000","last_clean":"2023-10-16T12:29:16.298686+0000","last_became_active":"2023-10-16T12:29:16.296300+0000","last_became_peered":"2023-10-16T12:29:16.296300+0000","last_unstale":"2023-10-16T12:29:16.298686+0000","last_undegraded":"2023-10-16T12:29:16.298686+0000","last_fullsized":"2023-10-16T12:29:16.298686+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:40:07.478573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":11,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.286316+0000","last_change":"2023-10-16T12:29:15.387886+0000","last_active":"2023-10-16T12:29:16.286316+0000","last_peered":"2023-10-16T12:29:16.286316+0000","last_clean":"2023-10-16T12:29:16.286316+0000","last_became_active":"2023-10-16T12:29:15.387641+0000","last_became_peered":"2023-10-16T12:29:15.387641+0000","last_unstale":"2023-10-16T12:29:16.286316+0000","last_undegraded":"2023-10-16T12:29:16.286316+0000","last_fullsized":"2023-10-16T12:29:16.286316+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_clean_scrub_stamp":"2023-10-16T12:29:14.359222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:24:38.015313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":9,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.301217+0000","last_change":"2023-10-16T12:29:16.298167+0000","last_active":"2023-10-16T12:29:16.301217+0000","last_peered":"2023-10-16T12:29:16.301217+0000","last_clean":"2023-10-16T12:29:16.301217+0000","last_became_active":"2023-10-16T12:29:16.297855+0000","last_became_peered":"2023-10-16T12:29:16.297855+0000","last_unstale":"2023-10-16T12:29:16.301217+0000","last_undegraded":"2023-10-16T12:29:16.301217+0000","last_fullsized":"2023-10-16T12:29:16.301217+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:15.364260+0000","last_clean_scrub_stamp":"2023-10-16T12:29:15.364260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:46:09.057491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.e","version":"127'2","reported_seq":13,"reported_epoch":127,"state":"active+clean","last_fresh":"2023-10-16T12:29:16.381173+0000","last_change":"2023-10-16T12:29:15.389554+0000","last_active":"2023-10-16T12:29:16.381173+0000","last_peered":"2023-10-16T12:29:16.381173+0000","last_clean":"2023-10-16T12:29:16.381173+0000","last_became_active":"2023-10-16T12:29:15.388188+0000","last_became_peered":"2023-10-16T12:29:15.388188+0000","last_unstale":"2023-10-16T12:29:16.381173+0000","last_undegraded":"2023-10-16T12:29:16.381173+0000","last_fullsized":"2023-10-16T12:29:16.381173+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:29:14.359222+0000","last_clean_scrub_stamp":"2023-10-16T12:29:14.359222+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:42:36.995029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"117'205","reported_seq":576,"reported_epoch":126,"state":"active+clean","last_fresh":"2023-10-16T12:29:15.377559+0000","last_change":"2023-10-16T12:10:34.746820+0000","last_active":"2023-10-16T12:29:15.377559+0000","last_peered":"2023-10-16T12:29:15.377559+0000","last_clean":"2023-10-16T12:29:15.377559+0000","last_became_active":"2023-10-16T12:10:34.745574+0000","last_became_peered":"2023-10-16T12:10:34.745574+0000","last_unstale":"2023-10-16T12:29:15.377559+0000","last_undegraded":"2023-10-16T12:29:15.377559+0000","last_fullsized":"2023-10-16T12:29:15.377559+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_clean_scrub_stamp":"2023-10-16T12:10:33.577111+0000","objects_scrubbed":0,"log_size":205,"log_dups_size":0,"ondisk_log_size":205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:14.461986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":884,"num_read_kb":1748,"num_write":319,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":16,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":15,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":884,"num_read_kb":1748,"num_write":319,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":205,"ondisk_log_size":205,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575071,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63836,"kb_used_data":2124,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94308004,"statfs":{"total":96636764160,"available":96571396096,"internally_reserved":0,"allocated":2174976,"data_stored":1382234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":63175115},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:28:12 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.59199999999999997,"15min":0.60399999999999998},"min":{"1min":0.39400000000000002,"5min":0.31,"15min":0.27400000000000002},"max":{"1min":4.2469999999999999,"5min":4.2469999999999999,"15min":4.2469999999999999},"last":0.5},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.56100000000000005,"15min":0.58299999999999996},"min":{"1min":0.40100000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.746,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.84099999999999997}]},{"osd":1,"last update":"Mon Oct 16 12:28:12 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.56499999999999995,"15min":0.60299999999999998},"min":{"1min":0.38100000000000001,"5min":0.35499999999999998,"15min":0.247},"max":{"1min":0.69299999999999995,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.58899999999999997,"15min":0.60599999999999998},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.32400000000000001},"max":{"1min":0.79100000000000004,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.66800000000000004}]},{"osd":2,"last update":"Mon Oct 16 12:28:12 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.63700000000000001,"15min":0.66000000000000003},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.36099999999999999},"max":{"1min":0.77300000000000002,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.64200000000000002,"15min":0.66700000000000004},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.376},"max":{"1min":0.76600000000000001,"5min":0.94599999999999995,"15min":0.96699999999999997},"last":0.75800000000000001}]},{"osd":3,"last update":"Mon Oct 16 12:28:12 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.59799999999999998,"15min":0.61699999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.80200000000000005,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.64000000000000001,"15min":0.63900000000000001},"min":{"1min":0.42999999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.82899999999999996,"5min":1.2509999999999999,"15min":1.2509999999999999},"last":0.57199999999999995}]},{"osd":4,"last update":"Mon Oct 16 12:28:12 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.55000000000000004,"15min":0.52700000000000002},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.89700000000000002,"5min":1.383,"15min":1.383},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.621,"5min":0.63100000000000001,"15min":0.60699999999999998},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.86699999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.59899999999999998}]},{"osd":5,"last update":"Mon Oct 16 12:28:12 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.57599999999999996,"15min":0.53900000000000003},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.74199999999999999,"5min":1.3109999999999999,"15min":1.3109999999999999},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.60399999999999998,"15min":0.58299999999999996},"min":{"1min":0.35399999999999998,"5min":0.33500000000000002,"15min":0.32400000000000001},"max":{"1min":0.84599999999999997,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":0.55500000000000005}]},{"osd":6,"last update":"Mon Oct 16 12:28:12 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.64500000000000002,"15min":0.62},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.80500000000000005,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.64800000000000002,"15min":0.626},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.83399999999999996,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.38200000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575069,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36864,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334976,"statfs":{"total":96636764160,"available":96599015424,"internally_reserved":0,"allocated":1560576,"data_stored":784583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.56799999999999995,"15min":0.61099999999999999},"min":{"1min":0.33800000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.80700000000000005,"5min":0.88200000000000001,"15min":1.196},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.58199999999999996,"15min":0.623},"min":{"1min":0.40799999999999997,"5min":0.38500000000000001,"15min":0.33600000000000002},"max":{"1min":0.78900000000000003,"5min":0.83399999999999996,"15min":0.997},"last":0.505}]},{"osd":1,"last update":"Mon Oct 16 12:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.53000000000000003,"15min":0.56699999999999995},"min":{"1min":0.40600000000000003,"5min":0.33100000000000002,"15min":0.29399999999999998},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.875},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.623,"5min":0.55500000000000005,"15min":0.58799999999999997},"min":{"1min":0.36699999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":1.091},"last":0.52700000000000002}]},{"osd":2,"last update":"Mon Oct 16 12:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.56499999999999995,"15min":0.59099999999999997},"min":{"1min":0.433,"5min":0.35599999999999998,"15min":0.34499999999999997},"max":{"1min":0.77200000000000002,"5min":0.81499999999999995,"15min":0.97099999999999997},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.55400000000000005,"15min":0.59999999999999998},"min":{"1min":0.41799999999999998,"5min":0.34999999999999998,"15min":0.27100000000000002},"max":{"1min":0.76500000000000001,"5min":0.85999999999999999,"15min":2.3879999999999999},"last":0.44700000000000001}]},{"osd":3,"last update":"Mon Oct 16 12:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.59499999999999997,"15min":0.625},"min":{"1min":0.38600000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.85499999999999998,"5min":0.88,"15min":0.93999999999999995},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.65800000000000003,"15min":0.68100000000000005},"min":{"1min":0.52700000000000002,"5min":0.41499999999999998,"15min":0.40899999999999997},"max":{"1min":0.83899999999999997,"5min":0.94499999999999995,"15min":1.3080000000000001},"last":0.58599999999999997}]},{"osd":4,"last update":"Mon Oct 16 12:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.53100000000000003,"15min":0.51900000000000002},"min":{"1min":0.35299999999999998,"5min":0.30599999999999999,"15min":0.29899999999999999},"max":{"1min":0.79100000000000004,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.623,"5min":0.59399999999999997,"15min":0.59999999999999998},"min":{"1min":0.34000000000000002,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.81599999999999995,"5min":0.82299999999999995,"15min":1.2250000000000001},"last":0.56699999999999995}]},{"osd":5,"last update":"Mon Oct 16 12:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.58899999999999997,"15min":0.57799999999999996},"min":{"1min":0.36899999999999999,"5min":0.30299999999999999,"15min":0.30099999999999999},"max":{"1min":0.86199999999999999,"5min":0.98799999999999999,"15min":1.0589999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.63300000000000001,"15min":0.63},"min":{"1min":0.45000000000000001,"5min":0.32100000000000001,"15min":0.311},"max":{"1min":0.88200000000000001,"5min":1.0029999999999999,"15min":1.3540000000000001},"last":0.63300000000000001}]},{"osd":7,"last update":"Mon Oct 16 12:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.57999999999999996,"15min":0.57699999999999996},"min":{"1min":0.42699999999999999,"5min":0.29699999999999999,"15min":0.26000000000000001},"max":{"1min":0.83299999999999996,"5min":0.92700000000000005,"15min":1.29},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.622,"15min":0.62},"min":{"1min":0.48399999999999999,"5min":0.32900000000000001,"15min":0.29799999999999999},"max":{"1min":0.89900000000000002,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.61399999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575069,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32708,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339132,"statfs":{"total":96636764160,"available":96603271168,"internally_reserved":0,"allocated":1560576,"data_stored":784583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":31914441},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:29:06 2023","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.375,"15min":0.38100000000000001},"min":{"1min":0.29499999999999998,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.66000000000000003},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.442,"15min":0.46100000000000002},"min":{"1min":0.31,"5min":0.28399999999999997,"15min":0.26700000000000002},"max":{"1min":0.66300000000000003,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.379}]},{"osd":2,"last update":"Mon Oct 16 12:29:06 2023","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.39700000000000002,"15min":0.42599999999999999},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.93000000000000005},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.45600000000000002,"15min":0.46700000000000003},"min":{"1min":0.34899999999999998,"5min":0.29099999999999998,"15min":0.28199999999999997},"max":{"1min":0.68300000000000005,"5min":0.69199999999999995,"15min":0.98099999999999998},"last":0.40000000000000002}]},{"osd":3,"last update":"Mon Oct 16 12:29:06 2023","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.38900000000000001,"15min":0.40300000000000002},"min":{"1min":0.308,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.60499999999999998,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.314},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.438,"15min":0.45600000000000002},"min":{"1min":0.36799999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.62,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.47899999999999998}]},{"osd":4,"last update":"Mon Oct 16 12:29:06 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.61099999999999999,"15min":0.63200000000000001},"min":{"1min":0.46600000000000003,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.878,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.432},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.623,"15min":0.64400000000000002},"min":{"1min":0.54500000000000004,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":1.089},"last":0.41599999999999998}]},{"osd":5,"last update":"Mon Oct 16 12:29:06 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.63900000000000001,"15min":0.66800000000000004},"min":{"1min":0.46800000000000003,"5min":0.42599999999999999,"15min":0.38300000000000001},"max":{"1min":1.0209999999999999,"5min":1.069,"15min":1.069},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.67900000000000005,"15min":0.68999999999999995},"min":{"1min":0.47499999999999998,"5min":0.439,"15min":0.32500000000000001},"max":{"1min":1.034,"5min":1.0620000000000001,"15min":1.0649999999999999},"last":0.53500000000000003}]},{"osd":6,"last update":"Mon Oct 16 12:29:06 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.63900000000000001,"15min":0.65800000000000003},"min":{"1min":0.48099999999999998,"5min":0.45600000000000002,"15min":0.378},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.64900000000000002,"15min":0.66500000000000004},"min":{"1min":0.46300000000000002,"5min":0.434,"15min":0.32900000000000001},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.048},"last":0.44800000000000001}]},{"osd":7,"last update":"Mon Oct 16 12:29:06 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.66200000000000003,"15min":0.68600000000000005},"min":{"1min":0.53500000000000003,"5min":0.40200000000000002,"15min":0.33700000000000002},"max":{"1min":0.90400000000000003,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":0.497},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.68300000000000005,"15min":0.70399999999999996},"min":{"1min":0.56999999999999995,"5min":0.45700000000000002,"15min":0.42099999999999999},"max":{"1min":1.006,"5min":1.006,"15min":1.167},"last":0.52300000000000002}]}]},{"osd":0,"up_from":13,"seq":55834575070,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37480,"kb_used_data":2132,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334360,"statfs":{"total":96636764160,"available":96598384640,"internally_reserved":0,"allocated":2183168,"data_stored":1383776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 12:28:58 2023","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.39700000000000002,"15min":0.39200000000000002},"min":{"1min":0.23899999999999999,"5min":0.219,"15min":0.219},"max":{"1min":0.59299999999999997,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.40500000000000003,"15min":0.41999999999999998},"min":{"1min":0.29399999999999998,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.66000000000000003,"5min":0.71799999999999997,"15min":0.73099999999999998},"last":0.58299999999999996}]},{"osd":2,"last update":"Mon Oct 16 12:28:58 2023","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.41899999999999998,"15min":0.42599999999999999},"min":{"1min":0.315,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.63600000000000001,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.44400000000000001,"15min":0.45400000000000001},"min":{"1min":0.32000000000000001,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.65500000000000003,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.54900000000000004}]},{"osd":3,"last update":"Mon Oct 16 12:28:58 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.42699999999999999,"15min":0.42599999999999999},"min":{"1min":0.34899999999999998,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.57499999999999996,"5min":0.62,"15min":0.79400000000000004},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.497,"5min":0.47899999999999998,"15min":0.48999999999999999},"min":{"1min":0.311,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.755,"5min":0.79200000000000004,"15min":0.878},"last":0.378}]},{"osd":4,"last update":"Mon Oct 16 12:28:58 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.63900000000000001,"15min":0.65200000000000002},"min":{"1min":0.378,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.0329999999999999,"5min":1.05,"15min":1.0960000000000001},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.63900000000000001,"15min":0.65800000000000003},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.34100000000000003},"max":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.1719999999999999},"last":0.432}]},{"osd":5,"last update":"Mon Oct 16 12:28:58 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.63900000000000001,"15min":0.65400000000000003},"min":{"1min":0.441,"5min":0.373,"15min":0.373},"max":{"1min":0.92200000000000004,"5min":0.97699999999999998,"15min":1.127},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.67400000000000004,"15min":0.68000000000000005},"min":{"1min":0.48699999999999999,"5min":0.39400000000000002,"15min":0.39000000000000001},"max":{"1min":0.875,"5min":1.0700000000000001,"15min":1.1160000000000001},"last":0.48999999999999999}]},{"osd":6,"last update":"Mon Oct 16 12:28:58 2023","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.67500000000000004,"15min":0.68400000000000005},"min":{"1min":0.51100000000000001,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":1.079},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.66700000000000004,"15min":0.68799999999999994},"min":{"1min":0.42799999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.95099999999999996,"5min":0.96399999999999997,"15min":1.1499999999999999},"last":0.752}]},{"osd":7,"last update":"Mon Oct 16 12:28:58 2023","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.69199999999999995,"15min":0.70499999999999996},"min":{"1min":0.45200000000000001,"5min":0.436,"15min":0.41499999999999998},"max":{"1min":0.98499999999999999,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.65800000000000003,"15min":0.68899999999999995},"min":{"1min":0.47699999999999998,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.91000000000000003,"5min":0.97499999999999998,"15min":1.0569999999999999},"last":0.47599999999999998}]}]},{"osd":2,"up_from":13,"seq":55834575070,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59096,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312744,"statfs":{"total":96636764160,"available":96576249856,"internally_reserved":0,"allocated":1576960,"data_stored":791740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:29:04 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.36899999999999999,"15min":0.38300000000000001},"min":{"1min":0.20799999999999999,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.52100000000000002,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.432,"15min":0.44700000000000001},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.53500000000000003,"5min":0.66500000000000004,"15min":0.82999999999999996},"last":0.30399999999999999}]},{"osd":1,"last update":"Mon Oct 16 12:29:04 2023","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.44500000000000001,"15min":0.46600000000000003},"min":{"1min":0.29399999999999998,"5min":0.27300000000000002,"15min":0.25600000000000001},"max":{"1min":0.64000000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.40899999999999997,"15min":0.432},"min":{"1min":0.28699999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.57599999999999996,"5min":0.59199999999999997,"15min":0.69699999999999995},"last":0.224}]},{"osd":3,"last update":"Mon Oct 16 12:29:04 2023","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.39500000000000002,"15min":0.41099999999999998},"min":{"1min":0.26300000000000001,"5min":0.214,"15min":0.214},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.89300000000000002},"last":0.25},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.498,"15min":0.51500000000000001},"min":{"1min":0.33900000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.74399999999999999,"5min":0.78000000000000003,"15min":0.79600000000000004},"last":0.35099999999999998}]},{"osd":4,"last update":"Mon Oct 16 12:29:04 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.65600000000000003,"15min":0.67300000000000004},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.92300000000000004,"5min":1.3839999999999999,"15min":1.3839999999999999},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.64100000000000001,"15min":0.66900000000000004},"min":{"1min":0.46500000000000002,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.94399999999999995,"5min":0.97599999999999998,"15min":1.0109999999999999},"last":0.435}]},{"osd":5,"last update":"Mon Oct 16 12:29:04 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.63700000000000001,"15min":0.66000000000000003},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.312},"max":{"1min":1.0109999999999999,"5min":1.093,"15min":1.093},"last":0.497},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.63200000000000001,"15min":0.65400000000000003},"min":{"1min":0.39900000000000002,"5min":0.33100000000000002,"15min":0.317},"max":{"1min":0.81899999999999995,"5min":1.331,"15min":1.331},"last":0.41999999999999998}]},{"osd":6,"last update":"Mon Oct 16 12:29:04 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.60499999999999998,"15min":0.628},"min":{"1min":0.436,"5min":0.34899999999999998,"15min":0.34699999999999998},"max":{"1min":0.82399999999999995,"5min":0.99299999999999999,"15min":1.0369999999999999},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.66600000000000004,"15min":0.68200000000000005},"min":{"1min":0.41499999999999998,"5min":0.40799999999999997,"15min":0.40000000000000002},"max":{"1min":0.96299999999999997,"5min":1.032,"15min":1.032},"last":0.47399999999999998}]},{"osd":7,"last update":"Mon Oct 16 12:29:04 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.65900000000000003,"15min":0.67000000000000004},"min":{"1min":0.438,"5min":0.40300000000000002,"15min":0.38400000000000001},"max":{"1min":0.86199999999999999,"5min":0.96399999999999997,"15min":1.016},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.66300000000000003,"15min":0.68200000000000005},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.0409999999999999,"5min":1.145,"15min":1.145},"last":0.44400000000000001}]}]},{"osd":3,"up_from":13,"seq":55834575069,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36888,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334952,"statfs":{"total":96636764160,"available":96598990848,"internally_reserved":0,"allocated":1576960,"data_stored":792288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:29:09 2023","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.39500000000000002,"15min":0.40000000000000002},"min":{"1min":0.25,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.628,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.434,"5min":0.39100000000000001,"15min":0.40600000000000003},"min":{"1min":0.28199999999999997,"5min":0.254,"15min":0.254},"max":{"1min":0.59499999999999997,"5min":0.61899999999999999,"15min":0.73099999999999998},"last":0.311}]},{"osd":1,"last update":"Mon Oct 16 12:29:09 2023","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.39600000000000002,"15min":0.41299999999999998},"min":{"1min":0.25700000000000001,"5min":0.245,"15min":0.24399999999999999},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70799999999999996},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.439,"15min":0.45900000000000002},"min":{"1min":0.28199999999999997,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.747},"last":0.41699999999999998}]},{"osd":2,"last update":"Mon Oct 16 12:29:09 2023","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.45600000000000002,"15min":0.47199999999999998},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29199999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.47899999999999998,"15min":0.498},"min":{"1min":0.35799999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.752},"last":0.36699999999999999}]},{"osd":4,"last update":"Mon Oct 16 12:29:09 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.61199999999999999,"15min":0.64300000000000002},"min":{"1min":0.378,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":1.1539999999999999},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.621,"15min":0.65800000000000003},"min":{"1min":0.39500000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.95399999999999996,"5min":0.96999999999999997,"15min":1.0409999999999999},"last":0.60699999999999998}]},{"osd":5,"last update":"Mon Oct 16 12:29:09 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.65300000000000002,"15min":0.68200000000000005},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.1080000000000001},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.63,"15min":0.66000000000000003},"min":{"1min":0.48299999999999998,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.96799999999999997,"5min":0.97299999999999998,"15min":1.022},"last":0.52900000000000003}]},{"osd":6,"last update":"Mon Oct 16 12:29:09 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.63500000000000001,"15min":0.66800000000000004},"min":{"1min":0.438,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.86599999999999999,"5min":0.94199999999999995,"15min":1.0760000000000001},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.67800000000000005,"15min":0.70499999999999996},"min":{"1min":0.49399999999999999,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.006,"5min":1.0349999999999999,"15min":1.0409999999999999},"last":0.629}]},{"osd":7,"last update":"Mon Oct 16 12:29:09 2023","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.65300000000000002,"15min":0.68400000000000005},"min":{"1min":0.52800000000000002,"5min":0.42399999999999999,"15min":0.38},"max":{"1min":1.0149999999999999,"5min":1.0189999999999999,"15min":1.2430000000000001},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.748,"5min":0.68600000000000005,"15min":0.70399999999999996},"min":{"1min":0.5,"5min":0.44,"15min":0.42699999999999999},"max":{"1min":1.002,"5min":1.03,"15min":1.1339999999999999},"last":0.51700000000000002}]}]},{"osd":4,"up_from":13,"seq":55834575068,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32720,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339120,"statfs":{"total":96636764160,"available":96603258880,"internally_reserved":0,"allocated":1581056,"data_stored":791798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.57099999999999995,"15min":0.58699999999999997},"min":{"1min":0.38900000000000001,"5min":0.30399999999999999,"15min":0.27900000000000003},"max":{"1min":0.78500000000000003,"5min":0.85099999999999998,"15min":0.94599999999999995},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.57299999999999995,"15min":0.59999999999999998},"min":{"1min":0.42699999999999999,"5min":0.29999999999999999,"15min":0.27600000000000002},"max":{"1min":0.76400000000000001,"5min":0.83199999999999996,"15min":0.88700000000000001},"last":0.66800000000000004}]},{"osd":1,"last update":"Mon Oct 16 12:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.58799999999999997,"15min":0.60999999999999999},"min":{"1min":0.39700000000000002,"5min":0.31900000000000001,"15min":0.28499999999999998},"max":{"1min":0.80200000000000005,"5min":0.82599999999999996,"15min":0.92600000000000005},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.61799999999999999},"min":{"1min":0.36899999999999999,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.78300000000000003,"5min":0.91700000000000004,"15min":0.98099999999999998},"last":0.69499999999999995}]},{"osd":2,"last update":"Mon Oct 16 12:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.59499999999999997,"15min":0.623},"min":{"1min":0.432,"5min":0.33800000000000002,"15min":0.32000000000000001},"max":{"1min":0.84299999999999997,"5min":0.85699999999999998,"15min":1.012},"last":0.505},{"interface":"front","average":{"1min":0.63,"5min":0.60599999999999998,"15min":0.625},"min":{"1min":0.42599999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.85599999999999998,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.65400000000000003}]},{"osd":3,"last update":"Mon Oct 16 12:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.57499999999999996,"15min":0.60499999999999998},"min":{"1min":0.46200000000000002,"5min":0.35599999999999998,"15min":0.33700000000000002},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.93200000000000005},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.63600000000000001,"15min":0.64400000000000002},"min":{"1min":0.46600000000000003,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.84999999999999998,"5min":0.90600000000000003,"15min":0.97199999999999998},"last":0.68100000000000005}]},{"osd":5,"last update":"Mon Oct 16 12:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.52000000000000002,"15min":0.503},"min":{"1min":0.35199999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.82599999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.52900000000000003,"15min":0.52500000000000002},"min":{"1min":0.34999999999999998,"5min":0.32100000000000001,"15min":0.308},"max":{"1min":0.86599999999999999,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.45800000000000002}]},{"osd":6,"last update":"Mon Oct 16 12:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.372,"5min":0.33800000000000002,"15min":0.32500000000000001},"max":{"1min":0.79800000000000004,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.64400000000000002,"15min":0.63600000000000001},"min":{"1min":0.46200000000000002,"5min":0.35499999999999998,"15min":0.32000000000000001},"max":{"1min":0.89300000000000002,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.40500000000000003}]},{"osd":7,"last update":"Mon Oct 16 12:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.58999999999999997,"15min":0.58199999999999996},"min":{"1min":0.41699999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.63500000000000001,"15min":0.629},"min":{"1min":0.434,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.90800000000000003,"5min":1.006,"15min":1.006},"last":0.437}]}]},{"osd":5,"up_from":13,"seq":55834575069,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54912,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316928,"statfs":{"total":96636764160,"available":96580534272,"internally_reserved":0,"allocated":1560576,"data_stored":784583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:29:03 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.58799999999999997},"min":{"1min":0.37,"5min":0.32500000000000001,"15min":0.28000000000000003},"max":{"1min":0.76200000000000001,"5min":0.84499999999999997,"15min":0.873},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.59899999999999998,"15min":0.61799999999999999},"min":{"1min":0.51800000000000002,"5min":0.313,"15min":0.27600000000000002},"max":{"1min":0.85799999999999998,"5min":0.85899999999999999,"15min":0.88800000000000001},"last":0.89600000000000002}]},{"osd":1,"last update":"Mon Oct 16 12:29:03 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.58699999999999997,"15min":0.61099999999999999},"min":{"1min":0.37,"5min":0.35999999999999999,"15min":0.28999999999999998},"max":{"1min":0.79600000000000004,"5min":0.89700000000000002,"15min":0.91300000000000003},"last":0.93799999999999994},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.59299999999999997,"15min":0.61799999999999999},"min":{"1min":0.40699999999999997,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.84399999999999997,"5min":0.85999999999999999,"15min":0.89000000000000001},"last":0.71999999999999997}]},{"osd":2,"last update":"Mon Oct 16 12:29:03 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.58799999999999997,"15min":0.621},"min":{"1min":0.371,"5min":0.36899999999999999,"15min":0.29099999999999998},"max":{"1min":0.876,"5min":0.876,"15min":0.94399999999999995},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.58799999999999997,"15min":0.60999999999999999},"min":{"1min":0.35399999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.77100000000000002,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.91400000000000003}]},{"osd":3,"last update":"Mon Oct 16 12:29:03 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.61599999999999999,"15min":0.63900000000000001},"min":{"1min":0.47199999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.97499999999999998},"last":0.755},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.627,"15min":0.64500000000000002},"min":{"1min":0.51500000000000001,"5min":0.40400000000000003,"15min":0.35999999999999999},"max":{"1min":0.83199999999999996,"5min":0.90800000000000003,"15min":0.93200000000000005},"last":0.64600000000000002}]},{"osd":4,"last update":"Mon Oct 16 12:29:03 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.499,"15min":0.50700000000000001},"min":{"1min":0.33000000000000002,"5min":0.28100000000000003,"15min":0.25800000000000001},"max":{"1min":0.81299999999999994,"5min":0.85699999999999998,"15min":0.86399999999999999},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.57799999999999996,"15min":0.57199999999999995},"min":{"1min":0.34999999999999998,"5min":0.33900000000000002,"15min":0.33500000000000002},"max":{"1min":0.86099999999999999,"5min":0.90200000000000002,"15min":1.347},"last":0.83999999999999997}]},{"osd":6,"last update":"Mon Oct 16 12:29:03 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.55800000000000005,"15min":0.55400000000000005},"min":{"1min":0.39400000000000002,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.86699999999999999,"5min":0.91800000000000004,"15min":0.93500000000000005},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.621,"5min":0.59599999999999997,"15min":0.59699999999999998},"min":{"1min":0.39200000000000002,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.88400000000000001,"5min":0.92600000000000005,"15min":0.92700000000000005},"last":0.51300000000000001}]},{"osd":7,"last update":"Mon Oct 16 12:29:03 2023","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.63200000000000001,"15min":0.621},"min":{"1min":0.41699999999999998,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.435,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.93899999999999995,"5min":0.95199999999999996,"15min":1.1180000000000001},"last":0.54100000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T12:29:20.944 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-16T12:29:20.944 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-16T12:29:21.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.109+0000 7f378bb9d700 1 Processor -- start 2023-10-16T12:29:21.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.109+0000 7f378bb9d700 1 -- start start 2023-10-16T12:29:21.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.110+0000 7f378bb9d700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f37840cf4c0 0x7f37840ca1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:21.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.110+0000 7f378bb9d700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f378415dda0 con 0x7f378406bad0 2023-10-16T12:29:21.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.110+0000 7f378bb9d700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f378415e080 con 0x7f3784159de0 2023-10-16T12:29:21.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.110+0000 7f378bb9d700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f378415ea20 con 0x7f37840cf4c0 2023-10-16T12:29:21.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.110+0000 7f378a13a700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f378406bad0 0x7f37840ca6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33116/0 (socket says 172.21.15.16:33116) 2023-10-16T12:29:21.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.110+0000 7f378a13a700 1 -- 172.21.15.16:0/988171582 learned_addr learned my addr 172.21.15.16:0/988171582 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:29:21.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.110+0000 7f3788937700 1 -- 172.21.15.16:0/988171582 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4086889572 0 0) 0x7f378415dda0 con 0x7f378406bad0 2023-10-16T12:29:21.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.110+0000 7f3789138700 1 --2- 172.21.15.16:0/988171582 >> v2:172.21.15.49:3300/0 conn(0x7f37840cf4c0 0x7f37840ca1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:21.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.110+0000 7f3788937700 1 -- 172.21.15.16:0/988171582 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37700035b0 con 0x7f378406bad0 2023-10-16T12:29:21.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.111+0000 7f3788937700 1 -- 172.21.15.16:0/988171582 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3873406720 0 0) 0x7f37700035b0 con 0x7f378406bad0 2023-10-16T12:29:21.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.111+0000 7f3788937700 1 -- 172.21.15.16:0/988171582 >> v2:172.21.15.49:3300/0 conn(0x7f37840cf4c0 msgr2=0x7f37840ca1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:29:21.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.111+0000 7f3788937700 1 --2- 172.21.15.16:0/988171582 >> v2:172.21.15.49:3300/0 conn(0x7f37840cf4c0 0x7f37840ca1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:21.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.111+0000 7f3788937700 1 -- 172.21.15.16:0/988171582 >> v1:172.21.15.49:6789/0 conn(0x7f3784159de0 legacy=0x7f378415a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:21.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.111+0000 7f3788937700 1 -- 172.21.15.16:0/988171582 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f378415e360 con 0x7f378406bad0 2023-10-16T12:29:21.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.111+0000 7f3788937700 1 -- 172.21.15.16:0/988171582 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f377c0019f0 con 0x7f378406bad0 2023-10-16T12:29:21.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.111+0000 7f3788937700 1 -- 172.21.15.16:0/988171582 >> v1:172.21.15.16:6789/0 conn(0x7f378406bad0 legacy=0x7f37840ca6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:21.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.111+0000 7f3788937700 1 --2- 172.21.15.16:0/988171582 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3770003a00 0x7f3770003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:21.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.111+0000 7f3788937700 1 --2- 172.21.15.16:0/988171582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3770004850 0x7f3770004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:21.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.111+0000 7f3788937700 1 --2- 172.21.15.16:0/988171582 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3770005210 0x7f377000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:21.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.111+0000 7f3788937700 1 -- 172.21.15.16:0/988171582 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f378415ea20 con 0x7f3770004850 2023-10-16T12:29:21.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.111+0000 7f3788937700 1 -- 172.21.15.16:0/988171582 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f377000c1b0 con 0x7f3770005210 2023-10-16T12:29:21.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.111+0000 7f3788937700 1 -- 172.21.15.16:0/988171582 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f377000d480 con 0x7f3770003a00 2023-10-16T12:29:21.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.111+0000 7f3789138700 1 --2- 172.21.15.16:0/988171582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3770004850 0x7f3770004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:21.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.111+0000 7f378a13a700 1 --2- 172.21.15.16:0/988171582 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3770005210 0x7f377000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:21.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.111+0000 7f3789939700 1 --2- 172.21.15.16:0/988171582 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3770003a00 0x7f3770003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:21.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.111+0000 7f3789138700 1 -- 172.21.15.16:0/988171582 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3770003a00 msgr2=0x7f3770003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:29:21.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.111+0000 7f3789138700 1 --2- 172.21.15.16:0/988171582 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3770003a00 0x7f3770003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:21.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.111+0000 7f3789138700 1 -- 172.21.15.16:0/988171582 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3770005210 msgr2=0x7f377000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:29:21.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.111+0000 7f3789138700 1 --2- 172.21.15.16:0/988171582 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3770005210 0x7f377000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:21.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.111+0000 7f3789138700 1 -- 172.21.15.16:0/988171582 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37700035b0 con 0x7f3770004850 2023-10-16T12:29:21.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.111+0000 7f378a13a700 1 -- 172.21.15.16:0/988171582 reap_dead start 2023-10-16T12:29:21.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.111+0000 7f3789138700 1 --2- 172.21.15.16:0/988171582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3770004850 0x7f3770004cd0 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f3780009b30 tx=0x7f3770005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e75137376cbf1bf4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:21.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.111+0000 7f3788937700 1 -- 172.21.15.16:0/988171582 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f378000e6c0 con 0x7f3770004850 2023-10-16T12:29:21.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.111+0000 7f3788937700 1 -- 172.21.15.16:0/988171582 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f378000e820 con 0x7f3770004850 2023-10-16T12:29:21.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.111+0000 7f3788937700 1 -- 172.21.15.16:0/988171582 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f378000eb90 con 0x7f3770004850 2023-10-16T12:29:21.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.112+0000 7f378bb9d700 1 -- 172.21.15.16:0/988171582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3770004850 msgr2=0x7f3770004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:21.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.112+0000 7f378bb9d700 1 --2- 172.21.15.16:0/988171582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3770004850 0x7f3770004cd0 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f3780009b30 tx=0x7f3770005880 comp rx=0 tx=0).stop 2023-10-16T12:29:21.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.112+0000 7f378bb9d700 1 -- 172.21.15.16:0/988171582 shutdown_connections 2023-10-16T12:29:21.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.112+0000 7f378bb9d700 1 --2- 172.21.15.16:0/988171582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3770004850 0x7f3770004cd0 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:21.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.112+0000 7f378bb9d700 1 -- 172.21.15.16:0/988171582 >> 172.21.15.16:0/988171582 conn(0x7f37840cc5e0 msgr2=0x7f37840cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:21.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.112+0000 7f378bb9d700 1 -- 172.21.15.16:0/988171582 shutdown_connections 2023-10-16T12:29:21.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.112+0000 7f378bb9d700 1 -- 172.21.15.16:0/988171582 wait complete. 2023-10-16T12:29:21.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.113+0000 7f378bb9d700 1 Processor -- start 2023-10-16T12:29:21.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.113+0000 7f378bb9d700 1 -- start start 2023-10-16T12:29:21.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.114+0000 7f378bb9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3770004850 0x7f37841f4040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:21.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.114+0000 7f378bb9d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f37840b4920 0x7f37840c5ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:21.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.114+0000 7f378bb9d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f378414f0f0 0x7f378406f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:21.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.114+0000 7f378bb9d700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f378414eaf0 con 0x7f3770004850 2023-10-16T12:29:21.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.114+0000 7f378bb9d700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37841f4580 con 0x7f37840b4920 2023-10-16T12:29:21.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.114+0000 7f378bb9d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37840b4e30 con 0x7f378414f0f0 2023-10-16T12:29:21.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.114+0000 7f3789939700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3770004850 0x7f37841f4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:21.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.114+0000 7f3789939700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3770004850 0x7f37841f4040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43850/0 (socket says 172.21.15.16:43850) 2023-10-16T12:29:21.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.114+0000 7f3789939700 1 -- 172.21.15.16:0/1081645580 learned_addr learned my addr 172.21.15.16:0/1081645580 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:29:21.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.114+0000 7f3789138700 1 --2- 172.21.15.16:0/1081645580 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f37840b4920 0x7f37840c5ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:21.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.114+0000 7f378a13a700 1 --2- 172.21.15.16:0/1081645580 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f378414f0f0 0x7f378406f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:21.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.115+0000 7f3789939700 1 -- 172.21.15.16:0/1081645580 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f378414f0f0 msgr2=0x7f378406f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:21.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.115+0000 7f3789939700 1 --2- 172.21.15.16:0/1081645580 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f378414f0f0 0x7f378406f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:21.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.115+0000 7f3789939700 1 -- 172.21.15.16:0/1081645580 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f37840b4920 msgr2=0x7f37840c5ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:21.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.115+0000 7f3789939700 1 --2- 172.21.15.16:0/1081645580 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f37840b4920 0x7f37840c5ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:21.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.115+0000 7f3789939700 1 -- 172.21.15.16:0/1081645580 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37840b5110 con 0x7f3770004850 2023-10-16T12:29:21.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.115+0000 7f378a13a700 1 --2- 172.21.15.16:0/1081645580 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f378414f0f0 0x7f378406f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:29:21.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.115+0000 7f3789939700 1 --2- 172.21.15.16:0/1081645580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3770004850 0x7f37841f4040 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f377400e7b0 tx=0x7f377400eac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:21.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.115+0000 7f377a7fc700 1 -- 172.21.15.16:0/1081645580 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37740052e0 con 0x7f3770004850 2023-10-16T12:29:21.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.115+0000 7f377a7fc700 1 -- 172.21.15.16:0/1081645580 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3774005440 con 0x7f3770004850 2023-10-16T12:29:21.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.115+0000 7f378bb9d700 1 -- 172.21.15.16:0/1081645580 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f378414b890 con 0x7f3770004850 2023-10-16T12:29:21.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.115+0000 7f377a7fc700 1 -- 172.21.15.16:0/1081645580 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37740038a0 con 0x7f3770004850 2023-10-16T12:29:21.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.115+0000 7f378bb9d700 1 -- 172.21.15.16:0/1081645580 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37840b53f0 con 0x7f3770004850 2023-10-16T12:29:21.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.116+0000 7f378bb9d700 1 -- 172.21.15.16:0/1081645580 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f378404ebb0 con 0x7f3770004850 2023-10-16T12:29:21.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.116+0000 7f377a7fc700 1 -- 172.21.15.16:0/1081645580 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 62) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3774003a00 con 0x7f3770004850 2023-10-16T12:29:21.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.117+0000 7f377a7fc700 1 --2- 172.21.15.16:0/1081645580 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f376806b020 0x7f376806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:21.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.117+0000 7f377a7fc700 1 -- 172.21.15.16:0/1081645580 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f3774088f40 con 0x7f3770004850 2023-10-16T12:29:21.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.117+0000 7f3789138700 1 --2- 172.21.15.16:0/1081645580 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f376806b020 0x7f376806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:21.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.117+0000 7f3789138700 1 --2- 172.21.15.16:0/1081645580 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f376806b020 0x7f376806d4e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f37800090b0 tx=0x7f378001e040 comp rx=0 tx=0).ready entity=mgr.7884 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:21.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.121+0000 7f377a7fc700 1 -- 172.21.15.16:0/1081645580 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3774053e40 con 0x7f3770004850 2023-10-16T12:29:21.165 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:29:21.168 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:29:21.361 INFO:tasks.ceph.mds.a.smithi016.stderr:2023-10-16T12:29:21.359+0000 7fd6de734700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T12:29:21.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.418+0000 7f378bb9d700 1 -- 172.21.15.16:0/1081645580 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f37840ce3d0 con 0x7f3770004850 2023-10-16T12:29:21.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.421+0000 7f377a7fc700 1 -- 172.21.15.16:0/1081645580 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v23) v1 ==== 190+0+147 (secure 0 0 0) 0x7f3774003cb0 con 0x7f3770004850 2023-10-16T12:29:21.423 INFO:teuthology.orchestra.run.smithi016.stdout:[client.0] 2023-10-16T12:29:21.423 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAPKC1l2l6PFxAA9QyXtBAHdGAXpwH0JfknuQ== 2023-10-16T12:29:21.423 INFO:teuthology.orchestra.run.smithi016.stdout: caps mds = "allow" 2023-10-16T12:29:21.423 INFO:teuthology.orchestra.run.smithi016.stdout: caps mon = "allow r" 2023-10-16T12:29:21.424 INFO:teuthology.orchestra.run.smithi016.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-16T12:29:21.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.423+0000 7f378bb9d700 1 -- 172.21.15.16:0/1081645580 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f376806b020 msgr2=0x7f376806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:21.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.423+0000 7f378bb9d700 1 --2- 172.21.15.16:0/1081645580 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f376806b020 0x7f376806d4e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f37800090b0 tx=0x7f378001e040 comp rx=0 tx=0).stop 2023-10-16T12:29:21.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.423+0000 7f378bb9d700 1 -- 172.21.15.16:0/1081645580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3770004850 msgr2=0x7f37841f4040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:21.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.423+0000 7f378bb9d700 1 --2- 172.21.15.16:0/1081645580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3770004850 0x7f37841f4040 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f377400e7b0 tx=0x7f377400eac0 comp rx=0 tx=0).stop 2023-10-16T12:29:21.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.423+0000 7f378bb9d700 1 -- 172.21.15.16:0/1081645580 shutdown_connections 2023-10-16T12:29:21.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.423+0000 7f378bb9d700 1 --2- 172.21.15.16:0/1081645580 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f376806b020 0x7f376806d4e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:21.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.423+0000 7f378bb9d700 1 --2- 172.21.15.16:0/1081645580 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f378414f0f0 0x7f378406f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:21.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.423+0000 7f378bb9d700 1 --2- 172.21.15.16:0/1081645580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3770004850 0x7f37841f4040 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:21.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.423+0000 7f378bb9d700 1 --2- 172.21.15.16:0/1081645580 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f37840b4920 0x7f37840c5ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:21.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.423+0000 7f378bb9d700 1 -- 172.21.15.16:0/1081645580 >> 172.21.15.16:0/1081645580 conn(0x7f37840cc5e0 msgr2=0x7f37840afe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:21.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.423+0000 7f378bb9d700 1 -- 172.21.15.16:0/1081645580 shutdown_connections 2023-10-16T12:29:21.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.423+0000 7f378bb9d700 1 -- 172.21.15.16:0/1081645580 wait complete. 2023-10-16T12:29:21.428 INFO:teuthology.orchestra.run.smithi016.stderr:updated caps for client.0 2023-10-16T12:29:21.437 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:29:21.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.562+0000 7f5de9991700 1 Processor -- start 2023-10-16T12:29:21.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.562+0000 7f5de9991700 1 -- start start 2023-10-16T12:29:21.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.563+0000 7f5de9991700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5de415d8d0 0x7f5de40d1c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:21.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.563+0000 7f5de9991700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f5de40d41c0 0x7f5de40d0040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:21.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.563+0000 7f5de9991700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5de406ba00 con 0x7f5de40d0580 2023-10-16T12:29:21.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.563+0000 7f5de9991700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5de40d0a10 con 0x7f5de415d8d0 2023-10-16T12:29:21.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.563+0000 7f5de9991700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5de40d2150 con 0x7f5de40d41c0 2023-10-16T12:29:21.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.563+0000 7f5de2ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5de415d8d0 0x7f5de40d1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:21.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.563+0000 7f5de2ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5de415d8d0 0x7f5de40d1c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43858/0 (socket says 172.21.15.16:43858) 2023-10-16T12:29:21.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.563+0000 7f5de37fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f5de40d0580 0x7f5de40ce240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33136/0 (socket says 172.21.15.16:33136) 2023-10-16T12:29:21.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.563+0000 7f5de2ffd700 1 -- 172.21.15.16:0/723355209 learned_addr learned my addr 172.21.15.16:0/723355209 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:29:21.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.564+0000 7f5de27fc700 1 --2- 172.21.15.16:0/723355209 >> v2:172.21.15.49:3300/0 conn(0x7f5de40d41c0 0x7f5de40d0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:21.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.564+0000 7f5de2ffd700 1 -- 172.21.15.16:0/723355209 >> v2:172.21.15.49:3300/0 conn(0x7f5de40d41c0 msgr2=0x7f5de40d0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:29:21.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.564+0000 7f5de2ffd700 1 --2- 172.21.15.16:0/723355209 >> v2:172.21.15.49:3300/0 conn(0x7f5de40d41c0 0x7f5de40d0040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:21.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.564+0000 7f5de2ffd700 1 -- 172.21.15.16:0/723355209 >> v1:172.21.15.16:6789/0 conn(0x7f5de40d0580 legacy=0x7f5de40ce240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:21.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.564+0000 7f5de2ffd700 1 -- 172.21.15.16:0/723355209 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5de40d26d0 con 0x7f5de415d8d0 2023-10-16T12:29:21.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.564+0000 7f5de1ffb700 1 -- 172.21.15.16:0/723355209 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2841029240 0 0) 0x7f5de406ba00 con 0x7f5de40d0580 2023-10-16T12:29:21.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.564+0000 7f5de2ffd700 1 --2- 172.21.15.16:0/723355209 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5de415d8d0 0x7f5de40d1c10 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f5de408da20 tx=0x7f5dd4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44e2d47c7c4e6ebf server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:21.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.564+0000 7f5de1ffb700 1 -- 172.21.15.16:0/723355209 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dd400b870 con 0x7f5de415d8d0 2023-10-16T12:29:21.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.564+0000 7f5de1ffb700 1 -- 172.21.15.16:0/723355209 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dd400b9d0 con 0x7f5de415d8d0 2023-10-16T12:29:21.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.564+0000 7f5de1ffb700 1 -- 172.21.15.16:0/723355209 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dd400bd60 con 0x7f5de415d8d0 2023-10-16T12:29:21.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.565+0000 7f5de9991700 1 -- 172.21.15.16:0/723355209 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5de415d8d0 msgr2=0x7f5de40d1c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:21.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.565+0000 7f5de9991700 1 --2- 172.21.15.16:0/723355209 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5de415d8d0 0x7f5de40d1c10 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f5de408da20 tx=0x7f5dd4009bd0 comp rx=0 tx=0).stop 2023-10-16T12:29:21.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.565+0000 7f5de9991700 1 -- 172.21.15.16:0/723355209 shutdown_connections 2023-10-16T12:29:21.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.565+0000 7f5de9991700 1 --2- 172.21.15.16:0/723355209 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5de415d8d0 0x7f5de40d1c10 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:21.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.565+0000 7f5de9991700 1 --2- 172.21.15.16:0/723355209 >> v2:172.21.15.49:3300/0 conn(0x7f5de40d41c0 0x7f5de40d0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:21.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.565+0000 7f5de9991700 1 -- 172.21.15.16:0/723355209 >> 172.21.15.16:0/723355209 conn(0x7f5de40d3630 msgr2=0x7f5de40d3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:21.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.565+0000 7f5de9991700 1 -- 172.21.15.16:0/723355209 shutdown_connections 2023-10-16T12:29:21.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.565+0000 7f5de9991700 1 -- 172.21.15.16:0/723355209 wait complete. 2023-10-16T12:29:21.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.566+0000 7f5de9991700 1 Processor -- start 2023-10-16T12:29:21.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.566+0000 7f5de9991700 1 -- start start 2023-10-16T12:29:21.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.567+0000 7f5de9991700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5de40d0580 0x7f5de40ccc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:21.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.567+0000 7f5de9991700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5de40d41c0 0x7f5de4155df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:21.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.567+0000 7f5de9991700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5de415d8d0 0x7f5de41e7410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:21.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.567+0000 7f5de9991700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5de41f3e20 con 0x7f5de40d0580 2023-10-16T12:29:21.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.567+0000 7f5de9991700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5de40cbea0 con 0x7f5de40d41c0 2023-10-16T12:29:21.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.567+0000 7f5de9991700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5de40cc180 con 0x7f5de415d8d0 2023-10-16T12:29:21.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.567+0000 7f5de2ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5de40d0580 0x7f5de40ccc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:21.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.567+0000 7f5de2ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5de40d0580 0x7f5de40ccc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43864/0 (socket says 172.21.15.16:43864) 2023-10-16T12:29:21.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.567+0000 7f5de2ffd700 1 -- 172.21.15.16:0/4155448314 learned_addr learned my addr 172.21.15.16:0/4155448314 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:29:21.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.567+0000 7f5de27fc700 1 --2- 172.21.15.16:0/4155448314 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5de40d41c0 0x7f5de4155df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:21.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.567+0000 7f5de37fe700 1 --2- 172.21.15.16:0/4155448314 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5de415d8d0 0x7f5de41e7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:21.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.567+0000 7f5de2ffd700 1 -- 172.21.15.16:0/4155448314 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5de415d8d0 msgr2=0x7f5de41e7410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:21.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.567+0000 7f5de2ffd700 1 --2- 172.21.15.16:0/4155448314 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5de415d8d0 0x7f5de41e7410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:21.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.567+0000 7f5de2ffd700 1 -- 172.21.15.16:0/4155448314 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5de40d41c0 msgr2=0x7f5de4155df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:21.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.567+0000 7f5de2ffd700 1 --2- 172.21.15.16:0/4155448314 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5de40d41c0 0x7f5de4155df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:21.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.567+0000 7f5de2ffd700 1 -- 172.21.15.16:0/4155448314 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dd4009550 con 0x7f5de40d0580 2023-10-16T12:29:21.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.568+0000 7f5de2ffd700 1 --2- 172.21.15.16:0/4155448314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5de40d0580 0x7f5de40ccc60 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f5dd4014040 tx=0x7f5dd4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:21.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.568+0000 7f5ddbfff700 1 -- 172.21.15.16:0/4155448314 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dd4004220 con 0x7f5de40d0580 2023-10-16T12:29:21.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.568+0000 7f5ddbfff700 1 -- 172.21.15.16:0/4155448314 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dd401e070 con 0x7f5de40d0580 2023-10-16T12:29:21.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.568+0000 7f5de9991700 1 -- 172.21.15.16:0/4155448314 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5de40cc460 con 0x7f5de40d0580 2023-10-16T12:29:21.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.568+0000 7f5ddbfff700 1 -- 172.21.15.16:0/4155448314 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dd4004580 con 0x7f5de40d0580 2023-10-16T12:29:21.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.568+0000 7f5de9991700 1 -- 172.21.15.16:0/4155448314 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5de40cd1a0 con 0x7f5de40d0580 2023-10-16T12:29:21.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.569+0000 7f5de9991700 1 -- 172.21.15.16:0/4155448314 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5de404ebb0 con 0x7f5de40d0580 2023-10-16T12:29:21.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.569+0000 7f5ddbfff700 1 -- 172.21.15.16:0/4155448314 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 62) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5dd40036a0 con 0x7f5de40d0580 2023-10-16T12:29:21.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.570+0000 7f5ddbfff700 1 --2- 172.21.15.16:0/4155448314 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f5dc406b020 0x7f5dc406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:21.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.570+0000 7f5ddbfff700 1 -- 172.21.15.16:0/4155448314 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f5dd4020070 con 0x7f5de40d0580 2023-10-16T12:29:21.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.570+0000 7f5de27fc700 1 --2- 172.21.15.16:0/4155448314 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f5dc406b020 0x7f5dc406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:21.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.570+0000 7f5de27fc700 1 --2- 172.21.15.16:0/4155448314 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f5dc406b020 0x7f5dc406d4e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5dcc00ab60 tx=0x7f5dcc008040 comp rx=0 tx=0).ready entity=mgr.7884 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:21.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.574+0000 7f5ddbfff700 1 -- 172.21.15.16:0/4155448314 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5dd4055820 con 0x7f5de40d0580 2023-10-16T12:29:21.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.822+0000 7f5de9991700 1 -- 172.21.15.16:0/4155448314 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5de406d1c0 con 0x7f5de40d0580 2023-10-16T12:29:21.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.823+0000 7f5ddbfff700 1 -- 172.21.15.16:0/4155448314 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+4909 (secure 0 0 0) 0x7f5dd4058e40 con 0x7f5de40d0580 2023-10-16T12:29:21.825 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:29:21.825 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":61,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6893,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/2971585675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":2971585675},{"type":"v1","addr":"172.21.15.49:6835","nonce":2971585675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":7723,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/3151669585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":3151669585},{"type":"v1","addr":"172.21.15.49:6834","nonce":3151669585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":8106,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/52924937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":52924937},{"type":"v1","addr":"172.21.15.16:6836","nonce":52924937}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:29:15.801502+0000","modified":"2023-10-16T12:29:18.806285+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8109},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8109":{"gid":8109,"name":"a","rank":0,"incarnation":57,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/3203451346","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":3203451346},{"type":"v1","addr":"172.21.15.16:6837","nonce":3203451346}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2023-10-16T12:29:21.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.825+0000 7f5de9991700 1 -- 172.21.15.16:0/4155448314 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f5dc406b020 msgr2=0x7f5dc406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:21.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.825+0000 7f5de9991700 1 --2- 172.21.15.16:0/4155448314 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f5dc406b020 0x7f5dc406d4e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5dcc00ab60 tx=0x7f5dcc008040 comp rx=0 tx=0).stop 2023-10-16T12:29:21.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.825+0000 7f5de9991700 1 -- 172.21.15.16:0/4155448314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5de40d0580 msgr2=0x7f5de40ccc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:21.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.825+0000 7f5de9991700 1 --2- 172.21.15.16:0/4155448314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5de40d0580 0x7f5de40ccc60 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f5dd4014040 tx=0x7f5dd4004750 comp rx=0 tx=0).stop 2023-10-16T12:29:21.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.825+0000 7f5de9991700 1 -- 172.21.15.16:0/4155448314 shutdown_connections 2023-10-16T12:29:21.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.825+0000 7f5de9991700 1 --2- 172.21.15.16:0/4155448314 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f5dc406b020 0x7f5dc406d4e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:21.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.825+0000 7f5de9991700 1 --2- 172.21.15.16:0/4155448314 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5de415d8d0 0x7f5de41e7410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:21.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.825+0000 7f5de9991700 1 --2- 172.21.15.16:0/4155448314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5de40d0580 0x7f5de40ccc60 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:21.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.825+0000 7f5de9991700 1 --2- 172.21.15.16:0/4155448314 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5de40d41c0 0x7f5de4155df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:21.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.825+0000 7f5de9991700 1 -- 172.21.15.16:0/4155448314 >> 172.21.15.16:0/4155448314 conn(0x7f5de40d3630 msgr2=0x7f5de415ca50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:21.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.825+0000 7f5de9991700 1 -- 172.21.15.16:0/4155448314 shutdown_connections 2023-10-16T12:29:21.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:21.825+0000 7f5de9991700 1 -- 172.21.15.16:0/4155448314 wait complete. 2023-10-16T12:29:21.830 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 61 2023-10-16T12:29:22.241 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 61, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T12:29:15.801502+0000', 'modified': '2023-10-16T12:29:18.806285+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8109}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8109': {'gid': 8109, 'name': 'a', 'rank': 0, 'incarnation': 57, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.16:6837/3203451346', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6835', 'nonce': 3203451346}, {'type': 'v1', 'addr': '172.21.15.16:6837', 'nonce': 3203451346}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [16], 'metadata_pool': 15, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-16T12:29:22.242 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-16T12:29:22.242 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-10-16T12:29:22.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.374+0000 7f62bb19d700 1 Processor -- start 2023-10-16T12:29:22.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.374+0000 7f62bb19d700 1 -- start start 2023-10-16T12:29:22.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.375+0000 7f62bb19d700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f62b40717e0 0x7f62b4071bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:22.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.375+0000 7f62bb19d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f62b406eba0 0x7f62b40cb820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:22.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.375+0000 7f62bb19d700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62b406fce0 con 0x7f62b415a580 2023-10-16T12:29:22.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.375+0000 7f62bb19d700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62b4072850 con 0x7f62b40717e0 2023-10-16T12:29:22.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.375+0000 7f62bb19d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62b4072100 con 0x7f62b406eba0 2023-10-16T12:29:22.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.376+0000 7f62b999a700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f62b415a580 0x7f62b415b320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33146/0 (socket says 172.21.15.16:33146) 2023-10-16T12:29:22.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.376+0000 7f62b999a700 1 -- 172.21.15.16:0/1130342127 learned_addr learned my addr 172.21.15.16:0/1130342127 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:29:22.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.376+0000 7f62ba19b700 1 --2- 172.21.15.16:0/1130342127 >> v2:172.21.15.49:3300/0 conn(0x7f62b40717e0 0x7f62b4071bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:22.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.376+0000 7f62b9199700 1 -- 172.21.15.16:0/1130342127 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3562768260 0 0) 0x7f62b406fce0 con 0x7f62b415a580 2023-10-16T12:29:22.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.376+0000 7f62ba99c700 1 --2- 172.21.15.16:0/1130342127 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f62b406eba0 0x7f62b40cb820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:22.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.376+0000 7f62b9199700 1 -- 172.21.15.16:0/1130342127 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62a00035b0 con 0x7f62b415a580 2023-10-16T12:29:22.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.376+0000 7f62b9199700 1 -- 172.21.15.16:0/1130342127 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1809578035 0 0) 0x7f62a00035b0 con 0x7f62b415a580 2023-10-16T12:29:22.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.376+0000 7f62b9199700 1 -- 172.21.15.16:0/1130342127 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f62b406eba0 msgr2=0x7f62b40cb820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:29:22.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.376+0000 7f62b9199700 1 --2- 172.21.15.16:0/1130342127 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f62b406eba0 0x7f62b40cb820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:22.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.376+0000 7f62b9199700 1 -- 172.21.15.16:0/1130342127 >> v2:172.21.15.49:3300/0 conn(0x7f62b40717e0 msgr2=0x7f62b4071bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:29:22.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.376+0000 7f62b9199700 1 --2- 172.21.15.16:0/1130342127 >> v2:172.21.15.49:3300/0 conn(0x7f62b40717e0 0x7f62b4071bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:22.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.376+0000 7f62b9199700 1 -- 172.21.15.16:0/1130342127 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62b406f0d0 con 0x7f62b415a580 2023-10-16T12:29:22.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.376+0000 7f62b9199700 1 -- 172.21.15.16:0/1130342127 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f62b0001670 con 0x7f62b415a580 2023-10-16T12:29:22.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.376+0000 7f62b9199700 1 -- 172.21.15.16:0/1130342127 >> v1:172.21.15.16:6789/0 conn(0x7f62b415a580 legacy=0x7f62b415b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:22.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.376+0000 7f62b9199700 1 --2- 172.21.15.16:0/1130342127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62a0003a00 0x7f62a0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:22.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.376+0000 7f62b9199700 1 --2- 172.21.15.16:0/1130342127 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f62b406eba0 0x7f62a0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:22.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.376+0000 7f62b9199700 1 --2- 172.21.15.16:0/1130342127 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f62a0004d90 0x7f62a0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:22.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.376+0000 7f62b9199700 1 -- 172.21.15.16:0/1130342127 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62b4072850 con 0x7f62a0003a00 2023-10-16T12:29:22.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.376+0000 7f62b9199700 1 -- 172.21.15.16:0/1130342127 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62b4072100 con 0x7f62a0004d90 2023-10-16T12:29:22.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.376+0000 7f62b9199700 1 -- 172.21.15.16:0/1130342127 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62a000dc20 con 0x7f62b406eba0 2023-10-16T12:29:22.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.376+0000 7f62ba19b700 1 --2- 172.21.15.16:0/1130342127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62a0003a00 0x7f62a0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:22.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.376+0000 7f62b999a700 1 --2- 172.21.15.16:0/1130342127 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f62b406eba0 0x7f62a0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:22.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.376+0000 7f62ba99c700 1 --2- 172.21.15.16:0/1130342127 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f62a0004d90 0x7f62a0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:22.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.377+0000 7f62ba19b700 1 -- 172.21.15.16:0/1130342127 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f62b406eba0 msgr2=0x7f62a0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:29:22.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.377+0000 7f62ba19b700 1 --2- 172.21.15.16:0/1130342127 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f62b406eba0 0x7f62a0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:22.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.377+0000 7f62ba19b700 1 -- 172.21.15.16:0/1130342127 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f62a0004d90 msgr2=0x7f62a0005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:29:22.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.377+0000 7f62ba19b700 1 --2- 172.21.15.16:0/1130342127 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f62a0004d90 0x7f62a0005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:22.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.377+0000 7f62ba19b700 1 -- 172.21.15.16:0/1130342127 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62a00035b0 con 0x7f62a0003a00 2023-10-16T12:29:22.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.377+0000 7f62b999a700 1 --2- 172.21.15.16:0/1130342127 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f62b406eba0 0x7f62a0004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:29:22.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.377+0000 7f62ba19b700 1 --2- 172.21.15.16:0/1130342127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62a0003a00 0x7f62a0003e20 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f62a4009b60 tx=0x7f62a0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e159df0da1221bb1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:22.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.377+0000 7f62b9199700 1 -- 172.21.15.16:0/1130342127 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62a400e6c0 con 0x7f62a0003a00 2023-10-16T12:29:22.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.377+0000 7f62b9199700 1 -- 172.21.15.16:0/1130342127 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62a400e820 con 0x7f62a0003a00 2023-10-16T12:29:22.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.377+0000 7f62b9199700 1 -- 172.21.15.16:0/1130342127 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62a400eb90 con 0x7f62a0003a00 2023-10-16T12:29:22.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.377+0000 7f62bb19d700 1 -- 172.21.15.16:0/1130342127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62a0003a00 msgr2=0x7f62a0003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:22.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.377+0000 7f62bb19d700 1 --2- 172.21.15.16:0/1130342127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62a0003a00 0x7f62a0003e20 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f62a4009b60 tx=0x7f62a0005880 comp rx=0 tx=0).stop 2023-10-16T12:29:22.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.378+0000 7f62ba99c700 1 -- 172.21.15.16:0/1130342127 reap_dead start 2023-10-16T12:29:22.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.378+0000 7f62bb19d700 1 -- 172.21.15.16:0/1130342127 shutdown_connections 2023-10-16T12:29:22.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.378+0000 7f62bb19d700 1 -- 172.21.15.16:0/1130342127 >> 172.21.15.16:0/1130342127 conn(0x7f62b40ccc80 msgr2=0x7f62b40cd090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:22.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.378+0000 7f62bb19d700 1 -- 172.21.15.16:0/1130342127 shutdown_connections 2023-10-16T12:29:22.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.378+0000 7f62bb19d700 1 -- 172.21.15.16:0/1130342127 wait complete. 2023-10-16T12:29:22.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.379+0000 7f62bb19d700 1 Processor -- start 2023-10-16T12:29:22.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.379+0000 7f62bb19d700 1 -- start start 2023-10-16T12:29:22.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.380+0000 7f62bb19d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f62a0003a00 0x7f62b415ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:22.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.380+0000 7f62bb19d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62b40b0a30 0x7f62b40c9760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:22.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.380+0000 7f62bb19d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f62b40c7790 0x7f62b41564e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:22.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.380+0000 7f62bb19d700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62b415cd60 con 0x7f62b40b0a30 2023-10-16T12:29:22.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.380+0000 7f62bb19d700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62b40b0eb0 con 0x7f62a0003a00 2023-10-16T12:29:22.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.380+0000 7f62bb19d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62b415c150 con 0x7f62b40c7790 2023-10-16T12:29:22.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.380+0000 7f62b999a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62b40b0a30 0x7f62b40c9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:22.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.380+0000 7f62b999a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62b40b0a30 0x7f62b40c9760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43886/0 (socket says 172.21.15.16:43886) 2023-10-16T12:29:22.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.380+0000 7f62b999a700 1 -- 172.21.15.16:0/3331270105 learned_addr learned my addr 172.21.15.16:0/3331270105 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:29:22.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.380+0000 7f62ba19b700 1 --2- 172.21.15.16:0/3331270105 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f62a0003a00 0x7f62b415ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:22.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.380+0000 7f62ba99c700 1 --2- 172.21.15.16:0/3331270105 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f62b40c7790 0x7f62b41564e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:22.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.380+0000 7f62b999a700 1 -- 172.21.15.16:0/3331270105 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f62b40c7790 msgr2=0x7f62b41564e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:22.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.380+0000 7f62b999a700 1 --2- 172.21.15.16:0/3331270105 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f62b40c7790 0x7f62b41564e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:22.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.380+0000 7f62b999a700 1 -- 172.21.15.16:0/3331270105 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f62a0003a00 msgr2=0x7f62b415ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:22.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.380+0000 7f62b999a700 1 --2- 172.21.15.16:0/3331270105 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f62a0003a00 0x7f62b415ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:22.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.380+0000 7f62b999a700 1 -- 172.21.15.16:0/3331270105 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62b415c430 con 0x7f62b40b0a30 2023-10-16T12:29:22.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.380+0000 7f62b999a700 1 --2- 172.21.15.16:0/3331270105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62b40b0a30 0x7f62b40c9760 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f62b0000fb0 tx=0x7f62b000eb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:22.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.380+0000 7f62aaffd700 1 -- 172.21.15.16:0/3331270105 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b0012590 con 0x7f62b40b0a30 2023-10-16T12:29:22.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.380+0000 7f62aaffd700 1 -- 172.21.15.16:0/3331270105 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62b00126f0 con 0x7f62b40b0a30 2023-10-16T12:29:22.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.380+0000 7f62bb19d700 1 -- 172.21.15.16:0/3331270105 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62b415c710 con 0x7f62b40b0a30 2023-10-16T12:29:22.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.381+0000 7f62aaffd700 1 -- 172.21.15.16:0/3331270105 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b0012a80 con 0x7f62b40b0a30 2023-10-16T12:29:22.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.381+0000 7f62bb19d700 1 -- 172.21.15.16:0/3331270105 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62b40c9ca0 con 0x7f62b40b0a30 2023-10-16T12:29:22.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.382+0000 7f62aaffd700 1 -- 172.21.15.16:0/3331270105 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 62) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f62b0012be0 con 0x7f62b40b0a30 2023-10-16T12:29:22.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.382+0000 7f62aaffd700 1 --2- 172.21.15.16:0/3331270105 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f629806b020 0x7f629806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:22.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.383+0000 7f62aaffd700 1 -- 172.21.15.16:0/3331270105 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f62b0089610 con 0x7f62b40b0a30 2023-10-16T12:29:22.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.383+0000 7f62ba19b700 1 --2- 172.21.15.16:0/3331270105 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f629806b020 0x7f629806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:22.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.383+0000 7f62ba19b700 1 --2- 172.21.15.16:0/3331270105 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f629806b020 0x7f629806d4e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f62a40090b0 tx=0x7f62a4008030 comp rx=0 tx=0).ready entity=mgr.7884 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:22.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.384+0000 7f62c85e9b80 1 Processor -- start 2023-10-16T12:29:22.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.385+0000 7f62c85e9b80 1 -- start start 2023-10-16T12:29:22.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.385+0000 7f62c85e9b80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563f62331e20 0x563f62332200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:22.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.385+0000 7f62c85e9b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f623051d0 0x563f62308000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:22.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.385+0000 7f62c85e9b80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x563f623030b0 0x563f6232f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:22.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.385+0000 7f62c85e9b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f6227bc20 con 0x563f623051d0 2023-10-16T12:29:22.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.385+0000 7f62c85e9b80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f62260c10 con 0x563f62331e20 2023-10-16T12:29:22.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.385+0000 7f62c85e9b80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x563f6225e530 con 0x563f623030b0 2023-10-16T12:29:22.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.385+0000 7f62b999a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f623051d0 0x563f62308000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:22.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.386+0000 7f62b999a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f623051d0 0x563f62308000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43894/0 (socket says 172.21.15.16:43894) 2023-10-16T12:29:22.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.386+0000 7f62b999a700 1 -- 172.21.15.16:0/3328706594 learned_addr learned my addr 172.21.15.16:0/3328706594 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:29:22.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.386+0000 7f62ba99c700 1 --2- 172.21.15.16:0/3328706594 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x563f623030b0 0x563f6232f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:22.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.386+0000 7f62ba19b700 1 --2- 172.21.15.16:0/3328706594 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563f62331e20 0x563f62332200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:22.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.386+0000 7f62b999a700 1 -- 172.21.15.16:0/3328706594 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x563f623030b0 msgr2=0x563f6232f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:29:22.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.386+0000 7f62b999a700 1 --2- 172.21.15.16:0/3328706594 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x563f623030b0 0x563f6232f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:22.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.386+0000 7f62b999a700 1 -- 172.21.15.16:0/3328706594 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563f62331e20 msgr2=0x563f62332200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:29:22.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.386+0000 7f62b999a700 1 --2- 172.21.15.16:0/3328706594 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563f62331e20 0x563f62332200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:22.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.386+0000 7f62b999a700 1 -- 172.21.15.16:0/3328706594 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563f622173a0 con 0x563f623051d0 2023-10-16T12:29:22.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.386+0000 7f62b999a700 1 --2- 172.21.15.16:0/3328706594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f623051d0 0x563f62308000 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f62b00878c0 tx=0x7f62b0012f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f525c09b2e7d0d0c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:22.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.386+0000 7f6297fff700 1 -- 172.21.15.16:0/3328706594 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b0060410 con 0x563f623051d0 2023-10-16T12:29:22.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.386+0000 7f6297fff700 1 -- 172.21.15.16:0/3328706594 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62b00564b0 con 0x563f623051d0 2023-10-16T12:29:22.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.386+0000 7f6297fff700 1 -- 172.21.15.16:0/3328706594 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b00114e0 con 0x563f623051d0 2023-10-16T12:29:22.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.387+0000 7f62c85e9b80 1 -- 172.21.15.16:0/3328706594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f623051d0 msgr2=0x563f62308000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:22.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.387+0000 7f62c85e9b80 1 --2- 172.21.15.16:0/3328706594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f623051d0 0x563f62308000 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f62b00878c0 tx=0x7f62b0012f60 comp rx=0 tx=0).stop 2023-10-16T12:29:22.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.387+0000 7f62c85e9b80 1 -- 172.21.15.16:0/3328706594 shutdown_connections 2023-10-16T12:29:22.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.387+0000 7f62c85e9b80 1 --2- 172.21.15.16:0/3328706594 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x563f623030b0 0x563f6232f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:22.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.387+0000 7f62c85e9b80 1 --2- 172.21.15.16:0/3328706594 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563f62331e20 0x563f62332200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:22.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.387+0000 7f62c85e9b80 1 --2- 172.21.15.16:0/3328706594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f623051d0 0x563f62308000 unknown :-1 s=CLOSED pgs=1394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:22.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.387+0000 7f62c85e9b80 1 -- 172.21.15.16:0/3328706594 >> 172.21.15.16:0/3328706594 conn(0x563f62234d80 msgr2=0x563f6234cc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:22.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.387+0000 7f62c85e9b80 1 -- 172.21.15.16:0/3328706594 shutdown_connections 2023-10-16T12:29:22.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.387+0000 7f62c85e9b80 1 -- 172.21.15.16:0/3328706594 wait complete. 2023-10-16T12:29:22.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.387+0000 7f62c85e9b80 1 -- start start 2023-10-16T12:29:22.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.388+0000 7f62c85e9b80 1 Processor -- start 2023-10-16T12:29:22.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.388+0000 7f62c85e9b80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x563f623030b0 0x563f6232c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:22.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.388+0000 7f62c85e9b80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563f623051d0 0x563f62332dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:22.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.388+0000 7f62c85e9b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f62331e20 0x563f62333310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:22.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.388+0000 7f62c85e9b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f62259e20 con 0x563f62331e20 2023-10-16T12:29:22.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.388+0000 7f62c85e9b80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f621f66c0 con 0x563f623051d0 2023-10-16T12:29:22.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.388+0000 7f62c85e9b80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x563f621f6360 con 0x563f623030b0 2023-10-16T12:29:22.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.389+0000 7f62c85e9b80 10 client.0 _ll_register_callbacks cb 0x563f620537b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T12:29:22.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.389+0000 7f62ba99c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f62331e20 0x563f62333310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:22.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.389+0000 7f62ba99c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f62331e20 0x563f62333310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43902/0 (socket says 172.21.15.16:43902) 2023-10-16T12:29:22.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.389+0000 7f62ba99c700 1 -- 172.21.15.16:0/253449948 learned_addr learned my addr 172.21.15.16:0/253449948 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:29:22.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.389+0000 7f62ba19b700 1 --2- 172.21.15.16:0/253449948 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x563f623030b0 0x563f6232c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:22.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.389+0000 7f62b999a700 1 --2- 172.21.15.16:0/253449948 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563f623051d0 0x563f62332dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:22.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.389+0000 7f62ba99c700 1 -- 172.21.15.16:0/253449948 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x563f623030b0 msgr2=0x563f6232c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:29:22.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.389+0000 7f62ba99c700 1 --2- 172.21.15.16:0/253449948 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x563f623030b0 0x563f6232c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:22.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.389+0000 7f62ba99c700 1 -- 172.21.15.16:0/253449948 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563f623051d0 msgr2=0x563f62332dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:29:22.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.389+0000 7f62ba99c700 1 --2- 172.21.15.16:0/253449948 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563f623051d0 0x563f62332dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:22.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.389+0000 7f62ba99c700 1 -- 172.21.15.16:0/253449948 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62b40c9ca0 con 0x563f62331e20 2023-10-16T12:29:22.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.389+0000 7f62ba99c700 1 --2- 172.21.15.16:0/253449948 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f62331e20 0x563f62333310 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f62b406d3a0 tx=0x7f62ac004550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c174c563acc65030 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:22.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.389+0000 7f62957fa700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2023-10-16T12:29:22.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.389+0000 7f62957fa700 1 -- 172.21.15.16:0/253449948 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62ac00e040 con 0x563f62331e20 2023-10-16T12:29:22.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.389+0000 7f62957fa700 1 -- 172.21.15.16:0/253449948 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(128..128 src has 1..128) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f62ac016870 con 0x563f62331e20 2023-10-16T12:29:22.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.389+0000 7f62c85e9b80 1 -- 172.21.15.16:0/253449948 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563f622fa5b0 con 0x563f62331e20 2023-10-16T12:29:22.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.389+0000 7f62957fa700 1 -- 172.21.15.16:0/253449948 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=61) v2 ==== 24+0+0 (secure 0 0 0) 0x7f62ac009be0 con 0x563f62331e20 2023-10-16T12:29:22.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.389+0000 7f62c85e9b80 10 client.8259 fetch_fsmap learned FSMap version 61 2023-10-16T12:29:22.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.390+0000 7f62c85e9b80 1 -- 172.21.15.16:0/253449948 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=61,monmap=2+}) v3 -- 0x563f62084d90 con 0x563f62331e20 2023-10-16T12:29:22.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.390+0000 7f62957fa700 1 -- 172.21.15.16:0/253449948 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x563f622fa5b0 con 0x563f62331e20 2023-10-16T12:29:22.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.390+0000 7f62957fa700 1 -- 172.21.15.16:0/253449948 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 61) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f62ac014050 con 0x563f62331e20 2023-10-16T12:29:22.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.390+0000 7f62c85e9b80 10 client.8259 fetch_fsmap finished waiting for FSMap version 61 2023-10-16T12:29:22.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.390+0000 7f62c85e9b80 10 client.8259 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-16T12:29:22.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.390+0000 7f62c85e9b80 20 client.8259 populate_metadata read hostname 'smithi016' 2023-10-16T12:29:22.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.390+0000 7f62c85e9b80 1 --2- 172.21.15.16:0/253449948 >> [v2:172.21.15.16:6835/3203451346,v1:172.21.15.16:6837/3203451346] conn(0x563f6232f0f0 0x563f6232f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:22.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.390+0000 7f62c85e9b80 4 client.8259 mds_command: new command op to 8109 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T12:29:22.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.390+0000 7f62c85e9b80 1 -- 172.21.15.16:0/253449948 --> [v2:172.21.15.16:6835/3203451346,v1:172.21.15.16:6837/3203451346] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563f62252e00 con 0x563f6232f0f0 2023-10-16T12:29:22.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.390+0000 7f62ba19b700 1 --2- 172.21.15.16:0/253449948 >> [v2:172.21.15.16:6835/3203451346,v1:172.21.15.16:6837/3203451346] conn(0x563f6232f0f0 0x563f6232f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:22.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.390+0000 7f62ba19b700 1 --2- 172.21.15.16:0/253449948 >> [v2:172.21.15.16:6835/3203451346,v1:172.21.15.16:6837/3203451346] conn(0x563f6232f0f0 0x563f6232f4d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=95f62f03533ac6e1 server_cookie=d05f177de485aaa6 in_seq=0 out_seq=0 2023-10-16T12:29:22.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.390+0000 7f62957fa700 10 client.8259 ms_handle_connect on v2:172.21.15.16:6835/3203451346 2023-10-16T12:29:22.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.391+0000 7f62957fa700 1 -- 172.21.15.16:0/253449948 <== mds.0 v2:172.21.15.16:6835/3203451346 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x563f62252e00 con 0x563f6232f0f0 2023-10-16T12:29:22.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.391+0000 7f62957fa700 10 client.8259 handle_command_reply: tid=0 2023-10-16T12:29:22.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.391+0000 7f62c85e9b80 1 client.8259 shutdown 2023-10-16T12:29:22.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.391+0000 7f62c85e9b80 1 -- 172.21.15.16:0/253449948 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f62331e20 msgr2=0x563f62333310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:22.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.391+0000 7f62c85e9b80 1 --2- 172.21.15.16:0/253449948 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f62331e20 0x563f62333310 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f62b406d3a0 tx=0x7f62ac004550 comp rx=0 tx=0).stop 2023-10-16T12:29:22.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.391+0000 7f62c85e9b80 1 -- 172.21.15.16:0/253449948 shutdown_connections 2023-10-16T12:29:22.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.391+0000 7f62c85e9b80 1 --2- 172.21.15.16:0/253449948 >> [v2:172.21.15.16:6835/3203451346,v1:172.21.15.16:6837/3203451346] conn(0x563f6232f0f0 0x563f6232f4d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:22.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.391+0000 7f62c85e9b80 1 --2- 172.21.15.16:0/253449948 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f62331e20 0x563f62333310 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:22.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.391+0000 7f62c85e9b80 1 --2- 172.21.15.16:0/253449948 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x563f623030b0 0x563f6232c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:22.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.391+0000 7f62c85e9b80 1 --2- 172.21.15.16:0/253449948 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563f623051d0 0x563f62332dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:22.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.391+0000 7f62957fa700 0 client.8259 ms_handle_reset on v2:172.21.15.16:6835/3203451346 2023-10-16T12:29:22.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.391+0000 7f62c85e9b80 1 -- 172.21.15.16:0/253449948 >> 172.21.15.16:0/253449948 conn(0x563f62234d80 msgr2=0x563f622fc730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:22.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.391+0000 7f62c85e9b80 1 -- 172.21.15.16:0/253449948 shutdown_connections 2023-10-16T12:29:22.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.391+0000 7f62c85e9b80 1 -- 172.21.15.16:0/253449948 wait complete. 2023-10-16T12:29:22.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.391+0000 7f62c85e9b80 20 client.8259 trim_cache size 0 max 16384 2023-10-16T12:29:22.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.409+0000 7f62c85e9b80 1 Processor -- start 2023-10-16T12:29:22.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.409+0000 7f62c85e9b80 1 -- start start 2023-10-16T12:29:22.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.410+0000 7f62c85e9b80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x563f623030b0 0x563f62334770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:22.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.410+0000 7f62c85e9b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f623051d0 0x563f62334cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:22.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.410+0000 7f62c85e9b80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563f62331e20 0x563f6234ff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:22.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.410+0000 7f62c85e9b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f620da8d0 con 0x563f623051d0 2023-10-16T12:29:22.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.410+0000 7f62c85e9b80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f62351940 con 0x563f62331e20 2023-10-16T12:29:22.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.410+0000 7f62c85e9b80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x563f62303800 con 0x563f623030b0 2023-10-16T12:29:22.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.410+0000 7f62b999a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f623051d0 0x563f62334cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:22.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.410+0000 7f62b999a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f623051d0 0x563f62334cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43908/0 (socket says 172.21.15.16:43908) 2023-10-16T12:29:22.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.410+0000 7f62b999a700 1 -- 172.21.15.16:0/2282348202 learned_addr learned my addr 172.21.15.16:0/2282348202 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:29:22.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.410+0000 7f62b999a700 1 -- 172.21.15.16:0/2282348202 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x563f623030b0 msgr2=0x563f62334770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:29:22.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.410+0000 7f62ba19b700 1 --2- 172.21.15.16:0/2282348202 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x563f623030b0 0x563f62334770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:22.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.410+0000 7f62b999a700 1 --2- 172.21.15.16:0/2282348202 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x563f623030b0 0x563f62334770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:22.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.410+0000 7f62b999a700 1 -- 172.21.15.16:0/2282348202 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563f62331e20 msgr2=0x563f6234ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:29:22.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.410+0000 7f62ba99c700 1 --2- 172.21.15.16:0/2282348202 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563f62331e20 0x563f6234ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:22.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.410+0000 7f62b999a700 1 --2- 172.21.15.16:0/2282348202 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563f62331e20 0x563f6234ff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:22.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.410+0000 7f62b999a700 1 -- 172.21.15.16:0/2282348202 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563f6206dc10 con 0x563f623051d0 2023-10-16T12:29:22.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.410+0000 7f62ba99c700 1 --2- 172.21.15.16:0/2282348202 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563f62331e20 0x563f6234ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:29:22.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.410+0000 7f62b999a700 1 --2- 172.21.15.16:0/2282348202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f623051d0 0x563f62334cb0 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f62b00589c0 tx=0x7f62b005e050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac9916bb3ae50149 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:22.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.410+0000 7f62a8d82700 1 -- 172.21.15.16:0/2282348202 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b0054450 con 0x563f623051d0 2023-10-16T12:29:22.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.410+0000 7f62a8d82700 1 -- 172.21.15.16:0/2282348202 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62b0060410 con 0x563f623051d0 2023-10-16T12:29:22.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.410+0000 7f62a8d82700 1 -- 172.21.15.16:0/2282348202 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b00114e0 con 0x563f623051d0 2023-10-16T12:29:22.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.411+0000 7f62c85e9b80 1 -- 172.21.15.16:0/2282348202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f623051d0 msgr2=0x563f62334cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:22.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.411+0000 7f62c85e9b80 1 --2- 172.21.15.16:0/2282348202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f623051d0 0x563f62334cb0 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f62b00589c0 tx=0x7f62b005e050 comp rx=0 tx=0).stop 2023-10-16T12:29:22.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.411+0000 7f62c85e9b80 1 -- 172.21.15.16:0/2282348202 shutdown_connections 2023-10-16T12:29:22.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.411+0000 7f62c85e9b80 1 --2- 172.21.15.16:0/2282348202 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563f62331e20 0x563f6234ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:22.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.411+0000 7f62c85e9b80 1 --2- 172.21.15.16:0/2282348202 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x563f623030b0 0x563f62334770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:22.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.411+0000 7f62c85e9b80 1 --2- 172.21.15.16:0/2282348202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f623051d0 0x563f62334cb0 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:22.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.411+0000 7f62c85e9b80 1 -- 172.21.15.16:0/2282348202 >> 172.21.15.16:0/2282348202 conn(0x563f62234d80 msgr2=0x563f6232c060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:22.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.411+0000 7f62c85e9b80 1 -- 172.21.15.16:0/2282348202 shutdown_connections 2023-10-16T12:29:22.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.411+0000 7f62c85e9b80 1 -- 172.21.15.16:0/2282348202 wait complete. 2023-10-16T12:29:22.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.412+0000 7f62c85e9b80 1 -- start start 2023-10-16T12:29:22.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.412+0000 7f62c85e9b80 1 Processor -- start 2023-10-16T12:29:22.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.413+0000 7f62c85e9b80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563f623030b0 0x563f623b3650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:22.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.413+0000 7f62c85e9b80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x563f623051d0 0x563f62330f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:22.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.413+0000 7f62c85e9b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f62331e20 0x563f62331490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:22.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.413+0000 7f62c85e9b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f62350240 con 0x563f62331e20 2023-10-16T12:29:22.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.413+0000 7f62c85e9b80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f623435b0 con 0x563f623030b0 2023-10-16T12:29:22.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.413+0000 7f62c85e9b80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x563f623432d0 con 0x563f623051d0 2023-10-16T12:29:22.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.413+0000 7f62c85e9b80 10 client.0 _ll_register_callbacks cb 0x7f62ac003e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T12:29:22.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.413+0000 7f62ba99c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f62331e20 0x563f62331490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:22.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.413+0000 7f62ba99c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f62331e20 0x563f62331490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43916/0 (socket says 172.21.15.16:43916) 2023-10-16T12:29:22.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.413+0000 7f62ba99c700 1 -- 172.21.15.16:0/2575659566 learned_addr learned my addr 172.21.15.16:0/2575659566 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:29:22.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.413+0000 7f62ba19b700 1 --2- 172.21.15.16:0/2575659566 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563f623030b0 0x563f623b3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:22.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.413+0000 7f62ba99c700 1 -- 172.21.15.16:0/2575659566 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x563f623051d0 msgr2=0x563f62330f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:29:22.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.413+0000 7f62b999a700 1 --2- 172.21.15.16:0/2575659566 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x563f623051d0 0x563f62330f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:22.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.413+0000 7f62ba99c700 1 --2- 172.21.15.16:0/2575659566 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x563f623051d0 0x563f62330f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:22.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.413+0000 7f62ba99c700 1 -- 172.21.15.16:0/2575659566 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563f623030b0 msgr2=0x563f623b3650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:29:22.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.413+0000 7f62ba99c700 1 --2- 172.21.15.16:0/2575659566 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563f623030b0 0x563f623b3650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:22.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.413+0000 7f62ba99c700 1 -- 172.21.15.16:0/2575659566 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563f622173a0 con 0x563f62331e20 2023-10-16T12:29:22.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.413+0000 7f62ba19b700 1 --2- 172.21.15.16:0/2575659566 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563f623030b0 0x563f623b3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:29:22.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.413+0000 7f62b999a700 1 --2- 172.21.15.16:0/2575659566 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x563f623051d0 0x563f62330f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:29:22.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.413+0000 7f62ba99c700 1 --2- 172.21.15.16:0/2575659566 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f62331e20 0x563f62331490 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7f62ac016920 tx=0x7f62ac004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=419ba8d31e640fe3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:22.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.413+0000 7f62957fa700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2023-10-16T12:29:22.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.413+0000 7f62957fa700 1 -- 172.21.15.16:0/2575659566 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62ac00e040 con 0x563f62331e20 2023-10-16T12:29:22.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.413+0000 7f62957fa700 1 -- 172.21.15.16:0/2575659566 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(128..128 src has 1..128) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f62ac009410 con 0x563f62331e20 2023-10-16T12:29:22.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.413+0000 7f62c85e9b80 1 -- 172.21.15.16:0/2575659566 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f62b00564b0 con 0x563f62331e20 2023-10-16T12:29:22.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.413+0000 7f62957fa700 1 -- 172.21.15.16:0/2575659566 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=61) v2 ==== 24+0+0 (secure 0 0 0) 0x7f62ac009bc0 con 0x563f62331e20 2023-10-16T12:29:22.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.414+0000 7f62c85e9b80 10 client.8265 fetch_fsmap learned FSMap version 61 2023-10-16T12:29:22.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.414+0000 7f62c85e9b80 1 -- 172.21.15.16:0/2575659566 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=61,monmap=2+}) v3 -- 0x563f620ab200 con 0x563f62331e20 2023-10-16T12:29:22.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.414+0000 7f62957fa700 1 -- 172.21.15.16:0/2575659566 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62b00564b0 con 0x563f62331e20 2023-10-16T12:29:22.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.414+0000 7f62957fa700 1 -- 172.21.15.16:0/2575659566 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 61) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f62ac014070 con 0x563f62331e20 2023-10-16T12:29:22.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.414+0000 7f62c85e9b80 10 client.8265 fetch_fsmap finished waiting for FSMap version 61 2023-10-16T12:29:22.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.414+0000 7f62c85e9b80 10 client.8265 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-16T12:29:22.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.414+0000 7f62c85e9b80 20 client.8265 populate_metadata read hostname 'smithi016' 2023-10-16T12:29:22.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.414+0000 7f62c85e9b80 1 --2- 172.21.15.16:0/2575659566 >> [v2:172.21.15.16:6835/3203451346,v1:172.21.15.16:6837/3203451346] conn(0x563f6232d240 0x563f6232d620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:22.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.414+0000 7f62c85e9b80 4 client.8265 mds_command: new command op to 8109 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T12:29:22.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.414+0000 7f62c85e9b80 1 -- 172.21.15.16:0/2575659566 --> [v2:172.21.15.16:6835/3203451346,v1:172.21.15.16:6837/3203451346] -- command(tid 0: {"prefix": "status"}) v1 -- 0x563f62252b20 con 0x563f6232d240 2023-10-16T12:29:22.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.414+0000 7f62ba19b700 1 --2- 172.21.15.16:0/2575659566 >> [v2:172.21.15.16:6835/3203451346,v1:172.21.15.16:6837/3203451346] conn(0x563f6232d240 0x563f6232d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:22.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.414+0000 7f62957fa700 10 client.8265 ms_handle_connect on v2:172.21.15.16:6835/3203451346 2023-10-16T12:29:22.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.414+0000 7f62ba19b700 1 --2- 172.21.15.16:0/2575659566 >> [v2:172.21.15.16:6835/3203451346,v1:172.21.15.16:6837/3203451346] conn(0x563f6232d240 0x563f6232d620 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=349764ac2c47d6f6 server_cookie=d07e44fc61d08cd7 in_seq=0 out_seq=0 2023-10-16T12:29:22.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.414+0000 7f62957fa700 1 -- 172.21.15.16:0/2575659566 <== mds.0 v2:172.21.15.16:6835/3203451346 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x563f62252b20 con 0x563f6232d240 2023-10-16T12:29:22.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.414+0000 7f62957fa700 10 client.8265 handle_command_reply: tid=0 2023-10-16T12:29:22.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.414+0000 7f62c85e9b80 1 client.8265 shutdown 2023-10-16T12:29:22.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.415+0000 7f62c85e9b80 1 -- 172.21.15.16:0/2575659566 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f62331e20 msgr2=0x563f62331490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:22.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.415+0000 7f62c85e9b80 1 --2- 172.21.15.16:0/2575659566 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f62331e20 0x563f62331490 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7f62ac016920 tx=0x7f62ac004b40 comp rx=0 tx=0).stop 2023-10-16T12:29:22.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.415+0000 7f62c85e9b80 1 -- 172.21.15.16:0/2575659566 shutdown_connections 2023-10-16T12:29:22.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.415+0000 7f62c85e9b80 1 --2- 172.21.15.16:0/2575659566 >> [v2:172.21.15.16:6835/3203451346,v1:172.21.15.16:6837/3203451346] conn(0x563f6232d240 0x563f6232d620 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:22.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.415+0000 7f62c85e9b80 1 --2- 172.21.15.16:0/2575659566 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f62331e20 0x563f62331490 unknown :-1 s=CLOSED pgs=1397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:22.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.415+0000 7f62c85e9b80 1 --2- 172.21.15.16:0/2575659566 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563f623030b0 0x563f623b3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:22.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.415+0000 7f62c85e9b80 1 --2- 172.21.15.16:0/2575659566 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x563f623051d0 0x563f62330f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:22.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.415+0000 7f62957fa700 0 client.8265 ms_handle_reset on v2:172.21.15.16:6835/3203451346 2023-10-16T12:29:22.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.415+0000 7f62c85e9b80 1 -- 172.21.15.16:0/2575659566 >> 172.21.15.16:0/2575659566 conn(0x563f62234d80 msgr2=0x563f623043e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:22.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.415+0000 7f62c85e9b80 1 -- 172.21.15.16:0/2575659566 shutdown_connections 2023-10-16T12:29:22.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.415+0000 7f62c85e9b80 1 -- 172.21.15.16:0/2575659566 wait complete. 2023-10-16T12:29:22.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.415+0000 7f62c85e9b80 20 client.8265 trim_cache size 0 max 16384 2023-10-16T12:29:22.439 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-10-16T12:29:22.439 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "5b28dfa1-b21f-474a-9b2e-9fb77af58254", 2023-10-16T12:29:22.439 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 0, 2023-10-16T12:29:22.440 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 8109, 2023-10-16T12:29:22.440 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2023-10-16T12:29:22.440 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2023-10-16T12:29:22.440 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2023-10-16T12:29:22.440 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 6.0446422010000003, 2023-10-16T12:29:22.441 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 61, 2023-10-16T12:29:22.441 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 128, 2023-10-16T12:29:22.441 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 128, 2023-10-16T12:29:22.441 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 11.934246583 2023-10-16T12:29:22.441 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-10-16T12:29:22.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.415+0000 7f62bb19d700 1 -- 172.21.15.16:0/3331270105 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f629806b020 msgr2=0x7f629806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:22.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.415+0000 7f62bb19d700 1 --2- 172.21.15.16:0/3331270105 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f629806b020 0x7f629806d4e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f62a40090b0 tx=0x7f62a4008030 comp rx=0 tx=0).stop 2023-10-16T12:29:22.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.415+0000 7f62bb19d700 1 -- 172.21.15.16:0/3331270105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62b40b0a30 msgr2=0x7f62b40c9760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:22.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.415+0000 7f62bb19d700 1 --2- 172.21.15.16:0/3331270105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62b40b0a30 0x7f62b40c9760 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f62b0000fb0 tx=0x7f62b000eb10 comp rx=0 tx=0).stop 2023-10-16T12:29:22.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.415+0000 7f62bb19d700 1 -- 172.21.15.16:0/3331270105 shutdown_connections 2023-10-16T12:29:22.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.415+0000 7f62bb19d700 1 --2- 172.21.15.16:0/3331270105 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f629806b020 0x7f629806d4e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:22.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.415+0000 7f62bb19d700 1 --2- 172.21.15.16:0/3331270105 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f62b40c7790 0x7f62b41564e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:22.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.415+0000 7f62bb19d700 1 --2- 172.21.15.16:0/3331270105 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f62a0003a00 0x7f62b415ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:22.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.415+0000 7f62bb19d700 1 --2- 172.21.15.16:0/3331270105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62b40b0a30 0x7f62b40c9760 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:22.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.415+0000 7f62bb19d700 1 -- 172.21.15.16:0/3331270105 >> 172.21.15.16:0/3331270105 conn(0x7f62b40ccc80 msgr2=0x7f62b40b2f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:22.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.415+0000 7f62bb19d700 1 -- 172.21.15.16:0/3331270105 shutdown_connections 2023-10-16T12:29:22.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.415+0000 7f62bb19d700 1 -- 172.21.15.16:0/3331270105 wait complete. 2023-10-16T12:29:22.445 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T12:29:22.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.568+0000 7f6c9c162700 1 Processor -- start 2023-10-16T12:29:22.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.569+0000 7f6c9c162700 1 -- start start 2023-10-16T12:29:22.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.569+0000 7f6c9c162700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6c9415a5c0 0x7f6c940cef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:22.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.569+0000 7f6c9c162700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f6c940d0ae0 0x7f6c940cc1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:22.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.569+0000 7f6c9c162700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c94072a50 con 0x7f6c940ce010 2023-10-16T12:29:22.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.569+0000 7f6c9c162700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c940cf440 con 0x7f6c9415a5c0 2023-10-16T12:29:22.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.569+0000 7f6c9c162700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c940cba30 con 0x7f6c940d0ae0 2023-10-16T12:29:22.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.569+0000 7f6c99efe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6c9415a5c0 0x7f6c940cef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:22.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.570+0000 7f6c996fd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f6c940ce010 0x7f6c940d03d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33196/0 (socket says 172.21.15.16:33196) 2023-10-16T12:29:22.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.570+0000 7f6c99efe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6c9415a5c0 0x7f6c940cef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43920/0 (socket says 172.21.15.16:43920) 2023-10-16T12:29:22.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.570+0000 7f6c996fd700 1 -- 172.21.15.16:0/179414394 learned_addr learned my addr 172.21.15.16:0/179414394 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:29:22.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.570+0000 7f6c9a6ff700 1 --2- 172.21.15.16:0/179414394 >> v2:172.21.15.49:3300/0 conn(0x7f6c940d0ae0 0x7f6c940cc1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:22.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.570+0000 7f6c99efe700 1 -- 172.21.15.16:0/179414394 >> v2:172.21.15.49:3300/0 conn(0x7f6c940d0ae0 msgr2=0x7f6c940cc1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:29:22.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.570+0000 7f6c99efe700 1 --2- 172.21.15.16:0/179414394 >> v2:172.21.15.49:3300/0 conn(0x7f6c940d0ae0 0x7f6c940cc1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:22.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.570+0000 7f6c99efe700 1 -- 172.21.15.16:0/179414394 >> v1:172.21.15.16:6789/0 conn(0x7f6c940ce010 legacy=0x7f6c940d03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:22.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.570+0000 7f6c99efe700 1 -- 172.21.15.16:0/179414394 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c940cf630 con 0x7f6c9415a5c0 2023-10-16T12:29:22.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.570+0000 7f6c98efc700 1 -- 172.21.15.16:0/179414394 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3113482845 0 0) 0x7f6c94072a50 con 0x7f6c940ce010 2023-10-16T12:29:22.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.570+0000 7f6c9a6ff700 1 --2- 172.21.15.16:0/179414394 >> v2:172.21.15.49:3300/0 conn(0x7f6c940d0ae0 0x7f6c940cc1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:29:22.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.570+0000 7f6c99efe700 1 --2- 172.21.15.16:0/179414394 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c9415a5c0 0x7f6c940cef00 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f6c94073480 tx=0x7f6c90009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da92442a072ca2d4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:22.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.570+0000 7f6c98efc700 1 -- 172.21.15.16:0/179414394 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c90011830 con 0x7f6c9415a5c0 2023-10-16T12:29:22.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.570+0000 7f6c98efc700 1 -- 172.21.15.16:0/179414394 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c90011990 con 0x7f6c9415a5c0 2023-10-16T12:29:22.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.570+0000 7f6c98efc700 1 -- 172.21.15.16:0/179414394 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c90011d20 con 0x7f6c9415a5c0 2023-10-16T12:29:22.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.571+0000 7f6c9c162700 1 -- 172.21.15.16:0/179414394 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c9415a5c0 msgr2=0x7f6c940cef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:22.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.571+0000 7f6c9c162700 1 --2- 172.21.15.16:0/179414394 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c9415a5c0 0x7f6c940cef00 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f6c94073480 tx=0x7f6c90009b20 comp rx=0 tx=0).stop 2023-10-16T12:29:22.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.571+0000 7f6c9c162700 1 -- 172.21.15.16:0/179414394 shutdown_connections 2023-10-16T12:29:22.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.571+0000 7f6c9c162700 1 --2- 172.21.15.16:0/179414394 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c9415a5c0 0x7f6c940cef00 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:22.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.571+0000 7f6c9c162700 1 --2- 172.21.15.16:0/179414394 >> v2:172.21.15.49:3300/0 conn(0x7f6c940d0ae0 0x7f6c940cc1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:22.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.571+0000 7f6c9c162700 1 -- 172.21.15.16:0/179414394 >> 172.21.15.16:0/179414394 conn(0x7f6c940b1fd0 msgr2=0x7f6c940b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:22.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.571+0000 7f6c9c162700 1 -- 172.21.15.16:0/179414394 shutdown_connections 2023-10-16T12:29:22.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.571+0000 7f6c9c162700 1 -- 172.21.15.16:0/179414394 wait complete. 2023-10-16T12:29:22.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.572+0000 7f6c9c162700 1 Processor -- start 2023-10-16T12:29:22.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.572+0000 7f6c9c162700 1 -- start start 2023-10-16T12:29:22.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.573+0000 7f6c9c162700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c940ce010 0x7f6c941f5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:22.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.573+0000 7f6c9c162700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6c940d0ae0 0x7f6c94157120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:22.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.573+0000 7f6c9c162700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6c9415a5c0 0x7f6c940cc490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:22.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.573+0000 7f6c9c162700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c941f5670 con 0x7f6c940ce010 2023-10-16T12:29:22.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.573+0000 7f6c9c162700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c9415bfb0 con 0x7f6c940d0ae0 2023-10-16T12:29:22.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.573+0000 7f6c9c162700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c9415c290 con 0x7f6c9415a5c0 2023-10-16T12:29:22.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.573+0000 7f6c99efe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c940ce010 0x7f6c941f5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:22.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.573+0000 7f6c99efe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c940ce010 0x7f6c941f5130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43926/0 (socket says 172.21.15.16:43926) 2023-10-16T12:29:22.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.573+0000 7f6c99efe700 1 -- 172.21.15.16:0/968868832 learned_addr learned my addr 172.21.15.16:0/968868832 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:29:22.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.573+0000 7f6c996fd700 1 --2- 172.21.15.16:0/968868832 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6c940d0ae0 0x7f6c94157120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:22.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.573+0000 7f6c9a6ff700 1 --2- 172.21.15.16:0/968868832 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6c9415a5c0 0x7f6c940cc490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:22.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.573+0000 7f6c99efe700 1 -- 172.21.15.16:0/968868832 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6c9415a5c0 msgr2=0x7f6c940cc490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:22.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.573+0000 7f6c99efe700 1 --2- 172.21.15.16:0/968868832 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6c9415a5c0 0x7f6c940cc490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:22.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.573+0000 7f6c99efe700 1 -- 172.21.15.16:0/968868832 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6c940d0ae0 msgr2=0x7f6c94157120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:22.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.573+0000 7f6c99efe700 1 --2- 172.21.15.16:0/968868832 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6c940d0ae0 0x7f6c94157120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:22.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.573+0000 7f6c99efe700 1 -- 172.21.15.16:0/968868832 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c900094a0 con 0x7f6c940ce010 2023-10-16T12:29:22.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.573+0000 7f6c99efe700 1 --2- 172.21.15.16:0/968868832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c940ce010 0x7f6c941f5130 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f6c900090b0 tx=0x7f6c90004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:22.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.574+0000 7f6c867fc700 1 -- 172.21.15.16:0/968868832 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c900036a0 con 0x7f6c940ce010 2023-10-16T12:29:22.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.574+0000 7f6c867fc700 1 -- 172.21.15.16:0/968868832 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c9001f070 con 0x7f6c940ce010 2023-10-16T12:29:22.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.574+0000 7f6c9c162700 1 -- 172.21.15.16:0/968868832 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c9415c570 con 0x7f6c940ce010 2023-10-16T12:29:22.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.574+0000 7f6c867fc700 1 -- 172.21.15.16:0/968868832 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c90003a60 con 0x7f6c940ce010 2023-10-16T12:29:22.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.574+0000 7f6c9c162700 1 -- 172.21.15.16:0/968868832 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c9415cb30 con 0x7f6c940ce010 2023-10-16T12:29:22.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.575+0000 7f6c9c162700 1 -- 172.21.15.16:0/968868832 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c9404ebb0 con 0x7f6c940ce010 2023-10-16T12:29:22.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.575+0000 7f6c867fc700 1 -- 172.21.15.16:0/968868832 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 62) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6c90003bc0 con 0x7f6c940ce010 2023-10-16T12:29:22.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.575+0000 7f6c867fc700 1 --2- 172.21.15.16:0/968868832 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f6c7806b020 0x7f6c7806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:22.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.575+0000 7f6c867fc700 1 -- 172.21.15.16:0/968868832 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f6c90021070 con 0x7f6c940ce010 2023-10-16T12:29:22.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.576+0000 7f6c996fd700 1 --2- 172.21.15.16:0/968868832 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f6c7806b020 0x7f6c7806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:22.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.576+0000 7f6c996fd700 1 --2- 172.21.15.16:0/968868832 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f6c7806b020 0x7f6c7806d4e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f6c9415bb80 tx=0x7f6c88007e70 comp rx=0 tx=0).ready entity=mgr.7884 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:22.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.580+0000 7f6c867fc700 1 -- 172.21.15.16:0/968868832 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6c90055ff0 con 0x7f6c940ce010 2023-10-16T12:29:22.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.831+0000 7f6c9c162700 1 -- 172.21.15.16:0/968868832 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f6c940cad70 con 0x7f6c940ce010 2023-10-16T12:29:22.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.831+0000 7f6c867fc700 1 -- 172.21.15.16:0/968868832 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v61) v1 ==== 53+0+73 (secure 0 0 0) 0x7f6c90059610 con 0x7f6c940ce010 2023-10-16T12:29:22.833 INFO:teuthology.orchestra.run.smithi016.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-16T12:29:22.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.833+0000 7f6c9c162700 1 -- 172.21.15.16:0/968868832 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f6c7806b020 msgr2=0x7f6c7806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:22.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.833+0000 7f6c9c162700 1 --2- 172.21.15.16:0/968868832 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f6c7806b020 0x7f6c7806d4e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f6c9415bb80 tx=0x7f6c88007e70 comp rx=0 tx=0).stop 2023-10-16T12:29:22.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.833+0000 7f6c9c162700 1 -- 172.21.15.16:0/968868832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c940ce010 msgr2=0x7f6c941f5130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:22.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.833+0000 7f6c9c162700 1 --2- 172.21.15.16:0/968868832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c940ce010 0x7f6c941f5130 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f6c900090b0 tx=0x7f6c90004750 comp rx=0 tx=0).stop 2023-10-16T12:29:22.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.833+0000 7f6c9c162700 1 -- 172.21.15.16:0/968868832 shutdown_connections 2023-10-16T12:29:22.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.833+0000 7f6c9c162700 1 --2- 172.21.15.16:0/968868832 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f6c7806b020 0x7f6c7806d4e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:22.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.833+0000 7f6c9c162700 1 --2- 172.21.15.16:0/968868832 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6c9415a5c0 0x7f6c940cc490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:22.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.833+0000 7f6c9c162700 1 --2- 172.21.15.16:0/968868832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c940ce010 0x7f6c941f5130 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:22.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.833+0000 7f6c9c162700 1 --2- 172.21.15.16:0/968868832 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6c940d0ae0 0x7f6c94157120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:22.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.833+0000 7f6c9c162700 1 -- 172.21.15.16:0/968868832 >> 172.21.15.16:0/968868832 conn(0x7f6c940b1fd0 msgr2=0x7f6c940ce4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:22.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.833+0000 7f6c9c162700 1 -- 172.21.15.16:0/968868832 shutdown_connections 2023-10-16T12:29:22.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:22.833+0000 7f6c9c162700 1 -- 172.21.15.16:0/968868832 wait complete. 2023-10-16T12:29:22.848 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-16T12:29:22.848 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T12:29:22.848 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi016.front.sepia.ceph.com 2023-10-16T12:29:22.849 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-16T12:29:22.849 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-16T12:29:22.849 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-16T12:29:22.850 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-16T12:29:22.850 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T12:29:22.850 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T12:29:22.851 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:29:22.851 DEBUG:teuthology.orchestra.run.smithi016:> ip addr 2023-10-16T12:29:22.869 INFO:teuthology.orchestra.run.smithi016.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T12:29:22.869 INFO:teuthology.orchestra.run.smithi016.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T12:29:22.870 INFO:teuthology.orchestra.run.smithi016.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T12:29:22.870 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:29:22.870 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 ::1/128 scope host 2023-10-16T12:29:22.870 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:29:22.870 INFO:teuthology.orchestra.run.smithi016.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T12:29:22.871 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e0 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:29:22.871 INFO:teuthology.orchestra.run.smithi016.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T12:29:22.871 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e1 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:29:22.871 INFO:teuthology.orchestra.run.smithi016.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T12:29:22.871 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:64 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:29:22.872 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.21.15.16/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-16T12:29:22.872 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft 41173sec preferred_lft 41173sec 2023-10-16T12:29:22.872 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::ec4:7aff:febd:1564/64 scope link 2023-10-16T12:29:22.872 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:29:22.872 INFO:teuthology.orchestra.run.smithi016.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T12:29:22.873 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:65 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:29:22.873 INFO:teuthology.orchestra.run.smithi016.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T12:29:22.873 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 36:cf:ea:25:db:f7 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:29:22.873 INFO:teuthology.orchestra.run.smithi016.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-16T12:29:22.873 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:29:22.873 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::cfc:c5ff:fe52:a786/64 scope link 2023-10-16T12:29:22.874 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:29:22.874 INFO:teuthology.orchestra.run.smithi016.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T12:29:22.874 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 6e:62:9d:a4:43:23 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T12:29:22.874 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::6c62:9dff:fea4:4323/64 scope link 2023-10-16T12:29:22.874 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:29:22.875 INFO:teuthology.orchestra.run.smithi016.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T12:29:22.875 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 36:cf:ea:25:db:f7 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-16T12:29:22.875 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::34cf:eaff:fe25:dbf7/64 scope link 2023-10-16T12:29:22.875 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:29:22.876 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:29:22.876 DEBUG:teuthology.orchestra.run.smithi016:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T12:29:22.935 INFO:teuthology.orchestra.run.smithi016.stdout:1 2023-10-16T12:29:22.937 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:29:22.937 DEBUG:teuthology.orchestra.run.smithi016:> route 2023-10-16T12:29:22.996 INFO:teuthology.orchestra.run.smithi016.stdout:Kernel IP routing table 2023-10-16T12:29:22.996 INFO:teuthology.orchestra.run.smithi016.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T12:29:22.996 INFO:teuthology.orchestra.run.smithi016.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-16T12:29:22.996 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-16T12:29:22.997 INFO:teuthology.orchestra.run.smithi016.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T12:29:22.997 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:29:22.997 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec bash -c ' 2023-10-16T12:29:22.998 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2023-10-16T12:29:22.998 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-16T12:29:22.999 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-16T12:29:22.999 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-16T12:29:22.999 DEBUG:teuthology.orchestra.run.smithi016:> ') 2023-10-16T12:29:23.103 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:29:23.103 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2023-10-16T12:29:23.121 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-16T12:29:23.121 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-16T12:29:23.122 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T12:29:23.122 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:29:23.123 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set brx.0 up 2023-10-16T12:29:23.189 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T12:29:23.189 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-16T12:29:23.246 INFO:teuthology.orchestra.run.smithi016.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-16T12:29:23.247 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T12:29:23.247 DEBUG:teuthology.orchestra.run.smithi016:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-16T12:29:23.303 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-16T12:29:23.304 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:29:23.304 DEBUG:teuthology.orchestra.run.smithi016:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T12:29:23.391 INFO:teuthology.orchestra.run.smithi016.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T12:29:23.391 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T12:29:23.391 INFO:teuthology.orchestra.run.smithi016.stdout:invalid new device string format 2023-10-16T12:29:23.400 INFO:teuthology.orchestra.run.smithi016.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T12:29:23.400 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T12:29:23.401 INFO:teuthology.orchestra.run.smithi016.stdout:invalid new device string format 2023-10-16T12:29:23.401 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: resolved to: "172.21.15.16:3300,172.21.15.49:3300,172.21.15.49:3301" 2023-10-16T12:29:23.401 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T12:29:23.474 INFO:teuthology.orchestra.run.smithi016.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T12:29:23.474 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T12:29:23.474 INFO:teuthology.orchestra.run.smithi016.stdout:invalid new device string format 2023-10-16T12:29:23.474 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: resolved to: "172.21.15.16:3300,172.21.15.49:3300,172.21.15.49:3301" 2023-10-16T12:29:23.475 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T12:29:23.475 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: resolved to: "172.21.15.16:6789,172.21.15.49:6789,172.21.15.49:6790" 2023-10-16T12:29:23.475 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.16:6789,172.21.15.49:6789,172.21.15.49:6790:/ 2023-10-16T12:29:23.475 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=5b28dfa1-b21f-474a-9b2e-9fb77af58254" will pass to kernel 2023-10-16T12:29:23.476 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-16T12:29:23.477 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:29:23.477 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-16T12:29:23.505 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:29:23.505 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T12:29:23.588 INFO:teuthology.orchestra.run.smithi016.stdout:client7825 2023-10-16T12:29:23.588 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:29:23.589 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:29:23.646 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-16T12:29:23.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.810+0000 7f818eda7700 1 Processor -- start 2023-10-16T12:29:23.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.810+0000 7f818eda7700 1 -- start start 2023-10-16T12:29:23.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.811+0000 7f818eda7700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f81880d0ae0 0x7f81880cef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:23.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.811+0000 7f818eda7700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8188072a50 con 0x7f81880ce010 2023-10-16T12:29:23.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.811+0000 7f818eda7700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81880cba30 con 0x7f818815a5c0 2023-10-16T12:29:23.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.811+0000 7f818eda7700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81880cf440 con 0x7f81880d0ae0 2023-10-16T12:29:23.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.811+0000 7f818d344700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f81880ce010 0x7f81880cc1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33216/0 (socket says 172.21.15.16:33216) 2023-10-16T12:29:23.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.811+0000 7f818d344700 1 -- 172.21.15.16:0/3174382605 learned_addr learned my addr 172.21.15.16:0/3174382605 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:29:23.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.811+0000 7f8187fff700 1 --2- 172.21.15.16:0/3174382605 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f81880d0ae0 0x7f81880cef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:23.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.811+0000 7f81877fe700 1 -- 172.21.15.16:0/3174382605 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 832066306 0 0) 0x7f8188072a50 con 0x7f81880ce010 2023-10-16T12:29:23.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.811+0000 7f81877fe700 1 -- 172.21.15.16:0/3174382605 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81780035b0 con 0x7f81880ce010 2023-10-16T12:29:23.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.811+0000 7f81877fe700 1 -- 172.21.15.16:0/3174382605 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1184985647 0 0) 0x7f81880cba30 con 0x7f818815a5c0 2023-10-16T12:29:23.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.811+0000 7f81877fe700 1 -- 172.21.15.16:0/3174382605 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8188072a50 con 0x7f818815a5c0 2023-10-16T12:29:23.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.811+0000 7f81877fe700 1 -- 172.21.15.16:0/3174382605 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 538736450 0 0) 0x7f81780035b0 con 0x7f81880ce010 2023-10-16T12:29:23.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.812+0000 7f81877fe700 1 -- 172.21.15.16:0/3174382605 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f81880d0ae0 msgr2=0x7f81880cef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:29:23.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.812+0000 7f81877fe700 1 --2- 172.21.15.16:0/3174382605 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f81880d0ae0 0x7f81880cef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:23.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.812+0000 7f81877fe700 1 -- 172.21.15.16:0/3174382605 >> v1:172.21.15.49:6789/0 conn(0x7f818815a5c0 legacy=0x7f81880d03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:23.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.812+0000 7f81877fe700 1 -- 172.21.15.16:0/3174382605 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81880cf630 con 0x7f81880ce010 2023-10-16T12:29:23.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.812+0000 7f81877fe700 1 -- 172.21.15.16:0/3174382605 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f81800019f0 con 0x7f81880ce010 2023-10-16T12:29:23.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.812+0000 7f81877fe700 1 -- 172.21.15.16:0/3174382605 >> v1:172.21.15.16:6789/0 conn(0x7f81880ce010 legacy=0x7f81880cc1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:23.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.812+0000 7f81877fe700 1 --2- 172.21.15.16:0/3174382605 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f81880d0ae0 0x7f8178003bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:23.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.812+0000 7f81877fe700 1 --2- 172.21.15.16:0/3174382605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8178004120 0x7f8178004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:23.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.812+0000 7f81877fe700 1 --2- 172.21.15.16:0/3174382605 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8178004d90 0x7f8178005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:23.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.812+0000 7f81877fe700 1 -- 172.21.15.16:0/3174382605 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f817800bac0 con 0x7f8178004120 2023-10-16T12:29:23.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.812+0000 7f81877fe700 1 -- 172.21.15.16:0/3174382605 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f817800bc30 con 0x7f8178004d90 2023-10-16T12:29:23.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.812+0000 7f81877fe700 1 -- 172.21.15.16:0/3174382605 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f817800bdf0 con 0x7f81880d0ae0 2023-10-16T12:29:23.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.812+0000 7f8187fff700 1 --2- 172.21.15.16:0/3174382605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8178004120 0x7f8178004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:23.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.812+0000 7f818cb43700 1 --2- 172.21.15.16:0/3174382605 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f81880d0ae0 0x7f8178003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:23.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.812+0000 7f818d344700 1 --2- 172.21.15.16:0/3174382605 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8178004d90 0x7f8178005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:23.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.812+0000 7f8187fff700 1 -- 172.21.15.16:0/3174382605 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f81880d0ae0 msgr2=0x7f8178003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:29:23.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.812+0000 7f8187fff700 1 --2- 172.21.15.16:0/3174382605 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f81880d0ae0 0x7f8178003bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:23.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.812+0000 7f8187fff700 1 -- 172.21.15.16:0/3174382605 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8178004d90 msgr2=0x7f8178005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:29:23.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.812+0000 7f8187fff700 1 --2- 172.21.15.16:0/3174382605 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8178004d90 0x7f8178005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:23.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.812+0000 7f8187fff700 1 -- 172.21.15.16:0/3174382605 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81780035b0 con 0x7f8178004120 2023-10-16T12:29:23.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.812+0000 7f8187fff700 1 --2- 172.21.15.16:0/3174382605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8178004120 0x7f8178004850 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f8174009b60 tx=0x7f8178005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23c16c8a5074a5dd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:23.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.812+0000 7f81877fe700 1 -- 172.21.15.16:0/3174382605 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f817400e6c0 con 0x7f8178004120 2023-10-16T12:29:23.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.812+0000 7f81877fe700 1 -- 172.21.15.16:0/3174382605 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f817400e820 con 0x7f8178004120 2023-10-16T12:29:23.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.812+0000 7f81877fe700 1 -- 172.21.15.16:0/3174382605 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f817400eb90 con 0x7f8178004120 2023-10-16T12:29:23.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.813+0000 7f818eda7700 1 -- 172.21.15.16:0/3174382605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8178004120 msgr2=0x7f8178004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:23.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.813+0000 7f818eda7700 1 --2- 172.21.15.16:0/3174382605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8178004120 0x7f8178004850 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f8174009b60 tx=0x7f8178005880 comp rx=0 tx=0).stop 2023-10-16T12:29:23.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.813+0000 7f818d344700 1 -- 172.21.15.16:0/3174382605 reap_dead start 2023-10-16T12:29:23.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.813+0000 7f818eda7700 1 -- 172.21.15.16:0/3174382605 shutdown_connections 2023-10-16T12:29:23.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.813+0000 7f818eda7700 1 -- 172.21.15.16:0/3174382605 >> 172.21.15.16:0/3174382605 conn(0x7f81880b1fd0 msgr2=0x7f81880b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:23.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.813+0000 7f818eda7700 1 -- 172.21.15.16:0/3174382605 shutdown_connections 2023-10-16T12:29:23.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.813+0000 7f818eda7700 1 -- 172.21.15.16:0/3174382605 wait complete. 2023-10-16T12:29:23.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.814+0000 7f818eda7700 1 Processor -- start 2023-10-16T12:29:23.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.814+0000 7f818eda7700 1 -- start start 2023-10-16T12:29:23.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.815+0000 7f818eda7700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8178004120 0x7f8188157720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:23.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.815+0000 7f818eda7700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f818815ccc0 0x7f81880b0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:23.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.815+0000 7f818eda7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81881f6dd0 0x7f81880b0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:23.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.815+0000 7f818eda7700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8188157c60 con 0x7f81881f6dd0 2023-10-16T12:29:23.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.815+0000 7f818eda7700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81881f72b0 con 0x7f8178004120 2023-10-16T12:29:23.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.815+0000 7f818eda7700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81880b3b40 con 0x7f818815ccc0 2023-10-16T12:29:23.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.815+0000 7f818d344700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81881f6dd0 0x7f81880b0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:23.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.815+0000 7f818d344700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81881f6dd0 0x7f81880b0b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43954/0 (socket says 172.21.15.16:43954) 2023-10-16T12:29:23.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.815+0000 7f818d344700 1 -- 172.21.15.16:0/1347380241 learned_addr learned my addr 172.21.15.16:0/1347380241 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:29:23.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.815+0000 7f818cb43700 1 --2- 172.21.15.16:0/1347380241 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8178004120 0x7f8188157720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:23.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.815+0000 7f8187fff700 1 --2- 172.21.15.16:0/1347380241 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f818815ccc0 0x7f81880b0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:23.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.816+0000 7f818d344700 1 -- 172.21.15.16:0/1347380241 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f818815ccc0 msgr2=0x7f81880b0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:23.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.816+0000 7f818d344700 1 --2- 172.21.15.16:0/1347380241 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f818815ccc0 0x7f81880b0650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:23.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.816+0000 7f818d344700 1 -- 172.21.15.16:0/1347380241 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8178004120 msgr2=0x7f8188157720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:23.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.816+0000 7f818d344700 1 --2- 172.21.15.16:0/1347380241 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8178004120 0x7f8188157720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:23.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.816+0000 7f818d344700 1 -- 172.21.15.16:0/1347380241 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81880aca00 con 0x7f81881f6dd0 2023-10-16T12:29:23.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.816+0000 7f818cb43700 1 --2- 172.21.15.16:0/1347380241 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8178004120 0x7f8188157720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:29:23.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.816+0000 7f818d344700 1 --2- 172.21.15.16:0/1347380241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81881f6dd0 0x7f81880b0b90 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f81880c3f40 tx=0x7f818000a5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:23.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.816+0000 7f81857fa700 1 -- 172.21.15.16:0/1347380241 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f818000ad00 con 0x7f81881f6dd0 2023-10-16T12:29:23.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.816+0000 7f81857fa700 1 -- 172.21.15.16:0/1347380241 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f818000ae60 con 0x7f81881f6dd0 2023-10-16T12:29:23.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.816+0000 7f818eda7700 1 -- 172.21.15.16:0/1347380241 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81880acce0 con 0x7f81881f6dd0 2023-10-16T12:29:23.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.816+0000 7f81857fa700 1 -- 172.21.15.16:0/1347380241 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8180013630 con 0x7f81881f6dd0 2023-10-16T12:29:23.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.816+0000 7f818eda7700 1 -- 172.21.15.16:0/1347380241 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81880ad2a0 con 0x7f81881f6dd0 2023-10-16T12:29:23.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.817+0000 7f818eda7700 1 -- 172.21.15.16:0/1347380241 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81880b2fc0 con 0x7f81881f6dd0 2023-10-16T12:29:23.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.817+0000 7f81857fa700 1 -- 172.21.15.16:0/1347380241 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 62) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8180013790 con 0x7f81881f6dd0 2023-10-16T12:29:23.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.818+0000 7f81857fa700 1 --2- 172.21.15.16:0/1347380241 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f816c06b020 0x7f816c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:23.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.818+0000 7f81857fa700 1 -- 172.21.15.16:0/1347380241 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f818008a350 con 0x7f81881f6dd0 2023-10-16T12:29:23.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.818+0000 7f818cb43700 1 --2- 172.21.15.16:0/1347380241 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f816c06b020 0x7f816c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:23.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.818+0000 7f818cb43700 1 --2- 172.21.15.16:0/1347380241 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f816c06b020 0x7f816c06d4e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f817c007fd0 tx=0x7f817c009040 comp rx=0 tx=0).ready entity=mgr.7884 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:23.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:23.822+0000 7f81857fa700 1 -- 172.21.15.16:0/1347380241 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f81880b2fc0 con 0x7f81881f6dd0 2023-10-16T12:29:24.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:24.055+0000 7f818eda7700 1 -- 172.21.15.16:0/1347380241 --> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f81880cfbf0 con 0x7f816c06b020 2023-10-16T12:29:24.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:24.057+0000 7f81857fa700 1 -- 172.21.15.16:0/1347380241 <== mgr.7884 v2:172.21.15.16:6832/2309613638 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f817c001430 con 0x7f816c06b020 2023-10-16T12:29:24.059 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2023-10-16T12:29:24.059 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:29:24.059 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cephfs" 2023-10-16T12:29:24.059 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:29:24.059 INFO:teuthology.orchestra.run.smithi016.stdout:] 2023-10-16T12:29:24.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:24.059+0000 7f818eda7700 1 -- 172.21.15.16:0/1347380241 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f816c06b020 msgr2=0x7f816c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:24.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:24.059+0000 7f818eda7700 1 --2- 172.21.15.16:0/1347380241 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f816c06b020 0x7f816c06d4e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f817c007fd0 tx=0x7f817c009040 comp rx=0 tx=0).stop 2023-10-16T12:29:24.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:24.059+0000 7f818eda7700 1 -- 172.21.15.16:0/1347380241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81881f6dd0 msgr2=0x7f81880b0b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:24.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:24.059+0000 7f818eda7700 1 --2- 172.21.15.16:0/1347380241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81881f6dd0 0x7f81880b0b90 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f81880c3f40 tx=0x7f818000a5b0 comp rx=0 tx=0).stop 2023-10-16T12:29:24.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:24.059+0000 7f818eda7700 1 -- 172.21.15.16:0/1347380241 shutdown_connections 2023-10-16T12:29:24.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:24.059+0000 7f818eda7700 1 --2- 172.21.15.16:0/1347380241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81881f6dd0 0x7f81880b0b90 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:24.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:24.059+0000 7f818eda7700 1 --2- 172.21.15.16:0/1347380241 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8178004120 0x7f8188157720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:24.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:24.059+0000 7f818eda7700 1 --2- 172.21.15.16:0/1347380241 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f816c06b020 0x7f816c06d4e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:24.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:24.059+0000 7f818eda7700 1 --2- 172.21.15.16:0/1347380241 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f818815ccc0 0x7f81880b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:24.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:24.059+0000 7f818eda7700 1 -- 172.21.15.16:0/1347380241 >> 172.21.15.16:0/1347380241 conn(0x7f81880b1fd0 msgr2=0x7f81881588e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:24.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:24.059+0000 7f818eda7700 1 -- 172.21.15.16:0/1347380241 shutdown_connections 2023-10-16T12:29:24.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:24.059+0000 7f818eda7700 1 -- 172.21.15.16:0/1347380241 wait complete. 2023-10-16T12:29:24.073 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-16T12:29:24.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:24.198+0000 7f8554ac5700 1 Processor -- start 2023-10-16T12:29:24.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:24.198+0000 7f8554ac5700 1 -- start start 2023-10-16T12:29:24.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:24.199+0000 7f8554ac5700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f85500ce5d0 0x7f85500cc190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:24.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:24.199+0000 7f8554ac5700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85500b2440 con 0x7f855015ec80 2023-10-16T12:29:24.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:24.199+0000 7f8554ac5700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85500d00e0 con 0x7f855015f060 2023-10-16T12:29:24.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:24.199+0000 7f8554ac5700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85500b2c50 con 0x7f85500ce5d0 2023-10-16T12:29:24.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:24.199+0000 7f854e59c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f855015ec80 0x7f85500cdec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33232/0 (socket says 172.21.15.16:33232) 2023-10-16T12:29:24.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:24.199+0000 7f854e59c700 1 -- 172.21.15.16:0/4047960686 learned_addr learned my addr 172.21.15.16:0/4047960686 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:29:24.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:24.200+0000 7f854dd9b700 1 --2- 172.21.15.16:0/4047960686 >> v2:172.21.15.49:3300/0 conn(0x7f85500ce5d0 0x7f85500cc190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:24.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:24.200+0000 7f854d59a700 1 -- 172.21.15.16:0/4047960686 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 895970616 0 0) 0x7f85500b2440 con 0x7f855015ec80 2023-10-16T12:29:24.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:24.200+0000 7f854d59a700 1 -- 172.21.15.16:0/4047960686 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85380035b0 con 0x7f855015ec80 2023-10-16T12:29:24.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:24.200+0000 7f854d59a700 1 -- 172.21.15.16:0/4047960686 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2608900310 0 0) 0x7f85380035b0 con 0x7f855015ec80 2023-10-16T12:29:24.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:24.200+0000 7f854dd9b700 1 -- 172.21.15.16:0/4047960686 >> v1:172.21.15.49:6789/0 conn(0x7f855015f060 legacy=0x7f85500cc6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:24.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:24.200+0000 7f854dd9b700 1 -- 172.21.15.16:0/4047960686 >> v1:172.21.15.16:6789/0 conn(0x7f855015ec80 legacy=0x7f85500cdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:24.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:24.200+0000 7f854dd9b700 1 -- 172.21.15.16:0/4047960686 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85500cef60 con 0x7f85500ce5d0 2023-10-16T12:29:24.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:24.200+0000 7f854dd9b700 1 --2- 172.21.15.16:0/4047960686 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f85500ce5d0 0x7f85500cc190 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f8540009870 tx=0x7f8540009b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3816394195b9ce43 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:24.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:24.200+0000 7f854d59a700 1 -- 172.21.15.16:0/4047960686 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8544001670 con 0x7f85500ce5d0 2023-10-16T12:29:24.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:24.200+0000 7f854d59a700 1 -- 172.21.15.16:0/4047960686 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8540004030 con 0x7f85500ce5d0 2023-10-16T12:29:24.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:24.200+0000 7f854d59a700 1 -- 172.21.15.16:0/4047960686 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85400043c0 con 0x7f85500ce5d0 2023-10-16T12:29:24.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:24.201+0000 7f8554ac5700 1 -- 172.21.15.16:0/4047960686 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f85500ce5d0 msgr2=0x7f85500cc190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:24.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:24.201+0000 7f8554ac5700 1 --2- 172.21.15.16:0/4047960686 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f85500ce5d0 0x7f85500cc190 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f8540009870 tx=0x7f8540009b80 comp rx=0 tx=0).stop 2023-10-16T12:29:24.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:24.201+0000 7f8554ac5700 1 -- 172.21.15.16:0/4047960686 shutdown_connections 2023-10-16T12:29:24.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:24.201+0000 7f8554ac5700 1 --2- 172.21.15.16:0/4047960686 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f85500ce5d0 0x7f85500cc190 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:24.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:24.201+0000 7f8554ac5700 1 -- 172.21.15.16:0/4047960686 >> 172.21.15.16:0/4047960686 conn(0x7f85500d15f0 msgr2=0x7f85500d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:24.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:24.201+0000 7f8554ac5700 1 -- 172.21.15.16:0/4047960686 shutdown_connections 2023-10-16T12:29:24.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:24.201+0000 7f8554ac5700 1 -- 172.21.15.16:0/4047960686 wait complete. 2023-10-16T12:29:24.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:24.202+0000 7f8554ac5700 1 Processor -- start 2023-10-16T12:29:24.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:24.202+0000 7f8554ac5700 1 -- start start 2023-10-16T12:29:24.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:24.203+0000 7f8554ac5700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f85500ce5d0 0x7f8550159c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:24.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:24.203+0000 7f8554ac5700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f855015ec80 0x7f85501577e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:24.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:24.203+0000 7f8554ac5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f855015f060 0x7f85501527e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:24.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:24.203+0000 7f8554ac5700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85500c57c0 con 0x7f855015f060 2023-10-16T12:29:24.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:24.203+0000 7f8554ac5700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f855015a140 con 0x7f85500ce5d0 2023-10-16T12:29:24.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:24.203+0000 7f8554ac5700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8550157d20 con 0x7f855015ec80 2023-10-16T12:29:24.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:24.203+0000 7f854ed9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f855015f060 0x7f85501527e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:24.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:24.203+0000 7f854ed9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f855015f060 0x7f85501527e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43966/0 (socket says 172.21.15.16:43966) 2023-10-16T12:29:24.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:24.203+0000 7f854ed9d700 1 -- 172.21.15.16:0/2862628249 learned_addr learned my addr 172.21.15.16:0/2862628249 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:29:24.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:24.203+0000 7f854e59c700 1 --2- 172.21.15.16:0/2862628249 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f85500ce5d0 0x7f8550159c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:24.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:24.203+0000 7f854dd9b700 1 --2- 172.21.15.16:0/2862628249 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f855015ec80 0x7f85501577e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:24.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:24.204+0000 7f854ed9d700 1 -- 172.21.15.16:0/2862628249 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f855015ec80 msgr2=0x7f85501577e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:24.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:24.204+0000 7f854ed9d700 1 --2- 172.21.15.16:0/2862628249 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f855015ec80 0x7f85501577e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:24.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:24.204+0000 7f854ed9d700 1 -- 172.21.15.16:0/2862628249 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f85500ce5d0 msgr2=0x7f8550159c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:24.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:24.204+0000 7f854ed9d700 1 --2- 172.21.15.16:0/2862628249 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f85500ce5d0 0x7f8550159c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:24.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:24.204+0000 7f854ed9d700 1 -- 172.21.15.16:0/2862628249 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85400094a0 con 0x7f855015f060 2023-10-16T12:29:24.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:24.204+0000 7f854ed9d700 1 --2- 172.21.15.16:0/2862628249 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f855015f060 0x7f85501527e0 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f854800cb20 tx=0x7f854800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:24.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:24.204+0000 7f853effd700 1 -- 172.21.15.16:0/2862628249 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8548003c00 con 0x7f855015f060 2023-10-16T12:29:24.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:24.204+0000 7f8554ac5700 1 -- 172.21.15.16:0/2862628249 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f855015a920 con 0x7f855015f060 2023-10-16T12:29:24.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:24.204+0000 7f853effd700 1 -- 172.21.15.16:0/2862628249 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8548003d60 con 0x7f855015f060 2023-10-16T12:29:24.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:24.204+0000 7f853effd700 1 -- 172.21.15.16:0/2862628249 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f854800a980 con 0x7f855015f060 2023-10-16T12:29:24.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:24.204+0000 7f8554ac5700 1 -- 172.21.15.16:0/2862628249 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f855015a360 con 0x7f855015f060 2023-10-16T12:29:24.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:24.205+0000 7f8554ac5700 1 -- 172.21.15.16:0/2862628249 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85500c5900 con 0x7f855015f060 2023-10-16T12:29:24.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:24.206+0000 7f853effd700 1 -- 172.21.15.16:0/2862628249 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 62) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f854800aae0 con 0x7f855015f060 2023-10-16T12:29:24.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:24.206+0000 7f853effd700 1 --2- 172.21.15.16:0/2862628249 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f853006b020 0x7f853006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:24.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:24.206+0000 7f853effd700 1 -- 172.21.15.16:0/2862628249 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f85480896d0 con 0x7f855015f060 2023-10-16T12:29:24.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:24.206+0000 7f854e59c700 1 --2- 172.21.15.16:0/2862628249 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f853006b020 0x7f853006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:24.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:24.207+0000 7f854e59c700 1 --2- 172.21.15.16:0/2862628249 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f853006b020 0x7f853006d4e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8544001eb0 tx=0x7f8544001e40 comp rx=0 tx=0).ready entity=mgr.7884 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:24.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:24.211+0000 7f853effd700 1 -- 172.21.15.16:0/2862628249 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f85500c5900 con 0x7f855015f060 2023-10-16T12:29:24.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:24.475+0000 7f8554ac5700 1 -- 172.21.15.16:0/2862628249 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f85500c5900 con 0x7f855015f060 2023-10-16T12:29:25.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.286+0000 7f853effd700 1 -- 172.21.15.16:0/2862628249 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v63) v1 ==== 92+0+0 (secure 0 0 0) 0x7f85500c5900 con 0x7f855015f060 2023-10-16T12:29:25.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.291+0000 7f8554ac5700 1 -- 172.21.15.16:0/2862628249 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f853006b020 msgr2=0x7f853006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:25.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.291+0000 7f8554ac5700 1 --2- 172.21.15.16:0/2862628249 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f853006b020 0x7f853006d4e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8544001eb0 tx=0x7f8544001e40 comp rx=0 tx=0).stop 2023-10-16T12:29:25.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.291+0000 7f8554ac5700 1 -- 172.21.15.16:0/2862628249 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f855015f060 msgr2=0x7f85501527e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:25.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.291+0000 7f8554ac5700 1 --2- 172.21.15.16:0/2862628249 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f855015f060 0x7f85501527e0 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f854800cb20 tx=0x7f854800ce30 comp rx=0 tx=0).stop 2023-10-16T12:29:25.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.293+0000 7f8554ac5700 1 -- 172.21.15.16:0/2862628249 shutdown_connections 2023-10-16T12:29:25.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.293+0000 7f8554ac5700 1 --2- 172.21.15.16:0/2862628249 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f855015f060 0x7f85501527e0 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:25.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.293+0000 7f8554ac5700 1 --2- 172.21.15.16:0/2862628249 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f85500ce5d0 0x7f8550159c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:25.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.293+0000 7f8554ac5700 1 --2- 172.21.15.16:0/2862628249 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f853006b020 0x7f853006d4e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:25.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.293+0000 7f8554ac5700 1 --2- 172.21.15.16:0/2862628249 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f855015ec80 0x7f85501577e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:25.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.293+0000 7f8554ac5700 1 -- 172.21.15.16:0/2862628249 >> 172.21.15.16:0/2862628249 conn(0x7f85500d15f0 msgr2=0x7f85500cd480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:25.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.293+0000 7f8554ac5700 1 -- 172.21.15.16:0/2862628249 shutdown_connections 2023-10-16T12:29:25.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.293+0000 7f8554ac5700 1 -- 172.21.15.16:0/2862628249 wait complete. 2023-10-16T12:29:25.312 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-16T12:29:25.440 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:29:25.438+0000 7fe838360200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T12:29:25.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.447+0000 7f7ab0149700 1 Processor -- start 2023-10-16T12:29:25.450 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:29:25.447+0000 7f16b8626200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T12:29:25.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.448+0000 7f7ab0149700 1 -- start start 2023-10-16T12:29:25.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.448+0000 7f7ab0149700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f7aa815a640 0x7f7aa80cb710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:25.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.448+0000 7f7ab0149700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7aa80c6770 0x7f7aa80cbc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:25.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.448+0000 7f7ab0149700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7aa815aa20 con 0x7f7aa815e140 2023-10-16T12:29:25.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.448+0000 7f7ab0149700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7aa806eb40 con 0x7f7aa815a640 2023-10-16T12:29:25.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.448+0000 7f7ab0149700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa806ee20 con 0x7f7aa80c6770 2023-10-16T12:29:25.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.449+0000 7f7aad6e4700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7aa80c6770 0x7f7aa80cbc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:25.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.449+0000 7f7aadee5700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f7aa815a640 0x7f7aa80cb710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:25.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.449+0000 7f7aae6e6700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7f7aa815e140 0x7f7aa80ca1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:40458/0 (socket says 172.21.15.16:40458) 2023-10-16T12:29:25.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.449+0000 7f7aae6e6700 1 -- 172.21.15.16:0/3350737662 learned_addr learned my addr 172.21.15.16:0/3350737662 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:29:25.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.449+0000 7f7aacee3700 1 -- 172.21.15.16:0/3350737662 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1250570571 0 0) 0x7f7aa0001490 con 0x7f7aa815e140 2023-10-16T12:29:25.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.449+0000 7f7aacee3700 1 -- 172.21.15.16:0/3350737662 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a940035b0 con 0x7f7aa815e140 2023-10-16T12:29:25.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.449+0000 7f7aadee5700 1 -- 172.21.15.16:0/3350737662 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7aa80c6770 msgr2=0x7f7aa80cbc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:29:25.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.449+0000 7f7aadee5700 1 --2- 172.21.15.16:0/3350737662 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7aa80c6770 0x7f7aa80cbc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:25.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.449+0000 7f7aadee5700 1 -- 172.21.15.16:0/3350737662 >> v1:172.21.15.49:6789/0 conn(0x7f7aa815e140 legacy=0x7f7aa80ca1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:25.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.449+0000 7f7aadee5700 1 -- 172.21.15.16:0/3350737662 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aa806f100 con 0x7f7aa815a640 2023-10-16T12:29:25.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.449+0000 7f7aad6e4700 1 --2- 172.21.15.16:0/3350737662 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7aa80c6770 0x7f7aa80cbc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:29:25.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.449+0000 7f7aadee5700 1 --2- 172.21.15.16:0/3350737662 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7aa815a640 0x7f7aa80cb710 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f7aa4009840 tx=0x7f7aa4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c6edc6cd2d7ccea server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:25.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.449+0000 7f7aacee3700 1 -- 172.21.15.16:0/3350737662 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa4011830 con 0x7f7aa815a640 2023-10-16T12:29:25.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.450+0000 7f7aacee3700 1 -- 172.21.15.16:0/3350737662 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7aa4011990 con 0x7f7aa815a640 2023-10-16T12:29:25.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.450+0000 7f7aacee3700 1 -- 172.21.15.16:0/3350737662 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa4011d20 con 0x7f7aa815a640 2023-10-16T12:29:25.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.450+0000 7f7ab0149700 1 -- 172.21.15.16:0/3350737662 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7aa815a640 msgr2=0x7f7aa80cb710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:25.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.450+0000 7f7ab0149700 1 --2- 172.21.15.16:0/3350737662 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7aa815a640 0x7f7aa80cb710 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f7aa4009840 tx=0x7f7aa4009b50 comp rx=0 tx=0).stop 2023-10-16T12:29:25.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.450+0000 7f7ab0149700 1 -- 172.21.15.16:0/3350737662 shutdown_connections 2023-10-16T12:29:25.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.450+0000 7f7ab0149700 1 --2- 172.21.15.16:0/3350737662 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7aa815a640 0x7f7aa80cb710 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:25.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.450+0000 7f7ab0149700 1 --2- 172.21.15.16:0/3350737662 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7aa80c6770 0x7f7aa80cbc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:25.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.450+0000 7f7ab0149700 1 -- 172.21.15.16:0/3350737662 >> 172.21.15.16:0/3350737662 conn(0x7f7aa80cc5e0 msgr2=0x7f7aa80cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:25.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.450+0000 7f7ab0149700 1 -- 172.21.15.16:0/3350737662 shutdown_connections 2023-10-16T12:29:25.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.450+0000 7f7ab0149700 1 -- 172.21.15.16:0/3350737662 wait complete. 2023-10-16T12:29:25.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.451+0000 7f7ab0149700 1 Processor -- start 2023-10-16T12:29:25.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.452+0000 7f7ab0149700 1 -- start start 2023-10-16T12:29:25.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.452+0000 7f7ab0149700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7aa80c6770 0x7f7aa8154140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:25.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.452+0000 7f7ab0149700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7aa815a640 0x7f7aa80c7100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:25.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.452+0000 7f7ab0149700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7aa815e140 0x7f7aa80c7640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:25.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.452+0000 7f7ab0149700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa80c8940 con 0x7f7aa815a640 2023-10-16T12:29:25.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.452+0000 7f7ab0149700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa80af2d0 con 0x7f7aa815e140 2023-10-16T12:29:25.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.452+0000 7f7ab0149700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa80c8d60 con 0x7f7aa80c6770 2023-10-16T12:29:25.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.452+0000 7f7aad6e4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7aa815a640 0x7f7aa80c7100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:25.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.452+0000 7f7aad6e4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7aa815a640 0x7f7aa80c7100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44006/0 (socket says 172.21.15.16:44006) 2023-10-16T12:29:25.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.452+0000 7f7aadee5700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7aa80c6770 0x7f7aa8154140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:25.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.452+0000 7f7aad6e4700 1 -- 172.21.15.16:0/1281741067 learned_addr learned my addr 172.21.15.16:0/1281741067 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:29:25.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.452+0000 7f7aae6e6700 1 --2- 172.21.15.16:0/1281741067 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7aa815e140 0x7f7aa80c7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:25.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.453+0000 7f7aadee5700 1 -- 172.21.15.16:0/1281741067 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7aa815e140 msgr2=0x7f7aa80c7640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:25.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.453+0000 7f7aadee5700 1 --2- 172.21.15.16:0/1281741067 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7aa815e140 0x7f7aa80c7640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:25.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.453+0000 7f7aadee5700 1 -- 172.21.15.16:0/1281741067 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7aa815a640 msgr2=0x7f7aa80c7100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:25.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.453+0000 7f7aadee5700 1 --2- 172.21.15.16:0/1281741067 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7aa815a640 0x7f7aa80c7100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:25.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.453+0000 7f7aadee5700 1 -- 172.21.15.16:0/1281741067 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a980094a0 con 0x7f7aa80c6770 2023-10-16T12:29:25.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.453+0000 7f7aad6e4700 1 --2- 172.21.15.16:0/1281741067 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7aa815a640 0x7f7aa80c7100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:29:25.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.453+0000 7f7aadee5700 1 --2- 172.21.15.16:0/1281741067 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7aa80c6770 0x7f7aa8154140 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f7aa4016000 tx=0x7f7aa4004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:25.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.453+0000 7f7a9e7fc700 1 -- 172.21.15.16:0/1281741067 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa40036a0 con 0x7f7aa80c6770 2023-10-16T12:29:25.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.453+0000 7f7a9e7fc700 1 -- 172.21.15.16:0/1281741067 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7aa401f070 con 0x7f7aa80c6770 2023-10-16T12:29:25.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.453+0000 7f7a9e7fc700 1 -- 172.21.15.16:0/1281741067 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa4003a60 con 0x7f7aa80c6770 2023-10-16T12:29:25.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.453+0000 7f7ab0149700 1 -- 172.21.15.16:0/1281741067 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7aa40094a0 con 0x7f7aa80c6770 2023-10-16T12:29:25.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.453+0000 7f7ab0149700 1 -- 172.21.15.16:0/1281741067 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7aa80c7fa0 con 0x7f7aa80c6770 2023-10-16T12:29:25.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.454+0000 7f7ab0149700 1 -- 172.21.15.16:0/1281741067 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7aa80c8a80 con 0x7f7aa80c6770 2023-10-16T12:29:25.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.455+0000 7f7a9e7fc700 1 -- 172.21.15.16:0/1281741067 <== mon.2 v2:172.21.15.49:3301/0 4 ==== mgrmap(e 63) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f7aa4003bc0 con 0x7f7aa80c6770 2023-10-16T12:29:25.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.456+0000 7f7a9e7fc700 1 --2- 172.21.15.16:0/1281741067 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f7a8c06b070 0x7f7a8c06d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:25.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.456+0000 7f7aad6e4700 1 -- 172.21.15.16:0/1281741067 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f7a8c06b070 msgr2=0x7f7a8c06d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2309613638 2023-10-16T12:29:25.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.456+0000 7f7a9e7fc700 1 -- 172.21.15.16:0/1281741067 <== mon.2 v2:172.21.15.49:3301/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f7aa4021070 con 0x7f7aa80c6770 2023-10-16T12:29:25.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.456+0000 7f7aad6e4700 1 --2- 172.21.15.16:0/1281741067 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f7a8c06b070 0x7f7a8c06d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:29:25.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.460+0000 7f7a9e7fc700 1 -- 172.21.15.16:0/1281741067 <== mon.2 v2:172.21.15.49:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7aa80c8a80 con 0x7f7aa80c6770 2023-10-16T12:29:25.510 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:29:25.509+0000 7fe838360200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T12:29:25.521 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:29:25.520+0000 7f16b8626200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T12:29:25.585 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:29:25.583+0000 7fe838360200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T12:29:25.597 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:29:25.595+0000 7f16b8626200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T12:29:25.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.657+0000 7f7aad6e4700 1 -- 172.21.15.16:0/1281741067 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f7a8c06b070 msgr2=0x7f7a8c06d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2309613638 2023-10-16T12:29:25.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.657+0000 7f7aad6e4700 1 --2- 172.21.15.16:0/1281741067 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f7a8c06b070 0x7f7a8c06d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:29:25.666 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:29:25.664+0000 7fe838360200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T12:29:25.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.667+0000 7f7ab0149700 1 -- 172.21.15.16:0/1281741067 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f7aa80c8a80 con 0x7f7aa80c6770 2023-10-16T12:29:25.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.671+0000 7f7a9e7fc700 1 -- 172.21.15.16:0/1281741067 <== mon.2 v2:172.21.15.49:3301/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v41)=0 v41) v1 ==== 138+0+0 (secure 0 0 0) 0x7f7aa80c8a80 con 0x7f7aa80c6770 2023-10-16T12:29:25.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.673+0000 7f7ab0149700 1 -- 172.21.15.16:0/1281741067 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f7a8c06b070 msgr2=0x7f7a8c06d530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:29:25.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.673+0000 7f7ab0149700 1 --2- 172.21.15.16:0/1281741067 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f7a8c06b070 0x7f7a8c06d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:25.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.673+0000 7f7ab0149700 1 -- 172.21.15.16:0/1281741067 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7aa80c6770 msgr2=0x7f7aa8154140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:25.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.673+0000 7f7ab0149700 1 --2- 172.21.15.16:0/1281741067 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7aa80c6770 0x7f7aa8154140 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f7aa4016000 tx=0x7f7aa4004750 comp rx=0 tx=0).stop 2023-10-16T12:29:25.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.673+0000 7f7ab0149700 1 -- 172.21.15.16:0/1281741067 shutdown_connections 2023-10-16T12:29:25.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.673+0000 7f7ab0149700 1 --2- 172.21.15.16:0/1281741067 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7aa815e140 0x7f7aa80c7640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:25.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.673+0000 7f7ab0149700 1 --2- 172.21.15.16:0/1281741067 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f7a8c06b070 0x7f7a8c06d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:25.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.673+0000 7f7ab0149700 1 --2- 172.21.15.16:0/1281741067 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7aa80c6770 0x7f7aa8154140 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:25.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.673+0000 7f7ab0149700 1 --2- 172.21.15.16:0/1281741067 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7aa815a640 0x7f7aa80c7100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:25.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.673+0000 7f7ab0149700 1 -- 172.21.15.16:0/1281741067 >> 172.21.15.16:0/1281741067 conn(0x7f7aa80cc5e0 msgr2=0x7f7aa8155fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:25.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.673+0000 7f7ab0149700 1 -- 172.21.15.16:0/1281741067 shutdown_connections 2023-10-16T12:29:25.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.674+0000 7f7ab0149700 1 -- 172.21.15.16:0/1281741067 wait complete. 2023-10-16T12:29:25.681 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:29:25.679+0000 7f16b8626200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T12:29:25.688 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-16T12:29:25.815 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:29:25.812+0000 7fe838360200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T12:29:25.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.816+0000 7f1013ed2700 1 Processor -- start 2023-10-16T12:29:25.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.816+0000 7f1013ed2700 1 -- start start 2023-10-16T12:29:25.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.817+0000 7f1013ed2700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f100c0cb880 0x7f100c0cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:25.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.817+0000 7f1013ed2700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f100c06da60 0x7f100c06def0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:25.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.817+0000 7f1013ed2700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f100c0cef70 con 0x7f100c15b070 2023-10-16T12:29:25.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.817+0000 7f1013ed2700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f100c06e430 con 0x7f100c0cb880 2023-10-16T12:29:25.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.817+0000 7f1013ed2700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f100c0ce360 con 0x7f100c06da60 2023-10-16T12:29:25.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.817+0000 7f101146d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f100c0cb880 0x7f100c0cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:25.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.817+0000 7f101146d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f100c0cb880 0x7f100c0cbd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44060/0 (socket says 172.21.15.16:44060) 2023-10-16T12:29:25.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.817+0000 7f101146d700 1 -- 172.21.15.16:0/1093158044 learned_addr learned my addr 172.21.15.16:0/1093158044 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:29:25.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.817+0000 7f101246f700 1 --2- 172.21.15.16:0/1093158044 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f100c06da60 0x7f100c06def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:25.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.817+0000 7f101146d700 1 -- 172.21.15.16:0/1093158044 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f100c06da60 msgr2=0x7f100c06def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:29:25.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.817+0000 7f101146d700 1 --2- 172.21.15.16:0/1093158044 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f100c06da60 0x7f100c06def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:25.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.817+0000 7f101146d700 1 -- 172.21.15.16:0/1093158044 >> v1:172.21.15.49:6789/0 conn(0x7f100c15b070 legacy=0x7f100c15b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:25.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.817+0000 7f101146d700 1 -- 172.21.15.16:0/1093158044 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f100c0ce640 con 0x7f100c0cb880 2023-10-16T12:29:25.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.817+0000 7f1010c6c700 1 -- 172.21.15.16:0/1093158044 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 885873839 0 0) 0x7f100c0cef70 con 0x7f100c15b070 2023-10-16T12:29:25.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.817+0000 7f101246f700 1 --2- 172.21.15.16:0/1093158044 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f100c06da60 0x7f100c06def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:29:25.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.817+0000 7f101146d700 1 --2- 172.21.15.16:0/1093158044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f100c0cb880 0x7f100c0cbd00 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc0098f0 tx=0x7f0ffc009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=958d66eb168c65e0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:25.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.818+0000 7f1010c6c700 1 -- 172.21.15.16:0/1093158044 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ffc011830 con 0x7f100c0cb880 2023-10-16T12:29:25.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.818+0000 7f1010c6c700 1 -- 172.21.15.16:0/1093158044 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ffc011990 con 0x7f100c0cb880 2023-10-16T12:29:25.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.818+0000 7f1010c6c700 1 -- 172.21.15.16:0/1093158044 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ffc011d20 con 0x7f100c0cb880 2023-10-16T12:29:25.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.818+0000 7f1013ed2700 1 -- 172.21.15.16:0/1093158044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f100c0cb880 msgr2=0x7f100c0cbd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:25.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.818+0000 7f1013ed2700 1 --2- 172.21.15.16:0/1093158044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f100c0cb880 0x7f100c0cbd00 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc0098f0 tx=0x7f0ffc009c00 comp rx=0 tx=0).stop 2023-10-16T12:29:25.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.818+0000 7f1013ed2700 1 -- 172.21.15.16:0/1093158044 shutdown_connections 2023-10-16T12:29:25.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.818+0000 7f1013ed2700 1 --2- 172.21.15.16:0/1093158044 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f100c06da60 0x7f100c06def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:25.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.818+0000 7f1013ed2700 1 --2- 172.21.15.16:0/1093158044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f100c0cb880 0x7f100c0cbd00 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:25.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.818+0000 7f1013ed2700 1 -- 172.21.15.16:0/1093158044 >> 172.21.15.16:0/1093158044 conn(0x7f100c0b1f40 msgr2=0x7f100c0b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:25.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.818+0000 7f1013ed2700 1 -- 172.21.15.16:0/1093158044 shutdown_connections 2023-10-16T12:29:25.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.818+0000 7f1013ed2700 1 -- 172.21.15.16:0/1093158044 wait complete. 2023-10-16T12:29:25.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.819+0000 7f1013ed2700 1 Processor -- start 2023-10-16T12:29:25.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.819+0000 7f1013ed2700 1 -- start start 2023-10-16T12:29:25.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.820+0000 7f1013ed2700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f100c06da60 0x7f100c155190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:25.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.820+0000 7f1013ed2700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f100c0cb880 0x7f100c0c52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:25.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.820+0000 7f1013ed2700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f100c15b070 0x7f100c152e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:25.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.820+0000 7f1013ed2700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f100c0c6d00 con 0x7f100c0cb880 2023-10-16T12:29:25.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.820+0000 7f1013ed2700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f100c0c6a20 con 0x7f100c15b070 2023-10-16T12:29:25.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.820+0000 7f1013ed2700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f100c0c75f0 con 0x7f100c06da60 2023-10-16T12:29:25.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.820+0000 7f101146d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f100c0cb880 0x7f100c0c52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:25.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.820+0000 7f101146d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f100c0cb880 0x7f100c0c52e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44066/0 (socket says 172.21.15.16:44066) 2023-10-16T12:29:25.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.820+0000 7f101146d700 1 -- 172.21.15.16:0/1101831732 learned_addr learned my addr 172.21.15.16:0/1101831732 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:29:25.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.820+0000 7f1011c6e700 1 --2- 172.21.15.16:0/1101831732 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f100c06da60 0x7f100c155190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:25.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.820+0000 7f101246f700 1 --2- 172.21.15.16:0/1101831732 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f100c15b070 0x7f100c152e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:25.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.821+0000 7f101146d700 1 -- 172.21.15.16:0/1101831732 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f100c06da60 msgr2=0x7f100c155190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:25.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.821+0000 7f101146d700 1 --2- 172.21.15.16:0/1101831732 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f100c06da60 0x7f100c155190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:25.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.821+0000 7f101146d700 1 -- 172.21.15.16:0/1101831732 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f100c15b070 msgr2=0x7f100c152e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:25.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.821+0000 7f101146d700 1 --2- 172.21.15.16:0/1101831732 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f100c15b070 0x7f100c152e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:25.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.821+0000 7f101146d700 1 -- 172.21.15.16:0/1101831732 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1004009a70 con 0x7f100c0cb880 2023-10-16T12:29:25.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.821+0000 7f101146d700 1 --2- 172.21.15.16:0/1101831732 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f100c0cb880 0x7f100c0c52e0 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc009c40 tx=0x7f0ffc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:25.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.821+0000 7f10027fc700 1 -- 172.21.15.16:0/1101831732 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ffc0045d0 con 0x7f100c0cb880 2023-10-16T12:29:25.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.821+0000 7f10027fc700 1 -- 172.21.15.16:0/1101831732 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ffc0036a0 con 0x7f100c0cb880 2023-10-16T12:29:25.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.821+0000 7f10027fc700 1 -- 172.21.15.16:0/1101831732 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ffc003a30 con 0x7f100c0cb880 2023-10-16T12:29:25.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.821+0000 7f1013ed2700 1 -- 172.21.15.16:0/1101831732 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ffc009550 con 0x7f100c0cb880 2023-10-16T12:29:25.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.821+0000 7f1013ed2700 1 -- 172.21.15.16:0/1101831732 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f100c0c7bb0 con 0x7f100c0cb880 2023-10-16T12:29:25.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.822+0000 7f1013ed2700 1 -- 172.21.15.16:0/1101831732 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f100c0c7d70 con 0x7f100c0cb880 2023-10-16T12:29:25.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.823+0000 7f10027fc700 1 -- 172.21.15.16:0/1101831732 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 63) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f0ffc003b90 con 0x7f100c0cb880 2023-10-16T12:29:25.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.823+0000 7f10027fc700 1 --2- 172.21.15.16:0/1101831732 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f0ff006b070 0x7f0ff006d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:25.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.823+0000 7f1011c6e700 1 -- 172.21.15.16:0/1101831732 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f0ff006b070 msgr2=0x7f0ff006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2309613638 2023-10-16T12:29:25.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.823+0000 7f10027fc700 1 -- 172.21.15.16:0/1101831732 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f0ffc08cc00 con 0x7f100c0cb880 2023-10-16T12:29:25.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.824+0000 7f1011c6e700 1 --2- 172.21.15.16:0/1101831732 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f0ff006b070 0x7f0ff006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:29:25.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:25.828+0000 7f10027fc700 1 -- 172.21.15.16:0/1101831732 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0ffc093080 con 0x7f100c0cb880 2023-10-16T12:29:25.838 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:29:25.836+0000 7f16b8626200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T12:29:26.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.025+0000 7f1011c6e700 1 -- 172.21.15.16:0/1101831732 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f0ff006b070 msgr2=0x7f0ff006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2309613638 2023-10-16T12:29:26.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.025+0000 7f1011c6e700 1 --2- 172.21.15.16:0/1101831732 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f0ff006b070 0x7f0ff006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:29:26.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.033+0000 7f1013ed2700 1 -- 172.21.15.16:0/1101831732 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f100c04ebb0 con 0x7f100c0cb880 2023-10-16T12:29:26.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.036+0000 7f10027fc700 1 -- 172.21.15.16:0/1101831732 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v42)=0 v42) v1 ==== 133+0+0 (secure 0 0 0) 0x7f0ffc015020 con 0x7f100c0cb880 2023-10-16T12:29:26.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.038+0000 7f1013ed2700 1 -- 172.21.15.16:0/1101831732 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f0ff006b070 msgr2=0x7f0ff006d530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:29:26.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.038+0000 7f1013ed2700 1 --2- 172.21.15.16:0/1101831732 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f0ff006b070 0x7f0ff006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:26.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.038+0000 7f1013ed2700 1 -- 172.21.15.16:0/1101831732 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f100c0cb880 msgr2=0x7f100c0c52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:26.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.038+0000 7f1013ed2700 1 --2- 172.21.15.16:0/1101831732 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f100c0cb880 0x7f100c0c52e0 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc009c40 tx=0x7f0ffc004750 comp rx=0 tx=0).stop 2023-10-16T12:29:26.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.038+0000 7f1013ed2700 1 -- 172.21.15.16:0/1101831732 shutdown_connections 2023-10-16T12:29:26.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.038+0000 7f1013ed2700 1 --2- 172.21.15.16:0/1101831732 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f100c15b070 0x7f100c152e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:26.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.039+0000 7f1013ed2700 1 --2- 172.21.15.16:0/1101831732 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7f0ff006b070 0x7f0ff006d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:26.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.039+0000 7f1013ed2700 1 --2- 172.21.15.16:0/1101831732 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f100c06da60 0x7f100c155190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:26.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.039+0000 7f1013ed2700 1 --2- 172.21.15.16:0/1101831732 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f100c0cb880 0x7f100c0c52e0 unknown :-1 s=CLOSED pgs=1409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:26.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.039+0000 7f1013ed2700 1 -- 172.21.15.16:0/1101831732 >> 172.21.15.16:0/1101831732 conn(0x7f100c0b1f40 msgr2=0x7f100c0b1670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:26.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.039+0000 7f1013ed2700 1 -- 172.21.15.16:0/1101831732 shutdown_connections 2023-10-16T12:29:26.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.039+0000 7f1013ed2700 1 -- 172.21.15.16:0/1101831732 wait complete. 2023-10-16T12:29:26.125 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:29:26.126 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-10-16T12:29:26.146 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1m 2023-10-16T12:29:26.155 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:29:26.153+0000 7fe838360200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T12:29:26.184 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:29:26.183+0000 7f16b8626200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T12:29:26.275 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:29:26.273+0000 7fe838360200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T12:29:26.309 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:29:26.308+0000 7f16b8626200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T12:29:26.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.314+0000 7fa0f9e9b700 1 Processor -- start 2023-10-16T12:29:26.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.314+0000 7fa0f9e9b700 1 -- start start 2023-10-16T12:29:26.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.315+0000 7fa0f9e9b700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fa0f40d4060 0x7fa0f40d1520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:26.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.315+0000 7fa0f9e9b700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0f40d4540 con 0x7fa0f415da10 2023-10-16T12:29:26.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.315+0000 7fa0f9e9b700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0f40ce7e0 con 0x7fa0f40d1d20 2023-10-16T12:29:26.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.315+0000 7fa0f9e9b700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0f40cf9b0 con 0x7fa0f40d4060 2023-10-16T12:29:26.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.315+0000 7fa0f37fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa0f415da10 0x7fa0f40cf2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33392/0 (socket says 172.21.15.16:33392) 2023-10-16T12:29:26.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.315+0000 7fa0f37fe700 1 -- 172.21.15.16:0/2509038549 learned_addr learned my addr 172.21.15.16:0/2509038549 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:29:26.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.315+0000 7fa0f2ffd700 1 --2- 172.21.15.16:0/2509038549 >> v2:172.21.15.49:3300/0 conn(0x7fa0f40d4060 0x7fa0f40d1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:26.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.315+0000 7fa0f27fc700 1 -- 172.21.15.16:0/2509038549 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4143196012 0 0) 0x7fa0f40d4540 con 0x7fa0f415da10 2023-10-16T12:29:26.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.315+0000 7fa0f27fc700 1 -- 172.21.15.16:0/2509038549 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0dc0035b0 con 0x7fa0f415da10 2023-10-16T12:29:26.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.315+0000 7fa0f27fc700 1 -- 172.21.15.16:0/2509038549 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 25274015 0 0) 0x7fa0f40ce7e0 con 0x7fa0f40d1d20 2023-10-16T12:29:26.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.315+0000 7fa0f27fc700 1 -- 172.21.15.16:0/2509038549 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0f40d4540 con 0x7fa0f40d1d20 2023-10-16T12:29:26.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.315+0000 7fa0f2ffd700 1 -- 172.21.15.16:0/2509038549 >> v1:172.21.15.49:6789/0 conn(0x7fa0f40d1d20 legacy=0x7fa0f406b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:26.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.315+0000 7fa0f2ffd700 1 -- 172.21.15.16:0/2509038549 >> v1:172.21.15.16:6789/0 conn(0x7fa0f415da10 legacy=0x7fa0f40cf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:26.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.315+0000 7fa0f2ffd700 1 -- 172.21.15.16:0/2509038549 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0f40ceac0 con 0x7fa0f40d4060 2023-10-16T12:29:26.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.316+0000 7fa0f2ffd700 1 --2- 172.21.15.16:0/2509038549 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa0f40d4060 0x7fa0f40d1520 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fa0e8009840 tx=0x7fa0e8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1b2b83491ec2ed45 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:26.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.316+0000 7fa0f27fc700 1 -- 172.21.15.16:0/2509038549 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0e8011830 con 0x7fa0f40d4060 2023-10-16T12:29:26.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.316+0000 7fa0f27fc700 1 -- 172.21.15.16:0/2509038549 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0e8011990 con 0x7fa0f40d4060 2023-10-16T12:29:26.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.316+0000 7fa0f27fc700 1 -- 172.21.15.16:0/2509038549 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0e8011d20 con 0x7fa0f40d4060 2023-10-16T12:29:26.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.316+0000 7fa0f9e9b700 1 -- 172.21.15.16:0/2509038549 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa0f40d4060 msgr2=0x7fa0f40d1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:26.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.316+0000 7fa0f9e9b700 1 --2- 172.21.15.16:0/2509038549 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa0f40d4060 0x7fa0f40d1520 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fa0e8009840 tx=0x7fa0e8009b50 comp rx=0 tx=0).stop 2023-10-16T12:29:26.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.316+0000 7fa0f9e9b700 1 -- 172.21.15.16:0/2509038549 shutdown_connections 2023-10-16T12:29:26.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.316+0000 7fa0f9e9b700 1 --2- 172.21.15.16:0/2509038549 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa0f40d4060 0x7fa0f40d1520 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:26.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.316+0000 7fa0f9e9b700 1 -- 172.21.15.16:0/2509038549 >> 172.21.15.16:0/2509038549 conn(0x7fa0f409c9f0 msgr2=0x7fa0f40d3740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:26.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.316+0000 7fa0f9e9b700 1 -- 172.21.15.16:0/2509038549 shutdown_connections 2023-10-16T12:29:26.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.316+0000 7fa0f9e9b700 1 -- 172.21.15.16:0/2509038549 wait complete. 2023-10-16T12:29:26.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.318+0000 7fa0f9e9b700 1 Processor -- start 2023-10-16T12:29:26.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.318+0000 7fa0f9e9b700 1 -- start start 2023-10-16T12:29:26.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.319+0000 7fa0f9e9b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa0f40d1d20 0x7fa0f4157cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:26.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.319+0000 7fa0f9e9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0f415da10 0x7fa0f40b3170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:26.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.319+0000 7fa0f9e9b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa0f414ae00 0x7fa0f40b3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:26.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.319+0000 7fa0f9e9b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f40b36b0 con 0x7fa0f415da10 2023-10-16T12:29:26.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.319+0000 7fa0f9e9b700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f40ccea0 con 0x7fa0f414ae00 2023-10-16T12:29:26.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.319+0000 7fa0f9e9b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f415fc90 con 0x7fa0f40d1d20 2023-10-16T12:29:26.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.319+0000 7fa0f2ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0f415da10 0x7fa0f40b3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:26.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.319+0000 7fa0f2ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0f415da10 0x7fa0f40b3170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44126/0 (socket says 172.21.15.16:44126) 2023-10-16T12:29:26.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.319+0000 7fa0f2ffd700 1 -- 172.21.15.16:0/4091482752 learned_addr learned my addr 172.21.15.16:0/4091482752 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:29:26.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.319+0000 7fa0f3fff700 1 --2- 172.21.15.16:0/4091482752 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa0f414ae00 0x7fa0f40b3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:26.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.319+0000 7fa0f37fe700 1 --2- 172.21.15.16:0/4091482752 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa0f40d1d20 0x7fa0f4157cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:26.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.319+0000 7fa0f2ffd700 1 -- 172.21.15.16:0/4091482752 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa0f40d1d20 msgr2=0x7fa0f4157cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:26.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.319+0000 7fa0f2ffd700 1 --2- 172.21.15.16:0/4091482752 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa0f40d1d20 0x7fa0f4157cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:26.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.319+0000 7fa0f2ffd700 1 -- 172.21.15.16:0/4091482752 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa0f414ae00 msgr2=0x7fa0f40b3dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:26.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.319+0000 7fa0f2ffd700 1 --2- 172.21.15.16:0/4091482752 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa0f414ae00 0x7fa0f40b3dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:26.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.319+0000 7fa0f2ffd700 1 -- 172.21.15.16:0/4091482752 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0e80094a0 con 0x7fa0f415da10 2023-10-16T12:29:26.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.319+0000 7fa0f2ffd700 1 --2- 172.21.15.16:0/4091482752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0f415da10 0x7fa0f40b3170 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7fa0e8009fd0 tx=0x7fa0e8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:26.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.319+0000 7fa0dbfff700 1 -- 172.21.15.16:0/4091482752 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0e801a8b0 con 0x7fa0f415da10 2023-10-16T12:29:26.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.319+0000 7fa0dbfff700 1 -- 172.21.15.16:0/4091482752 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0e801aa10 con 0x7fa0f415da10 2023-10-16T12:29:26.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.319+0000 7fa0dbfff700 1 -- 172.21.15.16:0/4091482752 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0e801ada0 con 0x7fa0f415da10 2023-10-16T12:29:26.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.319+0000 7fa0f9e9b700 1 -- 172.21.15.16:0/4091482752 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0f40cc2d0 con 0x7fa0f415da10 2023-10-16T12:29:26.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.320+0000 7fa0f9e9b700 1 -- 172.21.15.16:0/4091482752 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0f40cc850 con 0x7fa0f415da10 2023-10-16T12:29:26.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.321+0000 7fa0f9e9b700 1 -- 172.21.15.16:0/4091482752 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0f40cd030 con 0x7fa0f415da10 2023-10-16T12:29:26.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.321+0000 7fa0dbfff700 1 -- 172.21.15.16:0/4091482752 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 63) v1 ==== 89348+0+0 (secure 0 0 0) 0x7fa0e80243f0 con 0x7fa0f415da10 2023-10-16T12:29:26.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.322+0000 7fa0dbfff700 1 --2- 172.21.15.16:0/4091482752 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fa0e006b070 0x7fa0e006d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:26.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.322+0000 7fa0f37fe700 1 -- 172.21.15.16:0/4091482752 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fa0e006b070 msgr2=0x7fa0e006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2309613638 2023-10-16T12:29:26.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.322+0000 7fa0dbfff700 1 -- 172.21.15.16:0/4091482752 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fa0e808ada0 con 0x7fa0f415da10 2023-10-16T12:29:26.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.322+0000 7fa0f37fe700 1 --2- 172.21.15.16:0/4091482752 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fa0e006b070 0x7fa0e006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:29:26.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.327+0000 7fa0dbfff700 1 -- 172.21.15.16:0/4091482752 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa0e8055d90 con 0x7fa0f415da10 2023-10-16T12:29:26.353 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:29:26.352+0000 7fe838360200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T12:29:26.389 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:29:26.388+0000 7f16b8626200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T12:29:26.438 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:29:26.436+0000 7fe838360200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T12:29:26.474 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:29:26.472+0000 7f16b8626200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T12:29:26.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.523+0000 7fa0f37fe700 1 -- 172.21.15.16:0/4091482752 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fa0e006b070 msgr2=0x7fa0e006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2309613638 2023-10-16T12:29:26.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.523+0000 7fa0f37fe700 1 --2- 172.21.15.16:0/4091482752 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fa0e006b070 0x7fa0e006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:29:26.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.558+0000 7fa0f9e9b700 1 -- 172.21.15.16:0/4091482752 --> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fa0f40b2d30 con 0x7fa0e006b070 2023-10-16T12:29:26.823 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:29:26.821+0000 7fe838360200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T12:29:26.862 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:29:26.860+0000 7f16b8626200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T12:29:26.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.924+0000 7fa0f37fe700 1 -- 172.21.15.16:0/4091482752 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fa0e006b070 msgr2=0x7fa0e006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2309613638 2023-10-16T12:29:26.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:26.924+0000 7fa0f37fe700 1 --2- 172.21.15.16:0/4091482752 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fa0e006b070 0x7fa0e006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-16T12:29:27.428 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:29:27.426+0000 7fe838360200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T12:29:27.471 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:29:27.469+0000 7f16b8626200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T12:29:27.500 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:29:27.498+0000 7fe838360200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T12:29:27.544 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:29:27.542+0000 7f16b8626200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T12:29:27.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:27.725+0000 7fa0f37fe700 1 -- 172.21.15.16:0/4091482752 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fa0e006b070 msgr2=0x7fa0e006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2309613638 2023-10-16T12:29:27.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:27.725+0000 7fa0f37fe700 1 --2- 172.21.15.16:0/4091482752 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fa0e006b070 0x7fa0e006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-16T12:29:28.161 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:29:28.159+0000 7fe838360200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:29:28.210 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:29:28.208+0000 7f16b8626200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:29:28.241 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:29:28.240+0000 7fe838360200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T12:29:28.291 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:29:28.290+0000 7f16b8626200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T12:29:28.392 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:29:28.390+0000 7fe838360200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:29:28.444 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:29:28.442+0000 7f16b8626200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:29:28.629 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:29:28.627+0000 7fe838360200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T12:29:28.683 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:29:28.682+0000 7f16b8626200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T12:29:28.705 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:29:28.703+0000 7fe838360200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T12:29:28.761 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:29:28.760+0000 7f16b8626200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T12:29:28.979 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:29:28.977+0000 7fe838360200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T12:29:29.038 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:29:29.036+0000 7f16b8626200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T12:29:29.208 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:29:29.206+0000 7fe838360200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T12:29:29.272 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:29:29.270+0000 7f16b8626200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T12:29:29.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:29.326+0000 7fa0f37fe700 1 -- 172.21.15.16:0/4091482752 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fa0e006b070 msgr2=0x7fa0e006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2309613638 2023-10-16T12:29:29.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:29.326+0000 7fa0f37fe700 1 --2- 172.21.15.16:0/4091482752 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fa0e006b070 0x7fa0e006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-10-16T12:29:29.363 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:29:29.361+0000 7fe838360200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T12:29:29.430 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:29:29.428+0000 7f16b8626200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T12:29:29.482 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:29:29.480+0000 7fe838360200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T12:29:29.551 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:29:29.549+0000 7f16b8626200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T12:29:29.643 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:29:29.641+0000 7fe838360200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T12:29:29.716 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:29:29.714+0000 7f16b8626200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T12:29:29.913 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:29:29.911+0000 7fe838360200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T12:29:29.989 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:29:29.987+0000 7f16b8626200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T12:29:30.079 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:29:30.077+0000 7fe838360200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T12:29:30.159 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:29:30.157+0000 7f16b8626200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T12:29:30.631 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:29:30.629+0000 7fe838360200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T12:29:30.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:30.698+0000 7fa0dbfff700 1 -- 172.21.15.16:0/4091482752 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 64) v1 ==== 89348+0+0 (secure 0 0 0) 0x7fa0e8056160 con 0x7fa0f415da10 2023-10-16T12:29:30.720 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:29:30.718+0000 7f16b8626200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T12:29:30.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:30.740+0000 7fa0dbfff700 1 -- 172.21.15.16:0/4091482752 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mgrmap(e 65) v1 ==== 88957+0+0 (secure 0 0 0) 0x7fa0e8088de0 con 0x7fa0f415da10 2023-10-16T12:29:30.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:30.740+0000 7fa0dbfff700 1 -- 172.21.15.16:0/4091482752 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fa0e006b070 msgr2=0x7fa0e006d530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:29:30.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:30.740+0000 7fa0dbfff700 1 --2- 172.21.15.16:0/4091482752 >> [v2:172.21.15.16:6832/2309613638,v1:172.21.15.16:6833/2309613638] conn(0x7fa0e006b070 0x7fa0e006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:30.892 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:29:30.889+0000 7f162b601700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:29:30.892 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:29:30.889+0000 7f162b601700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:29:30.892 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:29:30.889+0000 7f162b601700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:29:30.892 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:29:30.889+0000 7f162b601700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:29:30.893 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:29:30.889+0000 7f162b601700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:29:30.893 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:29:30.889+0000 7f162e607700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:29:30.893 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:29:30.889+0000 7f162e607700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:29:30.893 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:29:30.889+0000 7f162e607700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:29:30.893 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:29:30.889+0000 7f162e607700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:29:30.894 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:29:30.889+0000 7f162e607700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:29:31.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.741+0000 7fa0dbfff700 1 -- 172.21.15.16:0/4091482752 <== mon.0 v2:172.21.15.16:3300/0 9 ==== mgrmap(e 66) v1 ==== 89084+0+0 (secure 0 0 0) 0x7fa0e8056a30 con 0x7fa0f415da10 2023-10-16T12:29:31.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.741+0000 7fa0dbfff700 1 --2- 172.21.15.16:0/4091482752 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7fa0e001d250 0x7fa0e00754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:31.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.741+0000 7fa0dbfff700 1 -- 172.21.15.16:0/4091482752 --> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fa0e001d630 con 0x7fa0e001d250 2023-10-16T12:29:31.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.742+0000 7fa0f37fe700 1 --2- 172.21.15.16:0/4091482752 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7fa0e001d250 0x7fa0e00754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:31.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.745+0000 7fa0f37fe700 1 --2- 172.21.15.16:0/4091482752 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7fa0e001d250 0x7fa0e00754a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fa0e4001000 tx=0x7fa0e4007c70 comp rx=0 tx=0).ready entity=mgr.7852 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:31.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.809+0000 7fa0dbfff700 1 -- 172.21.15.16:0/4091482752 <== mgr.7852 v2:172.21.15.16:6832/2550537276 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fa0e001d630 con 0x7fa0e001d250 2023-10-16T12:29:31.812 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule set for path snap_test_dir1 2023-10-16T12:29:31.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.811+0000 7fa0f9e9b700 1 -- 172.21.15.16:0/4091482752 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7fa0e001d250 msgr2=0x7fa0e00754a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:31.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.811+0000 7fa0f9e9b700 1 --2- 172.21.15.16:0/4091482752 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7fa0e001d250 0x7fa0e00754a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fa0e4001000 tx=0x7fa0e4007c70 comp rx=0 tx=0).stop 2023-10-16T12:29:31.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.811+0000 7fa0f9e9b700 1 -- 172.21.15.16:0/4091482752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0f415da10 msgr2=0x7fa0f40b3170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:31.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.811+0000 7fa0f9e9b700 1 --2- 172.21.15.16:0/4091482752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0f415da10 0x7fa0f40b3170 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7fa0e8009fd0 tx=0x7fa0e8004750 comp rx=0 tx=0).stop 2023-10-16T12:29:31.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.811+0000 7fa0f3fff700 1 -- 172.21.15.16:0/4091482752 reap_dead start 2023-10-16T12:29:31.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.811+0000 7fa0f9e9b700 1 -- 172.21.15.16:0/4091482752 shutdown_connections 2023-10-16T12:29:31.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.811+0000 7fa0f9e9b700 1 -- 172.21.15.16:0/4091482752 >> 172.21.15.16:0/4091482752 conn(0x7fa0f409c9f0 msgr2=0x7fa0f415e100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:31.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.811+0000 7fa0f9e9b700 1 -- 172.21.15.16:0/4091482752 shutdown_connections 2023-10-16T12:29:31.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.811+0000 7fa0f9e9b700 1 -- 172.21.15.16:0/4091482752 wait complete. 2023-10-16T12:29:31.827 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2023-10-16T12:29:31.827 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2023-10-16-12_30_00 in ~29s... 2023-10-16T12:29:31.828 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m'], expected_retention: [] 2023-10-16T12:29:31.828 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json snap_test_dir1 2023-10-16T12:29:31.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.957+0000 7f482fbbf700 1 Processor -- start 2023-10-16T12:29:31.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.958+0000 7f482fbbf700 1 -- start start 2023-10-16T12:29:31.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.958+0000 7f482fbbf700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f482806e570 0x7f482815e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:31.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.958+0000 7f482fbbf700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f482806a7e0 0x7f482806ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:31.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.958+0000 7f482fbbf700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f482815b6a0 con 0x7f482815b2c0 2023-10-16T12:29:31.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.958+0000 7f482fbbf700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f482806b1b0 con 0x7f482806a7e0 2023-10-16T12:29:31.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.958+0000 7f482fbbf700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4828161b20 con 0x7f482806e570 2023-10-16T12:29:31.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.959+0000 7f482d15a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f482806e570 0x7f482815e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:31.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.959+0000 7f482e15c700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f482806a7e0 0x7f482806ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:31.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.959+0000 7f482d15a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f482806e570 0x7f482815e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:60792/0 (socket says 172.21.15.16:60792) 2023-10-16T12:29:31.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.959+0000 7f482e15c700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f482806a7e0 0x7f482806ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:43366/0 (socket says 172.21.15.16:43366) 2023-10-16T12:29:31.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.959+0000 7f482d15a700 1 -- 172.21.15.16:0/2760677685 learned_addr learned my addr 172.21.15.16:0/2760677685 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:29:31.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.959+0000 7f482d95b700 1 --1- 172.21.15.16:0/2760677685 >> v1:172.21.15.49:6789/0 conn(0x7f482815b2c0 0x7f4828161340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:40756/0 (socket says 172.21.15.16:40756) 2023-10-16T12:29:31.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.959+0000 7f482d15a700 1 -- 172.21.15.16:0/2760677685 >> v2:172.21.15.49:3300/0 conn(0x7f482806a7e0 msgr2=0x7f482806ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:29:31.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.959+0000 7f482d15a700 1 --2- 172.21.15.16:0/2760677685 >> v2:172.21.15.49:3300/0 conn(0x7f482806a7e0 0x7f482806ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:31.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.959+0000 7f482d15a700 1 -- 172.21.15.16:0/2760677685 >> v1:172.21.15.49:6789/0 conn(0x7f482815b2c0 legacy=0x7f4828161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:31.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.959+0000 7f482d15a700 1 -- 172.21.15.16:0/2760677685 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4828161e00 con 0x7f482806e570 2023-10-16T12:29:31.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.959+0000 7f482c959700 1 -- 172.21.15.16:0/2760677685 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3631456453 0 0) 0x7f482815b6a0 con 0x7f482815b2c0 2023-10-16T12:29:31.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.959+0000 7f482e15c700 1 --2- 172.21.15.16:0/2760677685 >> v2:172.21.15.49:3300/0 conn(0x7f482806a7e0 0x7f482806ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:29:31.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.959+0000 7f482d15a700 1 --2- 172.21.15.16:0/2760677685 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f482806e570 0x7f482815e5a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f48240098f0 tx=0x7f4824009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=816723e42f305c2f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:31.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.960+0000 7f482c959700 1 -- 172.21.15.16:0/2760677685 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4824011830 con 0x7f482806e570 2023-10-16T12:29:31.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.960+0000 7f482c959700 1 -- 172.21.15.16:0/2760677685 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4824011990 con 0x7f482806e570 2023-10-16T12:29:31.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.960+0000 7f482c959700 1 -- 172.21.15.16:0/2760677685 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4824011d20 con 0x7f482806e570 2023-10-16T12:29:31.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.960+0000 7f482fbbf700 1 -- 172.21.15.16:0/2760677685 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f482806e570 msgr2=0x7f482815e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:31.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.960+0000 7f482fbbf700 1 --2- 172.21.15.16:0/2760677685 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f482806e570 0x7f482815e5a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f48240098f0 tx=0x7f4824009c00 comp rx=0 tx=0).stop 2023-10-16T12:29:31.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.960+0000 7f482fbbf700 1 -- 172.21.15.16:0/2760677685 shutdown_connections 2023-10-16T12:29:31.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.960+0000 7f482fbbf700 1 --2- 172.21.15.16:0/2760677685 >> v2:172.21.15.49:3300/0 conn(0x7f482806a7e0 0x7f482806ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:31.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.960+0000 7f482fbbf700 1 --2- 172.21.15.16:0/2760677685 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f482806e570 0x7f482815e5a0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:31.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.960+0000 7f482fbbf700 1 -- 172.21.15.16:0/2760677685 >> 172.21.15.16:0/2760677685 conn(0x7f48280b1f40 msgr2=0x7f48280b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:31.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.960+0000 7f482fbbf700 1 -- 172.21.15.16:0/2760677685 shutdown_connections 2023-10-16T12:29:31.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.960+0000 7f482fbbf700 1 -- 172.21.15.16:0/2760677685 wait complete. 2023-10-16T12:29:31.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.961+0000 7f482fbbf700 1 Processor -- start 2023-10-16T12:29:31.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.961+0000 7f482fbbf700 1 -- start start 2023-10-16T12:29:31.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.962+0000 7f482fbbf700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f482806a7e0 0x7f48281f27d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:31.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.962+0000 7f482fbbf700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f482806e570 0x7f48280c3e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:31.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.962+0000 7f482fbbf700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f482815b2c0 0x7f48280c4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:31.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.962+0000 7f482fbbf700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f482815f370 con 0x7f482806a7e0 2023-10-16T12:29:31.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.962+0000 7f482fbbf700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48280b5fa0 con 0x7f482806e570 2023-10-16T12:29:31.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.962+0000 7f482fbbf700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f482815e4a0 con 0x7f482815b2c0 2023-10-16T12:29:31.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.962+0000 7f482d95b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f482806a7e0 0x7f48281f27d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:31.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.962+0000 7f482d95b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f482806a7e0 0x7f48281f27d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44306/0 (socket says 172.21.15.16:44306) 2023-10-16T12:29:31.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.962+0000 7f482d95b700 1 -- 172.21.15.16:0/2924499485 learned_addr learned my addr 172.21.15.16:0/2924499485 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:29:31.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.962+0000 7f482d15a700 1 --2- 172.21.15.16:0/2924499485 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f482806e570 0x7f48280c3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:31.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.962+0000 7f482e15c700 1 --2- 172.21.15.16:0/2924499485 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f482815b2c0 0x7f48280c4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:31.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.963+0000 7f482d95b700 1 -- 172.21.15.16:0/2924499485 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f482815b2c0 msgr2=0x7f48280c4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:31.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.963+0000 7f482d95b700 1 --2- 172.21.15.16:0/2924499485 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f482815b2c0 0x7f48280c4350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:31.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.963+0000 7f482d95b700 1 -- 172.21.15.16:0/2924499485 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f482806e570 msgr2=0x7f48280c3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:31.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.963+0000 7f482d95b700 1 --2- 172.21.15.16:0/2924499485 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f482806e570 0x7f48280c3e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:31.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.963+0000 7f482d95b700 1 -- 172.21.15.16:0/2924499485 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4820009a70 con 0x7f482806a7e0 2023-10-16T12:29:31.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.963+0000 7f482e15c700 1 --2- 172.21.15.16:0/2924499485 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f482815b2c0 0x7f48280c4350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:29:31.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.963+0000 7f482d15a700 1 --2- 172.21.15.16:0/2924499485 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f482806e570 0x7f48280c3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:29:31.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.963+0000 7f482d95b700 1 --2- 172.21.15.16:0/2924499485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f482806a7e0 0x7f48281f27d0 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f481c0011e0 tx=0x7f481c00ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:31.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.963+0000 7f481a7fc700 1 -- 172.21.15.16:0/2924499485 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f481c004c80 con 0x7f482806a7e0 2023-10-16T12:29:31.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.963+0000 7f481a7fc700 1 -- 172.21.15.16:0/2924499485 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f481c004de0 con 0x7f482806a7e0 2023-10-16T12:29:31.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.963+0000 7f482fbbf700 1 -- 172.21.15.16:0/2924499485 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4824009550 con 0x7f482806a7e0 2023-10-16T12:29:31.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.963+0000 7f481a7fc700 1 -- 172.21.15.16:0/2924499485 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f481c007310 con 0x7f482806a7e0 2023-10-16T12:29:31.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.963+0000 7f482fbbf700 1 -- 172.21.15.16:0/2924499485 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f482815e9d0 con 0x7f482806a7e0 2023-10-16T12:29:31.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.964+0000 7f482fbbf700 1 -- 172.21.15.16:0/2924499485 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f482815f4b0 con 0x7f482806a7e0 2023-10-16T12:29:31.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.964+0000 7f481a7fc700 1 -- 172.21.15.16:0/2924499485 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 66) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f481c007470 con 0x7f482806a7e0 2023-10-16T12:29:31.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.965+0000 7f481a7fc700 1 --2- 172.21.15.16:0/2924499485 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7f480c06ada0 0x7f480c06d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:29:31.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.965+0000 7f481a7fc700 1 -- 172.21.15.16:0/2924499485 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f481c0891c0 con 0x7f482806a7e0 2023-10-16T12:29:31.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.965+0000 7f482d15a700 1 --2- 172.21.15.16:0/2924499485 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7f480c06ada0 0x7f480c06d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:29:31.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.965+0000 7f482d15a700 1 --2- 172.21.15.16:0/2924499485 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7f480c06ada0 0x7f480c06d260 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f4824009c40 tx=0x7f4824008030 comp rx=0 tx=0).ready entity=mgr.7852 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:29:31.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:31.970+0000 7f481a7fc700 1 -- 172.21.15.16:0/2924499485 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f481c054220 con 0x7f482806a7e0 2023-10-16T12:29:32.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:32.200+0000 7f482fbbf700 1 -- 172.21.15.16:0/2924499485 --> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f482815e080 con 0x7f480c06ada0 2023-10-16T12:29:32.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:32.209+0000 7f481a7fc700 1 -- 172.21.15.16:0/2924499485 <== mgr.7852 v2:172.21.15.16:6832/2550537276 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7f4824011b80 con 0x7f480c06ada0 2023-10-16T12:29:32.212 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:29:32.212 INFO:teuthology.orchestra.run.smithi016.stdout:{"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{}]} 2023-10-16T12:29:32.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:32.212+0000 7f482fbbf700 1 -- 172.21.15.16:0/2924499485 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7f480c06ada0 msgr2=0x7f480c06d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:32.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:32.212+0000 7f482fbbf700 1 --2- 172.21.15.16:0/2924499485 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7f480c06ada0 0x7f480c06d260 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f4824009c40 tx=0x7f4824008030 comp rx=0 tx=0).stop 2023-10-16T12:29:32.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:32.212+0000 7f482fbbf700 1 -- 172.21.15.16:0/2924499485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f482806a7e0 msgr2=0x7f48281f27d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:29:32.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:32.212+0000 7f482fbbf700 1 --2- 172.21.15.16:0/2924499485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f482806a7e0 0x7f48281f27d0 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f481c0011e0 tx=0x7f481c00ca30 comp rx=0 tx=0).stop 2023-10-16T12:29:32.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:32.212+0000 7f482fbbf700 1 -- 172.21.15.16:0/2924499485 shutdown_connections 2023-10-16T12:29:32.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:32.212+0000 7f482fbbf700 1 --2- 172.21.15.16:0/2924499485 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7f480c06ada0 0x7f480c06d260 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:32.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:32.212+0000 7f482fbbf700 1 --2- 172.21.15.16:0/2924499485 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f482815b2c0 0x7f48280c4350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:32.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:32.212+0000 7f482fbbf700 1 --2- 172.21.15.16:0/2924499485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f482806a7e0 0x7f48281f27d0 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:32.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:32.212+0000 7f482fbbf700 1 --2- 172.21.15.16:0/2924499485 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f482806e570 0x7f48280c3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:29:32.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:32.212+0000 7f482fbbf700 1 -- 172.21.15.16:0/2924499485 >> 172.21.15.16:0/2924499485 conn(0x7f48280b1f40 msgr2=0x7f4828159210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:29:32.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:32.212+0000 7f482fbbf700 1 -- 172.21.15.16:0/2924499485 shutdown_connections 2023-10-16T12:29:32.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:29:32.212+0000 7f482fbbf700 1 -- 172.21.15.16:0/2924499485 wait complete. 2023-10-16T12:29:32.226 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{}]} 2023-10-16T12:29:32.227 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1m'], 'retention': [{}]} 2023-10-16T12:29:32.227 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:29:32.227 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:29:32.246 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:29:32.247 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:29:32.247 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:29:33.249 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:29:33.249 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:29:33.273 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:29:33.273 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:29:33.274 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:29:34.275 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:29:34.275 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:29:34.297 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:29:34.298 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:29:34.298 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:29:35.299 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:29:35.300 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:29:35.322 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:29:35.322 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:29:35.322 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:29:36.324 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:29:36.324 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:29:36.347 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:29:36.347 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:29:36.348 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:29:37.349 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:29:37.350 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:29:37.372 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:29:37.372 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:29:37.373 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:29:38.374 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:29:38.375 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:29:38.397 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:29:38.397 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:29:38.397 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:29:39.399 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:29:39.399 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:29:39.422 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:29:39.422 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:29:39.423 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:29:40.424 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:29:40.424 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:29:40.447 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:29:40.447 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:29:40.448 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:29:41.449 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:29:41.449 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:29:41.472 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:29:41.473 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:29:41.473 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:29:42.474 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:29:42.475 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:29:42.497 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:29:42.498 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:29:42.498 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:29:43.499 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:29:43.500 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:29:43.522 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:29:43.523 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:29:43.523 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:29:44.524 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:29:44.525 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:29:44.547 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:29:44.548 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:29:44.548 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:29:45.549 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:29:45.550 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:29:45.572 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:29:45.573 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:29:45.573 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:29:46.574 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:29:46.574 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:29:46.596 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:29:46.597 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:29:46.597 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:29:47.599 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:29:47.599 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:29:47.622 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:29:47.622 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:29:47.623 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:29:48.623 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:29:48.624 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:29:48.647 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:29:48.647 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:29:48.647 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:29:49.648 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:29:49.649 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:29:49.671 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:29:49.672 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:29:49.672 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:29:50.673 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:29:50.673 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:29:50.696 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:29:50.696 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:29:50.697 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:29:51.229 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:29:51.233 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:29:51.698 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:29:51.698 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:29:51.720 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:29:51.721 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:29:51.721 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:29:52.722 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:29:52.722 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:29:52.745 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:29:52.746 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:29:52.746 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:29:53.748 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:29:53.748 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:29:53.771 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:29:53.771 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:29:53.771 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:29:54.772 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:29:54.773 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:29:54.796 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:29:54.796 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:29:54.796 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:29:55.797 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:29:55.798 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:29:55.820 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:29:55.821 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:29:55.821 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:29:56.822 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:29:56.823 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:29:56.845 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:29:56.845 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:29:56.846 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:29:57.847 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:29:57.848 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:29:57.870 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:29:57.871 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:29:57.871 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:29:58.872 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:29:58.872 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:29:58.895 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:29:58.895 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:29:58.896 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:29:59.897 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:29:59.898 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:29:59.920 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T12:29:59.920 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T12:29:59.921 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:30:00.821 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:00.819+0000 7f161dde6700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:30:00.822 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:00.819+0000 7f161dde6700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:30:00.822 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:00.819+0000 7f161dde6700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:30:00.822 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:00.819+0000 7f161dde6700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:30:00.822 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:00.819+0000 7f161dde6700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:30:00.922 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:30:00.923 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:30:00.941 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_30_00_UTC 2023-10-16T12:30:00.942 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-12_30_00_UTC'} 2023-10-16T12:30:00.942 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2023-10-16-12_30_00_UTC'} 2023-10-16T12:30:00.943 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T12:30:00.943 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2023-10-16-12_30_00_UTC'] 2023-10-16T12:30:00.943 DEBUG:tasks.cephfs.test_snap_schedules:exec=1697459371.8277152, now = 1697459400.94385, timo = 29 2023-10-16T12:30:01.945 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2023-10-16T12:30:02.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.086+0000 7f432272b700 1 Processor -- start 2023-10-16T12:30:02.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.087+0000 7f432272b700 1 -- start start 2023-10-16T12:30:02.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.087+0000 7f432272b700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f431c15ec80 0x7f431c0cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:02.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.087+0000 7f432272b700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f431c0b2440 con 0x7f431c0d2110 2023-10-16T12:30:02.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.087+0000 7f432272b700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f431c0d00e0 con 0x7f431c15f060 2023-10-16T12:30:02.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.087+0000 7f432272b700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f431c0b2c50 con 0x7f431c15ec80 2023-10-16T12:30:02.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.088+0000 7f4320cc8700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f431c0d2110 0x7f431c0b4b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33616/0 (socket says 172.21.15.16:33616) 2023-10-16T12:30:02.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.088+0000 7f4320cc8700 1 -- 172.21.15.16:0/3199119693 learned_addr learned my addr 172.21.15.16:0/3199119693 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:30:02.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.088+0000 7f431affd700 1 -- 172.21.15.16:0/3199119693 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2772920643 0 0) 0x7f431c0b2440 con 0x7f431c0d2110 2023-10-16T12:30:02.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.088+0000 7f431bfff700 1 --2- 172.21.15.16:0/3199119693 >> v2:172.21.15.49:3300/0 conn(0x7f431c15ec80 0x7f431c0cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:02.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.088+0000 7f431affd700 1 -- 172.21.15.16:0/3199119693 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43080035b0 con 0x7f431c0d2110 2023-10-16T12:30:02.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.088+0000 7f431affd700 1 -- 172.21.15.16:0/3199119693 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 943908804 0 0) 0x7f43080035b0 con 0x7f431c0d2110 2023-10-16T12:30:02.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.088+0000 7f431affd700 1 -- 172.21.15.16:0/3199119693 >> v2:172.21.15.49:3300/0 conn(0x7f431c15ec80 msgr2=0x7f431c0cdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:02.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.088+0000 7f431affd700 1 --2- 172.21.15.16:0/3199119693 >> v2:172.21.15.49:3300/0 conn(0x7f431c15ec80 0x7f431c0cdec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:02.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.088+0000 7f431affd700 1 -- 172.21.15.16:0/3199119693 >> v1:172.21.15.49:6789/0 conn(0x7f431c15f060 legacy=0x7f431c0b4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:02.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.088+0000 7f431affd700 1 -- 172.21.15.16:0/3199119693 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f431c0cef60 con 0x7f431c0d2110 2023-10-16T12:30:02.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.088+0000 7f431affd700 1 -- 172.21.15.16:0/3199119693 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f43140019f0 con 0x7f431c0d2110 2023-10-16T12:30:02.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.088+0000 7f431affd700 1 -- 172.21.15.16:0/3199119693 >> v1:172.21.15.16:6789/0 conn(0x7f431c0d2110 legacy=0x7f431c0b4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:02.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.088+0000 7f431affd700 1 --2- 172.21.15.16:0/3199119693 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4308003a00 0x7f4308003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:02.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.088+0000 7f431affd700 1 --2- 172.21.15.16:0/3199119693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4308004850 0x7f4308004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:02.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.088+0000 7f431affd700 1 --2- 172.21.15.16:0/3199119693 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4308005210 0x7f430800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:02.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.088+0000 7f431affd700 1 -- 172.21.15.16:0/3199119693 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f431c0b2c50 con 0x7f4308004850 2023-10-16T12:30:02.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.088+0000 7f431affd700 1 -- 172.21.15.16:0/3199119693 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f430800c0b0 con 0x7f4308005210 2023-10-16T12:30:02.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.088+0000 7f431affd700 1 -- 172.21.15.16:0/3199119693 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f430800c270 con 0x7f4308003a00 2023-10-16T12:30:02.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.088+0000 7f431b7fe700 1 --2- 172.21.15.16:0/3199119693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4308004850 0x7f4308004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:02.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.089+0000 7f431bfff700 1 --2- 172.21.15.16:0/3199119693 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4308003a00 0x7f4308003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:02.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.089+0000 7f4320cc8700 1 --2- 172.21.15.16:0/3199119693 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4308005210 0x7f430800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:02.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.089+0000 7f431b7fe700 1 -- 172.21.15.16:0/3199119693 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4308003a00 msgr2=0x7f4308003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:02.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.089+0000 7f431b7fe700 1 --2- 172.21.15.16:0/3199119693 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4308003a00 0x7f4308003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:02.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.089+0000 7f431b7fe700 1 -- 172.21.15.16:0/3199119693 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4308005210 msgr2=0x7f430800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:02.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.089+0000 7f431b7fe700 1 --2- 172.21.15.16:0/3199119693 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4308005210 0x7f430800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:02.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.089+0000 7f4320cc8700 1 -- 172.21.15.16:0/3199119693 reap_dead start 2023-10-16T12:30:02.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.089+0000 7f431b7fe700 1 -- 172.21.15.16:0/3199119693 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43080035b0 con 0x7f4308004850 2023-10-16T12:30:02.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.089+0000 7f431b7fe700 1 --2- 172.21.15.16:0/3199119693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4308004850 0x7f4308004cd0 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f431c15f460 tx=0x7f4308005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1208b2dcaa0add63 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:02.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.089+0000 7f431affd700 1 -- 172.21.15.16:0/3199119693 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43040052d0 con 0x7f4308004850 2023-10-16T12:30:02.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.089+0000 7f431affd700 1 -- 172.21.15.16:0/3199119693 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4304005450 con 0x7f4308004850 2023-10-16T12:30:02.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.089+0000 7f431affd700 1 -- 172.21.15.16:0/3199119693 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43040038c0 con 0x7f4308004850 2023-10-16T12:30:02.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.090+0000 7f432272b700 1 -- 172.21.15.16:0/3199119693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4308004850 msgr2=0x7f4308004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:02.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.090+0000 7f432272b700 1 --2- 172.21.15.16:0/3199119693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4308004850 0x7f4308004cd0 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f431c15f460 tx=0x7f4308005880 comp rx=0 tx=0).stop 2023-10-16T12:30:02.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.090+0000 7f432272b700 1 -- 172.21.15.16:0/3199119693 shutdown_connections 2023-10-16T12:30:02.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.090+0000 7f432272b700 1 --2- 172.21.15.16:0/3199119693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4308004850 0x7f4308004cd0 unknown :-1 s=CLOSED pgs=1429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:02.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.090+0000 7f432272b700 1 -- 172.21.15.16:0/3199119693 >> 172.21.15.16:0/3199119693 conn(0x7f431c0d15f0 msgr2=0x7f431c0d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:02.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.090+0000 7f432272b700 1 -- 172.21.15.16:0/3199119693 shutdown_connections 2023-10-16T12:30:02.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.090+0000 7f432272b700 1 -- 172.21.15.16:0/3199119693 wait complete. 2023-10-16T12:30:02.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.091+0000 7f432272b700 1 Processor -- start 2023-10-16T12:30:02.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.092+0000 7f432272b700 1 -- start start 2023-10-16T12:30:02.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.092+0000 7f432272b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4308004850 0x7f431c155030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:02.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.092+0000 7f432272b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f431c0cb560 0x7f431c158310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:02.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.092+0000 7f432272b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f431c1511a0 0x7f431c151620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:02.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.092+0000 7f432272b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f431c158850 con 0x7f431c1511a0 2023-10-16T12:30:02.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.092+0000 7f432272b700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f431c154590 con 0x7f4308004850 2023-10-16T12:30:02.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.092+0000 7f432272b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f431c158db0 con 0x7f431c0cb560 2023-10-16T12:30:02.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.093+0000 7f4320cc8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f431c1511a0 0x7f431c151620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:02.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.093+0000 7f4320cc8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f431c1511a0 0x7f431c151620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44356/0 (socket says 172.21.15.16:44356) 2023-10-16T12:30:02.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.093+0000 7f4320cc8700 1 -- 172.21.15.16:0/3894864395 learned_addr learned my addr 172.21.15.16:0/3894864395 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:02.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.093+0000 7f431b7fe700 1 --2- 172.21.15.16:0/3894864395 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f431c0cb560 0x7f431c158310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:02.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.093+0000 7f431bfff700 1 --2- 172.21.15.16:0/3894864395 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4308004850 0x7f431c155030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:02.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.093+0000 7f4320cc8700 1 -- 172.21.15.16:0/3894864395 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f431c0cb560 msgr2=0x7f431c158310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:02.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.093+0000 7f4320cc8700 1 --2- 172.21.15.16:0/3894864395 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f431c0cb560 0x7f431c158310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:02.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.093+0000 7f4320cc8700 1 -- 172.21.15.16:0/3894864395 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4308004850 msgr2=0x7f431c155030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:02.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.093+0000 7f4320cc8700 1 --2- 172.21.15.16:0/3894864395 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4308004850 0x7f431c155030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:02.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.093+0000 7f4320cc8700 1 -- 172.21.15.16:0/3894864395 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f431c158ad0 con 0x7f431c1511a0 2023-10-16T12:30:02.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.093+0000 7f431bfff700 1 --2- 172.21.15.16:0/3894864395 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4308004850 0x7f431c155030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:30:02.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.093+0000 7f4320cc8700 1 --2- 172.21.15.16:0/3894864395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f431c1511a0 0x7f431c151620 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f431c06ea20 tx=0x7f431400a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:02.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.093+0000 7f4318ff9700 1 -- 172.21.15.16:0/3894864395 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f431400ade0 con 0x7f431c1511a0 2023-10-16T12:30:02.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.093+0000 7f4318ff9700 1 -- 172.21.15.16:0/3894864395 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43140133f0 con 0x7f431c1511a0 2023-10-16T12:30:02.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.093+0000 7f432272b700 1 -- 172.21.15.16:0/3894864395 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f431c154830 con 0x7f431c1511a0 2023-10-16T12:30:02.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.093+0000 7f4318ff9700 1 -- 172.21.15.16:0/3894864395 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4314013760 con 0x7f431c1511a0 2023-10-16T12:30:02.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.093+0000 7f432272b700 1 -- 172.21.15.16:0/3894864395 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f431c15a250 con 0x7f431c1511a0 2023-10-16T12:30:02.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.094+0000 7f432272b700 1 -- 172.21.15.16:0/3894864395 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f431c1543f0 con 0x7f431c1511a0 2023-10-16T12:30:02.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.095+0000 7f4318ff9700 1 -- 172.21.15.16:0/3894864395 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 69) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f43140138c0 con 0x7f431c1511a0 2023-10-16T12:30:02.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.095+0000 7f4318ff9700 1 --2- 172.21.15.16:0/3894864395 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7f42fc06b1c0 0x7f42fc06d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:02.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.095+0000 7f4318ff9700 1 -- 172.21.15.16:0/3894864395 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f431408a590 con 0x7f431c1511a0 2023-10-16T12:30:02.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.095+0000 7f431bfff700 1 --2- 172.21.15.16:0/3894864395 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7f42fc06b1c0 0x7f42fc06d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:02.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.096+0000 7f431bfff700 1 --2- 172.21.15.16:0/3894864395 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7f42fc06b1c0 0x7f42fc06d680 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f430c009ae0 tx=0x7f430c008040 comp rx=0 tx=0).ready entity=mgr.7852 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:02.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.100+0000 7f4318ff9700 1 -- 172.21.15.16:0/3894864395 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f431c1543f0 con 0x7f431c1511a0 2023-10-16T12:30:02.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.332+0000 7f432272b700 1 -- 172.21.15.16:0/3894864395 --> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f431c0b2970 con 0x7f42fc06b1c0 2023-10-16T12:30:02.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.396+0000 7f4318ff9700 1 -- 172.21.15.16:0/3894864395 <== mgr.7852 v2:172.21.15.16:6832/2550537276 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f430c0137d0 con 0x7f42fc06b1c0 2023-10-16T12:30:02.399 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule removed for path snap_test_dir1 2023-10-16T12:30:02.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.399+0000 7f432272b700 1 -- 172.21.15.16:0/3894864395 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7f42fc06b1c0 msgr2=0x7f42fc06d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:02.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.399+0000 7f432272b700 1 --2- 172.21.15.16:0/3894864395 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7f42fc06b1c0 0x7f42fc06d680 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f430c009ae0 tx=0x7f430c008040 comp rx=0 tx=0).stop 2023-10-16T12:30:02.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.399+0000 7f432272b700 1 -- 172.21.15.16:0/3894864395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f431c1511a0 msgr2=0x7f431c151620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:02.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.399+0000 7f432272b700 1 --2- 172.21.15.16:0/3894864395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f431c1511a0 0x7f431c151620 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f431c06ea20 tx=0x7f431400a5f0 comp rx=0 tx=0).stop 2023-10-16T12:30:02.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.399+0000 7f432272b700 1 -- 172.21.15.16:0/3894864395 shutdown_connections 2023-10-16T12:30:02.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.399+0000 7f432272b700 1 --2- 172.21.15.16:0/3894864395 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7f42fc06b1c0 0x7f42fc06d680 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:02.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.399+0000 7f432272b700 1 --2- 172.21.15.16:0/3894864395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f431c1511a0 0x7f431c151620 unknown :-1 s=CLOSED pgs=1430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:02.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.399+0000 7f432272b700 1 --2- 172.21.15.16:0/3894864395 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4308004850 0x7f431c155030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:02.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.399+0000 7f432272b700 1 --2- 172.21.15.16:0/3894864395 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f431c0cb560 0x7f431c158310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:02.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.399+0000 7f432272b700 1 -- 172.21.15.16:0/3894864395 >> 172.21.15.16:0/3894864395 conn(0x7f431c0d15f0 msgr2=0x7f431c153710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:02.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.399+0000 7f432272b700 1 -- 172.21.15.16:0/3894864395 shutdown_connections 2023-10-16T12:30:02.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.399+0000 7f432272b700 1 -- 172.21.15.16:0/3894864395 wait complete. 2023-10-16T12:30:02.413 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2023-10-16T12:30:02.414 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:30:02.414 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T12:30:02.433 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_30_00_UTC 2023-10-16T12:30:02.433 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-12_30_00_UTC 2023-10-16T12:30:02.434 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:30:02.434 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-12_30_00_UTC) 2023-10-16T12:30:02.496 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:30:02.496 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2023-10-16T12:30:02.557 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-16T12:30:02.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.725+0000 7f27c1b2f700 1 Processor -- start 2023-10-16T12:30:02.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.725+0000 7f27c1b2f700 1 -- start start 2023-10-16T12:30:02.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.726+0000 7f27c1b2f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f27bc06e170 0x7f27bc0cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:02.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.726+0000 7f27c1b2f700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27bc0cef70 con 0x7f27bc0cb880 2023-10-16T12:30:02.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.726+0000 7f27c1b2f700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27bc0ce360 con 0x7f27bc15b070 2023-10-16T12:30:02.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.726+0000 7f27c1b2f700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27bc0ce640 con 0x7f27bc06e170 2023-10-16T12:30:02.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.726+0000 7f27bbfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f27bc06e170 0x7f27bc0cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:02.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.726+0000 7f27baffd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f27bc0cb880 0x7f27bc06da60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33636/0 (socket says 172.21.15.16:33636) 2023-10-16T12:30:02.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.726+0000 7f27baffd700 1 -- 172.21.15.16:0/2419409130 learned_addr learned my addr 172.21.15.16:0/2419409130 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:30:02.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.726+0000 7f27bbfff700 1 -- 172.21.15.16:0/2419409130 >> v1:172.21.15.49:6789/0 conn(0x7f27bc15b070 legacy=0x7f27bc15b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:02.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.726+0000 7f27bbfff700 1 -- 172.21.15.16:0/2419409130 >> v1:172.21.15.16:6789/0 conn(0x7f27bc0cb880 legacy=0x7f27bc06da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:02.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.726+0000 7f27bbfff700 1 -- 172.21.15.16:0/2419409130 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27bc0ce920 con 0x7f27bc06e170 2023-10-16T12:30:02.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.726+0000 7f27ba7fc700 1 -- 172.21.15.16:0/2419409130 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 911098011 0 0) 0x7f27bc0cef70 con 0x7f27bc0cb880 2023-10-16T12:30:02.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.727+0000 7f27bbfff700 1 --2- 172.21.15.16:0/2419409130 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27bc06e170 0x7f27bc0cbd00 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f27b4009d90 tx=0x7f27b400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f841c456ea7f16ef server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:02.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.727+0000 7f27ba7fc700 1 -- 172.21.15.16:0/2419409130 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27b400e070 con 0x7f27bc06e170 2023-10-16T12:30:02.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.727+0000 7f27ba7fc700 1 -- 172.21.15.16:0/2419409130 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27b400bbd0 con 0x7f27bc06e170 2023-10-16T12:30:02.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.727+0000 7f27ba7fc700 1 -- 172.21.15.16:0/2419409130 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27b4012430 con 0x7f27bc06e170 2023-10-16T12:30:02.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.727+0000 7f27c1b2f700 1 -- 172.21.15.16:0/2419409130 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27bc06e170 msgr2=0x7f27bc0cbd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:02.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.727+0000 7f27c1b2f700 1 --2- 172.21.15.16:0/2419409130 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27bc06e170 0x7f27bc0cbd00 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f27b4009d90 tx=0x7f27b400b330 comp rx=0 tx=0).stop 2023-10-16T12:30:02.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.727+0000 7f27c1b2f700 1 -- 172.21.15.16:0/2419409130 shutdown_connections 2023-10-16T12:30:02.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.727+0000 7f27c1b2f700 1 --2- 172.21.15.16:0/2419409130 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27bc06e170 0x7f27bc0cbd00 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:02.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.727+0000 7f27c1b2f700 1 -- 172.21.15.16:0/2419409130 >> 172.21.15.16:0/2419409130 conn(0x7f27bc0b1f40 msgr2=0x7f27bc0b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:02.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.727+0000 7f27c1b2f700 1 -- 172.21.15.16:0/2419409130 shutdown_connections 2023-10-16T12:30:02.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.727+0000 7f27c1b2f700 1 -- 172.21.15.16:0/2419409130 wait complete. 2023-10-16T12:30:02.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.729+0000 7f27c1b2f700 1 Processor -- start 2023-10-16T12:30:02.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.729+0000 7f27c1b2f700 1 -- start start 2023-10-16T12:30:02.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.729+0000 7f27c1b2f700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f27bc06e170 0x7f27bc152230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:02.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.729+0000 7f27c1b2f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27bc0cb880 0x7f27bc1528d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:02.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.729+0000 7f27c1b2f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f27bc15b070 0x7f27bc0ae600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:02.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.730+0000 7f27c1b2f700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27bc152770 con 0x7f27bc0cb880 2023-10-16T12:30:02.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.730+0000 7f27c1b2f700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27bc152e10 con 0x7f27bc06e170 2023-10-16T12:30:02.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.730+0000 7f27c1b2f700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27bc0ad2e0 con 0x7f27bc15b070 2023-10-16T12:30:02.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.730+0000 7f27baffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27bc0cb880 0x7f27bc1528d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:02.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.730+0000 7f27baffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27bc0cb880 0x7f27bc1528d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44368/0 (socket says 172.21.15.16:44368) 2023-10-16T12:30:02.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.730+0000 7f27baffd700 1 -- 172.21.15.16:0/3996843305 learned_addr learned my addr 172.21.15.16:0/3996843305 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:02.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.730+0000 7f27bbfff700 1 --2- 172.21.15.16:0/3996843305 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f27bc15b070 0x7f27bc0ae600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:02.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.730+0000 7f27baffd700 1 -- 172.21.15.16:0/3996843305 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f27bc15b070 msgr2=0x7f27bc0ae600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:02.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.730+0000 7f27bb7fe700 1 --2- 172.21.15.16:0/3996843305 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f27bc06e170 0x7f27bc152230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:02.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.730+0000 7f27baffd700 1 --2- 172.21.15.16:0/3996843305 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f27bc15b070 0x7f27bc0ae600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:02.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.730+0000 7f27baffd700 1 -- 172.21.15.16:0/3996843305 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f27bc06e170 msgr2=0x7f27bc152230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:02.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.730+0000 7f27baffd700 1 --2- 172.21.15.16:0/3996843305 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f27bc06e170 0x7f27bc152230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:02.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.730+0000 7f27baffd700 1 -- 172.21.15.16:0/3996843305 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27b4009a70 con 0x7f27bc0cb880 2023-10-16T12:30:02.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.730+0000 7f27bbfff700 1 --2- 172.21.15.16:0/3996843305 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f27bc15b070 0x7f27bc0ae600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:30:02.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.730+0000 7f27bb7fe700 1 --2- 172.21.15.16:0/3996843305 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f27bc06e170 0x7f27bc152230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:30:02.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.730+0000 7f27baffd700 1 --2- 172.21.15.16:0/3996843305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27bc0cb880 0x7f27bc1528d0 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f27a400c520 tx=0x7f27a400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:02.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.730+0000 7f27b3fff700 1 -- 172.21.15.16:0/3996843305 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27a4004c80 con 0x7f27bc0cb880 2023-10-16T12:30:02.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.730+0000 7f27b3fff700 1 -- 172.21.15.16:0/3996843305 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27a4004e00 con 0x7f27bc0cb880 2023-10-16T12:30:02.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.730+0000 7f27c1b2f700 1 -- 172.21.15.16:0/3996843305 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27bc0ad5c0 con 0x7f27bc0cb880 2023-10-16T12:30:02.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.730+0000 7f27b3fff700 1 -- 172.21.15.16:0/3996843305 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27a4007310 con 0x7f27bc0cb880 2023-10-16T12:30:02.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.730+0000 7f27c1b2f700 1 -- 172.21.15.16:0/3996843305 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27bc0adb80 con 0x7f27bc0cb880 2023-10-16T12:30:02.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.731+0000 7f27c1b2f700 1 -- 172.21.15.16:0/3996843305 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27bc04ebb0 con 0x7f27bc0cb880 2023-10-16T12:30:02.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.732+0000 7f27b3fff700 1 -- 172.21.15.16:0/3996843305 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 69) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f27a4007470 con 0x7f27bc0cb880 2023-10-16T12:30:02.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.732+0000 7f27b3fff700 1 --2- 172.21.15.16:0/3996843305 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7f279c06b190 0x7f279c06d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:02.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.732+0000 7f27bb7fe700 1 --2- 172.21.15.16:0/3996843305 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7f279c06b190 0x7f279c06d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:02.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.732+0000 7f27b3fff700 1 -- 172.21.15.16:0/3996843305 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f27a4089ac0 con 0x7f27bc0cb880 2023-10-16T12:30:02.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.733+0000 7f27bb7fe700 1 --2- 172.21.15.16:0/3996843305 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7f279c06b190 0x7f279c06d650 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f27ac000f60 tx=0x7f27ac009040 comp rx=0 tx=0).ready entity=mgr.7852 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:02.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:02.737+0000 7f27b3fff700 1 -- 172.21.15.16:0/3996843305 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f27a4054550 con 0x7f27bc0cb880 2023-10-16T12:30:03.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.005+0000 7f27c1b2f700 1 -- 172.21.15.16:0/3996843305 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f27bc0cd6c0 con 0x7f27bc0cb880 2023-10-16T12:30:03.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.327+0000 7f27b3fff700 1 -- 172.21.15.16:0/3996843305 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v70) v1 ==== 93+0+0 (secure 0 0 0) 0x7f27a4057b70 con 0x7f27bc0cb880 2023-10-16T12:30:03.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.331+0000 7f27c1b2f700 1 -- 172.21.15.16:0/3996843305 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7f279c06b190 msgr2=0x7f279c06d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:03.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.331+0000 7f27c1b2f700 1 --2- 172.21.15.16:0/3996843305 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7f279c06b190 0x7f279c06d650 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f27ac000f60 tx=0x7f27ac009040 comp rx=0 tx=0).stop 2023-10-16T12:30:03.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.331+0000 7f27c1b2f700 1 -- 172.21.15.16:0/3996843305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27bc0cb880 msgr2=0x7f27bc1528d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:03.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.331+0000 7f27c1b2f700 1 --2- 172.21.15.16:0/3996843305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27bc0cb880 0x7f27bc1528d0 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f27a400c520 tx=0x7f27a400c8e0 comp rx=0 tx=0).stop 2023-10-16T12:30:03.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.332+0000 7f27c1b2f700 1 -- 172.21.15.16:0/3996843305 shutdown_connections 2023-10-16T12:30:03.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.332+0000 7f27c1b2f700 1 --2- 172.21.15.16:0/3996843305 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7f279c06b190 0x7f279c06d650 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f27ac000f60 tx=0x7f27ac009040 comp rx=0 tx=0).stop 2023-10-16T12:30:03.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.332+0000 7f27c1b2f700 1 --2- 172.21.15.16:0/3996843305 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f27bc15b070 0x7f27bc0ae600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:03.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.332+0000 7f27c1b2f700 1 --2- 172.21.15.16:0/3996843305 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f27bc06e170 0x7f27bc152230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:03.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.332+0000 7f27c1b2f700 1 --2- 172.21.15.16:0/3996843305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27bc0cb880 0x7f27bc1528d0 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:03.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.332+0000 7f27c1b2f700 1 -- 172.21.15.16:0/3996843305 >> 172.21.15.16:0/3996843305 conn(0x7f27bc0b1f40 msgr2=0x7f27bc0cf300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:03.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.332+0000 7f27c1b2f700 1 -- 172.21.15.16:0/3996843305 shutdown_connections 2023-10-16T12:30:03.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.332+0000 7f27c1b2f700 1 -- 172.21.15.16:0/3996843305 wait complete. 2023-10-16T12:30:03.349 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T12:30:03.350 DEBUG:teuthology.orchestra.run.smithi016:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T12:30:03.397 DEBUG:teuthology.orchestra.run.smithi049:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T12:30:03.445 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T12:30:03.446 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:30:03.446 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:30:03.466 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:30:03.467 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:30:03.485 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:30:03.483+0000 7f71a592b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T12:30:03.486 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:03.483+0000 7fc68f8d0200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T12:30:03.505 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:30:03.554 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:03.553+0000 7fc68f8d0200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T12:30:03.555 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:30:03.553+0000 7f71a592b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T12:30:03.630 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:30:03.628+0000 7f71a592b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T12:30:03.632 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:03.631+0000 7fc68f8d0200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T12:30:03.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.674+0000 7f3efa5f0700 1 Processor -- start 2023-10-16T12:30:03.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.674+0000 7f3efa5f0700 1 -- start start 2023-10-16T12:30:03.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.675+0000 7f3efa5f0700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3ef415d900 0x7f3ef40d2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:03.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.675+0000 7f3efa5f0700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f3ef40ce9f0 0x7f3ef40cdc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:03.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.675+0000 7f3efa5f0700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ef40ce1a0 con 0x7f3ef40d2860 2023-10-16T12:30:03.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.675+0000 7f3efa5f0700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ef40cee70 con 0x7f3ef415d900 2023-10-16T12:30:03.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.675+0000 7f3efa5f0700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ef40ce400 con 0x7f3ef40ce9f0 2023-10-16T12:30:03.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.675+0000 7f3ef3fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3ef415d900 0x7f3ef40d2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:03.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.675+0000 7f3ef3fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3ef415d900 0x7f3ef40d2040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44418/0 (socket says 172.21.15.16:44418) 2023-10-16T12:30:03.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.675+0000 7f3ef8b8d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f3ef40d2860 0x7f3ef40b5120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33696/0 (socket says 172.21.15.16:33696) 2023-10-16T12:30:03.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.675+0000 7f3ef3fff700 1 -- 172.21.15.16:0/2351177756 learned_addr learned my addr 172.21.15.16:0/2351177756 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:03.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.675+0000 7f3ef37fe700 1 --2- 172.21.15.16:0/2351177756 >> v2:172.21.15.49:3300/0 conn(0x7f3ef40ce9f0 0x7f3ef40cdc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:03.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.676+0000 7f3ef3fff700 1 -- 172.21.15.16:0/2351177756 >> v2:172.21.15.49:3300/0 conn(0x7f3ef40ce9f0 msgr2=0x7f3ef40cdc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:03.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.676+0000 7f3ef3fff700 1 --2- 172.21.15.16:0/2351177756 >> v2:172.21.15.49:3300/0 conn(0x7f3ef40ce9f0 0x7f3ef40cdc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:03.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.676+0000 7f3ef3fff700 1 -- 172.21.15.16:0/2351177756 >> v1:172.21.15.16:6789/0 conn(0x7f3ef40d2860 legacy=0x7f3ef40b5120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:03.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.676+0000 7f3ef3fff700 1 -- 172.21.15.16:0/2351177756 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ef415dce0 con 0x7f3ef415d900 2023-10-16T12:30:03.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.676+0000 7f3ef2ffd700 1 -- 172.21.15.16:0/2351177756 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1490231487 0 0) 0x7f3ef40ce1a0 con 0x7f3ef40d2860 2023-10-16T12:30:03.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.676+0000 7f3ef3fff700 1 --2- 172.21.15.16:0/2351177756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ef415d900 0x7f3ef40d2040 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f3ef40b9810 tx=0x7f3ee4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8e9918cf902049f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:03.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.676+0000 7f3ef2ffd700 1 -- 172.21.15.16:0/2351177756 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ee4011830 con 0x7f3ef415d900 2023-10-16T12:30:03.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.676+0000 7f3ef2ffd700 1 -- 172.21.15.16:0/2351177756 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ee4011990 con 0x7f3ef415d900 2023-10-16T12:30:03.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.676+0000 7f3ef2ffd700 1 -- 172.21.15.16:0/2351177756 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ee4011d20 con 0x7f3ef415d900 2023-10-16T12:30:03.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.677+0000 7f3efa5f0700 1 -- 172.21.15.16:0/2351177756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ef415d900 msgr2=0x7f3ef40d2040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:03.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.677+0000 7f3efa5f0700 1 --2- 172.21.15.16:0/2351177756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ef415d900 0x7f3ef40d2040 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f3ef40b9810 tx=0x7f3ee4009bd0 comp rx=0 tx=0).stop 2023-10-16T12:30:03.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.677+0000 7f3efa5f0700 1 -- 172.21.15.16:0/2351177756 shutdown_connections 2023-10-16T12:30:03.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.677+0000 7f3efa5f0700 1 --2- 172.21.15.16:0/2351177756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ef415d900 0x7f3ef40d2040 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:03.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.677+0000 7f3efa5f0700 1 --2- 172.21.15.16:0/2351177756 >> v2:172.21.15.49:3300/0 conn(0x7f3ef40ce9f0 0x7f3ef40cdc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:03.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.677+0000 7f3efa5f0700 1 -- 172.21.15.16:0/2351177756 >> 172.21.15.16:0/2351177756 conn(0x7f3ef4072330 msgr2=0x7f3ef4072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:03.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.677+0000 7f3efa5f0700 1 -- 172.21.15.16:0/2351177756 shutdown_connections 2023-10-16T12:30:03.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.677+0000 7f3efa5f0700 1 -- 172.21.15.16:0/2351177756 wait complete. 2023-10-16T12:30:03.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.678+0000 7f3efa5f0700 1 Processor -- start 2023-10-16T12:30:03.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.678+0000 7f3efa5f0700 1 -- start start 2023-10-16T12:30:03.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.679+0000 7f3efa5f0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ef40ce9f0 0x7f3ef414ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:03.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.679+0000 7f3efa5f0700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3ef40d2860 0x7f3ef415f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:03.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.679+0000 7f3efa5f0700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3ef415d900 0x7f3ef40cbbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:03.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.679+0000 7f3efa5f0700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ef415fcd0 con 0x7f3ef40ce9f0 2023-10-16T12:30:03.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.679+0000 7f3efa5f0700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ef40c8b90 con 0x7f3ef40d2860 2023-10-16T12:30:03.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.679+0000 7f3efa5f0700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ef40c8e70 con 0x7f3ef415d900 2023-10-16T12:30:03.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.679+0000 7f3ef3fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ef40ce9f0 0x7f3ef414ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:03.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.679+0000 7f3ef3fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ef40ce9f0 0x7f3ef414ac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44424/0 (socket says 172.21.15.16:44424) 2023-10-16T12:30:03.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.679+0000 7f3ef3fff700 1 -- 172.21.15.16:0/1177930593 learned_addr learned my addr 172.21.15.16:0/1177930593 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:03.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.679+0000 7f3ef8b8d700 1 --2- 172.21.15.16:0/1177930593 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3ef415d900 0x7f3ef40cbbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:03.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.679+0000 7f3ef37fe700 1 --2- 172.21.15.16:0/1177930593 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3ef40d2860 0x7f3ef415f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:03.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.680+0000 7f3ef3fff700 1 -- 172.21.15.16:0/1177930593 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3ef415d900 msgr2=0x7f3ef40cbbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:03.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.680+0000 7f3ef3fff700 1 --2- 172.21.15.16:0/1177930593 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3ef415d900 0x7f3ef40cbbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:03.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.680+0000 7f3ef3fff700 1 -- 172.21.15.16:0/1177930593 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3ef40d2860 msgr2=0x7f3ef415f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:03.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.680+0000 7f3ef3fff700 1 --2- 172.21.15.16:0/1177930593 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3ef40d2860 0x7f3ef415f210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:03.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.680+0000 7f3ef3fff700 1 -- 172.21.15.16:0/1177930593 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ee4009550 con 0x7f3ef40ce9f0 2023-10-16T12:30:03.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.680+0000 7f3ef3fff700 1 --2- 172.21.15.16:0/1177930593 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ef40ce9f0 0x7f3ef414ac40 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f3ee4009c10 tx=0x7f3ee4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:03.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.680+0000 7f3ef0ff9700 1 -- 172.21.15.16:0/1177930593 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ee4004510 con 0x7f3ef40ce9f0 2023-10-16T12:30:03.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.680+0000 7f3ef0ff9700 1 -- 172.21.15.16:0/1177930593 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ee401e070 con 0x7f3ef40ce9f0 2023-10-16T12:30:03.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.680+0000 7f3ef0ff9700 1 -- 172.21.15.16:0/1177930593 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ee40038e0 con 0x7f3ef40ce9f0 2023-10-16T12:30:03.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.680+0000 7f3efa5f0700 1 -- 172.21.15.16:0/1177930593 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ef40c9150 con 0x7f3ef40ce9f0 2023-10-16T12:30:03.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.680+0000 7f3efa5f0700 1 -- 172.21.15.16:0/1177930593 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ef40c8070 con 0x7f3ef40ce9f0 2023-10-16T12:30:03.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.681+0000 7f3efa5f0700 1 -- 172.21.15.16:0/1177930593 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ef404ebb0 con 0x7f3ef40ce9f0 2023-10-16T12:30:03.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.682+0000 7f3ef0ff9700 1 -- 172.21.15.16:0/1177930593 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 70) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f3ee4003a40 con 0x7f3ef40ce9f0 2023-10-16T12:30:03.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.682+0000 7f3ef0ff9700 1 --2- 172.21.15.16:0/1177930593 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7f3ee006b140 0x7f3ee006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:03.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.682+0000 7f3ef37fe700 1 -- 172.21.15.16:0/1177930593 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7f3ee006b140 msgr2=0x7f3ee006d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2550537276 2023-10-16T12:30:03.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.682+0000 7f3ef0ff9700 1 -- 172.21.15.16:0/1177930593 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f3ee408aaf0 con 0x7f3ef40ce9f0 2023-10-16T12:30:03.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.682+0000 7f3ef37fe700 1 --2- 172.21.15.16:0/1177930593 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7f3ee006b140 0x7f3ee006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:30:03.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.687+0000 7f3ef0ff9700 1 -- 172.21.15.16:0/1177930593 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3ee40556e0 con 0x7f3ef40ce9f0 2023-10-16T12:30:03.711 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:30:03.709+0000 7f71a592b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T12:30:03.718 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:03.716+0000 7fc68f8d0200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T12:30:03.859 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:30:03.857+0000 7f71a592b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T12:30:03.867 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:03.866+0000 7fc68f8d0200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T12:30:03.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.883+0000 7f3ef37fe700 1 -- 172.21.15.16:0/1177930593 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7f3ee006b140 msgr2=0x7f3ee006d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2550537276 2023-10-16T12:30:03.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.883+0000 7f3ef37fe700 1 --2- 172.21.15.16:0/1177930593 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7f3ee006b140 0x7f3ee006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:30:03.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.891+0000 7f3efa5f0700 1 -- 172.21.15.16:0/1177930593 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3ef40c6d30 con 0x7f3ef40ce9f0 2023-10-16T12:30:03.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.892+0000 7f3ef0ff9700 1 -- 172.21.15.16:0/1177930593 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v129) v1 ==== 74+0+17150 (secure 0 0 0) 0x7f3ee4020090 con 0x7f3ef40ce9f0 2023-10-16T12:30:03.894 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:30:03.895 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":129,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:29:30.732485+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:29:14.232554+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2023-10-16T12:29:14.764132+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/32319707":"2023-10-17T12:29:30.732445+0000","172.21.15.16:0/1058620826":"2023-10-17T12:29:30.732445+0000","172.21.15.16:0/3098064509":"2023-10-17T12:29:30.732445+0000","172.21.15.16:0/3814896496":"2023-10-17T12:29:30.732445+0000","172.21.15.16:0/2973963882":"2023-10-17T12:29:30.732445+0000","172.21.15.16:6833/2309613638":"2023-10-17T12:29:30.732445+0000","172.21.15.16:6832/2309613638":"2023-10-17T12:29:30.732445+0000","172.21.15.16:0/1074148914":"2023-10-17T12:29:30.732445+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:30:03.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.894+0000 7f3efa5f0700 1 -- 172.21.15.16:0/1177930593 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7f3ee006b140 msgr2=0x7f3ee006d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:30:03.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.894+0000 7f3efa5f0700 1 --2- 172.21.15.16:0/1177930593 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7f3ee006b140 0x7f3ee006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:03.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.894+0000 7f3efa5f0700 1 -- 172.21.15.16:0/1177930593 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ef40ce9f0 msgr2=0x7f3ef414ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:03.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.894+0000 7f3efa5f0700 1 --2- 172.21.15.16:0/1177930593 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ef40ce9f0 0x7f3ef414ac40 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f3ee4009c10 tx=0x7f3ee4004750 comp rx=0 tx=0).stop 2023-10-16T12:30:03.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.894+0000 7f3efa5f0700 1 -- 172.21.15.16:0/1177930593 shutdown_connections 2023-10-16T12:30:03.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.894+0000 7f3efa5f0700 1 --2- 172.21.15.16:0/1177930593 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7f3ee006b140 0x7f3ee006d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:03.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.894+0000 7f3efa5f0700 1 --2- 172.21.15.16:0/1177930593 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3ef415d900 0x7f3ef40cbbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:03.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.894+0000 7f3efa5f0700 1 --2- 172.21.15.16:0/1177930593 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ef40ce9f0 0x7f3ef414ac40 unknown :-1 s=CLOSED pgs=1438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:03.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.894+0000 7f3efa5f0700 1 --2- 172.21.15.16:0/1177930593 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3ef40d2860 0x7f3ef415f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:03.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.894+0000 7f3efa5f0700 1 -- 172.21.15.16:0/1177930593 >> 172.21.15.16:0/1177930593 conn(0x7f3ef4072330 msgr2=0x7f3ef406b800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:03.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.894+0000 7f3efa5f0700 1 -- 172.21.15.16:0/1177930593 shutdown_connections 2023-10-16T12:30:03.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:03.894+0000 7f3efa5f0700 1 -- 172.21.15.16:0/1177930593 wait complete. 2023-10-16T12:30:03.909 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T12:29:14.232554+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '127', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T12:29:14.764132+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '128', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T12:30:03.910 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:30:04.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.076+0000 7fc8d37e3700 1 Processor -- start 2023-10-16T12:30:04.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.076+0000 7fc8d37e3700 1 -- start start 2023-10-16T12:30:04.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.077+0000 7fc8d37e3700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc8cc15ec80 0x7fc8cc0cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:04.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.077+0000 7fc8d37e3700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc8cc15f060 0x7fc8cc0ce400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:04.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.077+0000 7fc8d37e3700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8cc0b2440 con 0x7fc8cc0d2110 2023-10-16T12:30:04.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.077+0000 7fc8d37e3700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8cc0b2c50 con 0x7fc8cc15f060 2023-10-16T12:30:04.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.077+0000 7fc8d37e3700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8cc0d00e0 con 0x7fc8cc15ec80 2023-10-16T12:30:04.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.077+0000 7fc8d0d7e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc8cc15f060 0x7fc8cc0ce400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:04.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.077+0000 7fc8d0d7e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc8cc15f060 0x7fc8cc0ce400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44458/0 (socket says 172.21.15.16:44458) 2023-10-16T12:30:04.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.077+0000 7fc8d0d7e700 1 -- 172.21.15.16:0/102271555 learned_addr learned my addr 172.21.15.16:0/102271555 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:04.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.077+0000 7fc8d157f700 1 --2- 172.21.15.16:0/102271555 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc8cc15ec80 0x7fc8cc0cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:04.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.077+0000 7fc8d0d7e700 1 -- 172.21.15.16:0/102271555 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc8cc15ec80 msgr2=0x7fc8cc0cdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:04.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.077+0000 7fc8d0d7e700 1 --2- 172.21.15.16:0/102271555 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc8cc15ec80 0x7fc8cc0cdec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:04.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.077+0000 7fc8d0d7e700 1 -- 172.21.15.16:0/102271555 >> v1:172.21.15.49:6789/0 conn(0x7fc8cc0d2110 legacy=0x7fc8cc0b4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:04.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.077+0000 7fc8d0d7e700 1 -- 172.21.15.16:0/102271555 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8cc0cef60 con 0x7fc8cc15f060 2023-10-16T12:30:04.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.078+0000 7fc8d0d7e700 1 --2- 172.21.15.16:0/102271555 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8cc15f060 0x7fc8cc0ce400 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7fc8bc0098f0 tx=0x7fc8bc009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be89a41782c445ed server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:04.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.078+0000 7fc8c3fff700 1 -- 172.21.15.16:0/102271555 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8bc011830 con 0x7fc8cc15f060 2023-10-16T12:30:04.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.078+0000 7fc8c3fff700 1 -- 172.21.15.16:0/102271555 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8bc011990 con 0x7fc8cc15f060 2023-10-16T12:30:04.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.078+0000 7fc8c3fff700 1 -- 172.21.15.16:0/102271555 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8bc011d20 con 0x7fc8cc15f060 2023-10-16T12:30:04.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.078+0000 7fc8d37e3700 1 -- 172.21.15.16:0/102271555 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8cc15f060 msgr2=0x7fc8cc0ce400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:04.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.078+0000 7fc8d37e3700 1 --2- 172.21.15.16:0/102271555 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8cc15f060 0x7fc8cc0ce400 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7fc8bc0098f0 tx=0x7fc8bc009c00 comp rx=0 tx=0).stop 2023-10-16T12:30:04.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.078+0000 7fc8d37e3700 1 -- 172.21.15.16:0/102271555 shutdown_connections 2023-10-16T12:30:04.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.078+0000 7fc8d37e3700 1 --2- 172.21.15.16:0/102271555 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc8cc15ec80 0x7fc8cc0cdec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:04.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.078+0000 7fc8d37e3700 1 --2- 172.21.15.16:0/102271555 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8cc15f060 0x7fc8cc0ce400 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:04.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.078+0000 7fc8d37e3700 1 -- 172.21.15.16:0/102271555 >> 172.21.15.16:0/102271555 conn(0x7fc8cc0d15f0 msgr2=0x7fc8cc0d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:04.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.079+0000 7fc8d37e3700 1 -- 172.21.15.16:0/102271555 shutdown_connections 2023-10-16T12:30:04.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.079+0000 7fc8d37e3700 1 -- 172.21.15.16:0/102271555 wait complete. 2023-10-16T12:30:04.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.080+0000 7fc8d37e3700 1 Processor -- start 2023-10-16T12:30:04.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.080+0000 7fc8d37e3700 1 -- start start 2023-10-16T12:30:04.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.080+0000 7fc8d37e3700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc8cc0d2110 0x7fc8cc06d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:04.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.080+0000 7fc8d37e3700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc8cc15ec80 0x7fc8cc15c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:04.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.080+0000 7fc8d37e3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8cc15f060 0x7fc8cc15c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:04.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.080+0000 7fc8d37e3700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8cc0c99e0 con 0x7fc8cc15f060 2023-10-16T12:30:04.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.080+0000 7fc8d37e3700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8cc06d770 con 0x7fc8cc0d2110 2023-10-16T12:30:04.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.080+0000 7fc8d37e3700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8cc0c9040 con 0x7fc8cc15ec80 2023-10-16T12:30:04.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.081+0000 7fc8d1d80700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8cc15f060 0x7fc8cc15c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:04.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.081+0000 7fc8d1d80700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8cc15f060 0x7fc8cc15c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44466/0 (socket says 172.21.15.16:44466) 2023-10-16T12:30:04.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.081+0000 7fc8d1d80700 1 -- 172.21.15.16:0/1002261031 learned_addr learned my addr 172.21.15.16:0/1002261031 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:04.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.081+0000 7fc8d157f700 1 --2- 172.21.15.16:0/1002261031 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc8cc0d2110 0x7fc8cc06d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:04.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.081+0000 7fc8d0d7e700 1 --2- 172.21.15.16:0/1002261031 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc8cc15ec80 0x7fc8cc15c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:04.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.081+0000 7fc8d1d80700 1 -- 172.21.15.16:0/1002261031 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc8cc15ec80 msgr2=0x7fc8cc15c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:04.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.081+0000 7fc8d1d80700 1 --2- 172.21.15.16:0/1002261031 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc8cc15ec80 0x7fc8cc15c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:04.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.081+0000 7fc8d1d80700 1 -- 172.21.15.16:0/1002261031 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc8cc0d2110 msgr2=0x7fc8cc06d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:04.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.081+0000 7fc8d1d80700 1 --2- 172.21.15.16:0/1002261031 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc8cc0d2110 0x7fc8cc06d230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:04.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.081+0000 7fc8d1d80700 1 -- 172.21.15.16:0/1002261031 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8bc009550 con 0x7fc8cc15f060 2023-10-16T12:30:04.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.081+0000 7fc8d0d7e700 1 --2- 172.21.15.16:0/1002261031 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc8cc15ec80 0x7fc8cc15c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:30:04.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.081+0000 7fc8d1d80700 1 --2- 172.21.15.16:0/1002261031 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8cc15f060 0x7fc8cc15c8c0 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7fc8c400cc50 tx=0x7fc8c400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:04.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.081+0000 7fc8c1ffb700 1 -- 172.21.15.16:0/1002261031 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c4003c00 con 0x7fc8cc15f060 2023-10-16T12:30:04.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.081+0000 7fc8c1ffb700 1 -- 172.21.15.16:0/1002261031 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8c4003d60 con 0x7fc8cc15f060 2023-10-16T12:30:04.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.081+0000 7fc8d37e3700 1 -- 172.21.15.16:0/1002261031 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8cc0c9320 con 0x7fc8cc15f060 2023-10-16T12:30:04.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.081+0000 7fc8c1ffb700 1 -- 172.21.15.16:0/1002261031 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c400aab0 con 0x7fc8cc15f060 2023-10-16T12:30:04.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.082+0000 7fc8d37e3700 1 -- 172.21.15.16:0/1002261031 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8cc0c8ca0 con 0x7fc8cc15f060 2023-10-16T12:30:04.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.083+0000 7fc8d37e3700 1 -- 172.21.15.16:0/1002261031 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8cc0c9b20 con 0x7fc8cc15f060 2023-10-16T12:30:04.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.083+0000 7fc8c1ffb700 1 -- 172.21.15.16:0/1002261031 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 70) v1 ==== 89445+0+0 (secure 0 0 0) 0x7fc8c400ac10 con 0x7fc8cc15f060 2023-10-16T12:30:04.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.084+0000 7fc8c1ffb700 1 --2- 172.21.15.16:0/1002261031 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7fc8b006b140 0x7fc8b006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:04.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.084+0000 7fc8d157f700 1 -- 172.21.15.16:0/1002261031 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7fc8b006b140 msgr2=0x7fc8b006d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2550537276 2023-10-16T12:30:04.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.084+0000 7fc8c1ffb700 1 -- 172.21.15.16:0/1002261031 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fc8c4089c70 con 0x7fc8cc15f060 2023-10-16T12:30:04.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.084+0000 7fc8d157f700 1 --2- 172.21.15.16:0/1002261031 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7fc8b006b140 0x7fc8b006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:30:04.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.088+0000 7fc8c1ffb700 1 -- 172.21.15.16:0/1002261031 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc8cc0c9b20 con 0x7fc8cc15f060 2023-10-16T12:30:04.201 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:30:04.199+0000 7f71a592b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T12:30:04.216 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:04.215+0000 7fc68f8d0200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T12:30:04.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.285+0000 7fc8d157f700 1 -- 172.21.15.16:0/1002261031 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7fc8b006b140 msgr2=0x7fc8b006d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2550537276 2023-10-16T12:30:04.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.285+0000 7fc8d157f700 1 --2- 172.21.15.16:0/1002261031 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7fc8b006b140 0x7fc8b006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:30:04.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.292+0000 7fc8d37e3700 1 -- 172.21.15.16:0/1002261031 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc8cc0c9b20 con 0x7fc8cc15f060 2023-10-16T12:30:04.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.293+0000 7fc8c1ffb700 1 -- 172.21.15.16:0/1002261031 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v129) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc8cc0c9b20 con 0x7fc8cc15f060 2023-10-16T12:30:04.297 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T12:30:04.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.295+0000 7fc8d37e3700 1 -- 172.21.15.16:0/1002261031 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7fc8b006b140 msgr2=0x7fc8b006d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:30:04.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.295+0000 7fc8d37e3700 1 --2- 172.21.15.16:0/1002261031 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7fc8b006b140 0x7fc8b006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:04.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.295+0000 7fc8d37e3700 1 -- 172.21.15.16:0/1002261031 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8cc15f060 msgr2=0x7fc8cc15c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:04.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.295+0000 7fc8d37e3700 1 --2- 172.21.15.16:0/1002261031 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8cc15f060 0x7fc8cc15c8c0 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7fc8c400cc50 tx=0x7fc8c400cf60 comp rx=0 tx=0).stop 2023-10-16T12:30:04.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.296+0000 7fc8d37e3700 1 -- 172.21.15.16:0/1002261031 shutdown_connections 2023-10-16T12:30:04.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.296+0000 7fc8d37e3700 1 --2- 172.21.15.16:0/1002261031 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7fc8b006b140 0x7fc8b006d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:04.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.296+0000 7fc8d37e3700 1 --2- 172.21.15.16:0/1002261031 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8cc15f060 0x7fc8cc15c8c0 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:04.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.296+0000 7fc8d37e3700 1 --2- 172.21.15.16:0/1002261031 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc8cc0d2110 0x7fc8cc06d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:04.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.296+0000 7fc8d37e3700 1 --2- 172.21.15.16:0/1002261031 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc8cc15ec80 0x7fc8cc15c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:04.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.296+0000 7fc8d37e3700 1 -- 172.21.15.16:0/1002261031 >> 172.21.15.16:0/1002261031 conn(0x7fc8cc0d15f0 msgr2=0x7fc8cc0b4170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:04.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.296+0000 7fc8d37e3700 1 -- 172.21.15.16:0/1002261031 shutdown_connections 2023-10-16T12:30:04.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.296+0000 7fc8d37e3700 1 -- 172.21.15.16:0/1002261031 wait complete. 2023-10-16T12:30:04.313 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T12:30:04.325 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:30:04.323+0000 7f71a592b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T12:30:04.344 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:04.342+0000 7fc68f8d0200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T12:30:04.402 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:30:04.400+0000 7f71a592b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T12:30:04.423 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:04.421+0000 7fc68f8d0200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T12:30:04.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.441+0000 7f083da1d700 1 Processor -- start 2023-10-16T12:30:04.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.442+0000 7f083da1d700 1 -- start start 2023-10-16T12:30:04.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.442+0000 7f083da1d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f083806a7e0 0x7f083806ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:04.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.442+0000 7f083da1d700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f083815ecb0 con 0x7f083815b2c0 2023-10-16T12:30:04.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.442+0000 7f083da1d700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f083815b6a0 con 0x7f083806e570 2023-10-16T12:30:04.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.442+0000 7f083da1d700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f083806b190 con 0x7f083806a7e0 2023-10-16T12:30:04.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.442+0000 7f0836ffd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f083815b2c0 0x7f0838161340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33746/0 (socket says 172.21.15.16:33746) 2023-10-16T12:30:04.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.443+0000 7f0836ffd700 1 -- 172.21.15.16:0/1129798000 learned_addr learned my addr 172.21.15.16:0/1129798000 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:30:04.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.443+0000 7f08377fe700 1 --2- 172.21.15.16:0/1129798000 >> v2:172.21.15.16:3300/0 conn(0x7f083806a7e0 0x7f083806ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:04.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.443+0000 7f0835ffb700 1 -- 172.21.15.16:0/1129798000 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 267971077 0 0) 0x7f083815ecb0 con 0x7f083815b2c0 2023-10-16T12:30:04.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.443+0000 7f0835ffb700 1 -- 172.21.15.16:0/1129798000 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08200035b0 con 0x7f083815b2c0 2023-10-16T12:30:04.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.443+0000 7f08377fe700 1 -- 172.21.15.16:0/1129798000 >> v1:172.21.15.49:6789/0 conn(0x7f083806e570 legacy=0x7f083815e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:04.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.443+0000 7f0835ffb700 1 -- 172.21.15.16:0/1129798000 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 578662258 0 0) 0x7f083815b6a0 con 0x7f083806e570 2023-10-16T12:30:04.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.443+0000 7f08377fe700 1 -- 172.21.15.16:0/1129798000 >> v1:172.21.15.16:6789/0 conn(0x7f083815b2c0 legacy=0x7f0838161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:04.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.443+0000 7f08377fe700 1 -- 172.21.15.16:0/1129798000 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0838161b20 con 0x7f083806a7e0 2023-10-16T12:30:04.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.443+0000 7f08377fe700 1 --2- 172.21.15.16:0/1129798000 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f083806a7e0 0x7f083806ac50 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f0830009d90 tx=0x7f083000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77f4759ac4da1318 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:04.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.443+0000 7f0835ffb700 1 -- 172.21.15.16:0/1129798000 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f083000e070 con 0x7f083806a7e0 2023-10-16T12:30:04.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.443+0000 7f0835ffb700 1 -- 172.21.15.16:0/1129798000 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f083000bb20 con 0x7f083806a7e0 2023-10-16T12:30:04.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.443+0000 7f0835ffb700 1 -- 172.21.15.16:0/1129798000 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f083000bed0 con 0x7f083806a7e0 2023-10-16T12:30:04.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.444+0000 7f083da1d700 1 -- 172.21.15.16:0/1129798000 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f083806a7e0 msgr2=0x7f083806ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:04.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.444+0000 7f083da1d700 1 --2- 172.21.15.16:0/1129798000 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f083806a7e0 0x7f083806ac50 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f0830009d90 tx=0x7f083000b330 comp rx=0 tx=0).stop 2023-10-16T12:30:04.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.444+0000 7f083da1d700 1 -- 172.21.15.16:0/1129798000 shutdown_connections 2023-10-16T12:30:04.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.444+0000 7f083da1d700 1 --2- 172.21.15.16:0/1129798000 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f083806a7e0 0x7f083806ac50 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:04.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.444+0000 7f083da1d700 1 -- 172.21.15.16:0/1129798000 >> 172.21.15.16:0/1129798000 conn(0x7f08380b1f40 msgr2=0x7f08380b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:04.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.444+0000 7f083da1d700 1 -- 172.21.15.16:0/1129798000 shutdown_connections 2023-10-16T12:30:04.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.444+0000 7f083da1d700 1 -- 172.21.15.16:0/1129798000 wait complete. 2023-10-16T12:30:04.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.446+0000 7f083da1d700 1 Processor -- start 2023-10-16T12:30:04.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.446+0000 7f083da1d700 1 -- start start 2023-10-16T12:30:04.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.447+0000 7f083da1d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f083806a7e0 0x7f08380cbde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:04.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.447+0000 7f083da1d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f083806e570 0x7f08380c4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:04.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.447+0000 7f083da1d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f083815b2c0 0x7f08380c5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:04.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.447+0000 7f083da1d700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08380c58b0 con 0x7f083806a7e0 2023-10-16T12:30:04.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.447+0000 7f083da1d700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08380c5b90 con 0x7f083806e570 2023-10-16T12:30:04.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.447+0000 7f083da1d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08380c5e70 con 0x7f083815b2c0 2023-10-16T12:30:04.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.447+0000 7f0836ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f083806a7e0 0x7f08380cbde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:04.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.447+0000 7f0836ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f083806a7e0 0x7f08380cbde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44478/0 (socket says 172.21.15.16:44478) 2023-10-16T12:30:04.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.447+0000 7f0836ffd700 1 -- 172.21.15.16:0/2682745950 learned_addr learned my addr 172.21.15.16:0/2682745950 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:04.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.447+0000 7f08367fc700 1 --2- 172.21.15.16:0/2682745950 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f083806e570 0x7f08380c4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:04.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.447+0000 7f08377fe700 1 --2- 172.21.15.16:0/2682745950 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f083815b2c0 0x7f08380c5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:04.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.447+0000 7f0836ffd700 1 -- 172.21.15.16:0/2682745950 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f083815b2c0 msgr2=0x7f08380c5230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:04.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.447+0000 7f0836ffd700 1 --2- 172.21.15.16:0/2682745950 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f083815b2c0 0x7f08380c5230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:04.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.447+0000 7f0836ffd700 1 -- 172.21.15.16:0/2682745950 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f083806e570 msgr2=0x7f08380c4cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:04.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.447+0000 7f0836ffd700 1 --2- 172.21.15.16:0/2682745950 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f083806e570 0x7f08380c4cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:04.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.447+0000 7f0836ffd700 1 -- 172.21.15.16:0/2682745950 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0830009a70 con 0x7f083806a7e0 2023-10-16T12:30:04.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.447+0000 7f0836ffd700 1 --2- 172.21.15.16:0/2682745950 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f083806a7e0 0x7f08380cbde0 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f082800c4d0 tx=0x7f082800c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:04.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.448+0000 7f081ffff700 1 -- 172.21.15.16:0/2682745950 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0828005d70 con 0x7f083806a7e0 2023-10-16T12:30:04.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.448+0000 7f081ffff700 1 -- 172.21.15.16:0/2682745950 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08280070d0 con 0x7f083806a7e0 2023-10-16T12:30:04.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.448+0000 7f083da1d700 1 -- 172.21.15.16:0/2682745950 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08380c6150 con 0x7f083806a7e0 2023-10-16T12:30:04.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.448+0000 7f081ffff700 1 -- 172.21.15.16:0/2682745950 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0828007440 con 0x7f083806a7e0 2023-10-16T12:30:04.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.448+0000 7f083da1d700 1 -- 172.21.15.16:0/2682745950 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0838159fb0 con 0x7f083806a7e0 2023-10-16T12:30:04.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.449+0000 7f083da1d700 1 -- 172.21.15.16:0/2682745950 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f083804ebb0 con 0x7f083806a7e0 2023-10-16T12:30:04.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.450+0000 7f081ffff700 1 -- 172.21.15.16:0/2682745950 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 70) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f08280075a0 con 0x7f083806a7e0 2023-10-16T12:30:04.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.450+0000 7f081ffff700 1 --2- 172.21.15.16:0/2682745950 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7f082406b140 0x7f082406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:04.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.450+0000 7f08367fc700 1 -- 172.21.15.16:0/2682745950 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7f082406b140 msgr2=0x7f082406d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2550537276 2023-10-16T12:30:04.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.450+0000 7f081ffff700 1 -- 172.21.15.16:0/2682745950 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f0828088f90 con 0x7f083806a7e0 2023-10-16T12:30:04.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.450+0000 7f08367fc700 1 --2- 172.21.15.16:0/2682745950 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7f082406b140 0x7f082406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:30:04.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.455+0000 7f081ffff700 1 -- 172.21.15.16:0/2682745950 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f08280bbfe0 con 0x7f083806a7e0 2023-10-16T12:30:04.487 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:30:04.485+0000 7f71a592b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T12:30:04.510 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:04.508+0000 7fc68f8d0200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T12:30:04.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.651+0000 7f08367fc700 1 -- 172.21.15.16:0/2682745950 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7f082406b140 msgr2=0x7f082406d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2550537276 2023-10-16T12:30:04.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.651+0000 7f08367fc700 1 --2- 172.21.15.16:0/2682745950 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7f082406b140 0x7f082406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:30:04.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.658+0000 7f083da1d700 1 -- 172.21.15.16:0/2682745950 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f08380b6370 con 0x7f083806a7e0 2023-10-16T12:30:04.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.658+0000 7f081ffff700 1 -- 172.21.15.16:0/2682745950 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v129) v1 ==== 104+0+11 (secure 0 0 0) 0x7f08280170a0 con 0x7f083806a7e0 2023-10-16T12:30:04.660 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-10-16T12:30:04.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.660+0000 7f083da1d700 1 -- 172.21.15.16:0/2682745950 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7f082406b140 msgr2=0x7f082406d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:30:04.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.660+0000 7f083da1d700 1 --2- 172.21.15.16:0/2682745950 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7f082406b140 0x7f082406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:04.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.660+0000 7f083da1d700 1 -- 172.21.15.16:0/2682745950 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f083806a7e0 msgr2=0x7f08380cbde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:04.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.660+0000 7f083da1d700 1 --2- 172.21.15.16:0/2682745950 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f083806a7e0 0x7f08380cbde0 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f082800c4d0 tx=0x7f082800c7e0 comp rx=0 tx=0).stop 2023-10-16T12:30:04.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.660+0000 7f083da1d700 1 -- 172.21.15.16:0/2682745950 shutdown_connections 2023-10-16T12:30:04.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.660+0000 7f083da1d700 1 --2- 172.21.15.16:0/2682745950 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7f082406b140 0x7f082406d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:04.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.660+0000 7f083da1d700 1 --2- 172.21.15.16:0/2682745950 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f083815b2c0 0x7f08380c5230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:04.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.660+0000 7f083da1d700 1 --2- 172.21.15.16:0/2682745950 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f083806a7e0 0x7f08380cbde0 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:04.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.660+0000 7f083da1d700 1 --2- 172.21.15.16:0/2682745950 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f083806e570 0x7f08380c4cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:04.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.660+0000 7f083da1d700 1 -- 172.21.15.16:0/2682745950 >> 172.21.15.16:0/2682745950 conn(0x7f08380b1f40 msgr2=0x7f08380cce00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:04.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.660+0000 7f083da1d700 1 -- 172.21.15.16:0/2682745950 shutdown_connections 2023-10-16T12:30:04.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.660+0000 7f083da1d700 1 -- 172.21.15.16:0/2682745950 wait complete. 2023-10-16T12:30:04.675 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T12:30:04.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.843+0000 7fa33194f700 1 Processor -- start 2023-10-16T12:30:04.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.843+0000 7fa33194f700 1 -- start start 2023-10-16T12:30:04.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.844+0000 7fa33194f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa32c072b30 0x7fa32c071a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:04.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.844+0000 7fa33194f700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fa32c15bb40 0x7fa32c06e310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:04.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.844+0000 7fa33194f700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa32c15c040 con 0x7fa32c0caa40 2023-10-16T12:30:04.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.844+0000 7fa33194f700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa32c15c320 con 0x7fa32c15bb40 2023-10-16T12:30:04.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.844+0000 7fa33194f700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa32c15c600 con 0x7fa32c072b30 2023-10-16T12:30:04.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.844+0000 7fa32affd700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa32c072b30 0x7fa32c071a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:04.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.844+0000 7fa32a7fc700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fa32c15bb40 0x7fa32c06e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:04.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.844+0000 7fa32b7fe700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7fa32c0caa40 0x7fa32c0ccf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:40974/0 (socket says 172.21.15.16:40974) 2023-10-16T12:30:04.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.844+0000 7fa32a7fc700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fa32c15bb40 0x7fa32c06e310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:43578/0 (socket says 172.21.15.16:43578) 2023-10-16T12:30:04.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.844+0000 7fa32affd700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa32c072b30 0x7fa32c071a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:32772/0 (socket says 172.21.15.16:32772) 2023-10-16T12:30:04.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.844+0000 7fa32a7fc700 1 -- 172.21.15.16:0/4004479175 learned_addr learned my addr 172.21.15.16:0/4004479175 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:04.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.844+0000 7fa32a7fc700 1 -- 172.21.15.16:0/4004479175 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa32c072b30 msgr2=0x7fa32c071a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:04.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.844+0000 7fa32a7fc700 1 --2- 172.21.15.16:0/4004479175 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa32c072b30 0x7fa32c071a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:04.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.844+0000 7fa32a7fc700 1 -- 172.21.15.16:0/4004479175 >> v1:172.21.15.49:6789/0 conn(0x7fa32c0caa40 legacy=0x7fa32c0ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:04.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.844+0000 7fa329ffb700 1 -- 172.21.15.16:0/4004479175 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1220369512 0 0) 0x7fa32c15c040 con 0x7fa32c0caa40 2023-10-16T12:30:04.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.844+0000 7fa32a7fc700 1 -- 172.21.15.16:0/4004479175 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa32c15c8e0 con 0x7fa32c15bb40 2023-10-16T12:30:04.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.845+0000 7fa32a7fc700 1 --2- 172.21.15.16:0/4004479175 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa32c15bb40 0x7fa32c06e310 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fa3200098f0 tx=0x7fa320009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d86cd9e1c2f90fd1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:04.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.845+0000 7fa329ffb700 1 -- 172.21.15.16:0/4004479175 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa320011830 con 0x7fa32c15bb40 2023-10-16T12:30:04.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.845+0000 7fa329ffb700 1 -- 172.21.15.16:0/4004479175 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa320011990 con 0x7fa32c15bb40 2023-10-16T12:30:04.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.845+0000 7fa329ffb700 1 -- 172.21.15.16:0/4004479175 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa320011d20 con 0x7fa32c15bb40 2023-10-16T12:30:04.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.845+0000 7fa33194f700 1 -- 172.21.15.16:0/4004479175 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa32c15bb40 msgr2=0x7fa32c06e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:04.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.845+0000 7fa33194f700 1 --2- 172.21.15.16:0/4004479175 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa32c15bb40 0x7fa32c06e310 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fa3200098f0 tx=0x7fa320009c00 comp rx=0 tx=0).stop 2023-10-16T12:30:04.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.845+0000 7fa33194f700 1 -- 172.21.15.16:0/4004479175 shutdown_connections 2023-10-16T12:30:04.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.845+0000 7fa33194f700 1 --2- 172.21.15.16:0/4004479175 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa32c072b30 0x7fa32c071a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:04.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.845+0000 7fa33194f700 1 --2- 172.21.15.16:0/4004479175 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa32c15bb40 0x7fa32c06e310 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:04.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.845+0000 7fa33194f700 1 -- 172.21.15.16:0/4004479175 >> 172.21.15.16:0/4004479175 conn(0x7fa32c0b1f40 msgr2=0x7fa32c0b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:04.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.846+0000 7fa33194f700 1 -- 172.21.15.16:0/4004479175 shutdown_connections 2023-10-16T12:30:04.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.846+0000 7fa33194f700 1 -- 172.21.15.16:0/4004479175 wait complete. 2023-10-16T12:30:04.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.847+0000 7fa33194f700 1 Processor -- start 2023-10-16T12:30:04.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.847+0000 7fa33194f700 1 -- start start 2023-10-16T12:30:04.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.848+0000 7fa33194f700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa32c072b30 0x7fa32c152230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:04.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.848+0000 7fa33194f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa32c0caa40 0x7fa32c157400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:04.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.848+0000 7fa33194f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa32c15bb40 0x7fa32c1f4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:04.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.848+0000 7fa33194f700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa32c1f06a0 con 0x7fa32c0caa40 2023-10-16T12:30:04.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.848+0000 7fa33194f700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa32c0c8eb0 con 0x7fa32c072b30 2023-10-16T12:30:04.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.848+0000 7fa33194f700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa32c0c88f0 con 0x7fa32c15bb40 2023-10-16T12:30:04.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.848+0000 7fa32a7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa32c0caa40 0x7fa32c157400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:04.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.848+0000 7fa32affd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa32c072b30 0x7fa32c152230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:04.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.848+0000 7fa32a7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa32c0caa40 0x7fa32c157400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44522/0 (socket says 172.21.15.16:44522) 2023-10-16T12:30:04.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.848+0000 7fa32a7fc700 1 -- 172.21.15.16:0/165731278 learned_addr learned my addr 172.21.15.16:0/165731278 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:04.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.848+0000 7fa32b7fe700 1 --2- 172.21.15.16:0/165731278 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa32c15bb40 0x7fa32c1f4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:04.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.848+0000 7fa32a7fc700 1 -- 172.21.15.16:0/165731278 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa32c15bb40 msgr2=0x7fa32c1f4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:04.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.848+0000 7fa32a7fc700 1 --2- 172.21.15.16:0/165731278 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa32c15bb40 0x7fa32c1f4720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:04.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.848+0000 7fa32a7fc700 1 -- 172.21.15.16:0/165731278 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa32c072b30 msgr2=0x7fa32c152230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:04.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.848+0000 7fa32a7fc700 1 --2- 172.21.15.16:0/165731278 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa32c072b30 0x7fa32c152230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:04.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.848+0000 7fa32a7fc700 1 -- 172.21.15.16:0/165731278 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa320009550 con 0x7fa32c0caa40 2023-10-16T12:30:04.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.848+0000 7fa32affd700 1 --2- 172.21.15.16:0/165731278 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa32c072b30 0x7fa32c152230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:30:04.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.848+0000 7fa32a7fc700 1 --2- 172.21.15.16:0/165731278 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa32c0caa40 0x7fa32c157400 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7fa320009c40 tx=0x7fa320004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:04.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.849+0000 7fa313fff700 1 -- 172.21.15.16:0/165731278 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa320004300 con 0x7fa32c0caa40 2023-10-16T12:30:04.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.849+0000 7fa313fff700 1 -- 172.21.15.16:0/165731278 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa32001f070 con 0x7fa32c0caa40 2023-10-16T12:30:04.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.849+0000 7fa33194f700 1 -- 172.21.15.16:0/165731278 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa32c0c8bd0 con 0x7fa32c0caa40 2023-10-16T12:30:04.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.849+0000 7fa313fff700 1 -- 172.21.15.16:0/165731278 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3200036a0 con 0x7fa32c0caa40 2023-10-16T12:30:04.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.849+0000 7fa33194f700 1 -- 172.21.15.16:0/165731278 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa32c1f4200 con 0x7fa32c0caa40 2023-10-16T12:30:04.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.850+0000 7fa33194f700 1 -- 172.21.15.16:0/165731278 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa32c04ebb0 con 0x7fa32c0caa40 2023-10-16T12:30:04.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.850+0000 7fa313fff700 1 -- 172.21.15.16:0/165731278 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 70) v1 ==== 89445+0+0 (secure 0 0 0) 0x7fa320021030 con 0x7fa32c0caa40 2023-10-16T12:30:04.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.851+0000 7fa313fff700 1 --2- 172.21.15.16:0/165731278 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7fa31806b170 0x7fa31806d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:04.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.851+0000 7fa32affd700 1 -- 172.21.15.16:0/165731278 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7fa31806b170 msgr2=0x7fa31806d630 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2550537276 2023-10-16T12:30:04.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.851+0000 7fa313fff700 1 -- 172.21.15.16:0/165731278 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fa32008b000 con 0x7fa32c0caa40 2023-10-16T12:30:04.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.851+0000 7fa32affd700 1 --2- 172.21.15.16:0/165731278 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7fa31806b170 0x7fa31806d630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:30:04.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:04.855+0000 7fa313fff700 1 -- 172.21.15.16:0/165731278 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa320055bc0 con 0x7fa32c0caa40 2023-10-16T12:30:04.874 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:30:04.873+0000 7f71a592b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T12:30:04.906 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:04.904+0000 7fc68f8d0200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T12:30:05.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.052+0000 7fa32affd700 1 -- 172.21.15.16:0/165731278 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7fa31806b170 msgr2=0x7fa31806d630 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2550537276 2023-10-16T12:30:05.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.052+0000 7fa32affd700 1 --2- 172.21.15.16:0/165731278 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7fa31806b170 0x7fa31806d630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:30:05.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.058+0000 7fa33194f700 1 -- 172.21.15.16:0/165731278 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa32c0b2d40 con 0x7fa32c0caa40 2023-10-16T12:30:05.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.058+0000 7fa313fff700 1 -- 172.21.15.16:0/165731278 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v129) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa3200150a0 con 0x7fa32c0caa40 2023-10-16T12:30:05.060 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-10-16T12:30:05.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.060+0000 7fa33194f700 1 -- 172.21.15.16:0/165731278 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7fa31806b170 msgr2=0x7fa31806d630 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:30:05.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.060+0000 7fa33194f700 1 --2- 172.21.15.16:0/165731278 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7fa31806b170 0x7fa31806d630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:05.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.060+0000 7fa33194f700 1 -- 172.21.15.16:0/165731278 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa32c0caa40 msgr2=0x7fa32c157400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:05.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.060+0000 7fa33194f700 1 --2- 172.21.15.16:0/165731278 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa32c0caa40 0x7fa32c157400 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7fa320009c40 tx=0x7fa320004750 comp rx=0 tx=0).stop 2023-10-16T12:30:05.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.060+0000 7fa33194f700 1 -- 172.21.15.16:0/165731278 shutdown_connections 2023-10-16T12:30:05.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.060+0000 7fa33194f700 1 --2- 172.21.15.16:0/165731278 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7fa31806b170 0x7fa31806d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:05.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.060+0000 7fa33194f700 1 --2- 172.21.15.16:0/165731278 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa32c15bb40 0x7fa32c1f4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:05.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.060+0000 7fa33194f700 1 --2- 172.21.15.16:0/165731278 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa32c072b30 0x7fa32c152230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:05.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.060+0000 7fa33194f700 1 --2- 172.21.15.16:0/165731278 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa32c0caa40 0x7fa32c157400 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:05.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.060+0000 7fa33194f700 1 -- 172.21.15.16:0/165731278 >> 172.21.15.16:0/165731278 conn(0x7fa32c0b1f40 msgr2=0x7fa32c0b2390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:05.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.060+0000 7fa33194f700 1 -- 172.21.15.16:0/165731278 shutdown_connections 2023-10-16T12:30:05.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.060+0000 7fa33194f700 1 -- 172.21.15.16:0/165731278 wait complete. 2023-10-16T12:30:05.076 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T12:30:05.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.200+0000 7fd96348e700 1 Processor -- start 2023-10-16T12:30:05.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.201+0000 7fd96348e700 1 -- start start 2023-10-16T12:30:05.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.202+0000 7fd96348e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd95c0cdfa0 0x7fd95c0b2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:05.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.202+0000 7fd96348e700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fd95c0ce380 0x7fd95c0b2980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:05.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.202+0000 7fd96348e700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd95c0d0fe0 con 0x7fd95c0d04a0 2023-10-16T12:30:05.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.202+0000 7fd96348e700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd95c0d1210 con 0x7fd95c0ce380 2023-10-16T12:30:05.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.202+0000 7fd96348e700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd95c0cbc30 con 0x7fd95c0cdfa0 2023-10-16T12:30:05.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.202+0000 7fd96122a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd95c0cdfa0 0x7fd95c0b2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:05.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.202+0000 7fd960a29700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fd95c0ce380 0x7fd95c0b2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:05.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.202+0000 7fd961a2b700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7fd95c0d04a0 0x7fd95c0d08d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:40988/0 (socket says 172.21.15.16:40988) 2023-10-16T12:30:05.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.202+0000 7fd961a2b700 1 -- 172.21.15.16:0/2985001946 learned_addr learned my addr 172.21.15.16:0/2985001946 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:30:05.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.202+0000 7fd960a29700 1 -- 172.21.15.16:0/2985001946 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd95c0cdfa0 msgr2=0x7fd95c0b2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:05.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.202+0000 7fd960a29700 1 --2- 172.21.15.16:0/2985001946 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd95c0cdfa0 0x7fd95c0b2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:05.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.202+0000 7fd960a29700 1 -- 172.21.15.16:0/2985001946 >> v1:172.21.15.49:6789/0 conn(0x7fd95c0d04a0 legacy=0x7fd95c0d08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:05.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.202+0000 7fd960a29700 1 -- 172.21.15.16:0/2985001946 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd95c0cbf10 con 0x7fd95c0ce380 2023-10-16T12:30:05.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.202+0000 7fd953fff700 1 -- 172.21.15.16:0/2985001946 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1897031540 0 0) 0x7fd95c0d0fe0 con 0x7fd95c0d04a0 2023-10-16T12:30:05.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.202+0000 7fd96122a700 1 --2- 172.21.15.16:0/2985001946 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd95c0cdfa0 0x7fd95c0b2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:30:05.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.202+0000 7fd960a29700 1 --2- 172.21.15.16:0/2985001946 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd95c0ce380 0x7fd95c0b2980 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fd94c009840 tx=0x7fd94c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e34cbafd77e18111 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:05.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.203+0000 7fd953fff700 1 -- 172.21.15.16:0/2985001946 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd94c011830 con 0x7fd95c0ce380 2023-10-16T12:30:05.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.203+0000 7fd953fff700 1 -- 172.21.15.16:0/2985001946 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd94c011990 con 0x7fd95c0ce380 2023-10-16T12:30:05.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.203+0000 7fd953fff700 1 -- 172.21.15.16:0/2985001946 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd94c011d20 con 0x7fd95c0ce380 2023-10-16T12:30:05.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.203+0000 7fd96348e700 1 -- 172.21.15.16:0/2985001946 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd95c0ce380 msgr2=0x7fd95c0b2980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:05.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.203+0000 7fd96348e700 1 --2- 172.21.15.16:0/2985001946 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd95c0ce380 0x7fd95c0b2980 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fd94c009840 tx=0x7fd94c009b50 comp rx=0 tx=0).stop 2023-10-16T12:30:05.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.203+0000 7fd96348e700 1 -- 172.21.15.16:0/2985001946 shutdown_connections 2023-10-16T12:30:05.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.203+0000 7fd96348e700 1 --2- 172.21.15.16:0/2985001946 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd95c0cdfa0 0x7fd95c0b2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:05.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.203+0000 7fd96348e700 1 --2- 172.21.15.16:0/2985001946 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd95c0ce380 0x7fd95c0b2980 secure :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fd94c009840 tx=0x7fd94c009b50 comp rx=0 tx=0).stop 2023-10-16T12:30:05.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.203+0000 7fd96348e700 1 -- 172.21.15.16:0/2985001946 >> 172.21.15.16:0/2985001946 conn(0x7fd95c0d14b0 msgr2=0x7fd95c0d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:05.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.203+0000 7fd96348e700 1 -- 172.21.15.16:0/2985001946 shutdown_connections 2023-10-16T12:30:05.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.203+0000 7fd96348e700 1 -- 172.21.15.16:0/2985001946 wait complete. 2023-10-16T12:30:05.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.205+0000 7fd96348e700 1 Processor -- start 2023-10-16T12:30:05.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.205+0000 7fd96348e700 1 -- start start 2023-10-16T12:30:05.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.206+0000 7fd96348e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd95c0cdfa0 0x7fd95c15f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:05.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.206+0000 7fd96348e700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd95c0d04a0 0x7fd95c15d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:05.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.206+0000 7fd96348e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd95c1f2f50 0x7fd95c15e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:05.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.206+0000 7fd96348e700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd95c1f3fe0 con 0x7fd95c0cdfa0 2023-10-16T12:30:05.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.206+0000 7fd96348e700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd95c15f970 con 0x7fd95c0d04a0 2023-10-16T12:30:05.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.206+0000 7fd96348e700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd95c1f3460 con 0x7fd95c1f2f50 2023-10-16T12:30:05.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.206+0000 7fd96122a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd95c0cdfa0 0x7fd95c15f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:05.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.206+0000 7fd96122a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd95c0cdfa0 0x7fd95c15f430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44534/0 (socket says 172.21.15.16:44534) 2023-10-16T12:30:05.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.206+0000 7fd96122a700 1 -- 172.21.15.16:0/3921415399 learned_addr learned my addr 172.21.15.16:0/3921415399 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:05.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.206+0000 7fd960a29700 1 --2- 172.21.15.16:0/3921415399 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd95c0d04a0 0x7fd95c15d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:05.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.206+0000 7fd961a2b700 1 --2- 172.21.15.16:0/3921415399 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd95c1f2f50 0x7fd95c15e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:05.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.207+0000 7fd96122a700 1 -- 172.21.15.16:0/3921415399 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd95c1f2f50 msgr2=0x7fd95c15e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:05.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.207+0000 7fd96122a700 1 --2- 172.21.15.16:0/3921415399 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd95c1f2f50 0x7fd95c15e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:05.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.207+0000 7fd96122a700 1 -- 172.21.15.16:0/3921415399 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd95c0d04a0 msgr2=0x7fd95c15d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:05.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.207+0000 7fd96122a700 1 --2- 172.21.15.16:0/3921415399 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd95c0d04a0 0x7fd95c15d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:05.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.207+0000 7fd96122a700 1 -- 172.21.15.16:0/3921415399 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9580094a0 con 0x7fd95c0cdfa0 2023-10-16T12:30:05.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.207+0000 7fd960a29700 1 --2- 172.21.15.16:0/3921415399 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd95c0d04a0 0x7fd95c15d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:30:05.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.207+0000 7fd96122a700 1 --2- 172.21.15.16:0/3921415399 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd95c0cdfa0 0x7fd95c15f430 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7fd95800ebe0 tx=0x7fd95800eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:05.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.207+0000 7fd951ffb700 1 -- 172.21.15.16:0/3921415399 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd95800c850 con 0x7fd95c0cdfa0 2023-10-16T12:30:05.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.207+0000 7fd951ffb700 1 -- 172.21.15.16:0/3921415399 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd95800c9b0 con 0x7fd95c0cdfa0 2023-10-16T12:30:05.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.207+0000 7fd96348e700 1 -- 172.21.15.16:0/3921415399 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd94c0094a0 con 0x7fd95c0cdfa0 2023-10-16T12:30:05.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.207+0000 7fd951ffb700 1 -- 172.21.15.16:0/3921415399 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd95800cd40 con 0x7fd95c0cdfa0 2023-10-16T12:30:05.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.207+0000 7fd96348e700 1 -- 172.21.15.16:0/3921415399 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd95c1f3a20 con 0x7fd95c0cdfa0 2023-10-16T12:30:05.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.208+0000 7fd96348e700 1 -- 172.21.15.16:0/3921415399 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd95c0c5210 con 0x7fd95c0cdfa0 2023-10-16T12:30:05.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.209+0000 7fd951ffb700 1 -- 172.21.15.16:0/3921415399 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 70) v1 ==== 89445+0+0 (secure 0 0 0) 0x7fd958013470 con 0x7fd95c0cdfa0 2023-10-16T12:30:05.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.209+0000 7fd951ffb700 1 --2- 172.21.15.16:0/3921415399 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7fd94006b140 0x7fd94006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:05.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.209+0000 7fd960a29700 1 -- 172.21.15.16:0/3921415399 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7fd94006b140 msgr2=0x7fd94006d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2550537276 2023-10-16T12:30:05.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.209+0000 7fd951ffb700 1 -- 172.21.15.16:0/3921415399 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fd95808a480 con 0x7fd95c0cdfa0 2023-10-16T12:30:05.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.209+0000 7fd960a29700 1 --2- 172.21.15.16:0/3921415399 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7fd94006b140 0x7fd94006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:30:05.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.213+0000 7fd951ffb700 1 -- 172.21.15.16:0/3921415399 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd958054ff0 con 0x7fd95c0cdfa0 2023-10-16T12:30:05.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.410+0000 7fd960a29700 1 -- 172.21.15.16:0/3921415399 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7fd94006b140 msgr2=0x7fd94006d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2550537276 2023-10-16T12:30:05.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.410+0000 7fd960a29700 1 --2- 172.21.15.16:0/3921415399 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7fd94006b140 0x7fd94006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:30:05.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.481+0000 7fd96348e700 1 -- 172.21.15.16:0/3921415399 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fd95c04ebb0 con 0x7fd95c0cdfa0 2023-10-16T12:30:05.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.481+0000 7fd951ffb700 1 -- 172.21.15.16:0/3921415399 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v129) v1 ==== 80+0+468 (secure 0 0 0) 0x7fd95800cea0 con 0x7fd95c0cdfa0 2023-10-16T12:30:05.484 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/32319707 2023-10-17T12:29:30.732445+0000 2023-10-16T12:30:05.484 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/1058620826 2023-10-17T12:29:30.732445+0000 2023-10-16T12:30:05.484 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3098064509 2023-10-17T12:29:30.732445+0000 2023-10-16T12:30:05.484 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3814896496 2023-10-17T12:29:30.732445+0000 2023-10-16T12:30:05.484 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2973963882 2023-10-17T12:29:30.732445+0000 2023-10-16T12:30:05.485 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6833/2309613638 2023-10-17T12:29:30.732445+0000 2023-10-16T12:30:05.485 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6832/2309613638 2023-10-17T12:29:30.732445+0000 2023-10-16T12:30:05.485 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/1074148914 2023-10-17T12:29:30.732445+0000 2023-10-16T12:30:05.486 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:30:05.481+0000 7f71a592b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T12:30:05.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.483+0000 7fd96348e700 1 -- 172.21.15.16:0/3921415399 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7fd94006b140 msgr2=0x7fd94006d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:30:05.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.483+0000 7fd96348e700 1 --2- 172.21.15.16:0/3921415399 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7fd94006b140 0x7fd94006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:05.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.483+0000 7fd96348e700 1 -- 172.21.15.16:0/3921415399 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd95c0cdfa0 msgr2=0x7fd95c15f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:05.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.483+0000 7fd96348e700 1 --2- 172.21.15.16:0/3921415399 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd95c0cdfa0 0x7fd95c15f430 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7fd95800ebe0 tx=0x7fd95800eef0 comp rx=0 tx=0).stop 2023-10-16T12:30:05.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.483+0000 7fd96348e700 1 -- 172.21.15.16:0/3921415399 shutdown_connections 2023-10-16T12:30:05.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.483+0000 7fd96348e700 1 --2- 172.21.15.16:0/3921415399 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7fd94006b140 0x7fd94006d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:05.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.483+0000 7fd96348e700 1 --2- 172.21.15.16:0/3921415399 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd95c1f2f50 0x7fd95c15e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:05.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.483+0000 7fd96348e700 1 --2- 172.21.15.16:0/3921415399 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd95c0cdfa0 0x7fd95c15f430 unknown :-1 s=CLOSED pgs=1445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:05.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.483+0000 7fd96348e700 1 --2- 172.21.15.16:0/3921415399 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd95c0d04a0 0x7fd95c15d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:05.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.483+0000 7fd96348e700 1 -- 172.21.15.16:0/3921415399 >> 172.21.15.16:0/3921415399 conn(0x7fd95c0d14b0 msgr2=0x7fd95c1609c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:05.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.484+0000 7fd96348e700 1 -- 172.21.15.16:0/3921415399 shutdown_connections 2023-10-16T12:30:05.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.484+0000 7fd96348e700 1 -- 172.21.15.16:0/3921415399 wait complete. 2023-10-16T12:30:05.489 INFO:teuthology.orchestra.run.smithi016.stderr:listed 8 entries 2023-10-16T12:30:05.499 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-16T12:30:05.500 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:30:05.500 DEBUG:teuthology.orchestra.run.smithi016:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-16T12:30:05.522 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:05.521+0000 7fc68f8d0200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T12:30:05.545 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2023-10-16T12:30:05.546 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:30:05.546 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-16T12:30:05.556 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:30:05.555+0000 7f71a592b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T12:30:05.600 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:05.598+0000 7fc68f8d0200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T12:30:05.603 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T12:30:05.603 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:30:05.620 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284204k,nr_inodes=4071051,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=753 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260540k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T12:30:05.621 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:30:05.621 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:30:05.660 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16287364k,nr_inodes=4071841,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20251 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3261172k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T12:30:05.661 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T12:30:05.661 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:30:05.678 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284204k,nr_inodes=4071051,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=753 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260540k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T12:30:05.679 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:30:05.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.825+0000 7ff38dbff700 1 Processor -- start 2023-10-16T12:30:05.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.825+0000 7ff38dbff700 1 -- start start 2023-10-16T12:30:05.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.826+0000 7ff38dbff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff38815b280 0x7ff3881613e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:05.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.826+0000 7ff38dbff700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff3880cc8a0 0x7ff38815e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:05.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.826+0000 7ff38dbff700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff388161c30 con 0x7ff38806a7e0 2023-10-16T12:30:05.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.826+0000 7ff38dbff700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff388161f10 con 0x7ff38815b280 2023-10-16T12:30:05.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.826+0000 7ff38dbff700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3881621f0 con 0x7ff3880cc8a0 2023-10-16T12:30:05.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.826+0000 7ff3877fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff38815b280 0x7ff3881613e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:05.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.826+0000 7ff3877fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff38815b280 0x7ff3881613e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44544/0 (socket says 172.21.15.16:44544) 2023-10-16T12:30:05.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.826+0000 7ff3877fe700 1 -- 172.21.15.16:0/4161857403 learned_addr learned my addr 172.21.15.16:0/4161857403 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:05.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.826+0000 7ff386ffd700 1 --2- 172.21.15.16:0/4161857403 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff3880cc8a0 0x7ff38815e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:05.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.826+0000 7ff3877fe700 1 -- 172.21.15.16:0/4161857403 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff3880cc8a0 msgr2=0x7ff38815e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:05.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.826+0000 7ff3877fe700 1 --2- 172.21.15.16:0/4161857403 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff3880cc8a0 0x7ff38815e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:05.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.826+0000 7ff3877fe700 1 -- 172.21.15.16:0/4161857403 >> v1:172.21.15.49:6789/0 conn(0x7ff38806a7e0 legacy=0x7ff3880d02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:05.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.826+0000 7ff3877fe700 1 -- 172.21.15.16:0/4161857403 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff38815df00 con 0x7ff38815b280 2023-10-16T12:30:05.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.826+0000 7ff3877fe700 1 --2- 172.21.15.16:0/4161857403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff38815b280 0x7ff3881613e0 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7ff3780098f0 tx=0x7ff378009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=443838117cf27622 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:05.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.826+0000 7ff3867fc700 1 -- 172.21.15.16:0/4161857403 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff378011830 con 0x7ff38815b280 2023-10-16T12:30:05.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.826+0000 7ff3867fc700 1 -- 172.21.15.16:0/4161857403 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff378011990 con 0x7ff38815b280 2023-10-16T12:30:05.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.827+0000 7ff3867fc700 1 -- 172.21.15.16:0/4161857403 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff378011d20 con 0x7ff38815b280 2023-10-16T12:30:05.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.827+0000 7ff38dbff700 1 -- 172.21.15.16:0/4161857403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff38815b280 msgr2=0x7ff3881613e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:05.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.827+0000 7ff38dbff700 1 --2- 172.21.15.16:0/4161857403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff38815b280 0x7ff3881613e0 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7ff3780098f0 tx=0x7ff378009c00 comp rx=0 tx=0).stop 2023-10-16T12:30:05.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.827+0000 7ff38dbff700 1 -- 172.21.15.16:0/4161857403 shutdown_connections 2023-10-16T12:30:05.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.827+0000 7ff38dbff700 1 --2- 172.21.15.16:0/4161857403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff38815b280 0x7ff3881613e0 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:05.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.827+0000 7ff38dbff700 1 --2- 172.21.15.16:0/4161857403 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff3880cc8a0 0x7ff38815e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:05.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.827+0000 7ff38dbff700 1 -- 172.21.15.16:0/4161857403 >> 172.21.15.16:0/4161857403 conn(0x7ff3880b1fd0 msgr2=0x7ff3880b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:05.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.827+0000 7ff38dbff700 1 -- 172.21.15.16:0/4161857403 shutdown_connections 2023-10-16T12:30:05.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.827+0000 7ff38dbff700 1 -- 172.21.15.16:0/4161857403 wait complete. 2023-10-16T12:30:05.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.828+0000 7ff38dbff700 1 Processor -- start 2023-10-16T12:30:05.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.829+0000 7ff38dbff700 1 -- start start 2023-10-16T12:30:05.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.829+0000 7ff38dbff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff38806a7e0 0x7ff3881525d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:05.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.829+0000 7ff38dbff700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff3880cc8a0 0x7ff38815cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:05.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.829+0000 7ff38dbff700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff38815b280 0x7ff38815d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:05.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.829+0000 7ff38dbff700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff38815b660 con 0x7ff38806a7e0 2023-10-16T12:30:05.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.829+0000 7ff38dbff700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff38815b8b0 con 0x7ff38815b280 2023-10-16T12:30:05.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.829+0000 7ff38dbff700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff38815bb90 con 0x7ff3880cc8a0 2023-10-16T12:30:05.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.829+0000 7ff3877fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff38806a7e0 0x7ff3881525d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:05.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.829+0000 7ff3877fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff38806a7e0 0x7ff3881525d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44550/0 (socket says 172.21.15.16:44550) 2023-10-16T12:30:05.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.829+0000 7ff3877fe700 1 -- 172.21.15.16:0/1887115583 learned_addr learned my addr 172.21.15.16:0/1887115583 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:05.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.829+0000 7ff387fff700 1 --2- 172.21.15.16:0/1887115583 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff38815b280 0x7ff38815d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:05.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.829+0000 7ff386ffd700 1 --2- 172.21.15.16:0/1887115583 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff3880cc8a0 0x7ff38815cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:05.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.830+0000 7ff3877fe700 1 -- 172.21.15.16:0/1887115583 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff3880cc8a0 msgr2=0x7ff38815cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:05.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.830+0000 7ff3877fe700 1 --2- 172.21.15.16:0/1887115583 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff3880cc8a0 0x7ff38815cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:05.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.830+0000 7ff3877fe700 1 -- 172.21.15.16:0/1887115583 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff38815b280 msgr2=0x7ff38815d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:05.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.830+0000 7ff3877fe700 1 --2- 172.21.15.16:0/1887115583 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff38815b280 0x7ff38815d0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:05.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.830+0000 7ff3877fe700 1 -- 172.21.15.16:0/1887115583 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff378009550 con 0x7ff38806a7e0 2023-10-16T12:30:05.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.830+0000 7ff3877fe700 1 --2- 172.21.15.16:0/1887115583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff38806a7e0 0x7ff3881525d0 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7ff378009c40 tx=0x7ff378004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:05.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.830+0000 7ff36ffff700 1 -- 172.21.15.16:0/1887115583 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff378004510 con 0x7ff38806a7e0 2023-10-16T12:30:05.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.830+0000 7ff36ffff700 1 -- 172.21.15.16:0/1887115583 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff37801e070 con 0x7ff38806a7e0 2023-10-16T12:30:05.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.830+0000 7ff36ffff700 1 -- 172.21.15.16:0/1887115583 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3780038e0 con 0x7ff38806a7e0 2023-10-16T12:30:05.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.830+0000 7ff38dbff700 1 -- 172.21.15.16:0/1887115583 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff38815be70 con 0x7ff38806a7e0 2023-10-16T12:30:05.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.830+0000 7ff38dbff700 1 -- 172.21.15.16:0/1887115583 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff38815c430 con 0x7ff38806a7e0 2023-10-16T12:30:05.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.831+0000 7ff38dbff700 1 -- 172.21.15.16:0/1887115583 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3881f3dd0 con 0x7ff38806a7e0 2023-10-16T12:30:05.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.832+0000 7ff36ffff700 1 -- 172.21.15.16:0/1887115583 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 70) v1 ==== 89445+0+0 (secure 0 0 0) 0x7ff378003a40 con 0x7ff38806a7e0 2023-10-16T12:30:05.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.832+0000 7ff36ffff700 1 --2- 172.21.15.16:0/1887115583 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7ff37406b140 0x7ff37406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:05.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.832+0000 7ff386ffd700 1 -- 172.21.15.16:0/1887115583 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7ff37406b140 msgr2=0x7ff37406d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2550537276 2023-10-16T12:30:05.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.832+0000 7ff36ffff700 1 -- 172.21.15.16:0/1887115583 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6401+0+0 (secure 0 0 0) 0x7ff37808aa60 con 0x7ff38806a7e0 2023-10-16T12:30:05.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.832+0000 7ff386ffd700 1 --2- 172.21.15.16:0/1887115583 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7ff37406b140 0x7ff37406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:30:05.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:05.837+0000 7ff36ffff700 1 -- 172.21.15.16:0/1887115583 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff3780555d0 con 0x7ff38806a7e0 2023-10-16T12:30:06.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.033+0000 7ff386ffd700 1 -- 172.21.15.16:0/1887115583 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7ff37406b140 msgr2=0x7ff37406d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2550537276 2023-10-16T12:30:06.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.033+0000 7ff386ffd700 1 --2- 172.21.15.16:0/1887115583 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7ff37406b140 0x7ff37406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:30:06.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.090+0000 7ff38dbff700 1 -- 172.21.15.16:0/1887115583 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff38804ebb0 con 0x7ff38806a7e0 2023-10-16T12:30:06.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.091+0000 7ff36ffff700 1 -- 172.21.15.16:0/1887115583 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+4909 (secure 0 0 0) 0x7ff378058bf0 con 0x7ff38806a7e0 2023-10-16T12:30:06.093 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:30:06.093 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":61,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6893,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/2971585675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":2971585675},{"type":"v1","addr":"172.21.15.49:6835","nonce":2971585675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":7723,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/3151669585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":3151669585},{"type":"v1","addr":"172.21.15.49:6834","nonce":3151669585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":8106,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/52924937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":52924937},{"type":"v1","addr":"172.21.15.16:6836","nonce":52924937}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:29:15.801502+0000","modified":"2023-10-16T12:29:18.806285+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8109},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8109":{"gid":8109,"name":"a","rank":0,"incarnation":57,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/3203451346","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":3203451346},{"type":"v1","addr":"172.21.15.16:6837","nonce":3203451346}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2023-10-16T12:30:06.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.093+0000 7ff38dbff700 1 -- 172.21.15.16:0/1887115583 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7ff37406b140 msgr2=0x7ff37406d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:30:06.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.093+0000 7ff38dbff700 1 --2- 172.21.15.16:0/1887115583 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7ff37406b140 0x7ff37406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:06.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.093+0000 7ff38dbff700 1 -- 172.21.15.16:0/1887115583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff38806a7e0 msgr2=0x7ff3881525d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:06.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.093+0000 7ff38dbff700 1 --2- 172.21.15.16:0/1887115583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff38806a7e0 0x7ff3881525d0 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7ff378009c40 tx=0x7ff378004750 comp rx=0 tx=0).stop 2023-10-16T12:30:06.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.093+0000 7ff38dbff700 1 -- 172.21.15.16:0/1887115583 shutdown_connections 2023-10-16T12:30:06.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.093+0000 7ff38dbff700 1 --2- 172.21.15.16:0/1887115583 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7ff37406b140 0x7ff37406d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:06.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.093+0000 7ff38dbff700 1 --2- 172.21.15.16:0/1887115583 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff38815b280 0x7ff38815d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:06.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.093+0000 7ff38dbff700 1 --2- 172.21.15.16:0/1887115583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff38806a7e0 0x7ff3881525d0 unknown :-1 s=CLOSED pgs=1447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:06.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.093+0000 7ff38dbff700 1 --2- 172.21.15.16:0/1887115583 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff3880cc8a0 0x7ff38815cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:06.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.093+0000 7ff38dbff700 1 -- 172.21.15.16:0/1887115583 >> 172.21.15.16:0/1887115583 conn(0x7ff3880b1fd0 msgr2=0x7ff3880cce10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:06.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.093+0000 7ff38dbff700 1 -- 172.21.15.16:0/1887115583 shutdown_connections 2023-10-16T12:30:06.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.093+0000 7ff38dbff700 1 -- 172.21.15.16:0/1887115583 wait complete. 2023-10-16T12:30:06.098 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 61 2023-10-16T12:30:06.108 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:30:06.225 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:30:06.223+0000 7f71a592b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:30:06.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.236+0000 7f3339f1c700 1 Processor -- start 2023-10-16T12:30:06.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.236+0000 7f3339f1c700 1 -- start start 2023-10-16T12:30:06.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.236+0000 7f3339f1c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f333415b280 0x7f33341613e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:06.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.236+0000 7f3339f1c700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f333406a7e0 0x7f333406ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:06.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.236+0000 7f3339f1c700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f333406b150 con 0x7f33340cc8a0 2023-10-16T12:30:06.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.236+0000 7f3339f1c700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3334161c30 con 0x7f333415b280 2023-10-16T12:30:06.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.236+0000 7f3339f1c700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3334161f10 con 0x7f333406a7e0 2023-10-16T12:30:06.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.237+0000 7f33337fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f333415b280 0x7f33341613e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:06.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.237+0000 7f332affd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f33340cc8a0 0x7f333415e700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33834/0 (socket says 172.21.15.16:33834) 2023-10-16T12:30:06.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.237+0000 7f33337fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f333415b280 0x7f33341613e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44560/0 (socket says 172.21.15.16:44560) 2023-10-16T12:30:06.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.237+0000 7f332affd700 1 -- 172.21.15.16:0/3762490619 learned_addr learned my addr 172.21.15.16:0/3762490619 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:30:06.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.237+0000 7f3333fff700 1 --2- 172.21.15.16:0/3762490619 >> v2:172.21.15.49:3300/0 conn(0x7f333406a7e0 0x7f333406ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:06.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.237+0000 7f33337fe700 1 -- 172.21.15.16:0/3762490619 >> v2:172.21.15.49:3300/0 conn(0x7f333406a7e0 msgr2=0x7f333406ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:06.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.237+0000 7f33337fe700 1 --2- 172.21.15.16:0/3762490619 >> v2:172.21.15.49:3300/0 conn(0x7f333406a7e0 0x7f333406ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:06.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.237+0000 7f33337fe700 1 -- 172.21.15.16:0/3762490619 >> v1:172.21.15.16:6789/0 conn(0x7f33340cc8a0 legacy=0x7f333415e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:06.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.237+0000 7f33337fe700 1 -- 172.21.15.16:0/3762490619 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33341621f0 con 0x7f333415b280 2023-10-16T12:30:06.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.237+0000 7f3332ffd700 1 -- 172.21.15.16:0/3762490619 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 213227678 0 0) 0x7f333406b150 con 0x7f33340cc8a0 2023-10-16T12:30:06.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.237+0000 7f33337fe700 1 --2- 172.21.15.16:0/3762490619 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f333415b280 0x7f33341613e0 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f33340cbed0 tx=0x7f3324009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aec10cffcd778272 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:06.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.237+0000 7f3332ffd700 1 -- 172.21.15.16:0/3762490619 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3324011830 con 0x7f333415b280 2023-10-16T12:30:06.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.237+0000 7f3332ffd700 1 -- 172.21.15.16:0/3762490619 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3324011990 con 0x7f333415b280 2023-10-16T12:30:06.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.237+0000 7f3332ffd700 1 -- 172.21.15.16:0/3762490619 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3324011d20 con 0x7f333415b280 2023-10-16T12:30:06.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.238+0000 7f3339f1c700 1 -- 172.21.15.16:0/3762490619 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f333415b280 msgr2=0x7f33341613e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:06.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.238+0000 7f3339f1c700 1 --2- 172.21.15.16:0/3762490619 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f333415b280 0x7f33341613e0 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f33340cbed0 tx=0x7f3324009b20 comp rx=0 tx=0).stop 2023-10-16T12:30:06.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.238+0000 7f3339f1c700 1 -- 172.21.15.16:0/3762490619 shutdown_connections 2023-10-16T12:30:06.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.238+0000 7f3339f1c700 1 --2- 172.21.15.16:0/3762490619 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f333415b280 0x7f33341613e0 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:06.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.238+0000 7f3339f1c700 1 --2- 172.21.15.16:0/3762490619 >> v2:172.21.15.49:3300/0 conn(0x7f333406a7e0 0x7f333406ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:06.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.238+0000 7f3339f1c700 1 -- 172.21.15.16:0/3762490619 >> 172.21.15.16:0/3762490619 conn(0x7f33340b1fd0 msgr2=0x7f33340b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:06.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.238+0000 7f3339f1c700 1 -- 172.21.15.16:0/3762490619 shutdown_connections 2023-10-16T12:30:06.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.238+0000 7f3339f1c700 1 -- 172.21.15.16:0/3762490619 wait complete. 2023-10-16T12:30:06.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.240+0000 7f3339f1c700 1 Processor -- start 2023-10-16T12:30:06.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.240+0000 7f3339f1c700 1 -- start start 2023-10-16T12:30:06.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.241+0000 7f3339f1c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f333406a7e0 0x7f3334148c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:06.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.241+0000 7f3339f1c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33340cc8a0 0x7f33340c7f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:06.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.241+0000 7f3339f1c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f333415b280 0x7f333415f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:06.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.241+0000 7f3339f1c700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33340c6060 con 0x7f33340cc8a0 2023-10-16T12:30:06.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.241+0000 7f3339f1c700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33340c5450 con 0x7f333406a7e0 2023-10-16T12:30:06.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.241+0000 7f3339f1c700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33340c5730 con 0x7f333415b280 2023-10-16T12:30:06.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.241+0000 7f332affd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33340cc8a0 0x7f33340c7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:06.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.241+0000 7f332affd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33340cc8a0 0x7f33340c7f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44568/0 (socket says 172.21.15.16:44568) 2023-10-16T12:30:06.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.241+0000 7f332affd700 1 -- 172.21.15.16:0/3012626906 learned_addr learned my addr 172.21.15.16:0/3012626906 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:06.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.241+0000 7f33337fe700 1 --2- 172.21.15.16:0/3012626906 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f333406a7e0 0x7f3334148c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:06.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.241+0000 7f3333fff700 1 --2- 172.21.15.16:0/3012626906 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f333415b280 0x7f333415f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:06.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.241+0000 7f332affd700 1 -- 172.21.15.16:0/3012626906 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f333415b280 msgr2=0x7f333415f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:06.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.241+0000 7f332affd700 1 --2- 172.21.15.16:0/3012626906 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f333415b280 0x7f333415f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:06.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.241+0000 7f332affd700 1 -- 172.21.15.16:0/3012626906 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f333406a7e0 msgr2=0x7f3334148c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:06.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.241+0000 7f332affd700 1 --2- 172.21.15.16:0/3012626906 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f333406a7e0 0x7f3334148c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:06.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.241+0000 7f332affd700 1 -- 172.21.15.16:0/3012626906 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33240094a0 con 0x7f33340cc8a0 2023-10-16T12:30:06.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.241+0000 7f33337fe700 1 --2- 172.21.15.16:0/3012626906 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f333406a7e0 0x7f3334148c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:30:06.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.241+0000 7f332affd700 1 --2- 172.21.15.16:0/3012626906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33340cc8a0 0x7f33340c7f20 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f331c00c540 tx=0x7f331c00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:06.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.241+0000 7f3330ff9700 1 -- 172.21.15.16:0/3012626906 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f331c004c80 con 0x7f33340cc8a0 2023-10-16T12:30:06.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.242+0000 7f3330ff9700 1 -- 172.21.15.16:0/3012626906 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f331c004de0 con 0x7f33340cc8a0 2023-10-16T12:30:06.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.242+0000 7f3339f1c700 1 -- 172.21.15.16:0/3012626906 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33340c5a10 con 0x7f33340cc8a0 2023-10-16T12:30:06.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.242+0000 7f3330ff9700 1 -- 172.21.15.16:0/3012626906 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f331c007310 con 0x7f33340cc8a0 2023-10-16T12:30:06.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.242+0000 7f3339f1c700 1 -- 172.21.15.16:0/3012626906 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3334160990 con 0x7f33340cc8a0 2023-10-16T12:30:06.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.243+0000 7f3339f1c700 1 -- 172.21.15.16:0/3012626906 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33340c6210 con 0x7f33340cc8a0 2023-10-16T12:30:06.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.243+0000 7f3330ff9700 1 -- 172.21.15.16:0/3012626906 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 70) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f331c007470 con 0x7f33340cc8a0 2023-10-16T12:30:06.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.243+0000 7f3330ff9700 1 --2- 172.21.15.16:0/3012626906 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7f331406b140 0x7f331406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:06.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.243+0000 7f33337fe700 1 -- 172.21.15.16:0/3012626906 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7f331406b140 msgr2=0x7f331406d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2550537276 2023-10-16T12:30:06.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.244+0000 7f3330ff9700 1 -- 172.21.15.16:0/3012626906 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f331c0898e0 con 0x7f33340cc8a0 2023-10-16T12:30:06.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.244+0000 7f33337fe700 1 --2- 172.21.15.16:0/3012626906 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7f331406b140 0x7f331406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:30:06.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.248+0000 7f3330ff9700 1 -- 172.21.15.16:0/3012626906 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f33340c6210 con 0x7f33340cc8a0 2023-10-16T12:30:06.291 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:06.289+0000 7fc68f8d0200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:30:06.307 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:30:06.305+0000 7f71a592b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T12:30:06.376 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:06.374+0000 7fc68f8d0200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T12:30:06.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.445+0000 7f33337fe700 1 -- 172.21.15.16:0/3012626906 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7f331406b140 msgr2=0x7f331406d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2550537276 2023-10-16T12:30:06.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.445+0000 7f33337fe700 1 --2- 172.21.15.16:0/3012626906 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7f331406b140 0x7f331406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:30:06.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.453+0000 7f3339f1c700 1 -- 172.21.15.16:0/3012626906 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f333404ebb0 con 0x7f33340cc8a0 2023-10-16T12:30:06.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.455+0000 7f3330ff9700 1 -- 172.21.15.16:0/3012626906 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v129) v1 ==== 74+0+17150 (secure 0 0 0) 0x7f331c054370 con 0x7f33340cc8a0 2023-10-16T12:30:06.457 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:30:06.457 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":129,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:29:30.732485+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:29:14.232554+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2023-10-16T12:29:14.764132+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/32319707":"2023-10-17T12:29:30.732445+0000","172.21.15.16:0/1058620826":"2023-10-17T12:29:30.732445+0000","172.21.15.16:0/3098064509":"2023-10-17T12:29:30.732445+0000","172.21.15.16:0/3814896496":"2023-10-17T12:29:30.732445+0000","172.21.15.16:0/2973963882":"2023-10-17T12:29:30.732445+0000","172.21.15.16:6833/2309613638":"2023-10-17T12:29:30.732445+0000","172.21.15.16:6832/2309613638":"2023-10-17T12:29:30.732445+0000","172.21.15.16:0/1074148914":"2023-10-17T12:29:30.732445+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:30:06.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.457+0000 7f3339f1c700 1 -- 172.21.15.16:0/3012626906 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7f331406b140 msgr2=0x7f331406d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:30:06.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.457+0000 7f3339f1c700 1 --2- 172.21.15.16:0/3012626906 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7f331406b140 0x7f331406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:06.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.457+0000 7f3339f1c700 1 -- 172.21.15.16:0/3012626906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33340cc8a0 msgr2=0x7f33340c7f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:06.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.457+0000 7f3339f1c700 1 --2- 172.21.15.16:0/3012626906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33340cc8a0 0x7f33340c7f20 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f331c00c540 tx=0x7f331c00c850 comp rx=0 tx=0).stop 2023-10-16T12:30:06.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.457+0000 7f3339f1c700 1 -- 172.21.15.16:0/3012626906 shutdown_connections 2023-10-16T12:30:06.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.457+0000 7f3339f1c700 1 --2- 172.21.15.16:0/3012626906 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7f331406b140 0x7f331406d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:06.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.457+0000 7f3339f1c700 1 --2- 172.21.15.16:0/3012626906 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f333415b280 0x7f333415f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:06.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.457+0000 7f3339f1c700 1 --2- 172.21.15.16:0/3012626906 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f333406a7e0 0x7f3334148c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:06.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.457+0000 7f3339f1c700 1 --2- 172.21.15.16:0/3012626906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33340cc8a0 0x7f33340c7f20 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:06.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.457+0000 7f3339f1c700 1 -- 172.21.15.16:0/3012626906 >> 172.21.15.16:0/3012626906 conn(0x7f33340b1fd0 msgr2=0x7f333415d2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:06.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.457+0000 7f3339f1c700 1 -- 172.21.15.16:0/3012626906 shutdown_connections 2023-10-16T12:30:06.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.457+0000 7f3339f1c700 1 -- 172.21.15.16:0/3012626906 wait complete. 2023-10-16T12:30:06.462 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:30:06.457+0000 7f71a592b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:30:06.472 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T12:29:14.232554+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '127', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T12:29:14.764132+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '128', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T12:30:06.473 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:30:06.532 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:06.530+0000 7fc68f8d0200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:30:06.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.600+0000 7f134c95c700 1 Processor -- start 2023-10-16T12:30:06.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.600+0000 7f134c95c700 1 -- start start 2023-10-16T12:30:06.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.601+0000 7f134c95c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f134815b2c0 0x7f1348161340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:06.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.601+0000 7f134c95c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f134806e570 0x7f134815e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:06.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.601+0000 7f134c95c700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f134806a7e0 0x7f134806ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:06.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.601+0000 7f134c95c700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f134806b1b0 con 0x7f134815b2c0 2023-10-16T12:30:06.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.601+0000 7f134c95c700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1348161880 con 0x7f134806a7e0 2023-10-16T12:30:06.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.601+0000 7f134c95c700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f134815b6a0 con 0x7f134806e570 2023-10-16T12:30:06.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.602+0000 7f134659c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f134815b2c0 0x7f1348161340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:06.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.602+0000 7f134659c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f134815b2c0 0x7f1348161340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44592/0 (socket says 172.21.15.16:44592) 2023-10-16T12:30:06.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.602+0000 7f134659c700 1 -- 172.21.15.16:0/4230717042 learned_addr learned my addr 172.21.15.16:0/4230717042 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:06.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.602+0000 7f1345d9b700 1 --2- 172.21.15.16:0/4230717042 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f134806e570 0x7f134815e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:06.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.602+0000 7f1346d9d700 1 --2- 172.21.15.16:0/4230717042 >> v2:172.21.15.49:3300/0 conn(0x7f134806a7e0 0x7f134806ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:06.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.602+0000 7f134659c700 1 -- 172.21.15.16:0/4230717042 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f134806e570 msgr2=0x7f134815e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:06.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.602+0000 7f134659c700 1 --2- 172.21.15.16:0/4230717042 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f134806e570 0x7f134815e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:06.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.602+0000 7f134659c700 1 -- 172.21.15.16:0/4230717042 >> v2:172.21.15.49:3300/0 conn(0x7f134806a7e0 msgr2=0x7f134806ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:06.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.602+0000 7f134659c700 1 --2- 172.21.15.16:0/4230717042 >> v2:172.21.15.49:3300/0 conn(0x7f134806a7e0 0x7f134806ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:06.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.602+0000 7f134659c700 1 -- 172.21.15.16:0/4230717042 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1348161b20 con 0x7f134815b2c0 2023-10-16T12:30:06.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.602+0000 7f134659c700 1 --2- 172.21.15.16:0/4230717042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f134815b2c0 0x7f1348161340 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f13480ca970 tx=0x7f1338009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a6e1053962d280d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:06.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.602+0000 7f134559a700 1 -- 172.21.15.16:0/4230717042 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1338011830 con 0x7f134815b2c0 2023-10-16T12:30:06.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.602+0000 7f134559a700 1 -- 172.21.15.16:0/4230717042 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1338011990 con 0x7f134815b2c0 2023-10-16T12:30:06.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.602+0000 7f134559a700 1 -- 172.21.15.16:0/4230717042 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1338011d20 con 0x7f134815b2c0 2023-10-16T12:30:06.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.603+0000 7f134c95c700 1 -- 172.21.15.16:0/4230717042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f134815b2c0 msgr2=0x7f1348161340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:06.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.603+0000 7f134c95c700 1 --2- 172.21.15.16:0/4230717042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f134815b2c0 0x7f1348161340 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f13480ca970 tx=0x7f1338009bd0 comp rx=0 tx=0).stop 2023-10-16T12:30:06.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.603+0000 7f134c95c700 1 -- 172.21.15.16:0/4230717042 shutdown_connections 2023-10-16T12:30:06.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.603+0000 7f134c95c700 1 --2- 172.21.15.16:0/4230717042 >> v2:172.21.15.49:3300/0 conn(0x7f134806a7e0 0x7f134806ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:06.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.603+0000 7f134c95c700 1 --2- 172.21.15.16:0/4230717042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f134815b2c0 0x7f1348161340 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:06.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.603+0000 7f134c95c700 1 --2- 172.21.15.16:0/4230717042 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f134806e570 0x7f134815e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:06.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.603+0000 7f134c95c700 1 -- 172.21.15.16:0/4230717042 >> 172.21.15.16:0/4230717042 conn(0x7f13480b1f40 msgr2=0x7f13480b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:06.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.603+0000 7f134c95c700 1 -- 172.21.15.16:0/4230717042 shutdown_connections 2023-10-16T12:30:06.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.603+0000 7f134c95c700 1 -- 172.21.15.16:0/4230717042 wait complete. 2023-10-16T12:30:06.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.604+0000 7f134c95c700 1 Processor -- start 2023-10-16T12:30:06.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.605+0000 7f134c95c700 1 -- start start 2023-10-16T12:30:06.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.605+0000 7f134c95c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f134806a7e0 0x7f13480b5b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:06.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.605+0000 7f134c95c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f134806e570 0x7f13480c6d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:06.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.605+0000 7f134c95c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f134815b2c0 0x7f13480b3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:06.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.605+0000 7f134c95c700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13480c7280 con 0x7f134806e570 2023-10-16T12:30:06.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.605+0000 7f134c95c700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13480b6150 con 0x7f134806a7e0 2023-10-16T12:30:06.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.605+0000 7f134c95c700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13480c8510 con 0x7f134815b2c0 2023-10-16T12:30:06.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.605+0000 7f1345d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f134806e570 0x7f13480c6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:06.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.605+0000 7f1345d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f134806e570 0x7f13480c6d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44600/0 (socket says 172.21.15.16:44600) 2023-10-16T12:30:06.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.605+0000 7f134659c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f134806a7e0 0x7f13480b5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:06.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.605+0000 7f1345d9b700 1 -- 172.21.15.16:0/1174121154 learned_addr learned my addr 172.21.15.16:0/1174121154 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:06.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.605+0000 7f1346d9d700 1 --2- 172.21.15.16:0/1174121154 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f134815b2c0 0x7f13480b3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:06.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.606+0000 7f134659c700 1 -- 172.21.15.16:0/1174121154 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f134815b2c0 msgr2=0x7f13480b3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:06.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.606+0000 7f134659c700 1 --2- 172.21.15.16:0/1174121154 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f134815b2c0 0x7f13480b3660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:06.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.606+0000 7f134659c700 1 -- 172.21.15.16:0/1174121154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f134806e570 msgr2=0x7f13480c6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:06.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.606+0000 7f134659c700 1 --2- 172.21.15.16:0/1174121154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f134806e570 0x7f13480c6d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:06.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.606+0000 7f134659c700 1 -- 172.21.15.16:0/1174121154 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1338009550 con 0x7f134806a7e0 2023-10-16T12:30:06.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.606+0000 7f1346d9d700 1 --2- 172.21.15.16:0/1174121154 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f134815b2c0 0x7f13480b3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:30:06.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.606+0000 7f1345d9b700 1 --2- 172.21.15.16:0/1174121154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f134806e570 0x7f13480c6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:30:06.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.606+0000 7f134659c700 1 --2- 172.21.15.16:0/1174121154 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f134806a7e0 0x7f13480b5b20 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f1338016000 tx=0x7f1338004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:06.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.606+0000 7f1336ffd700 1 -- 172.21.15.16:0/1174121154 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1338004240 con 0x7f134806a7e0 2023-10-16T12:30:06.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.606+0000 7f1336ffd700 1 -- 172.21.15.16:0/1174121154 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f133801f070 con 0x7f134806a7e0 2023-10-16T12:30:06.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.606+0000 7f134c95c700 1 -- 172.21.15.16:0/1174121154 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1348152e60 con 0x7f134806a7e0 2023-10-16T12:30:06.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.606+0000 7f1336ffd700 1 -- 172.21.15.16:0/1174121154 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13380045e0 con 0x7f134806a7e0 2023-10-16T12:30:06.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.606+0000 7f134c95c700 1 -- 172.21.15.16:0/1174121154 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1348153420 con 0x7f134806a7e0 2023-10-16T12:30:06.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.607+0000 7f134c95c700 1 -- 172.21.15.16:0/1174121154 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13480c8730 con 0x7f134806a7e0 2023-10-16T12:30:06.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.608+0000 7f1336ffd700 1 -- 172.21.15.16:0/1174121154 <== mon.1 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 70) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f1338021030 con 0x7f134806a7e0 2023-10-16T12:30:06.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.608+0000 7f1336ffd700 1 --2- 172.21.15.16:0/1174121154 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7f132806b140 0x7f132806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:06.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.608+0000 7f1345d9b700 1 -- 172.21.15.16:0/1174121154 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7f132806b140 msgr2=0x7f132806d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2550537276 2023-10-16T12:30:06.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.608+0000 7f1336ffd700 1 -- 172.21.15.16:0/1174121154 <== mon.1 v2:172.21.15.49:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f133808ac80 con 0x7f134806a7e0 2023-10-16T12:30:06.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.608+0000 7f1345d9b700 1 --2- 172.21.15.16:0/1174121154 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7f132806b140 0x7f132806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:30:06.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.613+0000 7f1336ffd700 1 -- 172.21.15.16:0/1174121154 <== mon.1 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f13480c8730 con 0x7f134806a7e0 2023-10-16T12:30:06.697 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:30:06.695+0000 7f71a592b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T12:30:06.774 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:30:06.772+0000 7f71a592b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T12:30:06.778 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:06.776+0000 7fc68f8d0200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T12:30:06.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.809+0000 7f1345d9b700 1 -- 172.21.15.16:0/1174121154 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7f132806b140 msgr2=0x7f132806d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2550537276 2023-10-16T12:30:06.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.809+0000 7f1345d9b700 1 --2- 172.21.15.16:0/1174121154 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7f132806b140 0x7f132806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:30:06.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.815+0000 7f134c95c700 1 -- 172.21.15.16:0/1174121154 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f13480c8730 con 0x7f134806a7e0 2023-10-16T12:30:06.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.815+0000 7f1336ffd700 1 -- 172.21.15.16:0/1174121154 <== mon.1 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v129) v1 ==== 93+0+10 (secure 0 0 0) 0x7f13480c8730 con 0x7f134806a7e0 2023-10-16T12:30:06.818 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T12:30:06.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.818+0000 7f134c95c700 1 -- 172.21.15.16:0/1174121154 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7f132806b140 msgr2=0x7f132806d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:30:06.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.818+0000 7f134c95c700 1 --2- 172.21.15.16:0/1174121154 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7f132806b140 0x7f132806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:06.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.818+0000 7f134c95c700 1 -- 172.21.15.16:0/1174121154 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f134806a7e0 msgr2=0x7f13480b5b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:06.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.818+0000 7f134c95c700 1 --2- 172.21.15.16:0/1174121154 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f134806a7e0 0x7f13480b5b20 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f1338016000 tx=0x7f1338004750 comp rx=0 tx=0).stop 2023-10-16T12:30:06.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.818+0000 7f134c95c700 1 -- 172.21.15.16:0/1174121154 shutdown_connections 2023-10-16T12:30:06.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.818+0000 7f134c95c700 1 --2- 172.21.15.16:0/1174121154 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7f132806b140 0x7f132806d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:06.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.818+0000 7f134c95c700 1 --2- 172.21.15.16:0/1174121154 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f134815b2c0 0x7f13480b3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:06.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.818+0000 7f134c95c700 1 --2- 172.21.15.16:0/1174121154 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f134806a7e0 0x7f13480b5b20 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:06.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.818+0000 7f134c95c700 1 --2- 172.21.15.16:0/1174121154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f134806e570 0x7f13480c6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:06.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.818+0000 7f134c95c700 1 -- 172.21.15.16:0/1174121154 >> 172.21.15.16:0/1174121154 conn(0x7f13480b1f40 msgr2=0x7f1348161120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:06.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.818+0000 7f134c95c700 1 -- 172.21.15.16:0/1174121154 shutdown_connections 2023-10-16T12:30:06.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.818+0000 7f134c95c700 1 -- 172.21.15.16:0/1174121154 wait complete. 2023-10-16T12:30:06.832 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T12:30:06.858 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:06.857+0000 7fc68f8d0200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T12:30:06.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.963+0000 7fab0bd72700 1 Processor -- start 2023-10-16T12:30:06.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.963+0000 7fab0bd72700 1 -- start start 2023-10-16T12:30:06.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.964+0000 7fab0bd72700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fab040cb880 0x7fab040cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:06.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.964+0000 7fab0bd72700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fab0406da60 0x7fab0406def0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:06.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.964+0000 7fab0bd72700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab040cef70 con 0x7fab0415b070 2023-10-16T12:30:06.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.964+0000 7fab0bd72700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab0406e430 con 0x7fab0406da60 2023-10-16T12:30:06.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.964+0000 7fab0bd72700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab040ce360 con 0x7fab040cb880 2023-10-16T12:30:06.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.964+0000 7fab0a30f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fab0406da60 0x7fab0406def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:06.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.964+0000 7fab0a30f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fab0406da60 0x7fab0406def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44612/0 (socket says 172.21.15.16:44612) 2023-10-16T12:30:06.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.964+0000 7fab0a30f700 1 -- 172.21.15.16:0/2961446287 learned_addr learned my addr 172.21.15.16:0/2961446287 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:06.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.964+0000 7fab0930d700 1 --2- 172.21.15.16:0/2961446287 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fab040cb880 0x7fab040cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:06.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.964+0000 7fab0a30f700 1 -- 172.21.15.16:0/2961446287 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fab040cb880 msgr2=0x7fab040cbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:06.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.964+0000 7fab0a30f700 1 --2- 172.21.15.16:0/2961446287 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fab040cb880 0x7fab040cbd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:06.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.964+0000 7fab0a30f700 1 -- 172.21.15.16:0/2961446287 >> v1:172.21.15.49:6789/0 conn(0x7fab0415b070 legacy=0x7fab0415b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:06.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.964+0000 7fab0a30f700 1 -- 172.21.15.16:0/2961446287 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab040ce640 con 0x7fab0406da60 2023-10-16T12:30:06.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.964+0000 7fab08b0c700 1 -- 172.21.15.16:0/2961446287 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 189351120 0 0) 0x7fab040cef70 con 0x7fab0415b070 2023-10-16T12:30:06.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.964+0000 7fab0a30f700 1 --2- 172.21.15.16:0/2961446287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab0406da60 0x7fab0406def0 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7faafc009e50 tx=0x7faafc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5eb162da24ddca64 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:06.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.965+0000 7fab08b0c700 1 -- 172.21.15.16:0/2961446287 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faafc00e070 con 0x7fab0406da60 2023-10-16T12:30:06.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.965+0000 7fab08b0c700 1 -- 172.21.15.16:0/2961446287 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faafc00b910 con 0x7fab0406da60 2023-10-16T12:30:06.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.965+0000 7fab08b0c700 1 -- 172.21.15.16:0/2961446287 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faafc00bcc0 con 0x7fab0406da60 2023-10-16T12:30:06.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.965+0000 7fab0bd72700 1 -- 172.21.15.16:0/2961446287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab0406da60 msgr2=0x7fab0406def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:06.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.965+0000 7fab0bd72700 1 --2- 172.21.15.16:0/2961446287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab0406da60 0x7fab0406def0 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7faafc009e50 tx=0x7faafc00b330 comp rx=0 tx=0).stop 2023-10-16T12:30:06.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.965+0000 7fab0bd72700 1 -- 172.21.15.16:0/2961446287 shutdown_connections 2023-10-16T12:30:06.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.965+0000 7fab0bd72700 1 --2- 172.21.15.16:0/2961446287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab0406da60 0x7fab0406def0 unknown :-1 s=CLOSED pgs=1452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:06.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.965+0000 7fab0bd72700 1 --2- 172.21.15.16:0/2961446287 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fab040cb880 0x7fab040cbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:06.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.965+0000 7fab0bd72700 1 -- 172.21.15.16:0/2961446287 >> 172.21.15.16:0/2961446287 conn(0x7fab040b1f40 msgr2=0x7fab040b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:06.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.966+0000 7fab0bd72700 1 -- 172.21.15.16:0/2961446287 shutdown_connections 2023-10-16T12:30:06.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.966+0000 7fab0bd72700 1 -- 172.21.15.16:0/2961446287 wait complete. 2023-10-16T12:30:06.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.967+0000 7fab0bd72700 1 Processor -- start 2023-10-16T12:30:06.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.967+0000 7fab0bd72700 1 -- start start 2023-10-16T12:30:06.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.967+0000 7fab0bd72700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fab0406da60 0x7fab041526f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:06.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.968+0000 7fab0bd72700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab040cb880 0x7fab040aeca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:06.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.968+0000 7fab0bd72700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fab0415b070 0x7fab0406a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:06.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.968+0000 7fab0bd72700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab04152c30 con 0x7fab040cb880 2023-10-16T12:30:06.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.968+0000 7fab0bd72700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab040ac8d0 con 0x7fab0406da60 2023-10-16T12:30:06.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.968+0000 7fab0bd72700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab040acb70 con 0x7fab0415b070 2023-10-16T12:30:06.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.968+0000 7fab0930d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab040cb880 0x7fab040aeca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:06.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.968+0000 7fab0930d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab040cb880 0x7fab040aeca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44616/0 (socket says 172.21.15.16:44616) 2023-10-16T12:30:06.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.968+0000 7fab0930d700 1 -- 172.21.15.16:0/3355835300 learned_addr learned my addr 172.21.15.16:0/3355835300 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:06.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.968+0000 7fab0a30f700 1 --2- 172.21.15.16:0/3355835300 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fab0415b070 0x7fab0406a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:06.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.968+0000 7fab09b0e700 1 --2- 172.21.15.16:0/3355835300 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fab0406da60 0x7fab041526f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:06.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.968+0000 7fab0930d700 1 -- 172.21.15.16:0/3355835300 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fab0415b070 msgr2=0x7fab0406a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:06.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.968+0000 7fab0930d700 1 --2- 172.21.15.16:0/3355835300 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fab0415b070 0x7fab0406a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:06.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.968+0000 7fab0930d700 1 -- 172.21.15.16:0/3355835300 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fab0406da60 msgr2=0x7fab041526f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:06.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.968+0000 7fab0930d700 1 --2- 172.21.15.16:0/3355835300 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fab0406da60 0x7fab041526f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:06.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.968+0000 7fab0930d700 1 -- 172.21.15.16:0/3355835300 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faafc009b30 con 0x7fab040cb880 2023-10-16T12:30:06.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.968+0000 7fab0930d700 1 --2- 172.21.15.16:0/3355835300 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab040cb880 0x7fab040aeca0 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7faaf400e9b0 tx=0x7faaf400ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:06.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.968+0000 7faafa7fc700 1 -- 172.21.15.16:0/3355835300 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaf4009710 con 0x7fab040cb880 2023-10-16T12:30:06.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.968+0000 7faafa7fc700 1 -- 172.21.15.16:0/3355835300 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaf400ee50 con 0x7fab040cb880 2023-10-16T12:30:06.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.968+0000 7faafa7fc700 1 -- 172.21.15.16:0/3355835300 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaf400ca50 con 0x7fab040cb880 2023-10-16T12:30:06.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.968+0000 7fab0bd72700 1 -- 172.21.15.16:0/3355835300 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab040ace50 con 0x7fab040cb880 2023-10-16T12:30:06.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.969+0000 7fab0bd72700 1 -- 172.21.15.16:0/3355835300 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab040ad410 con 0x7fab040cb880 2023-10-16T12:30:06.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.970+0000 7fab0bd72700 1 -- 172.21.15.16:0/3355835300 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab040ad780 con 0x7fab040cb880 2023-10-16T12:30:06.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.970+0000 7faafa7fc700 1 -- 172.21.15.16:0/3355835300 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 70) v1 ==== 89445+0+0 (secure 0 0 0) 0x7faaf400cbb0 con 0x7fab040cb880 2023-10-16T12:30:06.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.970+0000 7faafa7fc700 1 --2- 172.21.15.16:0/3355835300 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7faae806b140 0x7faae806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:06.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.971+0000 7fab09b0e700 1 -- 172.21.15.16:0/3355835300 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7faae806b140 msgr2=0x7faae806d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2550537276 2023-10-16T12:30:06.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.971+0000 7faafa7fc700 1 -- 172.21.15.16:0/3355835300 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6401+0+0 (secure 0 0 0) 0x7faaf4089cb0 con 0x7fab040cb880 2023-10-16T12:30:06.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.971+0000 7fab09b0e700 1 --2- 172.21.15.16:0/3355835300 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7faae806b140 0x7faae806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:30:06.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:06.975+0000 7faafa7fc700 1 -- 172.21.15.16:0/3355835300 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7faaf4054740 con 0x7fab040cb880 2023-10-16T12:30:07.050 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:30:07.048+0000 7f71a592b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T12:30:07.141 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:07.140+0000 7fc68f8d0200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T12:30:07.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.172+0000 7fab09b0e700 1 -- 172.21.15.16:0/3355835300 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7faae806b140 msgr2=0x7faae806d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2550537276 2023-10-16T12:30:07.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.172+0000 7fab09b0e700 1 --2- 172.21.15.16:0/3355835300 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7faae806b140 0x7faae806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:30:07.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.181+0000 7fab0bd72700 1 -- 172.21.15.16:0/3355835300 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fab0404ebb0 con 0x7fab040cb880 2023-10-16T12:30:07.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.181+0000 7faafa7fc700 1 -- 172.21.15.16:0/3355835300 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v129) v1 ==== 104+0+11 (secure 0 0 0) 0x7faaf4021020 con 0x7fab040cb880 2023-10-16T12:30:07.184 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-10-16T12:30:07.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.183+0000 7fab0bd72700 1 -- 172.21.15.16:0/3355835300 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7faae806b140 msgr2=0x7faae806d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:30:07.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.183+0000 7fab0bd72700 1 --2- 172.21.15.16:0/3355835300 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7faae806b140 0x7faae806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:07.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.183+0000 7fab0bd72700 1 -- 172.21.15.16:0/3355835300 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab040cb880 msgr2=0x7fab040aeca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:07.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.183+0000 7fab0bd72700 1 --2- 172.21.15.16:0/3355835300 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab040cb880 0x7fab040aeca0 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7faaf400e9b0 tx=0x7faaf400ecc0 comp rx=0 tx=0).stop 2023-10-16T12:30:07.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.183+0000 7fab0bd72700 1 -- 172.21.15.16:0/3355835300 shutdown_connections 2023-10-16T12:30:07.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.183+0000 7fab0bd72700 1 --2- 172.21.15.16:0/3355835300 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7faae806b140 0x7faae806d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:07.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.183+0000 7fab0bd72700 1 --2- 172.21.15.16:0/3355835300 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fab0415b070 0x7fab0406a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:07.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.183+0000 7fab0bd72700 1 --2- 172.21.15.16:0/3355835300 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fab0406da60 0x7fab041526f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:07.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.183+0000 7fab0bd72700 1 --2- 172.21.15.16:0/3355835300 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab040cb880 0x7fab040aeca0 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:07.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.183+0000 7fab0bd72700 1 -- 172.21.15.16:0/3355835300 >> 172.21.15.16:0/3355835300 conn(0x7fab040b1f40 msgr2=0x7fab04153a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:07.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.184+0000 7fab0bd72700 1 -- 172.21.15.16:0/3355835300 shutdown_connections 2023-10-16T12:30:07.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.184+0000 7fab0bd72700 1 -- 172.21.15.16:0/3355835300 wait complete. 2023-10-16T12:30:07.198 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T12:30:07.282 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:30:07.280+0000 7f71a592b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T12:30:07.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.366+0000 7ff9074df700 1 Processor -- start 2023-10-16T12:30:07.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.367+0000 7ff9074df700 1 -- start start 2023-10-16T12:30:07.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.368+0000 7ff9074df700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7ff900160ee0 0x7ff9000aeb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:07.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.368+0000 7ff9074df700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9000af4a0 con 0x7ff9000ce890 2023-10-16T12:30:07.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.368+0000 7ff9074df700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff90006aa10 con 0x7ff9000af040 2023-10-16T12:30:07.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.368+0000 7ff9074df700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9000cf770 con 0x7ff900160ee0 2023-10-16T12:30:07.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.368+0000 7ff905a7c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff9000ce890 0x7ff90006aea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33900/0 (socket says 172.21.15.16:33900) 2023-10-16T12:30:07.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.368+0000 7ff905a7c700 1 -- 172.21.15.16:0/988946373 learned_addr learned my addr 172.21.15.16:0/988946373 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:30:07.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.368+0000 7ff90527b700 1 --2- 172.21.15.16:0/988946373 >> v2:172.21.15.49:3300/0 conn(0x7ff900160ee0 0x7ff9000aeb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:07.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.368+0000 7ff8f7fff700 1 -- 172.21.15.16:0/988946373 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 294643301 0 0) 0x7ff9000af4a0 con 0x7ff9000ce890 2023-10-16T12:30:07.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.368+0000 7ff8f7fff700 1 -- 172.21.15.16:0/988946373 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8ec0035b0 con 0x7ff9000ce890 2023-10-16T12:30:07.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.369+0000 7ff90527b700 1 -- 172.21.15.16:0/988946373 >> v1:172.21.15.49:6789/0 conn(0x7ff9000af040 legacy=0x7ff9000ce180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:07.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.369+0000 7ff90527b700 1 -- 172.21.15.16:0/988946373 >> v1:172.21.15.16:6789/0 conn(0x7ff9000ce890 legacy=0x7ff90006aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:07.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.369+0000 7ff8f7fff700 1 -- 172.21.15.16:0/988946373 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2738267032 0 0) 0x7ff90006aa10 con 0x7ff9000af040 2023-10-16T12:30:07.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.369+0000 7ff90527b700 1 -- 172.21.15.16:0/988946373 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9000ceed0 con 0x7ff900160ee0 2023-10-16T12:30:07.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.369+0000 7ff90527b700 1 --2- 172.21.15.16:0/988946373 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff900160ee0 0x7ff9000aeb00 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7ff8fc009840 tx=0x7ff8fc009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=969c0e6ee4815f1e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:07.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.369+0000 7ff8f7fff700 1 -- 172.21.15.16:0/988946373 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8fc011830 con 0x7ff900160ee0 2023-10-16T12:30:07.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.369+0000 7ff8f7fff700 1 -- 172.21.15.16:0/988946373 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8fc011990 con 0x7ff900160ee0 2023-10-16T12:30:07.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.369+0000 7ff8f7fff700 1 -- 172.21.15.16:0/988946373 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8fc011d20 con 0x7ff900160ee0 2023-10-16T12:30:07.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.370+0000 7ff9074df700 1 -- 172.21.15.16:0/988946373 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff900160ee0 msgr2=0x7ff9000aeb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:07.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.370+0000 7ff9074df700 1 --2- 172.21.15.16:0/988946373 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff900160ee0 0x7ff9000aeb00 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7ff8fc009840 tx=0x7ff8fc009b50 comp rx=0 tx=0).stop 2023-10-16T12:30:07.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.370+0000 7ff9074df700 1 -- 172.21.15.16:0/988946373 shutdown_connections 2023-10-16T12:30:07.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.370+0000 7ff9074df700 1 --2- 172.21.15.16:0/988946373 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff900160ee0 0x7ff9000aeb00 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:07.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.370+0000 7ff9074df700 1 -- 172.21.15.16:0/988946373 >> 172.21.15.16:0/988946373 conn(0x7ff9000cc5e0 msgr2=0x7ff9000cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:07.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.370+0000 7ff9074df700 1 -- 172.21.15.16:0/988946373 shutdown_connections 2023-10-16T12:30:07.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.370+0000 7ff9074df700 1 -- 172.21.15.16:0/988946373 wait complete. 2023-10-16T12:30:07.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.371+0000 7ff9074df700 1 Processor -- start 2023-10-16T12:30:07.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.371+0000 7ff9074df700 1 -- start start 2023-10-16T12:30:07.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.372+0000 7ff9074df700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff9000af040 0x7ff90015d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:07.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.372+0000 7ff9074df700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9000ce890 0x7ff90015dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:07.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.372+0000 7ff9074df700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff900160ee0 0x7ff900155150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:07.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.372+0000 7ff9074df700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff90015f320 con 0x7ff9000ce890 2023-10-16T12:30:07.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.372+0000 7ff9074df700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff90015f740 con 0x7ff9000af040 2023-10-16T12:30:07.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.372+0000 7ff9074df700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff90015e400 con 0x7ff900160ee0 2023-10-16T12:30:07.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.372+0000 7ff904a7a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9000ce890 0x7ff90015dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:07.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.372+0000 7ff904a7a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9000ce890 0x7ff90015dec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44630/0 (socket says 172.21.15.16:44630) 2023-10-16T12:30:07.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.372+0000 7ff904a7a700 1 -- 172.21.15.16:0/3205368378 learned_addr learned my addr 172.21.15.16:0/3205368378 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:07.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.372+0000 7ff90527b700 1 --2- 172.21.15.16:0/3205368378 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff9000af040 0x7ff90015d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:07.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.373+0000 7ff905a7c700 1 --2- 172.21.15.16:0/3205368378 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff900160ee0 0x7ff900155150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:07.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.373+0000 7ff904a7a700 1 -- 172.21.15.16:0/3205368378 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff900160ee0 msgr2=0x7ff900155150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:07.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.373+0000 7ff904a7a700 1 --2- 172.21.15.16:0/3205368378 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff900160ee0 0x7ff900155150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:07.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.373+0000 7ff904a7a700 1 -- 172.21.15.16:0/3205368378 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff9000af040 msgr2=0x7ff90015d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:07.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.373+0000 7ff904a7a700 1 --2- 172.21.15.16:0/3205368378 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff9000af040 0x7ff90015d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:07.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.373+0000 7ff904a7a700 1 -- 172.21.15.16:0/3205368378 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8fc0094a0 con 0x7ff9000ce890 2023-10-16T12:30:07.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.373+0000 7ff904a7a700 1 --2- 172.21.15.16:0/3205368378 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9000ce890 0x7ff90015dec0 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7ff8f00011e0 tx=0x7ff8f000ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:07.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.373+0000 7ff8f5ffb700 1 -- 172.21.15.16:0/3205368378 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8f00054e0 con 0x7ff9000ce890 2023-10-16T12:30:07.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.373+0000 7ff8f5ffb700 1 -- 172.21.15.16:0/3205368378 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8f0003680 con 0x7ff9000ce890 2023-10-16T12:30:07.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.373+0000 7ff9074df700 1 -- 172.21.15.16:0/3205368378 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff90015e6a0 con 0x7ff9000ce890 2023-10-16T12:30:07.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.373+0000 7ff8f5ffb700 1 -- 172.21.15.16:0/3205368378 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8f0003a30 con 0x7ff9000ce890 2023-10-16T12:30:07.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.373+0000 7ff9074df700 1 -- 172.21.15.16:0/3205368378 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff90015ec60 con 0x7ff9000ce890 2023-10-16T12:30:07.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.375+0000 7ff9074df700 1 -- 172.21.15.16:0/3205368378 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff90015f460 con 0x7ff9000ce890 2023-10-16T12:30:07.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.375+0000 7ff8f5ffb700 1 -- 172.21.15.16:0/3205368378 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 70) v1 ==== 89445+0+0 (secure 0 0 0) 0x7ff8f0003b90 con 0x7ff9000ce890 2023-10-16T12:30:07.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.375+0000 7ff8f5ffb700 1 --2- 172.21.15.16:0/3205368378 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7ff8e406b140 0x7ff8e406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:07.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.375+0000 7ff90527b700 1 -- 172.21.15.16:0/3205368378 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7ff8e406b140 msgr2=0x7ff8e406d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2550537276 2023-10-16T12:30:07.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.375+0000 7ff8f5ffb700 1 -- 172.21.15.16:0/3205368378 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6401+0+0 (secure 0 0 0) 0x7ff8f0089090 con 0x7ff9000ce890 2023-10-16T12:30:07.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.375+0000 7ff90527b700 1 --2- 172.21.15.16:0/3205368378 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7ff8e406b140 0x7ff8e406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:30:07.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.380+0000 7ff8f5ffb700 1 -- 172.21.15.16:0/3205368378 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff90015f460 con 0x7ff9000ce890 2023-10-16T12:30:07.395 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:07.385+0000 7fc68f8d0200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T12:30:07.440 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:30:07.438+0000 7f71a592b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T12:30:07.548 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:07.546+0000 7fc68f8d0200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T12:30:07.560 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:30:07.558+0000 7f71a592b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T12:30:07.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.576+0000 7ff90527b700 1 -- 172.21.15.16:0/3205368378 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7ff8e406b140 msgr2=0x7ff8e406d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2550537276 2023-10-16T12:30:07.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.576+0000 7ff90527b700 1 --2- 172.21.15.16:0/3205368378 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7ff8e406b140 0x7ff8e406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:30:07.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.585+0000 7ff9074df700 1 -- 172.21.15.16:0/3205368378 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff90015f460 con 0x7ff9000ce890 2023-10-16T12:30:07.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.585+0000 7ff8f5ffb700 1 -- 172.21.15.16:0/3205368378 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v129) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff90015f460 con 0x7ff9000ce890 2023-10-16T12:30:07.587 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-10-16T12:30:07.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.587+0000 7ff9074df700 1 -- 172.21.15.16:0/3205368378 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7ff8e406b140 msgr2=0x7ff8e406d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:30:07.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.587+0000 7ff9074df700 1 --2- 172.21.15.16:0/3205368378 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7ff8e406b140 0x7ff8e406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:07.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.587+0000 7ff9074df700 1 -- 172.21.15.16:0/3205368378 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9000ce890 msgr2=0x7ff90015dec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:07.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.587+0000 7ff9074df700 1 --2- 172.21.15.16:0/3205368378 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9000ce890 0x7ff90015dec0 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7ff8f00011e0 tx=0x7ff8f000ca30 comp rx=0 tx=0).stop 2023-10-16T12:30:07.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.587+0000 7ff9074df700 1 -- 172.21.15.16:0/3205368378 shutdown_connections 2023-10-16T12:30:07.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.587+0000 7ff9074df700 1 --2- 172.21.15.16:0/3205368378 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7ff8e406b140 0x7ff8e406d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:07.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.587+0000 7ff9074df700 1 --2- 172.21.15.16:0/3205368378 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff900160ee0 0x7ff900155150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:07.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.587+0000 7ff9074df700 1 --2- 172.21.15.16:0/3205368378 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff9000af040 0x7ff90015d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:07.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.587+0000 7ff9074df700 1 --2- 172.21.15.16:0/3205368378 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9000ce890 0x7ff90015dec0 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:07.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.587+0000 7ff9074df700 1 -- 172.21.15.16:0/3205368378 >> 172.21.15.16:0/3205368378 conn(0x7ff9000cc5e0 msgr2=0x7ff90006ec30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:07.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.587+0000 7ff9074df700 1 -- 172.21.15.16:0/3205368378 shutdown_connections 2023-10-16T12:30:07.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.587+0000 7ff9074df700 1 -- 172.21.15.16:0/3205368378 wait complete. 2023-10-16T12:30:07.602 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:30:07.673 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:07.671+0000 7fc68f8d0200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T12:30:07.723 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:30:07.720+0000 7f71a592b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T12:30:07.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.726+0000 7f420c0df700 1 Processor -- start 2023-10-16T12:30:07.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.726+0000 7f420c0df700 1 -- start start 2023-10-16T12:30:07.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.727+0000 7f420c0df700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f420415a5c0 0x7f42040cef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:07.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.727+0000 7f420c0df700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f42040ce010 0x7f42040d03d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:07.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.727+0000 7f420c0df700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f42040d0910 0x7f420415e210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:07.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.727+0000 7f420c0df700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42040d0dc0 con 0x7f420415a5c0 2023-10-16T12:30:07.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.727+0000 7f420c0df700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42040cf440 con 0x7f42040ce010 2023-10-16T12:30:07.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.727+0000 7f420c0df700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4204072a50 con 0x7f42040d0910 2023-10-16T12:30:07.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.727+0000 7f4209e7b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f420415a5c0 0x7f42040cef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:07.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.727+0000 7f4209e7b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f420415a5c0 0x7f42040cef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44638/0 (socket says 172.21.15.16:44638) 2023-10-16T12:30:07.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.727+0000 7f4209e7b700 1 -- 172.21.15.16:0/1859506702 learned_addr learned my addr 172.21.15.16:0/1859506702 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:07.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.727+0000 7f420a67c700 1 --2- 172.21.15.16:0/1859506702 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f42040d0910 0x7f420415e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:07.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.727+0000 7f420967a700 1 --2- 172.21.15.16:0/1859506702 >> v2:172.21.15.49:3300/0 conn(0x7f42040ce010 0x7f42040d03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:07.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.728+0000 7f4209e7b700 1 -- 172.21.15.16:0/1859506702 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f42040d0910 msgr2=0x7f420415e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:07.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.728+0000 7f4209e7b700 1 --2- 172.21.15.16:0/1859506702 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f42040d0910 0x7f420415e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:07.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.728+0000 7f4209e7b700 1 -- 172.21.15.16:0/1859506702 >> v2:172.21.15.49:3300/0 conn(0x7f42040ce010 msgr2=0x7f42040d03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:07.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.728+0000 7f4209e7b700 1 --2- 172.21.15.16:0/1859506702 >> v2:172.21.15.49:3300/0 conn(0x7f42040ce010 0x7f42040d03d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:07.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.728+0000 7f4209e7b700 1 -- 172.21.15.16:0/1859506702 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42040cba30 con 0x7f420415a5c0 2023-10-16T12:30:07.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.728+0000 7f4209e7b700 1 --2- 172.21.15.16:0/1859506702 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f420415a5c0 0x7f42040cef00 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f4204067660 tx=0x7f420000be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac3c08c581da5773 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:07.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.728+0000 7f4208e79700 1 -- 172.21.15.16:0/1859506702 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f420000dc70 con 0x7f420415a5c0 2023-10-16T12:30:07.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.728+0000 7f4208e79700 1 -- 172.21.15.16:0/1859506702 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f420000ddd0 con 0x7f420415a5c0 2023-10-16T12:30:07.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.728+0000 7f4208e79700 1 -- 172.21.15.16:0/1859506702 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4200010690 con 0x7f420415a5c0 2023-10-16T12:30:07.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.729+0000 7f420c0df700 1 -- 172.21.15.16:0/1859506702 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f420415a5c0 msgr2=0x7f42040cef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:07.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.729+0000 7f420c0df700 1 --2- 172.21.15.16:0/1859506702 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f420415a5c0 0x7f42040cef00 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f4204067660 tx=0x7f420000be70 comp rx=0 tx=0).stop 2023-10-16T12:30:07.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.729+0000 7f420c0df700 1 -- 172.21.15.16:0/1859506702 shutdown_connections 2023-10-16T12:30:07.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.729+0000 7f420c0df700 1 --2- 172.21.15.16:0/1859506702 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f42040d0910 0x7f420415e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:07.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.729+0000 7f420c0df700 1 --2- 172.21.15.16:0/1859506702 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f420415a5c0 0x7f42040cef00 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:07.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.729+0000 7f420c0df700 1 --2- 172.21.15.16:0/1859506702 >> v2:172.21.15.49:3300/0 conn(0x7f42040ce010 0x7f42040d03d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:07.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.729+0000 7f420c0df700 1 -- 172.21.15.16:0/1859506702 >> 172.21.15.16:0/1859506702 conn(0x7f42040b1fd0 msgr2=0x7f42040b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:07.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.729+0000 7f420c0df700 1 -- 172.21.15.16:0/1859506702 shutdown_connections 2023-10-16T12:30:07.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.729+0000 7f420c0df700 1 -- 172.21.15.16:0/1859506702 wait complete. 2023-10-16T12:30:07.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.730+0000 7f420c0df700 1 Processor -- start 2023-10-16T12:30:07.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.730+0000 7f420c0df700 1 -- start start 2023-10-16T12:30:07.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.731+0000 7f420c0df700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f42040ce010 0x7f4204154640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:07.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.731+0000 7f420c0df700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f42040d0910 0x7f42040c9260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:07.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.731+0000 7f420c0df700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f420415a5c0 0x7f42040b30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:07.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.731+0000 7f420c0df700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42040c84a0 con 0x7f420415a5c0 2023-10-16T12:30:07.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.731+0000 7f420c0df700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42040c8780 con 0x7f42040ce010 2023-10-16T12:30:07.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.731+0000 7f420c0df700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42040c8a60 con 0x7f42040d0910 2023-10-16T12:30:07.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.731+0000 7f420a67c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f420415a5c0 0x7f42040b30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:07.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.731+0000 7f4209e7b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f42040ce010 0x7f4204154640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:07.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.731+0000 7f420a67c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f420415a5c0 0x7f42040b30c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44648/0 (socket says 172.21.15.16:44648) 2023-10-16T12:30:07.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.731+0000 7f420967a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f42040d0910 0x7f42040c9260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:07.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.731+0000 7f420a67c700 1 -- 172.21.15.16:0/905631815 learned_addr learned my addr 172.21.15.16:0/905631815 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:07.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.731+0000 7f420a67c700 1 -- 172.21.15.16:0/905631815 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f42040d0910 msgr2=0x7f42040c9260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:07.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.731+0000 7f420a67c700 1 --2- 172.21.15.16:0/905631815 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f42040d0910 0x7f42040c9260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:07.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.731+0000 7f420a67c700 1 -- 172.21.15.16:0/905631815 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f42040ce010 msgr2=0x7f4204154640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:07.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.731+0000 7f420a67c700 1 --2- 172.21.15.16:0/905631815 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f42040ce010 0x7f4204154640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:07.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.731+0000 7f420a67c700 1 -- 172.21.15.16:0/905631815 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f420000b7f0 con 0x7f420415a5c0 2023-10-16T12:30:07.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.732+0000 7f420a67c700 1 --2- 172.21.15.16:0/905631815 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f420415a5c0 0x7f42040b30c0 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f41fc00efc0 tx=0x7f41fc00c0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:07.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.732+0000 7f41fa7fc700 1 -- 172.21.15.16:0/905631815 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41fc00caa0 con 0x7f420415a5c0 2023-10-16T12:30:07.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.732+0000 7f41fa7fc700 1 -- 172.21.15.16:0/905631815 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41fc00f040 con 0x7f420415a5c0 2023-10-16T12:30:07.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.732+0000 7f420c0df700 1 -- 172.21.15.16:0/905631815 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42040c8d40 con 0x7f420415a5c0 2023-10-16T12:30:07.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.732+0000 7f41fa7fc700 1 -- 172.21.15.16:0/905631815 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41fc00ce40 con 0x7f420415a5c0 2023-10-16T12:30:07.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.732+0000 7f420c0df700 1 -- 172.21.15.16:0/905631815 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42041f36d0 con 0x7f420415a5c0 2023-10-16T12:30:07.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.733+0000 7f420c0df700 1 -- 172.21.15.16:0/905631815 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42040c85e0 con 0x7f420415a5c0 2023-10-16T12:30:07.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.734+0000 7f41fa7fc700 1 -- 172.21.15.16:0/905631815 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 70) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f41fc014480 con 0x7f420415a5c0 2023-10-16T12:30:07.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.734+0000 7f41fa7fc700 1 --2- 172.21.15.16:0/905631815 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7f41e806b140 0x7f41e806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:07.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.734+0000 7f4209e7b700 1 -- 172.21.15.16:0/905631815 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7f41e806b140 msgr2=0x7f41e806d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2550537276 2023-10-16T12:30:07.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.734+0000 7f41fa7fc700 1 -- 172.21.15.16:0/905631815 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f41fc08a480 con 0x7f420415a5c0 2023-10-16T12:30:07.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.734+0000 7f4209e7b700 1 --2- 172.21.15.16:0/905631815 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7f41e806b140 0x7f41e806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:30:07.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.738+0000 7f41fa7fc700 1 -- 172.21.15.16:0/905631815 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f42040c85e0 con 0x7f420415a5c0 2023-10-16T12:30:07.841 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:07.840+0000 7fc68f8d0200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T12:30:07.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.935+0000 7f4209e7b700 1 -- 172.21.15.16:0/905631815 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7f41e806b140 msgr2=0x7f41e806d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2550537276 2023-10-16T12:30:07.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.935+0000 7f4209e7b700 1 --2- 172.21.15.16:0/905631815 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7f41e806b140 0x7f41e806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:30:07.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.991+0000 7f420c0df700 1 -- 172.21.15.16:0/905631815 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f42040c8f80 con 0x7f420415a5c0 2023-10-16T12:30:07.994 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:30:07.992+0000 7f71a592b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T12:30:07.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.992+0000 7f41fa7fc700 1 -- 172.21.15.16:0/905631815 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+4909 (secure 0 0 0) 0x7f42040c8f80 con 0x7f420415a5c0 2023-10-16T12:30:07.994 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:30:07.995 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":61,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6893,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/2971585675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":2971585675},{"type":"v1","addr":"172.21.15.49:6835","nonce":2971585675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":7723,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/3151669585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":3151669585},{"type":"v1","addr":"172.21.15.49:6834","nonce":3151669585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":8106,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/52924937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":52924937},{"type":"v1","addr":"172.21.15.16:6836","nonce":52924937}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:29:15.801502+0000","modified":"2023-10-16T12:29:18.806285+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8109},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8109":{"gid":8109,"name":"a","rank":0,"incarnation":57,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/3203451346","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":3203451346},{"type":"v1","addr":"172.21.15.16:6837","nonce":3203451346}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2023-10-16T12:30:07.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.994+0000 7f420c0df700 1 -- 172.21.15.16:0/905631815 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7f41e806b140 msgr2=0x7f41e806d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:30:07.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.994+0000 7f420c0df700 1 --2- 172.21.15.16:0/905631815 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7f41e806b140 0x7f41e806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:07.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.994+0000 7f420c0df700 1 -- 172.21.15.16:0/905631815 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f420415a5c0 msgr2=0x7f42040b30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:07.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.994+0000 7f420c0df700 1 --2- 172.21.15.16:0/905631815 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f420415a5c0 0x7f42040b30c0 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f41fc00efc0 tx=0x7f41fc00c0c0 comp rx=0 tx=0).stop 2023-10-16T12:30:07.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.994+0000 7f420c0df700 1 -- 172.21.15.16:0/905631815 shutdown_connections 2023-10-16T12:30:07.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.994+0000 7f420c0df700 1 --2- 172.21.15.16:0/905631815 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7f41e806b140 0x7f41e806d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:07.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.994+0000 7f420c0df700 1 --2- 172.21.15.16:0/905631815 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f420415a5c0 0x7f42040b30c0 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:07.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.994+0000 7f420c0df700 1 --2- 172.21.15.16:0/905631815 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f42040ce010 0x7f4204154640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:07.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.994+0000 7f420c0df700 1 --2- 172.21.15.16:0/905631815 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f42040d0910 0x7f42040c9260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:07.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.994+0000 7f420c0df700 1 -- 172.21.15.16:0/905631815 >> 172.21.15.16:0/905631815 conn(0x7f42040b1fd0 msgr2=0x7f42040cd030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:07.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.994+0000 7f420c0df700 1 -- 172.21.15.16:0/905631815 shutdown_connections 2023-10-16T12:30:07.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:07.994+0000 7f420c0df700 1 -- 172.21.15.16:0/905631815 wait complete. 2023-10-16T12:30:07.999 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 61 2023-10-16T12:30:08.009 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:30:08.120 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:08.118+0000 7fc68f8d0200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T12:30:08.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.133+0000 7fe1e01df700 1 Processor -- start 2023-10-16T12:30:08.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.133+0000 7fe1e01df700 1 -- start start 2023-10-16T12:30:08.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.134+0000 7fe1e01df700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe1d80d41c0 0x7fe1d80d0040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:08.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.134+0000 7fe1e01df700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1d806ba00 con 0x7fe1d80d0580 2023-10-16T12:30:08.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.134+0000 7fe1e01df700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1d80d26d0 con 0x7fe1d815d8d0 2023-10-16T12:30:08.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.134+0000 7fe1e01df700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1d80d0a10 con 0x7fe1d80d41c0 2023-10-16T12:30:08.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.134+0000 7fe1dd77a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe1d80d41c0 0x7fe1d80d0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:08.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.134+0000 7fe1de77c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe1d80d0580 0x7fe1d806c4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33932/0 (socket says 172.21.15.16:33932) 2023-10-16T12:30:08.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.134+0000 7fe1dd77a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe1d80d41c0 0x7fe1d80d0040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44656/0 (socket says 172.21.15.16:44656) 2023-10-16T12:30:08.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.134+0000 7fe1de77c700 1 -- 172.21.15.16:0/2419947612 learned_addr learned my addr 172.21.15.16:0/2419947612 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:30:08.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.135+0000 7fe1dd77a700 1 -- 172.21.15.16:0/2419947612 >> v1:172.21.15.49:6789/0 conn(0x7fe1d815d8d0 legacy=0x7fe1d80d1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:08.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.135+0000 7fe1dd77a700 1 -- 172.21.15.16:0/2419947612 >> v1:172.21.15.16:6789/0 conn(0x7fe1d80d0580 legacy=0x7fe1d806c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:08.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.135+0000 7fe1dcf79700 1 -- 172.21.15.16:0/2419947612 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2338326611 0 0) 0x7fe1d806ba00 con 0x7fe1d80d0580 2023-10-16T12:30:08.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.135+0000 7fe1dd77a700 1 -- 172.21.15.16:0/2419947612 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1d80d23f0 con 0x7fe1d80d41c0 2023-10-16T12:30:08.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.135+0000 7fe1dd77a700 1 --2- 172.21.15.16:0/2419947612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1d80d41c0 0x7fe1d80d0040 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7fe1c8009840 tx=0x7fe1c8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea6e2d8a252acc3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:08.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.135+0000 7fe1dcf79700 1 -- 172.21.15.16:0/2419947612 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1c8011830 con 0x7fe1d80d41c0 2023-10-16T12:30:08.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.135+0000 7fe1dcf79700 1 -- 172.21.15.16:0/2419947612 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1c8011990 con 0x7fe1d80d41c0 2023-10-16T12:30:08.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.135+0000 7fe1dcf79700 1 -- 172.21.15.16:0/2419947612 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1c8011d20 con 0x7fe1d80d41c0 2023-10-16T12:30:08.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.136+0000 7fe1e01df700 1 -- 172.21.15.16:0/2419947612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1d80d41c0 msgr2=0x7fe1d80d0040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:08.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.136+0000 7fe1e01df700 1 --2- 172.21.15.16:0/2419947612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1d80d41c0 0x7fe1d80d0040 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7fe1c8009840 tx=0x7fe1c8009b50 comp rx=0 tx=0).stop 2023-10-16T12:30:08.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.136+0000 7fe1e01df700 1 -- 172.21.15.16:0/2419947612 shutdown_connections 2023-10-16T12:30:08.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.136+0000 7fe1e01df700 1 --2- 172.21.15.16:0/2419947612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1d80d41c0 0x7fe1d80d0040 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:08.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.136+0000 7fe1e01df700 1 -- 172.21.15.16:0/2419947612 >> 172.21.15.16:0/2419947612 conn(0x7fe1d80d3630 msgr2=0x7fe1d80d3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:08.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.136+0000 7fe1e01df700 1 -- 172.21.15.16:0/2419947612 shutdown_connections 2023-10-16T12:30:08.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.136+0000 7fe1e01df700 1 -- 172.21.15.16:0/2419947612 wait complete. 2023-10-16T12:30:08.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.137+0000 7fe1e01df700 1 Processor -- start 2023-10-16T12:30:08.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.137+0000 7fe1e01df700 1 -- start start 2023-10-16T12:30:08.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.138+0000 7fe1e01df700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1d80d0580 0x7fe1d815a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:08.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.138+0000 7fe1e01df700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe1d80d41c0 0x7fe1d81f2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:08.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.138+0000 7fe1e01df700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe1d815d8d0 0x7fe1d80cc630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:08.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.138+0000 7fe1e01df700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d80c6de0 con 0x7fe1d80d0580 2023-10-16T12:30:08.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.138+0000 7fe1e01df700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d80c73a0 con 0x7fe1d815d8d0 2023-10-16T12:30:08.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.138+0000 7fe1e01df700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d80cd0f0 con 0x7fe1d80d41c0 2023-10-16T12:30:08.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.138+0000 7fe1ddf7b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1d80d0580 0x7fe1d815a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:08.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.138+0000 7fe1dd77a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe1d80d41c0 0x7fe1d81f2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:08.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.138+0000 7fe1de77c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe1d815d8d0 0x7fe1d80cc630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:08.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.138+0000 7fe1ddf7b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1d80d0580 0x7fe1d815a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44660/0 (socket says 172.21.15.16:44660) 2023-10-16T12:30:08.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.138+0000 7fe1dd77a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe1d80d41c0 0x7fe1d81f2ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:32920/0 (socket says 172.21.15.16:32920) 2023-10-16T12:30:08.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.138+0000 7fe1ddf7b700 1 -- 172.21.15.16:0/1179291968 learned_addr learned my addr 172.21.15.16:0/1179291968 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:08.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.138+0000 7fe1ddf7b700 1 -- 172.21.15.16:0/1179291968 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe1d80d41c0 msgr2=0x7fe1d81f2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:08.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.138+0000 7fe1ddf7b700 1 --2- 172.21.15.16:0/1179291968 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe1d80d41c0 0x7fe1d81f2ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:08.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.138+0000 7fe1ddf7b700 1 -- 172.21.15.16:0/1179291968 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe1d815d8d0 msgr2=0x7fe1d80cc630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:08.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.138+0000 7fe1ddf7b700 1 --2- 172.21.15.16:0/1179291968 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe1d815d8d0 0x7fe1d80cc630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:08.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.138+0000 7fe1ddf7b700 1 -- 172.21.15.16:0/1179291968 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1c80094a0 con 0x7fe1d80d0580 2023-10-16T12:30:08.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.139+0000 7fe1ddf7b700 1 --2- 172.21.15.16:0/1179291968 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1d80d0580 0x7fe1d815a410 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7fe1d400c480 tx=0x7fe1d400c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:08.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.139+0000 7fe1ce7fc700 1 -- 172.21.15.16:0/1179291968 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d40052d0 con 0x7fe1d80d0580 2023-10-16T12:30:08.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.139+0000 7fe1ce7fc700 1 -- 172.21.15.16:0/1179291968 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1d4005430 con 0x7fe1d80d0580 2023-10-16T12:30:08.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.139+0000 7fe1ce7fc700 1 -- 172.21.15.16:0/1179291968 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d40038c0 con 0x7fe1d80d0580 2023-10-16T12:30:08.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.139+0000 7fe1e01df700 1 -- 172.21.15.16:0/1179291968 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1d80cd3d0 con 0x7fe1d80d0580 2023-10-16T12:30:08.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.139+0000 7fe1e01df700 1 -- 172.21.15.16:0/1179291968 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1d815a950 con 0x7fe1d80d0580 2023-10-16T12:30:08.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.140+0000 7fe1e01df700 1 -- 172.21.15.16:0/1179291968 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1d804ebb0 con 0x7fe1d80d0580 2023-10-16T12:30:08.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.140+0000 7fe1ce7fc700 1 -- 172.21.15.16:0/1179291968 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 70) v1 ==== 89445+0+0 (secure 0 0 0) 0x7fe1d4003a20 con 0x7fe1d80d0580 2023-10-16T12:30:08.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.141+0000 7fe1ce7fc700 1 --2- 172.21.15.16:0/1179291968 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7fe1bc06b140 0x7fe1bc06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:08.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.141+0000 7fe1dd77a700 1 -- 172.21.15.16:0/1179291968 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7fe1bc06b140 msgr2=0x7fe1bc06d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2550537276 2023-10-16T12:30:08.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.141+0000 7fe1ce7fc700 1 -- 172.21.15.16:0/1179291968 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fe1d4088f90 con 0x7fe1d80d0580 2023-10-16T12:30:08.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.141+0000 7fe1dd77a700 1 --2- 172.21.15.16:0/1179291968 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7fe1bc06b140 0x7fe1bc06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:30:08.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.145+0000 7fe1ce7fc700 1 -- 172.21.15.16:0/1179291968 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe1d40bbfe0 con 0x7fe1d80d0580 2023-10-16T12:30:08.162 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:30:08.160+0000 7f71a592b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T12:30:08.296 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:08.294+0000 7fc68f8d0200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T12:30:08.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.342+0000 7fe1dd77a700 1 -- 172.21.15.16:0/1179291968 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7fe1bc06b140 msgr2=0x7fe1bc06d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2550537276 2023-10-16T12:30:08.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.342+0000 7fe1dd77a700 1 --2- 172.21.15.16:0/1179291968 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7fe1bc06b140 0x7fe1bc06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:30:08.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.351+0000 7fe1e01df700 1 -- 172.21.15.16:0/1179291968 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe1d80b1210 con 0x7fe1d80d0580 2023-10-16T12:30:08.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.352+0000 7fe1ce7fc700 1 -- 172.21.15.16:0/1179291968 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v129) v1 ==== 74+0+17150 (secure 0 0 0) 0x7fe1d4017020 con 0x7fe1d80d0580 2023-10-16T12:30:08.354 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:30:08.355 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":129,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:29:30.732485+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:29:14.232554+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2023-10-16T12:29:14.764132+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/32319707":"2023-10-17T12:29:30.732445+0000","172.21.15.16:0/1058620826":"2023-10-17T12:29:30.732445+0000","172.21.15.16:0/3098064509":"2023-10-17T12:29:30.732445+0000","172.21.15.16:0/3814896496":"2023-10-17T12:29:30.732445+0000","172.21.15.16:0/2973963882":"2023-10-17T12:29:30.732445+0000","172.21.15.16:6833/2309613638":"2023-10-17T12:29:30.732445+0000","172.21.15.16:6832/2309613638":"2023-10-17T12:29:30.732445+0000","172.21.15.16:0/1074148914":"2023-10-17T12:29:30.732445+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:30:08.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.355+0000 7fe1e01df700 1 -- 172.21.15.16:0/1179291968 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7fe1bc06b140 msgr2=0x7fe1bc06d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:30:08.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.355+0000 7fe1e01df700 1 --2- 172.21.15.16:0/1179291968 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7fe1bc06b140 0x7fe1bc06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:08.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.355+0000 7fe1e01df700 1 -- 172.21.15.16:0/1179291968 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1d80d0580 msgr2=0x7fe1d815a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:08.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.355+0000 7fe1e01df700 1 --2- 172.21.15.16:0/1179291968 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1d80d0580 0x7fe1d815a410 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7fe1d400c480 tx=0x7fe1d400c790 comp rx=0 tx=0).stop 2023-10-16T12:30:08.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.355+0000 7fe1e01df700 1 -- 172.21.15.16:0/1179291968 shutdown_connections 2023-10-16T12:30:08.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.355+0000 7fe1e01df700 1 --2- 172.21.15.16:0/1179291968 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7fe1bc06b140 0x7fe1bc06d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:08.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.355+0000 7fe1e01df700 1 --2- 172.21.15.16:0/1179291968 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe1d815d8d0 0x7fe1d80cc630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:08.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.355+0000 7fe1e01df700 1 --2- 172.21.15.16:0/1179291968 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1d80d0580 0x7fe1d815a410 unknown :-1 s=CLOSED pgs=1460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:08.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.355+0000 7fe1e01df700 1 --2- 172.21.15.16:0/1179291968 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe1d80d41c0 0x7fe1d81f2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:08.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.355+0000 7fe1e01df700 1 -- 172.21.15.16:0/1179291968 >> 172.21.15.16:0/1179291968 conn(0x7fe1d80d3630 msgr2=0x7fe1d8160c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:08.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.355+0000 7fe1e01df700 1 -- 172.21.15.16:0/1179291968 shutdown_connections 2023-10-16T12:30:08.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.355+0000 7fe1e01df700 1 -- 172.21.15.16:0/1179291968 wait complete. 2023-10-16T12:30:08.369 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-16T12:30:08.370 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:30:08.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.494+0000 7f6cf0b36700 1 Processor -- start 2023-10-16T12:30:08.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.494+0000 7f6cf0b36700 1 -- start start 2023-10-16T12:30:08.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.495+0000 7f6cf0b36700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f6cec0cdfa0 0x7f6cec0b2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:08.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.495+0000 7f6cf0b36700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6cec0ce380 0x7f6cec0b2980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:08.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.495+0000 7f6cf0b36700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cec0d0fc0 con 0x7f6cec0d04a0 2023-10-16T12:30:08.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.495+0000 7f6cf0b36700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cec0d1210 con 0x7f6cec0ce380 2023-10-16T12:30:08.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.495+0000 7f6cf0b36700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cec0cbc30 con 0x7f6cec0cdfa0 2023-10-16T12:30:08.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.495+0000 7f6ce9d9b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6cec0ce380 0x7f6cec0b2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:08.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.495+0000 7f6ce9d9b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6cec0ce380 0x7f6cec0b2980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44672/0 (socket says 172.21.15.16:44672) 2023-10-16T12:30:08.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.495+0000 7f6cead9d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f6cec0d04a0 0x7f6cec0d08b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33948/0 (socket says 172.21.15.16:33948) 2023-10-16T12:30:08.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.495+0000 7f6ce9d9b700 1 -- 172.21.15.16:0/2641058515 learned_addr learned my addr 172.21.15.16:0/2641058515 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:08.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.495+0000 7f6cea59c700 1 --2- 172.21.15.16:0/2641058515 >> v2:172.21.15.49:3300/0 conn(0x7f6cec0cdfa0 0x7f6cec0b2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:08.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.496+0000 7f6ce9d9b700 1 -- 172.21.15.16:0/2641058515 >> v2:172.21.15.49:3300/0 conn(0x7f6cec0cdfa0 msgr2=0x7f6cec0b2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:08.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.496+0000 7f6ce9d9b700 1 --2- 172.21.15.16:0/2641058515 >> v2:172.21.15.49:3300/0 conn(0x7f6cec0cdfa0 0x7f6cec0b2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:08.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.496+0000 7f6ce9d9b700 1 -- 172.21.15.16:0/2641058515 >> v1:172.21.15.16:6789/0 conn(0x7f6cec0d04a0 legacy=0x7f6cec0d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:08.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.496+0000 7f6ce9d9b700 1 -- 172.21.15.16:0/2641058515 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cec0cbf10 con 0x7f6cec0ce380 2023-10-16T12:30:08.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.496+0000 7f6ce959a700 1 -- 172.21.15.16:0/2641058515 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3242533739 0 0) 0x7f6cec0d0fc0 con 0x7f6cec0d04a0 2023-10-16T12:30:08.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.496+0000 7f6ce9d9b700 1 --2- 172.21.15.16:0/2641058515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cec0ce380 0x7f6cec0b2980 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f6cec067350 tx=0x7f6ce0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b1ac1ca46e9f48 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:08.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.496+0000 7f6ce959a700 1 -- 172.21.15.16:0/2641058515 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce0011830 con 0x7f6cec0ce380 2023-10-16T12:30:08.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.496+0000 7f6ce959a700 1 -- 172.21.15.16:0/2641058515 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ce0011990 con 0x7f6cec0ce380 2023-10-16T12:30:08.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.496+0000 7f6ce959a700 1 -- 172.21.15.16:0/2641058515 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce0011d20 con 0x7f6cec0ce380 2023-10-16T12:30:08.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.496+0000 7f6cf0b36700 1 -- 172.21.15.16:0/2641058515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cec0ce380 msgr2=0x7f6cec0b2980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:08.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.496+0000 7f6cf0b36700 1 --2- 172.21.15.16:0/2641058515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cec0ce380 0x7f6cec0b2980 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f6cec067350 tx=0x7f6ce0009bd0 comp rx=0 tx=0).stop 2023-10-16T12:30:08.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.497+0000 7f6cf0b36700 1 -- 172.21.15.16:0/2641058515 shutdown_connections 2023-10-16T12:30:08.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.497+0000 7f6cf0b36700 1 --2- 172.21.15.16:0/2641058515 >> v2:172.21.15.49:3300/0 conn(0x7f6cec0cdfa0 0x7f6cec0b2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:08.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.497+0000 7f6cf0b36700 1 --2- 172.21.15.16:0/2641058515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cec0ce380 0x7f6cec0b2980 unknown :-1 s=CLOSED pgs=1462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:08.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.497+0000 7f6cf0b36700 1 -- 172.21.15.16:0/2641058515 >> 172.21.15.16:0/2641058515 conn(0x7f6cec0d14b0 msgr2=0x7f6cec0d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:08.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.497+0000 7f6cf0b36700 1 -- 172.21.15.16:0/2641058515 shutdown_connections 2023-10-16T12:30:08.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.497+0000 7f6cf0b36700 1 -- 172.21.15.16:0/2641058515 wait complete. 2023-10-16T12:30:08.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.498+0000 7f6cf0b36700 1 Processor -- start 2023-10-16T12:30:08.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.498+0000 7f6cf0b36700 1 -- start start 2023-10-16T12:30:08.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.499+0000 7f6cf0b36700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cec0cdfa0 0x7f6cec155a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:08.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.499+0000 7f6cf0b36700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6cec0ce380 0x7f6cec15f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:08.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.499+0000 7f6cf0b36700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6cec0d04a0 0x7f6cec15f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:08.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.499+0000 7f6cf0b36700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cec15e970 con 0x7f6cec0cdfa0 2023-10-16T12:30:08.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.499+0000 7f6cf0b36700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cec15dcf0 con 0x7f6cec0ce380 2023-10-16T12:30:08.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.499+0000 7f6cf0b36700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cec15dfd0 con 0x7f6cec0d04a0 2023-10-16T12:30:08.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.499+0000 7f6cea59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cec0cdfa0 0x7f6cec155a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:08.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.499+0000 7f6cea59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cec0cdfa0 0x7f6cec155a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44676/0 (socket says 172.21.15.16:44676) 2023-10-16T12:30:08.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.499+0000 7f6cea59c700 1 -- 172.21.15.16:0/4022824730 learned_addr learned my addr 172.21.15.16:0/4022824730 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:08.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.499+0000 7f6cead9d700 1 --2- 172.21.15.16:0/4022824730 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6cec0d04a0 0x7f6cec15f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:08.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.499+0000 7f6ce9d9b700 1 --2- 172.21.15.16:0/4022824730 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6cec0ce380 0x7f6cec15f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:08.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.499+0000 7f6cea59c700 1 -- 172.21.15.16:0/4022824730 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6cec0d04a0 msgr2=0x7f6cec15f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:08.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.499+0000 7f6cea59c700 1 --2- 172.21.15.16:0/4022824730 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6cec0d04a0 0x7f6cec15f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:08.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.499+0000 7f6cea59c700 1 -- 172.21.15.16:0/4022824730 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6cec0ce380 msgr2=0x7f6cec15f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:08.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.499+0000 7f6cea59c700 1 --2- 172.21.15.16:0/4022824730 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6cec0ce380 0x7f6cec15f380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:08.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.499+0000 7f6cea59c700 1 -- 172.21.15.16:0/4022824730 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ce0009550 con 0x7f6cec0cdfa0 2023-10-16T12:30:08.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.499+0000 7f6cea59c700 1 --2- 172.21.15.16:0/4022824730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cec0cdfa0 0x7f6cec155a70 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f6cdc00e9b0 tx=0x7f6cdc00ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:08.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.499+0000 7f6cdaffd700 1 -- 172.21.15.16:0/4022824730 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cdc009710 con 0x7f6cec0cdfa0 2023-10-16T12:30:08.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.500+0000 7f6cdaffd700 1 -- 172.21.15.16:0/4022824730 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cdc00ee50 con 0x7f6cec0cdfa0 2023-10-16T12:30:08.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.500+0000 7f6cf0b36700 1 -- 172.21.15.16:0/4022824730 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cec15e2b0 con 0x7f6cec0cdfa0 2023-10-16T12:30:08.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.500+0000 7f6cdaffd700 1 -- 172.21.15.16:0/4022824730 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cdc00ca50 con 0x7f6cec0cdfa0 2023-10-16T12:30:08.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.500+0000 7f6cf0b36700 1 -- 172.21.15.16:0/4022824730 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cec155fb0 con 0x7f6cec0cdfa0 2023-10-16T12:30:08.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.500+0000 7f6cf0b36700 1 -- 172.21.15.16:0/4022824730 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cec04ebb0 con 0x7f6cec0cdfa0 2023-10-16T12:30:08.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.502+0000 7f6cdaffd700 1 -- 172.21.15.16:0/4022824730 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 70) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f6cdc00cbb0 con 0x7f6cec0cdfa0 2023-10-16T12:30:08.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.502+0000 7f6cdaffd700 1 --2- 172.21.15.16:0/4022824730 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7f6ccc06b140 0x7f6ccc06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:08.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.502+0000 7f6ce9d9b700 1 -- 172.21.15.16:0/4022824730 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7f6ccc06b140 msgr2=0x7f6ccc06d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2550537276 2023-10-16T12:30:08.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.502+0000 7f6cdaffd700 1 -- 172.21.15.16:0/4022824730 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f6cdc08a350 con 0x7f6cec0cdfa0 2023-10-16T12:30:08.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.502+0000 7f6ce9d9b700 1 --2- 172.21.15.16:0/4022824730 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7f6ccc06b140 0x7f6ccc06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:30:08.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.506+0000 7f6cdaffd700 1 -- 172.21.15.16:0/4022824730 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6cdc054f50 con 0x7f6cec0cdfa0 2023-10-16T12:30:08.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.703+0000 7f6ce9d9b700 1 -- 172.21.15.16:0/4022824730 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7f6ccc06b140 msgr2=0x7f6ccc06d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2550537276 2023-10-16T12:30:08.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.703+0000 7f6ce9d9b700 1 --2- 172.21.15.16:0/4022824730 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7f6ccc06b140 0x7f6ccc06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:30:08.719 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:30:08.717+0000 7f71a592b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T12:30:08.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.770+0000 7f6cf0b36700 1 -- 172.21.15.16:0/4022824730 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6cec0cfc00 con 0x7f6cec0cdfa0 2023-10-16T12:30:08.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.771+0000 7f6cdaffd700 1 -- 172.21.15.16:0/4022824730 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+4909 (secure 0 0 0) 0x7f6cdc00ce60 con 0x7f6cec0cdfa0 2023-10-16T12:30:08.773 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:30:08.773 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":61,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6893,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/2971585675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":2971585675},{"type":"v1","addr":"172.21.15.49:6835","nonce":2971585675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":7723,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/3151669585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":3151669585},{"type":"v1","addr":"172.21.15.49:6834","nonce":3151669585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":8106,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/52924937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":52924937},{"type":"v1","addr":"172.21.15.16:6836","nonce":52924937}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:29:15.801502+0000","modified":"2023-10-16T12:29:18.806285+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8109},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8109":{"gid":8109,"name":"a","rank":0,"incarnation":57,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/3203451346","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":3203451346},{"type":"v1","addr":"172.21.15.16:6837","nonce":3203451346}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2023-10-16T12:30:08.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.773+0000 7f6cf0b36700 1 -- 172.21.15.16:0/4022824730 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7f6ccc06b140 msgr2=0x7f6ccc06d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:30:08.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.773+0000 7f6cf0b36700 1 --2- 172.21.15.16:0/4022824730 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7f6ccc06b140 0x7f6ccc06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:08.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.773+0000 7f6cf0b36700 1 -- 172.21.15.16:0/4022824730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cec0cdfa0 msgr2=0x7f6cec155a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:08.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.773+0000 7f6cf0b36700 1 --2- 172.21.15.16:0/4022824730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cec0cdfa0 0x7f6cec155a70 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f6cdc00e9b0 tx=0x7f6cdc00ecc0 comp rx=0 tx=0).stop 2023-10-16T12:30:08.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.773+0000 7f6cf0b36700 1 -- 172.21.15.16:0/4022824730 shutdown_connections 2023-10-16T12:30:08.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.773+0000 7f6cf0b36700 1 --2- 172.21.15.16:0/4022824730 >> [v2:172.21.15.16:6832/2550537276,v1:172.21.15.16:6833/2550537276] conn(0x7f6ccc06b140 0x7f6ccc06d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:08.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.773+0000 7f6cf0b36700 1 --2- 172.21.15.16:0/4022824730 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6cec0d04a0 0x7f6cec15f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:08.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.773+0000 7f6cf0b36700 1 --2- 172.21.15.16:0/4022824730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cec0cdfa0 0x7f6cec155a70 unknown :-1 s=CLOSED pgs=1463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:08.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.773+0000 7f6cf0b36700 1 --2- 172.21.15.16:0/4022824730 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6cec0ce380 0x7f6cec15f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:08.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.773+0000 7f6cf0b36700 1 -- 172.21.15.16:0/4022824730 >> 172.21.15.16:0/4022824730 conn(0x7f6cec0d14b0 msgr2=0x7f6cec156920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:08.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.773+0000 7f6cf0b36700 1 -- 172.21.15.16:0/4022824730 shutdown_connections 2023-10-16T12:30:08.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.773+0000 7f6cf0b36700 1 -- 172.21.15.16:0/4022824730 wait complete. 2023-10-16T12:30:08.778 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 61 2023-10-16T12:30:08.788 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:30:08.884 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:08.882+0000 7fc68f8d0200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T12:30:08.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.919+0000 7f96cd1c7700 1 Processor -- start 2023-10-16T12:30:08.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.919+0000 7f96cd1c7700 1 -- start start 2023-10-16T12:30:08.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.919+0000 7f96cd1c7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f96c815db40 0x7f96c80cb980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:08.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.919+0000 7f96cd1c7700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f96c80ce870 0x7f96c806ac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:08.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.919+0000 7f96cd1c7700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96c80cedb0 con 0x7f96c806a7e0 2023-10-16T12:30:08.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.919+0000 7f96cd1c7700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96c806b1a0 con 0x7f96c815db40 2023-10-16T12:30:08.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.919+0000 7f96cd1c7700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96c815df20 con 0x7f96c80ce870 2023-10-16T12:30:08.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.919+0000 7f96c6d9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f96c815db40 0x7f96c80cb980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:08.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.919+0000 7f96c6d9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f96c815db40 0x7f96c80cb980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44706/0 (socket says 172.21.15.16:44706) 2023-10-16T12:30:08.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.920+0000 7f96c6d9d700 1 -- 172.21.15.16:0/3431567361 learned_addr learned my addr 172.21.15.16:0/3431567361 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:08.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.920+0000 7f96c759e700 1 --2- 172.21.15.16:0/3431567361 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f96c80ce870 0x7f96c806ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:08.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.920+0000 7f96c6d9d700 1 -- 172.21.15.16:0/3431567361 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f96c80ce870 msgr2=0x7f96c806ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:08.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.920+0000 7f96c6d9d700 1 --2- 172.21.15.16:0/3431567361 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f96c80ce870 0x7f96c806ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:08.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.920+0000 7f96c6d9d700 1 -- 172.21.15.16:0/3431567361 >> v1:172.21.15.49:6789/0 conn(0x7f96c806a7e0 legacy=0x7f96c80ce160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:08.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.920+0000 7f96c6d9d700 1 -- 172.21.15.16:0/3431567361 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96c80d12c0 con 0x7f96c815db40 2023-10-16T12:30:08.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.920+0000 7f96c759e700 1 --2- 172.21.15.16:0/3431567361 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f96c80ce870 0x7f96c806ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:30:08.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.920+0000 7f96c6d9d700 1 --2- 172.21.15.16:0/3431567361 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f96c815db40 0x7f96c80cb980 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f96b80098f0 tx=0x7f96b8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bda40ddcc486e899 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:08.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.920+0000 7f96c5d9b700 1 -- 172.21.15.16:0/3431567361 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96b800b8a0 con 0x7f96c815db40 2023-10-16T12:30:08.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.920+0000 7f96c5d9b700 1 -- 172.21.15.16:0/3431567361 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96b800ba00 con 0x7f96c815db40 2023-10-16T12:30:08.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.920+0000 7f96c5d9b700 1 -- 172.21.15.16:0/3431567361 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96b800bd90 con 0x7f96c815db40 2023-10-16T12:30:08.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.921+0000 7f96cd1c7700 1 -- 172.21.15.16:0/3431567361 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f96c815db40 msgr2=0x7f96c80cb980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:08.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.921+0000 7f96cd1c7700 1 --2- 172.21.15.16:0/3431567361 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f96c815db40 0x7f96c80cb980 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f96b80098f0 tx=0x7f96b8009c00 comp rx=0 tx=0).stop 2023-10-16T12:30:08.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.921+0000 7f96cd1c7700 1 -- 172.21.15.16:0/3431567361 shutdown_connections 2023-10-16T12:30:08.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.921+0000 7f96cd1c7700 1 --2- 172.21.15.16:0/3431567361 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f96c80ce870 0x7f96c806ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:08.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.921+0000 7f96cd1c7700 1 --2- 172.21.15.16:0/3431567361 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f96c815db40 0x7f96c80cb980 unknown :-1 s=CLOSED pgs=1465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:08.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.921+0000 7f96cd1c7700 1 -- 172.21.15.16:0/3431567361 >> 172.21.15.16:0/3431567361 conn(0x7f96c80b1fd0 msgr2=0x7f96c80b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:08.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.921+0000 7f96cd1c7700 1 -- 172.21.15.16:0/3431567361 shutdown_connections 2023-10-16T12:30:08.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.921+0000 7f96cd1c7700 1 -- 172.21.15.16:0/3431567361 wait complete. 2023-10-16T12:30:08.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.922+0000 7f96cd1c7700 1 Processor -- start 2023-10-16T12:30:08.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.922+0000 7f96cd1c7700 1 -- start start 2023-10-16T12:30:08.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.923+0000 7f96cd1c7700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f96c806a7e0 0x7f96c81f3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:08.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.923+0000 7f96cd1c7700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f96c80ce870 0x7f96c80ae0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:08.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.923+0000 7f96cd1c7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f96c815db40 0x7f96c80c9520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:08.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.923+0000 7f96cd1c7700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96c81f3bd0 con 0x7f96c815db40 2023-10-16T12:30:08.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.923+0000 7f96cd1c7700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96c81f61e0 con 0x7f96c806a7e0 2023-10-16T12:30:08.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.923+0000 7f96cd1c7700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96c80ae620 con 0x7f96c80ce870 2023-10-16T12:30:08.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.923+0000 7f96c759e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f96c815db40 0x7f96c80c9520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:08.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.923+0000 7f96c6d9d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f96c806a7e0 0x7f96c81f3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:08.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.923+0000 7f96c6d9d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f96c806a7e0 0x7f96c81f3690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:43774/0 (socket says 172.21.15.16:43774) 2023-10-16T12:30:08.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.923+0000 7f96c759e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f96c815db40 0x7f96c80c9520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44716/0 (socket says 172.21.15.16:44716) 2023-10-16T12:30:08.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.923+0000 7f96c6d9d700 1 -- 172.21.15.16:0/376988467 learned_addr learned my addr 172.21.15.16:0/376988467 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:08.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.923+0000 7f96c659c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f96c80ce870 0x7f96c80ae0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:08.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.923+0000 7f96c6d9d700 1 -- 172.21.15.16:0/376988467 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f96c80ce870 msgr2=0x7f96c80ae0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:08.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.923+0000 7f96c6d9d700 1 --2- 172.21.15.16:0/376988467 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f96c80ce870 0x7f96c80ae0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:08.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.923+0000 7f96c6d9d700 1 -- 172.21.15.16:0/376988467 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f96c815db40 msgr2=0x7f96c80c9520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:08.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.923+0000 7f96c6d9d700 1 --2- 172.21.15.16:0/376988467 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f96c815db40 0x7f96c80c9520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:08.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.923+0000 7f96c6d9d700 1 -- 172.21.15.16:0/376988467 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96b8009550 con 0x7f96c806a7e0 2023-10-16T12:30:08.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.923+0000 7f96c759e700 1 --2- 172.21.15.16:0/376988467 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f96c815db40 0x7f96c80c9520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:30:08.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.924+0000 7f96c6d9d700 1 --2- 172.21.15.16:0/376988467 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f96c806a7e0 0x7f96c81f3690 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f96b8009c40 tx=0x7f96b8004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:08.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.924+0000 7f96af7fe700 1 -- 172.21.15.16:0/376988467 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96b8004300 con 0x7f96c806a7e0 2023-10-16T12:30:08.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.924+0000 7f96af7fe700 1 -- 172.21.15.16:0/376988467 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96b801f070 con 0x7f96c806a7e0 2023-10-16T12:30:08.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.924+0000 7f96af7fe700 1 -- 172.21.15.16:0/376988467 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96b80036a0 con 0x7f96c806a7e0 2023-10-16T12:30:08.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.924+0000 7f96cd1c7700 1 -- 172.21.15.16:0/376988467 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96c80c4470 con 0x7f96c806a7e0 2023-10-16T12:30:08.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.924+0000 7f96cd1c7700 1 -- 172.21.15.16:0/376988467 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96c81f6760 con 0x7f96c806a7e0 2023-10-16T12:30:08.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.925+0000 7f96cd1c7700 1 -- 172.21.15.16:0/376988467 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96c804ebb0 con 0x7f96c806a7e0 2023-10-16T12:30:08.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.925+0000 7f96af7fe700 1 -- 172.21.15.16:0/376988467 <== mon.1 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 71) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f96b8021030 con 0x7f96c806a7e0 2023-10-16T12:30:08.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.926+0000 7f96af7fe700 1 -- 172.21.15.16:0/376988467 <== mon.1 v2:172.21.15.49:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6831+0+0 (secure 0 0 0) 0x7f96b808abb0 con 0x7f96c806a7e0 2023-10-16T12:30:08.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:08.930+0000 7f96af7fe700 1 -- 172.21.15.16:0/376988467 <== mon.1 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f96b8055c80 con 0x7f96c806a7e0 2023-10-16T12:30:08.970 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:08.967+0000 7fc60b0fc700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:30:08.970 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:08.967+0000 7fc60b0fc700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:30:08.970 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:08.967+0000 7fc60b0fc700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:30:08.970 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:08.967+0000 7fc60b0fc700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:30:08.971 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:08.967+0000 7fc60b0fc700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:30:08.971 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:08.968+0000 7fc6080f6700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:30:08.971 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:08.968+0000 7fc6080f6700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:30:08.971 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:08.968+0000 7fc6080f6700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:30:08.971 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:08.968+0000 7fc6080f6700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:30:08.971 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:08.968+0000 7fc6080f6700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:30:09.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.190+0000 7f96cd1c7700 1 -- 172.21.15.16:0/376988467 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f96c815d000 con 0x7f96c806a7e0 2023-10-16T12:30:09.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.190+0000 7f96af7fe700 1 -- 172.21.15.16:0/376988467 <== mon.1 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+4909 (secure 0 0 0) 0x7f96b80592a0 con 0x7f96c806a7e0 2023-10-16T12:30:09.193 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:30:09.193 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":61,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6893,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/2971585675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":2971585675},{"type":"v1","addr":"172.21.15.49:6835","nonce":2971585675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":7723,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/3151669585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":3151669585},{"type":"v1","addr":"172.21.15.49:6834","nonce":3151669585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":8106,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/52924937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":52924937},{"type":"v1","addr":"172.21.15.16:6836","nonce":52924937}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:29:15.801502+0000","modified":"2023-10-16T12:29:18.806285+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8109},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8109":{"gid":8109,"name":"a","rank":0,"incarnation":57,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/3203451346","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":3203451346},{"type":"v1","addr":"172.21.15.16:6837","nonce":3203451346}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2023-10-16T12:30:09.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.192+0000 7f96cd1c7700 1 -- 172.21.15.16:0/376988467 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f96c806a7e0 msgr2=0x7f96c81f3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:09.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.192+0000 7f96cd1c7700 1 --2- 172.21.15.16:0/376988467 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f96c806a7e0 0x7f96c81f3690 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f96b8009c40 tx=0x7f96b8004750 comp rx=0 tx=0).stop 2023-10-16T12:30:09.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.192+0000 7f96cd1c7700 1 -- 172.21.15.16:0/376988467 shutdown_connections 2023-10-16T12:30:09.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.192+0000 7f96cd1c7700 1 --2- 172.21.15.16:0/376988467 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f96c815db40 0x7f96c80c9520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:09.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.192+0000 7f96cd1c7700 1 --2- 172.21.15.16:0/376988467 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f96c806a7e0 0x7f96c81f3690 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:09.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.192+0000 7f96cd1c7700 1 --2- 172.21.15.16:0/376988467 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f96c80ce870 0x7f96c80ae0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:09.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.193+0000 7f96cd1c7700 1 -- 172.21.15.16:0/376988467 >> 172.21.15.16:0/376988467 conn(0x7f96c80b1fd0 msgr2=0x7f96c80ce130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:09.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.193+0000 7f96cd1c7700 1 -- 172.21.15.16:0/376988467 shutdown_connections 2023-10-16T12:30:09.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.193+0000 7f96cd1c7700 1 -- 172.21.15.16:0/376988467 wait complete. 2023-10-16T12:30:09.197 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 61 2023-10-16T12:30:09.207 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:30:09.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.329+0000 7f6b4bb9e700 1 Processor -- start 2023-10-16T12:30:09.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.330+0000 7f6b4bb9e700 1 -- start start 2023-10-16T12:30:09.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.330+0000 7f6b4bb9e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6b440cf880 0x7f6b440b52f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:09.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.330+0000 7f6b4bb9e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6b440b5830 0x7f6b4415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:09.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.330+0000 7f6b4bb9e700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b4415bc80 con 0x7f6b440cf4a0 2023-10-16T12:30:09.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.330+0000 7f6b4bb9e700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b4415b070 con 0x7f6b440cf880 2023-10-16T12:30:09.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.330+0000 7f6b4bb9e700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b4415b350 con 0x7f6b440b5830 2023-10-16T12:30:09.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.331+0000 7f6b49139700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6b440cf880 0x7f6b440b52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:09.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.331+0000 7f6b49139700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6b440cf880 0x7f6b440b52f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44772/0 (socket says 172.21.15.16:44772) 2023-10-16T12:30:09.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.331+0000 7f6b49139700 1 -- 172.21.15.16:0/499709818 learned_addr learned my addr 172.21.15.16:0/499709818 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:09.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.331+0000 7f6b49139700 1 -- 172.21.15.16:0/499709818 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6b440b5830 msgr2=0x7f6b4415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:09.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.331+0000 7f6b4a13b700 1 --2- 172.21.15.16:0/499709818 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6b440b5830 0x7f6b4415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:09.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.331+0000 7f6b49139700 1 --2- 172.21.15.16:0/499709818 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6b440b5830 0x7f6b4415a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:09.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.331+0000 7f6b49139700 1 -- 172.21.15.16:0/499709818 >> v1:172.21.15.16:6789/0 conn(0x7f6b440cf4a0 legacy=0x7f6b440726e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:09.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.331+0000 7f6b49139700 1 -- 172.21.15.16:0/499709818 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b4415b630 con 0x7f6b440cf880 2023-10-16T12:30:09.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.331+0000 7f6b49139700 1 --2- 172.21.15.16:0/499709818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b440cf880 0x7f6b440b52f0 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7f6b400098f0 tx=0x7f6b40009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4922cb83f234b1c6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:09.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.331+0000 7f6b48938700 1 -- 172.21.15.16:0/499709818 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b4000b8a0 con 0x7f6b440cf880 2023-10-16T12:30:09.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.331+0000 7f6b48938700 1 -- 172.21.15.16:0/499709818 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b4000ba00 con 0x7f6b440cf880 2023-10-16T12:30:09.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.331+0000 7f6b48938700 1 -- 172.21.15.16:0/499709818 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b4000bd90 con 0x7f6b440cf880 2023-10-16T12:30:09.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.332+0000 7f6b4bb9e700 1 -- 172.21.15.16:0/499709818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b440cf880 msgr2=0x7f6b440b52f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:09.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.332+0000 7f6b4bb9e700 1 --2- 172.21.15.16:0/499709818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b440cf880 0x7f6b440b52f0 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7f6b400098f0 tx=0x7f6b40009c00 comp rx=0 tx=0).stop 2023-10-16T12:30:09.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.332+0000 7f6b4bb9e700 1 -- 172.21.15.16:0/499709818 shutdown_connections 2023-10-16T12:30:09.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.332+0000 7f6b4bb9e700 1 --2- 172.21.15.16:0/499709818 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6b440b5830 0x7f6b4415a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:09.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.332+0000 7f6b4bb9e700 1 --2- 172.21.15.16:0/499709818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b440cf880 0x7f6b440b52f0 unknown :-1 s=CLOSED pgs=1473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:09.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.332+0000 7f6b4bb9e700 1 -- 172.21.15.16:0/499709818 >> 172.21.15.16:0/499709818 conn(0x7f6b440b1fd0 msgr2=0x7f6b440b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:09.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.332+0000 7f6b4bb9e700 1 -- 172.21.15.16:0/499709818 shutdown_connections 2023-10-16T12:30:09.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.332+0000 7f6b4bb9e700 1 -- 172.21.15.16:0/499709818 wait complete. 2023-10-16T12:30:09.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.333+0000 7f6b4bb9e700 1 Processor -- start 2023-10-16T12:30:09.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.333+0000 7f6b4bb9e700 1 -- start start 2023-10-16T12:30:09.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.334+0000 7f6b4bb9e700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6b440b5830 0x7f6b44150c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:09.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.334+0000 7f6b4bb9e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b440cf4a0 0x7f6b441679f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:09.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.334+0000 7f6b4bb9e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6b440cf880 0x7f6b44167f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:09.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.334+0000 7f6b4bb9e700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b440c3f70 con 0x7f6b440cf4a0 2023-10-16T12:30:09.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.334+0000 7f6b4bb9e700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b440c41c0 con 0x7f6b440b5830 2023-10-16T12:30:09.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.334+0000 7f6b4bb9e700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b440c44a0 con 0x7f6b440cf880 2023-10-16T12:30:09.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.335+0000 7f6b49139700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b440cf4a0 0x7f6b441679f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:09.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.335+0000 7f6b49139700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b440cf4a0 0x7f6b441679f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44778/0 (socket says 172.21.15.16:44778) 2023-10-16T12:30:09.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.335+0000 7f6b49139700 1 -- 172.21.15.16:0/597657783 learned_addr learned my addr 172.21.15.16:0/597657783 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:09.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.335+0000 7f6b4993a700 1 --2- 172.21.15.16:0/597657783 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6b440b5830 0x7f6b44150c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:09.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.335+0000 7f6b4a13b700 1 --2- 172.21.15.16:0/597657783 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6b440cf880 0x7f6b44167f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:09.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.335+0000 7f6b49139700 1 -- 172.21.15.16:0/597657783 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6b440cf880 msgr2=0x7f6b44167f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:09.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.335+0000 7f6b49139700 1 --2- 172.21.15.16:0/597657783 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6b440cf880 0x7f6b44167f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:09.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.335+0000 7f6b49139700 1 -- 172.21.15.16:0/597657783 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6b440b5830 msgr2=0x7f6b44150c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:09.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.335+0000 7f6b49139700 1 --2- 172.21.15.16:0/597657783 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6b440b5830 0x7f6b44150c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:09.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.335+0000 7f6b49139700 1 -- 172.21.15.16:0/597657783 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b40009550 con 0x7f6b440cf4a0 2023-10-16T12:30:09.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.335+0000 7f6b49139700 1 --2- 172.21.15.16:0/597657783 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b440cf4a0 0x7f6b441679f0 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7f6b40009130 tx=0x7f6b40004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:09.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.337+0000 7f6b3a7fc700 1 -- 172.21.15.16:0/597657783 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b400036a0 con 0x7f6b440cf4a0 2023-10-16T12:30:09.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.337+0000 7f6b3a7fc700 1 -- 172.21.15.16:0/597657783 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b4001e070 con 0x7f6b440cf4a0 2023-10-16T12:30:09.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.337+0000 7f6b4bb9e700 1 -- 172.21.15.16:0/597657783 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b440c4780 con 0x7f6b440cf4a0 2023-10-16T12:30:09.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.337+0000 7f6b3a7fc700 1 -- 172.21.15.16:0/597657783 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b40003a40 con 0x7f6b440cf4a0 2023-10-16T12:30:09.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.337+0000 7f6b4bb9e700 1 -- 172.21.15.16:0/597657783 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b441511b0 con 0x7f6b440cf4a0 2023-10-16T12:30:09.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.338+0000 7f6b4bb9e700 1 -- 172.21.15.16:0/597657783 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b440c4ca0 con 0x7f6b440cf4a0 2023-10-16T12:30:09.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.338+0000 7f6b3a7fc700 1 -- 172.21.15.16:0/597657783 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 71) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f6b40003ba0 con 0x7f6b440cf4a0 2023-10-16T12:30:09.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.339+0000 7f6b3a7fc700 1 -- 172.21.15.16:0/597657783 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6831+0+0 (secure 0 0 0) 0x7f6b40020070 con 0x7f6b440cf4a0 2023-10-16T12:30:09.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.343+0000 7f6b3a7fc700 1 -- 172.21.15.16:0/597657783 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6b440c4ca0 con 0x7f6b440cf4a0 2023-10-16T12:30:09.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.548+0000 7f6b4bb9e700 1 -- 172.21.15.16:0/597657783 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6b440c4ca0 con 0x7f6b440cf4a0 2023-10-16T12:30:09.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.549+0000 7f6b3a7fc700 1 -- 172.21.15.16:0/597657783 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v130) v1 ==== 74+0+17773 (secure 0 0 0) 0x7f6b440c4ca0 con 0x7f6b440cf4a0 2023-10-16T12:30:09.551 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:30:09.551 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":130,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:30:08.896893+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:29:14.232554+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2023-10-16T12:29:14.764132+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/2093940709":"2023-10-17T12:30:08.896838+0000","172.21.15.16:0/3924602056":"2023-10-17T12:30:08.896838+0000","172.21.15.16:0/3368698219":"2023-10-17T12:30:08.896838+0000","172.21.15.16:0/10052963":"2023-10-17T12:30:08.896838+0000","172.21.15.16:0/667000197":"2023-10-17T12:30:08.896838+0000","172.21.15.16:0/1018845767":"2023-10-17T12:30:08.896838+0000","172.21.15.16:0/3485123903":"2023-10-17T12:30:08.896838+0000","172.21.15.16:6832/2550537276":"2023-10-17T12:30:08.896838+0000","172.21.15.16:0/32319707":"2023-10-17T12:29:30.732445+0000","172.21.15.16:0/1058620826":"2023-10-17T12:29:30.732445+0000","172.21.15.16:0/3098064509":"2023-10-17T12:29:30.732445+0000","172.21.15.16:0/3814896496":"2023-10-17T12:29:30.732445+0000","172.21.15.16:0/2973963882":"2023-10-17T12:29:30.732445+0000","172.21.15.16:6833/2550537276":"2023-10-17T12:30:08.896838+0000","172.21.15.16:6833/2309613638":"2023-10-17T12:29:30.732445+0000","172.21.15.16:0/4081830650":"2023-10-17T12:30:08.896838+0000","172.21.15.16:6832/2309613638":"2023-10-17T12:29:30.732445+0000","172.21.15.16:0/1074148914":"2023-10-17T12:29:30.732445+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:30:09.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.551+0000 7f6b4bb9e700 1 -- 172.21.15.16:0/597657783 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b440cf4a0 msgr2=0x7f6b441679f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:09.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.551+0000 7f6b4bb9e700 1 --2- 172.21.15.16:0/597657783 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b440cf4a0 0x7f6b441679f0 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7f6b40009130 tx=0x7f6b40004750 comp rx=0 tx=0).stop 2023-10-16T12:30:09.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.551+0000 7f6b4bb9e700 1 -- 172.21.15.16:0/597657783 shutdown_connections 2023-10-16T12:30:09.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.551+0000 7f6b4bb9e700 1 --2- 172.21.15.16:0/597657783 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6b440cf880 0x7f6b44167f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:09.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.551+0000 7f6b4bb9e700 1 --2- 172.21.15.16:0/597657783 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6b440b5830 0x7f6b44150c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:09.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.551+0000 7f6b4bb9e700 1 --2- 172.21.15.16:0/597657783 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b440cf4a0 0x7f6b441679f0 unknown :-1 s=CLOSED pgs=1474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:09.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.551+0000 7f6b4bb9e700 1 -- 172.21.15.16:0/597657783 >> 172.21.15.16:0/597657783 conn(0x7f6b440b1fd0 msgr2=0x7f6b4415ccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:09.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.551+0000 7f6b4bb9e700 1 -- 172.21.15.16:0/597657783 shutdown_connections 2023-10-16T12:30:09.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.551+0000 7f6b4bb9e700 1 -- 172.21.15.16:0/597657783 wait complete. 2023-10-16T12:30:09.565 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-16T12:30:09.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.689+0000 7f31035c8700 1 Processor -- start 2023-10-16T12:30:09.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.690+0000 7f31035c8700 1 -- start start 2023-10-16T12:30:09.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.690+0000 7f31035c8700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f30fc0cf4a0 0x7f30fc0726e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:09.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.690+0000 7f31035c8700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f30fc0b5a00 0x7f30fc15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:09.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.690+0000 7f31035c8700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30fc15bc80 con 0x7f30fc0cf880 2023-10-16T12:30:09.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.690+0000 7f31035c8700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30fc0b5e60 con 0x7f30fc0b5a00 2023-10-16T12:30:09.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.690+0000 7f31035c8700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30fc072c20 con 0x7f30fc0cf4a0 2023-10-16T12:30:09.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.691+0000 7f3101b65700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f30fc0b5a00 0x7f30fc15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:09.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.691+0000 7f3101364700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f30fc0cf4a0 0x7f30fc0726e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:09.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.691+0000 7f3101b65700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f30fc0b5a00 0x7f30fc15a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:43848/0 (socket says 172.21.15.16:43848) 2023-10-16T12:30:09.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.691+0000 7f3101364700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f30fc0cf4a0 0x7f30fc0726e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:33040/0 (socket says 172.21.15.16:33040) 2023-10-16T12:30:09.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.691+0000 7f3100b63700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7f30fc0cf880 0x7f30fc0b52f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:41238/0 (socket says 172.21.15.16:41238) 2023-10-16T12:30:09.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.691+0000 7f3101b65700 1 -- 172.21.15.16:0/2729114575 learned_addr learned my addr 172.21.15.16:0/2729114575 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:09.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.691+0000 7f3101b65700 1 -- 172.21.15.16:0/2729114575 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f30fc0cf4a0 msgr2=0x7f30fc0726e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:09.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.691+0000 7f3101b65700 1 --2- 172.21.15.16:0/2729114575 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f30fc0cf4a0 0x7f30fc0726e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:09.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.691+0000 7f3101b65700 1 -- 172.21.15.16:0/2729114575 >> v1:172.21.15.49:6789/0 conn(0x7f30fc0cf880 legacy=0x7f30fc0b52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:09.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.691+0000 7f3101b65700 1 -- 172.21.15.16:0/2729114575 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30fc15b070 con 0x7f30fc0b5a00 2023-10-16T12:30:09.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.691+0000 7f30f3fff700 1 -- 172.21.15.16:0/2729114575 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2222829410 0 0) 0x7f30fc15bc80 con 0x7f30fc0cf880 2023-10-16T12:30:09.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.691+0000 7f3101364700 1 --2- 172.21.15.16:0/2729114575 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f30fc0cf4a0 0x7f30fc0726e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:30:09.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.692+0000 7f3101b65700 1 --2- 172.21.15.16:0/2729114575 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f30fc0b5a00 0x7f30fc15a090 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f30f400d150 tx=0x7f30f400d460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=647d4287a54240fd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:09.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.692+0000 7f30f3fff700 1 -- 172.21.15.16:0/2729114575 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30f4010070 con 0x7f30fc0b5a00 2023-10-16T12:30:09.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.692+0000 7f30f3fff700 1 -- 172.21.15.16:0/2729114575 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30f400dd10 con 0x7f30fc0b5a00 2023-10-16T12:30:09.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.692+0000 7f30f3fff700 1 -- 172.21.15.16:0/2729114575 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30f400b310 con 0x7f30fc0b5a00 2023-10-16T12:30:09.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.692+0000 7f31035c8700 1 -- 172.21.15.16:0/2729114575 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f30fc0b5a00 msgr2=0x7f30fc15a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:09.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.692+0000 7f31035c8700 1 --2- 172.21.15.16:0/2729114575 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f30fc0b5a00 0x7f30fc15a090 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f30f400d150 tx=0x7f30f400d460 comp rx=0 tx=0).stop 2023-10-16T12:30:09.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.692+0000 7f31035c8700 1 -- 172.21.15.16:0/2729114575 shutdown_connections 2023-10-16T12:30:09.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.692+0000 7f31035c8700 1 --2- 172.21.15.16:0/2729114575 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f30fc0b5a00 0x7f30fc15a090 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:09.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.692+0000 7f31035c8700 1 --2- 172.21.15.16:0/2729114575 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f30fc0cf4a0 0x7f30fc0726e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:09.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.692+0000 7f31035c8700 1 -- 172.21.15.16:0/2729114575 >> 172.21.15.16:0/2729114575 conn(0x7f30fc0b1fd0 msgr2=0x7f30fc0b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:09.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.692+0000 7f31035c8700 1 -- 172.21.15.16:0/2729114575 shutdown_connections 2023-10-16T12:30:09.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.692+0000 7f31035c8700 1 -- 172.21.15.16:0/2729114575 wait complete. 2023-10-16T12:30:09.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.693+0000 7f31035c8700 1 Processor -- start 2023-10-16T12:30:09.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.694+0000 7f31035c8700 1 -- start start 2023-10-16T12:30:09.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.694+0000 7f31035c8700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f30fc0b5a00 0x7f30fc14c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:09.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.694+0000 7f31035c8700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f30fc0cf4a0 0x7f30fc152050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:09.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.694+0000 7f31035c8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30fc0cf880 0x7f30fc0ac9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:09.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.694+0000 7f31035c8700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30fc0b3d80 con 0x7f30fc0cf880 2023-10-16T12:30:09.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.694+0000 7f31035c8700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30fc14c820 con 0x7f30fc0cf4a0 2023-10-16T12:30:09.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.694+0000 7f31035c8700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30fc0af810 con 0x7f30fc0b5a00 2023-10-16T12:30:09.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.694+0000 7f3101b65700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30fc0cf880 0x7f30fc0ac9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:09.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.694+0000 7f3101b65700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30fc0cf880 0x7f30fc0ac9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44792/0 (socket says 172.21.15.16:44792) 2023-10-16T12:30:09.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.694+0000 7f3101b65700 1 -- 172.21.15.16:0/3710024722 learned_addr learned my addr 172.21.15.16:0/3710024722 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:09.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.695+0000 7f3101364700 1 --2- 172.21.15.16:0/3710024722 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f30fc0b5a00 0x7f30fc14c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:09.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.695+0000 7f3100b63700 1 --2- 172.21.15.16:0/3710024722 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f30fc0cf4a0 0x7f30fc152050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:09.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.695+0000 7f3101b65700 1 -- 172.21.15.16:0/3710024722 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f30fc0b5a00 msgr2=0x7f30fc14c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:09.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.695+0000 7f3101b65700 1 --2- 172.21.15.16:0/3710024722 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f30fc0b5a00 0x7f30fc14c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:09.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.695+0000 7f3101b65700 1 -- 172.21.15.16:0/3710024722 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f30fc0cf4a0 msgr2=0x7f30fc152050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:09.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.695+0000 7f3101b65700 1 --2- 172.21.15.16:0/3710024722 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f30fc0cf4a0 0x7f30fc152050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:09.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.695+0000 7f3101b65700 1 -- 172.21.15.16:0/3710024722 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30ec0094a0 con 0x7f30fc0cf880 2023-10-16T12:30:09.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.695+0000 7f3100b63700 1 --2- 172.21.15.16:0/3710024722 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f30fc0cf4a0 0x7f30fc152050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:30:09.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.695+0000 7f3101b65700 1 --2- 172.21.15.16:0/3710024722 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30fc0cf880 0x7f30fc0ac9f0 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f30f400d7b0 tx=0x7f30f4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:09.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.695+0000 7f30f1ffb700 1 -- 172.21.15.16:0/3710024722 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30f4009a10 con 0x7f30fc0cf880 2023-10-16T12:30:09.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.695+0000 7f30f1ffb700 1 -- 172.21.15.16:0/3710024722 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30f4009b70 con 0x7f30fc0cf880 2023-10-16T12:30:09.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.695+0000 7f31035c8700 1 -- 172.21.15.16:0/3710024722 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30f400bd40 con 0x7f30fc0cf880 2023-10-16T12:30:09.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.695+0000 7f30f1ffb700 1 -- 172.21.15.16:0/3710024722 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30f4004040 con 0x7f30fc0cf880 2023-10-16T12:30:09.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.695+0000 7f31035c8700 1 -- 172.21.15.16:0/3710024722 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30fc0b3730 con 0x7f30fc0cf880 2023-10-16T12:30:09.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.696+0000 7f31035c8700 1 -- 172.21.15.16:0/3710024722 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30fc04ebb0 con 0x7f30fc0cf880 2023-10-16T12:30:09.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.697+0000 7f30f1ffb700 1 -- 172.21.15.16:0/3710024722 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 71) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f30f40041a0 con 0x7f30fc0cf880 2023-10-16T12:30:09.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.697+0000 7f30f1ffb700 1 -- 172.21.15.16:0/3710024722 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6831+0+0 (secure 0 0 0) 0x7f30f408aed0 con 0x7f30fc0cf880 2023-10-16T12:30:09.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.702+0000 7f30f1ffb700 1 -- 172.21.15.16:0/3710024722 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f30f4055f70 con 0x7f30fc0cf880 2023-10-16T12:30:09.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.911+0000 7f30f1ffb700 1 -- 172.21.15.16:0/3710024722 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 72) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f30f4055a20 con 0x7f30fc0cf880 2023-10-16T12:30:09.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.912+0000 7f30f1ffb700 1 --2- 172.21.15.16:0/3710024722 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f30e0070bc0 0x7f30e0072fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:09.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.912+0000 7f3101364700 1 --2- 172.21.15.16:0/3710024722 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f30e0070bc0 0x7f30e0072fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:09.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.913+0000 7f3101364700 1 --2- 172.21.15.16:0/3710024722 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f30e0070bc0 0x7f30e0072fb0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f30ec009d70 tx=0x7f30ec009210 comp rx=0 tx=0).ready entity=mgr.8388 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:09.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.963+0000 7f31035c8700 1 -- 172.21.15.16:0/3710024722 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f30fc152590 con 0x7f30fc0cf880 2023-10-16T12:30:09.966 INFO:tasks.ceph.mon.a.smithi016.stderr:2023-10-16T12:30:09.964+0000 7f9fc8535700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-16T12:30:09.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.965+0000 7f30f1ffb700 1 -- 172.21.15.16:0/3710024722 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v62) v1 ==== 161+0+0 (secure 0 0 0) 0x7f30f406f030 con 0x7f30fc0cf880 2023-10-16T12:30:09.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.969+0000 7f31035c8700 1 -- 172.21.15.16:0/3710024722 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f30e0070bc0 msgr2=0x7f30e0072fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:09.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.969+0000 7f31035c8700 1 --2- 172.21.15.16:0/3710024722 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f30e0070bc0 0x7f30e0072fb0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f30ec009d70 tx=0x7f30ec009210 comp rx=0 tx=0).stop 2023-10-16T12:30:09.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.969+0000 7f31035c8700 1 -- 172.21.15.16:0/3710024722 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30fc0cf880 msgr2=0x7f30fc0ac9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:09.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.969+0000 7f31035c8700 1 --2- 172.21.15.16:0/3710024722 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30fc0cf880 0x7f30fc0ac9f0 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f30f400d7b0 tx=0x7f30f4004ad0 comp rx=0 tx=0).stop 2023-10-16T12:30:09.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.969+0000 7f31035c8700 1 -- 172.21.15.16:0/3710024722 shutdown_connections 2023-10-16T12:30:09.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.969+0000 7f31035c8700 1 --2- 172.21.15.16:0/3710024722 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30fc0cf880 0x7f30fc0ac9f0 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:09.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.969+0000 7f31035c8700 1 --2- 172.21.15.16:0/3710024722 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f30fc0b5a00 0x7f30fc14c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:09.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.969+0000 7f31035c8700 1 --2- 172.21.15.16:0/3710024722 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f30e0070bc0 0x7f30e0072fb0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:09.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.969+0000 7f31035c8700 1 --2- 172.21.15.16:0/3710024722 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f30fc0cf4a0 0x7f30fc152050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:09.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.969+0000 7f31035c8700 1 -- 172.21.15.16:0/3710024722 >> 172.21.15.16:0/3710024722 conn(0x7f30fc0b1fd0 msgr2=0x7f30fc0afe40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:09.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.969+0000 7f31035c8700 1 -- 172.21.15.16:0/3710024722 shutdown_connections 2023-10-16T12:30:09.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:09.969+0000 7f31035c8700 1 -- 172.21.15.16:0/3710024722 wait complete. 2023-10-16T12:30:09.974 INFO:teuthology.orchestra.run.smithi016.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-16T12:30:09.979 INFO:tasks.ceph.mds.a.smithi016.stderr: -8905> 2023-10-16T12:29:21.359+0000 7fd6de734700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T12:30:09.988 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-16T12:30:10.017 INFO:tasks.ceph.mds.a.smithi016.stdout:starting mds.a at 2023-10-16T12:30:10.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:10.118+0000 7f66a64fe700 1 Processor -- start 2023-10-16T12:30:10.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:10.118+0000 7f66a64fe700 1 -- start start 2023-10-16T12:30:10.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:10.119+0000 7f66a64fe700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f66a00ce5d0 0x7f66a00cc190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:10.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:10.119+0000 7f66a64fe700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66a00b2440 con 0x7f66a015f060 2023-10-16T12:30:10.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:10.119+0000 7f66a64fe700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66a00d00e0 con 0x7f66a015ec80 2023-10-16T12:30:10.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:10.119+0000 7f66a64fe700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66a00b2c50 con 0x7f66a00ce5d0 2023-10-16T12:30:10.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:10.119+0000 7f66a4a9b700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f66a015f060 0x7f66a00cc6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34108/0 (socket says 172.21.15.16:34108) 2023-10-16T12:30:10.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:10.119+0000 7f66a4a9b700 1 -- 172.21.15.16:0/1011571664 learned_addr learned my addr 172.21.15.16:0/1011571664 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:30:10.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:10.119+0000 7f669f7fe700 1 --2- 172.21.15.16:0/1011571664 >> v2:172.21.15.49:3300/0 conn(0x7f66a00ce5d0 0x7f66a00cc190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:10.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:10.119+0000 7f669effd700 1 -- 172.21.15.16:0/1011571664 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 770643028 0 0) 0x7f66a00b2440 con 0x7f66a015f060 2023-10-16T12:30:10.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:10.119+0000 7f669effd700 1 -- 172.21.15.16:0/1011571664 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f668c0035b0 con 0x7f66a015f060 2023-10-16T12:30:10.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:10.119+0000 7f669effd700 1 -- 172.21.15.16:0/1011571664 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3469548728 0 0) 0x7f66a00d00e0 con 0x7f66a015ec80 2023-10-16T12:30:10.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:10.119+0000 7f669effd700 1 -- 172.21.15.16:0/1011571664 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66a00b2440 con 0x7f66a015ec80 2023-10-16T12:30:10.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:10.120+0000 7f669effd700 1 -- 172.21.15.16:0/1011571664 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2130843715 0 0) 0x7f668c0035b0 con 0x7f66a015f060 2023-10-16T12:30:10.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:10.120+0000 7f669f7fe700 1 -- 172.21.15.16:0/1011571664 >> v1:172.21.15.49:6789/0 conn(0x7f66a015ec80 legacy=0x7f66a00cdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:10.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:10.120+0000 7f669f7fe700 1 -- 172.21.15.16:0/1011571664 >> v1:172.21.15.16:6789/0 conn(0x7f66a015f060 legacy=0x7f66a00cc6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:10.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:10.120+0000 7f669f7fe700 1 -- 172.21.15.16:0/1011571664 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66a00cef60 con 0x7f66a00ce5d0 2023-10-16T12:30:10.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:10.120+0000 7f669f7fe700 1 --2- 172.21.15.16:0/1011571664 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f66a00ce5d0 0x7f66a00cc190 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f6688009870 tx=0x7f6688009b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bc419e907f1e9590 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:10.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:10.120+0000 7f669effd700 1 -- 172.21.15.16:0/1011571664 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66980019f0 con 0x7f66a00ce5d0 2023-10-16T12:30:10.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:10.120+0000 7f669effd700 1 -- 172.21.15.16:0/1011571664 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6688004030 con 0x7f66a00ce5d0 2023-10-16T12:30:10.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:10.120+0000 7f669effd700 1 -- 172.21.15.16:0/1011571664 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66880043c0 con 0x7f66a00ce5d0 2023-10-16T12:30:10.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:10.120+0000 7f66a64fe700 1 -- 172.21.15.16:0/1011571664 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f66a00ce5d0 msgr2=0x7f66a00cc190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:10.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:10.120+0000 7f66a64fe700 1 --2- 172.21.15.16:0/1011571664 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f66a00ce5d0 0x7f66a00cc190 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f6688009870 tx=0x7f6688009b80 comp rx=0 tx=0).stop 2023-10-16T12:30:10.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:10.121+0000 7f66a64fe700 1 -- 172.21.15.16:0/1011571664 shutdown_connections 2023-10-16T12:30:10.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:10.121+0000 7f66a64fe700 1 --2- 172.21.15.16:0/1011571664 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f66a00ce5d0 0x7f66a00cc190 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:10.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:10.121+0000 7f66a64fe700 1 -- 172.21.15.16:0/1011571664 >> 172.21.15.16:0/1011571664 conn(0x7f66a00d15f0 msgr2=0x7f66a00d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:10.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:10.121+0000 7f66a64fe700 1 -- 172.21.15.16:0/1011571664 shutdown_connections 2023-10-16T12:30:10.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:10.121+0000 7f66a64fe700 1 -- 172.21.15.16:0/1011571664 wait complete. 2023-10-16T12:30:10.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:10.122+0000 7f66a64fe700 1 Processor -- start 2023-10-16T12:30:10.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:10.122+0000 7f66a64fe700 1 -- start start 2023-10-16T12:30:10.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:10.123+0000 7f66a64fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f66a00ce5d0 0x7f66a00c77a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:10.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:10.123+0000 7f66a64fe700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f66a015ec80 0x7f66a01f4820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:10.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:10.123+0000 7f66a64fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66a015f060 0x7f66a015c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:10.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:10.123+0000 7f66a64fe700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66a01f57c0 con 0x7f66a015f060 2023-10-16T12:30:10.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:10.123+0000 7f66a64fe700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66a00c6d00 con 0x7f66a00ce5d0 2023-10-16T12:30:10.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:10.123+0000 7f66a64fe700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66a01f5200 con 0x7f66a015ec80 2023-10-16T12:30:10.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:10.123+0000 7f66a4a9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66a015f060 0x7f66a015c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:10.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:10.123+0000 7f66a4a9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66a015f060 0x7f66a015c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44840/0 (socket says 172.21.15.16:44840) 2023-10-16T12:30:10.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:10.123+0000 7f66a4a9b700 1 -- 172.21.15.16:0/3412083623 learned_addr learned my addr 172.21.15.16:0/3412083623 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:10.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:10.123+0000 7f669ffff700 1 --2- 172.21.15.16:0/3412083623 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f66a00ce5d0 0x7f66a00c77a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:10.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:10.123+0000 7f669f7fe700 1 --2- 172.21.15.16:0/3412083623 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f66a015ec80 0x7f66a01f4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:10.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:10.123+0000 7f66a4a9b700 1 -- 172.21.15.16:0/3412083623 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f66a015ec80 msgr2=0x7f66a01f4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:10.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:10.123+0000 7f66a4a9b700 1 --2- 172.21.15.16:0/3412083623 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f66a015ec80 0x7f66a01f4820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:10.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:10.123+0000 7f66a4a9b700 1 -- 172.21.15.16:0/3412083623 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f66a00ce5d0 msgr2=0x7f66a00c77a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:10.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:10.123+0000 7f66a4a9b700 1 --2- 172.21.15.16:0/3412083623 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f66a00ce5d0 0x7f66a00c77a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:10.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:10.123+0000 7f66a4a9b700 1 -- 172.21.15.16:0/3412083623 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66880094a0 con 0x7f66a015f060 2023-10-16T12:30:10.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:10.123+0000 7f66a4a9b700 1 --2- 172.21.15.16:0/3412083623 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66a015f060 0x7f66a015c8f0 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f6698001350 tx=0x7f669800cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:10.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:10.123+0000 7f669cff9700 1 -- 172.21.15.16:0/3412083623 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f669800ab70 con 0x7f66a015f060 2023-10-16T12:30:10.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:10.123+0000 7f669cff9700 1 -- 172.21.15.16:0/3412083623 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f669800acd0 con 0x7f66a015f060 2023-10-16T12:30:10.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:10.123+0000 7f669cff9700 1 -- 172.21.15.16:0/3412083623 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6698007310 con 0x7f66a015f060 2023-10-16T12:30:10.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:10.123+0000 7f66a64fe700 1 -- 172.21.15.16:0/3412083623 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66a01f54e0 con 0x7f66a015f060 2023-10-16T12:30:10.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:10.124+0000 7f66a64fe700 1 -- 172.21.15.16:0/3412083623 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66a00c7280 con 0x7f66a015f060 2023-10-16T12:30:10.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:10.124+0000 7f66a64fe700 1 -- 172.21.15.16:0/3412083623 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66a0066a30 con 0x7f66a015f060 2023-10-16T12:30:10.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:10.125+0000 7f669cff9700 1 -- 172.21.15.16:0/3412083623 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 72) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f6698007470 con 0x7f66a015f060 2023-10-16T12:30:10.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:10.125+0000 7f669cff9700 1 --2- 172.21.15.16:0/3412083623 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f668006ad50 0x7f668006d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:10.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:10.125+0000 7f669cff9700 1 -- 172.21.15.16:0/3412083623 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6945+0+0 (secure 0 0 0) 0x7f66980891a0 con 0x7f66a015f060 2023-10-16T12:30:10.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:10.125+0000 7f669ffff700 1 --2- 172.21.15.16:0/3412083623 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f668006ad50 0x7f668006d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:10.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:10.126+0000 7f669ffff700 1 --2- 172.21.15.16:0/3412083623 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f668006ad50 0x7f668006d210 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f66900011e0 tx=0x7f6690009040 comp rx=0 tx=0).ready entity=mgr.8388 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:10.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:10.130+0000 7f669cff9700 1 -- 172.21.15.16:0/3412083623 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f66980540b0 con 0x7f66a015f060 2023-10-16T12:30:10.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:10.383+0000 7f66a64fe700 1 -- 172.21.15.16:0/3412083623 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f66a004ebb0 con 0x7f66a015f060 2023-10-16T12:30:10.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:10.966+0000 7f669cff9700 1 -- 172.21.15.16:0/3412083623 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v63) v1 ==== 104+0+0 (secure 0 0 0) 0x7f66980576d0 con 0x7f66a015f060 2023-10-16T12:30:10.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:10.970+0000 7f66a64fe700 1 -- 172.21.15.16:0/3412083623 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f668006ad50 msgr2=0x7f668006d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:10.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:10.970+0000 7f66a64fe700 1 --2- 172.21.15.16:0/3412083623 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f668006ad50 0x7f668006d210 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f66900011e0 tx=0x7f6690009040 comp rx=0 tx=0).stop 2023-10-16T12:30:10.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:10.970+0000 7f66a64fe700 1 -- 172.21.15.16:0/3412083623 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66a015f060 msgr2=0x7f66a015c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:10.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:10.970+0000 7f66a64fe700 1 --2- 172.21.15.16:0/3412083623 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66a015f060 0x7f66a015c8f0 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f6698001350 tx=0x7f669800cf50 comp rx=0 tx=0).stop 2023-10-16T12:30:10.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:10.970+0000 7f66a64fe700 1 -- 172.21.15.16:0/3412083623 shutdown_connections 2023-10-16T12:30:10.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:10.970+0000 7f66a64fe700 1 --2- 172.21.15.16:0/3412083623 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66a015f060 0x7f66a015c8f0 unknown :-1 s=CLOSED pgs=1477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:10.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:10.970+0000 7f66a64fe700 1 --2- 172.21.15.16:0/3412083623 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f668006ad50 0x7f668006d210 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:10.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:10.970+0000 7f66a64fe700 1 --2- 172.21.15.16:0/3412083623 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f66a00ce5d0 0x7f66a00c77a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:10.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:10.970+0000 7f66a64fe700 1 --2- 172.21.15.16:0/3412083623 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f66a015ec80 0x7f66a01f4820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:10.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:10.970+0000 7f66a64fe700 1 -- 172.21.15.16:0/3412083623 >> 172.21.15.16:0/3412083623 conn(0x7f66a00d15f0 msgr2=0x7f66a0153cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:10.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:10.970+0000 7f66a64fe700 1 -- 172.21.15.16:0/3412083623 shutdown_connections 2023-10-16T12:30:10.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:10.970+0000 7f66a64fe700 1 -- 172.21.15.16:0/3412083623 wait complete. 2023-10-16T12:30:10.989 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-16T12:30:11.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:11.158+0000 7f18c21dc700 1 Processor -- start 2023-10-16T12:30:11.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:11.159+0000 7f18c21dc700 1 -- start start 2023-10-16T12:30:11.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:11.159+0000 7f18c21dc700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f18bc15b500 0x7f18bc15fab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:11.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:11.159+0000 7f18c21dc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f18bc06e590 0x7f18bc0ca930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:11.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:11.159+0000 7f18c21dc700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f18bc0cae70 0x7f18bc071970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:11.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:11.159+0000 7f18c21dc700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18bc160090 con 0x7f18bc06e590 2023-10-16T12:30:11.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:11.159+0000 7f18c21dc700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18bc0cb320 con 0x7f18bc15b500 2023-10-16T12:30:11.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:11.159+0000 7f18c21dc700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18bc0cc800 con 0x7f18bc0cae70 2023-10-16T12:30:11.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:11.159+0000 7f18baffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f18bc06e590 0x7f18bc0ca930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:11.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:11.159+0000 7f18baffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f18bc06e590 0x7f18bc0ca930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44864/0 (socket says 172.21.15.16:44864) 2023-10-16T12:30:11.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:11.159+0000 7f18baffd700 1 -- 172.21.15.16:0/413112789 learned_addr learned my addr 172.21.15.16:0/413112789 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:11.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:11.160+0000 7f18bbfff700 1 --2- 172.21.15.16:0/413112789 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f18bc0cae70 0x7f18bc071970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:11.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:11.160+0000 7f18baffd700 1 -- 172.21.15.16:0/413112789 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f18bc0cae70 msgr2=0x7f18bc071970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:11.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:11.160+0000 7f18baffd700 1 --2- 172.21.15.16:0/413112789 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f18bc0cae70 0x7f18bc071970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:11.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:11.160+0000 7f18bb7fe700 1 --2- 172.21.15.16:0/413112789 >> v2:172.21.15.49:3300/0 conn(0x7f18bc15b500 0x7f18bc15fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:11.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:11.160+0000 7f18baffd700 1 -- 172.21.15.16:0/413112789 >> v2:172.21.15.49:3300/0 conn(0x7f18bc15b500 msgr2=0x7f18bc15fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:11.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:11.160+0000 7f18baffd700 1 --2- 172.21.15.16:0/413112789 >> v2:172.21.15.49:3300/0 conn(0x7f18bc15b500 0x7f18bc15fab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:11.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:11.160+0000 7f18baffd700 1 -- 172.21.15.16:0/413112789 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18bc06b220 con 0x7f18bc06e590 2023-10-16T12:30:11.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:11.160+0000 7f18bbfff700 1 --2- 172.21.15.16:0/413112789 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f18bc0cae70 0x7f18bc071970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:30:11.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:11.160+0000 7f18baffd700 1 --2- 172.21.15.16:0/413112789 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18bc06e590 0x7f18bc0ca930 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7f18bc073080 tx=0x7f18a4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b87275ebe376cc3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:11.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:11.160+0000 7f18ba7fc700 1 -- 172.21.15.16:0/413112789 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a400b870 con 0x7f18bc06e590 2023-10-16T12:30:11.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:11.160+0000 7f18ba7fc700 1 -- 172.21.15.16:0/413112789 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18a400b9d0 con 0x7f18bc06e590 2023-10-16T12:30:11.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:11.160+0000 7f18ba7fc700 1 -- 172.21.15.16:0/413112789 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a400bd60 con 0x7f18bc06e590 2023-10-16T12:30:11.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:11.161+0000 7f18c21dc700 1 -- 172.21.15.16:0/413112789 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18bc06e590 msgr2=0x7f18bc0ca930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:11.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:11.161+0000 7f18c21dc700 1 --2- 172.21.15.16:0/413112789 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18bc06e590 0x7f18bc0ca930 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7f18bc073080 tx=0x7f18a4009bd0 comp rx=0 tx=0).stop 2023-10-16T12:30:11.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:11.161+0000 7f18c21dc700 1 -- 172.21.15.16:0/413112789 shutdown_connections 2023-10-16T12:30:11.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:11.161+0000 7f18c21dc700 1 --2- 172.21.15.16:0/413112789 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f18bc0cae70 0x7f18bc071970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:11.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:11.161+0000 7f18c21dc700 1 --2- 172.21.15.16:0/413112789 >> v2:172.21.15.49:3300/0 conn(0x7f18bc15b500 0x7f18bc15fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:11.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:11.161+0000 7f18c21dc700 1 --2- 172.21.15.16:0/413112789 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18bc06e590 0x7f18bc0ca930 unknown :-1 s=CLOSED pgs=1480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:11.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:11.161+0000 7f18c21dc700 1 -- 172.21.15.16:0/413112789 >> 172.21.15.16:0/413112789 conn(0x7f18bc06f430 msgr2=0x7f18bc06f810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:11.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:11.161+0000 7f18c21dc700 1 -- 172.21.15.16:0/413112789 shutdown_connections 2023-10-16T12:30:11.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:11.161+0000 7f18c21dc700 1 -- 172.21.15.16:0/413112789 wait complete. 2023-10-16T12:30:11.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:11.162+0000 7f18c21dc700 1 Processor -- start 2023-10-16T12:30:11.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:11.162+0000 7f18c21dc700 1 -- start start 2023-10-16T12:30:11.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:11.163+0000 7f18c21dc700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f18bc06e590 0x7f18bc0ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:11.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:11.163+0000 7f18c21dc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18bc0cae70 0x7f18bc0ae000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:11.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:11.163+0000 7f18c21dc700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f18bc15b500 0x7f18bc158890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:11.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:11.163+0000 7f18c21dc700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18bc0b3050 con 0x7f18bc0cae70 2023-10-16T12:30:11.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:11.163+0000 7f18c21dc700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18bc0ae540 con 0x7f18bc15b500 2023-10-16T12:30:11.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:11.163+0000 7f18c21dc700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18bc0ad800 con 0x7f18bc06e590 2023-10-16T12:30:11.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:11.163+0000 7f18baffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18bc0cae70 0x7f18bc0ae000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:11.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:11.163+0000 7f18baffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18bc0cae70 0x7f18bc0ae000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44868/0 (socket says 172.21.15.16:44868) 2023-10-16T12:30:11.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:11.163+0000 7f18baffd700 1 -- 172.21.15.16:0/552348199 learned_addr learned my addr 172.21.15.16:0/552348199 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:11.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:11.163+0000 7f18bb7fe700 1 --2- 172.21.15.16:0/552348199 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f18bc06e590 0x7f18bc0ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:11.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:11.163+0000 7f18bbfff700 1 --2- 172.21.15.16:0/552348199 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f18bc15b500 0x7f18bc158890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:11.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:11.163+0000 7f18baffd700 1 -- 172.21.15.16:0/552348199 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f18bc06e590 msgr2=0x7f18bc0ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:11.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:11.163+0000 7f18baffd700 1 --2- 172.21.15.16:0/552348199 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f18bc06e590 0x7f18bc0ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:11.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:11.163+0000 7f18baffd700 1 -- 172.21.15.16:0/552348199 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f18bc15b500 msgr2=0x7f18bc158890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:11.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:11.163+0000 7f18baffd700 1 --2- 172.21.15.16:0/552348199 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f18bc15b500 0x7f18bc158890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:11.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:11.163+0000 7f18baffd700 1 -- 172.21.15.16:0/552348199 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18a4009550 con 0x7f18bc0cae70 2023-10-16T12:30:11.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:11.163+0000 7f18baffd700 1 --2- 172.21.15.16:0/552348199 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18bc0cae70 0x7f18bc0ae000 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f18a4014040 tx=0x7f18a4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:11.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:11.164+0000 7f18b3fff700 1 -- 172.21.15.16:0/552348199 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a4004250 con 0x7f18bc0cae70 2023-10-16T12:30:11.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:11.164+0000 7f18b3fff700 1 -- 172.21.15.16:0/552348199 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18a401e070 con 0x7f18bc0cae70 2023-10-16T12:30:11.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:11.164+0000 7f18c21dc700 1 -- 172.21.15.16:0/552348199 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18bc0acf60 con 0x7f18bc0cae70 2023-10-16T12:30:11.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:11.164+0000 7f18b3fff700 1 -- 172.21.15.16:0/552348199 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a40036a0 con 0x7f18bc0cae70 2023-10-16T12:30:11.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:11.164+0000 7f18c21dc700 1 -- 172.21.15.16:0/552348199 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18bc0ad520 con 0x7f18bc0cae70 2023-10-16T12:30:11.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:11.165+0000 7f18c21dc700 1 -- 172.21.15.16:0/552348199 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18bc0b3190 con 0x7f18bc0cae70 2023-10-16T12:30:11.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:11.165+0000 7f18b3fff700 1 -- 172.21.15.16:0/552348199 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 72) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f18a4020030 con 0x7f18bc0cae70 2023-10-16T12:30:11.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:11.166+0000 7f18b3fff700 1 --2- 172.21.15.16:0/552348199 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f189c06ad50 0x7f189c06d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:11.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:11.166+0000 7f18b3fff700 1 -- 172.21.15.16:0/552348199 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6945+0+0 (secure 0 0 0) 0x7f18a408a7a0 con 0x7f18bc0cae70 2023-10-16T12:30:11.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:11.166+0000 7f18bb7fe700 1 --2- 172.21.15.16:0/552348199 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f189c06ad50 0x7f189c06d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:11.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:11.166+0000 7f18bb7fe700 1 --2- 172.21.15.16:0/552348199 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f189c06ad50 0x7f189c06d210 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f18ac00aae0 tx=0x7f18ac008040 comp rx=0 tx=0).ready entity=mgr.8388 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:11.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:11.170+0000 7f18b3fff700 1 -- 172.21.15.16:0/552348199 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f18bc0b3190 con 0x7f18bc0cae70 2023-10-16T12:30:11.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:11.343+0000 7f18b3fff700 1 -- 172.21.15.16:0/552348199 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 73) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f18a408ccf0 con 0x7f18bc0cae70 2023-10-16T12:30:11.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:11.380+0000 7f18c21dc700 1 -- 172.21.15.16:0/552348199 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f18bc0b3190 con 0x7f18bc0cae70 2023-10-16T12:30:12.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.345+0000 7f18b3fff700 1 -- 172.21.15.16:0/552348199 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v133) v1 ==== 181+0+0 (secure 0 0 0) 0x7f18bc0b3190 con 0x7f18bc0cae70 2023-10-16T12:30:12.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.350+0000 7f18c21dc700 1 -- 172.21.15.16:0/552348199 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f189c06ad50 msgr2=0x7f189c06d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:12.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.350+0000 7f18c21dc700 1 --2- 172.21.15.16:0/552348199 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f189c06ad50 0x7f189c06d210 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f18ac00aae0 tx=0x7f18ac008040 comp rx=0 tx=0).stop 2023-10-16T12:30:12.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.350+0000 7f18c21dc700 1 -- 172.21.15.16:0/552348199 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18bc0cae70 msgr2=0x7f18bc0ae000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:12.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.350+0000 7f18c21dc700 1 --2- 172.21.15.16:0/552348199 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18bc0cae70 0x7f18bc0ae000 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f18a4014040 tx=0x7f18a4004750 comp rx=0 tx=0).stop 2023-10-16T12:30:12.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.351+0000 7f18c21dc700 1 -- 172.21.15.16:0/552348199 shutdown_connections 2023-10-16T12:30:12.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.351+0000 7f18c21dc700 1 --2- 172.21.15.16:0/552348199 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f189c06ad50 0x7f189c06d210 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:12.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.351+0000 7f18c21dc700 1 --2- 172.21.15.16:0/552348199 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f18bc15b500 0x7f18bc158890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:12.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.351+0000 7f18c21dc700 1 --2- 172.21.15.16:0/552348199 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f18bc06e590 0x7f18bc0ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:12.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.351+0000 7f18c21dc700 1 --2- 172.21.15.16:0/552348199 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18bc0cae70 0x7f18bc0ae000 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:12.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.351+0000 7f18c21dc700 1 -- 172.21.15.16:0/552348199 >> 172.21.15.16:0/552348199 conn(0x7f18bc06f430 msgr2=0x7f18bc06faf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:12.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.351+0000 7f18c21dc700 1 -- 172.21.15.16:0/552348199 shutdown_connections 2023-10-16T12:30:12.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.352+0000 7f18c21dc700 1 -- 172.21.15.16:0/552348199 wait complete. 2023-10-16T12:30:12.356 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' removed 2023-10-16T12:30:12.370 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-16T12:30:12.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.505+0000 7fc62291c700 1 Processor -- start 2023-10-16T12:30:12.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.505+0000 7fc62291c700 1 -- start start 2023-10-16T12:30:12.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.505+0000 7fc62291c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc61c15a3d0 0x7fc61c06d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:12.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.505+0000 7fc62291c700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fc61c15e0b0 0x7fc61c0ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:12.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.505+0000 7fc62291c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc61c0cf430 0x7fc61c0cdeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:12.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.505+0000 7fc62291c700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc61c15f0c0 con 0x7fc61c0cf430 2023-10-16T12:30:12.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.505+0000 7fc62291c700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc61c072990 con 0x7fc61c15e0b0 2023-10-16T12:30:12.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.505+0000 7fc62291c700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc61c06de50 con 0x7fc61c15a3d0 2023-10-16T12:30:12.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.506+0000 7fc620eb9700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc61c0cf430 0x7fc61c0cdeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:12.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.506+0000 7fc620eb9700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc61c0cf430 0x7fc61c0cdeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44878/0 (socket says 172.21.15.16:44878) 2023-10-16T12:30:12.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.506+0000 7fc620eb9700 1 -- 172.21.15.16:0/150428905 learned_addr learned my addr 172.21.15.16:0/150428905 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:12.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.506+0000 7fc61b7fe700 1 --2- 172.21.15.16:0/150428905 >> v2:172.21.15.49:3300/0 conn(0x7fc61c15e0b0 0x7fc61c0ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:12.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.506+0000 7fc61bfff700 1 --2- 172.21.15.16:0/150428905 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc61c15a3d0 0x7fc61c06d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:12.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.506+0000 7fc620eb9700 1 -- 172.21.15.16:0/150428905 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc61c15a3d0 msgr2=0x7fc61c06d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:12.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.506+0000 7fc620eb9700 1 --2- 172.21.15.16:0/150428905 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc61c15a3d0 0x7fc61c06d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:12.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.506+0000 7fc620eb9700 1 -- 172.21.15.16:0/150428905 >> v2:172.21.15.49:3300/0 conn(0x7fc61c15e0b0 msgr2=0x7fc61c0ceef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:12.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.506+0000 7fc620eb9700 1 --2- 172.21.15.16:0/150428905 >> v2:172.21.15.49:3300/0 conn(0x7fc61c15e0b0 0x7fc61c0ceef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:12.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.506+0000 7fc620eb9700 1 -- 172.21.15.16:0/150428905 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc61c0cb870 con 0x7fc61c0cf430 2023-10-16T12:30:12.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.506+0000 7fc620eb9700 1 --2- 172.21.15.16:0/150428905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc61c0cf430 0x7fc61c0cdeb0 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7fc61c0cde60 tx=0x7fc61400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aca23090e5befb2f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:12.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.506+0000 7fc61affd700 1 -- 172.21.15.16:0/150428905 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc61400e070 con 0x7fc61c0cf430 2023-10-16T12:30:12.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.506+0000 7fc61affd700 1 -- 172.21.15.16:0/150428905 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc61400b9c0 con 0x7fc61c0cf430 2023-10-16T12:30:12.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.506+0000 7fc61affd700 1 -- 172.21.15.16:0/150428905 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc61400bd70 con 0x7fc61c0cf430 2023-10-16T12:30:12.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.507+0000 7fc62291c700 1 -- 172.21.15.16:0/150428905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc61c0cf430 msgr2=0x7fc61c0cdeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:12.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.507+0000 7fc62291c700 1 --2- 172.21.15.16:0/150428905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc61c0cf430 0x7fc61c0cdeb0 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7fc61c0cde60 tx=0x7fc61400b330 comp rx=0 tx=0).stop 2023-10-16T12:30:12.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.507+0000 7fc62291c700 1 -- 172.21.15.16:0/150428905 shutdown_connections 2023-10-16T12:30:12.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.507+0000 7fc62291c700 1 --2- 172.21.15.16:0/150428905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc61c0cf430 0x7fc61c0cdeb0 unknown :-1 s=CLOSED pgs=1482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:12.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.507+0000 7fc62291c700 1 --2- 172.21.15.16:0/150428905 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc61c15a3d0 0x7fc61c06d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:12.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.507+0000 7fc62291c700 1 --2- 172.21.15.16:0/150428905 >> v2:172.21.15.49:3300/0 conn(0x7fc61c15e0b0 0x7fc61c0ceef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:12.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.507+0000 7fc62291c700 1 -- 172.21.15.16:0/150428905 >> 172.21.15.16:0/150428905 conn(0x7fc61c06f430 msgr2=0x7fc61c06f810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:12.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.507+0000 7fc62291c700 1 -- 172.21.15.16:0/150428905 shutdown_connections 2023-10-16T12:30:12.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.507+0000 7fc62291c700 1 -- 172.21.15.16:0/150428905 wait complete. 2023-10-16T12:30:12.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.508+0000 7fc62291c700 1 Processor -- start 2023-10-16T12:30:12.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.508+0000 7fc62291c700 1 -- start start 2023-10-16T12:30:12.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.509+0000 7fc62291c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc61c0cf430 0x7fc61c0c73b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:12.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.509+0000 7fc62291c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc61c15a3d0 0x7fc61c15d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:12.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.509+0000 7fc62291c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc61c15e0b0 0x7fc61c1f6580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:12.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.509+0000 7fc62291c700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc61c0c7a50 con 0x7fc61c15a3d0 2023-10-16T12:30:12.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.509+0000 7fc62291c700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc61c0ae700 con 0x7fc61c15e0b0 2023-10-16T12:30:12.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.509+0000 7fc62291c700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc61c0af660 con 0x7fc61c0cf430 2023-10-16T12:30:12.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.509+0000 7fc61b7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc61c15a3d0 0x7fc61c15d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:12.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.509+0000 7fc61b7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc61c15a3d0 0x7fc61c15d270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44882/0 (socket says 172.21.15.16:44882) 2023-10-16T12:30:12.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.509+0000 7fc61b7fe700 1 -- 172.21.15.16:0/970543292 learned_addr learned my addr 172.21.15.16:0/970543292 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:12.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.509+0000 7fc61bfff700 1 --2- 172.21.15.16:0/970543292 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc61c0cf430 0x7fc61c0c73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:12.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.509+0000 7fc620eb9700 1 --2- 172.21.15.16:0/970543292 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc61c15e0b0 0x7fc61c1f6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:12.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.510+0000 7fc61b7fe700 1 -- 172.21.15.16:0/970543292 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc61c0cf430 msgr2=0x7fc61c0c73b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:12.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.510+0000 7fc61b7fe700 1 --2- 172.21.15.16:0/970543292 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc61c0cf430 0x7fc61c0c73b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:12.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.510+0000 7fc61b7fe700 1 -- 172.21.15.16:0/970543292 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc61c15e0b0 msgr2=0x7fc61c1f6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:12.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.510+0000 7fc61b7fe700 1 --2- 172.21.15.16:0/970543292 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc61c15e0b0 0x7fc61c1f6580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:12.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.510+0000 7fc61b7fe700 1 -- 172.21.15.16:0/970543292 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc614009b30 con 0x7fc61c15a3d0 2023-10-16T12:30:12.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.510+0000 7fc61b7fe700 1 --2- 172.21.15.16:0/970543292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc61c15a3d0 0x7fc61c15d270 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7fc61000d830 tx=0x7fc61000db40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:12.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.510+0000 7fc618ff9700 1 -- 172.21.15.16:0/970543292 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6100125f0 con 0x7fc61c15a3d0 2023-10-16T12:30:12.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.510+0000 7fc618ff9700 1 -- 172.21.15.16:0/970543292 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc610012750 con 0x7fc61c15a3d0 2023-10-16T12:30:12.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.510+0000 7fc618ff9700 1 -- 172.21.15.16:0/970543292 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc610012ae0 con 0x7fc61c15a3d0 2023-10-16T12:30:12.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.510+0000 7fc62291c700 1 -- 172.21.15.16:0/970543292 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc61c0af0a0 con 0x7fc61c15a3d0 2023-10-16T12:30:12.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.510+0000 7fc62291c700 1 -- 172.21.15.16:0/970543292 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc61c0aeb20 con 0x7fc61c15a3d0 2023-10-16T12:30:12.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.511+0000 7fc62291c700 1 -- 172.21.15.16:0/970543292 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc61c0aed40 con 0x7fc61c15a3d0 2023-10-16T12:30:12.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.512+0000 7fc618ff9700 1 -- 172.21.15.16:0/970543292 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 73) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fc610012c40 con 0x7fc61c15a3d0 2023-10-16T12:30:12.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.512+0000 7fc618ff9700 1 --2- 172.21.15.16:0/970543292 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7fc60806af90 0x7fc60806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:12.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.512+0000 7fc618ff9700 1 -- 172.21.15.16:0/970543292 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 6507+0+0 (secure 0 0 0) 0x7fc6100897e0 con 0x7fc61c15a3d0 2023-10-16T12:30:12.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.512+0000 7fc61bfff700 1 --2- 172.21.15.16:0/970543292 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7fc60806af90 0x7fc60806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:12.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.512+0000 7fc61bfff700 1 --2- 172.21.15.16:0/970543292 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7fc60806af90 0x7fc60806d450 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc60c006b20 tx=0x7fc60c008040 comp rx=0 tx=0).ready entity=mgr.8388 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:12.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.517+0000 7fc618ff9700 1 -- 172.21.15.16:0/970543292 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc61c0aed40 con 0x7fc61c15a3d0 2023-10-16T12:30:12.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:12.720+0000 7fc62291c700 1 -- 172.21.15.16:0/970543292 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc61c0aed40 con 0x7fc61c15a3d0 2023-10-16T12:30:13.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.352+0000 7fc618ff9700 1 -- 172.21.15.16:0/970543292 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v134) v1 ==== 169+0+0 (secure 0 0 0) 0x7fc61c0aed40 con 0x7fc61c15a3d0 2023-10-16T12:30:13.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.360+0000 7fc618ff9700 1 -- 172.21.15.16:0/970543292 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mgrmap(e 74) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc6100549e0 con 0x7fc61c15a3d0 2023-10-16T12:30:13.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.360+0000 7fc62291c700 1 -- 172.21.15.16:0/970543292 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7fc60806af90 msgr2=0x7fc60806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:13.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.360+0000 7fc62291c700 1 --2- 172.21.15.16:0/970543292 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7fc60806af90 0x7fc60806d450 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc60c006b20 tx=0x7fc60c008040 comp rx=0 tx=0).stop 2023-10-16T12:30:13.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.360+0000 7fc62291c700 1 -- 172.21.15.16:0/970543292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc61c15a3d0 msgr2=0x7fc61c15d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:13.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.360+0000 7fc62291c700 1 --2- 172.21.15.16:0/970543292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc61c15a3d0 0x7fc61c15d270 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7fc61000d830 tx=0x7fc61000db40 comp rx=0 tx=0).stop 2023-10-16T12:30:13.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.361+0000 7fc62291c700 1 -- 172.21.15.16:0/970543292 shutdown_connections 2023-10-16T12:30:13.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.361+0000 7fc62291c700 1 --2- 172.21.15.16:0/970543292 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7fc60806af90 0x7fc60806d450 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:13.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.361+0000 7fc62291c700 1 --2- 172.21.15.16:0/970543292 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc61c15e0b0 0x7fc61c1f6580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:13.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.361+0000 7fc62291c700 1 --2- 172.21.15.16:0/970543292 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc61c0cf430 0x7fc61c0c73b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:13.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.361+0000 7fc62291c700 1 --2- 172.21.15.16:0/970543292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc61c15a3d0 0x7fc61c15d270 unknown :-1 s=CLOSED pgs=1483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:13.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.361+0000 7fc62291c700 1 -- 172.21.15.16:0/970543292 >> 172.21.15.16:0/970543292 conn(0x7fc61c06f430 msgr2=0x7fc61c06a7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:13.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.361+0000 7fc62291c700 1 -- 172.21.15.16:0/970543292 shutdown_connections 2023-10-16T12:30:13.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.361+0000 7fc62291c700 1 -- 172.21.15.16:0/970543292 wait complete. 2023-10-16T12:30:13.365 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' removed 2023-10-16T12:30:13.380 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-16T12:30:13.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.520+0000 7f381306f700 1 Processor -- start 2023-10-16T12:30:13.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.521+0000 7f381306f700 1 -- start start 2023-10-16T12:30:13.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.522+0000 7f381306f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f380c153d50 0x7f380c06dd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:13.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.522+0000 7f381306f700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f380c15af40 con 0x7f380c06e2d0 2023-10-16T12:30:13.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.522+0000 7f381306f700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f380c15b190 con 0x7f380c071920 2023-10-16T12:30:13.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.522+0000 7f381306f700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f380c06e780 con 0x7f380c153d50 2023-10-16T12:30:13.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.522+0000 7f380bfff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f380c06e2d0 0x7f380c071210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34162/0 (socket says 172.21.15.16:34162) 2023-10-16T12:30:13.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.522+0000 7f380bfff700 1 -- 172.21.15.16:0/4171838354 learned_addr learned my addr 172.21.15.16:0/4171838354 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:30:13.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.523+0000 7f3810e0b700 1 --2- 172.21.15.16:0/4171838354 >> v2:172.21.15.16:3300/0 conn(0x7f380c153d50 0x7f380c06dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:13.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.523+0000 7f3810e0b700 1 -- 172.21.15.16:0/4171838354 >> v1:172.21.15.49:6789/0 conn(0x7f380c071920 legacy=0x7f380c15bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:13.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.523+0000 7f3810e0b700 1 -- 172.21.15.16:0/4171838354 >> v1:172.21.15.16:6789/0 conn(0x7f380c06e2d0 legacy=0x7f380c071210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:13.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.523+0000 7f3810e0b700 1 -- 172.21.15.16:0/4171838354 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f380c15b470 con 0x7f380c153d50 2023-10-16T12:30:13.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.524+0000 7f3810e0b700 1 --2- 172.21.15.16:0/4171838354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f380c153d50 0x7f380c06dd90 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7f38000091b0 tx=0x7f38000094c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26f9af7288f4f9e0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:13.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.524+0000 7f380b7fe700 1 -- 172.21.15.16:0/4171838354 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3800004230 con 0x7f380c153d50 2023-10-16T12:30:13.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.524+0000 7f380b7fe700 1 -- 172.21.15.16:0/4171838354 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3800004390 con 0x7f380c153d50 2023-10-16T12:30:13.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.524+0000 7f380b7fe700 1 -- 172.21.15.16:0/4171838354 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f380000f740 con 0x7f380c153d50 2023-10-16T12:30:13.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.525+0000 7f381306f700 1 -- 172.21.15.16:0/4171838354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f380c153d50 msgr2=0x7f380c06dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:13.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.525+0000 7f381306f700 1 --2- 172.21.15.16:0/4171838354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f380c153d50 0x7f380c06dd90 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7f38000091b0 tx=0x7f38000094c0 comp rx=0 tx=0).stop 2023-10-16T12:30:13.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.525+0000 7f381306f700 1 -- 172.21.15.16:0/4171838354 shutdown_connections 2023-10-16T12:30:13.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.525+0000 7f381306f700 1 --2- 172.21.15.16:0/4171838354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f380c153d50 0x7f380c06dd90 secure :-1 s=CLOSED pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7f38000091b0 tx=0x7f38000094c0 comp rx=0 tx=0).stop 2023-10-16T12:30:13.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.525+0000 7f381306f700 1 -- 172.21.15.16:0/4171838354 >> 172.21.15.16:0/4171838354 conn(0x7f380c06c980 msgr2=0x7f380c06cd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:13.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.526+0000 7f381306f700 1 -- 172.21.15.16:0/4171838354 shutdown_connections 2023-10-16T12:30:13.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.526+0000 7f381306f700 1 -- 172.21.15.16:0/4171838354 wait complete. 2023-10-16T12:30:13.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.528+0000 7f381306f700 1 Processor -- start 2023-10-16T12:30:13.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.528+0000 7f381306f700 1 -- start start 2023-10-16T12:30:13.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.529+0000 7f381306f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f380c06e2d0 0x7f380c0cdbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:13.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.529+0000 7f381306f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f380c071920 0x7f380c156060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:13.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.529+0000 7f381306f700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f380c159040 0x7f380c1565a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:13.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.529+0000 7f381306f700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f380c158c20 con 0x7f380c071920 2023-10-16T12:30:13.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.529+0000 7f381306f700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f380c157fe0 con 0x7f380c159040 2023-10-16T12:30:13.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.529+0000 7f381306f700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f380c158280 con 0x7f380c06e2d0 2023-10-16T12:30:13.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.529+0000 7f380bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f380c071920 0x7f380c156060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:13.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.530+0000 7f380bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f380c071920 0x7f380c156060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44896/0 (socket says 172.21.15.16:44896) 2023-10-16T12:30:13.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.530+0000 7f380bfff700 1 -- 172.21.15.16:0/613508708 learned_addr learned my addr 172.21.15.16:0/613508708 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:13.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.530+0000 7f3810e0b700 1 --2- 172.21.15.16:0/613508708 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f380c06e2d0 0x7f380c0cdbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:13.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.530+0000 7f381160c700 1 --2- 172.21.15.16:0/613508708 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f380c159040 0x7f380c1565a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:13.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.530+0000 7f380bfff700 1 -- 172.21.15.16:0/613508708 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f380c06e2d0 msgr2=0x7f380c0cdbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:13.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.530+0000 7f380bfff700 1 --2- 172.21.15.16:0/613508708 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f380c06e2d0 0x7f380c0cdbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:13.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.530+0000 7f380bfff700 1 -- 172.21.15.16:0/613508708 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f380c159040 msgr2=0x7f380c1565a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:13.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.530+0000 7f380bfff700 1 --2- 172.21.15.16:0/613508708 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f380c159040 0x7f380c1565a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:13.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.530+0000 7f380bfff700 1 -- 172.21.15.16:0/613508708 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3800009d30 con 0x7f380c071920 2023-10-16T12:30:13.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.530+0000 7f380bfff700 1 --2- 172.21.15.16:0/613508708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f380c071920 0x7f380c156060 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f37fc00c520 tx=0x7f37fc00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:13.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.530+0000 7f38097fa700 1 -- 172.21.15.16:0/613508708 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37fc005d70 con 0x7f380c071920 2023-10-16T12:30:13.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.530+0000 7f38097fa700 1 -- 172.21.15.16:0/613508708 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37fc0070d0 con 0x7f380c071920 2023-10-16T12:30:13.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.530+0000 7f381306f700 1 -- 172.21.15.16:0/613508708 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f380c158560 con 0x7f380c071920 2023-10-16T12:30:13.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.530+0000 7f38097fa700 1 -- 172.21.15.16:0/613508708 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37fc007440 con 0x7f380c071920 2023-10-16T12:30:13.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.530+0000 7f381306f700 1 -- 172.21.15.16:0/613508708 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f380c0ce0f0 con 0x7f380c071920 2023-10-16T12:30:13.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.531+0000 7f381306f700 1 -- 172.21.15.16:0/613508708 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f380c158d60 con 0x7f380c071920 2023-10-16T12:30:13.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.532+0000 7f38097fa700 1 -- 172.21.15.16:0/613508708 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 74) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f37fc0075a0 con 0x7f380c071920 2023-10-16T12:30:13.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.532+0000 7f38097fa700 1 --2- 172.21.15.16:0/613508708 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f37f806b020 0x7f37f806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:13.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.532+0000 7f38097fa700 1 -- 172.21.15.16:0/613508708 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 6093+0+0 (secure 0 0 0) 0x7f37fc088b50 con 0x7f380c071920 2023-10-16T12:30:13.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.532+0000 7f3810e0b700 1 --2- 172.21.15.16:0/613508708 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f37f806b020 0x7f37f806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:13.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.533+0000 7f3810e0b700 1 --2- 172.21.15.16:0/613508708 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f37f806b020 0x7f37f806d4e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f380c0b1e60 tx=0x7f3800008030 comp rx=0 tx=0).ready entity=mgr.8388 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:13.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.537+0000 7f38097fa700 1 -- 172.21.15.16:0/613508708 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f380c158d60 con 0x7f380c071920 2023-10-16T12:30:13.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.741+0000 7f381306f700 1 -- 172.21.15.16:0/613508708 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f380c158d60 con 0x7f380c071920 2023-10-16T12:30:13.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.745+0000 7f38097fa700 1 -- 172.21.15.16:0/613508708 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v43) v1 ==== 115+0+0 (secure 0 0 0) 0x7f380c158d60 con 0x7f380c071920 2023-10-16T12:30:13.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.747+0000 7f381306f700 1 -- 172.21.15.16:0/613508708 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f37f806b020 msgr2=0x7f37f806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:13.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.747+0000 7f381306f700 1 --2- 172.21.15.16:0/613508708 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f37f806b020 0x7f37f806d4e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f380c0b1e60 tx=0x7f3800008030 comp rx=0 tx=0).stop 2023-10-16T12:30:13.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.747+0000 7f381306f700 1 -- 172.21.15.16:0/613508708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f380c071920 msgr2=0x7f380c156060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:13.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.747+0000 7f381306f700 1 --2- 172.21.15.16:0/613508708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f380c071920 0x7f380c156060 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f37fc00c520 tx=0x7f37fc00c830 comp rx=0 tx=0).stop 2023-10-16T12:30:13.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.747+0000 7f381306f700 1 -- 172.21.15.16:0/613508708 shutdown_connections 2023-10-16T12:30:13.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.747+0000 7f381306f700 1 --2- 172.21.15.16:0/613508708 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f37f806b020 0x7f37f806d4e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:13.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.747+0000 7f381306f700 1 --2- 172.21.15.16:0/613508708 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f380c159040 0x7f380c1565a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:13.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.747+0000 7f381306f700 1 --2- 172.21.15.16:0/613508708 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f380c06e2d0 0x7f380c0cdbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:13.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.747+0000 7f381306f700 1 --2- 172.21.15.16:0/613508708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f380c071920 0x7f380c156060 unknown :-1 s=CLOSED pgs=1486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:13.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.747+0000 7f381306f700 1 -- 172.21.15.16:0/613508708 >> 172.21.15.16:0/613508708 conn(0x7f380c06c980 msgr2=0x7f380c0b1ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:13.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.747+0000 7f381306f700 1 -- 172.21.15.16:0/613508708 shutdown_connections 2023-10-16T12:30:13.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.747+0000 7f381306f700 1 -- 172.21.15.16:0/613508708 wait complete. 2023-10-16T12:30:13.762 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-16T12:30:13.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.888+0000 7efc12ae9700 1 Processor -- start 2023-10-16T12:30:13.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.888+0000 7efc12ae9700 1 -- start start 2023-10-16T12:30:13.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.888+0000 7efc12ae9700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7efc0c15b2c0 0x7efc0c161340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:13.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.888+0000 7efc12ae9700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7efc0c06e570 0x7efc0c15e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:13.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.888+0000 7efc12ae9700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc0c161880 con 0x7efc0c06a7e0 2023-10-16T12:30:13.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.888+0000 7efc12ae9700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc0c15b6a0 con 0x7efc0c06e570 2023-10-16T12:30:13.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.888+0000 7efc12ae9700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc0c161b20 con 0x7efc0c15b2c0 2023-10-16T12:30:13.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.889+0000 7efc0bfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7efc0c06e570 0x7efc0c15e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:13.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.889+0000 7efc0bfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7efc0c06e570 0x7efc0c15e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44902/0 (socket says 172.21.15.16:44902) 2023-10-16T12:30:13.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.889+0000 7efc0bfff700 1 -- 172.21.15.16:0/577288993 learned_addr learned my addr 172.21.15.16:0/577288993 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:13.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.889+0000 7efc10885700 1 --2- 172.21.15.16:0/577288993 >> v2:172.21.15.49:3300/0 conn(0x7efc0c15b2c0 0x7efc0c161340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:13.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.889+0000 7efc0bfff700 1 -- 172.21.15.16:0/577288993 >> v2:172.21.15.49:3300/0 conn(0x7efc0c15b2c0 msgr2=0x7efc0c161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:13.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.889+0000 7efc0bfff700 1 --2- 172.21.15.16:0/577288993 >> v2:172.21.15.49:3300/0 conn(0x7efc0c15b2c0 0x7efc0c161340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:13.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.889+0000 7efc0bfff700 1 -- 172.21.15.16:0/577288993 >> v1:172.21.15.49:6789/0 conn(0x7efc0c06a7e0 legacy=0x7efc0c071a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:13.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.889+0000 7efc0bfff700 1 -- 172.21.15.16:0/577288993 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc0c161e00 con 0x7efc0c06e570 2023-10-16T12:30:13.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.889+0000 7efc10885700 1 --2- 172.21.15.16:0/577288993 >> v2:172.21.15.49:3300/0 conn(0x7efc0c15b2c0 0x7efc0c161340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:30:13.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.889+0000 7efc0bfff700 1 --2- 172.21.15.16:0/577288993 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc0c06e570 0x7efc0c15e5a0 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7efc0c06b4a0 tx=0x7efbf8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e79bba46235969f1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:13.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.889+0000 7efc0b7fe700 1 -- 172.21.15.16:0/577288993 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf800b870 con 0x7efc0c06e570 2023-10-16T12:30:13.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.889+0000 7efc0b7fe700 1 -- 172.21.15.16:0/577288993 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbf800b9d0 con 0x7efc0c06e570 2023-10-16T12:30:13.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.889+0000 7efc0b7fe700 1 -- 172.21.15.16:0/577288993 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf800bd60 con 0x7efc0c06e570 2023-10-16T12:30:13.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.890+0000 7efc12ae9700 1 -- 172.21.15.16:0/577288993 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc0c06e570 msgr2=0x7efc0c15e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:13.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.890+0000 7efc12ae9700 1 --2- 172.21.15.16:0/577288993 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc0c06e570 0x7efc0c15e5a0 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7efc0c06b4a0 tx=0x7efbf8009bd0 comp rx=0 tx=0).stop 2023-10-16T12:30:13.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.890+0000 7efc12ae9700 1 -- 172.21.15.16:0/577288993 shutdown_connections 2023-10-16T12:30:13.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.890+0000 7efc12ae9700 1 --2- 172.21.15.16:0/577288993 >> v2:172.21.15.49:3300/0 conn(0x7efc0c15b2c0 0x7efc0c161340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:13.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.890+0000 7efc12ae9700 1 --2- 172.21.15.16:0/577288993 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc0c06e570 0x7efc0c15e5a0 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:13.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.890+0000 7efc12ae9700 1 -- 172.21.15.16:0/577288993 >> 172.21.15.16:0/577288993 conn(0x7efc0c0b1f40 msgr2=0x7efc0c0b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:13.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.890+0000 7efc12ae9700 1 -- 172.21.15.16:0/577288993 shutdown_connections 2023-10-16T12:30:13.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.890+0000 7efc12ae9700 1 -- 172.21.15.16:0/577288993 wait complete. 2023-10-16T12:30:13.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.891+0000 7efc12ae9700 1 Processor -- start 2023-10-16T12:30:13.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.892+0000 7efc12ae9700 1 -- start start 2023-10-16T12:30:13.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.893+0000 7efc12ae9700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efc0c06a7e0 0x7efc0c14c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:13.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.893+0000 7efc12ae9700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efc0c06e570 0x7efc0c0b5320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:13.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.893+0000 7efc12ae9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc0c15b2c0 0x7efc0c0c74c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:13.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.893+0000 7efc12ae9700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc0c14cc00 con 0x7efc0c15b2c0 2023-10-16T12:30:13.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.893+0000 7efc12ae9700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc0c14cee0 con 0x7efc0c06e570 2023-10-16T12:30:13.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.893+0000 7efc12ae9700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc0c14d1c0 con 0x7efc0c06a7e0 2023-10-16T12:30:13.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.893+0000 7efc11086700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc0c15b2c0 0x7efc0c0c74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:13.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.893+0000 7efc11086700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc0c15b2c0 0x7efc0c0c74c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44910/0 (socket says 172.21.15.16:44910) 2023-10-16T12:30:13.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.893+0000 7efc11086700 1 -- 172.21.15.16:0/2242403794 learned_addr learned my addr 172.21.15.16:0/2242403794 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:13.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.893+0000 7efc10885700 1 --2- 172.21.15.16:0/2242403794 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efc0c06a7e0 0x7efc0c14c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:13.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.893+0000 7efc0bfff700 1 --2- 172.21.15.16:0/2242403794 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efc0c06e570 0x7efc0c0b5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:13.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.893+0000 7efc11086700 1 -- 172.21.15.16:0/2242403794 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efc0c06a7e0 msgr2=0x7efc0c14c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:13.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.893+0000 7efc11086700 1 --2- 172.21.15.16:0/2242403794 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efc0c06a7e0 0x7efc0c14c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:13.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.893+0000 7efc11086700 1 -- 172.21.15.16:0/2242403794 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efc0c06e570 msgr2=0x7efc0c0b5320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:13.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.893+0000 7efc11086700 1 --2- 172.21.15.16:0/2242403794 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efc0c06e570 0x7efc0c0b5320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:13.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.893+0000 7efc11086700 1 -- 172.21.15.16:0/2242403794 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbf8009550 con 0x7efc0c15b2c0 2023-10-16T12:30:13.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.893+0000 7efc11086700 1 --2- 172.21.15.16:0/2242403794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc0c15b2c0 0x7efc0c0c74c0 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7efc0400cb50 tx=0x7efc0400cf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:13.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.893+0000 7efc097fa700 1 -- 172.21.15.16:0/2242403794 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc04003c00 con 0x7efc0c15b2c0 2023-10-16T12:30:13.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.893+0000 7efc097fa700 1 -- 172.21.15.16:0/2242403794 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc04003d60 con 0x7efc0c15b2c0 2023-10-16T12:30:13.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.893+0000 7efc12ae9700 1 -- 172.21.15.16:0/2242403794 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc0c14d4a0 con 0x7efc0c15b2c0 2023-10-16T12:30:13.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.893+0000 7efc097fa700 1 -- 172.21.15.16:0/2242403794 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc0400ab70 con 0x7efc0c15b2c0 2023-10-16T12:30:13.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.894+0000 7efc12ae9700 1 -- 172.21.15.16:0/2242403794 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc0c0b5860 con 0x7efc0c15b2c0 2023-10-16T12:30:13.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.895+0000 7efc12ae9700 1 -- 172.21.15.16:0/2242403794 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc0c14d9c0 con 0x7efc0c15b2c0 2023-10-16T12:30:13.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.895+0000 7efc097fa700 1 -- 172.21.15.16:0/2242403794 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 74) v1 ==== 89331+0+0 (secure 0 0 0) 0x7efc0400acd0 con 0x7efc0c15b2c0 2023-10-16T12:30:13.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.895+0000 7efc097fa700 1 --2- 172.21.15.16:0/2242403794 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7efbf006b020 0x7efbf006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:13.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.895+0000 7efc097fa700 1 -- 172.21.15.16:0/2242403794 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 6093+0+0 (secure 0 0 0) 0x7efc040898b0 con 0x7efc0c15b2c0 2023-10-16T12:30:13.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.895+0000 7efc10885700 1 --2- 172.21.15.16:0/2242403794 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7efbf006b020 0x7efbf006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:13.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.896+0000 7efc10885700 1 --2- 172.21.15.16:0/2242403794 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7efbf006b020 0x7efbf006d4e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7efc00009990 tx=0x7efc00008040 comp rx=0 tx=0).ready entity=mgr.8388 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:13.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:13.900+0000 7efc097fa700 1 -- 172.21.15.16:0/2242403794 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7efc0c14d9c0 con 0x7efc0c15b2c0 2023-10-16T12:30:14.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.110+0000 7efc12ae9700 1 -- 172.21.15.16:0/2242403794 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7efc0c14d9c0 con 0x7efc0c15b2c0 2023-10-16T12:30:14.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.113+0000 7efc097fa700 1 -- 172.21.15.16:0/2242403794 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v44) v1 ==== 104+0+0 (secure 0 0 0) 0x7efc0c14d9c0 con 0x7efc0c15b2c0 2023-10-16T12:30:14.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.115+0000 7efc12ae9700 1 -- 172.21.15.16:0/2242403794 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7efbf006b020 msgr2=0x7efbf006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:14.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.115+0000 7efc12ae9700 1 --2- 172.21.15.16:0/2242403794 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7efbf006b020 0x7efbf006d4e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7efc00009990 tx=0x7efc00008040 comp rx=0 tx=0).stop 2023-10-16T12:30:14.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.115+0000 7efc12ae9700 1 -- 172.21.15.16:0/2242403794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc0c15b2c0 msgr2=0x7efc0c0c74c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:14.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.115+0000 7efc12ae9700 1 --2- 172.21.15.16:0/2242403794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc0c15b2c0 0x7efc0c0c74c0 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7efc0400cb50 tx=0x7efc0400cf10 comp rx=0 tx=0).stop 2023-10-16T12:30:14.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.115+0000 7efc12ae9700 1 -- 172.21.15.16:0/2242403794 shutdown_connections 2023-10-16T12:30:14.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.115+0000 7efc12ae9700 1 --2- 172.21.15.16:0/2242403794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc0c15b2c0 0x7efc0c0c74c0 unknown :-1 s=CLOSED pgs=1488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:14.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.115+0000 7efc12ae9700 1 --2- 172.21.15.16:0/2242403794 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efc0c06a7e0 0x7efc0c14c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:14.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.115+0000 7efc12ae9700 1 --2- 172.21.15.16:0/2242403794 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7efbf006b020 0x7efbf006d4e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:14.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.115+0000 7efc12ae9700 1 --2- 172.21.15.16:0/2242403794 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efc0c06e570 0x7efc0c0b5320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:14.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.115+0000 7efc12ae9700 1 -- 172.21.15.16:0/2242403794 >> 172.21.15.16:0/2242403794 conn(0x7efc0c0b1f40 msgr2=0x7efc0c0cce00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:14.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.115+0000 7efc12ae9700 1 -- 172.21.15.16:0/2242403794 shutdown_connections 2023-10-16T12:30:14.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.115+0000 7efc12ae9700 1 -- 172.21.15.16:0/2242403794 wait complete. 2023-10-16T12:30:14.129 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-16T12:30:14.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.255+0000 7f6e6da3b700 1 Processor -- start 2023-10-16T12:30:14.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.255+0000 7f6e6da3b700 1 -- start start 2023-10-16T12:30:14.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.256+0000 7f6e6da3b700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f6e6815b790 0x7f6e680cda60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:14.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.256+0000 7f6e6da3b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6e6806e070 0x7f6e680ce850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:14.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.256+0000 7f6e6da3b700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e680ce0e0 con 0x7f6e6815bb70 2023-10-16T12:30:14.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.256+0000 7f6e6da3b700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e6806a7e0 con 0x7f6e6806e070 2023-10-16T12:30:14.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.256+0000 7f6e6da3b700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e6815c410 con 0x7f6e6815b790 2023-10-16T12:30:14.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.256+0000 7f6e677fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6e6806e070 0x7f6e680ce850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:14.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.256+0000 7f6e667fc700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f6e6815bb70 0x7f6e6806d960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34188/0 (socket says 172.21.15.16:34188) 2023-10-16T12:30:14.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.256+0000 7f6e677fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6e6806e070 0x7f6e680ce850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44918/0 (socket says 172.21.15.16:44918) 2023-10-16T12:30:14.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.256+0000 7f6e667fc700 1 -- 172.21.15.16:0/2788884960 learned_addr learned my addr 172.21.15.16:0/2788884960 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:30:14.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.257+0000 7f6e66ffd700 1 --2- 172.21.15.16:0/2788884960 >> v2:172.21.15.49:3300/0 conn(0x7f6e6815b790 0x7f6e680cda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:14.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.257+0000 7f6e677fe700 1 -- 172.21.15.16:0/2788884960 >> v2:172.21.15.49:3300/0 conn(0x7f6e6815b790 msgr2=0x7f6e680cda60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:14.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.257+0000 7f6e677fe700 1 --2- 172.21.15.16:0/2788884960 >> v2:172.21.15.49:3300/0 conn(0x7f6e6815b790 0x7f6e680cda60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:14.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.257+0000 7f6e677fe700 1 -- 172.21.15.16:0/2788884960 >> v1:172.21.15.16:6789/0 conn(0x7f6e6815bb70 legacy=0x7f6e6806d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:14.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.257+0000 7f6e677fe700 1 -- 172.21.15.16:0/2788884960 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e6815c6f0 con 0x7f6e6806e070 2023-10-16T12:30:14.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.257+0000 7f6e65ffb700 1 -- 172.21.15.16:0/2788884960 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2924880567 0 0) 0x7f6e680ce0e0 con 0x7f6e6815bb70 2023-10-16T12:30:14.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.257+0000 7f6e677fe700 1 --2- 172.21.15.16:0/2788884960 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e6806e070 0x7f6e680ce850 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7f6e68072220 tx=0x7f6e6000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ff9fe850b8400ac server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:14.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.257+0000 7f6e65ffb700 1 -- 172.21.15.16:0/2788884960 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e6000e070 con 0x7f6e6806e070 2023-10-16T12:30:14.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.257+0000 7f6e65ffb700 1 -- 172.21.15.16:0/2788884960 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e6000bbd0 con 0x7f6e6806e070 2023-10-16T12:30:14.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.257+0000 7f6e65ffb700 1 -- 172.21.15.16:0/2788884960 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e60012430 con 0x7f6e6806e070 2023-10-16T12:30:14.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.257+0000 7f6e6da3b700 1 -- 172.21.15.16:0/2788884960 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e6806e070 msgr2=0x7f6e680ce850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:14.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.257+0000 7f6e6da3b700 1 --2- 172.21.15.16:0/2788884960 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e6806e070 0x7f6e680ce850 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7f6e68072220 tx=0x7f6e6000b330 comp rx=0 tx=0).stop 2023-10-16T12:30:14.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.258+0000 7f6e6da3b700 1 -- 172.21.15.16:0/2788884960 shutdown_connections 2023-10-16T12:30:14.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.258+0000 7f6e6da3b700 1 --2- 172.21.15.16:0/2788884960 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e6806e070 0x7f6e680ce850 unknown :-1 s=CLOSED pgs=1490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:14.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.258+0000 7f6e6da3b700 1 --2- 172.21.15.16:0/2788884960 >> v2:172.21.15.49:3300/0 conn(0x7f6e6815b790 0x7f6e680cda60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:14.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.258+0000 7f6e6da3b700 1 -- 172.21.15.16:0/2788884960 >> 172.21.15.16:0/2788884960 conn(0x7f6e680cc5e0 msgr2=0x7f6e680cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:14.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.258+0000 7f6e6da3b700 1 -- 172.21.15.16:0/2788884960 shutdown_connections 2023-10-16T12:30:14.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.258+0000 7f6e6da3b700 1 -- 172.21.15.16:0/2788884960 wait complete. 2023-10-16T12:30:14.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.259+0000 7f6e6da3b700 1 Processor -- start 2023-10-16T12:30:14.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.259+0000 7f6e6da3b700 1 -- start start 2023-10-16T12:30:14.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.260+0000 7f6e6da3b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e6806e070 0x7f6e680b1070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:14.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.260+0000 7f6e6da3b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6e6815b790 0x7f6e68154a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:14.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.260+0000 7f6e6da3b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6e6815bb70 0x7f6e6815f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:14.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.260+0000 7f6e6da3b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e68158730 con 0x7f6e6806e070 2023-10-16T12:30:14.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.260+0000 7f6e6da3b700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e68158170 con 0x7f6e6815b790 2023-10-16T12:30:14.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.260+0000 7f6e6da3b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6815cc10 con 0x7f6e6815bb70 2023-10-16T12:30:14.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.260+0000 7f6e66ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e6806e070 0x7f6e680b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:14.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.260+0000 7f6e66ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e6806e070 0x7f6e680b1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44920/0 (socket says 172.21.15.16:44920) 2023-10-16T12:30:14.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.260+0000 7f6e66ffd700 1 -- 172.21.15.16:0/3695475281 learned_addr learned my addr 172.21.15.16:0/3695475281 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:14.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.260+0000 7f6e667fc700 1 --2- 172.21.15.16:0/3695475281 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6e6815b790 0x7f6e68154a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:14.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.260+0000 7f6e677fe700 1 --2- 172.21.15.16:0/3695475281 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6e6815bb70 0x7f6e6815f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:14.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.260+0000 7f6e66ffd700 1 -- 172.21.15.16:0/3695475281 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6e6815bb70 msgr2=0x7f6e6815f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:14.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.260+0000 7f6e66ffd700 1 --2- 172.21.15.16:0/3695475281 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6e6815bb70 0x7f6e6815f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:14.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.260+0000 7f6e66ffd700 1 -- 172.21.15.16:0/3695475281 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6e6815b790 msgr2=0x7f6e68154a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:14.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.260+0000 7f6e66ffd700 1 --2- 172.21.15.16:0/3695475281 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6e6815b790 0x7f6e68154a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:14.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.260+0000 7f6e66ffd700 1 -- 172.21.15.16:0/3695475281 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e60009a70 con 0x7f6e6806e070 2023-10-16T12:30:14.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.260+0000 7f6e667fc700 1 --2- 172.21.15.16:0/3695475281 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6e6815b790 0x7f6e68154a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:30:14.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.260+0000 7f6e66ffd700 1 --2- 172.21.15.16:0/3695475281 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e6806e070 0x7f6e680b1070 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7f6e5000e810 tx=0x7f6e5000ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:14.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.260+0000 7f6e5ffff700 1 -- 172.21.15.16:0/3695475281 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e50013610 con 0x7f6e6806e070 2023-10-16T12:30:14.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.261+0000 7f6e5ffff700 1 -- 172.21.15.16:0/3695475281 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e50013770 con 0x7f6e6806e070 2023-10-16T12:30:14.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.261+0000 7f6e6da3b700 1 -- 172.21.15.16:0/3695475281 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e6815cef0 con 0x7f6e6806e070 2023-10-16T12:30:14.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.261+0000 7f6e5ffff700 1 -- 172.21.15.16:0/3695475281 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e50013b20 con 0x7f6e6806e070 2023-10-16T12:30:14.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.261+0000 7f6e6da3b700 1 -- 172.21.15.16:0/3695475281 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e68152c90 con 0x7f6e6806e070 2023-10-16T12:30:14.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.261+0000 7f6e6da3b700 1 -- 172.21.15.16:0/3695475281 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e6804ebb0 con 0x7f6e6806e070 2023-10-16T12:30:14.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.262+0000 7f6e5ffff700 1 -- 172.21.15.16:0/3695475281 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 74) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6e50013c80 con 0x7f6e6806e070 2023-10-16T12:30:14.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.263+0000 7f6e5ffff700 1 --2- 172.21.15.16:0/3695475281 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f6e4806b020 0x7f6e4806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:14.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.263+0000 7f6e667fc700 1 --2- 172.21.15.16:0/3695475281 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f6e4806b020 0x7f6e4806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:14.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.263+0000 7f6e5ffff700 1 -- 172.21.15.16:0/3695475281 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 6093+0+0 (secure 0 0 0) 0x7f6e500896e0 con 0x7f6e6806e070 2023-10-16T12:30:14.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.263+0000 7f6e667fc700 1 --2- 172.21.15.16:0/3695475281 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f6e4806b020 0x7f6e4806d4e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f6e68066670 tx=0x7f6e58007d30 comp rx=0 tx=0).ready entity=mgr.8388 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:14.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.267+0000 7f6e5ffff700 1 -- 172.21.15.16:0/3695475281 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6e50054360 con 0x7f6e6806e070 2023-10-16T12:30:14.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.473+0000 7f6e6da3b700 1 -- 172.21.15.16:0/3695475281 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f6e6815d0e0 con 0x7f6e6806e070 2023-10-16T12:30:14.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.476+0000 7f6e5ffff700 1 -- 172.21.15.16:0/3695475281 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v45) v1 ==== 120+0+0 (secure 0 0 0) 0x7f6e50057980 con 0x7f6e6806e070 2023-10-16T12:30:14.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.478+0000 7f6e6da3b700 1 -- 172.21.15.16:0/3695475281 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f6e4806b020 msgr2=0x7f6e4806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:14.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.478+0000 7f6e6da3b700 1 --2- 172.21.15.16:0/3695475281 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f6e4806b020 0x7f6e4806d4e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f6e68066670 tx=0x7f6e58007d30 comp rx=0 tx=0).stop 2023-10-16T12:30:14.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.478+0000 7f6e6da3b700 1 -- 172.21.15.16:0/3695475281 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e6806e070 msgr2=0x7f6e680b1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:14.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.478+0000 7f6e6da3b700 1 --2- 172.21.15.16:0/3695475281 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e6806e070 0x7f6e680b1070 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7f6e5000e810 tx=0x7f6e5000ebd0 comp rx=0 tx=0).stop 2023-10-16T12:30:14.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.478+0000 7f6e6da3b700 1 -- 172.21.15.16:0/3695475281 shutdown_connections 2023-10-16T12:30:14.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.478+0000 7f6e6da3b700 1 --2- 172.21.15.16:0/3695475281 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6e6815bb70 0x7f6e6815f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:14.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.478+0000 7f6e6da3b700 1 --2- 172.21.15.16:0/3695475281 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e6806e070 0x7f6e680b1070 unknown :-1 s=CLOSED pgs=1491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:14.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.478+0000 7f6e6da3b700 1 --2- 172.21.15.16:0/3695475281 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f6e4806b020 0x7f6e4806d4e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:14.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.478+0000 7f6e6da3b700 1 --2- 172.21.15.16:0/3695475281 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6e6815b790 0x7f6e68154a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:14.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.478+0000 7f6e6da3b700 1 -- 172.21.15.16:0/3695475281 >> 172.21.15.16:0/3695475281 conn(0x7f6e680cc5e0 msgr2=0x7f6e6806ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:14.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.478+0000 7f6e6da3b700 1 -- 172.21.15.16:0/3695475281 shutdown_connections 2023-10-16T12:30:14.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.478+0000 7f6e6da3b700 1 -- 172.21.15.16:0/3695475281 wait complete. 2023-10-16T12:30:14.493 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule' 2023-10-16T12:30:14.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.617+0000 7f91b94f8700 1 Processor -- start 2023-10-16T12:30:14.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.617+0000 7f91b94f8700 1 -- start start 2023-10-16T12:30:14.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.618+0000 7f91b94f8700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f91b415d8d0 0x7f91b40d1c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:14.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.618+0000 7f91b94f8700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f91b40d0750 0x7f91b406c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:14.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.618+0000 7f91b94f8700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91b406ba00 con 0x7f91b40d41c0 2023-10-16T12:30:14.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.618+0000 7f91b94f8700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91b40d2150 con 0x7f91b415d8d0 2023-10-16T12:30:14.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.618+0000 7f91b94f8700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91b40d26d0 con 0x7f91b40d0750 2023-10-16T12:30:14.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.618+0000 7f91b2ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f91b415d8d0 0x7f91b40d1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:14.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.618+0000 7f91b2ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f91b415d8d0 0x7f91b40d1c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44928/0 (socket says 172.21.15.16:44928) 2023-10-16T12:30:14.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.618+0000 7f91b27fc700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f91b40d41c0 0x7f91b40d0040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34202/0 (socket says 172.21.15.16:34202) 2023-10-16T12:30:14.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.618+0000 7f91b2ffd700 1 -- 172.21.15.16:0/3090033619 learned_addr learned my addr 172.21.15.16:0/3090033619 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:14.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.618+0000 7f91b37fe700 1 --2- 172.21.15.16:0/3090033619 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f91b40d0750 0x7f91b406c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:14.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.618+0000 7f91b2ffd700 1 -- 172.21.15.16:0/3090033619 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f91b40d0750 msgr2=0x7f91b406c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:14.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.618+0000 7f91b2ffd700 1 --2- 172.21.15.16:0/3090033619 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f91b40d0750 0x7f91b406c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:14.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.618+0000 7f91b2ffd700 1 -- 172.21.15.16:0/3090033619 >> v1:172.21.15.16:6789/0 conn(0x7f91b40d41c0 legacy=0x7f91b40d0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:14.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.618+0000 7f91b2ffd700 1 -- 172.21.15.16:0/3090033619 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91b40d23f0 con 0x7f91b415d8d0 2023-10-16T12:30:14.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.618+0000 7f91b1ffb700 1 -- 172.21.15.16:0/3090033619 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3383635504 0 0) 0x7f91b406ba00 con 0x7f91b40d41c0 2023-10-16T12:30:14.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.619+0000 7f91b2ffd700 1 --2- 172.21.15.16:0/3090033619 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91b415d8d0 0x7f91b40d1c10 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7f91a80098f0 tx=0x7f91a8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c860e7768277e9c8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:14.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.619+0000 7f91b1ffb700 1 -- 172.21.15.16:0/3090033619 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91a800b8a0 con 0x7f91b415d8d0 2023-10-16T12:30:14.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.619+0000 7f91b1ffb700 1 -- 172.21.15.16:0/3090033619 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91a800ba00 con 0x7f91b415d8d0 2023-10-16T12:30:14.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.619+0000 7f91b1ffb700 1 -- 172.21.15.16:0/3090033619 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91a800bd90 con 0x7f91b415d8d0 2023-10-16T12:30:14.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.619+0000 7f91b94f8700 1 -- 172.21.15.16:0/3090033619 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91b415d8d0 msgr2=0x7f91b40d1c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:14.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.619+0000 7f91b94f8700 1 --2- 172.21.15.16:0/3090033619 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91b415d8d0 0x7f91b40d1c10 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7f91a80098f0 tx=0x7f91a8009c00 comp rx=0 tx=0).stop 2023-10-16T12:30:14.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.619+0000 7f91b94f8700 1 -- 172.21.15.16:0/3090033619 shutdown_connections 2023-10-16T12:30:14.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.619+0000 7f91b94f8700 1 --2- 172.21.15.16:0/3090033619 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f91b40d0750 0x7f91b406c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:14.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.619+0000 7f91b94f8700 1 --2- 172.21.15.16:0/3090033619 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91b415d8d0 0x7f91b40d1c10 unknown :-1 s=CLOSED pgs=1493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:14.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.619+0000 7f91b94f8700 1 -- 172.21.15.16:0/3090033619 >> 172.21.15.16:0/3090033619 conn(0x7f91b40d3630 msgr2=0x7f91b40d3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:14.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.619+0000 7f91b94f8700 1 -- 172.21.15.16:0/3090033619 shutdown_connections 2023-10-16T12:30:14.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.619+0000 7f91b94f8700 1 -- 172.21.15.16:0/3090033619 wait complete. 2023-10-16T12:30:14.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.621+0000 7f91b94f8700 1 Processor -- start 2023-10-16T12:30:14.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.621+0000 7f91b94f8700 1 -- start start 2023-10-16T12:30:14.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.621+0000 7f91b94f8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91b40d0750 0x7f91b40cb410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:14.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.621+0000 7f91b94f8700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f91b40d41c0 0x7f91b41e43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:14.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.621+0000 7f91b94f8700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f91b415d8d0 0x7f91b4159bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:14.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.621+0000 7f91b94f8700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91b41e4920 con 0x7f91b40d0750 2023-10-16T12:30:14.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.621+0000 7f91b94f8700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91b41e3fb0 con 0x7f91b40d41c0 2023-10-16T12:30:14.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.621+0000 7f91b94f8700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91b41e39f0 con 0x7f91b415d8d0 2023-10-16T12:30:14.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.621+0000 7f91b2ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91b40d0750 0x7f91b40cb410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:14.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.621+0000 7f91b2ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91b40d0750 0x7f91b40cb410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44934/0 (socket says 172.21.15.16:44934) 2023-10-16T12:30:14.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.621+0000 7f91b2ffd700 1 -- 172.21.15.16:0/4270791560 learned_addr learned my addr 172.21.15.16:0/4270791560 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:14.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.622+0000 7f91b2ffd700 1 -- 172.21.15.16:0/4270791560 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f91b415d8d0 msgr2=0x7f91b4159bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:14.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.622+0000 7f91b2ffd700 1 --2- 172.21.15.16:0/4270791560 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f91b415d8d0 0x7f91b4159bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:14.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.622+0000 7f91b2ffd700 1 -- 172.21.15.16:0/4270791560 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f91b40d41c0 msgr2=0x7f91b41e43e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:14.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.622+0000 7f91b2ffd700 1 --2- 172.21.15.16:0/4270791560 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f91b40d41c0 0x7f91b41e43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:14.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.622+0000 7f91b2ffd700 1 -- 172.21.15.16:0/4270791560 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91a8009550 con 0x7f91b40d0750 2023-10-16T12:30:14.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.622+0000 7f91b2ffd700 1 --2- 172.21.15.16:0/4270791560 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91b40d0750 0x7f91b40cb410 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7f91a8009c40 tx=0x7f91a8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:14.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.622+0000 7f919b7fe700 1 -- 172.21.15.16:0/4270791560 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91a80036a0 con 0x7f91b40d0750 2023-10-16T12:30:14.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.622+0000 7f919b7fe700 1 -- 172.21.15.16:0/4270791560 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91a801e070 con 0x7f91b40d0750 2023-10-16T12:30:14.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.622+0000 7f91b94f8700 1 -- 172.21.15.16:0/4270791560 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91b41e3cd0 con 0x7f91b40d0750 2023-10-16T12:30:14.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.622+0000 7f919b7fe700 1 -- 172.21.15.16:0/4270791560 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91a8003a20 con 0x7f91b40d0750 2023-10-16T12:30:14.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.622+0000 7f91b94f8700 1 -- 172.21.15.16:0/4270791560 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91b40c8610 con 0x7f91b40d0750 2023-10-16T12:30:14.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.623+0000 7f91b94f8700 1 -- 172.21.15.16:0/4270791560 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91b40c8190 con 0x7f91b40d0750 2023-10-16T12:30:14.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.624+0000 7f919b7fe700 1 -- 172.21.15.16:0/4270791560 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 74) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f91a8003b80 con 0x7f91b40d0750 2023-10-16T12:30:14.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.624+0000 7f919b7fe700 1 --2- 172.21.15.16:0/4270791560 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f91a006b020 0x7f91a006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:14.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.624+0000 7f919b7fe700 1 -- 172.21.15.16:0/4270791560 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 6093+0+0 (secure 0 0 0) 0x7f91a8020070 con 0x7f91b40d0750 2023-10-16T12:30:14.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.624+0000 7f91b27fc700 1 --2- 172.21.15.16:0/4270791560 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f91a006b020 0x7f91a006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:14.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.625+0000 7f91b27fc700 1 --2- 172.21.15.16:0/4270791560 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f91a006b020 0x7f91a006d4e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f91a40011e0 tx=0x7f91a40079d0 comp rx=0 tx=0).ready entity=mgr.8388 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:14.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.628+0000 7f919b7fe700 1 -- 172.21.15.16:0/4270791560 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f91a8055fe0 con 0x7f91b40d0750 2023-10-16T12:30:14.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:14.915+0000 7f91b94f8700 1 -- 172.21.15.16:0/4270791560 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]} v 0) v1 -- 0x7f91b40c8190 con 0x7f91b40d0750 2023-10-16T12:30:15.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.354+0000 7f919b7fe700 1 -- 172.21.15.16:0/4270791560 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]}]=0 v747) v1 ==== 148+0+0 (secure 0 0 0) 0x7f91b40c8190 con 0x7f91b40d0750 2023-10-16T12:30:15.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.357+0000 7f91b94f8700 1 -- 172.21.15.16:0/4270791560 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f91a006b020 msgr2=0x7f91a006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:15.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.357+0000 7f91b94f8700 1 --2- 172.21.15.16:0/4270791560 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f91a006b020 0x7f91a006d4e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f91a40011e0 tx=0x7f91a40079d0 comp rx=0 tx=0).stop 2023-10-16T12:30:15.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.357+0000 7f91b94f8700 1 -- 172.21.15.16:0/4270791560 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91b40d0750 msgr2=0x7f91b40cb410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:15.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.357+0000 7f91b94f8700 1 --2- 172.21.15.16:0/4270791560 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91b40d0750 0x7f91b40cb410 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7f91a8009c40 tx=0x7f91a8004750 comp rx=0 tx=0).stop 2023-10-16T12:30:15.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.357+0000 7f91b94f8700 1 -- 172.21.15.16:0/4270791560 shutdown_connections 2023-10-16T12:30:15.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.357+0000 7f91b94f8700 1 --2- 172.21.15.16:0/4270791560 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f91b415d8d0 0x7f91b4159bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:15.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.357+0000 7f91b94f8700 1 --2- 172.21.15.16:0/4270791560 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91b40d0750 0x7f91b40cb410 unknown :-1 s=CLOSED pgs=1494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:15.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.357+0000 7f91b94f8700 1 --2- 172.21.15.16:0/4270791560 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f91a006b020 0x7f91a006d4e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:15.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.357+0000 7f91b94f8700 1 --2- 172.21.15.16:0/4270791560 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f91b40d41c0 0x7f91b41e43e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:15.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.357+0000 7f91b94f8700 1 -- 172.21.15.16:0/4270791560 >> 172.21.15.16:0/4270791560 conn(0x7f91b40d3630 msgr2=0x7f91b415c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:15.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.357+0000 7f91b94f8700 1 -- 172.21.15.16:0/4270791560 shutdown_connections 2023-10-16T12:30:15.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.357+0000 7f91b94f8700 1 -- 172.21.15.16:0/4270791560 wait complete. 2023-10-16T12:30:15.374 INFO:tasks.cephfs_test_runner:Test existence of a scheduled snapshot ... ok 2023-10-16T12:30:15.374 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_all_periods (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-16T12:30:15.375 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:30:15.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.507+0000 7f70c4533700 1 Processor -- start 2023-10-16T12:30:15.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.507+0000 7f70c4533700 1 -- start start 2023-10-16T12:30:15.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.508+0000 7f70c4533700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f70bc15a530 0x7f70bc0b32d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:15.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.508+0000 7f70c4533700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f70bc0cf940 0x7f70bc0cbf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:15.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.508+0000 7f70c4533700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70bc0cba90 con 0x7f70bc072560 2023-10-16T12:30:15.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.508+0000 7f70c4533700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70bc0cd230 con 0x7f70bc15a530 2023-10-16T12:30:15.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.508+0000 7f70c4533700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70bc0cd510 con 0x7f70bc0cf940 2023-10-16T12:30:15.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.508+0000 7f70c22cf700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f70bc15a530 0x7f70bc0b32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:15.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.508+0000 7f70c22cf700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f70bc15a530 0x7f70bc0b32d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44942/0 (socket says 172.21.15.16:44942) 2023-10-16T12:30:15.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.508+0000 7f70c22cf700 1 -- 172.21.15.16:0/2531867929 learned_addr learned my addr 172.21.15.16:0/2531867929 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:15.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.508+0000 7f70c2ad0700 1 --2- 172.21.15.16:0/2531867929 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f70bc0cf940 0x7f70bc0cbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:15.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.508+0000 7f70c22cf700 1 -- 172.21.15.16:0/2531867929 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f70bc0cf940 msgr2=0x7f70bc0cbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:15.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.508+0000 7f70c22cf700 1 --2- 172.21.15.16:0/2531867929 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f70bc0cf940 0x7f70bc0cbf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:15.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.508+0000 7f70c22cf700 1 -- 172.21.15.16:0/2531867929 >> v1:172.21.15.49:6789/0 conn(0x7f70bc072560 legacy=0x7f70bc0cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:15.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.508+0000 7f70c22cf700 1 -- 172.21.15.16:0/2531867929 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70bc0cd7f0 con 0x7f70bc15a530 2023-10-16T12:30:15.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.508+0000 7f70c2ad0700 1 --2- 172.21.15.16:0/2531867929 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f70bc0cf940 0x7f70bc0cbf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:30:15.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.508+0000 7f70c22cf700 1 --2- 172.21.15.16:0/2531867929 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70bc15a530 0x7f70bc0b32d0 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f70ac0098f0 tx=0x7f70ac009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ffaea2dc45eb4d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:15.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.509+0000 7f70c12cd700 1 -- 172.21.15.16:0/2531867929 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70ac00b8a0 con 0x7f70bc15a530 2023-10-16T12:30:15.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.509+0000 7f70c12cd700 1 -- 172.21.15.16:0/2531867929 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70ac00ba00 con 0x7f70bc15a530 2023-10-16T12:30:15.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.509+0000 7f70c12cd700 1 -- 172.21.15.16:0/2531867929 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70ac00bd90 con 0x7f70bc15a530 2023-10-16T12:30:15.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.509+0000 7f70c4533700 1 -- 172.21.15.16:0/2531867929 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70bc15a530 msgr2=0x7f70bc0b32d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:15.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.509+0000 7f70c4533700 1 --2- 172.21.15.16:0/2531867929 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70bc15a530 0x7f70bc0b32d0 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f70ac0098f0 tx=0x7f70ac009c00 comp rx=0 tx=0).stop 2023-10-16T12:30:15.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.509+0000 7f70c4533700 1 -- 172.21.15.16:0/2531867929 shutdown_connections 2023-10-16T12:30:15.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.509+0000 7f70c4533700 1 --2- 172.21.15.16:0/2531867929 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f70bc0cf940 0x7f70bc0cbf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:15.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.509+0000 7f70c4533700 1 --2- 172.21.15.16:0/2531867929 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70bc15a530 0x7f70bc0b32d0 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:15.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.509+0000 7f70c4533700 1 -- 172.21.15.16:0/2531867929 >> 172.21.15.16:0/2531867929 conn(0x7f70bc0b1fd0 msgr2=0x7f70bc0b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:15.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.509+0000 7f70c4533700 1 -- 172.21.15.16:0/2531867929 shutdown_connections 2023-10-16T12:30:15.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.509+0000 7f70c4533700 1 -- 172.21.15.16:0/2531867929 wait complete. 2023-10-16T12:30:15.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.511+0000 7f70c4533700 1 Processor -- start 2023-10-16T12:30:15.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.511+0000 7f70c4533700 1 -- start start 2023-10-16T12:30:15.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.512+0000 7f70c4533700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70bc072560 0x7f70bc15d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:15.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.512+0000 7f70c4533700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f70bc0cf940 0x7f70bc0c7ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:15.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.512+0000 7f70c4533700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f70bc15a530 0x7f70bc14f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:15.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.512+0000 7f70c4533700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70bc0c94f0 con 0x7f70bc072560 2023-10-16T12:30:15.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.512+0000 7f70c4533700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70bc0c88e0 con 0x7f70bc0cf940 2023-10-16T12:30:15.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.512+0000 7f70c4533700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70bc0c8bc0 con 0x7f70bc15a530 2023-10-16T12:30:15.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.512+0000 7f70c22cf700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70bc072560 0x7f70bc15d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:15.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.512+0000 7f70c22cf700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70bc072560 0x7f70bc15d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44948/0 (socket says 172.21.15.16:44948) 2023-10-16T12:30:15.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.512+0000 7f70c22cf700 1 -- 172.21.15.16:0/3646701748 learned_addr learned my addr 172.21.15.16:0/3646701748 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:15.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.512+0000 7f70c2ad0700 1 --2- 172.21.15.16:0/3646701748 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f70bc15a530 0x7f70bc14f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:15.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.512+0000 7f70c1ace700 1 --2- 172.21.15.16:0/3646701748 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f70bc0cf940 0x7f70bc0c7ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:15.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.512+0000 7f70c22cf700 1 -- 172.21.15.16:0/3646701748 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f70bc15a530 msgr2=0x7f70bc14f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:15.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.512+0000 7f70c22cf700 1 --2- 172.21.15.16:0/3646701748 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f70bc15a530 0x7f70bc14f210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:15.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.512+0000 7f70c22cf700 1 -- 172.21.15.16:0/3646701748 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f70bc0cf940 msgr2=0x7f70bc0c7ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:15.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.512+0000 7f70c22cf700 1 --2- 172.21.15.16:0/3646701748 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f70bc0cf940 0x7f70bc0c7ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:15.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.512+0000 7f70c22cf700 1 -- 172.21.15.16:0/3646701748 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70ac009550 con 0x7f70bc072560 2023-10-16T12:30:15.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.512+0000 7f70c2ad0700 1 --2- 172.21.15.16:0/3646701748 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f70bc15a530 0x7f70bc14f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:30:15.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.512+0000 7f70c22cf700 1 --2- 172.21.15.16:0/3646701748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70bc072560 0x7f70bc15d090 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f70ac009c40 tx=0x7f70ac004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:15.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.513+0000 7f70b2ffd700 1 -- 172.21.15.16:0/3646701748 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70ac0036a0 con 0x7f70bc072560 2023-10-16T12:30:15.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.513+0000 7f70b2ffd700 1 -- 172.21.15.16:0/3646701748 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70ac01e070 con 0x7f70bc072560 2023-10-16T12:30:15.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.513+0000 7f70c4533700 1 -- 172.21.15.16:0/3646701748 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70bc0c8ea0 con 0x7f70bc072560 2023-10-16T12:30:15.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.513+0000 7f70b2ffd700 1 -- 172.21.15.16:0/3646701748 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70ac003a60 con 0x7f70bc072560 2023-10-16T12:30:15.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.513+0000 7f70c4533700 1 -- 172.21.15.16:0/3646701748 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70bc15d5d0 con 0x7f70bc072560 2023-10-16T12:30:15.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.514+0000 7f70c4533700 1 -- 172.21.15.16:0/3646701748 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70bc04ebb0 con 0x7f70bc072560 2023-10-16T12:30:15.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.514+0000 7f70b2ffd700 1 -- 172.21.15.16:0/3646701748 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 74) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f70ac003bc0 con 0x7f70bc072560 2023-10-16T12:30:15.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.514+0000 7f70b2ffd700 1 --2- 172.21.15.16:0/3646701748 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f70a006b020 0x7f70a006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:15.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.515+0000 7f70b2ffd700 1 -- 172.21.15.16:0/3646701748 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 6093+0+0 (secure 0 0 0) 0x7f70ac020070 con 0x7f70bc072560 2023-10-16T12:30:15.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.515+0000 7f70c1ace700 1 --2- 172.21.15.16:0/3646701748 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f70a006b020 0x7f70a006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:15.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.515+0000 7f70c1ace700 1 --2- 172.21.15.16:0/3646701748 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f70a006b020 0x7f70a006d4e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f70bc0b1b20 tx=0x7f70b8009040 comp rx=0 tx=0).ready entity=mgr.8388 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:15.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.519+0000 7f70b2ffd700 1 -- 172.21.15.16:0/3646701748 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f70ac055c10 con 0x7f70bc072560 2023-10-16T12:30:15.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.725+0000 7f70c4533700 1 -- 172.21.15.16:0/3646701748 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f70bc0687c0 con 0x7f70bc072560 2023-10-16T12:30:15.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.726+0000 7f70b2ffd700 1 -- 172.21.15.16:0/3646701748 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v134) v1 ==== 74+0+14304 (secure 0 0 0) 0x7f70ac059230 con 0x7f70bc072560 2023-10-16T12:30:15.728 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:30:15.728 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":134,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:30:13.347253+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6837/3203451346":"2023-10-17T12:30:09.964825+0000","172.21.15.16:6835/3203451346":"2023-10-17T12:30:09.964825+0000","172.21.15.16:0/2093940709":"2023-10-17T12:30:08.896838+0000","172.21.15.16:0/3924602056":"2023-10-17T12:30:08.896838+0000","172.21.15.16:0/3368698219":"2023-10-17T12:30:08.896838+0000","172.21.15.16:0/10052963":"2023-10-17T12:30:08.896838+0000","172.21.15.16:0/667000197":"2023-10-17T12:30:08.896838+0000","172.21.15.16:0/1018845767":"2023-10-17T12:30:08.896838+0000","172.21.15.16:0/3485123903":"2023-10-17T12:30:08.896838+0000","172.21.15.16:6832/2550537276":"2023-10-17T12:30:08.896838+0000","172.21.15.16:0/32319707":"2023-10-17T12:29:30.732445+0000","172.21.15.16:0/1058620826":"2023-10-17T12:29:30.732445+0000","172.21.15.16:0/3098064509":"2023-10-17T12:29:30.732445+0000","172.21.15.16:0/3814896496":"2023-10-17T12:29:30.732445+0000","172.21.15.16:0/2973963882":"2023-10-17T12:29:30.732445+0000","172.21.15.16:6833/2550537276":"2023-10-17T12:30:08.896838+0000","172.21.15.16:6833/2309613638":"2023-10-17T12:29:30.732445+0000","172.21.15.16:0/4081830650":"2023-10-17T12:30:08.896838+0000","172.21.15.16:6832/2309613638":"2023-10-17T12:29:30.732445+0000","172.21.15.16:0/1074148914":"2023-10-17T12:29:30.732445+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:30:15.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.728+0000 7f70c4533700 1 -- 172.21.15.16:0/3646701748 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f70a006b020 msgr2=0x7f70a006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:15.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.728+0000 7f70c4533700 1 --2- 172.21.15.16:0/3646701748 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f70a006b020 0x7f70a006d4e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f70bc0b1b20 tx=0x7f70b8009040 comp rx=0 tx=0).stop 2023-10-16T12:30:15.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.728+0000 7f70c4533700 1 -- 172.21.15.16:0/3646701748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70bc072560 msgr2=0x7f70bc15d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:15.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.728+0000 7f70c4533700 1 --2- 172.21.15.16:0/3646701748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70bc072560 0x7f70bc15d090 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f70ac009c40 tx=0x7f70ac004750 comp rx=0 tx=0).stop 2023-10-16T12:30:15.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.728+0000 7f70c4533700 1 -- 172.21.15.16:0/3646701748 shutdown_connections 2023-10-16T12:30:15.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.728+0000 7f70c4533700 1 --2- 172.21.15.16:0/3646701748 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f70bc15a530 0x7f70bc14f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:15.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.728+0000 7f70c4533700 1 --2- 172.21.15.16:0/3646701748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70bc072560 0x7f70bc15d090 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:15.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.728+0000 7f70c4533700 1 --2- 172.21.15.16:0/3646701748 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f70a006b020 0x7f70a006d4e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:15.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.728+0000 7f70c4533700 1 --2- 172.21.15.16:0/3646701748 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f70bc0cf940 0x7f70bc0c7ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:15.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.728+0000 7f70c4533700 1 -- 172.21.15.16:0/3646701748 >> 172.21.15.16:0/3646701748 conn(0x7f70bc0b1fd0 msgr2=0x7f70bc0cf140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:15.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.728+0000 7f70c4533700 1 -- 172.21.15.16:0/3646701748 shutdown_connections 2023-10-16T12:30:15.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.728+0000 7f70c4533700 1 -- 172.21.15.16:0/3646701748 wait complete. 2023-10-16T12:30:15.742 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T12:30:15.743 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:30:15.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.869+0000 7f1877e17700 1 Processor -- start 2023-10-16T12:30:15.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.870+0000 7f1877e17700 1 -- start start 2023-10-16T12:30:15.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.870+0000 7f1877e17700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f187006aef0 0x7f18700ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:15.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.870+0000 7f1877e17700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18701613c0 con 0x7f1870160fe0 2023-10-16T12:30:15.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.870+0000 7f1877e17700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18700adb00 con 0x7f187006da30 2023-10-16T12:30:15.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.871+0000 7f1877e17700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18700cf680 con 0x7f187006aef0 2023-10-16T12:30:15.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.871+0000 7f18753b2700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f187006aef0 0x7f18700ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:15.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.871+0000 7f1875bb3700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f1870160fe0 0x7f187006a7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34230/0 (socket says 172.21.15.16:34230) 2023-10-16T12:30:15.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.871+0000 7f18753b2700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f187006aef0 0x7f18700ac8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44958/0 (socket says 172.21.15.16:44958) 2023-10-16T12:30:15.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.871+0000 7f1875bb3700 1 -- 172.21.15.16:0/1152817823 learned_addr learned my addr 172.21.15.16:0/1152817823 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:30:15.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.871+0000 7f18753b2700 1 -- 172.21.15.16:0/1152817823 >> v1:172.21.15.49:6789/0 conn(0x7f187006da30 legacy=0x7f18700adf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:15.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.871+0000 7f18753b2700 1 -- 172.21.15.16:0/1152817823 >> v1:172.21.15.16:6789/0 conn(0x7f1870160fe0 legacy=0x7f187006a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:15.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.871+0000 7f18753b2700 1 -- 172.21.15.16:0/1152817823 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18700acef0 con 0x7f187006aef0 2023-10-16T12:30:15.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.871+0000 7f18753b2700 1 --2- 172.21.15.16:0/1152817823 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f187006aef0 0x7f18700ac8d0 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f1864009840 tx=0x7f1864009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2d47540696caac5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:15.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.871+0000 7f1874bb1700 1 -- 172.21.15.16:0/1152817823 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1864004230 con 0x7f187006aef0 2023-10-16T12:30:15.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.871+0000 7f1874bb1700 1 -- 172.21.15.16:0/1152817823 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1864004390 con 0x7f187006aef0 2023-10-16T12:30:15.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.871+0000 7f1874bb1700 1 -- 172.21.15.16:0/1152817823 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18640036a0 con 0x7f187006aef0 2023-10-16T12:30:15.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.872+0000 7f1877e17700 1 -- 172.21.15.16:0/1152817823 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f187006aef0 msgr2=0x7f18700ac8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:15.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.872+0000 7f1877e17700 1 --2- 172.21.15.16:0/1152817823 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f187006aef0 0x7f18700ac8d0 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f1864009840 tx=0x7f1864009b50 comp rx=0 tx=0).stop 2023-10-16T12:30:15.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.872+0000 7f1877e17700 1 -- 172.21.15.16:0/1152817823 shutdown_connections 2023-10-16T12:30:15.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.872+0000 7f1877e17700 1 --2- 172.21.15.16:0/1152817823 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f187006aef0 0x7f18700ac8d0 unknown :-1 s=CLOSED pgs=1498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:15.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.872+0000 7f1877e17700 1 -- 172.21.15.16:0/1152817823 >> 172.21.15.16:0/1152817823 conn(0x7f18700b1ed0 msgr2=0x7f18700b22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:15.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.872+0000 7f1877e17700 1 -- 172.21.15.16:0/1152817823 shutdown_connections 2023-10-16T12:30:15.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.872+0000 7f1877e17700 1 -- 172.21.15.16:0/1152817823 wait complete. 2023-10-16T12:30:15.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.873+0000 7f1877e17700 1 Processor -- start 2023-10-16T12:30:15.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.873+0000 7f1877e17700 1 -- start start 2023-10-16T12:30:15.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.874+0000 7f1877e17700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f187006aef0 0x7f18700c5c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:15.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.874+0000 7f1877e17700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f187006da30 0x7f1870153170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:15.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.874+0000 7f1877e17700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1870160fe0 0x7f18700b3730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:15.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.874+0000 7f1877e17700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18700b2fc0 con 0x7f1870160fe0 2023-10-16T12:30:15.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.874+0000 7f1877e17700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18700c6180 con 0x7f187006da30 2023-10-16T12:30:15.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.874+0000 7f1877e17700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18700c6400 con 0x7f187006aef0 2023-10-16T12:30:15.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.874+0000 7f18763b4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1870160fe0 0x7f18700b3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:15.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.874+0000 7f18763b4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1870160fe0 0x7f18700b3730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44966/0 (socket says 172.21.15.16:44966) 2023-10-16T12:30:15.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.874+0000 7f18763b4700 1 -- 172.21.15.16:0/51622433 learned_addr learned my addr 172.21.15.16:0/51622433 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:15.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.874+0000 7f1875bb3700 1 --2- 172.21.15.16:0/51622433 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f187006aef0 0x7f18700c5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:15.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.874+0000 7f18753b2700 1 --2- 172.21.15.16:0/51622433 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f187006da30 0x7f1870153170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:15.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.874+0000 7f18763b4700 1 -- 172.21.15.16:0/51622433 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f187006aef0 msgr2=0x7f18700c5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:15.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.874+0000 7f18763b4700 1 --2- 172.21.15.16:0/51622433 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f187006aef0 0x7f18700c5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:15.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.874+0000 7f18763b4700 1 -- 172.21.15.16:0/51622433 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f187006da30 msgr2=0x7f1870153170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:15.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.874+0000 7f18763b4700 1 --2- 172.21.15.16:0/51622433 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f187006da30 0x7f1870153170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:15.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.874+0000 7f18763b4700 1 -- 172.21.15.16:0/51622433 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18640094a0 con 0x7f1870160fe0 2023-10-16T12:30:15.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.874+0000 7f18763b4700 1 --2- 172.21.15.16:0/51622433 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1870160fe0 0x7f18700b3730 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f186800cb60 tx=0x7f186800ce70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:15.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.875+0000 7f18627fc700 1 -- 172.21.15.16:0/51622433 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f186800a9a0 con 0x7f1870160fe0 2023-10-16T12:30:15.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.875+0000 7f18627fc700 1 -- 172.21.15.16:0/51622433 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f186800ab00 con 0x7f1870160fe0 2023-10-16T12:30:15.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.875+0000 7f1877e17700 1 -- 172.21.15.16:0/51622433 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18700c66e0 con 0x7f1870160fe0 2023-10-16T12:30:15.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.875+0000 7f18627fc700 1 -- 172.21.15.16:0/51622433 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1868012420 con 0x7f1870160fe0 2023-10-16T12:30:15.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.875+0000 7f1877e17700 1 -- 172.21.15.16:0/51622433 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18700b3210 con 0x7f1870160fe0 2023-10-16T12:30:15.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.876+0000 7f1877e17700 1 -- 172.21.15.16:0/51622433 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f187004ebb0 con 0x7f1870160fe0 2023-10-16T12:30:15.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.876+0000 7f18627fc700 1 -- 172.21.15.16:0/51622433 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 74) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1868012580 con 0x7f1870160fe0 2023-10-16T12:30:15.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.876+0000 7f18627fc700 1 --2- 172.21.15.16:0/51622433 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f185406b020 0x7f185406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:15.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.877+0000 7f18627fc700 1 -- 172.21.15.16:0/51622433 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 6093+0+0 (secure 0 0 0) 0x7f1868089eb0 con 0x7f1870160fe0 2023-10-16T12:30:15.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.877+0000 7f1875bb3700 1 --2- 172.21.15.16:0/51622433 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f185406b020 0x7f185406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:15.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.877+0000 7f1875bb3700 1 --2- 172.21.15.16:0/51622433 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f185406b020 0x7f185406d4e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f186c0010e0 tx=0x7f186c007d90 comp rx=0 tx=0).ready entity=mgr.8388 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:15.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:15.881+0000 7f18627fc700 1 -- 172.21.15.16:0/51622433 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1868054d60 con 0x7f1870160fe0 2023-10-16T12:30:16.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:16.084+0000 7f1877e17700 1 -- 172.21.15.16:0/51622433 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f18700aed20 con 0x7f1870160fe0 2023-10-16T12:30:16.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:16.085+0000 7f18627fc700 1 -- 172.21.15.16:0/51622433 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v134) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1868016020 con 0x7f1870160fe0 2023-10-16T12:30:16.087 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T12:30:16.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:16.087+0000 7f1877e17700 1 -- 172.21.15.16:0/51622433 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f185406b020 msgr2=0x7f185406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:16.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:16.087+0000 7f1877e17700 1 --2- 172.21.15.16:0/51622433 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f185406b020 0x7f185406d4e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f186c0010e0 tx=0x7f186c007d90 comp rx=0 tx=0).stop 2023-10-16T12:30:16.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:16.087+0000 7f1877e17700 1 -- 172.21.15.16:0/51622433 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1870160fe0 msgr2=0x7f18700b3730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:16.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:16.087+0000 7f1877e17700 1 --2- 172.21.15.16:0/51622433 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1870160fe0 0x7f18700b3730 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f186800cb60 tx=0x7f186800ce70 comp rx=0 tx=0).stop 2023-10-16T12:30:16.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:16.087+0000 7f1877e17700 1 -- 172.21.15.16:0/51622433 shutdown_connections 2023-10-16T12:30:16.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:16.087+0000 7f1877e17700 1 --2- 172.21.15.16:0/51622433 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1870160fe0 0x7f18700b3730 unknown :-1 s=CLOSED pgs=1499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:16.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:16.087+0000 7f1877e17700 1 --2- 172.21.15.16:0/51622433 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f187006aef0 0x7f18700c5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:16.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:16.087+0000 7f1877e17700 1 --2- 172.21.15.16:0/51622433 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f185406b020 0x7f185406d4e0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:16.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:16.087+0000 7f1877e17700 1 --2- 172.21.15.16:0/51622433 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f187006da30 0x7f1870153170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:16.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:16.087+0000 7f1877e17700 1 -- 172.21.15.16:0/51622433 >> 172.21.15.16:0/51622433 conn(0x7f18700b1ed0 msgr2=0x7f18700af520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:16.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:16.087+0000 7f1877e17700 1 -- 172.21.15.16:0/51622433 shutdown_connections 2023-10-16T12:30:16.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:16.087+0000 7f1877e17700 1 -- 172.21.15.16:0/51622433 wait complete. 2023-10-16T12:30:16.101 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods' 2023-10-16T12:30:16.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:16.225+0000 7ff93fec8700 1 Processor -- start 2023-10-16T12:30:16.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:16.225+0000 7ff93fec8700 1 -- start start 2023-10-16T12:30:16.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:16.226+0000 7ff93fec8700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7ff93815b9f0 0x7ff9380b29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:16.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:16.226+0000 7ff93fec8700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9380c3490 con 0x7ff9380cf280 2023-10-16T12:30:16.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:16.226+0000 7ff93fec8700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9380cf730 con 0x7ff93806d940 2023-10-16T12:30:16.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:16.226+0000 7ff93fec8700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9380cdad0 con 0x7ff93815b9f0 2023-10-16T12:30:16.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:16.226+0000 7ff93e465700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff9380cf280 0x7ff93806b7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34248/0 (socket says 172.21.15.16:34248) 2023-10-16T12:30:16.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:16.226+0000 7ff93e465700 1 -- 172.21.15.16:0/3033568619 learned_addr learned my addr 172.21.15.16:0/3033568619 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:30:16.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:16.226+0000 7ff93dc64700 1 --2- 172.21.15.16:0/3033568619 >> v2:172.21.15.49:3300/0 conn(0x7ff93815b9f0 0x7ff9380b29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:16.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:16.226+0000 7ff93cc62700 1 -- 172.21.15.16:0/3033568619 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2686932433 0 0) 0x7ff9380c3490 con 0x7ff9380cf280 2023-10-16T12:30:16.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:16.226+0000 7ff93cc62700 1 -- 172.21.15.16:0/3033568619 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9240035b0 con 0x7ff9380cf280 2023-10-16T12:30:16.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:16.226+0000 7ff93cc62700 1 -- 172.21.15.16:0/3033568619 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 291122004 0 0) 0x7ff9380cf730 con 0x7ff93806d940 2023-10-16T12:30:16.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:16.226+0000 7ff93cc62700 1 -- 172.21.15.16:0/3033568619 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9380c3490 con 0x7ff93806d940 2023-10-16T12:30:16.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:16.226+0000 7ff93cc62700 1 -- 172.21.15.16:0/3033568619 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1296761593 0 0) 0x7ff9240035b0 con 0x7ff9380cf280 2023-10-16T12:30:16.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:16.226+0000 7ff93dc64700 1 -- 172.21.15.16:0/3033568619 >> v1:172.21.15.49:6789/0 conn(0x7ff93806d940 legacy=0x7ff9380ceb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:16.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:16.226+0000 7ff93dc64700 1 -- 172.21.15.16:0/3033568619 >> v1:172.21.15.16:6789/0 conn(0x7ff9380cf280 legacy=0x7ff93806b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:16.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:16.226+0000 7ff93dc64700 1 -- 172.21.15.16:0/3033568619 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9380cddb0 con 0x7ff93815b9f0 2023-10-16T12:30:16.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:16.227+0000 7ff93dc64700 1 --2- 172.21.15.16:0/3033568619 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff93815b9f0 0x7ff9380b29a0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7ff928009840 tx=0x7ff928009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=200498d61b8a5400 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:16.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:16.227+0000 7ff93cc62700 1 -- 172.21.15.16:0/3033568619 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9300019f0 con 0x7ff93815b9f0 2023-10-16T12:30:16.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:16.227+0000 7ff93cc62700 1 -- 172.21.15.16:0/3033568619 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff928004030 con 0x7ff93815b9f0 2023-10-16T12:30:16.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:16.227+0000 7ff93cc62700 1 -- 172.21.15.16:0/3033568619 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9280043a0 con 0x7ff93815b9f0 2023-10-16T12:30:16.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:16.227+0000 7ff93fec8700 1 -- 172.21.15.16:0/3033568619 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff93815b9f0 msgr2=0x7ff9380b29a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:16.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:16.227+0000 7ff93fec8700 1 --2- 172.21.15.16:0/3033568619 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff93815b9f0 0x7ff9380b29a0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7ff928009840 tx=0x7ff928009b50 comp rx=0 tx=0).stop 2023-10-16T12:30:16.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:16.227+0000 7ff93fec8700 1 -- 172.21.15.16:0/3033568619 shutdown_connections 2023-10-16T12:30:16.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:16.227+0000 7ff93fec8700 1 --2- 172.21.15.16:0/3033568619 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff93815b9f0 0x7ff9380b29a0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:16.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:16.227+0000 7ff93fec8700 1 -- 172.21.15.16:0/3033568619 >> 172.21.15.16:0/3033568619 conn(0x7ff9380cc5b0 msgr2=0x7ff9380cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:16.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:16.227+0000 7ff93fec8700 1 -- 172.21.15.16:0/3033568619 shutdown_connections 2023-10-16T12:30:16.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:16.227+0000 7ff93fec8700 1 -- 172.21.15.16:0/3033568619 wait complete. 2023-10-16T12:30:16.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:16.229+0000 7ff93fec8700 1 Processor -- start 2023-10-16T12:30:16.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:16.229+0000 7ff93fec8700 1 -- start start 2023-10-16T12:30:16.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:16.229+0000 7ff93fec8700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff93806d940 0x7ff9380c9000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:16.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:16.229+0000 7ff93fec8700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff9380cf280 0x7ff9380b35b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:16.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:16.229+0000 7ff93fec8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff938157c70 0x7ff9380cb920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:16.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:16.229+0000 7ff93fec8700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9380cb490 con 0x7ff938157c70 2023-10-16T12:30:16.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:16.229+0000 7ff93fec8700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9380c9540 con 0x7ff9380cf280 2023-10-16T12:30:16.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:16.229+0000 7ff93fec8700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9380ca880 con 0x7ff93806d940 2023-10-16T12:30:16.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:16.230+0000 7ff93e465700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff938157c70 0x7ff9380cb920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:16.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:16.230+0000 7ff93e465700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff938157c70 0x7ff9380cb920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44980/0 (socket says 172.21.15.16:44980) 2023-10-16T12:30:16.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:16.230+0000 7ff93e465700 1 -- 172.21.15.16:0/3731930819 learned_addr learned my addr 172.21.15.16:0/3731930819 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:16.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:16.230+0000 7ff93dc64700 1 --2- 172.21.15.16:0/3731930819 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff93806d940 0x7ff9380c9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:16.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:16.230+0000 7ff93e465700 1 -- 172.21.15.16:0/3731930819 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff93806d940 msgr2=0x7ff9380c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:16.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:16.230+0000 7ff93e465700 1 --2- 172.21.15.16:0/3731930819 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff93806d940 0x7ff9380c9000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:16.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:16.230+0000 7ff93e465700 1 -- 172.21.15.16:0/3731930819 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff9380cf280 msgr2=0x7ff9380b35b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:16.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:16.230+0000 7ff93d463700 1 --2- 172.21.15.16:0/3731930819 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff9380cf280 0x7ff9380b35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:16.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:16.230+0000 7ff93e465700 1 --2- 172.21.15.16:0/3731930819 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff9380cf280 0x7ff9380b35b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:16.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:16.230+0000 7ff93e465700 1 -- 172.21.15.16:0/3731930819 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9280094a0 con 0x7ff938157c70 2023-10-16T12:30:16.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:16.230+0000 7ff93d463700 1 --2- 172.21.15.16:0/3731930819 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff9380cf280 0x7ff9380b35b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:30:16.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:16.230+0000 7ff93e465700 1 --2- 172.21.15.16:0/3731930819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff938157c70 0x7ff9380cb920 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7ff93000cbc0 tx=0x7ff93000cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:16.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:16.230+0000 7ff92e7fc700 1 -- 172.21.15.16:0/3731930819 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff93000abb0 con 0x7ff938157c70 2023-10-16T12:30:16.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:16.230+0000 7ff92e7fc700 1 -- 172.21.15.16:0/3731930819 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff93000ad30 con 0x7ff938157c70 2023-10-16T12:30:16.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:16.230+0000 7ff92e7fc700 1 -- 172.21.15.16:0/3731930819 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff93000aea0 con 0x7ff938157c70 2023-10-16T12:30:16.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:16.230+0000 7ff93fec8700 1 -- 172.21.15.16:0/3731930819 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9380cab60 con 0x7ff938157c70 2023-10-16T12:30:16.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:16.230+0000 7ff93fec8700 1 -- 172.21.15.16:0/3731930819 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9380cb120 con 0x7ff938157c70 2023-10-16T12:30:16.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:16.231+0000 7ff93fec8700 1 -- 172.21.15.16:0/3731930819 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9380cb640 con 0x7ff938157c70 2023-10-16T12:30:16.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:16.232+0000 7ff92e7fc700 1 -- 172.21.15.16:0/3731930819 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 74) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff930007310 con 0x7ff938157c70 2023-10-16T12:30:16.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:16.232+0000 7ff92e7fc700 1 --2- 172.21.15.16:0/3731930819 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7ff91c06b050 0x7ff91c06d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:16.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:16.232+0000 7ff92e7fc700 1 -- 172.21.15.16:0/3731930819 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 6093+0+0 (secure 0 0 0) 0x7ff9300893a0 con 0x7ff938157c70 2023-10-16T12:30:16.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:16.232+0000 7ff93dc64700 1 --2- 172.21.15.16:0/3731930819 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7ff91c06b050 0x7ff91c06d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:16.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:16.233+0000 7ff93dc64700 1 --2- 172.21.15.16:0/3731930819 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7ff91c06b050 0x7ff91c06d510 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff928009b90 tx=0x7ff928013040 comp rx=0 tx=0).ready entity=mgr.8388 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:16.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:16.237+0000 7ff92e7fc700 1 -- 172.21.15.16:0/3731930819 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff9380cb640 con 0x7ff938157c70 2023-10-16T12:30:16.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:16.524+0000 7ff93fec8700 1 -- 172.21.15.16:0/3731930819 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods"]} v 0) v1 -- 0x7ff93804ebb0 con 0x7ff938157c70 2023-10-16T12:30:17.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.360+0000 7ff92e7fc700 1 -- 172.21.15.16:0/3731930819 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods"]}]=0 v749) v1 ==== 163+0+0 (secure 0 0 0) 0x7ff9300541c0 con 0x7ff938157c70 2023-10-16T12:30:17.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.363+0000 7ff93fec8700 1 -- 172.21.15.16:0/3731930819 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7ff91c06b050 msgr2=0x7ff91c06d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:17.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.363+0000 7ff93fec8700 1 --2- 172.21.15.16:0/3731930819 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7ff91c06b050 0x7ff91c06d510 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff928009b90 tx=0x7ff928013040 comp rx=0 tx=0).stop 2023-10-16T12:30:17.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.363+0000 7ff93fec8700 1 -- 172.21.15.16:0/3731930819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff938157c70 msgr2=0x7ff9380cb920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:17.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.363+0000 7ff93fec8700 1 --2- 172.21.15.16:0/3731930819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff938157c70 0x7ff9380cb920 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7ff93000cbc0 tx=0x7ff93000cf80 comp rx=0 tx=0).stop 2023-10-16T12:30:17.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.363+0000 7ff93fec8700 1 -- 172.21.15.16:0/3731930819 shutdown_connections 2023-10-16T12:30:17.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.363+0000 7ff93fec8700 1 --2- 172.21.15.16:0/3731930819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff938157c70 0x7ff9380cb920 unknown :-1 s=CLOSED pgs=1501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:17.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.363+0000 7ff93fec8700 1 --2- 172.21.15.16:0/3731930819 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff93806d940 0x7ff9380c9000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:17.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.363+0000 7ff93fec8700 1 --2- 172.21.15.16:0/3731930819 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7ff91c06b050 0x7ff91c06d510 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:17.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.363+0000 7ff93fec8700 1 --2- 172.21.15.16:0/3731930819 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff9380cf280 0x7ff9380b35b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:17.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.363+0000 7ff93fec8700 1 -- 172.21.15.16:0/3731930819 >> 172.21.15.16:0/3731930819 conn(0x7ff9380cc5b0 msgr2=0x7ff93815c810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:17.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.363+0000 7ff93fec8700 1 -- 172.21.15.16:0/3731930819 shutdown_connections 2023-10-16T12:30:17.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.363+0000 7ff93fec8700 1 -- 172.21.15.16:0/3731930819 wait complete. 2023-10-16T12:30:17.381 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-16T12:30:17.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.515+0000 7fec29e83700 1 Processor -- start 2023-10-16T12:30:17.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.515+0000 7fec29e83700 1 -- start start 2023-10-16T12:30:17.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.516+0000 7fec29e83700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fec2406e570 0x7fec2415e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:17.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.516+0000 7fec29e83700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec2415b6a0 con 0x7fec2406a7e0 2023-10-16T12:30:17.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.516+0000 7fec29e83700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec24161b20 con 0x7fec2415b2c0 2023-10-16T12:30:17.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.516+0000 7fec29e83700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec24161e00 con 0x7fec2406e570 2023-10-16T12:30:17.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.516+0000 7fec23fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fec2406a7e0 0x7fec24071a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34262/0 (socket says 172.21.15.16:34262) 2023-10-16T12:30:17.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.516+0000 7fec23fff700 1 -- 172.21.15.16:0/1479793833 learned_addr learned my addr 172.21.15.16:0/1479793833 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:30:17.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.516+0000 7fec22ffd700 1 --2- 172.21.15.16:0/1479793833 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fec2406e570 0x7fec2415e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:17.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.516+0000 7fec227fc700 1 -- 172.21.15.16:0/1479793833 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3733551193 0 0) 0x7fec2415b6a0 con 0x7fec2406a7e0 2023-10-16T12:30:17.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.516+0000 7fec227fc700 1 -- 172.21.15.16:0/1479793833 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec0c0035b0 con 0x7fec2406a7e0 2023-10-16T12:30:17.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.516+0000 7fec227fc700 1 -- 172.21.15.16:0/1479793833 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 402987563 0 0) 0x7fec24161b20 con 0x7fec2415b2c0 2023-10-16T12:30:17.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.516+0000 7fec227fc700 1 -- 172.21.15.16:0/1479793833 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec2415b6a0 con 0x7fec2415b2c0 2023-10-16T12:30:17.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.516+0000 7fec227fc700 1 -- 172.21.15.16:0/1479793833 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1289708771 0 0) 0x7fec0c0035b0 con 0x7fec2406a7e0 2023-10-16T12:30:17.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.516+0000 7fec227fc700 1 -- 172.21.15.16:0/1479793833 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fec2406e570 msgr2=0x7fec2415e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:17.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.516+0000 7fec227fc700 1 --2- 172.21.15.16:0/1479793833 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fec2406e570 0x7fec2415e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:17.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.516+0000 7fec227fc700 1 -- 172.21.15.16:0/1479793833 >> v1:172.21.15.49:6789/0 conn(0x7fec2415b2c0 legacy=0x7fec24161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:17.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.516+0000 7fec227fc700 1 -- 172.21.15.16:0/1479793833 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec241620e0 con 0x7fec2406a7e0 2023-10-16T12:30:17.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.516+0000 7fec227fc700 1 -- 172.21.15.16:0/1479793833 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7fec1c0019f0 con 0x7fec2406a7e0 2023-10-16T12:30:17.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.516+0000 7fec227fc700 1 -- 172.21.15.16:0/1479793833 >> v1:172.21.15.16:6789/0 conn(0x7fec2406a7e0 legacy=0x7fec24071a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:17.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.516+0000 7fec227fc700 1 --2- 172.21.15.16:0/1479793833 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec0c003bc0 0x7fec0c003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:17.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.516+0000 7fec227fc700 1 --2- 172.21.15.16:0/1479793833 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fec0c004850 0x7fec0c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:17.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.516+0000 7fec227fc700 1 --2- 172.21.15.16:0/1479793833 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fec2406e570 0x7fec0c005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:17.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.516+0000 7fec227fc700 1 -- 172.21.15.16:0/1479793833 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec24161e00 con 0x7fec0c003bc0 2023-10-16T12:30:17.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.516+0000 7fec227fc700 1 -- 172.21.15.16:0/1479793833 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec0c00bb20 con 0x7fec0c004850 2023-10-16T12:30:17.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.516+0000 7fec227fc700 1 -- 172.21.15.16:0/1479793833 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec0c00bce0 con 0x7fec2406e570 2023-10-16T12:30:17.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.517+0000 7fec237fe700 1 --2- 172.21.15.16:0/1479793833 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec0c003bc0 0x7fec0c003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:17.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.517+0000 7fec22ffd700 1 --2- 172.21.15.16:0/1479793833 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fec0c004850 0x7fec0c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:17.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.517+0000 7fec23fff700 1 --2- 172.21.15.16:0/1479793833 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fec2406e570 0x7fec0c005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:17.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.517+0000 7fec237fe700 1 -- 172.21.15.16:0/1479793833 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fec2406e570 msgr2=0x7fec0c005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:17.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.517+0000 7fec237fe700 1 --2- 172.21.15.16:0/1479793833 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fec2406e570 0x7fec0c005230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:17.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.517+0000 7fec237fe700 1 -- 172.21.15.16:0/1479793833 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fec0c004850 msgr2=0x7fec0c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:17.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.517+0000 7fec237fe700 1 --2- 172.21.15.16:0/1479793833 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fec0c004850 0x7fec0c004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:17.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.517+0000 7fec237fe700 1 -- 172.21.15.16:0/1479793833 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec0c0035b0 con 0x7fec0c003bc0 2023-10-16T12:30:17.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.517+0000 7fec22ffd700 1 --2- 172.21.15.16:0/1479793833 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fec0c004850 0x7fec0c004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:30:17.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.517+0000 7fec237fe700 1 --2- 172.21.15.16:0/1479793833 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec0c003bc0 0x7fec0c003fc0 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7fec14001940 tx=0x7fec0c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bea6e1f081f16d7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:17.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.517+0000 7fec227fc700 1 -- 172.21.15.16:0/1479793833 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec14001430 con 0x7fec0c003bc0 2023-10-16T12:30:17.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.517+0000 7fec227fc700 1 -- 172.21.15.16:0/1479793833 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec14001d70 con 0x7fec0c003bc0 2023-10-16T12:30:17.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.517+0000 7fec227fc700 1 -- 172.21.15.16:0/1479793833 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec14005510 con 0x7fec0c003bc0 2023-10-16T12:30:17.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.518+0000 7fec29e83700 1 -- 172.21.15.16:0/1479793833 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec0c003bc0 msgr2=0x7fec0c003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:17.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.518+0000 7fec29e83700 1 --2- 172.21.15.16:0/1479793833 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec0c003bc0 0x7fec0c003fc0 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7fec14001940 tx=0x7fec0c005880 comp rx=0 tx=0).stop 2023-10-16T12:30:17.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.518+0000 7fec23fff700 1 -- 172.21.15.16:0/1479793833 reap_dead start 2023-10-16T12:30:17.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.518+0000 7fec29e83700 1 -- 172.21.15.16:0/1479793833 shutdown_connections 2023-10-16T12:30:17.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.518+0000 7fec29e83700 1 -- 172.21.15.16:0/1479793833 >> 172.21.15.16:0/1479793833 conn(0x7fec240b1f40 msgr2=0x7fec240b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:17.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.518+0000 7fec29e83700 1 -- 172.21.15.16:0/1479793833 shutdown_connections 2023-10-16T12:30:17.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.518+0000 7fec29e83700 1 -- 172.21.15.16:0/1479793833 wait complete. 2023-10-16T12:30:17.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.519+0000 7fec29e83700 1 Processor -- start 2023-10-16T12:30:17.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.519+0000 7fec29e83700 1 -- start start 2023-10-16T12:30:17.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.520+0000 7fec29e83700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec0c003bc0 0x7fec240c3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:17.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.520+0000 7fec29e83700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fec2415e780 0x7fec2415ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:17.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.520+0000 7fec29e83700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fec240d0030 0x7fec240d04b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:17.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.520+0000 7fec29e83700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec240d09f0 con 0x7fec0c003bc0 2023-10-16T12:30:17.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.520+0000 7fec29e83700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec240c4e40 con 0x7fec2415e780 2023-10-16T12:30:17.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.520+0000 7fec29e83700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec240c4230 con 0x7fec240d0030 2023-10-16T12:30:17.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.520+0000 7fec237fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec0c003bc0 0x7fec240c3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:17.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.520+0000 7fec237fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec0c003bc0 0x7fec240c3bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44996/0 (socket says 172.21.15.16:44996) 2023-10-16T12:30:17.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.520+0000 7fec237fe700 1 -- 172.21.15.16:0/928907500 learned_addr learned my addr 172.21.15.16:0/928907500 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:17.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.520+0000 7fec23fff700 1 --2- 172.21.15.16:0/928907500 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fec240d0030 0x7fec240d04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:17.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.520+0000 7fec22ffd700 1 --2- 172.21.15.16:0/928907500 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fec2415e780 0x7fec2415ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:17.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.520+0000 7fec237fe700 1 -- 172.21.15.16:0/928907500 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fec240d0030 msgr2=0x7fec240d04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:17.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.520+0000 7fec237fe700 1 --2- 172.21.15.16:0/928907500 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fec240d0030 0x7fec240d04b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:17.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.520+0000 7fec237fe700 1 -- 172.21.15.16:0/928907500 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fec2415e780 msgr2=0x7fec2415ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:17.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.520+0000 7fec237fe700 1 --2- 172.21.15.16:0/928907500 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fec2415e780 0x7fec2415ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:17.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.520+0000 7fec237fe700 1 -- 172.21.15.16:0/928907500 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec240c4510 con 0x7fec0c003bc0 2023-10-16T12:30:17.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.520+0000 7fec22ffd700 1 --2- 172.21.15.16:0/928907500 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fec2415e780 0x7fec2415ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:30:17.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.521+0000 7fec237fe700 1 --2- 172.21.15.16:0/928907500 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec0c003bc0 0x7fec240c3bc0 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7fec0c00baf0 tx=0x7fec140012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:17.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.521+0000 7fec0bfff700 1 -- 172.21.15.16:0/928907500 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec14006c70 con 0x7fec0c003bc0 2023-10-16T12:30:17.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.521+0000 7fec0bfff700 1 -- 172.21.15.16:0/928907500 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec14006dd0 con 0x7fec0c003bc0 2023-10-16T12:30:17.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.521+0000 7fec29e83700 1 -- 172.21.15.16:0/928907500 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec240c47f0 con 0x7fec0c003bc0 2023-10-16T12:30:17.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.521+0000 7fec0bfff700 1 -- 172.21.15.16:0/928907500 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec14007260 con 0x7fec0c003bc0 2023-10-16T12:30:17.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.521+0000 7fec29e83700 1 -- 172.21.15.16:0/928907500 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec240c52d0 con 0x7fec0c003bc0 2023-10-16T12:30:17.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.522+0000 7fec29e83700 1 -- 172.21.15.16:0/928907500 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec2415dcd0 con 0x7fec0c003bc0 2023-10-16T12:30:17.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.522+0000 7fec0bfff700 1 -- 172.21.15.16:0/928907500 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 74) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fec140073c0 con 0x7fec0c003bc0 2023-10-16T12:30:17.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.523+0000 7fec0bfff700 1 --2- 172.21.15.16:0/928907500 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7fec1006b020 0x7fec1006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:17.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.523+0000 7fec0bfff700 1 -- 172.21.15.16:0/928907500 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 6093+0+0 (secure 0 0 0) 0x7fec1408a920 con 0x7fec0c003bc0 2023-10-16T12:30:17.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.523+0000 7fec22ffd700 1 --2- 172.21.15.16:0/928907500 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7fec1006b020 0x7fec1006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:17.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.523+0000 7fec22ffd700 1 --2- 172.21.15.16:0/928907500 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7fec1006b020 0x7fec1006d4e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fec240c3860 tx=0x7fec18008040 comp rx=0 tx=0).ready entity=mgr.8388 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:17.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.527+0000 7fec0bfff700 1 -- 172.21.15.16:0/928907500 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fec14055750 con 0x7fec0c003bc0 2023-10-16T12:30:17.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.729+0000 7fec29e83700 1 -- 172.21.15.16:0/928907500 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fec240d09f0 con 0x7fec0c003bc0 2023-10-16T12:30:17.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.732+0000 7fec0bfff700 1 -- 172.21.15.16:0/928907500 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v46)=0 v46) v1 ==== 122+0+0 (secure 0 0 0) 0x7fec14058d70 con 0x7fec0c003bc0 2023-10-16T12:30:17.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.735+0000 7fec29e83700 1 -- 172.21.15.16:0/928907500 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7fec1006b020 msgr2=0x7fec1006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:17.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.735+0000 7fec29e83700 1 --2- 172.21.15.16:0/928907500 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7fec1006b020 0x7fec1006d4e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fec240c3860 tx=0x7fec18008040 comp rx=0 tx=0).stop 2023-10-16T12:30:17.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.735+0000 7fec29e83700 1 -- 172.21.15.16:0/928907500 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec0c003bc0 msgr2=0x7fec240c3bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:17.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.735+0000 7fec29e83700 1 --2- 172.21.15.16:0/928907500 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec0c003bc0 0x7fec240c3bc0 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7fec0c00baf0 tx=0x7fec140012d0 comp rx=0 tx=0).stop 2023-10-16T12:30:17.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.735+0000 7fec29e83700 1 -- 172.21.15.16:0/928907500 shutdown_connections 2023-10-16T12:30:17.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.735+0000 7fec29e83700 1 --2- 172.21.15.16:0/928907500 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fec240d0030 0x7fec240d04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:17.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.735+0000 7fec29e83700 1 --2- 172.21.15.16:0/928907500 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec0c003bc0 0x7fec240c3bc0 unknown :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:17.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.735+0000 7fec29e83700 1 --2- 172.21.15.16:0/928907500 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7fec1006b020 0x7fec1006d4e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:17.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.735+0000 7fec29e83700 1 --2- 172.21.15.16:0/928907500 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fec2415e780 0x7fec2415ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:17.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.735+0000 7fec29e83700 1 -- 172.21.15.16:0/928907500 >> 172.21.15.16:0/928907500 conn(0x7fec240b1f40 msgr2=0x7fec240b4f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:17.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.735+0000 7fec29e83700 1 -- 172.21.15.16:0/928907500 shutdown_connections 2023-10-16T12:30:17.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:17.735+0000 7fec29e83700 1 -- 172.21.15.16:0/928907500 wait complete. 2023-10-16T12:30:17.749 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T12:30:17.749 DEBUG:teuthology.orchestra.run.smithi016:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T12:30:17.792 DEBUG:teuthology.orchestra.run.smithi049:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T12:30:17.841 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T12:30:17.842 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:30:17.842 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:30:17.860 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16287364k,nr_inodes=4071841,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20251 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3261172k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T12:30:17.860 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T12:30:17.860 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:30:17.880 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284204k,nr_inodes=4071051,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=753 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260540k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T12:30:17.881 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:30:17.881 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:30:17.916 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16287364k,nr_inodes=4071841,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20251 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3261172k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T12:30:17.917 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T12:30:17.917 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:30:17.940 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284204k,nr_inodes=4071051,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=753 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260540k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T12:30:17.941 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-16T12:30:17.941 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:30:18.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.079+0000 7f34df55a700 1 Processor -- start 2023-10-16T12:30:18.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.079+0000 7f34df55a700 1 -- start start 2023-10-16T12:30:18.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.080+0000 7f34df55a700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f34d815db40 0x7f34d80cb980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:18.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.080+0000 7f34df55a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f34d80ce870 0x7f34d806ac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:18.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.080+0000 7f34df55a700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34d80cedb0 con 0x7f34d806a7e0 2023-10-16T12:30:18.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.080+0000 7f34df55a700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34d806b1a0 con 0x7f34d80ce870 2023-10-16T12:30:18.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.080+0000 7f34df55a700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34d815df20 con 0x7f34d815db40 2023-10-16T12:30:18.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.080+0000 7f34ddaf7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f34d80ce870 0x7f34d806ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:18.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.080+0000 7f34ddaf7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f34d80ce870 0x7f34d806ac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45008/0 (socket says 172.21.15.16:45008) 2023-10-16T12:30:18.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.080+0000 7f34ddaf7700 1 -- 172.21.15.16:0/2760215474 learned_addr learned my addr 172.21.15.16:0/2760215474 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:18.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.080+0000 7f34dd2f6700 1 --2- 172.21.15.16:0/2760215474 >> v2:172.21.15.49:3300/0 conn(0x7f34d815db40 0x7f34d80cb980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:18.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.081+0000 7f34ddaf7700 1 -- 172.21.15.16:0/2760215474 >> v2:172.21.15.49:3300/0 conn(0x7f34d815db40 msgr2=0x7f34d80cb980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:18.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.081+0000 7f34ddaf7700 1 --2- 172.21.15.16:0/2760215474 >> v2:172.21.15.49:3300/0 conn(0x7f34d815db40 0x7f34d80cb980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:18.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.081+0000 7f34ddaf7700 1 -- 172.21.15.16:0/2760215474 >> v1:172.21.15.49:6789/0 conn(0x7f34d806a7e0 legacy=0x7f34d80ce160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:18.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.081+0000 7f34ddaf7700 1 -- 172.21.15.16:0/2760215474 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34d80d12c0 con 0x7f34d80ce870 2023-10-16T12:30:18.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.081+0000 7f34dd2f6700 1 --2- 172.21.15.16:0/2760215474 >> v2:172.21.15.49:3300/0 conn(0x7f34d815db40 0x7f34d80cb980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:30:18.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.081+0000 7f34ddaf7700 1 --2- 172.21.15.16:0/2760215474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34d80ce870 0x7f34d806ac60 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f34d806f340 tx=0x7f34d000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e68df4ed4985775 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:18.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.081+0000 7f34cffff700 1 -- 172.21.15.16:0/2760215474 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34d000e070 con 0x7f34d80ce870 2023-10-16T12:30:18.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.081+0000 7f34cffff700 1 -- 172.21.15.16:0/2760215474 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34d0003a20 con 0x7f34d80ce870 2023-10-16T12:30:18.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.081+0000 7f34cffff700 1 -- 172.21.15.16:0/2760215474 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34d0003dd0 con 0x7f34d80ce870 2023-10-16T12:30:18.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.081+0000 7f34df55a700 1 -- 172.21.15.16:0/2760215474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34d80ce870 msgr2=0x7f34d806ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:18.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.081+0000 7f34df55a700 1 --2- 172.21.15.16:0/2760215474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34d80ce870 0x7f34d806ac60 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f34d806f340 tx=0x7f34d000b330 comp rx=0 tx=0).stop 2023-10-16T12:30:18.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.081+0000 7f34df55a700 1 -- 172.21.15.16:0/2760215474 shutdown_connections 2023-10-16T12:30:18.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.081+0000 7f34df55a700 1 --2- 172.21.15.16:0/2760215474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34d80ce870 0x7f34d806ac60 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:18.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.081+0000 7f34df55a700 1 --2- 172.21.15.16:0/2760215474 >> v2:172.21.15.49:3300/0 conn(0x7f34d815db40 0x7f34d80cb980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:18.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.081+0000 7f34df55a700 1 -- 172.21.15.16:0/2760215474 >> 172.21.15.16:0/2760215474 conn(0x7f34d80b1fd0 msgr2=0x7f34d80b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:18.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.082+0000 7f34df55a700 1 -- 172.21.15.16:0/2760215474 shutdown_connections 2023-10-16T12:30:18.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.082+0000 7f34df55a700 1 -- 172.21.15.16:0/2760215474 wait complete. 2023-10-16T12:30:18.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.083+0000 7f34df55a700 1 Processor -- start 2023-10-16T12:30:18.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.083+0000 7f34df55a700 1 -- start start 2023-10-16T12:30:18.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.083+0000 7f34df55a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f34d806a7e0 0x7f34d81f4280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:18.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.083+0000 7f34df55a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34d80ce870 0x7f34d80c6970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:18.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.083+0000 7f34df55a700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f34d815db40 0x7f34d81f5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:18.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.083+0000 7f34df55a700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34d81f64f0 con 0x7f34d80ce870 2023-10-16T12:30:18.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.083+0000 7f34df55a700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34d80aca00 con 0x7f34d815db40 2023-10-16T12:30:18.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.083+0000 7f34df55a700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34d81f4040 con 0x7f34d806a7e0 2023-10-16T12:30:18.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.084+0000 7f34dcaf5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34d80ce870 0x7f34d80c6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:18.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.084+0000 7f34dcaf5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34d80ce870 0x7f34d80c6970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45012/0 (socket says 172.21.15.16:45012) 2023-10-16T12:30:18.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.084+0000 7f34dcaf5700 1 -- 172.21.15.16:0/4254190554 learned_addr learned my addr 172.21.15.16:0/4254190554 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:18.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.084+0000 7f34ddaf7700 1 --2- 172.21.15.16:0/4254190554 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f34d815db40 0x7f34d81f5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:18.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.084+0000 7f34dd2f6700 1 --2- 172.21.15.16:0/4254190554 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f34d806a7e0 0x7f34d81f4280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:18.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.084+0000 7f34dcaf5700 1 -- 172.21.15.16:0/4254190554 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f34d806a7e0 msgr2=0x7f34d81f4280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:18.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.084+0000 7f34dcaf5700 1 --2- 172.21.15.16:0/4254190554 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f34d806a7e0 0x7f34d81f4280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:18.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.084+0000 7f34dcaf5700 1 -- 172.21.15.16:0/4254190554 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f34d815db40 msgr2=0x7f34d81f5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:18.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.084+0000 7f34dcaf5700 1 --2- 172.21.15.16:0/4254190554 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f34d815db40 0x7f34d81f5fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:18.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.084+0000 7f34dcaf5700 1 -- 172.21.15.16:0/4254190554 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34d0009b30 con 0x7f34d80ce870 2023-10-16T12:30:18.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.084+0000 7f34ddaf7700 1 --2- 172.21.15.16:0/4254190554 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f34d815db40 0x7f34d81f5fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:30:18.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.084+0000 7f34dcaf5700 1 --2- 172.21.15.16:0/4254190554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34d80ce870 0x7f34d80c6970 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7f34d400b4d0 tx=0x7f34d400b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:18.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.084+0000 7f34cdffb700 1 -- 172.21.15.16:0/4254190554 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34d4011830 con 0x7f34d80ce870 2023-10-16T12:30:18.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.084+0000 7f34cdffb700 1 -- 172.21.15.16:0/4254190554 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34d4011990 con 0x7f34d80ce870 2023-10-16T12:30:18.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.084+0000 7f34cdffb700 1 -- 172.21.15.16:0/4254190554 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34d4011d40 con 0x7f34d80ce870 2023-10-16T12:30:18.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.084+0000 7f34df55a700 1 -- 172.21.15.16:0/4254190554 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34d81f37a0 con 0x7f34d80ce870 2023-10-16T12:30:18.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.085+0000 7f34df55a700 1 -- 172.21.15.16:0/4254190554 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34d81f3d60 con 0x7f34d80ce870 2023-10-16T12:30:18.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.085+0000 7f34df55a700 1 -- 172.21.15.16:0/4254190554 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34d804ebb0 con 0x7f34d80ce870 2023-10-16T12:30:18.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.086+0000 7f34cdffb700 1 -- 172.21.15.16:0/4254190554 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 74) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f34d401a400 con 0x7f34d80ce870 2023-10-16T12:30:18.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.086+0000 7f34cdffb700 1 --2- 172.21.15.16:0/4254190554 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f34bc06b020 0x7f34bc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:18.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.087+0000 7f34dd2f6700 1 --2- 172.21.15.16:0/4254190554 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f34bc06b020 0x7f34bc06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:18.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.087+0000 7f34cdffb700 1 -- 172.21.15.16:0/4254190554 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 6093+0+0 (secure 0 0 0) 0x7f34d4089540 con 0x7f34d80ce870 2023-10-16T12:30:18.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.087+0000 7f34dd2f6700 1 --2- 172.21.15.16:0/4254190554 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f34bc06b020 0x7f34bc06d4e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f34c8006fd0 tx=0x7f34c8008040 comp rx=0 tx=0).ready entity=mgr.8388 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:18.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.091+0000 7f34cdffb700 1 -- 172.21.15.16:0/4254190554 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f34d4054360 con 0x7f34d80ce870 2023-10-16T12:30:18.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.341+0000 7f34df55a700 1 -- 172.21.15.16:0/4254190554 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f34d80cb450 con 0x7f34d80ce870 2023-10-16T12:30:18.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.342+0000 7f34cdffb700 1 -- 172.21.15.16:0/4254190554 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+3693 (secure 0 0 0) 0x7f34d4057980 con 0x7f34d80ce870 2023-10-16T12:30:18.344 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:30:18.344 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":63,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6893,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/2971585675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":2971585675},{"type":"v1","addr":"172.21.15.49:6835","nonce":2971585675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":7163,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/1410302567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":1410302567},{"type":"v1","addr":"172.21.15.16:6837","nonce":1410302567}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7723,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/3151669585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":3151669585},{"type":"v1","addr":"172.21.15.49:6834","nonce":3151669585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":8106,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/52924937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":52924937},{"type":"v1","addr":"172.21.15.16:6836","nonce":52924937}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[]} 2023-10-16T12:30:18.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.344+0000 7f34df55a700 1 -- 172.21.15.16:0/4254190554 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f34bc06b020 msgr2=0x7f34bc06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:18.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.344+0000 7f34df55a700 1 --2- 172.21.15.16:0/4254190554 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f34bc06b020 0x7f34bc06d4e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f34c8006fd0 tx=0x7f34c8008040 comp rx=0 tx=0).stop 2023-10-16T12:30:18.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.344+0000 7f34df55a700 1 -- 172.21.15.16:0/4254190554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34d80ce870 msgr2=0x7f34d80c6970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:18.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.344+0000 7f34df55a700 1 --2- 172.21.15.16:0/4254190554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34d80ce870 0x7f34d80c6970 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7f34d400b4d0 tx=0x7f34d400b7e0 comp rx=0 tx=0).stop 2023-10-16T12:30:18.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.344+0000 7f34df55a700 1 -- 172.21.15.16:0/4254190554 shutdown_connections 2023-10-16T12:30:18.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.344+0000 7f34df55a700 1 --2- 172.21.15.16:0/4254190554 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f34bc06b020 0x7f34bc06d4e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:18.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.344+0000 7f34df55a700 1 --2- 172.21.15.16:0/4254190554 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f34d815db40 0x7f34d81f5fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:18.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.344+0000 7f34df55a700 1 --2- 172.21.15.16:0/4254190554 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f34d806a7e0 0x7f34d81f4280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:18.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.344+0000 7f34df55a700 1 --2- 172.21.15.16:0/4254190554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34d80ce870 0x7f34d80c6970 unknown :-1 s=CLOSED pgs=1506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:18.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.344+0000 7f34df55a700 1 -- 172.21.15.16:0/4254190554 >> 172.21.15.16:0/4254190554 conn(0x7f34d80b1fd0 msgr2=0x7f34d80ce130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:18.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.344+0000 7f34df55a700 1 -- 172.21.15.16:0/4254190554 shutdown_connections 2023-10-16T12:30:18.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.344+0000 7f34df55a700 1 -- 172.21.15.16:0/4254190554 wait complete. 2023-10-16T12:30:18.349 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 63 2023-10-16T12:30:18.358 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-16T12:30:18.359 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-16T12:30:18.359 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-16T12:30:18.359 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T12:30:18.360 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-16T12:30:18.360 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-16T12:30:18.360 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-16T12:30:18.360 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T12:30:18.361 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-16T12:30:18.361 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-16T12:30:18.361 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-16T12:30:18.361 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T12:30:18.362 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-16T12:30:18.362 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-16T12:30:18.362 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-16T12:30:18.363 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T12:30:18.368 INFO:tasks.ceph.mds.a:Stopped 2023-10-16T12:30:18.368 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-16T12:30:18.370 INFO:tasks.ceph.mds.c:Stopped 2023-10-16T12:30:18.370 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-16T12:30:18.372 INFO:tasks.ceph.mds.b:Stopped 2023-10-16T12:30:18.373 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-16T12:30:18.374 INFO:tasks.ceph.mds.d:Stopped 2023-10-16T12:30:18.374 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-16T12:30:18.412 INFO:tasks.ceph.mds.a:Started 2023-10-16T12:30:18.413 DEBUG:teuthology.parallel:result is None 2023-10-16T12:30:18.414 INFO:tasks.ceph.mds.c:Started 2023-10-16T12:30:18.414 DEBUG:teuthology.parallel:result is None 2023-10-16T12:30:18.416 INFO:tasks.ceph.mds.b:Started 2023-10-16T12:30:18.416 DEBUG:teuthology.parallel:result is None 2023-10-16T12:30:18.417 INFO:tasks.ceph.mds.d:Started 2023-10-16T12:30:18.417 DEBUG:teuthology.parallel:result is None 2023-10-16T12:30:18.418 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-16T12:30:18.495 INFO:tasks.ceph.mds.a.smithi016.stdout:starting mds.a at 2023-10-16T12:30:18.495 INFO:tasks.ceph.mds.c.smithi016.stdout:starting mds.c at 2023-10-16T12:30:18.504 INFO:tasks.ceph.mds.d.smithi049.stdout:starting mds.d at 2023-10-16T12:30:18.504 INFO:tasks.ceph.mds.b.smithi049.stdout:starting mds.b at 2023-10-16T12:30:18.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.549+0000 7fd750d2f700 1 Processor -- start 2023-10-16T12:30:18.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.549+0000 7fd750d2f700 1 -- start start 2023-10-16T12:30:18.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.550+0000 7fd750d2f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd74c15d230 0x7fd74c0b5a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:18.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.550+0000 7fd750d2f700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd74c15bc80 con 0x7fd74c0cf880 2023-10-16T12:30:18.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.550+0000 7fd750d2f700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd74c15b070 con 0x7fd74c0cf4a0 2023-10-16T12:30:18.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.550+0000 7fd750d2f700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd74c15b350 con 0x7fd74c15d230 2023-10-16T12:30:18.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.550+0000 7fd749d9b700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd74c0cf880 0x7fd74c0b52f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34316/0 (socket says 172.21.15.16:34316) 2023-10-16T12:30:18.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.550+0000 7fd749d9b700 1 -- 172.21.15.16:0/4188678419 learned_addr learned my addr 172.21.15.16:0/4188678419 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:30:18.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.550+0000 7fd74ad9d700 1 --2- 172.21.15.16:0/4188678419 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd74c15d230 0x7fd74c0b5a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:18.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.551+0000 7fd74959a700 1 -- 172.21.15.16:0/4188678419 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1877312984 0 0) 0x7fd74c15bc80 con 0x7fd74c0cf880 2023-10-16T12:30:18.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.551+0000 7fd74959a700 1 -- 172.21.15.16:0/4188678419 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7340035b0 con 0x7fd74c0cf880 2023-10-16T12:30:18.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.551+0000 7fd74959a700 1 -- 172.21.15.16:0/4188678419 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3054281548 0 0) 0x7fd74c15b070 con 0x7fd74c0cf4a0 2023-10-16T12:30:18.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.551+0000 7fd74959a700 1 -- 172.21.15.16:0/4188678419 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd74c15bc80 con 0x7fd74c0cf4a0 2023-10-16T12:30:18.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.551+0000 7fd74ad9d700 1 -- 172.21.15.16:0/4188678419 >> v1:172.21.15.49:6789/0 conn(0x7fd74c0cf4a0 legacy=0x7fd74c0726e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:18.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.551+0000 7fd74ad9d700 1 -- 172.21.15.16:0/4188678419 >> v1:172.21.15.16:6789/0 conn(0x7fd74c0cf880 legacy=0x7fd74c0b52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:18.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.551+0000 7fd74959a700 1 -- 172.21.15.16:0/4188678419 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3443170338 0 0) 0x7fd7340035b0 con 0x7fd74c0cf880 2023-10-16T12:30:18.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.551+0000 7fd74ad9d700 1 -- 172.21.15.16:0/4188678419 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd74c15b630 con 0x7fd74c15d230 2023-10-16T12:30:18.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.551+0000 7fd74ad9d700 1 --2- 172.21.15.16:0/4188678419 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd74c15d230 0x7fd74c0b5a00 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd744009dc0 tx=0x7fd74400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ba543b2af6a992a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:18.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.551+0000 7fd74959a700 1 -- 172.21.15.16:0/4188678419 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd74400e070 con 0x7fd74c15d230 2023-10-16T12:30:18.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.551+0000 7fd74959a700 1 -- 172.21.15.16:0/4188678419 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd744003a20 con 0x7fd74c15d230 2023-10-16T12:30:18.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.551+0000 7fd74959a700 1 -- 172.21.15.16:0/4188678419 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd73c001670 con 0x7fd74c15d230 2023-10-16T12:30:18.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.552+0000 7fd750d2f700 1 -- 172.21.15.16:0/4188678419 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd74c15d230 msgr2=0x7fd74c0b5a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:18.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.552+0000 7fd750d2f700 1 --2- 172.21.15.16:0/4188678419 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd74c15d230 0x7fd74c0b5a00 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd744009dc0 tx=0x7fd74400b330 comp rx=0 tx=0).stop 2023-10-16T12:30:18.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.552+0000 7fd750d2f700 1 -- 172.21.15.16:0/4188678419 shutdown_connections 2023-10-16T12:30:18.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.552+0000 7fd750d2f700 1 --2- 172.21.15.16:0/4188678419 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd74c15d230 0x7fd74c0b5a00 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:18.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.552+0000 7fd750d2f700 1 -- 172.21.15.16:0/4188678419 >> 172.21.15.16:0/4188678419 conn(0x7fd74c0b1fd0 msgr2=0x7fd74c0b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:18.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.552+0000 7fd750d2f700 1 -- 172.21.15.16:0/4188678419 shutdown_connections 2023-10-16T12:30:18.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.552+0000 7fd750d2f700 1 -- 172.21.15.16:0/4188678419 wait complete. 2023-10-16T12:30:18.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.553+0000 7fd750d2f700 1 Processor -- start 2023-10-16T12:30:18.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.553+0000 7fd750d2f700 1 -- start start 2023-10-16T12:30:18.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.554+0000 7fd750d2f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd74c0cf4a0 0x7fd74c152040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:18.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.554+0000 7fd750d2f700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd74c0cf880 0x7fd74c0c45d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:18.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.554+0000 7fd750d2f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd74c15d230 0x7fd74c0c4b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:18.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.554+0000 7fd750d2f700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd74c158530 con 0x7fd74c0cf4a0 2023-10-16T12:30:18.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.554+0000 7fd750d2f700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd74c152580 con 0x7fd74c0cf880 2023-10-16T12:30:18.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.554+0000 7fd750d2f700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd74c158250 con 0x7fd74c15d230 2023-10-16T12:30:18.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.554+0000 7fd74a59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd74c0cf4a0 0x7fd74c152040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:18.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.554+0000 7fd74a59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd74c0cf4a0 0x7fd74c152040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45048/0 (socket says 172.21.15.16:45048) 2023-10-16T12:30:18.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.554+0000 7fd74a59c700 1 -- 172.21.15.16:0/2727717064 learned_addr learned my addr 172.21.15.16:0/2727717064 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:18.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.554+0000 7fd749d9b700 1 --2- 172.21.15.16:0/2727717064 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd74c0cf880 0x7fd74c0c45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:18.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.554+0000 7fd74ad9d700 1 --2- 172.21.15.16:0/2727717064 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd74c15d230 0x7fd74c0c4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:18.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.554+0000 7fd74a59c700 1 -- 172.21.15.16:0/2727717064 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd74c15d230 msgr2=0x7fd74c0c4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:18.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.554+0000 7fd74a59c700 1 --2- 172.21.15.16:0/2727717064 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd74c15d230 0x7fd74c0c4b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:18.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.554+0000 7fd74a59c700 1 -- 172.21.15.16:0/2727717064 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd74c0cf880 msgr2=0x7fd74c0c45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:18.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.554+0000 7fd74a59c700 1 --2- 172.21.15.16:0/2727717064 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd74c0cf880 0x7fd74c0c45d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:18.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.554+0000 7fd74a59c700 1 -- 172.21.15.16:0/2727717064 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd744009a70 con 0x7fd74c0cf4a0 2023-10-16T12:30:18.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.555+0000 7fd74ad9d700 1 --2- 172.21.15.16:0/2727717064 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd74c15d230 0x7fd74c0c4b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:30:18.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.555+0000 7fd74a59c700 1 --2- 172.21.15.16:0/2727717064 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd74c0cf4a0 0x7fd74c152040 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7fd74000c520 tx=0x7fd74000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:18.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.555+0000 7fd73affd700 1 -- 172.21.15.16:0/2727717064 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7400052d0 con 0x7fd74c0cf4a0 2023-10-16T12:30:18.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.555+0000 7fd73affd700 1 -- 172.21.15.16:0/2727717064 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd740005430 con 0x7fd74c0cf4a0 2023-10-16T12:30:18.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.555+0000 7fd750d2f700 1 -- 172.21.15.16:0/2727717064 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd74c1efcc0 con 0x7fd74c0cf4a0 2023-10-16T12:30:18.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.555+0000 7fd73affd700 1 -- 172.21.15.16:0/2727717064 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7400038a0 con 0x7fd74c0cf4a0 2023-10-16T12:30:18.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.555+0000 7fd750d2f700 1 -- 172.21.15.16:0/2727717064 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd74c1f0280 con 0x7fd74c0cf4a0 2023-10-16T12:30:18.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.556+0000 7fd750d2f700 1 -- 172.21.15.16:0/2727717064 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd74c1efb20 con 0x7fd74c0cf4a0 2023-10-16T12:30:18.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.556+0000 7fd73affd700 1 -- 172.21.15.16:0/2727717064 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 74) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd740003a00 con 0x7fd74c0cf4a0 2023-10-16T12:30:18.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.557+0000 7fd73affd700 1 --2- 172.21.15.16:0/2727717064 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7fd72c06b020 0x7fd72c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:18.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.557+0000 7fd73affd700 1 -- 172.21.15.16:0/2727717064 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 6093+0+0 (secure 0 0 0) 0x7fd740088d70 con 0x7fd74c0cf4a0 2023-10-16T12:30:18.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.557+0000 7fd749d9b700 1 --2- 172.21.15.16:0/2727717064 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7fd72c06b020 0x7fd72c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:18.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.557+0000 7fd749d9b700 1 --2- 172.21.15.16:0/2727717064 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7fd72c06b020 0x7fd72c06d4e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd73c000f60 tx=0x7fd73c000bc0 comp rx=0 tx=0).ready entity=mgr.8388 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:18.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.561+0000 7fd73affd700 1 -- 172.21.15.16:0/2727717064 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd74c1efb20 con 0x7fd74c0cf4a0 2023-10-16T12:30:18.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:18.781+0000 7fd750d2f700 1 -- 172.21.15.16:0/2727717064 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fd74c1efb20 con 0x7fd74c0cf4a0 2023-10-16T12:30:19.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.370+0000 7fd73affd700 1 -- 172.21.15.16:0/2727717064 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v136) v1 ==== 97+0+0 (secure 0 0 0) 0x7fd74c1efb20 con 0x7fd74c0cf4a0 2023-10-16T12:30:19.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.374+0000 7fd750d2f700 1 -- 172.21.15.16:0/2727717064 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7fd72c06b020 msgr2=0x7fd72c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:19.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.374+0000 7fd750d2f700 1 --2- 172.21.15.16:0/2727717064 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7fd72c06b020 0x7fd72c06d4e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd73c000f60 tx=0x7fd73c000bc0 comp rx=0 tx=0).stop 2023-10-16T12:30:19.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.374+0000 7fd750d2f700 1 -- 172.21.15.16:0/2727717064 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd74c0cf4a0 msgr2=0x7fd74c152040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:19.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.374+0000 7fd750d2f700 1 --2- 172.21.15.16:0/2727717064 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd74c0cf4a0 0x7fd74c152040 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7fd74000c520 tx=0x7fd74000c830 comp rx=0 tx=0).stop 2023-10-16T12:30:19.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.374+0000 7fd750d2f700 1 -- 172.21.15.16:0/2727717064 shutdown_connections 2023-10-16T12:30:19.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.374+0000 7fd750d2f700 1 --2- 172.21.15.16:0/2727717064 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd74c15d230 0x7fd74c0c4b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:19.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.374+0000 7fd750d2f700 1 --2- 172.21.15.16:0/2727717064 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd74c0cf4a0 0x7fd74c152040 unknown :-1 s=CLOSED pgs=1516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:19.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.374+0000 7fd750d2f700 1 --2- 172.21.15.16:0/2727717064 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7fd72c06b020 0x7fd72c06d4e0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:19.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.374+0000 7fd750d2f700 1 --2- 172.21.15.16:0/2727717064 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd74c0cf880 0x7fd74c0c45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:19.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.374+0000 7fd750d2f700 1 -- 172.21.15.16:0/2727717064 >> 172.21.15.16:0/2727717064 conn(0x7fd74c0b1fd0 msgr2=0x7fd74c158e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:19.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.374+0000 7fd750d2f700 1 -- 172.21.15.16:0/2727717064 shutdown_connections 2023-10-16T12:30:19.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.374+0000 7fd750d2f700 1 -- 172.21.15.16:0/2727717064 wait complete. 2023-10-16T12:30:19.380 INFO:teuthology.orchestra.run.smithi016.stderr: removed all blocklist entries 2023-10-16T12:30:19.393 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-16T12:30:19.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.523+0000 7f7e1fe4a700 1 Processor -- start 2023-10-16T12:30:19.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.524+0000 7f7e1fe4a700 1 -- start start 2023-10-16T12:30:19.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.525+0000 7f7e1fe4a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7e1815b630 0x7f7e180d0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:19.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.525+0000 7f7e1fe4a700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f7e180ce4f0 0x7f7e180cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:19.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.525+0000 7f7e1fe4a700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e180b0970 con 0x7f7e1815ba10 2023-10-16T12:30:19.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.525+0000 7f7e1fe4a700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e180cd690 con 0x7f7e180ce4f0 2023-10-16T12:30:19.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.525+0000 7f7e1fe4a700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e180cd970 con 0x7f7e1815b630 2023-10-16T12:30:19.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.525+0000 7f7e1dbe6700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7e1815b630 0x7f7e180d0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:19.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.525+0000 7f7e1e3e7700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f7e180ce4f0 0x7f7e180cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:19.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.525+0000 7f7e1d3e5700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7f7e1815ba10 0x7f7e180cdde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:41514/0 (socket says 172.21.15.16:41514) 2023-10-16T12:30:19.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.525+0000 7f7e1d3e5700 1 -- 172.21.15.16:0/3655208752 learned_addr learned my addr 172.21.15.16:0/3655208752 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:30:19.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.525+0000 7f7e1e3e7700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f7e180ce4f0 0x7f7e180cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:44124/0 (socket says 172.21.15.16:44124) 2023-10-16T12:30:19.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.526+0000 7f7e1dbe6700 1 -- 172.21.15.16:0/3655208752 >> v2:172.21.15.49:3300/0 conn(0x7f7e180ce4f0 msgr2=0x7f7e180cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:19.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.526+0000 7f7e1dbe6700 1 --2- 172.21.15.16:0/3655208752 >> v2:172.21.15.49:3300/0 conn(0x7f7e180ce4f0 0x7f7e180cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:19.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.526+0000 7f7e1dbe6700 1 -- 172.21.15.16:0/3655208752 >> v1:172.21.15.49:6789/0 conn(0x7f7e1815ba10 legacy=0x7f7e180cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:19.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.526+0000 7f7e1dbe6700 1 -- 172.21.15.16:0/3655208752 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e180b0c30 con 0x7f7e1815b630 2023-10-16T12:30:19.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.526+0000 7f7e1e3e7700 1 --2- 172.21.15.16:0/3655208752 >> v2:172.21.15.49:3300/0 conn(0x7f7e180ce4f0 0x7f7e180cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:30:19.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.526+0000 7f7e1dbe6700 1 --2- 172.21.15.16:0/3655208752 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7e1815b630 0x7f7e180d0380 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7e08009840 tx=0x7f7e08009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=daef3de1b5d5fa64 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:19.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.526+0000 7f7e1cbe4700 1 -- 172.21.15.16:0/3655208752 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e08011830 con 0x7f7e1815b630 2023-10-16T12:30:19.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.526+0000 7f7e1cbe4700 1 -- 172.21.15.16:0/3655208752 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e08011990 con 0x7f7e1815b630 2023-10-16T12:30:19.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.526+0000 7f7e1cbe4700 1 -- 172.21.15.16:0/3655208752 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e08011d20 con 0x7f7e1815b630 2023-10-16T12:30:19.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.527+0000 7f7e1fe4a700 1 -- 172.21.15.16:0/3655208752 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7e1815b630 msgr2=0x7f7e180d0380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:19.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.527+0000 7f7e1fe4a700 1 --2- 172.21.15.16:0/3655208752 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7e1815b630 0x7f7e180d0380 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7e08009840 tx=0x7f7e08009b50 comp rx=0 tx=0).stop 2023-10-16T12:30:19.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.527+0000 7f7e1fe4a700 1 -- 172.21.15.16:0/3655208752 shutdown_connections 2023-10-16T12:30:19.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.527+0000 7f7e1fe4a700 1 --2- 172.21.15.16:0/3655208752 >> v2:172.21.15.49:3300/0 conn(0x7f7e180ce4f0 0x7f7e180cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:19.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.527+0000 7f7e1fe4a700 1 --2- 172.21.15.16:0/3655208752 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7e1815b630 0x7f7e180d0380 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:19.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.527+0000 7f7e1fe4a700 1 -- 172.21.15.16:0/3655208752 >> 172.21.15.16:0/3655208752 conn(0x7f7e180d14b0 msgr2=0x7f7e180d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:19.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.527+0000 7f7e1fe4a700 1 -- 172.21.15.16:0/3655208752 shutdown_connections 2023-10-16T12:30:19.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.527+0000 7f7e1fe4a700 1 -- 172.21.15.16:0/3655208752 wait complete. 2023-10-16T12:30:19.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.528+0000 7f7e1fe4a700 1 Processor -- start 2023-10-16T12:30:19.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.528+0000 7f7e1fe4a700 1 -- start start 2023-10-16T12:30:19.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.529+0000 7f7e1fe4a700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7e180ce4f0 0x7f7e180c78a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:19.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.529+0000 7f7e1fe4a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7e1815b630 0x7f7e1815cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:19.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.529+0000 7f7e1fe4a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e1815ba10 0x7f7e1815d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:19.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.529+0000 7f7e1fe4a700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e180c8580 con 0x7f7e1815ba10 2023-10-16T12:30:19.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.529+0000 7f7e1fe4a700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e180c7de0 con 0x7f7e180ce4f0 2023-10-16T12:30:19.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.529+0000 7f7e1fe4a700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e180c5f90 con 0x7f7e1815b630 2023-10-16T12:30:19.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.529+0000 7f7e1e3e7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e1815ba10 0x7f7e1815d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:19.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.529+0000 7f7e1e3e7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e1815ba10 0x7f7e1815d310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45068/0 (socket says 172.21.15.16:45068) 2023-10-16T12:30:19.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.529+0000 7f7e1e3e7700 1 -- 172.21.15.16:0/882584589 learned_addr learned my addr 172.21.15.16:0/882584589 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:19.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.529+0000 7f7e1dbe6700 1 --2- 172.21.15.16:0/882584589 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7e180ce4f0 0x7f7e180c78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:19.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.529+0000 7f7e1d3e5700 1 --2- 172.21.15.16:0/882584589 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7e1815b630 0x7f7e1815cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:19.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.529+0000 7f7e1e3e7700 1 -- 172.21.15.16:0/882584589 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7e1815b630 msgr2=0x7f7e1815cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:19.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.529+0000 7f7e1e3e7700 1 --2- 172.21.15.16:0/882584589 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7e1815b630 0x7f7e1815cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:19.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.529+0000 7f7e1e3e7700 1 -- 172.21.15.16:0/882584589 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7e180ce4f0 msgr2=0x7f7e180c78a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:19.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.529+0000 7f7e1e3e7700 1 --2- 172.21.15.16:0/882584589 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7e180ce4f0 0x7f7e180c78a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:19.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.529+0000 7f7e1e3e7700 1 -- 172.21.15.16:0/882584589 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e10009a70 con 0x7f7e1815ba10 2023-10-16T12:30:19.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.529+0000 7f7e1e3e7700 1 --2- 172.21.15.16:0/882584589 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e1815ba10 0x7f7e1815d310 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f7e1000efb0 tx=0x7f7e1000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:19.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.530+0000 7f7e0e7fc700 1 -- 172.21.15.16:0/882584589 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e1000cc90 con 0x7f7e1815ba10 2023-10-16T12:30:19.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.530+0000 7f7e0e7fc700 1 -- 172.21.15.16:0/882584589 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e1000f040 con 0x7f7e1815ba10 2023-10-16T12:30:19.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.530+0000 7f7e0e7fc700 1 -- 172.21.15.16:0/882584589 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e10014480 con 0x7f7e1815ba10 2023-10-16T12:30:19.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.530+0000 7f7e1fe4a700 1 -- 172.21.15.16:0/882584589 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e080094a0 con 0x7f7e1815ba10 2023-10-16T12:30:19.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.530+0000 7f7e1fe4a700 1 -- 172.21.15.16:0/882584589 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e180c6550 con 0x7f7e1815ba10 2023-10-16T12:30:19.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.531+0000 7f7e1fe4a700 1 -- 172.21.15.16:0/882584589 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e1804ebb0 con 0x7f7e1815ba10 2023-10-16T12:30:19.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.531+0000 7f7e0e7fc700 1 -- 172.21.15.16:0/882584589 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 74) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7e100145e0 con 0x7f7e1815ba10 2023-10-16T12:30:19.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.531+0000 7f7e0e7fc700 1 --2- 172.21.15.16:0/882584589 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f7dfc06b020 0x7f7dfc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:19.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.532+0000 7f7e0e7fc700 1 -- 172.21.15.16:0/882584589 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 5233+0+0 (secure 0 0 0) 0x7f7e1008a330 con 0x7f7e1815ba10 2023-10-16T12:30:19.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.532+0000 7f7e1dbe6700 1 --2- 172.21.15.16:0/882584589 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f7dfc06b020 0x7f7dfc06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:19.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.532+0000 7f7e1dbe6700 1 --2- 172.21.15.16:0/882584589 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f7dfc06b020 0x7f7dfc06d4e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7e08000c00 tx=0x7f7e08008030 comp rx=0 tx=0).ready entity=mgr.8388 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:19.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.536+0000 7f7e0e7fc700 1 -- 172.21.15.16:0/882584589 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7e10055380 con 0x7f7e1815ba10 2023-10-16T12:30:19.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.820+0000 7f7e1fe4a700 1 -- 172.21.15.16:0/882584589 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f7e180c4d70 con 0x7f7e1815ba10 2023-10-16T12:30:19.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.821+0000 7f7e0e7fc700 1 -- 172.21.15.16:0/882584589 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v23) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f7e10009d50 con 0x7f7e1815ba10 2023-10-16T12:30:19.823 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:30:19.823 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-10-16T12:30:19.823 INFO:teuthology.orchestra.run.smithi016.stdout: "auth_dump": [ 2023-10-16T12:30:19.824 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:30:19.824 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.a", 2023-10-16T12:30:19.824 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOKC1l/rBtOhAA/ob5WUvpz8Glp4E+xPZOYA==", 2023-10-16T12:30:19.824 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:30:19.824 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T12:30:19.824 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-10-16T12:30:19.825 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-10-16T12:30:19.825 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:30:19.825 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:30:19.825 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:30:19.825 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:30:19.826 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.b", 2023-10-16T12:30:19.826 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1lKAVsCxAAvhXuLY6i1+Iz1yq5BVPMNQ==", 2023-10-16T12:30:19.826 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:30:19.826 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T12:30:19.826 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-10-16T12:30:19.827 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-10-16T12:30:19.827 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:30:19.827 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:30:19.827 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:30:19.827 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:30:19.827 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.c", 2023-10-16T12:30:19.828 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1lvBowBhAA3waJNYhdgs8PGCYqYNYUog==", 2023-10-16T12:30:19.828 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:30:19.828 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T12:30:19.828 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-10-16T12:30:19.828 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-10-16T12:30:19.829 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:30:19.829 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:30:19.829 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:30:19.829 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:30:19.829 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.d", 2023-10-16T12:30:19.829 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1lS/fnEhAAfNYgQ5BW86JMjYm/bBfnaQ==", 2023-10-16T12:30:19.830 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:30:19.830 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T12:30:19.830 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-10-16T12:30:19.830 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-10-16T12:30:19.830 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:30:19.831 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:30:19.831 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:30:19.831 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:30:19.831 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.0", 2023-10-16T12:30:19.831 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAQKC1lbxllOhAAZeLdp2GMl6HqQPaQjKlNqw==", 2023-10-16T12:30:19.832 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:30:19.832 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T12:30:19.832 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T12:30:19.832 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:30:19.832 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:30:19.832 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:30:19.833 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:30:19.833 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.1", 2023-10-16T12:30:19.833 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQATKC1lnU6KOhAAHxvlbmzBptf1keGgL+Rbkg==", 2023-10-16T12:30:19.833 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:30:19.833 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T12:30:19.834 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T12:30:19.834 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:30:19.834 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:30:19.834 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:30:19.834 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:30:19.834 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.2", 2023-10-16T12:30:19.835 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAXKC1l9jFyARAAr/NAwtWhBMut9sAAoDEGWg==", 2023-10-16T12:30:19.835 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:30:19.835 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T12:30:19.835 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T12:30:19.835 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:30:19.836 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:30:19.836 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:30:19.836 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:30:19.836 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.3", 2023-10-16T12:30:19.836 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAaKC1lbw/lARAAGJ/9DNb5/7kLFSq7vrLtXg==", 2023-10-16T12:30:19.837 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:30:19.837 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T12:30:19.837 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T12:30:19.837 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:30:19.837 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:30:19.838 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:30:19.838 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:30:19.838 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.4", 2023-10-16T12:30:19.838 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAeKC1lZb7jERAAYpvb1xcSli9CTq7L8ChLOA==", 2023-10-16T12:30:19.838 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:30:19.838 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T12:30:19.839 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T12:30:19.839 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:30:19.839 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:30:19.839 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:30:19.839 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:30:19.840 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.5", 2023-10-16T12:30:19.840 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAhKC1l/4HuEBAAqfnRqjknij9KES/fqiyIBQ==", 2023-10-16T12:30:19.840 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:30:19.840 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T12:30:19.840 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T12:30:19.841 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:30:19.841 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:30:19.841 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:30:19.841 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:30:19.841 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.6", 2023-10-16T12:30:19.842 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAkKC1lzoUkEBAA+Q82OiBMNag8+R316WQtUw==", 2023-10-16T12:30:19.842 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:30:19.842 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T12:30:19.842 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T12:30:19.842 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:30:19.842 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:30:19.843 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:30:19.843 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:30:19.843 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.7", 2023-10-16T12:30:19.843 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAnKC1ljUu4DxAAc+G4anJANC2GhLJ8wb6gYg==", 2023-10-16T12:30:19.843 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:30:19.844 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T12:30:19.844 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T12:30:19.844 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:30:19.844 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:30:19.844 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:30:19.845 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:30:19.845 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.0", 2023-10-16T12:30:19.845 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1l2l6PFxAA9QyXtBAHdGAXpwH0JfknuQ==", 2023-10-16T12:30:19.845 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:30:19.845 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T12:30:19.845 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow r", 2023-10-16T12:30:19.846 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-16T12:30:19.846 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:30:19.846 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:30:19.846 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:30:19.846 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.1", 2023-10-16T12:30:19.847 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1lAcHFIxAAcNt2bmv2R6j5L3DYkdw7eQ==", 2023-10-16T12:30:19.847 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:30:19.847 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T12:30:19.847 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-10-16T12:30:19.847 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-10-16T12:30:19.847 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-10-16T12:30:19.848 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:30:19.848 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:30:19.848 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:30:19.848 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.2", 2023-10-16T12:30:19.848 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1lA6wnGxAAdtaqYhtIJu8qENMrD8xvVA==", 2023-10-16T12:30:19.849 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:30:19.849 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T12:30:19.849 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-10-16T12:30:19.849 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-10-16T12:30:19.849 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-10-16T12:30:19.849 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:30:19.850 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:30:19.850 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:30:19.850 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.3", 2023-10-16T12:30:19.850 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1l/ddAJxAA3fNKKn9XkXas1BlDN+ggjw==", 2023-10-16T12:30:19.850 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:30:19.851 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T12:30:19.851 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-10-16T12:30:19.851 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-10-16T12:30:19.851 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-10-16T12:30:19.851 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:30:19.851 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:30:19.852 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:30:19.852 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.admin", 2023-10-16T12:30:19.852 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOKC1ld5jUHBAAc6esnCce3dSEZn3wVjhIlQ==", 2023-10-16T12:30:19.852 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:30:19.852 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2023-10-16T12:30:19.853 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-10-16T12:30:19.853 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-10-16T12:30:19.853 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:30:19.853 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:30:19.853 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:30:19.853 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:30:19.854 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mgr.x", 2023-10-16T12:30:19.854 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOKC1l6b3vNhAAmj5xbfid2sZM2KfHmAf1Wg==", 2023-10-16T12:30:19.854 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:30:19.854 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2023-10-16T12:30:19.854 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile mgr", 2023-10-16T12:30:19.855 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:30:19.855 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:30:19.855 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:30:19.855 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:30:19.855 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mgr.y", 2023-10-16T12:30:19.855 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOKC1lLz2nMhAA8NqFYZ5nDKkzPBMrkwsVuQ==", 2023-10-16T12:30:19.856 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:30:19.856 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2023-10-16T12:30:19.856 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile mgr", 2023-10-16T12:30:19.856 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:30:19.856 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:30:19.857 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:30:19.857 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-10-16T12:30:19.857 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-10-16T12:30:19.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.823+0000 7f7e1fe4a700 1 -- 172.21.15.16:0/882584589 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f7dfc06b020 msgr2=0x7f7dfc06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:19.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.823+0000 7f7e1fe4a700 1 --2- 172.21.15.16:0/882584589 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f7dfc06b020 0x7f7dfc06d4e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7e08000c00 tx=0x7f7e08008030 comp rx=0 tx=0).stop 2023-10-16T12:30:19.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.823+0000 7f7e1fe4a700 1 -- 172.21.15.16:0/882584589 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e1815ba10 msgr2=0x7f7e1815d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:19.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.823+0000 7f7e1fe4a700 1 --2- 172.21.15.16:0/882584589 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e1815ba10 0x7f7e1815d310 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f7e1000efb0 tx=0x7f7e1000c3a0 comp rx=0 tx=0).stop 2023-10-16T12:30:19.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.823+0000 7f7e1fe4a700 1 -- 172.21.15.16:0/882584589 shutdown_connections 2023-10-16T12:30:19.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.823+0000 7f7e1fe4a700 1 --2- 172.21.15.16:0/882584589 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e1815ba10 0x7f7e1815d310 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:19.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.823+0000 7f7e1fe4a700 1 --2- 172.21.15.16:0/882584589 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f7dfc06b020 0x7f7dfc06d4e0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:19.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.823+0000 7f7e1fe4a700 1 --2- 172.21.15.16:0/882584589 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7e180ce4f0 0x7f7e180c78a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:19.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.823+0000 7f7e1fe4a700 1 --2- 172.21.15.16:0/882584589 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7e1815b630 0x7f7e1815cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:19.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.823+0000 7f7e1fe4a700 1 -- 172.21.15.16:0/882584589 >> 172.21.15.16:0/882584589 conn(0x7f7e180d14b0 msgr2=0x7f7e1815a940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:19.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.823+0000 7f7e1fe4a700 1 -- 172.21.15.16:0/882584589 shutdown_connections 2023-10-16T12:30:19.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.823+0000 7f7e1fe4a700 1 -- 172.21.15.16:0/882584589 wait complete. 2023-10-16T12:30:19.861 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:30:19.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.985+0000 7f9b7cdda700 1 Processor -- start 2023-10-16T12:30:19.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.985+0000 7f9b7cdda700 1 -- start start 2023-10-16T12:30:19.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.986+0000 7f9b7cdda700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9b7815b280 0x7f9b781613e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:19.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.986+0000 7f9b7cdda700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f9b7806a7e0 0x7f9b7806ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:19.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.986+0000 7f9b7cdda700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b7806b150 con 0x7f9b780cc8a0 2023-10-16T12:30:19.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.986+0000 7f9b7cdda700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b78161c30 con 0x7f9b7815b280 2023-10-16T12:30:19.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.986+0000 7f9b7cdda700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b78161f10 con 0x7f9b7806a7e0 2023-10-16T12:30:19.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.986+0000 7f9b7659c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9b7815b280 0x7f9b781613e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:19.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.986+0000 7f9b7659c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9b7815b280 0x7f9b781613e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45072/0 (socket says 172.21.15.16:45072) 2023-10-16T12:30:19.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.986+0000 7f9b7659c700 1 -- 172.21.15.16:0/9968847 learned_addr learned my addr 172.21.15.16:0/9968847 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:19.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.986+0000 7f9b7659c700 1 -- 172.21.15.16:0/9968847 >> v2:172.21.15.49:3300/0 conn(0x7f9b7806a7e0 msgr2=0x7f9b7806ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:19.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.986+0000 7f9b76d9d700 1 --2- 172.21.15.16:0/9968847 >> v2:172.21.15.49:3300/0 conn(0x7f9b7806a7e0 0x7f9b7806ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:19.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.986+0000 7f9b7659c700 1 --2- 172.21.15.16:0/9968847 >> v2:172.21.15.49:3300/0 conn(0x7f9b7806a7e0 0x7f9b7806ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:19.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.986+0000 7f9b7659c700 1 -- 172.21.15.16:0/9968847 >> v1:172.21.15.49:6789/0 conn(0x7f9b780cc8a0 legacy=0x7f9b7815e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:19.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.986+0000 7f9b7659c700 1 -- 172.21.15.16:0/9968847 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b781621f0 con 0x7f9b7815b280 2023-10-16T12:30:19.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.986+0000 7f9b76d9d700 1 --2- 172.21.15.16:0/9968847 >> v2:172.21.15.49:3300/0 conn(0x7f9b7806a7e0 0x7f9b7806ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:30:19.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.987+0000 7f9b7659c700 1 --2- 172.21.15.16:0/9968847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b7815b280 0x7f9b781613e0 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f9b780cbed0 tx=0x7f9b6c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc2470570808d2eb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:19.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.987+0000 7f9b7559a700 1 -- 172.21.15.16:0/9968847 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b6c004230 con 0x7f9b7815b280 2023-10-16T12:30:19.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.987+0000 7f9b7559a700 1 -- 172.21.15.16:0/9968847 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b6c004390 con 0x7f9b7815b280 2023-10-16T12:30:19.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.987+0000 7f9b7559a700 1 -- 172.21.15.16:0/9968847 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b6c0036a0 con 0x7f9b7815b280 2023-10-16T12:30:19.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.987+0000 7f9b7cdda700 1 -- 172.21.15.16:0/9968847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b7815b280 msgr2=0x7f9b781613e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:19.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.987+0000 7f9b7cdda700 1 --2- 172.21.15.16:0/9968847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b7815b280 0x7f9b781613e0 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f9b780cbed0 tx=0x7f9b6c009bd0 comp rx=0 tx=0).stop 2023-10-16T12:30:19.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.987+0000 7f9b7cdda700 1 -- 172.21.15.16:0/9968847 shutdown_connections 2023-10-16T12:30:19.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.987+0000 7f9b7cdda700 1 --2- 172.21.15.16:0/9968847 >> v2:172.21.15.49:3300/0 conn(0x7f9b7806a7e0 0x7f9b7806ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:19.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.987+0000 7f9b7cdda700 1 --2- 172.21.15.16:0/9968847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b7815b280 0x7f9b781613e0 unknown :-1 s=CLOSED pgs=1518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:19.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.987+0000 7f9b7cdda700 1 -- 172.21.15.16:0/9968847 >> 172.21.15.16:0/9968847 conn(0x7f9b780b1fd0 msgr2=0x7f9b780b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:19.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.987+0000 7f9b7cdda700 1 -- 172.21.15.16:0/9968847 shutdown_connections 2023-10-16T12:30:19.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.987+0000 7f9b7cdda700 1 -- 172.21.15.16:0/9968847 wait complete. 2023-10-16T12:30:19.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.989+0000 7f9b7cdda700 1 Processor -- start 2023-10-16T12:30:19.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.989+0000 7f9b7cdda700 1 -- start start 2023-10-16T12:30:19.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.989+0000 7f9b7cdda700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9b7806a7e0 0x7f9b780ca110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:19.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.989+0000 7f9b7cdda700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b780cc8a0 0x7f9b780c7c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:19.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.989+0000 7f9b7cdda700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9b7815b280 0x7f9b78072030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:19.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.989+0000 7f9b7cdda700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b780c81d0 con 0x7f9b780cc8a0 2023-10-16T12:30:19.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.989+0000 7f9b7cdda700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b78149540 con 0x7f9b7806a7e0 2023-10-16T12:30:19.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.989+0000 7f9b7cdda700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b78149820 con 0x7f9b7815b280 2023-10-16T12:30:19.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.990+0000 7f9b75d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b780cc8a0 0x7f9b780c7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:19.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.990+0000 7f9b75d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b780cc8a0 0x7f9b780c7c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45080/0 (socket says 172.21.15.16:45080) 2023-10-16T12:30:19.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.990+0000 7f9b75d9b700 1 -- 172.21.15.16:0/629845732 learned_addr learned my addr 172.21.15.16:0/629845732 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:19.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.990+0000 7f9b7659c700 1 --2- 172.21.15.16:0/629845732 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9b7806a7e0 0x7f9b780ca110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:19.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.990+0000 7f9b76d9d700 1 --2- 172.21.15.16:0/629845732 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9b7815b280 0x7f9b78072030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:20.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.990+0000 7f9b75d9b700 1 -- 172.21.15.16:0/629845732 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9b7815b280 msgr2=0x7f9b78072030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:20.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.990+0000 7f9b75d9b700 1 --2- 172.21.15.16:0/629845732 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9b7815b280 0x7f9b78072030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:20.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.990+0000 7f9b75d9b700 1 -- 172.21.15.16:0/629845732 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9b7806a7e0 msgr2=0x7f9b780ca110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:20.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.990+0000 7f9b75d9b700 1 --2- 172.21.15.16:0/629845732 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9b7806a7e0 0x7f9b780ca110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:20.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.990+0000 7f9b75d9b700 1 -- 172.21.15.16:0/629845732 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b6c009550 con 0x7f9b780cc8a0 2023-10-16T12:30:20.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.990+0000 7f9b7659c700 1 --2- 172.21.15.16:0/629845732 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9b7806a7e0 0x7f9b780ca110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:30:20.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.990+0000 7f9b75d9b700 1 --2- 172.21.15.16:0/629845732 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b780cc8a0 0x7f9b780c7c90 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f9b6800c4a0 tx=0x7f9b6800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:20.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.990+0000 7f9b66ffd700 1 -- 172.21.15.16:0/629845732 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b68011790 con 0x7f9b780cc8a0 2023-10-16T12:30:20.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.990+0000 7f9b66ffd700 1 -- 172.21.15.16:0/629845732 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b680118f0 con 0x7f9b780cc8a0 2023-10-16T12:30:20.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.990+0000 7f9b7cdda700 1 -- 172.21.15.16:0/629845732 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b78149b00 con 0x7f9b780cc8a0 2023-10-16T12:30:20.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.990+0000 7f9b66ffd700 1 -- 172.21.15.16:0/629845732 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b68011ca0 con 0x7f9b780cc8a0 2023-10-16T12:30:20.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.990+0000 7f9b7cdda700 1 -- 172.21.15.16:0/629845732 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b780c7730 con 0x7f9b780cc8a0 2023-10-16T12:30:20.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.991+0000 7f9b7cdda700 1 -- 172.21.15.16:0/629845732 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b7804ebb0 con 0x7f9b780cc8a0 2023-10-16T12:30:20.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.992+0000 7f9b66ffd700 1 -- 172.21.15.16:0/629845732 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 74) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9b6801c460 con 0x7f9b780cc8a0 2023-10-16T12:30:20.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.992+0000 7f9b66ffd700 1 --2- 172.21.15.16:0/629845732 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f9b5806b020 0x7f9b5806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:20.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.992+0000 7f9b66ffd700 1 -- 172.21.15.16:0/629845732 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 5233+0+0 (secure 0 0 0) 0x7f9b68089470 con 0x7f9b780cc8a0 2023-10-16T12:30:20.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.992+0000 7f9b7659c700 1 --2- 172.21.15.16:0/629845732 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f9b5806b020 0x7f9b5806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:20.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.993+0000 7f9b7659c700 1 --2- 172.21.15.16:0/629845732 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f9b5806b020 0x7f9b5806d4e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9b6c009130 tx=0x7f9b6c008030 comp rx=0 tx=0).ready entity=mgr.8388 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:20.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:19.997+0000 7f9b66ffd700 1 -- 172.21.15.16:0/629845732 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9b68054550 con 0x7f9b780cc8a0 2023-10-16T12:30:20.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.201+0000 7f9b7cdda700 1 -- 172.21.15.16:0/629845732 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9b780c36e0 con 0x7f9b780cc8a0 2023-10-16T12:30:20.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.202+0000 7f9b66ffd700 1 -- 172.21.15.16:0/629845732 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v138) v1 ==== 74+0+13052 (secure 0 0 0) 0x7f9b68057b70 con 0x7f9b780cc8a0 2023-10-16T12:30:20.204 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:30:20.205 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":138,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:30:19.378419+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:30:20.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.204+0000 7f9b7cdda700 1 -- 172.21.15.16:0/629845732 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f9b5806b020 msgr2=0x7f9b5806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:20.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.204+0000 7f9b7cdda700 1 --2- 172.21.15.16:0/629845732 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f9b5806b020 0x7f9b5806d4e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9b6c009130 tx=0x7f9b6c008030 comp rx=0 tx=0).stop 2023-10-16T12:30:20.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.204+0000 7f9b7cdda700 1 -- 172.21.15.16:0/629845732 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b780cc8a0 msgr2=0x7f9b780c7c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:20.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.204+0000 7f9b7cdda700 1 --2- 172.21.15.16:0/629845732 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b780cc8a0 0x7f9b780c7c90 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f9b6800c4a0 tx=0x7f9b6800c860 comp rx=0 tx=0).stop 2023-10-16T12:30:20.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.204+0000 7f9b7cdda700 1 -- 172.21.15.16:0/629845732 shutdown_connections 2023-10-16T12:30:20.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.204+0000 7f9b7cdda700 1 --2- 172.21.15.16:0/629845732 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9b7815b280 0x7f9b78072030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:20.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.204+0000 7f9b7cdda700 1 --2- 172.21.15.16:0/629845732 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f9b5806b020 0x7f9b5806d4e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:20.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.204+0000 7f9b7cdda700 1 --2- 172.21.15.16:0/629845732 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9b7806a7e0 0x7f9b780ca110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:20.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.204+0000 7f9b7cdda700 1 --2- 172.21.15.16:0/629845732 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b780cc8a0 0x7f9b780c7c90 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:20.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.204+0000 7f9b7cdda700 1 -- 172.21.15.16:0/629845732 >> 172.21.15.16:0/629845732 conn(0x7f9b780b1fd0 msgr2=0x7f9b780cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:20.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.205+0000 7f9b7cdda700 1 -- 172.21.15.16:0/629845732 shutdown_connections 2023-10-16T12:30:20.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.205+0000 7f9b7cdda700 1 -- 172.21.15.16:0/629845732 wait complete. 2023-10-16T12:30:20.219 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T12:30:20.220 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:30:20.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.346+0000 7febac04e700 1 Processor -- start 2023-10-16T12:30:20.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.347+0000 7febac04e700 1 -- start start 2023-10-16T12:30:20.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.347+0000 7febac04e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7feba406e170 0x7feba40cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:20.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.347+0000 7febac04e700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feba40cef70 con 0x7feba40cb880 2023-10-16T12:30:20.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.347+0000 7febac04e700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feba40ce360 con 0x7feba415b070 2023-10-16T12:30:20.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.347+0000 7febac04e700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feba40ce640 con 0x7feba406e170 2023-10-16T12:30:20.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.347+0000 7feba95e9700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7feba40cb880 0x7feba406da60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34362/0 (socket says 172.21.15.16:34362) 2023-10-16T12:30:20.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.347+0000 7feba95e9700 1 -- 172.21.15.16:0/1281637191 learned_addr learned my addr 172.21.15.16:0/1281637191 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:30:20.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.348+0000 7feba8de8700 1 -- 172.21.15.16:0/1281637191 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3147670281 0 0) 0x7feba40cef70 con 0x7feba40cb880 2023-10-16T12:30:20.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.348+0000 7feba8de8700 1 -- 172.21.15.16:0/1281637191 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb900035b0 con 0x7feba40cb880 2023-10-16T12:30:20.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.348+0000 7febaa5eb700 1 --2- 172.21.15.16:0/1281637191 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7feba406e170 0x7feba40cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:20.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.348+0000 7feba8de8700 1 -- 172.21.15.16:0/1281637191 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2050763725 0 0) 0x7feb900035b0 con 0x7feba40cb880 2023-10-16T12:30:20.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.348+0000 7feba8de8700 1 -- 172.21.15.16:0/1281637191 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7feba406e170 msgr2=0x7feba40cbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:20.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.348+0000 7feba8de8700 1 --2- 172.21.15.16:0/1281637191 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7feba406e170 0x7feba40cbd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:20.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.348+0000 7feba8de8700 1 -- 172.21.15.16:0/1281637191 >> v1:172.21.15.49:6789/0 conn(0x7feba415b070 legacy=0x7feba415b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:20.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.348+0000 7feba8de8700 1 -- 172.21.15.16:0/1281637191 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feba40ce920 con 0x7feba40cb880 2023-10-16T12:30:20.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.348+0000 7feba8de8700 1 -- 172.21.15.16:0/1281637191 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7feba0001670 con 0x7feba40cb880 2023-10-16T12:30:20.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.348+0000 7feba8de8700 1 -- 172.21.15.16:0/1281637191 >> v1:172.21.15.16:6789/0 conn(0x7feba40cb880 legacy=0x7feba406da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:20.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.348+0000 7feba8de8700 1 --2- 172.21.15.16:0/1281637191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb90003a30 0x7feb90003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:20.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.348+0000 7feba8de8700 1 --2- 172.21.15.16:0/1281637191 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feb90004850 0x7feb90004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:20.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.348+0000 7feba8de8700 1 --2- 172.21.15.16:0/1281637191 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7feba406e170 0x7feb9000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:20.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.348+0000 7feba8de8700 1 -- 172.21.15.16:0/1281637191 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feba40ce640 con 0x7feb90003a30 2023-10-16T12:30:20.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.348+0000 7feba8de8700 1 -- 172.21.15.16:0/1281637191 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb900055a0 con 0x7feb90004850 2023-10-16T12:30:20.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.348+0000 7feba8de8700 1 -- 172.21.15.16:0/1281637191 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb900056e0 con 0x7feba406e170 2023-10-16T12:30:20.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.348+0000 7feba9dea700 1 --2- 172.21.15.16:0/1281637191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb90003a30 0x7feb90003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:20.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.348+0000 7feba95e9700 1 --2- 172.21.15.16:0/1281637191 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feb90004850 0x7feb90004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:20.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.348+0000 7febaa5eb700 1 --2- 172.21.15.16:0/1281637191 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7feba406e170 0x7feb9000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:20.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.348+0000 7feba9dea700 1 -- 172.21.15.16:0/1281637191 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7feba406e170 msgr2=0x7feb9000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:20.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.349+0000 7feba9dea700 1 --2- 172.21.15.16:0/1281637191 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7feba406e170 0x7feb9000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:20.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.349+0000 7feba9dea700 1 -- 172.21.15.16:0/1281637191 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feb90004850 msgr2=0x7feb90004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:20.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.349+0000 7feba9dea700 1 --2- 172.21.15.16:0/1281637191 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feb90004850 0x7feb90004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:20.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.349+0000 7feba9dea700 1 -- 172.21.15.16:0/1281637191 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb900035b0 con 0x7feb90003a30 2023-10-16T12:30:20.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.349+0000 7feba9dea700 1 --2- 172.21.15.16:0/1281637191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb90003a30 0x7feb90003e50 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7feb94001830 tx=0x7feb90005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7959a589f3db72d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:20.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.349+0000 7feba8de8700 1 -- 172.21.15.16:0/1281637191 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb940012f0 con 0x7feb90003a30 2023-10-16T12:30:20.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.349+0000 7feba8de8700 1 -- 172.21.15.16:0/1281637191 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb940052f0 con 0x7feb90003a30 2023-10-16T12:30:20.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.349+0000 7feba8de8700 1 -- 172.21.15.16:0/1281637191 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb94006860 con 0x7feb90003a30 2023-10-16T12:30:20.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.349+0000 7febac04e700 1 -- 172.21.15.16:0/1281637191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb90003a30 msgr2=0x7feb90003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:20.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.349+0000 7febac04e700 1 --2- 172.21.15.16:0/1281637191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb90003a30 0x7feb90003e50 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7feb94001830 tx=0x7feb90005880 comp rx=0 tx=0).stop 2023-10-16T12:30:20.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.349+0000 7febaa5eb700 1 -- 172.21.15.16:0/1281637191 reap_dead start 2023-10-16T12:30:20.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.349+0000 7febac04e700 1 -- 172.21.15.16:0/1281637191 shutdown_connections 2023-10-16T12:30:20.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.349+0000 7febac04e700 1 -- 172.21.15.16:0/1281637191 >> 172.21.15.16:0/1281637191 conn(0x7feba40b1f40 msgr2=0x7feba40b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:20.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.350+0000 7febac04e700 1 -- 172.21.15.16:0/1281637191 shutdown_connections 2023-10-16T12:30:20.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.350+0000 7febac04e700 1 -- 172.21.15.16:0/1281637191 wait complete. 2023-10-16T12:30:20.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.351+0000 7febac04e700 1 Processor -- start 2023-10-16T12:30:20.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.351+0000 7febac04e700 1 -- start start 2023-10-16T12:30:20.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.351+0000 7febac04e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feba4150900 0x7feba415cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:20.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.351+0000 7febac04e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7feba4152fb0 0x7feba40c9750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:20.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.351+0000 7febac04e700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feba4159370 0x7feba41597f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:20.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.352+0000 7febac04e700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feba415c5d0 con 0x7feba4150900 2023-10-16T12:30:20.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.352+0000 7febac04e700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feba4159d30 con 0x7feba4159370 2023-10-16T12:30:20.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.352+0000 7febac04e700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feba415cb50 con 0x7feba4152fb0 2023-10-16T12:30:20.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.352+0000 7feba9dea700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feba4150900 0x7feba415cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:20.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.352+0000 7feba9dea700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feba4150900 0x7feba415cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45098/0 (socket says 172.21.15.16:45098) 2023-10-16T12:30:20.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.352+0000 7feba9dea700 1 -- 172.21.15.16:0/2381689089 learned_addr learned my addr 172.21.15.16:0/2381689089 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:20.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.352+0000 7febaa5eb700 1 --2- 172.21.15.16:0/2381689089 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feba4159370 0x7feba41597f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:20.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.352+0000 7feba9dea700 1 -- 172.21.15.16:0/2381689089 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7feba4152fb0 msgr2=0x7feba40c9750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:20.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.352+0000 7feba9dea700 1 --2- 172.21.15.16:0/2381689089 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7feba4152fb0 0x7feba40c9750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:20.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.352+0000 7feba9dea700 1 -- 172.21.15.16:0/2381689089 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feba4159370 msgr2=0x7feba41597f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:20.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.352+0000 7feba9dea700 1 --2- 172.21.15.16:0/2381689089 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feba4159370 0x7feba41597f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:20.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.352+0000 7feba9dea700 1 -- 172.21.15.16:0/2381689089 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feba40cd480 con 0x7feba4150900 2023-10-16T12:30:20.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.352+0000 7febaa5eb700 1 --2- 172.21.15.16:0/2381689089 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feba4159370 0x7feba41597f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:30:20.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.352+0000 7feba9dea700 1 --2- 172.21.15.16:0/2381689089 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feba4150900 0x7feba415cfe0 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7feb94004000 tx=0x7feb94001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:20.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.352+0000 7feb9a7fc700 1 -- 172.21.15.16:0/2381689089 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb94006c70 con 0x7feba4150900 2023-10-16T12:30:20.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.352+0000 7feb9a7fc700 1 -- 172.21.15.16:0/2381689089 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb94006dd0 con 0x7feba4150900 2023-10-16T12:30:20.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.352+0000 7febac04e700 1 -- 172.21.15.16:0/2381689089 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feba40c9c90 con 0x7feba4150900 2023-10-16T12:30:20.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.352+0000 7feb9a7fc700 1 -- 172.21.15.16:0/2381689089 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb940038c0 con 0x7feba4150900 2023-10-16T12:30:20.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.352+0000 7febac04e700 1 -- 172.21.15.16:0/2381689089 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feba40ca210 con 0x7feba4150900 2023-10-16T12:30:20.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.353+0000 7febac04e700 1 -- 172.21.15.16:0/2381689089 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feba40ca430 con 0x7feba4150900 2023-10-16T12:30:20.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.354+0000 7feb9a7fc700 1 -- 172.21.15.16:0/2381689089 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 74) v1 ==== 89331+0+0 (secure 0 0 0) 0x7feb94003a20 con 0x7feba4150900 2023-10-16T12:30:20.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.354+0000 7feb9a7fc700 1 --2- 172.21.15.16:0/2381689089 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7feb8806b050 0x7feb8806d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:20.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.354+0000 7feba95e9700 1 --2- 172.21.15.16:0/2381689089 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7feb8806b050 0x7feb8806d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:20.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.354+0000 7feb9a7fc700 1 -- 172.21.15.16:0/2381689089 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 5233+0+0 (secure 0 0 0) 0x7feb9408a3d0 con 0x7feba4150900 2023-10-16T12:30:20.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.355+0000 7feba95e9700 1 --2- 172.21.15.16:0/2381689089 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7feb8806b050 0x7feb8806d510 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7feba0001200 tx=0x7feba0001130 comp rx=0 tx=0).ready entity=mgr.8388 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:20.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.359+0000 7feb9a7fc700 1 -- 172.21.15.16:0/2381689089 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7feb94055550 con 0x7feba4150900 2023-10-16T12:30:20.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.564+0000 7febac04e700 1 -- 172.21.15.16:0/2381689089 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7feba404ebb0 con 0x7feba4150900 2023-10-16T12:30:20.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.564+0000 7feb9a7fc700 1 -- 172.21.15.16:0/2381689089 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v138) v1 ==== 93+0+10 (secure 0 0 0) 0x7feb94058b70 con 0x7feba4150900 2023-10-16T12:30:20.567 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T12:30:20.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.566+0000 7febac04e700 1 -- 172.21.15.16:0/2381689089 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7feb8806b050 msgr2=0x7feb8806d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:20.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.566+0000 7febac04e700 1 --2- 172.21.15.16:0/2381689089 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7feb8806b050 0x7feb8806d510 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7feba0001200 tx=0x7feba0001130 comp rx=0 tx=0).stop 2023-10-16T12:30:20.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.566+0000 7febac04e700 1 -- 172.21.15.16:0/2381689089 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feba4150900 msgr2=0x7feba415cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:20.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.566+0000 7febac04e700 1 --2- 172.21.15.16:0/2381689089 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feba4150900 0x7feba415cfe0 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7feb94004000 tx=0x7feb94001120 comp rx=0 tx=0).stop 2023-10-16T12:30:20.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.567+0000 7febac04e700 1 -- 172.21.15.16:0/2381689089 shutdown_connections 2023-10-16T12:30:20.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.567+0000 7febac04e700 1 --2- 172.21.15.16:0/2381689089 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7feb8806b050 0x7feb8806d510 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:20.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.567+0000 7febac04e700 1 --2- 172.21.15.16:0/2381689089 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feba4159370 0x7feba41597f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:20.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.567+0000 7febac04e700 1 --2- 172.21.15.16:0/2381689089 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feba4150900 0x7feba415cfe0 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:20.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.567+0000 7febac04e700 1 --2- 172.21.15.16:0/2381689089 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7feba4152fb0 0x7feba40c9750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:20.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.567+0000 7febac04e700 1 -- 172.21.15.16:0/2381689089 >> 172.21.15.16:0/2381689089 conn(0x7feba40b1f40 msgr2=0x7feba40c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:20.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.567+0000 7febac04e700 1 -- 172.21.15.16:0/2381689089 shutdown_connections 2023-10-16T12:30:20.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.567+0000 7febac04e700 1 -- 172.21.15.16:0/2381689089 wait complete. 2023-10-16T12:30:20.581 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-16T12:30:20.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.704+0000 7f48771f3700 1 Processor -- start 2023-10-16T12:30:20.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.704+0000 7f48771f3700 1 -- start start 2023-10-16T12:30:20.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.704+0000 7f48771f3700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f4870160eb0 0x7f487006d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:20.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.704+0000 7f48771f3700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f48700cab70 0x7f48700caff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:20.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.704+0000 7f48771f3700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f487006b1f0 0x7f487006b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:20.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.704+0000 7f48771f3700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48700af370 con 0x7f487006b1f0 2023-10-16T12:30:20.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.704+0000 7f48771f3700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48700cb530 con 0x7f4870160eb0 2023-10-16T12:30:20.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.704+0000 7f48771f3700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f487006c150 con 0x7f48700cab70 2023-10-16T12:30:20.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.705+0000 7f4875790700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f487006b1f0 0x7f487006b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:20.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.705+0000 7f4875790700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f487006b1f0 0x7f487006b6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45112/0 (socket says 172.21.15.16:45112) 2023-10-16T12:30:20.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.705+0000 7f4875790700 1 -- 172.21.15.16:0/2542294230 learned_addr learned my addr 172.21.15.16:0/2542294230 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:20.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.705+0000 7f486ffff700 1 --2- 172.21.15.16:0/2542294230 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f48700cab70 0x7f48700caff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:20.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.705+0000 7f4875790700 1 -- 172.21.15.16:0/2542294230 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f48700cab70 msgr2=0x7f48700caff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:20.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.705+0000 7f4875790700 1 --2- 172.21.15.16:0/2542294230 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f48700cab70 0x7f48700caff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:20.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.705+0000 7f4875790700 1 -- 172.21.15.16:0/2542294230 >> v2:172.21.15.49:3300/0 conn(0x7f4870160eb0 msgr2=0x7f487006d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:20.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.705+0000 7f4875790700 1 --2- 172.21.15.16:0/2542294230 >> v2:172.21.15.49:3300/0 conn(0x7f4870160eb0 0x7f487006d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:20.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.705+0000 7f4875790700 1 -- 172.21.15.16:0/2542294230 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f487006bbe0 con 0x7f487006b1f0 2023-10-16T12:30:20.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.705+0000 7f4875790700 1 --2- 172.21.15.16:0/2542294230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f487006b1f0 0x7f487006b6a0 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f48700cd6b0 tx=0x7f486800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88ca52e212e7f32f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:20.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.705+0000 7f486f7fe700 1 -- 172.21.15.16:0/2542294230 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f486800e070 con 0x7f487006b1f0 2023-10-16T12:30:20.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.705+0000 7f486f7fe700 1 -- 172.21.15.16:0/2542294230 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f486800b9c0 con 0x7f487006b1f0 2023-10-16T12:30:20.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.705+0000 7f486f7fe700 1 -- 172.21.15.16:0/2542294230 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f486800bd70 con 0x7f487006b1f0 2023-10-16T12:30:20.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.706+0000 7f48771f3700 1 -- 172.21.15.16:0/2542294230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f487006b1f0 msgr2=0x7f487006b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:20.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.706+0000 7f48771f3700 1 --2- 172.21.15.16:0/2542294230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f487006b1f0 0x7f487006b6a0 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f48700cd6b0 tx=0x7f486800b330 comp rx=0 tx=0).stop 2023-10-16T12:30:20.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.706+0000 7f48771f3700 1 -- 172.21.15.16:0/2542294230 shutdown_connections 2023-10-16T12:30:20.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.706+0000 7f48771f3700 1 --2- 172.21.15.16:0/2542294230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f487006b1f0 0x7f487006b6a0 unknown :-1 s=CLOSED pgs=1523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:20.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.706+0000 7f48771f3700 1 --2- 172.21.15.16:0/2542294230 >> v2:172.21.15.49:3300/0 conn(0x7f4870160eb0 0x7f487006d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:20.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.706+0000 7f48771f3700 1 --2- 172.21.15.16:0/2542294230 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f48700cab70 0x7f48700caff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:20.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.706+0000 7f48771f3700 1 -- 172.21.15.16:0/2542294230 >> 172.21.15.16:0/2542294230 conn(0x7f48700cc5b0 msgr2=0x7f48700cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:20.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.706+0000 7f48771f3700 1 -- 172.21.15.16:0/2542294230 shutdown_connections 2023-10-16T12:30:20.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.706+0000 7f48771f3700 1 -- 172.21.15.16:0/2542294230 wait complete. 2023-10-16T12:30:20.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.707+0000 7f48771f3700 1 Processor -- start 2023-10-16T12:30:20.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.707+0000 7f48771f3700 1 -- start start 2023-10-16T12:30:20.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.709+0000 7f48771f3700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f487006b1f0 0x7f48700c53a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:20.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.709+0000 7f48771f3700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f48700cab70 0x7f48700af610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:20.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.709+0000 7f48771f3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4870160eb0 0x7f48700afb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:20.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.709+0000 7f48771f3700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4870158d10 con 0x7f4870160eb0 2023-10-16T12:30:20.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.709+0000 7f48771f3700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48700c58e0 con 0x7f48700cab70 2023-10-16T12:30:20.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.709+0000 7f48771f3700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4870158090 con 0x7f487006b1f0 2023-10-16T12:30:20.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.709+0000 7f4875790700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4870160eb0 0x7f48700afb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:20.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.709+0000 7f4875790700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4870160eb0 0x7f48700afb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45118/0 (socket says 172.21.15.16:45118) 2023-10-16T12:30:20.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.709+0000 7f4875790700 1 -- 172.21.15.16:0/563742951 learned_addr learned my addr 172.21.15.16:0/563742951 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:20.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.709+0000 7f4874f8f700 1 --2- 172.21.15.16:0/563742951 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f487006b1f0 0x7f48700c53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:20.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.709+0000 7f486ffff700 1 --2- 172.21.15.16:0/563742951 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f48700cab70 0x7f48700af610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:20.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.709+0000 7f4875790700 1 -- 172.21.15.16:0/563742951 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f487006b1f0 msgr2=0x7f48700c53a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:20.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.709+0000 7f4875790700 1 --2- 172.21.15.16:0/563742951 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f487006b1f0 0x7f48700c53a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:20.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.709+0000 7f4875790700 1 -- 172.21.15.16:0/563742951 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f48700cab70 msgr2=0x7f48700af610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:20.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.709+0000 7f4875790700 1 --2- 172.21.15.16:0/563742951 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f48700cab70 0x7f48700af610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:20.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.709+0000 7f4875790700 1 -- 172.21.15.16:0/563742951 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4868009b30 con 0x7f4870160eb0 2023-10-16T12:30:20.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.709+0000 7f4875790700 1 --2- 172.21.15.16:0/563742951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4870160eb0 0x7f48700afb50 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f486800b370 tx=0x7f4868004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:20.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.709+0000 7f486d7fa700 1 -- 172.21.15.16:0/563742951 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4868004310 con 0x7f4870160eb0 2023-10-16T12:30:20.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.709+0000 7f486d7fa700 1 -- 172.21.15.16:0/563742951 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4868004470 con 0x7f4870160eb0 2023-10-16T12:30:20.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.709+0000 7f48771f3700 1 -- 172.21.15.16:0/563742951 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4870158370 con 0x7f4870160eb0 2023-10-16T12:30:20.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.709+0000 7f486d7fa700 1 -- 172.21.15.16:0/563742951 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4868004820 con 0x7f4870160eb0 2023-10-16T12:30:20.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.709+0000 7f48771f3700 1 -- 172.21.15.16:0/563742951 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4870158930 con 0x7f4870160eb0 2023-10-16T12:30:20.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.710+0000 7f48771f3700 1 -- 172.21.15.16:0/563742951 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4870158e50 con 0x7f4870160eb0 2023-10-16T12:30:20.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.711+0000 7f486d7fa700 1 -- 172.21.15.16:0/563742951 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 74) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f486801b430 con 0x7f4870160eb0 2023-10-16T12:30:20.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.711+0000 7f486d7fa700 1 --2- 172.21.15.16:0/563742951 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f485406b020 0x7f485406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:20.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.711+0000 7f486d7fa700 1 -- 172.21.15.16:0/563742951 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 5233+0+0 (secure 0 0 0) 0x7f486808ac60 con 0x7f4870160eb0 2023-10-16T12:30:20.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.711+0000 7f4874f8f700 1 --2- 172.21.15.16:0/563742951 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f485406b020 0x7f485406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:20.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.712+0000 7f4874f8f700 1 --2- 172.21.15.16:0/563742951 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f485406b020 0x7f485406d4e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4864006fd0 tx=0x7f4864008040 comp rx=0 tx=0).ready entity=mgr.8388 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:20.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.716+0000 7f486d7fa700 1 -- 172.21.15.16:0/563742951 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4870158e50 con 0x7f4870160eb0 2023-10-16T12:30:20.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.924+0000 7f48771f3700 1 -- 172.21.15.16:0/563742951 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f4870158e50 con 0x7f4870160eb0 2023-10-16T12:30:20.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.925+0000 7f486d7fa700 1 -- 172.21.15.16:0/563742951 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v138) v1 ==== 84+0+65 (secure 0 0 0) 0x7f4870158e50 con 0x7f4870160eb0 2023-10-16T12:30:20.927 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:30:20.927 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2023-10-16T12:30:20.927 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:30:20.928 INFO:teuthology.orchestra.run.smithi016.stdout: "poolnum": 1, 2023-10-16T12:30:20.928 INFO:teuthology.orchestra.run.smithi016.stdout: "poolname": ".mgr" 2023-10-16T12:30:20.928 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:30:20.928 INFO:teuthology.orchestra.run.smithi016.stdout:] 2023-10-16T12:30:20.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.927+0000 7f48771f3700 1 -- 172.21.15.16:0/563742951 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f485406b020 msgr2=0x7f485406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:20.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.927+0000 7f48771f3700 1 --2- 172.21.15.16:0/563742951 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f485406b020 0x7f485406d4e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4864006fd0 tx=0x7f4864008040 comp rx=0 tx=0).stop 2023-10-16T12:30:20.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.927+0000 7f48771f3700 1 -- 172.21.15.16:0/563742951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4870160eb0 msgr2=0x7f48700afb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:20.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.927+0000 7f48771f3700 1 --2- 172.21.15.16:0/563742951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4870160eb0 0x7f48700afb50 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f486800b370 tx=0x7f4868004ad0 comp rx=0 tx=0).stop 2023-10-16T12:30:20.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.927+0000 7f48771f3700 1 -- 172.21.15.16:0/563742951 shutdown_connections 2023-10-16T12:30:20.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.927+0000 7f48771f3700 1 --2- 172.21.15.16:0/563742951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4870160eb0 0x7f48700afb50 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:20.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.927+0000 7f48771f3700 1 --2- 172.21.15.16:0/563742951 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f487006b1f0 0x7f48700c53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:20.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.927+0000 7f48771f3700 1 --2- 172.21.15.16:0/563742951 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f485406b020 0x7f485406d4e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:20.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.927+0000 7f48771f3700 1 --2- 172.21.15.16:0/563742951 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f48700cab70 0x7f48700af610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:20.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.927+0000 7f48771f3700 1 -- 172.21.15.16:0/563742951 >> 172.21.15.16:0/563742951 conn(0x7f48700cc5b0 msgr2=0x7f48700686c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:20.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.927+0000 7f48771f3700 1 -- 172.21.15.16:0/563742951 shutdown_connections 2023-10-16T12:30:20.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:20.927+0000 7f48771f3700 1 -- 172.21.15.16:0/563742951 wait complete. 2023-10-16T12:30:20.941 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-16T12:30:20.941 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-16T12:30:21.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.069+0000 7fdf0c052700 1 Processor -- start 2023-10-16T12:30:21.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.069+0000 7fdf0c052700 1 -- start start 2023-10-16T12:30:21.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.070+0000 7fdf0c052700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdf0415b500 0x7fdf0415fab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:21.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.070+0000 7fdf0c052700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf0406b220 con 0x7fdf0406e590 2023-10-16T12:30:21.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.070+0000 7fdf0c052700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf0406a980 con 0x7fdf040cb040 2023-10-16T12:30:21.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.070+0000 7fdf0c052700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf04160090 con 0x7fdf0415b500 2023-10-16T12:30:21.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.070+0000 7fdf09dee700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdf0415b500 0x7fdf0415fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:21.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.070+0000 7fdf095ed700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fdf0406e590 0x7fdf040ca930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34398/0 (socket says 172.21.15.16:34398) 2023-10-16T12:30:21.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.070+0000 7fdf095ed700 1 -- 172.21.15.16:0/1980457712 learned_addr learned my addr 172.21.15.16:0/1980457712 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:30:21.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.071+0000 7fdf09dee700 1 -- 172.21.15.16:0/1980457712 >> v1:172.21.15.49:6789/0 conn(0x7fdf040cb040 legacy=0x7fdf040ccc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:21.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.071+0000 7fdf08dec700 1 -- 172.21.15.16:0/1980457712 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3095955558 0 0) 0x7fdf0406b220 con 0x7fdf0406e590 2023-10-16T12:30:21.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.071+0000 7fdf09dee700 1 -- 172.21.15.16:0/1980457712 >> v1:172.21.15.16:6789/0 conn(0x7fdf0406e590 legacy=0x7fdf040ca930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:21.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.071+0000 7fdf09dee700 1 -- 172.21.15.16:0/1980457712 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf0406ac60 con 0x7fdf0415b500 2023-10-16T12:30:21.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.071+0000 7fdf09dee700 1 --2- 172.21.15.16:0/1980457712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf0415b500 0x7fdf0415fab0 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7fdef4009840 tx=0x7fdef4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f316335d20b73b5d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:21.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.071+0000 7fdf08dec700 1 -- 172.21.15.16:0/1980457712 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdef4011830 con 0x7fdf0415b500 2023-10-16T12:30:21.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.071+0000 7fdf08dec700 1 -- 172.21.15.16:0/1980457712 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdef4011990 con 0x7fdf0415b500 2023-10-16T12:30:21.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.071+0000 7fdf08dec700 1 -- 172.21.15.16:0/1980457712 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdef4011d20 con 0x7fdf0415b500 2023-10-16T12:30:21.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.072+0000 7fdf0c052700 1 -- 172.21.15.16:0/1980457712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf0415b500 msgr2=0x7fdf0415fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:21.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.072+0000 7fdf0c052700 1 --2- 172.21.15.16:0/1980457712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf0415b500 0x7fdf0415fab0 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7fdef4009840 tx=0x7fdef4009b50 comp rx=0 tx=0).stop 2023-10-16T12:30:21.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.072+0000 7fdf0c052700 1 -- 172.21.15.16:0/1980457712 shutdown_connections 2023-10-16T12:30:21.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.072+0000 7fdf0c052700 1 --2- 172.21.15.16:0/1980457712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf0415b500 0x7fdf0415fab0 unknown :-1 s=CLOSED pgs=1526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:21.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.072+0000 7fdf0c052700 1 -- 172.21.15.16:0/1980457712 >> 172.21.15.16:0/1980457712 conn(0x7fdf0406f430 msgr2=0x7fdf0406f810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:21.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.072+0000 7fdf0c052700 1 -- 172.21.15.16:0/1980457712 shutdown_connections 2023-10-16T12:30:21.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.072+0000 7fdf0c052700 1 -- 172.21.15.16:0/1980457712 wait complete. 2023-10-16T12:30:21.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.073+0000 7fdf0c052700 1 Processor -- start 2023-10-16T12:30:21.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.073+0000 7fdf0c052700 1 -- start start 2023-10-16T12:30:21.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.074+0000 7fdf0c052700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdf0406e590 0x7fdf04155960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:21.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.074+0000 7fdf0c052700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf040cb040 0x7fdf0415d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:21.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.074+0000 7fdf0c052700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdf0415b500 0x7fdf040b0b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:21.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.074+0000 7fdf0c052700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf0415c540 con 0x7fdf040cb040 2023-10-16T12:30:21.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.074+0000 7fdf0c052700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf0415c820 con 0x7fdf0406e590 2023-10-16T12:30:21.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.074+0000 7fdf0c052700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf0415cb00 con 0x7fdf0415b500 2023-10-16T12:30:21.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.074+0000 7fdf095ed700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf040cb040 0x7fdf0415d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:21.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.074+0000 7fdf095ed700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf040cb040 0x7fdf0415d5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45130/0 (socket says 172.21.15.16:45130) 2023-10-16T12:30:21.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.074+0000 7fdf095ed700 1 -- 172.21.15.16:0/3895090823 learned_addr learned my addr 172.21.15.16:0/3895090823 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:21.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.074+0000 7fdf09dee700 1 --2- 172.21.15.16:0/3895090823 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdf0406e590 0x7fdf04155960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:21.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.074+0000 7fdf0a5ef700 1 --2- 172.21.15.16:0/3895090823 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdf0415b500 0x7fdf040b0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:21.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.074+0000 7fdf095ed700 1 -- 172.21.15.16:0/3895090823 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdf0415b500 msgr2=0x7fdf040b0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:21.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.074+0000 7fdf095ed700 1 --2- 172.21.15.16:0/3895090823 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdf0415b500 0x7fdf040b0b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:21.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.074+0000 7fdf095ed700 1 -- 172.21.15.16:0/3895090823 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdf0406e590 msgr2=0x7fdf04155960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:21.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.074+0000 7fdf095ed700 1 --2- 172.21.15.16:0/3895090823 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdf0406e590 0x7fdf04155960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:21.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.074+0000 7fdf095ed700 1 -- 172.21.15.16:0/3895090823 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdef40094a0 con 0x7fdf040cb040 2023-10-16T12:30:21.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.074+0000 7fdf0a5ef700 1 --2- 172.21.15.16:0/3895090823 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdf0415b500 0x7fdf040b0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:30:21.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.074+0000 7fdf095ed700 1 --2- 172.21.15.16:0/3895090823 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf040cb040 0x7fdf0415d5e0 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7fdf00001eb0 tx=0x7fdf0000c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:21.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.074+0000 7fdefa7fc700 1 -- 172.21.15.16:0/3895090823 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf00004c80 con 0x7fdf040cb040 2023-10-16T12:30:21.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.075+0000 7fdf0c052700 1 -- 172.21.15.16:0/3895090823 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf0415cde0 con 0x7fdf040cb040 2023-10-16T12:30:21.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.075+0000 7fdefa7fc700 1 -- 172.21.15.16:0/3895090823 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf00004de0 con 0x7fdf040cb040 2023-10-16T12:30:21.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.075+0000 7fdf0c052700 1 -- 172.21.15.16:0/3895090823 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf04155ea0 con 0x7fdf040cb040 2023-10-16T12:30:21.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.075+0000 7fdefa7fc700 1 -- 172.21.15.16:0/3895090823 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf000072d0 con 0x7fdf040cb040 2023-10-16T12:30:21.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.076+0000 7fdf0c052700 1 -- 172.21.15.16:0/3895090823 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdeec0052f0 con 0x7fdf040cb040 2023-10-16T12:30:21.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.076+0000 7fdefa7fc700 1 -- 172.21.15.16:0/3895090823 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 74) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fdf00007430 con 0x7fdf040cb040 2023-10-16T12:30:21.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.076+0000 7fdefa7fc700 1 --2- 172.21.15.16:0/3895090823 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7fdee806b020 0x7fdee806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:21.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.076+0000 7fdefa7fc700 1 -- 172.21.15.16:0/3895090823 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 5233+0+0 (secure 0 0 0) 0x7fdf00089000 con 0x7fdf040cb040 2023-10-16T12:30:21.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.076+0000 7fdf09dee700 1 --2- 172.21.15.16:0/3895090823 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7fdee806b020 0x7fdee806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:21.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.077+0000 7fdf09dee700 1 --2- 172.21.15.16:0/3895090823 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7fdee806b020 0x7fdee806d4e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fdef40090b0 tx=0x7fdef401c040 comp rx=0 tx=0).ready entity=mgr.8388 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:21.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.081+0000 7fdefa7fc700 1 -- 172.21.15.16:0/3895090823 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdf00054180 con 0x7fdf040cb040 2023-10-16T12:30:21.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.285+0000 7fdf0c052700 1 -- 172.21.15.16:0/3895090823 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fdeec005640 con 0x7fdf040cb040 2023-10-16T12:30:21.292 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:30:21.295 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:30:21.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.375+0000 7fdefa7fc700 1 -- 172.21.15.16:0/3895090823 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v139) v1 ==== 138+0+0 (secure 0 0 0) 0x7fdf000577a0 con 0x7fdf040cb040 2023-10-16T12:30:21.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.378+0000 7fdf0c052700 1 -- 172.21.15.16:0/3895090823 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7fdee806b020 msgr2=0x7fdee806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:21.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.378+0000 7fdf0c052700 1 --2- 172.21.15.16:0/3895090823 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7fdee806b020 0x7fdee806d4e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fdef40090b0 tx=0x7fdef401c040 comp rx=0 tx=0).stop 2023-10-16T12:30:21.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.378+0000 7fdf0c052700 1 -- 172.21.15.16:0/3895090823 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf040cb040 msgr2=0x7fdf0415d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:21.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.378+0000 7fdf0c052700 1 --2- 172.21.15.16:0/3895090823 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf040cb040 0x7fdf0415d5e0 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7fdf00001eb0 tx=0x7fdf0000c800 comp rx=0 tx=0).stop 2023-10-16T12:30:21.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.379+0000 7fdf0c052700 1 -- 172.21.15.16:0/3895090823 shutdown_connections 2023-10-16T12:30:21.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.379+0000 7fdf0c052700 1 --2- 172.21.15.16:0/3895090823 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdf0415b500 0x7fdf040b0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:21.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.379+0000 7fdf0c052700 1 --2- 172.21.15.16:0/3895090823 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7fdee806b020 0x7fdee806d4e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:21.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.379+0000 7fdf0c052700 1 --2- 172.21.15.16:0/3895090823 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdf0406e590 0x7fdf04155960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:21.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.379+0000 7fdf0c052700 1 --2- 172.21.15.16:0/3895090823 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf040cb040 0x7fdf0415d5e0 unknown :-1 s=CLOSED pgs=1527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:21.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.379+0000 7fdf0c052700 1 -- 172.21.15.16:0/3895090823 >> 172.21.15.16:0/3895090823 conn(0x7fdf0406f430 msgr2=0x7fdf0415f030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:21.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.379+0000 7fdf0c052700 1 -- 172.21.15.16:0/3895090823 shutdown_connections 2023-10-16T12:30:21.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.379+0000 7fdf0c052700 1 -- 172.21.15.16:0/3895090823 wait complete. 2023-10-16T12:30:21.386 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' created 2023-10-16T12:30:21.396 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-16T12:30:21.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.566+0000 7f080f6e0700 1 Processor -- start 2023-10-16T12:30:21.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.566+0000 7f080f6e0700 1 -- start start 2023-10-16T12:30:21.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.567+0000 7f080f6e0700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f080815b070 0x7f080815b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:21.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.567+0000 7f080f6e0700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f08080cb880 0x7f08080cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:21.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.567+0000 7f080f6e0700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08080cef70 con 0x7f080806da60 2023-10-16T12:30:21.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.567+0000 7f080f6e0700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08080ce360 con 0x7f080815b070 2023-10-16T12:30:21.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.567+0000 7f080f6e0700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08080ce640 con 0x7f08080cb880 2023-10-16T12:30:21.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.567+0000 7f080d47c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f080815b070 0x7f080815b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:21.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.567+0000 7f080d47c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f080815b070 0x7f080815b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45136/0 (socket says 172.21.15.16:45136) 2023-10-16T12:30:21.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.567+0000 7f080d47c700 1 -- 172.21.15.16:0/3642213290 learned_addr learned my addr 172.21.15.16:0/3642213290 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:21.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.567+0000 7f080cc7b700 1 --2- 172.21.15.16:0/3642213290 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f08080cb880 0x7f08080cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:21.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.567+0000 7f080d47c700 1 -- 172.21.15.16:0/3642213290 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f08080cb880 msgr2=0x7f08080cbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:21.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.567+0000 7f080d47c700 1 --2- 172.21.15.16:0/3642213290 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f08080cb880 0x7f08080cbd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:21.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.567+0000 7f080d47c700 1 -- 172.21.15.16:0/3642213290 >> v1:172.21.15.49:6789/0 conn(0x7f080806da60 legacy=0x7f08080cf400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:21.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.567+0000 7f080d47c700 1 -- 172.21.15.16:0/3642213290 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08080ce920 con 0x7f080815b070 2023-10-16T12:30:21.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.568+0000 7f080d47c700 1 --2- 172.21.15.16:0/3642213290 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f080815b070 0x7f080815b450 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f07f80098f0 tx=0x7f07f8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6d071ede291360b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:21.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.568+0000 7f07fffff700 1 -- 172.21.15.16:0/3642213290 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07f8004230 con 0x7f080815b070 2023-10-16T12:30:21.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.568+0000 7f07fffff700 1 -- 172.21.15.16:0/3642213290 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07f8004390 con 0x7f080815b070 2023-10-16T12:30:21.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.568+0000 7f07fffff700 1 -- 172.21.15.16:0/3642213290 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07f80036a0 con 0x7f080815b070 2023-10-16T12:30:21.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.568+0000 7f080f6e0700 1 -- 172.21.15.16:0/3642213290 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f080815b070 msgr2=0x7f080815b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:21.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.568+0000 7f080f6e0700 1 --2- 172.21.15.16:0/3642213290 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f080815b070 0x7f080815b450 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f07f80098f0 tx=0x7f07f8009c00 comp rx=0 tx=0).stop 2023-10-16T12:30:21.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.568+0000 7f080f6e0700 1 -- 172.21.15.16:0/3642213290 shutdown_connections 2023-10-16T12:30:21.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.568+0000 7f080f6e0700 1 --2- 172.21.15.16:0/3642213290 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f080815b070 0x7f080815b450 unknown :-1 s=CLOSED pgs=1528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:21.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.568+0000 7f080f6e0700 1 --2- 172.21.15.16:0/3642213290 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f08080cb880 0x7f08080cbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:21.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.568+0000 7f080f6e0700 1 -- 172.21.15.16:0/3642213290 >> 172.21.15.16:0/3642213290 conn(0x7f08080b1f40 msgr2=0x7f08080b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:21.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.568+0000 7f080f6e0700 1 -- 172.21.15.16:0/3642213290 shutdown_connections 2023-10-16T12:30:21.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.568+0000 7f080f6e0700 1 -- 172.21.15.16:0/3642213290 wait complete. 2023-10-16T12:30:21.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.570+0000 7f080f6e0700 1 Processor -- start 2023-10-16T12:30:21.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.570+0000 7f080f6e0700 1 -- start start 2023-10-16T12:30:21.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.570+0000 7f080f6e0700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f080806da60 0x7f08081465a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:21.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.570+0000 7f080f6e0700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f08080cb880 0x7f08081f4b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:21.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.570+0000 7f080f6e0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f080815b070 0x7f08081f0460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:21.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.570+0000 7f080f6e0700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0808146c30 con 0x7f080815b070 2023-10-16T12:30:21.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.570+0000 7f080f6e0700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0808146f10 con 0x7f080806da60 2023-10-16T12:30:21.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.570+0000 7f080f6e0700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08081471f0 con 0x7f08080cb880 2023-10-16T12:30:21.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.571+0000 7f080dc7d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f080815b070 0x7f08081f0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:21.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.571+0000 7f080dc7d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f080815b070 0x7f08081f0460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45146/0 (socket says 172.21.15.16:45146) 2023-10-16T12:30:21.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.571+0000 7f080dc7d700 1 -- 172.21.15.16:0/2687609228 learned_addr learned my addr 172.21.15.16:0/2687609228 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:21.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.571+0000 7f080cc7b700 1 --2- 172.21.15.16:0/2687609228 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f08080cb880 0x7f08081f4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:21.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.571+0000 7f080d47c700 1 --2- 172.21.15.16:0/2687609228 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f080806da60 0x7f08081465a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:21.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.571+0000 7f080dc7d700 1 -- 172.21.15.16:0/2687609228 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f08080cb880 msgr2=0x7f08081f4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:21.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.571+0000 7f080dc7d700 1 --2- 172.21.15.16:0/2687609228 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f08080cb880 0x7f08081f4b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:21.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.571+0000 7f080dc7d700 1 -- 172.21.15.16:0/2687609228 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f080806da60 msgr2=0x7f08081465a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:21.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.571+0000 7f080dc7d700 1 --2- 172.21.15.16:0/2687609228 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f080806da60 0x7f08081465a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:21.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.571+0000 7f080dc7d700 1 -- 172.21.15.16:0/2687609228 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07f8009550 con 0x7f080815b070 2023-10-16T12:30:21.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.571+0000 7f080dc7d700 1 --2- 172.21.15.16:0/2687609228 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f080815b070 0x7f08081f0460 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f080000cc10 tx=0x7f080000cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:21.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.571+0000 7f07fdffb700 1 -- 172.21.15.16:0/2687609228 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f080000aa60 con 0x7f080815b070 2023-10-16T12:30:21.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.571+0000 7f07fdffb700 1 -- 172.21.15.16:0/2687609228 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f080000abc0 con 0x7f080815b070 2023-10-16T12:30:21.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.571+0000 7f07fdffb700 1 -- 172.21.15.16:0/2687609228 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0800012420 con 0x7f080815b070 2023-10-16T12:30:21.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.571+0000 7f080f6e0700 1 -- 172.21.15.16:0/2687609228 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08081474d0 con 0x7f080815b070 2023-10-16T12:30:21.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.571+0000 7f080f6e0700 1 -- 172.21.15.16:0/2687609228 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08081f5050 con 0x7f080815b070 2023-10-16T12:30:21.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.572+0000 7f080f6e0700 1 -- 172.21.15.16:0/2687609228 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0808144830 con 0x7f080815b070 2023-10-16T12:30:21.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.573+0000 7f07fdffb700 1 -- 172.21.15.16:0/2687609228 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 74) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0800012580 con 0x7f080815b070 2023-10-16T12:30:21.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.573+0000 7f07fdffb700 1 --2- 172.21.15.16:0/2687609228 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f07ec06b020 0x7f07ec06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:21.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.573+0000 7f07fdffb700 1 -- 172.21.15.16:0/2687609228 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5603+0+0 (secure 0 0 0) 0x7f0800089a20 con 0x7f080815b070 2023-10-16T12:30:21.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.573+0000 7f080d47c700 1 --2- 172.21.15.16:0/2687609228 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f07ec06b020 0x7f07ec06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:21.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.574+0000 7f080d47c700 1 --2- 172.21.15.16:0/2687609228 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f07ec06b020 0x7f07ec06d4e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f07f8009130 tx=0x7f07f8008030 comp rx=0 tx=0).ready entity=mgr.8388 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:21.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.578+0000 7f07fdffb700 1 -- 172.21.15.16:0/2687609228 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0808144830 con 0x7f080815b070 2023-10-16T12:30:21.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:21.781+0000 7f080f6e0700 1 -- 172.21.15.16:0/2687609228 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f0808144830 con 0x7f080815b070 2023-10-16T12:30:22.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:22.378+0000 7f07fdffb700 1 -- 172.21.15.16:0/2687609228 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v140) v1 ==== 170+0+0 (secure 0 0 0) 0x7f0808144830 con 0x7f080815b070 2023-10-16T12:30:22.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:22.382+0000 7f080f6e0700 1 -- 172.21.15.16:0/2687609228 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f07ec06b020 msgr2=0x7f07ec06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:22.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:22.382+0000 7f080f6e0700 1 --2- 172.21.15.16:0/2687609228 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f07ec06b020 0x7f07ec06d4e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f07f8009130 tx=0x7f07f8008030 comp rx=0 tx=0).stop 2023-10-16T12:30:22.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:22.382+0000 7f080f6e0700 1 -- 172.21.15.16:0/2687609228 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f080815b070 msgr2=0x7f08081f0460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:22.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:22.382+0000 7f080f6e0700 1 --2- 172.21.15.16:0/2687609228 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f080815b070 0x7f08081f0460 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f080000cc10 tx=0x7f080000cf20 comp rx=0 tx=0).stop 2023-10-16T12:30:22.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:22.383+0000 7f080f6e0700 1 -- 172.21.15.16:0/2687609228 shutdown_connections 2023-10-16T12:30:22.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:22.383+0000 7f080f6e0700 1 --2- 172.21.15.16:0/2687609228 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f080815b070 0x7f08081f0460 unknown :-1 s=CLOSED pgs=1529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:22.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:22.383+0000 7f080f6e0700 1 --2- 172.21.15.16:0/2687609228 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f07ec06b020 0x7f07ec06d4e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:22.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:22.383+0000 7f080f6e0700 1 --2- 172.21.15.16:0/2687609228 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f080806da60 0x7f08081465a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:22.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:22.383+0000 7f080f6e0700 1 --2- 172.21.15.16:0/2687609228 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f08080cb880 0x7f08081f4b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:22.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:22.383+0000 7f080f6e0700 1 -- 172.21.15.16:0/2687609228 >> 172.21.15.16:0/2687609228 conn(0x7f08080b1f40 msgr2=0x7f08081586f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:22.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:22.384+0000 7f080f6e0700 1 -- 172.21.15.16:0/2687609228 shutdown_connections 2023-10-16T12:30:22.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:22.384+0000 7f080f6e0700 1 -- 172.21.15.16:0/2687609228 wait complete. 2023-10-16T12:30:22.398 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' created 2023-10-16T12:30:22.416 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-16T12:30:22.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:22.547+0000 7ff18dd63700 1 Processor -- start 2023-10-16T12:30:22.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:22.547+0000 7ff18dd63700 1 -- start start 2023-10-16T12:30:22.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:22.548+0000 7ff18dd63700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff18806e570 0x7ff18815e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:22.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:22.548+0000 7ff18dd63700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff18815b6a0 con 0x7ff18806a7e0 2023-10-16T12:30:22.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:22.548+0000 7ff18dd63700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff188161b20 con 0x7ff18815b2c0 2023-10-16T12:30:22.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:22.548+0000 7ff18dd63700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff188161e00 con 0x7ff18806e570 2023-10-16T12:30:22.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:22.548+0000 7ff186ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff18806e570 0x7ff18815e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:22.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:22.548+0000 7ff187fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff18806a7e0 0x7ff188071a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34428/0 (socket says 172.21.15.16:34428) 2023-10-16T12:30:22.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:22.549+0000 7ff187fff700 1 -- 172.21.15.16:0/372743275 learned_addr learned my addr 172.21.15.16:0/372743275 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:30:22.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:22.549+0000 7ff186ffd700 1 -- 172.21.15.16:0/372743275 >> v1:172.21.15.49:6789/0 conn(0x7ff18815b2c0 legacy=0x7ff188161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:22.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:22.549+0000 7ff186ffd700 1 -- 172.21.15.16:0/372743275 >> v1:172.21.15.16:6789/0 conn(0x7ff18806a7e0 legacy=0x7ff188071a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:22.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:22.549+0000 7ff186ffd700 1 -- 172.21.15.16:0/372743275 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1881620e0 con 0x7ff18806e570 2023-10-16T12:30:22.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:22.549+0000 7ff1867fc700 1 -- 172.21.15.16:0/372743275 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1676191362 0 0) 0x7ff18815b6a0 con 0x7ff18806a7e0 2023-10-16T12:30:22.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:22.549+0000 7ff186ffd700 1 --2- 172.21.15.16:0/372743275 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff18806e570 0x7ff18815e5a0 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7ff170009840 tx=0x7ff170009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb088a0d6f1f9d71 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:22.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:22.549+0000 7ff1867fc700 1 -- 172.21.15.16:0/372743275 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff170004230 con 0x7ff18806e570 2023-10-16T12:30:22.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:22.549+0000 7ff1867fc700 1 -- 172.21.15.16:0/372743275 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff170004390 con 0x7ff18806e570 2023-10-16T12:30:22.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:22.549+0000 7ff1867fc700 1 -- 172.21.15.16:0/372743275 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1700036a0 con 0x7ff18806e570 2023-10-16T12:30:22.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:22.550+0000 7ff18dd63700 1 -- 172.21.15.16:0/372743275 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff18806e570 msgr2=0x7ff18815e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:22.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:22.550+0000 7ff18dd63700 1 --2- 172.21.15.16:0/372743275 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff18806e570 0x7ff18815e5a0 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7ff170009840 tx=0x7ff170009b50 comp rx=0 tx=0).stop 2023-10-16T12:30:22.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:22.550+0000 7ff18dd63700 1 -- 172.21.15.16:0/372743275 shutdown_connections 2023-10-16T12:30:22.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:22.550+0000 7ff18dd63700 1 --2- 172.21.15.16:0/372743275 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff18806e570 0x7ff18815e5a0 unknown :-1 s=CLOSED pgs=1531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:22.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:22.550+0000 7ff18dd63700 1 -- 172.21.15.16:0/372743275 >> 172.21.15.16:0/372743275 conn(0x7ff1880b1f40 msgr2=0x7ff1880b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:22.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:22.550+0000 7ff18dd63700 1 -- 172.21.15.16:0/372743275 shutdown_connections 2023-10-16T12:30:22.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:22.550+0000 7ff18dd63700 1 -- 172.21.15.16:0/372743275 wait complete. 2023-10-16T12:30:22.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:22.551+0000 7ff18dd63700 1 Processor -- start 2023-10-16T12:30:22.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:22.551+0000 7ff18dd63700 1 -- start start 2023-10-16T12:30:22.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:22.552+0000 7ff18dd63700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff18806a7e0 0x7ff18815a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:22.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:22.552+0000 7ff18dd63700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff18806e570 0x7ff1881f2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:22.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:22.552+0000 7ff18dd63700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff18815b2c0 0x7ff1881f3410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:22.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:22.552+0000 7ff18dd63700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff18815aa20 con 0x7ff18815b2c0 2023-10-16T12:30:22.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:22.552+0000 7ff18dd63700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1881f3950 con 0x7ff18806e570 2023-10-16T12:30:22.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:22.552+0000 7ff18dd63700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1880b4d90 con 0x7ff18806a7e0 2023-10-16T12:30:22.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:22.552+0000 7ff187fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff18815b2c0 0x7ff1881f3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:22.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:22.552+0000 7ff187fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff18815b2c0 0x7ff1881f3410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45160/0 (socket says 172.21.15.16:45160) 2023-10-16T12:30:22.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:22.552+0000 7ff187fff700 1 -- 172.21.15.16:0/3440180714 learned_addr learned my addr 172.21.15.16:0/3440180714 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:22.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:22.552+0000 7ff1877fe700 1 --2- 172.21.15.16:0/3440180714 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff18806a7e0 0x7ff18815a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:22.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:22.552+0000 7ff186ffd700 1 --2- 172.21.15.16:0/3440180714 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff18806e570 0x7ff1881f2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:22.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:22.552+0000 7ff187fff700 1 -- 172.21.15.16:0/3440180714 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff18806a7e0 msgr2=0x7ff18815a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:22.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:22.552+0000 7ff187fff700 1 --2- 172.21.15.16:0/3440180714 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff18806a7e0 0x7ff18815a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:22.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:22.552+0000 7ff187fff700 1 -- 172.21.15.16:0/3440180714 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff18806e570 msgr2=0x7ff1881f2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:22.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:22.552+0000 7ff187fff700 1 --2- 172.21.15.16:0/3440180714 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff18806e570 0x7ff1881f2ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:22.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:22.552+0000 7ff187fff700 1 -- 172.21.15.16:0/3440180714 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1700094a0 con 0x7ff18815b2c0 2023-10-16T12:30:22.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:22.552+0000 7ff186ffd700 1 --2- 172.21.15.16:0/3440180714 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff18806e570 0x7ff1881f2ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:30:22.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:22.552+0000 7ff187fff700 1 --2- 172.21.15.16:0/3440180714 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff18815b2c0 0x7ff1881f3410 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7ff18000cb90 tx=0x7ff18000cea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:22.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:22.553+0000 7ff17ffff700 1 -- 172.21.15.16:0/3440180714 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff18000aa80 con 0x7ff18815b2c0 2023-10-16T12:30:22.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:22.553+0000 7ff17ffff700 1 -- 172.21.15.16:0/3440180714 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff18000abe0 con 0x7ff18815b2c0 2023-10-16T12:30:22.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:22.553+0000 7ff17ffff700 1 -- 172.21.15.16:0/3440180714 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1800070d0 con 0x7ff18815b2c0 2023-10-16T12:30:22.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:22.553+0000 7ff18dd63700 1 -- 172.21.15.16:0/3440180714 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1880b5070 con 0x7ff18815b2c0 2023-10-16T12:30:22.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:22.553+0000 7ff18dd63700 1 -- 172.21.15.16:0/3440180714 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1880b5630 con 0x7ff18815b2c0 2023-10-16T12:30:22.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:22.554+0000 7ff18dd63700 1 -- 172.21.15.16:0/3440180714 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff18804ebb0 con 0x7ff18815b2c0 2023-10-16T12:30:22.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:22.554+0000 7ff17ffff700 1 -- 172.21.15.16:0/3440180714 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 74) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff180007230 con 0x7ff18815b2c0 2023-10-16T12:30:22.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:22.555+0000 7ff17ffff700 1 --2- 172.21.15.16:0/3440180714 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7ff16806b050 0x7ff16806d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:22.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:22.555+0000 7ff17ffff700 1 -- 172.21.15.16:0/3440180714 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5985+0+0 (secure 0 0 0) 0x7ff1800892a0 con 0x7ff18815b2c0 2023-10-16T12:30:22.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:22.555+0000 7ff1877fe700 1 --2- 172.21.15.16:0/3440180714 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7ff16806b050 0x7ff16806d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:22.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:22.555+0000 7ff1877fe700 1 --2- 172.21.15.16:0/3440180714 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7ff16806b050 0x7ff16806d510 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ff1880cbff0 tx=0x7ff178007ec0 comp rx=0 tx=0).ready entity=mgr.8388 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:22.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:22.560+0000 7ff17ffff700 1 -- 172.21.15.16:0/3440180714 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff1800541b0 con 0x7ff18815b2c0 2023-10-16T12:30:22.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:22.806+0000 7ff18dd63700 1 -- 172.21.15.16:0/3440180714 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7ff1880b6370 con 0x7ff18815b2c0 2023-10-16T12:30:23.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.403+0000 7ff17ffff700 1 -- 172.21.15.16:0/3440180714 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 17 and data pool 18 v66) v1 ==== 174+0+0 (secure 0 0 0) 0x7ff1800577d0 con 0x7ff18815b2c0 2023-10-16T12:30:23.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.417+0000 7ff18dd63700 1 -- 172.21.15.16:0/3440180714 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7ff16806b050 msgr2=0x7ff16806d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:23.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.417+0000 7ff18dd63700 1 --2- 172.21.15.16:0/3440180714 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7ff16806b050 0x7ff16806d510 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ff1880cbff0 tx=0x7ff178007ec0 comp rx=0 tx=0).stop 2023-10-16T12:30:23.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.417+0000 7ff18dd63700 1 -- 172.21.15.16:0/3440180714 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff18815b2c0 msgr2=0x7ff1881f3410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:23.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.417+0000 7ff18dd63700 1 --2- 172.21.15.16:0/3440180714 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff18815b2c0 0x7ff1881f3410 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7ff18000cb90 tx=0x7ff18000cea0 comp rx=0 tx=0).stop 2023-10-16T12:30:23.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.417+0000 7ff18dd63700 1 -- 172.21.15.16:0/3440180714 shutdown_connections 2023-10-16T12:30:23.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.417+0000 7ff18dd63700 1 --2- 172.21.15.16:0/3440180714 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff18815b2c0 0x7ff1881f3410 unknown :-1 s=CLOSED pgs=1532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:23.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.417+0000 7ff18dd63700 1 --2- 172.21.15.16:0/3440180714 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff18806a7e0 0x7ff18815a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:23.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.417+0000 7ff18dd63700 1 --2- 172.21.15.16:0/3440180714 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7ff16806b050 0x7ff16806d510 secure :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ff1880cbff0 tx=0x7ff178007ec0 comp rx=0 tx=0).stop 2023-10-16T12:30:23.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.417+0000 7ff18dd63700 1 --2- 172.21.15.16:0/3440180714 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff18806e570 0x7ff1881f2ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:23.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.417+0000 7ff18dd63700 1 -- 172.21.15.16:0/3440180714 >> 172.21.15.16:0/3440180714 conn(0x7ff1880b1f40 msgr2=0x7ff1880cc2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:23.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.417+0000 7ff18dd63700 1 -- 172.21.15.16:0/3440180714 shutdown_connections 2023-10-16T12:30:23.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.418+0000 7ff18dd63700 1 -- 172.21.15.16:0/3440180714 wait complete. 2023-10-16T12:30:23.423 INFO:teuthology.orchestra.run.smithi016.stderr:new fs with metadata pool 17 and data pool 18 2023-10-16T12:30:23.442 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:30:23.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.573+0000 7f396e41b700 1 Processor -- start 2023-10-16T12:30:23.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.573+0000 7f396e41b700 1 -- start start 2023-10-16T12:30:23.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.573+0000 7f396e41b700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f396815b280 0x7f39681613e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:23.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.573+0000 7f396e41b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f396806a7e0 0x7f396806ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:23.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.573+0000 7f396e41b700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f396806b150 con 0x7f39680cc8a0 2023-10-16T12:30:23.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.573+0000 7f396e41b700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3968161c30 con 0x7f396806a7e0 2023-10-16T12:30:23.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.573+0000 7f396e41b700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3968161f10 con 0x7f396815b280 2023-10-16T12:30:23.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.574+0000 7f396c9b8700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f396806a7e0 0x7f396806ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:23.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.574+0000 7f396c9b8700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f396806a7e0 0x7f396806ac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45168/0 (socket says 172.21.15.16:45168) 2023-10-16T12:30:23.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.574+0000 7f396c9b8700 1 -- 172.21.15.16:0/2181146770 learned_addr learned my addr 172.21.15.16:0/2181146770 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:23.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.574+0000 7f396c9b8700 1 -- 172.21.15.16:0/2181146770 >> v2:172.21.15.49:3300/0 conn(0x7f396815b280 msgr2=0x7f39681613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:23.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.574+0000 7f396c9b8700 1 --2- 172.21.15.16:0/2181146770 >> v2:172.21.15.49:3300/0 conn(0x7f396815b280 0x7f39681613e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:23.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.574+0000 7f396c9b8700 1 -- 172.21.15.16:0/2181146770 >> v1:172.21.15.49:6789/0 conn(0x7f39680cc8a0 legacy=0x7f396815e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:23.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.574+0000 7f396c9b8700 1 -- 172.21.15.16:0/2181146770 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39681621f0 con 0x7f396806a7e0 2023-10-16T12:30:23.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.574+0000 7f396c9b8700 1 --2- 172.21.15.16:0/2181146770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f396806a7e0 0x7f396806ac10 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f396806b8a0 tx=0x7f396000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59356df020b80d5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:23.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.574+0000 7f3966ffd700 1 -- 172.21.15.16:0/2181146770 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f396000e070 con 0x7f396806a7e0 2023-10-16T12:30:23.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.574+0000 7f3966ffd700 1 -- 172.21.15.16:0/2181146770 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3960003a20 con 0x7f396806a7e0 2023-10-16T12:30:23.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.574+0000 7f3966ffd700 1 -- 172.21.15.16:0/2181146770 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3960003dd0 con 0x7f396806a7e0 2023-10-16T12:30:23.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.575+0000 7f396e41b700 1 -- 172.21.15.16:0/2181146770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f396806a7e0 msgr2=0x7f396806ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:23.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.575+0000 7f396e41b700 1 --2- 172.21.15.16:0/2181146770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f396806a7e0 0x7f396806ac10 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f396806b8a0 tx=0x7f396000b330 comp rx=0 tx=0).stop 2023-10-16T12:30:23.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.575+0000 7f396e41b700 1 -- 172.21.15.16:0/2181146770 shutdown_connections 2023-10-16T12:30:23.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.575+0000 7f396e41b700 1 --2- 172.21.15.16:0/2181146770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f396806a7e0 0x7f396806ac10 unknown :-1 s=CLOSED pgs=1533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:23.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.575+0000 7f396e41b700 1 --2- 172.21.15.16:0/2181146770 >> v2:172.21.15.49:3300/0 conn(0x7f396815b280 0x7f39681613e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:23.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.575+0000 7f396e41b700 1 -- 172.21.15.16:0/2181146770 >> 172.21.15.16:0/2181146770 conn(0x7f39680b1fd0 msgr2=0x7f39680b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:23.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.575+0000 7f396e41b700 1 -- 172.21.15.16:0/2181146770 shutdown_connections 2023-10-16T12:30:23.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.575+0000 7f396e41b700 1 -- 172.21.15.16:0/2181146770 wait complete. 2023-10-16T12:30:23.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.576+0000 7f396e41b700 1 Processor -- start 2023-10-16T12:30:23.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.576+0000 7f396e41b700 1 -- start start 2023-10-16T12:30:23.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.577+0000 7f396e41b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f396806a7e0 0x7f39680c8a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:23.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.577+0000 7f396e41b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39680cc8a0 0x7f3968158b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:23.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.577+0000 7f396e41b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f396815b280 0x7f39681f60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:23.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.577+0000 7f396e41b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39680c8f70 con 0x7f39680cc8a0 2023-10-16T12:30:23.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.577+0000 7f396e41b700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39681f7080 con 0x7f396806a7e0 2023-10-16T12:30:23.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.577+0000 7f396e41b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39681590a0 con 0x7f396815b280 2023-10-16T12:30:23.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.577+0000 7f39677fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39680cc8a0 0x7f3968158b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:23.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.577+0000 7f39677fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39680cc8a0 0x7f3968158b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45172/0 (socket says 172.21.15.16:45172) 2023-10-16T12:30:23.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.577+0000 7f39677fe700 1 -- 172.21.15.16:0/739706789 learned_addr learned my addr 172.21.15.16:0/739706789 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:23.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.577+0000 7f3967fff700 1 --2- 172.21.15.16:0/739706789 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f396806a7e0 0x7f39680c8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:23.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.577+0000 7f396c9b8700 1 --2- 172.21.15.16:0/739706789 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f396815b280 0x7f39681f60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:23.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.577+0000 7f39677fe700 1 -- 172.21.15.16:0/739706789 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f396815b280 msgr2=0x7f39681f60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:23.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.577+0000 7f39677fe700 1 --2- 172.21.15.16:0/739706789 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f396815b280 0x7f39681f60e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:23.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.577+0000 7f39677fe700 1 -- 172.21.15.16:0/739706789 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f396806a7e0 msgr2=0x7f39680c8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:23.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.577+0000 7f39677fe700 1 --2- 172.21.15.16:0/739706789 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f396806a7e0 0x7f39680c8a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:23.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.577+0000 7f39677fe700 1 -- 172.21.15.16:0/739706789 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3960009b30 con 0x7f39680cc8a0 2023-10-16T12:30:23.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.577+0000 7f3967fff700 1 --2- 172.21.15.16:0/739706789 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f396806a7e0 0x7f39680c8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:30:23.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.577+0000 7f396c9b8700 1 --2- 172.21.15.16:0/739706789 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f396815b280 0x7f39681f60e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:30:23.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.577+0000 7f39677fe700 1 --2- 172.21.15.16:0/739706789 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39680cc8a0 0x7f3968158b60 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f395c00c4d0 tx=0x7f395c00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:23.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.577+0000 7f3964ff9700 1 -- 172.21.15.16:0/739706789 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f395c0054e0 con 0x7f39680cc8a0 2023-10-16T12:30:23.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.578+0000 7f3964ff9700 1 -- 172.21.15.16:0/739706789 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f395c003680 con 0x7f39680cc8a0 2023-10-16T12:30:23.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.578+0000 7f3964ff9700 1 -- 172.21.15.16:0/739706789 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f395c003a10 con 0x7f39680cc8a0 2023-10-16T12:30:23.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.578+0000 7f396e41b700 1 -- 172.21.15.16:0/739706789 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39681f6ac0 con 0x7f39680cc8a0 2023-10-16T12:30:23.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.578+0000 7f396e41b700 1 -- 172.21.15.16:0/739706789 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39680aca70 con 0x7f39680cc8a0 2023-10-16T12:30:23.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.579+0000 7f396e41b700 1 -- 172.21.15.16:0/739706789 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39680ac8d0 con 0x7f39680cc8a0 2023-10-16T12:30:23.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.579+0000 7f3964ff9700 1 -- 172.21.15.16:0/739706789 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 74) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f395c003b70 con 0x7f39680cc8a0 2023-10-16T12:30:23.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.579+0000 7f3964ff9700 1 --2- 172.21.15.16:0/739706789 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f395406b020 0x7f395406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:23.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.580+0000 7f3964ff9700 1 -- 172.21.15.16:0/739706789 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6085+0+0 (secure 0 0 0) 0x7f395c088a30 con 0x7f39680cc8a0 2023-10-16T12:30:23.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.580+0000 7f3967fff700 1 --2- 172.21.15.16:0/739706789 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f395406b020 0x7f395406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:23.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.580+0000 7f3967fff700 1 --2- 172.21.15.16:0/739706789 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f395406b020 0x7f395406d4e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3958007900 tx=0x7f3958008040 comp rx=0 tx=0).ready entity=mgr.8388 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:23.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.584+0000 7f3964ff9700 1 -- 172.21.15.16:0/739706789 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f39680ac8d0 con 0x7f39680cc8a0 2023-10-16T12:30:23.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.791+0000 7f396e41b700 1 -- 172.21.15.16:0/739706789 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f39680ac8d0 con 0x7f39680cc8a0 2023-10-16T12:30:23.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.792+0000 7f3964ff9700 1 -- 172.21.15.16:0/739706789 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v141) v1 ==== 74+0+16777 (secure 0 0 0) 0x7f39680ac8d0 con 0x7f39680cc8a0 2023-10-16T12:30:23.794 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:30:23.795 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":141,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:30:22.807611+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:30:21.318498+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"141","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2023-10-16T12:30:21.813382+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"141","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:30:23.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.794+0000 7f396e41b700 1 -- 172.21.15.16:0/739706789 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f395406b020 msgr2=0x7f395406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:23.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.794+0000 7f396e41b700 1 --2- 172.21.15.16:0/739706789 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f395406b020 0x7f395406d4e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3958007900 tx=0x7f3958008040 comp rx=0 tx=0).stop 2023-10-16T12:30:23.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.794+0000 7f396e41b700 1 -- 172.21.15.16:0/739706789 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39680cc8a0 msgr2=0x7f3968158b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:23.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.794+0000 7f396e41b700 1 --2- 172.21.15.16:0/739706789 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39680cc8a0 0x7f3968158b60 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f395c00c4d0 tx=0x7f395c00c890 comp rx=0 tx=0).stop 2023-10-16T12:30:23.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.794+0000 7f396e41b700 1 -- 172.21.15.16:0/739706789 shutdown_connections 2023-10-16T12:30:23.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.794+0000 7f396e41b700 1 --2- 172.21.15.16:0/739706789 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f396815b280 0x7f39681f60e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:23.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.794+0000 7f396e41b700 1 --2- 172.21.15.16:0/739706789 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f395406b020 0x7f395406d4e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:23.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.794+0000 7f396e41b700 1 --2- 172.21.15.16:0/739706789 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f396806a7e0 0x7f39680c8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:23.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.794+0000 7f396e41b700 1 --2- 172.21.15.16:0/739706789 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39680cc8a0 0x7f3968158b60 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:23.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.794+0000 7f396e41b700 1 -- 172.21.15.16:0/739706789 >> 172.21.15.16:0/739706789 conn(0x7f39680b1fd0 msgr2=0x7f3968159c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:23.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.794+0000 7f396e41b700 1 -- 172.21.15.16:0/739706789 shutdown_connections 2023-10-16T12:30:23.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.794+0000 7f396e41b700 1 -- 172.21.15.16:0/739706789 wait complete. 2023-10-16T12:30:23.809 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:30:23.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.934+0000 7f5926b4f700 1 Processor -- start 2023-10-16T12:30:23.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.934+0000 7f5926b4f700 1 -- start start 2023-10-16T12:30:23.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.935+0000 7f5926b4f700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f592015b860 0x7f592006a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:23.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.935+0000 7f5926b4f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f59200b2be0 0x7f592006ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:23.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.935+0000 7f5926b4f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f59200cd770 0x7f59200719e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:23.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.935+0000 7f5926b4f700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59200d03d0 con 0x7f59200cd770 2023-10-16T12:30:23.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.935+0000 7f5926b4f700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59200d11c0 con 0x7f592015b860 2023-10-16T12:30:23.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.935+0000 7f5926b4f700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59200cd360 con 0x7f59200b2be0 2023-10-16T12:30:23.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.936+0000 7f59250ec700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f59200cd770 0x7f59200719e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:23.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.936+0000 7f59250ec700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f59200cd770 0x7f59200719e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45182/0 (socket says 172.21.15.16:45182) 2023-10-16T12:30:23.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.936+0000 7f59250ec700 1 -- 172.21.15.16:0/3412570058 learned_addr learned my addr 172.21.15.16:0/3412570058 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:23.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.936+0000 7f591ffff700 1 --2- 172.21.15.16:0/3412570058 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f59200b2be0 0x7f592006ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:23.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.936+0000 7f59248eb700 1 --2- 172.21.15.16:0/3412570058 >> v2:172.21.15.49:3300/0 conn(0x7f592015b860 0x7f592006a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:23.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.936+0000 7f59250ec700 1 -- 172.21.15.16:0/3412570058 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f59200b2be0 msgr2=0x7f592006ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:23.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.936+0000 7f59250ec700 1 --2- 172.21.15.16:0/3412570058 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f59200b2be0 0x7f592006ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:23.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.936+0000 7f59250ec700 1 -- 172.21.15.16:0/3412570058 >> v2:172.21.15.49:3300/0 conn(0x7f592015b860 msgr2=0x7f592006a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:23.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.936+0000 7f59250ec700 1 --2- 172.21.15.16:0/3412570058 >> v2:172.21.15.49:3300/0 conn(0x7f592015b860 0x7f592006a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:23.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.936+0000 7f59250ec700 1 -- 172.21.15.16:0/3412570058 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59200d05b0 con 0x7f59200cd770 2023-10-16T12:30:23.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.936+0000 7f59248eb700 1 --2- 172.21.15.16:0/3412570058 >> v2:172.21.15.49:3300/0 conn(0x7f592015b860 0x7f592006a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:30:23.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.936+0000 7f59250ec700 1 --2- 172.21.15.16:0/3412570058 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59200cd770 0x7f59200719e0 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f5920072e20 tx=0x7f591800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2624de05e482f28 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:23.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.936+0000 7f591f7fe700 1 -- 172.21.15.16:0/3412570058 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f591800e070 con 0x7f59200cd770 2023-10-16T12:30:23.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.936+0000 7f591f7fe700 1 -- 172.21.15.16:0/3412570058 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f591800b9c0 con 0x7f59200cd770 2023-10-16T12:30:23.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.936+0000 7f591f7fe700 1 -- 172.21.15.16:0/3412570058 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f591800bd70 con 0x7f59200cd770 2023-10-16T12:30:23.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.937+0000 7f5926b4f700 1 -- 172.21.15.16:0/3412570058 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59200cd770 msgr2=0x7f59200719e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:23.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.937+0000 7f5926b4f700 1 --2- 172.21.15.16:0/3412570058 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59200cd770 0x7f59200719e0 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f5920072e20 tx=0x7f591800b330 comp rx=0 tx=0).stop 2023-10-16T12:30:23.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.937+0000 7f5926b4f700 1 -- 172.21.15.16:0/3412570058 shutdown_connections 2023-10-16T12:30:23.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.937+0000 7f5926b4f700 1 --2- 172.21.15.16:0/3412570058 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59200cd770 0x7f59200719e0 unknown :-1 s=CLOSED pgs=1535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:23.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.937+0000 7f5926b4f700 1 --2- 172.21.15.16:0/3412570058 >> v2:172.21.15.49:3300/0 conn(0x7f592015b860 0x7f592006a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:23.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.937+0000 7f5926b4f700 1 --2- 172.21.15.16:0/3412570058 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f59200b2be0 0x7f592006ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:23.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.937+0000 7f5926b4f700 1 -- 172.21.15.16:0/3412570058 >> 172.21.15.16:0/3412570058 conn(0x7f59200b1fd0 msgr2=0x7f59200b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:23.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.937+0000 7f5926b4f700 1 -- 172.21.15.16:0/3412570058 shutdown_connections 2023-10-16T12:30:23.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.937+0000 7f5926b4f700 1 -- 172.21.15.16:0/3412570058 wait complete. 2023-10-16T12:30:23.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.938+0000 7f5926b4f700 1 Processor -- start 2023-10-16T12:30:23.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.938+0000 7f5926b4f700 1 -- start start 2023-10-16T12:30:23.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.939+0000 7f5926b4f700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f59200b2be0 0x7f59200a90a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:23.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.939+0000 7f5926b4f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59200cd770 0x7f59201f48e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:23.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.939+0000 7f5926b4f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f592015b860 0x7f5920158f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:23.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.939+0000 7f5926b4f700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59201f0970 con 0x7f59200cd770 2023-10-16T12:30:23.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.939+0000 7f5926b4f700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59200c8f10 con 0x7f59200b2be0 2023-10-16T12:30:23.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.939+0000 7f5926b4f700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59201f4e20 con 0x7f592015b860 2023-10-16T12:30:23.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.939+0000 7f591ffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59200cd770 0x7f59201f48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:23.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.939+0000 7f591ffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59200cd770 0x7f59201f48e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45186/0 (socket says 172.21.15.16:45186) 2023-10-16T12:30:23.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.939+0000 7f591ffff700 1 -- 172.21.15.16:0/2359071074 learned_addr learned my addr 172.21.15.16:0/2359071074 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:23.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.939+0000 7f59248eb700 1 --2- 172.21.15.16:0/2359071074 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f59200b2be0 0x7f59200a90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:23.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.939+0000 7f59250ec700 1 --2- 172.21.15.16:0/2359071074 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f592015b860 0x7f5920158f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:23.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.939+0000 7f591ffff700 1 -- 172.21.15.16:0/2359071074 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f592015b860 msgr2=0x7f5920158f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:23.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.939+0000 7f591ffff700 1 --2- 172.21.15.16:0/2359071074 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f592015b860 0x7f5920158f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:23.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.939+0000 7f591ffff700 1 -- 172.21.15.16:0/2359071074 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f59200b2be0 msgr2=0x7f59200a90a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:23.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.939+0000 7f591ffff700 1 --2- 172.21.15.16:0/2359071074 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f59200b2be0 0x7f59200a90a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:23.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.939+0000 7f591ffff700 1 -- 172.21.15.16:0/2359071074 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5918009b30 con 0x7f59200cd770 2023-10-16T12:30:23.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.940+0000 7f591ffff700 1 --2- 172.21.15.16:0/2359071074 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59200cd770 0x7f59201f48e0 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f590c00d800 tx=0x7f590c00db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:23.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.940+0000 7f591d7fa700 1 -- 172.21.15.16:0/2359071074 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f590c0125f0 con 0x7f59200cd770 2023-10-16T12:30:23.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.940+0000 7f591d7fa700 1 -- 172.21.15.16:0/2359071074 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f590c012790 con 0x7f59200cd770 2023-10-16T12:30:23.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.940+0000 7f5926b4f700 1 -- 172.21.15.16:0/2359071074 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59200c8300 con 0x7f59200cd770 2023-10-16T12:30:23.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.940+0000 7f591d7fa700 1 -- 172.21.15.16:0/2359071074 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f590c012b40 con 0x7f59200cd770 2023-10-16T12:30:23.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.940+0000 7f5926b4f700 1 -- 172.21.15.16:0/2359071074 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59200c88c0 con 0x7f59200cd770 2023-10-16T12:30:23.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.941+0000 7f5926b4f700 1 -- 172.21.15.16:0/2359071074 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59200c90a0 con 0x7f59200cd770 2023-10-16T12:30:23.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.941+0000 7f591d7fa700 1 -- 172.21.15.16:0/2359071074 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 74) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f590c012ca0 con 0x7f59200cd770 2023-10-16T12:30:23.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.942+0000 7f591d7fa700 1 --2- 172.21.15.16:0/2359071074 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f590406b020 0x7f590406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:23.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.942+0000 7f591d7fa700 1 -- 172.21.15.16:0/2359071074 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6085+0+0 (secure 0 0 0) 0x7f590c089a00 con 0x7f59200cd770 2023-10-16T12:30:23.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.942+0000 7f59248eb700 1 --2- 172.21.15.16:0/2359071074 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f590406b020 0x7f590406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:23.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.942+0000 7f59248eb700 1 --2- 172.21.15.16:0/2359071074 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f590406b020 0x7f590406d4e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5914006fd0 tx=0x7f5914008040 comp rx=0 tx=0).ready entity=mgr.8388 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:23.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:23.947+0000 7f591d7fa700 1 -- 172.21.15.16:0/2359071074 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f590c054830 con 0x7f59200cd770 2023-10-16T12:30:24.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.151+0000 7f5926b4f700 1 -- 172.21.15.16:0/2359071074 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f592004ebb0 con 0x7f59200cd770 2023-10-16T12:30:24.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.152+0000 7f591d7fa700 1 -- 172.21.15.16:0/2359071074 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v141) v1 ==== 74+0+16777 (secure 0 0 0) 0x7f590c057e50 con 0x7f59200cd770 2023-10-16T12:30:24.155 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:30:24.155 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":141,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:30:22.807611+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:30:21.318498+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"141","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2023-10-16T12:30:21.813382+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"141","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:30:24.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.155+0000 7f5926b4f700 1 -- 172.21.15.16:0/2359071074 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f590406b020 msgr2=0x7f590406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:24.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.155+0000 7f5926b4f700 1 --2- 172.21.15.16:0/2359071074 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f590406b020 0x7f590406d4e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5914006fd0 tx=0x7f5914008040 comp rx=0 tx=0).stop 2023-10-16T12:30:24.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.155+0000 7f5926b4f700 1 -- 172.21.15.16:0/2359071074 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59200cd770 msgr2=0x7f59201f48e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:24.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.155+0000 7f5926b4f700 1 --2- 172.21.15.16:0/2359071074 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59200cd770 0x7f59201f48e0 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f590c00d800 tx=0x7f590c00db10 comp rx=0 tx=0).stop 2023-10-16T12:30:24.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.155+0000 7f5926b4f700 1 -- 172.21.15.16:0/2359071074 shutdown_connections 2023-10-16T12:30:24.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.155+0000 7f5926b4f700 1 --2- 172.21.15.16:0/2359071074 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f592015b860 0x7f5920158f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:24.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.155+0000 7f5926b4f700 1 --2- 172.21.15.16:0/2359071074 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f590406b020 0x7f590406d4e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:24.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.155+0000 7f5926b4f700 1 --2- 172.21.15.16:0/2359071074 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f59200b2be0 0x7f59200a90a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:24.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.155+0000 7f5926b4f700 1 --2- 172.21.15.16:0/2359071074 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59200cd770 0x7f59201f48e0 unknown :-1 s=CLOSED pgs=1536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:24.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.155+0000 7f5926b4f700 1 -- 172.21.15.16:0/2359071074 >> 172.21.15.16:0/2359071074 conn(0x7f59200b1fd0 msgr2=0x7f59200b54e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:24.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.155+0000 7f5926b4f700 1 -- 172.21.15.16:0/2359071074 shutdown_connections 2023-10-16T12:30:24.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.155+0000 7f5926b4f700 1 -- 172.21.15.16:0/2359071074 wait complete. 2023-10-16T12:30:24.169 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-16T12:30:24.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.296+0000 7fac4d807700 1 Processor -- start 2023-10-16T12:30:24.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.296+0000 7fac4d807700 1 -- start start 2023-10-16T12:30:24.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.296+0000 7fac4d807700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fac480ce6a0 0x7fac4806c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:24.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.296+0000 7fac4d807700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fac480d2110 0x7fac4806c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:24.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.296+0000 7fac4d807700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac480b20f0 con 0x7fac480b2620 2023-10-16T12:30:24.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.296+0000 7fac4d807700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac480af1f0 con 0x7fac480d2110 2023-10-16T12:30:24.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.296+0000 7fac4d807700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac480d03d0 con 0x7fac480ce6a0 2023-10-16T12:30:24.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.297+0000 7fac467fc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fac480d2110 0x7fac4806c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:24.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.297+0000 7fac477fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fac480b2620 0x7fac480b0f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34470/0 (socket says 172.21.15.16:34470) 2023-10-16T12:30:24.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.297+0000 7fac467fc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fac480d2110 0x7fac4806c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45194/0 (socket says 172.21.15.16:45194) 2023-10-16T12:30:24.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.297+0000 7fac477fe700 1 -- 172.21.15.16:0/3900886271 learned_addr learned my addr 172.21.15.16:0/3900886271 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:30:24.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.297+0000 7fac46ffd700 1 --2- 172.21.15.16:0/3900886271 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fac480ce6a0 0x7fac4806c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:24.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.297+0000 7fac45ffb700 1 -- 172.21.15.16:0/3900886271 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4059973768 0 0) 0x7fac480b20f0 con 0x7fac480b2620 2023-10-16T12:30:24.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.297+0000 7fac45ffb700 1 -- 172.21.15.16:0/3900886271 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac300035b0 con 0x7fac480b2620 2023-10-16T12:30:24.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.297+0000 7fac467fc700 1 -- 172.21.15.16:0/3900886271 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fac480ce6a0 msgr2=0x7fac4806c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:24.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.297+0000 7fac467fc700 1 --2- 172.21.15.16:0/3900886271 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fac480ce6a0 0x7fac4806c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:24.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.297+0000 7fac467fc700 1 -- 172.21.15.16:0/3900886271 >> v1:172.21.15.16:6789/0 conn(0x7fac480b2620 legacy=0x7fac480b0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:24.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.297+0000 7fac467fc700 1 -- 172.21.15.16:0/3900886271 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac480b2340 con 0x7fac480d2110 2023-10-16T12:30:24.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.297+0000 7fac46ffd700 1 --2- 172.21.15.16:0/3900886271 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fac480ce6a0 0x7fac4806c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:30:24.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.297+0000 7fac467fc700 1 --2- 172.21.15.16:0/3900886271 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac480d2110 0x7fac4806c840 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7fac3c009840 tx=0x7fac3c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81b683c1e4306b07 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:24.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.297+0000 7fac45ffb700 1 -- 172.21.15.16:0/3900886271 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac3c011830 con 0x7fac480d2110 2023-10-16T12:30:24.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.297+0000 7fac45ffb700 1 -- 172.21.15.16:0/3900886271 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac3c011990 con 0x7fac480d2110 2023-10-16T12:30:24.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.297+0000 7fac45ffb700 1 -- 172.21.15.16:0/3900886271 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac3c011d00 con 0x7fac480d2110 2023-10-16T12:30:24.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.298+0000 7fac4d807700 1 -- 172.21.15.16:0/3900886271 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac480d2110 msgr2=0x7fac4806c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:24.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.298+0000 7fac4d807700 1 --2- 172.21.15.16:0/3900886271 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac480d2110 0x7fac4806c840 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7fac3c009840 tx=0x7fac3c009b50 comp rx=0 tx=0).stop 2023-10-16T12:30:24.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.298+0000 7fac4d807700 1 -- 172.21.15.16:0/3900886271 shutdown_connections 2023-10-16T12:30:24.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.298+0000 7fac4d807700 1 --2- 172.21.15.16:0/3900886271 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fac480ce6a0 0x7fac4806c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:24.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.298+0000 7fac4d807700 1 --2- 172.21.15.16:0/3900886271 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac480d2110 0x7fac4806c840 unknown :-1 s=CLOSED pgs=1538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:24.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.298+0000 7fac4d807700 1 -- 172.21.15.16:0/3900886271 >> 172.21.15.16:0/3900886271 conn(0x7fac480d15f0 msgr2=0x7fac480d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:24.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.298+0000 7fac4d807700 1 -- 172.21.15.16:0/3900886271 shutdown_connections 2023-10-16T12:30:24.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.298+0000 7fac4d807700 1 -- 172.21.15.16:0/3900886271 wait complete. 2023-10-16T12:30:24.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.299+0000 7fac4d807700 1 Processor -- start 2023-10-16T12:30:24.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.299+0000 7fac4d807700 1 -- start start 2023-10-16T12:30:24.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.300+0000 7fac4d807700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac480b2620 0x7fac480b4170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:24.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.300+0000 7fac4d807700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fac480ce6a0 0x7fac48151a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:24.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.300+0000 7fac4d807700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fac480d2110 0x7fac480b6580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:24.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.300+0000 7fac4d807700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac48151f40 con 0x7fac480b2620 2023-10-16T12:30:24.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.300+0000 7fac4d807700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac480c6460 con 0x7fac480ce6a0 2023-10-16T12:30:24.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.300+0000 7fac4d807700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac480b4870 con 0x7fac480d2110 2023-10-16T12:30:24.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.300+0000 7fac46ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac480b2620 0x7fac480b4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:24.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.300+0000 7fac46ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac480b2620 0x7fac480b4170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45198/0 (socket says 172.21.15.16:45198) 2023-10-16T12:30:24.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.300+0000 7fac46ffd700 1 -- 172.21.15.16:0/1289121818 learned_addr learned my addr 172.21.15.16:0/1289121818 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:24.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.300+0000 7fac467fc700 1 --2- 172.21.15.16:0/1289121818 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fac480ce6a0 0x7fac48151a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:24.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.300+0000 7fac477fe700 1 --2- 172.21.15.16:0/1289121818 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fac480d2110 0x7fac480b6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:24.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.301+0000 7fac46ffd700 1 -- 172.21.15.16:0/1289121818 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fac480d2110 msgr2=0x7fac480b6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:24.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.301+0000 7fac46ffd700 1 --2- 172.21.15.16:0/1289121818 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fac480d2110 0x7fac480b6580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:24.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.301+0000 7fac46ffd700 1 -- 172.21.15.16:0/1289121818 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fac480ce6a0 msgr2=0x7fac48151a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:24.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.301+0000 7fac46ffd700 1 --2- 172.21.15.16:0/1289121818 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fac480ce6a0 0x7fac48151a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:24.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.301+0000 7fac46ffd700 1 -- 172.21.15.16:0/1289121818 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac3c0094a0 con 0x7fac480b2620 2023-10-16T12:30:24.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.301+0000 7fac46ffd700 1 --2- 172.21.15.16:0/1289121818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac480b2620 0x7fac480b4170 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7fac3800e810 tx=0x7fac3800eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:24.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.301+0000 7fac2ffff700 1 -- 172.21.15.16:0/1289121818 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac3800c070 con 0x7fac480b2620 2023-10-16T12:30:24.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.301+0000 7fac2ffff700 1 -- 172.21.15.16:0/1289121818 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac38004230 con 0x7fac480b2620 2023-10-16T12:30:24.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.301+0000 7fac4d807700 1 -- 172.21.15.16:0/1289121818 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac480b4b50 con 0x7fac480b2620 2023-10-16T12:30:24.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.301+0000 7fac2ffff700 1 -- 172.21.15.16:0/1289121818 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac380036a0 con 0x7fac480b2620 2023-10-16T12:30:24.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.301+0000 7fac4d807700 1 -- 172.21.15.16:0/1289121818 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac480b5110 con 0x7fac480b2620 2023-10-16T12:30:24.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.302+0000 7fac4d807700 1 -- 172.21.15.16:0/1289121818 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac4804ebb0 con 0x7fac480b2620 2023-10-16T12:30:24.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.302+0000 7fac2ffff700 1 -- 172.21.15.16:0/1289121818 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 74) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fac38003800 con 0x7fac480b2620 2023-10-16T12:30:24.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.303+0000 7fac2ffff700 1 --2- 172.21.15.16:0/1289121818 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7fac3406b020 0x7fac3406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:24.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.303+0000 7fac2ffff700 1 -- 172.21.15.16:0/1289121818 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6085+0+0 (secure 0 0 0) 0x7fac38089700 con 0x7fac480b2620 2023-10-16T12:30:24.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.303+0000 7fac467fc700 1 --2- 172.21.15.16:0/1289121818 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7fac3406b020 0x7fac3406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:24.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.303+0000 7fac467fc700 1 --2- 172.21.15.16:0/1289121818 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7fac3406b020 0x7fac3406d4e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fac3c000c00 tx=0x7fac3c01c040 comp rx=0 tx=0).ready entity=mgr.8388 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:24.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.307+0000 7fac2ffff700 1 -- 172.21.15.16:0/1289121818 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fac38054370 con 0x7fac480b2620 2023-10-16T12:30:24.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:24.567+0000 7fac4d807700 1 -- 172.21.15.16:0/1289121818 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fac480d19d0 con 0x7fac480b2620 2023-10-16T12:30:25.379 INFO:tasks.ceph.mds.b.smithi049.stderr:2023-10-16T12:30:25.377+0000 7fa149c9f700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T12:30:25.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:25.416+0000 7fac2ffff700 1 -- 172.21.15.16:0/1289121818 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v69) v1 ==== 118+0+0 (secure 0 0 0) 0x7fac38057990 con 0x7fac480b2620 2023-10-16T12:30:25.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:25.419+0000 7fac4d807700 1 -- 172.21.15.16:0/1289121818 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7fac3406b020 msgr2=0x7fac3406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:25.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:25.419+0000 7fac4d807700 1 --2- 172.21.15.16:0/1289121818 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7fac3406b020 0x7fac3406d4e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fac3c000c00 tx=0x7fac3c01c040 comp rx=0 tx=0).stop 2023-10-16T12:30:25.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:25.419+0000 7fac4d807700 1 -- 172.21.15.16:0/1289121818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac480b2620 msgr2=0x7fac480b4170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:25.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:25.419+0000 7fac4d807700 1 --2- 172.21.15.16:0/1289121818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac480b2620 0x7fac480b4170 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7fac3800e810 tx=0x7fac3800eb20 comp rx=0 tx=0).stop 2023-10-16T12:30:25.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:25.419+0000 7fac4d807700 1 -- 172.21.15.16:0/1289121818 shutdown_connections 2023-10-16T12:30:25.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:25.419+0000 7fac4d807700 1 --2- 172.21.15.16:0/1289121818 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fac480d2110 0x7fac480b6580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:25.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:25.419+0000 7fac4d807700 1 --2- 172.21.15.16:0/1289121818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac480b2620 0x7fac480b4170 unknown :-1 s=CLOSED pgs=1539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:25.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:25.419+0000 7fac4d807700 1 --2- 172.21.15.16:0/1289121818 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7fac3406b020 0x7fac3406d4e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:25.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:25.419+0000 7fac4d807700 1 --2- 172.21.15.16:0/1289121818 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fac480ce6a0 0x7fac48151a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:25.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:25.419+0000 7fac4d807700 1 -- 172.21.15.16:0/1289121818 >> 172.21.15.16:0/1289121818 conn(0x7fac480d15f0 msgr2=0x7fac480cdc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:25.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:25.419+0000 7fac4d807700 1 -- 172.21.15.16:0/1289121818 shutdown_connections 2023-10-16T12:30:25.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:25.419+0000 7fac4d807700 1 -- 172.21.15.16:0/1289121818 wait complete. 2023-10-16T12:30:25.436 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-16T12:30:25.437 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-16T12:30:25.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:25.570+0000 7fa3d451d700 1 Processor -- start 2023-10-16T12:30:25.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:25.570+0000 7fa3d451d700 1 -- start start 2023-10-16T12:30:25.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:25.571+0000 7fa3d451d700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fa3cc0af210 0x7fa3cc0ce180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:25.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:25.571+0000 7fa3d451d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa3cc0ce6c0 0x7fa3cc06aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:25.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:25.571+0000 7fa3d451d700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3cc06aa10 con 0x7fa3cc160ee0 2023-10-16T12:30:25.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:25.571+0000 7fa3d451d700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3cc0cf770 con 0x7fa3cc0af210 2023-10-16T12:30:25.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:25.571+0000 7fa3d451d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3cc0ceed0 con 0x7fa3cc0ce6c0 2023-10-16T12:30:25.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:25.571+0000 7fa3d22b9700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa3cc160ee0 0x7fa3cc0aeb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34480/0 (socket says 172.21.15.16:34480) 2023-10-16T12:30:25.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:25.571+0000 7fa3d22b9700 1 -- 172.21.15.16:0/1910251760 learned_addr learned my addr 172.21.15.16:0/1910251760 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:30:25.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:25.571+0000 7fa3d1ab8700 1 --2- 172.21.15.16:0/1910251760 >> v2:172.21.15.49:3300/0 conn(0x7fa3cc0af210 0x7fa3cc0ce180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:25.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:25.571+0000 7fa3d2aba700 1 --2- 172.21.15.16:0/1910251760 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa3cc0ce6c0 0x7fa3cc06aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:25.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:25.571+0000 7fa3d12b7700 1 -- 172.21.15.16:0/1910251760 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2129424988 0 0) 0x7fa3cc06aa10 con 0x7fa3cc160ee0 2023-10-16T12:30:25.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:25.571+0000 7fa3d12b7700 1 -- 172.21.15.16:0/1910251760 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3b80035b0 con 0x7fa3cc160ee0 2023-10-16T12:30:25.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:25.572+0000 7fa3d1ab8700 1 -- 172.21.15.16:0/1910251760 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa3cc0ce6c0 msgr2=0x7fa3cc06aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:25.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:25.572+0000 7fa3d1ab8700 1 --2- 172.21.15.16:0/1910251760 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa3cc0ce6c0 0x7fa3cc06aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:25.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:25.572+0000 7fa3d1ab8700 1 -- 172.21.15.16:0/1910251760 >> v1:172.21.15.16:6789/0 conn(0x7fa3cc160ee0 legacy=0x7fa3cc0aeb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:25.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:25.572+0000 7fa3d1ab8700 1 -- 172.21.15.16:0/1910251760 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3cc0cf1b0 con 0x7fa3cc0af210 2023-10-16T12:30:25.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:25.572+0000 7fa3d1ab8700 1 --2- 172.21.15.16:0/1910251760 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa3cc0af210 0x7fa3cc0ce180 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fa3c8009840 tx=0x7fa3c8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=842dee3bb631a2d3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:25.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:25.572+0000 7fa3d12b7700 1 -- 172.21.15.16:0/1910251760 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c8011830 con 0x7fa3cc0af210 2023-10-16T12:30:25.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:25.572+0000 7fa3d12b7700 1 -- 172.21.15.16:0/1910251760 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3c8011990 con 0x7fa3cc0af210 2023-10-16T12:30:25.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:25.572+0000 7fa3d12b7700 1 -- 172.21.15.16:0/1910251760 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c8011d00 con 0x7fa3cc0af210 2023-10-16T12:30:25.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:25.572+0000 7fa3d451d700 1 -- 172.21.15.16:0/1910251760 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa3cc0af210 msgr2=0x7fa3cc0ce180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:25.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:25.572+0000 7fa3d451d700 1 --2- 172.21.15.16:0/1910251760 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa3cc0af210 0x7fa3cc0ce180 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fa3c8009840 tx=0x7fa3c8009b50 comp rx=0 tx=0).stop 2023-10-16T12:30:25.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:25.573+0000 7fa3d451d700 1 -- 172.21.15.16:0/1910251760 shutdown_connections 2023-10-16T12:30:25.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:25.573+0000 7fa3d451d700 1 --2- 172.21.15.16:0/1910251760 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa3cc0ce6c0 0x7fa3cc06aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:25.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:25.573+0000 7fa3d451d700 1 --2- 172.21.15.16:0/1910251760 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa3cc0af210 0x7fa3cc0ce180 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:25.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:25.573+0000 7fa3d451d700 1 -- 172.21.15.16:0/1910251760 >> 172.21.15.16:0/1910251760 conn(0x7fa3cc0cc5e0 msgr2=0x7fa3cc0cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:25.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:25.573+0000 7fa3d451d700 1 -- 172.21.15.16:0/1910251760 shutdown_connections 2023-10-16T12:30:25.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:25.573+0000 7fa3d451d700 1 -- 172.21.15.16:0/1910251760 wait complete. 2023-10-16T12:30:25.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:25.574+0000 7fa3d451d700 1 Processor -- start 2023-10-16T12:30:25.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:25.574+0000 7fa3d451d700 1 -- start start 2023-10-16T12:30:25.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:25.575+0000 7fa3d451d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3cc0af210 0x7fa3cc15d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:25.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:25.575+0000 7fa3d451d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa3cc0ce6c0 0x7fa3cc144b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:25.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:25.575+0000 7fa3d451d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa3cc160ee0 0x7fa3cc147d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:25.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:25.575+0000 7fa3d451d700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3cc0c6620 con 0x7fa3cc0af210 2023-10-16T12:30:25.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:25.575+0000 7fa3d451d700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3cc148270 con 0x7fa3cc0ce6c0 2023-10-16T12:30:25.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:25.575+0000 7fa3d451d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3cc15dc60 con 0x7fa3cc160ee0 2023-10-16T12:30:25.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:25.575+0000 7fa3d22b9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3cc0af210 0x7fa3cc15d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:25.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:25.575+0000 7fa3d22b9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3cc0af210 0x7fa3cc15d5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45212/0 (socket says 172.21.15.16:45212) 2023-10-16T12:30:25.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:25.575+0000 7fa3d22b9700 1 -- 172.21.15.16:0/21377729 learned_addr learned my addr 172.21.15.16:0/21377729 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:25.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:25.575+0000 7fa3d1ab8700 1 --2- 172.21.15.16:0/21377729 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa3cc0ce6c0 0x7fa3cc144b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:25.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:25.575+0000 7fa3d2aba700 1 --2- 172.21.15.16:0/21377729 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa3cc160ee0 0x7fa3cc147d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:25.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:25.575+0000 7fa3d22b9700 1 -- 172.21.15.16:0/21377729 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa3cc160ee0 msgr2=0x7fa3cc147d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:25.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:25.575+0000 7fa3d22b9700 1 --2- 172.21.15.16:0/21377729 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa3cc160ee0 0x7fa3cc147d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:25.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:25.575+0000 7fa3d22b9700 1 -- 172.21.15.16:0/21377729 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa3cc0ce6c0 msgr2=0x7fa3cc144b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:25.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:25.575+0000 7fa3d22b9700 1 --2- 172.21.15.16:0/21377729 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa3cc0ce6c0 0x7fa3cc144b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:25.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:25.575+0000 7fa3d22b9700 1 -- 172.21.15.16:0/21377729 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3c80094a0 con 0x7fa3cc0af210 2023-10-16T12:30:25.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:25.575+0000 7fa3d2aba700 1 --2- 172.21.15.16:0/21377729 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa3cc160ee0 0x7fa3cc147d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:30:25.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:25.575+0000 7fa3d22b9700 1 --2- 172.21.15.16:0/21377729 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3cc0af210 0x7fa3cc15d5a0 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7fa3bc00c4d0 tx=0x7fa3bc00c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:25.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:25.575+0000 7fa3c2ffd700 1 -- 172.21.15.16:0/21377729 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3bc005d70 con 0x7fa3cc0af210 2023-10-16T12:30:25.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:25.575+0000 7fa3c2ffd700 1 -- 172.21.15.16:0/21377729 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3bc0070d0 con 0x7fa3cc0af210 2023-10-16T12:30:25.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:25.575+0000 7fa3d451d700 1 -- 172.21.15.16:0/21377729 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3cc15df40 con 0x7fa3cc0af210 2023-10-16T12:30:25.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:25.575+0000 7fa3c2ffd700 1 -- 172.21.15.16:0/21377729 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3bc007440 con 0x7fa3cc0af210 2023-10-16T12:30:25.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:25.575+0000 7fa3d451d700 1 -- 172.21.15.16:0/21377729 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3cc15e500 con 0x7fa3cc0af210 2023-10-16T12:30:25.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:25.576+0000 7fa3d451d700 1 -- 172.21.15.16:0/21377729 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3cc04ebb0 con 0x7fa3cc0af210 2023-10-16T12:30:25.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:25.577+0000 7fa3c2ffd700 1 -- 172.21.15.16:0/21377729 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 74) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa3bc0075a0 con 0x7fa3cc0af210 2023-10-16T12:30:25.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:25.577+0000 7fa3c2ffd700 1 --2- 172.21.15.16:0/21377729 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7fa3b006b020 0x7fa3b006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:25.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:25.577+0000 7fa3c2ffd700 1 -- 172.21.15.16:0/21377729 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 6085+0+0 (secure 0 0 0) 0x7fa3bc088d90 con 0x7fa3cc0af210 2023-10-16T12:30:25.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:25.578+0000 7fa3d1ab8700 1 --2- 172.21.15.16:0/21377729 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7fa3b006b020 0x7fa3b006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:25.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:25.578+0000 7fa3d1ab8700 1 --2- 172.21.15.16:0/21377729 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7fa3b006b020 0x7fa3b006d4e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa3c8000c00 tx=0x7fa3c801c040 comp rx=0 tx=0).ready entity=mgr.8388 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:25.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:25.582+0000 7fa3c2ffd700 1 -- 172.21.15.16:0/21377729 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa3bc053bc0 con 0x7fa3cc0af210 2023-10-16T12:30:25.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:25.835+0000 7fa3d451d700 1 -- 172.21.15.16:0/21377729 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fa3cc0cf460 con 0x7fa3cc0af210 2023-10-16T12:30:26.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.417+0000 7fa3c2ffd700 1 -- 172.21.15.16:0/21377729 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v70) v1 ==== 122+0+0 (secure 0 0 0) 0x7fa3bc0571e0 con 0x7fa3cc0af210 2023-10-16T12:30:26.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.420+0000 7fa3d451d700 1 -- 172.21.15.16:0/21377729 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7fa3b006b020 msgr2=0x7fa3b006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:26.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.420+0000 7fa3d451d700 1 --2- 172.21.15.16:0/21377729 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7fa3b006b020 0x7fa3b006d4e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa3c8000c00 tx=0x7fa3c801c040 comp rx=0 tx=0).stop 2023-10-16T12:30:26.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.420+0000 7fa3d451d700 1 -- 172.21.15.16:0/21377729 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3cc0af210 msgr2=0x7fa3cc15d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:26.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.420+0000 7fa3d451d700 1 --2- 172.21.15.16:0/21377729 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3cc0af210 0x7fa3cc15d5a0 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7fa3bc00c4d0 tx=0x7fa3bc00c7e0 comp rx=0 tx=0).stop 2023-10-16T12:30:26.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.420+0000 7fa3d451d700 1 -- 172.21.15.16:0/21377729 shutdown_connections 2023-10-16T12:30:26.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.420+0000 7fa3d451d700 1 --2- 172.21.15.16:0/21377729 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa3cc160ee0 0x7fa3cc147d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:26.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.420+0000 7fa3d451d700 1 --2- 172.21.15.16:0/21377729 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3cc0af210 0x7fa3cc15d5a0 unknown :-1 s=CLOSED pgs=1541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:26.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.420+0000 7fa3d451d700 1 --2- 172.21.15.16:0/21377729 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7fa3b006b020 0x7fa3b006d4e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:26.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.420+0000 7fa3d451d700 1 --2- 172.21.15.16:0/21377729 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa3cc0ce6c0 0x7fa3cc144b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:26.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.420+0000 7fa3d451d700 1 -- 172.21.15.16:0/21377729 >> 172.21.15.16:0/21377729 conn(0x7fa3cc0cc5e0 msgr2=0x7fa3cc0b0cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:26.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.420+0000 7fa3d451d700 1 -- 172.21.15.16:0/21377729 shutdown_connections 2023-10-16T12:30:26.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.420+0000 7fa3d451d700 1 -- 172.21.15.16:0/21377729 wait complete. 2023-10-16T12:30:26.437 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-16T12:30:26.438 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:30:26.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.569+0000 7f597584c700 1 Processor -- start 2023-10-16T12:30:26.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.569+0000 7f597584c700 1 -- start start 2023-10-16T12:30:26.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.569+0000 7f597584c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f597015d8d0 0x7f59700d1c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:26.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.569+0000 7f597584c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f59700d41c0 0x7f59700d0040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:26.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.570+0000 7f597584c700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f59700d0580 0x7f59700ce240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:26.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.570+0000 7f597584c700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59700d0a10 con 0x7f597015d8d0 2023-10-16T12:30:26.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.570+0000 7f597584c700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f597006ba00 con 0x7f59700d0580 2023-10-16T12:30:26.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.570+0000 7f597584c700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59700d2150 con 0x7f59700d41c0 2023-10-16T12:30:26.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.570+0000 7f596effd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f597015d8d0 0x7f59700d1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:26.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.570+0000 7f596effd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f597015d8d0 0x7f59700d1c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45220/0 (socket says 172.21.15.16:45220) 2023-10-16T12:30:26.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.570+0000 7f596effd700 1 -- 172.21.15.16:0/931998975 learned_addr learned my addr 172.21.15.16:0/931998975 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:26.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.570+0000 7f596effd700 1 -- 172.21.15.16:0/931998975 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f59700d41c0 msgr2=0x7f59700d0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:26.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.570+0000 7f596effd700 1 --2- 172.21.15.16:0/931998975 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f59700d41c0 0x7f59700d0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:26.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.570+0000 7f596effd700 1 -- 172.21.15.16:0/931998975 >> v2:172.21.15.49:3300/0 conn(0x7f59700d0580 msgr2=0x7f59700ce240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:26.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.570+0000 7f596effd700 1 --2- 172.21.15.16:0/931998975 >> v2:172.21.15.49:3300/0 conn(0x7f59700d0580 0x7f59700ce240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:26.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.570+0000 7f596effd700 1 -- 172.21.15.16:0/931998975 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59700d26d0 con 0x7f597015d8d0 2023-10-16T12:30:26.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.570+0000 7f596effd700 1 --2- 172.21.15.16:0/931998975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f597015d8d0 0x7f59700d1c10 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7f59700c70f0 tx=0x7f5960009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2972ae752a6cfaa2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:26.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.570+0000 7f596dffb700 1 -- 172.21.15.16:0/931998975 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f596000b870 con 0x7f597015d8d0 2023-10-16T12:30:26.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.570+0000 7f596dffb700 1 -- 172.21.15.16:0/931998975 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f596000b9d0 con 0x7f597015d8d0 2023-10-16T12:30:26.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.570+0000 7f596dffb700 1 -- 172.21.15.16:0/931998975 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f596000bd60 con 0x7f597015d8d0 2023-10-16T12:30:26.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.571+0000 7f597584c700 1 -- 172.21.15.16:0/931998975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f597015d8d0 msgr2=0x7f59700d1c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:26.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.571+0000 7f597584c700 1 --2- 172.21.15.16:0/931998975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f597015d8d0 0x7f59700d1c10 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7f59700c70f0 tx=0x7f5960009bd0 comp rx=0 tx=0).stop 2023-10-16T12:30:26.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.571+0000 7f597584c700 1 -- 172.21.15.16:0/931998975 shutdown_connections 2023-10-16T12:30:26.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.571+0000 7f597584c700 1 --2- 172.21.15.16:0/931998975 >> v2:172.21.15.49:3300/0 conn(0x7f59700d0580 0x7f59700ce240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:26.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.571+0000 7f597584c700 1 --2- 172.21.15.16:0/931998975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f597015d8d0 0x7f59700d1c10 unknown :-1 s=CLOSED pgs=1542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:26.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.571+0000 7f597584c700 1 --2- 172.21.15.16:0/931998975 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f59700d41c0 0x7f59700d0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:26.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.571+0000 7f597584c700 1 -- 172.21.15.16:0/931998975 >> 172.21.15.16:0/931998975 conn(0x7f59700d3630 msgr2=0x7f59700d3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:26.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.571+0000 7f597584c700 1 -- 172.21.15.16:0/931998975 shutdown_connections 2023-10-16T12:30:26.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.571+0000 7f597584c700 1 -- 172.21.15.16:0/931998975 wait complete. 2023-10-16T12:30:26.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.572+0000 7f597584c700 1 Processor -- start 2023-10-16T12:30:26.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.572+0000 7f597584c700 1 -- start start 2023-10-16T12:30:26.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.573+0000 7f597584c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59700d0580 0x7f59700cd040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:26.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.573+0000 7f597584c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f59700d41c0 0x7f59701e6950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:26.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.573+0000 7f597584c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f597015d8d0 0x7f59701e6e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:26.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.573+0000 7f597584c700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f597015b1a0 con 0x7f59700d0580 2023-10-16T12:30:26.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.573+0000 7f597584c700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59700cab10 con 0x7f59700d41c0 2023-10-16T12:30:26.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.573+0000 7f597584c700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59701e5c60 con 0x7f597015d8d0 2023-10-16T12:30:26.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.573+0000 7f596effd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59700d0580 0x7f59700cd040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:26.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.573+0000 7f596effd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59700d0580 0x7f59700cd040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45226/0 (socket says 172.21.15.16:45226) 2023-10-16T12:30:26.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.573+0000 7f596effd700 1 -- 172.21.15.16:0/2173874314 learned_addr learned my addr 172.21.15.16:0/2173874314 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:26.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.574+0000 7f596f7fe700 1 --2- 172.21.15.16:0/2173874314 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f597015d8d0 0x7f59701e6e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:26.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.574+0000 7f596effd700 1 -- 172.21.15.16:0/2173874314 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f597015d8d0 msgr2=0x7f59701e6e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:26.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.574+0000 7f596effd700 1 --2- 172.21.15.16:0/2173874314 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f597015d8d0 0x7f59701e6e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:26.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.574+0000 7f596effd700 1 -- 172.21.15.16:0/2173874314 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f59700d41c0 msgr2=0x7f59701e6950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:26.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.574+0000 7f596effd700 1 --2- 172.21.15.16:0/2173874314 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f59700d41c0 0x7f59701e6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:26.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.574+0000 7f596effd700 1 -- 172.21.15.16:0/2173874314 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5960009550 con 0x7f59700d0580 2023-10-16T12:30:26.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.574+0000 7f596effd700 1 --2- 172.21.15.16:0/2173874314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59700d0580 0x7f59700cd040 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7f597008db90 tx=0x7f5960004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:26.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.574+0000 7f5967fff700 1 -- 172.21.15.16:0/2173874314 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5960004310 con 0x7f59700d0580 2023-10-16T12:30:26.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.574+0000 7f5967fff700 1 -- 172.21.15.16:0/2173874314 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f596001f070 con 0x7f59700d0580 2023-10-16T12:30:26.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.574+0000 7f597584c700 1 -- 172.21.15.16:0/2173874314 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59701e5f40 con 0x7f59700d0580 2023-10-16T12:30:26.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.574+0000 7f5967fff700 1 -- 172.21.15.16:0/2173874314 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59600036a0 con 0x7f59700d0580 2023-10-16T12:30:26.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.574+0000 7f597584c700 1 -- 172.21.15.16:0/2173874314 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59701e6220 con 0x7f59700d0580 2023-10-16T12:30:26.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.575+0000 7f597584c700 1 -- 172.21.15.16:0/2173874314 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5970067870 con 0x7f59700d0580 2023-10-16T12:30:26.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.575+0000 7f5967fff700 1 -- 172.21.15.16:0/2173874314 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 74) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5960021030 con 0x7f59700d0580 2023-10-16T12:30:26.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.576+0000 7f5967fff700 1 --2- 172.21.15.16:0/2173874314 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f595006b020 0x7f595006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:26.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.576+0000 7f5967fff700 1 -- 172.21.15.16:0/2173874314 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 6085+0+0 (secure 0 0 0) 0x7f596008add0 con 0x7f59700d0580 2023-10-16T12:30:26.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.576+0000 7f596e7fc700 1 --2- 172.21.15.16:0/2173874314 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f595006b020 0x7f595006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:26.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.576+0000 7f596e7fc700 1 --2- 172.21.15.16:0/2173874314 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f595006b020 0x7f595006d4e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5958006fd0 tx=0x7f5958006d40 comp rx=0 tx=0).ready entity=mgr.8388 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:26.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.581+0000 7f5967fff700 1 -- 172.21.15.16:0/2173874314 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5960055da0 con 0x7f59700d0580 2023-10-16T12:30:26.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.830+0000 7f597584c700 1 -- 172.21.15.16:0/2173874314 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f597006bc70 con 0x7f59700d0580 2023-10-16T12:30:26.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.831+0000 7f5967fff700 1 -- 172.21.15.16:0/2173874314 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+4915 (secure 0 0 0) 0x7f59600593c0 con 0x7f59700d0580 2023-10-16T12:30:26.834 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:30:26.834 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":70,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7220,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/4274593615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":4274593615},{"type":"v1","addr":"172.21.15.16:6837","nonce":4274593615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":8625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/1624531381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1624531381},{"type":"v1","addr":"172.21.15.16:6836","nonce":1624531381}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":8637,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/1629597454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":1629597454},{"type":"v1","addr":"172.21.15.49:6834","nonce":1629597454}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:30:22.807943+0000","modified":"2023-10-16T12:30:25.836865+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8640},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8640":{"gid":8640,"name":"b","rank":0,"incarnation":66,"state":"up:active","state_seq":3,"addr":"172.21.15.49:6835/2008024326","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":2008024326},{"type":"v1","addr":"172.21.15.49:6835","nonce":2008024326}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2023-10-16T12:30:26.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.833+0000 7f597584c700 1 -- 172.21.15.16:0/2173874314 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f595006b020 msgr2=0x7f595006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:26.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.833+0000 7f597584c700 1 --2- 172.21.15.16:0/2173874314 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f595006b020 0x7f595006d4e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5958006fd0 tx=0x7f5958006d40 comp rx=0 tx=0).stop 2023-10-16T12:30:26.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.833+0000 7f597584c700 1 -- 172.21.15.16:0/2173874314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59700d0580 msgr2=0x7f59700cd040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:26.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.833+0000 7f597584c700 1 --2- 172.21.15.16:0/2173874314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59700d0580 0x7f59700cd040 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7f597008db90 tx=0x7f5960004750 comp rx=0 tx=0).stop 2023-10-16T12:30:26.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.833+0000 7f597584c700 1 -- 172.21.15.16:0/2173874314 shutdown_connections 2023-10-16T12:30:26.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.833+0000 7f597584c700 1 --2- 172.21.15.16:0/2173874314 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f597015d8d0 0x7f59701e6e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:26.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.833+0000 7f597584c700 1 --2- 172.21.15.16:0/2173874314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59700d0580 0x7f59700cd040 unknown :-1 s=CLOSED pgs=1543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:26.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.833+0000 7f597584c700 1 --2- 172.21.15.16:0/2173874314 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f595006b020 0x7f595006d4e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:26.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.833+0000 7f597584c700 1 --2- 172.21.15.16:0/2173874314 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f59700d41c0 0x7f59701e6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:26.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.833+0000 7f597584c700 1 -- 172.21.15.16:0/2173874314 >> 172.21.15.16:0/2173874314 conn(0x7f59700d3630 msgr2=0x7f597015ca70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:26.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.833+0000 7f597584c700 1 -- 172.21.15.16:0/2173874314 shutdown_connections 2023-10-16T12:30:26.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.833+0000 7f597584c700 1 -- 172.21.15.16:0/2173874314 wait complete. 2023-10-16T12:30:26.841 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 70 2023-10-16T12:30:26.848 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:30:26.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.973+0000 7f15a26f1700 1 Processor -- start 2023-10-16T12:30:26.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.973+0000 7f15a26f1700 1 -- start start 2023-10-16T12:30:26.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.974+0000 7f15a26f1700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f159c0cc8a0 0x7f159c15e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:26.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.974+0000 7f15a26f1700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f159c06a7e0 0x7f159c06ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:26.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.974+0000 7f15a26f1700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f159c06b170 con 0x7f159c15b280 2023-10-16T12:30:26.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.974+0000 7f15a26f1700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f159c161c30 con 0x7f159c06a7e0 2023-10-16T12:30:26.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.974+0000 7f15a26f1700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f159c161f10 con 0x7f159c0cc8a0 2023-10-16T12:30:26.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.974+0000 7f159bfff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f159c15b280 0x7f159c1613e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34508/0 (socket says 172.21.15.16:34508) 2023-10-16T12:30:26.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.974+0000 7f159bfff700 1 -- 172.21.15.16:0/3926722378 learned_addr learned my addr 172.21.15.16:0/3926722378 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:30:26.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.974+0000 7f15a0c8e700 1 --2- 172.21.15.16:0/3926722378 >> v2:172.21.15.16:3300/0 conn(0x7f159c06a7e0 0x7f159c06ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:26.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.974+0000 7f159b7fe700 1 --2- 172.21.15.16:0/3926722378 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f159c0cc8a0 0x7f159c15e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:26.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.974+0000 7f159affd700 1 -- 172.21.15.16:0/3926722378 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 164549849 0 0) 0x7f158c001110 con 0x7f159c15b280 2023-10-16T12:30:26.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.974+0000 7f159affd700 1 -- 172.21.15.16:0/3926722378 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15840035b0 con 0x7f159c15b280 2023-10-16T12:30:26.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.974+0000 7f15a0c8e700 1 -- 172.21.15.16:0/3926722378 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f159c0cc8a0 msgr2=0x7f159c15e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:26.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.974+0000 7f15a0c8e700 1 --2- 172.21.15.16:0/3926722378 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f159c0cc8a0 0x7f159c15e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:26.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.974+0000 7f15a0c8e700 1 -- 172.21.15.16:0/3926722378 >> v1:172.21.15.16:6789/0 conn(0x7f159c15b280 legacy=0x7f159c1613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:26.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.974+0000 7f15a0c8e700 1 -- 172.21.15.16:0/3926722378 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f159c1621f0 con 0x7f159c06a7e0 2023-10-16T12:30:26.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.974+0000 7f159b7fe700 1 --2- 172.21.15.16:0/3926722378 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f159c0cc8a0 0x7f159c15e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:30:26.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.974+0000 7f15a0c8e700 1 --2- 172.21.15.16:0/3926722378 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f159c06a7e0 0x7f159c06ac30 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f1594009d90 tx=0x7f159400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f772264cf0b6b30 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:26.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.975+0000 7f159affd700 1 -- 172.21.15.16:0/3926722378 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f159400e070 con 0x7f159c06a7e0 2023-10-16T12:30:26.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.975+0000 7f159affd700 1 -- 172.21.15.16:0/3926722378 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f159400bb20 con 0x7f159c06a7e0 2023-10-16T12:30:26.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.975+0000 7f159affd700 1 -- 172.21.15.16:0/3926722378 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1594012430 con 0x7f159c06a7e0 2023-10-16T12:30:26.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.975+0000 7f15a26f1700 1 -- 172.21.15.16:0/3926722378 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f159c06a7e0 msgr2=0x7f159c06ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:26.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.975+0000 7f15a26f1700 1 --2- 172.21.15.16:0/3926722378 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f159c06a7e0 0x7f159c06ac30 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f1594009d90 tx=0x7f159400b330 comp rx=0 tx=0).stop 2023-10-16T12:30:26.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.975+0000 7f15a26f1700 1 -- 172.21.15.16:0/3926722378 shutdown_connections 2023-10-16T12:30:26.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.975+0000 7f15a26f1700 1 --2- 172.21.15.16:0/3926722378 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f159c06a7e0 0x7f159c06ac30 unknown :-1 s=CLOSED pgs=1545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:26.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.975+0000 7f15a26f1700 1 --2- 172.21.15.16:0/3926722378 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f159c0cc8a0 0x7f159c15e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:26.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.975+0000 7f15a26f1700 1 -- 172.21.15.16:0/3926722378 >> 172.21.15.16:0/3926722378 conn(0x7f159c0b1fd0 msgr2=0x7f159c0b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:26.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.975+0000 7f15a26f1700 1 -- 172.21.15.16:0/3926722378 shutdown_connections 2023-10-16T12:30:26.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.975+0000 7f15a26f1700 1 -- 172.21.15.16:0/3926722378 wait complete. 2023-10-16T12:30:26.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.976+0000 7f15a26f1700 1 Processor -- start 2023-10-16T12:30:26.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.977+0000 7f15a26f1700 1 -- start start 2023-10-16T12:30:26.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.977+0000 7f15a26f1700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f159c06a7e0 0x7f159c0c80f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:26.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.977+0000 7f15a26f1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f159c0cc8a0 0x7f159c0b05a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:26.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.977+0000 7f15a26f1700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f159c15b280 0x7f159c156b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:26.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.977+0000 7f15a26f1700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f159c0b1690 con 0x7f159c0cc8a0 2023-10-16T12:30:26.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.977+0000 7f15a26f1700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f159c0b0df0 con 0x7f159c06a7e0 2023-10-16T12:30:26.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.977+0000 7f15a26f1700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f159c0b10d0 con 0x7f159c15b280 2023-10-16T12:30:26.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.977+0000 7f159b7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f159c0cc8a0 0x7f159c0b05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:26.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.977+0000 7f159b7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f159c0cc8a0 0x7f159c0b05a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45242/0 (socket says 172.21.15.16:45242) 2023-10-16T12:30:26.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.978+0000 7f159b7fe700 1 -- 172.21.15.16:0/665504758 learned_addr learned my addr 172.21.15.16:0/665504758 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:26.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.978+0000 7f159b7fe700 1 -- 172.21.15.16:0/665504758 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f159c15b280 msgr2=0x7f159c156b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:26.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.978+0000 7f159b7fe700 1 --2- 172.21.15.16:0/665504758 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f159c15b280 0x7f159c156b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:26.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.978+0000 7f159b7fe700 1 -- 172.21.15.16:0/665504758 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f159c06a7e0 msgr2=0x7f159c0c80f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:26.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.978+0000 7f159b7fe700 1 --2- 172.21.15.16:0/665504758 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f159c06a7e0 0x7f159c0c80f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:26.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.978+0000 7f159b7fe700 1 -- 172.21.15.16:0/665504758 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1594009a70 con 0x7f159c0cc8a0 2023-10-16T12:30:26.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.978+0000 7f159b7fe700 1 --2- 172.21.15.16:0/665504758 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f159c0cc8a0 0x7f159c0b05a0 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f159000e810 tx=0x7f159000ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:26.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.978+0000 7f1598ff9700 1 -- 172.21.15.16:0/665504758 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1590004230 con 0x7f159c0cc8a0 2023-10-16T12:30:26.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.978+0000 7f1598ff9700 1 -- 172.21.15.16:0/665504758 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1590004390 con 0x7f159c0cc8a0 2023-10-16T12:30:26.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.978+0000 7f15a26f1700 1 -- 172.21.15.16:0/665504758 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f159c0b13b0 con 0x7f159c0cc8a0 2023-10-16T12:30:26.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.978+0000 7f1598ff9700 1 -- 172.21.15.16:0/665504758 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15900036a0 con 0x7f159c0cc8a0 2023-10-16T12:30:26.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.978+0000 7f15a26f1700 1 -- 172.21.15.16:0/665504758 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f159c0b0ae0 con 0x7f159c0cc8a0 2023-10-16T12:30:26.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.979+0000 7f15a26f1700 1 -- 172.21.15.16:0/665504758 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f159c1f3c80 con 0x7f159c0cc8a0 2023-10-16T12:30:26.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.980+0000 7f1598ff9700 1 -- 172.21.15.16:0/665504758 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 74) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1590003800 con 0x7f159c0cc8a0 2023-10-16T12:30:26.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.980+0000 7f1598ff9700 1 --2- 172.21.15.16:0/665504758 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f158806b020 0x7f158806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:26.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.980+0000 7f1598ff9700 1 -- 172.21.15.16:0/665504758 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 6085+0+0 (secure 0 0 0) 0x7f15900894c0 con 0x7f159c0cc8a0 2023-10-16T12:30:26.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.980+0000 7f159bfff700 1 --2- 172.21.15.16:0/665504758 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f158806b020 0x7f158806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:26.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.980+0000 7f159bfff700 1 --2- 172.21.15.16:0/665504758 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f158806b020 0x7f158806d4e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f158c005910 tx=0x7f158c001150 comp rx=0 tx=0).ready entity=mgr.8388 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:26.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:26.985+0000 7f1598ff9700 1 -- 172.21.15.16:0/665504758 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f159c1f3c80 con 0x7f159c0cc8a0 2023-10-16T12:30:27.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.189+0000 7f15a26f1700 1 -- 172.21.15.16:0/665504758 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f159c1f4060 con 0x7f159c0cc8a0 2023-10-16T12:30:27.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.190+0000 7f1598ff9700 1 -- 172.21.15.16:0/665504758 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v142) v1 ==== 74+0+16764 (secure 0 0 0) 0x7f159c1f4060 con 0x7f159c0cc8a0 2023-10-16T12:30:27.192 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:30:27.193 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":142,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:30:24.394120+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:30:21.318498+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"141","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2023-10-16T12:30:21.813382+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"142","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:30:27.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.192+0000 7f15a26f1700 1 -- 172.21.15.16:0/665504758 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f158806b020 msgr2=0x7f158806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:27.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.192+0000 7f15a26f1700 1 --2- 172.21.15.16:0/665504758 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f158806b020 0x7f158806d4e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f158c005910 tx=0x7f158c001150 comp rx=0 tx=0).stop 2023-10-16T12:30:27.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.192+0000 7f15a26f1700 1 -- 172.21.15.16:0/665504758 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f159c0cc8a0 msgr2=0x7f159c0b05a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:27.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.192+0000 7f15a26f1700 1 --2- 172.21.15.16:0/665504758 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f159c0cc8a0 0x7f159c0b05a0 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f159000e810 tx=0x7f159000ebd0 comp rx=0 tx=0).stop 2023-10-16T12:30:27.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.192+0000 7f15a26f1700 1 -- 172.21.15.16:0/665504758 shutdown_connections 2023-10-16T12:30:27.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.192+0000 7f15a26f1700 1 --2- 172.21.15.16:0/665504758 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f159c15b280 0x7f159c156b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:27.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.192+0000 7f15a26f1700 1 --2- 172.21.15.16:0/665504758 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f158806b020 0x7f158806d4e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:27.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.192+0000 7f15a26f1700 1 --2- 172.21.15.16:0/665504758 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f159c06a7e0 0x7f159c0c80f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:27.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.192+0000 7f15a26f1700 1 --2- 172.21.15.16:0/665504758 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f159c0cc8a0 0x7f159c0b05a0 unknown :-1 s=CLOSED pgs=1546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:27.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.192+0000 7f15a26f1700 1 -- 172.21.15.16:0/665504758 >> 172.21.15.16:0/665504758 conn(0x7f159c0b1fd0 msgr2=0x7f159c071f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:27.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.192+0000 7f15a26f1700 1 -- 172.21.15.16:0/665504758 shutdown_connections 2023-10-16T12:30:27.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.193+0000 7f15a26f1700 1 -- 172.21.15.16:0/665504758 wait complete. 2023-10-16T12:30:27.207 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-16T12:30:27.208 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T12:30:27.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.331+0000 7f82e14f0700 1 Processor -- start 2023-10-16T12:30:27.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.331+0000 7f82e14f0700 1 -- start start 2023-10-16T12:30:27.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.332+0000 7f82e14f0700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f82dc15d8d0 0x7f82dc0d1c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:27.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.332+0000 7f82e14f0700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f82dc0d41c0 0x7f82dc0d0040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:27.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.332+0000 7f82e14f0700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82dc06ba00 con 0x7f82dc0d0580 2023-10-16T12:30:27.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.332+0000 7f82e14f0700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82dc0d0a10 con 0x7f82dc0d41c0 2023-10-16T12:30:27.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.332+0000 7f82e14f0700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82dc0d2150 con 0x7f82dc15d8d0 2023-10-16T12:30:27.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.332+0000 7f82db7fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f82dc0d0580 0x7f82dc0ce240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34526/0 (socket says 172.21.15.16:34526) 2023-10-16T12:30:27.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.332+0000 7f82db7fe700 1 -- 172.21.15.16:0/2194780692 learned_addr learned my addr 172.21.15.16:0/2194780692 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:30:27.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.332+0000 7f82da7fc700 1 --2- 172.21.15.16:0/2194780692 >> v2:172.21.15.49:3300/0 conn(0x7f82dc0d41c0 0x7f82dc0d0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:27.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.332+0000 7f82daffd700 1 --2- 172.21.15.16:0/2194780692 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f82dc15d8d0 0x7f82dc0d1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:27.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.332+0000 7f82d9ffb700 1 -- 172.21.15.16:0/2194780692 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3537283908 0 0) 0x7f82dc06ba00 con 0x7f82dc0d0580 2023-10-16T12:30:27.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.332+0000 7f82d9ffb700 1 -- 172.21.15.16:0/2194780692 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82c40035b0 con 0x7f82dc0d0580 2023-10-16T12:30:27.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.332+0000 7f82d9ffb700 1 -- 172.21.15.16:0/2194780692 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2028307028 0 0) 0x7f82c40035b0 con 0x7f82dc0d0580 2023-10-16T12:30:27.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.332+0000 7f82d9ffb700 1 -- 172.21.15.16:0/2194780692 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f82dc15d8d0 msgr2=0x7f82dc0d1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:27.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.333+0000 7f82d9ffb700 1 --2- 172.21.15.16:0/2194780692 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f82dc15d8d0 0x7f82dc0d1c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:27.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.333+0000 7f82d9ffb700 1 -- 172.21.15.16:0/2194780692 >> v2:172.21.15.49:3300/0 conn(0x7f82dc0d41c0 msgr2=0x7f82dc0d0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:27.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.333+0000 7f82d9ffb700 1 --2- 172.21.15.16:0/2194780692 >> v2:172.21.15.49:3300/0 conn(0x7f82dc0d41c0 0x7f82dc0d0040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:27.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.333+0000 7f82d9ffb700 1 -- 172.21.15.16:0/2194780692 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82dc0d26d0 con 0x7f82dc0d0580 2023-10-16T12:30:27.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.333+0000 7f82d9ffb700 1 -- 172.21.15.16:0/2194780692 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f82d40019f0 con 0x7f82dc0d0580 2023-10-16T12:30:27.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.333+0000 7f82d9ffb700 1 -- 172.21.15.16:0/2194780692 >> v1:172.21.15.16:6789/0 conn(0x7f82dc0d0580 legacy=0x7f82dc0ce240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:27.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.333+0000 7f82d9ffb700 1 --2- 172.21.15.16:0/2194780692 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f82c4003a00 0x7f82c4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:27.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.333+0000 7f82d9ffb700 1 --2- 172.21.15.16:0/2194780692 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82c4004850 0x7f82c4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:27.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.333+0000 7f82d9ffb700 1 --2- 172.21.15.16:0/2194780692 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f82dc15d8d0 0x7f82c4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:27.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.333+0000 7f82d9ffb700 1 -- 172.21.15.16:0/2194780692 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82dc0d0a10 con 0x7f82c4004850 2023-10-16T12:30:27.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.333+0000 7f82d9ffb700 1 -- 172.21.15.16:0/2194780692 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82dc0d2150 con 0x7f82c4003a00 2023-10-16T12:30:27.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.333+0000 7f82d9ffb700 1 -- 172.21.15.16:0/2194780692 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82c400ba10 con 0x7f82dc15d8d0 2023-10-16T12:30:27.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.333+0000 7f82da7fc700 1 --2- 172.21.15.16:0/2194780692 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82c4004850 0x7f82c4004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:27.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.333+0000 7f82db7fe700 1 --2- 172.21.15.16:0/2194780692 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f82dc15d8d0 0x7f82c4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:27.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.333+0000 7f82daffd700 1 --2- 172.21.15.16:0/2194780692 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f82c4003a00 0x7f82c4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:27.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.333+0000 7f82da7fc700 1 -- 172.21.15.16:0/2194780692 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f82dc15d8d0 msgr2=0x7f82c4005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:27.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.333+0000 7f82da7fc700 1 --2- 172.21.15.16:0/2194780692 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f82dc15d8d0 0x7f82c4005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:27.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.333+0000 7f82da7fc700 1 -- 172.21.15.16:0/2194780692 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f82c4003a00 msgr2=0x7f82c4003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:27.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.333+0000 7f82da7fc700 1 --2- 172.21.15.16:0/2194780692 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f82c4003a00 0x7f82c4003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:27.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.333+0000 7f82da7fc700 1 -- 172.21.15.16:0/2194780692 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82c40035b0 con 0x7f82c4004850 2023-10-16T12:30:27.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.333+0000 7f82db7fe700 1 --2- 172.21.15.16:0/2194780692 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f82dc15d8d0 0x7f82c4005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:30:27.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.333+0000 7f82da7fc700 1 --2- 172.21.15.16:0/2194780692 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82c4004850 0x7f82c4004cd0 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f82cc009b60 tx=0x7f82c4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f999bb00d1256374 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:27.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.333+0000 7f82d9ffb700 1 -- 172.21.15.16:0/2194780692 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82cc00e6c0 con 0x7f82c4004850 2023-10-16T12:30:27.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.333+0000 7f82d9ffb700 1 -- 172.21.15.16:0/2194780692 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82cc00e820 con 0x7f82c4004850 2023-10-16T12:30:27.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.333+0000 7f82d9ffb700 1 -- 172.21.15.16:0/2194780692 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82cc00eb90 con 0x7f82c4004850 2023-10-16T12:30:27.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.334+0000 7f82e14f0700 1 -- 172.21.15.16:0/2194780692 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82c4004850 msgr2=0x7f82c4004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:27.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.334+0000 7f82e14f0700 1 --2- 172.21.15.16:0/2194780692 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82c4004850 0x7f82c4004cd0 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f82cc009b60 tx=0x7f82c4005880 comp rx=0 tx=0).stop 2023-10-16T12:30:27.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.334+0000 7f82db7fe700 1 -- 172.21.15.16:0/2194780692 reap_dead start 2023-10-16T12:30:27.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.334+0000 7f82e14f0700 1 -- 172.21.15.16:0/2194780692 shutdown_connections 2023-10-16T12:30:27.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.334+0000 7f82e14f0700 1 -- 172.21.15.16:0/2194780692 >> 172.21.15.16:0/2194780692 conn(0x7f82dc0d3630 msgr2=0x7f82dc0d3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:27.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.334+0000 7f82e14f0700 1 -- 172.21.15.16:0/2194780692 shutdown_connections 2023-10-16T12:30:27.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.334+0000 7f82e14f0700 1 -- 172.21.15.16:0/2194780692 wait complete. 2023-10-16T12:30:27.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.336+0000 7f82e14f0700 1 Processor -- start 2023-10-16T12:30:27.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.336+0000 7f82e14f0700 1 -- start start 2023-10-16T12:30:27.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.337+0000 7f82e14f0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82c4004850 0x7f82dc15ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:27.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.337+0000 7f82e14f0700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f82dc09d310 0x7f82dc155ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:27.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.337+0000 7f82e14f0700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f82dc15c540 0x7f82dc0ceaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:27.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.337+0000 7f82e14f0700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82dc15ce60 con 0x7f82c4004850 2023-10-16T12:30:27.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.337+0000 7f82e14f0700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82dc157dd0 con 0x7f82dc09d310 2023-10-16T12:30:27.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.337+0000 7f82e14f0700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82dc1580b0 con 0x7f82dc15c540 2023-10-16T12:30:27.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.337+0000 7f82daffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82c4004850 0x7f82dc15ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:27.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.337+0000 7f82daffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82c4004850 0x7f82dc15ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45260/0 (socket says 172.21.15.16:45260) 2023-10-16T12:30:27.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.337+0000 7f82daffd700 1 -- 172.21.15.16:0/842814443 learned_addr learned my addr 172.21.15.16:0/842814443 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:27.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.337+0000 7f82daffd700 1 -- 172.21.15.16:0/842814443 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f82dc15c540 msgr2=0x7f82dc0ceaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:27.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.337+0000 7f82daffd700 1 --2- 172.21.15.16:0/842814443 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f82dc15c540 0x7f82dc0ceaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:27.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.337+0000 7f82daffd700 1 -- 172.21.15.16:0/842814443 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f82dc09d310 msgr2=0x7f82dc155ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:27.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.337+0000 7f82daffd700 1 --2- 172.21.15.16:0/842814443 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f82dc09d310 0x7f82dc155ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:27.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.337+0000 7f82daffd700 1 -- 172.21.15.16:0/842814443 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82dc158390 con 0x7f82c4004850 2023-10-16T12:30:27.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.337+0000 7f82daffd700 1 --2- 172.21.15.16:0/842814443 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82c4004850 0x7f82dc15ac70 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7f82d00115e0 tx=0x7f82d00119a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:27.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.337+0000 7f82c37fe700 1 -- 172.21.15.16:0/842814443 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d0015870 con 0x7f82c4004850 2023-10-16T12:30:27.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.337+0000 7f82c37fe700 1 -- 172.21.15.16:0/842814443 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82d00159d0 con 0x7f82c4004850 2023-10-16T12:30:27.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.337+0000 7f82e14f0700 1 -- 172.21.15.16:0/842814443 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82dc158670 con 0x7f82c4004850 2023-10-16T12:30:27.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.337+0000 7f82c37fe700 1 -- 172.21.15.16:0/842814443 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d0015d40 con 0x7f82c4004850 2023-10-16T12:30:27.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.337+0000 7f82e14f0700 1 -- 172.21.15.16:0/842814443 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82dc15c9f0 con 0x7f82c4004850 2023-10-16T12:30:27.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.338+0000 7f82e14f0700 1 -- 172.21.15.16:0/842814443 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82dc1589e0 con 0x7f82c4004850 2023-10-16T12:30:27.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.339+0000 7f82c37fe700 1 -- 172.21.15.16:0/842814443 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 74) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f82d0013400 con 0x7f82c4004850 2023-10-16T12:30:27.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.339+0000 7f82c37fe700 1 --2- 172.21.15.16:0/842814443 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f82c806b020 0x7f82c806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:27.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.339+0000 7f82c37fe700 1 -- 172.21.15.16:0/842814443 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 6085+0+0 (secure 0 0 0) 0x7f82d0089cd0 con 0x7f82c4004850 2023-10-16T12:30:27.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.339+0000 7f82da7fc700 1 --2- 172.21.15.16:0/842814443 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f82c806b020 0x7f82c806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:27.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.340+0000 7f82da7fc700 1 --2- 172.21.15.16:0/842814443 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f82c806b020 0x7f82c806d4e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f82dc15b1b0 tx=0x7f82cc013000 comp rx=0 tx=0).ready entity=mgr.8388 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:27.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.344+0000 7f82c37fe700 1 -- 172.21.15.16:0/842814443 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f82d0054a50 con 0x7f82c4004850 2023-10-16T12:30:27.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.545+0000 7f82e14f0700 1 -- 172.21.15.16:0/842814443 --> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f82dc06bca0 con 0x7f82c806b020 2023-10-16T12:30:27.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.554+0000 7f82c37fe700 1 -- 172.21.15.16:0/842814443 <== mgr.8388 v2:172.21.15.16:6832/258291157 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289926 (secure 0 0 0) 0x7f82cc00fa20 con 0x7f82c806b020 2023-10-16T12:30:27.559 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:30:27.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.557+0000 7f82e14f0700 1 -- 172.21.15.16:0/842814443 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f82c806b020 msgr2=0x7f82c806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:27.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.557+0000 7f82e14f0700 1 --2- 172.21.15.16:0/842814443 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f82c806b020 0x7f82c806d4e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f82dc15b1b0 tx=0x7f82cc013000 comp rx=0 tx=0).stop 2023-10-16T12:30:27.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.557+0000 7f82e14f0700 1 -- 172.21.15.16:0/842814443 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82c4004850 msgr2=0x7f82dc15ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:27.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.557+0000 7f82e14f0700 1 --2- 172.21.15.16:0/842814443 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82c4004850 0x7f82dc15ac70 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7f82d00115e0 tx=0x7f82d00119a0 comp rx=0 tx=0).stop 2023-10-16T12:30:27.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.557+0000 7f82e14f0700 1 -- 172.21.15.16:0/842814443 shutdown_connections 2023-10-16T12:30:27.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.557+0000 7f82e14f0700 1 --2- 172.21.15.16:0/842814443 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f82dc15c540 0x7f82dc0ceaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:27.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.557+0000 7f82e14f0700 1 --2- 172.21.15.16:0/842814443 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82c4004850 0x7f82dc15ac70 unknown :-1 s=CLOSED pgs=1549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:27.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.557+0000 7f82e14f0700 1 --2- 172.21.15.16:0/842814443 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f82c806b020 0x7f82c806d4e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:27.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.557+0000 7f82e14f0700 1 --2- 172.21.15.16:0/842814443 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f82dc09d310 0x7f82dc155ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:27.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.557+0000 7f82e14f0700 1 -- 172.21.15.16:0/842814443 >> 172.21.15.16:0/842814443 conn(0x7f82dc0d3630 msgr2=0x7f82dc0686c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:27.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.557+0000 7f82e14f0700 1 -- 172.21.15.16:0/842814443 shutdown_connections 2023-10-16T12:30:27.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.557+0000 7f82e14f0700 1 -- 172.21.15.16:0/842814443 wait complete. 2023-10-16T12:30:27.564 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-10-16T12:30:27.572 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-10-16T12:30:26.915181+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1076,"num_read_kb":2160,"num_write":403,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":265,"ondisk_log_size":265,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":115,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":347672,"kb_used_data":14932,"kb_used_omap":12,"kb_used_meta":332595,"kb_avail":754627048,"statfs":{"total":773094113280,"available":772738097152,"internally_reserved":0,"allocated":15290368,"data_stored":8269448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12740,"internal_metadata":340577852},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.537810"},"pg_stats":[{"pgid":"18.3f","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.427791+0000","last_change":"2023-10-16T12:30:23.421734+0000","last_active":"2023-10-16T12:30:23.427791+0000","last_peered":"2023-10-16T12:30:23.427791+0000","last_clean":"2023-10-16T12:30:23.427791+0000","last_became_active":"2023-10-16T12:30:23.421440+0000","last_became_peered":"2023-10-16T12:30:23.421440+0000","last_unstale":"2023-10-16T12:30:23.427791+0000","last_undegraded":"2023-10-16T12:30:23.427791+0000","last_fullsized":"2023-10-16T12:30:23.427791+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:48:30.457561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3e","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.416122+0000","last_change":"2023-10-16T12:30:23.412272+0000","last_active":"2023-10-16T12:30:23.416122+0000","last_peered":"2023-10-16T12:30:23.416122+0000","last_clean":"2023-10-16T12:30:23.416122+0000","last_became_active":"2023-10-16T12:30:23.411962+0000","last_became_peered":"2023-10-16T12:30:23.411962+0000","last_unstale":"2023-10-16T12:30:23.416122+0000","last_undegraded":"2023-10-16T12:30:23.416122+0000","last_fullsized":"2023-10-16T12:30:23.416122+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:01:51.764162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3d","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.422041+0000","last_change":"2023-10-16T12:30:23.414309+0000","last_active":"2023-10-16T12:30:23.422041+0000","last_peered":"2023-10-16T12:30:23.422041+0000","last_clean":"2023-10-16T12:30:23.422041+0000","last_became_active":"2023-10-16T12:30:23.413931+0000","last_became_peered":"2023-10-16T12:30:23.413931+0000","last_unstale":"2023-10-16T12:30:23.422041+0000","last_undegraded":"2023-10-16T12:30:23.422041+0000","last_fullsized":"2023-10-16T12:30:23.422041+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:57:53.835205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3c","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T12:30:24.398830+0000","last_change":"2023-10-16T12:30:23.422354+0000","last_active":"2023-10-16T12:30:24.398830+0000","last_peered":"2023-10-16T12:30:24.398830+0000","last_clean":"2023-10-16T12:30:24.398830+0000","last_became_active":"2023-10-16T12:30:23.422053+0000","last_became_peered":"2023-10-16T12:30:23.422053+0000","last_unstale":"2023-10-16T12:30:24.398830+0000","last_undegraded":"2023-10-16T12:30:24.398830+0000","last_fullsized":"2023-10-16T12:30:24.398830+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:01:51.764162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3b","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.426678+0000","last_change":"2023-10-16T12:30:23.423801+0000","last_active":"2023-10-16T12:30:23.426678+0000","last_peered":"2023-10-16T12:30:23.426678+0000","last_clean":"2023-10-16T12:30:23.426678+0000","last_became_active":"2023-10-16T12:30:23.423517+0000","last_became_peered":"2023-10-16T12:30:23.423517+0000","last_unstale":"2023-10-16T12:30:23.426678+0000","last_undegraded":"2023-10-16T12:30:23.426678+0000","last_fullsized":"2023-10-16T12:30:23.426678+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:36:28.673423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3a","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T12:30:24.397614+0000","last_change":"2023-10-16T12:30:23.420780+0000","last_active":"2023-10-16T12:30:24.397614+0000","last_peered":"2023-10-16T12:30:24.397614+0000","last_clean":"2023-10-16T12:30:24.397614+0000","last_became_active":"2023-10-16T12:30:23.420343+0000","last_became_peered":"2023-10-16T12:30:23.420343+0000","last_unstale":"2023-10-16T12:30:24.397614+0000","last_undegraded":"2023-10-16T12:30:24.397614+0000","last_fullsized":"2023-10-16T12:30:24.397614+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:39:22.549338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.39","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T12:30:24.397458+0000","last_change":"2023-10-16T12:30:23.409187+0000","last_active":"2023-10-16T12:30:24.397458+0000","last_peered":"2023-10-16T12:30:24.397458+0000","last_clean":"2023-10-16T12:30:24.397458+0000","last_became_active":"2023-10-16T12:30:23.408890+0000","last_became_peered":"2023-10-16T12:30:23.408890+0000","last_unstale":"2023-10-16T12:30:24.397458+0000","last_undegraded":"2023-10-16T12:30:24.397458+0000","last_fullsized":"2023-10-16T12:30:24.397458+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:25:33.134269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.38","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.428621+0000","last_change":"2023-10-16T12:30:23.418765+0000","last_active":"2023-10-16T12:30:23.428621+0000","last_peered":"2023-10-16T12:30:23.428621+0000","last_clean":"2023-10-16T12:30:23.428621+0000","last_became_active":"2023-10-16T12:30:23.418486+0000","last_became_peered":"2023-10-16T12:30:23.418486+0000","last_unstale":"2023-10-16T12:30:23.428621+0000","last_undegraded":"2023-10-16T12:30:23.428621+0000","last_fullsized":"2023-10-16T12:30:23.428621+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:28:10.860320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.37","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T12:30:24.398926+0000","last_change":"2023-10-16T12:30:23.412518+0000","last_active":"2023-10-16T12:30:24.398926+0000","last_peered":"2023-10-16T12:30:24.398926+0000","last_clean":"2023-10-16T12:30:24.398926+0000","last_became_active":"2023-10-16T12:30:23.412187+0000","last_became_peered":"2023-10-16T12:30:23.412187+0000","last_unstale":"2023-10-16T12:30:24.398926+0000","last_undegraded":"2023-10-16T12:30:24.398926+0000","last_fullsized":"2023-10-16T12:30:24.398926+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:19:42.068670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.36","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.413045+0000","last_change":"2023-10-16T12:30:23.406924+0000","last_active":"2023-10-16T12:30:23.413045+0000","last_peered":"2023-10-16T12:30:23.413045+0000","last_clean":"2023-10-16T12:30:23.413045+0000","last_became_active":"2023-10-16T12:30:23.406632+0000","last_became_peered":"2023-10-16T12:30:23.406632+0000","last_unstale":"2023-10-16T12:30:23.413045+0000","last_undegraded":"2023-10-16T12:30:23.413045+0000","last_fullsized":"2023-10-16T12:30:23.413045+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:03:32.170008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.35","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.432907+0000","last_change":"2023-10-16T12:30:23.430887+0000","last_active":"2023-10-16T12:30:23.432907+0000","last_peered":"2023-10-16T12:30:23.432907+0000","last_clean":"2023-10-16T12:30:23.432907+0000","last_became_active":"2023-10-16T12:30:23.430578+0000","last_became_peered":"2023-10-16T12:30:23.430578+0000","last_unstale":"2023-10-16T12:30:23.432907+0000","last_undegraded":"2023-10-16T12:30:23.432907+0000","last_fullsized":"2023-10-16T12:30:23.432907+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:31:26.337544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.34","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T12:30:24.398166+0000","last_change":"2023-10-16T12:30:23.426707+0000","last_active":"2023-10-16T12:30:24.398166+0000","last_peered":"2023-10-16T12:30:24.398166+0000","last_clean":"2023-10-16T12:30:24.398166+0000","last_became_active":"2023-10-16T12:30:23.426419+0000","last_became_peered":"2023-10-16T12:30:23.426419+0000","last_unstale":"2023-10-16T12:30:24.398166+0000","last_undegraded":"2023-10-16T12:30:24.398166+0000","last_fullsized":"2023-10-16T12:30:24.398166+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:40:03.973069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.33","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.416433+0000","last_change":"2023-10-16T12:30:23.414195+0000","last_active":"2023-10-16T12:30:23.416433+0000","last_peered":"2023-10-16T12:30:23.416433+0000","last_clean":"2023-10-16T12:30:23.416433+0000","last_became_active":"2023-10-16T12:30:23.413919+0000","last_became_peered":"2023-10-16T12:30:23.413919+0000","last_unstale":"2023-10-16T12:30:23.416433+0000","last_undegraded":"2023-10-16T12:30:23.416433+0000","last_fullsized":"2023-10-16T12:30:23.416433+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:55:23.016394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.32","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.422173+0000","last_change":"2023-10-16T12:30:23.419708+0000","last_active":"2023-10-16T12:30:23.422173+0000","last_peered":"2023-10-16T12:30:23.422173+0000","last_clean":"2023-10-16T12:30:23.422173+0000","last_became_active":"2023-10-16T12:30:23.419167+0000","last_became_peered":"2023-10-16T12:30:23.419167+0000","last_unstale":"2023-10-16T12:30:23.422173+0000","last_undegraded":"2023-10-16T12:30:23.422173+0000","last_fullsized":"2023-10-16T12:30:23.422173+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:18:36.577516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.31","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.427006+0000","last_change":"2023-10-16T12:30:23.426251+0000","last_active":"2023-10-16T12:30:23.427006+0000","last_peered":"2023-10-16T12:30:23.427006+0000","last_clean":"2023-10-16T12:30:23.427006+0000","last_became_active":"2023-10-16T12:30:23.425968+0000","last_became_peered":"2023-10-16T12:30:23.425968+0000","last_unstale":"2023-10-16T12:30:23.427006+0000","last_undegraded":"2023-10-16T12:30:23.427006+0000","last_fullsized":"2023-10-16T12:30:23.427006+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:25:39.680586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.30","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T12:30:24.397443+0000","last_change":"2023-10-16T12:30:23.411584+0000","last_active":"2023-10-16T12:30:24.397443+0000","last_peered":"2023-10-16T12:30:24.397443+0000","last_clean":"2023-10-16T12:30:24.397443+0000","last_became_active":"2023-10-16T12:30:23.410532+0000","last_became_peered":"2023-10-16T12:30:23.410532+0000","last_unstale":"2023-10-16T12:30:24.397443+0000","last_undegraded":"2023-10-16T12:30:24.397443+0000","last_fullsized":"2023-10-16T12:30:24.397443+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:30:01.541809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.2f","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.418154+0000","last_change":"2023-10-16T12:30:23.416877+0000","last_active":"2023-10-16T12:30:23.418154+0000","last_peered":"2023-10-16T12:30:23.418154+0000","last_clean":"2023-10-16T12:30:23.418154+0000","last_became_active":"2023-10-16T12:30:23.416298+0000","last_became_peered":"2023-10-16T12:30:23.416298+0000","last_unstale":"2023-10-16T12:30:23.418154+0000","last_undegraded":"2023-10-16T12:30:23.418154+0000","last_fullsized":"2023-10-16T12:30:23.418154+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:21:12.661644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.2e","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.416147+0000","last_change":"2023-10-16T12:30:23.412392+0000","last_active":"2023-10-16T12:30:23.416147+0000","last_peered":"2023-10-16T12:30:23.416147+0000","last_clean":"2023-10-16T12:30:23.416147+0000","last_became_active":"2023-10-16T12:30:23.411945+0000","last_became_peered":"2023-10-16T12:30:23.411945+0000","last_unstale":"2023-10-16T12:30:23.416147+0000","last_undegraded":"2023-10-16T12:30:23.416147+0000","last_fullsized":"2023-10-16T12:30:23.416147+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:13:43.938813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.2d","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.408745+0000","last_change":"2023-10-16T12:30:23.406925+0000","last_active":"2023-10-16T12:30:23.408745+0000","last_peered":"2023-10-16T12:30:23.408745+0000","last_clean":"2023-10-16T12:30:23.408745+0000","last_became_active":"2023-10-16T12:30:23.406496+0000","last_became_peered":"2023-10-16T12:30:23.406496+0000","last_unstale":"2023-10-16T12:30:23.408745+0000","last_undegraded":"2023-10-16T12:30:23.408745+0000","last_fullsized":"2023-10-16T12:30:23.408745+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:26:39.397930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.2c","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.414239+0000","last_change":"2023-10-16T12:30:23.411765+0000","last_active":"2023-10-16T12:30:23.414239+0000","last_peered":"2023-10-16T12:30:23.414239+0000","last_clean":"2023-10-16T12:30:23.414239+0000","last_became_active":"2023-10-16T12:30:23.411313+0000","last_became_peered":"2023-10-16T12:30:23.411313+0000","last_unstale":"2023-10-16T12:30:23.414239+0000","last_undegraded":"2023-10-16T12:30:23.414239+0000","last_fullsized":"2023-10-16T12:30:23.414239+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:49:51.894838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.2b","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T12:30:24.398527+0000","last_change":"2023-10-16T12:30:23.411265+0000","last_active":"2023-10-16T12:30:24.398527+0000","last_peered":"2023-10-16T12:30:24.398527+0000","last_clean":"2023-10-16T12:30:24.398527+0000","last_became_active":"2023-10-16T12:30:23.410849+0000","last_became_peered":"2023-10-16T12:30:23.410849+0000","last_unstale":"2023-10-16T12:30:24.398527+0000","last_undegraded":"2023-10-16T12:30:24.398527+0000","last_fullsized":"2023-10-16T12:30:24.398527+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:14:15.737566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.2a","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.430208+0000","last_change":"2023-10-16T12:30:23.426390+0000","last_active":"2023-10-16T12:30:23.430208+0000","last_peered":"2023-10-16T12:30:23.430208+0000","last_clean":"2023-10-16T12:30:23.430208+0000","last_became_active":"2023-10-16T12:30:23.422353+0000","last_became_peered":"2023-10-16T12:30:23.422353+0000","last_unstale":"2023-10-16T12:30:23.430208+0000","last_undegraded":"2023-10-16T12:30:23.430208+0000","last_fullsized":"2023-10-16T12:30:23.430208+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:16:06.545565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.29","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T12:30:24.398493+0000","last_change":"2023-10-16T12:30:23.412572+0000","last_active":"2023-10-16T12:30:24.398493+0000","last_peered":"2023-10-16T12:30:24.398493+0000","last_clean":"2023-10-16T12:30:24.398493+0000","last_became_active":"2023-10-16T12:30:23.412241+0000","last_became_peered":"2023-10-16T12:30:23.412241+0000","last_unstale":"2023-10-16T12:30:24.398493+0000","last_undegraded":"2023-10-16T12:30:24.398493+0000","last_fullsized":"2023-10-16T12:30:24.398493+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:13:43.938813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.28","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.417023+0000","last_change":"2023-10-16T12:30:23.415883+0000","last_active":"2023-10-16T12:30:23.417023+0000","last_peered":"2023-10-16T12:30:23.417023+0000","last_clean":"2023-10-16T12:30:23.417023+0000","last_became_active":"2023-10-16T12:30:23.415439+0000","last_became_peered":"2023-10-16T12:30:23.415439+0000","last_unstale":"2023-10-16T12:30:23.417023+0000","last_undegraded":"2023-10-16T12:30:23.417023+0000","last_fullsized":"2023-10-16T12:30:23.417023+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:01:59.976335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.27","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.430312+0000","last_change":"2023-10-16T12:30:23.428654+0000","last_active":"2023-10-16T12:30:23.430312+0000","last_peered":"2023-10-16T12:30:23.430312+0000","last_clean":"2023-10-16T12:30:23.430312+0000","last_became_active":"2023-10-16T12:30:23.428335+0000","last_became_peered":"2023-10-16T12:30:23.428335+0000","last_unstale":"2023-10-16T12:30:23.430312+0000","last_undegraded":"2023-10-16T12:30:23.430312+0000","last_fullsized":"2023-10-16T12:30:23.430312+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:03:03.394597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.26","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.409086+0000","last_change":"2023-10-16T12:30:23.407300+0000","last_active":"2023-10-16T12:30:23.409086+0000","last_peered":"2023-10-16T12:30:23.409086+0000","last_clean":"2023-10-16T12:30:23.409086+0000","last_became_active":"2023-10-16T12:30:23.406831+0000","last_became_peered":"2023-10-16T12:30:23.406831+0000","last_unstale":"2023-10-16T12:30:23.409086+0000","last_undegraded":"2023-10-16T12:30:23.409086+0000","last_fullsized":"2023-10-16T12:30:23.409086+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:45:11.054235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.25","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.408726+0000","last_change":"2023-10-16T12:30:23.406657+0000","last_active":"2023-10-16T12:30:23.408726+0000","last_peered":"2023-10-16T12:30:23.408726+0000","last_clean":"2023-10-16T12:30:23.408726+0000","last_became_active":"2023-10-16T12:30:23.406075+0000","last_became_peered":"2023-10-16T12:30:23.406075+0000","last_unstale":"2023-10-16T12:30:23.408726+0000","last_undegraded":"2023-10-16T12:30:23.408726+0000","last_fullsized":"2023-10-16T12:30:23.408726+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:25:58.262389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.24","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T12:30:24.398011+0000","last_change":"2023-10-16T12:30:23.418156+0000","last_active":"2023-10-16T12:30:24.398011+0000","last_peered":"2023-10-16T12:30:24.398011+0000","last_clean":"2023-10-16T12:30:24.398011+0000","last_became_active":"2023-10-16T12:30:23.417863+0000","last_became_peered":"2023-10-16T12:30:23.417863+0000","last_unstale":"2023-10-16T12:30:24.398011+0000","last_undegraded":"2023-10-16T12:30:24.398011+0000","last_fullsized":"2023-10-16T12:30:24.398011+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:00:00.022734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.23","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.415416+0000","last_change":"2023-10-16T12:30:23.412045+0000","last_active":"2023-10-16T12:30:23.415416+0000","last_peered":"2023-10-16T12:30:23.415416+0000","last_clean":"2023-10-16T12:30:23.415416+0000","last_became_active":"2023-10-16T12:30:23.411753+0000","last_became_peered":"2023-10-16T12:30:23.411753+0000","last_unstale":"2023-10-16T12:30:23.415416+0000","last_undegraded":"2023-10-16T12:30:23.415416+0000","last_fullsized":"2023-10-16T12:30:23.415416+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:38:36.180207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.22","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.429522+0000","last_change":"2023-10-16T12:30:23.427792+0000","last_active":"2023-10-16T12:30:23.429522+0000","last_peered":"2023-10-16T12:30:23.429522+0000","last_clean":"2023-10-16T12:30:23.429522+0000","last_became_active":"2023-10-16T12:30:23.427492+0000","last_became_peered":"2023-10-16T12:30:23.427492+0000","last_unstale":"2023-10-16T12:30:23.429522+0000","last_undegraded":"2023-10-16T12:30:23.429522+0000","last_fullsized":"2023-10-16T12:30:23.429522+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:50:47.311871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.21","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.415396+0000","last_change":"2023-10-16T12:30:23.413300+0000","last_active":"2023-10-16T12:30:23.415396+0000","last_peered":"2023-10-16T12:30:23.415396+0000","last_clean":"2023-10-16T12:30:23.415396+0000","last_became_active":"2023-10-16T12:30:23.412975+0000","last_became_peered":"2023-10-16T12:30:23.412975+0000","last_unstale":"2023-10-16T12:30:23.415396+0000","last_undegraded":"2023-10-16T12:30:23.415396+0000","last_fullsized":"2023-10-16T12:30:23.415396+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:19:07.800422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.20","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.416222+0000","last_change":"2023-10-16T12:30:23.415666+0000","last_active":"2023-10-16T12:30:23.416222+0000","last_peered":"2023-10-16T12:30:23.416222+0000","last_clean":"2023-10-16T12:30:23.416222+0000","last_became_active":"2023-10-16T12:30:23.415128+0000","last_became_peered":"2023-10-16T12:30:23.415128+0000","last_unstale":"2023-10-16T12:30:23.416222+0000","last_undegraded":"2023-10-16T12:30:23.416222+0000","last_fullsized":"2023-10-16T12:30:23.416222+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:39:42.101547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.416875+0000","last_change":"2023-10-16T12:30:23.410285+0000","last_active":"2023-10-16T12:30:23.416875+0000","last_peered":"2023-10-16T12:30:23.416875+0000","last_clean":"2023-10-16T12:30:23.416875+0000","last_became_active":"2023-10-16T12:30:23.409872+0000","last_became_peered":"2023-10-16T12:30:23.409872+0000","last_unstale":"2023-10-16T12:30:23.416875+0000","last_undegraded":"2023-10-16T12:30:23.416875+0000","last_fullsized":"2023-10-16T12:30:23.416875+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:26:39.397930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1f","version":"141'4","reported_seq":15,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.427207+0000","last_change":"2023-10-16T12:30:22.394531+0000","last_active":"2023-10-16T12:30:23.427207+0000","last_peered":"2023-10-16T12:30:23.427207+0000","last_clean":"2023-10-16T12:30:23.427207+0000","last_became_active":"2023-10-16T12:30:22.394243+0000","last_became_peered":"2023-10-16T12:30:22.394243+0000","last_unstale":"2023-10-16T12:30:23.427207+0000","last_undegraded":"2023-10-16T12:30:23.427207+0000","last_fullsized":"2023-10-16T12:30:23.427207+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_clean_scrub_stamp":"2023-10-16T12:30:21.373132+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:11:58.279552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1d","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T12:30:24.397696+0000","last_change":"2023-10-16T12:30:23.414959+0000","last_active":"2023-10-16T12:30:24.397696+0000","last_peered":"2023-10-16T12:30:24.397696+0000","last_clean":"2023-10-16T12:30:24.397696+0000","last_became_active":"2023-10-16T12:30:23.414690+0000","last_became_peered":"2023-10-16T12:30:23.414690+0000","last_unstale":"2023-10-16T12:30:24.397696+0000","last_undegraded":"2023-10-16T12:30:24.397696+0000","last_fullsized":"2023-10-16T12:30:24.397696+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:47:31.216350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1e","version":"141'1","reported_seq":12,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.432623+0000","last_change":"2023-10-16T12:30:22.399129+0000","last_active":"2023-10-16T12:30:23.432623+0000","last_peered":"2023-10-16T12:30:23.432623+0000","last_clean":"2023-10-16T12:30:23.432623+0000","last_became_active":"2023-10-16T12:30:22.398739+0000","last_became_peered":"2023-10-16T12:30:22.398739+0000","last_unstale":"2023-10-16T12:30:23.432623+0000","last_undegraded":"2023-10-16T12:30:23.432623+0000","last_fullsized":"2023-10-16T12:30:23.432623+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_clean_scrub_stamp":"2023-10-16T12:30:21.373132+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:25:38.676933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.415708+0000","last_change":"2023-10-16T12:30:23.414335+0000","last_active":"2023-10-16T12:30:23.415708+0000","last_peered":"2023-10-16T12:30:23.415708+0000","last_clean":"2023-10-16T12:30:23.415708+0000","last_became_active":"2023-10-16T12:30:23.414023+0000","last_became_peered":"2023-10-16T12:30:23.414023+0000","last_unstale":"2023-10-16T12:30:23.415708+0000","last_undegraded":"2023-10-16T12:30:23.415708+0000","last_fullsized":"2023-10-16T12:30:23.415708+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:01:41.278198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"141'2","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T12:30:24.398638+0000","last_change":"2023-10-16T12:30:22.401175+0000","last_active":"2023-10-16T12:30:24.398638+0000","last_peered":"2023-10-16T12:30:24.398638+0000","last_clean":"2023-10-16T12:30:24.398638+0000","last_became_active":"2023-10-16T12:30:22.400471+0000","last_became_peered":"2023-10-16T12:30:22.400471+0000","last_unstale":"2023-10-16T12:30:24.398638+0000","last_undegraded":"2023-10-16T12:30:24.398638+0000","last_fullsized":"2023-10-16T12:30:24.398638+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_clean_scrub_stamp":"2023-10-16T12:30:21.373132+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:01:58.972682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1f","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T12:30:24.398946+0000","last_change":"2023-10-16T12:30:23.406133+0000","last_active":"2023-10-16T12:30:24.398946+0000","last_peered":"2023-10-16T12:30:24.398946+0000","last_clean":"2023-10-16T12:30:24.398946+0000","last_became_active":"2023-10-16T12:30:23.405819+0000","last_became_peered":"2023-10-16T12:30:23.405819+0000","last_unstale":"2023-10-16T12:30:24.398946+0000","last_undegraded":"2023-10-16T12:30:24.398946+0000","last_fullsized":"2023-10-16T12:30:24.398946+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:52:29.142123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"141'2","reported_seq":13,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.431875+0000","last_change":"2023-10-16T12:30:22.400661+0000","last_active":"2023-10-16T12:30:23.431875+0000","last_peered":"2023-10-16T12:30:23.431875+0000","last_clean":"2023-10-16T12:30:23.431875+0000","last_became_active":"2023-10-16T12:30:22.400329+0000","last_became_peered":"2023-10-16T12:30:22.400329+0000","last_unstale":"2023-10-16T12:30:23.431875+0000","last_undegraded":"2023-10-16T12:30:23.431875+0000","last_fullsized":"2023-10-16T12:30:23.431875+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_clean_scrub_stamp":"2023-10-16T12:30:21.373132+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:28:15.424497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.416641+0000","last_change":"2023-10-16T12:30:23.413223+0000","last_active":"2023-10-16T12:30:23.416641+0000","last_peered":"2023-10-16T12:30:23.416641+0000","last_clean":"2023-10-16T12:30:23.416641+0000","last_became_active":"2023-10-16T12:30:23.412934+0000","last_became_peered":"2023-10-16T12:30:23.412934+0000","last_unstale":"2023-10-16T12:30:23.416641+0000","last_undegraded":"2023-10-16T12:30:23.416641+0000","last_fullsized":"2023-10-16T12:30:23.416641+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:48:36.975059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1b","version":"141'1","reported_seq":12,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.432734+0000","last_change":"2023-10-16T12:30:22.404893+0000","last_active":"2023-10-16T12:30:23.432734+0000","last_peered":"2023-10-16T12:30:23.432734+0000","last_clean":"2023-10-16T12:30:23.432734+0000","last_became_active":"2023-10-16T12:30:22.404656+0000","last_became_peered":"2023-10-16T12:30:22.404656+0000","last_unstale":"2023-10-16T12:30:23.432734+0000","last_undegraded":"2023-10-16T12:30:23.432734+0000","last_fullsized":"2023-10-16T12:30:23.432734+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_clean_scrub_stamp":"2023-10-16T12:30:21.373132+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:41:23.912012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.426584+0000","last_change":"2023-10-16T12:30:23.425076+0000","last_active":"2023-10-16T12:30:23.426584+0000","last_peered":"2023-10-16T12:30:23.426584+0000","last_clean":"2023-10-16T12:30:23.426584+0000","last_became_active":"2023-10-16T12:30:23.424782+0000","last_became_peered":"2023-10-16T12:30:23.424782+0000","last_unstale":"2023-10-16T12:30:23.426584+0000","last_undegraded":"2023-10-16T12:30:23.426584+0000","last_fullsized":"2023-10-16T12:30:23.426584+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:04:26.539127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"141'1","reported_seq":13,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T12:30:24.397671+0000","last_change":"2023-10-16T12:30:22.403094+0000","last_active":"2023-10-16T12:30:24.397671+0000","last_peered":"2023-10-16T12:30:24.397671+0000","last_clean":"2023-10-16T12:30:24.397671+0000","last_became_active":"2023-10-16T12:30:22.402806+0000","last_became_peered":"2023-10-16T12:30:22.402806+0000","last_unstale":"2023-10-16T12:30:24.397671+0000","last_undegraded":"2023-10-16T12:30:24.397671+0000","last_fullsized":"2023-10-16T12:30:24.397671+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_clean_scrub_stamp":"2023-10-16T12:30:21.373132+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:16:13.015811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.1a","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.422157+0000","last_change":"2023-10-16T12:30:23.419589+0000","last_active":"2023-10-16T12:30:23.422157+0000","last_peered":"2023-10-16T12:30:23.422157+0000","last_clean":"2023-10-16T12:30:23.422157+0000","last_became_active":"2023-10-16T12:30:23.418858+0000","last_became_peered":"2023-10-16T12:30:23.418858+0000","last_unstale":"2023-10-16T12:30:23.422157+0000","last_undegraded":"2023-10-16T12:30:23.422157+0000","last_fullsized":"2023-10-16T12:30:23.422157+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:01:59.976335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":11,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.402414+0000","last_change":"2023-10-16T12:30:22.392831+0000","last_active":"2023-10-16T12:30:23.402414+0000","last_peered":"2023-10-16T12:30:23.402414+0000","last_clean":"2023-10-16T12:30:23.402414+0000","last_became_active":"2023-10-16T12:30:22.392535+0000","last_became_peered":"2023-10-16T12:30:22.392535+0000","last_unstale":"2023-10-16T12:30:23.402414+0000","last_undegraded":"2023-10-16T12:30:23.402414+0000","last_fullsized":"2023-10-16T12:30:23.402414+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_clean_scrub_stamp":"2023-10-16T12:30:21.373132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:50:46.308219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1b","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.416415+0000","last_change":"2023-10-16T12:30:23.412874+0000","last_active":"2023-10-16T12:30:23.416415+0000","last_peered":"2023-10-16T12:30:23.416415+0000","last_clean":"2023-10-16T12:30:23.416415+0000","last_became_active":"2023-10-16T12:30:23.412574+0000","last_became_peered":"2023-10-16T12:30:23.412574+0000","last_unstale":"2023-10-16T12:30:23.416415+0000","last_undegraded":"2023-10-16T12:30:23.416415+0000","last_fullsized":"2023-10-16T12:30:23.416415+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:25:58.262389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T12:30:24.397644+0000","last_change":"2023-10-16T12:30:22.402099+0000","last_active":"2023-10-16T12:30:24.397644+0000","last_peered":"2023-10-16T12:30:24.397644+0000","last_clean":"2023-10-16T12:30:24.397644+0000","last_became_active":"2023-10-16T12:30:22.401822+0000","last_became_peered":"2023-10-16T12:30:22.401822+0000","last_unstale":"2023-10-16T12:30:24.397644+0000","last_undegraded":"2023-10-16T12:30:24.397644+0000","last_fullsized":"2023-10-16T12:30:24.397644+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_clean_scrub_stamp":"2023-10-16T12:30:21.373132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:24:49.411085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.410157+0000","last_change":"2023-10-16T12:30:23.407550+0000","last_active":"2023-10-16T12:30:23.410157+0000","last_peered":"2023-10-16T12:30:23.410157+0000","last_clean":"2023-10-16T12:30:23.410157+0000","last_became_active":"2023-10-16T12:30:23.407281+0000","last_became_peered":"2023-10-16T12:30:23.407281+0000","last_unstale":"2023-10-16T12:30:23.410157+0000","last_undegraded":"2023-10-16T12:30:23.410157+0000","last_fullsized":"2023-10-16T12:30:23.410157+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:03:32.170008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":11,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.401651+0000","last_change":"2023-10-16T12:30:22.404324+0000","last_active":"2023-10-16T12:30:23.401651+0000","last_peered":"2023-10-16T12:30:23.401651+0000","last_clean":"2023-10-16T12:30:23.401651+0000","last_became_active":"2023-10-16T12:30:22.403985+0000","last_became_peered":"2023-10-16T12:30:22.403985+0000","last_unstale":"2023-10-16T12:30:23.401651+0000","last_undegraded":"2023-10-16T12:30:23.401651+0000","last_fullsized":"2023-10-16T12:30:23.401651+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_clean_scrub_stamp":"2023-10-16T12:30:21.373132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:41:13.487445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.429669+0000","last_change":"2023-10-16T12:30:23.422644+0000","last_active":"2023-10-16T12:30:23.429669+0000","last_peered":"2023-10-16T12:30:23.429669+0000","last_clean":"2023-10-16T12:30:23.429669+0000","last_became_active":"2023-10-16T12:30:23.417231+0000","last_became_peered":"2023-10-16T12:30:23.417231+0000","last_unstale":"2023-10-16T12:30:23.429669+0000","last_undegraded":"2023-10-16T12:30:23.429669+0000","last_fullsized":"2023-10-16T12:30:23.429669+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:53:19.541680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":11,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.401306+0000","last_change":"2023-10-16T12:30:22.390167+0000","last_active":"2023-10-16T12:30:23.401306+0000","last_peered":"2023-10-16T12:30:23.401306+0000","last_clean":"2023-10-16T12:30:23.401306+0000","last_became_active":"2023-10-16T12:30:22.389859+0000","last_became_peered":"2023-10-16T12:30:22.389859+0000","last_unstale":"2023-10-16T12:30:23.401306+0000","last_undegraded":"2023-10-16T12:30:23.401306+0000","last_fullsized":"2023-10-16T12:30:23.401306+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_clean_scrub_stamp":"2023-10-16T12:30:21.373132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:47:42.020252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.422052+0000","last_change":"2023-10-16T12:30:23.417841+0000","last_active":"2023-10-16T12:30:23.422052+0000","last_peered":"2023-10-16T12:30:23.422052+0000","last_clean":"2023-10-16T12:30:23.422052+0000","last_became_active":"2023-10-16T12:30:23.417561+0000","last_became_peered":"2023-10-16T12:30:23.417561+0000","last_unstale":"2023-10-16T12:30:23.422052+0000","last_undegraded":"2023-10-16T12:30:23.422052+0000","last_fullsized":"2023-10-16T12:30:23.422052+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:26:57.487238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.15","version":"141'3","reported_seq":14,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.429932+0000","last_change":"2023-10-16T12:30:22.389296+0000","last_active":"2023-10-16T12:30:23.429932+0000","last_peered":"2023-10-16T12:30:23.429932+0000","last_clean":"2023-10-16T12:30:23.429932+0000","last_became_active":"2023-10-16T12:30:22.389000+0000","last_became_peered":"2023-10-16T12:30:22.389000+0000","last_unstale":"2023-10-16T12:30:23.429932+0000","last_undegraded":"2023-10-16T12:30:23.429932+0000","last_fullsized":"2023-10-16T12:30:23.429932+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_clean_scrub_stamp":"2023-10-16T12:30:21.373132+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:28:09.856667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.415758+0000","last_change":"2023-10-16T12:30:23.414377+0000","last_active":"2023-10-16T12:30:23.415758+0000","last_peered":"2023-10-16T12:30:23.415758+0000","last_clean":"2023-10-16T12:30:23.415758+0000","last_became_active":"2023-10-16T12:30:23.413644+0000","last_became_peered":"2023-10-16T12:30:23.413644+0000","last_unstale":"2023-10-16T12:30:23.415758+0000","last_undegraded":"2023-10-16T12:30:23.415758+0000","last_fullsized":"2023-10-16T12:30:23.415758+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:40:20.116892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.14","version":"141'6","reported_seq":17,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.434528+0000","last_change":"2023-10-16T12:30:22.399154+0000","last_active":"2023-10-16T12:30:23.434528+0000","last_peered":"2023-10-16T12:30:23.434528+0000","last_clean":"2023-10-16T12:30:23.434528+0000","last_became_active":"2023-10-16T12:30:22.396265+0000","last_became_peered":"2023-10-16T12:30:22.396265+0000","last_unstale":"2023-10-16T12:30:23.434528+0000","last_undegraded":"2023-10-16T12:30:23.434528+0000","last_fullsized":"2023-10-16T12:30:23.434528+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_clean_scrub_stamp":"2023-10-16T12:30:21.373132+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:50:00.915742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.417721+0000","last_change":"2023-10-16T12:30:23.415605+0000","last_active":"2023-10-16T12:30:23.417721+0000","last_peered":"2023-10-16T12:30:23.417721+0000","last_clean":"2023-10-16T12:30:23.417721+0000","last_became_active":"2023-10-16T12:30:23.414998+0000","last_became_peered":"2023-10-16T12:30:23.414998+0000","last_unstale":"2023-10-16T12:30:23.417721+0000","last_undegraded":"2023-10-16T12:30:23.417721+0000","last_fullsized":"2023-10-16T12:30:23.417721+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:48:36.975059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.13","version":"141'2","reported_seq":13,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.431220+0000","last_change":"2023-10-16T12:30:22.395871+0000","last_active":"2023-10-16T12:30:23.431220+0000","last_peered":"2023-10-16T12:30:23.431220+0000","last_clean":"2023-10-16T12:30:23.431220+0000","last_became_active":"2023-10-16T12:30:22.395554+0000","last_became_peered":"2023-10-16T12:30:22.395554+0000","last_unstale":"2023-10-16T12:30:23.431220+0000","last_undegraded":"2023-10-16T12:30:23.431220+0000","last_fullsized":"2023-10-16T12:30:23.431220+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_clean_scrub_stamp":"2023-10-16T12:30:21.373132+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:09:46.026182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.409767+0000","last_change":"2023-10-16T12:30:23.401955+0000","last_active":"2023-10-16T12:30:23.409767+0000","last_peered":"2023-10-16T12:30:23.409767+0000","last_clean":"2023-10-16T12:30:23.409767+0000","last_became_active":"2023-10-16T12:30:23.400556+0000","last_became_peered":"2023-10-16T12:30:23.400556+0000","last_unstale":"2023-10-16T12:30:23.409767+0000","last_undegraded":"2023-10-16T12:30:23.409767+0000","last_fullsized":"2023-10-16T12:30:23.409767+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:11:59.283205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":11,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.408068+0000","last_change":"2023-10-16T12:30:22.403014+0000","last_active":"2023-10-16T12:30:23.408068+0000","last_peered":"2023-10-16T12:30:23.408068+0000","last_clean":"2023-10-16T12:30:23.408068+0000","last_became_active":"2023-10-16T12:30:22.402718+0000","last_became_peered":"2023-10-16T12:30:22.402718+0000","last_unstale":"2023-10-16T12:30:23.408068+0000","last_undegraded":"2023-10-16T12:30:23.408068+0000","last_fullsized":"2023-10-16T12:30:23.408068+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_clean_scrub_stamp":"2023-10-16T12:30:21.373132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:19:41.065017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.432281+0000","last_change":"2023-10-16T12:30:23.427340+0000","last_active":"2023-10-16T12:30:23.432281+0000","last_peered":"2023-10-16T12:30:23.432281+0000","last_clean":"2023-10-16T12:30:23.432281+0000","last_became_active":"2023-10-16T12:30:23.427069+0000","last_became_peered":"2023-10-16T12:30:23.427069+0000","last_unstale":"2023-10-16T12:30:23.432281+0000","last_undegraded":"2023-10-16T12:30:23.432281+0000","last_fullsized":"2023-10-16T12:30:23.432281+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:54:51.225200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":11,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.407329+0000","last_change":"2023-10-16T12:30:22.396094+0000","last_active":"2023-10-16T12:30:23.407329+0000","last_peered":"2023-10-16T12:30:23.407329+0000","last_clean":"2023-10-16T12:30:23.407329+0000","last_became_active":"2023-10-16T12:30:22.395797+0000","last_became_peered":"2023-10-16T12:30:22.395797+0000","last_unstale":"2023-10-16T12:30:23.407329+0000","last_undegraded":"2023-10-16T12:30:23.407329+0000","last_fullsized":"2023-10-16T12:30:23.407329+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_clean_scrub_stamp":"2023-10-16T12:30:21.373132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:14:14.733913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.415342+0000","last_change":"2023-10-16T12:30:23.413330+0000","last_active":"2023-10-16T12:30:23.415342+0000","last_peered":"2023-10-16T12:30:23.415342+0000","last_clean":"2023-10-16T12:30:23.415342+0000","last_became_active":"2023-10-16T12:30:23.413021+0000","last_became_peered":"2023-10-16T12:30:23.413021+0000","last_unstale":"2023-10-16T12:30:23.415342+0000","last_undegraded":"2023-10-16T12:30:23.415342+0000","last_fullsized":"2023-10-16T12:30:23.415342+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:31:54.709924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.f","version":"141'1","reported_seq":12,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.430568+0000","last_change":"2023-10-16T12:30:22.405161+0000","last_active":"2023-10-16T12:30:23.430568+0000","last_peered":"2023-10-16T12:30:23.430568+0000","last_clean":"2023-10-16T12:30:23.430568+0000","last_became_active":"2023-10-16T12:30:22.404864+0000","last_became_peered":"2023-10-16T12:30:22.404864+0000","last_unstale":"2023-10-16T12:30:23.430568+0000","last_undegraded":"2023-10-16T12:30:23.430568+0000","last_fullsized":"2023-10-16T12:30:23.430568+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_clean_scrub_stamp":"2023-10-16T12:30:21.373132+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:47:15.066363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.426240+0000","last_change":"2023-10-16T12:30:23.421801+0000","last_active":"2023-10-16T12:30:23.426240+0000","last_peered":"2023-10-16T12:30:23.426240+0000","last_clean":"2023-10-16T12:30:23.426240+0000","last_became_active":"2023-10-16T12:30:23.421510+0000","last_became_peered":"2023-10-16T12:30:23.421510+0000","last_unstale":"2023-10-16T12:30:23.426240+0000","last_undegraded":"2023-10-16T12:30:23.426240+0000","last_fullsized":"2023-10-16T12:30:23.426240+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:16:40.062055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.e","version":"141'2","reported_seq":13,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.432158+0000","last_change":"2023-10-16T12:30:22.406039+0000","last_active":"2023-10-16T12:30:23.432158+0000","last_peered":"2023-10-16T12:30:23.432158+0000","last_clean":"2023-10-16T12:30:23.432158+0000","last_became_active":"2023-10-16T12:30:22.405639+0000","last_became_peered":"2023-10-16T12:30:22.405639+0000","last_unstale":"2023-10-16T12:30:23.432158+0000","last_undegraded":"2023-10-16T12:30:23.432158+0000","last_fullsized":"2023-10-16T12:30:23.432158+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_clean_scrub_stamp":"2023-10-16T12:30:21.373132+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:13:23.770593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T12:30:24.398134+0000","last_change":"2023-10-16T12:30:23.406908+0000","last_active":"2023-10-16T12:30:24.398134+0000","last_peered":"2023-10-16T12:30:24.398134+0000","last_clean":"2023-10-16T12:30:24.398134+0000","last_became_active":"2023-10-16T12:30:23.406284+0000","last_became_peered":"2023-10-16T12:30:23.406284+0000","last_unstale":"2023-10-16T12:30:24.398134+0000","last_undegraded":"2023-10-16T12:30:24.398134+0000","last_fullsized":"2023-10-16T12:30:24.398134+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:19:31.176214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.d","version":"141'1","reported_seq":13,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T12:30:24.398640+0000","last_change":"2023-10-16T12:30:22.398802+0000","last_active":"2023-10-16T12:30:24.398640+0000","last_peered":"2023-10-16T12:30:24.398640+0000","last_clean":"2023-10-16T12:30:24.398640+0000","last_became_active":"2023-10-16T12:30:22.398415+0000","last_became_peered":"2023-10-16T12:30:22.398415+0000","last_unstale":"2023-10-16T12:30:24.398640+0000","last_undegraded":"2023-10-16T12:30:24.398640+0000","last_fullsized":"2023-10-16T12:30:24.398640+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_clean_scrub_stamp":"2023-10-16T12:30:21.373132+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:09:46.026182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.415779+0000","last_change":"2023-10-16T12:30:23.414503+0000","last_active":"2023-10-16T12:30:23.415779+0000","last_peered":"2023-10-16T12:30:23.415779+0000","last_clean":"2023-10-16T12:30:23.415779+0000","last_became_active":"2023-10-16T12:30:23.413975+0000","last_became_peered":"2023-10-16T12:30:23.413975+0000","last_unstale":"2023-10-16T12:30:23.415779+0000","last_undegraded":"2023-10-16T12:30:23.415779+0000","last_fullsized":"2023-10-16T12:30:23.415779+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:50:01.919395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T12:30:24.398912+0000","last_change":"2023-10-16T12:30:22.404429+0000","last_active":"2023-10-16T12:30:24.398912+0000","last_peered":"2023-10-16T12:30:24.398912+0000","last_clean":"2023-10-16T12:30:24.398912+0000","last_became_active":"2023-10-16T12:30:22.404105+0000","last_became_peered":"2023-10-16T12:30:22.404105+0000","last_unstale":"2023-10-16T12:30:24.398912+0000","last_undegraded":"2023-10-16T12:30:24.398912+0000","last_fullsized":"2023-10-16T12:30:24.398912+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_clean_scrub_stamp":"2023-10-16T12:30:21.373132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:03:02.390944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T12:30:24.397438+0000","last_change":"2023-10-16T12:30:23.413901+0000","last_active":"2023-10-16T12:30:24.397438+0000","last_peered":"2023-10-16T12:30:24.397438+0000","last_clean":"2023-10-16T12:30:24.397438+0000","last_became_active":"2023-10-16T12:30:23.413176+0000","last_became_peered":"2023-10-16T12:30:23.413176+0000","last_unstale":"2023-10-16T12:30:24.397438+0000","last_undegraded":"2023-10-16T12:30:24.397438+0000","last_fullsized":"2023-10-16T12:30:24.397438+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:00:33.801393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.b","version":"141'2","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T12:30:24.397519+0000","last_change":"2023-10-16T12:30:22.396880+0000","last_active":"2023-10-16T12:30:24.397519+0000","last_peered":"2023-10-16T12:30:24.397519+0000","last_clean":"2023-10-16T12:30:24.397519+0000","last_became_active":"2023-10-16T12:30:22.396167+0000","last_became_peered":"2023-10-16T12:30:22.396167+0000","last_unstale":"2023-10-16T12:30:24.397519+0000","last_undegraded":"2023-10-16T12:30:24.397519+0000","last_fullsized":"2023-10-16T12:30:24.397519+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_clean_scrub_stamp":"2023-10-16T12:30:21.373132+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:54:59.412222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.422072+0000","last_change":"2023-10-16T12:30:23.416839+0000","last_active":"2023-10-16T12:30:23.422072+0000","last_peered":"2023-10-16T12:30:23.422072+0000","last_clean":"2023-10-16T12:30:23.422072+0000","last_became_active":"2023-10-16T12:30:23.416564+0000","last_became_peered":"2023-10-16T12:30:23.416564+0000","last_unstale":"2023-10-16T12:30:23.422072+0000","last_undegraded":"2023-10-16T12:30:23.422072+0000","last_fullsized":"2023-10-16T12:30:23.422072+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:40:24.254569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T12:30:24.398808+0000","last_change":"2023-10-16T12:30:22.396419+0000","last_active":"2023-10-16T12:30:24.398808+0000","last_peered":"2023-10-16T12:30:24.398808+0000","last_clean":"2023-10-16T12:30:24.398808+0000","last_became_active":"2023-10-16T12:30:22.396141+0000","last_became_peered":"2023-10-16T12:30:22.396141+0000","last_unstale":"2023-10-16T12:30:24.398808+0000","last_undegraded":"2023-10-16T12:30:24.398808+0000","last_fullsized":"2023-10-16T12:30:24.398808+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_clean_scrub_stamp":"2023-10-16T12:30:21.373132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:31:53.706271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.422134+0000","last_change":"2023-10-16T12:30:23.419461+0000","last_active":"2023-10-16T12:30:23.422134+0000","last_peered":"2023-10-16T12:30:23.422134+0000","last_clean":"2023-10-16T12:30:23.422134+0000","last_became_active":"2023-10-16T12:30:23.418554+0000","last_became_peered":"2023-10-16T12:30:23.418554+0000","last_unstale":"2023-10-16T12:30:23.422134+0000","last_undegraded":"2023-10-16T12:30:23.422134+0000","last_fullsized":"2023-10-16T12:30:23.422134+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:47:43.023905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":11,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.407483+0000","last_change":"2023-10-16T12:30:22.406381+0000","last_active":"2023-10-16T12:30:23.407483+0000","last_peered":"2023-10-16T12:30:23.407483+0000","last_clean":"2023-10-16T12:30:23.407483+0000","last_became_active":"2023-10-16T12:30:22.406085+0000","last_became_peered":"2023-10-16T12:30:22.406085+0000","last_unstale":"2023-10-16T12:30:23.407483+0000","last_undegraded":"2023-10-16T12:30:23.407483+0000","last_fullsized":"2023-10-16T12:30:23.407483+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_clean_scrub_stamp":"2023-10-16T12:30:21.373132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:10:13.452435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.b","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.415370+0000","last_change":"2023-10-16T12:30:23.413305+0000","last_active":"2023-10-16T12:30:23.415370+0000","last_peered":"2023-10-16T12:30:23.415370+0000","last_clean":"2023-10-16T12:30:23.415370+0000","last_became_active":"2023-10-16T12:30:23.412990+0000","last_became_peered":"2023-10-16T12:30:23.412990+0000","last_unstale":"2023-10-16T12:30:23.415370+0000","last_undegraded":"2023-10-16T12:30:23.415370+0000","last_fullsized":"2023-10-16T12:30:23.415370+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:34:41.863547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T12:30:24.398000+0000","last_change":"2023-10-16T12:30:22.400832+0000","last_active":"2023-10-16T12:30:24.398000+0000","last_peered":"2023-10-16T12:30:24.398000+0000","last_clean":"2023-10-16T12:30:24.398000+0000","last_became_active":"2023-10-16T12:30:22.400416+0000","last_became_peered":"2023-10-16T12:30:22.400416+0000","last_unstale":"2023-10-16T12:30:24.398000+0000","last_undegraded":"2023-10-16T12:30:24.398000+0000","last_fullsized":"2023-10-16T12:30:24.398000+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_clean_scrub_stamp":"2023-10-16T12:30:21.373132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:39:41.097894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T12:30:24.398050+0000","last_change":"2023-10-16T12:30:23.414165+0000","last_active":"2023-10-16T12:30:24.398050+0000","last_peered":"2023-10-16T12:30:24.398050+0000","last_clean":"2023-10-16T12:30:24.398050+0000","last_became_active":"2023-10-16T12:30:23.413596+0000","last_became_peered":"2023-10-16T12:30:23.413596+0000","last_unstale":"2023-10-16T12:30:24.398050+0000","last_undegraded":"2023-10-16T12:30:24.398050+0000","last_fullsized":"2023-10-16T12:30:24.398050+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:52:28.294872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.7","version":"141'2","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T12:30:24.398245+0000","last_change":"2023-10-16T12:30:22.400351+0000","last_active":"2023-10-16T12:30:24.398245+0000","last_peered":"2023-10-16T12:30:24.398245+0000","last_clean":"2023-10-16T12:30:24.398245+0000","last_became_active":"2023-10-16T12:30:22.400061+0000","last_became_peered":"2023-10-16T12:30:22.400061+0000","last_unstale":"2023-10-16T12:30:24.398245+0000","last_undegraded":"2023-10-16T12:30:24.398245+0000","last_fullsized":"2023-10-16T12:30:24.398245+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_clean_scrub_stamp":"2023-10-16T12:30:21.373132+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:20:04.815147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.413225+0000","last_change":"2023-10-16T12:30:23.406733+0000","last_active":"2023-10-16T12:30:23.413225+0000","last_peered":"2023-10-16T12:30:23.413225+0000","last_clean":"2023-10-16T12:30:23.413225+0000","last_became_active":"2023-10-16T12:30:23.406406+0000","last_became_peered":"2023-10-16T12:30:23.406406+0000","last_unstale":"2023-10-16T12:30:23.413225+0000","last_undegraded":"2023-10-16T12:30:23.413225+0000","last_fullsized":"2023-10-16T12:30:23.413225+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:41:14.491098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":11,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.401492+0000","last_change":"2023-10-16T12:30:22.404263+0000","last_active":"2023-10-16T12:30:23.401492+0000","last_peered":"2023-10-16T12:30:23.401492+0000","last_clean":"2023-10-16T12:30:23.401492+0000","last_became_active":"2023-10-16T12:30:22.403947+0000","last_became_peered":"2023-10-16T12:30:22.403947+0000","last_unstale":"2023-10-16T12:30:23.401492+0000","last_undegraded":"2023-10-16T12:30:23.401492+0000","last_fullsized":"2023-10-16T12:30:23.401492+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_clean_scrub_stamp":"2023-10-16T12:30:21.373132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:28:15.424497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T12:30:24.398651+0000","last_change":"2023-10-16T12:30:23.418901+0000","last_active":"2023-10-16T12:30:24.398651+0000","last_peered":"2023-10-16T12:30:24.398651+0000","last_clean":"2023-10-16T12:30:24.398651+0000","last_became_active":"2023-10-16T12:30:23.418620+0000","last_became_peered":"2023-10-16T12:30:23.418620+0000","last_unstale":"2023-10-16T12:30:24.398651+0000","last_undegraded":"2023-10-16T12:30:24.398651+0000","last_fullsized":"2023-10-16T12:30:24.398651+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:10:14.456088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":11,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.403501+0000","last_change":"2023-10-16T12:30:22.405830+0000","last_active":"2023-10-16T12:30:23.403501+0000","last_peered":"2023-10-16T12:30:23.403501+0000","last_clean":"2023-10-16T12:30:23.403501+0000","last_became_active":"2023-10-16T12:30:22.405610+0000","last_became_peered":"2023-10-16T12:30:22.405610+0000","last_unstale":"2023-10-16T12:30:23.403501+0000","last_undegraded":"2023-10-16T12:30:23.403501+0000","last_fullsized":"2023-10-16T12:30:23.403501+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_clean_scrub_stamp":"2023-10-16T12:30:21.373132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:21:11.657992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.419009+0000","last_change":"2023-10-16T12:30:23.417810+0000","last_active":"2023-10-16T12:30:23.419009+0000","last_peered":"2023-10-16T12:30:23.419009+0000","last_clean":"2023-10-16T12:30:23.419009+0000","last_became_active":"2023-10-16T12:30:23.417530+0000","last_became_peered":"2023-10-16T12:30:23.417530+0000","last_unstale":"2023-10-16T12:30:23.419009+0000","last_undegraded":"2023-10-16T12:30:23.419009+0000","last_fullsized":"2023-10-16T12:30:23.419009+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:16:55.739332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T12:30:24.398074+0000","last_change":"2023-10-16T12:30:22.406512+0000","last_active":"2023-10-16T12:30:24.398074+0000","last_peered":"2023-10-16T12:30:24.398074+0000","last_clean":"2023-10-16T12:30:24.398074+0000","last_became_active":"2023-10-16T12:30:22.403874+0000","last_became_peered":"2023-10-16T12:30:22.403874+0000","last_unstale":"2023-10-16T12:30:24.398074+0000","last_undegraded":"2023-10-16T12:30:24.398074+0000","last_fullsized":"2023-10-16T12:30:24.398074+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_clean_scrub_stamp":"2023-10-16T12:30:21.373132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:41:04.465715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T12:30:24.397612+0000","last_change":"2023-10-16T12:30:23.426097+0000","last_active":"2023-10-16T12:30:24.397612+0000","last_peered":"2023-10-16T12:30:24.397612+0000","last_clean":"2023-10-16T12:30:24.397612+0000","last_became_active":"2023-10-16T12:30:23.425735+0000","last_became_peered":"2023-10-16T12:30:23.425735+0000","last_unstale":"2023-10-16T12:30:24.397612+0000","last_undegraded":"2023-10-16T12:30:24.397612+0000","last_fullsized":"2023-10-16T12:30:24.397612+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:16:31.182873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":11,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.423323+0000","last_change":"2023-10-16T12:30:22.394626+0000","last_active":"2023-10-16T12:30:23.423323+0000","last_peered":"2023-10-16T12:30:23.423323+0000","last_clean":"2023-10-16T12:30:23.423323+0000","last_became_active":"2023-10-16T12:30:22.394351+0000","last_became_peered":"2023-10-16T12:30:22.394351+0000","last_unstale":"2023-10-16T12:30:23.423323+0000","last_undegraded":"2023-10-16T12:30:23.423323+0000","last_fullsized":"2023-10-16T12:30:23.423323+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_clean_scrub_stamp":"2023-10-16T12:30:21.373132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:21:57.419693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.422323+0000","last_change":"2023-10-16T12:30:23.420117+0000","last_active":"2023-10-16T12:30:23.422323+0000","last_peered":"2023-10-16T12:30:23.422323+0000","last_clean":"2023-10-16T12:30:23.422323+0000","last_became_active":"2023-10-16T12:30:23.418417+0000","last_became_peered":"2023-10-16T12:30:23.418417+0000","last_unstale":"2023-10-16T12:30:23.422323+0000","last_undegraded":"2023-10-16T12:30:23.422323+0000","last_fullsized":"2023-10-16T12:30:23.422323+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:34:07.114770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":11,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.426712+0000","last_change":"2023-10-16T12:30:22.398876+0000","last_active":"2023-10-16T12:30:23.426712+0000","last_peered":"2023-10-16T12:30:23.426712+0000","last_clean":"2023-10-16T12:30:23.426712+0000","last_became_active":"2023-10-16T12:30:22.398006+0000","last_became_peered":"2023-10-16T12:30:22.398006+0000","last_unstale":"2023-10-16T12:30:23.426712+0000","last_undegraded":"2023-10-16T12:30:23.426712+0000","last_fullsized":"2023-10-16T12:30:23.426712+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_clean_scrub_stamp":"2023-10-16T12:30:21.373132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:25:32.064591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T12:30:24.398692+0000","last_change":"2023-10-16T12:30:23.411352+0000","last_active":"2023-10-16T12:30:24.398692+0000","last_peered":"2023-10-16T12:30:24.398692+0000","last_clean":"2023-10-16T12:30:24.398692+0000","last_became_active":"2023-10-16T12:30:23.411044+0000","last_became_peered":"2023-10-16T12:30:23.411044+0000","last_unstale":"2023-10-16T12:30:24.398692+0000","last_undegraded":"2023-10-16T12:30:24.398692+0000","last_fullsized":"2023-10-16T12:30:24.398692+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:26:57.487238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T12:30:24.397483+0000","last_change":"2023-10-16T12:30:22.401410+0000","last_active":"2023-10-16T12:30:24.397483+0000","last_peered":"2023-10-16T12:30:24.397483+0000","last_clean":"2023-10-16T12:30:24.397483+0000","last_became_active":"2023-10-16T12:30:22.401101+0000","last_became_peered":"2023-10-16T12:30:22.401101+0000","last_unstale":"2023-10-16T12:30:24.397483+0000","last_undegraded":"2023-10-16T12:30:24.397483+0000","last_fullsized":"2023-10-16T12:30:24.397483+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_clean_scrub_stamp":"2023-10-16T12:30:21.373132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:13:21.620462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.409943+0000","last_change":"2023-10-16T12:30:23.403923+0000","last_active":"2023-10-16T12:30:23.409943+0000","last_peered":"2023-10-16T12:30:23.409943+0000","last_clean":"2023-10-16T12:30:23.409943+0000","last_became_active":"2023-10-16T12:30:23.403627+0000","last_became_peered":"2023-10-16T12:30:23.403627+0000","last_unstale":"2023-10-16T12:30:23.409943+0000","last_undegraded":"2023-10-16T12:30:23.409943+0000","last_fullsized":"2023-10-16T12:30:23.409943+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:48:30.457561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T12:30:24.398014+0000","last_change":"2023-10-16T12:30:22.401281+0000","last_active":"2023-10-16T12:30:24.398014+0000","last_peered":"2023-10-16T12:30:24.398014+0000","last_clean":"2023-10-16T12:30:24.398014+0000","last_became_active":"2023-10-16T12:30:22.400837+0000","last_became_peered":"2023-10-16T12:30:22.400837+0000","last_unstale":"2023-10-16T12:30:24.398014+0000","last_undegraded":"2023-10-16T12:30:24.398014+0000","last_fullsized":"2023-10-16T12:30:24.398014+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_clean_scrub_stamp":"2023-10-16T12:30:21.373132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:53:18.538027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T12:30:24.398739+0000","last_change":"2023-10-16T12:30:23.412234+0000","last_active":"2023-10-16T12:30:24.398739+0000","last_peered":"2023-10-16T12:30:24.398739+0000","last_clean":"2023-10-16T12:30:24.398739+0000","last_became_active":"2023-10-16T12:30:23.411539+0000","last_became_peered":"2023-10-16T12:30:23.411539+0000","last_unstale":"2023-10-16T12:30:24.398739+0000","last_undegraded":"2023-10-16T12:30:24.398739+0000","last_fullsized":"2023-10-16T12:30:24.398739+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:18:36.577516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.10","version":"141'8","reported_seq":19,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.433481+0000","last_change":"2023-10-16T12:30:22.395953+0000","last_active":"2023-10-16T12:30:23.433481+0000","last_peered":"2023-10-16T12:30:23.433481+0000","last_clean":"2023-10-16T12:30:23.433481+0000","last_became_active":"2023-10-16T12:30:22.395623+0000","last_became_peered":"2023-10-16T12:30:22.395623+0000","last_unstale":"2023-10-16T12:30:23.433481+0000","last_undegraded":"2023-10-16T12:30:23.433481+0000","last_fullsized":"2023-10-16T12:30:23.433481+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_clean_scrub_stamp":"2023-10-16T12:30:21.373132+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:03:02.390944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"132'227","reported_seq":666,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.397265+0000","last_change":"2023-10-16T12:10:34.746820+0000","last_active":"2023-10-16T12:30:23.397265+0000","last_peered":"2023-10-16T12:30:23.397265+0000","last_clean":"2023-10-16T12:30:23.397265+0000","last_became_active":"2023-10-16T12:10:34.745574+0000","last_became_peered":"2023-10-16T12:10:34.745574+0000","last_unstale":"2023-10-16T12:30:23.397265+0000","last_undegraded":"2023-10-16T12:30:23.397265+0000","last_fullsized":"2023-10-16T12:30:23.397265+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_clean_scrub_stamp":"2023-10-16T12:10:33.577111+0000","objects_scrubbed":0,"log_size":227,"log_dups_size":0,"ondisk_log_size":227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:14.461986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1076,"num_read_kb":2160,"num_write":359,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":18,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1076,"num_read_kb":2160,"num_write":359,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":227,"ondisk_log_size":227,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575083,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47388,"kb_used_data":2308,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324452,"statfs":{"total":96636764160,"available":96588238848,"internally_reserved":0,"allocated":2363392,"data_stored":1475796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":46135755},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:30:18 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.61199999999999999,"15min":0.60199999999999998},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.27400000000000002},"max":{"1min":0.89300000000000002,"5min":4.2469999999999999,"15min":4.2469999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.57199999999999995,"15min":0.58199999999999996},"min":{"1min":0.35299999999999998,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.84899999999999998,"5min":0.85299999999999998,"15min":0.89900000000000002},"last":0.59099999999999997}]},{"osd":1,"last update":"Mon Oct 16 12:30:18 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.58499999999999996,"15min":0.59999999999999998},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.247},"max":{"1min":0.80200000000000005,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.58999999999999997,"15min":0.60099999999999998},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.32400000000000001},"max":{"1min":0.79200000000000004,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.497}]},{"osd":2,"last update":"Mon Oct 16 12:30:18 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.64400000000000002,"15min":0.65700000000000003},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.88600000000000001,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.64800000000000002,"15min":0.66300000000000003},"min":{"1min":0.44800000000000001,"5min":0.38100000000000001,"15min":0.376},"max":{"1min":0.88300000000000001,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.65500000000000003}]},{"osd":3,"last update":"Mon Oct 16 12:30:18 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.61099999999999999,"15min":0.61499999999999999},"min":{"1min":0.377,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.63,"15min":0.63300000000000001},"min":{"1min":0.41199999999999998,"5min":0.372,"15min":0.372},"max":{"1min":0.81000000000000005,"5min":0.996,"15min":1.2509999999999999},"last":0.66700000000000004}]},{"osd":4,"last update":"Mon Oct 16 12:30:18 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.54400000000000004,"15min":0.53000000000000003},"min":{"1min":0.38600000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.80300000000000005,"5min":0.89700000000000002,"15min":1.383},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.627,"5min":0.63500000000000001,"15min":0.61499999999999999},"min":{"1min":0.39800000000000002,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.88200000000000001,"5min":0.93600000000000005,"15min":1.0109999999999999},"last":0.68600000000000005}]},{"osd":5,"last update":"Mon Oct 16 12:30:18 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.56899999999999995,"15min":0.54000000000000004},"min":{"1min":0.311,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.77100000000000002,"5min":0.93400000000000005,"15min":1.3109999999999999},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.628,"5min":0.60799999999999998,"15min":0.58799999999999997},"min":{"1min":0.378,"5min":0.34699999999999998,"15min":0.32400000000000001},"max":{"1min":0.86099999999999999,"5min":0.93600000000000005,"15min":1.4359999999999999},"last":0.76000000000000001}]},{"osd":6,"last update":"Mon Oct 16 12:30:18 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.64800000000000002,"15min":0.625},"min":{"1min":0.41499999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.93000000000000005,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.64900000000000002,"15min":0.63},"min":{"1min":0.35099999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.91600000000000004,"5min":0.97199999999999998,"15min":1.1259999999999999},"last":0.745}]}]},{"osd":6,"up_from":13,"seq":55834575081,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37060,"kb_used_data":1716,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334780,"statfs":{"total":96636764160,"available":96598814720,"internally_reserved":0,"allocated":1757184,"data_stored":885234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:30:07 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.57499999999999996,"15min":0.60899999999999999},"min":{"1min":0.35199999999999998,"5min":0.29299999999999998,"15min":0.24299999999999999},"max":{"1min":0.87,"5min":0.88200000000000001,"15min":1.196},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.59699999999999998,"15min":0.621},"min":{"1min":0.436,"5min":0.38500000000000001,"15min":0.33600000000000002},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.997},"last":0.56499999999999995}]},{"osd":1,"last update":"Mon Oct 16 12:30:07 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.53300000000000003,"15min":0.56000000000000005},"min":{"1min":0.35699999999999998,"5min":0.33100000000000002,"15min":0.29399999999999998},"max":{"1min":0.745,"5min":0.82199999999999995,"15min":0.875},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.57399999999999995,"15min":0.58599999999999997},"min":{"1min":0.40400000000000003,"5min":0.29799999999999999,"15min":0.29499999999999998},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":1.091},"last":0.65600000000000003}]},{"osd":2,"last update":"Mon Oct 16 12:30:07 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56499999999999995,"15min":0.58999999999999997},"min":{"1min":0.377,"5min":0.35599999999999998,"15min":0.34499999999999997},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.97099999999999997},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.56899999999999995,"15min":0.59499999999999997},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.27100000000000002},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":2.3879999999999999},"last":0.50600000000000001}]},{"osd":3,"last update":"Mon Oct 16 12:30:07 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.60699999999999998,"15min":0.624},"min":{"1min":0.41599999999999998,"5min":0.37,"15min":0.34200000000000003},"max":{"1min":0.82299999999999995,"5min":0.88,"15min":0.93999999999999995},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.66700000000000004,"15min":0.67800000000000005},"min":{"1min":0.504,"5min":0.41899999999999998,"15min":0.40899999999999997},"max":{"1min":0.90300000000000002,"5min":0.94499999999999995,"15min":1.3080000000000001},"last":0.77800000000000002}]},{"osd":4,"last update":"Mon Oct 16 12:30:07 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.52200000000000002,"15min":0.51600000000000001},"min":{"1min":0.35899999999999999,"5min":0.30599999999999999,"15min":0.29899999999999999},"max":{"1min":0.65700000000000003,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.60099999999999998,"15min":0.59699999999999998},"min":{"1min":0.38700000000000001,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.79400000000000004,"5min":0.82299999999999995,"15min":1.2250000000000001},"last":0.41399999999999998}]},{"osd":5,"last update":"Mon Oct 16 12:30:07 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.57899999999999996,"15min":0.57699999999999996},"min":{"1min":0.41499999999999998,"5min":0.30299999999999999,"15min":0.30099999999999999},"max":{"1min":0.70399999999999996,"5min":0.98799999999999999,"15min":1.0589999999999999},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.64300000000000002,"15min":0.626},"min":{"1min":0.33400000000000002,"5min":0.32100000000000001,"15min":0.311},"max":{"1min":0.84299999999999997,"5min":1.0029999999999999,"15min":1.3540000000000001},"last":0.57899999999999996}]},{"osd":7,"last update":"Mon Oct 16 12:30:07 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.58299999999999996,"15min":0.57599999999999996},"min":{"1min":0.42499999999999999,"5min":0.29699999999999999,"15min":0.26000000000000001},"max":{"1min":0.68799999999999994,"5min":0.92700000000000005,"15min":1.29},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.623,"5min":0.63400000000000001,"15min":0.61899999999999999},"min":{"1min":0.436,"5min":0.32900000000000001,"15min":0.29799999999999999},"max":{"1min":0.873,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.48899999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575081,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37056,"kb_used_data":1716,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334784,"statfs":{"total":96636764160,"available":96598818816,"internally_reserved":0,"allocated":1757184,"data_stored":884674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":36174281},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.374,"15min":0.38},"min":{"1min":0.26600000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.627,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.434,"15min":0.46000000000000002},"min":{"1min":0.28899999999999998,"5min":0.28399999999999997,"15min":0.26700000000000002},"max":{"1min":0.60499999999999998,"5min":0.66300000000000003,"15min":1.0189999999999999},"last":0.35699999999999998}]},{"osd":2,"last update":"Mon Oct 16 12:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.39800000000000002,"15min":0.42599999999999999},"min":{"1min":0.26900000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.59199999999999997,"5min":0.65300000000000002,"15min":0.93000000000000005},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.44,"5min":0.45000000000000001,"15min":0.46700000000000003},"min":{"1min":0.33700000000000002,"5min":0.29099999999999998,"15min":0.28199999999999997},"max":{"1min":0.61699999999999999,"5min":0.69199999999999995,"15min":0.98099999999999998},"last":0.376}]},{"osd":3,"last update":"Mon Oct 16 12:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.38200000000000001,"15min":0.40300000000000002},"min":{"1min":0.247,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.56999999999999995,"5min":0.69599999999999995,"15min":0.98799999999999999},"last":0.311},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.42999999999999999,"15min":0.45300000000000001},"min":{"1min":0.30299999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.751,"5min":0.751,"15min":0.91200000000000003},"last":0.29299999999999998}]},{"osd":4,"last update":"Mon Oct 16 12:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.60399999999999998,"15min":0.627},"min":{"1min":0.40500000000000003,"5min":0.35499999999999998,"15min":0.35199999999999998},"max":{"1min":0.74099999999999999,"5min":0.878,"15min":1.0409999999999999},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.624,"15min":0.64200000000000002},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.39100000000000001},"max":{"1min":0.86699999999999999,"5min":0.95399999999999996,"15min":0.98599999999999999},"last":0.48199999999999998}]},{"osd":5,"last update":"Mon Oct 16 12:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.63900000000000001,"15min":0.66300000000000003},"min":{"1min":0.42899999999999999,"5min":0.42599999999999999,"15min":0.38300000000000001},"max":{"1min":0.95399999999999996,"5min":1.069,"15min":1.069},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.66300000000000003,"15min":0.68400000000000005},"min":{"1min":0.437,"5min":0.437,"15min":0.38900000000000001},"max":{"1min":0.93899999999999995,"5min":1.034,"15min":1.0620000000000001},"last":0.39300000000000002}]},{"osd":6,"last update":"Mon Oct 16 12:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.63400000000000001,"15min":0.65100000000000002},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.82599999999999996,"5min":1.044,"15min":1.044},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.63100000000000001,"15min":0.65800000000000003},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.32900000000000001},"max":{"1min":0.90400000000000003,"5min":1.0049999999999999,"15min":1.048},"last":0.436}]},{"osd":7,"last update":"Mon Oct 16 12:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.64800000000000002,"15min":0.68300000000000005},"min":{"1min":0.45800000000000002,"5min":0.42099999999999999,"15min":0.33700000000000002},"max":{"1min":0.80600000000000005,"5min":1.0469999999999999,"15min":1.3400000000000001},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.627,"5min":0.67600000000000005,"15min":0.69699999999999995},"min":{"1min":0.47299999999999998,"5min":0.45700000000000002,"15min":0.42099999999999999},"max":{"1min":0.89100000000000001,"5min":1.006,"15min":1.02},"last":0.45300000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575082,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37652,"kb_used_data":2308,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334188,"statfs":{"total":96636764160,"available":96598208512,"internally_reserved":0,"allocated":2363392,"data_stored":1481718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 12:29:58 2023","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.39500000000000002,"15min":0.39200000000000002},"min":{"1min":0.27100000000000002,"5min":0.219,"15min":0.219},"max":{"1min":0.57099999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.40699999999999997,"15min":0.41799999999999998},"min":{"1min":0.251,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.73099999999999998},"last":0.35499999999999998}]},{"osd":2,"last update":"Mon Oct 16 12:29:58 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.42299999999999999,"15min":0.42499999999999999},"min":{"1min":0.314,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.44800000000000001,"15min":0.45300000000000001},"min":{"1min":0.29999999999999999,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.57299999999999995,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.438}]},{"osd":3,"last update":"Mon Oct 16 12:29:58 2023","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.42799999999999999,"15min":0.42599999999999999},"min":{"1min":0.28299999999999997,"5min":0.26300000000000001,"15min":0.23699999999999999},"max":{"1min":0.56399999999999995,"5min":0.62,"15min":0.79400000000000004},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.48299999999999998,"15min":0.48799999999999999},"min":{"1min":0.314,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.63400000000000001,"5min":0.79200000000000004,"15min":0.878},"last":0.41499999999999998}]},{"osd":4,"last update":"Mon Oct 16 12:29:58 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.64000000000000001,"15min":0.65100000000000002},"min":{"1min":0.40200000000000002,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.82499999999999996,"5min":1.05,"15min":1.0960000000000001},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.63,"15min":0.65600000000000003},"min":{"1min":0.38800000000000001,"5min":0.35599999999999998,"15min":0.34100000000000003},"max":{"1min":0.97099999999999997,"5min":1.0549999999999999,"15min":1.1719999999999999},"last":0.496}]},{"osd":5,"last update":"Mon Oct 16 12:29:58 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.629,"15min":0.65100000000000002},"min":{"1min":0.39200000000000002,"5min":0.373,"15min":0.373},"max":{"1min":0.88100000000000001,"5min":0.97699999999999998,"15min":1.127},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.66800000000000004,"15min":0.67800000000000005},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39000000000000001},"max":{"1min":0.90000000000000002,"5min":1.0700000000000001,"15min":1.1160000000000001},"last":0.59999999999999998}]},{"osd":6,"last update":"Mon Oct 16 12:29:58 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.67200000000000004,"15min":0.68500000000000005},"min":{"1min":0.42699999999999999,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.85599999999999998,"5min":0.96899999999999997,"15min":1.079},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.66600000000000004,"15min":0.68700000000000006},"min":{"1min":0.40999999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.86699999999999999,"5min":0.96399999999999997,"15min":1.1499999999999999},"last":0.64500000000000002}]},{"osd":7,"last update":"Mon Oct 16 12:29:58 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.68500000000000005,"15min":0.69999999999999996},"min":{"1min":0.47299999999999998,"5min":0.436,"15min":0.41499999999999998},"max":{"1min":0.83199999999999996,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.64800000000000002,"15min":0.68500000000000005},"min":{"1min":0.438,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.93200000000000005,"5min":0.97499999999999998,"15min":1.0569999999999999},"last":0.58599999999999997}]}]},{"osd":2,"up_from":13,"seq":55834575082,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63420,"kb_used_data":1712,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94308420,"statfs":{"total":96636764160,"available":96571822080,"internally_reserved":0,"allocated":1753088,"data_stored":884652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":63175110},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:30:05 2023","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.36899999999999999,"15min":0.379},"min":{"1min":0.222,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.51200000000000001,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.42699999999999999,"15min":0.443},"min":{"1min":0.28599999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.497,"5min":0.66500000000000004,"15min":0.82999999999999996},"last":0.39900000000000002}]},{"osd":1,"last update":"Mon Oct 16 12:30:05 2023","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.441,"15min":0.46100000000000002},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.64500000000000002,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.439},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.40500000000000003,"15min":0.42999999999999999},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.57199999999999995,"5min":0.59199999999999997,"15min":0.69699999999999995},"last":0.26200000000000001}]},{"osd":3,"last update":"Mon Oct 16 12:30:05 2023","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.39400000000000002,"15min":0.40899999999999997},"min":{"1min":0.25,"5min":0.214,"15min":0.214},"max":{"1min":0.47799999999999998,"5min":0.69399999999999995,"15min":0.89300000000000002},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.498,"15min":0.51200000000000001},"min":{"1min":0.33200000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.74099999999999999,"5min":0.78000000000000003,"15min":0.79600000000000004},"last":0.501}]},{"osd":4,"last update":"Mon Oct 16 12:30:05 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.63800000000000001,"15min":0.66800000000000004},"min":{"1min":0.35499999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.753,"5min":0.93799999999999994,"15min":1.3839999999999999},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.628,"15min":0.66500000000000004},"min":{"1min":0.42199999999999999,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.79900000000000004,"5min":0.97599999999999998,"15min":1.0109999999999999},"last":0.60199999999999998}]},{"osd":5,"last update":"Mon Oct 16 12:30:05 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.63500000000000001,"15min":0.65500000000000003},"min":{"1min":0.41599999999999998,"5min":0.35999999999999999,"15min":0.312},"max":{"1min":0.86699999999999999,"5min":1.093,"15min":1.093},"last":0.624},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.61299999999999999,"15min":0.64700000000000002},"min":{"1min":0.40999999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.70499999999999996,"5min":1.0089999999999999,"15min":1.331},"last":0.46300000000000002}]},{"osd":6,"last update":"Mon Oct 16 12:30:05 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59599999999999997,"15min":0.626},"min":{"1min":0.42099999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.76200000000000001,"5min":0.99299999999999999,"15min":1.0369999999999999},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.65100000000000002,"15min":0.67600000000000005},"min":{"1min":0.437,"5min":0.40799999999999997,"15min":0.40000000000000002},"max":{"1min":0.83799999999999997,"5min":1.032,"15min":1.032},"last":0.68400000000000005}]},{"osd":7,"last update":"Mon Oct 16 12:30:05 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.64800000000000002,"15min":0.66600000000000004},"min":{"1min":0.432,"5min":0.40300000000000002,"15min":0.38400000000000001},"max":{"1min":0.78300000000000003,"5min":0.96399999999999997,"15min":1.016},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.65000000000000002,"15min":0.67800000000000005},"min":{"1min":0.44400000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.86199999999999999,"5min":1.145,"15min":1.145},"last":0.56999999999999995}]}]},{"osd":3,"up_from":13,"seq":55834575081,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37068,"kb_used_data":1728,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334772,"statfs":{"total":96636764160,"available":96598806528,"internally_reserved":0,"allocated":1769472,"data_stored":886298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:30:14 2023","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.38900000000000001,"15min":0.39700000000000002},"min":{"1min":0.22500000000000001,"5min":0.216,"15min":0.21099999999999999},"max":{"1min":0.46999999999999997,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.39200000000000002,"15min":0.40799999999999997},"min":{"1min":0.28000000000000003,"5min":0.254,"15min":0.254},"max":{"1min":0.54000000000000004,"5min":0.61899999999999999,"15min":0.73099999999999998},"last":0.35699999999999998}]},{"osd":1,"last update":"Mon Oct 16 12:30:14 2023","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.39900000000000002,"15min":0.41299999999999998},"min":{"1min":0.26400000000000001,"5min":0.245,"15min":0.24399999999999999},"max":{"1min":0.57499999999999996,"5min":0.70699999999999996,"15min":0.70799999999999996},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.438,"15min":0.45800000000000002},"min":{"1min":0.29499999999999998,"5min":0.27000000000000002,"15min":0.25800000000000001},"max":{"1min":0.60399999999999998,"5min":0.72699999999999998,"15min":0.747},"last":0.33700000000000002}]},{"osd":2,"last update":"Mon Oct 16 12:30:14 2023","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.45300000000000001,"15min":0.46999999999999997},"min":{"1min":0.313,"5min":0.29799999999999999,"15min":0.29199999999999998},"max":{"1min":0.63,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.47899999999999998,"15min":0.498},"min":{"1min":0.311,"5min":0.30599999999999999,"15min":0.27100000000000002},"max":{"1min":0.72799999999999998,"5min":0.74299999999999999,"15min":0.752},"last":0.38}]},{"osd":4,"last update":"Mon Oct 16 12:30:14 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.623,"15min":0.64500000000000002},"min":{"1min":0.39200000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.89600000000000002,"5min":0.98299999999999998,"15min":1.1539999999999999},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.622,"15min":0.65600000000000003},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.92100000000000004,"5min":0.95399999999999996,"15min":1.0409999999999999},"last":0.64400000000000002}]},{"osd":5,"last update":"Mon Oct 16 12:30:14 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.65200000000000002,"15min":0.68100000000000005},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.89400000000000002,"5min":1.0369999999999999,"15min":1.1080000000000001},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.63200000000000001,"15min":0.66000000000000003},"min":{"1min":0.45200000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.89000000000000001,"5min":0.97299999999999998,"15min":1.022},"last":0.59799999999999998}]},{"osd":6,"last update":"Mon Oct 16 12:30:14 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63300000000000001,"15min":0.66500000000000004},"min":{"1min":0.437,"5min":0.42999999999999999,"15min":0.36899999999999999},"max":{"1min":0.875,"5min":0.94199999999999995,"15min":1.0760000000000001},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.67400000000000004,"15min":0.70199999999999996},"min":{"1min":0.434,"5min":0.434,"15min":0.38600000000000001},"max":{"1min":0.89100000000000001,"5min":1.0349999999999999,"15min":1.0409999999999999},"last":0.57799999999999996}]},{"osd":7,"last update":"Mon Oct 16 12:30:14 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.64700000000000002,"15min":0.67900000000000005},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.38},"max":{"1min":0.85999999999999999,"5min":1.0149999999999999,"15min":1.2430000000000001},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.68500000000000005,"15min":0.70499999999999996},"min":{"1min":0.503,"5min":0.44,"15min":0.42699999999999999},"max":{"1min":0.88100000000000001,"5min":1.002,"15min":1.1339999999999999},"last":0.61499999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575080,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32916,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338924,"statfs":{"total":96636764160,"available":96603058176,"internally_reserved":0,"allocated":1765376,"data_stored":886264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.56599999999999995,"15min":0.58999999999999997},"min":{"1min":0.41099999999999998,"5min":0.30399999999999999,"15min":0.27900000000000003},"max":{"1min":0.752,"5min":0.78500000000000003,"15min":0.94599999999999995},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.57199999999999995,"15min":0.59699999999999998},"min":{"1min":0.42399999999999999,"5min":0.29999999999999999,"15min":0.27600000000000002},"max":{"1min":0.73299999999999998,"5min":0.83199999999999996,"15min":0.88700000000000001},"last":0.60599999999999998}]},{"osd":1,"last update":"Mon Oct 16 12:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.58599999999999997,"15min":0.60799999999999998},"min":{"1min":0.32900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.76200000000000001,"5min":0.82599999999999996,"15min":0.92600000000000005},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.61399999999999999},"min":{"1min":0.434,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.98099999999999998},"last":0.76700000000000002}]},{"osd":2,"last update":"Mon Oct 16 12:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.58399999999999996,"15min":0.61799999999999999},"min":{"1min":0.34999999999999998,"5min":0.33800000000000002,"15min":0.32000000000000001},"max":{"1min":0.745,"5min":0.84299999999999997,"15min":1.012},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.59699999999999998,"15min":0.624},"min":{"1min":0.38200000000000001,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.76000000000000001,"5min":0.85599999999999998,"15min":0.94199999999999995},"last":0.71899999999999997}]},{"osd":3,"last update":"Mon Oct 16 12:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.56899999999999995,"15min":0.60199999999999998},"min":{"1min":0.41799999999999998,"5min":0.35599999999999998,"15min":0.33700000000000002},"max":{"1min":0.70699999999999996,"5min":0.82299999999999995,"15min":0.93200000000000005},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.63,"5min":0.63300000000000001,"15min":0.64300000000000002},"min":{"1min":0.47099999999999997,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.80000000000000004,"5min":0.90600000000000003,"15min":0.97199999999999998},"last":0.69799999999999995}]},{"osd":5,"last update":"Mon Oct 16 12:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.52100000000000002,"15min":0.50600000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.72199999999999998,"5min":0.82599999999999996,"15min":0.95699999999999996},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.53000000000000003,"15min":0.52500000000000002},"min":{"1min":0.373,"5min":0.33900000000000002,"15min":0.308},"max":{"1min":0.73399999999999999,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.67800000000000005}]},{"osd":6,"last update":"Mon Oct 16 12:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.55600000000000005,"15min":0.55700000000000005},"min":{"1min":0.39000000000000001,"5min":0.33800000000000002,"15min":0.32500000000000001},"max":{"1min":0.70699999999999996,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.628},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.63800000000000001,"15min":0.63700000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.32000000000000001},"max":{"1min":0.84499999999999997,"5min":0.89300000000000002,"15min":0.99299999999999999},"last":0.82399999999999995}]},{"osd":7,"last update":"Mon Oct 16 12:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59199999999999997,"15min":0.58499999999999996},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.77700000000000002,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.629,"15min":0.629},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.29999999999999999},"max":{"1min":0.82899999999999996,"5min":1.006,"15min":1.006},"last":0.751}]}]},{"osd":5,"up_from":13,"seq":55834575081,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55112,"kb_used_data":1720,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316728,"statfs":{"total":96636764160,"available":96580329472,"internally_reserved":0,"allocated":1761280,"data_stored":884812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:30:04 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.55500000000000005,"15min":0.58599999999999997},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.28000000000000003},"max":{"1min":0.81399999999999995,"5min":0.84499999999999997,"15min":0.873},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.59399999999999997,"15min":0.61499999999999999},"min":{"1min":0.34599999999999997,"5min":0.34399999999999997,"15min":0.27600000000000002},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.64900000000000002}]},{"osd":1,"last update":"Mon Oct 16 12:30:04 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.58999999999999997,"15min":0.60999999999999999},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.28999999999999998},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57899999999999996,"15min":0.61199999999999999},"min":{"1min":0.31900000000000001,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.78600000000000003,"5min":0.84799999999999998,"15min":0.89000000000000001},"last":0.63500000000000001}]},{"osd":2,"last update":"Mon Oct 16 12:30:04 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.59299999999999997,"15min":0.62},"min":{"1min":0.40899999999999997,"5min":0.36899999999999999,"15min":0.29099999999999998},"max":{"1min":0.82499999999999996,"5min":0.876,"15min":0.94399999999999995},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.60799999999999998},"min":{"1min":0.40400000000000003,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.72699999999999998}]},{"osd":3,"last update":"Mon Oct 16 12:30:04 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.60999999999999999,"15min":0.63200000000000001},"min":{"1min":0.39700000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.76900000000000002,"5min":0.90400000000000003,"15min":0.97499999999999998},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.625,"15min":0.64300000000000002},"min":{"1min":0.45100000000000001,"5min":0.40400000000000003,"15min":0.35999999999999999},"max":{"1min":0.75,"5min":0.88700000000000001,"15min":0.93200000000000005},"last":0.69999999999999996}]},{"osd":4,"last update":"Mon Oct 16 12:30:04 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49299999999999999,"15min":0.50800000000000001},"min":{"1min":0.35999999999999999,"5min":0.28100000000000003,"15min":0.25800000000000001},"max":{"1min":0.69399999999999995,"5min":0.81299999999999994,"15min":0.86399999999999999},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.36399999999999999,"5min":0.33900000000000002,"15min":0.33500000000000002},"max":{"1min":0.83999999999999997,"5min":0.876,"15min":1.347},"last":0.53800000000000003}]},{"osd":6,"last update":"Mon Oct 16 12:30:04 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.371,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.58099999999999996,"15min":0.59799999999999998},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.74299999999999999,"5min":0.88400000000000001,"15min":0.92700000000000005},"last":0.61499999999999999}]},{"osd":7,"last update":"Mon Oct 16 12:30:04 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.621,"15min":0.621},"min":{"1min":0.41699999999999998,"5min":0.41499999999999998,"15min":0.32300000000000001},"max":{"1min":0.76800000000000002,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.64800000000000002,"15min":0.65700000000000003},"min":{"1min":0.438,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.86799999999999999,"5min":0.93899999999999995,"15min":1.1180000000000001},"last":0.66300000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T12:30:27.578 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T12:30:27.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.735+0000 7f2d88d40700 1 Processor -- start 2023-10-16T12:30:27.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.735+0000 7f2d88d40700 1 -- start start 2023-10-16T12:30:27.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.736+0000 7f2d88d40700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2d840ce4f0 0x7f2d840d0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:27.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.736+0000 7f2d88d40700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2d8415ba10 0x7f2d840cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:27.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.736+0000 7f2d88d40700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d840b0970 con 0x7f2d8415b630 2023-10-16T12:30:27.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.736+0000 7f2d88d40700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d840cd690 con 0x7f2d8415ba10 2023-10-16T12:30:27.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.736+0000 7f2d88d40700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d840cd970 con 0x7f2d840ce4f0 2023-10-16T12:30:27.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.736+0000 7f2d82d9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2d8415ba10 0x7f2d840cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:27.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.736+0000 7f2d8259c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2d8415b630 0x7f2d840cdde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34542/0 (socket says 172.21.15.16:34542) 2023-10-16T12:30:27.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.736+0000 7f2d8259c700 1 -- 172.21.15.16:0/2291187267 learned_addr learned my addr 172.21.15.16:0/2291187267 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:30:27.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.737+0000 7f2d81d9b700 1 --2- 172.21.15.16:0/2291187267 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2d840ce4f0 0x7f2d840d0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:27.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.737+0000 7f2d82d9d700 1 -- 172.21.15.16:0/2291187267 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2d840ce4f0 msgr2=0x7f2d840d0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:27.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.737+0000 7f2d82d9d700 1 --2- 172.21.15.16:0/2291187267 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2d840ce4f0 0x7f2d840d0380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:27.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.737+0000 7f2d82d9d700 1 -- 172.21.15.16:0/2291187267 >> v1:172.21.15.16:6789/0 conn(0x7f2d8415b630 legacy=0x7f2d840cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:27.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.737+0000 7f2d82d9d700 1 -- 172.21.15.16:0/2291187267 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d840b0c30 con 0x7f2d8415ba10 2023-10-16T12:30:27.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.737+0000 7f2d8159a700 1 -- 172.21.15.16:0/2291187267 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3062888494 0 0) 0x7f2d840b0970 con 0x7f2d8415b630 2023-10-16T12:30:27.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.737+0000 7f2d81d9b700 1 --2- 172.21.15.16:0/2291187267 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2d840ce4f0 0x7f2d840d0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:30:27.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.737+0000 7f2d82d9d700 1 --2- 172.21.15.16:0/2291187267 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d8415ba10 0x7f2d840cf290 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c009d90 tx=0x7f2d7c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e3cb6d4cf1e466d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:27.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.737+0000 7f2d8159a700 1 -- 172.21.15.16:0/2291187267 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d7c00e070 con 0x7f2d8415ba10 2023-10-16T12:30:27.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.737+0000 7f2d8159a700 1 -- 172.21.15.16:0/2291187267 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d7c00bbd0 con 0x7f2d8415ba10 2023-10-16T12:30:27.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.737+0000 7f2d8159a700 1 -- 172.21.15.16:0/2291187267 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d7c012430 con 0x7f2d8415ba10 2023-10-16T12:30:27.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.738+0000 7f2d88d40700 1 -- 172.21.15.16:0/2291187267 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d8415ba10 msgr2=0x7f2d840cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:27.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.738+0000 7f2d88d40700 1 --2- 172.21.15.16:0/2291187267 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d8415ba10 0x7f2d840cf290 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c009d90 tx=0x7f2d7c00b330 comp rx=0 tx=0).stop 2023-10-16T12:30:27.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.738+0000 7f2d88d40700 1 -- 172.21.15.16:0/2291187267 shutdown_connections 2023-10-16T12:30:27.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.738+0000 7f2d88d40700 1 --2- 172.21.15.16:0/2291187267 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d8415ba10 0x7f2d840cf290 unknown :-1 s=CLOSED pgs=1551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:27.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.738+0000 7f2d88d40700 1 --2- 172.21.15.16:0/2291187267 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2d840ce4f0 0x7f2d840d0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:27.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.738+0000 7f2d88d40700 1 -- 172.21.15.16:0/2291187267 >> 172.21.15.16:0/2291187267 conn(0x7f2d840d14b0 msgr2=0x7f2d840d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:27.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.738+0000 7f2d88d40700 1 -- 172.21.15.16:0/2291187267 shutdown_connections 2023-10-16T12:30:27.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.738+0000 7f2d88d40700 1 -- 172.21.15.16:0/2291187267 wait complete. 2023-10-16T12:30:27.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.739+0000 7f2d88d40700 1 Processor -- start 2023-10-16T12:30:27.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.739+0000 7f2d88d40700 1 -- start start 2023-10-16T12:30:27.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.740+0000 7f2d88d40700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2d840ce4f0 0x7f2d84158de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:27.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.740+0000 7f2d88d40700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d8415b630 0x7f2d84157080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:27.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.740+0000 7f2d88d40700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2d8415ba10 0x7f2d840c69c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:27.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.740+0000 7f2d88d40700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d841575c0 con 0x7f2d8415b630 2023-10-16T12:30:27.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.740+0000 7f2d88d40700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d84158950 con 0x7f2d8415ba10 2023-10-16T12:30:27.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.740+0000 7f2d88d40700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d840c7460 con 0x7f2d840ce4f0 2023-10-16T12:30:27.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.740+0000 7f2d81d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d8415b630 0x7f2d84157080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:27.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.740+0000 7f2d81d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d8415b630 0x7f2d84157080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45276/0 (socket says 172.21.15.16:45276) 2023-10-16T12:30:27.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.740+0000 7f2d81d9b700 1 -- 172.21.15.16:0/3391911726 learned_addr learned my addr 172.21.15.16:0/3391911726 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:27.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.740+0000 7f2d82d9d700 1 --2- 172.21.15.16:0/3391911726 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2d8415ba10 0x7f2d840c69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:27.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.740+0000 7f2d8259c700 1 --2- 172.21.15.16:0/3391911726 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2d840ce4f0 0x7f2d84158de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:27.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.740+0000 7f2d81d9b700 1 -- 172.21.15.16:0/3391911726 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2d840ce4f0 msgr2=0x7f2d84158de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:27.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.740+0000 7f2d81d9b700 1 --2- 172.21.15.16:0/3391911726 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2d840ce4f0 0x7f2d84158de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:27.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.740+0000 7f2d81d9b700 1 -- 172.21.15.16:0/3391911726 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2d8415ba10 msgr2=0x7f2d840c69c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:27.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.740+0000 7f2d81d9b700 1 --2- 172.21.15.16:0/3391911726 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2d8415ba10 0x7f2d840c69c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:27.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.740+0000 7f2d81d9b700 1 -- 172.21.15.16:0/3391911726 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d7c009a70 con 0x7f2d8415b630 2023-10-16T12:30:27.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.740+0000 7f2d82d9d700 1 --2- 172.21.15.16:0/3391911726 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2d8415ba10 0x7f2d840c69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:30:27.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.741+0000 7f2d81d9b700 1 --2- 172.21.15.16:0/3391911726 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d8415b630 0x7f2d84157080 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f2d7400c750 tx=0x7f2d7400ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:27.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.741+0000 7f2d72ffd700 1 -- 172.21.15.16:0/3391911726 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d74011800 con 0x7f2d8415b630 2023-10-16T12:30:27.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.741+0000 7f2d72ffd700 1 -- 172.21.15.16:0/3391911726 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d74011960 con 0x7f2d8415b630 2023-10-16T12:30:27.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.741+0000 7f2d88d40700 1 -- 172.21.15.16:0/3391911726 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d84158300 con 0x7f2d8415b630 2023-10-16T12:30:27.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.741+0000 7f2d72ffd700 1 -- 172.21.15.16:0/3391911726 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d74011cd0 con 0x7f2d8415b630 2023-10-16T12:30:27.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.741+0000 7f2d88d40700 1 -- 172.21.15.16:0/3391911726 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d840c7740 con 0x7f2d8415b630 2023-10-16T12:30:27.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.742+0000 7f2d88d40700 1 -- 172.21.15.16:0/3391911726 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d84158ae0 con 0x7f2d8415b630 2023-10-16T12:30:27.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.742+0000 7f2d72ffd700 1 -- 172.21.15.16:0/3391911726 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 74) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2d74021350 con 0x7f2d8415b630 2023-10-16T12:30:27.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.743+0000 7f2d72ffd700 1 --2- 172.21.15.16:0/3391911726 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f2d6406b020 0x7f2d6406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:27.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.743+0000 7f2d72ffd700 1 -- 172.21.15.16:0/3391911726 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 6085+0+0 (secure 0 0 0) 0x7f2d74089890 con 0x7f2d8415b630 2023-10-16T12:30:27.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.743+0000 7f2d8259c700 1 --2- 172.21.15.16:0/3391911726 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f2d6406b020 0x7f2d6406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:27.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.743+0000 7f2d8259c700 1 --2- 172.21.15.16:0/3391911726 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f2d6406b020 0x7f2d6406d4e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2d78007fd0 tx=0x7f2d78007de0 comp rx=0 tx=0).ready entity=mgr.8388 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:27.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.747+0000 7f2d72ffd700 1 -- 172.21.15.16:0/3391911726 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2d74054630 con 0x7f2d8415b630 2023-10-16T12:30:27.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.943+0000 7f2d88d40700 1 -- 172.21.15.16:0/3391911726 --> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2d840b0f10 con 0x7f2d6406b020 2023-10-16T12:30:27.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.951+0000 7f2d72ffd700 1 -- 172.21.15.16:0/3391911726 <== mgr.8388 v2:172.21.15.16:6832/258291157 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289926 (secure 0 0 0) 0x7f2d78001370 con 0x7f2d6406b020 2023-10-16T12:30:27.956 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:30:27.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.955+0000 7f2d88d40700 1 -- 172.21.15.16:0/3391911726 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f2d6406b020 msgr2=0x7f2d6406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:27.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.955+0000 7f2d88d40700 1 --2- 172.21.15.16:0/3391911726 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f2d6406b020 0x7f2d6406d4e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2d78007fd0 tx=0x7f2d78007de0 comp rx=0 tx=0).stop 2023-10-16T12:30:27.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.955+0000 7f2d88d40700 1 -- 172.21.15.16:0/3391911726 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d8415b630 msgr2=0x7f2d84157080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:27.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.955+0000 7f2d88d40700 1 --2- 172.21.15.16:0/3391911726 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d8415b630 0x7f2d84157080 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f2d7400c750 tx=0x7f2d7400ca60 comp rx=0 tx=0).stop 2023-10-16T12:30:27.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.955+0000 7f2d88d40700 1 -- 172.21.15.16:0/3391911726 shutdown_connections 2023-10-16T12:30:27.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.955+0000 7f2d88d40700 1 --2- 172.21.15.16:0/3391911726 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f2d6406b020 0x7f2d6406d4e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:27.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.955+0000 7f2d88d40700 1 --2- 172.21.15.16:0/3391911726 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2d8415ba10 0x7f2d840c69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:27.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.955+0000 7f2d88d40700 1 --2- 172.21.15.16:0/3391911726 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2d840ce4f0 0x7f2d84158de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:27.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.955+0000 7f2d88d40700 1 --2- 172.21.15.16:0/3391911726 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d8415b630 0x7f2d84157080 unknown :-1 s=CLOSED pgs=1552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:27.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.955+0000 7f2d88d40700 1 -- 172.21.15.16:0/3391911726 >> 172.21.15.16:0/3391911726 conn(0x7f2d840d14b0 msgr2=0x7f2d840686c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:27.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.955+0000 7f2d88d40700 1 -- 172.21.15.16:0/3391911726 shutdown_connections 2023-10-16T12:30:27.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:27.955+0000 7f2d88d40700 1 -- 172.21.15.16:0/3391911726 wait complete. 2023-10-16T12:30:27.961 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-10-16T12:30:27.969 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-10-16T12:30:26.915181+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1076,"num_read_kb":2160,"num_write":403,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":265,"ondisk_log_size":265,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":115,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":347672,"kb_used_data":14932,"kb_used_omap":12,"kb_used_meta":332595,"kb_avail":754627048,"statfs":{"total":773094113280,"available":772738097152,"internally_reserved":0,"allocated":15290368,"data_stored":8269448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12740,"internal_metadata":340577852},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.537810"},"pg_stats":[{"pgid":"18.3f","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.427791+0000","last_change":"2023-10-16T12:30:23.421734+0000","last_active":"2023-10-16T12:30:23.427791+0000","last_peered":"2023-10-16T12:30:23.427791+0000","last_clean":"2023-10-16T12:30:23.427791+0000","last_became_active":"2023-10-16T12:30:23.421440+0000","last_became_peered":"2023-10-16T12:30:23.421440+0000","last_unstale":"2023-10-16T12:30:23.427791+0000","last_undegraded":"2023-10-16T12:30:23.427791+0000","last_fullsized":"2023-10-16T12:30:23.427791+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:48:30.457561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3e","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.416122+0000","last_change":"2023-10-16T12:30:23.412272+0000","last_active":"2023-10-16T12:30:23.416122+0000","last_peered":"2023-10-16T12:30:23.416122+0000","last_clean":"2023-10-16T12:30:23.416122+0000","last_became_active":"2023-10-16T12:30:23.411962+0000","last_became_peered":"2023-10-16T12:30:23.411962+0000","last_unstale":"2023-10-16T12:30:23.416122+0000","last_undegraded":"2023-10-16T12:30:23.416122+0000","last_fullsized":"2023-10-16T12:30:23.416122+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:01:51.764162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3d","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.422041+0000","last_change":"2023-10-16T12:30:23.414309+0000","last_active":"2023-10-16T12:30:23.422041+0000","last_peered":"2023-10-16T12:30:23.422041+0000","last_clean":"2023-10-16T12:30:23.422041+0000","last_became_active":"2023-10-16T12:30:23.413931+0000","last_became_peered":"2023-10-16T12:30:23.413931+0000","last_unstale":"2023-10-16T12:30:23.422041+0000","last_undegraded":"2023-10-16T12:30:23.422041+0000","last_fullsized":"2023-10-16T12:30:23.422041+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:57:53.835205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3c","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T12:30:24.398830+0000","last_change":"2023-10-16T12:30:23.422354+0000","last_active":"2023-10-16T12:30:24.398830+0000","last_peered":"2023-10-16T12:30:24.398830+0000","last_clean":"2023-10-16T12:30:24.398830+0000","last_became_active":"2023-10-16T12:30:23.422053+0000","last_became_peered":"2023-10-16T12:30:23.422053+0000","last_unstale":"2023-10-16T12:30:24.398830+0000","last_undegraded":"2023-10-16T12:30:24.398830+0000","last_fullsized":"2023-10-16T12:30:24.398830+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:01:51.764162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3b","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.426678+0000","last_change":"2023-10-16T12:30:23.423801+0000","last_active":"2023-10-16T12:30:23.426678+0000","last_peered":"2023-10-16T12:30:23.426678+0000","last_clean":"2023-10-16T12:30:23.426678+0000","last_became_active":"2023-10-16T12:30:23.423517+0000","last_became_peered":"2023-10-16T12:30:23.423517+0000","last_unstale":"2023-10-16T12:30:23.426678+0000","last_undegraded":"2023-10-16T12:30:23.426678+0000","last_fullsized":"2023-10-16T12:30:23.426678+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:36:28.673423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3a","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T12:30:24.397614+0000","last_change":"2023-10-16T12:30:23.420780+0000","last_active":"2023-10-16T12:30:24.397614+0000","last_peered":"2023-10-16T12:30:24.397614+0000","last_clean":"2023-10-16T12:30:24.397614+0000","last_became_active":"2023-10-16T12:30:23.420343+0000","last_became_peered":"2023-10-16T12:30:23.420343+0000","last_unstale":"2023-10-16T12:30:24.397614+0000","last_undegraded":"2023-10-16T12:30:24.397614+0000","last_fullsized":"2023-10-16T12:30:24.397614+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:39:22.549338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.39","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T12:30:24.397458+0000","last_change":"2023-10-16T12:30:23.409187+0000","last_active":"2023-10-16T12:30:24.397458+0000","last_peered":"2023-10-16T12:30:24.397458+0000","last_clean":"2023-10-16T12:30:24.397458+0000","last_became_active":"2023-10-16T12:30:23.408890+0000","last_became_peered":"2023-10-16T12:30:23.408890+0000","last_unstale":"2023-10-16T12:30:24.397458+0000","last_undegraded":"2023-10-16T12:30:24.397458+0000","last_fullsized":"2023-10-16T12:30:24.397458+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:25:33.134269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.38","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.428621+0000","last_change":"2023-10-16T12:30:23.418765+0000","last_active":"2023-10-16T12:30:23.428621+0000","last_peered":"2023-10-16T12:30:23.428621+0000","last_clean":"2023-10-16T12:30:23.428621+0000","last_became_active":"2023-10-16T12:30:23.418486+0000","last_became_peered":"2023-10-16T12:30:23.418486+0000","last_unstale":"2023-10-16T12:30:23.428621+0000","last_undegraded":"2023-10-16T12:30:23.428621+0000","last_fullsized":"2023-10-16T12:30:23.428621+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:28:10.860320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.37","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T12:30:24.398926+0000","last_change":"2023-10-16T12:30:23.412518+0000","last_active":"2023-10-16T12:30:24.398926+0000","last_peered":"2023-10-16T12:30:24.398926+0000","last_clean":"2023-10-16T12:30:24.398926+0000","last_became_active":"2023-10-16T12:30:23.412187+0000","last_became_peered":"2023-10-16T12:30:23.412187+0000","last_unstale":"2023-10-16T12:30:24.398926+0000","last_undegraded":"2023-10-16T12:30:24.398926+0000","last_fullsized":"2023-10-16T12:30:24.398926+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:19:42.068670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.36","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.413045+0000","last_change":"2023-10-16T12:30:23.406924+0000","last_active":"2023-10-16T12:30:23.413045+0000","last_peered":"2023-10-16T12:30:23.413045+0000","last_clean":"2023-10-16T12:30:23.413045+0000","last_became_active":"2023-10-16T12:30:23.406632+0000","last_became_peered":"2023-10-16T12:30:23.406632+0000","last_unstale":"2023-10-16T12:30:23.413045+0000","last_undegraded":"2023-10-16T12:30:23.413045+0000","last_fullsized":"2023-10-16T12:30:23.413045+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:03:32.170008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.35","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.432907+0000","last_change":"2023-10-16T12:30:23.430887+0000","last_active":"2023-10-16T12:30:23.432907+0000","last_peered":"2023-10-16T12:30:23.432907+0000","last_clean":"2023-10-16T12:30:23.432907+0000","last_became_active":"2023-10-16T12:30:23.430578+0000","last_became_peered":"2023-10-16T12:30:23.430578+0000","last_unstale":"2023-10-16T12:30:23.432907+0000","last_undegraded":"2023-10-16T12:30:23.432907+0000","last_fullsized":"2023-10-16T12:30:23.432907+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:31:26.337544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.34","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T12:30:24.398166+0000","last_change":"2023-10-16T12:30:23.426707+0000","last_active":"2023-10-16T12:30:24.398166+0000","last_peered":"2023-10-16T12:30:24.398166+0000","last_clean":"2023-10-16T12:30:24.398166+0000","last_became_active":"2023-10-16T12:30:23.426419+0000","last_became_peered":"2023-10-16T12:30:23.426419+0000","last_unstale":"2023-10-16T12:30:24.398166+0000","last_undegraded":"2023-10-16T12:30:24.398166+0000","last_fullsized":"2023-10-16T12:30:24.398166+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:40:03.973069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.33","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.416433+0000","last_change":"2023-10-16T12:30:23.414195+0000","last_active":"2023-10-16T12:30:23.416433+0000","last_peered":"2023-10-16T12:30:23.416433+0000","last_clean":"2023-10-16T12:30:23.416433+0000","last_became_active":"2023-10-16T12:30:23.413919+0000","last_became_peered":"2023-10-16T12:30:23.413919+0000","last_unstale":"2023-10-16T12:30:23.416433+0000","last_undegraded":"2023-10-16T12:30:23.416433+0000","last_fullsized":"2023-10-16T12:30:23.416433+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:55:23.016394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.32","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.422173+0000","last_change":"2023-10-16T12:30:23.419708+0000","last_active":"2023-10-16T12:30:23.422173+0000","last_peered":"2023-10-16T12:30:23.422173+0000","last_clean":"2023-10-16T12:30:23.422173+0000","last_became_active":"2023-10-16T12:30:23.419167+0000","last_became_peered":"2023-10-16T12:30:23.419167+0000","last_unstale":"2023-10-16T12:30:23.422173+0000","last_undegraded":"2023-10-16T12:30:23.422173+0000","last_fullsized":"2023-10-16T12:30:23.422173+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:18:36.577516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.31","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.427006+0000","last_change":"2023-10-16T12:30:23.426251+0000","last_active":"2023-10-16T12:30:23.427006+0000","last_peered":"2023-10-16T12:30:23.427006+0000","last_clean":"2023-10-16T12:30:23.427006+0000","last_became_active":"2023-10-16T12:30:23.425968+0000","last_became_peered":"2023-10-16T12:30:23.425968+0000","last_unstale":"2023-10-16T12:30:23.427006+0000","last_undegraded":"2023-10-16T12:30:23.427006+0000","last_fullsized":"2023-10-16T12:30:23.427006+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:25:39.680586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.30","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T12:30:24.397443+0000","last_change":"2023-10-16T12:30:23.411584+0000","last_active":"2023-10-16T12:30:24.397443+0000","last_peered":"2023-10-16T12:30:24.397443+0000","last_clean":"2023-10-16T12:30:24.397443+0000","last_became_active":"2023-10-16T12:30:23.410532+0000","last_became_peered":"2023-10-16T12:30:23.410532+0000","last_unstale":"2023-10-16T12:30:24.397443+0000","last_undegraded":"2023-10-16T12:30:24.397443+0000","last_fullsized":"2023-10-16T12:30:24.397443+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:30:01.541809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.2f","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.418154+0000","last_change":"2023-10-16T12:30:23.416877+0000","last_active":"2023-10-16T12:30:23.418154+0000","last_peered":"2023-10-16T12:30:23.418154+0000","last_clean":"2023-10-16T12:30:23.418154+0000","last_became_active":"2023-10-16T12:30:23.416298+0000","last_became_peered":"2023-10-16T12:30:23.416298+0000","last_unstale":"2023-10-16T12:30:23.418154+0000","last_undegraded":"2023-10-16T12:30:23.418154+0000","last_fullsized":"2023-10-16T12:30:23.418154+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:21:12.661644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.2e","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.416147+0000","last_change":"2023-10-16T12:30:23.412392+0000","last_active":"2023-10-16T12:30:23.416147+0000","last_peered":"2023-10-16T12:30:23.416147+0000","last_clean":"2023-10-16T12:30:23.416147+0000","last_became_active":"2023-10-16T12:30:23.411945+0000","last_became_peered":"2023-10-16T12:30:23.411945+0000","last_unstale":"2023-10-16T12:30:23.416147+0000","last_undegraded":"2023-10-16T12:30:23.416147+0000","last_fullsized":"2023-10-16T12:30:23.416147+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:13:43.938813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.2d","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.408745+0000","last_change":"2023-10-16T12:30:23.406925+0000","last_active":"2023-10-16T12:30:23.408745+0000","last_peered":"2023-10-16T12:30:23.408745+0000","last_clean":"2023-10-16T12:30:23.408745+0000","last_became_active":"2023-10-16T12:30:23.406496+0000","last_became_peered":"2023-10-16T12:30:23.406496+0000","last_unstale":"2023-10-16T12:30:23.408745+0000","last_undegraded":"2023-10-16T12:30:23.408745+0000","last_fullsized":"2023-10-16T12:30:23.408745+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:26:39.397930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.2c","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.414239+0000","last_change":"2023-10-16T12:30:23.411765+0000","last_active":"2023-10-16T12:30:23.414239+0000","last_peered":"2023-10-16T12:30:23.414239+0000","last_clean":"2023-10-16T12:30:23.414239+0000","last_became_active":"2023-10-16T12:30:23.411313+0000","last_became_peered":"2023-10-16T12:30:23.411313+0000","last_unstale":"2023-10-16T12:30:23.414239+0000","last_undegraded":"2023-10-16T12:30:23.414239+0000","last_fullsized":"2023-10-16T12:30:23.414239+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:49:51.894838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.2b","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T12:30:24.398527+0000","last_change":"2023-10-16T12:30:23.411265+0000","last_active":"2023-10-16T12:30:24.398527+0000","last_peered":"2023-10-16T12:30:24.398527+0000","last_clean":"2023-10-16T12:30:24.398527+0000","last_became_active":"2023-10-16T12:30:23.410849+0000","last_became_peered":"2023-10-16T12:30:23.410849+0000","last_unstale":"2023-10-16T12:30:24.398527+0000","last_undegraded":"2023-10-16T12:30:24.398527+0000","last_fullsized":"2023-10-16T12:30:24.398527+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:14:15.737566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.2a","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.430208+0000","last_change":"2023-10-16T12:30:23.426390+0000","last_active":"2023-10-16T12:30:23.430208+0000","last_peered":"2023-10-16T12:30:23.430208+0000","last_clean":"2023-10-16T12:30:23.430208+0000","last_became_active":"2023-10-16T12:30:23.422353+0000","last_became_peered":"2023-10-16T12:30:23.422353+0000","last_unstale":"2023-10-16T12:30:23.430208+0000","last_undegraded":"2023-10-16T12:30:23.430208+0000","last_fullsized":"2023-10-16T12:30:23.430208+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:16:06.545565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.29","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T12:30:24.398493+0000","last_change":"2023-10-16T12:30:23.412572+0000","last_active":"2023-10-16T12:30:24.398493+0000","last_peered":"2023-10-16T12:30:24.398493+0000","last_clean":"2023-10-16T12:30:24.398493+0000","last_became_active":"2023-10-16T12:30:23.412241+0000","last_became_peered":"2023-10-16T12:30:23.412241+0000","last_unstale":"2023-10-16T12:30:24.398493+0000","last_undegraded":"2023-10-16T12:30:24.398493+0000","last_fullsized":"2023-10-16T12:30:24.398493+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:13:43.938813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.28","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.417023+0000","last_change":"2023-10-16T12:30:23.415883+0000","last_active":"2023-10-16T12:30:23.417023+0000","last_peered":"2023-10-16T12:30:23.417023+0000","last_clean":"2023-10-16T12:30:23.417023+0000","last_became_active":"2023-10-16T12:30:23.415439+0000","last_became_peered":"2023-10-16T12:30:23.415439+0000","last_unstale":"2023-10-16T12:30:23.417023+0000","last_undegraded":"2023-10-16T12:30:23.417023+0000","last_fullsized":"2023-10-16T12:30:23.417023+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:01:59.976335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.27","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.430312+0000","last_change":"2023-10-16T12:30:23.428654+0000","last_active":"2023-10-16T12:30:23.430312+0000","last_peered":"2023-10-16T12:30:23.430312+0000","last_clean":"2023-10-16T12:30:23.430312+0000","last_became_active":"2023-10-16T12:30:23.428335+0000","last_became_peered":"2023-10-16T12:30:23.428335+0000","last_unstale":"2023-10-16T12:30:23.430312+0000","last_undegraded":"2023-10-16T12:30:23.430312+0000","last_fullsized":"2023-10-16T12:30:23.430312+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:03:03.394597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.26","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.409086+0000","last_change":"2023-10-16T12:30:23.407300+0000","last_active":"2023-10-16T12:30:23.409086+0000","last_peered":"2023-10-16T12:30:23.409086+0000","last_clean":"2023-10-16T12:30:23.409086+0000","last_became_active":"2023-10-16T12:30:23.406831+0000","last_became_peered":"2023-10-16T12:30:23.406831+0000","last_unstale":"2023-10-16T12:30:23.409086+0000","last_undegraded":"2023-10-16T12:30:23.409086+0000","last_fullsized":"2023-10-16T12:30:23.409086+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:45:11.054235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.25","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.408726+0000","last_change":"2023-10-16T12:30:23.406657+0000","last_active":"2023-10-16T12:30:23.408726+0000","last_peered":"2023-10-16T12:30:23.408726+0000","last_clean":"2023-10-16T12:30:23.408726+0000","last_became_active":"2023-10-16T12:30:23.406075+0000","last_became_peered":"2023-10-16T12:30:23.406075+0000","last_unstale":"2023-10-16T12:30:23.408726+0000","last_undegraded":"2023-10-16T12:30:23.408726+0000","last_fullsized":"2023-10-16T12:30:23.408726+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:25:58.262389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.24","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T12:30:24.398011+0000","last_change":"2023-10-16T12:30:23.418156+0000","last_active":"2023-10-16T12:30:24.398011+0000","last_peered":"2023-10-16T12:30:24.398011+0000","last_clean":"2023-10-16T12:30:24.398011+0000","last_became_active":"2023-10-16T12:30:23.417863+0000","last_became_peered":"2023-10-16T12:30:23.417863+0000","last_unstale":"2023-10-16T12:30:24.398011+0000","last_undegraded":"2023-10-16T12:30:24.398011+0000","last_fullsized":"2023-10-16T12:30:24.398011+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:00:00.022734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.23","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.415416+0000","last_change":"2023-10-16T12:30:23.412045+0000","last_active":"2023-10-16T12:30:23.415416+0000","last_peered":"2023-10-16T12:30:23.415416+0000","last_clean":"2023-10-16T12:30:23.415416+0000","last_became_active":"2023-10-16T12:30:23.411753+0000","last_became_peered":"2023-10-16T12:30:23.411753+0000","last_unstale":"2023-10-16T12:30:23.415416+0000","last_undegraded":"2023-10-16T12:30:23.415416+0000","last_fullsized":"2023-10-16T12:30:23.415416+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:38:36.180207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.22","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.429522+0000","last_change":"2023-10-16T12:30:23.427792+0000","last_active":"2023-10-16T12:30:23.429522+0000","last_peered":"2023-10-16T12:30:23.429522+0000","last_clean":"2023-10-16T12:30:23.429522+0000","last_became_active":"2023-10-16T12:30:23.427492+0000","last_became_peered":"2023-10-16T12:30:23.427492+0000","last_unstale":"2023-10-16T12:30:23.429522+0000","last_undegraded":"2023-10-16T12:30:23.429522+0000","last_fullsized":"2023-10-16T12:30:23.429522+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:50:47.311871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.21","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.415396+0000","last_change":"2023-10-16T12:30:23.413300+0000","last_active":"2023-10-16T12:30:23.415396+0000","last_peered":"2023-10-16T12:30:23.415396+0000","last_clean":"2023-10-16T12:30:23.415396+0000","last_became_active":"2023-10-16T12:30:23.412975+0000","last_became_peered":"2023-10-16T12:30:23.412975+0000","last_unstale":"2023-10-16T12:30:23.415396+0000","last_undegraded":"2023-10-16T12:30:23.415396+0000","last_fullsized":"2023-10-16T12:30:23.415396+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:19:07.800422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.20","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.416222+0000","last_change":"2023-10-16T12:30:23.415666+0000","last_active":"2023-10-16T12:30:23.416222+0000","last_peered":"2023-10-16T12:30:23.416222+0000","last_clean":"2023-10-16T12:30:23.416222+0000","last_became_active":"2023-10-16T12:30:23.415128+0000","last_became_peered":"2023-10-16T12:30:23.415128+0000","last_unstale":"2023-10-16T12:30:23.416222+0000","last_undegraded":"2023-10-16T12:30:23.416222+0000","last_fullsized":"2023-10-16T12:30:23.416222+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:39:42.101547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.416875+0000","last_change":"2023-10-16T12:30:23.410285+0000","last_active":"2023-10-16T12:30:23.416875+0000","last_peered":"2023-10-16T12:30:23.416875+0000","last_clean":"2023-10-16T12:30:23.416875+0000","last_became_active":"2023-10-16T12:30:23.409872+0000","last_became_peered":"2023-10-16T12:30:23.409872+0000","last_unstale":"2023-10-16T12:30:23.416875+0000","last_undegraded":"2023-10-16T12:30:23.416875+0000","last_fullsized":"2023-10-16T12:30:23.416875+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:26:39.397930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1f","version":"141'4","reported_seq":15,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.427207+0000","last_change":"2023-10-16T12:30:22.394531+0000","last_active":"2023-10-16T12:30:23.427207+0000","last_peered":"2023-10-16T12:30:23.427207+0000","last_clean":"2023-10-16T12:30:23.427207+0000","last_became_active":"2023-10-16T12:30:22.394243+0000","last_became_peered":"2023-10-16T12:30:22.394243+0000","last_unstale":"2023-10-16T12:30:23.427207+0000","last_undegraded":"2023-10-16T12:30:23.427207+0000","last_fullsized":"2023-10-16T12:30:23.427207+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_clean_scrub_stamp":"2023-10-16T12:30:21.373132+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:11:58.279552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1d","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T12:30:24.397696+0000","last_change":"2023-10-16T12:30:23.414959+0000","last_active":"2023-10-16T12:30:24.397696+0000","last_peered":"2023-10-16T12:30:24.397696+0000","last_clean":"2023-10-16T12:30:24.397696+0000","last_became_active":"2023-10-16T12:30:23.414690+0000","last_became_peered":"2023-10-16T12:30:23.414690+0000","last_unstale":"2023-10-16T12:30:24.397696+0000","last_undegraded":"2023-10-16T12:30:24.397696+0000","last_fullsized":"2023-10-16T12:30:24.397696+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:47:31.216350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1e","version":"141'1","reported_seq":12,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.432623+0000","last_change":"2023-10-16T12:30:22.399129+0000","last_active":"2023-10-16T12:30:23.432623+0000","last_peered":"2023-10-16T12:30:23.432623+0000","last_clean":"2023-10-16T12:30:23.432623+0000","last_became_active":"2023-10-16T12:30:22.398739+0000","last_became_peered":"2023-10-16T12:30:22.398739+0000","last_unstale":"2023-10-16T12:30:23.432623+0000","last_undegraded":"2023-10-16T12:30:23.432623+0000","last_fullsized":"2023-10-16T12:30:23.432623+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_clean_scrub_stamp":"2023-10-16T12:30:21.373132+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:25:38.676933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.415708+0000","last_change":"2023-10-16T12:30:23.414335+0000","last_active":"2023-10-16T12:30:23.415708+0000","last_peered":"2023-10-16T12:30:23.415708+0000","last_clean":"2023-10-16T12:30:23.415708+0000","last_became_active":"2023-10-16T12:30:23.414023+0000","last_became_peered":"2023-10-16T12:30:23.414023+0000","last_unstale":"2023-10-16T12:30:23.415708+0000","last_undegraded":"2023-10-16T12:30:23.415708+0000","last_fullsized":"2023-10-16T12:30:23.415708+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:01:41.278198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"141'2","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T12:30:24.398638+0000","last_change":"2023-10-16T12:30:22.401175+0000","last_active":"2023-10-16T12:30:24.398638+0000","last_peered":"2023-10-16T12:30:24.398638+0000","last_clean":"2023-10-16T12:30:24.398638+0000","last_became_active":"2023-10-16T12:30:22.400471+0000","last_became_peered":"2023-10-16T12:30:22.400471+0000","last_unstale":"2023-10-16T12:30:24.398638+0000","last_undegraded":"2023-10-16T12:30:24.398638+0000","last_fullsized":"2023-10-16T12:30:24.398638+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_clean_scrub_stamp":"2023-10-16T12:30:21.373132+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:01:58.972682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1f","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T12:30:24.398946+0000","last_change":"2023-10-16T12:30:23.406133+0000","last_active":"2023-10-16T12:30:24.398946+0000","last_peered":"2023-10-16T12:30:24.398946+0000","last_clean":"2023-10-16T12:30:24.398946+0000","last_became_active":"2023-10-16T12:30:23.405819+0000","last_became_peered":"2023-10-16T12:30:23.405819+0000","last_unstale":"2023-10-16T12:30:24.398946+0000","last_undegraded":"2023-10-16T12:30:24.398946+0000","last_fullsized":"2023-10-16T12:30:24.398946+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:52:29.142123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"141'2","reported_seq":13,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.431875+0000","last_change":"2023-10-16T12:30:22.400661+0000","last_active":"2023-10-16T12:30:23.431875+0000","last_peered":"2023-10-16T12:30:23.431875+0000","last_clean":"2023-10-16T12:30:23.431875+0000","last_became_active":"2023-10-16T12:30:22.400329+0000","last_became_peered":"2023-10-16T12:30:22.400329+0000","last_unstale":"2023-10-16T12:30:23.431875+0000","last_undegraded":"2023-10-16T12:30:23.431875+0000","last_fullsized":"2023-10-16T12:30:23.431875+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_clean_scrub_stamp":"2023-10-16T12:30:21.373132+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:28:15.424497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.416641+0000","last_change":"2023-10-16T12:30:23.413223+0000","last_active":"2023-10-16T12:30:23.416641+0000","last_peered":"2023-10-16T12:30:23.416641+0000","last_clean":"2023-10-16T12:30:23.416641+0000","last_became_active":"2023-10-16T12:30:23.412934+0000","last_became_peered":"2023-10-16T12:30:23.412934+0000","last_unstale":"2023-10-16T12:30:23.416641+0000","last_undegraded":"2023-10-16T12:30:23.416641+0000","last_fullsized":"2023-10-16T12:30:23.416641+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:48:36.975059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1b","version":"141'1","reported_seq":12,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.432734+0000","last_change":"2023-10-16T12:30:22.404893+0000","last_active":"2023-10-16T12:30:23.432734+0000","last_peered":"2023-10-16T12:30:23.432734+0000","last_clean":"2023-10-16T12:30:23.432734+0000","last_became_active":"2023-10-16T12:30:22.404656+0000","last_became_peered":"2023-10-16T12:30:22.404656+0000","last_unstale":"2023-10-16T12:30:23.432734+0000","last_undegraded":"2023-10-16T12:30:23.432734+0000","last_fullsized":"2023-10-16T12:30:23.432734+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_clean_scrub_stamp":"2023-10-16T12:30:21.373132+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:41:23.912012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.426584+0000","last_change":"2023-10-16T12:30:23.425076+0000","last_active":"2023-10-16T12:30:23.426584+0000","last_peered":"2023-10-16T12:30:23.426584+0000","last_clean":"2023-10-16T12:30:23.426584+0000","last_became_active":"2023-10-16T12:30:23.424782+0000","last_became_peered":"2023-10-16T12:30:23.424782+0000","last_unstale":"2023-10-16T12:30:23.426584+0000","last_undegraded":"2023-10-16T12:30:23.426584+0000","last_fullsized":"2023-10-16T12:30:23.426584+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:04:26.539127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"141'1","reported_seq":13,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T12:30:24.397671+0000","last_change":"2023-10-16T12:30:22.403094+0000","last_active":"2023-10-16T12:30:24.397671+0000","last_peered":"2023-10-16T12:30:24.397671+0000","last_clean":"2023-10-16T12:30:24.397671+0000","last_became_active":"2023-10-16T12:30:22.402806+0000","last_became_peered":"2023-10-16T12:30:22.402806+0000","last_unstale":"2023-10-16T12:30:24.397671+0000","last_undegraded":"2023-10-16T12:30:24.397671+0000","last_fullsized":"2023-10-16T12:30:24.397671+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_clean_scrub_stamp":"2023-10-16T12:30:21.373132+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:16:13.015811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.1a","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.422157+0000","last_change":"2023-10-16T12:30:23.419589+0000","last_active":"2023-10-16T12:30:23.422157+0000","last_peered":"2023-10-16T12:30:23.422157+0000","last_clean":"2023-10-16T12:30:23.422157+0000","last_became_active":"2023-10-16T12:30:23.418858+0000","last_became_peered":"2023-10-16T12:30:23.418858+0000","last_unstale":"2023-10-16T12:30:23.422157+0000","last_undegraded":"2023-10-16T12:30:23.422157+0000","last_fullsized":"2023-10-16T12:30:23.422157+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:01:59.976335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":11,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.402414+0000","last_change":"2023-10-16T12:30:22.392831+0000","last_active":"2023-10-16T12:30:23.402414+0000","last_peered":"2023-10-16T12:30:23.402414+0000","last_clean":"2023-10-16T12:30:23.402414+0000","last_became_active":"2023-10-16T12:30:22.392535+0000","last_became_peered":"2023-10-16T12:30:22.392535+0000","last_unstale":"2023-10-16T12:30:23.402414+0000","last_undegraded":"2023-10-16T12:30:23.402414+0000","last_fullsized":"2023-10-16T12:30:23.402414+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_clean_scrub_stamp":"2023-10-16T12:30:21.373132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:50:46.308219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1b","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.416415+0000","last_change":"2023-10-16T12:30:23.412874+0000","last_active":"2023-10-16T12:30:23.416415+0000","last_peered":"2023-10-16T12:30:23.416415+0000","last_clean":"2023-10-16T12:30:23.416415+0000","last_became_active":"2023-10-16T12:30:23.412574+0000","last_became_peered":"2023-10-16T12:30:23.412574+0000","last_unstale":"2023-10-16T12:30:23.416415+0000","last_undegraded":"2023-10-16T12:30:23.416415+0000","last_fullsized":"2023-10-16T12:30:23.416415+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:25:58.262389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T12:30:24.397644+0000","last_change":"2023-10-16T12:30:22.402099+0000","last_active":"2023-10-16T12:30:24.397644+0000","last_peered":"2023-10-16T12:30:24.397644+0000","last_clean":"2023-10-16T12:30:24.397644+0000","last_became_active":"2023-10-16T12:30:22.401822+0000","last_became_peered":"2023-10-16T12:30:22.401822+0000","last_unstale":"2023-10-16T12:30:24.397644+0000","last_undegraded":"2023-10-16T12:30:24.397644+0000","last_fullsized":"2023-10-16T12:30:24.397644+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_clean_scrub_stamp":"2023-10-16T12:30:21.373132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:24:49.411085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.410157+0000","last_change":"2023-10-16T12:30:23.407550+0000","last_active":"2023-10-16T12:30:23.410157+0000","last_peered":"2023-10-16T12:30:23.410157+0000","last_clean":"2023-10-16T12:30:23.410157+0000","last_became_active":"2023-10-16T12:30:23.407281+0000","last_became_peered":"2023-10-16T12:30:23.407281+0000","last_unstale":"2023-10-16T12:30:23.410157+0000","last_undegraded":"2023-10-16T12:30:23.410157+0000","last_fullsized":"2023-10-16T12:30:23.410157+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:03:32.170008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":11,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.401651+0000","last_change":"2023-10-16T12:30:22.404324+0000","last_active":"2023-10-16T12:30:23.401651+0000","last_peered":"2023-10-16T12:30:23.401651+0000","last_clean":"2023-10-16T12:30:23.401651+0000","last_became_active":"2023-10-16T12:30:22.403985+0000","last_became_peered":"2023-10-16T12:30:22.403985+0000","last_unstale":"2023-10-16T12:30:23.401651+0000","last_undegraded":"2023-10-16T12:30:23.401651+0000","last_fullsized":"2023-10-16T12:30:23.401651+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_clean_scrub_stamp":"2023-10-16T12:30:21.373132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:41:13.487445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.429669+0000","last_change":"2023-10-16T12:30:23.422644+0000","last_active":"2023-10-16T12:30:23.429669+0000","last_peered":"2023-10-16T12:30:23.429669+0000","last_clean":"2023-10-16T12:30:23.429669+0000","last_became_active":"2023-10-16T12:30:23.417231+0000","last_became_peered":"2023-10-16T12:30:23.417231+0000","last_unstale":"2023-10-16T12:30:23.429669+0000","last_undegraded":"2023-10-16T12:30:23.429669+0000","last_fullsized":"2023-10-16T12:30:23.429669+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:53:19.541680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":11,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.401306+0000","last_change":"2023-10-16T12:30:22.390167+0000","last_active":"2023-10-16T12:30:23.401306+0000","last_peered":"2023-10-16T12:30:23.401306+0000","last_clean":"2023-10-16T12:30:23.401306+0000","last_became_active":"2023-10-16T12:30:22.389859+0000","last_became_peered":"2023-10-16T12:30:22.389859+0000","last_unstale":"2023-10-16T12:30:23.401306+0000","last_undegraded":"2023-10-16T12:30:23.401306+0000","last_fullsized":"2023-10-16T12:30:23.401306+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_clean_scrub_stamp":"2023-10-16T12:30:21.373132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:47:42.020252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.422052+0000","last_change":"2023-10-16T12:30:23.417841+0000","last_active":"2023-10-16T12:30:23.422052+0000","last_peered":"2023-10-16T12:30:23.422052+0000","last_clean":"2023-10-16T12:30:23.422052+0000","last_became_active":"2023-10-16T12:30:23.417561+0000","last_became_peered":"2023-10-16T12:30:23.417561+0000","last_unstale":"2023-10-16T12:30:23.422052+0000","last_undegraded":"2023-10-16T12:30:23.422052+0000","last_fullsized":"2023-10-16T12:30:23.422052+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:26:57.487238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.15","version":"141'3","reported_seq":14,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.429932+0000","last_change":"2023-10-16T12:30:22.389296+0000","last_active":"2023-10-16T12:30:23.429932+0000","last_peered":"2023-10-16T12:30:23.429932+0000","last_clean":"2023-10-16T12:30:23.429932+0000","last_became_active":"2023-10-16T12:30:22.389000+0000","last_became_peered":"2023-10-16T12:30:22.389000+0000","last_unstale":"2023-10-16T12:30:23.429932+0000","last_undegraded":"2023-10-16T12:30:23.429932+0000","last_fullsized":"2023-10-16T12:30:23.429932+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_clean_scrub_stamp":"2023-10-16T12:30:21.373132+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:28:09.856667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.415758+0000","last_change":"2023-10-16T12:30:23.414377+0000","last_active":"2023-10-16T12:30:23.415758+0000","last_peered":"2023-10-16T12:30:23.415758+0000","last_clean":"2023-10-16T12:30:23.415758+0000","last_became_active":"2023-10-16T12:30:23.413644+0000","last_became_peered":"2023-10-16T12:30:23.413644+0000","last_unstale":"2023-10-16T12:30:23.415758+0000","last_undegraded":"2023-10-16T12:30:23.415758+0000","last_fullsized":"2023-10-16T12:30:23.415758+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:40:20.116892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.14","version":"141'6","reported_seq":17,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.434528+0000","last_change":"2023-10-16T12:30:22.399154+0000","last_active":"2023-10-16T12:30:23.434528+0000","last_peered":"2023-10-16T12:30:23.434528+0000","last_clean":"2023-10-16T12:30:23.434528+0000","last_became_active":"2023-10-16T12:30:22.396265+0000","last_became_peered":"2023-10-16T12:30:22.396265+0000","last_unstale":"2023-10-16T12:30:23.434528+0000","last_undegraded":"2023-10-16T12:30:23.434528+0000","last_fullsized":"2023-10-16T12:30:23.434528+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_clean_scrub_stamp":"2023-10-16T12:30:21.373132+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:50:00.915742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.417721+0000","last_change":"2023-10-16T12:30:23.415605+0000","last_active":"2023-10-16T12:30:23.417721+0000","last_peered":"2023-10-16T12:30:23.417721+0000","last_clean":"2023-10-16T12:30:23.417721+0000","last_became_active":"2023-10-16T12:30:23.414998+0000","last_became_peered":"2023-10-16T12:30:23.414998+0000","last_unstale":"2023-10-16T12:30:23.417721+0000","last_undegraded":"2023-10-16T12:30:23.417721+0000","last_fullsized":"2023-10-16T12:30:23.417721+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:48:36.975059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.13","version":"141'2","reported_seq":13,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.431220+0000","last_change":"2023-10-16T12:30:22.395871+0000","last_active":"2023-10-16T12:30:23.431220+0000","last_peered":"2023-10-16T12:30:23.431220+0000","last_clean":"2023-10-16T12:30:23.431220+0000","last_became_active":"2023-10-16T12:30:22.395554+0000","last_became_peered":"2023-10-16T12:30:22.395554+0000","last_unstale":"2023-10-16T12:30:23.431220+0000","last_undegraded":"2023-10-16T12:30:23.431220+0000","last_fullsized":"2023-10-16T12:30:23.431220+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_clean_scrub_stamp":"2023-10-16T12:30:21.373132+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:09:46.026182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.409767+0000","last_change":"2023-10-16T12:30:23.401955+0000","last_active":"2023-10-16T12:30:23.409767+0000","last_peered":"2023-10-16T12:30:23.409767+0000","last_clean":"2023-10-16T12:30:23.409767+0000","last_became_active":"2023-10-16T12:30:23.400556+0000","last_became_peered":"2023-10-16T12:30:23.400556+0000","last_unstale":"2023-10-16T12:30:23.409767+0000","last_undegraded":"2023-10-16T12:30:23.409767+0000","last_fullsized":"2023-10-16T12:30:23.409767+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:11:59.283205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":11,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.408068+0000","last_change":"2023-10-16T12:30:22.403014+0000","last_active":"2023-10-16T12:30:23.408068+0000","last_peered":"2023-10-16T12:30:23.408068+0000","last_clean":"2023-10-16T12:30:23.408068+0000","last_became_active":"2023-10-16T12:30:22.402718+0000","last_became_peered":"2023-10-16T12:30:22.402718+0000","last_unstale":"2023-10-16T12:30:23.408068+0000","last_undegraded":"2023-10-16T12:30:23.408068+0000","last_fullsized":"2023-10-16T12:30:23.408068+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_clean_scrub_stamp":"2023-10-16T12:30:21.373132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:19:41.065017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.432281+0000","last_change":"2023-10-16T12:30:23.427340+0000","last_active":"2023-10-16T12:30:23.432281+0000","last_peered":"2023-10-16T12:30:23.432281+0000","last_clean":"2023-10-16T12:30:23.432281+0000","last_became_active":"2023-10-16T12:30:23.427069+0000","last_became_peered":"2023-10-16T12:30:23.427069+0000","last_unstale":"2023-10-16T12:30:23.432281+0000","last_undegraded":"2023-10-16T12:30:23.432281+0000","last_fullsized":"2023-10-16T12:30:23.432281+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:54:51.225200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":11,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.407329+0000","last_change":"2023-10-16T12:30:22.396094+0000","last_active":"2023-10-16T12:30:23.407329+0000","last_peered":"2023-10-16T12:30:23.407329+0000","last_clean":"2023-10-16T12:30:23.407329+0000","last_became_active":"2023-10-16T12:30:22.395797+0000","last_became_peered":"2023-10-16T12:30:22.395797+0000","last_unstale":"2023-10-16T12:30:23.407329+0000","last_undegraded":"2023-10-16T12:30:23.407329+0000","last_fullsized":"2023-10-16T12:30:23.407329+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_clean_scrub_stamp":"2023-10-16T12:30:21.373132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:14:14.733913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.415342+0000","last_change":"2023-10-16T12:30:23.413330+0000","last_active":"2023-10-16T12:30:23.415342+0000","last_peered":"2023-10-16T12:30:23.415342+0000","last_clean":"2023-10-16T12:30:23.415342+0000","last_became_active":"2023-10-16T12:30:23.413021+0000","last_became_peered":"2023-10-16T12:30:23.413021+0000","last_unstale":"2023-10-16T12:30:23.415342+0000","last_undegraded":"2023-10-16T12:30:23.415342+0000","last_fullsized":"2023-10-16T12:30:23.415342+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:31:54.709924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.f","version":"141'1","reported_seq":12,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.430568+0000","last_change":"2023-10-16T12:30:22.405161+0000","last_active":"2023-10-16T12:30:23.430568+0000","last_peered":"2023-10-16T12:30:23.430568+0000","last_clean":"2023-10-16T12:30:23.430568+0000","last_became_active":"2023-10-16T12:30:22.404864+0000","last_became_peered":"2023-10-16T12:30:22.404864+0000","last_unstale":"2023-10-16T12:30:23.430568+0000","last_undegraded":"2023-10-16T12:30:23.430568+0000","last_fullsized":"2023-10-16T12:30:23.430568+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_clean_scrub_stamp":"2023-10-16T12:30:21.373132+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:47:15.066363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.426240+0000","last_change":"2023-10-16T12:30:23.421801+0000","last_active":"2023-10-16T12:30:23.426240+0000","last_peered":"2023-10-16T12:30:23.426240+0000","last_clean":"2023-10-16T12:30:23.426240+0000","last_became_active":"2023-10-16T12:30:23.421510+0000","last_became_peered":"2023-10-16T12:30:23.421510+0000","last_unstale":"2023-10-16T12:30:23.426240+0000","last_undegraded":"2023-10-16T12:30:23.426240+0000","last_fullsized":"2023-10-16T12:30:23.426240+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:16:40.062055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.e","version":"141'2","reported_seq":13,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.432158+0000","last_change":"2023-10-16T12:30:22.406039+0000","last_active":"2023-10-16T12:30:23.432158+0000","last_peered":"2023-10-16T12:30:23.432158+0000","last_clean":"2023-10-16T12:30:23.432158+0000","last_became_active":"2023-10-16T12:30:22.405639+0000","last_became_peered":"2023-10-16T12:30:22.405639+0000","last_unstale":"2023-10-16T12:30:23.432158+0000","last_undegraded":"2023-10-16T12:30:23.432158+0000","last_fullsized":"2023-10-16T12:30:23.432158+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_clean_scrub_stamp":"2023-10-16T12:30:21.373132+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:13:23.770593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T12:30:24.398134+0000","last_change":"2023-10-16T12:30:23.406908+0000","last_active":"2023-10-16T12:30:24.398134+0000","last_peered":"2023-10-16T12:30:24.398134+0000","last_clean":"2023-10-16T12:30:24.398134+0000","last_became_active":"2023-10-16T12:30:23.406284+0000","last_became_peered":"2023-10-16T12:30:23.406284+0000","last_unstale":"2023-10-16T12:30:24.398134+0000","last_undegraded":"2023-10-16T12:30:24.398134+0000","last_fullsized":"2023-10-16T12:30:24.398134+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:19:31.176214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.d","version":"141'1","reported_seq":13,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T12:30:24.398640+0000","last_change":"2023-10-16T12:30:22.398802+0000","last_active":"2023-10-16T12:30:24.398640+0000","last_peered":"2023-10-16T12:30:24.398640+0000","last_clean":"2023-10-16T12:30:24.398640+0000","last_became_active":"2023-10-16T12:30:22.398415+0000","last_became_peered":"2023-10-16T12:30:22.398415+0000","last_unstale":"2023-10-16T12:30:24.398640+0000","last_undegraded":"2023-10-16T12:30:24.398640+0000","last_fullsized":"2023-10-16T12:30:24.398640+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_clean_scrub_stamp":"2023-10-16T12:30:21.373132+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:09:46.026182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.415779+0000","last_change":"2023-10-16T12:30:23.414503+0000","last_active":"2023-10-16T12:30:23.415779+0000","last_peered":"2023-10-16T12:30:23.415779+0000","last_clean":"2023-10-16T12:30:23.415779+0000","last_became_active":"2023-10-16T12:30:23.413975+0000","last_became_peered":"2023-10-16T12:30:23.413975+0000","last_unstale":"2023-10-16T12:30:23.415779+0000","last_undegraded":"2023-10-16T12:30:23.415779+0000","last_fullsized":"2023-10-16T12:30:23.415779+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:50:01.919395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T12:30:24.398912+0000","last_change":"2023-10-16T12:30:22.404429+0000","last_active":"2023-10-16T12:30:24.398912+0000","last_peered":"2023-10-16T12:30:24.398912+0000","last_clean":"2023-10-16T12:30:24.398912+0000","last_became_active":"2023-10-16T12:30:22.404105+0000","last_became_peered":"2023-10-16T12:30:22.404105+0000","last_unstale":"2023-10-16T12:30:24.398912+0000","last_undegraded":"2023-10-16T12:30:24.398912+0000","last_fullsized":"2023-10-16T12:30:24.398912+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_clean_scrub_stamp":"2023-10-16T12:30:21.373132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:03:02.390944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T12:30:24.397438+0000","last_change":"2023-10-16T12:30:23.413901+0000","last_active":"2023-10-16T12:30:24.397438+0000","last_peered":"2023-10-16T12:30:24.397438+0000","last_clean":"2023-10-16T12:30:24.397438+0000","last_became_active":"2023-10-16T12:30:23.413176+0000","last_became_peered":"2023-10-16T12:30:23.413176+0000","last_unstale":"2023-10-16T12:30:24.397438+0000","last_undegraded":"2023-10-16T12:30:24.397438+0000","last_fullsized":"2023-10-16T12:30:24.397438+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:00:33.801393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.b","version":"141'2","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T12:30:24.397519+0000","last_change":"2023-10-16T12:30:22.396880+0000","last_active":"2023-10-16T12:30:24.397519+0000","last_peered":"2023-10-16T12:30:24.397519+0000","last_clean":"2023-10-16T12:30:24.397519+0000","last_became_active":"2023-10-16T12:30:22.396167+0000","last_became_peered":"2023-10-16T12:30:22.396167+0000","last_unstale":"2023-10-16T12:30:24.397519+0000","last_undegraded":"2023-10-16T12:30:24.397519+0000","last_fullsized":"2023-10-16T12:30:24.397519+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_clean_scrub_stamp":"2023-10-16T12:30:21.373132+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:54:59.412222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.422072+0000","last_change":"2023-10-16T12:30:23.416839+0000","last_active":"2023-10-16T12:30:23.422072+0000","last_peered":"2023-10-16T12:30:23.422072+0000","last_clean":"2023-10-16T12:30:23.422072+0000","last_became_active":"2023-10-16T12:30:23.416564+0000","last_became_peered":"2023-10-16T12:30:23.416564+0000","last_unstale":"2023-10-16T12:30:23.422072+0000","last_undegraded":"2023-10-16T12:30:23.422072+0000","last_fullsized":"2023-10-16T12:30:23.422072+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:40:24.254569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T12:30:24.398808+0000","last_change":"2023-10-16T12:30:22.396419+0000","last_active":"2023-10-16T12:30:24.398808+0000","last_peered":"2023-10-16T12:30:24.398808+0000","last_clean":"2023-10-16T12:30:24.398808+0000","last_became_active":"2023-10-16T12:30:22.396141+0000","last_became_peered":"2023-10-16T12:30:22.396141+0000","last_unstale":"2023-10-16T12:30:24.398808+0000","last_undegraded":"2023-10-16T12:30:24.398808+0000","last_fullsized":"2023-10-16T12:30:24.398808+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_clean_scrub_stamp":"2023-10-16T12:30:21.373132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:31:53.706271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.422134+0000","last_change":"2023-10-16T12:30:23.419461+0000","last_active":"2023-10-16T12:30:23.422134+0000","last_peered":"2023-10-16T12:30:23.422134+0000","last_clean":"2023-10-16T12:30:23.422134+0000","last_became_active":"2023-10-16T12:30:23.418554+0000","last_became_peered":"2023-10-16T12:30:23.418554+0000","last_unstale":"2023-10-16T12:30:23.422134+0000","last_undegraded":"2023-10-16T12:30:23.422134+0000","last_fullsized":"2023-10-16T12:30:23.422134+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:47:43.023905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":11,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.407483+0000","last_change":"2023-10-16T12:30:22.406381+0000","last_active":"2023-10-16T12:30:23.407483+0000","last_peered":"2023-10-16T12:30:23.407483+0000","last_clean":"2023-10-16T12:30:23.407483+0000","last_became_active":"2023-10-16T12:30:22.406085+0000","last_became_peered":"2023-10-16T12:30:22.406085+0000","last_unstale":"2023-10-16T12:30:23.407483+0000","last_undegraded":"2023-10-16T12:30:23.407483+0000","last_fullsized":"2023-10-16T12:30:23.407483+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_clean_scrub_stamp":"2023-10-16T12:30:21.373132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:10:13.452435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.b","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.415370+0000","last_change":"2023-10-16T12:30:23.413305+0000","last_active":"2023-10-16T12:30:23.415370+0000","last_peered":"2023-10-16T12:30:23.415370+0000","last_clean":"2023-10-16T12:30:23.415370+0000","last_became_active":"2023-10-16T12:30:23.412990+0000","last_became_peered":"2023-10-16T12:30:23.412990+0000","last_unstale":"2023-10-16T12:30:23.415370+0000","last_undegraded":"2023-10-16T12:30:23.415370+0000","last_fullsized":"2023-10-16T12:30:23.415370+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:34:41.863547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T12:30:24.398000+0000","last_change":"2023-10-16T12:30:22.400832+0000","last_active":"2023-10-16T12:30:24.398000+0000","last_peered":"2023-10-16T12:30:24.398000+0000","last_clean":"2023-10-16T12:30:24.398000+0000","last_became_active":"2023-10-16T12:30:22.400416+0000","last_became_peered":"2023-10-16T12:30:22.400416+0000","last_unstale":"2023-10-16T12:30:24.398000+0000","last_undegraded":"2023-10-16T12:30:24.398000+0000","last_fullsized":"2023-10-16T12:30:24.398000+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_clean_scrub_stamp":"2023-10-16T12:30:21.373132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:39:41.097894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T12:30:24.398050+0000","last_change":"2023-10-16T12:30:23.414165+0000","last_active":"2023-10-16T12:30:24.398050+0000","last_peered":"2023-10-16T12:30:24.398050+0000","last_clean":"2023-10-16T12:30:24.398050+0000","last_became_active":"2023-10-16T12:30:23.413596+0000","last_became_peered":"2023-10-16T12:30:23.413596+0000","last_unstale":"2023-10-16T12:30:24.398050+0000","last_undegraded":"2023-10-16T12:30:24.398050+0000","last_fullsized":"2023-10-16T12:30:24.398050+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:52:28.294872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.7","version":"141'2","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T12:30:24.398245+0000","last_change":"2023-10-16T12:30:22.400351+0000","last_active":"2023-10-16T12:30:24.398245+0000","last_peered":"2023-10-16T12:30:24.398245+0000","last_clean":"2023-10-16T12:30:24.398245+0000","last_became_active":"2023-10-16T12:30:22.400061+0000","last_became_peered":"2023-10-16T12:30:22.400061+0000","last_unstale":"2023-10-16T12:30:24.398245+0000","last_undegraded":"2023-10-16T12:30:24.398245+0000","last_fullsized":"2023-10-16T12:30:24.398245+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_clean_scrub_stamp":"2023-10-16T12:30:21.373132+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:20:04.815147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.413225+0000","last_change":"2023-10-16T12:30:23.406733+0000","last_active":"2023-10-16T12:30:23.413225+0000","last_peered":"2023-10-16T12:30:23.413225+0000","last_clean":"2023-10-16T12:30:23.413225+0000","last_became_active":"2023-10-16T12:30:23.406406+0000","last_became_peered":"2023-10-16T12:30:23.406406+0000","last_unstale":"2023-10-16T12:30:23.413225+0000","last_undegraded":"2023-10-16T12:30:23.413225+0000","last_fullsized":"2023-10-16T12:30:23.413225+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:41:14.491098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":11,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.401492+0000","last_change":"2023-10-16T12:30:22.404263+0000","last_active":"2023-10-16T12:30:23.401492+0000","last_peered":"2023-10-16T12:30:23.401492+0000","last_clean":"2023-10-16T12:30:23.401492+0000","last_became_active":"2023-10-16T12:30:22.403947+0000","last_became_peered":"2023-10-16T12:30:22.403947+0000","last_unstale":"2023-10-16T12:30:23.401492+0000","last_undegraded":"2023-10-16T12:30:23.401492+0000","last_fullsized":"2023-10-16T12:30:23.401492+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_clean_scrub_stamp":"2023-10-16T12:30:21.373132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:28:15.424497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T12:30:24.398651+0000","last_change":"2023-10-16T12:30:23.418901+0000","last_active":"2023-10-16T12:30:24.398651+0000","last_peered":"2023-10-16T12:30:24.398651+0000","last_clean":"2023-10-16T12:30:24.398651+0000","last_became_active":"2023-10-16T12:30:23.418620+0000","last_became_peered":"2023-10-16T12:30:23.418620+0000","last_unstale":"2023-10-16T12:30:24.398651+0000","last_undegraded":"2023-10-16T12:30:24.398651+0000","last_fullsized":"2023-10-16T12:30:24.398651+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:10:14.456088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":11,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.403501+0000","last_change":"2023-10-16T12:30:22.405830+0000","last_active":"2023-10-16T12:30:23.403501+0000","last_peered":"2023-10-16T12:30:23.403501+0000","last_clean":"2023-10-16T12:30:23.403501+0000","last_became_active":"2023-10-16T12:30:22.405610+0000","last_became_peered":"2023-10-16T12:30:22.405610+0000","last_unstale":"2023-10-16T12:30:23.403501+0000","last_undegraded":"2023-10-16T12:30:23.403501+0000","last_fullsized":"2023-10-16T12:30:23.403501+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_clean_scrub_stamp":"2023-10-16T12:30:21.373132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:21:11.657992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.419009+0000","last_change":"2023-10-16T12:30:23.417810+0000","last_active":"2023-10-16T12:30:23.419009+0000","last_peered":"2023-10-16T12:30:23.419009+0000","last_clean":"2023-10-16T12:30:23.419009+0000","last_became_active":"2023-10-16T12:30:23.417530+0000","last_became_peered":"2023-10-16T12:30:23.417530+0000","last_unstale":"2023-10-16T12:30:23.419009+0000","last_undegraded":"2023-10-16T12:30:23.419009+0000","last_fullsized":"2023-10-16T12:30:23.419009+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:16:55.739332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T12:30:24.398074+0000","last_change":"2023-10-16T12:30:22.406512+0000","last_active":"2023-10-16T12:30:24.398074+0000","last_peered":"2023-10-16T12:30:24.398074+0000","last_clean":"2023-10-16T12:30:24.398074+0000","last_became_active":"2023-10-16T12:30:22.403874+0000","last_became_peered":"2023-10-16T12:30:22.403874+0000","last_unstale":"2023-10-16T12:30:24.398074+0000","last_undegraded":"2023-10-16T12:30:24.398074+0000","last_fullsized":"2023-10-16T12:30:24.398074+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_clean_scrub_stamp":"2023-10-16T12:30:21.373132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:41:04.465715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T12:30:24.397612+0000","last_change":"2023-10-16T12:30:23.426097+0000","last_active":"2023-10-16T12:30:24.397612+0000","last_peered":"2023-10-16T12:30:24.397612+0000","last_clean":"2023-10-16T12:30:24.397612+0000","last_became_active":"2023-10-16T12:30:23.425735+0000","last_became_peered":"2023-10-16T12:30:23.425735+0000","last_unstale":"2023-10-16T12:30:24.397612+0000","last_undegraded":"2023-10-16T12:30:24.397612+0000","last_fullsized":"2023-10-16T12:30:24.397612+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:16:31.182873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":11,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.423323+0000","last_change":"2023-10-16T12:30:22.394626+0000","last_active":"2023-10-16T12:30:23.423323+0000","last_peered":"2023-10-16T12:30:23.423323+0000","last_clean":"2023-10-16T12:30:23.423323+0000","last_became_active":"2023-10-16T12:30:22.394351+0000","last_became_peered":"2023-10-16T12:30:22.394351+0000","last_unstale":"2023-10-16T12:30:23.423323+0000","last_undegraded":"2023-10-16T12:30:23.423323+0000","last_fullsized":"2023-10-16T12:30:23.423323+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_clean_scrub_stamp":"2023-10-16T12:30:21.373132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:21:57.419693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.422323+0000","last_change":"2023-10-16T12:30:23.420117+0000","last_active":"2023-10-16T12:30:23.422323+0000","last_peered":"2023-10-16T12:30:23.422323+0000","last_clean":"2023-10-16T12:30:23.422323+0000","last_became_active":"2023-10-16T12:30:23.418417+0000","last_became_peered":"2023-10-16T12:30:23.418417+0000","last_unstale":"2023-10-16T12:30:23.422323+0000","last_undegraded":"2023-10-16T12:30:23.422323+0000","last_fullsized":"2023-10-16T12:30:23.422323+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:34:07.114770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":11,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.426712+0000","last_change":"2023-10-16T12:30:22.398876+0000","last_active":"2023-10-16T12:30:23.426712+0000","last_peered":"2023-10-16T12:30:23.426712+0000","last_clean":"2023-10-16T12:30:23.426712+0000","last_became_active":"2023-10-16T12:30:22.398006+0000","last_became_peered":"2023-10-16T12:30:22.398006+0000","last_unstale":"2023-10-16T12:30:23.426712+0000","last_undegraded":"2023-10-16T12:30:23.426712+0000","last_fullsized":"2023-10-16T12:30:23.426712+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_clean_scrub_stamp":"2023-10-16T12:30:21.373132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:25:32.064591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T12:30:24.398692+0000","last_change":"2023-10-16T12:30:23.411352+0000","last_active":"2023-10-16T12:30:24.398692+0000","last_peered":"2023-10-16T12:30:24.398692+0000","last_clean":"2023-10-16T12:30:24.398692+0000","last_became_active":"2023-10-16T12:30:23.411044+0000","last_became_peered":"2023-10-16T12:30:23.411044+0000","last_unstale":"2023-10-16T12:30:24.398692+0000","last_undegraded":"2023-10-16T12:30:24.398692+0000","last_fullsized":"2023-10-16T12:30:24.398692+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:26:57.487238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T12:30:24.397483+0000","last_change":"2023-10-16T12:30:22.401410+0000","last_active":"2023-10-16T12:30:24.397483+0000","last_peered":"2023-10-16T12:30:24.397483+0000","last_clean":"2023-10-16T12:30:24.397483+0000","last_became_active":"2023-10-16T12:30:22.401101+0000","last_became_peered":"2023-10-16T12:30:22.401101+0000","last_unstale":"2023-10-16T12:30:24.397483+0000","last_undegraded":"2023-10-16T12:30:24.397483+0000","last_fullsized":"2023-10-16T12:30:24.397483+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_clean_scrub_stamp":"2023-10-16T12:30:21.373132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:13:21.620462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":9,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.409943+0000","last_change":"2023-10-16T12:30:23.403923+0000","last_active":"2023-10-16T12:30:23.409943+0000","last_peered":"2023-10-16T12:30:23.409943+0000","last_clean":"2023-10-16T12:30:23.409943+0000","last_became_active":"2023-10-16T12:30:23.403627+0000","last_became_peered":"2023-10-16T12:30:23.403627+0000","last_unstale":"2023-10-16T12:30:23.409943+0000","last_undegraded":"2023-10-16T12:30:23.409943+0000","last_fullsized":"2023-10-16T12:30:23.409943+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:48:30.457561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T12:30:24.398014+0000","last_change":"2023-10-16T12:30:22.401281+0000","last_active":"2023-10-16T12:30:24.398014+0000","last_peered":"2023-10-16T12:30:24.398014+0000","last_clean":"2023-10-16T12:30:24.398014+0000","last_became_active":"2023-10-16T12:30:22.400837+0000","last_became_peered":"2023-10-16T12:30:22.400837+0000","last_unstale":"2023-10-16T12:30:24.398014+0000","last_undegraded":"2023-10-16T12:30:24.398014+0000","last_fullsized":"2023-10-16T12:30:24.398014+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_clean_scrub_stamp":"2023-10-16T12:30:21.373132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:53:18.538027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"0'0","reported_seq":11,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-16T12:30:24.398739+0000","last_change":"2023-10-16T12:30:23.412234+0000","last_active":"2023-10-16T12:30:24.398739+0000","last_peered":"2023-10-16T12:30:24.398739+0000","last_clean":"2023-10-16T12:30:24.398739+0000","last_became_active":"2023-10-16T12:30:23.411539+0000","last_became_peered":"2023-10-16T12:30:23.411539+0000","last_unstale":"2023-10-16T12:30:24.398739+0000","last_undegraded":"2023-10-16T12:30:24.398739+0000","last_fullsized":"2023-10-16T12:30:24.398739+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:22.376785+0000","last_clean_scrub_stamp":"2023-10-16T12:30:22.376785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:18:36.577516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.10","version":"141'8","reported_seq":19,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.433481+0000","last_change":"2023-10-16T12:30:22.395953+0000","last_active":"2023-10-16T12:30:23.433481+0000","last_peered":"2023-10-16T12:30:23.433481+0000","last_clean":"2023-10-16T12:30:23.433481+0000","last_became_active":"2023-10-16T12:30:22.395623+0000","last_became_peered":"2023-10-16T12:30:22.395623+0000","last_unstale":"2023-10-16T12:30:23.433481+0000","last_undegraded":"2023-10-16T12:30:23.433481+0000","last_fullsized":"2023-10-16T12:30:23.433481+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:30:21.373132+0000","last_clean_scrub_stamp":"2023-10-16T12:30:21.373132+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:03:02.390944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"132'227","reported_seq":666,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-16T12:30:23.397265+0000","last_change":"2023-10-16T12:10:34.746820+0000","last_active":"2023-10-16T12:30:23.397265+0000","last_peered":"2023-10-16T12:30:23.397265+0000","last_clean":"2023-10-16T12:30:23.397265+0000","last_became_active":"2023-10-16T12:10:34.745574+0000","last_became_peered":"2023-10-16T12:10:34.745574+0000","last_unstale":"2023-10-16T12:30:23.397265+0000","last_undegraded":"2023-10-16T12:30:23.397265+0000","last_fullsized":"2023-10-16T12:30:23.397265+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_clean_scrub_stamp":"2023-10-16T12:10:33.577111+0000","objects_scrubbed":0,"log_size":227,"log_dups_size":0,"ondisk_log_size":227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:14.461986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1076,"num_read_kb":2160,"num_write":359,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":18,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1076,"num_read_kb":2160,"num_write":359,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":227,"ondisk_log_size":227,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575083,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47388,"kb_used_data":2308,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324452,"statfs":{"total":96636764160,"available":96588238848,"internally_reserved":0,"allocated":2363392,"data_stored":1475796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":46135755},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:30:18 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.61199999999999999,"15min":0.60199999999999998},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.27400000000000002},"max":{"1min":0.89300000000000002,"5min":4.2469999999999999,"15min":4.2469999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.57199999999999995,"15min":0.58199999999999996},"min":{"1min":0.35299999999999998,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.84899999999999998,"5min":0.85299999999999998,"15min":0.89900000000000002},"last":0.59099999999999997}]},{"osd":1,"last update":"Mon Oct 16 12:30:18 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.58499999999999996,"15min":0.59999999999999998},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.247},"max":{"1min":0.80200000000000005,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.58999999999999997,"15min":0.60099999999999998},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.32400000000000001},"max":{"1min":0.79200000000000004,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.497}]},{"osd":2,"last update":"Mon Oct 16 12:30:18 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.64400000000000002,"15min":0.65700000000000003},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.88600000000000001,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.64800000000000002,"15min":0.66300000000000003},"min":{"1min":0.44800000000000001,"5min":0.38100000000000001,"15min":0.376},"max":{"1min":0.88300000000000001,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.65500000000000003}]},{"osd":3,"last update":"Mon Oct 16 12:30:18 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.61099999999999999,"15min":0.61499999999999999},"min":{"1min":0.377,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.63,"15min":0.63300000000000001},"min":{"1min":0.41199999999999998,"5min":0.372,"15min":0.372},"max":{"1min":0.81000000000000005,"5min":0.996,"15min":1.2509999999999999},"last":0.66700000000000004}]},{"osd":4,"last update":"Mon Oct 16 12:30:18 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.54400000000000004,"15min":0.53000000000000003},"min":{"1min":0.38600000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.80300000000000005,"5min":0.89700000000000002,"15min":1.383},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.627,"5min":0.63500000000000001,"15min":0.61499999999999999},"min":{"1min":0.39800000000000002,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.88200000000000001,"5min":0.93600000000000005,"15min":1.0109999999999999},"last":0.68600000000000005}]},{"osd":5,"last update":"Mon Oct 16 12:30:18 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.56899999999999995,"15min":0.54000000000000004},"min":{"1min":0.311,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.77100000000000002,"5min":0.93400000000000005,"15min":1.3109999999999999},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.628,"5min":0.60799999999999998,"15min":0.58799999999999997},"min":{"1min":0.378,"5min":0.34699999999999998,"15min":0.32400000000000001},"max":{"1min":0.86099999999999999,"5min":0.93600000000000005,"15min":1.4359999999999999},"last":0.76000000000000001}]},{"osd":6,"last update":"Mon Oct 16 12:30:18 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.64800000000000002,"15min":0.625},"min":{"1min":0.41499999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.93000000000000005,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.64900000000000002,"15min":0.63},"min":{"1min":0.35099999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.91600000000000004,"5min":0.97199999999999998,"15min":1.1259999999999999},"last":0.745}]}]},{"osd":6,"up_from":13,"seq":55834575081,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37060,"kb_used_data":1716,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334780,"statfs":{"total":96636764160,"available":96598814720,"internally_reserved":0,"allocated":1757184,"data_stored":885234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:30:07 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.57499999999999996,"15min":0.60899999999999999},"min":{"1min":0.35199999999999998,"5min":0.29299999999999998,"15min":0.24299999999999999},"max":{"1min":0.87,"5min":0.88200000000000001,"15min":1.196},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.59699999999999998,"15min":0.621},"min":{"1min":0.436,"5min":0.38500000000000001,"15min":0.33600000000000002},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.997},"last":0.56499999999999995}]},{"osd":1,"last update":"Mon Oct 16 12:30:07 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.53300000000000003,"15min":0.56000000000000005},"min":{"1min":0.35699999999999998,"5min":0.33100000000000002,"15min":0.29399999999999998},"max":{"1min":0.745,"5min":0.82199999999999995,"15min":0.875},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.57399999999999995,"15min":0.58599999999999997},"min":{"1min":0.40400000000000003,"5min":0.29799999999999999,"15min":0.29499999999999998},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":1.091},"last":0.65600000000000003}]},{"osd":2,"last update":"Mon Oct 16 12:30:07 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56499999999999995,"15min":0.58999999999999997},"min":{"1min":0.377,"5min":0.35599999999999998,"15min":0.34499999999999997},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.97099999999999997},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.56899999999999995,"15min":0.59499999999999997},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.27100000000000002},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":2.3879999999999999},"last":0.50600000000000001}]},{"osd":3,"last update":"Mon Oct 16 12:30:07 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.60699999999999998,"15min":0.624},"min":{"1min":0.41599999999999998,"5min":0.37,"15min":0.34200000000000003},"max":{"1min":0.82299999999999995,"5min":0.88,"15min":0.93999999999999995},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.66700000000000004,"15min":0.67800000000000005},"min":{"1min":0.504,"5min":0.41899999999999998,"15min":0.40899999999999997},"max":{"1min":0.90300000000000002,"5min":0.94499999999999995,"15min":1.3080000000000001},"last":0.77800000000000002}]},{"osd":4,"last update":"Mon Oct 16 12:30:07 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.52200000000000002,"15min":0.51600000000000001},"min":{"1min":0.35899999999999999,"5min":0.30599999999999999,"15min":0.29899999999999999},"max":{"1min":0.65700000000000003,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.60099999999999998,"15min":0.59699999999999998},"min":{"1min":0.38700000000000001,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.79400000000000004,"5min":0.82299999999999995,"15min":1.2250000000000001},"last":0.41399999999999998}]},{"osd":5,"last update":"Mon Oct 16 12:30:07 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.57899999999999996,"15min":0.57699999999999996},"min":{"1min":0.41499999999999998,"5min":0.30299999999999999,"15min":0.30099999999999999},"max":{"1min":0.70399999999999996,"5min":0.98799999999999999,"15min":1.0589999999999999},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.64300000000000002,"15min":0.626},"min":{"1min":0.33400000000000002,"5min":0.32100000000000001,"15min":0.311},"max":{"1min":0.84299999999999997,"5min":1.0029999999999999,"15min":1.3540000000000001},"last":0.57899999999999996}]},{"osd":7,"last update":"Mon Oct 16 12:30:07 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.58299999999999996,"15min":0.57599999999999996},"min":{"1min":0.42499999999999999,"5min":0.29699999999999999,"15min":0.26000000000000001},"max":{"1min":0.68799999999999994,"5min":0.92700000000000005,"15min":1.29},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.623,"5min":0.63400000000000001,"15min":0.61899999999999999},"min":{"1min":0.436,"5min":0.32900000000000001,"15min":0.29799999999999999},"max":{"1min":0.873,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.48899999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575081,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37056,"kb_used_data":1716,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334784,"statfs":{"total":96636764160,"available":96598818816,"internally_reserved":0,"allocated":1757184,"data_stored":884674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":36174281},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.374,"15min":0.38},"min":{"1min":0.26600000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.627,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.434,"15min":0.46000000000000002},"min":{"1min":0.28899999999999998,"5min":0.28399999999999997,"15min":0.26700000000000002},"max":{"1min":0.60499999999999998,"5min":0.66300000000000003,"15min":1.0189999999999999},"last":0.35699999999999998}]},{"osd":2,"last update":"Mon Oct 16 12:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.39800000000000002,"15min":0.42599999999999999},"min":{"1min":0.26900000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.59199999999999997,"5min":0.65300000000000002,"15min":0.93000000000000005},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.44,"5min":0.45000000000000001,"15min":0.46700000000000003},"min":{"1min":0.33700000000000002,"5min":0.29099999999999998,"15min":0.28199999999999997},"max":{"1min":0.61699999999999999,"5min":0.69199999999999995,"15min":0.98099999999999998},"last":0.376}]},{"osd":3,"last update":"Mon Oct 16 12:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.38200000000000001,"15min":0.40300000000000002},"min":{"1min":0.247,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.56999999999999995,"5min":0.69599999999999995,"15min":0.98799999999999999},"last":0.311},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.42999999999999999,"15min":0.45300000000000001},"min":{"1min":0.30299999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.751,"5min":0.751,"15min":0.91200000000000003},"last":0.29299999999999998}]},{"osd":4,"last update":"Mon Oct 16 12:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.60399999999999998,"15min":0.627},"min":{"1min":0.40500000000000003,"5min":0.35499999999999998,"15min":0.35199999999999998},"max":{"1min":0.74099999999999999,"5min":0.878,"15min":1.0409999999999999},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.624,"15min":0.64200000000000002},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.39100000000000001},"max":{"1min":0.86699999999999999,"5min":0.95399999999999996,"15min":0.98599999999999999},"last":0.48199999999999998}]},{"osd":5,"last update":"Mon Oct 16 12:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.63900000000000001,"15min":0.66300000000000003},"min":{"1min":0.42899999999999999,"5min":0.42599999999999999,"15min":0.38300000000000001},"max":{"1min":0.95399999999999996,"5min":1.069,"15min":1.069},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.66300000000000003,"15min":0.68400000000000005},"min":{"1min":0.437,"5min":0.437,"15min":0.38900000000000001},"max":{"1min":0.93899999999999995,"5min":1.034,"15min":1.0620000000000001},"last":0.39300000000000002}]},{"osd":6,"last update":"Mon Oct 16 12:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.63400000000000001,"15min":0.65100000000000002},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.82599999999999996,"5min":1.044,"15min":1.044},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.63100000000000001,"15min":0.65800000000000003},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.32900000000000001},"max":{"1min":0.90400000000000003,"5min":1.0049999999999999,"15min":1.048},"last":0.436}]},{"osd":7,"last update":"Mon Oct 16 12:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.64800000000000002,"15min":0.68300000000000005},"min":{"1min":0.45800000000000002,"5min":0.42099999999999999,"15min":0.33700000000000002},"max":{"1min":0.80600000000000005,"5min":1.0469999999999999,"15min":1.3400000000000001},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.627,"5min":0.67600000000000005,"15min":0.69699999999999995},"min":{"1min":0.47299999999999998,"5min":0.45700000000000002,"15min":0.42099999999999999},"max":{"1min":0.89100000000000001,"5min":1.006,"15min":1.02},"last":0.45300000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575082,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37652,"kb_used_data":2308,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334188,"statfs":{"total":96636764160,"available":96598208512,"internally_reserved":0,"allocated":2363392,"data_stored":1481718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 12:29:58 2023","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.39500000000000002,"15min":0.39200000000000002},"min":{"1min":0.27100000000000002,"5min":0.219,"15min":0.219},"max":{"1min":0.57099999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.40699999999999997,"15min":0.41799999999999998},"min":{"1min":0.251,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.73099999999999998},"last":0.35499999999999998}]},{"osd":2,"last update":"Mon Oct 16 12:29:58 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.42299999999999999,"15min":0.42499999999999999},"min":{"1min":0.314,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.44800000000000001,"15min":0.45300000000000001},"min":{"1min":0.29999999999999999,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.57299999999999995,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.438}]},{"osd":3,"last update":"Mon Oct 16 12:29:58 2023","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.42799999999999999,"15min":0.42599999999999999},"min":{"1min":0.28299999999999997,"5min":0.26300000000000001,"15min":0.23699999999999999},"max":{"1min":0.56399999999999995,"5min":0.62,"15min":0.79400000000000004},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.48299999999999998,"15min":0.48799999999999999},"min":{"1min":0.314,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.63400000000000001,"5min":0.79200000000000004,"15min":0.878},"last":0.41499999999999998}]},{"osd":4,"last update":"Mon Oct 16 12:29:58 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.64000000000000001,"15min":0.65100000000000002},"min":{"1min":0.40200000000000002,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.82499999999999996,"5min":1.05,"15min":1.0960000000000001},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.63,"15min":0.65600000000000003},"min":{"1min":0.38800000000000001,"5min":0.35599999999999998,"15min":0.34100000000000003},"max":{"1min":0.97099999999999997,"5min":1.0549999999999999,"15min":1.1719999999999999},"last":0.496}]},{"osd":5,"last update":"Mon Oct 16 12:29:58 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.629,"15min":0.65100000000000002},"min":{"1min":0.39200000000000002,"5min":0.373,"15min":0.373},"max":{"1min":0.88100000000000001,"5min":0.97699999999999998,"15min":1.127},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.66800000000000004,"15min":0.67800000000000005},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39000000000000001},"max":{"1min":0.90000000000000002,"5min":1.0700000000000001,"15min":1.1160000000000001},"last":0.59999999999999998}]},{"osd":6,"last update":"Mon Oct 16 12:29:58 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.67200000000000004,"15min":0.68500000000000005},"min":{"1min":0.42699999999999999,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.85599999999999998,"5min":0.96899999999999997,"15min":1.079},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.66600000000000004,"15min":0.68700000000000006},"min":{"1min":0.40999999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.86699999999999999,"5min":0.96399999999999997,"15min":1.1499999999999999},"last":0.64500000000000002}]},{"osd":7,"last update":"Mon Oct 16 12:29:58 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.68500000000000005,"15min":0.69999999999999996},"min":{"1min":0.47299999999999998,"5min":0.436,"15min":0.41499999999999998},"max":{"1min":0.83199999999999996,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.64800000000000002,"15min":0.68500000000000005},"min":{"1min":0.438,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.93200000000000005,"5min":0.97499999999999998,"15min":1.0569999999999999},"last":0.58599999999999997}]}]},{"osd":2,"up_from":13,"seq":55834575082,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63420,"kb_used_data":1712,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94308420,"statfs":{"total":96636764160,"available":96571822080,"internally_reserved":0,"allocated":1753088,"data_stored":884652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":63175110},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:30:05 2023","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.36899999999999999,"15min":0.379},"min":{"1min":0.222,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.51200000000000001,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.42699999999999999,"15min":0.443},"min":{"1min":0.28599999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.497,"5min":0.66500000000000004,"15min":0.82999999999999996},"last":0.39900000000000002}]},{"osd":1,"last update":"Mon Oct 16 12:30:05 2023","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.441,"15min":0.46100000000000002},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.64500000000000002,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.439},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.40500000000000003,"15min":0.42999999999999999},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.57199999999999995,"5min":0.59199999999999997,"15min":0.69699999999999995},"last":0.26200000000000001}]},{"osd":3,"last update":"Mon Oct 16 12:30:05 2023","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.39400000000000002,"15min":0.40899999999999997},"min":{"1min":0.25,"5min":0.214,"15min":0.214},"max":{"1min":0.47799999999999998,"5min":0.69399999999999995,"15min":0.89300000000000002},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.498,"15min":0.51200000000000001},"min":{"1min":0.33200000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.74099999999999999,"5min":0.78000000000000003,"15min":0.79600000000000004},"last":0.501}]},{"osd":4,"last update":"Mon Oct 16 12:30:05 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.63800000000000001,"15min":0.66800000000000004},"min":{"1min":0.35499999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.753,"5min":0.93799999999999994,"15min":1.3839999999999999},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.628,"15min":0.66500000000000004},"min":{"1min":0.42199999999999999,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.79900000000000004,"5min":0.97599999999999998,"15min":1.0109999999999999},"last":0.60199999999999998}]},{"osd":5,"last update":"Mon Oct 16 12:30:05 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.63500000000000001,"15min":0.65500000000000003},"min":{"1min":0.41599999999999998,"5min":0.35999999999999999,"15min":0.312},"max":{"1min":0.86699999999999999,"5min":1.093,"15min":1.093},"last":0.624},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.61299999999999999,"15min":0.64700000000000002},"min":{"1min":0.40999999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.70499999999999996,"5min":1.0089999999999999,"15min":1.331},"last":0.46300000000000002}]},{"osd":6,"last update":"Mon Oct 16 12:30:05 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59599999999999997,"15min":0.626},"min":{"1min":0.42099999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.76200000000000001,"5min":0.99299999999999999,"15min":1.0369999999999999},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.65100000000000002,"15min":0.67600000000000005},"min":{"1min":0.437,"5min":0.40799999999999997,"15min":0.40000000000000002},"max":{"1min":0.83799999999999997,"5min":1.032,"15min":1.032},"last":0.68400000000000005}]},{"osd":7,"last update":"Mon Oct 16 12:30:05 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.64800000000000002,"15min":0.66600000000000004},"min":{"1min":0.432,"5min":0.40300000000000002,"15min":0.38400000000000001},"max":{"1min":0.78300000000000003,"5min":0.96399999999999997,"15min":1.016},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.65000000000000002,"15min":0.67800000000000005},"min":{"1min":0.44400000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.86199999999999999,"5min":1.145,"15min":1.145},"last":0.56999999999999995}]}]},{"osd":3,"up_from":13,"seq":55834575081,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37068,"kb_used_data":1728,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334772,"statfs":{"total":96636764160,"available":96598806528,"internally_reserved":0,"allocated":1769472,"data_stored":886298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:30:14 2023","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.38900000000000001,"15min":0.39700000000000002},"min":{"1min":0.22500000000000001,"5min":0.216,"15min":0.21099999999999999},"max":{"1min":0.46999999999999997,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.39200000000000002,"15min":0.40799999999999997},"min":{"1min":0.28000000000000003,"5min":0.254,"15min":0.254},"max":{"1min":0.54000000000000004,"5min":0.61899999999999999,"15min":0.73099999999999998},"last":0.35699999999999998}]},{"osd":1,"last update":"Mon Oct 16 12:30:14 2023","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.39900000000000002,"15min":0.41299999999999998},"min":{"1min":0.26400000000000001,"5min":0.245,"15min":0.24399999999999999},"max":{"1min":0.57499999999999996,"5min":0.70699999999999996,"15min":0.70799999999999996},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.438,"15min":0.45800000000000002},"min":{"1min":0.29499999999999998,"5min":0.27000000000000002,"15min":0.25800000000000001},"max":{"1min":0.60399999999999998,"5min":0.72699999999999998,"15min":0.747},"last":0.33700000000000002}]},{"osd":2,"last update":"Mon Oct 16 12:30:14 2023","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.45300000000000001,"15min":0.46999999999999997},"min":{"1min":0.313,"5min":0.29799999999999999,"15min":0.29199999999999998},"max":{"1min":0.63,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.47899999999999998,"15min":0.498},"min":{"1min":0.311,"5min":0.30599999999999999,"15min":0.27100000000000002},"max":{"1min":0.72799999999999998,"5min":0.74299999999999999,"15min":0.752},"last":0.38}]},{"osd":4,"last update":"Mon Oct 16 12:30:14 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.623,"15min":0.64500000000000002},"min":{"1min":0.39200000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.89600000000000002,"5min":0.98299999999999998,"15min":1.1539999999999999},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.622,"15min":0.65600000000000003},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.92100000000000004,"5min":0.95399999999999996,"15min":1.0409999999999999},"last":0.64400000000000002}]},{"osd":5,"last update":"Mon Oct 16 12:30:14 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.65200000000000002,"15min":0.68100000000000005},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.89400000000000002,"5min":1.0369999999999999,"15min":1.1080000000000001},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.63200000000000001,"15min":0.66000000000000003},"min":{"1min":0.45200000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.89000000000000001,"5min":0.97299999999999998,"15min":1.022},"last":0.59799999999999998}]},{"osd":6,"last update":"Mon Oct 16 12:30:14 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63300000000000001,"15min":0.66500000000000004},"min":{"1min":0.437,"5min":0.42999999999999999,"15min":0.36899999999999999},"max":{"1min":0.875,"5min":0.94199999999999995,"15min":1.0760000000000001},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.67400000000000004,"15min":0.70199999999999996},"min":{"1min":0.434,"5min":0.434,"15min":0.38600000000000001},"max":{"1min":0.89100000000000001,"5min":1.0349999999999999,"15min":1.0409999999999999},"last":0.57799999999999996}]},{"osd":7,"last update":"Mon Oct 16 12:30:14 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.64700000000000002,"15min":0.67900000000000005},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.38},"max":{"1min":0.85999999999999999,"5min":1.0149999999999999,"15min":1.2430000000000001},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.68500000000000005,"15min":0.70499999999999996},"min":{"1min":0.503,"5min":0.44,"15min":0.42699999999999999},"max":{"1min":0.88100000000000001,"5min":1.002,"15min":1.1339999999999999},"last":0.61499999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575080,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32916,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338924,"statfs":{"total":96636764160,"available":96603058176,"internally_reserved":0,"allocated":1765376,"data_stored":886264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.56599999999999995,"15min":0.58999999999999997},"min":{"1min":0.41099999999999998,"5min":0.30399999999999999,"15min":0.27900000000000003},"max":{"1min":0.752,"5min":0.78500000000000003,"15min":0.94599999999999995},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.57199999999999995,"15min":0.59699999999999998},"min":{"1min":0.42399999999999999,"5min":0.29999999999999999,"15min":0.27600000000000002},"max":{"1min":0.73299999999999998,"5min":0.83199999999999996,"15min":0.88700000000000001},"last":0.60599999999999998}]},{"osd":1,"last update":"Mon Oct 16 12:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.58599999999999997,"15min":0.60799999999999998},"min":{"1min":0.32900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.76200000000000001,"5min":0.82599999999999996,"15min":0.92600000000000005},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.61399999999999999},"min":{"1min":0.434,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.98099999999999998},"last":0.76700000000000002}]},{"osd":2,"last update":"Mon Oct 16 12:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.58399999999999996,"15min":0.61799999999999999},"min":{"1min":0.34999999999999998,"5min":0.33800000000000002,"15min":0.32000000000000001},"max":{"1min":0.745,"5min":0.84299999999999997,"15min":1.012},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.59699999999999998,"15min":0.624},"min":{"1min":0.38200000000000001,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.76000000000000001,"5min":0.85599999999999998,"15min":0.94199999999999995},"last":0.71899999999999997}]},{"osd":3,"last update":"Mon Oct 16 12:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.56899999999999995,"15min":0.60199999999999998},"min":{"1min":0.41799999999999998,"5min":0.35599999999999998,"15min":0.33700000000000002},"max":{"1min":0.70699999999999996,"5min":0.82299999999999995,"15min":0.93200000000000005},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.63,"5min":0.63300000000000001,"15min":0.64300000000000002},"min":{"1min":0.47099999999999997,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.80000000000000004,"5min":0.90600000000000003,"15min":0.97199999999999998},"last":0.69799999999999995}]},{"osd":5,"last update":"Mon Oct 16 12:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.52100000000000002,"15min":0.50600000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.72199999999999998,"5min":0.82599999999999996,"15min":0.95699999999999996},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.53000000000000003,"15min":0.52500000000000002},"min":{"1min":0.373,"5min":0.33900000000000002,"15min":0.308},"max":{"1min":0.73399999999999999,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.67800000000000005}]},{"osd":6,"last update":"Mon Oct 16 12:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.55600000000000005,"15min":0.55700000000000005},"min":{"1min":0.39000000000000001,"5min":0.33800000000000002,"15min":0.32500000000000001},"max":{"1min":0.70699999999999996,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.628},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.63800000000000001,"15min":0.63700000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.32000000000000001},"max":{"1min":0.84499999999999997,"5min":0.89300000000000002,"15min":0.99299999999999999},"last":0.82399999999999995}]},{"osd":7,"last update":"Mon Oct 16 12:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59199999999999997,"15min":0.58499999999999996},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.77700000000000002,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.629,"15min":0.629},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.29999999999999999},"max":{"1min":0.82899999999999996,"5min":1.006,"15min":1.006},"last":0.751}]}]},{"osd":5,"up_from":13,"seq":55834575081,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55112,"kb_used_data":1720,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316728,"statfs":{"total":96636764160,"available":96580329472,"internally_reserved":0,"allocated":1761280,"data_stored":884812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:30:04 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.55500000000000005,"15min":0.58599999999999997},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.28000000000000003},"max":{"1min":0.81399999999999995,"5min":0.84499999999999997,"15min":0.873},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.59399999999999997,"15min":0.61499999999999999},"min":{"1min":0.34599999999999997,"5min":0.34399999999999997,"15min":0.27600000000000002},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.64900000000000002}]},{"osd":1,"last update":"Mon Oct 16 12:30:04 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.58999999999999997,"15min":0.60999999999999999},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.28999999999999998},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57899999999999996,"15min":0.61199999999999999},"min":{"1min":0.31900000000000001,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.78600000000000003,"5min":0.84799999999999998,"15min":0.89000000000000001},"last":0.63500000000000001}]},{"osd":2,"last update":"Mon Oct 16 12:30:04 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.59299999999999997,"15min":0.62},"min":{"1min":0.40899999999999997,"5min":0.36899999999999999,"15min":0.29099999999999998},"max":{"1min":0.82499999999999996,"5min":0.876,"15min":0.94399999999999995},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.60799999999999998},"min":{"1min":0.40400000000000003,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.72699999999999998}]},{"osd":3,"last update":"Mon Oct 16 12:30:04 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.60999999999999999,"15min":0.63200000000000001},"min":{"1min":0.39700000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.76900000000000002,"5min":0.90400000000000003,"15min":0.97499999999999998},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.625,"15min":0.64300000000000002},"min":{"1min":0.45100000000000001,"5min":0.40400000000000003,"15min":0.35999999999999999},"max":{"1min":0.75,"5min":0.88700000000000001,"15min":0.93200000000000005},"last":0.69999999999999996}]},{"osd":4,"last update":"Mon Oct 16 12:30:04 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49299999999999999,"15min":0.50800000000000001},"min":{"1min":0.35999999999999999,"5min":0.28100000000000003,"15min":0.25800000000000001},"max":{"1min":0.69399999999999995,"5min":0.81299999999999994,"15min":0.86399999999999999},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.36399999999999999,"5min":0.33900000000000002,"15min":0.33500000000000002},"max":{"1min":0.83999999999999997,"5min":0.876,"15min":1.347},"last":0.53800000000000003}]},{"osd":6,"last update":"Mon Oct 16 12:30:04 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.371,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.58099999999999996,"15min":0.59799999999999998},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.74299999999999999,"5min":0.88400000000000001,"15min":0.92700000000000005},"last":0.61499999999999999}]},{"osd":7,"last update":"Mon Oct 16 12:30:04 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.621,"15min":0.621},"min":{"1min":0.41699999999999998,"5min":0.41499999999999998,"15min":0.32300000000000001},"max":{"1min":0.76800000000000002,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.64800000000000002,"15min":0.65700000000000003},"min":{"1min":0.438,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.86799999999999999,"5min":0.93899999999999995,"15min":1.1180000000000001},"last":0.66300000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T12:30:27.975 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-16T12:30:27.976 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-16T12:30:28.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.139+0000 7f03e2ed7700 1 Processor -- start 2023-10-16T12:30:28.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.140+0000 7f03e2ed7700 1 -- start start 2023-10-16T12:30:28.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.140+0000 7f03e2ed7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f03dc159de0 0x7f03dc15a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:28.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.140+0000 7f03e2ed7700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f03dc0cf4c0 0x7f03dc06aac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:28.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.140+0000 7f03e2ed7700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03dc15dda0 con 0x7f03dc06b000 2023-10-16T12:30:28.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.140+0000 7f03e2ed7700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03dc15ea20 con 0x7f03dc159de0 2023-10-16T12:30:28.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.140+0000 7f03e2ed7700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03dc15e080 con 0x7f03dc0cf4c0 2023-10-16T12:30:28.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.141+0000 7f03e0c73700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f03dc159de0 0x7f03dc15a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:28.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.141+0000 7f03e1474700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f03dc06b000 0x7f03dc06b4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34560/0 (socket says 172.21.15.16:34560) 2023-10-16T12:30:28.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.141+0000 7f03e1474700 1 -- 172.21.15.16:0/1696556264 learned_addr learned my addr 172.21.15.16:0/1696556264 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:30:28.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.141+0000 7f03dbfff700 1 --2- 172.21.15.16:0/1696556264 >> v2:172.21.15.49:3300/0 conn(0x7f03dc0cf4c0 0x7f03dc06aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:28.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.141+0000 7f03e0c73700 1 -- 172.21.15.16:0/1696556264 >> v2:172.21.15.49:3300/0 conn(0x7f03dc0cf4c0 msgr2=0x7f03dc06aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:28.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.141+0000 7f03e0c73700 1 --2- 172.21.15.16:0/1696556264 >> v2:172.21.15.49:3300/0 conn(0x7f03dc0cf4c0 0x7f03dc06aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:28.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.141+0000 7f03e0c73700 1 -- 172.21.15.16:0/1696556264 >> v1:172.21.15.16:6789/0 conn(0x7f03dc06b000 legacy=0x7f03dc06b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:28.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.141+0000 7f03e0c73700 1 -- 172.21.15.16:0/1696556264 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03dc15e360 con 0x7f03dc159de0 2023-10-16T12:30:28.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.141+0000 7f03db7fe700 1 -- 172.21.15.16:0/1696556264 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 730359403 0 0) 0x7f03dc15dda0 con 0x7f03dc06b000 2023-10-16T12:30:28.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.141+0000 7f03e0c73700 1 --2- 172.21.15.16:0/1696556264 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03dc159de0 0x7f03dc15a1c0 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f03dc0cd3f0 tx=0x7f03d0009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d860a8e4026690a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:28.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.141+0000 7f03db7fe700 1 -- 172.21.15.16:0/1696556264 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03d0011830 con 0x7f03dc159de0 2023-10-16T12:30:28.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.141+0000 7f03db7fe700 1 -- 172.21.15.16:0/1696556264 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03d0011990 con 0x7f03dc159de0 2023-10-16T12:30:28.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.141+0000 7f03db7fe700 1 -- 172.21.15.16:0/1696556264 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03d0011d20 con 0x7f03dc159de0 2023-10-16T12:30:28.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.142+0000 7f03e2ed7700 1 -- 172.21.15.16:0/1696556264 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03dc159de0 msgr2=0x7f03dc15a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:28.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.142+0000 7f03e2ed7700 1 --2- 172.21.15.16:0/1696556264 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03dc159de0 0x7f03dc15a1c0 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f03dc0cd3f0 tx=0x7f03d0009b20 comp rx=0 tx=0).stop 2023-10-16T12:30:28.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.142+0000 7f03e2ed7700 1 -- 172.21.15.16:0/1696556264 shutdown_connections 2023-10-16T12:30:28.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.142+0000 7f03e2ed7700 1 --2- 172.21.15.16:0/1696556264 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03dc159de0 0x7f03dc15a1c0 unknown :-1 s=CLOSED pgs=1554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:28.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.142+0000 7f03e2ed7700 1 --2- 172.21.15.16:0/1696556264 >> v2:172.21.15.49:3300/0 conn(0x7f03dc0cf4c0 0x7f03dc06aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:28.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.142+0000 7f03e2ed7700 1 -- 172.21.15.16:0/1696556264 >> 172.21.15.16:0/1696556264 conn(0x7f03dc0cc5e0 msgr2=0x7f03dc0cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:28.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.142+0000 7f03e2ed7700 1 -- 172.21.15.16:0/1696556264 shutdown_connections 2023-10-16T12:30:28.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.142+0000 7f03e2ed7700 1 -- 172.21.15.16:0/1696556264 wait complete. 2023-10-16T12:30:28.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.143+0000 7f03e2ed7700 1 Processor -- start 2023-10-16T12:30:28.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.143+0000 7f03e2ed7700 1 -- start start 2023-10-16T12:30:28.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.144+0000 7f03e2ed7700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f03dc06b000 0x7f03dc158f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:28.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.144+0000 7f03e2ed7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03dc0cf4c0 0x7f03dc150920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:28.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.144+0000 7f03e2ed7700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f03dc159de0 0x7f03dc0ca750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:28.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.144+0000 7f03e2ed7700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03dc14ac60 con 0x7f03dc0cf4c0 2023-10-16T12:30:28.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.145+0000 7f03e2ed7700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03dc14a050 con 0x7f03dc159de0 2023-10-16T12:30:28.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.145+0000 7f03e2ed7700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03dc14a330 con 0x7f03dc06b000 2023-10-16T12:30:28.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.145+0000 7f03dbfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03dc0cf4c0 0x7f03dc150920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:28.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.145+0000 7f03dbfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03dc0cf4c0 0x7f03dc150920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45290/0 (socket says 172.21.15.16:45290) 2023-10-16T12:30:28.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.145+0000 7f03dbfff700 1 -- 172.21.15.16:0/1987858251 learned_addr learned my addr 172.21.15.16:0/1987858251 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:28.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.145+0000 7f03e1474700 1 --2- 172.21.15.16:0/1987858251 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f03dc159de0 0x7f03dc0ca750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:28.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.145+0000 7f03e0c73700 1 --2- 172.21.15.16:0/1987858251 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f03dc06b000 0x7f03dc158f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:28.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.145+0000 7f03dbfff700 1 -- 172.21.15.16:0/1987858251 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f03dc06b000 msgr2=0x7f03dc158f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:28.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.145+0000 7f03dbfff700 1 --2- 172.21.15.16:0/1987858251 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f03dc06b000 0x7f03dc158f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:28.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.145+0000 7f03dbfff700 1 -- 172.21.15.16:0/1987858251 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f03dc159de0 msgr2=0x7f03dc0ca750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:28.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.145+0000 7f03dbfff700 1 --2- 172.21.15.16:0/1987858251 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f03dc159de0 0x7f03dc0ca750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:28.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.145+0000 7f03dbfff700 1 -- 172.21.15.16:0/1987858251 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03d00094a0 con 0x7f03dc0cf4c0 2023-10-16T12:30:28.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.145+0000 7f03dbfff700 1 --2- 172.21.15.16:0/1987858251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03dc0cf4c0 0x7f03dc150920 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f03c8009f10 tx=0x7f03c800eaf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:28.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.145+0000 7f03d97fa700 1 -- 172.21.15.16:0/1987858251 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c8011920 con 0x7f03dc0cf4c0 2023-10-16T12:30:28.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.145+0000 7f03d97fa700 1 -- 172.21.15.16:0/1987858251 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03c8011a80 con 0x7f03dc0cf4c0 2023-10-16T12:30:28.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.145+0000 7f03e2ed7700 1 -- 172.21.15.16:0/1987858251 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03dc14a610 con 0x7f03dc0cf4c0 2023-10-16T12:30:28.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.145+0000 7f03d97fa700 1 -- 172.21.15.16:0/1987858251 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c8011e10 con 0x7f03dc0cf4c0 2023-10-16T12:30:28.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.145+0000 7f03e2ed7700 1 -- 172.21.15.16:0/1987858251 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03dc14b0f0 con 0x7f03dc0cf4c0 2023-10-16T12:30:28.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.146+0000 7f03e2ed7700 1 -- 172.21.15.16:0/1987858251 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03dc04ebb0 con 0x7f03dc0cf4c0 2023-10-16T12:30:28.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.147+0000 7f03d97fa700 1 -- 172.21.15.16:0/1987858251 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 74) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f03c8019440 con 0x7f03dc0cf4c0 2023-10-16T12:30:28.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.147+0000 7f03d97fa700 1 --2- 172.21.15.16:0/1987858251 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f03c006b020 0x7f03c006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:28.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.147+0000 7f03d97fa700 1 -- 172.21.15.16:0/1987858251 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 6085+0+0 (secure 0 0 0) 0x7f03c808b1c0 con 0x7f03dc0cf4c0 2023-10-16T12:30:28.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.147+0000 7f03e0c73700 1 --2- 172.21.15.16:0/1987858251 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f03c006b020 0x7f03c006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:28.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.148+0000 7f03e0c73700 1 --2- 172.21.15.16:0/1987858251 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f03c006b020 0x7f03c006d4e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f03d00090b0 tx=0x7f03d0008030 comp rx=0 tx=0).ready entity=mgr.8388 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:28.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.152+0000 7f03d97fa700 1 -- 172.21.15.16:0/1987858251 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f03c8053fe0 con 0x7f03dc0cf4c0 2023-10-16T12:30:28.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.437+0000 7f03e2ed7700 1 -- 172.21.15.16:0/1987858251 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f03dc0ce3d0 con 0x7f03dc0cf4c0 2023-10-16T12:30:28.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.438+0000 7f03d97fa700 1 -- 172.21.15.16:0/1987858251 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v25) v1 ==== 190+0+147 (secure 0 0 0) 0x7f03c801e0b0 con 0x7f03dc0cf4c0 2023-10-16T12:30:28.441 INFO:teuthology.orchestra.run.smithi016.stdout:[client.0] 2023-10-16T12:30:28.441 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAPKC1l2l6PFxAA9QyXtBAHdGAXpwH0JfknuQ== 2023-10-16T12:30:28.441 INFO:teuthology.orchestra.run.smithi016.stdout: caps mds = "allow" 2023-10-16T12:30:28.441 INFO:teuthology.orchestra.run.smithi016.stdout: caps mon = "allow r" 2023-10-16T12:30:28.441 INFO:teuthology.orchestra.run.smithi016.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-16T12:30:28.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.441+0000 7f03e2ed7700 1 -- 172.21.15.16:0/1987858251 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f03c006b020 msgr2=0x7f03c006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:28.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.441+0000 7f03e2ed7700 1 --2- 172.21.15.16:0/1987858251 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f03c006b020 0x7f03c006d4e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f03d00090b0 tx=0x7f03d0008030 comp rx=0 tx=0).stop 2023-10-16T12:30:28.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.441+0000 7f03e2ed7700 1 -- 172.21.15.16:0/1987858251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03dc0cf4c0 msgr2=0x7f03dc150920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:28.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.441+0000 7f03e2ed7700 1 --2- 172.21.15.16:0/1987858251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03dc0cf4c0 0x7f03dc150920 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f03c8009f10 tx=0x7f03c800eaf0 comp rx=0 tx=0).stop 2023-10-16T12:30:28.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.441+0000 7f03e2ed7700 1 -- 172.21.15.16:0/1987858251 shutdown_connections 2023-10-16T12:30:28.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.441+0000 7f03e2ed7700 1 --2- 172.21.15.16:0/1987858251 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f03c006b020 0x7f03c006d4e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:28.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.441+0000 7f03e2ed7700 1 --2- 172.21.15.16:0/1987858251 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f03dc159de0 0x7f03dc0ca750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:28.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.441+0000 7f03e2ed7700 1 --2- 172.21.15.16:0/1987858251 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f03dc06b000 0x7f03dc158f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:28.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.441+0000 7f03e2ed7700 1 --2- 172.21.15.16:0/1987858251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03dc0cf4c0 0x7f03dc150920 unknown :-1 s=CLOSED pgs=1555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:28.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.441+0000 7f03e2ed7700 1 -- 172.21.15.16:0/1987858251 >> 172.21.15.16:0/1987858251 conn(0x7f03dc0cc5e0 msgr2=0x7f03dc0cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:28.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.441+0000 7f03e2ed7700 1 -- 172.21.15.16:0/1987858251 shutdown_connections 2023-10-16T12:30:28.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.441+0000 7f03e2ed7700 1 -- 172.21.15.16:0/1987858251 wait complete. 2023-10-16T12:30:28.445 INFO:teuthology.orchestra.run.smithi016.stderr:updated caps for client.0 2023-10-16T12:30:28.455 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:30:28.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.579+0000 7f5972510700 1 Processor -- start 2023-10-16T12:30:28.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.579+0000 7f5972510700 1 -- start start 2023-10-16T12:30:28.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.580+0000 7f5972510700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f596c15b280 0x7f596c1613e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:28.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.580+0000 7f5972510700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f596c0cc8a0 0x7f596c15e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:28.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.580+0000 7f5972510700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f596c06a7e0 0x7f596c06ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:28.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.580+0000 7f5972510700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f596c06b190 con 0x7f596c0cc8a0 2023-10-16T12:30:28.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.580+0000 7f5972510700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f596c161c30 con 0x7f596c15b280 2023-10-16T12:30:28.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.580+0000 7f5972510700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f596c161f10 con 0x7f596c06a7e0 2023-10-16T12:30:28.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.580+0000 7f596b7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f596c0cc8a0 0x7f596c15e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:28.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.580+0000 7f596b7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f596c0cc8a0 0x7f596c15e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45296/0 (socket says 172.21.15.16:45296) 2023-10-16T12:30:28.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.580+0000 7f596b7fe700 1 -- 172.21.15.16:0/1221631848 learned_addr learned my addr 172.21.15.16:0/1221631848 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:28.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.580+0000 7f596bfff700 1 --2- 172.21.15.16:0/1221631848 >> v2:172.21.15.49:3300/0 conn(0x7f596c15b280 0x7f596c1613e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:28.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.580+0000 7f5970aad700 1 --2- 172.21.15.16:0/1221631848 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f596c06a7e0 0x7f596c06ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:28.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.580+0000 7f596b7fe700 1 -- 172.21.15.16:0/1221631848 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f596c06a7e0 msgr2=0x7f596c06ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:28.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.580+0000 7f596b7fe700 1 --2- 172.21.15.16:0/1221631848 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f596c06a7e0 0x7f596c06ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:28.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.580+0000 7f596b7fe700 1 -- 172.21.15.16:0/1221631848 >> v2:172.21.15.49:3300/0 conn(0x7f596c15b280 msgr2=0x7f596c1613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:28.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.580+0000 7f596b7fe700 1 --2- 172.21.15.16:0/1221631848 >> v2:172.21.15.49:3300/0 conn(0x7f596c15b280 0x7f596c1613e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:28.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.580+0000 7f596b7fe700 1 -- 172.21.15.16:0/1221631848 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f596c1621f0 con 0x7f596c0cc8a0 2023-10-16T12:30:28.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.580+0000 7f596b7fe700 1 --2- 172.21.15.16:0/1221631848 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f596c0cc8a0 0x7f596c15e700 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f596c06b8a0 tx=0x7f5960009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93bd0d4c727660c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:28.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.580+0000 7f596affd700 1 -- 172.21.15.16:0/1221631848 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5960011830 con 0x7f596c0cc8a0 2023-10-16T12:30:28.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.580+0000 7f596affd700 1 -- 172.21.15.16:0/1221631848 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5960011990 con 0x7f596c0cc8a0 2023-10-16T12:30:28.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.581+0000 7f596affd700 1 -- 172.21.15.16:0/1221631848 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5960011d20 con 0x7f596c0cc8a0 2023-10-16T12:30:28.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.581+0000 7f5972510700 1 -- 172.21.15.16:0/1221631848 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f596c0cc8a0 msgr2=0x7f596c15e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:28.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.581+0000 7f5972510700 1 --2- 172.21.15.16:0/1221631848 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f596c0cc8a0 0x7f596c15e700 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f596c06b8a0 tx=0x7f5960009bd0 comp rx=0 tx=0).stop 2023-10-16T12:30:28.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.581+0000 7f5972510700 1 -- 172.21.15.16:0/1221631848 shutdown_connections 2023-10-16T12:30:28.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.581+0000 7f5972510700 1 --2- 172.21.15.16:0/1221631848 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f596c06a7e0 0x7f596c06ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:28.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.581+0000 7f5972510700 1 --2- 172.21.15.16:0/1221631848 >> v2:172.21.15.49:3300/0 conn(0x7f596c15b280 0x7f596c1613e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:28.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.581+0000 7f5972510700 1 --2- 172.21.15.16:0/1221631848 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f596c0cc8a0 0x7f596c15e700 unknown :-1 s=CLOSED pgs=1556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:28.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.581+0000 7f5972510700 1 -- 172.21.15.16:0/1221631848 >> 172.21.15.16:0/1221631848 conn(0x7f596c0b1fd0 msgr2=0x7f596c0b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:28.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.581+0000 7f5972510700 1 -- 172.21.15.16:0/1221631848 shutdown_connections 2023-10-16T12:30:28.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.581+0000 7f5972510700 1 -- 172.21.15.16:0/1221631848 wait complete. 2023-10-16T12:30:28.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.583+0000 7f5972510700 1 Processor -- start 2023-10-16T12:30:28.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.583+0000 7f5972510700 1 -- start start 2023-10-16T12:30:28.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.584+0000 7f5972510700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f596c06a7e0 0x7f596c0c9eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:28.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.584+0000 7f5972510700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f596c0cc8a0 0x7f596c0c9710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:28.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.584+0000 7f5972510700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f596c15b280 0x7f596c15f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:28.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.584+0000 7f5972510700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f596c0c7510 con 0x7f596c06a7e0 2023-10-16T12:30:28.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.584+0000 7f5972510700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f596c160c70 con 0x7f596c0cc8a0 2023-10-16T12:30:28.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.584+0000 7f5972510700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f596c0c41f0 con 0x7f596c15b280 2023-10-16T12:30:28.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.584+0000 7f596bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f596c06a7e0 0x7f596c0c9eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:28.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.584+0000 7f596bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f596c06a7e0 0x7f596c0c9eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45302/0 (socket says 172.21.15.16:45302) 2023-10-16T12:30:28.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.584+0000 7f596bfff700 1 -- 172.21.15.16:0/3292692340 learned_addr learned my addr 172.21.15.16:0/3292692340 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:28.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.584+0000 7f596b7fe700 1 --2- 172.21.15.16:0/3292692340 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f596c0cc8a0 0x7f596c0c9710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:28.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.584+0000 7f5970aad700 1 --2- 172.21.15.16:0/3292692340 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f596c15b280 0x7f596c15f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:28.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.584+0000 7f596bfff700 1 -- 172.21.15.16:0/3292692340 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f596c15b280 msgr2=0x7f596c15f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:28.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.584+0000 7f596bfff700 1 --2- 172.21.15.16:0/3292692340 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f596c15b280 0x7f596c15f650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:28.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.584+0000 7f596bfff700 1 -- 172.21.15.16:0/3292692340 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f596c0cc8a0 msgr2=0x7f596c0c9710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:28.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.584+0000 7f596bfff700 1 --2- 172.21.15.16:0/3292692340 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f596c0cc8a0 0x7f596c0c9710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:28.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.584+0000 7f596bfff700 1 -- 172.21.15.16:0/3292692340 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5960009550 con 0x7f596c06a7e0 2023-10-16T12:30:28.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.584+0000 7f596bfff700 1 --2- 172.21.15.16:0/3292692340 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f596c06a7e0 0x7f596c0c9eb0 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7f595c00e870 tx=0x7f595c00ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:28.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.585+0000 7f5968ff9700 1 -- 172.21.15.16:0/3292692340 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f595c009590 con 0x7f596c06a7e0 2023-10-16T12:30:28.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.585+0000 7f5968ff9700 1 -- 172.21.15.16:0/3292692340 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f595c00ee70 con 0x7f596c06a7e0 2023-10-16T12:30:28.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.585+0000 7f5972510700 1 -- 172.21.15.16:0/3292692340 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f596c0c44d0 con 0x7f596c06a7e0 2023-10-16T12:30:28.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.585+0000 7f5968ff9700 1 -- 172.21.15.16:0/3292692340 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f595c00ca50 con 0x7f596c06a7e0 2023-10-16T12:30:28.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.585+0000 7f5972510700 1 -- 172.21.15.16:0/3292692340 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f596c0c7760 con 0x7f596c06a7e0 2023-10-16T12:30:28.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.586+0000 7f5972510700 1 -- 172.21.15.16:0/3292692340 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f596c0ce740 con 0x7f596c06a7e0 2023-10-16T12:30:28.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.586+0000 7f5968ff9700 1 -- 172.21.15.16:0/3292692340 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 74) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f595c00cbb0 con 0x7f596c06a7e0 2023-10-16T12:30:28.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.586+0000 7f5968ff9700 1 --2- 172.21.15.16:0/3292692340 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f595806b020 0x7f595806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:28.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.587+0000 7f5968ff9700 1 -- 172.21.15.16:0/3292692340 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 6085+0+0 (secure 0 0 0) 0x7f595c0898e0 con 0x7f596c06a7e0 2023-10-16T12:30:28.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.587+0000 7f596b7fe700 1 --2- 172.21.15.16:0/3292692340 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f595806b020 0x7f595806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:28.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.587+0000 7f596b7fe700 1 --2- 172.21.15.16:0/3292692340 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f595806b020 0x7f595806d4e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f596c06b8a0 tx=0x7f5960008030 comp rx=0 tx=0).ready entity=mgr.8388 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:28.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.591+0000 7f5968ff9700 1 -- 172.21.15.16:0/3292692340 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f595c054700 con 0x7f596c06a7e0 2023-10-16T12:30:28.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.839+0000 7f5972510700 1 -- 172.21.15.16:0/3292692340 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f596c0c7ad0 con 0x7f596c06a7e0 2023-10-16T12:30:28.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.839+0000 7f5968ff9700 1 -- 172.21.15.16:0/3292692340 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+4915 (secure 0 0 0) 0x7f595c017090 con 0x7f596c06a7e0 2023-10-16T12:30:28.842 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:30:28.842 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":70,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7220,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/4274593615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":4274593615},{"type":"v1","addr":"172.21.15.16:6837","nonce":4274593615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":8625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/1624531381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1624531381},{"type":"v1","addr":"172.21.15.16:6836","nonce":1624531381}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":8637,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/1629597454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":1629597454},{"type":"v1","addr":"172.21.15.49:6834","nonce":1629597454}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:30:22.807943+0000","modified":"2023-10-16T12:30:25.836865+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8640},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8640":{"gid":8640,"name":"b","rank":0,"incarnation":66,"state":"up:active","state_seq":3,"addr":"172.21.15.49:6835/2008024326","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":2008024326},{"type":"v1","addr":"172.21.15.49:6835","nonce":2008024326}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2023-10-16T12:30:28.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.841+0000 7f5972510700 1 -- 172.21.15.16:0/3292692340 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f595806b020 msgr2=0x7f595806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:28.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.841+0000 7f5972510700 1 --2- 172.21.15.16:0/3292692340 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f595806b020 0x7f595806d4e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f596c06b8a0 tx=0x7f5960008030 comp rx=0 tx=0).stop 2023-10-16T12:30:28.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.841+0000 7f5972510700 1 -- 172.21.15.16:0/3292692340 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f596c06a7e0 msgr2=0x7f596c0c9eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:28.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.841+0000 7f5972510700 1 --2- 172.21.15.16:0/3292692340 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f596c06a7e0 0x7f596c0c9eb0 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7f595c00e870 tx=0x7f595c00ec30 comp rx=0 tx=0).stop 2023-10-16T12:30:28.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.841+0000 7f5972510700 1 -- 172.21.15.16:0/3292692340 shutdown_connections 2023-10-16T12:30:28.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.841+0000 7f5972510700 1 --2- 172.21.15.16:0/3292692340 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f596c15b280 0x7f596c15f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:28.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.841+0000 7f5972510700 1 --2- 172.21.15.16:0/3292692340 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f596c06a7e0 0x7f596c0c9eb0 unknown :-1 s=CLOSED pgs=1557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:28.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.841+0000 7f5972510700 1 --2- 172.21.15.16:0/3292692340 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f595806b020 0x7f595806d4e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:28.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.841+0000 7f5972510700 1 --2- 172.21.15.16:0/3292692340 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f596c0cc8a0 0x7f596c0c9710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:28.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.841+0000 7f5972510700 1 -- 172.21.15.16:0/3292692340 >> 172.21.15.16:0/3292692340 conn(0x7f596c0b1fd0 msgr2=0x7f596c0cb690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:28.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.842+0000 7f5972510700 1 -- 172.21.15.16:0/3292692340 shutdown_connections 2023-10-16T12:30:28.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:28.842+0000 7f5972510700 1 -- 172.21.15.16:0/3292692340 wait complete. 2023-10-16T12:30:28.847 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 70 2023-10-16T12:30:29.258 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 70, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T12:30:22.807943+0000', 'modified': '2023-10-16T12:30:25.836865+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8640}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8640': {'gid': 8640, 'name': 'b', 'rank': 0, 'incarnation': 66, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.49:6835/2008024326', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6833', 'nonce': 2008024326}, {'type': 'v1', 'addr': '172.21.15.49:6835', 'nonce': 2008024326}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [18], 'metadata_pool': 17, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-16T12:30:29.259 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-16T12:30:29.259 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-10-16T12:30:29.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.388+0000 7f38712ef700 1 Processor -- start 2023-10-16T12:30:29.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.388+0000 7f38712ef700 1 -- start start 2023-10-16T12:30:29.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.389+0000 7f38712ef700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f386c0d41c0 0x7f386c0d0040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:29.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.389+0000 7f38712ef700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f386c0d0580 0x7f386c06c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:29.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.389+0000 7f38712ef700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f386c06ba00 con 0x7f386c15d8d0 2023-10-16T12:30:29.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.389+0000 7f38712ef700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f386c0d0a10 con 0x7f386c0d41c0 2023-10-16T12:30:29.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.389+0000 7f38712ef700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f386c0d26d0 con 0x7f386c0d0580 2023-10-16T12:30:29.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.389+0000 7f386affd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f386c15d8d0 0x7f386c0d1c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34584/0 (socket says 172.21.15.16:34584) 2023-10-16T12:30:29.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.389+0000 7f386affd700 1 -- 172.21.15.16:0/2221175652 learned_addr learned my addr 172.21.15.16:0/2221175652 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:30:29.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.389+0000 7f386a7fc700 1 --2- 172.21.15.16:0/2221175652 >> v2:172.21.15.49:3300/0 conn(0x7f386c0d41c0 0x7f386c0d0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:29.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.389+0000 7f3869ffb700 1 -- 172.21.15.16:0/2221175652 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 164719604 0 0) 0x7f386c06ba00 con 0x7f386c15d8d0 2023-10-16T12:30:29.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.389+0000 7f386b7fe700 1 --2- 172.21.15.16:0/2221175652 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f386c0d0580 0x7f386c06c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:29.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.389+0000 7f3869ffb700 1 -- 172.21.15.16:0/2221175652 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38540035b0 con 0x7f386c15d8d0 2023-10-16T12:30:29.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.389+0000 7f3869ffb700 1 -- 172.21.15.16:0/2221175652 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2499893688 0 0) 0x7f38540035b0 con 0x7f386c15d8d0 2023-10-16T12:30:29.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.389+0000 7f3869ffb700 1 -- 172.21.15.16:0/2221175652 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f386c0d0580 msgr2=0x7f386c06c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:29.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.389+0000 7f3869ffb700 1 --2- 172.21.15.16:0/2221175652 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f386c0d0580 0x7f386c06c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:29.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.389+0000 7f3869ffb700 1 -- 172.21.15.16:0/2221175652 >> v2:172.21.15.49:3300/0 conn(0x7f386c0d41c0 msgr2=0x7f386c0d0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:29.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.389+0000 7f3869ffb700 1 --2- 172.21.15.16:0/2221175652 >> v2:172.21.15.49:3300/0 conn(0x7f386c0d41c0 0x7f386c0d0040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:29.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.390+0000 7f3869ffb700 1 -- 172.21.15.16:0/2221175652 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f386c0d23f0 con 0x7f386c15d8d0 2023-10-16T12:30:29.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.390+0000 7f3869ffb700 1 -- 172.21.15.16:0/2221175652 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f385c001670 con 0x7f386c15d8d0 2023-10-16T12:30:29.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.390+0000 7f3869ffb700 1 -- 172.21.15.16:0/2221175652 >> v1:172.21.15.16:6789/0 conn(0x7f386c15d8d0 legacy=0x7f386c0d1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:29.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.390+0000 7f3869ffb700 1 --2- 172.21.15.16:0/2221175652 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3854003a00 0x7f3854003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:29.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.390+0000 7f3869ffb700 1 --2- 172.21.15.16:0/2221175652 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f386c0d0580 0x7f3854004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:29.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.390+0000 7f3869ffb700 1 --2- 172.21.15.16:0/2221175652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3854004d90 0x7f3854005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:29.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.390+0000 7f3869ffb700 1 -- 172.21.15.16:0/2221175652 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f386c0d0a10 con 0x7f3854004d90 2023-10-16T12:30:29.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.390+0000 7f3869ffb700 1 -- 172.21.15.16:0/2221175652 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f386c0d26d0 con 0x7f3854003a00 2023-10-16T12:30:29.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.390+0000 7f3869ffb700 1 -- 172.21.15.16:0/2221175652 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f385400ba10 con 0x7f386c0d0580 2023-10-16T12:30:29.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.390+0000 7f386b7fe700 1 --2- 172.21.15.16:0/2221175652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3854004d90 0x7f3854005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:29.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.390+0000 7f386affd700 1 --2- 172.21.15.16:0/2221175652 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3854003a00 0x7f3854003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:29.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.390+0000 7f386a7fc700 1 --2- 172.21.15.16:0/2221175652 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f386c0d0580 0x7f3854004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:29.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.390+0000 7f386b7fe700 1 -- 172.21.15.16:0/2221175652 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f386c0d0580 msgr2=0x7f3854004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:29.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.390+0000 7f386b7fe700 1 --2- 172.21.15.16:0/2221175652 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f386c0d0580 0x7f3854004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:29.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.390+0000 7f386b7fe700 1 -- 172.21.15.16:0/2221175652 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3854003a00 msgr2=0x7f3854003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:29.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.390+0000 7f386b7fe700 1 --2- 172.21.15.16:0/2221175652 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3854003a00 0x7f3854003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:29.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.390+0000 7f386b7fe700 1 -- 172.21.15.16:0/2221175652 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38540035b0 con 0x7f3854004d90 2023-10-16T12:30:29.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.390+0000 7f386b7fe700 1 --2- 172.21.15.16:0/2221175652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3854004d90 0x7f3854005210 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f386400e170 tx=0x7f3854005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ba242d2851b7388 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:29.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.390+0000 7f3869ffb700 1 -- 172.21.15.16:0/2221175652 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f386400ec90 con 0x7f3854004d90 2023-10-16T12:30:29.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.390+0000 7f3869ffb700 1 -- 172.21.15.16:0/2221175652 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f386400f040 con 0x7f3854004d90 2023-10-16T12:30:29.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.390+0000 7f3869ffb700 1 -- 172.21.15.16:0/2221175652 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3864014480 con 0x7f3854004d90 2023-10-16T12:30:29.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.391+0000 7f38712ef700 1 -- 172.21.15.16:0/2221175652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3854004d90 msgr2=0x7f3854005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:29.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.391+0000 7f38712ef700 1 --2- 172.21.15.16:0/2221175652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3854004d90 0x7f3854005210 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f386400e170 tx=0x7f3854005880 comp rx=0 tx=0).stop 2023-10-16T12:30:29.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.391+0000 7f386b7fe700 1 -- 172.21.15.16:0/2221175652 reap_dead start 2023-10-16T12:30:29.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.391+0000 7f38712ef700 1 -- 172.21.15.16:0/2221175652 shutdown_connections 2023-10-16T12:30:29.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.391+0000 7f38712ef700 1 -- 172.21.15.16:0/2221175652 >> 172.21.15.16:0/2221175652 conn(0x7f386c0d3630 msgr2=0x7f386c0d3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:29.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.391+0000 7f38712ef700 1 -- 172.21.15.16:0/2221175652 shutdown_connections 2023-10-16T12:30:29.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.391+0000 7f38712ef700 1 -- 172.21.15.16:0/2221175652 wait complete. 2023-10-16T12:30:29.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.392+0000 7f38712ef700 1 Processor -- start 2023-10-16T12:30:29.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.392+0000 7f38712ef700 1 -- start start 2023-10-16T12:30:29.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.393+0000 7f38712ef700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f386c15ae60 0x7f386c09e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:29.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.393+0000 7f38712ef700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f386c15eb40 0x7f386c15efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:29.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.393+0000 7f38712ef700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f386c1f65f0 0x7f386c1f6a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:29.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.393+0000 7f38712ef700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f386c09fe30 con 0x7f386c1f65f0 2023-10-16T12:30:29.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.393+0000 7f38712ef700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f386c15f500 con 0x7f386c15eb40 2023-10-16T12:30:29.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.393+0000 7f38712ef700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f386c0ca8c0 con 0x7f386c15ae60 2023-10-16T12:30:29.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.393+0000 7f386b7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f386c1f65f0 0x7f386c1f6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:29.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.393+0000 7f386b7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f386c1f65f0 0x7f386c1f6a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45326/0 (socket says 172.21.15.16:45326) 2023-10-16T12:30:29.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.393+0000 7f386b7fe700 1 -- 172.21.15.16:0/1342541884 learned_addr learned my addr 172.21.15.16:0/1342541884 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:29.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.393+0000 7f386a7fc700 1 --2- 172.21.15.16:0/1342541884 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f386c15eb40 0x7f386c15efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:29.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.393+0000 7f386affd700 1 --2- 172.21.15.16:0/1342541884 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f386c15ae60 0x7f386c09e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:29.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.394+0000 7f386b7fe700 1 -- 172.21.15.16:0/1342541884 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f386c15ae60 msgr2=0x7f386c09e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:29.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.394+0000 7f386b7fe700 1 --2- 172.21.15.16:0/1342541884 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f386c15ae60 0x7f386c09e310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:29.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.394+0000 7f386b7fe700 1 -- 172.21.15.16:0/1342541884 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f386c15eb40 msgr2=0x7f386c15efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:29.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.394+0000 7f386b7fe700 1 --2- 172.21.15.16:0/1342541884 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f386c15eb40 0x7f386c15efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:29.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.394+0000 7f386b7fe700 1 -- 172.21.15.16:0/1342541884 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f386c0caba0 con 0x7f386c1f65f0 2023-10-16T12:30:29.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.394+0000 7f386affd700 1 --2- 172.21.15.16:0/1342541884 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f386c15ae60 0x7f386c09e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:30:29.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.394+0000 7f386b7fe700 1 --2- 172.21.15.16:0/1342541884 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f386c1f65f0 0x7f386c1f6a70 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7f3854004710 tx=0x7f3864004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:29.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.394+0000 7f38537fe700 1 -- 172.21.15.16:0/1342541884 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3864007000 con 0x7f386c1f65f0 2023-10-16T12:30:29.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.394+0000 7f38537fe700 1 -- 172.21.15.16:0/1342541884 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f386400ee80 con 0x7f386c1f65f0 2023-10-16T12:30:29.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.394+0000 7f38712ef700 1 -- 172.21.15.16:0/1342541884 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f386c0cae80 con 0x7f386c1f65f0 2023-10-16T12:30:29.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.394+0000 7f38537fe700 1 -- 172.21.15.16:0/1342541884 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3864003a20 con 0x7f386c1f65f0 2023-10-16T12:30:29.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.394+0000 7f38712ef700 1 -- 172.21.15.16:0/1342541884 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f386c0cb440 con 0x7f386c1f65f0 2023-10-16T12:30:29.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.396+0000 7f38537fe700 1 -- 172.21.15.16:0/1342541884 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 74) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3864024020 con 0x7f386c1f65f0 2023-10-16T12:30:29.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.396+0000 7f38537fe700 1 --2- 172.21.15.16:0/1342541884 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f385806b050 0x7f385806d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:29.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.396+0000 7f38537fe700 1 -- 172.21.15.16:0/1342541884 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 6085+0+0 (secure 0 0 0) 0x7f386408b080 con 0x7f386c1f65f0 2023-10-16T12:30:29.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.396+0000 7f386affd700 1 --2- 172.21.15.16:0/1342541884 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f385806b050 0x7f385806d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:29.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.397+0000 7f386affd700 1 --2- 172.21.15.16:0/1342541884 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f385806b050 0x7f385806d510 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f385c0048d0 tx=0x7f385c009040 comp rx=0 tx=0).ready entity=mgr.8388 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:29.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.398+0000 7f387d4d9b80 1 Processor -- start 2023-10-16T12:30:29.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.398+0000 7f387d4d9b80 1 -- start start 2023-10-16T12:30:29.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.399+0000 7f387d4d9b80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55a283bfc220 0x55a283c03a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:29.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.399+0000 7f387d4d9b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a283c01630 0x55a283c03fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:29.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.399+0000 7f387d4d9b80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a283bd7010 0x55a283bd76c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:29.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.399+0000 7f387d4d9b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a283b4dc20 con 0x55a283c01630 2023-10-16T12:30:29.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.399+0000 7f387d4d9b80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a283b32c10 con 0x55a283bd7010 2023-10-16T12:30:29.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.399+0000 7f387d4d9b80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a283b30530 con 0x55a283bfc220 2023-10-16T12:30:29.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.399+0000 7f386a7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a283c01630 0x55a283c03fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:29.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.399+0000 7f386a7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a283c01630 0x55a283c03fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45332/0 (socket says 172.21.15.16:45332) 2023-10-16T12:30:29.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.399+0000 7f386a7fc700 1 -- 172.21.15.16:0/4244494656 learned_addr learned my addr 172.21.15.16:0/4244494656 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:29.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.399+0000 7f386b7fe700 1 --2- 172.21.15.16:0/4244494656 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a283bd7010 0x55a283bd76c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:29.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.399+0000 7f386affd700 1 --2- 172.21.15.16:0/4244494656 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55a283bfc220 0x55a283c03a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:29.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.399+0000 7f386a7fc700 1 -- 172.21.15.16:0/4244494656 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55a283bfc220 msgr2=0x55a283c03a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:29.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.399+0000 7f386a7fc700 1 --2- 172.21.15.16:0/4244494656 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55a283bfc220 0x55a283c03a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:29.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.399+0000 7f386a7fc700 1 -- 172.21.15.16:0/4244494656 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a283bd7010 msgr2=0x55a283bd76c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:29.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.399+0000 7f386a7fc700 1 --2- 172.21.15.16:0/4244494656 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a283bd7010 0x55a283bd76c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:29.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.399+0000 7f386a7fc700 1 -- 172.21.15.16:0/4244494656 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a283ae93a0 con 0x55a283c01630 2023-10-16T12:30:29.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.400+0000 7f386a7fc700 1 --2- 172.21.15.16:0/4244494656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a283c01630 0x55a283c03fb0 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f3860009db0 tx=0x7f3860009de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd0c5ae8781a0fd0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:29.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.400+0000 7f3850ff9700 1 -- 172.21.15.16:0/4244494656 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3860014ab0 con 0x55a283c01630 2023-10-16T12:30:29.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.400+0000 7f3850ff9700 1 -- 172.21.15.16:0/4244494656 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3860014c10 con 0x55a283c01630 2023-10-16T12:30:29.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.400+0000 7f3850ff9700 1 -- 172.21.15.16:0/4244494656 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f386001d440 con 0x55a283c01630 2023-10-16T12:30:29.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.400+0000 7f387d4d9b80 1 -- 172.21.15.16:0/4244494656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a283c01630 msgr2=0x55a283c03fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:29.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.400+0000 7f387d4d9b80 1 --2- 172.21.15.16:0/4244494656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a283c01630 0x55a283c03fb0 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f3860009db0 tx=0x7f3860009de0 comp rx=0 tx=0).stop 2023-10-16T12:30:29.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.400+0000 7f387d4d9b80 1 -- 172.21.15.16:0/4244494656 shutdown_connections 2023-10-16T12:30:29.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.400+0000 7f387d4d9b80 1 --2- 172.21.15.16:0/4244494656 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a283bd7010 0x55a283bd76c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:29.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.400+0000 7f387d4d9b80 1 --2- 172.21.15.16:0/4244494656 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55a283bfc220 0x55a283c03a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:29.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.400+0000 7f387d4d9b80 1 --2- 172.21.15.16:0/4244494656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a283c01630 0x55a283c03fb0 unknown :-1 s=CLOSED pgs=1561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:29.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.400+0000 7f387d4d9b80 1 -- 172.21.15.16:0/4244494656 >> 172.21.15.16:0/4244494656 conn(0x55a283b06d80 msgr2=0x55a283c21ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:29.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.401+0000 7f387d4d9b80 1 -- 172.21.15.16:0/4244494656 shutdown_connections 2023-10-16T12:30:29.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.401+0000 7f387d4d9b80 1 -- 172.21.15.16:0/4244494656 wait complete. 2023-10-16T12:30:29.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.401+0000 7f387d4d9b80 1 -- start start 2023-10-16T12:30:29.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.401+0000 7f387d4d9b80 1 Processor -- start 2023-10-16T12:30:29.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.402+0000 7f387d4d9b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a283bd7010 0x55a283bfb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:29.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.402+0000 7f387d4d9b80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55a283bfc220 0x55a283bfcf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:29.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.402+0000 7f387d4d9b80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a283c01630 0x55a283bfd4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:29.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.402+0000 7f387d4d9b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a283b2be20 con 0x55a283bd7010 2023-10-16T12:30:29.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.402+0000 7f387d4d9b80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a283ac86c0 con 0x55a283c01630 2023-10-16T12:30:29.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.402+0000 7f387d4d9b80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a283ac8360 con 0x55a283bfc220 2023-10-16T12:30:29.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.402+0000 7f387d4d9b80 10 client.0 _ll_register_callbacks cb 0x55a2839257b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T12:30:29.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.402+0000 7f386affd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a283bd7010 0x55a283bfb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:29.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.402+0000 7f386affd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a283bd7010 0x55a283bfb1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45336/0 (socket says 172.21.15.16:45336) 2023-10-16T12:30:29.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.402+0000 7f386affd700 1 -- 172.21.15.16:0/122694655 learned_addr learned my addr 172.21.15.16:0/122694655 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:29.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.402+0000 7f386affd700 1 -- 172.21.15.16:0/122694655 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55a283bfc220 msgr2=0x55a283bfcf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:29.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.402+0000 7f386b7fe700 1 --2- 172.21.15.16:0/122694655 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a283c01630 0x55a283bfd4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:29.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.402+0000 7f386affd700 1 --2- 172.21.15.16:0/122694655 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55a283bfc220 0x55a283bfcf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:29.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.402+0000 7f386affd700 1 -- 172.21.15.16:0/122694655 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a283c01630 msgr2=0x55a283bfd4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:29.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.402+0000 7f386affd700 1 --2- 172.21.15.16:0/122694655 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a283c01630 0x55a283bfd4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:29.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.402+0000 7f386affd700 1 -- 172.21.15.16:0/122694655 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38600099e0 con 0x55a283bd7010 2023-10-16T12:30:29.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.403+0000 7f386affd700 1 --2- 172.21.15.16:0/122694655 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a283bd7010 0x55a283bfb1d0 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f385c015a60 tx=0x7f385c000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77dfdf67b60a8f45 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:29.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.403+0000 7f3849ffb700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2023-10-16T12:30:29.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.403+0000 7f3849ffb700 1 -- 172.21.15.16:0/122694655 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f385c007810 con 0x55a283bd7010 2023-10-16T12:30:29.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.403+0000 7f3849ffb700 1 -- 172.21.15.16:0/122694655 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(142..142 src has 1..142) v4 ==== 6085+0+0 (secure 0 0 0) 0x7f385c01a040 con 0x55a283bd7010 2023-10-16T12:30:29.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.403+0000 7f387d4d9b80 1 -- 172.21.15.16:0/122694655 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a283bcc5b0 con 0x55a283bd7010 2023-10-16T12:30:29.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.403+0000 7f3849ffb700 1 -- 172.21.15.16:0/122694655 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=70) v2 ==== 24+0+0 (secure 0 0 0) 0x7f385c014a70 con 0x55a283bd7010 2023-10-16T12:30:29.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.403+0000 7f387d4d9b80 10 client.8775 fetch_fsmap learned FSMap version 70 2023-10-16T12:30:29.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.403+0000 7f387d4d9b80 1 -- 172.21.15.16:0/122694655 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=70,monmap=2+}) v3 -- 0x55a283956d90 con 0x55a283bd7010 2023-10-16T12:30:29.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.403+0000 7f3849ffb700 1 -- 172.21.15.16:0/122694655 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55a283bcc5b0 con 0x55a283bd7010 2023-10-16T12:30:29.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.403+0000 7f3849ffb700 1 -- 172.21.15.16:0/122694655 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 70) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f385c014bd0 con 0x55a283bd7010 2023-10-16T12:30:29.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.403+0000 7f387d4d9b80 10 client.8775 fetch_fsmap finished waiting for FSMap version 70 2023-10-16T12:30:29.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.403+0000 7f387d4d9b80 10 client.8775 resolve_mds: resolved name 'b' to daemon mds.b 2023-10-16T12:30:29.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.403+0000 7f387d4d9b80 20 client.8775 populate_metadata read hostname 'smithi016' 2023-10-16T12:30:29.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.403+0000 7f387d4d9b80 1 --2- 172.21.15.16:0/122694655 >> [v2:172.21.15.49:6833/2008024326,v1:172.21.15.49:6835/2008024326] conn(0x55a283bce430 0x55a283c04e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:29.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.403+0000 7f387d4d9b80 4 client.8775 mds_command: new command op to 8640 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T12:30:29.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.403+0000 7f387d4d9b80 1 -- 172.21.15.16:0/122694655 --> [v2:172.21.15.49:6833/2008024326,v1:172.21.15.49:6835/2008024326] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a283b24e00 con 0x55a283bce430 2023-10-16T12:30:29.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.403+0000 7f386a7fc700 1 --2- 172.21.15.16:0/122694655 >> [v2:172.21.15.49:6833/2008024326,v1:172.21.15.49:6835/2008024326] conn(0x55a283bce430 0x55a283c04e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:29.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.404+0000 7f386a7fc700 1 --2- 172.21.15.16:0/122694655 >> [v2:172.21.15.49:6833/2008024326,v1:172.21.15.49:6835/2008024326] conn(0x55a283bce430 0x55a283c04e90 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca780397708af077 server_cookie=caec32c606455fdf in_seq=0 out_seq=0 2023-10-16T12:30:29.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.404+0000 7f3849ffb700 10 client.8775 ms_handle_connect on v2:172.21.15.49:6833/2008024326 2023-10-16T12:30:29.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.404+0000 7f3849ffb700 1 -- 172.21.15.16:0/122694655 <== mds.0 v2:172.21.15.49:6833/2008024326 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55a283b24e00 con 0x55a283bce430 2023-10-16T12:30:29.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.404+0000 7f3849ffb700 10 client.8775 handle_command_reply: tid=0 2023-10-16T12:30:29.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.405+0000 7f387d4d9b80 1 client.8775 shutdown 2023-10-16T12:30:29.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.405+0000 7f387d4d9b80 1 -- 172.21.15.16:0/122694655 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a283bd7010 msgr2=0x55a283bfb1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:29.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.405+0000 7f387d4d9b80 1 --2- 172.21.15.16:0/122694655 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a283bd7010 0x55a283bfb1d0 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f385c015a60 tx=0x7f385c000fe0 comp rx=0 tx=0).stop 2023-10-16T12:30:29.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.405+0000 7f387d4d9b80 1 -- 172.21.15.16:0/122694655 shutdown_connections 2023-10-16T12:30:29.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.405+0000 7f387d4d9b80 1 --2- 172.21.15.16:0/122694655 >> [v2:172.21.15.49:6833/2008024326,v1:172.21.15.49:6835/2008024326] conn(0x55a283bce430 0x55a283c04e90 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:29.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.405+0000 7f387d4d9b80 1 --2- 172.21.15.16:0/122694655 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a283c01630 0x55a283bfd4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:29.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.405+0000 7f387d4d9b80 1 --2- 172.21.15.16:0/122694655 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a283bd7010 0x55a283bfb1d0 unknown :-1 s=CLOSED pgs=1562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:29.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.405+0000 7f387d4d9b80 1 --2- 172.21.15.16:0/122694655 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55a283bfc220 0x55a283bfcf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:29.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.405+0000 7f3849ffb700 0 client.8775 ms_handle_reset on v2:172.21.15.49:6833/2008024326 2023-10-16T12:30:29.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.405+0000 7f387d4d9b80 1 -- 172.21.15.16:0/122694655 >> 172.21.15.16:0/122694655 conn(0x55a283b06d80 msgr2=0x55a283c1f520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:29.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.405+0000 7f387d4d9b80 1 -- 172.21.15.16:0/122694655 shutdown_connections 2023-10-16T12:30:29.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.405+0000 7f387d4d9b80 1 -- 172.21.15.16:0/122694655 wait complete. 2023-10-16T12:30:29.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.405+0000 7f387d4d9b80 20 client.8775 trim_cache size 0 max 16384 2023-10-16T12:30:29.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.423+0000 7f387d4d9b80 1 Processor -- start 2023-10-16T12:30:29.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.423+0000 7f387d4d9b80 1 -- start start 2023-10-16T12:30:29.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.423+0000 7f387d4d9b80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55a283bd7010 0x55a283bd5b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:29.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.423+0000 7f387d4d9b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a283bfc220 0x55a283bfb870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:29.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.423+0000 7f387d4d9b80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a283c01630 0x55a283bd73f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:29.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.423+0000 7f387d4d9b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a2839ac8d0 con 0x55a283bfc220 2023-10-16T12:30:29.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.423+0000 7f387d4d9b80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a283c22e50 con 0x55a283c01630 2023-10-16T12:30:29.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.423+0000 7f387d4d9b80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a283c05500 con 0x55a283bd7010 2023-10-16T12:30:29.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.423+0000 7f386a7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a283bfc220 0x55a283bfb870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:29.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.423+0000 7f386a7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a283bfc220 0x55a283bfb870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45346/0 (socket says 172.21.15.16:45346) 2023-10-16T12:30:29.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.423+0000 7f386a7fc700 1 -- 172.21.15.16:0/1073772448 learned_addr learned my addr 172.21.15.16:0/1073772448 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:29.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.424+0000 7f386affd700 1 --2- 172.21.15.16:0/1073772448 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55a283bd7010 0x55a283bd5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:29.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.424+0000 7f386b7fe700 1 --2- 172.21.15.16:0/1073772448 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a283c01630 0x55a283bd73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:29.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.424+0000 7f386a7fc700 1 -- 172.21.15.16:0/1073772448 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55a283bd7010 msgr2=0x55a283bd5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:29.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.424+0000 7f386a7fc700 1 --2- 172.21.15.16:0/1073772448 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55a283bd7010 0x55a283bd5b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:29.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.424+0000 7f386a7fc700 1 -- 172.21.15.16:0/1073772448 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a283c01630 msgr2=0x55a283bd73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:29.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.424+0000 7f386a7fc700 1 --2- 172.21.15.16:0/1073772448 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a283c01630 0x55a283bd73f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:29.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.424+0000 7f386a7fc700 1 -- 172.21.15.16:0/1073772448 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a28393fc10 con 0x55a283bfc220 2023-10-16T12:30:29.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.424+0000 7f386b7fe700 1 --2- 172.21.15.16:0/1073772448 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a283c01630 0x55a283bd73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:30:29.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.424+0000 7f386a7fc700 1 --2- 172.21.15.16:0/1073772448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a283bfc220 0x55a283bfb870 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f3860020040 tx=0x7f3860017070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53288bfec95d509c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:29.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.425+0000 7f38517fa700 1 -- 172.21.15.16:0/1073772448 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3860028be0 con 0x55a283bfc220 2023-10-16T12:30:29.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.425+0000 7f38517fa700 1 -- 172.21.15.16:0/1073772448 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3860028d40 con 0x55a283bfc220 2023-10-16T12:30:29.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.425+0000 7f38517fa700 1 -- 172.21.15.16:0/1073772448 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38600282d0 con 0x55a283bfc220 2023-10-16T12:30:29.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.426+0000 7f387d4d9b80 1 -- 172.21.15.16:0/1073772448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a283bfc220 msgr2=0x55a283bfb870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:29.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.426+0000 7f387d4d9b80 1 --2- 172.21.15.16:0/1073772448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a283bfc220 0x55a283bfb870 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f3860020040 tx=0x7f3860017070 comp rx=0 tx=0).stop 2023-10-16T12:30:29.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.426+0000 7f387d4d9b80 1 -- 172.21.15.16:0/1073772448 shutdown_connections 2023-10-16T12:30:29.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.426+0000 7f387d4d9b80 1 --2- 172.21.15.16:0/1073772448 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a283c01630 0x55a283bd73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:29.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.426+0000 7f387d4d9b80 1 --2- 172.21.15.16:0/1073772448 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55a283bd7010 0x55a283bd5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:29.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.426+0000 7f387d4d9b80 1 --2- 172.21.15.16:0/1073772448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a283bfc220 0x55a283bfb870 unknown :-1 s=CLOSED pgs=1563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:29.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.426+0000 7f387d4d9b80 1 -- 172.21.15.16:0/1073772448 >> 172.21.15.16:0/1073772448 conn(0x55a283b06d80 msgr2=0x55a283bfafe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:29.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.426+0000 7f387d4d9b80 1 -- 172.21.15.16:0/1073772448 shutdown_connections 2023-10-16T12:30:29.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.426+0000 7f387d4d9b80 1 -- 172.21.15.16:0/1073772448 wait complete. 2023-10-16T12:30:29.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.426+0000 7f387d4d9b80 1 -- start start 2023-10-16T12:30:29.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.426+0000 7f387d4d9b80 1 Processor -- start 2023-10-16T12:30:29.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.427+0000 7f387d4d9b80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55a283bd7010 0x55a283c6f190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:29.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.427+0000 7f387d4d9b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a283bfc220 0x55a283c86010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:29.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.427+0000 7f387d4d9b80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a283c01630 0x55a283bd73f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:29.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.427+0000 7f387d4d9b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a283c1d730 con 0x55a283bfc220 2023-10-16T12:30:29.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.427+0000 7f387d4d9b80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a283bd89e0 con 0x55a283c01630 2023-10-16T12:30:29.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.427+0000 7f387d4d9b80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a283c868f0 con 0x55a283bd7010 2023-10-16T12:30:29.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.427+0000 7f387d4d9b80 10 client.0 _ll_register_callbacks cb 0x55a2839257b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T12:30:29.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.427+0000 7f386a7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a283bfc220 0x55a283c86010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:29.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.427+0000 7f386a7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a283bfc220 0x55a283c86010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45352/0 (socket says 172.21.15.16:45352) 2023-10-16T12:30:29.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.427+0000 7f386a7fc700 1 -- 172.21.15.16:0/4250162931 learned_addr learned my addr 172.21.15.16:0/4250162931 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:29.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.427+0000 7f386affd700 1 --2- 172.21.15.16:0/4250162931 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55a283bd7010 0x55a283c6f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:29.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.427+0000 7f386b7fe700 1 --2- 172.21.15.16:0/4250162931 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a283c01630 0x55a283bd73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:29.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.428+0000 7f386a7fc700 1 -- 172.21.15.16:0/4250162931 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55a283bd7010 msgr2=0x55a283c6f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:29.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.428+0000 7f386a7fc700 1 --2- 172.21.15.16:0/4250162931 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55a283bd7010 0x55a283c6f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:29.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.428+0000 7f386a7fc700 1 -- 172.21.15.16:0/4250162931 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a283c01630 msgr2=0x55a283bd73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:29.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.428+0000 7f386a7fc700 1 --2- 172.21.15.16:0/4250162931 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a283c01630 0x55a283bd73f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:29.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.428+0000 7f386a7fc700 1 -- 172.21.15.16:0/4250162931 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a283ae93a0 con 0x55a283bfc220 2023-10-16T12:30:29.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.428+0000 7f386affd700 1 --2- 172.21.15.16:0/4250162931 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55a283bd7010 0x55a283c6f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:30:29.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.428+0000 7f386a7fc700 1 --2- 172.21.15.16:0/4250162931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a283bfc220 0x55a283c86010 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f3860009250 tx=0x7f385c014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5218d6a87142ddd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:29.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.428+0000 7f3849ffb700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2023-10-16T12:30:29.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.428+0000 7f3849ffb700 1 -- 172.21.15.16:0/4250162931 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38600282d0 con 0x55a283bfc220 2023-10-16T12:30:29.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.428+0000 7f3849ffb700 1 -- 172.21.15.16:0/4250162931 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(142..142 src has 1..142) v4 ==== 6085+0+0 (secure 0 0 0) 0x7f3860028430 con 0x55a283bfc220 2023-10-16T12:30:29.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.428+0000 7f387d4d9b80 1 -- 172.21.15.16:0/4250162931 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3860014c10 con 0x55a283bfc220 2023-10-16T12:30:29.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.428+0000 7f3849ffb700 1 -- 172.21.15.16:0/4250162931 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=70) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3860004040 con 0x55a283bfc220 2023-10-16T12:30:29.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.428+0000 7f387d4d9b80 10 client.8781 fetch_fsmap learned FSMap version 70 2023-10-16T12:30:29.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.428+0000 7f387d4d9b80 1 -- 172.21.15.16:0/4250162931 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=70,monmap=2+}) v3 -- 0x55a28397d200 con 0x55a283bfc220 2023-10-16T12:30:29.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.428+0000 7f3849ffb700 1 -- 172.21.15.16:0/4250162931 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3860014c10 con 0x55a283bfc220 2023-10-16T12:30:29.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.428+0000 7f3849ffb700 1 -- 172.21.15.16:0/4250162931 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 70) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f386002f070 con 0x55a283bfc220 2023-10-16T12:30:29.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.428+0000 7f387d4d9b80 10 client.8781 fetch_fsmap finished waiting for FSMap version 70 2023-10-16T12:30:29.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.428+0000 7f387d4d9b80 10 client.8781 resolve_mds: resolved name 'b' to daemon mds.b 2023-10-16T12:30:29.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.428+0000 7f387d4d9b80 20 client.8781 populate_metadata read hostname 'smithi016' 2023-10-16T12:30:29.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.428+0000 7f387d4d9b80 1 --2- 172.21.15.16:0/4250162931 >> [v2:172.21.15.49:6833/2008024326,v1:172.21.15.49:6835/2008024326] conn(0x55a283bf9960 0x55a283bf9d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:29.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.428+0000 7f387d4d9b80 4 client.8781 mds_command: new command op to 8640 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T12:30:29.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.428+0000 7f387d4d9b80 1 -- 172.21.15.16:0/4250162931 --> [v2:172.21.15.49:6833/2008024326,v1:172.21.15.49:6835/2008024326] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55a283b24b20 con 0x55a283bf9960 2023-10-16T12:30:29.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.429+0000 7f386affd700 1 --2- 172.21.15.16:0/4250162931 >> [v2:172.21.15.49:6833/2008024326,v1:172.21.15.49:6835/2008024326] conn(0x55a283bf9960 0x55a283bf9d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:29.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.429+0000 7f3849ffb700 10 client.8781 ms_handle_connect on v2:172.21.15.49:6833/2008024326 2023-10-16T12:30:29.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.429+0000 7f386affd700 1 --2- 172.21.15.16:0/4250162931 >> [v2:172.21.15.49:6833/2008024326,v1:172.21.15.49:6835/2008024326] conn(0x55a283bf9960 0x55a283bf9d40 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9f999996bc0119e7 server_cookie=3cc8c7e0d052abb0 in_seq=0 out_seq=0 2023-10-16T12:30:29.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.429+0000 7f3849ffb700 1 -- 172.21.15.16:0/4250162931 <== mds.0 v2:172.21.15.49:6833/2008024326 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x55a283b24b20 con 0x55a283bf9960 2023-10-16T12:30:29.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.429+0000 7f3849ffb700 10 client.8781 handle_command_reply: tid=0 2023-10-16T12:30:29.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.429+0000 7f387d4d9b80 1 client.8781 shutdown 2023-10-16T12:30:29.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.429+0000 7f387d4d9b80 1 -- 172.21.15.16:0/4250162931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a283bfc220 msgr2=0x55a283c86010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:29.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.429+0000 7f387d4d9b80 1 --2- 172.21.15.16:0/4250162931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a283bfc220 0x55a283c86010 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f3860009250 tx=0x7f385c014000 comp rx=0 tx=0).stop 2023-10-16T12:30:29.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.429+0000 7f387d4d9b80 1 -- 172.21.15.16:0/4250162931 shutdown_connections 2023-10-16T12:30:29.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.429+0000 7f387d4d9b80 1 --2- 172.21.15.16:0/4250162931 >> [v2:172.21.15.49:6833/2008024326,v1:172.21.15.49:6835/2008024326] conn(0x55a283bf9960 0x55a283bf9d40 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:29.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.429+0000 7f387d4d9b80 1 --2- 172.21.15.16:0/4250162931 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a283c01630 0x55a283bd73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:29.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.429+0000 7f3849ffb700 0 client.8781 ms_handle_reset on v2:172.21.15.49:6833/2008024326 2023-10-16T12:30:29.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.429+0000 7f387d4d9b80 1 --2- 172.21.15.16:0/4250162931 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55a283bd7010 0x55a283c6f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:29.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.429+0000 7f387d4d9b80 1 --2- 172.21.15.16:0/4250162931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a283bfc220 0x55a283c86010 unknown :-1 s=CLOSED pgs=1564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:29.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.429+0000 7f387d4d9b80 1 -- 172.21.15.16:0/4250162931 >> 172.21.15.16:0/4250162931 conn(0x55a283b06d80 msgr2=0x55a283c2d480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:29.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.429+0000 7f387d4d9b80 1 -- 172.21.15.16:0/4250162931 shutdown_connections 2023-10-16T12:30:29.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.429+0000 7f387d4d9b80 1 -- 172.21.15.16:0/4250162931 wait complete. 2023-10-16T12:30:29.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.430+0000 7f387d4d9b80 20 client.8781 trim_cache size 0 max 16384 2023-10-16T12:30:29.461 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-10-16T12:30:29.461 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "5b28dfa1-b21f-474a-9b2e-9fb77af58254", 2023-10-16T12:30:29.461 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 0, 2023-10-16T12:30:29.461 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 8640, 2023-10-16T12:30:29.462 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2023-10-16T12:30:29.462 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2023-10-16T12:30:29.462 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2023-10-16T12:30:29.462 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 6.0244763150000002, 2023-10-16T12:30:29.462 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 70, 2023-10-16T12:30:29.463 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 142, 2023-10-16T12:30:29.463 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 142, 2023-10-16T12:30:29.463 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 10.925184923 2023-10-16T12:30:29.463 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-10-16T12:30:29.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.430+0000 7f38712ef700 1 -- 172.21.15.16:0/1342541884 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f385806b050 msgr2=0x7f385806d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:29.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.430+0000 7f38712ef700 1 --2- 172.21.15.16:0/1342541884 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f385806b050 0x7f385806d510 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f385c0048d0 tx=0x7f385c009040 comp rx=0 tx=0).stop 2023-10-16T12:30:29.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.430+0000 7f38712ef700 1 -- 172.21.15.16:0/1342541884 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f386c1f65f0 msgr2=0x7f386c1f6a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:29.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.430+0000 7f38712ef700 1 --2- 172.21.15.16:0/1342541884 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f386c1f65f0 0x7f386c1f6a70 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7f3854004710 tx=0x7f3864004b40 comp rx=0 tx=0).stop 2023-10-16T12:30:29.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.430+0000 7f38712ef700 1 -- 172.21.15.16:0/1342541884 shutdown_connections 2023-10-16T12:30:29.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.430+0000 7f38712ef700 1 --2- 172.21.15.16:0/1342541884 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f386c1f65f0 0x7f386c1f6a70 unknown :-1 s=CLOSED pgs=1560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:29.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.430+0000 7f38712ef700 1 --2- 172.21.15.16:0/1342541884 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f386c15ae60 0x7f386c09e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:29.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.430+0000 7f38712ef700 1 --2- 172.21.15.16:0/1342541884 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f385806b050 0x7f385806d510 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:29.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.430+0000 7f38712ef700 1 --2- 172.21.15.16:0/1342541884 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f386c15eb40 0x7f386c15efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:29.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.430+0000 7f38712ef700 1 -- 172.21.15.16:0/1342541884 >> 172.21.15.16:0/1342541884 conn(0x7f386c0d3630 msgr2=0x7f386c0b18f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:29.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.430+0000 7f38712ef700 1 -- 172.21.15.16:0/1342541884 shutdown_connections 2023-10-16T12:30:29.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.430+0000 7f38712ef700 1 -- 172.21.15.16:0/1342541884 wait complete. 2023-10-16T12:30:29.467 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T12:30:29.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.591+0000 7f9df4971700 1 Processor -- start 2023-10-16T12:30:29.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.591+0000 7f9df4971700 1 -- start start 2023-10-16T12:30:29.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.592+0000 7f9df4971700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9df015b9f0 0x7f9df00b29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:29.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.592+0000 7f9df4971700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9df00c3490 con 0x7f9df00cf280 2023-10-16T12:30:29.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.592+0000 7f9df4971700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9df00cf730 con 0x7f9df006d940 2023-10-16T12:30:29.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.592+0000 7f9df4971700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9df00cdad0 con 0x7f9df015b9f0 2023-10-16T12:30:29.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.592+0000 7f9dee59c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9df015b9f0 0x7f9df00b29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:29.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.592+0000 7f9deed9d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f9df00cf280 0x7f9df006b7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34636/0 (socket says 172.21.15.16:34636) 2023-10-16T12:30:29.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.592+0000 7f9deed9d700 1 -- 172.21.15.16:0/595116568 learned_addr learned my addr 172.21.15.16:0/595116568 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:30:29.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.593+0000 7f9dee59c700 1 -- 172.21.15.16:0/595116568 >> v1:172.21.15.49:6789/0 conn(0x7f9df006d940 legacy=0x7f9df00ceb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:29.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.593+0000 7f9dee59c700 1 -- 172.21.15.16:0/595116568 >> v1:172.21.15.16:6789/0 conn(0x7f9df00cf280 legacy=0x7f9df006b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:29.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.593+0000 7f9dee59c700 1 -- 172.21.15.16:0/595116568 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9df00cddb0 con 0x7f9df015b9f0 2023-10-16T12:30:29.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.593+0000 7f9ded59a700 1 -- 172.21.15.16:0/595116568 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 632291960 0 0) 0x7f9df00c3490 con 0x7f9df00cf280 2023-10-16T12:30:29.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.593+0000 7f9dee59c700 1 --2- 172.21.15.16:0/595116568 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9df015b9f0 0x7f9df00b29a0 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f9de0009840 tx=0x7f9de0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=245b98b3619e6555 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:29.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.593+0000 7f9ded59a700 1 -- 172.21.15.16:0/595116568 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de0011830 con 0x7f9df015b9f0 2023-10-16T12:30:29.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.593+0000 7f9ded59a700 1 -- 172.21.15.16:0/595116568 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9de0011990 con 0x7f9df015b9f0 2023-10-16T12:30:29.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.593+0000 7f9ded59a700 1 -- 172.21.15.16:0/595116568 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de0011d20 con 0x7f9df015b9f0 2023-10-16T12:30:29.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.594+0000 7f9df4971700 1 -- 172.21.15.16:0/595116568 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9df015b9f0 msgr2=0x7f9df00b29a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:29.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.594+0000 7f9df4971700 1 --2- 172.21.15.16:0/595116568 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9df015b9f0 0x7f9df00b29a0 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f9de0009840 tx=0x7f9de0009b50 comp rx=0 tx=0).stop 2023-10-16T12:30:29.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.594+0000 7f9df4971700 1 -- 172.21.15.16:0/595116568 shutdown_connections 2023-10-16T12:30:29.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.594+0000 7f9df4971700 1 --2- 172.21.15.16:0/595116568 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9df015b9f0 0x7f9df00b29a0 unknown :-1 s=CLOSED pgs=1566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:29.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.594+0000 7f9df4971700 1 -- 172.21.15.16:0/595116568 >> 172.21.15.16:0/595116568 conn(0x7f9df00cc5b0 msgr2=0x7f9df00cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:29.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.594+0000 7f9df4971700 1 -- 172.21.15.16:0/595116568 shutdown_connections 2023-10-16T12:30:29.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.594+0000 7f9df4971700 1 -- 172.21.15.16:0/595116568 wait complete. 2023-10-16T12:30:29.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.595+0000 7f9df4971700 1 Processor -- start 2023-10-16T12:30:29.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.595+0000 7f9df4971700 1 -- start start 2023-10-16T12:30:29.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.596+0000 7f9df4971700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9df006d940 0x7f9df015e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:29.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.596+0000 7f9df4971700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9df00cf280 0x7f9df015cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:29.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.596+0000 7f9df4971700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9df015d240 0x7f9df00c3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:29.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.596+0000 7f9df4971700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df00c9be0 con 0x7f9df015d240 2023-10-16T12:30:29.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.596+0000 7f9df4971700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df015bf40 con 0x7f9df006d940 2023-10-16T12:30:29.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.596+0000 7f9df4971700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9df015c220 con 0x7f9df00cf280 2023-10-16T12:30:29.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.596+0000 7f9deed9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9df015d240 0x7f9df00c3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:29.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.596+0000 7f9deed9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9df015d240 0x7f9df00c3670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45368/0 (socket says 172.21.15.16:45368) 2023-10-16T12:30:29.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.596+0000 7f9deed9d700 1 -- 172.21.15.16:0/3379302761 learned_addr learned my addr 172.21.15.16:0/3379302761 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:29.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.596+0000 7f9dee59c700 1 --2- 172.21.15.16:0/3379302761 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9df006d940 0x7f9df015e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:29.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.596+0000 7f9dedd9b700 1 --2- 172.21.15.16:0/3379302761 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9df00cf280 0x7f9df015cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:29.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.596+0000 7f9deed9d700 1 -- 172.21.15.16:0/3379302761 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9df00cf280 msgr2=0x7f9df015cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:29.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.596+0000 7f9deed9d700 1 --2- 172.21.15.16:0/3379302761 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9df00cf280 0x7f9df015cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:29.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.596+0000 7f9deed9d700 1 -- 172.21.15.16:0/3379302761 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9df006d940 msgr2=0x7f9df015e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:29.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.596+0000 7f9deed9d700 1 --2- 172.21.15.16:0/3379302761 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9df006d940 0x7f9df015e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:29.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.596+0000 7f9deed9d700 1 -- 172.21.15.16:0/3379302761 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9de00094a0 con 0x7f9df015d240 2023-10-16T12:30:29.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.596+0000 7f9deed9d700 1 --2- 172.21.15.16:0/3379302761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9df015d240 0x7f9df00c3670 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f9de800cba0 tx=0x7f9de800ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:29.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.596+0000 7f9de6ffd700 1 -- 172.21.15.16:0/3379302761 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de800aaa0 con 0x7f9df015d240 2023-10-16T12:30:29.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.597+0000 7f9de6ffd700 1 -- 172.21.15.16:0/3379302761 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9de800ac20 con 0x7f9df015d240 2023-10-16T12:30:29.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.597+0000 7f9df4971700 1 -- 172.21.15.16:0/3379302761 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9df015c500 con 0x7f9df015d240 2023-10-16T12:30:29.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.597+0000 7f9de6ffd700 1 -- 172.21.15.16:0/3379302761 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de80070d0 con 0x7f9df015d240 2023-10-16T12:30:29.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.597+0000 7f9df4971700 1 -- 172.21.15.16:0/3379302761 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9df015e5c0 con 0x7f9df015d240 2023-10-16T12:30:29.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.598+0000 7f9df4971700 1 -- 172.21.15.16:0/3379302761 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9df015ca20 con 0x7f9df015d240 2023-10-16T12:30:29.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.598+0000 7f9de6ffd700 1 -- 172.21.15.16:0/3379302761 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 74) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9de8007230 con 0x7f9df015d240 2023-10-16T12:30:29.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.598+0000 7f9de6ffd700 1 --2- 172.21.15.16:0/3379302761 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f9dd006b020 0x7f9dd006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:29.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.598+0000 7f9de6ffd700 1 -- 172.21.15.16:0/3379302761 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 6085+0+0 (secure 0 0 0) 0x7f9de8089330 con 0x7f9df015d240 2023-10-16T12:30:29.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.598+0000 7f9dee59c700 1 --2- 172.21.15.16:0/3379302761 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f9dd006b020 0x7f9dd006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:29.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.599+0000 7f9dee59c700 1 --2- 172.21.15.16:0/3379302761 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f9dd006b020 0x7f9dd006d4e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9de00090b0 tx=0x7f9de001c040 comp rx=0 tx=0).ready entity=mgr.8388 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:29.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.603+0000 7f9de6ffd700 1 -- 172.21.15.16:0/3379302761 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9de8054160 con 0x7f9df015d240 2023-10-16T12:30:29.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.855+0000 7f9df4971700 1 -- 172.21.15.16:0/3379302761 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f9df006f0f0 con 0x7f9df015d240 2023-10-16T12:30:29.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.855+0000 7f9de6ffd700 1 -- 172.21.15.16:0/3379302761 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v70) v1 ==== 53+0+73 (secure 0 0 0) 0x7f9de8057780 con 0x7f9df015d240 2023-10-16T12:30:29.858 INFO:teuthology.orchestra.run.smithi016.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-16T12:30:29.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.857+0000 7f9df4971700 1 -- 172.21.15.16:0/3379302761 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f9dd006b020 msgr2=0x7f9dd006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:29.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.857+0000 7f9df4971700 1 --2- 172.21.15.16:0/3379302761 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f9dd006b020 0x7f9dd006d4e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9de00090b0 tx=0x7f9de001c040 comp rx=0 tx=0).stop 2023-10-16T12:30:29.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.857+0000 7f9df4971700 1 -- 172.21.15.16:0/3379302761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9df015d240 msgr2=0x7f9df00c3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:29.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.857+0000 7f9df4971700 1 --2- 172.21.15.16:0/3379302761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9df015d240 0x7f9df00c3670 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f9de800cba0 tx=0x7f9de800ceb0 comp rx=0 tx=0).stop 2023-10-16T12:30:29.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.857+0000 7f9df4971700 1 -- 172.21.15.16:0/3379302761 shutdown_connections 2023-10-16T12:30:29.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.857+0000 7f9df4971700 1 --2- 172.21.15.16:0/3379302761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9df015d240 0x7f9df00c3670 unknown :-1 s=CLOSED pgs=1567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:29.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.857+0000 7f9df4971700 1 --2- 172.21.15.16:0/3379302761 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f9dd006b020 0x7f9dd006d4e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:29.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.857+0000 7f9df4971700 1 --2- 172.21.15.16:0/3379302761 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9df006d940 0x7f9df015e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:29.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.857+0000 7f9df4971700 1 --2- 172.21.15.16:0/3379302761 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9df00cf280 0x7f9df015cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:29.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.857+0000 7f9df4971700 1 -- 172.21.15.16:0/3379302761 >> 172.21.15.16:0/3379302761 conn(0x7f9df00cc5b0 msgr2=0x7f9df015a320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:29.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.857+0000 7f9df4971700 1 -- 172.21.15.16:0/3379302761 shutdown_connections 2023-10-16T12:30:29.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:29.857+0000 7f9df4971700 1 -- 172.21.15.16:0/3379302761 wait complete. 2023-10-16T12:30:29.872 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-16T12:30:29.872 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T12:30:29.873 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi016.front.sepia.ceph.com 2023-10-16T12:30:29.874 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-16T12:30:29.874 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-16T12:30:29.875 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-16T12:30:29.875 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-16T12:30:29.876 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T12:30:29.876 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T12:30:29.877 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:30:29.878 DEBUG:teuthology.orchestra.run.smithi016:> ip addr 2023-10-16T12:30:29.897 INFO:teuthology.orchestra.run.smithi016.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T12:30:29.898 INFO:teuthology.orchestra.run.smithi016.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T12:30:29.898 INFO:teuthology.orchestra.run.smithi016.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T12:30:29.898 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:30:29.899 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 ::1/128 scope host 2023-10-16T12:30:29.899 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:30:29.899 INFO:teuthology.orchestra.run.smithi016.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T12:30:29.900 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e0 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:30:29.900 INFO:teuthology.orchestra.run.smithi016.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T12:30:29.900 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e1 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:30:29.901 INFO:teuthology.orchestra.run.smithi016.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T12:30:29.901 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:64 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:30:29.901 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.21.15.16/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-16T12:30:29.902 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft 41106sec preferred_lft 41106sec 2023-10-16T12:30:29.902 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::ec4:7aff:febd:1564/64 scope link 2023-10-16T12:30:29.902 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:30:29.903 INFO:teuthology.orchestra.run.smithi016.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T12:30:29.903 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:65 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:30:29.903 INFO:teuthology.orchestra.run.smithi016.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T12:30:29.904 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 36:cf:ea:25:db:f7 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:30:29.904 INFO:teuthology.orchestra.run.smithi016.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-16T12:30:29.904 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:30:29.905 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::cfc:c5ff:fe52:a786/64 scope link 2023-10-16T12:30:29.905 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:30:29.905 INFO:teuthology.orchestra.run.smithi016.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T12:30:29.906 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 6e:62:9d:a4:43:23 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T12:30:29.906 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::6c62:9dff:fea4:4323/64 scope link 2023-10-16T12:30:29.906 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:30:29.907 INFO:teuthology.orchestra.run.smithi016.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T12:30:29.907 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 36:cf:ea:25:db:f7 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-16T12:30:29.907 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::34cf:eaff:fe25:dbf7/64 scope link 2023-10-16T12:30:29.908 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:30:29.908 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:30:29.909 DEBUG:teuthology.orchestra.run.smithi016:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T12:30:29.964 INFO:teuthology.orchestra.run.smithi016.stdout:1 2023-10-16T12:30:29.966 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:30:29.966 DEBUG:teuthology.orchestra.run.smithi016:> route 2023-10-16T12:30:30.025 INFO:teuthology.orchestra.run.smithi016.stdout:Kernel IP routing table 2023-10-16T12:30:30.025 INFO:teuthology.orchestra.run.smithi016.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T12:30:30.026 INFO:teuthology.orchestra.run.smithi016.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-16T12:30:30.026 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-16T12:30:30.026 INFO:teuthology.orchestra.run.smithi016.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T12:30:30.027 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:30:30.028 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec bash -c ' 2023-10-16T12:30:30.028 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2023-10-16T12:30:30.029 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-16T12:30:30.030 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-16T12:30:30.030 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-16T12:30:30.031 DEBUG:teuthology.orchestra.run.smithi016:> ') 2023-10-16T12:30:30.131 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:30:30.132 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2023-10-16T12:30:30.149 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-16T12:30:30.149 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-16T12:30:30.150 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T12:30:30.150 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:30:30.151 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set brx.0 up 2023-10-16T12:30:30.217 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T12:30:30.218 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-16T12:30:30.275 INFO:teuthology.orchestra.run.smithi016.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-16T12:30:30.275 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T12:30:30.275 DEBUG:teuthology.orchestra.run.smithi016:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-16T12:30:30.331 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-16T12:30:30.331 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:30:30.332 DEBUG:teuthology.orchestra.run.smithi016:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T12:30:30.418 INFO:teuthology.orchestra.run.smithi016.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T12:30:30.418 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T12:30:30.418 INFO:teuthology.orchestra.run.smithi016.stdout:invalid new device string format 2023-10-16T12:30:30.427 INFO:teuthology.orchestra.run.smithi016.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T12:30:30.427 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T12:30:30.428 INFO:teuthology.orchestra.run.smithi016.stdout:invalid new device string format 2023-10-16T12:30:30.428 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: resolved to: "172.21.15.16:3300,172.21.15.49:3300,172.21.15.49:3301" 2023-10-16T12:30:30.428 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T12:30:30.464 INFO:teuthology.orchestra.run.smithi016.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T12:30:30.465 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T12:30:30.465 INFO:teuthology.orchestra.run.smithi016.stdout:invalid new device string format 2023-10-16T12:30:30.465 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: resolved to: "172.21.15.16:3300,172.21.15.49:3300,172.21.15.49:3301" 2023-10-16T12:30:30.465 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T12:30:30.466 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: resolved to: "172.21.15.16:6789,172.21.15.49:6789,172.21.15.49:6790" 2023-10-16T12:30:30.466 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.16:6789,172.21.15.49:6789,172.21.15.49:6790:/ 2023-10-16T12:30:30.466 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=5b28dfa1-b21f-474a-9b2e-9fb77af58254" will pass to kernel 2023-10-16T12:30:30.486 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-16T12:30:30.486 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:30:30.486 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-16T12:30:30.515 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:30:30.515 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T12:30:30.603 INFO:teuthology.orchestra.run.smithi016.stdout:client8793 2023-10-16T12:30:30.603 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:30:30.604 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:30:30.621 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-16T12:30:30.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:30.785+0000 7f4a86d87700 1 Processor -- start 2023-10-16T12:30:30.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:30.785+0000 7f4a86d87700 1 -- start start 2023-10-16T12:30:30.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:30.786+0000 7f4a86d87700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f4a800d2f70 0x7f4a800d2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:30.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:30.786+0000 7f4a86d87700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4a800ce9f0 0x7f4a800cdc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:30.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:30.786+0000 7f4a86d87700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a800ce1a0 con 0x7f4a8015d900 2023-10-16T12:30:30.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:30.786+0000 7f4a86d87700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a800ce400 con 0x7f4a800ce9f0 2023-10-16T12:30:30.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:30.786+0000 7f4a86d87700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a800cee40 con 0x7f4a800d2f70 2023-10-16T12:30:30.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:30.786+0000 7f4a85324700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4a800ce9f0 0x7f4a800cdc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:30.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:30.786+0000 7f4a85324700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4a800ce9f0 0x7f4a800cdc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45380/0 (socket says 172.21.15.16:45380) 2023-10-16T12:30:30.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:30.786+0000 7f4a85324700 1 -- 172.21.15.16:0/840195797 learned_addr learned my addr 172.21.15.16:0/840195797 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:30.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:30.786+0000 7f4a7ffff700 1 --2- 172.21.15.16:0/840195797 >> v2:172.21.15.49:3300/0 conn(0x7f4a800d2f70 0x7f4a800d2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:30.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:30.787+0000 7f4a85324700 1 -- 172.21.15.16:0/840195797 >> v2:172.21.15.49:3300/0 conn(0x7f4a800d2f70 msgr2=0x7f4a800d2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:30.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:30.787+0000 7f4a85324700 1 --2- 172.21.15.16:0/840195797 >> v2:172.21.15.49:3300/0 conn(0x7f4a800d2f70 0x7f4a800d2040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:30.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:30.787+0000 7f4a85324700 1 -- 172.21.15.16:0/840195797 >> v1:172.21.15.49:6789/0 conn(0x7f4a8015d900 legacy=0x7f4a800d2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:30.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:30.787+0000 7f4a85324700 1 -- 172.21.15.16:0/840195797 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a8015dce0 con 0x7f4a800ce9f0 2023-10-16T12:30:30.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:30.787+0000 7f4a7ffff700 1 --2- 172.21.15.16:0/840195797 >> v2:172.21.15.49:3300/0 conn(0x7f4a800d2f70 0x7f4a800d2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:30:30.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:30.787+0000 7f4a85324700 1 --2- 172.21.15.16:0/840195797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a800ce9f0 0x7f4a800cdc60 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f4a800b9810 tx=0x7f4a7800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4d1e2c30bb80b9f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:30.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:30.787+0000 7f4a7f7fe700 1 -- 172.21.15.16:0/840195797 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a7800e070 con 0x7f4a800ce9f0 2023-10-16T12:30:30.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:30.787+0000 7f4a7f7fe700 1 -- 172.21.15.16:0/840195797 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a7800b9c0 con 0x7f4a800ce9f0 2023-10-16T12:30:30.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:30.787+0000 7f4a7f7fe700 1 -- 172.21.15.16:0/840195797 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a7800bd70 con 0x7f4a800ce9f0 2023-10-16T12:30:30.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:30.787+0000 7f4a86d87700 1 -- 172.21.15.16:0/840195797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a800ce9f0 msgr2=0x7f4a800cdc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:30.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:30.787+0000 7f4a86d87700 1 --2- 172.21.15.16:0/840195797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a800ce9f0 0x7f4a800cdc60 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f4a800b9810 tx=0x7f4a7800b330 comp rx=0 tx=0).stop 2023-10-16T12:30:30.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:30.787+0000 7f4a86d87700 1 -- 172.21.15.16:0/840195797 shutdown_connections 2023-10-16T12:30:30.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:30.787+0000 7f4a86d87700 1 --2- 172.21.15.16:0/840195797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a800ce9f0 0x7f4a800cdc60 unknown :-1 s=CLOSED pgs=1569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:30.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:30.787+0000 7f4a86d87700 1 --2- 172.21.15.16:0/840195797 >> v2:172.21.15.49:3300/0 conn(0x7f4a800d2f70 0x7f4a800d2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:30.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:30.787+0000 7f4a86d87700 1 -- 172.21.15.16:0/840195797 >> 172.21.15.16:0/840195797 conn(0x7f4a80072330 msgr2=0x7f4a80072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:30.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:30.788+0000 7f4a86d87700 1 -- 172.21.15.16:0/840195797 shutdown_connections 2023-10-16T12:30:30.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:30.788+0000 7f4a86d87700 1 -- 172.21.15.16:0/840195797 wait complete. 2023-10-16T12:30:30.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:30.789+0000 7f4a86d87700 1 Processor -- start 2023-10-16T12:30:30.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:30.789+0000 7f4a86d87700 1 -- start start 2023-10-16T12:30:30.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:30.790+0000 7f4a86d87700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4a800ce9f0 0x7f4a800c6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:30.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:30.790+0000 7f4a86d87700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a800d2f70 0x7f4a800c8f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:30.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:30.790+0000 7f4a86d87700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4a8015d900 0x7f4a800c9440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:30.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:30.790+0000 7f4a86d87700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a800b7850 con 0x7f4a800d2f70 2023-10-16T12:30:30.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:30.790+0000 7f4a86d87700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a800b7570 con 0x7f4a8015d900 2023-10-16T12:30:30.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:30.790+0000 7f4a86d87700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a8015e050 con 0x7f4a800ce9f0 2023-10-16T12:30:30.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:30.790+0000 7f4a7ffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a800d2f70 0x7f4a800c8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:30.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:30.790+0000 7f4a7ffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a800d2f70 0x7f4a800c8f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45384/0 (socket says 172.21.15.16:45384) 2023-10-16T12:30:30.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:30.790+0000 7f4a7ffff700 1 -- 172.21.15.16:0/736682480 learned_addr learned my addr 172.21.15.16:0/736682480 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:30.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:30.790+0000 7f4a85324700 1 --2- 172.21.15.16:0/736682480 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4a8015d900 0x7f4a800c9440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:30.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:30.790+0000 7f4a84b23700 1 --2- 172.21.15.16:0/736682480 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4a800ce9f0 0x7f4a800c6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:30.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:30.790+0000 7f4a7ffff700 1 -- 172.21.15.16:0/736682480 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4a800ce9f0 msgr2=0x7f4a800c6180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:30.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:30.790+0000 7f4a7ffff700 1 --2- 172.21.15.16:0/736682480 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4a800ce9f0 0x7f4a800c6180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:30.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:30.790+0000 7f4a7ffff700 1 -- 172.21.15.16:0/736682480 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4a8015d900 msgr2=0x7f4a800c9440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:30.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:30.790+0000 7f4a7ffff700 1 --2- 172.21.15.16:0/736682480 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4a8015d900 0x7f4a800c9440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:30.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:30.790+0000 7f4a7ffff700 1 -- 172.21.15.16:0/736682480 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a78009b30 con 0x7f4a800d2f70 2023-10-16T12:30:30.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:30.790+0000 7f4a7ffff700 1 --2- 172.21.15.16:0/736682480 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a800d2f70 0x7f4a800c8f00 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f4a6c00e810 tx=0x7f4a6c00eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:30.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:30.790+0000 7f4a7d7fa700 1 -- 172.21.15.16:0/736682480 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a6c00c070 con 0x7f4a800d2f70 2023-10-16T12:30:30.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:30.790+0000 7f4a7d7fa700 1 -- 172.21.15.16:0/736682480 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a6c004230 con 0x7f4a800d2f70 2023-10-16T12:30:30.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:30.790+0000 7f4a86d87700 1 -- 172.21.15.16:0/736682480 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a8015e330 con 0x7f4a800d2f70 2023-10-16T12:30:30.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:30.790+0000 7f4a7d7fa700 1 -- 172.21.15.16:0/736682480 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a6c0036a0 con 0x7f4a800d2f70 2023-10-16T12:30:30.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:30.790+0000 7f4a86d87700 1 -- 172.21.15.16:0/736682480 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a800c66c0 con 0x7f4a800d2f70 2023-10-16T12:30:30.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:30.791+0000 7f4a86d87700 1 -- 172.21.15.16:0/736682480 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a8015e980 con 0x7f4a800d2f70 2023-10-16T12:30:30.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:30.792+0000 7f4a7d7fa700 1 -- 172.21.15.16:0/736682480 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 74) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4a6c003800 con 0x7f4a800d2f70 2023-10-16T12:30:30.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:30.792+0000 7f4a7d7fa700 1 --2- 172.21.15.16:0/736682480 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f4a6406b020 0x7f4a6406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:30.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:30.792+0000 7f4a7d7fa700 1 -- 172.21.15.16:0/736682480 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 6085+0+0 (secure 0 0 0) 0x7f4a6c089700 con 0x7f4a800d2f70 2023-10-16T12:30:30.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:30.793+0000 7f4a84b23700 1 --2- 172.21.15.16:0/736682480 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f4a6406b020 0x7f4a6406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:30.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:30.793+0000 7f4a84b23700 1 --2- 172.21.15.16:0/736682480 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f4a6406b020 0x7f4a6406d4e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4a74007fd0 tx=0x7f4a74009040 comp rx=0 tx=0).ready entity=mgr.8388 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:30.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:30.797+0000 7f4a7d7fa700 1 -- 172.21.15.16:0/736682480 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4a6c054370 con 0x7f4a800d2f70 2023-10-16T12:30:31.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.030+0000 7f4a86d87700 1 -- 172.21.15.16:0/736682480 --> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f4a8015ded0 con 0x7f4a6406b020 2023-10-16T12:30:31.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.031+0000 7f4a7d7fa700 1 -- 172.21.15.16:0/736682480 <== mgr.8388 v2:172.21.15.16:6832/258291157 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f4a740014c0 con 0x7f4a6406b020 2023-10-16T12:30:31.033 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2023-10-16T12:30:31.033 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:30:31.034 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cephfs" 2023-10-16T12:30:31.034 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:30:31.034 INFO:teuthology.orchestra.run.smithi016.stdout:] 2023-10-16T12:30:31.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.033+0000 7f4a86d87700 1 -- 172.21.15.16:0/736682480 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f4a6406b020 msgr2=0x7f4a6406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:31.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.033+0000 7f4a86d87700 1 --2- 172.21.15.16:0/736682480 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f4a6406b020 0x7f4a6406d4e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4a74007fd0 tx=0x7f4a74009040 comp rx=0 tx=0).stop 2023-10-16T12:30:31.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.034+0000 7f4a86d87700 1 -- 172.21.15.16:0/736682480 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a800d2f70 msgr2=0x7f4a800c8f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:31.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.034+0000 7f4a86d87700 1 --2- 172.21.15.16:0/736682480 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a800d2f70 0x7f4a800c8f00 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f4a6c00e810 tx=0x7f4a6c00eb20 comp rx=0 tx=0).stop 2023-10-16T12:30:31.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.034+0000 7f4a86d87700 1 -- 172.21.15.16:0/736682480 shutdown_connections 2023-10-16T12:30:31.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.034+0000 7f4a86d87700 1 --2- 172.21.15.16:0/736682480 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f4a6406b020 0x7f4a6406d4e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:31.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.034+0000 7f4a86d87700 1 --2- 172.21.15.16:0/736682480 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4a8015d900 0x7f4a800c9440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:31.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.034+0000 7f4a86d87700 1 --2- 172.21.15.16:0/736682480 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4a800ce9f0 0x7f4a800c6180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:31.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.034+0000 7f4a86d87700 1 --2- 172.21.15.16:0/736682480 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a800d2f70 0x7f4a800c8f00 unknown :-1 s=CLOSED pgs=1570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:31.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.034+0000 7f4a86d87700 1 -- 172.21.15.16:0/736682480 >> 172.21.15.16:0/736682480 conn(0x7f4a80072330 msgr2=0x7f4a800cab10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:31.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.034+0000 7f4a86d87700 1 -- 172.21.15.16:0/736682480 shutdown_connections 2023-10-16T12:30:31.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.034+0000 7f4a86d87700 1 -- 172.21.15.16:0/736682480 wait complete. 2023-10-16T12:30:31.048 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-16T12:30:31.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.173+0000 7f9817ddf700 1 Processor -- start 2023-10-16T12:30:31.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.174+0000 7f9817ddf700 1 -- start start 2023-10-16T12:30:31.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.174+0000 7f9817ddf700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f981015afe0 0x7f981015b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:31.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.174+0000 7f9817ddf700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f981006e170 0x7f98100cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:31.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.174+0000 7f9817ddf700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98100ce360 con 0x7f98100cb880 2023-10-16T12:30:31.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.174+0000 7f9817ddf700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98100ce640 con 0x7f981006e170 2023-10-16T12:30:31.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.174+0000 7f9817ddf700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98100ce920 con 0x7f981015afe0 2023-10-16T12:30:31.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.174+0000 7f981537a700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f98100cb880 0x7f981006da60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34666/0 (socket says 172.21.15.16:34666) 2023-10-16T12:30:31.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.174+0000 7f981537a700 1 -- 172.21.15.16:0/1554816985 learned_addr learned my addr 172.21.15.16:0/1554816985 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:30:31.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.175+0000 7f981637c700 1 --2- 172.21.15.16:0/1554816985 >> v2:172.21.15.49:3300/0 conn(0x7f981006e170 0x7f98100cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:31.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.175+0000 7f9815b7b700 1 --2- 172.21.15.16:0/1554816985 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f981015afe0 0x7f981015b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:31.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.175+0000 7f9814b79700 1 -- 172.21.15.16:0/1554816985 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1314100641 0 0) 0x7f98100ce360 con 0x7f98100cb880 2023-10-16T12:30:31.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.175+0000 7f9814b79700 1 -- 172.21.15.16:0/1554816985 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97fc0035b0 con 0x7f98100cb880 2023-10-16T12:30:31.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.175+0000 7f9814b79700 1 -- 172.21.15.16:0/1554816985 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3231135037 0 0) 0x7f97fc0035b0 con 0x7f98100cb880 2023-10-16T12:30:31.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.175+0000 7f9814b79700 1 -- 172.21.15.16:0/1554816985 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f981015afe0 msgr2=0x7f981015b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:31.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.175+0000 7f9814b79700 1 --2- 172.21.15.16:0/1554816985 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f981015afe0 0x7f981015b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:31.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.175+0000 7f9814b79700 1 -- 172.21.15.16:0/1554816985 >> v2:172.21.15.49:3300/0 conn(0x7f981006e170 msgr2=0x7f98100cbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:31.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.175+0000 7f9814b79700 1 --2- 172.21.15.16:0/1554816985 >> v2:172.21.15.49:3300/0 conn(0x7f981006e170 0x7f98100cbd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:31.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.175+0000 7f9814b79700 1 -- 172.21.15.16:0/1554816985 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98100cec00 con 0x7f98100cb880 2023-10-16T12:30:31.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.175+0000 7f9814b79700 1 -- 172.21.15.16:0/1554816985 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f9804001670 con 0x7f98100cb880 2023-10-16T12:30:31.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.175+0000 7f981637c700 1 --2- 172.21.15.16:0/1554816985 >> v2:172.21.15.49:3300/0 conn(0x7f981006e170 0x7f98100cbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:30:31.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.175+0000 7f9814b79700 1 -- 172.21.15.16:0/1554816985 >> v1:172.21.15.16:6789/0 conn(0x7f98100cb880 legacy=0x7f981006da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:31.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.175+0000 7f9814b79700 1 --2- 172.21.15.16:0/1554816985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97fc003a00 0x7f97fc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:31.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.175+0000 7f9814b79700 1 --2- 172.21.15.16:0/1554816985 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f981015afe0 0x7f97fc004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:31.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.175+0000 7f9814b79700 1 --2- 172.21.15.16:0/1554816985 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f97fc004d90 0x7f97fc005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:31.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.175+0000 7f9814b79700 1 -- 172.21.15.16:0/1554816985 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98100ce640 con 0x7f97fc003a00 2023-10-16T12:30:31.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.175+0000 7f9814b79700 1 -- 172.21.15.16:0/1554816985 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98100ce920 con 0x7f97fc004d90 2023-10-16T12:30:31.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.175+0000 7f9814b79700 1 -- 172.21.15.16:0/1554816985 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97fc00ba10 con 0x7f981015afe0 2023-10-16T12:30:31.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.175+0000 7f9815b7b700 1 --2- 172.21.15.16:0/1554816985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97fc003a00 0x7f97fc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:31.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.175+0000 7f981637c700 1 --2- 172.21.15.16:0/1554816985 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f97fc004d90 0x7f97fc005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:31.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.175+0000 7f981537a700 1 --2- 172.21.15.16:0/1554816985 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f981015afe0 0x7f97fc004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:31.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.175+0000 7f9815b7b700 1 -- 172.21.15.16:0/1554816985 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f981015afe0 msgr2=0x7f97fc004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:31.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.175+0000 7f9815b7b700 1 --2- 172.21.15.16:0/1554816985 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f981015afe0 0x7f97fc004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:31.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.175+0000 7f9815b7b700 1 -- 172.21.15.16:0/1554816985 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f97fc004d90 msgr2=0x7f97fc005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:31.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.175+0000 7f9815b7b700 1 --2- 172.21.15.16:0/1554816985 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f97fc004d90 0x7f97fc005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:31.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.175+0000 7f9815b7b700 1 -- 172.21.15.16:0/1554816985 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97fc0035b0 con 0x7f97fc003a00 2023-10-16T12:30:31.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.175+0000 7f981637c700 1 --2- 172.21.15.16:0/1554816985 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f97fc004d90 0x7f97fc005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:30:31.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.176+0000 7f9815b7b700 1 --2- 172.21.15.16:0/1554816985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97fc003a00 0x7f97fc003e20 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f980c009b60 tx=0x7f97fc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38a636736de9954b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:31.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.176+0000 7f9814b79700 1 -- 172.21.15.16:0/1554816985 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f980c00e6c0 con 0x7f97fc003a00 2023-10-16T12:30:31.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.176+0000 7f9814b79700 1 -- 172.21.15.16:0/1554816985 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f980c00e820 con 0x7f97fc003a00 2023-10-16T12:30:31.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.176+0000 7f9814b79700 1 -- 172.21.15.16:0/1554816985 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f980c00eb90 con 0x7f97fc003a00 2023-10-16T12:30:31.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.176+0000 7f9817ddf700 1 -- 172.21.15.16:0/1554816985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97fc003a00 msgr2=0x7f97fc003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:31.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.176+0000 7f9817ddf700 1 --2- 172.21.15.16:0/1554816985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97fc003a00 0x7f97fc003e20 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f980c009b60 tx=0x7f97fc005880 comp rx=0 tx=0).stop 2023-10-16T12:30:31.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.176+0000 7f981637c700 1 -- 172.21.15.16:0/1554816985 reap_dead start 2023-10-16T12:30:31.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.176+0000 7f9817ddf700 1 -- 172.21.15.16:0/1554816985 shutdown_connections 2023-10-16T12:30:31.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.176+0000 7f9817ddf700 1 -- 172.21.15.16:0/1554816985 >> 172.21.15.16:0/1554816985 conn(0x7f98100b1f40 msgr2=0x7f98100b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:31.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.176+0000 7f9817ddf700 1 -- 172.21.15.16:0/1554816985 shutdown_connections 2023-10-16T12:30:31.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.176+0000 7f9817ddf700 1 -- 172.21.15.16:0/1554816985 wait complete. 2023-10-16T12:30:31.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.178+0000 7f9817ddf700 1 Processor -- start 2023-10-16T12:30:31.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.178+0000 7f9817ddf700 1 -- start start 2023-10-16T12:30:31.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.178+0000 7f9817ddf700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97fc003a00 0x7f98100b1160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:31.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.178+0000 7f9817ddf700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f98100afc30 0x7f98101f2560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:31.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.178+0000 7f9817ddf700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f98100af060 0x7f9810146380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:31.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.178+0000 7f9817ddf700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98100b00c0 con 0x7f97fc003a00 2023-10-16T12:30:31.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.178+0000 7f9817ddf700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98100b03a0 con 0x7f98100afc30 2023-10-16T12:30:31.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.178+0000 7f9817ddf700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98100b0680 con 0x7f98100af060 2023-10-16T12:30:31.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.179+0000 7f9815b7b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97fc003a00 0x7f98100b1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:31.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.179+0000 7f9815b7b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97fc003a00 0x7f98100b1160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45402/0 (socket says 172.21.15.16:45402) 2023-10-16T12:30:31.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.179+0000 7f9815b7b700 1 -- 172.21.15.16:0/1370994755 learned_addr learned my addr 172.21.15.16:0/1370994755 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:31.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.179+0000 7f981637c700 1 --2- 172.21.15.16:0/1370994755 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f98100af060 0x7f9810146380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:31.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.179+0000 7f981537a700 1 --2- 172.21.15.16:0/1370994755 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f98100afc30 0x7f98101f2560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:31.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.179+0000 7f9815b7b700 1 -- 172.21.15.16:0/1370994755 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f98100af060 msgr2=0x7f9810146380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:31.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.179+0000 7f9815b7b700 1 --2- 172.21.15.16:0/1370994755 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f98100af060 0x7f9810146380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:31.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.179+0000 7f9815b7b700 1 -- 172.21.15.16:0/1370994755 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f98100afc30 msgr2=0x7f98101f2560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:31.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.179+0000 7f9815b7b700 1 --2- 172.21.15.16:0/1370994755 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f98100afc30 0x7f98101f2560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:31.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.179+0000 7f9815b7b700 1 -- 172.21.15.16:0/1370994755 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98100b0960 con 0x7f97fc003a00 2023-10-16T12:30:31.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.179+0000 7f9815b7b700 1 --2- 172.21.15.16:0/1370994755 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97fc003a00 0x7f98100b1160 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7f97fc0046e0 tx=0x7f980c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:31.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.179+0000 7f98027fc700 1 -- 172.21.15.16:0/1370994755 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f980c00e6c0 con 0x7f97fc003a00 2023-10-16T12:30:31.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.179+0000 7f98027fc700 1 -- 172.21.15.16:0/1370994755 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f980c003ab0 con 0x7f97fc003a00 2023-10-16T12:30:31.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.179+0000 7f9817ddf700 1 -- 172.21.15.16:0/1370994755 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98100b0c40 con 0x7f97fc003a00 2023-10-16T12:30:31.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.179+0000 7f98027fc700 1 -- 172.21.15.16:0/1370994755 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f980c003e60 con 0x7f97fc003a00 2023-10-16T12:30:31.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.179+0000 7f9817ddf700 1 -- 172.21.15.16:0/1370994755 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98100af4e0 con 0x7f97fc003a00 2023-10-16T12:30:31.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.180+0000 7f9817ddf700 1 -- 172.21.15.16:0/1370994755 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98100b0fb0 con 0x7f97fc003a00 2023-10-16T12:30:31.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.181+0000 7f98027fc700 1 -- 172.21.15.16:0/1370994755 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 74) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f980c013470 con 0x7f97fc003a00 2023-10-16T12:30:31.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.181+0000 7f98027fc700 1 --2- 172.21.15.16:0/1370994755 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f97f406b020 0x7f97f406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:31.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.181+0000 7f98027fc700 1 -- 172.21.15.16:0/1370994755 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 6085+0+0 (secure 0 0 0) 0x7f980c08b280 con 0x7f97fc003a00 2023-10-16T12:30:31.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.181+0000 7f981537a700 1 --2- 172.21.15.16:0/1370994755 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f97f406b020 0x7f97f406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:31.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.182+0000 7f981537a700 1 --2- 172.21.15.16:0/1370994755 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f97f406b020 0x7f97f406d4e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9804007fd0 tx=0x7f9804007980 comp rx=0 tx=0).ready entity=mgr.8388 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:31.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.186+0000 7f98027fc700 1 -- 172.21.15.16:0/1370994755 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f980c056300 con 0x7f97fc003a00 2023-10-16T12:30:31.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:31.456+0000 7f9817ddf700 1 -- 172.21.15.16:0/1370994755 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f981004ebb0 con 0x7f97fc003a00 2023-10-16T12:30:32.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.450+0000 7f98027fc700 1 -- 172.21.15.16:0/1370994755 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v75) v1 ==== 92+0+0 (secure 0 0 0) 0x7f980c059920 con 0x7f97fc003a00 2023-10-16T12:30:32.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.454+0000 7f9817ddf700 1 -- 172.21.15.16:0/1370994755 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f97f406b020 msgr2=0x7f97f406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:32.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.454+0000 7f9817ddf700 1 --2- 172.21.15.16:0/1370994755 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f97f406b020 0x7f97f406d4e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9804007fd0 tx=0x7f9804007980 comp rx=0 tx=0).stop 2023-10-16T12:30:32.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.454+0000 7f9817ddf700 1 -- 172.21.15.16:0/1370994755 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97fc003a00 msgr2=0x7f98100b1160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:32.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.454+0000 7f9817ddf700 1 --2- 172.21.15.16:0/1370994755 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97fc003a00 0x7f98100b1160 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7f97fc0046e0 tx=0x7f980c0047c0 comp rx=0 tx=0).stop 2023-10-16T12:30:32.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.454+0000 7f9817ddf700 1 -- 172.21.15.16:0/1370994755 shutdown_connections 2023-10-16T12:30:32.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.454+0000 7f9817ddf700 1 --2- 172.21.15.16:0/1370994755 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f98100af060 0x7f9810146380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:32.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.454+0000 7f9817ddf700 1 --2- 172.21.15.16:0/1370994755 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97fc003a00 0x7f98100b1160 unknown :-1 s=CLOSED pgs=1573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:32.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.454+0000 7f9817ddf700 1 --2- 172.21.15.16:0/1370994755 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f97f406b020 0x7f97f406d4e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:32.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.454+0000 7f9817ddf700 1 --2- 172.21.15.16:0/1370994755 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f98100afc30 0x7f98101f2560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:32.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.454+0000 7f9817ddf700 1 -- 172.21.15.16:0/1370994755 >> 172.21.15.16:0/1370994755 conn(0x7f98100b1f40 msgr2=0x7f9810153b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:32.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.455+0000 7f9817ddf700 1 -- 172.21.15.16:0/1370994755 shutdown_connections 2023-10-16T12:30:32.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.455+0000 7f9817ddf700 1 -- 172.21.15.16:0/1370994755 wait complete. 2023-10-16T12:30:32.478 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-16T12:30:32.604 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:30:32.602+0000 7f8c2bb99200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T12:30:32.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.613+0000 7f76c34fd700 1 Processor -- start 2023-10-16T12:30:32.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.613+0000 7f76c34fd700 1 -- start start 2023-10-16T12:30:32.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.614+0000 7f76c34fd700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f76bc15b070 0x7f76bc15b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:32.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.614+0000 7f76c34fd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f76bc06e170 0x7f76bc0cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:32.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.614+0000 7f76c34fd700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76bc0cef70 con 0x7f76bc0cb880 2023-10-16T12:30:32.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.614+0000 7f76c34fd700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76bc0ce360 con 0x7f76bc06e170 2023-10-16T12:30:32.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.614+0000 7f76c34fd700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76bc0ce640 con 0x7f76bc15b070 2023-10-16T12:30:32.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.614+0000 7f76c1a9a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f76bc06e170 0x7f76bc0cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:32.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.614+0000 7f76c0a98700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f76bc0cb880 0x7f76bc06da60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34712/0 (socket says 172.21.15.16:34712) 2023-10-16T12:30:32.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.614+0000 7f76c0a98700 1 -- 172.21.15.16:0/1147630314 learned_addr learned my addr 172.21.15.16:0/1147630314 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:30:32.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.614+0000 7f76c1a9a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f76bc06e170 0x7f76bc0cbd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45440/0 (socket says 172.21.15.16:45440) 2023-10-16T12:30:32.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.614+0000 7f76c1299700 1 --2- 172.21.15.16:0/1147630314 >> v2:172.21.15.49:3300/0 conn(0x7f76bc15b070 0x7f76bc15b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:32.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.615+0000 7f76c1a9a700 1 -- 172.21.15.16:0/1147630314 >> v2:172.21.15.49:3300/0 conn(0x7f76bc15b070 msgr2=0x7f76bc15b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:32.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.615+0000 7f76c1a9a700 1 --2- 172.21.15.16:0/1147630314 >> v2:172.21.15.49:3300/0 conn(0x7f76bc15b070 0x7f76bc15b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:32.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.615+0000 7f76c1a9a700 1 -- 172.21.15.16:0/1147630314 >> v1:172.21.15.16:6789/0 conn(0x7f76bc0cb880 legacy=0x7f76bc06da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:32.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.615+0000 7f76b3fff700 1 -- 172.21.15.16:0/1147630314 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 598768898 0 0) 0x7f76bc0cef70 con 0x7f76bc0cb880 2023-10-16T12:30:32.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.615+0000 7f76c1a9a700 1 -- 172.21.15.16:0/1147630314 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76bc0ce920 con 0x7f76bc06e170 2023-10-16T12:30:32.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.615+0000 7f76c1a9a700 1 --2- 172.21.15.16:0/1147630314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76bc06e170 0x7f76bc0cbd00 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f76bc0705a0 tx=0x7f76b400d330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6d5eaaf50f06df7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:32.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.615+0000 7f76b3fff700 1 -- 172.21.15.16:0/1147630314 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76b4010070 con 0x7f76bc06e170 2023-10-16T12:30:32.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.615+0000 7f76b3fff700 1 -- 172.21.15.16:0/1147630314 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76b400b2e0 con 0x7f76bc06e170 2023-10-16T12:30:32.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.615+0000 7f76b3fff700 1 -- 172.21.15.16:0/1147630314 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76b400dc10 con 0x7f76bc06e170 2023-10-16T12:30:32.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.616+0000 7f76c34fd700 1 -- 172.21.15.16:0/1147630314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76bc06e170 msgr2=0x7f76bc0cbd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:32.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.616+0000 7f76c34fd700 1 --2- 172.21.15.16:0/1147630314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76bc06e170 0x7f76bc0cbd00 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f76bc0705a0 tx=0x7f76b400d330 comp rx=0 tx=0).stop 2023-10-16T12:30:32.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.616+0000 7f76c34fd700 1 -- 172.21.15.16:0/1147630314 shutdown_connections 2023-10-16T12:30:32.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.616+0000 7f76c34fd700 1 --2- 172.21.15.16:0/1147630314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76bc06e170 0x7f76bc0cbd00 unknown :-1 s=CLOSED pgs=1578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:32.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.616+0000 7f76c34fd700 1 --2- 172.21.15.16:0/1147630314 >> v2:172.21.15.49:3300/0 conn(0x7f76bc15b070 0x7f76bc15b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:32.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.616+0000 7f76c34fd700 1 -- 172.21.15.16:0/1147630314 >> 172.21.15.16:0/1147630314 conn(0x7f76bc0b1f40 msgr2=0x7f76bc0b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:32.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.616+0000 7f76c34fd700 1 -- 172.21.15.16:0/1147630314 shutdown_connections 2023-10-16T12:30:32.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.616+0000 7f76c34fd700 1 -- 172.21.15.16:0/1147630314 wait complete. 2023-10-16T12:30:32.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.618+0000 7f76c34fd700 1 Processor -- start 2023-10-16T12:30:32.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.618+0000 7f76c34fd700 1 -- start start 2023-10-16T12:30:32.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.619+0000 7f76c34fd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f76bc06e170 0x7f76bc1539c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:32.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.619+0000 7f76c34fd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76bc0cb880 0x7f76bc1579a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:32.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.619+0000 7f76c34fd700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f76bc15b070 0x7f76bc0b1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:32.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.619+0000 7f76c34fd700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76bc1f6c10 con 0x7f76bc0cb880 2023-10-16T12:30:32.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.619+0000 7f76c34fd700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76bc1f6000 con 0x7f76bc06e170 2023-10-16T12:30:32.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.619+0000 7f76c34fd700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76bc1f62e0 con 0x7f76bc15b070 2023-10-16T12:30:32.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.619+0000 7f76c0a98700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76bc0cb880 0x7f76bc1579a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:32.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.619+0000 7f76c0a98700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76bc0cb880 0x7f76bc1579a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45444/0 (socket says 172.21.15.16:45444) 2023-10-16T12:30:32.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.619+0000 7f76c0a98700 1 -- 172.21.15.16:0/1368216003 learned_addr learned my addr 172.21.15.16:0/1368216003 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:32.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.619+0000 7f76c1299700 1 --2- 172.21.15.16:0/1368216003 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f76bc06e170 0x7f76bc1539c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:32.628 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:32.619+0000 7f98ef0b1200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T12:30:32.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.619+0000 7f76c1a9a700 1 --2- 172.21.15.16:0/1368216003 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f76bc15b070 0x7f76bc0b1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:32.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.620+0000 7f76c0a98700 1 -- 172.21.15.16:0/1368216003 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f76bc15b070 msgr2=0x7f76bc0b1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:32.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.620+0000 7f76c0a98700 1 --2- 172.21.15.16:0/1368216003 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f76bc15b070 0x7f76bc0b1120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:32.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.620+0000 7f76c0a98700 1 -- 172.21.15.16:0/1368216003 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f76bc06e170 msgr2=0x7f76bc1539c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:32.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.620+0000 7f76c0a98700 1 --2- 172.21.15.16:0/1368216003 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f76bc06e170 0x7f76bc1539c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:32.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.620+0000 7f76c0a98700 1 -- 172.21.15.16:0/1368216003 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76b400bc80 con 0x7f76bc0cb880 2023-10-16T12:30:32.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.620+0000 7f76c1299700 1 --2- 172.21.15.16:0/1368216003 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f76bc06e170 0x7f76bc1539c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:30:32.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.620+0000 7f76c0a98700 1 --2- 172.21.15.16:0/1368216003 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76bc0cb880 0x7f76bc1579a0 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f76ac0011e0 tx=0x7f76ac00ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:32.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.620+0000 7f76b1ffb700 1 -- 172.21.15.16:0/1368216003 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76ac004c80 con 0x7f76bc0cb880 2023-10-16T12:30:32.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.620+0000 7f76b1ffb700 1 -- 172.21.15.16:0/1368216003 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76ac004e00 con 0x7f76bc0cb880 2023-10-16T12:30:32.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.620+0000 7f76c34fd700 1 -- 172.21.15.16:0/1368216003 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76bc1f65c0 con 0x7f76bc0cb880 2023-10-16T12:30:32.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.620+0000 7f76b1ffb700 1 -- 172.21.15.16:0/1368216003 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76ac007310 con 0x7f76bc0cb880 2023-10-16T12:30:32.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.620+0000 7f76c34fd700 1 -- 172.21.15.16:0/1368216003 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76bc153f00 con 0x7f76bc0cb880 2023-10-16T12:30:32.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.621+0000 7f76c34fd700 1 -- 172.21.15.16:0/1368216003 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76bc04ebb0 con 0x7f76bc0cb880 2023-10-16T12:30:32.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.622+0000 7f76b1ffb700 1 -- 172.21.15.16:0/1368216003 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 75) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f76ac007470 con 0x7f76bc0cb880 2023-10-16T12:30:32.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.622+0000 7f76b1ffb700 1 --2- 172.21.15.16:0/1368216003 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f76a006b070 0x7f76a006d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:32.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.622+0000 7f76c1299700 1 -- 172.21.15.16:0/1368216003 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f76a006b070 msgr2=0x7f76a006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/258291157 2023-10-16T12:30:32.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.623+0000 7f76b1ffb700 1 -- 172.21.15.16:0/1368216003 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 6085+0+0 (secure 0 0 0) 0x7f76ac0895e0 con 0x7f76bc0cb880 2023-10-16T12:30:32.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.623+0000 7f76c1299700 1 --2- 172.21.15.16:0/1368216003 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f76a006b070 0x7f76a006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:30:32.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.627+0000 7f76b1ffb700 1 -- 172.21.15.16:0/1368216003 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f76ac054400 con 0x7f76bc0cb880 2023-10-16T12:30:32.673 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:30:32.671+0000 7f8c2bb99200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T12:30:32.693 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:32.691+0000 7f98ef0b1200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T12:30:32.748 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:30:32.746+0000 7f8c2bb99200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T12:30:32.769 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:32.767+0000 7f98ef0b1200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T12:30:32.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.824+0000 7f76c1299700 1 -- 172.21.15.16:0/1368216003 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f76a006b070 msgr2=0x7f76a006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/258291157 2023-10-16T12:30:32.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.824+0000 7f76c1299700 1 --2- 172.21.15.16:0/1368216003 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f76a006b070 0x7f76a006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:30:32.830 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:30:32.828+0000 7f8c2bb99200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T12:30:32.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.832+0000 7f76c34fd700 1 -- 172.21.15.16:0/1368216003 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f76bc071970 con 0x7f76bc0cb880 2023-10-16T12:30:32.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.835+0000 7f76b1ffb700 1 -- 172.21.15.16:0/1368216003 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v47)=0 v47) v1 ==== 138+0+0 (secure 0 0 0) 0x7f76ac057a20 con 0x7f76bc0cb880 2023-10-16T12:30:32.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.837+0000 7f76c34fd700 1 -- 172.21.15.16:0/1368216003 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f76a006b070 msgr2=0x7f76a006d530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:30:32.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.837+0000 7f76c34fd700 1 --2- 172.21.15.16:0/1368216003 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f76a006b070 0x7f76a006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:32.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.837+0000 7f76c34fd700 1 -- 172.21.15.16:0/1368216003 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76bc0cb880 msgr2=0x7f76bc1579a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:32.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.837+0000 7f76c34fd700 1 --2- 172.21.15.16:0/1368216003 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76bc0cb880 0x7f76bc1579a0 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f76ac0011e0 tx=0x7f76ac00ca30 comp rx=0 tx=0).stop 2023-10-16T12:30:32.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.837+0000 7f76c34fd700 1 -- 172.21.15.16:0/1368216003 shutdown_connections 2023-10-16T12:30:32.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.837+0000 7f76c34fd700 1 --2- 172.21.15.16:0/1368216003 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f76bc15b070 0x7f76bc0b1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:32.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.837+0000 7f76c34fd700 1 --2- 172.21.15.16:0/1368216003 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f76a006b070 0x7f76a006d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:32.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.837+0000 7f76c34fd700 1 --2- 172.21.15.16:0/1368216003 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f76bc06e170 0x7f76bc1539c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:32.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.837+0000 7f76c34fd700 1 --2- 172.21.15.16:0/1368216003 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76bc0cb880 0x7f76bc1579a0 unknown :-1 s=CLOSED pgs=1579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:32.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.837+0000 7f76c34fd700 1 -- 172.21.15.16:0/1368216003 >> 172.21.15.16:0/1368216003 conn(0x7f76bc0b1f40 msgr2=0x7f76bc154850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:32.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.837+0000 7f76c34fd700 1 -- 172.21.15.16:0/1368216003 shutdown_connections 2023-10-16T12:30:32.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.837+0000 7f76c34fd700 1 -- 172.21.15.16:0/1368216003 wait complete. 2023-10-16T12:30:32.852 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-16T12:30:32.853 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:32.850+0000 7f98ef0b1200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T12:30:32.978 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:30:32.976+0000 7f8c2bb99200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T12:30:32.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.980+0000 7fdc17480700 1 Processor -- start 2023-10-16T12:30:32.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.980+0000 7fdc17480700 1 -- start start 2023-10-16T12:30:32.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.981+0000 7fdc17480700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdc1015b2c0 0x7fdc10161340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:32.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.981+0000 7fdc17480700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc1015ecb0 con 0x7fdc1006a7e0 2023-10-16T12:30:32.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.981+0000 7fdc17480700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc10161880 con 0x7fdc1006e570 2023-10-16T12:30:32.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.981+0000 7fdc17480700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc1015b6a0 con 0x7fdc1015b2c0 2023-10-16T12:30:32.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.981+0000 7fdc1521c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdc1015b2c0 0x7fdc10161340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:32.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.981+0000 7fdc15a1d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fdc1006a7e0 0x7fdc10071a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34774/0 (socket says 172.21.15.16:34774) 2023-10-16T12:30:32.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.981+0000 7fdc15a1d700 1 -- 172.21.15.16:0/1899909239 learned_addr learned my addr 172.21.15.16:0/1899909239 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:30:32.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.982+0000 7fdc1521c700 1 -- 172.21.15.16:0/1899909239 >> v1:172.21.15.49:6789/0 conn(0x7fdc1006e570 legacy=0x7fdc1015e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:32.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.982+0000 7fdc1521c700 1 -- 172.21.15.16:0/1899909239 >> v1:172.21.15.16:6789/0 conn(0x7fdc1006a7e0 legacy=0x7fdc10071a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:32.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.982+0000 7fdc1521c700 1 -- 172.21.15.16:0/1899909239 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc10161b20 con 0x7fdc1015b2c0 2023-10-16T12:30:32.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.982+0000 7fdc07fff700 1 -- 172.21.15.16:0/1899909239 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1505348867 0 0) 0x7fdc1015ecb0 con 0x7fdc1006a7e0 2023-10-16T12:30:32.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.982+0000 7fdc1521c700 1 --2- 172.21.15.16:0/1899909239 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc1015b2c0 0x7fdc10161340 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c009840 tx=0x7fdc0c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d11dea07416ae3a9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:32.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.982+0000 7fdc07fff700 1 -- 172.21.15.16:0/1899909239 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc0c011830 con 0x7fdc1015b2c0 2023-10-16T12:30:32.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.982+0000 7fdc07fff700 1 -- 172.21.15.16:0/1899909239 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc0c011990 con 0x7fdc1015b2c0 2023-10-16T12:30:32.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.982+0000 7fdc07fff700 1 -- 172.21.15.16:0/1899909239 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc0c011d20 con 0x7fdc1015b2c0 2023-10-16T12:30:32.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.983+0000 7fdc17480700 1 -- 172.21.15.16:0/1899909239 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc1015b2c0 msgr2=0x7fdc10161340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:32.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.983+0000 7fdc17480700 1 --2- 172.21.15.16:0/1899909239 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc1015b2c0 0x7fdc10161340 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c009840 tx=0x7fdc0c009b50 comp rx=0 tx=0).stop 2023-10-16T12:30:32.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.983+0000 7fdc17480700 1 -- 172.21.15.16:0/1899909239 shutdown_connections 2023-10-16T12:30:32.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.983+0000 7fdc17480700 1 --2- 172.21.15.16:0/1899909239 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc1015b2c0 0x7fdc10161340 unknown :-1 s=CLOSED pgs=1581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:32.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.983+0000 7fdc17480700 1 -- 172.21.15.16:0/1899909239 >> 172.21.15.16:0/1899909239 conn(0x7fdc100b1f40 msgr2=0x7fdc100b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:32.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.983+0000 7fdc17480700 1 -- 172.21.15.16:0/1899909239 shutdown_connections 2023-10-16T12:30:32.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.983+0000 7fdc17480700 1 -- 172.21.15.16:0/1899909239 wait complete. 2023-10-16T12:30:32.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.984+0000 7fdc17480700 1 Processor -- start 2023-10-16T12:30:32.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.984+0000 7fdc17480700 1 -- start start 2023-10-16T12:30:32.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.985+0000 7fdc17480700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdc1006a7e0 0x7fdc101577d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:32.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.985+0000 7fdc17480700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdc1006e570 0x7fdc100c6be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:32.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.985+0000 7fdc17480700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc1015b2c0 0x7fdc10159190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:32.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.985+0000 7fdc17480700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc100b5f30 con 0x7fdc1015b2c0 2023-10-16T12:30:32.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.985+0000 7fdc17480700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc10157d10 con 0x7fdc1006a7e0 2023-10-16T12:30:32.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.985+0000 7fdc17480700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc101f2cb0 con 0x7fdc1006e570 2023-10-16T12:30:32.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.985+0000 7fdc15a1d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc1015b2c0 0x7fdc10159190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:32.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.985+0000 7fdc1521c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdc1006a7e0 0x7fdc101577d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:32.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.985+0000 7fdc1521c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdc1006a7e0 0x7fdc101577d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:44564/0 (socket says 172.21.15.16:44564) 2023-10-16T12:30:32.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.985+0000 7fdc15a1d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc1015b2c0 0x7fdc10159190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45506/0 (socket says 172.21.15.16:45506) 2023-10-16T12:30:32.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.985+0000 7fdc1521c700 1 -- 172.21.15.16:0/2356736340 learned_addr learned my addr 172.21.15.16:0/2356736340 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:32.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.985+0000 7fdc14a1b700 1 --2- 172.21.15.16:0/2356736340 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdc1006e570 0x7fdc100c6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:32.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.985+0000 7fdc15a1d700 1 -- 172.21.15.16:0/2356736340 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdc1006e570 msgr2=0x7fdc100c6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:32.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.985+0000 7fdc15a1d700 1 --2- 172.21.15.16:0/2356736340 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdc1006e570 0x7fdc100c6be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:32.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.985+0000 7fdc15a1d700 1 -- 172.21.15.16:0/2356736340 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdc1006a7e0 msgr2=0x7fdc101577d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:32.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.985+0000 7fdc15a1d700 1 --2- 172.21.15.16:0/2356736340 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdc1006a7e0 0x7fdc101577d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:32.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.985+0000 7fdc15a1d700 1 -- 172.21.15.16:0/2356736340 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc0c0094a0 con 0x7fdc1015b2c0 2023-10-16T12:30:32.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.986+0000 7fdc14a1b700 1 --2- 172.21.15.16:0/2356736340 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdc1006e570 0x7fdc100c6be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:30:32.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.986+0000 7fdc1521c700 1 --2- 172.21.15.16:0/2356736340 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdc1006a7e0 0x7fdc101577d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:30:32.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.986+0000 7fdc15a1d700 1 --2- 172.21.15.16:0/2356736340 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc1015b2c0 0x7fdc10159190 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7fdc0800cbe0 tx=0x7fdc0800cef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:32.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.986+0000 7fdc05ffb700 1 -- 172.21.15.16:0/2356736340 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc0800aa40 con 0x7fdc1015b2c0 2023-10-16T12:30:32.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.986+0000 7fdc05ffb700 1 -- 172.21.15.16:0/2356736340 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc0800abc0 con 0x7fdc1015b2c0 2023-10-16T12:30:32.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.986+0000 7fdc17480700 1 -- 172.21.15.16:0/2356736340 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc100b5c50 con 0x7fdc1015b2c0 2023-10-16T12:30:32.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.986+0000 7fdc05ffb700 1 -- 172.21.15.16:0/2356736340 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc080070d0 con 0x7fdc1015b2c0 2023-10-16T12:30:32.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.986+0000 7fdc17480700 1 -- 172.21.15.16:0/2356736340 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc101f31a0 con 0x7fdc1015b2c0 2023-10-16T12:30:32.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.987+0000 7fdc17480700 1 -- 172.21.15.16:0/2356736340 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc100735f0 con 0x7fdc1015b2c0 2023-10-16T12:30:32.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.987+0000 7fdc05ffb700 1 -- 172.21.15.16:0/2356736340 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 75) v1 ==== 89348+0+0 (secure 0 0 0) 0x7fdc08007230 con 0x7fdc1015b2c0 2023-10-16T12:30:32.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.988+0000 7fdc05ffb700 1 --2- 172.21.15.16:0/2356736340 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7fdbf406b070 0x7fdbf406d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:32.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.988+0000 7fdc1521c700 1 -- 172.21.15.16:0/2356736340 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7fdbf406b070 msgr2=0x7fdbf406d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/258291157 2023-10-16T12:30:32.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.988+0000 7fdc05ffb700 1 -- 172.21.15.16:0/2356736340 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 6085+0+0 (secure 0 0 0) 0x7fdc08089610 con 0x7fdc1015b2c0 2023-10-16T12:30:32.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.988+0000 7fdc1521c700 1 --2- 172.21.15.16:0/2356736340 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7fdbf406b070 0x7fdbf406d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:30:32.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:32.992+0000 7fdc05ffb700 1 -- 172.21.15.16:0/2356736340 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdc08054430 con 0x7fdc1015b2c0 2023-10-16T12:30:33.007 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:33.005+0000 7f98ef0b1200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T12:30:33.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.189+0000 7fdc1521c700 1 -- 172.21.15.16:0/2356736340 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7fdbf406b070 msgr2=0x7fdbf406d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/258291157 2023-10-16T12:30:33.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.189+0000 7fdc1521c700 1 --2- 172.21.15.16:0/2356736340 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7fdbf406b070 0x7fdbf406d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:30:33.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.197+0000 7fdc17480700 1 -- 172.21.15.16:0/2356736340 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fdc1004ebb0 con 0x7fdc1015b2c0 2023-10-16T12:30:33.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.200+0000 7fdc05ffb700 1 -- 172.21.15.16:0/2356736340 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v48)=0 v48) v1 ==== 133+0+0 (secure 0 0 0) 0x7fdc08057a50 con 0x7fdc1015b2c0 2023-10-16T12:30:33.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.202+0000 7fdc17480700 1 -- 172.21.15.16:0/2356736340 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7fdbf406b070 msgr2=0x7fdbf406d530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:30:33.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.202+0000 7fdc17480700 1 --2- 172.21.15.16:0/2356736340 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7fdbf406b070 0x7fdbf406d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:33.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.202+0000 7fdc17480700 1 -- 172.21.15.16:0/2356736340 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc1015b2c0 msgr2=0x7fdc10159190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:33.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.202+0000 7fdc17480700 1 --2- 172.21.15.16:0/2356736340 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc1015b2c0 0x7fdc10159190 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7fdc0800cbe0 tx=0x7fdc0800cef0 comp rx=0 tx=0).stop 2023-10-16T12:30:33.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.202+0000 7fdc17480700 1 -- 172.21.15.16:0/2356736340 shutdown_connections 2023-10-16T12:30:33.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.202+0000 7fdc17480700 1 --2- 172.21.15.16:0/2356736340 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc1015b2c0 0x7fdc10159190 unknown :-1 s=CLOSED pgs=1582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:33.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.202+0000 7fdc17480700 1 --2- 172.21.15.16:0/2356736340 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7fdbf406b070 0x7fdbf406d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:33.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.202+0000 7fdc17480700 1 --2- 172.21.15.16:0/2356736340 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdc1006a7e0 0x7fdc101577d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:33.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.202+0000 7fdc17480700 1 --2- 172.21.15.16:0/2356736340 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdc1006e570 0x7fdc100c6be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:33.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.202+0000 7fdc17480700 1 -- 172.21.15.16:0/2356736340 >> 172.21.15.16:0/2356736340 conn(0x7fdc100b1f40 msgr2=0x7fdc100cb220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:33.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.202+0000 7fdc17480700 1 -- 172.21.15.16:0/2356736340 shutdown_connections 2023-10-16T12:30:33.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.202+0000 7fdc17480700 1 -- 172.21.15.16:0/2356736340 wait complete. 2023-10-16T12:30:33.217 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:30:33.218 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/minutes) 2023-10-16T12:30:33.242 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/minutes 1m 2023-10-16T12:30:33.323 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:30:33.321+0000 7f8c2bb99200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T12:30:33.357 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:33.355+0000 7f98ef0b1200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T12:30:33.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.410+0000 7f3e941fa700 1 Processor -- start 2023-10-16T12:30:33.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.410+0000 7f3e941fa700 1 -- start start 2023-10-16T12:30:33.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.411+0000 7f3e941fa700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3e8c0cf4c0 0x7f3e8c0ca1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:33.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.411+0000 7f3e941fa700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f3e8c06bad0 0x7f3e8c0ca6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:33.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.411+0000 7f3e941fa700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e8c15dda0 con 0x7f3e8c159de0 2023-10-16T12:30:33.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.411+0000 7f3e941fa700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e8c15ea20 con 0x7f3e8c06bad0 2023-10-16T12:30:33.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.411+0000 7f3e941fa700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e8c15e080 con 0x7f3e8c0cf4c0 2023-10-16T12:30:33.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.412+0000 7f3e92797700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f3e8c06bad0 0x7f3e8c0ca6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:33.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.412+0000 7f3e91795700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3e8c0cf4c0 0x7f3e8c0ca1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:33.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.412+0000 7f3e91f96700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7f3e8c159de0 0x7f3e8c15a1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:41996/0 (socket says 172.21.15.16:41996) 2023-10-16T12:30:33.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.412+0000 7f3e91f96700 1 -- 172.21.15.16:0/780594727 learned_addr learned my addr 172.21.15.16:0/780594727 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:30:33.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.412+0000 7f3e90f94700 1 -- 172.21.15.16:0/780594727 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3175603140 0 0) 0x7f3e8c15dda0 con 0x7f3e8c159de0 2023-10-16T12:30:33.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.412+0000 7f3e90f94700 1 -- 172.21.15.16:0/780594727 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e780035b0 con 0x7f3e8c159de0 2023-10-16T12:30:33.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.412+0000 7f3e91795700 1 -- 172.21.15.16:0/780594727 >> v2:172.21.15.49:3300/0 conn(0x7f3e8c06bad0 msgr2=0x7f3e8c0ca6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:33.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.412+0000 7f3e91795700 1 --2- 172.21.15.16:0/780594727 >> v2:172.21.15.49:3300/0 conn(0x7f3e8c06bad0 0x7f3e8c0ca6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:33.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.412+0000 7f3e91795700 1 -- 172.21.15.16:0/780594727 >> v1:172.21.15.49:6789/0 conn(0x7f3e8c159de0 legacy=0x7f3e8c15a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:33.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.412+0000 7f3e91795700 1 -- 172.21.15.16:0/780594727 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e8c15e360 con 0x7f3e8c0cf4c0 2023-10-16T12:30:33.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.412+0000 7f3e92797700 1 --2- 172.21.15.16:0/780594727 >> v2:172.21.15.49:3300/0 conn(0x7f3e8c06bad0 0x7f3e8c0ca6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:30:33.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.412+0000 7f3e91795700 1 --2- 172.21.15.16:0/780594727 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3e8c0cf4c0 0x7f3e8c0ca1b0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f3e7c009840 tx=0x7f3e7c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7523c49ba9f427d9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:33.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.412+0000 7f3e90f94700 1 -- 172.21.15.16:0/780594727 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e7c011830 con 0x7f3e8c0cf4c0 2023-10-16T12:30:33.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.412+0000 7f3e90f94700 1 -- 172.21.15.16:0/780594727 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e7c011990 con 0x7f3e8c0cf4c0 2023-10-16T12:30:33.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.412+0000 7f3e90f94700 1 -- 172.21.15.16:0/780594727 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e7c011d20 con 0x7f3e8c0cf4c0 2023-10-16T12:30:33.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.413+0000 7f3e941fa700 1 -- 172.21.15.16:0/780594727 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3e8c0cf4c0 msgr2=0x7f3e8c0ca1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:33.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.413+0000 7f3e941fa700 1 --2- 172.21.15.16:0/780594727 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3e8c0cf4c0 0x7f3e8c0ca1b0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f3e7c009840 tx=0x7f3e7c009b50 comp rx=0 tx=0).stop 2023-10-16T12:30:33.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.413+0000 7f3e941fa700 1 -- 172.21.15.16:0/780594727 shutdown_connections 2023-10-16T12:30:33.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.413+0000 7f3e941fa700 1 --2- 172.21.15.16:0/780594727 >> v2:172.21.15.49:3300/0 conn(0x7f3e8c06bad0 0x7f3e8c0ca6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:33.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.413+0000 7f3e941fa700 1 --2- 172.21.15.16:0/780594727 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3e8c0cf4c0 0x7f3e8c0ca1b0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:33.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.413+0000 7f3e941fa700 1 -- 172.21.15.16:0/780594727 >> 172.21.15.16:0/780594727 conn(0x7f3e8c0cc5e0 msgr2=0x7f3e8c0cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:33.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.413+0000 7f3e941fa700 1 -- 172.21.15.16:0/780594727 shutdown_connections 2023-10-16T12:30:33.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.413+0000 7f3e941fa700 1 -- 172.21.15.16:0/780594727 wait complete. 2023-10-16T12:30:33.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.415+0000 7f3e941fa700 1 Processor -- start 2023-10-16T12:30:33.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.415+0000 7f3e941fa700 1 -- start start 2023-10-16T12:30:33.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.416+0000 7f3e941fa700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3e8c06bad0 0x7f3e8c14cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:33.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.416+0000 7f3e941fa700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3e8c0cf4c0 0x7f3e8c15bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:33.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.416+0000 7f3e941fa700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e8c159de0 0x7f3e8c1f2e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:33.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.416+0000 7f3e941fa700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e8c155330 con 0x7f3e8c159de0 2023-10-16T12:30:33.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.416+0000 7f3e941fa700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e8c0adfd0 con 0x7f3e8c0cf4c0 2023-10-16T12:30:33.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.416+0000 7f3e941fa700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e8c154990 con 0x7f3e8c06bad0 2023-10-16T12:30:33.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.416+0000 7f3e92797700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e8c159de0 0x7f3e8c1f2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:33.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.416+0000 7f3e91795700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3e8c0cf4c0 0x7f3e8c15bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:33.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.416+0000 7f3e91f96700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3e8c06bad0 0x7f3e8c14cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:33.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.416+0000 7f3e92797700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e8c159de0 0x7f3e8c1f2e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45550/0 (socket says 172.21.15.16:45550) 2023-10-16T12:30:33.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.416+0000 7f3e92797700 1 -- 172.21.15.16:0/2000945890 learned_addr learned my addr 172.21.15.16:0/2000945890 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:33.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.416+0000 7f3e92797700 1 -- 172.21.15.16:0/2000945890 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3e8c06bad0 msgr2=0x7f3e8c14cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:33.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.416+0000 7f3e92797700 1 --2- 172.21.15.16:0/2000945890 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3e8c06bad0 0x7f3e8c14cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:33.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.416+0000 7f3e92797700 1 -- 172.21.15.16:0/2000945890 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3e8c0cf4c0 msgr2=0x7f3e8c15bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:33.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.416+0000 7f3e92797700 1 --2- 172.21.15.16:0/2000945890 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3e8c0cf4c0 0x7f3e8c15bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:33.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.416+0000 7f3e92797700 1 -- 172.21.15.16:0/2000945890 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e84009a70 con 0x7f3e8c159de0 2023-10-16T12:30:33.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.416+0000 7f3e91f96700 1 --2- 172.21.15.16:0/2000945890 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3e8c06bad0 0x7f3e8c14cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:30:33.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.416+0000 7f3e92797700 1 --2- 172.21.15.16:0/2000945890 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e8c159de0 0x7f3e8c1f2e70 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f3e84009eb0 tx=0x7f3e8400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:33.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.417+0000 7f3e827fc700 1 -- 172.21.15.16:0/2000945890 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e84009d50 con 0x7f3e8c159de0 2023-10-16T12:30:33.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.417+0000 7f3e941fa700 1 -- 172.21.15.16:0/2000945890 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e7c0094a0 con 0x7f3e8c159de0 2023-10-16T12:30:33.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.417+0000 7f3e827fc700 1 -- 172.21.15.16:0/2000945890 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e8400f040 con 0x7f3e8c159de0 2023-10-16T12:30:33.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.417+0000 7f3e827fc700 1 -- 172.21.15.16:0/2000945890 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e8400ceb0 con 0x7f3e8c159de0 2023-10-16T12:30:33.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.417+0000 7f3e941fa700 1 -- 172.21.15.16:0/2000945890 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e8c154f50 con 0x7f3e8c159de0 2023-10-16T12:30:33.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.418+0000 7f3e941fa700 1 -- 172.21.15.16:0/2000945890 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e8c04ebb0 con 0x7f3e8c159de0 2023-10-16T12:30:33.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.418+0000 7f3e827fc700 1 -- 172.21.15.16:0/2000945890 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 75) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f3e84014480 con 0x7f3e8c159de0 2023-10-16T12:30:33.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.419+0000 7f3e827fc700 1 --2- 172.21.15.16:0/2000945890 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f3e7006b070 0x7f3e7006d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:33.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.419+0000 7f3e91f96700 1 -- 172.21.15.16:0/2000945890 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f3e7006b070 msgr2=0x7f3e7006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/258291157 2023-10-16T12:30:33.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.419+0000 7f3e827fc700 1 -- 172.21.15.16:0/2000945890 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 6085+0+0 (secure 0 0 0) 0x7f3e8408a660 con 0x7f3e8c159de0 2023-10-16T12:30:33.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.419+0000 7f3e91f96700 1 --2- 172.21.15.16:0/2000945890 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f3e7006b070 0x7f3e7006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:30:33.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.423+0000 7f3e827fc700 1 -- 172.21.15.16:0/2000945890 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3e84055400 con 0x7f3e8c159de0 2023-10-16T12:30:33.444 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:30:33.442+0000 7f8c2bb99200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T12:30:33.482 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:33.480+0000 7f98ef0b1200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T12:30:33.522 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:30:33.520+0000 7f8c2bb99200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T12:30:33.560 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:33.558+0000 7f98ef0b1200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T12:30:33.607 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:30:33.604+0000 7f8c2bb99200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T12:30:33.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.620+0000 7f3e91f96700 1 -- 172.21.15.16:0/2000945890 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f3e7006b070 msgr2=0x7f3e7006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/258291157 2023-10-16T12:30:33.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.620+0000 7f3e91f96700 1 --2- 172.21.15.16:0/2000945890 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f3e7006b070 0x7f3e7006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:30:33.646 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:33.644+0000 7f98ef0b1200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T12:30:33.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:33.657+0000 7f3e941fa700 1 -- 172.21.15.16:0/2000945890 --> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/minutes", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f3e8c15da30 con 0x7f3e7006b070 2023-10-16T12:30:33.994 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:30:33.992+0000 7f8c2bb99200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T12:30:34.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:34.021+0000 7f3e91f96700 1 -- 172.21.15.16:0/2000945890 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f3e7006b070 msgr2=0x7f3e7006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/258291157 2023-10-16T12:30:34.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:34.021+0000 7f3e91f96700 1 --2- 172.21.15.16:0/2000945890 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f3e7006b070 0x7f3e7006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-16T12:30:34.036 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:34.034+0000 7f98ef0b1200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T12:30:34.601 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:30:34.599+0000 7f8c2bb99200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T12:30:34.649 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:34.647+0000 7f98ef0b1200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T12:30:34.674 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:30:34.672+0000 7f8c2bb99200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T12:30:34.723 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:34.721+0000 7f98ef0b1200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T12:30:34.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:34.822+0000 7f3e91f96700 1 -- 172.21.15.16:0/2000945890 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f3e7006b070 msgr2=0x7f3e7006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/258291157 2023-10-16T12:30:34.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:34.822+0000 7f3e91f96700 1 --2- 172.21.15.16:0/2000945890 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f3e7006b070 0x7f3e7006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-16T12:30:35.337 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:30:35.335+0000 7f8c2bb99200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:30:35.396 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:35.394+0000 7f98ef0b1200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:30:35.418 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:30:35.416+0000 7f8c2bb99200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T12:30:35.478 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:35.476+0000 7f98ef0b1200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T12:30:35.569 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:30:35.567+0000 7f8c2bb99200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:30:35.633 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:35.631+0000 7f98ef0b1200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:30:35.805 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:30:35.803+0000 7f8c2bb99200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T12:30:35.874 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:35.872+0000 7f98ef0b1200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T12:30:35.882 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:30:35.880+0000 7f8c2bb99200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T12:30:35.953 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:35.951+0000 7f98ef0b1200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T12:30:36.158 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:30:36.156+0000 7f8c2bb99200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T12:30:36.232 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:36.230+0000 7f98ef0b1200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T12:30:36.390 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:30:36.388+0000 7f8c2bb99200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T12:30:36.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:36.423+0000 7f3e91f96700 1 -- 172.21.15.16:0/2000945890 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f3e7006b070 msgr2=0x7f3e7006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/258291157 2023-10-16T12:30:36.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:36.423+0000 7f3e91f96700 1 --2- 172.21.15.16:0/2000945890 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f3e7006b070 0x7f3e7006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-10-16T12:30:36.469 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:36.467+0000 7f98ef0b1200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T12:30:36.547 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:30:36.545+0000 7f8c2bb99200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T12:30:36.628 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:36.627+0000 7f98ef0b1200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T12:30:36.667 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:30:36.665+0000 7f8c2bb99200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T12:30:36.750 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:36.748+0000 7f98ef0b1200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T12:30:36.831 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:30:36.828+0000 7f8c2bb99200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T12:30:36.915 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:36.914+0000 7f98ef0b1200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T12:30:37.102 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:30:37.100+0000 7f8c2bb99200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T12:30:37.190 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:37.188+0000 7f98ef0b1200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T12:30:37.270 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:30:37.268+0000 7f8c2bb99200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T12:30:37.362 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:37.360+0000 7f98ef0b1200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T12:30:37.826 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:30:37.824+0000 7f8c2bb99200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T12:30:37.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:37.894+0000 7f3e827fc700 1 -- 172.21.15.16:0/2000945890 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 76) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f3e84088a00 con 0x7f3e8c159de0 2023-10-16T12:30:37.929 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:37.927+0000 7f98ef0b1200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T12:30:37.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:37.947+0000 7f3e827fc700 1 -- 172.21.15.16:0/2000945890 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mgrmap(e 77) v1 ==== 88957+0+0 (secure 0 0 0) 0x7f3e840557d0 con 0x7f3e8c159de0 2023-10-16T12:30:37.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:37.947+0000 7f3e827fc700 1 -- 172.21.15.16:0/2000945890 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f3e7006b070 msgr2=0x7f3e7006d530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:30:37.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:37.947+0000 7f3e827fc700 1 --2- 172.21.15.16:0/2000945890 >> [v2:172.21.15.16:6832/258291157,v1:172.21.15.16:6833/258291157] conn(0x7f3e7006b070 0x7f3e7006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:38.093 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:38.091+0000 7f9865092700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:30:38.093 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:38.091+0000 7f9865092700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:30:38.093 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:38.091+0000 7f9865092700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:30:38.094 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:38.091+0000 7f9865092700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:30:38.094 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:38.091+0000 7f986208c700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:30:38.094 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:38.091+0000 7f986208c700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:30:38.094 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:38.091+0000 7f986208c700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:30:38.094 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:38.091+0000 7f986208c700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:30:38.095 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:38.091+0000 7f986208c700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:30:38.095 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:38.091+0000 7f9865092700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:30:38.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:38.951+0000 7f3e827fc700 1 -- 172.21.15.16:0/2000945890 <== mon.0 v2:172.21.15.16:3300/0 9 ==== mgrmap(e 78) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f3e840560a0 con 0x7f3e8c159de0 2023-10-16T12:30:38.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:38.952+0000 7f3e827fc700 1 --2- 172.21.15.16:0/2000945890 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f3e7001d250 0x7f3e70075500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:38.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:38.952+0000 7f3e827fc700 1 -- 172.21.15.16:0/2000945890 --> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/minutes", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f3e7001d630 con 0x7f3e7001d250 2023-10-16T12:30:38.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:38.955+0000 7f3e91f96700 1 --2- 172.21.15.16:0/2000945890 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f3e7001d250 0x7f3e70075500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:38.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:38.957+0000 7f3e91f96700 1 --2- 172.21.15.16:0/2000945890 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f3e7001d250 0x7f3e70075500 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3e88007fd0 tx=0x7f3e88007dc0 comp rx=0 tx=0).ready entity=mgr.8820 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:39.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.021+0000 7f3e827fc700 1 -- 172.21.15.16:0/2000945890 <== mgr.8820 v2:172.21.15.16:6832/3783161600 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+45 (secure 0 0 0) 0x7f3e7001d630 con 0x7f3e7001d250 2023-10-16T12:30:39.024 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule set for path snap_test_dir1/minutes 2023-10-16T12:30:39.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.023+0000 7f3e941fa700 1 -- 172.21.15.16:0/2000945890 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f3e7001d250 msgr2=0x7f3e70075500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:39.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.023+0000 7f3e941fa700 1 --2- 172.21.15.16:0/2000945890 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f3e7001d250 0x7f3e70075500 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3e88007fd0 tx=0x7f3e88007dc0 comp rx=0 tx=0).stop 2023-10-16T12:30:39.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.023+0000 7f3e941fa700 1 -- 172.21.15.16:0/2000945890 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e8c159de0 msgr2=0x7f3e8c1f2e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:39.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.023+0000 7f3e941fa700 1 --2- 172.21.15.16:0/2000945890 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e8c159de0 0x7f3e8c1f2e70 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f3e84009eb0 tx=0x7f3e8400c3a0 comp rx=0 tx=0).stop 2023-10-16T12:30:39.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.023+0000 7f3e92797700 1 -- 172.21.15.16:0/2000945890 reap_dead start 2023-10-16T12:30:39.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.024+0000 7f3e941fa700 1 -- 172.21.15.16:0/2000945890 shutdown_connections 2023-10-16T12:30:39.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.024+0000 7f3e941fa700 1 -- 172.21.15.16:0/2000945890 >> 172.21.15.16:0/2000945890 conn(0x7f3e8c0cc5e0 msgr2=0x7f3e8c158760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:39.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.024+0000 7f3e941fa700 1 -- 172.21.15.16:0/2000945890 shutdown_connections 2023-10-16T12:30:39.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.024+0000 7f3e941fa700 1 -- 172.21.15.16:0/2000945890 wait complete. 2023-10-16T12:30:39.039 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/minutes 2023-10-16T12:30:39.040 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:30:39.040 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/hourly) 2023-10-16T12:30:39.059 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/hourly 1h 2023-10-16T12:30:39.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.228+0000 7f0f1dcea700 1 Processor -- start 2023-10-16T12:30:39.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.228+0000 7f0f1dcea700 1 -- start start 2023-10-16T12:30:39.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.229+0000 7f0f1dcea700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f0f1815b070 0x7f0f1815b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:39.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.229+0000 7f0f1dcea700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f180cb880 0x7f0f180cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:39.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.229+0000 7f0f1dcea700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f180cef70 con 0x7f0f1806da60 2023-10-16T12:30:39.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.229+0000 7f0f1dcea700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f180ce360 con 0x7f0f1815b070 2023-10-16T12:30:39.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.229+0000 7f0f1dcea700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f180ce640 con 0x7f0f180cb880 2023-10-16T12:30:39.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.229+0000 7f0f17fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0f1806da60 0x7f0f180cf400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35014/0 (socket says 172.21.15.16:35014) 2023-10-16T12:30:39.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.229+0000 7f0f17fff700 1 -- 172.21.15.16:0/299492642 learned_addr learned my addr 172.21.15.16:0/299492642 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:30:39.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.229+0000 7f0f16ffd700 1 --2- 172.21.15.16:0/299492642 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f180cb880 0x7f0f180cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:39.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.229+0000 7f0f177fe700 1 --2- 172.21.15.16:0/299492642 >> v2:172.21.15.49:3300/0 conn(0x7f0f1815b070 0x7f0f1815b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:39.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.229+0000 7f0f167fc700 1 -- 172.21.15.16:0/299492642 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1713496626 0 0) 0x7f0f180cef70 con 0x7f0f1806da60 2023-10-16T12:30:39.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.229+0000 7f0f167fc700 1 -- 172.21.15.16:0/299492642 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f0c0035b0 con 0x7f0f1806da60 2023-10-16T12:30:39.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.230+0000 7f0f167fc700 1 -- 172.21.15.16:0/299492642 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2325891161 0 0) 0x7f0f0c0035b0 con 0x7f0f1806da60 2023-10-16T12:30:39.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.230+0000 7f0f167fc700 1 -- 172.21.15.16:0/299492642 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f180cb880 msgr2=0x7f0f180cbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:39.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.230+0000 7f0f167fc700 1 --2- 172.21.15.16:0/299492642 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f180cb880 0x7f0f180cbd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:39.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.230+0000 7f0f167fc700 1 -- 172.21.15.16:0/299492642 >> v2:172.21.15.49:3300/0 conn(0x7f0f1815b070 msgr2=0x7f0f1815b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:39.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.230+0000 7f0f167fc700 1 --2- 172.21.15.16:0/299492642 >> v2:172.21.15.49:3300/0 conn(0x7f0f1815b070 0x7f0f1815b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:39.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.230+0000 7f0f167fc700 1 -- 172.21.15.16:0/299492642 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f180ce920 con 0x7f0f1806da60 2023-10-16T12:30:39.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.230+0000 7f0f167fc700 1 -- 172.21.15.16:0/299492642 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f0f100019f0 con 0x7f0f1806da60 2023-10-16T12:30:39.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.230+0000 7f0f167fc700 1 -- 172.21.15.16:0/299492642 >> v1:172.21.15.16:6789/0 conn(0x7f0f1806da60 legacy=0x7f0f180cf400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:39.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.230+0000 7f0f167fc700 1 --2- 172.21.15.16:0/299492642 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f0c003a00 0x7f0f0c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:39.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.230+0000 7f0f167fc700 1 --2- 172.21.15.16:0/299492642 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f180cb880 0x7f0f0c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:39.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.230+0000 7f0f167fc700 1 --2- 172.21.15.16:0/299492642 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f0c004d90 0x7f0f0c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:39.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.230+0000 7f0f167fc700 1 -- 172.21.15.16:0/299492642 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f180ce360 con 0x7f0f0c003a00 2023-10-16T12:30:39.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.230+0000 7f0f167fc700 1 -- 172.21.15.16:0/299492642 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f180ce640 con 0x7f0f0c004d90 2023-10-16T12:30:39.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.230+0000 7f0f167fc700 1 -- 172.21.15.16:0/299492642 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f0c00dc20 con 0x7f0f180cb880 2023-10-16T12:30:39.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.230+0000 7f0f177fe700 1 --2- 172.21.15.16:0/299492642 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f0c003a00 0x7f0f0c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:39.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.230+0000 7f0f16ffd700 1 --2- 172.21.15.16:0/299492642 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f180cb880 0x7f0f0c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:39.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.230+0000 7f0f17fff700 1 --2- 172.21.15.16:0/299492642 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f0c004d90 0x7f0f0c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:39.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.230+0000 7f0f177fe700 1 -- 172.21.15.16:0/299492642 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f180cb880 msgr2=0x7f0f0c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:39.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.230+0000 7f0f177fe700 1 --2- 172.21.15.16:0/299492642 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f180cb880 0x7f0f0c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:39.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.230+0000 7f0f177fe700 1 -- 172.21.15.16:0/299492642 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f0c004d90 msgr2=0x7f0f0c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:39.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.230+0000 7f0f177fe700 1 --2- 172.21.15.16:0/299492642 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f0c004d90 0x7f0f0c005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:39.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.230+0000 7f0f177fe700 1 -- 172.21.15.16:0/299492642 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f0c0035b0 con 0x7f0f0c003a00 2023-10-16T12:30:39.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.230+0000 7f0f17fff700 1 --2- 172.21.15.16:0/299492642 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f0c004d90 0x7f0f0c005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:30:39.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.230+0000 7f0f177fe700 1 --2- 172.21.15.16:0/299492642 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f0c003a00 0x7f0f0c003e20 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f0f08009b60 tx=0x7f0f0c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3098d5c741939751 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:39.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.230+0000 7f0f167fc700 1 -- 172.21.15.16:0/299492642 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f0800e6c0 con 0x7f0f0c003a00 2023-10-16T12:30:39.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.231+0000 7f0f167fc700 1 -- 172.21.15.16:0/299492642 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f0800e820 con 0x7f0f0c003a00 2023-10-16T12:30:39.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.231+0000 7f0f167fc700 1 -- 172.21.15.16:0/299492642 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f0800eb90 con 0x7f0f0c003a00 2023-10-16T12:30:39.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.231+0000 7f0f1dcea700 1 -- 172.21.15.16:0/299492642 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f0c003a00 msgr2=0x7f0f0c003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:39.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.231+0000 7f0f1dcea700 1 --2- 172.21.15.16:0/299492642 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f0c003a00 0x7f0f0c003e20 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f0f08009b60 tx=0x7f0f0c005880 comp rx=0 tx=0).stop 2023-10-16T12:30:39.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.231+0000 7f0f17fff700 1 -- 172.21.15.16:0/299492642 reap_dead start 2023-10-16T12:30:39.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.231+0000 7f0f1dcea700 1 -- 172.21.15.16:0/299492642 shutdown_connections 2023-10-16T12:30:39.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.231+0000 7f0f1dcea700 1 -- 172.21.15.16:0/299492642 >> 172.21.15.16:0/299492642 conn(0x7f0f180b1f40 msgr2=0x7f0f180b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:39.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.231+0000 7f0f1dcea700 1 -- 172.21.15.16:0/299492642 shutdown_connections 2023-10-16T12:30:39.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.231+0000 7f0f1dcea700 1 -- 172.21.15.16:0/299492642 wait complete. 2023-10-16T12:30:39.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.232+0000 7f0f1dcea700 1 Processor -- start 2023-10-16T12:30:39.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.232+0000 7f0f1dcea700 1 -- start start 2023-10-16T12:30:39.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.233+0000 7f0f1dcea700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f0c003a00 0x7f0f181591b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:39.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.233+0000 7f0f1dcea700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f18152e70 0x7f0f181596f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:39.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.233+0000 7f0f1dcea700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f180ae890 0x7f0f180aed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:39.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.233+0000 7f0f1dcea700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1815c880 con 0x7f0f18152e70 2023-10-16T12:30:39.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.233+0000 7f0f1dcea700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1815bc70 con 0x7f0f180ae890 2023-10-16T12:30:39.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.233+0000 7f0f1dcea700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1815bf50 con 0x7f0f0c003a00 2023-10-16T12:30:39.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.233+0000 7f0f16ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f18152e70 0x7f0f181596f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:39.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.233+0000 7f0f16ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f18152e70 0x7f0f181596f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45750/0 (socket says 172.21.15.16:45750) 2023-10-16T12:30:39.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.233+0000 7f0f16ffd700 1 -- 172.21.15.16:0/3120301993 learned_addr learned my addr 172.21.15.16:0/3120301993 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:39.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.233+0000 7f0f17fff700 1 --2- 172.21.15.16:0/3120301993 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f180ae890 0x7f0f180aed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:39.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.233+0000 7f0f177fe700 1 --2- 172.21.15.16:0/3120301993 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f0c003a00 0x7f0f181591b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:39.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.234+0000 7f0f16ffd700 1 -- 172.21.15.16:0/3120301993 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f0c003a00 msgr2=0x7f0f181591b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:39.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.234+0000 7f0f16ffd700 1 --2- 172.21.15.16:0/3120301993 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f0c003a00 0x7f0f181591b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:39.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.234+0000 7f0f16ffd700 1 -- 172.21.15.16:0/3120301993 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f180ae890 msgr2=0x7f0f180aed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:39.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.234+0000 7f0f16ffd700 1 --2- 172.21.15.16:0/3120301993 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f180ae890 0x7f0f180aed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:39.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.234+0000 7f0f16ffd700 1 -- 172.21.15.16:0/3120301993 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f1815c230 con 0x7f0f18152e70 2023-10-16T12:30:39.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.234+0000 7f0f16ffd700 1 --2- 172.21.15.16:0/3120301993 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f18152e70 0x7f0f181596f0 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f0f00011aa0 tx=0x7f0f00011db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:39.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.234+0000 7f0f07fff700 1 -- 172.21.15.16:0/3120301993 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f00011ee0 con 0x7f0f18152e70 2023-10-16T12:30:39.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.234+0000 7f0f07fff700 1 -- 172.21.15.16:0/3120301993 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f0000f940 con 0x7f0f18152e70 2023-10-16T12:30:39.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.234+0000 7f0f07fff700 1 -- 172.21.15.16:0/3120301993 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f0000fcf0 con 0x7f0f18152e70 2023-10-16T12:30:39.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.234+0000 7f0f1dcea700 1 -- 172.21.15.16:0/3120301993 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f1815c510 con 0x7f0f18152e70 2023-10-16T12:30:39.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.234+0000 7f0f1dcea700 1 -- 172.21.15.16:0/3120301993 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f181f6640 con 0x7f0f18152e70 2023-10-16T12:30:39.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.235+0000 7f0f1dcea700 1 -- 172.21.15.16:0/3120301993 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f1804ebb0 con 0x7f0f18152e70 2023-10-16T12:30:39.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.236+0000 7f0f07fff700 1 -- 172.21.15.16:0/3120301993 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 78) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f0f00014440 con 0x7f0f18152e70 2023-10-16T12:30:39.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.236+0000 7f0f07fff700 1 --2- 172.21.15.16:0/3120301993 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f0ef806ada0 0x7f0ef806d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:39.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.236+0000 7f0f07fff700 1 -- 172.21.15.16:0/3120301993 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6429+0+0 (secure 0 0 0) 0x7f0f00089b50 con 0x7f0f18152e70 2023-10-16T12:30:39.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.236+0000 7f0f177fe700 1 --2- 172.21.15.16:0/3120301993 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f0ef806ada0 0x7f0ef806d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:39.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.237+0000 7f0f177fe700 1 --2- 172.21.15.16:0/3120301993 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f0ef806ada0 0x7f0ef806d260 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0f080090b0 tx=0x7f0f08008030 comp rx=0 tx=0).ready entity=mgr.8820 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:39.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.241+0000 7f0f07fff700 1 -- 172.21.15.16:0/3120301993 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0f00054d30 con 0x7f0f18152e70 2023-10-16T12:30:39.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.474+0000 7f0f1dcea700 1 -- 172.21.15.16:0/3120301993 --> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/hourly", "snap_schedule": "1h", "target": ["mon-mgr", ""]}) v1 -- 0x7f0f180728d0 con 0x7f0ef806ada0 2023-10-16T12:30:39.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.541+0000 7f0f07fff700 1 -- 172.21.15.16:0/3120301993 <== mgr.8820 v2:172.21.15.16:6832/3783161600 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+44 (secure 0 0 0) 0x7f0f08022040 con 0x7f0ef806ada0 2023-10-16T12:30:39.543 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule set for path snap_test_dir1/hourly 2023-10-16T12:30:39.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.543+0000 7f0f1dcea700 1 -- 172.21.15.16:0/3120301993 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f0ef806ada0 msgr2=0x7f0ef806d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:39.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.543+0000 7f0f1dcea700 1 --2- 172.21.15.16:0/3120301993 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f0ef806ada0 0x7f0ef806d260 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0f080090b0 tx=0x7f0f08008030 comp rx=0 tx=0).stop 2023-10-16T12:30:39.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.543+0000 7f0f1dcea700 1 -- 172.21.15.16:0/3120301993 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f18152e70 msgr2=0x7f0f181596f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:39.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.543+0000 7f0f1dcea700 1 --2- 172.21.15.16:0/3120301993 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f18152e70 0x7f0f181596f0 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f0f00011aa0 tx=0x7f0f00011db0 comp rx=0 tx=0).stop 2023-10-16T12:30:39.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.543+0000 7f0f1dcea700 1 -- 172.21.15.16:0/3120301993 shutdown_connections 2023-10-16T12:30:39.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.543+0000 7f0f1dcea700 1 --2- 172.21.15.16:0/3120301993 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f0ef806ada0 0x7f0ef806d260 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:39.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.543+0000 7f0f1dcea700 1 --2- 172.21.15.16:0/3120301993 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f180ae890 0x7f0f180aed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:39.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.543+0000 7f0f1dcea700 1 --2- 172.21.15.16:0/3120301993 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f0c003a00 0x7f0f181591b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:39.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.543+0000 7f0f1dcea700 1 --2- 172.21.15.16:0/3120301993 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f18152e70 0x7f0f181596f0 unknown :-1 s=CLOSED pgs=1596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:39.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.543+0000 7f0f1dcea700 1 -- 172.21.15.16:0/3120301993 >> 172.21.15.16:0/3120301993 conn(0x7f0f180b1f40 msgr2=0x7f0f181548f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:39.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.543+0000 7f0f1dcea700 1 -- 172.21.15.16:0/3120301993 shutdown_connections 2023-10-16T12:30:39.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.543+0000 7f0f1dcea700 1 -- 172.21.15.16:0/3120301993 wait complete. 2023-10-16T12:30:39.557 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/hourly 2023-10-16T12:30:39.558 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:30:39.558 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/daily) 2023-10-16T12:30:39.577 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/daily 1d 2023-10-16T12:30:39.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.743+0000 7f1c823a6700 1 Processor -- start 2023-10-16T12:30:39.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.743+0000 7f1c823a6700 1 -- start start 2023-10-16T12:30:39.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.744+0000 7f1c823a6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1c7c15ec80 0x7f1c7c0cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:39.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.744+0000 7f1c823a6700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f1c7c0d2110 0x7f1c7c0cd1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:39.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.744+0000 7f1c823a6700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c7c0b2440 con 0x7f1c7c15f060 2023-10-16T12:30:39.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.744+0000 7f1c823a6700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c7c0b2c50 con 0x7f1c7c15ec80 2023-10-16T12:30:39.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.744+0000 7f1c823a6700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c7c0d00e0 con 0x7f1c7c0d2110 2023-10-16T12:30:39.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.744+0000 7f1c7bfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1c7c15ec80 0x7f1c7c0cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:39.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.744+0000 7f1c7b7fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f1c7c15f060 0x7f1c7c0b4440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35030/0 (socket says 172.21.15.16:35030) 2023-10-16T12:30:39.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.744+0000 7f1c7b7fe700 1 -- 172.21.15.16:0/969713032 learned_addr learned my addr 172.21.15.16:0/969713032 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:30:39.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.745+0000 7f1c80943700 1 --2- 172.21.15.16:0/969713032 >> v2:172.21.15.49:3300/0 conn(0x7f1c7c0d2110 0x7f1c7c0cd1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:39.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.745+0000 7f1c7bfff700 1 -- 172.21.15.16:0/969713032 >> v2:172.21.15.49:3300/0 conn(0x7f1c7c0d2110 msgr2=0x7f1c7c0cd1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:39.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.745+0000 7f1c7bfff700 1 --2- 172.21.15.16:0/969713032 >> v2:172.21.15.49:3300/0 conn(0x7f1c7c0d2110 0x7f1c7c0cd1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:39.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.745+0000 7f1c7bfff700 1 -- 172.21.15.16:0/969713032 >> v1:172.21.15.16:6789/0 conn(0x7f1c7c15f060 legacy=0x7f1c7c0b4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:39.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.745+0000 7f1c7affd700 1 -- 172.21.15.16:0/969713032 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3210098088 0 0) 0x7f1c7c0b2440 con 0x7f1c7c15f060 2023-10-16T12:30:39.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.745+0000 7f1c7bfff700 1 -- 172.21.15.16:0/969713032 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c7c0cef60 con 0x7f1c7c15ec80 2023-10-16T12:30:39.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.745+0000 7f1c7bfff700 1 --2- 172.21.15.16:0/969713032 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c7c15ec80 0x7f1c7c0cdec0 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c08bcb0 tx=0x7f1c6c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=768ec4eb540b306e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:39.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.745+0000 7f1c7affd700 1 -- 172.21.15.16:0/969713032 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c6c011830 con 0x7f1c7c15ec80 2023-10-16T12:30:39.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.745+0000 7f1c7affd700 1 -- 172.21.15.16:0/969713032 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c6c011990 con 0x7f1c7c15ec80 2023-10-16T12:30:39.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.745+0000 7f1c7affd700 1 -- 172.21.15.16:0/969713032 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c6c011d20 con 0x7f1c7c15ec80 2023-10-16T12:30:39.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.746+0000 7f1c823a6700 1 -- 172.21.15.16:0/969713032 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c7c15ec80 msgr2=0x7f1c7c0cdec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:39.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.746+0000 7f1c823a6700 1 --2- 172.21.15.16:0/969713032 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c7c15ec80 0x7f1c7c0cdec0 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c08bcb0 tx=0x7f1c6c009b20 comp rx=0 tx=0).stop 2023-10-16T12:30:39.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.746+0000 7f1c823a6700 1 -- 172.21.15.16:0/969713032 shutdown_connections 2023-10-16T12:30:39.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.746+0000 7f1c823a6700 1 --2- 172.21.15.16:0/969713032 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c7c15ec80 0x7f1c7c0cdec0 unknown :-1 s=CLOSED pgs=1598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:39.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.746+0000 7f1c823a6700 1 --2- 172.21.15.16:0/969713032 >> v2:172.21.15.49:3300/0 conn(0x7f1c7c0d2110 0x7f1c7c0cd1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:39.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.746+0000 7f1c823a6700 1 -- 172.21.15.16:0/969713032 >> 172.21.15.16:0/969713032 conn(0x7f1c7c0d15f0 msgr2=0x7f1c7c0d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:39.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.746+0000 7f1c823a6700 1 -- 172.21.15.16:0/969713032 shutdown_connections 2023-10-16T12:30:39.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.746+0000 7f1c823a6700 1 -- 172.21.15.16:0/969713032 wait complete. 2023-10-16T12:30:39.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.747+0000 7f1c823a6700 1 Processor -- start 2023-10-16T12:30:39.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.747+0000 7f1c823a6700 1 -- start start 2023-10-16T12:30:39.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.748+0000 7f1c823a6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c7c0d2110 0x7f1c7c06fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:39.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.748+0000 7f1c823a6700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1c7c15ec80 0x7f1c7c1543a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:39.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.748+0000 7f1c823a6700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1c7c15f060 0x7f1c7c1f68f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:39.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.748+0000 7f1c823a6700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c7c154d60 con 0x7f1c7c0d2110 2023-10-16T12:30:39.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.748+0000 7f1c823a6700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c7c0c9da0 con 0x7f1c7c15f060 2023-10-16T12:30:39.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.748+0000 7f1c823a6700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c7c0c94b0 con 0x7f1c7c15ec80 2023-10-16T12:30:39.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.748+0000 7f1c7bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c7c0d2110 0x7f1c7c06fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:39.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.748+0000 7f1c7bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c7c0d2110 0x7f1c7c06fa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45760/0 (socket says 172.21.15.16:45760) 2023-10-16T12:30:39.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.748+0000 7f1c7bfff700 1 -- 172.21.15.16:0/912743572 learned_addr learned my addr 172.21.15.16:0/912743572 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:39.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.748+0000 7f1c80943700 1 --2- 172.21.15.16:0/912743572 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1c7c15f060 0x7f1c7c1f68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:39.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.748+0000 7f1c7b7fe700 1 --2- 172.21.15.16:0/912743572 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1c7c15ec80 0x7f1c7c1543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:39.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.748+0000 7f1c7bfff700 1 -- 172.21.15.16:0/912743572 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1c7c15ec80 msgr2=0x7f1c7c1543a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:39.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.748+0000 7f1c7bfff700 1 --2- 172.21.15.16:0/912743572 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1c7c15ec80 0x7f1c7c1543a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:39.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.748+0000 7f1c7bfff700 1 -- 172.21.15.16:0/912743572 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1c7c15f060 msgr2=0x7f1c7c1f68f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:39.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.748+0000 7f1c7bfff700 1 --2- 172.21.15.16:0/912743572 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1c7c15f060 0x7f1c7c1f68f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:39.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.748+0000 7f1c7bfff700 1 -- 172.21.15.16:0/912743572 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c6c0094a0 con 0x7f1c7c0d2110 2023-10-16T12:30:39.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.748+0000 7f1c80943700 1 --2- 172.21.15.16:0/912743572 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1c7c15f060 0x7f1c7c1f68f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:30:39.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.748+0000 7f1c7bfff700 1 --2- 172.21.15.16:0/912743572 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c7c0d2110 0x7f1c7c06fa00 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f1c6c009f20 tx=0x7f1c6c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:39.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.749+0000 7f1c78ff9700 1 -- 172.21.15.16:0/912743572 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c6c0038a0 con 0x7f1c7c0d2110 2023-10-16T12:30:39.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.749+0000 7f1c78ff9700 1 -- 172.21.15.16:0/912743572 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c6c01f070 con 0x7f1c7c0d2110 2023-10-16T12:30:39.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.749+0000 7f1c78ff9700 1 -- 172.21.15.16:0/912743572 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c6c003c40 con 0x7f1c7c0d2110 2023-10-16T12:30:39.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.749+0000 7f1c823a6700 1 -- 172.21.15.16:0/912743572 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c7c154a80 con 0x7f1c7c0d2110 2023-10-16T12:30:39.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.749+0000 7f1c823a6700 1 -- 172.21.15.16:0/912743572 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c7c0c9a30 con 0x7f1c7c0d2110 2023-10-16T12:30:39.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.750+0000 7f1c823a6700 1 -- 172.21.15.16:0/912743572 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c7c06ff40 con 0x7f1c7c0d2110 2023-10-16T12:30:39.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.750+0000 7f1c78ff9700 1 -- 172.21.15.16:0/912743572 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 78) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f1c6c026480 con 0x7f1c7c0d2110 2023-10-16T12:30:39.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.750+0000 7f1c78ff9700 1 --2- 172.21.15.16:0/912743572 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f1c6806ada0 0x7f1c6806d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:39.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.750+0000 7f1c78ff9700 1 -- 172.21.15.16:0/912743572 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6429+0+0 (secure 0 0 0) 0x7f1c6c021070 con 0x7f1c7c0d2110 2023-10-16T12:30:39.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.751+0000 7f1c7b7fe700 1 --2- 172.21.15.16:0/912743572 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f1c6806ada0 0x7f1c6806d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:39.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.751+0000 7f1c7b7fe700 1 --2- 172.21.15.16:0/912743572 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f1c6806ada0 0x7f1c6806d260 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c0b47a0 tx=0x7f1c70007c90 comp rx=0 tx=0).ready entity=mgr.8820 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:39.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.755+0000 7f1c78ff9700 1 -- 172.21.15.16:0/912743572 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1c6c056040 con 0x7f1c7c0d2110 2023-10-16T12:30:39.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:39.989+0000 7f1c823a6700 1 -- 172.21.15.16:0/912743572 --> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/daily", "snap_schedule": "1d", "target": ["mon-mgr", ""]}) v1 -- 0x7f1c7c0b2970 con 0x7f1c6806ada0 2023-10-16T12:30:40.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.057+0000 7f1c78ff9700 1 -- 172.21.15.16:0/912743572 <== mgr.8820 v2:172.21.15.16:6832/3783161600 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f1c70013420 con 0x7f1c6806ada0 2023-10-16T12:30:40.060 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule set for path snap_test_dir1/daily 2023-10-16T12:30:40.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.060+0000 7f1c823a6700 1 -- 172.21.15.16:0/912743572 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f1c6806ada0 msgr2=0x7f1c6806d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:40.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.060+0000 7f1c823a6700 1 --2- 172.21.15.16:0/912743572 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f1c6806ada0 0x7f1c6806d260 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c0b47a0 tx=0x7f1c70007c90 comp rx=0 tx=0).stop 2023-10-16T12:30:40.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.060+0000 7f1c823a6700 1 -- 172.21.15.16:0/912743572 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c7c0d2110 msgr2=0x7f1c7c06fa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:40.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.060+0000 7f1c823a6700 1 --2- 172.21.15.16:0/912743572 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c7c0d2110 0x7f1c7c06fa00 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f1c6c009f20 tx=0x7f1c6c004750 comp rx=0 tx=0).stop 2023-10-16T12:30:40.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.060+0000 7f1c823a6700 1 -- 172.21.15.16:0/912743572 shutdown_connections 2023-10-16T12:30:40.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.060+0000 7f1c823a6700 1 --2- 172.21.15.16:0/912743572 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f1c6806ada0 0x7f1c6806d260 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:40.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.060+0000 7f1c823a6700 1 --2- 172.21.15.16:0/912743572 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1c7c15f060 0x7f1c7c1f68f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:40.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.060+0000 7f1c823a6700 1 --2- 172.21.15.16:0/912743572 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c7c0d2110 0x7f1c7c06fa00 unknown :-1 s=CLOSED pgs=1599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:40.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.060+0000 7f1c823a6700 1 --2- 172.21.15.16:0/912743572 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1c7c15ec80 0x7f1c7c1543a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:40.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.060+0000 7f1c823a6700 1 -- 172.21.15.16:0/912743572 >> 172.21.15.16:0/912743572 conn(0x7f1c7c0d15f0 msgr2=0x7f1c7c156f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:40.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.060+0000 7f1c823a6700 1 -- 172.21.15.16:0/912743572 shutdown_connections 2023-10-16T12:30:40.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.060+0000 7f1c823a6700 1 -- 172.21.15.16:0/912743572 wait complete. 2023-10-16T12:30:40.074 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/daily 2023-10-16T12:30:40.075 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:30:40.075 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/weekly) 2023-10-16T12:30:40.093 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/weekly 1w 2023-10-16T12:30:40.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.260+0000 7ff4b1f23700 1 Processor -- start 2023-10-16T12:30:40.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.260+0000 7ff4b1f23700 1 -- start start 2023-10-16T12:30:40.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.261+0000 7ff4b1f23700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff4ac15b2c0 0x7ff4ac161340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:40.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.261+0000 7ff4b1f23700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7ff4ac06e570 0x7ff4ac15e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:40.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.261+0000 7ff4b1f23700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4ac161880 con 0x7ff4ac06a7e0 2023-10-16T12:30:40.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.261+0000 7ff4b1f23700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4ac15b6a0 con 0x7ff4ac15b2c0 2023-10-16T12:30:40.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.261+0000 7ff4b1f23700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4ac161b20 con 0x7ff4ac06e570 2023-10-16T12:30:40.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.261+0000 7ff4ab7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff4ac15b2c0 0x7ff4ac161340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:40.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.261+0000 7ff4ab7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff4ac15b2c0 0x7ff4ac161340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45784/0 (socket says 172.21.15.16:45784) 2023-10-16T12:30:40.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.261+0000 7ff4ab7fe700 1 -- 172.21.15.16:0/330618401 learned_addr learned my addr 172.21.15.16:0/330618401 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:40.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.261+0000 7ff4aaffd700 1 --2- 172.21.15.16:0/330618401 >> v2:172.21.15.49:3300/0 conn(0x7ff4ac06e570 0x7ff4ac15e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:40.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.261+0000 7ff4ab7fe700 1 -- 172.21.15.16:0/330618401 >> v2:172.21.15.49:3300/0 conn(0x7ff4ac06e570 msgr2=0x7ff4ac15e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:40.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.261+0000 7ff4ab7fe700 1 --2- 172.21.15.16:0/330618401 >> v2:172.21.15.49:3300/0 conn(0x7ff4ac06e570 0x7ff4ac15e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:40.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.261+0000 7ff4ab7fe700 1 -- 172.21.15.16:0/330618401 >> v1:172.21.15.49:6789/0 conn(0x7ff4ac06a7e0 legacy=0x7ff4ac071a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:40.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.261+0000 7ff4ab7fe700 1 -- 172.21.15.16:0/330618401 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4ac161e00 con 0x7ff4ac15b2c0 2023-10-16T12:30:40.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.261+0000 7ff4ab7fe700 1 --2- 172.21.15.16:0/330618401 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4ac15b2c0 0x7ff4ac161340 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac070500 tx=0x7ff49c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f0cb7b51baf6ce1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:40.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.262+0000 7ff4aa7fc700 1 -- 172.21.15.16:0/330618401 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff49c00b870 con 0x7ff4ac15b2c0 2023-10-16T12:30:40.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.262+0000 7ff4aa7fc700 1 -- 172.21.15.16:0/330618401 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff49c00b9d0 con 0x7ff4ac15b2c0 2023-10-16T12:30:40.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.262+0000 7ff4aa7fc700 1 -- 172.21.15.16:0/330618401 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff49c00bd60 con 0x7ff4ac15b2c0 2023-10-16T12:30:40.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.262+0000 7ff4b1f23700 1 -- 172.21.15.16:0/330618401 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4ac15b2c0 msgr2=0x7ff4ac161340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:40.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.262+0000 7ff4b1f23700 1 --2- 172.21.15.16:0/330618401 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4ac15b2c0 0x7ff4ac161340 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac070500 tx=0x7ff49c009bd0 comp rx=0 tx=0).stop 2023-10-16T12:30:40.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.262+0000 7ff4b1f23700 1 -- 172.21.15.16:0/330618401 shutdown_connections 2023-10-16T12:30:40.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.262+0000 7ff4b1f23700 1 --2- 172.21.15.16:0/330618401 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4ac15b2c0 0x7ff4ac161340 unknown :-1 s=CLOSED pgs=1602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:40.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.262+0000 7ff4b1f23700 1 --2- 172.21.15.16:0/330618401 >> v2:172.21.15.49:3300/0 conn(0x7ff4ac06e570 0x7ff4ac15e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:40.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.262+0000 7ff4b1f23700 1 -- 172.21.15.16:0/330618401 >> 172.21.15.16:0/330618401 conn(0x7ff4ac0b1f40 msgr2=0x7ff4ac0b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:40.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.262+0000 7ff4b1f23700 1 -- 172.21.15.16:0/330618401 shutdown_connections 2023-10-16T12:30:40.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.263+0000 7ff4b1f23700 1 -- 172.21.15.16:0/330618401 wait complete. 2023-10-16T12:30:40.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.264+0000 7ff4b1f23700 1 Processor -- start 2023-10-16T12:30:40.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.264+0000 7ff4b1f23700 1 -- start start 2023-10-16T12:30:40.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.265+0000 7ff4b1f23700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff4ac06a7e0 0x7ff4ac0c9720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:40.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.265+0000 7ff4b1f23700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff4ac06e570 0x7ff4ac1f4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:40.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.265+0000 7ff4b1f23700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4ac15b2c0 0x7ff4ac1f4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:40.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.265+0000 7ff4b1f23700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4ac0b5a40 con 0x7ff4ac15b2c0 2023-10-16T12:30:40.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.265+0000 7ff4b1f23700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4ac0c9290 con 0x7ff4ac06e570 2023-10-16T12:30:40.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.265+0000 7ff4b1f23700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4ac0b5c50 con 0x7ff4ac06a7e0 2023-10-16T12:30:40.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.265+0000 7ff4abfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4ac15b2c0 0x7ff4ac1f4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:40.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.265+0000 7ff4abfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4ac15b2c0 0x7ff4ac1f4ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45794/0 (socket says 172.21.15.16:45794) 2023-10-16T12:30:40.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.265+0000 7ff4abfff700 1 -- 172.21.15.16:0/1836438852 learned_addr learned my addr 172.21.15.16:0/1836438852 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:40.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.266+0000 7ff4aaffd700 1 --2- 172.21.15.16:0/1836438852 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff4ac06e570 0x7ff4ac1f4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:40.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.266+0000 7ff4ab7fe700 1 --2- 172.21.15.16:0/1836438852 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff4ac06a7e0 0x7ff4ac0c9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:40.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.266+0000 7ff4abfff700 1 -- 172.21.15.16:0/1836438852 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff4ac06a7e0 msgr2=0x7ff4ac0c9720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:40.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.266+0000 7ff4abfff700 1 --2- 172.21.15.16:0/1836438852 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff4ac06a7e0 0x7ff4ac0c9720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:40.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.266+0000 7ff4abfff700 1 -- 172.21.15.16:0/1836438852 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff4ac06e570 msgr2=0x7ff4ac1f4580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:40.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.266+0000 7ff4abfff700 1 --2- 172.21.15.16:0/1836438852 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff4ac06e570 0x7ff4ac1f4580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:40.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.266+0000 7ff4abfff700 1 -- 172.21.15.16:0/1836438852 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff49c009550 con 0x7ff4ac15b2c0 2023-10-16T12:30:40.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.266+0000 7ff4aaffd700 1 --2- 172.21.15.16:0/1836438852 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff4ac06e570 0x7ff4ac1f4580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:30:40.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.266+0000 7ff4abfff700 1 --2- 172.21.15.16:0/1836438852 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4ac15b2c0 0x7ff4ac1f4ac0 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7ff4a400cb80 tx=0x7ff4a400cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:40.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.266+0000 7ff493fff700 1 -- 172.21.15.16:0/1836438852 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4a4003c00 con 0x7ff4ac15b2c0 2023-10-16T12:30:40.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.266+0000 7ff493fff700 1 -- 172.21.15.16:0/1836438852 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4a4003d60 con 0x7ff4ac15b2c0 2023-10-16T12:30:40.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.266+0000 7ff493fff700 1 -- 172.21.15.16:0/1836438852 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4a400abb0 con 0x7ff4ac15b2c0 2023-10-16T12:30:40.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.266+0000 7ff4b1f23700 1 -- 172.21.15.16:0/1836438852 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4ac0b5f30 con 0x7ff4ac15b2c0 2023-10-16T12:30:40.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.266+0000 7ff4b1f23700 1 -- 172.21.15.16:0/1836438852 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4ac14ff10 con 0x7ff4ac15b2c0 2023-10-16T12:30:40.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.267+0000 7ff4b1f23700 1 -- 172.21.15.16:0/1836438852 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4ac0c9420 con 0x7ff4ac15b2c0 2023-10-16T12:30:40.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.268+0000 7ff493fff700 1 -- 172.21.15.16:0/1836438852 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 78) v1 ==== 89084+0+0 (secure 0 0 0) 0x7ff4a400ad10 con 0x7ff4ac15b2c0 2023-10-16T12:30:40.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.268+0000 7ff493fff700 1 --2- 172.21.15.16:0/1836438852 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7ff49806ada0 0x7ff49806d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:40.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.268+0000 7ff493fff700 1 -- 172.21.15.16:0/1836438852 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6429+0+0 (secure 0 0 0) 0x7ff4a4089660 con 0x7ff4ac15b2c0 2023-10-16T12:30:40.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.268+0000 7ff4ab7fe700 1 --2- 172.21.15.16:0/1836438852 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7ff49806ada0 0x7ff49806d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:40.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.268+0000 7ff4ab7fe700 1 --2- 172.21.15.16:0/1836438852 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7ff49806ada0 0x7ff49806d260 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff49c000c00 tx=0x7ff49c008030 comp rx=0 tx=0).ready entity=mgr.8820 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:40.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.272+0000 7ff493fff700 1 -- 172.21.15.16:0/1836438852 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff4a40546b0 con 0x7ff4ac15b2c0 2023-10-16T12:30:40.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.500+0000 7ff4b1f23700 1 -- 172.21.15.16:0/1836438852 --> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/weekly", "snap_schedule": "1w", "target": ["mon-mgr", ""]}) v1 -- 0x7ff4ac15e080 con 0x7ff49806ada0 2023-10-16T12:30:40.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.565+0000 7ff493fff700 1 -- 172.21.15.16:0/1836438852 <== mgr.8820 v2:172.21.15.16:6832/3783161600 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+44 (secure 0 0 0) 0x7ff49c00bb40 con 0x7ff49806ada0 2023-10-16T12:30:40.567 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule set for path snap_test_dir1/weekly 2023-10-16T12:30:40.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.567+0000 7ff4b1f23700 1 -- 172.21.15.16:0/1836438852 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7ff49806ada0 msgr2=0x7ff49806d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:40.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.567+0000 7ff4b1f23700 1 --2- 172.21.15.16:0/1836438852 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7ff49806ada0 0x7ff49806d260 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff49c000c00 tx=0x7ff49c008030 comp rx=0 tx=0).stop 2023-10-16T12:30:40.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.567+0000 7ff4b1f23700 1 -- 172.21.15.16:0/1836438852 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4ac15b2c0 msgr2=0x7ff4ac1f4ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:40.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.567+0000 7ff4b1f23700 1 --2- 172.21.15.16:0/1836438852 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4ac15b2c0 0x7ff4ac1f4ac0 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7ff4a400cb80 tx=0x7ff4a400cf40 comp rx=0 tx=0).stop 2023-10-16T12:30:40.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.567+0000 7ff4b1f23700 1 -- 172.21.15.16:0/1836438852 shutdown_connections 2023-10-16T12:30:40.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.567+0000 7ff4b1f23700 1 --2- 172.21.15.16:0/1836438852 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7ff49806ada0 0x7ff49806d260 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:40.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.567+0000 7ff4b1f23700 1 --2- 172.21.15.16:0/1836438852 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4ac15b2c0 0x7ff4ac1f4ac0 unknown :-1 s=CLOSED pgs=1603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:40.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.567+0000 7ff4b1f23700 1 --2- 172.21.15.16:0/1836438852 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff4ac06a7e0 0x7ff4ac0c9720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:40.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.567+0000 7ff4b1f23700 1 --2- 172.21.15.16:0/1836438852 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff4ac06e570 0x7ff4ac1f4580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:40.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.567+0000 7ff4b1f23700 1 -- 172.21.15.16:0/1836438852 >> 172.21.15.16:0/1836438852 conn(0x7ff4ac0b1f40 msgr2=0x7ff4ac0cd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:40.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.567+0000 7ff4b1f23700 1 -- 172.21.15.16:0/1836438852 shutdown_connections 2023-10-16T12:30:40.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.567+0000 7ff4b1f23700 1 -- 172.21.15.16:0/1836438852 wait complete. 2023-10-16T12:30:40.581 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/weekly 2023-10-16T12:30:40.582 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:30:40.582 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/monthly) 2023-10-16T12:30:40.601 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/monthly 1M 2023-10-16T12:30:40.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.766+0000 7f6794455700 1 Processor -- start 2023-10-16T12:30:40.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.766+0000 7f6794455700 1 -- start start 2023-10-16T12:30:40.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.767+0000 7f6794455700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f678c159e40 0x7f678c070cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:40.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.767+0000 7f6794455700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f678c0cb940 con 0x7f678c0cbdd0 2023-10-16T12:30:40.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.767+0000 7f6794455700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f678c0ceb60 con 0x7f678c071920 2023-10-16T12:30:40.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.767+0000 7f6794455700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f678c0ce880 con 0x7f678c159e40 2023-10-16T12:30:40.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.767+0000 7f67919f0700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f678c0cbdd0 0x7f678c071210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35072/0 (socket says 172.21.15.16:35072) 2023-10-16T12:30:40.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.767+0000 7f67919f0700 1 -- 172.21.15.16:0/1886361165 learned_addr learned my addr 172.21.15.16:0/1886361165 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:30:40.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.767+0000 7f67911ef700 1 -- 172.21.15.16:0/1886361165 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2286877585 0 0) 0x7f678c0cb940 con 0x7f678c0cbdd0 2023-10-16T12:30:40.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.767+0000 7f67921f1700 1 --2- 172.21.15.16:0/1886361165 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f678c159e40 0x7f678c070cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:40.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.767+0000 7f67911ef700 1 -- 172.21.15.16:0/1886361165 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67780035b0 con 0x7f678c0cbdd0 2023-10-16T12:30:40.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.768+0000 7f67911ef700 1 -- 172.21.15.16:0/1886361165 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2924700840 0 0) 0x7f67780035b0 con 0x7f678c0cbdd0 2023-10-16T12:30:40.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.768+0000 7f67911ef700 1 -- 172.21.15.16:0/1886361165 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f678c159e40 msgr2=0x7f678c070cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:40.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.768+0000 7f67911ef700 1 --2- 172.21.15.16:0/1886361165 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f678c159e40 0x7f678c070cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:40.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.768+0000 7f67911ef700 1 -- 172.21.15.16:0/1886361165 >> v1:172.21.15.49:6789/0 conn(0x7f678c071920 legacy=0x7f678c0cd3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:40.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.768+0000 7f67911ef700 1 -- 172.21.15.16:0/1886361165 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f678c0cb2f0 con 0x7f678c0cbdd0 2023-10-16T12:30:40.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.768+0000 7f67911ef700 1 -- 172.21.15.16:0/1886361165 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f6788001670 con 0x7f678c0cbdd0 2023-10-16T12:30:40.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.768+0000 7f67911ef700 1 -- 172.21.15.16:0/1886361165 >> v1:172.21.15.16:6789/0 conn(0x7f678c0cbdd0 legacy=0x7f678c071210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:40.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.768+0000 7f67911ef700 1 --2- 172.21.15.16:0/1886361165 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f678c159e40 0x7f6778003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:40.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.768+0000 7f67911ef700 1 --2- 172.21.15.16:0/1886361165 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6778003fb0 0x7f6778004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:40.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.768+0000 7f67911ef700 1 --2- 172.21.15.16:0/1886361165 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6778004d90 0x7f6778005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:40.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.768+0000 7f67911ef700 1 -- 172.21.15.16:0/1886361165 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f678c0ce880 con 0x7f6778003fb0 2023-10-16T12:30:40.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.768+0000 7f67911ef700 1 -- 172.21.15.16:0/1886361165 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f677800cc00 con 0x7f6778004d90 2023-10-16T12:30:40.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.768+0000 7f67911ef700 1 -- 172.21.15.16:0/1886361165 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f677800cdc0 con 0x7f678c159e40 2023-10-16T12:30:40.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.768+0000 7f67919f0700 1 --2- 172.21.15.16:0/1886361165 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6778003fb0 0x7f6778004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:40.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.768+0000 7f67921f1700 1 --2- 172.21.15.16:0/1886361165 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f678c159e40 0x7f6778003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:40.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.768+0000 7f67929f2700 1 --2- 172.21.15.16:0/1886361165 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6778004d90 0x7f6778005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:40.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.768+0000 7f67919f0700 1 -- 172.21.15.16:0/1886361165 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f678c159e40 msgr2=0x7f6778003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:40.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.768+0000 7f67919f0700 1 --2- 172.21.15.16:0/1886361165 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f678c159e40 0x7f6778003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:40.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.768+0000 7f67919f0700 1 -- 172.21.15.16:0/1886361165 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6778004d90 msgr2=0x7f6778005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:40.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.768+0000 7f67919f0700 1 --2- 172.21.15.16:0/1886361165 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6778004d90 0x7f6778005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:40.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.768+0000 7f67919f0700 1 -- 172.21.15.16:0/1886361165 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67780035b0 con 0x7f6778003fb0 2023-10-16T12:30:40.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.768+0000 7f67919f0700 1 --2- 172.21.15.16:0/1886361165 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6778003fb0 0x7f6778004850 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7f678800c210 tx=0x7f6778005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=914b64eb94720f27 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:40.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.769+0000 7f67911ef700 1 -- 172.21.15.16:0/1886361165 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6788007c10 con 0x7f6778003fb0 2023-10-16T12:30:40.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.769+0000 7f67911ef700 1 -- 172.21.15.16:0/1886361165 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f678800cdb0 con 0x7f6778003fb0 2023-10-16T12:30:40.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.769+0000 7f67911ef700 1 -- 172.21.15.16:0/1886361165 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6788005ba0 con 0x7f6778003fb0 2023-10-16T12:30:40.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.769+0000 7f6794455700 1 -- 172.21.15.16:0/1886361165 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6778003fb0 msgr2=0x7f6778004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:40.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.769+0000 7f6794455700 1 --2- 172.21.15.16:0/1886361165 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6778003fb0 0x7f6778004850 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7f678800c210 tx=0x7f6778005880 comp rx=0 tx=0).stop 2023-10-16T12:30:40.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.769+0000 7f67929f2700 1 -- 172.21.15.16:0/1886361165 reap_dead start 2023-10-16T12:30:40.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.769+0000 7f6794455700 1 -- 172.21.15.16:0/1886361165 shutdown_connections 2023-10-16T12:30:40.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.769+0000 7f6794455700 1 -- 172.21.15.16:0/1886361165 >> 172.21.15.16:0/1886361165 conn(0x7f678c06cdb0 msgr2=0x7f678c06d1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:40.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.769+0000 7f6794455700 1 -- 172.21.15.16:0/1886361165 shutdown_connections 2023-10-16T12:30:40.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.769+0000 7f6794455700 1 -- 172.21.15.16:0/1886361165 wait complete. 2023-10-16T12:30:40.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.770+0000 7f6794455700 1 Processor -- start 2023-10-16T12:30:40.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.770+0000 7f6794455700 1 -- start start 2023-10-16T12:30:40.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.771+0000 7f6794455700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6778003fb0 0x7f678c1540c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:40.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.771+0000 7f6794455700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f678c1583b0 0x7f678c154870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:40.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.771+0000 7f6794455700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f678c158830 0x7f678c154db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:40.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.771+0000 7f6794455700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f678c155530 con 0x7f678c158830 2023-10-16T12:30:40.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.771+0000 7f6794455700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f678c1570c0 con 0x7f6778003fb0 2023-10-16T12:30:40.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.771+0000 7f6794455700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f678c157f20 con 0x7f678c1583b0 2023-10-16T12:30:40.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.771+0000 7f67929f2700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f678c158830 0x7f678c154db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:40.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.771+0000 7f67929f2700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f678c158830 0x7f678c154db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45814/0 (socket says 172.21.15.16:45814) 2023-10-16T12:30:40.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.771+0000 7f67929f2700 1 -- 172.21.15.16:0/386353339 learned_addr learned my addr 172.21.15.16:0/386353339 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:40.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.771+0000 7f67919f0700 1 --2- 172.21.15.16:0/386353339 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f678c1583b0 0x7f678c154870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:40.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.772+0000 7f67921f1700 1 --2- 172.21.15.16:0/386353339 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6778003fb0 0x7f678c1540c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:40.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.772+0000 7f67929f2700 1 -- 172.21.15.16:0/386353339 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f678c1583b0 msgr2=0x7f678c154870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:40.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.772+0000 7f67929f2700 1 --2- 172.21.15.16:0/386353339 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f678c1583b0 0x7f678c154870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:40.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.772+0000 7f67929f2700 1 -- 172.21.15.16:0/386353339 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6778003fb0 msgr2=0x7f678c1540c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:40.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.772+0000 7f67929f2700 1 --2- 172.21.15.16:0/386353339 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6778003fb0 0x7f678c1540c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:40.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.772+0000 7f67929f2700 1 -- 172.21.15.16:0/386353339 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f678c157310 con 0x7f678c158830 2023-10-16T12:30:40.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.772+0000 7f67921f1700 1 --2- 172.21.15.16:0/386353339 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6778003fb0 0x7f678c1540c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:30:40.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.772+0000 7f67929f2700 1 --2- 172.21.15.16:0/386353339 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f678c158830 0x7f678c154db0 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f678c15db00 tx=0x7f678400d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:40.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.772+0000 7f677effd700 1 -- 172.21.15.16:0/386353339 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6784003c10 con 0x7f678c158830 2023-10-16T12:30:40.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.772+0000 7f677effd700 1 -- 172.21.15.16:0/386353339 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6784003d90 con 0x7f678c158830 2023-10-16T12:30:40.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.772+0000 7f6794455700 1 -- 172.21.15.16:0/386353339 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f678c1575f0 con 0x7f678c158830 2023-10-16T12:30:40.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.772+0000 7f677effd700 1 -- 172.21.15.16:0/386353339 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f678400dbd0 con 0x7f678c158830 2023-10-16T12:30:40.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.772+0000 7f6794455700 1 -- 172.21.15.16:0/386353339 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f678c157bb0 con 0x7f678c158830 2023-10-16T12:30:40.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.773+0000 7f6794455700 1 -- 172.21.15.16:0/386353339 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f678c04ebb0 con 0x7f678c158830 2023-10-16T12:30:40.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.773+0000 7f677effd700 1 -- 172.21.15.16:0/386353339 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 78) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f678400dd30 con 0x7f678c158830 2023-10-16T12:30:40.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.774+0000 7f677effd700 1 --2- 172.21.15.16:0/386353339 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f677006ada0 0x7f677006d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:40.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.774+0000 7f677effd700 1 -- 172.21.15.16:0/386353339 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6429+0+0 (secure 0 0 0) 0x7f67840897e0 con 0x7f678c158830 2023-10-16T12:30:40.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.774+0000 7f67921f1700 1 --2- 172.21.15.16:0/386353339 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f677006ada0 0x7f677006d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:40.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.774+0000 7f67921f1700 1 --2- 172.21.15.16:0/386353339 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f677006ada0 0x7f677006d260 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f678c0cb120 tx=0x7f6780008040 comp rx=0 tx=0).ready entity=mgr.8820 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:40.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.778+0000 7f677effd700 1 -- 172.21.15.16:0/386353339 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6784054900 con 0x7f678c158830 2023-10-16T12:30:40.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:40.960+0000 7f677effd700 1 -- 172.21.15.16:0/386353339 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 79) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f6784054cd0 con 0x7f678c158830 2023-10-16T12:30:41.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.013+0000 7f6794455700 1 -- 172.21.15.16:0/386353339 --> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/monthly", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f678c0cb5d0 con 0x7f677006ada0 2023-10-16T12:30:41.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.073+0000 7f677effd700 1 -- 172.21.15.16:0/386353339 <== mgr.8820 v2:172.21.15.16:6832/3783161600 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+45 (secure 0 0 0) 0x7f6780013830 con 0x7f677006ada0 2023-10-16T12:30:41.076 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule set for path snap_test_dir1/monthly 2023-10-16T12:30:41.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.075+0000 7f6794455700 1 -- 172.21.15.16:0/386353339 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f677006ada0 msgr2=0x7f677006d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:41.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.075+0000 7f6794455700 1 --2- 172.21.15.16:0/386353339 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f677006ada0 0x7f677006d260 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f678c0cb120 tx=0x7f6780008040 comp rx=0 tx=0).stop 2023-10-16T12:30:41.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.075+0000 7f6794455700 1 -- 172.21.15.16:0/386353339 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f678c158830 msgr2=0x7f678c154db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:41.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.075+0000 7f6794455700 1 --2- 172.21.15.16:0/386353339 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f678c158830 0x7f678c154db0 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f678c15db00 tx=0x7f678400d430 comp rx=0 tx=0).stop 2023-10-16T12:30:41.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.075+0000 7f6794455700 1 -- 172.21.15.16:0/386353339 shutdown_connections 2023-10-16T12:30:41.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.075+0000 7f6794455700 1 --2- 172.21.15.16:0/386353339 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f677006ada0 0x7f677006d260 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:41.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.075+0000 7f6794455700 1 --2- 172.21.15.16:0/386353339 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f678c158830 0x7f678c154db0 unknown :-1 s=CLOSED pgs=1606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:41.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.075+0000 7f6794455700 1 --2- 172.21.15.16:0/386353339 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6778003fb0 0x7f678c1540c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:41.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.075+0000 7f6794455700 1 --2- 172.21.15.16:0/386353339 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f678c1583b0 0x7f678c154870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:41.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.075+0000 7f6794455700 1 -- 172.21.15.16:0/386353339 >> 172.21.15.16:0/386353339 conn(0x7f678c06cdb0 msgr2=0x7f678c06e5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:41.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.075+0000 7f6794455700 1 -- 172.21.15.16:0/386353339 shutdown_connections 2023-10-16T12:30:41.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.076+0000 7f6794455700 1 -- 172.21.15.16:0/386353339 wait complete. 2023-10-16T12:30:41.091 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/monthly 2023-10-16T12:30:41.091 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:30:41.092 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/yearly) 2023-10-16T12:30:41.111 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/yearly 1Y 2023-10-16T12:30:41.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.276+0000 7fdaf8980700 1 Processor -- start 2023-10-16T12:30:41.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.276+0000 7fdaf8980700 1 -- start start 2023-10-16T12:30:41.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.277+0000 7fdaf8980700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdaf40cb880 0x7fdaf40cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:41.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.277+0000 7fdaf8980700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fdaf406da60 0x7fdaf406def0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:41.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.277+0000 7fdaf8980700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaf40cef70 con 0x7fdaf415b070 2023-10-16T12:30:41.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.277+0000 7fdaf8980700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdaf406e430 con 0x7fdaf406da60 2023-10-16T12:30:41.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.277+0000 7fdaf8980700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf40ce360 con 0x7fdaf40cb880 2023-10-16T12:30:41.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.277+0000 7fdaf259c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fdaf415b070 0x7fdaf415b450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35092/0 (socket says 172.21.15.16:35092) 2023-10-16T12:30:41.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.277+0000 7fdaf259c700 1 -- 172.21.15.16:0/2054259860 learned_addr learned my addr 172.21.15.16:0/2054259860 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:30:41.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.277+0000 7fdaf1d9b700 1 --2- 172.21.15.16:0/2054259860 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdaf40cb880 0x7fdaf40cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:41.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.277+0000 7fdaf2d9d700 1 --2- 172.21.15.16:0/2054259860 >> v2:172.21.15.49:3300/0 conn(0x7fdaf406da60 0x7fdaf406def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:41.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.277+0000 7fdaf159a700 1 -- 172.21.15.16:0/2054259860 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 780676372 0 0) 0x7fdaf40cef70 con 0x7fdaf415b070 2023-10-16T12:30:41.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.277+0000 7fdaf159a700 1 -- 172.21.15.16:0/2054259860 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdae00035b0 con 0x7fdaf415b070 2023-10-16T12:30:41.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.277+0000 7fdaf1d9b700 1 -- 172.21.15.16:0/2054259860 >> v2:172.21.15.49:3300/0 conn(0x7fdaf406da60 msgr2=0x7fdaf406def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:41.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.277+0000 7fdaf1d9b700 1 --2- 172.21.15.16:0/2054259860 >> v2:172.21.15.49:3300/0 conn(0x7fdaf406da60 0x7fdaf406def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:41.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.277+0000 7fdaf1d9b700 1 -- 172.21.15.16:0/2054259860 >> v1:172.21.15.16:6789/0 conn(0x7fdaf415b070 legacy=0x7fdaf415b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:41.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.277+0000 7fdaf159a700 1 -- 172.21.15.16:0/2054259860 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1690923491 0 0) 0x7fdae00035b0 con 0x7fdaf415b070 2023-10-16T12:30:41.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.277+0000 7fdaf1d9b700 1 -- 172.21.15.16:0/2054259860 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaf40ce640 con 0x7fdaf40cb880 2023-10-16T12:30:41.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.277+0000 7fdaf2d9d700 1 --2- 172.21.15.16:0/2054259860 >> v2:172.21.15.49:3300/0 conn(0x7fdaf406da60 0x7fdaf406def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:30:41.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.278+0000 7fdaf1d9b700 1 --2- 172.21.15.16:0/2054259860 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdaf40cb880 0x7fdaf40cbd00 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fdadc009840 tx=0x7fdadc009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d074129da72160a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:41.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.278+0000 7fdaf159a700 1 -- 172.21.15.16:0/2054259860 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdae4001670 con 0x7fdaf40cb880 2023-10-16T12:30:41.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.278+0000 7fdaf159a700 1 -- 172.21.15.16:0/2054259860 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdadc004030 con 0x7fdaf40cb880 2023-10-16T12:30:41.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.278+0000 7fdaf159a700 1 -- 172.21.15.16:0/2054259860 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdadc0043c0 con 0x7fdaf40cb880 2023-10-16T12:30:41.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.278+0000 7fdaf8980700 1 -- 172.21.15.16:0/2054259860 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdaf40cb880 msgr2=0x7fdaf40cbd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:41.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.278+0000 7fdaf8980700 1 --2- 172.21.15.16:0/2054259860 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdaf40cb880 0x7fdaf40cbd00 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fdadc009840 tx=0x7fdadc009b50 comp rx=0 tx=0).stop 2023-10-16T12:30:41.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.278+0000 7fdaf8980700 1 -- 172.21.15.16:0/2054259860 shutdown_connections 2023-10-16T12:30:41.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.278+0000 7fdaf8980700 1 --2- 172.21.15.16:0/2054259860 >> v2:172.21.15.49:3300/0 conn(0x7fdaf406da60 0x7fdaf406def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:41.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.278+0000 7fdaf8980700 1 --2- 172.21.15.16:0/2054259860 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdaf40cb880 0x7fdaf40cbd00 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:41.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.278+0000 7fdaf8980700 1 -- 172.21.15.16:0/2054259860 >> 172.21.15.16:0/2054259860 conn(0x7fdaf40b1f40 msgr2=0x7fdaf40b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:41.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.278+0000 7fdaf8980700 1 -- 172.21.15.16:0/2054259860 shutdown_connections 2023-10-16T12:30:41.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.278+0000 7fdaf8980700 1 -- 172.21.15.16:0/2054259860 wait complete. 2023-10-16T12:30:41.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.280+0000 7fdaf8980700 1 Processor -- start 2023-10-16T12:30:41.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.280+0000 7fdaf8980700 1 -- start start 2023-10-16T12:30:41.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.280+0000 7fdaf8980700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdaf406da60 0x7fdaf4156e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:41.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.280+0000 7fdaf8980700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdaf40cb880 0x7fdaf40ad830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:41.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.280+0000 7fdaf8980700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdaf415b070 0x7fdaf40cbca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:41.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.280+0000 7fdaf8980700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf40ca410 con 0x7fdaf40cb880 2023-10-16T12:30:41.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.280+0000 7fdaf8980700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf4157360 con 0x7fdaf406da60 2023-10-16T12:30:41.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.280+0000 7fdaf8980700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf40c8280 con 0x7fdaf415b070 2023-10-16T12:30:41.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.281+0000 7fdaf1d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdaf40cb880 0x7fdaf40ad830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:41.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.281+0000 7fdaf1d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdaf40cb880 0x7fdaf40ad830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45826/0 (socket says 172.21.15.16:45826) 2023-10-16T12:30:41.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.281+0000 7fdaf1d9b700 1 -- 172.21.15.16:0/211283609 learned_addr learned my addr 172.21.15.16:0/211283609 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:41.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.281+0000 7fdaf2d9d700 1 --2- 172.21.15.16:0/211283609 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdaf415b070 0x7fdaf40cbca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:41.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.281+0000 7fdaf259c700 1 --2- 172.21.15.16:0/211283609 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdaf406da60 0x7fdaf4156e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:41.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.281+0000 7fdaf1d9b700 1 -- 172.21.15.16:0/211283609 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdaf415b070 msgr2=0x7fdaf40cbca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:41.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.281+0000 7fdaf1d9b700 1 --2- 172.21.15.16:0/211283609 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdaf415b070 0x7fdaf40cbca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:41.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.281+0000 7fdaf1d9b700 1 -- 172.21.15.16:0/211283609 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdaf406da60 msgr2=0x7fdaf4156e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:41.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.281+0000 7fdaf1d9b700 1 --2- 172.21.15.16:0/211283609 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdaf406da60 0x7fdaf4156e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:41.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.281+0000 7fdaf1d9b700 1 -- 172.21.15.16:0/211283609 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaec009a70 con 0x7fdaf40cb880 2023-10-16T12:30:41.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.281+0000 7fdaf1d9b700 1 --2- 172.21.15.16:0/211283609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdaf40cb880 0x7fdaf40ad830 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7fdadc00a010 tx=0x7fdadc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:41.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.281+0000 7fdaeaffd700 1 -- 172.21.15.16:0/211283609 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdadc003ca0 con 0x7fdaf40cb880 2023-10-16T12:30:41.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.281+0000 7fdaeaffd700 1 -- 172.21.15.16:0/211283609 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdadc01f070 con 0x7fdaf40cb880 2023-10-16T12:30:41.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.281+0000 7fdaf8980700 1 -- 172.21.15.16:0/211283609 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdadc0094a0 con 0x7fdaf40cb880 2023-10-16T12:30:41.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.281+0000 7fdaeaffd700 1 -- 172.21.15.16:0/211283609 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdadc011870 con 0x7fdaf40cb880 2023-10-16T12:30:41.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.281+0000 7fdaf8980700 1 -- 172.21.15.16:0/211283609 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdaf40acd50 con 0x7fdaf40cb880 2023-10-16T12:30:41.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.282+0000 7fdaf8980700 1 -- 172.21.15.16:0/211283609 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdaf40cdfc0 con 0x7fdaf40cb880 2023-10-16T12:30:41.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.283+0000 7fdaeaffd700 1 -- 172.21.15.16:0/211283609 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 79) v1 ==== 89349+0+0 (secure 0 0 0) 0x7fdadc0119d0 con 0x7fdaf40cb880 2023-10-16T12:30:41.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.283+0000 7fdaeaffd700 1 --2- 172.21.15.16:0/211283609 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7fdad406b070 0x7fdad406d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:41.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.283+0000 7fdaeaffd700 1 -- 172.21.15.16:0/211283609 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6429+0+0 (secure 0 0 0) 0x7fdadc021070 con 0x7fdaf40cb880 2023-10-16T12:30:41.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.283+0000 7fdaf259c700 1 --2- 172.21.15.16:0/211283609 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7fdad406b070 0x7fdad406d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:41.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.284+0000 7fdaf259c700 1 --2- 172.21.15.16:0/211283609 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7fdad406b070 0x7fdad406d530 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fdae4000f60 tx=0x7fdae4001e00 comp rx=0 tx=0).ready entity=mgr.8820 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:41.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.288+0000 7fdaeaffd700 1 -- 172.21.15.16:0/211283609 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdadc0bdfe0 con 0x7fdaf40cb880 2023-10-16T12:30:41.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.520+0000 7fdaf8980700 1 -- 172.21.15.16:0/211283609 --> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/yearly", "snap_schedule": "1Y", "target": ["mon-mgr", ""]}) v1 -- 0x7fdaf40cec00 con 0x7fdad406b070 2023-10-16T12:30:41.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.589+0000 7fdaeaffd700 1 -- 172.21.15.16:0/211283609 <== mgr.8820 v2:172.21.15.16:6832/3783161600 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+44 (secure 0 0 0) 0x7fdae40016d0 con 0x7fdad406b070 2023-10-16T12:30:41.593 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule set for path snap_test_dir1/yearly 2023-10-16T12:30:41.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.591+0000 7fdaf8980700 1 -- 172.21.15.16:0/211283609 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7fdad406b070 msgr2=0x7fdad406d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:41.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.592+0000 7fdaf8980700 1 --2- 172.21.15.16:0/211283609 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7fdad406b070 0x7fdad406d530 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fdae4000f60 tx=0x7fdae4001e00 comp rx=0 tx=0).stop 2023-10-16T12:30:41.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.592+0000 7fdaf8980700 1 -- 172.21.15.16:0/211283609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdaf40cb880 msgr2=0x7fdaf40ad830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:41.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.592+0000 7fdaf8980700 1 --2- 172.21.15.16:0/211283609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdaf40cb880 0x7fdaf40ad830 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7fdadc00a010 tx=0x7fdadc004750 comp rx=0 tx=0).stop 2023-10-16T12:30:41.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.592+0000 7fdaf8980700 1 -- 172.21.15.16:0/211283609 shutdown_connections 2023-10-16T12:30:41.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.592+0000 7fdaf8980700 1 --2- 172.21.15.16:0/211283609 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7fdad406b070 0x7fdad406d530 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:41.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.592+0000 7fdaf8980700 1 --2- 172.21.15.16:0/211283609 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdaf415b070 0x7fdaf40cbca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:41.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.592+0000 7fdaf8980700 1 --2- 172.21.15.16:0/211283609 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdaf406da60 0x7fdaf4156e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:41.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.592+0000 7fdaf8980700 1 --2- 172.21.15.16:0/211283609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdaf40cb880 0x7fdaf40ad830 unknown :-1 s=CLOSED pgs=1608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:41.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.592+0000 7fdaf8980700 1 -- 172.21.15.16:0/211283609 >> 172.21.15.16:0/211283609 conn(0x7fdaf40b1f40 msgr2=0x7fdaf40d0e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:41.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.592+0000 7fdaf8980700 1 -- 172.21.15.16:0/211283609 shutdown_connections 2023-10-16T12:30:41.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.592+0000 7fdaf8980700 1 -- 172.21.15.16:0/211283609 wait complete. 2023-10-16T12:30:41.607 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/yearly 2023-10-16T12:30:41.607 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:30:41.608 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/bad_period_spec) 2023-10-16T12:30:41.626 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec 1X 2023-10-16T12:30:41.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.792+0000 7f28456cd700 1 Processor -- start 2023-10-16T12:30:41.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.792+0000 7f28456cd700 1 -- start start 2023-10-16T12:30:41.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.793+0000 7f28456cd700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f28400cea60 0x7f28400b2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:41.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.793+0000 7f28456cd700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28400b2f30 con 0x7f28400b8fa0 2023-10-16T12:30:41.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.793+0000 7f28456cd700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28400d1c40 con 0x7f284015d8d0 2023-10-16T12:30:41.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.793+0000 7f28456cd700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28400d0bb0 con 0x7f28400cea60 2023-10-16T12:30:41.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.793+0000 7f283e7fc700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f28400b8fa0 0x7f28400ce350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35108/0 (socket says 172.21.15.16:35108) 2023-10-16T12:30:41.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.793+0000 7f283e7fc700 1 -- 172.21.15.16:0/3140445012 learned_addr learned my addr 172.21.15.16:0/3140445012 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:30:41.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.793+0000 7f283dffb700 1 -- 172.21.15.16:0/3140445012 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2937759311 0 0) 0x7f28400b2f30 con 0x7f28400b8fa0 2023-10-16T12:30:41.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.793+0000 7f283f7fe700 1 --2- 172.21.15.16:0/3140445012 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f28400cea60 0x7f28400b2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:41.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.793+0000 7f283dffb700 1 -- 172.21.15.16:0/3140445012 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28280035b0 con 0x7f28400b8fa0 2023-10-16T12:30:41.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.793+0000 7f283dffb700 1 -- 172.21.15.16:0/3140445012 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1402642312 0 0) 0x7f28280035b0 con 0x7f28400b8fa0 2023-10-16T12:30:41.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.793+0000 7f283dffb700 1 -- 172.21.15.16:0/3140445012 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f28400cea60 msgr2=0x7f28400b2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:41.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.793+0000 7f283dffb700 1 --2- 172.21.15.16:0/3140445012 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f28400cea60 0x7f28400b2830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:41.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.793+0000 7f283dffb700 1 -- 172.21.15.16:0/3140445012 >> v1:172.21.15.49:6789/0 conn(0x7f284015d8d0 legacy=0x7f28400b6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:41.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.793+0000 7f283dffb700 1 -- 172.21.15.16:0/3140445012 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28400d1ee0 con 0x7f28400b8fa0 2023-10-16T12:30:41.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.793+0000 7f283dffb700 1 -- 172.21.15.16:0/3140445012 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f2834001670 con 0x7f28400b8fa0 2023-10-16T12:30:41.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.793+0000 7f283dffb700 1 -- 172.21.15.16:0/3140445012 >> v1:172.21.15.16:6789/0 conn(0x7f28400b8fa0 legacy=0x7f28400ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:41.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.793+0000 7f283dffb700 1 --2- 172.21.15.16:0/3140445012 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2828003a30 0x7f2828003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:41.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.793+0000 7f283dffb700 1 --2- 172.21.15.16:0/3140445012 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f28400cea60 0x7f2828004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:41.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.793+0000 7f283dffb700 1 --2- 172.21.15.16:0/3140445012 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2828004d90 0x7f2828005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:41.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.793+0000 7f283dffb700 1 -- 172.21.15.16:0/3140445012 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28400d0bb0 con 0x7f2828003a30 2023-10-16T12:30:41.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.793+0000 7f283dffb700 1 -- 172.21.15.16:0/3140445012 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f282800bb70 con 0x7f2828004d90 2023-10-16T12:30:41.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.793+0000 7f283dffb700 1 -- 172.21.15.16:0/3140445012 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f282800bd30 con 0x7f28400cea60 2023-10-16T12:30:41.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.794+0000 7f283effd700 1 --2- 172.21.15.16:0/3140445012 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2828003a30 0x7f2828003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:41.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.794+0000 7f283e7fc700 1 --2- 172.21.15.16:0/3140445012 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f28400cea60 0x7f2828004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:41.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.794+0000 7f283f7fe700 1 --2- 172.21.15.16:0/3140445012 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2828004d90 0x7f2828005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:41.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.794+0000 7f283effd700 1 -- 172.21.15.16:0/3140445012 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f28400cea60 msgr2=0x7f2828004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:41.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.794+0000 7f283effd700 1 --2- 172.21.15.16:0/3140445012 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f28400cea60 0x7f2828004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:41.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.794+0000 7f283effd700 1 -- 172.21.15.16:0/3140445012 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2828004d90 msgr2=0x7f2828005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:41.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.794+0000 7f283effd700 1 --2- 172.21.15.16:0/3140445012 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2828004d90 0x7f2828005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:41.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.794+0000 7f283effd700 1 -- 172.21.15.16:0/3140445012 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28280035b0 con 0x7f2828003a30 2023-10-16T12:30:41.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.794+0000 7f283effd700 1 --2- 172.21.15.16:0/3140445012 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2828003a30 0x7f2828003e50 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7f28300017c0 tx=0x7f2828005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78d46afda028ab9c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:41.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.794+0000 7f283dffb700 1 -- 172.21.15.16:0/3140445012 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28300052d0 con 0x7f2828003a30 2023-10-16T12:30:41.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.794+0000 7f283dffb700 1 -- 172.21.15.16:0/3140445012 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2830005430 con 0x7f2828003a30 2023-10-16T12:30:41.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.794+0000 7f283dffb700 1 -- 172.21.15.16:0/3140445012 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28300038c0 con 0x7f2828003a30 2023-10-16T12:30:41.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.795+0000 7f28456cd700 1 -- 172.21.15.16:0/3140445012 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2828003a30 msgr2=0x7f2828003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:41.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.795+0000 7f28456cd700 1 --2- 172.21.15.16:0/3140445012 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2828003a30 0x7f2828003e50 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7f28300017c0 tx=0x7f2828005880 comp rx=0 tx=0).stop 2023-10-16T12:30:41.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.795+0000 7f283f7fe700 1 -- 172.21.15.16:0/3140445012 reap_dead start 2023-10-16T12:30:41.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.795+0000 7f28456cd700 1 -- 172.21.15.16:0/3140445012 shutdown_connections 2023-10-16T12:30:41.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.795+0000 7f28456cd700 1 -- 172.21.15.16:0/3140445012 >> 172.21.15.16:0/3140445012 conn(0x7f2840072330 msgr2=0x7f2840072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:41.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.795+0000 7f28456cd700 1 -- 172.21.15.16:0/3140445012 shutdown_connections 2023-10-16T12:30:41.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.795+0000 7f28456cd700 1 -- 172.21.15.16:0/3140445012 wait complete. 2023-10-16T12:30:41.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.796+0000 7f28456cd700 1 Processor -- start 2023-10-16T12:30:41.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.796+0000 7f28456cd700 1 -- start start 2023-10-16T12:30:41.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.797+0000 7f28456cd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2828003a30 0x7f28401f3a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:41.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.797+0000 7f28456cd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2840159a20 0x7f28401f65a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:41.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.797+0000 7f28456cd700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f28401f6ae0 0x7f28400b36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:41.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.797+0000 7f28456cd700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2840159ea0 con 0x7f2828003a30 2023-10-16T12:30:41.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.797+0000 7f28456cd700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28401f2fb0 con 0x7f2840159a20 2023-10-16T12:30:41.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.797+0000 7f28456cd700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28401f6ff0 con 0x7f28401f6ae0 2023-10-16T12:30:41.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.797+0000 7f283effd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2828003a30 0x7f28401f3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:41.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.797+0000 7f283effd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2828003a30 0x7f28401f3a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45844/0 (socket says 172.21.15.16:45844) 2023-10-16T12:30:41.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.797+0000 7f283effd700 1 -- 172.21.15.16:0/82377123 learned_addr learned my addr 172.21.15.16:0/82377123 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:41.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.797+0000 7f283f7fe700 1 --2- 172.21.15.16:0/82377123 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f28401f6ae0 0x7f28400b36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:41.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.797+0000 7f283e7fc700 1 --2- 172.21.15.16:0/82377123 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2840159a20 0x7f28401f65a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:41.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.797+0000 7f283effd700 1 -- 172.21.15.16:0/82377123 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f28401f6ae0 msgr2=0x7f28400b36f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:41.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.797+0000 7f283effd700 1 --2- 172.21.15.16:0/82377123 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f28401f6ae0 0x7f28400b36f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:41.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.797+0000 7f283effd700 1 -- 172.21.15.16:0/82377123 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2840159a20 msgr2=0x7f28401f65a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:41.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.797+0000 7f283effd700 1 --2- 172.21.15.16:0/82377123 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2840159a20 0x7f28401f65a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:41.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.797+0000 7f283effd700 1 -- 172.21.15.16:0/82377123 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28401f72d0 con 0x7f2828003a30 2023-10-16T12:30:41.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.797+0000 7f283f7fe700 1 --2- 172.21.15.16:0/82377123 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f28401f6ae0 0x7f28400b36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:30:41.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.797+0000 7f283e7fc700 1 --2- 172.21.15.16:0/82377123 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2840159a20 0x7f28401f65a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:30:41.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.798+0000 7f283effd700 1 --2- 172.21.15.16:0/82377123 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2828003a30 0x7f28401f3a70 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f2830000c00 tx=0x7f2830001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:41.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.798+0000 7f28277fe700 1 -- 172.21.15.16:0/82377123 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2830007450 con 0x7f2828003a30 2023-10-16T12:30:41.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.798+0000 7f28277fe700 1 -- 172.21.15.16:0/82377123 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28300075b0 con 0x7f2828003a30 2023-10-16T12:30:41.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.798+0000 7f28456cd700 1 -- 172.21.15.16:0/82377123 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28401f3270 con 0x7f2828003a30 2023-10-16T12:30:41.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.798+0000 7f28277fe700 1 -- 172.21.15.16:0/82377123 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28300038c0 con 0x7f2828003a30 2023-10-16T12:30:41.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.798+0000 7f28456cd700 1 -- 172.21.15.16:0/82377123 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28400cd680 con 0x7f2828003a30 2023-10-16T12:30:41.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.799+0000 7f28456cd700 1 -- 172.21.15.16:0/82377123 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28400cd810 con 0x7f2828003a30 2023-10-16T12:30:41.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.799+0000 7f28277fe700 1 -- 172.21.15.16:0/82377123 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 79) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f2830003a20 con 0x7f2828003a30 2023-10-16T12:30:41.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.799+0000 7f28277fe700 1 --2- 172.21.15.16:0/82377123 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f282c06b0a0 0x7f282c06d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:41.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.800+0000 7f28277fe700 1 -- 172.21.15.16:0/82377123 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6429+0+0 (secure 0 0 0) 0x7f283008a5c0 con 0x7f2828003a30 2023-10-16T12:30:41.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.800+0000 7f283e7fc700 1 --2- 172.21.15.16:0/82377123 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f282c06b0a0 0x7f282c06d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:41.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.800+0000 7f283e7fc700 1 --2- 172.21.15.16:0/82377123 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f282c06b0a0 0x7f282c06d560 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f28400d35b0 tx=0x7f2834009040 comp rx=0 tx=0).ready entity=mgr.8820 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:41.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:41.804+0000 7f28277fe700 1 -- 172.21.15.16:0/82377123 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2830055320 con 0x7f2828003a30 2023-10-16T12:30:42.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.038+0000 7f28456cd700 1 -- 172.21.15.16:0/82377123 --> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "1X", "target": ["mon-mgr", ""]}) v1 -- 0x7f28400d24a0 con 0x7f282c06b0a0 2023-10-16T12:30:42.042 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:42.039+0000 7f986c0e0700 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - multiplier should be one of h,d,w,M,Y 2023-10-16T12:30:42.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.039+0000 7f28277fe700 1 -- 172.21.15.16:0/82377123 <== mgr.8820 v2:172.21.15.16:6832/3783161600 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - multiplier should be one of h,d,w,M,Y) v1 ==== 74+0+0 (secure 0 0 0) 0x7f2834001b20 con 0x7f282c06b0a0 2023-10-16T12:30:42.044 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENOENT: invalid schedule specified - multiplier should be one of h,d,w,M,Y 2023-10-16T12:30:42.047 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-10-16T12:30:42.048 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec 1MM 2023-10-16T12:30:42.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.174+0000 7fa59c1c5700 1 Processor -- start 2023-10-16T12:30:42.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.174+0000 7fa59c1c5700 1 -- start start 2023-10-16T12:30:42.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.175+0000 7fa59c1c5700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa59415b790 0x7fa5940cda60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:42.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.175+0000 7fa59c1c5700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5940ce0e0 con 0x7fa59406e070 2023-10-16T12:30:42.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.175+0000 7fa59c1c5700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa59406a7e0 con 0x7fa59415bb70 2023-10-16T12:30:42.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.175+0000 7fa59c1c5700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa59415c410 con 0x7fa59415b790 2023-10-16T12:30:42.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.175+0000 7fa59a762700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa59406e070 0x7fa5940ce850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35128/0 (socket says 172.21.15.16:35128) 2023-10-16T12:30:42.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.175+0000 7fa59a762700 1 -- 172.21.15.16:0/4075124429 learned_addr learned my addr 172.21.15.16:0/4075124429 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:30:42.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.175+0000 7fa599f61700 1 --2- 172.21.15.16:0/4075124429 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa59415b790 0x7fa5940cda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:42.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.175+0000 7fa598f5f700 1 -- 172.21.15.16:0/4075124429 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3782994760 0 0) 0x7fa5940ce0e0 con 0x7fa59406e070 2023-10-16T12:30:42.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.176+0000 7fa598f5f700 1 -- 172.21.15.16:0/4075124429 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5800035b0 con 0x7fa59406e070 2023-10-16T12:30:42.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.176+0000 7fa598f5f700 1 -- 172.21.15.16:0/4075124429 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2790732136 0 0) 0x7fa5800035b0 con 0x7fa59406e070 2023-10-16T12:30:42.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.176+0000 7fa598f5f700 1 -- 172.21.15.16:0/4075124429 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa59415b790 msgr2=0x7fa5940cda60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:42.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.176+0000 7fa598f5f700 1 --2- 172.21.15.16:0/4075124429 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa59415b790 0x7fa5940cda60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:42.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.176+0000 7fa598f5f700 1 -- 172.21.15.16:0/4075124429 >> v1:172.21.15.49:6789/0 conn(0x7fa59415bb70 legacy=0x7fa59406d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:42.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.176+0000 7fa598f5f700 1 -- 172.21.15.16:0/4075124429 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa59415c6f0 con 0x7fa59406e070 2023-10-16T12:30:42.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.176+0000 7fa598f5f700 1 -- 172.21.15.16:0/4075124429 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7fa58c0019f0 con 0x7fa59406e070 2023-10-16T12:30:42.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.176+0000 7fa598f5f700 1 -- 172.21.15.16:0/4075124429 >> v1:172.21.15.16:6789/0 conn(0x7fa59406e070 legacy=0x7fa5940ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:42.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.176+0000 7fa598f5f700 1 --2- 172.21.15.16:0/4075124429 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa580003a30 0x7fa580003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:42.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.176+0000 7fa598f5f700 1 --2- 172.21.15.16:0/4075124429 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa59415b790 0x7fa580004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:42.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.176+0000 7fa598f5f700 1 --2- 172.21.15.16:0/4075124429 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa580004d90 0x7fa580005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:42.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.176+0000 7fa598f5f700 1 -- 172.21.15.16:0/4075124429 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa59415c410 con 0x7fa580004d90 2023-10-16T12:30:42.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.176+0000 7fa598f5f700 1 -- 172.21.15.16:0/4075124429 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa58000ba10 con 0x7fa580003a30 2023-10-16T12:30:42.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.176+0000 7fa598f5f700 1 -- 172.21.15.16:0/4075124429 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa58000bbd0 con 0x7fa59415b790 2023-10-16T12:30:42.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.176+0000 7fa59a762700 1 --2- 172.21.15.16:0/4075124429 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa580004d90 0x7fa580005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:42.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.176+0000 7fa599f61700 1 --2- 172.21.15.16:0/4075124429 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa580003a30 0x7fa580003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:42.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.176+0000 7fa599760700 1 --2- 172.21.15.16:0/4075124429 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa59415b790 0x7fa580004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:42.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.176+0000 7fa59a762700 1 -- 172.21.15.16:0/4075124429 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa59415b790 msgr2=0x7fa580004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:42.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.176+0000 7fa59a762700 1 --2- 172.21.15.16:0/4075124429 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa59415b790 0x7fa580004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:42.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.176+0000 7fa59a762700 1 -- 172.21.15.16:0/4075124429 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa580003a30 msgr2=0x7fa580003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:42.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.176+0000 7fa59a762700 1 --2- 172.21.15.16:0/4075124429 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa580003a30 0x7fa580003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:42.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.176+0000 7fa59a762700 1 -- 172.21.15.16:0/4075124429 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5800035b0 con 0x7fa580004d90 2023-10-16T12:30:42.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.176+0000 7fa59a762700 1 --2- 172.21.15.16:0/4075124429 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa580004d90 0x7fa580005210 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7fa5940cc450 tx=0x7fa580005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6a468cc0d2a0b37 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:42.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.177+0000 7fa598f5f700 1 -- 172.21.15.16:0/4075124429 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa58c012600 con 0x7fa580004d90 2023-10-16T12:30:42.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.177+0000 7fa598f5f700 1 -- 172.21.15.16:0/4075124429 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa58c012760 con 0x7fa580004d90 2023-10-16T12:30:42.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.177+0000 7fa598f5f700 1 -- 172.21.15.16:0/4075124429 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa58c012b10 con 0x7fa580004d90 2023-10-16T12:30:42.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.177+0000 7fa59c1c5700 1 -- 172.21.15.16:0/4075124429 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa580004d90 msgr2=0x7fa580005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:42.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.177+0000 7fa59c1c5700 1 --2- 172.21.15.16:0/4075124429 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa580004d90 0x7fa580005210 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7fa5940cc450 tx=0x7fa580005880 comp rx=0 tx=0).stop 2023-10-16T12:30:42.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.177+0000 7fa59a762700 1 -- 172.21.15.16:0/4075124429 reap_dead start 2023-10-16T12:30:42.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.177+0000 7fa59c1c5700 1 -- 172.21.15.16:0/4075124429 shutdown_connections 2023-10-16T12:30:42.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.177+0000 7fa59c1c5700 1 -- 172.21.15.16:0/4075124429 >> 172.21.15.16:0/4075124429 conn(0x7fa5940cc5e0 msgr2=0x7fa5940cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:42.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.177+0000 7fa59c1c5700 1 -- 172.21.15.16:0/4075124429 shutdown_connections 2023-10-16T12:30:42.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.177+0000 7fa59c1c5700 1 -- 172.21.15.16:0/4075124429 wait complete. 2023-10-16T12:30:42.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.179+0000 7fa59c1c5700 1 Processor -- start 2023-10-16T12:30:42.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.179+0000 7fa59c1c5700 1 -- start start 2023-10-16T12:30:42.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.179+0000 7fa59c1c5700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa580004d90 0x7fa5940b4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:42.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.179+0000 7fa59c1c5700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa594153350 0x7fa5941f6ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:42.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.179+0000 7fa59c1c5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5941537b0 0x7fa5941f7220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:42.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.179+0000 7fa59c1c5700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5940b33b0 con 0x7fa5941537b0 2023-10-16T12:30:42.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.179+0000 7fa59c1c5700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5940b3690 con 0x7fa594153350 2023-10-16T12:30:42.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.179+0000 7fa59c1c5700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5940b3970 con 0x7fa580004d90 2023-10-16T12:30:42.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.179+0000 7fa59a762700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5941537b0 0x7fa5941f7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:42.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.179+0000 7fa59a762700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5941537b0 0x7fa5941f7220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45866/0 (socket says 172.21.15.16:45866) 2023-10-16T12:30:42.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.180+0000 7fa59a762700 1 -- 172.21.15.16:0/3049003449 learned_addr learned my addr 172.21.15.16:0/3049003449 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:42.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.180+0000 7fa599760700 1 --2- 172.21.15.16:0/3049003449 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa594153350 0x7fa5941f6ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:42.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.180+0000 7fa599f61700 1 --2- 172.21.15.16:0/3049003449 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa580004d90 0x7fa5940b4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:42.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.180+0000 7fa59a762700 1 -- 172.21.15.16:0/3049003449 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa580004d90 msgr2=0x7fa5940b4430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:42.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.180+0000 7fa59a762700 1 --2- 172.21.15.16:0/3049003449 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa580004d90 0x7fa5940b4430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:42.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.180+0000 7fa59a762700 1 -- 172.21.15.16:0/3049003449 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa594153350 msgr2=0x7fa5941f6ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:42.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.180+0000 7fa59a762700 1 --2- 172.21.15.16:0/3049003449 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa594153350 0x7fa5941f6ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:42.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.180+0000 7fa59a762700 1 -- 172.21.15.16:0/3049003449 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5940b3c50 con 0x7fa5941537b0 2023-10-16T12:30:42.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.180+0000 7fa59a762700 1 --2- 172.21.15.16:0/3049003449 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5941537b0 0x7fa5941f7220 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7fa58c012000 tx=0x7fa58c001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:42.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.180+0000 7fa58a7fc700 1 -- 172.21.15.16:0/3049003449 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa58c0019c0 con 0x7fa5941537b0 2023-10-16T12:30:42.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.180+0000 7fa58a7fc700 1 -- 172.21.15.16:0/3049003449 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa58c01e430 con 0x7fa5941537b0 2023-10-16T12:30:42.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.180+0000 7fa59c1c5700 1 -- 172.21.15.16:0/3049003449 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5940b3f30 con 0x7fa5941537b0 2023-10-16T12:30:42.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.180+0000 7fa58a7fc700 1 -- 172.21.15.16:0/3049003449 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa58c01e7e0 con 0x7fa5941537b0 2023-10-16T12:30:42.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.180+0000 7fa59c1c5700 1 -- 172.21.15.16:0/3049003449 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5941f7760 con 0x7fa5941537b0 2023-10-16T12:30:42.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.181+0000 7fa59c1c5700 1 -- 172.21.15.16:0/3049003449 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa59404ebb0 con 0x7fa5941537b0 2023-10-16T12:30:42.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.182+0000 7fa58a7fc700 1 -- 172.21.15.16:0/3049003449 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 79) v1 ==== 89349+0+0 (secure 0 0 0) 0x7fa58c01e940 con 0x7fa5941537b0 2023-10-16T12:30:42.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.182+0000 7fa58a7fc700 1 --2- 172.21.15.16:0/3049003449 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7fa57806b070 0x7fa57806d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:42.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.182+0000 7fa58a7fc700 1 -- 172.21.15.16:0/3049003449 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6429+0+0 (secure 0 0 0) 0x7fa58c08be40 con 0x7fa5941537b0 2023-10-16T12:30:42.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.182+0000 7fa599f61700 1 --2- 172.21.15.16:0/3049003449 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7fa57806b070 0x7fa57806d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:42.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.183+0000 7fa599f61700 1 --2- 172.21.15.16:0/3049003449 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7fa57806b070 0x7fa57806d530 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa584009ba0 tx=0x7fa584008040 comp rx=0 tx=0).ready entity=mgr.8820 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:42.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.187+0000 7fa58a7fc700 1 -- 172.21.15.16:0/3049003449 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa58c056cc0 con 0x7fa5941537b0 2023-10-16T12:30:42.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.419+0000 7fa59c1c5700 1 -- 172.21.15.16:0/3049003449 --> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "1MM", "target": ["mon-mgr", ""]}) v1 -- 0x7fa5940ce330 con 0x7fa57806b070 2023-10-16T12:30:42.423 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:42.420+0000 7f986c0e0700 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2023-10-16T12:30:42.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.420+0000 7fa58a7fc700 1 -- 172.21.15.16:0/3049003449 <== mgr.8820 v2:172.21.15.16:6832/3783161600 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc.) v1 ==== 138+0+0 (secure 0 0 0) 0x7fa5840097f0 con 0x7fa57806b070 2023-10-16T12:30:42.425 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENOENT: invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2023-10-16T12:30:42.428 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-10-16T12:30:42.429 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec 1 2023-10-16T12:30:42.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.555+0000 7f14998b1700 1 Processor -- start 2023-10-16T12:30:42.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.555+0000 7f14998b1700 1 -- start start 2023-10-16T12:30:42.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.556+0000 7f14998b1700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f14940ce6a0 0x7f149406c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:42.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.556+0000 7f14998b1700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f14940d2110 0x7f149406c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:42.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.556+0000 7f14998b1700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f14940b2620 0x7f14940b0f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:42.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.556+0000 7f14998b1700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14940b20f0 con 0x7f14940d2110 2023-10-16T12:30:42.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.556+0000 7f14998b1700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14940d03d0 con 0x7f14940ce6a0 2023-10-16T12:30:42.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.556+0000 7f14998b1700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14940b2340 con 0x7f14940b2620 2023-10-16T12:30:42.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.556+0000 7f14927fc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f14940d2110 0x7f149406c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:42.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.556+0000 7f14927fc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f14940d2110 0x7f149406c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45872/0 (socket says 172.21.15.16:45872) 2023-10-16T12:30:42.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.556+0000 7f14927fc700 1 -- 172.21.15.16:0/3049185094 learned_addr learned my addr 172.21.15.16:0/3049185094 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:42.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.556+0000 7f1492ffd700 1 --2- 172.21.15.16:0/3049185094 >> v2:172.21.15.49:3300/0 conn(0x7f14940ce6a0 0x7f149406c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:42.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.556+0000 7f14937fe700 1 --2- 172.21.15.16:0/3049185094 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f14940b2620 0x7f14940b0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:42.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.556+0000 7f14927fc700 1 -- 172.21.15.16:0/3049185094 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f14940b2620 msgr2=0x7f14940b0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:42.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.556+0000 7f14927fc700 1 --2- 172.21.15.16:0/3049185094 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f14940b2620 0x7f14940b0f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:42.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.556+0000 7f14927fc700 1 -- 172.21.15.16:0/3049185094 >> v2:172.21.15.49:3300/0 conn(0x7f14940ce6a0 msgr2=0x7f149406c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:42.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.556+0000 7f14927fc700 1 --2- 172.21.15.16:0/3049185094 >> v2:172.21.15.49:3300/0 conn(0x7f14940ce6a0 0x7f149406c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:42.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.556+0000 7f14927fc700 1 -- 172.21.15.16:0/3049185094 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f149406bb00 con 0x7f14940d2110 2023-10-16T12:30:42.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.556+0000 7f14927fc700 1 --2- 172.21.15.16:0/3049185094 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14940d2110 0x7f149406c840 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f1494067b50 tx=0x7f1488009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55ec12d023df3edb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:42.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.556+0000 7f1491ffb700 1 -- 172.21.15.16:0/3049185094 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1488011830 con 0x7f14940d2110 2023-10-16T12:30:42.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.557+0000 7f1491ffb700 1 -- 172.21.15.16:0/3049185094 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1488011990 con 0x7f14940d2110 2023-10-16T12:30:42.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.557+0000 7f1491ffb700 1 -- 172.21.15.16:0/3049185094 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1488011d20 con 0x7f14940d2110 2023-10-16T12:30:42.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.557+0000 7f14998b1700 1 -- 172.21.15.16:0/3049185094 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14940d2110 msgr2=0x7f149406c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:42.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.557+0000 7f14998b1700 1 --2- 172.21.15.16:0/3049185094 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14940d2110 0x7f149406c840 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f1494067b50 tx=0x7f1488009bd0 comp rx=0 tx=0).stop 2023-10-16T12:30:42.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.557+0000 7f14998b1700 1 -- 172.21.15.16:0/3049185094 shutdown_connections 2023-10-16T12:30:42.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.557+0000 7f14998b1700 1 --2- 172.21.15.16:0/3049185094 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f14940b2620 0x7f14940b0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:42.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.557+0000 7f14998b1700 1 --2- 172.21.15.16:0/3049185094 >> v2:172.21.15.49:3300/0 conn(0x7f14940ce6a0 0x7f149406c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:42.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.557+0000 7f14998b1700 1 --2- 172.21.15.16:0/3049185094 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14940d2110 0x7f149406c840 unknown :-1 s=CLOSED pgs=1615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:42.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.557+0000 7f14998b1700 1 -- 172.21.15.16:0/3049185094 >> 172.21.15.16:0/3049185094 conn(0x7f14940d15f0 msgr2=0x7f14940d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:42.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.557+0000 7f14998b1700 1 -- 172.21.15.16:0/3049185094 shutdown_connections 2023-10-16T12:30:42.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.557+0000 7f14998b1700 1 -- 172.21.15.16:0/3049185094 wait complete. 2023-10-16T12:30:42.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.558+0000 7f14998b1700 1 Processor -- start 2023-10-16T12:30:42.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.558+0000 7f14998b1700 1 -- start start 2023-10-16T12:30:42.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.559+0000 7f14998b1700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f14940b2620 0x7f149414fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:42.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.559+0000 7f14998b1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14940ce6a0 0x7f14940c7010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:42.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.559+0000 7f14998b1700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f14940d2110 0x7f14940c7550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:42.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.559+0000 7f14998b1700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14940ae280 con 0x7f14940ce6a0 2023-10-16T12:30:42.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.559+0000 7f14998b1700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14940c6130 con 0x7f14940b2620 2023-10-16T12:30:42.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.559+0000 7f14998b1700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14940c6410 con 0x7f14940d2110 2023-10-16T12:30:42.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.559+0000 7f14927fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14940ce6a0 0x7f14940c7010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:42.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.559+0000 7f14927fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14940ce6a0 0x7f14940c7010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45878/0 (socket says 172.21.15.16:45878) 2023-10-16T12:30:42.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.559+0000 7f14927fc700 1 -- 172.21.15.16:0/396961264 learned_addr learned my addr 172.21.15.16:0/396961264 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:42.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.560+0000 7f1492ffd700 1 --2- 172.21.15.16:0/396961264 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f14940b2620 0x7f149414fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:42.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.560+0000 7f14937fe700 1 --2- 172.21.15.16:0/396961264 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f14940d2110 0x7f14940c7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:42.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.560+0000 7f14927fc700 1 -- 172.21.15.16:0/396961264 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f14940d2110 msgr2=0x7f14940c7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:42.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.560+0000 7f14927fc700 1 --2- 172.21.15.16:0/396961264 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f14940d2110 0x7f14940c7550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:42.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.560+0000 7f14927fc700 1 -- 172.21.15.16:0/396961264 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f14940b2620 msgr2=0x7f149414fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:42.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.560+0000 7f14927fc700 1 --2- 172.21.15.16:0/396961264 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f14940b2620 0x7f149414fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:42.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.560+0000 7f14927fc700 1 -- 172.21.15.16:0/396961264 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1488009550 con 0x7f14940ce6a0 2023-10-16T12:30:42.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.560+0000 7f14927fc700 1 --2- 172.21.15.16:0/396961264 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14940ce6a0 0x7f14940c7010 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f1488009c10 tx=0x7f1488004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:42.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.560+0000 7f147bfff700 1 -- 172.21.15.16:0/396961264 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1488004510 con 0x7f14940ce6a0 2023-10-16T12:30:42.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.560+0000 7f147bfff700 1 -- 172.21.15.16:0/396961264 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f148801e070 con 0x7f14940ce6a0 2023-10-16T12:30:42.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.560+0000 7f14998b1700 1 -- 172.21.15.16:0/396961264 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14940c66f0 con 0x7f14940ce6a0 2023-10-16T12:30:42.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.560+0000 7f147bfff700 1 -- 172.21.15.16:0/396961264 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14880038e0 con 0x7f14940ce6a0 2023-10-16T12:30:42.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.560+0000 7f14998b1700 1 -- 172.21.15.16:0/396961264 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14940c53c0 con 0x7f14940ce6a0 2023-10-16T12:30:42.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.561+0000 7f14998b1700 1 -- 172.21.15.16:0/396961264 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14940ae450 con 0x7f14940ce6a0 2023-10-16T12:30:42.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.562+0000 7f147bfff700 1 -- 172.21.15.16:0/396961264 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 79) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f1488003a40 con 0x7f14940ce6a0 2023-10-16T12:30:42.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.562+0000 7f147bfff700 1 --2- 172.21.15.16:0/396961264 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f148006b070 0x7f148006d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:42.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.562+0000 7f147bfff700 1 -- 172.21.15.16:0/396961264 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6429+0+0 (secure 0 0 0) 0x7f148808a7e0 con 0x7f14940ce6a0 2023-10-16T12:30:42.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.562+0000 7f1492ffd700 1 --2- 172.21.15.16:0/396961264 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f148006b070 0x7f148006d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:42.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.562+0000 7f1492ffd700 1 --2- 172.21.15.16:0/396961264 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f148006b070 0x7f148006d530 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f14941f6d10 tx=0x7f1484008040 comp rx=0 tx=0).ready entity=mgr.8820 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:42.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.566+0000 7f147bfff700 1 -- 172.21.15.16:0/396961264 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f14940ae450 con 0x7f14940ce6a0 2023-10-16T12:30:42.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.794+0000 7f14998b1700 1 -- 172.21.15.16:0/396961264 --> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f14940c6580 con 0x7f148006b070 2023-10-16T12:30:42.798 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:42.795+0000 7f986c0e0700 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2023-10-16T12:30:42.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.795+0000 7f147bfff700 1 -- 172.21.15.16:0/396961264 <== mgr.8820 v2:172.21.15.16:6832/3783161600 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc.) v1 ==== 138+0+0 (secure 0 0 0) 0x7f14940c6580 con 0x7f148006b070 2023-10-16T12:30:42.800 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENOENT: invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2023-10-16T12:30:42.804 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-10-16T12:30:42.804 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec M 2023-10-16T12:30:42.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.930+0000 7fe3b7010700 1 Processor -- start 2023-10-16T12:30:42.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.930+0000 7fe3b7010700 1 -- start start 2023-10-16T12:30:42.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.931+0000 7fe3b7010700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe3b006ca10 0x7fe3b00ae8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:42.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.931+0000 7fe3b7010700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3b00b20f0 con 0x7fe3b00d2110 2023-10-16T12:30:42.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.931+0000 7fe3b7010700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3b00d03d0 con 0x7fe3b00ce6a0 2023-10-16T12:30:42.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.931+0000 7fe3b7010700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b00b2340 con 0x7fe3b006ca10 2023-10-16T12:30:42.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.931+0000 7fe3affff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe3b00d2110 0x7fe3b00ae1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35160/0 (socket says 172.21.15.16:35160) 2023-10-16T12:30:42.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.931+0000 7fe3affff700 1 -- 172.21.15.16:0/2578193184 learned_addr learned my addr 172.21.15.16:0/2578193184 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:30:42.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.931+0000 7fe3b55ad700 1 --2- 172.21.15.16:0/2578193184 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe3b006ca10 0x7fe3b00ae8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:42.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.931+0000 7fe3af7fe700 1 -- 172.21.15.16:0/2578193184 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3382854739 0 0) 0x7fe3b00b20f0 con 0x7fe3b00d2110 2023-10-16T12:30:42.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.931+0000 7fe3af7fe700 1 -- 172.21.15.16:0/2578193184 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3a00035b0 con 0x7fe3b00d2110 2023-10-16T12:30:42.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.931+0000 7fe3af7fe700 1 -- 172.21.15.16:0/2578193184 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2778477863 0 0) 0x7fe3a00035b0 con 0x7fe3b00d2110 2023-10-16T12:30:42.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.931+0000 7fe3af7fe700 1 -- 172.21.15.16:0/2578193184 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe3b006ca10 msgr2=0x7fe3b00ae8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:42.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.931+0000 7fe3af7fe700 1 --2- 172.21.15.16:0/2578193184 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe3b006ca10 0x7fe3b00ae8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:42.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.931+0000 7fe3af7fe700 1 -- 172.21.15.16:0/2578193184 >> v1:172.21.15.49:6789/0 conn(0x7fe3b00ce6a0 legacy=0x7fe3b006c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:42.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.931+0000 7fe3af7fe700 1 -- 172.21.15.16:0/2578193184 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3b006bb00 con 0x7fe3b00d2110 2023-10-16T12:30:42.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.931+0000 7fe3af7fe700 1 -- 172.21.15.16:0/2578193184 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7fe39c001670 con 0x7fe3b00d2110 2023-10-16T12:30:42.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.932+0000 7fe3af7fe700 1 -- 172.21.15.16:0/2578193184 >> v1:172.21.15.16:6789/0 conn(0x7fe3b00d2110 legacy=0x7fe3b00ae1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:42.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.932+0000 7fe3af7fe700 1 --2- 172.21.15.16:0/2578193184 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe3b006ca10 0x7fe3a0003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:42.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.932+0000 7fe3af7fe700 1 --2- 172.21.15.16:0/2578193184 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe3a0003f70 0x7fe3a0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:42.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.932+0000 7fe3af7fe700 1 --2- 172.21.15.16:0/2578193184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3a0004d90 0x7fe3a0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:42.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.932+0000 7fe3af7fe700 1 -- 172.21.15.16:0/2578193184 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b00b2340 con 0x7fe3a0004d90 2023-10-16T12:30:42.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.932+0000 7fe3af7fe700 1 -- 172.21.15.16:0/2578193184 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a000ba70 con 0x7fe3a0003f70 2023-10-16T12:30:42.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.932+0000 7fe3af7fe700 1 -- 172.21.15.16:0/2578193184 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a000bc30 con 0x7fe3b006ca10 2023-10-16T12:30:42.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.932+0000 7fe3b55ad700 1 --2- 172.21.15.16:0/2578193184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3a0004d90 0x7fe3a0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:42.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.932+0000 7fe3affff700 1 --2- 172.21.15.16:0/2578193184 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe3a0003f70 0x7fe3a0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:42.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.932+0000 7fe3b4dac700 1 --2- 172.21.15.16:0/2578193184 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe3b006ca10 0x7fe3a0003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:42.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.932+0000 7fe3b55ad700 1 -- 172.21.15.16:0/2578193184 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe3b006ca10 msgr2=0x7fe3a0003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:42.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.932+0000 7fe3b55ad700 1 --2- 172.21.15.16:0/2578193184 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe3b006ca10 0x7fe3a0003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:42.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.932+0000 7fe3b55ad700 1 -- 172.21.15.16:0/2578193184 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe3a0003f70 msgr2=0x7fe3a0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:42.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.932+0000 7fe3b55ad700 1 --2- 172.21.15.16:0/2578193184 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe3a0003f70 0x7fe3a0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:42.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.932+0000 7fe3b55ad700 1 -- 172.21.15.16:0/2578193184 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3a00035b0 con 0x7fe3a0004d90 2023-10-16T12:30:42.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.932+0000 7fe3affff700 1 --2- 172.21.15.16:0/2578193184 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe3a0003f70 0x7fe3a0004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:30:42.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.932+0000 7fe3b4dac700 1 --2- 172.21.15.16:0/2578193184 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe3b006ca10 0x7fe3a0003a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:30:42.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.932+0000 7fe3b55ad700 1 --2- 172.21.15.16:0/2578193184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3a0004d90 0x7fe3a0005210 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7fe3a8009fc0 tx=0x7fe3a0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97833b93f36e9d7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:42.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.932+0000 7fe3af7fe700 1 -- 172.21.15.16:0/2578193184 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a800ec30 con 0x7fe3a0004d90 2023-10-16T12:30:42.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.932+0000 7fe3af7fe700 1 -- 172.21.15.16:0/2578193184 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3a800f040 con 0x7fe3a0004d90 2023-10-16T12:30:42.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.932+0000 7fe3af7fe700 1 -- 172.21.15.16:0/2578193184 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a8014480 con 0x7fe3a0004d90 2023-10-16T12:30:42.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.933+0000 7fe3b7010700 1 -- 172.21.15.16:0/2578193184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3a0004d90 msgr2=0x7fe3a0005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:42.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.933+0000 7fe3b7010700 1 --2- 172.21.15.16:0/2578193184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3a0004d90 0x7fe3a0005210 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7fe3a8009fc0 tx=0x7fe3a0005880 comp rx=0 tx=0).stop 2023-10-16T12:30:42.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.933+0000 7fe3b55ad700 1 -- 172.21.15.16:0/2578193184 reap_dead start 2023-10-16T12:30:42.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.933+0000 7fe3b7010700 1 -- 172.21.15.16:0/2578193184 shutdown_connections 2023-10-16T12:30:42.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.933+0000 7fe3b7010700 1 -- 172.21.15.16:0/2578193184 >> 172.21.15.16:0/2578193184 conn(0x7fe3b00d15f0 msgr2=0x7fe3b00d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:42.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.933+0000 7fe3b7010700 1 -- 172.21.15.16:0/2578193184 shutdown_connections 2023-10-16T12:30:42.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.933+0000 7fe3b7010700 1 -- 172.21.15.16:0/2578193184 wait complete. 2023-10-16T12:30:42.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.934+0000 7fe3b7010700 1 Processor -- start 2023-10-16T12:30:42.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.934+0000 7fe3b7010700 1 -- start start 2023-10-16T12:30:42.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.935+0000 7fe3b7010700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe3a0004d90 0x7fe3b015ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:42.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.935+0000 7fe3b7010700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe3b00cd130 0x7fe3b00cd5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:42.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.935+0000 7fe3b7010700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3b01f6810 0x7fe3b01f6c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:42.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.935+0000 7fe3b7010700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b00cdaf0 con 0x7fe3b01f6810 2023-10-16T12:30:42.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.935+0000 7fe3b7010700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b01f7580 con 0x7fe3b00cd130 2023-10-16T12:30:42.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.935+0000 7fe3b7010700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b00c6170 con 0x7fe3a0004d90 2023-10-16T12:30:42.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.935+0000 7fe3b55ad700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3b01f6810 0x7fe3b01f6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:42.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.935+0000 7fe3b55ad700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3b01f6810 0x7fe3b01f6c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45900/0 (socket says 172.21.15.16:45900) 2023-10-16T12:30:42.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.935+0000 7fe3b55ad700 1 -- 172.21.15.16:0/307688787 learned_addr learned my addr 172.21.15.16:0/307688787 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:42.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.935+0000 7fe3affff700 1 --2- 172.21.15.16:0/307688787 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe3b00cd130 0x7fe3b00cd5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:42.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.935+0000 7fe3b4dac700 1 --2- 172.21.15.16:0/307688787 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe3a0004d90 0x7fe3b015ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:42.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.935+0000 7fe3b55ad700 1 -- 172.21.15.16:0/307688787 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe3a0004d90 msgr2=0x7fe3b015ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:42.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.935+0000 7fe3b55ad700 1 --2- 172.21.15.16:0/307688787 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe3a0004d90 0x7fe3b015ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:42.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.935+0000 7fe3b55ad700 1 -- 172.21.15.16:0/307688787 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe3b00cd130 msgr2=0x7fe3b00cd5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:42.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.935+0000 7fe3b55ad700 1 --2- 172.21.15.16:0/307688787 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe3b00cd130 0x7fe3b00cd5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:42.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.935+0000 7fe3b55ad700 1 -- 172.21.15.16:0/307688787 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3b00c6450 con 0x7fe3b01f6810 2023-10-16T12:30:42.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.935+0000 7fe3affff700 1 --2- 172.21.15.16:0/307688787 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe3b00cd130 0x7fe3b00cd5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:30:42.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.935+0000 7fe3b4dac700 1 --2- 172.21.15.16:0/307688787 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe3a0004d90 0x7fe3b015ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:30:42.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.936+0000 7fe3b55ad700 1 --2- 172.21.15.16:0/307688787 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3b01f6810 0x7fe3b01f6c90 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7fe3a8017040 tx=0x7fe3a8009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:42.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.936+0000 7fe3ad7fa700 1 -- 172.21.15.16:0/307688787 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a8007000 con 0x7fe3b01f6810 2023-10-16T12:30:42.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.936+0000 7fe3ad7fa700 1 -- 172.21.15.16:0/307688787 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3a8003e50 con 0x7fe3b01f6810 2023-10-16T12:30:42.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.936+0000 7fe3b7010700 1 -- 172.21.15.16:0/307688787 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3b00c6730 con 0x7fe3b01f6810 2023-10-16T12:30:42.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.936+0000 7fe3ad7fa700 1 -- 172.21.15.16:0/307688787 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a8014b50 con 0x7fe3b01f6810 2023-10-16T12:30:42.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.936+0000 7fe3b7010700 1 -- 172.21.15.16:0/307688787 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3b015b060 con 0x7fe3b01f6810 2023-10-16T12:30:42.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.937+0000 7fe3b7010700 1 -- 172.21.15.16:0/307688787 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3b00c6c50 con 0x7fe3b01f6810 2023-10-16T12:30:42.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.937+0000 7fe3ad7fa700 1 -- 172.21.15.16:0/307688787 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 79) v1 ==== 89349+0+0 (secure 0 0 0) 0x7fe3a8023020 con 0x7fe3b01f6810 2023-10-16T12:30:42.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.938+0000 7fe3ad7fa700 1 --2- 172.21.15.16:0/307688787 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7fe39406b070 0x7fe39406d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:42.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.938+0000 7fe3ad7fa700 1 -- 172.21.15.16:0/307688787 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6429+0+0 (secure 0 0 0) 0x7fe3a808b100 con 0x7fe3b01f6810 2023-10-16T12:30:42.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.938+0000 7fe3b4dac700 1 --2- 172.21.15.16:0/307688787 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7fe39406b070 0x7fe39406d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:42.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.938+0000 7fe3b4dac700 1 --2- 172.21.15.16:0/307688787 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7fe39406b070 0x7fe39406d530 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe3a4007b50 tx=0x7fe3a4009040 comp rx=0 tx=0).ready entity=mgr.8820 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:42.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.942+0000 7fe3ad7fa700 1 -- 172.21.15.16:0/307688787 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe3b00c6c50 con 0x7fe3b01f6810 2023-10-16T12:30:42.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:42.963+0000 7fe3ad7fa700 1 -- 172.21.15.16:0/307688787 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 80) v1 ==== 89405+0+0 (secure 0 0 0) 0x7fe3a8056350 con 0x7fe3b01f6810 2023-10-16T12:30:43.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.177+0000 7fe3b7010700 1 -- 172.21.15.16:0/307688787 --> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "M", "target": ["mon-mgr", ""]}) v1 -- 0x7fe3b00d0690 con 0x7fe39406b070 2023-10-16T12:30:43.180 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:43.178+0000 7f986c0e0700 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2023-10-16T12:30:43.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.178+0000 7fe3ad7fa700 1 -- 172.21.15.16:0/307688787 <== mgr.8820 v2:172.21.15.16:6832/3783161600 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc.) v1 ==== 138+0+0 (secure 0 0 0) 0x7fe3a40013f0 con 0x7fe39406b070 2023-10-16T12:30:43.182 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENOENT: invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2023-10-16T12:30:43.186 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-10-16T12:30:43.187 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec -1m 2023-10-16T12:30:43.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.312+0000 7f51c0cb5700 1 Processor -- start 2023-10-16T12:30:43.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.312+0000 7f51c0cb5700 1 -- start start 2023-10-16T12:30:43.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.314+0000 7f51c0cb5700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f51bc15b070 0x7f51bc15b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:43.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.314+0000 7f51c0cb5700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f51bc0cb880 0x7f51bc0cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:43.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.314+0000 7f51c0cb5700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51bc0cef70 con 0x7f51bc06da60 2023-10-16T12:30:43.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.314+0000 7f51c0cb5700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51bc0ce360 con 0x7f51bc0cb880 2023-10-16T12:30:43.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.314+0000 7f51c0cb5700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51bc0ce640 con 0x7f51bc15b070 2023-10-16T12:30:43.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.314+0000 7f51b9d9b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f51bc0cb880 0x7f51bc0cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:43.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.314+0000 7f51b9d9b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f51bc0cb880 0x7f51bc0cbd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45906/0 (socket says 172.21.15.16:45906) 2023-10-16T12:30:43.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.314+0000 7f51b9d9b700 1 -- 172.21.15.16:0/3397366476 learned_addr learned my addr 172.21.15.16:0/3397366476 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:43.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.314+0000 7f51ba59c700 1 --2- 172.21.15.16:0/3397366476 >> v2:172.21.15.49:3300/0 conn(0x7f51bc15b070 0x7f51bc15b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:43.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.314+0000 7f51b9d9b700 1 -- 172.21.15.16:0/3397366476 >> v2:172.21.15.49:3300/0 conn(0x7f51bc15b070 msgr2=0x7f51bc15b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:43.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.314+0000 7f51b9d9b700 1 --2- 172.21.15.16:0/3397366476 >> v2:172.21.15.49:3300/0 conn(0x7f51bc15b070 0x7f51bc15b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:43.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.314+0000 7f51b9d9b700 1 -- 172.21.15.16:0/3397366476 >> v1:172.21.15.49:6789/0 conn(0x7f51bc06da60 legacy=0x7f51bc0cf400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:43.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.314+0000 7f51b9d9b700 1 -- 172.21.15.16:0/3397366476 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51bc0ce920 con 0x7f51bc0cb880 2023-10-16T12:30:43.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.314+0000 7f51ba59c700 1 --2- 172.21.15.16:0/3397366476 >> v2:172.21.15.49:3300/0 conn(0x7f51bc15b070 0x7f51bc15b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:30:43.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.314+0000 7f51b9d9b700 1 --2- 172.21.15.16:0/3397366476 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51bc0cb880 0x7f51bc0cbd00 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7f51bc0705a0 tx=0x7f51b0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd3af3785da4e038 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:43.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.315+0000 7f51b959a700 1 -- 172.21.15.16:0/3397366476 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51b0004230 con 0x7f51bc0cb880 2023-10-16T12:30:43.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.315+0000 7f51b959a700 1 -- 172.21.15.16:0/3397366476 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51b0004390 con 0x7f51bc0cb880 2023-10-16T12:30:43.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.315+0000 7f51b959a700 1 -- 172.21.15.16:0/3397366476 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51b00036a0 con 0x7f51bc0cb880 2023-10-16T12:30:43.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.315+0000 7f51c0cb5700 1 -- 172.21.15.16:0/3397366476 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51bc0cb880 msgr2=0x7f51bc0cbd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:43.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.315+0000 7f51c0cb5700 1 --2- 172.21.15.16:0/3397366476 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51bc0cb880 0x7f51bc0cbd00 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7f51bc0705a0 tx=0x7f51b0009bd0 comp rx=0 tx=0).stop 2023-10-16T12:30:43.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.315+0000 7f51c0cb5700 1 -- 172.21.15.16:0/3397366476 shutdown_connections 2023-10-16T12:30:43.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.315+0000 7f51c0cb5700 1 --2- 172.21.15.16:0/3397366476 >> v2:172.21.15.49:3300/0 conn(0x7f51bc15b070 0x7f51bc15b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:43.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.315+0000 7f51c0cb5700 1 --2- 172.21.15.16:0/3397366476 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51bc0cb880 0x7f51bc0cbd00 unknown :-1 s=CLOSED pgs=1620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:43.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.315+0000 7f51c0cb5700 1 -- 172.21.15.16:0/3397366476 >> 172.21.15.16:0/3397366476 conn(0x7f51bc0b1f40 msgr2=0x7f51bc0b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:43.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.315+0000 7f51c0cb5700 1 -- 172.21.15.16:0/3397366476 shutdown_connections 2023-10-16T12:30:43.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.315+0000 7f51c0cb5700 1 -- 172.21.15.16:0/3397366476 wait complete. 2023-10-16T12:30:43.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.317+0000 7f51c0cb5700 1 Processor -- start 2023-10-16T12:30:43.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.317+0000 7f51c0cb5700 1 -- start start 2023-10-16T12:30:43.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.317+0000 7f51c0cb5700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f51bc06da60 0x7f51bc153880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:43.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.317+0000 7f51c0cb5700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f51bc0cb880 0x7f51bc159fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:43.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.317+0000 7f51c0cb5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51bc15b070 0x7f51bc0b5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:43.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.317+0000 7f51c0cb5700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51bc15a6a0 con 0x7f51bc15b070 2023-10-16T12:30:43.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.317+0000 7f51c0cb5700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51bc0c72f0 con 0x7f51bc06da60 2023-10-16T12:30:43.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.317+0000 7f51c0cb5700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51bc1570b0 con 0x7f51bc0cb880 2023-10-16T12:30:43.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.317+0000 7f51bad9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51bc15b070 0x7f51bc0b5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:43.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.317+0000 7f51bad9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51bc15b070 0x7f51bc0b5240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45914/0 (socket says 172.21.15.16:45914) 2023-10-16T12:30:43.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.318+0000 7f51bad9d700 1 -- 172.21.15.16:0/2960316501 learned_addr learned my addr 172.21.15.16:0/2960316501 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:43.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.318+0000 7f51ba59c700 1 --2- 172.21.15.16:0/2960316501 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f51bc06da60 0x7f51bc153880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:43.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.318+0000 7f51b9d9b700 1 --2- 172.21.15.16:0/2960316501 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f51bc0cb880 0x7f51bc159fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:43.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.318+0000 7f51bad9d700 1 -- 172.21.15.16:0/2960316501 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f51bc0cb880 msgr2=0x7f51bc159fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:43.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.318+0000 7f51bad9d700 1 --2- 172.21.15.16:0/2960316501 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f51bc0cb880 0x7f51bc159fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:43.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.318+0000 7f51bad9d700 1 -- 172.21.15.16:0/2960316501 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f51bc06da60 msgr2=0x7f51bc153880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:43.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.318+0000 7f51bad9d700 1 --2- 172.21.15.16:0/2960316501 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f51bc06da60 0x7f51bc153880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:43.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.318+0000 7f51bad9d700 1 -- 172.21.15.16:0/2960316501 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51b0009550 con 0x7f51bc15b070 2023-10-16T12:30:43.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.318+0000 7f51b9d9b700 1 --2- 172.21.15.16:0/2960316501 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f51bc0cb880 0x7f51bc159fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:30:43.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.318+0000 7f51ba59c700 1 --2- 172.21.15.16:0/2960316501 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f51bc06da60 0x7f51bc153880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:30:43.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.318+0000 7f51bad9d700 1 --2- 172.21.15.16:0/2960316501 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51bc15b070 0x7f51bc0b5240 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7f51b400cc10 tx=0x7f51b400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:43.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.318+0000 7f51aaffd700 1 -- 172.21.15.16:0/2960316501 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51b400abb0 con 0x7f51bc15b070 2023-10-16T12:30:43.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.318+0000 7f51aaffd700 1 -- 172.21.15.16:0/2960316501 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51b400ad10 con 0x7f51bc15b070 2023-10-16T12:30:43.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.318+0000 7f51c0cb5700 1 -- 172.21.15.16:0/2960316501 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51bc157390 con 0x7f51bc15b070 2023-10-16T12:30:43.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.318+0000 7f51aaffd700 1 -- 172.21.15.16:0/2960316501 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51b4012660 con 0x7f51bc15b070 2023-10-16T12:30:43.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.318+0000 7f51c0cb5700 1 -- 172.21.15.16:0/2960316501 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51bc157950 con 0x7f51bc15b070 2023-10-16T12:30:43.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.319+0000 7f51c0cb5700 1 -- 172.21.15.16:0/2960316501 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51bc0c83d0 con 0x7f51bc15b070 2023-10-16T12:30:43.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.320+0000 7f51aaffd700 1 -- 172.21.15.16:0/2960316501 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 80) v1 ==== 89405+0+0 (secure 0 0 0) 0x7f51b40127c0 con 0x7f51bc15b070 2023-10-16T12:30:43.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.320+0000 7f51aaffd700 1 --2- 172.21.15.16:0/2960316501 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f519c06b100 0x7f519c06d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:43.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.320+0000 7f51aaffd700 1 -- 172.21.15.16:0/2960316501 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6429+0+0 (secure 0 0 0) 0x7f51b408a300 con 0x7f51bc15b070 2023-10-16T12:30:43.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.320+0000 7f51ba59c700 1 --2- 172.21.15.16:0/2960316501 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f519c06b100 0x7f519c06d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:43.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.321+0000 7f51ba59c700 1 --2- 172.21.15.16:0/2960316501 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f519c06b100 0x7f519c06d5c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f51ac00aa20 tx=0x7f51ac008040 comp rx=0 tx=0).ready entity=mgr.8820 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:43.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.325+0000 7f51aaffd700 1 -- 172.21.15.16:0/2960316501 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f51b4054ef0 con 0x7f51bc15b070 2023-10-16T12:30:43.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.553+0000 7f51c0cb5700 1 -- 172.21.15.16:0/2960316501 --> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "-1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f51bc0728d0 con 0x7f519c06b100 2023-10-16T12:30:43.557 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:43.554+0000 7f986c0e0700 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period must be a non-zero positive value e.g. 1h or 4d etc. 2023-10-16T12:30:43.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.554+0000 7f51aaffd700 1 -- 172.21.15.16:0/2960316501 <== mgr.8820 v2:172.21.15.16:6832/3783161600 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period must be a non-zero positive value e.g. 1h or 4d etc.) v1 ==== 96+0+0 (secure 0 0 0) 0x7f51ac00a6f0 con 0x7f519c06b100 2023-10-16T12:30:43.559 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENOENT: invalid schedule specified - period must be a non-zero positive value e.g. 1h or 4d etc. 2023-10-16T12:30:43.562 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-10-16T12:30:43.563 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec '' 2023-10-16T12:30:43.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.688+0000 7f2e74630700 1 Processor -- start 2023-10-16T12:30:43.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.688+0000 7f2e74630700 1 -- start start 2023-10-16T12:30:43.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.689+0000 7f2e74630700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2e6c15f060 0x7f2e6c0cc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:43.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.689+0000 7f2e74630700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e6c0b2440 con 0x7f2e6c15ec80 2023-10-16T12:30:43.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.689+0000 7f2e74630700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e6c0d00e0 con 0x7f2e6c0ce5d0 2023-10-16T12:30:43.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.689+0000 7f2e74630700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e6c0b2c50 con 0x7f2e6c15f060 2023-10-16T12:30:43.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.689+0000 7f2e723cc700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2e6c15ec80 0x7f2e6c0cdec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35192/0 (socket says 172.21.15.16:35192) 2023-10-16T12:30:43.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.689+0000 7f2e723cc700 1 -- 172.21.15.16:0/2498802205 learned_addr learned my addr 172.21.15.16:0/2498802205 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:30:43.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.689+0000 7f2e72bcd700 1 --2- 172.21.15.16:0/2498802205 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2e6c15f060 0x7f2e6c0cc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:43.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.690+0000 7f2e713ca700 1 -- 172.21.15.16:0/2498802205 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3324916045 0 0) 0x7f2e6c0b2440 con 0x7f2e6c15ec80 2023-10-16T12:30:43.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.690+0000 7f2e713ca700 1 -- 172.21.15.16:0/2498802205 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e580035b0 con 0x7f2e6c15ec80 2023-10-16T12:30:43.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.690+0000 7f2e713ca700 1 -- 172.21.15.16:0/2498802205 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1994644264 0 0) 0x7f2e580035b0 con 0x7f2e6c15ec80 2023-10-16T12:30:43.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.690+0000 7f2e713ca700 1 -- 172.21.15.16:0/2498802205 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2e6c15f060 msgr2=0x7f2e6c0cc8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:43.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.690+0000 7f2e713ca700 1 --2- 172.21.15.16:0/2498802205 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2e6c15f060 0x7f2e6c0cc8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:43.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.690+0000 7f2e713ca700 1 -- 172.21.15.16:0/2498802205 >> v1:172.21.15.49:6789/0 conn(0x7f2e6c0ce5d0 legacy=0x7f2e6c0cc190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:43.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.690+0000 7f2e713ca700 1 -- 172.21.15.16:0/2498802205 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e6c0cef60 con 0x7f2e6c15ec80 2023-10-16T12:30:43.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.690+0000 7f2e713ca700 1 -- 172.21.15.16:0/2498802205 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f2e5c001670 con 0x7f2e6c15ec80 2023-10-16T12:30:43.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.690+0000 7f2e713ca700 1 -- 172.21.15.16:0/2498802205 >> v1:172.21.15.16:6789/0 conn(0x7f2e6c15ec80 legacy=0x7f2e6c0cdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:43.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.690+0000 7f2e713ca700 1 --2- 172.21.15.16:0/2498802205 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2e58003a30 0x7f2e58003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:43.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.690+0000 7f2e713ca700 1 --2- 172.21.15.16:0/2498802205 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2e6c15f060 0x7f2e58004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:43.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.690+0000 7f2e713ca700 1 --2- 172.21.15.16:0/2498802205 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e58004d90 0x7f2e58005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:43.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.690+0000 7f2e713ca700 1 -- 172.21.15.16:0/2498802205 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e6c0b2c50 con 0x7f2e58004d90 2023-10-16T12:30:43.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.690+0000 7f2e713ca700 1 -- 172.21.15.16:0/2498802205 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e5800ba40 con 0x7f2e58003a30 2023-10-16T12:30:43.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.690+0000 7f2e713ca700 1 -- 172.21.15.16:0/2498802205 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e5800bc00 con 0x7f2e6c15f060 2023-10-16T12:30:43.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.690+0000 7f2e72bcd700 1 --2- 172.21.15.16:0/2498802205 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e58004d90 0x7f2e58005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:43.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.690+0000 7f2e723cc700 1 --2- 172.21.15.16:0/2498802205 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2e58003a30 0x7f2e58003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:43.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.690+0000 7f2e71bcb700 1 --2- 172.21.15.16:0/2498802205 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2e6c15f060 0x7f2e58004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:43.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.690+0000 7f2e72bcd700 1 -- 172.21.15.16:0/2498802205 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2e6c15f060 msgr2=0x7f2e58004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:43.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.690+0000 7f2e72bcd700 1 --2- 172.21.15.16:0/2498802205 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2e6c15f060 0x7f2e58004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:43.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.690+0000 7f2e72bcd700 1 -- 172.21.15.16:0/2498802205 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2e58003a30 msgr2=0x7f2e58003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:43.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.690+0000 7f2e72bcd700 1 --2- 172.21.15.16:0/2498802205 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2e58003a30 0x7f2e58003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:43.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.690+0000 7f2e72bcd700 1 -- 172.21.15.16:0/2498802205 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e580035b0 con 0x7f2e58004d90 2023-10-16T12:30:43.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.691+0000 7f2e72bcd700 1 --2- 172.21.15.16:0/2498802205 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e58004d90 0x7f2e58005210 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f2e6c0cbf20 tx=0x7f2e58005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16e61f8460a642f1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:43.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.691+0000 7f2e713ca700 1 -- 172.21.15.16:0/2498802205 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e6400ebd0 con 0x7f2e58004d90 2023-10-16T12:30:43.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.691+0000 7f2e713ca700 1 -- 172.21.15.16:0/2498802205 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e6400f040 con 0x7f2e58004d90 2023-10-16T12:30:43.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.691+0000 7f2e713ca700 1 -- 172.21.15.16:0/2498802205 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e64014480 con 0x7f2e58004d90 2023-10-16T12:30:43.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.691+0000 7f2e74630700 1 -- 172.21.15.16:0/2498802205 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e58004d90 msgr2=0x7f2e58005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:43.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.691+0000 7f2e74630700 1 --2- 172.21.15.16:0/2498802205 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e58004d90 0x7f2e58005210 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f2e6c0cbf20 tx=0x7f2e58005880 comp rx=0 tx=0).stop 2023-10-16T12:30:43.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.691+0000 7f2e72bcd700 1 -- 172.21.15.16:0/2498802205 reap_dead start 2023-10-16T12:30:43.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.691+0000 7f2e74630700 1 -- 172.21.15.16:0/2498802205 shutdown_connections 2023-10-16T12:30:43.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.691+0000 7f2e74630700 1 -- 172.21.15.16:0/2498802205 >> 172.21.15.16:0/2498802205 conn(0x7f2e6c0d15f0 msgr2=0x7f2e6c0d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:43.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.691+0000 7f2e74630700 1 -- 172.21.15.16:0/2498802205 shutdown_connections 2023-10-16T12:30:43.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.691+0000 7f2e74630700 1 -- 172.21.15.16:0/2498802205 wait complete. 2023-10-16T12:30:43.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.693+0000 7f2e74630700 1 Processor -- start 2023-10-16T12:30:43.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.693+0000 7f2e74630700 1 -- start start 2023-10-16T12:30:43.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.693+0000 7f2e74630700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2e58004d90 0x7f2e6c156ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:43.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.693+0000 7f2e74630700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2e6c150cc0 0x7f2e6c151140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:43.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.693+0000 7f2e74630700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e6c150170 0x7f2e6c1505f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:43.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.693+0000 7f2e74630700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e6c150b30 con 0x7f2e6c150170 2023-10-16T12:30:43.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.693+0000 7f2e74630700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e6c1548f0 con 0x7f2e6c150cc0 2023-10-16T12:30:43.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.693+0000 7f2e74630700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e6c1592b0 con 0x7f2e58004d90 2023-10-16T12:30:43.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.693+0000 7f2e72bcd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e6c150170 0x7f2e6c1505f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:43.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.694+0000 7f2e72bcd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e6c150170 0x7f2e6c1505f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45932/0 (socket says 172.21.15.16:45932) 2023-10-16T12:30:43.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.694+0000 7f2e72bcd700 1 -- 172.21.15.16:0/1141929734 learned_addr learned my addr 172.21.15.16:0/1141929734 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:43.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.694+0000 7f2e723cc700 1 --2- 172.21.15.16:0/1141929734 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2e58004d90 0x7f2e6c156ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:43.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.694+0000 7f2e71bcb700 1 --2- 172.21.15.16:0/1141929734 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2e6c150cc0 0x7f2e6c151140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:43.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.694+0000 7f2e72bcd700 1 -- 172.21.15.16:0/1141929734 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2e58004d90 msgr2=0x7f2e6c156ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:43.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.694+0000 7f2e72bcd700 1 --2- 172.21.15.16:0/1141929734 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2e58004d90 0x7f2e6c156ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:43.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.694+0000 7f2e72bcd700 1 -- 172.21.15.16:0/1141929734 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2e6c150cc0 msgr2=0x7f2e6c151140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:43.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.694+0000 7f2e72bcd700 1 --2- 172.21.15.16:0/1141929734 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2e6c150cc0 0x7f2e6c151140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:43.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.694+0000 7f2e72bcd700 1 -- 172.21.15.16:0/1141929734 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e6c152000 con 0x7f2e6c150170 2023-10-16T12:30:43.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.694+0000 7f2e72bcd700 1 --2- 172.21.15.16:0/1141929734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e6c150170 0x7f2e6c1505f0 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f2e6400e700 tx=0x7f2e64004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:43.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.694+0000 7f2e62ffd700 1 -- 172.21.15.16:0/1141929734 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e64007000 con 0x7f2e6c150170 2023-10-16T12:30:43.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.694+0000 7f2e62ffd700 1 -- 172.21.15.16:0/1141929734 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e64003c40 con 0x7f2e6c150170 2023-10-16T12:30:43.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.694+0000 7f2e62ffd700 1 -- 172.21.15.16:0/1141929734 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e640149c0 con 0x7f2e6c150170 2023-10-16T12:30:43.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.694+0000 7f2e74630700 1 -- 172.21.15.16:0/1141929734 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e6c159570 con 0x7f2e6c150170 2023-10-16T12:30:43.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.694+0000 7f2e74630700 1 -- 172.21.15.16:0/1141929734 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e6c151a40 con 0x7f2e6c150170 2023-10-16T12:30:43.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.695+0000 7f2e74630700 1 -- 172.21.15.16:0/1141929734 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e6c159700 con 0x7f2e6c150170 2023-10-16T12:30:43.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.696+0000 7f2e62ffd700 1 -- 172.21.15.16:0/1141929734 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 80) v1 ==== 89405+0+0 (secure 0 0 0) 0x7f2e64023070 con 0x7f2e6c150170 2023-10-16T12:30:43.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.696+0000 7f2e62ffd700 1 --2- 172.21.15.16:0/1141929734 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f2e5006b100 0x7f2e5006d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:43.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.696+0000 7f2e62ffd700 1 -- 172.21.15.16:0/1141929734 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6429+0+0 (secure 0 0 0) 0x7f2e6408aa70 con 0x7f2e6c150170 2023-10-16T12:30:43.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.696+0000 7f2e723cc700 1 --2- 172.21.15.16:0/1141929734 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f2e5006b100 0x7f2e5006d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:43.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.697+0000 7f2e723cc700 1 --2- 172.21.15.16:0/1141929734 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f2e5006b100 0x7f2e5006d5c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f2e5c000c00 tx=0x7f2e5c000f80 comp rx=0 tx=0).ready entity=mgr.8820 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:43.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.701+0000 7f2e62ffd700 1 -- 172.21.15.16:0/1141929734 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2e64055780 con 0x7f2e6c150170 2023-10-16T12:30:43.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.932+0000 7f2e74630700 1 -- 172.21.15.16:0/1141929734 --> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "", "target": ["mon-mgr", ""]}) v1 -- 0x7f2e6c0b2970 con 0x7f2e5006b100 2023-10-16T12:30:43.935 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:43.933+0000 7f986c0e0700 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2023-10-16T12:30:43.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:43.933+0000 7f2e62ffd700 1 -- 172.21.15.16:0/1141929734 <== mgr.8820 v2:172.21.15.16:6832/3783161600 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc.) v1 ==== 138+0+0 (secure 0 0 0) 0x7f2e5c0070c0 con 0x7f2e5006b100 2023-10-16T12:30:43.937 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENOENT: invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2023-10-16T12:30:43.946 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-10-16T12:30:43.951 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:30:43.952 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/minutes) 2023-10-16T12:30:43.978 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:30:43.979 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/hourly) 2023-10-16T12:30:44.041 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:30:44.042 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/daily) 2023-10-16T12:30:44.102 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:30:44.103 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/weekly) 2023-10-16T12:30:44.168 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:30:44.169 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/monthly) 2023-10-16T12:30:44.242 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:30:44.243 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/yearly) 2023-10-16T12:30:44.310 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:30:44.311 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/bad_period_spec) 2023-10-16T12:30:44.371 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-16T12:30:44.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:44.539+0000 7f0ba6a60700 1 Processor -- start 2023-10-16T12:30:44.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:44.539+0000 7f0ba6a60700 1 -- start start 2023-10-16T12:30:44.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:44.540+0000 7f0ba6a60700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f0ba015da10 0x7f0ba00d1520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:44.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:44.540+0000 7f0ba6a60700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0ba00d1d20 0x7f0ba00d21b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:44.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:44.540+0000 7f0ba6a60700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ba00d4540 con 0x7f0ba00d4060 2023-10-16T12:30:44.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:44.540+0000 7f0ba6a60700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ba00cf9b0 con 0x7f0ba00d1d20 2023-10-16T12:30:44.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:44.540+0000 7f0ba6a60700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ba00d26f0 con 0x7f0ba015da10 2023-10-16T12:30:44.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:44.540+0000 7f0ba4ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0ba00d1d20 0x7f0ba00d21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:44.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:44.540+0000 7f0ba4ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0ba00d1d20 0x7f0ba00d21b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45938/0 (socket says 172.21.15.16:45938) 2023-10-16T12:30:44.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:44.540+0000 7f0ba4ffd700 1 -- 172.21.15.16:0/710108778 learned_addr learned my addr 172.21.15.16:0/710108778 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:44.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:44.540+0000 7f0b9ffff700 1 --2- 172.21.15.16:0/710108778 >> v2:172.21.15.49:3300/0 conn(0x7f0ba015da10 0x7f0ba00d1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:44.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:44.540+0000 7f0ba4ffd700 1 -- 172.21.15.16:0/710108778 >> v2:172.21.15.49:3300/0 conn(0x7f0ba015da10 msgr2=0x7f0ba00d1520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:44.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:44.540+0000 7f0ba4ffd700 1 --2- 172.21.15.16:0/710108778 >> v2:172.21.15.49:3300/0 conn(0x7f0ba015da10 0x7f0ba00d1520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:44.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:44.540+0000 7f0ba4ffd700 1 -- 172.21.15.16:0/710108778 >> v1:172.21.15.16:6789/0 conn(0x7f0ba00d4060 legacy=0x7f0ba00cf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:44.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:44.540+0000 7f0ba4ffd700 1 -- 172.21.15.16:0/710108778 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ba00ce7e0 con 0x7f0ba00d1d20 2023-10-16T12:30:44.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:44.540+0000 7f0b9ffff700 1 --2- 172.21.15.16:0/710108778 >> v2:172.21.15.49:3300/0 conn(0x7f0ba015da10 0x7f0ba00d1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:30:44.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:44.540+0000 7f0ba4ffd700 1 --2- 172.21.15.16:0/710108778 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ba00d1d20 0x7f0ba00d21b0 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7f0ba008b540 tx=0x7f0b9800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d7a1160bc2e6b66 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:44.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:44.541+0000 7f0b9effd700 1 -- 172.21.15.16:0/710108778 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b9800e070 con 0x7f0ba00d1d20 2023-10-16T12:30:44.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:44.541+0000 7f0b9effd700 1 -- 172.21.15.16:0/710108778 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b9800b9c0 con 0x7f0ba00d1d20 2023-10-16T12:30:44.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:44.541+0000 7f0b9effd700 1 -- 172.21.15.16:0/710108778 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b9800bd70 con 0x7f0ba00d1d20 2023-10-16T12:30:44.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:44.541+0000 7f0ba6a60700 1 -- 172.21.15.16:0/710108778 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ba00d1d20 msgr2=0x7f0ba00d21b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:44.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:44.541+0000 7f0ba6a60700 1 --2- 172.21.15.16:0/710108778 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ba00d1d20 0x7f0ba00d21b0 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7f0ba008b540 tx=0x7f0b9800b330 comp rx=0 tx=0).stop 2023-10-16T12:30:44.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:44.541+0000 7f0ba6a60700 1 -- 172.21.15.16:0/710108778 shutdown_connections 2023-10-16T12:30:44.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:44.541+0000 7f0ba6a60700 1 --2- 172.21.15.16:0/710108778 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ba00d1d20 0x7f0ba00d21b0 unknown :-1 s=CLOSED pgs=1626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:44.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:44.541+0000 7f0ba6a60700 1 --2- 172.21.15.16:0/710108778 >> v2:172.21.15.49:3300/0 conn(0x7f0ba015da10 0x7f0ba00d1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:44.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:44.541+0000 7f0ba6a60700 1 -- 172.21.15.16:0/710108778 >> 172.21.15.16:0/710108778 conn(0x7f0ba009c9f0 msgr2=0x7f0ba00d3740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:44.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:44.541+0000 7f0ba6a60700 1 -- 172.21.15.16:0/710108778 shutdown_connections 2023-10-16T12:30:44.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:44.541+0000 7f0ba6a60700 1 -- 172.21.15.16:0/710108778 wait complete. 2023-10-16T12:30:44.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:44.543+0000 7f0ba6a60700 1 Processor -- start 2023-10-16T12:30:44.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:44.543+0000 7f0ba6a60700 1 -- start start 2023-10-16T12:30:44.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:44.543+0000 7f0ba6a60700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ba00d1d20 0x7f0ba015c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:44.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:44.543+0000 7f0ba6a60700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0ba00d4060 0x7f0ba00ca080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:44.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:44.543+0000 7f0ba6a60700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0ba015da10 0x7f0ba00b8670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:44.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:44.543+0000 7f0ba6a60700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ba00cbbf0 con 0x7f0ba00d1d20 2023-10-16T12:30:44.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:44.543+0000 7f0ba6a60700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ba00ca5c0 con 0x7f0ba015da10 2023-10-16T12:30:44.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:44.543+0000 7f0ba6a60700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ba00cb2c0 con 0x7f0ba00d4060 2023-10-16T12:30:44.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:44.544+0000 7f0b9ffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ba00d1d20 0x7f0ba015c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:44.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:44.544+0000 7f0b9ffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ba00d1d20 0x7f0ba015c240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45942/0 (socket says 172.21.15.16:45942) 2023-10-16T12:30:44.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:44.544+0000 7f0b9ffff700 1 -- 172.21.15.16:0/3184719717 learned_addr learned my addr 172.21.15.16:0/3184719717 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:44.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:44.544+0000 7f0b9ffff700 1 -- 172.21.15.16:0/3184719717 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0ba00d4060 msgr2=0x7f0ba00ca080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:44.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:44.544+0000 7f0b9ffff700 1 --2- 172.21.15.16:0/3184719717 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0ba00d4060 0x7f0ba00ca080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:44.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:44.544+0000 7f0b9ffff700 1 -- 172.21.15.16:0/3184719717 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0ba015da10 msgr2=0x7f0ba00b8670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:44.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:44.544+0000 7f0b9ffff700 1 --2- 172.21.15.16:0/3184719717 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0ba015da10 0x7f0ba00b8670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:44.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:44.544+0000 7f0b9ffff700 1 -- 172.21.15.16:0/3184719717 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b98009b30 con 0x7f0ba00d1d20 2023-10-16T12:30:44.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:44.544+0000 7f0b9ffff700 1 --2- 172.21.15.16:0/3184719717 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ba00d1d20 0x7f0ba015c240 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7f0b9000c7f0 tx=0x7f0b9000cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:44.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:44.544+0000 7f0b9cff9700 1 -- 172.21.15.16:0/3184719717 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b90005c20 con 0x7f0ba00d1d20 2023-10-16T12:30:44.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:44.544+0000 7f0b9cff9700 1 -- 172.21.15.16:0/3184719717 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b90005d80 con 0x7f0ba00d1d20 2023-10-16T12:30:44.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:44.544+0000 7f0ba6a60700 1 -- 172.21.15.16:0/3184719717 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ba00cb5a0 con 0x7f0ba00d1d20 2023-10-16T12:30:44.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:44.544+0000 7f0b9cff9700 1 -- 172.21.15.16:0/3184719717 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b90011820 con 0x7f0ba00d1d20 2023-10-16T12:30:44.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:44.544+0000 7f0ba6a60700 1 -- 172.21.15.16:0/3184719717 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ba00cc080 con 0x7f0ba00d1d20 2023-10-16T12:30:44.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:44.545+0000 7f0ba6a60700 1 -- 172.21.15.16:0/3184719717 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ba00ced80 con 0x7f0ba00d1d20 2023-10-16T12:30:44.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:44.546+0000 7f0b9cff9700 1 -- 172.21.15.16:0/3184719717 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 80) v1 ==== 89405+0+0 (secure 0 0 0) 0x7f0b90011980 con 0x7f0ba00d1d20 2023-10-16T12:30:44.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:44.546+0000 7f0b9cff9700 1 --2- 172.21.15.16:0/3184719717 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f0b8006b130 0x7f0b8006d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:44.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:44.546+0000 7f0b9cff9700 1 -- 172.21.15.16:0/3184719717 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6429+0+0 (secure 0 0 0) 0x7f0b90089bd0 con 0x7f0ba00d1d20 2023-10-16T12:30:44.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:44.546+0000 7f0b9f7fe700 1 --2- 172.21.15.16:0/3184719717 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f0b8006b130 0x7f0b8006d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:44.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:44.547+0000 7f0b9f7fe700 1 --2- 172.21.15.16:0/3184719717 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f0b8006b130 0x7f0b8006d5f0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0ba00b92c0 tx=0x7f0b88007a40 comp rx=0 tx=0).ready entity=mgr.8820 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:44.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:44.551+0000 7f0b9cff9700 1 -- 172.21.15.16:0/3184719717 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0ba00ced80 con 0x7f0ba00d1d20 2023-10-16T12:30:44.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:44.813+0000 7f0ba6a60700 1 -- 172.21.15.16:0/3184719717 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f0ba00ced80 con 0x7f0ba00d1d20 2023-10-16T12:30:45.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.337+0000 7f0b9cff9700 1 -- 172.21.15.16:0/3184719717 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v81) v1 ==== 93+0+0 (secure 0 0 0) 0x7f0ba00ced80 con 0x7f0ba00d1d20 2023-10-16T12:30:45.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.340+0000 7f0ba6a60700 1 -- 172.21.15.16:0/3184719717 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f0b8006b130 msgr2=0x7f0b8006d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:45.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.340+0000 7f0ba6a60700 1 --2- 172.21.15.16:0/3184719717 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f0b8006b130 0x7f0b8006d5f0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0ba00b92c0 tx=0x7f0b88007a40 comp rx=0 tx=0).stop 2023-10-16T12:30:45.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.340+0000 7f0ba6a60700 1 -- 172.21.15.16:0/3184719717 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ba00d1d20 msgr2=0x7f0ba015c240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:45.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.340+0000 7f0ba6a60700 1 --2- 172.21.15.16:0/3184719717 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ba00d1d20 0x7f0ba015c240 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7f0b9000c7f0 tx=0x7f0b9000cbb0 comp rx=0 tx=0).stop 2023-10-16T12:30:45.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.340+0000 7f0ba6a60700 1 -- 172.21.15.16:0/3184719717 shutdown_connections 2023-10-16T12:30:45.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.340+0000 7f0ba6a60700 1 --2- 172.21.15.16:0/3184719717 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f0b8006b130 0x7f0b8006d5f0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:45.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.340+0000 7f0ba6a60700 1 --2- 172.21.15.16:0/3184719717 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0ba015da10 0x7f0ba00b8670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:45.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.340+0000 7f0ba6a60700 1 --2- 172.21.15.16:0/3184719717 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ba00d1d20 0x7f0ba015c240 unknown :-1 s=CLOSED pgs=1627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:45.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.340+0000 7f0ba6a60700 1 --2- 172.21.15.16:0/3184719717 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0ba00d4060 0x7f0ba00ca080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:45.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.340+0000 7f0ba6a60700 1 -- 172.21.15.16:0/3184719717 >> 172.21.15.16:0/3184719717 conn(0x7f0ba009c9f0 msgr2=0x7f0ba00d0330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:45.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.341+0000 7f0ba6a60700 1 -- 172.21.15.16:0/3184719717 shutdown_connections 2023-10-16T12:30:45.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.341+0000 7f0ba6a60700 1 -- 172.21.15.16:0/3184719717 wait complete. 2023-10-16T12:30:45.361 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T12:30:45.362 DEBUG:teuthology.orchestra.run.smithi016:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T12:30:45.413 DEBUG:teuthology.orchestra.run.smithi049:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T12:30:45.456 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T12:30:45.456 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:30:45.457 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:30:45.475 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:30:45.476 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:30:45.493 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:30:45.492+0000 7f3b079a5200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T12:30:45.499 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:45.496+0000 7f6bd6932200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T12:30:45.517 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:30:45.563 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:30:45.562+0000 7f3b079a5200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T12:30:45.569 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:45.567+0000 7f6bd6932200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T12:30:45.639 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:30:45.637+0000 7f3b079a5200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T12:30:45.647 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:45.644+0000 7f6bd6932200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T12:30:45.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.681+0000 7f5347468700 1 Processor -- start 2023-10-16T12:30:45.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.681+0000 7f5347468700 1 -- start start 2023-10-16T12:30:45.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.682+0000 7f5347468700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f534015d900 0x7f53400d2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:45.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.682+0000 7f5347468700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f53400d2f70 0x7f53400cdc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:45.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.682+0000 7f5347468700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53400ce1a0 con 0x7f53400ce9f0 2023-10-16T12:30:45.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.682+0000 7f5347468700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53400cee70 con 0x7f53400d2f70 2023-10-16T12:30:45.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.682+0000 7f5347468700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53400ce400 con 0x7f534015d900 2023-10-16T12:30:45.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.682+0000 7f5345a05700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f53400d2f70 0x7f53400cdc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:45.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.682+0000 7f5345a05700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f53400d2f70 0x7f53400cdc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46018/0 (socket says 172.21.15.16:46018) 2023-10-16T12:30:45.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.682+0000 7f5345a05700 1 -- 172.21.15.16:0/3485183944 learned_addr learned my addr 172.21.15.16:0/3485183944 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:45.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.682+0000 7f5345204700 1 --2- 172.21.15.16:0/3485183944 >> v2:172.21.15.49:3300/0 conn(0x7f534015d900 0x7f53400d2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:45.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.682+0000 7f5345a05700 1 -- 172.21.15.16:0/3485183944 >> v2:172.21.15.49:3300/0 conn(0x7f534015d900 msgr2=0x7f53400d2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:45.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.682+0000 7f5345a05700 1 --2- 172.21.15.16:0/3485183944 >> v2:172.21.15.49:3300/0 conn(0x7f534015d900 0x7f53400d2040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:45.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.682+0000 7f5345a05700 1 -- 172.21.15.16:0/3485183944 >> v1:172.21.15.16:6789/0 conn(0x7f53400ce9f0 legacy=0x7f53400d2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:45.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.682+0000 7f5337fff700 1 -- 172.21.15.16:0/3485183944 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2153343702 0 0) 0x7f53400ce1a0 con 0x7f53400ce9f0 2023-10-16T12:30:45.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.682+0000 7f5345a05700 1 -- 172.21.15.16:0/3485183944 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f534015dce0 con 0x7f53400d2f70 2023-10-16T12:30:45.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.682+0000 7f5345a05700 1 --2- 172.21.15.16:0/3485183944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53400d2f70 0x7f53400cdc60 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7f53400d4710 tx=0x7f533800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=863a38fe98010a4b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:45.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.683+0000 7f5337fff700 1 -- 172.21.15.16:0/3485183944 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f533800e070 con 0x7f53400d2f70 2023-10-16T12:30:45.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.683+0000 7f5337fff700 1 -- 172.21.15.16:0/3485183944 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f533800b9c0 con 0x7f53400d2f70 2023-10-16T12:30:45.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.683+0000 7f5337fff700 1 -- 172.21.15.16:0/3485183944 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f533800bd70 con 0x7f53400d2f70 2023-10-16T12:30:45.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.683+0000 7f5347468700 1 -- 172.21.15.16:0/3485183944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53400d2f70 msgr2=0x7f53400cdc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:45.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.683+0000 7f5347468700 1 --2- 172.21.15.16:0/3485183944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53400d2f70 0x7f53400cdc60 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7f53400d4710 tx=0x7f533800b330 comp rx=0 tx=0).stop 2023-10-16T12:30:45.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.683+0000 7f5347468700 1 -- 172.21.15.16:0/3485183944 shutdown_connections 2023-10-16T12:30:45.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.683+0000 7f5347468700 1 --2- 172.21.15.16:0/3485183944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53400d2f70 0x7f53400cdc60 unknown :-1 s=CLOSED pgs=1631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:45.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.683+0000 7f5347468700 1 --2- 172.21.15.16:0/3485183944 >> v2:172.21.15.49:3300/0 conn(0x7f534015d900 0x7f53400d2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:45.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.683+0000 7f5347468700 1 -- 172.21.15.16:0/3485183944 >> 172.21.15.16:0/3485183944 conn(0x7f5340072330 msgr2=0x7f5340072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:45.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.683+0000 7f5347468700 1 -- 172.21.15.16:0/3485183944 shutdown_connections 2023-10-16T12:30:45.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.683+0000 7f5347468700 1 -- 172.21.15.16:0/3485183944 wait complete. 2023-10-16T12:30:45.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.685+0000 7f5347468700 1 Processor -- start 2023-10-16T12:30:45.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.685+0000 7f5347468700 1 -- start start 2023-10-16T12:30:45.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.686+0000 7f5347468700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f53400ce9f0 0x7f53400b8e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:45.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.686+0000 7f5347468700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53400d2f70 0x7f534015ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:45.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.686+0000 7f5347468700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f534015d900 0x7f534014cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:45.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.686+0000 7f5347468700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f534015e410 con 0x7f53400d2f70 2023-10-16T12:30:45.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.686+0000 7f5347468700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f534015e6b0 con 0x7f53400ce9f0 2023-10-16T12:30:45.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.686+0000 7f5347468700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f534015e990 con 0x7f534015d900 2023-10-16T12:30:45.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.686+0000 7f5344a03700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53400d2f70 0x7f534015ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:45.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.686+0000 7f5345204700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f53400ce9f0 0x7f53400b8e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:45.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.686+0000 7f5345204700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f53400ce9f0 0x7f53400b8e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:45082/0 (socket says 172.21.15.16:45082) 2023-10-16T12:30:45.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.686+0000 7f5345204700 1 -- 172.21.15.16:0/2855412909 learned_addr learned my addr 172.21.15.16:0/2855412909 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:45.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.686+0000 7f5344a03700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53400d2f70 0x7f534015ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46022/0 (socket says 172.21.15.16:46022) 2023-10-16T12:30:45.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.686+0000 7f5345a05700 1 --2- 172.21.15.16:0/2855412909 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f534015d900 0x7f534014cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:45.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.687+0000 7f5344a03700 1 -- 172.21.15.16:0/2855412909 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f534015d900 msgr2=0x7f534014cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:45.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.687+0000 7f5344a03700 1 --2- 172.21.15.16:0/2855412909 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f534015d900 0x7f534014cab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:45.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.687+0000 7f5344a03700 1 -- 172.21.15.16:0/2855412909 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f53400ce9f0 msgr2=0x7f53400b8e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:45.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.687+0000 7f5344a03700 1 --2- 172.21.15.16:0/2855412909 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f53400ce9f0 0x7f53400b8e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:45.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.687+0000 7f5344a03700 1 -- 172.21.15.16:0/2855412909 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5338009b30 con 0x7f53400d2f70 2023-10-16T12:30:45.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.687+0000 7f5345204700 1 --2- 172.21.15.16:0/2855412909 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f53400ce9f0 0x7f53400b8e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:30:45.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.687+0000 7f5344a03700 1 --2- 172.21.15.16:0/2855412909 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53400d2f70 0x7f534015ded0 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7f533000c3e0 tx=0x7f533000c6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:45.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.687+0000 7f5335ffb700 1 -- 172.21.15.16:0/2855412909 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53300052d0 con 0x7f53400d2f70 2023-10-16T12:30:45.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.687+0000 7f5335ffb700 1 -- 172.21.15.16:0/2855412909 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5330005430 con 0x7f53400d2f70 2023-10-16T12:30:45.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.687+0000 7f5347468700 1 -- 172.21.15.16:0/2855412909 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f534015ec70 con 0x7f53400d2f70 2023-10-16T12:30:45.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.687+0000 7f5335ffb700 1 -- 172.21.15.16:0/2855412909 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53300038a0 con 0x7f53400d2f70 2023-10-16T12:30:45.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.687+0000 7f5347468700 1 -- 172.21.15.16:0/2855412909 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f534015f230 con 0x7f53400d2f70 2023-10-16T12:30:45.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.688+0000 7f5347468700 1 -- 172.21.15.16:0/2855412909 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f534004ebb0 con 0x7f53400d2f70 2023-10-16T12:30:45.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.689+0000 7f5335ffb700 1 -- 172.21.15.16:0/2855412909 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 81) v1 ==== 89388+0+0 (secure 0 0 0) 0x7f5330003a00 con 0x7f53400d2f70 2023-10-16T12:30:45.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.689+0000 7f5335ffb700 1 --2- 172.21.15.16:0/2855412909 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f532406b0b0 0x7f532406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:45.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.689+0000 7f5345204700 1 -- 172.21.15.16:0/2855412909 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f532406b0b0 msgr2=0x7f532406d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3783161600 2023-10-16T12:30:45.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.689+0000 7f5335ffb700 1 -- 172.21.15.16:0/2855412909 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6429+0+0 (secure 0 0 0) 0x7f5330088ec0 con 0x7f53400d2f70 2023-10-16T12:30:45.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.689+0000 7f5345204700 1 --2- 172.21.15.16:0/2855412909 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f532406b0b0 0x7f532406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:30:45.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.694+0000 7f5335ffb700 1 -- 172.21.15.16:0/2855412909 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5330053a20 con 0x7f53400d2f70 2023-10-16T12:30:45.720 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:30:45.719+0000 7f3b079a5200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T12:30:45.733 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:45.730+0000 7f6bd6932200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T12:30:45.869 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:30:45.868+0000 7f3b079a5200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T12:30:45.882 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:45.880+0000 7f6bd6932200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T12:30:45.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.890+0000 7f5345204700 1 -- 172.21.15.16:0/2855412909 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f532406b0b0 msgr2=0x7f532406d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3783161600 2023-10-16T12:30:45.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.890+0000 7f5345204700 1 --2- 172.21.15.16:0/2855412909 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f532406b0b0 0x7f532406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:30:45.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.900+0000 7f5347468700 1 -- 172.21.15.16:0/2855412909 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f534014cff0 con 0x7f53400d2f70 2023-10-16T12:30:45.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.901+0000 7f5335ffb700 1 -- 172.21.15.16:0/2855412909 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v143) v1 ==== 74+0+17263 (secure 0 0 0) 0x7f5330017020 con 0x7f53400d2f70 2023-10-16T12:30:45.905 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:30:45.905 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":143,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:30:37.941062+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:30:21.318498+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"141","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2023-10-16T12:30:21.813382+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"142","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/2901217010":"2023-10-17T12:30:37.941028+0000","172.21.15.16:6833/258291157":"2023-10-17T12:30:37.941028+0000","172.21.15.16:6832/258291157":"2023-10-17T12:30:37.941028+0000","172.21.15.16:0/4123017371":"2023-10-17T12:30:37.941028+0000","172.21.15.16:0/2703050386":"2023-10-17T12:30:37.941028+0000","172.21.15.16:0/3188488837":"2023-10-17T12:30:37.941028+0000","172.21.15.16:0/2621298501":"2023-10-17T12:30:37.941028+0000","172.21.15.16:0/1781882931":"2023-10-17T12:30:37.941028+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:30:45.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.903+0000 7f5347468700 1 -- 172.21.15.16:0/2855412909 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f532406b0b0 msgr2=0x7f532406d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:30:45.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.903+0000 7f5347468700 1 --2- 172.21.15.16:0/2855412909 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f532406b0b0 0x7f532406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:45.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.903+0000 7f5347468700 1 -- 172.21.15.16:0/2855412909 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53400d2f70 msgr2=0x7f534015ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:45.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.903+0000 7f5347468700 1 --2- 172.21.15.16:0/2855412909 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53400d2f70 0x7f534015ded0 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7f533000c3e0 tx=0x7f533000c6f0 comp rx=0 tx=0).stop 2023-10-16T12:30:45.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.903+0000 7f5347468700 1 -- 172.21.15.16:0/2855412909 shutdown_connections 2023-10-16T12:30:45.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.903+0000 7f5347468700 1 --2- 172.21.15.16:0/2855412909 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f532406b0b0 0x7f532406d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:45.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.903+0000 7f5347468700 1 --2- 172.21.15.16:0/2855412909 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f534015d900 0x7f534014cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:45.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.903+0000 7f5347468700 1 --2- 172.21.15.16:0/2855412909 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f53400ce9f0 0x7f53400b8e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:45.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.903+0000 7f5347468700 1 --2- 172.21.15.16:0/2855412909 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53400d2f70 0x7f534015ded0 unknown :-1 s=CLOSED pgs=1632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:45.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.903+0000 7f5347468700 1 -- 172.21.15.16:0/2855412909 >> 172.21.15.16:0/2855412909 conn(0x7f5340072330 msgr2=0x7f5340158110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:45.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.904+0000 7f5347468700 1 -- 172.21.15.16:0/2855412909 shutdown_connections 2023-10-16T12:30:45.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:45.904+0000 7f5347468700 1 -- 172.21.15.16:0/2855412909 wait complete. 2023-10-16T12:30:45.920 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T12:30:21.318498+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '141', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T12:30:21.813382+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '142', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T12:30:45.921 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:30:46.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.087+0000 7f1181ef9700 1 Processor -- start 2023-10-16T12:30:46.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.087+0000 7f1181ef9700 1 -- start start 2023-10-16T12:30:46.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.087+0000 7f1181ef9700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f117c0ce5d0 0x7f117c0cc190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:46.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.087+0000 7f1181ef9700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f117c15f060 0x7f117c0cc6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:46.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.087+0000 7f1181ef9700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f117c0b2440 con 0x7f117c15ec80 2023-10-16T12:30:46.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.087+0000 7f1181ef9700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f117c0b2c50 con 0x7f117c0ce5d0 2023-10-16T12:30:46.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.087+0000 7f1181ef9700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f117c0d00e0 con 0x7f117c15f060 2023-10-16T12:30:46.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.088+0000 7f117bfff700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f117c15f060 0x7f117c0cc6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:46.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.088+0000 7f117affd700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f117c0ce5d0 0x7f117c0cc190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:46.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.088+0000 7f117b7fe700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7f117c15ec80 0x7f117c0cdec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:42530/0 (socket says 172.21.15.16:42530) 2023-10-16T12:30:46.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.088+0000 7f117b7fe700 1 -- 172.21.15.16:0/373737325 learned_addr learned my addr 172.21.15.16:0/373737325 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:30:46.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.088+0000 7f117affd700 1 -- 172.21.15.16:0/373737325 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f117c15f060 msgr2=0x7f117c0cc6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:46.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.088+0000 7f117affd700 1 --2- 172.21.15.16:0/373737325 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f117c15f060 0x7f117c0cc6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:46.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.088+0000 7f117affd700 1 -- 172.21.15.16:0/373737325 >> v1:172.21.15.49:6789/0 conn(0x7f117c15ec80 legacy=0x7f117c0cdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:46.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.088+0000 7f117affd700 1 -- 172.21.15.16:0/373737325 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f117c0cef60 con 0x7f117c0ce5d0 2023-10-16T12:30:46.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.088+0000 7f117a7fc700 1 -- 172.21.15.16:0/373737325 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4282574359 0 0) 0x7f117c0b2440 con 0x7f117c15ec80 2023-10-16T12:30:46.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.088+0000 7f117affd700 1 --2- 172.21.15.16:0/373737325 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f117c0ce5d0 0x7f117c0cc190 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f1170009840 tx=0x7f1170009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=84616dd49f5fd042 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:46.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.088+0000 7f117a7fc700 1 -- 172.21.15.16:0/373737325 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1170011830 con 0x7f117c0ce5d0 2023-10-16T12:30:46.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.088+0000 7f117a7fc700 1 -- 172.21.15.16:0/373737325 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1170011990 con 0x7f117c0ce5d0 2023-10-16T12:30:46.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.088+0000 7f117a7fc700 1 -- 172.21.15.16:0/373737325 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1170011d20 con 0x7f117c0ce5d0 2023-10-16T12:30:46.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.089+0000 7f1181ef9700 1 -- 172.21.15.16:0/373737325 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f117c0ce5d0 msgr2=0x7f117c0cc190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:46.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.089+0000 7f1181ef9700 1 --2- 172.21.15.16:0/373737325 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f117c0ce5d0 0x7f117c0cc190 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f1170009840 tx=0x7f1170009b50 comp rx=0 tx=0).stop 2023-10-16T12:30:46.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.089+0000 7f1181ef9700 1 -- 172.21.15.16:0/373737325 shutdown_connections 2023-10-16T12:30:46.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.089+0000 7f1181ef9700 1 --2- 172.21.15.16:0/373737325 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f117c15f060 0x7f117c0cc6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:46.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.089+0000 7f1181ef9700 1 --2- 172.21.15.16:0/373737325 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f117c0ce5d0 0x7f117c0cc190 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:46.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.089+0000 7f1181ef9700 1 -- 172.21.15.16:0/373737325 >> 172.21.15.16:0/373737325 conn(0x7f117c0d15f0 msgr2=0x7f117c0d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:46.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.089+0000 7f1181ef9700 1 -- 172.21.15.16:0/373737325 shutdown_connections 2023-10-16T12:30:46.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.089+0000 7f1181ef9700 1 -- 172.21.15.16:0/373737325 wait complete. 2023-10-16T12:30:46.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.090+0000 7f1181ef9700 1 Processor -- start 2023-10-16T12:30:46.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.091+0000 7f1181ef9700 1 -- start start 2023-10-16T12:30:46.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.091+0000 7f1181ef9700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f117c0ce5d0 0x7f117c1e51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:46.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.091+0000 7f1181ef9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f117c15ec80 0x7f117c0d19d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:46.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.091+0000 7f1181ef9700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f117c15f060 0x7f117c0d1f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:46.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.091+0000 7f1181ef9700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f117c0c6850 con 0x7f117c15ec80 2023-10-16T12:30:46.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.091+0000 7f1181ef9700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f117c1e4510 con 0x7f117c15f060 2023-10-16T12:30:46.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.091+0000 7f1181ef9700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f117c1e47f0 con 0x7f117c0ce5d0 2023-10-16T12:30:46.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.091+0000 7f117affd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f117c15ec80 0x7f117c0d19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:46.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.091+0000 7f117b7fe700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f117c0ce5d0 0x7f117c1e51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:46.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.091+0000 7f117affd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f117c15ec80 0x7f117c0d19d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46082/0 (socket says 172.21.15.16:46082) 2023-10-16T12:30:46.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.091+0000 7f117affd700 1 -- 172.21.15.16:0/2254998839 learned_addr learned my addr 172.21.15.16:0/2254998839 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:46.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.091+0000 7f117b7fe700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f117c0ce5d0 0x7f117c1e51f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:34338/0 (socket says 172.21.15.16:34338) 2023-10-16T12:30:46.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.091+0000 7f117bfff700 1 --2- 172.21.15.16:0/2254998839 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f117c15f060 0x7f117c0d1f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:46.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.092+0000 7f117affd700 1 -- 172.21.15.16:0/2254998839 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f117c0ce5d0 msgr2=0x7f117c1e51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:46.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.092+0000 7f117affd700 1 --2- 172.21.15.16:0/2254998839 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f117c0ce5d0 0x7f117c1e51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:46.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.092+0000 7f117affd700 1 -- 172.21.15.16:0/2254998839 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f117c15f060 msgr2=0x7f117c0d1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:46.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.092+0000 7f117affd700 1 --2- 172.21.15.16:0/2254998839 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f117c15f060 0x7f117c0d1f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:46.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.092+0000 7f117affd700 1 -- 172.21.15.16:0/2254998839 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11700094a0 con 0x7f117c15ec80 2023-10-16T12:30:46.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.092+0000 7f117bfff700 1 --2- 172.21.15.16:0/2254998839 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f117c15f060 0x7f117c0d1f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:30:46.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.092+0000 7f117affd700 1 --2- 172.21.15.16:0/2254998839 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f117c15ec80 0x7f117c0d19d0 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f1170009fd0 tx=0x7f1170004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:46.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.092+0000 7f1163fff700 1 -- 172.21.15.16:0/2254998839 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11700036a0 con 0x7f117c15ec80 2023-10-16T12:30:46.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.092+0000 7f1163fff700 1 -- 172.21.15.16:0/2254998839 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f117001f070 con 0x7f117c15ec80 2023-10-16T12:30:46.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.092+0000 7f1181ef9700 1 -- 172.21.15.16:0/2254998839 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f117c1e4ad0 con 0x7f117c15ec80 2023-10-16T12:30:46.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.092+0000 7f1163fff700 1 -- 172.21.15.16:0/2254998839 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1170003a60 con 0x7f117c15ec80 2023-10-16T12:30:46.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.092+0000 7f1181ef9700 1 -- 172.21.15.16:0/2254998839 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f117c1e41a0 con 0x7f117c15ec80 2023-10-16T12:30:46.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.093+0000 7f1181ef9700 1 -- 172.21.15.16:0/2254998839 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f117c0c69c0 con 0x7f117c15ec80 2023-10-16T12:30:46.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.094+0000 7f1163fff700 1 -- 172.21.15.16:0/2254998839 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 81) v1 ==== 89388+0+0 (secure 0 0 0) 0x7f1170003bc0 con 0x7f117c15ec80 2023-10-16T12:30:46.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.095+0000 7f1163fff700 1 --2- 172.21.15.16:0/2254998839 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f116806b0b0 0x7f116806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:46.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.095+0000 7f117b7fe700 1 -- 172.21.15.16:0/2254998839 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f116806b0b0 msgr2=0x7f116806d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3783161600 2023-10-16T12:30:46.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.095+0000 7f1163fff700 1 -- 172.21.15.16:0/2254998839 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6429+0+0 (secure 0 0 0) 0x7f1170021070 con 0x7f117c15ec80 2023-10-16T12:30:46.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.095+0000 7f117b7fe700 1 --2- 172.21.15.16:0/2254998839 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f116806b0b0 0x7f116806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:30:46.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.099+0000 7f1163fff700 1 -- 172.21.15.16:0/2254998839 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1170055ec0 con 0x7f117c15ec80 2023-10-16T12:30:46.214 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:30:46.213+0000 7f3b079a5200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T12:30:46.235 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:46.232+0000 7f6bd6932200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T12:30:46.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.296+0000 7f117b7fe700 1 -- 172.21.15.16:0/2254998839 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f116806b0b0 msgr2=0x7f116806d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3783161600 2023-10-16T12:30:46.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.296+0000 7f117b7fe700 1 --2- 172.21.15.16:0/2254998839 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f116806b0b0 0x7f116806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:30:46.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.301+0000 7f1181ef9700 1 -- 172.21.15.16:0/2254998839 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f117c04ebb0 con 0x7f117c15ec80 2023-10-16T12:30:46.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.301+0000 7f1163fff700 1 -- 172.21.15.16:0/2254998839 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v143) v1 ==== 93+0+10 (secure 0 0 0) 0x7f11700594e0 con 0x7f117c15ec80 2023-10-16T12:30:46.304 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T12:30:46.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.303+0000 7f1181ef9700 1 -- 172.21.15.16:0/2254998839 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f116806b0b0 msgr2=0x7f116806d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:30:46.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.303+0000 7f1181ef9700 1 --2- 172.21.15.16:0/2254998839 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f116806b0b0 0x7f116806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:46.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.303+0000 7f1181ef9700 1 -- 172.21.15.16:0/2254998839 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f117c15ec80 msgr2=0x7f117c0d19d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:46.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.303+0000 7f1181ef9700 1 --2- 172.21.15.16:0/2254998839 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f117c15ec80 0x7f117c0d19d0 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f1170009fd0 tx=0x7f1170004750 comp rx=0 tx=0).stop 2023-10-16T12:30:46.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.303+0000 7f1181ef9700 1 -- 172.21.15.16:0/2254998839 shutdown_connections 2023-10-16T12:30:46.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.303+0000 7f1181ef9700 1 --2- 172.21.15.16:0/2254998839 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f116806b0b0 0x7f116806d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:46.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.303+0000 7f1181ef9700 1 --2- 172.21.15.16:0/2254998839 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f117c15f060 0x7f117c0d1f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:46.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.303+0000 7f1181ef9700 1 --2- 172.21.15.16:0/2254998839 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f117c0ce5d0 0x7f117c1e51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:46.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.303+0000 7f1181ef9700 1 --2- 172.21.15.16:0/2254998839 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f117c15ec80 0x7f117c0d19d0 unknown :-1 s=CLOSED pgs=1633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:46.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.303+0000 7f1181ef9700 1 -- 172.21.15.16:0/2254998839 >> 172.21.15.16:0/2254998839 conn(0x7f117c0d15f0 msgr2=0x7f117c06f730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:46.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.303+0000 7f1181ef9700 1 -- 172.21.15.16:0/2254998839 shutdown_connections 2023-10-16T12:30:46.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.303+0000 7f1181ef9700 1 -- 172.21.15.16:0/2254998839 wait complete. 2023-10-16T12:30:46.320 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T12:30:46.334 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:30:46.333+0000 7f3b079a5200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T12:30:46.360 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:46.357+0000 7f6bd6932200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T12:30:46.414 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:30:46.413+0000 7f3b079a5200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T12:30:46.441 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:46.438+0000 7f6bd6932200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T12:30:46.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.450+0000 7f489f966700 1 Processor -- start 2023-10-16T12:30:46.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.451+0000 7f489f966700 1 -- start start 2023-10-16T12:30:46.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.452+0000 7f489f966700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f48980ce6a0 0x7f489806c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:46.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.452+0000 7f489f966700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f48980b2620 0x7f489806c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:46.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.452+0000 7f489f966700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48980b20f0 con 0x7f48980d2110 2023-10-16T12:30:46.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.452+0000 7f489f966700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48980af1f0 con 0x7f48980ce6a0 2023-10-16T12:30:46.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.452+0000 7f489f966700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48980d03d0 con 0x7f48980b2620 2023-10-16T12:30:46.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.452+0000 7f489d702700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f48980ce6a0 0x7f489806c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:46.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.452+0000 7f489d702700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f48980ce6a0 0x7f489806c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46090/0 (socket says 172.21.15.16:46090) 2023-10-16T12:30:46.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.452+0000 7f489d702700 1 -- 172.21.15.16:0/1393726523 learned_addr learned my addr 172.21.15.16:0/1393726523 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:46.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.452+0000 7f489df03700 1 --2- 172.21.15.16:0/1393726523 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f48980b2620 0x7f489806c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:46.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.452+0000 7f489d702700 1 -- 172.21.15.16:0/1393726523 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f48980b2620 msgr2=0x7f489806c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:46.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.452+0000 7f489d702700 1 --2- 172.21.15.16:0/1393726523 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f48980b2620 0x7f489806c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:46.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.452+0000 7f489d702700 1 -- 172.21.15.16:0/1393726523 >> v1:172.21.15.49:6789/0 conn(0x7f48980d2110 legacy=0x7f48980b0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:46.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.452+0000 7f488ffff700 1 -- 172.21.15.16:0/1393726523 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3722475782 0 0) 0x7f48980b20f0 con 0x7f48980d2110 2023-10-16T12:30:46.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.452+0000 7f489d702700 1 -- 172.21.15.16:0/1393726523 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48980b2340 con 0x7f48980ce6a0 2023-10-16T12:30:46.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.453+0000 7f489d702700 1 --2- 172.21.15.16:0/1393726523 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48980ce6a0 0x7f489806c300 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7f48940098f0 tx=0x7f4894009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9da81b8518afb25a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:46.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.453+0000 7f488ffff700 1 -- 172.21.15.16:0/1393726523 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4894011830 con 0x7f48980ce6a0 2023-10-16T12:30:46.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.453+0000 7f488ffff700 1 -- 172.21.15.16:0/1393726523 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4894011990 con 0x7f48980ce6a0 2023-10-16T12:30:46.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.453+0000 7f488ffff700 1 -- 172.21.15.16:0/1393726523 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4894011d20 con 0x7f48980ce6a0 2023-10-16T12:30:46.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.453+0000 7f489f966700 1 -- 172.21.15.16:0/1393726523 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48980ce6a0 msgr2=0x7f489806c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:46.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.453+0000 7f489f966700 1 --2- 172.21.15.16:0/1393726523 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48980ce6a0 0x7f489806c300 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7f48940098f0 tx=0x7f4894009c00 comp rx=0 tx=0).stop 2023-10-16T12:30:46.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.453+0000 7f489f966700 1 -- 172.21.15.16:0/1393726523 shutdown_connections 2023-10-16T12:30:46.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.453+0000 7f489f966700 1 --2- 172.21.15.16:0/1393726523 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f48980b2620 0x7f489806c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:46.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.453+0000 7f489f966700 1 --2- 172.21.15.16:0/1393726523 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48980ce6a0 0x7f489806c300 unknown :-1 s=CLOSED pgs=1634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:46.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.453+0000 7f489f966700 1 -- 172.21.15.16:0/1393726523 >> 172.21.15.16:0/1393726523 conn(0x7f48980d15f0 msgr2=0x7f48980d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:46.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.453+0000 7f489f966700 1 -- 172.21.15.16:0/1393726523 shutdown_connections 2023-10-16T12:30:46.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.453+0000 7f489f966700 1 -- 172.21.15.16:0/1393726523 wait complete. 2023-10-16T12:30:46.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.455+0000 7f489f966700 1 Processor -- start 2023-10-16T12:30:46.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.455+0000 7f489f966700 1 -- start start 2023-10-16T12:30:46.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.455+0000 7f489f966700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48980b2620 0x7f48981f70f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:46.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.455+0000 7f489f966700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f48980ce6a0 0x7f489814f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:46.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.455+0000 7f489f966700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f48980d2110 0x7f48981f3160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:46.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.455+0000 7f489f966700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48980c6410 con 0x7f48980b2620 2023-10-16T12:30:46.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.455+0000 7f489f966700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48980c7ad0 con 0x7f48980d2110 2023-10-16T12:30:46.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.455+0000 7f489f966700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48981f3ff0 con 0x7f48980ce6a0 2023-10-16T12:30:46.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.456+0000 7f489d702700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48980b2620 0x7f48981f70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:46.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.456+0000 7f489df03700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f48980d2110 0x7f48981f3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:46.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.456+0000 7f489d702700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48980b2620 0x7f48981f70f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46096/0 (socket says 172.21.15.16:46096) 2023-10-16T12:30:46.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.456+0000 7f489cf01700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f48980ce6a0 0x7f489814f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:46.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.456+0000 7f489d702700 1 -- 172.21.15.16:0/1519501593 learned_addr learned my addr 172.21.15.16:0/1519501593 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:46.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.456+0000 7f489d702700 1 -- 172.21.15.16:0/1519501593 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f48980ce6a0 msgr2=0x7f489814f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:46.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.456+0000 7f489d702700 1 --2- 172.21.15.16:0/1519501593 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f48980ce6a0 0x7f489814f3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:46.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.456+0000 7f489d702700 1 -- 172.21.15.16:0/1519501593 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f48980d2110 msgr2=0x7f48981f3160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:46.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.456+0000 7f489d702700 1 --2- 172.21.15.16:0/1519501593 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f48980d2110 0x7f48981f3160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:46.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.456+0000 7f489d702700 1 -- 172.21.15.16:0/1519501593 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4894009550 con 0x7f48980b2620 2023-10-16T12:30:46.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.456+0000 7f489d702700 1 --2- 172.21.15.16:0/1519501593 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48980b2620 0x7f48981f70f0 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7f4894009c40 tx=0x7f4894004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:46.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.456+0000 7f488dffb700 1 -- 172.21.15.16:0/1519501593 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4894004550 con 0x7f48980b2620 2023-10-16T12:30:46.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.456+0000 7f488dffb700 1 -- 172.21.15.16:0/1519501593 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f489401f070 con 0x7f48980b2620 2023-10-16T12:30:46.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.456+0000 7f488dffb700 1 -- 172.21.15.16:0/1519501593 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48940038e0 con 0x7f48980b2620 2023-10-16T12:30:46.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.456+0000 7f489f966700 1 -- 172.21.15.16:0/1519501593 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48980c6ec0 con 0x7f48980b2620 2023-10-16T12:30:46.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.457+0000 7f489f966700 1 -- 172.21.15.16:0/1519501593 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48980c7480 con 0x7f48980b2620 2023-10-16T12:30:46.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.457+0000 7f489f966700 1 -- 172.21.15.16:0/1519501593 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48981f7630 con 0x7f48980b2620 2023-10-16T12:30:46.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.458+0000 7f488dffb700 1 -- 172.21.15.16:0/1519501593 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 81) v1 ==== 89388+0+0 (secure 0 0 0) 0x7f4894003a40 con 0x7f48980b2620 2023-10-16T12:30:46.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.458+0000 7f488dffb700 1 --2- 172.21.15.16:0/1519501593 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f487c06b0b0 0x7f487c06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:46.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.458+0000 7f489cf01700 1 -- 172.21.15.16:0/1519501593 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f487c06b0b0 msgr2=0x7f487c06d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3783161600 2023-10-16T12:30:46.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.458+0000 7f488dffb700 1 -- 172.21.15.16:0/1519501593 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6429+0+0 (secure 0 0 0) 0x7f4894021070 con 0x7f48980b2620 2023-10-16T12:30:46.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.458+0000 7f489cf01700 1 --2- 172.21.15.16:0/1519501593 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f487c06b0b0 0x7f487c06d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:30:46.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.463+0000 7f488dffb700 1 -- 172.21.15.16:0/1519501593 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4894055ef0 con 0x7f48980b2620 2023-10-16T12:30:46.499 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:30:46.498+0000 7f3b079a5200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T12:30:46.528 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:46.526+0000 7f6bd6932200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T12:30:46.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.659+0000 7f489cf01700 1 -- 172.21.15.16:0/1519501593 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f487c06b0b0 msgr2=0x7f487c06d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3783161600 2023-10-16T12:30:46.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.659+0000 7f489cf01700 1 --2- 172.21.15.16:0/1519501593 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f487c06b0b0 0x7f487c06d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:30:46.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.669+0000 7f489f966700 1 -- 172.21.15.16:0/1519501593 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f489804ebb0 con 0x7f48980b2620 2023-10-16T12:30:46.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.669+0000 7f488dffb700 1 -- 172.21.15.16:0/1519501593 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v143) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4894015020 con 0x7f48980b2620 2023-10-16T12:30:46.672 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-10-16T12:30:46.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.672+0000 7f489f966700 1 -- 172.21.15.16:0/1519501593 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f487c06b0b0 msgr2=0x7f487c06d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:30:46.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.672+0000 7f489f966700 1 --2- 172.21.15.16:0/1519501593 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f487c06b0b0 0x7f487c06d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:46.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.672+0000 7f489f966700 1 -- 172.21.15.16:0/1519501593 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48980b2620 msgr2=0x7f48981f70f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:46.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.672+0000 7f489f966700 1 --2- 172.21.15.16:0/1519501593 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48980b2620 0x7f48981f70f0 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7f4894009c40 tx=0x7f4894004750 comp rx=0 tx=0).stop 2023-10-16T12:30:46.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.672+0000 7f489f966700 1 -- 172.21.15.16:0/1519501593 shutdown_connections 2023-10-16T12:30:46.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.672+0000 7f489f966700 1 --2- 172.21.15.16:0/1519501593 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f487c06b0b0 0x7f487c06d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:46.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.672+0000 7f489f966700 1 --2- 172.21.15.16:0/1519501593 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f48980d2110 0x7f48981f3160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:46.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.672+0000 7f489f966700 1 --2- 172.21.15.16:0/1519501593 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48980b2620 0x7f48981f70f0 unknown :-1 s=CLOSED pgs=1635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:46.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.672+0000 7f489f966700 1 --2- 172.21.15.16:0/1519501593 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f48980ce6a0 0x7f489814f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:46.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.672+0000 7f489f966700 1 -- 172.21.15.16:0/1519501593 >> 172.21.15.16:0/1519501593 conn(0x7f48980d15f0 msgr2=0x7f48980b0f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:46.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.672+0000 7f489f966700 1 -- 172.21.15.16:0/1519501593 shutdown_connections 2023-10-16T12:30:46.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.672+0000 7f489f966700 1 -- 172.21.15.16:0/1519501593 wait complete. 2023-10-16T12:30:46.687 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T12:30:46.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.857+0000 7fa3ae41f700 1 Processor -- start 2023-10-16T12:30:46.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.857+0000 7fa3ae41f700 1 -- start start 2023-10-16T12:30:46.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.858+0000 7fa3ae41f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa3a80ce6a0 0x7fa3a806c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:46.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.858+0000 7fa3ae41f700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fa3a80d2110 0x7fa3a806c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:46.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.858+0000 7fa3ae41f700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3a80b20f0 con 0x7fa3a80b2620 2023-10-16T12:30:46.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.858+0000 7fa3ae41f700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3a80af1f0 con 0x7fa3a80ce6a0 2023-10-16T12:30:46.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.858+0000 7fa3ae41f700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3a80d03d0 con 0x7fa3a80d2110 2023-10-16T12:30:46.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.858+0000 7fa3a7fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa3a80ce6a0 0x7fa3a806c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:46.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.858+0000 7fa3ac9bc700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa3a80b2620 0x7fa3a80b0f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35428/0 (socket says 172.21.15.16:35428) 2023-10-16T12:30:46.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.858+0000 7fa3a7fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa3a80ce6a0 0x7fa3a806c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46150/0 (socket says 172.21.15.16:46150) 2023-10-16T12:30:46.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.858+0000 7fa3ac9bc700 1 -- 172.21.15.16:0/3652808797 learned_addr learned my addr 172.21.15.16:0/3652808797 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:30:46.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.858+0000 7fa3a77fe700 1 --2- 172.21.15.16:0/3652808797 >> v2:172.21.15.49:3300/0 conn(0x7fa3a80d2110 0x7fa3a806c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:46.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.859+0000 7fa3a7fff700 1 -- 172.21.15.16:0/3652808797 >> v2:172.21.15.49:3300/0 conn(0x7fa3a80d2110 msgr2=0x7fa3a806c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:46.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.859+0000 7fa3a7fff700 1 --2- 172.21.15.16:0/3652808797 >> v2:172.21.15.49:3300/0 conn(0x7fa3a80d2110 0x7fa3a806c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:46.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.859+0000 7fa3a7fff700 1 -- 172.21.15.16:0/3652808797 >> v1:172.21.15.16:6789/0 conn(0x7fa3a80b2620 legacy=0x7fa3a80b0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:46.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.859+0000 7fa3a6ffd700 1 -- 172.21.15.16:0/3652808797 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 123152308 0 0) 0x7fa3a80b20f0 con 0x7fa3a80b2620 2023-10-16T12:30:46.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.859+0000 7fa3a7fff700 1 -- 172.21.15.16:0/3652808797 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3a80b2340 con 0x7fa3a80ce6a0 2023-10-16T12:30:46.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.859+0000 7fa3a7fff700 1 --2- 172.21.15.16:0/3652808797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3a80ce6a0 0x7fa3a806c300 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7fa3a80cf0f0 tx=0x7fa398009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd2fbe9dbe5b14fc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:46.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.859+0000 7fa3a6ffd700 1 -- 172.21.15.16:0/3652808797 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa398011830 con 0x7fa3a80ce6a0 2023-10-16T12:30:46.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.859+0000 7fa3a6ffd700 1 -- 172.21.15.16:0/3652808797 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa398011990 con 0x7fa3a80ce6a0 2023-10-16T12:30:46.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.859+0000 7fa3a6ffd700 1 -- 172.21.15.16:0/3652808797 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa398011d20 con 0x7fa3a80ce6a0 2023-10-16T12:30:46.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.860+0000 7fa3ae41f700 1 -- 172.21.15.16:0/3652808797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3a80ce6a0 msgr2=0x7fa3a806c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:46.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.860+0000 7fa3ae41f700 1 --2- 172.21.15.16:0/3652808797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3a80ce6a0 0x7fa3a806c300 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7fa3a80cf0f0 tx=0x7fa398009b20 comp rx=0 tx=0).stop 2023-10-16T12:30:46.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.860+0000 7fa3ae41f700 1 -- 172.21.15.16:0/3652808797 shutdown_connections 2023-10-16T12:30:46.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.860+0000 7fa3ae41f700 1 --2- 172.21.15.16:0/3652808797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3a80ce6a0 0x7fa3a806c300 unknown :-1 s=CLOSED pgs=1637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:46.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.860+0000 7fa3ae41f700 1 --2- 172.21.15.16:0/3652808797 >> v2:172.21.15.49:3300/0 conn(0x7fa3a80d2110 0x7fa3a806c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:46.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.860+0000 7fa3ae41f700 1 -- 172.21.15.16:0/3652808797 >> 172.21.15.16:0/3652808797 conn(0x7fa3a80d15f0 msgr2=0x7fa3a80d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:46.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.860+0000 7fa3ae41f700 1 -- 172.21.15.16:0/3652808797 shutdown_connections 2023-10-16T12:30:46.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.860+0000 7fa3ae41f700 1 -- 172.21.15.16:0/3652808797 wait complete. 2023-10-16T12:30:46.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.861+0000 7fa3ae41f700 1 Processor -- start 2023-10-16T12:30:46.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.861+0000 7fa3ae41f700 1 -- start start 2023-10-16T12:30:46.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.862+0000 7fa3ae41f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa3a80b2620 0x7fa3a80c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:46.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.862+0000 7fa3ae41f700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa3a80ce6a0 0x7fa3a806ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:46.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.862+0000 7fa3ae41f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3a80d2110 0x7fa3a80ca760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:46.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.862+0000 7fa3ae41f700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3a80c61e0 con 0x7fa3a80d2110 2023-10-16T12:30:46.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.862+0000 7fa3ae41f700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3a80cb140 con 0x7fa3a80ce6a0 2023-10-16T12:30:46.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.862+0000 7fa3ae41f700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3a80c51c0 con 0x7fa3a80b2620 2023-10-16T12:30:46.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.862+0000 7fa3ac9bc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3a80d2110 0x7fa3a80ca760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:46.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.862+0000 7fa3ac9bc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3a80d2110 0x7fa3a80ca760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46160/0 (socket says 172.21.15.16:46160) 2023-10-16T12:30:46.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.862+0000 7fa3ac9bc700 1 -- 172.21.15.16:0/1947279886 learned_addr learned my addr 172.21.15.16:0/1947279886 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:46.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.862+0000 7fa3a77fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa3a80ce6a0 0x7fa3a806ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:46.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.862+0000 7fa3a7fff700 1 --2- 172.21.15.16:0/1947279886 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa3a80b2620 0x7fa3a80c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:46.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.862+0000 7fa3ac9bc700 1 -- 172.21.15.16:0/1947279886 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa3a80b2620 msgr2=0x7fa3a80c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:46.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.862+0000 7fa3ac9bc700 1 --2- 172.21.15.16:0/1947279886 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa3a80b2620 0x7fa3a80c5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:46.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.862+0000 7fa3ac9bc700 1 -- 172.21.15.16:0/1947279886 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa3a80ce6a0 msgr2=0x7fa3a806ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:46.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.862+0000 7fa3ac9bc700 1 --2- 172.21.15.16:0/1947279886 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa3a80ce6a0 0x7fa3a806ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:46.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.862+0000 7fa3ac9bc700 1 -- 172.21.15.16:0/1947279886 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3980094a0 con 0x7fa3a80d2110 2023-10-16T12:30:46.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.863+0000 7fa3ac9bc700 1 --2- 172.21.15.16:0/1947279886 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3a80d2110 0x7fa3a80ca760 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7fa3a0002230 tx=0x7fa3a000cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:46.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.863+0000 7fa3a4ff9700 1 -- 172.21.15.16:0/1947279886 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a000aac0 con 0x7fa3a80d2110 2023-10-16T12:30:46.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.863+0000 7fa3a4ff9700 1 -- 172.21.15.16:0/1947279886 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3a000ac40 con 0x7fa3a80d2110 2023-10-16T12:30:46.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.863+0000 7fa3ae41f700 1 -- 172.21.15.16:0/1947279886 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3a80c54a0 con 0x7fa3a80d2110 2023-10-16T12:30:46.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.863+0000 7fa3a4ff9700 1 -- 172.21.15.16:0/1947279886 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a00070d0 con 0x7fa3a80d2110 2023-10-16T12:30:46.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.863+0000 7fa3ae41f700 1 -- 172.21.15.16:0/1947279886 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3a80cb420 con 0x7fa3a80d2110 2023-10-16T12:30:46.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.864+0000 7fa3ae41f700 1 -- 172.21.15.16:0/1947279886 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3a80c59c0 con 0x7fa3a80d2110 2023-10-16T12:30:46.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.865+0000 7fa3a4ff9700 1 -- 172.21.15.16:0/1947279886 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 81) v1 ==== 89388+0+0 (secure 0 0 0) 0x7fa3a0007250 con 0x7fa3a80d2110 2023-10-16T12:30:46.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.866+0000 7fa3a4ff9700 1 --2- 172.21.15.16:0/1947279886 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7fa39406b0b0 0x7fa39406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:46.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.866+0000 7fa3a7fff700 1 -- 172.21.15.16:0/1947279886 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7fa39406b0b0 msgr2=0x7fa39406d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3783161600 2023-10-16T12:30:46.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.866+0000 7fa3a4ff9700 1 -- 172.21.15.16:0/1947279886 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6429+0+0 (secure 0 0 0) 0x7fa3a0089ab0 con 0x7fa3a80d2110 2023-10-16T12:30:46.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.866+0000 7fa3a7fff700 1 --2- 172.21.15.16:0/1947279886 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7fa39406b0b0 0x7fa39406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:30:46.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:46.870+0000 7fa3a4ff9700 1 -- 172.21.15.16:0/1947279886 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa3a0054680 con 0x7fa3a80d2110 2023-10-16T12:30:46.890 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:30:46.889+0000 7f3b079a5200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T12:30:46.927 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:46.925+0000 7f6bd6932200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T12:30:47.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.067+0000 7fa3a7fff700 1 -- 172.21.15.16:0/1947279886 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7fa39406b0b0 msgr2=0x7fa39406d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3783161600 2023-10-16T12:30:47.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.067+0000 7fa3a7fff700 1 --2- 172.21.15.16:0/1947279886 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7fa39406b0b0 0x7fa39406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:30:47.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.076+0000 7fa3ae41f700 1 -- 172.21.15.16:0/1947279886 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa3a80c59c0 con 0x7fa3a80d2110 2023-10-16T12:30:47.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.077+0000 7fa3a4ff9700 1 -- 172.21.15.16:0/1947279886 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v143) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa3a80c59c0 con 0x7fa3a80d2110 2023-10-16T12:30:47.080 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-10-16T12:30:47.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.079+0000 7fa3ae41f700 1 -- 172.21.15.16:0/1947279886 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7fa39406b0b0 msgr2=0x7fa39406d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:30:47.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.079+0000 7fa3ae41f700 1 --2- 172.21.15.16:0/1947279886 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7fa39406b0b0 0x7fa39406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:47.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.079+0000 7fa3ae41f700 1 -- 172.21.15.16:0/1947279886 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3a80d2110 msgr2=0x7fa3a80ca760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:47.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.079+0000 7fa3ae41f700 1 --2- 172.21.15.16:0/1947279886 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3a80d2110 0x7fa3a80ca760 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7fa3a0002230 tx=0x7fa3a000cf50 comp rx=0 tx=0).stop 2023-10-16T12:30:47.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.079+0000 7fa3ae41f700 1 -- 172.21.15.16:0/1947279886 shutdown_connections 2023-10-16T12:30:47.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.079+0000 7fa3ae41f700 1 --2- 172.21.15.16:0/1947279886 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7fa39406b0b0 0x7fa39406d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:47.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.079+0000 7fa3ae41f700 1 --2- 172.21.15.16:0/1947279886 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3a80d2110 0x7fa3a80ca760 unknown :-1 s=CLOSED pgs=1638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:47.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.079+0000 7fa3ae41f700 1 --2- 172.21.15.16:0/1947279886 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa3a80b2620 0x7fa3a80c5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:47.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.079+0000 7fa3ae41f700 1 --2- 172.21.15.16:0/1947279886 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa3a80ce6a0 0x7fa3a806ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:47.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.079+0000 7fa3ae41f700 1 -- 172.21.15.16:0/1947279886 >> 172.21.15.16:0/1947279886 conn(0x7fa3a80d15f0 msgr2=0x7fa3a80af330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:47.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.079+0000 7fa3ae41f700 1 -- 172.21.15.16:0/1947279886 shutdown_connections 2023-10-16T12:30:47.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.079+0000 7fa3ae41f700 1 -- 172.21.15.16:0/1947279886 wait complete. 2023-10-16T12:30:47.094 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T12:30:47.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.218+0000 7fba43712700 1 Processor -- start 2023-10-16T12:30:47.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.219+0000 7fba43712700 1 -- start start 2023-10-16T12:30:47.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.219+0000 7fba43712700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fba3c0ce4f0 0x7fba3c0d0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:47.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.219+0000 7fba43712700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fba3c15ba10 0x7fba3c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:47.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.219+0000 7fba43712700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba3c0b0970 con 0x7fba3c15b630 2023-10-16T12:30:47.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.220+0000 7fba43712700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba3c0cd690 con 0x7fba3c15ba10 2023-10-16T12:30:47.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.220+0000 7fba43712700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba3c0cd970 con 0x7fba3c0ce4f0 2023-10-16T12:30:47.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.220+0000 7fba414ae700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fba3c15b630 0x7fba3c0cdde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35440/0 (socket says 172.21.15.16:35440) 2023-10-16T12:30:47.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.220+0000 7fba41caf700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fba3c15ba10 0x7fba3c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:47.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.220+0000 7fba414ae700 1 -- 172.21.15.16:0/541828517 learned_addr learned my addr 172.21.15.16:0/541828517 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:30:47.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.220+0000 7fba40cad700 1 --2- 172.21.15.16:0/541828517 >> v2:172.21.15.49:3300/0 conn(0x7fba3c0ce4f0 0x7fba3c0d0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:47.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.220+0000 7fba33fff700 1 -- 172.21.15.16:0/541828517 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2011745934 0 0) 0x7fba3c0b0970 con 0x7fba3c15b630 2023-10-16T12:30:47.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.220+0000 7fba33fff700 1 -- 172.21.15.16:0/541828517 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba280035b0 con 0x7fba3c15b630 2023-10-16T12:30:47.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.220+0000 7fba41caf700 1 -- 172.21.15.16:0/541828517 >> v2:172.21.15.49:3300/0 conn(0x7fba3c0ce4f0 msgr2=0x7fba3c0d0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:47.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.220+0000 7fba41caf700 1 --2- 172.21.15.16:0/541828517 >> v2:172.21.15.49:3300/0 conn(0x7fba3c0ce4f0 0x7fba3c0d0380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:47.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.220+0000 7fba41caf700 1 -- 172.21.15.16:0/541828517 >> v1:172.21.15.16:6789/0 conn(0x7fba3c15b630 legacy=0x7fba3c0cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:47.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.220+0000 7fba41caf700 1 -- 172.21.15.16:0/541828517 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba3c0b0c30 con 0x7fba3c15ba10 2023-10-16T12:30:47.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.220+0000 7fba41caf700 1 --2- 172.21.15.16:0/541828517 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba3c15ba10 0x7fba3c0cf290 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7fba3c0d2460 tx=0x7fba3400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92db634237381ae8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:47.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.220+0000 7fba33fff700 1 -- 172.21.15.16:0/541828517 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba3400e070 con 0x7fba3c15ba10 2023-10-16T12:30:47.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.220+0000 7fba33fff700 1 -- 172.21.15.16:0/541828517 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba3400bb20 con 0x7fba3c15ba10 2023-10-16T12:30:47.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.221+0000 7fba33fff700 1 -- 172.21.15.16:0/541828517 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba34012430 con 0x7fba3c15ba10 2023-10-16T12:30:47.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.221+0000 7fba43712700 1 -- 172.21.15.16:0/541828517 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba3c15ba10 msgr2=0x7fba3c0cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:47.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.221+0000 7fba43712700 1 --2- 172.21.15.16:0/541828517 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba3c15ba10 0x7fba3c0cf290 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7fba3c0d2460 tx=0x7fba3400b330 comp rx=0 tx=0).stop 2023-10-16T12:30:47.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.221+0000 7fba43712700 1 -- 172.21.15.16:0/541828517 shutdown_connections 2023-10-16T12:30:47.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.221+0000 7fba43712700 1 --2- 172.21.15.16:0/541828517 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba3c15ba10 0x7fba3c0cf290 unknown :-1 s=CLOSED pgs=1640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:47.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.221+0000 7fba43712700 1 --2- 172.21.15.16:0/541828517 >> v2:172.21.15.49:3300/0 conn(0x7fba3c0ce4f0 0x7fba3c0d0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:47.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.221+0000 7fba43712700 1 -- 172.21.15.16:0/541828517 >> 172.21.15.16:0/541828517 conn(0x7fba3c0d14b0 msgr2=0x7fba3c0d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:47.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.221+0000 7fba43712700 1 -- 172.21.15.16:0/541828517 shutdown_connections 2023-10-16T12:30:47.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.221+0000 7fba43712700 1 -- 172.21.15.16:0/541828517 wait complete. 2023-10-16T12:30:47.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.222+0000 7fba43712700 1 Processor -- start 2023-10-16T12:30:47.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.223+0000 7fba43712700 1 -- start start 2023-10-16T12:30:47.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.223+0000 7fba43712700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fba3c0ce4f0 0x7fba3c0b4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:47.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.223+0000 7fba43712700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fba3c15b630 0x7fba3c0b4840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:47.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.223+0000 7fba43712700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba3c15ba10 0x7fba3c0b61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:47.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.223+0000 7fba43712700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba3c15c4b0 con 0x7fba3c15ba10 2023-10-16T12:30:47.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.223+0000 7fba43712700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba3c15c790 con 0x7fba3c0ce4f0 2023-10-16T12:30:47.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.223+0000 7fba43712700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba3c15ca70 con 0x7fba3c15b630 2023-10-16T12:30:47.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.223+0000 7fba41caf700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba3c15ba10 0x7fba3c0b61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:47.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.223+0000 7fba41caf700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba3c15ba10 0x7fba3c0b61d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46176/0 (socket says 172.21.15.16:46176) 2023-10-16T12:30:47.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.223+0000 7fba414ae700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fba3c0ce4f0 0x7fba3c0b4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:47.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.223+0000 7fba41caf700 1 -- 172.21.15.16:0/4070396504 learned_addr learned my addr 172.21.15.16:0/4070396504 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:47.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.223+0000 7fba40cad700 1 --2- 172.21.15.16:0/4070396504 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fba3c15b630 0x7fba3c0b4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:47.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.224+0000 7fba41caf700 1 -- 172.21.15.16:0/4070396504 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fba3c15b630 msgr2=0x7fba3c0b4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:47.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.224+0000 7fba41caf700 1 --2- 172.21.15.16:0/4070396504 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fba3c15b630 0x7fba3c0b4840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:47.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.224+0000 7fba41caf700 1 -- 172.21.15.16:0/4070396504 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fba3c0ce4f0 msgr2=0x7fba3c0b4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:47.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.224+0000 7fba41caf700 1 --2- 172.21.15.16:0/4070396504 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fba3c0ce4f0 0x7fba3c0b4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:47.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.224+0000 7fba41caf700 1 -- 172.21.15.16:0/4070396504 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba34009a70 con 0x7fba3c15ba10 2023-10-16T12:30:47.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.224+0000 7fba41caf700 1 --2- 172.21.15.16:0/4070396504 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba3c15ba10 0x7fba3c0b61d0 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7fba34014040 tx=0x7fba34004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:47.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.224+0000 7fba31ffb700 1 -- 172.21.15.16:0/4070396504 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba34012430 con 0x7fba3c15ba10 2023-10-16T12:30:47.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.224+0000 7fba31ffb700 1 -- 172.21.15.16:0/4070396504 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba34020070 con 0x7fba3c15ba10 2023-10-16T12:30:47.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.224+0000 7fba43712700 1 -- 172.21.15.16:0/4070396504 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba3c15cd50 con 0x7fba3c15ba10 2023-10-16T12:30:47.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.224+0000 7fba31ffb700 1 -- 172.21.15.16:0/4070396504 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba340098f0 con 0x7fba3c15ba10 2023-10-16T12:30:47.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.224+0000 7fba43712700 1 -- 172.21.15.16:0/4070396504 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba3c0b6710 con 0x7fba3c15ba10 2023-10-16T12:30:47.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.226+0000 7fba43712700 1 -- 172.21.15.16:0/4070396504 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba3c15d270 con 0x7fba3c15ba10 2023-10-16T12:30:47.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.226+0000 7fba31ffb700 1 -- 172.21.15.16:0/4070396504 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 81) v1 ==== 89388+0+0 (secure 0 0 0) 0x7fba34004040 con 0x7fba3c15ba10 2023-10-16T12:30:47.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.226+0000 7fba31ffb700 1 --2- 172.21.15.16:0/4070396504 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7fba2006b0b0 0x7fba2006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:47.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.226+0000 7fba414ae700 1 -- 172.21.15.16:0/4070396504 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7fba2006b0b0 msgr2=0x7fba2006d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3783161600 2023-10-16T12:30:47.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.226+0000 7fba31ffb700 1 -- 172.21.15.16:0/4070396504 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6429+0+0 (secure 0 0 0) 0x7fba34022070 con 0x7fba3c15ba10 2023-10-16T12:30:47.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.226+0000 7fba414ae700 1 --2- 172.21.15.16:0/4070396504 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7fba2006b0b0 0x7fba2006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:30:47.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.231+0000 7fba31ffb700 1 -- 172.21.15.16:0/4070396504 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fba3c15d270 con 0x7fba3c15ba10 2023-10-16T12:30:47.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.427+0000 7fba414ae700 1 -- 172.21.15.16:0/4070396504 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7fba2006b0b0 msgr2=0x7fba2006d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3783161600 2023-10-16T12:30:47.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.427+0000 7fba414ae700 1 --2- 172.21.15.16:0/4070396504 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7fba2006b0b0 0x7fba2006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:30:47.503 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:30:47.502+0000 7f3b079a5200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T12:30:47.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.525+0000 7fba43712700 1 -- 172.21.15.16:0/4070396504 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fba3c15d270 con 0x7fba3c15ba10 2023-10-16T12:30:47.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.526+0000 7fba31ffb700 1 -- 172.21.15.16:0/4070396504 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v143) v1 ==== 80+0+468 (secure 0 0 0) 0x7fba3c15d270 con 0x7fba3c15ba10 2023-10-16T12:30:47.529 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2901217010 2023-10-17T12:30:37.941028+0000 2023-10-16T12:30:47.529 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6833/258291157 2023-10-17T12:30:37.941028+0000 2023-10-16T12:30:47.530 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6832/258291157 2023-10-17T12:30:37.941028+0000 2023-10-16T12:30:47.530 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/4123017371 2023-10-17T12:30:37.941028+0000 2023-10-16T12:30:47.530 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2703050386 2023-10-17T12:30:37.941028+0000 2023-10-16T12:30:47.530 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3188488837 2023-10-17T12:30:37.941028+0000 2023-10-16T12:30:47.531 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2621298501 2023-10-17T12:30:37.941028+0000 2023-10-16T12:30:47.531 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/1781882931 2023-10-17T12:30:37.941028+0000 2023-10-16T12:30:47.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.528+0000 7fba43712700 1 -- 172.21.15.16:0/4070396504 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7fba2006b0b0 msgr2=0x7fba2006d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:30:47.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.528+0000 7fba43712700 1 --2- 172.21.15.16:0/4070396504 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7fba2006b0b0 0x7fba2006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:47.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.528+0000 7fba43712700 1 -- 172.21.15.16:0/4070396504 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba3c15ba10 msgr2=0x7fba3c0b61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:47.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.528+0000 7fba43712700 1 --2- 172.21.15.16:0/4070396504 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba3c15ba10 0x7fba3c0b61d0 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7fba34014040 tx=0x7fba34004ad0 comp rx=0 tx=0).stop 2023-10-16T12:30:47.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.528+0000 7fba43712700 1 -- 172.21.15.16:0/4070396504 shutdown_connections 2023-10-16T12:30:47.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.528+0000 7fba43712700 1 --2- 172.21.15.16:0/4070396504 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7fba2006b0b0 0x7fba2006d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:47.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.528+0000 7fba43712700 1 --2- 172.21.15.16:0/4070396504 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba3c15ba10 0x7fba3c0b61d0 unknown :-1 s=CLOSED pgs=1641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:47.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.528+0000 7fba43712700 1 --2- 172.21.15.16:0/4070396504 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fba3c0ce4f0 0x7fba3c0b4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:47.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.528+0000 7fba43712700 1 --2- 172.21.15.16:0/4070396504 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fba3c15b630 0x7fba3c0b4840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:47.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.528+0000 7fba43712700 1 -- 172.21.15.16:0/4070396504 >> 172.21.15.16:0/4070396504 conn(0x7fba3c0d14b0 msgr2=0x7fba3c15a9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:47.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.528+0000 7fba43712700 1 -- 172.21.15.16:0/4070396504 shutdown_connections 2023-10-16T12:30:47.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.528+0000 7fba43712700 1 -- 172.21.15.16:0/4070396504 wait complete. 2023-10-16T12:30:47.535 INFO:teuthology.orchestra.run.smithi016.stderr:listed 8 entries 2023-10-16T12:30:47.545 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-16T12:30:47.545 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:30:47.546 DEBUG:teuthology.orchestra.run.smithi016:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-16T12:30:47.577 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:30:47.576+0000 7f3b079a5200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T12:30:47.591 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:47.588+0000 7f6bd6932200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T12:30:47.601 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2023-10-16T12:30:47.601 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:30:47.602 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-16T12:30:47.620 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T12:30:47.621 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:30:47.639 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284204k,nr_inodes=4071051,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=753 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260540k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T12:30:47.639 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:30:47.639 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:30:47.669 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:47.666+0000 7f6bd6932200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T12:30:47.678 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16287364k,nr_inodes=4071841,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20251 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3261172k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T12:30:47.679 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T12:30:47.679 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:30:47.697 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284204k,nr_inodes=4071051,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=753 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260540k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T12:30:47.698 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:30:47.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.843+0000 7fd6ac8a3700 1 Processor -- start 2023-10-16T12:30:47.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.843+0000 7fd6ac8a3700 1 -- start start 2023-10-16T12:30:47.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.844+0000 7fd6ac8a3700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fd6a815d900 0x7fd6a80d2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:47.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.844+0000 7fd6ac8a3700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd6a80d2f70 0x7fd6a80cdc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:47.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.844+0000 7fd6ac8a3700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6a80ce1a0 con 0x7fd6a80ce9f0 2023-10-16T12:30:47.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.844+0000 7fd6ac8a3700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6a80cee70 con 0x7fd6a80d2f70 2023-10-16T12:30:47.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.844+0000 7fd6ac8a3700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6a80ce400 con 0x7fd6a815d900 2023-10-16T12:30:47.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.844+0000 7fd6a6d9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd6a80d2f70 0x7fd6a80cdc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:47.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.844+0000 7fd6a6d9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd6a80d2f70 0x7fd6a80cdc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46186/0 (socket says 172.21.15.16:46186) 2023-10-16T12:30:47.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.844+0000 7fd6a6d9d700 1 -- 172.21.15.16:0/225885387 learned_addr learned my addr 172.21.15.16:0/225885387 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:47.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.844+0000 7fd6a659c700 1 --2- 172.21.15.16:0/225885387 >> v2:172.21.15.49:3300/0 conn(0x7fd6a815d900 0x7fd6a80d2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:47.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.845+0000 7fd6a6d9d700 1 -- 172.21.15.16:0/225885387 >> v2:172.21.15.49:3300/0 conn(0x7fd6a815d900 msgr2=0x7fd6a80d2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:47.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.845+0000 7fd6a6d9d700 1 --2- 172.21.15.16:0/225885387 >> v2:172.21.15.49:3300/0 conn(0x7fd6a815d900 0x7fd6a80d2040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:47.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.845+0000 7fd6a6d9d700 1 -- 172.21.15.16:0/225885387 >> v1:172.21.15.49:6789/0 conn(0x7fd6a80ce9f0 legacy=0x7fd6a80d2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:47.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.845+0000 7fd6a559a700 1 -- 172.21.15.16:0/225885387 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3823922191 0 0) 0x7fd6a80ce1a0 con 0x7fd6a80ce9f0 2023-10-16T12:30:47.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.845+0000 7fd6a6d9d700 1 -- 172.21.15.16:0/225885387 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6a815dce0 con 0x7fd6a80d2f70 2023-10-16T12:30:47.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.845+0000 7fd6a6d9d700 1 --2- 172.21.15.16:0/225885387 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6a80d2f70 0x7fd6a80cdc60 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7fd6a80d4710 tx=0x7fd6a000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a02d18cd60ccaa0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:47.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.845+0000 7fd6a559a700 1 -- 172.21.15.16:0/225885387 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a000e070 con 0x7fd6a80d2f70 2023-10-16T12:30:47.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.845+0000 7fd6a559a700 1 -- 172.21.15.16:0/225885387 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6a000b910 con 0x7fd6a80d2f70 2023-10-16T12:30:47.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.845+0000 7fd6a559a700 1 -- 172.21.15.16:0/225885387 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a000bcc0 con 0x7fd6a80d2f70 2023-10-16T12:30:47.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.846+0000 7fd6ac8a3700 1 -- 172.21.15.16:0/225885387 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6a80d2f70 msgr2=0x7fd6a80cdc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:47.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.846+0000 7fd6ac8a3700 1 --2- 172.21.15.16:0/225885387 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6a80d2f70 0x7fd6a80cdc60 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7fd6a80d4710 tx=0x7fd6a000b330 comp rx=0 tx=0).stop 2023-10-16T12:30:47.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.846+0000 7fd6ac8a3700 1 -- 172.21.15.16:0/225885387 shutdown_connections 2023-10-16T12:30:47.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.846+0000 7fd6ac8a3700 1 --2- 172.21.15.16:0/225885387 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6a80d2f70 0x7fd6a80cdc60 unknown :-1 s=CLOSED pgs=1642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:47.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.846+0000 7fd6ac8a3700 1 --2- 172.21.15.16:0/225885387 >> v2:172.21.15.49:3300/0 conn(0x7fd6a815d900 0x7fd6a80d2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:47.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.846+0000 7fd6ac8a3700 1 -- 172.21.15.16:0/225885387 >> 172.21.15.16:0/225885387 conn(0x7fd6a8072330 msgr2=0x7fd6a8072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:47.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.846+0000 7fd6ac8a3700 1 -- 172.21.15.16:0/225885387 shutdown_connections 2023-10-16T12:30:47.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.846+0000 7fd6ac8a3700 1 -- 172.21.15.16:0/225885387 wait complete. 2023-10-16T12:30:47.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.847+0000 7fd6ac8a3700 1 Processor -- start 2023-10-16T12:30:47.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.847+0000 7fd6ac8a3700 1 -- start start 2023-10-16T12:30:47.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.848+0000 7fd6ac8a3700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd6a80ce9f0 0x7fd6a81f2480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:47.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.848+0000 7fd6ac8a3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6a80d2f70 0x7fd6a80cb0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:47.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.848+0000 7fd6ac8a3700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd6a815d900 0x7fd6a815c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:47.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.848+0000 7fd6ac8a3700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a814ee40 con 0x7fd6a80d2f70 2023-10-16T12:30:47.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.848+0000 7fd6ac8a3700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a80cb630 con 0x7fd6a80ce9f0 2023-10-16T12:30:47.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.848+0000 7fd6ac8a3700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a80c75d0 con 0x7fd6a815d900 2023-10-16T12:30:47.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.848+0000 7fd6a5d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6a80d2f70 0x7fd6a80cb0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:47.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.848+0000 7fd6a5d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6a80d2f70 0x7fd6a80cb0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46188/0 (socket says 172.21.15.16:46188) 2023-10-16T12:30:47.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.848+0000 7fd6a5d9b700 1 -- 172.21.15.16:0/4010811087 learned_addr learned my addr 172.21.15.16:0/4010811087 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:47.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.848+0000 7fd6a659c700 1 --2- 172.21.15.16:0/4010811087 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd6a80ce9f0 0x7fd6a81f2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:47.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.848+0000 7fd6a6d9d700 1 --2- 172.21.15.16:0/4010811087 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd6a815d900 0x7fd6a815c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:47.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.848+0000 7fd6a5d9b700 1 -- 172.21.15.16:0/4010811087 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd6a815d900 msgr2=0x7fd6a815c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:47.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.848+0000 7fd6a5d9b700 1 --2- 172.21.15.16:0/4010811087 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd6a815d900 0x7fd6a815c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:47.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.848+0000 7fd6a5d9b700 1 -- 172.21.15.16:0/4010811087 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd6a80ce9f0 msgr2=0x7fd6a81f2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:47.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.848+0000 7fd6a5d9b700 1 --2- 172.21.15.16:0/4010811087 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd6a80ce9f0 0x7fd6a81f2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:47.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.848+0000 7fd6a5d9b700 1 -- 172.21.15.16:0/4010811087 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6a0009b30 con 0x7fd6a80d2f70 2023-10-16T12:30:47.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.849+0000 7fd6a5d9b700 1 --2- 172.21.15.16:0/4010811087 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6a80d2f70 0x7fd6a80cb0f0 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7fd69c00c550 tx=0x7fd69c00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:47.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.849+0000 7fd696ffd700 1 -- 172.21.15.16:0/4010811087 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd69c0054e0 con 0x7fd6a80d2f70 2023-10-16T12:30:47.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.849+0000 7fd696ffd700 1 -- 172.21.15.16:0/4010811087 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd69c003680 con 0x7fd6a80d2f70 2023-10-16T12:30:47.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.849+0000 7fd696ffd700 1 -- 172.21.15.16:0/4010811087 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd69c003a30 con 0x7fd6a80d2f70 2023-10-16T12:30:47.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.849+0000 7fd6ac8a3700 1 -- 172.21.15.16:0/4010811087 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6a80c78b0 con 0x7fd6a80d2f70 2023-10-16T12:30:47.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.849+0000 7fd6ac8a3700 1 -- 172.21.15.16:0/4010811087 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6a80c7e70 con 0x7fd6a80d2f70 2023-10-16T12:30:47.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.850+0000 7fd6ac8a3700 1 -- 172.21.15.16:0/4010811087 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6a80c8390 con 0x7fd6a80d2f70 2023-10-16T12:30:47.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.850+0000 7fd696ffd700 1 -- 172.21.15.16:0/4010811087 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 81) v1 ==== 89388+0+0 (secure 0 0 0) 0x7fd69c003b90 con 0x7fd6a80d2f70 2023-10-16T12:30:47.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.851+0000 7fd696ffd700 1 --2- 172.21.15.16:0/4010811087 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7fd68806b0b0 0x7fd68806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:47.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.851+0000 7fd6a659c700 1 -- 172.21.15.16:0/4010811087 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7fd68806b0b0 msgr2=0x7fd68806d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3783161600 2023-10-16T12:30:47.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.851+0000 7fd696ffd700 1 -- 172.21.15.16:0/4010811087 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6429+0+0 (secure 0 0 0) 0x7fd69c088f00 con 0x7fd6a80d2f70 2023-10-16T12:30:47.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.851+0000 7fd6a659c700 1 --2- 172.21.15.16:0/4010811087 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7fd68806b0b0 0x7fd68806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:30:47.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:47.855+0000 7fd696ffd700 1 -- 172.21.15.16:0/4010811087 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd6a80c8390 con 0x7fd6a80d2f70 2023-10-16T12:30:48.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.052+0000 7fd6a659c700 1 -- 172.21.15.16:0/4010811087 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7fd68806b0b0 msgr2=0x7fd68806d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3783161600 2023-10-16T12:30:48.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.052+0000 7fd6a659c700 1 --2- 172.21.15.16:0/4010811087 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7fd68806b0b0 0x7fd68806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:30:48.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.106+0000 7fd6ac8a3700 1 -- 172.21.15.16:0/4010811087 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd6a80c8390 con 0x7fd6a80d2f70 2023-10-16T12:30:48.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.107+0000 7fd696ffd700 1 -- 172.21.15.16:0/4010811087 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+4915 (secure 0 0 0) 0x7fd6a80c8390 con 0x7fd6a80d2f70 2023-10-16T12:30:48.110 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:30:48.110 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":70,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7220,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/4274593615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":4274593615},{"type":"v1","addr":"172.21.15.16:6837","nonce":4274593615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":8625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/1624531381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1624531381},{"type":"v1","addr":"172.21.15.16:6836","nonce":1624531381}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":8637,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/1629597454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":1629597454},{"type":"v1","addr":"172.21.15.49:6834","nonce":1629597454}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:30:22.807943+0000","modified":"2023-10-16T12:30:25.836865+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8640},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8640":{"gid":8640,"name":"b","rank":0,"incarnation":66,"state":"up:active","state_seq":3,"addr":"172.21.15.49:6835/2008024326","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":2008024326},{"type":"v1","addr":"172.21.15.49:6835","nonce":2008024326}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2023-10-16T12:30:48.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.109+0000 7fd6ac8a3700 1 -- 172.21.15.16:0/4010811087 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7fd68806b0b0 msgr2=0x7fd68806d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:30:48.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.109+0000 7fd6ac8a3700 1 --2- 172.21.15.16:0/4010811087 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7fd68806b0b0 0x7fd68806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:48.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.109+0000 7fd6ac8a3700 1 -- 172.21.15.16:0/4010811087 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6a80d2f70 msgr2=0x7fd6a80cb0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:48.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.109+0000 7fd6ac8a3700 1 --2- 172.21.15.16:0/4010811087 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6a80d2f70 0x7fd6a80cb0f0 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7fd69c00c550 tx=0x7fd69c00c860 comp rx=0 tx=0).stop 2023-10-16T12:30:48.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.109+0000 7fd6ac8a3700 1 -- 172.21.15.16:0/4010811087 shutdown_connections 2023-10-16T12:30:48.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.109+0000 7fd6ac8a3700 1 --2- 172.21.15.16:0/4010811087 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7fd68806b0b0 0x7fd68806d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:48.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.109+0000 7fd6ac8a3700 1 --2- 172.21.15.16:0/4010811087 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd6a815d900 0x7fd6a815c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:48.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.109+0000 7fd6ac8a3700 1 --2- 172.21.15.16:0/4010811087 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd6a80ce9f0 0x7fd6a81f2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:48.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.109+0000 7fd6ac8a3700 1 --2- 172.21.15.16:0/4010811087 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6a80d2f70 0x7fd6a80cb0f0 unknown :-1 s=CLOSED pgs=1643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:48.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.109+0000 7fd6ac8a3700 1 -- 172.21.15.16:0/4010811087 >> 172.21.15.16:0/4010811087 conn(0x7fd6a8072330 msgr2=0x7fd6a8154c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:48.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.109+0000 7fd6ac8a3700 1 -- 172.21.15.16:0/4010811087 shutdown_connections 2023-10-16T12:30:48.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.109+0000 7fd6ac8a3700 1 -- 172.21.15.16:0/4010811087 wait complete. 2023-10-16T12:30:48.117 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 70 2023-10-16T12:30:48.125 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:30:48.253 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:30:48.252+0000 7f3b079a5200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:30:48.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.252+0000 7f5a32b64700 1 Processor -- start 2023-10-16T12:30:48.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.252+0000 7f5a32b64700 1 -- start start 2023-10-16T12:30:48.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.252+0000 7f5a32b64700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5a2c15b630 0x7f5a2c0d0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:48.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.252+0000 7f5a32b64700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f5a2c15ba10 0x7f5a2c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:48.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.252+0000 7f5a32b64700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a2c0b0970 con 0x7f5a2c0cdde0 2023-10-16T12:30:48.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.252+0000 7f5a32b64700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a2c0cd690 con 0x7f5a2c15ba10 2023-10-16T12:30:48.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.252+0000 7f5a32b64700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a2c0cd970 con 0x7f5a2c15b630 2023-10-16T12:30:48.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.253+0000 7f5a31101700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f5a2c0cdde0 0x7f5a2c0cc2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35476/0 (socket says 172.21.15.16:35476) 2023-10-16T12:30:48.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.253+0000 7f5a31101700 1 -- 172.21.15.16:0/1317887942 learned_addr learned my addr 172.21.15.16:0/1317887942 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:30:48.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.253+0000 7f5a2bfff700 1 --2- 172.21.15.16:0/1317887942 >> v2:172.21.15.49:3300/0 conn(0x7f5a2c15ba10 0x7f5a2c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:48.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.253+0000 7f5a30900700 1 --2- 172.21.15.16:0/1317887942 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5a2c15b630 0x7f5a2c0d0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:48.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.253+0000 7f5a2b7fe700 1 -- 172.21.15.16:0/1317887942 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 486591605 0 0) 0x7f5a2c0b0970 con 0x7f5a2c0cdde0 2023-10-16T12:30:48.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.253+0000 7f5a2b7fe700 1 -- 172.21.15.16:0/1317887942 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a1c0035b0 con 0x7f5a2c0cdde0 2023-10-16T12:30:48.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.253+0000 7f5a2bfff700 1 -- 172.21.15.16:0/1317887942 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5a2c15b630 msgr2=0x7f5a2c0d0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:48.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.253+0000 7f5a2bfff700 1 --2- 172.21.15.16:0/1317887942 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5a2c15b630 0x7f5a2c0d0380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:48.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.253+0000 7f5a2bfff700 1 -- 172.21.15.16:0/1317887942 >> v1:172.21.15.16:6789/0 conn(0x7f5a2c0cdde0 legacy=0x7f5a2c0cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:48.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.253+0000 7f5a2bfff700 1 -- 172.21.15.16:0/1317887942 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a2c0b0c30 con 0x7f5a2c15ba10 2023-10-16T12:30:48.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.253+0000 7f5a30900700 1 --2- 172.21.15.16:0/1317887942 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5a2c15b630 0x7f5a2c0d0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:30:48.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.253+0000 7f5a2bfff700 1 --2- 172.21.15.16:0/1317887942 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5a2c15ba10 0x7f5a2c0cf290 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f5a18009840 tx=0x7f5a18009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c543bae3ae39cb18 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:48.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.253+0000 7f5a2b7fe700 1 -- 172.21.15.16:0/1317887942 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a18011830 con 0x7f5a2c15ba10 2023-10-16T12:30:48.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.254+0000 7f5a2b7fe700 1 -- 172.21.15.16:0/1317887942 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a18011990 con 0x7f5a2c15ba10 2023-10-16T12:30:48.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.254+0000 7f5a2b7fe700 1 -- 172.21.15.16:0/1317887942 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a18011d20 con 0x7f5a2c15ba10 2023-10-16T12:30:48.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.254+0000 7f5a32b64700 1 -- 172.21.15.16:0/1317887942 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5a2c15ba10 msgr2=0x7f5a2c0cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:48.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.254+0000 7f5a32b64700 1 --2- 172.21.15.16:0/1317887942 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5a2c15ba10 0x7f5a2c0cf290 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f5a18009840 tx=0x7f5a18009b50 comp rx=0 tx=0).stop 2023-10-16T12:30:48.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.254+0000 7f5a32b64700 1 -- 172.21.15.16:0/1317887942 shutdown_connections 2023-10-16T12:30:48.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.254+0000 7f5a32b64700 1 --2- 172.21.15.16:0/1317887942 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5a2c15b630 0x7f5a2c0d0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:48.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.254+0000 7f5a32b64700 1 --2- 172.21.15.16:0/1317887942 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5a2c15ba10 0x7f5a2c0cf290 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:48.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.254+0000 7f5a32b64700 1 -- 172.21.15.16:0/1317887942 >> 172.21.15.16:0/1317887942 conn(0x7f5a2c0d14b0 msgr2=0x7f5a2c0d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:48.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.254+0000 7f5a32b64700 1 -- 172.21.15.16:0/1317887942 shutdown_connections 2023-10-16T12:30:48.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.254+0000 7f5a32b64700 1 -- 172.21.15.16:0/1317887942 wait complete. 2023-10-16T12:30:48.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.256+0000 7f5a32b64700 1 Processor -- start 2023-10-16T12:30:48.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.256+0000 7f5a32b64700 1 -- start start 2023-10-16T12:30:48.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.257+0000 7f5a32b64700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a2c0cdde0 0x7f5a2c14a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:48.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.257+0000 7f5a32b64700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5a2c15b630 0x7f5a2c0b4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:48.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.257+0000 7f5a32b64700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5a2c15ba10 0x7f5a2c0cc2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:48.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.257+0000 7f5a32b64700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a2c14ae90 con 0x7f5a2c0cdde0 2023-10-16T12:30:48.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.257+0000 7f5a32b64700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a2c0b4c70 con 0x7f5a2c15ba10 2023-10-16T12:30:48.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.257+0000 7f5a32b64700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a2c0b4f50 con 0x7f5a2c15b630 2023-10-16T12:30:48.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.257+0000 7f5a30900700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a2c0cdde0 0x7f5a2c14a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:48.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.257+0000 7f5a31101700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5a2c15ba10 0x7f5a2c0cc2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:48.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.257+0000 7f5a30900700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a2c0cdde0 0x7f5a2c14a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46204/0 (socket says 172.21.15.16:46204) 2023-10-16T12:30:48.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.257+0000 7f5a30900700 1 -- 172.21.15.16:0/3037213867 learned_addr learned my addr 172.21.15.16:0/3037213867 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:48.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.257+0000 7f5a2bfff700 1 --2- 172.21.15.16:0/3037213867 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5a2c15b630 0x7f5a2c0b4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:48.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.258+0000 7f5a30900700 1 -- 172.21.15.16:0/3037213867 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5a2c15b630 msgr2=0x7f5a2c0b4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:48.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.258+0000 7f5a30900700 1 --2- 172.21.15.16:0/3037213867 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5a2c15b630 0x7f5a2c0b4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:48.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.258+0000 7f5a30900700 1 -- 172.21.15.16:0/3037213867 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5a2c15ba10 msgr2=0x7f5a2c0cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:48.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.258+0000 7f5a30900700 1 --2- 172.21.15.16:0/3037213867 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5a2c15ba10 0x7f5a2c0cc2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:48.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.258+0000 7f5a30900700 1 -- 172.21.15.16:0/3037213867 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a200094a0 con 0x7f5a2c0cdde0 2023-10-16T12:30:48.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.258+0000 7f5a31101700 1 --2- 172.21.15.16:0/3037213867 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5a2c15ba10 0x7f5a2c0cc2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:30:48.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.258+0000 7f5a30900700 1 --2- 172.21.15.16:0/3037213867 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a2c0cdde0 0x7f5a2c14a950 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7f5a2000ebe0 tx=0x7f5a2000eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:48.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.258+0000 7f5a297fa700 1 -- 172.21.15.16:0/3037213867 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a2000c850 con 0x7f5a2c0cdde0 2023-10-16T12:30:48.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.258+0000 7f5a32b64700 1 -- 172.21.15.16:0/3037213867 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a180094a0 con 0x7f5a2c0cdde0 2023-10-16T12:30:48.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.258+0000 7f5a32b64700 1 -- 172.21.15.16:0/3037213867 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a2c0b5510 con 0x7f5a2c0cdde0 2023-10-16T12:30:48.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.258+0000 7f5a297fa700 1 -- 172.21.15.16:0/3037213867 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a2000c9b0 con 0x7f5a2c0cdde0 2023-10-16T12:30:48.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.258+0000 7f5a297fa700 1 -- 172.21.15.16:0/3037213867 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a2000cd20 con 0x7f5a2c0cdde0 2023-10-16T12:30:48.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.259+0000 7f5a32b64700 1 -- 172.21.15.16:0/3037213867 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a0c0052f0 con 0x7f5a2c0cdde0 2023-10-16T12:30:48.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.260+0000 7f5a297fa700 1 -- 172.21.15.16:0/3037213867 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 81) v1 ==== 89388+0+0 (secure 0 0 0) 0x7f5a20013470 con 0x7f5a2c0cdde0 2023-10-16T12:30:48.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.261+0000 7f5a297fa700 1 --2- 172.21.15.16:0/3037213867 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f5a1006b0b0 0x7f5a1006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:48.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.261+0000 7f5a2bfff700 1 -- 172.21.15.16:0/3037213867 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f5a1006b0b0 msgr2=0x7f5a1006d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3783161600 2023-10-16T12:30:48.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.261+0000 7f5a297fa700 1 -- 172.21.15.16:0/3037213867 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6429+0+0 (secure 0 0 0) 0x7f5a20089f90 con 0x7f5a2c0cdde0 2023-10-16T12:30:48.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.261+0000 7f5a2bfff700 1 --2- 172.21.15.16:0/3037213867 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f5a1006b0b0 0x7f5a1006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:30:48.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.265+0000 7f5a297fa700 1 -- 172.21.15.16:0/3037213867 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5a20054a60 con 0x7f5a2c0cdde0 2023-10-16T12:30:48.335 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:30:48.334+0000 7f3b079a5200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T12:30:48.361 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:48.358+0000 7f6bd6932200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:30:48.446 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:48.444+0000 7f6bd6932200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T12:30:48.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.462+0000 7f5a2bfff700 1 -- 172.21.15.16:0/3037213867 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f5a1006b0b0 msgr2=0x7f5a1006d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3783161600 2023-10-16T12:30:48.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.462+0000 7f5a2bfff700 1 --2- 172.21.15.16:0/3037213867 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f5a1006b0b0 0x7f5a1006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:30:48.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.476+0000 7f5a32b64700 1 -- 172.21.15.16:0/3037213867 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5a0c005870 con 0x7f5a2c0cdde0 2023-10-16T12:30:48.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.477+0000 7f5a297fa700 1 -- 172.21.15.16:0/3037213867 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v143) v1 ==== 74+0+17263 (secure 0 0 0) 0x7f5a20058080 con 0x7f5a2c0cdde0 2023-10-16T12:30:48.480 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:30:48.481 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":143,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:30:37.941062+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:30:21.318498+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"141","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2023-10-16T12:30:21.813382+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"142","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/2901217010":"2023-10-17T12:30:37.941028+0000","172.21.15.16:6833/258291157":"2023-10-17T12:30:37.941028+0000","172.21.15.16:6832/258291157":"2023-10-17T12:30:37.941028+0000","172.21.15.16:0/4123017371":"2023-10-17T12:30:37.941028+0000","172.21.15.16:0/2703050386":"2023-10-17T12:30:37.941028+0000","172.21.15.16:0/3188488837":"2023-10-17T12:30:37.941028+0000","172.21.15.16:0/2621298501":"2023-10-17T12:30:37.941028+0000","172.21.15.16:0/1781882931":"2023-10-17T12:30:37.941028+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:30:48.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.479+0000 7f5a32b64700 1 -- 172.21.15.16:0/3037213867 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f5a1006b0b0 msgr2=0x7f5a1006d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:30:48.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.479+0000 7f5a32b64700 1 --2- 172.21.15.16:0/3037213867 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f5a1006b0b0 0x7f5a1006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:48.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.479+0000 7f5a32b64700 1 -- 172.21.15.16:0/3037213867 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a2c0cdde0 msgr2=0x7f5a2c14a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:48.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.479+0000 7f5a32b64700 1 --2- 172.21.15.16:0/3037213867 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a2c0cdde0 0x7f5a2c14a950 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7f5a2000ebe0 tx=0x7f5a2000eef0 comp rx=0 tx=0).stop 2023-10-16T12:30:48.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.479+0000 7f5a32b64700 1 -- 172.21.15.16:0/3037213867 shutdown_connections 2023-10-16T12:30:48.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.479+0000 7f5a32b64700 1 --2- 172.21.15.16:0/3037213867 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f5a1006b0b0 0x7f5a1006d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:48.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.479+0000 7f5a32b64700 1 --2- 172.21.15.16:0/3037213867 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5a2c15ba10 0x7f5a2c0cc2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:48.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.479+0000 7f5a32b64700 1 --2- 172.21.15.16:0/3037213867 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a2c0cdde0 0x7f5a2c14a950 unknown :-1 s=CLOSED pgs=1645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:48.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.479+0000 7f5a32b64700 1 --2- 172.21.15.16:0/3037213867 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5a2c15b630 0x7f5a2c0b4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:48.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.479+0000 7f5a32b64700 1 -- 172.21.15.16:0/3037213867 >> 172.21.15.16:0/3037213867 conn(0x7f5a2c0d14b0 msgr2=0x7f5a2c070410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:48.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.479+0000 7f5a32b64700 1 -- 172.21.15.16:0/3037213867 shutdown_connections 2023-10-16T12:30:48.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.479+0000 7f5a32b64700 1 -- 172.21.15.16:0/3037213867 wait complete. 2023-10-16T12:30:48.491 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:30:48.490+0000 7f3b079a5200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:30:48.495 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T12:30:21.318498+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '141', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T12:30:21.813382+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '142', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T12:30:48.497 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:30:48.606 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:48.603+0000 7f6bd6932200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:30:48.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.624+0000 7fe7d9838700 1 Processor -- start 2023-10-16T12:30:48.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.625+0000 7fe7d9838700 1 -- start start 2023-10-16T12:30:48.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.625+0000 7fe7d9838700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7d415ec80 0x7fe7d40cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:48.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.625+0000 7fe7d9838700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe7d40d2110 0x7fe7d40cd1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:48.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.625+0000 7fe7d9838700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7d40b2440 con 0x7fe7d415f060 2023-10-16T12:30:48.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.625+0000 7fe7d9838700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7d40b2c50 con 0x7fe7d40d2110 2023-10-16T12:30:48.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.625+0000 7fe7d9838700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d40d00e0 con 0x7fe7d415ec80 2023-10-16T12:30:48.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.626+0000 7fe7d37fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe7d40d2110 0x7fe7d40cd1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:48.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.626+0000 7fe7d27fc700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe7d415f060 0x7fe7d40b4440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35518/0 (socket says 172.21.15.16:35518) 2023-10-16T12:30:48.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.626+0000 7fe7d27fc700 1 -- 172.21.15.16:0/3511717133 learned_addr learned my addr 172.21.15.16:0/3511717133 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:30:48.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.626+0000 7fe7d2ffd700 1 --2- 172.21.15.16:0/3511717133 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7d415ec80 0x7fe7d40cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:48.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.626+0000 7fe7d37fe700 1 -- 172.21.15.16:0/3511717133 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7d415ec80 msgr2=0x7fe7d40cdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:48.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.626+0000 7fe7d37fe700 1 --2- 172.21.15.16:0/3511717133 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7d415ec80 0x7fe7d40cdec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:48.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.626+0000 7fe7d37fe700 1 -- 172.21.15.16:0/3511717133 >> v1:172.21.15.16:6789/0 conn(0x7fe7d415f060 legacy=0x7fe7d40b4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:48.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.626+0000 7fe7d37fe700 1 -- 172.21.15.16:0/3511717133 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7d40cef60 con 0x7fe7d40d2110 2023-10-16T12:30:48.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.626+0000 7fe7d37fe700 1 --2- 172.21.15.16:0/3511717133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7d40d2110 0x7fe7d40cd1a0 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc009d90 tx=0x7fe7cc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81dcf26099e7c225 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:48.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.626+0000 7fe7d1ffb700 1 -- 172.21.15.16:0/3511717133 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7cc00e070 con 0x7fe7d40d2110 2023-10-16T12:30:48.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.626+0000 7fe7d1ffb700 1 -- 172.21.15.16:0/3511717133 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7cc00bb20 con 0x7fe7d40d2110 2023-10-16T12:30:48.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.626+0000 7fe7d1ffb700 1 -- 172.21.15.16:0/3511717133 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7cc012430 con 0x7fe7d40d2110 2023-10-16T12:30:48.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.627+0000 7fe7d9838700 1 -- 172.21.15.16:0/3511717133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7d40d2110 msgr2=0x7fe7d40cd1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:48.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.627+0000 7fe7d9838700 1 --2- 172.21.15.16:0/3511717133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7d40d2110 0x7fe7d40cd1a0 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc009d90 tx=0x7fe7cc00b330 comp rx=0 tx=0).stop 2023-10-16T12:30:48.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.627+0000 7fe7d9838700 1 -- 172.21.15.16:0/3511717133 shutdown_connections 2023-10-16T12:30:48.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.627+0000 7fe7d9838700 1 --2- 172.21.15.16:0/3511717133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7d40d2110 0x7fe7d40cd1a0 unknown :-1 s=CLOSED pgs=1647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:48.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.627+0000 7fe7d9838700 1 --2- 172.21.15.16:0/3511717133 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7d415ec80 0x7fe7d40cdec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:48.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.627+0000 7fe7d9838700 1 -- 172.21.15.16:0/3511717133 >> 172.21.15.16:0/3511717133 conn(0x7fe7d40d15f0 msgr2=0x7fe7d40d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:48.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.627+0000 7fe7d9838700 1 -- 172.21.15.16:0/3511717133 shutdown_connections 2023-10-16T12:30:48.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.627+0000 7fe7d9838700 1 -- 172.21.15.16:0/3511717133 wait complete. 2023-10-16T12:30:48.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.629+0000 7fe7d9838700 1 Processor -- start 2023-10-16T12:30:48.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.629+0000 7fe7d9838700 1 -- start start 2023-10-16T12:30:48.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.630+0000 7fe7d9838700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7d40d2110 0x7fe7d40c53a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:48.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.630+0000 7fe7d9838700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7d415ec80 0x7fe7d414d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:48.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.630+0000 7fe7d9838700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7d415f060 0x7fe7d415d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:48.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.630+0000 7fe7d9838700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d40c86c0 con 0x7fe7d415ec80 2023-10-16T12:30:48.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.630+0000 7fe7d9838700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d40c8910 con 0x7fe7d40d2110 2023-10-16T12:30:48.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.630+0000 7fe7d9838700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d40c8bf0 con 0x7fe7d415f060 2023-10-16T12:30:48.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.630+0000 7fe7d27fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7d415ec80 0x7fe7d414d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:48.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.630+0000 7fe7d27fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7d415ec80 0x7fe7d414d8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46250/0 (socket says 172.21.15.16:46250) 2023-10-16T12:30:48.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.630+0000 7fe7d27fc700 1 -- 172.21.15.16:0/391336229 learned_addr learned my addr 172.21.15.16:0/391336229 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:48.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.631+0000 7fe7d37fe700 1 --2- 172.21.15.16:0/391336229 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7d415f060 0x7fe7d415d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:48.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.631+0000 7fe7d2ffd700 1 --2- 172.21.15.16:0/391336229 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7d40d2110 0x7fe7d40c53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:48.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.631+0000 7fe7d27fc700 1 -- 172.21.15.16:0/391336229 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7d415f060 msgr2=0x7fe7d415d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:48.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.631+0000 7fe7d27fc700 1 --2- 172.21.15.16:0/391336229 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7d415f060 0x7fe7d415d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:48.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.631+0000 7fe7d27fc700 1 -- 172.21.15.16:0/391336229 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7d40d2110 msgr2=0x7fe7d40c53a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:48.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.631+0000 7fe7d27fc700 1 --2- 172.21.15.16:0/391336229 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7d40d2110 0x7fe7d40c53a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:48.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.631+0000 7fe7d27fc700 1 -- 172.21.15.16:0/391336229 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7cc009a70 con 0x7fe7d415ec80 2023-10-16T12:30:48.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.631+0000 7fe7d27fc700 1 --2- 172.21.15.16:0/391336229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7d415ec80 0x7fe7d414d8b0 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7fe7c800c450 tx=0x7fe7c800c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:48.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.631+0000 7fe7bbfff700 1 -- 172.21.15.16:0/391336229 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7c8005d70 con 0x7fe7d415ec80 2023-10-16T12:30:48.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.631+0000 7fe7bbfff700 1 -- 172.21.15.16:0/391336229 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7c800ce60 con 0x7fe7d415ec80 2023-10-16T12:30:48.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.631+0000 7fe7bbfff700 1 -- 172.21.15.16:0/391336229 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7c80072d0 con 0x7fe7d415ec80 2023-10-16T12:30:48.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.631+0000 7fe7d9838700 1 -- 172.21.15.16:0/391336229 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7d40c8ed0 con 0x7fe7d415ec80 2023-10-16T12:30:48.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.631+0000 7fe7d9838700 1 -- 172.21.15.16:0/391336229 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7d414ddf0 con 0x7fe7d415ec80 2023-10-16T12:30:48.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.632+0000 7fe7d9838700 1 -- 172.21.15.16:0/391336229 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7d40c93f0 con 0x7fe7d415ec80 2023-10-16T12:30:48.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.633+0000 7fe7bbfff700 1 -- 172.21.15.16:0/391336229 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 81) v1 ==== 89388+0+0 (secure 0 0 0) 0x7fe7c8007430 con 0x7fe7d415ec80 2023-10-16T12:30:48.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.633+0000 7fe7bbfff700 1 --2- 172.21.15.16:0/391336229 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7fe7c006b0b0 0x7fe7c006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:48.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.634+0000 7fe7d2ffd700 1 -- 172.21.15.16:0/391336229 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7fe7c006b0b0 msgr2=0x7fe7c006d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3783161600 2023-10-16T12:30:48.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.634+0000 7fe7bbfff700 1 -- 172.21.15.16:0/391336229 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6429+0+0 (secure 0 0 0) 0x7fe7c8088d20 con 0x7fe7d415ec80 2023-10-16T12:30:48.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.634+0000 7fe7d2ffd700 1 --2- 172.21.15.16:0/391336229 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7fe7c006b0b0 0x7fe7c006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:30:48.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.638+0000 7fe7bbfff700 1 -- 172.21.15.16:0/391336229 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe7d40c93f0 con 0x7fe7d415ec80 2023-10-16T12:30:48.731 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:30:48.730+0000 7f3b079a5200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T12:30:48.809 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:30:48.808+0000 7f3b079a5200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T12:30:48.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.835+0000 7fe7d2ffd700 1 -- 172.21.15.16:0/391336229 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7fe7c006b0b0 msgr2=0x7fe7c006d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3783161600 2023-10-16T12:30:48.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.835+0000 7fe7d2ffd700 1 --2- 172.21.15.16:0/391336229 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7fe7c006b0b0 0x7fe7c006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:30:48.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.842+0000 7fe7d9838700 1 -- 172.21.15.16:0/391336229 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe7d40c93f0 con 0x7fe7d415ec80 2023-10-16T12:30:48.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.843+0000 7fe7bbfff700 1 -- 172.21.15.16:0/391336229 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v143) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe7d40c93f0 con 0x7fe7d415ec80 2023-10-16T12:30:48.846 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T12:30:48.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.844+0000 7fe7d9838700 1 -- 172.21.15.16:0/391336229 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7fe7c006b0b0 msgr2=0x7fe7c006d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:30:48.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.845+0000 7fe7d9838700 1 --2- 172.21.15.16:0/391336229 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7fe7c006b0b0 0x7fe7c006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:48.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.845+0000 7fe7d9838700 1 -- 172.21.15.16:0/391336229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7d415ec80 msgr2=0x7fe7d414d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:48.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.845+0000 7fe7d9838700 1 --2- 172.21.15.16:0/391336229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7d415ec80 0x7fe7d414d8b0 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7fe7c800c450 tx=0x7fe7c800c760 comp rx=0 tx=0).stop 2023-10-16T12:30:48.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.845+0000 7fe7d9838700 1 -- 172.21.15.16:0/391336229 shutdown_connections 2023-10-16T12:30:48.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.845+0000 7fe7d9838700 1 --2- 172.21.15.16:0/391336229 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7fe7c006b0b0 0x7fe7c006d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:48.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.845+0000 7fe7d9838700 1 --2- 172.21.15.16:0/391336229 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7d415f060 0x7fe7d415d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:48.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.845+0000 7fe7d9838700 1 --2- 172.21.15.16:0/391336229 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7d40d2110 0x7fe7d40c53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:48.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.845+0000 7fe7d9838700 1 --2- 172.21.15.16:0/391336229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7d415ec80 0x7fe7d414d8b0 unknown :-1 s=CLOSED pgs=1648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:48.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.845+0000 7fe7d9838700 1 -- 172.21.15.16:0/391336229 >> 172.21.15.16:0/391336229 conn(0x7fe7d40d15f0 msgr2=0x7fe7d40b4170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:48.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.845+0000 7fe7d9838700 1 -- 172.21.15.16:0/391336229 shutdown_connections 2023-10-16T12:30:48.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.845+0000 7fe7d9838700 1 -- 172.21.15.16:0/391336229 wait complete. 2023-10-16T12:30:48.853 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:48.850+0000 7f6bd6932200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T12:30:48.861 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T12:30:48.937 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:48.934+0000 7f6bd6932200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T12:30:48.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.990+0000 7f9679817700 1 Processor -- start 2023-10-16T12:30:48.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.990+0000 7f9679817700 1 -- start start 2023-10-16T12:30:48.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.991+0000 7f9679817700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f967415da10 0x7f96740d1520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:48.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.991+0000 7f9679817700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f96740d4060 0x7f96740cf2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:48.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.991+0000 7f9679817700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f96740d1d20 0x7f96740d2170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:48.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.991+0000 7f9679817700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96740d26b0 con 0x7f96740d1d20 2023-10-16T12:30:48.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.991+0000 7f9679817700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96740d4540 con 0x7f967415da10 2023-10-16T12:30:48.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.991+0000 7f9679817700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96740ce7e0 con 0x7f96740d4060 2023-10-16T12:30:48.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.991+0000 7f96737fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f96740d1d20 0x7f96740d2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:48.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.991+0000 7f96737fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f96740d1d20 0x7f96740d2170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46262/0 (socket says 172.21.15.16:46262) 2023-10-16T12:30:48.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.991+0000 7f96737fe700 1 -- 172.21.15.16:0/970069952 learned_addr learned my addr 172.21.15.16:0/970069952 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:48.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.991+0000 7f96727fc700 1 --2- 172.21.15.16:0/970069952 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f96740d4060 0x7f96740cf2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:48.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.991+0000 7f9672ffd700 1 --2- 172.21.15.16:0/970069952 >> v2:172.21.15.49:3300/0 conn(0x7f967415da10 0x7f96740d1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:48.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.992+0000 7f96737fe700 1 -- 172.21.15.16:0/970069952 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f96740d4060 msgr2=0x7f96740cf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:48.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.992+0000 7f96737fe700 1 --2- 172.21.15.16:0/970069952 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f96740d4060 0x7f96740cf2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:49.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.992+0000 7f96737fe700 1 -- 172.21.15.16:0/970069952 >> v2:172.21.15.49:3300/0 conn(0x7f967415da10 msgr2=0x7f96740d1520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:49.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.992+0000 7f96737fe700 1 --2- 172.21.15.16:0/970069952 >> v2:172.21.15.49:3300/0 conn(0x7f967415da10 0x7f96740d1520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:49.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.992+0000 7f96737fe700 1 -- 172.21.15.16:0/970069952 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96740ceac0 con 0x7f96740d1d20 2023-10-16T12:30:49.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.992+0000 7f96737fe700 1 --2- 172.21.15.16:0/970069952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f96740d1d20 0x7f96740d2170 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7f967408b540 tx=0x7f966c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4544d924414309b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:49.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.992+0000 7f9671ffb700 1 -- 172.21.15.16:0/970069952 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f966c00e070 con 0x7f96740d1d20 2023-10-16T12:30:49.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.992+0000 7f9671ffb700 1 -- 172.21.15.16:0/970069952 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f966c00b910 con 0x7f96740d1d20 2023-10-16T12:30:49.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.992+0000 7f9671ffb700 1 -- 172.21.15.16:0/970069952 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f966c00bcc0 con 0x7f96740d1d20 2023-10-16T12:30:49.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.993+0000 7f9679817700 1 -- 172.21.15.16:0/970069952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f96740d1d20 msgr2=0x7f96740d2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:49.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.993+0000 7f9679817700 1 --2- 172.21.15.16:0/970069952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f96740d1d20 0x7f96740d2170 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7f967408b540 tx=0x7f966c00b330 comp rx=0 tx=0).stop 2023-10-16T12:30:49.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.993+0000 7f9679817700 1 -- 172.21.15.16:0/970069952 shutdown_connections 2023-10-16T12:30:49.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.993+0000 7f9679817700 1 --2- 172.21.15.16:0/970069952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f96740d1d20 0x7f96740d2170 unknown :-1 s=CLOSED pgs=1649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:49.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.993+0000 7f9679817700 1 --2- 172.21.15.16:0/970069952 >> v2:172.21.15.49:3300/0 conn(0x7f967415da10 0x7f96740d1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:49.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.993+0000 7f9679817700 1 --2- 172.21.15.16:0/970069952 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f96740d4060 0x7f96740cf2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:49.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.993+0000 7f9679817700 1 -- 172.21.15.16:0/970069952 >> 172.21.15.16:0/970069952 conn(0x7f967409c9f0 msgr2=0x7f96740d3740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:49.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.993+0000 7f9679817700 1 -- 172.21.15.16:0/970069952 shutdown_connections 2023-10-16T12:30:49.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.993+0000 7f9679817700 1 -- 172.21.15.16:0/970069952 wait complete. 2023-10-16T12:30:49.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.994+0000 7f9679817700 1 Processor -- start 2023-10-16T12:30:49.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.994+0000 7f9679817700 1 -- start start 2023-10-16T12:30:49.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.995+0000 7f9679817700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f96740d1d20 0x7f967415af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:49.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.995+0000 7f9679817700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f96740d4060 0x7f96740ca810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:49.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.995+0000 7f9679817700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f967415da10 0x7f967414daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:49.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.995+0000 7f9679817700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96740b3d80 con 0x7f96740d1d20 2023-10-16T12:30:49.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.995+0000 7f9679817700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96740b2eb0 con 0x7f967415da10 2023-10-16T12:30:49.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.995+0000 7f9679817700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96740b3170 con 0x7f96740d4060 2023-10-16T12:30:49.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.995+0000 7f9672ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f96740d1d20 0x7f967415af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:49.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.995+0000 7f9672ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f96740d1d20 0x7f967415af80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46264/0 (socket says 172.21.15.16:46264) 2023-10-16T12:30:49.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.995+0000 7f9672ffd700 1 -- 172.21.15.16:0/2408338001 learned_addr learned my addr 172.21.15.16:0/2408338001 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:49.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.995+0000 7f96737fe700 1 --2- 172.21.15.16:0/2408338001 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f967415da10 0x7f967414daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:49.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.995+0000 7f96727fc700 1 --2- 172.21.15.16:0/2408338001 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f96740d4060 0x7f96740ca810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:49.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.996+0000 7f96727fc700 1 -- 172.21.15.16:0/2408338001 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f967415da10 msgr2=0x7f967414daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:49.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.996+0000 7f96727fc700 1 --2- 172.21.15.16:0/2408338001 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f967415da10 0x7f967414daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:49.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.996+0000 7f96727fc700 1 -- 172.21.15.16:0/2408338001 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f96740d1d20 msgr2=0x7f967415af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:49.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.996+0000 7f96727fc700 1 --2- 172.21.15.16:0/2408338001 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f96740d1d20 0x7f967415af80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:49.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.996+0000 7f96727fc700 1 -- 172.21.15.16:0/2408338001 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f966c009b30 con 0x7f96740d4060 2023-10-16T12:30:49.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.996+0000 7f9672ffd700 1 --2- 172.21.15.16:0/2408338001 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f96740d1d20 0x7f967415af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:30:49.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.996+0000 7f96727fc700 1 --2- 172.21.15.16:0/2408338001 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f96740d4060 0x7f96740ca810 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f966800e840 tx=0x7f966800eb50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:49.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.996+0000 7f965bfff700 1 -- 172.21.15.16:0/2408338001 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9668013680 con 0x7f96740d4060 2023-10-16T12:30:49.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.996+0000 7f965bfff700 1 -- 172.21.15.16:0/2408338001 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96680137e0 con 0x7f96740d4060 2023-10-16T12:30:49.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.996+0000 7f9679817700 1 -- 172.21.15.16:0/2408338001 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96740b3450 con 0x7f96740d4060 2023-10-16T12:30:49.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.996+0000 7f965bfff700 1 -- 172.21.15.16:0/2408338001 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9668013b50 con 0x7f96740d4060 2023-10-16T12:30:49.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.996+0000 7f9679817700 1 -- 172.21.15.16:0/2408338001 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96740b3a10 con 0x7f96740d4060 2023-10-16T12:30:49.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.997+0000 7f9679817700 1 -- 172.21.15.16:0/2408338001 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96740b4490 con 0x7f96740d4060 2023-10-16T12:30:49.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.998+0000 7f965bfff700 1 -- 172.21.15.16:0/2408338001 <== mon.2 v2:172.21.15.49:3301/0 4 ==== mgrmap(e 81) v1 ==== 89388+0+0 (secure 0 0 0) 0x7f9668013cb0 con 0x7f96740d4060 2023-10-16T12:30:49.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.998+0000 7f965bfff700 1 --2- 172.21.15.16:0/2408338001 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f966006b0b0 0x7f966006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:49.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.998+0000 7f9672ffd700 1 -- 172.21.15.16:0/2408338001 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f966006b0b0 msgr2=0x7f966006d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3783161600 2023-10-16T12:30:49.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.998+0000 7f965bfff700 1 -- 172.21.15.16:0/2408338001 <== mon.2 v2:172.21.15.49:3301/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6429+0+0 (secure 0 0 0) 0x7f9668089b10 con 0x7f96740d4060 2023-10-16T12:30:49.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:48.998+0000 7f9672ffd700 1 --2- 172.21.15.16:0/2408338001 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f966006b0b0 0x7f966006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:30:49.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.006+0000 7f965bfff700 1 -- 172.21.15.16:0/2408338001 <== mon.2 v2:172.21.15.49:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f96680547e0 con 0x7f96740d4060 2023-10-16T12:30:49.089 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:30:49.088+0000 7f3b079a5200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T12:30:49.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.199+0000 7f9672ffd700 1 -- 172.21.15.16:0/2408338001 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f966006b0b0 msgr2=0x7f966006d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3783161600 2023-10-16T12:30:49.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.199+0000 7f9672ffd700 1 --2- 172.21.15.16:0/2408338001 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f966006b0b0 0x7f966006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:30:49.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.210+0000 7f9679817700 1 -- 172.21.15.16:0/2408338001 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f967404ebb0 con 0x7f96740d4060 2023-10-16T12:30:49.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.210+0000 7f965bfff700 1 -- 172.21.15.16:0/2408338001 <== mon.2 v2:172.21.15.49:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v143) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9668057e00 con 0x7f96740d4060 2023-10-16T12:30:49.214 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-10-16T12:30:49.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.213+0000 7f9679817700 1 -- 172.21.15.16:0/2408338001 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f966006b0b0 msgr2=0x7f966006d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:30:49.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.213+0000 7f9679817700 1 --2- 172.21.15.16:0/2408338001 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f966006b0b0 0x7f966006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:49.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.213+0000 7f9679817700 1 -- 172.21.15.16:0/2408338001 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f96740d4060 msgr2=0x7f96740ca810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:49.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.213+0000 7f9679817700 1 --2- 172.21.15.16:0/2408338001 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f96740d4060 0x7f96740ca810 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f966800e840 tx=0x7f966800eb50 comp rx=0 tx=0).stop 2023-10-16T12:30:49.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.213+0000 7f9679817700 1 -- 172.21.15.16:0/2408338001 shutdown_connections 2023-10-16T12:30:49.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.213+0000 7f9679817700 1 --2- 172.21.15.16:0/2408338001 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f966006b0b0 0x7f966006d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:49.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.213+0000 7f9679817700 1 --2- 172.21.15.16:0/2408338001 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f967415da10 0x7f967414daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:49.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.213+0000 7f9679817700 1 --2- 172.21.15.16:0/2408338001 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f96740d1d20 0x7f967415af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:49.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.213+0000 7f9679817700 1 --2- 172.21.15.16:0/2408338001 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f96740d4060 0x7f96740ca810 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:49.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.213+0000 7f9679817700 1 -- 172.21.15.16:0/2408338001 >> 172.21.15.16:0/2408338001 conn(0x7f967409c9f0 msgr2=0x7f967415d040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:49.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.213+0000 7f9679817700 1 -- 172.21.15.16:0/2408338001 shutdown_connections 2023-10-16T12:30:49.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.213+0000 7f9679817700 1 -- 172.21.15.16:0/2408338001 wait complete. 2023-10-16T12:30:49.219 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:49.216+0000 7f6bd6932200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T12:30:49.228 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T12:30:49.322 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:30:49.321+0000 7f3b079a5200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T12:30:49.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.355+0000 7f5cefbec700 1 Processor -- start 2023-10-16T12:30:49.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.355+0000 7f5cefbec700 1 -- start start 2023-10-16T12:30:49.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.356+0000 7f5cefbec700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5ce815b2c0 0x7f5ce8161340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:49.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.356+0000 7f5cefbec700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f5ce806a7e0 0x7f5ce806ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:49.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.356+0000 7f5cefbec700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ce815ecb0 con 0x7f5ce806e570 2023-10-16T12:30:49.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.356+0000 7f5cefbec700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ce806b1b0 con 0x7f5ce806a7e0 2023-10-16T12:30:49.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.356+0000 7f5cefbec700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce8161880 con 0x7f5ce815b2c0 2023-10-16T12:30:49.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.356+0000 7f5ced187700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f5ce806e570 0x7f5ce815e5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35548/0 (socket says 172.21.15.16:35548) 2023-10-16T12:30:49.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.356+0000 7f5ced187700 1 -- 172.21.15.16:0/2877302085 learned_addr learned my addr 172.21.15.16:0/2877302085 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:30:49.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.356+0000 7f5ced988700 1 --2- 172.21.15.16:0/2877302085 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5ce815b2c0 0x7f5ce8161340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:49.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.356+0000 7f5cee189700 1 --2- 172.21.15.16:0/2877302085 >> v2:172.21.15.49:3300/0 conn(0x7f5ce806a7e0 0x7f5ce806ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:49.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.357+0000 7f5cec986700 1 -- 172.21.15.16:0/2877302085 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2479947557 0 0) 0x7f5ce815ecb0 con 0x7f5ce806e570 2023-10-16T12:30:49.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.357+0000 7f5cec986700 1 -- 172.21.15.16:0/2877302085 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5cd40035b0 con 0x7f5ce806e570 2023-10-16T12:30:49.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.357+0000 7f5cee189700 1 -- 172.21.15.16:0/2877302085 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5ce815b2c0 msgr2=0x7f5ce8161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:49.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.357+0000 7f5cee189700 1 --2- 172.21.15.16:0/2877302085 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5ce815b2c0 0x7f5ce8161340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:49.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.357+0000 7f5cee189700 1 -- 172.21.15.16:0/2877302085 >> v1:172.21.15.16:6789/0 conn(0x7f5ce806e570 legacy=0x7f5ce815e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:49.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.357+0000 7f5cee189700 1 -- 172.21.15.16:0/2877302085 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ce815b6a0 con 0x7f5ce806a7e0 2023-10-16T12:30:49.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.357+0000 7f5ced988700 1 --2- 172.21.15.16:0/2877302085 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5ce815b2c0 0x7f5ce8161340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:30:49.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.357+0000 7f5cee189700 1 --2- 172.21.15.16:0/2877302085 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5ce806a7e0 0x7f5ce806ac70 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f5ce0009d90 tx=0x7f5ce000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6909139401a618c4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:49.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.357+0000 7f5cec986700 1 -- 172.21.15.16:0/2877302085 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ce000e070 con 0x7f5ce806a7e0 2023-10-16T12:30:49.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.357+0000 7f5cec986700 1 -- 172.21.15.16:0/2877302085 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ce000bb20 con 0x7f5ce806a7e0 2023-10-16T12:30:49.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.357+0000 7f5cec986700 1 -- 172.21.15.16:0/2877302085 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ce0012430 con 0x7f5ce806a7e0 2023-10-16T12:30:49.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.358+0000 7f5cefbec700 1 -- 172.21.15.16:0/2877302085 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5ce806a7e0 msgr2=0x7f5ce806ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:49.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.358+0000 7f5cefbec700 1 --2- 172.21.15.16:0/2877302085 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5ce806a7e0 0x7f5ce806ac70 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f5ce0009d90 tx=0x7f5ce000b330 comp rx=0 tx=0).stop 2023-10-16T12:30:49.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.358+0000 7f5cefbec700 1 -- 172.21.15.16:0/2877302085 shutdown_connections 2023-10-16T12:30:49.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.358+0000 7f5cefbec700 1 --2- 172.21.15.16:0/2877302085 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5ce815b2c0 0x7f5ce8161340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:49.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.358+0000 7f5cefbec700 1 --2- 172.21.15.16:0/2877302085 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5ce806a7e0 0x7f5ce806ac70 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:49.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.358+0000 7f5cefbec700 1 -- 172.21.15.16:0/2877302085 >> 172.21.15.16:0/2877302085 conn(0x7f5ce80b1f40 msgr2=0x7f5ce80b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:49.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.358+0000 7f5cefbec700 1 -- 172.21.15.16:0/2877302085 shutdown_connections 2023-10-16T12:30:49.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.358+0000 7f5cefbec700 1 -- 172.21.15.16:0/2877302085 wait complete. 2023-10-16T12:30:49.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.360+0000 7f5cefbec700 1 Processor -- start 2023-10-16T12:30:49.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.360+0000 7f5cefbec700 1 -- start start 2023-10-16T12:30:49.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.361+0000 7f5cefbec700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5ce806a7e0 0x7f5ce815e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:49.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.361+0000 7f5cefbec700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ce806e570 0x7f5ce80cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:49.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.361+0000 7f5cefbec700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5ce815b2c0 0x7f5ce81563c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:49.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.361+0000 7f5cefbec700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce815eda0 con 0x7f5ce806e570 2023-10-16T12:30:49.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.361+0000 7f5cefbec700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce815f080 con 0x7f5ce806a7e0 2023-10-16T12:30:49.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.361+0000 7f5cefbec700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce815f360 con 0x7f5ce815b2c0 2023-10-16T12:30:49.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.361+0000 7f5ced187700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ce806e570 0x7f5ce80cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:49.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.361+0000 7f5ced187700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ce806e570 0x7f5ce80cbd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46282/0 (socket says 172.21.15.16:46282) 2023-10-16T12:30:49.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.361+0000 7f5ced187700 1 -- 172.21.15.16:0/3125091117 learned_addr learned my addr 172.21.15.16:0/3125091117 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:49.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.361+0000 7f5ced988700 1 --2- 172.21.15.16:0/3125091117 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5ce806a7e0 0x7f5ce815e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:49.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.361+0000 7f5cee189700 1 --2- 172.21.15.16:0/3125091117 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5ce815b2c0 0x7f5ce81563c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:49.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.361+0000 7f5ced187700 1 -- 172.21.15.16:0/3125091117 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5ce815b2c0 msgr2=0x7f5ce81563c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:49.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.361+0000 7f5ced187700 1 --2- 172.21.15.16:0/3125091117 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5ce815b2c0 0x7f5ce81563c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:49.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.361+0000 7f5ced187700 1 -- 172.21.15.16:0/3125091117 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5ce806a7e0 msgr2=0x7f5ce815e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:49.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.361+0000 7f5ced187700 1 --2- 172.21.15.16:0/3125091117 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5ce806a7e0 0x7f5ce815e4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:49.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.361+0000 7f5ced187700 1 -- 172.21.15.16:0/3125091117 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ce40094a0 con 0x7f5ce806e570 2023-10-16T12:30:49.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.361+0000 7f5ced187700 1 --2- 172.21.15.16:0/3125091117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ce806e570 0x7f5ce80cbd00 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7f5cd8001eb0 tx=0x7f5cd800c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:49.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.361+0000 7f5cde7fc700 1 -- 172.21.15.16:0/3125091117 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cd8005d70 con 0x7f5ce806e570 2023-10-16T12:30:49.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.361+0000 7f5cde7fc700 1 -- 172.21.15.16:0/3125091117 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cd80070d0 con 0x7f5ce806e570 2023-10-16T12:30:49.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.361+0000 7f5cefbec700 1 -- 172.21.15.16:0/3125091117 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ce0009a70 con 0x7f5ce806e570 2023-10-16T12:30:49.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.361+0000 7f5cde7fc700 1 -- 172.21.15.16:0/3125091117 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cd8007460 con 0x7f5ce806e570 2023-10-16T12:30:49.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.361+0000 7f5cefbec700 1 -- 172.21.15.16:0/3125091117 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ce815f920 con 0x7f5ce806e570 2023-10-16T12:30:49.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.363+0000 7f5cefbec700 1 -- 172.21.15.16:0/3125091117 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ce804ebb0 con 0x7f5ce806e570 2023-10-16T12:30:49.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.363+0000 7f5cde7fc700 1 -- 172.21.15.16:0/3125091117 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 81) v1 ==== 89388+0+0 (secure 0 0 0) 0x7f5cd80075c0 con 0x7f5ce806e570 2023-10-16T12:30:49.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.363+0000 7f5cde7fc700 1 --2- 172.21.15.16:0/3125091117 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f5ccc06b0b0 0x7f5ccc06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:49.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.363+0000 7f5ced988700 1 -- 172.21.15.16:0/3125091117 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f5ccc06b0b0 msgr2=0x7f5ccc06d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3783161600 2023-10-16T12:30:49.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.363+0000 7f5cde7fc700 1 -- 172.21.15.16:0/3125091117 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6429+0+0 (secure 0 0 0) 0x7f5cd8088fc0 con 0x7f5ce806e570 2023-10-16T12:30:49.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.363+0000 7f5ced988700 1 --2- 172.21.15.16:0/3125091117 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f5ccc06b0b0 0x7f5ccc06d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:30:49.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.368+0000 7f5cde7fc700 1 -- 172.21.15.16:0/3125091117 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5cd80bbfe0 con 0x7f5ce806e570 2023-10-16T12:30:49.464 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:49.461+0000 7f6bd6932200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T12:30:49.480 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:30:49.479+0000 7f3b079a5200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T12:30:49.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.564+0000 7f5ced988700 1 -- 172.21.15.16:0/3125091117 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f5ccc06b0b0 msgr2=0x7f5ccc06d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3783161600 2023-10-16T12:30:49.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.564+0000 7f5ced988700 1 --2- 172.21.15.16:0/3125091117 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f5ccc06b0b0 0x7f5ccc06d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:30:49.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.574+0000 7f5cefbec700 1 -- 172.21.15.16:0/3125091117 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5ce80b6370 con 0x7f5ce806e570 2023-10-16T12:30:49.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.574+0000 7f5cde7fc700 1 -- 172.21.15.16:0/3125091117 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v143) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5cd80170a0 con 0x7f5ce806e570 2023-10-16T12:30:49.577 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-10-16T12:30:49.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.578+0000 7f5cefbec700 1 -- 172.21.15.16:0/3125091117 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f5ccc06b0b0 msgr2=0x7f5ccc06d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:30:49.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.578+0000 7f5cefbec700 1 --2- 172.21.15.16:0/3125091117 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f5ccc06b0b0 0x7f5ccc06d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:49.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.578+0000 7f5cefbec700 1 -- 172.21.15.16:0/3125091117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ce806e570 msgr2=0x7f5ce80cbd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:49.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.578+0000 7f5cefbec700 1 --2- 172.21.15.16:0/3125091117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ce806e570 0x7f5ce80cbd00 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7f5cd8001eb0 tx=0x7f5cd800c7b0 comp rx=0 tx=0).stop 2023-10-16T12:30:49.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.578+0000 7f5cefbec700 1 -- 172.21.15.16:0/3125091117 shutdown_connections 2023-10-16T12:30:49.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.578+0000 7f5cefbec700 1 --2- 172.21.15.16:0/3125091117 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f5ccc06b0b0 0x7f5ccc06d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:49.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.578+0000 7f5cefbec700 1 --2- 172.21.15.16:0/3125091117 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5ce815b2c0 0x7f5ce81563c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:49.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.578+0000 7f5cefbec700 1 --2- 172.21.15.16:0/3125091117 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5ce806a7e0 0x7f5ce815e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:49.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.578+0000 7f5cefbec700 1 --2- 172.21.15.16:0/3125091117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ce806e570 0x7f5ce80cbd00 unknown :-1 s=CLOSED pgs=1651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:49.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.578+0000 7f5cefbec700 1 -- 172.21.15.16:0/3125091117 >> 172.21.15.16:0/3125091117 conn(0x7f5ce80b1f40 msgr2=0x7f5ce80cd5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:49.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.578+0000 7f5cefbec700 1 -- 172.21.15.16:0/3125091117 shutdown_connections 2023-10-16T12:30:49.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.578+0000 7f5cefbec700 1 -- 172.21.15.16:0/3125091117 wait complete. 2023-10-16T12:30:49.598 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:30:49.600 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:30:49.599+0000 7f3b079a5200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T12:30:49.635 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:49.632+0000 7f6bd6932200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T12:30:49.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.721+0000 7fcd28530700 1 Processor -- start 2023-10-16T12:30:49.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.721+0000 7fcd28530700 1 -- start start 2023-10-16T12:30:49.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.722+0000 7fcd28530700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcd2015db40 0x7fcd200cb980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:49.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.722+0000 7fcd28530700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcd200ce870 0x7fcd2006ac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:49.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.722+0000 7fcd28530700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd200cedb0 con 0x7fcd2006a7e0 2023-10-16T12:30:49.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.722+0000 7fcd28530700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd2006b1a0 con 0x7fcd200ce870 2023-10-16T12:30:49.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.722+0000 7fcd28530700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd2015df20 con 0x7fcd2015db40 2023-10-16T12:30:49.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.722+0000 7fcd26acd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcd200ce870 0x7fcd2006ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:49.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.722+0000 7fcd25acb700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fcd2006a7e0 0x7fcd200ce160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35566/0 (socket says 172.21.15.16:35566) 2023-10-16T12:30:49.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.722+0000 7fcd25acb700 1 -- 172.21.15.16:0/2860526756 learned_addr learned my addr 172.21.15.16:0/2860526756 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:30:49.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.722+0000 7fcd262cc700 1 --2- 172.21.15.16:0/2860526756 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcd2015db40 0x7fcd200cb980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:49.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.722+0000 7fcd252ca700 1 -- 172.21.15.16:0/2860526756 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 765496986 0 0) 0x7fcd200cedb0 con 0x7fcd2006a7e0 2023-10-16T12:30:49.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.722+0000 7fcd252ca700 1 -- 172.21.15.16:0/2860526756 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd0c0035b0 con 0x7fcd2006a7e0 2023-10-16T12:30:49.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.722+0000 7fcd26acd700 1 -- 172.21.15.16:0/2860526756 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcd2015db40 msgr2=0x7fcd200cb980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:49.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.722+0000 7fcd26acd700 1 --2- 172.21.15.16:0/2860526756 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcd2015db40 0x7fcd200cb980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:49.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.722+0000 7fcd26acd700 1 -- 172.21.15.16:0/2860526756 >> v1:172.21.15.16:6789/0 conn(0x7fcd2006a7e0 legacy=0x7fcd200ce160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:49.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.722+0000 7fcd26acd700 1 -- 172.21.15.16:0/2860526756 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd200d12c0 con 0x7fcd200ce870 2023-10-16T12:30:49.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.722+0000 7fcd26acd700 1 --2- 172.21.15.16:0/2860526756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd200ce870 0x7fcd2006ac60 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7fcd18009d90 tx=0x7fcd1800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6d472eed03cf2f0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:49.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.723+0000 7fcd252ca700 1 -- 172.21.15.16:0/2860526756 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd1800e070 con 0x7fcd200ce870 2023-10-16T12:30:49.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.723+0000 7fcd252ca700 1 -- 172.21.15.16:0/2860526756 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd1800bb20 con 0x7fcd200ce870 2023-10-16T12:30:49.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.723+0000 7fcd252ca700 1 -- 172.21.15.16:0/2860526756 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd18012430 con 0x7fcd200ce870 2023-10-16T12:30:49.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.723+0000 7fcd28530700 1 -- 172.21.15.16:0/2860526756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd200ce870 msgr2=0x7fcd2006ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:49.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.723+0000 7fcd28530700 1 --2- 172.21.15.16:0/2860526756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd200ce870 0x7fcd2006ac60 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7fcd18009d90 tx=0x7fcd1800b330 comp rx=0 tx=0).stop 2023-10-16T12:30:49.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.723+0000 7fcd28530700 1 -- 172.21.15.16:0/2860526756 shutdown_connections 2023-10-16T12:30:49.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.723+0000 7fcd28530700 1 --2- 172.21.15.16:0/2860526756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd200ce870 0x7fcd2006ac60 unknown :-1 s=CLOSED pgs=1653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:49.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.723+0000 7fcd28530700 1 --2- 172.21.15.16:0/2860526756 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcd2015db40 0x7fcd200cb980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:49.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.723+0000 7fcd28530700 1 -- 172.21.15.16:0/2860526756 >> 172.21.15.16:0/2860526756 conn(0x7fcd200b1fd0 msgr2=0x7fcd200b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:49.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.723+0000 7fcd28530700 1 -- 172.21.15.16:0/2860526756 shutdown_connections 2023-10-16T12:30:49.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.723+0000 7fcd28530700 1 -- 172.21.15.16:0/2860526756 wait complete. 2023-10-16T12:30:49.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.725+0000 7fcd28530700 1 Processor -- start 2023-10-16T12:30:49.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.725+0000 7fcd28530700 1 -- start start 2023-10-16T12:30:49.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.725+0000 7fcd28530700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd2006a7e0 0x7fcd200c8f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:49.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.725+0000 7fcd28530700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcd200ce870 0x7fcd20157570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:49.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.725+0000 7fcd28530700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcd2015db40 0x7fcd200c7510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:49.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.725+0000 7fcd28530700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd20157ab0 con 0x7fcd2006a7e0 2023-10-16T12:30:49.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.725+0000 7fcd28530700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd200c7ed0 con 0x7fcd200ce870 2023-10-16T12:30:49.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.725+0000 7fcd28530700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd200c81b0 con 0x7fcd2015db40 2023-10-16T12:30:49.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.726+0000 7fcd262cc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd2006a7e0 0x7fcd200c8f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:49.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.726+0000 7fcd262cc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd2006a7e0 0x7fcd200c8f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46296/0 (socket says 172.21.15.16:46296) 2023-10-16T12:30:49.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.726+0000 7fcd262cc700 1 -- 172.21.15.16:0/3872416940 learned_addr learned my addr 172.21.15.16:0/3872416940 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:49.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.726+0000 7fcd25acb700 1 --2- 172.21.15.16:0/3872416940 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcd200ce870 0x7fcd20157570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:49.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.726+0000 7fcd26acd700 1 --2- 172.21.15.16:0/3872416940 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcd2015db40 0x7fcd200c7510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:49.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.726+0000 7fcd262cc700 1 -- 172.21.15.16:0/3872416940 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcd2015db40 msgr2=0x7fcd200c7510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:49.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.726+0000 7fcd262cc700 1 --2- 172.21.15.16:0/3872416940 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcd2015db40 0x7fcd200c7510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:49.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.726+0000 7fcd262cc700 1 -- 172.21.15.16:0/3872416940 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcd200ce870 msgr2=0x7fcd20157570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:49.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.726+0000 7fcd262cc700 1 --2- 172.21.15.16:0/3872416940 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcd200ce870 0x7fcd20157570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:49.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.726+0000 7fcd262cc700 1 -- 172.21.15.16:0/3872416940 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd18009a70 con 0x7fcd2006a7e0 2023-10-16T12:30:49.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.726+0000 7fcd262cc700 1 --2- 172.21.15.16:0/3872416940 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd2006a7e0 0x7fcd200c8f60 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c00e9b0 tx=0x7fcd1c00ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:49.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.726+0000 7fcd16ffd700 1 -- 172.21.15.16:0/3872416940 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd1c009710 con 0x7fcd2006a7e0 2023-10-16T12:30:49.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.726+0000 7fcd16ffd700 1 -- 172.21.15.16:0/3872416940 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd1c00ee50 con 0x7fcd2006a7e0 2023-10-16T12:30:49.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.726+0000 7fcd28530700 1 -- 172.21.15.16:0/3872416940 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd200c8490 con 0x7fcd2006a7e0 2023-10-16T12:30:49.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.726+0000 7fcd16ffd700 1 -- 172.21.15.16:0/3872416940 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd1c00ca50 con 0x7fcd2006a7e0 2023-10-16T12:30:49.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.726+0000 7fcd28530700 1 -- 172.21.15.16:0/3872416940 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd20157da0 con 0x7fcd2006a7e0 2023-10-16T12:30:49.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.728+0000 7fcd28530700 1 -- 172.21.15.16:0/3872416940 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd2004ebb0 con 0x7fcd2006a7e0 2023-10-16T12:30:49.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.728+0000 7fcd16ffd700 1 -- 172.21.15.16:0/3872416940 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 81) v1 ==== 89388+0+0 (secure 0 0 0) 0x7fcd1c00cbb0 con 0x7fcd2006a7e0 2023-10-16T12:30:49.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.728+0000 7fcd16ffd700 1 --2- 172.21.15.16:0/3872416940 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7fcd0406b0b0 0x7fcd0406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:49.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.728+0000 7fcd25acb700 1 -- 172.21.15.16:0/3872416940 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7fcd0406b0b0 msgr2=0x7fcd0406d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3783161600 2023-10-16T12:30:49.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.728+0000 7fcd16ffd700 1 -- 172.21.15.16:0/3872416940 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6429+0+0 (secure 0 0 0) 0x7fcd1c089b50 con 0x7fcd2006a7e0 2023-10-16T12:30:49.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.728+0000 7fcd25acb700 1 --2- 172.21.15.16:0/3872416940 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7fcd0406b0b0 0x7fcd0406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:30:49.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.733+0000 7fcd16ffd700 1 -- 172.21.15.16:0/3872416940 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcd1c054740 con 0x7fcd2006a7e0 2023-10-16T12:30:49.762 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:30:49.761+0000 7f3b079a5200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T12:30:49.769 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:49.767+0000 7f6bd6932200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T12:30:49.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.929+0000 7fcd25acb700 1 -- 172.21.15.16:0/3872416940 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7fcd0406b0b0 msgr2=0x7fcd0406d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3783161600 2023-10-16T12:30:49.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.929+0000 7fcd25acb700 1 --2- 172.21.15.16:0/3872416940 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7fcd0406b0b0 0x7fcd0406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:30:49.937 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:49.935+0000 7f6bd6932200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T12:30:49.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.995+0000 7fcd28530700 1 -- 172.21.15.16:0/3872416940 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcd200cbf70 con 0x7fcd2006a7e0 2023-10-16T12:30:49.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.996+0000 7fcd16ffd700 1 -- 172.21.15.16:0/3872416940 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+4915 (secure 0 0 0) 0x7fcd1c021090 con 0x7fcd2006a7e0 2023-10-16T12:30:49.999 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:30:49.999 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":70,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7220,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/4274593615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":4274593615},{"type":"v1","addr":"172.21.15.16:6837","nonce":4274593615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":8625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/1624531381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1624531381},{"type":"v1","addr":"172.21.15.16:6836","nonce":1624531381}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":8637,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/1629597454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":1629597454},{"type":"v1","addr":"172.21.15.49:6834","nonce":1629597454}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:30:22.807943+0000","modified":"2023-10-16T12:30:25.836865+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8640},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8640":{"gid":8640,"name":"b","rank":0,"incarnation":66,"state":"up:active","state_seq":3,"addr":"172.21.15.49:6835/2008024326","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":2008024326},{"type":"v1","addr":"172.21.15.49:6835","nonce":2008024326}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2023-10-16T12:30:50.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.998+0000 7fcd28530700 1 -- 172.21.15.16:0/3872416940 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7fcd0406b0b0 msgr2=0x7fcd0406d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:30:50.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.998+0000 7fcd28530700 1 --2- 172.21.15.16:0/3872416940 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7fcd0406b0b0 0x7fcd0406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:50.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.998+0000 7fcd28530700 1 -- 172.21.15.16:0/3872416940 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd2006a7e0 msgr2=0x7fcd200c8f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:50.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.998+0000 7fcd28530700 1 --2- 172.21.15.16:0/3872416940 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd2006a7e0 0x7fcd200c8f60 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c00e9b0 tx=0x7fcd1c00ecc0 comp rx=0 tx=0).stop 2023-10-16T12:30:50.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.998+0000 7fcd28530700 1 -- 172.21.15.16:0/3872416940 shutdown_connections 2023-10-16T12:30:50.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.998+0000 7fcd28530700 1 --2- 172.21.15.16:0/3872416940 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7fcd0406b0b0 0x7fcd0406d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:50.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.998+0000 7fcd28530700 1 --2- 172.21.15.16:0/3872416940 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcd2015db40 0x7fcd200c7510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:50.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.998+0000 7fcd28530700 1 --2- 172.21.15.16:0/3872416940 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd2006a7e0 0x7fcd200c8f60 unknown :-1 s=CLOSED pgs=1654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:50.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.998+0000 7fcd28530700 1 --2- 172.21.15.16:0/3872416940 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcd200ce870 0x7fcd20157570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:50.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.998+0000 7fcd28530700 1 -- 172.21.15.16:0/3872416940 >> 172.21.15.16:0/3872416940 conn(0x7fcd200b1fd0 msgr2=0x7fcd2015a890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:50.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.998+0000 7fcd28530700 1 -- 172.21.15.16:0/3872416940 shutdown_connections 2023-10-16T12:30:50.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:49.998+0000 7fcd28530700 1 -- 172.21.15.16:0/3872416940 wait complete. 2023-10-16T12:30:50.004 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 70 2023-10-16T12:30:50.015 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:30:50.034 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:30:50.032+0000 7f3b079a5200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T12:30:50.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.180+0000 7f3102421700 1 Processor -- start 2023-10-16T12:30:50.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.180+0000 7f3102421700 1 -- start start 2023-10-16T12:30:50.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.181+0000 7f3102421700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f30fc0ce380 0x7f30fc0d0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:50.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.181+0000 7f3102421700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30fc0d1210 con 0x7f30fc0b2b50 2023-10-16T12:30:50.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.181+0000 7f3102421700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30fc0cbc30 con 0x7f30fc0cdfa0 2023-10-16T12:30:50.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.181+0000 7f3102421700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30fc0cbf10 con 0x7f30fc0ce380 2023-10-16T12:30:50.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.181+0000 7f31009be700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f30fc0ce380 0x7f30fc0d0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:50.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.181+0000 7f30fb7fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f30fc0b2b50 0x7f30fc0d04a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35580/0 (socket says 172.21.15.16:35580) 2023-10-16T12:30:50.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.181+0000 7f30fb7fe700 1 -- 172.21.15.16:0/4240000632 learned_addr learned my addr 172.21.15.16:0/4240000632 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:30:50.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.181+0000 7f31009be700 1 -- 172.21.15.16:0/4240000632 >> v1:172.21.15.49:6789/0 conn(0x7f30fc0cdfa0 legacy=0x7f30fc0b2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:50.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.181+0000 7f31009be700 1 -- 172.21.15.16:0/4240000632 >> v1:172.21.15.16:6789/0 conn(0x7f30fc0b2b50 legacy=0x7f30fc0d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:50.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.181+0000 7f31009be700 1 -- 172.21.15.16:0/4240000632 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30fc0cc1f0 con 0x7f30fc0ce380 2023-10-16T12:30:50.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.181+0000 7f30faffd700 1 -- 172.21.15.16:0/4240000632 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1363759844 0 0) 0x7f30fc0d1210 con 0x7f30fc0b2b50 2023-10-16T12:30:50.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.181+0000 7f31009be700 1 --2- 172.21.15.16:0/4240000632 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30fc0ce380 0x7f30fc0d0bb0 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7f30f4009d90 tx=0x7f30f400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=475a687ae39a7458 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:50.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.181+0000 7f30faffd700 1 -- 172.21.15.16:0/4240000632 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30f400e070 con 0x7f30fc0ce380 2023-10-16T12:30:50.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.182+0000 7f30faffd700 1 -- 172.21.15.16:0/4240000632 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30f400bbd0 con 0x7f30fc0ce380 2023-10-16T12:30:50.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.182+0000 7f30faffd700 1 -- 172.21.15.16:0/4240000632 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30f4012430 con 0x7f30fc0ce380 2023-10-16T12:30:50.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.182+0000 7f3102421700 1 -- 172.21.15.16:0/4240000632 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30fc0ce380 msgr2=0x7f30fc0d0bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:50.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.182+0000 7f3102421700 1 --2- 172.21.15.16:0/4240000632 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30fc0ce380 0x7f30fc0d0bb0 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7f30f4009d90 tx=0x7f30f400b330 comp rx=0 tx=0).stop 2023-10-16T12:30:50.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.182+0000 7f3102421700 1 -- 172.21.15.16:0/4240000632 shutdown_connections 2023-10-16T12:30:50.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.182+0000 7f3102421700 1 --2- 172.21.15.16:0/4240000632 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30fc0ce380 0x7f30fc0d0bb0 unknown :-1 s=CLOSED pgs=1656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:50.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.182+0000 7f3102421700 1 -- 172.21.15.16:0/4240000632 >> 172.21.15.16:0/4240000632 conn(0x7f30fc0d14b0 msgr2=0x7f30fc0d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:50.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.182+0000 7f3102421700 1 -- 172.21.15.16:0/4240000632 shutdown_connections 2023-10-16T12:30:50.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.182+0000 7f3102421700 1 -- 172.21.15.16:0/4240000632 wait complete. 2023-10-16T12:30:50.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.184+0000 7f3102421700 1 Processor -- start 2023-10-16T12:30:50.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.184+0000 7f3102421700 1 -- start start 2023-10-16T12:30:50.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.185+0000 7f3102421700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f30fc0b2b50 0x7f30fc14f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:50.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.185+0000 7f3102421700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30fc0cdfa0 0x7f30fc151fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:50.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.185+0000 7f3102421700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f30fc0ce380 0x7f30fc1f27d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:50.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.185+0000 7f3102421700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30fc1507f0 con 0x7f30fc0cdfa0 2023-10-16T12:30:50.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.185+0000 7f3102421700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30fc14f260 con 0x7f30fc0b2b50 2023-10-16T12:30:50.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.185+0000 7f3102421700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30fc152520 con 0x7f30fc0ce380 2023-10-16T12:30:50.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.185+0000 7f30fb7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30fc0cdfa0 0x7f30fc151fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:50.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.185+0000 7f30fb7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30fc0cdfa0 0x7f30fc151fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46310/0 (socket says 172.21.15.16:46310) 2023-10-16T12:30:50.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.185+0000 7f30fb7fe700 1 -- 172.21.15.16:0/3397835925 learned_addr learned my addr 172.21.15.16:0/3397835925 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:50.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.185+0000 7f30fbfff700 1 --2- 172.21.15.16:0/3397835925 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f30fc0b2b50 0x7f30fc14f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:50.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.185+0000 7f31009be700 1 --2- 172.21.15.16:0/3397835925 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f30fc0ce380 0x7f30fc1f27d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:50.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.186+0000 7f30fb7fe700 1 -- 172.21.15.16:0/3397835925 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f30fc0ce380 msgr2=0x7f30fc1f27d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:50.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.186+0000 7f30fb7fe700 1 --2- 172.21.15.16:0/3397835925 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f30fc0ce380 0x7f30fc1f27d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:50.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.186+0000 7f30fb7fe700 1 -- 172.21.15.16:0/3397835925 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f30fc0b2b50 msgr2=0x7f30fc14f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:50.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.186+0000 7f30fb7fe700 1 --2- 172.21.15.16:0/3397835925 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f30fc0b2b50 0x7f30fc14f520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:50.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.186+0000 7f30fb7fe700 1 -- 172.21.15.16:0/3397835925 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30f4009a70 con 0x7f30fc0cdfa0 2023-10-16T12:30:50.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.186+0000 7f30fbfff700 1 --2- 172.21.15.16:0/3397835925 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f30fc0b2b50 0x7f30fc14f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:30:50.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.186+0000 7f30fb7fe700 1 --2- 172.21.15.16:0/3397835925 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30fc0cdfa0 0x7f30fc151fe0 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7f30f000c520 tx=0x7f30f000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:50.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.186+0000 7f30f8ff9700 1 -- 172.21.15.16:0/3397835925 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30f0004c80 con 0x7f30fc0cdfa0 2023-10-16T12:30:50.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.186+0000 7f30f8ff9700 1 -- 172.21.15.16:0/3397835925 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30f0004e00 con 0x7f30fc0cdfa0 2023-10-16T12:30:50.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.186+0000 7f3102421700 1 -- 172.21.15.16:0/3397835925 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30fc14fbe0 con 0x7f30fc0cdfa0 2023-10-16T12:30:50.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.186+0000 7f30f8ff9700 1 -- 172.21.15.16:0/3397835925 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30f0007310 con 0x7f30fc0cdfa0 2023-10-16T12:30:50.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.186+0000 7f3102421700 1 -- 172.21.15.16:0/3397835925 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30fc14ff70 con 0x7f30fc0cdfa0 2023-10-16T12:30:50.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.187+0000 7f3102421700 1 -- 172.21.15.16:0/3397835925 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30fc1509a0 con 0x7f30fc0cdfa0 2023-10-16T12:30:50.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.188+0000 7f30f8ff9700 1 -- 172.21.15.16:0/3397835925 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 81) v1 ==== 89388+0+0 (secure 0 0 0) 0x7f30f0007470 con 0x7f30fc0cdfa0 2023-10-16T12:30:50.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.188+0000 7f30f8ff9700 1 --2- 172.21.15.16:0/3397835925 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f30e806b0b0 0x7f30e806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:50.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.188+0000 7f30fbfff700 1 -- 172.21.15.16:0/3397835925 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f30e806b0b0 msgr2=0x7f30e806d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3783161600 2023-10-16T12:30:50.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.188+0000 7f30f8ff9700 1 -- 172.21.15.16:0/3397835925 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6429+0+0 (secure 0 0 0) 0x7f30f0089710 con 0x7f30fc0cdfa0 2023-10-16T12:30:50.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.188+0000 7f30fbfff700 1 --2- 172.21.15.16:0/3397835925 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f30e806b0b0 0x7f30e806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:30:50.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.197+0000 7f30f8ff9700 1 -- 172.21.15.16:0/3397835925 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f30fc1509a0 con 0x7f30fc0cdfa0 2023-10-16T12:30:50.202 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:30:50.200+0000 7f3b079a5200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T12:30:50.221 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:50.218+0000 7f6bd6932200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T12:30:50.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.389+0000 7f30fbfff700 1 -- 172.21.15.16:0/3397835925 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f30e806b0b0 msgr2=0x7f30e806d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3783161600 2023-10-16T12:30:50.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.389+0000 7f30fbfff700 1 --2- 172.21.15.16:0/3397835925 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f30e806b0b0 0x7f30e806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:30:50.394 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:50.391+0000 7f6bd6932200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T12:30:50.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.401+0000 7f3102421700 1 -- 172.21.15.16:0/3397835925 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f30fc04ebb0 con 0x7f30fc0cdfa0 2023-10-16T12:30:50.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.402+0000 7f30f8ff9700 1 -- 172.21.15.16:0/3397835925 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v143) v1 ==== 74+0+17263 (secure 0 0 0) 0x7f30f0054300 con 0x7f30fc0cdfa0 2023-10-16T12:30:50.406 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:30:50.406 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":143,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:30:37.941062+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:30:21.318498+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"141","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2023-10-16T12:30:21.813382+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"142","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/2901217010":"2023-10-17T12:30:37.941028+0000","172.21.15.16:6833/258291157":"2023-10-17T12:30:37.941028+0000","172.21.15.16:6832/258291157":"2023-10-17T12:30:37.941028+0000","172.21.15.16:0/4123017371":"2023-10-17T12:30:37.941028+0000","172.21.15.16:0/2703050386":"2023-10-17T12:30:37.941028+0000","172.21.15.16:0/3188488837":"2023-10-17T12:30:37.941028+0000","172.21.15.16:0/2621298501":"2023-10-17T12:30:37.941028+0000","172.21.15.16:0/1781882931":"2023-10-17T12:30:37.941028+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:30:50.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.405+0000 7f3102421700 1 -- 172.21.15.16:0/3397835925 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f30e806b0b0 msgr2=0x7f30e806d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:30:50.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.405+0000 7f3102421700 1 --2- 172.21.15.16:0/3397835925 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f30e806b0b0 0x7f30e806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:50.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.405+0000 7f3102421700 1 -- 172.21.15.16:0/3397835925 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30fc0cdfa0 msgr2=0x7f30fc151fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:50.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.405+0000 7f3102421700 1 --2- 172.21.15.16:0/3397835925 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30fc0cdfa0 0x7f30fc151fe0 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7f30f000c520 tx=0x7f30f000c830 comp rx=0 tx=0).stop 2023-10-16T12:30:50.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.405+0000 7f3102421700 1 -- 172.21.15.16:0/3397835925 shutdown_connections 2023-10-16T12:30:50.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.405+0000 7f3102421700 1 --2- 172.21.15.16:0/3397835925 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f30e806b0b0 0x7f30e806d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:50.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.405+0000 7f3102421700 1 --2- 172.21.15.16:0/3397835925 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f30fc0ce380 0x7f30fc1f27d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:50.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.405+0000 7f3102421700 1 --2- 172.21.15.16:0/3397835925 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f30fc0b2b50 0x7f30fc14f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:50.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.405+0000 7f3102421700 1 --2- 172.21.15.16:0/3397835925 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30fc0cdfa0 0x7f30fc151fe0 unknown :-1 s=CLOSED pgs=1657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:50.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.405+0000 7f3102421700 1 -- 172.21.15.16:0/3397835925 >> 172.21.15.16:0/3397835925 conn(0x7f30fc0d14b0 msgr2=0x7f30fc0b2400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:50.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.405+0000 7f3102421700 1 -- 172.21.15.16:0/3397835925 shutdown_connections 2023-10-16T12:30:50.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.405+0000 7f3102421700 1 -- 172.21.15.16:0/3397835925 wait complete. 2023-10-16T12:30:50.421 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-16T12:30:50.421 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:30:50.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.546+0000 7f6bb649a700 1 Processor -- start 2023-10-16T12:30:50.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.546+0000 7f6bb649a700 1 -- start start 2023-10-16T12:30:50.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.546+0000 7f6bb649a700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f6bb00b2b50 0x7f6bb00d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:50.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.546+0000 7f6bb649a700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bb00d1210 con 0x7f6bb00cdfa0 2023-10-16T12:30:50.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.546+0000 7f6bb649a700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bb00cbc30 con 0x7f6bb00ce380 2023-10-16T12:30:50.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.546+0000 7f6bb649a700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bb00cbf10 con 0x7f6bb00b2b50 2023-10-16T12:30:50.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.546+0000 7f6baffff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f6bb00cdfa0 0x7f6bb00b2440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35594/0 (socket says 172.21.15.16:35594) 2023-10-16T12:30:50.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.546+0000 7f6baffff700 1 -- 172.21.15.16:0/3125012417 learned_addr learned my addr 172.21.15.16:0/3125012417 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:30:50.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.546+0000 7f6baf7fe700 1 --2- 172.21.15.16:0/3125012417 >> v2:172.21.15.49:3300/0 conn(0x7f6bb00b2b50 0x7f6bb00d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:50.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.546+0000 7f6baeffd700 1 -- 172.21.15.16:0/3125012417 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 440619515 0 0) 0x7f6bb00d1210 con 0x7f6bb00cdfa0 2023-10-16T12:30:50.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.546+0000 7f6baeffd700 1 -- 172.21.15.16:0/3125012417 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b980035b0 con 0x7f6bb00cdfa0 2023-10-16T12:30:50.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.546+0000 7f6baeffd700 1 -- 172.21.15.16:0/3125012417 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2448497007 0 0) 0x7f6bb00cbc30 con 0x7f6bb00ce380 2023-10-16T12:30:50.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.546+0000 7f6baeffd700 1 -- 172.21.15.16:0/3125012417 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bb00d1210 con 0x7f6bb00ce380 2023-10-16T12:30:50.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.546+0000 7f6baf7fe700 1 -- 172.21.15.16:0/3125012417 >> v1:172.21.15.49:6789/0 conn(0x7f6bb00ce380 legacy=0x7f6bb00d09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:50.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.546+0000 7f6baf7fe700 1 -- 172.21.15.16:0/3125012417 >> v1:172.21.15.16:6789/0 conn(0x7f6bb00cdfa0 legacy=0x7f6bb00b2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:50.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.546+0000 7f6baf7fe700 1 -- 172.21.15.16:0/3125012417 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bb00cc1f0 con 0x7f6bb00b2b50 2023-10-16T12:30:50.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.546+0000 7f6baf7fe700 1 --2- 172.21.15.16:0/3125012417 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6bb00b2b50 0x7f6bb00d04a0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f6ba4009840 tx=0x7f6ba4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3e327468cc282473 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:50.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.547+0000 7f6baeffd700 1 -- 172.21.15.16:0/3125012417 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba4011830 con 0x7f6bb00b2b50 2023-10-16T12:30:50.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.547+0000 7f6baeffd700 1 -- 172.21.15.16:0/3125012417 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ba4011990 con 0x7f6bb00b2b50 2023-10-16T12:30:50.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.547+0000 7f6baeffd700 1 -- 172.21.15.16:0/3125012417 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba4011d20 con 0x7f6bb00b2b50 2023-10-16T12:30:50.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.547+0000 7f6bb649a700 1 -- 172.21.15.16:0/3125012417 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6bb00b2b50 msgr2=0x7f6bb00d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:50.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.547+0000 7f6bb649a700 1 --2- 172.21.15.16:0/3125012417 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6bb00b2b50 0x7f6bb00d04a0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f6ba4009840 tx=0x7f6ba4009b50 comp rx=0 tx=0).stop 2023-10-16T12:30:50.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.547+0000 7f6bb649a700 1 -- 172.21.15.16:0/3125012417 shutdown_connections 2023-10-16T12:30:50.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.547+0000 7f6bb649a700 1 --2- 172.21.15.16:0/3125012417 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6bb00b2b50 0x7f6bb00d04a0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:50.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.547+0000 7f6bb649a700 1 -- 172.21.15.16:0/3125012417 >> 172.21.15.16:0/3125012417 conn(0x7f6bb00d14b0 msgr2=0x7f6bb00d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:50.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.547+0000 7f6bb649a700 1 -- 172.21.15.16:0/3125012417 shutdown_connections 2023-10-16T12:30:50.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.547+0000 7f6bb649a700 1 -- 172.21.15.16:0/3125012417 wait complete. 2023-10-16T12:30:50.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.548+0000 7f6bb649a700 1 Processor -- start 2023-10-16T12:30:50.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.549+0000 7f6bb649a700 1 -- start start 2023-10-16T12:30:50.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.549+0000 7f6bb649a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6bb00b2b50 0x7f6bb015db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:50.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.549+0000 7f6bb649a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bb00cdfa0 0x7f6bb00b4660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:50.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.549+0000 7f6bb649a700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6bb00ce380 0x7f6bb00b4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:50.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.549+0000 7f6bb649a700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb00caee0 con 0x7f6bb00cdfa0 2023-10-16T12:30:50.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.549+0000 7f6bb649a700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb01f3e10 con 0x7f6bb00ce380 2023-10-16T12:30:50.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.549+0000 7f6bb649a700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb009eee0 con 0x7f6bb00b2b50 2023-10-16T12:30:50.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.549+0000 7f6baf7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bb00cdfa0 0x7f6bb00b4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:50.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.549+0000 7f6baf7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bb00cdfa0 0x7f6bb00b4660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46328/0 (socket says 172.21.15.16:46328) 2023-10-16T12:30:50.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.549+0000 7f6baf7fe700 1 -- 172.21.15.16:0/2775208593 learned_addr learned my addr 172.21.15.16:0/2775208593 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:50.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.549+0000 7f6baffff700 1 --2- 172.21.15.16:0/2775208593 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6bb00b2b50 0x7f6bb015db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:50.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.549+0000 7f6bb4a37700 1 --2- 172.21.15.16:0/2775208593 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6bb00ce380 0x7f6bb00b4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:50.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.550+0000 7f6baf7fe700 1 -- 172.21.15.16:0/2775208593 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6bb00b2b50 msgr2=0x7f6bb015db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:50.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.550+0000 7f6baf7fe700 1 --2- 172.21.15.16:0/2775208593 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6bb00b2b50 0x7f6bb015db10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:50.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.550+0000 7f6baf7fe700 1 -- 172.21.15.16:0/2775208593 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6bb00ce380 msgr2=0x7f6bb00b4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:50.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.550+0000 7f6baf7fe700 1 --2- 172.21.15.16:0/2775208593 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6bb00ce380 0x7f6bb00b4ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:50.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.550+0000 7f6baf7fe700 1 -- 172.21.15.16:0/2775208593 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ba40094a0 con 0x7f6bb00cdfa0 2023-10-16T12:30:50.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.550+0000 7f6baf7fe700 1 --2- 172.21.15.16:0/2775208593 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bb00cdfa0 0x7f6bb00b4660 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f6ba40090b0 tx=0x7f6ba4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:50.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.550+0000 7f6bacff9700 1 -- 172.21.15.16:0/2775208593 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba401a6e0 con 0x7f6bb00cdfa0 2023-10-16T12:30:50.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.550+0000 7f6bacff9700 1 -- 172.21.15.16:0/2775208593 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ba401a840 con 0x7f6bb00cdfa0 2023-10-16T12:30:50.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.550+0000 7f6bb649a700 1 -- 172.21.15.16:0/2775208593 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bb01f2df0 con 0x7f6bb00cdfa0 2023-10-16T12:30:50.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.550+0000 7f6bacff9700 1 -- 172.21.15.16:0/2775208593 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba401abb0 con 0x7f6bb00cdfa0 2023-10-16T12:30:50.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.550+0000 7f6bb649a700 1 -- 172.21.15.16:0/2775208593 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bb01f3050 con 0x7f6bb00cdfa0 2023-10-16T12:30:50.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.551+0000 7f6bb649a700 1 -- 172.21.15.16:0/2775208593 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bb004ebb0 con 0x7f6bb00cdfa0 2023-10-16T12:30:50.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.552+0000 7f6bacff9700 1 -- 172.21.15.16:0/2775208593 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 81) v1 ==== 89388+0+0 (secure 0 0 0) 0x7f6ba4022070 con 0x7f6bb00cdfa0 2023-10-16T12:30:50.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.553+0000 7f6bacff9700 1 --2- 172.21.15.16:0/2775208593 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f6b9c06b0b0 0x7f6b9c06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:50.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.553+0000 7f6baffff700 1 -- 172.21.15.16:0/2775208593 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f6b9c06b0b0 msgr2=0x7f6b9c06d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3783161600 2023-10-16T12:30:50.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.553+0000 7f6bacff9700 1 -- 172.21.15.16:0/2775208593 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6429+0+0 (secure 0 0 0) 0x7f6ba408ad20 con 0x7f6bb00cdfa0 2023-10-16T12:30:50.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.553+0000 7f6baffff700 1 --2- 172.21.15.16:0/2775208593 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f6b9c06b0b0 0x7f6b9c06d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:30:50.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.558+0000 7f6bacff9700 1 -- 172.21.15.16:0/2775208593 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6ba4055990 con 0x7f6bb00cdfa0 2023-10-16T12:30:50.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.754+0000 7f6baffff700 1 -- 172.21.15.16:0/2775208593 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f6b9c06b0b0 msgr2=0x7f6b9c06d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3783161600 2023-10-16T12:30:50.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.754+0000 7f6baffff700 1 --2- 172.21.15.16:0/2775208593 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f6b9c06b0b0 0x7f6b9c06d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:30:50.761 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:30:50.759+0000 7f3b079a5200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T12:30:50.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.808+0000 7f6bb649a700 1 -- 172.21.15.16:0/2775208593 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6bb00cb870 con 0x7f6bb00cdfa0 2023-10-16T12:30:50.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.809+0000 7f6bacff9700 1 -- 172.21.15.16:0/2775208593 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+4915 (secure 0 0 0) 0x7f6ba4029020 con 0x7f6bb00cdfa0 2023-10-16T12:30:50.812 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:30:50.812 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":70,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7220,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/4274593615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":4274593615},{"type":"v1","addr":"172.21.15.16:6837","nonce":4274593615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":8625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/1624531381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1624531381},{"type":"v1","addr":"172.21.15.16:6836","nonce":1624531381}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":8637,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/1629597454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":1629597454},{"type":"v1","addr":"172.21.15.49:6834","nonce":1629597454}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:30:22.807943+0000","modified":"2023-10-16T12:30:25.836865+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8640},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8640":{"gid":8640,"name":"b","rank":0,"incarnation":66,"state":"up:active","state_seq":3,"addr":"172.21.15.49:6835/2008024326","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":2008024326},{"type":"v1","addr":"172.21.15.49:6835","nonce":2008024326}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2023-10-16T12:30:50.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.811+0000 7f6bb649a700 1 -- 172.21.15.16:0/2775208593 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f6b9c06b0b0 msgr2=0x7f6b9c06d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:30:50.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.811+0000 7f6bb649a700 1 --2- 172.21.15.16:0/2775208593 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f6b9c06b0b0 0x7f6b9c06d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:50.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.811+0000 7f6bb649a700 1 -- 172.21.15.16:0/2775208593 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bb00cdfa0 msgr2=0x7f6bb00b4660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:50.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.811+0000 7f6bb649a700 1 --2- 172.21.15.16:0/2775208593 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bb00cdfa0 0x7f6bb00b4660 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f6ba40090b0 tx=0x7f6ba4004750 comp rx=0 tx=0).stop 2023-10-16T12:30:50.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.811+0000 7f6bb649a700 1 -- 172.21.15.16:0/2775208593 shutdown_connections 2023-10-16T12:30:50.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.811+0000 7f6bb649a700 1 --2- 172.21.15.16:0/2775208593 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7f6b9c06b0b0 0x7f6b9c06d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:50.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.811+0000 7f6bb649a700 1 --2- 172.21.15.16:0/2775208593 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6bb00ce380 0x7f6bb00b4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:50.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.811+0000 7f6bb649a700 1 --2- 172.21.15.16:0/2775208593 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6bb00b2b50 0x7f6bb015db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:50.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.811+0000 7f6bb649a700 1 --2- 172.21.15.16:0/2775208593 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bb00cdfa0 0x7f6bb00b4660 unknown :-1 s=CLOSED pgs=1659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:50.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.811+0000 7f6bb649a700 1 -- 172.21.15.16:0/2775208593 >> 172.21.15.16:0/2775208593 conn(0x7f6bb00d14b0 msgr2=0x7f6bb006eba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:50.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.811+0000 7f6bb649a700 1 -- 172.21.15.16:0/2775208593 shutdown_connections 2023-10-16T12:30:50.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.811+0000 7f6bb649a700 1 -- 172.21.15.16:0/2775208593 wait complete. 2023-10-16T12:30:50.817 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 70 2023-10-16T12:30:50.827 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:30:50.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.951+0000 7fa087207700 1 Processor -- start 2023-10-16T12:30:50.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.951+0000 7fa087207700 1 -- start start 2023-10-16T12:30:50.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.951+0000 7fa087207700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa0800d4380 0x7fa0800d2200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:50.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.951+0000 7fa087207700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0800d1d70 con 0x7fa080160d40 2023-10-16T12:30:50.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.951+0000 7fa087207700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0800cf6f0 con 0x7fa0800cffa0 2023-10-16T12:30:50.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.951+0000 7fa087207700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0800cf500 con 0x7fa0800d4380 2023-10-16T12:30:50.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.952+0000 7fa084fa3700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa080160d40 0x7fa0800cedf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35610/0 (socket says 172.21.15.16:35610) 2023-10-16T12:30:50.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.952+0000 7fa084fa3700 1 -- 172.21.15.16:0/929848467 learned_addr learned my addr 172.21.15.16:0/929848467 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:30:50.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.952+0000 7fa07f7fe700 1 -- 172.21.15.16:0/929848467 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1013876506 0 0) 0x7fa0800d1d70 con 0x7fa080160d40 2023-10-16T12:30:50.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.952+0000 7fa07f7fe700 1 -- 172.21.15.16:0/929848467 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0700035b0 con 0x7fa080160d40 2023-10-16T12:30:50.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.952+0000 7fa07ffff700 1 --2- 172.21.15.16:0/929848467 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa0800d4380 0x7fa0800d2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:50.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.952+0000 7fa07f7fe700 1 -- 172.21.15.16:0/929848467 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2217110739 0 0) 0x7fa0800cf6f0 con 0x7fa0800cffa0 2023-10-16T12:30:50.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.952+0000 7fa07f7fe700 1 -- 172.21.15.16:0/929848467 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0800d1d70 con 0x7fa0800cffa0 2023-10-16T12:30:50.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.952+0000 7fa07f7fe700 1 -- 172.21.15.16:0/929848467 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2257459431 0 0) 0x7fa0700035b0 con 0x7fa080160d40 2023-10-16T12:30:50.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.952+0000 7fa07f7fe700 1 -- 172.21.15.16:0/929848467 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa0800d4380 msgr2=0x7fa0800d2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:50.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.952+0000 7fa07f7fe700 1 --2- 172.21.15.16:0/929848467 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa0800d4380 0x7fa0800d2200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:50.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.952+0000 7fa07f7fe700 1 -- 172.21.15.16:0/929848467 >> v1:172.21.15.49:6789/0 conn(0x7fa0800cffa0 legacy=0x7fa08006b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:50.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.952+0000 7fa07f7fe700 1 -- 172.21.15.16:0/929848467 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa080161120 con 0x7fa080160d40 2023-10-16T12:30:50.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.952+0000 7fa07f7fe700 1 -- 172.21.15.16:0/929848467 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7fa0740017f0 con 0x7fa080160d40 2023-10-16T12:30:50.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.952+0000 7fa07f7fe700 1 -- 172.21.15.16:0/929848467 >> v1:172.21.15.16:6789/0 conn(0x7fa080160d40 legacy=0x7fa0800cedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:50.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.952+0000 7fa07f7fe700 1 --2- 172.21.15.16:0/929848467 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa070003bc0 0x7fa070003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:50.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.952+0000 7fa07f7fe700 1 --2- 172.21.15.16:0/929848467 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa070004850 0x7fa070004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:50.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.952+0000 7fa07f7fe700 1 --2- 172.21.15.16:0/929848467 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa0800d4380 0x7fa070005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:50.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.952+0000 7fa07f7fe700 1 -- 172.21.15.16:0/929848467 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa07000baf0 con 0x7fa070003bc0 2023-10-16T12:30:50.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.952+0000 7fa07f7fe700 1 -- 172.21.15.16:0/929848467 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa07000bc60 con 0x7fa070004850 2023-10-16T12:30:50.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.952+0000 7fa07f7fe700 1 -- 172.21.15.16:0/929848467 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa07000be20 con 0x7fa0800d4380 2023-10-16T12:30:50.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.952+0000 7fa084fa3700 1 --2- 172.21.15.16:0/929848467 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa070003bc0 0x7fa070003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:50.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.952+0000 7fa07ffff700 1 --2- 172.21.15.16:0/929848467 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa070004850 0x7fa070004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:50.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.953+0000 7fa0857a4700 1 --2- 172.21.15.16:0/929848467 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa0800d4380 0x7fa070005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:50.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.953+0000 7fa084fa3700 1 -- 172.21.15.16:0/929848467 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa0800d4380 msgr2=0x7fa070005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:50.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.953+0000 7fa084fa3700 1 --2- 172.21.15.16:0/929848467 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa0800d4380 0x7fa070005230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:50.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.953+0000 7fa084fa3700 1 -- 172.21.15.16:0/929848467 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa070004850 msgr2=0x7fa070004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:50.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.953+0000 7fa084fa3700 1 --2- 172.21.15.16:0/929848467 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa070004850 0x7fa070004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:50.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.953+0000 7fa084fa3700 1 -- 172.21.15.16:0/929848467 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0700035b0 con 0x7fa070003bc0 2023-10-16T12:30:50.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.953+0000 7fa07ffff700 1 --2- 172.21.15.16:0/929848467 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa070004850 0x7fa070004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:30:50.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.953+0000 7fa084fa3700 1 --2- 172.21.15.16:0/929848467 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa070003bc0 0x7fa070003fc0 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7fa07400c240 tx=0x7fa070005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31c435564a0364da server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:50.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.953+0000 7fa07f7fe700 1 -- 172.21.15.16:0/929848467 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa07400cd80 con 0x7fa070003bc0 2023-10-16T12:30:50.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.953+0000 7fa07f7fe700 1 -- 172.21.15.16:0/929848467 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa074001c70 con 0x7fa070003bc0 2023-10-16T12:30:50.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.953+0000 7fa07f7fe700 1 -- 172.21.15.16:0/929848467 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa074007310 con 0x7fa070003bc0 2023-10-16T12:30:50.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.954+0000 7fa087207700 1 -- 172.21.15.16:0/929848467 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa070003bc0 msgr2=0x7fa070003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:50.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.954+0000 7fa087207700 1 --2- 172.21.15.16:0/929848467 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa070003bc0 0x7fa070003fc0 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7fa07400c240 tx=0x7fa070005880 comp rx=0 tx=0).stop 2023-10-16T12:30:50.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.954+0000 7fa0857a4700 1 -- 172.21.15.16:0/929848467 reap_dead start 2023-10-16T12:30:50.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.954+0000 7fa087207700 1 -- 172.21.15.16:0/929848467 shutdown_connections 2023-10-16T12:30:50.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.954+0000 7fa087207700 1 -- 172.21.15.16:0/929848467 >> 172.21.15.16:0/929848467 conn(0x7fa08009c9f0 msgr2=0x7fa0800d3740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:50.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.954+0000 7fa087207700 1 -- 172.21.15.16:0/929848467 shutdown_connections 2023-10-16T12:30:50.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.954+0000 7fa087207700 1 -- 172.21.15.16:0/929848467 wait complete. 2023-10-16T12:30:50.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.955+0000 7fa087207700 1 Processor -- start 2023-10-16T12:30:50.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.955+0000 7fa087207700 1 -- start start 2023-10-16T12:30:50.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.956+0000 7fa087207700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa070003bc0 0x7fa0800b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:50.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.956+0000 7fa087207700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa08015a5d0 0x7fa08015aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:50.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.956+0000 7fa087207700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa0800ca810 0x7fa0800caca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:50.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.956+0000 7fa087207700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0800cb1e0 con 0x7fa070003bc0 2023-10-16T12:30:50.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.956+0000 7fa087207700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0800b2080 con 0x7fa08015a5d0 2023-10-16T12:30:50.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.956+0000 7fa087207700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0800c9950 con 0x7fa0800ca810 2023-10-16T12:30:50.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.956+0000 7fa084fa3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa070003bc0 0x7fa0800b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:50.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.956+0000 7fa084fa3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa070003bc0 0x7fa0800b1b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46348/0 (socket says 172.21.15.16:46348) 2023-10-16T12:30:50.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.956+0000 7fa084fa3700 1 -- 172.21.15.16:0/4260654880 learned_addr learned my addr 172.21.15.16:0/4260654880 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:50.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.956+0000 7fa07ffff700 1 --2- 172.21.15.16:0/4260654880 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa08015a5d0 0x7fa08015aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:50.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.956+0000 7fa0857a4700 1 --2- 172.21.15.16:0/4260654880 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa0800ca810 0x7fa0800caca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:50.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.956+0000 7fa084fa3700 1 -- 172.21.15.16:0/4260654880 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa0800ca810 msgr2=0x7fa0800caca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:50.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.956+0000 7fa084fa3700 1 --2- 172.21.15.16:0/4260654880 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa0800ca810 0x7fa0800caca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:50.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.956+0000 7fa084fa3700 1 -- 172.21.15.16:0/4260654880 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa08015a5d0 msgr2=0x7fa08015aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:50.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.956+0000 7fa084fa3700 1 --2- 172.21.15.16:0/4260654880 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa08015a5d0 0x7fa08015aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:50.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.956+0000 7fa084fa3700 1 -- 172.21.15.16:0/4260654880 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0800b2290 con 0x7fa070003bc0 2023-10-16T12:30:50.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.957+0000 7fa07ffff700 1 --2- 172.21.15.16:0/4260654880 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa08015a5d0 0x7fa08015aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:30:50.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.957+0000 7fa084fa3700 1 --2- 172.21.15.16:0/4260654880 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa070003bc0 0x7fa0800b1b40 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7fa074001a30 tx=0x7fa074001b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:50.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.957+0000 7fa07d7fa700 1 -- 172.21.15.16:0/4260654880 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa074005d40 con 0x7fa070003bc0 2023-10-16T12:30:50.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.957+0000 7fa07d7fa700 1 -- 172.21.15.16:0/4260654880 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0740070d0 con 0x7fa070003bc0 2023-10-16T12:30:50.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.957+0000 7fa07d7fa700 1 -- 172.21.15.16:0/4260654880 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa074005ea0 con 0x7fa070003bc0 2023-10-16T12:30:50.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.957+0000 7fa087207700 1 -- 172.21.15.16:0/4260654880 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0800cc930 con 0x7fa070003bc0 2023-10-16T12:30:50.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.957+0000 7fa087207700 1 -- 172.21.15.16:0/4260654880 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0800ccef0 con 0x7fa070003bc0 2023-10-16T12:30:50.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.958+0000 7fa087207700 1 -- 172.21.15.16:0/4260654880 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa08004ec20 con 0x7fa070003bc0 2023-10-16T12:30:50.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.958+0000 7fa07d7fa700 1 -- 172.21.15.16:0/4260654880 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 81) v1 ==== 89388+0+0 (secure 0 0 0) 0x7fa074019070 con 0x7fa070003bc0 2023-10-16T12:30:50.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.959+0000 7fa07d7fa700 1 --2- 172.21.15.16:0/4260654880 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7fa06406b0b0 0x7fa06406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:50.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.959+0000 7fa07ffff700 1 -- 172.21.15.16:0/4260654880 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7fa06406b0b0 msgr2=0x7fa06406d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3783161600 2023-10-16T12:30:50.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.959+0000 7fa07d7fa700 1 -- 172.21.15.16:0/4260654880 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6429+0+0 (secure 0 0 0) 0x7fa07408aa50 con 0x7fa070003bc0 2023-10-16T12:30:50.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.959+0000 7fa07ffff700 1 --2- 172.21.15.16:0/4260654880 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7fa06406b0b0 0x7fa06406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:30:50.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:50.964+0000 7fa07d7fa700 1 -- 172.21.15.16:0/4260654880 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa074055640 con 0x7fa070003bc0 2023-10-16T12:30:50.984 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:50.981+0000 7f6bd6932200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T12:30:51.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.001+0000 7fa07d7fa700 1 -- 172.21.15.16:0/4260654880 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 82) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fa074007c90 con 0x7fa070003bc0 2023-10-16T12:30:51.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.001+0000 7fa07d7fa700 1 -- 172.21.15.16:0/4260654880 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7fa06406b0b0 msgr2=0x7fa06406d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:30:51.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.001+0000 7fa07d7fa700 1 --2- 172.21.15.16:0/4260654880 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7fa06406b0b0 0x7fa06406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:51.065 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:51.062+0000 7f6b5215e700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:30:51.065 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:51.062+0000 7f6b5215e700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:30:51.065 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:51.062+0000 7f6b5215e700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:30:51.066 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:51.062+0000 7f6b5215e700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:30:51.066 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:51.062+0000 7f6b5215e700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:30:51.066 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:51.062+0000 7f6b4f158700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:30:51.066 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:51.062+0000 7f6b4f158700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:30:51.066 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:51.062+0000 7f6b4f158700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:30:51.067 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:51.062+0000 7f6b4f158700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:30:51.067 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:30:51.062+0000 7f6b4f158700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:30:51.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.231+0000 7fa087207700 1 -- 172.21.15.16:0/4260654880 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa0800d0ea0 con 0x7fa070003bc0 2023-10-16T12:30:51.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.232+0000 7fa07d7fa700 1 -- 172.21.15.16:0/4260654880 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+4915 (secure 0 0 0) 0x7fa07406d220 con 0x7fa070003bc0 2023-10-16T12:30:51.235 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:30:51.236 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":70,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7220,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/4274593615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":4274593615},{"type":"v1","addr":"172.21.15.16:6837","nonce":4274593615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":8625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/1624531381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1624531381},{"type":"v1","addr":"172.21.15.16:6836","nonce":1624531381}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":8637,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/1629597454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":1629597454},{"type":"v1","addr":"172.21.15.49:6834","nonce":1629597454}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:30:22.807943+0000","modified":"2023-10-16T12:30:25.836865+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8640},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8640":{"gid":8640,"name":"b","rank":0,"incarnation":66,"state":"up:active","state_seq":3,"addr":"172.21.15.49:6835/2008024326","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":2008024326},{"type":"v1","addr":"172.21.15.49:6835","nonce":2008024326}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2023-10-16T12:30:51.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.234+0000 7fa087207700 1 -- 172.21.15.16:0/4260654880 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa070003bc0 msgr2=0x7fa0800b1b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:51.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.234+0000 7fa087207700 1 --2- 172.21.15.16:0/4260654880 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa070003bc0 0x7fa0800b1b40 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7fa074001a30 tx=0x7fa074001b00 comp rx=0 tx=0).stop 2023-10-16T12:30:51.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.234+0000 7fa087207700 1 -- 172.21.15.16:0/4260654880 shutdown_connections 2023-10-16T12:30:51.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.234+0000 7fa087207700 1 --2- 172.21.15.16:0/4260654880 >> [v2:172.21.15.16:6832/3783161600,v1:172.21.15.16:6833/3783161600] conn(0x7fa06406b0b0 0x7fa06406d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:51.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.234+0000 7fa087207700 1 --2- 172.21.15.16:0/4260654880 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa0800ca810 0x7fa0800caca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:51.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.234+0000 7fa087207700 1 --2- 172.21.15.16:0/4260654880 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa070003bc0 0x7fa0800b1b40 unknown :-1 s=CLOSED pgs=1662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:51.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.234+0000 7fa087207700 1 --2- 172.21.15.16:0/4260654880 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa08015a5d0 0x7fa08015aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:51.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.234+0000 7fa087207700 1 -- 172.21.15.16:0/4260654880 >> 172.21.15.16:0/4260654880 conn(0x7fa08009c9f0 msgr2=0x7fa080161350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:51.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.234+0000 7fa087207700 1 -- 172.21.15.16:0/4260654880 shutdown_connections 2023-10-16T12:30:51.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.234+0000 7fa087207700 1 -- 172.21.15.16:0/4260654880 wait complete. 2023-10-16T12:30:51.239 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 70 2023-10-16T12:30:51.249 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:30:51.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.374+0000 7f0de4020700 1 Processor -- start 2023-10-16T12:30:51.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.374+0000 7f0de4020700 1 -- start start 2023-10-16T12:30:51.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.375+0000 7f0de4020700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0ddc15b630 0x7f0ddc0d0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:51.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.375+0000 7f0de4020700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0ddc0ce4f0 0x7f0ddc0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:51.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.375+0000 7f0de4020700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ddc0b0970 con 0x7f0ddc15ba10 2023-10-16T12:30:51.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.375+0000 7f0de4020700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ddc0cd690 con 0x7f0ddc0ce4f0 2023-10-16T12:30:51.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.375+0000 7f0de4020700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ddc0cd970 con 0x7f0ddc15b630 2023-10-16T12:30:51.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.375+0000 7f0de25bd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0ddc0ce4f0 0x7f0ddc0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:51.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.375+0000 7f0de15bb700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0ddc15ba10 0x7f0ddc0cdde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35704/0 (socket says 172.21.15.16:35704) 2023-10-16T12:30:51.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.375+0000 7f0de15bb700 1 -- 172.21.15.16:0/2205238095 learned_addr learned my addr 172.21.15.16:0/2205238095 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:30:51.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.375+0000 7f0de1dbc700 1 --2- 172.21.15.16:0/2205238095 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0ddc15b630 0x7f0ddc0d0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:51.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.375+0000 7f0de25bd700 1 -- 172.21.15.16:0/2205238095 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0ddc15b630 msgr2=0x7f0ddc0d0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:51.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.375+0000 7f0de25bd700 1 --2- 172.21.15.16:0/2205238095 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0ddc15b630 0x7f0ddc0d0380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:51.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.375+0000 7f0de25bd700 1 -- 172.21.15.16:0/2205238095 >> v1:172.21.15.16:6789/0 conn(0x7f0ddc15ba10 legacy=0x7f0ddc0cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:51.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.375+0000 7f0de0dba700 1 -- 172.21.15.16:0/2205238095 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2268879604 0 0) 0x7f0ddc0b0970 con 0x7f0ddc15ba10 2023-10-16T12:30:51.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.375+0000 7f0de25bd700 1 -- 172.21.15.16:0/2205238095 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ddc0b0c30 con 0x7f0ddc0ce4f0 2023-10-16T12:30:51.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.375+0000 7f0de25bd700 1 --2- 172.21.15.16:0/2205238095 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ddc0ce4f0 0x7f0ddc0cf290 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7f0dd4009d90 tx=0x7f0dd400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79f785c4eb02810 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:51.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.375+0000 7f0de0dba700 1 -- 172.21.15.16:0/2205238095 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dd400e070 con 0x7f0ddc0ce4f0 2023-10-16T12:30:51.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.375+0000 7f0de0dba700 1 -- 172.21.15.16:0/2205238095 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dd400bb20 con 0x7f0ddc0ce4f0 2023-10-16T12:30:51.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.375+0000 7f0de0dba700 1 -- 172.21.15.16:0/2205238095 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dd4012430 con 0x7f0ddc0ce4f0 2023-10-16T12:30:51.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.376+0000 7f0de4020700 1 -- 172.21.15.16:0/2205238095 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ddc0ce4f0 msgr2=0x7f0ddc0cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:51.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.376+0000 7f0de4020700 1 --2- 172.21.15.16:0/2205238095 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ddc0ce4f0 0x7f0ddc0cf290 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7f0dd4009d90 tx=0x7f0dd400b330 comp rx=0 tx=0).stop 2023-10-16T12:30:51.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.376+0000 7f0de4020700 1 -- 172.21.15.16:0/2205238095 shutdown_connections 2023-10-16T12:30:51.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.376+0000 7f0de4020700 1 --2- 172.21.15.16:0/2205238095 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ddc0ce4f0 0x7f0ddc0cf290 unknown :-1 s=CLOSED pgs=1673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:51.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.376+0000 7f0de4020700 1 --2- 172.21.15.16:0/2205238095 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0ddc15b630 0x7f0ddc0d0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:51.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.376+0000 7f0de4020700 1 -- 172.21.15.16:0/2205238095 >> 172.21.15.16:0/2205238095 conn(0x7f0ddc0d14b0 msgr2=0x7f0ddc0d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:51.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.376+0000 7f0de4020700 1 -- 172.21.15.16:0/2205238095 shutdown_connections 2023-10-16T12:30:51.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.376+0000 7f0de4020700 1 -- 172.21.15.16:0/2205238095 wait complete. 2023-10-16T12:30:51.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.377+0000 7f0de4020700 1 Processor -- start 2023-10-16T12:30:51.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.377+0000 7f0de4020700 1 -- start start 2023-10-16T12:30:51.385 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:30:51.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.378+0000 7f0de4020700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ddc0ce4f0 0x7f0ddc1f27b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:51.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.378+0000 7f0de4020700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0ddc15b630 0x7f0ddc0b4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:51.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.378+0000 7f0de4020700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0ddc15ba10 0x7f0ddc1f62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:51.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.378+0000 7f0de4020700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ddc06c5e0 con 0x7f0ddc0ce4f0 2023-10-16T12:30:51.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.378+0000 7f0de4020700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ddc06c8c0 con 0x7f0ddc15b630 2023-10-16T12:30:51.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.378+0000 7f0de4020700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ddc06cba0 con 0x7f0ddc15ba10 2023-10-16T12:30:51.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.378+0000 7f0de1dbc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ddc0ce4f0 0x7f0ddc1f27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:51.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.378+0000 7f0de1dbc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ddc0ce4f0 0x7f0ddc1f27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46436/0 (socket says 172.21.15.16:46436) 2023-10-16T12:30:51.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.378+0000 7f0de1dbc700 1 -- 172.21.15.16:0/4246815890 learned_addr learned my addr 172.21.15.16:0/4246815890 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:51.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.378+0000 7f0de25bd700 1 --2- 172.21.15.16:0/4246815890 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0ddc15ba10 0x7f0ddc1f62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:51.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.378+0000 7f0de15bb700 1 --2- 172.21.15.16:0/4246815890 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0ddc15b630 0x7f0ddc0b4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:51.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.379+0000 7f0de1dbc700 1 -- 172.21.15.16:0/4246815890 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0ddc15ba10 msgr2=0x7f0ddc1f62a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:51.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.379+0000 7f0de1dbc700 1 --2- 172.21.15.16:0/4246815890 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0ddc15ba10 0x7f0ddc1f62a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:51.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.379+0000 7f0de1dbc700 1 -- 172.21.15.16:0/4246815890 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0ddc15b630 msgr2=0x7f0ddc0b4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:51.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.379+0000 7f0de1dbc700 1 --2- 172.21.15.16:0/4246815890 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0ddc15b630 0x7f0ddc0b4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:51.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.379+0000 7f0de1dbc700 1 -- 172.21.15.16:0/4246815890 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dd4009a70 con 0x7f0ddc0ce4f0 2023-10-16T12:30:51.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.379+0000 7f0de15bb700 1 --2- 172.21.15.16:0/4246815890 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0ddc15b630 0x7f0ddc0b4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:30:51.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.379+0000 7f0de1dbc700 1 --2- 172.21.15.16:0/4246815890 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ddc0ce4f0 0x7f0ddc1f27b0 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7f0dd800e9d0 tx=0x7f0dd800ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:51.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.379+0000 7f0dd27fc700 1 -- 172.21.15.16:0/4246815890 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dd8009730 con 0x7f0ddc0ce4f0 2023-10-16T12:30:51.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.379+0000 7f0dd27fc700 1 -- 172.21.15.16:0/4246815890 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dd800c850 con 0x7f0ddc0ce4f0 2023-10-16T12:30:51.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.379+0000 7f0de4020700 1 -- 172.21.15.16:0/4246815890 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ddc06ce80 con 0x7f0ddc0ce4f0 2023-10-16T12:30:51.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.379+0000 7f0dd27fc700 1 -- 172.21.15.16:0/4246815890 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dd800cbe0 con 0x7f0ddc0ce4f0 2023-10-16T12:30:51.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.379+0000 7f0de4020700 1 -- 172.21.15.16:0/4246815890 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ddc06c2d0 con 0x7f0ddc0ce4f0 2023-10-16T12:30:51.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.380+0000 7f0de4020700 1 -- 172.21.15.16:0/4246815890 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ddc15bf70 con 0x7f0ddc0ce4f0 2023-10-16T12:30:51.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.381+0000 7f0dd27fc700 1 -- 172.21.15.16:0/4246815890 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 82) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f0dd800cd40 con 0x7f0ddc0ce4f0 2023-10-16T12:30:51.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.381+0000 7f0dd27fc700 1 -- 172.21.15.16:0/4246815890 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6816+0+0 (secure 0 0 0) 0x7f0dd8089ac0 con 0x7f0ddc0ce4f0 2023-10-16T12:30:51.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.385+0000 7f0dd27fc700 1 -- 172.21.15.16:0/4246815890 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0dd8054c30 con 0x7f0ddc0ce4f0 2023-10-16T12:30:51.393 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:30:51.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.595+0000 7f0de4020700 1 -- 172.21.15.16:0/4246815890 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0ddc15bf70 con 0x7f0ddc0ce4f0 2023-10-16T12:30:51.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.596+0000 7f0dd27fc700 1 -- 172.21.15.16:0/4246815890 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v144) v1 ==== 74+0+17825 (secure 0 0 0) 0x7f0ddc15bf70 con 0x7f0ddc0ce4f0 2023-10-16T12:30:51.599 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:30:51.599 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":144,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:30:50.995921+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:30:21.318498+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"141","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2023-10-16T12:30:21.813382+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"142","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/3944675301":"2023-10-17T12:30:50.995881+0000","172.21.15.16:0/503385959":"2023-10-17T12:30:50.995881+0000","172.21.15.16:0/3093596020":"2023-10-17T12:30:50.995881+0000","172.21.15.16:0/1418518570":"2023-10-17T12:30:50.995881+0000","172.21.15.16:0/2205033750":"2023-10-17T12:30:50.995881+0000","172.21.15.16:0/647659834":"2023-10-17T12:30:50.995881+0000","172.21.15.16:6832/3783161600":"2023-10-17T12:30:50.995881+0000","172.21.15.16:0/2901217010":"2023-10-17T12:30:37.941028+0000","172.21.15.16:6833/3783161600":"2023-10-17T12:30:50.995881+0000","172.21.15.16:6833/258291157":"2023-10-17T12:30:37.941028+0000","172.21.15.16:6832/258291157":"2023-10-17T12:30:37.941028+0000","172.21.15.16:0/4123017371":"2023-10-17T12:30:37.941028+0000","172.21.15.16:0/2703050386":"2023-10-17T12:30:37.941028+0000","172.21.15.16:0/1995641428":"2023-10-17T12:30:50.995881+0000","172.21.15.16:0/3188488837":"2023-10-17T12:30:37.941028+0000","172.21.15.16:0/2621298501":"2023-10-17T12:30:37.941028+0000","172.21.15.16:0/1781882931":"2023-10-17T12:30:37.941028+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:30:51.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.598+0000 7f0de4020700 1 -- 172.21.15.16:0/4246815890 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ddc0ce4f0 msgr2=0x7f0ddc1f27b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:51.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.598+0000 7f0de4020700 1 --2- 172.21.15.16:0/4246815890 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ddc0ce4f0 0x7f0ddc1f27b0 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7f0dd800e9d0 tx=0x7f0dd800ece0 comp rx=0 tx=0).stop 2023-10-16T12:30:51.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.598+0000 7f0de4020700 1 -- 172.21.15.16:0/4246815890 shutdown_connections 2023-10-16T12:30:51.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.598+0000 7f0de4020700 1 --2- 172.21.15.16:0/4246815890 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0ddc15ba10 0x7f0ddc1f62a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:51.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.598+0000 7f0de4020700 1 --2- 172.21.15.16:0/4246815890 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ddc0ce4f0 0x7f0ddc1f27b0 unknown :-1 s=CLOSED pgs=1674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:51.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.598+0000 7f0de4020700 1 --2- 172.21.15.16:0/4246815890 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0ddc15b630 0x7f0ddc0b4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:51.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.598+0000 7f0de4020700 1 -- 172.21.15.16:0/4246815890 >> 172.21.15.16:0/4246815890 conn(0x7f0ddc0d14b0 msgr2=0x7f0ddc0d1890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:51.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.598+0000 7f0de4020700 1 -- 172.21.15.16:0/4246815890 shutdown_connections 2023-10-16T12:30:51.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.598+0000 7f0de4020700 1 -- 172.21.15.16:0/4246815890 wait complete. 2023-10-16T12:30:51.612 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-16T12:30:51.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.736+0000 7f3cb47dd700 1 Processor -- start 2023-10-16T12:30:51.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.736+0000 7f3cb47dd700 1 -- start start 2023-10-16T12:30:51.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.737+0000 7f3cb47dd700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f3cac15b760 0x7f3cac06a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:51.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.737+0000 7f3cb47dd700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3cac0b2be0 0x7f3cac06ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:51.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.737+0000 7f3cb47dd700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cac0d03b0 con 0x7f3cac0cd6e0 2023-10-16T12:30:51.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.737+0000 7f3cb47dd700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cac15bb40 con 0x7f3cac15b760 2023-10-16T12:30:51.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.737+0000 7f3cb47dd700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cac0cb760 con 0x7f3cac0b2be0 2023-10-16T12:30:51.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.738+0000 7f3cb2579700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f3cac15b760 0x7f3cac06a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:51.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.738+0000 7f3cb1d78700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3cac0b2be0 0x7f3cac06ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:51.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.738+0000 7f3cb2d7a700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7f3cac0cd6e0 0x7f3cac0719e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:42900/0 (socket says 172.21.15.16:42900) 2023-10-16T12:30:51.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.738+0000 7f3cb2d7a700 1 -- 172.21.15.16:0/415961802 learned_addr learned my addr 172.21.15.16:0/415961802 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:30:51.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.738+0000 7f3cb1577700 1 -- 172.21.15.16:0/415961802 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1773924127 0 0) 0x7f3cac0d03b0 con 0x7f3cac0cd6e0 2023-10-16T12:30:51.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.738+0000 7f3cb1577700 1 -- 172.21.15.16:0/415961802 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c980035b0 con 0x7f3cac0cd6e0 2023-10-16T12:30:51.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.738+0000 7f3cb2579700 1 -- 172.21.15.16:0/415961802 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3cac0b2be0 msgr2=0x7f3cac06ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:51.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.738+0000 7f3cb2579700 1 --2- 172.21.15.16:0/415961802 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3cac0b2be0 0x7f3cac06ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:51.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.738+0000 7f3cb2579700 1 -- 172.21.15.16:0/415961802 >> v1:172.21.15.49:6789/0 conn(0x7f3cac0cd6e0 legacy=0x7f3cac0719e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:51.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.738+0000 7f3cb2579700 1 -- 172.21.15.16:0/415961802 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cac0d0600 con 0x7f3cac15b760 2023-10-16T12:30:51.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.738+0000 7f3cb1d78700 1 --2- 172.21.15.16:0/415961802 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3cac0b2be0 0x7f3cac06ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:30:51.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.738+0000 7f3cb2579700 1 --2- 172.21.15.16:0/415961802 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3cac15b760 0x7f3cac06a7e0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f3ca8009840 tx=0x7f3ca8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9ba37832a3c542ef server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:51.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.738+0000 7f3cb1577700 1 -- 172.21.15.16:0/415961802 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ca8011830 con 0x7f3cac15b760 2023-10-16T12:30:51.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.738+0000 7f3cb1577700 1 -- 172.21.15.16:0/415961802 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ca8011990 con 0x7f3cac15b760 2023-10-16T12:30:51.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.738+0000 7f3cb1577700 1 -- 172.21.15.16:0/415961802 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ca8011d00 con 0x7f3cac15b760 2023-10-16T12:30:51.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.739+0000 7f3cb47dd700 1 -- 172.21.15.16:0/415961802 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3cac15b760 msgr2=0x7f3cac06a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:51.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.739+0000 7f3cb47dd700 1 --2- 172.21.15.16:0/415961802 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3cac15b760 0x7f3cac06a7e0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f3ca8009840 tx=0x7f3ca8009b50 comp rx=0 tx=0).stop 2023-10-16T12:30:51.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.739+0000 7f3cb47dd700 1 -- 172.21.15.16:0/415961802 shutdown_connections 2023-10-16T12:30:51.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.739+0000 7f3cb47dd700 1 --2- 172.21.15.16:0/415961802 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3cac15b760 0x7f3cac06a7e0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:51.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.739+0000 7f3cb47dd700 1 --2- 172.21.15.16:0/415961802 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3cac0b2be0 0x7f3cac06ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:51.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.739+0000 7f3cb47dd700 1 -- 172.21.15.16:0/415961802 >> 172.21.15.16:0/415961802 conn(0x7f3cac0b1fd0 msgr2=0x7f3cac0b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:51.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.739+0000 7f3cb47dd700 1 -- 172.21.15.16:0/415961802 shutdown_connections 2023-10-16T12:30:51.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.739+0000 7f3cb47dd700 1 -- 172.21.15.16:0/415961802 wait complete. 2023-10-16T12:30:51.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.740+0000 7f3cb47dd700 1 Processor -- start 2023-10-16T12:30:51.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.740+0000 7f3cb47dd700 1 -- start start 2023-10-16T12:30:51.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.741+0000 7f3cb47dd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3cac0b2be0 0x7f3cac1f6860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:51.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.741+0000 7f3cb47dd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3cac0cd6e0 0x7f3cac0b18d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:51.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.741+0000 7f3cb47dd700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3cac15b760 0x7f3cac158960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:51.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.741+0000 7f3cb47dd700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cac0ae6b0 con 0x7f3cac0b2be0 2023-10-16T12:30:51.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.741+0000 7f3cb47dd700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cac0b1e10 con 0x7f3cac0cd6e0 2023-10-16T12:30:51.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.741+0000 7f3cb47dd700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cac0ae350 con 0x7f3cac15b760 2023-10-16T12:30:51.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.741+0000 7f3cb2579700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3cac0b2be0 0x7f3cac1f6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:51.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.741+0000 7f3cb2579700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3cac0b2be0 0x7f3cac1f6860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46446/0 (socket says 172.21.15.16:46446) 2023-10-16T12:30:51.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.741+0000 7f3cb2579700 1 -- 172.21.15.16:0/1262206048 learned_addr learned my addr 172.21.15.16:0/1262206048 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:51.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.741+0000 7f3cb1d78700 1 --2- 172.21.15.16:0/1262206048 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3cac0cd6e0 0x7f3cac0b18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:51.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.741+0000 7f3cb2d7a700 1 --2- 172.21.15.16:0/1262206048 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3cac15b760 0x7f3cac158960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:51.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.742+0000 7f3cb2579700 1 -- 172.21.15.16:0/1262206048 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3cac15b760 msgr2=0x7f3cac158960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:51.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.742+0000 7f3cb2579700 1 --2- 172.21.15.16:0/1262206048 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3cac15b760 0x7f3cac158960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:51.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.742+0000 7f3cb2579700 1 -- 172.21.15.16:0/1262206048 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3cac0cd6e0 msgr2=0x7f3cac0b18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:51.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.742+0000 7f3cb2579700 1 --2- 172.21.15.16:0/1262206048 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3cac0cd6e0 0x7f3cac0b18d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:51.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.742+0000 7f3cb2579700 1 -- 172.21.15.16:0/1262206048 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c9c0094a0 con 0x7f3cac0b2be0 2023-10-16T12:30:51.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.742+0000 7f3cb2579700 1 --2- 172.21.15.16:0/1262206048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3cac0b2be0 0x7f3cac1f6860 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7f3ca80090b0 tx=0x7f3ca8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:51.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.742+0000 7f3ca2ffd700 1 -- 172.21.15.16:0/1262206048 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ca8004550 con 0x7f3cac0b2be0 2023-10-16T12:30:51.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.742+0000 7f3ca2ffd700 1 -- 172.21.15.16:0/1262206048 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ca801f070 con 0x7f3cac0b2be0 2023-10-16T12:30:51.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.742+0000 7f3cb47dd700 1 -- 172.21.15.16:0/1262206048 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ca80094a0 con 0x7f3cac0b2be0 2023-10-16T12:30:51.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.742+0000 7f3ca2ffd700 1 -- 172.21.15.16:0/1262206048 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ca80038c0 con 0x7f3cac0b2be0 2023-10-16T12:30:51.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.742+0000 7f3cb47dd700 1 -- 172.21.15.16:0/1262206048 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cac1f7240 con 0x7f3cac0b2be0 2023-10-16T12:30:51.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.743+0000 7f3cb47dd700 1 -- 172.21.15.16:0/1262206048 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cac0ae7f0 con 0x7f3cac0b2be0 2023-10-16T12:30:51.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.743+0000 7f3ca2ffd700 1 -- 172.21.15.16:0/1262206048 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 82) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f3ca8003a20 con 0x7f3cac0b2be0 2023-10-16T12:30:51.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.744+0000 7f3ca2ffd700 1 -- 172.21.15.16:0/1262206048 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6816+0+0 (secure 0 0 0) 0x7f3ca8021070 con 0x7f3cac0b2be0 2023-10-16T12:30:51.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:51.748+0000 7f3ca2ffd700 1 -- 172.21.15.16:0/1262206048 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3cac0ae7f0 con 0x7f3cac0b2be0 2023-10-16T12:30:52.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.000+0000 7f3cb47dd700 1 -- 172.21.15.16:0/1262206048 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f3cac0ae7f0 con 0x7f3cac0b2be0 2023-10-16T12:30:52.003 INFO:tasks.ceph.mon.a.smithi016.stderr:2023-10-16T12:30:52.001+0000 7f9fc8535700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-16T12:30:52.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.005+0000 7f3ca2ffd700 1 -- 172.21.15.16:0/1262206048 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 83) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f3ca80557b0 con 0x7f3cac0b2be0 2023-10-16T12:30:52.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.006+0000 7f3ca2ffd700 1 --2- 172.21.15.16:0/1262206048 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f3c90070d40 0x7f3c90073130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:52.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.006+0000 7f3cb1d78700 1 --2- 172.21.15.16:0/1262206048 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f3c90070d40 0x7f3c90073130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:52.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.008+0000 7f3cb1d78700 1 --2- 172.21.15.16:0/1262206048 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f3c90070d40 0x7f3c90073130 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3cac1e4a20 tx=0x7f3c9c009210 comp rx=0 tx=0).ready entity=mgr.8973 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:52.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.011+0000 7f3ca2ffd700 1 -- 172.21.15.16:0/1262206048 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v71) v1 ==== 161+0+0 (secure 0 0 0) 0x7f3ca806de40 con 0x7f3cac0b2be0 2023-10-16T12:30:52.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.015+0000 7f3cb47dd700 1 -- 172.21.15.16:0/1262206048 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f3c90070d40 msgr2=0x7f3c90073130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:52.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.015+0000 7f3cb47dd700 1 --2- 172.21.15.16:0/1262206048 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f3c90070d40 0x7f3c90073130 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3cac1e4a20 tx=0x7f3c9c009210 comp rx=0 tx=0).stop 2023-10-16T12:30:52.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.015+0000 7f3cb47dd700 1 -- 172.21.15.16:0/1262206048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3cac0b2be0 msgr2=0x7f3cac1f6860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:52.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.015+0000 7f3cb47dd700 1 --2- 172.21.15.16:0/1262206048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3cac0b2be0 0x7f3cac1f6860 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7f3ca80090b0 tx=0x7f3ca8004750 comp rx=0 tx=0).stop 2023-10-16T12:30:52.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.015+0000 7f3cb47dd700 1 -- 172.21.15.16:0/1262206048 shutdown_connections 2023-10-16T12:30:52.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.015+0000 7f3cb47dd700 1 --2- 172.21.15.16:0/1262206048 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f3c90070d40 0x7f3c90073130 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:52.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.015+0000 7f3cb47dd700 1 --2- 172.21.15.16:0/1262206048 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3cac15b760 0x7f3cac158960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:52.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.015+0000 7f3cb47dd700 1 --2- 172.21.15.16:0/1262206048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3cac0b2be0 0x7f3cac1f6860 unknown :-1 s=CLOSED pgs=1675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:52.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.015+0000 7f3cb47dd700 1 --2- 172.21.15.16:0/1262206048 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3cac0cd6e0 0x7f3cac0b18d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:52.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.015+0000 7f3cb47dd700 1 -- 172.21.15.16:0/1262206048 >> 172.21.15.16:0/1262206048 conn(0x7f3cac0b1fd0 msgr2=0x7f3cac0b35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:52.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.015+0000 7f3cb47dd700 1 -- 172.21.15.16:0/1262206048 shutdown_connections 2023-10-16T12:30:52.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.015+0000 7f3cb47dd700 1 -- 172.21.15.16:0/1262206048 wait complete. 2023-10-16T12:30:52.020 INFO:tasks.ceph.mds.b.smithi049.stderr: -9876> 2023-10-16T12:30:25.377+0000 7fa149c9f700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T12:30:52.021 INFO:teuthology.orchestra.run.smithi016.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-16T12:30:52.030 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-16T12:30:52.055 INFO:tasks.ceph.mds.b.smithi049.stdout:starting mds.b at 2023-10-16T12:30:52.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.155+0000 7fcbec04f700 1 Processor -- start 2023-10-16T12:30:52.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.155+0000 7fcbec04f700 1 -- start start 2023-10-16T12:30:52.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.156+0000 7fcbec04f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcbe415a640 0x7fcbe40cb710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:52.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.156+0000 7fcbec04f700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbe415aa20 con 0x7fcbe40c6770 2023-10-16T12:30:52.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.156+0000 7fcbec04f700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbe40cafd0 con 0x7fcbe415e140 2023-10-16T12:30:52.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.156+0000 7fcbec04f700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe406eb40 con 0x7fcbe415a640 2023-10-16T12:30:52.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.156+0000 7fcbe95ea700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fcbe40c6770 0x7fcbe40ca1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35750/0 (socket says 172.21.15.16:35750) 2023-10-16T12:30:52.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.156+0000 7fcbe95ea700 1 -- 172.21.15.16:0/2120157875 learned_addr learned my addr 172.21.15.16:0/2120157875 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:30:52.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.156+0000 7fcbe9deb700 1 --2- 172.21.15.16:0/2120157875 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcbe415a640 0x7fcbe40cb710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:52.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.156+0000 7fcbe8de9700 1 -- 172.21.15.16:0/2120157875 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2024463193 0 0) 0x7fcbe0001110 con 0x7fcbe40c6770 2023-10-16T12:30:52.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.156+0000 7fcbe8de9700 1 -- 172.21.15.16:0/2120157875 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbd00035b0 con 0x7fcbe40c6770 2023-10-16T12:30:52.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.156+0000 7fcbe8de9700 1 -- 172.21.15.16:0/2120157875 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4024986636 0 0) 0x7fcbe40cafd0 con 0x7fcbe415e140 2023-10-16T12:30:52.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.157+0000 7fcbe8de9700 1 -- 172.21.15.16:0/2120157875 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbe0001110 con 0x7fcbe415e140 2023-10-16T12:30:52.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.157+0000 7fcbe8de9700 1 -- 172.21.15.16:0/2120157875 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 647654686 0 0) 0x7fcbd00035b0 con 0x7fcbe40c6770 2023-10-16T12:30:52.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.157+0000 7fcbe8de9700 1 -- 172.21.15.16:0/2120157875 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcbe415a640 msgr2=0x7fcbe40cb710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:52.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.157+0000 7fcbe8de9700 1 --2- 172.21.15.16:0/2120157875 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcbe415a640 0x7fcbe40cb710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:52.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.157+0000 7fcbe8de9700 1 -- 172.21.15.16:0/2120157875 >> v1:172.21.15.49:6789/0 conn(0x7fcbe415e140 legacy=0x7fcbe40ca8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:52.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.157+0000 7fcbe8de9700 1 -- 172.21.15.16:0/2120157875 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbe406ee20 con 0x7fcbe40c6770 2023-10-16T12:30:52.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.157+0000 7fcbe8de9700 1 -- 172.21.15.16:0/2120157875 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7fcbe0001800 con 0x7fcbe40c6770 2023-10-16T12:30:52.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.157+0000 7fcbe8de9700 1 -- 172.21.15.16:0/2120157875 >> v1:172.21.15.16:6789/0 conn(0x7fcbe40c6770 legacy=0x7fcbe40ca1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:52.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.157+0000 7fcbe8de9700 1 --2- 172.21.15.16:0/2120157875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbd0003bc0 0x7fcbd0003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:52.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.157+0000 7fcbe8de9700 1 --2- 172.21.15.16:0/2120157875 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcbe415a640 0x7fcbd0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:52.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.157+0000 7fcbe8de9700 1 --2- 172.21.15.16:0/2120157875 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcbd0004d90 0x7fcbd0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:52.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.157+0000 7fcbe8de9700 1 -- 172.21.15.16:0/2120157875 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe406eb40 con 0x7fcbd0003bc0 2023-10-16T12:30:52.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.157+0000 7fcbe8de9700 1 -- 172.21.15.16:0/2120157875 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd000bac0 con 0x7fcbd0004d90 2023-10-16T12:30:52.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.157+0000 7fcbe8de9700 1 -- 172.21.15.16:0/2120157875 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd000bc80 con 0x7fcbe415a640 2023-10-16T12:30:52.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.157+0000 7fcbe9deb700 1 --2- 172.21.15.16:0/2120157875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbd0003bc0 0x7fcbd0003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:52.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.157+0000 7fcbe95ea700 1 --2- 172.21.15.16:0/2120157875 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcbe415a640 0x7fcbd0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:52.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.157+0000 7fcbea5ec700 1 --2- 172.21.15.16:0/2120157875 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcbd0004d90 0x7fcbd0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:52.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.157+0000 7fcbe9deb700 1 -- 172.21.15.16:0/2120157875 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcbe415a640 msgr2=0x7fcbd0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:52.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.157+0000 7fcbe9deb700 1 --2- 172.21.15.16:0/2120157875 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcbe415a640 0x7fcbd0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:52.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.157+0000 7fcbe9deb700 1 -- 172.21.15.16:0/2120157875 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcbd0004d90 msgr2=0x7fcbd0005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:52.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.157+0000 7fcbe9deb700 1 --2- 172.21.15.16:0/2120157875 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcbd0004d90 0x7fcbd0005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:52.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.157+0000 7fcbe9deb700 1 -- 172.21.15.16:0/2120157875 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbd00035b0 con 0x7fcbd0003bc0 2023-10-16T12:30:52.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.157+0000 7fcbe9deb700 1 --2- 172.21.15.16:0/2120157875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbd0003bc0 0x7fcbd0003fc0 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7fcbd4009b60 tx=0x7fcbd0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d232e3974d2fb865 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:52.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.157+0000 7fcbe8de9700 1 -- 172.21.15.16:0/2120157875 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbd400e6c0 con 0x7fcbd0003bc0 2023-10-16T12:30:52.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.157+0000 7fcbe8de9700 1 -- 172.21.15.16:0/2120157875 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbd400e820 con 0x7fcbd0003bc0 2023-10-16T12:30:52.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.157+0000 7fcbe8de9700 1 -- 172.21.15.16:0/2120157875 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbd400eb90 con 0x7fcbd0003bc0 2023-10-16T12:30:52.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.158+0000 7fcbec04f700 1 -- 172.21.15.16:0/2120157875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbd0003bc0 msgr2=0x7fcbd0003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:52.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.158+0000 7fcbec04f700 1 --2- 172.21.15.16:0/2120157875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbd0003bc0 0x7fcbd0003fc0 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7fcbd4009b60 tx=0x7fcbd0005880 comp rx=0 tx=0).stop 2023-10-16T12:30:52.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.158+0000 7fcbea5ec700 1 -- 172.21.15.16:0/2120157875 reap_dead start 2023-10-16T12:30:52.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.158+0000 7fcbec04f700 1 -- 172.21.15.16:0/2120157875 shutdown_connections 2023-10-16T12:30:52.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.158+0000 7fcbec04f700 1 -- 172.21.15.16:0/2120157875 >> 172.21.15.16:0/2120157875 conn(0x7fcbe40cc5e0 msgr2=0x7fcbe40cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:52.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.158+0000 7fcbec04f700 1 -- 172.21.15.16:0/2120157875 shutdown_connections 2023-10-16T12:30:52.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.158+0000 7fcbec04f700 1 -- 172.21.15.16:0/2120157875 wait complete. 2023-10-16T12:30:52.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.159+0000 7fcbec04f700 1 Processor -- start 2023-10-16T12:30:52.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.159+0000 7fcbec04f700 1 -- start start 2023-10-16T12:30:52.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.160+0000 7fcbec04f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbd0003bc0 0x7fcbe4154eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:52.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.160+0000 7fcbec04f700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcbe40c51f0 0x7fcbe415d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:52.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.160+0000 7fcbec04f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcbe40711d0 0x7fcbe4071680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:52.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.160+0000 7fcbec04f700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe4071bc0 con 0x7fcbd0003bc0 2023-10-16T12:30:52.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.160+0000 7fcbec04f700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe41f6040 con 0x7fcbe40c51f0 2023-10-16T12:30:52.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.160+0000 7fcbec04f700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe41f6290 con 0x7fcbe40711d0 2023-10-16T12:30:52.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.160+0000 7fcbe9deb700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbd0003bc0 0x7fcbe4154eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:52.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.160+0000 7fcbe9deb700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbd0003bc0 0x7fcbe4154eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46488/0 (socket says 172.21.15.16:46488) 2023-10-16T12:30:52.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.160+0000 7fcbe9deb700 1 -- 172.21.15.16:0/4197202519 learned_addr learned my addr 172.21.15.16:0/4197202519 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:52.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.160+0000 7fcbe95ea700 1 --2- 172.21.15.16:0/4197202519 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcbe40c51f0 0x7fcbe415d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:52.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.160+0000 7fcbea5ec700 1 --2- 172.21.15.16:0/4197202519 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcbe40711d0 0x7fcbe4071680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:52.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.160+0000 7fcbe9deb700 1 -- 172.21.15.16:0/4197202519 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcbe40711d0 msgr2=0x7fcbe4071680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:52.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.160+0000 7fcbe9deb700 1 --2- 172.21.15.16:0/4197202519 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcbe40711d0 0x7fcbe4071680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:52.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.160+0000 7fcbe9deb700 1 -- 172.21.15.16:0/4197202519 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcbe40c51f0 msgr2=0x7fcbe415d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:52.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.160+0000 7fcbe9deb700 1 --2- 172.21.15.16:0/4197202519 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcbe40c51f0 0x7fcbe415d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:52.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.160+0000 7fcbe9deb700 1 -- 172.21.15.16:0/4197202519 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbe41f6570 con 0x7fcbd0003bc0 2023-10-16T12:30:52.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.161+0000 7fcbe9deb700 1 --2- 172.21.15.16:0/4197202519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbd0003bc0 0x7fcbe4154eb0 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7fcbd0005830 tx=0x7fcbd40047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:52.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.161+0000 7fcbda7fc700 1 -- 172.21.15.16:0/4197202519 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbd400e6c0 con 0x7fcbd0003bc0 2023-10-16T12:30:52.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.161+0000 7fcbda7fc700 1 -- 172.21.15.16:0/4197202519 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbd40038c0 con 0x7fcbd0003bc0 2023-10-16T12:30:52.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.161+0000 7fcbec04f700 1 -- 172.21.15.16:0/4197202519 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbe41f6850 con 0x7fcbd0003bc0 2023-10-16T12:30:52.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.161+0000 7fcbda7fc700 1 -- 172.21.15.16:0/4197202519 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbd4003c70 con 0x7fcbd0003bc0 2023-10-16T12:30:52.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.161+0000 7fcbec04f700 1 -- 172.21.15.16:0/4197202519 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbe415d840 con 0x7fcbd0003bc0 2023-10-16T12:30:52.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.162+0000 7fcbec04f700 1 -- 172.21.15.16:0/4197202519 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbe415d9d0 con 0x7fcbd0003bc0 2023-10-16T12:30:52.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.162+0000 7fcbda7fc700 1 -- 172.21.15.16:0/4197202519 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 83) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fcbd4013470 con 0x7fcbd0003bc0 2023-10-16T12:30:52.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.162+0000 7fcbda7fc700 1 --2- 172.21.15.16:0/4197202519 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7fcbc806ad50 0x7fcbc806d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:52.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.163+0000 7fcbda7fc700 1 -- 172.21.15.16:0/4197202519 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6902+0+0 (secure 0 0 0) 0x7fcbd408af30 con 0x7fcbd0003bc0 2023-10-16T12:30:52.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.163+0000 7fcbe95ea700 1 --2- 172.21.15.16:0/4197202519 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7fcbc806ad50 0x7fcbc806d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:52.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.163+0000 7fcbe95ea700 1 --2- 172.21.15.16:0/4197202519 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7fcbc806ad50 0x7fcbc806d210 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fcbe40b1530 tx=0x7fcbe0000f70 comp rx=0 tx=0).ready entity=mgr.8973 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:52.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.167+0000 7fcbda7fc700 1 -- 172.21.15.16:0/4197202519 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcbd4055f90 con 0x7fcbd0003bc0 2023-10-16T12:30:52.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:52.417+0000 7fcbec04f700 1 -- 172.21.15.16:0/4197202519 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fcbe404ebb0 con 0x7fcbd0003bc0 2023-10-16T12:30:53.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.015+0000 7fcbda7fc700 1 -- 172.21.15.16:0/4197202519 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v72) v1 ==== 104+0+0 (secure 0 0 0) 0x7fcbd40595b0 con 0x7fcbd0003bc0 2023-10-16T12:30:53.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.018+0000 7fcbec04f700 1 -- 172.21.15.16:0/4197202519 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7fcbc806ad50 msgr2=0x7fcbc806d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:53.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.018+0000 7fcbec04f700 1 --2- 172.21.15.16:0/4197202519 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7fcbc806ad50 0x7fcbc806d210 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fcbe40b1530 tx=0x7fcbe0000f70 comp rx=0 tx=0).stop 2023-10-16T12:30:53.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.018+0000 7fcbec04f700 1 -- 172.21.15.16:0/4197202519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbd0003bc0 msgr2=0x7fcbe4154eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:53.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.018+0000 7fcbec04f700 1 --2- 172.21.15.16:0/4197202519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbd0003bc0 0x7fcbe4154eb0 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7fcbd0005830 tx=0x7fcbd40047c0 comp rx=0 tx=0).stop 2023-10-16T12:30:53.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.018+0000 7fcbec04f700 1 -- 172.21.15.16:0/4197202519 shutdown_connections 2023-10-16T12:30:53.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.018+0000 7fcbec04f700 1 --2- 172.21.15.16:0/4197202519 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7fcbc806ad50 0x7fcbc806d210 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:53.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.018+0000 7fcbec04f700 1 --2- 172.21.15.16:0/4197202519 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcbe40711d0 0x7fcbe4071680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:53.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.018+0000 7fcbec04f700 1 --2- 172.21.15.16:0/4197202519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcbd0003bc0 0x7fcbe4154eb0 unknown :-1 s=CLOSED pgs=1679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:53.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.018+0000 7fcbec04f700 1 --2- 172.21.15.16:0/4197202519 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcbe40c51f0 0x7fcbe415d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:53.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.018+0000 7fcbec04f700 1 -- 172.21.15.16:0/4197202519 >> 172.21.15.16:0/4197202519 conn(0x7fcbe40cc5e0 msgr2=0x7fcbe40ce550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:53.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.018+0000 7fcbec04f700 1 -- 172.21.15.16:0/4197202519 shutdown_connections 2023-10-16T12:30:53.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.018+0000 7fcbec04f700 1 -- 172.21.15.16:0/4197202519 wait complete. 2023-10-16T12:30:53.035 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-16T12:30:53.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.174+0000 7f9c6aaac700 1 Processor -- start 2023-10-16T12:30:53.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.174+0000 7f9c6aaac700 1 -- start start 2023-10-16T12:30:53.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.175+0000 7f9c6aaac700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9c640cf600 0x7f9c6406d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:53.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.175+0000 7f9c6aaac700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c640cb870 con 0x7f9c6415e0b0 2023-10-16T12:30:53.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.175+0000 7f9c6aaac700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c6406e040 con 0x7f9c6415a3d0 2023-10-16T12:30:53.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.175+0000 7f9c6aaac700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c6415f0c0 con 0x7f9c640cf600 2023-10-16T12:30:53.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.175+0000 7f9c69049700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f9c6415e0b0 0x7f9c640cbd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35792/0 (socket says 172.21.15.16:35792) 2023-10-16T12:30:53.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.175+0000 7f9c69049700 1 -- 172.21.15.16:0/3823827504 learned_addr learned my addr 172.21.15.16:0/3823827504 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:30:53.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.175+0000 7f9c637fe700 1 -- 172.21.15.16:0/3823827504 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 397288009 0 0) 0x7f9c640cb870 con 0x7f9c6415e0b0 2023-10-16T12:30:53.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.175+0000 7f9c63fff700 1 --2- 172.21.15.16:0/3823827504 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9c640cf600 0x7f9c6406d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:53.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.175+0000 7f9c637fe700 1 -- 172.21.15.16:0/3823827504 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c540035b0 con 0x7f9c6415e0b0 2023-10-16T12:30:53.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.176+0000 7f9c637fe700 1 -- 172.21.15.16:0/3823827504 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3557424829 0 0) 0x7f9c540035b0 con 0x7f9c6415e0b0 2023-10-16T12:30:53.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.176+0000 7f9c637fe700 1 -- 172.21.15.16:0/3823827504 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9c640cf600 msgr2=0x7f9c6406d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:53.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.176+0000 7f9c637fe700 1 --2- 172.21.15.16:0/3823827504 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9c640cf600 0x7f9c6406d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:53.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.176+0000 7f9c637fe700 1 -- 172.21.15.16:0/3823827504 >> v1:172.21.15.49:6789/0 conn(0x7f9c6415a3d0 legacy=0x7f9c640ceef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:53.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.176+0000 7f9c637fe700 1 -- 172.21.15.16:0/3823827504 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c6406e320 con 0x7f9c6415e0b0 2023-10-16T12:30:53.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.176+0000 7f9c637fe700 1 -- 172.21.15.16:0/3823827504 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f9c5c0019f0 con 0x7f9c6415e0b0 2023-10-16T12:30:53.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.176+0000 7f9c637fe700 1 -- 172.21.15.16:0/3823827504 >> v1:172.21.15.16:6789/0 conn(0x7f9c6415e0b0 legacy=0x7f9c640cbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:53.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.176+0000 7f9c637fe700 1 --2- 172.21.15.16:0/3823827504 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9c54003a30 0x7f9c54003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:53.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.176+0000 7f9c637fe700 1 --2- 172.21.15.16:0/3823827504 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c54004850 0x7f9c54004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:53.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.176+0000 7f9c637fe700 1 --2- 172.21.15.16:0/3823827504 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9c640cf600 0x7f9c54005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:53.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.176+0000 7f9c637fe700 1 -- 172.21.15.16:0/3823827504 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c6415f0c0 con 0x7f9c54004850 2023-10-16T12:30:53.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.176+0000 7f9c637fe700 1 -- 172.21.15.16:0/3823827504 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c5400bac0 con 0x7f9c54003a30 2023-10-16T12:30:53.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.176+0000 7f9c637fe700 1 -- 172.21.15.16:0/3823827504 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c5400bc80 con 0x7f9c640cf600 2023-10-16T12:30:53.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.176+0000 7f9c63fff700 1 --2- 172.21.15.16:0/3823827504 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c54004850 0x7f9c54004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:53.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.176+0000 7f9c68848700 1 --2- 172.21.15.16:0/3823827504 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9c54003a30 0x7f9c54003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:53.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.176+0000 7f9c69049700 1 --2- 172.21.15.16:0/3823827504 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9c640cf600 0x7f9c54005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:53.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.176+0000 7f9c63fff700 1 -- 172.21.15.16:0/3823827504 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9c640cf600 msgr2=0x7f9c54005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:53.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.176+0000 7f9c63fff700 1 --2- 172.21.15.16:0/3823827504 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9c640cf600 0x7f9c54005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:53.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.176+0000 7f9c63fff700 1 -- 172.21.15.16:0/3823827504 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9c54003a30 msgr2=0x7f9c54003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:53.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.176+0000 7f9c63fff700 1 --2- 172.21.15.16:0/3823827504 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9c54003a30 0x7f9c54003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:53.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.176+0000 7f9c63fff700 1 -- 172.21.15.16:0/3823827504 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c540035b0 con 0x7f9c54004850 2023-10-16T12:30:53.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.176+0000 7f9c68848700 1 --2- 172.21.15.16:0/3823827504 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9c54003a30 0x7f9c54003e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:30:53.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.176+0000 7f9c69049700 1 --2- 172.21.15.16:0/3823827504 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9c640cf600 0x7f9c54005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:30:53.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.176+0000 7f9c63fff700 1 --2- 172.21.15.16:0/3823827504 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c54004850 0x7f9c54004cd0 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7f9c50009b30 tx=0x7f9c54005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=643fcd94a0a61980 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:53.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.176+0000 7f9c637fe700 1 -- 172.21.15.16:0/3823827504 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c5000e6c0 con 0x7f9c54004850 2023-10-16T12:30:53.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.176+0000 7f9c637fe700 1 -- 172.21.15.16:0/3823827504 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c5000e820 con 0x7f9c54004850 2023-10-16T12:30:53.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.176+0000 7f9c637fe700 1 -- 172.21.15.16:0/3823827504 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c5000eb90 con 0x7f9c54004850 2023-10-16T12:30:53.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.177+0000 7f9c6aaac700 1 -- 172.21.15.16:0/3823827504 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c54004850 msgr2=0x7f9c54004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:53.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.177+0000 7f9c6aaac700 1 --2- 172.21.15.16:0/3823827504 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c54004850 0x7f9c54004cd0 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7f9c50009b30 tx=0x7f9c54005880 comp rx=0 tx=0).stop 2023-10-16T12:30:53.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.177+0000 7f9c69049700 1 -- 172.21.15.16:0/3823827504 reap_dead start 2023-10-16T12:30:53.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.177+0000 7f9c6aaac700 1 -- 172.21.15.16:0/3823827504 shutdown_connections 2023-10-16T12:30:53.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.177+0000 7f9c6aaac700 1 -- 172.21.15.16:0/3823827504 >> 172.21.15.16:0/3823827504 conn(0x7f9c6406f430 msgr2=0x7f9c6406f810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:53.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.177+0000 7f9c6aaac700 1 -- 172.21.15.16:0/3823827504 shutdown_connections 2023-10-16T12:30:53.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.177+0000 7f9c6aaac700 1 -- 172.21.15.16:0/3823827504 wait complete. 2023-10-16T12:30:53.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.178+0000 7f9c6aaac700 1 Processor -- start 2023-10-16T12:30:53.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.178+0000 7f9c6aaac700 1 -- start start 2023-10-16T12:30:53.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.179+0000 7f9c6aaac700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9c54004850 0x7f9c64155980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:53.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.179+0000 7f9c6aaac700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9c641f4370 0x7f9c64154750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:53.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.179+0000 7f9c6aaac700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c64157960 0x7f9c64157df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:53.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.179+0000 7f9c6aaac700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c64158330 con 0x7f9c64157960 2023-10-16T12:30:53.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.179+0000 7f9c6aaac700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c64155ec0 con 0x7f9c641f4370 2023-10-16T12:30:53.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.179+0000 7f9c6aaac700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c64156a00 con 0x7f9c54004850 2023-10-16T12:30:53.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.179+0000 7f9c69049700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c64157960 0x7f9c64157df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:53.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.179+0000 7f9c69049700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c64157960 0x7f9c64157df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46530/0 (socket says 172.21.15.16:46530) 2023-10-16T12:30:53.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.179+0000 7f9c69049700 1 -- 172.21.15.16:0/4008259250 learned_addr learned my addr 172.21.15.16:0/4008259250 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:53.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.179+0000 7f9c68848700 1 --2- 172.21.15.16:0/4008259250 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9c54004850 0x7f9c64155980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:53.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.179+0000 7f9c63fff700 1 --2- 172.21.15.16:0/4008259250 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9c641f4370 0x7f9c64154750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:53.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.180+0000 7f9c69049700 1 -- 172.21.15.16:0/4008259250 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9c54004850 msgr2=0x7f9c64155980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:53.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.180+0000 7f9c69049700 1 --2- 172.21.15.16:0/4008259250 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9c54004850 0x7f9c64155980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:53.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.180+0000 7f9c69049700 1 -- 172.21.15.16:0/4008259250 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9c641f4370 msgr2=0x7f9c64154750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:53.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.180+0000 7f9c69049700 1 --2- 172.21.15.16:0/4008259250 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9c641f4370 0x7f9c64154750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:53.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.180+0000 7f9c69049700 1 -- 172.21.15.16:0/4008259250 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c641f4800 con 0x7f9c64157960 2023-10-16T12:30:53.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.180+0000 7f9c63fff700 1 --2- 172.21.15.16:0/4008259250 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9c641f4370 0x7f9c64154750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:30:53.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.180+0000 7f9c69049700 1 --2- 172.21.15.16:0/4008259250 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c64157960 0x7f9c64157df0 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7f9c64070520 tx=0x7f9c5c00e440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:53.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.180+0000 7f9c617fa700 1 -- 172.21.15.16:0/4008259250 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c5c00ecd0 con 0x7f9c64157960 2023-10-16T12:30:53.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.180+0000 7f9c617fa700 1 -- 172.21.15.16:0/4008259250 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c5c00ee30 con 0x7f9c64157960 2023-10-16T12:30:53.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.180+0000 7f9c6aaac700 1 -- 172.21.15.16:0/4008259250 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c641f4ae0 con 0x7f9c64157960 2023-10-16T12:30:53.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.180+0000 7f9c617fa700 1 -- 172.21.15.16:0/4008259250 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c5c013660 con 0x7f9c64157960 2023-10-16T12:30:53.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.180+0000 7f9c6aaac700 1 -- 172.21.15.16:0/4008259250 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c641f50a0 con 0x7f9c64157960 2023-10-16T12:30:53.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.181+0000 7f9c6aaac700 1 -- 172.21.15.16:0/4008259250 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c6404ec20 con 0x7f9c64157960 2023-10-16T12:30:53.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.182+0000 7f9c617fa700 1 -- 172.21.15.16:0/4008259250 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 83) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f9c5c0137c0 con 0x7f9c64157960 2023-10-16T12:30:53.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.182+0000 7f9c617fa700 1 --2- 172.21.15.16:0/4008259250 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f9c4806ad50 0x7f9c4806d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:53.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.182+0000 7f9c617fa700 1 -- 172.21.15.16:0/4008259250 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6902+0+0 (secure 0 0 0) 0x7f9c5c089cc0 con 0x7f9c64157960 2023-10-16T12:30:53.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.182+0000 7f9c68848700 1 --2- 172.21.15.16:0/4008259250 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f9c4806ad50 0x7f9c4806d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:53.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.182+0000 7f9c68848700 1 --2- 172.21.15.16:0/4008259250 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f9c4806ad50 0x7f9c4806d210 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f9c58007fd0 tx=0x7f9c58009040 comp rx=0 tx=0).ready entity=mgr.8973 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:53.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.186+0000 7f9c617fa700 1 -- 172.21.15.16:0/4008259250 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9c5c054c00 con 0x7f9c64157960 2023-10-16T12:30:53.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.367+0000 7f9c617fa700 1 -- 172.21.15.16:0/4008259250 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 84) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f9c5c054fd0 con 0x7f9c64157960 2023-10-16T12:30:53.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:53.392+0000 7f9c6aaac700 1 -- 172.21.15.16:0/4008259250 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f9c640b6260 con 0x7f9c64157960 2023-10-16T12:30:54.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:54.365+0000 7f9c617fa700 1 -- 172.21.15.16:0/4008259250 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v146) v1 ==== 181+0+0 (secure 0 0 0) 0x7f9c5c073630 con 0x7f9c64157960 2023-10-16T12:30:54.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:54.371+0000 7f9c6aaac700 1 -- 172.21.15.16:0/4008259250 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f9c4806ad50 msgr2=0x7f9c4806d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:54.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:54.371+0000 7f9c6aaac700 1 --2- 172.21.15.16:0/4008259250 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f9c4806ad50 0x7f9c4806d210 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f9c58007fd0 tx=0x7f9c58009040 comp rx=0 tx=0).stop 2023-10-16T12:30:54.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:54.371+0000 7f9c6aaac700 1 -- 172.21.15.16:0/4008259250 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c64157960 msgr2=0x7f9c64157df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:54.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:54.371+0000 7f9c6aaac700 1 --2- 172.21.15.16:0/4008259250 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c64157960 0x7f9c64157df0 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7f9c64070520 tx=0x7f9c5c00e440 comp rx=0 tx=0).stop 2023-10-16T12:30:54.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:54.372+0000 7f9c6aaac700 1 -- 172.21.15.16:0/4008259250 shutdown_connections 2023-10-16T12:30:54.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:54.372+0000 7f9c6aaac700 1 --2- 172.21.15.16:0/4008259250 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f9c4806ad50 0x7f9c4806d210 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:54.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:54.372+0000 7f9c6aaac700 1 --2- 172.21.15.16:0/4008259250 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c64157960 0x7f9c64157df0 unknown :-1 s=CLOSED pgs=1684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:54.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:54.372+0000 7f9c6aaac700 1 --2- 172.21.15.16:0/4008259250 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9c54004850 0x7f9c64155980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:54.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:54.372+0000 7f9c6aaac700 1 --2- 172.21.15.16:0/4008259250 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9c641f4370 0x7f9c64154750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:54.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:54.372+0000 7f9c6aaac700 1 -- 172.21.15.16:0/4008259250 >> 172.21.15.16:0/4008259250 conn(0x7f9c6406f430 msgr2=0x7f9c6406a7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:54.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:54.372+0000 7f9c6aaac700 1 -- 172.21.15.16:0/4008259250 shutdown_connections 2023-10-16T12:30:54.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:54.372+0000 7f9c6aaac700 1 -- 172.21.15.16:0/4008259250 wait complete. 2023-10-16T12:30:54.377 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' removed 2023-10-16T12:30:54.392 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-16T12:30:54.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:54.530+0000 7f7c213a9700 1 Processor -- start 2023-10-16T12:30:54.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:54.530+0000 7f7c213a9700 1 -- start start 2023-10-16T12:30:54.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:54.531+0000 7f7c213a9700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7c1c15d7a0 0x7f7c1c0b7240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:54.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:54.531+0000 7f7c213a9700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f7c1c0b7780 0x7f7c1c0ce260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:54.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:54.531+0000 7f7c213a9700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c1c0cebd0 con 0x7f7c1c0ce7a0 2023-10-16T12:30:54.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:54.531+0000 7f7c213a9700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c1c0b8980 con 0x7f7c1c15d7a0 2023-10-16T12:30:54.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:54.531+0000 7f7c213a9700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c1c0b91b0 con 0x7f7c1c0b7780 2023-10-16T12:30:54.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:54.531+0000 7f7c1affd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7c1c15d7a0 0x7f7c1c0b7240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:54.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:54.531+0000 7f7c1b7fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f7c1c0ce7a0 0x7f7c1c0b5010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35812/0 (socket says 172.21.15.16:35812) 2023-10-16T12:30:54.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:54.531+0000 7f7c1b7fe700 1 -- 172.21.15.16:0/844049564 learned_addr learned my addr 172.21.15.16:0/844049564 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:30:54.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:54.531+0000 7f7c1a7fc700 1 --2- 172.21.15.16:0/844049564 >> v2:172.21.15.49:3300/0 conn(0x7f7c1c0b7780 0x7f7c1c0ce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:54.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:54.531+0000 7f7c1affd700 1 -- 172.21.15.16:0/844049564 >> v2:172.21.15.49:3300/0 conn(0x7f7c1c0b7780 msgr2=0x7f7c1c0ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:54.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:54.531+0000 7f7c1affd700 1 --2- 172.21.15.16:0/844049564 >> v2:172.21.15.49:3300/0 conn(0x7f7c1c0b7780 0x7f7c1c0ce260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:54.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:54.531+0000 7f7c1affd700 1 -- 172.21.15.16:0/844049564 >> v1:172.21.15.16:6789/0 conn(0x7f7c1c0ce7a0 legacy=0x7f7c1c0b5010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:54.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:54.531+0000 7f7c19ffb700 1 -- 172.21.15.16:0/844049564 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3790378519 0 0) 0x7f7c1c0cebd0 con 0x7f7c1c0ce7a0 2023-10-16T12:30:54.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:54.531+0000 7f7c1affd700 1 -- 172.21.15.16:0/844049564 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c1c0b7d00 con 0x7f7c1c15d7a0 2023-10-16T12:30:54.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:54.531+0000 7f7c1a7fc700 1 --2- 172.21.15.16:0/844049564 >> v2:172.21.15.49:3300/0 conn(0x7f7c1c0b7780 0x7f7c1c0ce260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:30:54.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:54.532+0000 7f7c1affd700 1 --2- 172.21.15.16:0/844049564 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c1c15d7a0 0x7f7c1c0b7240 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f7c1c0ca610 tx=0x7f7c10009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93fd8c1005f90ff8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:54.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:54.532+0000 7f7c19ffb700 1 -- 172.21.15.16:0/844049564 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c10011830 con 0x7f7c1c15d7a0 2023-10-16T12:30:54.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:54.532+0000 7f7c19ffb700 1 -- 172.21.15.16:0/844049564 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c10011990 con 0x7f7c1c15d7a0 2023-10-16T12:30:54.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:54.532+0000 7f7c19ffb700 1 -- 172.21.15.16:0/844049564 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c10011d20 con 0x7f7c1c15d7a0 2023-10-16T12:30:54.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:54.532+0000 7f7c213a9700 1 -- 172.21.15.16:0/844049564 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c1c15d7a0 msgr2=0x7f7c1c0b7240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:54.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:54.532+0000 7f7c213a9700 1 --2- 172.21.15.16:0/844049564 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c1c15d7a0 0x7f7c1c0b7240 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f7c1c0ca610 tx=0x7f7c10009b20 comp rx=0 tx=0).stop 2023-10-16T12:30:54.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:54.532+0000 7f7c213a9700 1 -- 172.21.15.16:0/844049564 shutdown_connections 2023-10-16T12:30:54.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:54.532+0000 7f7c213a9700 1 --2- 172.21.15.16:0/844049564 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c1c15d7a0 0x7f7c1c0b7240 unknown :-1 s=CLOSED pgs=1686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:54.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:54.532+0000 7f7c213a9700 1 --2- 172.21.15.16:0/844049564 >> v2:172.21.15.49:3300/0 conn(0x7f7c1c0b7780 0x7f7c1c0ce260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:54.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:54.532+0000 7f7c213a9700 1 -- 172.21.15.16:0/844049564 >> 172.21.15.16:0/844049564 conn(0x7f7c1c072330 msgr2=0x7f7c1c072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:54.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:54.532+0000 7f7c213a9700 1 -- 172.21.15.16:0/844049564 shutdown_connections 2023-10-16T12:30:54.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:54.532+0000 7f7c213a9700 1 -- 172.21.15.16:0/844049564 wait complete. 2023-10-16T12:30:54.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:54.534+0000 7f7c213a9700 1 Processor -- start 2023-10-16T12:30:54.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:54.534+0000 7f7c213a9700 1 -- start start 2023-10-16T12:30:54.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:54.534+0000 7f7c213a9700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7c1c0b7780 0x7f7c1c152a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:54.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:54.535+0000 7f7c213a9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c1c0ce7a0 0x7f7c1c155550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:54.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:54.535+0000 7f7c213a9700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7c1c15d7a0 0x7f7c1c0c6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:54.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:54.535+0000 7f7c213a9700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c1c155fc0 con 0x7f7c1c0ce7a0 2023-10-16T12:30:54.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:54.535+0000 7f7c213a9700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c1c0ca790 con 0x7f7c1c15d7a0 2023-10-16T12:30:54.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:54.535+0000 7f7c213a9700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c1c0caa30 con 0x7f7c1c0b7780 2023-10-16T12:30:54.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:54.535+0000 7f7c1a7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c1c0ce7a0 0x7f7c1c155550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:54.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:54.535+0000 7f7c1a7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c1c0ce7a0 0x7f7c1c155550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46542/0 (socket says 172.21.15.16:46542) 2023-10-16T12:30:54.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:54.535+0000 7f7c1a7fc700 1 -- 172.21.15.16:0/2526134007 learned_addr learned my addr 172.21.15.16:0/2526134007 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:54.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:54.535+0000 7f7c1b7fe700 1 --2- 172.21.15.16:0/2526134007 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7c1c15d7a0 0x7f7c1c0c6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:54.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:54.535+0000 7f7c1affd700 1 --2- 172.21.15.16:0/2526134007 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7c1c0b7780 0x7f7c1c152a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:54.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:54.535+0000 7f7c1a7fc700 1 -- 172.21.15.16:0/2526134007 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7c1c0b7780 msgr2=0x7f7c1c152a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:54.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:54.535+0000 7f7c1a7fc700 1 --2- 172.21.15.16:0/2526134007 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7c1c0b7780 0x7f7c1c152a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:54.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:54.535+0000 7f7c1a7fc700 1 -- 172.21.15.16:0/2526134007 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7c1c15d7a0 msgr2=0x7f7c1c0c6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:54.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:54.535+0000 7f7c1a7fc700 1 --2- 172.21.15.16:0/2526134007 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7c1c15d7a0 0x7f7c1c0c6ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:54.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:54.535+0000 7f7c1a7fc700 1 -- 172.21.15.16:0/2526134007 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c100094a0 con 0x7f7c1c0ce7a0 2023-10-16T12:30:54.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:54.535+0000 7f7c1a7fc700 1 --2- 172.21.15.16:0/2526134007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c1c0ce7a0 0x7f7c1c155550 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c00c750 tx=0x7f7c0c00cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:54.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:54.535+0000 7f7c037fe700 1 -- 172.21.15.16:0/2526134007 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c0c011800 con 0x7f7c1c0ce7a0 2023-10-16T12:30:54.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:54.535+0000 7f7c037fe700 1 -- 172.21.15.16:0/2526134007 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c0c011960 con 0x7f7c1c0ce7a0 2023-10-16T12:30:54.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:54.535+0000 7f7c213a9700 1 -- 172.21.15.16:0/2526134007 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c1c0cad10 con 0x7f7c1c0ce7a0 2023-10-16T12:30:54.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:54.535+0000 7f7c037fe700 1 -- 172.21.15.16:0/2526134007 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c0c011cd0 con 0x7f7c1c0ce7a0 2023-10-16T12:30:54.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:54.535+0000 7f7c213a9700 1 -- 172.21.15.16:0/2526134007 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c1c152f60 con 0x7f7c1c0ce7a0 2023-10-16T12:30:54.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:54.536+0000 7f7c213a9700 1 -- 172.21.15.16:0/2526134007 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c1c0cb230 con 0x7f7c1c0ce7a0 2023-10-16T12:30:54.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:54.537+0000 7f7c037fe700 1 -- 172.21.15.16:0/2526134007 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 84) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f7c0c021350 con 0x7f7c1c0ce7a0 2023-10-16T12:30:54.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:54.537+0000 7f7c037fe700 1 --2- 172.21.15.16:0/2526134007 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f7c0806af90 0x7f7c0806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:54.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:54.537+0000 7f7c037fe700 1 -- 172.21.15.16:0/2526134007 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 6464+0+0 (secure 0 0 0) 0x7f7c0c089950 con 0x7f7c1c0ce7a0 2023-10-16T12:30:54.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:54.537+0000 7f7c1affd700 1 --2- 172.21.15.16:0/2526134007 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f7c0806af90 0x7f7c0806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:54.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:54.538+0000 7f7c1affd700 1 --2- 172.21.15.16:0/2526134007 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f7c0806af90 0x7f7c0806d450 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7c100090b0 tx=0x7f7c1001c040 comp rx=0 tx=0).ready entity=mgr.8973 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:54.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:54.542+0000 7f7c037fe700 1 -- 172.21.15.16:0/2526134007 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7c1c0cb230 con 0x7f7c1c0ce7a0 2023-10-16T12:30:54.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:54.745+0000 7f7c213a9700 1 -- 172.21.15.16:0/2526134007 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f7c1c0cb230 con 0x7f7c1c0ce7a0 2023-10-16T12:30:55.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.379+0000 7f7c037fe700 1 -- 172.21.15.16:0/2526134007 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v147) v1 ==== 169+0+0 (secure 0 0 0) 0x7f7c1c0cb230 con 0x7f7c1c0ce7a0 2023-10-16T12:30:55.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.386+0000 7f7c213a9700 1 -- 172.21.15.16:0/2526134007 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f7c0806af90 msgr2=0x7f7c0806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:55.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.386+0000 7f7c213a9700 1 --2- 172.21.15.16:0/2526134007 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f7c0806af90 0x7f7c0806d450 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7c100090b0 tx=0x7f7c1001c040 comp rx=0 tx=0).stop 2023-10-16T12:30:55.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.387+0000 7f7c213a9700 1 -- 172.21.15.16:0/2526134007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c1c0ce7a0 msgr2=0x7f7c1c155550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:55.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.387+0000 7f7c213a9700 1 --2- 172.21.15.16:0/2526134007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c1c0ce7a0 0x7f7c1c155550 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c00c750 tx=0x7f7c0c00cb10 comp rx=0 tx=0).stop 2023-10-16T12:30:55.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.387+0000 7f7c213a9700 1 -- 172.21.15.16:0/2526134007 shutdown_connections 2023-10-16T12:30:55.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.387+0000 7f7c213a9700 1 --2- 172.21.15.16:0/2526134007 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f7c0806af90 0x7f7c0806d450 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:55.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.387+0000 7f7c213a9700 1 --2- 172.21.15.16:0/2526134007 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7c1c15d7a0 0x7f7c1c0c6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:55.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.387+0000 7f7c213a9700 1 --2- 172.21.15.16:0/2526134007 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7c1c0b7780 0x7f7c1c152a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:55.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.387+0000 7f7c213a9700 1 --2- 172.21.15.16:0/2526134007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c1c0ce7a0 0x7f7c1c155550 unknown :-1 s=CLOSED pgs=1687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:55.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.387+0000 7f7c213a9700 1 -- 172.21.15.16:0/2526134007 >> 172.21.15.16:0/2526134007 conn(0x7f7c1c072330 msgr2=0x7f7c1c15f800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:55.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.388+0000 7f7c213a9700 1 -- 172.21.15.16:0/2526134007 shutdown_connections 2023-10-16T12:30:55.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.389+0000 7f7c213a9700 1 -- 172.21.15.16:0/2526134007 wait complete. 2023-10-16T12:30:55.395 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' removed 2023-10-16T12:30:55.406 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-16T12:30:55.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.539+0000 7f08c1a73700 1 Processor -- start 2023-10-16T12:30:55.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.539+0000 7f08c1a73700 1 -- start start 2023-10-16T12:30:55.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.539+0000 7f08c1a73700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f08bc06aef0 0x7f08bc0ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:55.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.539+0000 7f08c1a73700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f08bc06da30 0x7f08bc0adf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:55.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.539+0000 7f08c1a73700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08bc1613c0 con 0x7f08bc160fe0 2023-10-16T12:30:55.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.539+0000 7f08c1a73700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08bc0cf680 con 0x7f08bc06da30 2023-10-16T12:30:55.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.539+0000 7f08c1a73700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08bc0adb00 con 0x7f08bc06aef0 2023-10-16T12:30:55.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.540+0000 7f08bbfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f08bc06da30 0x7f08bc0adf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:55.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.540+0000 7f08bbfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f08bc06da30 0x7f08bc0adf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46552/0 (socket says 172.21.15.16:46552) 2023-10-16T12:30:55.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.540+0000 7f08bbfff700 1 -- 172.21.15.16:0/3321996541 learned_addr learned my addr 172.21.15.16:0/3321996541 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:55.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.540+0000 7f08baffd700 1 --2- 172.21.15.16:0/3321996541 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f08bc06aef0 0x7f08bc0ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:55.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.540+0000 7f08bbfff700 1 -- 172.21.15.16:0/3321996541 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f08bc06aef0 msgr2=0x7f08bc0ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:55.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.540+0000 7f08bbfff700 1 --2- 172.21.15.16:0/3321996541 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f08bc06aef0 0x7f08bc0ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:55.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.540+0000 7f08bbfff700 1 -- 172.21.15.16:0/3321996541 >> v1:172.21.15.49:6789/0 conn(0x7f08bc160fe0 legacy=0x7f08bc06a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:55.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.540+0000 7f08bbfff700 1 -- 172.21.15.16:0/3321996541 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08bc0acef0 con 0x7f08bc06da30 2023-10-16T12:30:55.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.540+0000 7f08bbfff700 1 --2- 172.21.15.16:0/3321996541 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08bc06da30 0x7f08bc0adf90 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7f08b4009e50 tx=0x7f08b400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f66e0d0f8c8b3bf server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:55.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.540+0000 7f08ba7fc700 1 -- 172.21.15.16:0/3321996541 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08b400e070 con 0x7f08bc06da30 2023-10-16T12:30:55.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.540+0000 7f08ba7fc700 1 -- 172.21.15.16:0/3321996541 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08b4003a20 con 0x7f08bc06da30 2023-10-16T12:30:55.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.540+0000 7f08ba7fc700 1 -- 172.21.15.16:0/3321996541 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08b4003dd0 con 0x7f08bc06da30 2023-10-16T12:30:55.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.541+0000 7f08c1a73700 1 -- 172.21.15.16:0/3321996541 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08bc06da30 msgr2=0x7f08bc0adf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:55.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.541+0000 7f08c1a73700 1 --2- 172.21.15.16:0/3321996541 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08bc06da30 0x7f08bc0adf90 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7f08b4009e50 tx=0x7f08b400b330 comp rx=0 tx=0).stop 2023-10-16T12:30:55.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.541+0000 7f08c1a73700 1 -- 172.21.15.16:0/3321996541 shutdown_connections 2023-10-16T12:30:55.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.541+0000 7f08c1a73700 1 --2- 172.21.15.16:0/3321996541 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08bc06da30 0x7f08bc0adf90 unknown :-1 s=CLOSED pgs=1688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:55.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.541+0000 7f08c1a73700 1 --2- 172.21.15.16:0/3321996541 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f08bc06aef0 0x7f08bc0ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:55.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.541+0000 7f08c1a73700 1 -- 172.21.15.16:0/3321996541 >> 172.21.15.16:0/3321996541 conn(0x7f08bc0b1ed0 msgr2=0x7f08bc0b22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:55.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.541+0000 7f08c1a73700 1 -- 172.21.15.16:0/3321996541 shutdown_connections 2023-10-16T12:30:55.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.541+0000 7f08c1a73700 1 -- 172.21.15.16:0/3321996541 wait complete. 2023-10-16T12:30:55.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.542+0000 7f08c1a73700 1 Processor -- start 2023-10-16T12:30:55.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.542+0000 7f08c1a73700 1 -- start start 2023-10-16T12:30:55.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.543+0000 7f08c1a73700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f08bc06aef0 0x7f08bc0b0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:55.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.543+0000 7f08c1a73700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08bc06da30 0x7f08bc147340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:55.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.543+0000 7f08c1a73700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f08bc160fe0 0x7f08bc147880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:55.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.543+0000 7f08c1a73700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08bc0c6a70 con 0x7f08bc06da30 2023-10-16T12:30:55.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.543+0000 7f08c1a73700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08bc0b4e90 con 0x7f08bc160fe0 2023-10-16T12:30:55.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.543+0000 7f08c1a73700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08bc144df0 con 0x7f08bc06aef0 2023-10-16T12:30:55.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.543+0000 7f08baffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08bc06da30 0x7f08bc147340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:55.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.543+0000 7f08baffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08bc06da30 0x7f08bc147340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46556/0 (socket says 172.21.15.16:46556) 2023-10-16T12:30:55.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.543+0000 7f08baffd700 1 -- 172.21.15.16:0/294286201 learned_addr learned my addr 172.21.15.16:0/294286201 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:55.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.543+0000 7f08bb7fe700 1 --2- 172.21.15.16:0/294286201 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f08bc06aef0 0x7f08bc0b0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:55.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.543+0000 7f08bbfff700 1 --2- 172.21.15.16:0/294286201 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f08bc160fe0 0x7f08bc147880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:55.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.543+0000 7f08baffd700 1 -- 172.21.15.16:0/294286201 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f08bc06aef0 msgr2=0x7f08bc0b0840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:55.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.543+0000 7f08baffd700 1 --2- 172.21.15.16:0/294286201 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f08bc06aef0 0x7f08bc0b0840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:55.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.543+0000 7f08baffd700 1 -- 172.21.15.16:0/294286201 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f08bc160fe0 msgr2=0x7f08bc147880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:55.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.543+0000 7f08baffd700 1 --2- 172.21.15.16:0/294286201 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f08bc160fe0 0x7f08bc147880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:55.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.543+0000 7f08baffd700 1 -- 172.21.15.16:0/294286201 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08b4009b30 con 0x7f08bc06da30 2023-10-16T12:30:55.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.544+0000 7f08baffd700 1 --2- 172.21.15.16:0/294286201 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08bc06da30 0x7f08bc147340 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7f08b000d800 tx=0x7f08b000db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:55.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.544+0000 7f08a3fff700 1 -- 172.21.15.16:0/294286201 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08b00125f0 con 0x7f08bc06da30 2023-10-16T12:30:55.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.544+0000 7f08a3fff700 1 -- 172.21.15.16:0/294286201 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08b0012750 con 0x7f08bc06da30 2023-10-16T12:30:55.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.544+0000 7f08c1a73700 1 -- 172.21.15.16:0/294286201 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08bc0b50c0 con 0x7f08bc06da30 2023-10-16T12:30:55.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.544+0000 7f08a3fff700 1 -- 172.21.15.16:0/294286201 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08b0012b00 con 0x7f08bc06da30 2023-10-16T12:30:55.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.544+0000 7f08c1a73700 1 -- 172.21.15.16:0/294286201 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08bc0b5680 con 0x7f08bc06da30 2023-10-16T12:30:55.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.545+0000 7f08c1a73700 1 -- 172.21.15.16:0/294286201 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08bc0b5cd0 con 0x7f08bc06da30 2023-10-16T12:30:55.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.545+0000 7f08a3fff700 1 -- 172.21.15.16:0/294286201 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 85) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f08b0012c60 con 0x7f08bc06da30 2023-10-16T12:30:55.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.546+0000 7f08a3fff700 1 --2- 172.21.15.16:0/294286201 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f08a806b020 0x7f08a806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:55.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.546+0000 7f08a3fff700 1 -- 172.21.15.16:0/294286201 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6050+0+0 (secure 0 0 0) 0x7f08b00898b0 con 0x7f08bc06da30 2023-10-16T12:30:55.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.546+0000 7f08bb7fe700 1 --2- 172.21.15.16:0/294286201 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f08a806b020 0x7f08a806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:55.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.546+0000 7f08bb7fe700 1 --2- 172.21.15.16:0/294286201 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f08a806b020 0x7f08a806d4e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f08ac0050c0 tx=0x7f08ac007f10 comp rx=0 tx=0).ready entity=mgr.8973 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:55.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.550+0000 7f08a3fff700 1 -- 172.21.15.16:0/294286201 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f08b00546f0 con 0x7f08bc06da30 2023-10-16T12:30:55.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.753+0000 7f08c1a73700 1 -- 172.21.15.16:0/294286201 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f08bc04ebb0 con 0x7f08bc06da30 2023-10-16T12:30:55.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.756+0000 7f08a3fff700 1 -- 172.21.15.16:0/294286201 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v49) v1 ==== 115+0+0 (secure 0 0 0) 0x7f08b0057d10 con 0x7f08bc06da30 2023-10-16T12:30:55.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.759+0000 7f08c1a73700 1 -- 172.21.15.16:0/294286201 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f08a806b020 msgr2=0x7f08a806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:55.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.759+0000 7f08c1a73700 1 --2- 172.21.15.16:0/294286201 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f08a806b020 0x7f08a806d4e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f08ac0050c0 tx=0x7f08ac007f10 comp rx=0 tx=0).stop 2023-10-16T12:30:55.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.759+0000 7f08c1a73700 1 -- 172.21.15.16:0/294286201 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08bc06da30 msgr2=0x7f08bc147340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:55.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.759+0000 7f08c1a73700 1 --2- 172.21.15.16:0/294286201 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08bc06da30 0x7f08bc147340 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7f08b000d800 tx=0x7f08b000db10 comp rx=0 tx=0).stop 2023-10-16T12:30:55.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.759+0000 7f08c1a73700 1 -- 172.21.15.16:0/294286201 shutdown_connections 2023-10-16T12:30:55.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.759+0000 7f08c1a73700 1 --2- 172.21.15.16:0/294286201 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f08a806b020 0x7f08a806d4e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:55.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.759+0000 7f08c1a73700 1 --2- 172.21.15.16:0/294286201 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f08bc160fe0 0x7f08bc147880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:55.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.759+0000 7f08c1a73700 1 --2- 172.21.15.16:0/294286201 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f08bc06aef0 0x7f08bc0b0840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:55.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.759+0000 7f08c1a73700 1 --2- 172.21.15.16:0/294286201 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08bc06da30 0x7f08bc147340 unknown :-1 s=CLOSED pgs=1689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:55.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.759+0000 7f08c1a73700 1 -- 172.21.15.16:0/294286201 >> 172.21.15.16:0/294286201 conn(0x7f08bc0b1ed0 msgr2=0x7f08bc0cde70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:55.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.759+0000 7f08c1a73700 1 -- 172.21.15.16:0/294286201 shutdown_connections 2023-10-16T12:30:55.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.759+0000 7f08c1a73700 1 -- 172.21.15.16:0/294286201 wait complete. 2023-10-16T12:30:55.773 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-16T12:30:55.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.899+0000 7ff671c4f700 1 Processor -- start 2023-10-16T12:30:55.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.899+0000 7ff671c4f700 1 -- start start 2023-10-16T12:30:55.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.899+0000 7ff671c4f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff66c0cb880 0x7ff66c0cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:55.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.899+0000 7ff671c4f700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff66c0cef70 con 0x7ff66c06da60 2023-10-16T12:30:55.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.899+0000 7ff671c4f700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff66c0ce360 con 0x7ff66c15b070 2023-10-16T12:30:55.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.899+0000 7ff671c4f700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff66c0ce640 con 0x7ff66c0cb880 2023-10-16T12:30:55.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.900+0000 7ff66bfff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff66c06da60 0x7ff66c0cf400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35840/0 (socket says 172.21.15.16:35840) 2023-10-16T12:30:55.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.900+0000 7ff66bfff700 1 -- 172.21.15.16:0/1350222395 learned_addr learned my addr 172.21.15.16:0/1350222395 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:30:55.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.900+0000 7ff66affd700 1 --2- 172.21.15.16:0/1350222395 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff66c0cb880 0x7ff66c0cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:55.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.900+0000 7ff66a7fc700 1 -- 172.21.15.16:0/1350222395 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 741977219 0 0) 0x7ff66c0cef70 con 0x7ff66c06da60 2023-10-16T12:30:55.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.900+0000 7ff66a7fc700 1 -- 172.21.15.16:0/1350222395 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6540035b0 con 0x7ff66c06da60 2023-10-16T12:30:55.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.900+0000 7ff66a7fc700 1 -- 172.21.15.16:0/1350222395 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 281880479 0 0) 0x7ff66c0ce360 con 0x7ff66c15b070 2023-10-16T12:30:55.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.900+0000 7ff66a7fc700 1 -- 172.21.15.16:0/1350222395 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff66c0cef70 con 0x7ff66c15b070 2023-10-16T12:30:55.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.900+0000 7ff66a7fc700 1 -- 172.21.15.16:0/1350222395 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3849799554 0 0) 0x7ff6540035b0 con 0x7ff66c06da60 2023-10-16T12:30:55.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.900+0000 7ff66a7fc700 1 -- 172.21.15.16:0/1350222395 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff66c0cb880 msgr2=0x7ff66c0cbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:55.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.900+0000 7ff66a7fc700 1 --2- 172.21.15.16:0/1350222395 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff66c0cb880 0x7ff66c0cbd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:55.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.900+0000 7ff66a7fc700 1 -- 172.21.15.16:0/1350222395 >> v1:172.21.15.49:6789/0 conn(0x7ff66c15b070 legacy=0x7ff66c15b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:55.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.900+0000 7ff66a7fc700 1 -- 172.21.15.16:0/1350222395 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff66c0ce920 con 0x7ff66c06da60 2023-10-16T12:30:55.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.900+0000 7ff66a7fc700 1 -- 172.21.15.16:0/1350222395 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7ff6640019f0 con 0x7ff66c06da60 2023-10-16T12:30:55.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.900+0000 7ff66a7fc700 1 -- 172.21.15.16:0/1350222395 >> v1:172.21.15.16:6789/0 conn(0x7ff66c06da60 legacy=0x7ff66c0cf400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:55.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.900+0000 7ff66a7fc700 1 --2- 172.21.15.16:0/1350222395 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff654003bc0 0x7ff654003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:55.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.900+0000 7ff66a7fc700 1 --2- 172.21.15.16:0/1350222395 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff66c0cb880 0x7ff654004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:55.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.900+0000 7ff66a7fc700 1 --2- 172.21.15.16:0/1350222395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff654004d90 0x7ff654005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:55.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.900+0000 7ff66a7fc700 1 -- 172.21.15.16:0/1350222395 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff66c0ce640 con 0x7ff654004d90 2023-10-16T12:30:55.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.900+0000 7ff66a7fc700 1 -- 172.21.15.16:0/1350222395 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff65400bb10 con 0x7ff654003bc0 2023-10-16T12:30:55.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.900+0000 7ff66a7fc700 1 -- 172.21.15.16:0/1350222395 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff65400bcd0 con 0x7ff66c0cb880 2023-10-16T12:30:55.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.900+0000 7ff66bfff700 1 --2- 172.21.15.16:0/1350222395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff654004d90 0x7ff654005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:55.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.900+0000 7ff66b7fe700 1 --2- 172.21.15.16:0/1350222395 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff654003bc0 0x7ff654003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:55.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.901+0000 7ff66affd700 1 --2- 172.21.15.16:0/1350222395 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff66c0cb880 0x7ff654004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:55.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.901+0000 7ff66bfff700 1 -- 172.21.15.16:0/1350222395 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff66c0cb880 msgr2=0x7ff654004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:55.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.901+0000 7ff66bfff700 1 --2- 172.21.15.16:0/1350222395 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff66c0cb880 0x7ff654004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:55.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.901+0000 7ff66bfff700 1 -- 172.21.15.16:0/1350222395 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff654003bc0 msgr2=0x7ff654003fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:55.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.901+0000 7ff66bfff700 1 --2- 172.21.15.16:0/1350222395 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff654003bc0 0x7ff654003fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:55.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.901+0000 7ff66bfff700 1 -- 172.21.15.16:0/1350222395 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6540035b0 con 0x7ff654004d90 2023-10-16T12:30:55.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.901+0000 7ff66b7fe700 1 --2- 172.21.15.16:0/1350222395 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff654003bc0 0x7ff654003fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:30:55.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.901+0000 7ff66bfff700 1 --2- 172.21.15.16:0/1350222395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff654004d90 0x7ff654005210 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7ff66400a7c0 tx=0x7ff654005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc51826d65512ea2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:55.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.901+0000 7ff66a7fc700 1 -- 172.21.15.16:0/1350222395 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff664012600 con 0x7ff654004d90 2023-10-16T12:30:55.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.901+0000 7ff66a7fc700 1 -- 172.21.15.16:0/1350222395 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff664012780 con 0x7ff654004d90 2023-10-16T12:30:55.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.901+0000 7ff66a7fc700 1 -- 172.21.15.16:0/1350222395 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff664012b30 con 0x7ff654004d90 2023-10-16T12:30:55.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.901+0000 7ff671c4f700 1 -- 172.21.15.16:0/1350222395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff654004d90 msgr2=0x7ff654005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:55.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.901+0000 7ff671c4f700 1 --2- 172.21.15.16:0/1350222395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff654004d90 0x7ff654005210 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7ff66400a7c0 tx=0x7ff654005880 comp rx=0 tx=0).stop 2023-10-16T12:30:55.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.901+0000 7ff66bfff700 1 -- 172.21.15.16:0/1350222395 reap_dead start 2023-10-16T12:30:55.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.902+0000 7ff671c4f700 1 -- 172.21.15.16:0/1350222395 shutdown_connections 2023-10-16T12:30:55.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.902+0000 7ff671c4f700 1 -- 172.21.15.16:0/1350222395 >> 172.21.15.16:0/1350222395 conn(0x7ff66c0b1f40 msgr2=0x7ff66c0b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:55.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.902+0000 7ff671c4f700 1 -- 172.21.15.16:0/1350222395 shutdown_connections 2023-10-16T12:30:55.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.902+0000 7ff671c4f700 1 -- 172.21.15.16:0/1350222395 wait complete. 2023-10-16T12:30:55.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.903+0000 7ff671c4f700 1 Processor -- start 2023-10-16T12:30:55.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.903+0000 7ff671c4f700 1 -- start start 2023-10-16T12:30:55.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.904+0000 7ff671c4f700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff66c1ee440 0x7ff66c155ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:55.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.904+0000 7ff671c4f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff66c0c6390 0x7ff66c0c6810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:55.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.904+0000 7ff671c4f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff66c0b4fc0 0x7ff66c0b5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:55.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.904+0000 7ff671c4f700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff66c0b4200 con 0x7ff66c0c6390 2023-10-16T12:30:55.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.904+0000 7ff671c4f700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff66c0b44e0 con 0x7ff66c1ee440 2023-10-16T12:30:55.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.904+0000 7ff671c4f700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff66c0b47c0 con 0x7ff66c0b4fc0 2023-10-16T12:30:55.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.904+0000 7ff66affd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff66c0c6390 0x7ff66c0c6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:55.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.904+0000 7ff66affd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff66c0c6390 0x7ff66c0c6810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46576/0 (socket says 172.21.15.16:46576) 2023-10-16T12:30:55.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.904+0000 7ff66affd700 1 -- 172.21.15.16:0/543036766 learned_addr learned my addr 172.21.15.16:0/543036766 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:55.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.904+0000 7ff66bfff700 1 --2- 172.21.15.16:0/543036766 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff66c0b4fc0 0x7ff66c0b5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:55.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.904+0000 7ff66b7fe700 1 --2- 172.21.15.16:0/543036766 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff66c1ee440 0x7ff66c155ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:55.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.904+0000 7ff66affd700 1 -- 172.21.15.16:0/543036766 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff66c0b4fc0 msgr2=0x7ff66c0b5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:55.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.904+0000 7ff66affd700 1 --2- 172.21.15.16:0/543036766 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff66c0b4fc0 0x7ff66c0b5470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:55.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.904+0000 7ff66affd700 1 -- 172.21.15.16:0/543036766 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff66c1ee440 msgr2=0x7ff66c155ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:55.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.904+0000 7ff66affd700 1 --2- 172.21.15.16:0/543036766 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff66c1ee440 0x7ff66c155ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:55.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.904+0000 7ff66affd700 1 -- 172.21.15.16:0/543036766 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff66c0b4aa0 con 0x7ff66c0c6390 2023-10-16T12:30:55.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.904+0000 7ff66affd700 1 --2- 172.21.15.16:0/543036766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff66c0c6390 0x7ff66c0c6810 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7ff660011660 tx=0x7ff660011a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:55.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.904+0000 7ff653fff700 1 -- 172.21.15.16:0/543036766 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff660015870 con 0x7ff66c0c6390 2023-10-16T12:30:55.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.904+0000 7ff653fff700 1 -- 172.21.15.16:0/543036766 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6600159d0 con 0x7ff66c0c6390 2023-10-16T12:30:55.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.904+0000 7ff671c4f700 1 -- 172.21.15.16:0/543036766 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff66c0b59b0 con 0x7ff66c0c6390 2023-10-16T12:30:55.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.904+0000 7ff653fff700 1 -- 172.21.15.16:0/543036766 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff660015d40 con 0x7ff66c0c6390 2023-10-16T12:30:55.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.905+0000 7ff671c4f700 1 -- 172.21.15.16:0/543036766 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff66c156220 con 0x7ff66c0c6390 2023-10-16T12:30:55.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.905+0000 7ff671c4f700 1 -- 172.21.15.16:0/543036766 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff66c0b4e10 con 0x7ff66c0c6390 2023-10-16T12:30:55.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.906+0000 7ff653fff700 1 -- 172.21.15.16:0/543036766 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 85) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff660013400 con 0x7ff66c0c6390 2023-10-16T12:30:55.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.906+0000 7ff653fff700 1 --2- 172.21.15.16:0/543036766 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7ff65806b020 0x7ff65806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:55.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.906+0000 7ff653fff700 1 -- 172.21.15.16:0/543036766 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6050+0+0 (secure 0 0 0) 0x7ff660089fc0 con 0x7ff66c0c6390 2023-10-16T12:30:55.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.906+0000 7ff66b7fe700 1 --2- 172.21.15.16:0/543036766 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7ff65806b020 0x7ff65806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:55.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.907+0000 7ff66b7fe700 1 --2- 172.21.15.16:0/543036766 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7ff65806b020 0x7ff65806d4e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff66c0b6010 tx=0x7ff65c009040 comp rx=0 tx=0).ready entity=mgr.8973 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:55.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:55.911+0000 7ff653fff700 1 -- 172.21.15.16:0/543036766 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff6600bcfe0 con 0x7ff66c0c6390 2023-10-16T12:30:56.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.124+0000 7ff671c4f700 1 -- 172.21.15.16:0/543036766 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7ff66c0b4340 con 0x7ff66c0c6390 2023-10-16T12:30:56.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.127+0000 7ff653fff700 1 -- 172.21.15.16:0/543036766 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v50) v1 ==== 104+0+0 (secure 0 0 0) 0x7ff66c0b4340 con 0x7ff66c0c6390 2023-10-16T12:30:56.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.129+0000 7ff671c4f700 1 -- 172.21.15.16:0/543036766 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7ff65806b020 msgr2=0x7ff65806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:56.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.129+0000 7ff671c4f700 1 --2- 172.21.15.16:0/543036766 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7ff65806b020 0x7ff65806d4e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff66c0b6010 tx=0x7ff65c009040 comp rx=0 tx=0).stop 2023-10-16T12:30:56.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.129+0000 7ff671c4f700 1 -- 172.21.15.16:0/543036766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff66c0c6390 msgr2=0x7ff66c0c6810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:56.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.129+0000 7ff671c4f700 1 --2- 172.21.15.16:0/543036766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff66c0c6390 0x7ff66c0c6810 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7ff660011660 tx=0x7ff660011a20 comp rx=0 tx=0).stop 2023-10-16T12:30:56.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.129+0000 7ff671c4f700 1 -- 172.21.15.16:0/543036766 shutdown_connections 2023-10-16T12:30:56.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.129+0000 7ff671c4f700 1 --2- 172.21.15.16:0/543036766 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7ff65806b020 0x7ff65806d4e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:56.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.130+0000 7ff671c4f700 1 --2- 172.21.15.16:0/543036766 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff66c0b4fc0 0x7ff66c0b5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:56.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.130+0000 7ff671c4f700 1 --2- 172.21.15.16:0/543036766 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff66c1ee440 0x7ff66c155ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:56.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.130+0000 7ff671c4f700 1 --2- 172.21.15.16:0/543036766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff66c0c6390 0x7ff66c0c6810 unknown :-1 s=CLOSED pgs=1692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:56.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.130+0000 7ff671c4f700 1 -- 172.21.15.16:0/543036766 >> 172.21.15.16:0/543036766 conn(0x7ff66c0b1f40 msgr2=0x7ff66c15e030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:56.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.130+0000 7ff671c4f700 1 -- 172.21.15.16:0/543036766 shutdown_connections 2023-10-16T12:30:56.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.130+0000 7ff671c4f700 1 -- 172.21.15.16:0/543036766 wait complete. 2023-10-16T12:30:56.144 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-16T12:30:56.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.269+0000 7f0901047700 1 Processor -- start 2023-10-16T12:30:56.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.269+0000 7f0901047700 1 -- start start 2023-10-16T12:30:56.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.270+0000 7f0901047700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f08fc0ca9b0 0x7f08fc06e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:56.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.270+0000 7f0901047700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f08fc15bb50 0x7f08fc0cad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:56.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.270+0000 7f0901047700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08fc0cc920 con 0x7f08fc15cd10 2023-10-16T12:30:56.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.270+0000 7f0901047700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08fc06ebf0 con 0x7f08fc0ca9b0 2023-10-16T12:30:56.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.270+0000 7f0901047700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08fc0cb2d0 con 0x7f08fc15bb50 2023-10-16T12:30:56.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.270+0000 7f08fad9d700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f08fc0ca9b0 0x7f08fc06e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:56.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.270+0000 7f08fa59c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f08fc15bb50 0x7f08fc0cad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:56.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.270+0000 7f08fad9d700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f08fc0ca9b0 0x7f08fc06e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:45644/0 (socket says 172.21.15.16:45644) 2023-10-16T12:30:56.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.270+0000 7f08fad9d700 1 -- 172.21.15.16:0/1394419884 learned_addr learned my addr 172.21.15.16:0/1394419884 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:56.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.270+0000 7f08fa59c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f08fc15bb50 0x7f08fc0cad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:34842/0 (socket says 172.21.15.16:34842) 2023-10-16T12:30:56.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.270+0000 7f08fb59e700 1 --1- 172.21.15.16:0/1394419884 >> v1:172.21.15.49:6789/0 conn(0x7f08fc15cd10 0x7f08fc0cd0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:43042/0 (socket says 172.21.15.16:43042) 2023-10-16T12:30:56.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.271+0000 7f08fad9d700 1 -- 172.21.15.16:0/1394419884 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f08fc15bb50 msgr2=0x7f08fc0cad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:56.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.271+0000 7f08fad9d700 1 --2- 172.21.15.16:0/1394419884 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f08fc15bb50 0x7f08fc0cad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:56.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.271+0000 7f08fad9d700 1 -- 172.21.15.16:0/1394419884 >> v1:172.21.15.49:6789/0 conn(0x7f08fc15cd10 legacy=0x7f08fc0cd0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:56.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.271+0000 7f08fad9d700 1 -- 172.21.15.16:0/1394419884 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08fc15c330 con 0x7f08fc0ca9b0 2023-10-16T12:30:56.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.271+0000 7f08f9d9b700 1 -- 172.21.15.16:0/1394419884 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2118767021 0 0) 0x7f08fc0cc920 con 0x7f08fc15cd10 2023-10-16T12:30:56.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.271+0000 7f08fa59c700 1 --2- 172.21.15.16:0/1394419884 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f08fc15bb50 0x7f08fc0cad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:30:56.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.271+0000 7f08fad9d700 1 --2- 172.21.15.16:0/1394419884 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f08fc0ca9b0 0x7f08fc06e6b0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f08ec006fd0 tx=0x7f08ec009bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7a549d94291a5027 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:56.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.271+0000 7f08f9d9b700 1 -- 172.21.15.16:0/1394419884 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08ec005e30 con 0x7f08fc0ca9b0 2023-10-16T12:30:56.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.271+0000 7f08f9d9b700 1 -- 172.21.15.16:0/1394419884 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08ec004030 con 0x7f08fc0ca9b0 2023-10-16T12:30:56.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.271+0000 7f08f9d9b700 1 -- 172.21.15.16:0/1394419884 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08ec0043e0 con 0x7f08fc0ca9b0 2023-10-16T12:30:56.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.271+0000 7f0901047700 1 -- 172.21.15.16:0/1394419884 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f08fc0ca9b0 msgr2=0x7f08fc06e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:56.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.271+0000 7f0901047700 1 --2- 172.21.15.16:0/1394419884 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f08fc0ca9b0 0x7f08fc06e6b0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f08ec006fd0 tx=0x7f08ec009bd0 comp rx=0 tx=0).stop 2023-10-16T12:30:56.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.272+0000 7f0901047700 1 -- 172.21.15.16:0/1394419884 shutdown_connections 2023-10-16T12:30:56.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.272+0000 7f0901047700 1 --2- 172.21.15.16:0/1394419884 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f08fc0ca9b0 0x7f08fc06e6b0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:56.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.272+0000 7f0901047700 1 --2- 172.21.15.16:0/1394419884 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f08fc15bb50 0x7f08fc0cad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:56.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.272+0000 7f0901047700 1 -- 172.21.15.16:0/1394419884 >> 172.21.15.16:0/1394419884 conn(0x7f08fc0b1ed0 msgr2=0x7f08fc0b22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:56.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.272+0000 7f0901047700 1 -- 172.21.15.16:0/1394419884 shutdown_connections 2023-10-16T12:30:56.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.272+0000 7f0901047700 1 -- 172.21.15.16:0/1394419884 wait complete. 2023-10-16T12:30:56.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.273+0000 7f0901047700 1 Processor -- start 2023-10-16T12:30:56.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.273+0000 7f0901047700 1 -- start start 2023-10-16T12:30:56.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.274+0000 7f0901047700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f08fc0ca9b0 0x7f08fc0cf9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:56.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.274+0000 7f0901047700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08fc15bb50 0x7f08fc153a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:56.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.274+0000 7f0901047700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f08fc15cd10 0x7f08fc0ada90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:56.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.274+0000 7f0901047700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08fc0cf5d0 con 0x7f08fc15bb50 2023-10-16T12:30:56.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.274+0000 7f0901047700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08fc0cff30 con 0x7f08fc0ca9b0 2023-10-16T12:30:56.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.274+0000 7f0901047700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08fc0ce950 con 0x7f08fc15cd10 2023-10-16T12:30:56.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.274+0000 7f08fa59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08fc15bb50 0x7f08fc153a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:56.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.274+0000 7f08fa59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08fc15bb50 0x7f08fc153a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46590/0 (socket says 172.21.15.16:46590) 2023-10-16T12:30:56.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.274+0000 7f08fa59c700 1 -- 172.21.15.16:0/362769349 learned_addr learned my addr 172.21.15.16:0/362769349 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:56.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.274+0000 7f08fb59e700 1 --2- 172.21.15.16:0/362769349 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f08fc15cd10 0x7f08fc0ada90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:56.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.274+0000 7f08fad9d700 1 --2- 172.21.15.16:0/362769349 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f08fc0ca9b0 0x7f08fc0cf9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:56.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.274+0000 7f08fa59c700 1 -- 172.21.15.16:0/362769349 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f08fc15cd10 msgr2=0x7f08fc0ada90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:56.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.274+0000 7f08fa59c700 1 --2- 172.21.15.16:0/362769349 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f08fc15cd10 0x7f08fc0ada90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:56.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.274+0000 7f08fa59c700 1 -- 172.21.15.16:0/362769349 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f08fc0ca9b0 msgr2=0x7f08fc0cf9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:56.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.274+0000 7f08fa59c700 1 --2- 172.21.15.16:0/362769349 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f08fc0ca9b0 0x7f08fc0cf9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:56.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.274+0000 7f08fa59c700 1 -- 172.21.15.16:0/362769349 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08e40094a0 con 0x7f08fc15bb50 2023-10-16T12:30:56.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.275+0000 7f08fa59c700 1 --2- 172.21.15.16:0/362769349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08fc15bb50 0x7f08fc153a50 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7f08e400ebe0 tx=0x7f08e400eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:56.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.275+0000 7f08f37fe700 1 -- 172.21.15.16:0/362769349 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08e400c850 con 0x7f08fc15bb50 2023-10-16T12:30:56.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.275+0000 7f08f37fe700 1 -- 172.21.15.16:0/362769349 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08e400c9b0 con 0x7f08fc15bb50 2023-10-16T12:30:56.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.275+0000 7f0901047700 1 -- 172.21.15.16:0/362769349 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08ec009500 con 0x7f08fc15bb50 2023-10-16T12:30:56.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.275+0000 7f08f37fe700 1 -- 172.21.15.16:0/362769349 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08e400cd40 con 0x7f08fc15bb50 2023-10-16T12:30:56.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.275+0000 7f0901047700 1 -- 172.21.15.16:0/362769349 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08fc0cef10 con 0x7f08fc15bb50 2023-10-16T12:30:56.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.276+0000 7f0901047700 1 -- 172.21.15.16:0/362769349 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08fc04ebb0 con 0x7f08fc15bb50 2023-10-16T12:30:56.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.276+0000 7f08f37fe700 1 -- 172.21.15.16:0/362769349 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 85) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f08e4013470 con 0x7f08fc15bb50 2023-10-16T12:30:56.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.277+0000 7f08f37fe700 1 --2- 172.21.15.16:0/362769349 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f08dc06b020 0x7f08dc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:56.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.277+0000 7f08f37fe700 1 -- 172.21.15.16:0/362769349 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6050+0+0 (secure 0 0 0) 0x7f08e40898a0 con 0x7f08fc15bb50 2023-10-16T12:30:56.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.277+0000 7f08fad9d700 1 --2- 172.21.15.16:0/362769349 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f08dc06b020 0x7f08dc06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:56.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.277+0000 7f08fad9d700 1 --2- 172.21.15.16:0/362769349 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f08dc06b020 0x7f08dc06d4e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f08ec006fd0 tx=0x7f08ec008070 comp rx=0 tx=0).ready entity=mgr.8973 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:56.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.281+0000 7f08f37fe700 1 -- 172.21.15.16:0/362769349 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f08e4054770 con 0x7f08fc15bb50 2023-10-16T12:30:56.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.491+0000 7f0901047700 1 -- 172.21.15.16:0/362769349 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f08fc15b470 con 0x7f08fc15bb50 2023-10-16T12:30:56.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.494+0000 7f08f37fe700 1 -- 172.21.15.16:0/362769349 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v51) v1 ==== 120+0+0 (secure 0 0 0) 0x7f08e400cea0 con 0x7f08fc15bb50 2023-10-16T12:30:56.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.497+0000 7f0901047700 1 -- 172.21.15.16:0/362769349 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f08dc06b020 msgr2=0x7f08dc06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:56.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.497+0000 7f0901047700 1 --2- 172.21.15.16:0/362769349 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f08dc06b020 0x7f08dc06d4e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f08ec006fd0 tx=0x7f08ec008070 comp rx=0 tx=0).stop 2023-10-16T12:30:56.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.497+0000 7f0901047700 1 -- 172.21.15.16:0/362769349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08fc15bb50 msgr2=0x7f08fc153a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:56.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.497+0000 7f0901047700 1 --2- 172.21.15.16:0/362769349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08fc15bb50 0x7f08fc153a50 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7f08e400ebe0 tx=0x7f08e400eef0 comp rx=0 tx=0).stop 2023-10-16T12:30:56.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.497+0000 7f0901047700 1 -- 172.21.15.16:0/362769349 shutdown_connections 2023-10-16T12:30:56.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.497+0000 7f0901047700 1 --2- 172.21.15.16:0/362769349 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f08dc06b020 0x7f08dc06d4e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:56.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.497+0000 7f0901047700 1 --2- 172.21.15.16:0/362769349 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f08fc15cd10 0x7f08fc0ada90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:56.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.497+0000 7f0901047700 1 --2- 172.21.15.16:0/362769349 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f08fc0ca9b0 0x7f08fc0cf9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:56.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.497+0000 7f0901047700 1 --2- 172.21.15.16:0/362769349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08fc15bb50 0x7f08fc153a50 unknown :-1 s=CLOSED pgs=1693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:56.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.497+0000 7f0901047700 1 -- 172.21.15.16:0/362769349 >> 172.21.15.16:0/362769349 conn(0x7f08fc0b1ed0 msgr2=0x7f08fc0b16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:56.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.497+0000 7f0901047700 1 -- 172.21.15.16:0/362769349 shutdown_connections 2023-10-16T12:30:56.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.497+0000 7f0901047700 1 -- 172.21.15.16:0/362769349 wait complete. 2023-10-16T12:30:56.512 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods' 2023-10-16T12:30:56.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.636+0000 7f87269d0700 1 Processor -- start 2023-10-16T12:30:56.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.636+0000 7f87269d0700 1 -- start start 2023-10-16T12:30:56.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.636+0000 7f87269d0700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f87200cc8a0 0x7f872015e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:56.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.636+0000 7f87269d0700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8720161c30 con 0x7f872015b280 2023-10-16T12:30:56.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.636+0000 7f87269d0700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8720161f10 con 0x7f872006a7e0 2023-10-16T12:30:56.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.636+0000 7f87269d0700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87201621f0 con 0x7f87200cc8a0 2023-10-16T12:30:56.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.636+0000 7f871ffff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f872015b280 0x7f87201613e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35870/0 (socket says 172.21.15.16:35870) 2023-10-16T12:30:56.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.637+0000 7f871ffff700 1 -- 172.21.15.16:0/3728094915 learned_addr learned my addr 172.21.15.16:0/3728094915 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:30:56.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.637+0000 7f871effd700 1 -- 172.21.15.16:0/3728094915 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3653634631 0 0) 0x7f8720161c30 con 0x7f872015b280 2023-10-16T12:30:56.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.637+0000 7f871effd700 1 -- 172.21.15.16:0/3728094915 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87080035b0 con 0x7f872015b280 2023-10-16T12:30:56.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.637+0000 7f871f7fe700 1 --2- 172.21.15.16:0/3728094915 >> v2:172.21.15.49:3300/0 conn(0x7f87200cc8a0 0x7f872015e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:56.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.637+0000 7f871effd700 1 -- 172.21.15.16:0/3728094915 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 373427842 0 0) 0x7f87080035b0 con 0x7f872015b280 2023-10-16T12:30:56.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.637+0000 7f871effd700 1 -- 172.21.15.16:0/3728094915 >> v2:172.21.15.49:3300/0 conn(0x7f87200cc8a0 msgr2=0x7f872015e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:56.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.637+0000 7f871effd700 1 --2- 172.21.15.16:0/3728094915 >> v2:172.21.15.49:3300/0 conn(0x7f87200cc8a0 0x7f872015e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:56.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.637+0000 7f871effd700 1 -- 172.21.15.16:0/3728094915 >> v1:172.21.15.49:6789/0 conn(0x7f872006a7e0 legacy=0x7f87200d02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:56.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.637+0000 7f871effd700 1 -- 172.21.15.16:0/3728094915 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f872015df00 con 0x7f872015b280 2023-10-16T12:30:56.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.637+0000 7f871effd700 1 -- 172.21.15.16:0/3728094915 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f8710001670 con 0x7f872015b280 2023-10-16T12:30:56.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.637+0000 7f871effd700 1 -- 172.21.15.16:0/3728094915 >> v1:172.21.15.16:6789/0 conn(0x7f872015b280 legacy=0x7f87201613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:56.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.637+0000 7f871effd700 1 --2- 172.21.15.16:0/3728094915 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8708003a00 0x7f8708003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:56.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.637+0000 7f871effd700 1 --2- 172.21.15.16:0/3728094915 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8708004850 0x7f8708004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:56.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.637+0000 7f871effd700 1 --2- 172.21.15.16:0/3728094915 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8708005210 0x7f870800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:56.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.637+0000 7f871effd700 1 -- 172.21.15.16:0/3728094915 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87201621f0 con 0x7f8708003a00 2023-10-16T12:30:56.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.637+0000 7f871effd700 1 -- 172.21.15.16:0/3728094915 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f870800c160 con 0x7f8708004850 2023-10-16T12:30:56.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.637+0000 7f871effd700 1 -- 172.21.15.16:0/3728094915 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f870800c320 con 0x7f8708005210 2023-10-16T12:30:56.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.637+0000 7f871ffff700 1 --2- 172.21.15.16:0/3728094915 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8708003a00 0x7f8708003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:56.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.637+0000 7f871f7fe700 1 --2- 172.21.15.16:0/3728094915 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8708004850 0x7f8708004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:56.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.637+0000 7f8724f6d700 1 --2- 172.21.15.16:0/3728094915 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8708005210 0x7f870800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:56.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.637+0000 7f871ffff700 1 -- 172.21.15.16:0/3728094915 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8708005210 msgr2=0x7f870800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:56.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.637+0000 7f871ffff700 1 --2- 172.21.15.16:0/3728094915 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8708005210 0x7f870800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:56.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.637+0000 7f871ffff700 1 -- 172.21.15.16:0/3728094915 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8708004850 msgr2=0x7f8708004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:56.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.637+0000 7f871ffff700 1 --2- 172.21.15.16:0/3728094915 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8708004850 0x7f8708004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:56.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.638+0000 7f871ffff700 1 -- 172.21.15.16:0/3728094915 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87080035b0 con 0x7f8708003a00 2023-10-16T12:30:56.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.638+0000 7f8724f6d700 1 -- 172.21.15.16:0/3728094915 reap_dead start 2023-10-16T12:30:56.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.638+0000 7f871ffff700 1 --2- 172.21.15.16:0/3728094915 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8708003a00 0x7f8708003e20 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f871000c240 tx=0x7f8708005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=170de3adc5863399 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:56.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.638+0000 7f871effd700 1 -- 172.21.15.16:0/3728094915 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8710007c10 con 0x7f8708003a00 2023-10-16T12:30:56.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.638+0000 7f871effd700 1 -- 172.21.15.16:0/3728094915 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f871000cde0 con 0x7f8708003a00 2023-10-16T12:30:56.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.638+0000 7f871effd700 1 -- 172.21.15.16:0/3728094915 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8710005ba0 con 0x7f8708003a00 2023-10-16T12:30:56.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.638+0000 7f87269d0700 1 -- 172.21.15.16:0/3728094915 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8708003a00 msgr2=0x7f8708003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:56.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.638+0000 7f87269d0700 1 --2- 172.21.15.16:0/3728094915 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8708003a00 0x7f8708003e20 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f871000c240 tx=0x7f8708005880 comp rx=0 tx=0).stop 2023-10-16T12:30:56.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.638+0000 7f87269d0700 1 -- 172.21.15.16:0/3728094915 shutdown_connections 2023-10-16T12:30:56.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.638+0000 7f87269d0700 1 --2- 172.21.15.16:0/3728094915 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8708003a00 0x7f8708003e20 unknown :-1 s=CLOSED pgs=1695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:56.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.638+0000 7f87269d0700 1 -- 172.21.15.16:0/3728094915 >> 172.21.15.16:0/3728094915 conn(0x7f87200b1fd0 msgr2=0x7f87200b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:56.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.638+0000 7f87269d0700 1 -- 172.21.15.16:0/3728094915 shutdown_connections 2023-10-16T12:30:56.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.638+0000 7f87269d0700 1 -- 172.21.15.16:0/3728094915 wait complete. 2023-10-16T12:30:56.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.640+0000 7f87269d0700 1 Processor -- start 2023-10-16T12:30:56.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.640+0000 7f87269d0700 1 -- start start 2023-10-16T12:30:56.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.641+0000 7f87269d0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8708003a00 0x7f8720158a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:56.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.641+0000 7f87269d0700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f87200c6a30 0x7f87201594d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:56.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.641+0000 7f87269d0700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f87200cb910 0x7f8720159a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:56.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.641+0000 7f87269d0700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87200ad630 con 0x7f8708003a00 2023-10-16T12:30:56.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.641+0000 7f87269d0700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8720158f80 con 0x7f87200cb910 2023-10-16T12:30:56.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.641+0000 7f87269d0700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f872015f6d0 con 0x7f87200c6a30 2023-10-16T12:30:56.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.641+0000 7f871ffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8708003a00 0x7f8720158a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:56.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.641+0000 7f871ffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8708003a00 0x7f8720158a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46608/0 (socket says 172.21.15.16:46608) 2023-10-16T12:30:56.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.641+0000 7f871ffff700 1 -- 172.21.15.16:0/2384955545 learned_addr learned my addr 172.21.15.16:0/2384955545 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:56.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.641+0000 7f871f7fe700 1 --2- 172.21.15.16:0/2384955545 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f87200c6a30 0x7f87201594d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:56.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.641+0000 7f8724f6d700 1 --2- 172.21.15.16:0/2384955545 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f87200cb910 0x7f8720159a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:56.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.641+0000 7f871ffff700 1 -- 172.21.15.16:0/2384955545 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f87200c6a30 msgr2=0x7f87201594d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:56.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.641+0000 7f871ffff700 1 --2- 172.21.15.16:0/2384955545 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f87200c6a30 0x7f87201594d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:56.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.642+0000 7f871ffff700 1 -- 172.21.15.16:0/2384955545 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f87200cb910 msgr2=0x7f8720159a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:56.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.642+0000 7f871ffff700 1 --2- 172.21.15.16:0/2384955545 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f87200cb910 0x7f8720159a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:56.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.642+0000 7f871ffff700 1 -- 172.21.15.16:0/2384955545 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87200c3840 con 0x7f8708003a00 2023-10-16T12:30:56.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.642+0000 7f871ffff700 1 --2- 172.21.15.16:0/2384955545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8708003a00 0x7f8720158a40 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f871000c830 tx=0x7f8710001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:56.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.642+0000 7f871cff9700 1 -- 172.21.15.16:0/2384955545 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8710001640 con 0x7f8708003a00 2023-10-16T12:30:56.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.642+0000 7f871cff9700 1 -- 172.21.15.16:0/2384955545 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87100070d0 con 0x7f8708003a00 2023-10-16T12:30:56.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.642+0000 7f87269d0700 1 -- 172.21.15.16:0/2384955545 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f872015f920 con 0x7f8708003a00 2023-10-16T12:30:56.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.642+0000 7f871cff9700 1 -- 172.21.15.16:0/2384955545 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87100074a0 con 0x7f8708003a00 2023-10-16T12:30:56.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.642+0000 7f87269d0700 1 -- 172.21.15.16:0/2384955545 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f872015fee0 con 0x7f8708003a00 2023-10-16T12:30:56.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.643+0000 7f87269d0700 1 -- 172.21.15.16:0/2384955545 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87200adb50 con 0x7f8708003a00 2023-10-16T12:30:56.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.643+0000 7f871cff9700 1 -- 172.21.15.16:0/2384955545 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 85) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8710007600 con 0x7f8708003a00 2023-10-16T12:30:56.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.644+0000 7f871cff9700 1 --2- 172.21.15.16:0/2384955545 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f870c06b020 0x7f870c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:56.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.644+0000 7f871cff9700 1 -- 172.21.15.16:0/2384955545 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6050+0+0 (secure 0 0 0) 0x7f871008abc0 con 0x7f8708003a00 2023-10-16T12:30:56.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.644+0000 7f871f7fe700 1 --2- 172.21.15.16:0/2384955545 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f870c06b020 0x7f870c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:56.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.644+0000 7f871f7fe700 1 --2- 172.21.15.16:0/2384955545 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f870c06b020 0x7f870c06d4e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f87200ca9f0 tx=0x7f8714008040 comp rx=0 tx=0).ready entity=mgr.8973 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:56.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.649+0000 7f871cff9700 1 -- 172.21.15.16:0/2384955545 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f87200adb50 con 0x7f8708003a00 2023-10-16T12:30:56.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:56.937+0000 7f87269d0700 1 -- 172.21.15.16:0/2384955545 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods"]} v 0) v1 -- 0x7f87200adb50 con 0x7f8708003a00 2023-10-16T12:30:57.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.381+0000 7f871cff9700 1 -- 172.21.15.16:0/2384955545 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods"]}]=0 v784) v1 ==== 160+0+0 (secure 0 0 0) 0x7f87200adb50 con 0x7f8708003a00 2023-10-16T12:30:57.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.384+0000 7f87269d0700 1 -- 172.21.15.16:0/2384955545 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f870c06b020 msgr2=0x7f870c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:57.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.384+0000 7f87269d0700 1 --2- 172.21.15.16:0/2384955545 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f870c06b020 0x7f870c06d4e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f87200ca9f0 tx=0x7f8714008040 comp rx=0 tx=0).stop 2023-10-16T12:30:57.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.384+0000 7f87269d0700 1 -- 172.21.15.16:0/2384955545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8708003a00 msgr2=0x7f8720158a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:57.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.384+0000 7f87269d0700 1 --2- 172.21.15.16:0/2384955545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8708003a00 0x7f8720158a40 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f871000c830 tx=0x7f8710001a00 comp rx=0 tx=0).stop 2023-10-16T12:30:57.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.384+0000 7f87269d0700 1 -- 172.21.15.16:0/2384955545 shutdown_connections 2023-10-16T12:30:57.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.384+0000 7f87269d0700 1 --2- 172.21.15.16:0/2384955545 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f870c06b020 0x7f870c06d4e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:57.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.384+0000 7f87269d0700 1 --2- 172.21.15.16:0/2384955545 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f87200cb910 0x7f8720159a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:57.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.384+0000 7f87269d0700 1 --2- 172.21.15.16:0/2384955545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8708003a00 0x7f8720158a40 unknown :-1 s=CLOSED pgs=1696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:57.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.384+0000 7f87269d0700 1 --2- 172.21.15.16:0/2384955545 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f87200c6a30 0x7f87201594d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:57.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.384+0000 7f87269d0700 1 -- 172.21.15.16:0/2384955545 >> 172.21.15.16:0/2384955545 conn(0x7f87200b1fd0 msgr2=0x7f872006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:57.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.384+0000 7f87269d0700 1 -- 172.21.15.16:0/2384955545 shutdown_connections 2023-10-16T12:30:57.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.384+0000 7f87269d0700 1 -- 172.21.15.16:0/2384955545 wait complete. 2023-10-16T12:30:57.401 INFO:tasks.cephfs_test_runner:test_snap_schedule_all_periods (tasks.cephfs.test_snap_schedules.TestSnapSchedules) ... ok 2023-10-16T12:30:57.402 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_for_number_of_snaps_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test that number of snaps retained are as per user spec. 2023-10-16T12:30:57.402 INFO:tasks.cephfs_test_runner:test_snap_schedule_for_number_of_snaps_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-16T12:30:57.403 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:30:57.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.533+0000 7f1b9dbf2700 1 Processor -- start 2023-10-16T12:30:57.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.533+0000 7f1b9dbf2700 1 -- start start 2023-10-16T12:30:57.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.534+0000 7f1b9dbf2700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1b9815b860 0x7f1b9806a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:57.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.534+0000 7f1b9dbf2700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1b980b4490 0x7f1b9806ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:57.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.534+0000 7f1b9dbf2700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b980d11c0 con 0x7f1b980b2be0 2023-10-16T12:30:57.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.534+0000 7f1b9dbf2700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b980d03d0 con 0x7f1b9815b860 2023-10-16T12:30:57.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.534+0000 7f1b9dbf2700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b980cd360 con 0x7f1b980b4490 2023-10-16T12:30:57.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.534+0000 7f1b977fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1b9815b860 0x7f1b9806a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:57.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.534+0000 7f1b977fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1b9815b860 0x7f1b9806a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46616/0 (socket says 172.21.15.16:46616) 2023-10-16T12:30:57.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.534+0000 7f1b977fe700 1 -- 172.21.15.16:0/3511111083 learned_addr learned my addr 172.21.15.16:0/3511111083 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:57.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.534+0000 7f1b97fff700 1 --2- 172.21.15.16:0/3511111083 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1b980b4490 0x7f1b9806ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:57.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.534+0000 7f1b977fe700 1 -- 172.21.15.16:0/3511111083 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1b980b4490 msgr2=0x7f1b9806ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:57.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.534+0000 7f1b977fe700 1 --2- 172.21.15.16:0/3511111083 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1b980b4490 0x7f1b9806ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:57.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.534+0000 7f1b977fe700 1 -- 172.21.15.16:0/3511111083 >> v1:172.21.15.49:6789/0 conn(0x7f1b980b2be0 legacy=0x7f1b980719e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:57.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.534+0000 7f1b977fe700 1 -- 172.21.15.16:0/3511111083 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b980d05b0 con 0x7f1b9815b860 2023-10-16T12:30:57.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.535+0000 7f1b977fe700 1 --2- 172.21.15.16:0/3511111083 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b9815b860 0x7f1b9806a7e0 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7f1b800098f0 tx=0x7f1b80009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=732157a5577ec4ae server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:57.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.535+0000 7f1b967fc700 1 -- 172.21.15.16:0/3511111083 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b8000b8a0 con 0x7f1b9815b860 2023-10-16T12:30:57.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.535+0000 7f1b967fc700 1 -- 172.21.15.16:0/3511111083 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b8000ba00 con 0x7f1b9815b860 2023-10-16T12:30:57.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.535+0000 7f1b967fc700 1 -- 172.21.15.16:0/3511111083 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b8000bd90 con 0x7f1b9815b860 2023-10-16T12:30:57.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.535+0000 7f1b9dbf2700 1 -- 172.21.15.16:0/3511111083 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b9815b860 msgr2=0x7f1b9806a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:57.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.535+0000 7f1b9dbf2700 1 --2- 172.21.15.16:0/3511111083 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b9815b860 0x7f1b9806a7e0 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7f1b800098f0 tx=0x7f1b80009c00 comp rx=0 tx=0).stop 2023-10-16T12:30:57.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.535+0000 7f1b9dbf2700 1 -- 172.21.15.16:0/3511111083 shutdown_connections 2023-10-16T12:30:57.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.535+0000 7f1b9dbf2700 1 --2- 172.21.15.16:0/3511111083 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1b980b4490 0x7f1b9806ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:57.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.535+0000 7f1b9dbf2700 1 --2- 172.21.15.16:0/3511111083 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b9815b860 0x7f1b9806a7e0 unknown :-1 s=CLOSED pgs=1697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:57.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.535+0000 7f1b9dbf2700 1 -- 172.21.15.16:0/3511111083 >> 172.21.15.16:0/3511111083 conn(0x7f1b980b1fd0 msgr2=0x7f1b980b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:57.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.535+0000 7f1b9dbf2700 1 -- 172.21.15.16:0/3511111083 shutdown_connections 2023-10-16T12:30:57.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.535+0000 7f1b9dbf2700 1 -- 172.21.15.16:0/3511111083 wait complete. 2023-10-16T12:30:57.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.537+0000 7f1b9dbf2700 1 Processor -- start 2023-10-16T12:30:57.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.537+0000 7f1b9dbf2700 1 -- start start 2023-10-16T12:30:57.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.537+0000 7f1b9dbf2700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1b980b2be0 0x7f1b98155d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:57.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.537+0000 7f1b9dbf2700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1b980b4490 0x7f1b980c9c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:57.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.537+0000 7f1b9dbf2700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b9815b860 0x7f1b9806d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:57.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.537+0000 7f1b9dbf2700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b980af4d0 con 0x7f1b9815b860 2023-10-16T12:30:57.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.537+0000 7f1b9dbf2700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b98156290 con 0x7f1b980b4490 2023-10-16T12:30:57.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.537+0000 7f1b9dbf2700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b980ae8c0 con 0x7f1b980b2be0 2023-10-16T12:30:57.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.538+0000 7f1b97fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b9815b860 0x7f1b9806d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:57.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.538+0000 7f1b97fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b9815b860 0x7f1b9806d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46626/0 (socket says 172.21.15.16:46626) 2023-10-16T12:30:57.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.538+0000 7f1b97fff700 1 -- 172.21.15.16:0/2336300020 learned_addr learned my addr 172.21.15.16:0/2336300020 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:57.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.538+0000 7f1b977fe700 1 --2- 172.21.15.16:0/2336300020 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1b980b2be0 0x7f1b98155d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:57.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.538+0000 7f1b96ffd700 1 --2- 172.21.15.16:0/2336300020 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1b980b4490 0x7f1b980c9c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:57.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.538+0000 7f1b97fff700 1 -- 172.21.15.16:0/2336300020 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1b980b2be0 msgr2=0x7f1b98155d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:57.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.538+0000 7f1b97fff700 1 --2- 172.21.15.16:0/2336300020 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1b980b2be0 0x7f1b98155d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:57.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.538+0000 7f1b97fff700 1 -- 172.21.15.16:0/2336300020 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1b980b4490 msgr2=0x7f1b980c9c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:57.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.538+0000 7f1b97fff700 1 --2- 172.21.15.16:0/2336300020 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1b980b4490 0x7f1b980c9c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:57.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.538+0000 7f1b97fff700 1 -- 172.21.15.16:0/2336300020 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b80009550 con 0x7f1b9815b860 2023-10-16T12:30:57.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.538+0000 7f1b96ffd700 1 --2- 172.21.15.16:0/2336300020 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1b980b4490 0x7f1b980c9c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:30:57.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.538+0000 7f1b97fff700 1 --2- 172.21.15.16:0/2336300020 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b9815b860 0x7f1b9806d9f0 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7f1b90007e30 tx=0x7f1b9000e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:57.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.538+0000 7f1b8ffff700 1 -- 172.21.15.16:0/2336300020 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b9000eea0 con 0x7f1b9815b860 2023-10-16T12:30:57.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.538+0000 7f1b8ffff700 1 -- 172.21.15.16:0/2336300020 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b90014420 con 0x7f1b9815b860 2023-10-16T12:30:57.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.538+0000 7f1b9dbf2700 1 -- 172.21.15.16:0/2336300020 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b980aeba0 con 0x7f1b9815b860 2023-10-16T12:30:57.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.538+0000 7f1b8ffff700 1 -- 172.21.15.16:0/2336300020 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b900147d0 con 0x7f1b9815b860 2023-10-16T12:30:57.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.538+0000 7f1b9dbf2700 1 -- 172.21.15.16:0/2336300020 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b980af160 con 0x7f1b9815b860 2023-10-16T12:30:57.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.539+0000 7f1b9dbf2700 1 -- 172.21.15.16:0/2336300020 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b9804ebb0 con 0x7f1b9815b860 2023-10-16T12:30:57.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.540+0000 7f1b8ffff700 1 -- 172.21.15.16:0/2336300020 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 85) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1b90014930 con 0x7f1b9815b860 2023-10-16T12:30:57.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.540+0000 7f1b8ffff700 1 --2- 172.21.15.16:0/2336300020 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f1b7806b020 0x7f1b7806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:57.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.540+0000 7f1b8ffff700 1 -- 172.21.15.16:0/2336300020 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6050+0+0 (secure 0 0 0) 0x7f1b9008a060 con 0x7f1b9815b860 2023-10-16T12:30:57.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.540+0000 7f1b977fe700 1 --2- 172.21.15.16:0/2336300020 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f1b7806b020 0x7f1b7806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:57.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.541+0000 7f1b977fe700 1 --2- 172.21.15.16:0/2336300020 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f1b7806b020 0x7f1b7806d4e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f1b80009840 tx=0x7f1b80008030 comp rx=0 tx=0).ready entity=mgr.8973 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:57.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.545+0000 7f1b8ffff700 1 -- 172.21.15.16:0/2336300020 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1b90054ea0 con 0x7f1b9815b860 2023-10-16T12:30:57.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.746+0000 7f1b9dbf2700 1 -- 172.21.15.16:0/2336300020 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1b980b0bf0 con 0x7f1b9815b860 2023-10-16T12:30:57.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.747+0000 7f1b8ffff700 1 -- 172.21.15.16:0/2336300020 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v147) v1 ==== 74+0+14243 (secure 0 0 0) 0x7f1b900584c0 con 0x7f1b9815b860 2023-10-16T12:30:57.749 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:30:57.750 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":147,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:30:55.375002+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.49:6835/2008024326":"2023-10-17T12:30:52.001889+0000","172.21.15.16:0/3944675301":"2023-10-17T12:30:50.995881+0000","172.21.15.16:0/503385959":"2023-10-17T12:30:50.995881+0000","172.21.15.16:0/3093596020":"2023-10-17T12:30:50.995881+0000","172.21.15.16:0/1418518570":"2023-10-17T12:30:50.995881+0000","172.21.15.16:0/2205033750":"2023-10-17T12:30:50.995881+0000","172.21.15.16:0/647659834":"2023-10-17T12:30:50.995881+0000","172.21.15.16:6832/3783161600":"2023-10-17T12:30:50.995881+0000","172.21.15.16:0/2901217010":"2023-10-17T12:30:37.941028+0000","172.21.15.16:6833/3783161600":"2023-10-17T12:30:50.995881+0000","172.21.15.16:6833/258291157":"2023-10-17T12:30:37.941028+0000","172.21.15.49:6833/2008024326":"2023-10-17T12:30:52.001889+0000","172.21.15.16:6832/258291157":"2023-10-17T12:30:37.941028+0000","172.21.15.16:0/4123017371":"2023-10-17T12:30:37.941028+0000","172.21.15.16:0/2703050386":"2023-10-17T12:30:37.941028+0000","172.21.15.16:0/1995641428":"2023-10-17T12:30:50.995881+0000","172.21.15.16:0/3188488837":"2023-10-17T12:30:37.941028+0000","172.21.15.16:0/2621298501":"2023-10-17T12:30:37.941028+0000","172.21.15.16:0/1781882931":"2023-10-17T12:30:37.941028+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:30:57.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.749+0000 7f1b9dbf2700 1 -- 172.21.15.16:0/2336300020 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f1b7806b020 msgr2=0x7f1b7806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:57.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.749+0000 7f1b9dbf2700 1 --2- 172.21.15.16:0/2336300020 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f1b7806b020 0x7f1b7806d4e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f1b80009840 tx=0x7f1b80008030 comp rx=0 tx=0).stop 2023-10-16T12:30:57.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.749+0000 7f1b9dbf2700 1 -- 172.21.15.16:0/2336300020 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b9815b860 msgr2=0x7f1b9806d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:57.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.749+0000 7f1b9dbf2700 1 --2- 172.21.15.16:0/2336300020 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b9815b860 0x7f1b9806d9f0 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7f1b90007e30 tx=0x7f1b9000e3c0 comp rx=0 tx=0).stop 2023-10-16T12:30:57.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.749+0000 7f1b9dbf2700 1 -- 172.21.15.16:0/2336300020 shutdown_connections 2023-10-16T12:30:57.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.749+0000 7f1b9dbf2700 1 --2- 172.21.15.16:0/2336300020 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f1b7806b020 0x7f1b7806d4e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:57.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.749+0000 7f1b9dbf2700 1 --2- 172.21.15.16:0/2336300020 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b9815b860 0x7f1b9806d9f0 unknown :-1 s=CLOSED pgs=1698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:57.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.749+0000 7f1b9dbf2700 1 --2- 172.21.15.16:0/2336300020 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1b980b2be0 0x7f1b98155d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:57.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.749+0000 7f1b9dbf2700 1 --2- 172.21.15.16:0/2336300020 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1b980b4490 0x7f1b980c9c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:57.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.749+0000 7f1b9dbf2700 1 -- 172.21.15.16:0/2336300020 >> 172.21.15.16:0/2336300020 conn(0x7f1b980b1fd0 msgr2=0x7f1b9815dc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:57.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.750+0000 7f1b9dbf2700 1 -- 172.21.15.16:0/2336300020 shutdown_connections 2023-10-16T12:30:57.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.750+0000 7f1b9dbf2700 1 -- 172.21.15.16:0/2336300020 wait complete. 2023-10-16T12:30:57.764 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T12:30:57.764 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:30:57.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.890+0000 7fbc931fa700 1 Processor -- start 2023-10-16T12:30:57.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.890+0000 7fbc931fa700 1 -- start start 2023-10-16T12:30:57.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.891+0000 7fbc931fa700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbc8c0d4060 0x7fbc8c0d1520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:57.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.891+0000 7fbc931fa700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbc8c0d1d20 0x7fbc8c0d21b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:57.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.891+0000 7fbc931fa700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc8c0d4540 con 0x7fbc8c15da10 2023-10-16T12:30:57.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.891+0000 7fbc931fa700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc8c0cf9b0 con 0x7fbc8c0d4060 2023-10-16T12:30:57.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.891+0000 7fbc931fa700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc8c0d26f0 con 0x7fbc8c0d1d20 2023-10-16T12:30:57.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.891+0000 7fbc8bfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbc8c0d4060 0x7fbc8c0d1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:57.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.891+0000 7fbc8bfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbc8c0d4060 0x7fbc8c0d1520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46632/0 (socket says 172.21.15.16:46632) 2023-10-16T12:30:57.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.891+0000 7fbc8bfff700 1 -- 172.21.15.16:0/240078270 learned_addr learned my addr 172.21.15.16:0/240078270 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:57.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.891+0000 7fbc91797700 1 --2- 172.21.15.16:0/240078270 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbc8c0d1d20 0x7fbc8c0d21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:57.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.891+0000 7fbc8bfff700 1 -- 172.21.15.16:0/240078270 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbc8c0d1d20 msgr2=0x7fbc8c0d21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:57.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.891+0000 7fbc8bfff700 1 --2- 172.21.15.16:0/240078270 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbc8c0d1d20 0x7fbc8c0d21b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:57.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.891+0000 7fbc8bfff700 1 -- 172.21.15.16:0/240078270 >> v1:172.21.15.49:6789/0 conn(0x7fbc8c15da10 legacy=0x7fbc8c0cf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:57.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.891+0000 7fbc8bfff700 1 -- 172.21.15.16:0/240078270 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc8c0ce7e0 con 0x7fbc8c0d4060 2023-10-16T12:30:57.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.891+0000 7fbc8bfff700 1 --2- 172.21.15.16:0/240078270 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc8c0d4060 0x7fbc8c0d1520 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7fbc780098f0 tx=0x7fbc78009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a966860bee3d9e6c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:57.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.892+0000 7fbc8b7fe700 1 -- 172.21.15.16:0/240078270 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc78004230 con 0x7fbc8c0d4060 2023-10-16T12:30:57.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.892+0000 7fbc8b7fe700 1 -- 172.21.15.16:0/240078270 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc78004390 con 0x7fbc8c0d4060 2023-10-16T12:30:57.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.892+0000 7fbc8b7fe700 1 -- 172.21.15.16:0/240078270 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc780036a0 con 0x7fbc8c0d4060 2023-10-16T12:30:57.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.892+0000 7fbc931fa700 1 -- 172.21.15.16:0/240078270 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc8c0d4060 msgr2=0x7fbc8c0d1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:57.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.892+0000 7fbc931fa700 1 --2- 172.21.15.16:0/240078270 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc8c0d4060 0x7fbc8c0d1520 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7fbc780098f0 tx=0x7fbc78009c00 comp rx=0 tx=0).stop 2023-10-16T12:30:57.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.892+0000 7fbc931fa700 1 -- 172.21.15.16:0/240078270 shutdown_connections 2023-10-16T12:30:57.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.892+0000 7fbc931fa700 1 --2- 172.21.15.16:0/240078270 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbc8c0d1d20 0x7fbc8c0d21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:57.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.892+0000 7fbc931fa700 1 --2- 172.21.15.16:0/240078270 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc8c0d4060 0x7fbc8c0d1520 unknown :-1 s=CLOSED pgs=1699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:57.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.892+0000 7fbc931fa700 1 -- 172.21.15.16:0/240078270 >> 172.21.15.16:0/240078270 conn(0x7fbc8c09c9f0 msgr2=0x7fbc8c0d3740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:57.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.892+0000 7fbc931fa700 1 -- 172.21.15.16:0/240078270 shutdown_connections 2023-10-16T12:30:57.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.892+0000 7fbc931fa700 1 -- 172.21.15.16:0/240078270 wait complete. 2023-10-16T12:30:57.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.893+0000 7fbc931fa700 1 Processor -- start 2023-10-16T12:30:57.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.893+0000 7fbc931fa700 1 -- start start 2023-10-16T12:30:57.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.894+0000 7fbc931fa700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbc8c0d1d20 0x7fbc8c0c7d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:57.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.894+0000 7fbc931fa700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbc8c0d4060 0x7fbc8c0b85e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:57.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.894+0000 7fbc931fa700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc8c15da10 0x7fbc8c14e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:57.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.894+0000 7fbc931fa700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc8c0c8240 con 0x7fbc8c15da10 2023-10-16T12:30:57.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.894+0000 7fbc931fa700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc8c0b5010 con 0x7fbc8c0d4060 2023-10-16T12:30:57.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.894+0000 7fbc931fa700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc8c0b52f0 con 0x7fbc8c0d1d20 2023-10-16T12:30:57.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.894+0000 7fbc91797700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc8c15da10 0x7fbc8c14e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:57.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.894+0000 7fbc91797700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc8c15da10 0x7fbc8c14e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46638/0 (socket says 172.21.15.16:46638) 2023-10-16T12:30:57.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.894+0000 7fbc91797700 1 -- 172.21.15.16:0/2146029713 learned_addr learned my addr 172.21.15.16:0/2146029713 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:57.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.895+0000 7fbc90f96700 1 --2- 172.21.15.16:0/2146029713 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbc8c0d1d20 0x7fbc8c0c7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:57.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.895+0000 7fbc8bfff700 1 --2- 172.21.15.16:0/2146029713 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbc8c0d4060 0x7fbc8c0b85e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:57.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.895+0000 7fbc91797700 1 -- 172.21.15.16:0/2146029713 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbc8c0d1d20 msgr2=0x7fbc8c0c7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:57.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.895+0000 7fbc91797700 1 --2- 172.21.15.16:0/2146029713 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbc8c0d1d20 0x7fbc8c0c7d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:57.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.895+0000 7fbc91797700 1 -- 172.21.15.16:0/2146029713 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbc8c0d4060 msgr2=0x7fbc8c0b85e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:57.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.895+0000 7fbc91797700 1 --2- 172.21.15.16:0/2146029713 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbc8c0d4060 0x7fbc8c0b85e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:57.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.895+0000 7fbc91797700 1 -- 172.21.15.16:0/2146029713 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc78009550 con 0x7fbc8c15da10 2023-10-16T12:30:57.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.895+0000 7fbc91797700 1 --2- 172.21.15.16:0/2146029713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc8c15da10 0x7fbc8c14e660 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7fbc8400cd70 tx=0x7fbc8400a570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:57.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.895+0000 7fbc897fa700 1 -- 172.21.15.16:0/2146029713 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc84003a20 con 0x7fbc8c15da10 2023-10-16T12:30:57.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.895+0000 7fbc897fa700 1 -- 172.21.15.16:0/2146029713 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc84003b80 con 0x7fbc8c15da10 2023-10-16T12:30:57.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.895+0000 7fbc897fa700 1 -- 172.21.15.16:0/2146029713 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc84004020 con 0x7fbc8c15da10 2023-10-16T12:30:57.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.895+0000 7fbc931fa700 1 -- 172.21.15.16:0/2146029713 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc8c0ca810 con 0x7fbc8c15da10 2023-10-16T12:30:57.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.895+0000 7fbc931fa700 1 -- 172.21.15.16:0/2146029713 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc8c0cadd0 con 0x7fbc8c15da10 2023-10-16T12:30:57.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.896+0000 7fbc931fa700 1 -- 172.21.15.16:0/2146029713 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc8c04ebb0 con 0x7fbc8c15da10 2023-10-16T12:30:57.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.897+0000 7fbc897fa700 1 -- 172.21.15.16:0/2146029713 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 85) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fbc84004180 con 0x7fbc8c15da10 2023-10-16T12:30:57.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.897+0000 7fbc897fa700 1 --2- 172.21.15.16:0/2146029713 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7fbc7006b020 0x7fbc7006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:57.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.897+0000 7fbc897fa700 1 -- 172.21.15.16:0/2146029713 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6050+0+0 (secure 0 0 0) 0x7fbc84012070 con 0x7fbc8c15da10 2023-10-16T12:30:57.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.897+0000 7fbc90f96700 1 --2- 172.21.15.16:0/2146029713 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7fbc7006b020 0x7fbc7006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:57.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.897+0000 7fbc90f96700 1 --2- 172.21.15.16:0/2146029713 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7fbc7006b020 0x7fbc7006d4e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fbc80006fd0 tx=0x7fbc80008040 comp rx=0 tx=0).ready entity=mgr.8973 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:57.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:57.901+0000 7fbc897fa700 1 -- 172.21.15.16:0/2146029713 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbc84054f30 con 0x7fbc8c15da10 2023-10-16T12:30:58.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.103+0000 7fbc931fa700 1 -- 172.21.15.16:0/2146029713 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbc8c0d1b00 con 0x7fbc8c15da10 2023-10-16T12:30:58.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.104+0000 7fbc897fa700 1 -- 172.21.15.16:0/2146029713 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v147) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbc84058550 con 0x7fbc8c15da10 2023-10-16T12:30:58.106 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T12:30:58.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.106+0000 7fbc931fa700 1 -- 172.21.15.16:0/2146029713 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7fbc7006b020 msgr2=0x7fbc7006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:58.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.106+0000 7fbc931fa700 1 --2- 172.21.15.16:0/2146029713 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7fbc7006b020 0x7fbc7006d4e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fbc80006fd0 tx=0x7fbc80008040 comp rx=0 tx=0).stop 2023-10-16T12:30:58.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.106+0000 7fbc931fa700 1 -- 172.21.15.16:0/2146029713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc8c15da10 msgr2=0x7fbc8c14e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:58.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.106+0000 7fbc931fa700 1 --2- 172.21.15.16:0/2146029713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc8c15da10 0x7fbc8c14e660 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7fbc8400cd70 tx=0x7fbc8400a570 comp rx=0 tx=0).stop 2023-10-16T12:30:58.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.106+0000 7fbc931fa700 1 -- 172.21.15.16:0/2146029713 shutdown_connections 2023-10-16T12:30:58.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.106+0000 7fbc931fa700 1 --2- 172.21.15.16:0/2146029713 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7fbc7006b020 0x7fbc7006d4e0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:58.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.106+0000 7fbc931fa700 1 --2- 172.21.15.16:0/2146029713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc8c15da10 0x7fbc8c14e660 unknown :-1 s=CLOSED pgs=1700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:58.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.106+0000 7fbc931fa700 1 --2- 172.21.15.16:0/2146029713 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbc8c0d1d20 0x7fbc8c0c7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:58.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.106+0000 7fbc931fa700 1 --2- 172.21.15.16:0/2146029713 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbc8c0d4060 0x7fbc8c0b85e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:58.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.106+0000 7fbc931fa700 1 -- 172.21.15.16:0/2146029713 >> 172.21.15.16:0/2146029713 conn(0x7fbc8c09c9f0 msgr2=0x7fbc8c0cf1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:58.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.106+0000 7fbc931fa700 1 -- 172.21.15.16:0/2146029713 shutdown_connections 2023-10-16T12:30:58.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.106+0000 7fbc931fa700 1 -- 172.21.15.16:0/2146029713 wait complete. 2023-10-16T12:30:58.120 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention' 2023-10-16T12:30:58.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.244+0000 7fa651bae700 1 Processor -- start 2023-10-16T12:30:58.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.244+0000 7fa651bae700 1 -- start start 2023-10-16T12:30:58.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.245+0000 7fa651bae700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa64c0cf880 0x7fa64c0b52f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:58.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.245+0000 7fa651bae700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa64c15bc80 con 0x7fa64c0b5830 2023-10-16T12:30:58.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.245+0000 7fa651bae700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa64c15b070 con 0x7fa64c0cf4a0 2023-10-16T12:30:58.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.245+0000 7fa651bae700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa64c15b350 con 0x7fa64c0cf880 2023-10-16T12:30:58.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.245+0000 7fa64bfff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa64c0b5830 0x7fa64c15a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35922/0 (socket says 172.21.15.16:35922) 2023-10-16T12:30:58.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.245+0000 7fa64bfff700 1 -- 172.21.15.16:0/2339101177 learned_addr learned my addr 172.21.15.16:0/2339101177 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:30:58.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.245+0000 7fa64affd700 1 --2- 172.21.15.16:0/2339101177 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa64c0cf880 0x7fa64c0b52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:58.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.245+0000 7fa64a7fc700 1 -- 172.21.15.16:0/2339101177 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3531523278 0 0) 0x7fa64c15bc80 con 0x7fa64c0b5830 2023-10-16T12:30:58.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.245+0000 7fa64a7fc700 1 -- 172.21.15.16:0/2339101177 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6340035b0 con 0x7fa64c0b5830 2023-10-16T12:30:58.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.245+0000 7fa64a7fc700 1 -- 172.21.15.16:0/2339101177 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2502956222 0 0) 0x7fa6340035b0 con 0x7fa64c0b5830 2023-10-16T12:30:58.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.245+0000 7fa64a7fc700 1 -- 172.21.15.16:0/2339101177 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa64c0cf880 msgr2=0x7fa64c0b52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:58.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.245+0000 7fa64a7fc700 1 --2- 172.21.15.16:0/2339101177 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa64c0cf880 0x7fa64c0b52f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:58.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.245+0000 7fa64a7fc700 1 -- 172.21.15.16:0/2339101177 >> v1:172.21.15.49:6789/0 conn(0x7fa64c0cf4a0 legacy=0x7fa64c0726e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:58.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.245+0000 7fa64a7fc700 1 -- 172.21.15.16:0/2339101177 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa64c15b630 con 0x7fa64c0b5830 2023-10-16T12:30:58.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.245+0000 7fa64a7fc700 1 -- 172.21.15.16:0/2339101177 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7fa6440019f0 con 0x7fa64c0b5830 2023-10-16T12:30:58.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.245+0000 7fa64a7fc700 1 -- 172.21.15.16:0/2339101177 >> v1:172.21.15.16:6789/0 conn(0x7fa64c0b5830 legacy=0x7fa64c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:58.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.245+0000 7fa64a7fc700 1 --2- 172.21.15.16:0/2339101177 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa634003a30 0x7fa634003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:58.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.245+0000 7fa64a7fc700 1 --2- 172.21.15.16:0/2339101177 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa6340043e0 0x7fa6340048b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:58.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.245+0000 7fa64a7fc700 1 --2- 172.21.15.16:0/2339101177 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa634005060 0x7fa63400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:58.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.245+0000 7fa64a7fc700 1 -- 172.21.15.16:0/2339101177 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa64c15b350 con 0x7fa634005060 2023-10-16T12:30:58.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.245+0000 7fa64a7fc700 1 -- 172.21.15.16:0/2339101177 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa63400bf80 con 0x7fa634003a30 2023-10-16T12:30:58.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.245+0000 7fa64a7fc700 1 -- 172.21.15.16:0/2339101177 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa63400c140 con 0x7fa6340043e0 2023-10-16T12:30:58.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.246+0000 7fa64affd700 1 --2- 172.21.15.16:0/2339101177 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa64c0cf880 0x7fa64c0b52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:30:58.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.246+0000 7fa64bfff700 1 --2- 172.21.15.16:0/2339101177 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa634005060 0x7fa63400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:58.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.246+0000 7fa64b7fe700 1 --2- 172.21.15.16:0/2339101177 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa634003a30 0x7fa634003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:58.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.246+0000 7fa64affd700 1 --2- 172.21.15.16:0/2339101177 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa6340043e0 0x7fa6340048b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:58.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.246+0000 7fa64bfff700 1 -- 172.21.15.16:0/2339101177 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa6340043e0 msgr2=0x7fa6340048b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:58.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.246+0000 7fa64bfff700 1 --2- 172.21.15.16:0/2339101177 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa6340043e0 0x7fa6340048b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:58.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.246+0000 7fa64bfff700 1 -- 172.21.15.16:0/2339101177 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa634003a30 msgr2=0x7fa634003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:58.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.246+0000 7fa64bfff700 1 --2- 172.21.15.16:0/2339101177 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa634003a30 0x7fa634003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:58.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.246+0000 7fa64bfff700 1 -- 172.21.15.16:0/2339101177 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6340035b0 con 0x7fa634005060 2023-10-16T12:30:58.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.246+0000 7fa64bfff700 1 --2- 172.21.15.16:0/2339101177 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa634005060 0x7fa63400ba10 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7fa64400a7d0 tx=0x7fa634005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d8d2536e4d898ec server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:58.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.246+0000 7fa64a7fc700 1 -- 172.21.15.16:0/2339101177 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa644012600 con 0x7fa634005060 2023-10-16T12:30:58.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.246+0000 7fa64a7fc700 1 -- 172.21.15.16:0/2339101177 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa644012760 con 0x7fa634005060 2023-10-16T12:30:58.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.246+0000 7fa64a7fc700 1 -- 172.21.15.16:0/2339101177 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa644012af0 con 0x7fa634005060 2023-10-16T12:30:58.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.247+0000 7fa651bae700 1 -- 172.21.15.16:0/2339101177 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa634005060 msgr2=0x7fa63400ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:58.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.247+0000 7fa651bae700 1 --2- 172.21.15.16:0/2339101177 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa634005060 0x7fa63400ba10 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7fa64400a7d0 tx=0x7fa634005880 comp rx=0 tx=0).stop 2023-10-16T12:30:58.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.247+0000 7fa64bfff700 1 -- 172.21.15.16:0/2339101177 reap_dead start 2023-10-16T12:30:58.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.247+0000 7fa651bae700 1 -- 172.21.15.16:0/2339101177 shutdown_connections 2023-10-16T12:30:58.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.247+0000 7fa651bae700 1 -- 172.21.15.16:0/2339101177 >> 172.21.15.16:0/2339101177 conn(0x7fa64c0b1fd0 msgr2=0x7fa64c0b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:58.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.247+0000 7fa651bae700 1 -- 172.21.15.16:0/2339101177 shutdown_connections 2023-10-16T12:30:58.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.247+0000 7fa651bae700 1 -- 172.21.15.16:0/2339101177 wait complete. 2023-10-16T12:30:58.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.248+0000 7fa651bae700 1 Processor -- start 2023-10-16T12:30:58.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.248+0000 7fa651bae700 1 -- start start 2023-10-16T12:30:58.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.249+0000 7fa651bae700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa64c0c6a50 0x7fa64c0c8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:58.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.249+0000 7fa651bae700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa64c0c9150 0x7fa64c159790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:58.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.249+0000 7fa651bae700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa64c0cd7a0 0x7fa64c159cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:58.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.249+0000 7fa651bae700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa64c0ca220 con 0x7fa64c0cd7a0 2023-10-16T12:30:58.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.249+0000 7fa651bae700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa64c0c9610 con 0x7fa64c0c9150 2023-10-16T12:30:58.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.249+0000 7fa651bae700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa64c0c98f0 con 0x7fa64c0c6a50 2023-10-16T12:30:58.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.249+0000 7fa64bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa64c0cd7a0 0x7fa64c159cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:58.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.249+0000 7fa64bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa64c0cd7a0 0x7fa64c159cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46658/0 (socket says 172.21.15.16:46658) 2023-10-16T12:30:58.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.249+0000 7fa64bfff700 1 -- 172.21.15.16:0/439437852 learned_addr learned my addr 172.21.15.16:0/439437852 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:58.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.249+0000 7fa64b7fe700 1 --2- 172.21.15.16:0/439437852 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa64c0c6a50 0x7fa64c0c8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:58.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.249+0000 7fa64affd700 1 --2- 172.21.15.16:0/439437852 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa64c0c9150 0x7fa64c159790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:58.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.249+0000 7fa64bfff700 1 -- 172.21.15.16:0/439437852 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa64c0c6a50 msgr2=0x7fa64c0c8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:58.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.249+0000 7fa64bfff700 1 --2- 172.21.15.16:0/439437852 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa64c0c6a50 0x7fa64c0c8c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:58.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.249+0000 7fa64bfff700 1 -- 172.21.15.16:0/439437852 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa64c0c9150 msgr2=0x7fa64c159790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:58.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.249+0000 7fa64bfff700 1 --2- 172.21.15.16:0/439437852 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa64c0c9150 0x7fa64c159790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:58.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.249+0000 7fa64bfff700 1 -- 172.21.15.16:0/439437852 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa64c0c9bd0 con 0x7fa64c0cd7a0 2023-10-16T12:30:58.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.250+0000 7fa64bfff700 1 --2- 172.21.15.16:0/439437852 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa64c0cd7a0 0x7fa64c159cd0 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7fa63400bf50 tx=0x7fa644001d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:58.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.250+0000 7fa633fff700 1 -- 172.21.15.16:0/439437852 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa64401e630 con 0x7fa64c0cd7a0 2023-10-16T12:30:58.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.250+0000 7fa633fff700 1 -- 172.21.15.16:0/439437852 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa64401e790 con 0x7fa64c0cd7a0 2023-10-16T12:30:58.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.250+0000 7fa651bae700 1 -- 172.21.15.16:0/439437852 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa64c0c9eb0 con 0x7fa64c0cd7a0 2023-10-16T12:30:58.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.250+0000 7fa633fff700 1 -- 172.21.15.16:0/439437852 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa64401eb40 con 0x7fa64c0cd7a0 2023-10-16T12:30:58.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.250+0000 7fa651bae700 1 -- 172.21.15.16:0/439437852 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa64c0cbad0 con 0x7fa64c0cd7a0 2023-10-16T12:30:58.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.251+0000 7fa651bae700 1 -- 172.21.15.16:0/439437852 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa64c04ebb0 con 0x7fa64c0cd7a0 2023-10-16T12:30:58.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.251+0000 7fa633fff700 1 -- 172.21.15.16:0/439437852 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 85) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa64401eca0 con 0x7fa64c0cd7a0 2023-10-16T12:30:58.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.252+0000 7fa633fff700 1 --2- 172.21.15.16:0/439437852 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7fa63806b020 0x7fa63806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:58.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.252+0000 7fa633fff700 1 -- 172.21.15.16:0/439437852 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6050+0+0 (secure 0 0 0) 0x7fa64408b6e0 con 0x7fa64c0cd7a0 2023-10-16T12:30:58.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.252+0000 7fa64b7fe700 1 --2- 172.21.15.16:0/439437852 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7fa63806b020 0x7fa63806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:58.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.252+0000 7fa64b7fe700 1 --2- 172.21.15.16:0/439437852 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7fa63806b020 0x7fa63806d4e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa63c000c00 tx=0x7fa63c009040 comp rx=0 tx=0).ready entity=mgr.8973 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:58.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.256+0000 7fa633fff700 1 -- 172.21.15.16:0/439437852 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa644056790 con 0x7fa64c0cd7a0 2023-10-16T12:30:58.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:58.544+0000 7fa651bae700 1 -- 172.21.15.16:0/439437852 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]} v 0) v1 -- 0x7fa64c0ca5b0 con 0x7fa64c0cd7a0 2023-10-16T12:30:59.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.387+0000 7fa633fff700 1 -- 172.21.15.16:0/439437852 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]}]=0 v786) v1 ==== 181+0+0 (secure 0 0 0) 0x7fa644059db0 con 0x7fa64c0cd7a0 2023-10-16T12:30:59.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.390+0000 7fa651bae700 1 -- 172.21.15.16:0/439437852 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7fa63806b020 msgr2=0x7fa63806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:59.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.390+0000 7fa651bae700 1 --2- 172.21.15.16:0/439437852 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7fa63806b020 0x7fa63806d4e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa63c000c00 tx=0x7fa63c009040 comp rx=0 tx=0).stop 2023-10-16T12:30:59.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.390+0000 7fa651bae700 1 -- 172.21.15.16:0/439437852 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa64c0cd7a0 msgr2=0x7fa64c159cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:59.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.390+0000 7fa651bae700 1 --2- 172.21.15.16:0/439437852 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa64c0cd7a0 0x7fa64c159cd0 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7fa63400bf50 tx=0x7fa644001d00 comp rx=0 tx=0).stop 2023-10-16T12:30:59.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.390+0000 7fa651bae700 1 -- 172.21.15.16:0/439437852 shutdown_connections 2023-10-16T12:30:59.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.390+0000 7fa651bae700 1 --2- 172.21.15.16:0/439437852 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7fa63806b020 0x7fa63806d4e0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:59.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.390+0000 7fa651bae700 1 --2- 172.21.15.16:0/439437852 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa64c0cd7a0 0x7fa64c159cd0 unknown :-1 s=CLOSED pgs=1703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:59.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.390+0000 7fa651bae700 1 --2- 172.21.15.16:0/439437852 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa64c0c6a50 0x7fa64c0c8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:59.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.390+0000 7fa651bae700 1 --2- 172.21.15.16:0/439437852 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa64c0c9150 0x7fa64c159790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:59.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.390+0000 7fa651bae700 1 -- 172.21.15.16:0/439437852 >> 172.21.15.16:0/439437852 conn(0x7fa64c0b1fd0 msgr2=0x7fa64c0b0330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:59.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.390+0000 7fa651bae700 1 -- 172.21.15.16:0/439437852 shutdown_connections 2023-10-16T12:30:59.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.390+0000 7fa651bae700 1 -- 172.21.15.16:0/439437852 wait complete. 2023-10-16T12:30:59.407 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-16T12:30:59.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.542+0000 7fdd453a9700 1 Processor -- start 2023-10-16T12:30:59.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.542+0000 7fdd453a9700 1 -- start start 2023-10-16T12:30:59.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.543+0000 7fdd453a9700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fdd4015d8d0 0x7fdd400b2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:59.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.543+0000 7fdd453a9700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdd400b8fa0 0x7fdd400b6720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:59.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.543+0000 7fdd453a9700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd400b2f30 con 0x7fdd400b6c60 2023-10-16T12:30:59.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.543+0000 7fdd453a9700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd400d0bb0 con 0x7fdd4015d8d0 2023-10-16T12:30:59.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.543+0000 7fdd453a9700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd4015dcb0 con 0x7fdd400b8fa0 2023-10-16T12:30:59.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.543+0000 7fdd3e7fc700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdd400b8fa0 0x7fdd400b6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:59.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.543+0000 7fdd3effd700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fdd4015d8d0 0x7fdd400b2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:59.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.543+0000 7fdd3f7fe700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7fdd400b6c60 0x7fdd400ce350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:43124/0 (socket says 172.21.15.16:43124) 2023-10-16T12:30:59.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.543+0000 7fdd3f7fe700 1 -- 172.21.15.16:0/1691489364 learned_addr learned my addr 172.21.15.16:0/1691489364 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:30:59.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.544+0000 7fdd3e7fc700 1 -- 172.21.15.16:0/1691489364 >> v2:172.21.15.49:3300/0 conn(0x7fdd4015d8d0 msgr2=0x7fdd400b2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:30:59.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.544+0000 7fdd3e7fc700 1 --2- 172.21.15.16:0/1691489364 >> v2:172.21.15.49:3300/0 conn(0x7fdd4015d8d0 0x7fdd400b2830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:59.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.544+0000 7fdd3e7fc700 1 -- 172.21.15.16:0/1691489364 >> v1:172.21.15.49:6789/0 conn(0x7fdd400b6c60 legacy=0x7fdd400ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:59.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.544+0000 7fdd3e7fc700 1 -- 172.21.15.16:0/1691489364 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd400d1c40 con 0x7fdd400b8fa0 2023-10-16T12:30:59.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.544+0000 7fdd3effd700 1 --2- 172.21.15.16:0/1691489364 >> v2:172.21.15.49:3300/0 conn(0x7fdd4015d8d0 0x7fdd400b2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:30:59.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.544+0000 7fdd3e7fc700 1 --2- 172.21.15.16:0/1691489364 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdd400b8fa0 0x7fdd400b6720 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fdd28009840 tx=0x7fdd28009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=349112946e7215a5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:59.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.544+0000 7fdd3dffb700 1 -- 172.21.15.16:0/1691489364 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd28011830 con 0x7fdd400b8fa0 2023-10-16T12:30:59.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.544+0000 7fdd3dffb700 1 -- 172.21.15.16:0/1691489364 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd28011990 con 0x7fdd400b8fa0 2023-10-16T12:30:59.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.544+0000 7fdd3dffb700 1 -- 172.21.15.16:0/1691489364 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd28011d20 con 0x7fdd400b8fa0 2023-10-16T12:30:59.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.544+0000 7fdd453a9700 1 -- 172.21.15.16:0/1691489364 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdd400b8fa0 msgr2=0x7fdd400b6720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:59.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.544+0000 7fdd453a9700 1 --2- 172.21.15.16:0/1691489364 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdd400b8fa0 0x7fdd400b6720 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fdd28009840 tx=0x7fdd28009b50 comp rx=0 tx=0).stop 2023-10-16T12:30:59.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.544+0000 7fdd453a9700 1 -- 172.21.15.16:0/1691489364 shutdown_connections 2023-10-16T12:30:59.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.544+0000 7fdd453a9700 1 --2- 172.21.15.16:0/1691489364 >> v2:172.21.15.49:3300/0 conn(0x7fdd4015d8d0 0x7fdd400b2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:59.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.544+0000 7fdd453a9700 1 --2- 172.21.15.16:0/1691489364 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdd400b8fa0 0x7fdd400b6720 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:59.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.545+0000 7fdd453a9700 1 -- 172.21.15.16:0/1691489364 >> 172.21.15.16:0/1691489364 conn(0x7fdd40072330 msgr2=0x7fdd40072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:59.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.545+0000 7fdd453a9700 1 -- 172.21.15.16:0/1691489364 shutdown_connections 2023-10-16T12:30:59.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.545+0000 7fdd453a9700 1 -- 172.21.15.16:0/1691489364 wait complete. 2023-10-16T12:30:59.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.546+0000 7fdd453a9700 1 Processor -- start 2023-10-16T12:30:59.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.546+0000 7fdd453a9700 1 -- start start 2023-10-16T12:30:59.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.547+0000 7fdd453a9700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdd400b6c60 0x7fdd400ad3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:59.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.547+0000 7fdd453a9700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdd400b8fa0 0x7fdd400ab2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:59.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.547+0000 7fdd453a9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd4015d8d0 0x7fdd401519f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:59.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.547+0000 7fdd453a9700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd400cbf90 con 0x7fdd4015d8d0 2023-10-16T12:30:59.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.547+0000 7fdd453a9700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd400abeb0 con 0x7fdd400b8fa0 2023-10-16T12:30:59.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.547+0000 7fdd453a9700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd400ad900 con 0x7fdd400b6c60 2023-10-16T12:30:59.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.547+0000 7fdd3f7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd4015d8d0 0x7fdd401519f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:59.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.547+0000 7fdd3f7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd4015d8d0 0x7fdd401519f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46674/0 (socket says 172.21.15.16:46674) 2023-10-16T12:30:59.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.547+0000 7fdd3f7fe700 1 -- 172.21.15.16:0/1257254147 learned_addr learned my addr 172.21.15.16:0/1257254147 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:30:59.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.547+0000 7fdd3e7fc700 1 --2- 172.21.15.16:0/1257254147 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdd400b8fa0 0x7fdd400ab2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:59.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.547+0000 7fdd3effd700 1 --2- 172.21.15.16:0/1257254147 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdd400b6c60 0x7fdd400ad3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:59.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.547+0000 7fdd3f7fe700 1 -- 172.21.15.16:0/1257254147 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdd400b6c60 msgr2=0x7fdd400ad3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:59.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.547+0000 7fdd3f7fe700 1 --2- 172.21.15.16:0/1257254147 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdd400b6c60 0x7fdd400ad3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:59.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.547+0000 7fdd3f7fe700 1 -- 172.21.15.16:0/1257254147 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdd400b8fa0 msgr2=0x7fdd400ab2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:59.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.547+0000 7fdd3f7fe700 1 --2- 172.21.15.16:0/1257254147 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdd400b8fa0 0x7fdd400ab2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:59.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.547+0000 7fdd3f7fe700 1 -- 172.21.15.16:0/1257254147 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd300094a0 con 0x7fdd4015d8d0 2023-10-16T12:30:59.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.547+0000 7fdd3effd700 1 --2- 172.21.15.16:0/1257254147 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdd400b6c60 0x7fdd400ad3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:30:59.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.547+0000 7fdd3f7fe700 1 --2- 172.21.15.16:0/1257254147 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd4015d8d0 0x7fdd401519f0 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7fdd3800cad0 tx=0x7fdd3800ce90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:59.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.547+0000 7fdd377fe700 1 -- 172.21.15.16:0/1257254147 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd3800aa60 con 0x7fdd4015d8d0 2023-10-16T12:30:59.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.547+0000 7fdd377fe700 1 -- 172.21.15.16:0/1257254147 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd3800abc0 con 0x7fdd4015d8d0 2023-10-16T12:30:59.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.547+0000 7fdd377fe700 1 -- 172.21.15.16:0/1257254147 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd38012400 con 0x7fdd4015d8d0 2023-10-16T12:30:59.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.548+0000 7fdd453a9700 1 -- 172.21.15.16:0/1257254147 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd280094a0 con 0x7fdd4015d8d0 2023-10-16T12:30:59.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.548+0000 7fdd453a9700 1 -- 172.21.15.16:0/1257254147 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd400cb8d0 con 0x7fdd4015d8d0 2023-10-16T12:30:59.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.549+0000 7fdd453a9700 1 -- 172.21.15.16:0/1257254147 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd4004ebb0 con 0x7fdd4015d8d0 2023-10-16T12:30:59.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.549+0000 7fdd377fe700 1 -- 172.21.15.16:0/1257254147 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 85) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fdd3801b3b0 con 0x7fdd4015d8d0 2023-10-16T12:30:59.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.549+0000 7fdd377fe700 1 --2- 172.21.15.16:0/1257254147 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7fdd2006b020 0x7fdd2006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:30:59.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.550+0000 7fdd377fe700 1 -- 172.21.15.16:0/1257254147 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6050+0+0 (secure 0 0 0) 0x7fdd38089360 con 0x7fdd4015d8d0 2023-10-16T12:30:59.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.550+0000 7fdd3effd700 1 --2- 172.21.15.16:0/1257254147 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7fdd2006b020 0x7fdd2006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:30:59.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.550+0000 7fdd3effd700 1 --2- 172.21.15.16:0/1257254147 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7fdd2006b020 0x7fdd2006d4e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fdd4015bc50 tx=0x7fdd30008040 comp rx=0 tx=0).ready entity=mgr.8973 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:30:59.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.554+0000 7fdd377fe700 1 -- 172.21.15.16:0/1257254147 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdd38054240 con 0x7fdd4015d8d0 2023-10-16T12:30:59.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.760+0000 7fdd453a9700 1 -- 172.21.15.16:0/1257254147 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fdd400b7550 con 0x7fdd4015d8d0 2023-10-16T12:30:59.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.763+0000 7fdd377fe700 1 -- 172.21.15.16:0/1257254147 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v52)=0 v52) v1 ==== 122+0+0 (secure 0 0 0) 0x7fdd38057860 con 0x7fdd4015d8d0 2023-10-16T12:30:59.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.765+0000 7fdd453a9700 1 -- 172.21.15.16:0/1257254147 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7fdd2006b020 msgr2=0x7fdd2006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:59.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.765+0000 7fdd453a9700 1 --2- 172.21.15.16:0/1257254147 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7fdd2006b020 0x7fdd2006d4e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fdd4015bc50 tx=0x7fdd30008040 comp rx=0 tx=0).stop 2023-10-16T12:30:59.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.765+0000 7fdd453a9700 1 -- 172.21.15.16:0/1257254147 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd4015d8d0 msgr2=0x7fdd401519f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:30:59.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.765+0000 7fdd453a9700 1 --2- 172.21.15.16:0/1257254147 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd4015d8d0 0x7fdd401519f0 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7fdd3800cad0 tx=0x7fdd3800ce90 comp rx=0 tx=0).stop 2023-10-16T12:30:59.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.766+0000 7fdd453a9700 1 -- 172.21.15.16:0/1257254147 shutdown_connections 2023-10-16T12:30:59.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.766+0000 7fdd453a9700 1 --2- 172.21.15.16:0/1257254147 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7fdd2006b020 0x7fdd2006d4e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:59.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.766+0000 7fdd453a9700 1 --2- 172.21.15.16:0/1257254147 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd4015d8d0 0x7fdd401519f0 unknown :-1 s=CLOSED pgs=1704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:59.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.766+0000 7fdd453a9700 1 --2- 172.21.15.16:0/1257254147 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdd400b6c60 0x7fdd400ad3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:59.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.766+0000 7fdd453a9700 1 --2- 172.21.15.16:0/1257254147 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdd400b8fa0 0x7fdd400ab2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:30:59.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.766+0000 7fdd453a9700 1 -- 172.21.15.16:0/1257254147 >> 172.21.15.16:0/1257254147 conn(0x7fdd40072330 msgr2=0x7fdd400b7970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:30:59.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.766+0000 7fdd453a9700 1 -- 172.21.15.16:0/1257254147 shutdown_connections 2023-10-16T12:30:59.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:30:59.766+0000 7fdd453a9700 1 -- 172.21.15.16:0/1257254147 wait complete. 2023-10-16T12:30:59.780 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T12:30:59.780 DEBUG:teuthology.orchestra.run.smithi016:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T12:30:59.822 DEBUG:teuthology.orchestra.run.smithi049:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T12:30:59.869 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T12:30:59.869 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:30:59.870 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:30:59.887 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16287364k,nr_inodes=4071841,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20251 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3261172k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T12:30:59.888 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T12:30:59.888 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:30:59.908 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284204k,nr_inodes=4071051,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=753 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260540k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T12:30:59.909 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:30:59.909 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:30:59.945 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16287364k,nr_inodes=4071841,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20251 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3261172k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T12:30:59.945 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T12:30:59.946 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:30:59.968 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284204k,nr_inodes=4071051,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=753 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260540k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T12:30:59.968 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-16T12:30:59.969 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:31:00.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.107+0000 7f42335c5700 1 Processor -- start 2023-10-16T12:31:00.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.107+0000 7f42335c5700 1 -- start start 2023-10-16T12:31:00.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.108+0000 7f42335c5700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f422c15b630 0x7f422c0cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:00.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.108+0000 7f42335c5700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f422c15ba10 0x7f422c0cdde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:00.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.108+0000 7f42335c5700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f422c0ce320 0x7f422c0cfca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:00.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.108+0000 7f42335c5700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f422c0cdb20 con 0x7f422c15b630 2023-10-16T12:31:00.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.108+0000 7f42335c5700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f422c0ce770 con 0x7f422c0ce320 2023-10-16T12:31:00.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.108+0000 7f42335c5700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f422c0cc2a0 con 0x7f422c15ba10 2023-10-16T12:31:00.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.108+0000 7f4231361700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f422c15b630 0x7f422c0cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:00.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.108+0000 7f4231361700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f422c15b630 0x7f422c0cd5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46678/0 (socket says 172.21.15.16:46678) 2023-10-16T12:31:00.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.108+0000 7f4231361700 1 -- 172.21.15.16:0/882127647 learned_addr learned my addr 172.21.15.16:0/882127647 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:31:00.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.108+0000 7f4230b60700 1 --2- 172.21.15.16:0/882127647 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f422c15ba10 0x7f422c0cdde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:00.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.108+0000 7f4231b62700 1 --2- 172.21.15.16:0/882127647 >> v2:172.21.15.49:3300/0 conn(0x7f422c0ce320 0x7f422c0cfca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:00.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.109+0000 7f4231361700 1 -- 172.21.15.16:0/882127647 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f422c15ba10 msgr2=0x7f422c0cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:31:00.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.109+0000 7f4231361700 1 --2- 172.21.15.16:0/882127647 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f422c15ba10 0x7f422c0cdde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:00.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.109+0000 7f4231361700 1 -- 172.21.15.16:0/882127647 >> v2:172.21.15.49:3300/0 conn(0x7f422c0ce320 msgr2=0x7f422c0cfca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:31:00.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.109+0000 7f4231361700 1 --2- 172.21.15.16:0/882127647 >> v2:172.21.15.49:3300/0 conn(0x7f422c0ce320 0x7f422c0cfca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:00.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.109+0000 7f4231361700 1 -- 172.21.15.16:0/882127647 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f422c0cc540 con 0x7f422c15b630 2023-10-16T12:31:00.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.109+0000 7f4231361700 1 --2- 172.21.15.16:0/882127647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f422c15b630 0x7f422c0cd5e0 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f422c06e990 tx=0x7f421c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eaa365f7706ce2dc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:00.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.109+0000 7f4223fff700 1 -- 172.21.15.16:0/882127647 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f421c005e30 con 0x7f422c15b630 2023-10-16T12:31:00.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.109+0000 7f4223fff700 1 -- 172.21.15.16:0/882127647 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f421c004030 con 0x7f422c15b630 2023-10-16T12:31:00.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.109+0000 7f4223fff700 1 -- 172.21.15.16:0/882127647 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f421c0043e0 con 0x7f422c15b630 2023-10-16T12:31:00.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.109+0000 7f42335c5700 1 -- 172.21.15.16:0/882127647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f422c15b630 msgr2=0x7f422c0cd5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:00.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.109+0000 7f42335c5700 1 --2- 172.21.15.16:0/882127647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f422c15b630 0x7f422c0cd5e0 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f422c06e990 tx=0x7f421c009bd0 comp rx=0 tx=0).stop 2023-10-16T12:31:00.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.109+0000 7f42335c5700 1 -- 172.21.15.16:0/882127647 shutdown_connections 2023-10-16T12:31:00.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.109+0000 7f42335c5700 1 --2- 172.21.15.16:0/882127647 >> v2:172.21.15.49:3300/0 conn(0x7f422c0ce320 0x7f422c0cfca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:00.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.109+0000 7f42335c5700 1 --2- 172.21.15.16:0/882127647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f422c15b630 0x7f422c0cd5e0 unknown :-1 s=CLOSED pgs=1705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:00.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.109+0000 7f42335c5700 1 --2- 172.21.15.16:0/882127647 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f422c15ba10 0x7f422c0cdde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:00.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.109+0000 7f42335c5700 1 -- 172.21.15.16:0/882127647 >> 172.21.15.16:0/882127647 conn(0x7f422c0d14b0 msgr2=0x7f422c0d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:31:00.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.110+0000 7f42335c5700 1 -- 172.21.15.16:0/882127647 shutdown_connections 2023-10-16T12:31:00.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.110+0000 7f42335c5700 1 -- 172.21.15.16:0/882127647 wait complete. 2023-10-16T12:31:00.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.111+0000 7f42335c5700 1 Processor -- start 2023-10-16T12:31:00.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.111+0000 7f42335c5700 1 -- start start 2023-10-16T12:31:00.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.112+0000 7f42335c5700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f422c0ce320 0x7f422c1f4070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:00.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.112+0000 7f42335c5700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f422c15b630 0x7f422c154a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:00.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.112+0000 7f42335c5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f422c15ba10 0x7f422c0ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:00.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.112+0000 7f42335c5700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f422c1f3be0 con 0x7f422c15ba10 2023-10-16T12:31:00.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.112+0000 7f42335c5700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f422c154fc0 con 0x7f422c15b630 2023-10-16T12:31:00.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.112+0000 7f42335c5700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f422c0cad20 con 0x7f422c0ce320 2023-10-16T12:31:00.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.112+0000 7f4231b62700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f422c15ba10 0x7f422c0ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:00.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.112+0000 7f4231b62700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f422c15ba10 0x7f422c0ca3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46688/0 (socket says 172.21.15.16:46688) 2023-10-16T12:31:00.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.112+0000 7f4231b62700 1 -- 172.21.15.16:0/2899332769 learned_addr learned my addr 172.21.15.16:0/2899332769 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:31:00.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.112+0000 7f4231361700 1 --2- 172.21.15.16:0/2899332769 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f422c0ce320 0x7f422c1f4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:00.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.112+0000 7f4230b60700 1 --2- 172.21.15.16:0/2899332769 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f422c15b630 0x7f422c154a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:00.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.113+0000 7f4231b62700 1 -- 172.21.15.16:0/2899332769 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f422c0ce320 msgr2=0x7f422c1f4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:00.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.113+0000 7f4231b62700 1 --2- 172.21.15.16:0/2899332769 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f422c0ce320 0x7f422c1f4070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:00.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.113+0000 7f4231b62700 1 -- 172.21.15.16:0/2899332769 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f422c15b630 msgr2=0x7f422c154a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:00.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.113+0000 7f4231b62700 1 --2- 172.21.15.16:0/2899332769 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f422c15b630 0x7f422c154a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:00.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.113+0000 7f4231b62700 1 -- 172.21.15.16:0/2899332769 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f421c009500 con 0x7f422c15ba10 2023-10-16T12:31:00.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.113+0000 7f4231b62700 1 --2- 172.21.15.16:0/2899332769 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f422c15ba10 0x7f422c0ca3b0 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7f4224007d20 tx=0x7f422400e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:00.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.113+0000 7f4221ffb700 1 -- 172.21.15.16:0/2899332769 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f422400ed30 con 0x7f422c15ba10 2023-10-16T12:31:00.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.113+0000 7f4221ffb700 1 -- 172.21.15.16:0/2899332769 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4224007e70 con 0x7f422c15ba10 2023-10-16T12:31:00.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.113+0000 7f42335c5700 1 -- 172.21.15.16:0/2899332769 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f422c1f3590 con 0x7f422c15ba10 2023-10-16T12:31:00.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.113+0000 7f4221ffb700 1 -- 172.21.15.16:0/2899332769 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4224014640 con 0x7f422c15ba10 2023-10-16T12:31:00.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.113+0000 7f42335c5700 1 -- 172.21.15.16:0/2899332769 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f422c0cb000 con 0x7f422c15ba10 2023-10-16T12:31:00.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.114+0000 7f42335c5700 1 -- 172.21.15.16:0/2899332769 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f422c067870 con 0x7f422c15ba10 2023-10-16T12:31:00.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.115+0000 7f4221ffb700 1 -- 172.21.15.16:0/2899332769 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 85) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f42240147a0 con 0x7f422c15ba10 2023-10-16T12:31:00.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.115+0000 7f4221ffb700 1 --2- 172.21.15.16:0/2899332769 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f421006b020 0x7f421006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:00.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.115+0000 7f4231361700 1 --2- 172.21.15.16:0/2899332769 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f421006b020 0x7f421006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:00.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.115+0000 7f4221ffb700 1 -- 172.21.15.16:0/2899332769 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6050+0+0 (secure 0 0 0) 0x7f4224089e40 con 0x7f422c15ba10 2023-10-16T12:31:00.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.116+0000 7f4231361700 1 --2- 172.21.15.16:0/2899332769 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f421006b020 0x7f421006d4e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f422c06e990 tx=0x7f421c008070 comp rx=0 tx=0).ready entity=mgr.8973 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:00.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.120+0000 7f4221ffb700 1 -- 172.21.15.16:0/2899332769 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4224054c80 con 0x7f422c15ba10 2023-10-16T12:31:00.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.367+0000 7f42335c5700 1 -- 172.21.15.16:0/2899332769 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f422c04ebb0 con 0x7f422c15ba10 2023-10-16T12:31:00.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.368+0000 7f4221ffb700 1 -- 172.21.15.16:0/2899332769 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+3699 (secure 0 0 0) 0x7f42240582a0 con 0x7f422c15ba10 2023-10-16T12:31:00.370 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:31:00.370 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":72,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7220,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/4274593615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":4274593615},{"type":"v1","addr":"172.21.15.16:6837","nonce":4274593615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":64},{"gid":7550,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/2210838269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":2210838269},{"type":"v1","addr":"172.21.15.49:6835","nonce":2210838269}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":72},{"gid":8625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/1624531381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1624531381},{"type":"v1","addr":"172.21.15.16:6836","nonce":1624531381}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":8637,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/1629597454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":1629597454},{"type":"v1","addr":"172.21.15.49:6834","nonce":1629597454}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65}],"filesystems":[]} 2023-10-16T12:31:00.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.370+0000 7f42335c5700 1 -- 172.21.15.16:0/2899332769 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f421006b020 msgr2=0x7f421006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:00.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.370+0000 7f42335c5700 1 --2- 172.21.15.16:0/2899332769 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f421006b020 0x7f421006d4e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f422c06e990 tx=0x7f421c008070 comp rx=0 tx=0).stop 2023-10-16T12:31:00.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.370+0000 7f42335c5700 1 -- 172.21.15.16:0/2899332769 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f422c15ba10 msgr2=0x7f422c0ca3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:00.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.370+0000 7f42335c5700 1 --2- 172.21.15.16:0/2899332769 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f422c15ba10 0x7f422c0ca3b0 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7f4224007d20 tx=0x7f422400e3c0 comp rx=0 tx=0).stop 2023-10-16T12:31:00.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.370+0000 7f42335c5700 1 -- 172.21.15.16:0/2899332769 shutdown_connections 2023-10-16T12:31:00.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.370+0000 7f42335c5700 1 --2- 172.21.15.16:0/2899332769 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f421006b020 0x7f421006d4e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:00.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.370+0000 7f42335c5700 1 --2- 172.21.15.16:0/2899332769 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f422c15ba10 0x7f422c0ca3b0 unknown :-1 s=CLOSED pgs=1706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:00.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.370+0000 7f42335c5700 1 --2- 172.21.15.16:0/2899332769 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f422c0ce320 0x7f422c1f4070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:00.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.370+0000 7f42335c5700 1 --2- 172.21.15.16:0/2899332769 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f422c15b630 0x7f422c154a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:00.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.370+0000 7f42335c5700 1 -- 172.21.15.16:0/2899332769 >> 172.21.15.16:0/2899332769 conn(0x7f422c0d14b0 msgr2=0x7f422c15e790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:31:00.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.370+0000 7f42335c5700 1 -- 172.21.15.16:0/2899332769 shutdown_connections 2023-10-16T12:31:00.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.370+0000 7f42335c5700 1 -- 172.21.15.16:0/2899332769 wait complete. 2023-10-16T12:31:00.375 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 72 2023-10-16T12:31:00.384 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-16T12:31:00.384 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-16T12:31:00.385 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-16T12:31:00.385 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T12:31:00.385 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-16T12:31:00.385 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-16T12:31:00.386 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-16T12:31:00.386 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T12:31:00.386 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-16T12:31:00.387 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-16T12:31:00.387 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-16T12:31:00.387 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T12:31:00.387 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-16T12:31:00.388 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-16T12:31:00.388 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-16T12:31:00.388 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T12:31:00.396 INFO:tasks.ceph.mds.a:Stopped 2023-10-16T12:31:00.396 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-16T12:31:00.397 INFO:tasks.ceph.mds.c:Stopped 2023-10-16T12:31:00.397 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-16T12:31:00.401 INFO:tasks.ceph.mds.b:Stopped 2023-10-16T12:31:00.401 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-16T12:31:00.402 INFO:tasks.ceph.mds.d:Stopped 2023-10-16T12:31:00.403 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-16T12:31:00.438 INFO:tasks.ceph.mds.a:Started 2023-10-16T12:31:00.438 DEBUG:teuthology.parallel:result is None 2023-10-16T12:31:00.439 INFO:tasks.ceph.mds.c:Started 2023-10-16T12:31:00.439 DEBUG:teuthology.parallel:result is None 2023-10-16T12:31:00.444 INFO:tasks.ceph.mds.b:Started 2023-10-16T12:31:00.445 DEBUG:teuthology.parallel:result is None 2023-10-16T12:31:00.445 INFO:tasks.ceph.mds.d:Started 2023-10-16T12:31:00.446 DEBUG:teuthology.parallel:result is None 2023-10-16T12:31:00.446 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-16T12:31:00.521 INFO:tasks.ceph.mds.c.smithi016.stdout:starting mds.c at 2023-10-16T12:31:00.521 INFO:tasks.ceph.mds.a.smithi016.stdout:starting mds.a at 2023-10-16T12:31:00.533 INFO:tasks.ceph.mds.b.smithi049.stdout:starting mds.b at 2023-10-16T12:31:00.533 INFO:tasks.ceph.mds.d.smithi049.stdout:starting mds.d at 2023-10-16T12:31:00.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.576+0000 7f0c0c05c700 1 Processor -- start 2023-10-16T12:31:00.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.576+0000 7f0c0c05c700 1 -- start start 2023-10-16T12:31:00.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.577+0000 7f0c0c05c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0c040d2f70 0x7f0c040d2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:00.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.577+0000 7f0c0c05c700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f0c040ce9f0 0x7f0c040cdc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:00.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.577+0000 7f0c0c05c700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c040ceea0 con 0x7f0c0415d900 2023-10-16T12:31:00.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.577+0000 7f0c0c05c700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c040ce1a0 con 0x7f0c040ce9f0 2023-10-16T12:31:00.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.577+0000 7f0c0c05c700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c040ce400 con 0x7f0c040d2f70 2023-10-16T12:31:00.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.577+0000 7f0c09df8700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0c0415d900 0x7f0c040d2860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35990/0 (socket says 172.21.15.16:35990) 2023-10-16T12:31:00.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.577+0000 7f0c09df8700 1 -- 172.21.15.16:0/3813257050 learned_addr learned my addr 172.21.15.16:0/3813257050 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:31:00.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.578+0000 7f0c095f7700 1 --2- 172.21.15.16:0/3813257050 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0c040d2f70 0x7f0c040d2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:00.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.578+0000 7f0c0a5f9700 1 --2- 172.21.15.16:0/3813257050 >> v2:172.21.15.49:3300/0 conn(0x7f0c040ce9f0 0x7f0c040cdc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:00.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.578+0000 7f0c08df6700 1 -- 172.21.15.16:0/3813257050 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3590142880 0 0) 0x7f0c040ceea0 con 0x7f0c0415d900 2023-10-16T12:31:00.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.578+0000 7f0c08df6700 1 -- 172.21.15.16:0/3813257050 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bf00035b0 con 0x7f0c0415d900 2023-10-16T12:31:00.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.578+0000 7f0c095f7700 1 -- 172.21.15.16:0/3813257050 >> v2:172.21.15.49:3300/0 conn(0x7f0c040ce9f0 msgr2=0x7f0c040cdc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:31:00.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.578+0000 7f0c095f7700 1 --2- 172.21.15.16:0/3813257050 >> v2:172.21.15.49:3300/0 conn(0x7f0c040ce9f0 0x7f0c040cdc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:00.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.578+0000 7f0c095f7700 1 -- 172.21.15.16:0/3813257050 >> v1:172.21.15.16:6789/0 conn(0x7f0c0415d900 legacy=0x7f0c040d2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:00.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.578+0000 7f0c095f7700 1 -- 172.21.15.16:0/3813257050 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c0415dce0 con 0x7f0c040d2f70 2023-10-16T12:31:00.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.578+0000 7f0c095f7700 1 --2- 172.21.15.16:0/3813257050 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0c040d2f70 0x7f0c040d2040 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f0c00009840 tx=0x7f0c00009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c5c29e91c175b36 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:00.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.578+0000 7f0c08df6700 1 -- 172.21.15.16:0/3813257050 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c00011830 con 0x7f0c040d2f70 2023-10-16T12:31:00.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.578+0000 7f0c08df6700 1 -- 172.21.15.16:0/3813257050 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c00011990 con 0x7f0c040d2f70 2023-10-16T12:31:00.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.578+0000 7f0c08df6700 1 -- 172.21.15.16:0/3813257050 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c00011d20 con 0x7f0c040d2f70 2023-10-16T12:31:00.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.579+0000 7f0c0c05c700 1 -- 172.21.15.16:0/3813257050 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0c040d2f70 msgr2=0x7f0c040d2040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:00.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.579+0000 7f0c0c05c700 1 --2- 172.21.15.16:0/3813257050 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0c040d2f70 0x7f0c040d2040 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f0c00009840 tx=0x7f0c00009b50 comp rx=0 tx=0).stop 2023-10-16T12:31:00.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.579+0000 7f0c0c05c700 1 -- 172.21.15.16:0/3813257050 shutdown_connections 2023-10-16T12:31:00.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.579+0000 7f0c0c05c700 1 --2- 172.21.15.16:0/3813257050 >> v2:172.21.15.49:3300/0 conn(0x7f0c040ce9f0 0x7f0c040cdc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:00.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.579+0000 7f0c0c05c700 1 --2- 172.21.15.16:0/3813257050 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0c040d2f70 0x7f0c040d2040 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:00.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.579+0000 7f0c0c05c700 1 -- 172.21.15.16:0/3813257050 >> 172.21.15.16:0/3813257050 conn(0x7f0c04072330 msgr2=0x7f0c04072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:31:00.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.579+0000 7f0c0c05c700 1 -- 172.21.15.16:0/3813257050 shutdown_connections 2023-10-16T12:31:00.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.579+0000 7f0c0c05c700 1 -- 172.21.15.16:0/3813257050 wait complete. 2023-10-16T12:31:00.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.580+0000 7f0c0c05c700 1 Processor -- start 2023-10-16T12:31:00.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.580+0000 7f0c0c05c700 1 -- start start 2023-10-16T12:31:00.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.581+0000 7f0c0c05c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0c040ce9f0 0x7f0c0414b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:00.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.581+0000 7f0c0c05c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0c040d2f70 0x7f0c04157e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:00.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.581+0000 7f0c0c05c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c0415d900 0x7f0c040c8360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:00.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.581+0000 7f0c0c05c700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c04156d80 con 0x7f0c0415d900 2023-10-16T12:31:00.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.581+0000 7f0c0c05c700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c04157060 con 0x7f0c040d2f70 2023-10-16T12:31:00.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.581+0000 7f0c0c05c700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c04157340 con 0x7f0c040ce9f0 2023-10-16T12:31:00.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.581+0000 7f0c0a5f9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c0415d900 0x7f0c040c8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:00.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.581+0000 7f0c0a5f9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c0415d900 0x7f0c040c8360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46726/0 (socket says 172.21.15.16:46726) 2023-10-16T12:31:00.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.581+0000 7f0c09df8700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0c040ce9f0 0x7f0c0414b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:00.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.581+0000 7f0c095f7700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0c040d2f70 0x7f0c04157e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:00.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.581+0000 7f0c0a5f9700 1 -- 172.21.15.16:0/733596253 learned_addr learned my addr 172.21.15.16:0/733596253 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:31:00.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.581+0000 7f0c0a5f9700 1 -- 172.21.15.16:0/733596253 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0c040ce9f0 msgr2=0x7f0c0414b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:00.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.581+0000 7f0c0a5f9700 1 --2- 172.21.15.16:0/733596253 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0c040ce9f0 0x7f0c0414b550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:00.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.581+0000 7f0c0a5f9700 1 -- 172.21.15.16:0/733596253 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0c040d2f70 msgr2=0x7f0c04157e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:00.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.581+0000 7f0c0a5f9700 1 --2- 172.21.15.16:0/733596253 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0c040d2f70 0x7f0c04157e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:00.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.581+0000 7f0c0a5f9700 1 -- 172.21.15.16:0/733596253 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c000094a0 con 0x7f0c0415d900 2023-10-16T12:31:00.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.581+0000 7f0c0a5f9700 1 --2- 172.21.15.16:0/733596253 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c0415d900 0x7f0c040c8360 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7f0bfc00efc0 tx=0x7f0bfc00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:00.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.582+0000 7f0bfa7fc700 1 -- 172.21.15.16:0/733596253 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bfc009ea0 con 0x7f0c0415d900 2023-10-16T12:31:00.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.582+0000 7f0bfa7fc700 1 -- 172.21.15.16:0/733596253 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bfc00f040 con 0x7f0c0415d900 2023-10-16T12:31:00.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.582+0000 7f0bfa7fc700 1 -- 172.21.15.16:0/733596253 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bfc00ced0 con 0x7f0c0415d900 2023-10-16T12:31:00.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.582+0000 7f0c0c05c700 1 -- 172.21.15.16:0/733596253 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c04157620 con 0x7f0c0415d900 2023-10-16T12:31:00.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.582+0000 7f0c0c05c700 1 -- 172.21.15.16:0/733596253 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c04158360 con 0x7f0c0415d900 2023-10-16T12:31:00.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.583+0000 7f0c0c05c700 1 -- 172.21.15.16:0/733596253 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c04157b40 con 0x7f0c0415d900 2023-10-16T12:31:00.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.583+0000 7f0bfa7fc700 1 -- 172.21.15.16:0/733596253 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 85) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0bfc014480 con 0x7f0c0415d900 2023-10-16T12:31:00.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.584+0000 7f0bfa7fc700 1 --2- 172.21.15.16:0/733596253 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f0be806b020 0x7f0be806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:00.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.584+0000 7f0c09df8700 1 --2- 172.21.15.16:0/733596253 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f0be806b020 0x7f0be806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:00.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.584+0000 7f0bfa7fc700 1 -- 172.21.15.16:0/733596253 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6050+0+0 (secure 0 0 0) 0x7f0bfc08a210 con 0x7f0c0415d900 2023-10-16T12:31:00.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.584+0000 7f0c09df8700 1 --2- 172.21.15.16:0/733596253 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f0be806b020 0x7f0be806d4e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0bf4007fd0 tx=0x7f0bf4007dc0 comp rx=0 tx=0).ready entity=mgr.8973 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:00.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.588+0000 7f0bfa7fc700 1 -- 172.21.15.16:0/733596253 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0c04157b40 con 0x7f0c0415d900 2023-10-16T12:31:00.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:00.808+0000 7f0c0c05c700 1 -- 172.21.15.16:0/733596253 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f0c04157b40 con 0x7f0c0415d900 2023-10-16T12:31:02.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.346+0000 7f0bfa7fc700 1 -- 172.21.15.16:0/733596253 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v152) v1 ==== 97+0+0 (secure 0 0 0) 0x7f0c04157b40 con 0x7f0c0415d900 2023-10-16T12:31:02.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.350+0000 7f0c0c05c700 1 -- 172.21.15.16:0/733596253 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f0be806b020 msgr2=0x7f0be806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:02.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.350+0000 7f0c0c05c700 1 --2- 172.21.15.16:0/733596253 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f0be806b020 0x7f0be806d4e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0bf4007fd0 tx=0x7f0bf4007dc0 comp rx=0 tx=0).stop 2023-10-16T12:31:02.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.350+0000 7f0c0c05c700 1 -- 172.21.15.16:0/733596253 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c0415d900 msgr2=0x7f0c040c8360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:02.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.350+0000 7f0c0c05c700 1 --2- 172.21.15.16:0/733596253 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c0415d900 0x7f0c040c8360 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7f0bfc00efc0 tx=0x7f0bfc00c3a0 comp rx=0 tx=0).stop 2023-10-16T12:31:02.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.350+0000 7f0c0c05c700 1 -- 172.21.15.16:0/733596253 shutdown_connections 2023-10-16T12:31:02.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.350+0000 7f0c0c05c700 1 --2- 172.21.15.16:0/733596253 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f0be806b020 0x7f0be806d4e0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:02.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.350+0000 7f0c0c05c700 1 --2- 172.21.15.16:0/733596253 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c0415d900 0x7f0c040c8360 unknown :-1 s=CLOSED pgs=1714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:02.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.350+0000 7f0c0c05c700 1 --2- 172.21.15.16:0/733596253 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0c040ce9f0 0x7f0c0414b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:02.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.350+0000 7f0c0c05c700 1 --2- 172.21.15.16:0/733596253 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0c040d2f70 0x7f0c04157e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:02.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.350+0000 7f0c0c05c700 1 -- 172.21.15.16:0/733596253 >> 172.21.15.16:0/733596253 conn(0x7f0c04072330 msgr2=0x7f0c04154ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:31:02.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.350+0000 7f0c0c05c700 1 -- 172.21.15.16:0/733596253 shutdown_connections 2023-10-16T12:31:02.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.350+0000 7f0c0c05c700 1 -- 172.21.15.16:0/733596253 wait complete. 2023-10-16T12:31:02.354 INFO:teuthology.orchestra.run.smithi016.stderr: removed all blocklist entries 2023-10-16T12:31:02.366 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-16T12:31:02.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.499+0000 7f1e3f914700 1 Processor -- start 2023-10-16T12:31:02.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.499+0000 7f1e3f914700 1 -- start start 2023-10-16T12:31:02.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.500+0000 7f1e3f914700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f1e380cc8a0 0x7f1e3815e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:02.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.500+0000 7f1e3f914700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1e3806a7e0 0x7f1e3806ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:02.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.500+0000 7f1e3f914700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e3806b150 con 0x7f1e3815b280 2023-10-16T12:31:02.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.500+0000 7f1e3f914700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e38161c30 con 0x7f1e3806a7e0 2023-10-16T12:31:02.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.500+0000 7f1e3f914700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e38161f10 con 0x7f1e380cc8a0 2023-10-16T12:31:02.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.500+0000 7f1e3deb1700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1e3806a7e0 0x7f1e3806ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:02.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.500+0000 7f1e3deb1700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1e3806a7e0 0x7f1e3806ac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46738/0 (socket says 172.21.15.16:46738) 2023-10-16T12:31:02.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.500+0000 7f1e3deb1700 1 -- 172.21.15.16:0/2555354452 learned_addr learned my addr 172.21.15.16:0/2555354452 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:31:02.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.500+0000 7f1e3deb1700 1 -- 172.21.15.16:0/2555354452 >> v2:172.21.15.49:3300/0 conn(0x7f1e380cc8a0 msgr2=0x7f1e3815e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:31:02.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.500+0000 7f1e3deb1700 1 --2- 172.21.15.16:0/2555354452 >> v2:172.21.15.49:3300/0 conn(0x7f1e380cc8a0 0x7f1e3815e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:02.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.500+0000 7f1e3deb1700 1 -- 172.21.15.16:0/2555354452 >> v1:172.21.15.49:6789/0 conn(0x7f1e3815b280 legacy=0x7f1e381613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:31:02.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.500+0000 7f1e3deb1700 1 -- 172.21.15.16:0/2555354452 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e381621f0 con 0x7f1e3806a7e0 2023-10-16T12:31:02.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.500+0000 7f1e3deb1700 1 --2- 172.21.15.16:0/2555354452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e3806a7e0 0x7f1e3806ac10 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7f1e38071970 tx=0x7f1e3000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a785fc6b7ce8b29a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:02.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.500+0000 7f1e2ffff700 1 -- 172.21.15.16:0/2555354452 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e3000e070 con 0x7f1e3806a7e0 2023-10-16T12:31:02.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.500+0000 7f1e2ffff700 1 -- 172.21.15.16:0/2555354452 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e30003a20 con 0x7f1e3806a7e0 2023-10-16T12:31:02.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.500+0000 7f1e2ffff700 1 -- 172.21.15.16:0/2555354452 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e30003dd0 con 0x7f1e3806a7e0 2023-10-16T12:31:02.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.501+0000 7f1e3f914700 1 -- 172.21.15.16:0/2555354452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e3806a7e0 msgr2=0x7f1e3806ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:02.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.501+0000 7f1e3f914700 1 --2- 172.21.15.16:0/2555354452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e3806a7e0 0x7f1e3806ac10 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7f1e38071970 tx=0x7f1e3000b330 comp rx=0 tx=0).stop 2023-10-16T12:31:02.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.501+0000 7f1e3f914700 1 -- 172.21.15.16:0/2555354452 shutdown_connections 2023-10-16T12:31:02.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.501+0000 7f1e3f914700 1 --2- 172.21.15.16:0/2555354452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e3806a7e0 0x7f1e3806ac10 unknown :-1 s=CLOSED pgs=1715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:02.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.501+0000 7f1e3f914700 1 --2- 172.21.15.16:0/2555354452 >> v2:172.21.15.49:3300/0 conn(0x7f1e380cc8a0 0x7f1e3815e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:02.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.501+0000 7f1e3f914700 1 -- 172.21.15.16:0/2555354452 >> 172.21.15.16:0/2555354452 conn(0x7f1e380b1fd0 msgr2=0x7f1e380b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:31:02.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.501+0000 7f1e3f914700 1 -- 172.21.15.16:0/2555354452 shutdown_connections 2023-10-16T12:31:02.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.501+0000 7f1e3f914700 1 -- 172.21.15.16:0/2555354452 wait complete. 2023-10-16T12:31:02.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.502+0000 7f1e3f914700 1 Processor -- start 2023-10-16T12:31:02.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.502+0000 7f1e3f914700 1 -- start start 2023-10-16T12:31:02.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.503+0000 7f1e3f914700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1e3806a7e0 0x7f1e381f2db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:02.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.503+0000 7f1e3f914700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e380cc8a0 0x7f1e380c4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:02.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.503+0000 7f1e3f914700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1e3815b280 0x7f1e380c8910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:02.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.503+0000 7f1e3f914700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e380c5390 con 0x7f1e380cc8a0 2023-10-16T12:31:02.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.503+0000 7f1e3f914700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e380cb550 con 0x7f1e3806a7e0 2023-10-16T12:31:02.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.503+0000 7f1e3f914700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e380c8e50 con 0x7f1e3815b280 2023-10-16T12:31:02.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.503+0000 7f1e3ceaf700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e380cc8a0 0x7f1e380c4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:02.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.503+0000 7f1e3ceaf700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e380cc8a0 0x7f1e380c4e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46742/0 (socket says 172.21.15.16:46742) 2023-10-16T12:31:02.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.503+0000 7f1e3ceaf700 1 -- 172.21.15.16:0/3156627764 learned_addr learned my addr 172.21.15.16:0/3156627764 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:31:02.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.503+0000 7f1e3d6b0700 1 --2- 172.21.15.16:0/3156627764 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1e3806a7e0 0x7f1e381f2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:02.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.503+0000 7f1e3deb1700 1 --2- 172.21.15.16:0/3156627764 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1e3815b280 0x7f1e380c8910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:02.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.503+0000 7f1e3ceaf700 1 -- 172.21.15.16:0/3156627764 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1e3815b280 msgr2=0x7f1e380c8910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:02.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.503+0000 7f1e3ceaf700 1 --2- 172.21.15.16:0/3156627764 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1e3815b280 0x7f1e380c8910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:02.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.503+0000 7f1e3ceaf700 1 -- 172.21.15.16:0/3156627764 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1e3806a7e0 msgr2=0x7f1e381f2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:02.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.503+0000 7f1e3ceaf700 1 --2- 172.21.15.16:0/3156627764 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1e3806a7e0 0x7f1e381f2db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:02.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.503+0000 7f1e3ceaf700 1 -- 172.21.15.16:0/3156627764 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e30009b30 con 0x7f1e380cc8a0 2023-10-16T12:31:02.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.503+0000 7f1e3deb1700 1 --2- 172.21.15.16:0/3156627764 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1e3815b280 0x7f1e380c8910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:31:02.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.504+0000 7f1e3ceaf700 1 --2- 172.21.15.16:0/3156627764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e380cc8a0 0x7f1e380c4e50 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7f1e2800b500 tx=0x7f1e2800b8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:02.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.504+0000 7f1e2dffb700 1 -- 172.21.15.16:0/3156627764 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e28011830 con 0x7f1e380cc8a0 2023-10-16T12:31:02.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.504+0000 7f1e2dffb700 1 -- 172.21.15.16:0/3156627764 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e280119b0 con 0x7f1e380cc8a0 2023-10-16T12:31:02.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.504+0000 7f1e3f914700 1 -- 172.21.15.16:0/3156627764 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e381f3410 con 0x7f1e380cc8a0 2023-10-16T12:31:02.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.504+0000 7f1e2dffb700 1 -- 172.21.15.16:0/3156627764 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e28011d60 con 0x7f1e380cc8a0 2023-10-16T12:31:02.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.504+0000 7f1e3f914700 1 -- 172.21.15.16:0/3156627764 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e381f39d0 con 0x7f1e380cc8a0 2023-10-16T12:31:02.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.505+0000 7f1e3f914700 1 -- 172.21.15.16:0/3156627764 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e3804ebb0 con 0x7f1e380cc8a0 2023-10-16T12:31:02.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.505+0000 7f1e2dffb700 1 -- 172.21.15.16:0/3156627764 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 85) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1e2801a400 con 0x7f1e380cc8a0 2023-10-16T12:31:02.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.506+0000 7f1e2dffb700 1 --2- 172.21.15.16:0/3156627764 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f1e1c06aff0 0x7f1e1c06d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:02.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.506+0000 7f1e2dffb700 1 -- 172.21.15.16:0/3156627764 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5233+0+0 (secure 0 0 0) 0x7f1e280895c0 con 0x7f1e380cc8a0 2023-10-16T12:31:02.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.506+0000 7f1e3d6b0700 1 --2- 172.21.15.16:0/3156627764 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f1e1c06aff0 0x7f1e1c06d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:02.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.506+0000 7f1e3d6b0700 1 --2- 172.21.15.16:0/3156627764 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f1e1c06aff0 0x7f1e1c06d4b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1e38071df0 tx=0x7f1e34008040 comp rx=0 tx=0).ready entity=mgr.8973 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:02.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.510+0000 7f1e2dffb700 1 -- 172.21.15.16:0/3156627764 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1e280547c0 con 0x7f1e380cc8a0 2023-10-16T12:31:02.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.795+0000 7f1e3f914700 1 -- 172.21.15.16:0/3156627764 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f1e380ca800 con 0x7f1e380cc8a0 2023-10-16T12:31:02.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.795+0000 7f1e2dffb700 1 -- 172.21.15.16:0/3156627764 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v25) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f1e28015090 con 0x7f1e380cc8a0 2023-10-16T12:31:02.798 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:31:02.798 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-10-16T12:31:02.798 INFO:teuthology.orchestra.run.smithi016.stdout: "auth_dump": [ 2023-10-16T12:31:02.799 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:31:02.799 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.a", 2023-10-16T12:31:02.799 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOKC1l/rBtOhAA/ob5WUvpz8Glp4E+xPZOYA==", 2023-10-16T12:31:02.799 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:31:02.800 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T12:31:02.800 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-10-16T12:31:02.800 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-10-16T12:31:02.800 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:31:02.800 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:31:02.801 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:31:02.801 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:31:02.801 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.b", 2023-10-16T12:31:02.801 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1lKAVsCxAAvhXuLY6i1+Iz1yq5BVPMNQ==", 2023-10-16T12:31:02.801 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:31:02.802 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T12:31:02.802 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-10-16T12:31:02.802 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-10-16T12:31:02.802 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:31:02.802 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:31:02.803 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:31:02.803 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:31:02.803 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.c", 2023-10-16T12:31:02.803 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1lvBowBhAA3waJNYhdgs8PGCYqYNYUog==", 2023-10-16T12:31:02.803 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:31:02.804 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T12:31:02.804 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-10-16T12:31:02.804 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-10-16T12:31:02.804 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:31:02.804 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:31:02.804 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:31:02.805 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:31:02.805 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.d", 2023-10-16T12:31:02.805 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1lS/fnEhAAfNYgQ5BW86JMjYm/bBfnaQ==", 2023-10-16T12:31:02.805 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:31:02.805 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T12:31:02.806 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-10-16T12:31:02.806 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-10-16T12:31:02.806 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:31:02.806 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:31:02.806 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:31:02.807 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:31:02.807 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.0", 2023-10-16T12:31:02.807 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAQKC1lbxllOhAAZeLdp2GMl6HqQPaQjKlNqw==", 2023-10-16T12:31:02.807 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:31:02.807 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T12:31:02.808 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T12:31:02.808 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:31:02.808 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:31:02.808 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:31:02.808 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:31:02.809 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.1", 2023-10-16T12:31:02.809 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQATKC1lnU6KOhAAHxvlbmzBptf1keGgL+Rbkg==", 2023-10-16T12:31:02.809 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:31:02.809 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T12:31:02.809 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T12:31:02.810 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:31:02.810 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:31:02.810 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:31:02.810 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:31:02.810 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.2", 2023-10-16T12:31:02.811 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAXKC1l9jFyARAAr/NAwtWhBMut9sAAoDEGWg==", 2023-10-16T12:31:02.811 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:31:02.811 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T12:31:02.811 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T12:31:02.811 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:31:02.812 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:31:02.812 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:31:02.812 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:31:02.812 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.3", 2023-10-16T12:31:02.812 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAaKC1lbw/lARAAGJ/9DNb5/7kLFSq7vrLtXg==", 2023-10-16T12:31:02.813 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:31:02.813 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T12:31:02.813 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T12:31:02.813 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:31:02.813 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:31:02.814 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:31:02.814 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:31:02.814 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.4", 2023-10-16T12:31:02.814 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAeKC1lZb7jERAAYpvb1xcSli9CTq7L8ChLOA==", 2023-10-16T12:31:02.814 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:31:02.815 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T12:31:02.815 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T12:31:02.815 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:31:02.815 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:31:02.815 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:31:02.816 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:31:02.816 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.5", 2023-10-16T12:31:02.816 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAhKC1l/4HuEBAAqfnRqjknij9KES/fqiyIBQ==", 2023-10-16T12:31:02.816 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:31:02.816 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T12:31:02.816 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T12:31:02.817 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:31:02.817 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:31:02.817 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:31:02.817 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:31:02.817 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.6", 2023-10-16T12:31:02.818 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAkKC1lzoUkEBAA+Q82OiBMNag8+R316WQtUw==", 2023-10-16T12:31:02.818 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:31:02.818 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T12:31:02.818 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T12:31:02.818 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:31:02.819 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:31:02.819 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:31:02.819 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:31:02.819 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.7", 2023-10-16T12:31:02.819 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAnKC1ljUu4DxAAc+G4anJANC2GhLJ8wb6gYg==", 2023-10-16T12:31:02.820 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:31:02.820 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T12:31:02.820 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T12:31:02.820 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:31:02.820 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:31:02.821 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:31:02.821 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:31:02.821 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.0", 2023-10-16T12:31:02.821 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1l2l6PFxAA9QyXtBAHdGAXpwH0JfknuQ==", 2023-10-16T12:31:02.822 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:31:02.822 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T12:31:02.822 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow r", 2023-10-16T12:31:02.822 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-16T12:31:02.822 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:31:02.823 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:31:02.823 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:31:02.823 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.1", 2023-10-16T12:31:02.823 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1lAcHFIxAAcNt2bmv2R6j5L3DYkdw7eQ==", 2023-10-16T12:31:02.823 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:31:02.824 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T12:31:02.824 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-10-16T12:31:02.824 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-10-16T12:31:02.824 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-10-16T12:31:02.824 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:31:02.824 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:31:02.825 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:31:02.825 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.2", 2023-10-16T12:31:02.825 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1lA6wnGxAAdtaqYhtIJu8qENMrD8xvVA==", 2023-10-16T12:31:02.825 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:31:02.825 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T12:31:02.826 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-10-16T12:31:02.826 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-10-16T12:31:02.826 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-10-16T12:31:02.826 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:31:02.826 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:31:02.827 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:31:02.827 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.3", 2023-10-16T12:31:02.827 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1l/ddAJxAA3fNKKn9XkXas1BlDN+ggjw==", 2023-10-16T12:31:02.827 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:31:02.827 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T12:31:02.828 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-10-16T12:31:02.828 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-10-16T12:31:02.828 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-10-16T12:31:02.828 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:31:02.828 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:31:02.829 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:31:02.829 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.admin", 2023-10-16T12:31:02.829 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOKC1ld5jUHBAAc6esnCce3dSEZn3wVjhIlQ==", 2023-10-16T12:31:02.829 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:31:02.829 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2023-10-16T12:31:02.830 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-10-16T12:31:02.830 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-10-16T12:31:02.830 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:31:02.830 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:31:02.830 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:31:02.831 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:31:02.831 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mgr.x", 2023-10-16T12:31:02.831 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOKC1l6b3vNhAAmj5xbfid2sZM2KfHmAf1Wg==", 2023-10-16T12:31:02.831 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:31:02.831 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2023-10-16T12:31:02.832 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile mgr", 2023-10-16T12:31:02.832 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:31:02.832 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:31:02.832 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T12:31:02.832 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:31:02.833 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mgr.y", 2023-10-16T12:31:02.833 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOKC1lLz2nMhAA8NqFYZ5nDKkzPBMrkwsVuQ==", 2023-10-16T12:31:02.833 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T12:31:02.833 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2023-10-16T12:31:02.833 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile mgr", 2023-10-16T12:31:02.834 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T12:31:02.834 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:31:02.834 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:31:02.834 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-10-16T12:31:02.834 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-10-16T12:31:02.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.798+0000 7f1e3f914700 1 -- 172.21.15.16:0/3156627764 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f1e1c06aff0 msgr2=0x7f1e1c06d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:02.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.798+0000 7f1e3f914700 1 --2- 172.21.15.16:0/3156627764 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f1e1c06aff0 0x7f1e1c06d4b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1e38071df0 tx=0x7f1e34008040 comp rx=0 tx=0).stop 2023-10-16T12:31:02.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.798+0000 7f1e3f914700 1 -- 172.21.15.16:0/3156627764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e380cc8a0 msgr2=0x7f1e380c4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:02.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.798+0000 7f1e3f914700 1 --2- 172.21.15.16:0/3156627764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e380cc8a0 0x7f1e380c4e50 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7f1e2800b500 tx=0x7f1e2800b8c0 comp rx=0 tx=0).stop 2023-10-16T12:31:02.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.798+0000 7f1e3f914700 1 -- 172.21.15.16:0/3156627764 shutdown_connections 2023-10-16T12:31:02.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.798+0000 7f1e3f914700 1 --2- 172.21.15.16:0/3156627764 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f1e1c06aff0 0x7f1e1c06d4b0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:02.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.798+0000 7f1e3f914700 1 --2- 172.21.15.16:0/3156627764 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1e3815b280 0x7f1e380c8910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:02.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.798+0000 7f1e3f914700 1 --2- 172.21.15.16:0/3156627764 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1e3806a7e0 0x7f1e381f2db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:02.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.798+0000 7f1e3f914700 1 --2- 172.21.15.16:0/3156627764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e380cc8a0 0x7f1e380c4e50 unknown :-1 s=CLOSED pgs=1716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:02.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.798+0000 7f1e3f914700 1 -- 172.21.15.16:0/3156627764 >> 172.21.15.16:0/3156627764 conn(0x7f1e380b1fd0 msgr2=0x7f1e380cce10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:31:02.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.798+0000 7f1e3f914700 1 -- 172.21.15.16:0/3156627764 shutdown_connections 2023-10-16T12:31:02.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.798+0000 7f1e3f914700 1 -- 172.21.15.16:0/3156627764 wait complete. 2023-10-16T12:31:02.839 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:31:02.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.962+0000 7f682e946700 1 Processor -- start 2023-10-16T12:31:02.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.962+0000 7f682e946700 1 -- start start 2023-10-16T12:31:02.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.963+0000 7f682e946700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f682806a7e0 0x7f682806ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:02.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.963+0000 7f682e946700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f682806b150 con 0x7f68280cc8a0 2023-10-16T12:31:02.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.963+0000 7f682e946700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6828161c30 con 0x7f682815b280 2023-10-16T12:31:02.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.963+0000 7f682e946700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6828161f10 con 0x7f682806a7e0 2023-10-16T12:31:02.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.963+0000 7f68277fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f68280cc8a0 0x7f682815e700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36024/0 (socket says 172.21.15.16:36024) 2023-10-16T12:31:02.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.963+0000 7f682cee3700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f682806a7e0 0x7f682806ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:02.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.963+0000 7f68277fe700 1 -- 172.21.15.16:0/1267730630 learned_addr learned my addr 172.21.15.16:0/1267730630 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:31:02.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.963+0000 7f682cee3700 1 -- 172.21.15.16:0/1267730630 >> v1:172.21.15.49:6789/0 conn(0x7f682815b280 legacy=0x7f68281613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:02.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.963+0000 7f682cee3700 1 -- 172.21.15.16:0/1267730630 >> v1:172.21.15.16:6789/0 conn(0x7f68280cc8a0 legacy=0x7f682815e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:02.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.963+0000 7f6826ffd700 1 -- 172.21.15.16:0/1267730630 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3112386751 0 0) 0x7f682806b150 con 0x7f68280cc8a0 2023-10-16T12:31:02.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.963+0000 7f682cee3700 1 -- 172.21.15.16:0/1267730630 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68281621f0 con 0x7f682806a7e0 2023-10-16T12:31:02.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.963+0000 7f682cee3700 1 --2- 172.21.15.16:0/1267730630 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f682806a7e0 0x7f682806ac10 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7f6820009d90 tx=0x7f682000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28b0a25ca624f146 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:02.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.964+0000 7f6826ffd700 1 -- 172.21.15.16:0/1267730630 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f682000e070 con 0x7f682806a7e0 2023-10-16T12:31:02.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.964+0000 7f6826ffd700 1 -- 172.21.15.16:0/1267730630 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f682000bbd0 con 0x7f682806a7e0 2023-10-16T12:31:02.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.964+0000 7f6826ffd700 1 -- 172.21.15.16:0/1267730630 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6820012430 con 0x7f682806a7e0 2023-10-16T12:31:02.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.964+0000 7f682e946700 1 -- 172.21.15.16:0/1267730630 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f682806a7e0 msgr2=0x7f682806ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:02.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.964+0000 7f682e946700 1 --2- 172.21.15.16:0/1267730630 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f682806a7e0 0x7f682806ac10 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7f6820009d90 tx=0x7f682000b330 comp rx=0 tx=0).stop 2023-10-16T12:31:02.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.964+0000 7f682e946700 1 -- 172.21.15.16:0/1267730630 shutdown_connections 2023-10-16T12:31:02.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.964+0000 7f682e946700 1 --2- 172.21.15.16:0/1267730630 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f682806a7e0 0x7f682806ac10 unknown :-1 s=CLOSED pgs=1718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:02.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.964+0000 7f682e946700 1 -- 172.21.15.16:0/1267730630 >> 172.21.15.16:0/1267730630 conn(0x7f68280b1fd0 msgr2=0x7f68280b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:31:02.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.964+0000 7f682e946700 1 -- 172.21.15.16:0/1267730630 shutdown_connections 2023-10-16T12:31:02.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.964+0000 7f682e946700 1 -- 172.21.15.16:0/1267730630 wait complete. 2023-10-16T12:31:02.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.965+0000 7f682e946700 1 Processor -- start 2023-10-16T12:31:02.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.966+0000 7f682e946700 1 -- start start 2023-10-16T12:31:02.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.966+0000 7f682e946700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f682806a7e0 0x7f68280c4980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:02.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.966+0000 7f682e946700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f68280cc8a0 0x7f68281f29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:02.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.966+0000 7f682e946700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f682815b280 0x7f682815f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:02.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.966+0000 7f682e946700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68281581f0 con 0x7f682806a7e0 2023-10-16T12:31:02.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.966+0000 7f682e946700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68280c5080 con 0x7f68280cc8a0 2023-10-16T12:31:02.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.966+0000 7f682e946700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68280c5360 con 0x7f682815b280 2023-10-16T12:31:02.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.966+0000 7f6827fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f682806a7e0 0x7f68280c4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:02.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.966+0000 7f6827fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f682806a7e0 0x7f68280c4980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46754/0 (socket says 172.21.15.16:46754) 2023-10-16T12:31:02.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.966+0000 7f6827fff700 1 -- 172.21.15.16:0/2858913708 learned_addr learned my addr 172.21.15.16:0/2858913708 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:31:02.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.967+0000 7f68277fe700 1 --2- 172.21.15.16:0/2858913708 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f68280cc8a0 0x7f68281f29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:02.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.967+0000 7f682cee3700 1 --2- 172.21.15.16:0/2858913708 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f682815b280 0x7f682815f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:02.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.967+0000 7f6827fff700 1 -- 172.21.15.16:0/2858913708 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f682815b280 msgr2=0x7f682815f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:02.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.967+0000 7f6827fff700 1 --2- 172.21.15.16:0/2858913708 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f682815b280 0x7f682815f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:02.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.967+0000 7f6827fff700 1 -- 172.21.15.16:0/2858913708 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f68280cc8a0 msgr2=0x7f68281f29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:02.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.967+0000 7f6827fff700 1 --2- 172.21.15.16:0/2858913708 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f68280cc8a0 0x7f68281f29e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:02.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.967+0000 7f6827fff700 1 -- 172.21.15.16:0/2858913708 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6820009a70 con 0x7f682806a7e0 2023-10-16T12:31:02.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.967+0000 7f68277fe700 1 --2- 172.21.15.16:0/2858913708 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f68280cc8a0 0x7f68281f29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:31:02.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.967+0000 7f682cee3700 1 --2- 172.21.15.16:0/2858913708 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f682815b280 0x7f682815f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:31:02.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.967+0000 7f6827fff700 1 --2- 172.21.15.16:0/2858913708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f682806a7e0 0x7f68280c4980 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7f681800c450 tx=0x7f681800c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:02.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.967+0000 7f6824ff9700 1 -- 172.21.15.16:0/2858913708 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68180052d0 con 0x7f682806a7e0 2023-10-16T12:31:02.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.967+0000 7f6824ff9700 1 -- 172.21.15.16:0/2858913708 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6818005430 con 0x7f682806a7e0 2023-10-16T12:31:02.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.967+0000 7f6824ff9700 1 -- 172.21.15.16:0/2858913708 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68180038c0 con 0x7f682806a7e0 2023-10-16T12:31:02.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.967+0000 7f682e946700 1 -- 172.21.15.16:0/2858913708 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68280c5640 con 0x7f682806a7e0 2023-10-16T12:31:02.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.967+0000 7f682e946700 1 -- 172.21.15.16:0/2858913708 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68280c5c00 con 0x7f682806a7e0 2023-10-16T12:31:02.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.968+0000 7f682e946700 1 -- 172.21.15.16:0/2858913708 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68280c5e20 con 0x7f682806a7e0 2023-10-16T12:31:02.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.969+0000 7f6824ff9700 1 -- 172.21.15.16:0/2858913708 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 85) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6818003a20 con 0x7f682806a7e0 2023-10-16T12:31:02.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.969+0000 7f6824ff9700 1 --2- 172.21.15.16:0/2858913708 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f681406b050 0x7f681406d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:02.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.969+0000 7f6824ff9700 1 -- 172.21.15.16:0/2858913708 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5233+0+0 (secure 0 0 0) 0x7f6818088890 con 0x7f682806a7e0 2023-10-16T12:31:02.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.969+0000 7f68277fe700 1 --2- 172.21.15.16:0/2858913708 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f681406b050 0x7f681406d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:02.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.970+0000 7f68277fe700 1 --2- 172.21.15.16:0/2858913708 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f681406b050 0x7f681406d510 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f68280d0f90 tx=0x7f681c009040 comp rx=0 tx=0).ready entity=mgr.8973 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:02.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:02.974+0000 7f6824ff9700 1 -- 172.21.15.16:0/2858913708 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6818053a00 con 0x7f682806a7e0 2023-10-16T12:31:03.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.179+0000 7f682e946700 1 -- 172.21.15.16:0/2858913708 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f682804ebb0 con 0x7f682806a7e0 2023-10-16T12:31:03.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.180+0000 7f6824ff9700 1 -- 172.21.15.16:0/2858913708 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v152) v1 ==== 74+0+13052 (secure 0 0 0) 0x7f6818057020 con 0x7f682806a7e0 2023-10-16T12:31:03.182 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:31:03.183 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":152,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:31:02.345821+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:31:03.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.182+0000 7f682e946700 1 -- 172.21.15.16:0/2858913708 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f681406b050 msgr2=0x7f681406d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:03.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.182+0000 7f682e946700 1 --2- 172.21.15.16:0/2858913708 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f681406b050 0x7f681406d510 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f68280d0f90 tx=0x7f681c009040 comp rx=0 tx=0).stop 2023-10-16T12:31:03.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.182+0000 7f682e946700 1 -- 172.21.15.16:0/2858913708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f682806a7e0 msgr2=0x7f68280c4980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:03.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.182+0000 7f682e946700 1 --2- 172.21.15.16:0/2858913708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f682806a7e0 0x7f68280c4980 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7f681800c450 tx=0x7f681800c810 comp rx=0 tx=0).stop 2023-10-16T12:31:03.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.182+0000 7f682e946700 1 -- 172.21.15.16:0/2858913708 shutdown_connections 2023-10-16T12:31:03.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.182+0000 7f682e946700 1 --2- 172.21.15.16:0/2858913708 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f681406b050 0x7f681406d510 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:03.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.182+0000 7f682e946700 1 --2- 172.21.15.16:0/2858913708 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f682815b280 0x7f682815f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:03.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.182+0000 7f682e946700 1 --2- 172.21.15.16:0/2858913708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f682806a7e0 0x7f68280c4980 unknown :-1 s=CLOSED pgs=1719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:03.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.182+0000 7f682e946700 1 --2- 172.21.15.16:0/2858913708 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f68280cc8a0 0x7f68281f29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:03.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.182+0000 7f682e946700 1 -- 172.21.15.16:0/2858913708 >> 172.21.15.16:0/2858913708 conn(0x7f68280b1fd0 msgr2=0x7f6828159a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:31:03.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.183+0000 7f682e946700 1 -- 172.21.15.16:0/2858913708 shutdown_connections 2023-10-16T12:31:03.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.183+0000 7f682e946700 1 -- 172.21.15.16:0/2858913708 wait complete. 2023-10-16T12:31:03.198 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T12:31:03.199 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T12:31:03.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.325+0000 7fdb3c507700 1 Processor -- start 2023-10-16T12:31:03.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.325+0000 7fdb3c507700 1 -- start start 2023-10-16T12:31:03.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.326+0000 7fdb3c507700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fdb3415ec80 0x7fdb340cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:03.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.326+0000 7fdb3c507700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdb3415f060 0x7fdb340ce400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:03.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.326+0000 7fdb3c507700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb340b2440 con 0x7fdb340d2110 2023-10-16T12:31:03.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.326+0000 7fdb3c507700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb340b2c50 con 0x7fdb3415ec80 2023-10-16T12:31:03.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.326+0000 7fdb3c507700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb340d00e0 con 0x7fdb3415f060 2023-10-16T12:31:03.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.326+0000 7fdb3aaa4700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fdb340d2110 0x7fdb340b4440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36036/0 (socket says 172.21.15.16:36036) 2023-10-16T12:31:03.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.326+0000 7fdb3aaa4700 1 -- 172.21.15.16:0/1643367980 learned_addr learned my addr 172.21.15.16:0/1643367980 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:31:03.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.327+0000 7fdb3a2a3700 1 --2- 172.21.15.16:0/1643367980 >> v2:172.21.15.49:3300/0 conn(0x7fdb3415ec80 0x7fdb340cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:03.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.327+0000 7fdb39aa2700 1 --2- 172.21.15.16:0/1643367980 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdb3415f060 0x7fdb340ce400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:03.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.327+0000 7fdb392a1700 1 -- 172.21.15.16:0/1643367980 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 223011559 0 0) 0x7fdb340b2440 con 0x7fdb340d2110 2023-10-16T12:31:03.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.327+0000 7fdb392a1700 1 -- 172.21.15.16:0/1643367980 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb200035b0 con 0x7fdb340d2110 2023-10-16T12:31:03.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.327+0000 7fdb392a1700 1 -- 172.21.15.16:0/1643367980 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3170947413 0 0) 0x7fdb200035b0 con 0x7fdb340d2110 2023-10-16T12:31:03.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.327+0000 7fdb392a1700 1 -- 172.21.15.16:0/1643367980 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdb3415f060 msgr2=0x7fdb340ce400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:31:03.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.327+0000 7fdb392a1700 1 --2- 172.21.15.16:0/1643367980 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdb3415f060 0x7fdb340ce400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:03.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.327+0000 7fdb392a1700 1 -- 172.21.15.16:0/1643367980 >> v2:172.21.15.49:3300/0 conn(0x7fdb3415ec80 msgr2=0x7fdb340cdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:31:03.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.327+0000 7fdb392a1700 1 --2- 172.21.15.16:0/1643367980 >> v2:172.21.15.49:3300/0 conn(0x7fdb3415ec80 0x7fdb340cdec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:03.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.327+0000 7fdb392a1700 1 -- 172.21.15.16:0/1643367980 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb340cef60 con 0x7fdb340d2110 2023-10-16T12:31:03.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.327+0000 7fdb392a1700 1 -- 172.21.15.16:0/1643367980 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7fdb2c0019f0 con 0x7fdb340d2110 2023-10-16T12:31:03.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.327+0000 7fdb392a1700 1 -- 172.21.15.16:0/1643367980 >> v1:172.21.15.16:6789/0 conn(0x7fdb340d2110 legacy=0x7fdb340b4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:03.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.327+0000 7fdb392a1700 1 --2- 172.21.15.16:0/1643367980 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb20003a00 0x7fdb20003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:03.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.327+0000 7fdb392a1700 1 --2- 172.21.15.16:0/1643367980 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdb20004850 0x7fdb20004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:03.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.327+0000 7fdb392a1700 1 --2- 172.21.15.16:0/1643367980 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdb3415f060 0x7fdb20005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:03.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.327+0000 7fdb392a1700 1 -- 172.21.15.16:0/1643367980 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb340b2c50 con 0x7fdb20003a00 2023-10-16T12:31:03.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.327+0000 7fdb392a1700 1 -- 172.21.15.16:0/1643367980 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb340d00e0 con 0x7fdb20004850 2023-10-16T12:31:03.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.327+0000 7fdb392a1700 1 -- 172.21.15.16:0/1643367980 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb2000ba10 con 0x7fdb3415f060 2023-10-16T12:31:03.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.327+0000 7fdb3a2a3700 1 --2- 172.21.15.16:0/1643367980 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb20003a00 0x7fdb20003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:03.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.327+0000 7fdb39aa2700 1 --2- 172.21.15.16:0/1643367980 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdb20004850 0x7fdb20004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:03.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.327+0000 7fdb3aaa4700 1 --2- 172.21.15.16:0/1643367980 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdb3415f060 0x7fdb20005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:03.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.327+0000 7fdb3a2a3700 1 -- 172.21.15.16:0/1643367980 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdb3415f060 msgr2=0x7fdb20005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:31:03.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.327+0000 7fdb3a2a3700 1 --2- 172.21.15.16:0/1643367980 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdb3415f060 0x7fdb20005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:03.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.327+0000 7fdb3a2a3700 1 -- 172.21.15.16:0/1643367980 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdb20004850 msgr2=0x7fdb20004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:31:03.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.327+0000 7fdb3a2a3700 1 --2- 172.21.15.16:0/1643367980 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdb20004850 0x7fdb20004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:03.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.327+0000 7fdb3a2a3700 1 -- 172.21.15.16:0/1643367980 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb200035b0 con 0x7fdb20003a00 2023-10-16T12:31:03.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.327+0000 7fdb3a2a3700 1 --2- 172.21.15.16:0/1643367980 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb20003a00 0x7fdb20003e20 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7fdb24009b60 tx=0x7fdb20005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b130e5c13dbd9aa server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:03.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.328+0000 7fdb392a1700 1 -- 172.21.15.16:0/1643367980 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb2400e6c0 con 0x7fdb20003a00 2023-10-16T12:31:03.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.328+0000 7fdb392a1700 1 -- 172.21.15.16:0/1643367980 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb2400e820 con 0x7fdb20003a00 2023-10-16T12:31:03.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.328+0000 7fdb392a1700 1 -- 172.21.15.16:0/1643367980 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb2400eb90 con 0x7fdb20003a00 2023-10-16T12:31:03.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.328+0000 7fdb3c507700 1 -- 172.21.15.16:0/1643367980 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb20003a00 msgr2=0x7fdb20003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:03.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.328+0000 7fdb3c507700 1 --2- 172.21.15.16:0/1643367980 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb20003a00 0x7fdb20003e20 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7fdb24009b60 tx=0x7fdb20005880 comp rx=0 tx=0).stop 2023-10-16T12:31:03.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.328+0000 7fdb3aaa4700 1 -- 172.21.15.16:0/1643367980 reap_dead start 2023-10-16T12:31:03.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.328+0000 7fdb3c507700 1 -- 172.21.15.16:0/1643367980 shutdown_connections 2023-10-16T12:31:03.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.328+0000 7fdb3c507700 1 -- 172.21.15.16:0/1643367980 >> 172.21.15.16:0/1643367980 conn(0x7fdb340d15f0 msgr2=0x7fdb340d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:31:03.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.328+0000 7fdb3c507700 1 -- 172.21.15.16:0/1643367980 shutdown_connections 2023-10-16T12:31:03.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.328+0000 7fdb3c507700 1 -- 172.21.15.16:0/1643367980 wait complete. 2023-10-16T12:31:03.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.330+0000 7fdb3c507700 1 Processor -- start 2023-10-16T12:31:03.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.330+0000 7fdb3c507700 1 -- start start 2023-10-16T12:31:03.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.330+0000 7fdb3c507700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb20003a00 0x7fdb340c7a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:03.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.330+0000 7fdb3c507700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdb340c86c0 0x7fdb340c8b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:03.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.330+0000 7fdb3c507700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdb3406fd40 0x7fdb340701d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:03.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.330+0000 7fdb3c507700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb3406eca0 con 0x7fdb20003a00 2023-10-16T12:31:03.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.330+0000 7fdb3c507700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb3406ef80 con 0x7fdb340c86c0 2023-10-16T12:31:03.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.330+0000 7fdb3c507700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb3406f260 con 0x7fdb3406fd40 2023-10-16T12:31:03.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.330+0000 7fdb3a2a3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb20003a00 0x7fdb340c7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:03.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.330+0000 7fdb3a2a3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb20003a00 0x7fdb340c7a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46772/0 (socket says 172.21.15.16:46772) 2023-10-16T12:31:03.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.330+0000 7fdb3a2a3700 1 -- 172.21.15.16:0/552538092 learned_addr learned my addr 172.21.15.16:0/552538092 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:31:03.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.331+0000 7fdb3aaa4700 1 --2- 172.21.15.16:0/552538092 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdb3406fd40 0x7fdb340701d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:03.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.331+0000 7fdb3a2a3700 1 -- 172.21.15.16:0/552538092 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdb3406fd40 msgr2=0x7fdb340701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:03.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.331+0000 7fdb3a2a3700 1 --2- 172.21.15.16:0/552538092 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdb3406fd40 0x7fdb340701d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:03.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.331+0000 7fdb3a2a3700 1 -- 172.21.15.16:0/552538092 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdb340c86c0 msgr2=0x7fdb340c8b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:03.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.331+0000 7fdb3a2a3700 1 --2- 172.21.15.16:0/552538092 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdb340c86c0 0x7fdb340c8b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:03.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.331+0000 7fdb3a2a3700 1 -- 172.21.15.16:0/552538092 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb3406f540 con 0x7fdb20003a00 2023-10-16T12:31:03.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.331+0000 7fdb3a2a3700 1 --2- 172.21.15.16:0/552538092 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb20003a00 0x7fdb340c7a30 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7fdb240090b0 tx=0x7fdb240047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:03.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.331+0000 7fdb2affd700 1 -- 172.21.15.16:0/552538092 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb2400e6c0 con 0x7fdb20003a00 2023-10-16T12:31:03.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.331+0000 7fdb2affd700 1 -- 172.21.15.16:0/552538092 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb24003ab0 con 0x7fdb20003a00 2023-10-16T12:31:03.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.331+0000 7fdb3c507700 1 -- 172.21.15.16:0/552538092 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb3406f820 con 0x7fdb20003a00 2023-10-16T12:31:03.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.331+0000 7fdb2affd700 1 -- 172.21.15.16:0/552538092 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb24003e60 con 0x7fdb20003a00 2023-10-16T12:31:03.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.331+0000 7fdb3c507700 1 -- 172.21.15.16:0/552538092 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb340c7f70 con 0x7fdb20003a00 2023-10-16T12:31:03.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.332+0000 7fdb3c507700 1 -- 172.21.15.16:0/552538092 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb3406fa40 con 0x7fdb20003a00 2023-10-16T12:31:03.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.333+0000 7fdb2affd700 1 -- 172.21.15.16:0/552538092 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 85) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fdb24013470 con 0x7fdb20003a00 2023-10-16T12:31:03.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.333+0000 7fdb2affd700 1 --2- 172.21.15.16:0/552538092 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7fdb1806b020 0x7fdb1806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:03.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.333+0000 7fdb39aa2700 1 --2- 172.21.15.16:0/552538092 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7fdb1806b020 0x7fdb1806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:03.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.333+0000 7fdb2affd700 1 -- 172.21.15.16:0/552538092 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5233+0+0 (secure 0 0 0) 0x7fdb2408b0c0 con 0x7fdb20003a00 2023-10-16T12:31:03.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.333+0000 7fdb39aa2700 1 --2- 172.21.15.16:0/552538092 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7fdb1806b020 0x7fdb1806d4e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fdb3415f460 tx=0x7fdb30009670 comp rx=0 tx=0).ready entity=mgr.8973 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:03.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.338+0000 7fdb2affd700 1 -- 172.21.15.16:0/552538092 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdb240562c0 con 0x7fdb20003a00 2023-10-16T12:31:03.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.539+0000 7fdb3c507700 1 -- 172.21.15.16:0/552538092 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fdb340713d0 con 0x7fdb20003a00 2023-10-16T12:31:03.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.539+0000 7fdb2affd700 1 -- 172.21.15.16:0/552538092 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v152) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdb240598e0 con 0x7fdb20003a00 2023-10-16T12:31:03.541 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T12:31:03.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.541+0000 7fdb3c507700 1 -- 172.21.15.16:0/552538092 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7fdb1806b020 msgr2=0x7fdb1806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:03.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.541+0000 7fdb3c507700 1 --2- 172.21.15.16:0/552538092 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7fdb1806b020 0x7fdb1806d4e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fdb3415f460 tx=0x7fdb30009670 comp rx=0 tx=0).stop 2023-10-16T12:31:03.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.541+0000 7fdb3c507700 1 -- 172.21.15.16:0/552538092 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb20003a00 msgr2=0x7fdb340c7a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:03.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.541+0000 7fdb3c507700 1 --2- 172.21.15.16:0/552538092 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb20003a00 0x7fdb340c7a30 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7fdb240090b0 tx=0x7fdb240047c0 comp rx=0 tx=0).stop 2023-10-16T12:31:03.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.541+0000 7fdb3c507700 1 -- 172.21.15.16:0/552538092 shutdown_connections 2023-10-16T12:31:03.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.541+0000 7fdb3c507700 1 --2- 172.21.15.16:0/552538092 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7fdb1806b020 0x7fdb1806d4e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:03.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.541+0000 7fdb3c507700 1 --2- 172.21.15.16:0/552538092 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdb3406fd40 0x7fdb340701d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:03.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.541+0000 7fdb3c507700 1 --2- 172.21.15.16:0/552538092 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb20003a00 0x7fdb340c7a30 unknown :-1 s=CLOSED pgs=1722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:03.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.541+0000 7fdb3c507700 1 --2- 172.21.15.16:0/552538092 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdb340c86c0 0x7fdb340c8b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:03.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.541+0000 7fdb3c507700 1 -- 172.21.15.16:0/552538092 >> 172.21.15.16:0/552538092 conn(0x7fdb340d15f0 msgr2=0x7fdb340cc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:31:03.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.541+0000 7fdb3c507700 1 -- 172.21.15.16:0/552538092 shutdown_connections 2023-10-16T12:31:03.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.541+0000 7fdb3c507700 1 -- 172.21.15.16:0/552538092 wait complete. 2023-10-16T12:31:03.555 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-16T12:31:03.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.678+0000 7f4cf5187700 1 Processor -- start 2023-10-16T12:31:03.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.678+0000 7f4cf5187700 1 -- start start 2023-10-16T12:31:03.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.679+0000 7f4cf5187700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4cf0159c90 0x7f4cf015d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:03.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.679+0000 7f4cf5187700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4cf015a070 0x7f4cf015dee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:03.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.679+0000 7f4cf5187700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cf006b1f0 con 0x7f4cf00cf580 2023-10-16T12:31:03.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.679+0000 7f4cf5187700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cf006b4b0 con 0x7f4cf0159c90 2023-10-16T12:31:03.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.679+0000 7f4cf5187700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf006b790 con 0x7f4cf015a070 2023-10-16T12:31:03.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.679+0000 7f4ceed9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4cf0159c90 0x7f4cf015d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:03.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.679+0000 7f4cef59e700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f4cf00cf580 0x7f4cf00ae520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36058/0 (socket says 172.21.15.16:36058) 2023-10-16T12:31:03.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.679+0000 7f4cef59e700 1 -- 172.21.15.16:0/4024277206 learned_addr learned my addr 172.21.15.16:0/4024277206 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:31:03.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.679+0000 7f4ceed9d700 1 -- 172.21.15.16:0/4024277206 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4cf015a070 msgr2=0x7f4cf015dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:31:03.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.679+0000 7f4cee59c700 1 --2- 172.21.15.16:0/4024277206 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4cf015a070 0x7f4cf015dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:03.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.679+0000 7f4cedd9b700 1 -- 172.21.15.16:0/4024277206 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 718576459 0 0) 0x7f4cf006b1f0 con 0x7f4cf00cf580 2023-10-16T12:31:03.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.679+0000 7f4ceed9d700 1 --2- 172.21.15.16:0/4024277206 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4cf015a070 0x7f4cf015dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:03.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.679+0000 7f4ceed9d700 1 -- 172.21.15.16:0/4024277206 >> v1:172.21.15.16:6789/0 conn(0x7f4cf00cf580 legacy=0x7f4cf00ae520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:03.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.679+0000 7f4ceed9d700 1 -- 172.21.15.16:0/4024277206 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cf006ba70 con 0x7f4cf0159c90 2023-10-16T12:31:03.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.679+0000 7f4cee59c700 1 --2- 172.21.15.16:0/4024277206 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4cf015a070 0x7f4cf015dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:31:03.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.679+0000 7f4ceed9d700 1 --2- 172.21.15.16:0/4024277206 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cf0159c90 0x7f4cf015d9a0 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7f4ce4009870 tx=0x7f4ce4009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=927f75e2122d1b24 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:03.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.680+0000 7f4cedd9b700 1 -- 172.21.15.16:0/4024277206 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ce4011830 con 0x7f4cf0159c90 2023-10-16T12:31:03.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.680+0000 7f4cedd9b700 1 -- 172.21.15.16:0/4024277206 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ce4011990 con 0x7f4cf0159c90 2023-10-16T12:31:03.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.680+0000 7f4cedd9b700 1 -- 172.21.15.16:0/4024277206 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ce4011d20 con 0x7f4cf0159c90 2023-10-16T12:31:03.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.680+0000 7f4cf5187700 1 -- 172.21.15.16:0/4024277206 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cf0159c90 msgr2=0x7f4cf015d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:03.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.680+0000 7f4cf5187700 1 --2- 172.21.15.16:0/4024277206 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cf0159c90 0x7f4cf015d9a0 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7f4ce4009870 tx=0x7f4ce4009b80 comp rx=0 tx=0).stop 2023-10-16T12:31:03.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.680+0000 7f4cf5187700 1 -- 172.21.15.16:0/4024277206 shutdown_connections 2023-10-16T12:31:03.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.680+0000 7f4cf5187700 1 --2- 172.21.15.16:0/4024277206 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cf0159c90 0x7f4cf015d9a0 unknown :-1 s=CLOSED pgs=1724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:03.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.680+0000 7f4cf5187700 1 --2- 172.21.15.16:0/4024277206 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4cf015a070 0x7f4cf015dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:03.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.680+0000 7f4cf5187700 1 -- 172.21.15.16:0/4024277206 >> 172.21.15.16:0/4024277206 conn(0x7f4cf00cc5b0 msgr2=0x7f4cf00cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:31:03.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.680+0000 7f4cf5187700 1 -- 172.21.15.16:0/4024277206 shutdown_connections 2023-10-16T12:31:03.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.680+0000 7f4cf5187700 1 -- 172.21.15.16:0/4024277206 wait complete. 2023-10-16T12:31:03.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.682+0000 7f4cf5187700 1 Processor -- start 2023-10-16T12:31:03.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.682+0000 7f4cf5187700 1 -- start start 2023-10-16T12:31:03.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.682+0000 7f4cf5187700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4cf00cf580 0x7f4cf01571d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:03.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.683+0000 7f4cf5187700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cf0159c90 0x7f4cf00ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:03.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.683+0000 7f4cf5187700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4cf015a070 0x7f4cf00c7610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:03.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.683+0000 7f4cf5187700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf006edf0 con 0x7f4cf0159c90 2023-10-16T12:31:03.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.683+0000 7f4cf5187700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf00c9230 con 0x7f4cf015a070 2023-10-16T12:31:03.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.683+0000 7f4cf5187700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf00c9510 con 0x7f4cf00cf580 2023-10-16T12:31:03.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.683+0000 7f4cee59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cf0159c90 0x7f4cf00ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:03.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.683+0000 7f4cee59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cf0159c90 0x7f4cf00ac8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46788/0 (socket says 172.21.15.16:46788) 2023-10-16T12:31:03.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.683+0000 7f4cee59c700 1 -- 172.21.15.16:0/3762548014 learned_addr learned my addr 172.21.15.16:0/3762548014 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:31:03.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.683+0000 7f4cee59c700 1 -- 172.21.15.16:0/3762548014 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4cf00cf580 msgr2=0x7f4cf01571d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:03.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.683+0000 7f4ceed9d700 1 --2- 172.21.15.16:0/3762548014 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4cf00cf580 0x7f4cf01571d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:03.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.683+0000 7f4cef59e700 1 --2- 172.21.15.16:0/3762548014 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4cf015a070 0x7f4cf00c7610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:03.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.683+0000 7f4cee59c700 1 --2- 172.21.15.16:0/3762548014 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4cf00cf580 0x7f4cf01571d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:03.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.683+0000 7f4cee59c700 1 -- 172.21.15.16:0/3762548014 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4cf015a070 msgr2=0x7f4cf00c7610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:03.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.683+0000 7f4cee59c700 1 --2- 172.21.15.16:0/3762548014 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4cf015a070 0x7f4cf00c7610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:03.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.683+0000 7f4cee59c700 1 -- 172.21.15.16:0/3762548014 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ce40094a0 con 0x7f4cf0159c90 2023-10-16T12:31:03.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.683+0000 7f4cee59c700 1 --2- 172.21.15.16:0/3762548014 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cf0159c90 0x7f4cf00ac8d0 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7f4ce000c7f0 tx=0x7f4ce000cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:03.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.683+0000 7f4cd77fe700 1 -- 172.21.15.16:0/3762548014 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ce0011800 con 0x7f4cf0159c90 2023-10-16T12:31:03.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.683+0000 7f4cd77fe700 1 -- 172.21.15.16:0/3762548014 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ce0011960 con 0x7f4cf0159c90 2023-10-16T12:31:03.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.683+0000 7f4cd77fe700 1 -- 172.21.15.16:0/3762548014 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ce0011d10 con 0x7f4cf0159c90 2023-10-16T12:31:03.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.683+0000 7f4cf5187700 1 -- 172.21.15.16:0/3762548014 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cf00c97f0 con 0x7f4cf0159c90 2023-10-16T12:31:03.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.683+0000 7f4cf5187700 1 -- 172.21.15.16:0/3762548014 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cf00c8f00 con 0x7f4cf0159c90 2023-10-16T12:31:03.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.684+0000 7f4cf5187700 1 -- 172.21.15.16:0/3762548014 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cf00c9a10 con 0x7f4cf0159c90 2023-10-16T12:31:03.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.685+0000 7f4cd77fe700 1 -- 172.21.15.16:0/3762548014 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 85) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4ce0018360 con 0x7f4cf0159c90 2023-10-16T12:31:03.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.685+0000 7f4cd77fe700 1 --2- 172.21.15.16:0/3762548014 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f4cdc06b020 0x7f4cdc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:03.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.685+0000 7f4cd77fe700 1 -- 172.21.15.16:0/3762548014 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5233+0+0 (secure 0 0 0) 0x7f4ce0089590 con 0x7f4cf0159c90 2023-10-16T12:31:03.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.685+0000 7f4ceed9d700 1 --2- 172.21.15.16:0/3762548014 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f4cdc06b020 0x7f4cdc06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:03.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.686+0000 7f4ceed9d700 1 --2- 172.21.15.16:0/3762548014 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f4cdc06b020 0x7f4cdc06d4e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4ce40090b0 tx=0x7f4ce4008030 comp rx=0 tx=0).ready entity=mgr.8973 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:03.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.690+0000 7f4cd77fe700 1 -- 172.21.15.16:0/3762548014 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4ce0054680 con 0x7f4cf0159c90 2023-10-16T12:31:03.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.896+0000 7f4cf5187700 1 -- 172.21.15.16:0/3762548014 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f4cf004ebb0 con 0x7f4cf0159c90 2023-10-16T12:31:03.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.897+0000 7f4cd77fe700 1 -- 172.21.15.16:0/3762548014 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v152) v1 ==== 84+0+65 (secure 0 0 0) 0x7f4ce0057ca0 con 0x7f4cf0159c90 2023-10-16T12:31:03.899 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:31:03.899 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2023-10-16T12:31:03.899 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:31:03.900 INFO:teuthology.orchestra.run.smithi016.stdout: "poolnum": 1, 2023-10-16T12:31:03.900 INFO:teuthology.orchestra.run.smithi016.stdout: "poolname": ".mgr" 2023-10-16T12:31:03.900 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:31:03.900 INFO:teuthology.orchestra.run.smithi016.stdout:] 2023-10-16T12:31:03.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.899+0000 7f4cf5187700 1 -- 172.21.15.16:0/3762548014 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f4cdc06b020 msgr2=0x7f4cdc06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:03.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.899+0000 7f4cf5187700 1 --2- 172.21.15.16:0/3762548014 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f4cdc06b020 0x7f4cdc06d4e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4ce40090b0 tx=0x7f4ce4008030 comp rx=0 tx=0).stop 2023-10-16T12:31:03.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.899+0000 7f4cf5187700 1 -- 172.21.15.16:0/3762548014 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cf0159c90 msgr2=0x7f4cf00ac8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:03.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.899+0000 7f4cf5187700 1 --2- 172.21.15.16:0/3762548014 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cf0159c90 0x7f4cf00ac8d0 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7f4ce000c7f0 tx=0x7f4ce000cbb0 comp rx=0 tx=0).stop 2023-10-16T12:31:03.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.899+0000 7f4cf5187700 1 -- 172.21.15.16:0/3762548014 shutdown_connections 2023-10-16T12:31:03.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.899+0000 7f4cf5187700 1 --2- 172.21.15.16:0/3762548014 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f4cdc06b020 0x7f4cdc06d4e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:03.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.899+0000 7f4cf5187700 1 --2- 172.21.15.16:0/3762548014 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4cf015a070 0x7f4cf00c7610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:03.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.899+0000 7f4cf5187700 1 --2- 172.21.15.16:0/3762548014 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4cf00cf580 0x7f4cf01571d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:03.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.899+0000 7f4cf5187700 1 --2- 172.21.15.16:0/3762548014 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cf0159c90 0x7f4cf00ac8d0 unknown :-1 s=CLOSED pgs=1725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:03.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.899+0000 7f4cf5187700 1 -- 172.21.15.16:0/3762548014 >> 172.21.15.16:0/3762548014 conn(0x7f4cf00cc5b0 msgr2=0x7f4cf0158d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:31:03.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.899+0000 7f4cf5187700 1 -- 172.21.15.16:0/3762548014 shutdown_connections 2023-10-16T12:31:03.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:03.899+0000 7f4cf5187700 1 -- 172.21.15.16:0/3762548014 wait complete. 2023-10-16T12:31:03.913 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-16T12:31:03.914 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-16T12:31:04.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.041+0000 7fdfcab6d700 1 Processor -- start 2023-10-16T12:31:04.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.041+0000 7fdfcab6d700 1 -- start start 2023-10-16T12:31:04.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.041+0000 7fdfcab6d700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fdfc4160e60 0x7fdfc415e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:04.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.042+0000 7fdfcab6d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdfc406d980 0x7fdfc415efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:04.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.042+0000 7fdfcab6d700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfc40cdeb0 con 0x7fdfc406a7e0 2023-10-16T12:31:04.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.042+0000 7fdfcab6d700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfc41613e0 con 0x7fdfc4160e60 2023-10-16T12:31:04.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.042+0000 7fdfcab6d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfc41582d0 con 0x7fdfc406d980 2023-10-16T12:31:04.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.042+0000 7fdfc910a700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fdfc406a7e0 0x7fdfc4161870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36072/0 (socket says 172.21.15.16:36072) 2023-10-16T12:31:04.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.042+0000 7fdfc910a700 1 -- 172.21.15.16:0/3698509297 learned_addr learned my addr 172.21.15.16:0/3698509297 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:31:04.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.042+0000 7fdfc37fe700 1 -- 172.21.15.16:0/3698509297 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 571918158 0 0) 0x7fdfc40cdeb0 con 0x7fdfc406a7e0 2023-10-16T12:31:04.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.042+0000 7fdfc37fe700 1 -- 172.21.15.16:0/3698509297 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfb40035b0 con 0x7fdfc406a7e0 2023-10-16T12:31:04.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.042+0000 7fdfc8909700 1 --2- 172.21.15.16:0/3698509297 >> v2:172.21.15.49:3300/0 conn(0x7fdfc4160e60 0x7fdfc415e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:04.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.042+0000 7fdfc3fff700 1 --2- 172.21.15.16:0/3698509297 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdfc406d980 0x7fdfc415efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:04.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.042+0000 7fdfc37fe700 1 -- 172.21.15.16:0/3698509297 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2786778949 0 0) 0x7fdfb40035b0 con 0x7fdfc406a7e0 2023-10-16T12:31:04.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.042+0000 7fdfc37fe700 1 -- 172.21.15.16:0/3698509297 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdfc406d980 msgr2=0x7fdfc415efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:31:04.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.042+0000 7fdfc37fe700 1 --2- 172.21.15.16:0/3698509297 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdfc406d980 0x7fdfc415efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:04.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.042+0000 7fdfc37fe700 1 -- 172.21.15.16:0/3698509297 >> v2:172.21.15.49:3300/0 conn(0x7fdfc4160e60 msgr2=0x7fdfc415e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:31:04.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.042+0000 7fdfc37fe700 1 --2- 172.21.15.16:0/3698509297 >> v2:172.21.15.49:3300/0 conn(0x7fdfc4160e60 0x7fdfc415e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:04.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.042+0000 7fdfc37fe700 1 -- 172.21.15.16:0/3698509297 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfc415edb0 con 0x7fdfc406a7e0 2023-10-16T12:31:04.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.042+0000 7fdfc37fe700 1 -- 172.21.15.16:0/3698509297 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7fdfbc0019f0 con 0x7fdfc406a7e0 2023-10-16T12:31:04.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.042+0000 7fdfc37fe700 1 -- 172.21.15.16:0/3698509297 >> v1:172.21.15.16:6789/0 conn(0x7fdfc406a7e0 legacy=0x7fdfc4161870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:04.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.042+0000 7fdfc37fe700 1 --2- 172.21.15.16:0/3698509297 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdfc406d980 0x7fdfb4003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:04.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.042+0000 7fdfc37fe700 1 --2- 172.21.15.16:0/3698509297 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdfb4003f80 0x7fdfb4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:04.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.042+0000 7fdfc37fe700 1 --2- 172.21.15.16:0/3698509297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfb4004d90 0x7fdfb4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:04.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.042+0000 7fdfc37fe700 1 -- 172.21.15.16:0/3698509297 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfc41613e0 con 0x7fdfb4004d90 2023-10-16T12:31:04.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.042+0000 7fdfc37fe700 1 -- 172.21.15.16:0/3698509297 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfc41582d0 con 0x7fdfb4003f80 2023-10-16T12:31:04.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.042+0000 7fdfc37fe700 1 -- 172.21.15.16:0/3698509297 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb400ba10 con 0x7fdfc406d980 2023-10-16T12:31:04.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.043+0000 7fdfc910a700 1 --2- 172.21.15.16:0/3698509297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfb4004d90 0x7fdfb4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:04.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.043+0000 7fdfc8909700 1 --2- 172.21.15.16:0/3698509297 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdfc406d980 0x7fdfb4003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:04.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.043+0000 7fdfc3fff700 1 --2- 172.21.15.16:0/3698509297 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdfb4003f80 0x7fdfb4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:04.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.043+0000 7fdfc910a700 1 -- 172.21.15.16:0/3698509297 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdfc406d980 msgr2=0x7fdfb4003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:31:04.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.043+0000 7fdfc910a700 1 --2- 172.21.15.16:0/3698509297 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdfc406d980 0x7fdfb4003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:04.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.043+0000 7fdfc910a700 1 -- 172.21.15.16:0/3698509297 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdfb4003f80 msgr2=0x7fdfb4004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:31:04.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.043+0000 7fdfc910a700 1 --2- 172.21.15.16:0/3698509297 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdfb4003f80 0x7fdfb4004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:04.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.043+0000 7fdfc910a700 1 -- 172.21.15.16:0/3698509297 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfb40035b0 con 0x7fdfb4004d90 2023-10-16T12:31:04.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.043+0000 7fdfc910a700 1 --2- 172.21.15.16:0/3698509297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfb4004d90 0x7fdfb4005210 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7fdfbc00a840 tx=0x7fdfb4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1d9a44dc80132c3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:04.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.043+0000 7fdfc37fe700 1 -- 172.21.15.16:0/3698509297 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfbc012600 con 0x7fdfb4004d90 2023-10-16T12:31:04.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.043+0000 7fdfc37fe700 1 -- 172.21.15.16:0/3698509297 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfbc012760 con 0x7fdfb4004d90 2023-10-16T12:31:04.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.043+0000 7fdfc37fe700 1 -- 172.21.15.16:0/3698509297 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfbc012b10 con 0x7fdfb4004d90 2023-10-16T12:31:04.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.044+0000 7fdfcab6d700 1 -- 172.21.15.16:0/3698509297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfb4004d90 msgr2=0x7fdfb4005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:04.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.044+0000 7fdfcab6d700 1 --2- 172.21.15.16:0/3698509297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfb4004d90 0x7fdfb4005210 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7fdfbc00a840 tx=0x7fdfb4005880 comp rx=0 tx=0).stop 2023-10-16T12:31:04.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.044+0000 7fdfc910a700 1 -- 172.21.15.16:0/3698509297 reap_dead start 2023-10-16T12:31:04.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.044+0000 7fdfcab6d700 1 -- 172.21.15.16:0/3698509297 shutdown_connections 2023-10-16T12:31:04.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.044+0000 7fdfcab6d700 1 -- 172.21.15.16:0/3698509297 >> 172.21.15.16:0/3698509297 conn(0x7fdfc406f430 msgr2=0x7fdfc406f810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:31:04.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.044+0000 7fdfcab6d700 1 -- 172.21.15.16:0/3698509297 shutdown_connections 2023-10-16T12:31:04.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.044+0000 7fdfcab6d700 1 -- 172.21.15.16:0/3698509297 wait complete. 2023-10-16T12:31:04.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.045+0000 7fdfcab6d700 1 Processor -- start 2023-10-16T12:31:04.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.045+0000 7fdfcab6d700 1 -- start start 2023-10-16T12:31:04.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.046+0000 7fdfcab6d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdfc4150330 0x7fdfc4156e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:04.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.046+0000 7fdfcab6d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfc41f65a0 0x7fdfc41573b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:04.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.046+0000 7fdfcab6d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdfc4150710 0x7fdfc40b4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:04.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.046+0000 7fdfcab6d700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfc40ad5f0 con 0x7fdfc41f65a0 2023-10-16T12:31:04.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.046+0000 7fdfcab6d700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfc40ac970 con 0x7fdfc4150710 2023-10-16T12:31:04.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.046+0000 7fdfcab6d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfc40acc50 con 0x7fdfc4150330 2023-10-16T12:31:04.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.046+0000 7fdfc3fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfc41f65a0 0x7fdfc41573b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:04.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.046+0000 7fdfc3fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfc41f65a0 0x7fdfc41573b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46808/0 (socket says 172.21.15.16:46808) 2023-10-16T12:31:04.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.046+0000 7fdfc3fff700 1 -- 172.21.15.16:0/1576671482 learned_addr learned my addr 172.21.15.16:0/1576671482 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:31:04.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.046+0000 7fdfc910a700 1 --2- 172.21.15.16:0/1576671482 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdfc4150710 0x7fdfc40b4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:04.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.046+0000 7fdfc8909700 1 --2- 172.21.15.16:0/1576671482 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdfc4150330 0x7fdfc4156e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:04.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.046+0000 7fdfc3fff700 1 -- 172.21.15.16:0/1576671482 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdfc4150330 msgr2=0x7fdfc4156e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:04.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.046+0000 7fdfc3fff700 1 --2- 172.21.15.16:0/1576671482 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdfc4150330 0x7fdfc4156e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:04.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.046+0000 7fdfc3fff700 1 -- 172.21.15.16:0/1576671482 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdfc4150710 msgr2=0x7fdfc40b4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:04.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.046+0000 7fdfc3fff700 1 --2- 172.21.15.16:0/1576671482 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdfc4150710 0x7fdfc40b4e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:04.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.046+0000 7fdfc3fff700 1 -- 172.21.15.16:0/1576671482 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfc40acf30 con 0x7fdfc41f65a0 2023-10-16T12:31:04.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.046+0000 7fdfc3fff700 1 --2- 172.21.15.16:0/1576671482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfc41f65a0 0x7fdfc41573b0 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7fdfb00119a0 tx=0x7fdfb0011cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:04.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.047+0000 7fdfc17fa700 1 -- 172.21.15.16:0/1576671482 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfb000f730 con 0x7fdfc41f65a0 2023-10-16T12:31:04.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.047+0000 7fdfc17fa700 1 -- 172.21.15.16:0/1576671482 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfb000f890 con 0x7fdfc41f65a0 2023-10-16T12:31:04.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.047+0000 7fdfcab6d700 1 -- 172.21.15.16:0/1576671482 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfc40ad210 con 0x7fdfc41f65a0 2023-10-16T12:31:04.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.047+0000 7fdfc17fa700 1 -- 172.21.15.16:0/1576671482 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfb000fc20 con 0x7fdfc41f65a0 2023-10-16T12:31:04.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.047+0000 7fdfcab6d700 1 -- 172.21.15.16:0/1576671482 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfc40b55b0 con 0x7fdfc41f65a0 2023-10-16T12:31:04.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.048+0000 7fdfcab6d700 1 -- 172.21.15.16:0/1576671482 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfc404ec20 con 0x7fdfc41f65a0 2023-10-16T12:31:04.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.048+0000 7fdfc17fa700 1 -- 172.21.15.16:0/1576671482 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 85) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fdfb000fd80 con 0x7fdfc41f65a0 2023-10-16T12:31:04.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.048+0000 7fdfc17fa700 1 --2- 172.21.15.16:0/1576671482 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7fdfa806b020 0x7fdfa806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:04.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.049+0000 7fdfc17fa700 1 -- 172.21.15.16:0/1576671482 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5233+0+0 (secure 0 0 0) 0x7fdfb0089ab0 con 0x7fdfc41f65a0 2023-10-16T12:31:04.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.049+0000 7fdfc8909700 1 --2- 172.21.15.16:0/1576671482 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7fdfa806b020 0x7fdfa806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:04.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.049+0000 7fdfc8909700 1 --2- 172.21.15.16:0/1576671482 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7fdfa806b020 0x7fdfa806d4e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fdfb8009ce0 tx=0x7fdfb8009210 comp rx=0 tx=0).ready entity=mgr.8973 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:04.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.053+0000 7fdfc17fa700 1 -- 172.21.15.16:0/1576671482 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdfb0054cb0 con 0x7fdfc41f65a0 2023-10-16T12:31:04.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.258+0000 7fdfcab6d700 1 -- 172.21.15.16:0/1576671482 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fdfc40b6260 con 0x7fdfc41f65a0 2023-10-16T12:31:04.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.404+0000 7fdfc17fa700 1 -- 172.21.15.16:0/1576671482 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v153) v1 ==== 138+0+0 (secure 0 0 0) 0x7fdfb00582d0 con 0x7fdfc41f65a0 2023-10-16T12:31:04.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.406+0000 7fdfcab6d700 1 -- 172.21.15.16:0/1576671482 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7fdfa806b020 msgr2=0x7fdfa806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:04.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.406+0000 7fdfcab6d700 1 --2- 172.21.15.16:0/1576671482 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7fdfa806b020 0x7fdfa806d4e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fdfb8009ce0 tx=0x7fdfb8009210 comp rx=0 tx=0).stop 2023-10-16T12:31:04.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.406+0000 7fdfcab6d700 1 -- 172.21.15.16:0/1576671482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfc41f65a0 msgr2=0x7fdfc41573b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:04.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.406+0000 7fdfcab6d700 1 --2- 172.21.15.16:0/1576671482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfc41f65a0 0x7fdfc41573b0 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7fdfb00119a0 tx=0x7fdfb0011cb0 comp rx=0 tx=0).stop 2023-10-16T12:31:04.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.406+0000 7fdfcab6d700 1 -- 172.21.15.16:0/1576671482 shutdown_connections 2023-10-16T12:31:04.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.406+0000 7fdfcab6d700 1 --2- 172.21.15.16:0/1576671482 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7fdfa806b020 0x7fdfa806d4e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:04.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.406+0000 7fdfcab6d700 1 --2- 172.21.15.16:0/1576671482 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdfc4150710 0x7fdfc40b4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:04.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.406+0000 7fdfcab6d700 1 --2- 172.21.15.16:0/1576671482 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdfc4150330 0x7fdfc4156e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:04.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.406+0000 7fdfcab6d700 1 --2- 172.21.15.16:0/1576671482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfc41f65a0 0x7fdfc41573b0 unknown :-1 s=CLOSED pgs=1728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:04.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.406+0000 7fdfcab6d700 1 -- 172.21.15.16:0/1576671482 >> 172.21.15.16:0/1576671482 conn(0x7fdfc406f430 msgr2=0x7fdfc415e0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:31:04.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.406+0000 7fdfcab6d700 1 -- 172.21.15.16:0/1576671482 shutdown_connections 2023-10-16T12:31:04.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.407+0000 7fdfcab6d700 1 -- 172.21.15.16:0/1576671482 wait complete. 2023-10-16T12:31:04.411 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' created 2023-10-16T12:31:04.424 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-16T12:31:04.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.555+0000 7f6e50077700 1 Processor -- start 2023-10-16T12:31:04.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.555+0000 7f6e50077700 1 -- start start 2023-10-16T12:31:04.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.556+0000 7f6e50077700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6e4815da30 0x7f6e480cdd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:04.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.556+0000 7f6e50077700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f6e4815de10 0x7f6e480ce290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:04.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.556+0000 7f6e50077700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e480b2160 con 0x7f6e480b2690 2023-10-16T12:31:04.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.556+0000 7f6e50077700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e480d11a0 con 0x7f6e4815da30 2023-10-16T12:31:04.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.556+0000 7f6e50077700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e480d0d10 con 0x7f6e4815de10 2023-10-16T12:31:04.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.556+0000 7f6e4de13700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6e4815da30 0x7f6e480cdd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:04.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.556+0000 7f6e4e614700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f6e480b2690 0x7f6e4806bc20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36092/0 (socket says 172.21.15.16:36092) 2023-10-16T12:31:04.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.556+0000 7f6e4e614700 1 -- 172.21.15.16:0/618380895 learned_addr learned my addr 172.21.15.16:0/618380895 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:31:04.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.556+0000 7f6e4d612700 1 --2- 172.21.15.16:0/618380895 >> v2:172.21.15.49:3300/0 conn(0x7f6e4815de10 0x7f6e480ce290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:04.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.556+0000 7f6e4de13700 1 -- 172.21.15.16:0/618380895 >> v2:172.21.15.49:3300/0 conn(0x7f6e4815de10 msgr2=0x7f6e480ce290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:31:04.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.556+0000 7f6e4de13700 1 --2- 172.21.15.16:0/618380895 >> v2:172.21.15.49:3300/0 conn(0x7f6e4815de10 0x7f6e480ce290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:04.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.556+0000 7f6e4de13700 1 -- 172.21.15.16:0/618380895 >> v1:172.21.15.16:6789/0 conn(0x7f6e480b2690 legacy=0x7f6e4806bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:04.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.557+0000 7f6e4de13700 1 -- 172.21.15.16:0/618380895 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e480b23b0 con 0x7f6e4815da30 2023-10-16T12:31:04.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.557+0000 7f6e4ce11700 1 -- 172.21.15.16:0/618380895 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2428398410 0 0) 0x7f6e480b2160 con 0x7f6e480b2690 2023-10-16T12:31:04.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.557+0000 7f6e4de13700 1 --2- 172.21.15.16:0/618380895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e4815da30 0x7f6e480cdd50 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7f6e48072370 tx=0x7f6e38009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c806469687f2c8ab server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:04.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.557+0000 7f6e4ce11700 1 -- 172.21.15.16:0/618380895 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e38011830 con 0x7f6e4815da30 2023-10-16T12:31:04.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.557+0000 7f6e4ce11700 1 -- 172.21.15.16:0/618380895 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e38011990 con 0x7f6e4815da30 2023-10-16T12:31:04.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.557+0000 7f6e4ce11700 1 -- 172.21.15.16:0/618380895 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e38011d20 con 0x7f6e4815da30 2023-10-16T12:31:04.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.557+0000 7f6e50077700 1 -- 172.21.15.16:0/618380895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e4815da30 msgr2=0x7f6e480cdd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:04.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.557+0000 7f6e50077700 1 --2- 172.21.15.16:0/618380895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e4815da30 0x7f6e480cdd50 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7f6e48072370 tx=0x7f6e38009b20 comp rx=0 tx=0).stop 2023-10-16T12:31:04.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.557+0000 7f6e50077700 1 -- 172.21.15.16:0/618380895 shutdown_connections 2023-10-16T12:31:04.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.557+0000 7f6e50077700 1 --2- 172.21.15.16:0/618380895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e4815da30 0x7f6e480cdd50 unknown :-1 s=CLOSED pgs=1730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:04.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.557+0000 7f6e50077700 1 --2- 172.21.15.16:0/618380895 >> v2:172.21.15.49:3300/0 conn(0x7f6e4815de10 0x7f6e480ce290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:04.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.557+0000 7f6e50077700 1 -- 172.21.15.16:0/618380895 >> 172.21.15.16:0/618380895 conn(0x7f6e480d14d0 msgr2=0x7f6e480d18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:31:04.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.558+0000 7f6e50077700 1 -- 172.21.15.16:0/618380895 shutdown_connections 2023-10-16T12:31:04.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.558+0000 7f6e50077700 1 -- 172.21.15.16:0/618380895 wait complete. 2023-10-16T12:31:04.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.559+0000 7f6e50077700 1 Processor -- start 2023-10-16T12:31:04.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.559+0000 7f6e50077700 1 -- start start 2023-10-16T12:31:04.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.559+0000 7f6e50077700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e480b2690 0x7f6e48151640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:04.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.559+0000 7f6e50077700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6e4815da30 0x7f6e480c6e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:04.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.559+0000 7f6e50077700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6e4815de10 0x7f6e481522e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:04.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.559+0000 7f6e50077700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e480c5ed0 con 0x7f6e480b2690 2023-10-16T12:31:04.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.559+0000 7f6e50077700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e48152160 con 0x7f6e4815da30 2023-10-16T12:31:04.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.559+0000 7f6e50077700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e480c5000 con 0x7f6e4815de10 2023-10-16T12:31:04.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.560+0000 7f6e4de13700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e480b2690 0x7f6e48151640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:04.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.560+0000 7f6e4de13700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e480b2690 0x7f6e48151640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46820/0 (socket says 172.21.15.16:46820) 2023-10-16T12:31:04.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.560+0000 7f6e4de13700 1 -- 172.21.15.16:0/2445794996 learned_addr learned my addr 172.21.15.16:0/2445794996 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:31:04.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.560+0000 7f6e4d612700 1 --2- 172.21.15.16:0/2445794996 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6e4815da30 0x7f6e480c6e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:04.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.560+0000 7f6e4de13700 1 -- 172.21.15.16:0/2445794996 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6e4815de10 msgr2=0x7f6e481522e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:04.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.560+0000 7f6e4e614700 1 --2- 172.21.15.16:0/2445794996 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6e4815de10 0x7f6e481522e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:04.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.560+0000 7f6e4de13700 1 --2- 172.21.15.16:0/2445794996 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6e4815de10 0x7f6e481522e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:04.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.560+0000 7f6e4de13700 1 -- 172.21.15.16:0/2445794996 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6e4815da30 msgr2=0x7f6e480c6e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:04.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.560+0000 7f6e4de13700 1 --2- 172.21.15.16:0/2445794996 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6e4815da30 0x7f6e480c6e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:04.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.560+0000 7f6e4de13700 1 -- 172.21.15.16:0/2445794996 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e380094a0 con 0x7f6e480b2690 2023-10-16T12:31:04.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.560+0000 7f6e4d612700 1 --2- 172.21.15.16:0/2445794996 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6e4815da30 0x7f6e480c6e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:31:04.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.560+0000 7f6e4de13700 1 --2- 172.21.15.16:0/2445794996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e480b2690 0x7f6e48151640 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7f6e380090b0 tx=0x7f6e38004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:04.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.560+0000 7f6e3e7fc700 1 -- 172.21.15.16:0/2445794996 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e380038a0 con 0x7f6e480b2690 2023-10-16T12:31:04.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.560+0000 7f6e3e7fc700 1 -- 172.21.15.16:0/2445794996 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e3801e070 con 0x7f6e480b2690 2023-10-16T12:31:04.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.560+0000 7f6e50077700 1 -- 172.21.15.16:0/2445794996 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e48151b80 con 0x7f6e480b2690 2023-10-16T12:31:04.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.560+0000 7f6e3e7fc700 1 -- 172.21.15.16:0/2445794996 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e38003c20 con 0x7f6e480b2690 2023-10-16T12:31:04.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.560+0000 7f6e50077700 1 -- 172.21.15.16:0/2445794996 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e480c5530 con 0x7f6e480b2690 2023-10-16T12:31:04.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.561+0000 7f6e50077700 1 -- 172.21.15.16:0/2445794996 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e4804ebb0 con 0x7f6e480b2690 2023-10-16T12:31:04.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.562+0000 7f6e3e7fc700 1 -- 172.21.15.16:0/2445794996 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 85) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6e3801a420 con 0x7f6e480b2690 2023-10-16T12:31:04.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.562+0000 7f6e3e7fc700 1 --2- 172.21.15.16:0/2445794996 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f6e2c06b020 0x7f6e2c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:04.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.562+0000 7f6e3e7fc700 1 -- 172.21.15.16:0/2445794996 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 5603+0+0 (secure 0 0 0) 0x7f6e3808a700 con 0x7f6e480b2690 2023-10-16T12:31:04.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.563+0000 7f6e4d612700 1 --2- 172.21.15.16:0/2445794996 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f6e2c06b020 0x7f6e2c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:04.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.563+0000 7f6e4d612700 1 --2- 172.21.15.16:0/2445794996 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f6e2c06b020 0x7f6e2c06d4e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6e44009b60 tx=0x7f6e44008040 comp rx=0 tx=0).ready entity=mgr.8973 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:04.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.567+0000 7f6e3e7fc700 1 -- 172.21.15.16:0/2445794996 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6e38055790 con 0x7f6e480b2690 2023-10-16T12:31:04.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:04.771+0000 7f6e50077700 1 -- 172.21.15.16:0/2445794996 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f6e4806f770 con 0x7f6e480b2690 2023-10-16T12:31:05.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.410+0000 7f6e3e7fc700 1 -- 172.21.15.16:0/2445794996 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v154) v1 ==== 170+0+0 (secure 0 0 0) 0x7f6e380200d0 con 0x7f6e480b2690 2023-10-16T12:31:05.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.414+0000 7f6e50077700 1 -- 172.21.15.16:0/2445794996 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f6e2c06b020 msgr2=0x7f6e2c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:05.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.414+0000 7f6e50077700 1 --2- 172.21.15.16:0/2445794996 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f6e2c06b020 0x7f6e2c06d4e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6e44009b60 tx=0x7f6e44008040 comp rx=0 tx=0).stop 2023-10-16T12:31:05.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.414+0000 7f6e50077700 1 -- 172.21.15.16:0/2445794996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e480b2690 msgr2=0x7f6e48151640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:05.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.414+0000 7f6e50077700 1 --2- 172.21.15.16:0/2445794996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e480b2690 0x7f6e48151640 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7f6e380090b0 tx=0x7f6e38004750 comp rx=0 tx=0).stop 2023-10-16T12:31:05.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.414+0000 7f6e50077700 1 -- 172.21.15.16:0/2445794996 shutdown_connections 2023-10-16T12:31:05.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.414+0000 7f6e50077700 1 --2- 172.21.15.16:0/2445794996 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f6e2c06b020 0x7f6e2c06d4e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:05.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.414+0000 7f6e50077700 1 --2- 172.21.15.16:0/2445794996 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6e4815de10 0x7f6e481522e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:05.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.414+0000 7f6e50077700 1 --2- 172.21.15.16:0/2445794996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e480b2690 0x7f6e48151640 unknown :-1 s=CLOSED pgs=1731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:05.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.414+0000 7f6e50077700 1 --2- 172.21.15.16:0/2445794996 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6e4815da30 0x7f6e480c6e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:05.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.414+0000 7f6e50077700 1 -- 172.21.15.16:0/2445794996 >> 172.21.15.16:0/2445794996 conn(0x7f6e480d14d0 msgr2=0x7f6e480cdb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:31:05.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.415+0000 7f6e50077700 1 -- 172.21.15.16:0/2445794996 shutdown_connections 2023-10-16T12:31:05.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.416+0000 7f6e50077700 1 -- 172.21.15.16:0/2445794996 wait complete. 2023-10-16T12:31:05.420 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' created 2023-10-16T12:31:05.444 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-16T12:31:05.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.576+0000 7f0dce365700 1 Processor -- start 2023-10-16T12:31:05.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.576+0000 7f0dce365700 1 -- start start 2023-10-16T12:31:05.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.576+0000 7f0dce365700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f0dc815b2c0 0x7f0dc8161340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:05.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.576+0000 7f0dce365700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0dc806e570 0x7f0dc815e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:05.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.576+0000 7f0dce365700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dc8161880 con 0x7f0dc806a7e0 2023-10-16T12:31:05.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.576+0000 7f0dce365700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0dc815b6a0 con 0x7f0dc806e570 2023-10-16T12:31:05.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.576+0000 7f0dce365700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0dc8161b20 con 0x7f0dc815b2c0 2023-10-16T12:31:05.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.577+0000 7f0dc77fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0dc806e570 0x7f0dc815e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:05.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.577+0000 7f0dcc902700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0dc806a7e0 0x7f0dc8071a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36106/0 (socket says 172.21.15.16:36106) 2023-10-16T12:31:05.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.577+0000 7f0dcc902700 1 -- 172.21.15.16:0/3486648708 learned_addr learned my addr 172.21.15.16:0/3486648708 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:31:05.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.577+0000 7f0dc7fff700 1 --2- 172.21.15.16:0/3486648708 >> v2:172.21.15.49:3300/0 conn(0x7f0dc815b2c0 0x7f0dc8161340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:05.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.577+0000 7f0dc77fe700 1 -- 172.21.15.16:0/3486648708 >> v2:172.21.15.49:3300/0 conn(0x7f0dc815b2c0 msgr2=0x7f0dc8161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:31:05.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.577+0000 7f0dc6ffd700 1 -- 172.21.15.16:0/3486648708 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2018501851 0 0) 0x7f0dc8161880 con 0x7f0dc806a7e0 2023-10-16T12:31:05.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.577+0000 7f0dc77fe700 1 --2- 172.21.15.16:0/3486648708 >> v2:172.21.15.49:3300/0 conn(0x7f0dc815b2c0 0x7f0dc8161340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:05.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.577+0000 7f0dc77fe700 1 -- 172.21.15.16:0/3486648708 >> v1:172.21.15.16:6789/0 conn(0x7f0dc806a7e0 legacy=0x7f0dc8071a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:05.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.577+0000 7f0dc77fe700 1 -- 172.21.15.16:0/3486648708 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dc8161e00 con 0x7f0dc806e570 2023-10-16T12:31:05.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.577+0000 7f0dc7fff700 1 --2- 172.21.15.16:0/3486648708 >> v2:172.21.15.49:3300/0 conn(0x7f0dc815b2c0 0x7f0dc8161340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:31:05.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.577+0000 7f0dc77fe700 1 --2- 172.21.15.16:0/3486648708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0dc806e570 0x7f0dc815e5a0 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7f0dc806b4a0 tx=0x7f0db8009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63e5bedff1177b77 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:05.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.577+0000 7f0dc6ffd700 1 -- 172.21.15.16:0/3486648708 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db8011830 con 0x7f0dc806e570 2023-10-16T12:31:05.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.577+0000 7f0dc6ffd700 1 -- 172.21.15.16:0/3486648708 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0db8011990 con 0x7f0dc806e570 2023-10-16T12:31:05.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.577+0000 7f0dc6ffd700 1 -- 172.21.15.16:0/3486648708 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db8011d20 con 0x7f0dc806e570 2023-10-16T12:31:05.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.578+0000 7f0dce365700 1 -- 172.21.15.16:0/3486648708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0dc806e570 msgr2=0x7f0dc815e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:05.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.578+0000 7f0dce365700 1 --2- 172.21.15.16:0/3486648708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0dc806e570 0x7f0dc815e5a0 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7f0dc806b4a0 tx=0x7f0db8009b20 comp rx=0 tx=0).stop 2023-10-16T12:31:05.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.578+0000 7f0dce365700 1 -- 172.21.15.16:0/3486648708 shutdown_connections 2023-10-16T12:31:05.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.578+0000 7f0dce365700 1 --2- 172.21.15.16:0/3486648708 >> v2:172.21.15.49:3300/0 conn(0x7f0dc815b2c0 0x7f0dc8161340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:05.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.578+0000 7f0dce365700 1 --2- 172.21.15.16:0/3486648708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0dc806e570 0x7f0dc815e5a0 unknown :-1 s=CLOSED pgs=1733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:05.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.578+0000 7f0dce365700 1 -- 172.21.15.16:0/3486648708 >> 172.21.15.16:0/3486648708 conn(0x7f0dc80b1f40 msgr2=0x7f0dc80b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:31:05.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.578+0000 7f0dce365700 1 -- 172.21.15.16:0/3486648708 shutdown_connections 2023-10-16T12:31:05.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.578+0000 7f0dce365700 1 -- 172.21.15.16:0/3486648708 wait complete. 2023-10-16T12:31:05.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.579+0000 7f0dce365700 1 Processor -- start 2023-10-16T12:31:05.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.579+0000 7f0dce365700 1 -- start start 2023-10-16T12:31:05.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.580+0000 7f0dce365700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0dc806a7e0 0x7f0dc80ca280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:05.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.580+0000 7f0dce365700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0dc806e570 0x7f0dc81f2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:05.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.580+0000 7f0dce365700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0dc815b2c0 0x7f0dc80c9180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:05.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.580+0000 7f0dce365700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc8156460 con 0x7f0dc806e570 2023-10-16T12:31:05.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.580+0000 7f0dce365700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc81547a0 con 0x7f0dc806a7e0 2023-10-16T12:31:05.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.580+0000 7f0dce365700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc81f3c30 con 0x7f0dc815b2c0 2023-10-16T12:31:05.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.580+0000 7f0dc77fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0dc806e570 0x7f0dc81f2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:05.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.580+0000 7f0dc77fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0dc806e570 0x7f0dc81f2cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46836/0 (socket says 172.21.15.16:46836) 2023-10-16T12:31:05.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.580+0000 7f0dc77fe700 1 -- 172.21.15.16:0/1789247096 learned_addr learned my addr 172.21.15.16:0/1789247096 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:31:05.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.580+0000 7f0dc7fff700 1 --2- 172.21.15.16:0/1789247096 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0dc806a7e0 0x7f0dc80ca280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:05.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.581+0000 7f0dcc902700 1 --2- 172.21.15.16:0/1789247096 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0dc815b2c0 0x7f0dc80c9180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:05.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.581+0000 7f0dc77fe700 1 -- 172.21.15.16:0/1789247096 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0dc815b2c0 msgr2=0x7f0dc80c9180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:05.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.581+0000 7f0dc77fe700 1 --2- 172.21.15.16:0/1789247096 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0dc815b2c0 0x7f0dc80c9180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:05.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.581+0000 7f0dc77fe700 1 -- 172.21.15.16:0/1789247096 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0dc806a7e0 msgr2=0x7f0dc80ca280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:05.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.581+0000 7f0dc77fe700 1 --2- 172.21.15.16:0/1789247096 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0dc806a7e0 0x7f0dc80ca280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:05.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.581+0000 7f0dc77fe700 1 -- 172.21.15.16:0/1789247096 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0db80094a0 con 0x7f0dc806e570 2023-10-16T12:31:05.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.581+0000 7f0dc7fff700 1 --2- 172.21.15.16:0/1789247096 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0dc806a7e0 0x7f0dc80ca280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:31:05.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.581+0000 7f0dcc902700 1 --2- 172.21.15.16:0/1789247096 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0dc815b2c0 0x7f0dc80c9180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:31:05.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.581+0000 7f0dc77fe700 1 --2- 172.21.15.16:0/1789247096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0dc806e570 0x7f0dc81f2cb0 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7f0db8000c00 tx=0x7f0db8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:05.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.581+0000 7f0dc4ff9700 1 -- 172.21.15.16:0/1789247096 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db80038a0 con 0x7f0dc806e570 2023-10-16T12:31:05.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.581+0000 7f0dc4ff9700 1 -- 172.21.15.16:0/1789247096 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0db801f070 con 0x7f0dc806e570 2023-10-16T12:31:05.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.581+0000 7f0dce365700 1 -- 172.21.15.16:0/1789247096 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dc80c96c0 con 0x7f0dc806e570 2023-10-16T12:31:05.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.581+0000 7f0dc4ff9700 1 -- 172.21.15.16:0/1789247096 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db8003c40 con 0x7f0dc806e570 2023-10-16T12:31:05.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.581+0000 7f0dce365700 1 -- 172.21.15.16:0/1789247096 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dc81f3670 con 0x7f0dc806e570 2023-10-16T12:31:05.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.582+0000 7f0dce365700 1 -- 172.21.15.16:0/1789247096 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dc804ebb0 con 0x7f0dc806e570 2023-10-16T12:31:05.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.583+0000 7f0dc4ff9700 1 -- 172.21.15.16:0/1789247096 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 85) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0db8026480 con 0x7f0dc806e570 2023-10-16T12:31:05.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.583+0000 7f0dc4ff9700 1 --2- 172.21.15.16:0/1789247096 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f0db406b020 0x7f0db406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:05.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.583+0000 7f0dc4ff9700 1 -- 172.21.15.16:0/1789247096 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 5985+0+0 (secure 0 0 0) 0x7f0db8021070 con 0x7f0dc806e570 2023-10-16T12:31:05.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.583+0000 7f0dc7fff700 1 --2- 172.21.15.16:0/1789247096 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f0db406b020 0x7f0db406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:05.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.584+0000 7f0dc7fff700 1 --2- 172.21.15.16:0/1789247096 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f0db406b020 0x7f0db406d4e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0dc80b1e90 tx=0x7f0dbc008040 comp rx=0 tx=0).ready entity=mgr.8973 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:05.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.587+0000 7f0dc4ff9700 1 -- 172.21.15.16:0/1789247096 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0db8056510 con 0x7f0dc806e570 2023-10-16T12:31:05.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:05.835+0000 7f0dce365700 1 -- 172.21.15.16:0/1789247096 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f0dc81620b0 con 0x7f0dc806e570 2023-10-16T12:31:06.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.412+0000 7f0dc4ff9700 1 -- 172.21.15.16:0/1789247096 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 19 and data pool 20 v74) v1 ==== 174+0+0 (secure 0 0 0) 0x7f0db8015020 con 0x7f0dc806e570 2023-10-16T12:31:06.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.415+0000 7f0dce365700 1 -- 172.21.15.16:0/1789247096 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f0db406b020 msgr2=0x7f0db406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:06.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.415+0000 7f0dce365700 1 --2- 172.21.15.16:0/1789247096 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f0db406b020 0x7f0db406d4e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0dc80b1e90 tx=0x7f0dbc008040 comp rx=0 tx=0).stop 2023-10-16T12:31:06.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.415+0000 7f0dce365700 1 -- 172.21.15.16:0/1789247096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0dc806e570 msgr2=0x7f0dc81f2cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:06.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.415+0000 7f0dce365700 1 --2- 172.21.15.16:0/1789247096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0dc806e570 0x7f0dc81f2cb0 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7f0db8000c00 tx=0x7f0db8004750 comp rx=0 tx=0).stop 2023-10-16T12:31:06.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.416+0000 7f0dce365700 1 -- 172.21.15.16:0/1789247096 shutdown_connections 2023-10-16T12:31:06.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.416+0000 7f0dce365700 1 --2- 172.21.15.16:0/1789247096 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f0db406b020 0x7f0db406d4e0 secure :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0dc80b1e90 tx=0x7f0dbc008040 comp rx=0 tx=0).stop 2023-10-16T12:31:06.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.416+0000 7f0dce365700 1 --2- 172.21.15.16:0/1789247096 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0dc815b2c0 0x7f0dc80c9180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:06.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.416+0000 7f0dce365700 1 --2- 172.21.15.16:0/1789247096 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0dc806a7e0 0x7f0dc80ca280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:06.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.416+0000 7f0dce365700 1 --2- 172.21.15.16:0/1789247096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0dc806e570 0x7f0dc81f2cb0 unknown :-1 s=CLOSED pgs=1734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:06.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.416+0000 7f0dce365700 1 -- 172.21.15.16:0/1789247096 >> 172.21.15.16:0/1789247096 conn(0x7f0dc80b1f40 msgr2=0x7f0dc80cc240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:31:06.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.416+0000 7f0dce365700 1 -- 172.21.15.16:0/1789247096 shutdown_connections 2023-10-16T12:31:06.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.416+0000 7f0dce365700 1 -- 172.21.15.16:0/1789247096 wait complete. 2023-10-16T12:31:06.420 INFO:teuthology.orchestra.run.smithi016.stderr:new fs with metadata pool 19 and data pool 20 2023-10-16T12:31:06.438 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:31:06.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.568+0000 7f18c6b33700 1 Processor -- start 2023-10-16T12:31:06.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.568+0000 7f18c6b33700 1 -- start start 2023-10-16T12:31:06.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.569+0000 7f18c6b33700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f18c015b280 0x7f18c01613e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:06.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.569+0000 7f18c6b33700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f18c006a7e0 0x7f18c006ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:06.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.569+0000 7f18c6b33700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18c0161c30 con 0x7f18c00cc8a0 2023-10-16T12:31:06.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.569+0000 7f18c6b33700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18c006b190 con 0x7f18c015b280 2023-10-16T12:31:06.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.569+0000 7f18c6b33700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18c0161f10 con 0x7f18c006a7e0 2023-10-16T12:31:06.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.569+0000 7f18c48cf700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f18c015b280 0x7f18c01613e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:06.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.569+0000 7f18bffff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f18c00cc8a0 0x7f18c015e700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36132/0 (socket says 172.21.15.16:36132) 2023-10-16T12:31:06.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.569+0000 7f18bffff700 1 -- 172.21.15.16:0/3651397896 learned_addr learned my addr 172.21.15.16:0/3651397896 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:31:06.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.570+0000 7f18c48cf700 1 -- 172.21.15.16:0/3651397896 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f18c006a7e0 msgr2=0x7f18c006ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:31:06.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.570+0000 7f18c50d0700 1 --2- 172.21.15.16:0/3651397896 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f18c006a7e0 0x7f18c006ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:06.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.570+0000 7f18c48cf700 1 --2- 172.21.15.16:0/3651397896 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f18c006a7e0 0x7f18c006ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:06.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.570+0000 7f18c48cf700 1 -- 172.21.15.16:0/3651397896 >> v1:172.21.15.16:6789/0 conn(0x7f18c00cc8a0 legacy=0x7f18c015e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:06.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.570+0000 7f18bf7fe700 1 -- 172.21.15.16:0/3651397896 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3229318872 0 0) 0x7f18c0161c30 con 0x7f18c00cc8a0 2023-10-16T12:31:06.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.570+0000 7f18c48cf700 1 -- 172.21.15.16:0/3651397896 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18c01621f0 con 0x7f18c015b280 2023-10-16T12:31:06.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.570+0000 7f18c50d0700 1 --2- 172.21.15.16:0/3651397896 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f18c006a7e0 0x7f18c006ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:31:06.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.570+0000 7f18c48cf700 1 --2- 172.21.15.16:0/3651397896 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18c015b280 0x7f18c01613e0 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7f18b4009840 tx=0x7f18b4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a297899b0b3acbc6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:06.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.570+0000 7f18bf7fe700 1 -- 172.21.15.16:0/3651397896 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18b4011830 con 0x7f18c015b280 2023-10-16T12:31:06.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.570+0000 7f18bf7fe700 1 -- 172.21.15.16:0/3651397896 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18b4011990 con 0x7f18c015b280 2023-10-16T12:31:06.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.570+0000 7f18bf7fe700 1 -- 172.21.15.16:0/3651397896 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18b4011d20 con 0x7f18c015b280 2023-10-16T12:31:06.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.570+0000 7f18c6b33700 1 -- 172.21.15.16:0/3651397896 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18c015b280 msgr2=0x7f18c01613e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:06.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.570+0000 7f18c6b33700 1 --2- 172.21.15.16:0/3651397896 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18c015b280 0x7f18c01613e0 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7f18b4009840 tx=0x7f18b4009b50 comp rx=0 tx=0).stop 2023-10-16T12:31:06.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.570+0000 7f18c6b33700 1 -- 172.21.15.16:0/3651397896 shutdown_connections 2023-10-16T12:31:06.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.570+0000 7f18c6b33700 1 --2- 172.21.15.16:0/3651397896 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f18c006a7e0 0x7f18c006ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:06.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.570+0000 7f18c6b33700 1 --2- 172.21.15.16:0/3651397896 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18c015b280 0x7f18c01613e0 unknown :-1 s=CLOSED pgs=1736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:06.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.570+0000 7f18c6b33700 1 -- 172.21.15.16:0/3651397896 >> 172.21.15.16:0/3651397896 conn(0x7f18c00b1fd0 msgr2=0x7f18c00b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:31:06.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.570+0000 7f18c6b33700 1 -- 172.21.15.16:0/3651397896 shutdown_connections 2023-10-16T12:31:06.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.571+0000 7f18c6b33700 1 -- 172.21.15.16:0/3651397896 wait complete. 2023-10-16T12:31:06.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.572+0000 7f18c6b33700 1 Processor -- start 2023-10-16T12:31:06.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.572+0000 7f18c6b33700 1 -- start start 2023-10-16T12:31:06.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.573+0000 7f18c6b33700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f18c006a7e0 0x7f18c00c8c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:06.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.573+0000 7f18c6b33700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18c00cc8a0 0x7f18c00c6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:06.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.573+0000 7f18c6b33700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f18c015b280 0x7f18c00c66b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:06.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.573+0000 7f18c6b33700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18c00ca380 con 0x7f18c00cc8a0 2023-10-16T12:31:06.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.573+0000 7f18c6b33700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18c00ca810 con 0x7f18c006a7e0 2023-10-16T12:31:06.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.573+0000 7f18c6b33700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18c00c9770 con 0x7f18c015b280 2023-10-16T12:31:06.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.573+0000 7f18bffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18c00cc8a0 0x7f18c00c6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:06.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.573+0000 7f18bffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18c00cc8a0 0x7f18c00c6170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46864/0 (socket says 172.21.15.16:46864) 2023-10-16T12:31:06.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.573+0000 7f18bffff700 1 -- 172.21.15.16:0/3536744814 learned_addr learned my addr 172.21.15.16:0/3536744814 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:31:06.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.573+0000 7f18c50d0700 1 --2- 172.21.15.16:0/3536744814 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f18c015b280 0x7f18c00c66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:06.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.573+0000 7f18c48cf700 1 --2- 172.21.15.16:0/3536744814 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f18c006a7e0 0x7f18c00c8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:06.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.573+0000 7f18bffff700 1 -- 172.21.15.16:0/3536744814 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f18c015b280 msgr2=0x7f18c00c66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:06.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.573+0000 7f18bffff700 1 --2- 172.21.15.16:0/3536744814 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f18c015b280 0x7f18c00c66b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:06.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.573+0000 7f18bffff700 1 -- 172.21.15.16:0/3536744814 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f18c006a7e0 msgr2=0x7f18c00c8c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:06.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.573+0000 7f18bffff700 1 --2- 172.21.15.16:0/3536744814 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f18c006a7e0 0x7f18c00c8c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:06.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.573+0000 7f18bffff700 1 -- 172.21.15.16:0/3536744814 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18b40094a0 con 0x7f18c00cc8a0 2023-10-16T12:31:06.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.573+0000 7f18bffff700 1 --2- 172.21.15.16:0/3536744814 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18c00cc8a0 0x7f18c00c6170 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f18ac00c540 tx=0x7f18ac00c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:06.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.573+0000 7f18bd7fa700 1 -- 172.21.15.16:0/3536744814 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18ac004c80 con 0x7f18c00cc8a0 2023-10-16T12:31:06.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.573+0000 7f18bd7fa700 1 -- 172.21.15.16:0/3536744814 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18ac004de0 con 0x7f18c00cc8a0 2023-10-16T12:31:06.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.573+0000 7f18bd7fa700 1 -- 172.21.15.16:0/3536744814 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18ac007310 con 0x7f18c00cc8a0 2023-10-16T12:31:06.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.573+0000 7f18c6b33700 1 -- 172.21.15.16:0/3536744814 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18c00c9a50 con 0x7f18c00cc8a0 2023-10-16T12:31:06.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.573+0000 7f18c6b33700 1 -- 172.21.15.16:0/3536744814 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18c00ca010 con 0x7f18c00cc8a0 2023-10-16T12:31:06.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.574+0000 7f18c6b33700 1 -- 172.21.15.16:0/3536744814 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18c004ebb0 con 0x7f18c00cc8a0 2023-10-16T12:31:06.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.575+0000 7f18bd7fa700 1 -- 172.21.15.16:0/3536744814 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 85) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f18ac007470 con 0x7f18c00cc8a0 2023-10-16T12:31:06.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.575+0000 7f18bd7fa700 1 --2- 172.21.15.16:0/3536744814 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f18a406b020 0x7f18a406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:06.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.575+0000 7f18bd7fa700 1 -- 172.21.15.16:0/3536744814 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 6085+0+0 (secure 0 0 0) 0x7f18ac089640 con 0x7f18c00cc8a0 2023-10-16T12:31:06.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.575+0000 7f18c48cf700 1 --2- 172.21.15.16:0/3536744814 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f18a406b020 0x7f18a406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:06.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.576+0000 7f18c48cf700 1 --2- 172.21.15.16:0/3536744814 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f18a406b020 0x7f18a406d4e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f18b40090b0 tx=0x7f18b401c040 comp rx=0 tx=0).ready entity=mgr.8973 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:06.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.580+0000 7f18bd7fa700 1 -- 172.21.15.16:0/3536744814 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f18ac054450 con 0x7f18c00cc8a0 2023-10-16T12:31:06.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.783+0000 7f18c6b33700 1 -- 172.21.15.16:0/3536744814 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f18c00ca4c0 con 0x7f18c00cc8a0 2023-10-16T12:31:06.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.784+0000 7f18bd7fa700 1 -- 172.21.15.16:0/3536744814 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v155) v1 ==== 74+0+16725 (secure 0 0 0) 0x7f18ac057a70 con 0x7f18c00cc8a0 2023-10-16T12:31:06.787 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:31:06.787 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":155,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:31:05.836577+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:31:04.290489+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2023-10-16T12:31:04.803486+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:31:06.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.786+0000 7f18c6b33700 1 -- 172.21.15.16:0/3536744814 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f18a406b020 msgr2=0x7f18a406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:06.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.786+0000 7f18c6b33700 1 --2- 172.21.15.16:0/3536744814 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f18a406b020 0x7f18a406d4e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f18b40090b0 tx=0x7f18b401c040 comp rx=0 tx=0).stop 2023-10-16T12:31:06.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.786+0000 7f18c6b33700 1 -- 172.21.15.16:0/3536744814 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18c00cc8a0 msgr2=0x7f18c00c6170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:06.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.786+0000 7f18c6b33700 1 --2- 172.21.15.16:0/3536744814 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18c00cc8a0 0x7f18c00c6170 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f18ac00c540 tx=0x7f18ac00c900 comp rx=0 tx=0).stop 2023-10-16T12:31:06.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.787+0000 7f18c6b33700 1 -- 172.21.15.16:0/3536744814 shutdown_connections 2023-10-16T12:31:06.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.787+0000 7f18c6b33700 1 --2- 172.21.15.16:0/3536744814 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f18a406b020 0x7f18a406d4e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:06.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.787+0000 7f18c6b33700 1 --2- 172.21.15.16:0/3536744814 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f18c015b280 0x7f18c00c66b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:06.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.787+0000 7f18c6b33700 1 --2- 172.21.15.16:0/3536744814 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f18c006a7e0 0x7f18c00c8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:06.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.787+0000 7f18c6b33700 1 --2- 172.21.15.16:0/3536744814 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18c00cc8a0 0x7f18c00c6170 unknown :-1 s=CLOSED pgs=1737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:06.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.787+0000 7f18c6b33700 1 -- 172.21.15.16:0/3536744814 >> 172.21.15.16:0/3536744814 conn(0x7f18c00b1fd0 msgr2=0x7f18c00d0270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:31:06.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.787+0000 7f18c6b33700 1 -- 172.21.15.16:0/3536744814 shutdown_connections 2023-10-16T12:31:06.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.787+0000 7f18c6b33700 1 -- 172.21.15.16:0/3536744814 wait complete. 2023-10-16T12:31:06.801 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:31:06.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.924+0000 7fad085a9700 1 Processor -- start 2023-10-16T12:31:06.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.924+0000 7fad085a9700 1 -- start start 2023-10-16T12:31:06.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.925+0000 7fad085a9700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fad0015b630 0x7fad000d0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:06.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.925+0000 7fad085a9700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad000b0970 con 0x7fad000ce4f0 2023-10-16T12:31:06.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.925+0000 7fad085a9700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad000cd690 con 0x7fad0015ba10 2023-10-16T12:31:06.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.925+0000 7fad085a9700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad000cd970 con 0x7fad0015b630 2023-10-16T12:31:06.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.925+0000 7fad06b46700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fad000ce4f0 0x7fad000b1430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36148/0 (socket says 172.21.15.16:36148) 2023-10-16T12:31:06.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.925+0000 7fad06b46700 1 -- 172.21.15.16:0/3406896937 learned_addr learned my addr 172.21.15.16:0/3406896937 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:31:06.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.925+0000 7fad06345700 1 --2- 172.21.15.16:0/3406896937 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fad0015b630 0x7fad000d0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:06.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.925+0000 7fad05343700 1 -- 172.21.15.16:0/3406896937 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3054800156 0 0) 0x7fad000b0970 con 0x7fad000ce4f0 2023-10-16T12:31:06.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.925+0000 7fad05343700 1 -- 172.21.15.16:0/3406896937 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facec0035b0 con 0x7fad000ce4f0 2023-10-16T12:31:06.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.925+0000 7fad05343700 1 -- 172.21.15.16:0/3406896937 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3650580732 0 0) 0x7fad000cd690 con 0x7fad0015ba10 2023-10-16T12:31:06.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.925+0000 7fad05343700 1 -- 172.21.15.16:0/3406896937 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad000b0970 con 0x7fad0015ba10 2023-10-16T12:31:06.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.926+0000 7fad05343700 1 -- 172.21.15.16:0/3406896937 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 349496132 0 0) 0x7facec0035b0 con 0x7fad000ce4f0 2023-10-16T12:31:06.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.926+0000 7fad05343700 1 -- 172.21.15.16:0/3406896937 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fad0015b630 msgr2=0x7fad000d0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:31:06.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.926+0000 7fad05343700 1 --2- 172.21.15.16:0/3406896937 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fad0015b630 0x7fad000d0380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:06.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.926+0000 7fad05343700 1 -- 172.21.15.16:0/3406896937 >> v1:172.21.15.49:6789/0 conn(0x7fad0015ba10 legacy=0x7fad000cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:06.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.926+0000 7fad05343700 1 -- 172.21.15.16:0/3406896937 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad000b0c30 con 0x7fad000ce4f0 2023-10-16T12:31:06.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.926+0000 7fad05343700 1 -- 172.21.15.16:0/3406896937 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7facf80019f0 con 0x7fad000ce4f0 2023-10-16T12:31:06.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.926+0000 7fad06345700 1 --2- 172.21.15.16:0/3406896937 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fad0015b630 0x7fad000d0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:31:06.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.926+0000 7fad05343700 1 -- 172.21.15.16:0/3406896937 >> v1:172.21.15.16:6789/0 conn(0x7fad000ce4f0 legacy=0x7fad000b1430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:06.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.926+0000 7fad05343700 1 --2- 172.21.15.16:0/3406896937 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7facec003bc0 0x7facec003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:06.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.926+0000 7fad05343700 1 --2- 172.21.15.16:0/3406896937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facec004850 0x7facec004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:06.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.926+0000 7fad05343700 1 --2- 172.21.15.16:0/3406896937 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fad0015b630 0x7facec005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:06.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.926+0000 7fad05343700 1 -- 172.21.15.16:0/3406896937 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad000cd970 con 0x7facec004850 2023-10-16T12:31:06.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.926+0000 7fad05343700 1 -- 172.21.15.16:0/3406896937 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facec00ba40 con 0x7facec003bc0 2023-10-16T12:31:06.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.926+0000 7fad05343700 1 -- 172.21.15.16:0/3406896937 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facec00bc00 con 0x7fad0015b630 2023-10-16T12:31:06.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.926+0000 7fad05b44700 1 --2- 172.21.15.16:0/3406896937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facec004850 0x7facec004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:06.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.926+0000 7fad06345700 1 --2- 172.21.15.16:0/3406896937 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7facec003bc0 0x7facec003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:06.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.926+0000 7fad06b46700 1 --2- 172.21.15.16:0/3406896937 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fad0015b630 0x7facec005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:06.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.926+0000 7fad05b44700 1 -- 172.21.15.16:0/3406896937 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fad0015b630 msgr2=0x7facec005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:31:06.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.926+0000 7fad05b44700 1 --2- 172.21.15.16:0/3406896937 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fad0015b630 0x7facec005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:06.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.926+0000 7fad05b44700 1 -- 172.21.15.16:0/3406896937 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7facec003bc0 msgr2=0x7facec003fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:31:06.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.926+0000 7fad05b44700 1 --2- 172.21.15.16:0/3406896937 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7facec003bc0 0x7facec003fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:06.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.926+0000 7fad05b44700 1 -- 172.21.15.16:0/3406896937 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facf40094a0 con 0x7facec004850 2023-10-16T12:31:06.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.926+0000 7fad06b46700 1 --2- 172.21.15.16:0/3406896937 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fad0015b630 0x7facec005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:31:06.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.926+0000 7fad05b44700 1 --2- 172.21.15.16:0/3406896937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facec004850 0x7facec004cd0 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7facfc007fd0 tx=0x7facec005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=779263b7f959be15 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:06.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.926+0000 7fad05343700 1 -- 172.21.15.16:0/3406896937 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facfc001430 con 0x7facec004850 2023-10-16T12:31:06.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.926+0000 7fad05343700 1 -- 172.21.15.16:0/3406896937 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facfc0052d0 con 0x7facec004850 2023-10-16T12:31:06.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.926+0000 7fad05343700 1 -- 172.21.15.16:0/3406896937 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facfc001d40 con 0x7facec004850 2023-10-16T12:31:06.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.927+0000 7fad085a9700 1 -- 172.21.15.16:0/3406896937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facec004850 msgr2=0x7facec004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:06.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.927+0000 7fad085a9700 1 --2- 172.21.15.16:0/3406896937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facec004850 0x7facec004cd0 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7facfc007fd0 tx=0x7facec005880 comp rx=0 tx=0).stop 2023-10-16T12:31:06.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.927+0000 7fad06b46700 1 -- 172.21.15.16:0/3406896937 reap_dead start 2023-10-16T12:31:06.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.927+0000 7fad085a9700 1 -- 172.21.15.16:0/3406896937 shutdown_connections 2023-10-16T12:31:06.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.927+0000 7fad085a9700 1 -- 172.21.15.16:0/3406896937 >> 172.21.15.16:0/3406896937 conn(0x7fad000d14b0 msgr2=0x7fad000d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:31:06.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.927+0000 7fad085a9700 1 -- 172.21.15.16:0/3406896937 shutdown_connections 2023-10-16T12:31:06.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.927+0000 7fad085a9700 1 -- 172.21.15.16:0/3406896937 wait complete. 2023-10-16T12:31:06.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.928+0000 7fad085a9700 1 Processor -- start 2023-10-16T12:31:06.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.928+0000 7fad085a9700 1 -- start start 2023-10-16T12:31:06.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.929+0000 7fad085a9700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7facec004850 0x7fad000b5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:06.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.929+0000 7fad085a9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad0015d990 0x7fad000d1b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:06.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.929+0000 7fad085a9700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fad001505b0 0x7fad00150a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:06.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.929+0000 7fad085a9700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad000b6880 con 0x7fad0015d990 2023-10-16T12:31:06.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.929+0000 7fad085a9700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad000b6ca0 con 0x7fad001505b0 2023-10-16T12:31:06.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.929+0000 7fad085a9700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad00150fa0 con 0x7facec004850 2023-10-16T12:31:06.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.929+0000 7fad05b44700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad0015d990 0x7fad000d1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:06.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.929+0000 7fad05b44700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad0015d990 0x7fad000d1b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46884/0 (socket says 172.21.15.16:46884) 2023-10-16T12:31:06.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.929+0000 7fad05b44700 1 -- 172.21.15.16:0/2697123881 learned_addr learned my addr 172.21.15.16:0/2697123881 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:31:06.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.929+0000 7fad06345700 1 --2- 172.21.15.16:0/2697123881 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7facec004850 0x7fad000b5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:06.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.929+0000 7fad06b46700 1 --2- 172.21.15.16:0/2697123881 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fad001505b0 0x7fad00150a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:06.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.929+0000 7fad05b44700 1 -- 172.21.15.16:0/2697123881 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7facec004850 msgr2=0x7fad000b5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:06.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.929+0000 7fad05b44700 1 --2- 172.21.15.16:0/2697123881 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7facec004850 0x7fad000b5540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:06.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.930+0000 7fad05b44700 1 -- 172.21.15.16:0/2697123881 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fad001505b0 msgr2=0x7fad00150a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:06.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.930+0000 7fad05b44700 1 --2- 172.21.15.16:0/2697123881 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fad001505b0 0x7fad00150a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:06.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.930+0000 7fad05b44700 1 -- 172.21.15.16:0/2697123881 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad000b5c00 con 0x7fad0015d990 2023-10-16T12:31:06.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.930+0000 7fad05b44700 1 --2- 172.21.15.16:0/2697123881 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad0015d990 0x7fad000d1b30 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7facfc00c3c0 tx=0x7facfc0012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:06.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.930+0000 7facf2ffd700 1 -- 172.21.15.16:0/2697123881 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facfc001d40 con 0x7fad0015d990 2023-10-16T12:31:06.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.930+0000 7facf2ffd700 1 -- 172.21.15.16:0/2697123881 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facfc0013d0 con 0x7fad0015d990 2023-10-16T12:31:06.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.930+0000 7fad085a9700 1 -- 172.21.15.16:0/2697123881 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad000b5ee0 con 0x7fad0015d990 2023-10-16T12:31:06.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.930+0000 7facf2ffd700 1 -- 172.21.15.16:0/2697123881 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facfc007280 con 0x7fad0015d990 2023-10-16T12:31:06.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.930+0000 7fad085a9700 1 -- 172.21.15.16:0/2697123881 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad000b64a0 con 0x7fad0015d990 2023-10-16T12:31:06.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.931+0000 7fad085a9700 1 -- 172.21.15.16:0/2697123881 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad0004ebb0 con 0x7fad0015d990 2023-10-16T12:31:06.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.931+0000 7facf2ffd700 1 -- 172.21.15.16:0/2697123881 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 85) v1 ==== 89331+0+0 (secure 0 0 0) 0x7facfc0073e0 con 0x7fad0015d990 2023-10-16T12:31:06.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.932+0000 7facf2ffd700 1 --2- 172.21.15.16:0/2697123881 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7face406b020 0x7face406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:06.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.932+0000 7facf2ffd700 1 -- 172.21.15.16:0/2697123881 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 6085+0+0 (secure 0 0 0) 0x7facfc08a870 con 0x7fad0015d990 2023-10-16T12:31:06.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.932+0000 7fad06345700 1 --2- 172.21.15.16:0/2697123881 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7face406b020 0x7face406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:06.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.932+0000 7fad06345700 1 --2- 172.21.15.16:0/2697123881 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7face406b020 0x7face406d4e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7facf4009d00 tx=0x7facf4009210 comp rx=0 tx=0).ready entity=mgr.8973 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:06.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:06.937+0000 7facf2ffd700 1 -- 172.21.15.16:0/2697123881 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7facfc055680 con 0x7fad0015d990 2023-10-16T12:31:07.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:07.141+0000 7fad085a9700 1 -- 172.21.15.16:0/2697123881 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fad0006b5d0 con 0x7fad0015d990 2023-10-16T12:31:07.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:07.142+0000 7facf2ffd700 1 -- 172.21.15.16:0/2697123881 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v155) v1 ==== 74+0+16725 (secure 0 0 0) 0x7facfc058ca0 con 0x7fad0015d990 2023-10-16T12:31:07.145 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:31:07.145 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":155,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:31:05.836577+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:31:04.290489+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2023-10-16T12:31:04.803486+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:31:07.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:07.144+0000 7fad085a9700 1 -- 172.21.15.16:0/2697123881 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7face406b020 msgr2=0x7face406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:07.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:07.144+0000 7fad085a9700 1 --2- 172.21.15.16:0/2697123881 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7face406b020 0x7face406d4e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7facf4009d00 tx=0x7facf4009210 comp rx=0 tx=0).stop 2023-10-16T12:31:07.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:07.144+0000 7fad085a9700 1 -- 172.21.15.16:0/2697123881 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad0015d990 msgr2=0x7fad000d1b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:07.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:07.144+0000 7fad085a9700 1 --2- 172.21.15.16:0/2697123881 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad0015d990 0x7fad000d1b30 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7facfc00c3c0 tx=0x7facfc0012d0 comp rx=0 tx=0).stop 2023-10-16T12:31:07.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:07.144+0000 7fad085a9700 1 -- 172.21.15.16:0/2697123881 shutdown_connections 2023-10-16T12:31:07.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:07.144+0000 7fad085a9700 1 --2- 172.21.15.16:0/2697123881 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7face406b020 0x7face406d4e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:07.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:07.144+0000 7fad085a9700 1 --2- 172.21.15.16:0/2697123881 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fad001505b0 0x7fad00150a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:07.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:07.144+0000 7fad085a9700 1 --2- 172.21.15.16:0/2697123881 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7facec004850 0x7fad000b5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:07.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:07.144+0000 7fad085a9700 1 --2- 172.21.15.16:0/2697123881 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad0015d990 0x7fad000d1b30 unknown :-1 s=CLOSED pgs=1740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:07.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:07.144+0000 7fad085a9700 1 -- 172.21.15.16:0/2697123881 >> 172.21.15.16:0/2697123881 conn(0x7fad000d14b0 msgr2=0x7fad000cf5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:31:07.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:07.145+0000 7fad085a9700 1 -- 172.21.15.16:0/2697123881 shutdown_connections 2023-10-16T12:31:07.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:07.145+0000 7fad085a9700 1 -- 172.21.15.16:0/2697123881 wait complete. 2023-10-16T12:31:07.159 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-16T12:31:07.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:07.285+0000 7f8998ba1700 1 Processor -- start 2023-10-16T12:31:07.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:07.286+0000 7f8998ba1700 1 -- start start 2023-10-16T12:31:07.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:07.286+0000 7f8998ba1700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f899415b490 0x7f899406b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:07.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:07.286+0000 7f8998ba1700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f899406b6e0 con 0x7f89940ce4e0 2023-10-16T12:31:07.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:07.286+0000 7f8998ba1700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8994161320 con 0x7f89940cd7a0 2023-10-16T12:31:07.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:07.286+0000 7f8998ba1700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8994161fa0 con 0x7f899415b490 2023-10-16T12:31:07.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:07.286+0000 7f89937fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f899415b490 0x7f899406b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:07.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:07.286+0000 7f8993fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f89940ce4e0 0x7f899415dc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36168/0 (socket says 172.21.15.16:36168) 2023-10-16T12:31:07.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:07.287+0000 7f8993fff700 1 -- 172.21.15.16:0/1239549339 learned_addr learned my addr 172.21.15.16:0/1239549339 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:31:07.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:07.287+0000 7f89937fe700 1 -- 172.21.15.16:0/1239549339 >> v1:172.21.15.49:6789/0 conn(0x7f89940cd7a0 legacy=0x7f89940cddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:31:07.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:07.287+0000 7f89937fe700 1 -- 172.21.15.16:0/1239549339 >> v1:172.21.15.16:6789/0 conn(0x7f89940ce4e0 legacy=0x7f899415dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:07.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:07.287+0000 7f89937fe700 1 -- 172.21.15.16:0/1239549339 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8994161600 con 0x7f899415b490 2023-10-16T12:31:07.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:07.287+0000 7f89937fe700 1 --2- 172.21.15.16:0/1239549339 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f899415b490 0x7f899406b1a0 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7f8984009840 tx=0x7f8984009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44fd66d41cdf82a5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:07.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:07.287+0000 7f89927fc700 1 -- 172.21.15.16:0/1239549339 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8984004230 con 0x7f899415b490 2023-10-16T12:31:07.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:07.287+0000 7f89927fc700 1 -- 172.21.15.16:0/1239549339 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8984004390 con 0x7f899415b490 2023-10-16T12:31:07.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:07.287+0000 7f89927fc700 1 -- 172.21.15.16:0/1239549339 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89840036a0 con 0x7f899415b490 2023-10-16T12:31:07.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:07.288+0000 7f8998ba1700 1 -- 172.21.15.16:0/1239549339 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f899415b490 msgr2=0x7f899406b1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:07.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:07.288+0000 7f8998ba1700 1 --2- 172.21.15.16:0/1239549339 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f899415b490 0x7f899406b1a0 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7f8984009840 tx=0x7f8984009b50 comp rx=0 tx=0).stop 2023-10-16T12:31:07.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:07.288+0000 7f8998ba1700 1 -- 172.21.15.16:0/1239549339 shutdown_connections 2023-10-16T12:31:07.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:07.288+0000 7f8998ba1700 1 --2- 172.21.15.16:0/1239549339 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f899415b490 0x7f899406b1a0 unknown :-1 s=CLOSED pgs=1742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:07.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:07.288+0000 7f8998ba1700 1 -- 172.21.15.16:0/1239549339 >> 172.21.15.16:0/1239549339 conn(0x7f89940ccc80 msgr2=0x7f89940cd090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:31:07.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:07.288+0000 7f8998ba1700 1 -- 172.21.15.16:0/1239549339 shutdown_connections 2023-10-16T12:31:07.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:07.288+0000 7f8998ba1700 1 -- 172.21.15.16:0/1239549339 wait complete. 2023-10-16T12:31:07.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:07.289+0000 7f8998ba1700 1 Processor -- start 2023-10-16T12:31:07.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:07.290+0000 7f8998ba1700 1 -- start start 2023-10-16T12:31:07.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:07.290+0000 7f8998ba1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89940cd7a0 0x7f89940c4d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:07.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:07.290+0000 7f8998ba1700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f89940ce4e0 0x7f89940c7700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:07.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:07.290+0000 7f8998ba1700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f899415b490 0x7f89940c9c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:07.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:07.290+0000 7f8998ba1700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89940c6270 con 0x7f89940cd7a0 2023-10-16T12:31:07.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:07.290+0000 7f8998ba1700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89940c6700 con 0x7f899415b490 2023-10-16T12:31:07.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:07.290+0000 7f8998ba1700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89940c5660 con 0x7f89940ce4e0 2023-10-16T12:31:07.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:07.290+0000 7f89937fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89940cd7a0 0x7f89940c4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:07.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:07.290+0000 7f89937fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89940cd7a0 0x7f89940c4d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46896/0 (socket says 172.21.15.16:46896) 2023-10-16T12:31:07.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:07.290+0000 7f89937fe700 1 -- 172.21.15.16:0/2920742955 learned_addr learned my addr 172.21.15.16:0/2920742955 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:31:07.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:07.291+0000 7f8993fff700 1 --2- 172.21.15.16:0/2920742955 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f899415b490 0x7f89940c9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:07.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:07.291+0000 7f89937fe700 1 -- 172.21.15.16:0/2920742955 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f89940ce4e0 msgr2=0x7f89940c7700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:07.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:07.291+0000 7f8992ffd700 1 --2- 172.21.15.16:0/2920742955 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f89940ce4e0 0x7f89940c7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:07.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:07.291+0000 7f89937fe700 1 --2- 172.21.15.16:0/2920742955 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f89940ce4e0 0x7f89940c7700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:07.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:07.291+0000 7f89937fe700 1 -- 172.21.15.16:0/2920742955 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f899415b490 msgr2=0x7f89940c9c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:07.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:07.291+0000 7f89937fe700 1 --2- 172.21.15.16:0/2920742955 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f899415b490 0x7f89940c9c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:07.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:07.291+0000 7f89937fe700 1 -- 172.21.15.16:0/2920742955 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89840094a0 con 0x7f89940cd7a0 2023-10-16T12:31:07.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:07.291+0000 7f8993fff700 1 --2- 172.21.15.16:0/2920742955 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f899415b490 0x7f89940c9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:31:07.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:07.291+0000 7f8992ffd700 1 --2- 172.21.15.16:0/2920742955 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f89940ce4e0 0x7f89940c7700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:31:07.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:07.291+0000 7f89937fe700 1 --2- 172.21.15.16:0/2920742955 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89940cd7a0 0x7f89940c4d80 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7f89840090b0 tx=0x7f89840047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:07.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:07.291+0000 7f898bfff700 1 -- 172.21.15.16:0/2920742955 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8984011630 con 0x7f89940cd7a0 2023-10-16T12:31:07.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:07.291+0000 7f898bfff700 1 -- 172.21.15.16:0/2920742955 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8984011790 con 0x7f89940cd7a0 2023-10-16T12:31:07.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:07.291+0000 7f8998ba1700 1 -- 172.21.15.16:0/2920742955 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89940c5940 con 0x7f89940cd7a0 2023-10-16T12:31:07.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:07.291+0000 7f898bfff700 1 -- 172.21.15.16:0/2920742955 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8984011b40 con 0x7f89940cd7a0 2023-10-16T12:31:07.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:07.291+0000 7f8998ba1700 1 -- 172.21.15.16:0/2920742955 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89940c5f00 con 0x7f89940cd7a0 2023-10-16T12:31:07.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:07.292+0000 7f8998ba1700 1 -- 172.21.15.16:0/2920742955 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f899404ebb0 con 0x7f89940cd7a0 2023-10-16T12:31:07.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:07.292+0000 7f898bfff700 1 -- 172.21.15.16:0/2920742955 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 85) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8984011ca0 con 0x7f89940cd7a0 2023-10-16T12:31:07.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:07.293+0000 7f898bfff700 1 --2- 172.21.15.16:0/2920742955 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f897406b020 0x7f897406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:07.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:07.293+0000 7f898bfff700 1 -- 172.21.15.16:0/2920742955 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 6085+0+0 (secure 0 0 0) 0x7f898408b130 con 0x7f89940cd7a0 2023-10-16T12:31:07.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:07.293+0000 7f8992ffd700 1 --2- 172.21.15.16:0/2920742955 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f897406b020 0x7f897406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:07.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:07.293+0000 7f8992ffd700 1 --2- 172.21.15.16:0/2920742955 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f897406b020 0x7f897406d4e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f89940c38a0 tx=0x7f897c009040 comp rx=0 tx=0).ready entity=mgr.8973 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:07.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:07.297+0000 7f898bfff700 1 -- 172.21.15.16:0/2920742955 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8984056100 con 0x7f89940cd7a0 2023-10-16T12:31:07.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:07.555+0000 7f8998ba1700 1 -- 172.21.15.16:0/2920742955 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f89940728c0 con 0x7f89940cd7a0 2023-10-16T12:31:08.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:08.425+0000 7f898bfff700 1 -- 172.21.15.16:0/2920742955 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v77) v1 ==== 118+0+0 (secure 0 0 0) 0x7f8984059720 con 0x7f89940cd7a0 2023-10-16T12:31:08.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:08.428+0000 7f8998ba1700 1 -- 172.21.15.16:0/2920742955 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f897406b020 msgr2=0x7f897406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:08.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:08.428+0000 7f8998ba1700 1 --2- 172.21.15.16:0/2920742955 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f897406b020 0x7f897406d4e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f89940c38a0 tx=0x7f897c009040 comp rx=0 tx=0).stop 2023-10-16T12:31:08.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:08.428+0000 7f8998ba1700 1 -- 172.21.15.16:0/2920742955 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89940cd7a0 msgr2=0x7f89940c4d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:08.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:08.428+0000 7f8998ba1700 1 --2- 172.21.15.16:0/2920742955 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89940cd7a0 0x7f89940c4d80 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7f89840090b0 tx=0x7f89840047c0 comp rx=0 tx=0).stop 2023-10-16T12:31:08.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:08.429+0000 7f8998ba1700 1 -- 172.21.15.16:0/2920742955 shutdown_connections 2023-10-16T12:31:08.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:08.429+0000 7f8998ba1700 1 --2- 172.21.15.16:0/2920742955 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f897406b020 0x7f897406d4e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:08.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:08.429+0000 7f8998ba1700 1 --2- 172.21.15.16:0/2920742955 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f899415b490 0x7f89940c9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:08.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:08.429+0000 7f8998ba1700 1 --2- 172.21.15.16:0/2920742955 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89940cd7a0 0x7f89940c4d80 unknown :-1 s=CLOSED pgs=1743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:08.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:08.429+0000 7f8998ba1700 1 --2- 172.21.15.16:0/2920742955 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f89940ce4e0 0x7f89940c7700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:08.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:08.429+0000 7f8998ba1700 1 -- 172.21.15.16:0/2920742955 >> 172.21.15.16:0/2920742955 conn(0x7f89940ccc80 msgr2=0x7f89940b14c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:31:08.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:08.429+0000 7f8998ba1700 1 -- 172.21.15.16:0/2920742955 shutdown_connections 2023-10-16T12:31:08.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:08.429+0000 7f8998ba1700 1 -- 172.21.15.16:0/2920742955 wait complete. 2023-10-16T12:31:08.446 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-16T12:31:08.446 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-16T12:31:08.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:08.580+0000 7f8e268d0700 1 Processor -- start 2023-10-16T12:31:08.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:08.580+0000 7f8e268d0700 1 -- start start 2023-10-16T12:31:08.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:08.581+0000 7f8e268d0700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f8e200ce5d0 0x7f8e200cc190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:08.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:08.581+0000 7f8e268d0700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e200b2440 con 0x7f8e2015f060 2023-10-16T12:31:08.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:08.581+0000 7f8e268d0700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e200d00e0 con 0x7f8e2015ec80 2023-10-16T12:31:08.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:08.581+0000 7f8e268d0700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e200b2c50 con 0x7f8e200ce5d0 2023-10-16T12:31:08.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:08.581+0000 7f8e24e6d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f8e2015f060 0x7f8e200cc6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36182/0 (socket says 172.21.15.16:36182) 2023-10-16T12:31:08.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:08.581+0000 7f8e24e6d700 1 -- 172.21.15.16:0/3056809352 learned_addr learned my addr 172.21.15.16:0/3056809352 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:31:08.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:08.581+0000 7f8e1f7fe700 1 --2- 172.21.15.16:0/3056809352 >> v2:172.21.15.49:3300/0 conn(0x7f8e200ce5d0 0x7f8e200cc190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:08.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:08.581+0000 7f8e1effd700 1 -- 172.21.15.16:0/3056809352 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4085251284 0 0) 0x7f8e200b2440 con 0x7f8e2015f060 2023-10-16T12:31:08.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:08.581+0000 7f8e1effd700 1 -- 172.21.15.16:0/3056809352 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e080035b0 con 0x7f8e2015f060 2023-10-16T12:31:08.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:08.581+0000 7f8e1effd700 1 -- 172.21.15.16:0/3056809352 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2681478701 0 0) 0x7f8e080035b0 con 0x7f8e2015f060 2023-10-16T12:31:08.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:08.581+0000 7f8e1f7fe700 1 -- 172.21.15.16:0/3056809352 >> v1:172.21.15.49:6789/0 conn(0x7f8e2015ec80 legacy=0x7f8e200cdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:08.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:08.581+0000 7f8e1f7fe700 1 -- 172.21.15.16:0/3056809352 >> v1:172.21.15.16:6789/0 conn(0x7f8e2015f060 legacy=0x7f8e200cc6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:08.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:08.581+0000 7f8e1f7fe700 1 -- 172.21.15.16:0/3056809352 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e200cef60 con 0x7f8e200ce5d0 2023-10-16T12:31:08.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:08.582+0000 7f8e1f7fe700 1 --2- 172.21.15.16:0/3056809352 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8e200ce5d0 0x7f8e200cc190 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f8e14009870 tx=0x7f8e14009b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1dfebe9d072dd0cd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:08.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:08.582+0000 7f8e1effd700 1 -- 172.21.15.16:0/3056809352 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e180019f0 con 0x7f8e200ce5d0 2023-10-16T12:31:08.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:08.582+0000 7f8e1effd700 1 -- 172.21.15.16:0/3056809352 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e14004030 con 0x7f8e200ce5d0 2023-10-16T12:31:08.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:08.582+0000 7f8e1effd700 1 -- 172.21.15.16:0/3056809352 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e140043c0 con 0x7f8e200ce5d0 2023-10-16T12:31:08.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:08.582+0000 7f8e268d0700 1 -- 172.21.15.16:0/3056809352 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8e200ce5d0 msgr2=0x7f8e200cc190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:08.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:08.582+0000 7f8e268d0700 1 --2- 172.21.15.16:0/3056809352 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8e200ce5d0 0x7f8e200cc190 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f8e14009870 tx=0x7f8e14009b80 comp rx=0 tx=0).stop 2023-10-16T12:31:08.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:08.582+0000 7f8e268d0700 1 -- 172.21.15.16:0/3056809352 shutdown_connections 2023-10-16T12:31:08.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:08.582+0000 7f8e268d0700 1 --2- 172.21.15.16:0/3056809352 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8e200ce5d0 0x7f8e200cc190 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:08.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:08.582+0000 7f8e268d0700 1 -- 172.21.15.16:0/3056809352 >> 172.21.15.16:0/3056809352 conn(0x7f8e200d15f0 msgr2=0x7f8e200d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:31:08.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:08.582+0000 7f8e268d0700 1 -- 172.21.15.16:0/3056809352 shutdown_connections 2023-10-16T12:31:08.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:08.582+0000 7f8e268d0700 1 -- 172.21.15.16:0/3056809352 wait complete. 2023-10-16T12:31:08.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:08.583+0000 7f8e268d0700 1 Processor -- start 2023-10-16T12:31:08.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:08.584+0000 7f8e268d0700 1 -- start start 2023-10-16T12:31:08.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:08.584+0000 7f8e268d0700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8e200ce5d0 0x7f8e2015c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:08.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:08.584+0000 7f8e268d0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e2015ec80 0x7f8e20151640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:08.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:08.584+0000 7f8e268d0700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8e2015f060 0x7f8e201e4660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:08.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:08.584+0000 7f8e268d0700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e200cb8d0 con 0x7f8e2015ec80 2023-10-16T12:31:08.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:08.584+0000 7f8e268d0700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e20151b80 con 0x7f8e200ce5d0 2023-10-16T12:31:08.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:08.584+0000 7f8e268d0700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e201e60a0 con 0x7f8e2015f060 2023-10-16T12:31:08.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:08.585+0000 7f8e1f7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e2015ec80 0x7f8e20151640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:08.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:08.585+0000 7f8e1f7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e2015ec80 0x7f8e20151640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46912/0 (socket says 172.21.15.16:46912) 2023-10-16T12:31:08.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:08.585+0000 7f8e1f7fe700 1 -- 172.21.15.16:0/2136944226 learned_addr learned my addr 172.21.15.16:0/2136944226 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:31:08.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:08.585+0000 7f8e24e6d700 1 --2- 172.21.15.16:0/2136944226 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8e2015f060 0x7f8e201e4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:08.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:08.585+0000 7f8e1ffff700 1 --2- 172.21.15.16:0/2136944226 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8e200ce5d0 0x7f8e2015c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:08.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:08.585+0000 7f8e1f7fe700 1 -- 172.21.15.16:0/2136944226 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8e2015f060 msgr2=0x7f8e201e4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:08.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:08.585+0000 7f8e1f7fe700 1 --2- 172.21.15.16:0/2136944226 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8e2015f060 0x7f8e201e4660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:08.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:08.585+0000 7f8e1f7fe700 1 -- 172.21.15.16:0/2136944226 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8e200ce5d0 msgr2=0x7f8e2015c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:08.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:08.585+0000 7f8e1f7fe700 1 --2- 172.21.15.16:0/2136944226 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8e200ce5d0 0x7f8e2015c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:08.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:08.585+0000 7f8e1f7fe700 1 -- 172.21.15.16:0/2136944226 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e140094a0 con 0x7f8e2015ec80 2023-10-16T12:31:08.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:08.585+0000 7f8e1ffff700 1 --2- 172.21.15.16:0/2136944226 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8e200ce5d0 0x7f8e2015c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:31:08.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:08.585+0000 7f8e24e6d700 1 --2- 172.21.15.16:0/2136944226 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8e2015f060 0x7f8e201e4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:31:08.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:08.585+0000 7f8e1f7fe700 1 --2- 172.21.15.16:0/2136944226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e2015ec80 0x7f8e20151640 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7f8e14000c00 tx=0x7f8e14004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:08.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:08.585+0000 7f8e1cff9700 1 -- 172.21.15.16:0/2136944226 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e14011830 con 0x7f8e2015ec80 2023-10-16T12:31:08.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:08.585+0000 7f8e1cff9700 1 -- 172.21.15.16:0/2136944226 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e1401e070 con 0x7f8e2015ec80 2023-10-16T12:31:08.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:08.585+0000 7f8e268d0700 1 -- 172.21.15.16:0/2136944226 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e200cb310 con 0x7f8e2015ec80 2023-10-16T12:31:08.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:08.585+0000 7f8e1cff9700 1 -- 172.21.15.16:0/2136944226 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e14011b90 con 0x7f8e2015ec80 2023-10-16T12:31:08.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:08.585+0000 7f8e268d0700 1 -- 172.21.15.16:0/2136944226 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e200c5230 con 0x7f8e2015ec80 2023-10-16T12:31:08.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:08.586+0000 7f8e268d0700 1 -- 172.21.15.16:0/2136944226 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e2004ebb0 con 0x7f8e2015ec80 2023-10-16T12:31:08.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:08.587+0000 7f8e1cff9700 1 -- 172.21.15.16:0/2136944226 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 85) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8e14011cf0 con 0x7f8e2015ec80 2023-10-16T12:31:08.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:08.587+0000 7f8e1cff9700 1 --2- 172.21.15.16:0/2136944226 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f8e0c06b020 0x7f8e0c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:08.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:08.587+0000 7f8e1cff9700 1 -- 172.21.15.16:0/2136944226 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 6085+0+0 (secure 0 0 0) 0x7f8e1408a710 con 0x7f8e2015ec80 2023-10-16T12:31:08.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:08.587+0000 7f8e1ffff700 1 --2- 172.21.15.16:0/2136944226 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f8e0c06b020 0x7f8e0c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:08.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:08.588+0000 7f8e1ffff700 1 --2- 172.21.15.16:0/2136944226 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f8e0c06b020 0x7f8e0c06d4e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f8e10000f60 tx=0x7f8e10009040 comp rx=0 tx=0).ready entity=mgr.8973 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:08.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:08.592+0000 7f8e1cff9700 1 -- 172.21.15.16:0/2136944226 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8e140555c0 con 0x7f8e2015ec80 2023-10-16T12:31:08.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:08.843+0000 7f8e268d0700 1 -- 172.21.15.16:0/2136944226 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f8e2006fa60 con 0x7f8e2015ec80 2023-10-16T12:31:09.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.430+0000 7f8e1cff9700 1 -- 172.21.15.16:0/2136944226 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v78) v1 ==== 122+0+0 (secure 0 0 0) 0x7f8e140200c0 con 0x7f8e2015ec80 2023-10-16T12:31:09.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.433+0000 7f8e268d0700 1 -- 172.21.15.16:0/2136944226 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f8e0c06b020 msgr2=0x7f8e0c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:09.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.433+0000 7f8e268d0700 1 --2- 172.21.15.16:0/2136944226 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f8e0c06b020 0x7f8e0c06d4e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f8e10000f60 tx=0x7f8e10009040 comp rx=0 tx=0).stop 2023-10-16T12:31:09.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.433+0000 7f8e268d0700 1 -- 172.21.15.16:0/2136944226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e2015ec80 msgr2=0x7f8e20151640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:09.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.433+0000 7f8e268d0700 1 --2- 172.21.15.16:0/2136944226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e2015ec80 0x7f8e20151640 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7f8e14000c00 tx=0x7f8e14004750 comp rx=0 tx=0).stop 2023-10-16T12:31:09.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.433+0000 7f8e268d0700 1 -- 172.21.15.16:0/2136944226 shutdown_connections 2023-10-16T12:31:09.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.433+0000 7f8e268d0700 1 --2- 172.21.15.16:0/2136944226 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f8e0c06b020 0x7f8e0c06d4e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:09.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.433+0000 7f8e268d0700 1 --2- 172.21.15.16:0/2136944226 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8e2015f060 0x7f8e201e4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:09.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.433+0000 7f8e268d0700 1 --2- 172.21.15.16:0/2136944226 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8e200ce5d0 0x7f8e2015c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:09.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.433+0000 7f8e268d0700 1 --2- 172.21.15.16:0/2136944226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e2015ec80 0x7f8e20151640 unknown :-1 s=CLOSED pgs=1745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:09.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.433+0000 7f8e268d0700 1 -- 172.21.15.16:0/2136944226 >> 172.21.15.16:0/2136944226 conn(0x7f8e200d15f0 msgr2=0x7f8e200b4690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:31:09.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.434+0000 7f8e268d0700 1 -- 172.21.15.16:0/2136944226 shutdown_connections 2023-10-16T12:31:09.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.434+0000 7f8e268d0700 1 -- 172.21.15.16:0/2136944226 wait complete. 2023-10-16T12:31:09.450 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-16T12:31:09.451 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:31:09.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.580+0000 7f49b457c700 1 Processor -- start 2023-10-16T12:31:09.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.581+0000 7f49b457c700 1 -- start start 2023-10-16T12:31:09.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.582+0000 7f49b457c700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f49ac0b2b50 0x7f49ac0d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:09.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.582+0000 7f49b457c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f49ac0ce380 0x7f49ac0d09e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:09.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.582+0000 7f49b457c700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49ac0d1210 con 0x7f49ac0cdfa0 2023-10-16T12:31:09.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.582+0000 7f49b457c700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49ac0cbc30 con 0x7f49ac0b2b50 2023-10-16T12:31:09.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.582+0000 7f49b457c700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49ac0cbf10 con 0x7f49ac0ce380 2023-10-16T12:31:09.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.582+0000 7f49b2318700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f49ac0cdfa0 0x7f49ac0b2440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36192/0 (socket says 172.21.15.16:36192) 2023-10-16T12:31:09.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.582+0000 7f49b2318700 1 -- 172.21.15.16:0/990656818 learned_addr learned my addr 172.21.15.16:0/990656818 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:31:09.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.582+0000 7f49b1b17700 1 --2- 172.21.15.16:0/990656818 >> v2:172.21.15.49:3300/0 conn(0x7f49ac0b2b50 0x7f49ac0d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:09.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.582+0000 7f49b2b19700 1 --2- 172.21.15.16:0/990656818 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f49ac0ce380 0x7f49ac0d09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:09.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.582+0000 7f49b1316700 1 -- 172.21.15.16:0/990656818 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1689751485 0 0) 0x7f49ac0d1210 con 0x7f49ac0cdfa0 2023-10-16T12:31:09.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.582+0000 7f49b1316700 1 -- 172.21.15.16:0/990656818 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49980035b0 con 0x7f49ac0cdfa0 2023-10-16T12:31:09.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.582+0000 7f49b2b19700 1 -- 172.21.15.16:0/990656818 >> v2:172.21.15.49:3300/0 conn(0x7f49ac0b2b50 msgr2=0x7f49ac0d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:31:09.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.582+0000 7f49b2b19700 1 --2- 172.21.15.16:0/990656818 >> v2:172.21.15.49:3300/0 conn(0x7f49ac0b2b50 0x7f49ac0d04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:09.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.582+0000 7f49b2b19700 1 -- 172.21.15.16:0/990656818 >> v1:172.21.15.16:6789/0 conn(0x7f49ac0cdfa0 legacy=0x7f49ac0b2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:09.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.582+0000 7f49b2b19700 1 -- 172.21.15.16:0/990656818 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49ac0cc1f0 con 0x7f49ac0ce380 2023-10-16T12:31:09.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.582+0000 7f49b1b17700 1 --2- 172.21.15.16:0/990656818 >> v2:172.21.15.49:3300/0 conn(0x7f49ac0b2b50 0x7f49ac0d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:31:09.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.582+0000 7f49b2b19700 1 --2- 172.21.15.16:0/990656818 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f49ac0ce380 0x7f49ac0d09e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f49a4009d90 tx=0x7f49a400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=62a3f8366b6acd7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:09.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.583+0000 7f49b1316700 1 -- 172.21.15.16:0/990656818 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a400e070 con 0x7f49ac0ce380 2023-10-16T12:31:09.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.583+0000 7f49b1316700 1 -- 172.21.15.16:0/990656818 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49a400bb20 con 0x7f49ac0ce380 2023-10-16T12:31:09.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.583+0000 7f49b1316700 1 -- 172.21.15.16:0/990656818 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a4012430 con 0x7f49ac0ce380 2023-10-16T12:31:09.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.583+0000 7f49b457c700 1 -- 172.21.15.16:0/990656818 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f49ac0ce380 msgr2=0x7f49ac0d09e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:09.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.583+0000 7f49b457c700 1 --2- 172.21.15.16:0/990656818 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f49ac0ce380 0x7f49ac0d09e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f49a4009d90 tx=0x7f49a400b330 comp rx=0 tx=0).stop 2023-10-16T12:31:09.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.583+0000 7f49b457c700 1 -- 172.21.15.16:0/990656818 shutdown_connections 2023-10-16T12:31:09.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.583+0000 7f49b457c700 1 --2- 172.21.15.16:0/990656818 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f49ac0ce380 0x7f49ac0d09e0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:09.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.583+0000 7f49b457c700 1 --2- 172.21.15.16:0/990656818 >> v2:172.21.15.49:3300/0 conn(0x7f49ac0b2b50 0x7f49ac0d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:09.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.583+0000 7f49b457c700 1 -- 172.21.15.16:0/990656818 >> 172.21.15.16:0/990656818 conn(0x7f49ac0d14b0 msgr2=0x7f49ac0d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:31:09.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.583+0000 7f49b457c700 1 -- 172.21.15.16:0/990656818 shutdown_connections 2023-10-16T12:31:09.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.583+0000 7f49b457c700 1 -- 172.21.15.16:0/990656818 wait complete. 2023-10-16T12:31:09.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.584+0000 7f49b457c700 1 Processor -- start 2023-10-16T12:31:09.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.585+0000 7f49b457c700 1 -- start start 2023-10-16T12:31:09.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.585+0000 7f49b457c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f49ac0b2b50 0x7f49ac09ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:09.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.585+0000 7f49b457c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49ac0cdfa0 0x7f49ac160510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:09.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.585+0000 7f49b457c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f49ac0ce380 0x7f49ac160a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:09.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.585+0000 7f49b457c700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49ac0c9610 con 0x7f49ac0cdfa0 2023-10-16T12:31:09.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.585+0000 7f49b457c700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49ac0c7eb0 con 0x7f49ac0ce380 2023-10-16T12:31:09.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.585+0000 7f49b457c700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49ac0c8cb0 con 0x7f49ac0b2b50 2023-10-16T12:31:09.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.585+0000 7f49b1b17700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49ac0cdfa0 0x7f49ac160510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:09.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.585+0000 7f49b1b17700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49ac0cdfa0 0x7f49ac160510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46924/0 (socket says 172.21.15.16:46924) 2023-10-16T12:31:09.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.585+0000 7f49b1b17700 1 -- 172.21.15.16:0/2945423683 learned_addr learned my addr 172.21.15.16:0/2945423683 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:31:09.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.585+0000 7f49b2318700 1 --2- 172.21.15.16:0/2945423683 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f49ac0b2b50 0x7f49ac09ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:09.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.586+0000 7f49b2b19700 1 --2- 172.21.15.16:0/2945423683 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f49ac0ce380 0x7f49ac160a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:09.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.586+0000 7f49b1b17700 1 -- 172.21.15.16:0/2945423683 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f49ac0b2b50 msgr2=0x7f49ac09ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:09.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.586+0000 7f49b1b17700 1 --2- 172.21.15.16:0/2945423683 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f49ac0b2b50 0x7f49ac09ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:09.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.586+0000 7f49b1b17700 1 -- 172.21.15.16:0/2945423683 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f49ac0ce380 msgr2=0x7f49ac160a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:09.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.586+0000 7f49b1b17700 1 --2- 172.21.15.16:0/2945423683 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f49ac0ce380 0x7f49ac160a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:09.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.586+0000 7f49b1b17700 1 -- 172.21.15.16:0/2945423683 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49a00094a0 con 0x7f49ac0cdfa0 2023-10-16T12:31:09.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.586+0000 7f49b1b17700 1 --2- 172.21.15.16:0/2945423683 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49ac0cdfa0 0x7f49ac160510 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7f49a000ebe0 tx=0x7f49a0009ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:09.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.586+0000 7f499effd700 1 -- 172.21.15.16:0/2945423683 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a000c910 con 0x7f49ac0cdfa0 2023-10-16T12:31:09.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.586+0000 7f499effd700 1 -- 172.21.15.16:0/2945423683 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49a000ca70 con 0x7f49ac0cdfa0 2023-10-16T12:31:09.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.586+0000 7f499effd700 1 -- 172.21.15.16:0/2945423683 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a000cde0 con 0x7f49ac0cdfa0 2023-10-16T12:31:09.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.586+0000 7f49b457c700 1 -- 172.21.15.16:0/2945423683 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49a4009a70 con 0x7f49ac0cdfa0 2023-10-16T12:31:09.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.586+0000 7f49b457c700 1 -- 172.21.15.16:0/2945423683 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49ac0c9230 con 0x7f49ac0cdfa0 2023-10-16T12:31:09.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.587+0000 7f49b457c700 1 -- 172.21.15.16:0/2945423683 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49ac0c9750 con 0x7f49ac0cdfa0 2023-10-16T12:31:09.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.588+0000 7f499effd700 1 -- 172.21.15.16:0/2945423683 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 85) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f49a0013470 con 0x7f49ac0cdfa0 2023-10-16T12:31:09.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.588+0000 7f499effd700 1 --2- 172.21.15.16:0/2945423683 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f499006b020 0x7f499006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:09.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.588+0000 7f499effd700 1 -- 172.21.15.16:0/2945423683 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 6085+0+0 (secure 0 0 0) 0x7f49a0089a90 con 0x7f49ac0cdfa0 2023-10-16T12:31:09.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.588+0000 7f49b2318700 1 --2- 172.21.15.16:0/2945423683 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f499006b020 0x7f499006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:09.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.588+0000 7f49b2318700 1 --2- 172.21.15.16:0/2945423683 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f499006b020 0x7f499006d4e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f49a80011e0 tx=0x7f49a8009040 comp rx=0 tx=0).ready entity=mgr.8973 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:09.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.593+0000 7f499effd700 1 -- 172.21.15.16:0/2945423683 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f49ac0c9750 con 0x7f49ac0cdfa0 2023-10-16T12:31:09.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.840+0000 7f49b457c700 1 -- 172.21.15.16:0/2945423683 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f49ac0c9750 con 0x7f49ac0cdfa0 2023-10-16T12:31:09.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.841+0000 7f499effd700 1 -- 172.21.15.16:0/2945423683 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+4915 (secure 0 0 0) 0x7f49ac0c9750 con 0x7f49ac0cdfa0 2023-10-16T12:31:09.843 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:31:09.843 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":78,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7631,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/3370728164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":3370728164},{"type":"v1","addr":"172.21.15.49:6834","nonce":3370728164}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":73},{"gid":8638,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2297042963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":2297042963},{"type":"v1","addr":"172.21.15.16:6837","nonce":2297042963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":73},{"gid":8647,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/1920044955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":1920044955},{"type":"v1","addr":"172.21.15.49:6835","nonce":1920044955}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":73}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:31:05.836924+0000","modified":"2023-10-16T12:31:08.845101+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9204":{"gid":9204,"name":"c","rank":0,"incarnation":74,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6836/1477717813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1477717813},{"type":"v1","addr":"172.21.15.16:6836","nonce":1477717813}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2023-10-16T12:31:09.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.843+0000 7f49b457c700 1 -- 172.21.15.16:0/2945423683 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f499006b020 msgr2=0x7f499006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:09.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.843+0000 7f49b457c700 1 --2- 172.21.15.16:0/2945423683 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f499006b020 0x7f499006d4e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f49a80011e0 tx=0x7f49a8009040 comp rx=0 tx=0).stop 2023-10-16T12:31:09.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.843+0000 7f49b457c700 1 -- 172.21.15.16:0/2945423683 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49ac0cdfa0 msgr2=0x7f49ac160510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:09.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.843+0000 7f49b457c700 1 --2- 172.21.15.16:0/2945423683 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49ac0cdfa0 0x7f49ac160510 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7f49a000ebe0 tx=0x7f49a0009ec0 comp rx=0 tx=0).stop 2023-10-16T12:31:09.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.843+0000 7f49b457c700 1 -- 172.21.15.16:0/2945423683 shutdown_connections 2023-10-16T12:31:09.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.843+0000 7f49b457c700 1 --2- 172.21.15.16:0/2945423683 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f499006b020 0x7f499006d4e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:09.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.843+0000 7f49b457c700 1 --2- 172.21.15.16:0/2945423683 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f49ac0ce380 0x7f49ac160a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:09.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.843+0000 7f49b457c700 1 --2- 172.21.15.16:0/2945423683 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f49ac0b2b50 0x7f49ac09ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:09.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.843+0000 7f49b457c700 1 --2- 172.21.15.16:0/2945423683 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49ac0cdfa0 0x7f49ac160510 unknown :-1 s=CLOSED pgs=1747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:09.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.843+0000 7f49b457c700 1 -- 172.21.15.16:0/2945423683 >> 172.21.15.16:0/2945423683 conn(0x7f49ac0d14b0 msgr2=0x7f49ac06c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:31:09.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.843+0000 7f49b457c700 1 -- 172.21.15.16:0/2945423683 shutdown_connections 2023-10-16T12:31:09.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.843+0000 7f49b457c700 1 -- 172.21.15.16:0/2945423683 wait complete. 2023-10-16T12:31:09.848 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 78 2023-10-16T12:31:09.857 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T12:31:09.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.980+0000 7fa7066bb700 1 Processor -- start 2023-10-16T12:31:09.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.980+0000 7fa7066bb700 1 -- start start 2023-10-16T12:31:09.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.981+0000 7fa7066bb700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fa70006aef0 0x7fa7000b3480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:09.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.981+0000 7fa7066bb700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa7000b2be0 0x7fa7000b39c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:09.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.981+0000 7fa7066bb700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7000d11c0 con 0x7fa70015b860 2023-10-16T12:31:09.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.981+0000 7fa7066bb700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7000d03d0 con 0x7fa70006aef0 2023-10-16T12:31:09.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.981+0000 7fa7066bb700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7000cd360 con 0x7fa7000b2be0 2023-10-16T12:31:09.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.981+0000 7fa6fffff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa70015b860 0x7fa70006a7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36204/0 (socket says 172.21.15.16:36204) 2023-10-16T12:31:09.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.981+0000 7fa6fffff700 1 -- 172.21.15.16:0/931657837 learned_addr learned my addr 172.21.15.16:0/931657837 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:31:09.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.981+0000 7fa704c58700 1 --2- 172.21.15.16:0/931657837 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa7000b2be0 0x7fa7000b39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:09.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.981+0000 7fa6ff7fe700 1 --2- 172.21.15.16:0/931657837 >> v2:172.21.15.49:3300/0 conn(0x7fa70006aef0 0x7fa7000b3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:09.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.981+0000 7fa6feffd700 1 -- 172.21.15.16:0/931657837 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4052179092 0 0) 0x7fa7000d11c0 con 0x7fa70015b860 2023-10-16T12:31:09.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.981+0000 7fa6feffd700 1 -- 172.21.15.16:0/931657837 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6ec0035b0 con 0x7fa70015b860 2023-10-16T12:31:09.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.981+0000 7fa704c58700 1 -- 172.21.15.16:0/931657837 >> v2:172.21.15.49:3300/0 conn(0x7fa70006aef0 msgr2=0x7fa7000b3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:31:09.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.981+0000 7fa704c58700 1 --2- 172.21.15.16:0/931657837 >> v2:172.21.15.49:3300/0 conn(0x7fa70006aef0 0x7fa7000b3480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:09.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.981+0000 7fa6feffd700 1 -- 172.21.15.16:0/931657837 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3030682272 0 0) 0x7fa6ec0035b0 con 0x7fa70015b860 2023-10-16T12:31:09.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.981+0000 7fa704c58700 1 -- 172.21.15.16:0/931657837 >> v1:172.21.15.16:6789/0 conn(0x7fa70015b860 legacy=0x7fa70006a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:09.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.981+0000 7fa704c58700 1 -- 172.21.15.16:0/931657837 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7000d05b0 con 0x7fa7000b2be0 2023-10-16T12:31:09.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.981+0000 7fa6ff7fe700 1 --2- 172.21.15.16:0/931657837 >> v2:172.21.15.49:3300/0 conn(0x7fa70006aef0 0x7fa7000b3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:31:09.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.982+0000 7fa704c58700 1 --2- 172.21.15.16:0/931657837 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa7000b2be0 0x7fa7000b39c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa6f8009d90 tx=0x7fa6f800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca306fcd3b399824 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:09.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.982+0000 7fa6feffd700 1 -- 172.21.15.16:0/931657837 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6f800e070 con 0x7fa7000b2be0 2023-10-16T12:31:09.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.982+0000 7fa6feffd700 1 -- 172.21.15.16:0/931657837 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6f8003a20 con 0x7fa7000b2be0 2023-10-16T12:31:09.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.982+0000 7fa6feffd700 1 -- 172.21.15.16:0/931657837 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6e8001670 con 0x7fa7000b2be0 2023-10-16T12:31:09.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.982+0000 7fa7066bb700 1 -- 172.21.15.16:0/931657837 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa7000b2be0 msgr2=0x7fa7000b39c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:09.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.982+0000 7fa7066bb700 1 --2- 172.21.15.16:0/931657837 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa7000b2be0 0x7fa7000b39c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa6f8009d90 tx=0x7fa6f800b330 comp rx=0 tx=0).stop 2023-10-16T12:31:09.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.982+0000 7fa7066bb700 1 -- 172.21.15.16:0/931657837 shutdown_connections 2023-10-16T12:31:09.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.982+0000 7fa7066bb700 1 --2- 172.21.15.16:0/931657837 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa7000b2be0 0x7fa7000b39c0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:09.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.982+0000 7fa7066bb700 1 --2- 172.21.15.16:0/931657837 >> v2:172.21.15.49:3300/0 conn(0x7fa70006aef0 0x7fa7000b3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:09.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.982+0000 7fa7066bb700 1 -- 172.21.15.16:0/931657837 >> 172.21.15.16:0/931657837 conn(0x7fa7000b1fd0 msgr2=0x7fa7000b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:31:09.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.982+0000 7fa7066bb700 1 -- 172.21.15.16:0/931657837 shutdown_connections 2023-10-16T12:31:09.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.982+0000 7fa7066bb700 1 -- 172.21.15.16:0/931657837 wait complete. 2023-10-16T12:31:09.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.984+0000 7fa7066bb700 1 Processor -- start 2023-10-16T12:31:09.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.984+0000 7fa7066bb700 1 -- start start 2023-10-16T12:31:09.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.984+0000 7fa7066bb700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa70006aef0 0x7fa700149a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:09.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.984+0000 7fa7066bb700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7000b2be0 0x7fa70014e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:09.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.984+0000 7fa7066bb700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa70015b860 0x7fa70014eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:09.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.984+0000 7fa7066bb700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa700149fd0 con 0x7fa7000b2be0 2023-10-16T12:31:09.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.984+0000 7fa7066bb700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa70014f700 con 0x7fa70015b860 2023-10-16T12:31:09.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.984+0000 7fa7066bb700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa70014f9e0 con 0x7fa70006aef0 2023-10-16T12:31:09.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.985+0000 7fa6ff7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7000b2be0 0x7fa70014e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:09.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.985+0000 7fa6ff7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7000b2be0 0x7fa70014e960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46938/0 (socket says 172.21.15.16:46938) 2023-10-16T12:31:09.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.985+0000 7fa6ff7fe700 1 -- 172.21.15.16:0/3941241588 learned_addr learned my addr 172.21.15.16:0/3941241588 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:31:09.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.985+0000 7fa704c58700 1 --2- 172.21.15.16:0/3941241588 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa70015b860 0x7fa70014eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:09.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.985+0000 7fa6fffff700 1 --2- 172.21.15.16:0/3941241588 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa70006aef0 0x7fa700149a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:09.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.985+0000 7fa6ff7fe700 1 -- 172.21.15.16:0/3941241588 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa70006aef0 msgr2=0x7fa700149a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:09.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.985+0000 7fa6ff7fe700 1 --2- 172.21.15.16:0/3941241588 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa70006aef0 0x7fa700149a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:09.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.985+0000 7fa6ff7fe700 1 -- 172.21.15.16:0/3941241588 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa70015b860 msgr2=0x7fa70014eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:09.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.985+0000 7fa6ff7fe700 1 --2- 172.21.15.16:0/3941241588 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa70015b860 0x7fa70014eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:09.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.985+0000 7fa6ff7fe700 1 -- 172.21.15.16:0/3941241588 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6f00094a0 con 0x7fa7000b2be0 2023-10-16T12:31:09.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.985+0000 7fa6ff7fe700 1 --2- 172.21.15.16:0/3941241588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7000b2be0 0x7fa70014e960 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7fa6f000ebe0 tx=0x7fa6f000eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:09.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.985+0000 7fa6fcff9700 1 -- 172.21.15.16:0/3941241588 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6f0009760 con 0x7fa7000b2be0 2023-10-16T12:31:09.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.985+0000 7fa6fcff9700 1 -- 172.21.15.16:0/3941241588 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6f000c850 con 0x7fa7000b2be0 2023-10-16T12:31:09.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.985+0000 7fa7066bb700 1 -- 172.21.15.16:0/3941241588 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6f8009a70 con 0x7fa7000b2be0 2023-10-16T12:31:09.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.985+0000 7fa6fcff9700 1 -- 172.21.15.16:0/3941241588 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6f000cbe0 con 0x7fa7000b2be0 2023-10-16T12:31:09.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.985+0000 7fa7066bb700 1 -- 172.21.15.16:0/3941241588 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa70014ffa0 con 0x7fa7000b2be0 2023-10-16T12:31:09.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.986+0000 7fa7066bb700 1 -- 172.21.15.16:0/3941241588 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa70004ebb0 con 0x7fa7000b2be0 2023-10-16T12:31:09.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.987+0000 7fa6fcff9700 1 -- 172.21.15.16:0/3941241588 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 85) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa6f000cd40 con 0x7fa7000b2be0 2023-10-16T12:31:09.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.987+0000 7fa6fcff9700 1 --2- 172.21.15.16:0/3941241588 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7fa6e006b020 0x7fa6e006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:09.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.987+0000 7fa6fcff9700 1 -- 172.21.15.16:0/3941241588 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 6085+0+0 (secure 0 0 0) 0x7fa6f0089870 con 0x7fa7000b2be0 2023-10-16T12:31:09.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.987+0000 7fa6fffff700 1 --2- 172.21.15.16:0/3941241588 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7fa6e006b020 0x7fa6e006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:09.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.988+0000 7fa6fffff700 1 --2- 172.21.15.16:0/3941241588 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7fa6e006b020 0x7fa6e006d4e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa7000af5b0 tx=0x7fa6e8001e40 comp rx=0 tx=0).ready entity=mgr.8973 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:09.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:09.992+0000 7fa6fcff9700 1 -- 172.21.15.16:0/3941241588 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa6f0054720 con 0x7fa7000b2be0 2023-10-16T12:31:10.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.196+0000 7fa7066bb700 1 -- 172.21.15.16:0/3941241588 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa7000cb2e0 con 0x7fa7000b2be0 2023-10-16T12:31:10.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.197+0000 7fa6fcff9700 1 -- 172.21.15.16:0/3941241588 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v156) v1 ==== 74+0+16712 (secure 0 0 0) 0x7fa6f0021090 con 0x7fa7000b2be0 2023-10-16T12:31:10.199 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:31:10.199 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":156,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T12:31:07.334950+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:31:04.290489+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2023-10-16T12:31:04.803486+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"156","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T12:31:10.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.199+0000 7fa7066bb700 1 -- 172.21.15.16:0/3941241588 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7fa6e006b020 msgr2=0x7fa6e006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:10.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.199+0000 7fa7066bb700 1 --2- 172.21.15.16:0/3941241588 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7fa6e006b020 0x7fa6e006d4e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa7000af5b0 tx=0x7fa6e8001e40 comp rx=0 tx=0).stop 2023-10-16T12:31:10.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.199+0000 7fa7066bb700 1 -- 172.21.15.16:0/3941241588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7000b2be0 msgr2=0x7fa70014e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:10.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.199+0000 7fa7066bb700 1 --2- 172.21.15.16:0/3941241588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7000b2be0 0x7fa70014e960 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7fa6f000ebe0 tx=0x7fa6f000eef0 comp rx=0 tx=0).stop 2023-10-16T12:31:10.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.199+0000 7fa7066bb700 1 -- 172.21.15.16:0/3941241588 shutdown_connections 2023-10-16T12:31:10.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.199+0000 7fa7066bb700 1 --2- 172.21.15.16:0/3941241588 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7fa6e006b020 0x7fa6e006d4e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:10.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.199+0000 7fa7066bb700 1 --2- 172.21.15.16:0/3941241588 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa70015b860 0x7fa70014eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:10.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.199+0000 7fa7066bb700 1 --2- 172.21.15.16:0/3941241588 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa70006aef0 0x7fa700149a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:10.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.199+0000 7fa7066bb700 1 --2- 172.21.15.16:0/3941241588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7000b2be0 0x7fa70014e960 unknown :-1 s=CLOSED pgs=1749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:10.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.199+0000 7fa7066bb700 1 -- 172.21.15.16:0/3941241588 >> 172.21.15.16:0/3941241588 conn(0x7fa7000b1fd0 msgr2=0x7fa7000b40c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:31:10.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.199+0000 7fa7066bb700 1 -- 172.21.15.16:0/3941241588 shutdown_connections 2023-10-16T12:31:10.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.199+0000 7fa7066bb700 1 -- 172.21.15.16:0/3941241588 wait complete. 2023-10-16T12:31:10.214 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-16T12:31:10.214 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T12:31:10.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.337+0000 7f4a27df2700 1 Processor -- start 2023-10-16T12:31:10.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.337+0000 7f4a27df2700 1 -- start start 2023-10-16T12:31:10.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.338+0000 7f4a27df2700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f4a2015b860 0x7f4a2006a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:10.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.338+0000 7f4a27df2700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4a200b2be0 0x7f4a2006ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:10.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.338+0000 7f4a27df2700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a200d11c0 con 0x7f4a200cd770 2023-10-16T12:31:10.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.338+0000 7f4a27df2700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a200d03d0 con 0x7f4a2015b860 2023-10-16T12:31:10.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.338+0000 7f4a27df2700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a200cd360 con 0x7f4a200b2be0 2023-10-16T12:31:10.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.338+0000 7f4a2638f700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f4a200cd770 0x7f4a200719e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36220/0 (socket says 172.21.15.16:36220) 2023-10-16T12:31:10.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.338+0000 7f4a2638f700 1 -- 172.21.15.16:0/393883495 learned_addr learned my addr 172.21.15.16:0/393883495 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:31:10.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.338+0000 7f4a25b8e700 1 --2- 172.21.15.16:0/393883495 >> v2:172.21.15.49:3300/0 conn(0x7f4a2015b860 0x7f4a2006a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:10.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.338+0000 7f4a2538d700 1 --2- 172.21.15.16:0/393883495 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4a200b2be0 0x7f4a2006ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:10.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.338+0000 7f4a24b8c700 1 -- 172.21.15.16:0/393883495 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2782616164 0 0) 0x7f4a200d11c0 con 0x7f4a200cd770 2023-10-16T12:31:10.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.338+0000 7f4a24b8c700 1 -- 172.21.15.16:0/393883495 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a0c0035b0 con 0x7f4a200cd770 2023-10-16T12:31:10.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.339+0000 7f4a25b8e700 1 -- 172.21.15.16:0/393883495 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4a200b2be0 msgr2=0x7f4a2006ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:31:10.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.339+0000 7f4a25b8e700 1 --2- 172.21.15.16:0/393883495 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4a200b2be0 0x7f4a2006ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:10.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.339+0000 7f4a24b8c700 1 -- 172.21.15.16:0/393883495 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2079026676 0 0) 0x7f4a0c0035b0 con 0x7f4a200cd770 2023-10-16T12:31:10.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.339+0000 7f4a25b8e700 1 -- 172.21.15.16:0/393883495 >> v1:172.21.15.16:6789/0 conn(0x7f4a200cd770 legacy=0x7f4a200719e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:10.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.339+0000 7f4a25b8e700 1 -- 172.21.15.16:0/393883495 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a200d05b0 con 0x7f4a2015b860 2023-10-16T12:31:10.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.339+0000 7f4a25b8e700 1 --2- 172.21.15.16:0/393883495 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4a2015b860 0x7f4a2006a7e0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f4a10009840 tx=0x7f4a10009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8028ee187dad69f0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:10.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.339+0000 7f4a24b8c700 1 -- 172.21.15.16:0/393883495 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a180019f0 con 0x7f4a2015b860 2023-10-16T12:31:10.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.339+0000 7f4a24b8c700 1 -- 172.21.15.16:0/393883495 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a10004030 con 0x7f4a2015b860 2023-10-16T12:31:10.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.339+0000 7f4a24b8c700 1 -- 172.21.15.16:0/393883495 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a100043c0 con 0x7f4a2015b860 2023-10-16T12:31:10.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.340+0000 7f4a27df2700 1 -- 172.21.15.16:0/393883495 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4a2015b860 msgr2=0x7f4a2006a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:10.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.340+0000 7f4a27df2700 1 --2- 172.21.15.16:0/393883495 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4a2015b860 0x7f4a2006a7e0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f4a10009840 tx=0x7f4a10009b50 comp rx=0 tx=0).stop 2023-10-16T12:31:10.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.340+0000 7f4a27df2700 1 -- 172.21.15.16:0/393883495 shutdown_connections 2023-10-16T12:31:10.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.340+0000 7f4a27df2700 1 --2- 172.21.15.16:0/393883495 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4a2015b860 0x7f4a2006a7e0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:10.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.340+0000 7f4a27df2700 1 --2- 172.21.15.16:0/393883495 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4a200b2be0 0x7f4a2006ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:10.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.340+0000 7f4a27df2700 1 -- 172.21.15.16:0/393883495 >> 172.21.15.16:0/393883495 conn(0x7f4a200b1fd0 msgr2=0x7f4a200b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:31:10.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.340+0000 7f4a27df2700 1 -- 172.21.15.16:0/393883495 shutdown_connections 2023-10-16T12:31:10.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.340+0000 7f4a27df2700 1 -- 172.21.15.16:0/393883495 wait complete. 2023-10-16T12:31:10.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.341+0000 7f4a27df2700 1 Processor -- start 2023-10-16T12:31:10.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.341+0000 7f4a27df2700 1 -- start start 2023-10-16T12:31:10.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.342+0000 7f4a27df2700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4a200b2be0 0x7f4a200adaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:10.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.342+0000 7f4a27df2700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a200cd770 0x7f4a200b43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:10.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.342+0000 7f4a27df2700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4a2015b860 0x7f4a201f3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:10.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.342+0000 7f4a27df2700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a20151450 con 0x7f4a200cd770 2023-10-16T12:31:10.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.342+0000 7f4a27df2700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a200acae0 con 0x7f4a200b2be0 2023-10-16T12:31:10.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.342+0000 7f4a27df2700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a200acd30 con 0x7f4a2015b860 2023-10-16T12:31:10.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.342+0000 7f4a2538d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a200cd770 0x7f4a200b43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:10.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.342+0000 7f4a2538d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a200cd770 0x7f4a200b43e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46950/0 (socket says 172.21.15.16:46950) 2023-10-16T12:31:10.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.342+0000 7f4a2538d700 1 -- 172.21.15.16:0/3367912029 learned_addr learned my addr 172.21.15.16:0/3367912029 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:31:10.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.342+0000 7f4a2638f700 1 --2- 172.21.15.16:0/3367912029 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4a2015b860 0x7f4a201f3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:10.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.342+0000 7f4a25b8e700 1 --2- 172.21.15.16:0/3367912029 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4a200b2be0 0x7f4a200adaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:10.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.342+0000 7f4a2538d700 1 -- 172.21.15.16:0/3367912029 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4a2015b860 msgr2=0x7f4a201f3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:10.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.342+0000 7f4a2538d700 1 --2- 172.21.15.16:0/3367912029 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4a2015b860 0x7f4a201f3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:10.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.342+0000 7f4a2538d700 1 -- 172.21.15.16:0/3367912029 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4a200b2be0 msgr2=0x7f4a200adaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:10.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.342+0000 7f4a2538d700 1 --2- 172.21.15.16:0/3367912029 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4a200b2be0 0x7f4a200adaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:10.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.342+0000 7f4a2538d700 1 -- 172.21.15.16:0/3367912029 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a100094a0 con 0x7f4a200cd770 2023-10-16T12:31:10.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.342+0000 7f4a2538d700 1 --2- 172.21.15.16:0/3367912029 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a200cd770 0x7f4a200b43e0 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7f4a1c00e870 tx=0x7f4a1c00ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:10.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.342+0000 7f4a167fc700 1 -- 172.21.15.16:0/3367912029 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a1c009590 con 0x7f4a200cd770 2023-10-16T12:31:10.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.342+0000 7f4a167fc700 1 -- 172.21.15.16:0/3367912029 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a1c00ee70 con 0x7f4a200cd770 2023-10-16T12:31:10.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.342+0000 7f4a27df2700 1 -- 172.21.15.16:0/3367912029 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a200ad010 con 0x7f4a200cd770 2023-10-16T12:31:10.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.342+0000 7f4a167fc700 1 -- 172.21.15.16:0/3367912029 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a1c00ca50 con 0x7f4a200cd770 2023-10-16T12:31:10.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.343+0000 7f4a27df2700 1 -- 172.21.15.16:0/3367912029 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a200ad5d0 con 0x7f4a200cd770 2023-10-16T12:31:10.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.343+0000 7f4a27df2700 1 -- 172.21.15.16:0/3367912029 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a2004ebb0 con 0x7f4a200cd770 2023-10-16T12:31:10.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.344+0000 7f4a167fc700 1 -- 172.21.15.16:0/3367912029 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 85) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4a1c00cbb0 con 0x7f4a200cd770 2023-10-16T12:31:10.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.344+0000 7f4a167fc700 1 --2- 172.21.15.16:0/3367912029 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f4a0406b020 0x7f4a0406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:10.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.344+0000 7f4a167fc700 1 -- 172.21.15.16:0/3367912029 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 6085+0+0 (secure 0 0 0) 0x7f4a1c089a10 con 0x7f4a200cd770 2023-10-16T12:31:10.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.344+0000 7f4a25b8e700 1 --2- 172.21.15.16:0/3367912029 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f4a0406b020 0x7f4a0406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:10.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.345+0000 7f4a25b8e700 1 --2- 172.21.15.16:0/3367912029 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f4a0406b020 0x7f4a0406d4e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4a100090b0 tx=0x7f4a10008030 comp rx=0 tx=0).ready entity=mgr.8973 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:10.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.349+0000 7f4a167fc700 1 -- 172.21.15.16:0/3367912029 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4a1c054700 con 0x7f4a200cd770 2023-10-16T12:31:10.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.549+0000 7f4a27df2700 1 -- 172.21.15.16:0/3367912029 --> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4a200d0b70 con 0x7f4a0406b020 2023-10-16T12:31:10.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.558+0000 7f4a167fc700 1 -- 172.21.15.16:0/3367912029 <== mgr.8973 v2:172.21.15.16:6832/4182163087 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289846 (secure 0 0 0) 0x7f4a100044a0 con 0x7f4a0406b020 2023-10-16T12:31:10.563 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:31:10.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.561+0000 7f4a27df2700 1 -- 172.21.15.16:0/3367912029 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f4a0406b020 msgr2=0x7f4a0406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:10.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.561+0000 7f4a27df2700 1 --2- 172.21.15.16:0/3367912029 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f4a0406b020 0x7f4a0406d4e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4a100090b0 tx=0x7f4a10008030 comp rx=0 tx=0).stop 2023-10-16T12:31:10.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.561+0000 7f4a27df2700 1 -- 172.21.15.16:0/3367912029 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a200cd770 msgr2=0x7f4a200b43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:10.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.561+0000 7f4a27df2700 1 --2- 172.21.15.16:0/3367912029 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a200cd770 0x7f4a200b43e0 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7f4a1c00e870 tx=0x7f4a1c00ec30 comp rx=0 tx=0).stop 2023-10-16T12:31:10.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.561+0000 7f4a27df2700 1 -- 172.21.15.16:0/3367912029 shutdown_connections 2023-10-16T12:31:10.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.561+0000 7f4a27df2700 1 --2- 172.21.15.16:0/3367912029 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f4a0406b020 0x7f4a0406d4e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:10.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.561+0000 7f4a27df2700 1 --2- 172.21.15.16:0/3367912029 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4a2015b860 0x7f4a201f3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:10.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.561+0000 7f4a27df2700 1 --2- 172.21.15.16:0/3367912029 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4a200b2be0 0x7f4a200adaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:10.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.561+0000 7f4a27df2700 1 --2- 172.21.15.16:0/3367912029 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a200cd770 0x7f4a200b43e0 unknown :-1 s=CLOSED pgs=1751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:10.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.561+0000 7f4a27df2700 1 -- 172.21.15.16:0/3367912029 >> 172.21.15.16:0/3367912029 conn(0x7f4a200b1fd0 msgr2=0x7f4a200c6c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:31:10.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.561+0000 7f4a27df2700 1 -- 172.21.15.16:0/3367912029 shutdown_connections 2023-10-16T12:31:10.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.561+0000 7f4a27df2700 1 -- 172.21.15.16:0/3367912029 wait complete. 2023-10-16T12:31:10.567 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-10-16T12:31:10.575 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-10-16T12:31:09.013953+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1268,"num_read_kb":2572,"num_write":443,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":287,"ondisk_log_size":287,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":353184,"kb_used_data":16296,"kb_used_omap":12,"kb_used_meta":336755,"kb_avail":754621536,"statfs":{"total":773094113280,"available":772732452864,"internally_reserved":0,"allocated":16687104,"data_stored":8993234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12740,"internal_metadata":344837692},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.608563"},"pg_stats":[{"pgid":"20.3f","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.353783+0000","last_change":"2023-10-16T12:31:06.350967+0000","last_active":"2023-10-16T12:31:06.353783+0000","last_peered":"2023-10-16T12:31:06.353783+0000","last_clean":"2023-10-16T12:31:06.353783+0000","last_became_active":"2023-10-16T12:31:06.350613+0000","last_became_peered":"2023-10-16T12:31:06.350613+0000","last_unstale":"2023-10-16T12:31:06.353783+0000","last_undegraded":"2023-10-16T12:31:06.353783+0000","last_fullsized":"2023-10-16T12:31:06.353783+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:33:07.560381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3e","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.357603+0000","last_change":"2023-10-16T12:31:06.356905+0000","last_active":"2023-10-16T12:31:06.357603+0000","last_peered":"2023-10-16T12:31:06.357603+0000","last_clean":"2023-10-16T12:31:06.357603+0000","last_became_active":"2023-10-16T12:31:06.356523+0000","last_became_peered":"2023-10-16T12:31:06.356523+0000","last_unstale":"2023-10-16T12:31:06.357603+0000","last_undegraded":"2023-10-16T12:31:06.357603+0000","last_fullsized":"2023-10-16T12:31:06.357603+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:13:38.834513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.3d","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.360784+0000","last_change":"2023-10-16T12:31:06.357773+0000","last_active":"2023-10-16T12:31:06.360784+0000","last_peered":"2023-10-16T12:31:06.360784+0000","last_clean":"2023-10-16T12:31:06.360784+0000","last_became_active":"2023-10-16T12:31:06.357147+0000","last_became_peered":"2023-10-16T12:31:06.357147+0000","last_unstale":"2023-10-16T12:31:06.360784+0000","last_undegraded":"2023-10-16T12:31:06.360784+0000","last_fullsized":"2023-10-16T12:31:06.360784+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:01:42.630197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.3c","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.351425+0000","last_change":"2023-10-16T12:31:06.350748+0000","last_active":"2023-10-16T12:31:06.351425+0000","last_peered":"2023-10-16T12:31:06.351425+0000","last_clean":"2023-10-16T12:31:06.351425+0000","last_became_active":"2023-10-16T12:31:06.350379+0000","last_became_peered":"2023-10-16T12:31:06.350379+0000","last_unstale":"2023-10-16T12:31:06.351425+0000","last_undegraded":"2023-10-16T12:31:06.351425+0000","last_fullsized":"2023-10-16T12:31:06.351425+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:47:02.786943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.3b","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.357245+0000","last_change":"2023-10-16T12:31:06.356535+0000","last_active":"2023-10-16T12:31:06.357245+0000","last_peered":"2023-10-16T12:31:06.357245+0000","last_clean":"2023-10-16T12:31:06.357245+0000","last_became_active":"2023-10-16T12:31:06.356207+0000","last_became_peered":"2023-10-16T12:31:06.356207+0000","last_unstale":"2023-10-16T12:31:06.357245+0000","last_undegraded":"2023-10-16T12:31:06.357245+0000","last_fullsized":"2023-10-16T12:31:06.357245+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:07:24.792309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.3a","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.354699+0000","last_change":"2023-10-16T12:31:06.352459+0000","last_active":"2023-10-16T12:31:06.354699+0000","last_peered":"2023-10-16T12:31:06.354699+0000","last_clean":"2023-10-16T12:31:06.354699+0000","last_became_active":"2023-10-16T12:31:06.352187+0000","last_became_peered":"2023-10-16T12:31:06.352187+0000","last_unstale":"2023-10-16T12:31:06.354699+0000","last_undegraded":"2023-10-16T12:31:06.354699+0000","last_fullsized":"2023-10-16T12:31:06.354699+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:47:24.253841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.39","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.366209+0000","last_change":"2023-10-16T12:31:06.362877+0000","last_active":"2023-10-16T12:31:06.366209+0000","last_peered":"2023-10-16T12:31:06.366209+0000","last_clean":"2023-10-16T12:31:06.366209+0000","last_became_active":"2023-10-16T12:31:06.362495+0000","last_became_peered":"2023-10-16T12:31:06.362495+0000","last_unstale":"2023-10-16T12:31:06.366209+0000","last_undegraded":"2023-10-16T12:31:06.366209+0000","last_fullsized":"2023-10-16T12:31:06.366209+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:47:02.786943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.38","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.361666+0000","last_change":"2023-10-16T12:31:06.359510+0000","last_active":"2023-10-16T12:31:06.361666+0000","last_peered":"2023-10-16T12:31:06.361666+0000","last_clean":"2023-10-16T12:31:06.361666+0000","last_became_active":"2023-10-16T12:31:06.359225+0000","last_became_peered":"2023-10-16T12:31:06.359225+0000","last_unstale":"2023-10-16T12:31:06.361666+0000","last_undegraded":"2023-10-16T12:31:06.361666+0000","last_fullsized":"2023-10-16T12:31:06.361666+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:48:38.236166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.37","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.354533+0000","last_change":"2023-10-16T12:31:06.352367+0000","last_active":"2023-10-16T12:31:06.354533+0000","last_peered":"2023-10-16T12:31:06.354533+0000","last_clean":"2023-10-16T12:31:06.354533+0000","last_became_active":"2023-10-16T12:31:06.351297+0000","last_became_peered":"2023-10-16T12:31:06.351297+0000","last_unstale":"2023-10-16T12:31:06.354533+0000","last_undegraded":"2023-10-16T12:31:06.354533+0000","last_fullsized":"2023-10-16T12:31:06.354533+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:40:14.200747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.36","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.358318+0000","last_change":"2023-10-16T12:31:06.357233+0000","last_active":"2023-10-16T12:31:06.358318+0000","last_peered":"2023-10-16T12:31:06.358318+0000","last_clean":"2023-10-16T12:31:06.358318+0000","last_became_active":"2023-10-16T12:31:06.356943+0000","last_became_peered":"2023-10-16T12:31:06.356943+0000","last_unstale":"2023-10-16T12:31:06.358318+0000","last_undegraded":"2023-10-16T12:31:06.358318+0000","last_fullsized":"2023-10-16T12:31:06.358318+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:24:28.210474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.35","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.349524+0000","last_change":"2023-10-16T12:31:06.348427+0000","last_active":"2023-10-16T12:31:06.349524+0000","last_peered":"2023-10-16T12:31:06.349524+0000","last_clean":"2023-10-16T12:31:06.349524+0000","last_became_active":"2023-10-16T12:31:06.347009+0000","last_became_peered":"2023-10-16T12:31:06.347009+0000","last_unstale":"2023-10-16T12:31:06.349524+0000","last_undegraded":"2023-10-16T12:31:06.349524+0000","last_fullsized":"2023-10-16T12:31:06.349524+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:20:01.477905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.34","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.353757+0000","last_change":"2023-10-16T12:31:06.351665+0000","last_active":"2023-10-16T12:31:06.353757+0000","last_peered":"2023-10-16T12:31:06.353757+0000","last_clean":"2023-10-16T12:31:06.353757+0000","last_became_active":"2023-10-16T12:31:06.351389+0000","last_became_peered":"2023-10-16T12:31:06.351389+0000","last_unstale":"2023-10-16T12:31:06.353757+0000","last_undegraded":"2023-10-16T12:31:06.353757+0000","last_fullsized":"2023-10-16T12:31:06.353757+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:50:54.016852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.33","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.356644+0000","last_change":"2023-10-16T12:31:06.354571+0000","last_active":"2023-10-16T12:31:06.356644+0000","last_peered":"2023-10-16T12:31:06.356644+0000","last_clean":"2023-10-16T12:31:06.356644+0000","last_became_active":"2023-10-16T12:31:06.354241+0000","last_became_peered":"2023-10-16T12:31:06.354241+0000","last_unstale":"2023-10-16T12:31:06.356644+0000","last_undegraded":"2023-10-16T12:31:06.356644+0000","last_fullsized":"2023-10-16T12:31:06.356644+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:19:50.829101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.32","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.363339+0000","last_change":"2023-10-16T12:31:06.361452+0000","last_active":"2023-10-16T12:31:06.363339+0000","last_peered":"2023-10-16T12:31:06.363339+0000","last_clean":"2023-10-16T12:31:06.363339+0000","last_became_active":"2023-10-16T12:31:06.361178+0000","last_became_peered":"2023-10-16T12:31:06.361178+0000","last_unstale":"2023-10-16T12:31:06.363339+0000","last_undegraded":"2023-10-16T12:31:06.363339+0000","last_fullsized":"2023-10-16T12:31:06.363339+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:22:05.786952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.31","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.363217+0000","last_change":"2023-10-16T12:31:06.361001+0000","last_active":"2023-10-16T12:31:06.363217+0000","last_peered":"2023-10-16T12:31:06.363217+0000","last_clean":"2023-10-16T12:31:06.363217+0000","last_became_active":"2023-10-16T12:31:06.360683+0000","last_became_peered":"2023-10-16T12:31:06.360683+0000","last_unstale":"2023-10-16T12:31:06.363217+0000","last_undegraded":"2023-10-16T12:31:06.363217+0000","last_fullsized":"2023-10-16T12:31:06.363217+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:30:58.887965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.30","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.356634+0000","last_change":"2023-10-16T12:31:06.355135+0000","last_active":"2023-10-16T12:31:06.356634+0000","last_peered":"2023-10-16T12:31:06.356634+0000","last_clean":"2023-10-16T12:31:06.356634+0000","last_became_active":"2023-10-16T12:31:06.354805+0000","last_became_peered":"2023-10-16T12:31:06.354805+0000","last_unstale":"2023-10-16T12:31:06.356634+0000","last_undegraded":"2023-10-16T12:31:06.356634+0000","last_fullsized":"2023-10-16T12:31:06.356634+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:13:20.091518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.2f","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.358084+0000","last_change":"2023-10-16T12:31:06.356443+0000","last_active":"2023-10-16T12:31:06.358084+0000","last_peered":"2023-10-16T12:31:06.358084+0000","last_clean":"2023-10-16T12:31:06.358084+0000","last_became_active":"2023-10-16T12:31:06.356084+0000","last_became_peered":"2023-10-16T12:31:06.356084+0000","last_unstale":"2023-10-16T12:31:06.358084+0000","last_undegraded":"2023-10-16T12:31:06.358084+0000","last_fullsized":"2023-10-16T12:31:06.358084+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:34:07.904108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.2e","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.360176+0000","last_change":"2023-10-16T12:31:06.359683+0000","last_active":"2023-10-16T12:31:06.360176+0000","last_peered":"2023-10-16T12:31:06.360176+0000","last_clean":"2023-10-16T12:31:06.360176+0000","last_became_active":"2023-10-16T12:31:06.359310+0000","last_became_peered":"2023-10-16T12:31:06.359310+0000","last_unstale":"2023-10-16T12:31:06.360176+0000","last_undegraded":"2023-10-16T12:31:06.360176+0000","last_fullsized":"2023-10-16T12:31:06.360176+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:40:12.392133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.2d","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.364934+0000","last_change":"2023-10-16T12:31:06.362863+0000","last_active":"2023-10-16T12:31:06.364934+0000","last_peered":"2023-10-16T12:31:06.364934+0000","last_clean":"2023-10-16T12:31:06.364934+0000","last_became_active":"2023-10-16T12:31:06.362299+0000","last_became_peered":"2023-10-16T12:31:06.362299+0000","last_unstale":"2023-10-16T12:31:06.364934+0000","last_undegraded":"2023-10-16T12:31:06.364934+0000","last_fullsized":"2023-10-16T12:31:06.364934+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:03:05.365328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.2c","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.363388+0000","last_change":"2023-10-16T12:31:06.360910+0000","last_active":"2023-10-16T12:31:06.363388+0000","last_peered":"2023-10-16T12:31:06.363388+0000","last_clean":"2023-10-16T12:31:06.363388+0000","last_became_active":"2023-10-16T12:31:06.360550+0000","last_became_peered":"2023-10-16T12:31:06.360550+0000","last_unstale":"2023-10-16T12:31:06.363388+0000","last_undegraded":"2023-10-16T12:31:06.363388+0000","last_fullsized":"2023-10-16T12:31:06.363388+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:14:05.652793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.2b","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.356234+0000","last_change":"2023-10-16T12:31:06.355026+0000","last_active":"2023-10-16T12:31:06.356234+0000","last_peered":"2023-10-16T12:31:06.356234+0000","last_clean":"2023-10-16T12:31:06.356234+0000","last_became_active":"2023-10-16T12:31:06.354578+0000","last_became_peered":"2023-10-16T12:31:06.354578+0000","last_unstale":"2023-10-16T12:31:06.356234+0000","last_undegraded":"2023-10-16T12:31:06.356234+0000","last_fullsized":"2023-10-16T12:31:06.356234+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:47:20.460113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.2a","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.345028+0000","last_change":"2023-10-16T12:31:06.342061+0000","last_active":"2023-10-16T12:31:06.345028+0000","last_peered":"2023-10-16T12:31:06.345028+0000","last_clean":"2023-10-16T12:31:06.345028+0000","last_became_active":"2023-10-16T12:31:06.341705+0000","last_became_peered":"2023-10-16T12:31:06.341705+0000","last_unstale":"2023-10-16T12:31:06.345028+0000","last_undegraded":"2023-10-16T12:31:06.345028+0000","last_fullsized":"2023-10-16T12:31:06.345028+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:40:54.188746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.29","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.364844+0000","last_change":"2023-10-16T12:31:06.363571+0000","last_active":"2023-10-16T12:31:06.364844+0000","last_peered":"2023-10-16T12:31:06.364844+0000","last_clean":"2023-10-16T12:31:06.364844+0000","last_became_active":"2023-10-16T12:31:06.363262+0000","last_became_peered":"2023-10-16T12:31:06.363262+0000","last_unstale":"2023-10-16T12:31:06.364844+0000","last_undegraded":"2023-10-16T12:31:06.364844+0000","last_fullsized":"2023-10-16T12:31:06.364844+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:56:10.808368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.28","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.354362+0000","last_change":"2023-10-16T12:31:06.352033+0000","last_active":"2023-10-16T12:31:06.354362+0000","last_peered":"2023-10-16T12:31:06.354362+0000","last_clean":"2023-10-16T12:31:06.354362+0000","last_became_active":"2023-10-16T12:31:06.351735+0000","last_became_peered":"2023-10-16T12:31:06.351735+0000","last_unstale":"2023-10-16T12:31:06.354362+0000","last_undegraded":"2023-10-16T12:31:06.354362+0000","last_fullsized":"2023-10-16T12:31:06.354362+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:36:31.736567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.27","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.361129+0000","last_change":"2023-10-16T12:31:06.359735+0000","last_active":"2023-10-16T12:31:06.361129+0000","last_peered":"2023-10-16T12:31:06.361129+0000","last_clean":"2023-10-16T12:31:06.361129+0000","last_became_active":"2023-10-16T12:31:06.359480+0000","last_became_peered":"2023-10-16T12:31:06.359480+0000","last_unstale":"2023-10-16T12:31:06.361129+0000","last_undegraded":"2023-10-16T12:31:06.361129+0000","last_fullsized":"2023-10-16T12:31:06.361129+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:47:50.218838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.26","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.361866+0000","last_change":"2023-10-16T12:31:06.361258+0000","last_active":"2023-10-16T12:31:06.361866+0000","last_peered":"2023-10-16T12:31:06.361866+0000","last_clean":"2023-10-16T12:31:06.361866+0000","last_became_active":"2023-10-16T12:31:06.360749+0000","last_became_peered":"2023-10-16T12:31:06.360749+0000","last_unstale":"2023-10-16T12:31:06.361866+0000","last_undegraded":"2023-10-16T12:31:06.361866+0000","last_fullsized":"2023-10-16T12:31:06.361866+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:18:49.322758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.25","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.364540+0000","last_change":"2023-10-16T12:31:06.360494+0000","last_active":"2023-10-16T12:31:06.364540+0000","last_peered":"2023-10-16T12:31:06.364540+0000","last_clean":"2023-10-16T12:31:06.364540+0000","last_became_active":"2023-10-16T12:31:06.360162+0000","last_became_peered":"2023-10-16T12:31:06.360162+0000","last_unstale":"2023-10-16T12:31:06.364540+0000","last_undegraded":"2023-10-16T12:31:06.364540+0000","last_fullsized":"2023-10-16T12:31:06.364540+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:44:22.408445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.24","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.358310+0000","last_change":"2023-10-16T12:31:06.356846+0000","last_active":"2023-10-16T12:31:06.358310+0000","last_peered":"2023-10-16T12:31:06.358310+0000","last_clean":"2023-10-16T12:31:06.358310+0000","last_became_active":"2023-10-16T12:31:06.356538+0000","last_became_peered":"2023-10-16T12:31:06.356538+0000","last_unstale":"2023-10-16T12:31:06.358310+0000","last_undegraded":"2023-10-16T12:31:06.358310+0000","last_fullsized":"2023-10-16T12:31:06.358310+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:24:31.850753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.23","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.356425+0000","last_change":"2023-10-16T12:31:06.355320+0000","last_active":"2023-10-16T12:31:06.356425+0000","last_peered":"2023-10-16T12:31:06.356425+0000","last_clean":"2023-10-16T12:31:06.356425+0000","last_became_active":"2023-10-16T12:31:06.354848+0000","last_became_peered":"2023-10-16T12:31:06.354848+0000","last_unstale":"2023-10-16T12:31:06.356425+0000","last_undegraded":"2023-10-16T12:31:06.356425+0000","last_fullsized":"2023-10-16T12:31:06.356425+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:26:59.264313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.22","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.356900+0000","last_change":"2023-10-16T12:31:06.356104+0000","last_active":"2023-10-16T12:31:06.356900+0000","last_peered":"2023-10-16T12:31:06.356900+0000","last_clean":"2023-10-16T12:31:06.356900+0000","last_became_active":"2023-10-16T12:31:06.355811+0000","last_became_peered":"2023-10-16T12:31:06.355811+0000","last_unstale":"2023-10-16T12:31:06.356900+0000","last_undegraded":"2023-10-16T12:31:06.356900+0000","last_fullsized":"2023-10-16T12:31:06.356900+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:00:09.057516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.21","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.359003+0000","last_change":"2023-10-16T12:31:06.356103+0000","last_active":"2023-10-16T12:31:06.359003+0000","last_peered":"2023-10-16T12:31:06.359003+0000","last_clean":"2023-10-16T12:31:06.359003+0000","last_became_active":"2023-10-16T12:31:06.355828+0000","last_became_peered":"2023-10-16T12:31:06.355828+0000","last_unstale":"2023-10-16T12:31:06.359003+0000","last_undegraded":"2023-10-16T12:31:06.359003+0000","last_fullsized":"2023-10-16T12:31:06.359003+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:40:14.200747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.20","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.353675+0000","last_change":"2023-10-16T12:31:06.346338+0000","last_active":"2023-10-16T12:31:06.353675+0000","last_peered":"2023-10-16T12:31:06.353675+0000","last_clean":"2023-10-16T12:31:06.353675+0000","last_became_active":"2023-10-16T12:31:06.346056+0000","last_became_peered":"2023-10-16T12:31:06.346056+0000","last_unstale":"2023-10-16T12:31:06.353675+0000","last_undegraded":"2023-10-16T12:31:06.353675+0000","last_fullsized":"2023-10-16T12:31:06.353675+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:14:16.142686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.361654+0000","last_change":"2023-10-16T12:31:06.358760+0000","last_active":"2023-10-16T12:31:06.361654+0000","last_peered":"2023-10-16T12:31:06.361654+0000","last_clean":"2023-10-16T12:31:06.361654+0000","last_became_active":"2023-10-16T12:31:06.358438+0000","last_became_peered":"2023-10-16T12:31:06.358438+0000","last_unstale":"2023-10-16T12:31:06.361654+0000","last_undegraded":"2023-10-16T12:31:06.361654+0000","last_fullsized":"2023-10-16T12:31:06.361654+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:34:07.904108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1f","version":"155'4","reported_seq":15,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.425714+0000","last_change":"2023-10-16T12:31:05.433796+0000","last_active":"2023-10-16T12:31:06.425714+0000","last_peered":"2023-10-16T12:31:06.425714+0000","last_clean":"2023-10-16T12:31:06.425714+0000","last_became_active":"2023-10-16T12:31:05.433502+0000","last_became_peered":"2023-10-16T12:31:05.433502+0000","last_unstale":"2023-10-16T12:31:06.425714+0000","last_undegraded":"2023-10-16T12:31:06.425714+0000","last_fullsized":"2023-10-16T12:31:06.425714+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_clean_scrub_stamp":"2023-10-16T12:31:04.400465+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:13:19.086520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.344998+0000","last_change":"2023-10-16T12:31:06.341593+0000","last_active":"2023-10-16T12:31:06.344998+0000","last_peered":"2023-10-16T12:31:06.344998+0000","last_clean":"2023-10-16T12:31:06.344998+0000","last_became_active":"2023-10-16T12:31:06.341243+0000","last_became_peered":"2023-10-16T12:31:06.341243+0000","last_unstale":"2023-10-16T12:31:06.344998+0000","last_undegraded":"2023-10-16T12:31:06.344998+0000","last_fullsized":"2023-10-16T12:31:06.344998+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:59:55.880992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1e","version":"155'1","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.422987+0000","last_change":"2023-10-16T12:31:05.434867+0000","last_active":"2023-10-16T12:31:06.422987+0000","last_peered":"2023-10-16T12:31:06.422987+0000","last_clean":"2023-10-16T12:31:06.422987+0000","last_became_active":"2023-10-16T12:31:05.434596+0000","last_became_peered":"2023-10-16T12:31:05.434596+0000","last_unstale":"2023-10-16T12:31:06.422987+0000","last_undegraded":"2023-10-16T12:31:06.422987+0000","last_fullsized":"2023-10-16T12:31:06.422987+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_clean_scrub_stamp":"2023-10-16T12:31:04.400465+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:55:42.439555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1a","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.364833+0000","last_change":"2023-10-16T12:31:06.363592+0000","last_active":"2023-10-16T12:31:06.364833+0000","last_peered":"2023-10-16T12:31:06.364833+0000","last_clean":"2023-10-16T12:31:06.364833+0000","last_became_active":"2023-10-16T12:31:06.363290+0000","last_became_peered":"2023-10-16T12:31:06.363290+0000","last_unstale":"2023-10-16T12:31:06.364833+0000","last_undegraded":"2023-10-16T12:31:06.364833+0000","last_fullsized":"2023-10-16T12:31:06.364833+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:58:04.540273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1d","version":"155'2","reported_seq":13,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.423046+0000","last_change":"2023-10-16T12:31:05.428337+0000","last_active":"2023-10-16T12:31:06.423046+0000","last_peered":"2023-10-16T12:31:06.423046+0000","last_clean":"2023-10-16T12:31:06.423046+0000","last_became_active":"2023-10-16T12:31:05.428026+0000","last_became_peered":"2023-10-16T12:31:05.428026+0000","last_unstale":"2023-10-16T12:31:06.423046+0000","last_undegraded":"2023-10-16T12:31:06.423046+0000","last_fullsized":"2023-10-16T12:31:06.423046+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_clean_scrub_stamp":"2023-10-16T12:31:04.400465+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:14:33.240346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1b","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.363178+0000","last_change":"2023-10-16T12:31:06.360877+0000","last_active":"2023-10-16T12:31:06.363178+0000","last_peered":"2023-10-16T12:31:06.363178+0000","last_clean":"2023-10-16T12:31:06.363178+0000","last_became_active":"2023-10-16T12:31:06.360545+0000","last_became_peered":"2023-10-16T12:31:06.360545+0000","last_unstale":"2023-10-16T12:31:06.363178+0000","last_undegraded":"2023-10-16T12:31:06.363178+0000","last_fullsized":"2023-10-16T12:31:06.363178+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:19:25.399025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1c","version":"155'2","reported_seq":13,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.425469+0000","last_change":"2023-10-16T12:31:05.427637+0000","last_active":"2023-10-16T12:31:06.425469+0000","last_peered":"2023-10-16T12:31:06.425469+0000","last_clean":"2023-10-16T12:31:06.425469+0000","last_became_active":"2023-10-16T12:31:05.427182+0000","last_became_peered":"2023-10-16T12:31:05.427182+0000","last_unstale":"2023-10-16T12:31:06.425469+0000","last_undegraded":"2023-10-16T12:31:06.425469+0000","last_fullsized":"2023-10-16T12:31:06.425469+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_clean_scrub_stamp":"2023-10-16T12:31:04.400465+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:40:25.407994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.359156+0000","last_change":"2023-10-16T12:31:06.356596+0000","last_active":"2023-10-16T12:31:06.359156+0000","last_peered":"2023-10-16T12:31:06.359156+0000","last_clean":"2023-10-16T12:31:06.359156+0000","last_became_active":"2023-10-16T12:31:06.355131+0000","last_became_peered":"2023-10-16T12:31:06.355131+0000","last_unstale":"2023-10-16T12:31:06.359156+0000","last_undegraded":"2023-10-16T12:31:06.359156+0000","last_fullsized":"2023-10-16T12:31:06.359156+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:37:35.697306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1b","version":"155'1","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.426257+0000","last_change":"2023-10-16T12:31:05.432941+0000","last_active":"2023-10-16T12:31:06.426257+0000","last_peered":"2023-10-16T12:31:06.426257+0000","last_clean":"2023-10-16T12:31:06.426257+0000","last_became_active":"2023-10-16T12:31:05.432608+0000","last_became_peered":"2023-10-16T12:31:05.432608+0000","last_unstale":"2023-10-16T12:31:06.426257+0000","last_undegraded":"2023-10-16T12:31:06.426257+0000","last_fullsized":"2023-10-16T12:31:06.426257+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_clean_scrub_stamp":"2023-10-16T12:31:04.400465+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:38:59.295383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1d","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.359830+0000","last_change":"2023-10-16T12:31:06.355659+0000","last_active":"2023-10-16T12:31:06.359830+0000","last_peered":"2023-10-16T12:31:06.359830+0000","last_clean":"2023-10-16T12:31:06.359830+0000","last_became_active":"2023-10-16T12:31:06.355357+0000","last_became_peered":"2023-10-16T12:31:06.355357+0000","last_unstale":"2023-10-16T12:31:06.359830+0000","last_undegraded":"2023-10-16T12:31:06.359830+0000","last_fullsized":"2023-10-16T12:31:06.359830+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:47:24.253841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1a","version":"155'1","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.422988+0000","last_change":"2023-10-16T12:31:05.426078+0000","last_active":"2023-10-16T12:31:06.422988+0000","last_peered":"2023-10-16T12:31:06.422988+0000","last_clean":"2023-10-16T12:31:06.422988+0000","last_became_active":"2023-10-16T12:31:05.425793+0000","last_became_peered":"2023-10-16T12:31:05.425793+0000","last_unstale":"2023-10-16T12:31:06.422988+0000","last_undegraded":"2023-10-16T12:31:06.422988+0000","last_fullsized":"2023-10-16T12:31:06.422988+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_clean_scrub_stamp":"2023-10-16T12:31:04.400465+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:53:34.201546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.363454+0000","last_change":"2023-10-16T12:31:06.361606+0000","last_active":"2023-10-16T12:31:06.363454+0000","last_peered":"2023-10-16T12:31:06.363454+0000","last_clean":"2023-10-16T12:31:06.363454+0000","last_became_active":"2023-10-16T12:31:06.361310+0000","last_became_peered":"2023-10-16T12:31:06.361310+0000","last_unstale":"2023-10-16T12:31:06.363454+0000","last_undegraded":"2023-10-16T12:31:06.363454+0000","last_fullsized":"2023-10-16T12:31:06.363454+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:29:31.557647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.346716+0000","last_change":"2023-10-16T12:31:05.436170+0000","last_active":"2023-10-16T12:31:06.346716+0000","last_peered":"2023-10-16T12:31:06.346716+0000","last_clean":"2023-10-16T12:31:06.346716+0000","last_became_active":"2023-10-16T12:31:05.435868+0000","last_became_peered":"2023-10-16T12:31:05.435868+0000","last_unstale":"2023-10-16T12:31:06.346716+0000","last_undegraded":"2023-10-16T12:31:06.346716+0000","last_fullsized":"2023-10-16T12:31:06.346716+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_clean_scrub_stamp":"2023-10-16T12:31:04.400465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:28:52.884000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1f","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.363469+0000","last_change":"2023-10-16T12:31:06.358925+0000","last_active":"2023-10-16T12:31:06.363469+0000","last_peered":"2023-10-16T12:31:06.363469+0000","last_clean":"2023-10-16T12:31:06.363469+0000","last_became_active":"2023-10-16T12:31:06.358375+0000","last_became_peered":"2023-10-16T12:31:06.358375+0000","last_unstale":"2023-10-16T12:31:06.363469+0000","last_undegraded":"2023-10-16T12:31:06.363469+0000","last_fullsized":"2023-10-16T12:31:06.363469+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:07:24.792309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.344475+0000","last_change":"2023-10-16T12:31:05.438603+0000","last_active":"2023-10-16T12:31:06.344475+0000","last_peered":"2023-10-16T12:31:06.344475+0000","last_clean":"2023-10-16T12:31:06.344475+0000","last_became_active":"2023-10-16T12:31:05.438302+0000","last_became_peered":"2023-10-16T12:31:05.438302+0000","last_unstale":"2023-10-16T12:31:06.344475+0000","last_undegraded":"2023-10-16T12:31:06.344475+0000","last_fullsized":"2023-10-16T12:31:06.344475+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_clean_scrub_stamp":"2023-10-16T12:31:04.400465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:36:30.731569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.355452+0000","last_change":"2023-10-16T12:31:06.354175+0000","last_active":"2023-10-16T12:31:06.355452+0000","last_peered":"2023-10-16T12:31:06.355452+0000","last_clean":"2023-10-16T12:31:06.355452+0000","last_became_active":"2023-10-16T12:31:06.353721+0000","last_became_peered":"2023-10-16T12:31:06.353721+0000","last_unstale":"2023-10-16T12:31:06.355452+0000","last_undegraded":"2023-10-16T12:31:06.355452+0000","last_fullsized":"2023-10-16T12:31:06.355452+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:35:32.424425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.339432+0000","last_change":"2023-10-16T12:31:05.436538+0000","last_active":"2023-10-16T12:31:06.339432+0000","last_peered":"2023-10-16T12:31:06.339432+0000","last_clean":"2023-10-16T12:31:06.339432+0000","last_became_active":"2023-10-16T12:31:05.436249+0000","last_became_peered":"2023-10-16T12:31:05.436249+0000","last_unstale":"2023-10-16T12:31:06.339432+0000","last_undegraded":"2023-10-16T12:31:06.339432+0000","last_fullsized":"2023-10-16T12:31:06.339432+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_clean_scrub_stamp":"2023-10-16T12:31:04.400465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:53:02.482566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.354312+0000","last_change":"2023-10-16T12:31:06.347358+0000","last_active":"2023-10-16T12:31:06.354312+0000","last_peered":"2023-10-16T12:31:06.354312+0000","last_clean":"2023-10-16T12:31:06.354312+0000","last_became_active":"2023-10-16T12:31:06.347082+0000","last_became_peered":"2023-10-16T12:31:06.347082+0000","last_unstale":"2023-10-16T12:31:06.354312+0000","last_undegraded":"2023-10-16T12:31:06.354312+0000","last_fullsized":"2023-10-16T12:31:06.354312+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:39:00.300381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.337990+0000","last_change":"2023-10-16T12:31:05.433498+0000","last_active":"2023-10-16T12:31:06.337990+0000","last_peered":"2023-10-16T12:31:06.337990+0000","last_clean":"2023-10-16T12:31:06.337990+0000","last_became_active":"2023-10-16T12:31:05.433210+0000","last_became_peered":"2023-10-16T12:31:05.433210+0000","last_unstale":"2023-10-16T12:31:06.337990+0000","last_undegraded":"2023-10-16T12:31:06.337990+0000","last_fullsized":"2023-10-16T12:31:06.337990+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_clean_scrub_stamp":"2023-10-16T12:31:04.400465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:48:36.284683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.351556+0000","last_change":"2023-10-16T12:31:06.350328+0000","last_active":"2023-10-16T12:31:06.351556+0000","last_peered":"2023-10-16T12:31:06.351556+0000","last_clean":"2023-10-16T12:31:06.351556+0000","last_became_active":"2023-10-16T12:31:06.350014+0000","last_became_peered":"2023-10-16T12:31:06.350014+0000","last_unstale":"2023-10-16T12:31:06.351556+0000","last_undegraded":"2023-10-16T12:31:06.351556+0000","last_fullsized":"2023-10-16T12:31:06.351556+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:18:23.655771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.15","version":"155'3","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.424976+0000","last_change":"2023-10-16T12:31:05.432652+0000","last_active":"2023-10-16T12:31:06.424976+0000","last_peered":"2023-10-16T12:31:06.424976+0000","last_clean":"2023-10-16T12:31:06.424976+0000","last_became_active":"2023-10-16T12:31:05.432320+0000","last_became_peered":"2023-10-16T12:31:05.432320+0000","last_unstale":"2023-10-16T12:31:06.424976+0000","last_undegraded":"2023-10-16T12:31:06.424976+0000","last_fullsized":"2023-10-16T12:31:06.424976+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_clean_scrub_stamp":"2023-10-16T12:31:04.400465+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:27:31.708027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.13","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.361773+0000","last_change":"2023-10-16T12:31:06.360763+0000","last_active":"2023-10-16T12:31:06.361773+0000","last_peered":"2023-10-16T12:31:06.361773+0000","last_clean":"2023-10-16T12:31:06.361773+0000","last_became_active":"2023-10-16T12:31:06.360390+0000","last_became_peered":"2023-10-16T12:31:06.360390+0000","last_unstale":"2023-10-16T12:31:06.361773+0000","last_undegraded":"2023-10-16T12:31:06.361773+0000","last_fullsized":"2023-10-16T12:31:06.361773+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:45:22.357913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.14","version":"155'6","reported_seq":17,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.428288+0000","last_change":"2023-10-16T12:31:05.436216+0000","last_active":"2023-10-16T12:31:06.428288+0000","last_peered":"2023-10-16T12:31:06.428288+0000","last_clean":"2023-10-16T12:31:06.428288+0000","last_became_active":"2023-10-16T12:31:05.435894+0000","last_became_peered":"2023-10-16T12:31:05.435894+0000","last_unstale":"2023-10-16T12:31:06.428288+0000","last_undegraded":"2023-10-16T12:31:06.428288+0000","last_fullsized":"2023-10-16T12:31:06.428288+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_clean_scrub_stamp":"2023-10-16T12:31:04.400465+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:44:10.145999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.14","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.357842+0000","last_change":"2023-10-16T12:31:06.351283+0000","last_active":"2023-10-16T12:31:06.357842+0000","last_peered":"2023-10-16T12:31:06.357842+0000","last_clean":"2023-10-16T12:31:06.357842+0000","last_became_active":"2023-10-16T12:31:06.350288+0000","last_became_peered":"2023-10-16T12:31:06.350288+0000","last_unstale":"2023-10-16T12:31:06.357842+0000","last_undegraded":"2023-10-16T12:31:06.357842+0000","last_fullsized":"2023-10-16T12:31:06.357842+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:43:20.718343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.13","version":"155'2","reported_seq":13,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.426392+0000","last_change":"2023-10-16T12:31:05.433808+0000","last_active":"2023-10-16T12:31:06.426392+0000","last_peered":"2023-10-16T12:31:06.426392+0000","last_clean":"2023-10-16T12:31:06.426392+0000","last_became_active":"2023-10-16T12:31:05.433528+0000","last_became_peered":"2023-10-16T12:31:05.433528+0000","last_unstale":"2023-10-16T12:31:06.426392+0000","last_undegraded":"2023-10-16T12:31:06.426392+0000","last_fullsized":"2023-10-16T12:31:06.426392+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_clean_scrub_stamp":"2023-10-16T12:31:04.400465+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:14:00.280428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.363505+0000","last_change":"2023-10-16T12:31:06.358586+0000","last_active":"2023-10-16T12:31:06.363505+0000","last_peered":"2023-10-16T12:31:06.363505+0000","last_clean":"2023-10-16T12:31:06.363505+0000","last_became_active":"2023-10-16T12:31:06.358127+0000","last_became_peered":"2023-10-16T12:31:06.358127+0000","last_unstale":"2023-10-16T12:31:06.363505+0000","last_undegraded":"2023-10-16T12:31:06.363505+0000","last_fullsized":"2023-10-16T12:31:06.363505+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:55:02.716579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.344255+0000","last_change":"2023-10-16T12:31:05.431094+0000","last_active":"2023-10-16T12:31:06.344255+0000","last_peered":"2023-10-16T12:31:06.344255+0000","last_clean":"2023-10-16T12:31:06.344255+0000","last_became_active":"2023-10-16T12:31:05.430795+0000","last_became_peered":"2023-10-16T12:31:05.430795+0000","last_unstale":"2023-10-16T12:31:06.344255+0000","last_undegraded":"2023-10-16T12:31:06.344255+0000","last_fullsized":"2023-10-16T12:31:06.344255+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_clean_scrub_stamp":"2023-10-16T12:31:04.400465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:27:34.917740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.357191+0000","last_change":"2023-10-16T12:31:06.355134+0000","last_active":"2023-10-16T12:31:06.357191+0000","last_peered":"2023-10-16T12:31:06.357191+0000","last_clean":"2023-10-16T12:31:06.357191+0000","last_became_active":"2023-10-16T12:31:06.354652+0000","last_became_peered":"2023-10-16T12:31:06.354652+0000","last_unstale":"2023-10-16T12:31:06.357191+0000","last_undegraded":"2023-10-16T12:31:06.357191+0000","last_fullsized":"2023-10-16T12:31:06.357191+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:14:31.147453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.10","version":"155'8","reported_seq":19,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.426550+0000","last_change":"2023-10-16T12:31:05.433311+0000","last_active":"2023-10-16T12:31:06.426550+0000","last_peered":"2023-10-16T12:31:06.426550+0000","last_clean":"2023-10-16T12:31:06.426550+0000","last_became_active":"2023-10-16T12:31:05.433012+0000","last_became_peered":"2023-10-16T12:31:05.433012+0000","last_unstale":"2023-10-16T12:31:06.426550+0000","last_undegraded":"2023-10-16T12:31:06.426550+0000","last_fullsized":"2023-10-16T12:31:06.426550+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_clean_scrub_stamp":"2023-10-16T12:31:04.400465+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:40:24.125227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.359180+0000","last_change":"2023-10-16T12:31:06.357784+0000","last_active":"2023-10-16T12:31:06.359180+0000","last_peered":"2023-10-16T12:31:06.359180+0000","last_clean":"2023-10-16T12:31:06.359180+0000","last_became_active":"2023-10-16T12:31:06.357171+0000","last_became_peered":"2023-10-16T12:31:06.357171+0000","last_unstale":"2023-10-16T12:31:06.359180+0000","last_undegraded":"2023-10-16T12:31:06.359180+0000","last_fullsized":"2023-10-16T12:31:06.359180+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:52:32.915036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.f","version":"155'1","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.422117+0000","last_change":"2023-10-16T12:31:05.433277+0000","last_active":"2023-10-16T12:31:06.422117+0000","last_peered":"2023-10-16T12:31:06.422117+0000","last_clean":"2023-10-16T12:31:06.422117+0000","last_became_active":"2023-10-16T12:31:05.432971+0000","last_became_peered":"2023-10-16T12:31:05.432971+0000","last_unstale":"2023-10-16T12:31:06.422117+0000","last_undegraded":"2023-10-16T12:31:06.422117+0000","last_fullsized":"2023-10-16T12:31:06.422117+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_clean_scrub_stamp":"2023-10-16T12:31:04.400465+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:40:25.407994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.366188+0000","last_change":"2023-10-16T12:31:06.362808+0000","last_active":"2023-10-16T12:31:06.366188+0000","last_peered":"2023-10-16T12:31:06.366188+0000","last_clean":"2023-10-16T12:31:06.366188+0000","last_became_active":"2023-10-16T12:31:06.362208+0000","last_became_peered":"2023-10-16T12:31:06.362208+0000","last_unstale":"2023-10-16T12:31:06.366188+0000","last_undegraded":"2023-10-16T12:31:06.366188+0000","last_fullsized":"2023-10-16T12:31:06.366188+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:22:48.953437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.e","version":"155'2","reported_seq":13,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.425228+0000","last_change":"2023-10-16T12:31:05.433817+0000","last_active":"2023-10-16T12:31:06.425228+0000","last_peered":"2023-10-16T12:31:06.425228+0000","last_clean":"2023-10-16T12:31:06.425228+0000","last_became_active":"2023-10-16T12:31:05.433523+0000","last_became_peered":"2023-10-16T12:31:05.433523+0000","last_unstale":"2023-10-16T12:31:06.425228+0000","last_undegraded":"2023-10-16T12:31:06.425228+0000","last_fullsized":"2023-10-16T12:31:06.425228+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_clean_scrub_stamp":"2023-10-16T12:31:04.400465+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:14:24.203363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.364394+0000","last_change":"2023-10-16T12:31:06.360271+0000","last_active":"2023-10-16T12:31:06.364394+0000","last_peered":"2023-10-16T12:31:06.364394+0000","last_clean":"2023-10-16T12:31:06.364394+0000","last_became_active":"2023-10-16T12:31:06.359911+0000","last_became_peered":"2023-10-16T12:31:06.359911+0000","last_unstale":"2023-10-16T12:31:06.364394+0000","last_undegraded":"2023-10-16T12:31:06.364394+0000","last_fullsized":"2023-10-16T12:31:06.364394+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:53:03.487564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.d","version":"155'1","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.425190+0000","last_change":"2023-10-16T12:31:05.435666+0000","last_active":"2023-10-16T12:31:06.425190+0000","last_peered":"2023-10-16T12:31:06.425190+0000","last_clean":"2023-10-16T12:31:06.425190+0000","last_became_active":"2023-10-16T12:31:05.435334+0000","last_became_peered":"2023-10-16T12:31:05.435334+0000","last_unstale":"2023-10-16T12:31:06.425190+0000","last_undegraded":"2023-10-16T12:31:06.425190+0000","last_fullsized":"2023-10-16T12:31:06.425190+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_clean_scrub_stamp":"2023-10-16T12:31:04.400465+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:41:02.140573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.b","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.355165+0000","last_change":"2023-10-16T12:31:06.351952+0000","last_active":"2023-10-16T12:31:06.355165+0000","last_peered":"2023-10-16T12:31:06.355165+0000","last_clean":"2023-10-16T12:31:06.355165+0000","last_became_active":"2023-10-16T12:31:06.351610+0000","last_became_peered":"2023-10-16T12:31:06.351610+0000","last_unstale":"2023-10-16T12:31:06.355165+0000","last_undegraded":"2023-10-16T12:31:06.355165+0000","last_fullsized":"2023-10-16T12:31:06.355165+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:30:03.812820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.349950+0000","last_change":"2023-10-16T12:31:05.425457+0000","last_active":"2023-10-16T12:31:06.349950+0000","last_peered":"2023-10-16T12:31:06.349950+0000","last_clean":"2023-10-16T12:31:06.349950+0000","last_became_active":"2023-10-16T12:31:05.425153+0000","last_became_peered":"2023-10-16T12:31:05.425153+0000","last_unstale":"2023-10-16T12:31:06.349950+0000","last_undegraded":"2023-10-16T12:31:06.349950+0000","last_fullsized":"2023-10-16T12:31:06.349950+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_clean_scrub_stamp":"2023-10-16T12:31:04.400465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:19:15.123877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.357306+0000","last_change":"2023-10-16T12:31:06.355992+0000","last_active":"2023-10-16T12:31:06.357306+0000","last_peered":"2023-10-16T12:31:06.357306+0000","last_clean":"2023-10-16T12:31:06.357306+0000","last_became_active":"2023-10-16T12:31:06.355672+0000","last_became_peered":"2023-10-16T12:31:06.355672+0000","last_unstale":"2023-10-16T12:31:06.357306+0000","last_undegraded":"2023-10-16T12:31:06.357306+0000","last_fullsized":"2023-10-16T12:31:06.357306+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:22:48.953437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.b","version":"155'2","reported_seq":13,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.424813+0000","last_change":"2023-10-16T12:31:05.427181+0000","last_active":"2023-10-16T12:31:06.424813+0000","last_peered":"2023-10-16T12:31:06.424813+0000","last_clean":"2023-10-16T12:31:06.424813+0000","last_became_active":"2023-10-16T12:31:05.426885+0000","last_became_peered":"2023-10-16T12:31:05.426885+0000","last_unstale":"2023-10-16T12:31:06.424813+0000","last_undegraded":"2023-10-16T12:31:06.424813+0000","last_fullsized":"2023-10-16T12:31:06.424813+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_clean_scrub_stamp":"2023-10-16T12:31:04.400465+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:35:23.887227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.d","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.363196+0000","last_change":"2023-10-16T12:31:06.361111+0000","last_active":"2023-10-16T12:31:06.363196+0000","last_peered":"2023-10-16T12:31:06.363196+0000","last_clean":"2023-10-16T12:31:06.363196+0000","last_became_active":"2023-10-16T12:31:06.360764+0000","last_became_peered":"2023-10-16T12:31:06.360764+0000","last_unstale":"2023-10-16T12:31:06.363196+0000","last_undegraded":"2023-10-16T12:31:06.363196+0000","last_fullsized":"2023-10-16T12:31:06.363196+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:51:15.459519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.337935+0000","last_change":"2023-10-16T12:31:05.418528+0000","last_active":"2023-10-16T12:31:06.337935+0000","last_peered":"2023-10-16T12:31:06.337935+0000","last_clean":"2023-10-16T12:31:06.337935+0000","last_became_active":"2023-10-16T12:31:05.418144+0000","last_became_peered":"2023-10-16T12:31:05.418144+0000","last_unstale":"2023-10-16T12:31:06.337935+0000","last_undegraded":"2023-10-16T12:31:06.337935+0000","last_fullsized":"2023-10-16T12:31:06.337935+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_clean_scrub_stamp":"2023-10-16T12:31:04.400465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:41:02.978606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.355354+0000","last_change":"2023-10-16T12:31:06.352614+0000","last_active":"2023-10-16T12:31:06.355354+0000","last_peered":"2023-10-16T12:31:06.355354+0000","last_clean":"2023-10-16T12:31:06.355354+0000","last_became_active":"2023-10-16T12:31:06.352316+0000","last_became_peered":"2023-10-16T12:31:06.352316+0000","last_unstale":"2023-10-16T12:31:06.355354+0000","last_undegraded":"2023-10-16T12:31:06.355354+0000","last_fullsized":"2023-10-16T12:31:06.355354+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:44:22.408445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.348093+0000","last_change":"2023-10-16T12:31:05.427444+0000","last_active":"2023-10-16T12:31:06.348093+0000","last_peered":"2023-10-16T12:31:06.348093+0000","last_clean":"2023-10-16T12:31:06.348093+0000","last_became_active":"2023-10-16T12:31:05.427137+0000","last_became_peered":"2023-10-16T12:31:05.427137+0000","last_unstale":"2023-10-16T12:31:06.348093+0000","last_undegraded":"2023-10-16T12:31:06.348093+0000","last_fullsized":"2023-10-16T12:31:06.348093+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_clean_scrub_stamp":"2023-10-16T12:31:04.400465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:29:30.552649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.353318+0000","last_change":"2023-10-16T12:31:06.352354+0000","last_active":"2023-10-16T12:31:06.353318+0000","last_peered":"2023-10-16T12:31:06.353318+0000","last_clean":"2023-10-16T12:31:06.353318+0000","last_became_active":"2023-10-16T12:31:06.352002+0000","last_became_peered":"2023-10-16T12:31:06.352002+0000","last_unstale":"2023-10-16T12:31:06.353318+0000","last_undegraded":"2023-10-16T12:31:06.353318+0000","last_fullsized":"2023-10-16T12:31:06.353318+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:15:25.187373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.346253+0000","last_change":"2023-10-16T12:31:05.433973+0000","last_active":"2023-10-16T12:31:06.346253+0000","last_peered":"2023-10-16T12:31:06.346253+0000","last_clean":"2023-10-16T12:31:06.346253+0000","last_became_active":"2023-10-16T12:31:05.433018+0000","last_became_peered":"2023-10-16T12:31:05.433018+0000","last_unstale":"2023-10-16T12:31:06.346253+0000","last_undegraded":"2023-10-16T12:31:06.346253+0000","last_fullsized":"2023-10-16T12:31:06.346253+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_clean_scrub_stamp":"2023-10-16T12:31:04.400465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:32:36.733604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.363418+0000","last_change":"2023-10-16T12:31:06.361296+0000","last_active":"2023-10-16T12:31:06.363418+0000","last_peered":"2023-10-16T12:31:06.363418+0000","last_clean":"2023-10-16T12:31:06.363418+0000","last_became_active":"2023-10-16T12:31:06.360937+0000","last_became_peered":"2023-10-16T12:31:06.360937+0000","last_unstale":"2023-10-16T12:31:06.363418+0000","last_undegraded":"2023-10-16T12:31:06.363418+0000","last_fullsized":"2023-10-16T12:31:06.363418+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:07:57.171342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.7","version":"155'2","reported_seq":13,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.426122+0000","last_change":"2023-10-16T12:31:05.431448+0000","last_active":"2023-10-16T12:31:06.426122+0000","last_peered":"2023-10-16T12:31:06.426122+0000","last_clean":"2023-10-16T12:31:06.426122+0000","last_became_active":"2023-10-16T12:31:05.431157+0000","last_became_peered":"2023-10-16T12:31:05.431157+0000","last_unstale":"2023-10-16T12:31:06.426122+0000","last_undegraded":"2023-10-16T12:31:06.426122+0000","last_fullsized":"2023-10-16T12:31:06.426122+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_clean_scrub_stamp":"2023-10-16T12:31:04.400465+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:20:08.795305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.361524+0000","last_change":"2023-10-16T12:31:06.360125+0000","last_active":"2023-10-16T12:31:06.361524+0000","last_peered":"2023-10-16T12:31:06.361524+0000","last_clean":"2023-10-16T12:31:06.361524+0000","last_became_active":"2023-10-16T12:31:06.359812+0000","last_became_peered":"2023-10-16T12:31:06.359812+0000","last_unstale":"2023-10-16T12:31:06.361524+0000","last_undegraded":"2023-10-16T12:31:06.361524+0000","last_fullsized":"2023-10-16T12:31:06.361524+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:10:42.229486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.351861+0000","last_change":"2023-10-16T12:31:05.423438+0000","last_active":"2023-10-16T12:31:06.351861+0000","last_peered":"2023-10-16T12:31:06.351861+0000","last_clean":"2023-10-16T12:31:06.351861+0000","last_became_active":"2023-10-16T12:31:05.423129+0000","last_became_peered":"2023-10-16T12:31:05.423129+0000","last_unstale":"2023-10-16T12:31:06.351861+0000","last_undegraded":"2023-10-16T12:31:06.351861+0000","last_fullsized":"2023-10-16T12:31:06.351861+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_clean_scrub_stamp":"2023-10-16T12:31:04.400465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:40:24.125227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.364961+0000","last_change":"2023-10-16T12:31:06.363278+0000","last_active":"2023-10-16T12:31:06.364961+0000","last_peered":"2023-10-16T12:31:06.364961+0000","last_clean":"2023-10-16T12:31:06.364961+0000","last_became_active":"2023-10-16T12:31:06.360426+0000","last_became_peered":"2023-10-16T12:31:06.360426+0000","last_unstale":"2023-10-16T12:31:06.364961+0000","last_undegraded":"2023-10-16T12:31:06.364961+0000","last_fullsized":"2023-10-16T12:31:06.364961+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:14:05.652793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.351859+0000","last_change":"2023-10-16T12:31:05.419046+0000","last_active":"2023-10-16T12:31:06.351859+0000","last_peered":"2023-10-16T12:31:06.351859+0000","last_clean":"2023-10-16T12:31:06.351859+0000","last_became_active":"2023-10-16T12:31:05.418758+0000","last_became_peered":"2023-10-16T12:31:05.418758+0000","last_unstale":"2023-10-16T12:31:06.351859+0000","last_undegraded":"2023-10-16T12:31:06.351859+0000","last_fullsized":"2023-10-16T12:31:06.351859+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_clean_scrub_stamp":"2023-10-16T12:31:04.400465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:19:49.824103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.358265+0000","last_change":"2023-10-16T12:31:06.356817+0000","last_active":"2023-10-16T12:31:06.358265+0000","last_peered":"2023-10-16T12:31:06.358265+0000","last_clean":"2023-10-16T12:31:06.358265+0000","last_became_active":"2023-10-16T12:31:06.356564+0000","last_became_peered":"2023-10-16T12:31:06.356564+0000","last_unstale":"2023-10-16T12:31:06.358265+0000","last_undegraded":"2023-10-16T12:31:06.358265+0000","last_fullsized":"2023-10-16T12:31:06.358265+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:49:38.103616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.346634+0000","last_change":"2023-10-16T12:31:05.429324+0000","last_active":"2023-10-16T12:31:06.346634+0000","last_peered":"2023-10-16T12:31:06.346634+0000","last_clean":"2023-10-16T12:31:06.346634+0000","last_became_active":"2023-10-16T12:31:05.428969+0000","last_became_peered":"2023-10-16T12:31:05.428969+0000","last_unstale":"2023-10-16T12:31:06.346634+0000","last_undegraded":"2023-10-16T12:31:06.346634+0000","last_fullsized":"2023-10-16T12:31:06.346634+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_clean_scrub_stamp":"2023-10-16T12:31:04.400465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:02:23.301879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.353418+0000","last_change":"2023-10-16T12:31:06.346768+0000","last_active":"2023-10-16T12:31:06.353418+0000","last_peered":"2023-10-16T12:31:06.353418+0000","last_clean":"2023-10-16T12:31:06.353418+0000","last_became_active":"2023-10-16T12:31:06.346276+0000","last_became_peered":"2023-10-16T12:31:06.346276+0000","last_unstale":"2023-10-16T12:31:06.353418+0000","last_undegraded":"2023-10-16T12:31:06.353418+0000","last_fullsized":"2023-10-16T12:31:06.353418+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:37:35.697306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.338363+0000","last_change":"2023-10-16T12:31:05.430525+0000","last_active":"2023-10-16T12:31:06.338363+0000","last_peered":"2023-10-16T12:31:06.338363+0000","last_clean":"2023-10-16T12:31:06.338363+0000","last_became_active":"2023-10-16T12:31:05.429303+0000","last_became_peered":"2023-10-16T12:31:05.429303+0000","last_unstale":"2023-10-16T12:31:06.338363+0000","last_undegraded":"2023-10-16T12:31:06.338363+0000","last_fullsized":"2023-10-16T12:31:06.338363+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_clean_scrub_stamp":"2023-10-16T12:31:04.400465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:53:58.286905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.353356+0000","last_change":"2023-10-16T12:31:06.346748+0000","last_active":"2023-10-16T12:31:06.353356+0000","last_peered":"2023-10-16T12:31:06.353356+0000","last_clean":"2023-10-16T12:31:06.353356+0000","last_became_active":"2023-10-16T12:31:06.345820+0000","last_became_peered":"2023-10-16T12:31:06.345820+0000","last_unstale":"2023-10-16T12:31:06.353356+0000","last_undegraded":"2023-10-16T12:31:06.353356+0000","last_fullsized":"2023-10-16T12:31:06.353356+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:35:24.892226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.346589+0000","last_change":"2023-10-16T12:31:05.428948+0000","last_active":"2023-10-16T12:31:06.346589+0000","last_peered":"2023-10-16T12:31:06.346589+0000","last_clean":"2023-10-16T12:31:06.346589+0000","last_became_active":"2023-10-16T12:31:05.428656+0000","last_became_peered":"2023-10-16T12:31:05.428656+0000","last_unstale":"2023-10-16T12:31:06.346589+0000","last_undegraded":"2023-10-16T12:31:06.346589+0000","last_fullsized":"2023-10-16T12:31:06.346589+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_clean_scrub_stamp":"2023-10-16T12:31:04.400465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:34:33.747687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.359355+0000","last_change":"2023-10-16T12:31:06.358792+0000","last_active":"2023-10-16T12:31:06.359355+0000","last_peered":"2023-10-16T12:31:06.359355+0000","last_clean":"2023-10-16T12:31:06.359355+0000","last_became_active":"2023-10-16T12:31:06.358515+0000","last_became_peered":"2023-10-16T12:31:06.358515+0000","last_unstale":"2023-10-16T12:31:06.359355+0000","last_undegraded":"2023-10-16T12:31:06.359355+0000","last_fullsized":"2023-10-16T12:31:06.359355+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:48:38.236166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.346566+0000","last_change":"2023-10-16T12:31:05.430784+0000","last_active":"2023-10-16T12:31:06.346566+0000","last_peered":"2023-10-16T12:31:06.346566+0000","last_clean":"2023-10-16T12:31:06.346566+0000","last_became_active":"2023-10-16T12:31:05.430493+0000","last_became_peered":"2023-10-16T12:31:05.430493+0000","last_unstale":"2023-10-16T12:31:06.346566+0000","last_undegraded":"2023-10-16T12:31:06.346566+0000","last_fullsized":"2023-10-16T12:31:06.346566+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_clean_scrub_stamp":"2023-10-16T12:31:04.400465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:54:01.565361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.7","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.364602+0000","last_change":"2023-10-16T12:31:06.361588+0000","last_active":"2023-10-16T12:31:06.364602+0000","last_peered":"2023-10-16T12:31:06.364602+0000","last_clean":"2023-10-16T12:31:06.364602+0000","last_became_active":"2023-10-16T12:31:06.361307+0000","last_became_peered":"2023-10-16T12:31:06.361307+0000","last_unstale":"2023-10-16T12:31:06.364602+0000","last_undegraded":"2023-10-16T12:31:06.364602+0000","last_fullsized":"2023-10-16T12:31:06.364602+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:40:12.392133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.349924+0000","last_change":"2023-10-16T12:31:05.435448+0000","last_active":"2023-10-16T12:31:06.349924+0000","last_peered":"2023-10-16T12:31:06.349924+0000","last_clean":"2023-10-16T12:31:06.349924+0000","last_became_active":"2023-10-16T12:31:05.435146+0000","last_became_peered":"2023-10-16T12:31:05.435146+0000","last_unstale":"2023-10-16T12:31:06.349924+0000","last_undegraded":"2023-10-16T12:31:06.349924+0000","last_fullsized":"2023-10-16T12:31:06.349924+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_clean_scrub_stamp":"2023-10-16T12:31:04.400465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:54:16.674958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.15","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.354008+0000","last_change":"2023-10-16T12:31:06.352816+0000","last_active":"2023-10-16T12:31:06.354008+0000","last_peered":"2023-10-16T12:31:06.354008+0000","last_clean":"2023-10-16T12:31:06.354008+0000","last_became_active":"2023-10-16T12:31:06.352514+0000","last_became_peered":"2023-10-16T12:31:06.352514+0000","last_unstale":"2023-10-16T12:31:06.354008+0000","last_undegraded":"2023-10-16T12:31:06.354008+0000","last_fullsized":"2023-10-16T12:31:06.354008+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:50:50.515903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.346405+0000","last_change":"2023-10-16T12:31:05.428882+0000","last_active":"2023-10-16T12:31:06.346405+0000","last_peered":"2023-10-16T12:31:06.346405+0000","last_clean":"2023-10-16T12:31:06.346405+0000","last_became_active":"2023-10-16T12:31:05.428587+0000","last_became_peered":"2023-10-16T12:31:05.428587+0000","last_unstale":"2023-10-16T12:31:06.346405+0000","last_undegraded":"2023-10-16T12:31:06.346405+0000","last_fullsized":"2023-10-16T12:31:06.346405+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_clean_scrub_stamp":"2023-10-16T12:31:04.400465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:11:08.350800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"145'249","reported_seq":756,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-16T12:31:05.420543+0000","last_change":"2023-10-16T12:10:34.746820+0000","last_active":"2023-10-16T12:31:05.420543+0000","last_peered":"2023-10-16T12:31:05.420543+0000","last_clean":"2023-10-16T12:31:05.420543+0000","last_became_active":"2023-10-16T12:10:34.745574+0000","last_became_peered":"2023-10-16T12:10:34.745574+0000","last_unstale":"2023-10-16T12:31:05.420543+0000","last_undegraded":"2023-10-16T12:31:05.420543+0000","last_fullsized":"2023-10-16T12:31:05.420543+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_clean_scrub_stamp":"2023-10-16T12:10:33.577111+0000","objects_scrubbed":0,"log_size":249,"log_dups_size":0,"ondisk_log_size":249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:14.461986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1268,"num_read_kb":2572,"num_write":399,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":20,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":19,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1268,"num_read_kb":2572,"num_write":399,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":249,"ondisk_log_size":249,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575090,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47552,"kb_used_data":2484,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324288,"statfs":{"total":96636764160,"available":96588070912,"internally_reserved":0,"allocated":2543616,"data_stored":1569829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":46135755},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:30:18 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.61199999999999999,"15min":0.60199999999999998},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.27400000000000002},"max":{"1min":0.89300000000000002,"5min":4.2469999999999999,"15min":4.2469999999999999},"last":0.752},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.57199999999999995,"15min":0.58199999999999996},"min":{"1min":0.35299999999999998,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.84899999999999998,"5min":0.85299999999999998,"15min":0.89900000000000002},"last":0.58499999999999996}]},{"osd":1,"last update":"Mon Oct 16 12:30:18 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.58499999999999996,"15min":0.59999999999999998},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.247},"max":{"1min":0.80200000000000005,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.58999999999999997,"15min":0.60099999999999998},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.32400000000000001},"max":{"1min":0.79200000000000004,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.53800000000000003}]},{"osd":2,"last update":"Mon Oct 16 12:30:18 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.64400000000000002,"15min":0.65700000000000003},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.88600000000000001,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.64800000000000002,"15min":0.66300000000000003},"min":{"1min":0.44800000000000001,"5min":0.38100000000000001,"15min":0.376},"max":{"1min":0.88300000000000001,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.71399999999999997}]},{"osd":3,"last update":"Mon Oct 16 12:30:18 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.61099999999999999,"15min":0.61499999999999999},"min":{"1min":0.377,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.63,"15min":0.63300000000000001},"min":{"1min":0.41199999999999998,"5min":0.372,"15min":0.372},"max":{"1min":0.81000000000000005,"5min":0.996,"15min":1.2509999999999999},"last":0.624}]},{"osd":4,"last update":"Mon Oct 16 12:30:18 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.54400000000000004,"15min":0.53000000000000003},"min":{"1min":0.38600000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.80300000000000005,"5min":0.89700000000000002,"15min":1.383},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.627,"5min":0.63500000000000001,"15min":0.61499999999999999},"min":{"1min":0.39800000000000002,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.88200000000000001,"5min":0.93600000000000005,"15min":1.0109999999999999},"last":0.501}]},{"osd":5,"last update":"Mon Oct 16 12:30:18 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.56899999999999995,"15min":0.54000000000000004},"min":{"1min":0.311,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.77100000000000002,"5min":0.93400000000000005,"15min":1.3109999999999999},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.628,"5min":0.60799999999999998,"15min":0.58799999999999997},"min":{"1min":0.378,"5min":0.34699999999999998,"15min":0.32400000000000001},"max":{"1min":0.86099999999999999,"5min":0.93600000000000005,"15min":1.4359999999999999},"last":0.46200000000000002}]},{"osd":6,"last update":"Mon Oct 16 12:30:18 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.64800000000000002,"15min":0.625},"min":{"1min":0.41499999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.93000000000000005,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.64900000000000002,"15min":0.63},"min":{"1min":0.35099999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.91600000000000004,"5min":0.97199999999999998,"15min":1.1259999999999999},"last":0.60299999999999998}]}]},{"osd":6,"up_from":13,"seq":55834575088,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37244,"kb_used_data":1896,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334596,"statfs":{"total":96636764160,"available":96598626304,"internally_reserved":0,"allocated":1941504,"data_stored":978431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:30:07 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.57499999999999996,"15min":0.60899999999999999},"min":{"1min":0.35199999999999998,"5min":0.29299999999999998,"15min":0.24299999999999999},"max":{"1min":0.87,"5min":0.88200000000000001,"15min":1.196},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.59699999999999998,"15min":0.621},"min":{"1min":0.436,"5min":0.38500000000000001,"15min":0.33600000000000002},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.997},"last":0.60399999999999998}]},{"osd":1,"last update":"Mon Oct 16 12:30:07 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.53300000000000003,"15min":0.56000000000000005},"min":{"1min":0.35699999999999998,"5min":0.33100000000000002,"15min":0.29399999999999998},"max":{"1min":0.745,"5min":0.82199999999999995,"15min":0.875},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.57399999999999995,"15min":0.58599999999999997},"min":{"1min":0.40400000000000003,"5min":0.29799999999999999,"15min":0.29499999999999998},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":1.091},"last":0.623}]},{"osd":2,"last update":"Mon Oct 16 12:30:07 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56499999999999995,"15min":0.58999999999999997},"min":{"1min":0.377,"5min":0.35599999999999998,"15min":0.34499999999999997},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.97099999999999997},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.56899999999999995,"15min":0.59499999999999997},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.27100000000000002},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":2.3879999999999999},"last":0.51100000000000001}]},{"osd":3,"last update":"Mon Oct 16 12:30:07 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.60699999999999998,"15min":0.624},"min":{"1min":0.41599999999999998,"5min":0.37,"15min":0.34200000000000003},"max":{"1min":0.82299999999999995,"5min":0.88,"15min":0.93999999999999995},"last":0.751},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.66700000000000004,"15min":0.67800000000000005},"min":{"1min":0.504,"5min":0.41899999999999998,"15min":0.40899999999999997},"max":{"1min":0.90300000000000002,"5min":0.94499999999999995,"15min":1.3080000000000001},"last":0.72199999999999998}]},{"osd":4,"last update":"Mon Oct 16 12:30:07 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.52200000000000002,"15min":0.51600000000000001},"min":{"1min":0.35899999999999999,"5min":0.30599999999999999,"15min":0.29899999999999999},"max":{"1min":0.65700000000000003,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.60099999999999998,"15min":0.59699999999999998},"min":{"1min":0.38700000000000001,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.79400000000000004,"5min":0.82299999999999995,"15min":1.2250000000000001},"last":0.67600000000000005}]},{"osd":5,"last update":"Mon Oct 16 12:30:07 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.57899999999999996,"15min":0.57699999999999996},"min":{"1min":0.41499999999999998,"5min":0.30299999999999999,"15min":0.30099999999999999},"max":{"1min":0.70399999999999996,"5min":0.98799999999999999,"15min":1.0589999999999999},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.64300000000000002,"15min":0.626},"min":{"1min":0.33400000000000002,"5min":0.32100000000000001,"15min":0.311},"max":{"1min":0.84299999999999997,"5min":1.0029999999999999,"15min":1.3540000000000001},"last":0.80500000000000005}]},{"osd":7,"last update":"Mon Oct 16 12:30:07 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.58299999999999996,"15min":0.57599999999999996},"min":{"1min":0.42499999999999999,"5min":0.29699999999999999,"15min":0.26000000000000001},"max":{"1min":0.68799999999999994,"5min":0.92700000000000005,"15min":1.29},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.623,"5min":0.63400000000000001,"15min":0.61899999999999999},"min":{"1min":0.436,"5min":0.32900000000000001,"15min":0.29799999999999999},"max":{"1min":0.873,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.82299999999999995}]}]},{"osd":2,"up_from":13,"seq":55834575089,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63604,"kb_used_data":1896,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94308236,"statfs":{"total":96636764160,"available":96571633664,"internally_reserved":0,"allocated":1941504,"data_stored":978375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":63175110},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:30:05 2023","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.36899999999999999,"15min":0.379},"min":{"1min":0.222,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.51200000000000001,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.42699999999999999,"15min":0.443},"min":{"1min":0.28599999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.497,"5min":0.66500000000000004,"15min":0.82999999999999996},"last":0.371}]},{"osd":1,"last update":"Mon Oct 16 12:30:05 2023","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.441,"15min":0.46100000000000002},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.64500000000000002,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.40500000000000003,"15min":0.42999999999999999},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.57199999999999995,"5min":0.59199999999999997,"15min":0.69699999999999995},"last":0.47699999999999998}]},{"osd":3,"last update":"Mon Oct 16 12:30:05 2023","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.39400000000000002,"15min":0.40899999999999997},"min":{"1min":0.25,"5min":0.214,"15min":0.214},"max":{"1min":0.47799999999999998,"5min":0.69399999999999995,"15min":0.89300000000000002},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.498,"15min":0.51200000000000001},"min":{"1min":0.33200000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.74099999999999999,"5min":0.78000000000000003,"15min":0.79600000000000004},"last":0.49399999999999999}]},{"osd":4,"last update":"Mon Oct 16 12:30:05 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.63800000000000001,"15min":0.66800000000000004},"min":{"1min":0.35499999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.753,"5min":0.93799999999999994,"15min":1.3839999999999999},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.628,"15min":0.66500000000000004},"min":{"1min":0.42199999999999999,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.79900000000000004,"5min":0.97599999999999998,"15min":1.0109999999999999},"last":0.88300000000000001}]},{"osd":5,"last update":"Mon Oct 16 12:30:05 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.63500000000000001,"15min":0.65500000000000003},"min":{"1min":0.41599999999999998,"5min":0.35999999999999999,"15min":0.312},"max":{"1min":0.86699999999999999,"5min":1.093,"15min":1.093},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.61299999999999999,"15min":0.64700000000000002},"min":{"1min":0.40999999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.70499999999999996,"5min":1.0089999999999999,"15min":1.331},"last":0.67100000000000004}]},{"osd":6,"last update":"Mon Oct 16 12:30:05 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59599999999999997,"15min":0.626},"min":{"1min":0.42099999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.76200000000000001,"5min":0.99299999999999999,"15min":1.0369999999999999},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.65100000000000002,"15min":0.67600000000000005},"min":{"1min":0.437,"5min":0.40799999999999997,"15min":0.40000000000000002},"max":{"1min":0.83799999999999997,"5min":1.032,"15min":1.032},"last":0.999}]},{"osd":7,"last update":"Mon Oct 16 12:30:05 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.64800000000000002,"15min":0.66600000000000004},"min":{"1min":0.432,"5min":0.40300000000000002,"15min":0.38400000000000001},"max":{"1min":0.78300000000000003,"5min":0.96399999999999997,"15min":1.016},"last":1.02},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.65000000000000002,"15min":0.67800000000000005},"min":{"1min":0.44400000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.86199999999999999,"5min":1.145,"15min":1.145},"last":0.93600000000000005}]}]},{"osd":0,"up_from":13,"seq":55834575089,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37820,"kb_used_data":2476,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334020,"statfs":{"total":96636764160,"available":96598036480,"internally_reserved":0,"allocated":2535424,"data_stored":1568183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 12:31:00 2023","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.38200000000000001,"15min":0.38900000000000001},"min":{"1min":0.22800000000000001,"5min":0.219,"15min":0.219},"max":{"1min":0.58699999999999997,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.38300000000000001,"15min":0.41399999999999998},"min":{"1min":0.251,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.51600000000000001,"5min":0.67000000000000004,"15min":0.73099999999999998},"last":0.51300000000000001}]},{"osd":2,"last update":"Mon Oct 16 12:31:00 2023","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.40100000000000002,"15min":0.42099999999999999},"min":{"1min":0.27300000000000002,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.51300000000000001,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.432,"15min":0.44900000000000001},"min":{"1min":0.29199999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.51800000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.55600000000000005}]},{"osd":3,"last update":"Mon Oct 16 12:31:00 2023","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.40699999999999997,"15min":0.42299999999999999},"min":{"1min":0.28599999999999998,"5min":0.26300000000000001,"15min":0.23699999999999999},"max":{"1min":0.47999999999999998,"5min":0.62,"15min":0.79400000000000004},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.439,"5min":0.46300000000000002,"15min":0.48399999999999999},"min":{"1min":0.317,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.63600000000000001,"5min":0.79200000000000004,"15min":0.878},"last":0.60999999999999999}]},{"osd":4,"last update":"Mon Oct 16 12:31:00 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.61599999999999999,"15min":0.64400000000000002},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.78800000000000003,"5min":1.05,"15min":1.0960000000000001},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.61899999999999999,"15min":0.65100000000000002},"min":{"1min":0.36599999999999999,"5min":0.35599999999999998,"15min":0.34100000000000003},"max":{"1min":1.004,"5min":1.0549999999999999,"15min":1.1719999999999999},"last":0.81499999999999995}]},{"osd":5,"last update":"Mon Oct 16 12:31:00 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.60499999999999998,"15min":0.64500000000000002},"min":{"1min":0.39900000000000002,"5min":0.373,"15min":0.373},"max":{"1min":0.73599999999999999,"5min":0.97699999999999998,"15min":1.127},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.64000000000000001,"15min":0.67200000000000004},"min":{"1min":0.45300000000000001,"5min":0.39300000000000002,"15min":0.39000000000000001},"max":{"1min":0.85399999999999998,"5min":1.0700000000000001,"15min":1.1160000000000001},"last":0.79600000000000004}]},{"osd":6,"last update":"Mon Oct 16 12:31:00 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.64400000000000002,"15min":0.67700000000000005},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.80300000000000005,"5min":0.96899999999999997,"15min":1.079},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.62,"5min":0.65000000000000002,"15min":0.68200000000000005},"min":{"1min":0.40600000000000003,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.83699999999999997,"5min":0.96399999999999997,"15min":1.1499999999999999},"last":0.68200000000000005}]},{"osd":7,"last update":"Mon Oct 16 12:31:00 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.65800000000000003,"15min":0.69299999999999995},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.78100000000000003,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.64200000000000002,"15min":0.68100000000000005},"min":{"1min":0.41799999999999998,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.96999999999999997,"5min":0.97499999999999998,"15min":1.0569999999999999},"last":0.77800000000000002}]}]},{"osd":1,"up_from":13,"seq":55834575088,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37216,"kb_used_data":1876,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334624,"statfs":{"total":96636764160,"available":96598654976,"internally_reserved":0,"allocated":1921024,"data_stored":970608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":36174281},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.374,"15min":0.38},"min":{"1min":0.26600000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.627,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.434,"15min":0.46000000000000002},"min":{"1min":0.28899999999999998,"5min":0.28399999999999997,"15min":0.26700000000000002},"max":{"1min":0.60499999999999998,"5min":0.66300000000000003,"15min":1.0189999999999999},"last":0.318}]},{"osd":2,"last update":"Mon Oct 16 12:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.39800000000000002,"15min":0.42599999999999999},"min":{"1min":0.26900000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.59199999999999997,"5min":0.65300000000000002,"15min":0.93000000000000005},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.44,"5min":0.45000000000000001,"15min":0.46700000000000003},"min":{"1min":0.33700000000000002,"5min":0.29099999999999998,"15min":0.28199999999999997},"max":{"1min":0.61699999999999999,"5min":0.69199999999999995,"15min":0.98099999999999998},"last":0.38400000000000001}]},{"osd":3,"last update":"Mon Oct 16 12:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.38200000000000001,"15min":0.40300000000000002},"min":{"1min":0.247,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.56999999999999995,"5min":0.69599999999999995,"15min":0.98799999999999999},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.42999999999999999,"15min":0.45300000000000001},"min":{"1min":0.30299999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.751,"5min":0.751,"15min":0.91200000000000003},"last":0.33900000000000002}]},{"osd":4,"last update":"Mon Oct 16 12:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.60399999999999998,"15min":0.627},"min":{"1min":0.40500000000000003,"5min":0.35499999999999998,"15min":0.35199999999999998},"max":{"1min":0.74099999999999999,"5min":0.878,"15min":1.0409999999999999},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.624,"15min":0.64200000000000002},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.39100000000000001},"max":{"1min":0.86699999999999999,"5min":0.95399999999999996,"15min":0.98599999999999999},"last":0.61599999999999999}]},{"osd":5,"last update":"Mon Oct 16 12:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.63900000000000001,"15min":0.66300000000000003},"min":{"1min":0.42899999999999999,"5min":0.42599999999999999,"15min":0.38300000000000001},"max":{"1min":0.95399999999999996,"5min":1.069,"15min":1.069},"last":0.745},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.66300000000000003,"15min":0.68400000000000005},"min":{"1min":0.437,"5min":0.437,"15min":0.38900000000000001},"max":{"1min":0.93899999999999995,"5min":1.034,"15min":1.0620000000000001},"last":0.58299999999999996}]},{"osd":6,"last update":"Mon Oct 16 12:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.63400000000000001,"15min":0.65100000000000002},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.82599999999999996,"5min":1.044,"15min":1.044},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.63100000000000001,"15min":0.65800000000000003},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.32900000000000001},"max":{"1min":0.90400000000000003,"5min":1.0049999999999999,"15min":1.048},"last":0.67700000000000005}]},{"osd":7,"last update":"Mon Oct 16 12:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.64800000000000002,"15min":0.68300000000000005},"min":{"1min":0.45800000000000002,"5min":0.42099999999999999,"15min":0.33700000000000002},"max":{"1min":0.80600000000000005,"5min":1.0469999999999999,"15min":1.3400000000000001},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.627,"5min":0.67600000000000005,"15min":0.69699999999999995},"min":{"1min":0.47299999999999998,"5min":0.45700000000000002,"15min":0.42099999999999999},"max":{"1min":0.89100000000000001,"5min":1.006,"15min":1.02},"last":0.70399999999999996}]}]},{"osd":3,"up_from":13,"seq":55834575088,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37248,"kb_used_data":1904,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334592,"statfs":{"total":96636764160,"available":96598622208,"internally_reserved":0,"allocated":1949696,"data_stored":979441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:30:14 2023","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.38900000000000001,"15min":0.39700000000000002},"min":{"1min":0.22500000000000001,"5min":0.216,"15min":0.21099999999999999},"max":{"1min":0.46999999999999997,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.39200000000000002,"15min":0.40799999999999997},"min":{"1min":0.28000000000000003,"5min":0.254,"15min":0.254},"max":{"1min":0.54000000000000004,"5min":0.61899999999999999,"15min":0.73099999999999998},"last":0.35999999999999999}]},{"osd":1,"last update":"Mon Oct 16 12:30:14 2023","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.39900000000000002,"15min":0.41299999999999998},"min":{"1min":0.26400000000000001,"5min":0.245,"15min":0.24399999999999999},"max":{"1min":0.57499999999999996,"5min":0.70699999999999996,"15min":0.70799999999999996},"last":0.378},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.438,"15min":0.45800000000000002},"min":{"1min":0.29499999999999998,"5min":0.27000000000000002,"15min":0.25800000000000001},"max":{"1min":0.60399999999999998,"5min":0.72699999999999998,"15min":0.747},"last":0.38700000000000001}]},{"osd":2,"last update":"Mon Oct 16 12:30:14 2023","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.45300000000000001,"15min":0.46999999999999997},"min":{"1min":0.313,"5min":0.29799999999999999,"15min":0.29199999999999998},"max":{"1min":0.63,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.47899999999999998,"15min":0.498},"min":{"1min":0.311,"5min":0.30599999999999999,"15min":0.27100000000000002},"max":{"1min":0.72799999999999998,"5min":0.74299999999999999,"15min":0.752},"last":0.42499999999999999}]},{"osd":4,"last update":"Mon Oct 16 12:30:14 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.623,"15min":0.64500000000000002},"min":{"1min":0.39200000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.89600000000000002,"5min":0.98299999999999998,"15min":1.1539999999999999},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.622,"15min":0.65600000000000003},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.92100000000000004,"5min":0.95399999999999996,"15min":1.0409999999999999},"last":0.68400000000000005}]},{"osd":5,"last update":"Mon Oct 16 12:30:14 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.65200000000000002,"15min":0.68100000000000005},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.89400000000000002,"5min":1.0369999999999999,"15min":1.1080000000000001},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.63200000000000001,"15min":0.66000000000000003},"min":{"1min":0.45200000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.89000000000000001,"5min":0.97299999999999998,"15min":1.022},"last":0.67300000000000004}]},{"osd":6,"last update":"Mon Oct 16 12:30:14 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63300000000000001,"15min":0.66500000000000004},"min":{"1min":0.437,"5min":0.42999999999999999,"15min":0.36899999999999999},"max":{"1min":0.875,"5min":0.94199999999999995,"15min":1.0760000000000001},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.67400000000000004,"15min":0.70199999999999996},"min":{"1min":0.434,"5min":0.434,"15min":0.38600000000000001},"max":{"1min":0.89100000000000001,"5min":1.0349999999999999,"15min":1.0409999999999999},"last":0.60699999999999998}]},{"osd":7,"last update":"Mon Oct 16 12:30:14 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.64700000000000002,"15min":0.67900000000000005},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.38},"max":{"1min":0.85999999999999999,"5min":1.0149999999999999,"15min":1.2430000000000001},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.68500000000000005,"15min":0.70499999999999996},"min":{"1min":0.503,"5min":0.44,"15min":0.42699999999999999},"max":{"1min":0.88100000000000001,"5min":1.002,"15min":1.1339999999999999},"last":0.621}]}]},{"osd":4,"up_from":13,"seq":55834575087,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37228,"kb_used_data":1888,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334612,"statfs":{"total":96636764160,"available":96598642688,"internally_reserved":0,"allocated":1933312,"data_stored":977759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.56599999999999995,"15min":0.58999999999999997},"min":{"1min":0.41099999999999998,"5min":0.30399999999999999,"15min":0.27900000000000003},"max":{"1min":0.752,"5min":0.78500000000000003,"15min":0.94599999999999995},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.57199999999999995,"15min":0.59699999999999998},"min":{"1min":0.42399999999999999,"5min":0.29999999999999999,"15min":0.27600000000000002},"max":{"1min":0.73299999999999998,"5min":0.83199999999999996,"15min":0.88700000000000001},"last":0.52600000000000002}]},{"osd":1,"last update":"Mon Oct 16 12:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.58599999999999997,"15min":0.60799999999999998},"min":{"1min":0.32900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.76200000000000001,"5min":0.82599999999999996,"15min":0.92600000000000005},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.61399999999999999},"min":{"1min":0.434,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.98099999999999998},"last":0.54800000000000004}]},{"osd":2,"last update":"Mon Oct 16 12:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.58399999999999996,"15min":0.61799999999999999},"min":{"1min":0.34999999999999998,"5min":0.33800000000000002,"15min":0.32000000000000001},"max":{"1min":0.745,"5min":0.84299999999999997,"15min":1.012},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.59699999999999998,"15min":0.624},"min":{"1min":0.38200000000000001,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.76000000000000001,"5min":0.85599999999999998,"15min":0.94199999999999995},"last":0.60499999999999998}]},{"osd":3,"last update":"Mon Oct 16 12:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.56899999999999995,"15min":0.60199999999999998},"min":{"1min":0.41799999999999998,"5min":0.35599999999999998,"15min":0.33700000000000002},"max":{"1min":0.70699999999999996,"5min":0.82299999999999995,"15min":0.93200000000000005},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.63,"5min":0.63300000000000001,"15min":0.64300000000000002},"min":{"1min":0.47099999999999997,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.80000000000000004,"5min":0.90600000000000003,"15min":0.97199999999999998},"last":0.50600000000000001}]},{"osd":5,"last update":"Mon Oct 16 12:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.52100000000000002,"15min":0.50600000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.72199999999999998,"5min":0.82599999999999996,"15min":0.95699999999999996},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.53000000000000003,"15min":0.52500000000000002},"min":{"1min":0.373,"5min":0.33900000000000002,"15min":0.308},"max":{"1min":0.73399999999999999,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.39600000000000002}]},{"osd":6,"last update":"Mon Oct 16 12:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.55600000000000005,"15min":0.55700000000000005},"min":{"1min":0.39000000000000001,"5min":0.33800000000000002,"15min":0.32500000000000001},"max":{"1min":0.70699999999999996,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.63800000000000001,"15min":0.63700000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.32000000000000001},"max":{"1min":0.84499999999999997,"5min":0.89300000000000002,"15min":0.99299999999999999},"last":0.48999999999999999}]},{"osd":7,"last update":"Mon Oct 16 12:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59199999999999997,"15min":0.58499999999999996},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.77700000000000002,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.629,"15min":0.629},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.29999999999999999},"max":{"1min":0.82899999999999996,"5min":1.006,"15min":1.006},"last":0.38}]}]},{"osd":5,"up_from":13,"seq":55834575088,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55272,"kb_used_data":1876,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316568,"statfs":{"total":96636764160,"available":96580165632,"internally_reserved":0,"allocated":1921024,"data_stored":970608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.53100000000000003,"15min":0.57799999999999996},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.73199999999999998,"5min":0.81399999999999995,"15min":0.873},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.58199999999999996,"15min":0.61099999999999999},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.27600000000000002},"max":{"1min":0.86099999999999999,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.67400000000000004}]},{"osd":1,"last update":"Mon Oct 16 12:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.58399999999999996,"15min":0.60699999999999998},"min":{"1min":0.40300000000000002,"5min":0.34499999999999997,"15min":0.28999999999999998},"max":{"1min":0.81499999999999995,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.55400000000000005,"15min":0.60399999999999998},"min":{"1min":0.34300000000000003,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.83399999999999996,"5min":0.84399999999999997,"15min":0.89000000000000001},"last":0.61799999999999999}]},{"osd":2,"last update":"Mon Oct 16 12:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57199999999999995,"15min":0.61299999999999999},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.29099999999999998},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.94399999999999995},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.56499999999999995,"15min":0.60299999999999998},"min":{"1min":0.34599999999999997,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.873,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.58899999999999997}]},{"osd":3,"last update":"Mon Oct 16 12:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.60099999999999998,"15min":0.628},"min":{"1min":0.41999999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.89300000000000002,"5min":0.90400000000000003,"15min":0.97499999999999998},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.621,"15min":0.64100000000000001},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.35999999999999999},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.72499999999999998}]},{"osd":4,"last update":"Mon Oct 16 12:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.501,"15min":0.51000000000000001},"min":{"1min":0.311,"5min":0.28100000000000003,"15min":0.25800000000000001},"max":{"1min":0.80300000000000005,"5min":0.81299999999999994,"15min":0.86399999999999999},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.57799999999999996,"15min":0.57299999999999995},"min":{"1min":0.44400000000000001,"5min":0.33900000000000002,"15min":0.33500000000000002},"max":{"1min":0.81999999999999995,"5min":0.876,"15min":1.347},"last":0.70299999999999996}]},{"osd":6,"last update":"Mon Oct 16 12:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.55700000000000005,"15min":0.55400000000000005},"min":{"1min":0.36399999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.85299999999999998,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.626,"5min":0.57999999999999996,"15min":0.59799999999999998},"min":{"1min":0.39500000000000002,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.83899999999999997,"5min":0.88400000000000001,"15min":0.92700000000000005},"last":0.65900000000000003}]},{"osd":7,"last update":"Mon Oct 16 12:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.63,"15min":0.627},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.32300000000000001},"max":{"1min":0.88300000000000001,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.64400000000000002,"15min":0.65700000000000003},"min":{"1min":0.47799999999999998,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":1.1180000000000001},"last":0.68799999999999994}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T12:31:10.582 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T12:31:10.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.738+0000 7fba075de700 1 Processor -- start 2023-10-16T12:31:10.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.738+0000 7fba075de700 1 -- start start 2023-10-16T12:31:10.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.739+0000 7fba075de700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fba000cf4a0 0x7fba000726e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:10.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.739+0000 7fba075de700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fba000cf880 0x7fba000b52f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:10.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.739+0000 7fba075de700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba0015bc80 con 0x7fba000b5830 2023-10-16T12:31:10.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.739+0000 7fba075de700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba00072c20 con 0x7fba000cf880 2023-10-16T12:31:10.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.739+0000 7fba075de700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba0015b070 con 0x7fba000cf4a0 2023-10-16T12:31:10.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.739+0000 7fba04b79700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fba000cf880 0x7fba000b52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:10.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.739+0000 7fba05b7b700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fba000b5830 0x7fba0015a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36234/0 (socket says 172.21.15.16:36234) 2023-10-16T12:31:10.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.739+0000 7fba05b7b700 1 -- 172.21.15.16:0/1135962970 learned_addr learned my addr 172.21.15.16:0/1135962970 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:31:10.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.739+0000 7fba0537a700 1 --2- 172.21.15.16:0/1135962970 >> v2:172.21.15.49:3300/0 conn(0x7fba000cf4a0 0x7fba000726e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:10.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.739+0000 7fba04b79700 1 -- 172.21.15.16:0/1135962970 >> v2:172.21.15.49:3300/0 conn(0x7fba000cf4a0 msgr2=0x7fba000726e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:31:10.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.739+0000 7fba04b79700 1 --2- 172.21.15.16:0/1135962970 >> v2:172.21.15.49:3300/0 conn(0x7fba000cf4a0 0x7fba000726e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:10.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.740+0000 7fba04b79700 1 -- 172.21.15.16:0/1135962970 >> v1:172.21.15.16:6789/0 conn(0x7fba000b5830 legacy=0x7fba0015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:10.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.740+0000 7fba04b79700 1 -- 172.21.15.16:0/1135962970 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba0015b350 con 0x7fba000cf880 2023-10-16T12:31:10.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.740+0000 7fb9f7fff700 1 -- 172.21.15.16:0/1135962970 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3030894461 0 0) 0x7fba0015bc80 con 0x7fba000b5830 2023-10-16T12:31:10.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.740+0000 7fba04b79700 1 --2- 172.21.15.16:0/1135962970 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba000cf880 0x7fba000b52f0 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7fba000b2d20 tx=0x7fb9fc009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d628bab7bf558d5a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:10.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.740+0000 7fb9f7fff700 1 -- 172.21.15.16:0/1135962970 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9fc011830 con 0x7fba000cf880 2023-10-16T12:31:10.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.740+0000 7fb9f7fff700 1 -- 172.21.15.16:0/1135962970 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9fc011990 con 0x7fba000cf880 2023-10-16T12:31:10.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.740+0000 7fb9f7fff700 1 -- 172.21.15.16:0/1135962970 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9fc011d20 con 0x7fba000cf880 2023-10-16T12:31:10.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.740+0000 7fba075de700 1 -- 172.21.15.16:0/1135962970 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba000cf880 msgr2=0x7fba000b52f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:10.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.740+0000 7fba075de700 1 --2- 172.21.15.16:0/1135962970 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba000cf880 0x7fba000b52f0 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7fba000b2d20 tx=0x7fb9fc009b20 comp rx=0 tx=0).stop 2023-10-16T12:31:10.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.740+0000 7fba075de700 1 -- 172.21.15.16:0/1135962970 shutdown_connections 2023-10-16T12:31:10.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.740+0000 7fba075de700 1 --2- 172.21.15.16:0/1135962970 >> v2:172.21.15.49:3300/0 conn(0x7fba000cf4a0 0x7fba000726e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:10.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.740+0000 7fba075de700 1 --2- 172.21.15.16:0/1135962970 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba000cf880 0x7fba000b52f0 unknown :-1 s=CLOSED pgs=1753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:10.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.740+0000 7fba075de700 1 -- 172.21.15.16:0/1135962970 >> 172.21.15.16:0/1135962970 conn(0x7fba000b1fd0 msgr2=0x7fba000b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:31:10.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.741+0000 7fba075de700 1 -- 172.21.15.16:0/1135962970 shutdown_connections 2023-10-16T12:31:10.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.741+0000 7fba075de700 1 -- 172.21.15.16:0/1135962970 wait complete. 2023-10-16T12:31:10.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.742+0000 7fba075de700 1 Processor -- start 2023-10-16T12:31:10.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.742+0000 7fba075de700 1 -- start start 2023-10-16T12:31:10.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.742+0000 7fba075de700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fba000b5830 0x7fba000c6620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:10.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.742+0000 7fba075de700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fba000cf4a0 0x7fba0014fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:10.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.742+0000 7fba075de700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba000cf880 0x7fba001511e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:10.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.743+0000 7fba075de700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba000c5e50 con 0x7fba000cf880 2023-10-16T12:31:10.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.743+0000 7fba075de700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba000c6ee0 con 0x7fba000cf4a0 2023-10-16T12:31:10.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.743+0000 7fba075de700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba00150d50 con 0x7fba000b5830 2023-10-16T12:31:10.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.743+0000 7fba05b7b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba000cf880 0x7fba001511e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:10.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.743+0000 7fba05b7b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba000cf880 0x7fba001511e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46966/0 (socket says 172.21.15.16:46966) 2023-10-16T12:31:10.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.743+0000 7fba05b7b700 1 -- 172.21.15.16:0/3515040587 learned_addr learned my addr 172.21.15.16:0/3515040587 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:31:10.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.743+0000 7fba04b79700 1 --2- 172.21.15.16:0/3515040587 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fba000cf4a0 0x7fba0014fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:10.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.743+0000 7fba0537a700 1 --2- 172.21.15.16:0/3515040587 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fba000b5830 0x7fba000c6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:10.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.743+0000 7fba05b7b700 1 -- 172.21.15.16:0/3515040587 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fba000b5830 msgr2=0x7fba000c6620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:10.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.743+0000 7fba05b7b700 1 --2- 172.21.15.16:0/3515040587 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fba000b5830 0x7fba000c6620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:10.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.743+0000 7fba05b7b700 1 -- 172.21.15.16:0/3515040587 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fba000cf4a0 msgr2=0x7fba0014fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:10.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.743+0000 7fba05b7b700 1 --2- 172.21.15.16:0/3515040587 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fba000cf4a0 0x7fba0014fa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:10.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.743+0000 7fba05b7b700 1 -- 172.21.15.16:0/3515040587 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9fc0094a0 con 0x7fba000cf880 2023-10-16T12:31:10.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.743+0000 7fba05b7b700 1 --2- 172.21.15.16:0/3515040587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba000cf880 0x7fba001511e0 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7fb9f800cb60 tx=0x7fb9f800cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:10.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.743+0000 7fb9f5ffb700 1 -- 172.21.15.16:0/3515040587 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9f800ab50 con 0x7fba000cf880 2023-10-16T12:31:10.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.743+0000 7fb9f5ffb700 1 -- 172.21.15.16:0/3515040587 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9f800acb0 con 0x7fba000cf880 2023-10-16T12:31:10.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.743+0000 7fb9f5ffb700 1 -- 172.21.15.16:0/3515040587 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9f8007310 con 0x7fba000cf880 2023-10-16T12:31:10.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.743+0000 7fba075de700 1 -- 172.21.15.16:0/3515040587 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba00150420 con 0x7fba000cf880 2023-10-16T12:31:10.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.743+0000 7fba075de700 1 -- 172.21.15.16:0/3515040587 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba001509e0 con 0x7fba000cf880 2023-10-16T12:31:10.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.745+0000 7fba075de700 1 -- 172.21.15.16:0/3515040587 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba00150ec0 con 0x7fba000cf880 2023-10-16T12:31:10.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.745+0000 7fb9f5ffb700 1 -- 172.21.15.16:0/3515040587 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 85) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb9f8007470 con 0x7fba000cf880 2023-10-16T12:31:10.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.745+0000 7fb9f5ffb700 1 --2- 172.21.15.16:0/3515040587 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7fb9e406b020 0x7fb9e406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:10.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.745+0000 7fb9f5ffb700 1 -- 172.21.15.16:0/3515040587 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 6085+0+0 (secure 0 0 0) 0x7fb9f80892c0 con 0x7fba000cf880 2023-10-16T12:31:10.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.745+0000 7fba0537a700 1 --2- 172.21.15.16:0/3515040587 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7fb9e406b020 0x7fb9e406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:10.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.746+0000 7fba0537a700 1 --2- 172.21.15.16:0/3515040587 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7fb9e406b020 0x7fb9e406d4e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb9f0009950 tx=0x7fb9f0008040 comp rx=0 tx=0).ready entity=mgr.8973 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:10.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.750+0000 7fb9f5ffb700 1 -- 172.21.15.16:0/3515040587 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb9f8054170 con 0x7fba000cf880 2023-10-16T12:31:10.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.952+0000 7fba075de700 1 -- 172.21.15.16:0/3515040587 --> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fba0015b910 con 0x7fb9e406b020 2023-10-16T12:31:10.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.960+0000 7fb9f5ffb700 1 -- 172.21.15.16:0/3515040587 <== mgr.8973 v2:172.21.15.16:6832/4182163087 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289846 (secure 0 0 0) 0x7fb9f00095e0 con 0x7fb9e406b020 2023-10-16T12:31:10.965 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:31:10.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.963+0000 7fba075de700 1 -- 172.21.15.16:0/3515040587 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7fb9e406b020 msgr2=0x7fb9e406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:10.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.963+0000 7fba075de700 1 --2- 172.21.15.16:0/3515040587 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7fb9e406b020 0x7fb9e406d4e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb9f0009950 tx=0x7fb9f0008040 comp rx=0 tx=0).stop 2023-10-16T12:31:10.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.963+0000 7fba075de700 1 -- 172.21.15.16:0/3515040587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba000cf880 msgr2=0x7fba001511e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:10.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.963+0000 7fba075de700 1 --2- 172.21.15.16:0/3515040587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba000cf880 0x7fba001511e0 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7fb9f800cb60 tx=0x7fb9f800cf20 comp rx=0 tx=0).stop 2023-10-16T12:31:10.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.964+0000 7fba075de700 1 -- 172.21.15.16:0/3515040587 shutdown_connections 2023-10-16T12:31:10.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.964+0000 7fba075de700 1 --2- 172.21.15.16:0/3515040587 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7fb9e406b020 0x7fb9e406d4e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:10.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.964+0000 7fba075de700 1 --2- 172.21.15.16:0/3515040587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba000cf880 0x7fba001511e0 unknown :-1 s=CLOSED pgs=1754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:10.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.964+0000 7fba075de700 1 --2- 172.21.15.16:0/3515040587 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fba000b5830 0x7fba000c6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:10.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.964+0000 7fba075de700 1 --2- 172.21.15.16:0/3515040587 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fba000cf4a0 0x7fba0014fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:10.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.964+0000 7fba075de700 1 -- 172.21.15.16:0/3515040587 >> 172.21.15.16:0/3515040587 conn(0x7fba000b1fd0 msgr2=0x7fba000c33d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:31:10.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.964+0000 7fba075de700 1 -- 172.21.15.16:0/3515040587 shutdown_connections 2023-10-16T12:31:10.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:10.964+0000 7fba075de700 1 -- 172.21.15.16:0/3515040587 wait complete. 2023-10-16T12:31:10.970 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-10-16T12:31:10.978 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-10-16T12:31:09.013953+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1268,"num_read_kb":2572,"num_write":443,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":287,"ondisk_log_size":287,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":353184,"kb_used_data":16296,"kb_used_omap":12,"kb_used_meta":336755,"kb_avail":754621536,"statfs":{"total":773094113280,"available":772732452864,"internally_reserved":0,"allocated":16687104,"data_stored":8993234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12740,"internal_metadata":344837692},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.608563"},"pg_stats":[{"pgid":"20.3f","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.353783+0000","last_change":"2023-10-16T12:31:06.350967+0000","last_active":"2023-10-16T12:31:06.353783+0000","last_peered":"2023-10-16T12:31:06.353783+0000","last_clean":"2023-10-16T12:31:06.353783+0000","last_became_active":"2023-10-16T12:31:06.350613+0000","last_became_peered":"2023-10-16T12:31:06.350613+0000","last_unstale":"2023-10-16T12:31:06.353783+0000","last_undegraded":"2023-10-16T12:31:06.353783+0000","last_fullsized":"2023-10-16T12:31:06.353783+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:33:07.560381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3e","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.357603+0000","last_change":"2023-10-16T12:31:06.356905+0000","last_active":"2023-10-16T12:31:06.357603+0000","last_peered":"2023-10-16T12:31:06.357603+0000","last_clean":"2023-10-16T12:31:06.357603+0000","last_became_active":"2023-10-16T12:31:06.356523+0000","last_became_peered":"2023-10-16T12:31:06.356523+0000","last_unstale":"2023-10-16T12:31:06.357603+0000","last_undegraded":"2023-10-16T12:31:06.357603+0000","last_fullsized":"2023-10-16T12:31:06.357603+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:13:38.834513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.3d","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.360784+0000","last_change":"2023-10-16T12:31:06.357773+0000","last_active":"2023-10-16T12:31:06.360784+0000","last_peered":"2023-10-16T12:31:06.360784+0000","last_clean":"2023-10-16T12:31:06.360784+0000","last_became_active":"2023-10-16T12:31:06.357147+0000","last_became_peered":"2023-10-16T12:31:06.357147+0000","last_unstale":"2023-10-16T12:31:06.360784+0000","last_undegraded":"2023-10-16T12:31:06.360784+0000","last_fullsized":"2023-10-16T12:31:06.360784+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:01:42.630197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.3c","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.351425+0000","last_change":"2023-10-16T12:31:06.350748+0000","last_active":"2023-10-16T12:31:06.351425+0000","last_peered":"2023-10-16T12:31:06.351425+0000","last_clean":"2023-10-16T12:31:06.351425+0000","last_became_active":"2023-10-16T12:31:06.350379+0000","last_became_peered":"2023-10-16T12:31:06.350379+0000","last_unstale":"2023-10-16T12:31:06.351425+0000","last_undegraded":"2023-10-16T12:31:06.351425+0000","last_fullsized":"2023-10-16T12:31:06.351425+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:47:02.786943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.3b","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.357245+0000","last_change":"2023-10-16T12:31:06.356535+0000","last_active":"2023-10-16T12:31:06.357245+0000","last_peered":"2023-10-16T12:31:06.357245+0000","last_clean":"2023-10-16T12:31:06.357245+0000","last_became_active":"2023-10-16T12:31:06.356207+0000","last_became_peered":"2023-10-16T12:31:06.356207+0000","last_unstale":"2023-10-16T12:31:06.357245+0000","last_undegraded":"2023-10-16T12:31:06.357245+0000","last_fullsized":"2023-10-16T12:31:06.357245+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:07:24.792309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.3a","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.354699+0000","last_change":"2023-10-16T12:31:06.352459+0000","last_active":"2023-10-16T12:31:06.354699+0000","last_peered":"2023-10-16T12:31:06.354699+0000","last_clean":"2023-10-16T12:31:06.354699+0000","last_became_active":"2023-10-16T12:31:06.352187+0000","last_became_peered":"2023-10-16T12:31:06.352187+0000","last_unstale":"2023-10-16T12:31:06.354699+0000","last_undegraded":"2023-10-16T12:31:06.354699+0000","last_fullsized":"2023-10-16T12:31:06.354699+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:47:24.253841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.39","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.366209+0000","last_change":"2023-10-16T12:31:06.362877+0000","last_active":"2023-10-16T12:31:06.366209+0000","last_peered":"2023-10-16T12:31:06.366209+0000","last_clean":"2023-10-16T12:31:06.366209+0000","last_became_active":"2023-10-16T12:31:06.362495+0000","last_became_peered":"2023-10-16T12:31:06.362495+0000","last_unstale":"2023-10-16T12:31:06.366209+0000","last_undegraded":"2023-10-16T12:31:06.366209+0000","last_fullsized":"2023-10-16T12:31:06.366209+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:47:02.786943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.38","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.361666+0000","last_change":"2023-10-16T12:31:06.359510+0000","last_active":"2023-10-16T12:31:06.361666+0000","last_peered":"2023-10-16T12:31:06.361666+0000","last_clean":"2023-10-16T12:31:06.361666+0000","last_became_active":"2023-10-16T12:31:06.359225+0000","last_became_peered":"2023-10-16T12:31:06.359225+0000","last_unstale":"2023-10-16T12:31:06.361666+0000","last_undegraded":"2023-10-16T12:31:06.361666+0000","last_fullsized":"2023-10-16T12:31:06.361666+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:48:38.236166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.37","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.354533+0000","last_change":"2023-10-16T12:31:06.352367+0000","last_active":"2023-10-16T12:31:06.354533+0000","last_peered":"2023-10-16T12:31:06.354533+0000","last_clean":"2023-10-16T12:31:06.354533+0000","last_became_active":"2023-10-16T12:31:06.351297+0000","last_became_peered":"2023-10-16T12:31:06.351297+0000","last_unstale":"2023-10-16T12:31:06.354533+0000","last_undegraded":"2023-10-16T12:31:06.354533+0000","last_fullsized":"2023-10-16T12:31:06.354533+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:40:14.200747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.36","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.358318+0000","last_change":"2023-10-16T12:31:06.357233+0000","last_active":"2023-10-16T12:31:06.358318+0000","last_peered":"2023-10-16T12:31:06.358318+0000","last_clean":"2023-10-16T12:31:06.358318+0000","last_became_active":"2023-10-16T12:31:06.356943+0000","last_became_peered":"2023-10-16T12:31:06.356943+0000","last_unstale":"2023-10-16T12:31:06.358318+0000","last_undegraded":"2023-10-16T12:31:06.358318+0000","last_fullsized":"2023-10-16T12:31:06.358318+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:24:28.210474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.35","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.349524+0000","last_change":"2023-10-16T12:31:06.348427+0000","last_active":"2023-10-16T12:31:06.349524+0000","last_peered":"2023-10-16T12:31:06.349524+0000","last_clean":"2023-10-16T12:31:06.349524+0000","last_became_active":"2023-10-16T12:31:06.347009+0000","last_became_peered":"2023-10-16T12:31:06.347009+0000","last_unstale":"2023-10-16T12:31:06.349524+0000","last_undegraded":"2023-10-16T12:31:06.349524+0000","last_fullsized":"2023-10-16T12:31:06.349524+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:20:01.477905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.34","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.353757+0000","last_change":"2023-10-16T12:31:06.351665+0000","last_active":"2023-10-16T12:31:06.353757+0000","last_peered":"2023-10-16T12:31:06.353757+0000","last_clean":"2023-10-16T12:31:06.353757+0000","last_became_active":"2023-10-16T12:31:06.351389+0000","last_became_peered":"2023-10-16T12:31:06.351389+0000","last_unstale":"2023-10-16T12:31:06.353757+0000","last_undegraded":"2023-10-16T12:31:06.353757+0000","last_fullsized":"2023-10-16T12:31:06.353757+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:50:54.016852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.33","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.356644+0000","last_change":"2023-10-16T12:31:06.354571+0000","last_active":"2023-10-16T12:31:06.356644+0000","last_peered":"2023-10-16T12:31:06.356644+0000","last_clean":"2023-10-16T12:31:06.356644+0000","last_became_active":"2023-10-16T12:31:06.354241+0000","last_became_peered":"2023-10-16T12:31:06.354241+0000","last_unstale":"2023-10-16T12:31:06.356644+0000","last_undegraded":"2023-10-16T12:31:06.356644+0000","last_fullsized":"2023-10-16T12:31:06.356644+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:19:50.829101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.32","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.363339+0000","last_change":"2023-10-16T12:31:06.361452+0000","last_active":"2023-10-16T12:31:06.363339+0000","last_peered":"2023-10-16T12:31:06.363339+0000","last_clean":"2023-10-16T12:31:06.363339+0000","last_became_active":"2023-10-16T12:31:06.361178+0000","last_became_peered":"2023-10-16T12:31:06.361178+0000","last_unstale":"2023-10-16T12:31:06.363339+0000","last_undegraded":"2023-10-16T12:31:06.363339+0000","last_fullsized":"2023-10-16T12:31:06.363339+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:22:05.786952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.31","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.363217+0000","last_change":"2023-10-16T12:31:06.361001+0000","last_active":"2023-10-16T12:31:06.363217+0000","last_peered":"2023-10-16T12:31:06.363217+0000","last_clean":"2023-10-16T12:31:06.363217+0000","last_became_active":"2023-10-16T12:31:06.360683+0000","last_became_peered":"2023-10-16T12:31:06.360683+0000","last_unstale":"2023-10-16T12:31:06.363217+0000","last_undegraded":"2023-10-16T12:31:06.363217+0000","last_fullsized":"2023-10-16T12:31:06.363217+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:30:58.887965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.30","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.356634+0000","last_change":"2023-10-16T12:31:06.355135+0000","last_active":"2023-10-16T12:31:06.356634+0000","last_peered":"2023-10-16T12:31:06.356634+0000","last_clean":"2023-10-16T12:31:06.356634+0000","last_became_active":"2023-10-16T12:31:06.354805+0000","last_became_peered":"2023-10-16T12:31:06.354805+0000","last_unstale":"2023-10-16T12:31:06.356634+0000","last_undegraded":"2023-10-16T12:31:06.356634+0000","last_fullsized":"2023-10-16T12:31:06.356634+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:13:20.091518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.2f","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.358084+0000","last_change":"2023-10-16T12:31:06.356443+0000","last_active":"2023-10-16T12:31:06.358084+0000","last_peered":"2023-10-16T12:31:06.358084+0000","last_clean":"2023-10-16T12:31:06.358084+0000","last_became_active":"2023-10-16T12:31:06.356084+0000","last_became_peered":"2023-10-16T12:31:06.356084+0000","last_unstale":"2023-10-16T12:31:06.358084+0000","last_undegraded":"2023-10-16T12:31:06.358084+0000","last_fullsized":"2023-10-16T12:31:06.358084+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:34:07.904108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.2e","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.360176+0000","last_change":"2023-10-16T12:31:06.359683+0000","last_active":"2023-10-16T12:31:06.360176+0000","last_peered":"2023-10-16T12:31:06.360176+0000","last_clean":"2023-10-16T12:31:06.360176+0000","last_became_active":"2023-10-16T12:31:06.359310+0000","last_became_peered":"2023-10-16T12:31:06.359310+0000","last_unstale":"2023-10-16T12:31:06.360176+0000","last_undegraded":"2023-10-16T12:31:06.360176+0000","last_fullsized":"2023-10-16T12:31:06.360176+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:40:12.392133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.2d","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.364934+0000","last_change":"2023-10-16T12:31:06.362863+0000","last_active":"2023-10-16T12:31:06.364934+0000","last_peered":"2023-10-16T12:31:06.364934+0000","last_clean":"2023-10-16T12:31:06.364934+0000","last_became_active":"2023-10-16T12:31:06.362299+0000","last_became_peered":"2023-10-16T12:31:06.362299+0000","last_unstale":"2023-10-16T12:31:06.364934+0000","last_undegraded":"2023-10-16T12:31:06.364934+0000","last_fullsized":"2023-10-16T12:31:06.364934+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:03:05.365328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.2c","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.363388+0000","last_change":"2023-10-16T12:31:06.360910+0000","last_active":"2023-10-16T12:31:06.363388+0000","last_peered":"2023-10-16T12:31:06.363388+0000","last_clean":"2023-10-16T12:31:06.363388+0000","last_became_active":"2023-10-16T12:31:06.360550+0000","last_became_peered":"2023-10-16T12:31:06.360550+0000","last_unstale":"2023-10-16T12:31:06.363388+0000","last_undegraded":"2023-10-16T12:31:06.363388+0000","last_fullsized":"2023-10-16T12:31:06.363388+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:14:05.652793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.2b","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.356234+0000","last_change":"2023-10-16T12:31:06.355026+0000","last_active":"2023-10-16T12:31:06.356234+0000","last_peered":"2023-10-16T12:31:06.356234+0000","last_clean":"2023-10-16T12:31:06.356234+0000","last_became_active":"2023-10-16T12:31:06.354578+0000","last_became_peered":"2023-10-16T12:31:06.354578+0000","last_unstale":"2023-10-16T12:31:06.356234+0000","last_undegraded":"2023-10-16T12:31:06.356234+0000","last_fullsized":"2023-10-16T12:31:06.356234+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:47:20.460113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.2a","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.345028+0000","last_change":"2023-10-16T12:31:06.342061+0000","last_active":"2023-10-16T12:31:06.345028+0000","last_peered":"2023-10-16T12:31:06.345028+0000","last_clean":"2023-10-16T12:31:06.345028+0000","last_became_active":"2023-10-16T12:31:06.341705+0000","last_became_peered":"2023-10-16T12:31:06.341705+0000","last_unstale":"2023-10-16T12:31:06.345028+0000","last_undegraded":"2023-10-16T12:31:06.345028+0000","last_fullsized":"2023-10-16T12:31:06.345028+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:40:54.188746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.29","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.364844+0000","last_change":"2023-10-16T12:31:06.363571+0000","last_active":"2023-10-16T12:31:06.364844+0000","last_peered":"2023-10-16T12:31:06.364844+0000","last_clean":"2023-10-16T12:31:06.364844+0000","last_became_active":"2023-10-16T12:31:06.363262+0000","last_became_peered":"2023-10-16T12:31:06.363262+0000","last_unstale":"2023-10-16T12:31:06.364844+0000","last_undegraded":"2023-10-16T12:31:06.364844+0000","last_fullsized":"2023-10-16T12:31:06.364844+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:56:10.808368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.28","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.354362+0000","last_change":"2023-10-16T12:31:06.352033+0000","last_active":"2023-10-16T12:31:06.354362+0000","last_peered":"2023-10-16T12:31:06.354362+0000","last_clean":"2023-10-16T12:31:06.354362+0000","last_became_active":"2023-10-16T12:31:06.351735+0000","last_became_peered":"2023-10-16T12:31:06.351735+0000","last_unstale":"2023-10-16T12:31:06.354362+0000","last_undegraded":"2023-10-16T12:31:06.354362+0000","last_fullsized":"2023-10-16T12:31:06.354362+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:36:31.736567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.27","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.361129+0000","last_change":"2023-10-16T12:31:06.359735+0000","last_active":"2023-10-16T12:31:06.361129+0000","last_peered":"2023-10-16T12:31:06.361129+0000","last_clean":"2023-10-16T12:31:06.361129+0000","last_became_active":"2023-10-16T12:31:06.359480+0000","last_became_peered":"2023-10-16T12:31:06.359480+0000","last_unstale":"2023-10-16T12:31:06.361129+0000","last_undegraded":"2023-10-16T12:31:06.361129+0000","last_fullsized":"2023-10-16T12:31:06.361129+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:47:50.218838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.26","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.361866+0000","last_change":"2023-10-16T12:31:06.361258+0000","last_active":"2023-10-16T12:31:06.361866+0000","last_peered":"2023-10-16T12:31:06.361866+0000","last_clean":"2023-10-16T12:31:06.361866+0000","last_became_active":"2023-10-16T12:31:06.360749+0000","last_became_peered":"2023-10-16T12:31:06.360749+0000","last_unstale":"2023-10-16T12:31:06.361866+0000","last_undegraded":"2023-10-16T12:31:06.361866+0000","last_fullsized":"2023-10-16T12:31:06.361866+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:18:49.322758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.25","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.364540+0000","last_change":"2023-10-16T12:31:06.360494+0000","last_active":"2023-10-16T12:31:06.364540+0000","last_peered":"2023-10-16T12:31:06.364540+0000","last_clean":"2023-10-16T12:31:06.364540+0000","last_became_active":"2023-10-16T12:31:06.360162+0000","last_became_peered":"2023-10-16T12:31:06.360162+0000","last_unstale":"2023-10-16T12:31:06.364540+0000","last_undegraded":"2023-10-16T12:31:06.364540+0000","last_fullsized":"2023-10-16T12:31:06.364540+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:44:22.408445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.24","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.358310+0000","last_change":"2023-10-16T12:31:06.356846+0000","last_active":"2023-10-16T12:31:06.358310+0000","last_peered":"2023-10-16T12:31:06.358310+0000","last_clean":"2023-10-16T12:31:06.358310+0000","last_became_active":"2023-10-16T12:31:06.356538+0000","last_became_peered":"2023-10-16T12:31:06.356538+0000","last_unstale":"2023-10-16T12:31:06.358310+0000","last_undegraded":"2023-10-16T12:31:06.358310+0000","last_fullsized":"2023-10-16T12:31:06.358310+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:24:31.850753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.23","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.356425+0000","last_change":"2023-10-16T12:31:06.355320+0000","last_active":"2023-10-16T12:31:06.356425+0000","last_peered":"2023-10-16T12:31:06.356425+0000","last_clean":"2023-10-16T12:31:06.356425+0000","last_became_active":"2023-10-16T12:31:06.354848+0000","last_became_peered":"2023-10-16T12:31:06.354848+0000","last_unstale":"2023-10-16T12:31:06.356425+0000","last_undegraded":"2023-10-16T12:31:06.356425+0000","last_fullsized":"2023-10-16T12:31:06.356425+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:26:59.264313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.22","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.356900+0000","last_change":"2023-10-16T12:31:06.356104+0000","last_active":"2023-10-16T12:31:06.356900+0000","last_peered":"2023-10-16T12:31:06.356900+0000","last_clean":"2023-10-16T12:31:06.356900+0000","last_became_active":"2023-10-16T12:31:06.355811+0000","last_became_peered":"2023-10-16T12:31:06.355811+0000","last_unstale":"2023-10-16T12:31:06.356900+0000","last_undegraded":"2023-10-16T12:31:06.356900+0000","last_fullsized":"2023-10-16T12:31:06.356900+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:00:09.057516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.21","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.359003+0000","last_change":"2023-10-16T12:31:06.356103+0000","last_active":"2023-10-16T12:31:06.359003+0000","last_peered":"2023-10-16T12:31:06.359003+0000","last_clean":"2023-10-16T12:31:06.359003+0000","last_became_active":"2023-10-16T12:31:06.355828+0000","last_became_peered":"2023-10-16T12:31:06.355828+0000","last_unstale":"2023-10-16T12:31:06.359003+0000","last_undegraded":"2023-10-16T12:31:06.359003+0000","last_fullsized":"2023-10-16T12:31:06.359003+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:40:14.200747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.20","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.353675+0000","last_change":"2023-10-16T12:31:06.346338+0000","last_active":"2023-10-16T12:31:06.353675+0000","last_peered":"2023-10-16T12:31:06.353675+0000","last_clean":"2023-10-16T12:31:06.353675+0000","last_became_active":"2023-10-16T12:31:06.346056+0000","last_became_peered":"2023-10-16T12:31:06.346056+0000","last_unstale":"2023-10-16T12:31:06.353675+0000","last_undegraded":"2023-10-16T12:31:06.353675+0000","last_fullsized":"2023-10-16T12:31:06.353675+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:14:16.142686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.361654+0000","last_change":"2023-10-16T12:31:06.358760+0000","last_active":"2023-10-16T12:31:06.361654+0000","last_peered":"2023-10-16T12:31:06.361654+0000","last_clean":"2023-10-16T12:31:06.361654+0000","last_became_active":"2023-10-16T12:31:06.358438+0000","last_became_peered":"2023-10-16T12:31:06.358438+0000","last_unstale":"2023-10-16T12:31:06.361654+0000","last_undegraded":"2023-10-16T12:31:06.361654+0000","last_fullsized":"2023-10-16T12:31:06.361654+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:34:07.904108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1f","version":"155'4","reported_seq":15,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.425714+0000","last_change":"2023-10-16T12:31:05.433796+0000","last_active":"2023-10-16T12:31:06.425714+0000","last_peered":"2023-10-16T12:31:06.425714+0000","last_clean":"2023-10-16T12:31:06.425714+0000","last_became_active":"2023-10-16T12:31:05.433502+0000","last_became_peered":"2023-10-16T12:31:05.433502+0000","last_unstale":"2023-10-16T12:31:06.425714+0000","last_undegraded":"2023-10-16T12:31:06.425714+0000","last_fullsized":"2023-10-16T12:31:06.425714+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_clean_scrub_stamp":"2023-10-16T12:31:04.400465+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:13:19.086520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.344998+0000","last_change":"2023-10-16T12:31:06.341593+0000","last_active":"2023-10-16T12:31:06.344998+0000","last_peered":"2023-10-16T12:31:06.344998+0000","last_clean":"2023-10-16T12:31:06.344998+0000","last_became_active":"2023-10-16T12:31:06.341243+0000","last_became_peered":"2023-10-16T12:31:06.341243+0000","last_unstale":"2023-10-16T12:31:06.344998+0000","last_undegraded":"2023-10-16T12:31:06.344998+0000","last_fullsized":"2023-10-16T12:31:06.344998+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:59:55.880992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1e","version":"155'1","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.422987+0000","last_change":"2023-10-16T12:31:05.434867+0000","last_active":"2023-10-16T12:31:06.422987+0000","last_peered":"2023-10-16T12:31:06.422987+0000","last_clean":"2023-10-16T12:31:06.422987+0000","last_became_active":"2023-10-16T12:31:05.434596+0000","last_became_peered":"2023-10-16T12:31:05.434596+0000","last_unstale":"2023-10-16T12:31:06.422987+0000","last_undegraded":"2023-10-16T12:31:06.422987+0000","last_fullsized":"2023-10-16T12:31:06.422987+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_clean_scrub_stamp":"2023-10-16T12:31:04.400465+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:55:42.439555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1a","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.364833+0000","last_change":"2023-10-16T12:31:06.363592+0000","last_active":"2023-10-16T12:31:06.364833+0000","last_peered":"2023-10-16T12:31:06.364833+0000","last_clean":"2023-10-16T12:31:06.364833+0000","last_became_active":"2023-10-16T12:31:06.363290+0000","last_became_peered":"2023-10-16T12:31:06.363290+0000","last_unstale":"2023-10-16T12:31:06.364833+0000","last_undegraded":"2023-10-16T12:31:06.364833+0000","last_fullsized":"2023-10-16T12:31:06.364833+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:58:04.540273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1d","version":"155'2","reported_seq":13,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.423046+0000","last_change":"2023-10-16T12:31:05.428337+0000","last_active":"2023-10-16T12:31:06.423046+0000","last_peered":"2023-10-16T12:31:06.423046+0000","last_clean":"2023-10-16T12:31:06.423046+0000","last_became_active":"2023-10-16T12:31:05.428026+0000","last_became_peered":"2023-10-16T12:31:05.428026+0000","last_unstale":"2023-10-16T12:31:06.423046+0000","last_undegraded":"2023-10-16T12:31:06.423046+0000","last_fullsized":"2023-10-16T12:31:06.423046+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_clean_scrub_stamp":"2023-10-16T12:31:04.400465+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:14:33.240346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1b","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.363178+0000","last_change":"2023-10-16T12:31:06.360877+0000","last_active":"2023-10-16T12:31:06.363178+0000","last_peered":"2023-10-16T12:31:06.363178+0000","last_clean":"2023-10-16T12:31:06.363178+0000","last_became_active":"2023-10-16T12:31:06.360545+0000","last_became_peered":"2023-10-16T12:31:06.360545+0000","last_unstale":"2023-10-16T12:31:06.363178+0000","last_undegraded":"2023-10-16T12:31:06.363178+0000","last_fullsized":"2023-10-16T12:31:06.363178+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:19:25.399025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1c","version":"155'2","reported_seq":13,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.425469+0000","last_change":"2023-10-16T12:31:05.427637+0000","last_active":"2023-10-16T12:31:06.425469+0000","last_peered":"2023-10-16T12:31:06.425469+0000","last_clean":"2023-10-16T12:31:06.425469+0000","last_became_active":"2023-10-16T12:31:05.427182+0000","last_became_peered":"2023-10-16T12:31:05.427182+0000","last_unstale":"2023-10-16T12:31:06.425469+0000","last_undegraded":"2023-10-16T12:31:06.425469+0000","last_fullsized":"2023-10-16T12:31:06.425469+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_clean_scrub_stamp":"2023-10-16T12:31:04.400465+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:40:25.407994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.359156+0000","last_change":"2023-10-16T12:31:06.356596+0000","last_active":"2023-10-16T12:31:06.359156+0000","last_peered":"2023-10-16T12:31:06.359156+0000","last_clean":"2023-10-16T12:31:06.359156+0000","last_became_active":"2023-10-16T12:31:06.355131+0000","last_became_peered":"2023-10-16T12:31:06.355131+0000","last_unstale":"2023-10-16T12:31:06.359156+0000","last_undegraded":"2023-10-16T12:31:06.359156+0000","last_fullsized":"2023-10-16T12:31:06.359156+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:37:35.697306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1b","version":"155'1","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.426257+0000","last_change":"2023-10-16T12:31:05.432941+0000","last_active":"2023-10-16T12:31:06.426257+0000","last_peered":"2023-10-16T12:31:06.426257+0000","last_clean":"2023-10-16T12:31:06.426257+0000","last_became_active":"2023-10-16T12:31:05.432608+0000","last_became_peered":"2023-10-16T12:31:05.432608+0000","last_unstale":"2023-10-16T12:31:06.426257+0000","last_undegraded":"2023-10-16T12:31:06.426257+0000","last_fullsized":"2023-10-16T12:31:06.426257+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_clean_scrub_stamp":"2023-10-16T12:31:04.400465+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:38:59.295383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1d","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.359830+0000","last_change":"2023-10-16T12:31:06.355659+0000","last_active":"2023-10-16T12:31:06.359830+0000","last_peered":"2023-10-16T12:31:06.359830+0000","last_clean":"2023-10-16T12:31:06.359830+0000","last_became_active":"2023-10-16T12:31:06.355357+0000","last_became_peered":"2023-10-16T12:31:06.355357+0000","last_unstale":"2023-10-16T12:31:06.359830+0000","last_undegraded":"2023-10-16T12:31:06.359830+0000","last_fullsized":"2023-10-16T12:31:06.359830+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:47:24.253841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1a","version":"155'1","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.422988+0000","last_change":"2023-10-16T12:31:05.426078+0000","last_active":"2023-10-16T12:31:06.422988+0000","last_peered":"2023-10-16T12:31:06.422988+0000","last_clean":"2023-10-16T12:31:06.422988+0000","last_became_active":"2023-10-16T12:31:05.425793+0000","last_became_peered":"2023-10-16T12:31:05.425793+0000","last_unstale":"2023-10-16T12:31:06.422988+0000","last_undegraded":"2023-10-16T12:31:06.422988+0000","last_fullsized":"2023-10-16T12:31:06.422988+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_clean_scrub_stamp":"2023-10-16T12:31:04.400465+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:53:34.201546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.363454+0000","last_change":"2023-10-16T12:31:06.361606+0000","last_active":"2023-10-16T12:31:06.363454+0000","last_peered":"2023-10-16T12:31:06.363454+0000","last_clean":"2023-10-16T12:31:06.363454+0000","last_became_active":"2023-10-16T12:31:06.361310+0000","last_became_peered":"2023-10-16T12:31:06.361310+0000","last_unstale":"2023-10-16T12:31:06.363454+0000","last_undegraded":"2023-10-16T12:31:06.363454+0000","last_fullsized":"2023-10-16T12:31:06.363454+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:29:31.557647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.346716+0000","last_change":"2023-10-16T12:31:05.436170+0000","last_active":"2023-10-16T12:31:06.346716+0000","last_peered":"2023-10-16T12:31:06.346716+0000","last_clean":"2023-10-16T12:31:06.346716+0000","last_became_active":"2023-10-16T12:31:05.435868+0000","last_became_peered":"2023-10-16T12:31:05.435868+0000","last_unstale":"2023-10-16T12:31:06.346716+0000","last_undegraded":"2023-10-16T12:31:06.346716+0000","last_fullsized":"2023-10-16T12:31:06.346716+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_clean_scrub_stamp":"2023-10-16T12:31:04.400465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:28:52.884000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1f","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.363469+0000","last_change":"2023-10-16T12:31:06.358925+0000","last_active":"2023-10-16T12:31:06.363469+0000","last_peered":"2023-10-16T12:31:06.363469+0000","last_clean":"2023-10-16T12:31:06.363469+0000","last_became_active":"2023-10-16T12:31:06.358375+0000","last_became_peered":"2023-10-16T12:31:06.358375+0000","last_unstale":"2023-10-16T12:31:06.363469+0000","last_undegraded":"2023-10-16T12:31:06.363469+0000","last_fullsized":"2023-10-16T12:31:06.363469+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:07:24.792309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.344475+0000","last_change":"2023-10-16T12:31:05.438603+0000","last_active":"2023-10-16T12:31:06.344475+0000","last_peered":"2023-10-16T12:31:06.344475+0000","last_clean":"2023-10-16T12:31:06.344475+0000","last_became_active":"2023-10-16T12:31:05.438302+0000","last_became_peered":"2023-10-16T12:31:05.438302+0000","last_unstale":"2023-10-16T12:31:06.344475+0000","last_undegraded":"2023-10-16T12:31:06.344475+0000","last_fullsized":"2023-10-16T12:31:06.344475+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_clean_scrub_stamp":"2023-10-16T12:31:04.400465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:36:30.731569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.355452+0000","last_change":"2023-10-16T12:31:06.354175+0000","last_active":"2023-10-16T12:31:06.355452+0000","last_peered":"2023-10-16T12:31:06.355452+0000","last_clean":"2023-10-16T12:31:06.355452+0000","last_became_active":"2023-10-16T12:31:06.353721+0000","last_became_peered":"2023-10-16T12:31:06.353721+0000","last_unstale":"2023-10-16T12:31:06.355452+0000","last_undegraded":"2023-10-16T12:31:06.355452+0000","last_fullsized":"2023-10-16T12:31:06.355452+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:35:32.424425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.339432+0000","last_change":"2023-10-16T12:31:05.436538+0000","last_active":"2023-10-16T12:31:06.339432+0000","last_peered":"2023-10-16T12:31:06.339432+0000","last_clean":"2023-10-16T12:31:06.339432+0000","last_became_active":"2023-10-16T12:31:05.436249+0000","last_became_peered":"2023-10-16T12:31:05.436249+0000","last_unstale":"2023-10-16T12:31:06.339432+0000","last_undegraded":"2023-10-16T12:31:06.339432+0000","last_fullsized":"2023-10-16T12:31:06.339432+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_clean_scrub_stamp":"2023-10-16T12:31:04.400465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:53:02.482566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.354312+0000","last_change":"2023-10-16T12:31:06.347358+0000","last_active":"2023-10-16T12:31:06.354312+0000","last_peered":"2023-10-16T12:31:06.354312+0000","last_clean":"2023-10-16T12:31:06.354312+0000","last_became_active":"2023-10-16T12:31:06.347082+0000","last_became_peered":"2023-10-16T12:31:06.347082+0000","last_unstale":"2023-10-16T12:31:06.354312+0000","last_undegraded":"2023-10-16T12:31:06.354312+0000","last_fullsized":"2023-10-16T12:31:06.354312+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:39:00.300381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.337990+0000","last_change":"2023-10-16T12:31:05.433498+0000","last_active":"2023-10-16T12:31:06.337990+0000","last_peered":"2023-10-16T12:31:06.337990+0000","last_clean":"2023-10-16T12:31:06.337990+0000","last_became_active":"2023-10-16T12:31:05.433210+0000","last_became_peered":"2023-10-16T12:31:05.433210+0000","last_unstale":"2023-10-16T12:31:06.337990+0000","last_undegraded":"2023-10-16T12:31:06.337990+0000","last_fullsized":"2023-10-16T12:31:06.337990+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_clean_scrub_stamp":"2023-10-16T12:31:04.400465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:48:36.284683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.351556+0000","last_change":"2023-10-16T12:31:06.350328+0000","last_active":"2023-10-16T12:31:06.351556+0000","last_peered":"2023-10-16T12:31:06.351556+0000","last_clean":"2023-10-16T12:31:06.351556+0000","last_became_active":"2023-10-16T12:31:06.350014+0000","last_became_peered":"2023-10-16T12:31:06.350014+0000","last_unstale":"2023-10-16T12:31:06.351556+0000","last_undegraded":"2023-10-16T12:31:06.351556+0000","last_fullsized":"2023-10-16T12:31:06.351556+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:18:23.655771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.15","version":"155'3","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.424976+0000","last_change":"2023-10-16T12:31:05.432652+0000","last_active":"2023-10-16T12:31:06.424976+0000","last_peered":"2023-10-16T12:31:06.424976+0000","last_clean":"2023-10-16T12:31:06.424976+0000","last_became_active":"2023-10-16T12:31:05.432320+0000","last_became_peered":"2023-10-16T12:31:05.432320+0000","last_unstale":"2023-10-16T12:31:06.424976+0000","last_undegraded":"2023-10-16T12:31:06.424976+0000","last_fullsized":"2023-10-16T12:31:06.424976+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_clean_scrub_stamp":"2023-10-16T12:31:04.400465+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:27:31.708027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.13","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.361773+0000","last_change":"2023-10-16T12:31:06.360763+0000","last_active":"2023-10-16T12:31:06.361773+0000","last_peered":"2023-10-16T12:31:06.361773+0000","last_clean":"2023-10-16T12:31:06.361773+0000","last_became_active":"2023-10-16T12:31:06.360390+0000","last_became_peered":"2023-10-16T12:31:06.360390+0000","last_unstale":"2023-10-16T12:31:06.361773+0000","last_undegraded":"2023-10-16T12:31:06.361773+0000","last_fullsized":"2023-10-16T12:31:06.361773+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:45:22.357913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.14","version":"155'6","reported_seq":17,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.428288+0000","last_change":"2023-10-16T12:31:05.436216+0000","last_active":"2023-10-16T12:31:06.428288+0000","last_peered":"2023-10-16T12:31:06.428288+0000","last_clean":"2023-10-16T12:31:06.428288+0000","last_became_active":"2023-10-16T12:31:05.435894+0000","last_became_peered":"2023-10-16T12:31:05.435894+0000","last_unstale":"2023-10-16T12:31:06.428288+0000","last_undegraded":"2023-10-16T12:31:06.428288+0000","last_fullsized":"2023-10-16T12:31:06.428288+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_clean_scrub_stamp":"2023-10-16T12:31:04.400465+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:44:10.145999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.14","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.357842+0000","last_change":"2023-10-16T12:31:06.351283+0000","last_active":"2023-10-16T12:31:06.357842+0000","last_peered":"2023-10-16T12:31:06.357842+0000","last_clean":"2023-10-16T12:31:06.357842+0000","last_became_active":"2023-10-16T12:31:06.350288+0000","last_became_peered":"2023-10-16T12:31:06.350288+0000","last_unstale":"2023-10-16T12:31:06.357842+0000","last_undegraded":"2023-10-16T12:31:06.357842+0000","last_fullsized":"2023-10-16T12:31:06.357842+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:43:20.718343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.13","version":"155'2","reported_seq":13,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.426392+0000","last_change":"2023-10-16T12:31:05.433808+0000","last_active":"2023-10-16T12:31:06.426392+0000","last_peered":"2023-10-16T12:31:06.426392+0000","last_clean":"2023-10-16T12:31:06.426392+0000","last_became_active":"2023-10-16T12:31:05.433528+0000","last_became_peered":"2023-10-16T12:31:05.433528+0000","last_unstale":"2023-10-16T12:31:06.426392+0000","last_undegraded":"2023-10-16T12:31:06.426392+0000","last_fullsized":"2023-10-16T12:31:06.426392+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_clean_scrub_stamp":"2023-10-16T12:31:04.400465+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:14:00.280428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.363505+0000","last_change":"2023-10-16T12:31:06.358586+0000","last_active":"2023-10-16T12:31:06.363505+0000","last_peered":"2023-10-16T12:31:06.363505+0000","last_clean":"2023-10-16T12:31:06.363505+0000","last_became_active":"2023-10-16T12:31:06.358127+0000","last_became_peered":"2023-10-16T12:31:06.358127+0000","last_unstale":"2023-10-16T12:31:06.363505+0000","last_undegraded":"2023-10-16T12:31:06.363505+0000","last_fullsized":"2023-10-16T12:31:06.363505+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:55:02.716579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.344255+0000","last_change":"2023-10-16T12:31:05.431094+0000","last_active":"2023-10-16T12:31:06.344255+0000","last_peered":"2023-10-16T12:31:06.344255+0000","last_clean":"2023-10-16T12:31:06.344255+0000","last_became_active":"2023-10-16T12:31:05.430795+0000","last_became_peered":"2023-10-16T12:31:05.430795+0000","last_unstale":"2023-10-16T12:31:06.344255+0000","last_undegraded":"2023-10-16T12:31:06.344255+0000","last_fullsized":"2023-10-16T12:31:06.344255+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_clean_scrub_stamp":"2023-10-16T12:31:04.400465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:27:34.917740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.357191+0000","last_change":"2023-10-16T12:31:06.355134+0000","last_active":"2023-10-16T12:31:06.357191+0000","last_peered":"2023-10-16T12:31:06.357191+0000","last_clean":"2023-10-16T12:31:06.357191+0000","last_became_active":"2023-10-16T12:31:06.354652+0000","last_became_peered":"2023-10-16T12:31:06.354652+0000","last_unstale":"2023-10-16T12:31:06.357191+0000","last_undegraded":"2023-10-16T12:31:06.357191+0000","last_fullsized":"2023-10-16T12:31:06.357191+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:14:31.147453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.10","version":"155'8","reported_seq":19,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.426550+0000","last_change":"2023-10-16T12:31:05.433311+0000","last_active":"2023-10-16T12:31:06.426550+0000","last_peered":"2023-10-16T12:31:06.426550+0000","last_clean":"2023-10-16T12:31:06.426550+0000","last_became_active":"2023-10-16T12:31:05.433012+0000","last_became_peered":"2023-10-16T12:31:05.433012+0000","last_unstale":"2023-10-16T12:31:06.426550+0000","last_undegraded":"2023-10-16T12:31:06.426550+0000","last_fullsized":"2023-10-16T12:31:06.426550+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_clean_scrub_stamp":"2023-10-16T12:31:04.400465+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:40:24.125227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.359180+0000","last_change":"2023-10-16T12:31:06.357784+0000","last_active":"2023-10-16T12:31:06.359180+0000","last_peered":"2023-10-16T12:31:06.359180+0000","last_clean":"2023-10-16T12:31:06.359180+0000","last_became_active":"2023-10-16T12:31:06.357171+0000","last_became_peered":"2023-10-16T12:31:06.357171+0000","last_unstale":"2023-10-16T12:31:06.359180+0000","last_undegraded":"2023-10-16T12:31:06.359180+0000","last_fullsized":"2023-10-16T12:31:06.359180+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T12:52:32.915036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.f","version":"155'1","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.422117+0000","last_change":"2023-10-16T12:31:05.433277+0000","last_active":"2023-10-16T12:31:06.422117+0000","last_peered":"2023-10-16T12:31:06.422117+0000","last_clean":"2023-10-16T12:31:06.422117+0000","last_became_active":"2023-10-16T12:31:05.432971+0000","last_became_peered":"2023-10-16T12:31:05.432971+0000","last_unstale":"2023-10-16T12:31:06.422117+0000","last_undegraded":"2023-10-16T12:31:06.422117+0000","last_fullsized":"2023-10-16T12:31:06.422117+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_clean_scrub_stamp":"2023-10-16T12:31:04.400465+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:40:25.407994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.366188+0000","last_change":"2023-10-16T12:31:06.362808+0000","last_active":"2023-10-16T12:31:06.366188+0000","last_peered":"2023-10-16T12:31:06.366188+0000","last_clean":"2023-10-16T12:31:06.366188+0000","last_became_active":"2023-10-16T12:31:06.362208+0000","last_became_peered":"2023-10-16T12:31:06.362208+0000","last_unstale":"2023-10-16T12:31:06.366188+0000","last_undegraded":"2023-10-16T12:31:06.366188+0000","last_fullsized":"2023-10-16T12:31:06.366188+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:22:48.953437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.e","version":"155'2","reported_seq":13,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.425228+0000","last_change":"2023-10-16T12:31:05.433817+0000","last_active":"2023-10-16T12:31:06.425228+0000","last_peered":"2023-10-16T12:31:06.425228+0000","last_clean":"2023-10-16T12:31:06.425228+0000","last_became_active":"2023-10-16T12:31:05.433523+0000","last_became_peered":"2023-10-16T12:31:05.433523+0000","last_unstale":"2023-10-16T12:31:06.425228+0000","last_undegraded":"2023-10-16T12:31:06.425228+0000","last_fullsized":"2023-10-16T12:31:06.425228+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_clean_scrub_stamp":"2023-10-16T12:31:04.400465+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:14:24.203363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.364394+0000","last_change":"2023-10-16T12:31:06.360271+0000","last_active":"2023-10-16T12:31:06.364394+0000","last_peered":"2023-10-16T12:31:06.364394+0000","last_clean":"2023-10-16T12:31:06.364394+0000","last_became_active":"2023-10-16T12:31:06.359911+0000","last_became_peered":"2023-10-16T12:31:06.359911+0000","last_unstale":"2023-10-16T12:31:06.364394+0000","last_undegraded":"2023-10-16T12:31:06.364394+0000","last_fullsized":"2023-10-16T12:31:06.364394+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:53:03.487564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.d","version":"155'1","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.425190+0000","last_change":"2023-10-16T12:31:05.435666+0000","last_active":"2023-10-16T12:31:06.425190+0000","last_peered":"2023-10-16T12:31:06.425190+0000","last_clean":"2023-10-16T12:31:06.425190+0000","last_became_active":"2023-10-16T12:31:05.435334+0000","last_became_peered":"2023-10-16T12:31:05.435334+0000","last_unstale":"2023-10-16T12:31:06.425190+0000","last_undegraded":"2023-10-16T12:31:06.425190+0000","last_fullsized":"2023-10-16T12:31:06.425190+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_clean_scrub_stamp":"2023-10-16T12:31:04.400465+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:41:02.140573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.b","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.355165+0000","last_change":"2023-10-16T12:31:06.351952+0000","last_active":"2023-10-16T12:31:06.355165+0000","last_peered":"2023-10-16T12:31:06.355165+0000","last_clean":"2023-10-16T12:31:06.355165+0000","last_became_active":"2023-10-16T12:31:06.351610+0000","last_became_peered":"2023-10-16T12:31:06.351610+0000","last_unstale":"2023-10-16T12:31:06.355165+0000","last_undegraded":"2023-10-16T12:31:06.355165+0000","last_fullsized":"2023-10-16T12:31:06.355165+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:30:03.812820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.349950+0000","last_change":"2023-10-16T12:31:05.425457+0000","last_active":"2023-10-16T12:31:06.349950+0000","last_peered":"2023-10-16T12:31:06.349950+0000","last_clean":"2023-10-16T12:31:06.349950+0000","last_became_active":"2023-10-16T12:31:05.425153+0000","last_became_peered":"2023-10-16T12:31:05.425153+0000","last_unstale":"2023-10-16T12:31:06.349950+0000","last_undegraded":"2023-10-16T12:31:06.349950+0000","last_fullsized":"2023-10-16T12:31:06.349950+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_clean_scrub_stamp":"2023-10-16T12:31:04.400465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:19:15.123877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.357306+0000","last_change":"2023-10-16T12:31:06.355992+0000","last_active":"2023-10-16T12:31:06.357306+0000","last_peered":"2023-10-16T12:31:06.357306+0000","last_clean":"2023-10-16T12:31:06.357306+0000","last_became_active":"2023-10-16T12:31:06.355672+0000","last_became_peered":"2023-10-16T12:31:06.355672+0000","last_unstale":"2023-10-16T12:31:06.357306+0000","last_undegraded":"2023-10-16T12:31:06.357306+0000","last_fullsized":"2023-10-16T12:31:06.357306+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:22:48.953437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.b","version":"155'2","reported_seq":13,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.424813+0000","last_change":"2023-10-16T12:31:05.427181+0000","last_active":"2023-10-16T12:31:06.424813+0000","last_peered":"2023-10-16T12:31:06.424813+0000","last_clean":"2023-10-16T12:31:06.424813+0000","last_became_active":"2023-10-16T12:31:05.426885+0000","last_became_peered":"2023-10-16T12:31:05.426885+0000","last_unstale":"2023-10-16T12:31:06.424813+0000","last_undegraded":"2023-10-16T12:31:06.424813+0000","last_fullsized":"2023-10-16T12:31:06.424813+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_clean_scrub_stamp":"2023-10-16T12:31:04.400465+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:35:23.887227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.d","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.363196+0000","last_change":"2023-10-16T12:31:06.361111+0000","last_active":"2023-10-16T12:31:06.363196+0000","last_peered":"2023-10-16T12:31:06.363196+0000","last_clean":"2023-10-16T12:31:06.363196+0000","last_became_active":"2023-10-16T12:31:06.360764+0000","last_became_peered":"2023-10-16T12:31:06.360764+0000","last_unstale":"2023-10-16T12:31:06.363196+0000","last_undegraded":"2023-10-16T12:31:06.363196+0000","last_fullsized":"2023-10-16T12:31:06.363196+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:51:15.459519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.337935+0000","last_change":"2023-10-16T12:31:05.418528+0000","last_active":"2023-10-16T12:31:06.337935+0000","last_peered":"2023-10-16T12:31:06.337935+0000","last_clean":"2023-10-16T12:31:06.337935+0000","last_became_active":"2023-10-16T12:31:05.418144+0000","last_became_peered":"2023-10-16T12:31:05.418144+0000","last_unstale":"2023-10-16T12:31:06.337935+0000","last_undegraded":"2023-10-16T12:31:06.337935+0000","last_fullsized":"2023-10-16T12:31:06.337935+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_clean_scrub_stamp":"2023-10-16T12:31:04.400465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:41:02.978606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.355354+0000","last_change":"2023-10-16T12:31:06.352614+0000","last_active":"2023-10-16T12:31:06.355354+0000","last_peered":"2023-10-16T12:31:06.355354+0000","last_clean":"2023-10-16T12:31:06.355354+0000","last_became_active":"2023-10-16T12:31:06.352316+0000","last_became_peered":"2023-10-16T12:31:06.352316+0000","last_unstale":"2023-10-16T12:31:06.355354+0000","last_undegraded":"2023-10-16T12:31:06.355354+0000","last_fullsized":"2023-10-16T12:31:06.355354+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:44:22.408445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.348093+0000","last_change":"2023-10-16T12:31:05.427444+0000","last_active":"2023-10-16T12:31:06.348093+0000","last_peered":"2023-10-16T12:31:06.348093+0000","last_clean":"2023-10-16T12:31:06.348093+0000","last_became_active":"2023-10-16T12:31:05.427137+0000","last_became_peered":"2023-10-16T12:31:05.427137+0000","last_unstale":"2023-10-16T12:31:06.348093+0000","last_undegraded":"2023-10-16T12:31:06.348093+0000","last_fullsized":"2023-10-16T12:31:06.348093+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_clean_scrub_stamp":"2023-10-16T12:31:04.400465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:29:30.552649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.353318+0000","last_change":"2023-10-16T12:31:06.352354+0000","last_active":"2023-10-16T12:31:06.353318+0000","last_peered":"2023-10-16T12:31:06.353318+0000","last_clean":"2023-10-16T12:31:06.353318+0000","last_became_active":"2023-10-16T12:31:06.352002+0000","last_became_peered":"2023-10-16T12:31:06.352002+0000","last_unstale":"2023-10-16T12:31:06.353318+0000","last_undegraded":"2023-10-16T12:31:06.353318+0000","last_fullsized":"2023-10-16T12:31:06.353318+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:15:25.187373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.346253+0000","last_change":"2023-10-16T12:31:05.433973+0000","last_active":"2023-10-16T12:31:06.346253+0000","last_peered":"2023-10-16T12:31:06.346253+0000","last_clean":"2023-10-16T12:31:06.346253+0000","last_became_active":"2023-10-16T12:31:05.433018+0000","last_became_peered":"2023-10-16T12:31:05.433018+0000","last_unstale":"2023-10-16T12:31:06.346253+0000","last_undegraded":"2023-10-16T12:31:06.346253+0000","last_fullsized":"2023-10-16T12:31:06.346253+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_clean_scrub_stamp":"2023-10-16T12:31:04.400465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:32:36.733604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.363418+0000","last_change":"2023-10-16T12:31:06.361296+0000","last_active":"2023-10-16T12:31:06.363418+0000","last_peered":"2023-10-16T12:31:06.363418+0000","last_clean":"2023-10-16T12:31:06.363418+0000","last_became_active":"2023-10-16T12:31:06.360937+0000","last_became_peered":"2023-10-16T12:31:06.360937+0000","last_unstale":"2023-10-16T12:31:06.363418+0000","last_undegraded":"2023-10-16T12:31:06.363418+0000","last_fullsized":"2023-10-16T12:31:06.363418+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:07:57.171342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.7","version":"155'2","reported_seq":13,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.426122+0000","last_change":"2023-10-16T12:31:05.431448+0000","last_active":"2023-10-16T12:31:06.426122+0000","last_peered":"2023-10-16T12:31:06.426122+0000","last_clean":"2023-10-16T12:31:06.426122+0000","last_became_active":"2023-10-16T12:31:05.431157+0000","last_became_peered":"2023-10-16T12:31:05.431157+0000","last_unstale":"2023-10-16T12:31:06.426122+0000","last_undegraded":"2023-10-16T12:31:06.426122+0000","last_fullsized":"2023-10-16T12:31:06.426122+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_clean_scrub_stamp":"2023-10-16T12:31:04.400465+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:20:08.795305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.361524+0000","last_change":"2023-10-16T12:31:06.360125+0000","last_active":"2023-10-16T12:31:06.361524+0000","last_peered":"2023-10-16T12:31:06.361524+0000","last_clean":"2023-10-16T12:31:06.361524+0000","last_became_active":"2023-10-16T12:31:06.359812+0000","last_became_peered":"2023-10-16T12:31:06.359812+0000","last_unstale":"2023-10-16T12:31:06.361524+0000","last_undegraded":"2023-10-16T12:31:06.361524+0000","last_fullsized":"2023-10-16T12:31:06.361524+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:10:42.229486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.351861+0000","last_change":"2023-10-16T12:31:05.423438+0000","last_active":"2023-10-16T12:31:06.351861+0000","last_peered":"2023-10-16T12:31:06.351861+0000","last_clean":"2023-10-16T12:31:06.351861+0000","last_became_active":"2023-10-16T12:31:05.423129+0000","last_became_peered":"2023-10-16T12:31:05.423129+0000","last_unstale":"2023-10-16T12:31:06.351861+0000","last_undegraded":"2023-10-16T12:31:06.351861+0000","last_fullsized":"2023-10-16T12:31:06.351861+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_clean_scrub_stamp":"2023-10-16T12:31:04.400465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:40:24.125227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.364961+0000","last_change":"2023-10-16T12:31:06.363278+0000","last_active":"2023-10-16T12:31:06.364961+0000","last_peered":"2023-10-16T12:31:06.364961+0000","last_clean":"2023-10-16T12:31:06.364961+0000","last_became_active":"2023-10-16T12:31:06.360426+0000","last_became_peered":"2023-10-16T12:31:06.360426+0000","last_unstale":"2023-10-16T12:31:06.364961+0000","last_undegraded":"2023-10-16T12:31:06.364961+0000","last_fullsized":"2023-10-16T12:31:06.364961+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:14:05.652793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.351859+0000","last_change":"2023-10-16T12:31:05.419046+0000","last_active":"2023-10-16T12:31:06.351859+0000","last_peered":"2023-10-16T12:31:06.351859+0000","last_clean":"2023-10-16T12:31:06.351859+0000","last_became_active":"2023-10-16T12:31:05.418758+0000","last_became_peered":"2023-10-16T12:31:05.418758+0000","last_unstale":"2023-10-16T12:31:06.351859+0000","last_undegraded":"2023-10-16T12:31:06.351859+0000","last_fullsized":"2023-10-16T12:31:06.351859+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_clean_scrub_stamp":"2023-10-16T12:31:04.400465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:19:49.824103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.358265+0000","last_change":"2023-10-16T12:31:06.356817+0000","last_active":"2023-10-16T12:31:06.358265+0000","last_peered":"2023-10-16T12:31:06.358265+0000","last_clean":"2023-10-16T12:31:06.358265+0000","last_became_active":"2023-10-16T12:31:06.356564+0000","last_became_peered":"2023-10-16T12:31:06.356564+0000","last_unstale":"2023-10-16T12:31:06.358265+0000","last_undegraded":"2023-10-16T12:31:06.358265+0000","last_fullsized":"2023-10-16T12:31:06.358265+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:49:38.103616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.346634+0000","last_change":"2023-10-16T12:31:05.429324+0000","last_active":"2023-10-16T12:31:06.346634+0000","last_peered":"2023-10-16T12:31:06.346634+0000","last_clean":"2023-10-16T12:31:06.346634+0000","last_became_active":"2023-10-16T12:31:05.428969+0000","last_became_peered":"2023-10-16T12:31:05.428969+0000","last_unstale":"2023-10-16T12:31:06.346634+0000","last_undegraded":"2023-10-16T12:31:06.346634+0000","last_fullsized":"2023-10-16T12:31:06.346634+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_clean_scrub_stamp":"2023-10-16T12:31:04.400465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:02:23.301879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.353418+0000","last_change":"2023-10-16T12:31:06.346768+0000","last_active":"2023-10-16T12:31:06.353418+0000","last_peered":"2023-10-16T12:31:06.353418+0000","last_clean":"2023-10-16T12:31:06.353418+0000","last_became_active":"2023-10-16T12:31:06.346276+0000","last_became_peered":"2023-10-16T12:31:06.346276+0000","last_unstale":"2023-10-16T12:31:06.353418+0000","last_undegraded":"2023-10-16T12:31:06.353418+0000","last_fullsized":"2023-10-16T12:31:06.353418+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:37:35.697306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.338363+0000","last_change":"2023-10-16T12:31:05.430525+0000","last_active":"2023-10-16T12:31:06.338363+0000","last_peered":"2023-10-16T12:31:06.338363+0000","last_clean":"2023-10-16T12:31:06.338363+0000","last_became_active":"2023-10-16T12:31:05.429303+0000","last_became_peered":"2023-10-16T12:31:05.429303+0000","last_unstale":"2023-10-16T12:31:06.338363+0000","last_undegraded":"2023-10-16T12:31:06.338363+0000","last_fullsized":"2023-10-16T12:31:06.338363+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_clean_scrub_stamp":"2023-10-16T12:31:04.400465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:53:58.286905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.353356+0000","last_change":"2023-10-16T12:31:06.346748+0000","last_active":"2023-10-16T12:31:06.353356+0000","last_peered":"2023-10-16T12:31:06.353356+0000","last_clean":"2023-10-16T12:31:06.353356+0000","last_became_active":"2023-10-16T12:31:06.345820+0000","last_became_peered":"2023-10-16T12:31:06.345820+0000","last_unstale":"2023-10-16T12:31:06.353356+0000","last_undegraded":"2023-10-16T12:31:06.353356+0000","last_fullsized":"2023-10-16T12:31:06.353356+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:35:24.892226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.346589+0000","last_change":"2023-10-16T12:31:05.428948+0000","last_active":"2023-10-16T12:31:06.346589+0000","last_peered":"2023-10-16T12:31:06.346589+0000","last_clean":"2023-10-16T12:31:06.346589+0000","last_became_active":"2023-10-16T12:31:05.428656+0000","last_became_peered":"2023-10-16T12:31:05.428656+0000","last_unstale":"2023-10-16T12:31:06.346589+0000","last_undegraded":"2023-10-16T12:31:06.346589+0000","last_fullsized":"2023-10-16T12:31:06.346589+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_clean_scrub_stamp":"2023-10-16T12:31:04.400465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:34:33.747687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.359355+0000","last_change":"2023-10-16T12:31:06.358792+0000","last_active":"2023-10-16T12:31:06.359355+0000","last_peered":"2023-10-16T12:31:06.359355+0000","last_clean":"2023-10-16T12:31:06.359355+0000","last_became_active":"2023-10-16T12:31:06.358515+0000","last_became_peered":"2023-10-16T12:31:06.358515+0000","last_unstale":"2023-10-16T12:31:06.359355+0000","last_undegraded":"2023-10-16T12:31:06.359355+0000","last_fullsized":"2023-10-16T12:31:06.359355+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:48:38.236166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.346566+0000","last_change":"2023-10-16T12:31:05.430784+0000","last_active":"2023-10-16T12:31:06.346566+0000","last_peered":"2023-10-16T12:31:06.346566+0000","last_clean":"2023-10-16T12:31:06.346566+0000","last_became_active":"2023-10-16T12:31:05.430493+0000","last_became_peered":"2023-10-16T12:31:05.430493+0000","last_unstale":"2023-10-16T12:31:06.346566+0000","last_undegraded":"2023-10-16T12:31:06.346566+0000","last_fullsized":"2023-10-16T12:31:06.346566+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_clean_scrub_stamp":"2023-10-16T12:31:04.400465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:54:01.565361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.7","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.364602+0000","last_change":"2023-10-16T12:31:06.361588+0000","last_active":"2023-10-16T12:31:06.364602+0000","last_peered":"2023-10-16T12:31:06.364602+0000","last_clean":"2023-10-16T12:31:06.364602+0000","last_became_active":"2023-10-16T12:31:06.361307+0000","last_became_peered":"2023-10-16T12:31:06.361307+0000","last_unstale":"2023-10-16T12:31:06.364602+0000","last_undegraded":"2023-10-16T12:31:06.364602+0000","last_fullsized":"2023-10-16T12:31:06.364602+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:40:12.392133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.349924+0000","last_change":"2023-10-16T12:31:05.435448+0000","last_active":"2023-10-16T12:31:06.349924+0000","last_peered":"2023-10-16T12:31:06.349924+0000","last_clean":"2023-10-16T12:31:06.349924+0000","last_became_active":"2023-10-16T12:31:05.435146+0000","last_became_peered":"2023-10-16T12:31:05.435146+0000","last_unstale":"2023-10-16T12:31:06.349924+0000","last_undegraded":"2023-10-16T12:31:06.349924+0000","last_fullsized":"2023-10-16T12:31:06.349924+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_clean_scrub_stamp":"2023-10-16T12:31:04.400465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:54:16.674958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.15","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.354008+0000","last_change":"2023-10-16T12:31:06.352816+0000","last_active":"2023-10-16T12:31:06.354008+0000","last_peered":"2023-10-16T12:31:06.354008+0000","last_clean":"2023-10-16T12:31:06.354008+0000","last_became_active":"2023-10-16T12:31:06.352514+0000","last_became_peered":"2023-10-16T12:31:06.352514+0000","last_unstale":"2023-10-16T12:31:06.354008+0000","last_undegraded":"2023-10-16T12:31:06.354008+0000","last_fullsized":"2023-10-16T12:31:06.354008+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:05.405463+0000","last_clean_scrub_stamp":"2023-10-16T12:31:05.405463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:50:50.515903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":11,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-16T12:31:06.346405+0000","last_change":"2023-10-16T12:31:05.428882+0000","last_active":"2023-10-16T12:31:06.346405+0000","last_peered":"2023-10-16T12:31:06.346405+0000","last_clean":"2023-10-16T12:31:06.346405+0000","last_became_active":"2023-10-16T12:31:05.428587+0000","last_became_peered":"2023-10-16T12:31:05.428587+0000","last_unstale":"2023-10-16T12:31:06.346405+0000","last_undegraded":"2023-10-16T12:31:06.346405+0000","last_fullsized":"2023-10-16T12:31:06.346405+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:31:04.400465+0000","last_clean_scrub_stamp":"2023-10-16T12:31:04.400465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:11:08.350800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"145'249","reported_seq":756,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-16T12:31:05.420543+0000","last_change":"2023-10-16T12:10:34.746820+0000","last_active":"2023-10-16T12:31:05.420543+0000","last_peered":"2023-10-16T12:31:05.420543+0000","last_clean":"2023-10-16T12:31:05.420543+0000","last_became_active":"2023-10-16T12:10:34.745574+0000","last_became_peered":"2023-10-16T12:10:34.745574+0000","last_unstale":"2023-10-16T12:31:05.420543+0000","last_undegraded":"2023-10-16T12:31:05.420543+0000","last_fullsized":"2023-10-16T12:31:05.420543+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_clean_scrub_stamp":"2023-10-16T12:10:33.577111+0000","objects_scrubbed":0,"log_size":249,"log_dups_size":0,"ondisk_log_size":249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:14.461986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1268,"num_read_kb":2572,"num_write":399,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":20,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":19,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1268,"num_read_kb":2572,"num_write":399,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":249,"ondisk_log_size":249,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575090,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47552,"kb_used_data":2484,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324288,"statfs":{"total":96636764160,"available":96588070912,"internally_reserved":0,"allocated":2543616,"data_stored":1569829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":46135755},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:30:18 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.61199999999999999,"15min":0.60199999999999998},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.27400000000000002},"max":{"1min":0.89300000000000002,"5min":4.2469999999999999,"15min":4.2469999999999999},"last":0.752},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.57199999999999995,"15min":0.58199999999999996},"min":{"1min":0.35299999999999998,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.84899999999999998,"5min":0.85299999999999998,"15min":0.89900000000000002},"last":0.58499999999999996}]},{"osd":1,"last update":"Mon Oct 16 12:30:18 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.58499999999999996,"15min":0.59999999999999998},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.247},"max":{"1min":0.80200000000000005,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.58999999999999997,"15min":0.60099999999999998},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.32400000000000001},"max":{"1min":0.79200000000000004,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.53800000000000003}]},{"osd":2,"last update":"Mon Oct 16 12:30:18 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.64400000000000002,"15min":0.65700000000000003},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.88600000000000001,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.64800000000000002,"15min":0.66300000000000003},"min":{"1min":0.44800000000000001,"5min":0.38100000000000001,"15min":0.376},"max":{"1min":0.88300000000000001,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.71399999999999997}]},{"osd":3,"last update":"Mon Oct 16 12:30:18 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.61099999999999999,"15min":0.61499999999999999},"min":{"1min":0.377,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.63,"15min":0.63300000000000001},"min":{"1min":0.41199999999999998,"5min":0.372,"15min":0.372},"max":{"1min":0.81000000000000005,"5min":0.996,"15min":1.2509999999999999},"last":0.624}]},{"osd":4,"last update":"Mon Oct 16 12:30:18 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.54400000000000004,"15min":0.53000000000000003},"min":{"1min":0.38600000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.80300000000000005,"5min":0.89700000000000002,"15min":1.383},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.627,"5min":0.63500000000000001,"15min":0.61499999999999999},"min":{"1min":0.39800000000000002,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.88200000000000001,"5min":0.93600000000000005,"15min":1.0109999999999999},"last":0.501}]},{"osd":5,"last update":"Mon Oct 16 12:30:18 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.56899999999999995,"15min":0.54000000000000004},"min":{"1min":0.311,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.77100000000000002,"5min":0.93400000000000005,"15min":1.3109999999999999},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.628,"5min":0.60799999999999998,"15min":0.58799999999999997},"min":{"1min":0.378,"5min":0.34699999999999998,"15min":0.32400000000000001},"max":{"1min":0.86099999999999999,"5min":0.93600000000000005,"15min":1.4359999999999999},"last":0.46200000000000002}]},{"osd":6,"last update":"Mon Oct 16 12:30:18 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.64800000000000002,"15min":0.625},"min":{"1min":0.41499999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.93000000000000005,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.64900000000000002,"15min":0.63},"min":{"1min":0.35099999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.91600000000000004,"5min":0.97199999999999998,"15min":1.1259999999999999},"last":0.60299999999999998}]}]},{"osd":6,"up_from":13,"seq":55834575088,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37244,"kb_used_data":1896,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334596,"statfs":{"total":96636764160,"available":96598626304,"internally_reserved":0,"allocated":1941504,"data_stored":978431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:30:07 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.57499999999999996,"15min":0.60899999999999999},"min":{"1min":0.35199999999999998,"5min":0.29299999999999998,"15min":0.24299999999999999},"max":{"1min":0.87,"5min":0.88200000000000001,"15min":1.196},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.59699999999999998,"15min":0.621},"min":{"1min":0.436,"5min":0.38500000000000001,"15min":0.33600000000000002},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.997},"last":0.60399999999999998}]},{"osd":1,"last update":"Mon Oct 16 12:30:07 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.53300000000000003,"15min":0.56000000000000005},"min":{"1min":0.35699999999999998,"5min":0.33100000000000002,"15min":0.29399999999999998},"max":{"1min":0.745,"5min":0.82199999999999995,"15min":0.875},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.57399999999999995,"15min":0.58599999999999997},"min":{"1min":0.40400000000000003,"5min":0.29799999999999999,"15min":0.29499999999999998},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":1.091},"last":0.623}]},{"osd":2,"last update":"Mon Oct 16 12:30:07 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56499999999999995,"15min":0.58999999999999997},"min":{"1min":0.377,"5min":0.35599999999999998,"15min":0.34499999999999997},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.97099999999999997},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.56899999999999995,"15min":0.59499999999999997},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.27100000000000002},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":2.3879999999999999},"last":0.51100000000000001}]},{"osd":3,"last update":"Mon Oct 16 12:30:07 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.60699999999999998,"15min":0.624},"min":{"1min":0.41599999999999998,"5min":0.37,"15min":0.34200000000000003},"max":{"1min":0.82299999999999995,"5min":0.88,"15min":0.93999999999999995},"last":0.751},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.66700000000000004,"15min":0.67800000000000005},"min":{"1min":0.504,"5min":0.41899999999999998,"15min":0.40899999999999997},"max":{"1min":0.90300000000000002,"5min":0.94499999999999995,"15min":1.3080000000000001},"last":0.72199999999999998}]},{"osd":4,"last update":"Mon Oct 16 12:30:07 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.52200000000000002,"15min":0.51600000000000001},"min":{"1min":0.35899999999999999,"5min":0.30599999999999999,"15min":0.29899999999999999},"max":{"1min":0.65700000000000003,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.60099999999999998,"15min":0.59699999999999998},"min":{"1min":0.38700000000000001,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.79400000000000004,"5min":0.82299999999999995,"15min":1.2250000000000001},"last":0.67600000000000005}]},{"osd":5,"last update":"Mon Oct 16 12:30:07 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.57899999999999996,"15min":0.57699999999999996},"min":{"1min":0.41499999999999998,"5min":0.30299999999999999,"15min":0.30099999999999999},"max":{"1min":0.70399999999999996,"5min":0.98799999999999999,"15min":1.0589999999999999},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.64300000000000002,"15min":0.626},"min":{"1min":0.33400000000000002,"5min":0.32100000000000001,"15min":0.311},"max":{"1min":0.84299999999999997,"5min":1.0029999999999999,"15min":1.3540000000000001},"last":0.80500000000000005}]},{"osd":7,"last update":"Mon Oct 16 12:30:07 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.58299999999999996,"15min":0.57599999999999996},"min":{"1min":0.42499999999999999,"5min":0.29699999999999999,"15min":0.26000000000000001},"max":{"1min":0.68799999999999994,"5min":0.92700000000000005,"15min":1.29},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.623,"5min":0.63400000000000001,"15min":0.61899999999999999},"min":{"1min":0.436,"5min":0.32900000000000001,"15min":0.29799999999999999},"max":{"1min":0.873,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.82299999999999995}]}]},{"osd":2,"up_from":13,"seq":55834575089,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63604,"kb_used_data":1896,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94308236,"statfs":{"total":96636764160,"available":96571633664,"internally_reserved":0,"allocated":1941504,"data_stored":978375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":63175110},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:30:05 2023","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.36899999999999999,"15min":0.379},"min":{"1min":0.222,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.51200000000000001,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.42699999999999999,"15min":0.443},"min":{"1min":0.28599999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.497,"5min":0.66500000000000004,"15min":0.82999999999999996},"last":0.371}]},{"osd":1,"last update":"Mon Oct 16 12:30:05 2023","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.441,"15min":0.46100000000000002},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.64500000000000002,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.40500000000000003,"15min":0.42999999999999999},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.57199999999999995,"5min":0.59199999999999997,"15min":0.69699999999999995},"last":0.47699999999999998}]},{"osd":3,"last update":"Mon Oct 16 12:30:05 2023","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.39400000000000002,"15min":0.40899999999999997},"min":{"1min":0.25,"5min":0.214,"15min":0.214},"max":{"1min":0.47799999999999998,"5min":0.69399999999999995,"15min":0.89300000000000002},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.498,"15min":0.51200000000000001},"min":{"1min":0.33200000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.74099999999999999,"5min":0.78000000000000003,"15min":0.79600000000000004},"last":0.49399999999999999}]},{"osd":4,"last update":"Mon Oct 16 12:30:05 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.63800000000000001,"15min":0.66800000000000004},"min":{"1min":0.35499999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.753,"5min":0.93799999999999994,"15min":1.3839999999999999},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.628,"15min":0.66500000000000004},"min":{"1min":0.42199999999999999,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.79900000000000004,"5min":0.97599999999999998,"15min":1.0109999999999999},"last":0.88300000000000001}]},{"osd":5,"last update":"Mon Oct 16 12:30:05 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.63500000000000001,"15min":0.65500000000000003},"min":{"1min":0.41599999999999998,"5min":0.35999999999999999,"15min":0.312},"max":{"1min":0.86699999999999999,"5min":1.093,"15min":1.093},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.61299999999999999,"15min":0.64700000000000002},"min":{"1min":0.40999999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.70499999999999996,"5min":1.0089999999999999,"15min":1.331},"last":0.67100000000000004}]},{"osd":6,"last update":"Mon Oct 16 12:30:05 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59599999999999997,"15min":0.626},"min":{"1min":0.42099999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.76200000000000001,"5min":0.99299999999999999,"15min":1.0369999999999999},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.65100000000000002,"15min":0.67600000000000005},"min":{"1min":0.437,"5min":0.40799999999999997,"15min":0.40000000000000002},"max":{"1min":0.83799999999999997,"5min":1.032,"15min":1.032},"last":0.999}]},{"osd":7,"last update":"Mon Oct 16 12:30:05 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.64800000000000002,"15min":0.66600000000000004},"min":{"1min":0.432,"5min":0.40300000000000002,"15min":0.38400000000000001},"max":{"1min":0.78300000000000003,"5min":0.96399999999999997,"15min":1.016},"last":1.02},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.65000000000000002,"15min":0.67800000000000005},"min":{"1min":0.44400000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.86199999999999999,"5min":1.145,"15min":1.145},"last":0.93600000000000005}]}]},{"osd":0,"up_from":13,"seq":55834575089,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37820,"kb_used_data":2476,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334020,"statfs":{"total":96636764160,"available":96598036480,"internally_reserved":0,"allocated":2535424,"data_stored":1568183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 12:31:00 2023","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.38200000000000001,"15min":0.38900000000000001},"min":{"1min":0.22800000000000001,"5min":0.219,"15min":0.219},"max":{"1min":0.58699999999999997,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.38300000000000001,"15min":0.41399999999999998},"min":{"1min":0.251,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.51600000000000001,"5min":0.67000000000000004,"15min":0.73099999999999998},"last":0.51300000000000001}]},{"osd":2,"last update":"Mon Oct 16 12:31:00 2023","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.40100000000000002,"15min":0.42099999999999999},"min":{"1min":0.27300000000000002,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.51300000000000001,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.432,"15min":0.44900000000000001},"min":{"1min":0.29199999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.51800000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.55600000000000005}]},{"osd":3,"last update":"Mon Oct 16 12:31:00 2023","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.40699999999999997,"15min":0.42299999999999999},"min":{"1min":0.28599999999999998,"5min":0.26300000000000001,"15min":0.23699999999999999},"max":{"1min":0.47999999999999998,"5min":0.62,"15min":0.79400000000000004},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.439,"5min":0.46300000000000002,"15min":0.48399999999999999},"min":{"1min":0.317,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.63600000000000001,"5min":0.79200000000000004,"15min":0.878},"last":0.60999999999999999}]},{"osd":4,"last update":"Mon Oct 16 12:31:00 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.61599999999999999,"15min":0.64400000000000002},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.78800000000000003,"5min":1.05,"15min":1.0960000000000001},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.61899999999999999,"15min":0.65100000000000002},"min":{"1min":0.36599999999999999,"5min":0.35599999999999998,"15min":0.34100000000000003},"max":{"1min":1.004,"5min":1.0549999999999999,"15min":1.1719999999999999},"last":0.81499999999999995}]},{"osd":5,"last update":"Mon Oct 16 12:31:00 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.60499999999999998,"15min":0.64500000000000002},"min":{"1min":0.39900000000000002,"5min":0.373,"15min":0.373},"max":{"1min":0.73599999999999999,"5min":0.97699999999999998,"15min":1.127},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.64000000000000001,"15min":0.67200000000000004},"min":{"1min":0.45300000000000001,"5min":0.39300000000000002,"15min":0.39000000000000001},"max":{"1min":0.85399999999999998,"5min":1.0700000000000001,"15min":1.1160000000000001},"last":0.79600000000000004}]},{"osd":6,"last update":"Mon Oct 16 12:31:00 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.64400000000000002,"15min":0.67700000000000005},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.80300000000000005,"5min":0.96899999999999997,"15min":1.079},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.62,"5min":0.65000000000000002,"15min":0.68200000000000005},"min":{"1min":0.40600000000000003,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.83699999999999997,"5min":0.96399999999999997,"15min":1.1499999999999999},"last":0.68200000000000005}]},{"osd":7,"last update":"Mon Oct 16 12:31:00 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.65800000000000003,"15min":0.69299999999999995},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.78100000000000003,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.64200000000000002,"15min":0.68100000000000005},"min":{"1min":0.41799999999999998,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.96999999999999997,"5min":0.97499999999999998,"15min":1.0569999999999999},"last":0.77800000000000002}]}]},{"osd":1,"up_from":13,"seq":55834575088,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37216,"kb_used_data":1876,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334624,"statfs":{"total":96636764160,"available":96598654976,"internally_reserved":0,"allocated":1921024,"data_stored":970608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":36174281},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.374,"15min":0.38},"min":{"1min":0.26600000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.627,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.434,"15min":0.46000000000000002},"min":{"1min":0.28899999999999998,"5min":0.28399999999999997,"15min":0.26700000000000002},"max":{"1min":0.60499999999999998,"5min":0.66300000000000003,"15min":1.0189999999999999},"last":0.318}]},{"osd":2,"last update":"Mon Oct 16 12:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.39800000000000002,"15min":0.42599999999999999},"min":{"1min":0.26900000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.59199999999999997,"5min":0.65300000000000002,"15min":0.93000000000000005},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.44,"5min":0.45000000000000001,"15min":0.46700000000000003},"min":{"1min":0.33700000000000002,"5min":0.29099999999999998,"15min":0.28199999999999997},"max":{"1min":0.61699999999999999,"5min":0.69199999999999995,"15min":0.98099999999999998},"last":0.38400000000000001}]},{"osd":3,"last update":"Mon Oct 16 12:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.38200000000000001,"15min":0.40300000000000002},"min":{"1min":0.247,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.56999999999999995,"5min":0.69599999999999995,"15min":0.98799999999999999},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.42999999999999999,"15min":0.45300000000000001},"min":{"1min":0.30299999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.751,"5min":0.751,"15min":0.91200000000000003},"last":0.33900000000000002}]},{"osd":4,"last update":"Mon Oct 16 12:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.60399999999999998,"15min":0.627},"min":{"1min":0.40500000000000003,"5min":0.35499999999999998,"15min":0.35199999999999998},"max":{"1min":0.74099999999999999,"5min":0.878,"15min":1.0409999999999999},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.624,"15min":0.64200000000000002},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.39100000000000001},"max":{"1min":0.86699999999999999,"5min":0.95399999999999996,"15min":0.98599999999999999},"last":0.61599999999999999}]},{"osd":5,"last update":"Mon Oct 16 12:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.63900000000000001,"15min":0.66300000000000003},"min":{"1min":0.42899999999999999,"5min":0.42599999999999999,"15min":0.38300000000000001},"max":{"1min":0.95399999999999996,"5min":1.069,"15min":1.069},"last":0.745},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.66300000000000003,"15min":0.68400000000000005},"min":{"1min":0.437,"5min":0.437,"15min":0.38900000000000001},"max":{"1min":0.93899999999999995,"5min":1.034,"15min":1.0620000000000001},"last":0.58299999999999996}]},{"osd":6,"last update":"Mon Oct 16 12:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.63400000000000001,"15min":0.65100000000000002},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.82599999999999996,"5min":1.044,"15min":1.044},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.63100000000000001,"15min":0.65800000000000003},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.32900000000000001},"max":{"1min":0.90400000000000003,"5min":1.0049999999999999,"15min":1.048},"last":0.67700000000000005}]},{"osd":7,"last update":"Mon Oct 16 12:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.64800000000000002,"15min":0.68300000000000005},"min":{"1min":0.45800000000000002,"5min":0.42099999999999999,"15min":0.33700000000000002},"max":{"1min":0.80600000000000005,"5min":1.0469999999999999,"15min":1.3400000000000001},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.627,"5min":0.67600000000000005,"15min":0.69699999999999995},"min":{"1min":0.47299999999999998,"5min":0.45700000000000002,"15min":0.42099999999999999},"max":{"1min":0.89100000000000001,"5min":1.006,"15min":1.02},"last":0.70399999999999996}]}]},{"osd":3,"up_from":13,"seq":55834575088,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37248,"kb_used_data":1904,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334592,"statfs":{"total":96636764160,"available":96598622208,"internally_reserved":0,"allocated":1949696,"data_stored":979441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:30:14 2023","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.38900000000000001,"15min":0.39700000000000002},"min":{"1min":0.22500000000000001,"5min":0.216,"15min":0.21099999999999999},"max":{"1min":0.46999999999999997,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.39200000000000002,"15min":0.40799999999999997},"min":{"1min":0.28000000000000003,"5min":0.254,"15min":0.254},"max":{"1min":0.54000000000000004,"5min":0.61899999999999999,"15min":0.73099999999999998},"last":0.35999999999999999}]},{"osd":1,"last update":"Mon Oct 16 12:30:14 2023","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.39900000000000002,"15min":0.41299999999999998},"min":{"1min":0.26400000000000001,"5min":0.245,"15min":0.24399999999999999},"max":{"1min":0.57499999999999996,"5min":0.70699999999999996,"15min":0.70799999999999996},"last":0.378},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.438,"15min":0.45800000000000002},"min":{"1min":0.29499999999999998,"5min":0.27000000000000002,"15min":0.25800000000000001},"max":{"1min":0.60399999999999998,"5min":0.72699999999999998,"15min":0.747},"last":0.38700000000000001}]},{"osd":2,"last update":"Mon Oct 16 12:30:14 2023","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.45300000000000001,"15min":0.46999999999999997},"min":{"1min":0.313,"5min":0.29799999999999999,"15min":0.29199999999999998},"max":{"1min":0.63,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.47899999999999998,"15min":0.498},"min":{"1min":0.311,"5min":0.30599999999999999,"15min":0.27100000000000002},"max":{"1min":0.72799999999999998,"5min":0.74299999999999999,"15min":0.752},"last":0.42499999999999999}]},{"osd":4,"last update":"Mon Oct 16 12:30:14 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.623,"15min":0.64500000000000002},"min":{"1min":0.39200000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.89600000000000002,"5min":0.98299999999999998,"15min":1.1539999999999999},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.622,"15min":0.65600000000000003},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.92100000000000004,"5min":0.95399999999999996,"15min":1.0409999999999999},"last":0.68400000000000005}]},{"osd":5,"last update":"Mon Oct 16 12:30:14 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.65200000000000002,"15min":0.68100000000000005},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.89400000000000002,"5min":1.0369999999999999,"15min":1.1080000000000001},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.63200000000000001,"15min":0.66000000000000003},"min":{"1min":0.45200000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.89000000000000001,"5min":0.97299999999999998,"15min":1.022},"last":0.67300000000000004}]},{"osd":6,"last update":"Mon Oct 16 12:30:14 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63300000000000001,"15min":0.66500000000000004},"min":{"1min":0.437,"5min":0.42999999999999999,"15min":0.36899999999999999},"max":{"1min":0.875,"5min":0.94199999999999995,"15min":1.0760000000000001},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.67400000000000004,"15min":0.70199999999999996},"min":{"1min":0.434,"5min":0.434,"15min":0.38600000000000001},"max":{"1min":0.89100000000000001,"5min":1.0349999999999999,"15min":1.0409999999999999},"last":0.60699999999999998}]},{"osd":7,"last update":"Mon Oct 16 12:30:14 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.64700000000000002,"15min":0.67900000000000005},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.38},"max":{"1min":0.85999999999999999,"5min":1.0149999999999999,"15min":1.2430000000000001},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.68500000000000005,"15min":0.70499999999999996},"min":{"1min":0.503,"5min":0.44,"15min":0.42699999999999999},"max":{"1min":0.88100000000000001,"5min":1.002,"15min":1.1339999999999999},"last":0.621}]}]},{"osd":4,"up_from":13,"seq":55834575087,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37228,"kb_used_data":1888,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334612,"statfs":{"total":96636764160,"available":96598642688,"internally_reserved":0,"allocated":1933312,"data_stored":977759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.56599999999999995,"15min":0.58999999999999997},"min":{"1min":0.41099999999999998,"5min":0.30399999999999999,"15min":0.27900000000000003},"max":{"1min":0.752,"5min":0.78500000000000003,"15min":0.94599999999999995},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.57199999999999995,"15min":0.59699999999999998},"min":{"1min":0.42399999999999999,"5min":0.29999999999999999,"15min":0.27600000000000002},"max":{"1min":0.73299999999999998,"5min":0.83199999999999996,"15min":0.88700000000000001},"last":0.52600000000000002}]},{"osd":1,"last update":"Mon Oct 16 12:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.58599999999999997,"15min":0.60799999999999998},"min":{"1min":0.32900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.76200000000000001,"5min":0.82599999999999996,"15min":0.92600000000000005},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.61399999999999999},"min":{"1min":0.434,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.98099999999999998},"last":0.54800000000000004}]},{"osd":2,"last update":"Mon Oct 16 12:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.58399999999999996,"15min":0.61799999999999999},"min":{"1min":0.34999999999999998,"5min":0.33800000000000002,"15min":0.32000000000000001},"max":{"1min":0.745,"5min":0.84299999999999997,"15min":1.012},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.59699999999999998,"15min":0.624},"min":{"1min":0.38200000000000001,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.76000000000000001,"5min":0.85599999999999998,"15min":0.94199999999999995},"last":0.60499999999999998}]},{"osd":3,"last update":"Mon Oct 16 12:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.56899999999999995,"15min":0.60199999999999998},"min":{"1min":0.41799999999999998,"5min":0.35599999999999998,"15min":0.33700000000000002},"max":{"1min":0.70699999999999996,"5min":0.82299999999999995,"15min":0.93200000000000005},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.63,"5min":0.63300000000000001,"15min":0.64300000000000002},"min":{"1min":0.47099999999999997,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.80000000000000004,"5min":0.90600000000000003,"15min":0.97199999999999998},"last":0.50600000000000001}]},{"osd":5,"last update":"Mon Oct 16 12:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.52100000000000002,"15min":0.50600000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.72199999999999998,"5min":0.82599999999999996,"15min":0.95699999999999996},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.53000000000000003,"15min":0.52500000000000002},"min":{"1min":0.373,"5min":0.33900000000000002,"15min":0.308},"max":{"1min":0.73399999999999999,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.39600000000000002}]},{"osd":6,"last update":"Mon Oct 16 12:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.55600000000000005,"15min":0.55700000000000005},"min":{"1min":0.39000000000000001,"5min":0.33800000000000002,"15min":0.32500000000000001},"max":{"1min":0.70699999999999996,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.63800000000000001,"15min":0.63700000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.32000000000000001},"max":{"1min":0.84499999999999997,"5min":0.89300000000000002,"15min":0.99299999999999999},"last":0.48999999999999999}]},{"osd":7,"last update":"Mon Oct 16 12:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59199999999999997,"15min":0.58499999999999996},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.77700000000000002,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.629,"15min":0.629},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.29999999999999999},"max":{"1min":0.82899999999999996,"5min":1.006,"15min":1.006},"last":0.38}]}]},{"osd":5,"up_from":13,"seq":55834575088,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55272,"kb_used_data":1876,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316568,"statfs":{"total":96636764160,"available":96580165632,"internally_reserved":0,"allocated":1921024,"data_stored":970608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 12:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.53100000000000003,"15min":0.57799999999999996},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.73199999999999998,"5min":0.81399999999999995,"15min":0.873},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.58199999999999996,"15min":0.61099999999999999},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.27600000000000002},"max":{"1min":0.86099999999999999,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.67400000000000004}]},{"osd":1,"last update":"Mon Oct 16 12:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.58399999999999996,"15min":0.60699999999999998},"min":{"1min":0.40300000000000002,"5min":0.34499999999999997,"15min":0.28999999999999998},"max":{"1min":0.81499999999999995,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.55400000000000005,"15min":0.60399999999999998},"min":{"1min":0.34300000000000003,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.83399999999999996,"5min":0.84399999999999997,"15min":0.89000000000000001},"last":0.61799999999999999}]},{"osd":2,"last update":"Mon Oct 16 12:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57199999999999995,"15min":0.61299999999999999},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.29099999999999998},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.94399999999999995},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.56499999999999995,"15min":0.60299999999999998},"min":{"1min":0.34599999999999997,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.873,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.58899999999999997}]},{"osd":3,"last update":"Mon Oct 16 12:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.60099999999999998,"15min":0.628},"min":{"1min":0.41999999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.89300000000000002,"5min":0.90400000000000003,"15min":0.97499999999999998},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.621,"15min":0.64100000000000001},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.35999999999999999},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.72499999999999998}]},{"osd":4,"last update":"Mon Oct 16 12:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.501,"15min":0.51000000000000001},"min":{"1min":0.311,"5min":0.28100000000000003,"15min":0.25800000000000001},"max":{"1min":0.80300000000000005,"5min":0.81299999999999994,"15min":0.86399999999999999},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.57799999999999996,"15min":0.57299999999999995},"min":{"1min":0.44400000000000001,"5min":0.33900000000000002,"15min":0.33500000000000002},"max":{"1min":0.81999999999999995,"5min":0.876,"15min":1.347},"last":0.70299999999999996}]},{"osd":6,"last update":"Mon Oct 16 12:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.55700000000000005,"15min":0.55400000000000005},"min":{"1min":0.36399999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.85299999999999998,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.626,"5min":0.57999999999999996,"15min":0.59799999999999998},"min":{"1min":0.39500000000000002,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.83899999999999997,"5min":0.88400000000000001,"15min":0.92700000000000005},"last":0.65900000000000003}]},{"osd":7,"last update":"Mon Oct 16 12:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.63,"15min":0.627},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.32300000000000001},"max":{"1min":0.88300000000000001,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.64400000000000002,"15min":0.65700000000000003},"min":{"1min":0.47799999999999998,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":1.1180000000000001},"last":0.68799999999999994}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T12:31:10.983 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-16T12:31:10.984 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-16T12:31:11.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.149+0000 7f7f8ebf0700 1 Processor -- start 2023-10-16T12:31:11.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.149+0000 7f7f8ebf0700 1 -- start start 2023-10-16T12:31:11.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.150+0000 7f7f8ebf0700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7f8815a640 0x7f7f880cb710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:11.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.150+0000 7f7f8ebf0700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7f8815e140 0x7f7f880cbc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:11.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.150+0000 7f7f8ebf0700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f8815aa20 con 0x7f7f880c6770 2023-10-16T12:31:11.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.150+0000 7f7f8ebf0700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f8806eb40 con 0x7f7f8815a640 2023-10-16T12:31:11.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.150+0000 7f7f8ebf0700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f8806ee20 con 0x7f7f8815e140 2023-10-16T12:31:11.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.150+0000 7f7f8c98c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7f8815a640 0x7f7f880cb710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:11.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.150+0000 7f7f87fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f7f880c6770 0x7f7f880ca1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36244/0 (socket says 172.21.15.16:36244) 2023-10-16T12:31:11.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.150+0000 7f7f87fff700 1 -- 172.21.15.16:0/2096335651 learned_addr learned my addr 172.21.15.16:0/2096335651 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:31:11.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.150+0000 7f7f8c98c700 1 -- 172.21.15.16:0/2096335651 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7f8815e140 msgr2=0x7f7f880cbc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:31:11.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.150+0000 7f7f8d18d700 1 --2- 172.21.15.16:0/2096335651 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7f8815e140 0x7f7f880cbc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:11.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.150+0000 7f7f8c98c700 1 --2- 172.21.15.16:0/2096335651 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7f8815e140 0x7f7f880cbc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:11.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.150+0000 7f7f8c98c700 1 -- 172.21.15.16:0/2096335651 >> v1:172.21.15.16:6789/0 conn(0x7f7f880c6770 legacy=0x7f7f880ca1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:11.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.150+0000 7f7f8c98c700 1 -- 172.21.15.16:0/2096335651 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f8806f100 con 0x7f7f8815a640 2023-10-16T12:31:11.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.150+0000 7f7f877fe700 1 -- 172.21.15.16:0/2096335651 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 637424837 0 0) 0x7f7f74001110 con 0x7f7f880c6770 2023-10-16T12:31:11.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.150+0000 7f7f8d18d700 1 --2- 172.21.15.16:0/2096335651 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7f8815e140 0x7f7f880cbc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:31:11.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.150+0000 7f7f8c98c700 1 --2- 172.21.15.16:0/2096335651 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f8815a640 0x7f7f880cb710 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7f7f7c009840 tx=0x7f7f7c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff0c2b6273bbc9b9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:11.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.150+0000 7f7f877fe700 1 -- 172.21.15.16:0/2096335651 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f7c011830 con 0x7f7f8815a640 2023-10-16T12:31:11.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.150+0000 7f7f877fe700 1 -- 172.21.15.16:0/2096335651 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f7c011990 con 0x7f7f8815a640 2023-10-16T12:31:11.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.150+0000 7f7f877fe700 1 -- 172.21.15.16:0/2096335651 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f7c011d20 con 0x7f7f8815a640 2023-10-16T12:31:11.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.151+0000 7f7f8ebf0700 1 -- 172.21.15.16:0/2096335651 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f8815a640 msgr2=0x7f7f880cb710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:11.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.151+0000 7f7f8ebf0700 1 --2- 172.21.15.16:0/2096335651 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f8815a640 0x7f7f880cb710 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7f7f7c009840 tx=0x7f7f7c009b50 comp rx=0 tx=0).stop 2023-10-16T12:31:11.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.151+0000 7f7f8ebf0700 1 -- 172.21.15.16:0/2096335651 shutdown_connections 2023-10-16T12:31:11.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.151+0000 7f7f8ebf0700 1 --2- 172.21.15.16:0/2096335651 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7f8815e140 0x7f7f880cbc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:11.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.151+0000 7f7f8ebf0700 1 --2- 172.21.15.16:0/2096335651 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f8815a640 0x7f7f880cb710 unknown :-1 s=CLOSED pgs=1756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:11.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.151+0000 7f7f8ebf0700 1 -- 172.21.15.16:0/2096335651 >> 172.21.15.16:0/2096335651 conn(0x7f7f880cc5e0 msgr2=0x7f7f880cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:31:11.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.151+0000 7f7f8ebf0700 1 -- 172.21.15.16:0/2096335651 shutdown_connections 2023-10-16T12:31:11.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.151+0000 7f7f8ebf0700 1 -- 172.21.15.16:0/2096335651 wait complete. 2023-10-16T12:31:11.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.152+0000 7f7f8ebf0700 1 Processor -- start 2023-10-16T12:31:11.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.152+0000 7f7f8ebf0700 1 -- start start 2023-10-16T12:31:11.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.153+0000 7f7f8ebf0700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7f880c6770 0x7f7f8815bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:11.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.153+0000 7f7f8ebf0700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7f8815a640 0x7f7f881568f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:11.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.153+0000 7f7f8ebf0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f8815e140 0x7f7f88156e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:11.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.153+0000 7f7f8ebf0700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f8815cee0 con 0x7f7f8815e140 2023-10-16T12:31:11.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.153+0000 7f7f8ebf0700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f8815c2d0 con 0x7f7f880c6770 2023-10-16T12:31:11.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.153+0000 7f7f8ebf0700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f8815c5b0 con 0x7f7f8815a640 2023-10-16T12:31:11.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.153+0000 7f7f8d18d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f8815e140 0x7f7f88156e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:11.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.153+0000 7f7f8d18d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f8815e140 0x7f7f88156e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46978/0 (socket says 172.21.15.16:46978) 2023-10-16T12:31:11.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.153+0000 7f7f8d18d700 1 -- 172.21.15.16:0/1007508709 learned_addr learned my addr 172.21.15.16:0/1007508709 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:31:11.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.153+0000 7f7f87fff700 1 --2- 172.21.15.16:0/1007508709 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7f8815a640 0x7f7f881568f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:11.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.153+0000 7f7f8c98c700 1 --2- 172.21.15.16:0/1007508709 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7f880c6770 0x7f7f8815bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:11.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.154+0000 7f7f8d18d700 1 -- 172.21.15.16:0/1007508709 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7f8815a640 msgr2=0x7f7f881568f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:11.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.154+0000 7f7f8d18d700 1 --2- 172.21.15.16:0/1007508709 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7f8815a640 0x7f7f881568f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:11.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.154+0000 7f7f8d18d700 1 -- 172.21.15.16:0/1007508709 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7f880c6770 msgr2=0x7f7f8815bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:11.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.154+0000 7f7f8d18d700 1 --2- 172.21.15.16:0/1007508709 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7f880c6770 0x7f7f8815bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:11.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.154+0000 7f7f8d18d700 1 -- 172.21.15.16:0/1007508709 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f7c0094a0 con 0x7f7f8815e140 2023-10-16T12:31:11.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.154+0000 7f7f87fff700 1 --2- 172.21.15.16:0/1007508709 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7f8815a640 0x7f7f881568f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:31:11.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.154+0000 7f7f8c98c700 1 --2- 172.21.15.16:0/1007508709 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7f880c6770 0x7f7f8815bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:31:11.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.154+0000 7f7f8d18d700 1 --2- 172.21.15.16:0/1007508709 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f8815e140 0x7f7f88156e30 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7f7f8000cd50 tx=0x7f7f8000a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:11.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.154+0000 7f7f857fa700 1 -- 172.21.15.16:0/1007508709 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f8000f480 con 0x7f7f8815e140 2023-10-16T12:31:11.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.154+0000 7f7f857fa700 1 -- 172.21.15.16:0/1007508709 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f8000f600 con 0x7f7f8815e140 2023-10-16T12:31:11.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.154+0000 7f7f8ebf0700 1 -- 172.21.15.16:0/1007508709 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f8815c890 con 0x7f7f8815e140 2023-10-16T12:31:11.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.154+0000 7f7f857fa700 1 -- 172.21.15.16:0/1007508709 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f8000f9b0 con 0x7f7f8815e140 2023-10-16T12:31:11.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.154+0000 7f7f8ebf0700 1 -- 172.21.15.16:0/1007508709 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f8815d370 con 0x7f7f8815e140 2023-10-16T12:31:11.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.155+0000 7f7f8ebf0700 1 -- 172.21.15.16:0/1007508709 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f8815d050 con 0x7f7f8815e140 2023-10-16T12:31:11.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.156+0000 7f7f857fa700 1 -- 172.21.15.16:0/1007508709 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 85) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7f8000b040 con 0x7f7f8815e140 2023-10-16T12:31:11.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.156+0000 7f7f857fa700 1 --2- 172.21.15.16:0/1007508709 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f7f6c06b020 0x7f7f6c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:11.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.156+0000 7f7f857fa700 1 -- 172.21.15.16:0/1007508709 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 6085+0+0 (secure 0 0 0) 0x7f7f8008a2b0 con 0x7f7f8815e140 2023-10-16T12:31:11.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.156+0000 7f7f8c98c700 1 --2- 172.21.15.16:0/1007508709 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f7f6c06b020 0x7f7f6c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:11.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.157+0000 7f7f8c98c700 1 --2- 172.21.15.16:0/1007508709 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f7f6c06b020 0x7f7f6c06d4e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7f7c0090b0 tx=0x7f7f7c008030 comp rx=0 tx=0).ready entity=mgr.8973 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:11.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.161+0000 7f7f857fa700 1 -- 172.21.15.16:0/1007508709 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7f80054f40 con 0x7f7f8815e140 2023-10-16T12:31:11.403 INFO:tasks.ceph.mds.c.smithi016.stderr:2023-10-16T12:31:11.402+0000 7f01e116c700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T12:31:11.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.448+0000 7f7f8ebf0700 1 -- 172.21.15.16:0/1007508709 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f7f8815d590 con 0x7f7f8815e140 2023-10-16T12:31:11.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.450+0000 7f7f857fa700 1 -- 172.21.15.16:0/1007508709 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v26) v1 ==== 190+0+147 (secure 0 0 0) 0x7f7f80058560 con 0x7f7f8815e140 2023-10-16T12:31:11.452 INFO:teuthology.orchestra.run.smithi016.stdout:[client.0] 2023-10-16T12:31:11.452 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAPKC1l2l6PFxAA9QyXtBAHdGAXpwH0JfknuQ== 2023-10-16T12:31:11.452 INFO:teuthology.orchestra.run.smithi016.stdout: caps mds = "allow" 2023-10-16T12:31:11.453 INFO:teuthology.orchestra.run.smithi016.stdout: caps mon = "allow r" 2023-10-16T12:31:11.453 INFO:teuthology.orchestra.run.smithi016.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-16T12:31:11.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.452+0000 7f7f8ebf0700 1 -- 172.21.15.16:0/1007508709 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f7f6c06b020 msgr2=0x7f7f6c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:11.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.452+0000 7f7f8ebf0700 1 --2- 172.21.15.16:0/1007508709 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f7f6c06b020 0x7f7f6c06d4e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7f7c0090b0 tx=0x7f7f7c008030 comp rx=0 tx=0).stop 2023-10-16T12:31:11.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.452+0000 7f7f8ebf0700 1 -- 172.21.15.16:0/1007508709 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f8815e140 msgr2=0x7f7f88156e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:11.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.452+0000 7f7f8ebf0700 1 --2- 172.21.15.16:0/1007508709 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f8815e140 0x7f7f88156e30 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7f7f8000cd50 tx=0x7f7f8000a560 comp rx=0 tx=0).stop 2023-10-16T12:31:11.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.452+0000 7f7f8ebf0700 1 -- 172.21.15.16:0/1007508709 shutdown_connections 2023-10-16T12:31:11.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.452+0000 7f7f8ebf0700 1 --2- 172.21.15.16:0/1007508709 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f7f6c06b020 0x7f7f6c06d4e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:11.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.452+0000 7f7f8ebf0700 1 --2- 172.21.15.16:0/1007508709 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f8815e140 0x7f7f88156e30 unknown :-1 s=CLOSED pgs=1757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:11.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.452+0000 7f7f8ebf0700 1 --2- 172.21.15.16:0/1007508709 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7f880c6770 0x7f7f8815bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:11.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.452+0000 7f7f8ebf0700 1 --2- 172.21.15.16:0/1007508709 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7f8815a640 0x7f7f881568f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:11.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.452+0000 7f7f8ebf0700 1 -- 172.21.15.16:0/1007508709 >> 172.21.15.16:0/1007508709 conn(0x7f7f880cc5e0 msgr2=0x7f7f880ced40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:31:11.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.452+0000 7f7f8ebf0700 1 -- 172.21.15.16:0/1007508709 shutdown_connections 2023-10-16T12:31:11.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.453+0000 7f7f8ebf0700 1 -- 172.21.15.16:0/1007508709 wait complete. 2023-10-16T12:31:11.457 INFO:teuthology.orchestra.run.smithi016.stderr:updated caps for client.0 2023-10-16T12:31:11.467 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T12:31:11.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.590+0000 7f2132326700 1 Processor -- start 2023-10-16T12:31:11.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.590+0000 7f2132326700 1 -- start start 2023-10-16T12:31:11.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.590+0000 7f2132326700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f212c0cf4a0 0x7f212c0726e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:11.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.590+0000 7f2132326700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f212c0b5a00 0x7f212c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:11.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.590+0000 7f2132326700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f212c15bc80 con 0x7f212c0cf880 2023-10-16T12:31:11.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.590+0000 7f2132326700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f212c0b5e40 con 0x7f212c0cf4a0 2023-10-16T12:31:11.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.590+0000 7f2132326700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f212c072c20 con 0x7f212c0b5a00 2023-10-16T12:31:11.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.591+0000 7f212bfff700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f212c0cf4a0 0x7f212c0726e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:11.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.591+0000 7f21308c3700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f212c0b5a00 0x7f212c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:11.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.591+0000 7f21308c3700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f212c0b5a00 0x7f212c15a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:35244/0 (socket says 172.21.15.16:35244) 2023-10-16T12:31:11.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.591+0000 7f212bfff700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f212c0cf4a0 0x7f212c0726e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:46044/0 (socket says 172.21.15.16:46044) 2023-10-16T12:31:11.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.591+0000 7f21308c3700 1 -- 172.21.15.16:0/1491065903 learned_addr learned my addr 172.21.15.16:0/1491065903 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:31:11.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.591+0000 7f212bfff700 1 -- 172.21.15.16:0/1491065903 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f212c0b5a00 msgr2=0x7f212c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:31:11.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.591+0000 7f212bfff700 1 --2- 172.21.15.16:0/1491065903 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f212c0b5a00 0x7f212c15a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:11.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.591+0000 7f212bfff700 1 -- 172.21.15.16:0/1491065903 >> v1:172.21.15.49:6789/0 conn(0x7f212c0cf880 legacy=0x7f212c0b52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:11.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.591+0000 7f212bfff700 1 -- 172.21.15.16:0/1491065903 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f212c15b070 con 0x7f212c0cf4a0 2023-10-16T12:31:11.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.591+0000 7f212affd700 1 -- 172.21.15.16:0/1491065903 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 312033883 0 0) 0x7f212c15bc80 con 0x7f212c0cf880 2023-10-16T12:31:11.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.591+0000 7f21308c3700 1 --2- 172.21.15.16:0/1491065903 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f212c0b5a00 0x7f212c15a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:31:11.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.592+0000 7f212bfff700 1 --2- 172.21.15.16:0/1491065903 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f212c0cf4a0 0x7f212c0726e0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f211c009840 tx=0x7f211c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8322d21a39965b92 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:11.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.592+0000 7f212affd700 1 -- 172.21.15.16:0/1491065903 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f211c011830 con 0x7f212c0cf4a0 2023-10-16T12:31:11.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.592+0000 7f212affd700 1 -- 172.21.15.16:0/1491065903 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f211c011990 con 0x7f212c0cf4a0 2023-10-16T12:31:11.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.592+0000 7f212affd700 1 -- 172.21.15.16:0/1491065903 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f211c011d20 con 0x7f212c0cf4a0 2023-10-16T12:31:11.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.592+0000 7f2132326700 1 -- 172.21.15.16:0/1491065903 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f212c0cf4a0 msgr2=0x7f212c0726e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:11.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.592+0000 7f2132326700 1 --2- 172.21.15.16:0/1491065903 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f212c0cf4a0 0x7f212c0726e0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f211c009840 tx=0x7f211c009b50 comp rx=0 tx=0).stop 2023-10-16T12:31:11.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.592+0000 7f2132326700 1 -- 172.21.15.16:0/1491065903 shutdown_connections 2023-10-16T12:31:11.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.592+0000 7f2132326700 1 --2- 172.21.15.16:0/1491065903 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f212c0b5a00 0x7f212c15a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:11.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.592+0000 7f2132326700 1 --2- 172.21.15.16:0/1491065903 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f212c0cf4a0 0x7f212c0726e0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:11.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.592+0000 7f2132326700 1 -- 172.21.15.16:0/1491065903 >> 172.21.15.16:0/1491065903 conn(0x7f212c0b1fd0 msgr2=0x7f212c0b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:31:11.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.592+0000 7f2132326700 1 -- 172.21.15.16:0/1491065903 shutdown_connections 2023-10-16T12:31:11.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.592+0000 7f2132326700 1 -- 172.21.15.16:0/1491065903 wait complete. 2023-10-16T12:31:11.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.593+0000 7f2132326700 1 Processor -- start 2023-10-16T12:31:11.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.594+0000 7f2132326700 1 -- start start 2023-10-16T12:31:11.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.594+0000 7f2132326700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f212c0b5a00 0x7f212c151de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:11.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.594+0000 7f2132326700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f212c0cf4a0 0x7f212c0ccd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:11.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.594+0000 7f2132326700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f212c0cf880 0x7f212c0c3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:11.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.594+0000 7f2132326700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f212c1531b0 con 0x7f212c0cf4a0 2023-10-16T12:31:11.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.594+0000 7f2132326700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f212c152320 con 0x7f212c0cf880 2023-10-16T12:31:11.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.594+0000 7f2132326700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f212c0ca880 con 0x7f212c0b5a00 2023-10-16T12:31:11.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.594+0000 7f212b7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f212c0cf4a0 0x7f212c0ccd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:11.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.595+0000 7f212b7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f212c0cf4a0 0x7f212c0ccd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46990/0 (socket says 172.21.15.16:46990) 2023-10-16T12:31:11.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.595+0000 7f212b7fe700 1 -- 172.21.15.16:0/1682611839 learned_addr learned my addr 172.21.15.16:0/1682611839 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:31:11.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.595+0000 7f21308c3700 1 --2- 172.21.15.16:0/1682611839 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f212c0cf880 0x7f212c0c3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:11.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.595+0000 7f212bfff700 1 --2- 172.21.15.16:0/1682611839 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f212c0b5a00 0x7f212c151de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:11.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.595+0000 7f212b7fe700 1 -- 172.21.15.16:0/1682611839 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f212c0b5a00 msgr2=0x7f212c151de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:11.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.595+0000 7f212b7fe700 1 --2- 172.21.15.16:0/1682611839 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f212c0b5a00 0x7f212c151de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:11.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.595+0000 7f212b7fe700 1 -- 172.21.15.16:0/1682611839 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f212c0cf880 msgr2=0x7f212c0c3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:11.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.595+0000 7f212b7fe700 1 --2- 172.21.15.16:0/1682611839 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f212c0cf880 0x7f212c0c3f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:11.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.595+0000 7f212b7fe700 1 -- 172.21.15.16:0/1682611839 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2124009b30 con 0x7f212c0cf4a0 2023-10-16T12:31:11.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.595+0000 7f21308c3700 1 --2- 172.21.15.16:0/1682611839 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f212c0cf880 0x7f212c0c3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:31:11.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.595+0000 7f212b7fe700 1 --2- 172.21.15.16:0/1682611839 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f212c0cf4a0 0x7f212c0ccd10 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7f212000c580 tx=0x7f212000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:11.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.595+0000 7f2128ff9700 1 -- 172.21.15.16:0/1682611839 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21200054e0 con 0x7f212c0cf4a0 2023-10-16T12:31:11.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.595+0000 7f2128ff9700 1 -- 172.21.15.16:0/1682611839 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2120003680 con 0x7f212c0cf4a0 2023-10-16T12:31:11.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.595+0000 7f2132326700 1 -- 172.21.15.16:0/1682611839 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f211c0094a0 con 0x7f212c0cf4a0 2023-10-16T12:31:11.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.595+0000 7f2128ff9700 1 -- 172.21.15.16:0/1682611839 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2120003a30 con 0x7f212c0cf4a0 2023-10-16T12:31:11.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.595+0000 7f2132326700 1 -- 172.21.15.16:0/1682611839 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f212c152af0 con 0x7f212c0cf4a0 2023-10-16T12:31:11.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.596+0000 7f2132326700 1 -- 172.21.15.16:0/1682611839 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f212c15b8c0 con 0x7f212c0cf4a0 2023-10-16T12:31:11.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.597+0000 7f2128ff9700 1 -- 172.21.15.16:0/1682611839 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 85) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2120003b90 con 0x7f212c0cf4a0 2023-10-16T12:31:11.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.597+0000 7f2128ff9700 1 --2- 172.21.15.16:0/1682611839 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f211806b020 0x7f211806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:11.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.597+0000 7f2128ff9700 1 -- 172.21.15.16:0/1682611839 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 6085+0+0 (secure 0 0 0) 0x7f2120088e40 con 0x7f212c0cf4a0 2023-10-16T12:31:11.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.597+0000 7f212bfff700 1 --2- 172.21.15.16:0/1682611839 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f211806b020 0x7f211806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:11.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.598+0000 7f212bfff700 1 --2- 172.21.15.16:0/1682611839 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f211806b020 0x7f211806d4e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f211c0090b0 tx=0x7f211c008030 comp rx=0 tx=0).ready entity=mgr.8973 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:11.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.602+0000 7f2128ff9700 1 -- 172.21.15.16:0/1682611839 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2120053b50 con 0x7f212c0cf4a0 2023-10-16T12:31:11.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.853+0000 7f2132326700 1 -- 172.21.15.16:0/1682611839 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f212c04ebb0 con 0x7f212c0cf4a0 2023-10-16T12:31:11.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.854+0000 7f2128ff9700 1 -- 172.21.15.16:0/1682611839 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+4915 (secure 0 0 0) 0x7f2120017090 con 0x7f212c0cf4a0 2023-10-16T12:31:11.856 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:31:11.856 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":78,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7631,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/3370728164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":3370728164},{"type":"v1","addr":"172.21.15.49:6834","nonce":3370728164}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":73},{"gid":8638,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2297042963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":2297042963},{"type":"v1","addr":"172.21.15.16:6837","nonce":2297042963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":73},{"gid":8647,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/1920044955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":1920044955},{"type":"v1","addr":"172.21.15.49:6835","nonce":1920044955}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":73}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:31:05.836924+0000","modified":"2023-10-16T12:31:08.845101+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9204":{"gid":9204,"name":"c","rank":0,"incarnation":74,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6836/1477717813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1477717813},{"type":"v1","addr":"172.21.15.16:6836","nonce":1477717813}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2023-10-16T12:31:11.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.856+0000 7f2132326700 1 -- 172.21.15.16:0/1682611839 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f211806b020 msgr2=0x7f211806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:11.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.856+0000 7f2132326700 1 --2- 172.21.15.16:0/1682611839 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f211806b020 0x7f211806d4e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f211c0090b0 tx=0x7f211c008030 comp rx=0 tx=0).stop 2023-10-16T12:31:11.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.856+0000 7f2132326700 1 -- 172.21.15.16:0/1682611839 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f212c0cf4a0 msgr2=0x7f212c0ccd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:11.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.856+0000 7f2132326700 1 --2- 172.21.15.16:0/1682611839 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f212c0cf4a0 0x7f212c0ccd10 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7f212000c580 tx=0x7f212000c940 comp rx=0 tx=0).stop 2023-10-16T12:31:11.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.856+0000 7f2132326700 1 -- 172.21.15.16:0/1682611839 shutdown_connections 2023-10-16T12:31:11.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.856+0000 7f2132326700 1 --2- 172.21.15.16:0/1682611839 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f211806b020 0x7f211806d4e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:11.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.856+0000 7f2132326700 1 --2- 172.21.15.16:0/1682611839 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f212c0cf880 0x7f212c0c3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:11.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.856+0000 7f2132326700 1 --2- 172.21.15.16:0/1682611839 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f212c0b5a00 0x7f212c151de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:11.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.856+0000 7f2132326700 1 --2- 172.21.15.16:0/1682611839 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f212c0cf4a0 0x7f212c0ccd10 unknown :-1 s=CLOSED pgs=1758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:11.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.856+0000 7f2132326700 1 -- 172.21.15.16:0/1682611839 >> 172.21.15.16:0/1682611839 conn(0x7f212c0b1fd0 msgr2=0x7f212c0afe40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:31:11.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.856+0000 7f2132326700 1 -- 172.21.15.16:0/1682611839 shutdown_connections 2023-10-16T12:31:11.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:11.856+0000 7f2132326700 1 -- 172.21.15.16:0/1682611839 wait complete. 2023-10-16T12:31:11.861 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 78 2023-10-16T12:31:12.273 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 78, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T12:31:05.836924+0000', 'modified': '2023-10-16T12:31:08.845101+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9204}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9204': {'gid': 9204, 'name': 'c', 'rank': 0, 'incarnation': 74, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.16:6836/1477717813', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6834', 'nonce': 1477717813}, {'type': 'v1', 'addr': '172.21.15.16:6836', 'nonce': 1477717813}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [20], 'metadata_pool': 19, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-16T12:31:12.273 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-16T12:31:12.274 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-10-16T12:31:12.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.405+0000 7f16b923c700 1 Processor -- start 2023-10-16T12:31:12.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.405+0000 7f16b923c700 1 -- start start 2023-10-16T12:31:12.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.406+0000 7f16b923c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f16b40cc8a0 0x7f16b415e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:12.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.406+0000 7f16b923c700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f16b406a7e0 0x7f16b406ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:12.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.406+0000 7f16b923c700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16b406b170 con 0x7f16b415b280 2023-10-16T12:31:12.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.406+0000 7f16b923c700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16b4161c30 con 0x7f16b406a7e0 2023-10-16T12:31:12.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.406+0000 7f16b923c700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16b4161f10 con 0x7f16b40cc8a0 2023-10-16T12:31:12.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.406+0000 7f16b2d9d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f16b415b280 0x7f16b41613e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36270/0 (socket says 172.21.15.16:36270) 2023-10-16T12:31:12.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.406+0000 7f16b2d9d700 1 -- 172.21.15.16:0/4059553818 learned_addr learned my addr 172.21.15.16:0/4059553818 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:31:12.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.406+0000 7f16b259c700 1 --2- 172.21.15.16:0/4059553818 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f16b40cc8a0 0x7f16b415e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:12.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.406+0000 7f16b359e700 1 --2- 172.21.15.16:0/4059553818 >> v2:172.21.15.49:3300/0 conn(0x7f16b406a7e0 0x7f16b406ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:12.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.406+0000 7f16b1d9b700 1 -- 172.21.15.16:0/4059553818 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 293875577 0 0) 0x7f16a8001110 con 0x7f16b415b280 2023-10-16T12:31:12.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.406+0000 7f16b1d9b700 1 -- 172.21.15.16:0/4059553818 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f169c0035b0 con 0x7f16b415b280 2023-10-16T12:31:12.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.406+0000 7f16b359e700 1 -- 172.21.15.16:0/4059553818 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f16b40cc8a0 msgr2=0x7f16b415e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:31:12.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.406+0000 7f16b359e700 1 --2- 172.21.15.16:0/4059553818 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f16b40cc8a0 0x7f16b415e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:12.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.406+0000 7f16b359e700 1 -- 172.21.15.16:0/4059553818 >> v1:172.21.15.16:6789/0 conn(0x7f16b415b280 legacy=0x7f16b41613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:12.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.406+0000 7f16b1d9b700 1 -- 172.21.15.16:0/4059553818 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 279280989 0 0) 0x7f169c0035b0 con 0x7f16b415b280 2023-10-16T12:31:12.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.407+0000 7f16b359e700 1 -- 172.21.15.16:0/4059553818 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16b41621f0 con 0x7f16b406a7e0 2023-10-16T12:31:12.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.407+0000 7f16b359e700 1 --2- 172.21.15.16:0/4059553818 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f16b406a7e0 0x7f16b406ac30 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f16ac009d90 tx=0x7f16ac00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2862db6842da50c9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:12.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.407+0000 7f16b1d9b700 1 -- 172.21.15.16:0/4059553818 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16ac00e070 con 0x7f16b406a7e0 2023-10-16T12:31:12.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.407+0000 7f16b1d9b700 1 -- 172.21.15.16:0/4059553818 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16ac003a20 con 0x7f16b406a7e0 2023-10-16T12:31:12.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.407+0000 7f16b1d9b700 1 -- 172.21.15.16:0/4059553818 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16a8001800 con 0x7f16b406a7e0 2023-10-16T12:31:12.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.407+0000 7f16b923c700 1 -- 172.21.15.16:0/4059553818 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f16b406a7e0 msgr2=0x7f16b406ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:12.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.407+0000 7f16b923c700 1 --2- 172.21.15.16:0/4059553818 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f16b406a7e0 0x7f16b406ac30 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f16ac009d90 tx=0x7f16ac00b330 comp rx=0 tx=0).stop 2023-10-16T12:31:12.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.407+0000 7f16b923c700 1 -- 172.21.15.16:0/4059553818 shutdown_connections 2023-10-16T12:31:12.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.407+0000 7f16b923c700 1 --2- 172.21.15.16:0/4059553818 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f16b406a7e0 0x7f16b406ac30 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:12.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.407+0000 7f16b923c700 1 --2- 172.21.15.16:0/4059553818 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f16b40cc8a0 0x7f16b415e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:12.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.407+0000 7f16b923c700 1 -- 172.21.15.16:0/4059553818 >> 172.21.15.16:0/4059553818 conn(0x7f16b40b1fd0 msgr2=0x7f16b40b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:31:12.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.408+0000 7f16b923c700 1 -- 172.21.15.16:0/4059553818 shutdown_connections 2023-10-16T12:31:12.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.408+0000 7f16b923c700 1 -- 172.21.15.16:0/4059553818 wait complete. 2023-10-16T12:31:12.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.409+0000 7f16b923c700 1 Processor -- start 2023-10-16T12:31:12.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.409+0000 7f16b923c700 1 -- start start 2023-10-16T12:31:12.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.410+0000 7f16b923c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16b406a7e0 0x7f16b4156bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:12.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.410+0000 7f16b923c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f16b40cc8a0 0x7f16b40b0f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:12.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.410+0000 7f16b923c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f16b415b280 0x7f16b40c65b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:12.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.410+0000 7f16b923c700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16b4157130 con 0x7f16b406a7e0 2023-10-16T12:31:12.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.410+0000 7f16b923c700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16b40c4960 con 0x7f16b415b280 2023-10-16T12:31:12.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.410+0000 7f16b923c700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16b40b0a70 con 0x7f16b40cc8a0 2023-10-16T12:31:12.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.410+0000 7f16b2d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16b406a7e0 0x7f16b4156bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:12.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.410+0000 7f16b2d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16b406a7e0 0x7f16b4156bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47004/0 (socket says 172.21.15.16:47004) 2023-10-16T12:31:12.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.410+0000 7f16b2d9d700 1 -- 172.21.15.16:0/2901575350 learned_addr learned my addr 172.21.15.16:0/2901575350 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:31:12.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.410+0000 7f16b259c700 1 --2- 172.21.15.16:0/2901575350 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f16b40cc8a0 0x7f16b40b0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:12.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.410+0000 7f16b359e700 1 --2- 172.21.15.16:0/2901575350 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f16b415b280 0x7f16b40c65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:12.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.410+0000 7f16b2d9d700 1 -- 172.21.15.16:0/2901575350 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f16b40cc8a0 msgr2=0x7f16b40b0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:12.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.410+0000 7f16b2d9d700 1 --2- 172.21.15.16:0/2901575350 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f16b40cc8a0 0x7f16b40b0f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:12.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.410+0000 7f16b2d9d700 1 -- 172.21.15.16:0/2901575350 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f16b415b280 msgr2=0x7f16b40c65b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:12.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.410+0000 7f16b2d9d700 1 --2- 172.21.15.16:0/2901575350 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f16b415b280 0x7f16b40c65b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:12.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.410+0000 7f16b2d9d700 1 -- 172.21.15.16:0/2901575350 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16ac009a70 con 0x7f16b406a7e0 2023-10-16T12:31:12.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.410+0000 7f16b2d9d700 1 --2- 172.21.15.16:0/2901575350 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16b406a7e0 0x7f16b4156bf0 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7f16a800c520 tx=0x7f16a800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:12.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.411+0000 7f169b7fe700 1 -- 172.21.15.16:0/2901575350 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16a8004c80 con 0x7f16b406a7e0 2023-10-16T12:31:12.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.411+0000 7f169b7fe700 1 -- 172.21.15.16:0/2901575350 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16a8004e00 con 0x7f16b406a7e0 2023-10-16T12:31:12.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.411+0000 7f16b923c700 1 -- 172.21.15.16:0/2901575350 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16b40c5170 con 0x7f16b406a7e0 2023-10-16T12:31:12.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.411+0000 7f169b7fe700 1 -- 172.21.15.16:0/2901575350 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16a8007310 con 0x7f16b406a7e0 2023-10-16T12:31:12.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.411+0000 7f16b923c700 1 -- 172.21.15.16:0/2901575350 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16b40b0720 con 0x7f16b406a7e0 2023-10-16T12:31:12.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.412+0000 7f169b7fe700 1 -- 172.21.15.16:0/2901575350 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 85) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f16a8007470 con 0x7f16b406a7e0 2023-10-16T12:31:12.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.413+0000 7f169b7fe700 1 --2- 172.21.15.16:0/2901575350 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f16a006b020 0x7f16a006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:12.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.413+0000 7f169b7fe700 1 -- 172.21.15.16:0/2901575350 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 6085+0+0 (secure 0 0 0) 0x7f16a8089420 con 0x7f16b406a7e0 2023-10-16T12:31:12.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.413+0000 7f16b259c700 1 --2- 172.21.15.16:0/2901575350 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f16a006b020 0x7f16a006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:12.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.413+0000 7f16b259c700 1 --2- 172.21.15.16:0/2901575350 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f16a006b020 0x7f16a006d4e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f16b41f61b0 tx=0x7f16a4009210 comp rx=0 tx=0).ready entity=mgr.8973 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:12.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.414+0000 7f16c5426b80 1 Processor -- start 2023-10-16T12:31:12.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.415+0000 7f16c5426b80 1 -- start start 2023-10-16T12:31:12.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.415+0000 7f16c5426b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55758c28b220 0x55758c292a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:12.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.415+0000 7f16c5426b80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55758c290630 0x55758c292fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:12.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.415+0000 7f16c5426b80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55758c266010 0x55758c2666c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:12.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.415+0000 7f16c5426b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55758c1dcc20 con 0x55758c28b220 2023-10-16T12:31:12.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.415+0000 7f16c5426b80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x55758c1c1c10 con 0x55758c266010 2023-10-16T12:31:12.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.415+0000 7f16c5426b80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x55758c1bf530 con 0x55758c290630 2023-10-16T12:31:12.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.415+0000 7f16b2d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55758c28b220 0x55758c292a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:12.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.415+0000 7f16b2d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55758c28b220 0x55758c292a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47010/0 (socket says 172.21.15.16:47010) 2023-10-16T12:31:12.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.415+0000 7f16b2d9d700 1 -- 172.21.15.16:0/366519090 learned_addr learned my addr 172.21.15.16:0/366519090 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:31:12.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.416+0000 7f16b259c700 1 --2- 172.21.15.16:0/366519090 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55758c290630 0x55758c292fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:12.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.416+0000 7f16b359e700 1 --2- 172.21.15.16:0/366519090 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55758c266010 0x55758c2666c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:12.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.416+0000 7f16b2d9d700 1 -- 172.21.15.16:0/366519090 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55758c290630 msgr2=0x55758c292fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:31:12.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.416+0000 7f16b2d9d700 1 --2- 172.21.15.16:0/366519090 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55758c290630 0x55758c292fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:12.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.416+0000 7f16b2d9d700 1 -- 172.21.15.16:0/366519090 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55758c266010 msgr2=0x55758c2666c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:31:12.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.416+0000 7f16b2d9d700 1 --2- 172.21.15.16:0/366519090 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55758c266010 0x55758c2666c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:12.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.416+0000 7f16b2d9d700 1 -- 172.21.15.16:0/366519090 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55758c1783a0 con 0x55758c28b220 2023-10-16T12:31:12.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.416+0000 7f16b2d9d700 1 --2- 172.21.15.16:0/366519090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55758c28b220 0x55758c292a70 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7f16a8054490 tx=0x7f16a8001530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f259233b2acf2ac7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:12.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.416+0000 7f1698ff9700 1 -- 172.21.15.16:0/366519090 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16a806f070 con 0x55758c28b220 2023-10-16T12:31:12.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.416+0000 7f1698ff9700 1 -- 172.21.15.16:0/366519090 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16a8057f90 con 0x55758c28b220 2023-10-16T12:31:12.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.416+0000 7f1698ff9700 1 -- 172.21.15.16:0/366519090 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16a805c0e0 con 0x55758c28b220 2023-10-16T12:31:12.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.417+0000 7f16c5426b80 1 -- 172.21.15.16:0/366519090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55758c28b220 msgr2=0x55758c292a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:12.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.417+0000 7f16c5426b80 1 --2- 172.21.15.16:0/366519090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55758c28b220 0x55758c292a70 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7f16a8054490 tx=0x7f16a8001530 comp rx=0 tx=0).stop 2023-10-16T12:31:12.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.417+0000 7f16c5426b80 1 -- 172.21.15.16:0/366519090 shutdown_connections 2023-10-16T12:31:12.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.417+0000 7f16c5426b80 1 --2- 172.21.15.16:0/366519090 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55758c266010 0x55758c2666c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:12.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.417+0000 7f16c5426b80 1 --2- 172.21.15.16:0/366519090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55758c28b220 0x55758c292a70 unknown :-1 s=CLOSED pgs=1761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:12.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.417+0000 7f16c5426b80 1 --2- 172.21.15.16:0/366519090 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55758c290630 0x55758c292fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:12.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.417+0000 7f16c5426b80 1 -- 172.21.15.16:0/366519090 >> 172.21.15.16:0/366519090 conn(0x55758c195d80 msgr2=0x55758c2b0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:31:12.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.417+0000 7f16c5426b80 1 -- 172.21.15.16:0/366519090 shutdown_connections 2023-10-16T12:31:12.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.417+0000 7f16c5426b80 1 -- 172.21.15.16:0/366519090 wait complete. 2023-10-16T12:31:12.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.417+0000 7f16c5426b80 1 -- start start 2023-10-16T12:31:12.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.418+0000 7f16c5426b80 1 Processor -- start 2023-10-16T12:31:12.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.418+0000 7f16c5426b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55758c266010 0x55758c28c370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:12.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.418+0000 7f16c5426b80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55758c28b220 0x55758c28c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:12.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.418+0000 7f16c5426b80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55758c290630 0x55758c266450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:12.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.418+0000 7f16c5426b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55758c1bae20 con 0x55758c266010 2023-10-16T12:31:12.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.418+0000 7f16c5426b80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x55758c1576c0 con 0x55758c28b220 2023-10-16T12:31:12.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.418+0000 7f16c5426b80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x55758c157360 con 0x55758c290630 2023-10-16T12:31:12.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.418+0000 7f16c5426b80 10 client.0 _ll_register_callbacks cb 0x55758bfb47b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T12:31:12.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.418+0000 7f16b2d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55758c266010 0x55758c28c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:12.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.419+0000 7f16b2d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55758c266010 0x55758c28c370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47016/0 (socket says 172.21.15.16:47016) 2023-10-16T12:31:12.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.419+0000 7f16b2d9d700 1 -- 172.21.15.16:0/3770982477 learned_addr learned my addr 172.21.15.16:0/3770982477 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:31:12.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.419+0000 7f16b259c700 1 --2- 172.21.15.16:0/3770982477 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55758c28b220 0x55758c28c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:12.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.419+0000 7f16b359e700 1 --2- 172.21.15.16:0/3770982477 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55758c290630 0x55758c266450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:12.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.419+0000 7f16b2d9d700 1 -- 172.21.15.16:0/3770982477 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55758c290630 msgr2=0x55758c266450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:31:12.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.419+0000 7f16b2d9d700 1 --2- 172.21.15.16:0/3770982477 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55758c290630 0x55758c266450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:12.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.419+0000 7f16b2d9d700 1 -- 172.21.15.16:0/3770982477 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55758c28b220 msgr2=0x55758c28c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:31:12.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.419+0000 7f16b2d9d700 1 --2- 172.21.15.16:0/3770982477 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55758c28b220 0x55758c28c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:12.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.419+0000 7f16b2d9d700 1 -- 172.21.15.16:0/3770982477 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16b40b0720 con 0x55758c266010 2023-10-16T12:31:12.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.419+0000 7f16b2d9d700 1 --2- 172.21.15.16:0/3770982477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55758c266010 0x55758c28c370 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7f16a8057c90 tx=0x7f16a8000fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eee6488c65b4a54d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:12.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.419+0000 7f1691ffb700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2023-10-16T12:31:12.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.419+0000 7f1691ffb700 1 -- 172.21.15.16:0/3770982477 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16a8086070 con 0x55758c266010 2023-10-16T12:31:12.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.419+0000 7f1691ffb700 1 -- 172.21.15.16:0/3770982477 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(156..156 src has 1..156) v4 ==== 6085+0+0 (secure 0 0 0) 0x7f16a8060290 con 0x55758c266010 2023-10-16T12:31:12.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.419+0000 7f16c5426b80 1 -- 172.21.15.16:0/3770982477 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55758c25b5b0 con 0x55758c266010 2023-10-16T12:31:12.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.419+0000 7f1691ffb700 1 -- 172.21.15.16:0/3770982477 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=78) v2 ==== 24+0+0 (secure 0 0 0) 0x7f16a806f040 con 0x55758c266010 2023-10-16T12:31:12.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.419+0000 7f16c5426b80 10 client.9363 fetch_fsmap learned FSMap version 78 2023-10-16T12:31:12.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.419+0000 7f16c5426b80 1 -- 172.21.15.16:0/3770982477 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=78,monmap=2+}) v3 -- 0x55758bfe5d90 con 0x55758c266010 2023-10-16T12:31:12.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.419+0000 7f1691ffb700 1 -- 172.21.15.16:0/3770982477 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55758c25b5b0 con 0x55758c266010 2023-10-16T12:31:12.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.420+0000 7f1691ffb700 1 -- 172.21.15.16:0/3770982477 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 78) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f16a80890c0 con 0x55758c266010 2023-10-16T12:31:12.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.420+0000 7f16c5426b80 10 client.9363 fetch_fsmap finished waiting for FSMap version 78 2023-10-16T12:31:12.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.420+0000 7f16c5426b80 10 client.9363 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-16T12:31:12.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.420+0000 7f16c5426b80 20 client.9363 populate_metadata read hostname 'smithi016' 2023-10-16T12:31:12.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.420+0000 7f16c5426b80 1 --2- 172.21.15.16:0/3770982477 >> [v2:172.21.15.16:6834/1477717813,v1:172.21.15.16:6836/1477717813] conn(0x55758c2bd490 0x55758c2bd870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:12.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.420+0000 7f16c5426b80 4 client.9363 mds_command: new command op to 9204 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T12:31:12.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.420+0000 7f16c5426b80 1 -- 172.21.15.16:0/3770982477 --> [v2:172.21.15.16:6834/1477717813,v1:172.21.15.16:6836/1477717813] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55758c1b3e00 con 0x55758c2bd490 2023-10-16T12:31:12.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.420+0000 7f16b259c700 1 --2- 172.21.15.16:0/3770982477 >> [v2:172.21.15.16:6834/1477717813,v1:172.21.15.16:6836/1477717813] conn(0x55758c2bd490 0x55758c2bd870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:12.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.420+0000 7f1691ffb700 10 client.9363 ms_handle_connect on v2:172.21.15.16:6834/1477717813 2023-10-16T12:31:12.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.420+0000 7f16b259c700 1 --2- 172.21.15.16:0/3770982477 >> [v2:172.21.15.16:6834/1477717813,v1:172.21.15.16:6836/1477717813] conn(0x55758c2bd490 0x55758c2bd870 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a6c39aea4cead59 server_cookie=c0e1db3b547be664 in_seq=0 out_seq=0 2023-10-16T12:31:12.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.421+0000 7f1691ffb700 1 -- 172.21.15.16:0/3770982477 <== mds.0 v2:172.21.15.16:6834/1477717813 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55758c1b3e00 con 0x55758c2bd490 2023-10-16T12:31:12.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.421+0000 7f1691ffb700 10 client.9363 handle_command_reply: tid=0 2023-10-16T12:31:12.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.421+0000 7f16c5426b80 1 client.9363 shutdown 2023-10-16T12:31:12.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.421+0000 7f16c5426b80 1 -- 172.21.15.16:0/3770982477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55758c266010 msgr2=0x55758c28c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:12.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.421+0000 7f16c5426b80 1 --2- 172.21.15.16:0/3770982477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55758c266010 0x55758c28c370 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7f16a8057c90 tx=0x7f16a8000fb0 comp rx=0 tx=0).stop 2023-10-16T12:31:12.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.421+0000 7f16c5426b80 1 -- 172.21.15.16:0/3770982477 shutdown_connections 2023-10-16T12:31:12.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.421+0000 7f16c5426b80 1 --2- 172.21.15.16:0/3770982477 >> [v2:172.21.15.16:6834/1477717813,v1:172.21.15.16:6836/1477717813] conn(0x55758c2bd490 0x55758c2bd870 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:12.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.421+0000 7f16c5426b80 1 --2- 172.21.15.16:0/3770982477 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55758c290630 0x55758c266450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:12.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.421+0000 7f16c5426b80 1 --2- 172.21.15.16:0/3770982477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55758c266010 0x55758c28c370 unknown :-1 s=CLOSED pgs=1762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:12.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.421+0000 7f16c5426b80 1 --2- 172.21.15.16:0/3770982477 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55758c28b220 0x55758c28c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:12.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.421+0000 7f1691ffb700 0 client.9363 ms_handle_reset on v2:172.21.15.16:6834/1477717813 2023-10-16T12:31:12.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.421+0000 7f16c5426b80 1 -- 172.21.15.16:0/3770982477 >> 172.21.15.16:0/3770982477 conn(0x55758c195d80 msgr2=0x55758c2b0f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:31:12.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.421+0000 7f16c5426b80 1 -- 172.21.15.16:0/3770982477 shutdown_connections 2023-10-16T12:31:12.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.421+0000 7f16c5426b80 1 -- 172.21.15.16:0/3770982477 wait complete. 2023-10-16T12:31:12.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.421+0000 7f16c5426b80 20 client.9363 trim_cache size 0 max 16384 2023-10-16T12:31:12.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.439+0000 7f16c5426b80 1 Processor -- start 2023-10-16T12:31:12.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.439+0000 7f16c5426b80 1 -- start start 2023-10-16T12:31:12.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.440+0000 7f16c5426b80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55758c266010 0x55758c2c6e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:12.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.440+0000 7f16c5426b80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55758c28b220 0x55758c292a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:12.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.440+0000 7f16c5426b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55758c290630 0x55758c292f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:12.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.440+0000 7f16c5426b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55758c03b8d0 con 0x55758c290630 2023-10-16T12:31:12.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.440+0000 7f16c5426b80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x55758c28dab0 con 0x55758c28b220 2023-10-16T12:31:12.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.440+0000 7f16c5426b80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x55758c290100 con 0x55758c266010 2023-10-16T12:31:12.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.440+0000 7f16b359e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55758c290630 0x55758c292f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:12.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.440+0000 7f16b359e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55758c290630 0x55758c292f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47028/0 (socket says 172.21.15.16:47028) 2023-10-16T12:31:12.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.440+0000 7f16b359e700 1 -- 172.21.15.16:0/935710868 learned_addr learned my addr 172.21.15.16:0/935710868 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:31:12.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.440+0000 7f16b259c700 1 --2- 172.21.15.16:0/935710868 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55758c28b220 0x55758c292a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:12.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.440+0000 7f16b2d9d700 1 --2- 172.21.15.16:0/935710868 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55758c266010 0x55758c2c6e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:12.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.440+0000 7f16b359e700 1 -- 172.21.15.16:0/935710868 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55758c266010 msgr2=0x55758c2c6e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:31:12.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.440+0000 7f16b359e700 1 --2- 172.21.15.16:0/935710868 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55758c266010 0x55758c2c6e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:12.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.440+0000 7f16b359e700 1 -- 172.21.15.16:0/935710868 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55758c28b220 msgr2=0x55758c292a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:31:12.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.440+0000 7f16b359e700 1 --2- 172.21.15.16:0/935710868 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55758c28b220 0x55758c292a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:12.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.440+0000 7f16b359e700 1 -- 172.21.15.16:0/935710868 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55758bfcec10 con 0x55758c290630 2023-10-16T12:31:12.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.440+0000 7f16b259c700 1 --2- 172.21.15.16:0/935710868 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55758c28b220 0x55758c292a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:31:12.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.440+0000 7f16b2d9d700 1 --2- 172.21.15.16:0/935710868 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55758c266010 0x55758c2c6e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:31:12.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.440+0000 7f16b359e700 1 --2- 172.21.15.16:0/935710868 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55758c290630 0x55758c292f60 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7f16ac00b680 tx=0x7f16ac010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c215a1c6ac0cae73 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:12.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.440+0000 7f16997fa700 1 -- 172.21.15.16:0/935710868 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16ac022070 con 0x55758c290630 2023-10-16T12:31:12.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.440+0000 7f16997fa700 1 -- 172.21.15.16:0/935710868 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16ac003e70 con 0x55758c290630 2023-10-16T12:31:12.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.440+0000 7f16997fa700 1 -- 172.21.15.16:0/935710868 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16ac00bc20 con 0x55758c290630 2023-10-16T12:31:12.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.441+0000 7f16c5426b80 1 -- 172.21.15.16:0/935710868 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55758c290630 msgr2=0x55758c292f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:12.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.441+0000 7f16c5426b80 1 --2- 172.21.15.16:0/935710868 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55758c290630 0x55758c292f60 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7f16ac00b680 tx=0x7f16ac010000 comp rx=0 tx=0).stop 2023-10-16T12:31:12.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.441+0000 7f16c5426b80 1 -- 172.21.15.16:0/935710868 shutdown_connections 2023-10-16T12:31:12.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.441+0000 7f16c5426b80 1 --2- 172.21.15.16:0/935710868 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55758c290630 0x55758c292f60 unknown :-1 s=CLOSED pgs=1763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:12.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.441+0000 7f16c5426b80 1 --2- 172.21.15.16:0/935710868 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55758c266010 0x55758c2c6e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:12.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.441+0000 7f16c5426b80 1 --2- 172.21.15.16:0/935710868 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55758c28b220 0x55758c292a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:12.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.441+0000 7f16c5426b80 1 -- 172.21.15.16:0/935710868 >> 172.21.15.16:0/935710868 conn(0x55758c195d80 msgr2=0x55758c25cf40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:31:12.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.441+0000 7f16c5426b80 1 -- 172.21.15.16:0/935710868 shutdown_connections 2023-10-16T12:31:12.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.441+0000 7f16c5426b80 1 -- 172.21.15.16:0/935710868 wait complete. 2023-10-16T12:31:12.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.442+0000 7f16c5426b80 1 -- start start 2023-10-16T12:31:12.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.442+0000 7f16c5426b80 1 Processor -- start 2023-10-16T12:31:12.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.443+0000 7f16c5426b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55758c266010 0x55758c263200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:12.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.443+0000 7f16c5426b80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55758c28b220 0x55758c2a86f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:12.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.443+0000 7f16c5426b80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55758c290630 0x55758c2f9760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:12.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.443+0000 7f16c5426b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55758c2a9170 con 0x55758c266010 2023-10-16T12:31:12.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.443+0000 7f16c5426b80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x55758c263890 con 0x55758c28b220 2023-10-16T12:31:12.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.443+0000 7f16c5426b80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x55758c2fa110 con 0x55758c290630 2023-10-16T12:31:12.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.443+0000 7f16c5426b80 10 client.0 _ll_register_callbacks cb 0x7f16a8057ad0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T12:31:12.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.443+0000 7f16b2d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55758c266010 0x55758c263200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:12.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.443+0000 7f16b2d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55758c266010 0x55758c263200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47030/0 (socket says 172.21.15.16:47030) 2023-10-16T12:31:12.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.443+0000 7f16b2d9d700 1 -- 172.21.15.16:0/2830511657 learned_addr learned my addr 172.21.15.16:0/2830511657 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:31:12.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.443+0000 7f16b259c700 1 --2- 172.21.15.16:0/2830511657 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55758c28b220 0x55758c2a86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:12.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.443+0000 7f16b359e700 1 --2- 172.21.15.16:0/2830511657 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55758c290630 0x55758c2f9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:12.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.443+0000 7f16b2d9d700 1 -- 172.21.15.16:0/2830511657 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55758c290630 msgr2=0x55758c2f9760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:31:12.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.443+0000 7f16b2d9d700 1 --2- 172.21.15.16:0/2830511657 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55758c290630 0x55758c2f9760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:12.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.443+0000 7f16b2d9d700 1 -- 172.21.15.16:0/2830511657 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55758c28b220 msgr2=0x55758c2a86f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:31:12.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.443+0000 7f16b2d9d700 1 --2- 172.21.15.16:0/2830511657 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55758c28b220 0x55758c2a86f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:12.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.443+0000 7f16b2d9d700 1 -- 172.21.15.16:0/2830511657 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16b41621f0 con 0x55758c266010 2023-10-16T12:31:12.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.443+0000 7f16b359e700 1 --2- 172.21.15.16:0/2830511657 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55758c290630 0x55758c2f9760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:31:12.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.444+0000 7f16b2d9d700 1 --2- 172.21.15.16:0/2830511657 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55758c266010 0x55758c263200 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7f16a8007fd0 tx=0x7f16a8015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9d1563bd5a773bb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:12.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.444+0000 7f1691ffb700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2023-10-16T12:31:12.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.444+0000 7f1691ffb700 1 -- 172.21.15.16:0/2830511657 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16a806f040 con 0x55758c266010 2023-10-16T12:31:12.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.444+0000 7f1691ffb700 1 -- 172.21.15.16:0/2830511657 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(156..156 src has 1..156) v4 ==== 6085+0+0 (secure 0 0 0) 0x7f16a8060290 con 0x55758c266010 2023-10-16T12:31:12.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.444+0000 7f16c5426b80 1 -- 172.21.15.16:0/2830511657 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f16a8057f90 con 0x55758c266010 2023-10-16T12:31:12.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.444+0000 7f1691ffb700 1 -- 172.21.15.16:0/2830511657 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=78) v2 ==== 24+0+0 (secure 0 0 0) 0x7f16a80581f0 con 0x55758c266010 2023-10-16T12:31:12.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.444+0000 7f16c5426b80 10 client.9369 fetch_fsmap learned FSMap version 78 2023-10-16T12:31:12.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.444+0000 7f16c5426b80 1 -- 172.21.15.16:0/2830511657 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=78,monmap=2+}) v3 -- 0x55758c00c200 con 0x55758c266010 2023-10-16T12:31:12.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.444+0000 7f1691ffb700 1 -- 172.21.15.16:0/2830511657 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16a8057f90 con 0x55758c266010 2023-10-16T12:31:12.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.444+0000 7f1691ffb700 1 -- 172.21.15.16:0/2830511657 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 78) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f16a8057480 con 0x55758c266010 2023-10-16T12:31:12.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.444+0000 7f16c5426b80 10 client.9369 fetch_fsmap finished waiting for FSMap version 78 2023-10-16T12:31:12.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.444+0000 7f16c5426b80 10 client.9369 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-16T12:31:12.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.444+0000 7f16c5426b80 20 client.9369 populate_metadata read hostname 'smithi016' 2023-10-16T12:31:12.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.444+0000 7f16c5426b80 1 --2- 172.21.15.16:0/2830511657 >> [v2:172.21.15.16:6834/1477717813,v1:172.21.15.16:6836/1477717813] conn(0x55758c2fa790 0x55758c292a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:12.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.444+0000 7f16c5426b80 4 client.9369 mds_command: new command op to 9204 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T12:31:12.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.444+0000 7f16c5426b80 1 -- 172.21.15.16:0/2830511657 --> [v2:172.21.15.16:6834/1477717813,v1:172.21.15.16:6836/1477717813] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55758c1b3b20 con 0x55758c2fa790 2023-10-16T12:31:12.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.444+0000 7f16b259c700 1 --2- 172.21.15.16:0/2830511657 >> [v2:172.21.15.16:6834/1477717813,v1:172.21.15.16:6836/1477717813] conn(0x55758c2fa790 0x55758c292a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:12.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.444+0000 7f1691ffb700 10 client.9369 ms_handle_connect on v2:172.21.15.16:6834/1477717813 2023-10-16T12:31:12.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.444+0000 7f16b259c700 1 --2- 172.21.15.16:0/2830511657 >> [v2:172.21.15.16:6834/1477717813,v1:172.21.15.16:6836/1477717813] conn(0x55758c2fa790 0x55758c292a20 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=db212bd9510a6aa2 server_cookie=8b40050a83182eea in_seq=0 out_seq=0 2023-10-16T12:31:12.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.445+0000 7f1691ffb700 1 -- 172.21.15.16:0/2830511657 <== mds.0 v2:172.21.15.16:6834/1477717813 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x55758c1b3b20 con 0x55758c2fa790 2023-10-16T12:31:12.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.445+0000 7f1691ffb700 10 client.9369 handle_command_reply: tid=0 2023-10-16T12:31:12.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.445+0000 7f16c5426b80 1 client.9369 shutdown 2023-10-16T12:31:12.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.445+0000 7f16c5426b80 1 -- 172.21.15.16:0/2830511657 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55758c266010 msgr2=0x55758c263200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:12.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.445+0000 7f16c5426b80 1 --2- 172.21.15.16:0/2830511657 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55758c266010 0x55758c263200 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7f16a8007fd0 tx=0x7f16a8015000 comp rx=0 tx=0).stop 2023-10-16T12:31:12.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.445+0000 7f16c5426b80 1 -- 172.21.15.16:0/2830511657 shutdown_connections 2023-10-16T12:31:12.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.445+0000 7f16c5426b80 1 --2- 172.21.15.16:0/2830511657 >> [v2:172.21.15.16:6834/1477717813,v1:172.21.15.16:6836/1477717813] conn(0x55758c2fa790 0x55758c292a20 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:12.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.445+0000 7f16c5426b80 1 --2- 172.21.15.16:0/2830511657 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55758c290630 0x55758c2f9760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:12.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.445+0000 7f16c5426b80 1 --2- 172.21.15.16:0/2830511657 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55758c266010 0x55758c263200 unknown :-1 s=CLOSED pgs=1764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:12.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.445+0000 7f1691ffb700 0 client.9369 ms_handle_reset on v2:172.21.15.16:6834/1477717813 2023-10-16T12:31:12.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.445+0000 7f16c5426b80 1 --2- 172.21.15.16:0/2830511657 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55758c28b220 0x55758c2a86f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:12.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.445+0000 7f16c5426b80 1 -- 172.21.15.16:0/2830511657 >> 172.21.15.16:0/2830511657 conn(0x55758c195d80 msgr2=0x55758c25cf40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:31:12.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.445+0000 7f16c5426b80 1 -- 172.21.15.16:0/2830511657 shutdown_connections 2023-10-16T12:31:12.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.445+0000 7f16c5426b80 1 -- 172.21.15.16:0/2830511657 wait complete. 2023-10-16T12:31:12.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.445+0000 7f16c5426b80 20 client.9369 trim_cache size 0 max 16384 2023-10-16T12:31:12.470 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-10-16T12:31:12.470 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "5b28dfa1-b21f-474a-9b2e-9fb77af58254", 2023-10-16T12:31:12.471 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 0, 2023-10-16T12:31:12.471 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 9204, 2023-10-16T12:31:12.471 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2023-10-16T12:31:12.471 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2023-10-16T12:31:12.471 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2023-10-16T12:31:12.472 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 6.0323183269999996, 2023-10-16T12:31:12.472 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 78, 2023-10-16T12:31:12.472 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 156, 2023-10-16T12:31:12.472 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 156, 2023-10-16T12:31:12.472 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 11.924173221 2023-10-16T12:31:12.473 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-10-16T12:31:12.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.445+0000 7f16b923c700 1 -- 172.21.15.16:0/2901575350 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f16a006b020 msgr2=0x7f16a006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:12.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.445+0000 7f16b923c700 1 --2- 172.21.15.16:0/2901575350 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f16a006b020 0x7f16a006d4e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f16b41f61b0 tx=0x7f16a4009210 comp rx=0 tx=0).stop 2023-10-16T12:31:12.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.445+0000 7f16b923c700 1 -- 172.21.15.16:0/2901575350 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16b406a7e0 msgr2=0x7f16b4156bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:12.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.445+0000 7f16b923c700 1 --2- 172.21.15.16:0/2901575350 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16b406a7e0 0x7f16b4156bf0 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7f16a800c520 tx=0x7f16a800c830 comp rx=0 tx=0).stop 2023-10-16T12:31:12.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.445+0000 7f16b923c700 1 -- 172.21.15.16:0/2901575350 shutdown_connections 2023-10-16T12:31:12.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.445+0000 7f16b923c700 1 --2- 172.21.15.16:0/2901575350 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f16a006b020 0x7f16a006d4e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:12.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.446+0000 7f16b923c700 1 --2- 172.21.15.16:0/2901575350 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f16b415b280 0x7f16b40c65b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:12.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.446+0000 7f16b923c700 1 --2- 172.21.15.16:0/2901575350 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16b406a7e0 0x7f16b4156bf0 unknown :-1 s=CLOSED pgs=1760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:12.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.446+0000 7f16b923c700 1 --2- 172.21.15.16:0/2901575350 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f16b40cc8a0 0x7f16b40b0f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:12.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.446+0000 7f16b923c700 1 -- 172.21.15.16:0/2901575350 >> 172.21.15.16:0/2901575350 conn(0x7f16b40b1fd0 msgr2=0x7f16b40ccd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:31:12.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.446+0000 7f16b923c700 1 -- 172.21.15.16:0/2901575350 shutdown_connections 2023-10-16T12:31:12.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.446+0000 7f16b923c700 1 -- 172.21.15.16:0/2901575350 wait complete. 2023-10-16T12:31:12.477 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T12:31:12.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.600+0000 7ff20ac2b700 1 Processor -- start 2023-10-16T12:31:12.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.600+0000 7ff20ac2b700 1 -- start start 2023-10-16T12:31:12.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.601+0000 7ff20ac2b700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7ff20406d350 0x7ff20406eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:12.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.601+0000 7ff20ac2b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff2040cbed0 0x7ff2040cde10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:12.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.601+0000 7ff20ac2b700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff20406f0a0 con 0x7ff20415a870 2023-10-16T12:31:12.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.601+0000 7ff20ac2b700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff20415ac50 con 0x7ff20406d350 2023-10-16T12:31:12.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.601+0000 7ff20ac2b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff20406f380 con 0x7ff2040cbed0 2023-10-16T12:31:12.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.601+0000 7ff203fff700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7ff20406d350 0x7ff20406eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:12.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.601+0000 7ff2091c8700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff2040cbed0 0x7ff2040cde10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:12.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.601+0000 7ff2091c8700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff2040cbed0 0x7ff2040cde10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:35300/0 (socket says 172.21.15.16:35300) 2023-10-16T12:31:12.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.601+0000 7ff203fff700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7ff20406d350 0x7ff20406eaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:46102/0 (socket says 172.21.15.16:46102) 2023-10-16T12:31:12.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.601+0000 7ff2091c8700 1 -- 172.21.15.16:0/3127109828 learned_addr learned my addr 172.21.15.16:0/3127109828 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:31:12.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.601+0000 7ff2089c7700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7ff20415a870 0x7ff2040cb7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:43494/0 (socket says 172.21.15.16:43494) 2023-10-16T12:31:12.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.601+0000 7ff203fff700 1 -- 172.21.15.16:0/3127109828 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff2040cbed0 msgr2=0x7ff2040cde10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:31:12.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.601+0000 7ff203fff700 1 --2- 172.21.15.16:0/3127109828 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff2040cbed0 0x7ff2040cde10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:12.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.601+0000 7ff203fff700 1 -- 172.21.15.16:0/3127109828 >> v1:172.21.15.49:6789/0 conn(0x7ff20415a870 legacy=0x7ff2040cb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:12.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.602+0000 7ff203fff700 1 -- 172.21.15.16:0/3127109828 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff20406f660 con 0x7ff20406d350 2023-10-16T12:31:12.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.602+0000 7ff203fff700 1 --2- 172.21.15.16:0/3127109828 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff20406d350 0x7ff20406eaa0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7ff1f0009840 tx=0x7ff1f0009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f4e45affa496d9b0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:12.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.602+0000 7ff2037fe700 1 -- 172.21.15.16:0/3127109828 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f0011830 con 0x7ff20406d350 2023-10-16T12:31:12.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.602+0000 7ff2037fe700 1 -- 172.21.15.16:0/3127109828 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1f0011990 con 0x7ff20406d350 2023-10-16T12:31:12.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.602+0000 7ff2037fe700 1 -- 172.21.15.16:0/3127109828 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f0011d20 con 0x7ff20406d350 2023-10-16T12:31:12.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.602+0000 7ff20ac2b700 1 -- 172.21.15.16:0/3127109828 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff20406d350 msgr2=0x7ff20406eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:12.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.602+0000 7ff20ac2b700 1 --2- 172.21.15.16:0/3127109828 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff20406d350 0x7ff20406eaa0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7ff1f0009840 tx=0x7ff1f0009b50 comp rx=0 tx=0).stop 2023-10-16T12:31:12.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.603+0000 7ff20ac2b700 1 -- 172.21.15.16:0/3127109828 shutdown_connections 2023-10-16T12:31:12.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.603+0000 7ff20ac2b700 1 --2- 172.21.15.16:0/3127109828 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff2040cbed0 0x7ff2040cde10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:12.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.603+0000 7ff20ac2b700 1 --2- 172.21.15.16:0/3127109828 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff20406d350 0x7ff20406eaa0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:12.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.603+0000 7ff20ac2b700 1 -- 172.21.15.16:0/3127109828 >> 172.21.15.16:0/3127109828 conn(0x7ff2040cc5b0 msgr2=0x7ff2040cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:31:12.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.603+0000 7ff20ac2b700 1 -- 172.21.15.16:0/3127109828 shutdown_connections 2023-10-16T12:31:12.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.603+0000 7ff20ac2b700 1 -- 172.21.15.16:0/3127109828 wait complete. 2023-10-16T12:31:12.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.604+0000 7ff20ac2b700 1 Processor -- start 2023-10-16T12:31:12.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.604+0000 7ff20ac2b700 1 -- start start 2023-10-16T12:31:12.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.605+0000 7ff20ac2b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff20406d350 0x7ff20414e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:12.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.605+0000 7ff20ac2b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff2040cbed0 0x7ff2040afe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:12.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.605+0000 7ff20ac2b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff20415a870 0x7ff2040b03c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:12.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.605+0000 7ff20ac2b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2040ad7c0 con 0x7ff20406d350 2023-10-16T12:31:12.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.605+0000 7ff20ac2b700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2040ac8d0 con 0x7ff20415a870 2023-10-16T12:31:12.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.605+0000 7ff20ac2b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2040acbb0 con 0x7ff2040cbed0 2023-10-16T12:31:12.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.605+0000 7ff2089c7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff20406d350 0x7ff20414e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:12.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.606+0000 7ff2089c7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff20406d350 0x7ff20414e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47044/0 (socket says 172.21.15.16:47044) 2023-10-16T12:31:12.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.606+0000 7ff2089c7700 1 -- 172.21.15.16:0/3369706188 learned_addr learned my addr 172.21.15.16:0/3369706188 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:31:12.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.606+0000 7ff203fff700 1 --2- 172.21.15.16:0/3369706188 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff2040cbed0 0x7ff2040afe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:12.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.606+0000 7ff2091c8700 1 --2- 172.21.15.16:0/3369706188 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff20415a870 0x7ff2040b03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:12.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.606+0000 7ff2089c7700 1 -- 172.21.15.16:0/3369706188 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff2040cbed0 msgr2=0x7ff2040afe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:12.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.606+0000 7ff2089c7700 1 --2- 172.21.15.16:0/3369706188 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff2040cbed0 0x7ff2040afe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:12.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.606+0000 7ff2089c7700 1 -- 172.21.15.16:0/3369706188 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff20415a870 msgr2=0x7ff2040b03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:12.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.606+0000 7ff2089c7700 1 --2- 172.21.15.16:0/3369706188 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff20415a870 0x7ff2040b03c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:12.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.606+0000 7ff2089c7700 1 -- 172.21.15.16:0/3369706188 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1f00094a0 con 0x7ff20406d350 2023-10-16T12:31:12.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.606+0000 7ff2089c7700 1 --2- 172.21.15.16:0/3369706188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff20406d350 0x7ff20414e3d0 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7ff1f800c430 tx=0x7ff1f800c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:12.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.606+0000 7ff2017fa700 1 -- 172.21.15.16:0/3369706188 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f8005cb0 con 0x7ff20406d350 2023-10-16T12:31:12.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.606+0000 7ff2017fa700 1 -- 172.21.15.16:0/3369706188 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1f8005e10 con 0x7ff20406d350 2023-10-16T12:31:12.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.606+0000 7ff20ac2b700 1 -- 172.21.15.16:0/3369706188 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2040ace90 con 0x7ff20406d350 2023-10-16T12:31:12.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.606+0000 7ff2017fa700 1 -- 172.21.15.16:0/3369706188 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f80072d0 con 0x7ff20406d350 2023-10-16T12:31:12.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.606+0000 7ff20ac2b700 1 -- 172.21.15.16:0/3369706188 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2040ad450 con 0x7ff20406d350 2023-10-16T12:31:12.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.607+0000 7ff20ac2b700 1 -- 172.21.15.16:0/3369706188 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff20404ebb0 con 0x7ff20406d350 2023-10-16T12:31:12.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.608+0000 7ff2017fa700 1 -- 172.21.15.16:0/3369706188 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 85) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff1f8007430 con 0x7ff20406d350 2023-10-16T12:31:12.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.608+0000 7ff2017fa700 1 --2- 172.21.15.16:0/3369706188 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7ff1e806b020 0x7ff1e806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:12.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.608+0000 7ff2017fa700 1 -- 172.21.15.16:0/3369706188 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 6085+0+0 (secure 0 0 0) 0x7ff1f8088b10 con 0x7ff20406d350 2023-10-16T12:31:12.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.608+0000 7ff203fff700 1 --2- 172.21.15.16:0/3369706188 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7ff1e806b020 0x7ff1e806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:12.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.608+0000 7ff203fff700 1 --2- 172.21.15.16:0/3369706188 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7ff1e806b020 0x7ff1e806d4e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff1f0000c00 tx=0x7ff1f0008030 comp rx=0 tx=0).ready entity=mgr.8973 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:12.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.613+0000 7ff2017fa700 1 -- 172.21.15.16:0/3369706188 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff1f8053a70 con 0x7ff20406d350 2023-10-16T12:31:12.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.862+0000 7ff20ac2b700 1 -- 172.21.15.16:0/3369706188 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7ff2040caf80 con 0x7ff20406d350 2023-10-16T12:31:12.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.862+0000 7ff2017fa700 1 -- 172.21.15.16:0/3369706188 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v78) v1 ==== 53+0+73 (secure 0 0 0) 0x7ff1f8017020 con 0x7ff20406d350 2023-10-16T12:31:12.864 INFO:teuthology.orchestra.run.smithi016.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-16T12:31:12.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.864+0000 7ff20ac2b700 1 -- 172.21.15.16:0/3369706188 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7ff1e806b020 msgr2=0x7ff1e806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:12.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.864+0000 7ff20ac2b700 1 --2- 172.21.15.16:0/3369706188 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7ff1e806b020 0x7ff1e806d4e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff1f0000c00 tx=0x7ff1f0008030 comp rx=0 tx=0).stop 2023-10-16T12:31:12.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.864+0000 7ff20ac2b700 1 -- 172.21.15.16:0/3369706188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff20406d350 msgr2=0x7ff20414e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:12.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.864+0000 7ff20ac2b700 1 --2- 172.21.15.16:0/3369706188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff20406d350 0x7ff20414e3d0 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7ff1f800c430 tx=0x7ff1f800c7f0 comp rx=0 tx=0).stop 2023-10-16T12:31:12.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.864+0000 7ff20ac2b700 1 -- 172.21.15.16:0/3369706188 shutdown_connections 2023-10-16T12:31:12.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.864+0000 7ff20ac2b700 1 --2- 172.21.15.16:0/3369706188 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7ff1e806b020 0x7ff1e806d4e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:12.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.864+0000 7ff20ac2b700 1 --2- 172.21.15.16:0/3369706188 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff20415a870 0x7ff2040b03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:12.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.864+0000 7ff20ac2b700 1 --2- 172.21.15.16:0/3369706188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff20406d350 0x7ff20414e3d0 unknown :-1 s=CLOSED pgs=1765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:12.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.864+0000 7ff20ac2b700 1 --2- 172.21.15.16:0/3369706188 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff2040cbed0 0x7ff2040afe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:12.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.864+0000 7ff20ac2b700 1 -- 172.21.15.16:0/3369706188 >> 172.21.15.16:0/3369706188 conn(0x7ff2040cc5b0 msgr2=0x7ff2040cb780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:31:12.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.864+0000 7ff20ac2b700 1 -- 172.21.15.16:0/3369706188 shutdown_connections 2023-10-16T12:31:12.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:12.864+0000 7ff20ac2b700 1 -- 172.21.15.16:0/3369706188 wait complete. 2023-10-16T12:31:12.878 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-16T12:31:12.879 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T12:31:12.879 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi016.front.sepia.ceph.com 2023-10-16T12:31:12.879 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-16T12:31:12.880 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-16T12:31:12.880 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-16T12:31:12.880 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-16T12:31:12.881 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T12:31:12.881 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T12:31:12.881 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:31:12.882 DEBUG:teuthology.orchestra.run.smithi016:> ip addr 2023-10-16T12:31:12.900 INFO:teuthology.orchestra.run.smithi016.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T12:31:12.900 INFO:teuthology.orchestra.run.smithi016.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T12:31:12.900 INFO:teuthology.orchestra.run.smithi016.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T12:31:12.900 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:31:12.901 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 ::1/128 scope host 2023-10-16T12:31:12.901 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:31:12.901 INFO:teuthology.orchestra.run.smithi016.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T12:31:12.901 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e0 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:31:12.902 INFO:teuthology.orchestra.run.smithi016.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T12:31:12.902 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e1 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:31:12.902 INFO:teuthology.orchestra.run.smithi016.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T12:31:12.902 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:64 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:31:12.902 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.21.15.16/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-16T12:31:12.903 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft 41063sec preferred_lft 41063sec 2023-10-16T12:31:12.903 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::ec4:7aff:febd:1564/64 scope link 2023-10-16T12:31:12.903 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:31:12.903 INFO:teuthology.orchestra.run.smithi016.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T12:31:12.903 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:65 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:31:12.903 INFO:teuthology.orchestra.run.smithi016.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T12:31:12.904 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 36:cf:ea:25:db:f7 brd ff:ff:ff:ff:ff:ff 2023-10-16T12:31:12.904 INFO:teuthology.orchestra.run.smithi016.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-16T12:31:12.904 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:31:12.904 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::cfc:c5ff:fe52:a786/64 scope link 2023-10-16T12:31:12.904 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:31:12.905 INFO:teuthology.orchestra.run.smithi016.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T12:31:12.905 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 6e:62:9d:a4:43:23 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T12:31:12.905 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::6c62:9dff:fea4:4323/64 scope link 2023-10-16T12:31:12.905 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:31:12.905 INFO:teuthology.orchestra.run.smithi016.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T12:31:12.906 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 36:cf:ea:25:db:f7 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-16T12:31:12.906 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::34cf:eaff:fe25:dbf7/64 scope link 2023-10-16T12:31:12.906 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T12:31:12.906 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:31:12.907 DEBUG:teuthology.orchestra.run.smithi016:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T12:31:12.965 INFO:teuthology.orchestra.run.smithi016.stdout:1 2023-10-16T12:31:12.967 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:31:12.967 DEBUG:teuthology.orchestra.run.smithi016:> route 2023-10-16T12:31:13.026 INFO:teuthology.orchestra.run.smithi016.stdout:Kernel IP routing table 2023-10-16T12:31:13.027 INFO:teuthology.orchestra.run.smithi016.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T12:31:13.027 INFO:teuthology.orchestra.run.smithi016.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-16T12:31:13.027 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-16T12:31:13.027 INFO:teuthology.orchestra.run.smithi016.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T12:31:13.027 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:31:13.028 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec bash -c ' 2023-10-16T12:31:13.028 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2023-10-16T12:31:13.028 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-16T12:31:13.029 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-16T12:31:13.029 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-16T12:31:13.029 DEBUG:teuthology.orchestra.run.smithi016:> ') 2023-10-16T12:31:13.133 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:31:13.134 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2023-10-16T12:31:13.151 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-16T12:31:13.151 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-16T12:31:13.151 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T12:31:13.151 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:31:13.152 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set brx.0 up 2023-10-16T12:31:13.217 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T12:31:13.218 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-16T12:31:13.275 INFO:teuthology.orchestra.run.smithi016.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-16T12:31:13.275 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T12:31:13.275 DEBUG:teuthology.orchestra.run.smithi016:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-16T12:31:13.331 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-16T12:31:13.332 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:31:13.332 DEBUG:teuthology.orchestra.run.smithi016:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T12:31:13.418 INFO:teuthology.orchestra.run.smithi016.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T12:31:13.419 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T12:31:13.419 INFO:teuthology.orchestra.run.smithi016.stdout:invalid new device string format 2023-10-16T12:31:13.428 INFO:teuthology.orchestra.run.smithi016.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T12:31:13.428 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T12:31:13.428 INFO:teuthology.orchestra.run.smithi016.stdout:invalid new device string format 2023-10-16T12:31:13.428 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: resolved to: "172.21.15.16:3300,172.21.15.49:3300,172.21.15.49:3301" 2023-10-16T12:31:13.429 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T12:31:13.501 INFO:teuthology.orchestra.run.smithi016.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T12:31:13.501 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T12:31:13.501 INFO:teuthology.orchestra.run.smithi016.stdout:invalid new device string format 2023-10-16T12:31:13.502 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: resolved to: "172.21.15.16:3300,172.21.15.49:3300,172.21.15.49:3301" 2023-10-16T12:31:13.502 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T12:31:13.502 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: resolved to: "172.21.15.16:6789,172.21.15.49:6789,172.21.15.49:6790" 2023-10-16T12:31:13.502 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.16:6789,172.21.15.49:6789,172.21.15.49:6790:/ 2023-10-16T12:31:13.502 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=5b28dfa1-b21f-474a-9b2e-9fb77af58254" will pass to kernel 2023-10-16T12:31:13.503 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-16T12:31:13.504 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:31:13.504 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-16T12:31:13.532 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:31:13.532 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T12:31:13.619 INFO:teuthology.orchestra.run.smithi016.stdout:client8746 2023-10-16T12:31:13.619 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T12:31:13.620 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T12:31:13.637 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-16T12:31:13.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:13.802+0000 7f4891367700 1 Processor -- start 2023-10-16T12:31:13.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:13.802+0000 7f4891367700 1 -- start start 2023-10-16T12:31:13.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:13.802+0000 7f4891367700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f488c15da40 0x7f488c0b2f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:13.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:13.803+0000 7f4891367700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f488c0b34c0 0x7f488c06bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:13.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:13.803+0000 7f4891367700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f488c06c210 0x7f488c070530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:13.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:13.803+0000 7f4891367700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f488c0cc4d0 con 0x7f488c06c210 2023-10-16T12:31:13.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:13.803+0000 7f4891367700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f488c0b3950 con 0x7f488c15da40 2023-10-16T12:31:13.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:13.803+0000 7f4891367700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f488c06ccc0 con 0x7f488c0b34c0 2023-10-16T12:31:13.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:13.803+0000 7f488b7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f488c06c210 0x7f488c070530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:13.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:13.803+0000 7f488b7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f488c06c210 0x7f488c070530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47060/0 (socket says 172.21.15.16:47060) 2023-10-16T12:31:13.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:13.803+0000 7f488b7fe700 1 -- 172.21.15.16:0/2136706751 learned_addr learned my addr 172.21.15.16:0/2136706751 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:31:13.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:13.803+0000 7f488b7fe700 1 -- 172.21.15.16:0/2136706751 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f488c0b34c0 msgr2=0x7f488c06bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:31:13.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:13.803+0000 7f488a7fc700 1 --2- 172.21.15.16:0/2136706751 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f488c0b34c0 0x7f488c06bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:13.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:13.803+0000 7f488affd700 1 --2- 172.21.15.16:0/2136706751 >> v2:172.21.15.49:3300/0 conn(0x7f488c15da40 0x7f488c0b2f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:13.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:13.803+0000 7f488b7fe700 1 --2- 172.21.15.16:0/2136706751 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f488c0b34c0 0x7f488c06bcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:13.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:13.803+0000 7f488b7fe700 1 -- 172.21.15.16:0/2136706751 >> v2:172.21.15.49:3300/0 conn(0x7f488c15da40 msgr2=0x7f488c0b2f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:31:13.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:13.803+0000 7f488b7fe700 1 --2- 172.21.15.16:0/2136706751 >> v2:172.21.15.49:3300/0 conn(0x7f488c15da40 0x7f488c0b2f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:13.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:13.803+0000 7f488b7fe700 1 -- 172.21.15.16:0/2136706751 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f488c0cbb70 con 0x7f488c06c210 2023-10-16T12:31:13.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:13.803+0000 7f488b7fe700 1 --2- 172.21.15.16:0/2136706751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f488c06c210 0x7f488c070530 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7f488c071540 tx=0x7f488400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f378757f84f6013b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:13.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:13.803+0000 7f4889ffb700 1 -- 172.21.15.16:0/2136706751 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f488400e070 con 0x7f488c06c210 2023-10-16T12:31:13.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:13.803+0000 7f4889ffb700 1 -- 172.21.15.16:0/2136706751 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f488400b980 con 0x7f488c06c210 2023-10-16T12:31:13.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:13.803+0000 7f4889ffb700 1 -- 172.21.15.16:0/2136706751 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f488400bd30 con 0x7f488c06c210 2023-10-16T12:31:13.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:13.804+0000 7f4891367700 1 -- 172.21.15.16:0/2136706751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f488c06c210 msgr2=0x7f488c070530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:13.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:13.804+0000 7f4891367700 1 --2- 172.21.15.16:0/2136706751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f488c06c210 0x7f488c070530 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7f488c071540 tx=0x7f488400b330 comp rx=0 tx=0).stop 2023-10-16T12:31:13.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:13.804+0000 7f4891367700 1 -- 172.21.15.16:0/2136706751 shutdown_connections 2023-10-16T12:31:13.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:13.804+0000 7f4891367700 1 --2- 172.21.15.16:0/2136706751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f488c06c210 0x7f488c070530 unknown :-1 s=CLOSED pgs=1766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:13.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:13.804+0000 7f4891367700 1 --2- 172.21.15.16:0/2136706751 >> v2:172.21.15.49:3300/0 conn(0x7f488c15da40 0x7f488c0b2f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:13.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:13.804+0000 7f4891367700 1 --2- 172.21.15.16:0/2136706751 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f488c0b34c0 0x7f488c06bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:13.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:13.804+0000 7f4891367700 1 -- 172.21.15.16:0/2136706751 >> 172.21.15.16:0/2136706751 conn(0x7f488c0cdf80 msgr2=0x7f488c0ce390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:31:13.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:13.804+0000 7f4891367700 1 -- 172.21.15.16:0/2136706751 shutdown_connections 2023-10-16T12:31:13.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:13.804+0000 7f4891367700 1 -- 172.21.15.16:0/2136706751 wait complete. 2023-10-16T12:31:13.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:13.805+0000 7f4891367700 1 Processor -- start 2023-10-16T12:31:13.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:13.805+0000 7f4891367700 1 -- start start 2023-10-16T12:31:13.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:13.806+0000 7f4891367700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f488c06c210 0x7f488c0c5f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:13.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:13.806+0000 7f4891367700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f488c0b34c0 0x7f488c14fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:13.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:13.806+0000 7f4891367700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f488c15da40 0x7f488c0b41a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:13.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:13.806+0000 7f4891367700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f488c0c7170 con 0x7f488c15da40 2023-10-16T12:31:13.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:13.806+0000 7f4891367700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f488c0c64f0 con 0x7f488c0b34c0 2023-10-16T12:31:13.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:13.806+0000 7f4891367700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f488c0c67d0 con 0x7f488c06c210 2023-10-16T12:31:13.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:13.806+0000 7f488b7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f488c15da40 0x7f488c0b41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:13.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:13.806+0000 7f488b7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f488c15da40 0x7f488c0b41a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47066/0 (socket says 172.21.15.16:47066) 2023-10-16T12:31:13.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:13.806+0000 7f488b7fe700 1 -- 172.21.15.16:0/3252250158 learned_addr learned my addr 172.21.15.16:0/3252250158 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:31:13.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:13.806+0000 7f488a7fc700 1 --2- 172.21.15.16:0/3252250158 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f488c0b34c0 0x7f488c14fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:13.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:13.806+0000 7f488affd700 1 --2- 172.21.15.16:0/3252250158 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f488c06c210 0x7f488c0c5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:13.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:13.806+0000 7f488b7fe700 1 -- 172.21.15.16:0/3252250158 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f488c06c210 msgr2=0x7f488c0c5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:13.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:13.806+0000 7f488b7fe700 1 --2- 172.21.15.16:0/3252250158 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f488c06c210 0x7f488c0c5f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:13.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:13.806+0000 7f488b7fe700 1 -- 172.21.15.16:0/3252250158 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f488c0b34c0 msgr2=0x7f488c14fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:13.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:13.806+0000 7f488b7fe700 1 --2- 172.21.15.16:0/3252250158 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f488c0b34c0 0x7f488c14fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:13.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:13.806+0000 7f488b7fe700 1 -- 172.21.15.16:0/3252250158 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4884009b30 con 0x7f488c15da40 2023-10-16T12:31:13.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:13.806+0000 7f488a7fc700 1 --2- 172.21.15.16:0/3252250158 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f488c0b34c0 0x7f488c14fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:31:13.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:13.806+0000 7f488affd700 1 --2- 172.21.15.16:0/3252250158 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f488c06c210 0x7f488c0c5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:31:13.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:13.807+0000 7f488b7fe700 1 --2- 172.21.15.16:0/3252250158 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f488c15da40 0x7f488c0b41a0 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7f488c0d2530 tx=0x7f4884004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:13.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:13.807+0000 7f48737fe700 1 -- 172.21.15.16:0/3252250158 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48840042d0 con 0x7f488c15da40 2023-10-16T12:31:13.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:13.807+0000 7f48737fe700 1 -- 172.21.15.16:0/3252250158 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4884004430 con 0x7f488c15da40 2023-10-16T12:31:13.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:13.807+0000 7f4891367700 1 -- 172.21.15.16:0/3252250158 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f488c0c6ab0 con 0x7f488c15da40 2023-10-16T12:31:13.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:13.807+0000 7f48737fe700 1 -- 172.21.15.16:0/3252250158 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48840047e0 con 0x7f488c15da40 2023-10-16T12:31:13.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:13.807+0000 7f4891367700 1 -- 172.21.15.16:0/3252250158 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f488c0b46e0 con 0x7f488c15da40 2023-10-16T12:31:13.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:13.808+0000 7f4891367700 1 -- 172.21.15.16:0/3252250158 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f488c04ebb0 con 0x7f488c15da40 2023-10-16T12:31:13.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:13.808+0000 7f48737fe700 1 -- 172.21.15.16:0/3252250158 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 85) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f488401c440 con 0x7f488c15da40 2023-10-16T12:31:13.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:13.809+0000 7f48737fe700 1 --2- 172.21.15.16:0/3252250158 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f487806b020 0x7f487806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:13.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:13.809+0000 7f48737fe700 1 -- 172.21.15.16:0/3252250158 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 6085+0+0 (secure 0 0 0) 0x7f488408b370 con 0x7f488c15da40 2023-10-16T12:31:13.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:13.809+0000 7f488affd700 1 --2- 172.21.15.16:0/3252250158 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f487806b020 0x7f487806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:13.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:13.809+0000 7f488affd700 1 --2- 172.21.15.16:0/3252250158 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f487806b020 0x7f487806d4e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f487c00a9b0 tx=0x7f487c008040 comp rx=0 tx=0).ready entity=mgr.8973 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:13.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:13.813+0000 7f48737fe700 1 -- 172.21.15.16:0/3252250158 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4884056110 con 0x7f488c15da40 2023-10-16T12:31:14.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.045+0000 7f4891367700 1 -- 172.21.15.16:0/3252250158 --> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f488c0cbe10 con 0x7f487806b020 2023-10-16T12:31:14.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.046+0000 7f48737fe700 1 -- 172.21.15.16:0/3252250158 <== mgr.8973 v2:172.21.15.16:6832/4182163087 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f487c00ae20 con 0x7f487806b020 2023-10-16T12:31:14.048 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2023-10-16T12:31:14.049 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T12:31:14.049 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cephfs" 2023-10-16T12:31:14.049 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T12:31:14.049 INFO:teuthology.orchestra.run.smithi016.stdout:] 2023-10-16T12:31:14.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.048+0000 7f4891367700 1 -- 172.21.15.16:0/3252250158 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f487806b020 msgr2=0x7f487806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:14.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.048+0000 7f4891367700 1 --2- 172.21.15.16:0/3252250158 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f487806b020 0x7f487806d4e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f487c00a9b0 tx=0x7f487c008040 comp rx=0 tx=0).stop 2023-10-16T12:31:14.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.048+0000 7f4891367700 1 -- 172.21.15.16:0/3252250158 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f488c15da40 msgr2=0x7f488c0b41a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:14.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.048+0000 7f4891367700 1 --2- 172.21.15.16:0/3252250158 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f488c15da40 0x7f488c0b41a0 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7f488c0d2530 tx=0x7f4884004ad0 comp rx=0 tx=0).stop 2023-10-16T12:31:14.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.048+0000 7f4891367700 1 -- 172.21.15.16:0/3252250158 shutdown_connections 2023-10-16T12:31:14.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.048+0000 7f4891367700 1 --2- 172.21.15.16:0/3252250158 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f487806b020 0x7f487806d4e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:14.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.048+0000 7f4891367700 1 --2- 172.21.15.16:0/3252250158 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f488c15da40 0x7f488c0b41a0 unknown :-1 s=CLOSED pgs=1767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:14.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.048+0000 7f4891367700 1 --2- 172.21.15.16:0/3252250158 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f488c06c210 0x7f488c0c5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:14.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.048+0000 7f4891367700 1 --2- 172.21.15.16:0/3252250158 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f488c0b34c0 0x7f488c14fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:14.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.048+0000 7f4891367700 1 -- 172.21.15.16:0/3252250158 >> 172.21.15.16:0/3252250158 conn(0x7f488c0cdf80 msgr2=0x7f488c0ccd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:31:14.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.048+0000 7f4891367700 1 -- 172.21.15.16:0/3252250158 shutdown_connections 2023-10-16T12:31:14.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.049+0000 7f4891367700 1 -- 172.21.15.16:0/3252250158 wait complete. 2023-10-16T12:31:14.062 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-16T12:31:14.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.188+0000 7ff437553700 1 Processor -- start 2023-10-16T12:31:14.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.188+0000 7ff437553700 1 -- start start 2023-10-16T12:31:14.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.189+0000 7ff437553700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff4300ce6a0 0x7ff43006c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:14.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.189+0000 7ff437553700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7ff4300d2110 0x7ff43006c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:14.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.189+0000 7ff437553700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff4300b2620 0x7ff4300b0f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:14.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.189+0000 7ff437553700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4300b20f0 con 0x7ff4300b2620 2023-10-16T12:31:14.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.189+0000 7ff437553700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4300d03d0 con 0x7ff4300d2110 2023-10-16T12:31:14.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.189+0000 7ff437553700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4300b2340 con 0x7ff4300ce6a0 2023-10-16T12:31:14.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.189+0000 7ff435af0700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff4300b2620 0x7ff4300b0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:14.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.189+0000 7ff435af0700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff4300b2620 0x7ff4300b0f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47072/0 (socket says 172.21.15.16:47072) 2023-10-16T12:31:14.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.189+0000 7ff435af0700 1 -- 172.21.15.16:0/2971444355 learned_addr learned my addr 172.21.15.16:0/2971444355 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:31:14.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.189+0000 7ff4352ef700 1 --2- 172.21.15.16:0/2971444355 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff4300ce6a0 0x7ff43006c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:14.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.189+0000 7ff434aee700 1 --2- 172.21.15.16:0/2971444355 >> v2:172.21.15.49:3300/0 conn(0x7ff4300d2110 0x7ff43006c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:14.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.189+0000 7ff435af0700 1 -- 172.21.15.16:0/2971444355 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff4300ce6a0 msgr2=0x7ff43006c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:31:14.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.189+0000 7ff435af0700 1 --2- 172.21.15.16:0/2971444355 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff4300ce6a0 0x7ff43006c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:14.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.189+0000 7ff435af0700 1 -- 172.21.15.16:0/2971444355 >> v2:172.21.15.49:3300/0 conn(0x7ff4300d2110 msgr2=0x7ff43006c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:31:14.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.189+0000 7ff435af0700 1 --2- 172.21.15.16:0/2971444355 >> v2:172.21.15.49:3300/0 conn(0x7ff4300d2110 0x7ff43006c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:14.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.189+0000 7ff435af0700 1 -- 172.21.15.16:0/2971444355 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff43006bb00 con 0x7ff4300b2620 2023-10-16T12:31:14.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.189+0000 7ff435af0700 1 --2- 172.21.15.16:0/2971444355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4300b2620 0x7ff4300b0f50 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7ff430067c50 tx=0x7ff42800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f7651baa407b993 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:14.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.190+0000 7ff427fff700 1 -- 172.21.15.16:0/2971444355 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff42800e070 con 0x7ff4300b2620 2023-10-16T12:31:14.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.190+0000 7ff427fff700 1 -- 172.21.15.16:0/2971444355 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff42800b9c0 con 0x7ff4300b2620 2023-10-16T12:31:14.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.190+0000 7ff427fff700 1 -- 172.21.15.16:0/2971444355 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff42800bd70 con 0x7ff4300b2620 2023-10-16T12:31:14.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.190+0000 7ff437553700 1 -- 172.21.15.16:0/2971444355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4300b2620 msgr2=0x7ff4300b0f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:14.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.190+0000 7ff437553700 1 --2- 172.21.15.16:0/2971444355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4300b2620 0x7ff4300b0f50 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7ff430067c50 tx=0x7ff42800b330 comp rx=0 tx=0).stop 2023-10-16T12:31:14.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.190+0000 7ff437553700 1 -- 172.21.15.16:0/2971444355 shutdown_connections 2023-10-16T12:31:14.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.190+0000 7ff437553700 1 --2- 172.21.15.16:0/2971444355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4300b2620 0x7ff4300b0f50 unknown :-1 s=CLOSED pgs=1768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:14.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.190+0000 7ff437553700 1 --2- 172.21.15.16:0/2971444355 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff4300ce6a0 0x7ff43006c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:14.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.190+0000 7ff437553700 1 --2- 172.21.15.16:0/2971444355 >> v2:172.21.15.49:3300/0 conn(0x7ff4300d2110 0x7ff43006c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:14.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.190+0000 7ff437553700 1 -- 172.21.15.16:0/2971444355 >> 172.21.15.16:0/2971444355 conn(0x7ff4300d15f0 msgr2=0x7ff4300d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:31:14.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.190+0000 7ff437553700 1 -- 172.21.15.16:0/2971444355 shutdown_connections 2023-10-16T12:31:14.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.190+0000 7ff437553700 1 -- 172.21.15.16:0/2971444355 wait complete. 2023-10-16T12:31:14.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.191+0000 7ff437553700 1 Processor -- start 2023-10-16T12:31:14.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.191+0000 7ff437553700 1 -- start start 2023-10-16T12:31:14.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.192+0000 7ff437553700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff4300b2620 0x7ff4301f39e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:14.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.192+0000 7ff437553700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff4300ce6a0 0x7ff43015adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:14.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.192+0000 7ff437553700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4300d2110 0x7ff43015d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:14.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.192+0000 7ff437553700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff43015d170 con 0x7ff4300d2110 2023-10-16T12:31:14.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.192+0000 7ff437553700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff43015bc50 con 0x7ff4300b2620 2023-10-16T12:31:14.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.192+0000 7ff437553700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4301f3f20 con 0x7ff4300ce6a0 2023-10-16T12:31:14.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.192+0000 7ff435af0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4300d2110 0x7ff43015d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:14.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.192+0000 7ff435af0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4300d2110 0x7ff43015d940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47078/0 (socket says 172.21.15.16:47078) 2023-10-16T12:31:14.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.192+0000 7ff435af0700 1 -- 172.21.15.16:0/3691826919 learned_addr learned my addr 172.21.15.16:0/3691826919 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:31:14.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.193+0000 7ff4352ef700 1 --2- 172.21.15.16:0/3691826919 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff4300b2620 0x7ff4301f39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:14.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.193+0000 7ff434aee700 1 --2- 172.21.15.16:0/3691826919 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff4300ce6a0 0x7ff43015adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:14.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.193+0000 7ff435af0700 1 -- 172.21.15.16:0/3691826919 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff4300ce6a0 msgr2=0x7ff43015adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:14.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.193+0000 7ff435af0700 1 --2- 172.21.15.16:0/3691826919 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff4300ce6a0 0x7ff43015adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:14.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.193+0000 7ff435af0700 1 -- 172.21.15.16:0/3691826919 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff4300b2620 msgr2=0x7ff4301f39e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:14.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.193+0000 7ff435af0700 1 --2- 172.21.15.16:0/3691826919 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff4300b2620 0x7ff4301f39e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:14.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.193+0000 7ff435af0700 1 -- 172.21.15.16:0/3691826919 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff428009b30 con 0x7ff4300d2110 2023-10-16T12:31:14.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.193+0000 7ff4352ef700 1 --2- 172.21.15.16:0/3691826919 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff4300b2620 0x7ff4301f39e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:31:14.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.193+0000 7ff434aee700 1 --2- 172.21.15.16:0/3691826919 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff4300ce6a0 0x7ff43015adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:31:14.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.193+0000 7ff435af0700 1 --2- 172.21.15.16:0/3691826919 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4300d2110 0x7ff43015d940 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7ff430067580 tx=0x7ff428004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:14.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.193+0000 7ff425ffb700 1 -- 172.21.15.16:0/3691826919 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff428004250 con 0x7ff4300d2110 2023-10-16T12:31:14.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.193+0000 7ff425ffb700 1 -- 172.21.15.16:0/3691826919 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4280043d0 con 0x7ff4300d2110 2023-10-16T12:31:14.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.193+0000 7ff437553700 1 -- 172.21.15.16:0/3691826919 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff43015b690 con 0x7ff4300d2110 2023-10-16T12:31:14.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.193+0000 7ff425ffb700 1 -- 172.21.15.16:0/3691826919 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff428004780 con 0x7ff4300d2110 2023-10-16T12:31:14.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.193+0000 7ff437553700 1 -- 172.21.15.16:0/3691826919 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff43006ed30 con 0x7ff4300d2110 2023-10-16T12:31:14.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.194+0000 7ff437553700 1 -- 172.21.15.16:0/3691826919 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff43004ebb0 con 0x7ff4300d2110 2023-10-16T12:31:14.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.195+0000 7ff425ffb700 1 -- 172.21.15.16:0/3691826919 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 85) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff42801c440 con 0x7ff4300d2110 2023-10-16T12:31:14.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.195+0000 7ff425ffb700 1 --2- 172.21.15.16:0/3691826919 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7ff41406b020 0x7ff41406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:14.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.195+0000 7ff425ffb700 1 -- 172.21.15.16:0/3691826919 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 6085+0+0 (secure 0 0 0) 0x7ff42808af30 con 0x7ff4300d2110 2023-10-16T12:31:14.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.195+0000 7ff4352ef700 1 --2- 172.21.15.16:0/3691826919 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7ff41406b020 0x7ff41406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:14.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.195+0000 7ff4352ef700 1 --2- 172.21.15.16:0/3691826919 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7ff41406b020 0x7ff41406d4e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff42c0099e0 tx=0x7ff42c008040 comp rx=0 tx=0).ready entity=mgr.8973 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:14.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.199+0000 7ff425ffb700 1 -- 172.21.15.16:0/3691826919 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff428055cd0 con 0x7ff4300d2110 2023-10-16T12:31:14.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:14.470+0000 7ff437553700 1 -- 172.21.15.16:0/3691826919 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7ff43006c0e0 con 0x7ff4300d2110 2023-10-16T12:31:15.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.347+0000 7ff425ffb700 1 -- 172.21.15.16:0/3691826919 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v86) v1 ==== 92+0+0 (secure 0 0 0) 0x7ff4280200a0 con 0x7ff4300d2110 2023-10-16T12:31:15.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.350+0000 7ff437553700 1 -- 172.21.15.16:0/3691826919 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7ff41406b020 msgr2=0x7ff41406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:15.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.350+0000 7ff437553700 1 --2- 172.21.15.16:0/3691826919 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7ff41406b020 0x7ff41406d4e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff42c0099e0 tx=0x7ff42c008040 comp rx=0 tx=0).stop 2023-10-16T12:31:15.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.350+0000 7ff437553700 1 -- 172.21.15.16:0/3691826919 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4300d2110 msgr2=0x7ff43015d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:15.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.350+0000 7ff437553700 1 --2- 172.21.15.16:0/3691826919 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4300d2110 0x7ff43015d940 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7ff430067580 tx=0x7ff428004ad0 comp rx=0 tx=0).stop 2023-10-16T12:31:15.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.351+0000 7ff437553700 1 -- 172.21.15.16:0/3691826919 shutdown_connections 2023-10-16T12:31:15.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.351+0000 7ff437553700 1 --2- 172.21.15.16:0/3691826919 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7ff41406b020 0x7ff41406d4e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:15.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.351+0000 7ff437553700 1 --2- 172.21.15.16:0/3691826919 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4300d2110 0x7ff43015d940 unknown :-1 s=CLOSED pgs=1769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:15.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.351+0000 7ff437553700 1 --2- 172.21.15.16:0/3691826919 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff4300b2620 0x7ff4301f39e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:15.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.351+0000 7ff437553700 1 --2- 172.21.15.16:0/3691826919 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff4300ce6a0 0x7ff43015adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:15.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.351+0000 7ff437553700 1 -- 172.21.15.16:0/3691826919 >> 172.21.15.16:0/3691826919 conn(0x7ff4300d15f0 msgr2=0x7ff4300b0f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:31:15.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.352+0000 7ff437553700 1 -- 172.21.15.16:0/3691826919 shutdown_connections 2023-10-16T12:31:15.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.352+0000 7ff437553700 1 -- 172.21.15.16:0/3691826919 wait complete. 2023-10-16T12:31:15.371 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-16T12:31:15.501 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:31:15.499+0000 7feb31c30200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T12:31:15.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.508+0000 7f72bbcb2700 1 Processor -- start 2023-10-16T12:31:15.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.509+0000 7f72bbcb2700 1 -- start start 2023-10-16T12:31:15.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.509+0000 7f72bbcb2700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f72b40ce5d0 0x7f72b40cc190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:15.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.509+0000 7f72bbcb2700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72b40b2440 con 0x7f72b415f060 2023-10-16T12:31:15.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.509+0000 7f72bbcb2700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72b40d00e0 con 0x7f72b415ec80 2023-10-16T12:31:15.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.509+0000 7f72bbcb2700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72b40b2c50 con 0x7f72b40ce5d0 2023-10-16T12:31:15.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.510+0000 7f72ba24f700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f72b415f060 0x7f72b40cc6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36386/0 (socket says 172.21.15.16:36386) 2023-10-16T12:31:15.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.510+0000 7f72ba24f700 1 -- 172.21.15.16:0/2218280037 learned_addr learned my addr 172.21.15.16:0/2218280037 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:31:15.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.510+0000 7f72b924d700 1 --2- 172.21.15.16:0/2218280037 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f72b40ce5d0 0x7f72b40cc190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:15.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.510+0000 7f72b8a4c700 1 -- 172.21.15.16:0/2218280037 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 21323705 0 0) 0x7f72b40b2440 con 0x7f72b415f060 2023-10-16T12:31:15.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.510+0000 7f72b8a4c700 1 -- 172.21.15.16:0/2218280037 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72a00035b0 con 0x7f72b415f060 2023-10-16T12:31:15.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.510+0000 7f72b8a4c700 1 -- 172.21.15.16:0/2218280037 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4166584715 0 0) 0x7f72b40d00e0 con 0x7f72b415ec80 2023-10-16T12:31:15.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.510+0000 7f72b8a4c700 1 -- 172.21.15.16:0/2218280037 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72b40b2440 con 0x7f72b415ec80 2023-10-16T12:31:15.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.510+0000 7f72b924d700 1 -- 172.21.15.16:0/2218280037 >> v1:172.21.15.49:6789/0 conn(0x7f72b415ec80 legacy=0x7f72b40cdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:15.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.510+0000 7f72b924d700 1 -- 172.21.15.16:0/2218280037 >> v1:172.21.15.16:6789/0 conn(0x7f72b415f060 legacy=0x7f72b40cc6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:15.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.510+0000 7f72b924d700 1 -- 172.21.15.16:0/2218280037 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72b40cef60 con 0x7f72b40ce5d0 2023-10-16T12:31:15.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.510+0000 7f72b924d700 1 --2- 172.21.15.16:0/2218280037 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f72b40ce5d0 0x7f72b40cc190 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f72a4009870 tx=0x7f72a4009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=13783f51a92957ae server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:15.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.510+0000 7f72b8a4c700 1 -- 172.21.15.16:0/2218280037 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72a4011830 con 0x7f72b40ce5d0 2023-10-16T12:31:15.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.511+0000 7f72b8a4c700 1 -- 172.21.15.16:0/2218280037 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72a4011990 con 0x7f72b40ce5d0 2023-10-16T12:31:15.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.511+0000 7f72b8a4c700 1 -- 172.21.15.16:0/2218280037 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72a4011d20 con 0x7f72b40ce5d0 2023-10-16T12:31:15.517 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:31:15.511+0000 7f8ae6a23200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T12:31:15.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.511+0000 7f72bbcb2700 1 -- 172.21.15.16:0/2218280037 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f72b40ce5d0 msgr2=0x7f72b40cc190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:15.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.511+0000 7f72bbcb2700 1 --2- 172.21.15.16:0/2218280037 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f72b40ce5d0 0x7f72b40cc190 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f72a4009870 tx=0x7f72a4009b80 comp rx=0 tx=0).stop 2023-10-16T12:31:15.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.511+0000 7f72bbcb2700 1 -- 172.21.15.16:0/2218280037 shutdown_connections 2023-10-16T12:31:15.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.511+0000 7f72bbcb2700 1 --2- 172.21.15.16:0/2218280037 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f72b40ce5d0 0x7f72b40cc190 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:15.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.511+0000 7f72bbcb2700 1 -- 172.21.15.16:0/2218280037 >> 172.21.15.16:0/2218280037 conn(0x7f72b40d15f0 msgr2=0x7f72b40d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:31:15.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.511+0000 7f72bbcb2700 1 -- 172.21.15.16:0/2218280037 shutdown_connections 2023-10-16T12:31:15.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.511+0000 7f72bbcb2700 1 -- 172.21.15.16:0/2218280037 wait complete. 2023-10-16T12:31:15.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.512+0000 7f72bbcb2700 1 Processor -- start 2023-10-16T12:31:15.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.513+0000 7f72bbcb2700 1 -- start start 2023-10-16T12:31:15.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.513+0000 7f72bbcb2700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72b40ce5d0 0x7f72b4152a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:15.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.513+0000 7f72bbcb2700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f72b415ec80 0x7f72b40d19d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:15.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.513+0000 7f72bbcb2700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f72b415f060 0x7f72b40d1f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:15.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.513+0000 7f72bbcb2700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72b40c5100 con 0x7f72b40ce5d0 2023-10-16T12:31:15.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.513+0000 7f72bbcb2700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72b40c5350 con 0x7f72b415ec80 2023-10-16T12:31:15.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.513+0000 7f72bbcb2700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72b40c5630 con 0x7f72b415f060 2023-10-16T12:31:15.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.513+0000 7f72b9a4e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72b40ce5d0 0x7f72b4152a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:15.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.513+0000 7f72b9a4e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72b40ce5d0 0x7f72b4152a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47114/0 (socket says 172.21.15.16:47114) 2023-10-16T12:31:15.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.513+0000 7f72b9a4e700 1 -- 172.21.15.16:0/1699480230 learned_addr learned my addr 172.21.15.16:0/1699480230 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:31:15.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.514+0000 7f72b924d700 1 --2- 172.21.15.16:0/1699480230 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f72b415ec80 0x7f72b40d19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:15.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.514+0000 7f72ba24f700 1 --2- 172.21.15.16:0/1699480230 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f72b415f060 0x7f72b40d1f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:15.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.514+0000 7f72b9a4e700 1 -- 172.21.15.16:0/1699480230 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f72b415f060 msgr2=0x7f72b40d1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:15.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.514+0000 7f72b9a4e700 1 --2- 172.21.15.16:0/1699480230 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f72b415f060 0x7f72b40d1f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:15.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.514+0000 7f72b9a4e700 1 -- 172.21.15.16:0/1699480230 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f72b415ec80 msgr2=0x7f72b40d19d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:15.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.514+0000 7f72b9a4e700 1 --2- 172.21.15.16:0/1699480230 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f72b415ec80 0x7f72b40d19d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:15.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.514+0000 7f72b9a4e700 1 -- 172.21.15.16:0/1699480230 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72a40094a0 con 0x7f72b40ce5d0 2023-10-16T12:31:15.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.514+0000 7f72b924d700 1 --2- 172.21.15.16:0/1699480230 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f72b415ec80 0x7f72b40d19d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T12:31:15.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.514+0000 7f72b9a4e700 1 --2- 172.21.15.16:0/1699480230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72b40ce5d0 0x7f72b4152a80 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7f72b000c520 tx=0x7f72b000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:15.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.514+0000 7f72aa7fc700 1 -- 172.21.15.16:0/1699480230 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72b00052d0 con 0x7f72b40ce5d0 2023-10-16T12:31:15.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.514+0000 7f72bbcb2700 1 -- 172.21.15.16:0/1699480230 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72b40c5910 con 0x7f72b40ce5d0 2023-10-16T12:31:15.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.514+0000 7f72aa7fc700 1 -- 172.21.15.16:0/1699480230 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72b0005430 con 0x7f72b40ce5d0 2023-10-16T12:31:15.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.514+0000 7f72aa7fc700 1 -- 172.21.15.16:0/1699480230 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72b00038c0 con 0x7f72b40ce5d0 2023-10-16T12:31:15.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.514+0000 7f72bbcb2700 1 -- 172.21.15.16:0/1699480230 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72b4152fc0 con 0x7f72b40ce5d0 2023-10-16T12:31:15.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.515+0000 7f72bbcb2700 1 -- 172.21.15.16:0/1699480230 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72b404ebb0 con 0x7f72b40ce5d0 2023-10-16T12:31:15.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.516+0000 7f72aa7fc700 1 -- 172.21.15.16:0/1699480230 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 86) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f72b0003a20 con 0x7f72b40ce5d0 2023-10-16T12:31:15.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.516+0000 7f72aa7fc700 1 --2- 172.21.15.16:0/1699480230 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f729806b070 0x7f729806d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:15.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.516+0000 7f72b924d700 1 -- 172.21.15.16:0/1699480230 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f729806b070 msgr2=0x7f729806d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/4182163087 2023-10-16T12:31:15.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.516+0000 7f72aa7fc700 1 -- 172.21.15.16:0/1699480230 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 6085+0+0 (secure 0 0 0) 0x7f72b0088d70 con 0x7f72b40ce5d0 2023-10-16T12:31:15.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.516+0000 7f72b924d700 1 --2- 172.21.15.16:0/1699480230 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f729806b070 0x7f729806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:31:15.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.525+0000 7f72aa7fc700 1 -- 172.21.15.16:0/1699480230 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f72b0053ba0 con 0x7f72b40ce5d0 2023-10-16T12:31:15.570 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:31:15.568+0000 7feb31c30200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T12:31:15.588 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:31:15.586+0000 7f8ae6a23200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T12:31:15.645 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:31:15.643+0000 7feb31c30200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T12:31:15.664 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:31:15.662+0000 7f8ae6a23200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T12:31:15.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.717+0000 7f72b924d700 1 -- 172.21.15.16:0/1699480230 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f729806b070 msgr2=0x7f729806d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/4182163087 2023-10-16T12:31:15.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.717+0000 7f72b924d700 1 --2- 172.21.15.16:0/1699480230 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f729806b070 0x7f729806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:31:15.726 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:31:15.724+0000 7feb31c30200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T12:31:15.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.729+0000 7f72bbcb2700 1 -- 172.21.15.16:0/1699480230 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f72b415d450 con 0x7f72b40ce5d0 2023-10-16T12:31:15.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.732+0000 7f72aa7fc700 1 -- 172.21.15.16:0/1699480230 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v53)=0 v53) v1 ==== 138+0+0 (secure 0 0 0) 0x7f72b00571c0 con 0x7f72b40ce5d0 2023-10-16T12:31:15.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.734+0000 7f72bbcb2700 1 -- 172.21.15.16:0/1699480230 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f729806b070 msgr2=0x7f729806d530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:31:15.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.734+0000 7f72bbcb2700 1 --2- 172.21.15.16:0/1699480230 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f729806b070 0x7f729806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:15.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.734+0000 7f72bbcb2700 1 -- 172.21.15.16:0/1699480230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72b40ce5d0 msgr2=0x7f72b4152a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:15.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.734+0000 7f72bbcb2700 1 --2- 172.21.15.16:0/1699480230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72b40ce5d0 0x7f72b4152a80 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7f72b000c520 tx=0x7f72b000c830 comp rx=0 tx=0).stop 2023-10-16T12:31:15.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.734+0000 7f72bbcb2700 1 -- 172.21.15.16:0/1699480230 shutdown_connections 2023-10-16T12:31:15.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.734+0000 7f72bbcb2700 1 --2- 172.21.15.16:0/1699480230 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f729806b070 0x7f729806d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:15.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.734+0000 7f72bbcb2700 1 --2- 172.21.15.16:0/1699480230 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f72b415f060 0x7f72b40d1f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:15.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.734+0000 7f72bbcb2700 1 --2- 172.21.15.16:0/1699480230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72b40ce5d0 0x7f72b4152a80 unknown :-1 s=CLOSED pgs=1771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:15.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.734+0000 7f72bbcb2700 1 --2- 172.21.15.16:0/1699480230 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f72b415ec80 0x7f72b40d19d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:15.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.734+0000 7f72bbcb2700 1 -- 172.21.15.16:0/1699480230 >> 172.21.15.16:0/1699480230 conn(0x7f72b40d15f0 msgr2=0x7f72b40b68c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:31:15.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.735+0000 7f72bbcb2700 1 -- 172.21.15.16:0/1699480230 shutdown_connections 2023-10-16T12:31:15.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.735+0000 7f72bbcb2700 1 -- 172.21.15.16:0/1699480230 wait complete. 2023-10-16T12:31:15.747 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:31:15.745+0000 7f8ae6a23200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T12:31:15.749 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-16T12:31:15.874 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:31:15.872+0000 7feb31c30200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T12:31:15.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.878+0000 7fccba283700 1 Processor -- start 2023-10-16T12:31:15.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.878+0000 7fccba283700 1 -- start start 2023-10-16T12:31:15.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.879+0000 7fccba283700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fccb4072b30 0x7fccb4071a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:15.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.879+0000 7fccba283700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fccb415bb40 0x7fccb406e310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:15.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.879+0000 7fccba283700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccb415c040 con 0x7fccb40caa40 2023-10-16T12:31:15.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.879+0000 7fccba283700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccb415c320 con 0x7fccb415bb40 2023-10-16T12:31:15.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.879+0000 7fccba283700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccb415c600 con 0x7fccb4072b30 2023-10-16T12:31:15.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.879+0000 7fccb37fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fccb415bb40 0x7fccb406e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:15.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.879+0000 7fccb8820700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fccb40caa40 0x7fccb40cae50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36428/0 (socket says 172.21.15.16:36428) 2023-10-16T12:31:15.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.879+0000 7fccb37fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fccb415bb40 0x7fccb406e310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47150/0 (socket says 172.21.15.16:47150) 2023-10-16T12:31:15.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.879+0000 7fccb8820700 1 -- 172.21.15.16:0/3150622328 learned_addr learned my addr 172.21.15.16:0/3150622328 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:31:15.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.879+0000 7fccb3fff700 1 --2- 172.21.15.16:0/3150622328 >> v2:172.21.15.49:3300/0 conn(0x7fccb4072b30 0x7fccb4071a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:15.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.879+0000 7fccb37fe700 1 -- 172.21.15.16:0/3150622328 >> v2:172.21.15.49:3300/0 conn(0x7fccb4072b30 msgr2=0x7fccb4071a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:31:15.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.879+0000 7fccb37fe700 1 --2- 172.21.15.16:0/3150622328 >> v2:172.21.15.49:3300/0 conn(0x7fccb4072b30 0x7fccb4071a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:15.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.879+0000 7fccb37fe700 1 -- 172.21.15.16:0/3150622328 >> v1:172.21.15.16:6789/0 conn(0x7fccb40caa40 legacy=0x7fccb40cae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:15.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.879+0000 7fccb2ffd700 1 -- 172.21.15.16:0/3150622328 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3693605329 0 0) 0x7fccb415c040 con 0x7fccb40caa40 2023-10-16T12:31:15.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.879+0000 7fccb37fe700 1 -- 172.21.15.16:0/3150622328 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccb415c8e0 con 0x7fccb415bb40 2023-10-16T12:31:15.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.879+0000 7fccb37fe700 1 --2- 172.21.15.16:0/3150622328 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccb415bb40 0x7fccb406e310 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7fccb406b780 tx=0x7fcca8009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3bc19e435a335b3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:15.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.879+0000 7fccb2ffd700 1 -- 172.21.15.16:0/3150622328 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca8011830 con 0x7fccb415bb40 2023-10-16T12:31:15.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.879+0000 7fccb2ffd700 1 -- 172.21.15.16:0/3150622328 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcca8011990 con 0x7fccb415bb40 2023-10-16T12:31:15.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.880+0000 7fccb2ffd700 1 -- 172.21.15.16:0/3150622328 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca8011d20 con 0x7fccb415bb40 2023-10-16T12:31:15.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.880+0000 7fccba283700 1 -- 172.21.15.16:0/3150622328 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccb415bb40 msgr2=0x7fccb406e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:15.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.880+0000 7fccba283700 1 --2- 172.21.15.16:0/3150622328 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccb415bb40 0x7fccb406e310 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7fccb406b780 tx=0x7fcca8009b20 comp rx=0 tx=0).stop 2023-10-16T12:31:15.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.880+0000 7fccba283700 1 -- 172.21.15.16:0/3150622328 shutdown_connections 2023-10-16T12:31:15.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.880+0000 7fccba283700 1 --2- 172.21.15.16:0/3150622328 >> v2:172.21.15.49:3300/0 conn(0x7fccb4072b30 0x7fccb4071a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:15.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.880+0000 7fccba283700 1 --2- 172.21.15.16:0/3150622328 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccb415bb40 0x7fccb406e310 unknown :-1 s=CLOSED pgs=1773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:15.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.880+0000 7fccba283700 1 -- 172.21.15.16:0/3150622328 >> 172.21.15.16:0/3150622328 conn(0x7fccb40b1f40 msgr2=0x7fccb40b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:31:15.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.880+0000 7fccba283700 1 -- 172.21.15.16:0/3150622328 shutdown_connections 2023-10-16T12:31:15.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.880+0000 7fccba283700 1 -- 172.21.15.16:0/3150622328 wait complete. 2023-10-16T12:31:15.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.881+0000 7fccba283700 1 Processor -- start 2023-10-16T12:31:15.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.882+0000 7fccba283700 1 -- start start 2023-10-16T12:31:15.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.882+0000 7fccba283700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fccb4072b30 0x7fccb414b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:15.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.882+0000 7fccba283700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fccb40caa40 0x7fccb40c8c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:15.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.882+0000 7fccba283700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccb415bb40 0x7fccb40c5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:15.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.882+0000 7fccba283700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb41f1080 con 0x7fccb415bb40 2023-10-16T12:31:15.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.882+0000 7fccba283700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb40c9160 con 0x7fccb4072b30 2023-10-16T12:31:15.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.882+0000 7fccba283700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccb415cf00 con 0x7fccb40caa40 2023-10-16T12:31:15.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.882+0000 7fccb8820700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccb415bb40 0x7fccb40c5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:15.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.883+0000 7fccb8820700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccb415bb40 0x7fccb40c5480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47160/0 (socket says 172.21.15.16:47160) 2023-10-16T12:31:15.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.883+0000 7fccb8820700 1 -- 172.21.15.16:0/2918249807 learned_addr learned my addr 172.21.15.16:0/2918249807 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:31:15.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.883+0000 7fccb37fe700 1 --2- 172.21.15.16:0/2918249807 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fccb40caa40 0x7fccb40c8c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:15.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.883+0000 7fccb3fff700 1 --2- 172.21.15.16:0/2918249807 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fccb4072b30 0x7fccb414b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:15.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.883+0000 7fccb8820700 1 -- 172.21.15.16:0/2918249807 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fccb40caa40 msgr2=0x7fccb40c8c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:15.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.883+0000 7fccb8820700 1 --2- 172.21.15.16:0/2918249807 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fccb40caa40 0x7fccb40c8c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:15.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.883+0000 7fccb8820700 1 -- 172.21.15.16:0/2918249807 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fccb4072b30 msgr2=0x7fccb414b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:15.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.883+0000 7fccb8820700 1 --2- 172.21.15.16:0/2918249807 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fccb4072b30 0x7fccb414b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:15.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.883+0000 7fccb8820700 1 -- 172.21.15.16:0/2918249807 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcca80094a0 con 0x7fccb415bb40 2023-10-16T12:31:15.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.883+0000 7fccb8820700 1 --2- 172.21.15.16:0/2918249807 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccb415bb40 0x7fccb40c5480 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7fccac002230 tx=0x7fccac00cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:15.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.883+0000 7fccb0ff9700 1 -- 172.21.15.16:0/2918249807 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccac00aac0 con 0x7fccb415bb40 2023-10-16T12:31:15.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.883+0000 7fccb0ff9700 1 -- 172.21.15.16:0/2918249807 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccac00ac40 con 0x7fccb415bb40 2023-10-16T12:31:15.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.883+0000 7fccba283700 1 -- 172.21.15.16:0/2918249807 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccb40c9840 con 0x7fccb415bb40 2023-10-16T12:31:15.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.883+0000 7fccb0ff9700 1 -- 172.21.15.16:0/2918249807 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccac0070d0 con 0x7fccb415bb40 2023-10-16T12:31:15.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.883+0000 7fccba283700 1 -- 172.21.15.16:0/2918249807 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccb40c9e00 con 0x7fccb415bb40 2023-10-16T12:31:15.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.884+0000 7fccba283700 1 -- 172.21.15.16:0/2918249807 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccb41f11f0 con 0x7fccb415bb40 2023-10-16T12:31:15.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.885+0000 7fccb0ff9700 1 -- 172.21.15.16:0/2918249807 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 86) v1 ==== 89348+0+0 (secure 0 0 0) 0x7fccac007250 con 0x7fccb415bb40 2023-10-16T12:31:15.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.885+0000 7fccb0ff9700 1 --2- 172.21.15.16:0/2918249807 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7fcca006b070 0x7fcca006d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:15.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.885+0000 7fccb3fff700 1 -- 172.21.15.16:0/2918249807 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7fcca006b070 msgr2=0x7fcca006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/4182163087 2023-10-16T12:31:15.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.885+0000 7fccb0ff9700 1 -- 172.21.15.16:0/2918249807 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 6085+0+0 (secure 0 0 0) 0x7fccac0893b0 con 0x7fccb415bb40 2023-10-16T12:31:15.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.885+0000 7fccb3fff700 1 --2- 172.21.15.16:0/2918249807 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7fcca006b070 0x7fcca006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:31:15.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:15.890+0000 7fccb0ff9700 1 -- 172.21.15.16:0/2918249807 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fccb41f11f0 con 0x7fccb415bb40 2023-10-16T12:31:15.903 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:31:15.902+0000 7f8ae6a23200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T12:31:16.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.086+0000 7fccb3fff700 1 -- 172.21.15.16:0/2918249807 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7fcca006b070 msgr2=0x7fcca006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/4182163087 2023-10-16T12:31:16.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.086+0000 7fccb3fff700 1 --2- 172.21.15.16:0/2918249807 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7fcca006b070 0x7fcca006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:31:16.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.096+0000 7fccba283700 1 -- 172.21.15.16:0/2918249807 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fccb41f11c0 con 0x7fccb415bb40 2023-10-16T12:31:16.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.099+0000 7fccb0ff9700 1 -- 172.21.15.16:0/2918249807 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v54)=0 v54) v1 ==== 133+0+0 (secure 0 0 0) 0x7fccac054150 con 0x7fccb415bb40 2023-10-16T12:31:16.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.101+0000 7fccba283700 1 -- 172.21.15.16:0/2918249807 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7fcca006b070 msgr2=0x7fcca006d530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:31:16.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.101+0000 7fccba283700 1 --2- 172.21.15.16:0/2918249807 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7fcca006b070 0x7fcca006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:16.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.101+0000 7fccba283700 1 -- 172.21.15.16:0/2918249807 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccb415bb40 msgr2=0x7fccb40c5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:16.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.101+0000 7fccba283700 1 --2- 172.21.15.16:0/2918249807 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccb415bb40 0x7fccb40c5480 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7fccac002230 tx=0x7fccac00cf50 comp rx=0 tx=0).stop 2023-10-16T12:31:16.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.101+0000 7fccba283700 1 -- 172.21.15.16:0/2918249807 shutdown_connections 2023-10-16T12:31:16.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.101+0000 7fccba283700 1 --2- 172.21.15.16:0/2918249807 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7fcca006b070 0x7fcca006d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:16.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.101+0000 7fccba283700 1 --2- 172.21.15.16:0/2918249807 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccb415bb40 0x7fccb40c5480 unknown :-1 s=CLOSED pgs=1774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:16.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.101+0000 7fccba283700 1 --2- 172.21.15.16:0/2918249807 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fccb4072b30 0x7fccb414b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:16.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.101+0000 7fccba283700 1 --2- 172.21.15.16:0/2918249807 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fccb40caa40 0x7fccb40c8c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:16.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.101+0000 7fccba283700 1 -- 172.21.15.16:0/2918249807 >> 172.21.15.16:0/2918249807 conn(0x7fccb40b1f40 msgr2=0x7fccb4159950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:31:16.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.101+0000 7fccba283700 1 -- 172.21.15.16:0/2918249807 shutdown_connections 2023-10-16T12:31:16.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.101+0000 7fccba283700 1 -- 172.21.15.16:0/2918249807 wait complete. 2023-10-16T12:31:16.116 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T12:31:16.117 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-10-16T12:31:16.137 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1 1m 2023-10-16T12:31:16.217 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:31:16.215+0000 7feb31c30200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T12:31:16.257 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:31:16.255+0000 7f8ae6a23200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T12:31:16.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.306+0000 7f16cf93a700 1 Processor -- start 2023-10-16T12:31:16.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.306+0000 7f16cf93a700 1 -- start start 2023-10-16T12:31:16.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.307+0000 7f16cf93a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f16c815da10 0x7f16c80d1520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:16.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.307+0000 7f16cf93a700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f16c80d4060 0x7f16c80cf2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:16.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.307+0000 7f16cf93a700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16c80d4540 con 0x7f16c80d1d20 2023-10-16T12:31:16.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.307+0000 7f16cf93a700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16c80ce7e0 con 0x7f16c815da10 2023-10-16T12:31:16.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.307+0000 7f16cf93a700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16c80ceac0 con 0x7f16c80d4060 2023-10-16T12:31:16.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.307+0000 7f16cd6d6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f16c815da10 0x7f16c80d1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:16.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.307+0000 7f16cded7700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f16c80d1d20 0x7f16c806b860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36478/0 (socket says 172.21.15.16:36478) 2023-10-16T12:31:16.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.307+0000 7f16cd6d6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f16c815da10 0x7f16c80d1520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47200/0 (socket says 172.21.15.16:47200) 2023-10-16T12:31:16.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.307+0000 7f16cded7700 1 -- 172.21.15.16:0/17444550 learned_addr learned my addr 172.21.15.16:0/17444550 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:31:16.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.307+0000 7f16cced5700 1 --2- 172.21.15.16:0/17444550 >> v2:172.21.15.49:3300/0 conn(0x7f16c80d4060 0x7f16c80cf2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:16.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.307+0000 7f16cd6d6700 1 -- 172.21.15.16:0/17444550 >> v2:172.21.15.49:3300/0 conn(0x7f16c80d4060 msgr2=0x7f16c80cf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:31:16.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.307+0000 7f16cd6d6700 1 --2- 172.21.15.16:0/17444550 >> v2:172.21.15.49:3300/0 conn(0x7f16c80d4060 0x7f16c80cf2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:16.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.307+0000 7f16cd6d6700 1 -- 172.21.15.16:0/17444550 >> v1:172.21.15.16:6789/0 conn(0x7f16c80d1d20 legacy=0x7f16c806b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:16.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.307+0000 7f16cd6d6700 1 -- 172.21.15.16:0/17444550 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16c80ceda0 con 0x7f16c815da10 2023-10-16T12:31:16.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.307+0000 7f16bffff700 1 -- 172.21.15.16:0/17444550 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 327299147 0 0) 0x7f16c80d4540 con 0x7f16c80d1d20 2023-10-16T12:31:16.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.308+0000 7f16cd6d6700 1 --2- 172.21.15.16:0/17444550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16c815da10 0x7f16c80d1520 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7f16c80cdf70 tx=0x7f16c4009b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43bfb068acafc3cc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:16.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.308+0000 7f16bffff700 1 -- 172.21.15.16:0/17444550 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16c4005e30 con 0x7f16c815da10 2023-10-16T12:31:16.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.308+0000 7f16bffff700 1 -- 172.21.15.16:0/17444550 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16c4006030 con 0x7f16c815da10 2023-10-16T12:31:16.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.308+0000 7f16bffff700 1 -- 172.21.15.16:0/17444550 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16c40063c0 con 0x7f16c815da10 2023-10-16T12:31:16.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.308+0000 7f16cf93a700 1 -- 172.21.15.16:0/17444550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16c815da10 msgr2=0x7f16c80d1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:16.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.308+0000 7f16cf93a700 1 --2- 172.21.15.16:0/17444550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16c815da10 0x7f16c80d1520 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7f16c80cdf70 tx=0x7f16c4009b00 comp rx=0 tx=0).stop 2023-10-16T12:31:16.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.309+0000 7f16cf93a700 1 -- 172.21.15.16:0/17444550 shutdown_connections 2023-10-16T12:31:16.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.309+0000 7f16cf93a700 1 --2- 172.21.15.16:0/17444550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16c815da10 0x7f16c80d1520 unknown :-1 s=CLOSED pgs=1776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:16.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.309+0000 7f16cf93a700 1 --2- 172.21.15.16:0/17444550 >> v2:172.21.15.49:3300/0 conn(0x7f16c80d4060 0x7f16c80cf2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:16.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.309+0000 7f16cf93a700 1 -- 172.21.15.16:0/17444550 >> 172.21.15.16:0/17444550 conn(0x7f16c809c9f0 msgr2=0x7f16c80d3740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:31:16.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.309+0000 7f16cf93a700 1 -- 172.21.15.16:0/17444550 shutdown_connections 2023-10-16T12:31:16.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.309+0000 7f16cf93a700 1 -- 172.21.15.16:0/17444550 wait complete. 2023-10-16T12:31:16.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.310+0000 7f16cf93a700 1 Processor -- start 2023-10-16T12:31:16.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.310+0000 7f16cf93a700 1 -- start start 2023-10-16T12:31:16.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.311+0000 7f16cf93a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f16c80d1d20 0x7f16c8157e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:16.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.311+0000 7f16cf93a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16c80d4060 0x7f16c815a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:16.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.311+0000 7f16cf93a700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f16c815da10 0x7f16c806bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:16.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.311+0000 7f16cf93a700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16c8159dc0 con 0x7f16c80d4060 2023-10-16T12:31:16.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.311+0000 7f16cf93a700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16c815b890 con 0x7f16c815da10 2023-10-16T12:31:16.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.311+0000 7f16cf93a700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16c8159140 con 0x7f16c80d1d20 2023-10-16T12:31:16.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.311+0000 7f16cced5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16c80d4060 0x7f16c815a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:16.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.311+0000 7f16cced5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16c80d4060 0x7f16c815a1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47208/0 (socket says 172.21.15.16:47208) 2023-10-16T12:31:16.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.311+0000 7f16cced5700 1 -- 172.21.15.16:0/573902437 learned_addr learned my addr 172.21.15.16:0/573902437 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:31:16.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.311+0000 7f16cded7700 1 --2- 172.21.15.16:0/573902437 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f16c815da10 0x7f16c806bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:16.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.311+0000 7f16cd6d6700 1 --2- 172.21.15.16:0/573902437 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f16c80d1d20 0x7f16c8157e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:16.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.311+0000 7f16cced5700 1 -- 172.21.15.16:0/573902437 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f16c80d1d20 msgr2=0x7f16c8157e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:16.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.311+0000 7f16cced5700 1 --2- 172.21.15.16:0/573902437 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f16c80d1d20 0x7f16c8157e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:16.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.311+0000 7f16cced5700 1 -- 172.21.15.16:0/573902437 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f16c815da10 msgr2=0x7f16c806bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:16.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.311+0000 7f16cced5700 1 --2- 172.21.15.16:0/573902437 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f16c815da10 0x7f16c806bef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:16.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.311+0000 7f16cced5700 1 -- 172.21.15.16:0/573902437 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16c4009430 con 0x7f16c80d4060 2023-10-16T12:31:16.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.311+0000 7f16cced5700 1 --2- 172.21.15.16:0/573902437 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16c80d4060 0x7f16c815a1e0 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7f16b800e980 tx=0x7f16b800ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:16.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.312+0000 7f16bdffb700 1 -- 172.21.15.16:0/573902437 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b8013680 con 0x7f16c80d4060 2023-10-16T12:31:16.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.312+0000 7f16bdffb700 1 -- 172.21.15.16:0/573902437 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16b80137e0 con 0x7f16c80d4060 2023-10-16T12:31:16.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.312+0000 7f16cf93a700 1 -- 172.21.15.16:0/573902437 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16c8159420 con 0x7f16c80d4060 2023-10-16T12:31:16.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.312+0000 7f16bdffb700 1 -- 172.21.15.16:0/573902437 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b8013b70 con 0x7f16c80d4060 2023-10-16T12:31:16.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.312+0000 7f16cf93a700 1 -- 172.21.15.16:0/573902437 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16c81599e0 con 0x7f16c80d4060 2023-10-16T12:31:16.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.313+0000 7f16cf93a700 1 -- 172.21.15.16:0/573902437 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16c8159f00 con 0x7f16c80d4060 2023-10-16T12:31:16.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.313+0000 7f16bdffb700 1 -- 172.21.15.16:0/573902437 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 86) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f16b8013cd0 con 0x7f16c80d4060 2023-10-16T12:31:16.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.314+0000 7f16bdffb700 1 --2- 172.21.15.16:0/573902437 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f16ac06b070 0x7f16ac06d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:16.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.314+0000 7f16cd6d6700 1 -- 172.21.15.16:0/573902437 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f16ac06b070 msgr2=0x7f16ac06d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/4182163087 2023-10-16T12:31:16.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.314+0000 7f16bdffb700 1 -- 172.21.15.16:0/573902437 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 6085+0+0 (secure 0 0 0) 0x7f16b80897e0 con 0x7f16c80d4060 2023-10-16T12:31:16.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.314+0000 7f16cd6d6700 1 --2- 172.21.15.16:0/573902437 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f16ac06b070 0x7f16ac06d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T12:31:16.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.318+0000 7f16bdffb700 1 -- 172.21.15.16:0/573902437 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f16c8159f00 con 0x7f16c80d4060 2023-10-16T12:31:16.341 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:31:16.339+0000 7feb31c30200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T12:31:16.383 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:31:16.381+0000 7f8ae6a23200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T12:31:16.419 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:31:16.417+0000 7feb31c30200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T12:31:16.462 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:31:16.461+0000 7f8ae6a23200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T12:31:16.503 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:31:16.501+0000 7feb31c30200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T12:31:16.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.515+0000 7f16cd6d6700 1 -- 172.21.15.16:0/573902437 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f16ac06b070 msgr2=0x7f16ac06d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/4182163087 2023-10-16T12:31:16.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.515+0000 7f16cd6d6700 1 --2- 172.21.15.16:0/573902437 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f16ac06b070 0x7f16ac06d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T12:31:16.549 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:31:16.547+0000 7f8ae6a23200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T12:31:16.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.554+0000 7f16cf93a700 1 -- 172.21.15.16:0/573902437 --> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f16c815ddf0 con 0x7f16ac06b070 2023-10-16T12:31:16.891 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:31:16.889+0000 7feb31c30200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T12:31:16.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.916+0000 7f16cd6d6700 1 -- 172.21.15.16:0/573902437 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f16ac06b070 msgr2=0x7f16ac06d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/4182163087 2023-10-16T12:31:16.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:16.916+0000 7f16cd6d6700 1 --2- 172.21.15.16:0/573902437 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f16ac06b070 0x7f16ac06d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-16T12:31:16.941 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:31:16.939+0000 7f8ae6a23200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T12:31:17.499 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:31:17.497+0000 7feb31c30200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T12:31:17.556 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:31:17.554+0000 7f8ae6a23200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T12:31:17.572 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:31:17.570+0000 7feb31c30200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T12:31:17.630 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:31:17.628+0000 7f8ae6a23200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T12:31:17.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:17.717+0000 7f16cd6d6700 1 -- 172.21.15.16:0/573902437 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f16ac06b070 msgr2=0x7f16ac06d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/4182163087 2023-10-16T12:31:17.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:17.717+0000 7f16cd6d6700 1 --2- 172.21.15.16:0/573902437 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f16ac06b070 0x7f16ac06d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-16T12:31:18.235 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:31:18.233+0000 7feb31c30200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:31:18.303 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:31:18.302+0000 7f8ae6a23200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:31:18.319 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:31:18.314+0000 7feb31c30200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T12:31:18.386 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:31:18.385+0000 7f8ae6a23200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T12:31:18.468 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:31:18.466+0000 7feb31c30200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:31:18.541 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:31:18.538+0000 7f8ae6a23200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T12:31:18.706 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:31:18.704+0000 7feb31c30200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T12:31:18.782 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:31:18.780+0000 7f8ae6a23200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T12:31:18.783 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:31:18.782+0000 7feb31c30200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T12:31:18.861 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:31:18.859+0000 7f8ae6a23200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T12:31:19.060 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:31:19.058+0000 7feb31c30200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T12:31:19.141 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:31:19.139+0000 7f8ae6a23200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T12:31:19.291 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:31:19.289+0000 7feb31c30200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T12:31:19.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:19.318+0000 7f16cd6d6700 1 -- 172.21.15.16:0/573902437 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f16ac06b070 msgr2=0x7f16ac06d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/4182163087 2023-10-16T12:31:19.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:19.318+0000 7f16cd6d6700 1 --2- 172.21.15.16:0/573902437 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f16ac06b070 0x7f16ac06d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-10-16T12:31:19.378 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:31:19.376+0000 7f8ae6a23200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T12:31:19.446 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:31:19.445+0000 7feb31c30200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T12:31:19.538 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:31:19.536+0000 7f8ae6a23200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T12:31:19.566 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:31:19.565+0000 7feb31c30200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T12:31:19.660 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:31:19.658+0000 7f8ae6a23200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T12:31:19.729 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:31:19.728+0000 7feb31c30200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T12:31:19.827 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:31:19.825+0000 7f8ae6a23200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T12:31:20.001 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:31:20.000+0000 7feb31c30200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T12:31:20.102 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:31:20.101+0000 7f8ae6a23200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T12:31:20.169 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:31:20.168+0000 7feb31c30200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T12:31:20.273 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:31:20.271+0000 7f8ae6a23200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T12:31:20.724 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T12:31:20.723+0000 7feb31c30200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T12:31:20.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:20.791+0000 7f16bdffb700 1 -- 172.21.15.16:0/573902437 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 87) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f16b8054a60 con 0x7f16c80d4060 2023-10-16T12:31:20.841 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:31:20.839+0000 7f8ae6a23200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T12:31:20.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:20.859+0000 7f16bdffb700 1 -- 172.21.15.16:0/573902437 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mgrmap(e 88) v1 ==== 88957+0+0 (secure 0 0 0) 0x7f16b8087b50 con 0x7f16c80d4060 2023-10-16T12:31:20.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:20.860+0000 7f16bdffb700 1 -- 172.21.15.16:0/573902437 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f16ac06b070 msgr2=0x7f16ac06d530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T12:31:20.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:20.860+0000 7f16bdffb700 1 --2- 172.21.15.16:0/573902437 >> [v2:172.21.15.16:6832/4182163087,v1:172.21.15.16:6833/4182163087] conn(0x7f16ac06b070 0x7f16ac06d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:21.005 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:31:21.003+0000 7f8a599fe700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:31:21.005 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:31:21.003+0000 7f8a599fe700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:31:21.005 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:31:21.003+0000 7f8a599fe700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:31:21.006 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:31:21.003+0000 7f8a599fe700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:31:21.006 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:31:21.003+0000 7f8a599fe700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:31:21.006 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:31:21.003+0000 7f8a5ca04700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:31:21.006 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:31:21.003+0000 7f8a5ca04700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:31:21.006 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:31:21.003+0000 7f8a5ca04700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:31:21.007 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:31:21.003+0000 7f8a5ca04700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:31:21.007 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:31:21.003+0000 7f8a5ca04700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:31:21.453 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:31:21.456 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:31:21.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:21.863+0000 7f16bdffb700 1 -- 172.21.15.16:0/573902437 <== mon.0 v2:172.21.15.16:3300/0 9 ==== mgrmap(e 89) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f16b8055330 con 0x7f16c80d4060 2023-10-16T12:31:21.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:21.864+0000 7f16bdffb700 1 --2- 172.21.15.16:0/573902437 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f16ac01d250 0x7f16ac075500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:21.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:21.864+0000 7f16bdffb700 1 -- 172.21.15.16:0/573902437 --> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f16ac01d630 con 0x7f16ac01d250 2023-10-16T12:31:21.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:21.865+0000 7f16cd6d6700 1 --2- 172.21.15.16:0/573902437 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f16ac01d250 0x7f16ac075500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:21.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:21.868+0000 7f16cd6d6700 1 --2- 172.21.15.16:0/573902437 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f16ac01d250 0x7f16ac075500 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f16c4009b40 tx=0x7f16c4008070 comp rx=0 tx=0).ready entity=mgr.8770 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:21.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:21.932+0000 7f16bdffb700 1 -- 172.21.15.16:0/573902437 <== mgr.8770 v2:172.21.15.16:6832/1503042987 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f16ac01d630 con 0x7f16ac01d250 2023-10-16T12:31:21.935 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule set for path /snap_test_dir1 2023-10-16T12:31:21.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:21.935+0000 7f16cf93a700 1 -- 172.21.15.16:0/573902437 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f16ac01d250 msgr2=0x7f16ac075500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:21.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:21.935+0000 7f16cf93a700 1 --2- 172.21.15.16:0/573902437 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f16ac01d250 0x7f16ac075500 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f16c4009b40 tx=0x7f16c4008070 comp rx=0 tx=0).stop 2023-10-16T12:31:21.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:21.935+0000 7f16cf93a700 1 -- 172.21.15.16:0/573902437 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16c80d4060 msgr2=0x7f16c815a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:21.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:21.935+0000 7f16cf93a700 1 --2- 172.21.15.16:0/573902437 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16c80d4060 0x7f16c815a1e0 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7f16b800e980 tx=0x7f16b800ec90 comp rx=0 tx=0).stop 2023-10-16T12:31:21.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:21.935+0000 7f16cded7700 1 -- 172.21.15.16:0/573902437 reap_dead start 2023-10-16T12:31:21.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:21.935+0000 7f16cf93a700 1 -- 172.21.15.16:0/573902437 shutdown_connections 2023-10-16T12:31:21.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:21.935+0000 7f16cf93a700 1 -- 172.21.15.16:0/573902437 >> 172.21.15.16:0/573902437 conn(0x7f16c809c9f0 msgr2=0x7f16c80d0150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:31:21.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:21.935+0000 7f16cf93a700 1 -- 172.21.15.16:0/573902437 shutdown_connections 2023-10-16T12:31:21.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:21.935+0000 7f16cf93a700 1 -- 172.21.15.16:0/573902437 wait complete. 2023-10-16T12:31:21.950 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1 2023-10-16T12:31:21.951 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention add /snap_test_dir1 55n 2023-10-16T12:31:22.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.080+0000 7f06b63f9700 1 Processor -- start 2023-10-16T12:31:22.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.080+0000 7f06b63f9700 1 -- start start 2023-10-16T12:31:22.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.081+0000 7f06b63f9700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f06b015b5a0 0x7f06b015f110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:22.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.081+0000 7f06b63f9700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f06b00ce550 0x7f06b00b4300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:22.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.081+0000 7f06b63f9700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06b00cf530 con 0x7f06b00cbb00 2023-10-16T12:31:22.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.081+0000 7f06b63f9700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06b00cf290 con 0x7f06b015b5a0 2023-10-16T12:31:22.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.081+0000 7f06b63f9700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06b015f650 con 0x7f06b00ce550 2023-10-16T12:31:22.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.081+0000 7f06affff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f06b015b5a0 0x7f06b015f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:22.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.081+0000 7f06b4996700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f06b00cbb00 0x7f06b00cf9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36662/0 (socket says 172.21.15.16:36662) 2023-10-16T12:31:22.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.081+0000 7f06b4996700 1 -- 172.21.15.16:0/338714752 learned_addr learned my addr 172.21.15.16:0/338714752 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:31:22.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.081+0000 7f06affff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f06b015b5a0 0x7f06b015f110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47388/0 (socket says 172.21.15.16:47388) 2023-10-16T12:31:22.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.082+0000 7f06af7fe700 1 --2- 172.21.15.16:0/338714752 >> v2:172.21.15.49:3300/0 conn(0x7f06b00ce550 0x7f06b00b4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:22.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.082+0000 7f06affff700 1 -- 172.21.15.16:0/338714752 >> v2:172.21.15.49:3300/0 conn(0x7f06b00ce550 msgr2=0x7f06b00b4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:31:22.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.082+0000 7f06affff700 1 --2- 172.21.15.16:0/338714752 >> v2:172.21.15.49:3300/0 conn(0x7f06b00ce550 0x7f06b00b4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:22.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.082+0000 7f06affff700 1 -- 172.21.15.16:0/338714752 >> v1:172.21.15.16:6789/0 conn(0x7f06b00cbb00 legacy=0x7f06b00cf9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:22.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.082+0000 7f06affff700 1 -- 172.21.15.16:0/338714752 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06b015f8f0 con 0x7f06b015b5a0 2023-10-16T12:31:22.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.082+0000 7f06affff700 1 --2- 172.21.15.16:0/338714752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06b015b5a0 0x7f06b015f110 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f06b00b09f0 tx=0x7f0698009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79d26f6c65da8011 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:22.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.082+0000 7f06aeffd700 1 -- 172.21.15.16:0/338714752 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0698011830 con 0x7f06b015b5a0 2023-10-16T12:31:22.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.082+0000 7f06aeffd700 1 -- 172.21.15.16:0/338714752 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0698011990 con 0x7f06b015b5a0 2023-10-16T12:31:22.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.082+0000 7f06aeffd700 1 -- 172.21.15.16:0/338714752 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0698011d20 con 0x7f06b015b5a0 2023-10-16T12:31:22.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.082+0000 7f06b63f9700 1 -- 172.21.15.16:0/338714752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06b015b5a0 msgr2=0x7f06b015f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:22.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.082+0000 7f06b63f9700 1 --2- 172.21.15.16:0/338714752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06b015b5a0 0x7f06b015f110 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f06b00b09f0 tx=0x7f0698009b20 comp rx=0 tx=0).stop 2023-10-16T12:31:22.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.083+0000 7f06b63f9700 1 -- 172.21.15.16:0/338714752 shutdown_connections 2023-10-16T12:31:22.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.083+0000 7f06b63f9700 1 --2- 172.21.15.16:0/338714752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06b015b5a0 0x7f06b015f110 unknown :-1 s=CLOSED pgs=1790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:22.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.083+0000 7f06b63f9700 1 --2- 172.21.15.16:0/338714752 >> v2:172.21.15.49:3300/0 conn(0x7f06b00ce550 0x7f06b00b4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:22.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.083+0000 7f06b63f9700 1 -- 172.21.15.16:0/338714752 >> 172.21.15.16:0/338714752 conn(0x7f06b006eb40 msgr2=0x7f06b006ef20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:31:22.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.083+0000 7f06b63f9700 1 -- 172.21.15.16:0/338714752 shutdown_connections 2023-10-16T12:31:22.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.083+0000 7f06b63f9700 1 -- 172.21.15.16:0/338714752 wait complete. 2023-10-16T12:31:22.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.084+0000 7f06b63f9700 1 Processor -- start 2023-10-16T12:31:22.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.084+0000 7f06b63f9700 1 -- start start 2023-10-16T12:31:22.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.085+0000 7f06b63f9700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f06b00cbb00 0x7f06b00c98a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:22.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.085+0000 7f06b63f9700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f06b00ce550 0x7f06b00cb110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:22.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.085+0000 7f06b63f9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06b015b5a0 0x7f06b00b6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:22.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.085+0000 7f06b63f9700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06b014bce0 con 0x7f06b015b5a0 2023-10-16T12:31:22.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.085+0000 7f06b63f9700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06b00cb7f0 con 0x7f06b00cbb00 2023-10-16T12:31:22.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.085+0000 7f06b63f9700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06b015cde0 con 0x7f06b00ce550 2023-10-16T12:31:22.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.085+0000 7f06b4996700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06b015b5a0 0x7f06b00b6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:22.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.086+0000 7f06b4996700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06b015b5a0 0x7f06b00b6260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47396/0 (socket says 172.21.15.16:47396) 2023-10-16T12:31:22.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.086+0000 7f06b4996700 1 -- 172.21.15.16:0/345789506 learned_addr learned my addr 172.21.15.16:0/345789506 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:31:22.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.086+0000 7f06affff700 1 --2- 172.21.15.16:0/345789506 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f06b00cbb00 0x7f06b00c98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:22.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.086+0000 7f06af7fe700 1 --2- 172.21.15.16:0/345789506 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f06b00ce550 0x7f06b00cb110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:22.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.086+0000 7f06b4996700 1 -- 172.21.15.16:0/345789506 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f06b00ce550 msgr2=0x7f06b00cb110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:22.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.086+0000 7f06b4996700 1 --2- 172.21.15.16:0/345789506 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f06b00ce550 0x7f06b00cb110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:22.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.086+0000 7f06b4996700 1 -- 172.21.15.16:0/345789506 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f06b00cbb00 msgr2=0x7f06b00c98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:22.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.086+0000 7f06b4996700 1 --2- 172.21.15.16:0/345789506 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f06b00cbb00 0x7f06b00c98a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:22.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.086+0000 7f06b4996700 1 -- 172.21.15.16:0/345789506 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06980094a0 con 0x7f06b015b5a0 2023-10-16T12:31:22.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.086+0000 7f06b4996700 1 --2- 172.21.15.16:0/345789506 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06b015b5a0 0x7f06b00b6260 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f06a800cbc0 tx=0x7f06a800ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:22.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.086+0000 7f06acff9700 1 -- 172.21.15.16:0/345789506 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a800a9e0 con 0x7f06b015b5a0 2023-10-16T12:31:22.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.086+0000 7f06acff9700 1 -- 172.21.15.16:0/345789506 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06a800ab40 con 0x7f06b015b5a0 2023-10-16T12:31:22.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.086+0000 7f06b63f9700 1 -- 172.21.15.16:0/345789506 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06b015d0c0 con 0x7f06b015b5a0 2023-10-16T12:31:22.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.086+0000 7f06acff9700 1 -- 172.21.15.16:0/345789506 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a8012400 con 0x7f06b015b5a0 2023-10-16T12:31:22.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.086+0000 7f06b63f9700 1 -- 172.21.15.16:0/345789506 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06b015d680 con 0x7f06b015b5a0 2023-10-16T12:31:22.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.087+0000 7f06b63f9700 1 -- 172.21.15.16:0/345789506 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06b004ec20 con 0x7f06b015b5a0 2023-10-16T12:31:22.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.088+0000 7f06acff9700 1 -- 172.21.15.16:0/345789506 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 89) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f06a801b3b0 con 0x7f06b015b5a0 2023-10-16T12:31:22.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.088+0000 7f06acff9700 1 --2- 172.21.15.16:0/345789506 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f069006ada0 0x7f069006d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:22.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.088+0000 7f06acff9700 1 -- 172.21.15.16:0/345789506 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 6429+0+0 (secure 0 0 0) 0x7f06a8089040 con 0x7f06b015b5a0 2023-10-16T12:31:22.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.088+0000 7f06affff700 1 --2- 172.21.15.16:0/345789506 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f069006ada0 0x7f069006d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:22.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.089+0000 7f06affff700 1 --2- 172.21.15.16:0/345789506 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f069006ada0 0x7f069006d260 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f06980090b0 tx=0x7f069801c040 comp rx=0 tx=0).ready entity=mgr.8770 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:22.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.093+0000 7f06acff9700 1 -- 172.21.15.16:0/345789506 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f06a8054190 con 0x7f06b015b5a0 2023-10-16T12:31:22.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.324+0000 7f06b63f9700 1 -- 172.21.15.16:0/345789506 --> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "path": "/snap_test_dir1", "retention_spec_or_period": "55n", "target": ["mon-mgr", ""]}) v1 -- 0x7f06b015feb0 con 0x7f069006ada0 2023-10-16T12:31:22.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.380+0000 7f06acff9700 1 -- 172.21.15.16:0/345789506 <== mgr.8770 v2:172.21.15.16:6832/1503042987 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+40 (secure 0 0 0) 0x7f0698011c70 con 0x7f069006ada0 2023-10-16T12:31:22.383 INFO:teuthology.orchestra.run.smithi016.stdout:Retention added to path /snap_test_dir1 2023-10-16T12:31:22.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.383+0000 7f06b63f9700 1 -- 172.21.15.16:0/345789506 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f069006ada0 msgr2=0x7f069006d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:22.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.383+0000 7f06b63f9700 1 --2- 172.21.15.16:0/345789506 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f069006ada0 0x7f069006d260 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f06980090b0 tx=0x7f069801c040 comp rx=0 tx=0).stop 2023-10-16T12:31:22.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.383+0000 7f06b63f9700 1 -- 172.21.15.16:0/345789506 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06b015b5a0 msgr2=0x7f06b00b6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:22.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.383+0000 7f06b63f9700 1 --2- 172.21.15.16:0/345789506 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06b015b5a0 0x7f06b00b6260 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f06a800cbc0 tx=0x7f06a800ced0 comp rx=0 tx=0).stop 2023-10-16T12:31:22.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.383+0000 7f06b63f9700 1 -- 172.21.15.16:0/345789506 shutdown_connections 2023-10-16T12:31:22.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.383+0000 7f06b63f9700 1 --2- 172.21.15.16:0/345789506 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f069006ada0 0x7f069006d260 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:22.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.383+0000 7f06b63f9700 1 --2- 172.21.15.16:0/345789506 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06b015b5a0 0x7f06b00b6260 unknown :-1 s=CLOSED pgs=1791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:22.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.383+0000 7f06b63f9700 1 --2- 172.21.15.16:0/345789506 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f06b00cbb00 0x7f06b00c98a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:22.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.383+0000 7f06b63f9700 1 --2- 172.21.15.16:0/345789506 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f06b00ce550 0x7f06b00cb110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:22.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.383+0000 7f06b63f9700 1 -- 172.21.15.16:0/345789506 >> 172.21.15.16:0/345789506 conn(0x7f06b006eb40 msgr2=0x7f06b00cf9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:31:22.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.383+0000 7f06b63f9700 1 -- 172.21.15.16:0/345789506 shutdown_connections 2023-10-16T12:31:22.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.383+0000 7f06b63f9700 1 -- 172.21.15.16:0/345789506 wait complete. 2023-10-16T12:31:22.397 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention added to path /snap_test_dir1 2023-10-16T12:31:22.397 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m'], expected_retention: [] 2023-10-16T12:31:22.398 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json /snap_test_dir1 2023-10-16T12:31:22.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.523+0000 7f36d7ba0700 1 Processor -- start 2023-10-16T12:31:22.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.523+0000 7f36d7ba0700 1 -- start start 2023-10-16T12:31:22.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.524+0000 7f36d7ba0700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f36d015d8d0 0x7f36d00b2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:22.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.524+0000 7f36d7ba0700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f36d00b8fa0 0x7f36d00b6720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:22.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.524+0000 7f36d7ba0700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36d00b2f30 con 0x7f36d00b6c60 2023-10-16T12:31:22.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.524+0000 7f36d7ba0700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36d00d0bb0 con 0x7f36d015d8d0 2023-10-16T12:31:22.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.524+0000 7f36d7ba0700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36d015dcb0 con 0x7f36d00b8fa0 2023-10-16T12:31:22.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.524+0000 7f36d593c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f36d015d8d0 0x7f36d00b2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:22.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.524+0000 7f36d613d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f36d00b6c60 0x7f36d00ce350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36678/0 (socket says 172.21.15.16:36678) 2023-10-16T12:31:22.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.524+0000 7f36d593c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f36d015d8d0 0x7f36d00b2830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47400/0 (socket says 172.21.15.16:47400) 2023-10-16T12:31:22.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.524+0000 7f36d613d700 1 -- 172.21.15.16:0/1865267954 learned_addr learned my addr 172.21.15.16:0/1865267954 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T12:31:22.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.525+0000 7f36d513b700 1 --2- 172.21.15.16:0/1865267954 >> v2:172.21.15.49:3300/0 conn(0x7f36d00b8fa0 0x7f36d00b6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:22.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.525+0000 7f36d593c700 1 -- 172.21.15.16:0/1865267954 >> v2:172.21.15.49:3300/0 conn(0x7f36d00b8fa0 msgr2=0x7f36d00b6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T12:31:22.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.525+0000 7f36d593c700 1 --2- 172.21.15.16:0/1865267954 >> v2:172.21.15.49:3300/0 conn(0x7f36d00b8fa0 0x7f36d00b6720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:22.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.525+0000 7f36d593c700 1 -- 172.21.15.16:0/1865267954 >> v1:172.21.15.16:6789/0 conn(0x7f36d00b6c60 legacy=0x7f36d00ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:22.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.525+0000 7f36d493a700 1 -- 172.21.15.16:0/1865267954 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 933325329 0 0) 0x7f36d00b2f30 con 0x7f36d00b6c60 2023-10-16T12:31:22.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.525+0000 7f36d593c700 1 -- 172.21.15.16:0/1865267954 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36d00d1c40 con 0x7f36d015d8d0 2023-10-16T12:31:22.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.525+0000 7f36d513b700 1 --2- 172.21.15.16:0/1865267954 >> v2:172.21.15.49:3300/0 conn(0x7f36d00b8fa0 0x7f36d00b6720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T12:31:22.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.525+0000 7f36d593c700 1 --2- 172.21.15.16:0/1865267954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36d015d8d0 0x7f36d00b2830 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7f36d00cdc10 tx=0x7f36c0009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=460e2f9c82782732 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:22.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.525+0000 7f36d493a700 1 -- 172.21.15.16:0/1865267954 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c0011830 con 0x7f36d015d8d0 2023-10-16T12:31:22.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.525+0000 7f36d493a700 1 -- 172.21.15.16:0/1865267954 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36c0011990 con 0x7f36d015d8d0 2023-10-16T12:31:22.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.525+0000 7f36d493a700 1 -- 172.21.15.16:0/1865267954 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c0011d20 con 0x7f36d015d8d0 2023-10-16T12:31:22.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.526+0000 7f36d7ba0700 1 -- 172.21.15.16:0/1865267954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36d015d8d0 msgr2=0x7f36d00b2830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:22.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.526+0000 7f36d7ba0700 1 --2- 172.21.15.16:0/1865267954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36d015d8d0 0x7f36d00b2830 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7f36d00cdc10 tx=0x7f36c0009b20 comp rx=0 tx=0).stop 2023-10-16T12:31:22.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.526+0000 7f36d7ba0700 1 -- 172.21.15.16:0/1865267954 shutdown_connections 2023-10-16T12:31:22.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.526+0000 7f36d7ba0700 1 --2- 172.21.15.16:0/1865267954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36d015d8d0 0x7f36d00b2830 unknown :-1 s=CLOSED pgs=1793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:22.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.526+0000 7f36d7ba0700 1 --2- 172.21.15.16:0/1865267954 >> v2:172.21.15.49:3300/0 conn(0x7f36d00b8fa0 0x7f36d00b6720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:22.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.526+0000 7f36d7ba0700 1 -- 172.21.15.16:0/1865267954 >> 172.21.15.16:0/1865267954 conn(0x7f36d0072330 msgr2=0x7f36d0072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:31:22.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.526+0000 7f36d7ba0700 1 -- 172.21.15.16:0/1865267954 shutdown_connections 2023-10-16T12:31:22.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.526+0000 7f36d7ba0700 1 -- 172.21.15.16:0/1865267954 wait complete. 2023-10-16T12:31:22.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.527+0000 7f36d7ba0700 1 Processor -- start 2023-10-16T12:31:22.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.527+0000 7f36d7ba0700 1 -- start start 2023-10-16T12:31:22.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.528+0000 7f36d7ba0700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f36d00b6c60 0x7f36d00cb750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:22.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.528+0000 7f36d7ba0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36d00b8fa0 0x7f36d0153520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:22.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.528+0000 7f36d7ba0700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f36d015d8d0 0x7f36d0153a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:22.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.528+0000 7f36d7ba0700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36d00ccb00 con 0x7f36d00b8fa0 2023-10-16T12:31:22.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.528+0000 7f36d7ba0700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36d00cbc90 con 0x7f36d015d8d0 2023-10-16T12:31:22.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.528+0000 7f36d7ba0700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36d00cbef0 con 0x7f36d00b6c60 2023-10-16T12:31:22.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.528+0000 7f36d513b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36d00b8fa0 0x7f36d0153520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:22.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.528+0000 7f36d513b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36d00b8fa0 0x7f36d0153520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47408/0 (socket says 172.21.15.16:47408) 2023-10-16T12:31:22.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.528+0000 7f36d513b700 1 -- 172.21.15.16:0/3065077252 learned_addr learned my addr 172.21.15.16:0/3065077252 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T12:31:22.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.528+0000 7f36d613d700 1 --2- 172.21.15.16:0/3065077252 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f36d015d8d0 0x7f36d0153a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:22.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.528+0000 7f36d593c700 1 --2- 172.21.15.16:0/3065077252 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f36d00b6c60 0x7f36d00cb750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:22.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.528+0000 7f36d513b700 1 -- 172.21.15.16:0/3065077252 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f36d00b6c60 msgr2=0x7f36d00cb750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:22.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.528+0000 7f36d513b700 1 --2- 172.21.15.16:0/3065077252 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f36d00b6c60 0x7f36d00cb750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:22.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.528+0000 7f36d513b700 1 -- 172.21.15.16:0/3065077252 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f36d015d8d0 msgr2=0x7f36d0153a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:22.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.528+0000 7f36d513b700 1 --2- 172.21.15.16:0/3065077252 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f36d015d8d0 0x7f36d0153a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:22.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.528+0000 7f36d513b700 1 -- 172.21.15.16:0/3065077252 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36c00094a0 con 0x7f36d00b8fa0 2023-10-16T12:31:22.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.528+0000 7f36d613d700 1 --2- 172.21.15.16:0/3065077252 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f36d015d8d0 0x7f36d0153a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T12:31:22.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.528+0000 7f36d513b700 1 --2- 172.21.15.16:0/3065077252 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36d00b8fa0 0x7f36d0153520 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7f36cc00c7f0 tx=0x7f36cc00cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:22.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.528+0000 7f36c67fc700 1 -- 172.21.15.16:0/3065077252 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36cc011800 con 0x7f36d00b8fa0 2023-10-16T12:31:22.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.529+0000 7f36c67fc700 1 -- 172.21.15.16:0/3065077252 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36cc011960 con 0x7f36d00b8fa0 2023-10-16T12:31:22.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.529+0000 7f36c67fc700 1 -- 172.21.15.16:0/3065077252 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36cc011cf0 con 0x7f36d00b8fa0 2023-10-16T12:31:22.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.529+0000 7f36d7ba0700 1 -- 172.21.15.16:0/3065077252 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36d00cc1d0 con 0x7f36d00b8fa0 2023-10-16T12:31:22.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.529+0000 7f36d7ba0700 1 -- 172.21.15.16:0/3065077252 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36d00cc790 con 0x7f36d00b8fa0 2023-10-16T12:31:22.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.530+0000 7f36d7ba0700 1 -- 172.21.15.16:0/3065077252 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36d015ec90 con 0x7f36d00b8fa0 2023-10-16T12:31:22.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.530+0000 7f36c67fc700 1 -- 172.21.15.16:0/3065077252 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 89) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f36cc018360 con 0x7f36d00b8fa0 2023-10-16T12:31:22.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.530+0000 7f36c67fc700 1 --2- 172.21.15.16:0/3065077252 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f36b406ada0 0x7f36b406d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T12:31:22.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.530+0000 7f36c67fc700 1 -- 172.21.15.16:0/3065077252 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 6429+0+0 (secure 0 0 0) 0x7f36cc0894a0 con 0x7f36d00b8fa0 2023-10-16T12:31:22.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.530+0000 7f36d593c700 1 --2- 172.21.15.16:0/3065077252 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f36b406ada0 0x7f36b406d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T12:31:22.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.531+0000 7f36d593c700 1 --2- 172.21.15.16:0/3065077252 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f36b406ada0 0x7f36b406d260 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f36c00090b0 tx=0x7f36c0008030 comp rx=0 tx=0).ready entity=mgr.8770 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T12:31:22.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.535+0000 7f36c67fc700 1 -- 172.21.15.16:0/3065077252 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f36cc054540 con 0x7f36d00b8fa0 2023-10-16T12:31:22.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.769+0000 7f36d7ba0700 1 -- 172.21.15.16:0/3065077252 --> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "/snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f36d00d21c0 con 0x7f36b406ada0 2023-10-16T12:31:22.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.779+0000 7f36c67fc700 1 -- 172.21.15.16:0/3065077252 <== mgr.8770 v2:172.21.15.16:6832/1503042987 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+74 (secure 0 0 0) 0x7f36c0011cb0 con 0x7f36b406ada0 2023-10-16T12:31:22.781 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T12:31:22.781 INFO:teuthology.orchestra.run.smithi016.stdout:{"path": "/snap_test_dir1", "schedule": ["1m"], "retention": [{"n": 55}]} 2023-10-16T12:31:22.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.781+0000 7f36d7ba0700 1 -- 172.21.15.16:0/3065077252 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f36b406ada0 msgr2=0x7f36b406d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:22.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.781+0000 7f36d7ba0700 1 --2- 172.21.15.16:0/3065077252 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f36b406ada0 0x7f36b406d260 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f36c00090b0 tx=0x7f36c0008030 comp rx=0 tx=0).stop 2023-10-16T12:31:22.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.781+0000 7f36d7ba0700 1 -- 172.21.15.16:0/3065077252 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36d00b8fa0 msgr2=0x7f36d0153520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T12:31:22.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.781+0000 7f36d7ba0700 1 --2- 172.21.15.16:0/3065077252 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36d00b8fa0 0x7f36d0153520 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7f36cc00c7f0 tx=0x7f36cc00cb00 comp rx=0 tx=0).stop 2023-10-16T12:31:22.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.781+0000 7f36d7ba0700 1 -- 172.21.15.16:0/3065077252 shutdown_connections 2023-10-16T12:31:22.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.781+0000 7f36d7ba0700 1 --2- 172.21.15.16:0/3065077252 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f36b406ada0 0x7f36b406d260 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:22.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.781+0000 7f36d7ba0700 1 --2- 172.21.15.16:0/3065077252 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f36d015d8d0 0x7f36d0153a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:22.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.781+0000 7f36d7ba0700 1 --2- 172.21.15.16:0/3065077252 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f36d00b6c60 0x7f36d00cb750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:22.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.781+0000 7f36d7ba0700 1 --2- 172.21.15.16:0/3065077252 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36d00b8fa0 0x7f36d0153520 unknown :-1 s=CLOSED pgs=1794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T12:31:22.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.781+0000 7f36d7ba0700 1 -- 172.21.15.16:0/3065077252 >> 172.21.15.16:0/3065077252 conn(0x7f36d0072330 msgr2=0x7f36d015ee90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T12:31:22.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.781+0000 7f36d7ba0700 1 -- 172.21.15.16:0/3065077252 shutdown_connections 2023-10-16T12:31:22.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T12:31:22.781+0000 7f36d7ba0700 1 -- 172.21.15.16:0/3065077252 wait complete. 2023-10-16T12:31:22.795 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "/snap_test_dir1", "schedule": ["1m"], "retention": [{"n": 55}]} 2023-10-16T12:31:22.796 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': '/snap_test_dir1', 'schedule': ['1m'], 'retention': [{'n': 55}]} 2023-10-16T12:31:51.517 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:31:51.520 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:32:00.393 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:32:00.391+0000 7f8a4b9e2700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:32:00.394 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:32:00.391+0000 7f8a4b9e2700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:32:00.394 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:32:00.391+0000 7f8a4b9e2700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:32:00.394 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:32:00.391+0000 7f8a4b9e2700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:32:00.394 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T12:32:00.391+0000 7f8a4b9e2700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T12:32:21.589 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:32:21.592 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:32:51.670 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:32:51.673 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:33:21.743 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:33:21.746 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:33:51.824 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:33:51.828 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:34:21.902 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:34:21.905 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:34:51.964 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:34:51.967 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:35:22.042 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:35:22.046 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:35:52.124 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:35:52.128 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:36:22.196 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:36:22.200 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:36:52.267 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:36:52.271 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:37:22.348 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:37:22.351 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:37:52.441 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:37:52.444 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:38:22.516 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:38:22.520 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:38:52.596 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:38:52.599 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:39:22.667 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:39:22.671 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:39:52.738 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:39:52.742 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:40:22.802 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:40:22.805 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:40:52.893 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:40:52.896 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:41:22.979 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:41:22.982 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:41:53.049 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:41:53.052 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:42:23.178 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:42:23.183 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:42:53.252 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:42:53.255 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:43:23.323 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:43:23.326 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:43:53.437 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:43:53.441 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:44:23.516 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:44:23.520 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:44:53.595 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:44:53.599 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:45:23.664 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:45:23.670 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:45:53.742 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:45:53.745 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:46:23.836 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:46:23.839 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:46:53.907 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:46:53.910 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:47:23.971 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:47:23.982 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:47:54.060 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:47:54.064 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:48:24.131 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:48:24.134 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:48:54.203 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:48:54.206 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:49:24.273 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:49:24.277 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:49:54.356 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:49:54.360 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:50:24.438 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:50:24.441 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:50:54.544 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:50:54.548 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:51:24.633 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:51:24.636 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:51:54.706 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:51:54.709 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:52:24.762 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:52:24.765 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:52:54.834 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:52:54.838 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:53:24.898 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:53:24.901 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:53:54.968 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:53:54.975 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:54:25.052 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:54:25.056 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:54:55.124 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:54:55.127 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:55:25.199 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:55:25.203 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:55:55.279 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:55:55.282 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:56:25.384 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:56:25.387 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:56:55.455 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:56:55.459 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:57:25.526 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:57:25.530 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:57:55.610 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:57:55.613 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:58:25.679 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:58:25.682 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:58:55.764 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:58:55.767 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:59:25.829 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:59:25.833 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:59:55.902 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T12:59:55.905 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:00:25.964 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:00:25.968 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:00:56.054 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:00:56.057 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:01:26.134 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:01:26.138 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:01:56.205 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:01:56.208 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:02:26.338 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:02:26.342 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:02:56.406 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:02:56.410 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:03:26.470 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:03:26.473 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:03:56.566 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:03:56.569 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:04:26.654 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:04:26.657 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:04:56.734 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:04:56.738 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:05:26.806 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:05:26.810 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:05:56.870 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:05:56.874 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:06:26.994 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:06:26.997 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:06:57.067 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:06:57.070 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:07:27.156 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:07:27.159 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:07:57.229 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:07:57.233 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:08:27.312 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:08:27.315 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:08:57.493 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:08:57.496 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:09:27.571 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:09:27.575 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:09:57.643 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:09:57.646 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:10:27.733 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:10:27.736 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:10:57.805 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:10:57.808 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:11:28.016 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:11:28.020 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:11:58.096 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:11:58.099 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:12:28.167 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:12:28.170 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:12:58.239 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:12:58.242 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:13:28.310 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:13:28.314 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:13:58.408 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:13:58.411 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:14:28.471 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:14:28.474 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:14:58.558 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:14:58.561 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:15:28.633 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:15:28.640 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:15:58.708 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:15:58.712 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:16:28.837 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:16:28.840 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:16:58.909 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:16:58.912 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:17:28.983 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:17:28.986 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:17:59.062 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:17:59.065 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:18:29.125 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:18:29.128 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:18:59.188 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:18:59.191 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:19:29.264 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:19:29.267 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:19:59.331 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:19:59.334 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:20:29.413 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:20:29.416 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:20:59.508 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:20:59.512 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:21:29.580 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:21:29.583 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:21:59.652 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:21:59.655 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:22:29.741 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:22:29.744 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:22:59.811 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:22:59.814 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:23:29.886 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:23:29.890 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:23:59.958 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:23:59.961 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:24:30.039 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:24:30.043 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:25:00.105 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:25:00.109 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:25:30.176 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:25:30.180 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:26:00.248 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:26:00.251 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:26:30.319 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:26:30.322 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:27:00.390 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:27:00.394 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:27:15.797 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:27:15.797 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T13:27:15.825 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_33_00_UTC 2023-10-16T13:27:15.825 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_34_00_UTC 2023-10-16T13:27:15.825 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_35_00_UTC 2023-10-16T13:27:15.825 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_36_00_UTC 2023-10-16T13:27:15.825 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_37_00_UTC 2023-10-16T13:27:15.826 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_38_00_UTC 2023-10-16T13:27:15.826 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_39_00_UTC 2023-10-16T13:27:15.826 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_40_00_UTC 2023-10-16T13:27:15.826 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_41_00_UTC 2023-10-16T13:27:15.826 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_42_00_UTC 2023-10-16T13:27:15.827 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_43_00_UTC 2023-10-16T13:27:15.827 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_44_00_UTC 2023-10-16T13:27:15.827 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_45_00_UTC 2023-10-16T13:27:15.827 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_46_00_UTC 2023-10-16T13:27:15.827 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_47_00_UTC 2023-10-16T13:27:15.828 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_48_00_UTC 2023-10-16T13:27:15.828 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_49_00_UTC 2023-10-16T13:27:15.828 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_50_00_UTC 2023-10-16T13:27:15.828 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_51_00_UTC 2023-10-16T13:27:15.828 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_52_00_UTC 2023-10-16T13:27:15.829 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_53_00_UTC 2023-10-16T13:27:15.829 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_54_00_UTC 2023-10-16T13:27:15.829 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_55_00_UTC 2023-10-16T13:27:15.829 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_56_00_UTC 2023-10-16T13:27:15.829 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_57_00_UTC 2023-10-16T13:27:15.829 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_58_00_UTC 2023-10-16T13:27:15.830 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_59_00_UTC 2023-10-16T13:27:15.830 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_00_00_UTC 2023-10-16T13:27:15.830 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_01_00_UTC 2023-10-16T13:27:15.830 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_02_00_UTC 2023-10-16T13:27:15.830 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_03_00_UTC 2023-10-16T13:27:15.831 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_04_00_UTC 2023-10-16T13:27:15.831 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_05_00_UTC 2023-10-16T13:27:15.831 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_06_00_UTC 2023-10-16T13:27:15.831 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_07_00_UTC 2023-10-16T13:27:15.831 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_08_00_UTC 2023-10-16T13:27:15.832 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_09_00_UTC 2023-10-16T13:27:15.832 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_10_00_UTC 2023-10-16T13:27:15.832 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_11_00_UTC 2023-10-16T13:27:15.832 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_12_00_UTC 2023-10-16T13:27:15.832 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_13_00_UTC 2023-10-16T13:27:15.833 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_14_00_UTC 2023-10-16T13:27:15.833 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_15_00_UTC 2023-10-16T13:27:15.833 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_16_00_UTC 2023-10-16T13:27:15.833 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_17_00_UTC 2023-10-16T13:27:15.833 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_18_00_UTC 2023-10-16T13:27:15.833 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_19_00_UTC 2023-10-16T13:27:15.834 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_20_00_UTC 2023-10-16T13:27:15.834 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_21_00_UTC 2023-10-16T13:27:15.834 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_22_00_UTC 2023-10-16T13:27:15.834 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_23_00_UTC 2023-10-16T13:27:15.834 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_24_00_UTC 2023-10-16T13:27:15.835 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_25_00_UTC 2023-10-16T13:27:15.835 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_26_00_UTC 2023-10-16T13:27:15.835 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_27_00_UTC 2023-10-16T13:27:15.835 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-16-12_33_00_UTC', 'scheduled-2023-10-16-12_34_00_UTC', 'scheduled-2023-10-16-12_35_00_UTC', 'scheduled-2023-10-16-12_36_00_UTC', 'scheduled-2023-10-16-12_37_00_UTC', 'scheduled-2023-10-16-12_38_00_UTC', 'scheduled-2023-10-16-12_39_00_UTC', 'scheduled-2023-10-16-12_40_00_UTC', 'scheduled-2023-10-16-12_41_00_UTC', 'scheduled-2023-10-16-12_42_00_UTC', 'scheduled-2023-10-16-12_43_00_UTC', 'scheduled-2023-10-16-12_44_00_UTC', 'scheduled-2023-10-16-12_45_00_UTC', 'scheduled-2023-10-16-12_46_00_UTC', 'scheduled-2023-10-16-12_47_00_UTC', 'scheduled-2023-10-16-12_48_00_UTC', 'scheduled-2023-10-16-12_49_00_UTC', 'scheduled-2023-10-16-12_50_00_UTC', 'scheduled-2023-10-16-12_51_00_UTC', 'scheduled-2023-10-16-12_52_00_UTC', 'scheduled-2023-10-16-12_53_00_UTC', 'scheduled-2023-10-16-12_54_00_UTC', 'scheduled-2023-10-16-12_55_00_UTC', 'scheduled-2023-10-16-12_56_00_UTC', 'scheduled-2023-10-16-12_57_00_UTC', 'scheduled-2023-10-16-12_58_00_UTC', 'scheduled-2023-10-16-12_59_00_UTC', 'scheduled-2023-10-16-13_00_00_UTC', 'scheduled-2023-10-16-13_01_00_UTC', 'scheduled-2023-10-16-13_02_00_UTC', 'scheduled-2023-10-16-13_03_00_UTC', 'scheduled-2023-10-16-13_04_00_UTC', 'scheduled-2023-10-16-13_05_00_UTC', 'scheduled-2023-10-16-13_06_00_UTC', 'scheduled-2023-10-16-13_07_00_UTC', 'scheduled-2023-10-16-13_08_00_UTC', 'scheduled-2023-10-16-13_09_00_UTC', 'scheduled-2023-10-16-13_10_00_UTC', 'scheduled-2023-10-16-13_11_00_UTC', 'scheduled-2023-10-16-13_12_00_UTC', 'scheduled-2023-10-16-13_13_00_UTC', 'scheduled-2023-10-16-13_14_00_UTC', 'scheduled-2023-10-16-13_15_00_UTC', 'scheduled-2023-10-16-13_16_00_UTC', 'scheduled-2023-10-16-13_17_00_UTC', 'scheduled-2023-10-16-13_18_00_UTC', 'scheduled-2023-10-16-13_19_00_UTC', 'scheduled-2023-10-16-13_20_00_UTC', 'scheduled-2023-10-16-13_21_00_UTC', 'scheduled-2023-10-16-13_22_00_UTC', 'scheduled-2023-10-16-13_23_00_UTC', 'scheduled-2023-10-16-13_24_00_UTC', 'scheduled-2023-10-16-13_25_00_UTC', 'scheduled-2023-10-16-13_26_00_UTC', 'scheduled-2023-10-16-13_27_00_UTC'] 2023-10-16T13:27:15.836 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1 2023-10-16T13:27:16.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:15.999+0000 7fd6d368d700 1 Processor -- start 2023-10-16T13:27:16.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:15.999+0000 7fd6d368d700 1 -- start start 2023-10-16T13:27:16.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.000+0000 7fd6d368d700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fd6cc15b2c0 0x7fd6cc161340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:16.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.000+0000 7fd6d368d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd6cc06a7e0 0x7fd6cc06ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:16.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.000+0000 7fd6d368d700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6cc15ecb0 con 0x7fd6cc06e570 2023-10-16T13:27:16.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.000+0000 7fd6d368d700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6cc06b1b0 con 0x7fd6cc15b2c0 2023-10-16T13:27:16.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.000+0000 7fd6d368d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6cc161880 con 0x7fd6cc06a7e0 2023-10-16T13:27:16.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.000+0000 7fd6d1c2a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd6cc06a7e0 0x7fd6cc06ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:16.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.000+0000 7fd6d0c28700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7fd6cc06e570 0x7fd6cc15e5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:44016/0 (socket says 172.21.15.16:44016) 2023-10-16T13:27:16.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.000+0000 7fd6d1429700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fd6cc15b2c0 0x7fd6cc161340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:16.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.000+0000 7fd6d0c28700 1 -- 172.21.15.16:0/3422705635 learned_addr learned my addr 172.21.15.16:0/3422705635 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:27:16.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.001+0000 7fd6d1c2a700 1 -- 172.21.15.16:0/3422705635 >> v2:172.21.15.49:3300/0 conn(0x7fd6cc15b2c0 msgr2=0x7fd6cc161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:16.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.001+0000 7fd6d1c2a700 1 --2- 172.21.15.16:0/3422705635 >> v2:172.21.15.49:3300/0 conn(0x7fd6cc15b2c0 0x7fd6cc161340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:16.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.001+0000 7fd6d1c2a700 1 -- 172.21.15.16:0/3422705635 >> v1:172.21.15.49:6789/0 conn(0x7fd6cc06e570 legacy=0x7fd6cc15e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:16.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.001+0000 7fd6d1c2a700 1 -- 172.21.15.16:0/3422705635 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6cc15b6a0 con 0x7fd6cc06a7e0 2023-10-16T13:27:16.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.001+0000 7fd6d1429700 1 --2- 172.21.15.16:0/3422705635 >> v2:172.21.15.49:3300/0 conn(0x7fd6cc15b2c0 0x7fd6cc161340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:27:16.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.001+0000 7fd6d1c2a700 1 --2- 172.21.15.16:0/3422705635 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd6cc06a7e0 0x7fd6cc06ac70 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fd6c4009d90 tx=0x7fd6c400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74ef28f82f1d53 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:16.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.001+0000 7fd6c3fff700 1 -- 172.21.15.16:0/3422705635 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6c400e070 con 0x7fd6cc06a7e0 2023-10-16T13:27:16.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.001+0000 7fd6c3fff700 1 -- 172.21.15.16:0/3422705635 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6c400bb20 con 0x7fd6cc06a7e0 2023-10-16T13:27:16.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.001+0000 7fd6c3fff700 1 -- 172.21.15.16:0/3422705635 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6c4012430 con 0x7fd6cc06a7e0 2023-10-16T13:27:16.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.002+0000 7fd6d368d700 1 -- 172.21.15.16:0/3422705635 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd6cc06a7e0 msgr2=0x7fd6cc06ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:16.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.002+0000 7fd6d368d700 1 --2- 172.21.15.16:0/3422705635 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd6cc06a7e0 0x7fd6cc06ac70 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fd6c4009d90 tx=0x7fd6c400b330 comp rx=0 tx=0).stop 2023-10-16T13:27:16.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.002+0000 7fd6d368d700 1 -- 172.21.15.16:0/3422705635 shutdown_connections 2023-10-16T13:27:16.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.002+0000 7fd6d368d700 1 --2- 172.21.15.16:0/3422705635 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd6cc06a7e0 0x7fd6cc06ac70 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:16.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.002+0000 7fd6d368d700 1 --2- 172.21.15.16:0/3422705635 >> v2:172.21.15.49:3300/0 conn(0x7fd6cc15b2c0 0x7fd6cc161340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:16.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.002+0000 7fd6d368d700 1 -- 172.21.15.16:0/3422705635 >> 172.21.15.16:0/3422705635 conn(0x7fd6cc0b1f40 msgr2=0x7fd6cc0b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:16.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.002+0000 7fd6d368d700 1 -- 172.21.15.16:0/3422705635 shutdown_connections 2023-10-16T13:27:16.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.002+0000 7fd6d368d700 1 -- 172.21.15.16:0/3422705635 wait complete. 2023-10-16T13:27:16.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.003+0000 7fd6d368d700 1 Processor -- start 2023-10-16T13:27:16.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.003+0000 7fd6d368d700 1 -- start start 2023-10-16T13:27:16.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.004+0000 7fd6d368d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6cc06a7e0 0x7fd6cc0c69d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:16.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.004+0000 7fd6d368d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd6cc06e570 0x7fd6cc1f2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:16.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.004+0000 7fd6d368d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd6cc15b2c0 0x7fd6cc1f2750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:16.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.004+0000 7fd6d368d700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6cc0ca750 con 0x7fd6cc06a7e0 2023-10-16T13:27:16.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.004+0000 7fd6d368d700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6cc0ca470 con 0x7fd6cc15b2c0 2023-10-16T13:27:16.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.004+0000 7fd6d368d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6cc0b4c40 con 0x7fd6cc06e570 2023-10-16T13:27:16.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.004+0000 7fd6d1429700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6cc06a7e0 0x7fd6cc0c69d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:16.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.004+0000 7fd6d1429700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6cc06a7e0 0x7fd6cc0c69d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47562/0 (socket says 172.21.15.16:47562) 2023-10-16T13:27:16.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.004+0000 7fd6d1429700 1 -- 172.21.15.16:0/1065674277 learned_addr learned my addr 172.21.15.16:0/1065674277 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:16.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.004+0000 7fd6d1c2a700 1 --2- 172.21.15.16:0/1065674277 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd6cc15b2c0 0x7fd6cc1f2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:16.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.004+0000 7fd6d0c28700 1 --2- 172.21.15.16:0/1065674277 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd6cc06e570 0x7fd6cc1f2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:16.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.004+0000 7fd6d1429700 1 -- 172.21.15.16:0/1065674277 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd6cc06e570 msgr2=0x7fd6cc1f2210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:16.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.004+0000 7fd6d1429700 1 --2- 172.21.15.16:0/1065674277 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd6cc06e570 0x7fd6cc1f2210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:16.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.004+0000 7fd6d1429700 1 -- 172.21.15.16:0/1065674277 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd6cc15b2c0 msgr2=0x7fd6cc1f2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:16.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.004+0000 7fd6d1429700 1 --2- 172.21.15.16:0/1065674277 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd6cc15b2c0 0x7fd6cc1f2750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:16.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.004+0000 7fd6d1429700 1 -- 172.21.15.16:0/1065674277 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6bc0094a0 con 0x7fd6cc06a7e0 2023-10-16T13:27:16.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.004+0000 7fd6d1c2a700 1 --2- 172.21.15.16:0/1065674277 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd6cc15b2c0 0x7fd6cc1f2750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:27:16.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.005+0000 7fd6d1429700 1 --2- 172.21.15.16:0/1065674277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6cc06a7e0 0x7fd6cc0c69d0 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc00ebe0 tx=0x7fd6bc00eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:16.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.005+0000 7fd6c1ffb700 1 -- 172.21.15.16:0/1065674277 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6bc00c850 con 0x7fd6cc06a7e0 2023-10-16T13:27:16.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.005+0000 7fd6c1ffb700 1 -- 172.21.15.16:0/1065674277 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6bc00c9b0 con 0x7fd6cc06a7e0 2023-10-16T13:27:16.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.005+0000 7fd6d368d700 1 -- 172.21.15.16:0/1065674277 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6c4009a70 con 0x7fd6cc06a7e0 2023-10-16T13:27:16.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.005+0000 7fd6c1ffb700 1 -- 172.21.15.16:0/1065674277 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6bc00cd40 con 0x7fd6cc06a7e0 2023-10-16T13:27:16.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.005+0000 7fd6d368d700 1 -- 172.21.15.16:0/1065674277 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6cc0b5200 con 0x7fd6cc06a7e0 2023-10-16T13:27:16.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.006+0000 7fd6d368d700 1 -- 172.21.15.16:0/1065674277 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6cc0b4aa0 con 0x7fd6cc06a7e0 2023-10-16T13:27:16.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.006+0000 7fd6c1ffb700 1 -- 172.21.15.16:0/1065674277 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 93) v1 ==== 89360+0+0 (secure 0 0 0) 0x7fd6bc013470 con 0x7fd6cc06a7e0 2023-10-16T13:27:16.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.007+0000 7fd6c1ffb700 1 --2- 172.21.15.16:0/1065674277 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7fd6b006b070 0x7fd6b006d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:16.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.007+0000 7fd6c1ffb700 1 -- 172.21.15.16:0/1065674277 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 6517+0+0 (secure 0 0 0) 0x7fd6bc089be0 con 0x7fd6cc06a7e0 2023-10-16T13:27:16.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.007+0000 7fd6d0c28700 1 --2- 172.21.15.16:0/1065674277 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7fd6b006b070 0x7fd6b006d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:16.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.007+0000 7fd6d0c28700 1 --2- 172.21.15.16:0/1065674277 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7fd6b006b070 0x7fd6b006d530 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd6c8005910 tx=0x7fd6c8009040 comp rx=0 tx=0).ready entity=mgr.8770 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:16.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.012+0000 7fd6c1ffb700 1 -- 172.21.15.16:0/1065674277 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd6cc0b4aa0 con 0x7fd6cc06a7e0 2023-10-16T13:27:16.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.241+0000 7fd6d368d700 1 -- 172.21.15.16:0/1065674277 --> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd6cc161e00 con 0x7fd6b006b070 2023-10-16T13:27:16.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.250+0000 7fd6c1ffb700 1 -- 172.21.15.16:0/1065674277 <== mgr.8770 v2:172.21.15.16:6832/1503042987 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+373 (secure 0 0 0) 0x7fd6c80014e0 con 0x7fd6b006b070 2023-10-16T13:27:16.253 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:27:16.253 INFO:teuthology.orchestra.run.smithi016.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1", "rel_path": "/snap_test_dir1", "schedule": "1m", "retention": {"n": 55}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T12:31:21", "first": "2023-10-16T12:32:00", "last": "2023-10-16T13:27:00", "last_pruned": "2023-10-16T13:27:00", "created_count": 56, "pruned_count": 1, "active": true}] 2023-10-16T13:27:16.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.252+0000 7fd6d368d700 1 -- 172.21.15.16:0/1065674277 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7fd6b006b070 msgr2=0x7fd6b006d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:16.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.252+0000 7fd6d368d700 1 --2- 172.21.15.16:0/1065674277 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7fd6b006b070 0x7fd6b006d530 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd6c8005910 tx=0x7fd6c8009040 comp rx=0 tx=0).stop 2023-10-16T13:27:16.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.252+0000 7fd6d368d700 1 -- 172.21.15.16:0/1065674277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6cc06a7e0 msgr2=0x7fd6cc0c69d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:16.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.252+0000 7fd6d368d700 1 --2- 172.21.15.16:0/1065674277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6cc06a7e0 0x7fd6cc0c69d0 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc00ebe0 tx=0x7fd6bc00eef0 comp rx=0 tx=0).stop 2023-10-16T13:27:16.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.253+0000 7fd6d368d700 1 -- 172.21.15.16:0/1065674277 shutdown_connections 2023-10-16T13:27:16.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.253+0000 7fd6d368d700 1 --2- 172.21.15.16:0/1065674277 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7fd6b006b070 0x7fd6b006d530 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:16.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.253+0000 7fd6d368d700 1 --2- 172.21.15.16:0/1065674277 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd6cc15b2c0 0x7fd6cc1f2750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:16.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.253+0000 7fd6d368d700 1 --2- 172.21.15.16:0/1065674277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6cc06a7e0 0x7fd6cc0c69d0 unknown :-1 s=CLOSED pgs=1799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:16.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.253+0000 7fd6d368d700 1 --2- 172.21.15.16:0/1065674277 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd6cc06e570 0x7fd6cc1f2210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:16.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.253+0000 7fd6d368d700 1 -- 172.21.15.16:0/1065674277 >> 172.21.15.16:0/1065674277 conn(0x7fd6cc0b1f40 msgr2=0x7fd6cc15e4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:16.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.253+0000 7fd6d368d700 1 -- 172.21.15.16:0/1065674277 shutdown_connections 2023-10-16T13:27:16.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.253+0000 7fd6d368d700 1 -- 172.21.15.16:0/1065674277 wait complete. 2023-10-16T13:27:16.267 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1", "rel_path": "/snap_test_dir1", "schedule": "1m", "retention": {"n": 55}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T12:31:21", "first": "2023-10-16T12:32:00", "last": "2023-10-16T13:27:00", "last_pruned": "2023-10-16T13:27:00", "created_count": 56, "pruned_count": 1, "active": true}] 2023-10-16T13:27:16.267 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1', 'rel_path': '/snap_test_dir1', 'schedule': '1m', 'retention': {'n': 55}, 'start': '2023-10-16T00:00:00', 'created': '2023-10-16T12:31:21', 'first': '2023-10-16T12:32:00', 'last': '2023-10-16T13:27:00', 'last_pruned': '2023-10-16T13:27:00', 'created_count': 56, 'pruned_count': 1, 'active': True} 2023-10-16T13:27:16.267 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 55 2023-10-16T13:27:16.268 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 56 2023-10-16T13:27:16.268 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1 2023-10-16T13:27:16.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.393+0000 7f4ed69d2700 1 Processor -- start 2023-10-16T13:27:16.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.393+0000 7f4ed69d2700 1 -- start start 2023-10-16T13:27:16.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.393+0000 7f4ed69d2700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4ed00ce6a0 0x7f4ed006c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:16.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.393+0000 7f4ed69d2700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4ed00d2110 0x7f4ed006c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:16.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.393+0000 7f4ed69d2700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ed00b20f0 con 0x7f4ed00b2620 2023-10-16T13:27:16.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.393+0000 7f4ed69d2700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ed00af1f0 con 0x7f4ed00ce6a0 2023-10-16T13:27:16.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.393+0000 7f4ed69d2700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed00d03d0 con 0x7f4ed00d2110 2023-10-16T13:27:16.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.394+0000 7f4ecffff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4ed00ce6a0 0x7f4ed006c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:16.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.394+0000 7f4ed4f6f700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f4ed00b2620 0x7f4ed00b0f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36848/0 (socket says 172.21.15.16:36848) 2023-10-16T13:27:16.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.394+0000 7f4ed4f6f700 1 -- 172.21.15.16:0/521441997 learned_addr learned my addr 172.21.15.16:0/521441997 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:27:16.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.394+0000 7f4ecf7fe700 1 --2- 172.21.15.16:0/521441997 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4ed00d2110 0x7f4ed006c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:16.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.394+0000 7f4ecffff700 1 -- 172.21.15.16:0/521441997 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4ed00d2110 msgr2=0x7f4ed006c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:16.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.394+0000 7f4ecffff700 1 --2- 172.21.15.16:0/521441997 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4ed00d2110 0x7f4ed006c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:16.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.394+0000 7f4ecffff700 1 -- 172.21.15.16:0/521441997 >> v1:172.21.15.16:6789/0 conn(0x7f4ed00b2620 legacy=0x7f4ed00b0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:16.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.394+0000 7f4ecffff700 1 -- 172.21.15.16:0/521441997 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ed00b2340 con 0x7f4ed00ce6a0 2023-10-16T13:27:16.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.394+0000 7f4eceffd700 1 -- 172.21.15.16:0/521441997 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3137431741 0 0) 0x7f4ed00b20f0 con 0x7f4ed00b2620 2023-10-16T13:27:16.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.394+0000 7f4ecffff700 1 --2- 172.21.15.16:0/521441997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ed00ce6a0 0x7f4ed006c300 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f4ec0009840 tx=0x7f4ec0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab0339a3156fceb9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:16.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.394+0000 7f4eceffd700 1 -- 172.21.15.16:0/521441997 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ec0011830 con 0x7f4ed00ce6a0 2023-10-16T13:27:16.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.394+0000 7f4eceffd700 1 -- 172.21.15.16:0/521441997 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ec0011990 con 0x7f4ed00ce6a0 2023-10-16T13:27:16.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.394+0000 7f4eceffd700 1 -- 172.21.15.16:0/521441997 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ec0011d20 con 0x7f4ed00ce6a0 2023-10-16T13:27:16.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.395+0000 7f4ed69d2700 1 -- 172.21.15.16:0/521441997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ed00ce6a0 msgr2=0x7f4ed006c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:16.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.395+0000 7f4ed69d2700 1 --2- 172.21.15.16:0/521441997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ed00ce6a0 0x7f4ed006c300 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f4ec0009840 tx=0x7f4ec0009b50 comp rx=0 tx=0).stop 2023-10-16T13:27:16.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.395+0000 7f4ed69d2700 1 -- 172.21.15.16:0/521441997 shutdown_connections 2023-10-16T13:27:16.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.395+0000 7f4ed69d2700 1 --2- 172.21.15.16:0/521441997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ed00ce6a0 0x7f4ed006c300 unknown :-1 s=CLOSED pgs=1801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:16.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.395+0000 7f4ed69d2700 1 --2- 172.21.15.16:0/521441997 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4ed00d2110 0x7f4ed006c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:16.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.395+0000 7f4ed69d2700 1 -- 172.21.15.16:0/521441997 >> 172.21.15.16:0/521441997 conn(0x7f4ed00d15f0 msgr2=0x7f4ed00d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:16.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.395+0000 7f4ed69d2700 1 -- 172.21.15.16:0/521441997 shutdown_connections 2023-10-16T13:27:16.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.395+0000 7f4ed69d2700 1 -- 172.21.15.16:0/521441997 wait complete. 2023-10-16T13:27:16.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.396+0000 7f4ed69d2700 1 Processor -- start 2023-10-16T13:27:16.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.396+0000 7f4ed69d2700 1 -- start start 2023-10-16T13:27:16.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.397+0000 7f4ed69d2700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4ed00b2620 0x7f4ed0154b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:16.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.397+0000 7f4ed69d2700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4ed00ce6a0 0x7f4ed00b6570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:16.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.397+0000 7f4ed69d2700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ed00d2110 0x7f4ed00b6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:16.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.397+0000 7f4ed69d2700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed0155070 con 0x7f4ed00d2110 2023-10-16T13:27:16.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.397+0000 7f4ed69d2700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed00c8dc0 con 0x7f4ed00b2620 2023-10-16T13:27:16.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.397+0000 7f4ed69d2700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed00b5030 con 0x7f4ed00ce6a0 2023-10-16T13:27:16.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.397+0000 7f4ed4f6f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ed00d2110 0x7f4ed00b6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:16.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.397+0000 7f4ed4f6f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ed00d2110 0x7f4ed00b6ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47580/0 (socket says 172.21.15.16:47580) 2023-10-16T13:27:16.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.397+0000 7f4ed4f6f700 1 -- 172.21.15.16:0/1370085776 learned_addr learned my addr 172.21.15.16:0/1370085776 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:16.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.397+0000 7f4ecf7fe700 1 --2- 172.21.15.16:0/1370085776 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4ed00ce6a0 0x7f4ed00b6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:16.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.397+0000 7f4ecffff700 1 --2- 172.21.15.16:0/1370085776 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4ed00b2620 0x7f4ed0154b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:16.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.397+0000 7f4ed4f6f700 1 -- 172.21.15.16:0/1370085776 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4ed00ce6a0 msgr2=0x7f4ed00b6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:16.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.397+0000 7f4ed4f6f700 1 --2- 172.21.15.16:0/1370085776 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4ed00ce6a0 0x7f4ed00b6570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:16.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.397+0000 7f4ed4f6f700 1 -- 172.21.15.16:0/1370085776 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4ed00b2620 msgr2=0x7f4ed0154b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:16.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.397+0000 7f4ed4f6f700 1 --2- 172.21.15.16:0/1370085776 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4ed00b2620 0x7f4ed0154b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:16.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.397+0000 7f4ed4f6f700 1 -- 172.21.15.16:0/1370085776 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ec00094a0 con 0x7f4ed00d2110 2023-10-16T13:27:16.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.398+0000 7f4ed4f6f700 1 --2- 172.21.15.16:0/1370085776 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ed00d2110 0x7f4ed00b6ab0 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7f4ec8002230 tx=0x7f4ec800cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:16.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.398+0000 7f4eccff9700 1 -- 172.21.15.16:0/1370085776 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ec800aac0 con 0x7f4ed00d2110 2023-10-16T13:27:16.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.398+0000 7f4eccff9700 1 -- 172.21.15.16:0/1370085776 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ec800ac40 con 0x7f4ed00d2110 2023-10-16T13:27:16.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.398+0000 7f4ed69d2700 1 -- 172.21.15.16:0/1370085776 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ed00b5310 con 0x7f4ed00d2110 2023-10-16T13:27:16.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.398+0000 7f4eccff9700 1 -- 172.21.15.16:0/1370085776 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ec80070d0 con 0x7f4ed00d2110 2023-10-16T13:27:16.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.398+0000 7f4ed69d2700 1 -- 172.21.15.16:0/1370085776 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ed00b58d0 con 0x7f4ed00d2110 2023-10-16T13:27:16.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.399+0000 7f4ed69d2700 1 -- 172.21.15.16:0/1370085776 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ed00c8fc0 con 0x7f4ed00d2110 2023-10-16T13:27:16.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.399+0000 7f4eccff9700 1 -- 172.21.15.16:0/1370085776 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 93) v1 ==== 89360+0+0 (secure 0 0 0) 0x7f4ec8007250 con 0x7f4ed00d2110 2023-10-16T13:27:16.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.400+0000 7f4eccff9700 1 --2- 172.21.15.16:0/1370085776 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f4ebc06b070 0x7f4ebc06d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:16.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.400+0000 7f4eccff9700 1 -- 172.21.15.16:0/1370085776 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f4ec8089870 con 0x7f4ed00d2110 2023-10-16T13:27:16.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.400+0000 7f4ecffff700 1 --2- 172.21.15.16:0/1370085776 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f4ebc06b070 0x7f4ebc06d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:16.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.400+0000 7f4ecffff700 1 --2- 172.21.15.16:0/1370085776 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f4ebc06b070 0x7f4ebc06d530 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f4ec00090b0 tx=0x7f4ec0008030 comp rx=0 tx=0).ready entity=mgr.8770 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:16.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.404+0000 7f4eccff9700 1 -- 172.21.15.16:0/1370085776 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4ed00c8fc0 con 0x7f4ed00d2110 2023-10-16T13:27:16.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.635+0000 7f4ed69d2700 1 -- 172.21.15.16:0/1370085776 --> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f4ed006bde0 con 0x7f4ebc06b070 2023-10-16T13:27:16.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.715+0000 7f4eccff9700 1 -- 172.21.15.16:0/1370085776 <== mgr.8770 v2:172.21.15.16:6832/1503042987 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+42 (secure 0 0 0) 0x7f4ec0011cb0 con 0x7f4ebc06b070 2023-10-16T13:27:16.718 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule removed for path /snap_test_dir1 2023-10-16T13:27:16.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.717+0000 7f4ed69d2700 1 -- 172.21.15.16:0/1370085776 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f4ebc06b070 msgr2=0x7f4ebc06d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:16.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.717+0000 7f4ed69d2700 1 --2- 172.21.15.16:0/1370085776 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f4ebc06b070 0x7f4ebc06d530 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f4ec00090b0 tx=0x7f4ec0008030 comp rx=0 tx=0).stop 2023-10-16T13:27:16.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.717+0000 7f4ed69d2700 1 -- 172.21.15.16:0/1370085776 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ed00d2110 msgr2=0x7f4ed00b6ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:16.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.717+0000 7f4ed69d2700 1 --2- 172.21.15.16:0/1370085776 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ed00d2110 0x7f4ed00b6ab0 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7f4ec8002230 tx=0x7f4ec800cf50 comp rx=0 tx=0).stop 2023-10-16T13:27:16.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.718+0000 7f4ed69d2700 1 -- 172.21.15.16:0/1370085776 shutdown_connections 2023-10-16T13:27:16.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.718+0000 7f4ed69d2700 1 --2- 172.21.15.16:0/1370085776 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f4ebc06b070 0x7f4ebc06d530 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:16.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.718+0000 7f4ed69d2700 1 --2- 172.21.15.16:0/1370085776 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ed00d2110 0x7f4ed00b6ab0 unknown :-1 s=CLOSED pgs=1802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:16.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.718+0000 7f4ed69d2700 1 --2- 172.21.15.16:0/1370085776 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4ed00b2620 0x7f4ed0154b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:16.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.718+0000 7f4ed69d2700 1 --2- 172.21.15.16:0/1370085776 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4ed00ce6a0 0x7f4ed00b6570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:16.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.718+0000 7f4ed69d2700 1 -- 172.21.15.16:0/1370085776 >> 172.21.15.16:0/1370085776 conn(0x7f4ed00d15f0 msgr2=0x7f4ed006c300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:16.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.718+0000 7f4ed69d2700 1 -- 172.21.15.16:0/1370085776 shutdown_connections 2023-10-16T13:27:16.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:16.718+0000 7f4ed69d2700 1 -- 172.21.15.16:0/1370085776 wait complete. 2023-10-16T13:27:16.732 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1 2023-10-16T13:27:16.732 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:27:16.733 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T13:27:16.752 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_33_00_UTC 2023-10-16T13:27:16.752 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_34_00_UTC 2023-10-16T13:27:16.753 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_35_00_UTC 2023-10-16T13:27:16.753 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_36_00_UTC 2023-10-16T13:27:16.753 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_37_00_UTC 2023-10-16T13:27:16.753 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_38_00_UTC 2023-10-16T13:27:16.753 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_39_00_UTC 2023-10-16T13:27:16.754 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_40_00_UTC 2023-10-16T13:27:16.754 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_41_00_UTC 2023-10-16T13:27:16.754 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_42_00_UTC 2023-10-16T13:27:16.754 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_43_00_UTC 2023-10-16T13:27:16.754 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_44_00_UTC 2023-10-16T13:27:16.755 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_45_00_UTC 2023-10-16T13:27:16.755 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_46_00_UTC 2023-10-16T13:27:16.755 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_47_00_UTC 2023-10-16T13:27:16.755 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_48_00_UTC 2023-10-16T13:27:16.755 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_49_00_UTC 2023-10-16T13:27:16.756 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_50_00_UTC 2023-10-16T13:27:16.756 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_51_00_UTC 2023-10-16T13:27:16.756 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_52_00_UTC 2023-10-16T13:27:16.756 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_53_00_UTC 2023-10-16T13:27:16.756 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_54_00_UTC 2023-10-16T13:27:16.757 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_55_00_UTC 2023-10-16T13:27:16.757 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_56_00_UTC 2023-10-16T13:27:16.757 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_57_00_UTC 2023-10-16T13:27:16.757 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_58_00_UTC 2023-10-16T13:27:16.757 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-12_59_00_UTC 2023-10-16T13:27:16.758 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_00_00_UTC 2023-10-16T13:27:16.758 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_01_00_UTC 2023-10-16T13:27:16.758 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_02_00_UTC 2023-10-16T13:27:16.758 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_03_00_UTC 2023-10-16T13:27:16.758 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_04_00_UTC 2023-10-16T13:27:16.759 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_05_00_UTC 2023-10-16T13:27:16.759 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_06_00_UTC 2023-10-16T13:27:16.759 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_07_00_UTC 2023-10-16T13:27:16.759 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_08_00_UTC 2023-10-16T13:27:16.759 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_09_00_UTC 2023-10-16T13:27:16.760 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_10_00_UTC 2023-10-16T13:27:16.760 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_11_00_UTC 2023-10-16T13:27:16.760 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_12_00_UTC 2023-10-16T13:27:16.760 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_13_00_UTC 2023-10-16T13:27:16.760 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_14_00_UTC 2023-10-16T13:27:16.761 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_15_00_UTC 2023-10-16T13:27:16.761 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_16_00_UTC 2023-10-16T13:27:16.761 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_17_00_UTC 2023-10-16T13:27:16.761 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_18_00_UTC 2023-10-16T13:27:16.761 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_19_00_UTC 2023-10-16T13:27:16.762 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_20_00_UTC 2023-10-16T13:27:16.762 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_21_00_UTC 2023-10-16T13:27:16.762 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_22_00_UTC 2023-10-16T13:27:16.762 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_23_00_UTC 2023-10-16T13:27:16.762 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_24_00_UTC 2023-10-16T13:27:16.763 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_25_00_UTC 2023-10-16T13:27:16.763 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_26_00_UTC 2023-10-16T13:27:16.763 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_27_00_UTC 2023-10-16T13:27:16.763 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-12_33_00_UTC 2023-10-16T13:27:16.764 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:27:16.764 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-12_33_00_UTC) 2023-10-16T13:27:16.815 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-12_34_00_UTC 2023-10-16T13:27:16.815 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:27:16.816 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-12_34_00_UTC) 2023-10-16T13:27:16.879 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-12_35_00_UTC 2023-10-16T13:27:16.879 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:27:16.880 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-12_35_00_UTC) 2023-10-16T13:27:16.943 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-12_36_00_UTC 2023-10-16T13:27:16.943 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:27:16.943 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-12_36_00_UTC) 2023-10-16T13:27:17.007 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-12_37_00_UTC 2023-10-16T13:27:17.008 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:27:17.008 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-12_37_00_UTC) 2023-10-16T13:27:17.076 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-12_38_00_UTC 2023-10-16T13:27:17.076 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:27:17.076 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-12_38_00_UTC) 2023-10-16T13:27:17.139 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-12_39_00_UTC 2023-10-16T13:27:17.139 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:27:17.140 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-12_39_00_UTC) 2023-10-16T13:27:17.201 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-12_40_00_UTC 2023-10-16T13:27:17.202 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:27:17.202 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-12_40_00_UTC) 2023-10-16T13:27:17.265 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-12_41_00_UTC 2023-10-16T13:27:17.266 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:27:17.266 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-12_41_00_UTC) 2023-10-16T13:27:17.328 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-12_42_00_UTC 2023-10-16T13:27:17.328 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:27:17.328 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-12_42_00_UTC) 2023-10-16T13:27:17.391 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-12_43_00_UTC 2023-10-16T13:27:17.391 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:27:17.392 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-12_43_00_UTC) 2023-10-16T13:27:17.454 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-12_44_00_UTC 2023-10-16T13:27:17.455 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:27:17.455 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-12_44_00_UTC) 2023-10-16T13:27:17.517 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-12_45_00_UTC 2023-10-16T13:27:17.517 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:27:17.518 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-12_45_00_UTC) 2023-10-16T13:27:17.580 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-12_46_00_UTC 2023-10-16T13:27:17.581 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:27:17.581 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-12_46_00_UTC) 2023-10-16T13:27:17.643 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-12_47_00_UTC 2023-10-16T13:27:17.643 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:27:17.643 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-12_47_00_UTC) 2023-10-16T13:27:17.706 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-12_48_00_UTC 2023-10-16T13:27:17.706 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:27:17.706 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-12_48_00_UTC) 2023-10-16T13:27:17.769 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-12_49_00_UTC 2023-10-16T13:27:17.769 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:27:17.770 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-12_49_00_UTC) 2023-10-16T13:27:17.832 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-12_50_00_UTC 2023-10-16T13:27:17.832 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:27:17.833 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-12_50_00_UTC) 2023-10-16T13:27:17.895 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-12_51_00_UTC 2023-10-16T13:27:17.895 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:27:17.896 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-12_51_00_UTC) 2023-10-16T13:27:17.958 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-12_52_00_UTC 2023-10-16T13:27:17.958 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:27:17.959 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-12_52_00_UTC) 2023-10-16T13:27:18.021 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-12_53_00_UTC 2023-10-16T13:27:18.021 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:27:18.022 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-12_53_00_UTC) 2023-10-16T13:27:18.094 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-12_54_00_UTC 2023-10-16T13:27:18.094 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:27:18.095 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-12_54_00_UTC) 2023-10-16T13:27:18.156 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-12_55_00_UTC 2023-10-16T13:27:18.157 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:27:18.157 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-12_55_00_UTC) 2023-10-16T13:27:18.219 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-12_56_00_UTC 2023-10-16T13:27:18.220 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:27:18.220 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-12_56_00_UTC) 2023-10-16T13:27:18.281 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-12_57_00_UTC 2023-10-16T13:27:18.282 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:27:18.282 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-12_57_00_UTC) 2023-10-16T13:27:18.344 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-12_58_00_UTC 2023-10-16T13:27:18.344 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:27:18.345 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-12_58_00_UTC) 2023-10-16T13:27:18.406 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-12_59_00_UTC 2023-10-16T13:27:18.407 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:27:18.407 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-12_59_00_UTC) 2023-10-16T13:27:18.470 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_00_00_UTC 2023-10-16T13:27:18.470 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:27:18.471 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_00_00_UTC) 2023-10-16T13:27:18.533 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_01_00_UTC 2023-10-16T13:27:18.533 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:27:18.533 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_01_00_UTC) 2023-10-16T13:27:18.596 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_02_00_UTC 2023-10-16T13:27:18.596 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:27:18.597 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_02_00_UTC) 2023-10-16T13:27:18.658 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_03_00_UTC 2023-10-16T13:27:18.659 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:27:18.659 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_03_00_UTC) 2023-10-16T13:27:18.723 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_04_00_UTC 2023-10-16T13:27:18.723 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:27:18.724 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_04_00_UTC) 2023-10-16T13:27:18.785 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_05_00_UTC 2023-10-16T13:27:18.786 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:27:18.786 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_05_00_UTC) 2023-10-16T13:27:18.850 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_06_00_UTC 2023-10-16T13:27:18.851 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:27:18.851 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_06_00_UTC) 2023-10-16T13:27:18.915 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_07_00_UTC 2023-10-16T13:27:18.916 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:27:18.916 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_07_00_UTC) 2023-10-16T13:27:18.979 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_08_00_UTC 2023-10-16T13:27:18.979 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:27:18.980 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_08_00_UTC) 2023-10-16T13:27:19.042 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_09_00_UTC 2023-10-16T13:27:19.042 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:27:19.043 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_09_00_UTC) 2023-10-16T13:27:19.138 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_10_00_UTC 2023-10-16T13:27:19.138 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:27:19.139 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_10_00_UTC) 2023-10-16T13:27:19.173 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_11_00_UTC 2023-10-16T13:27:19.174 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:27:19.174 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_11_00_UTC) 2023-10-16T13:27:19.245 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_12_00_UTC 2023-10-16T13:27:19.246 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:27:19.246 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_12_00_UTC) 2023-10-16T13:27:19.308 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_13_00_UTC 2023-10-16T13:27:19.309 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:27:19.309 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_13_00_UTC) 2023-10-16T13:27:19.371 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_14_00_UTC 2023-10-16T13:27:19.371 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:27:19.372 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_14_00_UTC) 2023-10-16T13:27:19.434 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_15_00_UTC 2023-10-16T13:27:19.435 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:27:19.435 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_15_00_UTC) 2023-10-16T13:27:19.499 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_16_00_UTC 2023-10-16T13:27:19.500 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:27:19.500 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_16_00_UTC) 2023-10-16T13:27:19.561 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_17_00_UTC 2023-10-16T13:27:19.562 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:27:19.562 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_17_00_UTC) 2023-10-16T13:27:19.623 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_18_00_UTC 2023-10-16T13:27:19.624 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:27:19.624 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_18_00_UTC) 2023-10-16T13:27:19.685 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_19_00_UTC 2023-10-16T13:27:19.685 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:27:19.686 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_19_00_UTC) 2023-10-16T13:27:19.747 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_20_00_UTC 2023-10-16T13:27:19.747 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:27:19.747 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_20_00_UTC) 2023-10-16T13:27:19.809 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_21_00_UTC 2023-10-16T13:27:19.810 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:27:19.810 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_21_00_UTC) 2023-10-16T13:27:19.871 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_22_00_UTC 2023-10-16T13:27:19.872 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:27:19.872 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_22_00_UTC) 2023-10-16T13:27:19.933 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_23_00_UTC 2023-10-16T13:27:19.934 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:27:19.935 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_23_00_UTC) 2023-10-16T13:27:19.996 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_24_00_UTC 2023-10-16T13:27:19.997 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:27:19.997 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_24_00_UTC) 2023-10-16T13:27:20.060 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_25_00_UTC 2023-10-16T13:27:20.060 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:27:20.061 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_25_00_UTC) 2023-10-16T13:27:20.135 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_26_00_UTC 2023-10-16T13:27:20.135 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:27:20.136 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_26_00_UTC) 2023-10-16T13:27:20.198 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_27_00_UTC 2023-10-16T13:27:20.199 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:27:20.199 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_27_00_UTC) 2023-10-16T13:27:20.261 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:27:20.261 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2023-10-16T13:27:20.321 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-16T13:27:20.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:20.506+0000 7f4ef6fd0700 1 Processor -- start 2023-10-16T13:27:20.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:20.506+0000 7f4ef6fd0700 1 -- start start 2023-10-16T13:27:20.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:20.507+0000 7f4ef6fd0700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4ef00d4060 0x7f4ef00d1520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:20.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:20.507+0000 7f4ef6fd0700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4ef00d1d20 0x7f4ef00d21b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:20.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:20.507+0000 7f4ef6fd0700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ef00d4540 con 0x7f4ef015da10 2023-10-16T13:27:20.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:20.507+0000 7f4ef6fd0700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ef00cf9b0 con 0x7f4ef00d4060 2023-10-16T13:27:20.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:20.507+0000 7f4ef6fd0700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef00d26f0 con 0x7f4ef00d1d20 2023-10-16T13:27:20.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:20.508+0000 7f4eeffff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4ef00d4060 0x7f4ef00d1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:20.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:20.508+0000 7f4eeffff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4ef00d4060 0x7f4ef00d1520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47586/0 (socket says 172.21.15.16:47586) 2023-10-16T13:27:20.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:20.508+0000 7f4eeffff700 1 -- 172.21.15.16:0/1798183859 learned_addr learned my addr 172.21.15.16:0/1798183859 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:20.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:20.508+0000 7f4ef556d700 1 --2- 172.21.15.16:0/1798183859 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4ef00d1d20 0x7f4ef00d21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:20.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:20.508+0000 7f4eeffff700 1 -- 172.21.15.16:0/1798183859 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4ef00d1d20 msgr2=0x7f4ef00d21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:20.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:20.508+0000 7f4eeffff700 1 --2- 172.21.15.16:0/1798183859 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4ef00d1d20 0x7f4ef00d21b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:20.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:20.508+0000 7f4eeffff700 1 -- 172.21.15.16:0/1798183859 >> v1:172.21.15.49:6789/0 conn(0x7f4ef015da10 legacy=0x7f4ef00cf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:20.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:20.508+0000 7f4eeffff700 1 -- 172.21.15.16:0/1798183859 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ef00ce7e0 con 0x7f4ef00d4060 2023-10-16T13:27:20.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:20.508+0000 7f4eeffff700 1 --2- 172.21.15.16:0/1798183859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ef00d4060 0x7f4ef00d1520 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7f4ed80098f0 tx=0x7f4ed8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7120dde4853f8db server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:20.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:20.508+0000 7f4eef7fe700 1 -- 172.21.15.16:0/1798183859 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ed800b8a0 con 0x7f4ef00d4060 2023-10-16T13:27:20.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:20.508+0000 7f4eef7fe700 1 -- 172.21.15.16:0/1798183859 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ed800ba00 con 0x7f4ef00d4060 2023-10-16T13:27:20.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:20.508+0000 7f4eef7fe700 1 -- 172.21.15.16:0/1798183859 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ed800bd90 con 0x7f4ef00d4060 2023-10-16T13:27:20.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:20.509+0000 7f4ef6fd0700 1 -- 172.21.15.16:0/1798183859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ef00d4060 msgr2=0x7f4ef00d1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:20.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:20.509+0000 7f4ef6fd0700 1 --2- 172.21.15.16:0/1798183859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ef00d4060 0x7f4ef00d1520 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7f4ed80098f0 tx=0x7f4ed8009c00 comp rx=0 tx=0).stop 2023-10-16T13:27:20.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:20.509+0000 7f4ef6fd0700 1 -- 172.21.15.16:0/1798183859 shutdown_connections 2023-10-16T13:27:20.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:20.509+0000 7f4ef6fd0700 1 --2- 172.21.15.16:0/1798183859 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4ef00d1d20 0x7f4ef00d21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:20.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:20.509+0000 7f4ef6fd0700 1 --2- 172.21.15.16:0/1798183859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ef00d4060 0x7f4ef00d1520 unknown :-1 s=CLOSED pgs=1803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:20.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:20.509+0000 7f4ef6fd0700 1 -- 172.21.15.16:0/1798183859 >> 172.21.15.16:0/1798183859 conn(0x7f4ef009c9f0 msgr2=0x7f4ef00d3740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:20.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:20.509+0000 7f4ef6fd0700 1 -- 172.21.15.16:0/1798183859 shutdown_connections 2023-10-16T13:27:20.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:20.509+0000 7f4ef6fd0700 1 -- 172.21.15.16:0/1798183859 wait complete. 2023-10-16T13:27:20.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:20.510+0000 7f4ef6fd0700 1 Processor -- start 2023-10-16T13:27:20.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:20.510+0000 7f4ef6fd0700 1 -- start start 2023-10-16T13:27:20.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:20.511+0000 7f4ef6fd0700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4ef00d1d20 0x7f4ef0157a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:20.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:20.511+0000 7f4ef6fd0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ef00d4060 0x7f4ef015c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:20.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:20.511+0000 7f4ef6fd0700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4ef015da10 0x7f4ef015c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:20.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:20.511+0000 7f4ef6fd0700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef00cc480 con 0x7f4ef00d4060 2023-10-16T13:27:20.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:20.511+0000 7f4ef6fd0700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef01f6890 con 0x7f4ef00d1d20 2023-10-16T13:27:20.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:20.511+0000 7f4ef6fd0700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef01f6b70 con 0x7f4ef015da10 2023-10-16T13:27:20.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:20.511+0000 7f4eeffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ef00d4060 0x7f4ef015c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:20.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:20.511+0000 7f4eeffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ef00d4060 0x7f4ef015c120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47592/0 (socket says 172.21.15.16:47592) 2023-10-16T13:27:20.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:20.511+0000 7f4ef4d6c700 1 --2- 172.21.15.16:0/3365383235 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4ef00d1d20 0x7f4ef0157a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:20.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:20.511+0000 7f4eeffff700 1 -- 172.21.15.16:0/3365383235 learned_addr learned my addr 172.21.15.16:0/3365383235 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:20.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:20.511+0000 7f4ef556d700 1 --2- 172.21.15.16:0/3365383235 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4ef015da10 0x7f4ef015c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:20.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:20.511+0000 7f4eeffff700 1 -- 172.21.15.16:0/3365383235 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4ef015da10 msgr2=0x7f4ef015c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:20.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:20.511+0000 7f4eeffff700 1 --2- 172.21.15.16:0/3365383235 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4ef015da10 0x7f4ef015c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:20.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:20.511+0000 7f4eeffff700 1 -- 172.21.15.16:0/3365383235 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4ef00d1d20 msgr2=0x7f4ef0157a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:20.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:20.511+0000 7f4eeffff700 1 --2- 172.21.15.16:0/3365383235 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4ef00d1d20 0x7f4ef0157a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:20.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:20.511+0000 7f4eeffff700 1 -- 172.21.15.16:0/3365383235 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ed8009550 con 0x7f4ef00d4060 2023-10-16T13:27:20.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:20.512+0000 7f4eeffff700 1 --2- 172.21.15.16:0/3365383235 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ef00d4060 0x7f4ef015c120 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7f4ed8000c00 tx=0x7f4ed8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:20.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:20.512+0000 7f4eed7fa700 1 -- 172.21.15.16:0/3365383235 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ed8004300 con 0x7f4ef00d4060 2023-10-16T13:27:20.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:20.512+0000 7f4eed7fa700 1 -- 172.21.15.16:0/3365383235 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ed801f070 con 0x7f4ef00d4060 2023-10-16T13:27:20.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:20.512+0000 7f4ef6fd0700 1 -- 172.21.15.16:0/3365383235 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ef01f6e50 con 0x7f4ef00d4060 2023-10-16T13:27:20.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:20.512+0000 7f4eed7fa700 1 -- 172.21.15.16:0/3365383235 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ed80036a0 con 0x7f4ef00d4060 2023-10-16T13:27:20.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:20.512+0000 7f4ef6fd0700 1 -- 172.21.15.16:0/3365383235 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ef01f7410 con 0x7f4ef00d4060 2023-10-16T13:27:20.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:20.513+0000 7f4ef6fd0700 1 -- 172.21.15.16:0/3365383235 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ef004ebb0 con 0x7f4ef00d4060 2023-10-16T13:27:20.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:20.513+0000 7f4eed7fa700 1 -- 172.21.15.16:0/3365383235 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 93) v1 ==== 89360+0+0 (secure 0 0 0) 0x7f4ed8021030 con 0x7f4ef00d4060 2023-10-16T13:27:20.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:20.514+0000 7f4eed7fa700 1 --2- 172.21.15.16:0/3365383235 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f4ed006b070 0x7f4ed006d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:20.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:20.514+0000 7f4eed7fa700 1 -- 172.21.15.16:0/3365383235 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f4ed808ae10 con 0x7f4ef00d4060 2023-10-16T13:27:20.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:20.514+0000 7f4ef4d6c700 1 --2- 172.21.15.16:0/3365383235 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f4ed006b070 0x7f4ed006d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:20.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:20.514+0000 7f4ef4d6c700 1 --2- 172.21.15.16:0/3365383235 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f4ed006b070 0x7f4ed006d530 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4ee8001330 tx=0x7f4ee8009040 comp rx=0 tx=0).ready entity=mgr.8770 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:20.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:20.518+0000 7f4eed7fa700 1 -- 172.21.15.16:0/3365383235 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4ed8055a90 con 0x7f4ef00d4060 2023-10-16T13:27:20.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:20.803+0000 7f4ef6fd0700 1 -- 172.21.15.16:0/3365383235 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f4ef00d1b00 con 0x7f4ef00d4060 2023-10-16T13:27:21.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.077+0000 7f4eed7fa700 1 -- 172.21.15.16:0/3365383235 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v94) v1 ==== 93+0+0 (secure 0 0 0) 0x7f4ed80590b0 con 0x7f4ef00d4060 2023-10-16T13:27:21.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.082+0000 7f4ef6fd0700 1 -- 172.21.15.16:0/3365383235 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f4ed006b070 msgr2=0x7f4ed006d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:21.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.082+0000 7f4ef6fd0700 1 --2- 172.21.15.16:0/3365383235 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f4ed006b070 0x7f4ed006d530 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4ee8001330 tx=0x7f4ee8009040 comp rx=0 tx=0).stop 2023-10-16T13:27:21.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.082+0000 7f4ef6fd0700 1 -- 172.21.15.16:0/3365383235 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ef00d4060 msgr2=0x7f4ef015c120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:21.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.083+0000 7f4ef6fd0700 1 --2- 172.21.15.16:0/3365383235 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ef00d4060 0x7f4ef015c120 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7f4ed8000c00 tx=0x7f4ed8004750 comp rx=0 tx=0).stop 2023-10-16T13:27:21.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.084+0000 7f4ef6fd0700 1 -- 172.21.15.16:0/3365383235 shutdown_connections 2023-10-16T13:27:21.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.084+0000 7f4ef6fd0700 1 --2- 172.21.15.16:0/3365383235 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f4ed006b070 0x7f4ed006d530 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:21.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.084+0000 7f4ef6fd0700 1 --2- 172.21.15.16:0/3365383235 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4ef015da10 0x7f4ef015c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:21.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.084+0000 7f4ef6fd0700 1 --2- 172.21.15.16:0/3365383235 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4ef00d1d20 0x7f4ef0157a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:21.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.084+0000 7f4ef6fd0700 1 --2- 172.21.15.16:0/3365383235 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ef00d4060 0x7f4ef015c120 unknown :-1 s=CLOSED pgs=1804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:21.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.084+0000 7f4ef6fd0700 1 -- 172.21.15.16:0/3365383235 >> 172.21.15.16:0/3365383235 conn(0x7f4ef009c9f0 msgr2=0x7f4ef00d0330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:21.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.092+0000 7f4ef6fd0700 1 -- 172.21.15.16:0/3365383235 shutdown_connections 2023-10-16T13:27:21.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.092+0000 7f4ef6fd0700 1 -- 172.21.15.16:0/3365383235 wait complete. 2023-10-16T13:27:21.115 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T13:27:21.116 DEBUG:teuthology.orchestra.run.smithi016:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T13:27:21.165 DEBUG:teuthology.orchestra.run.smithi049:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T13:27:21.212 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T13:27:21.213 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T13:27:21.213 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:27:21.231 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T13:27:21.232 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:27:21.246 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:27:21.244+0000 7f52d4a73200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T13:27:21.248 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:27:21.246+0000 7fbf61b58200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T13:27:21.264 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:27:21.319 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:27:21.316+0000 7fbf61b58200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T13:27:21.325 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:27:21.322+0000 7f52d4a73200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T13:27:21.398 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:27:21.395+0000 7fbf61b58200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T13:27:21.424 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:27:21.422+0000 7f52d4a73200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T13:27:21.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.466+0000 7fa14e754700 1 Processor -- start 2023-10-16T13:27:21.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.466+0000 7fa14e754700 1 -- start start 2023-10-16T13:27:21.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.467+0000 7fa14e754700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa1480b2be0 0x7fa1480b3b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:21.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.467+0000 7fa14e754700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1480d11c0 con 0x7fa14806aef0 2023-10-16T13:27:21.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.467+0000 7fa14e754700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1480cd360 con 0x7fa14815b860 2023-10-16T13:27:21.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.467+0000 7fa14e754700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1480d03d0 con 0x7fa1480b2be0 2023-10-16T13:27:21.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.467+0000 7fa14ccf1700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa1480b2be0 0x7fa1480b3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:21.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.467+0000 7fa14ccf1700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa1480b2be0 0x7fa1480b3b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47656/0 (socket says 172.21.15.16:47656) 2023-10-16T13:27:21.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.467+0000 7fa14ccf1700 1 -- 172.21.15.16:0/474783346 learned_addr learned my addr 172.21.15.16:0/474783346 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:21.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.467+0000 7fa1477fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa14806aef0 0x7fa1480b3480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36926/0 (socket says 172.21.15.16:36926) 2023-10-16T13:27:21.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.467+0000 7fa14ccf1700 1 -- 172.21.15.16:0/474783346 >> v1:172.21.15.49:6789/0 conn(0x7fa14815b860 legacy=0x7fa14806a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:21.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.467+0000 7fa14ccf1700 1 -- 172.21.15.16:0/474783346 >> v1:172.21.15.16:6789/0 conn(0x7fa14806aef0 legacy=0x7fa1480b3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:21.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.467+0000 7fa14ccf1700 1 -- 172.21.15.16:0/474783346 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1480d05b0 con 0x7fa1480b2be0 2023-10-16T13:27:21.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.467+0000 7fa14ccf1700 1 --2- 172.21.15.16:0/474783346 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1480b2be0 0x7fa1480b3b90 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7fa140009e50 tx=0x7fa14000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=995635487031e949 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:21.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.467+0000 7fa146ffd700 1 -- 172.21.15.16:0/474783346 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa14000e070 con 0x7fa1480b2be0 2023-10-16T13:27:21.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.468+0000 7fa146ffd700 1 -- 172.21.15.16:0/474783346 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa14000b9c0 con 0x7fa1480b2be0 2023-10-16T13:27:21.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.468+0000 7fa146ffd700 1 -- 172.21.15.16:0/474783346 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa14000bd70 con 0x7fa1480b2be0 2023-10-16T13:27:21.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.468+0000 7fa14e754700 1 -- 172.21.15.16:0/474783346 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1480b2be0 msgr2=0x7fa1480b3b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:21.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.468+0000 7fa14e754700 1 --2- 172.21.15.16:0/474783346 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1480b2be0 0x7fa1480b3b90 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7fa140009e50 tx=0x7fa14000b330 comp rx=0 tx=0).stop 2023-10-16T13:27:21.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.468+0000 7fa14e754700 1 -- 172.21.15.16:0/474783346 shutdown_connections 2023-10-16T13:27:21.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.468+0000 7fa14e754700 1 --2- 172.21.15.16:0/474783346 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1480b2be0 0x7fa1480b3b90 unknown :-1 s=CLOSED pgs=1810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:21.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.468+0000 7fa14e754700 1 -- 172.21.15.16:0/474783346 >> 172.21.15.16:0/474783346 conn(0x7fa1480b1fd0 msgr2=0x7fa1480b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:21.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.468+0000 7fa14e754700 1 -- 172.21.15.16:0/474783346 shutdown_connections 2023-10-16T13:27:21.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.468+0000 7fa14e754700 1 -- 172.21.15.16:0/474783346 wait complete. 2023-10-16T13:27:21.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.469+0000 7fa14e754700 1 Processor -- start 2023-10-16T13:27:21.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.470+0000 7fa14e754700 1 -- start start 2023-10-16T13:27:21.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.470+0000 7fa14e754700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa14806aef0 0x7fa14814cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:21.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.470+0000 7fa14e754700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa1480b2be0 0x7fa14814d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:21.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.470+0000 7fa14e754700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa14815b860 0x7fa1480b05a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:21.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.470+0000 7fa14e754700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1480c5300 con 0x7fa14815b860 2023-10-16T13:27:21.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.470+0000 7fa14e754700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1480c5510 con 0x7fa14806aef0 2023-10-16T13:27:21.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.470+0000 7fa14e754700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1480c57f0 con 0x7fa1480b2be0 2023-10-16T13:27:21.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.470+0000 7fa14ccf1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa14815b860 0x7fa1480b05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:21.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.471+0000 7fa14ccf1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa14815b860 0x7fa1480b05a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47662/0 (socket says 172.21.15.16:47662) 2023-10-16T13:27:21.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.471+0000 7fa14ccf1700 1 -- 172.21.15.16:0/3492275539 learned_addr learned my addr 172.21.15.16:0/3492275539 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:21.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.471+0000 7fa147fff700 1 --2- 172.21.15.16:0/3492275539 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa14806aef0 0x7fa14814cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:21.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.471+0000 7fa1477fe700 1 --2- 172.21.15.16:0/3492275539 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa1480b2be0 0x7fa14814d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:21.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.471+0000 7fa14ccf1700 1 -- 172.21.15.16:0/3492275539 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa1480b2be0 msgr2=0x7fa14814d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:21.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.471+0000 7fa14ccf1700 1 --2- 172.21.15.16:0/3492275539 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa1480b2be0 0x7fa14814d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:21.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.471+0000 7fa14ccf1700 1 -- 172.21.15.16:0/3492275539 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa14806aef0 msgr2=0x7fa14814cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:21.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.471+0000 7fa14ccf1700 1 --2- 172.21.15.16:0/3492275539 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa14806aef0 0x7fa14814cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:21.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.471+0000 7fa14ccf1700 1 -- 172.21.15.16:0/3492275539 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa140009b30 con 0x7fa14815b860 2023-10-16T13:27:21.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.471+0000 7fa14ccf1700 1 --2- 172.21.15.16:0/3492275539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa14815b860 0x7fa1480b05a0 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7fa140012000 tx=0x7fa140004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:21.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.471+0000 7fa144ff9700 1 -- 172.21.15.16:0/3492275539 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa140004840 con 0x7fa14815b860 2023-10-16T13:27:21.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.471+0000 7fa144ff9700 1 -- 172.21.15.16:0/3492275539 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1400099b0 con 0x7fa14815b860 2023-10-16T13:27:21.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.471+0000 7fa14e754700 1 -- 172.21.15.16:0/3492275539 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1480c5ad0 con 0x7fa14815b860 2023-10-16T13:27:21.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.471+0000 7fa144ff9700 1 -- 172.21.15.16:0/3492275539 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa14001c680 con 0x7fa14815b860 2023-10-16T13:27:21.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.471+0000 7fa14e754700 1 -- 172.21.15.16:0/3492275539 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1480c62d0 con 0x7fa14815b860 2023-10-16T13:27:21.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.472+0000 7fa14e754700 1 -- 172.21.15.16:0/3492275539 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1480c5f90 con 0x7fa14815b860 2023-10-16T13:27:21.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.473+0000 7fa144ff9700 1 -- 172.21.15.16:0/3492275539 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 94) v1 ==== 89343+0+0 (secure 0 0 0) 0x7fa14001c7e0 con 0x7fa14815b860 2023-10-16T13:27:21.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.473+0000 7fa144ff9700 1 --2- 172.21.15.16:0/3492275539 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7fa13406b020 0x7fa13406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:21.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.473+0000 7fa147fff700 1 -- 172.21.15.16:0/3492275539 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7fa13406b020 msgr2=0x7fa13406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1503042987 2023-10-16T13:27:21.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.473+0000 7fa144ff9700 1 -- 172.21.15.16:0/3492275539 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 6517+0+0 (secure 0 0 0) 0x7fa14008b310 con 0x7fa14815b860 2023-10-16T13:27:21.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.474+0000 7fa147fff700 1 --2- 172.21.15.16:0/3492275539 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7fa13406b020 0x7fa13406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:27:21.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.478+0000 7fa144ff9700 1 -- 172.21.15.16:0/3492275539 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa140055dd0 con 0x7fa14815b860 2023-10-16T13:27:21.488 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:27:21.486+0000 7fbf61b58200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T13:27:21.524 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:27:21.522+0000 7f52d4a73200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T13:27:21.649 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:27:21.647+0000 7fbf61b58200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T13:27:21.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.675+0000 7fa147fff700 1 -- 172.21.15.16:0/3492275539 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7fa13406b020 msgr2=0x7fa13406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1503042987 2023-10-16T13:27:21.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.675+0000 7fa147fff700 1 --2- 172.21.15.16:0/3492275539 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7fa13406b020 0x7fa13406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:27:21.680 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:27:21.677+0000 7f52d4a73200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T13:27:21.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.695+0000 7fa14e754700 1 -- 172.21.15.16:0/3492275539 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa14804ebb0 con 0x7fa14815b860 2023-10-16T13:27:21.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.697+0000 7fa144ff9700 1 -- 172.21.15.16:0/3492275539 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v163) v1 ==== 74+0+17380 (secure 0 0 0) 0x7fa140088400 con 0x7fa14815b860 2023-10-16T13:27:21.700 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:27:21.700 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":163,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:27:21.069543+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:31:04.290489+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2023-10-16T12:31:04.803486+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"163","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":112,"snap_epoch":163,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/3195044045":"2023-10-17T12:31:20.854761+0000","172.21.15.16:6833/4182163087":"2023-10-17T12:31:20.854761+0000","172.21.15.16:6832/4182163087":"2023-10-17T12:31:20.854761+0000","172.21.15.16:0/4217694331":"2023-10-17T12:31:20.854761+0000","172.21.15.16:0/2516829556":"2023-10-17T12:31:20.854761+0000","172.21.15.16:0/2546955595":"2023-10-17T12:31:20.854761+0000","172.21.15.16:0/1851287351":"2023-10-17T12:31:20.854761+0000","172.21.15.16:0/3755256116":"2023-10-17T12:31:20.854761+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":54},{"begin":59,"length":54}]}],"new_removed_snaps":[{"pool":20,"snaps":[{"begin":38,"length":19},{"begin":94,"length":19}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:27:21.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.699+0000 7fa14e754700 1 -- 172.21.15.16:0/3492275539 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7fa13406b020 msgr2=0x7fa13406d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:27:21.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.699+0000 7fa14e754700 1 --2- 172.21.15.16:0/3492275539 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7fa13406b020 0x7fa13406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:21.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.699+0000 7fa14e754700 1 -- 172.21.15.16:0/3492275539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa14815b860 msgr2=0x7fa1480b05a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:21.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.699+0000 7fa14e754700 1 --2- 172.21.15.16:0/3492275539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa14815b860 0x7fa1480b05a0 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7fa140012000 tx=0x7fa140004ad0 comp rx=0 tx=0).stop 2023-10-16T13:27:21.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.699+0000 7fa14e754700 1 -- 172.21.15.16:0/3492275539 shutdown_connections 2023-10-16T13:27:21.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.699+0000 7fa14e754700 1 --2- 172.21.15.16:0/3492275539 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7fa13406b020 0x7fa13406d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:21.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.699+0000 7fa14e754700 1 --2- 172.21.15.16:0/3492275539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa14815b860 0x7fa1480b05a0 unknown :-1 s=CLOSED pgs=1811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:21.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.699+0000 7fa14e754700 1 --2- 172.21.15.16:0/3492275539 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa14806aef0 0x7fa14814cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:21.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.699+0000 7fa14e754700 1 --2- 172.21.15.16:0/3492275539 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa1480b2be0 0x7fa14814d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:21.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.699+0000 7fa14e754700 1 -- 172.21.15.16:0/3492275539 >> 172.21.15.16:0/3492275539 conn(0x7fa1480b1fd0 msgr2=0x7fa148159860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:21.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.699+0000 7fa14e754700 1 -- 172.21.15.16:0/3492275539 shutdown_connections 2023-10-16T13:27:21.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.699+0000 7fa14e754700 1 -- 172.21.15.16:0/3492275539 wait complete. 2023-10-16T13:27:21.715 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 19, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T12:31:04.290489+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '155', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 20, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T12:31:04.803486+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '163', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 112, 'snap_epoch': 163, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T13:27:21.716 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T13:27:21.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.896+0000 7fded4787700 1 Processor -- start 2023-10-16T13:27:21.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.896+0000 7fded4787700 1 -- start start 2023-10-16T13:27:21.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.897+0000 7fded4787700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdecc15f060 0x7fdecc0cc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:21.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.897+0000 7fded4787700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdecc0b2440 con 0x7fdecc0ce5d0 2023-10-16T13:27:21.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.897+0000 7fded4787700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdecc0d00e0 con 0x7fdecc15ec80 2023-10-16T13:27:21.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.897+0000 7fded4787700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdecc0b2c50 con 0x7fdecc15f060 2023-10-16T13:27:21.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.898+0000 7fded1d22700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fdecc0ce5d0 0x7fdecc0cc190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36970/0 (socket says 172.21.15.16:36970) 2023-10-16T13:27:21.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.898+0000 7fded1d22700 1 -- 172.21.15.16:0/457156731 learned_addr learned my addr 172.21.15.16:0/457156731 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:27:21.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.898+0000 7fded2d24700 1 --2- 172.21.15.16:0/457156731 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdecc15f060 0x7fdecc0cc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:21.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.898+0000 7fded1521700 1 -- 172.21.15.16:0/457156731 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3119535277 0 0) 0x7fdecc0b2440 con 0x7fdecc0ce5d0 2023-10-16T13:27:21.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.898+0000 7fded1521700 1 -- 172.21.15.16:0/457156731 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdeb80035b0 con 0x7fdecc0ce5d0 2023-10-16T13:27:21.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.898+0000 7fded1521700 1 -- 172.21.15.16:0/457156731 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4250459116 0 0) 0x7fdecc0d00e0 con 0x7fdecc15ec80 2023-10-16T13:27:21.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.898+0000 7fded1521700 1 -- 172.21.15.16:0/457156731 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdecc0b2440 con 0x7fdecc15ec80 2023-10-16T13:27:21.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.898+0000 7fded2d24700 1 -- 172.21.15.16:0/457156731 >> v1:172.21.15.49:6789/0 conn(0x7fdecc15ec80 legacy=0x7fdecc0cdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:21.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.898+0000 7fded2d24700 1 -- 172.21.15.16:0/457156731 >> v1:172.21.15.16:6789/0 conn(0x7fdecc0ce5d0 legacy=0x7fdecc0cc190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:21.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.898+0000 7fded2d24700 1 -- 172.21.15.16:0/457156731 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdecc0cef60 con 0x7fdecc15f060 2023-10-16T13:27:21.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.898+0000 7fded2d24700 1 --2- 172.21.15.16:0/457156731 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdecc15f060 0x7fdecc0cc8a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fdec4009dc0 tx=0x7fdec400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dffcd1039e0ced67 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:21.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.898+0000 7fded1521700 1 -- 172.21.15.16:0/457156731 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdec400e070 con 0x7fdecc15f060 2023-10-16T13:27:21.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.898+0000 7fded1521700 1 -- 172.21.15.16:0/457156731 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdec400bb20 con 0x7fdecc15f060 2023-10-16T13:27:21.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.898+0000 7fded1521700 1 -- 172.21.15.16:0/457156731 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdec4012430 con 0x7fdecc15f060 2023-10-16T13:27:21.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.899+0000 7fded4787700 1 -- 172.21.15.16:0/457156731 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdecc15f060 msgr2=0x7fdecc0cc8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:21.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.899+0000 7fded4787700 1 --2- 172.21.15.16:0/457156731 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdecc15f060 0x7fdecc0cc8a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fdec4009dc0 tx=0x7fdec400b330 comp rx=0 tx=0).stop 2023-10-16T13:27:21.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.899+0000 7fded4787700 1 -- 172.21.15.16:0/457156731 shutdown_connections 2023-10-16T13:27:21.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.899+0000 7fded4787700 1 --2- 172.21.15.16:0/457156731 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdecc15f060 0x7fdecc0cc8a0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:21.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.899+0000 7fded4787700 1 -- 172.21.15.16:0/457156731 >> 172.21.15.16:0/457156731 conn(0x7fdecc0d15f0 msgr2=0x7fdecc0d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:21.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.899+0000 7fded4787700 1 -- 172.21.15.16:0/457156731 shutdown_connections 2023-10-16T13:27:21.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.899+0000 7fded4787700 1 -- 172.21.15.16:0/457156731 wait complete. 2023-10-16T13:27:21.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.900+0000 7fded4787700 1 Processor -- start 2023-10-16T13:27:21.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.900+0000 7fded4787700 1 -- start start 2023-10-16T13:27:21.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.901+0000 7fded4787700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdecc0ce5d0 0x7fdecc0c86c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:21.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.901+0000 7fded4787700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdecc15ec80 0x7fdecc06f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:21.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.901+0000 7fded4787700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdecc15f060 0x7fdecc06f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:21.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.901+0000 7fded4787700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdecc1e5090 con 0x7fdecc15ec80 2023-10-16T13:27:21.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.901+0000 7fded4787700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdecc1e41a0 con 0x7fdecc0ce5d0 2023-10-16T13:27:21.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.901+0000 7fded4787700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdecc1e4480 con 0x7fdecc15f060 2023-10-16T13:27:21.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.901+0000 7fded1d22700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdecc15ec80 0x7fdecc06f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:21.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.901+0000 7fded1d22700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdecc15ec80 0x7fdecc06f160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47702/0 (socket says 172.21.15.16:47702) 2023-10-16T13:27:21.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.901+0000 7fded1d22700 1 -- 172.21.15.16:0/521530428 learned_addr learned my addr 172.21.15.16:0/521530428 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:21.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.901+0000 7fded2d24700 1 --2- 172.21.15.16:0/521530428 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdecc15f060 0x7fdecc06f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:21.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.901+0000 7fded2523700 1 --2- 172.21.15.16:0/521530428 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdecc0ce5d0 0x7fdecc0c86c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:21.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.901+0000 7fded1d22700 1 -- 172.21.15.16:0/521530428 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdecc15f060 msgr2=0x7fdecc06f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:21.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.901+0000 7fded1d22700 1 --2- 172.21.15.16:0/521530428 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdecc15f060 0x7fdecc06f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:21.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.901+0000 7fded1d22700 1 -- 172.21.15.16:0/521530428 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdecc0ce5d0 msgr2=0x7fdecc0c86c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:21.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.901+0000 7fded1d22700 1 --2- 172.21.15.16:0/521530428 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdecc0ce5d0 0x7fdecc0c86c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:21.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.901+0000 7fded1d22700 1 -- 172.21.15.16:0/521530428 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdec4009a70 con 0x7fdecc15ec80 2023-10-16T13:27:21.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.902+0000 7fded1d22700 1 --2- 172.21.15.16:0/521530428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdecc15ec80 0x7fdecc06f160 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7fdebc001eb0 tx=0x7fdebc00c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:21.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.902+0000 7fdec2ffd700 1 -- 172.21.15.16:0/521530428 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdebc005d70 con 0x7fdecc15ec80 2023-10-16T13:27:21.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.902+0000 7fdec2ffd700 1 -- 172.21.15.16:0/521530428 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdebc0070d0 con 0x7fdecc15ec80 2023-10-16T13:27:21.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.902+0000 7fded4787700 1 -- 172.21.15.16:0/521530428 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdecc1e4760 con 0x7fdecc15ec80 2023-10-16T13:27:21.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.902+0000 7fdec2ffd700 1 -- 172.21.15.16:0/521530428 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdebc007460 con 0x7fdecc15ec80 2023-10-16T13:27:21.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.902+0000 7fded4787700 1 -- 172.21.15.16:0/521530428 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdecc1e4d20 con 0x7fdecc15ec80 2023-10-16T13:27:21.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.903+0000 7fded4787700 1 -- 172.21.15.16:0/521530428 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdecc1e4ee0 con 0x7fdecc15ec80 2023-10-16T13:27:21.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.904+0000 7fdec2ffd700 1 -- 172.21.15.16:0/521530428 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 94) v1 ==== 89343+0+0 (secure 0 0 0) 0x7fdebc0075c0 con 0x7fdecc15ec80 2023-10-16T13:27:21.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.904+0000 7fdec2ffd700 1 --2- 172.21.15.16:0/521530428 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7fdeb006b020 0x7fdeb006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:21.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.904+0000 7fded2523700 1 -- 172.21.15.16:0/521530428 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7fdeb006b020 msgr2=0x7fdeb006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1503042987 2023-10-16T13:27:21.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.904+0000 7fdec2ffd700 1 -- 172.21.15.16:0/521530428 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 6517+0+0 (secure 0 0 0) 0x7fdebc088e00 con 0x7fdecc15ec80 2023-10-16T13:27:21.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.904+0000 7fded2523700 1 --2- 172.21.15.16:0/521530428 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7fdeb006b020 0x7fdeb006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:27:21.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:21.909+0000 7fdec2ffd700 1 -- 172.21.15.16:0/521530428 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdebc053b00 con 0x7fdecc15ec80 2023-10-16T13:27:22.028 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:27:22.025+0000 7fbf61b58200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T13:27:22.090 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:27:22.088+0000 7f52d4a73200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T13:27:22.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.105+0000 7fded2523700 1 -- 172.21.15.16:0/521530428 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7fdeb006b020 msgr2=0x7fdeb006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1503042987 2023-10-16T13:27:22.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.105+0000 7fded2523700 1 --2- 172.21.15.16:0/521530428 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7fdeb006b020 0x7fdeb006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:27:22.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.158+0000 7fded4787700 1 -- 172.21.15.16:0/521530428 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fdecc08bb80 con 0x7fdecc15ec80 2023-10-16T13:27:22.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.158+0000 7fdec2ffd700 1 -- 172.21.15.16:0/521530428 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v164) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdebc0170a0 con 0x7fdecc15ec80 2023-10-16T13:27:22.161 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T13:27:22.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.161+0000 7fded4787700 1 -- 172.21.15.16:0/521530428 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7fdeb006b020 msgr2=0x7fdeb006d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:27:22.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.161+0000 7fded4787700 1 --2- 172.21.15.16:0/521530428 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7fdeb006b020 0x7fdeb006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:22.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.161+0000 7fded4787700 1 -- 172.21.15.16:0/521530428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdecc15ec80 msgr2=0x7fdecc06f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:22.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.161+0000 7fded4787700 1 --2- 172.21.15.16:0/521530428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdecc15ec80 0x7fdecc06f160 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7fdebc001eb0 tx=0x7fdebc00c7b0 comp rx=0 tx=0).stop 2023-10-16T13:27:22.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.161+0000 7fded4787700 1 -- 172.21.15.16:0/521530428 shutdown_connections 2023-10-16T13:27:22.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.161+0000 7fded4787700 1 --2- 172.21.15.16:0/521530428 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7fdeb006b020 0x7fdeb006d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:22.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.161+0000 7fded4787700 1 --2- 172.21.15.16:0/521530428 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdecc15f060 0x7fdecc06f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:22.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.161+0000 7fded4787700 1 --2- 172.21.15.16:0/521530428 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdecc0ce5d0 0x7fdecc0c86c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:22.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.161+0000 7fded4787700 1 --2- 172.21.15.16:0/521530428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdecc15ec80 0x7fdecc06f160 unknown :-1 s=CLOSED pgs=1813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:22.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.161+0000 7fded4787700 1 -- 172.21.15.16:0/521530428 >> 172.21.15.16:0/521530428 conn(0x7fdecc0d15f0 msgr2=0x7fdecc153c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:22.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.161+0000 7fded4787700 1 -- 172.21.15.16:0/521530428 shutdown_connections 2023-10-16T13:27:22.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.161+0000 7fded4787700 1 -- 172.21.15.16:0/521530428 wait complete. 2023-10-16T13:27:22.177 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T13:27:22.178 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:27:22.175+0000 7fbf61b58200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T13:27:22.214 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:27:22.212+0000 7f52d4a73200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T13:27:22.254 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:27:22.252+0000 7fbf61b58200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T13:27:22.293 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:27:22.291+0000 7f52d4a73200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T13:27:22.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.308+0000 7fbfa1869700 1 Processor -- start 2023-10-16T13:27:22.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.308+0000 7fbfa1869700 1 -- start start 2023-10-16T13:27:22.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.309+0000 7fbfa1869700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fbf9c0ce6a0 0x7fbf9c06c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:22.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.309+0000 7fbfa1869700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbf9c0b2620 0x7fbf9c06c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:22.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.309+0000 7fbfa1869700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf9c0b20f0 con 0x7fbf9c0d2110 2023-10-16T13:27:22.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.309+0000 7fbfa1869700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf9c0af1f0 con 0x7fbf9c0b2620 2023-10-16T13:27:22.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.309+0000 7fbfa1869700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf9c0d03d0 con 0x7fbf9c0ce6a0 2023-10-16T13:27:22.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.309+0000 7fbf9b7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbf9c0b2620 0x7fbf9c06c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:22.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.309+0000 7fbf9a7fc700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fbf9c0d2110 0x7fbf9c0b0f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36986/0 (socket says 172.21.15.16:36986) 2023-10-16T13:27:22.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.309+0000 7fbf9a7fc700 1 -- 172.21.15.16:0/4259513628 learned_addr learned my addr 172.21.15.16:0/4259513628 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:27:22.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.309+0000 7fbf9affd700 1 --2- 172.21.15.16:0/4259513628 >> v2:172.21.15.49:3300/0 conn(0x7fbf9c0ce6a0 0x7fbf9c06c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:22.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.309+0000 7fbf9b7fe700 1 -- 172.21.15.16:0/4259513628 >> v2:172.21.15.49:3300/0 conn(0x7fbf9c0ce6a0 msgr2=0x7fbf9c06c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:22.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.309+0000 7fbf9b7fe700 1 --2- 172.21.15.16:0/4259513628 >> v2:172.21.15.49:3300/0 conn(0x7fbf9c0ce6a0 0x7fbf9c06c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:22.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.309+0000 7fbf9b7fe700 1 -- 172.21.15.16:0/4259513628 >> v1:172.21.15.16:6789/0 conn(0x7fbf9c0d2110 legacy=0x7fbf9c0b0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:22.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.309+0000 7fbf99ffb700 1 -- 172.21.15.16:0/4259513628 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 560471571 0 0) 0x7fbf9c0b20f0 con 0x7fbf9c0d2110 2023-10-16T13:27:22.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.309+0000 7fbf9b7fe700 1 -- 172.21.15.16:0/4259513628 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf9c0b2340 con 0x7fbf9c0b2620 2023-10-16T13:27:22.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.310+0000 7fbf9b7fe700 1 --2- 172.21.15.16:0/4259513628 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf9c0b2620 0x7fbf9c06c840 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c067b50 tx=0x7fbf9400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=568ef31d2bbbbe6a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:22.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.310+0000 7fbf99ffb700 1 -- 172.21.15.16:0/4259513628 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf9400e070 con 0x7fbf9c0b2620 2023-10-16T13:27:22.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.310+0000 7fbf99ffb700 1 -- 172.21.15.16:0/4259513628 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf9400bb20 con 0x7fbf9c0b2620 2023-10-16T13:27:22.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.310+0000 7fbf99ffb700 1 -- 172.21.15.16:0/4259513628 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf94012430 con 0x7fbf9c0b2620 2023-10-16T13:27:22.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.310+0000 7fbfa1869700 1 -- 172.21.15.16:0/4259513628 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf9c0b2620 msgr2=0x7fbf9c06c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:22.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.310+0000 7fbfa1869700 1 --2- 172.21.15.16:0/4259513628 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf9c0b2620 0x7fbf9c06c840 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c067b50 tx=0x7fbf9400b330 comp rx=0 tx=0).stop 2023-10-16T13:27:22.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.310+0000 7fbfa1869700 1 -- 172.21.15.16:0/4259513628 shutdown_connections 2023-10-16T13:27:22.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.310+0000 7fbfa1869700 1 --2- 172.21.15.16:0/4259513628 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf9c0b2620 0x7fbf9c06c840 unknown :-1 s=CLOSED pgs=1815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:22.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.310+0000 7fbfa1869700 1 --2- 172.21.15.16:0/4259513628 >> v2:172.21.15.49:3300/0 conn(0x7fbf9c0ce6a0 0x7fbf9c06c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:22.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.310+0000 7fbfa1869700 1 -- 172.21.15.16:0/4259513628 >> 172.21.15.16:0/4259513628 conn(0x7fbf9c0d15f0 msgr2=0x7fbf9c0d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:22.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.310+0000 7fbfa1869700 1 -- 172.21.15.16:0/4259513628 shutdown_connections 2023-10-16T13:27:22.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.310+0000 7fbfa1869700 1 -- 172.21.15.16:0/4259513628 wait complete. 2023-10-16T13:27:22.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.312+0000 7fbfa1869700 1 Processor -- start 2023-10-16T13:27:22.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.312+0000 7fbfa1869700 1 -- start start 2023-10-16T13:27:22.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.313+0000 7fbfa1869700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbf9c0b2620 0x7fbf9c14ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:22.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.313+0000 7fbfa1869700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbf9c0ce6a0 0x7fbf9c0afff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:22.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.313+0000 7fbfa1869700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf9c0d2110 0x7fbf9c0b0530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:22.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.313+0000 7fbfa1869700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf9c151550 con 0x7fbf9c0d2110 2023-10-16T13:27:22.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.313+0000 7fbfa1869700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf9c1504b0 con 0x7fbf9c0b2620 2023-10-16T13:27:22.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.313+0000 7fbfa1869700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf9c150790 con 0x7fbf9c0ce6a0 2023-10-16T13:27:22.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.313+0000 7fbf9b7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf9c0d2110 0x7fbf9c0b0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:22.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.313+0000 7fbf9b7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf9c0d2110 0x7fbf9c0b0530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47720/0 (socket says 172.21.15.16:47720) 2023-10-16T13:27:22.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.313+0000 7fbf9b7fe700 1 -- 172.21.15.16:0/2554195048 learned_addr learned my addr 172.21.15.16:0/2554195048 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:22.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.313+0000 7fbf9affd700 1 --2- 172.21.15.16:0/2554195048 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbf9c0b2620 0x7fbf9c14ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:22.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.313+0000 7fbf9a7fc700 1 --2- 172.21.15.16:0/2554195048 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbf9c0ce6a0 0x7fbf9c0afff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:22.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.313+0000 7fbf9b7fe700 1 -- 172.21.15.16:0/2554195048 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbf9c0ce6a0 msgr2=0x7fbf9c0afff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:22.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.313+0000 7fbf9b7fe700 1 --2- 172.21.15.16:0/2554195048 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbf9c0ce6a0 0x7fbf9c0afff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:22.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.313+0000 7fbf9b7fe700 1 -- 172.21.15.16:0/2554195048 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbf9c0b2620 msgr2=0x7fbf9c14ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:22.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.313+0000 7fbf9b7fe700 1 --2- 172.21.15.16:0/2554195048 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbf9c0b2620 0x7fbf9c14ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:22.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.313+0000 7fbf9b7fe700 1 -- 172.21.15.16:0/2554195048 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf94009a70 con 0x7fbf9c0d2110 2023-10-16T13:27:22.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.313+0000 7fbf9affd700 1 --2- 172.21.15.16:0/2554195048 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbf9c0b2620 0x7fbf9c14ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:27:22.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.313+0000 7fbf9b7fe700 1 --2- 172.21.15.16:0/2554195048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf9c0d2110 0x7fbf9c0b0530 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7fbf9400b370 tx=0x7fbf94004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:22.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.313+0000 7fbf83fff700 1 -- 172.21.15.16:0/2554195048 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf94012430 con 0x7fbf9c0d2110 2023-10-16T13:27:22.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.313+0000 7fbf83fff700 1 -- 172.21.15.16:0/2554195048 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf94020070 con 0x7fbf9c0d2110 2023-10-16T13:27:22.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.313+0000 7fbfa1869700 1 -- 172.21.15.16:0/2554195048 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf9c150a70 con 0x7fbf9c0d2110 2023-10-16T13:27:22.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.313+0000 7fbf83fff700 1 -- 172.21.15.16:0/2554195048 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf94004280 con 0x7fbf9c0d2110 2023-10-16T13:27:22.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.314+0000 7fbfa1869700 1 -- 172.21.15.16:0/2554195048 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf9c151030 con 0x7fbf9c0d2110 2023-10-16T13:27:22.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.315+0000 7fbfa1869700 1 -- 172.21.15.16:0/2554195048 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf9c04ebb0 con 0x7fbf9c0d2110 2023-10-16T13:27:22.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.315+0000 7fbf83fff700 1 -- 172.21.15.16:0/2554195048 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 94) v1 ==== 89343+0+0 (secure 0 0 0) 0x7fbf940043e0 con 0x7fbf9c0d2110 2023-10-16T13:27:22.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.316+0000 7fbf83fff700 1 --2- 172.21.15.16:0/2554195048 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7fbf8806b020 0x7fbf8806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:22.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.316+0000 7fbf9affd700 1 -- 172.21.15.16:0/2554195048 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7fbf8806b020 msgr2=0x7fbf8806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1503042987 2023-10-16T13:27:22.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.316+0000 7fbf83fff700 1 -- 172.21.15.16:0/2554195048 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 6517+0+0 (secure 0 0 0) 0x7fbf94022070 con 0x7fbf9c0d2110 2023-10-16T13:27:22.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.316+0000 7fbf9affd700 1 --2- 172.21.15.16:0/2554195048 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7fbf8806b020 0x7fbf8806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:27:22.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.320+0000 7fbf83fff700 1 -- 172.21.15.16:0/2554195048 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbf94056390 con 0x7fbf9c0d2110 2023-10-16T13:27:22.339 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:27:22.337+0000 7fbf61b58200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T13:27:22.382 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:27:22.380+0000 7f52d4a73200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T13:27:22.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.517+0000 7fbf9affd700 1 -- 172.21.15.16:0/2554195048 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7fbf8806b020 msgr2=0x7fbf8806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1503042987 2023-10-16T13:27:22.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.517+0000 7fbf9affd700 1 --2- 172.21.15.16:0/2554195048 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7fbf8806b020 0x7fbf8806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:27:22.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.529+0000 7fbfa1869700 1 -- 172.21.15.16:0/2554195048 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fbf9c06bde0 con 0x7fbf9c0d2110 2023-10-16T13:27:22.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.529+0000 7fbf83fff700 1 -- 172.21.15.16:0/2554195048 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v164) v1 ==== 104+0+11 (secure 0 0 0) 0x7fbf94012e60 con 0x7fbf9c0d2110 2023-10-16T13:27:22.532 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-10-16T13:27:22.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.531+0000 7fbfa1869700 1 -- 172.21.15.16:0/2554195048 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7fbf8806b020 msgr2=0x7fbf8806d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:27:22.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.531+0000 7fbfa1869700 1 --2- 172.21.15.16:0/2554195048 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7fbf8806b020 0x7fbf8806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:22.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.531+0000 7fbfa1869700 1 -- 172.21.15.16:0/2554195048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf9c0d2110 msgr2=0x7fbf9c0b0530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:22.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.531+0000 7fbfa1869700 1 --2- 172.21.15.16:0/2554195048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf9c0d2110 0x7fbf9c0b0530 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7fbf9400b370 tx=0x7fbf94004ad0 comp rx=0 tx=0).stop 2023-10-16T13:27:22.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.531+0000 7fbfa1869700 1 -- 172.21.15.16:0/2554195048 shutdown_connections 2023-10-16T13:27:22.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.531+0000 7fbfa1869700 1 --2- 172.21.15.16:0/2554195048 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7fbf8806b020 0x7fbf8806d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:22.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.531+0000 7fbfa1869700 1 --2- 172.21.15.16:0/2554195048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf9c0d2110 0x7fbf9c0b0530 unknown :-1 s=CLOSED pgs=1816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:22.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.531+0000 7fbfa1869700 1 --2- 172.21.15.16:0/2554195048 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbf9c0b2620 0x7fbf9c14ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:22.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.531+0000 7fbfa1869700 1 --2- 172.21.15.16:0/2554195048 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbf9c0ce6a0 0x7fbf9c0afff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:22.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.531+0000 7fbfa1869700 1 -- 172.21.15.16:0/2554195048 >> 172.21.15.16:0/2554195048 conn(0x7fbf9c0d15f0 msgr2=0x7fbf9c06c300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:22.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.531+0000 7fbfa1869700 1 -- 172.21.15.16:0/2554195048 shutdown_connections 2023-10-16T13:27:22.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.531+0000 7fbfa1869700 1 -- 172.21.15.16:0/2554195048 wait complete. 2023-10-16T13:27:22.546 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T13:27:22.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.714+0000 7f0bb4447700 1 Processor -- start 2023-10-16T13:27:22.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.714+0000 7f0bb4447700 1 -- start start 2023-10-16T13:27:22.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.715+0000 7f0bb4447700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0bac15b070 0x7f0bac15b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:22.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.715+0000 7f0bb4447700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0bac0cb880 0x7f0bac0cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:22.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.715+0000 7f0bb4447700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f0bac06da60 0x7f0bac06def0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:22.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.715+0000 7f0bb4447700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bac06e430 con 0x7f0bac15b070 2023-10-16T13:27:22.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.715+0000 7f0bb4447700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bac0cef70 con 0x7f0bac06da60 2023-10-16T13:27:22.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.715+0000 7f0bb4447700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bac0ce360 con 0x7f0bac0cb880 2023-10-16T13:27:22.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.715+0000 7f0bb21e3700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0bac15b070 0x7f0bac15b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:22.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.715+0000 7f0bb21e3700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0bac15b070 0x7f0bac15b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47742/0 (socket says 172.21.15.16:47742) 2023-10-16T13:27:22.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.715+0000 7f0bb21e3700 1 -- 172.21.15.16:0/2802709220 learned_addr learned my addr 172.21.15.16:0/2802709220 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:22.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.715+0000 7f0bb19e2700 1 --2- 172.21.15.16:0/2802709220 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0bac0cb880 0x7f0bac0cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:22.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.715+0000 7f0bb29e4700 1 --2- 172.21.15.16:0/2802709220 >> v2:172.21.15.49:3300/0 conn(0x7f0bac06da60 0x7f0bac06def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:22.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.715+0000 7f0bb21e3700 1 -- 172.21.15.16:0/2802709220 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0bac0cb880 msgr2=0x7f0bac0cbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:22.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.715+0000 7f0bb21e3700 1 --2- 172.21.15.16:0/2802709220 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0bac0cb880 0x7f0bac0cbd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:22.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.715+0000 7f0bb21e3700 1 -- 172.21.15.16:0/2802709220 >> v2:172.21.15.49:3300/0 conn(0x7f0bac06da60 msgr2=0x7f0bac06def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:22.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.715+0000 7f0bb21e3700 1 --2- 172.21.15.16:0/2802709220 >> v2:172.21.15.49:3300/0 conn(0x7f0bac06da60 0x7f0bac06def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:22.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.715+0000 7f0bb21e3700 1 -- 172.21.15.16:0/2802709220 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bac0ce640 con 0x7f0bac15b070 2023-10-16T13:27:22.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.716+0000 7f0bb21e3700 1 --2- 172.21.15.16:0/2802709220 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bac15b070 0x7f0bac15b450 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7f0bac072fc0 tx=0x7f0ba8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3227e0cd6790daa6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:22.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.716+0000 7f0bb11e1700 1 -- 172.21.15.16:0/2802709220 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ba8011830 con 0x7f0bac15b070 2023-10-16T13:27:22.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.716+0000 7f0bb11e1700 1 -- 172.21.15.16:0/2802709220 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ba8011990 con 0x7f0bac15b070 2023-10-16T13:27:22.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.716+0000 7f0bb11e1700 1 -- 172.21.15.16:0/2802709220 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ba8011d20 con 0x7f0bac15b070 2023-10-16T13:27:22.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.716+0000 7f0bb4447700 1 -- 172.21.15.16:0/2802709220 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bac15b070 msgr2=0x7f0bac15b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:22.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.716+0000 7f0bb4447700 1 --2- 172.21.15.16:0/2802709220 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bac15b070 0x7f0bac15b450 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7f0bac072fc0 tx=0x7f0ba8009bd0 comp rx=0 tx=0).stop 2023-10-16T13:27:22.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.716+0000 7f0bb4447700 1 -- 172.21.15.16:0/2802709220 shutdown_connections 2023-10-16T13:27:22.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.716+0000 7f0bb4447700 1 --2- 172.21.15.16:0/2802709220 >> v2:172.21.15.49:3300/0 conn(0x7f0bac06da60 0x7f0bac06def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:22.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.716+0000 7f0bb4447700 1 --2- 172.21.15.16:0/2802709220 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bac15b070 0x7f0bac15b450 unknown :-1 s=CLOSED pgs=1817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:22.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.716+0000 7f0bb4447700 1 --2- 172.21.15.16:0/2802709220 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0bac0cb880 0x7f0bac0cbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:22.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.716+0000 7f0bb4447700 1 -- 172.21.15.16:0/2802709220 >> 172.21.15.16:0/2802709220 conn(0x7f0bac0b1f40 msgr2=0x7f0bac0b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:22.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.717+0000 7f0bb4447700 1 -- 172.21.15.16:0/2802709220 shutdown_connections 2023-10-16T13:27:22.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.717+0000 7f0bb4447700 1 -- 172.21.15.16:0/2802709220 wait complete. 2023-10-16T13:27:22.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.718+0000 7f0bb4447700 1 Processor -- start 2023-10-16T13:27:22.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.718+0000 7f0bb4447700 1 -- start start 2023-10-16T13:27:22.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.719+0000 7f0bb4447700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bac06da60 0x7f0bac0c9350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:22.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.719+0000 7f0bb4447700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0bac0cb880 0x7f0bac1f2740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:22.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.719+0000 7f0bb4447700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0bac15b070 0x7f0bac1589d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:22.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.719+0000 7f0bb4447700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bac15d4f0 con 0x7f0bac06da60 2023-10-16T13:27:22.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.719+0000 7f0bb4447700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bac15cf30 con 0x7f0bac0cb880 2023-10-16T13:27:22.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.719+0000 7f0bb4447700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bac15d210 con 0x7f0bac15b070 2023-10-16T13:27:22.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.719+0000 7f0bb21e3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bac06da60 0x7f0bac0c9350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:22.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.719+0000 7f0bb21e3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bac06da60 0x7f0bac0c9350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47748/0 (socket says 172.21.15.16:47748) 2023-10-16T13:27:22.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.719+0000 7f0bb21e3700 1 -- 172.21.15.16:0/1949599493 learned_addr learned my addr 172.21.15.16:0/1949599493 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:22.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.719+0000 7f0bb19e2700 1 --2- 172.21.15.16:0/1949599493 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0bac0cb880 0x7f0bac1f2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:22.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.719+0000 7f0bb29e4700 1 --2- 172.21.15.16:0/1949599493 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0bac15b070 0x7f0bac1589d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:22.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.719+0000 7f0bb21e3700 1 -- 172.21.15.16:0/1949599493 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0bac15b070 msgr2=0x7f0bac1589d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:22.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.719+0000 7f0bb21e3700 1 --2- 172.21.15.16:0/1949599493 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0bac15b070 0x7f0bac1589d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:22.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.719+0000 7f0bb21e3700 1 -- 172.21.15.16:0/1949599493 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0bac0cb880 msgr2=0x7f0bac1f2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:22.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.719+0000 7f0bb21e3700 1 --2- 172.21.15.16:0/1949599493 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0bac0cb880 0x7f0bac1f2740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:22.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.719+0000 7f0bb21e3700 1 -- 172.21.15.16:0/1949599493 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ba8009550 con 0x7f0bac06da60 2023-10-16T13:27:22.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.719+0000 7f0bb29e4700 1 --2- 172.21.15.16:0/1949599493 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0bac15b070 0x7f0bac1589d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:27:22.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.719+0000 7f0bb21e3700 1 --2- 172.21.15.16:0/1949599493 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bac06da60 0x7f0bac0c9350 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7f0ba8009c10 tx=0x7f0ba8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:22.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.719+0000 7f0ba2ffd700 1 -- 172.21.15.16:0/1949599493 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ba8004510 con 0x7f0bac06da60 2023-10-16T13:27:22.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.719+0000 7f0ba2ffd700 1 -- 172.21.15.16:0/1949599493 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ba801e070 con 0x7f0bac06da60 2023-10-16T13:27:22.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.719+0000 7f0bb4447700 1 -- 172.21.15.16:0/1949599493 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bac0aca70 con 0x7f0bac06da60 2023-10-16T13:27:22.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.719+0000 7f0ba2ffd700 1 -- 172.21.15.16:0/1949599493 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ba80038e0 con 0x7f0bac06da60 2023-10-16T13:27:22.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.720+0000 7f0bb4447700 1 -- 172.21.15.16:0/1949599493 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bac0c9890 con 0x7f0bac06da60 2023-10-16T13:27:22.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.721+0000 7f0bb4447700 1 -- 172.21.15.16:0/1949599493 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bac0ac8d0 con 0x7f0bac06da60 2023-10-16T13:27:22.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.722+0000 7f0ba2ffd700 1 -- 172.21.15.16:0/1949599493 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 94) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f0ba8003a40 con 0x7f0bac06da60 2023-10-16T13:27:22.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.722+0000 7f0ba2ffd700 1 --2- 172.21.15.16:0/1949599493 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f0b9006b020 0x7f0b9006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:22.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.722+0000 7f0bb19e2700 1 -- 172.21.15.16:0/1949599493 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f0b9006b020 msgr2=0x7f0b9006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1503042987 2023-10-16T13:27:22.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.722+0000 7f0ba2ffd700 1 -- 172.21.15.16:0/1949599493 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f0ba808a820 con 0x7f0bac06da60 2023-10-16T13:27:22.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.722+0000 7f0bb19e2700 1 --2- 172.21.15.16:0/1949599493 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f0b9006b020 0x7f0b9006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:27:22.734 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:27:22.729+0000 7fbf61b58200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T13:27:22.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.727+0000 7f0ba2ffd700 1 -- 172.21.15.16:0/1949599493 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0bac0ac8d0 con 0x7f0bac06da60 2023-10-16T13:27:22.786 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:27:22.784+0000 7f52d4a73200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T13:27:22.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.923+0000 7f0bb19e2700 1 -- 172.21.15.16:0/1949599493 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f0b9006b020 msgr2=0x7f0b9006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1503042987 2023-10-16T13:27:22.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.923+0000 7f0bb19e2700 1 --2- 172.21.15.16:0/1949599493 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f0b9006b020 0x7f0b9006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:27:22.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.932+0000 7f0bb4447700 1 -- 172.21.15.16:0/1949599493 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0bac0ac8d0 con 0x7f0bac06da60 2023-10-16T13:27:22.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.933+0000 7f0ba2ffd700 1 -- 172.21.15.16:0/1949599493 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v164) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0bac0ac8d0 con 0x7f0bac06da60 2023-10-16T13:27:22.936 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-10-16T13:27:22.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.935+0000 7f0bb4447700 1 -- 172.21.15.16:0/1949599493 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f0b9006b020 msgr2=0x7f0b9006d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:27:22.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.935+0000 7f0bb4447700 1 --2- 172.21.15.16:0/1949599493 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f0b9006b020 0x7f0b9006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:22.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.935+0000 7f0bb4447700 1 -- 172.21.15.16:0/1949599493 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bac06da60 msgr2=0x7f0bac0c9350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:22.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.935+0000 7f0bb4447700 1 --2- 172.21.15.16:0/1949599493 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bac06da60 0x7f0bac0c9350 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7f0ba8009c10 tx=0x7f0ba8004750 comp rx=0 tx=0).stop 2023-10-16T13:27:22.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.935+0000 7f0bb4447700 1 -- 172.21.15.16:0/1949599493 shutdown_connections 2023-10-16T13:27:22.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.935+0000 7f0bb4447700 1 --2- 172.21.15.16:0/1949599493 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f0b9006b020 0x7f0b9006d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:22.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.935+0000 7f0bb4447700 1 --2- 172.21.15.16:0/1949599493 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0bac15b070 0x7f0bac1589d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:22.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.935+0000 7f0bb4447700 1 --2- 172.21.15.16:0/1949599493 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bac06da60 0x7f0bac0c9350 unknown :-1 s=CLOSED pgs=1818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:22.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.935+0000 7f0bb4447700 1 --2- 172.21.15.16:0/1949599493 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0bac0cb880 0x7f0bac1f2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:22.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.935+0000 7f0bb4447700 1 -- 172.21.15.16:0/1949599493 >> 172.21.15.16:0/1949599493 conn(0x7f0bac0b1f40 msgr2=0x7f0bac0cf400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:22.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.935+0000 7f0bb4447700 1 -- 172.21.15.16:0/1949599493 shutdown_connections 2023-10-16T13:27:22.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:22.935+0000 7f0bb4447700 1 -- 172.21.15.16:0/1949599493 wait complete. 2023-10-16T13:27:22.950 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T13:27:23.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.077+0000 7fb10a399700 1 Processor -- start 2023-10-16T13:27:23.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.077+0000 7fb10a399700 1 -- start start 2023-10-16T13:27:23.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.078+0000 7fb10a399700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb10415b630 0x7fb1040cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:23.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.078+0000 7fb10a399700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fb10415ba10 0x7fb1040cdde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:23.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.078+0000 7fb10a399700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb1040ce320 0x7fb1040cfca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:23.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.078+0000 7fb10a399700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1040cdb20 con 0x7fb10415b630 2023-10-16T13:27:23.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.078+0000 7fb10a399700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1040ce750 con 0x7fb10415ba10 2023-10-16T13:27:23.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.078+0000 7fb10a399700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1040cc2a0 con 0x7fb1040ce320 2023-10-16T13:27:23.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.078+0000 7fb103fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb10415b630 0x7fb1040cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:23.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.079+0000 7fb103fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb10415b630 0x7fb1040cd5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47758/0 (socket says 172.21.15.16:47758) 2023-10-16T13:27:23.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.079+0000 7fb103fff700 1 -- 172.21.15.16:0/2392971902 learned_addr learned my addr 172.21.15.16:0/2392971902 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:23.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.079+0000 7fb108936700 1 --2- 172.21.15.16:0/2392971902 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb1040ce320 0x7fb1040cfca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:23.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.079+0000 7fb1037fe700 1 --2- 172.21.15.16:0/2392971902 >> v2:172.21.15.49:3300/0 conn(0x7fb10415ba10 0x7fb1040cdde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:23.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.079+0000 7fb103fff700 1 -- 172.21.15.16:0/2392971902 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb1040ce320 msgr2=0x7fb1040cfca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:23.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.079+0000 7fb103fff700 1 --2- 172.21.15.16:0/2392971902 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb1040ce320 0x7fb1040cfca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:23.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.079+0000 7fb103fff700 1 -- 172.21.15.16:0/2392971902 >> v2:172.21.15.49:3300/0 conn(0x7fb10415ba10 msgr2=0x7fb1040cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:23.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.079+0000 7fb103fff700 1 --2- 172.21.15.16:0/2392971902 >> v2:172.21.15.49:3300/0 conn(0x7fb10415ba10 0x7fb1040cdde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:23.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.079+0000 7fb103fff700 1 -- 172.21.15.16:0/2392971902 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1040cc540 con 0x7fb10415b630 2023-10-16T13:27:23.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.079+0000 7fb1037fe700 1 --2- 172.21.15.16:0/2392971902 >> v2:172.21.15.49:3300/0 conn(0x7fb10415ba10 0x7fb1040cdde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:27:23.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.079+0000 7fb108936700 1 --2- 172.21.15.16:0/2392971902 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb1040ce320 0x7fb1040cfca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:27:23.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.079+0000 7fb103fff700 1 --2- 172.21.15.16:0/2392971902 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb10415b630 0x7fb1040cd5e0 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7fb1040cc070 tx=0x7fb0f4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd7e1b758e84c5cc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:23.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.079+0000 7fb102ffd700 1 -- 172.21.15.16:0/2392971902 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0f4005e30 con 0x7fb10415b630 2023-10-16T13:27:23.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.079+0000 7fb102ffd700 1 -- 172.21.15.16:0/2392971902 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0f4004030 con 0x7fb10415b630 2023-10-16T13:27:23.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.079+0000 7fb102ffd700 1 -- 172.21.15.16:0/2392971902 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0f40043e0 con 0x7fb10415b630 2023-10-16T13:27:23.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.080+0000 7fb10a399700 1 -- 172.21.15.16:0/2392971902 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb10415b630 msgr2=0x7fb1040cd5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:23.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.080+0000 7fb10a399700 1 --2- 172.21.15.16:0/2392971902 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb10415b630 0x7fb1040cd5e0 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7fb1040cc070 tx=0x7fb0f4009bd0 comp rx=0 tx=0).stop 2023-10-16T13:27:23.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.080+0000 7fb10a399700 1 -- 172.21.15.16:0/2392971902 shutdown_connections 2023-10-16T13:27:23.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.080+0000 7fb10a399700 1 --2- 172.21.15.16:0/2392971902 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb1040ce320 0x7fb1040cfca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:23.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.080+0000 7fb10a399700 1 --2- 172.21.15.16:0/2392971902 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb10415b630 0x7fb1040cd5e0 unknown :-1 s=CLOSED pgs=1819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:23.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.080+0000 7fb10a399700 1 --2- 172.21.15.16:0/2392971902 >> v2:172.21.15.49:3300/0 conn(0x7fb10415ba10 0x7fb1040cdde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:23.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.080+0000 7fb10a399700 1 -- 172.21.15.16:0/2392971902 >> 172.21.15.16:0/2392971902 conn(0x7fb1040d14b0 msgr2=0x7fb1040d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:23.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.080+0000 7fb10a399700 1 -- 172.21.15.16:0/2392971902 shutdown_connections 2023-10-16T13:27:23.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.080+0000 7fb10a399700 1 -- 172.21.15.16:0/2392971902 wait complete. 2023-10-16T13:27:23.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.082+0000 7fb10a399700 1 Processor -- start 2023-10-16T13:27:23.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.082+0000 7fb10a399700 1 -- start start 2023-10-16T13:27:23.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.083+0000 7fb10a399700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb1040ce320 0x7fb1041f2240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:23.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.083+0000 7fb10a399700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb10415b630 0x7fb10409d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:23.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.083+0000 7fb10a399700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb10415ba10 0x7fb1041f4670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:23.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.083+0000 7fb10a399700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb10409e2c0 con 0x7fb10415b630 2023-10-16T13:27:23.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.083+0000 7fb10a399700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1040c9670 con 0x7fb10415ba10 2023-10-16T13:27:23.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.083+0000 7fb10a399700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1040c9950 con 0x7fb1040ce320 2023-10-16T13:27:23.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.083+0000 7fb1037fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb10415b630 0x7fb10409d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:23.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.083+0000 7fb103fff700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb1040ce320 0x7fb1041f2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:23.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.083+0000 7fb1037fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb10415b630 0x7fb10409d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47766/0 (socket says 172.21.15.16:47766) 2023-10-16T13:27:23.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.083+0000 7fb1037fe700 1 -- 172.21.15.16:0/1080529673 learned_addr learned my addr 172.21.15.16:0/1080529673 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:23.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.083+0000 7fb108936700 1 --2- 172.21.15.16:0/1080529673 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb10415ba10 0x7fb1041f4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:23.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.084+0000 7fb1037fe700 1 -- 172.21.15.16:0/1080529673 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb1040ce320 msgr2=0x7fb1041f2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:23.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.084+0000 7fb1037fe700 1 --2- 172.21.15.16:0/1080529673 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb1040ce320 0x7fb1041f2240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:23.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.084+0000 7fb1037fe700 1 -- 172.21.15.16:0/1080529673 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb10415ba10 msgr2=0x7fb1041f4670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:23.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.084+0000 7fb1037fe700 1 --2- 172.21.15.16:0/1080529673 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb10415ba10 0x7fb1041f4670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:23.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.084+0000 7fb1037fe700 1 -- 172.21.15.16:0/1080529673 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0f4009500 con 0x7fb10415b630 2023-10-16T13:27:23.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.084+0000 7fb1037fe700 1 --2- 172.21.15.16:0/1080529673 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb10415b630 0x7fb10409d820 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7fb0f800e810 tx=0x7fb0f800eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:23.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.084+0000 7fb100ff9700 1 -- 172.21.15.16:0/1080529673 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0f800c070 con 0x7fb10415b630 2023-10-16T13:27:23.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.084+0000 7fb10a399700 1 -- 172.21.15.16:0/1080529673 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1040c9c30 con 0x7fb10415b630 2023-10-16T13:27:23.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.084+0000 7fb100ff9700 1 -- 172.21.15.16:0/1080529673 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0f8004230 con 0x7fb10415b630 2023-10-16T13:27:23.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.084+0000 7fb100ff9700 1 -- 172.21.15.16:0/1080529673 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0f80036a0 con 0x7fb10415b630 2023-10-16T13:27:23.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.084+0000 7fb10a399700 1 -- 172.21.15.16:0/1080529673 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1041f2780 con 0x7fb10415b630 2023-10-16T13:27:23.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.085+0000 7fb10a399700 1 -- 172.21.15.16:0/1080529673 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb10404ebb0 con 0x7fb10415b630 2023-10-16T13:27:23.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.086+0000 7fb100ff9700 1 -- 172.21.15.16:0/1080529673 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 94) v1 ==== 89343+0+0 (secure 0 0 0) 0x7fb0f8003800 con 0x7fb10415b630 2023-10-16T13:27:23.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.086+0000 7fb100ff9700 1 --2- 172.21.15.16:0/1080529673 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7fb0f006b020 0x7fb0f006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:23.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.086+0000 7fb103fff700 1 -- 172.21.15.16:0/1080529673 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7fb0f006b020 msgr2=0x7fb0f006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1503042987 2023-10-16T13:27:23.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.086+0000 7fb100ff9700 1 -- 172.21.15.16:0/1080529673 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 6517+0+0 (secure 0 0 0) 0x7fb0f80898b0 con 0x7fb10415b630 2023-10-16T13:27:23.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.086+0000 7fb103fff700 1 --2- 172.21.15.16:0/1080529673 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7fb0f006b020 0x7fb0f006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:27:23.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.091+0000 7fb100ff9700 1 -- 172.21.15.16:0/1080529673 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb0f8054390 con 0x7fb10415b630 2023-10-16T13:27:23.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.287+0000 7fb103fff700 1 -- 172.21.15.16:0/1080529673 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7fb0f006b020 msgr2=0x7fb0f006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1503042987 2023-10-16T13:27:23.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.287+0000 7fb103fff700 1 --2- 172.21.15.16:0/1080529673 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7fb0f006b020 0x7fb0f006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:27:23.346 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:27:23.344+0000 7fbf61b58200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T13:27:23.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.358+0000 7fb10a399700 1 -- 172.21.15.16:0/1080529673 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fb1040cb870 con 0x7fb10415b630 2023-10-16T13:27:23.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.359+0000 7fb100ff9700 1 -- 172.21.15.16:0/1080529673 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v164) v1 ==== 80+0+470 (secure 0 0 0) 0x7fb0f80579b0 con 0x7fb10415b630 2023-10-16T13:27:23.361 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3195044045 2023-10-17T12:31:20.854761+0000 2023-10-16T13:27:23.362 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6833/4182163087 2023-10-17T12:31:20.854761+0000 2023-10-16T13:27:23.362 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6832/4182163087 2023-10-17T12:31:20.854761+0000 2023-10-16T13:27:23.362 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/4217694331 2023-10-17T12:31:20.854761+0000 2023-10-16T13:27:23.362 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2516829556 2023-10-17T12:31:20.854761+0000 2023-10-16T13:27:23.362 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2546955595 2023-10-17T12:31:20.854761+0000 2023-10-16T13:27:23.363 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/1851287351 2023-10-17T12:31:20.854761+0000 2023-10-16T13:27:23.363 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3755256116 2023-10-17T12:31:20.854761+0000 2023-10-16T13:27:23.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.361+0000 7fb10a399700 1 -- 172.21.15.16:0/1080529673 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7fb0f006b020 msgr2=0x7fb0f006d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:27:23.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.361+0000 7fb10a399700 1 --2- 172.21.15.16:0/1080529673 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7fb0f006b020 0x7fb0f006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:23.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.361+0000 7fb10a399700 1 -- 172.21.15.16:0/1080529673 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb10415b630 msgr2=0x7fb10409d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:23.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.361+0000 7fb10a399700 1 --2- 172.21.15.16:0/1080529673 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb10415b630 0x7fb10409d820 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7fb0f800e810 tx=0x7fb0f800eb20 comp rx=0 tx=0).stop 2023-10-16T13:27:23.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.361+0000 7fb10a399700 1 -- 172.21.15.16:0/1080529673 shutdown_connections 2023-10-16T13:27:23.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.361+0000 7fb10a399700 1 --2- 172.21.15.16:0/1080529673 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7fb0f006b020 0x7fb0f006d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:23.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.361+0000 7fb10a399700 1 --2- 172.21.15.16:0/1080529673 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb10415ba10 0x7fb1041f4670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:23.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.361+0000 7fb10a399700 1 --2- 172.21.15.16:0/1080529673 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb1040ce320 0x7fb1041f2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:23.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.361+0000 7fb10a399700 1 --2- 172.21.15.16:0/1080529673 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb10415b630 0x7fb10409d820 unknown :-1 s=CLOSED pgs=1820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:23.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.361+0000 7fb10a399700 1 -- 172.21.15.16:0/1080529673 >> 172.21.15.16:0/1080529673 conn(0x7fb1040d14b0 msgr2=0x7fb1040b54c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:23.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.361+0000 7fb10a399700 1 -- 172.21.15.16:0/1080529673 shutdown_connections 2023-10-16T13:27:23.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.361+0000 7fb10a399700 1 -- 172.21.15.16:0/1080529673 wait complete. 2023-10-16T13:27:23.366 INFO:teuthology.orchestra.run.smithi016.stderr:listed 8 entries 2023-10-16T13:27:23.377 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-16T13:27:23.377 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:27:23.378 DEBUG:teuthology.orchestra.run.smithi016:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-16T13:27:23.412 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:27:23.410+0000 7f52d4a73200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T13:27:23.419 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:27:23.417+0000 7fbf61b58200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T13:27:23.435 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2023-10-16T13:27:23.435 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:27:23.436 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-16T13:27:23.488 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:27:23.486+0000 7f52d4a73200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T13:27:23.493 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T13:27:23.493 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:27:23.511 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284204k,nr_inodes=4071051,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=753 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260540k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T13:27:23.512 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T13:27:23.512 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:27:23.551 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16287364k,nr_inodes=4071841,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20251 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3261172k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T13:27:23.551 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T13:27:23.552 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:27:23.571 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284204k,nr_inodes=4071051,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=753 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260540k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T13:27:23.572 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:27:23.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.717+0000 7f4095bc9700 1 Processor -- start 2023-10-16T13:27:23.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.717+0000 7f4095bc9700 1 -- start start 2023-10-16T13:27:23.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.718+0000 7f4095bc9700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f409015b9f0 0x7f40900b29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:23.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.718+0000 7f4095bc9700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f409006d940 0x7f409006ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:23.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.718+0000 7f4095bc9700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40900c3490 con 0x7f40900ceb70 2023-10-16T13:27:23.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.718+0000 7f4095bc9700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f409006e300 con 0x7f409006d940 2023-10-16T13:27:23.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.718+0000 7f4095bc9700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40900cdad0 con 0x7f409015b9f0 2023-10-16T13:27:23.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.718+0000 7f408ffff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f40900ceb70 0x7f40900cefd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37052/0 (socket says 172.21.15.16:37052) 2023-10-16T13:27:23.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.718+0000 7f408ffff700 1 -- 172.21.15.16:0/783276793 learned_addr learned my addr 172.21.15.16:0/783276793 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:27:23.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.718+0000 7f408f7fe700 1 --2- 172.21.15.16:0/783276793 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f409015b9f0 0x7f40900b29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:23.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.718+0000 7f408effd700 1 --2- 172.21.15.16:0/783276793 >> v2:172.21.15.49:3300/0 conn(0x7f409006d940 0x7f409006ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:23.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.718+0000 7f408e7fc700 1 -- 172.21.15.16:0/783276793 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2378075998 0 0) 0x7f40900c3490 con 0x7f40900ceb70 2023-10-16T13:27:23.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.718+0000 7f408e7fc700 1 -- 172.21.15.16:0/783276793 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f407c0035b0 con 0x7f40900ceb70 2023-10-16T13:27:23.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.719+0000 7f408f7fe700 1 -- 172.21.15.16:0/783276793 >> v2:172.21.15.49:3300/0 conn(0x7f409006d940 msgr2=0x7f409006ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:23.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.719+0000 7f408f7fe700 1 --2- 172.21.15.16:0/783276793 >> v2:172.21.15.49:3300/0 conn(0x7f409006d940 0x7f409006ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:23.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.719+0000 7f408f7fe700 1 -- 172.21.15.16:0/783276793 >> v1:172.21.15.16:6789/0 conn(0x7f40900ceb70 legacy=0x7f40900cefd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:23.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.719+0000 7f408f7fe700 1 -- 172.21.15.16:0/783276793 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40900cddb0 con 0x7f409015b9f0 2023-10-16T13:27:23.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.719+0000 7f408effd700 1 --2- 172.21.15.16:0/783276793 >> v2:172.21.15.49:3300/0 conn(0x7f409006d940 0x7f409006ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:27:23.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.719+0000 7f408f7fe700 1 --2- 172.21.15.16:0/783276793 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f409015b9f0 0x7f40900b29a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4080009840 tx=0x7f4080009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dcb46b651f1f6eb8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:23.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.719+0000 7f408e7fc700 1 -- 172.21.15.16:0/783276793 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4080011830 con 0x7f409015b9f0 2023-10-16T13:27:23.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.719+0000 7f408e7fc700 1 -- 172.21.15.16:0/783276793 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4080011990 con 0x7f409015b9f0 2023-10-16T13:27:23.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.719+0000 7f408e7fc700 1 -- 172.21.15.16:0/783276793 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4080011d20 con 0x7f409015b9f0 2023-10-16T13:27:23.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.720+0000 7f4095bc9700 1 -- 172.21.15.16:0/783276793 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f409015b9f0 msgr2=0x7f40900b29a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:23.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.720+0000 7f4095bc9700 1 --2- 172.21.15.16:0/783276793 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f409015b9f0 0x7f40900b29a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4080009840 tx=0x7f4080009b50 comp rx=0 tx=0).stop 2023-10-16T13:27:23.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.720+0000 7f4095bc9700 1 -- 172.21.15.16:0/783276793 shutdown_connections 2023-10-16T13:27:23.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.720+0000 7f4095bc9700 1 --2- 172.21.15.16:0/783276793 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f409015b9f0 0x7f40900b29a0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:23.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.720+0000 7f4095bc9700 1 --2- 172.21.15.16:0/783276793 >> v2:172.21.15.49:3300/0 conn(0x7f409006d940 0x7f409006ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:23.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.720+0000 7f4095bc9700 1 -- 172.21.15.16:0/783276793 >> 172.21.15.16:0/783276793 conn(0x7f40900cc5b0 msgr2=0x7f40900cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:23.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.720+0000 7f4095bc9700 1 -- 172.21.15.16:0/783276793 shutdown_connections 2023-10-16T13:27:23.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.720+0000 7f4095bc9700 1 -- 172.21.15.16:0/783276793 wait complete. 2023-10-16T13:27:23.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.721+0000 7f4095bc9700 1 Processor -- start 2023-10-16T13:27:23.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.721+0000 7f4095bc9700 1 -- start start 2023-10-16T13:27:23.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.722+0000 7f4095bc9700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f409006d940 0x7f409015e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:23.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.722+0000 7f4095bc9700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f40900ceb70 0x7f409015e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:23.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.722+0000 7f4095bc9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40901f4650 0x7f40900c3620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:23.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.722+0000 7f4095bc9700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40900c6c00 con 0x7f40901f4650 2023-10-16T13:27:23.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.722+0000 7f4095bc9700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40901f4bc0 con 0x7f40900ceb70 2023-10-16T13:27:23.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.722+0000 7f4095bc9700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40901f4ea0 con 0x7f409006d940 2023-10-16T13:27:23.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.722+0000 7f408ffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40901f4650 0x7f40900c3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:23.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.722+0000 7f408ffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40901f4650 0x7f40900c3620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47784/0 (socket says 172.21.15.16:47784) 2023-10-16T13:27:23.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.722+0000 7f408effd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f40900ceb70 0x7f409015e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:23.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.722+0000 7f408ffff700 1 -- 172.21.15.16:0/2166865582 learned_addr learned my addr 172.21.15.16:0/2166865582 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:23.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.722+0000 7f408f7fe700 1 --2- 172.21.15.16:0/2166865582 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f409006d940 0x7f409015e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:23.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.722+0000 7f408ffff700 1 -- 172.21.15.16:0/2166865582 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f409006d940 msgr2=0x7f409015e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:23.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.722+0000 7f408ffff700 1 --2- 172.21.15.16:0/2166865582 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f409006d940 0x7f409015e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:23.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.722+0000 7f408ffff700 1 -- 172.21.15.16:0/2166865582 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f40900ceb70 msgr2=0x7f409015e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:23.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.722+0000 7f408ffff700 1 --2- 172.21.15.16:0/2166865582 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f40900ceb70 0x7f409015e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:23.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.722+0000 7f408ffff700 1 -- 172.21.15.16:0/2166865582 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40780094a0 con 0x7f40901f4650 2023-10-16T13:27:23.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.722+0000 7f408ffff700 1 --2- 172.21.15.16:0/2166865582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40901f4650 0x7f40900c3620 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7f408800caa0 tx=0x7f408800cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:23.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.723+0000 7f4087fff700 1 -- 172.21.15.16:0/2166865582 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f408800a9e0 con 0x7f40901f4650 2023-10-16T13:27:23.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.723+0000 7f4087fff700 1 -- 172.21.15.16:0/2166865582 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f408800ab40 con 0x7f40901f4650 2023-10-16T13:27:23.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.723+0000 7f4095bc9700 1 -- 172.21.15.16:0/2166865582 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40800094a0 con 0x7f40901f4650 2023-10-16T13:27:23.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.723+0000 7f4087fff700 1 -- 172.21.15.16:0/2166865582 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4088012400 con 0x7f40901f4650 2023-10-16T13:27:23.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.723+0000 7f4095bc9700 1 -- 172.21.15.16:0/2166865582 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40901f5460 con 0x7f40901f4650 2023-10-16T13:27:23.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.724+0000 7f4095bc9700 1 -- 172.21.15.16:0/2166865582 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f409004ebb0 con 0x7f40901f4650 2023-10-16T13:27:23.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.724+0000 7f4087fff700 1 -- 172.21.15.16:0/2166865582 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 94) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f408801b3b0 con 0x7f40901f4650 2023-10-16T13:27:23.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.725+0000 7f4087fff700 1 --2- 172.21.15.16:0/2166865582 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f407006b020 0x7f407006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:23.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.725+0000 7f408f7fe700 1 -- 172.21.15.16:0/2166865582 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f407006b020 msgr2=0x7f407006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1503042987 2023-10-16T13:27:23.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.725+0000 7f4087fff700 1 -- 172.21.15.16:0/2166865582 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f40880897a0 con 0x7f40901f4650 2023-10-16T13:27:23.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.725+0000 7f408f7fe700 1 --2- 172.21.15.16:0/2166865582 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f407006b020 0x7f407006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:27:23.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.729+0000 7f4087fff700 1 -- 172.21.15.16:0/2166865582 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f40880543b0 con 0x7f40901f4650 2023-10-16T13:27:23.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.926+0000 7f408f7fe700 1 -- 172.21.15.16:0/2166865582 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f407006b020 msgr2=0x7f407006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1503042987 2023-10-16T13:27:23.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.926+0000 7f408f7fe700 1 --2- 172.21.15.16:0/2166865582 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f407006b020 0x7f407006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:27:23.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.981+0000 7f4095bc9700 1 -- 172.21.15.16:0/2166865582 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f409006b6c0 con 0x7f40901f4650 2023-10-16T13:27:23.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.981+0000 7f4087fff700 1 -- 172.21.15.16:0/2166865582 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+4915 (secure 0 0 0) 0x7f40880579d0 con 0x7f40901f4650 2023-10-16T13:27:23.985 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:27:23.986 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":78,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7631,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/3370728164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":3370728164},{"type":"v1","addr":"172.21.15.49:6834","nonce":3370728164}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":73},{"gid":8638,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2297042963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":2297042963},{"type":"v1","addr":"172.21.15.16:6837","nonce":2297042963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":73},{"gid":8647,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/1920044955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":1920044955},{"type":"v1","addr":"172.21.15.49:6835","nonce":1920044955}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":73}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:31:05.836924+0000","modified":"2023-10-16T12:31:08.845101+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9204":{"gid":9204,"name":"c","rank":0,"incarnation":74,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6836/1477717813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1477717813},{"type":"v1","addr":"172.21.15.16:6836","nonce":1477717813}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2023-10-16T13:27:23.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.984+0000 7f4095bc9700 1 -- 172.21.15.16:0/2166865582 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f407006b020 msgr2=0x7f407006d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:27:23.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.984+0000 7f4095bc9700 1 --2- 172.21.15.16:0/2166865582 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f407006b020 0x7f407006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:23.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.984+0000 7f4095bc9700 1 -- 172.21.15.16:0/2166865582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40901f4650 msgr2=0x7f40900c3620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:23.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.984+0000 7f4095bc9700 1 --2- 172.21.15.16:0/2166865582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40901f4650 0x7f40900c3620 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7f408800caa0 tx=0x7f408800cdb0 comp rx=0 tx=0).stop 2023-10-16T13:27:23.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.984+0000 7f4095bc9700 1 -- 172.21.15.16:0/2166865582 shutdown_connections 2023-10-16T13:27:23.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.984+0000 7f4095bc9700 1 --2- 172.21.15.16:0/2166865582 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f407006b020 0x7f407006d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:23.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.984+0000 7f4095bc9700 1 --2- 172.21.15.16:0/2166865582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40901f4650 0x7f40900c3620 unknown :-1 s=CLOSED pgs=1822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:23.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.984+0000 7f4095bc9700 1 --2- 172.21.15.16:0/2166865582 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f409006d940 0x7f409015e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:23.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.984+0000 7f4095bc9700 1 --2- 172.21.15.16:0/2166865582 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f40900ceb70 0x7f409015e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:23.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.984+0000 7f4095bc9700 1 -- 172.21.15.16:0/2166865582 >> 172.21.15.16:0/2166865582 conn(0x7f40900cc5b0 msgr2=0x7f409006b8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:23.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.984+0000 7f4095bc9700 1 -- 172.21.15.16:0/2166865582 shutdown_connections 2023-10-16T13:27:23.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:23.984+0000 7f4095bc9700 1 -- 172.21.15.16:0/2166865582 wait complete. 2023-10-16T13:27:23.992 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 78 2023-10-16T13:27:24.000 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:27:24.106 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:27:24.104+0000 7fbf61b58200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:27:24.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.130+0000 7f16cb309700 1 Processor -- start 2023-10-16T13:27:24.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.130+0000 7f16cb309700 1 -- start start 2023-10-16T13:27:24.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.131+0000 7f16cb309700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f16c415b280 0x7f16c41613e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:24.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.131+0000 7f16cb309700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f16c40cc8a0 0x7f16c415e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:24.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.131+0000 7f16cb309700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16c4161c30 con 0x7f16c406a7e0 2023-10-16T13:27:24.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.131+0000 7f16cb309700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16c4161f10 con 0x7f16c40cc8a0 2023-10-16T13:27:24.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.131+0000 7f16cb309700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16c41621f0 con 0x7f16c415b280 2023-10-16T13:27:24.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.131+0000 7f16c88a4700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f16c40cc8a0 0x7f16c415e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:24.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.131+0000 7f16c88a4700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f16c40cc8a0 0x7f16c415e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47816/0 (socket says 172.21.15.16:47816) 2023-10-16T13:27:24.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.131+0000 7f16c98a6700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7f16c406a7e0 0x7f16c40d02b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:44274/0 (socket says 172.21.15.16:44274) 2023-10-16T13:27:24.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.131+0000 7f16c88a4700 1 -- 172.21.15.16:0/3449506364 learned_addr learned my addr 172.21.15.16:0/3449506364 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:24.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.131+0000 7f16c90a5700 1 --2- 172.21.15.16:0/3449506364 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f16c415b280 0x7f16c41613e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:24.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.132+0000 7f16c88a4700 1 -- 172.21.15.16:0/3449506364 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f16c415b280 msgr2=0x7f16c41613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:24.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.132+0000 7f16c88a4700 1 --2- 172.21.15.16:0/3449506364 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f16c415b280 0x7f16c41613e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:24.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.132+0000 7f16c88a4700 1 -- 172.21.15.16:0/3449506364 >> v1:172.21.15.49:6789/0 conn(0x7f16c406a7e0 legacy=0x7f16c40d02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:24.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.132+0000 7f16c88a4700 1 -- 172.21.15.16:0/3449506364 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16c415df00 con 0x7f16c40cc8a0 2023-10-16T13:27:24.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.132+0000 7f16bbfff700 1 -- 172.21.15.16:0/3449506364 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2562976593 0 0) 0x7f16c4161c30 con 0x7f16c406a7e0 2023-10-16T13:27:24.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.132+0000 7f16c88a4700 1 --2- 172.21.15.16:0/3449506364 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16c40cc8a0 0x7f16c415e700 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7f16c00098f0 tx=0x7f16c0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8766f361d24f951d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:24.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.132+0000 7f16bbfff700 1 -- 172.21.15.16:0/3449506364 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16c0011830 con 0x7f16c40cc8a0 2023-10-16T13:27:24.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.132+0000 7f16bbfff700 1 -- 172.21.15.16:0/3449506364 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16c0011990 con 0x7f16c40cc8a0 2023-10-16T13:27:24.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.132+0000 7f16bbfff700 1 -- 172.21.15.16:0/3449506364 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16c0011d20 con 0x7f16c40cc8a0 2023-10-16T13:27:24.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.132+0000 7f16cb309700 1 -- 172.21.15.16:0/3449506364 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16c40cc8a0 msgr2=0x7f16c415e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:24.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.132+0000 7f16cb309700 1 --2- 172.21.15.16:0/3449506364 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16c40cc8a0 0x7f16c415e700 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7f16c00098f0 tx=0x7f16c0009c00 comp rx=0 tx=0).stop 2023-10-16T13:27:24.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.133+0000 7f16cb309700 1 -- 172.21.15.16:0/3449506364 shutdown_connections 2023-10-16T13:27:24.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.133+0000 7f16cb309700 1 --2- 172.21.15.16:0/3449506364 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f16c415b280 0x7f16c41613e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:24.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.133+0000 7f16cb309700 1 --2- 172.21.15.16:0/3449506364 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16c40cc8a0 0x7f16c415e700 unknown :-1 s=CLOSED pgs=1823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:24.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.133+0000 7f16cb309700 1 -- 172.21.15.16:0/3449506364 >> 172.21.15.16:0/3449506364 conn(0x7f16c40b1fd0 msgr2=0x7f16c40b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:24.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.133+0000 7f16cb309700 1 -- 172.21.15.16:0/3449506364 shutdown_connections 2023-10-16T13:27:24.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.133+0000 7f16cb309700 1 -- 172.21.15.16:0/3449506364 wait complete. 2023-10-16T13:27:24.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.134+0000 7f16cb309700 1 Processor -- start 2023-10-16T13:27:24.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.134+0000 7f16cb309700 1 -- start start 2023-10-16T13:27:24.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.135+0000 7f16cb309700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16c406a7e0 0x7f16c4152740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:24.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.135+0000 7f16cb309700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f16c40cc8a0 0x7f16c4159180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:24.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.135+0000 7f16cb309700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f16c415b280 0x7f16c40cb110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:24.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.135+0000 7f16cb309700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16c41f61c0 con 0x7f16c406a7e0 2023-10-16T13:27:24.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.135+0000 7f16cb309700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16c4159bd0 con 0x7f16c415b280 2023-10-16T13:27:24.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.135+0000 7f16cb309700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16c41f6410 con 0x7f16c40cc8a0 2023-10-16T13:27:24.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.135+0000 7f16c90a5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16c406a7e0 0x7f16c4152740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:24.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.135+0000 7f16c88a4700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f16c40cc8a0 0x7f16c4159180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:24.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.135+0000 7f16c90a5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16c406a7e0 0x7f16c4152740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47820/0 (socket says 172.21.15.16:47820) 2023-10-16T13:27:24.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.135+0000 7f16c90a5700 1 -- 172.21.15.16:0/536780147 learned_addr learned my addr 172.21.15.16:0/536780147 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:24.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.135+0000 7f16c98a6700 1 --2- 172.21.15.16:0/536780147 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f16c415b280 0x7f16c40cb110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:24.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.135+0000 7f16c88a4700 1 -- 172.21.15.16:0/536780147 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f16c415b280 msgr2=0x7f16c40cb110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:24.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.135+0000 7f16c88a4700 1 --2- 172.21.15.16:0/536780147 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f16c415b280 0x7f16c40cb110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:24.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.135+0000 7f16c88a4700 1 -- 172.21.15.16:0/536780147 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16c406a7e0 msgr2=0x7f16c4152740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:24.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.135+0000 7f16c88a4700 1 --2- 172.21.15.16:0/536780147 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16c406a7e0 0x7f16c4152740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:24.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.135+0000 7f16c88a4700 1 -- 172.21.15.16:0/536780147 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16c0009550 con 0x7f16c40cc8a0 2023-10-16T13:27:24.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.135+0000 7f16c90a5700 1 --2- 172.21.15.16:0/536780147 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16c406a7e0 0x7f16c4152740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:27:24.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.135+0000 7f16c88a4700 1 --2- 172.21.15.16:0/536780147 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f16c40cc8a0 0x7f16c4159180 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f16c0016000 tx=0x7f16c0004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:24.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.136+0000 7f16b9ffb700 1 -- 172.21.15.16:0/536780147 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16c0004300 con 0x7f16c40cc8a0 2023-10-16T13:27:24.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.136+0000 7f16b9ffb700 1 -- 172.21.15.16:0/536780147 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16c001f070 con 0x7f16c40cc8a0 2023-10-16T13:27:24.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.136+0000 7f16cb309700 1 -- 172.21.15.16:0/536780147 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16c41f66f0 con 0x7f16c40cc8a0 2023-10-16T13:27:24.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.136+0000 7f16b9ffb700 1 -- 172.21.15.16:0/536780147 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16c00036a0 con 0x7f16c40cc8a0 2023-10-16T13:27:24.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.136+0000 7f16cb309700 1 -- 172.21.15.16:0/536780147 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16c41f6ef0 con 0x7f16c40cc8a0 2023-10-16T13:27:24.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.137+0000 7f16cb309700 1 -- 172.21.15.16:0/536780147 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16c41f6c10 con 0x7f16c40cc8a0 2023-10-16T13:27:24.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.137+0000 7f16b9ffb700 1 -- 172.21.15.16:0/536780147 <== mon.2 v2:172.21.15.49:3301/0 4 ==== mgrmap(e 94) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f16c0021030 con 0x7f16c40cc8a0 2023-10-16T13:27:24.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.138+0000 7f16b9ffb700 1 --2- 172.21.15.16:0/536780147 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f16a806b020 0x7f16a806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:24.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.138+0000 7f16c90a5700 1 -- 172.21.15.16:0/536780147 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f16a806b020 msgr2=0x7f16a806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1503042987 2023-10-16T13:27:24.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.138+0000 7f16b9ffb700 1 -- 172.21.15.16:0/536780147 <== mon.2 v2:172.21.15.49:3301/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f16c008af40 con 0x7f16c40cc8a0 2023-10-16T13:27:24.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.138+0000 7f16c90a5700 1 --2- 172.21.15.16:0/536780147 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f16a806b020 0x7f16a806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:27:24.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.142+0000 7f16b9ffb700 1 -- 172.21.15.16:0/536780147 <== mon.2 v2:172.21.15.49:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f16c41f6c10 con 0x7f16c40cc8a0 2023-10-16T13:27:24.180 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:27:24.178+0000 7f52d4a73200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:27:24.188 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:27:24.186+0000 7fbf61b58200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T13:27:24.264 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:27:24.261+0000 7f52d4a73200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T13:27:24.340 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:27:24.337+0000 7fbf61b58200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:27:24.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.339+0000 7f16c90a5700 1 -- 172.21.15.16:0/536780147 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f16a806b020 msgr2=0x7f16a806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1503042987 2023-10-16T13:27:24.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.339+0000 7f16c90a5700 1 --2- 172.21.15.16:0/536780147 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f16a806b020 0x7f16a806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:27:24.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.348+0000 7f16cb309700 1 -- 172.21.15.16:0/536780147 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f16c41f6c10 con 0x7f16c40cc8a0 2023-10-16T13:27:24.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.349+0000 7f16b9ffb700 1 -- 172.21.15.16:0/536780147 <== mon.2 v2:172.21.15.49:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v164) v1 ==== 74+0+17379 (secure 0 0 0) 0x7f16c41f6c10 con 0x7f16c40cc8a0 2023-10-16T13:27:24.352 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:27:24.353 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":164,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:27:22.071686+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:31:04.290489+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2023-10-16T12:31:04.803486+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"164","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":113,"snap_epoch":164,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/3195044045":"2023-10-17T12:31:20.854761+0000","172.21.15.16:6833/4182163087":"2023-10-17T12:31:20.854761+0000","172.21.15.16:6832/4182163087":"2023-10-17T12:31:20.854761+0000","172.21.15.16:0/4217694331":"2023-10-17T12:31:20.854761+0000","172.21.15.16:0/2516829556":"2023-10-17T12:31:20.854761+0000","172.21.15.16:0/2546955595":"2023-10-17T12:31:20.854761+0000","172.21.15.16:0/1851287351":"2023-10-17T12:31:20.854761+0000","172.21.15.16:0/3755256116":"2023-10-17T12:31:20.854761+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[{"pool":20,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:27:24.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.352+0000 7f16cb309700 1 -- 172.21.15.16:0/536780147 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f16a806b020 msgr2=0x7f16a806d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:27:24.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.352+0000 7f16cb309700 1 --2- 172.21.15.16:0/536780147 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f16a806b020 0x7f16a806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:24.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.352+0000 7f16cb309700 1 -- 172.21.15.16:0/536780147 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f16c40cc8a0 msgr2=0x7f16c4159180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:24.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.352+0000 7f16cb309700 1 --2- 172.21.15.16:0/536780147 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f16c40cc8a0 0x7f16c4159180 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f16c0016000 tx=0x7f16c0004750 comp rx=0 tx=0).stop 2023-10-16T13:27:24.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.352+0000 7f16cb309700 1 -- 172.21.15.16:0/536780147 shutdown_connections 2023-10-16T13:27:24.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.352+0000 7f16cb309700 1 --2- 172.21.15.16:0/536780147 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f16a806b020 0x7f16a806d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:24.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.352+0000 7f16cb309700 1 --2- 172.21.15.16:0/536780147 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f16c415b280 0x7f16c40cb110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:24.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.352+0000 7f16cb309700 1 --2- 172.21.15.16:0/536780147 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16c406a7e0 0x7f16c4152740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:24.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.352+0000 7f16cb309700 1 --2- 172.21.15.16:0/536780147 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f16c40cc8a0 0x7f16c4159180 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:24.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.352+0000 7f16cb309700 1 -- 172.21.15.16:0/536780147 >> 172.21.15.16:0/536780147 conn(0x7f16c40b1fd0 msgr2=0x7f16c40cd950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:24.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.352+0000 7f16cb309700 1 -- 172.21.15.16:0/536780147 shutdown_connections 2023-10-16T13:27:24.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.352+0000 7f16cb309700 1 -- 172.21.15.16:0/536780147 wait complete. 2023-10-16T13:27:24.367 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 19, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T12:31:04.290489+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '155', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 20, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T12:31:04.803486+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '164', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 113, 'snap_epoch': 164, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T13:27:24.368 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T13:27:24.422 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:27:24.419+0000 7f52d4a73200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:27:24.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.496+0000 7f61a805a700 1 Processor -- start 2023-10-16T13:27:24.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.496+0000 7f61a805a700 1 -- start start 2023-10-16T13:27:24.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.497+0000 7f61a805a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f61a015da10 0x7f61a00d1520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:24.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.497+0000 7f61a805a700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f61a00d4060 0x7f61a00cf2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:24.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.497+0000 7f61a805a700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61a00d4540 con 0x7f61a00d1d20 2023-10-16T13:27:24.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.497+0000 7f61a805a700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61a00ce7e0 con 0x7f61a00d4060 2023-10-16T13:27:24.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.497+0000 7f61a805a700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61a00ceac0 con 0x7f61a015da10 2023-10-16T13:27:24.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.497+0000 7f61a5df6700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f61a015da10 0x7f61a00d1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:24.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.497+0000 7f61a55f5700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f61a00d4060 0x7f61a00cf2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:24.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.497+0000 7f61a65f7700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7f61a00d1d20 0x7f61a00d2130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:44288/0 (socket says 172.21.15.16:44288) 2023-10-16T13:27:24.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.497+0000 7f61a5df6700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f61a015da10 0x7f61a00d1520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:36088/0 (socket says 172.21.15.16:36088) 2023-10-16T13:27:24.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.497+0000 7f61a55f5700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f61a00d4060 0x7f61a00cf2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:46892/0 (socket says 172.21.15.16:46892) 2023-10-16T13:27:24.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.497+0000 7f61a65f7700 1 -- 172.21.15.16:0/147130077 learned_addr learned my addr 172.21.15.16:0/147130077 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:27:24.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.498+0000 7f61a55f5700 1 -- 172.21.15.16:0/147130077 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f61a015da10 msgr2=0x7f61a00d1520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:24.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.498+0000 7f61a55f5700 1 --2- 172.21.15.16:0/147130077 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f61a015da10 0x7f61a00d1520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:24.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.498+0000 7f61a55f5700 1 -- 172.21.15.16:0/147130077 >> v1:172.21.15.49:6789/0 conn(0x7f61a00d1d20 legacy=0x7f61a00d2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:24.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.498+0000 7f61a4df4700 1 -- 172.21.15.16:0/147130077 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4192568903 0 0) 0x7f61a00d4540 con 0x7f61a00d1d20 2023-10-16T13:27:24.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.498+0000 7f61a55f5700 1 -- 172.21.15.16:0/147130077 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61a00ceda0 con 0x7f61a00d4060 2023-10-16T13:27:24.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.498+0000 7f61a5df6700 1 --2- 172.21.15.16:0/147130077 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f61a015da10 0x7f61a00d1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:27:24.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.498+0000 7f61a55f5700 1 --2- 172.21.15.16:0/147130077 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f61a00d4060 0x7f61a00cf2a0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f6190009840 tx=0x7f6190009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=605529eb6d1309fc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:24.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.498+0000 7f61a4df4700 1 -- 172.21.15.16:0/147130077 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6190011830 con 0x7f61a00d4060 2023-10-16T13:27:24.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.498+0000 7f61a4df4700 1 -- 172.21.15.16:0/147130077 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6190011990 con 0x7f61a00d4060 2023-10-16T13:27:24.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.498+0000 7f61a4df4700 1 -- 172.21.15.16:0/147130077 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6190011d20 con 0x7f61a00d4060 2023-10-16T13:27:24.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.499+0000 7f61a805a700 1 -- 172.21.15.16:0/147130077 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f61a00d4060 msgr2=0x7f61a00cf2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:24.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.499+0000 7f61a805a700 1 --2- 172.21.15.16:0/147130077 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f61a00d4060 0x7f61a00cf2a0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f6190009840 tx=0x7f6190009b50 comp rx=0 tx=0).stop 2023-10-16T13:27:24.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.499+0000 7f61a805a700 1 -- 172.21.15.16:0/147130077 shutdown_connections 2023-10-16T13:27:24.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.499+0000 7f61a805a700 1 --2- 172.21.15.16:0/147130077 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f61a015da10 0x7f61a00d1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:24.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.499+0000 7f61a805a700 1 --2- 172.21.15.16:0/147130077 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f61a00d4060 0x7f61a00cf2a0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:24.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.499+0000 7f61a805a700 1 -- 172.21.15.16:0/147130077 >> 172.21.15.16:0/147130077 conn(0x7f61a009c9f0 msgr2=0x7f61a00d3740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:24.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.499+0000 7f61a805a700 1 -- 172.21.15.16:0/147130077 shutdown_connections 2023-10-16T13:27:24.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.499+0000 7f61a805a700 1 -- 172.21.15.16:0/147130077 wait complete. 2023-10-16T13:27:24.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.500+0000 7f61a805a700 1 Processor -- start 2023-10-16T13:27:24.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.500+0000 7f61a805a700 1 -- start start 2023-10-16T13:27:24.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.501+0000 7f61a805a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f61a00d1d20 0x7f61a00b5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:24.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.501+0000 7f61a805a700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f61a00d4060 0x7f61a014dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:24.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.501+0000 7f61a805a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61a015da10 0x7f61a00b3950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:24.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.501+0000 7f61a805a700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61a00b5a80 con 0x7f61a015da10 2023-10-16T13:27:24.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.501+0000 7f61a805a700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61a014e130 con 0x7f61a00d4060 2023-10-16T13:27:24.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.501+0000 7f61a805a700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61a00b3e90 con 0x7f61a00d1d20 2023-10-16T13:27:24.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.501+0000 7f61a65f7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61a015da10 0x7f61a00b3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:24.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.501+0000 7f61a55f5700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f61a00d4060 0x7f61a014dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:24.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.501+0000 7f61a5df6700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f61a00d1d20 0x7f61a00b5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:24.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.501+0000 7f61a65f7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61a015da10 0x7f61a00b3950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47838/0 (socket says 172.21.15.16:47838) 2023-10-16T13:27:24.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.501+0000 7f61a55f5700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f61a00d4060 0x7f61a014dbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:46898/0 (socket says 172.21.15.16:46898) 2023-10-16T13:27:24.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.501+0000 7f61a5df6700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f61a00d1d20 0x7f61a00b5540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:36092/0 (socket says 172.21.15.16:36092) 2023-10-16T13:27:24.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.501+0000 7f61a55f5700 1 -- 172.21.15.16:0/3731372351 learned_addr learned my addr 172.21.15.16:0/3731372351 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:24.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.501+0000 7f61a55f5700 1 -- 172.21.15.16:0/3731372351 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f61a00d1d20 msgr2=0x7f61a00b5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:24.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.501+0000 7f61a55f5700 1 --2- 172.21.15.16:0/3731372351 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f61a00d1d20 0x7f61a00b5540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:24.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.501+0000 7f61a55f5700 1 -- 172.21.15.16:0/3731372351 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61a015da10 msgr2=0x7f61a00b3950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:24.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.501+0000 7f61a55f5700 1 --2- 172.21.15.16:0/3731372351 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61a015da10 0x7f61a00b3950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:24.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.501+0000 7f61a55f5700 1 -- 172.21.15.16:0/3731372351 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f619c009430 con 0x7f61a00d4060 2023-10-16T13:27:24.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.501+0000 7f61a5df6700 1 --2- 172.21.15.16:0/3731372351 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f61a00d1d20 0x7f61a00b5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:27:24.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.501+0000 7f61a65f7700 1 --2- 172.21.15.16:0/3731372351 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61a015da10 0x7f61a00b3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:27:24.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.502+0000 7f61a55f5700 1 --2- 172.21.15.16:0/3731372351 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f61a00d4060 0x7f61a014dbf0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f61900090b0 tx=0x7f6190004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:24.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.502+0000 7f61967fc700 1 -- 172.21.15.16:0/3731372351 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61900036a0 con 0x7f61a00d4060 2023-10-16T13:27:24.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.502+0000 7f61967fc700 1 -- 172.21.15.16:0/3731372351 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f619001f070 con 0x7f61a00d4060 2023-10-16T13:27:24.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.502+0000 7f61a805a700 1 -- 172.21.15.16:0/3731372351 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61900094a0 con 0x7f61a00d4060 2023-10-16T13:27:24.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.502+0000 7f61967fc700 1 -- 172.21.15.16:0/3731372351 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6190003a60 con 0x7f61a00d4060 2023-10-16T13:27:24.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.502+0000 7f61a805a700 1 -- 172.21.15.16:0/3731372351 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61a00b43a0 con 0x7f61a00d4060 2023-10-16T13:27:24.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.503+0000 7f61a805a700 1 -- 172.21.15.16:0/3731372351 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61a004ebb0 con 0x7f61a00d4060 2023-10-16T13:27:24.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.503+0000 7f61967fc700 1 -- 172.21.15.16:0/3731372351 <== mon.1 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 94) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f6190003bc0 con 0x7f61a00d4060 2023-10-16T13:27:24.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.503+0000 7f61967fc700 1 --2- 172.21.15.16:0/3731372351 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f618406b020 0x7f618406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:24.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.504+0000 7f61a5df6700 1 -- 172.21.15.16:0/3731372351 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f618406b020 msgr2=0x7f618406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1503042987 2023-10-16T13:27:24.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.504+0000 7f61967fc700 1 -- 172.21.15.16:0/3731372351 <== mon.1 v2:172.21.15.49:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f6190021070 con 0x7f61a00d4060 2023-10-16T13:27:24.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.504+0000 7f61a5df6700 1 --2- 172.21.15.16:0/3731372351 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f618406b020 0x7f618406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:27:24.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.508+0000 7f61967fc700 1 -- 172.21.15.16:0/3731372351 <== mon.1 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6190055f10 con 0x7f61a00d4060 2023-10-16T13:27:24.579 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:27:24.577+0000 7fbf61b58200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T13:27:24.657 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:27:24.655+0000 7fbf61b58200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T13:27:24.670 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:27:24.667+0000 7f52d4a73200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T13:27:24.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.705+0000 7f61a5df6700 1 -- 172.21.15.16:0/3731372351 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f618406b020 msgr2=0x7f618406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1503042987 2023-10-16T13:27:24.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.705+0000 7f61a5df6700 1 --2- 172.21.15.16:0/3731372351 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f618406b020 0x7f618406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:27:24.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.711+0000 7f61a805a700 1 -- 172.21.15.16:0/3731372351 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f61a00b6090 con 0x7f61a00d4060 2023-10-16T13:27:24.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.712+0000 7f61967fc700 1 -- 172.21.15.16:0/3731372351 <== mon.1 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v164) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6190059530 con 0x7f61a00d4060 2023-10-16T13:27:24.714 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T13:27:24.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.714+0000 7f61a805a700 1 -- 172.21.15.16:0/3731372351 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f618406b020 msgr2=0x7f618406d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:27:24.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.714+0000 7f61a805a700 1 --2- 172.21.15.16:0/3731372351 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f618406b020 0x7f618406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:24.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.714+0000 7f61a805a700 1 -- 172.21.15.16:0/3731372351 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f61a00d4060 msgr2=0x7f61a014dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:24.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.714+0000 7f61a805a700 1 --2- 172.21.15.16:0/3731372351 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f61a00d4060 0x7f61a014dbf0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f61900090b0 tx=0x7f6190004750 comp rx=0 tx=0).stop 2023-10-16T13:27:24.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.714+0000 7f61a805a700 1 -- 172.21.15.16:0/3731372351 shutdown_connections 2023-10-16T13:27:24.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.714+0000 7f61a805a700 1 --2- 172.21.15.16:0/3731372351 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f618406b020 0x7f618406d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:24.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.714+0000 7f61a805a700 1 --2- 172.21.15.16:0/3731372351 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61a015da10 0x7f61a00b3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:24.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.714+0000 7f61a805a700 1 --2- 172.21.15.16:0/3731372351 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f61a00d1d20 0x7f61a00b5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:24.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.714+0000 7f61a805a700 1 --2- 172.21.15.16:0/3731372351 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f61a00d4060 0x7f61a014dbf0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:24.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.714+0000 7f61a805a700 1 -- 172.21.15.16:0/3731372351 >> 172.21.15.16:0/3731372351 conn(0x7f61a009c9f0 msgr2=0x7f61a006b810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:24.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.714+0000 7f61a805a700 1 -- 172.21.15.16:0/3731372351 shutdown_connections 2023-10-16T13:27:24.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.714+0000 7f61a805a700 1 -- 172.21.15.16:0/3731372351 wait complete. 2023-10-16T13:27:24.729 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T13:27:24.753 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:27:24.750+0000 7f52d4a73200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T13:27:24.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.862+0000 7f2cdcafd700 1 Processor -- start 2023-10-16T13:27:24.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.863+0000 7f2cdcafd700 1 -- start start 2023-10-16T13:27:24.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.864+0000 7f2cdcafd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2cd815ec80 0x7f2cd80cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:24.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.864+0000 7f2cdcafd700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f2cd80d2110 0x7f2cd80cd1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:24.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.864+0000 7f2cdcafd700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cd80b2440 con 0x7f2cd815f060 2023-10-16T13:27:24.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.864+0000 7f2cdcafd700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cd80b2c50 con 0x7f2cd815ec80 2023-10-16T13:27:24.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.864+0000 7f2cdcafd700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cd80d00e0 con 0x7f2cd80d2110 2023-10-16T13:27:24.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.864+0000 7f2cd659c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2cd815ec80 0x7f2cd80cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:24.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.864+0000 7f2cd5d9b700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2cd815f060 0x7f2cd80b4440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37120/0 (socket says 172.21.15.16:37120) 2023-10-16T13:27:24.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.864+0000 7f2cd659c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2cd815ec80 0x7f2cd80cdec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47844/0 (socket says 172.21.15.16:47844) 2023-10-16T13:27:24.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.864+0000 7f2cd5d9b700 1 -- 172.21.15.16:0/78612446 learned_addr learned my addr 172.21.15.16:0/78612446 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:27:24.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.864+0000 7f2cd6d9d700 1 --2- 172.21.15.16:0/78612446 >> v2:172.21.15.49:3300/0 conn(0x7f2cd80d2110 0x7f2cd80cd1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:24.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.864+0000 7f2cd659c700 1 -- 172.21.15.16:0/78612446 >> v2:172.21.15.49:3300/0 conn(0x7f2cd80d2110 msgr2=0x7f2cd80cd1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:24.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.864+0000 7f2cd659c700 1 --2- 172.21.15.16:0/78612446 >> v2:172.21.15.49:3300/0 conn(0x7f2cd80d2110 0x7f2cd80cd1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:24.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.864+0000 7f2cd659c700 1 -- 172.21.15.16:0/78612446 >> v1:172.21.15.16:6789/0 conn(0x7f2cd815f060 legacy=0x7f2cd80b4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:24.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.864+0000 7f2cd659c700 1 -- 172.21.15.16:0/78612446 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cd80cef60 con 0x7f2cd815ec80 2023-10-16T13:27:24.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.864+0000 7f2cd559a700 1 -- 172.21.15.16:0/78612446 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1408986246 0 0) 0x7f2cd80b2440 con 0x7f2cd815f060 2023-10-16T13:27:24.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.865+0000 7f2cd659c700 1 --2- 172.21.15.16:0/78612446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2cd815ec80 0x7f2cd80cdec0 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7f2cd808bcb0 tx=0x7f2cc8009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40b0804d04adc636 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:24.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.865+0000 7f2cd559a700 1 -- 172.21.15.16:0/78612446 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cc8011830 con 0x7f2cd815ec80 2023-10-16T13:27:24.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.865+0000 7f2cd559a700 1 -- 172.21.15.16:0/78612446 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cc8011990 con 0x7f2cd815ec80 2023-10-16T13:27:24.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.865+0000 7f2cd559a700 1 -- 172.21.15.16:0/78612446 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cc8011d20 con 0x7f2cd815ec80 2023-10-16T13:27:24.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.865+0000 7f2cdcafd700 1 -- 172.21.15.16:0/78612446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2cd815ec80 msgr2=0x7f2cd80cdec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:24.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.865+0000 7f2cdcafd700 1 --2- 172.21.15.16:0/78612446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2cd815ec80 0x7f2cd80cdec0 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7f2cd808bcb0 tx=0x7f2cc8009b20 comp rx=0 tx=0).stop 2023-10-16T13:27:24.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.865+0000 7f2cdcafd700 1 -- 172.21.15.16:0/78612446 shutdown_connections 2023-10-16T13:27:24.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.866+0000 7f2cdcafd700 1 --2- 172.21.15.16:0/78612446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2cd815ec80 0x7f2cd80cdec0 unknown :-1 s=CLOSED pgs=1825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:24.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.866+0000 7f2cdcafd700 1 --2- 172.21.15.16:0/78612446 >> v2:172.21.15.49:3300/0 conn(0x7f2cd80d2110 0x7f2cd80cd1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:24.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.866+0000 7f2cdcafd700 1 -- 172.21.15.16:0/78612446 >> 172.21.15.16:0/78612446 conn(0x7f2cd80d15f0 msgr2=0x7f2cd80d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:24.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.866+0000 7f2cdcafd700 1 -- 172.21.15.16:0/78612446 shutdown_connections 2023-10-16T13:27:24.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.866+0000 7f2cdcafd700 1 -- 172.21.15.16:0/78612446 wait complete. 2023-10-16T13:27:24.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.867+0000 7f2cdcafd700 1 Processor -- start 2023-10-16T13:27:24.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.867+0000 7f2cdcafd700 1 -- start start 2023-10-16T13:27:24.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.868+0000 7f2cdcafd700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2cd80d2110 0x7f2cd815b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:24.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.868+0000 7f2cdcafd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2cd815ec80 0x7f2cd80c9b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:24.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.868+0000 7f2cdcafd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2cd815f060 0x7f2cd8152900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:24.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.868+0000 7f2cdcafd700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd81f42e0 con 0x7f2cd815ec80 2023-10-16T13:27:24.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.868+0000 7f2cdcafd700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd81f45c0 con 0x7f2cd815f060 2023-10-16T13:27:24.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.868+0000 7f2cdcafd700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd81f48a0 con 0x7f2cd80d2110 2023-10-16T13:27:24.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.869+0000 7f2cd5d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2cd815ec80 0x7f2cd80c9b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:24.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.869+0000 7f2cd5d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2cd815ec80 0x7f2cd80c9b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47852/0 (socket says 172.21.15.16:47852) 2023-10-16T13:27:24.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.869+0000 7f2cd5d9b700 1 -- 172.21.15.16:0/165697706 learned_addr learned my addr 172.21.15.16:0/165697706 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:24.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.869+0000 7f2cd659c700 1 --2- 172.21.15.16:0/165697706 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2cd80d2110 0x7f2cd815b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:24.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.869+0000 7f2cd6d9d700 1 --2- 172.21.15.16:0/165697706 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2cd815f060 0x7f2cd8152900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:24.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.869+0000 7f2cd6d9d700 1 -- 172.21.15.16:0/165697706 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2cd80d2110 msgr2=0x7f2cd815b6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:24.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.869+0000 7f2cd6d9d700 1 --2- 172.21.15.16:0/165697706 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2cd80d2110 0x7f2cd815b6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:24.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.869+0000 7f2cd6d9d700 1 -- 172.21.15.16:0/165697706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2cd815ec80 msgr2=0x7f2cd80c9b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:24.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.869+0000 7f2cd6d9d700 1 --2- 172.21.15.16:0/165697706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2cd815ec80 0x7f2cd80c9b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:24.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.869+0000 7f2cd6d9d700 1 -- 172.21.15.16:0/165697706 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cc80094a0 con 0x7f2cd815f060 2023-10-16T13:27:24.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.869+0000 7f2cd5d9b700 1 --2- 172.21.15.16:0/165697706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2cd815ec80 0x7f2cd80c9b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:27:24.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.869+0000 7f2cd6d9d700 1 --2- 172.21.15.16:0/165697706 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2cd815f060 0x7f2cd8152900 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f2cd000efc0 tx=0x7f2cd000c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:24.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.869+0000 7f2cc6ffd700 1 -- 172.21.15.16:0/165697706 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd000cc90 con 0x7f2cd815f060 2023-10-16T13:27:24.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.869+0000 7f2cc6ffd700 1 -- 172.21.15.16:0/165697706 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cd000f040 con 0x7f2cd815f060 2023-10-16T13:27:24.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.869+0000 7f2cc6ffd700 1 -- 172.21.15.16:0/165697706 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd00146c0 con 0x7f2cd815f060 2023-10-16T13:27:24.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.869+0000 7f2cdcafd700 1 -- 172.21.15.16:0/165697706 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cd81f4b80 con 0x7f2cd815f060 2023-10-16T13:27:24.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.870+0000 7f2cdcafd700 1 -- 172.21.15.16:0/165697706 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cd81f5140 con 0x7f2cd815f060 2023-10-16T13:27:24.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.871+0000 7f2cdcafd700 1 -- 172.21.15.16:0/165697706 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cd8154600 con 0x7f2cd815f060 2023-10-16T13:27:24.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.871+0000 7f2cc6ffd700 1 -- 172.21.15.16:0/165697706 <== mon.1 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 94) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f2cd0014820 con 0x7f2cd815f060 2023-10-16T13:27:24.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.872+0000 7f2cc6ffd700 1 --2- 172.21.15.16:0/165697706 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f2cb806b020 0x7f2cb806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:24.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.872+0000 7f2cd659c700 1 -- 172.21.15.16:0/165697706 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f2cb806b020 msgr2=0x7f2cb806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1503042987 2023-10-16T13:27:24.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.872+0000 7f2cc6ffd700 1 -- 172.21.15.16:0/165697706 <== mon.1 v2:172.21.15.49:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f2cd008a670 con 0x7f2cd815f060 2023-10-16T13:27:24.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.872+0000 7f2cd659c700 1 --2- 172.21.15.16:0/165697706 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f2cb806b020 0x7f2cb806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:27:24.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:24.876+0000 7f2cc6ffd700 1 -- 172.21.15.16:0/165697706 <== mon.1 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2cd8154600 con 0x7f2cd815f060 2023-10-16T13:27:24.935 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:27:24.933+0000 7fbf61b58200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T13:27:25.038 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:27:25.035+0000 7f52d4a73200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T13:27:25.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.073+0000 7f2cd659c700 1 -- 172.21.15.16:0/165697706 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f2cb806b020 msgr2=0x7f2cb806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1503042987 2023-10-16T13:27:25.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.073+0000 7f2cd659c700 1 --2- 172.21.15.16:0/165697706 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f2cb806b020 0x7f2cb806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:27:25.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.079+0000 7f2cdcafd700 1 -- 172.21.15.16:0/165697706 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2cd8154600 con 0x7f2cd815f060 2023-10-16T13:27:25.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.080+0000 7f2cc6ffd700 1 -- 172.21.15.16:0/165697706 <== mon.1 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v164) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2cd8154600 con 0x7f2cd815f060 2023-10-16T13:27:25.082 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-10-16T13:27:25.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.082+0000 7f2cdcafd700 1 -- 172.21.15.16:0/165697706 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f2cb806b020 msgr2=0x7f2cb806d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:27:25.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.082+0000 7f2cdcafd700 1 --2- 172.21.15.16:0/165697706 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f2cb806b020 0x7f2cb806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:25.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.082+0000 7f2cdcafd700 1 -- 172.21.15.16:0/165697706 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2cd815f060 msgr2=0x7f2cd8152900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:25.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.082+0000 7f2cdcafd700 1 --2- 172.21.15.16:0/165697706 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2cd815f060 0x7f2cd8152900 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f2cd000efc0 tx=0x7f2cd000c3a0 comp rx=0 tx=0).stop 2023-10-16T13:27:25.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.082+0000 7f2cdcafd700 1 -- 172.21.15.16:0/165697706 shutdown_connections 2023-10-16T13:27:25.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.082+0000 7f2cdcafd700 1 --2- 172.21.15.16:0/165697706 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f2cb806b020 0x7f2cb806d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:25.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.082+0000 7f2cdcafd700 1 --2- 172.21.15.16:0/165697706 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2cd815f060 0x7f2cd8152900 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:25.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.082+0000 7f2cdcafd700 1 --2- 172.21.15.16:0/165697706 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2cd80d2110 0x7f2cd815b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:25.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.082+0000 7f2cdcafd700 1 --2- 172.21.15.16:0/165697706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2cd815ec80 0x7f2cd80c9b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:25.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.082+0000 7f2cdcafd700 1 -- 172.21.15.16:0/165697706 >> 172.21.15.16:0/165697706 conn(0x7f2cd80d15f0 msgr2=0x7f2cd8156eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:25.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.082+0000 7f2cdcafd700 1 -- 172.21.15.16:0/165697706 shutdown_connections 2023-10-16T13:27:25.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.082+0000 7f2cdcafd700 1 -- 172.21.15.16:0/165697706 wait complete. 2023-10-16T13:27:25.098 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T13:27:25.169 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:27:25.166+0000 7fbf61b58200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T13:27:25.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.266+0000 7f7bd9b5c700 1 Processor -- start 2023-10-16T13:27:25.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.266+0000 7f7bd9b5c700 1 -- start start 2023-10-16T13:27:25.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.267+0000 7f7bd9b5c700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f7bd40cda60 0x7f7bd406d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:25.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.267+0000 7f7bd9b5c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7bd406dea0 0x7f7bd40ce850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:25.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.267+0000 7f7bd9b5c700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bd40ce0e0 con 0x7f7bd415b790 2023-10-16T13:27:25.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.267+0000 7f7bd9b5c700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bd406e2b0 con 0x7f7bd406dea0 2023-10-16T13:27:25.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.267+0000 7f7bd9b5c700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bd406a7e0 con 0x7f7bd40cda60 2023-10-16T13:27:25.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.267+0000 7f7bd3fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7bd406dea0 0x7f7bd40ce850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:25.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.267+0000 7f7bd3fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7bd406dea0 0x7f7bd40ce850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47864/0 (socket says 172.21.15.16:47864) 2023-10-16T13:27:25.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.267+0000 7f7bd37fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f7bd415b790 0x7f7bd415bb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37134/0 (socket says 172.21.15.16:37134) 2023-10-16T13:27:25.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.267+0000 7f7bd3fff700 1 -- 172.21.15.16:0/663496266 learned_addr learned my addr 172.21.15.16:0/663496266 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:25.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.267+0000 7f7bd3fff700 1 -- 172.21.15.16:0/663496266 >> v2:172.21.15.49:3300/0 conn(0x7f7bd40cda60 msgr2=0x7f7bd406d960 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-16T13:27:25.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.267+0000 7f7bd3fff700 1 --2- 172.21.15.16:0/663496266 >> v2:172.21.15.49:3300/0 conn(0x7f7bd40cda60 0x7f7bd406d960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:25.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.267+0000 7f7bd3fff700 1 -- 172.21.15.16:0/663496266 >> v1:172.21.15.16:6789/0 conn(0x7f7bd415b790 legacy=0x7f7bd415bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:25.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.267+0000 7f7bd3fff700 1 -- 172.21.15.16:0/663496266 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bd415c410 con 0x7f7bd406dea0 2023-10-16T13:27:25.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.267+0000 7f7bd3fff700 1 --2- 172.21.15.16:0/663496266 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bd406dea0 0x7f7bd40ce850 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7f7bd40ca140 tx=0x7f7bcc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ea8121ddddaf7c5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:25.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.268+0000 7f7bd27fc700 1 -- 172.21.15.16:0/663496266 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bcc00b870 con 0x7f7bd406dea0 2023-10-16T13:27:25.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.268+0000 7f7bd27fc700 1 -- 172.21.15.16:0/663496266 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bcc00b9d0 con 0x7f7bd406dea0 2023-10-16T13:27:25.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.268+0000 7f7bd27fc700 1 -- 172.21.15.16:0/663496266 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bcc00bd60 con 0x7f7bd406dea0 2023-10-16T13:27:25.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.269+0000 7f7bd9b5c700 1 -- 172.21.15.16:0/663496266 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bd406dea0 msgr2=0x7f7bd40ce850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:25.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.269+0000 7f7bd9b5c700 1 --2- 172.21.15.16:0/663496266 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bd406dea0 0x7f7bd40ce850 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7f7bd40ca140 tx=0x7f7bcc009bd0 comp rx=0 tx=0).stop 2023-10-16T13:27:25.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.269+0000 7f7bd9b5c700 1 -- 172.21.15.16:0/663496266 shutdown_connections 2023-10-16T13:27:25.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.269+0000 7f7bd9b5c700 1 --2- 172.21.15.16:0/663496266 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bd406dea0 0x7f7bd40ce850 unknown :-1 s=CLOSED pgs=1827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:25.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.269+0000 7f7bd9b5c700 1 --2- 172.21.15.16:0/663496266 >> v2:172.21.15.49:3300/0 conn(0x7f7bd40cda60 0x7f7bd406d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:25.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.269+0000 7f7bd9b5c700 1 -- 172.21.15.16:0/663496266 >> 172.21.15.16:0/663496266 conn(0x7f7bd40cc5e0 msgr2=0x7f7bd40cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:25.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.269+0000 7f7bd9b5c700 1 -- 172.21.15.16:0/663496266 shutdown_connections 2023-10-16T13:27:25.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.269+0000 7f7bd9b5c700 1 -- 172.21.15.16:0/663496266 wait complete. 2023-10-16T13:27:25.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.270+0000 7f7bd9b5c700 1 Processor -- start 2023-10-16T13:27:25.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.270+0000 7f7bd9b5c700 1 -- start start 2023-10-16T13:27:25.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.271+0000 7f7bd9b5c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7bd406dea0 0x7f7bd40c8de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:25.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.271+0000 7f7bd9b5c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7bd40cda60 0x7f7bd40b3850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:25.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.271+0000 7f7bd9b5c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bd415b790 0x7f7bd40b3d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:25.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.271+0000 7f7bd9b5c700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bd40c9ba0 con 0x7f7bd415b790 2023-10-16T13:27:25.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.271+0000 7f7bd9b5c700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bd415fa50 con 0x7f7bd40cda60 2023-10-16T13:27:25.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.271+0000 7f7bd9b5c700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bd40c9320 con 0x7f7bd406dea0 2023-10-16T13:27:25.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.271+0000 7f7bd3fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bd415b790 0x7f7bd40b3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:25.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.271+0000 7f7bd3fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bd415b790 0x7f7bd40b3d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47870/0 (socket says 172.21.15.16:47870) 2023-10-16T13:27:25.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.271+0000 7f7bd3fff700 1 -- 172.21.15.16:0/2197439027 learned_addr learned my addr 172.21.15.16:0/2197439027 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:25.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.271+0000 7f7bd37fe700 1 --2- 172.21.15.16:0/2197439027 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7bd406dea0 0x7f7bd40c8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:25.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.271+0000 7f7bd2ffd700 1 --2- 172.21.15.16:0/2197439027 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7bd40cda60 0x7f7bd40b3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:25.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.271+0000 7f7bd3fff700 1 -- 172.21.15.16:0/2197439027 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7bd406dea0 msgr2=0x7f7bd40c8de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:25.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.271+0000 7f7bd3fff700 1 --2- 172.21.15.16:0/2197439027 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7bd406dea0 0x7f7bd40c8de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:25.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.271+0000 7f7bd3fff700 1 -- 172.21.15.16:0/2197439027 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7bd40cda60 msgr2=0x7f7bd40b3850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:25.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.271+0000 7f7bd3fff700 1 --2- 172.21.15.16:0/2197439027 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7bd40cda60 0x7f7bd40b3850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:25.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.271+0000 7f7bd3fff700 1 -- 172.21.15.16:0/2197439027 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bcc009550 con 0x7f7bd415b790 2023-10-16T13:27:25.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.271+0000 7f7bd3fff700 1 --2- 172.21.15.16:0/2197439027 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bd415b790 0x7f7bd40b3d90 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7f7bcc014040 tx=0x7f7bcc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:25.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.272+0000 7f7bcbfff700 1 -- 172.21.15.16:0/2197439027 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bcc004520 con 0x7f7bd415b790 2023-10-16T13:27:25.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.272+0000 7f7bcbfff700 1 -- 172.21.15.16:0/2197439027 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bcc01e070 con 0x7f7bd415b790 2023-10-16T13:27:25.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.272+0000 7f7bd9b5c700 1 -- 172.21.15.16:0/2197439027 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bd40c95e0 con 0x7f7bd415b790 2023-10-16T13:27:25.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.272+0000 7f7bcbfff700 1 -- 172.21.15.16:0/2197439027 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bcc0038a0 con 0x7f7bd415b790 2023-10-16T13:27:25.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.272+0000 7f7bd9b5c700 1 -- 172.21.15.16:0/2197439027 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bd415dd50 con 0x7f7bd415b790 2023-10-16T13:27:25.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.273+0000 7f7bd9b5c700 1 -- 172.21.15.16:0/2197439027 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bd404ebb0 con 0x7f7bd415b790 2023-10-16T13:27:25.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.273+0000 7f7bcbfff700 1 -- 172.21.15.16:0/2197439027 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 94) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f7bcc003a00 con 0x7f7bd415b790 2023-10-16T13:27:25.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.274+0000 7f7bcbfff700 1 --2- 172.21.15.16:0/2197439027 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f7bb406b020 0x7f7bb406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:25.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.274+0000 7f7bd37fe700 1 -- 172.21.15.16:0/2197439027 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f7bb406b020 msgr2=0x7f7bb406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1503042987 2023-10-16T13:27:25.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.274+0000 7f7bcbfff700 1 -- 172.21.15.16:0/2197439027 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f7bcc020070 con 0x7f7bd415b790 2023-10-16T13:27:25.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.274+0000 7f7bd37fe700 1 --2- 172.21.15.16:0/2197439027 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f7bb406b020 0x7f7bb406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:27:25.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.278+0000 7f7bcbfff700 1 -- 172.21.15.16:0/2197439027 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7bcc055d00 con 0x7f7bd415b790 2023-10-16T13:27:25.285 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:27:25.282+0000 7f52d4a73200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T13:27:25.325 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:27:25.322+0000 7fbf61b58200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T13:27:25.445 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:27:25.443+0000 7fbf61b58200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T13:27:25.446 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:27:25.443+0000 7f52d4a73200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T13:27:25.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.475+0000 7f7bd37fe700 1 -- 172.21.15.16:0/2197439027 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f7bb406b020 msgr2=0x7f7bb406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1503042987 2023-10-16T13:27:25.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.475+0000 7f7bd37fe700 1 --2- 172.21.15.16:0/2197439027 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f7bb406b020 0x7f7bb406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:27:25.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.486+0000 7f7bd9b5c700 1 -- 172.21.15.16:0/2197439027 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7bd4159db0 con 0x7f7bd415b790 2023-10-16T13:27:25.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.486+0000 7f7bcbfff700 1 -- 172.21.15.16:0/2197439027 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v164) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7bcc003cb0 con 0x7f7bd415b790 2023-10-16T13:27:25.489 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-10-16T13:27:25.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.489+0000 7f7bd9b5c700 1 -- 172.21.15.16:0/2197439027 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f7bb406b020 msgr2=0x7f7bb406d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:27:25.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.489+0000 7f7bd9b5c700 1 --2- 172.21.15.16:0/2197439027 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f7bb406b020 0x7f7bb406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:25.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.489+0000 7f7bd9b5c700 1 -- 172.21.15.16:0/2197439027 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bd415b790 msgr2=0x7f7bd40b3d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:25.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.489+0000 7f7bd9b5c700 1 --2- 172.21.15.16:0/2197439027 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bd415b790 0x7f7bd40b3d90 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7f7bcc014040 tx=0x7f7bcc004750 comp rx=0 tx=0).stop 2023-10-16T13:27:25.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.489+0000 7f7bd9b5c700 1 -- 172.21.15.16:0/2197439027 shutdown_connections 2023-10-16T13:27:25.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.489+0000 7f7bd9b5c700 1 --2- 172.21.15.16:0/2197439027 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f7bb406b020 0x7f7bb406d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:25.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.489+0000 7f7bd9b5c700 1 --2- 172.21.15.16:0/2197439027 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bd415b790 0x7f7bd40b3d90 unknown :-1 s=CLOSED pgs=1828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:25.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.489+0000 7f7bd9b5c700 1 --2- 172.21.15.16:0/2197439027 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7bd406dea0 0x7f7bd40c8de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:25.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.489+0000 7f7bd9b5c700 1 --2- 172.21.15.16:0/2197439027 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7bd40cda60 0x7f7bd40b3850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:25.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.489+0000 7f7bd9b5c700 1 -- 172.21.15.16:0/2197439027 >> 172.21.15.16:0/2197439027 conn(0x7f7bd40cc5e0 msgr2=0x7f7bd41538c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:25.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.489+0000 7f7bd9b5c700 1 -- 172.21.15.16:0/2197439027 shutdown_connections 2023-10-16T13:27:25.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.489+0000 7f7bd9b5c700 1 -- 172.21.15.16:0/2197439027 wait complete. 2023-10-16T13:27:25.504 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:27:25.570 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:27:25.568+0000 7f52d4a73200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T13:27:25.609 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:27:25.607+0000 7fbf61b58200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T13:27:25.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.629+0000 7f72e8e47700 1 Processor -- start 2023-10-16T13:27:25.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.629+0000 7f72e8e47700 1 -- start start 2023-10-16T13:27:25.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.630+0000 7f72e8e47700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f72e415d8d0 0x7f72e40d1c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:25.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.630+0000 7f72e8e47700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f72e40d0750 0x7f72e406c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:25.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.630+0000 7f72e8e47700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72e406ba00 con 0x7f72e40d41c0 2023-10-16T13:27:25.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.630+0000 7f72e8e47700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72e40d2150 con 0x7f72e415d8d0 2023-10-16T13:27:25.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.630+0000 7f72e8e47700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72e40d26d0 con 0x7f72e40d0750 2023-10-16T13:27:25.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.630+0000 7f72e259c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f72e415d8d0 0x7f72e40d1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:25.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.630+0000 7f72e259c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f72e415d8d0 0x7f72e40d1c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47876/0 (socket says 172.21.15.16:47876) 2023-10-16T13:27:25.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.630+0000 7f72e259c700 1 -- 172.21.15.16:0/3538704647 learned_addr learned my addr 172.21.15.16:0/3538704647 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:25.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.630+0000 7f72e2d9d700 1 --2- 172.21.15.16:0/3538704647 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f72e40d0750 0x7f72e406c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:25.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.630+0000 7f72e259c700 1 -- 172.21.15.16:0/3538704647 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f72e40d0750 msgr2=0x7f72e406c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:25.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.630+0000 7f72e259c700 1 --2- 172.21.15.16:0/3538704647 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f72e40d0750 0x7f72e406c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:25.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.630+0000 7f72e259c700 1 -- 172.21.15.16:0/3538704647 >> v1:172.21.15.49:6789/0 conn(0x7f72e40d41c0 legacy=0x7f72e40d0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:25.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.630+0000 7f72e259c700 1 -- 172.21.15.16:0/3538704647 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72e40d23f0 con 0x7f72e415d8d0 2023-10-16T13:27:25.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.630+0000 7f72e159a700 1 -- 172.21.15.16:0/3538704647 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1848654806 0 0) 0x7f72e406ba00 con 0x7f72e40d41c0 2023-10-16T13:27:25.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.630+0000 7f72e259c700 1 --2- 172.21.15.16:0/3538704647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72e415d8d0 0x7f72e40d1c10 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7f72d40098f0 tx=0x7f72d4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5d3f5af86009aee server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:25.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.631+0000 7f72e159a700 1 -- 172.21.15.16:0/3538704647 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72d4011830 con 0x7f72e415d8d0 2023-10-16T13:27:25.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.631+0000 7f72e159a700 1 -- 172.21.15.16:0/3538704647 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72d4011990 con 0x7f72e415d8d0 2023-10-16T13:27:25.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.631+0000 7f72e159a700 1 -- 172.21.15.16:0/3538704647 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72d4011d20 con 0x7f72e415d8d0 2023-10-16T13:27:25.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.631+0000 7f72e8e47700 1 -- 172.21.15.16:0/3538704647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72e415d8d0 msgr2=0x7f72e40d1c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:25.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.631+0000 7f72e8e47700 1 --2- 172.21.15.16:0/3538704647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72e415d8d0 0x7f72e40d1c10 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7f72d40098f0 tx=0x7f72d4009c00 comp rx=0 tx=0).stop 2023-10-16T13:27:25.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.631+0000 7f72e8e47700 1 -- 172.21.15.16:0/3538704647 shutdown_connections 2023-10-16T13:27:25.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.631+0000 7f72e8e47700 1 --2- 172.21.15.16:0/3538704647 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f72e40d0750 0x7f72e406c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:25.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.631+0000 7f72e8e47700 1 --2- 172.21.15.16:0/3538704647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72e415d8d0 0x7f72e40d1c10 unknown :-1 s=CLOSED pgs=1829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:25.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.631+0000 7f72e8e47700 1 -- 172.21.15.16:0/3538704647 >> 172.21.15.16:0/3538704647 conn(0x7f72e40d3630 msgr2=0x7f72e40d3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:25.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.631+0000 7f72e8e47700 1 -- 172.21.15.16:0/3538704647 shutdown_connections 2023-10-16T13:27:25.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.631+0000 7f72e8e47700 1 -- 172.21.15.16:0/3538704647 wait complete. 2023-10-16T13:27:25.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.633+0000 7f72e8e47700 1 Processor -- start 2023-10-16T13:27:25.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.633+0000 7f72e8e47700 1 -- start start 2023-10-16T13:27:25.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.634+0000 7f72e8e47700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f72e40d0750 0x7f72e41e42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:25.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.634+0000 7f72e8e47700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f72e40d41c0 0x7f72e40c8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:25.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.634+0000 7f72e8e47700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72e415d8d0 0x7f72e415a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:25.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.634+0000 7f72e8e47700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72e41e6340 con 0x7f72e415d8d0 2023-10-16T13:27:25.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.634+0000 7f72e8e47700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72e41e4fc0 con 0x7f72e40d0750 2023-10-16T13:27:25.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.634+0000 7f72e8e47700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72e41e52a0 con 0x7f72e40d41c0 2023-10-16T13:27:25.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.634+0000 7f72e2d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72e415d8d0 0x7f72e415a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:25.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.634+0000 7f72e2d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72e415d8d0 0x7f72e415a630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47886/0 (socket says 172.21.15.16:47886) 2023-10-16T13:27:25.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.634+0000 7f72e2d9d700 1 -- 172.21.15.16:0/1417895554 learned_addr learned my addr 172.21.15.16:0/1417895554 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:25.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.634+0000 7f72e259c700 1 --2- 172.21.15.16:0/1417895554 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f72e40d0750 0x7f72e41e42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:25.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.634+0000 7f72e1d9b700 1 --2- 172.21.15.16:0/1417895554 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f72e40d41c0 0x7f72e40c8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:25.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.634+0000 7f72e2d9d700 1 -- 172.21.15.16:0/1417895554 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f72e40d41c0 msgr2=0x7f72e40c8bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:25.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.634+0000 7f72e2d9d700 1 --2- 172.21.15.16:0/1417895554 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f72e40d41c0 0x7f72e40c8bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:25.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.634+0000 7f72e2d9d700 1 -- 172.21.15.16:0/1417895554 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f72e40d0750 msgr2=0x7f72e41e42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:25.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.634+0000 7f72e2d9d700 1 --2- 172.21.15.16:0/1417895554 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f72e40d0750 0x7f72e41e42c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:25.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.634+0000 7f72e2d9d700 1 -- 172.21.15.16:0/1417895554 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72d4009550 con 0x7f72e415d8d0 2023-10-16T13:27:25.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.634+0000 7f72e2d9d700 1 --2- 172.21.15.16:0/1417895554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72e415d8d0 0x7f72e415a630 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7f72dc00efc0 tx=0x7f72dc00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:25.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.634+0000 7f72d2ffd700 1 -- 172.21.15.16:0/1417895554 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72dc009ea0 con 0x7f72e415d8d0 2023-10-16T13:27:25.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.634+0000 7f72d2ffd700 1 -- 172.21.15.16:0/1417895554 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72dc00f040 con 0x7f72e415d8d0 2023-10-16T13:27:25.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.634+0000 7f72e8e47700 1 -- 172.21.15.16:0/1417895554 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72e41e5580 con 0x7f72e415d8d0 2023-10-16T13:27:25.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.634+0000 7f72d2ffd700 1 -- 172.21.15.16:0/1417895554 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72dc00ced0 con 0x7f72e415d8d0 2023-10-16T13:27:25.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.635+0000 7f72e8e47700 1 -- 172.21.15.16:0/1417895554 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72e41e5b40 con 0x7f72e415d8d0 2023-10-16T13:27:25.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.635+0000 7f72e8e47700 1 -- 172.21.15.16:0/1417895554 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72e404ebb0 con 0x7f72e415d8d0 2023-10-16T13:27:25.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.636+0000 7f72d2ffd700 1 -- 172.21.15.16:0/1417895554 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 94) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f72dc014480 con 0x7f72e415d8d0 2023-10-16T13:27:25.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.637+0000 7f72d2ffd700 1 --2- 172.21.15.16:0/1417895554 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f72c406b020 0x7f72c406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:25.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.637+0000 7f72e259c700 1 -- 172.21.15.16:0/1417895554 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f72c406b020 msgr2=0x7f72c406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1503042987 2023-10-16T13:27:25.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.637+0000 7f72d2ffd700 1 -- 172.21.15.16:0/1417895554 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f72dc08a2d0 con 0x7f72e415d8d0 2023-10-16T13:27:25.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.637+0000 7f72e259c700 1 --2- 172.21.15.16:0/1417895554 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f72c406b020 0x7f72c406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:27:25.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.641+0000 7f72d2ffd700 1 -- 172.21.15.16:0/1417895554 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f72dc054ec0 con 0x7f72e415d8d0 2023-10-16T13:27:25.743 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:27:25.740+0000 7f52d4a73200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T13:27:25.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.838+0000 7f72e259c700 1 -- 172.21.15.16:0/1417895554 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f72c406b020 msgr2=0x7f72c406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1503042987 2023-10-16T13:27:25.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.838+0000 7f72e259c700 1 --2- 172.21.15.16:0/1417895554 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f72c406b020 0x7f72c406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:27:25.882 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:27:25.879+0000 7fbf61b58200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T13:27:25.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.891+0000 7f72e8e47700 1 -- 172.21.15.16:0/1417895554 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f72e40d2e70 con 0x7f72e415d8d0 2023-10-16T13:27:25.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.892+0000 7f72d2ffd700 1 -- 172.21.15.16:0/1417895554 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+4915 (secure 0 0 0) 0x7f72dc0584e0 con 0x7f72e415d8d0 2023-10-16T13:27:25.894 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:27:25.895 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":78,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7631,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/3370728164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":3370728164},{"type":"v1","addr":"172.21.15.49:6834","nonce":3370728164}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":73},{"gid":8638,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2297042963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":2297042963},{"type":"v1","addr":"172.21.15.16:6837","nonce":2297042963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":73},{"gid":8647,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/1920044955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":1920044955},{"type":"v1","addr":"172.21.15.49:6835","nonce":1920044955}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":73}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:31:05.836924+0000","modified":"2023-10-16T12:31:08.845101+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9204":{"gid":9204,"name":"c","rank":0,"incarnation":74,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6836/1477717813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1477717813},{"type":"v1","addr":"172.21.15.16:6836","nonce":1477717813}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2023-10-16T13:27:25.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.894+0000 7f72e8e47700 1 -- 172.21.15.16:0/1417895554 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f72c406b020 msgr2=0x7f72c406d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:27:25.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.894+0000 7f72e8e47700 1 --2- 172.21.15.16:0/1417895554 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f72c406b020 0x7f72c406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:25.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.894+0000 7f72e8e47700 1 -- 172.21.15.16:0/1417895554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72e415d8d0 msgr2=0x7f72e415a630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:25.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.894+0000 7f72e8e47700 1 --2- 172.21.15.16:0/1417895554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72e415d8d0 0x7f72e415a630 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7f72dc00efc0 tx=0x7f72dc00c3a0 comp rx=0 tx=0).stop 2023-10-16T13:27:25.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.894+0000 7f72e8e47700 1 -- 172.21.15.16:0/1417895554 shutdown_connections 2023-10-16T13:27:25.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.894+0000 7f72e8e47700 1 --2- 172.21.15.16:0/1417895554 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f72c406b020 0x7f72c406d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:25.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.894+0000 7f72e8e47700 1 --2- 172.21.15.16:0/1417895554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72e415d8d0 0x7f72e415a630 unknown :-1 s=CLOSED pgs=1830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:25.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.894+0000 7f72e8e47700 1 --2- 172.21.15.16:0/1417895554 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f72e40d0750 0x7f72e41e42c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:25.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.894+0000 7f72e8e47700 1 --2- 172.21.15.16:0/1417895554 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f72e40d41c0 0x7f72e40c8bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:25.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.894+0000 7f72e8e47700 1 -- 172.21.15.16:0/1417895554 >> 172.21.15.16:0/1417895554 conn(0x7f72e40d3630 msgr2=0x7f72e415ca70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:25.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.894+0000 7f72e8e47700 1 -- 172.21.15.16:0/1417895554 shutdown_connections 2023-10-16T13:27:25.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:25.894+0000 7f72e8e47700 1 -- 172.21.15.16:0/1417895554 wait complete. 2023-10-16T13:27:25.899 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 78 2023-10-16T13:27:25.910 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:27:26.025 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:27:26.022+0000 7f52d4a73200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T13:27:26.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.035+0000 7f6c2a122700 1 Processor -- start 2023-10-16T13:27:26.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.035+0000 7f6c2a122700 1 -- start start 2023-10-16T13:27:26.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.036+0000 7f6c2a122700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6c240cdfa0 0x7f6c240b2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:26.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.036+0000 7f6c2a122700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f6c240ce380 0x7f6c240b2980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:26.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.036+0000 7f6c2a122700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c240d0fe0 con 0x7f6c240d04a0 2023-10-16T13:27:26.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.036+0000 7f6c2a122700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c240d1210 con 0x7f6c240ce380 2023-10-16T13:27:26.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.036+0000 7f6c2a122700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c240cbc30 con 0x7f6c240cdfa0 2023-10-16T13:27:26.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.036+0000 7f6c22ffd700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f6c240ce380 0x7f6c240b2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:26.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.036+0000 7f6c237fe700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6c240cdfa0 0x7f6c240b2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:26.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.036+0000 7f6c22ffd700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f6c240ce380 0x7f6c240b2980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:46956/0 (socket says 172.21.15.16:46956) 2023-10-16T13:27:26.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.036+0000 7f6c22ffd700 1 -- 172.21.15.16:0/4087286735 learned_addr learned my addr 172.21.15.16:0/4087286735 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:26.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.036+0000 7f6c237fe700 1 --2- 172.21.15.16:0/4087286735 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6c240cdfa0 0x7f6c240b2440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:36150/0 (socket says 172.21.15.16:36150) 2023-10-16T13:27:26.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.036+0000 7f6c237fe700 1 -- 172.21.15.16:0/4087286735 >> v2:172.21.15.49:3300/0 conn(0x7f6c240ce380 msgr2=0x7f6c240b2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:26.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.036+0000 7f6c237fe700 1 --2- 172.21.15.16:0/4087286735 >> v2:172.21.15.49:3300/0 conn(0x7f6c240ce380 0x7f6c240b2980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:26.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.036+0000 7f6c237fe700 1 -- 172.21.15.16:0/4087286735 >> v1:172.21.15.49:6789/0 conn(0x7f6c240d04a0 legacy=0x7f6c240d08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:26.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.036+0000 7f6c237fe700 1 -- 172.21.15.16:0/4087286735 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c240cbf10 con 0x7f6c240cdfa0 2023-10-16T13:27:26.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.036+0000 7f6c237fe700 1 --2- 172.21.15.16:0/4087286735 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6c240cdfa0 0x7f6c240b2440 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6c14009840 tx=0x7f6c14009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e05352b77b75394 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:26.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.037+0000 7f6c227fc700 1 -- 172.21.15.16:0/4087286735 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c14011830 con 0x7f6c240cdfa0 2023-10-16T13:27:26.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.037+0000 7f6c227fc700 1 -- 172.21.15.16:0/4087286735 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c14011990 con 0x7f6c240cdfa0 2023-10-16T13:27:26.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.037+0000 7f6c227fc700 1 -- 172.21.15.16:0/4087286735 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c14011d20 con 0x7f6c240cdfa0 2023-10-16T13:27:26.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.037+0000 7f6c2a122700 1 -- 172.21.15.16:0/4087286735 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6c240cdfa0 msgr2=0x7f6c240b2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:26.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.037+0000 7f6c2a122700 1 --2- 172.21.15.16:0/4087286735 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6c240cdfa0 0x7f6c240b2440 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6c14009840 tx=0x7f6c14009b50 comp rx=0 tx=0).stop 2023-10-16T13:27:26.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.037+0000 7f6c2a122700 1 -- 172.21.15.16:0/4087286735 shutdown_connections 2023-10-16T13:27:26.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.037+0000 7f6c2a122700 1 --2- 172.21.15.16:0/4087286735 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6c240cdfa0 0x7f6c240b2440 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:26.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.037+0000 7f6c2a122700 1 --2- 172.21.15.16:0/4087286735 >> v2:172.21.15.49:3300/0 conn(0x7f6c240ce380 0x7f6c240b2980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:26.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.037+0000 7f6c2a122700 1 -- 172.21.15.16:0/4087286735 >> 172.21.15.16:0/4087286735 conn(0x7f6c240d14b0 msgr2=0x7f6c240d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:26.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.037+0000 7f6c2a122700 1 -- 172.21.15.16:0/4087286735 shutdown_connections 2023-10-16T13:27:26.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.037+0000 7f6c2a122700 1 -- 172.21.15.16:0/4087286735 wait complete. 2023-10-16T13:27:26.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.039+0000 7f6c2a122700 1 Processor -- start 2023-10-16T13:27:26.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.039+0000 7f6c2a122700 1 -- start start 2023-10-16T13:27:26.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.039+0000 7f6c2a122700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6c240cdfa0 0x7f6c241518e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:26.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.039+0000 7f6c2a122700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6c240ce380 0x7f6c240cf920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:26.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.039+0000 7f6c2a122700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c240d04a0 0x7f6c240cfe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:26.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.039+0000 7f6c2a122700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c2415dd60 con 0x7f6c240d04a0 2023-10-16T13:27:26.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.040+0000 7f6c2a122700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c24151e20 con 0x7f6c240cdfa0 2023-10-16T13:27:26.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.040+0000 7f6c2a122700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c240c67c0 con 0x7f6c240ce380 2023-10-16T13:27:26.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.040+0000 7f6c23fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c240d04a0 0x7f6c240cfe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:26.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.040+0000 7f6c237fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6c240cdfa0 0x7f6c241518e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:26.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.040+0000 7f6c22ffd700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6c240ce380 0x7f6c240cf920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:26.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.040+0000 7f6c237fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6c240cdfa0 0x7f6c241518e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:46960/0 (socket says 172.21.15.16:46960) 2023-10-16T13:27:26.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.040+0000 7f6c22ffd700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6c240ce380 0x7f6c240cf920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:36158/0 (socket says 172.21.15.16:36158) 2023-10-16T13:27:26.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.040+0000 7f6c23fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c240d04a0 0x7f6c240cfe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47902/0 (socket says 172.21.15.16:47902) 2023-10-16T13:27:26.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.040+0000 7f6c237fe700 1 -- 172.21.15.16:0/602882635 learned_addr learned my addr 172.21.15.16:0/602882635 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:26.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.040+0000 7f6c22ffd700 1 -- 172.21.15.16:0/602882635 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6c240cdfa0 msgr2=0x7f6c241518e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:26.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.040+0000 7f6c22ffd700 1 --2- 172.21.15.16:0/602882635 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6c240cdfa0 0x7f6c241518e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:26.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.040+0000 7f6c22ffd700 1 -- 172.21.15.16:0/602882635 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c240d04a0 msgr2=0x7f6c240cfe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:26.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.040+0000 7f6c22ffd700 1 --2- 172.21.15.16:0/602882635 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c240d04a0 0x7f6c240cfe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:26.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.040+0000 7f6c22ffd700 1 -- 172.21.15.16:0/602882635 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c140094a0 con 0x7f6c240ce380 2023-10-16T13:27:26.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.040+0000 7f6c237fe700 1 --2- 172.21.15.16:0/602882635 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6c240cdfa0 0x7f6c241518e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:27:26.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.040+0000 7f6c22ffd700 1 --2- 172.21.15.16:0/602882635 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6c240ce380 0x7f6c240cf920 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f6c1800e8e0 tx=0x7f6c1800ebf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:26.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.040+0000 7f6c0bfff700 1 -- 172.21.15.16:0/602882635 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c18013680 con 0x7f6c240ce380 2023-10-16T13:27:26.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.040+0000 7f6c0bfff700 1 -- 172.21.15.16:0/602882635 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c180137e0 con 0x7f6c240ce380 2023-10-16T13:27:26.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.040+0000 7f6c2a122700 1 -- 172.21.15.16:0/602882635 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c2415f180 con 0x7f6c240ce380 2023-10-16T13:27:26.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.040+0000 7f6c0bfff700 1 -- 172.21.15.16:0/602882635 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c18013b70 con 0x7f6c240ce380 2023-10-16T13:27:26.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.041+0000 7f6c2a122700 1 -- 172.21.15.16:0/602882635 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c2415eea0 con 0x7f6c240ce380 2023-10-16T13:27:26.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.041+0000 7f6c2a122700 1 -- 172.21.15.16:0/602882635 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c240c7040 con 0x7f6c240ce380 2023-10-16T13:27:26.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.042+0000 7f6c0bfff700 1 -- 172.21.15.16:0/602882635 <== mon.2 v2:172.21.15.49:3301/0 4 ==== mgrmap(e 94) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f6c18013cd0 con 0x7f6c240ce380 2023-10-16T13:27:26.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.042+0000 7f6c0bfff700 1 --2- 172.21.15.16:0/602882635 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f6c1006b020 0x7f6c1006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:26.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.042+0000 7f6c237fe700 1 -- 172.21.15.16:0/602882635 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f6c1006b020 msgr2=0x7f6c1006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1503042987 2023-10-16T13:27:26.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.042+0000 7f6c0bfff700 1 -- 172.21.15.16:0/602882635 <== mon.2 v2:172.21.15.49:3301/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f6c180899a0 con 0x7f6c240ce380 2023-10-16T13:27:26.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.042+0000 7f6c237fe700 1 --2- 172.21.15.16:0/602882635 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f6c1006b020 0x7f6c1006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:27:26.055 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:27:26.048+0000 7fbf61b58200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T13:27:26.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.047+0000 7f6c0bfff700 1 -- 172.21.15.16:0/602882635 <== mon.2 v2:172.21.15.49:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6c18054620 con 0x7f6c240ce380 2023-10-16T13:27:26.202 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:27:26.200+0000 7f52d4a73200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T13:27:26.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.243+0000 7f6c237fe700 1 -- 172.21.15.16:0/602882635 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f6c1006b020 msgr2=0x7f6c1006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1503042987 2023-10-16T13:27:26.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.243+0000 7f6c237fe700 1 --2- 172.21.15.16:0/602882635 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f6c1006b020 0x7f6c1006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:27:26.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.253+0000 7f6c2a122700 1 -- 172.21.15.16:0/602882635 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6c2404ebb0 con 0x7f6c240ce380 2023-10-16T13:27:26.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.255+0000 7f6c0bfff700 1 -- 172.21.15.16:0/602882635 <== mon.2 v2:172.21.15.49:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v164) v1 ==== 74+0+17379 (secure 0 0 0) 0x7f6c1805e070 con 0x7f6c240ce380 2023-10-16T13:27:26.258 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:27:26.258 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":164,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:27:22.071686+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:31:04.290489+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2023-10-16T12:31:04.803486+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"164","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":113,"snap_epoch":164,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/3195044045":"2023-10-17T12:31:20.854761+0000","172.21.15.16:6833/4182163087":"2023-10-17T12:31:20.854761+0000","172.21.15.16:6832/4182163087":"2023-10-17T12:31:20.854761+0000","172.21.15.16:0/4217694331":"2023-10-17T12:31:20.854761+0000","172.21.15.16:0/2516829556":"2023-10-17T12:31:20.854761+0000","172.21.15.16:0/2546955595":"2023-10-17T12:31:20.854761+0000","172.21.15.16:0/1851287351":"2023-10-17T12:31:20.854761+0000","172.21.15.16:0/3755256116":"2023-10-17T12:31:20.854761+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[{"pool":20,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:27:26.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.257+0000 7f6c2a122700 1 -- 172.21.15.16:0/602882635 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f6c1006b020 msgr2=0x7f6c1006d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:27:26.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.257+0000 7f6c2a122700 1 --2- 172.21.15.16:0/602882635 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f6c1006b020 0x7f6c1006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:26.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.257+0000 7f6c2a122700 1 -- 172.21.15.16:0/602882635 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6c240ce380 msgr2=0x7f6c240cf920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:26.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.257+0000 7f6c2a122700 1 --2- 172.21.15.16:0/602882635 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6c240ce380 0x7f6c240cf920 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f6c1800e8e0 tx=0x7f6c1800ebf0 comp rx=0 tx=0).stop 2023-10-16T13:27:26.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.257+0000 7f6c2a122700 1 -- 172.21.15.16:0/602882635 shutdown_connections 2023-10-16T13:27:26.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.257+0000 7f6c2a122700 1 --2- 172.21.15.16:0/602882635 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f6c1006b020 0x7f6c1006d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:26.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.257+0000 7f6c2a122700 1 --2- 172.21.15.16:0/602882635 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c240d04a0 0x7f6c240cfe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:26.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.257+0000 7f6c2a122700 1 --2- 172.21.15.16:0/602882635 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6c240cdfa0 0x7f6c241518e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:26.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.257+0000 7f6c2a122700 1 --2- 172.21.15.16:0/602882635 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6c240ce380 0x7f6c240cf920 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:26.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.257+0000 7f6c2a122700 1 -- 172.21.15.16:0/602882635 >> 172.21.15.16:0/602882635 conn(0x7f6c240d14b0 msgr2=0x7f6c2415f9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:26.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.257+0000 7f6c2a122700 1 -- 172.21.15.16:0/602882635 shutdown_connections 2023-10-16T13:27:26.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.257+0000 7f6c2a122700 1 -- 172.21.15.16:0/602882635 wait complete. 2023-10-16T13:27:26.273 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-16T13:27:26.274 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:27:26.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.397+0000 7f0884654700 1 Processor -- start 2023-10-16T13:27:26.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.397+0000 7f0884654700 1 -- start start 2023-10-16T13:27:26.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.398+0000 7f0884654700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f087c0d2040 0x7f087c0ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:26.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.398+0000 7f0884654700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f087c0d2490 con 0x7f087c0d2f70 2023-10-16T13:27:26.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.398+0000 7f0884654700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f087c0ce400 con 0x7f087c15d900 2023-10-16T13:27:26.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.398+0000 7f0884654700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f087c15dce0 con 0x7f087c0d2040 2023-10-16T13:27:26.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.398+0000 7f0882bf1700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f087c0d2040 0x7f087c0ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:26.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.398+0000 7f0882bf1700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f087c0d2040 0x7f087c0ce9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47912/0 (socket says 172.21.15.16:47912) 2023-10-16T13:27:26.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.398+0000 7f0882bf1700 1 -- 172.21.15.16:0/3255648881 learned_addr learned my addr 172.21.15.16:0/3255648881 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:26.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.398+0000 7f0881bef700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f087c0d2f70 0x7f087c0b6790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37184/0 (socket says 172.21.15.16:37184) 2023-10-16T13:27:26.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.399+0000 7f0882bf1700 1 -- 172.21.15.16:0/3255648881 >> v1:172.21.15.49:6789/0 conn(0x7f087c15d900 legacy=0x7f087c0d2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:26.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.399+0000 7f0882bf1700 1 -- 172.21.15.16:0/3255648881 >> v1:172.21.15.16:6789/0 conn(0x7f087c0d2f70 legacy=0x7f087c0b6790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:26.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.399+0000 7f0882bf1700 1 -- 172.21.15.16:0/3255648881 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f087c0d1280 con 0x7f087c0d2040 2023-10-16T13:27:26.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.399+0000 7f08813ee700 1 -- 172.21.15.16:0/3255648881 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1661259693 0 0) 0x7f087c0d2490 con 0x7f087c0d2f70 2023-10-16T13:27:26.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.399+0000 7f0882bf1700 1 --2- 172.21.15.16:0/3255648881 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f087c0d2040 0x7f087c0ce9f0 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7f0874009e50 tx=0x7f087400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58cc2a7d24c3e276 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:26.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.399+0000 7f08813ee700 1 -- 172.21.15.16:0/3255648881 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f087400e070 con 0x7f087c0d2040 2023-10-16T13:27:26.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.399+0000 7f08813ee700 1 -- 172.21.15.16:0/3255648881 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f087400b9c0 con 0x7f087c0d2040 2023-10-16T13:27:26.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.399+0000 7f08813ee700 1 -- 172.21.15.16:0/3255648881 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f087400bd70 con 0x7f087c0d2040 2023-10-16T13:27:26.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.400+0000 7f0884654700 1 -- 172.21.15.16:0/3255648881 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f087c0d2040 msgr2=0x7f087c0ce9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:26.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.400+0000 7f0884654700 1 --2- 172.21.15.16:0/3255648881 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f087c0d2040 0x7f087c0ce9f0 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7f0874009e50 tx=0x7f087400b330 comp rx=0 tx=0).stop 2023-10-16T13:27:26.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.400+0000 7f0884654700 1 -- 172.21.15.16:0/3255648881 shutdown_connections 2023-10-16T13:27:26.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.400+0000 7f0884654700 1 --2- 172.21.15.16:0/3255648881 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f087c0d2040 0x7f087c0ce9f0 unknown :-1 s=CLOSED pgs=1832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:26.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.400+0000 7f0884654700 1 -- 172.21.15.16:0/3255648881 >> 172.21.15.16:0/3255648881 conn(0x7f087c072330 msgr2=0x7f087c072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:26.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.400+0000 7f0884654700 1 -- 172.21.15.16:0/3255648881 shutdown_connections 2023-10-16T13:27:26.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.400+0000 7f0884654700 1 -- 172.21.15.16:0/3255648881 wait complete. 2023-10-16T13:27:26.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.401+0000 7f0884654700 1 Processor -- start 2023-10-16T13:27:26.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.401+0000 7f0884654700 1 -- start start 2023-10-16T13:27:26.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.402+0000 7f0884654700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f087c0d2040 0x7f087c15e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:26.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.402+0000 7f0884654700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f087c0d2f70 0x7f087c1f2b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:26.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.402+0000 7f0884654700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f087c15d900 0x7f087c157650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:26.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.402+0000 7f0884654700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f087c157230 con 0x7f087c0d2f70 2023-10-16T13:27:26.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.402+0000 7f0884654700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f087c1565f0 con 0x7f087c0d2040 2023-10-16T13:27:26.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.402+0000 7f0884654700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f087c156890 con 0x7f087c15d900 2023-10-16T13:27:26.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.402+0000 7f0881bef700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f087c0d2f70 0x7f087c1f2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:26.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.402+0000 7f08823f0700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f087c0d2040 0x7f087c15e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:26.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.402+0000 7f0881bef700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f087c0d2f70 0x7f087c1f2b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47916/0 (socket says 172.21.15.16:47916) 2023-10-16T13:27:26.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.402+0000 7f08823f0700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f087c0d2040 0x7f087c15e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:46976/0 (socket says 172.21.15.16:46976) 2023-10-16T13:27:26.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.402+0000 7f0881bef700 1 -- 172.21.15.16:0/393679085 learned_addr learned my addr 172.21.15.16:0/393679085 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:26.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.402+0000 7f0882bf1700 1 --2- 172.21.15.16:0/393679085 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f087c15d900 0x7f087c157650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:26.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.402+0000 7f0881bef700 1 -- 172.21.15.16:0/393679085 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f087c15d900 msgr2=0x7f087c157650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:26.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.402+0000 7f0881bef700 1 --2- 172.21.15.16:0/393679085 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f087c15d900 0x7f087c157650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:26.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.402+0000 7f0881bef700 1 -- 172.21.15.16:0/393679085 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f087c0d2040 msgr2=0x7f087c15e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:26.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.402+0000 7f0881bef700 1 --2- 172.21.15.16:0/393679085 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f087c0d2040 0x7f087c15e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:26.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.402+0000 7f0881bef700 1 -- 172.21.15.16:0/393679085 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0874009b30 con 0x7f087c0d2f70 2023-10-16T13:27:26.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.402+0000 7f0881bef700 1 --2- 172.21.15.16:0/393679085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f087c0d2f70 0x7f087c1f2b30 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7f086c00c520 tx=0x7f086c00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:26.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.403+0000 7f0872ffd700 1 -- 172.21.15.16:0/393679085 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f086c004c80 con 0x7f087c0d2f70 2023-10-16T13:27:26.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.403+0000 7f0872ffd700 1 -- 172.21.15.16:0/393679085 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f086c004e00 con 0x7f087c0d2f70 2023-10-16T13:27:26.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.403+0000 7f0872ffd700 1 -- 172.21.15.16:0/393679085 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f086c007310 con 0x7f087c0d2f70 2023-10-16T13:27:26.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.403+0000 7f0884654700 1 -- 172.21.15.16:0/393679085 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f087c156b70 con 0x7f087c0d2f70 2023-10-16T13:27:26.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.403+0000 7f0884654700 1 -- 172.21.15.16:0/393679085 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f087c15e920 con 0x7f087c0d2f70 2023-10-16T13:27:26.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.403+0000 7f0884654700 1 -- 172.21.15.16:0/393679085 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f087c157370 con 0x7f087c0d2f70 2023-10-16T13:27:26.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.404+0000 7f0872ffd700 1 -- 172.21.15.16:0/393679085 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 94) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f086c007470 con 0x7f087c0d2f70 2023-10-16T13:27:26.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.404+0000 7f0872ffd700 1 --2- 172.21.15.16:0/393679085 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f086006b020 0x7f086006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:26.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.405+0000 7f08823f0700 1 -- 172.21.15.16:0/393679085 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f086006b020 msgr2=0x7f086006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1503042987 2023-10-16T13:27:26.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.405+0000 7f0872ffd700 1 -- 172.21.15.16:0/393679085 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f086c0896d0 con 0x7f087c0d2f70 2023-10-16T13:27:26.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.405+0000 7f08823f0700 1 --2- 172.21.15.16:0/393679085 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f086006b020 0x7f086006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:27:26.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.413+0000 7f0872ffd700 1 -- 172.21.15.16:0/393679085 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f087c157370 con 0x7f087c0d2f70 2023-10-16T13:27:26.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.606+0000 7f08823f0700 1 -- 172.21.15.16:0/393679085 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f086006b020 msgr2=0x7f086006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1503042987 2023-10-16T13:27:26.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.606+0000 7f08823f0700 1 --2- 172.21.15.16:0/393679085 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f086006b020 0x7f086006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:27:26.609 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:27:26.606+0000 7fbf61b58200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T13:27:26.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.664+0000 7f0884654700 1 -- 172.21.15.16:0/393679085 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f087c157370 con 0x7f087c0d2f70 2023-10-16T13:27:26.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.664+0000 7f0872ffd700 1 -- 172.21.15.16:0/393679085 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+4915 (secure 0 0 0) 0x7f087c157370 con 0x7f087c0d2f70 2023-10-16T13:27:26.667 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:27:26.667 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":78,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7631,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/3370728164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":3370728164},{"type":"v1","addr":"172.21.15.49:6834","nonce":3370728164}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":73},{"gid":8638,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2297042963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":2297042963},{"type":"v1","addr":"172.21.15.16:6837","nonce":2297042963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":73},{"gid":8647,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/1920044955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":1920044955},{"type":"v1","addr":"172.21.15.49:6835","nonce":1920044955}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":73}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:31:05.836924+0000","modified":"2023-10-16T12:31:08.845101+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9204":{"gid":9204,"name":"c","rank":0,"incarnation":74,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6836/1477717813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1477717813},{"type":"v1","addr":"172.21.15.16:6836","nonce":1477717813}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2023-10-16T13:27:26.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.666+0000 7f0884654700 1 -- 172.21.15.16:0/393679085 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f086006b020 msgr2=0x7f086006d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:27:26.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.666+0000 7f0884654700 1 --2- 172.21.15.16:0/393679085 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f086006b020 0x7f086006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:26.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.666+0000 7f0884654700 1 -- 172.21.15.16:0/393679085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f087c0d2f70 msgr2=0x7f087c1f2b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:26.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.666+0000 7f0884654700 1 --2- 172.21.15.16:0/393679085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f087c0d2f70 0x7f087c1f2b30 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7f086c00c520 tx=0x7f086c00c830 comp rx=0 tx=0).stop 2023-10-16T13:27:26.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.666+0000 7f0884654700 1 -- 172.21.15.16:0/393679085 shutdown_connections 2023-10-16T13:27:26.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.666+0000 7f0884654700 1 --2- 172.21.15.16:0/393679085 >> [v2:172.21.15.16:6832/1503042987,v1:172.21.15.16:6833/1503042987] conn(0x7f086006b020 0x7f086006d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:26.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.666+0000 7f0884654700 1 --2- 172.21.15.16:0/393679085 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f087c15d900 0x7f087c157650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:26.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.666+0000 7f0884654700 1 --2- 172.21.15.16:0/393679085 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f087c0d2040 0x7f087c15e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:26.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.666+0000 7f0884654700 1 --2- 172.21.15.16:0/393679085 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f087c0d2f70 0x7f087c1f2b30 unknown :-1 s=CLOSED pgs=1833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:26.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.666+0000 7f0884654700 1 -- 172.21.15.16:0/393679085 >> 172.21.15.16:0/393679085 conn(0x7f087c072330 msgr2=0x7f087c15bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:26.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.666+0000 7f0884654700 1 -- 172.21.15.16:0/393679085 shutdown_connections 2023-10-16T13:27:26.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.666+0000 7f0884654700 1 -- 172.21.15.16:0/393679085 wait complete. 2023-10-16T13:27:26.672 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 78 2023-10-16T13:27:26.682 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:27:26.786 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:27:26.784+0000 7f52d4a73200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T13:27:26.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.809+0000 7f60c3aea700 1 Processor -- start 2023-10-16T13:27:26.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.809+0000 7f60c3aea700 1 -- start start 2023-10-16T13:27:26.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.811+0000 7f60c3aea700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f60bc144990 0x7f60bc06c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:26.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.811+0000 7f60c3aea700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60bc1447c0 con 0x7f60bc15f890 2023-10-16T13:27:26.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.811+0000 7f60c3aea700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60bc06d680 con 0x7f60bc0cc980 2023-10-16T13:27:26.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.811+0000 7f60c3aea700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60bc06cde0 con 0x7f60bc144990 2023-10-16T13:27:26.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.811+0000 7f60c1085700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f60bc144990 0x7f60bc06c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:26.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.811+0000 7f60c1085700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f60bc144990 0x7f60bc06c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47932/0 (socket says 172.21.15.16:47932) 2023-10-16T13:27:26.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.811+0000 7f60c1085700 1 -- 172.21.15.16:0/1923838406 learned_addr learned my addr 172.21.15.16:0/1923838406 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:26.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.811+0000 7f60c1085700 1 -- 172.21.15.16:0/1923838406 >> v1:172.21.15.49:6789/0 conn(0x7f60bc0cc980 legacy=0x7f60bc070f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:26.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.811+0000 7f60c1085700 1 -- 172.21.15.16:0/1923838406 >> v1:172.21.15.16:6789/0 conn(0x7f60bc15f890 legacy=0x7f60bc0cf1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:26.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.811+0000 7f60c1085700 1 -- 172.21.15.16:0/1923838406 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60bc06d0c0 con 0x7f60bc144990 2023-10-16T13:27:26.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.812+0000 7f60c1085700 1 --2- 172.21.15.16:0/1923838406 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60bc144990 0x7f60bc06c800 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7f60ac009920 tx=0x7f60ac009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a0adf6b777af0b7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:26.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.812+0000 7f60c0884700 1 -- 172.21.15.16:0/1923838406 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60ac00ba20 con 0x7f60bc144990 2023-10-16T13:27:26.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.813+0000 7f60c0884700 1 -- 172.21.15.16:0/1923838406 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60ac00bb80 con 0x7f60bc144990 2023-10-16T13:27:26.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.813+0000 7f60c0884700 1 -- 172.21.15.16:0/1923838406 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60ac009e70 con 0x7f60bc144990 2023-10-16T13:27:26.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.813+0000 7f60c3aea700 1 -- 172.21.15.16:0/1923838406 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60bc144990 msgr2=0x7f60bc06c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:26.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.813+0000 7f60c3aea700 1 --2- 172.21.15.16:0/1923838406 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60bc144990 0x7f60bc06c800 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7f60ac009920 tx=0x7f60ac009c30 comp rx=0 tx=0).stop 2023-10-16T13:27:26.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.813+0000 7f60c3aea700 1 -- 172.21.15.16:0/1923838406 shutdown_connections 2023-10-16T13:27:26.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.813+0000 7f60c3aea700 1 --2- 172.21.15.16:0/1923838406 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60bc144990 0x7f60bc06c800 unknown :-1 s=CLOSED pgs=1836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:26.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.813+0000 7f60c3aea700 1 -- 172.21.15.16:0/1923838406 >> 172.21.15.16:0/1923838406 conn(0x7f60bc06a7e0 msgr2=0x7f60bc06abf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:26.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.814+0000 7f60c3aea700 1 -- 172.21.15.16:0/1923838406 shutdown_connections 2023-10-16T13:27:26.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.814+0000 7f60c3aea700 1 -- 172.21.15.16:0/1923838406 wait complete. 2023-10-16T13:27:26.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.815+0000 7f60c3aea700 1 Processor -- start 2023-10-16T13:27:26.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.815+0000 7f60c3aea700 1 -- start start 2023-10-16T13:27:26.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.816+0000 7f60c3aea700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f60bc0cc980 0x7f60bc15b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:26.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.816+0000 7f60c3aea700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60bc144990 0x7f60bc0b3110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:26.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.816+0000 7f60c3aea700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f60bc15f890 0x7f60bc1e6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:26.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.816+0000 7f60c3aea700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60bc0b3880 con 0x7f60bc144990 2023-10-16T13:27:26.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.816+0000 7f60c3aea700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60bc15b790 con 0x7f60bc15f890 2023-10-16T13:27:26.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.816+0000 7f60c3aea700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60bc13e360 con 0x7f60bc0cc980 2023-10-16T13:27:26.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.816+0000 7f60c1085700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60bc144990 0x7f60bc0b3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:26.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.816+0000 7f60c1085700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60bc144990 0x7f60bc0b3110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47940/0 (socket says 172.21.15.16:47940) 2023-10-16T13:27:26.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.816+0000 7f60c1085700 1 -- 172.21.15.16:0/2563557510 learned_addr learned my addr 172.21.15.16:0/2563557510 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:26.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.816+0000 7f60c1886700 1 --2- 172.21.15.16:0/2563557510 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f60bc0cc980 0x7f60bc15b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:26.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.816+0000 7f60c2087700 1 --2- 172.21.15.16:0/2563557510 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f60bc15f890 0x7f60bc1e6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:26.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.816+0000 7f60c1085700 1 -- 172.21.15.16:0/2563557510 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f60bc0cc980 msgr2=0x7f60bc15b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:26.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.816+0000 7f60c1085700 1 --2- 172.21.15.16:0/2563557510 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f60bc0cc980 0x7f60bc15b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:26.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.816+0000 7f60c1085700 1 -- 172.21.15.16:0/2563557510 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f60bc15f890 msgr2=0x7f60bc1e6360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:26.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.816+0000 7f60c1085700 1 --2- 172.21.15.16:0/2563557510 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f60bc15f890 0x7f60bc1e6360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:26.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.816+0000 7f60c1085700 1 -- 172.21.15.16:0/2563557510 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60ac009550 con 0x7f60bc144990 2023-10-16T13:27:26.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.817+0000 7f60c1085700 1 --2- 172.21.15.16:0/2563557510 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60bc144990 0x7f60bc0b3110 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7f60ac000c00 tx=0x7f60ac004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:26.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.817+0000 7f60b27fc700 1 -- 172.21.15.16:0/2563557510 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60ac0044d0 con 0x7f60bc144990 2023-10-16T13:27:26.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.817+0000 7f60b27fc700 1 -- 172.21.15.16:0/2563557510 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60ac01f070 con 0x7f60bc144990 2023-10-16T13:27:26.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.817+0000 7f60c3aea700 1 -- 172.21.15.16:0/2563557510 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60bc0b3650 con 0x7f60bc144990 2023-10-16T13:27:26.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.817+0000 7f60b27fc700 1 -- 172.21.15.16:0/2563557510 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60ac0038e0 con 0x7f60bc144990 2023-10-16T13:27:26.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.817+0000 7f60c3aea700 1 -- 172.21.15.16:0/2563557510 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60bc1e5880 con 0x7f60bc144990 2023-10-16T13:27:26.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.818+0000 7f60c3aea700 1 -- 172.21.15.16:0/2563557510 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60bc04ebb0 con 0x7f60bc144990 2023-10-16T13:27:26.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.819+0000 7f60b27fc700 1 -- 172.21.15.16:0/2563557510 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 95) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f60ac003a40 con 0x7f60bc144990 2023-10-16T13:27:26.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.819+0000 7f60b27fc700 1 -- 172.21.15.16:0/2563557510 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 6817+0+0 (secure 0 0 0) 0x7f60ac021070 con 0x7f60bc144990 2023-10-16T13:27:26.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:26.824+0000 7f60b27fc700 1 -- 172.21.15.16:0/2563557510 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f60ac055d70 con 0x7f60bc144990 2023-10-16T13:27:26.873 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:27:26.871+0000 7f525029f700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:27:26.874 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:27:26.871+0000 7f525029f700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:27:26.874 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:27:26.871+0000 7f525029f700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:27:26.874 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:27:26.871+0000 7f525029f700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:27:26.874 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:27:26.871+0000 7f525029f700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:27:26.875 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:27:26.871+0000 7f524d299700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:27:26.875 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:27:26.871+0000 7f524d299700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:27:26.875 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:27:26.871+0000 7f524d299700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:27:26.875 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:27:26.871+0000 7f524d299700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:27:26.875 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:27:26.871+0000 7f524d299700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:27:27.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.084+0000 7f60c3aea700 1 -- 172.21.15.16:0/2563557510 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f60bc15efb0 con 0x7f60bc144990 2023-10-16T13:27:27.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.085+0000 7f60b27fc700 1 -- 172.21.15.16:0/2563557510 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+4915 (secure 0 0 0) 0x7f60ac015020 con 0x7f60bc144990 2023-10-16T13:27:27.090 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:27:27.090 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":78,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7631,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/3370728164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":3370728164},{"type":"v1","addr":"172.21.15.49:6834","nonce":3370728164}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":73},{"gid":8638,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2297042963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":2297042963},{"type":"v1","addr":"172.21.15.16:6837","nonce":2297042963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":73},{"gid":8647,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/1920044955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":1920044955},{"type":"v1","addr":"172.21.15.49:6835","nonce":1920044955}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":73}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T12:31:05.836924+0000","modified":"2023-10-16T12:31:08.845101+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9204":{"gid":9204,"name":"c","rank":0,"incarnation":74,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6836/1477717813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1477717813},{"type":"v1","addr":"172.21.15.16:6836","nonce":1477717813}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2023-10-16T13:27:27.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.087+0000 7f60c3aea700 1 -- 172.21.15.16:0/2563557510 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60bc144990 msgr2=0x7f60bc0b3110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:27.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.087+0000 7f60c3aea700 1 --2- 172.21.15.16:0/2563557510 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60bc144990 0x7f60bc0b3110 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7f60ac000c00 tx=0x7f60ac004750 comp rx=0 tx=0).stop 2023-10-16T13:27:27.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.087+0000 7f60c3aea700 1 -- 172.21.15.16:0/2563557510 shutdown_connections 2023-10-16T13:27:27.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.087+0000 7f60c3aea700 1 --2- 172.21.15.16:0/2563557510 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f60bc15f890 0x7f60bc1e6360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:27.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.087+0000 7f60c3aea700 1 --2- 172.21.15.16:0/2563557510 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f60bc0cc980 0x7f60bc15b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:27.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.087+0000 7f60c3aea700 1 --2- 172.21.15.16:0/2563557510 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60bc144990 0x7f60bc0b3110 unknown :-1 s=CLOSED pgs=1837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:27.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.087+0000 7f60c3aea700 1 -- 172.21.15.16:0/2563557510 >> 172.21.15.16:0/2563557510 conn(0x7f60bc06a7e0 msgr2=0x7f60bc15da10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:27.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.087+0000 7f60c3aea700 1 -- 172.21.15.16:0/2563557510 shutdown_connections 2023-10-16T13:27:27.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.087+0000 7f60c3aea700 1 -- 172.21.15.16:0/2563557510 wait complete. 2023-10-16T13:27:27.093 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 78 2023-10-16T13:27:27.102 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:27:27.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.224+0000 7f124acf9700 1 Processor -- start 2023-10-16T13:27:27.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.225+0000 7f124acf9700 1 -- start start 2023-10-16T13:27:27.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.225+0000 7f124acf9700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f124415d750 0x7f12440cf920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:27.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.225+0000 7f124acf9700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1244161490 0x7f12441618c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:27.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.225+0000 7f124acf9700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1244161e00 con 0x7f124415db30 2023-10-16T13:27:27.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.225+0000 7f124acf9700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12440cfe60 con 0x7f124415d750 2023-10-16T13:27:27.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.225+0000 7f124acf9700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12440d0070 con 0x7f1244161490 2023-10-16T13:27:27.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.225+0000 7f1248a95700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f124415d750 0x7f12440cf920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:27.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.226+0000 7f1248a95700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f124415d750 0x7f12440cf920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48008/0 (socket says 172.21.15.16:48008) 2023-10-16T13:27:27.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.226+0000 7f1248a95700 1 -- 172.21.15.16:0/2447305517 learned_addr learned my addr 172.21.15.16:0/2447305517 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:27.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.226+0000 7f1249296700 1 --2- 172.21.15.16:0/2447305517 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1244161490 0x7f12441618c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:27.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.226+0000 7f1248a95700 1 -- 172.21.15.16:0/2447305517 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1244161490 msgr2=0x7f12441618c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:27.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.226+0000 7f1248a95700 1 --2- 172.21.15.16:0/2447305517 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1244161490 0x7f12441618c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:27.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.226+0000 7f1248a95700 1 -- 172.21.15.16:0/2447305517 >> v1:172.21.15.16:6789/0 conn(0x7f124415db30 legacy=0x7f124406c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:27.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.226+0000 7f1248a95700 1 -- 172.21.15.16:0/2447305517 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12440d0350 con 0x7f124415d750 2023-10-16T13:27:27.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.226+0000 7f1248a95700 1 --2- 172.21.15.16:0/2447305517 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f124415d750 0x7f12440cf920 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7f1238006fd0 tx=0x7f1238009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86dccfebfa3c3419 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:27.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.226+0000 7f12437fe700 1 -- 172.21.15.16:0/2447305517 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1238005e30 con 0x7f124415d750 2023-10-16T13:27:27.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.226+0000 7f12437fe700 1 -- 172.21.15.16:0/2447305517 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1238004030 con 0x7f124415d750 2023-10-16T13:27:27.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.226+0000 7f12437fe700 1 -- 172.21.15.16:0/2447305517 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12380043e0 con 0x7f124415d750 2023-10-16T13:27:27.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.227+0000 7f124acf9700 1 -- 172.21.15.16:0/2447305517 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f124415d750 msgr2=0x7f12440cf920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:27.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.227+0000 7f124acf9700 1 --2- 172.21.15.16:0/2447305517 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f124415d750 0x7f12440cf920 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7f1238006fd0 tx=0x7f1238009bd0 comp rx=0 tx=0).stop 2023-10-16T13:27:27.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.227+0000 7f124acf9700 1 -- 172.21.15.16:0/2447305517 shutdown_connections 2023-10-16T13:27:27.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.227+0000 7f124acf9700 1 --2- 172.21.15.16:0/2447305517 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1244161490 0x7f12441618c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:27.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.227+0000 7f124acf9700 1 --2- 172.21.15.16:0/2447305517 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f124415d750 0x7f12440cf920 unknown :-1 s=CLOSED pgs=1848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:27.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.227+0000 7f124acf9700 1 -- 172.21.15.16:0/2447305517 >> 172.21.15.16:0/2447305517 conn(0x7f12440d14b0 msgr2=0x7f12440d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:27.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.227+0000 7f124acf9700 1 -- 172.21.15.16:0/2447305517 shutdown_connections 2023-10-16T13:27:27.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.227+0000 7f124acf9700 1 -- 172.21.15.16:0/2447305517 wait complete. 2023-10-16T13:27:27.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.228+0000 7f124acf9700 1 Processor -- start 2023-10-16T13:27:27.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.228+0000 7f124acf9700 1 -- start start 2023-10-16T13:27:27.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.229+0000 7f124acf9700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f124415d750 0x7f124409ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:27.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.229+0000 7f124acf9700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f124415db30 0x7f12440b57f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:27.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.229+0000 7f124acf9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1244161490 0x7f12440c5090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:27.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.229+0000 7f124acf9700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f124409b360 con 0x7f1244161490 2023-10-16T13:27:27.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.229+0000 7f124acf9700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12440c7a20 con 0x7f124415db30 2023-10-16T13:27:27.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.229+0000 7f124acf9700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12440c7d00 con 0x7f124415d750 2023-10-16T13:27:27.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.229+0000 7f1249296700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1244161490 0x7f12440c5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:27.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.229+0000 7f1248a95700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f124415d750 0x7f124409ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:27.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.229+0000 7f1249296700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1244161490 0x7f12440c5090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48018/0 (socket says 172.21.15.16:48018) 2023-10-16T13:27:27.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.229+0000 7f1248a95700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f124415d750 0x7f124409ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:36272/0 (socket says 172.21.15.16:36272) 2023-10-16T13:27:27.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.229+0000 7f1249296700 1 -- 172.21.15.16:0/2856402494 learned_addr learned my addr 172.21.15.16:0/2856402494 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:27.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.229+0000 7f1243fff700 1 --2- 172.21.15.16:0/2856402494 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f124415db30 0x7f12440b57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:27.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.229+0000 7f1249296700 1 -- 172.21.15.16:0/2856402494 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f124415d750 msgr2=0x7f124409ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:27.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.229+0000 7f1249296700 1 --2- 172.21.15.16:0/2856402494 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f124415d750 0x7f124409ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:27.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.229+0000 7f1249296700 1 -- 172.21.15.16:0/2856402494 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f124415db30 msgr2=0x7f12440b57f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:27.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.229+0000 7f1249296700 1 --2- 172.21.15.16:0/2856402494 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f124415db30 0x7f12440b57f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:27.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.229+0000 7f1249296700 1 -- 172.21.15.16:0/2856402494 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1238009500 con 0x7f1244161490 2023-10-16T13:27:27.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.230+0000 7f1249296700 1 --2- 172.21.15.16:0/2856402494 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1244161490 0x7f12440c5090 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f123c007e40 tx=0x7f123c00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:27.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.230+0000 7f12417fa700 1 -- 172.21.15.16:0/2856402494 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f123c00ed30 con 0x7f1244161490 2023-10-16T13:27:27.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.230+0000 7f12417fa700 1 -- 172.21.15.16:0/2856402494 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f123c014420 con 0x7f1244161490 2023-10-16T13:27:27.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.230+0000 7f124acf9700 1 -- 172.21.15.16:0/2856402494 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12440b4d10 con 0x7f1244161490 2023-10-16T13:27:27.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.230+0000 7f12417fa700 1 -- 172.21.15.16:0/2856402494 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f123c0147d0 con 0x7f1244161490 2023-10-16T13:27:27.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.230+0000 7f124acf9700 1 -- 172.21.15.16:0/2856402494 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12440b52d0 con 0x7f1244161490 2023-10-16T13:27:27.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.231+0000 7f124acf9700 1 -- 172.21.15.16:0/2856402494 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f124404ebb0 con 0x7f1244161490 2023-10-16T13:27:27.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.231+0000 7f12417fa700 1 -- 172.21.15.16:0/2856402494 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 95) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f123c014930 con 0x7f1244161490 2023-10-16T13:27:27.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.232+0000 7f12417fa700 1 -- 172.21.15.16:0/2856402494 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 6817+0+0 (secure 0 0 0) 0x7f123c089bc0 con 0x7f1244161490 2023-10-16T13:27:27.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.236+0000 7f12417fa700 1 -- 172.21.15.16:0/2856402494 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f123c054be0 con 0x7f1244161490 2023-10-16T13:27:27.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.440+0000 7f124acf9700 1 -- 172.21.15.16:0/2856402494 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f124406b5d0 con 0x7f1244161490 2023-10-16T13:27:27.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.441+0000 7f12417fa700 1 -- 172.21.15.16:0/2856402494 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v165) v1 ==== 74+0+17808 (secure 0 0 0) 0x7f123c058200 con 0x7f1244161490 2023-10-16T13:27:27.444 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:27:27.444 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":165,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:27:26.798718+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2023-10-16T12:31:04.290489+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2023-10-16T12:31:04.803486+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"164","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":113,"snap_epoch":164,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/756925648":"2023-10-17T13:27:26.798652+0000","172.21.15.16:0/1677462197":"2023-10-17T13:27:26.798652+0000","172.21.15.16:6833/1503042987":"2023-10-17T13:27:26.798652+0000","172.21.15.16:6832/1503042987":"2023-10-17T13:27:26.798652+0000","172.21.15.16:0/719467386":"2023-10-17T13:27:26.798652+0000","172.21.15.16:0/3389751869":"2023-10-17T13:27:26.798652+0000","172.21.15.16:0/354924692":"2023-10-17T13:27:26.798652+0000","172.21.15.16:0/1922950917":"2023-10-17T13:27:26.798652+0000","172.21.15.16:0/3195044045":"2023-10-17T12:31:20.854761+0000","172.21.15.16:6833/4182163087":"2023-10-17T12:31:20.854761+0000","172.21.15.16:6832/4182163087":"2023-10-17T12:31:20.854761+0000","172.21.15.16:0/4217694331":"2023-10-17T12:31:20.854761+0000","172.21.15.16:0/2516829556":"2023-10-17T12:31:20.854761+0000","172.21.15.16:0/2546955595":"2023-10-17T12:31:20.854761+0000","172.21.15.16:0/1851287351":"2023-10-17T12:31:20.854761+0000","172.21.15.16:0/3755256116":"2023-10-17T12:31:20.854761+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:27:27.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.443+0000 7f124acf9700 1 -- 172.21.15.16:0/2856402494 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1244161490 msgr2=0x7f12440c5090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:27.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.443+0000 7f124acf9700 1 --2- 172.21.15.16:0/2856402494 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1244161490 0x7f12440c5090 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f123c007e40 tx=0x7f123c00e3c0 comp rx=0 tx=0).stop 2023-10-16T13:27:27.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.444+0000 7f124acf9700 1 -- 172.21.15.16:0/2856402494 shutdown_connections 2023-10-16T13:27:27.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.444+0000 7f124acf9700 1 --2- 172.21.15.16:0/2856402494 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1244161490 0x7f12440c5090 unknown :-1 s=CLOSED pgs=1849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:27.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.444+0000 7f124acf9700 1 --2- 172.21.15.16:0/2856402494 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f124415d750 0x7f124409ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:27.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.444+0000 7f124acf9700 1 --2- 172.21.15.16:0/2856402494 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f124415db30 0x7f12440b57f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:27.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.444+0000 7f124acf9700 1 -- 172.21.15.16:0/2856402494 >> 172.21.15.16:0/2856402494 conn(0x7f12440d14b0 msgr2=0x7f12440d19b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:27.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.444+0000 7f124acf9700 1 -- 172.21.15.16:0/2856402494 shutdown_connections 2023-10-16T13:27:27.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.444+0000 7f124acf9700 1 -- 172.21.15.16:0/2856402494 wait complete. 2023-10-16T13:27:27.458 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-16T13:27:27.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.581+0000 7fb8bcfd7700 1 Processor -- start 2023-10-16T13:27:27.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.581+0000 7fb8bcfd7700 1 -- start start 2023-10-16T13:27:27.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.582+0000 7fb8bcfd7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb8b815ba10 0x7fb8b80d0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:27.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.582+0000 7fb8bcfd7700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8b80b0970 con 0x7fb8b80ce4f0 2023-10-16T13:27:27.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.582+0000 7fb8bcfd7700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8b80cd690 con 0x7fb8b815b630 2023-10-16T13:27:27.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.582+0000 7fb8bcfd7700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8b80cd970 con 0x7fb8b815ba10 2023-10-16T13:27:27.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.582+0000 7fb8b6d9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb8b815ba10 0x7fb8b80d0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:27.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.582+0000 7fb8b6d9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb8b815ba10 0x7fb8b80d0380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48022/0 (socket says 172.21.15.16:48022) 2023-10-16T13:27:27.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.582+0000 7fb8b6d9d700 1 -- 172.21.15.16:0/3235238539 learned_addr learned my addr 172.21.15.16:0/3235238539 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:27.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.583+0000 7fb8b6d9d700 1 -- 172.21.15.16:0/3235238539 >> v1:172.21.15.49:6789/0 conn(0x7fb8b815b630 legacy=0x7fb8b80cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:27.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.583+0000 7fb8b6d9d700 1 -- 172.21.15.16:0/3235238539 >> v1:172.21.15.16:6789/0 conn(0x7fb8b80ce4f0 legacy=0x7fb8b80b1430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:27.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.583+0000 7fb8b6d9d700 1 -- 172.21.15.16:0/3235238539 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8b80b0c30 con 0x7fb8b815ba10 2023-10-16T13:27:27.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.583+0000 7fb8b6d9d700 1 --2- 172.21.15.16:0/3235238539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8b815ba10 0x7fb8b80d0380 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7fb8b0009e50 tx=0x7fb8b000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7c2a0e9969dbfdc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:27.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.583+0000 7fb8b559a700 1 -- 172.21.15.16:0/3235238539 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8b000e070 con 0x7fb8b815ba10 2023-10-16T13:27:27.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.583+0000 7fb8b559a700 1 -- 172.21.15.16:0/3235238539 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8b000b9c0 con 0x7fb8b815ba10 2023-10-16T13:27:27.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.583+0000 7fb8b559a700 1 -- 172.21.15.16:0/3235238539 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8b000bd70 con 0x7fb8b815ba10 2023-10-16T13:27:27.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.583+0000 7fb8bcfd7700 1 -- 172.21.15.16:0/3235238539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8b815ba10 msgr2=0x7fb8b80d0380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:27.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.583+0000 7fb8bcfd7700 1 --2- 172.21.15.16:0/3235238539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8b815ba10 0x7fb8b80d0380 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7fb8b0009e50 tx=0x7fb8b000b330 comp rx=0 tx=0).stop 2023-10-16T13:27:27.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.584+0000 7fb8bcfd7700 1 -- 172.21.15.16:0/3235238539 shutdown_connections 2023-10-16T13:27:27.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.584+0000 7fb8bcfd7700 1 --2- 172.21.15.16:0/3235238539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8b815ba10 0x7fb8b80d0380 unknown :-1 s=CLOSED pgs=1851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:27.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.584+0000 7fb8bcfd7700 1 -- 172.21.15.16:0/3235238539 >> 172.21.15.16:0/3235238539 conn(0x7fb8b80d14b0 msgr2=0x7fb8b80d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:27.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.584+0000 7fb8bcfd7700 1 -- 172.21.15.16:0/3235238539 shutdown_connections 2023-10-16T13:27:27.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.584+0000 7fb8bcfd7700 1 -- 172.21.15.16:0/3235238539 wait complete. 2023-10-16T13:27:27.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.585+0000 7fb8bcfd7700 1 Processor -- start 2023-10-16T13:27:27.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.585+0000 7fb8bcfd7700 1 -- start start 2023-10-16T13:27:27.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.586+0000 7fb8bcfd7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8b80ce4f0 0x7fb8b8150bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:27.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.586+0000 7fb8bcfd7700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb8b815b630 0x7fb8b80ca160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:27.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.586+0000 7fb8bcfd7700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb8b815ba10 0x7fb8b80b4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:27.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.586+0000 7fb8bcfd7700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b8151110 con 0x7fb8b80ce4f0 2023-10-16T13:27:27.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.586+0000 7fb8bcfd7700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b815d860 con 0x7fb8b815ba10 2023-10-16T13:27:27.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.586+0000 7fb8bcfd7700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b80ca6a0 con 0x7fb8b815b630 2023-10-16T13:27:27.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.586+0000 7fb8b659c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8b80ce4f0 0x7fb8b8150bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:27.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.586+0000 7fb8b659c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8b80ce4f0 0x7fb8b8150bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48026/0 (socket says 172.21.15.16:48026) 2023-10-16T13:27:27.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.586+0000 7fb8b659c700 1 -- 172.21.15.16:0/2277211127 learned_addr learned my addr 172.21.15.16:0/2277211127 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:27.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.586+0000 7fb8b6d9d700 1 --2- 172.21.15.16:0/2277211127 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb8b815ba10 0x7fb8b80b4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:27.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.586+0000 7fb8b5d9b700 1 --2- 172.21.15.16:0/2277211127 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb8b815b630 0x7fb8b80ca160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:27.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.586+0000 7fb8b659c700 1 -- 172.21.15.16:0/2277211127 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb8b815b630 msgr2=0x7fb8b80ca160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:27.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.586+0000 7fb8b659c700 1 --2- 172.21.15.16:0/2277211127 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb8b815b630 0x7fb8b80ca160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:27.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.586+0000 7fb8b659c700 1 -- 172.21.15.16:0/2277211127 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb8b815ba10 msgr2=0x7fb8b80b4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:27.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.586+0000 7fb8b659c700 1 --2- 172.21.15.16:0/2277211127 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb8b815ba10 0x7fb8b80b4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:27.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.586+0000 7fb8b659c700 1 -- 172.21.15.16:0/2277211127 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8b0009b30 con 0x7fb8b80ce4f0 2023-10-16T13:27:27.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.586+0000 7fb8b659c700 1 --2- 172.21.15.16:0/2277211127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8b80ce4f0 0x7fb8b8150bd0 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7fb8a800c4e0 tx=0x7fb8a800c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:27.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.586+0000 7fb8a6ffd700 1 -- 172.21.15.16:0/2277211127 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8a80052d0 con 0x7fb8b80ce4f0 2023-10-16T13:27:27.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.586+0000 7fb8a6ffd700 1 -- 172.21.15.16:0/2277211127 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8a8005430 con 0x7fb8b80ce4f0 2023-10-16T13:27:27.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.586+0000 7fb8bcfd7700 1 -- 172.21.15.16:0/2277211127 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8b815cf50 con 0x7fb8b80ce4f0 2023-10-16T13:27:27.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.586+0000 7fb8a6ffd700 1 -- 172.21.15.16:0/2277211127 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8a80038c0 con 0x7fb8b80ce4f0 2023-10-16T13:27:27.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.587+0000 7fb8bcfd7700 1 -- 172.21.15.16:0/2277211127 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8b815d210 con 0x7fb8b80ce4f0 2023-10-16T13:27:27.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.588+0000 7fb8bcfd7700 1 -- 172.21.15.16:0/2277211127 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8b804ebb0 con 0x7fb8b80ce4f0 2023-10-16T13:27:27.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.588+0000 7fb8a6ffd700 1 -- 172.21.15.16:0/2277211127 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 95) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fb8a8003a20 con 0x7fb8b80ce4f0 2023-10-16T13:27:27.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.589+0000 7fb8a6ffd700 1 -- 172.21.15.16:0/2277211127 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 6817+0+0 (secure 0 0 0) 0x7fb8a8088940 con 0x7fb8b80ce4f0 2023-10-16T13:27:27.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.593+0000 7fb8a6ffd700 1 -- 172.21.15.16:0/2277211127 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb8a80538f0 con 0x7fb8b80ce4f0 2023-10-16T13:27:27.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.809+0000 7fb8a6ffd700 1 -- 172.21.15.16:0/2277211127 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 96) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fb8a80533a0 con 0x7fb8b80ce4f0 2023-10-16T13:27:27.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.810+0000 7fb8a6ffd700 1 --2- 172.21.15.16:0/2277211127 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7fb898070cc0 0x7fb8980730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:27.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.810+0000 7fb8b5d9b700 1 --2- 172.21.15.16:0/2277211127 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7fb898070cc0 0x7fb8980730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:27.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.812+0000 7fb8b5d9b700 1 --2- 172.21.15.16:0/2277211127 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7fb898070cc0 0x7fb8980730b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb8b8159880 tx=0x7fb8ac007ec0 comp rx=0 tx=0).ready entity=mgr.9510 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:27.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.850+0000 7fb8bcfd7700 1 -- 172.21.15.16:0/2277211127 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fb8b806b5d0 con 0x7fb8b80ce4f0 2023-10-16T13:27:27.853 INFO:tasks.ceph.mon.a.smithi016.stderr:2023-10-16T13:27:27.851+0000 7f9fc8535700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-16T13:27:27.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.852+0000 7fb8a6ffd700 1 -- 172.21.15.16:0/2277211127 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v79) v1 ==== 161+0+0 (secure 0 0 0) 0x7fb8a8072f20 con 0x7fb8b80ce4f0 2023-10-16T13:27:27.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.855+0000 7fb8bcfd7700 1 -- 172.21.15.16:0/2277211127 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7fb898070cc0 msgr2=0x7fb8980730b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:27.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.855+0000 7fb8bcfd7700 1 --2- 172.21.15.16:0/2277211127 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7fb898070cc0 0x7fb8980730b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb8b8159880 tx=0x7fb8ac007ec0 comp rx=0 tx=0).stop 2023-10-16T13:27:27.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.856+0000 7fb8bcfd7700 1 -- 172.21.15.16:0/2277211127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8b80ce4f0 msgr2=0x7fb8b8150bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:27.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.856+0000 7fb8bcfd7700 1 --2- 172.21.15.16:0/2277211127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8b80ce4f0 0x7fb8b8150bd0 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7fb8a800c4e0 tx=0x7fb8a800c7f0 comp rx=0 tx=0).stop 2023-10-16T13:27:27.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.856+0000 7fb8bcfd7700 1 -- 172.21.15.16:0/2277211127 shutdown_connections 2023-10-16T13:27:27.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.856+0000 7fb8bcfd7700 1 --2- 172.21.15.16:0/2277211127 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7fb898070cc0 0x7fb8980730b0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:27.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.856+0000 7fb8bcfd7700 1 --2- 172.21.15.16:0/2277211127 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb8b815ba10 0x7fb8b80b4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:27.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.856+0000 7fb8bcfd7700 1 --2- 172.21.15.16:0/2277211127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8b80ce4f0 0x7fb8b8150bd0 unknown :-1 s=CLOSED pgs=1852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:27.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.856+0000 7fb8bcfd7700 1 --2- 172.21.15.16:0/2277211127 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb8b815b630 0x7fb8b80ca160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:27.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.856+0000 7fb8bcfd7700 1 -- 172.21.15.16:0/2277211127 >> 172.21.15.16:0/2277211127 conn(0x7fb8b80d14b0 msgr2=0x7fb8b80d1890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:27.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.856+0000 7fb8bcfd7700 1 -- 172.21.15.16:0/2277211127 shutdown_connections 2023-10-16T13:27:27.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.856+0000 7fb8bcfd7700 1 -- 172.21.15.16:0/2277211127 wait complete. 2023-10-16T13:27:27.861 INFO:teuthology.orchestra.run.smithi016.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-16T13:27:27.871 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-16T13:27:27.896 INFO:tasks.ceph.mds.c.smithi016.stdout:starting mds.c at 2023-10-16T13:27:27.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.997+0000 7f276171f700 1 Processor -- start 2023-10-16T13:27:28.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.997+0000 7f276171f700 1 -- start start 2023-10-16T13:27:28.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.998+0000 7f276171f700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f275c15d8d0 0x7f275c0b2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:28.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.998+0000 7f276171f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f275c0b8fa0 0x7f275c0b6720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:28.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.998+0000 7f276171f700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f275c0b2f30 con 0x7f275c0b6c60 2023-10-16T13:27:28.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.998+0000 7f276171f700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f275c0d0bb0 con 0x7f275c15d8d0 2023-10-16T13:27:28.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.998+0000 7f276171f700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f275c15dcb0 con 0x7f275c0b8fa0 2023-10-16T13:27:28.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.998+0000 7f275affd700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f275c15d8d0 0x7f275c0b2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:28.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.998+0000 7f275a7fc700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f275c0b8fa0 0x7f275c0b6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:28.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.999+0000 7f275affd700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f275c15d8d0 0x7f275c0b2830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:47130/0 (socket says 172.21.15.16:47130) 2023-10-16T13:27:28.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.999+0000 7f275affd700 1 -- 172.21.15.16:0/2700365334 learned_addr learned my addr 172.21.15.16:0/2700365334 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:28.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.999+0000 7f275affd700 1 -- 172.21.15.16:0/2700365334 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f275c0b8fa0 msgr2=0x7f275c0b6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:28.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.999+0000 7f275affd700 1 --2- 172.21.15.16:0/2700365334 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f275c0b8fa0 0x7f275c0b6720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:28.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.999+0000 7f275affd700 1 -- 172.21.15.16:0/2700365334 >> v1:172.21.15.49:6789/0 conn(0x7f275c0b6c60 legacy=0x7f275c0ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:28.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.999+0000 7f275affd700 1 -- 172.21.15.16:0/2700365334 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f275c0d1c40 con 0x7f275c15d8d0 2023-10-16T13:27:28.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.999+0000 7f275affd700 1 --2- 172.21.15.16:0/2700365334 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f275c15d8d0 0x7f275c0b2830 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f27500098f0 tx=0x7f2750009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f52bca1e4719d5f0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:28.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.999+0000 7f2759ffb700 1 -- 172.21.15.16:0/2700365334 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2750011830 con 0x7f275c15d8d0 2023-10-16T13:27:28.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.999+0000 7f2759ffb700 1 -- 172.21.15.16:0/2700365334 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2750011990 con 0x7f275c15d8d0 2023-10-16T13:27:28.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:27.999+0000 7f2759ffb700 1 -- 172.21.15.16:0/2700365334 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2750011d20 con 0x7f275c15d8d0 2023-10-16T13:27:28.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:28.000+0000 7f276171f700 1 -- 172.21.15.16:0/2700365334 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f275c15d8d0 msgr2=0x7f275c0b2830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:28.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:28.000+0000 7f276171f700 1 --2- 172.21.15.16:0/2700365334 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f275c15d8d0 0x7f275c0b2830 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f27500098f0 tx=0x7f2750009c00 comp rx=0 tx=0).stop 2023-10-16T13:27:28.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:28.000+0000 7f276171f700 1 -- 172.21.15.16:0/2700365334 shutdown_connections 2023-10-16T13:27:28.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:28.000+0000 7f276171f700 1 --2- 172.21.15.16:0/2700365334 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f275c15d8d0 0x7f275c0b2830 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:28.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:28.000+0000 7f276171f700 1 --2- 172.21.15.16:0/2700365334 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f275c0b8fa0 0x7f275c0b6720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:28.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:28.000+0000 7f276171f700 1 -- 172.21.15.16:0/2700365334 >> 172.21.15.16:0/2700365334 conn(0x7f275c072330 msgr2=0x7f275c072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:28.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:28.000+0000 7f276171f700 1 -- 172.21.15.16:0/2700365334 shutdown_connections 2023-10-16T13:27:28.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:28.000+0000 7f276171f700 1 -- 172.21.15.16:0/2700365334 wait complete. 2023-10-16T13:27:28.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:28.001+0000 7f276171f700 1 Processor -- start 2023-10-16T13:27:28.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:28.002+0000 7f276171f700 1 -- start start 2023-10-16T13:27:28.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:28.002+0000 7f276171f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f275c0b6c60 0x7f275c0d0170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:28.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:28.003+0000 7f276171f700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f275c0b8fa0 0x7f275c1f6c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:28.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:28.003+0000 7f276171f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f275c15d8d0 0x7f275c0b4170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:28.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:28.003+0000 7f276171f700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f275c1f64d0 con 0x7f275c15d8d0 2023-10-16T13:27:28.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:28.003+0000 7f276171f700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f275c1f37c0 con 0x7f275c0b8fa0 2023-10-16T13:27:28.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:28.003+0000 7f276171f700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f275c1f3aa0 con 0x7f275c0b6c60 2023-10-16T13:27:28.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:28.003+0000 7f275b7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f275c15d8d0 0x7f275c0b4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:28.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:28.003+0000 7f275b7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f275c15d8d0 0x7f275c0b4170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48078/0 (socket says 172.21.15.16:48078) 2023-10-16T13:27:28.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:28.003+0000 7f275b7fe700 1 -- 172.21.15.16:0/297061462 learned_addr learned my addr 172.21.15.16:0/297061462 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:28.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:28.003+0000 7f275a7fc700 1 --2- 172.21.15.16:0/297061462 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f275c0b8fa0 0x7f275c1f6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:28.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:28.003+0000 7f275affd700 1 --2- 172.21.15.16:0/297061462 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f275c0b6c60 0x7f275c0d0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:28.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:28.003+0000 7f275b7fe700 1 -- 172.21.15.16:0/297061462 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f275c0b6c60 msgr2=0x7f275c0d0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:28.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:28.003+0000 7f275b7fe700 1 --2- 172.21.15.16:0/297061462 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f275c0b6c60 0x7f275c0d0170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:28.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:28.003+0000 7f275b7fe700 1 -- 172.21.15.16:0/297061462 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f275c0b8fa0 msgr2=0x7f275c1f6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:28.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:28.003+0000 7f275b7fe700 1 --2- 172.21.15.16:0/297061462 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f275c0b8fa0 0x7f275c1f6c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:28.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:28.003+0000 7f275b7fe700 1 -- 172.21.15.16:0/297061462 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2750009550 con 0x7f275c15d8d0 2023-10-16T13:27:28.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:28.003+0000 7f275b7fe700 1 --2- 172.21.15.16:0/297061462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f275c15d8d0 0x7f275c0b4170 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7f275400ca70 tx=0x7f275400cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:28.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:28.003+0000 7f27437fe700 1 -- 172.21.15.16:0/297061462 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2754003c00 con 0x7f275c15d8d0 2023-10-16T13:27:28.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:28.003+0000 7f27437fe700 1 -- 172.21.15.16:0/297061462 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2754003d60 con 0x7f275c15d8d0 2023-10-16T13:27:28.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:28.003+0000 7f276171f700 1 -- 172.21.15.16:0/297061462 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f275c1f3d80 con 0x7f275c15d8d0 2023-10-16T13:27:28.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:28.003+0000 7f27437fe700 1 -- 172.21.15.16:0/297061462 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f275400aa20 con 0x7f275c15d8d0 2023-10-16T13:27:28.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:28.004+0000 7f276171f700 1 -- 172.21.15.16:0/297061462 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f275c0d06b0 con 0x7f275c15d8d0 2023-10-16T13:27:28.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:28.004+0000 7f276171f700 1 -- 172.21.15.16:0/297061462 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f275c1f6b00 con 0x7f275c15d8d0 2023-10-16T13:27:28.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:28.005+0000 7f27437fe700 1 -- 172.21.15.16:0/297061462 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 96) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f275400ab80 con 0x7f275c15d8d0 2023-10-16T13:27:28.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:28.005+0000 7f27437fe700 1 --2- 172.21.15.16:0/297061462 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f274806ad50 0x7f274806d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:28.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:28.005+0000 7f27437fe700 1 -- 172.21.15.16:0/297061462 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(166..166 src has 1..166) v4 ==== 6903+0+0 (secure 0 0 0) 0x7f27540895e0 con 0x7f275c15d8d0 2023-10-16T13:27:28.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:28.005+0000 7f275affd700 1 --2- 172.21.15.16:0/297061462 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f274806ad50 0x7f274806d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:28.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:28.006+0000 7f275affd700 1 --2- 172.21.15.16:0/297061462 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f274806ad50 0x7f274806d210 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f2750009c40 tx=0x7f2750008030 comp rx=0 tx=0).ready entity=mgr.9510 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:28.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:28.010+0000 7f27437fe700 1 -- 172.21.15.16:0/297061462 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2754054450 con 0x7f275c15d8d0 2023-10-16T13:27:28.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:28.261+0000 7f276171f700 1 -- 172.21.15.16:0/297061462 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f275c04ebb0 con 0x7f275c15d8d0 2023-10-16T13:27:28.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:28.853+0000 7f27437fe700 1 -- 172.21.15.16:0/297061462 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v80) v1 ==== 104+0+0 (secure 0 0 0) 0x7f2754057a70 con 0x7f275c15d8d0 2023-10-16T13:27:28.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:28.857+0000 7f276171f700 1 -- 172.21.15.16:0/297061462 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f274806ad50 msgr2=0x7f274806d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:28.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:28.857+0000 7f276171f700 1 --2- 172.21.15.16:0/297061462 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f274806ad50 0x7f274806d210 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f2750009c40 tx=0x7f2750008030 comp rx=0 tx=0).stop 2023-10-16T13:27:28.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:28.857+0000 7f276171f700 1 -- 172.21.15.16:0/297061462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f275c15d8d0 msgr2=0x7f275c0b4170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:28.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:28.857+0000 7f276171f700 1 --2- 172.21.15.16:0/297061462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f275c15d8d0 0x7f275c0b4170 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7f275400ca70 tx=0x7f275400cd80 comp rx=0 tx=0).stop 2023-10-16T13:27:28.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:28.857+0000 7f276171f700 1 -- 172.21.15.16:0/297061462 shutdown_connections 2023-10-16T13:27:28.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:28.857+0000 7f276171f700 1 --2- 172.21.15.16:0/297061462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f275c15d8d0 0x7f275c0b4170 unknown :-1 s=CLOSED pgs=1855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:28.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:28.857+0000 7f276171f700 1 --2- 172.21.15.16:0/297061462 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f275c0b6c60 0x7f275c0d0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:28.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:28.857+0000 7f276171f700 1 --2- 172.21.15.16:0/297061462 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f274806ad50 0x7f274806d210 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:28.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:28.857+0000 7f276171f700 1 --2- 172.21.15.16:0/297061462 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f275c0b8fa0 0x7f275c1f6c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:28.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:28.857+0000 7f276171f700 1 -- 172.21.15.16:0/297061462 >> 172.21.15.16:0/297061462 conn(0x7f275c072330 msgr2=0x7f275c0b7d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:28.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:28.857+0000 7f276171f700 1 -- 172.21.15.16:0/297061462 shutdown_connections 2023-10-16T13:27:28.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:28.857+0000 7f276171f700 1 -- 172.21.15.16:0/297061462 wait complete. 2023-10-16T13:27:28.876 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-16T13:27:29.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.050+0000 7f000a164700 1 Processor -- start 2023-10-16T13:27:29.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.050+0000 7f000a164700 1 -- start start 2023-10-16T13:27:29.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.051+0000 7f000a164700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f000406d980 0x7f000415efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:29.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.051+0000 7f000a164700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00040cdeb0 con 0x7f000406a7e0 2023-10-16T13:27:29.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.051+0000 7f000a164700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00041613e0 con 0x7f0004160e60 2023-10-16T13:27:29.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.051+0000 7f000a164700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00041582d0 con 0x7f000406d980 2023-10-16T13:27:29.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.051+0000 7f0003fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f000406a7e0 0x7f0004161870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37380/0 (socket says 172.21.15.16:37380) 2023-10-16T13:27:29.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.051+0000 7f0003fff700 1 -- 172.21.15.16:0/1339321018 learned_addr learned my addr 172.21.15.16:0/1339321018 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:27:29.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.051+0000 7f00027fc700 1 -- 172.21.15.16:0/1339321018 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1281386518 0 0) 0x7f00040cdeb0 con 0x7f000406a7e0 2023-10-16T13:27:29.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.051+0000 7f00027fc700 1 -- 172.21.15.16:0/1339321018 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efff00035b0 con 0x7f000406a7e0 2023-10-16T13:27:29.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.051+0000 7f0002ffd700 1 --2- 172.21.15.16:0/1339321018 >> v2:172.21.15.49:3300/0 conn(0x7f000406d980 0x7f000415efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:29.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.052+0000 7f00027fc700 1 -- 172.21.15.16:0/1339321018 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 97342510 0 0) 0x7efff00035b0 con 0x7f000406a7e0 2023-10-16T13:27:29.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.052+0000 7f00027fc700 1 -- 172.21.15.16:0/1339321018 >> v2:172.21.15.49:3300/0 conn(0x7f000406d980 msgr2=0x7f000415efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:29.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.052+0000 7f00027fc700 1 --2- 172.21.15.16:0/1339321018 >> v2:172.21.15.49:3300/0 conn(0x7f000406d980 0x7f000415efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:29.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.052+0000 7f00027fc700 1 -- 172.21.15.16:0/1339321018 >> v1:172.21.15.49:6789/0 conn(0x7f0004160e60 legacy=0x7f000415e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:29.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.052+0000 7f00027fc700 1 -- 172.21.15.16:0/1339321018 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00040ce100 con 0x7f000406a7e0 2023-10-16T13:27:29.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.052+0000 7f00027fc700 1 -- 172.21.15.16:0/1339321018 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7efffc0019f0 con 0x7f000406a7e0 2023-10-16T13:27:29.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.052+0000 7f00027fc700 1 -- 172.21.15.16:0/1339321018 >> v1:172.21.15.16:6789/0 conn(0x7f000406a7e0 legacy=0x7f0004161870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:29.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.052+0000 7f00027fc700 1 --2- 172.21.15.16:0/1339321018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efff0003a00 0x7efff0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:29.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.052+0000 7f00027fc700 1 --2- 172.21.15.16:0/1339321018 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efff0004850 0x7efff0004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:29.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.052+0000 7f00027fc700 1 --2- 172.21.15.16:0/1339321018 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efff0005210 0x7efff000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:29.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.052+0000 7f00027fc700 1 -- 172.21.15.16:0/1339321018 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00041582d0 con 0x7efff0003a00 2023-10-16T13:27:29.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.052+0000 7f00027fc700 1 -- 172.21.15.16:0/1339321018 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efff000c1b0 con 0x7efff0004850 2023-10-16T13:27:29.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.052+0000 7f00027fc700 1 -- 172.21.15.16:0/1339321018 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efff000c370 con 0x7efff0005210 2023-10-16T13:27:29.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.052+0000 7f00037fe700 1 --2- 172.21.15.16:0/1339321018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efff0003a00 0x7efff0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:29.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.052+0000 7f0002ffd700 1 --2- 172.21.15.16:0/1339321018 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efff0004850 0x7efff0004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:29.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.052+0000 7f0003fff700 1 --2- 172.21.15.16:0/1339321018 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efff0005210 0x7efff000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:29.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.052+0000 7f00037fe700 1 -- 172.21.15.16:0/1339321018 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efff0005210 msgr2=0x7efff000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:29.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.052+0000 7f00037fe700 1 --2- 172.21.15.16:0/1339321018 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efff0005210 0x7efff000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:29.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.052+0000 7f00037fe700 1 -- 172.21.15.16:0/1339321018 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efff0004850 msgr2=0x7efff0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:29.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.052+0000 7f00037fe700 1 --2- 172.21.15.16:0/1339321018 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efff0004850 0x7efff0004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:29.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.052+0000 7f00037fe700 1 -- 172.21.15.16:0/1339321018 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efff00035b0 con 0x7efff0003a00 2023-10-16T13:27:29.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.052+0000 7f0003fff700 1 -- 172.21.15.16:0/1339321018 reap_dead start 2023-10-16T13:27:29.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.052+0000 7f00037fe700 1 --2- 172.21.15.16:0/1339321018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efff0003a00 0x7efff0003e20 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7effec0017c0 tx=0x7efff0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d082948a97c535e7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:29.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.052+0000 7f00027fc700 1 -- 172.21.15.16:0/1339321018 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effec0052d0 con 0x7efff0003a00 2023-10-16T13:27:29.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.053+0000 7f00027fc700 1 -- 172.21.15.16:0/1339321018 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effec005430 con 0x7efff0003a00 2023-10-16T13:27:29.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.053+0000 7f00027fc700 1 -- 172.21.15.16:0/1339321018 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effec0038c0 con 0x7efff0003a00 2023-10-16T13:27:29.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.053+0000 7f000a164700 1 -- 172.21.15.16:0/1339321018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efff0003a00 msgr2=0x7efff0003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:29.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.053+0000 7f000a164700 1 --2- 172.21.15.16:0/1339321018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efff0003a00 0x7efff0003e20 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7effec0017c0 tx=0x7efff0005880 comp rx=0 tx=0).stop 2023-10-16T13:27:29.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.053+0000 7f000a164700 1 -- 172.21.15.16:0/1339321018 shutdown_connections 2023-10-16T13:27:29.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.053+0000 7f000a164700 1 --2- 172.21.15.16:0/1339321018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efff0003a00 0x7efff0003e20 unknown :-1 s=CLOSED pgs=1858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:29.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.053+0000 7f000a164700 1 -- 172.21.15.16:0/1339321018 >> 172.21.15.16:0/1339321018 conn(0x7f000406f430 msgr2=0x7f000406f810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:29.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.053+0000 7f000a164700 1 -- 172.21.15.16:0/1339321018 shutdown_connections 2023-10-16T13:27:29.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.053+0000 7f000a164700 1 -- 172.21.15.16:0/1339321018 wait complete. 2023-10-16T13:27:29.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.055+0000 7f000a164700 1 Processor -- start 2023-10-16T13:27:29.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.055+0000 7f000a164700 1 -- start start 2023-10-16T13:27:29.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.056+0000 7f000a164700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efff0003a00 0x7f00040b50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:29.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.056+0000 7f000a164700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f000414c510 0x7f00040c4a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:29.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.056+0000 7f000a164700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f00040c4f50 0x7f000415e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:29.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.056+0000 7f000a164700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00040b5620 con 0x7f000414c510 2023-10-16T13:27:29.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.056+0000 7f000a164700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f000414d090 con 0x7efff0003a00 2023-10-16T13:27:29.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.056+0000 7f000a164700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00040b3df0 con 0x7f00040c4f50 2023-10-16T13:27:29.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.056+0000 7f0002ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f000414c510 0x7f00040c4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:29.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.056+0000 7f0002ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f000414c510 0x7f00040c4a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48116/0 (socket says 172.21.15.16:48116) 2023-10-16T13:27:29.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.056+0000 7f0002ffd700 1 -- 172.21.15.16:0/1658752191 learned_addr learned my addr 172.21.15.16:0/1658752191 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:29.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.056+0000 7f00037fe700 1 --2- 172.21.15.16:0/1658752191 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efff0003a00 0x7f00040b50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:29.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.056+0000 7f0003fff700 1 --2- 172.21.15.16:0/1658752191 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f00040c4f50 0x7f000415e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:29.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.056+0000 7f0002ffd700 1 -- 172.21.15.16:0/1658752191 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f00040c4f50 msgr2=0x7f000415e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:29.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.056+0000 7f0002ffd700 1 --2- 172.21.15.16:0/1658752191 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f00040c4f50 0x7f000415e100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:29.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.056+0000 7f0002ffd700 1 -- 172.21.15.16:0/1658752191 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efff0003a00 msgr2=0x7f00040b50e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:29.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.056+0000 7f0002ffd700 1 --2- 172.21.15.16:0/1658752191 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efff0003a00 0x7f00040b50e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:29.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.056+0000 7f0002ffd700 1 -- 172.21.15.16:0/1658752191 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00040b4c50 con 0x7f000414c510 2023-10-16T13:27:29.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.056+0000 7f0003fff700 1 --2- 172.21.15.16:0/1658752191 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f00040c4f50 0x7f000415e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:27:29.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.056+0000 7f0002ffd700 1 --2- 172.21.15.16:0/1658752191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f000414c510 0x7f00040c4a10 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7efff4011960 tx=0x7efff4011c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:29.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.057+0000 7efffbfff700 1 -- 172.21.15.16:0/1658752191 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efff4011eb0 con 0x7f000414c510 2023-10-16T13:27:29.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.057+0000 7efffbfff700 1 -- 172.21.15.16:0/1658752191 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efff4004240 con 0x7f000414c510 2023-10-16T13:27:29.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.057+0000 7f000a164700 1 -- 172.21.15.16:0/1658752191 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00040b4040 con 0x7f000414c510 2023-10-16T13:27:29.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.057+0000 7efffbfff700 1 -- 172.21.15.16:0/1658752191 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efff40045b0 con 0x7f000414c510 2023-10-16T13:27:29.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.057+0000 7f000a164700 1 -- 172.21.15.16:0/1658752191 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00040b4600 con 0x7f000414c510 2023-10-16T13:27:29.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.058+0000 7f000a164700 1 -- 172.21.15.16:0/1658752191 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00040b4de0 con 0x7f000414c510 2023-10-16T13:27:29.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.058+0000 7efffbfff700 1 -- 172.21.15.16:0/1658752191 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 96) v1 ==== 89067+0+0 (secure 0 0 0) 0x7efff400f730 con 0x7f000414c510 2023-10-16T13:27:29.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.059+0000 7efffbfff700 1 --2- 172.21.15.16:0/1658752191 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7effe406ad50 0x7effe406d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:29.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.059+0000 7f00037fe700 1 --2- 172.21.15.16:0/1658752191 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7effe406ad50 0x7effe406d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:29.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.059+0000 7efffbfff700 1 -- 172.21.15.16:0/1658752191 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(166..166 src has 1..166) v4 ==== 6903+0+0 (secure 0 0 0) 0x7efff4089970 con 0x7f000414c510 2023-10-16T13:27:29.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.059+0000 7f00037fe700 1 --2- 172.21.15.16:0/1658752191 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7effe406ad50 0x7effe406d210 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7effec005040 tx=0x7effec009030 comp rx=0 tx=0).ready entity=mgr.9510 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:29.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.063+0000 7efffbfff700 1 -- 172.21.15.16:0/1658752191 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7efff40548b0 con 0x7f000414c510 2023-10-16T13:27:29.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.266+0000 7f000a164700 1 -- 172.21.15.16:0/1658752191 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f000404ec20 con 0x7f000414c510 2023-10-16T13:27:29.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:29.816+0000 7efffbfff700 1 -- 172.21.15.16:0/1658752191 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 97) v1 ==== 89275+0+0 (secure 0 0 0) 0x7efff4054c80 con 0x7f000414c510 2023-10-16T13:27:30.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:30.093+0000 7efffbfff700 1 -- 172.21.15.16:0/1658752191 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v167) v1 ==== 181+0+0 (secure 0 0 0) 0x7efff4085620 con 0x7f000414c510 2023-10-16T13:27:30.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:30.097+0000 7f000a164700 1 -- 172.21.15.16:0/1658752191 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7effe406ad50 msgr2=0x7effe406d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:30.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:30.097+0000 7f000a164700 1 --2- 172.21.15.16:0/1658752191 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7effe406ad50 0x7effe406d210 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7effec005040 tx=0x7effec009030 comp rx=0 tx=0).stop 2023-10-16T13:27:30.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:30.097+0000 7f000a164700 1 -- 172.21.15.16:0/1658752191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f000414c510 msgr2=0x7f00040c4a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:30.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:30.097+0000 7f000a164700 1 --2- 172.21.15.16:0/1658752191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f000414c510 0x7f00040c4a10 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7efff4011960 tx=0x7efff4011c70 comp rx=0 tx=0).stop 2023-10-16T13:27:30.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:30.097+0000 7f000a164700 1 -- 172.21.15.16:0/1658752191 shutdown_connections 2023-10-16T13:27:30.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:30.097+0000 7f000a164700 1 --2- 172.21.15.16:0/1658752191 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f00040c4f50 0x7f000415e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:30.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:30.097+0000 7f000a164700 1 --2- 172.21.15.16:0/1658752191 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7effe406ad50 0x7effe406d210 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:30.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:30.097+0000 7f000a164700 1 --2- 172.21.15.16:0/1658752191 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efff0003a00 0x7f00040b50e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:30.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:30.097+0000 7f000a164700 1 --2- 172.21.15.16:0/1658752191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f000414c510 0x7f00040c4a10 unknown :-1 s=CLOSED pgs=1859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:30.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:30.097+0000 7f000a164700 1 -- 172.21.15.16:0/1658752191 >> 172.21.15.16:0/1658752191 conn(0x7f000406f430 msgr2=0x7f000415f990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:30.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:30.098+0000 7f000a164700 1 -- 172.21.15.16:0/1658752191 shutdown_connections 2023-10-16T13:27:30.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:30.098+0000 7f000a164700 1 -- 172.21.15.16:0/1658752191 wait complete. 2023-10-16T13:27:30.103 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' removed 2023-10-16T13:27:30.116 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-16T13:27:30.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:30.251+0000 7f14af85b700 1 Processor -- start 2023-10-16T13:27:30.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:30.251+0000 7f14af85b700 1 -- start start 2023-10-16T13:27:30.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:30.252+0000 7f14af85b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f14a815d890 0x7f14a80ce5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:30.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:30.252+0000 7f14af85b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f14a80ceb20 0x7f14a80cffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:30.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:30.252+0000 7f14af85b700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f14a80d04e0 0x7f14a80b5120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:30.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:30.252+0000 7f14af85b700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14a80b6800 con 0x7f14a80ceb20 2023-10-16T13:27:30.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:30.252+0000 7f14af85b700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14a80d0950 con 0x7f14a80d04e0 2023-10-16T13:27:30.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:30.252+0000 7f14af85b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14a80d2580 con 0x7f14a815d890 2023-10-16T13:27:30.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:30.252+0000 7f14acdf6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f14a80ceb20 0x7f14a80cffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:30.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:30.252+0000 7f14acdf6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f14a80ceb20 0x7f14a80cffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48124/0 (socket says 172.21.15.16:48124) 2023-10-16T13:27:30.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:30.252+0000 7f14acdf6700 1 -- 172.21.15.16:0/3365697143 learned_addr learned my addr 172.21.15.16:0/3365697143 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:30.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:30.252+0000 7f14acdf6700 1 -- 172.21.15.16:0/3365697143 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f14a815d890 msgr2=0x7f14a80ce5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:30.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:30.252+0000 7f14acdf6700 1 --2- 172.21.15.16:0/3365697143 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f14a815d890 0x7f14a80ce5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:30.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:30.252+0000 7f14acdf6700 1 -- 172.21.15.16:0/3365697143 >> v2:172.21.15.49:3300/0 conn(0x7f14a80d04e0 msgr2=0x7f14a80b5120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:30.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:30.252+0000 7f14acdf6700 1 --2- 172.21.15.16:0/3365697143 >> v2:172.21.15.49:3300/0 conn(0x7f14a80d04e0 0x7f14a80b5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:30.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:30.252+0000 7f14acdf6700 1 -- 172.21.15.16:0/3365697143 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14a80d33e0 con 0x7f14a80ceb20 2023-10-16T13:27:30.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:30.252+0000 7f14acdf6700 1 --2- 172.21.15.16:0/3365697143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14a80ceb20 0x7f14a80cffa0 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7f14a80ce3e0 tx=0x7f1498009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd501c8357c2eb9a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:30.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:30.253+0000 7f149ffff700 1 -- 172.21.15.16:0/3365697143 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f149800b870 con 0x7f14a80ceb20 2023-10-16T13:27:30.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:30.253+0000 7f149ffff700 1 -- 172.21.15.16:0/3365697143 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f149800b9d0 con 0x7f14a80ceb20 2023-10-16T13:27:30.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:30.253+0000 7f149ffff700 1 -- 172.21.15.16:0/3365697143 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f149800bd60 con 0x7f14a80ceb20 2023-10-16T13:27:30.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:30.253+0000 7f14af85b700 1 -- 172.21.15.16:0/3365697143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14a80ceb20 msgr2=0x7f14a80cffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:30.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:30.253+0000 7f14af85b700 1 --2- 172.21.15.16:0/3365697143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14a80ceb20 0x7f14a80cffa0 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7f14a80ce3e0 tx=0x7f1498009bd0 comp rx=0 tx=0).stop 2023-10-16T13:27:30.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:30.253+0000 7f14af85b700 1 -- 172.21.15.16:0/3365697143 shutdown_connections 2023-10-16T13:27:30.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:30.253+0000 7f14af85b700 1 --2- 172.21.15.16:0/3365697143 >> v2:172.21.15.49:3300/0 conn(0x7f14a80d04e0 0x7f14a80b5120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:30.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:30.253+0000 7f14af85b700 1 --2- 172.21.15.16:0/3365697143 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f14a815d890 0x7f14a80ce5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:30.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:30.253+0000 7f14af85b700 1 --2- 172.21.15.16:0/3365697143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14a80ceb20 0x7f14a80cffa0 unknown :-1 s=CLOSED pgs=1860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:30.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:30.253+0000 7f14af85b700 1 -- 172.21.15.16:0/3365697143 >> 172.21.15.16:0/3365697143 conn(0x7f14a8072360 msgr2=0x7f14a8072770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:30.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:30.253+0000 7f14af85b700 1 -- 172.21.15.16:0/3365697143 shutdown_connections 2023-10-16T13:27:30.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:30.253+0000 7f14af85b700 1 -- 172.21.15.16:0/3365697143 wait complete. 2023-10-16T13:27:30.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:30.254+0000 7f14af85b700 1 Processor -- start 2023-10-16T13:27:30.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:30.255+0000 7f14af85b700 1 -- start start 2023-10-16T13:27:30.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:30.255+0000 7f14af85b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f14a80ceb20 0x7f14a8153f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:30.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:30.255+0000 7f14af85b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14a80d04e0 0x7f14a81f4200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:30.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:30.255+0000 7f14af85b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f14a815d890 0x7f14a8156dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:30.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:30.255+0000 7f14af85b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14a81544a0 con 0x7f14a80d04e0 2023-10-16T13:27:30.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:30.255+0000 7f14af85b700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14a80ca880 con 0x7f14a80ceb20 2023-10-16T13:27:30.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:30.255+0000 7f14af85b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14a80cab60 con 0x7f14a815d890 2023-10-16T13:27:30.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:30.255+0000 7f14acdf6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14a80d04e0 0x7f14a81f4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:30.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:30.255+0000 7f14acdf6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14a80d04e0 0x7f14a81f4200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48130/0 (socket says 172.21.15.16:48130) 2023-10-16T13:27:30.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:30.255+0000 7f14acdf6700 1 -- 172.21.15.16:0/3894113679 learned_addr learned my addr 172.21.15.16:0/3894113679 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:30.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:30.256+0000 7f14ad5f7700 1 --2- 172.21.15.16:0/3894113679 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f14a80ceb20 0x7f14a8153f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:30.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:30.256+0000 7f14acdf6700 1 -- 172.21.15.16:0/3894113679 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f14a815d890 msgr2=0x7f14a8156dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:30.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:30.256+0000 7f14acdf6700 1 --2- 172.21.15.16:0/3894113679 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f14a815d890 0x7f14a8156dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:30.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:30.256+0000 7f14acdf6700 1 -- 172.21.15.16:0/3894113679 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f14a80ceb20 msgr2=0x7f14a8153f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:30.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:30.256+0000 7f14acdf6700 1 --2- 172.21.15.16:0/3894113679 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f14a80ceb20 0x7f14a8153f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:30.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:30.256+0000 7f14acdf6700 1 -- 172.21.15.16:0/3894113679 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1498009550 con 0x7f14a80d04e0 2023-10-16T13:27:30.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:30.256+0000 7f14acdf6700 1 --2- 172.21.15.16:0/3894113679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14a80d04e0 0x7f14a81f4200 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7f14a80d0fa0 tx=0x7f1498004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:30.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:30.256+0000 7f149dffb700 1 -- 172.21.15.16:0/3894113679 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1498004310 con 0x7f14a80d04e0 2023-10-16T13:27:30.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:30.256+0000 7f149dffb700 1 -- 172.21.15.16:0/3894113679 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f149801f070 con 0x7f14a80d04e0 2023-10-16T13:27:30.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:30.256+0000 7f149dffb700 1 -- 172.21.15.16:0/3894113679 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14980036a0 con 0x7f14a80d04e0 2023-10-16T13:27:30.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:30.256+0000 7f14af85b700 1 -- 172.21.15.16:0/3894113679 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14a80cae40 con 0x7f14a80d04e0 2023-10-16T13:27:30.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:30.256+0000 7f14af85b700 1 -- 172.21.15.16:0/3894113679 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14a80cb400 con 0x7f14a80d04e0 2023-10-16T13:27:30.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:30.257+0000 7f14af85b700 1 -- 172.21.15.16:0/3894113679 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14a804ec20 con 0x7f14a80d04e0 2023-10-16T13:27:30.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:30.257+0000 7f149dffb700 1 -- 172.21.15.16:0/3894113679 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 97) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f1498021030 con 0x7f14a80d04e0 2023-10-16T13:27:30.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:30.258+0000 7f149dffb700 1 --2- 172.21.15.16:0/3894113679 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f148c06af90 0x7f148c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:30.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:30.258+0000 7f149dffb700 1 -- 172.21.15.16:0/3894113679 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 6465+0+0 (secure 0 0 0) 0x7f149808adc0 con 0x7f14a80d04e0 2023-10-16T13:27:30.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:30.258+0000 7f14ad5f7700 1 --2- 172.21.15.16:0/3894113679 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f148c06af90 0x7f148c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:30.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:30.258+0000 7f14ad5f7700 1 --2- 172.21.15.16:0/3894113679 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f148c06af90 0x7f148c06d450 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f14a81f6590 tx=0x7f14a4008040 comp rx=0 tx=0).ready entity=mgr.9510 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:30.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:30.262+0000 7f149dffb700 1 -- 172.21.15.16:0/3894113679 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1498055c10 con 0x7f14a80d04e0 2023-10-16T13:27:30.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:30.464+0000 7f14af85b700 1 -- 172.21.15.16:0/3894113679 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f14a80c9f70 con 0x7f14a80d04e0 2023-10-16T13:27:30.474 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:27:30.477 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:27:31.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.099+0000 7f149dffb700 1 -- 172.21.15.16:0/3894113679 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v168) v1 ==== 169+0+0 (secure 0 0 0) 0x7f1498059230 con 0x7f14a80d04e0 2023-10-16T13:27:31.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.106+0000 7f149dffb700 1 -- 172.21.15.16:0/3894113679 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1498055fe0 con 0x7f14a80d04e0 2023-10-16T13:27:31.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.107+0000 7f14af85b700 1 -- 172.21.15.16:0/3894113679 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f148c06af90 msgr2=0x7f148c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:31.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.107+0000 7f14af85b700 1 --2- 172.21.15.16:0/3894113679 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f148c06af90 0x7f148c06d450 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f14a81f6590 tx=0x7f14a4008040 comp rx=0 tx=0).stop 2023-10-16T13:27:31.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.107+0000 7f14af85b700 1 -- 172.21.15.16:0/3894113679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14a80d04e0 msgr2=0x7f14a81f4200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:31.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.107+0000 7f14af85b700 1 --2- 172.21.15.16:0/3894113679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14a80d04e0 0x7f14a81f4200 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7f14a80d0fa0 tx=0x7f1498004750 comp rx=0 tx=0).stop 2023-10-16T13:27:31.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.107+0000 7f14af85b700 1 -- 172.21.15.16:0/3894113679 shutdown_connections 2023-10-16T13:27:31.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.107+0000 7f14af85b700 1 --2- 172.21.15.16:0/3894113679 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f14a815d890 0x7f14a8156dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:31.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.107+0000 7f14af85b700 1 --2- 172.21.15.16:0/3894113679 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f148c06af90 0x7f148c06d450 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:31.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.107+0000 7f14af85b700 1 --2- 172.21.15.16:0/3894113679 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f14a80ceb20 0x7f14a8153f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:31.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.107+0000 7f14af85b700 1 --2- 172.21.15.16:0/3894113679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14a80d04e0 0x7f14a81f4200 unknown :-1 s=CLOSED pgs=1861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:31.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.108+0000 7f14af85b700 1 -- 172.21.15.16:0/3894113679 >> 172.21.15.16:0/3894113679 conn(0x7f14a8072360 msgr2=0x7f14a8159a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:31.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.108+0000 7f14af85b700 1 -- 172.21.15.16:0/3894113679 shutdown_connections 2023-10-16T13:27:31.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.108+0000 7f14af85b700 1 -- 172.21.15.16:0/3894113679 wait complete. 2023-10-16T13:27:31.113 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' removed 2023-10-16T13:27:31.127 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-16T13:27:31.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.260+0000 7f3f7eb6f700 1 Processor -- start 2023-10-16T13:27:31.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.260+0000 7f3f7eb6f700 1 -- start start 2023-10-16T13:27:31.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.261+0000 7f3f7eb6f700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f3f7806e570 0x7f3f7815e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:31.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.261+0000 7f3f7eb6f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3f7806a7e0 0x7f3f7806ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:31.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.261+0000 7f3f7eb6f700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f7815b6a0 con 0x7f3f7815b2c0 2023-10-16T13:27:31.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.261+0000 7f3f7eb6f700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f7806b190 con 0x7f3f7806a7e0 2023-10-16T13:27:31.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.261+0000 7f3f7eb6f700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f78161b20 con 0x7f3f7806e570 2023-10-16T13:27:31.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.261+0000 7f3f7d10c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3f7806a7e0 0x7f3f7806ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:31.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.261+0000 7f3f7d10c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3f7806a7e0 0x7f3f7806ac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48138/0 (socket says 172.21.15.16:48138) 2023-10-16T13:27:31.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.261+0000 7f3f7c90b700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f3f7815b2c0 0x7f3f78161340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37410/0 (socket says 172.21.15.16:37410) 2023-10-16T13:27:31.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.261+0000 7f3f7d10c700 1 -- 172.21.15.16:0/4037389319 learned_addr learned my addr 172.21.15.16:0/4037389319 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:31.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.261+0000 7f3f77fff700 1 --2- 172.21.15.16:0/4037389319 >> v2:172.21.15.49:3300/0 conn(0x7f3f7806e570 0x7f3f7815e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:31.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.261+0000 7f3f7d10c700 1 -- 172.21.15.16:0/4037389319 >> v2:172.21.15.49:3300/0 conn(0x7f3f7806e570 msgr2=0x7f3f7815e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:31.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.261+0000 7f3f7d10c700 1 --2- 172.21.15.16:0/4037389319 >> v2:172.21.15.49:3300/0 conn(0x7f3f7806e570 0x7f3f7815e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:31.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.261+0000 7f3f7d10c700 1 -- 172.21.15.16:0/4037389319 >> v1:172.21.15.16:6789/0 conn(0x7f3f7815b2c0 legacy=0x7f3f78161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:31.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.261+0000 7f3f7d10c700 1 -- 172.21.15.16:0/4037389319 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f78161e00 con 0x7f3f7806a7e0 2023-10-16T13:27:31.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.261+0000 7f3f777fe700 1 -- 172.21.15.16:0/4037389319 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 956710004 0 0) 0x7f3f7815b6a0 con 0x7f3f7815b2c0 2023-10-16T13:27:31.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.262+0000 7f3f7d10c700 1 --2- 172.21.15.16:0/4037389319 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f7806a7e0 0x7f3f7806ac50 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7f3f780d03c0 tx=0x7f3f7000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99b18d15488731e9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:31.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.262+0000 7f3f777fe700 1 -- 172.21.15.16:0/4037389319 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f7000e070 con 0x7f3f7806a7e0 2023-10-16T13:27:31.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.262+0000 7f3f777fe700 1 -- 172.21.15.16:0/4037389319 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f7000b9c0 con 0x7f3f7806a7e0 2023-10-16T13:27:31.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.262+0000 7f3f777fe700 1 -- 172.21.15.16:0/4037389319 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f7000bd70 con 0x7f3f7806a7e0 2023-10-16T13:27:31.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.262+0000 7f3f7eb6f700 1 -- 172.21.15.16:0/4037389319 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f7806a7e0 msgr2=0x7f3f7806ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:31.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.262+0000 7f3f7eb6f700 1 --2- 172.21.15.16:0/4037389319 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f7806a7e0 0x7f3f7806ac50 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7f3f780d03c0 tx=0x7f3f7000b330 comp rx=0 tx=0).stop 2023-10-16T13:27:31.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.262+0000 7f3f7eb6f700 1 -- 172.21.15.16:0/4037389319 shutdown_connections 2023-10-16T13:27:31.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.262+0000 7f3f7eb6f700 1 --2- 172.21.15.16:0/4037389319 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f7806a7e0 0x7f3f7806ac50 unknown :-1 s=CLOSED pgs=1863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:31.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.262+0000 7f3f7eb6f700 1 --2- 172.21.15.16:0/4037389319 >> v2:172.21.15.49:3300/0 conn(0x7f3f7806e570 0x7f3f7815e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:31.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.262+0000 7f3f7eb6f700 1 -- 172.21.15.16:0/4037389319 >> 172.21.15.16:0/4037389319 conn(0x7f3f780b1f40 msgr2=0x7f3f780b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:31.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.262+0000 7f3f7eb6f700 1 -- 172.21.15.16:0/4037389319 shutdown_connections 2023-10-16T13:27:31.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.262+0000 7f3f7eb6f700 1 -- 172.21.15.16:0/4037389319 wait complete. 2023-10-16T13:27:31.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.263+0000 7f3f7eb6f700 1 Processor -- start 2023-10-16T13:27:31.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.264+0000 7f3f7eb6f700 1 -- start start 2023-10-16T13:27:31.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.264+0000 7f3f7eb6f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f7806a7e0 0x7f3f7814f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:31.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.264+0000 7f3f7eb6f700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3f7806e570 0x7f3f780c83f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:31.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.264+0000 7f3f7eb6f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3f7815b2c0 0x7f3f780c5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:31.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.264+0000 7f3f7eb6f700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f780c3e30 con 0x7f3f7806a7e0 2023-10-16T13:27:31.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.264+0000 7f3f7eb6f700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f780c8930 con 0x7f3f7806e570 2023-10-16T13:27:31.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.264+0000 7f3f7eb6f700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f780c6a70 con 0x7f3f7815b2c0 2023-10-16T13:27:31.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.264+0000 7f3f7c90b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f7806a7e0 0x7f3f7814f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:31.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.264+0000 7f3f7c90b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f7806a7e0 0x7f3f7814f9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48142/0 (socket says 172.21.15.16:48142) 2023-10-16T13:27:31.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.264+0000 7f3f7c90b700 1 -- 172.21.15.16:0/2549835114 learned_addr learned my addr 172.21.15.16:0/2549835114 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:31.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.265+0000 7f3f77fff700 1 --2- 172.21.15.16:0/2549835114 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3f7806e570 0x7f3f780c83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:31.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.265+0000 7f3f7d10c700 1 --2- 172.21.15.16:0/2549835114 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3f7815b2c0 0x7f3f780c5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:31.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.265+0000 7f3f7c90b700 1 -- 172.21.15.16:0/2549835114 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3f7815b2c0 msgr2=0x7f3f780c5780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:31.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.265+0000 7f3f7c90b700 1 --2- 172.21.15.16:0/2549835114 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3f7815b2c0 0x7f3f780c5780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:31.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.265+0000 7f3f7c90b700 1 -- 172.21.15.16:0/2549835114 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3f7806e570 msgr2=0x7f3f780c83f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:31.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.265+0000 7f3f7c90b700 1 --2- 172.21.15.16:0/2549835114 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3f7806e570 0x7f3f780c83f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:31.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.265+0000 7f3f7c90b700 1 -- 172.21.15.16:0/2549835114 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f70009b30 con 0x7f3f7806a7e0 2023-10-16T13:27:31.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.265+0000 7f3f7c90b700 1 --2- 172.21.15.16:0/2549835114 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f7806a7e0 0x7f3f7814f9a0 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7f3f6c0011e0 tx=0x7f3f6c00cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:31.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.265+0000 7f3f757fa700 1 -- 172.21.15.16:0/2549835114 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f6c004c80 con 0x7f3f7806a7e0 2023-10-16T13:27:31.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.265+0000 7f3f757fa700 1 -- 172.21.15.16:0/2549835114 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f6c004e00 con 0x7f3f7806a7e0 2023-10-16T13:27:31.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.265+0000 7f3f7eb6f700 1 -- 172.21.15.16:0/2549835114 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f780c61d0 con 0x7f3f7806a7e0 2023-10-16T13:27:31.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.265+0000 7f3f757fa700 1 -- 172.21.15.16:0/2549835114 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f6c007310 con 0x7f3f7806a7e0 2023-10-16T13:27:31.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.265+0000 7f3f7eb6f700 1 -- 172.21.15.16:0/2549835114 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f780c6790 con 0x7f3f7806a7e0 2023-10-16T13:27:31.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.266+0000 7f3f7eb6f700 1 -- 172.21.15.16:0/2549835114 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f7804ebb0 con 0x7f3f7806a7e0 2023-10-16T13:27:31.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.267+0000 7f3f757fa700 1 -- 172.21.15.16:0/2549835114 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3f6c007470 con 0x7f3f7806a7e0 2023-10-16T13:27:31.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.267+0000 7f3f757fa700 1 --2- 172.21.15.16:0/2549835114 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f3f5c06b020 0x7f3f5c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:31.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.267+0000 7f3f757fa700 1 -- 172.21.15.16:0/2549835114 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 6051+0+0 (secure 0 0 0) 0x7f3f6c0896d0 con 0x7f3f7806a7e0 2023-10-16T13:27:31.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.267+0000 7f3f77fff700 1 --2- 172.21.15.16:0/2549835114 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f3f5c06b020 0x7f3f5c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:31.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.268+0000 7f3f77fff700 1 --2- 172.21.15.16:0/2549835114 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f3f5c06b020 0x7f3f5c06d4e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3f780c7570 tx=0x7f3f64008040 comp rx=0 tx=0).ready entity=mgr.9510 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:31.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.272+0000 7f3f757fa700 1 -- 172.21.15.16:0/2549835114 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3f6c054510 con 0x7f3f7806a7e0 2023-10-16T13:27:31.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.477+0000 7f3f7eb6f700 1 -- 172.21.15.16:0/2549835114 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f3f780c3b90 con 0x7f3f7806a7e0 2023-10-16T13:27:31.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.480+0000 7f3f757fa700 1 -- 172.21.15.16:0/2549835114 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v55) v1 ==== 115+0+0 (secure 0 0 0) 0x7f3f6c057b30 con 0x7f3f7806a7e0 2023-10-16T13:27:31.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.482+0000 7f3f7eb6f700 1 -- 172.21.15.16:0/2549835114 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f3f5c06b020 msgr2=0x7f3f5c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:31.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.482+0000 7f3f7eb6f700 1 --2- 172.21.15.16:0/2549835114 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f3f5c06b020 0x7f3f5c06d4e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3f780c7570 tx=0x7f3f64008040 comp rx=0 tx=0).stop 2023-10-16T13:27:31.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.482+0000 7f3f7eb6f700 1 -- 172.21.15.16:0/2549835114 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f7806a7e0 msgr2=0x7f3f7814f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:31.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.482+0000 7f3f7eb6f700 1 --2- 172.21.15.16:0/2549835114 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f7806a7e0 0x7f3f7814f9a0 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7f3f6c0011e0 tx=0x7f3f6c00cae0 comp rx=0 tx=0).stop 2023-10-16T13:27:31.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.482+0000 7f3f7eb6f700 1 -- 172.21.15.16:0/2549835114 shutdown_connections 2023-10-16T13:27:31.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.482+0000 7f3f7eb6f700 1 --2- 172.21.15.16:0/2549835114 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3f7815b2c0 0x7f3f780c5780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:31.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.482+0000 7f3f7eb6f700 1 --2- 172.21.15.16:0/2549835114 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f7806a7e0 0x7f3f7814f9a0 unknown :-1 s=CLOSED pgs=1864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:31.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.482+0000 7f3f7eb6f700 1 --2- 172.21.15.16:0/2549835114 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f3f5c06b020 0x7f3f5c06d4e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:31.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.482+0000 7f3f7eb6f700 1 --2- 172.21.15.16:0/2549835114 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3f7806e570 0x7f3f780c83f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:31.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.482+0000 7f3f7eb6f700 1 -- 172.21.15.16:0/2549835114 >> 172.21.15.16:0/2549835114 conn(0x7f3f780b1f40 msgr2=0x7f3f780cce00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:31.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.482+0000 7f3f7eb6f700 1 -- 172.21.15.16:0/2549835114 shutdown_connections 2023-10-16T13:27:31.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.482+0000 7f3f7eb6f700 1 -- 172.21.15.16:0/2549835114 wait complete. 2023-10-16T13:27:31.497 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-16T13:27:31.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.622+0000 7f3800f91700 1 Processor -- start 2023-10-16T13:27:31.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.622+0000 7f3800f91700 1 -- start start 2023-10-16T13:27:31.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.623+0000 7f3800f91700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f37fc0cb150 0x7f37fc071a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:31.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.623+0000 7f3800f91700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37fc15c040 con 0x7f37fc072b30 2023-10-16T13:27:31.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.623+0000 7f3800f91700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37fc15c320 con 0x7f37fc15bb40 2023-10-16T13:27:31.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.623+0000 7f3800f91700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37fc06ea20 con 0x7f37fc0cb150 2023-10-16T13:27:31.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.623+0000 7f37fa59c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f37fc072b30 0x7f37fc06e310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37424/0 (socket says 172.21.15.16:37424) 2023-10-16T13:27:31.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.623+0000 7f37fa59c700 1 -- 172.21.15.16:0/120770934 learned_addr learned my addr 172.21.15.16:0/120770934 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:27:31.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.623+0000 7f37fad9d700 1 --2- 172.21.15.16:0/120770934 >> v2:172.21.15.49:3300/0 conn(0x7f37fc0cb150 0x7f37fc071a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:31.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.623+0000 7f37f959a700 1 -- 172.21.15.16:0/120770934 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3055294534 0 0) 0x7f37fc15c040 con 0x7f37fc072b30 2023-10-16T13:27:31.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.623+0000 7f37f959a700 1 -- 172.21.15.16:0/120770934 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37e40035b0 con 0x7f37fc072b30 2023-10-16T13:27:31.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.624+0000 7f37f959a700 1 -- 172.21.15.16:0/120770934 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 985472845 0 0) 0x7f37fc15c320 con 0x7f37fc15bb40 2023-10-16T13:27:31.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.624+0000 7f37f959a700 1 -- 172.21.15.16:0/120770934 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37fc15c040 con 0x7f37fc15bb40 2023-10-16T13:27:31.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.624+0000 7f37f959a700 1 -- 172.21.15.16:0/120770934 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 349852808 0 0) 0x7f37e40035b0 con 0x7f37fc072b30 2023-10-16T13:27:31.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.624+0000 7f37f959a700 1 -- 172.21.15.16:0/120770934 >> v2:172.21.15.49:3300/0 conn(0x7f37fc0cb150 msgr2=0x7f37fc071a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:31.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.624+0000 7f37f959a700 1 --2- 172.21.15.16:0/120770934 >> v2:172.21.15.49:3300/0 conn(0x7f37fc0cb150 0x7f37fc071a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:31.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.624+0000 7f37f959a700 1 -- 172.21.15.16:0/120770934 >> v1:172.21.15.49:6789/0 conn(0x7f37fc15bb40 legacy=0x7f37fc0caa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:31.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.624+0000 7f37f959a700 1 -- 172.21.15.16:0/120770934 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37fc15c600 con 0x7f37fc072b30 2023-10-16T13:27:31.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.624+0000 7f37f959a700 1 -- 172.21.15.16:0/120770934 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f37f0001670 con 0x7f37fc072b30 2023-10-16T13:27:31.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.624+0000 7f37f959a700 1 -- 172.21.15.16:0/120770934 >> v1:172.21.15.16:6789/0 conn(0x7f37fc072b30 legacy=0x7f37fc06e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:31.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.624+0000 7f37f959a700 1 --2- 172.21.15.16:0/120770934 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37e4003b90 0x7f37e4003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:31.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.624+0000 7f37f959a700 1 --2- 172.21.15.16:0/120770934 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f37e4004850 0x7f37e4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:31.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.624+0000 7f37f959a700 1 --2- 172.21.15.16:0/120770934 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f37e4005210 0x7f37e400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:31.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.624+0000 7f37f959a700 1 -- 172.21.15.16:0/120770934 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37fc06ea20 con 0x7f37e4003b90 2023-10-16T13:27:31.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.624+0000 7f37f959a700 1 -- 172.21.15.16:0/120770934 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37e400c090 con 0x7f37e4005210 2023-10-16T13:27:31.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.624+0000 7f37f959a700 1 -- 172.21.15.16:0/120770934 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37e400c250 con 0x7f37e4004850 2023-10-16T13:27:31.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.624+0000 7f37fa59c700 1 --2- 172.21.15.16:0/120770934 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37e4003b90 0x7f37e4003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:31.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.624+0000 7f37fad9d700 1 --2- 172.21.15.16:0/120770934 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f37e4005210 0x7f37e400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:31.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.624+0000 7f37f9d9b700 1 --2- 172.21.15.16:0/120770934 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f37e4004850 0x7f37e4004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:31.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.624+0000 7f37fa59c700 1 -- 172.21.15.16:0/120770934 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f37e4004850 msgr2=0x7f37e4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:31.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.624+0000 7f37fa59c700 1 --2- 172.21.15.16:0/120770934 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f37e4004850 0x7f37e4004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:31.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.624+0000 7f37fa59c700 1 -- 172.21.15.16:0/120770934 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f37e4005210 msgr2=0x7f37e400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:31.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.624+0000 7f37fa59c700 1 --2- 172.21.15.16:0/120770934 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f37e4005210 0x7f37e400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:31.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.624+0000 7f37fad9d700 1 -- 172.21.15.16:0/120770934 reap_dead start 2023-10-16T13:27:31.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.624+0000 7f37fa59c700 1 -- 172.21.15.16:0/120770934 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37e40035b0 con 0x7f37e4003b90 2023-10-16T13:27:31.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.624+0000 7f37fa59c700 1 --2- 172.21.15.16:0/120770934 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37e4003b90 0x7f37e4003f90 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7f37f000c210 tx=0x7f37e4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b92dc001f0a44c5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:31.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.625+0000 7f37f959a700 1 -- 172.21.15.16:0/120770934 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37f000cdb0 con 0x7f37e4003b90 2023-10-16T13:27:31.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.625+0000 7f37f959a700 1 -- 172.21.15.16:0/120770934 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37f0001af0 con 0x7f37e4003b90 2023-10-16T13:27:31.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.625+0000 7f37f959a700 1 -- 172.21.15.16:0/120770934 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37f0005ba0 con 0x7f37e4003b90 2023-10-16T13:27:31.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.625+0000 7f3800f91700 1 -- 172.21.15.16:0/120770934 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37e4003b90 msgr2=0x7f37e4003f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:31.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.625+0000 7f3800f91700 1 --2- 172.21.15.16:0/120770934 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37e4003b90 0x7f37e4003f90 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7f37f000c210 tx=0x7f37e4005880 comp rx=0 tx=0).stop 2023-10-16T13:27:31.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.625+0000 7f3800f91700 1 -- 172.21.15.16:0/120770934 shutdown_connections 2023-10-16T13:27:31.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.625+0000 7f3800f91700 1 --2- 172.21.15.16:0/120770934 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37e4003b90 0x7f37e4003f90 unknown :-1 s=CLOSED pgs=1866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:31.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.625+0000 7f3800f91700 1 -- 172.21.15.16:0/120770934 >> 172.21.15.16:0/120770934 conn(0x7f37fc0b1f40 msgr2=0x7f37fc0b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:31.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.625+0000 7f3800f91700 1 -- 172.21.15.16:0/120770934 shutdown_connections 2023-10-16T13:27:31.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.625+0000 7f3800f91700 1 -- 172.21.15.16:0/120770934 wait complete. 2023-10-16T13:27:31.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.627+0000 7f3800f91700 1 Processor -- start 2023-10-16T13:27:31.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.627+0000 7f3800f91700 1 -- start start 2023-10-16T13:27:31.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.627+0000 7f3800f91700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f37e4003b90 0x7f37fc154c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:31.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.627+0000 7f3800f91700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37fc15cb20 0x7f37fc0b4080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:31.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.627+0000 7f3800f91700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f37fc15f7e0 0x7f37fc157af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:31.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.628+0000 7f3800f91700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37fc0c8640 con 0x7f37fc15cb20 2023-10-16T13:27:31.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.628+0000 7f3800f91700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37fc0c8920 con 0x7f37fc15f7e0 2023-10-16T13:27:31.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.628+0000 7f3800f91700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37fc0b45c0 con 0x7f37e4003b90 2023-10-16T13:27:31.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.628+0000 7f37f9d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37fc15cb20 0x7f37fc0b4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:31.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.628+0000 7f37f9d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37fc15cb20 0x7f37fc0b4080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48164/0 (socket says 172.21.15.16:48164) 2023-10-16T13:27:31.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.628+0000 7f37f9d9b700 1 -- 172.21.15.16:0/2116676813 learned_addr learned my addr 172.21.15.16:0/2116676813 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:31.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.628+0000 7f37fa59c700 1 --2- 172.21.15.16:0/2116676813 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f37e4003b90 0x7f37fc154c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:31.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.628+0000 7f37fad9d700 1 --2- 172.21.15.16:0/2116676813 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f37fc15f7e0 0x7f37fc157af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:31.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.628+0000 7f37f9d9b700 1 -- 172.21.15.16:0/2116676813 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f37e4003b90 msgr2=0x7f37fc154c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:31.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.628+0000 7f37f9d9b700 1 --2- 172.21.15.16:0/2116676813 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f37e4003b90 0x7f37fc154c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:31.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.628+0000 7f37f9d9b700 1 -- 172.21.15.16:0/2116676813 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f37fc15f7e0 msgr2=0x7f37fc157af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:31.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.628+0000 7f37f9d9b700 1 --2- 172.21.15.16:0/2116676813 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f37fc15f7e0 0x7f37fc157af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:31.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.628+0000 7f37f9d9b700 1 -- 172.21.15.16:0/2116676813 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37fc15d090 con 0x7f37fc15cb20 2023-10-16T13:27:31.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.628+0000 7f37f9d9b700 1 --2- 172.21.15.16:0/2116676813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37fc15cb20 0x7f37fc0b4080 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7f37ec00e5b0 tx=0x7f37ec00e970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:31.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.628+0000 7f37eaffd700 1 -- 172.21.15.16:0/2116676813 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37ec0052e0 con 0x7f37fc15cb20 2023-10-16T13:27:31.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.628+0000 7f37eaffd700 1 -- 172.21.15.16:0/2116676813 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37ec005440 con 0x7f37fc15cb20 2023-10-16T13:27:31.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.628+0000 7f3800f91700 1 -- 172.21.15.16:0/2116676813 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37fc15d370 con 0x7f37fc15cb20 2023-10-16T13:27:31.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.628+0000 7f37eaffd700 1 -- 172.21.15.16:0/2116676813 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37ec0038c0 con 0x7f37fc15cb20 2023-10-16T13:27:31.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.628+0000 7f3800f91700 1 -- 172.21.15.16:0/2116676813 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37fc0b1b80 con 0x7f37fc15cb20 2023-10-16T13:27:31.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.629+0000 7f3800f91700 1 -- 172.21.15.16:0/2116676813 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37fc0b1d40 con 0x7f37fc15cb20 2023-10-16T13:27:31.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.630+0000 7f37eaffd700 1 -- 172.21.15.16:0/2116676813 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f37ec003a20 con 0x7f37fc15cb20 2023-10-16T13:27:31.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.630+0000 7f37eaffd700 1 --2- 172.21.15.16:0/2116676813 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f37dc06b020 0x7f37dc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:31.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.630+0000 7f37eaffd700 1 -- 172.21.15.16:0/2116676813 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 6051+0+0 (secure 0 0 0) 0x7f37ec089050 con 0x7f37fc15cb20 2023-10-16T13:27:31.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.630+0000 7f37fa59c700 1 --2- 172.21.15.16:0/2116676813 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f37dc06b020 0x7f37dc06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:31.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.631+0000 7f37fa59c700 1 --2- 172.21.15.16:0/2116676813 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f37dc06b020 0x7f37dc06d4e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f37f0000c00 tx=0x7f37f000a000 comp rx=0 tx=0).ready entity=mgr.9510 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:31.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.635+0000 7f37eaffd700 1 -- 172.21.15.16:0/2116676813 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f37ec053f20 con 0x7f37fc15cb20 2023-10-16T13:27:31.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.842+0000 7f3800f91700 1 -- 172.21.15.16:0/2116676813 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f37fc15d890 con 0x7f37fc15cb20 2023-10-16T13:27:31.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.845+0000 7f37eaffd700 1 -- 172.21.15.16:0/2116676813 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v56) v1 ==== 104+0+0 (secure 0 0 0) 0x7f37ec057540 con 0x7f37fc15cb20 2023-10-16T13:27:31.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.847+0000 7f3800f91700 1 -- 172.21.15.16:0/2116676813 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f37dc06b020 msgr2=0x7f37dc06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:31.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.847+0000 7f3800f91700 1 --2- 172.21.15.16:0/2116676813 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f37dc06b020 0x7f37dc06d4e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f37f0000c00 tx=0x7f37f000a000 comp rx=0 tx=0).stop 2023-10-16T13:27:31.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.847+0000 7f3800f91700 1 -- 172.21.15.16:0/2116676813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37fc15cb20 msgr2=0x7f37fc0b4080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:31.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.847+0000 7f3800f91700 1 --2- 172.21.15.16:0/2116676813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37fc15cb20 0x7f37fc0b4080 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7f37ec00e5b0 tx=0x7f37ec00e970 comp rx=0 tx=0).stop 2023-10-16T13:27:31.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.847+0000 7f3800f91700 1 -- 172.21.15.16:0/2116676813 shutdown_connections 2023-10-16T13:27:31.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.847+0000 7f3800f91700 1 --2- 172.21.15.16:0/2116676813 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f37dc06b020 0x7f37dc06d4e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:31.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.847+0000 7f3800f91700 1 --2- 172.21.15.16:0/2116676813 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f37fc15f7e0 0x7f37fc157af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:31.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.847+0000 7f3800f91700 1 --2- 172.21.15.16:0/2116676813 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f37e4003b90 0x7f37fc154c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:31.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.847+0000 7f3800f91700 1 --2- 172.21.15.16:0/2116676813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37fc15cb20 0x7f37fc0b4080 unknown :-1 s=CLOSED pgs=1867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:31.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.847+0000 7f3800f91700 1 -- 172.21.15.16:0/2116676813 >> 172.21.15.16:0/2116676813 conn(0x7f37fc0b1f40 msgr2=0x7f37fc0c3f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:31.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.848+0000 7f3800f91700 1 -- 172.21.15.16:0/2116676813 shutdown_connections 2023-10-16T13:27:31.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.848+0000 7f3800f91700 1 -- 172.21.15.16:0/2116676813 wait complete. 2023-10-16T13:27:31.862 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-16T13:27:31.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.987+0000 7f824a160700 1 Processor -- start 2023-10-16T13:27:31.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.987+0000 7f824a160700 1 -- start start 2023-10-16T13:27:31.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.988+0000 7f824a160700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f82440cbe20 0x7f82440cda60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:31.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.988+0000 7f824a160700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f82440c6770 0x7f82440cdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:31.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.988+0000 7f824a160700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f824415aa20 con 0x7f824415a640 2023-10-16T13:27:31.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.988+0000 7f824a160700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f824406eb40 con 0x7f82440c6770 2023-10-16T13:27:31.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.988+0000 7f824a160700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f824406ee20 con 0x7f82440cbe20 2023-10-16T13:27:31.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.988+0000 7f82437fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f824415a640 0x7f82440cb710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37444/0 (socket says 172.21.15.16:37444) 2023-10-16T13:27:31.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.988+0000 7f82437fe700 1 -- 172.21.15.16:0/1026002333 learned_addr learned my addr 172.21.15.16:0/1026002333 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:27:31.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.989+0000 7f8243fff700 1 --2- 172.21.15.16:0/1026002333 >> v2:172.21.15.49:3300/0 conn(0x7f82440c6770 0x7f82440cdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:31.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.989+0000 7f8242ffd700 1 --2- 172.21.15.16:0/1026002333 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f82440cbe20 0x7f82440cda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:31.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.989+0000 7f82427fc700 1 -- 172.21.15.16:0/1026002333 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 813331724 0 0) 0x7f8234001110 con 0x7f824415a640 2023-10-16T13:27:31.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.989+0000 7f82427fc700 1 -- 172.21.15.16:0/1026002333 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82300035b0 con 0x7f824415a640 2023-10-16T13:27:31.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.989+0000 7f8242ffd700 1 -- 172.21.15.16:0/1026002333 >> v2:172.21.15.49:3300/0 conn(0x7f82440c6770 msgr2=0x7f82440cdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:31.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.989+0000 7f8242ffd700 1 --2- 172.21.15.16:0/1026002333 >> v2:172.21.15.49:3300/0 conn(0x7f82440c6770 0x7f82440cdfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:31.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.989+0000 7f8242ffd700 1 -- 172.21.15.16:0/1026002333 >> v1:172.21.15.16:6789/0 conn(0x7f824415a640 legacy=0x7f82440cb710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:31.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.989+0000 7f8242ffd700 1 -- 172.21.15.16:0/1026002333 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f824406f100 con 0x7f82440cbe20 2023-10-16T13:27:31.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.989+0000 7f8243fff700 1 --2- 172.21.15.16:0/1026002333 >> v2:172.21.15.49:3300/0 conn(0x7f82440c6770 0x7f82440cdfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:27:31.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.989+0000 7f8242ffd700 1 --2- 172.21.15.16:0/1026002333 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f82440cbe20 0x7f82440cda60 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f822c009840 tx=0x7f822c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=400f670de630fab5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:31.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.989+0000 7f82427fc700 1 -- 172.21.15.16:0/1026002333 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f822c011830 con 0x7f82440cbe20 2023-10-16T13:27:31.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.989+0000 7f82427fc700 1 -- 172.21.15.16:0/1026002333 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f822c011990 con 0x7f82440cbe20 2023-10-16T13:27:32.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.989+0000 7f82427fc700 1 -- 172.21.15.16:0/1026002333 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f822c011d20 con 0x7f82440cbe20 2023-10-16T13:27:32.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.990+0000 7f824a160700 1 -- 172.21.15.16:0/1026002333 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f82440cbe20 msgr2=0x7f82440cda60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:32.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.990+0000 7f824a160700 1 --2- 172.21.15.16:0/1026002333 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f82440cbe20 0x7f82440cda60 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f822c009840 tx=0x7f822c009b50 comp rx=0 tx=0).stop 2023-10-16T13:27:32.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.990+0000 7f824a160700 1 -- 172.21.15.16:0/1026002333 shutdown_connections 2023-10-16T13:27:32.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.990+0000 7f824a160700 1 --2- 172.21.15.16:0/1026002333 >> v2:172.21.15.49:3300/0 conn(0x7f82440c6770 0x7f82440cdfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:32.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.990+0000 7f824a160700 1 --2- 172.21.15.16:0/1026002333 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f82440cbe20 0x7f82440cda60 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:32.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.990+0000 7f824a160700 1 -- 172.21.15.16:0/1026002333 >> 172.21.15.16:0/1026002333 conn(0x7f82440cc5e0 msgr2=0x7f82440cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:32.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.990+0000 7f824a160700 1 -- 172.21.15.16:0/1026002333 shutdown_connections 2023-10-16T13:27:32.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.990+0000 7f824a160700 1 -- 172.21.15.16:0/1026002333 wait complete. 2023-10-16T13:27:32.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.991+0000 7f824a160700 1 Processor -- start 2023-10-16T13:27:32.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.991+0000 7f824a160700 1 -- start start 2023-10-16T13:27:32.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.992+0000 7f824a160700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f82440c6770 0x7f82440c3820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:32.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.992+0000 7f824a160700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82440cbe20 0x7f82440cab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:32.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.992+0000 7f824a160700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f824415a640 0x7f82440cb050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:32.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.992+0000 7f824a160700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82440c80d0 con 0x7f82440cbe20 2023-10-16T13:27:32.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.992+0000 7f824a160700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82440c7550 con 0x7f824415a640 2023-10-16T13:27:32.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.992+0000 7f824a160700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82440c7830 con 0x7f82440c6770 2023-10-16T13:27:32.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.992+0000 7f8242ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82440cbe20 0x7f82440cab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:32.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.992+0000 7f8242ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82440cbe20 0x7f82440cab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48178/0 (socket says 172.21.15.16:48178) 2023-10-16T13:27:32.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.992+0000 7f8242ffd700 1 -- 172.21.15.16:0/4054685631 learned_addr learned my addr 172.21.15.16:0/4054685631 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:32.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.992+0000 7f82437fe700 1 --2- 172.21.15.16:0/4054685631 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f82440c6770 0x7f82440c3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:32.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.992+0000 7f8242ffd700 1 -- 172.21.15.16:0/4054685631 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f82440c6770 msgr2=0x7f82440c3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:32.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.992+0000 7f8242ffd700 1 --2- 172.21.15.16:0/4054685631 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f82440c6770 0x7f82440c3820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:32.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.992+0000 7f8242ffd700 1 -- 172.21.15.16:0/4054685631 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f824415a640 msgr2=0x7f82440cb050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:32.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.992+0000 7f8242ffd700 1 --2- 172.21.15.16:0/4054685631 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f824415a640 0x7f82440cb050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:32.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.992+0000 7f8242ffd700 1 -- 172.21.15.16:0/4054685631 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f823c009a70 con 0x7f82440cbe20 2023-10-16T13:27:32.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.993+0000 7f8242ffd700 1 --2- 172.21.15.16:0/4054685631 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82440cbe20 0x7f82440cab10 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7f822c000c00 tx=0x7f822c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:32.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.993+0000 7f823bfff700 1 -- 172.21.15.16:0/4054685631 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f822c0036a0 con 0x7f82440cbe20 2023-10-16T13:27:32.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.993+0000 7f823bfff700 1 -- 172.21.15.16:0/4054685631 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f822c01f070 con 0x7f82440cbe20 2023-10-16T13:27:32.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.993+0000 7f824a160700 1 -- 172.21.15.16:0/4054685631 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f822c0094a0 con 0x7f82440cbe20 2023-10-16T13:27:32.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.993+0000 7f823bfff700 1 -- 172.21.15.16:0/4054685631 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f822c003a40 con 0x7f82440cbe20 2023-10-16T13:27:32.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.993+0000 7f824a160700 1 -- 172.21.15.16:0/4054685631 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82440c7df0 con 0x7f82440cbe20 2023-10-16T13:27:32.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.994+0000 7f824a160700 1 -- 172.21.15.16:0/4054685631 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82440cd0d0 con 0x7f82440cbe20 2023-10-16T13:27:32.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.994+0000 7f823bfff700 1 -- 172.21.15.16:0/4054685631 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f822c003ba0 con 0x7f82440cbe20 2023-10-16T13:27:32.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.995+0000 7f823bfff700 1 --2- 172.21.15.16:0/4054685631 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f822406b020 0x7f822406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:32.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.995+0000 7f823bfff700 1 -- 172.21.15.16:0/4054685631 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 6051+0+0 (secure 0 0 0) 0x7f822c021070 con 0x7f82440cbe20 2023-10-16T13:27:32.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.995+0000 7f82437fe700 1 --2- 172.21.15.16:0/4054685631 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f822406b020 0x7f822406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:32.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.995+0000 7f82437fe700 1 --2- 172.21.15.16:0/4054685631 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f822406b020 0x7f822406d4e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8234001370 tx=0x7f8234009040 comp rx=0 tx=0).ready entity=mgr.9510 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:32.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:31.999+0000 7f823bfff700 1 -- 172.21.15.16:0/4054685631 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f822c055fb0 con 0x7f82440cbe20 2023-10-16T13:27:32.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:32.205+0000 7f824a160700 1 -- 172.21.15.16:0/4054685631 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f824404ebb0 con 0x7f82440cbe20 2023-10-16T13:27:32.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:32.208+0000 7f823bfff700 1 -- 172.21.15.16:0/4054685631 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v57) v1 ==== 120+0+0 (secure 0 0 0) 0x7f822c0595d0 con 0x7f82440cbe20 2023-10-16T13:27:32.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:32.210+0000 7f824a160700 1 -- 172.21.15.16:0/4054685631 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f822406b020 msgr2=0x7f822406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:32.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:32.210+0000 7f824a160700 1 --2- 172.21.15.16:0/4054685631 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f822406b020 0x7f822406d4e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8234001370 tx=0x7f8234009040 comp rx=0 tx=0).stop 2023-10-16T13:27:32.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:32.210+0000 7f824a160700 1 -- 172.21.15.16:0/4054685631 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82440cbe20 msgr2=0x7f82440cab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:32.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:32.210+0000 7f824a160700 1 --2- 172.21.15.16:0/4054685631 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82440cbe20 0x7f82440cab10 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7f822c000c00 tx=0x7f822c004750 comp rx=0 tx=0).stop 2023-10-16T13:27:32.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:32.210+0000 7f824a160700 1 -- 172.21.15.16:0/4054685631 shutdown_connections 2023-10-16T13:27:32.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:32.210+0000 7f824a160700 1 --2- 172.21.15.16:0/4054685631 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f822406b020 0x7f822406d4e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:32.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:32.210+0000 7f824a160700 1 --2- 172.21.15.16:0/4054685631 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f824415a640 0x7f82440cb050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:32.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:32.210+0000 7f824a160700 1 --2- 172.21.15.16:0/4054685631 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f82440c6770 0x7f82440c3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:32.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:32.210+0000 7f824a160700 1 --2- 172.21.15.16:0/4054685631 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82440cbe20 0x7f82440cab10 unknown :-1 s=CLOSED pgs=1869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:32.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:32.210+0000 7f824a160700 1 -- 172.21.15.16:0/4054685631 >> 172.21.15.16:0/4054685631 conn(0x7f82440cc5e0 msgr2=0x7f82440cda60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:32.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:32.211+0000 7f824a160700 1 -- 172.21.15.16:0/4054685631 shutdown_connections 2023-10-16T13:27:32.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:32.211+0000 7f824a160700 1 -- 172.21.15.16:0/4054685631 wait complete. 2023-10-16T13:27:32.226 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention' 2023-10-16T13:27:32.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:32.349+0000 7f6c27a65700 1 Processor -- start 2023-10-16T13:27:32.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:32.349+0000 7f6c27a65700 1 -- start start 2023-10-16T13:27:32.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:32.350+0000 7f6c27a65700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6c2015a5c0 0x7f6c200cef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:32.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:32.350+0000 7f6c27a65700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6c200ce010 0x7f6c200d03d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:32.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:32.350+0000 7f6c27a65700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c20072a50 con 0x7f6c200d0910 2023-10-16T13:27:32.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:32.350+0000 7f6c27a65700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c200cf440 con 0x7f6c200ce010 2023-10-16T13:27:32.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:32.350+0000 7f6c27a65700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c200d0da0 con 0x7f6c2015a5c0 2023-10-16T13:27:32.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:32.350+0000 7f6c25000700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6c200ce010 0x7f6c200d03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:32.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:32.350+0000 7f6c25000700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6c200ce010 0x7f6c200d03d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48186/0 (socket says 172.21.15.16:48186) 2023-10-16T13:27:32.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:32.350+0000 7f6c25000700 1 -- 172.21.15.16:0/1937321931 learned_addr learned my addr 172.21.15.16:0/1937321931 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:32.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:32.350+0000 7f6c25801700 1 --2- 172.21.15.16:0/1937321931 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6c2015a5c0 0x7f6c200cef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:32.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:32.350+0000 7f6c25000700 1 -- 172.21.15.16:0/1937321931 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6c2015a5c0 msgr2=0x7f6c200cef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:32.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:32.350+0000 7f6c25000700 1 --2- 172.21.15.16:0/1937321931 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6c2015a5c0 0x7f6c200cef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:32.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:32.350+0000 7f6c25000700 1 -- 172.21.15.16:0/1937321931 >> v1:172.21.15.16:6789/0 conn(0x7f6c200d0910 legacy=0x7f6c2015e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:32.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:32.350+0000 7f6c25000700 1 -- 172.21.15.16:0/1937321931 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c200cba30 con 0x7f6c200ce010 2023-10-16T13:27:32.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:32.351+0000 7f6c25000700 1 --2- 172.21.15.16:0/1937321931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c200ce010 0x7f6c200d03d0 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7f6c100098f0 tx=0x7f6c10009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3d40ffddf7e0d03 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:32.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:32.351+0000 7f6c17fff700 1 -- 172.21.15.16:0/1937321931 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c1000b8a0 con 0x7f6c200ce010 2023-10-16T13:27:32.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:32.351+0000 7f6c17fff700 1 -- 172.21.15.16:0/1937321931 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c1000ba00 con 0x7f6c200ce010 2023-10-16T13:27:32.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:32.351+0000 7f6c17fff700 1 -- 172.21.15.16:0/1937321931 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c1000bd90 con 0x7f6c200ce010 2023-10-16T13:27:32.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:32.351+0000 7f6c27a65700 1 -- 172.21.15.16:0/1937321931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c200ce010 msgr2=0x7f6c200d03d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:32.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:32.351+0000 7f6c27a65700 1 --2- 172.21.15.16:0/1937321931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c200ce010 0x7f6c200d03d0 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7f6c100098f0 tx=0x7f6c10009c00 comp rx=0 tx=0).stop 2023-10-16T13:27:32.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:32.351+0000 7f6c27a65700 1 -- 172.21.15.16:0/1937321931 shutdown_connections 2023-10-16T13:27:32.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:32.351+0000 7f6c27a65700 1 --2- 172.21.15.16:0/1937321931 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6c2015a5c0 0x7f6c200cef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:32.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:32.351+0000 7f6c27a65700 1 --2- 172.21.15.16:0/1937321931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c200ce010 0x7f6c200d03d0 unknown :-1 s=CLOSED pgs=1871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:32.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:32.351+0000 7f6c27a65700 1 -- 172.21.15.16:0/1937321931 >> 172.21.15.16:0/1937321931 conn(0x7f6c200b1fd0 msgr2=0x7f6c200b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:32.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:32.352+0000 7f6c27a65700 1 -- 172.21.15.16:0/1937321931 shutdown_connections 2023-10-16T13:27:32.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:32.352+0000 7f6c27a65700 1 -- 172.21.15.16:0/1937321931 wait complete. 2023-10-16T13:27:32.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:32.353+0000 7f6c27a65700 1 Processor -- start 2023-10-16T13:27:32.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:32.353+0000 7f6c27a65700 1 -- start start 2023-10-16T13:27:32.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:32.353+0000 7f6c27a65700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c200ce010 0x7f6c201f1630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:32.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:32.353+0000 7f6c27a65700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6c200d0910 0x7f6c2015c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:32.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:32.354+0000 7f6c27a65700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6c2015a5c0 0x7f6c2015cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:32.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:32.354+0000 7f6c27a65700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c201f1e30 con 0x7f6c200ce010 2023-10-16T13:27:32.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:32.354+0000 7f6c27a65700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c201f1b70 con 0x7f6c200d0910 2023-10-16T13:27:32.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:32.354+0000 7f6c27a65700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c201497c0 con 0x7f6c2015a5c0 2023-10-16T13:27:32.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:32.354+0000 7f6c25801700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c200ce010 0x7f6c201f1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:32.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:32.354+0000 7f6c25801700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c200ce010 0x7f6c201f1630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48190/0 (socket says 172.21.15.16:48190) 2023-10-16T13:27:32.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:32.354+0000 7f6c25801700 1 -- 172.21.15.16:0/1199535856 learned_addr learned my addr 172.21.15.16:0/1199535856 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:32.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:32.354+0000 7f6c26002700 1 --2- 172.21.15.16:0/1199535856 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6c2015a5c0 0x7f6c2015cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:32.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:32.354+0000 7f6c25000700 1 --2- 172.21.15.16:0/1199535856 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6c200d0910 0x7f6c2015c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:32.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:32.354+0000 7f6c25801700 1 -- 172.21.15.16:0/1199535856 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6c2015a5c0 msgr2=0x7f6c2015cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:32.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:32.354+0000 7f6c25801700 1 --2- 172.21.15.16:0/1199535856 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6c2015a5c0 0x7f6c2015cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:32.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:32.354+0000 7f6c25801700 1 -- 172.21.15.16:0/1199535856 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6c200d0910 msgr2=0x7f6c2015c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:32.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:32.354+0000 7f6c25801700 1 --2- 172.21.15.16:0/1199535856 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6c200d0910 0x7f6c2015c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:32.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:32.354+0000 7f6c25801700 1 -- 172.21.15.16:0/1199535856 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c10009550 con 0x7f6c200ce010 2023-10-16T13:27:32.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:32.354+0000 7f6c25801700 1 --2- 172.21.15.16:0/1199535856 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c200ce010 0x7f6c201f1630 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7f6c1c00c7f0 tx=0x7f6c1c00cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:32.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:32.354+0000 7f6c15ffb700 1 -- 172.21.15.16:0/1199535856 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c1c011800 con 0x7f6c200ce010 2023-10-16T13:27:32.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:32.354+0000 7f6c15ffb700 1 -- 172.21.15.16:0/1199535856 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c1c011960 con 0x7f6c200ce010 2023-10-16T13:27:32.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:32.354+0000 7f6c27a65700 1 -- 172.21.15.16:0/1199535856 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c20149aa0 con 0x7f6c200ce010 2023-10-16T13:27:32.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:32.354+0000 7f6c15ffb700 1 -- 172.21.15.16:0/1199535856 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c1c011cf0 con 0x7f6c200ce010 2023-10-16T13:27:32.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:32.355+0000 7f6c27a65700 1 -- 172.21.15.16:0/1199535856 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c2014a060 con 0x7f6c200ce010 2023-10-16T13:27:32.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:32.355+0000 7f6c27a65700 1 -- 172.21.15.16:0/1199535856 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c2004ebb0 con 0x7f6c200ce010 2023-10-16T13:27:32.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:32.356+0000 7f6c15ffb700 1 -- 172.21.15.16:0/1199535856 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6c1c018360 con 0x7f6c200ce010 2023-10-16T13:27:32.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:32.356+0000 7f6c15ffb700 1 --2- 172.21.15.16:0/1199535856 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f6c0406b020 0x7f6c0406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:32.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:32.356+0000 7f6c15ffb700 1 -- 172.21.15.16:0/1199535856 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 6051+0+0 (secure 0 0 0) 0x7f6c1c089960 con 0x7f6c200ce010 2023-10-16T13:27:32.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:32.356+0000 7f6c25000700 1 --2- 172.21.15.16:0/1199535856 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f6c0406b020 0x7f6c0406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:32.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:32.357+0000 7f6c25000700 1 --2- 172.21.15.16:0/1199535856 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f6c0406b020 0x7f6c0406d4e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6c10014040 tx=0x7f6c10008030 comp rx=0 tx=0).ready entity=mgr.9510 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:32.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:32.361+0000 7f6c15ffb700 1 -- 172.21.15.16:0/1199535856 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6c1c0546a0 con 0x7f6c200ce010 2023-10-16T13:27:32.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:32.656+0000 7f6c27a65700 1 -- 172.21.15.16:0/1199535856 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]} v 0) v1 -- 0x7f6c200b23b0 con 0x7f6c200ce010 2023-10-16T13:27:33.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.101+0000 7f6c15ffb700 1 -- 172.21.15.16:0/1199535856 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]}]=0 v2502) v1 ==== 178+0+0 (secure 0 0 0) 0x7f6c1c057cc0 con 0x7f6c200ce010 2023-10-16T13:27:33.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.104+0000 7f6c27a65700 1 -- 172.21.15.16:0/1199535856 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f6c0406b020 msgr2=0x7f6c0406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:33.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.104+0000 7f6c27a65700 1 --2- 172.21.15.16:0/1199535856 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f6c0406b020 0x7f6c0406d4e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6c10014040 tx=0x7f6c10008030 comp rx=0 tx=0).stop 2023-10-16T13:27:33.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.104+0000 7f6c27a65700 1 -- 172.21.15.16:0/1199535856 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c200ce010 msgr2=0x7f6c201f1630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:33.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.104+0000 7f6c27a65700 1 --2- 172.21.15.16:0/1199535856 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c200ce010 0x7f6c201f1630 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7f6c1c00c7f0 tx=0x7f6c1c00cbb0 comp rx=0 tx=0).stop 2023-10-16T13:27:33.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.104+0000 7f6c27a65700 1 -- 172.21.15.16:0/1199535856 shutdown_connections 2023-10-16T13:27:33.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.104+0000 7f6c27a65700 1 --2- 172.21.15.16:0/1199535856 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6c2015a5c0 0x7f6c2015cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:33.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.104+0000 7f6c27a65700 1 --2- 172.21.15.16:0/1199535856 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c200ce010 0x7f6c201f1630 unknown :-1 s=CLOSED pgs=1872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:33.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.104+0000 7f6c27a65700 1 --2- 172.21.15.16:0/1199535856 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f6c0406b020 0x7f6c0406d4e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:33.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.104+0000 7f6c27a65700 1 --2- 172.21.15.16:0/1199535856 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6c200d0910 0x7f6c2015c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:33.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.104+0000 7f6c27a65700 1 -- 172.21.15.16:0/1199535856 >> 172.21.15.16:0/1199535856 conn(0x7f6c200b1fd0 msgr2=0x7f6c200cec40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:33.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.105+0000 7f6c27a65700 1 -- 172.21.15.16:0/1199535856 shutdown_connections 2023-10-16T13:27:33.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.105+0000 7f6c27a65700 1 -- 172.21.15.16:0/1199535856 wait complete. 2023-10-16T13:27:33.122 INFO:tasks.cephfs_test_runner:Test that number of snaps retained are as per user spec. ... ok 2023-10-16T13:27:33.123 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_list_post_schedule_remove (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test listing snap schedules post removal of a schedule 2023-10-16T13:27:33.123 INFO:tasks.cephfs_test_runner:test_snap_schedule_list_post_schedule_remove (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-16T13:27:33.124 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:27:33.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.253+0000 7f176c0e0700 1 Processor -- start 2023-10-16T13:27:33.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.253+0000 7f176c0e0700 1 -- start start 2023-10-16T13:27:33.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.254+0000 7f176c0e0700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f17640cc8a0 0x7f176415e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:33.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.254+0000 7f176c0e0700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1764161c30 con 0x7f176406a7e0 2023-10-16T13:27:33.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.254+0000 7f176c0e0700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1764161f10 con 0x7f176415b280 2023-10-16T13:27:33.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.254+0000 7f176c0e0700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17641621f0 con 0x7f17640cc8a0 2023-10-16T13:27:33.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.254+0000 7f176a67d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f176406a7e0 0x7f17640d02b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37476/0 (socket says 172.21.15.16:37476) 2023-10-16T13:27:33.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.254+0000 7f176a67d700 1 -- 172.21.15.16:0/3423120502 learned_addr learned my addr 172.21.15.16:0/3423120502 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:27:33.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.254+0000 7f1768e7a700 1 -- 172.21.15.16:0/3423120502 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 319086367 0 0) 0x7f1764161c30 con 0x7f176406a7e0 2023-10-16T13:27:33.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.254+0000 7f1768e7a700 1 -- 172.21.15.16:0/3423120502 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17500035b0 con 0x7f176406a7e0 2023-10-16T13:27:33.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.254+0000 7f176967b700 1 --2- 172.21.15.16:0/3423120502 >> v2:172.21.15.49:3300/0 conn(0x7f17640cc8a0 0x7f176415e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:33.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.254+0000 7f1768e7a700 1 -- 172.21.15.16:0/3423120502 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2702790115 0 0) 0x7f17500035b0 con 0x7f176406a7e0 2023-10-16T13:27:33.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.254+0000 7f1768e7a700 1 -- 172.21.15.16:0/3423120502 >> v2:172.21.15.49:3300/0 conn(0x7f17640cc8a0 msgr2=0x7f176415e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:33.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.254+0000 7f1768e7a700 1 --2- 172.21.15.16:0/3423120502 >> v2:172.21.15.49:3300/0 conn(0x7f17640cc8a0 0x7f176415e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:33.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.254+0000 7f1768e7a700 1 -- 172.21.15.16:0/3423120502 >> v1:172.21.15.49:6789/0 conn(0x7f176415b280 legacy=0x7f17641613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:33.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.254+0000 7f1768e7a700 1 -- 172.21.15.16:0/3423120502 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f176415df00 con 0x7f176406a7e0 2023-10-16T13:27:33.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.254+0000 7f1768e7a700 1 -- 172.21.15.16:0/3423120502 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f175c0019f0 con 0x7f176406a7e0 2023-10-16T13:27:33.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.255+0000 7f1768e7a700 1 -- 172.21.15.16:0/3423120502 >> v1:172.21.15.16:6789/0 conn(0x7f176406a7e0 legacy=0x7f17640d02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:33.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.255+0000 7f1768e7a700 1 --2- 172.21.15.16:0/3423120502 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1750003a00 0x7f1750003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:33.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.255+0000 7f1768e7a700 1 --2- 172.21.15.16:0/3423120502 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1750004850 0x7f1750004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:33.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.255+0000 7f1768e7a700 1 --2- 172.21.15.16:0/3423120502 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1750005210 0x7f175000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:33.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.255+0000 7f1768e7a700 1 -- 172.21.15.16:0/3423120502 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17641621f0 con 0x7f1750004850 2023-10-16T13:27:33.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.255+0000 7f1768e7a700 1 -- 172.21.15.16:0/3423120502 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f175000c0b0 con 0x7f1750005210 2023-10-16T13:27:33.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.255+0000 7f1768e7a700 1 -- 172.21.15.16:0/3423120502 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f175000c270 con 0x7f1750003a00 2023-10-16T13:27:33.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.255+0000 7f176967b700 1 --2- 172.21.15.16:0/3423120502 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1750004850 0x7f1750004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:33.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.255+0000 7f176a67d700 1 --2- 172.21.15.16:0/3423120502 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1750005210 0x7f175000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:33.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.255+0000 7f1769e7c700 1 --2- 172.21.15.16:0/3423120502 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1750003a00 0x7f1750003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:33.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.255+0000 7f176967b700 1 -- 172.21.15.16:0/3423120502 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1750003a00 msgr2=0x7f1750003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:33.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.255+0000 7f176967b700 1 --2- 172.21.15.16:0/3423120502 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1750003a00 0x7f1750003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:33.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.255+0000 7f176967b700 1 -- 172.21.15.16:0/3423120502 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1750005210 msgr2=0x7f175000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:33.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.255+0000 7f176967b700 1 --2- 172.21.15.16:0/3423120502 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1750005210 0x7f175000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:33.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.255+0000 7f176a67d700 1 -- 172.21.15.16:0/3423120502 reap_dead start 2023-10-16T13:27:33.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.255+0000 7f176967b700 1 -- 172.21.15.16:0/3423120502 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17500035b0 con 0x7f1750004850 2023-10-16T13:27:33.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.255+0000 7f176967b700 1 --2- 172.21.15.16:0/3423120502 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1750004850 0x7f1750004cd0 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7f1754009b30 tx=0x7f1750005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b25eaf0682f1af88 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:33.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.255+0000 7f1768e7a700 1 -- 172.21.15.16:0/3423120502 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f175400e6c0 con 0x7f1750004850 2023-10-16T13:27:33.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.255+0000 7f1768e7a700 1 -- 172.21.15.16:0/3423120502 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f175400e820 con 0x7f1750004850 2023-10-16T13:27:33.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.255+0000 7f1768e7a700 1 -- 172.21.15.16:0/3423120502 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f175400eb90 con 0x7f1750004850 2023-10-16T13:27:33.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.256+0000 7f176c0e0700 1 -- 172.21.15.16:0/3423120502 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1750004850 msgr2=0x7f1750004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:33.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.256+0000 7f176c0e0700 1 --2- 172.21.15.16:0/3423120502 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1750004850 0x7f1750004cd0 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7f1754009b30 tx=0x7f1750005880 comp rx=0 tx=0).stop 2023-10-16T13:27:33.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.256+0000 7f176c0e0700 1 -- 172.21.15.16:0/3423120502 shutdown_connections 2023-10-16T13:27:33.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.256+0000 7f176c0e0700 1 --2- 172.21.15.16:0/3423120502 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1750004850 0x7f1750004cd0 unknown :-1 s=CLOSED pgs=1874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:33.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.256+0000 7f176c0e0700 1 -- 172.21.15.16:0/3423120502 >> 172.21.15.16:0/3423120502 conn(0x7f17640b1fd0 msgr2=0x7f17640b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:33.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.256+0000 7f176c0e0700 1 -- 172.21.15.16:0/3423120502 shutdown_connections 2023-10-16T13:27:33.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.256+0000 7f176c0e0700 1 -- 172.21.15.16:0/3423120502 wait complete. 2023-10-16T13:27:33.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.257+0000 7f176c0e0700 1 Processor -- start 2023-10-16T13:27:33.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.257+0000 7f176c0e0700 1 -- start start 2023-10-16T13:27:33.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.258+0000 7f176c0e0700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1750004850 0x7f1764159dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:33.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.258+0000 7f176c0e0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17641f2be0 0x7f176415f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:33.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.258+0000 7f176c0e0700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f176415fae0 0x7f17640c3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:33.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.258+0000 7f176c0e0700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f176415ff60 con 0x7f17641f2be0 2023-10-16T13:27:33.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.258+0000 7f176c0e0700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17640c58e0 con 0x7f1750004850 2023-10-16T13:27:33.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.258+0000 7f176c0e0700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17640c5b30 con 0x7f176415fae0 2023-10-16T13:27:33.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.258+0000 7f176967b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17641f2be0 0x7f176415f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:33.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.258+0000 7f176967b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17641f2be0 0x7f176415f5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48212/0 (socket says 172.21.15.16:48212) 2023-10-16T13:27:33.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.258+0000 7f176967b700 1 -- 172.21.15.16:0/3114124498 learned_addr learned my addr 172.21.15.16:0/3114124498 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:33.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.258+0000 7f1769e7c700 1 --2- 172.21.15.16:0/3114124498 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1750004850 0x7f1764159dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:33.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.258+0000 7f176a67d700 1 --2- 172.21.15.16:0/3114124498 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f176415fae0 0x7f17640c3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:33.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.258+0000 7f176967b700 1 -- 172.21.15.16:0/3114124498 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f176415fae0 msgr2=0x7f17640c3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:33.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.258+0000 7f176967b700 1 --2- 172.21.15.16:0/3114124498 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f176415fae0 0x7f17640c3f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:33.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.258+0000 7f176967b700 1 -- 172.21.15.16:0/3114124498 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1750004850 msgr2=0x7f1764159dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:33.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.258+0000 7f176967b700 1 --2- 172.21.15.16:0/3114124498 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1750004850 0x7f1764159dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:33.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.258+0000 7f176967b700 1 -- 172.21.15.16:0/3114124498 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17640c5e10 con 0x7f17641f2be0 2023-10-16T13:27:33.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.258+0000 7f1769e7c700 1 --2- 172.21.15.16:0/3114124498 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1750004850 0x7f1764159dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:27:33.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.258+0000 7f176967b700 1 --2- 172.21.15.16:0/3114124498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17641f2be0 0x7f176415f5a0 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7f175000c080 tx=0x7f17540047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:33.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.259+0000 7f175a7fc700 1 -- 172.21.15.16:0/3114124498 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f175400e6c0 con 0x7f17641f2be0 2023-10-16T13:27:33.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.259+0000 7f175a7fc700 1 -- 172.21.15.16:0/3114124498 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1754013c00 con 0x7f17641f2be0 2023-10-16T13:27:33.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.259+0000 7f176c0e0700 1 -- 172.21.15.16:0/3114124498 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17640c60f0 con 0x7f17641f2be0 2023-10-16T13:27:33.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.259+0000 7f175a7fc700 1 -- 172.21.15.16:0/3114124498 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f175401c450 con 0x7f17641f2be0 2023-10-16T13:27:33.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.259+0000 7f176c0e0700 1 -- 172.21.15.16:0/3114124498 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17640c66b0 con 0x7f17641f2be0 2023-10-16T13:27:33.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.260+0000 7f176c0e0700 1 -- 172.21.15.16:0/3114124498 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17640ad780 con 0x7f17641f2be0 2023-10-16T13:27:33.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.260+0000 7f175a7fc700 1 -- 172.21.15.16:0/3114124498 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1754024070 con 0x7f17641f2be0 2023-10-16T13:27:33.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.260+0000 7f175a7fc700 1 --2- 172.21.15.16:0/3114124498 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f174806b020 0x7f174806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:33.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.261+0000 7f175a7fc700 1 -- 172.21.15.16:0/3114124498 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 6051+0+0 (secure 0 0 0) 0x7f175408b140 con 0x7f17641f2be0 2023-10-16T13:27:33.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.261+0000 7f1769e7c700 1 --2- 172.21.15.16:0/3114124498 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f174806b020 0x7f174806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:33.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.261+0000 7f1769e7c700 1 --2- 172.21.15.16:0/3114124498 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f174806b020 0x7f174806d4e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f1760000c00 tx=0x7f1760009040 comp rx=0 tx=0).ready entity=mgr.9510 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:33.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.265+0000 7f175a7fc700 1 -- 172.21.15.16:0/3114124498 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1754056130 con 0x7f17641f2be0 2023-10-16T13:27:33.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.468+0000 7f176c0e0700 1 -- 172.21.15.16:0/3114124498 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f176404ebb0 con 0x7f17641f2be0 2023-10-16T13:27:33.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.469+0000 7f175a7fc700 1 -- 172.21.15.16:0/3114124498 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v168) v1 ==== 74+0+14253 (secure 0 0 0) 0x7f1754059750 con 0x7f17641f2be0 2023-10-16T13:27:33.472 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:27:33.472 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":168,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:27:31.095195+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6836/1477717813":"2023-10-17T13:27:27.851634+0000","172.21.15.16:0/756925648":"2023-10-17T13:27:26.798652+0000","172.21.15.16:0/1677462197":"2023-10-17T13:27:26.798652+0000","172.21.15.16:6833/1503042987":"2023-10-17T13:27:26.798652+0000","172.21.15.16:6832/1503042987":"2023-10-17T13:27:26.798652+0000","172.21.15.16:0/719467386":"2023-10-17T13:27:26.798652+0000","172.21.15.16:0/3389751869":"2023-10-17T13:27:26.798652+0000","172.21.15.16:0/354924692":"2023-10-17T13:27:26.798652+0000","172.21.15.16:0/1922950917":"2023-10-17T13:27:26.798652+0000","172.21.15.16:0/3195044045":"2023-10-17T12:31:20.854761+0000","172.21.15.16:6833/4182163087":"2023-10-17T12:31:20.854761+0000","172.21.15.16:6832/4182163087":"2023-10-17T12:31:20.854761+0000","172.21.15.16:0/4217694331":"2023-10-17T12:31:20.854761+0000","172.21.15.16:6834/1477717813":"2023-10-17T13:27:27.851634+0000","172.21.15.16:0/2516829556":"2023-10-17T12:31:20.854761+0000","172.21.15.16:0/2546955595":"2023-10-17T12:31:20.854761+0000","172.21.15.16:0/1851287351":"2023-10-17T12:31:20.854761+0000","172.21.15.16:0/3755256116":"2023-10-17T12:31:20.854761+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:27:33.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.471+0000 7f176c0e0700 1 -- 172.21.15.16:0/3114124498 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f174806b020 msgr2=0x7f174806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:33.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.471+0000 7f176c0e0700 1 --2- 172.21.15.16:0/3114124498 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f174806b020 0x7f174806d4e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f1760000c00 tx=0x7f1760009040 comp rx=0 tx=0).stop 2023-10-16T13:27:33.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.471+0000 7f176c0e0700 1 -- 172.21.15.16:0/3114124498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17641f2be0 msgr2=0x7f176415f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:33.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.471+0000 7f176c0e0700 1 --2- 172.21.15.16:0/3114124498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17641f2be0 0x7f176415f5a0 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7f175000c080 tx=0x7f17540047c0 comp rx=0 tx=0).stop 2023-10-16T13:27:33.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.471+0000 7f176c0e0700 1 -- 172.21.15.16:0/3114124498 shutdown_connections 2023-10-16T13:27:33.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.471+0000 7f176c0e0700 1 --2- 172.21.15.16:0/3114124498 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f176415fae0 0x7f17640c3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:33.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.471+0000 7f176c0e0700 1 --2- 172.21.15.16:0/3114124498 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f174806b020 0x7f174806d4e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:33.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.471+0000 7f176c0e0700 1 --2- 172.21.15.16:0/3114124498 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1750004850 0x7f1764159dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:33.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.471+0000 7f176c0e0700 1 --2- 172.21.15.16:0/3114124498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17641f2be0 0x7f176415f5a0 unknown :-1 s=CLOSED pgs=1875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:33.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.471+0000 7f176c0e0700 1 -- 172.21.15.16:0/3114124498 >> 172.21.15.16:0/3114124498 conn(0x7f17640b1fd0 msgr2=0x7f17640cb2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:33.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.471+0000 7f176c0e0700 1 -- 172.21.15.16:0/3114124498 shutdown_connections 2023-10-16T13:27:33.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.471+0000 7f176c0e0700 1 -- 172.21.15.16:0/3114124498 wait complete. 2023-10-16T13:27:33.486 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T13:27:33.487 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T13:27:33.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.612+0000 7f426de75700 1 Processor -- start 2023-10-16T13:27:33.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.613+0000 7f426de75700 1 -- start start 2023-10-16T13:27:33.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.613+0000 7f426de75700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f426815b070 0x7f426815b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:33.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.613+0000 7f426de75700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f42680cb880 0x7f42680cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:33.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.613+0000 7f426de75700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42680cef70 con 0x7f426806da60 2023-10-16T13:27:33.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.613+0000 7f426de75700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42680ce360 con 0x7f42680cb880 2023-10-16T13:27:33.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.613+0000 7f426de75700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42680ce640 con 0x7f426815b070 2023-10-16T13:27:33.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.613+0000 7f4266ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f42680cb880 0x7f42680cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:33.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.613+0000 7f4266ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f42680cb880 0x7f42680cbd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48220/0 (socket says 172.21.15.16:48220) 2023-10-16T13:27:33.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.613+0000 7f4267fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f426806da60 0x7f42680cf400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37496/0 (socket says 172.21.15.16:37496) 2023-10-16T13:27:33.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.613+0000 7f4266ffd700 1 -- 172.21.15.16:0/3030765833 learned_addr learned my addr 172.21.15.16:0/3030765833 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:33.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.614+0000 7f42677fe700 1 --2- 172.21.15.16:0/3030765833 >> v2:172.21.15.49:3300/0 conn(0x7f426815b070 0x7f426815b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:33.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.614+0000 7f4266ffd700 1 -- 172.21.15.16:0/3030765833 >> v2:172.21.15.49:3300/0 conn(0x7f426815b070 msgr2=0x7f426815b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:33.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.614+0000 7f4266ffd700 1 --2- 172.21.15.16:0/3030765833 >> v2:172.21.15.49:3300/0 conn(0x7f426815b070 0x7f426815b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:33.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.614+0000 7f4266ffd700 1 -- 172.21.15.16:0/3030765833 >> v1:172.21.15.16:6789/0 conn(0x7f426806da60 legacy=0x7f42680cf400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:33.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.614+0000 7f4266ffd700 1 -- 172.21.15.16:0/3030765833 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42680ce920 con 0x7f42680cb880 2023-10-16T13:27:33.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.614+0000 7f42667fc700 1 -- 172.21.15.16:0/3030765833 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3163049302 0 0) 0x7f42680cef70 con 0x7f426806da60 2023-10-16T13:27:33.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.614+0000 7f42677fe700 1 --2- 172.21.15.16:0/3030765833 >> v2:172.21.15.49:3300/0 conn(0x7f426815b070 0x7f426815b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:27:33.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.614+0000 7f4266ffd700 1 --2- 172.21.15.16:0/3030765833 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42680cb880 0x7f42680cbd00 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7f42680705a0 tx=0x7f4250009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dc1efea77d6121d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:33.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.614+0000 7f42667fc700 1 -- 172.21.15.16:0/3030765833 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f425000b870 con 0x7f42680cb880 2023-10-16T13:27:33.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.614+0000 7f42667fc700 1 -- 172.21.15.16:0/3030765833 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f425000b9d0 con 0x7f42680cb880 2023-10-16T13:27:33.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.614+0000 7f42667fc700 1 -- 172.21.15.16:0/3030765833 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f425000bd60 con 0x7f42680cb880 2023-10-16T13:27:33.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.615+0000 7f426de75700 1 -- 172.21.15.16:0/3030765833 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42680cb880 msgr2=0x7f42680cbd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:33.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.615+0000 7f426de75700 1 --2- 172.21.15.16:0/3030765833 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42680cb880 0x7f42680cbd00 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7f42680705a0 tx=0x7f4250009bd0 comp rx=0 tx=0).stop 2023-10-16T13:27:33.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.615+0000 7f426de75700 1 -- 172.21.15.16:0/3030765833 shutdown_connections 2023-10-16T13:27:33.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.615+0000 7f426de75700 1 --2- 172.21.15.16:0/3030765833 >> v2:172.21.15.49:3300/0 conn(0x7f426815b070 0x7f426815b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:33.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.615+0000 7f426de75700 1 --2- 172.21.15.16:0/3030765833 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42680cb880 0x7f42680cbd00 unknown :-1 s=CLOSED pgs=1877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:33.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.615+0000 7f426de75700 1 -- 172.21.15.16:0/3030765833 >> 172.21.15.16:0/3030765833 conn(0x7f42680b1f40 msgr2=0x7f42680b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:33.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.615+0000 7f426de75700 1 -- 172.21.15.16:0/3030765833 shutdown_connections 2023-10-16T13:27:33.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.615+0000 7f426de75700 1 -- 172.21.15.16:0/3030765833 wait complete. 2023-10-16T13:27:33.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.616+0000 7f426de75700 1 Processor -- start 2023-10-16T13:27:33.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.616+0000 7f426de75700 1 -- start start 2023-10-16T13:27:33.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.617+0000 7f426de75700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f426806da60 0x7f42681641d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:33.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.617+0000 7f426de75700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f42680cb880 0x7f4268164950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:33.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.617+0000 7f426de75700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f426815b070 0x7f426815cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:33.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.617+0000 7f426de75700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42681675d0 con 0x7f426806da60 2023-10-16T13:27:33.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.617+0000 7f426de75700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4268164e90 con 0x7f426815b070 2023-10-16T13:27:33.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.617+0000 7f426de75700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4268166fa0 con 0x7f42680cb880 2023-10-16T13:27:33.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.617+0000 7f42677fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f426806da60 0x7f42681641d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:33.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.617+0000 7f42677fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f426806da60 0x7f42681641d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48224/0 (socket says 172.21.15.16:48224) 2023-10-16T13:27:33.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.617+0000 7f42677fe700 1 -- 172.21.15.16:0/1544560392 learned_addr learned my addr 172.21.15.16:0/1544560392 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:33.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.617+0000 7f4267fff700 1 --2- 172.21.15.16:0/1544560392 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f426815b070 0x7f426815cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:33.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.617+0000 7f4266ffd700 1 --2- 172.21.15.16:0/1544560392 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f42680cb880 0x7f4268164950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:33.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.617+0000 7f42677fe700 1 -- 172.21.15.16:0/1544560392 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f42680cb880 msgr2=0x7f4268164950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:33.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.617+0000 7f42677fe700 1 --2- 172.21.15.16:0/1544560392 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f42680cb880 0x7f4268164950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:33.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.617+0000 7f42677fe700 1 -- 172.21.15.16:0/1544560392 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f426815b070 msgr2=0x7f426815cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:33.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.617+0000 7f42677fe700 1 --2- 172.21.15.16:0/1544560392 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f426815b070 0x7f426815cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:33.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.617+0000 7f42677fe700 1 -- 172.21.15.16:0/1544560392 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4250009550 con 0x7f426806da60 2023-10-16T13:27:33.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.617+0000 7f4266ffd700 1 --2- 172.21.15.16:0/1544560392 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f42680cb880 0x7f4268164950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:27:33.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.617+0000 7f4267fff700 1 --2- 172.21.15.16:0/1544560392 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f426815b070 0x7f426815cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:27:33.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.618+0000 7f42677fe700 1 --2- 172.21.15.16:0/1544560392 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f426806da60 0x7f42681641d0 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7f425800c7f0 tx=0x7f425800cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:33.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.618+0000 7f425ffff700 1 -- 172.21.15.16:0/1544560392 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4258011800 con 0x7f426806da60 2023-10-16T13:27:33.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.618+0000 7f425ffff700 1 -- 172.21.15.16:0/1544560392 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4258011960 con 0x7f426806da60 2023-10-16T13:27:33.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.618+0000 7f426de75700 1 -- 172.21.15.16:0/1544560392 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4268165130 con 0x7f426806da60 2023-10-16T13:27:33.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.618+0000 7f425ffff700 1 -- 172.21.15.16:0/1544560392 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4258011d10 con 0x7f426806da60 2023-10-16T13:27:33.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.618+0000 7f426de75700 1 -- 172.21.15.16:0/1544560392 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4268167260 con 0x7f426806da60 2023-10-16T13:27:33.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.619+0000 7f426de75700 1 -- 172.21.15.16:0/1544560392 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42681678b0 con 0x7f426806da60 2023-10-16T13:27:33.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.619+0000 7f425ffff700 1 -- 172.21.15.16:0/1544560392 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4258018360 con 0x7f426806da60 2023-10-16T13:27:33.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.620+0000 7f425ffff700 1 --2- 172.21.15.16:0/1544560392 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f424806b020 0x7f424806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:33.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.620+0000 7f425ffff700 1 -- 172.21.15.16:0/1544560392 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 6051+0+0 (secure 0 0 0) 0x7f4258089960 con 0x7f426806da60 2023-10-16T13:27:33.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.620+0000 7f4266ffd700 1 --2- 172.21.15.16:0/1544560392 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f424806b020 0x7f424806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:33.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.620+0000 7f4266ffd700 1 --2- 172.21.15.16:0/1544560392 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f424806b020 0x7f424806d4e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f42680c70b0 tx=0x7f4250008030 comp rx=0 tx=0).ready entity=mgr.9510 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:33.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.624+0000 7f425ffff700 1 -- 172.21.15.16:0/1544560392 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4258054690 con 0x7f426806da60 2023-10-16T13:27:33.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.827+0000 7f426de75700 1 -- 172.21.15.16:0/1544560392 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f426804ebb0 con 0x7f426806da60 2023-10-16T13:27:33.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.828+0000 7f425ffff700 1 -- 172.21.15.16:0/1544560392 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v168) v1 ==== 93+0+10 (secure 0 0 0) 0x7f425801f0a0 con 0x7f426806da60 2023-10-16T13:27:33.831 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T13:27:33.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.830+0000 7f426de75700 1 -- 172.21.15.16:0/1544560392 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f424806b020 msgr2=0x7f424806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:33.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.830+0000 7f426de75700 1 --2- 172.21.15.16:0/1544560392 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f424806b020 0x7f424806d4e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f42680c70b0 tx=0x7f4250008030 comp rx=0 tx=0).stop 2023-10-16T13:27:33.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.830+0000 7f426de75700 1 -- 172.21.15.16:0/1544560392 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f426806da60 msgr2=0x7f42681641d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:33.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.830+0000 7f426de75700 1 --2- 172.21.15.16:0/1544560392 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f426806da60 0x7f42681641d0 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7f425800c7f0 tx=0x7f425800cbb0 comp rx=0 tx=0).stop 2023-10-16T13:27:33.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.830+0000 7f426de75700 1 -- 172.21.15.16:0/1544560392 shutdown_connections 2023-10-16T13:27:33.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.830+0000 7f426de75700 1 --2- 172.21.15.16:0/1544560392 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f424806b020 0x7f424806d4e0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:33.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.830+0000 7f426de75700 1 --2- 172.21.15.16:0/1544560392 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f426815b070 0x7f426815cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:33.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.830+0000 7f426de75700 1 --2- 172.21.15.16:0/1544560392 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f426806da60 0x7f42681641d0 unknown :-1 s=CLOSED pgs=1878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:33.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.830+0000 7f426de75700 1 --2- 172.21.15.16:0/1544560392 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f42680cb880 0x7f4268164950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:33.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.830+0000 7f426de75700 1 -- 172.21.15.16:0/1544560392 >> 172.21.15.16:0/1544560392 conn(0x7f42680b1f40 msgr2=0x7f42680c6920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:33.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.830+0000 7f426de75700 1 -- 172.21.15.16:0/1544560392 shutdown_connections 2023-10-16T13:27:33.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.830+0000 7f426de75700 1 -- 172.21.15.16:0/1544560392 wait complete. 2023-10-16T13:27:33.846 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove' 2023-10-16T13:27:33.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.969+0000 7f87b3264700 1 Processor -- start 2023-10-16T13:27:33.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.970+0000 7f87b3264700 1 -- start start 2023-10-16T13:27:33.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.970+0000 7f87b3264700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f87ac0cf880 0x7f87ac0b52f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:33.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.970+0000 7f87b3264700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f87ac0b5830 0x7f87ac15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:33.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.970+0000 7f87b3264700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87ac15bc80 con 0x7f87ac0cf4a0 2023-10-16T13:27:33.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.970+0000 7f87b3264700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87ac15b070 con 0x7f87ac0cf880 2023-10-16T13:27:33.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.970+0000 7f87b3264700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87ac15b350 con 0x7f87ac0b5830 2023-10-16T13:27:33.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.971+0000 7f87abfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f87ac0cf880 0x7f87ac0b52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:33.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.971+0000 7f87abfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f87ac0cf880 0x7f87ac0b52f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48232/0 (socket says 172.21.15.16:48232) 2023-10-16T13:27:33.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.971+0000 7f87abfff700 1 -- 172.21.15.16:0/922288456 learned_addr learned my addr 172.21.15.16:0/922288456 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:33.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.971+0000 7f87b1801700 1 --2- 172.21.15.16:0/922288456 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f87ac0b5830 0x7f87ac15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:33.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.971+0000 7f87abfff700 1 -- 172.21.15.16:0/922288456 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f87ac0b5830 msgr2=0x7f87ac15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:33.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.971+0000 7f87abfff700 1 --2- 172.21.15.16:0/922288456 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f87ac0b5830 0x7f87ac15a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:33.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.971+0000 7f87abfff700 1 -- 172.21.15.16:0/922288456 >> v1:172.21.15.49:6789/0 conn(0x7f87ac0cf4a0 legacy=0x7f87ac0726e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:33.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.971+0000 7f87abfff700 1 -- 172.21.15.16:0/922288456 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87ac15b630 con 0x7f87ac0cf880 2023-10-16T13:27:33.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.971+0000 7f87abfff700 1 --2- 172.21.15.16:0/922288456 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87ac0cf880 0x7f87ac0b52f0 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7f87980098f0 tx=0x7f8798009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b04a112fdc3749e5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:33.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.971+0000 7f87ab7fe700 1 -- 172.21.15.16:0/922288456 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f879800b8a0 con 0x7f87ac0cf880 2023-10-16T13:27:33.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.971+0000 7f87ab7fe700 1 -- 172.21.15.16:0/922288456 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f879800ba00 con 0x7f87ac0cf880 2023-10-16T13:27:33.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.971+0000 7f87ab7fe700 1 -- 172.21.15.16:0/922288456 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f879800bd90 con 0x7f87ac0cf880 2023-10-16T13:27:33.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.972+0000 7f87b3264700 1 -- 172.21.15.16:0/922288456 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87ac0cf880 msgr2=0x7f87ac0b52f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:33.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.972+0000 7f87b3264700 1 --2- 172.21.15.16:0/922288456 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87ac0cf880 0x7f87ac0b52f0 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7f87980098f0 tx=0x7f8798009c00 comp rx=0 tx=0).stop 2023-10-16T13:27:33.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.972+0000 7f87b3264700 1 -- 172.21.15.16:0/922288456 shutdown_connections 2023-10-16T13:27:33.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.972+0000 7f87b3264700 1 --2- 172.21.15.16:0/922288456 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f87ac0b5830 0x7f87ac15a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:33.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.972+0000 7f87b3264700 1 --2- 172.21.15.16:0/922288456 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87ac0cf880 0x7f87ac0b52f0 unknown :-1 s=CLOSED pgs=1879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:33.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.972+0000 7f87b3264700 1 -- 172.21.15.16:0/922288456 >> 172.21.15.16:0/922288456 conn(0x7f87ac0b1fd0 msgr2=0x7f87ac0b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:33.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.972+0000 7f87b3264700 1 -- 172.21.15.16:0/922288456 shutdown_connections 2023-10-16T13:27:33.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.972+0000 7f87b3264700 1 -- 172.21.15.16:0/922288456 wait complete. 2023-10-16T13:27:33.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.973+0000 7f87b3264700 1 Processor -- start 2023-10-16T13:27:33.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.973+0000 7f87b3264700 1 -- start start 2023-10-16T13:27:33.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.974+0000 7f87b3264700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f87ac0b5830 0x7f87ac14ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:33.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.974+0000 7f87b3264700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87ac0cf4a0 0x7f87ac0c63f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:33.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.974+0000 7f87b3264700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f87ac0cf880 0x7f87ac0c8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:33.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.974+0000 7f87b3264700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87ac159fe0 con 0x7f87ac0cf4a0 2023-10-16T13:27:33.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.974+0000 7f87b3264700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87ac0c79d0 con 0x7f87ac0cf880 2023-10-16T13:27:33.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.974+0000 7f87b3264700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87ac0c7cb0 con 0x7f87ac0b5830 2023-10-16T13:27:33.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.974+0000 7f87abfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87ac0cf4a0 0x7f87ac0c63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:33.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.974+0000 7f87abfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87ac0cf4a0 0x7f87ac0c63f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48240/0 (socket says 172.21.15.16:48240) 2023-10-16T13:27:33.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.974+0000 7f87abfff700 1 -- 172.21.15.16:0/278680108 learned_addr learned my addr 172.21.15.16:0/278680108 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:33.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.974+0000 7f87b1000700 1 --2- 172.21.15.16:0/278680108 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f87ac0b5830 0x7f87ac14ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:33.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.975+0000 7f87b1801700 1 --2- 172.21.15.16:0/278680108 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f87ac0cf880 0x7f87ac0c8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:33.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.975+0000 7f87abfff700 1 -- 172.21.15.16:0/278680108 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f87ac0b5830 msgr2=0x7f87ac14ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:33.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.975+0000 7f87abfff700 1 --2- 172.21.15.16:0/278680108 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f87ac0b5830 0x7f87ac14ee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:33.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.975+0000 7f87abfff700 1 -- 172.21.15.16:0/278680108 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f87ac0cf880 msgr2=0x7f87ac0c8a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:33.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.975+0000 7f87abfff700 1 --2- 172.21.15.16:0/278680108 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f87ac0cf880 0x7f87ac0c8a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:33.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.975+0000 7f87abfff700 1 -- 172.21.15.16:0/278680108 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8798009550 con 0x7f87ac0cf4a0 2023-10-16T13:27:33.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.975+0000 7f87b1801700 1 --2- 172.21.15.16:0/278680108 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f87ac0cf880 0x7f87ac0c8a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:27:33.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.975+0000 7f87abfff700 1 --2- 172.21.15.16:0/278680108 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87ac0cf4a0 0x7f87ac0c63f0 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7f8798009c40 tx=0x7f8798004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:33.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.975+0000 7f87a97fa700 1 -- 172.21.15.16:0/278680108 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87980045d0 con 0x7f87ac0cf4a0 2023-10-16T13:27:33.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.975+0000 7f87a97fa700 1 -- 172.21.15.16:0/278680108 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f879801e070 con 0x7f87ac0cf4a0 2023-10-16T13:27:33.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.975+0000 7f87b3264700 1 -- 172.21.15.16:0/278680108 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87ac0c7f90 con 0x7f87ac0cf4a0 2023-10-16T13:27:33.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.975+0000 7f87a97fa700 1 -- 172.21.15.16:0/278680108 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87980038e0 con 0x7f87ac0cf4a0 2023-10-16T13:27:33.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.975+0000 7f87b3264700 1 -- 172.21.15.16:0/278680108 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87ac0c8550 con 0x7f87ac0cf4a0 2023-10-16T13:27:33.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.976+0000 7f87b3264700 1 -- 172.21.15.16:0/278680108 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87ac04ebb0 con 0x7f87ac0cf4a0 2023-10-16T13:27:33.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.976+0000 7f87a97fa700 1 -- 172.21.15.16:0/278680108 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8798003a40 con 0x7f87ac0cf4a0 2023-10-16T13:27:33.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.977+0000 7f87a97fa700 1 --2- 172.21.15.16:0/278680108 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f879006b050 0x7f879006d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:33.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.977+0000 7f87a97fa700 1 -- 172.21.15.16:0/278680108 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 6051+0+0 (secure 0 0 0) 0x7f8798020070 con 0x7f87ac0cf4a0 2023-10-16T13:27:33.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.977+0000 7f87b1000700 1 --2- 172.21.15.16:0/278680108 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f879006b050 0x7f879006d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:33.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.977+0000 7f87b1000700 1 --2- 172.21.15.16:0/278680108 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f879006b050 0x7f879006d510 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f87a0007fd0 tx=0x7f87a0009040 comp rx=0 tx=0).ready entity=mgr.9510 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:33.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:33.981+0000 7f87a97fa700 1 -- 172.21.15.16:0/278680108 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8798055c20 con 0x7f87ac0cf4a0 2023-10-16T13:27:34.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:34.267+0000 7f87b3264700 1 -- 172.21.15.16:0/278680108 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]} v 0) v1 -- 0x7f87ac0cd7f0 con 0x7f87ac0cf4a0 2023-10-16T13:27:35.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.107+0000 7f87a97fa700 1 -- 172.21.15.16:0/278680108 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]}]=0 v2504) v1 ==== 177+0+0 (secure 0 0 0) 0x7f8798015020 con 0x7f87ac0cf4a0 2023-10-16T13:27:35.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.110+0000 7f87b3264700 1 -- 172.21.15.16:0/278680108 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f879006b050 msgr2=0x7f879006d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:35.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.110+0000 7f87b3264700 1 --2- 172.21.15.16:0/278680108 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f879006b050 0x7f879006d510 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f87a0007fd0 tx=0x7f87a0009040 comp rx=0 tx=0).stop 2023-10-16T13:27:35.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.110+0000 7f87b3264700 1 -- 172.21.15.16:0/278680108 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87ac0cf4a0 msgr2=0x7f87ac0c63f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:35.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.110+0000 7f87b3264700 1 --2- 172.21.15.16:0/278680108 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87ac0cf4a0 0x7f87ac0c63f0 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7f8798009c40 tx=0x7f8798004750 comp rx=0 tx=0).stop 2023-10-16T13:27:35.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.110+0000 7f87b3264700 1 -- 172.21.15.16:0/278680108 shutdown_connections 2023-10-16T13:27:35.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.110+0000 7f87b3264700 1 --2- 172.21.15.16:0/278680108 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f879006b050 0x7f879006d510 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:35.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.110+0000 7f87b3264700 1 --2- 172.21.15.16:0/278680108 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f87ac0cf880 0x7f87ac0c8a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:35.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.110+0000 7f87b3264700 1 --2- 172.21.15.16:0/278680108 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f87ac0b5830 0x7f87ac14ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:35.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.110+0000 7f87b3264700 1 --2- 172.21.15.16:0/278680108 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87ac0cf4a0 0x7f87ac0c63f0 unknown :-1 s=CLOSED pgs=1880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:35.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.110+0000 7f87b3264700 1 -- 172.21.15.16:0/278680108 >> 172.21.15.16:0/278680108 conn(0x7f87ac0b1fd0 msgr2=0x7f87ac0b4f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:35.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.110+0000 7f87b3264700 1 -- 172.21.15.16:0/278680108 shutdown_connections 2023-10-16T13:27:35.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.110+0000 7f87b3264700 1 -- 172.21.15.16:0/278680108 wait complete. 2023-10-16T13:27:35.128 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-16T13:27:35.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.264+0000 7fd500c52700 1 Processor -- start 2023-10-16T13:27:35.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.264+0000 7fd500c52700 1 -- start start 2023-10-16T13:27:35.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.265+0000 7fd500c52700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fd4fc0cb6f0 0x7fd4fc0c54b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:35.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.265+0000 7fd500c52700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd4fc15f170 0x7fd4fc0c59f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:35.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.265+0000 7fd500c52700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4fc0cee50 con 0x7fd4fc0cde90 2023-10-16T13:27:35.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.265+0000 7fd500c52700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4fc0c5020 con 0x7fd4fc15f170 2023-10-16T13:27:35.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.265+0000 7fd500c52700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4fc0d2340 con 0x7fd4fc0cb6f0 2023-10-16T13:27:35.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.265+0000 7fd4fad9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd4fc15f170 0x7fd4fc0c59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:35.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.265+0000 7fd4fad9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd4fc15f170 0x7fd4fc0c59f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48250/0 (socket says 172.21.15.16:48250) 2023-10-16T13:27:35.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.265+0000 7fd4fad9d700 1 -- 172.21.15.16:0/2715285001 learned_addr learned my addr 172.21.15.16:0/2715285001 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:35.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.265+0000 7fd4f9d9b700 1 --2- 172.21.15.16:0/2715285001 >> v2:172.21.15.49:3300/0 conn(0x7fd4fc0cb6f0 0x7fd4fc0c54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:35.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.265+0000 7fd4fad9d700 1 -- 172.21.15.16:0/2715285001 >> v2:172.21.15.49:3300/0 conn(0x7fd4fc0cb6f0 msgr2=0x7fd4fc0c54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:35.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.265+0000 7fd4fad9d700 1 --2- 172.21.15.16:0/2715285001 >> v2:172.21.15.49:3300/0 conn(0x7fd4fc0cb6f0 0x7fd4fc0c54b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:35.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.265+0000 7fd4fad9d700 1 -- 172.21.15.16:0/2715285001 >> v1:172.21.15.49:6789/0 conn(0x7fd4fc0cde90 legacy=0x7fd4fc15ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:35.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.265+0000 7fd4fad9d700 1 -- 172.21.15.16:0/2715285001 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4fc0ccdd0 con 0x7fd4fc15f170 2023-10-16T13:27:35.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.265+0000 7fd4f9d9b700 1 --2- 172.21.15.16:0/2715285001 >> v2:172.21.15.49:3300/0 conn(0x7fd4fc0cb6f0 0x7fd4fc0c54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:27:35.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.265+0000 7fd4fad9d700 1 --2- 172.21.15.16:0/2715285001 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4fc15f170 0x7fd4fc0c59f0 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7fd4fc0b2f50 tx=0x7fd4f400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6acc2febdb8d5403 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:35.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.266+0000 7fd4f959a700 1 -- 172.21.15.16:0/2715285001 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4f400e070 con 0x7fd4fc15f170 2023-10-16T13:27:35.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.266+0000 7fd4f959a700 1 -- 172.21.15.16:0/2715285001 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4f400b9c0 con 0x7fd4fc15f170 2023-10-16T13:27:35.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.266+0000 7fd4f959a700 1 -- 172.21.15.16:0/2715285001 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4f400bd70 con 0x7fd4fc15f170 2023-10-16T13:27:35.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.266+0000 7fd500c52700 1 -- 172.21.15.16:0/2715285001 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4fc15f170 msgr2=0x7fd4fc0c59f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:35.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.266+0000 7fd500c52700 1 --2- 172.21.15.16:0/2715285001 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4fc15f170 0x7fd4fc0c59f0 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7fd4fc0b2f50 tx=0x7fd4f400b330 comp rx=0 tx=0).stop 2023-10-16T13:27:35.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.266+0000 7fd500c52700 1 -- 172.21.15.16:0/2715285001 shutdown_connections 2023-10-16T13:27:35.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.266+0000 7fd500c52700 1 --2- 172.21.15.16:0/2715285001 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4fc15f170 0x7fd4fc0c59f0 unknown :-1 s=CLOSED pgs=1881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:35.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.266+0000 7fd500c52700 1 --2- 172.21.15.16:0/2715285001 >> v2:172.21.15.49:3300/0 conn(0x7fd4fc0cb6f0 0x7fd4fc0c54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:35.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.266+0000 7fd500c52700 1 -- 172.21.15.16:0/2715285001 >> 172.21.15.16:0/2715285001 conn(0x7fd4fc0d14d0 msgr2=0x7fd4fc0d18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:35.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.266+0000 7fd500c52700 1 -- 172.21.15.16:0/2715285001 shutdown_connections 2023-10-16T13:27:35.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.266+0000 7fd500c52700 1 -- 172.21.15.16:0/2715285001 wait complete. 2023-10-16T13:27:35.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.268+0000 7fd500c52700 1 Processor -- start 2023-10-16T13:27:35.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.268+0000 7fd500c52700 1 -- start start 2023-10-16T13:27:35.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.268+0000 7fd500c52700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4fc0cb6f0 0x7fd4fc15a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:35.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.268+0000 7fd500c52700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd4fc0cde90 0x7fd4fc0c67f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:35.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.269+0000 7fd500c52700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd4fc15f170 0x7fd4fc0c6d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:35.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.269+0000 7fd500c52700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4fc1f3f50 con 0x7fd4fc0cb6f0 2023-10-16T13:27:35.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.269+0000 7fd500c52700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4fc1f2f80 con 0x7fd4fc0cde90 2023-10-16T13:27:35.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.269+0000 7fd500c52700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4fc1f3ac0 con 0x7fd4fc15f170 2023-10-16T13:27:35.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.269+0000 7fd4fa59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4fc0cb6f0 0x7fd4fc15a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:35.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.269+0000 7fd4fa59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4fc0cb6f0 0x7fd4fc15a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48252/0 (socket says 172.21.15.16:48252) 2023-10-16T13:27:35.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.269+0000 7fd4fa59c700 1 -- 172.21.15.16:0/3121306517 learned_addr learned my addr 172.21.15.16:0/3121306517 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:35.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.269+0000 7fd4fad9d700 1 --2- 172.21.15.16:0/3121306517 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd4fc15f170 0x7fd4fc0c6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:35.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.269+0000 7fd4f9d9b700 1 --2- 172.21.15.16:0/3121306517 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd4fc0cde90 0x7fd4fc0c67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:35.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.269+0000 7fd4fa59c700 1 -- 172.21.15.16:0/3121306517 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd4fc15f170 msgr2=0x7fd4fc0c6d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:35.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.269+0000 7fd4fa59c700 1 --2- 172.21.15.16:0/3121306517 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd4fc15f170 0x7fd4fc0c6d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:35.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.269+0000 7fd4fa59c700 1 -- 172.21.15.16:0/3121306517 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd4fc0cde90 msgr2=0x7fd4fc0c67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:35.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.269+0000 7fd4fa59c700 1 --2- 172.21.15.16:0/3121306517 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd4fc0cde90 0x7fd4fc0c67f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:35.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.269+0000 7fd4fa59c700 1 -- 172.21.15.16:0/3121306517 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4f4009b30 con 0x7fd4fc0cb6f0 2023-10-16T13:27:35.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.269+0000 7fd4f9d9b700 1 --2- 172.21.15.16:0/3121306517 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd4fc0cde90 0x7fd4fc0c67f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:27:35.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.269+0000 7fd4fad9d700 1 --2- 172.21.15.16:0/3121306517 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd4fc15f170 0x7fd4fc0c6d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:27:35.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.269+0000 7fd4fa59c700 1 --2- 172.21.15.16:0/3121306517 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4fc0cb6f0 0x7fd4fc15a7d0 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7fd4ec00c490 tx=0x7fd4ec00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:35.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.269+0000 7fd4eaffd700 1 -- 172.21.15.16:0/3121306517 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4ec0052d0 con 0x7fd4fc0cb6f0 2023-10-16T13:27:35.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.269+0000 7fd4eaffd700 1 -- 172.21.15.16:0/3121306517 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4ec005450 con 0x7fd4fc0cb6f0 2023-10-16T13:27:35.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.269+0000 7fd500c52700 1 -- 172.21.15.16:0/3121306517 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4fc1f3190 con 0x7fd4fc0cb6f0 2023-10-16T13:27:35.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.269+0000 7fd4eaffd700 1 -- 172.21.15.16:0/3121306517 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4ec006aa0 con 0x7fd4fc0cb6f0 2023-10-16T13:27:35.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.269+0000 7fd500c52700 1 -- 172.21.15.16:0/3121306517 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4fc1f3750 con 0x7fd4fc0cb6f0 2023-10-16T13:27:35.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.270+0000 7fd500c52700 1 -- 172.21.15.16:0/3121306517 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4fc04ebb0 con 0x7fd4fc0cb6f0 2023-10-16T13:27:35.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.271+0000 7fd4eaffd700 1 -- 172.21.15.16:0/3121306517 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd4ec003680 con 0x7fd4fc0cb6f0 2023-10-16T13:27:35.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.271+0000 7fd4eaffd700 1 --2- 172.21.15.16:0/3121306517 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7fd4dc06b020 0x7fd4dc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:35.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.271+0000 7fd4eaffd700 1 -- 172.21.15.16:0/3121306517 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 6051+0+0 (secure 0 0 0) 0x7fd4ec088b70 con 0x7fd4fc0cb6f0 2023-10-16T13:27:35.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.271+0000 7fd4f9d9b700 1 --2- 172.21.15.16:0/3121306517 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7fd4dc06b020 0x7fd4dc06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:35.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.272+0000 7fd4f9d9b700 1 --2- 172.21.15.16:0/3121306517 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7fd4dc06b020 0x7fd4dc06d4e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fd4fc0d00b0 tx=0x7fd4f0008040 comp rx=0 tx=0).ready entity=mgr.9510 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:35.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.276+0000 7fd4eaffd700 1 -- 172.21.15.16:0/3121306517 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd4ec0539b0 con 0x7fd4fc0cb6f0 2023-10-16T13:27:35.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.479+0000 7fd500c52700 1 -- 172.21.15.16:0/3121306517 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fd4fc0ce510 con 0x7fd4fc0cb6f0 2023-10-16T13:27:35.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.482+0000 7fd4eaffd700 1 -- 172.21.15.16:0/3121306517 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v58)=0 v58) v1 ==== 122+0+0 (secure 0 0 0) 0x7fd4ec056fd0 con 0x7fd4fc0cb6f0 2023-10-16T13:27:35.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.484+0000 7fd500c52700 1 -- 172.21.15.16:0/3121306517 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7fd4dc06b020 msgr2=0x7fd4dc06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:35.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.484+0000 7fd500c52700 1 --2- 172.21.15.16:0/3121306517 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7fd4dc06b020 0x7fd4dc06d4e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fd4fc0d00b0 tx=0x7fd4f0008040 comp rx=0 tx=0).stop 2023-10-16T13:27:35.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.484+0000 7fd500c52700 1 -- 172.21.15.16:0/3121306517 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4fc0cb6f0 msgr2=0x7fd4fc15a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:35.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.484+0000 7fd500c52700 1 --2- 172.21.15.16:0/3121306517 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4fc0cb6f0 0x7fd4fc15a7d0 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7fd4ec00c490 tx=0x7fd4ec00c850 comp rx=0 tx=0).stop 2023-10-16T13:27:35.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.484+0000 7fd500c52700 1 -- 172.21.15.16:0/3121306517 shutdown_connections 2023-10-16T13:27:35.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.484+0000 7fd500c52700 1 --2- 172.21.15.16:0/3121306517 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd4fc15f170 0x7fd4fc0c6d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:35.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.484+0000 7fd500c52700 1 --2- 172.21.15.16:0/3121306517 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4fc0cb6f0 0x7fd4fc15a7d0 unknown :-1 s=CLOSED pgs=1882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:35.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.484+0000 7fd500c52700 1 --2- 172.21.15.16:0/3121306517 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7fd4dc06b020 0x7fd4dc06d4e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:35.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.484+0000 7fd500c52700 1 --2- 172.21.15.16:0/3121306517 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd4fc0cde90 0x7fd4fc0c67f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:35.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.484+0000 7fd500c52700 1 -- 172.21.15.16:0/3121306517 >> 172.21.15.16:0/3121306517 conn(0x7fd4fc0d14d0 msgr2=0x7fd4fc15ea40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:35.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.484+0000 7fd500c52700 1 -- 172.21.15.16:0/3121306517 shutdown_connections 2023-10-16T13:27:35.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.485+0000 7fd500c52700 1 -- 172.21.15.16:0/3121306517 wait complete. 2023-10-16T13:27:35.499 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T13:27:35.500 DEBUG:teuthology.orchestra.run.smithi016:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T13:27:35.540 DEBUG:teuthology.orchestra.run.smithi049:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T13:27:35.592 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T13:27:35.593 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T13:27:35.593 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:27:35.611 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16287364k,nr_inodes=4071841,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20251 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3261172k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T13:27:35.611 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T13:27:35.612 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:27:35.633 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284204k,nr_inodes=4071051,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=753 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260540k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T13:27:35.633 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T13:27:35.633 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:27:35.668 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16287364k,nr_inodes=4071841,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20251 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3261172k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T13:27:35.668 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T13:27:35.669 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:27:35.693 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284204k,nr_inodes=4071051,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=753 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260540k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T13:27:35.694 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-16T13:27:35.694 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:27:35.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.830+0000 7f456f255700 1 Processor -- start 2023-10-16T13:27:35.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.830+0000 7f456f255700 1 -- start start 2023-10-16T13:27:35.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.831+0000 7f456f255700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f45680ce640 0x7f45680cd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:35.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.831+0000 7f456f255700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45680cf4f0 con 0x7f45680b50c0 2023-10-16T13:27:35.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.831+0000 7f456f255700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f456815f570 con 0x7f45680b3270 2023-10-16T13:27:35.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.831+0000 7f456f255700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45680cef50 con 0x7f45680ce640 2023-10-16T13:27:35.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.831+0000 7f456cff1700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f45680b50c0 0x7f45680cd1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37534/0 (socket says 172.21.15.16:37534) 2023-10-16T13:27:35.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.831+0000 7f456cff1700 1 -- 172.21.15.16:0/3178969187 learned_addr learned my addr 172.21.15.16:0/3178969187 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:27:35.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.831+0000 7f45677fe700 1 -- 172.21.15.16:0/3178969187 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4117312966 0 0) 0x7f45680cf4f0 con 0x7f45680b50c0 2023-10-16T13:27:35.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.832+0000 7f45677fe700 1 -- 172.21.15.16:0/3178969187 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45580035b0 con 0x7f45680b50c0 2023-10-16T13:27:35.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.832+0000 7f4567fff700 1 --2- 172.21.15.16:0/3178969187 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f45680ce640 0x7f45680cd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:35.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.832+0000 7f45677fe700 1 -- 172.21.15.16:0/3178969187 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2018196964 0 0) 0x7f45580035b0 con 0x7f45680b50c0 2023-10-16T13:27:35.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.832+0000 7f45677fe700 1 -- 172.21.15.16:0/3178969187 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f45680ce640 msgr2=0x7f45680cd8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:35.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.832+0000 7f45677fe700 1 --2- 172.21.15.16:0/3178969187 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f45680ce640 0x7f45680cd8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:35.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.832+0000 7f45677fe700 1 -- 172.21.15.16:0/3178969187 >> v1:172.21.15.49:6789/0 conn(0x7f45680b3270 legacy=0x7f45680b3720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:35.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.832+0000 7f45677fe700 1 -- 172.21.15.16:0/3178969187 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45680b54a0 con 0x7f45680b50c0 2023-10-16T13:27:35.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.832+0000 7f45677fe700 1 -- 172.21.15.16:0/3178969187 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f455c0017f0 con 0x7f45680b50c0 2023-10-16T13:27:35.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.832+0000 7f45677fe700 1 -- 172.21.15.16:0/3178969187 >> v1:172.21.15.16:6789/0 conn(0x7f45680b50c0 legacy=0x7f45680cd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:35.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.832+0000 7f45677fe700 1 --2- 172.21.15.16:0/3178969187 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f45680ce640 0x7f4558003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:35.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.832+0000 7f45677fe700 1 --2- 172.21.15.16:0/3178969187 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4558003fb0 0x7f4558004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:35.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.832+0000 7f45677fe700 1 --2- 172.21.15.16:0/3178969187 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4558004d90 0x7f4558005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:35.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.832+0000 7f45677fe700 1 -- 172.21.15.16:0/3178969187 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45680cef50 con 0x7f4558004d90 2023-10-16T13:27:35.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.832+0000 7f45677fe700 1 -- 172.21.15.16:0/3178969187 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f455800bba0 con 0x7f4558003fb0 2023-10-16T13:27:35.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.832+0000 7f45677fe700 1 -- 172.21.15.16:0/3178969187 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f455800bd60 con 0x7f45680ce640 2023-10-16T13:27:35.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.832+0000 7f456d7f2700 1 --2- 172.21.15.16:0/3178969187 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4558004d90 0x7f4558005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:35.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.832+0000 7f456cff1700 1 --2- 172.21.15.16:0/3178969187 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f45680ce640 0x7f4558003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:35.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.832+0000 7f4567fff700 1 --2- 172.21.15.16:0/3178969187 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4558003fb0 0x7f4558004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:35.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.832+0000 7f456d7f2700 1 -- 172.21.15.16:0/3178969187 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f45680ce640 msgr2=0x7f4558003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:35.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.832+0000 7f456d7f2700 1 --2- 172.21.15.16:0/3178969187 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f45680ce640 0x7f4558003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:35.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.832+0000 7f456d7f2700 1 -- 172.21.15.16:0/3178969187 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4558003fb0 msgr2=0x7f4558004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:35.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.832+0000 7f456d7f2700 1 --2- 172.21.15.16:0/3178969187 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4558003fb0 0x7f4558004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:35.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.832+0000 7f456d7f2700 1 -- 172.21.15.16:0/3178969187 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45580035b0 con 0x7f4558004d90 2023-10-16T13:27:35.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.832+0000 7f456d7f2700 1 --2- 172.21.15.16:0/3178969187 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4558004d90 0x7f4558005210 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7f456000c190 tx=0x7f4558005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7eb6952a0f5bb6f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:35.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.833+0000 7f45677fe700 1 -- 172.21.15.16:0/3178969187 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4560001db0 con 0x7f4558004d90 2023-10-16T13:27:35.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.833+0000 7f45677fe700 1 -- 172.21.15.16:0/3178969187 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f456000ccd0 con 0x7f4558004d90 2023-10-16T13:27:35.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.833+0000 7f45677fe700 1 -- 172.21.15.16:0/3178969187 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4560003c40 con 0x7f4558004d90 2023-10-16T13:27:35.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.833+0000 7f456f255700 1 -- 172.21.15.16:0/3178969187 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4558004d90 msgr2=0x7f4558005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:35.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.833+0000 7f456f255700 1 --2- 172.21.15.16:0/3178969187 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4558004d90 0x7f4558005210 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7f456000c190 tx=0x7f4558005880 comp rx=0 tx=0).stop 2023-10-16T13:27:35.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.833+0000 7f456d7f2700 1 -- 172.21.15.16:0/3178969187 reap_dead start 2023-10-16T13:27:35.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.833+0000 7f456f255700 1 -- 172.21.15.16:0/3178969187 shutdown_connections 2023-10-16T13:27:35.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.833+0000 7f456f255700 1 -- 172.21.15.16:0/3178969187 >> 172.21.15.16:0/3178969187 conn(0x7f45680cc120 msgr2=0x7f45680cc530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:35.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.833+0000 7f456f255700 1 -- 172.21.15.16:0/3178969187 shutdown_connections 2023-10-16T13:27:35.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.833+0000 7f456f255700 1 -- 172.21.15.16:0/3178969187 wait complete. 2023-10-16T13:27:35.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.834+0000 7f456f255700 1 Processor -- start 2023-10-16T13:27:35.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.834+0000 7f456f255700 1 -- start start 2023-10-16T13:27:35.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.835+0000 7f456f255700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4558004d90 0x7f45680cb260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:35.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.835+0000 7f456f255700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f45681e5de0 0x7f4568157b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:35.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.835+0000 7f456f255700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f45680af2c0 0x7f456815dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:35.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.835+0000 7f456f255700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4568158070 con 0x7f4558004d90 2023-10-16T13:27:35.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.835+0000 7f456f255700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45681e6260 con 0x7f45680af2c0 2023-10-16T13:27:35.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.835+0000 7f456f255700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45680cb8b0 con 0x7f45681e5de0 2023-10-16T13:27:35.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.835+0000 7f456cff1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4558004d90 0x7f45680cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:35.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.835+0000 7f456cff1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4558004d90 0x7f45680cb260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48272/0 (socket says 172.21.15.16:48272) 2023-10-16T13:27:35.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.835+0000 7f456cff1700 1 -- 172.21.15.16:0/2100449163 learned_addr learned my addr 172.21.15.16:0/2100449163 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:35.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.836+0000 7f456d7f2700 1 --2- 172.21.15.16:0/2100449163 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f45680af2c0 0x7f456815dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:35.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.836+0000 7f4567fff700 1 --2- 172.21.15.16:0/2100449163 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f45681e5de0 0x7f4568157b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:35.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.836+0000 7f456cff1700 1 -- 172.21.15.16:0/2100449163 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f45681e5de0 msgr2=0x7f4568157b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:35.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.836+0000 7f456cff1700 1 --2- 172.21.15.16:0/2100449163 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f45681e5de0 0x7f4568157b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:35.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.836+0000 7f456cff1700 1 -- 172.21.15.16:0/2100449163 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f45680af2c0 msgr2=0x7f456815dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:35.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.836+0000 7f456cff1700 1 --2- 172.21.15.16:0/2100449163 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f45680af2c0 0x7f456815dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:35.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.836+0000 7f456cff1700 1 -- 172.21.15.16:0/2100449163 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45680cbb90 con 0x7f4558004d90 2023-10-16T13:27:35.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.836+0000 7f4567fff700 1 --2- 172.21.15.16:0/2100449163 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f45681e5de0 0x7f4568157b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:27:35.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.836+0000 7f456d7f2700 1 --2- 172.21.15.16:0/2100449163 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f45680af2c0 0x7f456815dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:27:35.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.836+0000 7f456cff1700 1 --2- 172.21.15.16:0/2100449163 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4558004d90 0x7f45680cb260 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7f455c007fd0 tx=0x7f455c00fe70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:35.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.836+0000 7f45657fa700 1 -- 172.21.15.16:0/2100449163 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f455c0072e0 con 0x7f4558004d90 2023-10-16T13:27:35.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.836+0000 7f45657fa700 1 -- 172.21.15.16:0/2100449163 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f455c007440 con 0x7f4558004d90 2023-10-16T13:27:35.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.836+0000 7f45657fa700 1 -- 172.21.15.16:0/2100449163 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f455c005830 con 0x7f4558004d90 2023-10-16T13:27:35.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.836+0000 7f456f255700 1 -- 172.21.15.16:0/2100449163 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45681e55e0 con 0x7f4558004d90 2023-10-16T13:27:35.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.836+0000 7f456f255700 1 -- 172.21.15.16:0/2100449163 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45680af740 con 0x7f4558004d90 2023-10-16T13:27:35.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.837+0000 7f456f255700 1 -- 172.21.15.16:0/2100449163 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f456804ebb0 con 0x7f4558004d90 2023-10-16T13:27:35.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.838+0000 7f45657fa700 1 -- 172.21.15.16:0/2100449163 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f455c005990 con 0x7f4558004d90 2023-10-16T13:27:35.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.838+0000 7f45657fa700 1 --2- 172.21.15.16:0/2100449163 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f454c06b020 0x7f454c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:35.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.838+0000 7f45657fa700 1 -- 172.21.15.16:0/2100449163 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 6051+0+0 (secure 0 0 0) 0x7f455c0899f0 con 0x7f4558004d90 2023-10-16T13:27:35.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.838+0000 7f4567fff700 1 --2- 172.21.15.16:0/2100449163 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f454c06b020 0x7f454c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:35.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.839+0000 7f4567fff700 1 --2- 172.21.15.16:0/2100449163 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f454c06b020 0x7f454c06d4e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f45680ad690 tx=0x7f4554008040 comp rx=0 tx=0).ready entity=mgr.9510 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:35.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:35.843+0000 7f45657fa700 1 -- 172.21.15.16:0/2100449163 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f455c054830 con 0x7f4558004d90 2023-10-16T13:27:36.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:36.093+0000 7f456f255700 1 -- 172.21.15.16:0/2100449163 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f45681e63a0 con 0x7f4558004d90 2023-10-16T13:27:36.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:36.093+0000 7f45657fa700 1 -- 172.21.15.16:0/2100449163 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+3699 (secure 0 0 0) 0x7f455c057e50 con 0x7f4558004d90 2023-10-16T13:27:36.096 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:27:36.097 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":80,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7631,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/3370728164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":3370728164},{"type":"v1","addr":"172.21.15.49:6834","nonce":3370728164}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":73},{"gid":8638,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2297042963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":2297042963},{"type":"v1","addr":"172.21.15.16:6837","nonce":2297042963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":73},{"gid":8647,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/1920044955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":1920044955},{"type":"v1","addr":"172.21.15.49:6835","nonce":1920044955}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":73},{"gid":9660,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/1635767122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1635767122},{"type":"v1","addr":"172.21.15.16:6836","nonce":1635767122}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80}],"filesystems":[]} 2023-10-16T13:27:36.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:36.095+0000 7f456f255700 1 -- 172.21.15.16:0/2100449163 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f454c06b020 msgr2=0x7f454c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:36.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:36.095+0000 7f456f255700 1 --2- 172.21.15.16:0/2100449163 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f454c06b020 0x7f454c06d4e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f45680ad690 tx=0x7f4554008040 comp rx=0 tx=0).stop 2023-10-16T13:27:36.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:36.095+0000 7f456f255700 1 -- 172.21.15.16:0/2100449163 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4558004d90 msgr2=0x7f45680cb260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:36.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:36.095+0000 7f456f255700 1 --2- 172.21.15.16:0/2100449163 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4558004d90 0x7f45680cb260 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7f455c007fd0 tx=0x7f455c00fe70 comp rx=0 tx=0).stop 2023-10-16T13:27:36.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:36.096+0000 7f456f255700 1 -- 172.21.15.16:0/2100449163 shutdown_connections 2023-10-16T13:27:36.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:36.096+0000 7f456f255700 1 --2- 172.21.15.16:0/2100449163 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f454c06b020 0x7f454c06d4e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:36.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:36.096+0000 7f456f255700 1 --2- 172.21.15.16:0/2100449163 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f45680af2c0 0x7f456815dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:36.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:36.096+0000 7f456f255700 1 --2- 172.21.15.16:0/2100449163 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4558004d90 0x7f45680cb260 unknown :-1 s=CLOSED pgs=1885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:36.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:36.096+0000 7f456f255700 1 --2- 172.21.15.16:0/2100449163 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f45681e5de0 0x7f4568157b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:36.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:36.096+0000 7f456f255700 1 -- 172.21.15.16:0/2100449163 >> 172.21.15.16:0/2100449163 conn(0x7f45680cc120 msgr2=0x7f45680c8f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:36.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:36.096+0000 7f456f255700 1 -- 172.21.15.16:0/2100449163 shutdown_connections 2023-10-16T13:27:36.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:36.096+0000 7f456f255700 1 -- 172.21.15.16:0/2100449163 wait complete. 2023-10-16T13:27:36.101 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 80 2023-10-16T13:27:36.111 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-16T13:27:36.112 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-16T13:27:36.112 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-16T13:27:36.112 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T13:27:36.113 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-16T13:27:36.113 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-16T13:27:36.113 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-16T13:27:36.113 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T13:27:36.114 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-16T13:27:36.114 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-16T13:27:36.114 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-16T13:27:36.114 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T13:27:36.115 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-16T13:27:36.115 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-16T13:27:36.115 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-16T13:27:36.116 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T13:27:36.121 INFO:tasks.ceph.mds.a:Stopped 2023-10-16T13:27:36.121 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-16T13:27:36.122 INFO:tasks.ceph.mds.c:Stopped 2023-10-16T13:27:36.123 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-16T13:27:36.126 INFO:tasks.ceph.mds.b:Stopped 2023-10-16T13:27:36.127 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-16T13:27:36.128 INFO:tasks.ceph.mds.d:Stopped 2023-10-16T13:27:36.128 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-16T13:27:36.164 INFO:tasks.ceph.mds.a:Started 2023-10-16T13:27:36.164 DEBUG:teuthology.parallel:result is None 2023-10-16T13:27:36.165 INFO:tasks.ceph.mds.c:Started 2023-10-16T13:27:36.165 DEBUG:teuthology.parallel:result is None 2023-10-16T13:27:36.169 INFO:tasks.ceph.mds.b:Started 2023-10-16T13:27:36.170 DEBUG:teuthology.parallel:result is None 2023-10-16T13:27:36.170 INFO:tasks.ceph.mds.d:Started 2023-10-16T13:27:36.171 DEBUG:teuthology.parallel:result is None 2023-10-16T13:27:36.171 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-16T13:27:36.247 INFO:tasks.ceph.mds.a.smithi016.stdout:starting mds.a at 2023-10-16T13:27:36.247 INFO:tasks.ceph.mds.c.smithi016.stdout:starting mds.c at 2023-10-16T13:27:36.261 INFO:tasks.ceph.mds.d.smithi049.stdout:starting mds.d at 2023-10-16T13:27:36.261 INFO:tasks.ceph.mds.b.smithi049.stdout:starting mds.b at 2023-10-16T13:27:36.262 INFO:tasks.ceph.mds.b.smithi049.stderr:2023-10-16T13:27:36.260+0000 7f2da4b0f700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6832/0: (98) Address already in use 2023-10-16T13:27:36.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:36.302+0000 7f19d2129700 1 Processor -- start 2023-10-16T13:27:36.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:36.302+0000 7f19d2129700 1 -- start start 2023-10-16T13:27:36.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:36.303+0000 7f19d2129700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f19cc15b630 0x7f19cc0d0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:36.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:36.303+0000 7f19d2129700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f19cc15ba10 0x7f19cc0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:36.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:36.303+0000 7f19d2129700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19cc0b0970 con 0x7f19cc0cdde0 2023-10-16T13:27:36.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:36.303+0000 7f19d2129700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19cc0cd690 con 0x7f19cc15ba10 2023-10-16T13:27:36.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:36.303+0000 7f19d2129700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19cc0cd970 con 0x7f19cc15b630 2023-10-16T13:27:36.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:36.303+0000 7f19caffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f19cc15ba10 0x7f19cc0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:36.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:36.303+0000 7f19caffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f19cc15ba10 0x7f19cc0cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48304/0 (socket says 172.21.15.16:48304) 2023-10-16T13:27:36.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:36.303+0000 7f19caffd700 1 -- 172.21.15.16:0/416515153 learned_addr learned my addr 172.21.15.16:0/416515153 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:36.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:36.303+0000 7f19cb7fe700 1 --2- 172.21.15.16:0/416515153 >> v2:172.21.15.49:3300/0 conn(0x7f19cc15b630 0x7f19cc0d0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:36.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:36.303+0000 7f19caffd700 1 -- 172.21.15.16:0/416515153 >> v2:172.21.15.49:3300/0 conn(0x7f19cc15b630 msgr2=0x7f19cc0d0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:36.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:36.303+0000 7f19caffd700 1 --2- 172.21.15.16:0/416515153 >> v2:172.21.15.49:3300/0 conn(0x7f19cc15b630 0x7f19cc0d0380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:36.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:36.303+0000 7f19caffd700 1 -- 172.21.15.16:0/416515153 >> v1:172.21.15.49:6789/0 conn(0x7f19cc0cdde0 legacy=0x7f19cc0ce1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:36.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:36.303+0000 7f19caffd700 1 -- 172.21.15.16:0/416515153 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19cc0b0c30 con 0x7f19cc15ba10 2023-10-16T13:27:36.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:36.303+0000 7f19caffd700 1 --2- 172.21.15.16:0/416515153 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19cc15ba10 0x7f19cc0cf290 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7f19cc067350 tx=0x7f19c0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a4e624976f7d8b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:36.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:36.303+0000 7f19ca7fc700 1 -- 172.21.15.16:0/416515153 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19c0011830 con 0x7f19cc15ba10 2023-10-16T13:27:36.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:36.303+0000 7f19ca7fc700 1 -- 172.21.15.16:0/416515153 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19c0011990 con 0x7f19cc15ba10 2023-10-16T13:27:36.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:36.303+0000 7f19ca7fc700 1 -- 172.21.15.16:0/416515153 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19c0011d20 con 0x7f19cc15ba10 2023-10-16T13:27:36.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:36.304+0000 7f19d2129700 1 -- 172.21.15.16:0/416515153 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19cc15ba10 msgr2=0x7f19cc0cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:36.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:36.304+0000 7f19d2129700 1 --2- 172.21.15.16:0/416515153 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19cc15ba10 0x7f19cc0cf290 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7f19cc067350 tx=0x7f19c0009bd0 comp rx=0 tx=0).stop 2023-10-16T13:27:36.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:36.304+0000 7f19d2129700 1 -- 172.21.15.16:0/416515153 shutdown_connections 2023-10-16T13:27:36.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:36.304+0000 7f19d2129700 1 --2- 172.21.15.16:0/416515153 >> v2:172.21.15.49:3300/0 conn(0x7f19cc15b630 0x7f19cc0d0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:36.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:36.304+0000 7f19d2129700 1 --2- 172.21.15.16:0/416515153 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19cc15ba10 0x7f19cc0cf290 unknown :-1 s=CLOSED pgs=1894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:36.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:36.304+0000 7f19d2129700 1 -- 172.21.15.16:0/416515153 >> 172.21.15.16:0/416515153 conn(0x7f19cc0d14b0 msgr2=0x7f19cc0d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:36.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:36.304+0000 7f19d2129700 1 -- 172.21.15.16:0/416515153 shutdown_connections 2023-10-16T13:27:36.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:36.304+0000 7f19d2129700 1 -- 172.21.15.16:0/416515153 wait complete. 2023-10-16T13:27:36.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:36.305+0000 7f19d2129700 1 Processor -- start 2023-10-16T13:27:36.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:36.305+0000 7f19d2129700 1 -- start start 2023-10-16T13:27:36.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:36.306+0000 7f19d2129700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f19cc0cdde0 0x7f19cc0b6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:36.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:36.306+0000 7f19d2129700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f19cc15b630 0x7f19cc1f6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:36.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:36.306+0000 7f19d2129700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19cc15ba10 0x7f19cc1f6780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:36.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:36.306+0000 7f19d2129700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19cc0c7f60 con 0x7f19cc15ba10 2023-10-16T13:27:36.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:36.306+0000 7f19d2129700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19cc0c7350 con 0x7f19cc15b630 2023-10-16T13:27:36.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:36.306+0000 7f19d2129700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19cc0c7630 con 0x7f19cc0cdde0 2023-10-16T13:27:36.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:36.306+0000 7f19cbfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19cc15ba10 0x7f19cc1f6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:36.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:36.306+0000 7f19cbfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19cc15ba10 0x7f19cc1f6780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48312/0 (socket says 172.21.15.16:48312) 2023-10-16T13:27:36.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:36.306+0000 7f19cbfff700 1 -- 172.21.15.16:0/727100422 learned_addr learned my addr 172.21.15.16:0/727100422 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:36.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:36.306+0000 7f19cb7fe700 1 --2- 172.21.15.16:0/727100422 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f19cc0cdde0 0x7f19cc0b6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:36.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:36.306+0000 7f19caffd700 1 --2- 172.21.15.16:0/727100422 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f19cc15b630 0x7f19cc1f6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:36.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:36.307+0000 7f19cbfff700 1 -- 172.21.15.16:0/727100422 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f19cc0cdde0 msgr2=0x7f19cc0b6eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:36.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:36.307+0000 7f19cbfff700 1 --2- 172.21.15.16:0/727100422 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f19cc0cdde0 0x7f19cc0b6eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:36.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:36.307+0000 7f19cbfff700 1 -- 172.21.15.16:0/727100422 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f19cc15b630 msgr2=0x7f19cc1f6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:36.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:36.307+0000 7f19cbfff700 1 --2- 172.21.15.16:0/727100422 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f19cc15b630 0x7f19cc1f6240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:36.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:36.307+0000 7f19cbfff700 1 -- 172.21.15.16:0/727100422 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19c0009550 con 0x7f19cc15ba10 2023-10-16T13:27:36.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:36.307+0000 7f19caffd700 1 --2- 172.21.15.16:0/727100422 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f19cc15b630 0x7f19cc1f6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:27:36.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:36.307+0000 7f19cbfff700 1 --2- 172.21.15.16:0/727100422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19cc15ba10 0x7f19cc1f6780 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7f19c400cc50 tx=0x7f19c400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:36.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:36.307+0000 7f19b3fff700 1 -- 172.21.15.16:0/727100422 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19c4003c00 con 0x7f19cc15ba10 2023-10-16T13:27:36.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:36.307+0000 7f19b3fff700 1 -- 172.21.15.16:0/727100422 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19c4003d60 con 0x7f19cc15ba10 2023-10-16T13:27:36.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:36.307+0000 7f19d2129700 1 -- 172.21.15.16:0/727100422 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19cc0c7910 con 0x7f19cc15ba10 2023-10-16T13:27:36.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:36.307+0000 7f19b3fff700 1 -- 172.21.15.16:0/727100422 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19c400ab60 con 0x7f19cc15ba10 2023-10-16T13:27:36.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:36.307+0000 7f19d2129700 1 -- 172.21.15.16:0/727100422 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19cc15c4f0 con 0x7f19cc15ba10 2023-10-16T13:27:36.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:36.308+0000 7f19d2129700 1 -- 172.21.15.16:0/727100422 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19cc06b8f0 con 0x7f19cc15ba10 2023-10-16T13:27:36.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:36.308+0000 7f19b3fff700 1 -- 172.21.15.16:0/727100422 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f19c400acc0 con 0x7f19cc15ba10 2023-10-16T13:27:36.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:36.309+0000 7f19b3fff700 1 --2- 172.21.15.16:0/727100422 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f19b806b020 0x7f19b806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:36.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:36.309+0000 7f19b3fff700 1 -- 172.21.15.16:0/727100422 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 6051+0+0 (secure 0 0 0) 0x7f19c4089920 con 0x7f19cc15ba10 2023-10-16T13:27:36.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:36.309+0000 7f19cb7fe700 1 --2- 172.21.15.16:0/727100422 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f19b806b020 0x7f19b806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:36.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:36.309+0000 7f19cb7fe700 1 --2- 172.21.15.16:0/727100422 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f19b806b020 0x7f19b806d4e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f19bc009be0 tx=0x7f19bc009210 comp rx=0 tx=0).ready entity=mgr.9510 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:36.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:36.313+0000 7f19b3fff700 1 -- 172.21.15.16:0/727100422 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f19cc06b8f0 con 0x7f19cc15ba10 2023-10-16T13:27:36.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:36.531+0000 7f19d2129700 1 -- 172.21.15.16:0/727100422 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f19cc06b8f0 con 0x7f19cc15ba10 2023-10-16T13:27:38.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.088+0000 7f19b3fff700 1 -- 172.21.15.16:0/727100422 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v173) v1 ==== 97+0+0 (secure 0 0 0) 0x7f19cc06b8f0 con 0x7f19cc15ba10 2023-10-16T13:27:38.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.092+0000 7f19d2129700 1 -- 172.21.15.16:0/727100422 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f19b806b020 msgr2=0x7f19b806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:38.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.092+0000 7f19d2129700 1 --2- 172.21.15.16:0/727100422 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f19b806b020 0x7f19b806d4e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f19bc009be0 tx=0x7f19bc009210 comp rx=0 tx=0).stop 2023-10-16T13:27:38.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.092+0000 7f19d2129700 1 -- 172.21.15.16:0/727100422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19cc15ba10 msgr2=0x7f19cc1f6780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:38.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.092+0000 7f19d2129700 1 --2- 172.21.15.16:0/727100422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19cc15ba10 0x7f19cc1f6780 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7f19c400cc50 tx=0x7f19c400cf60 comp rx=0 tx=0).stop 2023-10-16T13:27:38.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.092+0000 7f19d2129700 1 -- 172.21.15.16:0/727100422 shutdown_connections 2023-10-16T13:27:38.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.092+0000 7f19d2129700 1 --2- 172.21.15.16:0/727100422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19cc15ba10 0x7f19cc1f6780 unknown :-1 s=CLOSED pgs=1895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:38.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.092+0000 7f19d2129700 1 --2- 172.21.15.16:0/727100422 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f19cc0cdde0 0x7f19cc0b6eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:38.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.092+0000 7f19d2129700 1 --2- 172.21.15.16:0/727100422 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f19b806b020 0x7f19b806d4e0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:38.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.092+0000 7f19d2129700 1 --2- 172.21.15.16:0/727100422 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f19cc15b630 0x7f19cc1f6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:38.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.092+0000 7f19d2129700 1 -- 172.21.15.16:0/727100422 >> 172.21.15.16:0/727100422 conn(0x7f19cc0d14b0 msgr2=0x7f19cc0b2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:38.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.092+0000 7f19d2129700 1 -- 172.21.15.16:0/727100422 shutdown_connections 2023-10-16T13:27:38.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.092+0000 7f19d2129700 1 -- 172.21.15.16:0/727100422 wait complete. 2023-10-16T13:27:38.097 INFO:teuthology.orchestra.run.smithi016.stderr: removed all blocklist entries 2023-10-16T13:27:38.110 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-16T13:27:38.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.240+0000 7f498babc700 1 Processor -- start 2023-10-16T13:27:38.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.240+0000 7f498babc700 1 -- start start 2023-10-16T13:27:38.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.241+0000 7f498babc700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f498415b280 0x7f49841613e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:38.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.241+0000 7f498babc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f49840cc8a0 0x7f498415e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:38.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.241+0000 7f498babc700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4984161c30 con 0x7f498406a7e0 2023-10-16T13:27:38.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.241+0000 7f498babc700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4984161f10 con 0x7f49840cc8a0 2023-10-16T13:27:38.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.241+0000 7f498babc700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49841621f0 con 0x7f498415b280 2023-10-16T13:27:38.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.241+0000 7f4989057700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f49840cc8a0 0x7f498415e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:38.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.241+0000 7f498a059700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f498406a7e0 0x7f49840d02b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37598/0 (socket says 172.21.15.16:37598) 2023-10-16T13:27:38.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.241+0000 7f498a059700 1 -- 172.21.15.16:0/3149760614 learned_addr learned my addr 172.21.15.16:0/3149760614 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:27:38.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.241+0000 7f4989858700 1 --2- 172.21.15.16:0/3149760614 >> v2:172.21.15.49:3300/0 conn(0x7f498415b280 0x7f49841613e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:38.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.241+0000 7f4989057700 1 -- 172.21.15.16:0/3149760614 >> v2:172.21.15.49:3300/0 conn(0x7f498415b280 msgr2=0x7f49841613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:38.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.241+0000 7f4989057700 1 --2- 172.21.15.16:0/3149760614 >> v2:172.21.15.49:3300/0 conn(0x7f498415b280 0x7f49841613e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:38.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.241+0000 7f4989057700 1 -- 172.21.15.16:0/3149760614 >> v1:172.21.15.16:6789/0 conn(0x7f498406a7e0 legacy=0x7f49840d02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:38.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.241+0000 7f4989057700 1 -- 172.21.15.16:0/3149760614 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f498415df00 con 0x7f49840cc8a0 2023-10-16T13:27:38.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.241+0000 7f4989057700 1 --2- 172.21.15.16:0/3149760614 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49840cc8a0 0x7f498415e700 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7f498406b8a0 tx=0x7f4978009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d964492b287cdf7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:38.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.241+0000 7f4988856700 1 -- 172.21.15.16:0/3149760614 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4978011830 con 0x7f49840cc8a0 2023-10-16T13:27:38.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.241+0000 7f4988856700 1 -- 172.21.15.16:0/3149760614 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4978011990 con 0x7f49840cc8a0 2023-10-16T13:27:38.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.241+0000 7f4988856700 1 -- 172.21.15.16:0/3149760614 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4978011d20 con 0x7f49840cc8a0 2023-10-16T13:27:38.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.242+0000 7f498babc700 1 -- 172.21.15.16:0/3149760614 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49840cc8a0 msgr2=0x7f498415e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:38.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.242+0000 7f498babc700 1 --2- 172.21.15.16:0/3149760614 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49840cc8a0 0x7f498415e700 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7f498406b8a0 tx=0x7f4978009b20 comp rx=0 tx=0).stop 2023-10-16T13:27:38.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.242+0000 7f498babc700 1 -- 172.21.15.16:0/3149760614 shutdown_connections 2023-10-16T13:27:38.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.242+0000 7f498babc700 1 --2- 172.21.15.16:0/3149760614 >> v2:172.21.15.49:3300/0 conn(0x7f498415b280 0x7f49841613e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:38.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.242+0000 7f498babc700 1 --2- 172.21.15.16:0/3149760614 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49840cc8a0 0x7f498415e700 unknown :-1 s=CLOSED pgs=1897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:38.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.242+0000 7f498babc700 1 -- 172.21.15.16:0/3149760614 >> 172.21.15.16:0/3149760614 conn(0x7f49840b1fd0 msgr2=0x7f49840b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:38.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.242+0000 7f498babc700 1 -- 172.21.15.16:0/3149760614 shutdown_connections 2023-10-16T13:27:38.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.242+0000 7f498babc700 1 -- 172.21.15.16:0/3149760614 wait complete. 2023-10-16T13:27:38.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.243+0000 7f498babc700 1 Processor -- start 2023-10-16T13:27:38.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.243+0000 7f498babc700 1 -- start start 2023-10-16T13:27:38.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.244+0000 7f498babc700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f498406a7e0 0x7f49840c3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:38.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.244+0000 7f498babc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49840cc8a0 0x7f49841f2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:38.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.244+0000 7f498babc700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f498415b280 0x7f49840ad2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:38.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.244+0000 7f498babc700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49841561e0 con 0x7f49840cc8a0 2023-10-16T13:27:38.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.244+0000 7f498babc700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49840b07a0 con 0x7f498415b280 2023-10-16T13:27:38.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.244+0000 7f498babc700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49840c44b0 con 0x7f498406a7e0 2023-10-16T13:27:38.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.244+0000 7f4989057700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49840cc8a0 0x7f49841f2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:38.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.244+0000 7f4989057700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49840cc8a0 0x7f49841f2aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48328/0 (socket says 172.21.15.16:48328) 2023-10-16T13:27:38.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.244+0000 7f4989057700 1 -- 172.21.15.16:0/3293924476 learned_addr learned my addr 172.21.15.16:0/3293924476 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:38.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.244+0000 7f4989057700 1 -- 172.21.15.16:0/3293924476 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f498406a7e0 msgr2=0x7f49840c3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:38.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.244+0000 7f4989858700 1 --2- 172.21.15.16:0/3293924476 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f498406a7e0 0x7f49840c3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:38.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.244+0000 7f498a059700 1 --2- 172.21.15.16:0/3293924476 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f498415b280 0x7f49840ad2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:38.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.244+0000 7f4989057700 1 --2- 172.21.15.16:0/3293924476 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f498406a7e0 0x7f49840c3f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:38.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.244+0000 7f4989057700 1 -- 172.21.15.16:0/3293924476 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f498415b280 msgr2=0x7f49840ad2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:38.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.245+0000 7f4989057700 1 --2- 172.21.15.16:0/3293924476 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f498415b280 0x7f49840ad2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:38.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.245+0000 7f4989057700 1 -- 172.21.15.16:0/3293924476 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49780094a0 con 0x7f49840cc8a0 2023-10-16T13:27:38.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.245+0000 7f4989057700 1 --2- 172.21.15.16:0/3293924476 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49840cc8a0 0x7f49841f2aa0 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7f4978009b60 tx=0x7f4978004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:38.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.245+0000 7f49767fc700 1 -- 172.21.15.16:0/3293924476 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49780036a0 con 0x7f49840cc8a0 2023-10-16T13:27:38.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.245+0000 7f49767fc700 1 -- 172.21.15.16:0/3293924476 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f497801f070 con 0x7f49840cc8a0 2023-10-16T13:27:38.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.245+0000 7f498babc700 1 -- 172.21.15.16:0/3293924476 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49841f2fe0 con 0x7f49840cc8a0 2023-10-16T13:27:38.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.245+0000 7f49767fc700 1 -- 172.21.15.16:0/3293924476 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4978003a60 con 0x7f49840cc8a0 2023-10-16T13:27:38.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.245+0000 7f498babc700 1 -- 172.21.15.16:0/3293924476 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49840afe70 con 0x7f49840cc8a0 2023-10-16T13:27:38.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.246+0000 7f498babc700 1 -- 172.21.15.16:0/3293924476 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f498404ebb0 con 0x7f49840cc8a0 2023-10-16T13:27:38.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.246+0000 7f49767fc700 1 -- 172.21.15.16:0/3293924476 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4978003bc0 con 0x7f49840cc8a0 2023-10-16T13:27:38.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.247+0000 7f49767fc700 1 --2- 172.21.15.16:0/3293924476 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f496806b020 0x7f496806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:38.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.247+0000 7f49767fc700 1 -- 172.21.15.16:0/3293924476 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 5277+0+0 (secure 0 0 0) 0x7f4978021070 con 0x7f49840cc8a0 2023-10-16T13:27:38.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.247+0000 7f4989858700 1 --2- 172.21.15.16:0/3293924476 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f496806b020 0x7f496806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:38.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.247+0000 7f4989858700 1 --2- 172.21.15.16:0/3293924476 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f496806b020 0x7f496806d4e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4980009950 tx=0x7f4980008040 comp rx=0 tx=0).ready entity=mgr.9510 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:38.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.251+0000 7f49767fc700 1 -- 172.21.15.16:0/3293924476 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4978056100 con 0x7f49840cc8a0 2023-10-16T13:27:38.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.542+0000 7f498babc700 1 -- 172.21.15.16:0/3293924476 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f49840af900 con 0x7f49840cc8a0 2023-10-16T13:27:38.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.542+0000 7f49767fc700 1 -- 172.21.15.16:0/3293924476 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v46) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f4978015020 con 0x7f49840cc8a0 2023-10-16T13:27:38.545 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:27:38.545 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-10-16T13:27:38.545 INFO:teuthology.orchestra.run.smithi016.stdout: "auth_dump": [ 2023-10-16T13:27:38.546 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:27:38.546 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.a", 2023-10-16T13:27:38.546 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOKC1l/rBtOhAA/ob5WUvpz8Glp4E+xPZOYA==", 2023-10-16T13:27:38.546 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:27:38.547 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T13:27:38.547 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-10-16T13:27:38.547 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-10-16T13:27:38.547 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:27:38.547 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:27:38.547 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:27:38.548 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:27:38.548 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.b", 2023-10-16T13:27:38.548 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1lKAVsCxAAvhXuLY6i1+Iz1yq5BVPMNQ==", 2023-10-16T13:27:38.548 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:27:38.548 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T13:27:38.548 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-10-16T13:27:38.549 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-10-16T13:27:38.549 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:27:38.549 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:27:38.549 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:27:38.549 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:27:38.550 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.c", 2023-10-16T13:27:38.550 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1lvBowBhAA3waJNYhdgs8PGCYqYNYUog==", 2023-10-16T13:27:38.550 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:27:38.550 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T13:27:38.550 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-10-16T13:27:38.551 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-10-16T13:27:38.551 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:27:38.551 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:27:38.551 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:27:38.551 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:27:38.551 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.d", 2023-10-16T13:27:38.552 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1lS/fnEhAAfNYgQ5BW86JMjYm/bBfnaQ==", 2023-10-16T13:27:38.552 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:27:38.552 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T13:27:38.552 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-10-16T13:27:38.552 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-10-16T13:27:38.552 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:27:38.553 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:27:38.553 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:27:38.553 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:27:38.553 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.0", 2023-10-16T13:27:38.553 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAQKC1lbxllOhAAZeLdp2GMl6HqQPaQjKlNqw==", 2023-10-16T13:27:38.554 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:27:38.554 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T13:27:38.554 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T13:27:38.554 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:27:38.554 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:27:38.555 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:27:38.555 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:27:38.555 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.1", 2023-10-16T13:27:38.555 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQATKC1lnU6KOhAAHxvlbmzBptf1keGgL+Rbkg==", 2023-10-16T13:27:38.555 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:27:38.555 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T13:27:38.556 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T13:27:38.556 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:27:38.556 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:27:38.556 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:27:38.556 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:27:38.556 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.2", 2023-10-16T13:27:38.557 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAXKC1l9jFyARAAr/NAwtWhBMut9sAAoDEGWg==", 2023-10-16T13:27:38.557 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:27:38.557 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T13:27:38.557 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T13:27:38.557 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:27:38.557 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:27:38.558 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:27:38.558 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:27:38.558 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.3", 2023-10-16T13:27:38.558 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAaKC1lbw/lARAAGJ/9DNb5/7kLFSq7vrLtXg==", 2023-10-16T13:27:38.558 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:27:38.559 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T13:27:38.559 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T13:27:38.559 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:27:38.559 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:27:38.559 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:27:38.559 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:27:38.560 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.4", 2023-10-16T13:27:38.560 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAeKC1lZb7jERAAYpvb1xcSli9CTq7L8ChLOA==", 2023-10-16T13:27:38.560 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:27:38.560 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T13:27:38.560 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T13:27:38.560 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:27:38.561 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:27:38.561 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:27:38.561 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:27:38.561 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.5", 2023-10-16T13:27:38.561 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAhKC1l/4HuEBAAqfnRqjknij9KES/fqiyIBQ==", 2023-10-16T13:27:38.561 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:27:38.562 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T13:27:38.562 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T13:27:38.562 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:27:38.562 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:27:38.562 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:27:38.563 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:27:38.563 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.6", 2023-10-16T13:27:38.563 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAkKC1lzoUkEBAA+Q82OiBMNag8+R316WQtUw==", 2023-10-16T13:27:38.563 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:27:38.563 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T13:27:38.563 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T13:27:38.564 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:27:38.564 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:27:38.564 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:27:38.564 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:27:38.564 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.7", 2023-10-16T13:27:38.564 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAnKC1ljUu4DxAAc+G4anJANC2GhLJ8wb6gYg==", 2023-10-16T13:27:38.565 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:27:38.565 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T13:27:38.565 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T13:27:38.565 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:27:38.565 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:27:38.565 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:27:38.566 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:27:38.566 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.0", 2023-10-16T13:27:38.566 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1l2l6PFxAA9QyXtBAHdGAXpwH0JfknuQ==", 2023-10-16T13:27:38.566 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:27:38.566 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T13:27:38.567 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow r", 2023-10-16T13:27:38.567 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-16T13:27:38.567 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:27:38.567 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:27:38.568 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:27:38.568 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.1", 2023-10-16T13:27:38.568 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1lAcHFIxAAcNt2bmv2R6j5L3DYkdw7eQ==", 2023-10-16T13:27:38.568 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:27:38.568 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T13:27:38.568 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-10-16T13:27:38.569 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-10-16T13:27:38.569 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-10-16T13:27:38.569 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:27:38.569 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:27:38.569 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:27:38.570 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.2", 2023-10-16T13:27:38.570 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1lA6wnGxAAdtaqYhtIJu8qENMrD8xvVA==", 2023-10-16T13:27:38.570 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:27:38.570 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T13:27:38.570 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-10-16T13:27:38.570 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-10-16T13:27:38.571 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-10-16T13:27:38.571 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:27:38.571 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:27:38.571 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:27:38.571 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.3", 2023-10-16T13:27:38.572 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1l/ddAJxAA3fNKKn9XkXas1BlDN+ggjw==", 2023-10-16T13:27:38.572 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:27:38.572 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T13:27:38.572 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-10-16T13:27:38.572 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-10-16T13:27:38.572 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-10-16T13:27:38.573 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:27:38.573 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:27:38.573 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:27:38.573 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.admin", 2023-10-16T13:27:38.573 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOKC1ld5jUHBAAc6esnCce3dSEZn3wVjhIlQ==", 2023-10-16T13:27:38.573 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:27:38.574 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2023-10-16T13:27:38.574 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-10-16T13:27:38.574 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-10-16T13:27:38.574 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:27:38.574 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:27:38.575 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:27:38.575 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:27:38.575 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mgr.x", 2023-10-16T13:27:38.575 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOKC1l6b3vNhAAmj5xbfid2sZM2KfHmAf1Wg==", 2023-10-16T13:27:38.575 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:27:38.576 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2023-10-16T13:27:38.576 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile mgr", 2023-10-16T13:27:38.576 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:27:38.576 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:27:38.577 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:27:38.577 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:27:38.577 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mgr.y", 2023-10-16T13:27:38.577 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOKC1lLz2nMhAA8NqFYZ5nDKkzPBMrkwsVuQ==", 2023-10-16T13:27:38.577 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:27:38.577 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2023-10-16T13:27:38.578 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile mgr", 2023-10-16T13:27:38.578 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:27:38.578 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:27:38.578 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:27:38.578 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-10-16T13:27:38.578 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-10-16T13:27:38.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.544+0000 7f498babc700 1 -- 172.21.15.16:0/3293924476 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f496806b020 msgr2=0x7f496806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:38.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.544+0000 7f498babc700 1 --2- 172.21.15.16:0/3293924476 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f496806b020 0x7f496806d4e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4980009950 tx=0x7f4980008040 comp rx=0 tx=0).stop 2023-10-16T13:27:38.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.544+0000 7f498babc700 1 -- 172.21.15.16:0/3293924476 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49840cc8a0 msgr2=0x7f49841f2aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:38.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.544+0000 7f498babc700 1 --2- 172.21.15.16:0/3293924476 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49840cc8a0 0x7f49841f2aa0 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7f4978009b60 tx=0x7f4978004750 comp rx=0 tx=0).stop 2023-10-16T13:27:38.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.544+0000 7f498babc700 1 -- 172.21.15.16:0/3293924476 shutdown_connections 2023-10-16T13:27:38.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.544+0000 7f498babc700 1 --2- 172.21.15.16:0/3293924476 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f496806b020 0x7f496806d4e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:38.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.544+0000 7f498babc700 1 --2- 172.21.15.16:0/3293924476 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f498415b280 0x7f49840ad2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:38.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.544+0000 7f498babc700 1 --2- 172.21.15.16:0/3293924476 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f498406a7e0 0x7f49840c3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:38.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.544+0000 7f498babc700 1 --2- 172.21.15.16:0/3293924476 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49840cc8a0 0x7f49841f2aa0 unknown :-1 s=CLOSED pgs=1898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:38.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.544+0000 7f498babc700 1 -- 172.21.15.16:0/3293924476 >> 172.21.15.16:0/3293924476 conn(0x7f49840b1fd0 msgr2=0x7f498415a870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:38.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.545+0000 7f498babc700 1 -- 172.21.15.16:0/3293924476 shutdown_connections 2023-10-16T13:27:38.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.545+0000 7f498babc700 1 -- 172.21.15.16:0/3293924476 wait complete. 2023-10-16T13:27:38.583 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:27:38.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.705+0000 7f419656f700 1 Processor -- start 2023-10-16T13:27:38.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.705+0000 7f419656f700 1 -- start start 2023-10-16T13:27:38.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.706+0000 7f419656f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4190161ba0 0x7f41900cf920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:38.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.706+0000 7f419656f700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41900d0070 con 0x7f419015d750 2023-10-16T13:27:38.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.706+0000 7f419656f700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41900d0350 con 0x7f419015db30 2023-10-16T13:27:38.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.706+0000 7f419656f700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41900cfe60 con 0x7f4190161ba0 2023-10-16T13:27:38.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.706+0000 7f418ffff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f419015d750 0x7f419006c960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37608/0 (socket says 172.21.15.16:37608) 2023-10-16T13:27:38.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.706+0000 7f4194b0c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4190161ba0 0x7f41900cf920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:38.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.706+0000 7f418ffff700 1 -- 172.21.15.16:0/624426778 learned_addr learned my addr 172.21.15.16:0/624426778 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:27:38.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.707+0000 7f4194b0c700 1 -- 172.21.15.16:0/624426778 >> v1:172.21.15.49:6789/0 conn(0x7f419015db30 legacy=0x7f4190161490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:38.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.707+0000 7f4194b0c700 1 -- 172.21.15.16:0/624426778 >> v1:172.21.15.16:6789/0 conn(0x7f419015d750 legacy=0x7f419006c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:38.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.707+0000 7f4194b0c700 1 -- 172.21.15.16:0/624426778 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41900d0630 con 0x7f4190161ba0 2023-10-16T13:27:38.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.707+0000 7f418effd700 1 -- 172.21.15.16:0/624426778 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3215502264 0 0) 0x7f41900d0070 con 0x7f419015d750 2023-10-16T13:27:38.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.707+0000 7f4194b0c700 1 --2- 172.21.15.16:0/624426778 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4190161ba0 0x7f41900cf920 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7f4188009d90 tx=0x7f418800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15a3b4320688044d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:38.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.707+0000 7f418effd700 1 -- 172.21.15.16:0/624426778 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f418800e070 con 0x7f4190161ba0 2023-10-16T13:27:38.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.707+0000 7f418effd700 1 -- 172.21.15.16:0/624426778 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f418800bb90 con 0x7f4190161ba0 2023-10-16T13:27:38.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.707+0000 7f418effd700 1 -- 172.21.15.16:0/624426778 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4188003a20 con 0x7f4190161ba0 2023-10-16T13:27:38.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.708+0000 7f419656f700 1 -- 172.21.15.16:0/624426778 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4190161ba0 msgr2=0x7f41900cf920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:38.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.708+0000 7f419656f700 1 --2- 172.21.15.16:0/624426778 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4190161ba0 0x7f41900cf920 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7f4188009d90 tx=0x7f418800b330 comp rx=0 tx=0).stop 2023-10-16T13:27:38.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.708+0000 7f419656f700 1 -- 172.21.15.16:0/624426778 shutdown_connections 2023-10-16T13:27:38.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.708+0000 7f419656f700 1 --2- 172.21.15.16:0/624426778 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4190161ba0 0x7f41900cf920 unknown :-1 s=CLOSED pgs=1900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:38.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.708+0000 7f419656f700 1 -- 172.21.15.16:0/624426778 >> 172.21.15.16:0/624426778 conn(0x7f41900d14b0 msgr2=0x7f41900d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:38.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.708+0000 7f419656f700 1 -- 172.21.15.16:0/624426778 shutdown_connections 2023-10-16T13:27:38.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.708+0000 7f419656f700 1 -- 172.21.15.16:0/624426778 wait complete. 2023-10-16T13:27:38.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.709+0000 7f419656f700 1 Processor -- start 2023-10-16T13:27:38.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.709+0000 7f419656f700 1 -- start start 2023-10-16T13:27:38.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.710+0000 7f419656f700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f419015d750 0x7f419009edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:38.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.710+0000 7f419656f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f419015db30 0x7f41900c4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:38.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.710+0000 7f419656f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4190161ba0 0x7f4190157a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:38.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.710+0000 7f419656f700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4190157f60 con 0x7f419015db30 2023-10-16T13:27:38.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.710+0000 7f419656f700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41900c5ad0 con 0x7f419015d750 2023-10-16T13:27:38.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.710+0000 7f419656f700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41900c5db0 con 0x7f4190161ba0 2023-10-16T13:27:38.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.710+0000 7f418f7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f419015db30 0x7f41900c4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:38.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.710+0000 7f418f7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f419015db30 0x7f41900c4fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48340/0 (socket says 172.21.15.16:48340) 2023-10-16T13:27:38.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.710+0000 7f418f7fe700 1 -- 172.21.15.16:0/1192428522 learned_addr learned my addr 172.21.15.16:0/1192428522 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:38.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.710+0000 7f418ffff700 1 --2- 172.21.15.16:0/1192428522 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f419015d750 0x7f419009edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:38.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.710+0000 7f4194b0c700 1 --2- 172.21.15.16:0/1192428522 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4190161ba0 0x7f4190157a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:38.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.710+0000 7f418f7fe700 1 -- 172.21.15.16:0/1192428522 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4190161ba0 msgr2=0x7f4190157a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:38.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.710+0000 7f418f7fe700 1 --2- 172.21.15.16:0/1192428522 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4190161ba0 0x7f4190157a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:38.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.710+0000 7f418f7fe700 1 -- 172.21.15.16:0/1192428522 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f419015d750 msgr2=0x7f419009edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:38.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.710+0000 7f418f7fe700 1 --2- 172.21.15.16:0/1192428522 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f419015d750 0x7f419009edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:38.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.710+0000 7f418f7fe700 1 -- 172.21.15.16:0/1192428522 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4188009a70 con 0x7f419015db30 2023-10-16T13:27:38.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.710+0000 7f418ffff700 1 --2- 172.21.15.16:0/1192428522 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f419015d750 0x7f419009edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:27:38.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.710+0000 7f418f7fe700 1 --2- 172.21.15.16:0/1192428522 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f419015db30 0x7f41900c4fe0 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7f417800c4a0 tx=0x7f417800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:38.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.711+0000 7f418cff9700 1 -- 172.21.15.16:0/1192428522 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4178011790 con 0x7f419015db30 2023-10-16T13:27:38.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.711+0000 7f418cff9700 1 -- 172.21.15.16:0/1192428522 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41780118f0 con 0x7f419015db30 2023-10-16T13:27:38.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.711+0000 7f418cff9700 1 -- 172.21.15.16:0/1192428522 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4178011ca0 con 0x7f419015db30 2023-10-16T13:27:38.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.711+0000 7f419656f700 1 -- 172.21.15.16:0/1192428522 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41900b53b0 con 0x7f419015db30 2023-10-16T13:27:38.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.711+0000 7f419656f700 1 -- 172.21.15.16:0/1192428522 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41900c6090 con 0x7f419015db30 2023-10-16T13:27:38.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.712+0000 7f419656f700 1 -- 172.21.15.16:0/1192428522 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41900b58d0 con 0x7f419015db30 2023-10-16T13:27:38.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.712+0000 7f418cff9700 1 -- 172.21.15.16:0/1192428522 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f417801c460 con 0x7f419015db30 2023-10-16T13:27:38.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.712+0000 7f418cff9700 1 --2- 172.21.15.16:0/1192428522 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f417006b020 0x7f417006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:38.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.713+0000 7f418cff9700 1 -- 172.21.15.16:0/1192428522 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 5277+0+0 (secure 0 0 0) 0x7f4178089370 con 0x7f419015db30 2023-10-16T13:27:38.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.713+0000 7f418ffff700 1 --2- 172.21.15.16:0/1192428522 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f417006b020 0x7f417006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:38.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.713+0000 7f418ffff700 1 --2- 172.21.15.16:0/1192428522 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f417006b020 0x7f417006d4e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f41800010e0 tx=0x7f4180009040 comp rx=0 tx=0).ready entity=mgr.9510 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:38.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.717+0000 7f418cff9700 1 -- 172.21.15.16:0/1192428522 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f41900b58d0 con 0x7f419015db30 2023-10-16T13:27:38.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.923+0000 7f419656f700 1 -- 172.21.15.16:0/1192428522 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f41900b58d0 con 0x7f419015db30 2023-10-16T13:27:38.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.923+0000 7f418cff9700 1 -- 172.21.15.16:0/1192428522 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v173) v1 ==== 74+0+13123 (secure 0 0 0) 0x7f41900b58d0 con 0x7f419015db30 2023-10-16T13:27:38.926 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:27:38.927 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":173,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:27:38.087654+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:27:38.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.926+0000 7f419656f700 1 -- 172.21.15.16:0/1192428522 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f417006b020 msgr2=0x7f417006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:38.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.926+0000 7f419656f700 1 --2- 172.21.15.16:0/1192428522 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f417006b020 0x7f417006d4e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f41800010e0 tx=0x7f4180009040 comp rx=0 tx=0).stop 2023-10-16T13:27:38.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.926+0000 7f419656f700 1 -- 172.21.15.16:0/1192428522 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f419015db30 msgr2=0x7f41900c4fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:38.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.926+0000 7f419656f700 1 --2- 172.21.15.16:0/1192428522 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f419015db30 0x7f41900c4fe0 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7f417800c4a0 tx=0x7f417800c860 comp rx=0 tx=0).stop 2023-10-16T13:27:38.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.926+0000 7f419656f700 1 -- 172.21.15.16:0/1192428522 shutdown_connections 2023-10-16T13:27:38.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.926+0000 7f419656f700 1 --2- 172.21.15.16:0/1192428522 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4190161ba0 0x7f4190157a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:38.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.926+0000 7f419656f700 1 --2- 172.21.15.16:0/1192428522 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f417006b020 0x7f417006d4e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:38.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.926+0000 7f419656f700 1 --2- 172.21.15.16:0/1192428522 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f419015d750 0x7f419009edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:38.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.926+0000 7f419656f700 1 --2- 172.21.15.16:0/1192428522 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f419015db30 0x7f41900c4fe0 unknown :-1 s=CLOSED pgs=1901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:38.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.926+0000 7f419656f700 1 -- 172.21.15.16:0/1192428522 >> 172.21.15.16:0/1192428522 conn(0x7f41900d14b0 msgr2=0x7f419015e000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:38.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.926+0000 7f419656f700 1 -- 172.21.15.16:0/1192428522 shutdown_connections 2023-10-16T13:27:38.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:38.926+0000 7f419656f700 1 -- 172.21.15.16:0/1192428522 wait complete. 2023-10-16T13:27:38.941 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T13:27:38.942 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T13:27:39.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.068+0000 7f78fed60700 1 Processor -- start 2023-10-16T13:27:39.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.068+0000 7f78fed60700 1 -- start start 2023-10-16T13:27:39.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.069+0000 7f78fed60700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f78f815b070 0x7f78f815b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:39.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.069+0000 7f78fed60700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78f80cef70 con 0x7f78f806e170 2023-10-16T13:27:39.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.069+0000 7f78fed60700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78f80ce360 con 0x7f78f80cb880 2023-10-16T13:27:39.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.069+0000 7f78fed60700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78f80ce640 con 0x7f78f815b070 2023-10-16T13:27:39.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.069+0000 7f78fd2fd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f78f806e170 0x7f78f80cf400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37624/0 (socket says 172.21.15.16:37624) 2023-10-16T13:27:39.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.069+0000 7f78fd2fd700 1 -- 172.21.15.16:0/3957959096 learned_addr learned my addr 172.21.15.16:0/3957959096 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:27:39.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.069+0000 7f78fcafc700 1 --2- 172.21.15.16:0/3957959096 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f78f815b070 0x7f78f815b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:39.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.069+0000 7f78f77fe700 1 -- 172.21.15.16:0/3957959096 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2886935676 0 0) 0x7f78f80cef70 con 0x7f78f806e170 2023-10-16T13:27:39.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.069+0000 7f78f77fe700 1 -- 172.21.15.16:0/3957959096 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78e80035b0 con 0x7f78f806e170 2023-10-16T13:27:39.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.069+0000 7f78f77fe700 1 -- 172.21.15.16:0/3957959096 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 340112767 0 0) 0x7f78f80ce360 con 0x7f78f80cb880 2023-10-16T13:27:39.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.069+0000 7f78f77fe700 1 -- 172.21.15.16:0/3957959096 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78f80cef70 con 0x7f78f80cb880 2023-10-16T13:27:39.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.069+0000 7f78fcafc700 1 -- 172.21.15.16:0/3957959096 >> v1:172.21.15.49:6789/0 conn(0x7f78f80cb880 legacy=0x7f78f806da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:39.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.069+0000 7f78fcafc700 1 -- 172.21.15.16:0/3957959096 >> v1:172.21.15.16:6789/0 conn(0x7f78f806e170 legacy=0x7f78f80cf400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:39.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.069+0000 7f78f77fe700 1 -- 172.21.15.16:0/3957959096 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2083284596 0 0) 0x7f78e80035b0 con 0x7f78f806e170 2023-10-16T13:27:39.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.069+0000 7f78fcafc700 1 -- 172.21.15.16:0/3957959096 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78f80ce920 con 0x7f78f815b070 2023-10-16T13:27:39.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.070+0000 7f78fcafc700 1 --2- 172.21.15.16:0/3957959096 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f78f815b070 0x7f78f815b450 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f78ec009870 tx=0x7f78ec009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=44bdf12b1152048a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:39.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.070+0000 7f78f77fe700 1 -- 172.21.15.16:0/3957959096 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f00019f0 con 0x7f78f815b070 2023-10-16T13:27:39.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.070+0000 7f78f77fe700 1 -- 172.21.15.16:0/3957959096 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78ec004030 con 0x7f78f815b070 2023-10-16T13:27:39.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.070+0000 7f78f77fe700 1 -- 172.21.15.16:0/3957959096 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78ec0043a0 con 0x7f78f815b070 2023-10-16T13:27:39.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.070+0000 7f78fed60700 1 -- 172.21.15.16:0/3957959096 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f78f815b070 msgr2=0x7f78f815b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:39.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.070+0000 7f78fed60700 1 --2- 172.21.15.16:0/3957959096 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f78f815b070 0x7f78f815b450 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f78ec009870 tx=0x7f78ec009b80 comp rx=0 tx=0).stop 2023-10-16T13:27:39.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.070+0000 7f78fed60700 1 -- 172.21.15.16:0/3957959096 shutdown_connections 2023-10-16T13:27:39.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.070+0000 7f78fed60700 1 --2- 172.21.15.16:0/3957959096 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f78f815b070 0x7f78f815b450 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:39.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.070+0000 7f78fed60700 1 -- 172.21.15.16:0/3957959096 >> 172.21.15.16:0/3957959096 conn(0x7f78f80b1f40 msgr2=0x7f78f80b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:39.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.070+0000 7f78fed60700 1 -- 172.21.15.16:0/3957959096 shutdown_connections 2023-10-16T13:27:39.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.070+0000 7f78fed60700 1 -- 172.21.15.16:0/3957959096 wait complete. 2023-10-16T13:27:39.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.071+0000 7f78fed60700 1 Processor -- start 2023-10-16T13:27:39.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.072+0000 7f78fed60700 1 -- start start 2023-10-16T13:27:39.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.072+0000 7f78fed60700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f78f806e170 0x7f78f806b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:39.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.072+0000 7f78fed60700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f78f80cb880 0x7f78f814c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:39.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.072+0000 7f78fed60700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78f815b070 0x7f78f8157140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:39.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.072+0000 7f78fed60700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f806a7e0 con 0x7f78f815b070 2023-10-16T13:27:39.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.072+0000 7f78fed60700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f814ce00 con 0x7f78f806e170 2023-10-16T13:27:39.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.072+0000 7f78fed60700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78f806ad60 con 0x7f78f80cb880 2023-10-16T13:27:39.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.073+0000 7f78fd2fd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78f815b070 0x7f78f8157140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:39.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.073+0000 7f78fd2fd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78f815b070 0x7f78f8157140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48356/0 (socket says 172.21.15.16:48356) 2023-10-16T13:27:39.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.073+0000 7f78fd2fd700 1 -- 172.21.15.16:0/4081461439 learned_addr learned my addr 172.21.15.16:0/4081461439 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:39.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.073+0000 7f78fcafc700 1 --2- 172.21.15.16:0/4081461439 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f78f806e170 0x7f78f806b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:39.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.073+0000 7f78f7fff700 1 --2- 172.21.15.16:0/4081461439 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f78f80cb880 0x7f78f814c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:39.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.073+0000 7f78fd2fd700 1 -- 172.21.15.16:0/4081461439 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f78f80cb880 msgr2=0x7f78f814c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:39.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.073+0000 7f78fd2fd700 1 --2- 172.21.15.16:0/4081461439 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f78f80cb880 0x7f78f814c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:39.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.073+0000 7f78fd2fd700 1 -- 172.21.15.16:0/4081461439 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f78f806e170 msgr2=0x7f78f806b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:39.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.073+0000 7f78fd2fd700 1 --2- 172.21.15.16:0/4081461439 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f78f806e170 0x7f78f806b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:39.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.073+0000 7f78fd2fd700 1 -- 172.21.15.16:0/4081461439 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78ec0094a0 con 0x7f78f815b070 2023-10-16T13:27:39.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.073+0000 7f78fcafc700 1 --2- 172.21.15.16:0/4081461439 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f78f806e170 0x7f78f806b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:27:39.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.073+0000 7f78fd2fd700 1 --2- 172.21.15.16:0/4081461439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78f815b070 0x7f78f8157140 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7f78f000cc70 tx=0x7f78f000a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:39.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.073+0000 7f78f57fa700 1 -- 172.21.15.16:0/4081461439 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f000ade0 con 0x7f78f815b070 2023-10-16T13:27:39.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.073+0000 7f78f57fa700 1 -- 172.21.15.16:0/4081461439 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78f000ce70 con 0x7f78f815b070 2023-10-16T13:27:39.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.073+0000 7f78fed60700 1 -- 172.21.15.16:0/4081461439 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78f8157b30 con 0x7f78f815b070 2023-10-16T13:27:39.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.073+0000 7f78f57fa700 1 -- 172.21.15.16:0/4081461439 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f00072f0 con 0x7f78f815b070 2023-10-16T13:27:39.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.073+0000 7f78fed60700 1 -- 172.21.15.16:0/4081461439 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78f806a9f0 con 0x7f78f815b070 2023-10-16T13:27:39.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.074+0000 7f78fed60700 1 -- 172.21.15.16:0/4081461439 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78f804ebb0 con 0x7f78f815b070 2023-10-16T13:27:39.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.075+0000 7f78f57fa700 1 -- 172.21.15.16:0/4081461439 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f78f0007450 con 0x7f78f815b070 2023-10-16T13:27:39.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.075+0000 7f78f57fa700 1 --2- 172.21.15.16:0/4081461439 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f78dc06b050 0x7f78dc06d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:39.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.075+0000 7f78fcafc700 1 --2- 172.21.15.16:0/4081461439 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f78dc06b050 0x7f78dc06d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:39.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.075+0000 7f78f57fa700 1 -- 172.21.15.16:0/4081461439 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 5277+0+0 (secure 0 0 0) 0x7f78f0089500 con 0x7f78f815b070 2023-10-16T13:27:39.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.076+0000 7f78fcafc700 1 --2- 172.21.15.16:0/4081461439 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f78dc06b050 0x7f78dc06d510 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f78ec009bc0 tx=0x7f78ec013040 comp rx=0 tx=0).ready entity=mgr.9510 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:39.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.080+0000 7f78f57fa700 1 -- 172.21.15.16:0/4081461439 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f78f0054650 con 0x7f78f815b070 2023-10-16T13:27:39.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.281+0000 7f78fed60700 1 -- 172.21.15.16:0/4081461439 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f78f80cd200 con 0x7f78f815b070 2023-10-16T13:27:39.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.281+0000 7f78f57fa700 1 -- 172.21.15.16:0/4081461439 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v173) v1 ==== 93+0+10 (secure 0 0 0) 0x7f78f0018020 con 0x7f78f815b070 2023-10-16T13:27:39.284 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T13:27:39.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.283+0000 7f78fed60700 1 -- 172.21.15.16:0/4081461439 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f78dc06b050 msgr2=0x7f78dc06d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:39.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.283+0000 7f78fed60700 1 --2- 172.21.15.16:0/4081461439 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f78dc06b050 0x7f78dc06d510 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f78ec009bc0 tx=0x7f78ec013040 comp rx=0 tx=0).stop 2023-10-16T13:27:39.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.283+0000 7f78fed60700 1 -- 172.21.15.16:0/4081461439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78f815b070 msgr2=0x7f78f8157140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:39.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.283+0000 7f78fed60700 1 --2- 172.21.15.16:0/4081461439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78f815b070 0x7f78f8157140 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7f78f000cc70 tx=0x7f78f000a340 comp rx=0 tx=0).stop 2023-10-16T13:27:39.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.283+0000 7f78fed60700 1 -- 172.21.15.16:0/4081461439 shutdown_connections 2023-10-16T13:27:39.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.283+0000 7f78fed60700 1 --2- 172.21.15.16:0/4081461439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78f815b070 0x7f78f8157140 unknown :-1 s=CLOSED pgs=1903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:39.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.283+0000 7f78fed60700 1 --2- 172.21.15.16:0/4081461439 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f78dc06b050 0x7f78dc06d510 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:39.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.283+0000 7f78fed60700 1 --2- 172.21.15.16:0/4081461439 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f78f806e170 0x7f78f806b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:39.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.283+0000 7f78fed60700 1 --2- 172.21.15.16:0/4081461439 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f78f80cb880 0x7f78f814c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:39.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.283+0000 7f78fed60700 1 -- 172.21.15.16:0/4081461439 >> 172.21.15.16:0/4081461439 conn(0x7f78f80b1f40 msgr2=0x7f78f8159f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:39.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.283+0000 7f78fed60700 1 -- 172.21.15.16:0/4081461439 shutdown_connections 2023-10-16T13:27:39.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.283+0000 7f78fed60700 1 -- 172.21.15.16:0/4081461439 wait complete. 2023-10-16T13:27:39.298 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-16T13:27:39.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.421+0000 7f165c35c700 1 Processor -- start 2023-10-16T13:27:39.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.421+0000 7f165c35c700 1 -- start start 2023-10-16T13:27:39.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.422+0000 7f165c35c700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f16540cf4a0 0x7f16540726e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:39.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.422+0000 7f165c35c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f16540cf880 0x7f16540b52f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:39.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.422+0000 7f165c35c700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f165415bc80 con 0x7f16540b5830 2023-10-16T13:27:39.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.422+0000 7f165c35c700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1654072c20 con 0x7f16540cf880 2023-10-16T13:27:39.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.422+0000 7f165c35c700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f165415b070 con 0x7f16540cf4a0 2023-10-16T13:27:39.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.422+0000 7f16598f7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f16540cf880 0x7f16540b52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:39.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.422+0000 7f165a8f9700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f16540b5830 0x7f165415a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37638/0 (socket says 172.21.15.16:37638) 2023-10-16T13:27:39.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.422+0000 7f165a8f9700 1 -- 172.21.15.16:0/3794596166 learned_addr learned my addr 172.21.15.16:0/3794596166 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:27:39.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.422+0000 7f165a0f8700 1 --2- 172.21.15.16:0/3794596166 >> v2:172.21.15.49:3300/0 conn(0x7f16540cf4a0 0x7f16540726e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:39.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.422+0000 7f16598f7700 1 -- 172.21.15.16:0/3794596166 >> v2:172.21.15.49:3300/0 conn(0x7f16540cf4a0 msgr2=0x7f16540726e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:39.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.422+0000 7f16598f7700 1 --2- 172.21.15.16:0/3794596166 >> v2:172.21.15.49:3300/0 conn(0x7f16540cf4a0 0x7f16540726e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:39.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.422+0000 7f16598f7700 1 -- 172.21.15.16:0/3794596166 >> v1:172.21.15.16:6789/0 conn(0x7f16540b5830 legacy=0x7f165415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:39.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.422+0000 7f16598f7700 1 -- 172.21.15.16:0/3794596166 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f165415b350 con 0x7f16540cf880 2023-10-16T13:27:39.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.422+0000 7f16590f6700 1 -- 172.21.15.16:0/3794596166 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3750415128 0 0) 0x7f165415bc80 con 0x7f16540b5830 2023-10-16T13:27:39.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.422+0000 7f165a0f8700 1 --2- 172.21.15.16:0/3794596166 >> v2:172.21.15.49:3300/0 conn(0x7f16540cf4a0 0x7f16540726e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:27:39.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.423+0000 7f16598f7700 1 --2- 172.21.15.16:0/3794596166 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16540cf880 0x7f16540b52f0 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7f16540b2d20 tx=0x7f1650009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e213ec608bbb6f6e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:39.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.423+0000 7f16590f6700 1 -- 172.21.15.16:0/3794596166 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1650011830 con 0x7f16540cf880 2023-10-16T13:27:39.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.423+0000 7f16590f6700 1 -- 172.21.15.16:0/3794596166 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1650011990 con 0x7f16540cf880 2023-10-16T13:27:39.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.423+0000 7f16590f6700 1 -- 172.21.15.16:0/3794596166 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1650011d20 con 0x7f16540cf880 2023-10-16T13:27:39.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.423+0000 7f165c35c700 1 -- 172.21.15.16:0/3794596166 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16540cf880 msgr2=0x7f16540b52f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:39.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.423+0000 7f165c35c700 1 --2- 172.21.15.16:0/3794596166 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16540cf880 0x7f16540b52f0 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7f16540b2d20 tx=0x7f1650009b20 comp rx=0 tx=0).stop 2023-10-16T13:27:39.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.423+0000 7f165c35c700 1 -- 172.21.15.16:0/3794596166 shutdown_connections 2023-10-16T13:27:39.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.423+0000 7f165c35c700 1 --2- 172.21.15.16:0/3794596166 >> v2:172.21.15.49:3300/0 conn(0x7f16540cf4a0 0x7f16540726e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:39.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.423+0000 7f165c35c700 1 --2- 172.21.15.16:0/3794596166 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16540cf880 0x7f16540b52f0 unknown :-1 s=CLOSED pgs=1905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:39.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.423+0000 7f165c35c700 1 -- 172.21.15.16:0/3794596166 >> 172.21.15.16:0/3794596166 conn(0x7f16540b1fd0 msgr2=0x7f16540b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:39.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.423+0000 7f165c35c700 1 -- 172.21.15.16:0/3794596166 shutdown_connections 2023-10-16T13:27:39.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.423+0000 7f165c35c700 1 -- 172.21.15.16:0/3794596166 wait complete. 2023-10-16T13:27:39.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.425+0000 7f165c35c700 1 Processor -- start 2023-10-16T13:27:39.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.425+0000 7f165c35c700 1 -- start start 2023-10-16T13:27:39.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.425+0000 7f165c35c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16540b5830 0x7f1654155040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:39.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.425+0000 7f165c35c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f16540cf4a0 0x7f16540cbc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:39.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.425+0000 7f165c35c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f16540cf880 0x7f16541521d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:39.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.425+0000 7f165c35c700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16541516d0 con 0x7f16540b5830 2023-10-16T13:27:39.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.425+0000 7f165c35c700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1654150da0 con 0x7f16540cf4a0 2023-10-16T13:27:39.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.425+0000 7f165c35c700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1654151080 con 0x7f16540cf880 2023-10-16T13:27:39.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.426+0000 7f165a0f8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16540b5830 0x7f1654155040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:39.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.426+0000 7f165a0f8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16540b5830 0x7f1654155040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48366/0 (socket says 172.21.15.16:48366) 2023-10-16T13:27:39.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.426+0000 7f165a0f8700 1 -- 172.21.15.16:0/213153102 learned_addr learned my addr 172.21.15.16:0/213153102 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:39.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.426+0000 7f165a0f8700 1 -- 172.21.15.16:0/213153102 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f16540cf880 msgr2=0x7f16541521d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:39.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.426+0000 7f165a0f8700 1 --2- 172.21.15.16:0/213153102 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f16540cf880 0x7f16541521d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:39.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.426+0000 7f165a0f8700 1 -- 172.21.15.16:0/213153102 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f16540cf4a0 msgr2=0x7f16540cbc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:39.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.426+0000 7f165a0f8700 1 --2- 172.21.15.16:0/213153102 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f16540cf4a0 0x7f16540cbc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:39.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.426+0000 7f165a0f8700 1 -- 172.21.15.16:0/213153102 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16500094a0 con 0x7f16540b5830 2023-10-16T13:27:39.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.426+0000 7f165a0f8700 1 --2- 172.21.15.16:0/213153102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16540b5830 0x7f1654155040 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7f164400c750 tx=0x7f164400cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:39.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.426+0000 7f164affd700 1 -- 172.21.15.16:0/213153102 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1644011800 con 0x7f16540b5830 2023-10-16T13:27:39.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.426+0000 7f164affd700 1 -- 172.21.15.16:0/213153102 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1644011960 con 0x7f16540b5830 2023-10-16T13:27:39.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.426+0000 7f165c35c700 1 -- 172.21.15.16:0/213153102 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1654151360 con 0x7f16540b5830 2023-10-16T13:27:39.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.426+0000 7f164affd700 1 -- 172.21.15.16:0/213153102 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1644011cd0 con 0x7f16540b5830 2023-10-16T13:27:39.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.426+0000 7f165c35c700 1 -- 172.21.15.16:0/213153102 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1654151e30 con 0x7f16540b5830 2023-10-16T13:27:39.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.427+0000 7f165c35c700 1 -- 172.21.15.16:0/213153102 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f165404ebb0 con 0x7f16540b5830 2023-10-16T13:27:39.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.428+0000 7f164affd700 1 -- 172.21.15.16:0/213153102 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1644021350 con 0x7f16540b5830 2023-10-16T13:27:39.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.428+0000 7f164affd700 1 --2- 172.21.15.16:0/213153102 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f163806b020 0x7f163806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:39.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.428+0000 7f164affd700 1 -- 172.21.15.16:0/213153102 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 5277+0+0 (secure 0 0 0) 0x7f16440895c0 con 0x7f16540b5830 2023-10-16T13:27:39.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.428+0000 7f16598f7700 1 --2- 172.21.15.16:0/213153102 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f163806b020 0x7f163806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:39.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.429+0000 7f16598f7700 1 --2- 172.21.15.16:0/213153102 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f163806b020 0x7f163806d4e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f16540ccce0 tx=0x7f1650010000 comp rx=0 tx=0).ready entity=mgr.9510 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:39.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.433+0000 7f164affd700 1 -- 172.21.15.16:0/213153102 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1644054690 con 0x7f16540b5830 2023-10-16T13:27:39.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.638+0000 7f165c35c700 1 -- 172.21.15.16:0/213153102 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f16540aff50 con 0x7f16540b5830 2023-10-16T13:27:39.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.638+0000 7f164affd700 1 -- 172.21.15.16:0/213153102 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v173) v1 ==== 84+0+65 (secure 0 0 0) 0x7f1644057cb0 con 0x7f16540b5830 2023-10-16T13:27:39.641 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:27:39.642 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2023-10-16T13:27:39.642 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:27:39.642 INFO:teuthology.orchestra.run.smithi016.stdout: "poolnum": 1, 2023-10-16T13:27:39.642 INFO:teuthology.orchestra.run.smithi016.stdout: "poolname": ".mgr" 2023-10-16T13:27:39.642 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:27:39.643 INFO:teuthology.orchestra.run.smithi016.stdout:] 2023-10-16T13:27:39.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.641+0000 7f165c35c700 1 -- 172.21.15.16:0/213153102 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f163806b020 msgr2=0x7f163806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:39.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.641+0000 7f165c35c700 1 --2- 172.21.15.16:0/213153102 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f163806b020 0x7f163806d4e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f16540ccce0 tx=0x7f1650010000 comp rx=0 tx=0).stop 2023-10-16T13:27:39.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.641+0000 7f165c35c700 1 -- 172.21.15.16:0/213153102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16540b5830 msgr2=0x7f1654155040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:39.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.641+0000 7f165c35c700 1 --2- 172.21.15.16:0/213153102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16540b5830 0x7f1654155040 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7f164400c750 tx=0x7f164400cb10 comp rx=0 tx=0).stop 2023-10-16T13:27:39.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.641+0000 7f165c35c700 1 -- 172.21.15.16:0/213153102 shutdown_connections 2023-10-16T13:27:39.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.641+0000 7f165c35c700 1 --2- 172.21.15.16:0/213153102 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f16540cf880 0x7f16541521d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:39.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.641+0000 7f165c35c700 1 --2- 172.21.15.16:0/213153102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16540b5830 0x7f1654155040 unknown :-1 s=CLOSED pgs=1906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:39.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.641+0000 7f165c35c700 1 --2- 172.21.15.16:0/213153102 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f163806b020 0x7f163806d4e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:39.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.641+0000 7f165c35c700 1 --2- 172.21.15.16:0/213153102 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f16540cf4a0 0x7f16540cbc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:39.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.641+0000 7f165c35c700 1 -- 172.21.15.16:0/213153102 >> 172.21.15.16:0/213153102 conn(0x7f16540b1fd0 msgr2=0x7f165415cfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:39.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.641+0000 7f165c35c700 1 -- 172.21.15.16:0/213153102 shutdown_connections 2023-10-16T13:27:39.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.641+0000 7f165c35c700 1 -- 172.21.15.16:0/213153102 wait complete. 2023-10-16T13:27:39.655 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-16T13:27:39.656 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-16T13:27:39.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.782+0000 7f520a87f700 1 Processor -- start 2023-10-16T13:27:39.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.782+0000 7f520a87f700 1 -- start start 2023-10-16T13:27:39.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.783+0000 7f520a87f700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f520415a3d0 0x7f520406d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:39.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.783+0000 7f520a87f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f52040cf600 0x7f52040cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:39.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.783+0000 7f520a87f700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52040cb870 con 0x7f520415e0b0 2023-10-16T13:27:39.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.783+0000 7f520a87f700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f520415f0c0 con 0x7f520415a3d0 2023-10-16T13:27:39.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.783+0000 7f520a87f700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5204072990 con 0x7f52040cf600 2023-10-16T13:27:39.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.783+0000 7f52037fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f520415e0b0 0x7f52040ceef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37648/0 (socket says 172.21.15.16:37648) 2023-10-16T13:27:39.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.783+0000 7f52037fe700 1 -- 172.21.15.16:0/1458771792 learned_addr learned my addr 172.21.15.16:0/1458771792 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:27:39.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.783+0000 7f5202ffd700 1 -- 172.21.15.16:0/1458771792 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1672184853 0 0) 0x7f52040cb870 con 0x7f520415e0b0 2023-10-16T13:27:39.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.783+0000 7f5208e1c700 1 --2- 172.21.15.16:0/1458771792 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f52040cf600 0x7f52040cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:39.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.783+0000 7f5203fff700 1 --2- 172.21.15.16:0/1458771792 >> v2:172.21.15.49:3300/0 conn(0x7f520415a3d0 0x7f520406d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:39.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.783+0000 7f5202ffd700 1 -- 172.21.15.16:0/1458771792 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51ec0035b0 con 0x7f520415e0b0 2023-10-16T13:27:39.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.784+0000 7f5202ffd700 1 -- 172.21.15.16:0/1458771792 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1670816831 0 0) 0x7f51ec0035b0 con 0x7f520415e0b0 2023-10-16T13:27:39.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.784+0000 7f5202ffd700 1 -- 172.21.15.16:0/1458771792 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f52040cf600 msgr2=0x7f52040cbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:39.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.784+0000 7f5202ffd700 1 --2- 172.21.15.16:0/1458771792 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f52040cf600 0x7f52040cbd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:39.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.784+0000 7f5202ffd700 1 -- 172.21.15.16:0/1458771792 >> v2:172.21.15.49:3300/0 conn(0x7f520415a3d0 msgr2=0x7f520406d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:39.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.784+0000 7f5202ffd700 1 --2- 172.21.15.16:0/1458771792 >> v2:172.21.15.49:3300/0 conn(0x7f520415a3d0 0x7f520406d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:39.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.784+0000 7f5202ffd700 1 -- 172.21.15.16:0/1458771792 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f520406e040 con 0x7f520415e0b0 2023-10-16T13:27:39.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.784+0000 7f5202ffd700 1 -- 172.21.15.16:0/1458771792 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f51f8001670 con 0x7f520415e0b0 2023-10-16T13:27:39.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.784+0000 7f5202ffd700 1 -- 172.21.15.16:0/1458771792 >> v1:172.21.15.16:6789/0 conn(0x7f520415e0b0 legacy=0x7f52040ceef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:39.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.784+0000 7f5202ffd700 1 --2- 172.21.15.16:0/1458771792 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f51ec003a00 0x7f51ec003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:39.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.784+0000 7f5202ffd700 1 --2- 172.21.15.16:0/1458771792 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51ec004850 0x7f51ec004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:39.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.784+0000 7f5202ffd700 1 --2- 172.21.15.16:0/1458771792 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f52040cf600 0x7f51ec005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:39.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.784+0000 7f5202ffd700 1 -- 172.21.15.16:0/1458771792 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f520415f0c0 con 0x7f51ec004850 2023-10-16T13:27:39.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.784+0000 7f5202ffd700 1 -- 172.21.15.16:0/1458771792 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51ec00ba10 con 0x7f51ec003a00 2023-10-16T13:27:39.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.784+0000 7f5202ffd700 1 -- 172.21.15.16:0/1458771792 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51ec00bb50 con 0x7f52040cf600 2023-10-16T13:27:39.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.784+0000 7f52037fe700 1 --2- 172.21.15.16:0/1458771792 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51ec004850 0x7f51ec004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:39.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.784+0000 7f5203fff700 1 --2- 172.21.15.16:0/1458771792 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f51ec003a00 0x7f51ec003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:39.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.784+0000 7f5208e1c700 1 --2- 172.21.15.16:0/1458771792 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f52040cf600 0x7f51ec005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:39.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.784+0000 7f52037fe700 1 -- 172.21.15.16:0/1458771792 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f52040cf600 msgr2=0x7f51ec005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:39.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.784+0000 7f52037fe700 1 --2- 172.21.15.16:0/1458771792 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f52040cf600 0x7f51ec005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:39.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.784+0000 7f52037fe700 1 -- 172.21.15.16:0/1458771792 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f51ec003a00 msgr2=0x7f51ec003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:39.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.784+0000 7f52037fe700 1 --2- 172.21.15.16:0/1458771792 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f51ec003a00 0x7f51ec003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:39.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.784+0000 7f52037fe700 1 -- 172.21.15.16:0/1458771792 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51ec0035b0 con 0x7f51ec004850 2023-10-16T13:27:39.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.784+0000 7f52037fe700 1 --2- 172.21.15.16:0/1458771792 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51ec004850 0x7f51ec004cd0 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7f51f800c350 tx=0x7f51ec005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=532cf4af0c35a380 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:39.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.784+0000 7f5202ffd700 1 -- 172.21.15.16:0/1458771792 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f800ce50 con 0x7f51ec004850 2023-10-16T13:27:39.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.785+0000 7f5202ffd700 1 -- 172.21.15.16:0/1458771792 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51f8005960 con 0x7f51ec004850 2023-10-16T13:27:39.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.785+0000 7f5202ffd700 1 -- 172.21.15.16:0/1458771792 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f8005d10 con 0x7f51ec004850 2023-10-16T13:27:39.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.785+0000 7f520a87f700 1 -- 172.21.15.16:0/1458771792 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51ec004850 msgr2=0x7f51ec004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:39.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.785+0000 7f520a87f700 1 --2- 172.21.15.16:0/1458771792 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51ec004850 0x7f51ec004cd0 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7f51f800c350 tx=0x7f51ec005880 comp rx=0 tx=0).stop 2023-10-16T13:27:39.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.785+0000 7f5208e1c700 1 -- 172.21.15.16:0/1458771792 reap_dead start 2023-10-16T13:27:39.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.785+0000 7f520a87f700 1 -- 172.21.15.16:0/1458771792 shutdown_connections 2023-10-16T13:27:39.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.785+0000 7f520a87f700 1 -- 172.21.15.16:0/1458771792 >> 172.21.15.16:0/1458771792 conn(0x7f520406f430 msgr2=0x7f520406f810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:39.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.785+0000 7f520a87f700 1 -- 172.21.15.16:0/1458771792 shutdown_connections 2023-10-16T13:27:39.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.785+0000 7f520a87f700 1 -- 172.21.15.16:0/1458771792 wait complete. 2023-10-16T13:27:39.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.786+0000 7f520a87f700 1 Processor -- start 2023-10-16T13:27:39.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.787+0000 7f520a87f700 1 -- start start 2023-10-16T13:27:39.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.787+0000 7f520a87f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f51ec004850 0x7f5204156e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:39.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.787+0000 7f520a87f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52040aede0 0x7f52040ad1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:39.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.787+0000 7f520a87f700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f52040ad730 0x7f52040c7b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:39.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.787+0000 7f520a87f700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52040c9dc0 con 0x7f52040aede0 2023-10-16T13:27:39.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.787+0000 7f520a87f700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52040add60 con 0x7f52040ad730 2023-10-16T13:27:39.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.787+0000 7f520a87f700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52040ae040 con 0x7f51ec004850 2023-10-16T13:27:39.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.787+0000 7f52037fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52040aede0 0x7f52040ad1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:39.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.787+0000 7f52037fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52040aede0 0x7f52040ad1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48388/0 (socket says 172.21.15.16:48388) 2023-10-16T13:27:39.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.787+0000 7f52037fe700 1 -- 172.21.15.16:0/3096748981 learned_addr learned my addr 172.21.15.16:0/3096748981 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:39.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.788+0000 7f5203fff700 1 --2- 172.21.15.16:0/3096748981 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f51ec004850 0x7f5204156e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:39.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.788+0000 7f5208e1c700 1 --2- 172.21.15.16:0/3096748981 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f52040ad730 0x7f52040c7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:39.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.788+0000 7f52037fe700 1 -- 172.21.15.16:0/3096748981 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f51ec004850 msgr2=0x7f5204156e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:39.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.788+0000 7f52037fe700 1 --2- 172.21.15.16:0/3096748981 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f51ec004850 0x7f5204156e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:39.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.788+0000 7f52037fe700 1 -- 172.21.15.16:0/3096748981 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f52040ad730 msgr2=0x7f52040c7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:39.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.788+0000 7f52037fe700 1 --2- 172.21.15.16:0/3096748981 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f52040ad730 0x7f52040c7b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:39.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.788+0000 7f52037fe700 1 -- 172.21.15.16:0/3096748981 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52040ae320 con 0x7f52040aede0 2023-10-16T13:27:39.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.788+0000 7f5203fff700 1 --2- 172.21.15.16:0/3096748981 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f51ec004850 0x7f5204156e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:27:39.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.788+0000 7f52037fe700 1 --2- 172.21.15.16:0/3096748981 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52040aede0 0x7f52040ad1f0 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7f51f8015040 tx=0x7f51f8001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:39.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.788+0000 7f5200ff9700 1 -- 172.21.15.16:0/3096748981 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f8001680 con 0x7f52040aede0 2023-10-16T13:27:39.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.788+0000 7f5200ff9700 1 -- 172.21.15.16:0/3096748981 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51f8004a80 con 0x7f52040aede0 2023-10-16T13:27:39.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.788+0000 7f520a87f700 1 -- 172.21.15.16:0/3096748981 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52040ae600 con 0x7f52040aede0 2023-10-16T13:27:39.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.788+0000 7f5200ff9700 1 -- 172.21.15.16:0/3096748981 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f8004e30 con 0x7f52040aede0 2023-10-16T13:27:39.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.788+0000 7f520a87f700 1 -- 172.21.15.16:0/3096748981 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5204157380 con 0x7f52040aede0 2023-10-16T13:27:39.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.789+0000 7f520a87f700 1 -- 172.21.15.16:0/3096748981 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5204157510 con 0x7f52040aede0 2023-10-16T13:27:39.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.789+0000 7f5200ff9700 1 -- 172.21.15.16:0/3096748981 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f51f80070d0 con 0x7f52040aede0 2023-10-16T13:27:39.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.790+0000 7f5200ff9700 1 --2- 172.21.15.16:0/3096748981 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f51f006b050 0x7f51f006d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:39.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.790+0000 7f5200ff9700 1 -- 172.21.15.16:0/3096748981 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 5277+0+0 (secure 0 0 0) 0x7f51f808a940 con 0x7f52040aede0 2023-10-16T13:27:39.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.790+0000 7f5203fff700 1 --2- 172.21.15.16:0/3096748981 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f51f006b050 0x7f51f006d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:39.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.790+0000 7f5203fff700 1 --2- 172.21.15.16:0/3096748981 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f51f006b050 0x7f51f006d510 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f51f4009c60 tx=0x7f51f4008040 comp rx=0 tx=0).ready entity=mgr.9510 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:39.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.794+0000 7f5200ff9700 1 -- 172.21.15.16:0/3096748981 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f51f8055be0 con 0x7f52040aede0 2023-10-16T13:27:40.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:39.998+0000 7f520a87f700 1 -- 172.21.15.16:0/3096748981 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f520404ec20 con 0x7f52040aede0 2023-10-16T13:27:40.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:40.123+0000 7f5200ff9700 1 -- 172.21.15.16:0/3096748981 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v174) v1 ==== 138+0+0 (secure 0 0 0) 0x7f51f8007da0 con 0x7f52040aede0 2023-10-16T13:27:40.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:40.127+0000 7f520a87f700 1 -- 172.21.15.16:0/3096748981 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f51f006b050 msgr2=0x7f51f006d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:40.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:40.127+0000 7f520a87f700 1 --2- 172.21.15.16:0/3096748981 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f51f006b050 0x7f51f006d510 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f51f4009c60 tx=0x7f51f4008040 comp rx=0 tx=0).stop 2023-10-16T13:27:40.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:40.127+0000 7f520a87f700 1 -- 172.21.15.16:0/3096748981 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52040aede0 msgr2=0x7f52040ad1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:40.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:40.127+0000 7f520a87f700 1 --2- 172.21.15.16:0/3096748981 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52040aede0 0x7f52040ad1f0 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7f51f8015040 tx=0x7f51f8001a00 comp rx=0 tx=0).stop 2023-10-16T13:27:40.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:40.128+0000 7f520a87f700 1 -- 172.21.15.16:0/3096748981 shutdown_connections 2023-10-16T13:27:40.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:40.128+0000 7f520a87f700 1 --2- 172.21.15.16:0/3096748981 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f51f006b050 0x7f51f006d510 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:40.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:40.128+0000 7f520a87f700 1 --2- 172.21.15.16:0/3096748981 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f52040ad730 0x7f52040c7b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:40.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:40.128+0000 7f520a87f700 1 --2- 172.21.15.16:0/3096748981 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f51ec004850 0x7f5204156e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:40.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:40.128+0000 7f520a87f700 1 --2- 172.21.15.16:0/3096748981 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52040aede0 0x7f52040ad1f0 unknown :-1 s=CLOSED pgs=1909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:40.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:40.128+0000 7f520a87f700 1 -- 172.21.15.16:0/3096748981 >> 172.21.15.16:0/3096748981 conn(0x7f520406f430 msgr2=0x7f52040cde90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:40.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:40.128+0000 7f520a87f700 1 -- 172.21.15.16:0/3096748981 shutdown_connections 2023-10-16T13:27:40.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:40.129+0000 7f520a87f700 1 -- 172.21.15.16:0/3096748981 wait complete. 2023-10-16T13:27:40.133 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' created 2023-10-16T13:27:40.144 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-16T13:27:40.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:40.274+0000 7fc1155c8700 1 Processor -- start 2023-10-16T13:27:40.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:40.274+0000 7fc1155c8700 1 -- start start 2023-10-16T13:27:40.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:40.275+0000 7fc1155c8700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fc11015ec80 0x7fc1100cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:40.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:40.275+0000 7fc1155c8700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc11015f060 0x7fc1100ce400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:40.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:40.275+0000 7fc1155c8700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1100b2440 con 0x7fc1100d2110 2023-10-16T13:27:40.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:40.275+0000 7fc1155c8700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1100b2c50 con 0x7fc11015f060 2023-10-16T13:27:40.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:40.275+0000 7fc1155c8700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1100d00e0 con 0x7fc11015ec80 2023-10-16T13:27:40.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:40.275+0000 7fc10f7fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc1100d2110 0x7fc1100b4440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37670/0 (socket says 172.21.15.16:37670) 2023-10-16T13:27:40.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:40.275+0000 7fc10e7fc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc11015f060 0x7fc1100ce400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:40.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:40.275+0000 7fc10f7fe700 1 -- 172.21.15.16:0/3460799609 learned_addr learned my addr 172.21.15.16:0/3460799609 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:27:40.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:40.276+0000 7fc10effd700 1 --2- 172.21.15.16:0/3460799609 >> v2:172.21.15.49:3300/0 conn(0x7fc11015ec80 0x7fc1100cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:40.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:40.276+0000 7fc10dffb700 1 -- 172.21.15.16:0/3460799609 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3655880061 0 0) 0x7fc1100b2440 con 0x7fc1100d2110 2023-10-16T13:27:40.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:40.276+0000 7fc10dffb700 1 -- 172.21.15.16:0/3460799609 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0f80035b0 con 0x7fc1100d2110 2023-10-16T13:27:40.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:40.276+0000 7fc10e7fc700 1 -- 172.21.15.16:0/3460799609 >> v2:172.21.15.49:3300/0 conn(0x7fc11015ec80 msgr2=0x7fc1100cdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:40.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:40.276+0000 7fc10e7fc700 1 --2- 172.21.15.16:0/3460799609 >> v2:172.21.15.49:3300/0 conn(0x7fc11015ec80 0x7fc1100cdec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:40.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:40.276+0000 7fc10e7fc700 1 -- 172.21.15.16:0/3460799609 >> v1:172.21.15.16:6789/0 conn(0x7fc1100d2110 legacy=0x7fc1100b4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:40.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:40.276+0000 7fc10e7fc700 1 -- 172.21.15.16:0/3460799609 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1100cef60 con 0x7fc11015f060 2023-10-16T13:27:40.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:40.276+0000 7fc10e7fc700 1 --2- 172.21.15.16:0/3460799609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc11015f060 0x7fc1100ce400 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7fc110067b20 tx=0x7fc100009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2319902e3a68e6f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:40.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:40.276+0000 7fc10dffb700 1 -- 172.21.15.16:0/3460799609 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc100011830 con 0x7fc11015f060 2023-10-16T13:27:40.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:40.276+0000 7fc10dffb700 1 -- 172.21.15.16:0/3460799609 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc100011990 con 0x7fc11015f060 2023-10-16T13:27:40.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:40.276+0000 7fc10dffb700 1 -- 172.21.15.16:0/3460799609 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc100011d00 con 0x7fc11015f060 2023-10-16T13:27:40.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:40.277+0000 7fc1155c8700 1 -- 172.21.15.16:0/3460799609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc11015f060 msgr2=0x7fc1100ce400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:40.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:40.277+0000 7fc1155c8700 1 --2- 172.21.15.16:0/3460799609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc11015f060 0x7fc1100ce400 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7fc110067b20 tx=0x7fc100009b20 comp rx=0 tx=0).stop 2023-10-16T13:27:40.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:40.277+0000 7fc1155c8700 1 -- 172.21.15.16:0/3460799609 shutdown_connections 2023-10-16T13:27:40.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:40.277+0000 7fc1155c8700 1 --2- 172.21.15.16:0/3460799609 >> v2:172.21.15.49:3300/0 conn(0x7fc11015ec80 0x7fc1100cdec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:40.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:40.277+0000 7fc1155c8700 1 --2- 172.21.15.16:0/3460799609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc11015f060 0x7fc1100ce400 unknown :-1 s=CLOSED pgs=1911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:40.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:40.277+0000 7fc1155c8700 1 -- 172.21.15.16:0/3460799609 >> 172.21.15.16:0/3460799609 conn(0x7fc1100d15f0 msgr2=0x7fc1100d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:40.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:40.277+0000 7fc1155c8700 1 -- 172.21.15.16:0/3460799609 shutdown_connections 2023-10-16T13:27:40.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:40.277+0000 7fc1155c8700 1 -- 172.21.15.16:0/3460799609 wait complete. 2023-10-16T13:27:40.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:40.278+0000 7fc1155c8700 1 Processor -- start 2023-10-16T13:27:40.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:40.278+0000 7fc1155c8700 1 -- start start 2023-10-16T13:27:40.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:40.279+0000 7fc1155c8700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc1100d2110 0x7fc1101f5860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:40.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:40.279+0000 7fc1155c8700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc11015ec80 0x7fc1101f2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:40.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:40.279+0000 7fc1155c8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc11015f060 0x7fc1100c7090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:40.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:40.279+0000 7fc1155c8700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1100c6960 con 0x7fc11015f060 2023-10-16T13:27:40.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:40.279+0000 7fc1155c8700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1100c8860 con 0x7fc1100d2110 2023-10-16T13:27:40.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:40.279+0000 7fc1155c8700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1100c8b40 con 0x7fc11015ec80 2023-10-16T13:27:40.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:40.279+0000 7fc10f7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc11015f060 0x7fc1100c7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:40.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:40.279+0000 7fc10f7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc11015f060 0x7fc1100c7090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48402/0 (socket says 172.21.15.16:48402) 2023-10-16T13:27:40.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:40.279+0000 7fc10f7fe700 1 -- 172.21.15.16:0/117761122 learned_addr learned my addr 172.21.15.16:0/117761122 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:40.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:40.279+0000 7fc10effd700 1 --2- 172.21.15.16:0/117761122 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc1100d2110 0x7fc1101f5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:40.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:40.279+0000 7fc10e7fc700 1 --2- 172.21.15.16:0/117761122 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc11015ec80 0x7fc1101f2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:40.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:40.279+0000 7fc10f7fe700 1 -- 172.21.15.16:0/117761122 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc11015ec80 msgr2=0x7fc1101f2ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:40.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:40.279+0000 7fc10f7fe700 1 --2- 172.21.15.16:0/117761122 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc11015ec80 0x7fc1101f2ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:40.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:40.279+0000 7fc10f7fe700 1 -- 172.21.15.16:0/117761122 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc1100d2110 msgr2=0x7fc1101f5860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:40.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:40.279+0000 7fc10f7fe700 1 --2- 172.21.15.16:0/117761122 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc1100d2110 0x7fc1101f5860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:40.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:40.279+0000 7fc10f7fe700 1 -- 172.21.15.16:0/117761122 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1000094a0 con 0x7fc11015f060 2023-10-16T13:27:40.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:40.279+0000 7fc10f7fe700 1 --2- 172.21.15.16:0/117761122 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc11015f060 0x7fc1100c7090 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7fc10800cb40 tx=0x7fc10800ce50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:40.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:40.280+0000 7fc0f77fe700 1 -- 172.21.15.16:0/117761122 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc10800a9e0 con 0x7fc11015f060 2023-10-16T13:27:40.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:40.280+0000 7fc0f77fe700 1 -- 172.21.15.16:0/117761122 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc10800ab40 con 0x7fc11015f060 2023-10-16T13:27:40.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:40.280+0000 7fc1155c8700 1 -- 172.21.15.16:0/117761122 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1100c8e20 con 0x7fc11015f060 2023-10-16T13:27:40.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:40.280+0000 7fc0f77fe700 1 -- 172.21.15.16:0/117761122 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc108012400 con 0x7fc11015f060 2023-10-16T13:27:40.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:40.280+0000 7fc1155c8700 1 -- 172.21.15.16:0/117761122 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1101f5da0 con 0x7fc11015f060 2023-10-16T13:27:40.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:40.281+0000 7fc1155c8700 1 -- 172.21.15.16:0/117761122 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1100c9340 con 0x7fc11015f060 2023-10-16T13:27:40.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:40.281+0000 7fc0f77fe700 1 -- 172.21.15.16:0/117761122 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc10801b3b0 con 0x7fc11015f060 2023-10-16T13:27:40.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:40.282+0000 7fc0f77fe700 1 --2- 172.21.15.16:0/117761122 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7fc0fc06b020 0x7fc0fc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:40.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:40.282+0000 7fc0f77fe700 1 -- 172.21.15.16:0/117761122 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 5647+0+0 (secure 0 0 0) 0x7fc108089720 con 0x7fc11015f060 2023-10-16T13:27:40.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:40.282+0000 7fc10effd700 1 --2- 172.21.15.16:0/117761122 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7fc0fc06b020 0x7fc0fc06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:40.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:40.282+0000 7fc10effd700 1 --2- 172.21.15.16:0/117761122 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7fc0fc06b020 0x7fc0fc06d4e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc1100c6cf0 tx=0x7fc104008040 comp rx=0 tx=0).ready entity=mgr.9510 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:40.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:40.286+0000 7fc0f77fe700 1 -- 172.21.15.16:0/117761122 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc108054890 con 0x7fc11015f060 2023-10-16T13:27:40.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:40.490+0000 7fc1155c8700 1 -- 172.21.15.16:0/117761122 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fc11004ebb0 con 0x7fc11015f060 2023-10-16T13:27:41.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.129+0000 7fc0f77fe700 1 -- 172.21.15.16:0/117761122 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v175) v1 ==== 170+0+0 (secure 0 0 0) 0x7fc108057eb0 con 0x7fc11015f060 2023-10-16T13:27:41.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.134+0000 7fc1155c8700 1 -- 172.21.15.16:0/117761122 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7fc0fc06b020 msgr2=0x7fc0fc06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:41.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.134+0000 7fc1155c8700 1 --2- 172.21.15.16:0/117761122 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7fc0fc06b020 0x7fc0fc06d4e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc1100c6cf0 tx=0x7fc104008040 comp rx=0 tx=0).stop 2023-10-16T13:27:41.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.134+0000 7fc1155c8700 1 -- 172.21.15.16:0/117761122 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc11015f060 msgr2=0x7fc1100c7090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:41.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.134+0000 7fc1155c8700 1 --2- 172.21.15.16:0/117761122 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc11015f060 0x7fc1100c7090 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7fc10800cb40 tx=0x7fc10800ce50 comp rx=0 tx=0).stop 2023-10-16T13:27:41.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.135+0000 7fc1155c8700 1 -- 172.21.15.16:0/117761122 shutdown_connections 2023-10-16T13:27:41.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.135+0000 7fc1155c8700 1 --2- 172.21.15.16:0/117761122 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc11015f060 0x7fc1100c7090 unknown :-1 s=CLOSED pgs=1912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:41.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.135+0000 7fc1155c8700 1 --2- 172.21.15.16:0/117761122 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7fc0fc06b020 0x7fc0fc06d4e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:41.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.135+0000 7fc1155c8700 1 --2- 172.21.15.16:0/117761122 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc1100d2110 0x7fc1101f5860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:41.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.135+0000 7fc1155c8700 1 --2- 172.21.15.16:0/117761122 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc11015ec80 0x7fc1101f2ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:41.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.135+0000 7fc1155c8700 1 -- 172.21.15.16:0/117761122 >> 172.21.15.16:0/117761122 conn(0x7fc1100d15f0 msgr2=0x7fc1100b6180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:41.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.136+0000 7fc1155c8700 1 -- 172.21.15.16:0/117761122 shutdown_connections 2023-10-16T13:27:41.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.137+0000 7fc1155c8700 1 -- 172.21.15.16:0/117761122 wait complete. 2023-10-16T13:27:41.146 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' created 2023-10-16T13:27:41.165 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-16T13:27:41.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.296+0000 7fd8b5610700 1 Processor -- start 2023-10-16T13:27:41.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.296+0000 7fd8b5610700 1 -- start start 2023-10-16T13:27:41.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.297+0000 7fd8b5610700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fd8b00ce6a0 0x7fd8b006c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:41.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.297+0000 7fd8b5610700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8b00b20f0 con 0x7fd8b00d2110 2023-10-16T13:27:41.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.297+0000 7fd8b5610700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8b00af1f0 con 0x7fd8b00b2620 2023-10-16T13:27:41.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.297+0000 7fd8b5610700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8b00b1d70 con 0x7fd8b00ce6a0 2023-10-16T13:27:41.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.297+0000 7fd8ae7fc700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd8b00d2110 0x7fd8b00b0f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37680/0 (socket says 172.21.15.16:37680) 2023-10-16T13:27:41.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.297+0000 7fd8ae7fc700 1 -- 172.21.15.16:0/1872264480 learned_addr learned my addr 172.21.15.16:0/1872264480 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:27:41.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.298+0000 7fd8adffb700 1 -- 172.21.15.16:0/1872264480 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 662596560 0 0) 0x7fd8b00b20f0 con 0x7fd8b00d2110 2023-10-16T13:27:41.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.298+0000 7fd8aeffd700 1 --2- 172.21.15.16:0/1872264480 >> v2:172.21.15.49:3300/0 conn(0x7fd8b00ce6a0 0x7fd8b006c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:41.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.298+0000 7fd8adffb700 1 -- 172.21.15.16:0/1872264480 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8980035b0 con 0x7fd8b00d2110 2023-10-16T13:27:41.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.298+0000 7fd8adffb700 1 -- 172.21.15.16:0/1872264480 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3988364361 0 0) 0x7fd8980035b0 con 0x7fd8b00d2110 2023-10-16T13:27:41.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.298+0000 7fd8adffb700 1 -- 172.21.15.16:0/1872264480 >> v2:172.21.15.49:3300/0 conn(0x7fd8b00ce6a0 msgr2=0x7fd8b006c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:41.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.298+0000 7fd8adffb700 1 --2- 172.21.15.16:0/1872264480 >> v2:172.21.15.49:3300/0 conn(0x7fd8b00ce6a0 0x7fd8b006c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:41.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.298+0000 7fd8adffb700 1 -- 172.21.15.16:0/1872264480 >> v1:172.21.15.49:6789/0 conn(0x7fd8b00b2620 legacy=0x7fd8b00b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:41.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.298+0000 7fd8adffb700 1 -- 172.21.15.16:0/1872264480 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8b00d03d0 con 0x7fd8b00d2110 2023-10-16T13:27:41.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.298+0000 7fd8adffb700 1 -- 172.21.15.16:0/1872264480 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7fd8a0001670 con 0x7fd8b00d2110 2023-10-16T13:27:41.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.298+0000 7fd8adffb700 1 -- 172.21.15.16:0/1872264480 >> v1:172.21.15.16:6789/0 conn(0x7fd8b00d2110 legacy=0x7fd8b00b0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:41.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.298+0000 7fd8adffb700 1 --2- 172.21.15.16:0/1872264480 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd898003a00 0x7fd898003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:41.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.298+0000 7fd8adffb700 1 --2- 172.21.15.16:0/1872264480 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd898004850 0x7fd898004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:41.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.298+0000 7fd8adffb700 1 --2- 172.21.15.16:0/1872264480 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd898005210 0x7fd89800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:41.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.298+0000 7fd8adffb700 1 -- 172.21.15.16:0/1872264480 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b00b1d70 con 0x7fd898003a00 2023-10-16T13:27:41.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.298+0000 7fd8adffb700 1 -- 172.21.15.16:0/1872264480 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd89800c0b0 con 0x7fd898004850 2023-10-16T13:27:41.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.298+0000 7fd8adffb700 1 -- 172.21.15.16:0/1872264480 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd89800c270 con 0x7fd898005210 2023-10-16T13:27:41.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.298+0000 7fd8aeffd700 1 --2- 172.21.15.16:0/1872264480 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd898003a00 0x7fd898003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:41.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.298+0000 7fd8ae7fc700 1 --2- 172.21.15.16:0/1872264480 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd898004850 0x7fd898004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:41.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.298+0000 7fd8af7fe700 1 --2- 172.21.15.16:0/1872264480 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd898005210 0x7fd89800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:41.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.299+0000 7fd8aeffd700 1 -- 172.21.15.16:0/1872264480 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd898005210 msgr2=0x7fd89800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:41.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.299+0000 7fd8aeffd700 1 --2- 172.21.15.16:0/1872264480 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd898005210 0x7fd89800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:41.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.299+0000 7fd8aeffd700 1 -- 172.21.15.16:0/1872264480 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd898004850 msgr2=0x7fd898004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:41.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.299+0000 7fd8aeffd700 1 --2- 172.21.15.16:0/1872264480 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd898004850 0x7fd898004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:41.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.299+0000 7fd8aeffd700 1 -- 172.21.15.16:0/1872264480 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8980035b0 con 0x7fd898003a00 2023-10-16T13:27:41.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.299+0000 7fd8af7fe700 1 --2- 172.21.15.16:0/1872264480 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd898005210 0x7fd89800ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:27:41.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.299+0000 7fd8af7fe700 1 -- 172.21.15.16:0/1872264480 reap_dead start 2023-10-16T13:27:41.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.299+0000 7fd8aeffd700 1 --2- 172.21.15.16:0/1872264480 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd898003a00 0x7fd898003e20 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7fd8a4009b60 tx=0x7fd898005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34e10a22f13d90a8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:41.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.299+0000 7fd8adffb700 1 -- 172.21.15.16:0/1872264480 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a400e6c0 con 0x7fd898003a00 2023-10-16T13:27:41.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.299+0000 7fd8adffb700 1 -- 172.21.15.16:0/1872264480 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8a400e820 con 0x7fd898003a00 2023-10-16T13:27:41.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.299+0000 7fd8adffb700 1 -- 172.21.15.16:0/1872264480 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a400eb90 con 0x7fd898003a00 2023-10-16T13:27:41.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.299+0000 7fd8b5610700 1 -- 172.21.15.16:0/1872264480 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd898003a00 msgr2=0x7fd898003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:41.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.299+0000 7fd8b5610700 1 --2- 172.21.15.16:0/1872264480 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd898003a00 0x7fd898003e20 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7fd8a4009b60 tx=0x7fd898005880 comp rx=0 tx=0).stop 2023-10-16T13:27:41.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.299+0000 7fd8b5610700 1 -- 172.21.15.16:0/1872264480 shutdown_connections 2023-10-16T13:27:41.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.299+0000 7fd8b5610700 1 --2- 172.21.15.16:0/1872264480 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd898003a00 0x7fd898003e20 unknown :-1 s=CLOSED pgs=1914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:41.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.299+0000 7fd8b5610700 1 -- 172.21.15.16:0/1872264480 >> 172.21.15.16:0/1872264480 conn(0x7fd8b00d15f0 msgr2=0x7fd8b00d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:41.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.300+0000 7fd8b5610700 1 -- 172.21.15.16:0/1872264480 shutdown_connections 2023-10-16T13:27:41.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.300+0000 7fd8b5610700 1 -- 172.21.15.16:0/1872264480 wait complete. 2023-10-16T13:27:41.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.301+0000 7fd8b5610700 1 Processor -- start 2023-10-16T13:27:41.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.301+0000 7fd8b5610700 1 -- start start 2023-10-16T13:27:41.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.302+0000 7fd8b5610700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd898003a00 0x7fd8b014f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:41.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.302+0000 7fd8b5610700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd8b01f6fd0 0x7fd8b00c5120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:41.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.302+0000 7fd8b5610700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd8b00c5660 0x7fd8b0152120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:41.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.302+0000 7fd8b5610700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b00c5ae0 con 0x7fd898003a00 2023-10-16T13:27:41.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.302+0000 7fd8b5610700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b00c70a0 con 0x7fd8b01f6fd0 2023-10-16T13:27:41.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.302+0000 7fd8b5610700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b014fd60 con 0x7fd8b00c5660 2023-10-16T13:27:41.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.302+0000 7fd8aeffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd898003a00 0x7fd8b014f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:41.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.302+0000 7fd8aeffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd898003a00 0x7fd8b014f820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48418/0 (socket says 172.21.15.16:48418) 2023-10-16T13:27:41.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.302+0000 7fd8aeffd700 1 -- 172.21.15.16:0/2910869180 learned_addr learned my addr 172.21.15.16:0/2910869180 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:41.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.302+0000 7fd8ae7fc700 1 --2- 172.21.15.16:0/2910869180 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd8b01f6fd0 0x7fd8b00c5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:41.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.302+0000 7fd8af7fe700 1 --2- 172.21.15.16:0/2910869180 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd8b00c5660 0x7fd8b0152120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:41.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.302+0000 7fd8aeffd700 1 -- 172.21.15.16:0/2910869180 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd8b00c5660 msgr2=0x7fd8b0152120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:41.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.302+0000 7fd8aeffd700 1 --2- 172.21.15.16:0/2910869180 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd8b00c5660 0x7fd8b0152120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:41.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.302+0000 7fd8aeffd700 1 -- 172.21.15.16:0/2910869180 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd8b01f6fd0 msgr2=0x7fd8b00c5120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:41.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.302+0000 7fd8aeffd700 1 --2- 172.21.15.16:0/2910869180 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd8b01f6fd0 0x7fd8b00c5120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:41.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.302+0000 7fd8aeffd700 1 -- 172.21.15.16:0/2910869180 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8b00c72d0 con 0x7fd898003a00 2023-10-16T13:27:41.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.302+0000 7fd8aeffd700 1 --2- 172.21.15.16:0/2910869180 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd898003a00 0x7fd8b014f820 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7fd89800c080 tx=0x7fd8a40047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:41.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.302+0000 7fd8977fe700 1 -- 172.21.15.16:0/2910869180 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a400e6c0 con 0x7fd898003a00 2023-10-16T13:27:41.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.303+0000 7fd8977fe700 1 -- 172.21.15.16:0/2910869180 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8a4003ad0 con 0x7fd898003a00 2023-10-16T13:27:41.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.303+0000 7fd8b5610700 1 -- 172.21.15.16:0/2910869180 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8b00c75b0 con 0x7fd898003a00 2023-10-16T13:27:41.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.303+0000 7fd8977fe700 1 -- 172.21.15.16:0/2910869180 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a4003e80 con 0x7fd898003a00 2023-10-16T13:27:41.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.303+0000 7fd8b5610700 1 -- 172.21.15.16:0/2910869180 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8b00c7b70 con 0x7fd898003a00 2023-10-16T13:27:41.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.303+0000 7fd8b5610700 1 -- 172.21.15.16:0/2910869180 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8b00c81c0 con 0x7fd898003a00 2023-10-16T13:27:41.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.304+0000 7fd8977fe700 1 -- 172.21.15.16:0/2910869180 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd8a4013470 con 0x7fd898003a00 2023-10-16T13:27:41.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.305+0000 7fd8977fe700 1 --2- 172.21.15.16:0/2910869180 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7fd89c06b020 0x7fd89c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:41.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.305+0000 7fd8977fe700 1 -- 172.21.15.16:0/2910869180 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fd8a408b910 con 0x7fd898003a00 2023-10-16T13:27:41.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.305+0000 7fd8ae7fc700 1 --2- 172.21.15.16:0/2910869180 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7fd89c06b020 0x7fd89c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:41.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.305+0000 7fd8ae7fc700 1 --2- 172.21.15.16:0/2910869180 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7fd89c06b020 0x7fd89c06d4e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd8a0001050 tx=0x7fd8a0009040 comp rx=0 tx=0).ready entity=mgr.9510 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:41.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.309+0000 7fd8977fe700 1 -- 172.21.15.16:0/2910869180 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd8a4056900 con 0x7fd898003a00 2023-10-16T13:27:41.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:41.560+0000 7fd8b5610700 1 -- 172.21.15.16:0/2910869180 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fd8b00c5a40 con 0x7fd898003a00 2023-10-16T13:27:42.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.129+0000 7fd8977fe700 1 -- 172.21.15.16:0/2910869180 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 21 and data pool 22 v82) v1 ==== 174+0+0 (secure 0 0 0) 0x7fd8a40170c0 con 0x7fd898003a00 2023-10-16T13:27:42.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.132+0000 7fd8b5610700 1 -- 172.21.15.16:0/2910869180 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7fd89c06b020 msgr2=0x7fd89c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:42.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.132+0000 7fd8b5610700 1 --2- 172.21.15.16:0/2910869180 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7fd89c06b020 0x7fd89c06d4e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd8a0001050 tx=0x7fd8a0009040 comp rx=0 tx=0).stop 2023-10-16T13:27:42.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.132+0000 7fd8b5610700 1 -- 172.21.15.16:0/2910869180 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd898003a00 msgr2=0x7fd8b014f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:42.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.132+0000 7fd8b5610700 1 --2- 172.21.15.16:0/2910869180 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd898003a00 0x7fd8b014f820 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7fd89800c080 tx=0x7fd8a40047c0 comp rx=0 tx=0).stop 2023-10-16T13:27:42.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.132+0000 7fd8b5610700 1 -- 172.21.15.16:0/2910869180 shutdown_connections 2023-10-16T13:27:42.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.132+0000 7fd8b5610700 1 --2- 172.21.15.16:0/2910869180 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd8b00c5660 0x7fd8b0152120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:42.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.132+0000 7fd8b5610700 1 --2- 172.21.15.16:0/2910869180 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd898003a00 0x7fd8b014f820 unknown :-1 s=CLOSED pgs=1915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:42.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.132+0000 7fd8b5610700 1 --2- 172.21.15.16:0/2910869180 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7fd89c06b020 0x7fd89c06d4e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:42.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.132+0000 7fd8b5610700 1 --2- 172.21.15.16:0/2910869180 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd8b01f6fd0 0x7fd8b00c5120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:42.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.132+0000 7fd8b5610700 1 -- 172.21.15.16:0/2910869180 >> 172.21.15.16:0/2910869180 conn(0x7fd8b00d15f0 msgr2=0x7fd8b00b0720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:42.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.132+0000 7fd8b5610700 1 -- 172.21.15.16:0/2910869180 shutdown_connections 2023-10-16T13:27:42.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.132+0000 7fd8b5610700 1 -- 172.21.15.16:0/2910869180 wait complete. 2023-10-16T13:27:42.139 INFO:teuthology.orchestra.run.smithi016.stderr:new fs with metadata pool 21 and data pool 22 2023-10-16T13:27:42.155 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:27:42.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.280+0000 7f2aa88e6700 1 Processor -- start 2023-10-16T13:27:42.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.281+0000 7f2aa88e6700 1 -- start start 2023-10-16T13:27:42.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.282+0000 7f2aa88e6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2aa40cf190 0x7f2aa415d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:42.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.282+0000 7f2aa88e6700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2aa4066780 con 0x7f2aa40cabf0 2023-10-16T13:27:42.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.282+0000 7f2aa88e6700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2aa40cf650 con 0x7f2aa415bc70 2023-10-16T13:27:42.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.282+0000 7f2aa88e6700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2aa40cf930 con 0x7f2aa40cf190 2023-10-16T13:27:42.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.282+0000 7f2aa2d9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2aa40cf190 0x7f2aa415d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:42.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.282+0000 7f2aa1d9b700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2aa40cabf0 0x7f2aa406b460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37700/0 (socket says 172.21.15.16:37700) 2023-10-16T13:27:42.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.282+0000 7f2aa1d9b700 1 -- 172.21.15.16:0/196043429 learned_addr learned my addr 172.21.15.16:0/196043429 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:27:42.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.282+0000 7f2aa2d9d700 1 -- 172.21.15.16:0/196043429 >> v1:172.21.15.49:6789/0 conn(0x7f2aa415bc70 legacy=0x7f2aa40ca4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:42.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.282+0000 7f2aa159a700 1 -- 172.21.15.16:0/196043429 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1524045028 0 0) 0x7f2aa4066780 con 0x7f2aa40cabf0 2023-10-16T13:27:42.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.282+0000 7f2aa2d9d700 1 -- 172.21.15.16:0/196043429 >> v1:172.21.15.16:6789/0 conn(0x7f2aa40cabf0 legacy=0x7f2aa406b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:42.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.282+0000 7f2aa2d9d700 1 -- 172.21.15.16:0/196043429 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aa406a980 con 0x7f2aa40cf190 2023-10-16T13:27:42.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.282+0000 7f2aa2d9d700 1 --2- 172.21.15.16:0/196043429 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2aa40cf190 0x7f2aa415d250 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c009d90 tx=0x7f2a9c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76da515192b9f837 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:42.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.282+0000 7f2aa159a700 1 -- 172.21.15.16:0/196043429 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a9c00e070 con 0x7f2aa40cf190 2023-10-16T13:27:42.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.282+0000 7f2aa159a700 1 -- 172.21.15.16:0/196043429 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a9c00bbd0 con 0x7f2aa40cf190 2023-10-16T13:27:42.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.282+0000 7f2aa159a700 1 -- 172.21.15.16:0/196043429 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a9c012430 con 0x7f2aa40cf190 2023-10-16T13:27:42.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.283+0000 7f2aa88e6700 1 -- 172.21.15.16:0/196043429 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2aa40cf190 msgr2=0x7f2aa415d250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:42.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.283+0000 7f2aa88e6700 1 --2- 172.21.15.16:0/196043429 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2aa40cf190 0x7f2aa415d250 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c009d90 tx=0x7f2a9c00b330 comp rx=0 tx=0).stop 2023-10-16T13:27:42.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.283+0000 7f2aa88e6700 1 -- 172.21.15.16:0/196043429 shutdown_connections 2023-10-16T13:27:42.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.283+0000 7f2aa88e6700 1 --2- 172.21.15.16:0/196043429 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2aa40cf190 0x7f2aa415d250 unknown :-1 s=CLOSED pgs=1917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:42.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.283+0000 7f2aa88e6700 1 -- 172.21.15.16:0/196043429 >> 172.21.15.16:0/196043429 conn(0x7f2aa40cc5e0 msgr2=0x7f2aa40cc9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:42.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.283+0000 7f2aa88e6700 1 -- 172.21.15.16:0/196043429 shutdown_connections 2023-10-16T13:27:42.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.283+0000 7f2aa88e6700 1 -- 172.21.15.16:0/196043429 wait complete. 2023-10-16T13:27:42.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.284+0000 7f2aa88e6700 1 Processor -- start 2023-10-16T13:27:42.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.284+0000 7f2aa88e6700 1 -- start start 2023-10-16T13:27:42.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.285+0000 7f2aa88e6700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2aa40cabf0 0x7f2aa414ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:42.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.285+0000 7f2aa88e6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2aa40cf190 0x7f2aa40af410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:42.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.285+0000 7f2aa88e6700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2aa415bc70 0x7f2aa40af950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:42.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.285+0000 7f2aa88e6700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa41f4290 con 0x7f2aa40cf190 2023-10-16T13:27:42.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.285+0000 7f2aa88e6700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa406e2f0 con 0x7f2aa415bc70 2023-10-16T13:27:42.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.285+0000 7f2aa88e6700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa406e5d0 con 0x7f2aa40cabf0 2023-10-16T13:27:42.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.285+0000 7f2aa1d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2aa40cf190 0x7f2aa40af410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:42.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.285+0000 7f2aa1d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2aa40cf190 0x7f2aa40af410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48434/0 (socket says 172.21.15.16:48434) 2023-10-16T13:27:42.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.285+0000 7f2aa1d9b700 1 -- 172.21.15.16:0/3913903798 learned_addr learned my addr 172.21.15.16:0/3913903798 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:42.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.285+0000 7f2aa2d9d700 1 --2- 172.21.15.16:0/3913903798 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2aa415bc70 0x7f2aa40af950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:42.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.285+0000 7f2aa259c700 1 --2- 172.21.15.16:0/3913903798 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2aa40cabf0 0x7f2aa414ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:42.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.286+0000 7f2aa1d9b700 1 -- 172.21.15.16:0/3913903798 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2aa40cabf0 msgr2=0x7f2aa414ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:42.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.286+0000 7f2aa1d9b700 1 --2- 172.21.15.16:0/3913903798 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2aa40cabf0 0x7f2aa414ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:42.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.286+0000 7f2aa1d9b700 1 -- 172.21.15.16:0/3913903798 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2aa415bc70 msgr2=0x7f2aa40af950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:42.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.286+0000 7f2aa1d9b700 1 --2- 172.21.15.16:0/3913903798 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2aa415bc70 0x7f2aa40af950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:42.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.286+0000 7f2aa1d9b700 1 -- 172.21.15.16:0/3913903798 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a9c009a70 con 0x7f2aa40cf190 2023-10-16T13:27:42.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.286+0000 7f2aa259c700 1 --2- 172.21.15.16:0/3913903798 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2aa40cabf0 0x7f2aa414ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:27:42.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.286+0000 7f2aa1d9b700 1 --2- 172.21.15.16:0/3913903798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2aa40cf190 0x7f2aa40af410 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7f2a9800c520 tx=0x7f2a9800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:42.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.286+0000 7f2a92ffd700 1 -- 172.21.15.16:0/3913903798 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a98004c80 con 0x7f2aa40cf190 2023-10-16T13:27:42.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.286+0000 7f2a92ffd700 1 -- 172.21.15.16:0/3913903798 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a98004de0 con 0x7f2aa40cf190 2023-10-16T13:27:42.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.286+0000 7f2aa88e6700 1 -- 172.21.15.16:0/3913903798 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2aa406e8b0 con 0x7f2aa40cf190 2023-10-16T13:27:42.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.286+0000 7f2a92ffd700 1 -- 172.21.15.16:0/3913903798 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a98007310 con 0x7f2aa40cf190 2023-10-16T13:27:42.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.286+0000 7f2aa88e6700 1 -- 172.21.15.16:0/3913903798 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2aa414f4a0 con 0x7f2aa40cf190 2023-10-16T13:27:42.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.287+0000 7f2aa88e6700 1 -- 172.21.15.16:0/3913903798 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2aa41f4460 con 0x7f2aa40cf190 2023-10-16T13:27:42.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.287+0000 7f2a92ffd700 1 -- 172.21.15.16:0/3913903798 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2a98007470 con 0x7f2aa40cf190 2023-10-16T13:27:42.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.288+0000 7f2a92ffd700 1 --2- 172.21.15.16:0/3913903798 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f2a8406b020 0x7f2a8406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:42.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.288+0000 7f2a92ffd700 1 -- 172.21.15.16:0/3913903798 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f2a98089580 con 0x7f2aa40cf190 2023-10-16T13:27:42.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.288+0000 7f2aa259c700 1 --2- 172.21.15.16:0/3913903798 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f2a8406b020 0x7f2a8406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:42.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.288+0000 7f2aa259c700 1 --2- 172.21.15.16:0/3913903798 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f2a8406b020 0x7f2a8406d4e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2aa40b4f50 tx=0x7f2a94009040 comp rx=0 tx=0).ready entity=mgr.9510 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:42.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.293+0000 7f2a92ffd700 1 -- 172.21.15.16:0/3913903798 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2a98054370 con 0x7f2aa40cf190 2023-10-16T13:27:42.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.495+0000 7f2aa88e6700 1 -- 172.21.15.16:0/3913903798 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2aa404ebb0 con 0x7f2aa40cf190 2023-10-16T13:27:42.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.496+0000 7f2a92ffd700 1 -- 172.21.15.16:0/3913903798 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v176) v1 ==== 74+0+16788 (secure 0 0 0) 0x7f2a98057990 con 0x7f2aa40cf190 2023-10-16T13:27:42.500 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:27:42.501 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":176,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:27:41.561488+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2023-10-16T13:27:40.032163+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"176","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2023-10-16T13:27:40.523975+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"176","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:27:42.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.498+0000 7f2aa88e6700 1 -- 172.21.15.16:0/3913903798 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f2a8406b020 msgr2=0x7f2a8406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:42.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.498+0000 7f2aa88e6700 1 --2- 172.21.15.16:0/3913903798 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f2a8406b020 0x7f2a8406d4e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2aa40b4f50 tx=0x7f2a94009040 comp rx=0 tx=0).stop 2023-10-16T13:27:42.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.498+0000 7f2aa88e6700 1 -- 172.21.15.16:0/3913903798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2aa40cf190 msgr2=0x7f2aa40af410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:42.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.498+0000 7f2aa88e6700 1 --2- 172.21.15.16:0/3913903798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2aa40cf190 0x7f2aa40af410 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7f2a9800c520 tx=0x7f2a9800c830 comp rx=0 tx=0).stop 2023-10-16T13:27:42.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.498+0000 7f2aa88e6700 1 -- 172.21.15.16:0/3913903798 shutdown_connections 2023-10-16T13:27:42.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.498+0000 7f2aa88e6700 1 --2- 172.21.15.16:0/3913903798 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f2a8406b020 0x7f2a8406d4e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:42.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.498+0000 7f2aa88e6700 1 --2- 172.21.15.16:0/3913903798 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2aa415bc70 0x7f2aa40af950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:42.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.498+0000 7f2aa88e6700 1 --2- 172.21.15.16:0/3913903798 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2aa40cabf0 0x7f2aa414ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:42.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.498+0000 7f2aa88e6700 1 --2- 172.21.15.16:0/3913903798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2aa40cf190 0x7f2aa40af410 unknown :-1 s=CLOSED pgs=1918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:42.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.498+0000 7f2aa88e6700 1 -- 172.21.15.16:0/3913903798 >> 172.21.15.16:0/3913903798 conn(0x7f2aa40cc5e0 msgr2=0x7f2aa40cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:42.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.499+0000 7f2aa88e6700 1 -- 172.21.15.16:0/3913903798 shutdown_connections 2023-10-16T13:27:42.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.499+0000 7f2aa88e6700 1 -- 172.21.15.16:0/3913903798 wait complete. 2023-10-16T13:27:42.515 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:27:42.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.636+0000 7f7147d5e700 1 Processor -- start 2023-10-16T13:27:42.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.637+0000 7f7147d5e700 1 -- start start 2023-10-16T13:27:42.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.637+0000 7f7147d5e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f714015b630 0x7f71400cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:42.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.637+0000 7f7147d5e700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f714015ba10 0x7f71400cdde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:42.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.637+0000 7f7147d5e700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71400ce770 con 0x7f71400ce320 2023-10-16T13:27:42.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.637+0000 7f7147d5e700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71400cdb20 con 0x7f714015ba10 2023-10-16T13:27:42.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.637+0000 7f7147d5e700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71400cc2a0 con 0x7f714015b630 2023-10-16T13:27:42.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.638+0000 7f71452f9700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f714015ba10 0x7f71400cdde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:42.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.638+0000 7f7145afa700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f714015b630 0x7f71400cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:42.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.638+0000 7f71452f9700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f714015ba10 0x7f71400cdde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:47502/0 (socket says 172.21.15.16:47502) 2023-10-16T13:27:42.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.638+0000 7f7145afa700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f714015b630 0x7f71400cd5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:36698/0 (socket says 172.21.15.16:36698) 2023-10-16T13:27:42.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.638+0000 7f71452f9700 1 -- 172.21.15.16:0/522304199 learned_addr learned my addr 172.21.15.16:0/522304199 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:42.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.638+0000 7f7144af8700 1 -- 172.21.15.16:0/522304199 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2103571727 0 0) 0x7f7138001490 con 0x7f71400ce320 2023-10-16T13:27:42.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.638+0000 7f7144af8700 1 -- 172.21.15.16:0/522304199 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f712c0035b0 con 0x7f71400ce320 2023-10-16T13:27:42.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.638+0000 7f7145afa700 1 -- 172.21.15.16:0/522304199 >> v2:172.21.15.49:3300/0 conn(0x7f714015ba10 msgr2=0x7f71400cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:42.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.638+0000 7f7145afa700 1 --2- 172.21.15.16:0/522304199 >> v2:172.21.15.49:3300/0 conn(0x7f714015ba10 0x7f71400cdde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:42.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.638+0000 7f7145afa700 1 -- 172.21.15.16:0/522304199 >> v1:172.21.15.49:6789/0 conn(0x7f71400ce320 legacy=0x7f71400cfca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:42.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.638+0000 7f7145afa700 1 -- 172.21.15.16:0/522304199 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71400cc540 con 0x7f714015b630 2023-10-16T13:27:42.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.638+0000 7f71452f9700 1 --2- 172.21.15.16:0/522304199 >> v2:172.21.15.49:3300/0 conn(0x7f714015ba10 0x7f71400cdde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:27:42.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.639+0000 7f7145afa700 1 --2- 172.21.15.16:0/522304199 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f714015b630 0x7f71400cd5e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f7130009820 tx=0x7f7130009b30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a3fe21b372b74dcb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:42.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.639+0000 7f7144af8700 1 -- 172.21.15.16:0/522304199 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7130005e30 con 0x7f714015b630 2023-10-16T13:27:42.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.639+0000 7f7144af8700 1 -- 172.21.15.16:0/522304199 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7130006030 con 0x7f714015b630 2023-10-16T13:27:42.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.639+0000 7f7144af8700 1 -- 172.21.15.16:0/522304199 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71300063c0 con 0x7f714015b630 2023-10-16T13:27:42.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.639+0000 7f7147d5e700 1 -- 172.21.15.16:0/522304199 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f714015b630 msgr2=0x7f71400cd5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:42.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.639+0000 7f7147d5e700 1 --2- 172.21.15.16:0/522304199 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f714015b630 0x7f71400cd5e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f7130009820 tx=0x7f7130009b30 comp rx=0 tx=0).stop 2023-10-16T13:27:42.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.639+0000 7f7147d5e700 1 -- 172.21.15.16:0/522304199 shutdown_connections 2023-10-16T13:27:42.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.639+0000 7f7147d5e700 1 --2- 172.21.15.16:0/522304199 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f714015b630 0x7f71400cd5e0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:42.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.639+0000 7f7147d5e700 1 --2- 172.21.15.16:0/522304199 >> v2:172.21.15.49:3300/0 conn(0x7f714015ba10 0x7f71400cdde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:42.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.639+0000 7f7147d5e700 1 -- 172.21.15.16:0/522304199 >> 172.21.15.16:0/522304199 conn(0x7f71400d14b0 msgr2=0x7f71400d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:42.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.640+0000 7f7147d5e700 1 -- 172.21.15.16:0/522304199 shutdown_connections 2023-10-16T13:27:42.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.640+0000 7f7147d5e700 1 -- 172.21.15.16:0/522304199 wait complete. 2023-10-16T13:27:42.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.641+0000 7f7147d5e700 1 Processor -- start 2023-10-16T13:27:42.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.641+0000 7f7147d5e700 1 -- start start 2023-10-16T13:27:42.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.641+0000 7f7147d5e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f71400ce320 0x7f71401f2b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:42.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.641+0000 7f7147d5e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f714015b630 0x7f71400c8580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:42.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.641+0000 7f7147d5e700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f714015ba10 0x7f71401f55b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:42.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.641+0000 7f7147d5e700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71401f5190 con 0x7f714015b630 2023-10-16T13:27:42.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.641+0000 7f7147d5e700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71401f4510 con 0x7f714015ba10 2023-10-16T13:27:42.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.641+0000 7f7147d5e700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71401f47f0 con 0x7f71400ce320 2023-10-16T13:27:42.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.642+0000 7f71452f9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f714015b630 0x7f71400c8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:42.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.642+0000 7f71452f9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f714015b630 0x7f71400c8580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48446/0 (socket says 172.21.15.16:48446) 2023-10-16T13:27:42.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.642+0000 7f71452f9700 1 -- 172.21.15.16:0/1650912101 learned_addr learned my addr 172.21.15.16:0/1650912101 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:42.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.642+0000 7f7145afa700 1 --2- 172.21.15.16:0/1650912101 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f71400ce320 0x7f71401f2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:42.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.642+0000 7f71462fb700 1 --2- 172.21.15.16:0/1650912101 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f714015ba10 0x7f71401f55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:42.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.642+0000 7f71452f9700 1 -- 172.21.15.16:0/1650912101 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f71400ce320 msgr2=0x7f71401f2b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:42.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.642+0000 7f71452f9700 1 --2- 172.21.15.16:0/1650912101 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f71400ce320 0x7f71401f2b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:42.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.642+0000 7f71452f9700 1 -- 172.21.15.16:0/1650912101 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f714015ba10 msgr2=0x7f71401f55b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:42.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.642+0000 7f71452f9700 1 --2- 172.21.15.16:0/1650912101 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f714015ba10 0x7f71401f55b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:42.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.642+0000 7f71452f9700 1 -- 172.21.15.16:0/1650912101 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f713c009550 con 0x7f714015b630 2023-10-16T13:27:42.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.642+0000 7f71452f9700 1 --2- 172.21.15.16:0/1650912101 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f714015b630 0x7f71400c8580 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7f713c00ebe0 tx=0x7f713c009e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:42.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.642+0000 7f71367fc700 1 -- 172.21.15.16:0/1650912101 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f713c00c8b0 con 0x7f714015b630 2023-10-16T13:27:42.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.642+0000 7f71367fc700 1 -- 172.21.15.16:0/1650912101 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f713c00ca10 con 0x7f714015b630 2023-10-16T13:27:42.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.642+0000 7f7147d5e700 1 -- 172.21.15.16:0/1650912101 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7130009430 con 0x7f714015b630 2023-10-16T13:27:42.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.642+0000 7f71367fc700 1 -- 172.21.15.16:0/1650912101 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f713c00cd80 con 0x7f714015b630 2023-10-16T13:27:42.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.642+0000 7f7147d5e700 1 -- 172.21.15.16:0/1650912101 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71401f4db0 con 0x7f714015b630 2023-10-16T13:27:42.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.643+0000 7f7147d5e700 1 -- 172.21.15.16:0/1650912101 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f714004ebb0 con 0x7f714015b630 2023-10-16T13:27:42.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.644+0000 7f71367fc700 1 -- 172.21.15.16:0/1650912101 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f713c013470 con 0x7f714015b630 2023-10-16T13:27:42.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.644+0000 7f71367fc700 1 --2- 172.21.15.16:0/1650912101 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f712406b020 0x7f712406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:42.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.644+0000 7f71367fc700 1 -- 172.21.15.16:0/1650912101 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f713c089a30 con 0x7f714015b630 2023-10-16T13:27:42.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.644+0000 7f7145afa700 1 --2- 172.21.15.16:0/1650912101 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f712406b020 0x7f712406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:42.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.645+0000 7f7145afa700 1 --2- 172.21.15.16:0/1650912101 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f712406b020 0x7f712406d4e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7130000c00 tx=0x7f7130008070 comp rx=0 tx=0).ready entity=mgr.9510 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:42.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.649+0000 7f71367fc700 1 -- 172.21.15.16:0/1650912101 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f713c0548b0 con 0x7f714015b630 2023-10-16T13:27:42.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.852+0000 7f7147d5e700 1 -- 172.21.15.16:0/1650912101 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f714006b5d0 con 0x7f714015b630 2023-10-16T13:27:42.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.853+0000 7f71367fc700 1 -- 172.21.15.16:0/1650912101 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v176) v1 ==== 74+0+16788 (secure 0 0 0) 0x7f713c021090 con 0x7f714015b630 2023-10-16T13:27:42.856 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:27:42.857 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":176,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:27:41.561488+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2023-10-16T13:27:40.032163+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"176","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2023-10-16T13:27:40.523975+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"176","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:27:42.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.855+0000 7f7147d5e700 1 -- 172.21.15.16:0/1650912101 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f712406b020 msgr2=0x7f712406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:42.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.855+0000 7f7147d5e700 1 --2- 172.21.15.16:0/1650912101 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f712406b020 0x7f712406d4e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7130000c00 tx=0x7f7130008070 comp rx=0 tx=0).stop 2023-10-16T13:27:42.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.855+0000 7f7147d5e700 1 -- 172.21.15.16:0/1650912101 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f714015b630 msgr2=0x7f71400c8580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:42.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.855+0000 7f7147d5e700 1 --2- 172.21.15.16:0/1650912101 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f714015b630 0x7f71400c8580 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7f713c00ebe0 tx=0x7f713c009e70 comp rx=0 tx=0).stop 2023-10-16T13:27:42.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.855+0000 7f7147d5e700 1 -- 172.21.15.16:0/1650912101 shutdown_connections 2023-10-16T13:27:42.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.855+0000 7f7147d5e700 1 --2- 172.21.15.16:0/1650912101 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f712406b020 0x7f712406d4e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:42.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.855+0000 7f7147d5e700 1 --2- 172.21.15.16:0/1650912101 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f714015ba10 0x7f71401f55b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:42.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.855+0000 7f7147d5e700 1 --2- 172.21.15.16:0/1650912101 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f71400ce320 0x7f71401f2b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:42.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.855+0000 7f7147d5e700 1 --2- 172.21.15.16:0/1650912101 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f714015b630 0x7f71400c8580 unknown :-1 s=CLOSED pgs=1919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:42.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.855+0000 7f7147d5e700 1 -- 172.21.15.16:0/1650912101 >> 172.21.15.16:0/1650912101 conn(0x7f71400d14b0 msgr2=0x7f714006eba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:42.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.855+0000 7f7147d5e700 1 -- 172.21.15.16:0/1650912101 shutdown_connections 2023-10-16T13:27:42.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.855+0000 7f7147d5e700 1 -- 172.21.15.16:0/1650912101 wait complete. 2023-10-16T13:27:42.871 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-16T13:27:42.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.996+0000 7f4841bcf700 1 Processor -- start 2023-10-16T13:27:43.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.996+0000 7f4841bcf700 1 -- start start 2023-10-16T13:27:43.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.997+0000 7f4841bcf700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f483c15ec80 0x7f483c0cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:43.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.997+0000 7f4841bcf700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f483c15f060 0x7f483c0ce400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:43.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.997+0000 7f4841bcf700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f483c0d2110 0x7f483c0b4440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:43.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.997+0000 7f4841bcf700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f483c0b2c50 con 0x7f483c15f060 2023-10-16T13:27:43.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.997+0000 7f4841bcf700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f483c0b2440 con 0x7f483c15ec80 2023-10-16T13:27:43.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.997+0000 7f4841bcf700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f483c0d00e0 con 0x7f483c0d2110 2023-10-16T13:27:43.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.997+0000 7f483affd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f483c15f060 0x7f483c0ce400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:43.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.997+0000 7f483affd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f483c15f060 0x7f483c0ce400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48454/0 (socket says 172.21.15.16:48454) 2023-10-16T13:27:43.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.997+0000 7f483affd700 1 -- 172.21.15.16:0/2065984747 learned_addr learned my addr 172.21.15.16:0/2065984747 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:43.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.997+0000 7f483b7fe700 1 --2- 172.21.15.16:0/2065984747 >> v2:172.21.15.49:3300/0 conn(0x7f483c15ec80 0x7f483c0cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:43.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.997+0000 7f483bfff700 1 --2- 172.21.15.16:0/2065984747 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f483c0d2110 0x7f483c0b4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:43.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.997+0000 7f483affd700 1 -- 172.21.15.16:0/2065984747 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f483c0d2110 msgr2=0x7f483c0b4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:43.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.997+0000 7f483affd700 1 --2- 172.21.15.16:0/2065984747 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f483c0d2110 0x7f483c0b4440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:43.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.997+0000 7f483affd700 1 -- 172.21.15.16:0/2065984747 >> v2:172.21.15.49:3300/0 conn(0x7f483c15ec80 msgr2=0x7f483c0cdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:43.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.997+0000 7f483affd700 1 --2- 172.21.15.16:0/2065984747 >> v2:172.21.15.49:3300/0 conn(0x7f483c15ec80 0x7f483c0cdec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:43.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.997+0000 7f483affd700 1 -- 172.21.15.16:0/2065984747 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f483c0cef60 con 0x7f483c15f060 2023-10-16T13:27:43.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.997+0000 7f483b7fe700 1 --2- 172.21.15.16:0/2065984747 >> v2:172.21.15.49:3300/0 conn(0x7f483c15ec80 0x7f483c0cdec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:27:43.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.997+0000 7f483affd700 1 --2- 172.21.15.16:0/2065984747 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f483c15f060 0x7f483c0ce400 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7f483c067b20 tx=0x7f482c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cf717a35085df20 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:43.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.998+0000 7f483a7fc700 1 -- 172.21.15.16:0/2065984747 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f482c00b870 con 0x7f483c15f060 2023-10-16T13:27:43.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.998+0000 7f483a7fc700 1 -- 172.21.15.16:0/2065984747 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f482c00b9d0 con 0x7f483c15f060 2023-10-16T13:27:43.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.998+0000 7f483a7fc700 1 -- 172.21.15.16:0/2065984747 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f482c00bd60 con 0x7f483c15f060 2023-10-16T13:27:43.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.998+0000 7f4841bcf700 1 -- 172.21.15.16:0/2065984747 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f483c15f060 msgr2=0x7f483c0ce400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:43.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.998+0000 7f4841bcf700 1 --2- 172.21.15.16:0/2065984747 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f483c15f060 0x7f483c0ce400 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7f483c067b20 tx=0x7f482c009bd0 comp rx=0 tx=0).stop 2023-10-16T13:27:43.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.998+0000 7f4841bcf700 1 -- 172.21.15.16:0/2065984747 shutdown_connections 2023-10-16T13:27:43.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.998+0000 7f4841bcf700 1 --2- 172.21.15.16:0/2065984747 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f483c0d2110 0x7f483c0b4440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:43.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.998+0000 7f4841bcf700 1 --2- 172.21.15.16:0/2065984747 >> v2:172.21.15.49:3300/0 conn(0x7f483c15ec80 0x7f483c0cdec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:43.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.998+0000 7f4841bcf700 1 --2- 172.21.15.16:0/2065984747 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f483c15f060 0x7f483c0ce400 unknown :-1 s=CLOSED pgs=1920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:43.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.998+0000 7f4841bcf700 1 -- 172.21.15.16:0/2065984747 >> 172.21.15.16:0/2065984747 conn(0x7f483c0d15f0 msgr2=0x7f483c0d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:43.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.998+0000 7f4841bcf700 1 -- 172.21.15.16:0/2065984747 shutdown_connections 2023-10-16T13:27:43.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.998+0000 7f4841bcf700 1 -- 172.21.15.16:0/2065984747 wait complete. 2023-10-16T13:27:43.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:42.999+0000 7f4841bcf700 1 Processor -- start 2023-10-16T13:27:43.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:43.000+0000 7f4841bcf700 1 -- start start 2023-10-16T13:27:43.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:43.000+0000 7f4841bcf700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f483c0d2110 0x7f483c159a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:43.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:43.000+0000 7f4841bcf700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f483c15ec80 0x7f483c0b6700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:43.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:43.000+0000 7f4841bcf700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f483c15f060 0x7f483c154840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:43.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:43.000+0000 7f4841bcf700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f483c0b54e0 con 0x7f483c0d2110 2023-10-16T13:27:43.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:43.000+0000 7f4841bcf700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f483c159fd0 con 0x7f483c15f060 2023-10-16T13:27:43.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:43.000+0000 7f4841bcf700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f483c06d010 con 0x7f483c15ec80 2023-10-16T13:27:43.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:43.001+0000 7f483b7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f483c0d2110 0x7f483c159a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:43.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:43.001+0000 7f483b7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f483c0d2110 0x7f483c159a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48458/0 (socket says 172.21.15.16:48458) 2023-10-16T13:27:43.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:43.001+0000 7f483b7fe700 1 -- 172.21.15.16:0/4071089043 learned_addr learned my addr 172.21.15.16:0/4071089043 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:43.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:43.001+0000 7f483affd700 1 --2- 172.21.15.16:0/4071089043 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f483c15ec80 0x7f483c0b6700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:43.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:43.001+0000 7f483bfff700 1 --2- 172.21.15.16:0/4071089043 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f483c15f060 0x7f483c154840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:43.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:43.001+0000 7f483b7fe700 1 -- 172.21.15.16:0/4071089043 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f483c15ec80 msgr2=0x7f483c0b6700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:43.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:43.001+0000 7f483b7fe700 1 --2- 172.21.15.16:0/4071089043 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f483c15ec80 0x7f483c0b6700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:43.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:43.001+0000 7f483b7fe700 1 -- 172.21.15.16:0/4071089043 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f483c15f060 msgr2=0x7f483c154840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:43.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:43.001+0000 7f483b7fe700 1 --2- 172.21.15.16:0/4071089043 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f483c15f060 0x7f483c154840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:43.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:43.001+0000 7f483b7fe700 1 -- 172.21.15.16:0/4071089043 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f482c009550 con 0x7f483c0d2110 2023-10-16T13:27:43.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:43.001+0000 7f483b7fe700 1 --2- 172.21.15.16:0/4071089043 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f483c0d2110 0x7f483c159a90 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7f482400c750 tx=0x7f482400cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:43.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:43.001+0000 7f4833fff700 1 -- 172.21.15.16:0/4071089043 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4824011800 con 0x7f483c0d2110 2023-10-16T13:27:43.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:43.001+0000 7f4833fff700 1 -- 172.21.15.16:0/4071089043 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4824011960 con 0x7f483c0d2110 2023-10-16T13:27:43.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:43.001+0000 7f4841bcf700 1 -- 172.21.15.16:0/4071089043 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f483c06d2f0 con 0x7f483c0d2110 2023-10-16T13:27:43.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:43.001+0000 7f4833fff700 1 -- 172.21.15.16:0/4071089043 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4824011cd0 con 0x7f483c0d2110 2023-10-16T13:27:43.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:43.001+0000 7f4841bcf700 1 -- 172.21.15.16:0/4071089043 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f483c06d8b0 con 0x7f483c0d2110 2023-10-16T13:27:43.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:43.002+0000 7f4841bcf700 1 -- 172.21.15.16:0/4071089043 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f483c04ebb0 con 0x7f483c0d2110 2023-10-16T13:27:43.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:43.003+0000 7f4833fff700 1 -- 172.21.15.16:0/4071089043 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4824021350 con 0x7f483c0d2110 2023-10-16T13:27:43.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:43.003+0000 7f4833fff700 1 --2- 172.21.15.16:0/4071089043 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f481c06b020 0x7f481c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:43.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:43.003+0000 7f4833fff700 1 -- 172.21.15.16:0/4071089043 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f4824089920 con 0x7f483c0d2110 2023-10-16T13:27:43.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:43.003+0000 7f483affd700 1 --2- 172.21.15.16:0/4071089043 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f481c06b020 0x7f481c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:43.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:43.004+0000 7f483affd700 1 --2- 172.21.15.16:0/4071089043 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f481c06b020 0x7f481c06d4e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f483c15f460 tx=0x7f482c008030 comp rx=0 tx=0).ready entity=mgr.9510 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:43.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:43.008+0000 7f4833fff700 1 -- 172.21.15.16:0/4071089043 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4824054690 con 0x7f483c0d2110 2023-10-16T13:27:43.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:43.265+0000 7f4841bcf700 1 -- 172.21.15.16:0/4071089043 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f483c0cf7a0 con 0x7f483c0d2110 2023-10-16T13:27:44.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:44.145+0000 7f4833fff700 1 -- 172.21.15.16:0/4071089043 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v85) v1 ==== 118+0+0 (secure 0 0 0) 0x7f482401e0b0 con 0x7f483c0d2110 2023-10-16T13:27:44.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:44.149+0000 7f4841bcf700 1 -- 172.21.15.16:0/4071089043 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f481c06b020 msgr2=0x7f481c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:44.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:44.149+0000 7f4841bcf700 1 --2- 172.21.15.16:0/4071089043 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f481c06b020 0x7f481c06d4e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f483c15f460 tx=0x7f482c008030 comp rx=0 tx=0).stop 2023-10-16T13:27:44.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:44.149+0000 7f4841bcf700 1 -- 172.21.15.16:0/4071089043 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f483c0d2110 msgr2=0x7f483c159a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:44.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:44.149+0000 7f4841bcf700 1 --2- 172.21.15.16:0/4071089043 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f483c0d2110 0x7f483c159a90 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7f482400c750 tx=0x7f482400cb10 comp rx=0 tx=0).stop 2023-10-16T13:27:44.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:44.149+0000 7f4841bcf700 1 -- 172.21.15.16:0/4071089043 shutdown_connections 2023-10-16T13:27:44.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:44.149+0000 7f4841bcf700 1 --2- 172.21.15.16:0/4071089043 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f481c06b020 0x7f481c06d4e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:44.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:44.149+0000 7f4841bcf700 1 --2- 172.21.15.16:0/4071089043 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f483c15f060 0x7f483c154840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:44.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:44.149+0000 7f4841bcf700 1 --2- 172.21.15.16:0/4071089043 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f483c0d2110 0x7f483c159a90 unknown :-1 s=CLOSED pgs=1921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:44.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:44.149+0000 7f4841bcf700 1 --2- 172.21.15.16:0/4071089043 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f483c15ec80 0x7f483c0b6700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:44.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:44.149+0000 7f4841bcf700 1 -- 172.21.15.16:0/4071089043 >> 172.21.15.16:0/4071089043 conn(0x7f483c0d15f0 msgr2=0x7f483c15c8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:44.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:44.149+0000 7f4841bcf700 1 -- 172.21.15.16:0/4071089043 shutdown_connections 2023-10-16T13:27:44.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:44.149+0000 7f4841bcf700 1 -- 172.21.15.16:0/4071089043 wait complete. 2023-10-16T13:27:44.169 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-16T13:27:44.170 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-16T13:27:44.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:44.305+0000 7f07bc00a700 1 Processor -- start 2023-10-16T13:27:44.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:44.305+0000 7f07bc00a700 1 -- start start 2023-10-16T13:27:44.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:44.306+0000 7f07bc00a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f07b40b36b0 0x7f07b406be40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:44.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:44.306+0000 7f07bc00a700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f07b4162390 0x7f07b406c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:44.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:44.306+0000 7f07bc00a700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07b4161f00 con 0x7f07b415d8c0 2023-10-16T13:27:44.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:44.306+0000 7f07bc00a700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07b415f180 con 0x7f07b4162390 2023-10-16T13:27:44.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:44.306+0000 7f07bc00a700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07b415ecb0 con 0x7f07b40b36b0 2023-10-16T13:27:44.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:44.306+0000 7f07b95a5700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f07b40b36b0 0x7f07b406be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:44.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:44.306+0000 7f07ba5a7700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f07b4162390 0x7f07b406c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:44.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:44.306+0000 7f07b9da6700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7f07b415d8c0 0x7f07b40b2fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:44922/0 (socket says 172.21.15.16:44922) 2023-10-16T13:27:44.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:44.306+0000 7f07b9da6700 1 -- 172.21.15.16:0/2465403046 learned_addr learned my addr 172.21.15.16:0/2465403046 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:27:44.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:44.307+0000 7f07b8da4700 1 -- 172.21.15.16:0/2465403046 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 484210505 0 0) 0x7f07b4161f00 con 0x7f07b415d8c0 2023-10-16T13:27:44.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:44.307+0000 7f07b8da4700 1 -- 172.21.15.16:0/2465403046 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07a00035b0 con 0x7f07b415d8c0 2023-10-16T13:27:44.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:44.307+0000 7f07ba5a7700 1 -- 172.21.15.16:0/2465403046 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f07b40b36b0 msgr2=0x7f07b406be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:44.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:44.307+0000 7f07ba5a7700 1 --2- 172.21.15.16:0/2465403046 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f07b40b36b0 0x7f07b406be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:44.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:44.307+0000 7f07ba5a7700 1 -- 172.21.15.16:0/2465403046 >> v1:172.21.15.49:6789/0 conn(0x7f07b415d8c0 legacy=0x7f07b40b2fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:44.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:44.307+0000 7f07ba5a7700 1 -- 172.21.15.16:0/2465403046 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07b415eea0 con 0x7f07b4162390 2023-10-16T13:27:44.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:44.307+0000 7f07b95a5700 1 --2- 172.21.15.16:0/2465403046 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f07b40b36b0 0x7f07b406be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:27:44.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:44.307+0000 7f07ba5a7700 1 --2- 172.21.15.16:0/2465403046 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f07b4162390 0x7f07b406c380 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f07ac009dc0 tx=0x7f07ac00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=564e554508485412 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:44.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:44.307+0000 7f07b8da4700 1 -- 172.21.15.16:0/2465403046 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07ac00e070 con 0x7f07b4162390 2023-10-16T13:27:44.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:44.307+0000 7f07b8da4700 1 -- 172.21.15.16:0/2465403046 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07ac00bae0 con 0x7f07b4162390 2023-10-16T13:27:44.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:44.307+0000 7f07b8da4700 1 -- 172.21.15.16:0/2465403046 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07ac00be50 con 0x7f07b4162390 2023-10-16T13:27:44.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:44.308+0000 7f07bc00a700 1 -- 172.21.15.16:0/2465403046 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f07b4162390 msgr2=0x7f07b406c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:44.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:44.308+0000 7f07bc00a700 1 --2- 172.21.15.16:0/2465403046 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f07b4162390 0x7f07b406c380 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f07ac009dc0 tx=0x7f07ac00b330 comp rx=0 tx=0).stop 2023-10-16T13:27:44.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:44.308+0000 7f07bc00a700 1 -- 172.21.15.16:0/2465403046 shutdown_connections 2023-10-16T13:27:44.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:44.308+0000 7f07bc00a700 1 --2- 172.21.15.16:0/2465403046 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f07b4162390 0x7f07b406c380 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:44.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:44.308+0000 7f07bc00a700 1 --2- 172.21.15.16:0/2465403046 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f07b40b36b0 0x7f07b406be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:44.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:44.308+0000 7f07bc00a700 1 -- 172.21.15.16:0/2465403046 >> 172.21.15.16:0/2465403046 conn(0x7f07b40cdfa0 msgr2=0x7f07b40ce3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:44.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:44.308+0000 7f07bc00a700 1 -- 172.21.15.16:0/2465403046 shutdown_connections 2023-10-16T13:27:44.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:44.308+0000 7f07bc00a700 1 -- 172.21.15.16:0/2465403046 wait complete. 2023-10-16T13:27:44.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:44.309+0000 7f07bc00a700 1 Processor -- start 2023-10-16T13:27:44.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:44.309+0000 7f07bc00a700 1 -- start start 2023-10-16T13:27:44.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:44.310+0000 7f07bc00a700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f07b40b36b0 0x7f07b40c4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:44.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:44.310+0000 7f07bc00a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07b415d8c0 0x7f07b414e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:44.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:44.310+0000 7f07bc00a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f07b4162390 0x7f07b40c6e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:44.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:44.310+0000 7f07bc00a700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07b41f40d0 con 0x7f07b415d8c0 2023-10-16T13:27:44.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:44.310+0000 7f07bc00a700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07b414eed0 con 0x7f07b40b36b0 2023-10-16T13:27:44.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:44.310+0000 7f07bc00a700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07b41f34c0 con 0x7f07b4162390 2023-10-16T13:27:44.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:44.310+0000 7f07b95a5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07b415d8c0 0x7f07b414e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:44.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:44.310+0000 7f07b95a5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07b415d8c0 0x7f07b414e990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48474/0 (socket says 172.21.15.16:48474) 2023-10-16T13:27:44.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:44.310+0000 7f07b95a5700 1 -- 172.21.15.16:0/3487962417 learned_addr learned my addr 172.21.15.16:0/3487962417 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:44.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:44.310+0000 7f07ba5a7700 1 --2- 172.21.15.16:0/3487962417 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f07b4162390 0x7f07b40c6e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:44.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:44.310+0000 7f07b9da6700 1 --2- 172.21.15.16:0/3487962417 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f07b40b36b0 0x7f07b40c4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:44.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:44.310+0000 7f07b95a5700 1 -- 172.21.15.16:0/3487962417 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f07b4162390 msgr2=0x7f07b40c6e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:44.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:44.310+0000 7f07b95a5700 1 --2- 172.21.15.16:0/3487962417 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f07b4162390 0x7f07b40c6e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:44.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:44.310+0000 7f07b95a5700 1 -- 172.21.15.16:0/3487962417 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f07b40b36b0 msgr2=0x7f07b40c4fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:44.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:44.310+0000 7f07b95a5700 1 --2- 172.21.15.16:0/3487962417 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f07b40b36b0 0x7f07b40c4fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:44.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:44.310+0000 7f07b95a5700 1 -- 172.21.15.16:0/3487962417 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07b00094a0 con 0x7f07b415d8c0 2023-10-16T13:27:44.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:44.310+0000 7f07b95a5700 1 --2- 172.21.15.16:0/3487962417 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07b415d8c0 0x7f07b414e990 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7f07b000ebe0 tx=0x7f07b000eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:44.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:44.310+0000 7f07aa7fc700 1 -- 172.21.15.16:0/3487962417 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07b000c850 con 0x7f07b415d8c0 2023-10-16T13:27:44.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:44.310+0000 7f07aa7fc700 1 -- 172.21.15.16:0/3487962417 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07b000c9b0 con 0x7f07b415d8c0 2023-10-16T13:27:44.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:44.310+0000 7f07bc00a700 1 -- 172.21.15.16:0/3487962417 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07ac009a70 con 0x7f07b415d8c0 2023-10-16T13:27:44.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:44.310+0000 7f07aa7fc700 1 -- 172.21.15.16:0/3487962417 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07b000cd40 con 0x7f07b415d8c0 2023-10-16T13:27:44.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:44.311+0000 7f07bc00a700 1 -- 172.21.15.16:0/3487962417 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07b41f3a80 con 0x7f07b415d8c0 2023-10-16T13:27:44.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:44.311+0000 7f07bc00a700 1 -- 172.21.15.16:0/3487962417 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07b404ebb0 con 0x7f07b415d8c0 2023-10-16T13:27:44.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:44.312+0000 7f07aa7fc700 1 -- 172.21.15.16:0/3487962417 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f07b0013470 con 0x7f07b415d8c0 2023-10-16T13:27:44.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:44.312+0000 7f07aa7fc700 1 --2- 172.21.15.16:0/3487962417 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f079806b020 0x7f079806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:44.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:44.313+0000 7f07b9da6700 1 --2- 172.21.15.16:0/3487962417 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f079806b020 0x7f079806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:44.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:44.313+0000 7f07aa7fc700 1 -- 172.21.15.16:0/3487962417 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f07b0089990 con 0x7f07b415d8c0 2023-10-16T13:27:44.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:44.313+0000 7f07b9da6700 1 --2- 172.21.15.16:0/3487962417 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f079806b020 0x7f079806d4e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f07a4007fd0 tx=0x7f07a4009040 comp rx=0 tx=0).ready entity=mgr.9510 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:44.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:44.317+0000 7f07aa7fc700 1 -- 172.21.15.16:0/3487962417 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f07b0054780 con 0x7f07b415d8c0 2023-10-16T13:27:44.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:44.570+0000 7f07bc00a700 1 -- 172.21.15.16:0/3487962417 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f07b40b74a0 con 0x7f07b415d8c0 2023-10-16T13:27:45.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.149+0000 7f07aa7fc700 1 -- 172.21.15.16:0/3487962417 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v86) v1 ==== 122+0+0 (secure 0 0 0) 0x7f07b00210c0 con 0x7f07b415d8c0 2023-10-16T13:27:45.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.152+0000 7f07bc00a700 1 -- 172.21.15.16:0/3487962417 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f079806b020 msgr2=0x7f079806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:45.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.152+0000 7f07bc00a700 1 --2- 172.21.15.16:0/3487962417 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f079806b020 0x7f079806d4e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f07a4007fd0 tx=0x7f07a4009040 comp rx=0 tx=0).stop 2023-10-16T13:27:45.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.152+0000 7f07bc00a700 1 -- 172.21.15.16:0/3487962417 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07b415d8c0 msgr2=0x7f07b414e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:45.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.152+0000 7f07bc00a700 1 --2- 172.21.15.16:0/3487962417 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07b415d8c0 0x7f07b414e990 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7f07b000ebe0 tx=0x7f07b000eef0 comp rx=0 tx=0).stop 2023-10-16T13:27:45.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.152+0000 7f07bc00a700 1 -- 172.21.15.16:0/3487962417 shutdown_connections 2023-10-16T13:27:45.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.152+0000 7f07bc00a700 1 --2- 172.21.15.16:0/3487962417 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f07b4162390 0x7f07b40c6e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:45.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.152+0000 7f07bc00a700 1 --2- 172.21.15.16:0/3487962417 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f079806b020 0x7f079806d4e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:45.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.152+0000 7f07bc00a700 1 --2- 172.21.15.16:0/3487962417 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f07b40b36b0 0x7f07b40c4fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:45.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.152+0000 7f07bc00a700 1 --2- 172.21.15.16:0/3487962417 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07b415d8c0 0x7f07b414e990 unknown :-1 s=CLOSED pgs=1922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:45.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.152+0000 7f07bc00a700 1 -- 172.21.15.16:0/3487962417 >> 172.21.15.16:0/3487962417 conn(0x7f07b40cdfa0 msgr2=0x7f07b4159df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:45.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.153+0000 7f07bc00a700 1 -- 172.21.15.16:0/3487962417 shutdown_connections 2023-10-16T13:27:45.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.153+0000 7f07bc00a700 1 -- 172.21.15.16:0/3487962417 wait complete. 2023-10-16T13:27:45.171 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-16T13:27:45.172 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:27:45.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.300+0000 7f8b728b7700 1 Processor -- start 2023-10-16T13:27:45.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.300+0000 7f8b728b7700 1 -- start start 2023-10-16T13:27:45.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.301+0000 7f8b728b7700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8b6c0cf880 0x7f8b6c0b52f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:45.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.301+0000 7f8b728b7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8b6c0b5830 0x7f8b6c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:45.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.301+0000 7f8b728b7700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b6c15bc80 con 0x7f8b6c0cf4a0 2023-10-16T13:27:45.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.301+0000 7f8b728b7700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b6c15b070 con 0x7f8b6c0b5830 2023-10-16T13:27:45.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.301+0000 7f8b728b7700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b6c15b350 con 0x7f8b6c0cf880 2023-10-16T13:27:45.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.301+0000 7f8b70e54700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8b6c0b5830 0x7f8b6c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:45.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.301+0000 7f8b6bfff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f8b6c0cf4a0 0x7f8b6c0726e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37754/0 (socket says 172.21.15.16:37754) 2023-10-16T13:27:45.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.301+0000 7f8b6bfff700 1 -- 172.21.15.16:0/2154654513 learned_addr learned my addr 172.21.15.16:0/2154654513 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:27:45.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.301+0000 7f8b6b7fe700 1 --2- 172.21.15.16:0/2154654513 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8b6c0cf880 0x7f8b6c0b52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:45.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.301+0000 7f8b70e54700 1 -- 172.21.15.16:0/2154654513 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8b6c0cf880 msgr2=0x7f8b6c0b52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:45.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.301+0000 7f8b70e54700 1 --2- 172.21.15.16:0/2154654513 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8b6c0cf880 0x7f8b6c0b52f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:45.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.301+0000 7f8b70e54700 1 -- 172.21.15.16:0/2154654513 >> v1:172.21.15.16:6789/0 conn(0x7f8b6c0cf4a0 legacy=0x7f8b6c0726e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:45.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.301+0000 7f8b70e54700 1 -- 172.21.15.16:0/2154654513 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b6c15b630 con 0x7f8b6c0b5830 2023-10-16T13:27:45.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.301+0000 7f8b6b7fe700 1 --2- 172.21.15.16:0/2154654513 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8b6c0cf880 0x7f8b6c0b52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:27:45.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.301+0000 7f8b70e54700 1 --2- 172.21.15.16:0/2154654513 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b6c0b5830 0x7f8b6c15a090 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7f8b64009d90 tx=0x7f8b6400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25660e5b88d9a29c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:45.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.301+0000 7f8b6affd700 1 -- 172.21.15.16:0/2154654513 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b6400bdd0 con 0x7f8b6c0b5830 2023-10-16T13:27:45.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.302+0000 7f8b6affd700 1 -- 172.21.15.16:0/2154654513 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b64011460 con 0x7f8b6c0b5830 2023-10-16T13:27:45.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.302+0000 7f8b6affd700 1 -- 172.21.15.16:0/2154654513 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b64011810 con 0x7f8b6c0b5830 2023-10-16T13:27:45.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.302+0000 7f8b728b7700 1 -- 172.21.15.16:0/2154654513 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b6c0b5830 msgr2=0x7f8b6c15a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:45.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.302+0000 7f8b728b7700 1 --2- 172.21.15.16:0/2154654513 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b6c0b5830 0x7f8b6c15a090 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7f8b64009d90 tx=0x7f8b6400b330 comp rx=0 tx=0).stop 2023-10-16T13:27:45.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.302+0000 7f8b728b7700 1 -- 172.21.15.16:0/2154654513 shutdown_connections 2023-10-16T13:27:45.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.302+0000 7f8b728b7700 1 --2- 172.21.15.16:0/2154654513 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b6c0b5830 0x7f8b6c15a090 unknown :-1 s=CLOSED pgs=1924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:45.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.302+0000 7f8b728b7700 1 --2- 172.21.15.16:0/2154654513 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8b6c0cf880 0x7f8b6c0b52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:45.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.302+0000 7f8b728b7700 1 -- 172.21.15.16:0/2154654513 >> 172.21.15.16:0/2154654513 conn(0x7f8b6c0b1fd0 msgr2=0x7f8b6c0b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:45.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.302+0000 7f8b728b7700 1 -- 172.21.15.16:0/2154654513 shutdown_connections 2023-10-16T13:27:45.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.302+0000 7f8b728b7700 1 -- 172.21.15.16:0/2154654513 wait complete. 2023-10-16T13:27:45.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.304+0000 7f8b728b7700 1 Processor -- start 2023-10-16T13:27:45.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.304+0000 7f8b728b7700 1 -- start start 2023-10-16T13:27:45.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.305+0000 7f8b728b7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b6c0b5830 0x7f8b6c0c6b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:45.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.305+0000 7f8b728b7700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8b6c0cf4a0 0x7f8b6c0c6080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:45.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.305+0000 7f8b728b7700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8b6c0cf880 0x7f8b6c14b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:45.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.305+0000 7f8b728b7700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b6c0c5800 con 0x7f8b6c0b5830 2023-10-16T13:27:45.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.305+0000 7f8b728b7700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b6c0c4950 con 0x7f8b6c0cf4a0 2023-10-16T13:27:45.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.305+0000 7f8b728b7700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b6c0c4bf0 con 0x7f8b6c0cf880 2023-10-16T13:27:45.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.305+0000 7f8b6bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b6c0b5830 0x7f8b6c0c6b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:45.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.305+0000 7f8b6bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b6c0b5830 0x7f8b6c0c6b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48486/0 (socket says 172.21.15.16:48486) 2023-10-16T13:27:45.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.305+0000 7f8b6bfff700 1 -- 172.21.15.16:0/531569727 learned_addr learned my addr 172.21.15.16:0/531569727 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:45.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.305+0000 7f8b6b7fe700 1 --2- 172.21.15.16:0/531569727 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8b6c0cf4a0 0x7f8b6c0c6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:45.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.305+0000 7f8b70e54700 1 --2- 172.21.15.16:0/531569727 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8b6c0cf880 0x7f8b6c14b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:45.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.305+0000 7f8b6bfff700 1 -- 172.21.15.16:0/531569727 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8b6c0cf880 msgr2=0x7f8b6c14b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:45.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.305+0000 7f8b6bfff700 1 --2- 172.21.15.16:0/531569727 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8b6c0cf880 0x7f8b6c14b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:45.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.305+0000 7f8b6bfff700 1 -- 172.21.15.16:0/531569727 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8b6c0cf4a0 msgr2=0x7f8b6c0c6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:45.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.305+0000 7f8b6bfff700 1 --2- 172.21.15.16:0/531569727 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8b6c0cf4a0 0x7f8b6c0c6080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:45.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.305+0000 7f8b6bfff700 1 -- 172.21.15.16:0/531569727 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b64009a70 con 0x7f8b6c0b5830 2023-10-16T13:27:45.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.305+0000 7f8b6bfff700 1 --2- 172.21.15.16:0/531569727 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b6c0b5830 0x7f8b6c0c6b10 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7f8b5400c520 tx=0x7f8b5400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:45.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.305+0000 7f8b68ff9700 1 -- 172.21.15.16:0/531569727 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b54005d70 con 0x7f8b6c0b5830 2023-10-16T13:27:45.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.305+0000 7f8b68ff9700 1 -- 172.21.15.16:0/531569727 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b540070d0 con 0x7f8b6c0b5830 2023-10-16T13:27:45.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.305+0000 7f8b728b7700 1 -- 172.21.15.16:0/531569727 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b6c0c4ed0 con 0x7f8b6c0b5830 2023-10-16T13:27:45.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.305+0000 7f8b68ff9700 1 -- 172.21.15.16:0/531569727 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b54007440 con 0x7f8b6c0b5830 2023-10-16T13:27:45.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.305+0000 7f8b728b7700 1 -- 172.21.15.16:0/531569727 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b6c0c5490 con 0x7f8b6c0b5830 2023-10-16T13:27:45.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.306+0000 7f8b728b7700 1 -- 172.21.15.16:0/531569727 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b6c0c5620 con 0x7f8b6c0b5830 2023-10-16T13:27:45.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.307+0000 7f8b68ff9700 1 -- 172.21.15.16:0/531569727 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8b540075a0 con 0x7f8b6c0b5830 2023-10-16T13:27:45.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.307+0000 7f8b68ff9700 1 --2- 172.21.15.16:0/531569727 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f8b4c06b020 0x7f8b4c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:45.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.307+0000 7f8b68ff9700 1 -- 172.21.15.16:0/531569727 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f8b54088db0 con 0x7f8b6c0b5830 2023-10-16T13:27:45.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.307+0000 7f8b6b7fe700 1 --2- 172.21.15.16:0/531569727 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f8b4c06b020 0x7f8b4c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:45.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.308+0000 7f8b6b7fe700 1 --2- 172.21.15.16:0/531569727 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f8b4c06b020 0x7f8b4c06d4e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8b6c0b2700 tx=0x7f8b5c008040 comp rx=0 tx=0).ready entity=mgr.9510 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:45.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.312+0000 7f8b68ff9700 1 -- 172.21.15.16:0/531569727 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8b5408d070 con 0x7f8b6c0b5830 2023-10-16T13:27:45.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.573+0000 7f8b728b7700 1 -- 172.21.15.16:0/531569727 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b6c04ebb0 con 0x7f8b6c0b5830 2023-10-16T13:27:45.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.573+0000 7f8b68ff9700 1 -- 172.21.15.16:0/531569727 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+4917 (secure 0 0 0) 0x7f8b54017090 con 0x7f8b6c0b5830 2023-10-16T13:27:45.577 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:27:45.577 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":86,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7964,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/4012137577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":4012137577},{"type":"v1","addr":"172.21.15.16:6837","nonce":4012137577}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81},{"gid":7970,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/3742568047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":3742568047},{"type":"v1","addr":"172.21.15.49:6835","nonce":3742568047}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81},{"gid":9756,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/3838743980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":3838743980},{"type":"v1","addr":"172.21.15.16:6836","nonce":3838743980}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:27:41.561818+0000","modified":"2023-10-16T13:27:44.572793+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9774},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9774":{"gid":9774,"name":"d","rank":0,"incarnation":82,"state":"up:active","state_seq":3,"addr":"172.21.15.49:6834/2218212001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":2218212001},{"type":"v1","addr":"172.21.15.49:6834","nonce":2218212001}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2023-10-16T13:27:45.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.576+0000 7f8b728b7700 1 -- 172.21.15.16:0/531569727 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f8b4c06b020 msgr2=0x7f8b4c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:45.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.576+0000 7f8b728b7700 1 --2- 172.21.15.16:0/531569727 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f8b4c06b020 0x7f8b4c06d4e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8b6c0b2700 tx=0x7f8b5c008040 comp rx=0 tx=0).stop 2023-10-16T13:27:45.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.576+0000 7f8b728b7700 1 -- 172.21.15.16:0/531569727 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b6c0b5830 msgr2=0x7f8b6c0c6b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:45.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.576+0000 7f8b728b7700 1 --2- 172.21.15.16:0/531569727 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b6c0b5830 0x7f8b6c0c6b10 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7f8b5400c520 tx=0x7f8b5400c8e0 comp rx=0 tx=0).stop 2023-10-16T13:27:45.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.576+0000 7f8b728b7700 1 -- 172.21.15.16:0/531569727 shutdown_connections 2023-10-16T13:27:45.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.576+0000 7f8b728b7700 1 --2- 172.21.15.16:0/531569727 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8b6c0cf880 0x7f8b6c14b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:45.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.576+0000 7f8b728b7700 1 --2- 172.21.15.16:0/531569727 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b6c0b5830 0x7f8b6c0c6b10 unknown :-1 s=CLOSED pgs=1925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:45.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.576+0000 7f8b728b7700 1 --2- 172.21.15.16:0/531569727 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f8b4c06b020 0x7f8b4c06d4e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:45.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.576+0000 7f8b728b7700 1 --2- 172.21.15.16:0/531569727 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8b6c0cf4a0 0x7f8b6c0c6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:45.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.576+0000 7f8b728b7700 1 -- 172.21.15.16:0/531569727 >> 172.21.15.16:0/531569727 conn(0x7f8b6c0b1fd0 msgr2=0x7f8b6c0afec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:45.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.576+0000 7f8b728b7700 1 -- 172.21.15.16:0/531569727 shutdown_connections 2023-10-16T13:27:45.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.576+0000 7f8b728b7700 1 -- 172.21.15.16:0/531569727 wait complete. 2023-10-16T13:27:45.582 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 86 2023-10-16T13:27:45.592 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:27:45.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.713+0000 7f18d332b700 1 Processor -- start 2023-10-16T13:27:45.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.713+0000 7f18d332b700 1 -- start start 2023-10-16T13:27:45.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.715+0000 7f18d332b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f18cc159290 0x7f18cc0cd810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:45.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.715+0000 7f18d332b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f18cc06dbe0 0x7f18cc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:45.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.715+0000 7f18d332b700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18cc0d1310 con 0x7f18cc071990 2023-10-16T13:27:45.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.715+0000 7f18d332b700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18cc06e5d0 con 0x7f18cc06dbe0 2023-10-16T13:27:45.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.715+0000 7f18d332b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18cc0cb600 con 0x7f18cc159290 2023-10-16T13:27:45.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.715+0000 7f18d18c8700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f18cc06dbe0 0x7f18cc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:45.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.715+0000 7f18d18c8700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f18cc06dbe0 0x7f18cc06e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48498/0 (socket says 172.21.15.16:48498) 2023-10-16T13:27:45.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.715+0000 7f18d18c8700 1 -- 172.21.15.16:0/595181094 learned_addr learned my addr 172.21.15.16:0/595181094 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:45.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.715+0000 7f18d10c7700 1 --2- 172.21.15.16:0/595181094 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f18cc159290 0x7f18cc0cd810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:45.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.715+0000 7f18d18c8700 1 -- 172.21.15.16:0/595181094 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f18cc159290 msgr2=0x7f18cc0cd810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:45.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.715+0000 7f18d18c8700 1 --2- 172.21.15.16:0/595181094 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f18cc159290 0x7f18cc0cd810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:45.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.715+0000 7f18d18c8700 1 -- 172.21.15.16:0/595181094 >> v1:172.21.15.49:6789/0 conn(0x7f18cc071990 legacy=0x7f18cc15bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:45.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.715+0000 7f18d18c8700 1 -- 172.21.15.16:0/595181094 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18cc159670 con 0x7f18cc06dbe0 2023-10-16T13:27:45.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.715+0000 7f18d18c8700 1 --2- 172.21.15.16:0/595181094 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18cc06dbe0 0x7f18cc06e090 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7f18c4009e80 tx=0x7f18c400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79e1f640dc56370e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:45.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.715+0000 7f18c3fff700 1 -- 172.21.15.16:0/595181094 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18c400e070 con 0x7f18cc06dbe0 2023-10-16T13:27:45.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.715+0000 7f18c3fff700 1 -- 172.21.15.16:0/595181094 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18c4003a20 con 0x7f18cc06dbe0 2023-10-16T13:27:45.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.716+0000 7f18c3fff700 1 -- 172.21.15.16:0/595181094 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18c4003dd0 con 0x7f18cc06dbe0 2023-10-16T13:27:45.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.716+0000 7f18d332b700 1 -- 172.21.15.16:0/595181094 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18cc06dbe0 msgr2=0x7f18cc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:45.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.716+0000 7f18d332b700 1 --2- 172.21.15.16:0/595181094 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18cc06dbe0 0x7f18cc06e090 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7f18c4009e80 tx=0x7f18c400b330 comp rx=0 tx=0).stop 2023-10-16T13:27:45.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.716+0000 7f18d332b700 1 -- 172.21.15.16:0/595181094 shutdown_connections 2023-10-16T13:27:45.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.716+0000 7f18d332b700 1 --2- 172.21.15.16:0/595181094 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18cc06dbe0 0x7f18cc06e090 unknown :-1 s=CLOSED pgs=1926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:45.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.716+0000 7f18d332b700 1 --2- 172.21.15.16:0/595181094 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f18cc159290 0x7f18cc0cd810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:45.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.716+0000 7f18d332b700 1 -- 172.21.15.16:0/595181094 >> 172.21.15.16:0/595181094 conn(0x7f18cc06bad0 msgr2=0x7f18cc06bee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:45.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.716+0000 7f18d332b700 1 -- 172.21.15.16:0/595181094 shutdown_connections 2023-10-16T13:27:45.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.716+0000 7f18d332b700 1 -- 172.21.15.16:0/595181094 wait complete. 2023-10-16T13:27:45.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.717+0000 7f18d332b700 1 Processor -- start 2023-10-16T13:27:45.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.717+0000 7f18d332b700 1 -- start start 2023-10-16T13:27:45.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.718+0000 7f18d332b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f18cc06dbe0 0x7f18cc1f6200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:45.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.718+0000 7f18d332b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18cc159290 0x7f18cc0c15d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:45.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.718+0000 7f18d332b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f18cc0cfcc0 0x7f18cc0bf930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:45.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.718+0000 7f18d332b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18cc1f6a50 con 0x7f18cc159290 2023-10-16T13:27:45.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.718+0000 7f18d332b700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18cc1f6d30 con 0x7f18cc06dbe0 2023-10-16T13:27:45.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.718+0000 7f18d332b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18cc1f7010 con 0x7f18cc0cfcc0 2023-10-16T13:27:45.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.718+0000 7f18d08c6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18cc159290 0x7f18cc0c15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:45.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.718+0000 7f18d08c6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18cc159290 0x7f18cc0c15d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48502/0 (socket says 172.21.15.16:48502) 2023-10-16T13:27:45.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.718+0000 7f18d08c6700 1 -- 172.21.15.16:0/3688921317 learned_addr learned my addr 172.21.15.16:0/3688921317 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:45.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.718+0000 7f18d18c8700 1 --2- 172.21.15.16:0/3688921317 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f18cc0cfcc0 0x7f18cc0bf930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:45.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.718+0000 7f18d10c7700 1 --2- 172.21.15.16:0/3688921317 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f18cc06dbe0 0x7f18cc1f6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:45.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.719+0000 7f18d08c6700 1 -- 172.21.15.16:0/3688921317 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f18cc0cfcc0 msgr2=0x7f18cc0bf930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:45.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.719+0000 7f18d08c6700 1 --2- 172.21.15.16:0/3688921317 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f18cc0cfcc0 0x7f18cc0bf930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:45.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.719+0000 7f18d08c6700 1 -- 172.21.15.16:0/3688921317 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f18cc06dbe0 msgr2=0x7f18cc1f6200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:45.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.719+0000 7f18d08c6700 1 --2- 172.21.15.16:0/3688921317 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f18cc06dbe0 0x7f18cc1f6200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:45.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.719+0000 7f18d08c6700 1 -- 172.21.15.16:0/3688921317 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18c4009b30 con 0x7f18cc159290 2023-10-16T13:27:45.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.719+0000 7f18d08c6700 1 --2- 172.21.15.16:0/3688921317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18cc159290 0x7f18cc0c15d0 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x7f18c800c4d0 tx=0x7f18c800c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:45.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.719+0000 7f18c1ffb700 1 -- 172.21.15.16:0/3688921317 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18c8011790 con 0x7f18cc159290 2023-10-16T13:27:45.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.719+0000 7f18c1ffb700 1 -- 172.21.15.16:0/3688921317 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18c80118f0 con 0x7f18cc159290 2023-10-16T13:27:45.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.719+0000 7f18d332b700 1 -- 172.21.15.16:0/3688921317 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18cc0c5970 con 0x7f18cc159290 2023-10-16T13:27:45.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.719+0000 7f18c1ffb700 1 -- 172.21.15.16:0/3688921317 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18c8011ca0 con 0x7f18cc159290 2023-10-16T13:27:45.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.719+0000 7f18d332b700 1 -- 172.21.15.16:0/3688921317 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18cc0c1b10 con 0x7f18cc159290 2023-10-16T13:27:45.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.720+0000 7f18d332b700 1 -- 172.21.15.16:0/3688921317 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18cc0c5b60 con 0x7f18cc159290 2023-10-16T13:27:45.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.720+0000 7f18c1ffb700 1 -- 172.21.15.16:0/3688921317 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f18c801c460 con 0x7f18cc159290 2023-10-16T13:27:45.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.721+0000 7f18c1ffb700 1 --2- 172.21.15.16:0/3688921317 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f18b006b020 0x7f18b006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:45.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.721+0000 7f18c1ffb700 1 -- 172.21.15.16:0/3688921317 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f18c80895f0 con 0x7f18cc159290 2023-10-16T13:27:45.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.721+0000 7f18d10c7700 1 --2- 172.21.15.16:0/3688921317 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f18b006b020 0x7f18b006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:45.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.721+0000 7f18d10c7700 1 --2- 172.21.15.16:0/3688921317 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f18b006b020 0x7f18b006d4e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f18bc006b20 tx=0x7f18bc008040 comp rx=0 tx=0).ready entity=mgr.9510 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:45.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.725+0000 7f18c1ffb700 1 -- 172.21.15.16:0/3688921317 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f18c80543e0 con 0x7f18cc159290 2023-10-16T13:27:45.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.932+0000 7f18d332b700 1 -- 172.21.15.16:0/3688921317 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f18cc04ebb0 con 0x7f18cc159290 2023-10-16T13:27:45.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.933+0000 7f18c1ffb700 1 -- 172.21.15.16:0/3688921317 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v177) v1 ==== 74+0+16775 (secure 0 0 0) 0x7f18c8057a00 con 0x7f18cc159290 2023-10-16T13:27:45.937 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:27:45.937 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":177,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:27:43.071088+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2023-10-16T13:27:40.032163+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"176","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2023-10-16T13:27:40.523975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"177","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:27:45.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.935+0000 7f18d332b700 1 -- 172.21.15.16:0/3688921317 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f18b006b020 msgr2=0x7f18b006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:45.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.935+0000 7f18d332b700 1 --2- 172.21.15.16:0/3688921317 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f18b006b020 0x7f18b006d4e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f18bc006b20 tx=0x7f18bc008040 comp rx=0 tx=0).stop 2023-10-16T13:27:45.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.935+0000 7f18d332b700 1 -- 172.21.15.16:0/3688921317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18cc159290 msgr2=0x7f18cc0c15d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:45.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.935+0000 7f18d332b700 1 --2- 172.21.15.16:0/3688921317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18cc159290 0x7f18cc0c15d0 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x7f18c800c4d0 tx=0x7f18c800c7e0 comp rx=0 tx=0).stop 2023-10-16T13:27:45.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.935+0000 7f18d332b700 1 -- 172.21.15.16:0/3688921317 shutdown_connections 2023-10-16T13:27:45.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.935+0000 7f18d332b700 1 --2- 172.21.15.16:0/3688921317 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f18cc0cfcc0 0x7f18cc0bf930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:45.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.935+0000 7f18d332b700 1 --2- 172.21.15.16:0/3688921317 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f18b006b020 0x7f18b006d4e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:45.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.935+0000 7f18d332b700 1 --2- 172.21.15.16:0/3688921317 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f18cc06dbe0 0x7f18cc1f6200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:45.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.935+0000 7f18d332b700 1 --2- 172.21.15.16:0/3688921317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18cc159290 0x7f18cc0c15d0 unknown :-1 s=CLOSED pgs=1927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:45.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.935+0000 7f18d332b700 1 -- 172.21.15.16:0/3688921317 >> 172.21.15.16:0/3688921317 conn(0x7f18cc06bad0 msgr2=0x7f18cc0c3400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:45.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.935+0000 7f18d332b700 1 -- 172.21.15.16:0/3688921317 shutdown_connections 2023-10-16T13:27:45.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:45.935+0000 7f18d332b700 1 -- 172.21.15.16:0/3688921317 wait complete. 2023-10-16T13:27:45.951 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-16T13:27:45.952 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T13:27:46.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.074+0000 7f769d2da700 1 Processor -- start 2023-10-16T13:27:46.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.074+0000 7f769d2da700 1 -- start start 2023-10-16T13:27:46.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.075+0000 7f769d2da700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f769815b280 0x7f76981613e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:46.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.075+0000 7f769d2da700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f76980cc8a0 0x7f769815e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:46.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.075+0000 7f769d2da700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7698161c30 con 0x7f769806a7e0 2023-10-16T13:27:46.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.075+0000 7f769d2da700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7698161f10 con 0x7f769815b280 2023-10-16T13:27:46.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.075+0000 7f769d2da700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76981621f0 con 0x7f76980cc8a0 2023-10-16T13:27:46.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.075+0000 7f76977fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f769806a7e0 0x7f76980d02b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37786/0 (socket says 172.21.15.16:37786) 2023-10-16T13:27:46.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.075+0000 7f76977fe700 1 -- 172.21.15.16:0/1396226043 learned_addr learned my addr 172.21.15.16:0/1396226043 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:27:46.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.075+0000 7f76967fc700 1 --2- 172.21.15.16:0/1396226043 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f76980cc8a0 0x7f769815e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:46.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.075+0000 7f7696ffd700 1 --2- 172.21.15.16:0/1396226043 >> v2:172.21.15.49:3300/0 conn(0x7f769815b280 0x7f76981613e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:46.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.075+0000 7f7695ffb700 1 -- 172.21.15.16:0/1396226043 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 970210523 0 0) 0x7f7698161c30 con 0x7f769806a7e0 2023-10-16T13:27:46.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.075+0000 7f7695ffb700 1 -- 172.21.15.16:0/1396226043 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76800035b0 con 0x7f769806a7e0 2023-10-16T13:27:46.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.075+0000 7f76967fc700 1 -- 172.21.15.16:0/1396226043 >> v2:172.21.15.49:3300/0 conn(0x7f769815b280 msgr2=0x7f76981613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:46.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.075+0000 7f76967fc700 1 --2- 172.21.15.16:0/1396226043 >> v2:172.21.15.49:3300/0 conn(0x7f769815b280 0x7f76981613e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:46.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.075+0000 7f76967fc700 1 -- 172.21.15.16:0/1396226043 >> v1:172.21.15.16:6789/0 conn(0x7f769806a7e0 legacy=0x7f76980d02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:46.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.075+0000 7f76967fc700 1 -- 172.21.15.16:0/1396226043 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f769815df00 con 0x7f76980cc8a0 2023-10-16T13:27:46.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.075+0000 7f76967fc700 1 --2- 172.21.15.16:0/1396226043 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f76980cc8a0 0x7f769815e700 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f768c009840 tx=0x7f768c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a00a91622b75ddae server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:46.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.076+0000 7f7695ffb700 1 -- 172.21.15.16:0/1396226043 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f768c011830 con 0x7f76980cc8a0 2023-10-16T13:27:46.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.076+0000 7f7695ffb700 1 -- 172.21.15.16:0/1396226043 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f768c011990 con 0x7f76980cc8a0 2023-10-16T13:27:46.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.076+0000 7f7695ffb700 1 -- 172.21.15.16:0/1396226043 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f768c011d20 con 0x7f76980cc8a0 2023-10-16T13:27:46.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.076+0000 7f769d2da700 1 -- 172.21.15.16:0/1396226043 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f76980cc8a0 msgr2=0x7f769815e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:46.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.076+0000 7f769d2da700 1 --2- 172.21.15.16:0/1396226043 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f76980cc8a0 0x7f769815e700 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f768c009840 tx=0x7f768c009b50 comp rx=0 tx=0).stop 2023-10-16T13:27:46.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.076+0000 7f769d2da700 1 -- 172.21.15.16:0/1396226043 shutdown_connections 2023-10-16T13:27:46.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.076+0000 7f769d2da700 1 --2- 172.21.15.16:0/1396226043 >> v2:172.21.15.49:3300/0 conn(0x7f769815b280 0x7f76981613e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:46.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.076+0000 7f769d2da700 1 --2- 172.21.15.16:0/1396226043 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f76980cc8a0 0x7f769815e700 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:46.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.076+0000 7f769d2da700 1 -- 172.21.15.16:0/1396226043 >> 172.21.15.16:0/1396226043 conn(0x7f76980b1fd0 msgr2=0x7f76980b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:46.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.076+0000 7f769d2da700 1 -- 172.21.15.16:0/1396226043 shutdown_connections 2023-10-16T13:27:46.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.076+0000 7f769d2da700 1 -- 172.21.15.16:0/1396226043 wait complete. 2023-10-16T13:27:46.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.078+0000 7f769d2da700 1 Processor -- start 2023-10-16T13:27:46.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.078+0000 7f769d2da700 1 -- start start 2023-10-16T13:27:46.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.079+0000 7f769d2da700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f769806a7e0 0x7f76981526b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:46.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.079+0000 7f769d2da700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f76980cc8a0 0x7f769815f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:46.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.079+0000 7f769d2da700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f769815b280 0x7f769815c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:46.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.079+0000 7f769d2da700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76980c4f00 con 0x7f769806a7e0 2023-10-16T13:27:46.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.079+0000 7f769d2da700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76980afc30 con 0x7f769815b280 2023-10-16T13:27:46.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.079+0000 7f769d2da700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76980c6310 con 0x7f76980cc8a0 2023-10-16T13:27:46.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.079+0000 7f7696ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f769806a7e0 0x7f76981526b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:46.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.079+0000 7f7696ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f769806a7e0 0x7f76981526b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48514/0 (socket says 172.21.15.16:48514) 2023-10-16T13:27:46.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.079+0000 7f7696ffd700 1 -- 172.21.15.16:0/3745279098 learned_addr learned my addr 172.21.15.16:0/3745279098 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:46.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.079+0000 7f76967fc700 1 --2- 172.21.15.16:0/3745279098 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f76980cc8a0 0x7f769815f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:46.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.079+0000 7f76977fe700 1 --2- 172.21.15.16:0/3745279098 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f769815b280 0x7f769815c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:46.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.079+0000 7f7696ffd700 1 -- 172.21.15.16:0/3745279098 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f76980cc8a0 msgr2=0x7f769815f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:46.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.079+0000 7f7696ffd700 1 --2- 172.21.15.16:0/3745279098 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f76980cc8a0 0x7f769815f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:46.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.079+0000 7f7696ffd700 1 -- 172.21.15.16:0/3745279098 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f769815b280 msgr2=0x7f769815c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:46.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.079+0000 7f7696ffd700 1 --2- 172.21.15.16:0/3745279098 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f769815b280 0x7f769815c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:46.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.079+0000 7f7696ffd700 1 -- 172.21.15.16:0/3745279098 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f768c0094a0 con 0x7f769806a7e0 2023-10-16T13:27:46.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.079+0000 7f76977fe700 1 --2- 172.21.15.16:0/3745279098 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f769815b280 0x7f769815c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:27:46.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.080+0000 7f7696ffd700 1 --2- 172.21.15.16:0/3745279098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f769806a7e0 0x7f76981526b0 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7f768800e9b0 tx=0x7f768800ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:46.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.080+0000 7f767f7fe700 1 -- 172.21.15.16:0/3745279098 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7688009710 con 0x7f769806a7e0 2023-10-16T13:27:46.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.080+0000 7f767f7fe700 1 -- 172.21.15.16:0/3745279098 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f768800c850 con 0x7f769806a7e0 2023-10-16T13:27:46.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.080+0000 7f769d2da700 1 -- 172.21.15.16:0/3745279098 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76980c65f0 con 0x7f769806a7e0 2023-10-16T13:27:46.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.080+0000 7f767f7fe700 1 -- 172.21.15.16:0/3745279098 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f768800cbc0 con 0x7f769806a7e0 2023-10-16T13:27:46.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.080+0000 7f769d2da700 1 -- 172.21.15.16:0/3745279098 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76980c6980 con 0x7f769806a7e0 2023-10-16T13:27:46.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.081+0000 7f769d2da700 1 -- 172.21.15.16:0/3745279098 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76980aff30 con 0x7f769806a7e0 2023-10-16T13:27:46.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.081+0000 7f767f7fe700 1 -- 172.21.15.16:0/3745279098 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f768800cd20 con 0x7f769806a7e0 2023-10-16T13:27:46.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.082+0000 7f767f7fe700 1 --2- 172.21.15.16:0/3745279098 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f768406b020 0x7f768406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:46.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.082+0000 7f767f7fe700 1 -- 172.21.15.16:0/3745279098 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f7688089970 con 0x7f769806a7e0 2023-10-16T13:27:46.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.082+0000 7f76967fc700 1 --2- 172.21.15.16:0/3745279098 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f768406b020 0x7f768406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:46.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.082+0000 7f76967fc700 1 --2- 172.21.15.16:0/3745279098 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f768406b020 0x7f768406d4e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f768c000c00 tx=0x7f768c008030 comp rx=0 tx=0).ready entity=mgr.9510 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:46.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.086+0000 7f767f7fe700 1 -- 172.21.15.16:0/3745279098 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f76880547f0 con 0x7f769806a7e0 2023-10-16T13:27:46.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.285+0000 7f769d2da700 1 -- 172.21.15.16:0/3745279098 --> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7698161920 con 0x7f768406b020 2023-10-16T13:27:46.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.294+0000 7f767f7fe700 1 -- 172.21.15.16:0/3745279098 <== mgr.9510 v2:172.21.15.16:6832/2329471558 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+290010 (secure 0 0 0) 0x7f768c011630 con 0x7f768406b020 2023-10-16T13:27:46.301 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:27:46.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.298+0000 7f769d2da700 1 -- 172.21.15.16:0/3745279098 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f768406b020 msgr2=0x7f768406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:46.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.298+0000 7f769d2da700 1 --2- 172.21.15.16:0/3745279098 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f768406b020 0x7f768406d4e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f768c000c00 tx=0x7f768c008030 comp rx=0 tx=0).stop 2023-10-16T13:27:46.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.298+0000 7f769d2da700 1 -- 172.21.15.16:0/3745279098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f769806a7e0 msgr2=0x7f76981526b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:46.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.298+0000 7f769d2da700 1 --2- 172.21.15.16:0/3745279098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f769806a7e0 0x7f76981526b0 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7f768800e9b0 tx=0x7f768800ed70 comp rx=0 tx=0).stop 2023-10-16T13:27:46.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.298+0000 7f769d2da700 1 -- 172.21.15.16:0/3745279098 shutdown_connections 2023-10-16T13:27:46.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.298+0000 7f769d2da700 1 --2- 172.21.15.16:0/3745279098 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f768406b020 0x7f768406d4e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:46.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.298+0000 7f769d2da700 1 --2- 172.21.15.16:0/3745279098 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f769815b280 0x7f769815c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:46.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.298+0000 7f769d2da700 1 --2- 172.21.15.16:0/3745279098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f769806a7e0 0x7f76981526b0 unknown :-1 s=CLOSED pgs=1929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:46.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.298+0000 7f769d2da700 1 --2- 172.21.15.16:0/3745279098 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f76980cc8a0 0x7f769815f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:46.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.298+0000 7f769d2da700 1 -- 172.21.15.16:0/3745279098 >> 172.21.15.16:0/3745279098 conn(0x7f76980b1fd0 msgr2=0x7f76980b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:46.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.298+0000 7f769d2da700 1 -- 172.21.15.16:0/3745279098 shutdown_connections 2023-10-16T13:27:46.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.298+0000 7f769d2da700 1 -- 172.21.15.16:0/3745279098 wait complete. 2023-10-16T13:27:46.306 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-10-16T13:27:46.313 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-10-16T13:27:44.817851+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1460,"num_read_kb":2984,"num_write":483,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":309,"ondisk_log_size":309,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":66,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":356632,"kb_used_data":18324,"kb_used_omap":12,"kb_used_meta":338163,"kb_avail":754618088,"statfs":{"total":773094113280,"available":772728922112,"internally_reserved":0,"allocated":18763776,"data_stored":10119168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12740,"internal_metadata":346279484},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.692909"},"pg_stats":[{"pgid":"22.3f","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.100873+0000","last_change":"2023-10-16T13:27:42.098307+0000","last_active":"2023-10-16T13:27:42.100873+0000","last_peered":"2023-10-16T13:27:42.100873+0000","last_clean":"2023-10-16T13:27:42.100873+0000","last_became_active":"2023-10-16T13:27:42.097986+0000","last_became_peered":"2023-10-16T13:27:42.097986+0000","last_unstale":"2023-10-16T13:27:42.100873+0000","last_undegraded":"2023-10-16T13:27:42.100873+0000","last_fullsized":"2023-10-16T13:27:42.100873+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:12:33.359605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.3e","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.088248+0000","last_change":"2023-10-16T13:27:42.085221+0000","last_active":"2023-10-16T13:27:42.088248+0000","last_peered":"2023-10-16T13:27:42.088248+0000","last_clean":"2023-10-16T13:27:42.088248+0000","last_became_active":"2023-10-16T13:27:42.084934+0000","last_became_peered":"2023-10-16T13:27:42.084934+0000","last_unstale":"2023-10-16T13:27:42.088248+0000","last_undegraded":"2023-10-16T13:27:42.088248+0000","last_fullsized":"2023-10-16T13:27:42.088248+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:16:25.214504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.3d","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.090236+0000","last_change":"2023-10-16T13:27:42.089490+0000","last_active":"2023-10-16T13:27:42.090236+0000","last_peered":"2023-10-16T13:27:42.090236+0000","last_clean":"2023-10-16T13:27:42.090236+0000","last_became_active":"2023-10-16T13:27:42.089198+0000","last_became_peered":"2023-10-16T13:27:42.089198+0000","last_unstale":"2023-10-16T13:27:42.090236+0000","last_undegraded":"2023-10-16T13:27:42.090236+0000","last_fullsized":"2023-10-16T13:27:42.090236+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:58:40.655812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.3c","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.091871+0000","last_change":"2023-10-16T13:27:42.087993+0000","last_active":"2023-10-16T13:27:42.091871+0000","last_peered":"2023-10-16T13:27:42.091871+0000","last_clean":"2023-10-16T13:27:42.091871+0000","last_became_active":"2023-10-16T13:27:42.087677+0000","last_became_peered":"2023-10-16T13:27:42.087677+0000","last_unstale":"2023-10-16T13:27:42.091871+0000","last_undegraded":"2023-10-16T13:27:42.091871+0000","last_fullsized":"2023-10-16T13:27:42.091871+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:03:17.841508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.3b","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.091522+0000","last_change":"2023-10-16T13:27:42.090897+0000","last_active":"2023-10-16T13:27:42.091522+0000","last_peered":"2023-10-16T13:27:42.091522+0000","last_clean":"2023-10-16T13:27:42.091522+0000","last_became_active":"2023-10-16T13:27:42.090581+0000","last_became_peered":"2023-10-16T13:27:42.090581+0000","last_unstale":"2023-10-16T13:27:42.091522+0000","last_undegraded":"2023-10-16T13:27:42.091522+0000","last_fullsized":"2023-10-16T13:27:42.091522+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:09:36.254368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3a","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.091586+0000","last_change":"2023-10-16T13:27:42.091034+0000","last_active":"2023-10-16T13:27:42.091586+0000","last_peered":"2023-10-16T13:27:42.091586+0000","last_clean":"2023-10-16T13:27:42.091586+0000","last_became_active":"2023-10-16T13:27:42.090505+0000","last_became_peered":"2023-10-16T13:27:42.090505+0000","last_unstale":"2023-10-16T13:27:42.091586+0000","last_undegraded":"2023-10-16T13:27:42.091586+0000","last_fullsized":"2023-10-16T13:27:42.091586+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:19:14.194081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.39","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.098882+0000","last_change":"2023-10-16T13:27:42.095313+0000","last_active":"2023-10-16T13:27:42.098882+0000","last_peered":"2023-10-16T13:27:42.098882+0000","last_clean":"2023-10-16T13:27:42.098882+0000","last_became_active":"2023-10-16T13:27:42.094911+0000","last_became_peered":"2023-10-16T13:27:42.094911+0000","last_unstale":"2023-10-16T13:27:42.098882+0000","last_undegraded":"2023-10-16T13:27:42.098882+0000","last_fullsized":"2023-10-16T13:27:42.098882+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:17:18.887726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.38","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.098304+0000","last_change":"2023-10-16T13:27:42.093051+0000","last_active":"2023-10-16T13:27:42.098304+0000","last_peered":"2023-10-16T13:27:42.098304+0000","last_clean":"2023-10-16T13:27:42.098304+0000","last_became_active":"2023-10-16T13:27:42.091399+0000","last_became_peered":"2023-10-16T13:27:42.091399+0000","last_unstale":"2023-10-16T13:27:42.098304+0000","last_undegraded":"2023-10-16T13:27:42.098304+0000","last_fullsized":"2023-10-16T13:27:42.098304+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:28:02.926570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.37","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.091534+0000","last_change":"2023-10-16T13:27:42.091029+0000","last_active":"2023-10-16T13:27:42.091534+0000","last_peered":"2023-10-16T13:27:42.091534+0000","last_clean":"2023-10-16T13:27:42.091534+0000","last_became_active":"2023-10-16T13:27:42.090163+0000","last_became_peered":"2023-10-16T13:27:42.090163+0000","last_unstale":"2023-10-16T13:27:42.091534+0000","last_undegraded":"2023-10-16T13:27:42.091534+0000","last_fullsized":"2023-10-16T13:27:42.091534+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:22:18.575043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.36","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.090855+0000","last_change":"2023-10-16T13:27:42.086147+0000","last_active":"2023-10-16T13:27:42.090855+0000","last_peered":"2023-10-16T13:27:42.090855+0000","last_clean":"2023-10-16T13:27:42.090855+0000","last_became_active":"2023-10-16T13:27:42.085841+0000","last_became_peered":"2023-10-16T13:27:42.085841+0000","last_unstale":"2023-10-16T13:27:42.090855+0000","last_undegraded":"2023-10-16T13:27:42.090855+0000","last_fullsized":"2023-10-16T13:27:42.090855+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:50.700408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.35","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.096868+0000","last_change":"2023-10-16T13:27:42.095274+0000","last_active":"2023-10-16T13:27:42.096868+0000","last_peered":"2023-10-16T13:27:42.096868+0000","last_clean":"2023-10-16T13:27:42.096868+0000","last_became_active":"2023-10-16T13:27:42.094986+0000","last_became_peered":"2023-10-16T13:27:42.094986+0000","last_unstale":"2023-10-16T13:27:42.096868+0000","last_undegraded":"2023-10-16T13:27:42.096868+0000","last_fullsized":"2023-10-16T13:27:42.096868+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:13:55.435394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.34","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.091848+0000","last_change":"2023-10-16T13:27:42.085399+0000","last_active":"2023-10-16T13:27:42.091848+0000","last_peered":"2023-10-16T13:27:42.091848+0000","last_clean":"2023-10-16T13:27:42.091848+0000","last_became_active":"2023-10-16T13:27:42.084777+0000","last_became_peered":"2023-10-16T13:27:42.084777+0000","last_unstale":"2023-10-16T13:27:42.091848+0000","last_undegraded":"2023-10-16T13:27:42.091848+0000","last_fullsized":"2023-10-16T13:27:42.091848+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:59:39.691551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.33","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.090960+0000","last_change":"2023-10-16T13:27:42.086264+0000","last_active":"2023-10-16T13:27:42.090960+0000","last_peered":"2023-10-16T13:27:42.090960+0000","last_clean":"2023-10-16T13:27:42.090960+0000","last_became_active":"2023-10-16T13:27:42.085967+0000","last_became_peered":"2023-10-16T13:27:42.085967+0000","last_unstale":"2023-10-16T13:27:42.090960+0000","last_undegraded":"2023-10-16T13:27:42.090960+0000","last_fullsized":"2023-10-16T13:27:42.090960+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:12:55.131919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.32","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.091478+0000","last_change":"2023-10-16T13:27:42.086362+0000","last_active":"2023-10-16T13:27:42.091478+0000","last_peered":"2023-10-16T13:27:42.091478+0000","last_clean":"2023-10-16T13:27:42.091478+0000","last_became_active":"2023-10-16T13:27:42.086071+0000","last_became_peered":"2023-10-16T13:27:42.086071+0000","last_unstale":"2023-10-16T13:27:42.091478+0000","last_undegraded":"2023-10-16T13:27:42.091478+0000","last_fullsized":"2023-10-16T13:27:42.091478+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:22:57.648967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.31","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.097960+0000","last_change":"2023-10-16T13:27:42.095644+0000","last_active":"2023-10-16T13:27:42.097960+0000","last_peered":"2023-10-16T13:27:42.097960+0000","last_clean":"2023-10-16T13:27:42.097960+0000","last_became_active":"2023-10-16T13:27:42.095348+0000","last_became_peered":"2023-10-16T13:27:42.095348+0000","last_unstale":"2023-10-16T13:27:42.097960+0000","last_undegraded":"2023-10-16T13:27:42.097960+0000","last_fullsized":"2023-10-16T13:27:42.097960+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:54:35.805923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.30","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.092656+0000","last_change":"2023-10-16T13:27:42.092368+0000","last_active":"2023-10-16T13:27:42.092656+0000","last_peered":"2023-10-16T13:27:42.092656+0000","last_clean":"2023-10-16T13:27:42.092656+0000","last_became_active":"2023-10-16T13:27:42.092054+0000","last_became_peered":"2023-10-16T13:27:42.092054+0000","last_unstale":"2023-10-16T13:27:42.092656+0000","last_undegraded":"2023-10-16T13:27:42.092656+0000","last_fullsized":"2023-10-16T13:27:42.092656+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:59:05.571247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.2f","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.096705+0000","last_change":"2023-10-16T13:27:42.095131+0000","last_active":"2023-10-16T13:27:42.096705+0000","last_peered":"2023-10-16T13:27:42.096705+0000","last_clean":"2023-10-16T13:27:42.096705+0000","last_became_active":"2023-10-16T13:27:42.094801+0000","last_became_peered":"2023-10-16T13:27:42.094801+0000","last_unstale":"2023-10-16T13:27:42.096705+0000","last_undegraded":"2023-10-16T13:27:42.096705+0000","last_fullsized":"2023-10-16T13:27:42.096705+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:28:44.934920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.2e","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.097774+0000","last_change":"2023-10-16T13:27:42.096463+0000","last_active":"2023-10-16T13:27:42.097774+0000","last_peered":"2023-10-16T13:27:42.097774+0000","last_clean":"2023-10-16T13:27:42.097774+0000","last_became_active":"2023-10-16T13:27:42.095950+0000","last_became_peered":"2023-10-16T13:27:42.095950+0000","last_unstale":"2023-10-16T13:27:42.097774+0000","last_undegraded":"2023-10-16T13:27:42.097774+0000","last_fullsized":"2023-10-16T13:27:42.097774+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:47:32.249424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2d","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.101197+0000","last_change":"2023-10-16T13:27:42.097037+0000","last_active":"2023-10-16T13:27:42.101197+0000","last_peered":"2023-10-16T13:27:42.101197+0000","last_clean":"2023-10-16T13:27:42.101197+0000","last_became_active":"2023-10-16T13:27:42.096591+0000","last_became_peered":"2023-10-16T13:27:42.096591+0000","last_unstale":"2023-10-16T13:27:42.101197+0000","last_undegraded":"2023-10-16T13:27:42.101197+0000","last_fullsized":"2023-10-16T13:27:42.101197+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:50.700408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.2c","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.090726+0000","last_change":"2023-10-16T13:27:42.087996+0000","last_active":"2023-10-16T13:27:42.090726+0000","last_peered":"2023-10-16T13:27:42.090726+0000","last_clean":"2023-10-16T13:27:42.090726+0000","last_became_active":"2023-10-16T13:27:42.087683+0000","last_became_peered":"2023-10-16T13:27:42.087683+0000","last_unstale":"2023-10-16T13:27:42.090726+0000","last_undegraded":"2023-10-16T13:27:42.090726+0000","last_fullsized":"2023-10-16T13:27:42.090726+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:13:33.886086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.2b","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.091154+0000","last_change":"2023-10-16T13:27:42.090377+0000","last_active":"2023-10-16T13:27:42.091154+0000","last_peered":"2023-10-16T13:27:42.091154+0000","last_clean":"2023-10-16T13:27:42.091154+0000","last_became_active":"2023-10-16T13:27:42.090052+0000","last_became_peered":"2023-10-16T13:27:42.090052+0000","last_unstale":"2023-10-16T13:27:42.091154+0000","last_undegraded":"2023-10-16T13:27:42.091154+0000","last_fullsized":"2023-10-16T13:27:42.091154+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:24:09.829869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.2a","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.101355+0000","last_change":"2023-10-16T13:27:42.099466+0000","last_active":"2023-10-16T13:27:42.101355+0000","last_peered":"2023-10-16T13:27:42.101355+0000","last_clean":"2023-10-16T13:27:42.101355+0000","last_became_active":"2023-10-16T13:27:42.098627+0000","last_became_peered":"2023-10-16T13:27:42.098627+0000","last_unstale":"2023-10-16T13:27:42.101355+0000","last_undegraded":"2023-10-16T13:27:42.101355+0000","last_fullsized":"2023-10-16T13:27:42.101355+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:21:10.011990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.29","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.101691+0000","last_change":"2023-10-16T13:27:42.100451+0000","last_active":"2023-10-16T13:27:42.101691+0000","last_peered":"2023-10-16T13:27:42.101691+0000","last_clean":"2023-10-16T13:27:42.101691+0000","last_became_active":"2023-10-16T13:27:42.100031+0000","last_became_peered":"2023-10-16T13:27:42.100031+0000","last_unstale":"2023-10-16T13:27:42.101691+0000","last_undegraded":"2023-10-16T13:27:42.101691+0000","last_fullsized":"2023-10-16T13:27:42.101691+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:08:01.317496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.28","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.086537+0000","last_change":"2023-10-16T13:27:42.083874+0000","last_active":"2023-10-16T13:27:42.086537+0000","last_peered":"2023-10-16T13:27:42.086537+0000","last_clean":"2023-10-16T13:27:42.086537+0000","last_became_active":"2023-10-16T13:27:42.083405+0000","last_became_peered":"2023-10-16T13:27:42.083405+0000","last_unstale":"2023-10-16T13:27:42.086537+0000","last_undegraded":"2023-10-16T13:27:42.086537+0000","last_fullsized":"2023-10-16T13:27:42.086537+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:09:17.969657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.27","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.096759+0000","last_change":"2023-10-16T13:27:42.092281+0000","last_active":"2023-10-16T13:27:42.096759+0000","last_peered":"2023-10-16T13:27:42.096759+0000","last_clean":"2023-10-16T13:27:42.096759+0000","last_became_active":"2023-10-16T13:27:42.091984+0000","last_became_peered":"2023-10-16T13:27:42.091984+0000","last_unstale":"2023-10-16T13:27:42.096759+0000","last_undegraded":"2023-10-16T13:27:42.096759+0000","last_fullsized":"2023-10-16T13:27:42.096759+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:08:07.749142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.26","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.097015+0000","last_change":"2023-10-16T13:27:42.089669+0000","last_active":"2023-10-16T13:27:42.097015+0000","last_peered":"2023-10-16T13:27:42.097015+0000","last_clean":"2023-10-16T13:27:42.097015+0000","last_became_active":"2023-10-16T13:27:42.089350+0000","last_became_peered":"2023-10-16T13:27:42.089350+0000","last_unstale":"2023-10-16T13:27:42.097015+0000","last_undegraded":"2023-10-16T13:27:42.097015+0000","last_fullsized":"2023-10-16T13:27:42.097015+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:41:28.464562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.25","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.097348+0000","last_change":"2023-10-16T13:27:42.094753+0000","last_active":"2023-10-16T13:27:42.097348+0000","last_peered":"2023-10-16T13:27:42.097348+0000","last_clean":"2023-10-16T13:27:42.097348+0000","last_became_active":"2023-10-16T13:27:42.094466+0000","last_became_peered":"2023-10-16T13:27:42.094466+0000","last_unstale":"2023-10-16T13:27:42.097348+0000","last_undegraded":"2023-10-16T13:27:42.097348+0000","last_fullsized":"2023-10-16T13:27:42.097348+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:47:49.660794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.24","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.087709+0000","last_change":"2023-10-16T13:27:42.086360+0000","last_active":"2023-10-16T13:27:42.087709+0000","last_peered":"2023-10-16T13:27:42.087709+0000","last_clean":"2023-10-16T13:27:42.087709+0000","last_became_active":"2023-10-16T13:27:42.086057+0000","last_became_peered":"2023-10-16T13:27:42.086057+0000","last_unstale":"2023-10-16T13:27:42.087709+0000","last_undegraded":"2023-10-16T13:27:42.087709+0000","last_fullsized":"2023-10-16T13:27:42.087709+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:52:58.973737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.23","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.097571+0000","last_change":"2023-10-16T13:27:42.092831+0000","last_active":"2023-10-16T13:27:42.097571+0000","last_peered":"2023-10-16T13:27:42.097571+0000","last_clean":"2023-10-16T13:27:42.097571+0000","last_became_active":"2023-10-16T13:27:42.092082+0000","last_became_peered":"2023-10-16T13:27:42.092082+0000","last_unstale":"2023-10-16T13:27:42.097571+0000","last_undegraded":"2023-10-16T13:27:42.097571+0000","last_fullsized":"2023-10-16T13:27:42.097571+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:27:52.441984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.22","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.098060+0000","last_change":"2023-10-16T13:27:42.097207+0000","last_active":"2023-10-16T13:27:42.098060+0000","last_peered":"2023-10-16T13:27:42.098060+0000","last_clean":"2023-10-16T13:27:42.098060+0000","last_became_active":"2023-10-16T13:27:42.096884+0000","last_became_peered":"2023-10-16T13:27:42.096884+0000","last_unstale":"2023-10-16T13:27:42.098060+0000","last_undegraded":"2023-10-16T13:27:42.098060+0000","last_fullsized":"2023-10-16T13:27:42.098060+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:39:10.604015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.21","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.091043+0000","last_change":"2023-10-16T13:27:42.088075+0000","last_active":"2023-10-16T13:27:42.091043+0000","last_peered":"2023-10-16T13:27:42.091043+0000","last_clean":"2023-10-16T13:27:42.091043+0000","last_became_active":"2023-10-16T13:27:42.087750+0000","last_became_peered":"2023-10-16T13:27:42.087750+0000","last_unstale":"2023-10-16T13:27:42.091043+0000","last_undegraded":"2023-10-16T13:27:42.091043+0000","last_fullsized":"2023-10-16T13:27:42.091043+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:20:26.554357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.20","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.099352+0000","last_change":"2023-10-16T13:27:42.095380+0000","last_active":"2023-10-16T13:27:42.099352+0000","last_peered":"2023-10-16T13:27:42.099352+0000","last_clean":"2023-10-16T13:27:42.099352+0000","last_became_active":"2023-10-16T13:27:42.095029+0000","last_became_peered":"2023-10-16T13:27:42.095029+0000","last_unstale":"2023-10-16T13:27:42.099352+0000","last_undegraded":"2023-10-16T13:27:42.099352+0000","last_fullsized":"2023-10-16T13:27:42.099352+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:40:57.689492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.087610+0000","last_change":"2023-10-16T13:27:42.085122+0000","last_active":"2023-10-16T13:27:42.087610+0000","last_peered":"2023-10-16T13:27:42.087610+0000","last_clean":"2023-10-16T13:27:42.087610+0000","last_became_active":"2023-10-16T13:27:42.084823+0000","last_became_peered":"2023-10-16T13:27:42.084823+0000","last_unstale":"2023-10-16T13:27:42.087610+0000","last_undegraded":"2023-10-16T13:27:42.087610+0000","last_fullsized":"2023-10-16T13:27:42.087610+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:08:14.962839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1f","version":"176'4","reported_seq":15,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.144058+0000","last_change":"2023-10-16T13:27:41.146091+0000","last_active":"2023-10-16T13:27:42.144058+0000","last_peered":"2023-10-16T13:27:42.144058+0000","last_clean":"2023-10-16T13:27:42.144058+0000","last_became_active":"2023-10-16T13:27:41.145307+0000","last_became_peered":"2023-10-16T13:27:41.145307+0000","last_unstale":"2023-10-16T13:27:42.144058+0000","last_undegraded":"2023-10-16T13:27:42.144058+0000","last_fullsized":"2023-10-16T13:27:42.144058+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_clean_scrub_stamp":"2023-10-16T13:27:40.119893+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:24:52.542805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.097923+0000","last_change":"2023-10-16T13:27:42.095553+0000","last_active":"2023-10-16T13:27:42.097923+0000","last_peered":"2023-10-16T13:27:42.097923+0000","last_clean":"2023-10-16T13:27:42.097923+0000","last_became_active":"2023-10-16T13:27:42.095246+0000","last_became_peered":"2023-10-16T13:27:42.095246+0000","last_unstale":"2023-10-16T13:27:42.097923+0000","last_undegraded":"2023-10-16T13:27:42.097923+0000","last_fullsized":"2023-10-16T13:27:42.097923+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:19:11.670430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1e","version":"176'1","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.141029+0000","last_change":"2023-10-16T13:27:41.152486+0000","last_active":"2023-10-16T13:27:42.141029+0000","last_peered":"2023-10-16T13:27:42.141029+0000","last_clean":"2023-10-16T13:27:42.141029+0000","last_became_active":"2023-10-16T13:27:41.152111+0000","last_became_peered":"2023-10-16T13:27:41.152111+0000","last_unstale":"2023-10-16T13:27:42.141029+0000","last_undegraded":"2023-10-16T13:27:42.141029+0000","last_fullsized":"2023-10-16T13:27:42.141029+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_clean_scrub_stamp":"2023-10-16T13:27:40.119893+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:47:48.655752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1e","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.097604+0000","last_change":"2023-10-16T13:27:42.091626+0000","last_active":"2023-10-16T13:27:42.097604+0000","last_peered":"2023-10-16T13:27:42.097604+0000","last_clean":"2023-10-16T13:27:42.097604+0000","last_became_active":"2023-10-16T13:27:42.091086+0000","last_became_peered":"2023-10-16T13:27:42.091086+0000","last_unstale":"2023-10-16T13:27:42.097604+0000","last_undegraded":"2023-10-16T13:27:42.097604+0000","last_fullsized":"2023-10-16T13:27:42.097604+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:11:18.208581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"176'2","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.143421+0000","last_change":"2023-10-16T13:27:41.151661+0000","last_active":"2023-10-16T13:27:42.143421+0000","last_peered":"2023-10-16T13:27:42.143421+0000","last_clean":"2023-10-16T13:27:42.143421+0000","last_became_active":"2023-10-16T13:27:41.151363+0000","last_became_peered":"2023-10-16T13:27:41.151363+0000","last_unstale":"2023-10-16T13:27:42.143421+0000","last_undegraded":"2023-10-16T13:27:42.143421+0000","last_fullsized":"2023-10-16T13:27:42.143421+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_clean_scrub_stamp":"2023-10-16T13:27:40.119893+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:24:08.824826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1f","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.101300+0000","last_change":"2023-10-16T13:27:42.099362+0000","last_active":"2023-10-16T13:27:42.101300+0000","last_peered":"2023-10-16T13:27:42.101300+0000","last_clean":"2023-10-16T13:27:42.101300+0000","last_became_active":"2023-10-16T13:27:42.099060+0000","last_became_peered":"2023-10-16T13:27:42.099060+0000","last_unstale":"2023-10-16T13:27:42.101300+0000","last_undegraded":"2023-10-16T13:27:42.101300+0000","last_fullsized":"2023-10-16T13:27:42.101300+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:09:36.254368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1c","version":"176'2","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.144285+0000","last_change":"2023-10-16T13:27:41.150315+0000","last_active":"2023-10-16T13:27:42.144285+0000","last_peered":"2023-10-16T13:27:42.144285+0000","last_clean":"2023-10-16T13:27:42.144285+0000","last_became_active":"2023-10-16T13:27:41.147732+0000","last_became_peered":"2023-10-16T13:27:41.147732+0000","last_unstale":"2023-10-16T13:27:42.144285+0000","last_undegraded":"2023-10-16T13:27:42.144285+0000","last_fullsized":"2023-10-16T13:27:42.144285+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_clean_scrub_stamp":"2023-10-16T13:27:40.119893+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:54:58.716839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.089183+0000","last_change":"2023-10-16T13:27:42.086114+0000","last_active":"2023-10-16T13:27:42.089183+0000","last_peered":"2023-10-16T13:27:42.089183+0000","last_clean":"2023-10-16T13:27:42.089183+0000","last_became_active":"2023-10-16T13:27:42.085826+0000","last_became_peered":"2023-10-16T13:27:42.085826+0000","last_unstale":"2023-10-16T13:27:42.089183+0000","last_undegraded":"2023-10-16T13:27:42.089183+0000","last_fullsized":"2023-10-16T13:27:42.089183+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:12:55.131919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.1b","version":"176'1","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.141441+0000","last_change":"2023-10-16T13:27:41.146905+0000","last_active":"2023-10-16T13:27:42.141441+0000","last_peered":"2023-10-16T13:27:42.141441+0000","last_clean":"2023-10-16T13:27:42.141441+0000","last_became_active":"2023-10-16T13:27:41.146574+0000","last_became_peered":"2023-10-16T13:27:41.146574+0000","last_unstale":"2023-10-16T13:27:42.141441+0000","last_undegraded":"2023-10-16T13:27:42.141441+0000","last_fullsized":"2023-10-16T13:27:42.141441+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_clean_scrub_stamp":"2023-10-16T13:27:40.119893+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:16:51.195967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.091855+0000","last_change":"2023-10-16T13:27:42.084487+0000","last_active":"2023-10-16T13:27:42.091855+0000","last_peered":"2023-10-16T13:27:42.091855+0000","last_clean":"2023-10-16T13:27:42.091855+0000","last_became_active":"2023-10-16T13:27:42.083614+0000","last_became_peered":"2023-10-16T13:27:42.083614+0000","last_unstale":"2023-10-16T13:27:42.091855+0000","last_undegraded":"2023-10-16T13:27:42.091855+0000","last_fullsized":"2023-10-16T13:27:42.091855+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:18:35.039185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1a","version":"176'1","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.142136+0000","last_change":"2023-10-16T13:27:41.148063+0000","last_active":"2023-10-16T13:27:42.142136+0000","last_peered":"2023-10-16T13:27:42.142136+0000","last_clean":"2023-10-16T13:27:42.142136+0000","last_became_active":"2023-10-16T13:27:41.147752+0000","last_became_peered":"2023-10-16T13:27:41.147752+0000","last_unstale":"2023-10-16T13:27:42.142136+0000","last_undegraded":"2023-10-16T13:27:42.142136+0000","last_fullsized":"2023-10-16T13:27:42.142136+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_clean_scrub_stamp":"2023-10-16T13:27:40.119893+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:13:32.881043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.089631+0000","last_change":"2023-10-16T13:27:42.088376+0000","last_active":"2023-10-16T13:27:42.089631+0000","last_peered":"2023-10-16T13:27:42.089631+0000","last_clean":"2023-10-16T13:27:42.089631+0000","last_became_active":"2023-10-16T13:27:42.088056+0000","last_became_peered":"2023-10-16T13:27:42.088056+0000","last_unstale":"2023-10-16T13:27:42.089631+0000","last_undegraded":"2023-10-16T13:27:42.089631+0000","last_fullsized":"2023-10-16T13:27:42.089631+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:55:15.868250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":11,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.086766+0000","last_change":"2023-10-16T13:27:41.153100+0000","last_active":"2023-10-16T13:27:42.086766+0000","last_peered":"2023-10-16T13:27:42.086766+0000","last_clean":"2023-10-16T13:27:42.086766+0000","last_became_active":"2023-10-16T13:27:41.152820+0000","last_became_peered":"2023-10-16T13:27:41.152820+0000","last_unstale":"2023-10-16T13:27:42.086766+0000","last_undegraded":"2023-10-16T13:27:42.086766+0000","last_fullsized":"2023-10-16T13:27:42.086766+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_clean_scrub_stamp":"2023-10-16T13:27:40.119893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:04:11.225946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1b","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.100932+0000","last_change":"2023-10-16T13:27:42.098791+0000","last_active":"2023-10-16T13:27:42.100932+0000","last_peered":"2023-10-16T13:27:42.100932+0000","last_clean":"2023-10-16T13:27:42.100932+0000","last_became_active":"2023-10-16T13:27:42.098456+0000","last_became_peered":"2023-10-16T13:27:42.098456+0000","last_unstale":"2023-10-16T13:27:42.100932+0000","last_undegraded":"2023-10-16T13:27:42.100932+0000","last_fullsized":"2023-10-16T13:27:42.100932+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:26:26.242188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":11,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.074653+0000","last_change":"2023-10-16T13:27:41.146877+0000","last_active":"2023-10-16T13:27:42.074653+0000","last_peered":"2023-10-16T13:27:42.074653+0000","last_clean":"2023-10-16T13:27:42.074653+0000","last_became_active":"2023-10-16T13:27:41.146529+0000","last_became_peered":"2023-10-16T13:27:41.146529+0000","last_unstale":"2023-10-16T13:27:42.074653+0000","last_undegraded":"2023-10-16T13:27:42.074653+0000","last_fullsized":"2023-10-16T13:27:42.074653+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_clean_scrub_stamp":"2023-10-16T13:27:40.119893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:11:56.094808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.14","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.101025+0000","last_change":"2023-10-16T13:27:42.096512+0000","last_active":"2023-10-16T13:27:42.101025+0000","last_peered":"2023-10-16T13:27:42.101025+0000","last_clean":"2023-10-16T13:27:42.101025+0000","last_became_active":"2023-10-16T13:27:42.096224+0000","last_became_peered":"2023-10-16T13:27:42.096224+0000","last_unstale":"2023-10-16T13:27:42.101025+0000","last_undegraded":"2023-10-16T13:27:42.101025+0000","last_fullsized":"2023-10-16T13:27:42.101025+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:39:53.629749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":11,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.085719+0000","last_change":"2023-10-16T13:27:41.156079+0000","last_active":"2023-10-16T13:27:42.085719+0000","last_peered":"2023-10-16T13:27:42.085719+0000","last_clean":"2023-10-16T13:27:42.085719+0000","last_became_active":"2023-10-16T13:27:41.155614+0000","last_became_peered":"2023-10-16T13:27:41.155614+0000","last_unstale":"2023-10-16T13:27:42.085719+0000","last_undegraded":"2023-10-16T13:27:42.085719+0000","last_fullsized":"2023-10-16T13:27:42.085719+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_clean_scrub_stamp":"2023-10-16T13:27:40.119893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:19:13.189038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.096849+0000","last_change":"2023-10-16T13:27:42.094983+0000","last_active":"2023-10-16T13:27:42.096849+0000","last_peered":"2023-10-16T13:27:42.096849+0000","last_clean":"2023-10-16T13:27:42.096849+0000","last_became_active":"2023-10-16T13:27:42.094609+0000","last_became_peered":"2023-10-16T13:27:42.094609+0000","last_unstale":"2023-10-16T13:27:42.096849+0000","last_undegraded":"2023-10-16T13:27:42.096849+0000","last_fullsized":"2023-10-16T13:27:42.096849+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:17:53.030835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":11,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.079400+0000","last_change":"2023-10-16T13:27:41.146063+0000","last_active":"2023-10-16T13:27:42.079400+0000","last_peered":"2023-10-16T13:27:42.079400+0000","last_clean":"2023-10-16T13:27:42.079400+0000","last_became_active":"2023-10-16T13:27:41.145651+0000","last_became_peered":"2023-10-16T13:27:41.145651+0000","last_unstale":"2023-10-16T13:27:42.079400+0000","last_undegraded":"2023-10-16T13:27:42.079400+0000","last_fullsized":"2023-10-16T13:27:42.079400+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_clean_scrub_stamp":"2023-10-16T13:27:40.119893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:23:38.428549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.097441+0000","last_change":"2023-10-16T13:27:42.096248+0000","last_active":"2023-10-16T13:27:42.097441+0000","last_peered":"2023-10-16T13:27:42.097441+0000","last_clean":"2023-10-16T13:27:42.097441+0000","last_became_active":"2023-10-16T13:27:42.095590+0000","last_became_peered":"2023-10-16T13:27:42.095590+0000","last_unstale":"2023-10-16T13:27:42.097441+0000","last_undegraded":"2023-10-16T13:27:42.097441+0000","last_fullsized":"2023-10-16T13:27:42.097441+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:27:13.595264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.15","version":"176'3","reported_seq":14,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.144116+0000","last_change":"2023-10-16T13:27:41.145596+0000","last_active":"2023-10-16T13:27:42.144116+0000","last_peered":"2023-10-16T13:27:42.144116+0000","last_clean":"2023-10-16T13:27:42.144116+0000","last_became_active":"2023-10-16T13:27:41.145220+0000","last_became_peered":"2023-10-16T13:27:41.145220+0000","last_unstale":"2023-10-16T13:27:42.144116+0000","last_undegraded":"2023-10-16T13:27:42.144116+0000","last_fullsized":"2023-10-16T13:27:42.144116+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_clean_scrub_stamp":"2023-10-16T13:27:40.119893+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:06:55.303636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.10","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.098342+0000","last_change":"2023-10-16T13:27:42.093717+0000","last_active":"2023-10-16T13:27:42.098342+0000","last_peered":"2023-10-16T13:27:42.098342+0000","last_clean":"2023-10-16T13:27:42.098342+0000","last_became_active":"2023-10-16T13:27:42.093362+0000","last_became_peered":"2023-10-16T13:27:42.093362+0000","last_unstale":"2023-10-16T13:27:42.098342+0000","last_undegraded":"2023-10-16T13:27:42.098342+0000","last_fullsized":"2023-10-16T13:27:42.098342+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:59:52.745820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.13","version":"176'2","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.143549+0000","last_change":"2023-10-16T13:27:41.145858+0000","last_active":"2023-10-16T13:27:42.143549+0000","last_peered":"2023-10-16T13:27:42.143549+0000","last_clean":"2023-10-16T13:27:42.143549+0000","last_became_active":"2023-10-16T13:27:41.145566+0000","last_became_peered":"2023-10-16T13:27:41.145566+0000","last_unstale":"2023-10-16T13:27:42.143549+0000","last_undegraded":"2023-10-16T13:27:42.143549+0000","last_fullsized":"2023-10-16T13:27:42.143549+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_clean_scrub_stamp":"2023-10-16T13:27:40.119893+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:18:21.429859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.085464+0000","last_change":"2023-10-16T13:27:42.084461+0000","last_active":"2023-10-16T13:27:42.085464+0000","last_peered":"2023-10-16T13:27:42.085464+0000","last_clean":"2023-10-16T13:27:42.085464+0000","last_became_active":"2023-10-16T13:27:42.083999+0000","last_became_peered":"2023-10-16T13:27:42.083999+0000","last_unstale":"2023-10-16T13:27:42.085464+0000","last_undegraded":"2023-10-16T13:27:42.085464+0000","last_fullsized":"2023-10-16T13:27:42.085464+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:35:31.624279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":11,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.080002+0000","last_change":"2023-10-16T13:27:41.153282+0000","last_active":"2023-10-16T13:27:42.080002+0000","last_peered":"2023-10-16T13:27:42.080002+0000","last_clean":"2023-10-16T13:27:42.080002+0000","last_became_active":"2023-10-16T13:27:41.147632+0000","last_became_peered":"2023-10-16T13:27:41.147632+0000","last_unstale":"2023-10-16T13:27:42.080002+0000","last_undegraded":"2023-10-16T13:27:42.080002+0000","last_fullsized":"2023-10-16T13:27:42.080002+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_clean_scrub_stamp":"2023-10-16T13:27:40.119893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:43:15.770153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.090714+0000","last_change":"2023-10-16T13:27:42.084817+0000","last_active":"2023-10-16T13:27:42.090714+0000","last_peered":"2023-10-16T13:27:42.090714+0000","last_clean":"2023-10-16T13:27:42.090714+0000","last_became_active":"2023-10-16T13:27:42.083457+0000","last_became_peered":"2023-10-16T13:27:42.083457+0000","last_unstale":"2023-10-16T13:27:42.090714+0000","last_undegraded":"2023-10-16T13:27:42.090714+0000","last_fullsized":"2023-10-16T13:27:42.090714+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:58:02.857014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":11,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.081089+0000","last_change":"2023-10-16T13:27:41.151560+0000","last_active":"2023-10-16T13:27:42.081089+0000","last_peered":"2023-10-16T13:27:42.081089+0000","last_clean":"2023-10-16T13:27:42.081089+0000","last_became_active":"2023-10-16T13:27:41.151104+0000","last_became_peered":"2023-10-16T13:27:41.151104+0000","last_unstale":"2023-10-16T13:27:42.081089+0000","last_undegraded":"2023-10-16T13:27:42.081089+0000","last_fullsized":"2023-10-16T13:27:42.081089+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_clean_scrub_stamp":"2023-10-16T13:27:40.119893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:29:05.731531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.083161+0000","last_change":"2023-10-16T13:27:42.080030+0000","last_active":"2023-10-16T13:27:42.083161+0000","last_peered":"2023-10-16T13:27:42.083161+0000","last_clean":"2023-10-16T13:27:42.083161+0000","last_became_active":"2023-10-16T13:27:42.079717+0000","last_became_peered":"2023-10-16T13:27:42.079717+0000","last_unstale":"2023-10-16T13:27:42.083161+0000","last_undegraded":"2023-10-16T13:27:42.083161+0000","last_fullsized":"2023-10-16T13:27:42.083161+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:31:43.138772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.10","version":"176'8","reported_seq":19,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.143973+0000","last_change":"2023-10-16T13:27:41.152454+0000","last_active":"2023-10-16T13:27:42.143973+0000","last_peered":"2023-10-16T13:27:42.143973+0000","last_clean":"2023-10-16T13:27:42.143973+0000","last_became_active":"2023-10-16T13:27:41.152053+0000","last_became_peered":"2023-10-16T13:27:41.152053+0000","last_unstale":"2023-10-16T13:27:42.143973+0000","last_undegraded":"2023-10-16T13:27:42.143973+0000","last_fullsized":"2023-10-16T13:27:42.143973+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_clean_scrub_stamp":"2023-10-16T13:27:40.119893+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:24:52.276480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.099681+0000","last_change":"2023-10-16T13:27:42.094617+0000","last_active":"2023-10-16T13:27:42.099681+0000","last_peered":"2023-10-16T13:27:42.099681+0000","last_clean":"2023-10-16T13:27:42.099681+0000","last_became_active":"2023-10-16T13:27:42.094303+0000","last_became_peered":"2023-10-16T13:27:42.094303+0000","last_unstale":"2023-10-16T13:27:42.099681+0000","last_undegraded":"2023-10-16T13:27:42.099681+0000","last_fullsized":"2023-10-16T13:27:42.099681+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:18:22.434902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.f","version":"176'1","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.140880+0000","last_change":"2023-10-16T13:27:41.154407+0000","last_active":"2023-10-16T13:27:42.140880+0000","last_peered":"2023-10-16T13:27:42.140880+0000","last_clean":"2023-10-16T13:27:42.140880+0000","last_became_active":"2023-10-16T13:27:41.154160+0000","last_became_peered":"2023-10-16T13:27:41.154160+0000","last_unstale":"2023-10-16T13:27:42.140880+0000","last_undegraded":"2023-10-16T13:27:42.140880+0000","last_fullsized":"2023-10-16T13:27:42.140880+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_clean_scrub_stamp":"2023-10-16T13:27:40.119893+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:10:58.951084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.d","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.101176+0000","last_change":"2023-10-16T13:27:42.096862+0000","last_active":"2023-10-16T13:27:42.101176+0000","last_peered":"2023-10-16T13:27:42.101176+0000","last_clean":"2023-10-16T13:27:42.101176+0000","last_became_active":"2023-10-16T13:27:42.096215+0000","last_became_peered":"2023-10-16T13:27:42.096215+0000","last_unstale":"2023-10-16T13:27:42.101176+0000","last_undegraded":"2023-10-16T13:27:42.101176+0000","last_fullsized":"2023-10-16T13:27:42.101176+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:24:53.547848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.e","version":"176'2","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.143080+0000","last_change":"2023-10-16T13:27:41.154236+0000","last_active":"2023-10-16T13:27:42.143080+0000","last_peered":"2023-10-16T13:27:42.143080+0000","last_clean":"2023-10-16T13:27:42.143080+0000","last_became_active":"2023-10-16T13:27:41.153914+0000","last_became_peered":"2023-10-16T13:27:41.153914+0000","last_unstale":"2023-10-16T13:27:42.143080+0000","last_undegraded":"2023-10-16T13:27:42.143080+0000","last_fullsized":"2023-10-16T13:27:42.143080+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_clean_scrub_stamp":"2023-10-16T13:27:40.119893+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:11:56.094808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.e","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.097036+0000","last_change":"2023-10-16T13:27:42.093054+0000","last_active":"2023-10-16T13:27:42.097036+0000","last_peered":"2023-10-16T13:27:42.097036+0000","last_clean":"2023-10-16T13:27:42.097036+0000","last_became_active":"2023-10-16T13:27:42.092757+0000","last_became_peered":"2023-10-16T13:27:42.092757+0000","last_unstale":"2023-10-16T13:27:42.097036+0000","last_undegraded":"2023-10-16T13:27:42.097036+0000","last_fullsized":"2023-10-16T13:27:42.097036+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:22:13.821974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.d","version":"176'1","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.142694+0000","last_change":"2023-10-16T13:27:41.150983+0000","last_active":"2023-10-16T13:27:42.142694+0000","last_peered":"2023-10-16T13:27:42.142694+0000","last_clean":"2023-10-16T13:27:42.142694+0000","last_became_active":"2023-10-16T13:27:41.150669+0000","last_became_peered":"2023-10-16T13:27:41.150669+0000","last_unstale":"2023-10-16T13:27:42.142694+0000","last_undegraded":"2023-10-16T13:27:42.142694+0000","last_fullsized":"2023-10-16T13:27:42.142694+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_clean_scrub_stamp":"2023-10-16T13:27:40.119893+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:38:01.595105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.f","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.099721+0000","last_change":"2023-10-16T13:27:42.096503+0000","last_active":"2023-10-16T13:27:42.099721+0000","last_peered":"2023-10-16T13:27:42.099721+0000","last_clean":"2023-10-16T13:27:42.099721+0000","last_became_active":"2023-10-16T13:27:42.096173+0000","last_became_peered":"2023-10-16T13:27:42.096173+0000","last_unstale":"2023-10-16T13:27:42.099721+0000","last_undegraded":"2023-10-16T13:27:42.099721+0000","last_fullsized":"2023-10-16T13:27:42.099721+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:10.823738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":11,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.080588+0000","last_change":"2023-10-16T13:27:41.150504+0000","last_active":"2023-10-16T13:27:42.080588+0000","last_peered":"2023-10-16T13:27:42.080588+0000","last_clean":"2023-10-16T13:27:42.080588+0000","last_became_active":"2023-10-16T13:27:41.149304+0000","last_became_peered":"2023-10-16T13:27:41.149304+0000","last_unstale":"2023-10-16T13:27:42.080588+0000","last_undegraded":"2023-10-16T13:27:42.080588+0000","last_fullsized":"2023-10-16T13:27:42.080588+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_clean_scrub_stamp":"2023-10-16T13:27:40.119893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:03:28.153513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.101270+0000","last_change":"2023-10-16T13:27:42.096736+0000","last_active":"2023-10-16T13:27:42.101270+0000","last_peered":"2023-10-16T13:27:42.101270+0000","last_clean":"2023-10-16T13:27:42.101270+0000","last_became_active":"2023-10-16T13:27:42.096417+0000","last_became_peered":"2023-10-16T13:27:42.096417+0000","last_unstale":"2023-10-16T13:27:42.101270+0000","last_undegraded":"2023-10-16T13:27:42.101270+0000","last_fullsized":"2023-10-16T13:27:42.101270+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:22:18.575043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.b","version":"176'2","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.143981+0000","last_change":"2023-10-16T13:27:41.149613+0000","last_active":"2023-10-16T13:27:42.143981+0000","last_peered":"2023-10-16T13:27:42.143981+0000","last_clean":"2023-10-16T13:27:42.143981+0000","last_became_active":"2023-10-16T13:27:41.148952+0000","last_became_peered":"2023-10-16T13:27:41.148952+0000","last_unstale":"2023-10-16T13:27:42.143981+0000","last_undegraded":"2023-10-16T13:27:42.143981+0000","last_fullsized":"2023-10-16T13:27:42.143981+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_clean_scrub_stamp":"2023-10-16T13:27:40.119893+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:56:46.969224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.097980+0000","last_change":"2023-10-16T13:27:42.096477+0000","last_active":"2023-10-16T13:27:42.097980+0000","last_peered":"2023-10-16T13:27:42.097980+0000","last_clean":"2023-10-16T13:27:42.097980+0000","last_became_active":"2023-10-16T13:27:42.095870+0000","last_became_peered":"2023-10-16T13:27:42.095870+0000","last_unstale":"2023-10-16T13:27:42.097980+0000","last_undegraded":"2023-10-16T13:27:42.097980+0000","last_fullsized":"2023-10-16T13:27:42.097980+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:22:35.735362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":11,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.085410+0000","last_change":"2023-10-16T13:27:41.152283+0000","last_active":"2023-10-16T13:27:42.085410+0000","last_peered":"2023-10-16T13:27:42.085410+0000","last_clean":"2023-10-16T13:27:42.085410+0000","last_became_active":"2023-10-16T13:27:41.151990+0000","last_became_peered":"2023-10-16T13:27:41.151990+0000","last_unstale":"2023-10-16T13:27:42.085410+0000","last_undegraded":"2023-10-16T13:27:42.085410+0000","last_fullsized":"2023-10-16T13:27:42.085410+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_clean_scrub_stamp":"2023-10-16T13:27:40.119893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:11:03.602352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.096884+0000","last_change":"2023-10-16T13:27:42.092196+0000","last_active":"2023-10-16T13:27:42.096884+0000","last_peered":"2023-10-16T13:27:42.096884+0000","last_clean":"2023-10-16T13:27:42.096884+0000","last_became_active":"2023-10-16T13:27:42.091909+0000","last_became_peered":"2023-10-16T13:27:42.091909+0000","last_unstale":"2023-10-16T13:27:42.096884+0000","last_undegraded":"2023-10-16T13:27:42.096884+0000","last_fullsized":"2023-10-16T13:27:42.096884+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:10:22.490244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":11,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.079521+0000","last_change":"2023-10-16T13:27:41.147022+0000","last_active":"2023-10-16T13:27:42.079521+0000","last_peered":"2023-10-16T13:27:42.079521+0000","last_clean":"2023-10-16T13:27:42.079521+0000","last_became_active":"2023-10-16T13:27:41.146669+0000","last_became_peered":"2023-10-16T13:27:41.146669+0000","last_unstale":"2023-10-16T13:27:42.079521+0000","last_undegraded":"2023-10-16T13:27:42.079521+0000","last_fullsized":"2023-10-16T13:27:42.079521+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_clean_scrub_stamp":"2023-10-16T13:27:40.119893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:58:01.851972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.097967+0000","last_change":"2023-10-16T13:27:42.095760+0000","last_active":"2023-10-16T13:27:42.097967+0000","last_peered":"2023-10-16T13:27:42.097967+0000","last_clean":"2023-10-16T13:27:42.097967+0000","last_became_active":"2023-10-16T13:27:42.095454+0000","last_became_peered":"2023-10-16T13:27:42.095454+0000","last_unstale":"2023-10-16T13:27:42.097967+0000","last_undegraded":"2023-10-16T13:27:42.097967+0000","last_fullsized":"2023-10-16T13:27:42.097967+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:28:03.681820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":11,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.087111+0000","last_change":"2023-10-16T13:27:41.148257+0000","last_active":"2023-10-16T13:27:42.087111+0000","last_peered":"2023-10-16T13:27:42.087111+0000","last_clean":"2023-10-16T13:27:42.087111+0000","last_became_active":"2023-10-16T13:27:41.145869+0000","last_became_peered":"2023-10-16T13:27:41.145869+0000","last_unstale":"2023-10-16T13:27:42.087111+0000","last_undegraded":"2023-10-16T13:27:42.087111+0000","last_fullsized":"2023-10-16T13:27:42.087111+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_clean_scrub_stamp":"2023-10-16T13:27:40.119893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:05:26.666331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.098733+0000","last_change":"2023-10-16T13:27:42.098423+0000","last_active":"2023-10-16T13:27:42.098733+0000","last_peered":"2023-10-16T13:27:42.098733+0000","last_clean":"2023-10-16T13:27:42.098733+0000","last_became_active":"2023-10-16T13:27:42.098164+0000","last_became_peered":"2023-10-16T13:27:42.098164+0000","last_unstale":"2023-10-16T13:27:42.098733+0000","last_undegraded":"2023-10-16T13:27:42.098733+0000","last_fullsized":"2023-10-16T13:27:42.098733+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:31:38.628796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.7","version":"176'2","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.142873+0000","last_change":"2023-10-16T13:27:41.149885+0000","last_active":"2023-10-16T13:27:42.142873+0000","last_peered":"2023-10-16T13:27:42.142873+0000","last_clean":"2023-10-16T13:27:42.142873+0000","last_became_active":"2023-10-16T13:27:41.147708+0000","last_became_peered":"2023-10-16T13:27:41.147708+0000","last_unstale":"2023-10-16T13:27:42.142873+0000","last_undegraded":"2023-10-16T13:27:42.142873+0000","last_fullsized":"2023-10-16T13:27:42.142873+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_clean_scrub_stamp":"2023-10-16T13:27:40.119893+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:01:24.701443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.091426+0000","last_change":"2023-10-16T13:27:42.086530+0000","last_active":"2023-10-16T13:27:42.091426+0000","last_peered":"2023-10-16T13:27:42.091426+0000","last_clean":"2023-10-16T13:27:42.091426+0000","last_became_active":"2023-10-16T13:27:42.086204+0000","last_became_peered":"2023-10-16T13:27:42.086204+0000","last_unstale":"2023-10-16T13:27:42.091426+0000","last_undegraded":"2023-10-16T13:27:42.091426+0000","last_fullsized":"2023-10-16T13:27:42.091426+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:29:14.085622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":11,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.079242+0000","last_change":"2023-10-16T13:27:41.154787+0000","last_active":"2023-10-16T13:27:42.079242+0000","last_peered":"2023-10-16T13:27:42.079242+0000","last_clean":"2023-10-16T13:27:42.079242+0000","last_became_active":"2023-10-16T13:27:41.154373+0000","last_became_peered":"2023-10-16T13:27:41.154373+0000","last_unstale":"2023-10-16T13:27:42.079242+0000","last_undegraded":"2023-10-16T13:27:42.079242+0000","last_fullsized":"2023-10-16T13:27:42.079242+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_clean_scrub_stamp":"2023-10-16T13:27:40.119893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:52:52.209689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.101302+0000","last_change":"2023-10-16T13:27:42.099248+0000","last_active":"2023-10-16T13:27:42.101302+0000","last_peered":"2023-10-16T13:27:42.101302+0000","last_clean":"2023-10-16T13:27:42.101302+0000","last_became_active":"2023-10-16T13:27:42.098477+0000","last_became_peered":"2023-10-16T13:27:42.098477+0000","last_unstale":"2023-10-16T13:27:42.101302+0000","last_undegraded":"2023-10-16T13:27:42.101302+0000","last_fullsized":"2023-10-16T13:27:42.101302+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:17:05.181133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":11,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.076376+0000","last_change":"2023-10-16T13:27:41.150678+0000","last_active":"2023-10-16T13:27:42.076376+0000","last_peered":"2023-10-16T13:27:42.076376+0000","last_clean":"2023-10-16T13:27:42.076376+0000","last_became_active":"2023-10-16T13:27:41.150185+0000","last_became_peered":"2023-10-16T13:27:41.150185+0000","last_unstale":"2023-10-16T13:27:42.076376+0000","last_undegraded":"2023-10-16T13:27:42.076376+0000","last_fullsized":"2023-10-16T13:27:42.076376+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_clean_scrub_stamp":"2023-10-16T13:27:40.119893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:05:37.745445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.085618+0000","last_change":"2023-10-16T13:27:42.083360+0000","last_active":"2023-10-16T13:27:42.085618+0000","last_peered":"2023-10-16T13:27:42.085618+0000","last_clean":"2023-10-16T13:27:42.085618+0000","last_became_active":"2023-10-16T13:27:42.083059+0000","last_became_peered":"2023-10-16T13:27:42.083059+0000","last_unstale":"2023-10-16T13:27:42.085618+0000","last_undegraded":"2023-10-16T13:27:42.085618+0000","last_fullsized":"2023-10-16T13:27:42.085618+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:51:51.674630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":11,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.075287+0000","last_change":"2023-10-16T13:27:41.148045+0000","last_active":"2023-10-16T13:27:42.075287+0000","last_peered":"2023-10-16T13:27:42.075287+0000","last_clean":"2023-10-16T13:27:42.075287+0000","last_became_active":"2023-10-16T13:27:41.146902+0000","last_became_peered":"2023-10-16T13:27:41.146902+0000","last_unstale":"2023-10-16T13:27:42.075287+0000","last_undegraded":"2023-10-16T13:27:42.075287+0000","last_fullsized":"2023-10-16T13:27:42.075287+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_clean_scrub_stamp":"2023-10-16T13:27:40.119893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:10:21.485201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.098283+0000","last_change":"2023-10-16T13:27:42.092375+0000","last_active":"2023-10-16T13:27:42.098283+0000","last_peered":"2023-10-16T13:27:42.098283+0000","last_clean":"2023-10-16T13:27:42.098283+0000","last_became_active":"2023-10-16T13:27:42.092091+0000","last_became_peered":"2023-10-16T13:27:42.092091+0000","last_unstale":"2023-10-16T13:27:42.098283+0000","last_undegraded":"2023-10-16T13:27:42.098283+0000","last_fullsized":"2023-10-16T13:27:42.098283+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:07:07.625727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":11,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.079917+0000","last_change":"2023-10-16T13:27:41.147629+0000","last_active":"2023-10-16T13:27:42.079917+0000","last_peered":"2023-10-16T13:27:42.079917+0000","last_clean":"2023-10-16T13:27:42.079917+0000","last_became_active":"2023-10-16T13:27:41.147127+0000","last_became_peered":"2023-10-16T13:27:41.147127+0000","last_unstale":"2023-10-16T13:27:42.079917+0000","last_undegraded":"2023-10-16T13:27:42.079917+0000","last_fullsized":"2023-10-16T13:27:42.079917+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_clean_scrub_stamp":"2023-10-16T13:27:40.119893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:37:16.544959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.091833+0000","last_change":"2023-10-16T13:27:42.083297+0000","last_active":"2023-10-16T13:27:42.091833+0000","last_peered":"2023-10-16T13:27:42.091833+0000","last_clean":"2023-10-16T13:27:42.091833+0000","last_became_active":"2023-10-16T13:27:42.082995+0000","last_became_peered":"2023-10-16T13:27:42.082995+0000","last_unstale":"2023-10-16T13:27:42.091833+0000","last_undegraded":"2023-10-16T13:27:42.091833+0000","last_fullsized":"2023-10-16T13:27:42.091833+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:19:54.622529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":11,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.081204+0000","last_change":"2023-10-16T13:27:41.137848+0000","last_active":"2023-10-16T13:27:42.081204+0000","last_peered":"2023-10-16T13:27:42.081204+0000","last_clean":"2023-10-16T13:27:42.081204+0000","last_became_active":"2023-10-16T13:27:41.137530+0000","last_became_peered":"2023-10-16T13:27:41.137530+0000","last_unstale":"2023-10-16T13:27:42.081204+0000","last_undegraded":"2023-10-16T13:27:42.081204+0000","last_fullsized":"2023-10-16T13:27:42.081204+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_clean_scrub_stamp":"2023-10-16T13:27:40.119893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:30:00.346674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.082904+0000","last_change":"2023-10-16T13:27:42.081718+0000","last_active":"2023-10-16T13:27:42.082904+0000","last_peered":"2023-10-16T13:27:42.082904+0000","last_clean":"2023-10-16T13:27:42.082904+0000","last_became_active":"2023-10-16T13:27:42.081414+0000","last_became_peered":"2023-10-16T13:27:42.081414+0000","last_unstale":"2023-10-16T13:27:42.082904+0000","last_undegraded":"2023-10-16T13:27:42.082904+0000","last_fullsized":"2023-10-16T13:27:42.082904+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:35:13.025389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":11,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.086849+0000","last_change":"2023-10-16T13:27:41.144713+0000","last_active":"2023-10-16T13:27:42.086849+0000","last_peered":"2023-10-16T13:27:42.086849+0000","last_clean":"2023-10-16T13:27:42.086849+0000","last_became_active":"2023-10-16T13:27:41.144410+0000","last_became_peered":"2023-10-16T13:27:41.144410+0000","last_unstale":"2023-10-16T13:27:42.086849+0000","last_undegraded":"2023-10-16T13:27:42.086849+0000","last_fullsized":"2023-10-16T13:27:42.086849+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_clean_scrub_stamp":"2023-10-16T13:27:40.119893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:12:12.493088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.097592+0000","last_change":"2023-10-16T13:27:42.093052+0000","last_active":"2023-10-16T13:27:42.097592+0000","last_peered":"2023-10-16T13:27:42.097592+0000","last_clean":"2023-10-16T13:27:42.097592+0000","last_became_active":"2023-10-16T13:27:42.091691+0000","last_became_peered":"2023-10-16T13:27:42.091691+0000","last_unstale":"2023-10-16T13:27:42.097592+0000","last_undegraded":"2023-10-16T13:27:42.097592+0000","last_fullsized":"2023-10-16T13:27:42.097592+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:13:38.435362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":11,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.085352+0000","last_change":"2023-10-16T13:27:41.155093+0000","last_active":"2023-10-16T13:27:42.085352+0000","last_peered":"2023-10-16T13:27:42.085352+0000","last_clean":"2023-10-16T13:27:42.085352+0000","last_became_active":"2023-10-16T13:27:41.154818+0000","last_became_peered":"2023-10-16T13:27:41.154818+0000","last_unstale":"2023-10-16T13:27:42.085352+0000","last_undegraded":"2023-10-16T13:27:42.085352+0000","last_fullsized":"2023-10-16T13:27:42.085352+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_clean_scrub_stamp":"2023-10-16T13:27:40.119893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:51:50.669587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.097790+0000","last_change":"2023-10-16T13:27:42.091585+0000","last_active":"2023-10-16T13:27:42.097790+0000","last_peered":"2023-10-16T13:27:42.097790+0000","last_clean":"2023-10-16T13:27:42.097790+0000","last_became_active":"2023-10-16T13:27:42.091244+0000","last_became_peered":"2023-10-16T13:27:42.091244+0000","last_unstale":"2023-10-16T13:27:42.097790+0000","last_undegraded":"2023-10-16T13:27:42.097790+0000","last_fullsized":"2023-10-16T13:27:42.097790+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:37:51.020671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.14","version":"176'6","reported_seq":17,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.145439+0000","last_change":"2023-10-16T13:27:41.149614+0000","last_active":"2023-10-16T13:27:42.145439+0000","last_peered":"2023-10-16T13:27:42.145439+0000","last_clean":"2023-10-16T13:27:42.145439+0000","last_became_active":"2023-10-16T13:27:41.149202+0000","last_became_peered":"2023-10-16T13:27:41.149202+0000","last_unstale":"2023-10-16T13:27:42.145439+0000","last_undegraded":"2023-10-16T13:27:42.145439+0000","last_fullsized":"2023-10-16T13:27:42.145439+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_clean_scrub_stamp":"2023-10-16T13:27:40.119893+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:38:44.970839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"166'271","reported_seq":881,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-16T13:27:41.138752+0000","last_change":"2023-10-16T12:10:34.746820+0000","last_active":"2023-10-16T13:27:41.138752+0000","last_peered":"2023-10-16T13:27:41.138752+0000","last_clean":"2023-10-16T13:27:41.138752+0000","last_became_active":"2023-10-16T12:10:34.745574+0000","last_became_peered":"2023-10-16T12:10:34.745574+0000","last_unstale":"2023-10-16T13:27:41.138752+0000","last_undegraded":"2023-10-16T13:27:41.138752+0000","last_fullsized":"2023-10-16T13:27:41.138752+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_clean_scrub_stamp":"2023-10-16T12:10:33.577111+0000","objects_scrubbed":0,"log_size":271,"log_dups_size":0,"ondisk_log_size":271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:14.461986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1460,"num_read_kb":2984,"num_write":439,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":21,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":22,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1460,"num_read_kb":2984,"num_write":439,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":271,"ondisk_log_size":271,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575767,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51944,"kb_used_data":2716,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319896,"statfs":{"total":96636764160,"available":96583573504,"internally_reserved":0,"allocated":2781184,"data_stored":1703782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":50395595},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:27:06 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.63200000000000001,"15min":0.626},"min":{"1min":0.45800000000000002,"5min":0.39800000000000002,"15min":0.30399999999999999},"max":{"1min":0.78400000000000003,"5min":0.874,"15min":0.88200000000000001},"last":3.218},{"interface":"front","average":{"1min":0.629,"5min":0.63,"15min":0.63100000000000001},"min":{"1min":0.39900000000000002,"5min":0.38500000000000001,"15min":0.375},"max":{"1min":0.80900000000000005,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":9.1270000000000007}]},{"osd":1,"last update":"Mon Oct 16 13:27:06 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.65000000000000002,"15min":0.63600000000000001},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.34000000000000002},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.91700000000000004},"last":8.7539999999999996},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.64100000000000001,"15min":0.63200000000000001},"min":{"1min":0.47299999999999998,"5min":0.41199999999999998,"15min":0.36899999999999999},"max":{"1min":0.81899999999999995,"5min":0.85999999999999999,"15min":0.99199999999999999},"last":9.173}]},{"osd":2,"last update":"Mon Oct 16 13:27:06 2023","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.70199999999999996,"15min":0.69299999999999995},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.38500000000000001},"max":{"1min":0.80500000000000005,"5min":0.92900000000000005,"15min":1.03},"last":6.5869999999999997},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.70199999999999996,"15min":0.69899999999999995},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.41099999999999998},"max":{"1min":0.88300000000000001,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":3.0640000000000001}]},{"osd":3,"last update":"Mon Oct 16 13:27:06 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.64900000000000002,"15min":0.64400000000000002},"min":{"1min":0.45700000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.83199999999999996,"5min":0.84699999999999998,"15min":0.86299999999999999},"last":3.1299999999999999},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.65900000000000003,"15min":0.65500000000000003},"min":{"1min":0.503,"5min":0.48399999999999999,"15min":0.40899999999999997},"max":{"1min":0.85099999999999998,"5min":0.93999999999999995,"15min":0.95199999999999996},"last":8.9920000000000009}]},{"osd":4,"last update":"Mon Oct 16 13:27:06 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.53200000000000003,"15min":0.52800000000000002},"min":{"1min":0.36899999999999999,"5min":0.32800000000000001,"15min":0.313},"max":{"1min":0.79900000000000004,"5min":0.84499999999999997,"15min":0.874},"last":6.3769999999999998},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.61899999999999999,"15min":0.62},"min":{"1min":0.47499999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.80600000000000005,"5min":0.90200000000000002,"15min":0.90900000000000003},"last":3.1779999999999999}]},{"osd":5,"last update":"Mon Oct 16 13:27:06 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.54800000000000004,"15min":0.54300000000000004},"min":{"1min":0.39400000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.81799999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":8.9209999999999994},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.59199999999999997,"15min":0.59399999999999997},"min":{"1min":0.377,"5min":0.377,"15min":0.376},"max":{"1min":0.84899999999999998,"5min":1.0329999999999999,"15min":1.135},"last":6.7560000000000002}]},{"osd":6,"last update":"Mon Oct 16 13:27:06 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.63500000000000001,"15min":0.63400000000000001},"min":{"1min":0.44400000000000001,"5min":0.41899999999999998,"15min":0.38500000000000001},"max":{"1min":0.88100000000000001,"5min":0.90200000000000002,"15min":0.96099999999999997},"last":8.8279999999999994},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.40000000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.84999999999999998,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":3.2599999999999998}]}]},{"osd":6,"up_from":13,"seq":55834575765,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41644,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330196,"statfs":{"total":96636764160,"available":96594120704,"internally_reserved":0,"allocated":2187264,"data_stored":1113414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:27:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.69599999999999995,"15min":0.67800000000000005},"min":{"1min":0.434,"5min":0.434,"15min":0.29899999999999999},"max":{"1min":0.86699999999999999,"5min":0.90800000000000003,"15min":0.91400000000000003},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.68799999999999994,"15min":0.68500000000000005},"min":{"1min":0.48999999999999999,"5min":0.42099999999999999,"15min":0.34699999999999998},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.93100000000000005},"last":0.5}]},{"osd":1,"last update":"Mon Oct 16 13:27:05 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.63700000000000001,"15min":0.63100000000000001},"min":{"1min":0.48599999999999999,"5min":0.36099999999999999,"15min":0.35499999999999998},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.626,"5min":0.63200000000000001,"15min":0.63600000000000001},"min":{"1min":0.44900000000000001,"5min":0.38400000000000001,"15min":0.34899999999999998},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.95599999999999996},"last":0.54400000000000004}]},{"osd":2,"last update":"Mon Oct 16 13:27:05 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.64700000000000002,"15min":0.63900000000000001},"min":{"1min":0.46100000000000002,"5min":0.44400000000000001,"15min":0.33500000000000002},"max":{"1min":0.84199999999999997,"5min":0.93899999999999995,"15min":0.94199999999999995},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.63500000000000001,"15min":0.63700000000000001},"min":{"1min":0.50800000000000001,"5min":0.39700000000000002,"15min":0.33100000000000002},"max":{"1min":0.79600000000000004,"5min":0.86399999999999999,"15min":0.93100000000000005},"last":0.59799999999999998}]},{"osd":3,"last update":"Mon Oct 16 13:27:05 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.69099999999999995,"15min":0.68700000000000006},"min":{"1min":0.60499999999999998,"5min":0.42599999999999999,"15min":0.374},"max":{"1min":0.874,"5min":1.288,"15min":1.288},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.746,"15min":0.72899999999999998},"min":{"1min":0.60499999999999998,"5min":0.504,"15min":0.32300000000000001},"max":{"1min":0.81999999999999995,"5min":1.351,"15min":1.351},"last":0.70299999999999996}]},{"osd":4,"last update":"Mon Oct 16 13:27:05 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.36099999999999999,"5min":0.34100000000000003,"15min":0.32000000000000001},"max":{"1min":0.78200000000000003,"5min":0.85599999999999998,"15min":0.86299999999999999},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.60999999999999999,"15min":0.60899999999999999},"min":{"1min":0.48899999999999999,"5min":0.35199999999999998,"15min":0.32000000000000001},"max":{"1min":0.82299999999999995,"5min":0.89600000000000002,"15min":0.90100000000000002},"last":0.66600000000000004}]},{"osd":5,"last update":"Mon Oct 16 13:27:05 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.56899999999999995,"15min":0.57199999999999995},"min":{"1min":0.41799999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.83199999999999996,"5min":0.85899999999999999,"15min":0.91800000000000004},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.64500000000000002,"15min":0.63},"min":{"1min":0.45500000000000002,"5min":0.34399999999999997,"15min":0.32700000000000001},"max":{"1min":0.82099999999999995,"5min":0.91000000000000003,"15min":0.93799999999999994},"last":0.73699999999999999}]},{"osd":7,"last update":"Mon Oct 16 13:27:05 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.58599999999999997,"15min":0.57499999999999996},"min":{"1min":0.41499999999999998,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.629,"15min":0.626},"min":{"1min":0.48599999999999999,"5min":0.39300000000000002,"15min":0.378},"max":{"1min":0.83999999999999997,"5min":0.92000000000000004,"15min":0.98899999999999999},"last":0.68999999999999995}]}]},{"osd":1,"up_from":13,"seq":55834575765,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41640,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330200,"statfs":{"total":96636764160,"available":96594124800,"internally_reserved":0,"allocated":2187264,"data_stored":1113414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":40434121},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:27:04 2023","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.41899999999999998,"15min":0.41399999999999998},"min":{"1min":0.27400000000000002,"5min":0.27000000000000002,"15min":0.245},"max":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.76600000000000001},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.505,"15min":0.49399999999999999},"min":{"1min":0.40400000000000003,"5min":0.35299999999999998,"15min":0.34699999999999998},"max":{"1min":0.69899999999999995,"5min":0.71199999999999997,"15min":0.74199999999999999},"last":0.374}]},{"osd":2,"last update":"Mon Oct 16 13:27:04 2023","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.46400000000000002,"15min":0.45400000000000001},"min":{"1min":0.34399999999999997,"5min":0.28999999999999998,"15min":0.28100000000000003},"max":{"1min":0.627,"5min":0.751,"15min":0.751},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.505,"5min":0.497,"15min":0.495},"min":{"1min":0.35999999999999999,"5min":0.32300000000000001,"15min":0.29299999999999998},"max":{"1min":0.67600000000000005,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.48299999999999998}]},{"osd":3,"last update":"Mon Oct 16 13:27:04 2023","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.435,"15min":0.42599999999999999},"min":{"1min":0.32600000000000001,"5min":0.309,"15min":0.29199999999999998},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.71099999999999997},"last":0.312},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.48899999999999999,"15min":0.48599999999999999},"min":{"1min":0.39000000000000001,"5min":0.311,"15min":0.31},"max":{"1min":0.65700000000000003,"5min":0.66900000000000004,"15min":0.75},"last":0.53000000000000003}]},{"osd":4,"last update":"Mon Oct 16 13:27:04 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.67200000000000004,"15min":0.66800000000000004},"min":{"1min":0.50600000000000001,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98899999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.69199999999999995,"15min":0.68100000000000005},"min":{"1min":0.51100000000000001,"5min":0.45900000000000002,"15min":0.42099999999999999},"max":{"1min":0.92000000000000004,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.629}]},{"osd":5,"last update":"Mon Oct 16 13:27:04 2023","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.71799999999999997,"15min":0.71599999999999997},"min":{"1min":0.55900000000000005,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":1.046},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.746,"15min":0.73399999999999999},"min":{"1min":0.61299999999999999,"5min":0.52500000000000002,"15min":0.47099999999999997},"max":{"1min":0.93600000000000005,"5min":0.98499999999999999,"15min":1.0089999999999999},"last":0.71199999999999997}]},{"osd":6,"last update":"Mon Oct 16 13:27:04 2023","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.68799999999999994,"15min":0.68700000000000006},"min":{"1min":0.52700000000000002,"5min":0.51100000000000001,"15min":0.45000000000000001},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.71799999999999997,"15min":0.71399999999999997},"min":{"1min":0.55000000000000004,"5min":0.46700000000000003,"15min":0.432},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.64400000000000002}]},{"osd":7,"last update":"Mon Oct 16 13:27:04 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.73999999999999999,"15min":0.73799999999999999},"min":{"1min":0.49299999999999999,"5min":0.48799999999999999,"15min":0.44},"max":{"1min":0.876,"5min":1.024,"15min":1.04},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.76500000000000001,"15min":0.75600000000000001},"min":{"1min":0.60499999999999998,"5min":0.56299999999999994,"15min":0.498},"max":{"1min":0.997,"5min":0.997,"15min":1.0269999999999999},"last":0.79700000000000004}]}]},{"osd":0,"up_from":13,"seq":55834575766,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42224,"kb_used_data":2716,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329616,"statfs":{"total":96636764160,"available":96593526784,"internally_reserved":0,"allocated":2781184,"data_stored":1703782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 13:26:57 2023","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.42399999999999999,"15min":0.41499999999999998},"min":{"1min":0.28299999999999997,"5min":0.26300000000000001,"15min":0.23499999999999999},"max":{"1min":0.61799999999999999,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.45400000000000001,"15min":0.45100000000000001},"min":{"1min":0.30199999999999999,"5min":0.29899999999999999,"15min":0.28199999999999997},"max":{"1min":0.63400000000000001,"5min":0.745,"15min":0.75800000000000001},"last":0.34399999999999997}]},{"osd":2,"last update":"Mon Oct 16 13:26:57 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.46000000000000002,"15min":0.45900000000000002},"min":{"1min":0.36899999999999999,"5min":0.30399999999999999,"15min":0.28599999999999998},"max":{"1min":0.61099999999999999,"5min":0.70799999999999996,"15min":0.73399999999999999},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.48899999999999999,"15min":0.48499999999999999},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.29499999999999998},"max":{"1min":0.61699999999999999,"5min":0.67100000000000004,"15min":0.75700000000000001},"last":0.42199999999999999}]},{"osd":3,"last update":"Mon Oct 16 13:26:57 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.45200000000000001,"15min":0.44900000000000001},"min":{"1min":0.311,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.60099999999999998,"5min":0.66100000000000003,"15min":0.70199999999999996},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.53200000000000003,"15min":0.52800000000000002},"min":{"1min":0.40899999999999997,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.78200000000000003,"5min":0.81299999999999994,"15min":0.82099999999999995},"last":0.622}]},{"osd":4,"last update":"Mon Oct 16 13:26:57 2023","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.71999999999999997,"15min":0.68700000000000006},"min":{"1min":0.61799999999999999,"5min":0.45500000000000002,"15min":0.40200000000000002},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0369999999999999},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.71199999999999997,"15min":0.68700000000000006},"min":{"1min":0.47999999999999998,"5min":0.42399999999999999,"15min":0.39000000000000001},"max":{"1min":0.92800000000000005,"5min":1.0429999999999999,"15min":1.0669999999999999},"last":0.55300000000000005}]},{"osd":5,"last update":"Mon Oct 16 13:26:57 2023","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.70699999999999996,"15min":0.68300000000000005},"min":{"1min":0.44900000000000001,"5min":0.438,"15min":0.40300000000000002},"max":{"1min":1.0049999999999999,"5min":1.034,"15min":1.034},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.73899999999999999,"15min":0.71699999999999997},"min":{"1min":0.51500000000000001,"5min":0.47899999999999998,"15min":0.46700000000000003},"max":{"1min":0.92100000000000004,"5min":1.109,"15min":1.109},"last":0.51800000000000002}]},{"osd":6,"last update":"Mon Oct 16 13:26:57 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.74299999999999999,"15min":0.72999999999999998},"min":{"1min":0.62,"5min":0.49199999999999999,"15min":0.42999999999999999},"max":{"1min":0.93300000000000005,"5min":1.0660000000000001,"15min":1.109},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.747,"15min":0.73099999999999998},"min":{"1min":0.52400000000000002,"5min":0.46999999999999997,"15min":0.44500000000000001},"max":{"1min":0.95299999999999996,"5min":1.056,"15min":1.0720000000000001},"last":0.50600000000000001}]},{"osd":7,"last update":"Mon Oct 16 13:26:57 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.75600000000000001,"15min":0.73699999999999999},"min":{"1min":0.58099999999999996,"5min":0.51800000000000002,"15min":0.5},"max":{"1min":0.94999999999999996,"5min":1.0149999999999999,"15min":1.0880000000000001},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.753,"5min":0.754,"15min":0.73299999999999998},"min":{"1min":0.56699999999999995,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.93200000000000005,"5min":1.0840000000000001,"15min":1.0920000000000001},"last":0.60599999999999998}]}]},{"osd":2,"up_from":13,"seq":55834575766,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51392,"kb_used_data":2156,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320448,"statfs":{"total":96636764160,"available":96584138752,"internally_reserved":0,"allocated":2207744,"data_stored":1120721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:26:52 2023","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.41399999999999998,"15min":0.41299999999999998},"min":{"1min":0.25700000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.63400000000000001,"5min":0.65600000000000003,"15min":0.77800000000000002},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.499,"5min":0.502,"15min":0.48799999999999999},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.32500000000000001},"max":{"1min":0.78100000000000003,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.30299999999999999}]},{"osd":1,"last update":"Mon Oct 16 13:26:52 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.53500000000000003,"15min":0.51900000000000002},"min":{"1min":0.40999999999999998,"5min":0.34999999999999998,"15min":0.32400000000000001},"max":{"1min":0.63100000000000001,"5min":0.79600000000000004,"15min":0.879},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.47599999999999998,"15min":0.47299999999999998},"min":{"1min":0.33500000000000002,"5min":0.30599999999999999,"15min":0.29699999999999999},"max":{"1min":0.57199999999999995,"5min":0.78200000000000003,"15min":0.79000000000000004},"last":0.35699999999999998}]},{"osd":3,"last update":"Mon Oct 16 13:26:52 2023","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.44400000000000001,"15min":0.45000000000000001},"min":{"1min":0.31,"5min":0.29499999999999998,"15min":0.29099999999999998},"max":{"1min":0.52600000000000002,"5min":0.66200000000000003,"15min":0.81200000000000006},"last":0.248},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.56499999999999995,"15min":0.55800000000000005},"min":{"1min":0.42599999999999999,"5min":0.38900000000000001,"15min":0.34499999999999997},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":1.008},"last":0.379}]},{"osd":4,"last update":"Mon Oct 16 13:26:52 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.70599999999999996,"15min":0.70499999999999996},"min":{"1min":0.53600000000000003,"5min":0.41599999999999998,"15min":0.379},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.71699999999999997,"15min":0.71099999999999997},"min":{"1min":0.504,"5min":0.48799999999999999,"15min":0.45700000000000002},"max":{"1min":0.996,"5min":0.996,"15min":1.0109999999999999},"last":0.753}]},{"osd":5,"last update":"Mon Oct 16 13:26:52 2023","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.72199999999999998,"15min":0.70899999999999996},"min":{"1min":0.54300000000000004,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.84399999999999997,"5min":0.95299999999999996,"15min":1.056},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.68300000000000005,"15min":0.68400000000000005},"min":{"1min":0.497,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.879,"5min":0.96899999999999997,"15min":1.0740000000000001},"last":0.58699999999999997}]},{"osd":6,"last update":"Mon Oct 16 13:26:52 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.78600000000000003,"5min":1.046,"15min":1.345},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.74199999999999999,"15min":0.73499999999999999},"min":{"1min":0.57199999999999995,"5min":0.51900000000000002,"15min":0.51600000000000001},"max":{"1min":0.95899999999999996,"5min":1.006,"15min":1.2430000000000001},"last":0.57699999999999996}]},{"osd":7,"last update":"Mon Oct 16 13:26:52 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.72699999999999998,"15min":0.72499999999999998},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.88800000000000001,"5min":1.0620000000000001,"15min":1.115},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.747,"15min":0.73799999999999999},"min":{"1min":0.60299999999999998,"5min":0.54600000000000004,"15min":0.51100000000000001},"max":{"1min":1.018,"5min":1.018,"15min":1.1759999999999999},"last":0.60599999999999998}]}]},{"osd":3,"up_from":13,"seq":55834575765,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47228,"kb_used_data":2156,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324612,"statfs":{"total":96636764160,"available":96588402688,"internally_reserved":0,"allocated":2207744,"data_stored":1121225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":46135752},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:26:57 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.40799999999999997,"15min":0.40999999999999998},"min":{"1min":0.312,"5min":0.27800000000000002,"15min":0.249},"max":{"1min":0.58499999999999996,"5min":0.629,"15min":0.73099999999999998},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.438,"5min":0.433,"15min":0.439},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.70299999999999996,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.36599999999999999}]},{"osd":1,"last update":"Mon Oct 16 13:26:57 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.45200000000000001,"15min":0.441},"min":{"1min":0.313,"5min":0.30299999999999999,"15min":0.27500000000000002},"max":{"1min":0.65600000000000003,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.48899999999999999,"15min":0.49099999999999999},"min":{"1min":0.35299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.69999999999999996,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.41099999999999998}]},{"osd":2,"last update":"Mon Oct 16 13:26:57 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.495,"15min":0.501},"min":{"1min":0.39600000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.67600000000000005,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.52300000000000002,"15min":0.53200000000000003},"min":{"1min":0.372,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73299999999999998,"5min":0.79300000000000004,"15min":0.89800000000000002},"last":0.46800000000000003}]},{"osd":4,"last update":"Mon Oct 16 13:26:57 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.66200000000000003,"15min":0.68000000000000005},"min":{"1min":0.46600000000000003,"5min":0.435,"15min":0.41799999999999998},"max":{"1min":0.88200000000000001,"5min":1.0640000000000001,"15min":1.1060000000000001},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.68400000000000005,"15min":0.69899999999999995},"min":{"1min":0.50900000000000001,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.97799999999999998,"5min":1.008,"15min":1.0860000000000001},"last":0.63500000000000001}]},{"osd":5,"last update":"Mon Oct 16 13:26:57 2023","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.70599999999999996,"15min":0.72099999999999997},"min":{"1min":0.53600000000000003,"5min":0.48499999999999999,"15min":0.47699999999999998},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.69499999999999995,"15min":0.70699999999999996},"min":{"1min":0.51200000000000001,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.73799999999999999}]},{"osd":6,"last update":"Mon Oct 16 13:26:57 2023","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.72199999999999998,"15min":0.71599999999999997},"min":{"1min":0.496,"5min":0.496,"15min":0.42799999999999999},"max":{"1min":0.97999999999999998,"5min":1.0469999999999999,"15min":1.0800000000000001},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.73999999999999999,"15min":0.751},"min":{"1min":0.57599999999999996,"5min":0.50600000000000001,"15min":0.46999999999999997},"max":{"1min":0.995,"5min":1.073,"15min":1.1220000000000001},"last":0.71999999999999997}]},{"osd":7,"last update":"Mon Oct 16 13:26:57 2023","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.73099999999999998,"15min":0.73499999999999999},"min":{"1min":0.53600000000000003,"5min":0.502,"15min":0.49099999999999999},"max":{"1min":1.0409999999999999,"5min":1.099,"15min":1.099},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.75600000000000001,"15min":0.76000000000000001},"min":{"1min":0.59799999999999998,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.054,"5min":1.0609999999999999,"15min":1.131},"last":0.70799999999999996}]}]},{"osd":4,"up_from":13,"seq":55834575764,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29180,"kb_used_data":2160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342660,"statfs":{"total":96636764160,"available":96606883840,"internally_reserved":0,"allocated":2211840,"data_stored":1122221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:26:57 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.62},"min":{"1min":0.40799999999999997,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.84199999999999997,"5min":0.85199999999999998,"15min":0.93600000000000005},"last":0.745},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.64600000000000002,"15min":0.63600000000000001},"min":{"1min":0.48599999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.76400000000000001}]},{"osd":1,"last update":"Mon Oct 16 13:26:57 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.65100000000000002,"15min":0.64500000000000002},"min":{"1min":0.51500000000000001,"5min":0.34300000000000003,"15min":0.33400000000000002},"max":{"1min":0.78500000000000003,"5min":0.878,"15min":0.93799999999999994},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64500000000000002,"15min":0.63900000000000001},"min":{"1min":0.45900000000000002,"5min":0.42699999999999999,"15min":0.30199999999999999},"max":{"1min":0.79300000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.68899999999999995}]},{"osd":2,"last update":"Mon Oct 16 13:26:57 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.32800000000000001},"max":{"1min":0.81299999999999994,"5min":0.90000000000000002,"15min":1.091},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.67800000000000005,"15min":0.65800000000000003},"min":{"1min":0.46400000000000002,"5min":0.38,"15min":0.38},"max":{"1min":0.82599999999999996,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.625}]},{"osd":3,"last update":"Mon Oct 16 13:26:57 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.64900000000000002,"15min":0.64800000000000002},"min":{"1min":0.441,"5min":0.41699999999999998,"15min":0.39400000000000002},"max":{"1min":0.80000000000000004,"5min":0.89200000000000002,"15min":0.95399999999999996},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.69499999999999995,"15min":0.67700000000000005},"min":{"1min":0.48999999999999999,"5min":0.41599999999999998,"15min":0.41499999999999998},"max":{"1min":0.80800000000000005,"5min":0.86799999999999999,"15min":0.98899999999999999},"last":0.77900000000000003}]},{"osd":5,"last update":"Mon Oct 16 13:26:57 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.495,"15min":0.49399999999999999},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.309},"max":{"1min":0.71799999999999997,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.499},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.53800000000000003,"15min":0.54000000000000004},"min":{"1min":0.40799999999999997,"5min":0.31,"15min":0.31},"max":{"1min":0.748,"5min":0.93000000000000005,"15min":0.97099999999999997},"last":0.56200000000000006}]},{"osd":6,"last update":"Mon Oct 16 13:26:57 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.55500000000000005,"15min":0.56399999999999995},"min":{"1min":0.43099999999999999,"5min":0.34499999999999997,"15min":0.32800000000000001},"max":{"1min":0.88600000000000001,"5min":0.90900000000000003,"15min":1.046},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.65200000000000002,"15min":0.65500000000000003},"min":{"1min":0.433,"5min":0.433,"15min":0.34999999999999998},"max":{"1min":0.88200000000000001,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.80300000000000005}]},{"osd":7,"last update":"Mon Oct 16 13:26:57 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.60199999999999998,"15min":0.59799999999999998},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.84799999999999998,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.65200000000000002,"15min":0.64900000000000002},"min":{"1min":0.40000000000000002,"5min":0.39500000000000002,"15min":0.38400000000000001},"max":{"1min":0.92000000000000004,"5min":0.94899999999999995,"15min":1.1359999999999999},"last":0.66500000000000004}]}]},{"osd":5,"up_from":13,"seq":55834575765,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51380,"kb_used_data":2148,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320460,"statfs":{"total":96636764160,"available":96584151040,"internally_reserved":0,"allocated":2199552,"data_stored":1120609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:26:47 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.63200000000000001,"15min":0.623},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.34599999999999997},"max":{"1min":0.83899999999999997,"5min":0.87,"15min":0.92900000000000005},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.622,"5min":0.64900000000000002,"15min":0.65000000000000002},"min":{"1min":0.432,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.83899999999999997,"5min":0.89100000000000001,"15min":0.94899999999999995},"last":0.27500000000000002}]},{"osd":1,"last update":"Mon Oct 16 13:26:47 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.65400000000000003,"15min":0.65200000000000002},"min":{"1min":0.40699999999999997,"5min":0.378,"15min":0.37},"max":{"1min":0.81799999999999995,"5min":0.91800000000000004,"15min":0.93899999999999995},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.66200000000000003,"15min":0.65600000000000003},"min":{"1min":0.51300000000000001,"5min":0.40000000000000002,"15min":0.38100000000000001},"max":{"1min":0.872,"5min":0.90200000000000002,"15min":0.93300000000000005},"last":0.47799999999999998}]},{"osd":2,"last update":"Mon Oct 16 13:26:47 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.67500000000000004,"15min":0.65600000000000003},"min":{"1min":0.441,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.93300000000000005,"5min":1.091,"15min":1.091},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.64300000000000002,"15min":0.64700000000000002},"min":{"1min":0.49299999999999999,"5min":0.42699999999999999,"15min":0.35799999999999998},"max":{"1min":0.80400000000000005,"5min":0.94099999999999995,"15min":0.95699999999999996},"last":0.38300000000000001}]},{"osd":3,"last update":"Mon Oct 16 13:26:47 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.66700000000000004,"15min":0.66200000000000003},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.95499999999999996,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.68899999999999995,"15min":0.68400000000000005},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.41099999999999998},"max":{"1min":0.86099999999999999,"5min":1.042,"15min":1.042},"last":1.357}]},{"osd":4,"last update":"Mon Oct 16 13:26:47 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.51300000000000001,"15min":0.51600000000000001},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.315},"max":{"1min":0.81399999999999995,"5min":0.877,"15min":0.877},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59199999999999997,"15min":0.57999999999999996},"min":{"1min":0.372,"5min":0.372,"15min":0.35199999999999998},"max":{"1min":0.872,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.45200000000000001}]},{"osd":6,"last update":"Mon Oct 16 13:26:47 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57399999999999995,"15min":0.56699999999999995},"min":{"1min":0.39800000000000002,"5min":0.34599999999999997,"15min":0.34399999999999997},"max":{"1min":0.80000000000000004,"5min":0.88700000000000001,"15min":0.93100000000000005},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.628,"15min":0.61299999999999999},"min":{"1min":0.40300000000000002,"5min":0.38700000000000001,"15min":0.312},"max":{"1min":0.88700000000000001,"5min":1.006,"15min":1.006},"last":0.54500000000000004}]},{"osd":7,"last update":"Mon Oct 16 13:26:47 2023","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.66300000000000003,"15min":0.64100000000000001},"min":{"1min":0.44600000000000001,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.91500000000000004,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.69299999999999995,"15min":0.67200000000000004},"min":{"1min":0.49299999999999999,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.90200000000000002,"5min":0.98899999999999999,"15min":1.022},"last":0.50700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T13:27:46.320 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T13:27:46.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.476+0000 7f1aa7898700 1 Processor -- start 2023-10-16T13:27:46.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.476+0000 7f1aa7898700 1 -- start start 2023-10-16T13:27:46.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.478+0000 7f1aa7898700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1aa00ce380 0x7f1aa00d0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:46.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.478+0000 7f1aa7898700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1aa00d1210 con 0x7f1aa00b2b50 2023-10-16T13:27:46.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.478+0000 7f1aa7898700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1aa00cbc30 con 0x7f1aa00cdfa0 2023-10-16T13:27:46.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.478+0000 7f1aa7898700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1aa00cbf10 con 0x7f1aa00ce380 2023-10-16T13:27:46.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.478+0000 7f1aa5e35700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1aa00ce380 0x7f1aa00d0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:46.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.478+0000 7f1aa4e33700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f1aa00b2b50 0x7f1aa00d04a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37796/0 (socket says 172.21.15.16:37796) 2023-10-16T13:27:46.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.478+0000 7f1aa4e33700 1 -- 172.21.15.16:0/490523097 learned_addr learned my addr 172.21.15.16:0/490523097 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:27:46.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.478+0000 7f1aa5e35700 1 -- 172.21.15.16:0/490523097 >> v1:172.21.15.49:6789/0 conn(0x7f1aa00cdfa0 legacy=0x7f1aa00b2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:46.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.478+0000 7f1aa5e35700 1 -- 172.21.15.16:0/490523097 >> v1:172.21.15.16:6789/0 conn(0x7f1aa00b2b50 legacy=0x7f1aa00d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:46.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.478+0000 7f1aa5e35700 1 -- 172.21.15.16:0/490523097 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aa00cc1f0 con 0x7f1aa00ce380 2023-10-16T13:27:46.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.478+0000 7f1a97fff700 1 -- 172.21.15.16:0/490523097 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 651265270 0 0) 0x7f1aa00d1210 con 0x7f1aa00b2b50 2023-10-16T13:27:46.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.478+0000 7f1aa5e35700 1 --2- 172.21.15.16:0/490523097 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1aa00ce380 0x7f1aa00d0bb0 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7f1a98009d90 tx=0x7f1a9800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eab4210884fca8d7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:46.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.478+0000 7f1a97fff700 1 -- 172.21.15.16:0/490523097 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a9800e070 con 0x7f1aa00ce380 2023-10-16T13:27:46.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.478+0000 7f1a97fff700 1 -- 172.21.15.16:0/490523097 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a9800bbd0 con 0x7f1aa00ce380 2023-10-16T13:27:46.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.478+0000 7f1a97fff700 1 -- 172.21.15.16:0/490523097 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a98012430 con 0x7f1aa00ce380 2023-10-16T13:27:46.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.479+0000 7f1aa7898700 1 -- 172.21.15.16:0/490523097 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1aa00ce380 msgr2=0x7f1aa00d0bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:46.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.479+0000 7f1aa7898700 1 --2- 172.21.15.16:0/490523097 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1aa00ce380 0x7f1aa00d0bb0 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7f1a98009d90 tx=0x7f1a9800b330 comp rx=0 tx=0).stop 2023-10-16T13:27:46.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.479+0000 7f1aa7898700 1 -- 172.21.15.16:0/490523097 shutdown_connections 2023-10-16T13:27:46.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.479+0000 7f1aa7898700 1 --2- 172.21.15.16:0/490523097 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1aa00ce380 0x7f1aa00d0bb0 unknown :-1 s=CLOSED pgs=1931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:46.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.479+0000 7f1aa7898700 1 -- 172.21.15.16:0/490523097 >> 172.21.15.16:0/490523097 conn(0x7f1aa00d14b0 msgr2=0x7f1aa00d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:46.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.479+0000 7f1aa7898700 1 -- 172.21.15.16:0/490523097 shutdown_connections 2023-10-16T13:27:46.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.479+0000 7f1aa7898700 1 -- 172.21.15.16:0/490523097 wait complete. 2023-10-16T13:27:46.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.480+0000 7f1aa7898700 1 Processor -- start 2023-10-16T13:27:46.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.480+0000 7f1aa7898700 1 -- start start 2023-10-16T13:27:46.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.481+0000 7f1aa7898700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1aa00b2b50 0x7f1aa01f2c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:46.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.481+0000 7f1aa7898700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1aa00cdfa0 0x7f1aa00c8580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:46.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.481+0000 7f1aa7898700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1aa00ce380 0x7f1aa00c8ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:46.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.481+0000 7f1aa7898700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa015bcc0 con 0x7f1aa00b2b50 2023-10-16T13:27:46.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.481+0000 7f1aa7898700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa00c9c50 con 0x7f1aa00cdfa0 2023-10-16T13:27:46.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.481+0000 7f1aa7898700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa01f3160 con 0x7f1aa00ce380 2023-10-16T13:27:46.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.481+0000 7f1aa5634700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1aa00b2b50 0x7f1aa01f2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:46.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.481+0000 7f1aa5634700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1aa00b2b50 0x7f1aa01f2c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48528/0 (socket says 172.21.15.16:48528) 2023-10-16T13:27:46.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.481+0000 7f1aa5634700 1 -- 172.21.15.16:0/4240773244 learned_addr learned my addr 172.21.15.16:0/4240773244 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:46.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.481+0000 7f1aa4e33700 1 --2- 172.21.15.16:0/4240773244 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1aa00cdfa0 0x7f1aa00c8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:46.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.481+0000 7f1aa5e35700 1 --2- 172.21.15.16:0/4240773244 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1aa00ce380 0x7f1aa00c8ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:46.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.481+0000 7f1aa5634700 1 -- 172.21.15.16:0/4240773244 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1aa00ce380 msgr2=0x7f1aa00c8ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:46.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.481+0000 7f1aa5634700 1 --2- 172.21.15.16:0/4240773244 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1aa00ce380 0x7f1aa00c8ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:46.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.481+0000 7f1aa5634700 1 -- 172.21.15.16:0/4240773244 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1aa00cdfa0 msgr2=0x7f1aa00c8580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:46.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.481+0000 7f1aa5634700 1 --2- 172.21.15.16:0/4240773244 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1aa00cdfa0 0x7f1aa00c8580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:46.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.481+0000 7f1aa5634700 1 -- 172.21.15.16:0/4240773244 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a98009a70 con 0x7f1aa00b2b50 2023-10-16T13:27:46.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.482+0000 7f1aa5634700 1 --2- 172.21.15.16:0/4240773244 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1aa00b2b50 0x7f1aa01f2c20 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7f1a9000c4e0 tx=0x7f1a9000c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:46.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.482+0000 7f1a95ffb700 1 -- 172.21.15.16:0/4240773244 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a900052d0 con 0x7f1aa00b2b50 2023-10-16T13:27:46.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.482+0000 7f1a95ffb700 1 -- 172.21.15.16:0/4240773244 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a90005430 con 0x7f1aa00b2b50 2023-10-16T13:27:46.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.482+0000 7f1aa7898700 1 -- 172.21.15.16:0/4240773244 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1aa00c90d0 con 0x7f1aa00b2b50 2023-10-16T13:27:46.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.482+0000 7f1a95ffb700 1 -- 172.21.15.16:0/4240773244 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a900038c0 con 0x7f1aa00b2b50 2023-10-16T13:27:46.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.482+0000 7f1aa7898700 1 -- 172.21.15.16:0/4240773244 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1aa00c9690 con 0x7f1aa00b2b50 2023-10-16T13:27:46.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.483+0000 7f1aa7898700 1 -- 172.21.15.16:0/4240773244 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1aa015be00 con 0x7f1aa00b2b50 2023-10-16T13:27:46.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.483+0000 7f1a95ffb700 1 -- 172.21.15.16:0/4240773244 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1a90003a20 con 0x7f1aa00b2b50 2023-10-16T13:27:46.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.484+0000 7f1a95ffb700 1 --2- 172.21.15.16:0/4240773244 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f1a8406b020 0x7f1a8406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:46.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.484+0000 7f1a95ffb700 1 -- 172.21.15.16:0/4240773244 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f1a90088c90 con 0x7f1aa00b2b50 2023-10-16T13:27:46.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.484+0000 7f1aa4e33700 1 --2- 172.21.15.16:0/4240773244 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f1a8406b020 0x7f1a8406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:46.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.485+0000 7f1aa4e33700 1 --2- 172.21.15.16:0/4240773244 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f1a8406b020 0x7f1a8406d4e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1a9c007b10 tx=0x7f1a9c007a60 comp rx=0 tx=0).ready entity=mgr.9510 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:46.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.489+0000 7f1a95ffb700 1 -- 172.21.15.16:0/4240773244 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1a90053a80 con 0x7f1aa00b2b50 2023-10-16T13:27:46.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.687+0000 7f1aa7898700 1 -- 172.21.15.16:0/4240773244 --> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1aa00cc4d0 con 0x7f1a8406b020 2023-10-16T13:27:46.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.695+0000 7f1a95ffb700 1 -- 172.21.15.16:0/4240773244 <== mgr.9510 v2:172.21.15.16:6832/2329471558 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+290010 (secure 0 0 0) 0x7f1a9c001330 con 0x7f1a8406b020 2023-10-16T13:27:46.702 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:27:46.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.699+0000 7f1aa7898700 1 -- 172.21.15.16:0/4240773244 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f1a8406b020 msgr2=0x7f1a8406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:46.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.699+0000 7f1aa7898700 1 --2- 172.21.15.16:0/4240773244 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f1a8406b020 0x7f1a8406d4e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1a9c007b10 tx=0x7f1a9c007a60 comp rx=0 tx=0).stop 2023-10-16T13:27:46.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.699+0000 7f1aa7898700 1 -- 172.21.15.16:0/4240773244 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1aa00b2b50 msgr2=0x7f1aa01f2c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:46.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.699+0000 7f1aa7898700 1 --2- 172.21.15.16:0/4240773244 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1aa00b2b50 0x7f1aa01f2c20 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7f1a9000c4e0 tx=0x7f1a9000c7f0 comp rx=0 tx=0).stop 2023-10-16T13:27:46.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.699+0000 7f1aa7898700 1 -- 172.21.15.16:0/4240773244 shutdown_connections 2023-10-16T13:27:46.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.699+0000 7f1aa7898700 1 --2- 172.21.15.16:0/4240773244 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1aa00ce380 0x7f1aa00c8ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:46.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.699+0000 7f1aa7898700 1 --2- 172.21.15.16:0/4240773244 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1aa00b2b50 0x7f1aa01f2c20 unknown :-1 s=CLOSED pgs=1932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:46.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.699+0000 7f1aa7898700 1 --2- 172.21.15.16:0/4240773244 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f1a8406b020 0x7f1a8406d4e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:46.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.699+0000 7f1aa7898700 1 --2- 172.21.15.16:0/4240773244 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1aa00cdfa0 0x7f1aa00c8580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:46.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.699+0000 7f1aa7898700 1 -- 172.21.15.16:0/4240773244 >> 172.21.15.16:0/4240773244 conn(0x7f1aa00d14b0 msgr2=0x7f1aa00b2400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:46.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.699+0000 7f1aa7898700 1 -- 172.21.15.16:0/4240773244 shutdown_connections 2023-10-16T13:27:46.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.699+0000 7f1aa7898700 1 -- 172.21.15.16:0/4240773244 wait complete. 2023-10-16T13:27:46.707 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-10-16T13:27:46.715 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-10-16T13:27:44.817851+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1460,"num_read_kb":2984,"num_write":483,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":309,"ondisk_log_size":309,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":66,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":356632,"kb_used_data":18324,"kb_used_omap":12,"kb_used_meta":338163,"kb_avail":754618088,"statfs":{"total":773094113280,"available":772728922112,"internally_reserved":0,"allocated":18763776,"data_stored":10119168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12740,"internal_metadata":346279484},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.692909"},"pg_stats":[{"pgid":"22.3f","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.100873+0000","last_change":"2023-10-16T13:27:42.098307+0000","last_active":"2023-10-16T13:27:42.100873+0000","last_peered":"2023-10-16T13:27:42.100873+0000","last_clean":"2023-10-16T13:27:42.100873+0000","last_became_active":"2023-10-16T13:27:42.097986+0000","last_became_peered":"2023-10-16T13:27:42.097986+0000","last_unstale":"2023-10-16T13:27:42.100873+0000","last_undegraded":"2023-10-16T13:27:42.100873+0000","last_fullsized":"2023-10-16T13:27:42.100873+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:12:33.359605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.3e","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.088248+0000","last_change":"2023-10-16T13:27:42.085221+0000","last_active":"2023-10-16T13:27:42.088248+0000","last_peered":"2023-10-16T13:27:42.088248+0000","last_clean":"2023-10-16T13:27:42.088248+0000","last_became_active":"2023-10-16T13:27:42.084934+0000","last_became_peered":"2023-10-16T13:27:42.084934+0000","last_unstale":"2023-10-16T13:27:42.088248+0000","last_undegraded":"2023-10-16T13:27:42.088248+0000","last_fullsized":"2023-10-16T13:27:42.088248+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:16:25.214504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.3d","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.090236+0000","last_change":"2023-10-16T13:27:42.089490+0000","last_active":"2023-10-16T13:27:42.090236+0000","last_peered":"2023-10-16T13:27:42.090236+0000","last_clean":"2023-10-16T13:27:42.090236+0000","last_became_active":"2023-10-16T13:27:42.089198+0000","last_became_peered":"2023-10-16T13:27:42.089198+0000","last_unstale":"2023-10-16T13:27:42.090236+0000","last_undegraded":"2023-10-16T13:27:42.090236+0000","last_fullsized":"2023-10-16T13:27:42.090236+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:58:40.655812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.3c","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.091871+0000","last_change":"2023-10-16T13:27:42.087993+0000","last_active":"2023-10-16T13:27:42.091871+0000","last_peered":"2023-10-16T13:27:42.091871+0000","last_clean":"2023-10-16T13:27:42.091871+0000","last_became_active":"2023-10-16T13:27:42.087677+0000","last_became_peered":"2023-10-16T13:27:42.087677+0000","last_unstale":"2023-10-16T13:27:42.091871+0000","last_undegraded":"2023-10-16T13:27:42.091871+0000","last_fullsized":"2023-10-16T13:27:42.091871+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:03:17.841508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.3b","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.091522+0000","last_change":"2023-10-16T13:27:42.090897+0000","last_active":"2023-10-16T13:27:42.091522+0000","last_peered":"2023-10-16T13:27:42.091522+0000","last_clean":"2023-10-16T13:27:42.091522+0000","last_became_active":"2023-10-16T13:27:42.090581+0000","last_became_peered":"2023-10-16T13:27:42.090581+0000","last_unstale":"2023-10-16T13:27:42.091522+0000","last_undegraded":"2023-10-16T13:27:42.091522+0000","last_fullsized":"2023-10-16T13:27:42.091522+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:09:36.254368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3a","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.091586+0000","last_change":"2023-10-16T13:27:42.091034+0000","last_active":"2023-10-16T13:27:42.091586+0000","last_peered":"2023-10-16T13:27:42.091586+0000","last_clean":"2023-10-16T13:27:42.091586+0000","last_became_active":"2023-10-16T13:27:42.090505+0000","last_became_peered":"2023-10-16T13:27:42.090505+0000","last_unstale":"2023-10-16T13:27:42.091586+0000","last_undegraded":"2023-10-16T13:27:42.091586+0000","last_fullsized":"2023-10-16T13:27:42.091586+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:19:14.194081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.39","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.098882+0000","last_change":"2023-10-16T13:27:42.095313+0000","last_active":"2023-10-16T13:27:42.098882+0000","last_peered":"2023-10-16T13:27:42.098882+0000","last_clean":"2023-10-16T13:27:42.098882+0000","last_became_active":"2023-10-16T13:27:42.094911+0000","last_became_peered":"2023-10-16T13:27:42.094911+0000","last_unstale":"2023-10-16T13:27:42.098882+0000","last_undegraded":"2023-10-16T13:27:42.098882+0000","last_fullsized":"2023-10-16T13:27:42.098882+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:17:18.887726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.38","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.098304+0000","last_change":"2023-10-16T13:27:42.093051+0000","last_active":"2023-10-16T13:27:42.098304+0000","last_peered":"2023-10-16T13:27:42.098304+0000","last_clean":"2023-10-16T13:27:42.098304+0000","last_became_active":"2023-10-16T13:27:42.091399+0000","last_became_peered":"2023-10-16T13:27:42.091399+0000","last_unstale":"2023-10-16T13:27:42.098304+0000","last_undegraded":"2023-10-16T13:27:42.098304+0000","last_fullsized":"2023-10-16T13:27:42.098304+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:28:02.926570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.37","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.091534+0000","last_change":"2023-10-16T13:27:42.091029+0000","last_active":"2023-10-16T13:27:42.091534+0000","last_peered":"2023-10-16T13:27:42.091534+0000","last_clean":"2023-10-16T13:27:42.091534+0000","last_became_active":"2023-10-16T13:27:42.090163+0000","last_became_peered":"2023-10-16T13:27:42.090163+0000","last_unstale":"2023-10-16T13:27:42.091534+0000","last_undegraded":"2023-10-16T13:27:42.091534+0000","last_fullsized":"2023-10-16T13:27:42.091534+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:22:18.575043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.36","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.090855+0000","last_change":"2023-10-16T13:27:42.086147+0000","last_active":"2023-10-16T13:27:42.090855+0000","last_peered":"2023-10-16T13:27:42.090855+0000","last_clean":"2023-10-16T13:27:42.090855+0000","last_became_active":"2023-10-16T13:27:42.085841+0000","last_became_peered":"2023-10-16T13:27:42.085841+0000","last_unstale":"2023-10-16T13:27:42.090855+0000","last_undegraded":"2023-10-16T13:27:42.090855+0000","last_fullsized":"2023-10-16T13:27:42.090855+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:50.700408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.35","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.096868+0000","last_change":"2023-10-16T13:27:42.095274+0000","last_active":"2023-10-16T13:27:42.096868+0000","last_peered":"2023-10-16T13:27:42.096868+0000","last_clean":"2023-10-16T13:27:42.096868+0000","last_became_active":"2023-10-16T13:27:42.094986+0000","last_became_peered":"2023-10-16T13:27:42.094986+0000","last_unstale":"2023-10-16T13:27:42.096868+0000","last_undegraded":"2023-10-16T13:27:42.096868+0000","last_fullsized":"2023-10-16T13:27:42.096868+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:13:55.435394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.34","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.091848+0000","last_change":"2023-10-16T13:27:42.085399+0000","last_active":"2023-10-16T13:27:42.091848+0000","last_peered":"2023-10-16T13:27:42.091848+0000","last_clean":"2023-10-16T13:27:42.091848+0000","last_became_active":"2023-10-16T13:27:42.084777+0000","last_became_peered":"2023-10-16T13:27:42.084777+0000","last_unstale":"2023-10-16T13:27:42.091848+0000","last_undegraded":"2023-10-16T13:27:42.091848+0000","last_fullsized":"2023-10-16T13:27:42.091848+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:59:39.691551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.33","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.090960+0000","last_change":"2023-10-16T13:27:42.086264+0000","last_active":"2023-10-16T13:27:42.090960+0000","last_peered":"2023-10-16T13:27:42.090960+0000","last_clean":"2023-10-16T13:27:42.090960+0000","last_became_active":"2023-10-16T13:27:42.085967+0000","last_became_peered":"2023-10-16T13:27:42.085967+0000","last_unstale":"2023-10-16T13:27:42.090960+0000","last_undegraded":"2023-10-16T13:27:42.090960+0000","last_fullsized":"2023-10-16T13:27:42.090960+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:12:55.131919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.32","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.091478+0000","last_change":"2023-10-16T13:27:42.086362+0000","last_active":"2023-10-16T13:27:42.091478+0000","last_peered":"2023-10-16T13:27:42.091478+0000","last_clean":"2023-10-16T13:27:42.091478+0000","last_became_active":"2023-10-16T13:27:42.086071+0000","last_became_peered":"2023-10-16T13:27:42.086071+0000","last_unstale":"2023-10-16T13:27:42.091478+0000","last_undegraded":"2023-10-16T13:27:42.091478+0000","last_fullsized":"2023-10-16T13:27:42.091478+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:22:57.648967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.31","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.097960+0000","last_change":"2023-10-16T13:27:42.095644+0000","last_active":"2023-10-16T13:27:42.097960+0000","last_peered":"2023-10-16T13:27:42.097960+0000","last_clean":"2023-10-16T13:27:42.097960+0000","last_became_active":"2023-10-16T13:27:42.095348+0000","last_became_peered":"2023-10-16T13:27:42.095348+0000","last_unstale":"2023-10-16T13:27:42.097960+0000","last_undegraded":"2023-10-16T13:27:42.097960+0000","last_fullsized":"2023-10-16T13:27:42.097960+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:54:35.805923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.30","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.092656+0000","last_change":"2023-10-16T13:27:42.092368+0000","last_active":"2023-10-16T13:27:42.092656+0000","last_peered":"2023-10-16T13:27:42.092656+0000","last_clean":"2023-10-16T13:27:42.092656+0000","last_became_active":"2023-10-16T13:27:42.092054+0000","last_became_peered":"2023-10-16T13:27:42.092054+0000","last_unstale":"2023-10-16T13:27:42.092656+0000","last_undegraded":"2023-10-16T13:27:42.092656+0000","last_fullsized":"2023-10-16T13:27:42.092656+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:59:05.571247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.2f","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.096705+0000","last_change":"2023-10-16T13:27:42.095131+0000","last_active":"2023-10-16T13:27:42.096705+0000","last_peered":"2023-10-16T13:27:42.096705+0000","last_clean":"2023-10-16T13:27:42.096705+0000","last_became_active":"2023-10-16T13:27:42.094801+0000","last_became_peered":"2023-10-16T13:27:42.094801+0000","last_unstale":"2023-10-16T13:27:42.096705+0000","last_undegraded":"2023-10-16T13:27:42.096705+0000","last_fullsized":"2023-10-16T13:27:42.096705+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:28:44.934920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.2e","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.097774+0000","last_change":"2023-10-16T13:27:42.096463+0000","last_active":"2023-10-16T13:27:42.097774+0000","last_peered":"2023-10-16T13:27:42.097774+0000","last_clean":"2023-10-16T13:27:42.097774+0000","last_became_active":"2023-10-16T13:27:42.095950+0000","last_became_peered":"2023-10-16T13:27:42.095950+0000","last_unstale":"2023-10-16T13:27:42.097774+0000","last_undegraded":"2023-10-16T13:27:42.097774+0000","last_fullsized":"2023-10-16T13:27:42.097774+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:47:32.249424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2d","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.101197+0000","last_change":"2023-10-16T13:27:42.097037+0000","last_active":"2023-10-16T13:27:42.101197+0000","last_peered":"2023-10-16T13:27:42.101197+0000","last_clean":"2023-10-16T13:27:42.101197+0000","last_became_active":"2023-10-16T13:27:42.096591+0000","last_became_peered":"2023-10-16T13:27:42.096591+0000","last_unstale":"2023-10-16T13:27:42.101197+0000","last_undegraded":"2023-10-16T13:27:42.101197+0000","last_fullsized":"2023-10-16T13:27:42.101197+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:50.700408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.2c","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.090726+0000","last_change":"2023-10-16T13:27:42.087996+0000","last_active":"2023-10-16T13:27:42.090726+0000","last_peered":"2023-10-16T13:27:42.090726+0000","last_clean":"2023-10-16T13:27:42.090726+0000","last_became_active":"2023-10-16T13:27:42.087683+0000","last_became_peered":"2023-10-16T13:27:42.087683+0000","last_unstale":"2023-10-16T13:27:42.090726+0000","last_undegraded":"2023-10-16T13:27:42.090726+0000","last_fullsized":"2023-10-16T13:27:42.090726+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:13:33.886086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.2b","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.091154+0000","last_change":"2023-10-16T13:27:42.090377+0000","last_active":"2023-10-16T13:27:42.091154+0000","last_peered":"2023-10-16T13:27:42.091154+0000","last_clean":"2023-10-16T13:27:42.091154+0000","last_became_active":"2023-10-16T13:27:42.090052+0000","last_became_peered":"2023-10-16T13:27:42.090052+0000","last_unstale":"2023-10-16T13:27:42.091154+0000","last_undegraded":"2023-10-16T13:27:42.091154+0000","last_fullsized":"2023-10-16T13:27:42.091154+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:24:09.829869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.2a","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.101355+0000","last_change":"2023-10-16T13:27:42.099466+0000","last_active":"2023-10-16T13:27:42.101355+0000","last_peered":"2023-10-16T13:27:42.101355+0000","last_clean":"2023-10-16T13:27:42.101355+0000","last_became_active":"2023-10-16T13:27:42.098627+0000","last_became_peered":"2023-10-16T13:27:42.098627+0000","last_unstale":"2023-10-16T13:27:42.101355+0000","last_undegraded":"2023-10-16T13:27:42.101355+0000","last_fullsized":"2023-10-16T13:27:42.101355+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:21:10.011990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.29","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.101691+0000","last_change":"2023-10-16T13:27:42.100451+0000","last_active":"2023-10-16T13:27:42.101691+0000","last_peered":"2023-10-16T13:27:42.101691+0000","last_clean":"2023-10-16T13:27:42.101691+0000","last_became_active":"2023-10-16T13:27:42.100031+0000","last_became_peered":"2023-10-16T13:27:42.100031+0000","last_unstale":"2023-10-16T13:27:42.101691+0000","last_undegraded":"2023-10-16T13:27:42.101691+0000","last_fullsized":"2023-10-16T13:27:42.101691+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:08:01.317496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.28","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.086537+0000","last_change":"2023-10-16T13:27:42.083874+0000","last_active":"2023-10-16T13:27:42.086537+0000","last_peered":"2023-10-16T13:27:42.086537+0000","last_clean":"2023-10-16T13:27:42.086537+0000","last_became_active":"2023-10-16T13:27:42.083405+0000","last_became_peered":"2023-10-16T13:27:42.083405+0000","last_unstale":"2023-10-16T13:27:42.086537+0000","last_undegraded":"2023-10-16T13:27:42.086537+0000","last_fullsized":"2023-10-16T13:27:42.086537+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:09:17.969657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.27","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.096759+0000","last_change":"2023-10-16T13:27:42.092281+0000","last_active":"2023-10-16T13:27:42.096759+0000","last_peered":"2023-10-16T13:27:42.096759+0000","last_clean":"2023-10-16T13:27:42.096759+0000","last_became_active":"2023-10-16T13:27:42.091984+0000","last_became_peered":"2023-10-16T13:27:42.091984+0000","last_unstale":"2023-10-16T13:27:42.096759+0000","last_undegraded":"2023-10-16T13:27:42.096759+0000","last_fullsized":"2023-10-16T13:27:42.096759+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:08:07.749142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.26","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.097015+0000","last_change":"2023-10-16T13:27:42.089669+0000","last_active":"2023-10-16T13:27:42.097015+0000","last_peered":"2023-10-16T13:27:42.097015+0000","last_clean":"2023-10-16T13:27:42.097015+0000","last_became_active":"2023-10-16T13:27:42.089350+0000","last_became_peered":"2023-10-16T13:27:42.089350+0000","last_unstale":"2023-10-16T13:27:42.097015+0000","last_undegraded":"2023-10-16T13:27:42.097015+0000","last_fullsized":"2023-10-16T13:27:42.097015+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:41:28.464562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.25","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.097348+0000","last_change":"2023-10-16T13:27:42.094753+0000","last_active":"2023-10-16T13:27:42.097348+0000","last_peered":"2023-10-16T13:27:42.097348+0000","last_clean":"2023-10-16T13:27:42.097348+0000","last_became_active":"2023-10-16T13:27:42.094466+0000","last_became_peered":"2023-10-16T13:27:42.094466+0000","last_unstale":"2023-10-16T13:27:42.097348+0000","last_undegraded":"2023-10-16T13:27:42.097348+0000","last_fullsized":"2023-10-16T13:27:42.097348+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:47:49.660794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.24","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.087709+0000","last_change":"2023-10-16T13:27:42.086360+0000","last_active":"2023-10-16T13:27:42.087709+0000","last_peered":"2023-10-16T13:27:42.087709+0000","last_clean":"2023-10-16T13:27:42.087709+0000","last_became_active":"2023-10-16T13:27:42.086057+0000","last_became_peered":"2023-10-16T13:27:42.086057+0000","last_unstale":"2023-10-16T13:27:42.087709+0000","last_undegraded":"2023-10-16T13:27:42.087709+0000","last_fullsized":"2023-10-16T13:27:42.087709+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:52:58.973737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.23","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.097571+0000","last_change":"2023-10-16T13:27:42.092831+0000","last_active":"2023-10-16T13:27:42.097571+0000","last_peered":"2023-10-16T13:27:42.097571+0000","last_clean":"2023-10-16T13:27:42.097571+0000","last_became_active":"2023-10-16T13:27:42.092082+0000","last_became_peered":"2023-10-16T13:27:42.092082+0000","last_unstale":"2023-10-16T13:27:42.097571+0000","last_undegraded":"2023-10-16T13:27:42.097571+0000","last_fullsized":"2023-10-16T13:27:42.097571+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:27:52.441984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.22","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.098060+0000","last_change":"2023-10-16T13:27:42.097207+0000","last_active":"2023-10-16T13:27:42.098060+0000","last_peered":"2023-10-16T13:27:42.098060+0000","last_clean":"2023-10-16T13:27:42.098060+0000","last_became_active":"2023-10-16T13:27:42.096884+0000","last_became_peered":"2023-10-16T13:27:42.096884+0000","last_unstale":"2023-10-16T13:27:42.098060+0000","last_undegraded":"2023-10-16T13:27:42.098060+0000","last_fullsized":"2023-10-16T13:27:42.098060+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:39:10.604015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.21","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.091043+0000","last_change":"2023-10-16T13:27:42.088075+0000","last_active":"2023-10-16T13:27:42.091043+0000","last_peered":"2023-10-16T13:27:42.091043+0000","last_clean":"2023-10-16T13:27:42.091043+0000","last_became_active":"2023-10-16T13:27:42.087750+0000","last_became_peered":"2023-10-16T13:27:42.087750+0000","last_unstale":"2023-10-16T13:27:42.091043+0000","last_undegraded":"2023-10-16T13:27:42.091043+0000","last_fullsized":"2023-10-16T13:27:42.091043+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:20:26.554357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.20","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.099352+0000","last_change":"2023-10-16T13:27:42.095380+0000","last_active":"2023-10-16T13:27:42.099352+0000","last_peered":"2023-10-16T13:27:42.099352+0000","last_clean":"2023-10-16T13:27:42.099352+0000","last_became_active":"2023-10-16T13:27:42.095029+0000","last_became_peered":"2023-10-16T13:27:42.095029+0000","last_unstale":"2023-10-16T13:27:42.099352+0000","last_undegraded":"2023-10-16T13:27:42.099352+0000","last_fullsized":"2023-10-16T13:27:42.099352+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:40:57.689492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.087610+0000","last_change":"2023-10-16T13:27:42.085122+0000","last_active":"2023-10-16T13:27:42.087610+0000","last_peered":"2023-10-16T13:27:42.087610+0000","last_clean":"2023-10-16T13:27:42.087610+0000","last_became_active":"2023-10-16T13:27:42.084823+0000","last_became_peered":"2023-10-16T13:27:42.084823+0000","last_unstale":"2023-10-16T13:27:42.087610+0000","last_undegraded":"2023-10-16T13:27:42.087610+0000","last_fullsized":"2023-10-16T13:27:42.087610+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:08:14.962839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1f","version":"176'4","reported_seq":15,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.144058+0000","last_change":"2023-10-16T13:27:41.146091+0000","last_active":"2023-10-16T13:27:42.144058+0000","last_peered":"2023-10-16T13:27:42.144058+0000","last_clean":"2023-10-16T13:27:42.144058+0000","last_became_active":"2023-10-16T13:27:41.145307+0000","last_became_peered":"2023-10-16T13:27:41.145307+0000","last_unstale":"2023-10-16T13:27:42.144058+0000","last_undegraded":"2023-10-16T13:27:42.144058+0000","last_fullsized":"2023-10-16T13:27:42.144058+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_clean_scrub_stamp":"2023-10-16T13:27:40.119893+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:24:52.542805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.097923+0000","last_change":"2023-10-16T13:27:42.095553+0000","last_active":"2023-10-16T13:27:42.097923+0000","last_peered":"2023-10-16T13:27:42.097923+0000","last_clean":"2023-10-16T13:27:42.097923+0000","last_became_active":"2023-10-16T13:27:42.095246+0000","last_became_peered":"2023-10-16T13:27:42.095246+0000","last_unstale":"2023-10-16T13:27:42.097923+0000","last_undegraded":"2023-10-16T13:27:42.097923+0000","last_fullsized":"2023-10-16T13:27:42.097923+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:19:11.670430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1e","version":"176'1","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.141029+0000","last_change":"2023-10-16T13:27:41.152486+0000","last_active":"2023-10-16T13:27:42.141029+0000","last_peered":"2023-10-16T13:27:42.141029+0000","last_clean":"2023-10-16T13:27:42.141029+0000","last_became_active":"2023-10-16T13:27:41.152111+0000","last_became_peered":"2023-10-16T13:27:41.152111+0000","last_unstale":"2023-10-16T13:27:42.141029+0000","last_undegraded":"2023-10-16T13:27:42.141029+0000","last_fullsized":"2023-10-16T13:27:42.141029+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_clean_scrub_stamp":"2023-10-16T13:27:40.119893+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:47:48.655752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1e","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.097604+0000","last_change":"2023-10-16T13:27:42.091626+0000","last_active":"2023-10-16T13:27:42.097604+0000","last_peered":"2023-10-16T13:27:42.097604+0000","last_clean":"2023-10-16T13:27:42.097604+0000","last_became_active":"2023-10-16T13:27:42.091086+0000","last_became_peered":"2023-10-16T13:27:42.091086+0000","last_unstale":"2023-10-16T13:27:42.097604+0000","last_undegraded":"2023-10-16T13:27:42.097604+0000","last_fullsized":"2023-10-16T13:27:42.097604+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:11:18.208581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"176'2","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.143421+0000","last_change":"2023-10-16T13:27:41.151661+0000","last_active":"2023-10-16T13:27:42.143421+0000","last_peered":"2023-10-16T13:27:42.143421+0000","last_clean":"2023-10-16T13:27:42.143421+0000","last_became_active":"2023-10-16T13:27:41.151363+0000","last_became_peered":"2023-10-16T13:27:41.151363+0000","last_unstale":"2023-10-16T13:27:42.143421+0000","last_undegraded":"2023-10-16T13:27:42.143421+0000","last_fullsized":"2023-10-16T13:27:42.143421+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_clean_scrub_stamp":"2023-10-16T13:27:40.119893+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:24:08.824826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1f","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.101300+0000","last_change":"2023-10-16T13:27:42.099362+0000","last_active":"2023-10-16T13:27:42.101300+0000","last_peered":"2023-10-16T13:27:42.101300+0000","last_clean":"2023-10-16T13:27:42.101300+0000","last_became_active":"2023-10-16T13:27:42.099060+0000","last_became_peered":"2023-10-16T13:27:42.099060+0000","last_unstale":"2023-10-16T13:27:42.101300+0000","last_undegraded":"2023-10-16T13:27:42.101300+0000","last_fullsized":"2023-10-16T13:27:42.101300+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:09:36.254368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1c","version":"176'2","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.144285+0000","last_change":"2023-10-16T13:27:41.150315+0000","last_active":"2023-10-16T13:27:42.144285+0000","last_peered":"2023-10-16T13:27:42.144285+0000","last_clean":"2023-10-16T13:27:42.144285+0000","last_became_active":"2023-10-16T13:27:41.147732+0000","last_became_peered":"2023-10-16T13:27:41.147732+0000","last_unstale":"2023-10-16T13:27:42.144285+0000","last_undegraded":"2023-10-16T13:27:42.144285+0000","last_fullsized":"2023-10-16T13:27:42.144285+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_clean_scrub_stamp":"2023-10-16T13:27:40.119893+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:54:58.716839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.089183+0000","last_change":"2023-10-16T13:27:42.086114+0000","last_active":"2023-10-16T13:27:42.089183+0000","last_peered":"2023-10-16T13:27:42.089183+0000","last_clean":"2023-10-16T13:27:42.089183+0000","last_became_active":"2023-10-16T13:27:42.085826+0000","last_became_peered":"2023-10-16T13:27:42.085826+0000","last_unstale":"2023-10-16T13:27:42.089183+0000","last_undegraded":"2023-10-16T13:27:42.089183+0000","last_fullsized":"2023-10-16T13:27:42.089183+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:12:55.131919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.1b","version":"176'1","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.141441+0000","last_change":"2023-10-16T13:27:41.146905+0000","last_active":"2023-10-16T13:27:42.141441+0000","last_peered":"2023-10-16T13:27:42.141441+0000","last_clean":"2023-10-16T13:27:42.141441+0000","last_became_active":"2023-10-16T13:27:41.146574+0000","last_became_peered":"2023-10-16T13:27:41.146574+0000","last_unstale":"2023-10-16T13:27:42.141441+0000","last_undegraded":"2023-10-16T13:27:42.141441+0000","last_fullsized":"2023-10-16T13:27:42.141441+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_clean_scrub_stamp":"2023-10-16T13:27:40.119893+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:16:51.195967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.091855+0000","last_change":"2023-10-16T13:27:42.084487+0000","last_active":"2023-10-16T13:27:42.091855+0000","last_peered":"2023-10-16T13:27:42.091855+0000","last_clean":"2023-10-16T13:27:42.091855+0000","last_became_active":"2023-10-16T13:27:42.083614+0000","last_became_peered":"2023-10-16T13:27:42.083614+0000","last_unstale":"2023-10-16T13:27:42.091855+0000","last_undegraded":"2023-10-16T13:27:42.091855+0000","last_fullsized":"2023-10-16T13:27:42.091855+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:18:35.039185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1a","version":"176'1","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.142136+0000","last_change":"2023-10-16T13:27:41.148063+0000","last_active":"2023-10-16T13:27:42.142136+0000","last_peered":"2023-10-16T13:27:42.142136+0000","last_clean":"2023-10-16T13:27:42.142136+0000","last_became_active":"2023-10-16T13:27:41.147752+0000","last_became_peered":"2023-10-16T13:27:41.147752+0000","last_unstale":"2023-10-16T13:27:42.142136+0000","last_undegraded":"2023-10-16T13:27:42.142136+0000","last_fullsized":"2023-10-16T13:27:42.142136+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_clean_scrub_stamp":"2023-10-16T13:27:40.119893+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:13:32.881043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.089631+0000","last_change":"2023-10-16T13:27:42.088376+0000","last_active":"2023-10-16T13:27:42.089631+0000","last_peered":"2023-10-16T13:27:42.089631+0000","last_clean":"2023-10-16T13:27:42.089631+0000","last_became_active":"2023-10-16T13:27:42.088056+0000","last_became_peered":"2023-10-16T13:27:42.088056+0000","last_unstale":"2023-10-16T13:27:42.089631+0000","last_undegraded":"2023-10-16T13:27:42.089631+0000","last_fullsized":"2023-10-16T13:27:42.089631+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:55:15.868250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":11,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.086766+0000","last_change":"2023-10-16T13:27:41.153100+0000","last_active":"2023-10-16T13:27:42.086766+0000","last_peered":"2023-10-16T13:27:42.086766+0000","last_clean":"2023-10-16T13:27:42.086766+0000","last_became_active":"2023-10-16T13:27:41.152820+0000","last_became_peered":"2023-10-16T13:27:41.152820+0000","last_unstale":"2023-10-16T13:27:42.086766+0000","last_undegraded":"2023-10-16T13:27:42.086766+0000","last_fullsized":"2023-10-16T13:27:42.086766+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_clean_scrub_stamp":"2023-10-16T13:27:40.119893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:04:11.225946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1b","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.100932+0000","last_change":"2023-10-16T13:27:42.098791+0000","last_active":"2023-10-16T13:27:42.100932+0000","last_peered":"2023-10-16T13:27:42.100932+0000","last_clean":"2023-10-16T13:27:42.100932+0000","last_became_active":"2023-10-16T13:27:42.098456+0000","last_became_peered":"2023-10-16T13:27:42.098456+0000","last_unstale":"2023-10-16T13:27:42.100932+0000","last_undegraded":"2023-10-16T13:27:42.100932+0000","last_fullsized":"2023-10-16T13:27:42.100932+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:26:26.242188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":11,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.074653+0000","last_change":"2023-10-16T13:27:41.146877+0000","last_active":"2023-10-16T13:27:42.074653+0000","last_peered":"2023-10-16T13:27:42.074653+0000","last_clean":"2023-10-16T13:27:42.074653+0000","last_became_active":"2023-10-16T13:27:41.146529+0000","last_became_peered":"2023-10-16T13:27:41.146529+0000","last_unstale":"2023-10-16T13:27:42.074653+0000","last_undegraded":"2023-10-16T13:27:42.074653+0000","last_fullsized":"2023-10-16T13:27:42.074653+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_clean_scrub_stamp":"2023-10-16T13:27:40.119893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:11:56.094808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.14","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.101025+0000","last_change":"2023-10-16T13:27:42.096512+0000","last_active":"2023-10-16T13:27:42.101025+0000","last_peered":"2023-10-16T13:27:42.101025+0000","last_clean":"2023-10-16T13:27:42.101025+0000","last_became_active":"2023-10-16T13:27:42.096224+0000","last_became_peered":"2023-10-16T13:27:42.096224+0000","last_unstale":"2023-10-16T13:27:42.101025+0000","last_undegraded":"2023-10-16T13:27:42.101025+0000","last_fullsized":"2023-10-16T13:27:42.101025+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:39:53.629749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":11,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.085719+0000","last_change":"2023-10-16T13:27:41.156079+0000","last_active":"2023-10-16T13:27:42.085719+0000","last_peered":"2023-10-16T13:27:42.085719+0000","last_clean":"2023-10-16T13:27:42.085719+0000","last_became_active":"2023-10-16T13:27:41.155614+0000","last_became_peered":"2023-10-16T13:27:41.155614+0000","last_unstale":"2023-10-16T13:27:42.085719+0000","last_undegraded":"2023-10-16T13:27:42.085719+0000","last_fullsized":"2023-10-16T13:27:42.085719+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_clean_scrub_stamp":"2023-10-16T13:27:40.119893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:19:13.189038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.096849+0000","last_change":"2023-10-16T13:27:42.094983+0000","last_active":"2023-10-16T13:27:42.096849+0000","last_peered":"2023-10-16T13:27:42.096849+0000","last_clean":"2023-10-16T13:27:42.096849+0000","last_became_active":"2023-10-16T13:27:42.094609+0000","last_became_peered":"2023-10-16T13:27:42.094609+0000","last_unstale":"2023-10-16T13:27:42.096849+0000","last_undegraded":"2023-10-16T13:27:42.096849+0000","last_fullsized":"2023-10-16T13:27:42.096849+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:17:53.030835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":11,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.079400+0000","last_change":"2023-10-16T13:27:41.146063+0000","last_active":"2023-10-16T13:27:42.079400+0000","last_peered":"2023-10-16T13:27:42.079400+0000","last_clean":"2023-10-16T13:27:42.079400+0000","last_became_active":"2023-10-16T13:27:41.145651+0000","last_became_peered":"2023-10-16T13:27:41.145651+0000","last_unstale":"2023-10-16T13:27:42.079400+0000","last_undegraded":"2023-10-16T13:27:42.079400+0000","last_fullsized":"2023-10-16T13:27:42.079400+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_clean_scrub_stamp":"2023-10-16T13:27:40.119893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:23:38.428549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.097441+0000","last_change":"2023-10-16T13:27:42.096248+0000","last_active":"2023-10-16T13:27:42.097441+0000","last_peered":"2023-10-16T13:27:42.097441+0000","last_clean":"2023-10-16T13:27:42.097441+0000","last_became_active":"2023-10-16T13:27:42.095590+0000","last_became_peered":"2023-10-16T13:27:42.095590+0000","last_unstale":"2023-10-16T13:27:42.097441+0000","last_undegraded":"2023-10-16T13:27:42.097441+0000","last_fullsized":"2023-10-16T13:27:42.097441+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:27:13.595264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.15","version":"176'3","reported_seq":14,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.144116+0000","last_change":"2023-10-16T13:27:41.145596+0000","last_active":"2023-10-16T13:27:42.144116+0000","last_peered":"2023-10-16T13:27:42.144116+0000","last_clean":"2023-10-16T13:27:42.144116+0000","last_became_active":"2023-10-16T13:27:41.145220+0000","last_became_peered":"2023-10-16T13:27:41.145220+0000","last_unstale":"2023-10-16T13:27:42.144116+0000","last_undegraded":"2023-10-16T13:27:42.144116+0000","last_fullsized":"2023-10-16T13:27:42.144116+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_clean_scrub_stamp":"2023-10-16T13:27:40.119893+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:06:55.303636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.10","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.098342+0000","last_change":"2023-10-16T13:27:42.093717+0000","last_active":"2023-10-16T13:27:42.098342+0000","last_peered":"2023-10-16T13:27:42.098342+0000","last_clean":"2023-10-16T13:27:42.098342+0000","last_became_active":"2023-10-16T13:27:42.093362+0000","last_became_peered":"2023-10-16T13:27:42.093362+0000","last_unstale":"2023-10-16T13:27:42.098342+0000","last_undegraded":"2023-10-16T13:27:42.098342+0000","last_fullsized":"2023-10-16T13:27:42.098342+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:59:52.745820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.13","version":"176'2","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.143549+0000","last_change":"2023-10-16T13:27:41.145858+0000","last_active":"2023-10-16T13:27:42.143549+0000","last_peered":"2023-10-16T13:27:42.143549+0000","last_clean":"2023-10-16T13:27:42.143549+0000","last_became_active":"2023-10-16T13:27:41.145566+0000","last_became_peered":"2023-10-16T13:27:41.145566+0000","last_unstale":"2023-10-16T13:27:42.143549+0000","last_undegraded":"2023-10-16T13:27:42.143549+0000","last_fullsized":"2023-10-16T13:27:42.143549+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_clean_scrub_stamp":"2023-10-16T13:27:40.119893+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:18:21.429859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.085464+0000","last_change":"2023-10-16T13:27:42.084461+0000","last_active":"2023-10-16T13:27:42.085464+0000","last_peered":"2023-10-16T13:27:42.085464+0000","last_clean":"2023-10-16T13:27:42.085464+0000","last_became_active":"2023-10-16T13:27:42.083999+0000","last_became_peered":"2023-10-16T13:27:42.083999+0000","last_unstale":"2023-10-16T13:27:42.085464+0000","last_undegraded":"2023-10-16T13:27:42.085464+0000","last_fullsized":"2023-10-16T13:27:42.085464+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:35:31.624279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":11,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.080002+0000","last_change":"2023-10-16T13:27:41.153282+0000","last_active":"2023-10-16T13:27:42.080002+0000","last_peered":"2023-10-16T13:27:42.080002+0000","last_clean":"2023-10-16T13:27:42.080002+0000","last_became_active":"2023-10-16T13:27:41.147632+0000","last_became_peered":"2023-10-16T13:27:41.147632+0000","last_unstale":"2023-10-16T13:27:42.080002+0000","last_undegraded":"2023-10-16T13:27:42.080002+0000","last_fullsized":"2023-10-16T13:27:42.080002+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_clean_scrub_stamp":"2023-10-16T13:27:40.119893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:43:15.770153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.090714+0000","last_change":"2023-10-16T13:27:42.084817+0000","last_active":"2023-10-16T13:27:42.090714+0000","last_peered":"2023-10-16T13:27:42.090714+0000","last_clean":"2023-10-16T13:27:42.090714+0000","last_became_active":"2023-10-16T13:27:42.083457+0000","last_became_peered":"2023-10-16T13:27:42.083457+0000","last_unstale":"2023-10-16T13:27:42.090714+0000","last_undegraded":"2023-10-16T13:27:42.090714+0000","last_fullsized":"2023-10-16T13:27:42.090714+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:58:02.857014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":11,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.081089+0000","last_change":"2023-10-16T13:27:41.151560+0000","last_active":"2023-10-16T13:27:42.081089+0000","last_peered":"2023-10-16T13:27:42.081089+0000","last_clean":"2023-10-16T13:27:42.081089+0000","last_became_active":"2023-10-16T13:27:41.151104+0000","last_became_peered":"2023-10-16T13:27:41.151104+0000","last_unstale":"2023-10-16T13:27:42.081089+0000","last_undegraded":"2023-10-16T13:27:42.081089+0000","last_fullsized":"2023-10-16T13:27:42.081089+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_clean_scrub_stamp":"2023-10-16T13:27:40.119893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:29:05.731531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.083161+0000","last_change":"2023-10-16T13:27:42.080030+0000","last_active":"2023-10-16T13:27:42.083161+0000","last_peered":"2023-10-16T13:27:42.083161+0000","last_clean":"2023-10-16T13:27:42.083161+0000","last_became_active":"2023-10-16T13:27:42.079717+0000","last_became_peered":"2023-10-16T13:27:42.079717+0000","last_unstale":"2023-10-16T13:27:42.083161+0000","last_undegraded":"2023-10-16T13:27:42.083161+0000","last_fullsized":"2023-10-16T13:27:42.083161+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:31:43.138772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.10","version":"176'8","reported_seq":19,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.143973+0000","last_change":"2023-10-16T13:27:41.152454+0000","last_active":"2023-10-16T13:27:42.143973+0000","last_peered":"2023-10-16T13:27:42.143973+0000","last_clean":"2023-10-16T13:27:42.143973+0000","last_became_active":"2023-10-16T13:27:41.152053+0000","last_became_peered":"2023-10-16T13:27:41.152053+0000","last_unstale":"2023-10-16T13:27:42.143973+0000","last_undegraded":"2023-10-16T13:27:42.143973+0000","last_fullsized":"2023-10-16T13:27:42.143973+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_clean_scrub_stamp":"2023-10-16T13:27:40.119893+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:24:52.276480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.099681+0000","last_change":"2023-10-16T13:27:42.094617+0000","last_active":"2023-10-16T13:27:42.099681+0000","last_peered":"2023-10-16T13:27:42.099681+0000","last_clean":"2023-10-16T13:27:42.099681+0000","last_became_active":"2023-10-16T13:27:42.094303+0000","last_became_peered":"2023-10-16T13:27:42.094303+0000","last_unstale":"2023-10-16T13:27:42.099681+0000","last_undegraded":"2023-10-16T13:27:42.099681+0000","last_fullsized":"2023-10-16T13:27:42.099681+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:18:22.434902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.f","version":"176'1","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.140880+0000","last_change":"2023-10-16T13:27:41.154407+0000","last_active":"2023-10-16T13:27:42.140880+0000","last_peered":"2023-10-16T13:27:42.140880+0000","last_clean":"2023-10-16T13:27:42.140880+0000","last_became_active":"2023-10-16T13:27:41.154160+0000","last_became_peered":"2023-10-16T13:27:41.154160+0000","last_unstale":"2023-10-16T13:27:42.140880+0000","last_undegraded":"2023-10-16T13:27:42.140880+0000","last_fullsized":"2023-10-16T13:27:42.140880+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_clean_scrub_stamp":"2023-10-16T13:27:40.119893+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:10:58.951084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.d","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.101176+0000","last_change":"2023-10-16T13:27:42.096862+0000","last_active":"2023-10-16T13:27:42.101176+0000","last_peered":"2023-10-16T13:27:42.101176+0000","last_clean":"2023-10-16T13:27:42.101176+0000","last_became_active":"2023-10-16T13:27:42.096215+0000","last_became_peered":"2023-10-16T13:27:42.096215+0000","last_unstale":"2023-10-16T13:27:42.101176+0000","last_undegraded":"2023-10-16T13:27:42.101176+0000","last_fullsized":"2023-10-16T13:27:42.101176+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:24:53.547848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.e","version":"176'2","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.143080+0000","last_change":"2023-10-16T13:27:41.154236+0000","last_active":"2023-10-16T13:27:42.143080+0000","last_peered":"2023-10-16T13:27:42.143080+0000","last_clean":"2023-10-16T13:27:42.143080+0000","last_became_active":"2023-10-16T13:27:41.153914+0000","last_became_peered":"2023-10-16T13:27:41.153914+0000","last_unstale":"2023-10-16T13:27:42.143080+0000","last_undegraded":"2023-10-16T13:27:42.143080+0000","last_fullsized":"2023-10-16T13:27:42.143080+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_clean_scrub_stamp":"2023-10-16T13:27:40.119893+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:11:56.094808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.e","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.097036+0000","last_change":"2023-10-16T13:27:42.093054+0000","last_active":"2023-10-16T13:27:42.097036+0000","last_peered":"2023-10-16T13:27:42.097036+0000","last_clean":"2023-10-16T13:27:42.097036+0000","last_became_active":"2023-10-16T13:27:42.092757+0000","last_became_peered":"2023-10-16T13:27:42.092757+0000","last_unstale":"2023-10-16T13:27:42.097036+0000","last_undegraded":"2023-10-16T13:27:42.097036+0000","last_fullsized":"2023-10-16T13:27:42.097036+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:22:13.821974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.d","version":"176'1","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.142694+0000","last_change":"2023-10-16T13:27:41.150983+0000","last_active":"2023-10-16T13:27:42.142694+0000","last_peered":"2023-10-16T13:27:42.142694+0000","last_clean":"2023-10-16T13:27:42.142694+0000","last_became_active":"2023-10-16T13:27:41.150669+0000","last_became_peered":"2023-10-16T13:27:41.150669+0000","last_unstale":"2023-10-16T13:27:42.142694+0000","last_undegraded":"2023-10-16T13:27:42.142694+0000","last_fullsized":"2023-10-16T13:27:42.142694+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_clean_scrub_stamp":"2023-10-16T13:27:40.119893+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:38:01.595105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.f","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.099721+0000","last_change":"2023-10-16T13:27:42.096503+0000","last_active":"2023-10-16T13:27:42.099721+0000","last_peered":"2023-10-16T13:27:42.099721+0000","last_clean":"2023-10-16T13:27:42.099721+0000","last_became_active":"2023-10-16T13:27:42.096173+0000","last_became_peered":"2023-10-16T13:27:42.096173+0000","last_unstale":"2023-10-16T13:27:42.099721+0000","last_undegraded":"2023-10-16T13:27:42.099721+0000","last_fullsized":"2023-10-16T13:27:42.099721+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:00:10.823738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":11,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.080588+0000","last_change":"2023-10-16T13:27:41.150504+0000","last_active":"2023-10-16T13:27:42.080588+0000","last_peered":"2023-10-16T13:27:42.080588+0000","last_clean":"2023-10-16T13:27:42.080588+0000","last_became_active":"2023-10-16T13:27:41.149304+0000","last_became_peered":"2023-10-16T13:27:41.149304+0000","last_unstale":"2023-10-16T13:27:42.080588+0000","last_undegraded":"2023-10-16T13:27:42.080588+0000","last_fullsized":"2023-10-16T13:27:42.080588+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_clean_scrub_stamp":"2023-10-16T13:27:40.119893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:03:28.153513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.101270+0000","last_change":"2023-10-16T13:27:42.096736+0000","last_active":"2023-10-16T13:27:42.101270+0000","last_peered":"2023-10-16T13:27:42.101270+0000","last_clean":"2023-10-16T13:27:42.101270+0000","last_became_active":"2023-10-16T13:27:42.096417+0000","last_became_peered":"2023-10-16T13:27:42.096417+0000","last_unstale":"2023-10-16T13:27:42.101270+0000","last_undegraded":"2023-10-16T13:27:42.101270+0000","last_fullsized":"2023-10-16T13:27:42.101270+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:22:18.575043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.b","version":"176'2","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.143981+0000","last_change":"2023-10-16T13:27:41.149613+0000","last_active":"2023-10-16T13:27:42.143981+0000","last_peered":"2023-10-16T13:27:42.143981+0000","last_clean":"2023-10-16T13:27:42.143981+0000","last_became_active":"2023-10-16T13:27:41.148952+0000","last_became_peered":"2023-10-16T13:27:41.148952+0000","last_unstale":"2023-10-16T13:27:42.143981+0000","last_undegraded":"2023-10-16T13:27:42.143981+0000","last_fullsized":"2023-10-16T13:27:42.143981+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_clean_scrub_stamp":"2023-10-16T13:27:40.119893+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:56:46.969224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.097980+0000","last_change":"2023-10-16T13:27:42.096477+0000","last_active":"2023-10-16T13:27:42.097980+0000","last_peered":"2023-10-16T13:27:42.097980+0000","last_clean":"2023-10-16T13:27:42.097980+0000","last_became_active":"2023-10-16T13:27:42.095870+0000","last_became_peered":"2023-10-16T13:27:42.095870+0000","last_unstale":"2023-10-16T13:27:42.097980+0000","last_undegraded":"2023-10-16T13:27:42.097980+0000","last_fullsized":"2023-10-16T13:27:42.097980+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:22:35.735362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":11,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.085410+0000","last_change":"2023-10-16T13:27:41.152283+0000","last_active":"2023-10-16T13:27:42.085410+0000","last_peered":"2023-10-16T13:27:42.085410+0000","last_clean":"2023-10-16T13:27:42.085410+0000","last_became_active":"2023-10-16T13:27:41.151990+0000","last_became_peered":"2023-10-16T13:27:41.151990+0000","last_unstale":"2023-10-16T13:27:42.085410+0000","last_undegraded":"2023-10-16T13:27:42.085410+0000","last_fullsized":"2023-10-16T13:27:42.085410+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_clean_scrub_stamp":"2023-10-16T13:27:40.119893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:11:03.602352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.096884+0000","last_change":"2023-10-16T13:27:42.092196+0000","last_active":"2023-10-16T13:27:42.096884+0000","last_peered":"2023-10-16T13:27:42.096884+0000","last_clean":"2023-10-16T13:27:42.096884+0000","last_became_active":"2023-10-16T13:27:42.091909+0000","last_became_peered":"2023-10-16T13:27:42.091909+0000","last_unstale":"2023-10-16T13:27:42.096884+0000","last_undegraded":"2023-10-16T13:27:42.096884+0000","last_fullsized":"2023-10-16T13:27:42.096884+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:10:22.490244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":11,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.079521+0000","last_change":"2023-10-16T13:27:41.147022+0000","last_active":"2023-10-16T13:27:42.079521+0000","last_peered":"2023-10-16T13:27:42.079521+0000","last_clean":"2023-10-16T13:27:42.079521+0000","last_became_active":"2023-10-16T13:27:41.146669+0000","last_became_peered":"2023-10-16T13:27:41.146669+0000","last_unstale":"2023-10-16T13:27:42.079521+0000","last_undegraded":"2023-10-16T13:27:42.079521+0000","last_fullsized":"2023-10-16T13:27:42.079521+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_clean_scrub_stamp":"2023-10-16T13:27:40.119893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:58:01.851972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.097967+0000","last_change":"2023-10-16T13:27:42.095760+0000","last_active":"2023-10-16T13:27:42.097967+0000","last_peered":"2023-10-16T13:27:42.097967+0000","last_clean":"2023-10-16T13:27:42.097967+0000","last_became_active":"2023-10-16T13:27:42.095454+0000","last_became_peered":"2023-10-16T13:27:42.095454+0000","last_unstale":"2023-10-16T13:27:42.097967+0000","last_undegraded":"2023-10-16T13:27:42.097967+0000","last_fullsized":"2023-10-16T13:27:42.097967+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:28:03.681820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":11,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.087111+0000","last_change":"2023-10-16T13:27:41.148257+0000","last_active":"2023-10-16T13:27:42.087111+0000","last_peered":"2023-10-16T13:27:42.087111+0000","last_clean":"2023-10-16T13:27:42.087111+0000","last_became_active":"2023-10-16T13:27:41.145869+0000","last_became_peered":"2023-10-16T13:27:41.145869+0000","last_unstale":"2023-10-16T13:27:42.087111+0000","last_undegraded":"2023-10-16T13:27:42.087111+0000","last_fullsized":"2023-10-16T13:27:42.087111+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_clean_scrub_stamp":"2023-10-16T13:27:40.119893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:05:26.666331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.098733+0000","last_change":"2023-10-16T13:27:42.098423+0000","last_active":"2023-10-16T13:27:42.098733+0000","last_peered":"2023-10-16T13:27:42.098733+0000","last_clean":"2023-10-16T13:27:42.098733+0000","last_became_active":"2023-10-16T13:27:42.098164+0000","last_became_peered":"2023-10-16T13:27:42.098164+0000","last_unstale":"2023-10-16T13:27:42.098733+0000","last_undegraded":"2023-10-16T13:27:42.098733+0000","last_fullsized":"2023-10-16T13:27:42.098733+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:31:38.628796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.7","version":"176'2","reported_seq":13,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.142873+0000","last_change":"2023-10-16T13:27:41.149885+0000","last_active":"2023-10-16T13:27:42.142873+0000","last_peered":"2023-10-16T13:27:42.142873+0000","last_clean":"2023-10-16T13:27:42.142873+0000","last_became_active":"2023-10-16T13:27:41.147708+0000","last_became_peered":"2023-10-16T13:27:41.147708+0000","last_unstale":"2023-10-16T13:27:42.142873+0000","last_undegraded":"2023-10-16T13:27:42.142873+0000","last_fullsized":"2023-10-16T13:27:42.142873+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_clean_scrub_stamp":"2023-10-16T13:27:40.119893+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:01:24.701443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.091426+0000","last_change":"2023-10-16T13:27:42.086530+0000","last_active":"2023-10-16T13:27:42.091426+0000","last_peered":"2023-10-16T13:27:42.091426+0000","last_clean":"2023-10-16T13:27:42.091426+0000","last_became_active":"2023-10-16T13:27:42.086204+0000","last_became_peered":"2023-10-16T13:27:42.086204+0000","last_unstale":"2023-10-16T13:27:42.091426+0000","last_undegraded":"2023-10-16T13:27:42.091426+0000","last_fullsized":"2023-10-16T13:27:42.091426+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:29:14.085622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":11,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.079242+0000","last_change":"2023-10-16T13:27:41.154787+0000","last_active":"2023-10-16T13:27:42.079242+0000","last_peered":"2023-10-16T13:27:42.079242+0000","last_clean":"2023-10-16T13:27:42.079242+0000","last_became_active":"2023-10-16T13:27:41.154373+0000","last_became_peered":"2023-10-16T13:27:41.154373+0000","last_unstale":"2023-10-16T13:27:42.079242+0000","last_undegraded":"2023-10-16T13:27:42.079242+0000","last_fullsized":"2023-10-16T13:27:42.079242+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_clean_scrub_stamp":"2023-10-16T13:27:40.119893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:52:52.209689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.101302+0000","last_change":"2023-10-16T13:27:42.099248+0000","last_active":"2023-10-16T13:27:42.101302+0000","last_peered":"2023-10-16T13:27:42.101302+0000","last_clean":"2023-10-16T13:27:42.101302+0000","last_became_active":"2023-10-16T13:27:42.098477+0000","last_became_peered":"2023-10-16T13:27:42.098477+0000","last_unstale":"2023-10-16T13:27:42.101302+0000","last_undegraded":"2023-10-16T13:27:42.101302+0000","last_fullsized":"2023-10-16T13:27:42.101302+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:17:05.181133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":11,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.076376+0000","last_change":"2023-10-16T13:27:41.150678+0000","last_active":"2023-10-16T13:27:42.076376+0000","last_peered":"2023-10-16T13:27:42.076376+0000","last_clean":"2023-10-16T13:27:42.076376+0000","last_became_active":"2023-10-16T13:27:41.150185+0000","last_became_peered":"2023-10-16T13:27:41.150185+0000","last_unstale":"2023-10-16T13:27:42.076376+0000","last_undegraded":"2023-10-16T13:27:42.076376+0000","last_fullsized":"2023-10-16T13:27:42.076376+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_clean_scrub_stamp":"2023-10-16T13:27:40.119893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:05:37.745445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.085618+0000","last_change":"2023-10-16T13:27:42.083360+0000","last_active":"2023-10-16T13:27:42.085618+0000","last_peered":"2023-10-16T13:27:42.085618+0000","last_clean":"2023-10-16T13:27:42.085618+0000","last_became_active":"2023-10-16T13:27:42.083059+0000","last_became_peered":"2023-10-16T13:27:42.083059+0000","last_unstale":"2023-10-16T13:27:42.085618+0000","last_undegraded":"2023-10-16T13:27:42.085618+0000","last_fullsized":"2023-10-16T13:27:42.085618+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:51:51.674630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":11,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.075287+0000","last_change":"2023-10-16T13:27:41.148045+0000","last_active":"2023-10-16T13:27:42.075287+0000","last_peered":"2023-10-16T13:27:42.075287+0000","last_clean":"2023-10-16T13:27:42.075287+0000","last_became_active":"2023-10-16T13:27:41.146902+0000","last_became_peered":"2023-10-16T13:27:41.146902+0000","last_unstale":"2023-10-16T13:27:42.075287+0000","last_undegraded":"2023-10-16T13:27:42.075287+0000","last_fullsized":"2023-10-16T13:27:42.075287+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_clean_scrub_stamp":"2023-10-16T13:27:40.119893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:10:21.485201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.098283+0000","last_change":"2023-10-16T13:27:42.092375+0000","last_active":"2023-10-16T13:27:42.098283+0000","last_peered":"2023-10-16T13:27:42.098283+0000","last_clean":"2023-10-16T13:27:42.098283+0000","last_became_active":"2023-10-16T13:27:42.092091+0000","last_became_peered":"2023-10-16T13:27:42.092091+0000","last_unstale":"2023-10-16T13:27:42.098283+0000","last_undegraded":"2023-10-16T13:27:42.098283+0000","last_fullsized":"2023-10-16T13:27:42.098283+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:07:07.625727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":11,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.079917+0000","last_change":"2023-10-16T13:27:41.147629+0000","last_active":"2023-10-16T13:27:42.079917+0000","last_peered":"2023-10-16T13:27:42.079917+0000","last_clean":"2023-10-16T13:27:42.079917+0000","last_became_active":"2023-10-16T13:27:41.147127+0000","last_became_peered":"2023-10-16T13:27:41.147127+0000","last_unstale":"2023-10-16T13:27:42.079917+0000","last_undegraded":"2023-10-16T13:27:42.079917+0000","last_fullsized":"2023-10-16T13:27:42.079917+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_clean_scrub_stamp":"2023-10-16T13:27:40.119893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:37:16.544959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.091833+0000","last_change":"2023-10-16T13:27:42.083297+0000","last_active":"2023-10-16T13:27:42.091833+0000","last_peered":"2023-10-16T13:27:42.091833+0000","last_clean":"2023-10-16T13:27:42.091833+0000","last_became_active":"2023-10-16T13:27:42.082995+0000","last_became_peered":"2023-10-16T13:27:42.082995+0000","last_unstale":"2023-10-16T13:27:42.091833+0000","last_undegraded":"2023-10-16T13:27:42.091833+0000","last_fullsized":"2023-10-16T13:27:42.091833+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:19:54.622529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":11,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.081204+0000","last_change":"2023-10-16T13:27:41.137848+0000","last_active":"2023-10-16T13:27:42.081204+0000","last_peered":"2023-10-16T13:27:42.081204+0000","last_clean":"2023-10-16T13:27:42.081204+0000","last_became_active":"2023-10-16T13:27:41.137530+0000","last_became_peered":"2023-10-16T13:27:41.137530+0000","last_unstale":"2023-10-16T13:27:42.081204+0000","last_undegraded":"2023-10-16T13:27:42.081204+0000","last_fullsized":"2023-10-16T13:27:42.081204+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_clean_scrub_stamp":"2023-10-16T13:27:40.119893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:30:00.346674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.082904+0000","last_change":"2023-10-16T13:27:42.081718+0000","last_active":"2023-10-16T13:27:42.082904+0000","last_peered":"2023-10-16T13:27:42.082904+0000","last_clean":"2023-10-16T13:27:42.082904+0000","last_became_active":"2023-10-16T13:27:42.081414+0000","last_became_peered":"2023-10-16T13:27:42.081414+0000","last_unstale":"2023-10-16T13:27:42.082904+0000","last_undegraded":"2023-10-16T13:27:42.082904+0000","last_fullsized":"2023-10-16T13:27:42.082904+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:35:13.025389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":11,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.086849+0000","last_change":"2023-10-16T13:27:41.144713+0000","last_active":"2023-10-16T13:27:42.086849+0000","last_peered":"2023-10-16T13:27:42.086849+0000","last_clean":"2023-10-16T13:27:42.086849+0000","last_became_active":"2023-10-16T13:27:41.144410+0000","last_became_peered":"2023-10-16T13:27:41.144410+0000","last_unstale":"2023-10-16T13:27:42.086849+0000","last_undegraded":"2023-10-16T13:27:42.086849+0000","last_fullsized":"2023-10-16T13:27:42.086849+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_clean_scrub_stamp":"2023-10-16T13:27:40.119893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:12:12.493088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.097592+0000","last_change":"2023-10-16T13:27:42.093052+0000","last_active":"2023-10-16T13:27:42.097592+0000","last_peered":"2023-10-16T13:27:42.097592+0000","last_clean":"2023-10-16T13:27:42.097592+0000","last_became_active":"2023-10-16T13:27:42.091691+0000","last_became_peered":"2023-10-16T13:27:42.091691+0000","last_unstale":"2023-10-16T13:27:42.097592+0000","last_undegraded":"2023-10-16T13:27:42.097592+0000","last_fullsized":"2023-10-16T13:27:42.097592+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:13:38.435362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":11,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.085352+0000","last_change":"2023-10-16T13:27:41.155093+0000","last_active":"2023-10-16T13:27:42.085352+0000","last_peered":"2023-10-16T13:27:42.085352+0000","last_clean":"2023-10-16T13:27:42.085352+0000","last_became_active":"2023-10-16T13:27:41.154818+0000","last_became_peered":"2023-10-16T13:27:41.154818+0000","last_unstale":"2023-10-16T13:27:42.085352+0000","last_undegraded":"2023-10-16T13:27:42.085352+0000","last_fullsized":"2023-10-16T13:27:42.085352+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_clean_scrub_stamp":"2023-10-16T13:27:40.119893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:51:50.669587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":9,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.097790+0000","last_change":"2023-10-16T13:27:42.091585+0000","last_active":"2023-10-16T13:27:42.097790+0000","last_peered":"2023-10-16T13:27:42.097790+0000","last_clean":"2023-10-16T13:27:42.097790+0000","last_became_active":"2023-10-16T13:27:42.091244+0000","last_became_peered":"2023-10-16T13:27:42.091244+0000","last_unstale":"2023-10-16T13:27:42.097790+0000","last_undegraded":"2023-10-16T13:27:42.097790+0000","last_fullsized":"2023-10-16T13:27:42.097790+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:41.124936+0000","last_clean_scrub_stamp":"2023-10-16T13:27:41.124936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:37:51.020671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.14","version":"176'6","reported_seq":17,"reported_epoch":176,"state":"active+clean","last_fresh":"2023-10-16T13:27:42.145439+0000","last_change":"2023-10-16T13:27:41.149614+0000","last_active":"2023-10-16T13:27:42.145439+0000","last_peered":"2023-10-16T13:27:42.145439+0000","last_clean":"2023-10-16T13:27:42.145439+0000","last_became_active":"2023-10-16T13:27:41.149202+0000","last_became_peered":"2023-10-16T13:27:41.149202+0000","last_unstale":"2023-10-16T13:27:42.145439+0000","last_undegraded":"2023-10-16T13:27:42.145439+0000","last_fullsized":"2023-10-16T13:27:42.145439+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:27:40.119893+0000","last_clean_scrub_stamp":"2023-10-16T13:27:40.119893+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:38:44.970839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"166'271","reported_seq":881,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-16T13:27:41.138752+0000","last_change":"2023-10-16T12:10:34.746820+0000","last_active":"2023-10-16T13:27:41.138752+0000","last_peered":"2023-10-16T13:27:41.138752+0000","last_clean":"2023-10-16T13:27:41.138752+0000","last_became_active":"2023-10-16T12:10:34.745574+0000","last_became_peered":"2023-10-16T12:10:34.745574+0000","last_unstale":"2023-10-16T13:27:41.138752+0000","last_undegraded":"2023-10-16T13:27:41.138752+0000","last_fullsized":"2023-10-16T13:27:41.138752+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_clean_scrub_stamp":"2023-10-16T12:10:33.577111+0000","objects_scrubbed":0,"log_size":271,"log_dups_size":0,"ondisk_log_size":271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:14.461986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1460,"num_read_kb":2984,"num_write":439,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":21,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":22,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1460,"num_read_kb":2984,"num_write":439,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":271,"ondisk_log_size":271,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575767,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51944,"kb_used_data":2716,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319896,"statfs":{"total":96636764160,"available":96583573504,"internally_reserved":0,"allocated":2781184,"data_stored":1703782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":50395595},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:27:06 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.63200000000000001,"15min":0.626},"min":{"1min":0.45800000000000002,"5min":0.39800000000000002,"15min":0.30399999999999999},"max":{"1min":0.78400000000000003,"5min":0.874,"15min":0.88200000000000001},"last":3.218},{"interface":"front","average":{"1min":0.629,"5min":0.63,"15min":0.63100000000000001},"min":{"1min":0.39900000000000002,"5min":0.38500000000000001,"15min":0.375},"max":{"1min":0.80900000000000005,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":9.1270000000000007}]},{"osd":1,"last update":"Mon Oct 16 13:27:06 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.65000000000000002,"15min":0.63600000000000001},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.34000000000000002},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.91700000000000004},"last":8.7539999999999996},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.64100000000000001,"15min":0.63200000000000001},"min":{"1min":0.47299999999999998,"5min":0.41199999999999998,"15min":0.36899999999999999},"max":{"1min":0.81899999999999995,"5min":0.85999999999999999,"15min":0.99199999999999999},"last":9.173}]},{"osd":2,"last update":"Mon Oct 16 13:27:06 2023","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.70199999999999996,"15min":0.69299999999999995},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.38500000000000001},"max":{"1min":0.80500000000000005,"5min":0.92900000000000005,"15min":1.03},"last":6.5869999999999997},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.70199999999999996,"15min":0.69899999999999995},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.41099999999999998},"max":{"1min":0.88300000000000001,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":3.0640000000000001}]},{"osd":3,"last update":"Mon Oct 16 13:27:06 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.64900000000000002,"15min":0.64400000000000002},"min":{"1min":0.45700000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.83199999999999996,"5min":0.84699999999999998,"15min":0.86299999999999999},"last":3.1299999999999999},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.65900000000000003,"15min":0.65500000000000003},"min":{"1min":0.503,"5min":0.48399999999999999,"15min":0.40899999999999997},"max":{"1min":0.85099999999999998,"5min":0.93999999999999995,"15min":0.95199999999999996},"last":8.9920000000000009}]},{"osd":4,"last update":"Mon Oct 16 13:27:06 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.53200000000000003,"15min":0.52800000000000002},"min":{"1min":0.36899999999999999,"5min":0.32800000000000001,"15min":0.313},"max":{"1min":0.79900000000000004,"5min":0.84499999999999997,"15min":0.874},"last":6.3769999999999998},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.61899999999999999,"15min":0.62},"min":{"1min":0.47499999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.80600000000000005,"5min":0.90200000000000002,"15min":0.90900000000000003},"last":3.1779999999999999}]},{"osd":5,"last update":"Mon Oct 16 13:27:06 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.54800000000000004,"15min":0.54300000000000004},"min":{"1min":0.39400000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.81799999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":8.9209999999999994},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.59199999999999997,"15min":0.59399999999999997},"min":{"1min":0.377,"5min":0.377,"15min":0.376},"max":{"1min":0.84899999999999998,"5min":1.0329999999999999,"15min":1.135},"last":6.7560000000000002}]},{"osd":6,"last update":"Mon Oct 16 13:27:06 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.63500000000000001,"15min":0.63400000000000001},"min":{"1min":0.44400000000000001,"5min":0.41899999999999998,"15min":0.38500000000000001},"max":{"1min":0.88100000000000001,"5min":0.90200000000000002,"15min":0.96099999999999997},"last":8.8279999999999994},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.40000000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.84999999999999998,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":3.2599999999999998}]}]},{"osd":6,"up_from":13,"seq":55834575765,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41644,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330196,"statfs":{"total":96636764160,"available":96594120704,"internally_reserved":0,"allocated":2187264,"data_stored":1113414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:27:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.69599999999999995,"15min":0.67800000000000005},"min":{"1min":0.434,"5min":0.434,"15min":0.29899999999999999},"max":{"1min":0.86699999999999999,"5min":0.90800000000000003,"15min":0.91400000000000003},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.68799999999999994,"15min":0.68500000000000005},"min":{"1min":0.48999999999999999,"5min":0.42099999999999999,"15min":0.34699999999999998},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.93100000000000005},"last":0.5}]},{"osd":1,"last update":"Mon Oct 16 13:27:05 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.63700000000000001,"15min":0.63100000000000001},"min":{"1min":0.48599999999999999,"5min":0.36099999999999999,"15min":0.35499999999999998},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.626,"5min":0.63200000000000001,"15min":0.63600000000000001},"min":{"1min":0.44900000000000001,"5min":0.38400000000000001,"15min":0.34899999999999998},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.95599999999999996},"last":0.54400000000000004}]},{"osd":2,"last update":"Mon Oct 16 13:27:05 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.64700000000000002,"15min":0.63900000000000001},"min":{"1min":0.46100000000000002,"5min":0.44400000000000001,"15min":0.33500000000000002},"max":{"1min":0.84199999999999997,"5min":0.93899999999999995,"15min":0.94199999999999995},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.63500000000000001,"15min":0.63700000000000001},"min":{"1min":0.50800000000000001,"5min":0.39700000000000002,"15min":0.33100000000000002},"max":{"1min":0.79600000000000004,"5min":0.86399999999999999,"15min":0.93100000000000005},"last":0.59799999999999998}]},{"osd":3,"last update":"Mon Oct 16 13:27:05 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.69099999999999995,"15min":0.68700000000000006},"min":{"1min":0.60499999999999998,"5min":0.42599999999999999,"15min":0.374},"max":{"1min":0.874,"5min":1.288,"15min":1.288},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.746,"15min":0.72899999999999998},"min":{"1min":0.60499999999999998,"5min":0.504,"15min":0.32300000000000001},"max":{"1min":0.81999999999999995,"5min":1.351,"15min":1.351},"last":0.70299999999999996}]},{"osd":4,"last update":"Mon Oct 16 13:27:05 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.36099999999999999,"5min":0.34100000000000003,"15min":0.32000000000000001},"max":{"1min":0.78200000000000003,"5min":0.85599999999999998,"15min":0.86299999999999999},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.60999999999999999,"15min":0.60899999999999999},"min":{"1min":0.48899999999999999,"5min":0.35199999999999998,"15min":0.32000000000000001},"max":{"1min":0.82299999999999995,"5min":0.89600000000000002,"15min":0.90100000000000002},"last":0.66600000000000004}]},{"osd":5,"last update":"Mon Oct 16 13:27:05 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.56899999999999995,"15min":0.57199999999999995},"min":{"1min":0.41799999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.83199999999999996,"5min":0.85899999999999999,"15min":0.91800000000000004},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.64500000000000002,"15min":0.63},"min":{"1min":0.45500000000000002,"5min":0.34399999999999997,"15min":0.32700000000000001},"max":{"1min":0.82099999999999995,"5min":0.91000000000000003,"15min":0.93799999999999994},"last":0.73699999999999999}]},{"osd":7,"last update":"Mon Oct 16 13:27:05 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.58599999999999997,"15min":0.57499999999999996},"min":{"1min":0.41499999999999998,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.629,"15min":0.626},"min":{"1min":0.48599999999999999,"5min":0.39300000000000002,"15min":0.378},"max":{"1min":0.83999999999999997,"5min":0.92000000000000004,"15min":0.98899999999999999},"last":0.68999999999999995}]}]},{"osd":1,"up_from":13,"seq":55834575765,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41640,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330200,"statfs":{"total":96636764160,"available":96594124800,"internally_reserved":0,"allocated":2187264,"data_stored":1113414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":40434121},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:27:04 2023","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.41899999999999998,"15min":0.41399999999999998},"min":{"1min":0.27400000000000002,"5min":0.27000000000000002,"15min":0.245},"max":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.76600000000000001},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.505,"15min":0.49399999999999999},"min":{"1min":0.40400000000000003,"5min":0.35299999999999998,"15min":0.34699999999999998},"max":{"1min":0.69899999999999995,"5min":0.71199999999999997,"15min":0.74199999999999999},"last":0.374}]},{"osd":2,"last update":"Mon Oct 16 13:27:04 2023","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.46400000000000002,"15min":0.45400000000000001},"min":{"1min":0.34399999999999997,"5min":0.28999999999999998,"15min":0.28100000000000003},"max":{"1min":0.627,"5min":0.751,"15min":0.751},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.505,"5min":0.497,"15min":0.495},"min":{"1min":0.35999999999999999,"5min":0.32300000000000001,"15min":0.29299999999999998},"max":{"1min":0.67600000000000005,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.48299999999999998}]},{"osd":3,"last update":"Mon Oct 16 13:27:04 2023","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.435,"15min":0.42599999999999999},"min":{"1min":0.32600000000000001,"5min":0.309,"15min":0.29199999999999998},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.71099999999999997},"last":0.312},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.48899999999999999,"15min":0.48599999999999999},"min":{"1min":0.39000000000000001,"5min":0.311,"15min":0.31},"max":{"1min":0.65700000000000003,"5min":0.66900000000000004,"15min":0.75},"last":0.53000000000000003}]},{"osd":4,"last update":"Mon Oct 16 13:27:04 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.67200000000000004,"15min":0.66800000000000004},"min":{"1min":0.50600000000000001,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98899999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.69199999999999995,"15min":0.68100000000000005},"min":{"1min":0.51100000000000001,"5min":0.45900000000000002,"15min":0.42099999999999999},"max":{"1min":0.92000000000000004,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.629}]},{"osd":5,"last update":"Mon Oct 16 13:27:04 2023","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.71799999999999997,"15min":0.71599999999999997},"min":{"1min":0.55900000000000005,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":1.046},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.746,"15min":0.73399999999999999},"min":{"1min":0.61299999999999999,"5min":0.52500000000000002,"15min":0.47099999999999997},"max":{"1min":0.93600000000000005,"5min":0.98499999999999999,"15min":1.0089999999999999},"last":0.71199999999999997}]},{"osd":6,"last update":"Mon Oct 16 13:27:04 2023","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.68799999999999994,"15min":0.68700000000000006},"min":{"1min":0.52700000000000002,"5min":0.51100000000000001,"15min":0.45000000000000001},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.71799999999999997,"15min":0.71399999999999997},"min":{"1min":0.55000000000000004,"5min":0.46700000000000003,"15min":0.432},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.64400000000000002}]},{"osd":7,"last update":"Mon Oct 16 13:27:04 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.73999999999999999,"15min":0.73799999999999999},"min":{"1min":0.49299999999999999,"5min":0.48799999999999999,"15min":0.44},"max":{"1min":0.876,"5min":1.024,"15min":1.04},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.76500000000000001,"15min":0.75600000000000001},"min":{"1min":0.60499999999999998,"5min":0.56299999999999994,"15min":0.498},"max":{"1min":0.997,"5min":0.997,"15min":1.0269999999999999},"last":0.79700000000000004}]}]},{"osd":0,"up_from":13,"seq":55834575766,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42224,"kb_used_data":2716,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329616,"statfs":{"total":96636764160,"available":96593526784,"internally_reserved":0,"allocated":2781184,"data_stored":1703782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 13:26:57 2023","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.42399999999999999,"15min":0.41499999999999998},"min":{"1min":0.28299999999999997,"5min":0.26300000000000001,"15min":0.23499999999999999},"max":{"1min":0.61799999999999999,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.45400000000000001,"15min":0.45100000000000001},"min":{"1min":0.30199999999999999,"5min":0.29899999999999999,"15min":0.28199999999999997},"max":{"1min":0.63400000000000001,"5min":0.745,"15min":0.75800000000000001},"last":0.34399999999999997}]},{"osd":2,"last update":"Mon Oct 16 13:26:57 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.46000000000000002,"15min":0.45900000000000002},"min":{"1min":0.36899999999999999,"5min":0.30399999999999999,"15min":0.28599999999999998},"max":{"1min":0.61099999999999999,"5min":0.70799999999999996,"15min":0.73399999999999999},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.48899999999999999,"15min":0.48499999999999999},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.29499999999999998},"max":{"1min":0.61699999999999999,"5min":0.67100000000000004,"15min":0.75700000000000001},"last":0.42199999999999999}]},{"osd":3,"last update":"Mon Oct 16 13:26:57 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.45200000000000001,"15min":0.44900000000000001},"min":{"1min":0.311,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.60099999999999998,"5min":0.66100000000000003,"15min":0.70199999999999996},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.53200000000000003,"15min":0.52800000000000002},"min":{"1min":0.40899999999999997,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.78200000000000003,"5min":0.81299999999999994,"15min":0.82099999999999995},"last":0.622}]},{"osd":4,"last update":"Mon Oct 16 13:26:57 2023","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.71999999999999997,"15min":0.68700000000000006},"min":{"1min":0.61799999999999999,"5min":0.45500000000000002,"15min":0.40200000000000002},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0369999999999999},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.71199999999999997,"15min":0.68700000000000006},"min":{"1min":0.47999999999999998,"5min":0.42399999999999999,"15min":0.39000000000000001},"max":{"1min":0.92800000000000005,"5min":1.0429999999999999,"15min":1.0669999999999999},"last":0.55300000000000005}]},{"osd":5,"last update":"Mon Oct 16 13:26:57 2023","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.70699999999999996,"15min":0.68300000000000005},"min":{"1min":0.44900000000000001,"5min":0.438,"15min":0.40300000000000002},"max":{"1min":1.0049999999999999,"5min":1.034,"15min":1.034},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.73899999999999999,"15min":0.71699999999999997},"min":{"1min":0.51500000000000001,"5min":0.47899999999999998,"15min":0.46700000000000003},"max":{"1min":0.92100000000000004,"5min":1.109,"15min":1.109},"last":0.51800000000000002}]},{"osd":6,"last update":"Mon Oct 16 13:26:57 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.74299999999999999,"15min":0.72999999999999998},"min":{"1min":0.62,"5min":0.49199999999999999,"15min":0.42999999999999999},"max":{"1min":0.93300000000000005,"5min":1.0660000000000001,"15min":1.109},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.747,"15min":0.73099999999999998},"min":{"1min":0.52400000000000002,"5min":0.46999999999999997,"15min":0.44500000000000001},"max":{"1min":0.95299999999999996,"5min":1.056,"15min":1.0720000000000001},"last":0.50600000000000001}]},{"osd":7,"last update":"Mon Oct 16 13:26:57 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.75600000000000001,"15min":0.73699999999999999},"min":{"1min":0.58099999999999996,"5min":0.51800000000000002,"15min":0.5},"max":{"1min":0.94999999999999996,"5min":1.0149999999999999,"15min":1.0880000000000001},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.753,"5min":0.754,"15min":0.73299999999999998},"min":{"1min":0.56699999999999995,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.93200000000000005,"5min":1.0840000000000001,"15min":1.0920000000000001},"last":0.60599999999999998}]}]},{"osd":2,"up_from":13,"seq":55834575766,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51392,"kb_used_data":2156,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320448,"statfs":{"total":96636764160,"available":96584138752,"internally_reserved":0,"allocated":2207744,"data_stored":1120721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:26:52 2023","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.41399999999999998,"15min":0.41299999999999998},"min":{"1min":0.25700000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.63400000000000001,"5min":0.65600000000000003,"15min":0.77800000000000002},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.499,"5min":0.502,"15min":0.48799999999999999},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.32500000000000001},"max":{"1min":0.78100000000000003,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.30299999999999999}]},{"osd":1,"last update":"Mon Oct 16 13:26:52 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.53500000000000003,"15min":0.51900000000000002},"min":{"1min":0.40999999999999998,"5min":0.34999999999999998,"15min":0.32400000000000001},"max":{"1min":0.63100000000000001,"5min":0.79600000000000004,"15min":0.879},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.47599999999999998,"15min":0.47299999999999998},"min":{"1min":0.33500000000000002,"5min":0.30599999999999999,"15min":0.29699999999999999},"max":{"1min":0.57199999999999995,"5min":0.78200000000000003,"15min":0.79000000000000004},"last":0.35699999999999998}]},{"osd":3,"last update":"Mon Oct 16 13:26:52 2023","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.44400000000000001,"15min":0.45000000000000001},"min":{"1min":0.31,"5min":0.29499999999999998,"15min":0.29099999999999998},"max":{"1min":0.52600000000000002,"5min":0.66200000000000003,"15min":0.81200000000000006},"last":0.248},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.56499999999999995,"15min":0.55800000000000005},"min":{"1min":0.42599999999999999,"5min":0.38900000000000001,"15min":0.34499999999999997},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":1.008},"last":0.379}]},{"osd":4,"last update":"Mon Oct 16 13:26:52 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.70599999999999996,"15min":0.70499999999999996},"min":{"1min":0.53600000000000003,"5min":0.41599999999999998,"15min":0.379},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.71699999999999997,"15min":0.71099999999999997},"min":{"1min":0.504,"5min":0.48799999999999999,"15min":0.45700000000000002},"max":{"1min":0.996,"5min":0.996,"15min":1.0109999999999999},"last":0.753}]},{"osd":5,"last update":"Mon Oct 16 13:26:52 2023","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.72199999999999998,"15min":0.70899999999999996},"min":{"1min":0.54300000000000004,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.84399999999999997,"5min":0.95299999999999996,"15min":1.056},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.68300000000000005,"15min":0.68400000000000005},"min":{"1min":0.497,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.879,"5min":0.96899999999999997,"15min":1.0740000000000001},"last":0.58699999999999997}]},{"osd":6,"last update":"Mon Oct 16 13:26:52 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.78600000000000003,"5min":1.046,"15min":1.345},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.74199999999999999,"15min":0.73499999999999999},"min":{"1min":0.57199999999999995,"5min":0.51900000000000002,"15min":0.51600000000000001},"max":{"1min":0.95899999999999996,"5min":1.006,"15min":1.2430000000000001},"last":0.57699999999999996}]},{"osd":7,"last update":"Mon Oct 16 13:26:52 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.72699999999999998,"15min":0.72499999999999998},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.88800000000000001,"5min":1.0620000000000001,"15min":1.115},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.747,"15min":0.73799999999999999},"min":{"1min":0.60299999999999998,"5min":0.54600000000000004,"15min":0.51100000000000001},"max":{"1min":1.018,"5min":1.018,"15min":1.1759999999999999},"last":0.60599999999999998}]}]},{"osd":3,"up_from":13,"seq":55834575765,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47228,"kb_used_data":2156,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324612,"statfs":{"total":96636764160,"available":96588402688,"internally_reserved":0,"allocated":2207744,"data_stored":1121225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":46135752},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:26:57 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.40799999999999997,"15min":0.40999999999999998},"min":{"1min":0.312,"5min":0.27800000000000002,"15min":0.249},"max":{"1min":0.58499999999999996,"5min":0.629,"15min":0.73099999999999998},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.438,"5min":0.433,"15min":0.439},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.70299999999999996,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.36599999999999999}]},{"osd":1,"last update":"Mon Oct 16 13:26:57 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.45200000000000001,"15min":0.441},"min":{"1min":0.313,"5min":0.30299999999999999,"15min":0.27500000000000002},"max":{"1min":0.65600000000000003,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.48899999999999999,"15min":0.49099999999999999},"min":{"1min":0.35299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.69999999999999996,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.41099999999999998}]},{"osd":2,"last update":"Mon Oct 16 13:26:57 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.495,"15min":0.501},"min":{"1min":0.39600000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.67600000000000005,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.52300000000000002,"15min":0.53200000000000003},"min":{"1min":0.372,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73299999999999998,"5min":0.79300000000000004,"15min":0.89800000000000002},"last":0.46800000000000003}]},{"osd":4,"last update":"Mon Oct 16 13:26:57 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.66200000000000003,"15min":0.68000000000000005},"min":{"1min":0.46600000000000003,"5min":0.435,"15min":0.41799999999999998},"max":{"1min":0.88200000000000001,"5min":1.0640000000000001,"15min":1.1060000000000001},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.68400000000000005,"15min":0.69899999999999995},"min":{"1min":0.50900000000000001,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.97799999999999998,"5min":1.008,"15min":1.0860000000000001},"last":0.63500000000000001}]},{"osd":5,"last update":"Mon Oct 16 13:26:57 2023","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.70599999999999996,"15min":0.72099999999999997},"min":{"1min":0.53600000000000003,"5min":0.48499999999999999,"15min":0.47699999999999998},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.69499999999999995,"15min":0.70699999999999996},"min":{"1min":0.51200000000000001,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.73799999999999999}]},{"osd":6,"last update":"Mon Oct 16 13:26:57 2023","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.72199999999999998,"15min":0.71599999999999997},"min":{"1min":0.496,"5min":0.496,"15min":0.42799999999999999},"max":{"1min":0.97999999999999998,"5min":1.0469999999999999,"15min":1.0800000000000001},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.73999999999999999,"15min":0.751},"min":{"1min":0.57599999999999996,"5min":0.50600000000000001,"15min":0.46999999999999997},"max":{"1min":0.995,"5min":1.073,"15min":1.1220000000000001},"last":0.71999999999999997}]},{"osd":7,"last update":"Mon Oct 16 13:26:57 2023","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.73099999999999998,"15min":0.73499999999999999},"min":{"1min":0.53600000000000003,"5min":0.502,"15min":0.49099999999999999},"max":{"1min":1.0409999999999999,"5min":1.099,"15min":1.099},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.75600000000000001,"15min":0.76000000000000001},"min":{"1min":0.59799999999999998,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.054,"5min":1.0609999999999999,"15min":1.131},"last":0.70799999999999996}]}]},{"osd":4,"up_from":13,"seq":55834575764,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29180,"kb_used_data":2160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342660,"statfs":{"total":96636764160,"available":96606883840,"internally_reserved":0,"allocated":2211840,"data_stored":1122221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:26:57 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.62},"min":{"1min":0.40799999999999997,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.84199999999999997,"5min":0.85199999999999998,"15min":0.93600000000000005},"last":0.745},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.64600000000000002,"15min":0.63600000000000001},"min":{"1min":0.48599999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.76400000000000001}]},{"osd":1,"last update":"Mon Oct 16 13:26:57 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.65100000000000002,"15min":0.64500000000000002},"min":{"1min":0.51500000000000001,"5min":0.34300000000000003,"15min":0.33400000000000002},"max":{"1min":0.78500000000000003,"5min":0.878,"15min":0.93799999999999994},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64500000000000002,"15min":0.63900000000000001},"min":{"1min":0.45900000000000002,"5min":0.42699999999999999,"15min":0.30199999999999999},"max":{"1min":0.79300000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.68899999999999995}]},{"osd":2,"last update":"Mon Oct 16 13:26:57 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.32800000000000001},"max":{"1min":0.81299999999999994,"5min":0.90000000000000002,"15min":1.091},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.67800000000000005,"15min":0.65800000000000003},"min":{"1min":0.46400000000000002,"5min":0.38,"15min":0.38},"max":{"1min":0.82599999999999996,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.625}]},{"osd":3,"last update":"Mon Oct 16 13:26:57 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.64900000000000002,"15min":0.64800000000000002},"min":{"1min":0.441,"5min":0.41699999999999998,"15min":0.39400000000000002},"max":{"1min":0.80000000000000004,"5min":0.89200000000000002,"15min":0.95399999999999996},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.69499999999999995,"15min":0.67700000000000005},"min":{"1min":0.48999999999999999,"5min":0.41599999999999998,"15min":0.41499999999999998},"max":{"1min":0.80800000000000005,"5min":0.86799999999999999,"15min":0.98899999999999999},"last":0.77900000000000003}]},{"osd":5,"last update":"Mon Oct 16 13:26:57 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.495,"15min":0.49399999999999999},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.309},"max":{"1min":0.71799999999999997,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.499},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.53800000000000003,"15min":0.54000000000000004},"min":{"1min":0.40799999999999997,"5min":0.31,"15min":0.31},"max":{"1min":0.748,"5min":0.93000000000000005,"15min":0.97099999999999997},"last":0.56200000000000006}]},{"osd":6,"last update":"Mon Oct 16 13:26:57 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.55500000000000005,"15min":0.56399999999999995},"min":{"1min":0.43099999999999999,"5min":0.34499999999999997,"15min":0.32800000000000001},"max":{"1min":0.88600000000000001,"5min":0.90900000000000003,"15min":1.046},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.65200000000000002,"15min":0.65500000000000003},"min":{"1min":0.433,"5min":0.433,"15min":0.34999999999999998},"max":{"1min":0.88200000000000001,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.80300000000000005}]},{"osd":7,"last update":"Mon Oct 16 13:26:57 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.60199999999999998,"15min":0.59799999999999998},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.84799999999999998,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.65200000000000002,"15min":0.64900000000000002},"min":{"1min":0.40000000000000002,"5min":0.39500000000000002,"15min":0.38400000000000001},"max":{"1min":0.92000000000000004,"5min":0.94899999999999995,"15min":1.1359999999999999},"last":0.66500000000000004}]}]},{"osd":5,"up_from":13,"seq":55834575765,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51380,"kb_used_data":2148,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320460,"statfs":{"total":96636764160,"available":96584151040,"internally_reserved":0,"allocated":2199552,"data_stored":1120609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:26:47 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.63200000000000001,"15min":0.623},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.34599999999999997},"max":{"1min":0.83899999999999997,"5min":0.87,"15min":0.92900000000000005},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.622,"5min":0.64900000000000002,"15min":0.65000000000000002},"min":{"1min":0.432,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.83899999999999997,"5min":0.89100000000000001,"15min":0.94899999999999995},"last":0.27500000000000002}]},{"osd":1,"last update":"Mon Oct 16 13:26:47 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.65400000000000003,"15min":0.65200000000000002},"min":{"1min":0.40699999999999997,"5min":0.378,"15min":0.37},"max":{"1min":0.81799999999999995,"5min":0.91800000000000004,"15min":0.93899999999999995},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.66200000000000003,"15min":0.65600000000000003},"min":{"1min":0.51300000000000001,"5min":0.40000000000000002,"15min":0.38100000000000001},"max":{"1min":0.872,"5min":0.90200000000000002,"15min":0.93300000000000005},"last":0.47799999999999998}]},{"osd":2,"last update":"Mon Oct 16 13:26:47 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.67500000000000004,"15min":0.65600000000000003},"min":{"1min":0.441,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.93300000000000005,"5min":1.091,"15min":1.091},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.64300000000000002,"15min":0.64700000000000002},"min":{"1min":0.49299999999999999,"5min":0.42699999999999999,"15min":0.35799999999999998},"max":{"1min":0.80400000000000005,"5min":0.94099999999999995,"15min":0.95699999999999996},"last":0.38300000000000001}]},{"osd":3,"last update":"Mon Oct 16 13:26:47 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.66700000000000004,"15min":0.66200000000000003},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.95499999999999996,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.68899999999999995,"15min":0.68400000000000005},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.41099999999999998},"max":{"1min":0.86099999999999999,"5min":1.042,"15min":1.042},"last":1.357}]},{"osd":4,"last update":"Mon Oct 16 13:26:47 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.51300000000000001,"15min":0.51600000000000001},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.315},"max":{"1min":0.81399999999999995,"5min":0.877,"15min":0.877},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59199999999999997,"15min":0.57999999999999996},"min":{"1min":0.372,"5min":0.372,"15min":0.35199999999999998},"max":{"1min":0.872,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.45200000000000001}]},{"osd":6,"last update":"Mon Oct 16 13:26:47 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57399999999999995,"15min":0.56699999999999995},"min":{"1min":0.39800000000000002,"5min":0.34599999999999997,"15min":0.34399999999999997},"max":{"1min":0.80000000000000004,"5min":0.88700000000000001,"15min":0.93100000000000005},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.628,"15min":0.61299999999999999},"min":{"1min":0.40300000000000002,"5min":0.38700000000000001,"15min":0.312},"max":{"1min":0.88700000000000001,"5min":1.006,"15min":1.006},"last":0.54500000000000004}]},{"osd":7,"last update":"Mon Oct 16 13:26:47 2023","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.66300000000000003,"15min":0.64100000000000001},"min":{"1min":0.44600000000000001,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.91500000000000004,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.69299999999999995,"15min":0.67200000000000004},"min":{"1min":0.49299999999999999,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.90200000000000002,"5min":0.98899999999999999,"15min":1.022},"last":0.50700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T13:27:46.721 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-16T13:27:46.722 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-16T13:27:46.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.884+0000 7f428b2bd700 1 Processor -- start 2023-10-16T13:27:46.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.884+0000 7f428b2bd700 1 -- start start 2023-10-16T13:27:46.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.885+0000 7f428b2bd700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f4284158050 0x7f42840cdf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:46.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.885+0000 7f428b2bd700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f428406eab0 con 0x7f42840ce470 2023-10-16T13:27:46.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.885+0000 7f428b2bd700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f428406e4f0 con 0x7f42840cc1d0 2023-10-16T13:27:46.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.885+0000 7f428b2bd700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f428406e7d0 con 0x7f4284158050 2023-10-16T13:27:46.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.885+0000 7f4288858700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f42840ce470 0x7f428406da30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37810/0 (socket says 172.21.15.16:37810) 2023-10-16T13:27:46.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.885+0000 7f4288858700 1 -- 172.21.15.16:0/4062561769 learned_addr learned my addr 172.21.15.16:0/4062561769 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:27:46.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.885+0000 7f4289059700 1 --2- 172.21.15.16:0/4062561769 >> v2:172.21.15.49:3300/0 conn(0x7f4284158050 0x7f42840cdf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:46.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.885+0000 7f427bfff700 1 -- 172.21.15.16:0/4062561769 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 615253045 0 0) 0x7f428406eab0 con 0x7f42840ce470 2023-10-16T13:27:46.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.885+0000 7f427bfff700 1 -- 172.21.15.16:0/4062561769 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42700035b0 con 0x7f42840ce470 2023-10-16T13:27:46.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.885+0000 7f4289059700 1 -- 172.21.15.16:0/4062561769 >> v1:172.21.15.49:6789/0 conn(0x7f42840cc1d0 legacy=0x7f42840cc660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:46.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.885+0000 7f4289059700 1 -- 172.21.15.16:0/4062561769 >> v1:172.21.15.16:6789/0 conn(0x7f42840ce470 legacy=0x7f428406da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:46.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.885+0000 7f427bfff700 1 -- 172.21.15.16:0/4062561769 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1295388905 0 0) 0x7f42700035b0 con 0x7f42840ce470 2023-10-16T13:27:46.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.885+0000 7f4289059700 1 -- 172.21.15.16:0/4062561769 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42840cb9d0 con 0x7f4284158050 2023-10-16T13:27:46.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.886+0000 7f4289059700 1 --2- 172.21.15.16:0/4062561769 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4284158050 0x7f42840cdf30 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f4274009820 tx=0x7f4274009b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=79633f3b600d7e75 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:46.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.886+0000 7f427bfff700 1 -- 172.21.15.16:0/4062561769 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4280001670 con 0x7f4284158050 2023-10-16T13:27:46.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.886+0000 7f427bfff700 1 -- 172.21.15.16:0/4062561769 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4274005c30 con 0x7f4284158050 2023-10-16T13:27:46.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.886+0000 7f427bfff700 1 -- 172.21.15.16:0/4062561769 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4274004030 con 0x7f4284158050 2023-10-16T13:27:46.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.886+0000 7f428b2bd700 1 -- 172.21.15.16:0/4062561769 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4284158050 msgr2=0x7f42840cdf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:46.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.886+0000 7f428b2bd700 1 --2- 172.21.15.16:0/4062561769 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4284158050 0x7f42840cdf30 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f4274009820 tx=0x7f4274009b30 comp rx=0 tx=0).stop 2023-10-16T13:27:46.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.886+0000 7f428b2bd700 1 -- 172.21.15.16:0/4062561769 shutdown_connections 2023-10-16T13:27:46.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.886+0000 7f428b2bd700 1 --2- 172.21.15.16:0/4062561769 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4284158050 0x7f42840cdf30 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:46.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.886+0000 7f428b2bd700 1 -- 172.21.15.16:0/4062561769 >> 172.21.15.16:0/4062561769 conn(0x7f42840b1ed0 msgr2=0x7f42840b22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:46.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.886+0000 7f428b2bd700 1 -- 172.21.15.16:0/4062561769 shutdown_connections 2023-10-16T13:27:46.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.886+0000 7f428b2bd700 1 -- 172.21.15.16:0/4062561769 wait complete. 2023-10-16T13:27:46.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.888+0000 7f428b2bd700 1 Processor -- start 2023-10-16T13:27:46.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.888+0000 7f428b2bd700 1 -- start start 2023-10-16T13:27:46.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.888+0000 7f428b2bd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42840cc1d0 0x7f4284157510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:46.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.888+0000 7f428b2bd700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f42840ce470 0x7f42840c49d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:46.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.888+0000 7f428b2bd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4284158050 0x7f42840c5c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:46.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.888+0000 7f428b2bd700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42840c4f10 con 0x7f42840cc1d0 2023-10-16T13:27:46.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.888+0000 7f428b2bd700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42840c3f30 con 0x7f4284158050 2023-10-16T13:27:46.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.888+0000 7f428b2bd700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42840c6460 con 0x7f42840ce470 2023-10-16T13:27:46.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.889+0000 7f4289059700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42840cc1d0 0x7f4284157510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:46.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.889+0000 7f4289059700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42840cc1d0 0x7f4284157510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48540/0 (socket says 172.21.15.16:48540) 2023-10-16T13:27:46.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.889+0000 7f4289059700 1 -- 172.21.15.16:0/195014305 learned_addr learned my addr 172.21.15.16:0/195014305 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:46.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.889+0000 7f4288858700 1 --2- 172.21.15.16:0/195014305 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f42840ce470 0x7f42840c49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:46.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.889+0000 7f428985a700 1 --2- 172.21.15.16:0/195014305 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4284158050 0x7f42840c5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:46.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.889+0000 7f4289059700 1 -- 172.21.15.16:0/195014305 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f42840ce470 msgr2=0x7f42840c49d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:46.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.889+0000 7f4289059700 1 --2- 172.21.15.16:0/195014305 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f42840ce470 0x7f42840c49d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:46.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.889+0000 7f4289059700 1 -- 172.21.15.16:0/195014305 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4284158050 msgr2=0x7f42840c5c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:46.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.889+0000 7f4289059700 1 --2- 172.21.15.16:0/195014305 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4284158050 0x7f42840c5c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:46.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.889+0000 7f4289059700 1 -- 172.21.15.16:0/195014305 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4274009430 con 0x7f42840cc1d0 2023-10-16T13:27:46.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.889+0000 7f4289059700 1 --2- 172.21.15.16:0/195014305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42840cc1d0 0x7f4284157510 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7f4284073550 tx=0x7f4274004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:46.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.889+0000 7f4279ffb700 1 -- 172.21.15.16:0/195014305 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4274003c30 con 0x7f42840cc1d0 2023-10-16T13:27:46.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.889+0000 7f4279ffb700 1 -- 172.21.15.16:0/195014305 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4274003d90 con 0x7f42840cc1d0 2023-10-16T13:27:46.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.889+0000 7f428b2bd700 1 -- 172.21.15.16:0/195014305 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42840c41d0 con 0x7f42840cc1d0 2023-10-16T13:27:46.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.889+0000 7f4279ffb700 1 -- 172.21.15.16:0/195014305 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f427401d650 con 0x7f42840cc1d0 2023-10-16T13:27:46.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.889+0000 7f428b2bd700 1 -- 172.21.15.16:0/195014305 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42840c6740 con 0x7f42840cc1d0 2023-10-16T13:27:46.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.890+0000 7f428b2bd700 1 -- 172.21.15.16:0/195014305 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42840c46f0 con 0x7f42840cc1d0 2023-10-16T13:27:46.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.891+0000 7f4279ffb700 1 -- 172.21.15.16:0/195014305 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f427401d7b0 con 0x7f42840cc1d0 2023-10-16T13:27:46.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.891+0000 7f4279ffb700 1 --2- 172.21.15.16:0/195014305 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f426806b020 0x7f426806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:46.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.891+0000 7f4279ffb700 1 -- 172.21.15.16:0/195014305 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f427408ad40 con 0x7f42840cc1d0 2023-10-16T13:27:46.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.892+0000 7f4288858700 1 --2- 172.21.15.16:0/195014305 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f426806b020 0x7f426806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:46.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.892+0000 7f4288858700 1 --2- 172.21.15.16:0/195014305 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f426806b020 0x7f426806d4e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f428406b8c0 tx=0x7f4280001ce0 comp rx=0 tx=0).ready entity=mgr.9510 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:46.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:46.896+0000 7f4279ffb700 1 -- 172.21.15.16:0/195014305 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4274055a90 con 0x7f42840cc1d0 2023-10-16T13:27:47.122 INFO:tasks.ceph.mds.d.smithi049.stderr:2023-10-16T13:27:47.120+0000 7f7186f98700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T13:27:47.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.184+0000 7f428b2bd700 1 -- 172.21.15.16:0/195014305 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f428404ebb0 con 0x7f42840cc1d0 2023-10-16T13:27:47.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.186+0000 7f4279ffb700 1 -- 172.21.15.16:0/195014305 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v47) v1 ==== 190+0+147 (secure 0 0 0) 0x7f42740160e0 con 0x7f42840cc1d0 2023-10-16T13:27:47.189 INFO:teuthology.orchestra.run.smithi016.stdout:[client.0] 2023-10-16T13:27:47.190 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAPKC1l2l6PFxAA9QyXtBAHdGAXpwH0JfknuQ== 2023-10-16T13:27:47.190 INFO:teuthology.orchestra.run.smithi016.stdout: caps mds = "allow" 2023-10-16T13:27:47.190 INFO:teuthology.orchestra.run.smithi016.stdout: caps mon = "allow r" 2023-10-16T13:27:47.190 INFO:teuthology.orchestra.run.smithi016.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-16T13:27:47.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.188+0000 7f428b2bd700 1 -- 172.21.15.16:0/195014305 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f426806b020 msgr2=0x7f426806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:47.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.188+0000 7f428b2bd700 1 --2- 172.21.15.16:0/195014305 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f426806b020 0x7f426806d4e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f428406b8c0 tx=0x7f4280001ce0 comp rx=0 tx=0).stop 2023-10-16T13:27:47.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.188+0000 7f428b2bd700 1 -- 172.21.15.16:0/195014305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42840cc1d0 msgr2=0x7f4284157510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:47.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.188+0000 7f428b2bd700 1 --2- 172.21.15.16:0/195014305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42840cc1d0 0x7f4284157510 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7f4284073550 tx=0x7f4274004750 comp rx=0 tx=0).stop 2023-10-16T13:27:47.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.188+0000 7f428b2bd700 1 -- 172.21.15.16:0/195014305 shutdown_connections 2023-10-16T13:27:47.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.188+0000 7f428b2bd700 1 --2- 172.21.15.16:0/195014305 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f426806b020 0x7f426806d4e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:47.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.188+0000 7f428b2bd700 1 --2- 172.21.15.16:0/195014305 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4284158050 0x7f42840c5c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:47.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.188+0000 7f428b2bd700 1 --2- 172.21.15.16:0/195014305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42840cc1d0 0x7f4284157510 unknown :-1 s=CLOSED pgs=1934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:47.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.188+0000 7f428b2bd700 1 --2- 172.21.15.16:0/195014305 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f42840ce470 0x7f42840c49d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:47.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.188+0000 7f428b2bd700 1 -- 172.21.15.16:0/195014305 >> 172.21.15.16:0/195014305 conn(0x7f42840b1ed0 msgr2=0x7f4284159e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:47.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.188+0000 7f428b2bd700 1 -- 172.21.15.16:0/195014305 shutdown_connections 2023-10-16T13:27:47.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.188+0000 7f428b2bd700 1 -- 172.21.15.16:0/195014305 wait complete. 2023-10-16T13:27:47.194 INFO:teuthology.orchestra.run.smithi016.stderr:updated caps for client.0 2023-10-16T13:27:47.204 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:27:47.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.325+0000 7f1236753700 1 Processor -- start 2023-10-16T13:27:47.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.325+0000 7f1236753700 1 -- start start 2023-10-16T13:27:47.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.326+0000 7f1236753700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f12300d03b0 0x7f12300cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:47.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.326+0000 7f1236753700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12300cc2a0 con 0x7f123015b630 2023-10-16T13:27:47.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.326+0000 7f1236753700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12300cc540 con 0x7f12300ce4f0 2023-10-16T13:27:47.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.326+0000 7f1236753700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12300cdb20 con 0x7f12300d03b0 2023-10-16T13:27:47.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.326+0000 7f1234cf0700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f12300d03b0 0x7f12300cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:47.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.326+0000 7f122ffff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f123015b630 0x7f12300cdde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37822/0 (socket says 172.21.15.16:37822) 2023-10-16T13:27:47.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.326+0000 7f122ffff700 1 -- 172.21.15.16:0/1425665022 learned_addr learned my addr 172.21.15.16:0/1425665022 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:27:47.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.327+0000 7f1234cf0700 1 -- 172.21.15.16:0/1425665022 >> v1:172.21.15.49:6789/0 conn(0x7f12300ce4f0 legacy=0x7f12300cfca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:47.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.327+0000 7f1234cf0700 1 -- 172.21.15.16:0/1425665022 >> v1:172.21.15.16:6789/0 conn(0x7f123015b630 legacy=0x7f12300cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:47.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.327+0000 7f1234cf0700 1 -- 172.21.15.16:0/1425665022 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12300cc820 con 0x7f12300d03b0 2023-10-16T13:27:47.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.327+0000 7f122effd700 1 -- 172.21.15.16:0/1425665022 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 862739104 0 0) 0x7f12300cc2a0 con 0x7f123015b630 2023-10-16T13:27:47.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.327+0000 7f1234cf0700 1 --2- 172.21.15.16:0/1425665022 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12300d03b0 0x7f12300cd5e0 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7f1228009d90 tx=0x7f122800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63c6e6311f4b4bc1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:47.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.327+0000 7f122effd700 1 -- 172.21.15.16:0/1425665022 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f122800e070 con 0x7f12300d03b0 2023-10-16T13:27:47.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.327+0000 7f122effd700 1 -- 172.21.15.16:0/1425665022 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f122800bbd0 con 0x7f12300d03b0 2023-10-16T13:27:47.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.327+0000 7f122effd700 1 -- 172.21.15.16:0/1425665022 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1228012430 con 0x7f12300d03b0 2023-10-16T13:27:47.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.327+0000 7f1236753700 1 -- 172.21.15.16:0/1425665022 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12300d03b0 msgr2=0x7f12300cd5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:47.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.327+0000 7f1236753700 1 --2- 172.21.15.16:0/1425665022 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12300d03b0 0x7f12300cd5e0 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7f1228009d90 tx=0x7f122800b330 comp rx=0 tx=0).stop 2023-10-16T13:27:47.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.328+0000 7f1236753700 1 -- 172.21.15.16:0/1425665022 shutdown_connections 2023-10-16T13:27:47.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.328+0000 7f1236753700 1 --2- 172.21.15.16:0/1425665022 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12300d03b0 0x7f12300cd5e0 unknown :-1 s=CLOSED pgs=1936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:47.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.328+0000 7f1236753700 1 -- 172.21.15.16:0/1425665022 >> 172.21.15.16:0/1425665022 conn(0x7f12300d14b0 msgr2=0x7f12300d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:47.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.328+0000 7f1236753700 1 -- 172.21.15.16:0/1425665022 shutdown_connections 2023-10-16T13:27:47.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.328+0000 7f1236753700 1 -- 172.21.15.16:0/1425665022 wait complete. 2023-10-16T13:27:47.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.329+0000 7f1236753700 1 Processor -- start 2023-10-16T13:27:47.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.329+0000 7f1236753700 1 -- start start 2023-10-16T13:27:47.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.330+0000 7f1236753700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12300ce4f0 0x7f1230151f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:47.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.330+0000 7f1236753700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f12300d03b0 0x7f123006cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:47.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.330+0000 7f1236753700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f123015b630 0x7f12300b44e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:47.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.330+0000 7f1236753700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1230151280 con 0x7f12300ce4f0 2023-10-16T13:27:47.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.330+0000 7f1236753700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1230152440 con 0x7f123015b630 2023-10-16T13:27:47.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.330+0000 7f1236753700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f123006d540 con 0x7f12300d03b0 2023-10-16T13:27:47.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.330+0000 7f122ffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12300ce4f0 0x7f1230151f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:47.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.330+0000 7f122ffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12300ce4f0 0x7f1230151f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48554/0 (socket says 172.21.15.16:48554) 2023-10-16T13:27:47.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.330+0000 7f122ffff700 1 -- 172.21.15.16:0/1148675729 learned_addr learned my addr 172.21.15.16:0/1148675729 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:47.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.330+0000 7f122f7fe700 1 --2- 172.21.15.16:0/1148675729 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f12300d03b0 0x7f123006cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:47.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.330+0000 7f1234cf0700 1 --2- 172.21.15.16:0/1148675729 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f123015b630 0x7f12300b44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:47.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.330+0000 7f122ffff700 1 -- 172.21.15.16:0/1148675729 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f12300d03b0 msgr2=0x7f123006cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:47.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.330+0000 7f122ffff700 1 --2- 172.21.15.16:0/1148675729 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f12300d03b0 0x7f123006cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:47.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.330+0000 7f122ffff700 1 -- 172.21.15.16:0/1148675729 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f123015b630 msgr2=0x7f12300b44e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:47.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.330+0000 7f122ffff700 1 --2- 172.21.15.16:0/1148675729 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f123015b630 0x7f12300b44e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:47.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.330+0000 7f122ffff700 1 -- 172.21.15.16:0/1148675729 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1228009a70 con 0x7f12300ce4f0 2023-10-16T13:27:47.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.330+0000 7f122ffff700 1 --2- 172.21.15.16:0/1148675729 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12300ce4f0 0x7f1230151f00 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7f122000c470 tx=0x7f122000c780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:47.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.331+0000 7f122cff9700 1 -- 172.21.15.16:0/1148675729 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12200074e0 con 0x7f12300ce4f0 2023-10-16T13:27:47.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.331+0000 7f122cff9700 1 -- 172.21.15.16:0/1148675729 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1220007640 con 0x7f12300ce4f0 2023-10-16T13:27:47.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.331+0000 7f1236753700 1 -- 172.21.15.16:0/1148675729 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f123006d820 con 0x7f12300ce4f0 2023-10-16T13:27:47.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.331+0000 7f122cff9700 1 -- 172.21.15.16:0/1148675729 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1220005a70 con 0x7f12300ce4f0 2023-10-16T13:27:47.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.331+0000 7f1236753700 1 -- 172.21.15.16:0/1148675729 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1230151700 con 0x7f12300ce4f0 2023-10-16T13:27:47.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.332+0000 7f1236753700 1 -- 172.21.15.16:0/1148675729 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f123004ebb0 con 0x7f12300ce4f0 2023-10-16T13:27:47.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.332+0000 7f122cff9700 1 -- 172.21.15.16:0/1148675729 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1220005bd0 con 0x7f12300ce4f0 2023-10-16T13:27:47.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.333+0000 7f122cff9700 1 --2- 172.21.15.16:0/1148675729 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f121006b020 0x7f121006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:47.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.333+0000 7f122cff9700 1 -- 172.21.15.16:0/1148675729 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f1220088c10 con 0x7f12300ce4f0 2023-10-16T13:27:47.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.333+0000 7f122f7fe700 1 --2- 172.21.15.16:0/1148675729 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f121006b020 0x7f121006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:47.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.333+0000 7f122f7fe700 1 --2- 172.21.15.16:0/1148675729 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f121006b020 0x7f121006d4e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1230154920 tx=0x7f1218009040 comp rx=0 tx=0).ready entity=mgr.9510 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:47.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.337+0000 7f122cff9700 1 -- 172.21.15.16:0/1148675729 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1220053a00 con 0x7f12300ce4f0 2023-10-16T13:27:47.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.585+0000 7f1236753700 1 -- 172.21.15.16:0/1148675729 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f123006b5d0 con 0x7f12300ce4f0 2023-10-16T13:27:47.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.585+0000 7f122cff9700 1 -- 172.21.15.16:0/1148675729 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+4917 (secure 0 0 0) 0x7f1220017020 con 0x7f12300ce4f0 2023-10-16T13:27:47.589 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:27:47.589 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":86,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7964,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/4012137577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":4012137577},{"type":"v1","addr":"172.21.15.16:6837","nonce":4012137577}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81},{"gid":7970,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/3742568047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":3742568047},{"type":"v1","addr":"172.21.15.49:6835","nonce":3742568047}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81},{"gid":9756,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/3838743980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":3838743980},{"type":"v1","addr":"172.21.15.16:6836","nonce":3838743980}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:27:41.561818+0000","modified":"2023-10-16T13:27:44.572793+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9774},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9774":{"gid":9774,"name":"d","rank":0,"incarnation":82,"state":"up:active","state_seq":3,"addr":"172.21.15.49:6834/2218212001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":2218212001},{"type":"v1","addr":"172.21.15.49:6834","nonce":2218212001}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2023-10-16T13:27:47.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.587+0000 7f1236753700 1 -- 172.21.15.16:0/1148675729 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f121006b020 msgr2=0x7f121006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:47.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.587+0000 7f1236753700 1 --2- 172.21.15.16:0/1148675729 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f121006b020 0x7f121006d4e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1230154920 tx=0x7f1218009040 comp rx=0 tx=0).stop 2023-10-16T13:27:47.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.587+0000 7f1236753700 1 -- 172.21.15.16:0/1148675729 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12300ce4f0 msgr2=0x7f1230151f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:47.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.587+0000 7f1236753700 1 --2- 172.21.15.16:0/1148675729 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12300ce4f0 0x7f1230151f00 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7f122000c470 tx=0x7f122000c780 comp rx=0 tx=0).stop 2023-10-16T13:27:47.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.587+0000 7f1236753700 1 -- 172.21.15.16:0/1148675729 shutdown_connections 2023-10-16T13:27:47.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.587+0000 7f1236753700 1 --2- 172.21.15.16:0/1148675729 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f121006b020 0x7f121006d4e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:47.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.587+0000 7f1236753700 1 --2- 172.21.15.16:0/1148675729 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f123015b630 0x7f12300b44e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:47.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.587+0000 7f1236753700 1 --2- 172.21.15.16:0/1148675729 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12300ce4f0 0x7f1230151f00 unknown :-1 s=CLOSED pgs=1937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:47.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.587+0000 7f1236753700 1 --2- 172.21.15.16:0/1148675729 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f12300d03b0 0x7f123006cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:47.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.587+0000 7f1236753700 1 -- 172.21.15.16:0/1148675729 >> 172.21.15.16:0/1148675729 conn(0x7f12300d14b0 msgr2=0x7f12300b5340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:47.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.587+0000 7f1236753700 1 -- 172.21.15.16:0/1148675729 shutdown_connections 2023-10-16T13:27:47.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:47.587+0000 7f1236753700 1 -- 172.21.15.16:0/1148675729 wait complete. 2023-10-16T13:27:47.594 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 86 2023-10-16T13:27:48.005 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 86, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T13:27:41.561818+0000', 'modified': '2023-10-16T13:27:44.572793+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9774}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9774': {'gid': 9774, 'name': 'd', 'rank': 0, 'incarnation': 82, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.49:6834/2218212001', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6832', 'nonce': 2218212001}, {'type': 'v1', 'addr': '172.21.15.49:6834', 'nonce': 2218212001}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-16T13:27:48.006 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-16T13:27:48.007 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2023-10-16T13:27:48.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.135+0000 7f0f9f9a5700 1 Processor -- start 2023-10-16T13:27:48.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.135+0000 7f0f9f9a5700 1 -- start start 2023-10-16T13:27:48.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.136+0000 7f0f9f9a5700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0f980cf4a0 0x7f0f980726e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:48.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.136+0000 7f0f9f9a5700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f0f980cf880 0x7f0f980b52f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:48.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.136+0000 7f0f9f9a5700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f9815bc80 con 0x7f0f980b5830 2023-10-16T13:27:48.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.136+0000 7f0f9f9a5700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f98072c20 con 0x7f0f980cf4a0 2023-10-16T13:27:48.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.136+0000 7f0f9f9a5700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f9815b070 con 0x7f0f980cf880 2023-10-16T13:27:48.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.136+0000 7f0f9d741700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0f980cf4a0 0x7f0f980726e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:48.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.136+0000 7f0f9d741700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0f980cf4a0 0x7f0f980726e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48562/0 (socket says 172.21.15.16:48562) 2023-10-16T13:27:48.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.136+0000 7f0f9d741700 1 -- 172.21.15.16:0/867723778 learned_addr learned my addr 172.21.15.16:0/867723778 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:48.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.136+0000 7f0f9cf40700 1 --2- 172.21.15.16:0/867723778 >> v2:172.21.15.49:3300/0 conn(0x7f0f980cf880 0x7f0f980b52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:48.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.137+0000 7f0f9d741700 1 -- 172.21.15.16:0/867723778 >> v2:172.21.15.49:3300/0 conn(0x7f0f980cf880 msgr2=0x7f0f980b52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:48.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.137+0000 7f0f9d741700 1 --2- 172.21.15.16:0/867723778 >> v2:172.21.15.49:3300/0 conn(0x7f0f980cf880 0x7f0f980b52f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:48.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.137+0000 7f0f9d741700 1 -- 172.21.15.16:0/867723778 >> v1:172.21.15.49:6789/0 conn(0x7f0f980b5830 legacy=0x7f0f9815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:48.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.137+0000 7f0f9d741700 1 -- 172.21.15.16:0/867723778 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f9815b350 con 0x7f0f980cf4a0 2023-10-16T13:27:48.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.137+0000 7f0f9d741700 1 --2- 172.21.15.16:0/867723778 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f980cf4a0 0x7f0f980726e0 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7f0f98071da0 tx=0x7f0f88009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc374095ce088dad server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:48.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.137+0000 7f0f8ffff700 1 -- 172.21.15.16:0/867723778 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f88011830 con 0x7f0f980cf4a0 2023-10-16T13:27:48.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.137+0000 7f0f8ffff700 1 -- 172.21.15.16:0/867723778 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f88011990 con 0x7f0f980cf4a0 2023-10-16T13:27:48.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.137+0000 7f0f8ffff700 1 -- 172.21.15.16:0/867723778 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f88011d20 con 0x7f0f980cf4a0 2023-10-16T13:27:48.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.137+0000 7f0f9f9a5700 1 -- 172.21.15.16:0/867723778 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f980cf4a0 msgr2=0x7f0f980726e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:48.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.137+0000 7f0f9f9a5700 1 --2- 172.21.15.16:0/867723778 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f980cf4a0 0x7f0f980726e0 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7f0f98071da0 tx=0x7f0f88009bd0 comp rx=0 tx=0).stop 2023-10-16T13:27:48.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.137+0000 7f0f9f9a5700 1 -- 172.21.15.16:0/867723778 shutdown_connections 2023-10-16T13:27:48.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.137+0000 7f0f9f9a5700 1 --2- 172.21.15.16:0/867723778 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f980cf4a0 0x7f0f980726e0 unknown :-1 s=CLOSED pgs=1938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:48.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.137+0000 7f0f9f9a5700 1 --2- 172.21.15.16:0/867723778 >> v2:172.21.15.49:3300/0 conn(0x7f0f980cf880 0x7f0f980b52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:48.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.137+0000 7f0f9f9a5700 1 -- 172.21.15.16:0/867723778 >> 172.21.15.16:0/867723778 conn(0x7f0f980b1fd0 msgr2=0x7f0f980b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:48.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.138+0000 7f0f9f9a5700 1 -- 172.21.15.16:0/867723778 shutdown_connections 2023-10-16T13:27:48.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.138+0000 7f0f9f9a5700 1 -- 172.21.15.16:0/867723778 wait complete. 2023-10-16T13:27:48.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.139+0000 7f0f9f9a5700 1 Processor -- start 2023-10-16T13:27:48.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.139+0000 7f0f9f9a5700 1 -- start start 2023-10-16T13:27:48.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.139+0000 7f0f9f9a5700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f980b5830 0x7f0f98154770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:48.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.139+0000 7f0f9f9a5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f980cf880 0x7f0f980cceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:48.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.140+0000 7f0f9f9a5700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f980c7d30 0x7f0f9806a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:48.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.140+0000 7f0f9f9a5700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f980c4d90 con 0x7f0f980cf880 2023-10-16T13:27:48.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.140+0000 7f0f9f9a5700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f980c4110 con 0x7f0f980c7d30 2023-10-16T13:27:48.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.140+0000 7f0f9f9a5700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f980c43f0 con 0x7f0f980b5830 2023-10-16T13:27:48.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.140+0000 7f0f9cf40700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f980cf880 0x7f0f980cceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:48.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.140+0000 7f0f9cf40700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f980cf880 0x7f0f980cceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48570/0 (socket says 172.21.15.16:48570) 2023-10-16T13:27:48.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.140+0000 7f0f9cf40700 1 -- 172.21.15.16:0/2210001833 learned_addr learned my addr 172.21.15.16:0/2210001833 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:48.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.140+0000 7f0f9d741700 1 --2- 172.21.15.16:0/2210001833 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f980b5830 0x7f0f98154770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:48.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.140+0000 7f0f9df42700 1 --2- 172.21.15.16:0/2210001833 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f980c7d30 0x7f0f9806a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:48.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.140+0000 7f0f9cf40700 1 -- 172.21.15.16:0/2210001833 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f980b5830 msgr2=0x7f0f98154770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:48.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.140+0000 7f0f9cf40700 1 --2- 172.21.15.16:0/2210001833 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f980b5830 0x7f0f98154770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:48.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.140+0000 7f0f9cf40700 1 -- 172.21.15.16:0/2210001833 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f980c7d30 msgr2=0x7f0f9806a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:48.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.140+0000 7f0f9cf40700 1 --2- 172.21.15.16:0/2210001833 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f980c7d30 0x7f0f9806a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:48.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.140+0000 7f0f9cf40700 1 -- 172.21.15.16:0/2210001833 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f88009550 con 0x7f0f980cf880 2023-10-16T13:27:48.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.140+0000 7f0f9cf40700 1 --2- 172.21.15.16:0/2210001833 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f980cf880 0x7f0f980cceb0 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7f0f9400e9b0 tx=0x7f0f9400ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:48.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.140+0000 7f0f8dffb700 1 -- 172.21.15.16:0/2210001833 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f94009710 con 0x7f0f980cf880 2023-10-16T13:27:48.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.140+0000 7f0f8dffb700 1 -- 172.21.15.16:0/2210001833 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f9400c850 con 0x7f0f980cf880 2023-10-16T13:27:48.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.140+0000 7f0f9f9a5700 1 -- 172.21.15.16:0/2210001833 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f980c46d0 con 0x7f0f980cf880 2023-10-16T13:27:48.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.140+0000 7f0f8dffb700 1 -- 172.21.15.16:0/2210001833 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f9400cbc0 con 0x7f0f980cf880 2023-10-16T13:27:48.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.141+0000 7f0f9f9a5700 1 -- 172.21.15.16:0/2210001833 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f98154cb0 con 0x7f0f980cf880 2023-10-16T13:27:48.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.142+0000 7f0f8dffb700 1 -- 172.21.15.16:0/2210001833 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0f9400cd20 con 0x7f0f980cf880 2023-10-16T13:27:48.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.142+0000 7f0f8dffb700 1 --2- 172.21.15.16:0/2210001833 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f0f7c06b020 0x7f0f7c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:48.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.142+0000 7f0f8dffb700 1 -- 172.21.15.16:0/2210001833 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f0f94089a20 con 0x7f0f980cf880 2023-10-16T13:27:48.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.142+0000 7f0f9d741700 1 --2- 172.21.15.16:0/2210001833 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f0f7c06b020 0x7f0f7c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:48.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.143+0000 7f0f9d741700 1 --2- 172.21.15.16:0/2210001833 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f0f7c06b020 0x7f0f7c06d4e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0f9806f300 tx=0x7f0f88008030 comp rx=0 tx=0).ready entity=mgr.9510 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:48.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.144+0000 7f0fabb8fb80 1 Processor -- start 2023-10-16T13:27:48.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.144+0000 7f0fabb8fb80 1 -- start start 2023-10-16T13:27:48.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.145+0000 7f0fabb8fb80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d1ba862220 0x55d1ba869a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:48.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.145+0000 7f0fabb8fb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d1ba867630 0x55d1ba869fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:48.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.145+0000 7f0fabb8fb80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55d1ba83d010 0x55d1ba83d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:48.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.145+0000 7f0fabb8fb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1ba7b3c20 con 0x55d1ba867630 2023-10-16T13:27:48.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.145+0000 7f0fabb8fb80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1ba798c10 con 0x55d1ba862220 2023-10-16T13:27:48.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.145+0000 7f0fabb8fb80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d1ba796530 con 0x55d1ba83d010 2023-10-16T13:27:48.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.145+0000 7f0f9cf40700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d1ba867630 0x55d1ba869fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:48.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.145+0000 7f0f9cf40700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d1ba867630 0x55d1ba869fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48578/0 (socket says 172.21.15.16:48578) 2023-10-16T13:27:48.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.145+0000 7f0f9cf40700 1 -- 172.21.15.16:0/1832010899 learned_addr learned my addr 172.21.15.16:0/1832010899 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:48.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.145+0000 7f0f9d741700 1 --2- 172.21.15.16:0/1832010899 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d1ba862220 0x55d1ba869a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:48.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.145+0000 7f0f9df42700 1 --2- 172.21.15.16:0/1832010899 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55d1ba83d010 0x55d1ba83d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:48.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.145+0000 7f0f9cf40700 1 -- 172.21.15.16:0/1832010899 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55d1ba83d010 msgr2=0x55d1ba83d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:48.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.145+0000 7f0f9cf40700 1 --2- 172.21.15.16:0/1832010899 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55d1ba83d010 0x55d1ba83d6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:48.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.145+0000 7f0f9cf40700 1 -- 172.21.15.16:0/1832010899 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d1ba862220 msgr2=0x55d1ba869a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:48.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.145+0000 7f0f9cf40700 1 --2- 172.21.15.16:0/1832010899 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d1ba862220 0x55d1ba869a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:48.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.145+0000 7f0f9cf40700 1 -- 172.21.15.16:0/1832010899 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d1ba74f3a0 con 0x55d1ba867630 2023-10-16T13:27:48.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.145+0000 7f0f9df42700 1 --2- 172.21.15.16:0/1832010899 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55d1ba83d010 0x55d1ba83d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:27:48.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.146+0000 7f0f9cf40700 1 --2- 172.21.15.16:0/1832010899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d1ba867630 0x55d1ba869fb0 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7f0f9400c430 tx=0x7f0f940584d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98edb3c857b7f9de server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:48.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.146+0000 7f0f82ffd700 1 -- 172.21.15.16:0/1832010899 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f940607d0 con 0x55d1ba867630 2023-10-16T13:27:48.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.146+0000 7f0f82ffd700 1 -- 172.21.15.16:0/1832010899 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f94054be0 con 0x55d1ba867630 2023-10-16T13:27:48.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.146+0000 7f0f82ffd700 1 -- 172.21.15.16:0/1832010899 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f94058930 con 0x55d1ba867630 2023-10-16T13:27:48.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.146+0000 7f0fabb8fb80 1 -- 172.21.15.16:0/1832010899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d1ba867630 msgr2=0x55d1ba869fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:48.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.146+0000 7f0fabb8fb80 1 --2- 172.21.15.16:0/1832010899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d1ba867630 0x55d1ba869fb0 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7f0f9400c430 tx=0x7f0f940584d0 comp rx=0 tx=0).stop 2023-10-16T13:27:48.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.146+0000 7f0fabb8fb80 1 -- 172.21.15.16:0/1832010899 shutdown_connections 2023-10-16T13:27:48.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.146+0000 7f0fabb8fb80 1 --2- 172.21.15.16:0/1832010899 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55d1ba83d010 0x55d1ba83d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:48.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.146+0000 7f0fabb8fb80 1 --2- 172.21.15.16:0/1832010899 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d1ba862220 0x55d1ba869a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:48.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.146+0000 7f0fabb8fb80 1 --2- 172.21.15.16:0/1832010899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d1ba867630 0x55d1ba869fb0 unknown :-1 s=CLOSED pgs=1940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:48.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.146+0000 7f0fabb8fb80 1 -- 172.21.15.16:0/1832010899 >> 172.21.15.16:0/1832010899 conn(0x55d1ba76cd80 msgr2=0x55d1ba887ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:48.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.146+0000 7f0fabb8fb80 1 -- 172.21.15.16:0/1832010899 shutdown_connections 2023-10-16T13:27:48.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.147+0000 7f0fabb8fb80 1 -- 172.21.15.16:0/1832010899 wait complete. 2023-10-16T13:27:48.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.147+0000 7f0fabb8fb80 1 -- start start 2023-10-16T13:27:48.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.147+0000 7f0fabb8fb80 1 Processor -- start 2023-10-16T13:27:48.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.148+0000 7f0fabb8fb80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55d1ba83d010 0x55d1ba83df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:48.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.148+0000 7f0fabb8fb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d1ba862220 0x55d1ba894ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:48.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.148+0000 7f0fabb8fb80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d1ba867630 0x55d1ba865d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:48.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.148+0000 7f0fabb8fb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1ba791e20 con 0x55d1ba862220 2023-10-16T13:27:48.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.148+0000 7f0fabb8fb80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1ba72e6c0 con 0x55d1ba867630 2023-10-16T13:27:48.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.148+0000 7f0fabb8fb80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d1ba72e360 con 0x55d1ba83d010 2023-10-16T13:27:48.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.148+0000 7f0fabb8fb80 10 client.0 _ll_register_callbacks cb 0x55d1ba58b7b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T13:27:48.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.148+0000 7f0f9cf40700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d1ba862220 0x55d1ba894ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:48.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.148+0000 7f0f9cf40700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d1ba862220 0x55d1ba894ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48584/0 (socket says 172.21.15.16:48584) 2023-10-16T13:27:48.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.148+0000 7f0f9cf40700 1 -- 172.21.15.16:0/2249820385 learned_addr learned my addr 172.21.15.16:0/2249820385 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:48.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.148+0000 7f0f9d741700 1 --2- 172.21.15.16:0/2249820385 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55d1ba83d010 0x55d1ba83df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:48.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.148+0000 7f0f9df42700 1 --2- 172.21.15.16:0/2249820385 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d1ba867630 0x55d1ba865d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:48.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.148+0000 7f0f9cf40700 1 -- 172.21.15.16:0/2249820385 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55d1ba83d010 msgr2=0x55d1ba83df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:48.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.148+0000 7f0f9cf40700 1 --2- 172.21.15.16:0/2249820385 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55d1ba83d010 0x55d1ba83df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:48.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.148+0000 7f0f9cf40700 1 -- 172.21.15.16:0/2249820385 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d1ba867630 msgr2=0x55d1ba865d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:48.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.148+0000 7f0f9cf40700 1 --2- 172.21.15.16:0/2249820385 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d1ba867630 0x55d1ba865d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:48.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.148+0000 7f0f9cf40700 1 -- 172.21.15.16:0/2249820385 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f98154cb0 con 0x55d1ba862220 2023-10-16T13:27:48.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.148+0000 7f0f9df42700 1 --2- 172.21.15.16:0/2249820385 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d1ba867630 0x55d1ba865d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:27:48.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.149+0000 7f0f9cf40700 1 --2- 172.21.15.16:0/2249820385 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d1ba862220 0x55d1ba894ee0 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7f0f940549d0 tx=0x7f0f94021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ef37363466e166 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:48.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.149+0000 7f0f77fff700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2023-10-16T13:27:48.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.149+0000 7f0f77fff700 1 -- 172.21.15.16:0/2249820385 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f94073030 con 0x55d1ba862220 2023-10-16T13:27:48.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.149+0000 7f0f77fff700 1 -- 172.21.15.16:0/2249820385 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(177..177 src has 1..177) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f0f94074070 con 0x55d1ba862220 2023-10-16T13:27:48.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.149+0000 7f0fabb8fb80 1 -- 172.21.15.16:0/2249820385 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d1ba8325b0 con 0x55d1ba862220 2023-10-16T13:27:48.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.149+0000 7f0f77fff700 1 -- 172.21.15.16:0/2249820385 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0f940607d0 con 0x55d1ba862220 2023-10-16T13:27:48.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.149+0000 7f0fabb8fb80 10 client.9912 fetch_fsmap learned FSMap version 86 2023-10-16T13:27:48.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.149+0000 7f0fabb8fb80 1 -- 172.21.15.16:0/2249820385 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=86,monmap=2+}) v3 -- 0x55d1ba5bcd90 con 0x55d1ba862220 2023-10-16T13:27:48.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.149+0000 7f0f77fff700 1 -- 172.21.15.16:0/2249820385 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55d1ba8325b0 con 0x55d1ba862220 2023-10-16T13:27:48.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.149+0000 7f0f77fff700 1 -- 172.21.15.16:0/2249820385 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 86) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f0f94060340 con 0x55d1ba862220 2023-10-16T13:27:48.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.149+0000 7f0fabb8fb80 10 client.9912 fetch_fsmap finished waiting for FSMap version 86 2023-10-16T13:27:48.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.149+0000 7f0fabb8fb80 10 client.9912 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-16T13:27:48.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.149+0000 7f0fabb8fb80 20 client.9912 populate_metadata read hostname 'smithi016' 2023-10-16T13:27:48.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.149+0000 7f0fabb8fb80 1 --2- 172.21.15.16:0/2249820385 >> [v2:172.21.15.49:6832/2218212001,v1:172.21.15.49:6834/2218212001] conn(0x55d1ba895420 0x55d1ba861020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:48.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.149+0000 7f0fabb8fb80 4 client.9912 mds_command: new command op to 9774 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T13:27:48.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.149+0000 7f0fabb8fb80 1 -- 172.21.15.16:0/2249820385 --> [v2:172.21.15.49:6832/2218212001,v1:172.21.15.49:6834/2218212001] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d1ba78ae00 con 0x55d1ba895420 2023-10-16T13:27:48.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.149+0000 7f0f9d741700 1 --2- 172.21.15.16:0/2249820385 >> [v2:172.21.15.49:6832/2218212001,v1:172.21.15.49:6834/2218212001] conn(0x55d1ba895420 0x55d1ba861020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:48.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.150+0000 7f0f77fff700 10 client.9912 ms_handle_connect on v2:172.21.15.49:6832/2218212001 2023-10-16T13:27:48.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.150+0000 7f0f9d741700 1 --2- 172.21.15.16:0/2249820385 >> [v2:172.21.15.49:6832/2218212001,v1:172.21.15.49:6834/2218212001] conn(0x55d1ba895420 0x55d1ba861020 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1bb2e1a38b2c41d3 server_cookie=6cd9ee03fbcc910e in_seq=0 out_seq=0 2023-10-16T13:27:48.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.151+0000 7f0f77fff700 1 -- 172.21.15.16:0/2249820385 <== mds.0 v2:172.21.15.49:6832/2218212001 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55d1ba78ae00 con 0x55d1ba895420 2023-10-16T13:27:48.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.151+0000 7f0f77fff700 10 client.9912 handle_command_reply: tid=0 2023-10-16T13:27:48.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.151+0000 7f0fabb8fb80 1 client.9912 shutdown 2023-10-16T13:27:48.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.151+0000 7f0fabb8fb80 1 -- 172.21.15.16:0/2249820385 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d1ba862220 msgr2=0x55d1ba894ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:48.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.151+0000 7f0fabb8fb80 1 --2- 172.21.15.16:0/2249820385 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d1ba862220 0x55d1ba894ee0 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7f0f940549d0 tx=0x7f0f94021000 comp rx=0 tx=0).stop 2023-10-16T13:27:48.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.151+0000 7f0fabb8fb80 1 -- 172.21.15.16:0/2249820385 shutdown_connections 2023-10-16T13:27:48.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.151+0000 7f0fabb8fb80 1 --2- 172.21.15.16:0/2249820385 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d1ba867630 0x55d1ba865d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:48.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.151+0000 7f0fabb8fb80 1 --2- 172.21.15.16:0/2249820385 >> [v2:172.21.15.49:6832/2218212001,v1:172.21.15.49:6834/2218212001] conn(0x55d1ba895420 0x55d1ba861020 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:48.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.151+0000 7f0fabb8fb80 1 --2- 172.21.15.16:0/2249820385 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55d1ba83d010 0x55d1ba83df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:48.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.151+0000 7f0fabb8fb80 1 --2- 172.21.15.16:0/2249820385 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d1ba862220 0x55d1ba894ee0 unknown :-1 s=CLOSED pgs=1941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:48.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.151+0000 7f0fabb8fb80 1 -- 172.21.15.16:0/2249820385 >> 172.21.15.16:0/2249820385 conn(0x55d1ba76cd80 msgr2=0x55d1ba83e9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:48.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.151+0000 7f0f77fff700 0 client.9912 ms_handle_reset on v2:172.21.15.49:6832/2218212001 2023-10-16T13:27:48.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.151+0000 7f0fabb8fb80 1 -- 172.21.15.16:0/2249820385 shutdown_connections 2023-10-16T13:27:48.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.151+0000 7f0fabb8fb80 1 -- 172.21.15.16:0/2249820385 wait complete. 2023-10-16T13:27:48.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.151+0000 7f0fabb8fb80 20 client.9912 trim_cache size 0 max 16384 2023-10-16T13:27:48.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.169+0000 7f0fabb8fb80 1 Processor -- start 2023-10-16T13:27:48.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.169+0000 7f0fabb8fb80 1 -- start start 2023-10-16T13:27:48.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.169+0000 7f0fabb8fb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d1ba83d010 0x55d1ba83a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:48.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.169+0000 7f0fabb8fb80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d1ba862220 0x55d1ba89da40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:48.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.169+0000 7f0fabb8fb80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55d1ba867630 0x55d1ba866910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:48.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.169+0000 7f0fabb8fb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1ba6128d0 con 0x55d1ba83d010 2023-10-16T13:27:48.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.169+0000 7f0fabb8fb80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1ba864c00 con 0x55d1ba862220 2023-10-16T13:27:48.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.169+0000 7f0fabb8fb80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d1ba840b30 con 0x55d1ba867630 2023-10-16T13:27:48.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.170+0000 7f0f9d741700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d1ba83d010 0x55d1ba83a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:48.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.170+0000 7f0f9d741700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d1ba83d010 0x55d1ba83a1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48590/0 (socket says 172.21.15.16:48590) 2023-10-16T13:27:48.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.170+0000 7f0f9d741700 1 -- 172.21.15.16:0/1287720206 learned_addr learned my addr 172.21.15.16:0/1287720206 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:48.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.170+0000 7f0f9df42700 1 --2- 172.21.15.16:0/1287720206 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55d1ba867630 0x55d1ba866910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:48.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.170+0000 7f0f9cf40700 1 --2- 172.21.15.16:0/1287720206 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d1ba862220 0x55d1ba89da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:48.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.170+0000 7f0f9d741700 1 -- 172.21.15.16:0/1287720206 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55d1ba867630 msgr2=0x55d1ba866910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:48.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.170+0000 7f0f9d741700 1 --2- 172.21.15.16:0/1287720206 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55d1ba867630 0x55d1ba866910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:48.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.170+0000 7f0f9d741700 1 -- 172.21.15.16:0/1287720206 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d1ba862220 msgr2=0x55d1ba89da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:48.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.170+0000 7f0f9d741700 1 --2- 172.21.15.16:0/1287720206 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d1ba862220 0x55d1ba89da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:48.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.170+0000 7f0f9d741700 1 -- 172.21.15.16:0/1287720206 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d1ba5a5c10 con 0x55d1ba83d010 2023-10-16T13:27:48.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.170+0000 7f0f9cf40700 1 --2- 172.21.15.16:0/1287720206 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d1ba862220 0x55d1ba89da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:27:48.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.170+0000 7f0f9d741700 1 --2- 172.21.15.16:0/1287720206 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d1ba83d010 0x55d1ba83a1e0 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7f0f88009520 tx=0x7f0f88010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47ec648141c460a9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:48.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.170+0000 7f0f837fe700 1 -- 172.21.15.16:0/1287720206 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f8802c040 con 0x55d1ba83d010 2023-10-16T13:27:48.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.170+0000 7f0f837fe700 1 -- 172.21.15.16:0/1287720206 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f88032970 con 0x55d1ba83d010 2023-10-16T13:27:48.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.170+0000 7f0f837fe700 1 -- 172.21.15.16:0/1287720206 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f88032ce0 con 0x55d1ba83d010 2023-10-16T13:27:48.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.171+0000 7f0fabb8fb80 1 -- 172.21.15.16:0/1287720206 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d1ba83d010 msgr2=0x55d1ba83a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:48.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.171+0000 7f0fabb8fb80 1 --2- 172.21.15.16:0/1287720206 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d1ba83d010 0x55d1ba83a1e0 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7f0f88009520 tx=0x7f0f88010000 comp rx=0 tx=0).stop 2023-10-16T13:27:48.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.171+0000 7f0fabb8fb80 1 -- 172.21.15.16:0/1287720206 shutdown_connections 2023-10-16T13:27:48.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.171+0000 7f0fabb8fb80 1 --2- 172.21.15.16:0/1287720206 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55d1ba867630 0x55d1ba866910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:48.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.171+0000 7f0fabb8fb80 1 --2- 172.21.15.16:0/1287720206 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d1ba83d010 0x55d1ba83a1e0 unknown :-1 s=CLOSED pgs=1942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:48.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.171+0000 7f0fabb8fb80 1 --2- 172.21.15.16:0/1287720206 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d1ba862220 0x55d1ba89da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:48.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.171+0000 7f0fabb8fb80 1 -- 172.21.15.16:0/1287720206 >> 172.21.15.16:0/1287720206 conn(0x55d1ba76cd80 msgr2=0x55d1ba83dd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:48.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.171+0000 7f0fabb8fb80 1 -- 172.21.15.16:0/1287720206 shutdown_connections 2023-10-16T13:27:48.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.171+0000 7f0fabb8fb80 1 -- 172.21.15.16:0/1287720206 wait complete. 2023-10-16T13:27:48.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.171+0000 7f0fabb8fb80 1 -- start start 2023-10-16T13:27:48.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.171+0000 7f0fabb8fb80 1 Processor -- start 2023-10-16T13:27:48.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.172+0000 7f0fabb8fb80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d1ba83d010 0x55d1ba8ef020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:48.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.172+0000 7f0fabb8fb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d1ba862220 0x55d1ba8eb200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:48.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.172+0000 7f0fabb8fb80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55d1ba867630 0x55d1ba880ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:48.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.172+0000 7f0fabb8fb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1ba8e2120 con 0x55d1ba862220 2023-10-16T13:27:48.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.172+0000 7f0fabb8fb80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1ba8ebd60 con 0x55d1ba83d010 2023-10-16T13:27:48.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.172+0000 7f0fabb8fb80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d1ba839ca0 con 0x55d1ba867630 2023-10-16T13:27:48.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.172+0000 7f0fabb8fb80 10 client.0 _ll_register_callbacks cb 0x55d1ba58b7b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T13:27:48.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.172+0000 7f0f9cf40700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d1ba862220 0x55d1ba8eb200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:48.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.172+0000 7f0f9cf40700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d1ba862220 0x55d1ba8eb200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48598/0 (socket says 172.21.15.16:48598) 2023-10-16T13:27:48.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.172+0000 7f0f9cf40700 1 -- 172.21.15.16:0/754274997 learned_addr learned my addr 172.21.15.16:0/754274997 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:48.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.172+0000 7f0f9df42700 1 --2- 172.21.15.16:0/754274997 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55d1ba867630 0x55d1ba880ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:48.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.172+0000 7f0f9d741700 1 --2- 172.21.15.16:0/754274997 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d1ba83d010 0x55d1ba8ef020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:48.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.173+0000 7f0f9cf40700 1 -- 172.21.15.16:0/754274997 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55d1ba867630 msgr2=0x55d1ba880ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:48.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.173+0000 7f0f9cf40700 1 --2- 172.21.15.16:0/754274997 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55d1ba867630 0x55d1ba880ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:48.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.173+0000 7f0f9cf40700 1 -- 172.21.15.16:0/754274997 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d1ba83d010 msgr2=0x55d1ba8ef020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:48.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.173+0000 7f0f9cf40700 1 --2- 172.21.15.16:0/754274997 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d1ba83d010 0x55d1ba8ef020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:48.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.173+0000 7f0f9cf40700 1 -- 172.21.15.16:0/754274997 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f9815b350 con 0x55d1ba862220 2023-10-16T13:27:48.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.173+0000 7f0f9d741700 1 --2- 172.21.15.16:0/754274997 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d1ba83d010 0x55d1ba8ef020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:27:48.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.173+0000 7f0f9df42700 1 --2- 172.21.15.16:0/754274997 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55d1ba867630 0x55d1ba880ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:27:48.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.173+0000 7f0f9cf40700 1 --2- 172.21.15.16:0/754274997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d1ba862220 0x55d1ba8eb200 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7f0f9400e980 tx=0x7f0f9401d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2e3270512a42ed7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:48.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.173+0000 7f0f77fff700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2023-10-16T13:27:48.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.173+0000 7f0f77fff700 1 -- 172.21.15.16:0/754274997 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f94073070 con 0x55d1ba862220 2023-10-16T13:27:48.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.173+0000 7f0f77fff700 1 -- 172.21.15.16:0/754274997 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(177..177 src has 1..177) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f0f94054d50 con 0x55d1ba862220 2023-10-16T13:27:48.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.173+0000 7f0fabb8fb80 1 -- 172.21.15.16:0/754274997 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0f94054be0 con 0x55d1ba862220 2023-10-16T13:27:48.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.173+0000 7f0f77fff700 1 -- 172.21.15.16:0/754274997 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0f94078020 con 0x55d1ba862220 2023-10-16T13:27:48.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.173+0000 7f0fabb8fb80 10 client.9918 fetch_fsmap learned FSMap version 86 2023-10-16T13:27:48.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.173+0000 7f0fabb8fb80 1 -- 172.21.15.16:0/754274997 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=86,monmap=2+}) v3 -- 0x55d1ba5e3200 con 0x55d1ba862220 2023-10-16T13:27:48.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.173+0000 7f0f77fff700 1 -- 172.21.15.16:0/754274997 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f94054be0 con 0x55d1ba862220 2023-10-16T13:27:48.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.173+0000 7f0f77fff700 1 -- 172.21.15.16:0/754274997 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 86) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f0f9405c170 con 0x55d1ba862220 2023-10-16T13:27:48.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.173+0000 7f0fabb8fb80 10 client.9918 fetch_fsmap finished waiting for FSMap version 86 2023-10-16T13:27:48.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.173+0000 7f0fabb8fb80 10 client.9918 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-16T13:27:48.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.173+0000 7f0fabb8fb80 20 client.9918 populate_metadata read hostname 'smithi016' 2023-10-16T13:27:48.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.173+0000 7f0fabb8fb80 1 --2- 172.21.15.16:0/754274997 >> [v2:172.21.15.49:6832/2218212001,v1:172.21.15.49:6834/2218212001] conn(0x55d1ba86d0b0 0x55d1ba881430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:48.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.173+0000 7f0fabb8fb80 4 client.9918 mds_command: new command op to 9774 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T13:27:48.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.173+0000 7f0fabb8fb80 1 -- 172.21.15.16:0/754274997 --> [v2:172.21.15.49:6832/2218212001,v1:172.21.15.49:6834/2218212001] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55d1ba78ab20 con 0x55d1ba86d0b0 2023-10-16T13:27:48.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.174+0000 7f0f9d741700 1 --2- 172.21.15.16:0/754274997 >> [v2:172.21.15.49:6832/2218212001,v1:172.21.15.49:6834/2218212001] conn(0x55d1ba86d0b0 0x55d1ba881430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:48.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.174+0000 7f0f9d741700 1 --2- 172.21.15.16:0/754274997 >> [v2:172.21.15.49:6832/2218212001,v1:172.21.15.49:6834/2218212001] conn(0x55d1ba86d0b0 0x55d1ba881430 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ae6b783169a163b server_cookie=198c4696aaac0882 in_seq=0 out_seq=0 2023-10-16T13:27:48.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.174+0000 7f0f77fff700 10 client.9918 ms_handle_connect on v2:172.21.15.49:6832/2218212001 2023-10-16T13:27:48.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.174+0000 7f0f77fff700 1 -- 172.21.15.16:0/754274997 <== mds.0 v2:172.21.15.49:6832/2218212001 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x55d1ba78ab20 con 0x55d1ba86d0b0 2023-10-16T13:27:48.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.174+0000 7f0f77fff700 10 client.9918 handle_command_reply: tid=0 2023-10-16T13:27:48.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.174+0000 7f0fabb8fb80 1 client.9918 shutdown 2023-10-16T13:27:48.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.174+0000 7f0fabb8fb80 1 -- 172.21.15.16:0/754274997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d1ba862220 msgr2=0x55d1ba8eb200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:48.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.174+0000 7f0fabb8fb80 1 --2- 172.21.15.16:0/754274997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d1ba862220 0x55d1ba8eb200 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7f0f9400e980 tx=0x7f0f9401d000 comp rx=0 tx=0).stop 2023-10-16T13:27:48.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.174+0000 7f0fabb8fb80 1 -- 172.21.15.16:0/754274997 shutdown_connections 2023-10-16T13:27:48.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.174+0000 7f0fabb8fb80 1 --2- 172.21.15.16:0/754274997 >> [v2:172.21.15.49:6832/2218212001,v1:172.21.15.49:6834/2218212001] conn(0x55d1ba86d0b0 0x55d1ba881430 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:48.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.174+0000 7f0fabb8fb80 1 --2- 172.21.15.16:0/754274997 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55d1ba867630 0x55d1ba880ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:48.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.174+0000 7f0fabb8fb80 1 --2- 172.21.15.16:0/754274997 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d1ba83d010 0x55d1ba8ef020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:48.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.174+0000 7f0f77fff700 0 client.9918 ms_handle_reset on v2:172.21.15.49:6832/2218212001 2023-10-16T13:27:48.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.174+0000 7f0fabb8fb80 1 --2- 172.21.15.16:0/754274997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d1ba862220 0x55d1ba8eb200 unknown :-1 s=CLOSED pgs=1943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:48.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.174+0000 7f0fabb8fb80 1 -- 172.21.15.16:0/754274997 >> 172.21.15.16:0/754274997 conn(0x55d1ba76cd80 msgr2=0x55d1ba83b850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:48.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.174+0000 7f0fabb8fb80 1 -- 172.21.15.16:0/754274997 shutdown_connections 2023-10-16T13:27:48.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.174+0000 7f0fabb8fb80 1 -- 172.21.15.16:0/754274997 wait complete. 2023-10-16T13:27:48.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.175+0000 7f0fabb8fb80 20 client.9918 trim_cache size 0 max 16384 2023-10-16T13:27:48.201 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-10-16T13:27:48.201 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "5b28dfa1-b21f-474a-9b2e-9fb77af58254", 2023-10-16T13:27:48.201 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 0, 2023-10-16T13:27:48.202 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 9774, 2023-10-16T13:27:48.202 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2023-10-16T13:27:48.202 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2023-10-16T13:27:48.202 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2023-10-16T13:27:48.202 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 6.0442322600000002, 2023-10-16T13:27:48.203 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 86, 2023-10-16T13:27:48.203 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 177, 2023-10-16T13:27:48.203 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 177, 2023-10-16T13:27:48.203 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 11.914795998000001 2023-10-16T13:27:48.203 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-10-16T13:27:48.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.175+0000 7f0f9f9a5700 1 -- 172.21.15.16:0/2210001833 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f0f7c06b020 msgr2=0x7f0f7c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:48.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.175+0000 7f0f9f9a5700 1 --2- 172.21.15.16:0/2210001833 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f0f7c06b020 0x7f0f7c06d4e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0f9806f300 tx=0x7f0f88008030 comp rx=0 tx=0).stop 2023-10-16T13:27:48.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.175+0000 7f0f9f9a5700 1 -- 172.21.15.16:0/2210001833 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f980cf880 msgr2=0x7f0f980cceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:48.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.175+0000 7f0f9f9a5700 1 --2- 172.21.15.16:0/2210001833 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f980cf880 0x7f0f980cceb0 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7f0f9400e9b0 tx=0x7f0f9400ed70 comp rx=0 tx=0).stop 2023-10-16T13:27:48.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.175+0000 7f0f9f9a5700 1 -- 172.21.15.16:0/2210001833 shutdown_connections 2023-10-16T13:27:48.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.175+0000 7f0f9f9a5700 1 --2- 172.21.15.16:0/2210001833 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f0f7c06b020 0x7f0f7c06d4e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:48.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.175+0000 7f0f9f9a5700 1 --2- 172.21.15.16:0/2210001833 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f980c7d30 0x7f0f9806a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:48.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.175+0000 7f0f9f9a5700 1 --2- 172.21.15.16:0/2210001833 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f980b5830 0x7f0f98154770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:48.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.175+0000 7f0f9f9a5700 1 --2- 172.21.15.16:0/2210001833 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f980cf880 0x7f0f980cceb0 unknown :-1 s=CLOSED pgs=1939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:48.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.175+0000 7f0f9f9a5700 1 -- 172.21.15.16:0/2210001833 >> 172.21.15.16:0/2210001833 conn(0x7f0f980b1fd0 msgr2=0x7f0f980afe40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:48.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.175+0000 7f0f9f9a5700 1 -- 172.21.15.16:0/2210001833 shutdown_connections 2023-10-16T13:27:48.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.175+0000 7f0f9f9a5700 1 -- 172.21.15.16:0/2210001833 wait complete. 2023-10-16T13:27:48.208 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T13:27:48.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.329+0000 7f90d7436700 1 Processor -- start 2023-10-16T13:27:48.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.330+0000 7f90d7436700 1 -- start start 2023-10-16T13:27:48.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.330+0000 7f90d7436700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f90d015a5c0 0x7f90d00cef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:48.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.330+0000 7f90d7436700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f90d00ce010 0x7f90d00d03d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:48.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.330+0000 7f90d7436700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90d0072a50 con 0x7f90d00d0910 2023-10-16T13:27:48.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.330+0000 7f90d7436700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90d00cf440 con 0x7f90d00ce010 2023-10-16T13:27:48.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.330+0000 7f90d7436700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90d00d0da0 con 0x7f90d015a5c0 2023-10-16T13:27:48.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.330+0000 7f90d49d1700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f90d00ce010 0x7f90d00d03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:48.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.330+0000 7f90d49d1700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f90d00ce010 0x7f90d00d03d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48606/0 (socket says 172.21.15.16:48606) 2023-10-16T13:27:48.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.331+0000 7f90d49d1700 1 -- 172.21.15.16:0/3137821288 learned_addr learned my addr 172.21.15.16:0/3137821288 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:48.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.331+0000 7f90d51d2700 1 --2- 172.21.15.16:0/3137821288 >> v2:172.21.15.49:3300/0 conn(0x7f90d015a5c0 0x7f90d00cef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:48.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.331+0000 7f90d49d1700 1 -- 172.21.15.16:0/3137821288 >> v2:172.21.15.49:3300/0 conn(0x7f90d015a5c0 msgr2=0x7f90d00cef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:48.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.331+0000 7f90d49d1700 1 --2- 172.21.15.16:0/3137821288 >> v2:172.21.15.49:3300/0 conn(0x7f90d015a5c0 0x7f90d00cef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:48.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.331+0000 7f90d49d1700 1 -- 172.21.15.16:0/3137821288 >> v1:172.21.15.49:6789/0 conn(0x7f90d00d0910 legacy=0x7f90d015e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:48.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.331+0000 7f90d49d1700 1 -- 172.21.15.16:0/3137821288 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90d00cba30 con 0x7f90d00ce010 2023-10-16T13:27:48.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.331+0000 7f90d51d2700 1 --2- 172.21.15.16:0/3137821288 >> v2:172.21.15.49:3300/0 conn(0x7f90d015a5c0 0x7f90d00cef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:27:48.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.331+0000 7f90d49d1700 1 --2- 172.21.15.16:0/3137821288 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90d00ce010 0x7f90d00d03d0 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7f90d00b2ce0 tx=0x7f90c4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad7fe20a97a63ab server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:48.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.331+0000 7f90c3fff700 1 -- 172.21.15.16:0/3137821288 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90c4004230 con 0x7f90d00ce010 2023-10-16T13:27:48.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.331+0000 7f90c3fff700 1 -- 172.21.15.16:0/3137821288 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90c4004390 con 0x7f90d00ce010 2023-10-16T13:27:48.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.331+0000 7f90c3fff700 1 -- 172.21.15.16:0/3137821288 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90c40036a0 con 0x7f90d00ce010 2023-10-16T13:27:48.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.332+0000 7f90d7436700 1 -- 172.21.15.16:0/3137821288 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90d00ce010 msgr2=0x7f90d00d03d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:48.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.332+0000 7f90d7436700 1 --2- 172.21.15.16:0/3137821288 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90d00ce010 0x7f90d00d03d0 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7f90d00b2ce0 tx=0x7f90c4009bd0 comp rx=0 tx=0).stop 2023-10-16T13:27:48.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.332+0000 7f90d7436700 1 -- 172.21.15.16:0/3137821288 shutdown_connections 2023-10-16T13:27:48.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.332+0000 7f90d7436700 1 --2- 172.21.15.16:0/3137821288 >> v2:172.21.15.49:3300/0 conn(0x7f90d015a5c0 0x7f90d00cef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:48.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.332+0000 7f90d7436700 1 --2- 172.21.15.16:0/3137821288 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90d00ce010 0x7f90d00d03d0 unknown :-1 s=CLOSED pgs=1944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:48.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.332+0000 7f90d7436700 1 -- 172.21.15.16:0/3137821288 >> 172.21.15.16:0/3137821288 conn(0x7f90d00b1fd0 msgr2=0x7f90d00b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:48.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.332+0000 7f90d7436700 1 -- 172.21.15.16:0/3137821288 shutdown_connections 2023-10-16T13:27:48.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.332+0000 7f90d7436700 1 -- 172.21.15.16:0/3137821288 wait complete. 2023-10-16T13:27:48.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.333+0000 7f90d7436700 1 Processor -- start 2023-10-16T13:27:48.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.333+0000 7f90d7436700 1 -- start start 2023-10-16T13:27:48.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.334+0000 7f90d7436700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f90d00ce010 0x7f90d0165960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:48.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.334+0000 7f90d7436700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f90d00d0910 0x7f90d00c9140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:48.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.334+0000 7f90d7436700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90d015a5c0 0x7f90d0150650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:48.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.334+0000 7f90d7436700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90d00ca210 con 0x7f90d015a5c0 2023-10-16T13:27:48.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.334+0000 7f90d7436700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90d00b30c0 con 0x7f90d00ce010 2023-10-16T13:27:48.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.334+0000 7f90d7436700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90d00c9c50 con 0x7f90d00d0910 2023-10-16T13:27:48.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.334+0000 7f90d59d3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90d015a5c0 0x7f90d0150650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:48.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.334+0000 7f90d59d3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90d015a5c0 0x7f90d0150650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48614/0 (socket says 172.21.15.16:48614) 2023-10-16T13:27:48.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.334+0000 7f90d59d3700 1 -- 172.21.15.16:0/3862170970 learned_addr learned my addr 172.21.15.16:0/3862170970 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:48.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.334+0000 7f90d49d1700 1 --2- 172.21.15.16:0/3862170970 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f90d00d0910 0x7f90d00c9140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:48.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.334+0000 7f90d51d2700 1 --2- 172.21.15.16:0/3862170970 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f90d00ce010 0x7f90d0165960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:48.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.334+0000 7f90d59d3700 1 -- 172.21.15.16:0/3862170970 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f90d00d0910 msgr2=0x7f90d00c9140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:48.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.334+0000 7f90d59d3700 1 --2- 172.21.15.16:0/3862170970 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f90d00d0910 0x7f90d00c9140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:48.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.334+0000 7f90d59d3700 1 -- 172.21.15.16:0/3862170970 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f90d00ce010 msgr2=0x7f90d0165960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:48.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.334+0000 7f90d59d3700 1 --2- 172.21.15.16:0/3862170970 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f90d00ce010 0x7f90d0165960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:48.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.334+0000 7f90d59d3700 1 -- 172.21.15.16:0/3862170970 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90c4009550 con 0x7f90d015a5c0 2023-10-16T13:27:48.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.334+0000 7f90d59d3700 1 --2- 172.21.15.16:0/3862170970 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90d015a5c0 0x7f90d0150650 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7f90c800b9e0 tx=0x7f90c800bcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:48.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.335+0000 7f90c1ffb700 1 -- 172.21.15.16:0/3862170970 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90c800d820 con 0x7f90d015a5c0 2023-10-16T13:27:48.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.335+0000 7f90c1ffb700 1 -- 172.21.15.16:0/3862170970 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90c800d980 con 0x7f90d015a5c0 2023-10-16T13:27:48.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.335+0000 7f90d7436700 1 -- 172.21.15.16:0/3862170970 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90d00c9f30 con 0x7f90d015a5c0 2023-10-16T13:27:48.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.335+0000 7f90c1ffb700 1 -- 172.21.15.16:0/3862170970 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90c800dd30 con 0x7f90d015a5c0 2023-10-16T13:27:48.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.335+0000 7f90d7436700 1 -- 172.21.15.16:0/3862170970 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90d00b35b0 con 0x7f90d015a5c0 2023-10-16T13:27:48.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.336+0000 7f90d7436700 1 -- 172.21.15.16:0/3862170970 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90d00b37d0 con 0x7f90d015a5c0 2023-10-16T13:27:48.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.336+0000 7f90c1ffb700 1 -- 172.21.15.16:0/3862170970 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f90c8015420 con 0x7f90d015a5c0 2023-10-16T13:27:48.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.336+0000 7f90c1ffb700 1 --2- 172.21.15.16:0/3862170970 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f90b406b020 0x7f90b406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:48.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.337+0000 7f90c1ffb700 1 -- 172.21.15.16:0/3862170970 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f90c8011070 con 0x7f90d015a5c0 2023-10-16T13:27:48.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.337+0000 7f90d51d2700 1 --2- 172.21.15.16:0/3862170970 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f90b406b020 0x7f90b406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:48.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.337+0000 7f90d51d2700 1 --2- 172.21.15.16:0/3862170970 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f90b406b020 0x7f90b406d4e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f90cc006fd0 tx=0x7f90cc008040 comp rx=0 tx=0).ready entity=mgr.9510 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:48.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.341+0000 7f90c1ffb700 1 -- 172.21.15.16:0/3862170970 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f90c80545f0 con 0x7f90d015a5c0 2023-10-16T13:27:48.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.593+0000 7f90d7436700 1 -- 172.21.15.16:0/3862170970 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f90d004ebb0 con 0x7f90d015a5c0 2023-10-16T13:27:48.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.593+0000 7f90c1ffb700 1 -- 172.21.15.16:0/3862170970 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v86) v1 ==== 53+0+73 (secure 0 0 0) 0x7f90c8057c10 con 0x7f90d015a5c0 2023-10-16T13:27:48.597 INFO:teuthology.orchestra.run.smithi016.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-16T13:27:48.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.595+0000 7f90d7436700 1 -- 172.21.15.16:0/3862170970 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f90b406b020 msgr2=0x7f90b406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:48.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.595+0000 7f90d7436700 1 --2- 172.21.15.16:0/3862170970 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f90b406b020 0x7f90b406d4e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f90cc006fd0 tx=0x7f90cc008040 comp rx=0 tx=0).stop 2023-10-16T13:27:48.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.595+0000 7f90d7436700 1 -- 172.21.15.16:0/3862170970 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90d015a5c0 msgr2=0x7f90d0150650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:48.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.595+0000 7f90d7436700 1 --2- 172.21.15.16:0/3862170970 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90d015a5c0 0x7f90d0150650 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7f90c800b9e0 tx=0x7f90c800bcf0 comp rx=0 tx=0).stop 2023-10-16T13:27:48.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.595+0000 7f90d7436700 1 -- 172.21.15.16:0/3862170970 shutdown_connections 2023-10-16T13:27:48.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.595+0000 7f90d7436700 1 --2- 172.21.15.16:0/3862170970 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90d015a5c0 0x7f90d0150650 unknown :-1 s=CLOSED pgs=1945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:48.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.595+0000 7f90d7436700 1 --2- 172.21.15.16:0/3862170970 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f90b406b020 0x7f90b406d4e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:48.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.595+0000 7f90d7436700 1 --2- 172.21.15.16:0/3862170970 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f90d00ce010 0x7f90d0165960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:48.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.595+0000 7f90d7436700 1 --2- 172.21.15.16:0/3862170970 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f90d00d0910 0x7f90d00c9140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:48.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.595+0000 7f90d7436700 1 -- 172.21.15.16:0/3862170970 >> 172.21.15.16:0/3862170970 conn(0x7f90d00b1fd0 msgr2=0x7f90d00afe10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:48.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.595+0000 7f90d7436700 1 -- 172.21.15.16:0/3862170970 shutdown_connections 2023-10-16T13:27:48.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:48.595+0000 7f90d7436700 1 -- 172.21.15.16:0/3862170970 wait complete. 2023-10-16T13:27:48.611 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-16T13:27:48.611 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T13:27:48.611 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi016.front.sepia.ceph.com 2023-10-16T13:27:48.612 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-16T13:27:48.612 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-16T13:27:48.612 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-16T13:27:48.613 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-16T13:27:48.613 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T13:27:48.613 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T13:27:48.614 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:27:48.614 DEBUG:teuthology.orchestra.run.smithi016:> ip addr 2023-10-16T13:27:48.632 INFO:teuthology.orchestra.run.smithi016.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T13:27:48.632 INFO:teuthology.orchestra.run.smithi016.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T13:27:48.632 INFO:teuthology.orchestra.run.smithi016.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T13:27:48.633 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T13:27:48.633 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 ::1/128 scope host 2023-10-16T13:27:48.633 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T13:27:48.633 INFO:teuthology.orchestra.run.smithi016.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T13:27:48.633 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e0 brd ff:ff:ff:ff:ff:ff 2023-10-16T13:27:48.633 INFO:teuthology.orchestra.run.smithi016.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T13:27:48.634 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e1 brd ff:ff:ff:ff:ff:ff 2023-10-16T13:27:48.634 INFO:teuthology.orchestra.run.smithi016.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T13:27:48.634 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:64 brd ff:ff:ff:ff:ff:ff 2023-10-16T13:27:48.634 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.21.15.16/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-16T13:27:48.635 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft 37667sec preferred_lft 37667sec 2023-10-16T13:27:48.635 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::ec4:7aff:febd:1564/64 scope link 2023-10-16T13:27:48.635 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T13:27:48.635 INFO:teuthology.orchestra.run.smithi016.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T13:27:48.635 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:65 brd ff:ff:ff:ff:ff:ff 2023-10-16T13:27:48.635 INFO:teuthology.orchestra.run.smithi016.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T13:27:48.636 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 36:cf:ea:25:db:f7 brd ff:ff:ff:ff:ff:ff 2023-10-16T13:27:48.636 INFO:teuthology.orchestra.run.smithi016.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-16T13:27:48.636 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T13:27:48.636 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::cfc:c5ff:fe52:a786/64 scope link 2023-10-16T13:27:48.636 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T13:27:48.637 INFO:teuthology.orchestra.run.smithi016.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T13:27:48.637 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 6e:62:9d:a4:43:23 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T13:27:48.637 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::6c62:9dff:fea4:4323/64 scope link 2023-10-16T13:27:48.637 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T13:27:48.637 INFO:teuthology.orchestra.run.smithi016.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T13:27:48.637 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 36:cf:ea:25:db:f7 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-16T13:27:48.638 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::34cf:eaff:fe25:dbf7/64 scope link 2023-10-16T13:27:48.638 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T13:27:48.638 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:27:48.639 DEBUG:teuthology.orchestra.run.smithi016:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T13:27:48.698 INFO:teuthology.orchestra.run.smithi016.stdout:1 2023-10-16T13:27:48.699 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:27:48.699 DEBUG:teuthology.orchestra.run.smithi016:> route 2023-10-16T13:27:48.758 INFO:teuthology.orchestra.run.smithi016.stdout:Kernel IP routing table 2023-10-16T13:27:48.759 INFO:teuthology.orchestra.run.smithi016.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T13:27:48.759 INFO:teuthology.orchestra.run.smithi016.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-16T13:27:48.759 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-16T13:27:48.759 INFO:teuthology.orchestra.run.smithi016.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T13:27:48.760 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:27:48.760 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec bash -c ' 2023-10-16T13:27:48.761 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2023-10-16T13:27:48.761 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-16T13:27:48.761 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-16T13:27:48.762 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-16T13:27:48.762 DEBUG:teuthology.orchestra.run.smithi016:> ') 2023-10-16T13:27:48.866 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:27:48.866 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2023-10-16T13:27:48.883 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-16T13:27:48.884 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-16T13:27:48.884 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T13:27:48.884 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:27:48.885 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set brx.0 up 2023-10-16T13:27:48.950 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T13:27:48.951 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-16T13:27:49.008 INFO:teuthology.orchestra.run.smithi016.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-16T13:27:49.009 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T13:27:49.009 DEBUG:teuthology.orchestra.run.smithi016:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-16T13:27:49.065 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-16T13:27:49.065 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:27:49.066 DEBUG:teuthology.orchestra.run.smithi016:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T13:27:49.153 INFO:teuthology.orchestra.run.smithi016.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T13:27:49.153 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T13:27:49.153 INFO:teuthology.orchestra.run.smithi016.stdout:invalid new device string format 2023-10-16T13:27:49.162 INFO:teuthology.orchestra.run.smithi016.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T13:27:49.163 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T13:27:49.163 INFO:teuthology.orchestra.run.smithi016.stdout:invalid new device string format 2023-10-16T13:27:49.163 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: resolved to: "172.21.15.16:3300,172.21.15.49:3300,172.21.15.49:3301" 2023-10-16T13:27:49.164 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T13:27:49.237 INFO:teuthology.orchestra.run.smithi016.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T13:27:49.237 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T13:27:49.237 INFO:teuthology.orchestra.run.smithi016.stdout:invalid new device string format 2023-10-16T13:27:49.237 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: resolved to: "172.21.15.16:3300,172.21.15.49:3300,172.21.15.49:3301" 2023-10-16T13:27:49.238 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T13:27:49.238 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: resolved to: "172.21.15.16:6789,172.21.15.49:6789,172.21.15.49:6790" 2023-10-16T13:27:49.238 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.16:6789,172.21.15.49:6789,172.21.15.49:6790:/ 2023-10-16T13:27:49.238 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=5b28dfa1-b21f-474a-9b2e-9fb77af58254" will pass to kernel 2023-10-16T13:27:49.239 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-16T13:27:49.240 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:27:49.240 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-16T13:27:49.268 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:27:49.269 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T13:27:49.359 INFO:teuthology.orchestra.run.smithi016.stdout:client8063 2023-10-16T13:27:49.359 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T13:27:49.360 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:27:49.379 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-16T13:27:49.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.544+0000 7fc99609b700 1 Processor -- start 2023-10-16T13:27:49.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.544+0000 7fc99609b700 1 -- start start 2023-10-16T13:27:49.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.545+0000 7fc99609b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc99015ad00 0x7fc99006c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:49.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.546+0000 7fc99609b700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9900b2120 con 0x7fc99006cd40 2023-10-16T13:27:49.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.546+0000 7fc99609b700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9900b4320 con 0x7fc9900b3e50 2023-10-16T13:27:49.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.546+0000 7fc99609b700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc99006d1d0 con 0x7fc99015ad00 2023-10-16T13:27:49.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.546+0000 7fc98f7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc99015ad00 0x7fc99006c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:49.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.546+0000 7fc98effd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc99006cd40 0x7fc9900b3740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37896/0 (socket says 172.21.15.16:37896) 2023-10-16T13:27:49.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.546+0000 7fc98effd700 1 -- 172.21.15.16:0/2764829407 learned_addr learned my addr 172.21.15.16:0/2764829407 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:27:49.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.546+0000 7fc98f7fe700 1 -- 172.21.15.16:0/2764829407 >> v1:172.21.15.49:6789/0 conn(0x7fc9900b3e50 legacy=0x7fc990070e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:49.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.546+0000 7fc98f7fe700 1 -- 172.21.15.16:0/2764829407 >> v1:172.21.15.16:6789/0 conn(0x7fc99006cd40 legacy=0x7fc9900b3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:49.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.546+0000 7fc98f7fe700 1 -- 172.21.15.16:0/2764829407 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9900b23e0 con 0x7fc99015ad00 2023-10-16T13:27:49.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.546+0000 7fc98e7fc700 1 -- 172.21.15.16:0/2764829407 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4121376805 0 0) 0x7fc9900b2120 con 0x7fc99006cd40 2023-10-16T13:27:49.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.547+0000 7fc98f7fe700 1 --2- 172.21.15.16:0/2764829407 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc99015ad00 0x7fc99006c800 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7fc980009870 tx=0x7fc980009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1753cd1aa057bdd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:49.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.547+0000 7fc98e7fc700 1 -- 172.21.15.16:0/2764829407 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc980004230 con 0x7fc99015ad00 2023-10-16T13:27:49.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.547+0000 7fc98e7fc700 1 -- 172.21.15.16:0/2764829407 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc980004390 con 0x7fc99015ad00 2023-10-16T13:27:49.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.547+0000 7fc98e7fc700 1 -- 172.21.15.16:0/2764829407 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9800036a0 con 0x7fc99015ad00 2023-10-16T13:27:49.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.548+0000 7fc99609b700 1 -- 172.21.15.16:0/2764829407 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc99015ad00 msgr2=0x7fc99006c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:49.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.548+0000 7fc99609b700 1 --2- 172.21.15.16:0/2764829407 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc99015ad00 0x7fc99006c800 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7fc980009870 tx=0x7fc980009b80 comp rx=0 tx=0).stop 2023-10-16T13:27:49.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.548+0000 7fc99609b700 1 -- 172.21.15.16:0/2764829407 shutdown_connections 2023-10-16T13:27:49.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.548+0000 7fc99609b700 1 --2- 172.21.15.16:0/2764829407 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc99015ad00 0x7fc99006c800 unknown :-1 s=CLOSED pgs=1947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:49.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.548+0000 7fc99609b700 1 -- 172.21.15.16:0/2764829407 >> 172.21.15.16:0/2764829407 conn(0x7fc9900b4cd0 msgr2=0x7fc99006a7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:49.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.548+0000 7fc99609b700 1 -- 172.21.15.16:0/2764829407 shutdown_connections 2023-10-16T13:27:49.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.548+0000 7fc99609b700 1 -- 172.21.15.16:0/2764829407 wait complete. 2023-10-16T13:27:49.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.549+0000 7fc99609b700 1 Processor -- start 2023-10-16T13:27:49.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.550+0000 7fc99609b700 1 -- start start 2023-10-16T13:27:49.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.550+0000 7fc99609b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc99006cd40 0x7fc990140ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:49.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.550+0000 7fc99609b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc9900b3e50 0x7fc990142e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:49.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.550+0000 7fc99609b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc99015ad00 0x7fc990167c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:49.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.550+0000 7fc99609b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc990143960 con 0x7fc99006cd40 2023-10-16T13:27:49.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.550+0000 7fc99609b700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc990143360 con 0x7fc99015ad00 2023-10-16T13:27:49.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.550+0000 7fc99609b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9900afe20 con 0x7fc9900b3e50 2023-10-16T13:27:49.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.551+0000 7fc98f7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc99006cd40 0x7fc990140ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:49.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.551+0000 7fc98f7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc99006cd40 0x7fc990140ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48628/0 (socket says 172.21.15.16:48628) 2023-10-16T13:27:49.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.551+0000 7fc98f7fe700 1 -- 172.21.15.16:0/2817634160 learned_addr learned my addr 172.21.15.16:0/2817634160 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:49.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.551+0000 7fc98effd700 1 --2- 172.21.15.16:0/2817634160 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc9900b3e50 0x7fc990142e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:49.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.551+0000 7fc98ffff700 1 --2- 172.21.15.16:0/2817634160 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc99015ad00 0x7fc990167c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:49.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.551+0000 7fc98f7fe700 1 -- 172.21.15.16:0/2817634160 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc9900b3e50 msgr2=0x7fc990142e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:49.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.551+0000 7fc98f7fe700 1 --2- 172.21.15.16:0/2817634160 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc9900b3e50 0x7fc990142e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:49.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.551+0000 7fc98f7fe700 1 -- 172.21.15.16:0/2817634160 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc99015ad00 msgr2=0x7fc990167c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:49.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.551+0000 7fc98f7fe700 1 --2- 172.21.15.16:0/2817634160 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc99015ad00 0x7fc990167c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:49.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.551+0000 7fc98f7fe700 1 -- 172.21.15.16:0/2817634160 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9800094a0 con 0x7fc99006cd40 2023-10-16T13:27:49.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.551+0000 7fc98f7fe700 1 --2- 172.21.15.16:0/2817634160 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc99006cd40 0x7fc990140ff0 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7fc9800090b0 tx=0x7fc9800047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:49.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.551+0000 7fc987fff700 1 -- 172.21.15.16:0/2817634160 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc980011830 con 0x7fc99006cd40 2023-10-16T13:27:49.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.551+0000 7fc987fff700 1 -- 172.21.15.16:0/2817634160 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc980011990 con 0x7fc99006cd40 2023-10-16T13:27:49.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.551+0000 7fc99609b700 1 -- 172.21.15.16:0/2817634160 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9900b0100 con 0x7fc99006cd40 2023-10-16T13:27:49.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.551+0000 7fc987fff700 1 -- 172.21.15.16:0/2817634160 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc980011d00 con 0x7fc99006cd40 2023-10-16T13:27:49.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.551+0000 7fc99609b700 1 -- 172.21.15.16:0/2817634160 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc990143e90 con 0x7fc99006cd40 2023-10-16T13:27:49.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.553+0000 7fc99609b700 1 -- 172.21.15.16:0/2817634160 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9900b02f0 con 0x7fc99006cd40 2023-10-16T13:27:49.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.553+0000 7fc987fff700 1 -- 172.21.15.16:0/2817634160 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc98001b440 con 0x7fc99006cd40 2023-10-16T13:27:49.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.554+0000 7fc987fff700 1 --2- 172.21.15.16:0/2817634160 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7fc97006b020 0x7fc97006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:49.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.554+0000 7fc987fff700 1 -- 172.21.15.16:0/2817634160 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 6129+0+0 (secure 0 0 0) 0x7fc98008b3f0 con 0x7fc99006cd40 2023-10-16T13:27:49.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.554+0000 7fc98effd700 1 --2- 172.21.15.16:0/2817634160 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7fc97006b020 0x7fc97006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:49.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.554+0000 7fc98effd700 1 --2- 172.21.15.16:0/2817634160 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7fc97006b020 0x7fc97006d4e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc9901687e0 tx=0x7fc978007be0 comp rx=0 tx=0).ready entity=mgr.9510 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:49.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.558+0000 7fc987fff700 1 -- 172.21.15.16:0/2817634160 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc980056310 con 0x7fc99006cd40 2023-10-16T13:27:49.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.794+0000 7fc99609b700 1 -- 172.21.15.16:0/2817634160 --> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fc9900b29a0 con 0x7fc97006b020 2023-10-16T13:27:49.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.795+0000 7fc987fff700 1 -- 172.21.15.16:0/2817634160 <== mgr.9510 v2:172.21.15.16:6832/2329471558 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fc978005aa0 con 0x7fc97006b020 2023-10-16T13:27:49.799 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2023-10-16T13:27:49.799 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:27:49.799 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cephfs" 2023-10-16T13:27:49.799 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:27:49.800 INFO:teuthology.orchestra.run.smithi016.stdout:] 2023-10-16T13:27:49.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.797+0000 7fc99609b700 1 -- 172.21.15.16:0/2817634160 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7fc97006b020 msgr2=0x7fc97006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:49.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.797+0000 7fc99609b700 1 --2- 172.21.15.16:0/2817634160 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7fc97006b020 0x7fc97006d4e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc9901687e0 tx=0x7fc978007be0 comp rx=0 tx=0).stop 2023-10-16T13:27:49.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.797+0000 7fc99609b700 1 -- 172.21.15.16:0/2817634160 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc99006cd40 msgr2=0x7fc990140ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:49.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.797+0000 7fc99609b700 1 --2- 172.21.15.16:0/2817634160 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc99006cd40 0x7fc990140ff0 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7fc9800090b0 tx=0x7fc9800047c0 comp rx=0 tx=0).stop 2023-10-16T13:27:49.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.798+0000 7fc99609b700 1 -- 172.21.15.16:0/2817634160 shutdown_connections 2023-10-16T13:27:49.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.798+0000 7fc99609b700 1 --2- 172.21.15.16:0/2817634160 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7fc97006b020 0x7fc97006d4e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:49.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.798+0000 7fc99609b700 1 --2- 172.21.15.16:0/2817634160 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc99015ad00 0x7fc990167c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:49.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.798+0000 7fc99609b700 1 --2- 172.21.15.16:0/2817634160 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc99006cd40 0x7fc990140ff0 unknown :-1 s=CLOSED pgs=1948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:49.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.798+0000 7fc99609b700 1 --2- 172.21.15.16:0/2817634160 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc9900b3e50 0x7fc990142e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:49.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.798+0000 7fc99609b700 1 -- 172.21.15.16:0/2817634160 >> 172.21.15.16:0/2817634160 conn(0x7fc9900b4cd0 msgr2=0x7fc990145c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:49.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.798+0000 7fc99609b700 1 -- 172.21.15.16:0/2817634160 shutdown_connections 2023-10-16T13:27:49.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.798+0000 7fc99609b700 1 -- 172.21.15.16:0/2817634160 wait complete. 2023-10-16T13:27:49.813 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-16T13:27:49.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.941+0000 7f87387c7700 1 Processor -- start 2023-10-16T13:27:49.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.941+0000 7f87387c7700 1 -- start start 2023-10-16T13:27:49.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.942+0000 7f87387c7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f873015da10 0x7f87300d1520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:49.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.942+0000 7f87387c7700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f87300d4060 0x7f87300cf2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:49.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.942+0000 7f87387c7700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f87300d1d20 0x7f87300d21b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:49.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.942+0000 7f87387c7700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87300d26f0 con 0x7f873015da10 2023-10-16T13:27:49.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.942+0000 7f87387c7700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87300d4540 con 0x7f87300d4060 2023-10-16T13:27:49.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.942+0000 7f87387c7700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87300ce7e0 con 0x7f87300d1d20 2023-10-16T13:27:49.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.942+0000 7f8736563700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f873015da10 0x7f87300d1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:49.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.942+0000 7f8736563700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f873015da10 0x7f87300d1520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48636/0 (socket says 172.21.15.16:48636) 2023-10-16T13:27:49.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.942+0000 7f8736563700 1 -- 172.21.15.16:0/3849510215 learned_addr learned my addr 172.21.15.16:0/3849510215 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:49.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.942+0000 7f8736563700 1 -- 172.21.15.16:0/3849510215 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f87300d1d20 msgr2=0x7f87300d21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:49.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.942+0000 7f8736563700 1 --2- 172.21.15.16:0/3849510215 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f87300d1d20 0x7f87300d21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:49.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.942+0000 7f8736563700 1 -- 172.21.15.16:0/3849510215 >> v2:172.21.15.49:3300/0 conn(0x7f87300d4060 msgr2=0x7f87300cf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:49.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.942+0000 7f8735d62700 1 --2- 172.21.15.16:0/3849510215 >> v2:172.21.15.49:3300/0 conn(0x7f87300d4060 0x7f87300cf2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:49.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.942+0000 7f8736563700 1 --2- 172.21.15.16:0/3849510215 >> v2:172.21.15.49:3300/0 conn(0x7f87300d4060 0x7f87300cf2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:49.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.942+0000 7f8736563700 1 -- 172.21.15.16:0/3849510215 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87300ceac0 con 0x7f873015da10 2023-10-16T13:27:49.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.943+0000 7f8736563700 1 --2- 172.21.15.16:0/3849510215 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f873015da10 0x7f87300d1520 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7f87300cdf70 tx=0x7f872c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a226a5a8a60604d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:49.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.943+0000 7f8735561700 1 -- 172.21.15.16:0/3849510215 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f872c005e30 con 0x7f873015da10 2023-10-16T13:27:49.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.943+0000 7f8735561700 1 -- 172.21.15.16:0/3849510215 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f872c004030 con 0x7f873015da10 2023-10-16T13:27:49.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.943+0000 7f8735561700 1 -- 172.21.15.16:0/3849510215 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f872c0043e0 con 0x7f873015da10 2023-10-16T13:27:49.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.943+0000 7f87387c7700 1 -- 172.21.15.16:0/3849510215 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f873015da10 msgr2=0x7f87300d1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:49.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.943+0000 7f87387c7700 1 --2- 172.21.15.16:0/3849510215 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f873015da10 0x7f87300d1520 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7f87300cdf70 tx=0x7f872c009bd0 comp rx=0 tx=0).stop 2023-10-16T13:27:49.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.944+0000 7f87387c7700 1 -- 172.21.15.16:0/3849510215 shutdown_connections 2023-10-16T13:27:49.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.944+0000 7f87387c7700 1 --2- 172.21.15.16:0/3849510215 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f87300d1d20 0x7f87300d21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:49.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.944+0000 7f87387c7700 1 --2- 172.21.15.16:0/3849510215 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f873015da10 0x7f87300d1520 unknown :-1 s=CLOSED pgs=1949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:49.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.944+0000 7f87387c7700 1 --2- 172.21.15.16:0/3849510215 >> v2:172.21.15.49:3300/0 conn(0x7f87300d4060 0x7f87300cf2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:49.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.944+0000 7f87387c7700 1 -- 172.21.15.16:0/3849510215 >> 172.21.15.16:0/3849510215 conn(0x7f873009c9f0 msgr2=0x7f87300d3740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:49.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.944+0000 7f87387c7700 1 -- 172.21.15.16:0/3849510215 shutdown_connections 2023-10-16T13:27:49.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.944+0000 7f87387c7700 1 -- 172.21.15.16:0/3849510215 wait complete. 2023-10-16T13:27:49.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.945+0000 7f87387c7700 1 Processor -- start 2023-10-16T13:27:49.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.945+0000 7f87387c7700 1 -- start start 2023-10-16T13:27:49.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.946+0000 7f87387c7700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f87300d1d20 0x7f87300cc1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:49.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.946+0000 7f87387c7700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f87300d4060 0x7f87300b7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:49.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.946+0000 7f87387c7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f873015da10 0x7f8730153000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:49.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.946+0000 7f87387c7700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87300b9220 con 0x7f873015da10 2023-10-16T13:27:49.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.946+0000 7f87387c7700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87300b8610 con 0x7f87300d1d20 2023-10-16T13:27:49.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.946+0000 7f87387c7700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87300b88f0 con 0x7f87300d4060 2023-10-16T13:27:49.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.946+0000 7f8736d64700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f873015da10 0x7f8730153000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:49.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.946+0000 7f8736d64700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f873015da10 0x7f8730153000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48646/0 (socket says 172.21.15.16:48646) 2023-10-16T13:27:49.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.946+0000 7f8736d64700 1 -- 172.21.15.16:0/2595151258 learned_addr learned my addr 172.21.15.16:0/2595151258 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:49.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.946+0000 7f8736563700 1 --2- 172.21.15.16:0/2595151258 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f87300d1d20 0x7f87300cc1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:49.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.946+0000 7f8735d62700 1 --2- 172.21.15.16:0/2595151258 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f87300d4060 0x7f87300b7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:49.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.946+0000 7f8736d64700 1 -- 172.21.15.16:0/2595151258 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f87300d4060 msgr2=0x7f87300b7dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:49.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.946+0000 7f8736d64700 1 --2- 172.21.15.16:0/2595151258 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f87300d4060 0x7f87300b7dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:49.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.946+0000 7f8736d64700 1 -- 172.21.15.16:0/2595151258 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f87300d1d20 msgr2=0x7f87300cc1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:49.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.946+0000 7f8736d64700 1 --2- 172.21.15.16:0/2595151258 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f87300d1d20 0x7f87300cc1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:49.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.946+0000 7f8736d64700 1 -- 172.21.15.16:0/2595151258 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f872c009500 con 0x7f873015da10 2023-10-16T13:27:49.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.946+0000 7f8735d62700 1 --2- 172.21.15.16:0/2595151258 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f87300d4060 0x7f87300b7dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:27:49.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.946+0000 7f8736563700 1 --2- 172.21.15.16:0/2595151258 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f87300d1d20 0x7f87300cc1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:27:49.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.946+0000 7f8736d64700 1 --2- 172.21.15.16:0/2595151258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f873015da10 0x7f8730153000 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7f872800b970 tx=0x7f872800bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:49.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.946+0000 7f8726ffd700 1 -- 172.21.15.16:0/2595151258 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f872800c9b0 con 0x7f873015da10 2023-10-16T13:27:49.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.946+0000 7f8726ffd700 1 -- 172.21.15.16:0/2595151258 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f872800cb30 con 0x7f873015da10 2023-10-16T13:27:49.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.946+0000 7f87387c7700 1 -- 172.21.15.16:0/2595151258 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87300b8bd0 con 0x7f873015da10 2023-10-16T13:27:49.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.946+0000 7f8726ffd700 1 -- 172.21.15.16:0/2595151258 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8728014440 con 0x7f873015da10 2023-10-16T13:27:49.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.947+0000 7f87387c7700 1 -- 172.21.15.16:0/2595151258 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87300b8300 con 0x7f873015da10 2023-10-16T13:27:49.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.948+0000 7f87387c7700 1 -- 172.21.15.16:0/2595151258 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87300b9390 con 0x7f873015da10 2023-10-16T13:27:49.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.949+0000 7f8726ffd700 1 -- 172.21.15.16:0/2595151258 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 98) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f872801d3b0 con 0x7f873015da10 2023-10-16T13:27:49.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.949+0000 7f8726ffd700 1 --2- 172.21.15.16:0/2595151258 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f871406b020 0x7f871406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:49.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.949+0000 7f8726ffd700 1 -- 172.21.15.16:0/2595151258 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f8728006e40 con 0x7f873015da10 2023-10-16T13:27:49.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.949+0000 7f8736563700 1 --2- 172.21.15.16:0/2595151258 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f871406b020 0x7f871406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:49.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.950+0000 7f8736563700 1 --2- 172.21.15.16:0/2595151258 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f871406b020 0x7f871406d4e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f87300cdf70 tx=0x7f872c008070 comp rx=0 tx=0).ready entity=mgr.9510 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:49.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:49.953+0000 7f8726ffd700 1 -- 172.21.15.16:0/2595151258 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f87280548b0 con 0x7f873015da10 2023-10-16T13:27:50.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:50.232+0000 7f87387c7700 1 -- 172.21.15.16:0/2595151258 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f873004ebb0 con 0x7f873015da10 2023-10-16T13:27:51.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.084+0000 7f8726ffd700 1 -- 172.21.15.16:0/2595151258 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v99) v1 ==== 92+0+0 (secure 0 0 0) 0x7f87280190a0 con 0x7f873015da10 2023-10-16T13:27:51.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.087+0000 7f87387c7700 1 -- 172.21.15.16:0/2595151258 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f871406b020 msgr2=0x7f871406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:51.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.087+0000 7f87387c7700 1 --2- 172.21.15.16:0/2595151258 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f871406b020 0x7f871406d4e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f87300cdf70 tx=0x7f872c008070 comp rx=0 tx=0).stop 2023-10-16T13:27:51.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.087+0000 7f87387c7700 1 -- 172.21.15.16:0/2595151258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f873015da10 msgr2=0x7f8730153000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:51.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.087+0000 7f87387c7700 1 --2- 172.21.15.16:0/2595151258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f873015da10 0x7f8730153000 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7f872800b970 tx=0x7f872800bd30 comp rx=0 tx=0).stop 2023-10-16T13:27:51.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.087+0000 7f87387c7700 1 -- 172.21.15.16:0/2595151258 shutdown_connections 2023-10-16T13:27:51.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.087+0000 7f87387c7700 1 --2- 172.21.15.16:0/2595151258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f873015da10 0x7f8730153000 unknown :-1 s=CLOSED pgs=1950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:51.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.087+0000 7f87387c7700 1 --2- 172.21.15.16:0/2595151258 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f871406b020 0x7f871406d4e0 secure :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f87300cdf70 tx=0x7f872c008070 comp rx=0 tx=0).stop 2023-10-16T13:27:51.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.087+0000 7f87387c7700 1 --2- 172.21.15.16:0/2595151258 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f87300d1d20 0x7f87300cc1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:51.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.087+0000 7f87387c7700 1 --2- 172.21.15.16:0/2595151258 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f87300d4060 0x7f87300b7dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:51.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.087+0000 7f87387c7700 1 -- 172.21.15.16:0/2595151258 >> 172.21.15.16:0/2595151258 conn(0x7f873009c9f0 msgr2=0x7f873006d110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:51.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.088+0000 7f87387c7700 1 -- 172.21.15.16:0/2595151258 shutdown_connections 2023-10-16T13:27:51.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.089+0000 7f87387c7700 1 -- 172.21.15.16:0/2595151258 wait complete. 2023-10-16T13:27:51.108 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-16T13:27:51.239 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:27:51.237+0000 7f55c0505200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T13:27:51.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.239+0000 7fb2465c2700 1 Processor -- start 2023-10-16T13:27:51.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.239+0000 7fb2465c2700 1 -- start start 2023-10-16T13:27:51.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.240+0000 7fb2465c2700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fb24015b070 0x7fb24015b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:51.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.240+0000 7fb2465c2700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2400cef70 con 0x7fb24006e170 2023-10-16T13:27:51.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.240+0000 7fb2465c2700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2400ce360 con 0x7fb2400cb880 2023-10-16T13:27:51.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.240+0000 7fb2465c2700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2400ce640 con 0x7fb24015b070 2023-10-16T13:27:51.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.240+0000 7fb244b5f700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb24006e170 0x7fb2400cf400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37954/0 (socket says 172.21.15.16:37954) 2023-10-16T13:27:51.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.240+0000 7fb244b5f700 1 -- 172.21.15.16:0/3586735651 learned_addr learned my addr 172.21.15.16:0/3586735651 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:27:51.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.240+0000 7fb23ffff700 1 --2- 172.21.15.16:0/3586735651 >> v2:172.21.15.49:3300/0 conn(0x7fb24015b070 0x7fb24015b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:51.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.241+0000 7fb23effd700 1 -- 172.21.15.16:0/3586735651 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1658851415 0 0) 0x7fb2400cef70 con 0x7fb24006e170 2023-10-16T13:27:51.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.241+0000 7fb23effd700 1 -- 172.21.15.16:0/3586735651 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2280035b0 con 0x7fb24006e170 2023-10-16T13:27:51.246 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:27:51.241+0000 7fcd78b3c200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T13:27:51.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.241+0000 7fb23effd700 1 -- 172.21.15.16:0/3586735651 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1810628098 0 0) 0x7fb2400ce360 con 0x7fb2400cb880 2023-10-16T13:27:51.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.241+0000 7fb23effd700 1 -- 172.21.15.16:0/3586735651 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2400cef70 con 0x7fb2400cb880 2023-10-16T13:27:51.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.241+0000 7fb23ffff700 1 -- 172.21.15.16:0/3586735651 >> v1:172.21.15.49:6789/0 conn(0x7fb2400cb880 legacy=0x7fb24006da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:51.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.241+0000 7fb23ffff700 1 -- 172.21.15.16:0/3586735651 >> v1:172.21.15.16:6789/0 conn(0x7fb24006e170 legacy=0x7fb2400cf400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:51.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.241+0000 7fb23ffff700 1 -- 172.21.15.16:0/3586735651 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2400ce920 con 0x7fb24015b070 2023-10-16T13:27:51.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.241+0000 7fb23ffff700 1 --2- 172.21.15.16:0/3586735651 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb24015b070 0x7fb24015b450 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fb230009840 tx=0x7fb230009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ccc9be7e6ef2508 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:51.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.241+0000 7fb23effd700 1 -- 172.21.15.16:0/3586735651 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb230011830 con 0x7fb24015b070 2023-10-16T13:27:51.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.241+0000 7fb23effd700 1 -- 172.21.15.16:0/3586735651 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb230011990 con 0x7fb24015b070 2023-10-16T13:27:51.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.241+0000 7fb23effd700 1 -- 172.21.15.16:0/3586735651 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb230011d20 con 0x7fb24015b070 2023-10-16T13:27:51.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.242+0000 7fb2465c2700 1 -- 172.21.15.16:0/3586735651 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb24015b070 msgr2=0x7fb24015b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:51.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.242+0000 7fb2465c2700 1 --2- 172.21.15.16:0/3586735651 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb24015b070 0x7fb24015b450 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fb230009840 tx=0x7fb230009b50 comp rx=0 tx=0).stop 2023-10-16T13:27:51.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.242+0000 7fb2465c2700 1 -- 172.21.15.16:0/3586735651 shutdown_connections 2023-10-16T13:27:51.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.242+0000 7fb2465c2700 1 --2- 172.21.15.16:0/3586735651 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb24015b070 0x7fb24015b450 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:51.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.242+0000 7fb2465c2700 1 -- 172.21.15.16:0/3586735651 >> 172.21.15.16:0/3586735651 conn(0x7fb2400b1f40 msgr2=0x7fb2400b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:51.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.242+0000 7fb2465c2700 1 -- 172.21.15.16:0/3586735651 shutdown_connections 2023-10-16T13:27:51.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.242+0000 7fb2465c2700 1 -- 172.21.15.16:0/3586735651 wait complete. 2023-10-16T13:27:51.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.244+0000 7fb2465c2700 1 Processor -- start 2023-10-16T13:27:51.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.244+0000 7fb2465c2700 1 -- start start 2023-10-16T13:27:51.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.246+0000 7fb2465c2700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb24006e170 0x7fb2400c74c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:51.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.246+0000 7fb2465c2700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb2400cb880 0x7fb2400b5c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:51.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.246+0000 7fb2465c2700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb24015b070 0x7fb24014bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:51.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.246+0000 7fb2465c2700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2400b6140 con 0x7fb24015b070 2023-10-16T13:27:51.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.246+0000 7fb2465c2700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2400c7aa0 con 0x7fb24006e170 2023-10-16T13:27:51.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.246+0000 7fb2465c2700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2401f67c0 con 0x7fb2400cb880 2023-10-16T13:27:51.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.246+0000 7fb244b5f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb24015b070 0x7fb24014bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:51.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.246+0000 7fb23ffff700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb24006e170 0x7fb2400c74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:51.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.246+0000 7fb23f7fe700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb2400cb880 0x7fb2400b5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:51.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.246+0000 7fb23ffff700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb24006e170 0x7fb2400c74c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:47744/0 (socket says 172.21.15.16:47744) 2023-10-16T13:27:51.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.246+0000 7fb23ffff700 1 -- 172.21.15.16:0/1912567649 learned_addr learned my addr 172.21.15.16:0/1912567649 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:51.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.246+0000 7fb23ffff700 1 -- 172.21.15.16:0/1912567649 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb2400cb880 msgr2=0x7fb2400b5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:51.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.246+0000 7fb23ffff700 1 --2- 172.21.15.16:0/1912567649 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb2400cb880 0x7fb2400b5c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:51.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.246+0000 7fb23ffff700 1 -- 172.21.15.16:0/1912567649 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb24015b070 msgr2=0x7fb24014bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:51.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.246+0000 7fb23ffff700 1 --2- 172.21.15.16:0/1912567649 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb24015b070 0x7fb24014bbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:51.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.246+0000 7fb23ffff700 1 -- 172.21.15.16:0/1912567649 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2300094a0 con 0x7fb24006e170 2023-10-16T13:27:51.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.246+0000 7fb23f7fe700 1 --2- 172.21.15.16:0/1912567649 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb2400cb880 0x7fb2400b5c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:27:51.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.246+0000 7fb23ffff700 1 --2- 172.21.15.16:0/1912567649 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb24006e170 0x7fb2400c74c0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fb230014040 tx=0x7fb230004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:51.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.247+0000 7fb23cff9700 1 -- 172.21.15.16:0/1912567649 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb23001a6e0 con 0x7fb24006e170 2023-10-16T13:27:51.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.247+0000 7fb23cff9700 1 -- 172.21.15.16:0/1912567649 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb23001a840 con 0x7fb24006e170 2023-10-16T13:27:51.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.247+0000 7fb2465c2700 1 -- 172.21.15.16:0/1912567649 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb24014c130 con 0x7fb24006e170 2023-10-16T13:27:51.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.247+0000 7fb23cff9700 1 -- 172.21.15.16:0/1912567649 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb23001abb0 con 0x7fb24006e170 2023-10-16T13:27:51.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.247+0000 7fb2465c2700 1 -- 172.21.15.16:0/1912567649 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2401f69d0 con 0x7fb24006e170 2023-10-16T13:27:51.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.248+0000 7fb2465c2700 1 -- 172.21.15.16:0/1912567649 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb24004ebb0 con 0x7fb24006e170 2023-10-16T13:27:51.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.248+0000 7fb23cff9700 1 -- 172.21.15.16:0/1912567649 <== mon.1 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 99) v1 ==== 89348+0+0 (secure 0 0 0) 0x7fb230022070 con 0x7fb24006e170 2023-10-16T13:27:51.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.249+0000 7fb23cff9700 1 --2- 172.21.15.16:0/1912567649 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7fb22c06b070 0x7fb22c06d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:51.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.249+0000 7fb23f7fe700 1 -- 172.21.15.16:0/1912567649 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7fb22c06b070 msgr2=0x7fb22c06d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2329471558 2023-10-16T13:27:51.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.249+0000 7fb23cff9700 1 -- 172.21.15.16:0/1912567649 <== mon.1 v2:172.21.15.49:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 6129+0+0 (secure 0 0 0) 0x7fb23008a870 con 0x7fb24006e170 2023-10-16T13:27:51.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.249+0000 7fb23f7fe700 1 --2- 172.21.15.16:0/1912567649 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7fb22c06b070 0x7fb22c06d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:27:51.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.254+0000 7fb23cff9700 1 -- 172.21.15.16:0/1912567649 <== mon.1 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb230055810 con 0x7fb24006e170 2023-10-16T13:27:51.309 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:27:51.307+0000 7f55c0505200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T13:27:51.318 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:27:51.315+0000 7fcd78b3c200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T13:27:51.385 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:27:51.383+0000 7f55c0505200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T13:27:51.394 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:27:51.391+0000 7fcd78b3c200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T13:27:51.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.450+0000 7fb23f7fe700 1 -- 172.21.15.16:0/1912567649 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7fb22c06b070 msgr2=0x7fb22c06d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2329471558 2023-10-16T13:27:51.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.450+0000 7fb23f7fe700 1 --2- 172.21.15.16:0/1912567649 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7fb22c06b070 0x7fb22c06d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:27:51.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.461+0000 7fb2465c2700 1 -- 172.21.15.16:0/1912567649 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7fb2400668b0 con 0x7fb24006e170 2023-10-16T13:27:51.467 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:27:51.464+0000 7f55c0505200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T13:27:51.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.466+0000 7fb23cff9700 1 -- 172.21.15.16:0/1912567649 <== mon.1 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v59)=0 v59) v1 ==== 138+0+0 (secure 0 0 0) 0x7fb230029020 con 0x7fb24006e170 2023-10-16T13:27:51.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.468+0000 7fb2465c2700 1 -- 172.21.15.16:0/1912567649 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7fb22c06b070 msgr2=0x7fb22c06d530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:27:51.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.468+0000 7fb2465c2700 1 --2- 172.21.15.16:0/1912567649 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7fb22c06b070 0x7fb22c06d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:51.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.468+0000 7fb2465c2700 1 -- 172.21.15.16:0/1912567649 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb24006e170 msgr2=0x7fb2400c74c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:51.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.468+0000 7fb2465c2700 1 --2- 172.21.15.16:0/1912567649 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb24006e170 0x7fb2400c74c0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fb230014040 tx=0x7fb230004750 comp rx=0 tx=0).stop 2023-10-16T13:27:51.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.468+0000 7fb2465c2700 1 -- 172.21.15.16:0/1912567649 shutdown_connections 2023-10-16T13:27:51.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.468+0000 7fb2465c2700 1 --2- 172.21.15.16:0/1912567649 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb24015b070 0x7fb24014bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:51.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.468+0000 7fb2465c2700 1 --2- 172.21.15.16:0/1912567649 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7fb22c06b070 0x7fb22c06d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:51.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.468+0000 7fb2465c2700 1 --2- 172.21.15.16:0/1912567649 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb24006e170 0x7fb2400c74c0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:51.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.468+0000 7fb2465c2700 1 --2- 172.21.15.16:0/1912567649 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb2400cb880 0x7fb2400b5c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:51.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.468+0000 7fb2465c2700 1 -- 172.21.15.16:0/1912567649 >> 172.21.15.16:0/1912567649 conn(0x7fb2400b1f40 msgr2=0x7fb2400cf400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:51.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.468+0000 7fb2465c2700 1 -- 172.21.15.16:0/1912567649 shutdown_connections 2023-10-16T13:27:51.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.468+0000 7fb2465c2700 1 -- 172.21.15.16:0/1912567649 wait complete. 2023-10-16T13:27:51.479 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:27:51.476+0000 7fcd78b3c200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T13:27:51.484 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-16T13:27:51.616 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:27:51.613+0000 7f55c0505200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T13:27:51.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.612+0000 7f0e4cc1d700 1 Processor -- start 2023-10-16T13:27:51.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.612+0000 7f0e4cc1d700 1 -- start start 2023-10-16T13:27:51.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.614+0000 7f0e4cc1d700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f0e4806e570 0x7f0e4815e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:51.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.614+0000 7f0e4cc1d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0e4806a7e0 0x7f0e4806ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:51.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.614+0000 7f0e4cc1d700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e4815b6a0 con 0x7f0e4815b2c0 2023-10-16T13:27:51.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.614+0000 7f0e4cc1d700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e4806b190 con 0x7f0e4806a7e0 2023-10-16T13:27:51.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.614+0000 7f0e4cc1d700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e48161b20 con 0x7f0e4806e570 2023-10-16T13:27:51.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.614+0000 7f0e4659c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0e4815b2c0 0x7f0e48161340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38000/0 (socket says 172.21.15.16:38000) 2023-10-16T13:27:51.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.614+0000 7f0e4659c700 1 -- 172.21.15.16:0/2778232712 learned_addr learned my addr 172.21.15.16:0/2778232712 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:27:51.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.614+0000 7f0e46d9d700 1 --2- 172.21.15.16:0/2778232712 >> v2:172.21.15.16:3300/0 conn(0x7f0e4806a7e0 0x7f0e4806ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:51.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.614+0000 7f0e45d9b700 1 --2- 172.21.15.16:0/2778232712 >> v2:172.21.15.49:3300/0 conn(0x7f0e4806e570 0x7f0e4815e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:51.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.614+0000 7f0e4559a700 1 -- 172.21.15.16:0/2778232712 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1046849307 0 0) 0x7f0e4815b6a0 con 0x7f0e4815b2c0 2023-10-16T13:27:51.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.614+0000 7f0e4559a700 1 -- 172.21.15.16:0/2778232712 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e300035b0 con 0x7f0e4815b2c0 2023-10-16T13:27:51.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.614+0000 7f0e46d9d700 1 -- 172.21.15.16:0/2778232712 >> v2:172.21.15.49:3300/0 conn(0x7f0e4806e570 msgr2=0x7f0e4815e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:51.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.614+0000 7f0e46d9d700 1 --2- 172.21.15.16:0/2778232712 >> v2:172.21.15.49:3300/0 conn(0x7f0e4806e570 0x7f0e4815e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:51.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.614+0000 7f0e46d9d700 1 -- 172.21.15.16:0/2778232712 >> v1:172.21.15.16:6789/0 conn(0x7f0e4815b2c0 legacy=0x7f0e48161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:51.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.614+0000 7f0e46d9d700 1 -- 172.21.15.16:0/2778232712 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e48161e00 con 0x7f0e4806a7e0 2023-10-16T13:27:51.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.614+0000 7f0e46d9d700 1 --2- 172.21.15.16:0/2778232712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e4806a7e0 0x7f0e4806ac50 secure :-1 s=READY pgs=1955 cs=0 l=1 rev1=1 crypto rx=0x7f0e480d03c0 tx=0x7f0e4000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f1933eb62dbecb8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:51.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.615+0000 7f0e4559a700 1 -- 172.21.15.16:0/2778232712 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e4000e070 con 0x7f0e4806a7e0 2023-10-16T13:27:51.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.615+0000 7f0e4559a700 1 -- 172.21.15.16:0/2778232712 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e4000bb20 con 0x7f0e4806a7e0 2023-10-16T13:27:51.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.615+0000 7f0e4559a700 1 -- 172.21.15.16:0/2778232712 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e40012430 con 0x7f0e4806a7e0 2023-10-16T13:27:51.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.616+0000 7f0e4cc1d700 1 -- 172.21.15.16:0/2778232712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e4806a7e0 msgr2=0x7f0e4806ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:51.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.616+0000 7f0e4cc1d700 1 --2- 172.21.15.16:0/2778232712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e4806a7e0 0x7f0e4806ac50 secure :-1 s=READY pgs=1955 cs=0 l=1 rev1=1 crypto rx=0x7f0e480d03c0 tx=0x7f0e4000b330 comp rx=0 tx=0).stop 2023-10-16T13:27:51.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.616+0000 7f0e4cc1d700 1 -- 172.21.15.16:0/2778232712 shutdown_connections 2023-10-16T13:27:51.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.616+0000 7f0e4cc1d700 1 --2- 172.21.15.16:0/2778232712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e4806a7e0 0x7f0e4806ac50 unknown :-1 s=CLOSED pgs=1955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:51.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.616+0000 7f0e4cc1d700 1 --2- 172.21.15.16:0/2778232712 >> v2:172.21.15.49:3300/0 conn(0x7f0e4806e570 0x7f0e4815e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:51.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.616+0000 7f0e4cc1d700 1 -- 172.21.15.16:0/2778232712 >> 172.21.15.16:0/2778232712 conn(0x7f0e480b1f40 msgr2=0x7f0e480b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:51.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.616+0000 7f0e4cc1d700 1 -- 172.21.15.16:0/2778232712 shutdown_connections 2023-10-16T13:27:51.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.616+0000 7f0e4cc1d700 1 -- 172.21.15.16:0/2778232712 wait complete. 2023-10-16T13:27:51.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.617+0000 7f0e4cc1d700 1 Processor -- start 2023-10-16T13:27:51.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.617+0000 7f0e4cc1d700 1 -- start start 2023-10-16T13:27:51.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.618+0000 7f0e4cc1d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e4806a7e0 0x7f0e480c79a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:51.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.618+0000 7f0e4cc1d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0e4806e570 0x7f0e481462c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:51.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.618+0000 7f0e4cc1d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0e4815b2c0 0x7f0e480c9c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:51.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.618+0000 7f0e4cc1d700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e480ca570 con 0x7f0e4806a7e0 2023-10-16T13:27:51.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.618+0000 7f0e4cc1d700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e4815f670 con 0x7f0e4806e570 2023-10-16T13:27:51.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.618+0000 7f0e4cc1d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e4815f950 con 0x7f0e4815b2c0 2023-10-16T13:27:51.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.618+0000 7f0e4659c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e4806a7e0 0x7f0e480c79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:51.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.618+0000 7f0e4659c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e4806a7e0 0x7f0e480c79a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48732/0 (socket says 172.21.15.16:48732) 2023-10-16T13:27:51.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.618+0000 7f0e4659c700 1 -- 172.21.15.16:0/417264025 learned_addr learned my addr 172.21.15.16:0/417264025 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:51.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.618+0000 7f0e45d9b700 1 --2- 172.21.15.16:0/417264025 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0e4806e570 0x7f0e481462c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:51.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.618+0000 7f0e46d9d700 1 --2- 172.21.15.16:0/417264025 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0e4815b2c0 0x7f0e480c9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:51.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.618+0000 7f0e4659c700 1 -- 172.21.15.16:0/417264025 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0e4815b2c0 msgr2=0x7f0e480c9c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:51.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.618+0000 7f0e4659c700 1 --2- 172.21.15.16:0/417264025 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0e4815b2c0 0x7f0e480c9c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:51.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.618+0000 7f0e4659c700 1 -- 172.21.15.16:0/417264025 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0e4806e570 msgr2=0x7f0e481462c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:51.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.618+0000 7f0e4659c700 1 --2- 172.21.15.16:0/417264025 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0e4806e570 0x7f0e481462c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:51.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.618+0000 7f0e4659c700 1 -- 172.21.15.16:0/417264025 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e40009a70 con 0x7f0e4806a7e0 2023-10-16T13:27:51.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.619+0000 7f0e4659c700 1 --2- 172.21.15.16:0/417264025 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e4806a7e0 0x7f0e480c79a0 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7f0e3800c700 tx=0x7f0e3800ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:51.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.619+0000 7f0e36ffd700 1 -- 172.21.15.16:0/417264025 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e38005d70 con 0x7f0e4806a7e0 2023-10-16T13:27:51.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.619+0000 7f0e36ffd700 1 -- 172.21.15.16:0/417264025 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e380070d0 con 0x7f0e4806a7e0 2023-10-16T13:27:51.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.619+0000 7f0e36ffd700 1 -- 172.21.15.16:0/417264025 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e38007460 con 0x7f0e4806a7e0 2023-10-16T13:27:51.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.619+0000 7f0e4cc1d700 1 -- 172.21.15.16:0/417264025 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e4815fc30 con 0x7f0e4806a7e0 2023-10-16T13:27:51.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.619+0000 7f0e4cc1d700 1 -- 172.21.15.16:0/417264025 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e481f6830 con 0x7f0e4806a7e0 2023-10-16T13:27:51.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.620+0000 7f0e4cc1d700 1 -- 172.21.15.16:0/417264025 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e4815e060 con 0x7f0e4806a7e0 2023-10-16T13:27:51.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.621+0000 7f0e36ffd700 1 -- 172.21.15.16:0/417264025 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 99) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f0e380075c0 con 0x7f0e4806a7e0 2023-10-16T13:27:51.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.621+0000 7f0e36ffd700 1 --2- 172.21.15.16:0/417264025 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f0e2806b070 0x7f0e2806d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:51.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.621+0000 7f0e45d9b700 1 -- 172.21.15.16:0/417264025 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f0e2806b070 msgr2=0x7f0e2806d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2329471558 2023-10-16T13:27:51.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.621+0000 7f0e36ffd700 1 -- 172.21.15.16:0/417264025 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f0e38088d10 con 0x7f0e4806a7e0 2023-10-16T13:27:51.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.621+0000 7f0e45d9b700 1 --2- 172.21.15.16:0/417264025 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f0e2806b070 0x7f0e2806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:27:51.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.626+0000 7f0e36ffd700 1 -- 172.21.15.16:0/417264025 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0e4815e060 con 0x7f0e4806a7e0 2023-10-16T13:27:51.636 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:27:51.632+0000 7fcd78b3c200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T13:27:51.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.822+0000 7f0e45d9b700 1 -- 172.21.15.16:0/417264025 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f0e2806b070 msgr2=0x7f0e2806d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2329471558 2023-10-16T13:27:51.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.822+0000 7f0e45d9b700 1 --2- 172.21.15.16:0/417264025 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f0e2806b070 0x7f0e2806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:27:51.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.831+0000 7f0e4cc1d700 1 -- 172.21.15.16:0/417264025 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f0e4815e060 con 0x7f0e4806a7e0 2023-10-16T13:27:51.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.834+0000 7f0e36ffd700 1 -- 172.21.15.16:0/417264025 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v60)=0 v60) v1 ==== 133+0+0 (secure 0 0 0) 0x7f0e4815e060 con 0x7f0e4806a7e0 2023-10-16T13:27:51.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.836+0000 7f0e4cc1d700 1 -- 172.21.15.16:0/417264025 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f0e2806b070 msgr2=0x7f0e2806d530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:27:51.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.836+0000 7f0e4cc1d700 1 --2- 172.21.15.16:0/417264025 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f0e2806b070 0x7f0e2806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:51.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.836+0000 7f0e4cc1d700 1 -- 172.21.15.16:0/417264025 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e4806a7e0 msgr2=0x7f0e480c79a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:51.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.836+0000 7f0e4cc1d700 1 --2- 172.21.15.16:0/417264025 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e4806a7e0 0x7f0e480c79a0 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7f0e3800c700 tx=0x7f0e3800ca10 comp rx=0 tx=0).stop 2023-10-16T13:27:51.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.836+0000 7f0e4cc1d700 1 -- 172.21.15.16:0/417264025 shutdown_connections 2023-10-16T13:27:51.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.836+0000 7f0e4cc1d700 1 --2- 172.21.15.16:0/417264025 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0e4815b2c0 0x7f0e480c9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:51.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.836+0000 7f0e4cc1d700 1 --2- 172.21.15.16:0/417264025 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e4806a7e0 0x7f0e480c79a0 unknown :-1 s=CLOSED pgs=1956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:51.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.836+0000 7f0e4cc1d700 1 --2- 172.21.15.16:0/417264025 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f0e2806b070 0x7f0e2806d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:51.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.836+0000 7f0e4cc1d700 1 --2- 172.21.15.16:0/417264025 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0e4806e570 0x7f0e481462c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:51.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.836+0000 7f0e4cc1d700 1 -- 172.21.15.16:0/417264025 >> 172.21.15.16:0/417264025 conn(0x7f0e480b1f40 msgr2=0x7f0e4815b930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:51.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.836+0000 7f0e4cc1d700 1 -- 172.21.15.16:0/417264025 shutdown_connections 2023-10-16T13:27:51.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:51.836+0000 7f0e4cc1d700 1 -- 172.21.15.16:0/417264025 wait complete. 2023-10-16T13:27:51.852 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:27:51.853 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-10-16T13:27:51.873 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1h 2023-10-16T13:27:51.958 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:27:51.956+0000 7f55c0505200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T13:27:51.997 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:27:51.994+0000 7fcd78b3c200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T13:27:52.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:52.044+0000 7f8cc566f700 1 Processor -- start 2023-10-16T13:27:52.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:52.044+0000 7f8cc566f700 1 -- start start 2023-10-16T13:27:52.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:52.044+0000 7f8cc566f700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f8cc015afe0 0x7f8cc015b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:52.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:52.044+0000 7f8cc566f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8cc00cb880 0x7f8cc00cbce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:52.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:52.044+0000 7f8cc566f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8cc006da60 0x7f8cc006def0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:52.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:52.044+0000 7f8cc566f700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cc00cc220 con 0x7f8cc00cb880 2023-10-16T13:27:52.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:52.044+0000 7f8cc566f700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cc006e430 con 0x7f8cc015afe0 2023-10-16T13:27:52.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:52.044+0000 7f8cc566f700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc00ce360 con 0x7f8cc006da60 2023-10-16T13:27:52.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:52.044+0000 7f8cbe7fc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8cc00cb880 0x7f8cc00cbce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:52.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:52.044+0000 7f8cbe7fc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8cc00cb880 0x7f8cc00cbce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48788/0 (socket says 172.21.15.16:48788) 2023-10-16T13:27:52.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:52.044+0000 7f8cbe7fc700 1 -- 172.21.15.16:0/850417059 learned_addr learned my addr 172.21.15.16:0/850417059 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:52.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:52.044+0000 7f8cbeffd700 1 --2- 172.21.15.16:0/850417059 >> v2:172.21.15.49:3300/0 conn(0x7f8cc015afe0 0x7f8cc015b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:52.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:52.044+0000 7f8cbf7fe700 1 --2- 172.21.15.16:0/850417059 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8cc006da60 0x7f8cc006def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:52.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:52.046+0000 7f8cbe7fc700 1 -- 172.21.15.16:0/850417059 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8cc006da60 msgr2=0x7f8cc006def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:52.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:52.046+0000 7f8cbe7fc700 1 --2- 172.21.15.16:0/850417059 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8cc006da60 0x7f8cc006def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:52.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:52.046+0000 7f8cbe7fc700 1 -- 172.21.15.16:0/850417059 >> v2:172.21.15.49:3300/0 conn(0x7f8cc015afe0 msgr2=0x7f8cc015b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:52.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:52.046+0000 7f8cbe7fc700 1 --2- 172.21.15.16:0/850417059 >> v2:172.21.15.49:3300/0 conn(0x7f8cc015afe0 0x7f8cc015b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:52.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:52.046+0000 7f8cbe7fc700 1 -- 172.21.15.16:0/850417059 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cc00ce640 con 0x7f8cc00cb880 2023-10-16T13:27:52.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:52.046+0000 7f8cbe7fc700 1 --2- 172.21.15.16:0/850417059 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cc00cb880 0x7f8cc00cbce0 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7f8cc00705a0 tx=0x7f8cb4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd6d45e96c60259d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:52.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:52.046+0000 7f8cbdffb700 1 -- 172.21.15.16:0/850417059 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cb4011830 con 0x7f8cc00cb880 2023-10-16T13:27:52.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:52.046+0000 7f8cbdffb700 1 -- 172.21.15.16:0/850417059 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cb4011990 con 0x7f8cc00cb880 2023-10-16T13:27:52.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:52.046+0000 7f8cbdffb700 1 -- 172.21.15.16:0/850417059 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cb4011d20 con 0x7f8cc00cb880 2023-10-16T13:27:52.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:52.046+0000 7f8cc566f700 1 -- 172.21.15.16:0/850417059 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cc00cb880 msgr2=0x7f8cc00cbce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:52.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:52.046+0000 7f8cc566f700 1 --2- 172.21.15.16:0/850417059 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cc00cb880 0x7f8cc00cbce0 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7f8cc00705a0 tx=0x7f8cb4009bd0 comp rx=0 tx=0).stop 2023-10-16T13:27:52.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:52.046+0000 7f8cc566f700 1 -- 172.21.15.16:0/850417059 shutdown_connections 2023-10-16T13:27:52.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:52.046+0000 7f8cc566f700 1 --2- 172.21.15.16:0/850417059 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8cc006da60 0x7f8cc006def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:52.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:52.046+0000 7f8cc566f700 1 --2- 172.21.15.16:0/850417059 >> v2:172.21.15.49:3300/0 conn(0x7f8cc015afe0 0x7f8cc015b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:52.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:52.046+0000 7f8cc566f700 1 --2- 172.21.15.16:0/850417059 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cc00cb880 0x7f8cc00cbce0 unknown :-1 s=CLOSED pgs=1957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:52.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:52.046+0000 7f8cc566f700 1 -- 172.21.15.16:0/850417059 >> 172.21.15.16:0/850417059 conn(0x7f8cc00b1f40 msgr2=0x7f8cc00b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:52.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:52.046+0000 7f8cc566f700 1 -- 172.21.15.16:0/850417059 shutdown_connections 2023-10-16T13:27:52.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:52.046+0000 7f8cc566f700 1 -- 172.21.15.16:0/850417059 wait complete. 2023-10-16T13:27:52.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:52.048+0000 7f8cc566f700 1 Processor -- start 2023-10-16T13:27:52.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:52.048+0000 7f8cc566f700 1 -- start start 2023-10-16T13:27:52.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:52.049+0000 7f8cc566f700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8cc006da60 0x7f8cc00af320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:52.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:52.049+0000 7f8cc566f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cc00cb880 0x7f8cc015a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:52.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:52.049+0000 7f8cc566f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8cc015afe0 0x7f8cc00ae1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:52.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:52.049+0000 7f8cc566f700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc00c8a80 con 0x7f8cc00cb880 2023-10-16T13:27:52.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:52.049+0000 7f8cc566f700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc00c8d60 con 0x7f8cc006da60 2023-10-16T13:27:52.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:52.049+0000 7f8cc566f700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc00c9040 con 0x7f8cc015afe0 2023-10-16T13:27:52.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:52.049+0000 7f8cbe7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cc00cb880 0x7f8cc015a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:52.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:52.049+0000 7f8cbe7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cc00cb880 0x7f8cc015a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48794/0 (socket says 172.21.15.16:48794) 2023-10-16T13:27:52.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:52.049+0000 7f8cbe7fc700 1 -- 172.21.15.16:0/188594107 learned_addr learned my addr 172.21.15.16:0/188594107 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:52.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:52.049+0000 7f8cbeffd700 1 --2- 172.21.15.16:0/188594107 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8cc006da60 0x7f8cc00af320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:52.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:52.049+0000 7f8cbf7fe700 1 --2- 172.21.15.16:0/188594107 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8cc015afe0 0x7f8cc00ae1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:52.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:52.049+0000 7f8cbe7fc700 1 -- 172.21.15.16:0/188594107 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8cc015afe0 msgr2=0x7f8cc00ae1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:52.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:52.049+0000 7f8cbe7fc700 1 --2- 172.21.15.16:0/188594107 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8cc015afe0 0x7f8cc00ae1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:52.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:52.049+0000 7f8cbe7fc700 1 -- 172.21.15.16:0/188594107 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8cc006da60 msgr2=0x7f8cc00af320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:52.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:52.049+0000 7f8cbe7fc700 1 --2- 172.21.15.16:0/188594107 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8cc006da60 0x7f8cc00af320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:52.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:52.049+0000 7f8cbe7fc700 1 -- 172.21.15.16:0/188594107 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cb4009550 con 0x7f8cc00cb880 2023-10-16T13:27:52.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:52.049+0000 7f8cbe7fc700 1 --2- 172.21.15.16:0/188594107 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cc00cb880 0x7f8cc015a010 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7f8cb4000c00 tx=0x7f8cb4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:52.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:52.049+0000 7f8ca77fe700 1 -- 172.21.15.16:0/188594107 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cb4004240 con 0x7f8cc00cb880 2023-10-16T13:27:52.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:52.049+0000 7f8cc566f700 1 -- 172.21.15.16:0/188594107 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cc00c9320 con 0x7f8cc00cb880 2023-10-16T13:27:52.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:52.049+0000 7f8ca77fe700 1 -- 172.21.15.16:0/188594107 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cb401f070 con 0x7f8cc00cb880 2023-10-16T13:27:52.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:52.049+0000 7f8ca77fe700 1 -- 172.21.15.16:0/188594107 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cb40045e0 con 0x7f8cc00cb880 2023-10-16T13:27:52.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:52.049+0000 7f8cc566f700 1 -- 172.21.15.16:0/188594107 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cc015a550 con 0x7f8cc00cb880 2023-10-16T13:27:52.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:52.051+0000 7f8cc566f700 1 -- 172.21.15.16:0/188594107 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cc0072880 con 0x7f8cc00cb880 2023-10-16T13:27:52.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:52.052+0000 7f8ca77fe700 1 -- 172.21.15.16:0/188594107 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 99) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f8cb4021030 con 0x7f8cc00cb880 2023-10-16T13:27:52.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:52.052+0000 7f8ca77fe700 1 --2- 172.21.15.16:0/188594107 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f8cac06b070 0x7f8cac06d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:52.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:52.052+0000 7f8cbeffd700 1 -- 172.21.15.16:0/188594107 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f8cac06b070 msgr2=0x7f8cac06d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2329471558 2023-10-16T13:27:52.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:52.052+0000 7f8ca77fe700 1 -- 172.21.15.16:0/188594107 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f8cb408aab0 con 0x7f8cc00cb880 2023-10-16T13:27:52.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:52.052+0000 7f8cbeffd700 1 --2- 172.21.15.16:0/188594107 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f8cac06b070 0x7f8cac06d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:27:52.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:52.061+0000 7f8ca77fe700 1 -- 172.21.15.16:0/188594107 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8cb40558b0 con 0x7f8cc00cb880 2023-10-16T13:27:52.082 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:27:52.079+0000 7f55c0505200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T13:27:52.125 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:27:52.122+0000 7fcd78b3c200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T13:27:52.159 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:27:52.157+0000 7f55c0505200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T13:27:52.202 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:27:52.200+0000 7fcd78b3c200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T13:27:52.243 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:27:52.241+0000 7f55c0505200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T13:27:52.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:52.253+0000 7f8cbeffd700 1 -- 172.21.15.16:0/188594107 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f8cac06b070 msgr2=0x7f8cac06d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2329471558 2023-10-16T13:27:52.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:52.253+0000 7f8cbeffd700 1 --2- 172.21.15.16:0/188594107 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f8cac06b070 0x7f8cac06d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:27:52.287 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:27:52.285+0000 7fcd78b3c200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T13:27:52.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:52.294+0000 7f8cc566f700 1 -- 172.21.15.16:0/188594107 --> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1h", "target": ["mon-mgr", ""]}) v1 -- 0x7f8cc00cec00 con 0x7f8cac06b070 2023-10-16T13:27:52.634 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:27:52.631+0000 7f55c0505200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T13:27:52.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:52.654+0000 7f8cbeffd700 1 -- 172.21.15.16:0/188594107 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f8cac06b070 msgr2=0x7f8cac06d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2329471558 2023-10-16T13:27:52.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:52.654+0000 7f8cbeffd700 1 --2- 172.21.15.16:0/188594107 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f8cac06b070 0x7f8cac06d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-16T13:27:52.676 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:27:52.673+0000 7fcd78b3c200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T13:27:53.244 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:27:53.242+0000 7f55c0505200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T13:27:53.287 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:27:53.285+0000 7fcd78b3c200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T13:27:53.318 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:27:53.315+0000 7f55c0505200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T13:27:53.361 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:27:53.359+0000 7fcd78b3c200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T13:27:53.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:53.455+0000 7f8cbeffd700 1 -- 172.21.15.16:0/188594107 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f8cac06b070 msgr2=0x7f8cac06d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2329471558 2023-10-16T13:27:53.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:53.455+0000 7f8cbeffd700 1 --2- 172.21.15.16:0/188594107 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f8cac06b070 0x7f8cac06d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-16T13:27:53.983 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:27:53.981+0000 7f55c0505200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:27:54.030 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:27:54.028+0000 7fcd78b3c200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:27:54.064 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:27:54.062+0000 7f55c0505200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T13:27:54.112 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:27:54.110+0000 7fcd78b3c200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T13:27:54.216 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:27:54.213+0000 7f55c0505200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:27:54.265 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:27:54.262+0000 7fcd78b3c200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:27:54.454 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:27:54.452+0000 7f55c0505200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T13:27:54.504 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:27:54.502+0000 7fcd78b3c200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T13:27:54.532 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:27:54.529+0000 7f55c0505200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T13:27:54.582 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:27:54.579+0000 7fcd78b3c200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T13:27:54.809 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:27:54.807+0000 7f55c0505200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T13:27:54.860 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:27:54.858+0000 7fcd78b3c200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T13:27:55.042 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:27:55.039+0000 7f55c0505200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T13:27:55.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:55.056+0000 7f8cbeffd700 1 -- 172.21.15.16:0/188594107 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f8cac06b070 msgr2=0x7f8cac06d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2329471558 2023-10-16T13:27:55.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:55.056+0000 7f8cbeffd700 1 --2- 172.21.15.16:0/188594107 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f8cac06b070 0x7f8cac06d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-10-16T13:27:55.094 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:27:55.091+0000 7fcd78b3c200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T13:27:55.199 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:27:55.197+0000 7f55c0505200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T13:27:55.251 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:27:55.249+0000 7fcd78b3c200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T13:27:55.319 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:27:55.317+0000 7f55c0505200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T13:27:55.372 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:27:55.370+0000 7fcd78b3c200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T13:27:55.482 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:27:55.480+0000 7f55c0505200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T13:27:55.537 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:27:55.535+0000 7fcd78b3c200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T13:27:55.754 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:27:55.751+0000 7f55c0505200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T13:27:55.811 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:27:55.808+0000 7fcd78b3c200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T13:27:55.921 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:27:55.919+0000 7f55c0505200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T13:27:55.980 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:27:55.978+0000 7fcd78b3c200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T13:27:56.478 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:27:56.475+0000 7f55c0505200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T13:27:56.541 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:27:56.539+0000 7fcd78b3c200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T13:27:56.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:56.545+0000 7f8ca77fe700 1 -- 172.21.15.16:0/188594107 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 100) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f8cb4055c80 con 0x7f8cc00cb880 2023-10-16T13:27:56.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:56.560+0000 7f8ca77fe700 1 -- 172.21.15.16:0/188594107 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mgrmap(e 101) v1 ==== 88957+0+0 (secure 0 0 0) 0x7f8cb4088df0 con 0x7f8cc00cb880 2023-10-16T13:27:56.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:56.560+0000 7f8ca77fe700 1 -- 172.21.15.16:0/188594107 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f8cac06b070 msgr2=0x7f8cac06d530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:27:56.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:56.560+0000 7f8ca77fe700 1 --2- 172.21.15.16:0/188594107 >> [v2:172.21.15.16:6832/2329471558,v1:172.21.15.16:6833/2329471558] conn(0x7f8cac06b070 0x7f8cac06d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:56.703 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:27:56.700+0000 7fcceeb1d700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:27:56.703 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:27:56.700+0000 7fcceeb1d700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:27:56.703 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:27:56.700+0000 7fcceeb1d700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:27:56.703 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:27:56.700+0000 7fcceeb1d700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:27:56.704 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:27:56.700+0000 7fcceeb1d700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:27:56.704 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:27:56.700+0000 7fccebb17700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:27:56.704 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:27:56.700+0000 7fccebb17700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:27:56.704 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:27:56.700+0000 7fccebb17700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:27:56.704 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:27:56.700+0000 7fccebb17700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:27:56.705 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:27:56.700+0000 7fccebb17700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:27:57.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:57.562+0000 7f8ca77fe700 1 -- 172.21.15.16:0/188594107 <== mon.0 v2:172.21.15.16:3300/0 9 ==== mgrmap(e 102) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f8cb4015cf0 con 0x7f8cc00cb880 2023-10-16T13:27:57.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:57.562+0000 7f8ca77fe700 1 --2- 172.21.15.16:0/188594107 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7f8cac027a00 0x7f8cac027de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:57.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:57.562+0000 7f8ca77fe700 1 -- 172.21.15.16:0/188594107 --> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1h", "target": ["mon-mgr", ""]}) v1 -- 0x7f8cb40045e0 con 0x7f8cac027a00 2023-10-16T13:27:57.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:57.564+0000 7f8cbeffd700 1 --2- 172.21.15.16:0/188594107 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7f8cac027a00 0x7f8cac027de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:57.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:57.566+0000 7f8cbeffd700 1 --2- 172.21.15.16:0/188594107 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7f8cac027a00 0x7f8cac027de0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f8cc00cab60 tx=0x7f8cb0009210 comp rx=0 tx=0).ready entity=mgr.9954 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:57.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:57.630+0000 7f8ca77fe700 1 -- 172.21.15.16:0/188594107 <== mgr.9954 v2:172.21.15.16:6832/3622553136 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f8cb40045e0 con 0x7f8cac027a00 2023-10-16T13:27:57.633 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule set for path snap_test_dir1 2023-10-16T13:27:57.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:57.632+0000 7f8cc566f700 1 -- 172.21.15.16:0/188594107 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7f8cac027a00 msgr2=0x7f8cac027de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:57.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:57.632+0000 7f8cc566f700 1 --2- 172.21.15.16:0/188594107 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7f8cac027a00 0x7f8cac027de0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f8cc00cab60 tx=0x7f8cb0009210 comp rx=0 tx=0).stop 2023-10-16T13:27:57.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:57.632+0000 7f8cc566f700 1 -- 172.21.15.16:0/188594107 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cc00cb880 msgr2=0x7f8cc015a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:57.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:57.632+0000 7f8cc566f700 1 --2- 172.21.15.16:0/188594107 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cc00cb880 0x7f8cc015a010 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7f8cb4000c00 tx=0x7f8cb4004750 comp rx=0 tx=0).stop 2023-10-16T13:27:57.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:57.632+0000 7f8cbf7fe700 1 -- 172.21.15.16:0/188594107 reap_dead start 2023-10-16T13:27:57.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:57.632+0000 7f8cc566f700 1 -- 172.21.15.16:0/188594107 shutdown_connections 2023-10-16T13:27:57.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:57.632+0000 7f8cc566f700 1 -- 172.21.15.16:0/188594107 >> 172.21.15.16:0/188594107 conn(0x7f8cc00b1f40 msgr2=0x7f8cc015d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:57.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:57.632+0000 7f8cc566f700 1 -- 172.21.15.16:0/188594107 shutdown_connections 2023-10-16T13:27:57.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:57.632+0000 7f8cc566f700 1 -- 172.21.15.16:0/188594107 wait complete. 2023-10-16T13:27:57.648 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2023-10-16T13:27:57.648 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2023-10-16T13:27:57.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:57.773+0000 7f8334186700 1 Processor -- start 2023-10-16T13:27:57.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:57.774+0000 7f8334186700 1 -- start start 2023-10-16T13:27:57.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:57.774+0000 7f8334186700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f832c0ce6a0 0x7f832c06c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:57.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:57.775+0000 7f8334186700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f832c0b2620 0x7f832c06c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:57.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:57.775+0000 7f8334186700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f832c0b20f0 con 0x7f832c0d2110 2023-10-16T13:27:57.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:57.775+0000 7f8334186700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f832c0af1f0 con 0x7f832c0b2620 2023-10-16T13:27:57.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:57.775+0000 7f8334186700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f832c0d03d0 con 0x7f832c0ce6a0 2023-10-16T13:27:57.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:57.775+0000 7f8332723700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f832c0b2620 0x7f832c06c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:57.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:57.775+0000 7f8332723700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f832c0b2620 0x7f832c06c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48964/0 (socket says 172.21.15.16:48964) 2023-10-16T13:27:57.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:57.775+0000 7f8331721700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f832c0d2110 0x7f832c0b0f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38234/0 (socket says 172.21.15.16:38234) 2023-10-16T13:27:57.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:57.775+0000 7f8332723700 1 -- 172.21.15.16:0/1556630988 learned_addr learned my addr 172.21.15.16:0/1556630988 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:57.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:57.775+0000 7f8331f22700 1 --2- 172.21.15.16:0/1556630988 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f832c0ce6a0 0x7f832c06c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:57.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:57.775+0000 7f8332723700 1 -- 172.21.15.16:0/1556630988 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f832c0ce6a0 msgr2=0x7f832c06c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:57.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:57.775+0000 7f8332723700 1 --2- 172.21.15.16:0/1556630988 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f832c0ce6a0 0x7f832c06c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:57.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:57.775+0000 7f8332723700 1 -- 172.21.15.16:0/1556630988 >> v1:172.21.15.16:6789/0 conn(0x7f832c0d2110 legacy=0x7f832c0b0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:57.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:57.775+0000 7f8332723700 1 -- 172.21.15.16:0/1556630988 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f832c0b2340 con 0x7f832c0b2620 2023-10-16T13:27:57.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:57.775+0000 7f8331f22700 1 --2- 172.21.15.16:0/1556630988 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f832c0ce6a0 0x7f832c06c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:27:57.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:57.775+0000 7f8332723700 1 --2- 172.21.15.16:0/1556630988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f832c0b2620 0x7f832c06c840 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7f8324009e50 tx=0x7f832400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86beb4690974371c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:57.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:57.775+0000 7f8330f20700 1 -- 172.21.15.16:0/1556630988 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f832400e070 con 0x7f832c0b2620 2023-10-16T13:27:57.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:57.775+0000 7f8330f20700 1 -- 172.21.15.16:0/1556630988 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f832400b9c0 con 0x7f832c0b2620 2023-10-16T13:27:57.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:57.775+0000 7f8330f20700 1 -- 172.21.15.16:0/1556630988 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f832400bd70 con 0x7f832c0b2620 2023-10-16T13:27:57.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:57.776+0000 7f8334186700 1 -- 172.21.15.16:0/1556630988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f832c0b2620 msgr2=0x7f832c06c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:57.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:57.776+0000 7f8334186700 1 --2- 172.21.15.16:0/1556630988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f832c0b2620 0x7f832c06c840 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7f8324009e50 tx=0x7f832400b330 comp rx=0 tx=0).stop 2023-10-16T13:27:57.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:57.776+0000 7f8334186700 1 -- 172.21.15.16:0/1556630988 shutdown_connections 2023-10-16T13:27:57.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:57.776+0000 7f8334186700 1 --2- 172.21.15.16:0/1556630988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f832c0b2620 0x7f832c06c840 unknown :-1 s=CLOSED pgs=1970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:57.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:57.776+0000 7f8334186700 1 --2- 172.21.15.16:0/1556630988 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f832c0ce6a0 0x7f832c06c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:57.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:57.776+0000 7f8334186700 1 -- 172.21.15.16:0/1556630988 >> 172.21.15.16:0/1556630988 conn(0x7f832c0d15f0 msgr2=0x7f832c0d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:57.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:57.776+0000 7f8334186700 1 -- 172.21.15.16:0/1556630988 shutdown_connections 2023-10-16T13:27:57.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:57.776+0000 7f8334186700 1 -- 172.21.15.16:0/1556630988 wait complete. 2023-10-16T13:27:57.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:57.778+0000 7f8334186700 1 Processor -- start 2023-10-16T13:27:57.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:57.778+0000 7f8334186700 1 -- start start 2023-10-16T13:27:57.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:57.778+0000 7f8334186700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f832c0b2620 0x7f832c0cae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:57.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:57.779+0000 7f8334186700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f832c0ce6a0 0x7f832c0ccee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:57.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:57.779+0000 7f8334186700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f832c0d2110 0x7f832c150670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:57.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:57.779+0000 7f8334186700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f832c152020 con 0x7f832c0d2110 2023-10-16T13:27:57.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:57.779+0000 7f8334186700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f832c151450 con 0x7f832c0b2620 2023-10-16T13:27:57.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:57.779+0000 7f8334186700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f832c1516f0 con 0x7f832c0ce6a0 2023-10-16T13:27:57.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:57.779+0000 7f8332723700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f832c0d2110 0x7f832c150670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:57.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:57.779+0000 7f8332723700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f832c0d2110 0x7f832c150670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48970/0 (socket says 172.21.15.16:48970) 2023-10-16T13:27:57.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:57.779+0000 7f8332723700 1 -- 172.21.15.16:0/3635434206 learned_addr learned my addr 172.21.15.16:0/3635434206 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:57.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:57.779+0000 7f8331721700 1 --2- 172.21.15.16:0/3635434206 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f832c0ce6a0 0x7f832c0ccee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:57.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:57.779+0000 7f8331f22700 1 --2- 172.21.15.16:0/3635434206 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f832c0b2620 0x7f832c0cae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:57.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:57.779+0000 7f8332723700 1 -- 172.21.15.16:0/3635434206 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f832c0ce6a0 msgr2=0x7f832c0ccee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:57.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:57.779+0000 7f8332723700 1 --2- 172.21.15.16:0/3635434206 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f832c0ce6a0 0x7f832c0ccee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:57.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:57.779+0000 7f8332723700 1 -- 172.21.15.16:0/3635434206 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f832c0b2620 msgr2=0x7f832c0cae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:57.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:57.779+0000 7f8332723700 1 --2- 172.21.15.16:0/3635434206 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f832c0b2620 0x7f832c0cae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:57.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:57.779+0000 7f8332723700 1 -- 172.21.15.16:0/3635434206 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8324009b30 con 0x7f832c0d2110 2023-10-16T13:27:57.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:57.779+0000 7f8332723700 1 --2- 172.21.15.16:0/3635434206 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f832c0d2110 0x7f832c150670 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7f832c067580 tx=0x7f8324004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:57.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:57.779+0000 7f831e7fc700 1 -- 172.21.15.16:0/3635434206 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83240092e0 con 0x7f832c0d2110 2023-10-16T13:27:57.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:57.779+0000 7f831e7fc700 1 -- 172.21.15.16:0/3635434206 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8324004960 con 0x7f832c0d2110 2023-10-16T13:27:57.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:57.779+0000 7f8334186700 1 -- 172.21.15.16:0/3635434206 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f832c1519d0 con 0x7f832c0d2110 2023-10-16T13:27:57.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:57.779+0000 7f831e7fc700 1 -- 172.21.15.16:0/3635434206 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f832401c680 con 0x7f832c0d2110 2023-10-16T13:27:57.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:57.779+0000 7f8334186700 1 -- 172.21.15.16:0/3635434206 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f832c0cb380 con 0x7f832c0d2110 2023-10-16T13:27:57.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:57.780+0000 7f8334186700 1 -- 172.21.15.16:0/3635434206 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f832c0cb540 con 0x7f832c0d2110 2023-10-16T13:27:57.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:57.781+0000 7f831e7fc700 1 -- 172.21.15.16:0/3635434206 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 102) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f832401c7e0 con 0x7f832c0d2110 2023-10-16T13:27:57.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:57.781+0000 7f831e7fc700 1 --2- 172.21.15.16:0/3635434206 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7f831006ada0 0x7f831006d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:57.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:57.781+0000 7f831e7fc700 1 -- 172.21.15.16:0/3635434206 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 6473+0+0 (secure 0 0 0) 0x7f832408b3f0 con 0x7f832c0d2110 2023-10-16T13:27:57.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:57.781+0000 7f8331f22700 1 --2- 172.21.15.16:0/3635434206 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7f831006ada0 0x7f831006d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:57.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:57.782+0000 7f8331f22700 1 --2- 172.21.15.16:0/3635434206 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7f831006ada0 0x7f831006d260 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8328006fd0 tx=0x7f8328008040 comp rx=0 tx=0).ready entity=mgr.9954 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:57.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:57.786+0000 7f831e7fc700 1 -- 172.21.15.16:0/3635434206 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8324056450 con 0x7f832c0d2110 2023-10-16T13:27:58.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.016+0000 7f8334186700 1 -- 172.21.15.16:0/3635434206 --> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f832c06bde0 con 0x7f831006ada0 2023-10-16T13:27:58.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.081+0000 7f831e7fc700 1 -- 172.21.15.16:0/3635434206 <== mgr.9954 v2:172.21.15.16:6832/3622553136 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f832800c050 con 0x7f831006ada0 2023-10-16T13:27:58.084 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule removed for path snap_test_dir1 2023-10-16T13:27:58.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.084+0000 7f8334186700 1 -- 172.21.15.16:0/3635434206 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7f831006ada0 msgr2=0x7f831006d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:58.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.084+0000 7f8334186700 1 --2- 172.21.15.16:0/3635434206 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7f831006ada0 0x7f831006d260 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8328006fd0 tx=0x7f8328008040 comp rx=0 tx=0).stop 2023-10-16T13:27:58.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.084+0000 7f8334186700 1 -- 172.21.15.16:0/3635434206 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f832c0d2110 msgr2=0x7f832c150670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:58.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.084+0000 7f8334186700 1 --2- 172.21.15.16:0/3635434206 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f832c0d2110 0x7f832c150670 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7f832c067580 tx=0x7f8324004ad0 comp rx=0 tx=0).stop 2023-10-16T13:27:58.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.084+0000 7f8334186700 1 -- 172.21.15.16:0/3635434206 shutdown_connections 2023-10-16T13:27:58.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.084+0000 7f8334186700 1 --2- 172.21.15.16:0/3635434206 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7f831006ada0 0x7f831006d260 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:58.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.084+0000 7f8334186700 1 --2- 172.21.15.16:0/3635434206 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f832c0d2110 0x7f832c150670 unknown :-1 s=CLOSED pgs=1971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:58.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.084+0000 7f8334186700 1 --2- 172.21.15.16:0/3635434206 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f832c0b2620 0x7f832c0cae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:58.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.084+0000 7f8334186700 1 --2- 172.21.15.16:0/3635434206 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f832c0ce6a0 0x7f832c0ccee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:58.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.084+0000 7f8334186700 1 -- 172.21.15.16:0/3635434206 >> 172.21.15.16:0/3635434206 conn(0x7f832c0d15f0 msgr2=0x7f832c0b0f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:58.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.084+0000 7f8334186700 1 -- 172.21.15.16:0/3635434206 shutdown_connections 2023-10-16T13:27:58.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.084+0000 7f8334186700 1 -- 172.21.15.16:0/3635434206 wait complete. 2023-10-16T13:27:58.100 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2023-10-16T13:27:58.100 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list snap_test_dir1 2023-10-16T13:27:58.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.224+0000 7f0f3a759700 1 Processor -- start 2023-10-16T13:27:58.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.224+0000 7f0f3a759700 1 -- start start 2023-10-16T13:27:58.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.225+0000 7f0f3a759700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0f34159de0 0x7f0f3415a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:58.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.225+0000 7f0f3a759700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f0f340cb1c0 0x7f0f340cb670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:58.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.225+0000 7f0f3a759700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f3415dda0 con 0x7f0f340cf4c0 2023-10-16T13:27:58.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.225+0000 7f0f3a759700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f3415ea20 con 0x7f0f34159de0 2023-10-16T13:27:58.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.225+0000 7f0f3a759700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f340cbbb0 con 0x7f0f340cb1c0 2023-10-16T13:27:58.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.225+0000 7f0f33fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0f34159de0 0x7f0f3415a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:58.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.225+0000 7f0f33fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0f34159de0 0x7f0f3415a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48974/0 (socket says 172.21.15.16:48974) 2023-10-16T13:27:58.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.225+0000 7f0f33fff700 1 -- 172.21.15.16:0/3126642582 learned_addr learned my addr 172.21.15.16:0/3126642582 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:58.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.225+0000 7f0f38cf6700 1 --2- 172.21.15.16:0/3126642582 >> v2:172.21.15.49:3300/0 conn(0x7f0f340cb1c0 0x7f0f340cb670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:58.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.226+0000 7f0f33fff700 1 -- 172.21.15.16:0/3126642582 >> v2:172.21.15.49:3300/0 conn(0x7f0f340cb1c0 msgr2=0x7f0f340cb670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:58.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.226+0000 7f0f33fff700 1 --2- 172.21.15.16:0/3126642582 >> v2:172.21.15.49:3300/0 conn(0x7f0f340cb1c0 0x7f0f340cb670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:58.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.226+0000 7f0f33fff700 1 -- 172.21.15.16:0/3126642582 >> v1:172.21.15.49:6789/0 conn(0x7f0f340cf4c0 legacy=0x7f0f3406aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:58.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.226+0000 7f0f33fff700 1 -- 172.21.15.16:0/3126642582 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f3415e080 con 0x7f0f34159de0 2023-10-16T13:27:58.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.226+0000 7f0f33fff700 1 --2- 172.21.15.16:0/3126642582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f34159de0 0x7f0f3415a1c0 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7f0f340ca140 tx=0x7f0f2400be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d832e29288d019c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:58.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.226+0000 7f0f32ffd700 1 -- 172.21.15.16:0/3126642582 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f2400db60 con 0x7f0f34159de0 2023-10-16T13:27:58.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.226+0000 7f0f32ffd700 1 -- 172.21.15.16:0/3126642582 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f2400dcc0 con 0x7f0f34159de0 2023-10-16T13:27:58.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.226+0000 7f0f32ffd700 1 -- 172.21.15.16:0/3126642582 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f24017630 con 0x7f0f34159de0 2023-10-16T13:27:58.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.226+0000 7f0f3a759700 1 -- 172.21.15.16:0/3126642582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f34159de0 msgr2=0x7f0f3415a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:58.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.226+0000 7f0f3a759700 1 --2- 172.21.15.16:0/3126642582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f34159de0 0x7f0f3415a1c0 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7f0f340ca140 tx=0x7f0f2400be70 comp rx=0 tx=0).stop 2023-10-16T13:27:58.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.227+0000 7f0f3a759700 1 -- 172.21.15.16:0/3126642582 shutdown_connections 2023-10-16T13:27:58.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.227+0000 7f0f3a759700 1 --2- 172.21.15.16:0/3126642582 >> v2:172.21.15.49:3300/0 conn(0x7f0f340cb1c0 0x7f0f340cb670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:58.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.227+0000 7f0f3a759700 1 --2- 172.21.15.16:0/3126642582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f34159de0 0x7f0f3415a1c0 unknown :-1 s=CLOSED pgs=1972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:58.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.227+0000 7f0f3a759700 1 -- 172.21.15.16:0/3126642582 >> 172.21.15.16:0/3126642582 conn(0x7f0f340cc5e0 msgr2=0x7f0f340cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:58.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.227+0000 7f0f3a759700 1 -- 172.21.15.16:0/3126642582 shutdown_connections 2023-10-16T13:27:58.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.227+0000 7f0f3a759700 1 -- 172.21.15.16:0/3126642582 wait complete. 2023-10-16T13:27:58.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.228+0000 7f0f3a759700 1 Processor -- start 2023-10-16T13:27:58.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.228+0000 7f0f3a759700 1 -- start start 2023-10-16T13:27:58.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.229+0000 7f0f3a759700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f340cb1c0 0x7f0f340c55e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:58.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.229+0000 7f0f3a759700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f340cf4c0 0x7f0f3415abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:58.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.229+0000 7f0f3a759700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f34159de0 0x7f0f341521b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:58.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.229+0000 7f0f3a759700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f3415c030 con 0x7f0f340cb1c0 2023-10-16T13:27:58.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.229+0000 7f0f3a759700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f3415b420 con 0x7f0f34159de0 2023-10-16T13:27:58.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.229+0000 7f0f3a759700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f3415b700 con 0x7f0f340cf4c0 2023-10-16T13:27:58.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.229+0000 7f0f33fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f340cb1c0 0x7f0f340c55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:58.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.229+0000 7f0f33fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f340cb1c0 0x7f0f340c55e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48980/0 (socket says 172.21.15.16:48980) 2023-10-16T13:27:58.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.229+0000 7f0f33fff700 1 -- 172.21.15.16:0/2730704435 learned_addr learned my addr 172.21.15.16:0/2730704435 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:58.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.229+0000 7f0f38cf6700 1 --2- 172.21.15.16:0/2730704435 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f34159de0 0x7f0f341521b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:58.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.229+0000 7f0f337fe700 1 --2- 172.21.15.16:0/2730704435 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f340cf4c0 0x7f0f3415abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:58.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.229+0000 7f0f33fff700 1 -- 172.21.15.16:0/2730704435 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f340cf4c0 msgr2=0x7f0f3415abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:58.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.229+0000 7f0f33fff700 1 --2- 172.21.15.16:0/2730704435 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f340cf4c0 0x7f0f3415abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:58.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.229+0000 7f0f33fff700 1 -- 172.21.15.16:0/2730704435 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f34159de0 msgr2=0x7f0f341521b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:58.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.229+0000 7f0f33fff700 1 --2- 172.21.15.16:0/2730704435 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f34159de0 0x7f0f341521b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:58.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.229+0000 7f0f33fff700 1 -- 172.21.15.16:0/2730704435 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f2400b7f0 con 0x7f0f340cb1c0 2023-10-16T13:27:58.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.229+0000 7f0f337fe700 1 --2- 172.21.15.16:0/2730704435 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f340cf4c0 0x7f0f3415abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:27:58.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.229+0000 7f0f33fff700 1 --2- 172.21.15.16:0/2730704435 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f340cb1c0 0x7f0f340c55e0 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7f0f24019040 tx=0x7f0f24004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:58.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.229+0000 7f0f30ff9700 1 -- 172.21.15.16:0/2730704435 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f240179e0 con 0x7f0f340cb1c0 2023-10-16T13:27:58.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.229+0000 7f0f30ff9700 1 -- 172.21.15.16:0/2730704435 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f24017b80 con 0x7f0f340cb1c0 2023-10-16T13:27:58.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.229+0000 7f0f3a759700 1 -- 172.21.15.16:0/2730704435 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f3415b9e0 con 0x7f0f340cb1c0 2023-10-16T13:27:58.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.229+0000 7f0f30ff9700 1 -- 172.21.15.16:0/2730704435 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f240036a0 con 0x7f0f340cb1c0 2023-10-16T13:27:58.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.230+0000 7f0f3a759700 1 -- 172.21.15.16:0/2730704435 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f3415b110 con 0x7f0f340cb1c0 2023-10-16T13:27:58.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.230+0000 7f0f3a759700 1 -- 172.21.15.16:0/2730704435 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f3415e620 con 0x7f0f340cb1c0 2023-10-16T13:27:58.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.231+0000 7f0f30ff9700 1 -- 172.21.15.16:0/2730704435 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 102) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f0f24015070 con 0x7f0f340cb1c0 2023-10-16T13:27:58.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.231+0000 7f0f30ff9700 1 --2- 172.21.15.16:0/2730704435 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7f0f1406ada0 0x7f0f1406d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:58.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.231+0000 7f0f30ff9700 1 -- 172.21.15.16:0/2730704435 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 6473+0+0 (secure 0 0 0) 0x7f0f2408d3b0 con 0x7f0f340cb1c0 2023-10-16T13:27:58.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.231+0000 7f0f337fe700 1 --2- 172.21.15.16:0/2730704435 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7f0f1406ada0 0x7f0f1406d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:58.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.232+0000 7f0f337fe700 1 --2- 172.21.15.16:0/2730704435 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7f0f1406ada0 0x7f0f1406d260 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0f340cf930 tx=0x7f0f1c009040 comp rx=0 tx=0).ready entity=mgr.9954 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:58.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.236+0000 7f0f30ff9700 1 -- 172.21.15.16:0/2730704435 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0f3415e620 con 0x7f0f340cb1c0 2023-10-16T13:27:58.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.466+0000 7f0f3a759700 1 -- 172.21.15.16:0/2730704435 --> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f0f3415e360 con 0x7f0f1406ada0 2023-10-16T13:27:58.475 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:27:58.473+0000 7fccf5b6b700 -1 mgr.server reply reply (2) No such file or directory SnapSchedule for snap_test_dir1 not found 2023-10-16T13:27:58.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.474+0000 7f0f30ff9700 1 -- 172.21.15.16:0/2730704435 <== mgr.9954 v2:172.21.15.16:6832/3622553136 1 ==== mgr_command_reply(tid 0: -2 SnapSchedule for snap_test_dir1 not found) v1 ==== 49+0+0 (secure 0 0 0) 0x7f0f1c0013c0 con 0x7f0f1406ada0 2023-10-16T13:27:58.477 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENOENT: SnapSchedule for snap_test_dir1 not found 2023-10-16T13:27:58.481 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-10-16T13:27:58.482 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:27:58.482 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2023-10-16T13:27:58.502 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-16T13:27:58.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.672+0000 7f72bd6d2700 1 Processor -- start 2023-10-16T13:27:58.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.672+0000 7f72bd6d2700 1 -- start start 2023-10-16T13:27:58.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.673+0000 7f72bd6d2700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f72b80cea60 0x7f72b80b2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:58.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.673+0000 7f72bd6d2700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72b80b2f30 con 0x7f72b80b8fa0 2023-10-16T13:27:58.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.673+0000 7f72bd6d2700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72b80d1c40 con 0x7f72b815d8d0 2023-10-16T13:27:58.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.673+0000 7f72bd6d2700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72b80d0bb0 con 0x7f72b80cea60 2023-10-16T13:27:58.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.673+0000 7f72b67fc700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f72b80b8fa0 0x7f72b80ce350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38282/0 (socket says 172.21.15.16:38282) 2023-10-16T13:27:58.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.673+0000 7f72b67fc700 1 -- 172.21.15.16:0/1211650583 learned_addr learned my addr 172.21.15.16:0/1211650583 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:27:58.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.673+0000 7f72b77fe700 1 --2- 172.21.15.16:0/1211650583 >> v2:172.21.15.49:3300/0 conn(0x7f72b80cea60 0x7f72b80b2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:58.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.673+0000 7f72b5ffb700 1 -- 172.21.15.16:0/1211650583 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3883387959 0 0) 0x7f72b80b2f30 con 0x7f72b80b8fa0 2023-10-16T13:27:58.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.673+0000 7f72b5ffb700 1 -- 172.21.15.16:0/1211650583 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72a00035b0 con 0x7f72b80b8fa0 2023-10-16T13:27:58.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.673+0000 7f72b77fe700 1 -- 172.21.15.16:0/1211650583 >> v1:172.21.15.49:6789/0 conn(0x7f72b815d8d0 legacy=0x7f72b80b6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:58.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.673+0000 7f72b77fe700 1 -- 172.21.15.16:0/1211650583 >> v1:172.21.15.16:6789/0 conn(0x7f72b80b8fa0 legacy=0x7f72b80ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:58.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.673+0000 7f72b77fe700 1 -- 172.21.15.16:0/1211650583 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72b80d1ee0 con 0x7f72b80cea60 2023-10-16T13:27:58.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.673+0000 7f72b77fe700 1 --2- 172.21.15.16:0/1211650583 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f72b80cea60 0x7f72b80b2830 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f72b0009d90 tx=0x7f72b000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1f6649faeb6d356a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:58.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.674+0000 7f72b5ffb700 1 -- 172.21.15.16:0/1211650583 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72b000e070 con 0x7f72b80cea60 2023-10-16T13:27:58.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.674+0000 7f72b5ffb700 1 -- 172.21.15.16:0/1211650583 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72b000bb20 con 0x7f72b80cea60 2023-10-16T13:27:58.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.674+0000 7f72b5ffb700 1 -- 172.21.15.16:0/1211650583 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72b000be90 con 0x7f72b80cea60 2023-10-16T13:27:58.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.674+0000 7f72bd6d2700 1 -- 172.21.15.16:0/1211650583 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f72b80cea60 msgr2=0x7f72b80b2830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:58.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.674+0000 7f72bd6d2700 1 --2- 172.21.15.16:0/1211650583 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f72b80cea60 0x7f72b80b2830 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f72b0009d90 tx=0x7f72b000b330 comp rx=0 tx=0).stop 2023-10-16T13:27:58.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.674+0000 7f72bd6d2700 1 -- 172.21.15.16:0/1211650583 shutdown_connections 2023-10-16T13:27:58.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.674+0000 7f72bd6d2700 1 --2- 172.21.15.16:0/1211650583 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f72b80cea60 0x7f72b80b2830 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:58.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.674+0000 7f72bd6d2700 1 -- 172.21.15.16:0/1211650583 >> 172.21.15.16:0/1211650583 conn(0x7f72b8072330 msgr2=0x7f72b8072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:58.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.674+0000 7f72bd6d2700 1 -- 172.21.15.16:0/1211650583 shutdown_connections 2023-10-16T13:27:58.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.674+0000 7f72bd6d2700 1 -- 172.21.15.16:0/1211650583 wait complete. 2023-10-16T13:27:58.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.676+0000 7f72bd6d2700 1 Processor -- start 2023-10-16T13:27:58.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.676+0000 7f72bd6d2700 1 -- start start 2023-10-16T13:27:58.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.677+0000 7f72bd6d2700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f72b80b8fa0 0x7f72b8156d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:58.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.677+0000 7f72bd6d2700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72b80cea60 0x7f72b81f70c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:58.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.677+0000 7f72bd6d2700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f72b815d8d0 0x7f72b80cb310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:58.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.677+0000 7f72bd6d2700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72b81579e0 con 0x7f72b80cea60 2023-10-16T13:27:58.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.677+0000 7f72bd6d2700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72b8157700 con 0x7f72b815d8d0 2023-10-16T13:27:58.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.677+0000 7f72bd6d2700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72b80ca830 con 0x7f72b80b8fa0 2023-10-16T13:27:58.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.677+0000 7f72b67fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72b80cea60 0x7f72b81f70c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:58.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.677+0000 7f72b67fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72b80cea60 0x7f72b81f70c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49014/0 (socket says 172.21.15.16:49014) 2023-10-16T13:27:58.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.677+0000 7f72b67fc700 1 -- 172.21.15.16:0/3386668345 learned_addr learned my addr 172.21.15.16:0/3386668345 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:58.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.677+0000 7f72b6ffd700 1 --2- 172.21.15.16:0/3386668345 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f72b80b8fa0 0x7f72b8156d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:58.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.677+0000 7f72b77fe700 1 --2- 172.21.15.16:0/3386668345 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f72b815d8d0 0x7f72b80cb310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:58.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.677+0000 7f72b67fc700 1 -- 172.21.15.16:0/3386668345 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f72b80b8fa0 msgr2=0x7f72b8156d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:58.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.677+0000 7f72b67fc700 1 --2- 172.21.15.16:0/3386668345 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f72b80b8fa0 0x7f72b8156d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:58.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.677+0000 7f72b67fc700 1 -- 172.21.15.16:0/3386668345 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f72b815d8d0 msgr2=0x7f72b80cb310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:58.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.677+0000 7f72b67fc700 1 --2- 172.21.15.16:0/3386668345 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f72b815d8d0 0x7f72b80cb310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:58.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.677+0000 7f72b67fc700 1 -- 172.21.15.16:0/3386668345 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72b0009a70 con 0x7f72b80cea60 2023-10-16T13:27:58.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.677+0000 7f72b67fc700 1 --2- 172.21.15.16:0/3386668345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72b80cea60 0x7f72b81f70c0 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7f72ac001eb0 tx=0x7f72ac00c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:58.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.678+0000 7f729f7fe700 1 -- 172.21.15.16:0/3386668345 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72ac005d70 con 0x7f72b80cea60 2023-10-16T13:27:58.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.678+0000 7f729f7fe700 1 -- 172.21.15.16:0/3386668345 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72ac0070d0 con 0x7f72b80cea60 2023-10-16T13:27:58.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.678+0000 7f72bd6d2700 1 -- 172.21.15.16:0/3386668345 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72b80cab10 con 0x7f72b80cea60 2023-10-16T13:27:58.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.678+0000 7f729f7fe700 1 -- 172.21.15.16:0/3386668345 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72ac007460 con 0x7f72b80cea60 2023-10-16T13:27:58.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.678+0000 7f72bd6d2700 1 -- 172.21.15.16:0/3386668345 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72b8157340 con 0x7f72b80cea60 2023-10-16T13:27:58.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.679+0000 7f72bd6d2700 1 -- 172.21.15.16:0/3386668345 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72b80cb030 con 0x7f72b80cea60 2023-10-16T13:27:58.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.679+0000 7f729f7fe700 1 -- 172.21.15.16:0/3386668345 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 102) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f72ac0075c0 con 0x7f72b80cea60 2023-10-16T13:27:58.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.679+0000 7f729f7fe700 1 --2- 172.21.15.16:0/3386668345 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7f72a406ada0 0x7f72a406d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:58.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.680+0000 7f729f7fe700 1 -- 172.21.15.16:0/3386668345 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 6473+0+0 (secure 0 0 0) 0x7f72ac088930 con 0x7f72b80cea60 2023-10-16T13:27:58.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.680+0000 7f72b6ffd700 1 --2- 172.21.15.16:0/3386668345 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7f72a406ada0 0x7f72a406d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:58.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.680+0000 7f72b6ffd700 1 --2- 172.21.15.16:0/3386668345 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7f72a406ada0 0x7f72a406d260 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f72a8007fd0 tx=0x7f72a8009040 comp rx=0 tx=0).ready entity=mgr.9954 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:58.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.684+0000 7f729f7fe700 1 -- 172.21.15.16:0/3386668345 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f72b80cb030 con 0x7f72b80cea60 2023-10-16T13:27:58.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:58.949+0000 7f72bd6d2700 1 -- 172.21.15.16:0/3386668345 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f72b80cb030 con 0x7f72b80cea60 2023-10-16T13:27:59.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:59.568+0000 7f729f7fe700 1 -- 172.21.15.16:0/3386668345 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v103) v1 ==== 93+0+0 (secure 0 0 0) 0x7f72b80cb030 con 0x7f72b80cea60 2023-10-16T13:27:59.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:59.572+0000 7f72bd6d2700 1 -- 172.21.15.16:0/3386668345 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7f72a406ada0 msgr2=0x7f72a406d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:59.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:59.572+0000 7f72bd6d2700 1 --2- 172.21.15.16:0/3386668345 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7f72a406ada0 0x7f72a406d260 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f72a8007fd0 tx=0x7f72a8009040 comp rx=0 tx=0).stop 2023-10-16T13:27:59.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:59.572+0000 7f72bd6d2700 1 -- 172.21.15.16:0/3386668345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72b80cea60 msgr2=0x7f72b81f70c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:59.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:59.572+0000 7f72bd6d2700 1 --2- 172.21.15.16:0/3386668345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72b80cea60 0x7f72b81f70c0 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7f72ac001eb0 tx=0x7f72ac00c7b0 comp rx=0 tx=0).stop 2023-10-16T13:27:59.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:59.573+0000 7f72bd6d2700 1 -- 172.21.15.16:0/3386668345 shutdown_connections 2023-10-16T13:27:59.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:59.573+0000 7f72bd6d2700 1 --2- 172.21.15.16:0/3386668345 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7f72a406ada0 0x7f72a406d260 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:59.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:59.573+0000 7f72bd6d2700 1 --2- 172.21.15.16:0/3386668345 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f72b815d8d0 0x7f72b80cb310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:59.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:59.573+0000 7f72bd6d2700 1 --2- 172.21.15.16:0/3386668345 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f72b80b8fa0 0x7f72b8156d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:59.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:59.573+0000 7f72bd6d2700 1 --2- 172.21.15.16:0/3386668345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72b80cea60 0x7f72b81f70c0 unknown :-1 s=CLOSED pgs=1977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:59.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:59.573+0000 7f72bd6d2700 1 -- 172.21.15.16:0/3386668345 >> 172.21.15.16:0/3386668345 conn(0x7f72b8072330 msgr2=0x7f72b80b3760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:59.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:59.573+0000 7f72bd6d2700 1 -- 172.21.15.16:0/3386668345 shutdown_connections 2023-10-16T13:27:59.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:59.573+0000 7f72bd6d2700 1 -- 172.21.15.16:0/3386668345 wait complete. 2023-10-16T13:27:59.591 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T13:27:59.591 DEBUG:teuthology.orchestra.run.smithi016:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T13:27:59.633 DEBUG:teuthology.orchestra.run.smithi049:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T13:27:59.678 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T13:27:59.679 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T13:27:59.679 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:27:59.697 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T13:27:59.697 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:27:59.725 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:27:59.723+0000 7f180423e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T13:27:59.728 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:27:59.725+0000 7f954f950200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T13:27:59.745 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:27:59.794 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:27:59.792+0000 7f180423e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T13:27:59.802 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:27:59.800+0000 7f954f950200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T13:27:59.869 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:27:59.867+0000 7f180423e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T13:27:59.880 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:27:59.878+0000 7f954f950200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T13:27:59.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:59.911+0000 7fcdcb27d700 1 Processor -- start 2023-10-16T13:27:59.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:59.911+0000 7fcdcb27d700 1 -- start start 2023-10-16T13:27:59.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:59.912+0000 7fcdcb27d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcdc40cc8a0 0x7fcdc415e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:59.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:59.912+0000 7fcdcb27d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcdc406a7e0 0x7fcdc406ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:59.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:59.912+0000 7fcdcb27d700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdc406b170 con 0x7fcdc415b280 2023-10-16T13:27:59.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:59.912+0000 7fcdcb27d700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdc4161c30 con 0x7fcdc406a7e0 2023-10-16T13:27:59.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:59.912+0000 7fcdcb27d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdc4161f10 con 0x7fcdc40cc8a0 2023-10-16T13:27:59.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:59.912+0000 7fcdc981a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcdc406a7e0 0x7fcdc406ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:59.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:59.912+0000 7fcdc9019700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fcdc415b280 0x7fcdc41613e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38340/0 (socket says 172.21.15.16:38340) 2023-10-16T13:27:59.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:59.912+0000 7fcdc9019700 1 -- 172.21.15.16:0/830342313 learned_addr learned my addr 172.21.15.16:0/830342313 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:27:59.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:59.912+0000 7fcdc8818700 1 --2- 172.21.15.16:0/830342313 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcdc40cc8a0 0x7fcdc415e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:59.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:59.912+0000 7fcdc981a700 1 -- 172.21.15.16:0/830342313 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcdc40cc8a0 msgr2=0x7fcdc415e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:27:59.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:59.912+0000 7fcdc981a700 1 --2- 172.21.15.16:0/830342313 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcdc40cc8a0 0x7fcdc415e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:59.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:59.912+0000 7fcdc981a700 1 -- 172.21.15.16:0/830342313 >> v1:172.21.15.16:6789/0 conn(0x7fcdc415b280 legacy=0x7fcdc41613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:59.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:59.912+0000 7fcdbbfff700 1 -- 172.21.15.16:0/830342313 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4069655212 0 0) 0x7fcdc0001110 con 0x7fcdc415b280 2023-10-16T13:27:59.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:59.912+0000 7fcdc981a700 1 -- 172.21.15.16:0/830342313 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdc41621f0 con 0x7fcdc406a7e0 2023-10-16T13:27:59.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:59.913+0000 7fcdc981a700 1 --2- 172.21.15.16:0/830342313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcdc406a7e0 0x7fcdc406ac30 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc009d90 tx=0x7fcdbc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5867a3b216bac8bb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:59.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:59.913+0000 7fcdbbfff700 1 -- 172.21.15.16:0/830342313 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdbc00e070 con 0x7fcdc406a7e0 2023-10-16T13:27:59.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:59.913+0000 7fcdbbfff700 1 -- 172.21.15.16:0/830342313 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdbc00bb20 con 0x7fcdc406a7e0 2023-10-16T13:27:59.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:59.913+0000 7fcdbbfff700 1 -- 172.21.15.16:0/830342313 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdbc012430 con 0x7fcdc406a7e0 2023-10-16T13:27:59.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:59.913+0000 7fcdcb27d700 1 -- 172.21.15.16:0/830342313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcdc406a7e0 msgr2=0x7fcdc406ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:59.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:59.913+0000 7fcdcb27d700 1 --2- 172.21.15.16:0/830342313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcdc406a7e0 0x7fcdc406ac30 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc009d90 tx=0x7fcdbc00b330 comp rx=0 tx=0).stop 2023-10-16T13:27:59.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:59.913+0000 7fcdcb27d700 1 -- 172.21.15.16:0/830342313 shutdown_connections 2023-10-16T13:27:59.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:59.913+0000 7fcdcb27d700 1 --2- 172.21.15.16:0/830342313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcdc406a7e0 0x7fcdc406ac30 unknown :-1 s=CLOSED pgs=1981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:59.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:59.913+0000 7fcdcb27d700 1 --2- 172.21.15.16:0/830342313 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcdc40cc8a0 0x7fcdc415e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:59.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:59.913+0000 7fcdcb27d700 1 -- 172.21.15.16:0/830342313 >> 172.21.15.16:0/830342313 conn(0x7fcdc40b1fd0 msgr2=0x7fcdc40b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:27:59.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:59.913+0000 7fcdcb27d700 1 -- 172.21.15.16:0/830342313 shutdown_connections 2023-10-16T13:27:59.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:59.914+0000 7fcdcb27d700 1 -- 172.21.15.16:0/830342313 wait complete. 2023-10-16T13:27:59.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:59.915+0000 7fcdcb27d700 1 Processor -- start 2023-10-16T13:27:59.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:59.915+0000 7fcdcb27d700 1 -- start start 2023-10-16T13:27:59.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:59.915+0000 7fcdcb27d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcdc406a7e0 0x7fcdc4156ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:59.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:59.915+0000 7fcdcb27d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcdc40cc8a0 0x7fcdc40c4c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:59.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:59.915+0000 7fcdcb27d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcdc415b280 0x7fcdc415d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:59.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:59.915+0000 7fcdcb27d700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdc415c1b0 con 0x7fcdc415b280 2023-10-16T13:27:59.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:59.915+0000 7fcdcb27d700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdc415c490 con 0x7fcdc406a7e0 2023-10-16T13:27:59.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:59.915+0000 7fcdcb27d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdc415c770 con 0x7fcdc40cc8a0 2023-10-16T13:27:59.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:59.916+0000 7fcdc981a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcdc415b280 0x7fcdc415d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:59.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:59.916+0000 7fcdc9019700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcdc406a7e0 0x7fcdc4156ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:59.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:59.916+0000 7fcdc981a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcdc415b280 0x7fcdc415d250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49076/0 (socket says 172.21.15.16:49076) 2023-10-16T13:27:59.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:59.916+0000 7fcdc8818700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcdc40cc8a0 0x7fcdc40c4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:27:59.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:59.916+0000 7fcdc981a700 1 -- 172.21.15.16:0/790402009 learned_addr learned my addr 172.21.15.16:0/790402009 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:27:59.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:59.916+0000 7fcdc981a700 1 -- 172.21.15.16:0/790402009 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcdc40cc8a0 msgr2=0x7fcdc40c4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:59.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:59.916+0000 7fcdc981a700 1 --2- 172.21.15.16:0/790402009 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcdc40cc8a0 0x7fcdc40c4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:59.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:59.916+0000 7fcdc981a700 1 -- 172.21.15.16:0/790402009 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcdc406a7e0 msgr2=0x7fcdc4156ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:27:59.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:59.916+0000 7fcdc981a700 1 --2- 172.21.15.16:0/790402009 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcdc406a7e0 0x7fcdc4156ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:27:59.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:59.916+0000 7fcdc981a700 1 -- 172.21.15.16:0/790402009 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdbc009a70 con 0x7fcdc415b280 2023-10-16T13:27:59.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:59.916+0000 7fcdc981a700 1 --2- 172.21.15.16:0/790402009 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcdc415b280 0x7fcdc415d250 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7fcdc40d1540 tx=0x7fcdbc004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:27:59.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:59.916+0000 7fcdb9ffb700 1 -- 172.21.15.16:0/790402009 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdbc012430 con 0x7fcdc415b280 2023-10-16T13:27:59.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:59.916+0000 7fcdb9ffb700 1 -- 172.21.15.16:0/790402009 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdbc020070 con 0x7fcdc415b280 2023-10-16T13:27:59.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:59.916+0000 7fcdcb27d700 1 -- 172.21.15.16:0/790402009 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdc415ca50 con 0x7fcdc415b280 2023-10-16T13:27:59.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:59.916+0000 7fcdb9ffb700 1 -- 172.21.15.16:0/790402009 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdbc004280 con 0x7fcdc415b280 2023-10-16T13:27:59.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:59.917+0000 7fcdcb27d700 1 -- 172.21.15.16:0/790402009 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdc40c5140 con 0x7fcdc415b280 2023-10-16T13:27:59.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:59.918+0000 7fcdcb27d700 1 -- 172.21.15.16:0/790402009 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdc415cf70 con 0x7fcdc415b280 2023-10-16T13:27:59.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:59.918+0000 7fcdb9ffb700 1 -- 172.21.15.16:0/790402009 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 103) v1 ==== 89332+0+0 (secure 0 0 0) 0x7fcdbc0043e0 con 0x7fcdc415b280 2023-10-16T13:27:59.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:59.919+0000 7fcdb9ffb700 1 --2- 172.21.15.16:0/790402009 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7fcda806b020 0x7fcda806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:27:59.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:59.919+0000 7fcdc9019700 1 -- 172.21.15.16:0/790402009 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7fcda806b020 msgr2=0x7fcda806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3622553136 2023-10-16T13:27:59.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:59.919+0000 7fcdb9ffb700 1 -- 172.21.15.16:0/790402009 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 6473+0+0 (secure 0 0 0) 0x7fcdbc022070 con 0x7fcdc415b280 2023-10-16T13:27:59.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:59.919+0000 7fcdc9019700 1 --2- 172.21.15.16:0/790402009 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7fcda806b020 0x7fcda806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:27:59.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:27:59.923+0000 7fcdb9ffb700 1 -- 172.21.15.16:0/790402009 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcdc415cf70 con 0x7fcdc415b280 2023-10-16T13:27:59.951 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:27:59.949+0000 7f180423e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T13:27:59.969 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:27:59.967+0000 7f954f950200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T13:28:00.099 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:28:00.097+0000 7f180423e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T13:28:00.119 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:28:00.117+0000 7f954f950200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T13:28:00.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.120+0000 7fcdc9019700 1 -- 172.21.15.16:0/790402009 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7fcda806b020 msgr2=0x7fcda806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3622553136 2023-10-16T13:28:00.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.120+0000 7fcdc9019700 1 --2- 172.21.15.16:0/790402009 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7fcda806b020 0x7fcda806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:28:00.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.128+0000 7fcdcb27d700 1 -- 172.21.15.16:0/790402009 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcdc415cf70 con 0x7fcdc415b280 2023-10-16T13:28:00.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.129+0000 7fcdb9ffb700 1 -- 172.21.15.16:0/790402009 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v178) v1 ==== 74+0+17273 (secure 0 0 0) 0x7fcdc415cf70 con 0x7fcdc415b280 2023-10-16T13:28:00.132 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:28:00.132 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":178,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:27:56.552748+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2023-10-16T13:27:40.032163+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"176","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2023-10-16T13:27:40.523975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"177","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/472482039":"2023-10-17T13:27:56.552715+0000","172.21.15.16:0/1822527107":"2023-10-17T13:27:56.552715+0000","172.21.15.16:0/3919013455":"2023-10-17T13:27:56.552715+0000","172.21.15.16:6833/2329471558":"2023-10-17T13:27:56.552715+0000","172.21.15.16:6832/2329471558":"2023-10-17T13:27:56.552715+0000","172.21.15.16:0/508475708":"2023-10-17T13:27:56.552715+0000","172.21.15.16:0/353895884":"2023-10-17T13:27:56.552715+0000","172.21.15.16:0/2733317665":"2023-10-17T13:27:56.552715+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:28:00.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.131+0000 7fcdcb27d700 1 -- 172.21.15.16:0/790402009 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7fcda806b020 msgr2=0x7fcda806d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:28:00.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.131+0000 7fcdcb27d700 1 --2- 172.21.15.16:0/790402009 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7fcda806b020 0x7fcda806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:00.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.131+0000 7fcdcb27d700 1 -- 172.21.15.16:0/790402009 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcdc415b280 msgr2=0x7fcdc415d250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:00.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.131+0000 7fcdcb27d700 1 --2- 172.21.15.16:0/790402009 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcdc415b280 0x7fcdc415d250 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7fcdc40d1540 tx=0x7fcdbc004ad0 comp rx=0 tx=0).stop 2023-10-16T13:28:00.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.131+0000 7fcdcb27d700 1 -- 172.21.15.16:0/790402009 shutdown_connections 2023-10-16T13:28:00.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.131+0000 7fcdcb27d700 1 --2- 172.21.15.16:0/790402009 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7fcda806b020 0x7fcda806d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:00.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.131+0000 7fcdcb27d700 1 --2- 172.21.15.16:0/790402009 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcdc415b280 0x7fcdc415d250 unknown :-1 s=CLOSED pgs=1982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:00.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.131+0000 7fcdcb27d700 1 --2- 172.21.15.16:0/790402009 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcdc406a7e0 0x7fcdc4156ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:00.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.131+0000 7fcdcb27d700 1 --2- 172.21.15.16:0/790402009 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcdc40cc8a0 0x7fcdc40c4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:00.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.131+0000 7fcdcb27d700 1 -- 172.21.15.16:0/790402009 >> 172.21.15.16:0/790402009 conn(0x7fcdc40b1fd0 msgr2=0x7fcdc40d0270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:00.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.131+0000 7fcdcb27d700 1 -- 172.21.15.16:0/790402009 shutdown_connections 2023-10-16T13:28:00.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.132+0000 7fcdcb27d700 1 -- 172.21.15.16:0/790402009 wait complete. 2023-10-16T13:28:00.147 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 21, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T13:27:40.032163+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '176', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 22, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T13:27:40.523975+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '177', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T13:28:00.148 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T13:28:00.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.315+0000 7f076d882700 1 Processor -- start 2023-10-16T13:28:00.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.315+0000 7f076d882700 1 -- start start 2023-10-16T13:28:00.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.316+0000 7f076d882700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f07680d2430 0x7f07680d1520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:00.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.316+0000 7f076d882700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07680d4540 con 0x7f07680d4060 2023-10-16T13:28:00.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.316+0000 7f076d882700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07680ce7e0 con 0x7f076815da10 2023-10-16T13:28:00.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.316+0000 7f076d882700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07680cf9b0 con 0x7f07680d2430 2023-10-16T13:28:00.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.316+0000 7f07667fc700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f07680d4060 0x7f07680d1d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38374/0 (socket says 172.21.15.16:38374) 2023-10-16T13:28:00.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.316+0000 7f07667fc700 1 -- 172.21.15.16:0/673360524 learned_addr learned my addr 172.21.15.16:0/673360524 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:28:00.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.316+0000 7f07677fe700 1 --2- 172.21.15.16:0/673360524 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f07680d2430 0x7f07680d1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:00.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.316+0000 7f0765ffb700 1 -- 172.21.15.16:0/673360524 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3013452936 0 0) 0x7f07680d4540 con 0x7f07680d4060 2023-10-16T13:28:00.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.316+0000 7f0765ffb700 1 -- 172.21.15.16:0/673360524 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07500035b0 con 0x7f07680d4060 2023-10-16T13:28:00.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.316+0000 7f0765ffb700 1 -- 172.21.15.16:0/673360524 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 97943880 0 0) 0x7f07680ce7e0 con 0x7f076815da10 2023-10-16T13:28:00.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.316+0000 7f0765ffb700 1 -- 172.21.15.16:0/673360524 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07680d4540 con 0x7f076815da10 2023-10-16T13:28:00.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.316+0000 7f07677fe700 1 -- 172.21.15.16:0/673360524 >> v1:172.21.15.49:6789/0 conn(0x7f076815da10 legacy=0x7f07680cf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:00.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.316+0000 7f07677fe700 1 -- 172.21.15.16:0/673360524 >> v1:172.21.15.16:6789/0 conn(0x7f07680d4060 legacy=0x7f07680d1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:00.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.316+0000 7f07677fe700 1 -- 172.21.15.16:0/673360524 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07680ceac0 con 0x7f07680d2430 2023-10-16T13:28:00.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.317+0000 7f07677fe700 1 --2- 172.21.15.16:0/673360524 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f07680d2430 0x7f07680d1520 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f0760009dc0 tx=0x7f076000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69b9e3c32f4a540f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:00.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.317+0000 7f0765ffb700 1 -- 172.21.15.16:0/673360524 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f076000e070 con 0x7f07680d2430 2023-10-16T13:28:00.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.317+0000 7f0765ffb700 1 -- 172.21.15.16:0/673360524 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f076000bb20 con 0x7f07680d2430 2023-10-16T13:28:00.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.317+0000 7f0765ffb700 1 -- 172.21.15.16:0/673360524 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f076000be90 con 0x7f07680d2430 2023-10-16T13:28:00.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.317+0000 7f076d882700 1 -- 172.21.15.16:0/673360524 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f07680d2430 msgr2=0x7f07680d1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:00.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.317+0000 7f076d882700 1 --2- 172.21.15.16:0/673360524 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f07680d2430 0x7f07680d1520 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f0760009dc0 tx=0x7f076000b330 comp rx=0 tx=0).stop 2023-10-16T13:28:00.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.317+0000 7f076d882700 1 -- 172.21.15.16:0/673360524 shutdown_connections 2023-10-16T13:28:00.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.317+0000 7f076d882700 1 --2- 172.21.15.16:0/673360524 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f07680d2430 0x7f07680d1520 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:00.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.317+0000 7f076d882700 1 -- 172.21.15.16:0/673360524 >> 172.21.15.16:0/673360524 conn(0x7f076809c9f0 msgr2=0x7f07680d3740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:00.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.317+0000 7f076d882700 1 -- 172.21.15.16:0/673360524 shutdown_connections 2023-10-16T13:28:00.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.317+0000 7f076d882700 1 -- 172.21.15.16:0/673360524 wait complete. 2023-10-16T13:28:00.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.319+0000 7f076d882700 1 Processor -- start 2023-10-16T13:28:00.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.319+0000 7f076d882700 1 -- start start 2023-10-16T13:28:00.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.319+0000 7f076d882700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f07680d2430 0x7f0768157c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:00.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.319+0000 7f076d882700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07680d4060 0x7f07681f3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:00.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.319+0000 7f076d882700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f076815da10 0x7f076815b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:00.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.319+0000 7f076d882700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07680cad10 con 0x7f07680d4060 2023-10-16T13:28:00.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.319+0000 7f076d882700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07680caff0 con 0x7f07680d2430 2023-10-16T13:28:00.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.319+0000 7f076d882700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07680cb2d0 con 0x7f076815da10 2023-10-16T13:28:00.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.320+0000 7f07667fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07680d4060 0x7f07681f3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:00.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.320+0000 7f07667fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07680d4060 0x7f07681f3140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49106/0 (socket says 172.21.15.16:49106) 2023-10-16T13:28:00.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.320+0000 7f07667fc700 1 -- 172.21.15.16:0/40028230 learned_addr learned my addr 172.21.15.16:0/40028230 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:00.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.320+0000 7f0766ffd700 1 --2- 172.21.15.16:0/40028230 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f07680d2430 0x7f0768157c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:00.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.320+0000 7f07677fe700 1 --2- 172.21.15.16:0/40028230 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f076815da10 0x7f076815b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:00.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.320+0000 7f07667fc700 1 -- 172.21.15.16:0/40028230 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f076815da10 msgr2=0x7f076815b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:00.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.320+0000 7f07667fc700 1 --2- 172.21.15.16:0/40028230 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f076815da10 0x7f076815b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:00.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.320+0000 7f07667fc700 1 -- 172.21.15.16:0/40028230 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f07680d2430 msgr2=0x7f0768157c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:00.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.320+0000 7f07667fc700 1 --2- 172.21.15.16:0/40028230 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f07680d2430 0x7f0768157c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:00.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.320+0000 7f07667fc700 1 -- 172.21.15.16:0/40028230 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0760009a70 con 0x7f07680d4060 2023-10-16T13:28:00.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.320+0000 7f07667fc700 1 --2- 172.21.15.16:0/40028230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07680d4060 0x7f07681f3140 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7f075c00c500 tx=0x7f075c00c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:00.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.320+0000 7f074ffff700 1 -- 172.21.15.16:0/40028230 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f075c005d70 con 0x7f07680d4060 2023-10-16T13:28:00.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.320+0000 7f074ffff700 1 -- 172.21.15.16:0/40028230 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f075c0070d0 con 0x7f07680d4060 2023-10-16T13:28:00.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.320+0000 7f076d882700 1 -- 172.21.15.16:0/40028230 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07680cb5b0 con 0x7f07680d4060 2023-10-16T13:28:00.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.320+0000 7f074ffff700 1 -- 172.21.15.16:0/40028230 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f075c007480 con 0x7f07680d4060 2023-10-16T13:28:00.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.320+0000 7f076d882700 1 -- 172.21.15.16:0/40028230 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0768158160 con 0x7f07680d4060 2023-10-16T13:28:00.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.321+0000 7f076d882700 1 -- 172.21.15.16:0/40028230 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07680cbad0 con 0x7f07680d4060 2023-10-16T13:28:00.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.322+0000 7f074ffff700 1 -- 172.21.15.16:0/40028230 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 103) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f075c0075e0 con 0x7f07680d4060 2023-10-16T13:28:00.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.322+0000 7f074ffff700 1 --2- 172.21.15.16:0/40028230 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7f075406b020 0x7f075406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:00.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.323+0000 7f0766ffd700 1 -- 172.21.15.16:0/40028230 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7f075406b020 msgr2=0x7f075406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3622553136 2023-10-16T13:28:00.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.323+0000 7f074ffff700 1 -- 172.21.15.16:0/40028230 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 6473+0+0 (secure 0 0 0) 0x7f075c088e50 con 0x7f07680d4060 2023-10-16T13:28:00.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.323+0000 7f0766ffd700 1 --2- 172.21.15.16:0/40028230 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7f075406b020 0x7f075406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:28:00.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.327+0000 7f074ffff700 1 -- 172.21.15.16:0/40028230 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f07680cbad0 con 0x7f07680d4060 2023-10-16T13:28:00.440 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:28:00.438+0000 7f180423e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T13:28:00.473 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:28:00.471+0000 7f954f950200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T13:28:00.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.524+0000 7f0766ffd700 1 -- 172.21.15.16:0/40028230 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7f075406b020 msgr2=0x7f075406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3622553136 2023-10-16T13:28:00.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.524+0000 7f0766ffd700 1 --2- 172.21.15.16:0/40028230 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7f075406b020 0x7f075406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:28:00.542 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:28:00.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.540+0000 7f076d882700 1 -- 172.21.15.16:0/40028230 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f07680cbad0 con 0x7f07680d4060 2023-10-16T13:28:00.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.540+0000 7f074ffff700 1 -- 172.21.15.16:0/40028230 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v178) v1 ==== 93+0+10 (secure 0 0 0) 0x7f07680cbad0 con 0x7f07680d4060 2023-10-16T13:28:00.544 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T13:28:00.546 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:28:00.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.543+0000 7f076d882700 1 -- 172.21.15.16:0/40028230 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7f075406b020 msgr2=0x7f075406d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:28:00.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.543+0000 7f076d882700 1 --2- 172.21.15.16:0/40028230 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7f075406b020 0x7f075406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:00.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.543+0000 7f076d882700 1 -- 172.21.15.16:0/40028230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07680d4060 msgr2=0x7f07681f3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:00.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.543+0000 7f076d882700 1 --2- 172.21.15.16:0/40028230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07680d4060 0x7f07681f3140 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7f075c00c500 tx=0x7f075c00c810 comp rx=0 tx=0).stop 2023-10-16T13:28:00.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.543+0000 7f076d882700 1 -- 172.21.15.16:0/40028230 shutdown_connections 2023-10-16T13:28:00.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.543+0000 7f076d882700 1 --2- 172.21.15.16:0/40028230 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7f075406b020 0x7f075406d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:00.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.543+0000 7f076d882700 1 --2- 172.21.15.16:0/40028230 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f076815da10 0x7f076815b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:00.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.543+0000 7f076d882700 1 --2- 172.21.15.16:0/40028230 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f07680d2430 0x7f0768157c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:00.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.543+0000 7f076d882700 1 --2- 172.21.15.16:0/40028230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07680d4060 0x7f07681f3140 unknown :-1 s=CLOSED pgs=1984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:00.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.543+0000 7f076d882700 1 -- 172.21.15.16:0/40028230 >> 172.21.15.16:0/40028230 conn(0x7f076809c9f0 msgr2=0x7f07680d0150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:00.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.543+0000 7f076d882700 1 -- 172.21.15.16:0/40028230 shutdown_connections 2023-10-16T13:28:00.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.543+0000 7f076d882700 1 -- 172.21.15.16:0/40028230 wait complete. 2023-10-16T13:28:00.560 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:28:00.558+0000 7f180423e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T13:28:00.565 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T13:28:00.606 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:28:00.604+0000 7f954f950200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T13:28:00.640 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:28:00.637+0000 7f180423e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T13:28:00.686 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:28:00.684+0000 7f954f950200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T13:28:00.724 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:28:00.722+0000 7f180423e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T13:28:00.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.732+0000 7f85195da700 1 Processor -- start 2023-10-16T13:28:00.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.732+0000 7f85195da700 1 -- start start 2023-10-16T13:28:00.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.732+0000 7f85195da700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f85140b8fa0 0x7f85140b2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:00.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.732+0000 7f85195da700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f85140ce350 0x7f85140ce780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:00.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.732+0000 7f85195da700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85140cecc0 con 0x7f851415d8d0 2023-10-16T13:28:00.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.732+0000 7f85195da700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85140d0bb0 con 0x7f85140b8fa0 2023-10-16T13:28:00.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.732+0000 7f85195da700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f851415dcb0 con 0x7f85140ce350 2023-10-16T13:28:00.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.733+0000 7f85127fc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f85140b8fa0 0x7f85140b2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:00.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.733+0000 7f85127fc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f85140b8fa0 0x7f85140b2830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49116/0 (socket says 172.21.15.16:49116) 2023-10-16T13:28:00.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.733+0000 7f85127fc700 1 -- 172.21.15.16:0/395238947 learned_addr learned my addr 172.21.15.16:0/395238947 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:00.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.733+0000 7f85137fe700 1 --2- 172.21.15.16:0/395238947 >> v2:172.21.15.49:3300/0 conn(0x7f85140ce350 0x7f85140ce780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:00.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.733+0000 7f85127fc700 1 -- 172.21.15.16:0/395238947 >> v2:172.21.15.49:3300/0 conn(0x7f85140ce350 msgr2=0x7f85140ce780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:28:00.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.733+0000 7f85127fc700 1 --2- 172.21.15.16:0/395238947 >> v2:172.21.15.49:3300/0 conn(0x7f85140ce350 0x7f85140ce780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:00.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.733+0000 7f85127fc700 1 -- 172.21.15.16:0/395238947 >> v1:172.21.15.49:6789/0 conn(0x7f851415d8d0 legacy=0x7f85140b6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:00.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.733+0000 7f85127fc700 1 -- 172.21.15.16:0/395238947 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85140b2f30 con 0x7f85140b8fa0 2023-10-16T13:28:00.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.733+0000 7f85127fc700 1 --2- 172.21.15.16:0/395238947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85140b8fa0 0x7f85140b2830 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7f85140d3c80 tx=0x7f8508009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70368a1dd8730205 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:00.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.733+0000 7f8511ffb700 1 -- 172.21.15.16:0/395238947 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f850800b870 con 0x7f85140b8fa0 2023-10-16T13:28:00.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.733+0000 7f8511ffb700 1 -- 172.21.15.16:0/395238947 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f850800b9d0 con 0x7f85140b8fa0 2023-10-16T13:28:00.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.733+0000 7f8511ffb700 1 -- 172.21.15.16:0/395238947 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f850800bd60 con 0x7f85140b8fa0 2023-10-16T13:28:00.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.734+0000 7f85195da700 1 -- 172.21.15.16:0/395238947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85140b8fa0 msgr2=0x7f85140b2830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:00.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.734+0000 7f85195da700 1 --2- 172.21.15.16:0/395238947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85140b8fa0 0x7f85140b2830 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7f85140d3c80 tx=0x7f8508009bd0 comp rx=0 tx=0).stop 2023-10-16T13:28:00.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.734+0000 7f85195da700 1 -- 172.21.15.16:0/395238947 shutdown_connections 2023-10-16T13:28:00.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.734+0000 7f85195da700 1 --2- 172.21.15.16:0/395238947 >> v2:172.21.15.49:3300/0 conn(0x7f85140ce350 0x7f85140ce780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:00.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.734+0000 7f85195da700 1 --2- 172.21.15.16:0/395238947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85140b8fa0 0x7f85140b2830 unknown :-1 s=CLOSED pgs=1985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:00.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.734+0000 7f85195da700 1 -- 172.21.15.16:0/395238947 >> 172.21.15.16:0/395238947 conn(0x7f8514072330 msgr2=0x7f8514072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:00.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.734+0000 7f85195da700 1 -- 172.21.15.16:0/395238947 shutdown_connections 2023-10-16T13:28:00.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.734+0000 7f85195da700 1 -- 172.21.15.16:0/395238947 wait complete. 2023-10-16T13:28:00.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.735+0000 7f85195da700 1 Processor -- start 2023-10-16T13:28:00.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.735+0000 7f85195da700 1 -- start start 2023-10-16T13:28:00.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.736+0000 7f85195da700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f85140b8fa0 0x7f8514152ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:00.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.736+0000 7f85195da700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85140ce350 0x7f85140c80c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:00.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.736+0000 7f85195da700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f851415d8d0 0x7f85140c8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:00.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.736+0000 7f85195da700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85140cb110 con 0x7f85140ce350 2023-10-16T13:28:00.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.736+0000 7f85195da700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8514153000 con 0x7f851415d8d0 2023-10-16T13:28:00.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.736+0000 7f85195da700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85140cb360 con 0x7f85140b8fa0 2023-10-16T13:28:00.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.736+0000 7f85127fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85140ce350 0x7f85140c80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:00.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.736+0000 7f85127fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85140ce350 0x7f85140c80c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49122/0 (socket says 172.21.15.16:49122) 2023-10-16T13:28:00.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.736+0000 7f85127fc700 1 -- 172.21.15.16:0/3399922736 learned_addr learned my addr 172.21.15.16:0/3399922736 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:00.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.736+0000 7f85137fe700 1 --2- 172.21.15.16:0/3399922736 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f851415d8d0 0x7f85140c8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:00.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.736+0000 7f8512ffd700 1 --2- 172.21.15.16:0/3399922736 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f85140b8fa0 0x7f8514152ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:00.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.736+0000 7f85127fc700 1 -- 172.21.15.16:0/3399922736 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f85140b8fa0 msgr2=0x7f8514152ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:00.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.736+0000 7f85127fc700 1 --2- 172.21.15.16:0/3399922736 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f85140b8fa0 0x7f8514152ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:00.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.736+0000 7f85127fc700 1 -- 172.21.15.16:0/3399922736 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f851415d8d0 msgr2=0x7f85140c8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:00.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.736+0000 7f85127fc700 1 --2- 172.21.15.16:0/3399922736 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f851415d8d0 0x7f85140c8600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:00.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.736+0000 7f85127fc700 1 -- 172.21.15.16:0/3399922736 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8508009550 con 0x7f85140ce350 2023-10-16T13:28:00.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.737+0000 7f85127fc700 1 --2- 172.21.15.16:0/3399922736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85140ce350 0x7f85140c80c0 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 crypto rx=0x7f8508000c00 tx=0x7f8508004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:00.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.737+0000 7f84fb7fe700 1 -- 172.21.15.16:0/3399922736 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8508004310 con 0x7f85140ce350 2023-10-16T13:28:00.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.737+0000 7f84fb7fe700 1 -- 172.21.15.16:0/3399922736 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f850801e070 con 0x7f85140ce350 2023-10-16T13:28:00.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.737+0000 7f85195da700 1 -- 172.21.15.16:0/3399922736 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85140cb640 con 0x7f85140ce350 2023-10-16T13:28:00.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.737+0000 7f84fb7fe700 1 -- 172.21.15.16:0/3399922736 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85080036a0 con 0x7f85140ce350 2023-10-16T13:28:00.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.737+0000 7f85195da700 1 -- 172.21.15.16:0/3399922736 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85140cbc00 con 0x7f85140ce350 2023-10-16T13:28:00.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.738+0000 7f85195da700 1 -- 172.21.15.16:0/3399922736 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f851404ebb0 con 0x7f85140ce350 2023-10-16T13:28:00.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.739+0000 7f84fb7fe700 1 -- 172.21.15.16:0/3399922736 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 103) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f8508020030 con 0x7f85140ce350 2023-10-16T13:28:00.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.739+0000 7f84fb7fe700 1 --2- 172.21.15.16:0/3399922736 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7f850006aff0 0x7f850006d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:00.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.739+0000 7f8512ffd700 1 -- 172.21.15.16:0/3399922736 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7f850006aff0 msgr2=0x7f850006d4b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3622553136 2023-10-16T13:28:00.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.739+0000 7f84fb7fe700 1 -- 172.21.15.16:0/3399922736 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 6473+0+0 (secure 0 0 0) 0x7f850808a8f0 con 0x7f85140ce350 2023-10-16T13:28:00.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.739+0000 7f8512ffd700 1 --2- 172.21.15.16:0/3399922736 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7f850006aff0 0x7f850006d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:28:00.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.743+0000 7f84fb7fe700 1 -- 172.21.15.16:0/3399922736 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f85080555a0 con 0x7f85140ce350 2023-10-16T13:28:00.775 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:28:00.772+0000 7f954f950200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T13:28:00.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.940+0000 7f8512ffd700 1 -- 172.21.15.16:0/3399922736 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7f850006aff0 msgr2=0x7f850006d4b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3622553136 2023-10-16T13:28:00.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.940+0000 7f8512ffd700 1 --2- 172.21.15.16:0/3399922736 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7f850006aff0 0x7f850006d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:28:00.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.946+0000 7f85195da700 1 -- 172.21.15.16:0/3399922736 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f85140b2b50 con 0x7f85140ce350 2023-10-16T13:28:00.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.947+0000 7f84fb7fe700 1 -- 172.21.15.16:0/3399922736 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v178) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8508003800 con 0x7f85140ce350 2023-10-16T13:28:00.949 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-10-16T13:28:00.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.949+0000 7f85195da700 1 -- 172.21.15.16:0/3399922736 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7f850006aff0 msgr2=0x7f850006d4b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:28:00.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.949+0000 7f85195da700 1 --2- 172.21.15.16:0/3399922736 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7f850006aff0 0x7f850006d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:00.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.949+0000 7f85195da700 1 -- 172.21.15.16:0/3399922736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85140ce350 msgr2=0x7f85140c80c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:00.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.949+0000 7f85195da700 1 --2- 172.21.15.16:0/3399922736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85140ce350 0x7f85140c80c0 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 crypto rx=0x7f8508000c00 tx=0x7f8508004750 comp rx=0 tx=0).stop 2023-10-16T13:28:00.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.949+0000 7f85195da700 1 -- 172.21.15.16:0/3399922736 shutdown_connections 2023-10-16T13:28:00.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.949+0000 7f85195da700 1 --2- 172.21.15.16:0/3399922736 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7f850006aff0 0x7f850006d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:00.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.949+0000 7f85195da700 1 --2- 172.21.15.16:0/3399922736 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f851415d8d0 0x7f85140c8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:00.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.949+0000 7f85195da700 1 --2- 172.21.15.16:0/3399922736 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f85140b8fa0 0x7f8514152ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:00.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.949+0000 7f85195da700 1 --2- 172.21.15.16:0/3399922736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85140ce350 0x7f85140c80c0 unknown :-1 s=CLOSED pgs=1986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:00.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.949+0000 7f85195da700 1 -- 172.21.15.16:0/3399922736 >> 172.21.15.16:0/3399922736 conn(0x7f8514072330 msgr2=0x7f85140d2ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:00.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.949+0000 7f85195da700 1 -- 172.21.15.16:0/3399922736 shutdown_connections 2023-10-16T13:28:00.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:00.949+0000 7f85195da700 1 -- 172.21.15.16:0/3399922736 wait complete. 2023-10-16T13:28:00.964 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T13:28:01.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.091+0000 7fdcd104f700 1 Processor -- start 2023-10-16T13:28:01.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.091+0000 7fdcd104f700 1 -- start start 2023-10-16T13:28:01.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.092+0000 7fdcd104f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdccc0ce5d0 0x7fdccc0cc190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:01.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.092+0000 7fdcd104f700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fdccc15f060 0x7fdccc0cc6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:01.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.092+0000 7fdcd104f700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdccc0b2440 con 0x7fdccc15ec80 2023-10-16T13:28:01.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.092+0000 7fdcd104f700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdccc0b2c50 con 0x7fdccc15f060 2023-10-16T13:28:01.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.092+0000 7fdcd104f700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdccc0d00e0 con 0x7fdccc0ce5d0 2023-10-16T13:28:01.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.092+0000 7fdcca59c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdccc0ce5d0 0x7fdccc0cc190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:01.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.092+0000 7fdccb59e700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fdccc15f060 0x7fdccc0cc6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:01.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.092+0000 7fdcca59c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdccc0ce5d0 0x7fdccc0cc190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:37408/0 (socket says 172.21.15.16:37408) 2023-10-16T13:28:01.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.092+0000 7fdccad9d700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7fdccc15ec80 0x7fdccc0cdec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:45604/0 (socket says 172.21.15.16:45604) 2023-10-16T13:28:01.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.092+0000 7fdccad9d700 1 -- 172.21.15.16:0/110730325 learned_addr learned my addr 172.21.15.16:0/110730325 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:28:01.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.092+0000 7fdcc9d9b700 1 -- 172.21.15.16:0/110730325 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2661986658 0 0) 0x7fdccc0b2440 con 0x7fdccc15ec80 2023-10-16T13:28:01.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.092+0000 7fdcca59c700 1 -- 172.21.15.16:0/110730325 >> v2:172.21.15.49:3300/0 conn(0x7fdccc15f060 msgr2=0x7fdccc0cc6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:28:01.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.092+0000 7fdcca59c700 1 --2- 172.21.15.16:0/110730325 >> v2:172.21.15.49:3300/0 conn(0x7fdccc15f060 0x7fdccc0cc6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:01.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.092+0000 7fdcca59c700 1 -- 172.21.15.16:0/110730325 >> v1:172.21.15.49:6789/0 conn(0x7fdccc15ec80 legacy=0x7fdccc0cdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:01.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.092+0000 7fdcca59c700 1 -- 172.21.15.16:0/110730325 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdccc0cef60 con 0x7fdccc0ce5d0 2023-10-16T13:28:01.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.093+0000 7fdcca59c700 1 --2- 172.21.15.16:0/110730325 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdccc0ce5d0 0x7fdccc0cc190 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fdcc0009840 tx=0x7fdcc0009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1fccd20680ae6ba6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:01.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.093+0000 7fdcc9d9b700 1 -- 172.21.15.16:0/110730325 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc0011830 con 0x7fdccc0ce5d0 2023-10-16T13:28:01.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.093+0000 7fdcc9d9b700 1 -- 172.21.15.16:0/110730325 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcc0011990 con 0x7fdccc0ce5d0 2023-10-16T13:28:01.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.093+0000 7fdcc9d9b700 1 -- 172.21.15.16:0/110730325 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc0011d20 con 0x7fdccc0ce5d0 2023-10-16T13:28:01.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.093+0000 7fdcd104f700 1 -- 172.21.15.16:0/110730325 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdccc0ce5d0 msgr2=0x7fdccc0cc190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:01.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.093+0000 7fdcd104f700 1 --2- 172.21.15.16:0/110730325 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdccc0ce5d0 0x7fdccc0cc190 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fdcc0009840 tx=0x7fdcc0009b50 comp rx=0 tx=0).stop 2023-10-16T13:28:01.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.093+0000 7fdcd104f700 1 -- 172.21.15.16:0/110730325 shutdown_connections 2023-10-16T13:28:01.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.093+0000 7fdcd104f700 1 --2- 172.21.15.16:0/110730325 >> v2:172.21.15.49:3300/0 conn(0x7fdccc15f060 0x7fdccc0cc6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:01.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.093+0000 7fdcd104f700 1 --2- 172.21.15.16:0/110730325 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdccc0ce5d0 0x7fdccc0cc190 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:01.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.093+0000 7fdcd104f700 1 -- 172.21.15.16:0/110730325 >> 172.21.15.16:0/110730325 conn(0x7fdccc0d15f0 msgr2=0x7fdccc0d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:01.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.093+0000 7fdcd104f700 1 -- 172.21.15.16:0/110730325 shutdown_connections 2023-10-16T13:28:01.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.093+0000 7fdcd104f700 1 -- 172.21.15.16:0/110730325 wait complete. 2023-10-16T13:28:01.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.095+0000 7fdcd104f700 1 Processor -- start 2023-10-16T13:28:01.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.095+0000 7fdcd104f700 1 -- start start 2023-10-16T13:28:01.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.096+0000 7fdcd104f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdccc0ce5d0 0x7fdccc0cb0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:01.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.096+0000 7fdcd104f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdccc15ec80 0x7fdccc1f4100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:01.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.096+0000 7fdcd104f700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdccc15f060 0x7fdccc1f4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:01.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.096+0000 7fdcd104f700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdccc15bbd0 con 0x7fdccc15ec80 2023-10-16T13:28:01.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.096+0000 7fdcd104f700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdccc0c77b0 con 0x7fdccc15f060 2023-10-16T13:28:01.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.096+0000 7fdcd104f700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdccc1e4c50 con 0x7fdccc0ce5d0 2023-10-16T13:28:01.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.096+0000 7fdccad9d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdccc0ce5d0 0x7fdccc0cb0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:01.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.096+0000 7fdccb59e700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdccc15f060 0x7fdccc1f4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:01.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.096+0000 7fdcca59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdccc15ec80 0x7fdccc1f4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:01.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.096+0000 7fdccb59e700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdccc15f060 0x7fdccc1f4640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:48218/0 (socket says 172.21.15.16:48218) 2023-10-16T13:28:01.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.096+0000 7fdccb59e700 1 -- 172.21.15.16:0/2754077308 learned_addr learned my addr 172.21.15.16:0/2754077308 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:01.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.096+0000 7fdccad9d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdccc0ce5d0 0x7fdccc0cb0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:37412/0 (socket says 172.21.15.16:37412) 2023-10-16T13:28:01.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.096+0000 7fdccb59e700 1 -- 172.21.15.16:0/2754077308 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdccc0ce5d0 msgr2=0x7fdccc0cb0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:01.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.096+0000 7fdccb59e700 1 --2- 172.21.15.16:0/2754077308 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdccc0ce5d0 0x7fdccc0cb0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:01.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.096+0000 7fdccb59e700 1 -- 172.21.15.16:0/2754077308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdccc15ec80 msgr2=0x7fdccc1f4100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:01.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.096+0000 7fdccb59e700 1 --2- 172.21.15.16:0/2754077308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdccc15ec80 0x7fdccc1f4100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:01.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.096+0000 7fdccb59e700 1 -- 172.21.15.16:0/2754077308 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcc00094a0 con 0x7fdccc15f060 2023-10-16T13:28:01.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.096+0000 7fdcca59c700 1 --2- 172.21.15.16:0/2754077308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdccc15ec80 0x7fdccc1f4100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:28:01.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.096+0000 7fdccb59e700 1 --2- 172.21.15.16:0/2754077308 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdccc15f060 0x7fdccc1f4640 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fdcc400efc0 tx=0x7fdcc4009f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:01.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.097+0000 7fdcb37fe700 1 -- 172.21.15.16:0/2754077308 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc4009b20 con 0x7fdccc15f060 2023-10-16T13:28:01.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.097+0000 7fdcb37fe700 1 -- 172.21.15.16:0/2754077308 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcc400f040 con 0x7fdccc15f060 2023-10-16T13:28:01.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.097+0000 7fdcd104f700 1 -- 172.21.15.16:0/2754077308 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdccc1e4f30 con 0x7fdccc15f060 2023-10-16T13:28:01.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.097+0000 7fdcb37fe700 1 -- 172.21.15.16:0/2754077308 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc400cc90 con 0x7fdccc15f060 2023-10-16T13:28:01.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.097+0000 7fdcd104f700 1 -- 172.21.15.16:0/2754077308 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdccc154d30 con 0x7fdccc15f060 2023-10-16T13:28:01.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.098+0000 7fdcd104f700 1 -- 172.21.15.16:0/2754077308 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdccc0c7b40 con 0x7fdccc15f060 2023-10-16T13:28:01.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.098+0000 7fdcb37fe700 1 -- 172.21.15.16:0/2754077308 <== mon.1 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 103) v1 ==== 89332+0+0 (secure 0 0 0) 0x7fdcc4014480 con 0x7fdccc15f060 2023-10-16T13:28:01.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.099+0000 7fdcb37fe700 1 --2- 172.21.15.16:0/2754077308 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7fdcb806b020 0x7fdcb806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:01.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.099+0000 7fdccad9d700 1 -- 172.21.15.16:0/2754077308 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7fdcb806b020 msgr2=0x7fdcb806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3622553136 2023-10-16T13:28:01.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.099+0000 7fdcb37fe700 1 -- 172.21.15.16:0/2754077308 <== mon.1 v2:172.21.15.49:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 6473+0+0 (secure 0 0 0) 0x7fdcc408a5d0 con 0x7fdccc15f060 2023-10-16T13:28:01.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.099+0000 7fdccad9d700 1 --2- 172.21.15.16:0/2754077308 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7fdcb806b020 0x7fdcb806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:28:01.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.103+0000 7fdcb37fe700 1 -- 172.21.15.16:0/2754077308 <== mon.1 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdccc0c7b40 con 0x7fdccc15f060 2023-10-16T13:28:01.112 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:28:01.109+0000 7f180423e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T13:28:01.174 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:28:01.172+0000 7f954f950200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T13:28:01.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.300+0000 7fdccad9d700 1 -- 172.21.15.16:0/2754077308 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7fdcb806b020 msgr2=0x7fdcb806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3622553136 2023-10-16T13:28:01.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.300+0000 7fdccad9d700 1 --2- 172.21.15.16:0/2754077308 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7fdcb806b020 0x7fdcb806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:28:01.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.308+0000 7fdcd104f700 1 -- 172.21.15.16:0/2754077308 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fdccc0c7b40 con 0x7fdccc15f060 2023-10-16T13:28:01.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.308+0000 7fdcb37fe700 1 -- 172.21.15.16:0/2754077308 <== mon.1 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v178) v1 ==== 100+0+11 (secure 0 0 0) 0x7fdccc0c7b40 con 0x7fdccc15f060 2023-10-16T13:28:01.311 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-10-16T13:28:01.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.311+0000 7fdcd104f700 1 -- 172.21.15.16:0/2754077308 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7fdcb806b020 msgr2=0x7fdcb806d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:28:01.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.311+0000 7fdcd104f700 1 --2- 172.21.15.16:0/2754077308 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7fdcb806b020 0x7fdcb806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:01.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.311+0000 7fdcd104f700 1 -- 172.21.15.16:0/2754077308 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdccc15f060 msgr2=0x7fdccc1f4640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:01.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.311+0000 7fdcd104f700 1 --2- 172.21.15.16:0/2754077308 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdccc15f060 0x7fdccc1f4640 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fdcc400efc0 tx=0x7fdcc4009f60 comp rx=0 tx=0).stop 2023-10-16T13:28:01.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.311+0000 7fdcd104f700 1 -- 172.21.15.16:0/2754077308 shutdown_connections 2023-10-16T13:28:01.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.311+0000 7fdcd104f700 1 --2- 172.21.15.16:0/2754077308 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7fdcb806b020 0x7fdcb806d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:01.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.311+0000 7fdcd104f700 1 --2- 172.21.15.16:0/2754077308 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdccc15f060 0x7fdccc1f4640 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:01.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.311+0000 7fdcd104f700 1 --2- 172.21.15.16:0/2754077308 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdccc0ce5d0 0x7fdccc0cb0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:01.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.311+0000 7fdcd104f700 1 --2- 172.21.15.16:0/2754077308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdccc15ec80 0x7fdccc1f4100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:01.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.311+0000 7fdcd104f700 1 -- 172.21.15.16:0/2754077308 >> 172.21.15.16:0/2754077308 conn(0x7fdccc0d15f0 msgr2=0x7fdccc0cf1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:01.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.311+0000 7fdcd104f700 1 -- 172.21.15.16:0/2754077308 shutdown_connections 2023-10-16T13:28:01.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.311+0000 7fdcd104f700 1 -- 172.21.15.16:0/2754077308 wait complete. 2023-10-16T13:28:01.326 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T13:28:01.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.451+0000 7fa51edec700 1 Processor -- start 2023-10-16T13:28:01.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.451+0000 7fa51edec700 1 -- start start 2023-10-16T13:28:01.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.452+0000 7fa51edec700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa51815b860 0x7fa51806a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:01.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.452+0000 7fa51edec700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5180d11c0 con 0x7fa5180b4490 2023-10-16T13:28:01.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.452+0000 7fa51edec700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa518072800 con 0x7fa5180b2be0 2023-10-16T13:28:01.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.452+0000 7fa51edec700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5180d03d0 con 0x7fa51815b860 2023-10-16T13:28:01.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.452+0000 7fa51d389700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa5180b4490 0x7fa5180720f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38442/0 (socket says 172.21.15.16:38442) 2023-10-16T13:28:01.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.452+0000 7fa51d389700 1 -- 172.21.15.16:0/2927149063 learned_addr learned my addr 172.21.15.16:0/2927149063 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:28:01.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.452+0000 7fa51cb88700 1 --2- 172.21.15.16:0/2927149063 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa51815b860 0x7fa51806a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:01.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.452+0000 7fa5177fe700 1 -- 172.21.15.16:0/2927149063 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1103437434 0 0) 0x7fa5180d11c0 con 0x7fa5180b4490 2023-10-16T13:28:01.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.452+0000 7fa5177fe700 1 -- 172.21.15.16:0/2927149063 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5080035b0 con 0x7fa5180b4490 2023-10-16T13:28:01.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.452+0000 7fa5177fe700 1 -- 172.21.15.16:0/2927149063 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2359799184 0 0) 0x7fa518072800 con 0x7fa5180b2be0 2023-10-16T13:28:01.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.452+0000 7fa5177fe700 1 -- 172.21.15.16:0/2927149063 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5180d11c0 con 0x7fa5180b2be0 2023-10-16T13:28:01.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.452+0000 7fa51cb88700 1 -- 172.21.15.16:0/2927149063 >> v1:172.21.15.49:6789/0 conn(0x7fa5180b2be0 legacy=0x7fa5180719e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:01.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.452+0000 7fa51cb88700 1 -- 172.21.15.16:0/2927149063 >> v1:172.21.15.16:6789/0 conn(0x7fa5180b4490 legacy=0x7fa5180720f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:01.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.452+0000 7fa51cb88700 1 -- 172.21.15.16:0/2927149063 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5180cd360 con 0x7fa51815b860 2023-10-16T13:28:01.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.453+0000 7fa51cb88700 1 --2- 172.21.15.16:0/2927149063 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa51815b860 0x7fa51806a7e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fa50c009870 tx=0x7fa50c009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=afdbd08351405217 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:01.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.453+0000 7fa5177fe700 1 -- 172.21.15.16:0/2927149063 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa50c011830 con 0x7fa51815b860 2023-10-16T13:28:01.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.453+0000 7fa5177fe700 1 -- 172.21.15.16:0/2927149063 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa50c011990 con 0x7fa51815b860 2023-10-16T13:28:01.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.453+0000 7fa5177fe700 1 -- 172.21.15.16:0/2927149063 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa50c011d20 con 0x7fa51815b860 2023-10-16T13:28:01.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.453+0000 7fa51edec700 1 -- 172.21.15.16:0/2927149063 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa51815b860 msgr2=0x7fa51806a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:01.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.453+0000 7fa51edec700 1 --2- 172.21.15.16:0/2927149063 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa51815b860 0x7fa51806a7e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fa50c009870 tx=0x7fa50c009b80 comp rx=0 tx=0).stop 2023-10-16T13:28:01.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.453+0000 7fa51edec700 1 -- 172.21.15.16:0/2927149063 shutdown_connections 2023-10-16T13:28:01.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.453+0000 7fa51edec700 1 --2- 172.21.15.16:0/2927149063 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa51815b860 0x7fa51806a7e0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:01.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.453+0000 7fa51edec700 1 -- 172.21.15.16:0/2927149063 >> 172.21.15.16:0/2927149063 conn(0x7fa5180b1fd0 msgr2=0x7fa5180b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:01.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.453+0000 7fa51edec700 1 -- 172.21.15.16:0/2927149063 shutdown_connections 2023-10-16T13:28:01.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.453+0000 7fa51edec700 1 -- 172.21.15.16:0/2927149063 wait complete. 2023-10-16T13:28:01.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.455+0000 7fa51edec700 1 Processor -- start 2023-10-16T13:28:01.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.455+0000 7fa51edec700 1 -- start start 2023-10-16T13:28:01.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.455+0000 7fa51edec700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5180b2be0 0x7fa5180a84c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:01.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.455+0000 7fa51edec700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa5180b4490 0x7fa51814a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:01.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.455+0000 7fa51edec700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa51815b860 0x7fa51814ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:01.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.455+0000 7fa51edec700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5180a8e10 con 0x7fa5180b2be0 2023-10-16T13:28:01.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.455+0000 7fa51edec700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5181f0b10 con 0x7fa51815b860 2023-10-16T13:28:01.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.455+0000 7fa51edec700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5181f0df0 con 0x7fa5180b4490 2023-10-16T13:28:01.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.456+0000 7fa51cb88700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5180b2be0 0x7fa5180a84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:01.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.456+0000 7fa51cb88700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5180b2be0 0x7fa5180a84c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49170/0 (socket says 172.21.15.16:49170) 2023-10-16T13:28:01.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.456+0000 7fa51cb88700 1 -- 172.21.15.16:0/4139682899 learned_addr learned my addr 172.21.15.16:0/4139682899 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:01.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.456+0000 7fa517fff700 1 --2- 172.21.15.16:0/4139682899 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa5180b4490 0x7fa51814a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:01.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.456+0000 7fa51d389700 1 --2- 172.21.15.16:0/4139682899 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa51815b860 0x7fa51814ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:01.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.456+0000 7fa51cb88700 1 -- 172.21.15.16:0/4139682899 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa5180b4490 msgr2=0x7fa51814a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:01.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.456+0000 7fa51cb88700 1 --2- 172.21.15.16:0/4139682899 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa5180b4490 0x7fa51814a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:01.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.456+0000 7fa51cb88700 1 -- 172.21.15.16:0/4139682899 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa51815b860 msgr2=0x7fa51814ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:01.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.456+0000 7fa51cb88700 1 --2- 172.21.15.16:0/4139682899 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa51815b860 0x7fa51814ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:01.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.456+0000 7fa51cb88700 1 -- 172.21.15.16:0/4139682899 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa50c0094a0 con 0x7fa5180b2be0 2023-10-16T13:28:01.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.456+0000 7fa51cb88700 1 --2- 172.21.15.16:0/4139682899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5180b2be0 0x7fa5180a84c0 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7fa50c0097c0 tx=0x7fa50c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:01.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.456+0000 7fa5157fa700 1 -- 172.21.15.16:0/4139682899 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa50c01a8f0 con 0x7fa5180b2be0 2023-10-16T13:28:01.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.456+0000 7fa5157fa700 1 -- 172.21.15.16:0/4139682899 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa50c01aa70 con 0x7fa5180b2be0 2023-10-16T13:28:01.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.456+0000 7fa51edec700 1 -- 172.21.15.16:0/4139682899 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5181f10d0 con 0x7fa5180b2be0 2023-10-16T13:28:01.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.456+0000 7fa5157fa700 1 -- 172.21.15.16:0/4139682899 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa50c01ae20 con 0x7fa5180b2be0 2023-10-16T13:28:01.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.456+0000 7fa51edec700 1 -- 172.21.15.16:0/4139682899 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5180a8a00 con 0x7fa5180b2be0 2023-10-16T13:28:01.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.458+0000 7fa51edec700 1 -- 172.21.15.16:0/4139682899 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5181f0970 con 0x7fa5180b2be0 2023-10-16T13:28:01.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.458+0000 7fa5157fa700 1 -- 172.21.15.16:0/4139682899 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 103) v1 ==== 89332+0+0 (secure 0 0 0) 0x7fa50c02e450 con 0x7fa5180b2be0 2023-10-16T13:28:01.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.458+0000 7fa5157fa700 1 --2- 172.21.15.16:0/4139682899 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7fa4fc06b020 0x7fa4fc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:01.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.458+0000 7fa517fff700 1 -- 172.21.15.16:0/4139682899 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7fa4fc06b020 msgr2=0x7fa4fc06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3622553136 2023-10-16T13:28:01.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.459+0000 7fa5157fa700 1 -- 172.21.15.16:0/4139682899 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 6473+0+0 (secure 0 0 0) 0x7fa50c08acd0 con 0x7fa5180b2be0 2023-10-16T13:28:01.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.459+0000 7fa517fff700 1 --2- 172.21.15.16:0/4139682899 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7fa4fc06b020 0x7fa4fc06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:28:01.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.463+0000 7fa5157fa700 1 -- 172.21.15.16:0/4139682899 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa5181f0970 con 0x7fa5180b2be0 2023-10-16T13:28:01.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.660+0000 7fa517fff700 1 -- 172.21.15.16:0/4139682899 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7fa4fc06b020 msgr2=0x7fa4fc06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3622553136 2023-10-16T13:28:01.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.660+0000 7fa517fff700 1 --2- 172.21.15.16:0/4139682899 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7fa4fc06b020 0x7fa4fc06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:28:01.722 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:28:01.719+0000 7f180423e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T13:28:01.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.742+0000 7fa51edec700 1 -- 172.21.15.16:0/4139682899 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fa5181f0970 con 0x7fa5180b2be0 2023-10-16T13:28:01.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.742+0000 7fa5157fa700 1 -- 172.21.15.16:0/4139682899 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v178) v1 ==== 80+0+467 (secure 0 0 0) 0x7fa5181f0970 con 0x7fa5180b2be0 2023-10-16T13:28:01.745 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/472482039 2023-10-17T13:27:56.552715+0000 2023-10-16T13:28:01.745 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/1822527107 2023-10-17T13:27:56.552715+0000 2023-10-16T13:28:01.745 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3919013455 2023-10-17T13:27:56.552715+0000 2023-10-16T13:28:01.745 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6833/2329471558 2023-10-17T13:27:56.552715+0000 2023-10-16T13:28:01.745 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6832/2329471558 2023-10-17T13:27:56.552715+0000 2023-10-16T13:28:01.746 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/508475708 2023-10-17T13:27:56.552715+0000 2023-10-16T13:28:01.746 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/353895884 2023-10-17T13:27:56.552715+0000 2023-10-16T13:28:01.746 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2733317665 2023-10-17T13:27:56.552715+0000 2023-10-16T13:28:01.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.744+0000 7fa51edec700 1 -- 172.21.15.16:0/4139682899 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7fa4fc06b020 msgr2=0x7fa4fc06d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:28:01.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.744+0000 7fa51edec700 1 --2- 172.21.15.16:0/4139682899 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7fa4fc06b020 0x7fa4fc06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:01.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.744+0000 7fa51edec700 1 -- 172.21.15.16:0/4139682899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5180b2be0 msgr2=0x7fa5180a84c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:01.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.744+0000 7fa51edec700 1 --2- 172.21.15.16:0/4139682899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5180b2be0 0x7fa5180a84c0 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7fa50c0097c0 tx=0x7fa50c004750 comp rx=0 tx=0).stop 2023-10-16T13:28:01.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.744+0000 7fa51edec700 1 -- 172.21.15.16:0/4139682899 shutdown_connections 2023-10-16T13:28:01.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.744+0000 7fa51edec700 1 --2- 172.21.15.16:0/4139682899 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7fa4fc06b020 0x7fa4fc06d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:01.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.744+0000 7fa51edec700 1 --2- 172.21.15.16:0/4139682899 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa51815b860 0x7fa51814ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:01.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.744+0000 7fa51edec700 1 --2- 172.21.15.16:0/4139682899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5180b2be0 0x7fa5180a84c0 unknown :-1 s=CLOSED pgs=1988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:01.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.744+0000 7fa51edec700 1 --2- 172.21.15.16:0/4139682899 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa5180b4490 0x7fa51814a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:01.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.744+0000 7fa51edec700 1 -- 172.21.15.16:0/4139682899 >> 172.21.15.16:0/4139682899 conn(0x7fa5180b1fd0 msgr2=0x7fa5180b4950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:01.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.744+0000 7fa51edec700 1 -- 172.21.15.16:0/4139682899 shutdown_connections 2023-10-16T13:28:01.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:01.744+0000 7fa51edec700 1 -- 172.21.15.16:0/4139682899 wait complete. 2023-10-16T13:28:01.752 INFO:teuthology.orchestra.run.smithi016.stderr:listed 8 entries 2023-10-16T13:28:01.761 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-16T13:28:01.761 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:28:01.762 DEBUG:teuthology.orchestra.run.smithi016:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-16T13:28:01.795 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:28:01.792+0000 7f180423e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T13:28:01.810 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:28:01.808+0000 7f954f950200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T13:28:01.819 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2023-10-16T13:28:01.820 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:28:01.820 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-16T13:28:01.837 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T13:28:01.837 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:28:01.855 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284204k,nr_inodes=4071051,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=753 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260540k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T13:28:01.855 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T13:28:01.856 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:28:01.887 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:28:01.885+0000 7f954f950200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T13:28:01.894 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16287364k,nr_inodes=4071841,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20251 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3261172k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T13:28:01.895 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T13:28:01.895 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:28:01.912 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284204k,nr_inodes=4071051,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=753 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260540k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T13:28:01.913 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:28:02.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.059+0000 7f1e7c580700 1 Processor -- start 2023-10-16T13:28:02.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.059+0000 7f1e7c580700 1 -- start start 2023-10-16T13:28:02.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.060+0000 7f1e7c580700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1e7415b630 0x7f1e740d0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:02.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.060+0000 7f1e7c580700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e740b0970 con 0x7f1e7415ba10 2023-10-16T13:28:02.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.060+0000 7f1e7c580700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e740cd690 con 0x7f1e740ce4f0 2023-10-16T13:28:02.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.060+0000 7f1e7c580700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e740cd970 con 0x7f1e7415b630 2023-10-16T13:28:02.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.060+0000 7f1e7a31c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1e7415b630 0x7f1e740d0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:02.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.060+0000 7f1e79b1b700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f1e7415ba10 0x7f1e740cdde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38454/0 (socket says 172.21.15.16:38454) 2023-10-16T13:28:02.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.061+0000 7f1e79b1b700 1 -- 172.21.15.16:0/2796230868 learned_addr learned my addr 172.21.15.16:0/2796230868 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:28:02.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.061+0000 7f1e7a31c700 1 -- 172.21.15.16:0/2796230868 >> v1:172.21.15.49:6789/0 conn(0x7f1e740ce4f0 legacy=0x7f1e740b1430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:02.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.061+0000 7f1e7a31c700 1 -- 172.21.15.16:0/2796230868 >> v1:172.21.15.16:6789/0 conn(0x7f1e7415ba10 legacy=0x7f1e740cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:02.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.061+0000 7f1e7a31c700 1 -- 172.21.15.16:0/2796230868 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e740b0c30 con 0x7f1e7415b630 2023-10-16T13:28:02.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.061+0000 7f1e7931a700 1 -- 172.21.15.16:0/2796230868 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 863100689 0 0) 0x7f1e740b0970 con 0x7f1e7415ba10 2023-10-16T13:28:02.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.061+0000 7f1e7a31c700 1 --2- 172.21.15.16:0/2796230868 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e7415b630 0x7f1e740d0380 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7f1e70009840 tx=0x7f1e70009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec69f0340c8dcfab server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:02.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.061+0000 7f1e7931a700 1 -- 172.21.15.16:0/2796230868 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e70011830 con 0x7f1e7415b630 2023-10-16T13:28:02.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.061+0000 7f1e7931a700 1 -- 172.21.15.16:0/2796230868 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e70011990 con 0x7f1e7415b630 2023-10-16T13:28:02.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.061+0000 7f1e7931a700 1 -- 172.21.15.16:0/2796230868 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e70011d20 con 0x7f1e7415b630 2023-10-16T13:28:02.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.062+0000 7f1e7c580700 1 -- 172.21.15.16:0/2796230868 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e7415b630 msgr2=0x7f1e740d0380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:02.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.062+0000 7f1e7c580700 1 --2- 172.21.15.16:0/2796230868 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e7415b630 0x7f1e740d0380 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7f1e70009840 tx=0x7f1e70009b50 comp rx=0 tx=0).stop 2023-10-16T13:28:02.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.062+0000 7f1e7c580700 1 -- 172.21.15.16:0/2796230868 shutdown_connections 2023-10-16T13:28:02.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.062+0000 7f1e7c580700 1 --2- 172.21.15.16:0/2796230868 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e7415b630 0x7f1e740d0380 unknown :-1 s=CLOSED pgs=1990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:02.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.062+0000 7f1e7c580700 1 -- 172.21.15.16:0/2796230868 >> 172.21.15.16:0/2796230868 conn(0x7f1e740d14b0 msgr2=0x7f1e740d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:02.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.062+0000 7f1e7c580700 1 -- 172.21.15.16:0/2796230868 shutdown_connections 2023-10-16T13:28:02.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.062+0000 7f1e7c580700 1 -- 172.21.15.16:0/2796230868 wait complete. 2023-10-16T13:28:02.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.064+0000 7f1e7c580700 1 Processor -- start 2023-10-16T13:28:02.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.064+0000 7f1e7c580700 1 -- start start 2023-10-16T13:28:02.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.065+0000 7f1e7c580700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e740ce4f0 0x7f1e741509b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:02.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.065+0000 7f1e7c580700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1e7415b630 0x7f1e740ca970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:02.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.065+0000 7f1e7c580700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1e7415ba10 0x7f1e740caeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:02.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.065+0000 7f1e7c580700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e7415e6f0 con 0x7f1e740ce4f0 2023-10-16T13:28:02.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.065+0000 7f1e7c580700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e7415bf20 con 0x7f1e7415ba10 2023-10-16T13:28:02.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.065+0000 7f1e7c580700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e7415c200 con 0x7f1e7415b630 2023-10-16T13:28:02.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.065+0000 7f1e7a31c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e740ce4f0 0x7f1e741509b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:02.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.065+0000 7f1e7a31c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e740ce4f0 0x7f1e741509b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49184/0 (socket says 172.21.15.16:49184) 2023-10-16T13:28:02.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.065+0000 7f1e7a31c700 1 -- 172.21.15.16:0/1460334973 learned_addr learned my addr 172.21.15.16:0/1460334973 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:02.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.065+0000 7f1e7ab1d700 1 --2- 172.21.15.16:0/1460334973 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1e7415ba10 0x7f1e740caeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:02.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.065+0000 7f1e79b1b700 1 --2- 172.21.15.16:0/1460334973 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1e7415b630 0x7f1e740ca970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:02.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.066+0000 7f1e7a31c700 1 -- 172.21.15.16:0/1460334973 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1e7415b630 msgr2=0x7f1e740ca970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:02.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.066+0000 7f1e7a31c700 1 --2- 172.21.15.16:0/1460334973 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1e7415b630 0x7f1e740ca970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:02.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.066+0000 7f1e7a31c700 1 -- 172.21.15.16:0/1460334973 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1e7415ba10 msgr2=0x7f1e740caeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:02.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.066+0000 7f1e7a31c700 1 --2- 172.21.15.16:0/1460334973 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1e7415ba10 0x7f1e740caeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:02.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.066+0000 7f1e7a31c700 1 -- 172.21.15.16:0/1460334973 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e700094a0 con 0x7f1e740ce4f0 2023-10-16T13:28:02.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.066+0000 7f1e7ab1d700 1 --2- 172.21.15.16:0/1460334973 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1e7415ba10 0x7f1e740caeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:28:02.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.066+0000 7f1e7a31c700 1 --2- 172.21.15.16:0/1460334973 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e740ce4f0 0x7f1e741509b0 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7f1e700090b0 tx=0x7f1e70004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:02.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.066+0000 7f1e6affd700 1 -- 172.21.15.16:0/1460334973 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e700045d0 con 0x7f1e740ce4f0 2023-10-16T13:28:02.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.066+0000 7f1e6affd700 1 -- 172.21.15.16:0/1460334973 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e7001f070 con 0x7f1e740ce4f0 2023-10-16T13:28:02.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.066+0000 7f1e7c580700 1 -- 172.21.15.16:0/1460334973 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e7415c4e0 con 0x7f1e740ce4f0 2023-10-16T13:28:02.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.066+0000 7f1e6affd700 1 -- 172.21.15.16:0/1460334973 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e700038a0 con 0x7f1e740ce4f0 2023-10-16T13:28:02.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.066+0000 7f1e7c580700 1 -- 172.21.15.16:0/1460334973 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e74150ef0 con 0x7f1e740ce4f0 2023-10-16T13:28:02.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.067+0000 7f1e7c580700 1 -- 172.21.15.16:0/1460334973 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e7415ca00 con 0x7f1e740ce4f0 2023-10-16T13:28:02.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.068+0000 7f1e6affd700 1 -- 172.21.15.16:0/1460334973 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 103) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f1e70003a00 con 0x7f1e740ce4f0 2023-10-16T13:28:02.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.068+0000 7f1e6affd700 1 --2- 172.21.15.16:0/1460334973 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7f1e5806b020 0x7f1e5806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:02.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.068+0000 7f1e79b1b700 1 -- 172.21.15.16:0/1460334973 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7f1e5806b020 msgr2=0x7f1e5806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3622553136 2023-10-16T13:28:02.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.068+0000 7f1e6affd700 1 -- 172.21.15.16:0/1460334973 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 6473+0+0 (secure 0 0 0) 0x7f1e70021070 con 0x7f1e740ce4f0 2023-10-16T13:28:02.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.068+0000 7f1e79b1b700 1 --2- 172.21.15.16:0/1460334973 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7f1e5806b020 0x7f1e5806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:28:02.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.073+0000 7f1e6affd700 1 -- 172.21.15.16:0/1460334973 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1e7415ca00 con 0x7f1e740ce4f0 2023-10-16T13:28:02.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.269+0000 7f1e79b1b700 1 -- 172.21.15.16:0/1460334973 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7f1e5806b020 msgr2=0x7f1e5806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3622553136 2023-10-16T13:28:02.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.269+0000 7f1e79b1b700 1 --2- 172.21.15.16:0/1460334973 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7f1e5806b020 0x7f1e5806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:28:02.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.326+0000 7f1e7c580700 1 -- 172.21.15.16:0/1460334973 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1e7415ca00 con 0x7f1e740ce4f0 2023-10-16T13:28:02.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.326+0000 7f1e6affd700 1 -- 172.21.15.16:0/1460334973 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+4917 (secure 0 0 0) 0x7f1e7415ca00 con 0x7f1e740ce4f0 2023-10-16T13:28:02.329 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:28:02.329 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":86,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7964,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/4012137577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":4012137577},{"type":"v1","addr":"172.21.15.16:6837","nonce":4012137577}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81},{"gid":7970,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/3742568047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":3742568047},{"type":"v1","addr":"172.21.15.49:6835","nonce":3742568047}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81},{"gid":9756,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/3838743980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":3838743980},{"type":"v1","addr":"172.21.15.16:6836","nonce":3838743980}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:27:41.561818+0000","modified":"2023-10-16T13:27:44.572793+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9774},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9774":{"gid":9774,"name":"d","rank":0,"incarnation":82,"state":"up:active","state_seq":3,"addr":"172.21.15.49:6834/2218212001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":2218212001},{"type":"v1","addr":"172.21.15.49:6834","nonce":2218212001}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2023-10-16T13:28:02.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.329+0000 7f1e7c580700 1 -- 172.21.15.16:0/1460334973 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7f1e5806b020 msgr2=0x7f1e5806d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:28:02.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.329+0000 7f1e7c580700 1 --2- 172.21.15.16:0/1460334973 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7f1e5806b020 0x7f1e5806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:02.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.329+0000 7f1e7c580700 1 -- 172.21.15.16:0/1460334973 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e740ce4f0 msgr2=0x7f1e741509b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:02.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.329+0000 7f1e7c580700 1 --2- 172.21.15.16:0/1460334973 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e740ce4f0 0x7f1e741509b0 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7f1e700090b0 tx=0x7f1e70004750 comp rx=0 tx=0).stop 2023-10-16T13:28:02.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.329+0000 7f1e7c580700 1 -- 172.21.15.16:0/1460334973 shutdown_connections 2023-10-16T13:28:02.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.329+0000 7f1e7c580700 1 --2- 172.21.15.16:0/1460334973 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7f1e5806b020 0x7f1e5806d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:02.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.329+0000 7f1e7c580700 1 --2- 172.21.15.16:0/1460334973 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1e7415ba10 0x7f1e740caeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:02.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.329+0000 7f1e7c580700 1 --2- 172.21.15.16:0/1460334973 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e740ce4f0 0x7f1e741509b0 unknown :-1 s=CLOSED pgs=1991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:02.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.329+0000 7f1e7c580700 1 --2- 172.21.15.16:0/1460334973 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1e7415b630 0x7f1e740ca970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:02.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.329+0000 7f1e7c580700 1 -- 172.21.15.16:0/1460334973 >> 172.21.15.16:0/1460334973 conn(0x7f1e740d14b0 msgr2=0x7f1e740b12f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:02.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.329+0000 7f1e7c580700 1 -- 172.21.15.16:0/1460334973 shutdown_connections 2023-10-16T13:28:02.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.329+0000 7f1e7c580700 1 -- 172.21.15.16:0/1460334973 wait complete. 2023-10-16T13:28:02.336 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 86 2023-10-16T13:28:02.344 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:28:02.464 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:28:02.461+0000 7f180423e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:28:02.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.472+0000 7fd0acd37700 1 Processor -- start 2023-10-16T13:28:02.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.472+0000 7fd0acd37700 1 -- start start 2023-10-16T13:28:02.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.474+0000 7fd0acd37700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd0a80d2f70 0x7fd0a80d2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:02.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.474+0000 7fd0acd37700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd0a80ce9f0 0x7fd0a80cdc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:02.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.474+0000 7fd0acd37700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0a80ce1a0 con 0x7fd0a815d900 2023-10-16T13:28:02.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.474+0000 7fd0acd37700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0a80cee80 con 0x7fd0a80d2f70 2023-10-16T13:28:02.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.474+0000 7fd0acd37700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0a80ce400 con 0x7fd0a80ce9f0 2023-10-16T13:28:02.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.474+0000 7fd0a5d9b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd0a80d2f70 0x7fd0a80d2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:02.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.474+0000 7fd0a659c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd0a815d900 0x7fd0a80d2860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38468/0 (socket says 172.21.15.16:38468) 2023-10-16T13:28:02.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.474+0000 7fd0a659c700 1 -- 172.21.15.16:0/1476511552 learned_addr learned my addr 172.21.15.16:0/1476511552 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:28:02.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.474+0000 7fd0a6d9d700 1 --2- 172.21.15.16:0/1476511552 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd0a80ce9f0 0x7fd0a80cdc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:02.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.474+0000 7fd0a5d9b700 1 -- 172.21.15.16:0/1476511552 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd0a80ce9f0 msgr2=0x7fd0a80cdc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:28:02.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.474+0000 7fd0a5d9b700 1 --2- 172.21.15.16:0/1476511552 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd0a80ce9f0 0x7fd0a80cdc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:02.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.474+0000 7fd0a5d9b700 1 -- 172.21.15.16:0/1476511552 >> v1:172.21.15.16:6789/0 conn(0x7fd0a815d900 legacy=0x7fd0a80d2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:02.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.474+0000 7fd0a559a700 1 -- 172.21.15.16:0/1476511552 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2614182839 0 0) 0x7fd0a80ce1a0 con 0x7fd0a815d900 2023-10-16T13:28:02.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.474+0000 7fd0a5d9b700 1 -- 172.21.15.16:0/1476511552 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0a815dce0 con 0x7fd0a80d2f70 2023-10-16T13:28:02.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.474+0000 7fd0a5d9b700 1 --2- 172.21.15.16:0/1476511552 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0a80d2f70 0x7fd0a80d2040 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7fd09c009840 tx=0x7fd09c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=361a3d9f111ee7c1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:02.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.475+0000 7fd0a559a700 1 -- 172.21.15.16:0/1476511552 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd09c011830 con 0x7fd0a80d2f70 2023-10-16T13:28:02.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.475+0000 7fd0a559a700 1 -- 172.21.15.16:0/1476511552 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd09c011990 con 0x7fd0a80d2f70 2023-10-16T13:28:02.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.475+0000 7fd0a559a700 1 -- 172.21.15.16:0/1476511552 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd09c011d20 con 0x7fd0a80d2f70 2023-10-16T13:28:02.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.475+0000 7fd0acd37700 1 -- 172.21.15.16:0/1476511552 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0a80d2f70 msgr2=0x7fd0a80d2040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:02.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.475+0000 7fd0acd37700 1 --2- 172.21.15.16:0/1476511552 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0a80d2f70 0x7fd0a80d2040 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7fd09c009840 tx=0x7fd09c009b50 comp rx=0 tx=0).stop 2023-10-16T13:28:02.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.475+0000 7fd0acd37700 1 -- 172.21.15.16:0/1476511552 shutdown_connections 2023-10-16T13:28:02.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.475+0000 7fd0acd37700 1 --2- 172.21.15.16:0/1476511552 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd0a80ce9f0 0x7fd0a80cdc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:02.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.475+0000 7fd0acd37700 1 --2- 172.21.15.16:0/1476511552 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0a80d2f70 0x7fd0a80d2040 unknown :-1 s=CLOSED pgs=1993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:02.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.475+0000 7fd0acd37700 1 -- 172.21.15.16:0/1476511552 >> 172.21.15.16:0/1476511552 conn(0x7fd0a8072330 msgr2=0x7fd0a8072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:02.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.475+0000 7fd0acd37700 1 -- 172.21.15.16:0/1476511552 shutdown_connections 2023-10-16T13:28:02.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.475+0000 7fd0acd37700 1 -- 172.21.15.16:0/1476511552 wait complete. 2023-10-16T13:28:02.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.477+0000 7fd0acd37700 1 Processor -- start 2023-10-16T13:28:02.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.477+0000 7fd0acd37700 1 -- start start 2023-10-16T13:28:02.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.478+0000 7fd0acd37700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0a80ce9f0 0x7fd0a815ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:02.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.478+0000 7fd0acd37700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd0a80d2f70 0x7fd0a8156dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:02.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.478+0000 7fd0acd37700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd0a815d900 0x7fd0a8157310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:02.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.478+0000 7fd0acd37700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0a81604c0 con 0x7fd0a80ce9f0 2023-10-16T13:28:02.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.478+0000 7fd0acd37700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0a815f8b0 con 0x7fd0a815d900 2023-10-16T13:28:02.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.478+0000 7fd0acd37700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0a815fb90 con 0x7fd0a80d2f70 2023-10-16T13:28:02.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.478+0000 7fd0a659c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0a80ce9f0 0x7fd0a815ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:02.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.478+0000 7fd0a5d9b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd0a80d2f70 0x7fd0a8156dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:02.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.478+0000 7fd0a659c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0a80ce9f0 0x7fd0a815ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49200/0 (socket says 172.21.15.16:49200) 2023-10-16T13:28:02.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.478+0000 7fd0a659c700 1 -- 172.21.15.16:0/848629815 learned_addr learned my addr 172.21.15.16:0/848629815 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:02.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.478+0000 7fd0a6d9d700 1 --2- 172.21.15.16:0/848629815 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd0a815d900 0x7fd0a8157310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:02.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.478+0000 7fd0a5d9b700 1 -- 172.21.15.16:0/848629815 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd0a815d900 msgr2=0x7fd0a8157310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:02.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.478+0000 7fd0a5d9b700 1 --2- 172.21.15.16:0/848629815 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd0a815d900 0x7fd0a8157310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:02.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.478+0000 7fd0a5d9b700 1 -- 172.21.15.16:0/848629815 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0a80ce9f0 msgr2=0x7fd0a815ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:02.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.478+0000 7fd0a5d9b700 1 --2- 172.21.15.16:0/848629815 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0a80ce9f0 0x7fd0a815ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:02.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.478+0000 7fd0a5d9b700 1 -- 172.21.15.16:0/848629815 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd09c0094a0 con 0x7fd0a80d2f70 2023-10-16T13:28:02.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.478+0000 7fd0a659c700 1 --2- 172.21.15.16:0/848629815 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0a80ce9f0 0x7fd0a815ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:28:02.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.478+0000 7fd0a5d9b700 1 --2- 172.21.15.16:0/848629815 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd0a80d2f70 0x7fd0a8156dd0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fd09c016000 tx=0x7fd09c004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:02.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.478+0000 7fd096ffd700 1 -- 172.21.15.16:0/848629815 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd09c0036a0 con 0x7fd0a80d2f70 2023-10-16T13:28:02.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.479+0000 7fd096ffd700 1 -- 172.21.15.16:0/848629815 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd09c01f070 con 0x7fd0a80d2f70 2023-10-16T13:28:02.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.479+0000 7fd096ffd700 1 -- 172.21.15.16:0/848629815 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd09c003a20 con 0x7fd0a80d2f70 2023-10-16T13:28:02.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.479+0000 7fd0acd37700 1 -- 172.21.15.16:0/848629815 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0a815fe70 con 0x7fd0a80d2f70 2023-10-16T13:28:02.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.479+0000 7fd0acd37700 1 -- 172.21.15.16:0/848629815 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0a80c6490 con 0x7fd0a80d2f70 2023-10-16T13:28:02.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.480+0000 7fd0acd37700 1 -- 172.21.15.16:0/848629815 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0a8160630 con 0x7fd0a80d2f70 2023-10-16T13:28:02.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.480+0000 7fd096ffd700 1 -- 172.21.15.16:0/848629815 <== mon.2 v2:172.21.15.49:3301/0 4 ==== mgrmap(e 103) v1 ==== 89332+0+0 (secure 0 0 0) 0x7fd09c003b80 con 0x7fd0a80d2f70 2023-10-16T13:28:02.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.480+0000 7fd096ffd700 1 --2- 172.21.15.16:0/848629815 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7fd08806b020 0x7fd08806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:02.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.480+0000 7fd0a659c700 1 -- 172.21.15.16:0/848629815 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7fd08806b020 msgr2=0x7fd08806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3622553136 2023-10-16T13:28:02.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.480+0000 7fd096ffd700 1 -- 172.21.15.16:0/848629815 <== mon.2 v2:172.21.15.49:3301/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 6473+0+0 (secure 0 0 0) 0x7fd09c021070 con 0x7fd0a80d2f70 2023-10-16T13:28:02.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.481+0000 7fd0a659c700 1 --2- 172.21.15.16:0/848629815 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7fd08806b020 0x7fd08806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:28:02.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.485+0000 7fd096ffd700 1 -- 172.21.15.16:0/848629815 <== mon.2 v2:172.21.15.49:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd09c055ec0 con 0x7fd0a80d2f70 2023-10-16T13:28:02.546 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:28:02.544+0000 7f180423e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T13:28:02.581 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:28:02.579+0000 7f954f950200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:28:02.665 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:28:02.663+0000 7f954f950200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T13:28:02.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.682+0000 7fd0a659c700 1 -- 172.21.15.16:0/848629815 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7fd08806b020 msgr2=0x7fd08806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3622553136 2023-10-16T13:28:02.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.682+0000 7fd0a659c700 1 --2- 172.21.15.16:0/848629815 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7fd08806b020 0x7fd08806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:28:02.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.693+0000 7fd0acd37700 1 -- 172.21.15.16:0/848629815 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd0a80c6620 con 0x7fd0a80d2f70 2023-10-16T13:28:02.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.694+0000 7fd096ffd700 1 -- 172.21.15.16:0/848629815 <== mon.2 v2:172.21.15.49:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v178) v1 ==== 74+0+17273 (secure 0 0 0) 0x7fd09c015090 con 0x7fd0a80d2f70 2023-10-16T13:28:02.697 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:28:02.697 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":178,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:27:56.552748+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2023-10-16T13:27:40.032163+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"176","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2023-10-16T13:27:40.523975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"177","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/472482039":"2023-10-17T13:27:56.552715+0000","172.21.15.16:0/1822527107":"2023-10-17T13:27:56.552715+0000","172.21.15.16:0/3919013455":"2023-10-17T13:27:56.552715+0000","172.21.15.16:6833/2329471558":"2023-10-17T13:27:56.552715+0000","172.21.15.16:6832/2329471558":"2023-10-17T13:27:56.552715+0000","172.21.15.16:0/508475708":"2023-10-17T13:27:56.552715+0000","172.21.15.16:0/353895884":"2023-10-17T13:27:56.552715+0000","172.21.15.16:0/2733317665":"2023-10-17T13:27:56.552715+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:28:02.698 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:28:02.696+0000 7f180423e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:28:02.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.697+0000 7fd0acd37700 1 -- 172.21.15.16:0/848629815 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7fd08806b020 msgr2=0x7fd08806d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:28:02.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.697+0000 7fd0acd37700 1 --2- 172.21.15.16:0/848629815 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7fd08806b020 0x7fd08806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:02.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.697+0000 7fd0acd37700 1 -- 172.21.15.16:0/848629815 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd0a80d2f70 msgr2=0x7fd0a8156dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:02.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.697+0000 7fd0acd37700 1 --2- 172.21.15.16:0/848629815 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd0a80d2f70 0x7fd0a8156dd0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fd09c016000 tx=0x7fd09c004750 comp rx=0 tx=0).stop 2023-10-16T13:28:02.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.697+0000 7fd0acd37700 1 -- 172.21.15.16:0/848629815 shutdown_connections 2023-10-16T13:28:02.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.697+0000 7fd0acd37700 1 --2- 172.21.15.16:0/848629815 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7fd08806b020 0x7fd08806d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:02.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.697+0000 7fd0acd37700 1 --2- 172.21.15.16:0/848629815 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd0a815d900 0x7fd0a8157310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:02.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.697+0000 7fd0acd37700 1 --2- 172.21.15.16:0/848629815 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0a80ce9f0 0x7fd0a815ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:02.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.697+0000 7fd0acd37700 1 --2- 172.21.15.16:0/848629815 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd0a80d2f70 0x7fd0a8156dd0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:02.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.697+0000 7fd0acd37700 1 -- 172.21.15.16:0/848629815 >> 172.21.15.16:0/848629815 conn(0x7fd0a8072330 msgr2=0x7fd0a80cc6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:02.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.698+0000 7fd0acd37700 1 -- 172.21.15.16:0/848629815 shutdown_connections 2023-10-16T13:28:02.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.698+0000 7fd0acd37700 1 -- 172.21.15.16:0/848629815 wait complete. 2023-10-16T13:28:02.717 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 21, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T13:27:40.032163+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '176', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 22, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T13:27:40.523975+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '177', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T13:28:02.718 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T13:28:02.833 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:28:02.830+0000 7f954f950200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:28:02.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.845+0000 7f54fe23b700 1 Processor -- start 2023-10-16T13:28:02.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.845+0000 7f54fe23b700 1 -- start start 2023-10-16T13:28:02.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.846+0000 7f54fe23b700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f54f8072b30 0x7f54f8071a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:02.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.846+0000 7f54fe23b700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54f815c040 con 0x7f54f815bb40 2023-10-16T13:28:02.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.846+0000 7f54fe23b700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54f815c320 con 0x7f54f80caa40 2023-10-16T13:28:02.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.846+0000 7f54fe23b700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54f806ea20 con 0x7f54f8072b30 2023-10-16T13:28:02.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.846+0000 7f54f6ffd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f54f815bb40 0x7f54f806e310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38502/0 (socket says 172.21.15.16:38502) 2023-10-16T13:28:02.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.846+0000 7f54f6ffd700 1 -- 172.21.15.16:0/2098572836 learned_addr learned my addr 172.21.15.16:0/2098572836 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:28:02.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.846+0000 7f54f77fe700 1 --2- 172.21.15.16:0/2098572836 >> v2:172.21.15.49:3300/0 conn(0x7f54f8072b30 0x7f54f8071a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:02.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.847+0000 7f54f67fc700 1 -- 172.21.15.16:0/2098572836 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 973397362 0 0) 0x7f54f815c040 con 0x7f54f815bb40 2023-10-16T13:28:02.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.847+0000 7f54f67fc700 1 -- 172.21.15.16:0/2098572836 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54e00035b0 con 0x7f54f815bb40 2023-10-16T13:28:02.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.847+0000 7f54f77fe700 1 -- 172.21.15.16:0/2098572836 >> v1:172.21.15.49:6789/0 conn(0x7f54f80caa40 legacy=0x7f54f80cae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:02.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.847+0000 7f54f77fe700 1 -- 172.21.15.16:0/2098572836 >> v1:172.21.15.16:6789/0 conn(0x7f54f815bb40 legacy=0x7f54f806e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:02.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.847+0000 7f54f67fc700 1 -- 172.21.15.16:0/2098572836 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3427290768 0 0) 0x7f54f815c320 con 0x7f54f80caa40 2023-10-16T13:28:02.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.847+0000 7f54f77fe700 1 -- 172.21.15.16:0/2098572836 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54f815c600 con 0x7f54f8072b30 2023-10-16T13:28:02.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.847+0000 7f54f77fe700 1 --2- 172.21.15.16:0/2098572836 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f54f8072b30 0x7f54f8071a70 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f54e8009d90 tx=0x7f54e800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b1d8492f15faf833 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:02.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.847+0000 7f54f67fc700 1 -- 172.21.15.16:0/2098572836 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54e800e070 con 0x7f54f8072b30 2023-10-16T13:28:02.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.847+0000 7f54f67fc700 1 -- 172.21.15.16:0/2098572836 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54e800bb20 con 0x7f54f8072b30 2023-10-16T13:28:02.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.847+0000 7f54f67fc700 1 -- 172.21.15.16:0/2098572836 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54e800bed0 con 0x7f54f8072b30 2023-10-16T13:28:02.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.848+0000 7f54fe23b700 1 -- 172.21.15.16:0/2098572836 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f54f8072b30 msgr2=0x7f54f8071a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:02.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.848+0000 7f54fe23b700 1 --2- 172.21.15.16:0/2098572836 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f54f8072b30 0x7f54f8071a70 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f54e8009d90 tx=0x7f54e800b330 comp rx=0 tx=0).stop 2023-10-16T13:28:02.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.848+0000 7f54fe23b700 1 -- 172.21.15.16:0/2098572836 shutdown_connections 2023-10-16T13:28:02.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.848+0000 7f54fe23b700 1 --2- 172.21.15.16:0/2098572836 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f54f8072b30 0x7f54f8071a70 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:02.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.848+0000 7f54fe23b700 1 -- 172.21.15.16:0/2098572836 >> 172.21.15.16:0/2098572836 conn(0x7f54f80b1f40 msgr2=0x7f54f80b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:02.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.848+0000 7f54fe23b700 1 -- 172.21.15.16:0/2098572836 shutdown_connections 2023-10-16T13:28:02.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.848+0000 7f54fe23b700 1 -- 172.21.15.16:0/2098572836 wait complete. 2023-10-16T13:28:02.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.849+0000 7f54fe23b700 1 Processor -- start 2023-10-16T13:28:02.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.849+0000 7f54fe23b700 1 -- start start 2023-10-16T13:28:02.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.850+0000 7f54fe23b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f54f8072b30 0x7f54f8154cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:02.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.850+0000 7f54fe23b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f54f80caa40 0x7f54f80ceee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:02.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.850+0000 7f54fe23b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54f815bb40 0x7f54f80cf420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:02.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.850+0000 7f54fe23b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54f80ce370 con 0x7f54f815bb40 2023-10-16T13:28:02.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.850+0000 7f54fe23b700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54f80b31d0 con 0x7f54f8072b30 2023-10-16T13:28:02.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.850+0000 7f54fe23b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54f80b34b0 con 0x7f54f80caa40 2023-10-16T13:28:02.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.850+0000 7f54f7fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54f815bb40 0x7f54f80cf420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:02.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.850+0000 7f54f77fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f54f8072b30 0x7f54f8154cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:02.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.850+0000 7f54f7fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54f815bb40 0x7f54f80cf420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49236/0 (socket says 172.21.15.16:49236) 2023-10-16T13:28:02.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.850+0000 7f54f7fff700 1 -- 172.21.15.16:0/777158643 learned_addr learned my addr 172.21.15.16:0/777158643 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:02.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.850+0000 7f54f6ffd700 1 --2- 172.21.15.16:0/777158643 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f54f80caa40 0x7f54f80ceee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:02.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.851+0000 7f54f7fff700 1 -- 172.21.15.16:0/777158643 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f54f80caa40 msgr2=0x7f54f80ceee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:02.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.851+0000 7f54f7fff700 1 --2- 172.21.15.16:0/777158643 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f54f80caa40 0x7f54f80ceee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:02.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.851+0000 7f54f7fff700 1 -- 172.21.15.16:0/777158643 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f54f8072b30 msgr2=0x7f54f8154cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:02.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.851+0000 7f54f7fff700 1 --2- 172.21.15.16:0/777158643 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f54f8072b30 0x7f54f8154cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:02.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.851+0000 7f54f7fff700 1 -- 172.21.15.16:0/777158643 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54e8009a70 con 0x7f54f815bb40 2023-10-16T13:28:02.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.851+0000 7f54f77fe700 1 --2- 172.21.15.16:0/777158643 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f54f8072b30 0x7f54f8154cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:28:02.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.851+0000 7f54f7fff700 1 --2- 172.21.15.16:0/777158643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54f815bb40 0x7f54f80cf420 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 crypto rx=0x7f54f000c4d0 tx=0x7f54f000c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:02.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.851+0000 7f54dffff700 1 -- 172.21.15.16:0/777158643 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54f00054e0 con 0x7f54f815bb40 2023-10-16T13:28:02.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.851+0000 7f54dffff700 1 -- 172.21.15.16:0/777158643 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54f0003680 con 0x7f54f815bb40 2023-10-16T13:28:02.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.851+0000 7f54fe23b700 1 -- 172.21.15.16:0/777158643 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54f80b3790 con 0x7f54f815bb40 2023-10-16T13:28:02.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.851+0000 7f54dffff700 1 -- 172.21.15.16:0/777158643 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54f0003a30 con 0x7f54f815bb40 2023-10-16T13:28:02.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.851+0000 7f54fe23b700 1 -- 172.21.15.16:0/777158643 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54f80b3d50 con 0x7f54f815bb40 2023-10-16T13:28:02.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.852+0000 7f54fe23b700 1 -- 172.21.15.16:0/777158643 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54f804ebb0 con 0x7f54f815bb40 2023-10-16T13:28:02.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.853+0000 7f54dffff700 1 -- 172.21.15.16:0/777158643 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 103) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f54f0003b90 con 0x7f54f815bb40 2023-10-16T13:28:02.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.853+0000 7f54dffff700 1 --2- 172.21.15.16:0/777158643 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7f54e406b020 0x7f54e406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:02.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.853+0000 7f54f77fe700 1 -- 172.21.15.16:0/777158643 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7f54e406b020 msgr2=0x7f54e406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3622553136 2023-10-16T13:28:02.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.854+0000 7f54dffff700 1 -- 172.21.15.16:0/777158643 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 6473+0+0 (secure 0 0 0) 0x7f54f0088ec0 con 0x7f54f815bb40 2023-10-16T13:28:02.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.854+0000 7f54f77fe700 1 --2- 172.21.15.16:0/777158643 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7f54e406b020 0x7f54e406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:28:02.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:02.858+0000 7f54dffff700 1 -- 172.21.15.16:0/777158643 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f54f0053ba0 con 0x7f54f815bb40 2023-10-16T13:28:02.935 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:28:02.933+0000 7f180423e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T13:28:03.013 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:28:03.010+0000 7f180423e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T13:28:03.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.055+0000 7f54f77fe700 1 -- 172.21.15.16:0/777158643 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7f54e406b020 msgr2=0x7f54e406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3622553136 2023-10-16T13:28:03.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.055+0000 7f54f77fe700 1 --2- 172.21.15.16:0/777158643 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7f54e406b020 0x7f54e406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:28:03.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.063+0000 7f54fe23b700 1 -- 172.21.15.16:0/777158643 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f54f80b2d40 con 0x7f54f815bb40 2023-10-16T13:28:03.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.064+0000 7f54dffff700 1 -- 172.21.15.16:0/777158643 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v178) v1 ==== 93+0+10 (secure 0 0 0) 0x7f54f00571c0 con 0x7f54f815bb40 2023-10-16T13:28:03.066 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T13:28:03.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.066+0000 7f54fe23b700 1 -- 172.21.15.16:0/777158643 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7f54e406b020 msgr2=0x7f54e406d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:28:03.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.066+0000 7f54fe23b700 1 --2- 172.21.15.16:0/777158643 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7f54e406b020 0x7f54e406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:03.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.066+0000 7f54fe23b700 1 -- 172.21.15.16:0/777158643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54f815bb40 msgr2=0x7f54f80cf420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:03.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.066+0000 7f54fe23b700 1 --2- 172.21.15.16:0/777158643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54f815bb40 0x7f54f80cf420 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 crypto rx=0x7f54f000c4d0 tx=0x7f54f000c7e0 comp rx=0 tx=0).stop 2023-10-16T13:28:03.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.066+0000 7f54fe23b700 1 -- 172.21.15.16:0/777158643 shutdown_connections 2023-10-16T13:28:03.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.066+0000 7f54fe23b700 1 --2- 172.21.15.16:0/777158643 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7f54e406b020 0x7f54e406d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:03.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.066+0000 7f54fe23b700 1 --2- 172.21.15.16:0/777158643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54f815bb40 0x7f54f80cf420 unknown :-1 s=CLOSED pgs=1995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:03.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.066+0000 7f54fe23b700 1 --2- 172.21.15.16:0/777158643 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f54f8072b30 0x7f54f8154cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:03.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.066+0000 7f54fe23b700 1 --2- 172.21.15.16:0/777158643 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f54f80caa40 0x7f54f80ceee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:03.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.066+0000 7f54fe23b700 1 -- 172.21.15.16:0/777158643 >> 172.21.15.16:0/777158643 conn(0x7f54f80b1f40 msgr2=0x7f54f80c3f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:03.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.066+0000 7f54fe23b700 1 -- 172.21.15.16:0/777158643 shutdown_connections 2023-10-16T13:28:03.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.066+0000 7f54fe23b700 1 -- 172.21.15.16:0/777158643 wait complete. 2023-10-16T13:28:03.079 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:28:03.077+0000 7f954f950200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T13:28:03.081 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T13:28:03.161 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:28:03.159+0000 7f954f950200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T13:28:03.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.209+0000 7fd99a637700 1 Processor -- start 2023-10-16T13:28:03.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.209+0000 7fd99a637700 1 -- start start 2023-10-16T13:28:03.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.209+0000 7fd99a637700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fd9940d4060 0x7fd9940d1520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:03.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.209+0000 7fd99a637700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd9940d1d20 0x7fd9940d21b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:03.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.210+0000 7fd99a637700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9940d4540 con 0x7fd99415da10 2023-10-16T13:28:03.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.210+0000 7fd99a637700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9940cf9b0 con 0x7fd9940d4060 2023-10-16T13:28:03.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.210+0000 7fd99a637700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9940d26f0 con 0x7fd9940d1d20 2023-10-16T13:28:03.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.210+0000 7fd993fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd99415da10 0x7fd9940cf2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38516/0 (socket says 172.21.15.16:38516) 2023-10-16T13:28:03.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.210+0000 7fd993fff700 1 -- 172.21.15.16:0/1470366148 learned_addr learned my addr 172.21.15.16:0/1470366148 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:28:03.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.210+0000 7fd998bd4700 1 --2- 172.21.15.16:0/1470366148 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd9940d1d20 0x7fd9940d21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:03.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.210+0000 7fd9937fe700 1 --2- 172.21.15.16:0/1470366148 >> v2:172.21.15.49:3300/0 conn(0x7fd9940d4060 0x7fd9940d1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:03.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.210+0000 7fd992ffd700 1 -- 172.21.15.16:0/1470366148 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 121367951 0 0) 0x7fd9940d4540 con 0x7fd99415da10 2023-10-16T13:28:03.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.210+0000 7fd992ffd700 1 -- 172.21.15.16:0/1470366148 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd97c0035b0 con 0x7fd99415da10 2023-10-16T13:28:03.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.210+0000 7fd9937fe700 1 -- 172.21.15.16:0/1470366148 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd9940d1d20 msgr2=0x7fd9940d21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:28:03.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.210+0000 7fd9937fe700 1 --2- 172.21.15.16:0/1470366148 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd9940d1d20 0x7fd9940d21b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:03.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.210+0000 7fd9937fe700 1 -- 172.21.15.16:0/1470366148 >> v1:172.21.15.16:6789/0 conn(0x7fd99415da10 legacy=0x7fd9940cf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:03.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.210+0000 7fd9937fe700 1 -- 172.21.15.16:0/1470366148 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9940ce7e0 con 0x7fd9940d4060 2023-10-16T13:28:03.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.210+0000 7fd998bd4700 1 --2- 172.21.15.16:0/1470366148 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd9940d1d20 0x7fd9940d21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:28:03.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.210+0000 7fd9937fe700 1 --2- 172.21.15.16:0/1470366148 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd9940d4060 0x7fd9940d1520 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fd988009840 tx=0x7fd988009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6ac351894f30625b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:03.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.210+0000 7fd992ffd700 1 -- 172.21.15.16:0/1470366148 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd988011830 con 0x7fd9940d4060 2023-10-16T13:28:03.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.211+0000 7fd992ffd700 1 -- 172.21.15.16:0/1470366148 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd988011990 con 0x7fd9940d4060 2023-10-16T13:28:03.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.211+0000 7fd992ffd700 1 -- 172.21.15.16:0/1470366148 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd988011d20 con 0x7fd9940d4060 2023-10-16T13:28:03.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.211+0000 7fd99a637700 1 -- 172.21.15.16:0/1470366148 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd9940d4060 msgr2=0x7fd9940d1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:03.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.211+0000 7fd99a637700 1 --2- 172.21.15.16:0/1470366148 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd9940d4060 0x7fd9940d1520 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fd988009840 tx=0x7fd988009b50 comp rx=0 tx=0).stop 2023-10-16T13:28:03.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.211+0000 7fd99a637700 1 -- 172.21.15.16:0/1470366148 shutdown_connections 2023-10-16T13:28:03.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.211+0000 7fd99a637700 1 --2- 172.21.15.16:0/1470366148 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd9940d1d20 0x7fd9940d21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:03.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.211+0000 7fd99a637700 1 --2- 172.21.15.16:0/1470366148 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd9940d4060 0x7fd9940d1520 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:03.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.211+0000 7fd99a637700 1 -- 172.21.15.16:0/1470366148 >> 172.21.15.16:0/1470366148 conn(0x7fd99409c9f0 msgr2=0x7fd9940d3740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:03.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.211+0000 7fd99a637700 1 -- 172.21.15.16:0/1470366148 shutdown_connections 2023-10-16T13:28:03.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.211+0000 7fd99a637700 1 -- 172.21.15.16:0/1470366148 wait complete. 2023-10-16T13:28:03.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.213+0000 7fd99a637700 1 Processor -- start 2023-10-16T13:28:03.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.213+0000 7fd99a637700 1 -- start start 2023-10-16T13:28:03.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.213+0000 7fd99a637700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd9940d1d20 0x7fd9940b55b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:03.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.213+0000 7fd99a637700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9940d4060 0x7fd9941f6f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:03.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.213+0000 7fd99a637700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd99415da10 0x7fd9941f7440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:03.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.213+0000 7fd99a637700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9941f7e30 con 0x7fd9940d4060 2023-10-16T13:28:03.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.214+0000 7fd99a637700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9940caf90 con 0x7fd9940d1d20 2023-10-16T13:28:03.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.214+0000 7fd99a637700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9941f7b50 con 0x7fd99415da10 2023-10-16T13:28:03.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.214+0000 7fd9937fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9940d4060 0x7fd9941f6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:03.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.214+0000 7fd9937fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9940d4060 0x7fd9941f6f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49250/0 (socket says 172.21.15.16:49250) 2023-10-16T13:28:03.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.214+0000 7fd9937fe700 1 -- 172.21.15.16:0/437575588 learned_addr learned my addr 172.21.15.16:0/437575588 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:03.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.214+0000 7fd993fff700 1 --2- 172.21.15.16:0/437575588 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd9940d1d20 0x7fd9940b55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:03.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.214+0000 7fd998bd4700 1 --2- 172.21.15.16:0/437575588 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd99415da10 0x7fd9941f7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:03.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.214+0000 7fd9937fe700 1 -- 172.21.15.16:0/437575588 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd99415da10 msgr2=0x7fd9941f7440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:03.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.214+0000 7fd9937fe700 1 --2- 172.21.15.16:0/437575588 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd99415da10 0x7fd9941f7440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:03.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.214+0000 7fd9937fe700 1 -- 172.21.15.16:0/437575588 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd9940d1d20 msgr2=0x7fd9940b55b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:03.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.214+0000 7fd9937fe700 1 --2- 172.21.15.16:0/437575588 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd9940d1d20 0x7fd9940b55b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:03.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.214+0000 7fd9937fe700 1 -- 172.21.15.16:0/437575588 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd98c009a70 con 0x7fd9940d4060 2023-10-16T13:28:03.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.214+0000 7fd993fff700 1 --2- 172.21.15.16:0/437575588 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd9940d1d20 0x7fd9940b55b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:28:03.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.214+0000 7fd9937fe700 1 --2- 172.21.15.16:0/437575588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9940d4060 0x7fd9941f6f00 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7fd988000c00 tx=0x7fd988004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:03.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.214+0000 7fd990ff9700 1 -- 172.21.15.16:0/437575588 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd988004550 con 0x7fd9940d4060 2023-10-16T13:28:03.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.214+0000 7fd990ff9700 1 -- 172.21.15.16:0/437575588 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd98801f070 con 0x7fd9940d4060 2023-10-16T13:28:03.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.214+0000 7fd99a637700 1 -- 172.21.15.16:0/437575588 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9880094a0 con 0x7fd9940d4060 2023-10-16T13:28:03.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.214+0000 7fd990ff9700 1 -- 172.21.15.16:0/437575588 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9880038e0 con 0x7fd9940d4060 2023-10-16T13:28:03.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.214+0000 7fd99a637700 1 -- 172.21.15.16:0/437575588 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9940cac20 con 0x7fd9940d4060 2023-10-16T13:28:03.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.215+0000 7fd99a637700 1 -- 172.21.15.16:0/437575588 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9940cadb0 con 0x7fd9940d4060 2023-10-16T13:28:03.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.216+0000 7fd990ff9700 1 -- 172.21.15.16:0/437575588 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 103) v1 ==== 89332+0+0 (secure 0 0 0) 0x7fd988003a40 con 0x7fd9940d4060 2023-10-16T13:28:03.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.216+0000 7fd990ff9700 1 --2- 172.21.15.16:0/437575588 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7fd98006b020 0x7fd98006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:03.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.216+0000 7fd993fff700 1 -- 172.21.15.16:0/437575588 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7fd98006b020 msgr2=0x7fd98006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3622553136 2023-10-16T13:28:03.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.216+0000 7fd990ff9700 1 -- 172.21.15.16:0/437575588 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 6473+0+0 (secure 0 0 0) 0x7fd988021070 con 0x7fd9940d4060 2023-10-16T13:28:03.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.217+0000 7fd993fff700 1 --2- 172.21.15.16:0/437575588 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7fd98006b020 0x7fd98006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:28:03.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.224+0000 7fd990ff9700 1 -- 172.21.15.16:0/437575588 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd9880bdfe0 con 0x7fd9940d4060 2023-10-16T13:28:03.290 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:28:03.287+0000 7f180423e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T13:28:03.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.418+0000 7fd993fff700 1 -- 172.21.15.16:0/437575588 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7fd98006b020 msgr2=0x7fd98006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3622553136 2023-10-16T13:28:03.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.418+0000 7fd993fff700 1 --2- 172.21.15.16:0/437575588 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7fd98006b020 0x7fd98006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:28:03.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.430+0000 7fd99a637700 1 -- 172.21.15.16:0/437575588 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd9940b1e10 con 0x7fd9940d4060 2023-10-16T13:28:03.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.430+0000 7fd990ff9700 1 -- 172.21.15.16:0/437575588 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v178) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd988055df0 con 0x7fd9940d4060 2023-10-16T13:28:03.433 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-10-16T13:28:03.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.432+0000 7fd99a637700 1 -- 172.21.15.16:0/437575588 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7fd98006b020 msgr2=0x7fd98006d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:28:03.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.432+0000 7fd99a637700 1 --2- 172.21.15.16:0/437575588 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7fd98006b020 0x7fd98006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:03.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.432+0000 7fd99a637700 1 -- 172.21.15.16:0/437575588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9940d4060 msgr2=0x7fd9941f6f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:03.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.432+0000 7fd99a637700 1 --2- 172.21.15.16:0/437575588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9940d4060 0x7fd9941f6f00 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7fd988000c00 tx=0x7fd988004750 comp rx=0 tx=0).stop 2023-10-16T13:28:03.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.432+0000 7fd99a637700 1 -- 172.21.15.16:0/437575588 shutdown_connections 2023-10-16T13:28:03.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.432+0000 7fd99a637700 1 --2- 172.21.15.16:0/437575588 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7fd98006b020 0x7fd98006d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:03.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.432+0000 7fd99a637700 1 --2- 172.21.15.16:0/437575588 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd99415da10 0x7fd9941f7440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:03.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.432+0000 7fd99a637700 1 --2- 172.21.15.16:0/437575588 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd9940d1d20 0x7fd9940b55b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:03.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.432+0000 7fd99a637700 1 --2- 172.21.15.16:0/437575588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9940d4060 0x7fd9941f6f00 unknown :-1 s=CLOSED pgs=1997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:03.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.432+0000 7fd99a637700 1 -- 172.21.15.16:0/437575588 >> 172.21.15.16:0/437575588 conn(0x7fd99409c9f0 msgr2=0x7fd99415de80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:03.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.432+0000 7fd99a637700 1 -- 172.21.15.16:0/437575588 shutdown_connections 2023-10-16T13:28:03.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.432+0000 7fd99a637700 1 -- 172.21.15.16:0/437575588 wait complete. 2023-10-16T13:28:03.447 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T13:28:03.448 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:28:03.445+0000 7f954f950200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T13:28:03.522 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:28:03.519+0000 7f180423e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T13:28:03.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.576+0000 7f8fd2aaf700 1 Processor -- start 2023-10-16T13:28:03.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.576+0000 7f8fd2aaf700 1 -- start start 2023-10-16T13:28:03.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.577+0000 7f8fd2aaf700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8fcc15cd10 0x7f8fcc0cd0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:03.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.577+0000 7f8fd2aaf700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fcc0cc920 con 0x7f8fcc0ca9b0 2023-10-16T13:28:03.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.577+0000 7f8fd2aaf700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fcc15c330 con 0x7f8fcc06e6b0 2023-10-16T13:28:03.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.577+0000 7f8fd2aaf700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fcc15c610 con 0x7f8fcc15cd10 2023-10-16T13:28:03.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.577+0000 7f8fd104c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8fcc15cd10 0x7f8fcc0cd0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:03.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.577+0000 7f8fd084b700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f8fcc0ca9b0 0x7f8fcc15bb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38532/0 (socket says 172.21.15.16:38532) 2023-10-16T13:28:03.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.577+0000 7f8fd084b700 1 -- 172.21.15.16:0/1289080756 learned_addr learned my addr 172.21.15.16:0/1289080756 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:28:03.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.577+0000 7f8fd104c700 1 -- 172.21.15.16:0/1289080756 >> v1:172.21.15.49:6789/0 conn(0x7f8fcc06e6b0 legacy=0x7f8fcc0cad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:03.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.577+0000 7f8fd104c700 1 -- 172.21.15.16:0/1289080756 >> v1:172.21.15.16:6789/0 conn(0x7f8fcc0ca9b0 legacy=0x7f8fcc15bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:03.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.577+0000 7f8fcb7fe700 1 -- 172.21.15.16:0/1289080756 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2253514276 0 0) 0x7f8fcc15c330 con 0x7f8fcc06e6b0 2023-10-16T13:28:03.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.577+0000 7f8fd104c700 1 -- 172.21.15.16:0/1289080756 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fcc15c8f0 con 0x7f8fcc15cd10 2023-10-16T13:28:03.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.577+0000 7f8fd104c700 1 --2- 172.21.15.16:0/1289080756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fcc15cd10 0x7f8fcc0cd0c0 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7f8fc4009d90 tx=0x7f8fc400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f846176e2244223f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:03.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.577+0000 7f8fcb7fe700 1 -- 172.21.15.16:0/1289080756 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc400e070 con 0x7f8fcc15cd10 2023-10-16T13:28:03.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.578+0000 7f8fcb7fe700 1 -- 172.21.15.16:0/1289080756 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fc400bad0 con 0x7f8fcc15cd10 2023-10-16T13:28:03.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.578+0000 7f8fcb7fe700 1 -- 172.21.15.16:0/1289080756 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc400be80 con 0x7f8fcc15cd10 2023-10-16T13:28:03.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.578+0000 7f8fd2aaf700 1 -- 172.21.15.16:0/1289080756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fcc15cd10 msgr2=0x7f8fcc0cd0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:03.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.578+0000 7f8fd2aaf700 1 --2- 172.21.15.16:0/1289080756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fcc15cd10 0x7f8fcc0cd0c0 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7f8fc4009d90 tx=0x7f8fc400b330 comp rx=0 tx=0).stop 2023-10-16T13:28:03.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.578+0000 7f8fd2aaf700 1 -- 172.21.15.16:0/1289080756 shutdown_connections 2023-10-16T13:28:03.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.578+0000 7f8fd2aaf700 1 --2- 172.21.15.16:0/1289080756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fcc15cd10 0x7f8fcc0cd0c0 unknown :-1 s=CLOSED pgs=1999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:03.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.578+0000 7f8fd2aaf700 1 -- 172.21.15.16:0/1289080756 >> 172.21.15.16:0/1289080756 conn(0x7f8fcc0b1ed0 msgr2=0x7f8fcc0b22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:03.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.578+0000 7f8fd2aaf700 1 -- 172.21.15.16:0/1289080756 shutdown_connections 2023-10-16T13:28:03.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.578+0000 7f8fd2aaf700 1 -- 172.21.15.16:0/1289080756 wait complete. 2023-10-16T13:28:03.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.579+0000 7f8fd2aaf700 1 Processor -- start 2023-10-16T13:28:03.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.580+0000 7f8fd2aaf700 1 -- start start 2023-10-16T13:28:03.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.580+0000 7f8fd2aaf700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8fcc06e6b0 0x7f8fcc14c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:03.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.580+0000 7f8fd2aaf700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8fcc0ca9b0 0x7f8fcc0cf050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:03.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.580+0000 7f8fd2aaf700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fcc15cd10 0x7f8fcc15dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:03.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.580+0000 7f8fd2aaf700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fcc1f2d80 con 0x7f8fcc15cd10 2023-10-16T13:28:03.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.580+0000 7f8fd2aaf700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fcc14cce0 con 0x7f8fcc06e6b0 2023-10-16T13:28:03.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.580+0000 7f8fd2aaf700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fcc1f2f90 con 0x7f8fcc0ca9b0 2023-10-16T13:28:03.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.581+0000 7f8fd104c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fcc15cd10 0x7f8fcc15dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:03.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.581+0000 7f8fd104c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fcc15cd10 0x7f8fcc15dcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49268/0 (socket says 172.21.15.16:49268) 2023-10-16T13:28:03.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.581+0000 7f8fd104c700 1 -- 172.21.15.16:0/466415611 learned_addr learned my addr 172.21.15.16:0/466415611 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:03.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.581+0000 7f8fd084b700 1 --2- 172.21.15.16:0/466415611 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8fcc06e6b0 0x7f8fcc14c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:03.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.581+0000 7f8fcbfff700 1 --2- 172.21.15.16:0/466415611 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8fcc0ca9b0 0x7f8fcc0cf050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:03.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.581+0000 7f8fd104c700 1 -- 172.21.15.16:0/466415611 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8fcc0ca9b0 msgr2=0x7f8fcc0cf050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:03.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.581+0000 7f8fd104c700 1 --2- 172.21.15.16:0/466415611 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8fcc0ca9b0 0x7f8fcc0cf050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:03.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.581+0000 7f8fd104c700 1 -- 172.21.15.16:0/466415611 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8fcc06e6b0 msgr2=0x7f8fcc14c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:03.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.581+0000 7f8fd104c700 1 --2- 172.21.15.16:0/466415611 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8fcc06e6b0 0x7f8fcc14c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:03.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.581+0000 7f8fd104c700 1 -- 172.21.15.16:0/466415611 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fc4009a70 con 0x7f8fcc15cd10 2023-10-16T13:28:03.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.581+0000 7f8fd104c700 1 --2- 172.21.15.16:0/466415611 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fcc15cd10 0x7f8fcc15dcc0 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7f8fc4009d60 tx=0x7f8fc4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:03.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.581+0000 7f8fc97fa700 1 -- 172.21.15.16:0/466415611 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc4012630 con 0x7f8fcc15cd10 2023-10-16T13:28:03.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.581+0000 7f8fc97fa700 1 -- 172.21.15.16:0/466415611 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fc4003e90 con 0x7f8fcc15cd10 2023-10-16T13:28:03.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.581+0000 7f8fd2aaf700 1 -- 172.21.15.16:0/466415611 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fcc1f3270 con 0x7f8fcc15cd10 2023-10-16T13:28:03.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.581+0000 7f8fc97fa700 1 -- 172.21.15.16:0/466415611 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc40129f0 con 0x7f8fcc15cd10 2023-10-16T13:28:03.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.581+0000 7f8fd2aaf700 1 -- 172.21.15.16:0/466415611 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fcc1f3830 con 0x7f8fcc15cd10 2023-10-16T13:28:03.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.582+0000 7f8fd2aaf700 1 -- 172.21.15.16:0/466415611 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fcc04ebb0 con 0x7f8fcc15cd10 2023-10-16T13:28:03.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.583+0000 7f8fc97fa700 1 -- 172.21.15.16:0/466415611 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 103) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f8fc4012b50 con 0x7f8fcc15cd10 2023-10-16T13:28:03.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.583+0000 7f8fc97fa700 1 --2- 172.21.15.16:0/466415611 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7f8fb006b020 0x7f8fb006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:03.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.583+0000 7f8fd084b700 1 -- 172.21.15.16:0/466415611 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7f8fb006b020 msgr2=0x7f8fb006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3622553136 2023-10-16T13:28:03.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.583+0000 7f8fc97fa700 1 -- 172.21.15.16:0/466415611 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 6473+0+0 (secure 0 0 0) 0x7f8fc408bdc0 con 0x7f8fcc15cd10 2023-10-16T13:28:03.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.584+0000 7f8fd084b700 1 --2- 172.21.15.16:0/466415611 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7f8fb006b020 0x7f8fb006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:28:03.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.592+0000 7f8fc97fa700 1 -- 172.21.15.16:0/466415611 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8fc405a470 con 0x7f8fcc15cd10 2023-10-16T13:28:03.680 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:28:03.677+0000 7f180423e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T13:28:03.694 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:28:03.691+0000 7f954f950200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T13:28:03.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.785+0000 7f8fd084b700 1 -- 172.21.15.16:0/466415611 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7f8fb006b020 msgr2=0x7f8fb006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3622553136 2023-10-16T13:28:03.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.785+0000 7f8fd084b700 1 --2- 172.21.15.16:0/466415611 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7f8fb006b020 0x7f8fb006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:28:03.799 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:28:03.796+0000 7f180423e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T13:28:03.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.816+0000 7f8fd2aaf700 1 -- 172.21.15.16:0/466415611 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8fcc0b2630 con 0x7f8fcc15cd10 2023-10-16T13:28:03.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.817+0000 7f8fc97fa700 1 -- 172.21.15.16:0/466415611 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v178) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8fc405a000 con 0x7f8fcc15cd10 2023-10-16T13:28:03.819 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-10-16T13:28:03.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.819+0000 7f8fd2aaf700 1 -- 172.21.15.16:0/466415611 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7f8fb006b020 msgr2=0x7f8fb006d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:28:03.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.819+0000 7f8fd2aaf700 1 --2- 172.21.15.16:0/466415611 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7f8fb006b020 0x7f8fb006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:03.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.819+0000 7f8fd2aaf700 1 -- 172.21.15.16:0/466415611 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fcc15cd10 msgr2=0x7f8fcc15dcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:03.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.819+0000 7f8fd2aaf700 1 --2- 172.21.15.16:0/466415611 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fcc15cd10 0x7f8fcc15dcc0 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7f8fc4009d60 tx=0x7f8fc4004ad0 comp rx=0 tx=0).stop 2023-10-16T13:28:03.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.819+0000 7f8fd2aaf700 1 -- 172.21.15.16:0/466415611 shutdown_connections 2023-10-16T13:28:03.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.819+0000 7f8fd2aaf700 1 --2- 172.21.15.16:0/466415611 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7f8fb006b020 0x7f8fb006d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:03.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.819+0000 7f8fd2aaf700 1 --2- 172.21.15.16:0/466415611 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fcc15cd10 0x7f8fcc15dcc0 unknown :-1 s=CLOSED pgs=2000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:03.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.819+0000 7f8fd2aaf700 1 --2- 172.21.15.16:0/466415611 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8fcc06e6b0 0x7f8fcc14c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:03.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.819+0000 7f8fd2aaf700 1 --2- 172.21.15.16:0/466415611 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8fcc0ca9b0 0x7f8fcc0cf050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:03.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.819+0000 7f8fd2aaf700 1 -- 172.21.15.16:0/466415611 >> 172.21.15.16:0/466415611 conn(0x7f8fcc0b1ed0 msgr2=0x7f8fcc0b4fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:03.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.819+0000 7f8fd2aaf700 1 -- 172.21.15.16:0/466415611 shutdown_connections 2023-10-16T13:28:03.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.819+0000 7f8fd2aaf700 1 -- 172.21.15.16:0/466415611 wait complete. 2023-10-16T13:28:03.834 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:28:03.871 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:28:03.869+0000 7f954f950200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T13:28:03.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.958+0000 7fe7cae6b700 1 Processor -- start 2023-10-16T13:28:03.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.958+0000 7fe7cae6b700 1 -- start start 2023-10-16T13:28:03.962 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:28:03.958+0000 7f180423e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T13:28:03.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.959+0000 7fe7cae6b700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fe7c415b630 0x7fe7c40d0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:03.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.959+0000 7fe7cae6b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe7c40ce4f0 0x7fe7c40cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:03.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.959+0000 7fe7cae6b700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7c40b0970 con 0x7fe7c415ba10 2023-10-16T13:28:03.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.959+0000 7fe7cae6b700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7c40cd690 con 0x7fe7c40ce4f0 2023-10-16T13:28:03.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.959+0000 7fe7cae6b700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7c40cd970 con 0x7fe7c415b630 2023-10-16T13:28:03.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.959+0000 7fe7c9408700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe7c40ce4f0 0x7fe7c40cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:03.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.959+0000 7fe7c9408700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe7c40ce4f0 0x7fe7c40cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49278/0 (socket says 172.21.15.16:49278) 2023-10-16T13:28:03.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.959+0000 7fe7c9408700 1 -- 172.21.15.16:0/3643459641 learned_addr learned my addr 172.21.15.16:0/3643459641 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:03.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.959+0000 7fe7c8c07700 1 --2- 172.21.15.16:0/3643459641 >> v2:172.21.15.49:3300/0 conn(0x7fe7c415b630 0x7fe7c40d0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:03.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.959+0000 7fe7c9408700 1 -- 172.21.15.16:0/3643459641 >> v2:172.21.15.49:3300/0 conn(0x7fe7c415b630 msgr2=0x7fe7c40d0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:28:03.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.959+0000 7fe7c9408700 1 --2- 172.21.15.16:0/3643459641 >> v2:172.21.15.49:3300/0 conn(0x7fe7c415b630 0x7fe7c40d0380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:03.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.959+0000 7fe7c9408700 1 -- 172.21.15.16:0/3643459641 >> v1:172.21.15.49:6789/0 conn(0x7fe7c415ba10 legacy=0x7fe7c40cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:03.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.959+0000 7fe7c9408700 1 -- 172.21.15.16:0/3643459641 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7c40b0c30 con 0x7fe7c40ce4f0 2023-10-16T13:28:03.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.960+0000 7fe7c9408700 1 --2- 172.21.15.16:0/3643459641 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7c40ce4f0 0x7fe7c40cf290 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7fe7c4067350 tx=0x7fe7bc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af2052f429cd975b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:03.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.960+0000 7fe7c37fe700 1 -- 172.21.15.16:0/3643459641 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7bc00e070 con 0x7fe7c40ce4f0 2023-10-16T13:28:03.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.960+0000 7fe7c37fe700 1 -- 172.21.15.16:0/3643459641 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7bc00b910 con 0x7fe7c40ce4f0 2023-10-16T13:28:03.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.960+0000 7fe7c37fe700 1 -- 172.21.15.16:0/3643459641 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7bc00bcc0 con 0x7fe7c40ce4f0 2023-10-16T13:28:03.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.960+0000 7fe7cae6b700 1 -- 172.21.15.16:0/3643459641 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7c40ce4f0 msgr2=0x7fe7c40cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:03.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.960+0000 7fe7cae6b700 1 --2- 172.21.15.16:0/3643459641 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7c40ce4f0 0x7fe7c40cf290 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7fe7c4067350 tx=0x7fe7bc00b330 comp rx=0 tx=0).stop 2023-10-16T13:28:03.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.960+0000 7fe7cae6b700 1 -- 172.21.15.16:0/3643459641 shutdown_connections 2023-10-16T13:28:03.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.960+0000 7fe7cae6b700 1 --2- 172.21.15.16:0/3643459641 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7c40ce4f0 0x7fe7c40cf290 unknown :-1 s=CLOSED pgs=2001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:03.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.960+0000 7fe7cae6b700 1 --2- 172.21.15.16:0/3643459641 >> v2:172.21.15.49:3300/0 conn(0x7fe7c415b630 0x7fe7c40d0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:03.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.960+0000 7fe7cae6b700 1 -- 172.21.15.16:0/3643459641 >> 172.21.15.16:0/3643459641 conn(0x7fe7c40d14b0 msgr2=0x7fe7c40d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:03.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.961+0000 7fe7cae6b700 1 -- 172.21.15.16:0/3643459641 shutdown_connections 2023-10-16T13:28:03.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.961+0000 7fe7cae6b700 1 -- 172.21.15.16:0/3643459641 wait complete. 2023-10-16T13:28:03.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.962+0000 7fe7cae6b700 1 Processor -- start 2023-10-16T13:28:03.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.962+0000 7fe7cae6b700 1 -- start start 2023-10-16T13:28:03.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.963+0000 7fe7cae6b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7c40ce4f0 0x7fe7c40c8580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:03.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.963+0000 7fe7cae6b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7c415b630 0x7fe7c41508b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:03.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.963+0000 7fe7cae6b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7c415ba10 0x7fe7c406eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:03.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.963+0000 7fe7cae6b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7c40b54d0 con 0x7fe7c40ce4f0 2023-10-16T13:28:03.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.963+0000 7fe7cae6b700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7c406c350 con 0x7fe7c415b630 2023-10-16T13:28:03.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.963+0000 7fe7cae6b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7c406ba20 con 0x7fe7c415ba10 2023-10-16T13:28:03.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.963+0000 7fe7c8c07700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7c40ce4f0 0x7fe7c40c8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:03.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.963+0000 7fe7c8c07700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7c40ce4f0 0x7fe7c40c8580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49280/0 (socket says 172.21.15.16:49280) 2023-10-16T13:28:03.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.963+0000 7fe7c8c07700 1 -- 172.21.15.16:0/1354517915 learned_addr learned my addr 172.21.15.16:0/1354517915 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:03.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.963+0000 7fe7c3fff700 1 --2- 172.21.15.16:0/1354517915 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7c415b630 0x7fe7c41508b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:03.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.963+0000 7fe7c9408700 1 --2- 172.21.15.16:0/1354517915 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7c415ba10 0x7fe7c406eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:03.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.963+0000 7fe7c8c07700 1 -- 172.21.15.16:0/1354517915 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7c415ba10 msgr2=0x7fe7c406eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:03.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.963+0000 7fe7c8c07700 1 --2- 172.21.15.16:0/1354517915 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7c415ba10 0x7fe7c406eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:03.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.963+0000 7fe7c8c07700 1 -- 172.21.15.16:0/1354517915 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7c415b630 msgr2=0x7fe7c41508b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:03.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.963+0000 7fe7c8c07700 1 --2- 172.21.15.16:0/1354517915 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7c415b630 0x7fe7c41508b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:03.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.963+0000 7fe7c8c07700 1 -- 172.21.15.16:0/1354517915 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7bc009b30 con 0x7fe7c40ce4f0 2023-10-16T13:28:03.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.963+0000 7fe7c8c07700 1 --2- 172.21.15.16:0/1354517915 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7c40ce4f0 0x7fe7c40c8580 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7fe7b800e9e0 tx=0x7fe7b800ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:03.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.963+0000 7fe7c17fa700 1 -- 172.21.15.16:0/1354517915 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7b8009710 con 0x7fe7c40ce4f0 2023-10-16T13:28:03.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.963+0000 7fe7c17fa700 1 -- 172.21.15.16:0/1354517915 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7b800ee80 con 0x7fe7c40ce4f0 2023-10-16T13:28:03.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.963+0000 7fe7cae6b700 1 -- 172.21.15.16:0/1354517915 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7c406bd00 con 0x7fe7c40ce4f0 2023-10-16T13:28:03.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.963+0000 7fe7c17fa700 1 -- 172.21.15.16:0/1354517915 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7b800ca50 con 0x7fe7c40ce4f0 2023-10-16T13:28:03.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.964+0000 7fe7cae6b700 1 -- 172.21.15.16:0/1354517915 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7c40c8ac0 con 0x7fe7c40ce4f0 2023-10-16T13:28:03.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.964+0000 7fe7cae6b700 1 -- 172.21.15.16:0/1354517915 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7c404ebb0 con 0x7fe7c40ce4f0 2023-10-16T13:28:03.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.965+0000 7fe7c17fa700 1 -- 172.21.15.16:0/1354517915 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 103) v1 ==== 89332+0+0 (secure 0 0 0) 0x7fe7b800cbb0 con 0x7fe7c40ce4f0 2023-10-16T13:28:03.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.966+0000 7fe7c17fa700 1 --2- 172.21.15.16:0/1354517915 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7fe7a806b020 0x7fe7a806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:03.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.966+0000 7fe7c3fff700 1 -- 172.21.15.16:0/1354517915 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7fe7a806b020 msgr2=0x7fe7a806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3622553136 2023-10-16T13:28:03.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.966+0000 7fe7c17fa700 1 -- 172.21.15.16:0/1354517915 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 6473+0+0 (secure 0 0 0) 0x7fe7b8089a40 con 0x7fe7c40ce4f0 2023-10-16T13:28:03.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.966+0000 7fe7c3fff700 1 --2- 172.21.15.16:0/1354517915 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7fe7a806b020 0x7fe7a806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:28:03.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:03.970+0000 7fe7c17fa700 1 -- 172.21.15.16:0/1354517915 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe7b80546d0 con 0x7fe7c40ce4f0 2023-10-16T13:28:03.999 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:28:03.997+0000 7f954f950200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T13:28:04.166 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:28:04.164+0000 7f954f950200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T13:28:04.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.167+0000 7fe7c3fff700 1 -- 172.21.15.16:0/1354517915 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7fe7a806b020 msgr2=0x7fe7a806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3622553136 2023-10-16T13:28:04.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.167+0000 7fe7c3fff700 1 --2- 172.21.15.16:0/1354517915 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7fe7a806b020 0x7fe7a806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:28:04.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.226+0000 7fe7cae6b700 1 -- 172.21.15.16:0/1354517915 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe7c40d05a0 con 0x7fe7c40ce4f0 2023-10-16T13:28:04.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.227+0000 7fe7c17fa700 1 -- 172.21.15.16:0/1354517915 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+4917 (secure 0 0 0) 0x7fe7b8021090 con 0x7fe7c40ce4f0 2023-10-16T13:28:04.229 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:28:04.230 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":86,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7964,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/4012137577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":4012137577},{"type":"v1","addr":"172.21.15.16:6837","nonce":4012137577}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81},{"gid":7970,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/3742568047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":3742568047},{"type":"v1","addr":"172.21.15.49:6835","nonce":3742568047}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81},{"gid":9756,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/3838743980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":3838743980},{"type":"v1","addr":"172.21.15.16:6836","nonce":3838743980}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:27:41.561818+0000","modified":"2023-10-16T13:27:44.572793+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9774},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9774":{"gid":9774,"name":"d","rank":0,"incarnation":82,"state":"up:active","state_seq":3,"addr":"172.21.15.49:6834/2218212001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":2218212001},{"type":"v1","addr":"172.21.15.49:6834","nonce":2218212001}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2023-10-16T13:28:04.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.229+0000 7fe7cae6b700 1 -- 172.21.15.16:0/1354517915 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7fe7a806b020 msgr2=0x7fe7a806d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:28:04.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.229+0000 7fe7cae6b700 1 --2- 172.21.15.16:0/1354517915 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7fe7a806b020 0x7fe7a806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:04.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.229+0000 7fe7cae6b700 1 -- 172.21.15.16:0/1354517915 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7c40ce4f0 msgr2=0x7fe7c40c8580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:04.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.229+0000 7fe7cae6b700 1 --2- 172.21.15.16:0/1354517915 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7c40ce4f0 0x7fe7c40c8580 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7fe7b800e9e0 tx=0x7fe7b800ecf0 comp rx=0 tx=0).stop 2023-10-16T13:28:04.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.229+0000 7fe7cae6b700 1 -- 172.21.15.16:0/1354517915 shutdown_connections 2023-10-16T13:28:04.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.229+0000 7fe7cae6b700 1 --2- 172.21.15.16:0/1354517915 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7fe7a806b020 0x7fe7a806d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:04.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.229+0000 7fe7cae6b700 1 --2- 172.21.15.16:0/1354517915 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7c415ba10 0x7fe7c406eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:04.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.229+0000 7fe7cae6b700 1 --2- 172.21.15.16:0/1354517915 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7c40ce4f0 0x7fe7c40c8580 unknown :-1 s=CLOSED pgs=2002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:04.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.229+0000 7fe7cae6b700 1 --2- 172.21.15.16:0/1354517915 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7c415b630 0x7fe7c41508b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:04.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.229+0000 7fe7cae6b700 1 -- 172.21.15.16:0/1354517915 >> 172.21.15.16:0/1354517915 conn(0x7fe7c40d14b0 msgr2=0x7fe7c40b00c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:04.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.229+0000 7fe7cae6b700 1 -- 172.21.15.16:0/1354517915 shutdown_connections 2023-10-16T13:28:04.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.229+0000 7fe7cae6b700 1 -- 172.21.15.16:0/1354517915 wait complete. 2023-10-16T13:28:04.235 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:28:04.231+0000 7f180423e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T13:28:04.235 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 86 2023-10-16T13:28:04.245 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:28:04.401 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:28:04.399+0000 7f180423e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T13:28:04.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.412+0000 7fa1c29b9700 1 Processor -- start 2023-10-16T13:28:04.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.412+0000 7fa1c29b9700 1 -- start start 2023-10-16T13:28:04.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.412+0000 7fa1c29b9700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fa1bc0cdfa0 0x7fa1bc0b2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:04.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.412+0000 7fa1c29b9700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa1bc0ce380 0x7fa1bc0b2980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:04.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.412+0000 7fa1c29b9700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1bc0d0fe0 con 0x7fa1bc0d04a0 2023-10-16T13:28:04.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.412+0000 7fa1c29b9700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1bc0d1210 con 0x7fa1bc0cdfa0 2023-10-16T13:28:04.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.412+0000 7fa1c29b9700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1bc0cbc30 con 0x7fa1bc0ce380 2023-10-16T13:28:04.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.413+0000 7fa1bb7fe700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa1bc0ce380 0x7fa1bc0b2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:04.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.413+0000 7fa1bbfff700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fa1bc0cdfa0 0x7fa1bc0b2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:04.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.413+0000 7fa1c0f56700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7fa1bc0d04a0 0x7fa1bc0d08d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:45750/0 (socket says 172.21.15.16:45750) 2023-10-16T13:28:04.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.413+0000 7fa1c0f56700 1 -- 172.21.15.16:0/1808753959 learned_addr learned my addr 172.21.15.16:0/1808753959 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:28:04.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.413+0000 7fa1bb7fe700 1 --2- 172.21.15.16:0/1808753959 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa1bc0ce380 0x7fa1bc0b2980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:37550/0 (socket says 172.21.15.16:37550) 2023-10-16T13:28:04.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.413+0000 7fa1bb7fe700 1 -- 172.21.15.16:0/1808753959 >> v2:172.21.15.49:3300/0 conn(0x7fa1bc0cdfa0 msgr2=0x7fa1bc0b2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:28:04.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.413+0000 7fa1bb7fe700 1 --2- 172.21.15.16:0/1808753959 >> v2:172.21.15.49:3300/0 conn(0x7fa1bc0cdfa0 0x7fa1bc0b2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:04.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.413+0000 7fa1bb7fe700 1 -- 172.21.15.16:0/1808753959 >> v1:172.21.15.49:6789/0 conn(0x7fa1bc0d04a0 legacy=0x7fa1bc0d08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:04.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.413+0000 7fa1bb7fe700 1 -- 172.21.15.16:0/1808753959 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1bc0cbf10 con 0x7fa1bc0ce380 2023-10-16T13:28:04.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.413+0000 7fa1bbfff700 1 --2- 172.21.15.16:0/1808753959 >> v2:172.21.15.49:3300/0 conn(0x7fa1bc0cdfa0 0x7fa1bc0b2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:28:04.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.413+0000 7fa1bb7fe700 1 --2- 172.21.15.16:0/1808753959 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa1bc0ce380 0x7fa1bc0b2980 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa1b0009840 tx=0x7fa1b0009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e9e7a5ed26e1ff92 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:04.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.413+0000 7fa1baffd700 1 -- 172.21.15.16:0/1808753959 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1b0011830 con 0x7fa1bc0ce380 2023-10-16T13:28:04.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.413+0000 7fa1baffd700 1 -- 172.21.15.16:0/1808753959 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1b0011990 con 0x7fa1bc0ce380 2023-10-16T13:28:04.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.413+0000 7fa1baffd700 1 -- 172.21.15.16:0/1808753959 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1b0011d20 con 0x7fa1bc0ce380 2023-10-16T13:28:04.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.414+0000 7fa1c29b9700 1 -- 172.21.15.16:0/1808753959 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa1bc0ce380 msgr2=0x7fa1bc0b2980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:04.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.414+0000 7fa1c29b9700 1 --2- 172.21.15.16:0/1808753959 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa1bc0ce380 0x7fa1bc0b2980 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa1b0009840 tx=0x7fa1b0009b50 comp rx=0 tx=0).stop 2023-10-16T13:28:04.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.414+0000 7fa1c29b9700 1 -- 172.21.15.16:0/1808753959 shutdown_connections 2023-10-16T13:28:04.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.414+0000 7fa1c29b9700 1 --2- 172.21.15.16:0/1808753959 >> v2:172.21.15.49:3300/0 conn(0x7fa1bc0cdfa0 0x7fa1bc0b2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:04.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.414+0000 7fa1c29b9700 1 --2- 172.21.15.16:0/1808753959 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa1bc0ce380 0x7fa1bc0b2980 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:04.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.414+0000 7fa1c29b9700 1 -- 172.21.15.16:0/1808753959 >> 172.21.15.16:0/1808753959 conn(0x7fa1bc0d14b0 msgr2=0x7fa1bc0d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:04.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.414+0000 7fa1c29b9700 1 -- 172.21.15.16:0/1808753959 shutdown_connections 2023-10-16T13:28:04.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.414+0000 7fa1c29b9700 1 -- 172.21.15.16:0/1808753959 wait complete. 2023-10-16T13:28:04.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.416+0000 7fa1c29b9700 1 Processor -- start 2023-10-16T13:28:04.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.416+0000 7fa1c29b9700 1 -- start start 2023-10-16T13:28:04.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.417+0000 7fa1c29b9700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa1bc0cdfa0 0x7fa1bc1528b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:04.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.417+0000 7fa1c29b9700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa1bc0ce380 0x7fa1bc0cadc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:04.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.417+0000 7fa1c29b9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1bc0d04a0 0x7fa1bc0cb300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:04.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.417+0000 7fa1c29b9700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1bc152df0 con 0x7fa1bc0d04a0 2023-10-16T13:28:04.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.417+0000 7fa1c29b9700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1bc0cb840 con 0x7fa1bc0cdfa0 2023-10-16T13:28:04.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.417+0000 7fa1c29b9700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1bc0c62d0 con 0x7fa1bc0ce380 2023-10-16T13:28:04.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.417+0000 7fa1bbfff700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa1bc0cdfa0 0x7fa1bc1528b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:04.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.417+0000 7fa1c0f56700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1bc0d04a0 0x7fa1bc0cb300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:04.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.417+0000 7fa1bb7fe700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa1bc0ce380 0x7fa1bc0cadc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:04.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.417+0000 7fa1bbfff700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa1bc0cdfa0 0x7fa1bc1528b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:48358/0 (socket says 172.21.15.16:48358) 2023-10-16T13:28:04.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.417+0000 7fa1bbfff700 1 -- 172.21.15.16:0/1444182047 learned_addr learned my addr 172.21.15.16:0/1444182047 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:04.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.417+0000 7fa1bbfff700 1 -- 172.21.15.16:0/1444182047 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa1bc0ce380 msgr2=0x7fa1bc0cadc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:04.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.417+0000 7fa1bbfff700 1 --2- 172.21.15.16:0/1444182047 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa1bc0ce380 0x7fa1bc0cadc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:04.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.417+0000 7fa1bbfff700 1 -- 172.21.15.16:0/1444182047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1bc0d04a0 msgr2=0x7fa1bc0cb300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:04.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.417+0000 7fa1bbfff700 1 --2- 172.21.15.16:0/1444182047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1bc0d04a0 0x7fa1bc0cb300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:04.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.417+0000 7fa1bbfff700 1 -- 172.21.15.16:0/1444182047 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1ac0094a0 con 0x7fa1bc0cdfa0 2023-10-16T13:28:04.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.417+0000 7fa1c0f56700 1 --2- 172.21.15.16:0/1444182047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1bc0d04a0 0x7fa1bc0cb300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:28:04.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.418+0000 7fa1bbfff700 1 --2- 172.21.15.16:0/1444182047 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa1bc0cdfa0 0x7fa1bc1528b0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac00ebe0 tx=0x7fa1ac00eef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:04.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.418+0000 7fa1b8ff9700 1 -- 172.21.15.16:0/1444182047 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1ac00c850 con 0x7fa1bc0cdfa0 2023-10-16T13:28:04.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.418+0000 7fa1b8ff9700 1 -- 172.21.15.16:0/1444182047 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1ac00c9b0 con 0x7fa1bc0cdfa0 2023-10-16T13:28:04.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.418+0000 7fa1c29b9700 1 -- 172.21.15.16:0/1444182047 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1b00094a0 con 0x7fa1bc0cdfa0 2023-10-16T13:28:04.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.418+0000 7fa1b8ff9700 1 -- 172.21.15.16:0/1444182047 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1ac00cd40 con 0x7fa1bc0cdfa0 2023-10-16T13:28:04.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.418+0000 7fa1c29b9700 1 -- 172.21.15.16:0/1444182047 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1bc0c6890 con 0x7fa1bc0cdfa0 2023-10-16T13:28:04.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.419+0000 7fa1c29b9700 1 -- 172.21.15.16:0/1444182047 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1bc04ebb0 con 0x7fa1bc0cdfa0 2023-10-16T13:28:04.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.419+0000 7fa1b8ff9700 1 -- 172.21.15.16:0/1444182047 <== mon.1 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 103) v1 ==== 89332+0+0 (secure 0 0 0) 0x7fa1ac013470 con 0x7fa1bc0cdfa0 2023-10-16T13:28:04.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.420+0000 7fa1b8ff9700 1 --2- 172.21.15.16:0/1444182047 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7fa1a806b020 0x7fa1a806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:04.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.420+0000 7fa1bb7fe700 1 -- 172.21.15.16:0/1444182047 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7fa1a806b020 msgr2=0x7fa1a806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3622553136 2023-10-16T13:28:04.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.420+0000 7fa1b8ff9700 1 -- 172.21.15.16:0/1444182047 <== mon.1 v2:172.21.15.49:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 6473+0+0 (secure 0 0 0) 0x7fa1ac089b40 con 0x7fa1bc0cdfa0 2023-10-16T13:28:04.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.420+0000 7fa1bb7fe700 1 --2- 172.21.15.16:0/1444182047 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7fa1a806b020 0x7fa1a806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:28:04.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.424+0000 7fa1b8ff9700 1 -- 172.21.15.16:0/1444182047 <== mon.1 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa1ac054760 con 0x7fa1bc0cdfa0 2023-10-16T13:28:04.448 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:28:04.446+0000 7f954f950200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T13:28:04.622 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:28:04.620+0000 7f954f950200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T13:28:04.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.621+0000 7fa1bb7fe700 1 -- 172.21.15.16:0/1444182047 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7fa1a806b020 msgr2=0x7fa1a806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3622553136 2023-10-16T13:28:04.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.621+0000 7fa1bb7fe700 1 --2- 172.21.15.16:0/1444182047 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7fa1a806b020 0x7fa1a806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:28:04.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.631+0000 7fa1c29b9700 1 -- 172.21.15.16:0/1444182047 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa1bc070660 con 0x7fa1bc0cdfa0 2023-10-16T13:28:04.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.633+0000 7fa1b8ff9700 1 -- 172.21.15.16:0/1444182047 <== mon.1 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v178) v1 ==== 74+0+17273 (secure 0 0 0) 0x7fa1ac00cea0 con 0x7fa1bc0cdfa0 2023-10-16T13:28:04.635 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:28:04.636 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":178,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:27:56.552748+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2023-10-16T13:27:40.032163+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"176","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2023-10-16T13:27:40.523975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"177","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/472482039":"2023-10-17T13:27:56.552715+0000","172.21.15.16:0/1822527107":"2023-10-17T13:27:56.552715+0000","172.21.15.16:0/3919013455":"2023-10-17T13:27:56.552715+0000","172.21.15.16:6833/2329471558":"2023-10-17T13:27:56.552715+0000","172.21.15.16:6832/2329471558":"2023-10-17T13:27:56.552715+0000","172.21.15.16:0/508475708":"2023-10-17T13:27:56.552715+0000","172.21.15.16:0/353895884":"2023-10-17T13:27:56.552715+0000","172.21.15.16:0/2733317665":"2023-10-17T13:27:56.552715+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:28:04.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.635+0000 7fa1c29b9700 1 -- 172.21.15.16:0/1444182047 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7fa1a806b020 msgr2=0x7fa1a806d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:28:04.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.635+0000 7fa1c29b9700 1 --2- 172.21.15.16:0/1444182047 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7fa1a806b020 0x7fa1a806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:04.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.635+0000 7fa1c29b9700 1 -- 172.21.15.16:0/1444182047 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa1bc0cdfa0 msgr2=0x7fa1bc1528b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:04.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.635+0000 7fa1c29b9700 1 --2- 172.21.15.16:0/1444182047 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa1bc0cdfa0 0x7fa1bc1528b0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac00ebe0 tx=0x7fa1ac00eef0 comp rx=0 tx=0).stop 2023-10-16T13:28:04.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.635+0000 7fa1c29b9700 1 -- 172.21.15.16:0/1444182047 shutdown_connections 2023-10-16T13:28:04.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.635+0000 7fa1c29b9700 1 --2- 172.21.15.16:0/1444182047 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7fa1a806b020 0x7fa1a806d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:04.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.635+0000 7fa1c29b9700 1 --2- 172.21.15.16:0/1444182047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1bc0d04a0 0x7fa1bc0cb300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:04.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.635+0000 7fa1c29b9700 1 --2- 172.21.15.16:0/1444182047 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa1bc0cdfa0 0x7fa1bc1528b0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:04.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.635+0000 7fa1c29b9700 1 --2- 172.21.15.16:0/1444182047 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa1bc0ce380 0x7fa1bc0cadc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:04.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.635+0000 7fa1c29b9700 1 -- 172.21.15.16:0/1444182047 >> 172.21.15.16:0/1444182047 conn(0x7fa1bc0d14b0 msgr2=0x7fa1bc06eba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:04.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.635+0000 7fa1c29b9700 1 -- 172.21.15.16:0/1444182047 shutdown_connections 2023-10-16T13:28:04.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.635+0000 7fa1c29b9700 1 -- 172.21.15.16:0/1444182047 wait complete. 2023-10-16T13:28:04.650 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-16T13:28:04.651 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:28:04.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.775+0000 7f433c744700 1 Processor -- start 2023-10-16T13:28:04.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.775+0000 7f433c744700 1 -- start start 2023-10-16T13:28:04.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.776+0000 7f433c744700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f43340cf4a0 0x7f43340726e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:04.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.776+0000 7f433c744700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f433415bc80 con 0x7f43340b5a00 2023-10-16T13:28:04.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.776+0000 7f433c744700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43340b5e70 con 0x7f43340cf880 2023-10-16T13:28:04.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.776+0000 7f433c744700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4334072c20 con 0x7f43340cf4a0 2023-10-16T13:28:04.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.776+0000 7f433ace1700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f43340b5a00 0x7f433415a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38584/0 (socket says 172.21.15.16:38584) 2023-10-16T13:28:04.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.776+0000 7f433ace1700 1 -- 172.21.15.16:0/1198056113 learned_addr learned my addr 172.21.15.16:0/1198056113 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:28:04.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.776+0000 7f433a4e0700 1 --2- 172.21.15.16:0/1198056113 >> v2:172.21.15.49:3300/0 conn(0x7f43340cf4a0 0x7f43340726e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:04.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.776+0000 7f43394de700 1 -- 172.21.15.16:0/1198056113 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4130726099 0 0) 0x7f433415bc80 con 0x7f43340b5a00 2023-10-16T13:28:04.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.776+0000 7f43394de700 1 -- 172.21.15.16:0/1198056113 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43200035b0 con 0x7f43340b5a00 2023-10-16T13:28:04.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.776+0000 7f43394de700 1 -- 172.21.15.16:0/1198056113 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2834628149 0 0) 0x7f43340b5e70 con 0x7f43340cf880 2023-10-16T13:28:04.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.776+0000 7f43394de700 1 -- 172.21.15.16:0/1198056113 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f433415bc80 con 0x7f43340cf880 2023-10-16T13:28:04.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.777+0000 7f433a4e0700 1 -- 172.21.15.16:0/1198056113 >> v1:172.21.15.49:6789/0 conn(0x7f43340cf880 legacy=0x7f43340b52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:04.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.777+0000 7f433a4e0700 1 -- 172.21.15.16:0/1198056113 >> v1:172.21.15.16:6789/0 conn(0x7f43340b5a00 legacy=0x7f433415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:04.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.777+0000 7f433a4e0700 1 -- 172.21.15.16:0/1198056113 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f433415b070 con 0x7f43340cf4a0 2023-10-16T13:28:04.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.777+0000 7f433a4e0700 1 --2- 172.21.15.16:0/1198056113 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f43340cf4a0 0x7f43340726e0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f4324009840 tx=0x7f4324009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7efe67e564b64359 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:04.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.777+0000 7f43394de700 1 -- 172.21.15.16:0/1198056113 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4324011830 con 0x7f43340cf4a0 2023-10-16T13:28:04.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.777+0000 7f43394de700 1 -- 172.21.15.16:0/1198056113 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4324011990 con 0x7f43340cf4a0 2023-10-16T13:28:04.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.777+0000 7f43394de700 1 -- 172.21.15.16:0/1198056113 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4324011d20 con 0x7f43340cf4a0 2023-10-16T13:28:04.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.778+0000 7f433c744700 1 -- 172.21.15.16:0/1198056113 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f43340cf4a0 msgr2=0x7f43340726e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:04.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.778+0000 7f433c744700 1 --2- 172.21.15.16:0/1198056113 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f43340cf4a0 0x7f43340726e0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f4324009840 tx=0x7f4324009b50 comp rx=0 tx=0).stop 2023-10-16T13:28:04.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.778+0000 7f433c744700 1 -- 172.21.15.16:0/1198056113 shutdown_connections 2023-10-16T13:28:04.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.778+0000 7f433c744700 1 --2- 172.21.15.16:0/1198056113 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f43340cf4a0 0x7f43340726e0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:04.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.778+0000 7f433c744700 1 -- 172.21.15.16:0/1198056113 >> 172.21.15.16:0/1198056113 conn(0x7f43340b1fd0 msgr2=0x7f43340b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:04.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.778+0000 7f433c744700 1 -- 172.21.15.16:0/1198056113 shutdown_connections 2023-10-16T13:28:04.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.778+0000 7f433c744700 1 -- 172.21.15.16:0/1198056113 wait complete. 2023-10-16T13:28:04.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.779+0000 7f433c744700 1 Processor -- start 2023-10-16T13:28:04.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.779+0000 7f433c744700 1 -- start start 2023-10-16T13:28:04.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.780+0000 7f433c744700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43340b5a00 0x7f433414aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:04.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.780+0000 7f433c744700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f43340cf4a0 0x7f43341448d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:04.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.780+0000 7f433c744700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f43340cf880 0x7f433406a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:04.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.780+0000 7f433c744700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43340c5d10 con 0x7f43340b5a00 2023-10-16T13:28:04.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.780+0000 7f433c744700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43340c4010 con 0x7f43340cf4a0 2023-10-16T13:28:04.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.780+0000 7f433c744700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43340c42f0 con 0x7f43340cf880 2023-10-16T13:28:04.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.780+0000 7f433a4e0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43340b5a00 0x7f433414aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:04.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.780+0000 7f433a4e0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43340b5a00 0x7f433414aac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49312/0 (socket says 172.21.15.16:49312) 2023-10-16T13:28:04.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.780+0000 7f433a4e0700 1 -- 172.21.15.16:0/3773636283 learned_addr learned my addr 172.21.15.16:0/3773636283 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:04.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.780+0000 7f4339cdf700 1 --2- 172.21.15.16:0/3773636283 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f43340cf4a0 0x7f43341448d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:04.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.780+0000 7f433ace1700 1 --2- 172.21.15.16:0/3773636283 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f43340cf880 0x7f433406a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:04.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.780+0000 7f433a4e0700 1 -- 172.21.15.16:0/3773636283 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f43340cf880 msgr2=0x7f433406a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:04.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.780+0000 7f433a4e0700 1 --2- 172.21.15.16:0/3773636283 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f43340cf880 0x7f433406a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:04.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.780+0000 7f433a4e0700 1 -- 172.21.15.16:0/3773636283 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f43340cf4a0 msgr2=0x7f43341448d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:04.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.780+0000 7f433a4e0700 1 --2- 172.21.15.16:0/3773636283 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f43340cf4a0 0x7f43341448d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:04.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.780+0000 7f433a4e0700 1 -- 172.21.15.16:0/3773636283 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43240094a0 con 0x7f43340b5a00 2023-10-16T13:28:04.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.780+0000 7f433a4e0700 1 --2- 172.21.15.16:0/3773636283 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43340b5a00 0x7f433414aac0 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7f43240090b0 tx=0x7f4324004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:04.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.781+0000 7f432affd700 1 -- 172.21.15.16:0/3773636283 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f432401a6e0 con 0x7f43340b5a00 2023-10-16T13:28:04.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.781+0000 7f432affd700 1 -- 172.21.15.16:0/3773636283 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f432401a840 con 0x7f43340b5a00 2023-10-16T13:28:04.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.781+0000 7f433c744700 1 -- 172.21.15.16:0/3773636283 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43340c5fb0 con 0x7f43340b5a00 2023-10-16T13:28:04.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.781+0000 7f432affd700 1 -- 172.21.15.16:0/3773636283 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f432401abb0 con 0x7f43340b5a00 2023-10-16T13:28:04.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.781+0000 7f433c744700 1 -- 172.21.15.16:0/3773636283 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4334144e10 con 0x7f43340b5a00 2023-10-16T13:28:04.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.782+0000 7f433c744700 1 -- 172.21.15.16:0/3773636283 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43340c5b70 con 0x7f43340b5a00 2023-10-16T13:28:04.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.782+0000 7f432affd700 1 -- 172.21.15.16:0/3773636283 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 103) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f4324022070 con 0x7f43340b5a00 2023-10-16T13:28:04.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.783+0000 7f432affd700 1 --2- 172.21.15.16:0/3773636283 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7f431806b020 0x7f431806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:04.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.783+0000 7f4339cdf700 1 -- 172.21.15.16:0/3773636283 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7f431806b020 msgr2=0x7f431806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3622553136 2023-10-16T13:28:04.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.783+0000 7f432affd700 1 -- 172.21.15.16:0/3773636283 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 6473+0+0 (secure 0 0 0) 0x7f432408ab10 con 0x7f43340b5a00 2023-10-16T13:28:04.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.783+0000 7f4339cdf700 1 --2- 172.21.15.16:0/3773636283 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7f431806b020 0x7f431806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:28:04.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.787+0000 7f432affd700 1 -- 172.21.15.16:0/3773636283 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f43340c5b70 con 0x7f43340b5a00 2023-10-16T13:28:04.958 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:28:04.956+0000 7f180423e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T13:28:04.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.984+0000 7f4339cdf700 1 -- 172.21.15.16:0/3773636283 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7f431806b020 msgr2=0x7f431806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3622553136 2023-10-16T13:28:04.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:04.984+0000 7f4339cdf700 1 --2- 172.21.15.16:0/3773636283 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7f431806b020 0x7f431806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:28:05.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.039+0000 7f433c744700 1 -- 172.21.15.16:0/3773636283 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f43340c5b70 con 0x7f43340b5a00 2023-10-16T13:28:05.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.039+0000 7f432affd700 1 -- 172.21.15.16:0/3773636283 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+4917 (secure 0 0 0) 0x7f43340c5b70 con 0x7f43340b5a00 2023-10-16T13:28:05.042 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:28:05.042 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":86,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7964,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/4012137577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":4012137577},{"type":"v1","addr":"172.21.15.16:6837","nonce":4012137577}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81},{"gid":7970,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/3742568047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":3742568047},{"type":"v1","addr":"172.21.15.49:6835","nonce":3742568047}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81},{"gid":9756,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/3838743980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":3838743980},{"type":"v1","addr":"172.21.15.16:6836","nonce":3838743980}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:27:41.561818+0000","modified":"2023-10-16T13:27:44.572793+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9774},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9774":{"gid":9774,"name":"d","rank":0,"incarnation":82,"state":"up:active","state_seq":3,"addr":"172.21.15.49:6834/2218212001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":2218212001},{"type":"v1","addr":"172.21.15.49:6834","nonce":2218212001}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2023-10-16T13:28:05.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.041+0000 7f433c744700 1 -- 172.21.15.16:0/3773636283 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7f431806b020 msgr2=0x7f431806d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:28:05.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.041+0000 7f433c744700 1 --2- 172.21.15.16:0/3773636283 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7f431806b020 0x7f431806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:05.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.041+0000 7f433c744700 1 -- 172.21.15.16:0/3773636283 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43340b5a00 msgr2=0x7f433414aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:05.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.041+0000 7f433c744700 1 --2- 172.21.15.16:0/3773636283 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43340b5a00 0x7f433414aac0 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7f43240090b0 tx=0x7f4324004750 comp rx=0 tx=0).stop 2023-10-16T13:28:05.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.041+0000 7f433c744700 1 -- 172.21.15.16:0/3773636283 shutdown_connections 2023-10-16T13:28:05.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.041+0000 7f433c744700 1 --2- 172.21.15.16:0/3773636283 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7f431806b020 0x7f431806d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:05.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.041+0000 7f433c744700 1 --2- 172.21.15.16:0/3773636283 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f43340cf880 0x7f433406a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:05.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.041+0000 7f433c744700 1 --2- 172.21.15.16:0/3773636283 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43340b5a00 0x7f433414aac0 unknown :-1 s=CLOSED pgs=2004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:05.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.041+0000 7f433c744700 1 --2- 172.21.15.16:0/3773636283 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f43340cf4a0 0x7f43341448d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:05.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.042+0000 7f433c744700 1 -- 172.21.15.16:0/3773636283 >> 172.21.15.16:0/3773636283 conn(0x7f43340b1fd0 msgr2=0x7f43340b4fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:05.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.042+0000 7f433c744700 1 -- 172.21.15.16:0/3773636283 shutdown_connections 2023-10-16T13:28:05.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.042+0000 7f433c744700 1 -- 172.21.15.16:0/3773636283 wait complete. 2023-10-16T13:28:05.047 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 86 2023-10-16T13:28:05.057 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:28:05.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.183+0000 7f9a85d62700 1 Processor -- start 2023-10-16T13:28:05.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.183+0000 7f9a85d62700 1 -- start start 2023-10-16T13:28:05.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.184+0000 7f9a85d62700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f9a800d0ae0 0x7f9a800cef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:05.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.184+0000 7f9a85d62700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9a800ce010 0x7f9a800cc1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:05.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.184+0000 7f9a85d62700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a80072a50 con 0x7f9a8015a5c0 2023-10-16T13:28:05.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.184+0000 7f9a85d62700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a800cf440 con 0x7f9a800ce010 2023-10-16T13:28:05.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.184+0000 7f9a85d62700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a800cba30 con 0x7f9a800d0ae0 2023-10-16T13:28:05.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.184+0000 7f9a7ffff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9a800ce010 0x7f9a800cc1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:05.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.184+0000 7f9a7f7fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f9a8015a5c0 0x7f9a800d03d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38596/0 (socket says 172.21.15.16:38596) 2023-10-16T13:28:05.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.184+0000 7f9a7f7fe700 1 -- 172.21.15.16:0/2434435637 learned_addr learned my addr 172.21.15.16:0/2434435637 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:28:05.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.184+0000 7f9a7effd700 1 --2- 172.21.15.16:0/2434435637 >> v2:172.21.15.49:3300/0 conn(0x7f9a800d0ae0 0x7f9a800cef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:05.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.184+0000 7f9a7ffff700 1 -- 172.21.15.16:0/2434435637 >> v2:172.21.15.49:3300/0 conn(0x7f9a800d0ae0 msgr2=0x7f9a800cef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:28:05.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.184+0000 7f9a7ffff700 1 --2- 172.21.15.16:0/2434435637 >> v2:172.21.15.49:3300/0 conn(0x7f9a800d0ae0 0x7f9a800cef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:05.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.184+0000 7f9a7ffff700 1 -- 172.21.15.16:0/2434435637 >> v1:172.21.15.16:6789/0 conn(0x7f9a8015a5c0 legacy=0x7f9a800d03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:05.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.184+0000 7f9a7ffff700 1 -- 172.21.15.16:0/2434435637 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a800cf630 con 0x7f9a800ce010 2023-10-16T13:28:05.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.184+0000 7f9a7e7fc700 1 -- 172.21.15.16:0/2434435637 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3280136032 0 0) 0x7f9a80072a50 con 0x7f9a8015a5c0 2023-10-16T13:28:05.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.184+0000 7f9a7ffff700 1 --2- 172.21.15.16:0/2434435637 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a800ce010 0x7f9a800cc1f0 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7f9a80067660 tx=0x7f9a7800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98d0fea31eb9ae0f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:05.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.184+0000 7f9a7e7fc700 1 -- 172.21.15.16:0/2434435637 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a7800e070 con 0x7f9a800ce010 2023-10-16T13:28:05.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.184+0000 7f9a7e7fc700 1 -- 172.21.15.16:0/2434435637 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a7800bb20 con 0x7f9a800ce010 2023-10-16T13:28:05.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.184+0000 7f9a7e7fc700 1 -- 172.21.15.16:0/2434435637 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a78012430 con 0x7f9a800ce010 2023-10-16T13:28:05.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.185+0000 7f9a85d62700 1 -- 172.21.15.16:0/2434435637 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a800ce010 msgr2=0x7f9a800cc1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:05.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.185+0000 7f9a85d62700 1 --2- 172.21.15.16:0/2434435637 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a800ce010 0x7f9a800cc1f0 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7f9a80067660 tx=0x7f9a7800b330 comp rx=0 tx=0).stop 2023-10-16T13:28:05.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.185+0000 7f9a85d62700 1 -- 172.21.15.16:0/2434435637 shutdown_connections 2023-10-16T13:28:05.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.185+0000 7f9a85d62700 1 --2- 172.21.15.16:0/2434435637 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a800ce010 0x7f9a800cc1f0 unknown :-1 s=CLOSED pgs=2006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:05.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.185+0000 7f9a85d62700 1 --2- 172.21.15.16:0/2434435637 >> v2:172.21.15.49:3300/0 conn(0x7f9a800d0ae0 0x7f9a800cef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:05.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.185+0000 7f9a85d62700 1 -- 172.21.15.16:0/2434435637 >> 172.21.15.16:0/2434435637 conn(0x7f9a800b1fd0 msgr2=0x7f9a800b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:05.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.185+0000 7f9a85d62700 1 -- 172.21.15.16:0/2434435637 shutdown_connections 2023-10-16T13:28:05.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.185+0000 7f9a85d62700 1 -- 172.21.15.16:0/2434435637 wait complete. 2023-10-16T13:28:05.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.187+0000 7f9a85d62700 1 Processor -- start 2023-10-16T13:28:05.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.187+0000 7f9a85d62700 1 -- start start 2023-10-16T13:28:05.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.188+0000 7f9a85d62700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9a800ce010 0x7f9a800c9e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:05.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.188+0000 7f9a85d62700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9a800d0ae0 0x7f9a800c3000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:05.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.188+0000 7f9a85d62700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a8015a5c0 0x7f9a801593a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:05.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.188+0000 7f9a85d62700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a800c7510 con 0x7f9a8015a5c0 2023-10-16T13:28:05.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.188+0000 7f9a85d62700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a800ca3c0 con 0x7f9a800d0ae0 2023-10-16T13:28:05.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.188+0000 7f9a85d62700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a800b54e0 con 0x7f9a800ce010 2023-10-16T13:28:05.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.188+0000 7f9a7ffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a8015a5c0 0x7f9a801593a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:05.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.188+0000 7f9a7effd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9a800d0ae0 0x7f9a800c3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:05.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.188+0000 7f9a7f7fe700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9a800ce010 0x7f9a800c9e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:05.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.188+0000 7f9a7ffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a8015a5c0 0x7f9a801593a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49332/0 (socket says 172.21.15.16:49332) 2023-10-16T13:28:05.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.188+0000 7f9a7ffff700 1 -- 172.21.15.16:0/697554284 learned_addr learned my addr 172.21.15.16:0/697554284 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:05.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.188+0000 7f9a7effd700 1 -- 172.21.15.16:0/697554284 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9a800ce010 msgr2=0x7f9a800c9e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:05.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.188+0000 7f9a7effd700 1 --2- 172.21.15.16:0/697554284 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9a800ce010 0x7f9a800c9e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:05.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.188+0000 7f9a7effd700 1 -- 172.21.15.16:0/697554284 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a8015a5c0 msgr2=0x7f9a801593a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:05.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.188+0000 7f9a7effd700 1 --2- 172.21.15.16:0/697554284 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a8015a5c0 0x7f9a801593a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:05.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.188+0000 7f9a7effd700 1 -- 172.21.15.16:0/697554284 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a78009a70 con 0x7f9a800d0ae0 2023-10-16T13:28:05.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.188+0000 7f9a7f7fe700 1 --2- 172.21.15.16:0/697554284 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9a800ce010 0x7f9a800c9e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:28:05.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.188+0000 7f9a7ffff700 1 --2- 172.21.15.16:0/697554284 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a8015a5c0 0x7f9a801593a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:28:05.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.189+0000 7f9a7effd700 1 --2- 172.21.15.16:0/697554284 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9a800d0ae0 0x7f9a800c3000 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f9a7400e900 tx=0x7f9a7400ec10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:05.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.189+0000 7f9a67fff700 1 -- 172.21.15.16:0/697554284 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a74013680 con 0x7f9a800d0ae0 2023-10-16T13:28:05.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.189+0000 7f9a67fff700 1 -- 172.21.15.16:0/697554284 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a740137e0 con 0x7f9a800d0ae0 2023-10-16T13:28:05.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.189+0000 7f9a85d62700 1 -- 172.21.15.16:0/697554284 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a800b5780 con 0x7f9a800d0ae0 2023-10-16T13:28:05.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.189+0000 7f9a67fff700 1 -- 172.21.15.16:0/697554284 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a74013b50 con 0x7f9a800d0ae0 2023-10-16T13:28:05.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.189+0000 7f9a85d62700 1 -- 172.21.15.16:0/697554284 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a800b5d40 con 0x7f9a800d0ae0 2023-10-16T13:28:05.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.190+0000 7f9a85d62700 1 -- 172.21.15.16:0/697554284 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a800c7680 con 0x7f9a800d0ae0 2023-10-16T13:28:05.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.191+0000 7f9a67fff700 1 -- 172.21.15.16:0/697554284 <== mon.1 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 103) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f9a74013cb0 con 0x7f9a800d0ae0 2023-10-16T13:28:05.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.191+0000 7f9a67fff700 1 --2- 172.21.15.16:0/697554284 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7f9a6c06b020 0x7f9a6c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:05.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.191+0000 7f9a7f7fe700 1 -- 172.21.15.16:0/697554284 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7f9a6c06b020 msgr2=0x7f9a6c06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3622553136 2023-10-16T13:28:05.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.191+0000 7f9a67fff700 1 -- 172.21.15.16:0/697554284 <== mon.1 v2:172.21.15.49:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 6473+0+0 (secure 0 0 0) 0x7f9a740899f0 con 0x7f9a800d0ae0 2023-10-16T13:28:05.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.191+0000 7f9a7f7fe700 1 --2- 172.21.15.16:0/697554284 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7f9a6c06b020 0x7f9a6c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:28:05.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.196+0000 7f9a67fff700 1 -- 172.21.15.16:0/697554284 <== mon.1 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9a74054690 con 0x7f9a800d0ae0 2023-10-16T13:28:05.210 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:28:05.208+0000 7f954f950200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T13:28:05.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.226+0000 7f9a67fff700 1 -- 172.21.15.16:0/697554284 <== mon.1 v2:172.21.15.49:3300/0 7 ==== mgrmap(e 104) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f9a74087c10 con 0x7f9a800d0ae0 2023-10-16T13:28:05.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.227+0000 7f9a67fff700 1 -- 172.21.15.16:0/697554284 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7f9a6c06b020 msgr2=0x7f9a6c06d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:28:05.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.227+0000 7f9a67fff700 1 --2- 172.21.15.16:0/697554284 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7f9a6c06b020 0x7f9a6c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:05.293 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:28:05.291+0000 7f94c8136700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:28:05.294 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:28:05.291+0000 7f94c8136700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:28:05.294 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:28:05.291+0000 7f94c8136700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:28:05.294 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:28:05.291+0000 7f94c8136700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:28:05.294 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:28:05.291+0000 7f94c8136700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:28:05.295 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:28:05.291+0000 7f94cb17c700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:28:05.295 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:28:05.291+0000 7f94cb17c700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:28:05.295 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:28:05.291+0000 7f94cb17c700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:28:05.295 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:28:05.291+0000 7f94cb17c700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:28:05.295 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:28:05.291+0000 7f94cb17c700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:28:05.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.463+0000 7f9a85d62700 1 -- 172.21.15.16:0/697554284 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9a8004ebb0 con 0x7f9a800d0ae0 2023-10-16T13:28:05.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.464+0000 7f9a67fff700 1 -- 172.21.15.16:0/697554284 <== mon.1 v2:172.21.15.49:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+4917 (secure 0 0 0) 0x7f9a740703c0 con 0x7f9a800d0ae0 2023-10-16T13:28:05.467 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:28:05.467 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":86,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7964,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/4012137577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":4012137577},{"type":"v1","addr":"172.21.15.16:6837","nonce":4012137577}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81},{"gid":7970,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/3742568047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":3742568047},{"type":"v1","addr":"172.21.15.49:6835","nonce":3742568047}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81},{"gid":9756,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/3838743980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":3838743980},{"type":"v1","addr":"172.21.15.16:6836","nonce":3838743980}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:27:41.561818+0000","modified":"2023-10-16T13:27:44.572793+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9774},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9774":{"gid":9774,"name":"d","rank":0,"incarnation":82,"state":"up:active","state_seq":3,"addr":"172.21.15.49:6834/2218212001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":2218212001},{"type":"v1","addr":"172.21.15.49:6834","nonce":2218212001}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2023-10-16T13:28:05.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.466+0000 7f9a85d62700 1 -- 172.21.15.16:0/697554284 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9a800d0ae0 msgr2=0x7f9a800c3000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:05.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.466+0000 7f9a85d62700 1 --2- 172.21.15.16:0/697554284 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9a800d0ae0 0x7f9a800c3000 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f9a7400e900 tx=0x7f9a7400ec10 comp rx=0 tx=0).stop 2023-10-16T13:28:05.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.466+0000 7f9a85d62700 1 -- 172.21.15.16:0/697554284 shutdown_connections 2023-10-16T13:28:05.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.466+0000 7f9a85d62700 1 --2- 172.21.15.16:0/697554284 >> [v2:172.21.15.16:6832/3622553136,v1:172.21.15.16:6833/3622553136] conn(0x7f9a6c06b020 0x7f9a6c06d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:05.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.466+0000 7f9a85d62700 1 --2- 172.21.15.16:0/697554284 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a8015a5c0 0x7f9a801593a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:05.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.466+0000 7f9a85d62700 1 --2- 172.21.15.16:0/697554284 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9a800ce010 0x7f9a800c9e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:05.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.466+0000 7f9a85d62700 1 --2- 172.21.15.16:0/697554284 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9a800d0ae0 0x7f9a800c3000 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:05.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.466+0000 7f9a85d62700 1 -- 172.21.15.16:0/697554284 >> 172.21.15.16:0/697554284 conn(0x7f9a800b1fd0 msgr2=0x7f9a800d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:05.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.467+0000 7f9a85d62700 1 -- 172.21.15.16:0/697554284 shutdown_connections 2023-10-16T13:28:05.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.467+0000 7f9a85d62700 1 -- 172.21.15.16:0/697554284 wait complete. 2023-10-16T13:28:05.471 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 86 2023-10-16T13:28:05.481 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:28:05.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.606+0000 7fad99edf700 1 Processor -- start 2023-10-16T13:28:05.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.606+0000 7fad99edf700 1 -- start start 2023-10-16T13:28:05.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.607+0000 7fad99edf700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fad940cc8a0 0x7fad9415e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:05.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.607+0000 7fad99edf700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad94161c30 con 0x7fad9406a7e0 2023-10-16T13:28:05.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.607+0000 7fad99edf700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad94161f10 con 0x7fad9415b280 2023-10-16T13:28:05.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.607+0000 7fad99edf700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad941621f0 con 0x7fad940cc8a0 2023-10-16T13:28:05.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.607+0000 7fad92ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fad940cc8a0 0x7fad9415e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:05.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.607+0000 7fad92ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fad940cc8a0 0x7fad9415e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49412/0 (socket says 172.21.15.16:49412) 2023-10-16T13:28:05.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.607+0000 7fad92ffd700 1 -- 172.21.15.16:0/1801127099 learned_addr learned my addr 172.21.15.16:0/1801127099 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:05.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.607+0000 7fad93fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fad9406a7e0 0x7fad940d02b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38688/0 (socket says 172.21.15.16:38688) 2023-10-16T13:28:05.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.607+0000 7fad92ffd700 1 -- 172.21.15.16:0/1801127099 >> v1:172.21.15.49:6789/0 conn(0x7fad9415b280 legacy=0x7fad941613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:05.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.607+0000 7fad92ffd700 1 -- 172.21.15.16:0/1801127099 >> v1:172.21.15.16:6789/0 conn(0x7fad9406a7e0 legacy=0x7fad940d02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:05.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.607+0000 7fad92ffd700 1 -- 172.21.15.16:0/1801127099 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad9415df00 con 0x7fad940cc8a0 2023-10-16T13:28:05.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.607+0000 7fad927fc700 1 -- 172.21.15.16:0/1801127099 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4052734723 0 0) 0x7fad94161c30 con 0x7fad9406a7e0 2023-10-16T13:28:05.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.607+0000 7fad92ffd700 1 --2- 172.21.15.16:0/1801127099 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad940cc8a0 0x7fad9415e700 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7fad840098f0 tx=0x7fad84009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7150db4d21cb05f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:05.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.608+0000 7fad927fc700 1 -- 172.21.15.16:0/1801127099 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad8400b8a0 con 0x7fad940cc8a0 2023-10-16T13:28:05.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.608+0000 7fad927fc700 1 -- 172.21.15.16:0/1801127099 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad8400ba00 con 0x7fad940cc8a0 2023-10-16T13:28:05.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.608+0000 7fad927fc700 1 -- 172.21.15.16:0/1801127099 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad8400bd90 con 0x7fad940cc8a0 2023-10-16T13:28:05.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.608+0000 7fad99edf700 1 -- 172.21.15.16:0/1801127099 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad940cc8a0 msgr2=0x7fad9415e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:05.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.608+0000 7fad99edf700 1 --2- 172.21.15.16:0/1801127099 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad940cc8a0 0x7fad9415e700 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7fad840098f0 tx=0x7fad84009c00 comp rx=0 tx=0).stop 2023-10-16T13:28:05.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.608+0000 7fad99edf700 1 -- 172.21.15.16:0/1801127099 shutdown_connections 2023-10-16T13:28:05.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.608+0000 7fad99edf700 1 --2- 172.21.15.16:0/1801127099 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad940cc8a0 0x7fad9415e700 unknown :-1 s=CLOSED pgs=2015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:05.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.608+0000 7fad99edf700 1 -- 172.21.15.16:0/1801127099 >> 172.21.15.16:0/1801127099 conn(0x7fad940b1fd0 msgr2=0x7fad940b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:05.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.608+0000 7fad99edf700 1 -- 172.21.15.16:0/1801127099 shutdown_connections 2023-10-16T13:28:05.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.608+0000 7fad99edf700 1 -- 172.21.15.16:0/1801127099 wait complete. 2023-10-16T13:28:05.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.610+0000 7fad99edf700 1 Processor -- start 2023-10-16T13:28:05.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.610+0000 7fad99edf700 1 -- start start 2023-10-16T13:28:05.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.611+0000 7fad99edf700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fad9406a7e0 0x7fad94160950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:05.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.611+0000 7fad99edf700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fad940cc8a0 0x7fad940ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:05.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.611+0000 7fad99edf700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad9415b280 0x7fad941f27c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:05.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.611+0000 7fad99edf700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad9415bba0 con 0x7fad9415b280 2023-10-16T13:28:05.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.611+0000 7fad99edf700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad9415fe40 con 0x7fad9406a7e0 2023-10-16T13:28:05.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.611+0000 7fad99edf700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad9415f5a0 con 0x7fad940cc8a0 2023-10-16T13:28:05.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.611+0000 7fad93fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad9415b280 0x7fad941f27c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:05.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.611+0000 7fad937fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fad9406a7e0 0x7fad94160950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:05.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.611+0000 7fad92ffd700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fad940cc8a0 0x7fad940ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:05.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.611+0000 7fad93fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad9415b280 0x7fad941f27c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49420/0 (socket says 172.21.15.16:49420) 2023-10-16T13:28:05.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.611+0000 7fad93fff700 1 -- 172.21.15.16:0/1379633623 learned_addr learned my addr 172.21.15.16:0/1379633623 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:05.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.611+0000 7fad93fff700 1 -- 172.21.15.16:0/1379633623 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fad940cc8a0 msgr2=0x7fad940ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:05.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.611+0000 7fad93fff700 1 --2- 172.21.15.16:0/1379633623 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fad940cc8a0 0x7fad940ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:05.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.611+0000 7fad93fff700 1 -- 172.21.15.16:0/1379633623 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fad9406a7e0 msgr2=0x7fad94160950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:05.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.611+0000 7fad93fff700 1 --2- 172.21.15.16:0/1379633623 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fad9406a7e0 0x7fad94160950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:05.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.611+0000 7fad93fff700 1 -- 172.21.15.16:0/1379633623 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad84009550 con 0x7fad9415b280 2023-10-16T13:28:05.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.611+0000 7fad93fff700 1 --2- 172.21.15.16:0/1379633623 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad9415b280 0x7fad941f27c0 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7fad8c002230 tx=0x7fad8c00cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:05.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.612+0000 7fad8bfff700 1 -- 172.21.15.16:0/1379633623 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad8c00aac0 con 0x7fad9415b280 2023-10-16T13:28:05.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.612+0000 7fad8bfff700 1 -- 172.21.15.16:0/1379633623 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad8c00ac40 con 0x7fad9415b280 2023-10-16T13:28:05.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.612+0000 7fad99edf700 1 -- 172.21.15.16:0/1379633623 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad9415f880 con 0x7fad9415b280 2023-10-16T13:28:05.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.612+0000 7fad8bfff700 1 -- 172.21.15.16:0/1379633623 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad8c0070d0 con 0x7fad9415b280 2023-10-16T13:28:05.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.612+0000 7fad99edf700 1 -- 172.21.15.16:0/1379633623 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad9415edc0 con 0x7fad9415b280 2023-10-16T13:28:05.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.613+0000 7fad99edf700 1 -- 172.21.15.16:0/1379633623 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad9415b900 con 0x7fad9415b280 2023-10-16T13:28:05.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.614+0000 7fad8bfff700 1 -- 172.21.15.16:0/1379633623 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 104) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fad8c007250 con 0x7fad9415b280 2023-10-16T13:28:05.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.614+0000 7fad8bfff700 1 -- 172.21.15.16:0/1379633623 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 6817+0+0 (secure 0 0 0) 0x7fad8c0890d0 con 0x7fad9415b280 2023-10-16T13:28:05.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.618+0000 7fad8bfff700 1 -- 172.21.15.16:0/1379633623 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fad8c054170 con 0x7fad9415b280 2023-10-16T13:28:05.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.826+0000 7fad99edf700 1 -- 172.21.15.16:0/1379633623 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fad9415bce0 con 0x7fad9415b280 2023-10-16T13:28:05.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.827+0000 7fad8bfff700 1 -- 172.21.15.16:0/1379633623 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v179) v1 ==== 74+0+17774 (secure 0 0 0) 0x7fad9415bce0 con 0x7fad9415b280 2023-10-16T13:28:05.830 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:28:05.830 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":179,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:28:05.221924+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2023-10-16T13:27:40.032163+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"176","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2023-10-16T13:27:40.523975+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"177","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/439929087":"2023-10-17T13:28:05.221887+0000","172.21.15.16:0/1761844403":"2023-10-17T13:28:05.221887+0000","172.21.15.16:0/2092694398":"2023-10-17T13:28:05.221887+0000","172.21.15.16:0/1458495329":"2023-10-17T13:28:05.221887+0000","172.21.15.16:6833/3622553136":"2023-10-17T13:28:05.221887+0000","172.21.15.16:6832/3622553136":"2023-10-17T13:28:05.221887+0000","172.21.15.16:0/3947060028":"2023-10-17T13:28:05.221887+0000","172.21.15.16:0/3444187941":"2023-10-17T13:28:05.221887+0000","172.21.15.16:0/472482039":"2023-10-17T13:27:56.552715+0000","172.21.15.16:0/1822527107":"2023-10-17T13:27:56.552715+0000","172.21.15.16:0/3919013455":"2023-10-17T13:27:56.552715+0000","172.21.15.16:6833/2329471558":"2023-10-17T13:27:56.552715+0000","172.21.15.16:6832/2329471558":"2023-10-17T13:27:56.552715+0000","172.21.15.16:0/508475708":"2023-10-17T13:27:56.552715+0000","172.21.15.16:0/353895884":"2023-10-17T13:27:56.552715+0000","172.21.15.16:0/2733317665":"2023-10-17T13:27:56.552715+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:28:05.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.829+0000 7fad99edf700 1 -- 172.21.15.16:0/1379633623 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad9415b280 msgr2=0x7fad941f27c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:05.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.829+0000 7fad99edf700 1 --2- 172.21.15.16:0/1379633623 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad9415b280 0x7fad941f27c0 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7fad8c002230 tx=0x7fad8c00cf50 comp rx=0 tx=0).stop 2023-10-16T13:28:05.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.829+0000 7fad99edf700 1 -- 172.21.15.16:0/1379633623 shutdown_connections 2023-10-16T13:28:05.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.829+0000 7fad99edf700 1 --2- 172.21.15.16:0/1379633623 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad9415b280 0x7fad941f27c0 unknown :-1 s=CLOSED pgs=2016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:05.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.829+0000 7fad99edf700 1 --2- 172.21.15.16:0/1379633623 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fad9406a7e0 0x7fad94160950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:05.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.829+0000 7fad99edf700 1 --2- 172.21.15.16:0/1379633623 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fad940cc8a0 0x7fad940ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:05.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.829+0000 7fad99edf700 1 -- 172.21.15.16:0/1379633623 >> 172.21.15.16:0/1379633623 conn(0x7fad940b1fd0 msgr2=0x7fad9415a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:05.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.829+0000 7fad99edf700 1 -- 172.21.15.16:0/1379633623 shutdown_connections 2023-10-16T13:28:05.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.829+0000 7fad99edf700 1 -- 172.21.15.16:0/1379633623 wait complete. 2023-10-16T13:28:05.844 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-16T13:28:05.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.967+0000 7fa215889700 1 Processor -- start 2023-10-16T13:28:05.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.967+0000 7fa215889700 1 -- start start 2023-10-16T13:28:05.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.968+0000 7fa215889700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa210159c90 0x7fa21015d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:05.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.968+0000 7fa215889700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa21015a070 0x7fa21015dee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:05.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.968+0000 7fa215889700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa21006b1f0 con 0x7fa2100cf580 2023-10-16T13:28:05.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.968+0000 7fa215889700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa21006b4b0 con 0x7fa21015a070 2023-10-16T13:28:05.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.968+0000 7fa215889700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa21006b790 con 0x7fa210159c90 2023-10-16T13:28:05.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.968+0000 7fa20e7fc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa21015a070 0x7fa21015dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:05.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.968+0000 7fa20f7fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa2100cf580 0x7fa2100ae520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38700/0 (socket says 172.21.15.16:38700) 2023-10-16T13:28:05.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.968+0000 7fa20f7fe700 1 -- 172.21.15.16:0/4105548172 learned_addr learned my addr 172.21.15.16:0/4105548172 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:28:05.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.968+0000 7fa20effd700 1 --2- 172.21.15.16:0/4105548172 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa210159c90 0x7fa21015d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:05.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.969+0000 7fa20e7fc700 1 -- 172.21.15.16:0/4105548172 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa210159c90 msgr2=0x7fa21015d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:28:05.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.969+0000 7fa20e7fc700 1 --2- 172.21.15.16:0/4105548172 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa210159c90 0x7fa21015d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:05.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.969+0000 7fa20e7fc700 1 -- 172.21.15.16:0/4105548172 >> v1:172.21.15.16:6789/0 conn(0x7fa2100cf580 legacy=0x7fa2100ae520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:05.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.969+0000 7fa20dffb700 1 -- 172.21.15.16:0/4105548172 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 727231985 0 0) 0x7fa21006b1f0 con 0x7fa2100cf580 2023-10-16T13:28:05.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.969+0000 7fa20e7fc700 1 -- 172.21.15.16:0/4105548172 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa21006ba70 con 0x7fa21015a070 2023-10-16T13:28:05.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.969+0000 7fa20e7fc700 1 --2- 172.21.15.16:0/4105548172 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa21015a070 0x7fa21015dee0 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7fa200009840 tx=0x7fa200009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=412d651591da822b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:05.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.969+0000 7fa20dffb700 1 -- 172.21.15.16:0/4105548172 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa20000f8a0 con 0x7fa21015a070 2023-10-16T13:28:05.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.969+0000 7fa20dffb700 1 -- 172.21.15.16:0/4105548172 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa20000fa00 con 0x7fa21015a070 2023-10-16T13:28:05.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.969+0000 7fa20dffb700 1 -- 172.21.15.16:0/4105548172 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa20000fd90 con 0x7fa21015a070 2023-10-16T13:28:05.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.970+0000 7fa215889700 1 -- 172.21.15.16:0/4105548172 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa21015a070 msgr2=0x7fa21015dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:05.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.970+0000 7fa215889700 1 --2- 172.21.15.16:0/4105548172 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa21015a070 0x7fa21015dee0 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7fa200009840 tx=0x7fa200009b50 comp rx=0 tx=0).stop 2023-10-16T13:28:05.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.970+0000 7fa215889700 1 -- 172.21.15.16:0/4105548172 shutdown_connections 2023-10-16T13:28:05.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.970+0000 7fa215889700 1 --2- 172.21.15.16:0/4105548172 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa210159c90 0x7fa21015d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:05.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.970+0000 7fa215889700 1 --2- 172.21.15.16:0/4105548172 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa21015a070 0x7fa21015dee0 unknown :-1 s=CLOSED pgs=2018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:05.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.970+0000 7fa215889700 1 -- 172.21.15.16:0/4105548172 >> 172.21.15.16:0/4105548172 conn(0x7fa2100cc5b0 msgr2=0x7fa2100cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:05.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.970+0000 7fa215889700 1 -- 172.21.15.16:0/4105548172 shutdown_connections 2023-10-16T13:28:05.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.970+0000 7fa215889700 1 -- 172.21.15.16:0/4105548172 wait complete. 2023-10-16T13:28:05.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.971+0000 7fa215889700 1 Processor -- start 2023-10-16T13:28:05.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.971+0000 7fa215889700 1 -- start start 2023-10-16T13:28:05.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.972+0000 7fa215889700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa2100cf580 0x7fa2101f2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:05.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.972+0000 7fa215889700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa210159c90 0x7fa2101562c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:05.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.972+0000 7fa215889700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa21015a070 0x7fa2100b37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:05.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.972+0000 7fa215889700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2100b47d0 con 0x7fa210159c90 2023-10-16T13:28:05.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.972+0000 7fa215889700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2101f3510 con 0x7fa21015a070 2023-10-16T13:28:05.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.972+0000 7fa215889700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2100b4210 con 0x7fa2100cf580 2023-10-16T13:28:05.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.972+0000 7fa20e7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa210159c90 0x7fa2101562c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:05.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.972+0000 7fa20e7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa210159c90 0x7fa2101562c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49430/0 (socket says 172.21.15.16:49430) 2023-10-16T13:28:05.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.972+0000 7fa20e7fc700 1 -- 172.21.15.16:0/3678242607 learned_addr learned my addr 172.21.15.16:0/3678242607 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:05.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.972+0000 7fa20effd700 1 --2- 172.21.15.16:0/3678242607 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa2100cf580 0x7fa2101f2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:05.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.972+0000 7fa20f7fe700 1 --2- 172.21.15.16:0/3678242607 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa21015a070 0x7fa2100b37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:05.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.972+0000 7fa20e7fc700 1 -- 172.21.15.16:0/3678242607 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa2100cf580 msgr2=0x7fa2101f2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:05.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.972+0000 7fa20e7fc700 1 --2- 172.21.15.16:0/3678242607 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa2100cf580 0x7fa2101f2fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:05.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.972+0000 7fa20e7fc700 1 -- 172.21.15.16:0/3678242607 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa21015a070 msgr2=0x7fa2100b37e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:05.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.972+0000 7fa20e7fc700 1 --2- 172.21.15.16:0/3678242607 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa21015a070 0x7fa2100b37e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:05.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.972+0000 7fa20e7fc700 1 -- 172.21.15.16:0/3678242607 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2000094a0 con 0x7fa210159c90 2023-10-16T13:28:05.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.972+0000 7fa20e7fc700 1 --2- 172.21.15.16:0/3678242607 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa210159c90 0x7fa2101562c0 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7fa200009fd0 tx=0x7fa200004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:05.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.973+0000 7fa1f7fff700 1 -- 172.21.15.16:0/3678242607 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2000038a0 con 0x7fa210159c90 2023-10-16T13:28:05.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.973+0000 7fa1f7fff700 1 -- 172.21.15.16:0/3678242607 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa200003a20 con 0x7fa210159c90 2023-10-16T13:28:05.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.973+0000 7fa1f7fff700 1 -- 172.21.15.16:0/3678242607 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa200003dd0 con 0x7fa210159c90 2023-10-16T13:28:05.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.973+0000 7fa215889700 1 -- 172.21.15.16:0/3678242607 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2100b44f0 con 0x7fa210159c90 2023-10-16T13:28:05.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.973+0000 7fa215889700 1 -- 172.21.15.16:0/3678242607 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2100c3ae0 con 0x7fa210159c90 2023-10-16T13:28:05.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.974+0000 7fa215889700 1 -- 172.21.15.16:0/3678242607 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa210066f10 con 0x7fa210159c90 2023-10-16T13:28:05.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.974+0000 7fa1f7fff700 1 -- 172.21.15.16:0/3678242607 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 104) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fa20002c430 con 0x7fa210159c90 2023-10-16T13:28:05.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.975+0000 7fa1f7fff700 1 -- 172.21.15.16:0/3678242607 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 6817+0+0 (secure 0 0 0) 0x7fa20008a8e0 con 0x7fa210159c90 2023-10-16T13:28:05.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:05.979+0000 7fa1f7fff700 1 -- 172.21.15.16:0/3678242607 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa200055980 con 0x7fa210159c90 2023-10-16T13:28:06.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.231+0000 7fa1f7fff700 1 -- 172.21.15.16:0/3678242607 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 105) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fa20008cd50 con 0x7fa210159c90 2023-10-16T13:28:06.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.232+0000 7fa1f7fff700 1 --2- 172.21.15.16:0/3678242607 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7fa1fc070cc0 0x7fa1fc0730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:06.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.233+0000 7fa20effd700 1 --2- 172.21.15.16:0/3678242607 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7fa1fc070cc0 0x7fa1fc0730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:06.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.234+0000 7fa20effd700 1 --2- 172.21.15.16:0/3678242607 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7fa1fc070cc0 0x7fa1fc0730b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa20400ace0 tx=0x7fa20400a320 comp rx=0 tx=0).ready entity=mgr.10047 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:06.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.240+0000 7fa215889700 1 -- 172.21.15.16:0/3678242607 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fa2100c3660 con 0x7fa210159c90 2023-10-16T13:28:06.243 INFO:tasks.ceph.mon.a.smithi016.stderr:2023-10-16T13:28:06.240+0000 7f9fc8535700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-16T13:28:06.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.242+0000 7fa1f7fff700 1 -- 172.21.15.16:0/3678242607 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v87) v1 ==== 161+0+0 (secure 0 0 0) 0x7fa2100c3660 con 0x7fa210159c90 2023-10-16T13:28:06.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.245+0000 7fa215889700 1 -- 172.21.15.16:0/3678242607 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7fa1fc070cc0 msgr2=0x7fa1fc0730b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:06.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.245+0000 7fa215889700 1 --2- 172.21.15.16:0/3678242607 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7fa1fc070cc0 0x7fa1fc0730b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa20400ace0 tx=0x7fa20400a320 comp rx=0 tx=0).stop 2023-10-16T13:28:06.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.245+0000 7fa215889700 1 -- 172.21.15.16:0/3678242607 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa210159c90 msgr2=0x7fa2101562c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:06.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.245+0000 7fa215889700 1 --2- 172.21.15.16:0/3678242607 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa210159c90 0x7fa2101562c0 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7fa200009fd0 tx=0x7fa200004750 comp rx=0 tx=0).stop 2023-10-16T13:28:06.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.245+0000 7fa215889700 1 -- 172.21.15.16:0/3678242607 shutdown_connections 2023-10-16T13:28:06.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.245+0000 7fa215889700 1 --2- 172.21.15.16:0/3678242607 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7fa1fc070cc0 0x7fa1fc0730b0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:06.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.245+0000 7fa215889700 1 --2- 172.21.15.16:0/3678242607 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa21015a070 0x7fa2100b37e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:06.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.245+0000 7fa215889700 1 --2- 172.21.15.16:0/3678242607 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa2100cf580 0x7fa2101f2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:06.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.245+0000 7fa215889700 1 --2- 172.21.15.16:0/3678242607 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa210159c90 0x7fa2101562c0 unknown :-1 s=CLOSED pgs=2019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:06.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.245+0000 7fa215889700 1 -- 172.21.15.16:0/3678242607 >> 172.21.15.16:0/3678242607 conn(0x7fa2100cc5b0 msgr2=0x7fa2100b08c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:06.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.245+0000 7fa215889700 1 -- 172.21.15.16:0/3678242607 shutdown_connections 2023-10-16T13:28:06.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.245+0000 7fa215889700 1 -- 172.21.15.16:0/3678242607 wait complete. 2023-10-16T13:28:06.251 INFO:tasks.ceph.mds.d.smithi049.stderr: -4753> 2023-10-16T13:27:47.120+0000 7f7186f98700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T13:28:06.251 INFO:teuthology.orchestra.run.smithi016.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-16T13:28:06.260 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-16T13:28:06.281 INFO:tasks.ceph.mds.d.smithi049.stdout:starting mds.d at 2023-10-16T13:28:06.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.386+0000 7f2dea1dd700 1 Processor -- start 2023-10-16T13:28:06.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.386+0000 7f2dea1dd700 1 -- start start 2023-10-16T13:28:06.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.387+0000 7f2dea1dd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2de4160ee0 0x7f2de40aeb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:06.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.387+0000 7f2dea1dd700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f2de40af040 0x7f2de40ce180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:06.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.387+0000 7f2dea1dd700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2de40af4a0 con 0x7f2de40ce6c0 2023-10-16T13:28:06.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.387+0000 7f2dea1dd700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2de40cf770 con 0x7f2de4160ee0 2023-10-16T13:28:06.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.387+0000 7f2dea1dd700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2de406aa10 con 0x7f2de40af040 2023-10-16T13:28:06.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.387+0000 7f2de37fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2de4160ee0 0x7f2de40aeb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:06.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.387+0000 7f2de37fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2de4160ee0 0x7f2de40aeb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49458/0 (socket says 172.21.15.16:49458) 2023-10-16T13:28:06.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.387+0000 7f2de37fe700 1 -- 172.21.15.16:0/3527323307 learned_addr learned my addr 172.21.15.16:0/3527323307 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:06.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.387+0000 7f2de2ffd700 1 --2- 172.21.15.16:0/3527323307 >> v2:172.21.15.49:3300/0 conn(0x7f2de40af040 0x7f2de40ce180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:06.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.387+0000 7f2de37fe700 1 -- 172.21.15.16:0/3527323307 >> v2:172.21.15.49:3300/0 conn(0x7f2de40af040 msgr2=0x7f2de40ce180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:28:06.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.387+0000 7f2de37fe700 1 --2- 172.21.15.16:0/3527323307 >> v2:172.21.15.49:3300/0 conn(0x7f2de40af040 0x7f2de40ce180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:06.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.387+0000 7f2de37fe700 1 -- 172.21.15.16:0/3527323307 >> v1:172.21.15.49:6789/0 conn(0x7f2de40ce6c0 legacy=0x7f2de41615b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:06.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.388+0000 7f2de37fe700 1 -- 172.21.15.16:0/3527323307 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2de40ceed0 con 0x7f2de4160ee0 2023-10-16T13:28:06.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.388+0000 7f2de37fe700 1 --2- 172.21.15.16:0/3527323307 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2de4160ee0 0x7f2de40aeb00 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 crypto rx=0x7f2de40d0c30 tx=0x7f2dd4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc4722ce73631d38 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:06.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.388+0000 7f2de27fc700 1 -- 172.21.15.16:0/3527323307 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd4011830 con 0x7f2de4160ee0 2023-10-16T13:28:06.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.388+0000 7f2de27fc700 1 -- 172.21.15.16:0/3527323307 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dd4011990 con 0x7f2de4160ee0 2023-10-16T13:28:06.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.388+0000 7f2de27fc700 1 -- 172.21.15.16:0/3527323307 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd4011d20 con 0x7f2de4160ee0 2023-10-16T13:28:06.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.388+0000 7f2dea1dd700 1 -- 172.21.15.16:0/3527323307 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2de4160ee0 msgr2=0x7f2de40aeb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:06.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.388+0000 7f2dea1dd700 1 --2- 172.21.15.16:0/3527323307 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2de4160ee0 0x7f2de40aeb00 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 crypto rx=0x7f2de40d0c30 tx=0x7f2dd4009bd0 comp rx=0 tx=0).stop 2023-10-16T13:28:06.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.388+0000 7f2dea1dd700 1 -- 172.21.15.16:0/3527323307 shutdown_connections 2023-10-16T13:28:06.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.388+0000 7f2dea1dd700 1 --2- 172.21.15.16:0/3527323307 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2de4160ee0 0x7f2de40aeb00 unknown :-1 s=CLOSED pgs=2023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:06.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.388+0000 7f2dea1dd700 1 --2- 172.21.15.16:0/3527323307 >> v2:172.21.15.49:3300/0 conn(0x7f2de40af040 0x7f2de40ce180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:06.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.388+0000 7f2dea1dd700 1 -- 172.21.15.16:0/3527323307 >> 172.21.15.16:0/3527323307 conn(0x7f2de40cc5e0 msgr2=0x7f2de40cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:06.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.388+0000 7f2dea1dd700 1 -- 172.21.15.16:0/3527323307 shutdown_connections 2023-10-16T13:28:06.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.388+0000 7f2dea1dd700 1 -- 172.21.15.16:0/3527323307 wait complete. 2023-10-16T13:28:06.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.390+0000 7f2dea1dd700 1 Processor -- start 2023-10-16T13:28:06.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.390+0000 7f2dea1dd700 1 -- start start 2023-10-16T13:28:06.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.391+0000 7f2dea1dd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2de40af040 0x7f2de41560d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:06.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.391+0000 7f2dea1dd700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2de40ce6c0 0x7f2de40ad920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:06.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.391+0000 7f2dea1dd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2de4160ee0 0x7f2de40ade60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:06.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.391+0000 7f2dea1dd700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2de415a130 con 0x7f2de40af040 2023-10-16T13:28:06.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.391+0000 7f2dea1dd700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2de40c9a00 con 0x7f2de4160ee0 2023-10-16T13:28:06.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.391+0000 7f2dea1dd700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2de40c89f0 con 0x7f2de40ce6c0 2023-10-16T13:28:06.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.391+0000 7f2de37fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2de40af040 0x7f2de41560d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:06.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.391+0000 7f2de37fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2de40af040 0x7f2de41560d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49466/0 (socket says 172.21.15.16:49466) 2023-10-16T13:28:06.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.391+0000 7f2de37fe700 1 -- 172.21.15.16:0/1202862628 learned_addr learned my addr 172.21.15.16:0/1202862628 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:06.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.392+0000 7f2de2ffd700 1 --2- 172.21.15.16:0/1202862628 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2de40ce6c0 0x7f2de40ad920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:06.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.392+0000 7f2de3fff700 1 --2- 172.21.15.16:0/1202862628 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2de4160ee0 0x7f2de40ade60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:06.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.392+0000 7f2de37fe700 1 -- 172.21.15.16:0/1202862628 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2de40ce6c0 msgr2=0x7f2de40ad920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:06.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.392+0000 7f2de37fe700 1 --2- 172.21.15.16:0/1202862628 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2de40ce6c0 0x7f2de40ad920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:06.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.392+0000 7f2de37fe700 1 -- 172.21.15.16:0/1202862628 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2de4160ee0 msgr2=0x7f2de40ade60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:06.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.392+0000 7f2de37fe700 1 --2- 172.21.15.16:0/1202862628 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2de4160ee0 0x7f2de40ade60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:06.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.392+0000 7f2de37fe700 1 -- 172.21.15.16:0/1202862628 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dd4009550 con 0x7f2de40af040 2023-10-16T13:28:06.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.392+0000 7f2de37fe700 1 --2- 172.21.15.16:0/1202862628 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2de40af040 0x7f2de41560d0 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7f2dd4009c10 tx=0x7f2dd4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:06.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.392+0000 7f2dcbfff700 1 -- 172.21.15.16:0/1202862628 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd40036a0 con 0x7f2de40af040 2023-10-16T13:28:06.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.392+0000 7f2dcbfff700 1 -- 172.21.15.16:0/1202862628 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dd401e070 con 0x7f2de40af040 2023-10-16T13:28:06.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.392+0000 7f2dcbfff700 1 -- 172.21.15.16:0/1202862628 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd4003a40 con 0x7f2de40af040 2023-10-16T13:28:06.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.392+0000 7f2dea1dd700 1 -- 172.21.15.16:0/1202862628 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2de40c8c40 con 0x7f2de40af040 2023-10-16T13:28:06.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.392+0000 7f2dea1dd700 1 -- 172.21.15.16:0/1202862628 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2de40c9200 con 0x7f2de40af040 2023-10-16T13:28:06.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.393+0000 7f2dea1dd700 1 -- 172.21.15.16:0/1202862628 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2de404ebb0 con 0x7f2de40af040 2023-10-16T13:28:06.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.393+0000 7f2dcbfff700 1 -- 172.21.15.16:0/1202862628 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 105) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f2dd4003ba0 con 0x7f2de40af040 2023-10-16T13:28:06.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.394+0000 7f2dcbfff700 1 --2- 172.21.15.16:0/1202862628 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f2dd006ad50 0x7f2dd006d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:06.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.394+0000 7f2dcbfff700 1 -- 172.21.15.16:0/1202862628 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 6903+0+0 (secure 0 0 0) 0x7f2dd408a800 con 0x7f2de40af040 2023-10-16T13:28:06.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.394+0000 7f2de2ffd700 1 --2- 172.21.15.16:0/1202862628 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f2dd006ad50 0x7f2dd006d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:06.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.394+0000 7f2de2ffd700 1 --2- 172.21.15.16:0/1202862628 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f2dd006ad50 0x7f2dd006d210 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f2de40c8510 tx=0x7f2dd8008040 comp rx=0 tx=0).ready entity=mgr.10047 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:06.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.398+0000 7f2dcbfff700 1 -- 172.21.15.16:0/1202862628 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2dd4055740 con 0x7f2de40af040 2023-10-16T13:28:06.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:06.652+0000 7f2dea1dd700 1 -- 172.21.15.16:0/1202862628 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f2de40cdba0 con 0x7f2de40af040 2023-10-16T13:28:07.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:07.243+0000 7f2dcbfff700 1 -- 172.21.15.16:0/1202862628 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v88) v1 ==== 104+0+0 (secure 0 0 0) 0x7f2dd4058d60 con 0x7f2de40af040 2023-10-16T13:28:07.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:07.246+0000 7f2dea1dd700 1 -- 172.21.15.16:0/1202862628 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f2dd006ad50 msgr2=0x7f2dd006d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:07.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:07.246+0000 7f2dea1dd700 1 --2- 172.21.15.16:0/1202862628 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f2dd006ad50 0x7f2dd006d210 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f2de40c8510 tx=0x7f2dd8008040 comp rx=0 tx=0).stop 2023-10-16T13:28:07.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:07.246+0000 7f2dea1dd700 1 -- 172.21.15.16:0/1202862628 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2de40af040 msgr2=0x7f2de41560d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:07.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:07.246+0000 7f2dea1dd700 1 --2- 172.21.15.16:0/1202862628 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2de40af040 0x7f2de41560d0 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7f2dd4009c10 tx=0x7f2dd4004750 comp rx=0 tx=0).stop 2023-10-16T13:28:07.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:07.246+0000 7f2dea1dd700 1 -- 172.21.15.16:0/1202862628 shutdown_connections 2023-10-16T13:28:07.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:07.246+0000 7f2dea1dd700 1 --2- 172.21.15.16:0/1202862628 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f2dd006ad50 0x7f2dd006d210 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:07.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:07.246+0000 7f2dea1dd700 1 --2- 172.21.15.16:0/1202862628 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2de4160ee0 0x7f2de40ade60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:07.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:07.246+0000 7f2dea1dd700 1 --2- 172.21.15.16:0/1202862628 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2de40af040 0x7f2de41560d0 unknown :-1 s=CLOSED pgs=2024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:07.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:07.246+0000 7f2dea1dd700 1 --2- 172.21.15.16:0/1202862628 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2de40ce6c0 0x7f2de40ad920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:07.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:07.246+0000 7f2dea1dd700 1 -- 172.21.15.16:0/1202862628 >> 172.21.15.16:0/1202862628 conn(0x7f2de40cc5e0 msgr2=0x7f2de40b0b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:07.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:07.246+0000 7f2dea1dd700 1 -- 172.21.15.16:0/1202862628 shutdown_connections 2023-10-16T13:28:07.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:07.246+0000 7f2dea1dd700 1 -- 172.21.15.16:0/1202862628 wait complete. 2023-10-16T13:28:07.264 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-16T13:28:07.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:07.402+0000 7f55b7eee700 1 Processor -- start 2023-10-16T13:28:07.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:07.402+0000 7f55b7eee700 1 -- start start 2023-10-16T13:28:07.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:07.403+0000 7f55b7eee700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f55b0159c10 0x7f55b0071140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:07.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:07.403+0000 7f55b7eee700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55b015e5d0 con 0x7f55b00aefc0 2023-10-16T13:28:07.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:07.403+0000 7f55b7eee700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55b015da00 con 0x7f55b0071680 2023-10-16T13:28:07.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:07.403+0000 7f55b7eee700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55b0071b00 con 0x7f55b0159c10 2023-10-16T13:28:07.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:07.403+0000 7f55b5c8a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f55b0159c10 0x7f55b0071140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:07.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:07.403+0000 7f55b648b700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f55b00aefc0 0x7f55b006aa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38764/0 (socket says 172.21.15.16:38764) 2023-10-16T13:28:07.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:07.403+0000 7f55b648b700 1 -- 172.21.15.16:0/215358320 learned_addr learned my addr 172.21.15.16:0/215358320 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:28:07.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:07.403+0000 7f55b5c8a700 1 -- 172.21.15.16:0/215358320 >> v1:172.21.15.49:6789/0 conn(0x7f55b0071680 legacy=0x7f55b00ae8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:07.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:07.403+0000 7f55b5c8a700 1 -- 172.21.15.16:0/215358320 >> v1:172.21.15.16:6789/0 conn(0x7f55b00aefc0 legacy=0x7f55b006aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:07.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:07.403+0000 7f55b5c8a700 1 -- 172.21.15.16:0/215358320 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55b015dca0 con 0x7f55b0159c10 2023-10-16T13:28:07.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:07.403+0000 7f55b4c88700 1 -- 172.21.15.16:0/215358320 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2759917585 0 0) 0x7f55b015e5d0 con 0x7f55b00aefc0 2023-10-16T13:28:07.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:07.403+0000 7f55b5c8a700 1 --2- 172.21.15.16:0/215358320 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55b0159c10 0x7f55b0071140 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 crypto rx=0x7f55ac009840 tx=0x7f55ac009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27ec16085985228f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:07.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:07.403+0000 7f55b4c88700 1 -- 172.21.15.16:0/215358320 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55ac011830 con 0x7f55b0159c10 2023-10-16T13:28:07.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:07.404+0000 7f55b4c88700 1 -- 172.21.15.16:0/215358320 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55ac011990 con 0x7f55b0159c10 2023-10-16T13:28:07.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:07.404+0000 7f55b4c88700 1 -- 172.21.15.16:0/215358320 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55ac011d20 con 0x7f55b0159c10 2023-10-16T13:28:07.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:07.404+0000 7f55b7eee700 1 -- 172.21.15.16:0/215358320 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55b0159c10 msgr2=0x7f55b0071140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:07.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:07.404+0000 7f55b7eee700 1 --2- 172.21.15.16:0/215358320 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55b0159c10 0x7f55b0071140 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 crypto rx=0x7f55ac009840 tx=0x7f55ac009b50 comp rx=0 tx=0).stop 2023-10-16T13:28:07.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:07.404+0000 7f55b7eee700 1 -- 172.21.15.16:0/215358320 shutdown_connections 2023-10-16T13:28:07.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:07.404+0000 7f55b7eee700 1 --2- 172.21.15.16:0/215358320 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55b0159c10 0x7f55b0071140 unknown :-1 s=CLOSED pgs=2028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:07.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:07.404+0000 7f55b7eee700 1 -- 172.21.15.16:0/215358320 >> 172.21.15.16:0/215358320 conn(0x7f55b00cfa90 msgr2=0x7f55b00cfea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:07.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:07.404+0000 7f55b7eee700 1 -- 172.21.15.16:0/215358320 shutdown_connections 2023-10-16T13:28:07.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:07.404+0000 7f55b7eee700 1 -- 172.21.15.16:0/215358320 wait complete. 2023-10-16T13:28:07.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:07.405+0000 7f55b7eee700 1 Processor -- start 2023-10-16T13:28:07.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:07.405+0000 7f55b7eee700 1 -- start start 2023-10-16T13:28:07.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:07.406+0000 7f55b7eee700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55b0071680 0x7f55b0151640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:07.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:07.406+0000 7f55b7eee700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f55b00aefc0 0x7f55b00ac8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:07.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:07.406+0000 7f55b7eee700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f55b0159c10 0x7f55b00c8670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:07.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:07.406+0000 7f55b7eee700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55b00c90c0 con 0x7f55b0071680 2023-10-16T13:28:07.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:07.406+0000 7f55b7eee700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55b01e37d0 con 0x7f55b0159c10 2023-10-16T13:28:07.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:07.406+0000 7f55b7eee700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55b01e3ab0 con 0x7f55b00aefc0 2023-10-16T13:28:07.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:07.406+0000 7f55b5c8a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55b0071680 0x7f55b0151640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:07.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:07.406+0000 7f55b5c8a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55b0071680 0x7f55b0151640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49496/0 (socket says 172.21.15.16:49496) 2023-10-16T13:28:07.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:07.406+0000 7f55b5c8a700 1 -- 172.21.15.16:0/870204996 learned_addr learned my addr 172.21.15.16:0/870204996 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:07.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:07.406+0000 7f55b5489700 1 --2- 172.21.15.16:0/870204996 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f55b00aefc0 0x7f55b00ac8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:07.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:07.406+0000 7f55b648b700 1 --2- 172.21.15.16:0/870204996 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f55b0159c10 0x7f55b00c8670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:07.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:07.406+0000 7f55b5c8a700 1 -- 172.21.15.16:0/870204996 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f55b00aefc0 msgr2=0x7f55b00ac8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:07.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:07.406+0000 7f55b5c8a700 1 --2- 172.21.15.16:0/870204996 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f55b00aefc0 0x7f55b00ac8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:07.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:07.406+0000 7f55b5c8a700 1 -- 172.21.15.16:0/870204996 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f55b0159c10 msgr2=0x7f55b00c8670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:07.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:07.406+0000 7f55b5c8a700 1 --2- 172.21.15.16:0/870204996 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f55b0159c10 0x7f55b00c8670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:07.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:07.406+0000 7f55b5c8a700 1 -- 172.21.15.16:0/870204996 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55ac0094a0 con 0x7f55b0071680 2023-10-16T13:28:07.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:07.407+0000 7f55b5c8a700 1 --2- 172.21.15.16:0/870204996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55b0071680 0x7f55b0151640 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 crypto rx=0x7f55ac014040 tx=0x7f55ac004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:07.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:07.407+0000 7f55a67fc700 1 -- 172.21.15.16:0/870204996 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55ac01a900 con 0x7f55b0071680 2023-10-16T13:28:07.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:07.407+0000 7f55a67fc700 1 -- 172.21.15.16:0/870204996 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55ac01aa60 con 0x7f55b0071680 2023-10-16T13:28:07.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:07.407+0000 7f55a67fc700 1 -- 172.21.15.16:0/870204996 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55ac01add0 con 0x7f55b0071680 2023-10-16T13:28:07.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:07.407+0000 7f55b7eee700 1 -- 172.21.15.16:0/870204996 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55b01e3d90 con 0x7f55b0071680 2023-10-16T13:28:07.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:07.407+0000 7f55b7eee700 1 -- 172.21.15.16:0/870204996 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55b00ace00 con 0x7f55b0071680 2023-10-16T13:28:07.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:07.408+0000 7f55b7eee700 1 -- 172.21.15.16:0/870204996 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55b01e42b0 con 0x7f55b0071680 2023-10-16T13:28:07.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:07.408+0000 7f55a67fc700 1 -- 172.21.15.16:0/870204996 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 105) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f55ac022070 con 0x7f55b0071680 2023-10-16T13:28:07.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:07.409+0000 7f55a67fc700 1 --2- 172.21.15.16:0/870204996 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f559406ad50 0x7f559406d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:07.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:07.409+0000 7f55a67fc700 1 -- 172.21.15.16:0/870204996 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 6903+0+0 (secure 0 0 0) 0x7f55ac08a560 con 0x7f55b0071680 2023-10-16T13:28:07.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:07.409+0000 7f55b5489700 1 --2- 172.21.15.16:0/870204996 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f559406ad50 0x7f559406d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:07.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:07.409+0000 7f55b5489700 1 --2- 172.21.15.16:0/870204996 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f559406ad50 0x7f559406d210 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f55b0159600 tx=0x7f55a0009040 comp rx=0 tx=0).ready entity=mgr.10047 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:07.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:07.413+0000 7f55a67fc700 1 -- 172.21.15.16:0/870204996 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f55b01e42b0 con 0x7f55b0071680 2023-10-16T13:28:07.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:07.617+0000 7f55b7eee700 1 -- 172.21.15.16:0/870204996 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f55b01e42b0 con 0x7f55b0071680 2023-10-16T13:28:08.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:08.239+0000 7f55a67fc700 1 -- 172.21.15.16:0/870204996 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 106) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f55ac055920 con 0x7f55b0071680 2023-10-16T13:28:08.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:08.581+0000 7f55a67fc700 1 -- 172.21.15.16:0/870204996 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v181) v1 ==== 181+0+0 (secure 0 0 0) 0x7f55ac06e860 con 0x7f55b0071680 2023-10-16T13:28:08.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:08.585+0000 7f55b7eee700 1 -- 172.21.15.16:0/870204996 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f559406ad50 msgr2=0x7f559406d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:08.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:08.585+0000 7f55b7eee700 1 --2- 172.21.15.16:0/870204996 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f559406ad50 0x7f559406d210 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f55b0159600 tx=0x7f55a0009040 comp rx=0 tx=0).stop 2023-10-16T13:28:08.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:08.585+0000 7f55b7eee700 1 -- 172.21.15.16:0/870204996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55b0071680 msgr2=0x7f55b0151640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:08.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:08.585+0000 7f55b7eee700 1 --2- 172.21.15.16:0/870204996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55b0071680 0x7f55b0151640 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 crypto rx=0x7f55ac014040 tx=0x7f55ac004750 comp rx=0 tx=0).stop 2023-10-16T13:28:08.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:08.585+0000 7f55b7eee700 1 -- 172.21.15.16:0/870204996 shutdown_connections 2023-10-16T13:28:08.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:08.585+0000 7f55b7eee700 1 --2- 172.21.15.16:0/870204996 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f559406ad50 0x7f559406d210 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:08.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:08.585+0000 7f55b7eee700 1 --2- 172.21.15.16:0/870204996 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f55b0159c10 0x7f55b00c8670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:08.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:08.585+0000 7f55b7eee700 1 --2- 172.21.15.16:0/870204996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55b0071680 0x7f55b0151640 unknown :-1 s=CLOSED pgs=2029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:08.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:08.585+0000 7f55b7eee700 1 --2- 172.21.15.16:0/870204996 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f55b00aefc0 0x7f55b00ac8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:08.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:08.585+0000 7f55b7eee700 1 -- 172.21.15.16:0/870204996 >> 172.21.15.16:0/870204996 conn(0x7f55b00cfa90 msgr2=0x7f55b00cc510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:08.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:08.585+0000 7f55b7eee700 1 -- 172.21.15.16:0/870204996 shutdown_connections 2023-10-16T13:28:08.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:08.586+0000 7f55b7eee700 1 -- 172.21.15.16:0/870204996 wait complete. 2023-10-16T13:28:08.591 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' removed 2023-10-16T13:28:08.603 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-16T13:28:08.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:08.737+0000 7fe69bec4700 1 Processor -- start 2023-10-16T13:28:08.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:08.737+0000 7fe69bec4700 1 -- start start 2023-10-16T13:28:08.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:08.738+0000 7fe69bec4700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe6940cf680 0x7fe69415e070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:08.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:08.738+0000 7fe69bec4700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fe6940cdef0 0x7fe6940cbde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:08.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:08.738+0000 7fe69bec4700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6940cb930 con 0x7fe69415da20 2023-10-16T13:28:08.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:08.738+0000 7fe69bec4700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe69415de00 con 0x7fe6940cf680 2023-10-16T13:28:08.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:08.738+0000 7fe69bec4700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe69415e650 con 0x7fe6940cdef0 2023-10-16T13:28:08.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:08.738+0000 7fe69945f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe6940cf680 0x7fe69415e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:08.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:08.738+0000 7fe69945f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe6940cf680 0x7fe69415e070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49506/0 (socket says 172.21.15.16:49506) 2023-10-16T13:28:08.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:08.738+0000 7fe69945f700 1 -- 172.21.15.16:0/1899000428 learned_addr learned my addr 172.21.15.16:0/1899000428 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:08.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:08.738+0000 7fe69a461700 1 --2- 172.21.15.16:0/1899000428 >> v2:172.21.15.49:3300/0 conn(0x7fe6940cdef0 0x7fe6940cbde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:08.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:08.738+0000 7fe69945f700 1 -- 172.21.15.16:0/1899000428 >> v2:172.21.15.49:3300/0 conn(0x7fe6940cdef0 msgr2=0x7fe6940cbde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:28:08.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:08.738+0000 7fe69945f700 1 --2- 172.21.15.16:0/1899000428 >> v2:172.21.15.49:3300/0 conn(0x7fe6940cdef0 0x7fe6940cbde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:08.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:08.738+0000 7fe69945f700 1 -- 172.21.15.16:0/1899000428 >> v1:172.21.15.49:6789/0 conn(0x7fe69415da20 legacy=0x7fe6940cef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:08.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:08.738+0000 7fe69945f700 1 -- 172.21.15.16:0/1899000428 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe69415e930 con 0x7fe6940cf680 2023-10-16T13:28:08.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:08.739+0000 7fe69945f700 1 --2- 172.21.15.16:0/1899000428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6940cf680 0x7fe69415e070 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7fe694072c10 tx=0x7fe690009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5317f675480cfc0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:08.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:08.739+0000 7fe698c5e700 1 -- 172.21.15.16:0/1899000428 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe69000b870 con 0x7fe6940cf680 2023-10-16T13:28:08.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:08.739+0000 7fe698c5e700 1 -- 172.21.15.16:0/1899000428 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe69000b9d0 con 0x7fe6940cf680 2023-10-16T13:28:08.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:08.739+0000 7fe698c5e700 1 -- 172.21.15.16:0/1899000428 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe69000bd60 con 0x7fe6940cf680 2023-10-16T13:28:08.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:08.739+0000 7fe69bec4700 1 -- 172.21.15.16:0/1899000428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6940cf680 msgr2=0x7fe69415e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:08.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:08.739+0000 7fe69bec4700 1 --2- 172.21.15.16:0/1899000428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6940cf680 0x7fe69415e070 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7fe694072c10 tx=0x7fe690009bd0 comp rx=0 tx=0).stop 2023-10-16T13:28:08.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:08.739+0000 7fe69bec4700 1 -- 172.21.15.16:0/1899000428 shutdown_connections 2023-10-16T13:28:08.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:08.739+0000 7fe69bec4700 1 --2- 172.21.15.16:0/1899000428 >> v2:172.21.15.49:3300/0 conn(0x7fe6940cdef0 0x7fe6940cbde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:08.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:08.739+0000 7fe69bec4700 1 --2- 172.21.15.16:0/1899000428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6940cf680 0x7fe69415e070 unknown :-1 s=CLOSED pgs=2030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:08.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:08.739+0000 7fe69bec4700 1 -- 172.21.15.16:0/1899000428 >> 172.21.15.16:0/1899000428 conn(0x7fe6940b1ed0 msgr2=0x7fe6940b22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:08.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:08.739+0000 7fe69bec4700 1 -- 172.21.15.16:0/1899000428 shutdown_connections 2023-10-16T13:28:08.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:08.739+0000 7fe69bec4700 1 -- 172.21.15.16:0/1899000428 wait complete. 2023-10-16T13:28:08.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:08.741+0000 7fe69bec4700 1 Processor -- start 2023-10-16T13:28:08.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:08.741+0000 7fe69bec4700 1 -- start start 2023-10-16T13:28:08.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:08.741+0000 7fe69bec4700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe6940cdef0 0x7fe6940af3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:08.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:08.741+0000 7fe69bec4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6940cf680 0x7fe6940c8bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:08.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:08.741+0000 7fe69bec4700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe69415da20 0x7fe69406a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:08.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:08.741+0000 7fe69bec4700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6940c9670 con 0x7fe6940cf680 2023-10-16T13:28:08.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:08.741+0000 7fe69bec4700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6940c9950 con 0x7fe6940cdef0 2023-10-16T13:28:08.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:08.741+0000 7fe69bec4700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6940c9c30 con 0x7fe69415da20 2023-10-16T13:28:08.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:08.741+0000 7fe69945f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6940cf680 0x7fe6940c8bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:08.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:08.741+0000 7fe69945f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6940cf680 0x7fe6940c8bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49512/0 (socket says 172.21.15.16:49512) 2023-10-16T13:28:08.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:08.741+0000 7fe69945f700 1 -- 172.21.15.16:0/4205221267 learned_addr learned my addr 172.21.15.16:0/4205221267 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:08.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:08.742+0000 7fe699c60700 1 --2- 172.21.15.16:0/4205221267 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe6940cdef0 0x7fe6940af3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:08.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:08.742+0000 7fe69a461700 1 --2- 172.21.15.16:0/4205221267 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe69415da20 0x7fe69406a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:08.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:08.742+0000 7fe69945f700 1 -- 172.21.15.16:0/4205221267 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe69415da20 msgr2=0x7fe69406a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:08.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:08.742+0000 7fe69945f700 1 --2- 172.21.15.16:0/4205221267 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe69415da20 0x7fe69406a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:08.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:08.742+0000 7fe69945f700 1 -- 172.21.15.16:0/4205221267 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe6940cdef0 msgr2=0x7fe6940af3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:08.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:08.742+0000 7fe69945f700 1 --2- 172.21.15.16:0/4205221267 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe6940cdef0 0x7fe6940af3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:08.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:08.742+0000 7fe69945f700 1 -- 172.21.15.16:0/4205221267 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe690009550 con 0x7fe6940cf680 2023-10-16T13:28:08.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:08.742+0000 7fe69a461700 1 --2- 172.21.15.16:0/4205221267 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe69415da20 0x7fe69406a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:28:08.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:08.742+0000 7fe69945f700 1 --2- 172.21.15.16:0/4205221267 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6940cf680 0x7fe6940c8bf0 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7fe690000c00 tx=0x7fe690004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:08.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:08.742+0000 7fe68a7fc700 1 -- 172.21.15.16:0/4205221267 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe690004520 con 0x7fe6940cf680 2023-10-16T13:28:08.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:08.742+0000 7fe68a7fc700 1 -- 172.21.15.16:0/4205221267 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe69001e070 con 0x7fe6940cf680 2023-10-16T13:28:08.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:08.742+0000 7fe69bec4700 1 -- 172.21.15.16:0/4205221267 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6940c9f10 con 0x7fe6940cf680 2023-10-16T13:28:08.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:08.742+0000 7fe68a7fc700 1 -- 172.21.15.16:0/4205221267 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6900038a0 con 0x7fe6940cf680 2023-10-16T13:28:08.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:08.742+0000 7fe69bec4700 1 -- 172.21.15.16:0/4205221267 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6940af930 con 0x7fe6940cf680 2023-10-16T13:28:08.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:08.743+0000 7fe69bec4700 1 -- 172.21.15.16:0/4205221267 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6940ca280 con 0x7fe6940cf680 2023-10-16T13:28:08.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:08.744+0000 7fe68a7fc700 1 -- 172.21.15.16:0/4205221267 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 106) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fe690003a00 con 0x7fe6940cf680 2023-10-16T13:28:08.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:08.744+0000 7fe68a7fc700 1 --2- 172.21.15.16:0/4205221267 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7fe67806af90 0x7fe67806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:08.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:08.744+0000 7fe68a7fc700 1 -- 172.21.15.16:0/4205221267 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 6465+0+0 (secure 0 0 0) 0x7fe690020070 con 0x7fe6940cf680 2023-10-16T13:28:08.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:08.744+0000 7fe699c60700 1 --2- 172.21.15.16:0/4205221267 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7fe67806af90 0x7fe67806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:08.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:08.744+0000 7fe699c60700 1 --2- 172.21.15.16:0/4205221267 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7fe67806af90 0x7fe67806d450 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe684007fd0 tx=0x7fe684009040 comp rx=0 tx=0).ready entity=mgr.10047 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:08.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:08.749+0000 7fe68a7fc700 1 -- 172.21.15.16:0/4205221267 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe6940ca280 con 0x7fe6940cf680 2023-10-16T13:28:08.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:08.954+0000 7fe69bec4700 1 -- 172.21.15.16:0/4205221267 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe6940ca280 con 0x7fe6940cf680 2023-10-16T13:28:09.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.587+0000 7fe68a7fc700 1 -- 172.21.15.16:0/4205221267 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v182) v1 ==== 169+0+0 (secure 0 0 0) 0x7fe6940ca280 con 0x7fe6940cf680 2023-10-16T13:28:09.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.594+0000 7fe69bec4700 1 -- 172.21.15.16:0/4205221267 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7fe67806af90 msgr2=0x7fe67806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:09.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.594+0000 7fe69bec4700 1 --2- 172.21.15.16:0/4205221267 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7fe67806af90 0x7fe67806d450 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe684007fd0 tx=0x7fe684009040 comp rx=0 tx=0).stop 2023-10-16T13:28:09.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.594+0000 7fe69bec4700 1 -- 172.21.15.16:0/4205221267 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6940cf680 msgr2=0x7fe6940c8bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:09.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.594+0000 7fe69bec4700 1 --2- 172.21.15.16:0/4205221267 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6940cf680 0x7fe6940c8bf0 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7fe690000c00 tx=0x7fe690004750 comp rx=0 tx=0).stop 2023-10-16T13:28:09.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.595+0000 7fe69bec4700 1 -- 172.21.15.16:0/4205221267 shutdown_connections 2023-10-16T13:28:09.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.595+0000 7fe69bec4700 1 --2- 172.21.15.16:0/4205221267 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe69415da20 0x7fe69406a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:09.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.595+0000 7fe69bec4700 1 --2- 172.21.15.16:0/4205221267 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7fe67806af90 0x7fe67806d450 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:09.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.595+0000 7fe69bec4700 1 --2- 172.21.15.16:0/4205221267 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe6940cdef0 0x7fe6940af3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:09.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.595+0000 7fe69bec4700 1 --2- 172.21.15.16:0/4205221267 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6940cf680 0x7fe6940c8bf0 unknown :-1 s=CLOSED pgs=2031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:09.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.595+0000 7fe69bec4700 1 -- 172.21.15.16:0/4205221267 >> 172.21.15.16:0/4205221267 conn(0x7fe6940b1ed0 msgr2=0x7fe6940b0d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:09.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.598+0000 7fe69bec4700 1 -- 172.21.15.16:0/4205221267 shutdown_connections 2023-10-16T13:28:09.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.598+0000 7fe69bec4700 1 -- 172.21.15.16:0/4205221267 wait complete. 2023-10-16T13:28:09.602 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' removed 2023-10-16T13:28:09.615 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-16T13:28:09.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.742+0000 7ff83eabb700 1 Processor -- start 2023-10-16T13:28:09.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.743+0000 7ff83eabb700 1 -- start start 2023-10-16T13:28:09.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.743+0000 7ff83eabb700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff83806e570 0x7ff83815e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:09.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.743+0000 7ff83eabb700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff83806a7e0 0x7ff83806ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:09.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.743+0000 7ff83eabb700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff83815b6a0 con 0x7ff83815b2c0 2023-10-16T13:28:09.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.743+0000 7ff83eabb700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff83806b1b0 con 0x7ff83806e570 2023-10-16T13:28:09.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.743+0000 7ff83eabb700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff838161b20 con 0x7ff83806a7e0 2023-10-16T13:28:09.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.744+0000 7ff837fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff83806e570 0x7ff83815e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:09.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.744+0000 7ff83c857700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff83815b2c0 0x7ff838161340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38792/0 (socket says 172.21.15.16:38792) 2023-10-16T13:28:09.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.744+0000 7ff83c857700 1 -- 172.21.15.16:0/3634434763 learned_addr learned my addr 172.21.15.16:0/3634434763 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:28:09.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.744+0000 7ff83d058700 1 --2- 172.21.15.16:0/3634434763 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff83806a7e0 0x7ff83806ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:09.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.744+0000 7ff837fff700 1 -- 172.21.15.16:0/3634434763 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff83806a7e0 msgr2=0x7ff83806ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:28:09.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.744+0000 7ff837fff700 1 --2- 172.21.15.16:0/3634434763 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff83806a7e0 0x7ff83806ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:09.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.744+0000 7ff837fff700 1 -- 172.21.15.16:0/3634434763 >> v1:172.21.15.16:6789/0 conn(0x7ff83815b2c0 legacy=0x7ff838161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:09.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.744+0000 7ff8377fe700 1 -- 172.21.15.16:0/3634434763 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1488202001 0 0) 0x7ff83815b6a0 con 0x7ff83815b2c0 2023-10-16T13:28:09.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.744+0000 7ff837fff700 1 -- 172.21.15.16:0/3634434763 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff838161e00 con 0x7ff83806e570 2023-10-16T13:28:09.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.744+0000 7ff837fff700 1 --2- 172.21.15.16:0/3634434763 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff83806e570 0x7ff83815e5a0 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 crypto rx=0x7ff824009840 tx=0x7ff824009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1345c5be468377e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:09.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.744+0000 7ff8377fe700 1 -- 172.21.15.16:0/3634434763 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff824011830 con 0x7ff83806e570 2023-10-16T13:28:09.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.744+0000 7ff8377fe700 1 -- 172.21.15.16:0/3634434763 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff824011990 con 0x7ff83806e570 2023-10-16T13:28:09.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.744+0000 7ff8377fe700 1 -- 172.21.15.16:0/3634434763 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff824011d20 con 0x7ff83806e570 2023-10-16T13:28:09.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.745+0000 7ff83eabb700 1 -- 172.21.15.16:0/3634434763 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff83806e570 msgr2=0x7ff83815e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:09.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.745+0000 7ff83eabb700 1 --2- 172.21.15.16:0/3634434763 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff83806e570 0x7ff83815e5a0 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 crypto rx=0x7ff824009840 tx=0x7ff824009b50 comp rx=0 tx=0).stop 2023-10-16T13:28:09.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.745+0000 7ff83eabb700 1 -- 172.21.15.16:0/3634434763 shutdown_connections 2023-10-16T13:28:09.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.745+0000 7ff83eabb700 1 --2- 172.21.15.16:0/3634434763 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff83806a7e0 0x7ff83806ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:09.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.745+0000 7ff83eabb700 1 --2- 172.21.15.16:0/3634434763 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff83806e570 0x7ff83815e5a0 unknown :-1 s=CLOSED pgs=2033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:09.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.745+0000 7ff83eabb700 1 -- 172.21.15.16:0/3634434763 >> 172.21.15.16:0/3634434763 conn(0x7ff8380b1f40 msgr2=0x7ff8380b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:09.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.745+0000 7ff83eabb700 1 -- 172.21.15.16:0/3634434763 shutdown_connections 2023-10-16T13:28:09.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.745+0000 7ff83eabb700 1 -- 172.21.15.16:0/3634434763 wait complete. 2023-10-16T13:28:09.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.746+0000 7ff83eabb700 1 Processor -- start 2023-10-16T13:28:09.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.746+0000 7ff83eabb700 1 -- start start 2023-10-16T13:28:09.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.747+0000 7ff83eabb700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff83806a7e0 0x7ff83815e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:09.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.747+0000 7ff83eabb700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff83806e570 0x7ff838160440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:09.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.747+0000 7ff83eabb700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff83815b2c0 0x7ff8380b50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:09.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.747+0000 7ff83eabb700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff83815f650 con 0x7ff83806e570 2023-10-16T13:28:09.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.747+0000 7ff83eabb700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff838158240 con 0x7ff83815b2c0 2023-10-16T13:28:09.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.747+0000 7ff83eabb700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff83815e9d0 con 0x7ff83806a7e0 2023-10-16T13:28:09.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.747+0000 7ff837fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff83806e570 0x7ff838160440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:09.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.747+0000 7ff837fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff83806e570 0x7ff838160440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49526/0 (socket says 172.21.15.16:49526) 2023-10-16T13:28:09.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.747+0000 7ff837fff700 1 -- 172.21.15.16:0/3778206277 learned_addr learned my addr 172.21.15.16:0/3778206277 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:09.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.747+0000 7ff837fff700 1 -- 172.21.15.16:0/3778206277 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff83806a7e0 msgr2=0x7ff83815e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:09.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.747+0000 7ff837fff700 1 --2- 172.21.15.16:0/3778206277 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff83806a7e0 0x7ff83815e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:09.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.747+0000 7ff837fff700 1 -- 172.21.15.16:0/3778206277 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff83815b2c0 msgr2=0x7ff8380b50d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:09.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.747+0000 7ff837fff700 1 --2- 172.21.15.16:0/3778206277 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff83815b2c0 0x7ff8380b50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:09.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.747+0000 7ff837fff700 1 -- 172.21.15.16:0/3778206277 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8240094a0 con 0x7ff83806e570 2023-10-16T13:28:09.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.747+0000 7ff837fff700 1 --2- 172.21.15.16:0/3778206277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff83806e570 0x7ff838160440 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 crypto rx=0x7ff838070620 tx=0x7ff824004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:09.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.748+0000 7ff8357fa700 1 -- 172.21.15.16:0/3778206277 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8240038a0 con 0x7ff83806e570 2023-10-16T13:28:09.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.748+0000 7ff8357fa700 1 -- 172.21.15.16:0/3778206277 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff82401f070 con 0x7ff83806e570 2023-10-16T13:28:09.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.748+0000 7ff83eabb700 1 -- 172.21.15.16:0/3778206277 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff83815ecb0 con 0x7ff83806e570 2023-10-16T13:28:09.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.748+0000 7ff8357fa700 1 -- 172.21.15.16:0/3778206277 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff824003c20 con 0x7ff83806e570 2023-10-16T13:28:09.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.748+0000 7ff83eabb700 1 -- 172.21.15.16:0/3778206277 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff83815f270 con 0x7ff83806e570 2023-10-16T13:28:09.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.749+0000 7ff83eabb700 1 -- 172.21.15.16:0/3778206277 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8380cf740 con 0x7ff83806e570 2023-10-16T13:28:09.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.749+0000 7ff8357fa700 1 -- 172.21.15.16:0/3778206277 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 107) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff82401a430 con 0x7ff83806e570 2023-10-16T13:28:09.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.750+0000 7ff8357fa700 1 --2- 172.21.15.16:0/3778206277 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7ff81c06b020 0x7ff81c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:09.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.750+0000 7ff8357fa700 1 -- 172.21.15.16:0/3778206277 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 6051+0+0 (secure 0 0 0) 0x7ff824021070 con 0x7ff83806e570 2023-10-16T13:28:09.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.750+0000 7ff83c857700 1 --2- 172.21.15.16:0/3778206277 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7ff81c06b020 0x7ff81c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:09.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.750+0000 7ff83c857700 1 --2- 172.21.15.16:0/3778206277 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7ff81c06b020 0x7ff81c06d4e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff82c0011e0 tx=0x7ff82c0079b0 comp rx=0 tx=0).ready entity=mgr.10047 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:09.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.754+0000 7ff8357fa700 1 -- 172.21.15.16:0/3778206277 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff8240560a0 con 0x7ff83806e570 2023-10-16T13:28:09.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.961+0000 7ff83eabb700 1 -- 172.21.15.16:0/3778206277 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7ff83815f790 con 0x7ff83806e570 2023-10-16T13:28:09.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.965+0000 7ff8357fa700 1 -- 172.21.15.16:0/3778206277 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v61) v1 ==== 115+0+0 (secure 0 0 0) 0x7ff83815f790 con 0x7ff83806e570 2023-10-16T13:28:09.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.967+0000 7ff83eabb700 1 -- 172.21.15.16:0/3778206277 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7ff81c06b020 msgr2=0x7ff81c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:09.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.967+0000 7ff83eabb700 1 --2- 172.21.15.16:0/3778206277 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7ff81c06b020 0x7ff81c06d4e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff82c0011e0 tx=0x7ff82c0079b0 comp rx=0 tx=0).stop 2023-10-16T13:28:09.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.967+0000 7ff83eabb700 1 -- 172.21.15.16:0/3778206277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff83806e570 msgr2=0x7ff838160440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:09.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.967+0000 7ff83eabb700 1 --2- 172.21.15.16:0/3778206277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff83806e570 0x7ff838160440 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 crypto rx=0x7ff838070620 tx=0x7ff824004750 comp rx=0 tx=0).stop 2023-10-16T13:28:09.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.967+0000 7ff83eabb700 1 -- 172.21.15.16:0/3778206277 shutdown_connections 2023-10-16T13:28:09.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.967+0000 7ff83eabb700 1 --2- 172.21.15.16:0/3778206277 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7ff81c06b020 0x7ff81c06d4e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:09.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.967+0000 7ff83eabb700 1 --2- 172.21.15.16:0/3778206277 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff83815b2c0 0x7ff8380b50d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:09.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.967+0000 7ff83eabb700 1 --2- 172.21.15.16:0/3778206277 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff83806a7e0 0x7ff83815e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:09.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.967+0000 7ff83eabb700 1 --2- 172.21.15.16:0/3778206277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff83806e570 0x7ff838160440 unknown :-1 s=CLOSED pgs=2034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:09.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.967+0000 7ff83eabb700 1 -- 172.21.15.16:0/3778206277 >> 172.21.15.16:0/3778206277 conn(0x7ff8380b1f40 msgr2=0x7ff838159c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:09.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.967+0000 7ff83eabb700 1 -- 172.21.15.16:0/3778206277 shutdown_connections 2023-10-16T13:28:09.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:09.967+0000 7ff83eabb700 1 -- 172.21.15.16:0/3778206277 wait complete. 2023-10-16T13:28:09.983 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-16T13:28:10.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.108+0000 7efe3e769700 1 Processor -- start 2023-10-16T13:28:10.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.108+0000 7efe3e769700 1 -- start start 2023-10-16T13:28:10.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.109+0000 7efe3e769700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7efe3806e570 0x7efe3815e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:10.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.109+0000 7efe3e769700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efe3806a7e0 0x7efe3806ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:10.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.109+0000 7efe3e769700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe3815b6a0 con 0x7efe3815b2c0 2023-10-16T13:28:10.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.109+0000 7efe3e769700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe3806b1b0 con 0x7efe3806e570 2023-10-16T13:28:10.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.109+0000 7efe3e769700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe38161b20 con 0x7efe3806a7e0 2023-10-16T13:28:10.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.109+0000 7efe37fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7efe3815b2c0 0x7efe38161340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38806/0 (socket says 172.21.15.16:38806) 2023-10-16T13:28:10.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.109+0000 7efe37fff700 1 -- 172.21.15.16:0/3638205514 learned_addr learned my addr 172.21.15.16:0/3638205514 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:28:10.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.109+0000 7efe3cd06700 1 --2- 172.21.15.16:0/3638205514 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efe3806a7e0 0x7efe3806ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:10.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.109+0000 7efe377fe700 1 --2- 172.21.15.16:0/3638205514 >> v2:172.21.15.49:3300/0 conn(0x7efe3806e570 0x7efe3815e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:10.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.109+0000 7efe36ffd700 1 -- 172.21.15.16:0/3638205514 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3137191517 0 0) 0x7efe3815b6a0 con 0x7efe3815b2c0 2023-10-16T13:28:10.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.109+0000 7efe36ffd700 1 -- 172.21.15.16:0/3638205514 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe240035b0 con 0x7efe3815b2c0 2023-10-16T13:28:10.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.110+0000 7efe36ffd700 1 -- 172.21.15.16:0/3638205514 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 20491229 0 0) 0x7efe240035b0 con 0x7efe3815b2c0 2023-10-16T13:28:10.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.110+0000 7efe36ffd700 1 -- 172.21.15.16:0/3638205514 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efe3806a7e0 msgr2=0x7efe3806ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:28:10.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.110+0000 7efe36ffd700 1 --2- 172.21.15.16:0/3638205514 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efe3806a7e0 0x7efe3806ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:10.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.110+0000 7efe36ffd700 1 -- 172.21.15.16:0/3638205514 >> v2:172.21.15.49:3300/0 conn(0x7efe3806e570 msgr2=0x7efe3815e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:28:10.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.110+0000 7efe36ffd700 1 --2- 172.21.15.16:0/3638205514 >> v2:172.21.15.49:3300/0 conn(0x7efe3806e570 0x7efe3815e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:10.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.110+0000 7efe36ffd700 1 -- 172.21.15.16:0/3638205514 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe38161e00 con 0x7efe3815b2c0 2023-10-16T13:28:10.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.110+0000 7efe36ffd700 1 -- 172.21.15.16:0/3638205514 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7efe28001670 con 0x7efe3815b2c0 2023-10-16T13:28:10.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.110+0000 7efe36ffd700 1 -- 172.21.15.16:0/3638205514 >> v1:172.21.15.16:6789/0 conn(0x7efe3815b2c0 legacy=0x7efe38161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:10.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.110+0000 7efe36ffd700 1 --2- 172.21.15.16:0/3638205514 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efe24003a00 0x7efe24003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:10.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.110+0000 7efe36ffd700 1 --2- 172.21.15.16:0/3638205514 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe240043f0 0x7efe240048d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:10.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.110+0000 7efe36ffd700 1 --2- 172.21.15.16:0/3638205514 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efe24004fe0 0x7efe2400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:10.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.110+0000 7efe36ffd700 1 -- 172.21.15.16:0/3638205514 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe3806b1b0 con 0x7efe240043f0 2023-10-16T13:28:10.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.110+0000 7efe36ffd700 1 -- 172.21.15.16:0/3638205514 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe38161b20 con 0x7efe24004fe0 2023-10-16T13:28:10.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.110+0000 7efe36ffd700 1 -- 172.21.15.16:0/3638205514 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe2400bf80 con 0x7efe24003a00 2023-10-16T13:28:10.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.110+0000 7efe3cd06700 1 --2- 172.21.15.16:0/3638205514 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efe3806a7e0 0x7efe3806ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:28:10.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.110+0000 7efe377fe700 1 --2- 172.21.15.16:0/3638205514 >> v2:172.21.15.49:3300/0 conn(0x7efe3806e570 0x7efe3815e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:28:10.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.110+0000 7efe37fff700 1 --2- 172.21.15.16:0/3638205514 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efe24003a00 0x7efe24003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:10.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.110+0000 7efe377fe700 1 --2- 172.21.15.16:0/3638205514 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe240043f0 0x7efe240048d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:10.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.110+0000 7efe3cd06700 1 --2- 172.21.15.16:0/3638205514 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efe24004fe0 0x7efe2400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:10.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.110+0000 7efe377fe700 1 -- 172.21.15.16:0/3638205514 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efe24003a00 msgr2=0x7efe24003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:28:10.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.110+0000 7efe377fe700 1 --2- 172.21.15.16:0/3638205514 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efe24003a00 0x7efe24003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:10.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.110+0000 7efe377fe700 1 -- 172.21.15.16:0/3638205514 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efe24004fe0 msgr2=0x7efe2400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:28:10.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.110+0000 7efe377fe700 1 --2- 172.21.15.16:0/3638205514 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efe24004fe0 0x7efe2400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:10.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.110+0000 7efe377fe700 1 -- 172.21.15.16:0/3638205514 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe240035b0 con 0x7efe240043f0 2023-10-16T13:28:10.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.110+0000 7efe3cd06700 1 --2- 172.21.15.16:0/3638205514 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efe24004fe0 0x7efe2400ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:28:10.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.110+0000 7efe377fe700 1 --2- 172.21.15.16:0/3638205514 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe240043f0 0x7efe240048d0 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7efe20009d10 tx=0x7efe24005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a34cba51778eae0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:10.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.110+0000 7efe36ffd700 1 -- 172.21.15.16:0/3638205514 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe20009e60 con 0x7efe240043f0 2023-10-16T13:28:10.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.110+0000 7efe36ffd700 1 -- 172.21.15.16:0/3638205514 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe20004030 con 0x7efe240043f0 2023-10-16T13:28:10.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.110+0000 7efe36ffd700 1 -- 172.21.15.16:0/3638205514 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe200043a0 con 0x7efe240043f0 2023-10-16T13:28:10.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.111+0000 7efe3e769700 1 -- 172.21.15.16:0/3638205514 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe240043f0 msgr2=0x7efe240048d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:10.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.111+0000 7efe3e769700 1 --2- 172.21.15.16:0/3638205514 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe240043f0 0x7efe240048d0 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7efe20009d10 tx=0x7efe24005880 comp rx=0 tx=0).stop 2023-10-16T13:28:10.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.111+0000 7efe3cd06700 1 -- 172.21.15.16:0/3638205514 reap_dead start 2023-10-16T13:28:10.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.111+0000 7efe3e769700 1 -- 172.21.15.16:0/3638205514 shutdown_connections 2023-10-16T13:28:10.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.111+0000 7efe3e769700 1 -- 172.21.15.16:0/3638205514 >> 172.21.15.16:0/3638205514 conn(0x7efe380b1f40 msgr2=0x7efe380b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:10.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.111+0000 7efe3e769700 1 -- 172.21.15.16:0/3638205514 shutdown_connections 2023-10-16T13:28:10.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.111+0000 7efe3e769700 1 -- 172.21.15.16:0/3638205514 wait complete. 2023-10-16T13:28:10.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.112+0000 7efe3e769700 1 Processor -- start 2023-10-16T13:28:10.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.112+0000 7efe3e769700 1 -- start start 2023-10-16T13:28:10.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.113+0000 7efe3e769700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe240043f0 0x7efe38161110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:10.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.113+0000 7efe3e769700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efe381f63a0 0x7efe381f6820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:10.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.113+0000 7efe3e769700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efe380c3ba0 0x7efe380c4020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:10.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.113+0000 7efe3e769700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe38160cf0 con 0x7efe240043f0 2023-10-16T13:28:10.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.113+0000 7efe3e769700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe380b4c30 con 0x7efe380c3ba0 2023-10-16T13:28:10.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.113+0000 7efe3e769700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe381f6d60 con 0x7efe381f63a0 2023-10-16T13:28:10.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.113+0000 7efe37fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe240043f0 0x7efe38161110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:10.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.113+0000 7efe37fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe240043f0 0x7efe38161110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49544/0 (socket says 172.21.15.16:49544) 2023-10-16T13:28:10.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.113+0000 7efe37fff700 1 -- 172.21.15.16:0/2756588691 learned_addr learned my addr 172.21.15.16:0/2756588691 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:10.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.113+0000 7efe3cd06700 1 --2- 172.21.15.16:0/2756588691 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efe380c3ba0 0x7efe380c4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:10.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.114+0000 7efe377fe700 1 --2- 172.21.15.16:0/2756588691 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efe381f63a0 0x7efe381f6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:10.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.114+0000 7efe37fff700 1 -- 172.21.15.16:0/2756588691 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efe381f63a0 msgr2=0x7efe381f6820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:10.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.114+0000 7efe37fff700 1 --2- 172.21.15.16:0/2756588691 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efe381f63a0 0x7efe381f6820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:10.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.114+0000 7efe37fff700 1 -- 172.21.15.16:0/2756588691 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efe380c3ba0 msgr2=0x7efe380c4020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:10.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.114+0000 7efe37fff700 1 --2- 172.21.15.16:0/2756588691 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efe380c3ba0 0x7efe380c4020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:10.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.114+0000 7efe37fff700 1 -- 172.21.15.16:0/2756588691 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe38160070 con 0x7efe240043f0 2023-10-16T13:28:10.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.114+0000 7efe377fe700 1 --2- 172.21.15.16:0/2756588691 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efe381f63a0 0x7efe381f6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:28:10.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.114+0000 7efe37fff700 1 --2- 172.21.15.16:0/2756588691 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe240043f0 0x7efe38161110 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 crypto rx=0x7efe28007340 tx=0x7efe28007650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:10.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.114+0000 7efe34ff9700 1 -- 172.21.15.16:0/2756588691 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe2800f660 con 0x7efe240043f0 2023-10-16T13:28:10.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.114+0000 7efe34ff9700 1 -- 172.21.15.16:0/2756588691 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe2800f800 con 0x7efe240043f0 2023-10-16T13:28:10.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.114+0000 7efe3e769700 1 -- 172.21.15.16:0/2756588691 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe38160350 con 0x7efe240043f0 2023-10-16T13:28:10.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.114+0000 7efe34ff9700 1 -- 172.21.15.16:0/2756588691 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe2800fbb0 con 0x7efe240043f0 2023-10-16T13:28:10.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.114+0000 7efe3e769700 1 -- 172.21.15.16:0/2756588691 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe38160910 con 0x7efe240043f0 2023-10-16T13:28:10.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.115+0000 7efe3e769700 1 -- 172.21.15.16:0/2756588691 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe38160ad0 con 0x7efe240043f0 2023-10-16T13:28:10.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.116+0000 7efe34ff9700 1 -- 172.21.15.16:0/2756588691 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 107) v1 ==== 89331+0+0 (secure 0 0 0) 0x7efe2800fd10 con 0x7efe240043f0 2023-10-16T13:28:10.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.116+0000 7efe34ff9700 1 --2- 172.21.15.16:0/2756588691 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7efe1806b020 0x7efe1806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:10.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.116+0000 7efe377fe700 1 --2- 172.21.15.16:0/2756588691 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7efe1806b020 0x7efe1806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:10.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.116+0000 7efe34ff9700 1 -- 172.21.15.16:0/2756588691 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 6051+0+0 (secure 0 0 0) 0x7efe280898a0 con 0x7efe240043f0 2023-10-16T13:28:10.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.117+0000 7efe377fe700 1 --2- 172.21.15.16:0/2756588691 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7efe1806b020 0x7efe1806d4e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7efe240047d0 tx=0x7efe20008030 comp rx=0 tx=0).ready entity=mgr.10047 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:10.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.121+0000 7efe34ff9700 1 -- 172.21.15.16:0/2756588691 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7efe280546e0 con 0x7efe240043f0 2023-10-16T13:28:10.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.334+0000 7efe3e769700 1 -- 172.21.15.16:0/2756588691 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7efe3804ebb0 con 0x7efe240043f0 2023-10-16T13:28:10.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.337+0000 7efe34ff9700 1 -- 172.21.15.16:0/2756588691 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v62) v1 ==== 104+0+0 (secure 0 0 0) 0x7efe28057d00 con 0x7efe240043f0 2023-10-16T13:28:10.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.339+0000 7efe3e769700 1 -- 172.21.15.16:0/2756588691 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7efe1806b020 msgr2=0x7efe1806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:10.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.339+0000 7efe3e769700 1 --2- 172.21.15.16:0/2756588691 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7efe1806b020 0x7efe1806d4e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7efe240047d0 tx=0x7efe20008030 comp rx=0 tx=0).stop 2023-10-16T13:28:10.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.339+0000 7efe3e769700 1 -- 172.21.15.16:0/2756588691 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe240043f0 msgr2=0x7efe38161110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:10.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.339+0000 7efe3e769700 1 --2- 172.21.15.16:0/2756588691 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe240043f0 0x7efe38161110 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 crypto rx=0x7efe28007340 tx=0x7efe28007650 comp rx=0 tx=0).stop 2023-10-16T13:28:10.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.339+0000 7efe3e769700 1 -- 172.21.15.16:0/2756588691 shutdown_connections 2023-10-16T13:28:10.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.339+0000 7efe3e769700 1 --2- 172.21.15.16:0/2756588691 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7efe1806b020 0x7efe1806d4e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:10.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.339+0000 7efe3e769700 1 --2- 172.21.15.16:0/2756588691 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efe380c3ba0 0x7efe380c4020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:10.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.339+0000 7efe3e769700 1 --2- 172.21.15.16:0/2756588691 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe240043f0 0x7efe38161110 unknown :-1 s=CLOSED pgs=2037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:10.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.339+0000 7efe3e769700 1 --2- 172.21.15.16:0/2756588691 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efe381f63a0 0x7efe381f6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:10.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.339+0000 7efe3e769700 1 -- 172.21.15.16:0/2756588691 >> 172.21.15.16:0/2756588691 conn(0x7efe380b1f40 msgr2=0x7efe38159440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:10.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.339+0000 7efe3e769700 1 -- 172.21.15.16:0/2756588691 shutdown_connections 2023-10-16T13:28:10.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.339+0000 7efe3e769700 1 -- 172.21.15.16:0/2756588691 wait complete. 2023-10-16T13:28:10.353 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-16T13:28:10.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.478+0000 7f30f52fe700 1 Processor -- start 2023-10-16T13:28:10.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.478+0000 7f30f52fe700 1 -- start start 2023-10-16T13:28:10.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.479+0000 7f30f52fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f30f015f060 0x7f30f00cc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:10.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.479+0000 7f30f52fe700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30f00b2440 con 0x7f30f00ce5d0 2023-10-16T13:28:10.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.479+0000 7f30f52fe700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30f00d00e0 con 0x7f30f015ec80 2023-10-16T13:28:10.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.479+0000 7f30f52fe700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30f00b2c50 con 0x7f30f015f060 2023-10-16T13:28:10.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.479+0000 7f30ef7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f30f015f060 0x7f30f00cc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:10.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.479+0000 7f30ee7fc700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f30f00ce5d0 0x7f30f00cc190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38826/0 (socket says 172.21.15.16:38826) 2023-10-16T13:28:10.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.479+0000 7f30ee7fc700 1 -- 172.21.15.16:0/1872415913 learned_addr learned my addr 172.21.15.16:0/1872415913 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:28:10.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.480+0000 7f30ef7fe700 1 -- 172.21.15.16:0/1872415913 >> v1:172.21.15.49:6789/0 conn(0x7f30f015ec80 legacy=0x7f30f00cdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:10.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.480+0000 7f30ef7fe700 1 -- 172.21.15.16:0/1872415913 >> v1:172.21.15.16:6789/0 conn(0x7f30f00ce5d0 legacy=0x7f30f00cc190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:10.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.480+0000 7f30ef7fe700 1 -- 172.21.15.16:0/1872415913 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30f00cef60 con 0x7f30f015f060 2023-10-16T13:28:10.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.480+0000 7f30ef7fe700 1 --2- 172.21.15.16:0/1872415913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30f015f060 0x7f30f00cc8a0 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7f30e8009dc0 tx=0x7f30e800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=562a4aea8c910644 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:10.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.480+0000 7f30edffb700 1 -- 172.21.15.16:0/1872415913 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30e800e070 con 0x7f30f015f060 2023-10-16T13:28:10.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.480+0000 7f30edffb700 1 -- 172.21.15.16:0/1872415913 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30e800bb90 con 0x7f30f015f060 2023-10-16T13:28:10.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.480+0000 7f30edffb700 1 -- 172.21.15.16:0/1872415913 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30e8012430 con 0x7f30f015f060 2023-10-16T13:28:10.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.481+0000 7f30f52fe700 1 -- 172.21.15.16:0/1872415913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30f015f060 msgr2=0x7f30f00cc8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:10.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.481+0000 7f30f52fe700 1 --2- 172.21.15.16:0/1872415913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30f015f060 0x7f30f00cc8a0 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7f30e8009dc0 tx=0x7f30e800b330 comp rx=0 tx=0).stop 2023-10-16T13:28:10.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.481+0000 7f30f52fe700 1 -- 172.21.15.16:0/1872415913 shutdown_connections 2023-10-16T13:28:10.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.481+0000 7f30f52fe700 1 --2- 172.21.15.16:0/1872415913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30f015f060 0x7f30f00cc8a0 unknown :-1 s=CLOSED pgs=2039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:10.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.481+0000 7f30f52fe700 1 -- 172.21.15.16:0/1872415913 >> 172.21.15.16:0/1872415913 conn(0x7f30f00d15f0 msgr2=0x7f30f00d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:10.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.481+0000 7f30f52fe700 1 -- 172.21.15.16:0/1872415913 shutdown_connections 2023-10-16T13:28:10.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.481+0000 7f30f52fe700 1 -- 172.21.15.16:0/1872415913 wait complete. 2023-10-16T13:28:10.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.482+0000 7f30f52fe700 1 Processor -- start 2023-10-16T13:28:10.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.482+0000 7f30f52fe700 1 -- start start 2023-10-16T13:28:10.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.483+0000 7f30f52fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30f00ce5d0 0x7f30f00c9e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:10.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.483+0000 7f30f52fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f30f015ec80 0x7f30f01e41a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:10.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.483+0000 7f30f52fe700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f30f015f060 0x7f30f00c8b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:10.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.483+0000 7f30f52fe700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30f01e55e0 con 0x7f30f00ce5d0 2023-10-16T13:28:10.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.483+0000 7f30f52fe700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30f01e49d0 con 0x7f30f015ec80 2023-10-16T13:28:10.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.483+0000 7f30f52fe700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30f01e4cb0 con 0x7f30f015f060 2023-10-16T13:28:10.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.484+0000 7f30eeffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30f00ce5d0 0x7f30f00c9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:10.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.484+0000 7f30eeffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30f00ce5d0 0x7f30f00c9e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49556/0 (socket says 172.21.15.16:49556) 2023-10-16T13:28:10.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.484+0000 7f30eeffd700 1 -- 172.21.15.16:0/4109883761 learned_addr learned my addr 172.21.15.16:0/4109883761 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:10.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.484+0000 7f30ee7fc700 1 --2- 172.21.15.16:0/4109883761 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f30f015ec80 0x7f30f01e41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:10.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.484+0000 7f30ef7fe700 1 --2- 172.21.15.16:0/4109883761 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f30f015f060 0x7f30f00c8b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:10.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.484+0000 7f30eeffd700 1 -- 172.21.15.16:0/4109883761 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f30f015f060 msgr2=0x7f30f00c8b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:10.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.484+0000 7f30eeffd700 1 --2- 172.21.15.16:0/4109883761 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f30f015f060 0x7f30f00c8b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:10.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.484+0000 7f30eeffd700 1 -- 172.21.15.16:0/4109883761 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f30f015ec80 msgr2=0x7f30f01e41a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:10.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.484+0000 7f30eeffd700 1 --2- 172.21.15.16:0/4109883761 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f30f015ec80 0x7f30f01e41a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:10.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.484+0000 7f30eeffd700 1 -- 172.21.15.16:0/4109883761 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30e8009a70 con 0x7f30f00ce5d0 2023-10-16T13:28:10.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.484+0000 7f30eeffd700 1 --2- 172.21.15.16:0/4109883761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30f00ce5d0 0x7f30f00c9e10 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7f30e400c4e0 tx=0x7f30e400c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:10.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.484+0000 7f30d77fe700 1 -- 172.21.15.16:0/4109883761 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30e40052d0 con 0x7f30f00ce5d0 2023-10-16T13:28:10.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.484+0000 7f30d77fe700 1 -- 172.21.15.16:0/4109883761 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30e4005430 con 0x7f30f00ce5d0 2023-10-16T13:28:10.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.484+0000 7f30f52fe700 1 -- 172.21.15.16:0/4109883761 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30f01e4f90 con 0x7f30f00ce5d0 2023-10-16T13:28:10.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.484+0000 7f30d77fe700 1 -- 172.21.15.16:0/4109883761 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30e4011a70 con 0x7f30f00ce5d0 2023-10-16T13:28:10.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.484+0000 7f30f52fe700 1 -- 172.21.15.16:0/4109883761 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30f01e46e0 con 0x7f30f00ce5d0 2023-10-16T13:28:10.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.485+0000 7f30f52fe700 1 -- 172.21.15.16:0/4109883761 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30f004ebb0 con 0x7f30f00ce5d0 2023-10-16T13:28:10.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.486+0000 7f30d77fe700 1 -- 172.21.15.16:0/4109883761 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 107) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f30e4011bd0 con 0x7f30f00ce5d0 2023-10-16T13:28:10.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.486+0000 7f30d77fe700 1 --2- 172.21.15.16:0/4109883761 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f30dc06b050 0x7f30dc06d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:10.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.486+0000 7f30d77fe700 1 -- 172.21.15.16:0/4109883761 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 6051+0+0 (secure 0 0 0) 0x7f30e408d070 con 0x7f30f00ce5d0 2023-10-16T13:28:10.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.486+0000 7f30ee7fc700 1 --2- 172.21.15.16:0/4109883761 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f30dc06b050 0x7f30dc06d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:10.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.487+0000 7f30ee7fc700 1 --2- 172.21.15.16:0/4109883761 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f30dc06b050 0x7f30dc06d510 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f30f01f3f80 tx=0x7f30e0009040 comp rx=0 tx=0).ready entity=mgr.10047 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:10.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.491+0000 7f30d77fe700 1 -- 172.21.15.16:0/4109883761 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f30e4057eb0 con 0x7f30f00ce5d0 2023-10-16T13:28:10.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.699+0000 7f30f52fe700 1 -- 172.21.15.16:0/4109883761 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f30f00b6220 con 0x7f30f00ce5d0 2023-10-16T13:28:10.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.702+0000 7f30d77fe700 1 -- 172.21.15.16:0/4109883761 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v63) v1 ==== 120+0+0 (secure 0 0 0) 0x7f30e405b4d0 con 0x7f30f00ce5d0 2023-10-16T13:28:10.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.704+0000 7f30f52fe700 1 -- 172.21.15.16:0/4109883761 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f30dc06b050 msgr2=0x7f30dc06d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:10.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.704+0000 7f30f52fe700 1 --2- 172.21.15.16:0/4109883761 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f30dc06b050 0x7f30dc06d510 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f30f01f3f80 tx=0x7f30e0009040 comp rx=0 tx=0).stop 2023-10-16T13:28:10.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.704+0000 7f30f52fe700 1 -- 172.21.15.16:0/4109883761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30f00ce5d0 msgr2=0x7f30f00c9e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:10.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.704+0000 7f30f52fe700 1 --2- 172.21.15.16:0/4109883761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30f00ce5d0 0x7f30f00c9e10 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7f30e400c4e0 tx=0x7f30e400c8a0 comp rx=0 tx=0).stop 2023-10-16T13:28:10.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.704+0000 7f30f52fe700 1 -- 172.21.15.16:0/4109883761 shutdown_connections 2023-10-16T13:28:10.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.704+0000 7f30f52fe700 1 --2- 172.21.15.16:0/4109883761 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f30f015f060 0x7f30f00c8b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:10.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.704+0000 7f30f52fe700 1 --2- 172.21.15.16:0/4109883761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30f00ce5d0 0x7f30f00c9e10 unknown :-1 s=CLOSED pgs=2040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:10.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.704+0000 7f30f52fe700 1 --2- 172.21.15.16:0/4109883761 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f30dc06b050 0x7f30dc06d510 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:10.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.704+0000 7f30f52fe700 1 --2- 172.21.15.16:0/4109883761 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f30f015ec80 0x7f30f01e41a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:10.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.705+0000 7f30f52fe700 1 -- 172.21.15.16:0/4109883761 >> 172.21.15.16:0/4109883761 conn(0x7f30f00d15f0 msgr2=0x7f30f0153b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:10.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.705+0000 7f30f52fe700 1 -- 172.21.15.16:0/4109883761 shutdown_connections 2023-10-16T13:28:10.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.705+0000 7f30f52fe700 1 -- 172.21.15.16:0/4109883761 wait complete. 2023-10-16T13:28:10.720 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove' 2023-10-16T13:28:10.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.843+0000 7f06d07f8700 1 Processor -- start 2023-10-16T13:28:10.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.843+0000 7f06d07f8700 1 -- start start 2023-10-16T13:28:10.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.844+0000 7f06d07f8700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f06c815b630 0x7f06c80cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:10.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.844+0000 7f06d07f8700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f06c815ba10 0x7f06c80cdde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:10.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.844+0000 7f06d07f8700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f06c80ce320 0x7f06c80cfca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:10.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.844+0000 7f06d07f8700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06c80cdb20 con 0x7f06c815ba10 2023-10-16T13:28:10.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.844+0000 7f06d07f8700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06c80ce770 con 0x7f06c80ce320 2023-10-16T13:28:10.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.844+0000 7f06d07f8700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06c80cc2a0 con 0x7f06c815b630 2023-10-16T13:28:10.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.844+0000 7f06cdd93700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f06c815ba10 0x7f06c80cdde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:10.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.844+0000 7f06cdd93700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f06c815ba10 0x7f06c80cdde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49566/0 (socket says 172.21.15.16:49566) 2023-10-16T13:28:10.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.844+0000 7f06cdd93700 1 -- 172.21.15.16:0/1400576057 learned_addr learned my addr 172.21.15.16:0/1400576057 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:10.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.844+0000 7f06ced95700 1 --2- 172.21.15.16:0/1400576057 >> v2:172.21.15.49:3300/0 conn(0x7f06c80ce320 0x7f06c80cfca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:10.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.844+0000 7f06cdd93700 1 -- 172.21.15.16:0/1400576057 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f06c815b630 msgr2=0x7f06c80cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:28:10.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.844+0000 7f06ce594700 1 --2- 172.21.15.16:0/1400576057 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f06c815b630 0x7f06c80cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:10.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.844+0000 7f06cdd93700 1 --2- 172.21.15.16:0/1400576057 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f06c815b630 0x7f06c80cd5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:10.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.844+0000 7f06cdd93700 1 -- 172.21.15.16:0/1400576057 >> v2:172.21.15.49:3300/0 conn(0x7f06c80ce320 msgr2=0x7f06c80cfca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:28:10.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.844+0000 7f06cdd93700 1 --2- 172.21.15.16:0/1400576057 >> v2:172.21.15.49:3300/0 conn(0x7f06c80ce320 0x7f06c80cfca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:10.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.844+0000 7f06cdd93700 1 -- 172.21.15.16:0/1400576057 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06c80cc540 con 0x7f06c815ba10 2023-10-16T13:28:10.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.844+0000 7f06ced95700 1 --2- 172.21.15.16:0/1400576057 >> v2:172.21.15.49:3300/0 conn(0x7f06c80ce320 0x7f06c80cfca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:28:10.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.844+0000 7f06ce594700 1 --2- 172.21.15.16:0/1400576057 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f06c815b630 0x7f06c80cd5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:28:10.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.844+0000 7f06cdd93700 1 --2- 172.21.15.16:0/1400576057 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06c815ba10 0x7f06c80cdde0 secure :-1 s=READY pgs=2041 cs=0 l=1 rev1=1 crypto rx=0x7f06c806e990 tx=0x7f06b8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b35f1e2b85fd4a7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:10.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.845+0000 7f06cd592700 1 -- 172.21.15.16:0/1400576057 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06b800b870 con 0x7f06c815ba10 2023-10-16T13:28:10.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.845+0000 7f06cd592700 1 -- 172.21.15.16:0/1400576057 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06b800b9d0 con 0x7f06c815ba10 2023-10-16T13:28:10.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.845+0000 7f06cd592700 1 -- 172.21.15.16:0/1400576057 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06b800bd60 con 0x7f06c815ba10 2023-10-16T13:28:10.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.845+0000 7f06d07f8700 1 -- 172.21.15.16:0/1400576057 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06c815ba10 msgr2=0x7f06c80cdde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:10.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.845+0000 7f06d07f8700 1 --2- 172.21.15.16:0/1400576057 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06c815ba10 0x7f06c80cdde0 secure :-1 s=READY pgs=2041 cs=0 l=1 rev1=1 crypto rx=0x7f06c806e990 tx=0x7f06b8009bd0 comp rx=0 tx=0).stop 2023-10-16T13:28:10.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.845+0000 7f06d07f8700 1 -- 172.21.15.16:0/1400576057 shutdown_connections 2023-10-16T13:28:10.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.845+0000 7f06d07f8700 1 --2- 172.21.15.16:0/1400576057 >> v2:172.21.15.49:3300/0 conn(0x7f06c80ce320 0x7f06c80cfca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:10.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.845+0000 7f06d07f8700 1 --2- 172.21.15.16:0/1400576057 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f06c815b630 0x7f06c80cd5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:10.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.845+0000 7f06d07f8700 1 --2- 172.21.15.16:0/1400576057 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06c815ba10 0x7f06c80cdde0 unknown :-1 s=CLOSED pgs=2041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:10.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.845+0000 7f06d07f8700 1 -- 172.21.15.16:0/1400576057 >> 172.21.15.16:0/1400576057 conn(0x7f06c80d14b0 msgr2=0x7f06c80d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:10.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.845+0000 7f06d07f8700 1 -- 172.21.15.16:0/1400576057 shutdown_connections 2023-10-16T13:28:10.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.845+0000 7f06d07f8700 1 -- 172.21.15.16:0/1400576057 wait complete. 2023-10-16T13:28:10.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.846+0000 7f06d07f8700 1 Processor -- start 2023-10-16T13:28:10.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.847+0000 7f06d07f8700 1 -- start start 2023-10-16T13:28:10.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.847+0000 7f06d07f8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06c80ce320 0x7f06c814fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:10.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.847+0000 7f06d07f8700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f06c815b630 0x7f06c80cacd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:10.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.847+0000 7f06d07f8700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f06c815ba10 0x7f06c806fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:10.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.847+0000 7f06d07f8700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06c809bee0 con 0x7f06c80ce320 2023-10-16T13:28:10.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.847+0000 7f06d07f8700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06c8150000 con 0x7f06c815b630 2023-10-16T13:28:10.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.847+0000 7f06d07f8700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06c809b2d0 con 0x7f06c815ba10 2023-10-16T13:28:10.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.847+0000 7f06ce594700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06c80ce320 0x7f06c814fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:10.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.847+0000 7f06ce594700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06c80ce320 0x7f06c814fac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49570/0 (socket says 172.21.15.16:49570) 2023-10-16T13:28:10.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.847+0000 7f06ce594700 1 -- 172.21.15.16:0/637638538 learned_addr learned my addr 172.21.15.16:0/637638538 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:10.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.848+0000 7f06ce594700 1 -- 172.21.15.16:0/637638538 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f06c815ba10 msgr2=0x7f06c806fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:10.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.848+0000 7f06ced95700 1 --2- 172.21.15.16:0/637638538 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f06c815ba10 0x7f06c806fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:10.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.848+0000 7f06ce594700 1 --2- 172.21.15.16:0/637638538 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f06c815ba10 0x7f06c806fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:10.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.848+0000 7f06ce594700 1 -- 172.21.15.16:0/637638538 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f06c815b630 msgr2=0x7f06c80cacd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:10.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.848+0000 7f06cdd93700 1 --2- 172.21.15.16:0/637638538 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f06c815b630 0x7f06c80cacd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:10.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.848+0000 7f06ce594700 1 --2- 172.21.15.16:0/637638538 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f06c815b630 0x7f06c80cacd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:10.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.848+0000 7f06ce594700 1 -- 172.21.15.16:0/637638538 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06b8009550 con 0x7f06c80ce320 2023-10-16T13:28:10.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.848+0000 7f06ce594700 1 --2- 172.21.15.16:0/637638538 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06c80ce320 0x7f06c814fac0 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7f06c400c7f0 tx=0x7f06c400cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:10.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.848+0000 7f06beffd700 1 -- 172.21.15.16:0/637638538 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06c4005c20 con 0x7f06c80ce320 2023-10-16T13:28:10.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.848+0000 7f06beffd700 1 -- 172.21.15.16:0/637638538 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06c4005d80 con 0x7f06c80ce320 2023-10-16T13:28:10.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.848+0000 7f06d07f8700 1 -- 172.21.15.16:0/637638538 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06c809b5b0 con 0x7f06c80ce320 2023-10-16T13:28:10.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.848+0000 7f06beffd700 1 -- 172.21.15.16:0/637638538 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06c4011820 con 0x7f06c80ce320 2023-10-16T13:28:10.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.848+0000 7f06d07f8700 1 -- 172.21.15.16:0/637638538 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06c809bb70 con 0x7f06c80ce320 2023-10-16T13:28:10.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.849+0000 7f06d07f8700 1 -- 172.21.15.16:0/637638538 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06c80b73b0 con 0x7f06c80ce320 2023-10-16T13:28:10.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.850+0000 7f06beffd700 1 -- 172.21.15.16:0/637638538 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 107) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f06c4011980 con 0x7f06c80ce320 2023-10-16T13:28:10.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.850+0000 7f06beffd700 1 --2- 172.21.15.16:0/637638538 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f06ac06b020 0x7f06ac06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:10.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.850+0000 7f06beffd700 1 -- 172.21.15.16:0/637638538 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 6051+0+0 (secure 0 0 0) 0x7f06c40897b0 con 0x7f06c80ce320 2023-10-16T13:28:10.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.850+0000 7f06cdd93700 1 --2- 172.21.15.16:0/637638538 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f06ac06b020 0x7f06ac06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:10.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.851+0000 7f06cdd93700 1 --2- 172.21.15.16:0/637638538 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f06ac06b020 0x7f06ac06d4e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f06c80b6f60 tx=0x7f06b8008030 comp rx=0 tx=0).ready entity=mgr.10047 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:10.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:10.855+0000 7f06beffd700 1 -- 172.21.15.16:0/637638538 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f06c40544e0 con 0x7f06c80ce320 2023-10-16T13:28:11.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.141+0000 7f06d07f8700 1 -- 172.21.15.16:0/637638538 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]} v 0) v1 -- 0x7f06c804ebb0 con 0x7f06c80ce320 2023-10-16T13:28:11.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.589+0000 7f06beffd700 1 -- 172.21.15.16:0/637638538 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]}]=0 v2536) v1 ==== 174+0+0 (secure 0 0 0) 0x7f06c4057b00 con 0x7f06c80ce320 2023-10-16T13:28:11.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.591+0000 7f06d07f8700 1 -- 172.21.15.16:0/637638538 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f06ac06b020 msgr2=0x7f06ac06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:11.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.591+0000 7f06d07f8700 1 --2- 172.21.15.16:0/637638538 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f06ac06b020 0x7f06ac06d4e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f06c80b6f60 tx=0x7f06b8008030 comp rx=0 tx=0).stop 2023-10-16T13:28:11.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.591+0000 7f06d07f8700 1 -- 172.21.15.16:0/637638538 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06c80ce320 msgr2=0x7f06c814fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:11.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.591+0000 7f06d07f8700 1 --2- 172.21.15.16:0/637638538 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06c80ce320 0x7f06c814fac0 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7f06c400c7f0 tx=0x7f06c400cbb0 comp rx=0 tx=0).stop 2023-10-16T13:28:11.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.592+0000 7f06d07f8700 1 -- 172.21.15.16:0/637638538 shutdown_connections 2023-10-16T13:28:11.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.592+0000 7f06d07f8700 1 --2- 172.21.15.16:0/637638538 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f06c815ba10 0x7f06c806fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:11.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.592+0000 7f06d07f8700 1 --2- 172.21.15.16:0/637638538 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06c80ce320 0x7f06c814fac0 unknown :-1 s=CLOSED pgs=2042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:11.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.592+0000 7f06d07f8700 1 --2- 172.21.15.16:0/637638538 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f06ac06b020 0x7f06ac06d4e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:11.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.592+0000 7f06d07f8700 1 --2- 172.21.15.16:0/637638538 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f06c815b630 0x7f06c80cacd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:11.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.592+0000 7f06d07f8700 1 -- 172.21.15.16:0/637638538 >> 172.21.15.16:0/637638538 conn(0x7f06c80d14b0 msgr2=0x7f06c80b48f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:11.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.592+0000 7f06d07f8700 1 -- 172.21.15.16:0/637638538 shutdown_connections 2023-10-16T13:28:11.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.592+0000 7f06d07f8700 1 -- 172.21.15.16:0/637638538 wait complete. 2023-10-16T13:28:11.608 INFO:tasks.cephfs_test_runner:Test listing snap schedules post removal of a schedule ... ok 2023-10-16T13:28:11.608 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_with_mgr_restart (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test that snap schedule is resumed after mgr restart 2023-10-16T13:28:11.608 INFO:tasks.cephfs_test_runner:test_snap_schedule_with_mgr_restart (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-16T13:28:11.609 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:28:11.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.735+0000 7fd2aa979700 1 Processor -- start 2023-10-16T13:28:11.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.735+0000 7fd2aa979700 1 -- start start 2023-10-16T13:28:11.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.736+0000 7fd2aa979700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd2a40cf880 0x7fd2a40b52f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:11.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.736+0000 7fd2aa979700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fd2a40b5830 0x7fd2a415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:11.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.736+0000 7fd2aa979700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2a415bc80 con 0x7fd2a40cf4a0 2023-10-16T13:28:11.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.736+0000 7fd2aa979700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2a415b070 con 0x7fd2a40cf880 2023-10-16T13:28:11.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.736+0000 7fd2aa979700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2a415b350 con 0x7fd2a40b5830 2023-10-16T13:28:11.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.736+0000 7fd2a37fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd2a40cf880 0x7fd2a40b52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:11.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.736+0000 7fd2a37fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd2a40cf880 0x7fd2a40b52f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49580/0 (socket says 172.21.15.16:49580) 2023-10-16T13:28:11.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.736+0000 7fd2a37fe700 1 -- 172.21.15.16:0/857255652 learned_addr learned my addr 172.21.15.16:0/857255652 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:11.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.736+0000 7fd2a8f16700 1 --2- 172.21.15.16:0/857255652 >> v2:172.21.15.49:3300/0 conn(0x7fd2a40b5830 0x7fd2a415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:11.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.736+0000 7fd2a37fe700 1 -- 172.21.15.16:0/857255652 >> v2:172.21.15.49:3300/0 conn(0x7fd2a40b5830 msgr2=0x7fd2a415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:28:11.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.736+0000 7fd2a37fe700 1 --2- 172.21.15.16:0/857255652 >> v2:172.21.15.49:3300/0 conn(0x7fd2a40b5830 0x7fd2a415a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:11.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.736+0000 7fd2a37fe700 1 -- 172.21.15.16:0/857255652 >> v1:172.21.15.49:6789/0 conn(0x7fd2a40cf4a0 legacy=0x7fd2a40726e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:28:11.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.736+0000 7fd2a37fe700 1 -- 172.21.15.16:0/857255652 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2a415b630 con 0x7fd2a40cf880 2023-10-16T13:28:11.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.736+0000 7fd2a8f16700 1 --2- 172.21.15.16:0/857255652 >> v2:172.21.15.49:3300/0 conn(0x7fd2a40b5830 0x7fd2a415a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:28:11.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.737+0000 7fd2a37fe700 1 --2- 172.21.15.16:0/857255652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2a40cf880 0x7fd2a40b52f0 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7fd2a4072df0 tx=0x7fd28c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e385d7f1423a602 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:11.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.737+0000 7fd2a2ffd700 1 -- 172.21.15.16:0/857255652 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd28c004230 con 0x7fd2a40cf880 2023-10-16T13:28:11.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.737+0000 7fd2a2ffd700 1 -- 172.21.15.16:0/857255652 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd28c004390 con 0x7fd2a40cf880 2023-10-16T13:28:11.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.737+0000 7fd2a2ffd700 1 -- 172.21.15.16:0/857255652 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd28c0036a0 con 0x7fd2a40cf880 2023-10-16T13:28:11.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.737+0000 7fd2aa979700 1 -- 172.21.15.16:0/857255652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2a40cf880 msgr2=0x7fd2a40b52f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:11.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.737+0000 7fd2aa979700 1 --2- 172.21.15.16:0/857255652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2a40cf880 0x7fd2a40b52f0 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7fd2a4072df0 tx=0x7fd28c009bd0 comp rx=0 tx=0).stop 2023-10-16T13:28:11.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.737+0000 7fd2aa979700 1 -- 172.21.15.16:0/857255652 shutdown_connections 2023-10-16T13:28:11.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.737+0000 7fd2aa979700 1 --2- 172.21.15.16:0/857255652 >> v2:172.21.15.49:3300/0 conn(0x7fd2a40b5830 0x7fd2a415a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:11.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.737+0000 7fd2aa979700 1 --2- 172.21.15.16:0/857255652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2a40cf880 0x7fd2a40b52f0 unknown :-1 s=CLOSED pgs=2043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:11.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.737+0000 7fd2aa979700 1 -- 172.21.15.16:0/857255652 >> 172.21.15.16:0/857255652 conn(0x7fd2a40b1fd0 msgr2=0x7fd2a40b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:11.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.737+0000 7fd2aa979700 1 -- 172.21.15.16:0/857255652 shutdown_connections 2023-10-16T13:28:11.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.737+0000 7fd2aa979700 1 -- 172.21.15.16:0/857255652 wait complete. 2023-10-16T13:28:11.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.738+0000 7fd2aa979700 1 Processor -- start 2023-10-16T13:28:11.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.739+0000 7fd2aa979700 1 -- start start 2023-10-16T13:28:11.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.739+0000 7fd2aa979700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd2a40b5830 0x7fd2a4156ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:11.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.739+0000 7fd2aa979700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd2a40cf4a0 0x7fd2a40c6720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:11.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.739+0000 7fd2aa979700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2a40cf880 0x7fd2a41f30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:11.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.739+0000 7fd2aa979700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a40af850 con 0x7fd2a40cf880 2023-10-16T13:28:11.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.739+0000 7fd2aa979700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a40aec40 con 0x7fd2a40b5830 2023-10-16T13:28:11.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.739+0000 7fd2aa979700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a40aef20 con 0x7fd2a40cf4a0 2023-10-16T13:28:11.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.739+0000 7fd2a8f16700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2a40cf880 0x7fd2a41f30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:11.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.740+0000 7fd2a8f16700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2a40cf880 0x7fd2a41f30c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49588/0 (socket says 172.21.15.16:49588) 2023-10-16T13:28:11.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.740+0000 7fd2a8f16700 1 -- 172.21.15.16:0/2813427116 learned_addr learned my addr 172.21.15.16:0/2813427116 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:11.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.740+0000 7fd2a3fff700 1 --2- 172.21.15.16:0/2813427116 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd2a40b5830 0x7fd2a4156ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:11.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.740+0000 7fd2a37fe700 1 --2- 172.21.15.16:0/2813427116 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd2a40cf4a0 0x7fd2a40c6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:11.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.740+0000 7fd2a8f16700 1 -- 172.21.15.16:0/2813427116 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd2a40cf4a0 msgr2=0x7fd2a40c6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:11.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.740+0000 7fd2a8f16700 1 --2- 172.21.15.16:0/2813427116 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd2a40cf4a0 0x7fd2a40c6720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:11.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.740+0000 7fd2a8f16700 1 -- 172.21.15.16:0/2813427116 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd2a40b5830 msgr2=0x7fd2a4156ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:11.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.740+0000 7fd2a8f16700 1 --2- 172.21.15.16:0/2813427116 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd2a40b5830 0x7fd2a4156ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:11.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.740+0000 7fd2a8f16700 1 -- 172.21.15.16:0/2813427116 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd28c009550 con 0x7fd2a40cf880 2023-10-16T13:28:11.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.740+0000 7fd2a3fff700 1 --2- 172.21.15.16:0/2813427116 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd2a40b5830 0x7fd2a4156ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:28:11.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.740+0000 7fd2a37fe700 1 --2- 172.21.15.16:0/2813427116 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd2a40cf4a0 0x7fd2a40c6720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:28:11.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.740+0000 7fd2a8f16700 1 --2- 172.21.15.16:0/2813427116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2a40cf880 0x7fd2a41f30c0 secure :-1 s=READY pgs=2044 cs=0 l=1 rev1=1 crypto rx=0x7fd29c00cef0 tx=0x7fd29c00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:11.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.740+0000 7fd2a0ff9700 1 -- 172.21.15.16:0/2813427116 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd29c00ae50 con 0x7fd2a40cf880 2023-10-16T13:28:11.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.740+0000 7fd2a0ff9700 1 -- 172.21.15.16:0/2813427116 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd29c015440 con 0x7fd2a40cf880 2023-10-16T13:28:11.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.740+0000 7fd2a0ff9700 1 -- 172.21.15.16:0/2813427116 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd29c0157f0 con 0x7fd2a40cf880 2023-10-16T13:28:11.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.740+0000 7fd2aa979700 1 -- 172.21.15.16:0/2813427116 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2a40af200 con 0x7fd2a40cf880 2023-10-16T13:28:11.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.740+0000 7fd2aa979700 1 -- 172.21.15.16:0/2813427116 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2a40c6c60 con 0x7fd2a40cf880 2023-10-16T13:28:11.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.741+0000 7fd2aa979700 1 -- 172.21.15.16:0/2813427116 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2a40afa00 con 0x7fd2a40cf880 2023-10-16T13:28:11.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.742+0000 7fd2a0ff9700 1 -- 172.21.15.16:0/2813427116 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 107) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd29c00d430 con 0x7fd2a40cf880 2023-10-16T13:28:11.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.742+0000 7fd2a0ff9700 1 --2- 172.21.15.16:0/2813427116 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7fd28406b020 0x7fd28406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:11.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.742+0000 7fd2a0ff9700 1 -- 172.21.15.16:0/2813427116 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 6051+0+0 (secure 0 0 0) 0x7fd29c089e40 con 0x7fd2a40cf880 2023-10-16T13:28:11.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.742+0000 7fd2a3fff700 1 --2- 172.21.15.16:0/2813427116 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7fd28406b020 0x7fd28406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:11.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.743+0000 7fd2a3fff700 1 --2- 172.21.15.16:0/2813427116 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7fd28406b020 0x7fd28406d4e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd2a415a7c0 tx=0x7fd294009040 comp rx=0 tx=0).ready entity=mgr.10047 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:11.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.747+0000 7fd2a0ff9700 1 -- 172.21.15.16:0/2813427116 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd2a40afa00 con 0x7fd2a40cf880 2023-10-16T13:28:11.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.952+0000 7fd2aa979700 1 -- 172.21.15.16:0/2813427116 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd2a404ebb0 con 0x7fd2a40cf880 2023-10-16T13:28:11.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.953+0000 7fd2a0ff9700 1 -- 172.21.15.16:0/2813427116 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v182) v1 ==== 74+0+14252 (secure 0 0 0) 0x7fd29c01a020 con 0x7fd2a40cf880 2023-10-16T13:28:11.955 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:28:11.955 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":182,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:28:09.583328+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.49:6834/2218212001":"2023-10-17T13:28:06.241503+0000","172.21.15.16:0/439929087":"2023-10-17T13:28:05.221887+0000","172.21.15.16:0/1761844403":"2023-10-17T13:28:05.221887+0000","172.21.15.16:0/2092694398":"2023-10-17T13:28:05.221887+0000","172.21.15.16:0/1458495329":"2023-10-17T13:28:05.221887+0000","172.21.15.16:6833/3622553136":"2023-10-17T13:28:05.221887+0000","172.21.15.16:6832/3622553136":"2023-10-17T13:28:05.221887+0000","172.21.15.16:0/3947060028":"2023-10-17T13:28:05.221887+0000","172.21.15.16:0/3444187941":"2023-10-17T13:28:05.221887+0000","172.21.15.16:0/472482039":"2023-10-17T13:27:56.552715+0000","172.21.15.49:6832/2218212001":"2023-10-17T13:28:06.241503+0000","172.21.15.16:0/1822527107":"2023-10-17T13:27:56.552715+0000","172.21.15.16:0/3919013455":"2023-10-17T13:27:56.552715+0000","172.21.15.16:6833/2329471558":"2023-10-17T13:27:56.552715+0000","172.21.15.16:6832/2329471558":"2023-10-17T13:27:56.552715+0000","172.21.15.16:0/508475708":"2023-10-17T13:27:56.552715+0000","172.21.15.16:0/353895884":"2023-10-17T13:27:56.552715+0000","172.21.15.16:0/2733317665":"2023-10-17T13:27:56.552715+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:28:11.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.955+0000 7fd2aa979700 1 -- 172.21.15.16:0/2813427116 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7fd28406b020 msgr2=0x7fd28406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:11.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.955+0000 7fd2aa979700 1 --2- 172.21.15.16:0/2813427116 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7fd28406b020 0x7fd28406d4e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd2a415a7c0 tx=0x7fd294009040 comp rx=0 tx=0).stop 2023-10-16T13:28:11.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.955+0000 7fd2aa979700 1 -- 172.21.15.16:0/2813427116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2a40cf880 msgr2=0x7fd2a41f30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:11.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.955+0000 7fd2aa979700 1 --2- 172.21.15.16:0/2813427116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2a40cf880 0x7fd2a41f30c0 secure :-1 s=READY pgs=2044 cs=0 l=1 rev1=1 crypto rx=0x7fd29c00cef0 tx=0x7fd29c00a340 comp rx=0 tx=0).stop 2023-10-16T13:28:11.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.955+0000 7fd2aa979700 1 -- 172.21.15.16:0/2813427116 shutdown_connections 2023-10-16T13:28:11.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.955+0000 7fd2aa979700 1 --2- 172.21.15.16:0/2813427116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2a40cf880 0x7fd2a41f30c0 unknown :-1 s=CLOSED pgs=2044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:11.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.955+0000 7fd2aa979700 1 --2- 172.21.15.16:0/2813427116 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7fd28406b020 0x7fd28406d4e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:11.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.955+0000 7fd2aa979700 1 --2- 172.21.15.16:0/2813427116 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd2a40b5830 0x7fd2a4156ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:11.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.955+0000 7fd2aa979700 1 --2- 172.21.15.16:0/2813427116 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd2a40cf4a0 0x7fd2a40c6720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:11.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.955+0000 7fd2aa979700 1 -- 172.21.15.16:0/2813427116 >> 172.21.15.16:0/2813427116 conn(0x7fd2a40b1fd0 msgr2=0x7fd2a4159c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:11.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.955+0000 7fd2aa979700 1 -- 172.21.15.16:0/2813427116 shutdown_connections 2023-10-16T13:28:11.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:11.955+0000 7fd2aa979700 1 -- 172.21.15.16:0/2813427116 wait complete. 2023-10-16T13:28:11.970 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T13:28:11.970 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T13:28:12.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.096+0000 7f2fdf3c1700 1 Processor -- start 2023-10-16T13:28:12.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.096+0000 7f2fdf3c1700 1 -- start start 2023-10-16T13:28:12.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.097+0000 7f2fdf3c1700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2fd815b790 0x7f2fd80cda60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:12.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.097+0000 7f2fdf3c1700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fd80ce0e0 con 0x7f2fd806e070 2023-10-16T13:28:12.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.097+0000 7f2fdf3c1700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fd806a7e0 con 0x7f2fd815bb70 2023-10-16T13:28:12.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.097+0000 7f2fdf3c1700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fd815c410 con 0x7f2fd815b790 2023-10-16T13:28:12.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.097+0000 7f2fdd15d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2fd815b790 0x7f2fd80cda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:12.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.097+0000 7f2fdd95e700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2fd806e070 0x7f2fd80ce850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38870/0 (socket says 172.21.15.16:38870) 2023-10-16T13:28:12.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.097+0000 7f2fdd95e700 1 -- 172.21.15.16:0/1507531592 learned_addr learned my addr 172.21.15.16:0/1507531592 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:28:12.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.097+0000 7f2fdd15d700 1 -- 172.21.15.16:0/1507531592 >> v1:172.21.15.49:6789/0 conn(0x7f2fd815bb70 legacy=0x7f2fd806d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:12.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.097+0000 7f2fdd15d700 1 -- 172.21.15.16:0/1507531592 >> v1:172.21.15.16:6789/0 conn(0x7f2fd806e070 legacy=0x7f2fd80ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:12.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.097+0000 7f2fcffff700 1 -- 172.21.15.16:0/1507531592 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 233778659 0 0) 0x7f2fd80ce0e0 con 0x7f2fd806e070 2023-10-16T13:28:12.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.097+0000 7f2fdd15d700 1 -- 172.21.15.16:0/1507531592 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fd815c6f0 con 0x7f2fd815b790 2023-10-16T13:28:12.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.097+0000 7f2fdd15d700 1 --2- 172.21.15.16:0/1507531592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2fd815b790 0x7f2fd80cda60 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7f2fd4009840 tx=0x7f2fd4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba36462e999d22c9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:12.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.098+0000 7f2fcffff700 1 -- 172.21.15.16:0/1507531592 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd4011830 con 0x7f2fd815b790 2023-10-16T13:28:12.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.098+0000 7f2fcffff700 1 -- 172.21.15.16:0/1507531592 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fd4011990 con 0x7f2fd815b790 2023-10-16T13:28:12.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.098+0000 7f2fcffff700 1 -- 172.21.15.16:0/1507531592 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd4011d20 con 0x7f2fd815b790 2023-10-16T13:28:12.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.098+0000 7f2fdf3c1700 1 -- 172.21.15.16:0/1507531592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2fd815b790 msgr2=0x7f2fd80cda60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:12.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.098+0000 7f2fdf3c1700 1 --2- 172.21.15.16:0/1507531592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2fd815b790 0x7f2fd80cda60 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7f2fd4009840 tx=0x7f2fd4009b50 comp rx=0 tx=0).stop 2023-10-16T13:28:12.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.098+0000 7f2fdf3c1700 1 -- 172.21.15.16:0/1507531592 shutdown_connections 2023-10-16T13:28:12.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.098+0000 7f2fdf3c1700 1 --2- 172.21.15.16:0/1507531592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2fd815b790 0x7f2fd80cda60 unknown :-1 s=CLOSED pgs=2046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:12.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.098+0000 7f2fdf3c1700 1 -- 172.21.15.16:0/1507531592 >> 172.21.15.16:0/1507531592 conn(0x7f2fd80cc5e0 msgr2=0x7f2fd80cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:12.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.098+0000 7f2fdf3c1700 1 -- 172.21.15.16:0/1507531592 shutdown_connections 2023-10-16T13:28:12.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.098+0000 7f2fdf3c1700 1 -- 172.21.15.16:0/1507531592 wait complete. 2023-10-16T13:28:12.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.100+0000 7f2fdf3c1700 1 Processor -- start 2023-10-16T13:28:12.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.100+0000 7f2fdf3c1700 1 -- start start 2023-10-16T13:28:12.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.100+0000 7f2fdf3c1700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2fd806e070 0x7f2fd81548b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:12.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.100+0000 7f2fdf3c1700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2fd815b790 0x7f2fd80c81e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:12.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.100+0000 7f2fdf3c1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2fd815bb70 0x7f2fd80c56d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:12.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.100+0000 7f2fdf3c1700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd80c34c0 con 0x7f2fd815bb70 2023-10-16T13:28:12.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.100+0000 7f2fdf3c1700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd80c6190 con 0x7f2fd806e070 2023-10-16T13:28:12.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.100+0000 7f2fdf3c1700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd80c3710 con 0x7f2fd815b790 2023-10-16T13:28:12.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.100+0000 7f2fdd95e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2fd815bb70 0x7f2fd80c56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:12.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.100+0000 7f2fdd95e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2fd815bb70 0x7f2fd80c56d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49602/0 (socket says 172.21.15.16:49602) 2023-10-16T13:28:12.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.100+0000 7f2fdd95e700 1 -- 172.21.15.16:0/1056354681 learned_addr learned my addr 172.21.15.16:0/1056354681 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:12.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.101+0000 7f2fdd15d700 1 --2- 172.21.15.16:0/1056354681 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2fd806e070 0x7f2fd81548b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:12.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.101+0000 7f2fdc95c700 1 --2- 172.21.15.16:0/1056354681 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2fd815b790 0x7f2fd80c81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:12.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.101+0000 7f2fdd95e700 1 -- 172.21.15.16:0/1056354681 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2fd815b790 msgr2=0x7f2fd80c81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:12.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.101+0000 7f2fdd95e700 1 --2- 172.21.15.16:0/1056354681 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2fd815b790 0x7f2fd80c81e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:12.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.101+0000 7f2fdd95e700 1 -- 172.21.15.16:0/1056354681 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2fd806e070 msgr2=0x7f2fd81548b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:12.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.101+0000 7f2fdd95e700 1 --2- 172.21.15.16:0/1056354681 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2fd806e070 0x7f2fd81548b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:12.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.101+0000 7f2fdd95e700 1 -- 172.21.15.16:0/1056354681 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fd40094a0 con 0x7f2fd815bb70 2023-10-16T13:28:12.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.101+0000 7f2fdc95c700 1 --2- 172.21.15.16:0/1056354681 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2fd815b790 0x7f2fd80c81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:28:12.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.101+0000 7f2fdd95e700 1 --2- 172.21.15.16:0/1056354681 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2fd815bb70 0x7f2fd80c56d0 secure :-1 s=READY pgs=2047 cs=0 l=1 rev1=1 crypto rx=0x7f2fd000cc70 tx=0x7f2fd000cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:12.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.101+0000 7f2fcdffb700 1 -- 172.21.15.16:0/1056354681 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd000ab50 con 0x7f2fd815bb70 2023-10-16T13:28:12.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.101+0000 7f2fcdffb700 1 -- 172.21.15.16:0/1056354681 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fd000acd0 con 0x7f2fd815bb70 2023-10-16T13:28:12.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.101+0000 7f2fdf3c1700 1 -- 172.21.15.16:0/1056354681 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fd80c39f0 con 0x7f2fd815bb70 2023-10-16T13:28:12.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.101+0000 7f2fcdffb700 1 -- 172.21.15.16:0/1056354681 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd0007310 con 0x7f2fd815bb70 2023-10-16T13:28:12.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.101+0000 7f2fdf3c1700 1 -- 172.21.15.16:0/1056354681 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fd8154df0 con 0x7f2fd815bb70 2023-10-16T13:28:12.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.102+0000 7f2fdf3c1700 1 -- 172.21.15.16:0/1056354681 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fd80c3f10 con 0x7f2fd815bb70 2023-10-16T13:28:12.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.103+0000 7f2fcdffb700 1 -- 172.21.15.16:0/1056354681 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 107) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2fd0007470 con 0x7f2fd815bb70 2023-10-16T13:28:12.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.103+0000 7f2fcdffb700 1 --2- 172.21.15.16:0/1056354681 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f2fbc06b020 0x7f2fbc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:12.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.103+0000 7f2fdd15d700 1 --2- 172.21.15.16:0/1056354681 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f2fbc06b020 0x7f2fbc06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:12.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.103+0000 7f2fcdffb700 1 -- 172.21.15.16:0/1056354681 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 6051+0+0 (secure 0 0 0) 0x7f2fd00896a0 con 0x7f2fd815bb70 2023-10-16T13:28:12.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.104+0000 7f2fdd15d700 1 --2- 172.21.15.16:0/1056354681 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f2fbc06b020 0x7f2fbc06d4e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2fd40090b0 tx=0x7f2fd401c040 comp rx=0 tx=0).ready entity=mgr.10047 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:12.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.108+0000 7f2fcdffb700 1 -- 172.21.15.16:0/1056354681 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2fd80c3f10 con 0x7f2fd815bb70 2023-10-16T13:28:12.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.309+0000 7f2fdf3c1700 1 -- 172.21.15.16:0/1056354681 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2fd80c3f10 con 0x7f2fd815bb70 2023-10-16T13:28:12.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.310+0000 7f2fcdffb700 1 -- 172.21.15.16:0/1056354681 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v182) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2fd0018020 con 0x7f2fd815bb70 2023-10-16T13:28:12.313 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T13:28:12.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.312+0000 7f2fdf3c1700 1 -- 172.21.15.16:0/1056354681 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f2fbc06b020 msgr2=0x7f2fbc06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:12.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.312+0000 7f2fdf3c1700 1 --2- 172.21.15.16:0/1056354681 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f2fbc06b020 0x7f2fbc06d4e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2fd40090b0 tx=0x7f2fd401c040 comp rx=0 tx=0).stop 2023-10-16T13:28:12.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.312+0000 7f2fdf3c1700 1 -- 172.21.15.16:0/1056354681 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2fd815bb70 msgr2=0x7f2fd80c56d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:12.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.312+0000 7f2fdf3c1700 1 --2- 172.21.15.16:0/1056354681 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2fd815bb70 0x7f2fd80c56d0 secure :-1 s=READY pgs=2047 cs=0 l=1 rev1=1 crypto rx=0x7f2fd000cc70 tx=0x7f2fd000cf80 comp rx=0 tx=0).stop 2023-10-16T13:28:12.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.312+0000 7f2fdf3c1700 1 -- 172.21.15.16:0/1056354681 shutdown_connections 2023-10-16T13:28:12.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.312+0000 7f2fdf3c1700 1 --2- 172.21.15.16:0/1056354681 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2fd815bb70 0x7f2fd80c56d0 unknown :-1 s=CLOSED pgs=2047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:12.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.312+0000 7f2fdf3c1700 1 --2- 172.21.15.16:0/1056354681 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f2fbc06b020 0x7f2fbc06d4e0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:12.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.312+0000 7f2fdf3c1700 1 --2- 172.21.15.16:0/1056354681 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2fd806e070 0x7f2fd81548b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:12.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.312+0000 7f2fdf3c1700 1 --2- 172.21.15.16:0/1056354681 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2fd815b790 0x7f2fd80c81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:12.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.312+0000 7f2fdf3c1700 1 -- 172.21.15.16:0/1056354681 >> 172.21.15.16:0/1056354681 conn(0x7f2fd80cc5e0 msgr2=0x7f2fd8155aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:12.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.312+0000 7f2fdf3c1700 1 -- 172.21.15.16:0/1056354681 shutdown_connections 2023-10-16T13:28:12.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.312+0000 7f2fdf3c1700 1 -- 172.21.15.16:0/1056354681 wait complete. 2023-10-16T13:28:12.327 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart' 2023-10-16T13:28:12.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.450+0000 7fa1fbbe6700 1 Processor -- start 2023-10-16T13:28:12.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.450+0000 7fa1fbbe6700 1 -- start start 2023-10-16T13:28:12.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.451+0000 7fa1fbbe6700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa1f406a7e0 0x7fa1f406ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:12.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.451+0000 7fa1fbbe6700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1f4161c30 con 0x7fa1f415b280 2023-10-16T13:28:12.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.451+0000 7fa1fbbe6700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1f4161f10 con 0x7fa1f40cc8a0 2023-10-16T13:28:12.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.451+0000 7fa1fbbe6700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f406b190 con 0x7fa1f406a7e0 2023-10-16T13:28:12.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.451+0000 7fa1f9982700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa1f415b280 0x7fa1f41613e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38880/0 (socket says 172.21.15.16:38880) 2023-10-16T13:28:12.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.451+0000 7fa1f9982700 1 -- 172.21.15.16:0/4104726586 learned_addr learned my addr 172.21.15.16:0/4104726586 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:28:12.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.451+0000 7fa1f8980700 1 -- 172.21.15.16:0/4104726586 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 285784869 0 0) 0x7fa1f4161c30 con 0x7fa1f415b280 2023-10-16T13:28:12.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.452+0000 7fa1f8980700 1 -- 172.21.15.16:0/4104726586 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1e00035b0 con 0x7fa1f415b280 2023-10-16T13:28:12.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.452+0000 7fa1fa183700 1 --2- 172.21.15.16:0/4104726586 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa1f406a7e0 0x7fa1f406ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:12.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.452+0000 7fa1f8980700 1 -- 172.21.15.16:0/4104726586 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 157588444 0 0) 0x7fa1e00035b0 con 0x7fa1f415b280 2023-10-16T13:28:12.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.452+0000 7fa1f8980700 1 -- 172.21.15.16:0/4104726586 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa1f406a7e0 msgr2=0x7fa1f406ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:28:12.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.452+0000 7fa1f8980700 1 --2- 172.21.15.16:0/4104726586 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa1f406a7e0 0x7fa1f406ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:12.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.452+0000 7fa1f8980700 1 -- 172.21.15.16:0/4104726586 >> v1:172.21.15.49:6789/0 conn(0x7fa1f40cc8a0 legacy=0x7fa1f415e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:12.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.452+0000 7fa1f8980700 1 -- 172.21.15.16:0/4104726586 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1f41621f0 con 0x7fa1f415b280 2023-10-16T13:28:12.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.452+0000 7fa1f8980700 1 -- 172.21.15.16:0/4104726586 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7fa1e4001670 con 0x7fa1f415b280 2023-10-16T13:28:12.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.452+0000 7fa1f8980700 1 -- 172.21.15.16:0/4104726586 >> v1:172.21.15.16:6789/0 conn(0x7fa1f415b280 legacy=0x7fa1f41613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:12.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.452+0000 7fa1f8980700 1 --2- 172.21.15.16:0/4104726586 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1e0003a30 0x7fa1e0003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:12.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.452+0000 7fa1f8980700 1 --2- 172.21.15.16:0/4104726586 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa1f406a7e0 0x7fa1e0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:12.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.452+0000 7fa1f8980700 1 --2- 172.21.15.16:0/4104726586 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa1e0004d90 0x7fa1e0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:12.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.452+0000 7fa1f8980700 1 -- 172.21.15.16:0/4104726586 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f406b190 con 0x7fa1e0003a30 2023-10-16T13:28:12.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.452+0000 7fa1f8980700 1 -- 172.21.15.16:0/4104726586 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e000bac0 con 0x7fa1e0004d90 2023-10-16T13:28:12.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.452+0000 7fa1f8980700 1 -- 172.21.15.16:0/4104726586 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e000bc80 con 0x7fa1f406a7e0 2023-10-16T13:28:12.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.452+0000 7fa1f9982700 1 --2- 172.21.15.16:0/4104726586 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1e0003a30 0x7fa1e0003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:12.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.452+0000 7fa1f9181700 1 --2- 172.21.15.16:0/4104726586 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa1f406a7e0 0x7fa1e0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:12.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.452+0000 7fa1fa183700 1 --2- 172.21.15.16:0/4104726586 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa1e0004d90 0x7fa1e0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:12.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.452+0000 7fa1f9982700 1 -- 172.21.15.16:0/4104726586 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa1f406a7e0 msgr2=0x7fa1e0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:28:12.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.452+0000 7fa1f9982700 1 --2- 172.21.15.16:0/4104726586 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa1f406a7e0 0x7fa1e0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:12.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.452+0000 7fa1f9982700 1 -- 172.21.15.16:0/4104726586 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa1e0004d90 msgr2=0x7fa1e0005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:28:12.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.452+0000 7fa1f9982700 1 --2- 172.21.15.16:0/4104726586 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa1e0004d90 0x7fa1e0005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:12.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.452+0000 7fa1f9982700 1 -- 172.21.15.16:0/4104726586 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1e00035b0 con 0x7fa1e0003a30 2023-10-16T13:28:12.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.452+0000 7fa1f9982700 1 --2- 172.21.15.16:0/4104726586 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1e0003a30 0x7fa1e0003e50 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 crypto rx=0x7fa1e400c210 tx=0x7fa1e0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ee80bf44d324597 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:12.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.453+0000 7fa1f8980700 1 -- 172.21.15.16:0/4104726586 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e4007c10 con 0x7fa1e0003a30 2023-10-16T13:28:12.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.453+0000 7fa1f8980700 1 -- 172.21.15.16:0/4104726586 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1e400cdd0 con 0x7fa1e0003a30 2023-10-16T13:28:12.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.453+0000 7fa1f8980700 1 -- 172.21.15.16:0/4104726586 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e4005ba0 con 0x7fa1e0003a30 2023-10-16T13:28:12.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.453+0000 7fa1fbbe6700 1 -- 172.21.15.16:0/4104726586 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1e0003a30 msgr2=0x7fa1e0003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:12.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.453+0000 7fa1fbbe6700 1 --2- 172.21.15.16:0/4104726586 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1e0003a30 0x7fa1e0003e50 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 crypto rx=0x7fa1e400c210 tx=0x7fa1e0005880 comp rx=0 tx=0).stop 2023-10-16T13:28:12.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.453+0000 7fa1fa183700 1 -- 172.21.15.16:0/4104726586 reap_dead start 2023-10-16T13:28:12.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.453+0000 7fa1fbbe6700 1 -- 172.21.15.16:0/4104726586 shutdown_connections 2023-10-16T13:28:12.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.453+0000 7fa1fbbe6700 1 -- 172.21.15.16:0/4104726586 >> 172.21.15.16:0/4104726586 conn(0x7fa1f40b1fd0 msgr2=0x7fa1f40b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:12.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.453+0000 7fa1fbbe6700 1 -- 172.21.15.16:0/4104726586 shutdown_connections 2023-10-16T13:28:12.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.453+0000 7fa1fbbe6700 1 -- 172.21.15.16:0/4104726586 wait complete. 2023-10-16T13:28:12.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.454+0000 7fa1fbbe6700 1 Processor -- start 2023-10-16T13:28:12.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.455+0000 7fa1fbbe6700 1 -- start start 2023-10-16T13:28:12.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.455+0000 7fa1fbbe6700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa1e0003a30 0x7fa1f40c7f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:12.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.455+0000 7fa1fbbe6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1f4160760 0x7fa1f40aee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:12.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.455+0000 7fa1fbbe6700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa1f415f650 0x7fa1f415fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:12.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.455+0000 7fa1fbbe6700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f4160010 con 0x7fa1f4160760 2023-10-16T13:28:12.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.455+0000 7fa1fbbe6700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f414d0b0 con 0x7fa1f415f650 2023-10-16T13:28:12.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.455+0000 7fa1fbbe6700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f414c4a0 con 0x7fa1e0003a30 2023-10-16T13:28:12.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.455+0000 7fa1f9181700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1f4160760 0x7fa1f40aee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:12.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.455+0000 7fa1f9181700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1f4160760 0x7fa1f40aee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49618/0 (socket says 172.21.15.16:49618) 2023-10-16T13:28:12.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.455+0000 7fa1f9181700 1 -- 172.21.15.16:0/2631505463 learned_addr learned my addr 172.21.15.16:0/2631505463 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:12.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.456+0000 7fa1f9982700 1 --2- 172.21.15.16:0/2631505463 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa1e0003a30 0x7fa1f40c7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:12.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.456+0000 7fa1fa183700 1 --2- 172.21.15.16:0/2631505463 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa1f415f650 0x7fa1f415fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:12.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.456+0000 7fa1f9181700 1 -- 172.21.15.16:0/2631505463 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa1e0003a30 msgr2=0x7fa1f40c7f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:12.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.456+0000 7fa1f9181700 1 --2- 172.21.15.16:0/2631505463 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa1e0003a30 0x7fa1f40c7f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:12.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.456+0000 7fa1f9181700 1 -- 172.21.15.16:0/2631505463 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa1f415f650 msgr2=0x7fa1f415fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:12.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.456+0000 7fa1f9181700 1 --2- 172.21.15.16:0/2631505463 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa1f415f650 0x7fa1f415fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:12.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.456+0000 7fa1f9181700 1 -- 172.21.15.16:0/2631505463 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1f414c780 con 0x7fa1f4160760 2023-10-16T13:28:12.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.456+0000 7fa1f9181700 1 --2- 172.21.15.16:0/2631505463 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1f4160760 0x7fa1f40aee20 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 crypto rx=0x7fa1f000e540 tx=0x7fa1f000e900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:12.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.456+0000 7fa1ea7fc700 1 -- 172.21.15.16:0/2631505463 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f00052e0 con 0x7fa1f4160760 2023-10-16T13:28:12.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.456+0000 7fa1ea7fc700 1 -- 172.21.15.16:0/2631505463 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1f0005440 con 0x7fa1f4160760 2023-10-16T13:28:12.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.456+0000 7fa1fbbe6700 1 -- 172.21.15.16:0/2631505463 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1f414ca60 con 0x7fa1f4160760 2023-10-16T13:28:12.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.456+0000 7fa1ea7fc700 1 -- 172.21.15.16:0/2631505463 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f00038a0 con 0x7fa1f4160760 2023-10-16T13:28:12.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.456+0000 7fa1fbbe6700 1 -- 172.21.15.16:0/2631505463 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1f4160be0 con 0x7fa1f4160760 2023-10-16T13:28:12.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.457+0000 7fa1fbbe6700 1 -- 172.21.15.16:0/2631505463 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1f404ebb0 con 0x7fa1f4160760 2023-10-16T13:28:12.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.458+0000 7fa1ea7fc700 1 -- 172.21.15.16:0/2631505463 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 107) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa1f0003a00 con 0x7fa1f4160760 2023-10-16T13:28:12.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.458+0000 7fa1ea7fc700 1 --2- 172.21.15.16:0/2631505463 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7fa1d806b020 0x7fa1d806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:12.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.458+0000 7fa1ea7fc700 1 -- 172.21.15.16:0/2631505463 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 6051+0+0 (secure 0 0 0) 0x7fa1f0088f50 con 0x7fa1f4160760 2023-10-16T13:28:12.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.458+0000 7fa1f9982700 1 --2- 172.21.15.16:0/2631505463 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7fa1d806b020 0x7fa1d806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:12.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.458+0000 7fa1f9982700 1 --2- 172.21.15.16:0/2631505463 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7fa1d806b020 0x7fa1d806d4e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa1e4015040 tx=0x7fa1e4007000 comp rx=0 tx=0).ready entity=mgr.10047 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:12.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.463+0000 7fa1ea7fc700 1 -- 172.21.15.16:0/2631505463 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa1f0053ed0 con 0x7fa1f4160760 2023-10-16T13:28:12.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:12.752+0000 7fa1fbbe6700 1 -- 172.21.15.16:0/2631505463 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]} v 0) v1 -- 0x7fa1f4071a80 con 0x7fa1f4160760 2023-10-16T13:28:13.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.595+0000 7fa1ea7fc700 1 -- 172.21.15.16:0/2631505463 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]}]=0 v2538) v1 ==== 168+0+0 (secure 0 0 0) 0x7fa1f00190e0 con 0x7fa1f4160760 2023-10-16T13:28:13.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.598+0000 7fa1fbbe6700 1 -- 172.21.15.16:0/2631505463 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7fa1d806b020 msgr2=0x7fa1d806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:13.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.598+0000 7fa1fbbe6700 1 --2- 172.21.15.16:0/2631505463 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7fa1d806b020 0x7fa1d806d4e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa1e4015040 tx=0x7fa1e4007000 comp rx=0 tx=0).stop 2023-10-16T13:28:13.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.598+0000 7fa1fbbe6700 1 -- 172.21.15.16:0/2631505463 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1f4160760 msgr2=0x7fa1f40aee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:13.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.598+0000 7fa1fbbe6700 1 --2- 172.21.15.16:0/2631505463 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1f4160760 0x7fa1f40aee20 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 crypto rx=0x7fa1f000e540 tx=0x7fa1f000e900 comp rx=0 tx=0).stop 2023-10-16T13:28:13.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.598+0000 7fa1fbbe6700 1 -- 172.21.15.16:0/2631505463 shutdown_connections 2023-10-16T13:28:13.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.598+0000 7fa1fbbe6700 1 --2- 172.21.15.16:0/2631505463 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7fa1d806b020 0x7fa1d806d4e0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:13.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.598+0000 7fa1fbbe6700 1 --2- 172.21.15.16:0/2631505463 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa1f415f650 0x7fa1f415fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:13.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.598+0000 7fa1fbbe6700 1 --2- 172.21.15.16:0/2631505463 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa1e0003a30 0x7fa1f40c7f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:13.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.598+0000 7fa1fbbe6700 1 --2- 172.21.15.16:0/2631505463 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1f4160760 0x7fa1f40aee20 unknown :-1 s=CLOSED pgs=2050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:13.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.598+0000 7fa1fbbe6700 1 -- 172.21.15.16:0/2631505463 >> 172.21.15.16:0/2631505463 conn(0x7fa1f40b1fd0 msgr2=0x7fa1f4072120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:13.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.598+0000 7fa1fbbe6700 1 -- 172.21.15.16:0/2631505463 shutdown_connections 2023-10-16T13:28:13.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.598+0000 7fa1fbbe6700 1 -- 172.21.15.16:0/2631505463 wait complete. 2023-10-16T13:28:13.616 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-16T13:28:13.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.748+0000 7f539de67700 1 Processor -- start 2023-10-16T13:28:13.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.748+0000 7f539de67700 1 -- start start 2023-10-16T13:28:13.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.749+0000 7f539de67700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f539815ec80 0x7f53980cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:13.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.749+0000 7f539de67700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f53980d2110 0x7f53980cd1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:13.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.749+0000 7f539de67700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53980b2440 con 0x7f539815f060 2023-10-16T13:28:13.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.749+0000 7f539de67700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53980b2c50 con 0x7f539815ec80 2023-10-16T13:28:13.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.749+0000 7f539de67700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53980d00e0 con 0x7f53980d2110 2023-10-16T13:28:13.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.749+0000 7f53977fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f539815ec80 0x7f53980cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:13.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.749+0000 7f53977fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f539815ec80 0x7f53980cdec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49626/0 (socket says 172.21.15.16:49626) 2023-10-16T13:28:13.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.749+0000 7f53977fe700 1 -- 172.21.15.16:0/9540170 learned_addr learned my addr 172.21.15.16:0/9540170 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:13.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.749+0000 7f5397fff700 1 --2- 172.21.15.16:0/9540170 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f53980d2110 0x7f53980cd1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:13.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.749+0000 7f53977fe700 1 -- 172.21.15.16:0/9540170 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f53980d2110 msgr2=0x7f53980cd1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:28:13.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.750+0000 7f53977fe700 1 --2- 172.21.15.16:0/9540170 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f53980d2110 0x7f53980cd1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:13.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.750+0000 7f53977fe700 1 -- 172.21.15.16:0/9540170 >> v1:172.21.15.49:6789/0 conn(0x7f539815f060 legacy=0x7f53980b4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:28:13.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.750+0000 7f53977fe700 1 -- 172.21.15.16:0/9540170 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53980cef60 con 0x7f539815ec80 2023-10-16T13:28:13.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.750+0000 7f53977fe700 1 --2- 172.21.15.16:0/9540170 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f539815ec80 0x7f53980cdec0 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7f53800098f0 tx=0x7f5380009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcb433c8c29aa873 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:13.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.750+0000 7f53967fc700 1 -- 172.21.15.16:0/9540170 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f538000b8a0 con 0x7f539815ec80 2023-10-16T13:28:13.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.750+0000 7f53967fc700 1 -- 172.21.15.16:0/9540170 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f538000ba00 con 0x7f539815ec80 2023-10-16T13:28:13.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.750+0000 7f53967fc700 1 -- 172.21.15.16:0/9540170 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f538000bd90 con 0x7f539815ec80 2023-10-16T13:28:13.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.750+0000 7f539de67700 1 -- 172.21.15.16:0/9540170 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f539815ec80 msgr2=0x7f53980cdec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:13.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.750+0000 7f539de67700 1 --2- 172.21.15.16:0/9540170 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f539815ec80 0x7f53980cdec0 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7f53800098f0 tx=0x7f5380009c00 comp rx=0 tx=0).stop 2023-10-16T13:28:13.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.751+0000 7f539de67700 1 -- 172.21.15.16:0/9540170 shutdown_connections 2023-10-16T13:28:13.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.751+0000 7f539de67700 1 --2- 172.21.15.16:0/9540170 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f53980d2110 0x7f53980cd1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:13.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.751+0000 7f539de67700 1 --2- 172.21.15.16:0/9540170 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f539815ec80 0x7f53980cdec0 unknown :-1 s=CLOSED pgs=2051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:13.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.751+0000 7f539de67700 1 -- 172.21.15.16:0/9540170 >> 172.21.15.16:0/9540170 conn(0x7f53980d15f0 msgr2=0x7f53980d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:13.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.751+0000 7f539de67700 1 -- 172.21.15.16:0/9540170 shutdown_connections 2023-10-16T13:28:13.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.751+0000 7f539de67700 1 -- 172.21.15.16:0/9540170 wait complete. 2023-10-16T13:28:13.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.752+0000 7f539de67700 1 Processor -- start 2023-10-16T13:28:13.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.752+0000 7f539de67700 1 -- start start 2023-10-16T13:28:13.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.753+0000 7f539de67700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53980d2110 0x7f53980c88f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:13.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.753+0000 7f539de67700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f539815ec80 0x7f53981f3d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:13.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.753+0000 7f539de67700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f539815f060 0x7f539815ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:13.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.753+0000 7f539de67700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5398157250 con 0x7f53980d2110 2023-10-16T13:28:13.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.753+0000 7f539de67700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5398159ef0 con 0x7f539815ec80 2023-10-16T13:28:13.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.753+0000 7f539de67700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5398156f70 con 0x7f539815f060 2023-10-16T13:28:13.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.753+0000 7f53977fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53980d2110 0x7f53980c88f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:13.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.753+0000 7f53977fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53980d2110 0x7f53980c88f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49632/0 (socket says 172.21.15.16:49632) 2023-10-16T13:28:13.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.753+0000 7f53977fe700 1 -- 172.21.15.16:0/3110885142 learned_addr learned my addr 172.21.15.16:0/3110885142 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:13.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.753+0000 7f5396ffd700 1 --2- 172.21.15.16:0/3110885142 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f539815ec80 0x7f53981f3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:13.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.753+0000 7f5397fff700 1 --2- 172.21.15.16:0/3110885142 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f539815f060 0x7f539815ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:13.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.753+0000 7f53977fe700 1 -- 172.21.15.16:0/3110885142 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f539815f060 msgr2=0x7f539815ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:13.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.753+0000 7f53977fe700 1 --2- 172.21.15.16:0/3110885142 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f539815f060 0x7f539815ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:13.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.753+0000 7f53977fe700 1 -- 172.21.15.16:0/3110885142 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f539815ec80 msgr2=0x7f53981f3d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:13.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.753+0000 7f53977fe700 1 --2- 172.21.15.16:0/3110885142 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f539815ec80 0x7f53981f3d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:13.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.753+0000 7f53977fe700 1 -- 172.21.15.16:0/3110885142 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5380009550 con 0x7f53980d2110 2023-10-16T13:28:13.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.753+0000 7f53977fe700 1 --2- 172.21.15.16:0/3110885142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53980d2110 0x7f53980c88f0 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 crypto rx=0x7f5380009c40 tx=0x7f5380004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:13.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.753+0000 7f538ffff700 1 -- 172.21.15.16:0/3110885142 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53800044d0 con 0x7f53980d2110 2023-10-16T13:28:13.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.754+0000 7f538ffff700 1 -- 172.21.15.16:0/3110885142 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f538001f070 con 0x7f53980d2110 2023-10-16T13:28:13.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.754+0000 7f539de67700 1 -- 172.21.15.16:0/3110885142 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5398159930 con 0x7f53980d2110 2023-10-16T13:28:13.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.754+0000 7f538ffff700 1 -- 172.21.15.16:0/3110885142 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53800038e0 con 0x7f53980d2110 2023-10-16T13:28:13.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.754+0000 7f539de67700 1 -- 172.21.15.16:0/3110885142 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5398156b90 con 0x7f53980d2110 2023-10-16T13:28:13.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.754+0000 7f539de67700 1 -- 172.21.15.16:0/3110885142 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f539804ebb0 con 0x7f53980d2110 2023-10-16T13:28:13.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.755+0000 7f538ffff700 1 -- 172.21.15.16:0/3110885142 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 107) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5380003a40 con 0x7f53980d2110 2023-10-16T13:28:13.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.755+0000 7f538ffff700 1 --2- 172.21.15.16:0/3110885142 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f537806b020 0x7f537806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:13.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.755+0000 7f538ffff700 1 -- 172.21.15.16:0/3110885142 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 6051+0+0 (secure 0 0 0) 0x7f5380021070 con 0x7f53980d2110 2023-10-16T13:28:13.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.755+0000 7f5396ffd700 1 --2- 172.21.15.16:0/3110885142 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f537806b020 0x7f537806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:13.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.756+0000 7f5396ffd700 1 --2- 172.21.15.16:0/3110885142 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f537806b020 0x7f537806d4e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f53980cb2b0 tx=0x7f5388009040 comp rx=0 tx=0).ready entity=mgr.10047 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:13.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.760+0000 7f538ffff700 1 -- 172.21.15.16:0/3110885142 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5380055ee0 con 0x7f53980d2110 2023-10-16T13:28:13.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.963+0000 7f539de67700 1 -- 172.21.15.16:0/3110885142 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f53980cfd40 con 0x7f53980d2110 2023-10-16T13:28:13.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.966+0000 7f538ffff700 1 -- 172.21.15.16:0/3110885142 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v64)=0 v64) v1 ==== 122+0+0 (secure 0 0 0) 0x7f5380015020 con 0x7f53980d2110 2023-10-16T13:28:13.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.969+0000 7f539de67700 1 -- 172.21.15.16:0/3110885142 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f537806b020 msgr2=0x7f537806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:13.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.969+0000 7f539de67700 1 --2- 172.21.15.16:0/3110885142 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f537806b020 0x7f537806d4e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f53980cb2b0 tx=0x7f5388009040 comp rx=0 tx=0).stop 2023-10-16T13:28:13.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.969+0000 7f539de67700 1 -- 172.21.15.16:0/3110885142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53980d2110 msgr2=0x7f53980c88f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:13.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.969+0000 7f539de67700 1 --2- 172.21.15.16:0/3110885142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53980d2110 0x7f53980c88f0 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 crypto rx=0x7f5380009c40 tx=0x7f5380004750 comp rx=0 tx=0).stop 2023-10-16T13:28:13.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.969+0000 7f539de67700 1 -- 172.21.15.16:0/3110885142 shutdown_connections 2023-10-16T13:28:13.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.969+0000 7f539de67700 1 --2- 172.21.15.16:0/3110885142 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f539815f060 0x7f539815ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:13.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.969+0000 7f539de67700 1 --2- 172.21.15.16:0/3110885142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53980d2110 0x7f53980c88f0 unknown :-1 s=CLOSED pgs=2052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:13.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.969+0000 7f539de67700 1 --2- 172.21.15.16:0/3110885142 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f537806b020 0x7f537806d4e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:13.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.969+0000 7f539de67700 1 --2- 172.21.15.16:0/3110885142 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f539815ec80 0x7f53981f3d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:13.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.969+0000 7f539de67700 1 -- 172.21.15.16:0/3110885142 >> 172.21.15.16:0/3110885142 conn(0x7f53980d15f0 msgr2=0x7f53980d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:13.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.969+0000 7f539de67700 1 -- 172.21.15.16:0/3110885142 shutdown_connections 2023-10-16T13:28:13.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:13.969+0000 7f539de67700 1 -- 172.21.15.16:0/3110885142 wait complete. 2023-10-16T13:28:13.984 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T13:28:13.984 DEBUG:teuthology.orchestra.run.smithi016:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T13:28:14.030 DEBUG:teuthology.orchestra.run.smithi049:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T13:28:14.080 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T13:28:14.081 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T13:28:14.081 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:28:14.100 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16287364k,nr_inodes=4071841,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20251 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3261172k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T13:28:14.101 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T13:28:14.102 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:28:14.122 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284204k,nr_inodes=4071051,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=753 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260540k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T13:28:14.122 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T13:28:14.123 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:28:14.157 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16287364k,nr_inodes=4071841,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20251 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3261172k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T13:28:14.158 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T13:28:14.158 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:28:14.182 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284204k,nr_inodes=4071051,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=753 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260540k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T13:28:14.183 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-16T13:28:14.183 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:28:14.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.320+0000 7fcb39dbe700 1 Processor -- start 2023-10-16T13:28:14.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.321+0000 7fcb39dbe700 1 -- start start 2023-10-16T13:28:14.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.321+0000 7fcb39dbe700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fcb340d41c0 0x7fcb340d0040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:14.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.321+0000 7fcb39dbe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcb340d0580 0x7fcb3406c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:14.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.321+0000 7fcb39dbe700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb3406ba00 con 0x7fcb3415d8d0 2023-10-16T13:28:14.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.321+0000 7fcb39dbe700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb340d0a10 con 0x7fcb340d0580 2023-10-16T13:28:14.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.321+0000 7fcb39dbe700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb340d26d0 con 0x7fcb340d41c0 2023-10-16T13:28:14.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.321+0000 7fcb33fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcb340d0580 0x7fcb3406c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:14.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.322+0000 7fcb33fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcb340d0580 0x7fcb3406c4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49644/0 (socket says 172.21.15.16:49644) 2023-10-16T13:28:14.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.322+0000 7fcb33fff700 1 -- 172.21.15.16:0/2856489064 learned_addr learned my addr 172.21.15.16:0/2856489064 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:14.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.322+0000 7fcb32ffd700 1 --2- 172.21.15.16:0/2856489064 >> v2:172.21.15.49:3300/0 conn(0x7fcb340d41c0 0x7fcb340d0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:14.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.322+0000 7fcb33fff700 1 -- 172.21.15.16:0/2856489064 >> v2:172.21.15.49:3300/0 conn(0x7fcb340d41c0 msgr2=0x7fcb340d0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:28:14.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.322+0000 7fcb33fff700 1 --2- 172.21.15.16:0/2856489064 >> v2:172.21.15.49:3300/0 conn(0x7fcb340d41c0 0x7fcb340d0040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:14.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.322+0000 7fcb33fff700 1 -- 172.21.15.16:0/2856489064 >> v1:172.21.15.49:6789/0 conn(0x7fcb3415d8d0 legacy=0x7fcb340d1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:28:14.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.322+0000 7fcb33fff700 1 -- 172.21.15.16:0/2856489064 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb340d23f0 con 0x7fcb340d0580 2023-10-16T13:28:14.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.322+0000 7fcb32ffd700 1 --2- 172.21.15.16:0/2856489064 >> v2:172.21.15.49:3300/0 conn(0x7fcb340d41c0 0x7fcb340d0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:28:14.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.322+0000 7fcb33fff700 1 --2- 172.21.15.16:0/2856489064 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb340d0580 0x7fcb3406c4a0 secure :-1 s=READY pgs=2053 cs=0 l=1 rev1=1 crypto rx=0x7fcb3408b580 tx=0x7fcb2c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f6095fa1eac02c1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:14.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.322+0000 7fcb327fc700 1 -- 172.21.15.16:0/2856489064 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb2c00e070 con 0x7fcb340d0580 2023-10-16T13:28:14.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.322+0000 7fcb327fc700 1 -- 172.21.15.16:0/2856489064 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb2c003a20 con 0x7fcb340d0580 2023-10-16T13:28:14.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.322+0000 7fcb327fc700 1 -- 172.21.15.16:0/2856489064 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb2c003dd0 con 0x7fcb340d0580 2023-10-16T13:28:14.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.323+0000 7fcb39dbe700 1 -- 172.21.15.16:0/2856489064 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb340d0580 msgr2=0x7fcb3406c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:14.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.323+0000 7fcb39dbe700 1 --2- 172.21.15.16:0/2856489064 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb340d0580 0x7fcb3406c4a0 secure :-1 s=READY pgs=2053 cs=0 l=1 rev1=1 crypto rx=0x7fcb3408b580 tx=0x7fcb2c00b330 comp rx=0 tx=0).stop 2023-10-16T13:28:14.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.323+0000 7fcb39dbe700 1 -- 172.21.15.16:0/2856489064 shutdown_connections 2023-10-16T13:28:14.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.323+0000 7fcb39dbe700 1 --2- 172.21.15.16:0/2856489064 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb340d0580 0x7fcb3406c4a0 unknown :-1 s=CLOSED pgs=2053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:14.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.323+0000 7fcb39dbe700 1 --2- 172.21.15.16:0/2856489064 >> v2:172.21.15.49:3300/0 conn(0x7fcb340d41c0 0x7fcb340d0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:14.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.323+0000 7fcb39dbe700 1 -- 172.21.15.16:0/2856489064 >> 172.21.15.16:0/2856489064 conn(0x7fcb340d3630 msgr2=0x7fcb340d3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:14.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.323+0000 7fcb39dbe700 1 -- 172.21.15.16:0/2856489064 shutdown_connections 2023-10-16T13:28:14.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.323+0000 7fcb39dbe700 1 -- 172.21.15.16:0/2856489064 wait complete. 2023-10-16T13:28:14.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.324+0000 7fcb39dbe700 1 Processor -- start 2023-10-16T13:28:14.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.324+0000 7fcb39dbe700 1 -- start start 2023-10-16T13:28:14.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.325+0000 7fcb39dbe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb340d0580 0x7fcb340cd420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:14.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.325+0000 7fcb39dbe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcb340d41c0 0x7fcb3414a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:14.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.325+0000 7fcb39dbe700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcb3415d8d0 0x7fcb340c8be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:14.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.325+0000 7fcb39dbe700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb3414ae30 con 0x7fcb340d0580 2023-10-16T13:28:14.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.325+0000 7fcb39dbe700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb340c9120 con 0x7fcb340d41c0 2023-10-16T13:28:14.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.325+0000 7fcb39dbe700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb340c93a0 con 0x7fcb3415d8d0 2023-10-16T13:28:14.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.325+0000 7fcb337fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb340d0580 0x7fcb340cd420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:14.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.325+0000 7fcb337fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb340d0580 0x7fcb340cd420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49646/0 (socket says 172.21.15.16:49646) 2023-10-16T13:28:14.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.325+0000 7fcb337fe700 1 -- 172.21.15.16:0/512497723 learned_addr learned my addr 172.21.15.16:0/512497723 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:14.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.325+0000 7fcb32ffd700 1 --2- 172.21.15.16:0/512497723 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcb340d41c0 0x7fcb3414a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:14.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.325+0000 7fcb33fff700 1 --2- 172.21.15.16:0/512497723 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcb3415d8d0 0x7fcb340c8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:14.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.325+0000 7fcb337fe700 1 -- 172.21.15.16:0/512497723 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcb3415d8d0 msgr2=0x7fcb340c8be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:14.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.325+0000 7fcb337fe700 1 --2- 172.21.15.16:0/512497723 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcb3415d8d0 0x7fcb340c8be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:14.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.325+0000 7fcb337fe700 1 -- 172.21.15.16:0/512497723 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcb340d41c0 msgr2=0x7fcb3414a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:14.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.325+0000 7fcb337fe700 1 --2- 172.21.15.16:0/512497723 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcb340d41c0 0x7fcb3414a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:14.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.325+0000 7fcb337fe700 1 -- 172.21.15.16:0/512497723 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb2c009b30 con 0x7fcb340d0580 2023-10-16T13:28:14.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.325+0000 7fcb33fff700 1 --2- 172.21.15.16:0/512497723 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcb3415d8d0 0x7fcb340c8be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:28:14.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.325+0000 7fcb32ffd700 1 --2- 172.21.15.16:0/512497723 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcb340d41c0 0x7fcb3414a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:28:14.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.326+0000 7fcb337fe700 1 --2- 172.21.15.16:0/512497723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb340d0580 0x7fcb340cd420 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x7fcb2400b4a0 tx=0x7fcb2400b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:14.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.326+0000 7fcb1bfff700 1 -- 172.21.15.16:0/512497723 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb24011830 con 0x7fcb340d0580 2023-10-16T13:28:14.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.326+0000 7fcb1bfff700 1 -- 172.21.15.16:0/512497723 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb24011990 con 0x7fcb340d0580 2023-10-16T13:28:14.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.326+0000 7fcb39dbe700 1 -- 172.21.15.16:0/512497723 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb340c9680 con 0x7fcb340d0580 2023-10-16T13:28:14.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.326+0000 7fcb1bfff700 1 -- 172.21.15.16:0/512497723 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb24011d40 con 0x7fcb340d0580 2023-10-16T13:28:14.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.326+0000 7fcb39dbe700 1 -- 172.21.15.16:0/512497723 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb340c9c40 con 0x7fcb340d0580 2023-10-16T13:28:14.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.327+0000 7fcb39dbe700 1 -- 172.21.15.16:0/512497723 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb3404ebb0 con 0x7fcb340d0580 2023-10-16T13:28:14.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.327+0000 7fcb1bfff700 1 -- 172.21.15.16:0/512497723 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 107) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fcb2401a400 con 0x7fcb340d0580 2023-10-16T13:28:14.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.328+0000 7fcb1bfff700 1 --2- 172.21.15.16:0/512497723 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7fcb2006b020 0x7fcb2006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:14.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.328+0000 7fcb1bfff700 1 -- 172.21.15.16:0/512497723 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 6051+0+0 (secure 0 0 0) 0x7fcb24089600 con 0x7fcb340d0580 2023-10-16T13:28:14.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.328+0000 7fcb32ffd700 1 --2- 172.21.15.16:0/512497723 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7fcb2006b020 0x7fcb2006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:14.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.328+0000 7fcb32ffd700 1 --2- 172.21.15.16:0/512497723 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7fcb2006b020 0x7fcb2006d4e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fcb340cad40 tx=0x7fcb28008040 comp rx=0 tx=0).ready entity=mgr.10047 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:14.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.332+0000 7fcb1bfff700 1 -- 172.21.15.16:0/512497723 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcb24054440 con 0x7fcb340d0580 2023-10-16T13:28:14.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.580+0000 7fcb39dbe700 1 -- 172.21.15.16:0/512497723 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcb340b1ec0 con 0x7fcb340d0580 2023-10-16T13:28:14.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.581+0000 7fcb1bfff700 1 -- 172.21.15.16:0/512497723 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3700 (secure 0 0 0) 0x7fcb24015e90 con 0x7fcb340d0580 2023-10-16T13:28:14.584 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:28:14.584 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":88,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7964,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/4012137577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":4012137577},{"type":"v1","addr":"172.21.15.16:6837","nonce":4012137577}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81},{"gid":7970,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/3742568047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":3742568047},{"type":"v1","addr":"172.21.15.49:6835","nonce":3742568047}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81},{"gid":9756,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/3838743980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":3838743980},{"type":"v1","addr":"172.21.15.16:6836","nonce":3838743980}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81},{"gid":10197,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/2221587450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":2221587450},{"type":"v1","addr":"172.21.15.49:6834","nonce":2221587450}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":88}],"filesystems":[]} 2023-10-16T13:28:14.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.583+0000 7fcb39dbe700 1 -- 172.21.15.16:0/512497723 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7fcb2006b020 msgr2=0x7fcb2006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:14.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.583+0000 7fcb39dbe700 1 --2- 172.21.15.16:0/512497723 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7fcb2006b020 0x7fcb2006d4e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fcb340cad40 tx=0x7fcb28008040 comp rx=0 tx=0).stop 2023-10-16T13:28:14.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.583+0000 7fcb39dbe700 1 -- 172.21.15.16:0/512497723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb340d0580 msgr2=0x7fcb340cd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:14.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.583+0000 7fcb39dbe700 1 --2- 172.21.15.16:0/512497723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb340d0580 0x7fcb340cd420 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x7fcb2400b4a0 tx=0x7fcb2400b860 comp rx=0 tx=0).stop 2023-10-16T13:28:14.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.583+0000 7fcb39dbe700 1 -- 172.21.15.16:0/512497723 shutdown_connections 2023-10-16T13:28:14.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.583+0000 7fcb39dbe700 1 --2- 172.21.15.16:0/512497723 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcb3415d8d0 0x7fcb340c8be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:14.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.583+0000 7fcb39dbe700 1 --2- 172.21.15.16:0/512497723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb340d0580 0x7fcb340cd420 unknown :-1 s=CLOSED pgs=2054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:14.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.583+0000 7fcb39dbe700 1 --2- 172.21.15.16:0/512497723 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7fcb2006b020 0x7fcb2006d4e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:14.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.583+0000 7fcb39dbe700 1 --2- 172.21.15.16:0/512497723 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcb340d41c0 0x7fcb3414a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:14.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.583+0000 7fcb39dbe700 1 -- 172.21.15.16:0/512497723 >> 172.21.15.16:0/512497723 conn(0x7fcb340d3630 msgr2=0x7fcb3406d200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:14.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.583+0000 7fcb39dbe700 1 -- 172.21.15.16:0/512497723 shutdown_connections 2023-10-16T13:28:14.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.583+0000 7fcb39dbe700 1 -- 172.21.15.16:0/512497723 wait complete. 2023-10-16T13:28:14.588 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 88 2023-10-16T13:28:14.598 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-16T13:28:14.598 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-16T13:28:14.599 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-16T13:28:14.599 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T13:28:14.599 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-16T13:28:14.600 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-16T13:28:14.600 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-16T13:28:14.600 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T13:28:14.601 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-16T13:28:14.601 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-16T13:28:14.601 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-16T13:28:14.602 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T13:28:14.602 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-16T13:28:14.602 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-16T13:28:14.602 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-16T13:28:14.603 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T13:28:14.608 INFO:tasks.ceph.mds.a:Stopped 2023-10-16T13:28:14.608 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-16T13:28:14.610 INFO:tasks.ceph.mds.c:Stopped 2023-10-16T13:28:14.610 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-16T13:28:14.613 INFO:tasks.ceph.mds.b:Stopped 2023-10-16T13:28:14.613 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-16T13:28:14.614 INFO:tasks.ceph.mds.d:Stopped 2023-10-16T13:28:14.614 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-16T13:28:14.652 INFO:tasks.ceph.mds.a:Started 2023-10-16T13:28:14.652 DEBUG:teuthology.parallel:result is None 2023-10-16T13:28:14.653 INFO:tasks.ceph.mds.c:Started 2023-10-16T13:28:14.654 DEBUG:teuthology.parallel:result is None 2023-10-16T13:28:14.655 INFO:tasks.ceph.mds.b:Started 2023-10-16T13:28:14.655 DEBUG:teuthology.parallel:result is None 2023-10-16T13:28:14.656 INFO:tasks.ceph.mds.d:Started 2023-10-16T13:28:14.656 DEBUG:teuthology.parallel:result is None 2023-10-16T13:28:14.657 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-16T13:28:14.733 INFO:tasks.ceph.mds.c.smithi016.stdout:starting mds.c at 2023-10-16T13:28:14.734 INFO:tasks.ceph.mds.a.smithi016.stdout:starting mds.a at 2023-10-16T13:28:14.746 INFO:tasks.ceph.mds.d.smithi049.stdout:starting mds.d at 2023-10-16T13:28:14.747 INFO:tasks.ceph.mds.b.smithi049.stdout:starting mds.b at 2023-10-16T13:28:14.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.786+0000 7faeec15c700 1 Processor -- start 2023-10-16T13:28:14.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.787+0000 7faeec15c700 1 -- start start 2023-10-16T13:28:14.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.788+0000 7faeec15c700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7faee415a5c0 0x7faee40cef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:14.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.788+0000 7faeec15c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faee40ce010 0x7faee40d03d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:14.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.788+0000 7faeec15c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7faee40d0910 0x7faee415e210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:14.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.788+0000 7faeec15c700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faee40cf440 con 0x7faee40d0910 2023-10-16T13:28:14.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.788+0000 7faeec15c700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faee40d0da0 con 0x7faee415a5c0 2023-10-16T13:28:14.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.788+0000 7faeec15c700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faee4072a50 con 0x7faee40ce010 2023-10-16T13:28:14.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.788+0000 7faeea6f9700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7faee40d0910 0x7faee415e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:14.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.788+0000 7faeea6f9700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7faee40d0910 0x7faee415e210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49680/0 (socket says 172.21.15.16:49680) 2023-10-16T13:28:14.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.788+0000 7faeea6f9700 1 -- 172.21.15.16:0/1502672887 learned_addr learned my addr 172.21.15.16:0/1502672887 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:14.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.788+0000 7faee9ef8700 1 --2- 172.21.15.16:0/1502672887 >> v2:172.21.15.49:3300/0 conn(0x7faee415a5c0 0x7faee40cef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:14.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.788+0000 7faee96f7700 1 --2- 172.21.15.16:0/1502672887 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faee40ce010 0x7faee40d03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:14.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.788+0000 7faeea6f9700 1 -- 172.21.15.16:0/1502672887 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faee40ce010 msgr2=0x7faee40d03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:28:14.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.788+0000 7faeea6f9700 1 --2- 172.21.15.16:0/1502672887 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faee40ce010 0x7faee40d03d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:14.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.788+0000 7faeea6f9700 1 -- 172.21.15.16:0/1502672887 >> v2:172.21.15.49:3300/0 conn(0x7faee415a5c0 msgr2=0x7faee40cef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:28:14.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.788+0000 7faeea6f9700 1 --2- 172.21.15.16:0/1502672887 >> v2:172.21.15.49:3300/0 conn(0x7faee415a5c0 0x7faee40cef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:14.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.788+0000 7faeea6f9700 1 -- 172.21.15.16:0/1502672887 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faee40cba30 con 0x7faee40d0910 2023-10-16T13:28:14.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.789+0000 7faeea6f9700 1 --2- 172.21.15.16:0/1502672887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faee40d0910 0x7faee415e210 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7faee40b2ce0 tx=0x7faedc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=179b72d65eb34f5f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:14.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.789+0000 7faee8ef6700 1 -- 172.21.15.16:0/1502672887 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faedc00e070 con 0x7faee40d0910 2023-10-16T13:28:14.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.789+0000 7faee8ef6700 1 -- 172.21.15.16:0/1502672887 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faedc00b910 con 0x7faee40d0910 2023-10-16T13:28:14.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.789+0000 7faee8ef6700 1 -- 172.21.15.16:0/1502672887 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faedc00bcc0 con 0x7faee40d0910 2023-10-16T13:28:14.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.789+0000 7faeec15c700 1 -- 172.21.15.16:0/1502672887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faee40d0910 msgr2=0x7faee415e210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:14.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.789+0000 7faeec15c700 1 --2- 172.21.15.16:0/1502672887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faee40d0910 0x7faee415e210 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7faee40b2ce0 tx=0x7faedc00b330 comp rx=0 tx=0).stop 2023-10-16T13:28:14.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.789+0000 7faeec15c700 1 -- 172.21.15.16:0/1502672887 shutdown_connections 2023-10-16T13:28:14.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.789+0000 7faeec15c700 1 --2- 172.21.15.16:0/1502672887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faee40d0910 0x7faee415e210 unknown :-1 s=CLOSED pgs=2060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:14.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.789+0000 7faeec15c700 1 --2- 172.21.15.16:0/1502672887 >> v2:172.21.15.49:3300/0 conn(0x7faee415a5c0 0x7faee40cef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:14.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.789+0000 7faeec15c700 1 --2- 172.21.15.16:0/1502672887 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faee40ce010 0x7faee40d03d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:14.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.789+0000 7faeec15c700 1 -- 172.21.15.16:0/1502672887 >> 172.21.15.16:0/1502672887 conn(0x7faee40b1fd0 msgr2=0x7faee40b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:14.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.789+0000 7faeec15c700 1 -- 172.21.15.16:0/1502672887 shutdown_connections 2023-10-16T13:28:14.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.789+0000 7faeec15c700 1 -- 172.21.15.16:0/1502672887 wait complete. 2023-10-16T13:28:14.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.791+0000 7faeec15c700 1 Processor -- start 2023-10-16T13:28:14.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.791+0000 7faeec15c700 1 -- start start 2023-10-16T13:28:14.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.791+0000 7faeec15c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faee40ce010 0x7faee40c6a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:14.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.791+0000 7faeec15c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faee40d0910 0x7faee40b5c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:14.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.791+0000 7faeec15c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faee415a5c0 0x7faee40c5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:14.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.792+0000 7faeec15c700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faee40add00 con 0x7faee415a5c0 2023-10-16T13:28:14.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.792+0000 7faeec15c700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faee40b61b0 con 0x7faee40ce010 2023-10-16T13:28:14.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.792+0000 7faeec15c700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faee40c6350 con 0x7faee40d0910 2023-10-16T13:28:14.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.792+0000 7faeea6f9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faee415a5c0 0x7faee40c5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:14.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.792+0000 7faeea6f9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faee415a5c0 0x7faee40c5b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49686/0 (socket says 172.21.15.16:49686) 2023-10-16T13:28:14.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.792+0000 7faeea6f9700 1 -- 172.21.15.16:0/1435721226 learned_addr learned my addr 172.21.15.16:0/1435721226 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:14.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.792+0000 7faee9ef8700 1 --2- 172.21.15.16:0/1435721226 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faee40ce010 0x7faee40c6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:14.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.792+0000 7faee96f7700 1 --2- 172.21.15.16:0/1435721226 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faee40d0910 0x7faee40b5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:14.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.792+0000 7faeea6f9700 1 -- 172.21.15.16:0/1435721226 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faee40d0910 msgr2=0x7faee40b5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:14.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.792+0000 7faeea6f9700 1 --2- 172.21.15.16:0/1435721226 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faee40d0910 0x7faee40b5c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:14.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.792+0000 7faeea6f9700 1 -- 172.21.15.16:0/1435721226 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faee40ce010 msgr2=0x7faee40c6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:14.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.792+0000 7faeea6f9700 1 --2- 172.21.15.16:0/1435721226 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faee40ce010 0x7faee40c6a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:14.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.792+0000 7faeea6f9700 1 -- 172.21.15.16:0/1435721226 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faedc009b30 con 0x7faee415a5c0 2023-10-16T13:28:14.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.792+0000 7faeea6f9700 1 --2- 172.21.15.16:0/1435721226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faee415a5c0 0x7faee40c5b70 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 crypto rx=0x7faee40737d0 tx=0x7faedc004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:14.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.792+0000 7faeda7fc700 1 -- 172.21.15.16:0/1435721226 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faedc0099b0 con 0x7faee415a5c0 2023-10-16T13:28:14.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.792+0000 7faeda7fc700 1 -- 172.21.15.16:0/1435721226 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faedc0092e0 con 0x7faee415a5c0 2023-10-16T13:28:14.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.792+0000 7faeec15c700 1 -- 172.21.15.16:0/1435721226 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faee40ad410 con 0x7faee415a5c0 2023-10-16T13:28:14.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.792+0000 7faeda7fc700 1 -- 172.21.15.16:0/1435721226 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faedc004470 con 0x7faee415a5c0 2023-10-16T13:28:14.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.792+0000 7faeec15c700 1 -- 172.21.15.16:0/1435721226 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faee40ad6b0 con 0x7faee415a5c0 2023-10-16T13:28:14.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.793+0000 7faeec15c700 1 -- 172.21.15.16:0/1435721226 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faee40adeb0 con 0x7faee415a5c0 2023-10-16T13:28:14.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.794+0000 7faeda7fc700 1 -- 172.21.15.16:0/1435721226 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 107) v1 ==== 89331+0+0 (secure 0 0 0) 0x7faedc0045d0 con 0x7faee415a5c0 2023-10-16T13:28:14.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.794+0000 7faeda7fc700 1 --2- 172.21.15.16:0/1435721226 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7faec806b020 0x7faec806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:14.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.794+0000 7faeda7fc700 1 -- 172.21.15.16:0/1435721226 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 6051+0+0 (secure 0 0 0) 0x7faedc08aae0 con 0x7faee415a5c0 2023-10-16T13:28:14.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.794+0000 7faee9ef8700 1 --2- 172.21.15.16:0/1435721226 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7faec806b020 0x7faec806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:14.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.795+0000 7faee9ef8700 1 --2- 172.21.15.16:0/1435721226 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7faec806b020 0x7faec806d4e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7faed4009be0 tx=0x7faed4009210 comp rx=0 tx=0).ready entity=mgr.10047 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:14.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:14.799+0000 7faeda7fc700 1 -- 172.21.15.16:0/1435721226 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7faee40adeb0 con 0x7faee415a5c0 2023-10-16T13:28:15.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.020+0000 7faeec15c700 1 -- 172.21.15.16:0/1435721226 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7faee40adeb0 con 0x7faee415a5c0 2023-10-16T13:28:15.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.605+0000 7faeda7fc700 1 -- 172.21.15.16:0/1435721226 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v184) v1 ==== 97+0+0 (secure 0 0 0) 0x7faee40adeb0 con 0x7faee415a5c0 2023-10-16T13:28:15.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.609+0000 7faeec15c700 1 -- 172.21.15.16:0/1435721226 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7faec806b020 msgr2=0x7faec806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:15.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.609+0000 7faeec15c700 1 --2- 172.21.15.16:0/1435721226 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7faec806b020 0x7faec806d4e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7faed4009be0 tx=0x7faed4009210 comp rx=0 tx=0).stop 2023-10-16T13:28:15.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.609+0000 7faeec15c700 1 -- 172.21.15.16:0/1435721226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faee415a5c0 msgr2=0x7faee40c5b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:15.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.609+0000 7faeec15c700 1 --2- 172.21.15.16:0/1435721226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faee415a5c0 0x7faee40c5b70 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 crypto rx=0x7faee40737d0 tx=0x7faedc004ad0 comp rx=0 tx=0).stop 2023-10-16T13:28:15.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.609+0000 7faeec15c700 1 -- 172.21.15.16:0/1435721226 shutdown_connections 2023-10-16T13:28:15.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.609+0000 7faeec15c700 1 --2- 172.21.15.16:0/1435721226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faee415a5c0 0x7faee40c5b70 unknown :-1 s=CLOSED pgs=2061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:15.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.609+0000 7faeec15c700 1 --2- 172.21.15.16:0/1435721226 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7faec806b020 0x7faec806d4e0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:15.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.609+0000 7faeec15c700 1 --2- 172.21.15.16:0/1435721226 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faee40ce010 0x7faee40c6a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:15.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.609+0000 7faeec15c700 1 --2- 172.21.15.16:0/1435721226 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faee40d0910 0x7faee40b5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:15.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.609+0000 7faeec15c700 1 -- 172.21.15.16:0/1435721226 >> 172.21.15.16:0/1435721226 conn(0x7faee40b1fd0 msgr2=0x7faee40b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:15.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.609+0000 7faeec15c700 1 -- 172.21.15.16:0/1435721226 shutdown_connections 2023-10-16T13:28:15.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.609+0000 7faeec15c700 1 -- 172.21.15.16:0/1435721226 wait complete. 2023-10-16T13:28:15.614 INFO:teuthology.orchestra.run.smithi016.stderr: removed all blocklist entries 2023-10-16T13:28:15.626 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-16T13:28:15.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.751+0000 7f27054c8700 1 Processor -- start 2023-10-16T13:28:15.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.751+0000 7f27054c8700 1 -- start start 2023-10-16T13:28:15.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.752+0000 7f27054c8700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f270015d8d0 0x7f27000d1c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:15.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.752+0000 7f27054c8700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f27000d41c0 0x7f27000d0040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:15.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.752+0000 7f27054c8700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f27000d0580 0x7f27000ce240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:15.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.752+0000 7f27054c8700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27000d0a10 con 0x7f270015d8d0 2023-10-16T13:28:15.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.752+0000 7f27054c8700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f270006ba00 con 0x7f27000d0580 2023-10-16T13:28:15.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.752+0000 7f27054c8700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27000d2150 con 0x7f27000d41c0 2023-10-16T13:28:15.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.752+0000 7f26feffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f270015d8d0 0x7f27000d1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:15.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.752+0000 7f26feffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f270015d8d0 0x7f27000d1c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49692/0 (socket says 172.21.15.16:49692) 2023-10-16T13:28:15.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.752+0000 7f26feffd700 1 -- 172.21.15.16:0/1106976383 learned_addr learned my addr 172.21.15.16:0/1106976383 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:15.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.752+0000 7f26fe7fc700 1 --2- 172.21.15.16:0/1106976383 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f27000d41c0 0x7f27000d0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:15.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.752+0000 7f26ff7fe700 1 --2- 172.21.15.16:0/1106976383 >> v2:172.21.15.49:3300/0 conn(0x7f27000d0580 0x7f27000ce240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:15.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.753+0000 7f26feffd700 1 -- 172.21.15.16:0/1106976383 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f27000d41c0 msgr2=0x7f27000d0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:28:15.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.753+0000 7f26feffd700 1 --2- 172.21.15.16:0/1106976383 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f27000d41c0 0x7f27000d0040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:15.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.753+0000 7f26feffd700 1 -- 172.21.15.16:0/1106976383 >> v2:172.21.15.49:3300/0 conn(0x7f27000d0580 msgr2=0x7f27000ce240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:28:15.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.753+0000 7f26feffd700 1 --2- 172.21.15.16:0/1106976383 >> v2:172.21.15.49:3300/0 conn(0x7f27000d0580 0x7f27000ce240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:15.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.753+0000 7f26feffd700 1 -- 172.21.15.16:0/1106976383 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27000d26d0 con 0x7f270015d8d0 2023-10-16T13:28:15.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.753+0000 7f26ff7fe700 1 --2- 172.21.15.16:0/1106976383 >> v2:172.21.15.49:3300/0 conn(0x7f27000d0580 0x7f27000ce240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:28:15.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.753+0000 7f26feffd700 1 --2- 172.21.15.16:0/1106976383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f270015d8d0 0x7f27000d1c10 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 crypto rx=0x7f27000c70f0 tx=0x7f26f4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2256f405ec02009c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:15.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.753+0000 7f26fdffb700 1 -- 172.21.15.16:0/1106976383 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26f400b870 con 0x7f270015d8d0 2023-10-16T13:28:15.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.753+0000 7f26fdffb700 1 -- 172.21.15.16:0/1106976383 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26f400b9d0 con 0x7f270015d8d0 2023-10-16T13:28:15.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.753+0000 7f26fdffb700 1 -- 172.21.15.16:0/1106976383 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26f400bd60 con 0x7f270015d8d0 2023-10-16T13:28:15.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.753+0000 7f27054c8700 1 -- 172.21.15.16:0/1106976383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f270015d8d0 msgr2=0x7f27000d1c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:15.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.753+0000 7f27054c8700 1 --2- 172.21.15.16:0/1106976383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f270015d8d0 0x7f27000d1c10 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 crypto rx=0x7f27000c70f0 tx=0x7f26f4009bd0 comp rx=0 tx=0).stop 2023-10-16T13:28:15.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.754+0000 7f27054c8700 1 -- 172.21.15.16:0/1106976383 shutdown_connections 2023-10-16T13:28:15.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.754+0000 7f27054c8700 1 --2- 172.21.15.16:0/1106976383 >> v2:172.21.15.49:3300/0 conn(0x7f27000d0580 0x7f27000ce240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:15.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.754+0000 7f27054c8700 1 --2- 172.21.15.16:0/1106976383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f270015d8d0 0x7f27000d1c10 unknown :-1 s=CLOSED pgs=2062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:15.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.754+0000 7f27054c8700 1 --2- 172.21.15.16:0/1106976383 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f27000d41c0 0x7f27000d0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:15.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.754+0000 7f27054c8700 1 -- 172.21.15.16:0/1106976383 >> 172.21.15.16:0/1106976383 conn(0x7f27000d3630 msgr2=0x7f27000d3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:15.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.754+0000 7f27054c8700 1 -- 172.21.15.16:0/1106976383 shutdown_connections 2023-10-16T13:28:15.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.754+0000 7f27054c8700 1 -- 172.21.15.16:0/1106976383 wait complete. 2023-10-16T13:28:15.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.755+0000 7f27054c8700 1 Processor -- start 2023-10-16T13:28:15.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.755+0000 7f27054c8700 1 -- start start 2023-10-16T13:28:15.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.756+0000 7f27054c8700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f27000d0580 0x7f2700149aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:15.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.756+0000 7f27054c8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27000d41c0 0x7f27001e7b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:15.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.756+0000 7f27054c8700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f270015d8d0 0x7f27000cc300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:15.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.756+0000 7f27054c8700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2700149fe0 con 0x7f27000d41c0 2023-10-16T13:28:15.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.756+0000 7f27054c8700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27000caee0 con 0x7f270015d8d0 2023-10-16T13:28:15.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.756+0000 7f27054c8700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27000c9720 con 0x7f27000d0580 2023-10-16T13:28:15.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.756+0000 7f26fe7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27000d41c0 0x7f27001e7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:15.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.756+0000 7f26feffd700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f27000d0580 0x7f2700149aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:15.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.756+0000 7f26fe7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27000d41c0 0x7f27001e7b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49700/0 (socket says 172.21.15.16:49700) 2023-10-16T13:28:15.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.756+0000 7f26ff7fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f270015d8d0 0x7f27000cc300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:15.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.756+0000 7f26fe7fc700 1 -- 172.21.15.16:0/2587562476 learned_addr learned my addr 172.21.15.16:0/2587562476 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:15.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.756+0000 7f26fe7fc700 1 -- 172.21.15.16:0/2587562476 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f27000d0580 msgr2=0x7f2700149aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:15.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.756+0000 7f26fe7fc700 1 --2- 172.21.15.16:0/2587562476 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f27000d0580 0x7f2700149aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:15.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.756+0000 7f26fe7fc700 1 -- 172.21.15.16:0/2587562476 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f270015d8d0 msgr2=0x7f27000cc300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:15.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.756+0000 7f26fe7fc700 1 --2- 172.21.15.16:0/2587562476 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f270015d8d0 0x7f27000cc300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:15.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.756+0000 7f26fe7fc700 1 -- 172.21.15.16:0/2587562476 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26f4009550 con 0x7f27000d41c0 2023-10-16T13:28:15.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.756+0000 7f26fe7fc700 1 --2- 172.21.15.16:0/2587562476 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27000d41c0 0x7f27001e7b70 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7f26f000e810 tx=0x7f26f000eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:15.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.757+0000 7f26e77fe700 1 -- 172.21.15.16:0/2587562476 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26f000c070 con 0x7f27000d41c0 2023-10-16T13:28:15.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.757+0000 7f26e77fe700 1 -- 172.21.15.16:0/2587562476 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26f0013610 con 0x7f27000d41c0 2023-10-16T13:28:15.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.757+0000 7f27054c8700 1 -- 172.21.15.16:0/2587562476 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27000cac00 con 0x7f27000d41c0 2023-10-16T13:28:15.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.757+0000 7f26e77fe700 1 -- 172.21.15.16:0/2587562476 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26f00139c0 con 0x7f27000d41c0 2023-10-16T13:28:15.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.757+0000 7f27054c8700 1 -- 172.21.15.16:0/2587562476 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27000c93b0 con 0x7f27000d41c0 2023-10-16T13:28:15.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.758+0000 7f27054c8700 1 -- 172.21.15.16:0/2587562476 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f270004ebb0 con 0x7f27000d41c0 2023-10-16T13:28:15.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.758+0000 7f26e77fe700 1 -- 172.21.15.16:0/2587562476 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 107) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f26f0013b20 con 0x7f27000d41c0 2023-10-16T13:28:15.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.759+0000 7f26e77fe700 1 --2- 172.21.15.16:0/2587562476 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f26ec06b020 0x7f26ec06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:15.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.759+0000 7f26e77fe700 1 -- 172.21.15.16:0/2587562476 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 5277+0+0 (secure 0 0 0) 0x7f26f00893d0 con 0x7f27000d41c0 2023-10-16T13:28:15.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.759+0000 7f26feffd700 1 --2- 172.21.15.16:0/2587562476 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f26ec06b020 0x7f26ec06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:15.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.759+0000 7f26feffd700 1 --2- 172.21.15.16:0/2587562476 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f26ec06b020 0x7f26ec06d4e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f26f4000c00 tx=0x7f26f4008030 comp rx=0 tx=0).ready entity=mgr.10047 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:15.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:15.763+0000 7f26e77fe700 1 -- 172.21.15.16:0/2587562476 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f26f0054420 con 0x7f27000d41c0 2023-10-16T13:28:16.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.049+0000 7f27054c8700 1 -- 172.21.15.16:0/2587562476 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f27000c9860 con 0x7f27000d41c0 2023-10-16T13:28:16.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.050+0000 7f26e77fe700 1 -- 172.21.15.16:0/2587562476 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v47) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f26f0057a40 con 0x7f27000d41c0 2023-10-16T13:28:16.053 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:28:16.053 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-10-16T13:28:16.053 INFO:teuthology.orchestra.run.smithi016.stdout: "auth_dump": [ 2023-10-16T13:28:16.053 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:28:16.054 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.a", 2023-10-16T13:28:16.054 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOKC1l/rBtOhAA/ob5WUvpz8Glp4E+xPZOYA==", 2023-10-16T13:28:16.054 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:28:16.054 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T13:28:16.054 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-10-16T13:28:16.055 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-10-16T13:28:16.055 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:28:16.055 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:28:16.055 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:28:16.055 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:28:16.056 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.b", 2023-10-16T13:28:16.056 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1lKAVsCxAAvhXuLY6i1+Iz1yq5BVPMNQ==", 2023-10-16T13:28:16.056 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:28:16.056 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T13:28:16.056 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-10-16T13:28:16.056 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-10-16T13:28:16.057 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:28:16.057 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:28:16.057 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:28:16.057 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:28:16.057 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.c", 2023-10-16T13:28:16.058 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1lvBowBhAA3waJNYhdgs8PGCYqYNYUog==", 2023-10-16T13:28:16.058 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:28:16.058 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T13:28:16.058 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-10-16T13:28:16.058 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-10-16T13:28:16.059 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:28:16.059 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:28:16.059 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:28:16.059 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:28:16.059 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.d", 2023-10-16T13:28:16.060 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1lS/fnEhAAfNYgQ5BW86JMjYm/bBfnaQ==", 2023-10-16T13:28:16.060 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:28:16.060 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T13:28:16.060 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-10-16T13:28:16.060 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-10-16T13:28:16.060 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:28:16.061 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:28:16.061 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:28:16.061 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:28:16.061 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.0", 2023-10-16T13:28:16.061 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAQKC1lbxllOhAAZeLdp2GMl6HqQPaQjKlNqw==", 2023-10-16T13:28:16.062 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:28:16.062 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T13:28:16.062 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T13:28:16.062 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:28:16.062 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:28:16.062 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:28:16.063 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:28:16.063 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.1", 2023-10-16T13:28:16.063 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQATKC1lnU6KOhAAHxvlbmzBptf1keGgL+Rbkg==", 2023-10-16T13:28:16.063 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:28:16.063 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T13:28:16.064 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T13:28:16.064 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:28:16.064 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:28:16.064 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:28:16.064 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:28:16.065 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.2", 2023-10-16T13:28:16.065 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAXKC1l9jFyARAAr/NAwtWhBMut9sAAoDEGWg==", 2023-10-16T13:28:16.065 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:28:16.065 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T13:28:16.065 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T13:28:16.065 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:28:16.066 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:28:16.066 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:28:16.066 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:28:16.066 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.3", 2023-10-16T13:28:16.066 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAaKC1lbw/lARAAGJ/9DNb5/7kLFSq7vrLtXg==", 2023-10-16T13:28:16.067 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:28:16.067 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T13:28:16.067 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T13:28:16.067 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:28:16.067 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:28:16.068 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:28:16.068 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:28:16.068 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.4", 2023-10-16T13:28:16.068 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAeKC1lZb7jERAAYpvb1xcSli9CTq7L8ChLOA==", 2023-10-16T13:28:16.068 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:28:16.069 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T13:28:16.069 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T13:28:16.069 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:28:16.069 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:28:16.069 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:28:16.069 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:28:16.070 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.5", 2023-10-16T13:28:16.070 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAhKC1l/4HuEBAAqfnRqjknij9KES/fqiyIBQ==", 2023-10-16T13:28:16.070 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:28:16.070 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T13:28:16.070 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T13:28:16.071 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:28:16.071 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:28:16.071 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:28:16.071 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:28:16.071 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.6", 2023-10-16T13:28:16.072 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAkKC1lzoUkEBAA+Q82OiBMNag8+R316WQtUw==", 2023-10-16T13:28:16.072 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:28:16.072 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T13:28:16.072 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T13:28:16.072 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:28:16.072 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:28:16.073 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:28:16.073 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:28:16.073 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.7", 2023-10-16T13:28:16.073 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAnKC1ljUu4DxAAc+G4anJANC2GhLJ8wb6gYg==", 2023-10-16T13:28:16.073 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:28:16.074 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T13:28:16.074 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T13:28:16.074 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:28:16.074 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:28:16.074 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:28:16.074 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:28:16.075 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.0", 2023-10-16T13:28:16.075 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1l2l6PFxAA9QyXtBAHdGAXpwH0JfknuQ==", 2023-10-16T13:28:16.075 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:28:16.075 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T13:28:16.075 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow r", 2023-10-16T13:28:16.076 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-16T13:28:16.076 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:28:16.076 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:28:16.076 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:28:16.076 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.1", 2023-10-16T13:28:16.076 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1lAcHFIxAAcNt2bmv2R6j5L3DYkdw7eQ==", 2023-10-16T13:28:16.077 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:28:16.077 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T13:28:16.077 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-10-16T13:28:16.077 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-10-16T13:28:16.077 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-10-16T13:28:16.078 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:28:16.078 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:28:16.078 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:28:16.078 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.2", 2023-10-16T13:28:16.078 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1lA6wnGxAAdtaqYhtIJu8qENMrD8xvVA==", 2023-10-16T13:28:16.078 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:28:16.079 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T13:28:16.079 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-10-16T13:28:16.079 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-10-16T13:28:16.079 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-10-16T13:28:16.079 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:28:16.080 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:28:16.080 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:28:16.080 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.3", 2023-10-16T13:28:16.080 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1l/ddAJxAA3fNKKn9XkXas1BlDN+ggjw==", 2023-10-16T13:28:16.080 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:28:16.080 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T13:28:16.081 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-10-16T13:28:16.081 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-10-16T13:28:16.081 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-10-16T13:28:16.081 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:28:16.081 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:28:16.082 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:28:16.082 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.admin", 2023-10-16T13:28:16.082 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOKC1ld5jUHBAAc6esnCce3dSEZn3wVjhIlQ==", 2023-10-16T13:28:16.082 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:28:16.082 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2023-10-16T13:28:16.083 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-10-16T13:28:16.083 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-10-16T13:28:16.083 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:28:16.083 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:28:16.083 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:28:16.083 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:28:16.084 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mgr.x", 2023-10-16T13:28:16.084 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOKC1l6b3vNhAAmj5xbfid2sZM2KfHmAf1Wg==", 2023-10-16T13:28:16.084 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:28:16.084 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2023-10-16T13:28:16.084 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile mgr", 2023-10-16T13:28:16.085 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:28:16.085 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:28:16.085 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:28:16.085 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:28:16.085 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mgr.y", 2023-10-16T13:28:16.085 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOKC1lLz2nMhAA8NqFYZ5nDKkzPBMrkwsVuQ==", 2023-10-16T13:28:16.086 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:28:16.086 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2023-10-16T13:28:16.086 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile mgr", 2023-10-16T13:28:16.086 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:28:16.086 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:28:16.087 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:28:16.089 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-10-16T13:28:16.089 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-10-16T13:28:16.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.052+0000 7f27054c8700 1 -- 172.21.15.16:0/2587562476 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f26ec06b020 msgr2=0x7f26ec06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:16.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.052+0000 7f27054c8700 1 --2- 172.21.15.16:0/2587562476 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f26ec06b020 0x7f26ec06d4e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f26f4000c00 tx=0x7f26f4008030 comp rx=0 tx=0).stop 2023-10-16T13:28:16.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.052+0000 7f27054c8700 1 -- 172.21.15.16:0/2587562476 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27000d41c0 msgr2=0x7f27001e7b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:16.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.052+0000 7f27054c8700 1 --2- 172.21.15.16:0/2587562476 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27000d41c0 0x7f27001e7b70 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7f26f000e810 tx=0x7f26f000eb20 comp rx=0 tx=0).stop 2023-10-16T13:28:16.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.052+0000 7f27054c8700 1 -- 172.21.15.16:0/2587562476 shutdown_connections 2023-10-16T13:28:16.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.052+0000 7f27054c8700 1 --2- 172.21.15.16:0/2587562476 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f26ec06b020 0x7f26ec06d4e0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:16.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.052+0000 7f27054c8700 1 --2- 172.21.15.16:0/2587562476 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f270015d8d0 0x7f27000cc300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:16.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.052+0000 7f27054c8700 1 --2- 172.21.15.16:0/2587562476 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f27000d0580 0x7f2700149aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:16.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.052+0000 7f27054c8700 1 --2- 172.21.15.16:0/2587562476 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27000d41c0 0x7f27001e7b70 unknown :-1 s=CLOSED pgs=2063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:16.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.052+0000 7f27054c8700 1 -- 172.21.15.16:0/2587562476 >> 172.21.15.16:0/2587562476 conn(0x7f27000d3630 msgr2=0x7f270015cae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:16.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.052+0000 7f27054c8700 1 -- 172.21.15.16:0/2587562476 shutdown_connections 2023-10-16T13:28:16.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.052+0000 7f27054c8700 1 -- 172.21.15.16:0/2587562476 wait complete. 2023-10-16T13:28:16.094 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:28:16.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.216+0000 7f35d2520700 1 Processor -- start 2023-10-16T13:28:16.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.217+0000 7f35d2520700 1 -- start start 2023-10-16T13:28:16.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.217+0000 7f35d2520700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f35cc15db30 0x7f35cc0cf920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:16.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.217+0000 7f35d2520700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35cc0d0070 con 0x7f35cc15d750 2023-10-16T13:28:16.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.217+0000 7f35d2520700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35cc0d0350 con 0x7f35cc161490 2023-10-16T13:28:16.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.217+0000 7f35d2520700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35cc0cfe60 con 0x7f35cc15db30 2023-10-16T13:28:16.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.218+0000 7f35cbfff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f35cc15d750 0x7f35cc06c960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38980/0 (socket says 172.21.15.16:38980) 2023-10-16T13:28:16.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.218+0000 7f35cbfff700 1 -- 172.21.15.16:0/2976083587 learned_addr learned my addr 172.21.15.16:0/2976083587 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:28:16.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.218+0000 7f35caffd700 1 -- 172.21.15.16:0/2976083587 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 264512680 0 0) 0x7f35cc0d0070 con 0x7f35cc15d750 2023-10-16T13:28:16.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.218+0000 7f35caffd700 1 -- 172.21.15.16:0/2976083587 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35b40035b0 con 0x7f35cc15d750 2023-10-16T13:28:16.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.218+0000 7f35cb7fe700 1 --2- 172.21.15.16:0/2976083587 >> v2:172.21.15.49:3300/0 conn(0x7f35cc15db30 0x7f35cc0cf920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:16.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.218+0000 7f35caffd700 1 -- 172.21.15.16:0/2976083587 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 685904405 0 0) 0x7f35b40035b0 con 0x7f35cc15d750 2023-10-16T13:28:16.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.218+0000 7f35caffd700 1 -- 172.21.15.16:0/2976083587 >> v2:172.21.15.49:3300/0 conn(0x7f35cc15db30 msgr2=0x7f35cc0cf920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:28:16.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.218+0000 7f35caffd700 1 --2- 172.21.15.16:0/2976083587 >> v2:172.21.15.49:3300/0 conn(0x7f35cc15db30 0x7f35cc0cf920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:16.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.218+0000 7f35caffd700 1 -- 172.21.15.16:0/2976083587 >> v1:172.21.15.49:6789/0 conn(0x7f35cc161490 legacy=0x7f35cc161870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:16.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.218+0000 7f35caffd700 1 -- 172.21.15.16:0/2976083587 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35cc0d0630 con 0x7f35cc15d750 2023-10-16T13:28:16.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.218+0000 7f35caffd700 1 -- 172.21.15.16:0/2976083587 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f35bc0017f0 con 0x7f35cc15d750 2023-10-16T13:28:16.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.218+0000 7f35caffd700 1 -- 172.21.15.16:0/2976083587 >> v1:172.21.15.16:6789/0 conn(0x7f35cc15d750 legacy=0x7f35cc06c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:16.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.218+0000 7f35caffd700 1 --2- 172.21.15.16:0/2976083587 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f35b4003a00 0x7f35b4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:16.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.218+0000 7f35caffd700 1 --2- 172.21.15.16:0/2976083587 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f35b4004850 0x7f35b4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:16.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.218+0000 7f35caffd700 1 --2- 172.21.15.16:0/2976083587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35b4005210 0x7f35b400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:16.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.218+0000 7f35caffd700 1 -- 172.21.15.16:0/2976083587 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35cc0cfe60 con 0x7f35b4005210 2023-10-16T13:28:16.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.218+0000 7f35caffd700 1 -- 172.21.15.16:0/2976083587 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35b400c0b0 con 0x7f35b4004850 2023-10-16T13:28:16.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.218+0000 7f35caffd700 1 -- 172.21.15.16:0/2976083587 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35b400c270 con 0x7f35b4003a00 2023-10-16T13:28:16.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.218+0000 7f35d0abd700 1 --2- 172.21.15.16:0/2976083587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35b4005210 0x7f35b400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:16.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.218+0000 7f35cbfff700 1 --2- 172.21.15.16:0/2976083587 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f35b4003a00 0x7f35b4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:16.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.218+0000 7f35cb7fe700 1 --2- 172.21.15.16:0/2976083587 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f35b4004850 0x7f35b4004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:16.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.219+0000 7f35d0abd700 1 -- 172.21.15.16:0/2976083587 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f35b4003a00 msgr2=0x7f35b4003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:28:16.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.219+0000 7f35d0abd700 1 --2- 172.21.15.16:0/2976083587 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f35b4003a00 0x7f35b4003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:16.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.219+0000 7f35d0abd700 1 -- 172.21.15.16:0/2976083587 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f35b4004850 msgr2=0x7f35b4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:28:16.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.219+0000 7f35d0abd700 1 --2- 172.21.15.16:0/2976083587 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f35b4004850 0x7f35b4004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:16.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.219+0000 7f35d0abd700 1 -- 172.21.15.16:0/2976083587 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35b40035b0 con 0x7f35b4005210 2023-10-16T13:28:16.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.219+0000 7f35cb7fe700 1 --2- 172.21.15.16:0/2976083587 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f35b4004850 0x7f35b4004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:28:16.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.219+0000 7f35cbfff700 1 --2- 172.21.15.16:0/2976083587 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f35b4003a00 0x7f35b4003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:28:16.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.219+0000 7f35d0abd700 1 -- 172.21.15.16:0/2976083587 reap_dead start 2023-10-16T13:28:16.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.219+0000 7f35d0abd700 1 --2- 172.21.15.16:0/2976083587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35b4005210 0x7f35b400ba10 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 crypto rx=0x7f35c400c190 tx=0x7f35b4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1f5f88a4f4c81dd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:16.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.219+0000 7f35caffd700 1 -- 172.21.15.16:0/2976083587 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35c4001db0 con 0x7f35b4005210 2023-10-16T13:28:16.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.219+0000 7f35caffd700 1 -- 172.21.15.16:0/2976083587 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35c400ccd0 con 0x7f35b4005210 2023-10-16T13:28:16.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.219+0000 7f35caffd700 1 -- 172.21.15.16:0/2976083587 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35c4003c40 con 0x7f35b4005210 2023-10-16T13:28:16.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.219+0000 7f35d2520700 1 -- 172.21.15.16:0/2976083587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35b4005210 msgr2=0x7f35b400ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:16.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.219+0000 7f35d2520700 1 --2- 172.21.15.16:0/2976083587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35b4005210 0x7f35b400ba10 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 crypto rx=0x7f35c400c190 tx=0x7f35b4005880 comp rx=0 tx=0).stop 2023-10-16T13:28:16.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.219+0000 7f35d2520700 1 -- 172.21.15.16:0/2976083587 shutdown_connections 2023-10-16T13:28:16.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.219+0000 7f35d2520700 1 --2- 172.21.15.16:0/2976083587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35b4005210 0x7f35b400ba10 unknown :-1 s=CLOSED pgs=2065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:16.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.219+0000 7f35d2520700 1 -- 172.21.15.16:0/2976083587 >> 172.21.15.16:0/2976083587 conn(0x7f35cc0d14b0 msgr2=0x7f35cc0d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:16.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.219+0000 7f35d2520700 1 -- 172.21.15.16:0/2976083587 shutdown_connections 2023-10-16T13:28:16.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.219+0000 7f35d2520700 1 -- 172.21.15.16:0/2976083587 wait complete. 2023-10-16T13:28:16.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.221+0000 7f35d2520700 1 Processor -- start 2023-10-16T13:28:16.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.221+0000 7f35d2520700 1 -- start start 2023-10-16T13:28:16.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.221+0000 7f35d2520700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f35b4005210 0x7f35cc0b4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:16.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.221+0000 7f35d2520700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f35cc1f2c10 0x7f35cc0b5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:16.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.222+0000 7f35d2520700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35cc1546d0 0x7f35cc1f6540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:16.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.222+0000 7f35d2520700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35cc0b5ec0 con 0x7f35cc1546d0 2023-10-16T13:28:16.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.222+0000 7f35d2520700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35cc0b48e0 con 0x7f35b4005210 2023-10-16T13:28:16.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.222+0000 7f35d2520700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35cc0b4bc0 con 0x7f35cc1f2c10 2023-10-16T13:28:16.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.222+0000 7f35d0abd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35cc1546d0 0x7f35cc1f6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:16.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.222+0000 7f35d0abd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35cc1546d0 0x7f35cc1f6540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49722/0 (socket says 172.21.15.16:49722) 2023-10-16T13:28:16.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.222+0000 7f35d0abd700 1 -- 172.21.15.16:0/645115616 learned_addr learned my addr 172.21.15.16:0/645115616 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:16.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.222+0000 7f35cb7fe700 1 --2- 172.21.15.16:0/645115616 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f35cc1f2c10 0x7f35cc0b5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:16.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.222+0000 7f35cbfff700 1 --2- 172.21.15.16:0/645115616 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f35b4005210 0x7f35cc0b4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:16.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.222+0000 7f35d0abd700 1 -- 172.21.15.16:0/645115616 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f35cc1f2c10 msgr2=0x7f35cc0b5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:16.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.222+0000 7f35d0abd700 1 --2- 172.21.15.16:0/645115616 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f35cc1f2c10 0x7f35cc0b5980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:16.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.222+0000 7f35d0abd700 1 -- 172.21.15.16:0/645115616 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f35b4005210 msgr2=0x7f35cc0b4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:16.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.222+0000 7f35d0abd700 1 --2- 172.21.15.16:0/645115616 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f35b4005210 0x7f35cc0b4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:16.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.222+0000 7f35d0abd700 1 -- 172.21.15.16:0/645115616 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35cc0b4ea0 con 0x7f35cc1546d0 2023-10-16T13:28:16.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.222+0000 7f35d0abd700 1 --2- 172.21.15.16:0/645115616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35cc1546d0 0x7f35cc1f6540 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 crypto rx=0x7f35cc0d1460 tx=0x7f35c40014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:16.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.222+0000 7f35c8ff9700 1 -- 172.21.15.16:0/645115616 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35c4005070 con 0x7f35cc1546d0 2023-10-16T13:28:16.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.222+0000 7f35c8ff9700 1 -- 172.21.15.16:0/645115616 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35c40051d0 con 0x7f35cc1546d0 2023-10-16T13:28:16.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.222+0000 7f35d2520700 1 -- 172.21.15.16:0/645115616 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35cc0b5180 con 0x7f35cc1546d0 2023-10-16T13:28:16.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.222+0000 7f35c8ff9700 1 -- 172.21.15.16:0/645115616 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35c4005380 con 0x7f35cc1546d0 2023-10-16T13:28:16.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.222+0000 7f35d2520700 1 -- 172.21.15.16:0/645115616 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35cc154b60 con 0x7f35cc1546d0 2023-10-16T13:28:16.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.223+0000 7f35d2520700 1 -- 172.21.15.16:0/645115616 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35cc0b56a0 con 0x7f35cc1546d0 2023-10-16T13:28:16.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.224+0000 7f35c8ff9700 1 -- 172.21.15.16:0/645115616 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 107) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f35c40054e0 con 0x7f35cc1546d0 2023-10-16T13:28:16.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.224+0000 7f35c8ff9700 1 --2- 172.21.15.16:0/645115616 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f35b806b020 0x7f35b806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:16.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.224+0000 7f35c8ff9700 1 -- 172.21.15.16:0/645115616 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 5277+0+0 (secure 0 0 0) 0x7f35c408a010 con 0x7f35cc1546d0 2023-10-16T13:28:16.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.224+0000 7f35cbfff700 1 --2- 172.21.15.16:0/645115616 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f35b806b020 0x7f35b806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:16.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.225+0000 7f35cbfff700 1 --2- 172.21.15.16:0/645115616 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f35b806b020 0x7f35b806d4e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f35bc001380 tx=0x7f35bc009040 comp rx=0 tx=0).ready entity=mgr.10047 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:16.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.229+0000 7f35c8ff9700 1 -- 172.21.15.16:0/645115616 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f35cc0b56a0 con 0x7f35cc1546d0 2023-10-16T13:28:16.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.432+0000 7f35d2520700 1 -- 172.21.15.16:0/645115616 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f35cc0b56a0 con 0x7f35cc1546d0 2023-10-16T13:28:16.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.433+0000 7f35c8ff9700 1 -- 172.21.15.16:0/645115616 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v186) v1 ==== 74+0+13123 (secure 0 0 0) 0x7f35cc0b56a0 con 0x7f35cc1546d0 2023-10-16T13:28:16.435 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:28:16.436 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":186,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:28:15.613102+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:28:16.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.435+0000 7f35d2520700 1 -- 172.21.15.16:0/645115616 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f35b806b020 msgr2=0x7f35b806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:16.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.435+0000 7f35d2520700 1 --2- 172.21.15.16:0/645115616 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f35b806b020 0x7f35b806d4e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f35bc001380 tx=0x7f35bc009040 comp rx=0 tx=0).stop 2023-10-16T13:28:16.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.435+0000 7f35d2520700 1 -- 172.21.15.16:0/645115616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35cc1546d0 msgr2=0x7f35cc1f6540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:16.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.435+0000 7f35d2520700 1 --2- 172.21.15.16:0/645115616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35cc1546d0 0x7f35cc1f6540 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 crypto rx=0x7f35cc0d1460 tx=0x7f35c40014a0 comp rx=0 tx=0).stop 2023-10-16T13:28:16.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.435+0000 7f35d2520700 1 -- 172.21.15.16:0/645115616 shutdown_connections 2023-10-16T13:28:16.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.435+0000 7f35d2520700 1 --2- 172.21.15.16:0/645115616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35cc1546d0 0x7f35cc1f6540 unknown :-1 s=CLOSED pgs=2066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:16.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.435+0000 7f35d2520700 1 --2- 172.21.15.16:0/645115616 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f35b806b020 0x7f35b806d4e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:16.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.435+0000 7f35d2520700 1 --2- 172.21.15.16:0/645115616 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f35b4005210 0x7f35cc0b4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:16.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.435+0000 7f35d2520700 1 --2- 172.21.15.16:0/645115616 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f35cc1f2c10 0x7f35cc0b5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:16.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.435+0000 7f35d2520700 1 -- 172.21.15.16:0/645115616 >> 172.21.15.16:0/645115616 conn(0x7f35cc0d14b0 msgr2=0x7f35cc0b3300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:16.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.435+0000 7f35d2520700 1 -- 172.21.15.16:0/645115616 shutdown_connections 2023-10-16T13:28:16.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.435+0000 7f35d2520700 1 -- 172.21.15.16:0/645115616 wait complete. 2023-10-16T13:28:16.450 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T13:28:16.451 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T13:28:16.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.576+0000 7f54690d3700 1 Processor -- start 2023-10-16T13:28:16.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.576+0000 7f54690d3700 1 -- start start 2023-10-16T13:28:16.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.577+0000 7f54690d3700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f546415b070 0x7f546415b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:16.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.577+0000 7f54690d3700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f546406e170 0x7f54640cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:16.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.577+0000 7f54690d3700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54640cef70 con 0x7f54640cb880 2023-10-16T13:28:16.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.577+0000 7f54690d3700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54640ce360 con 0x7f546415b070 2023-10-16T13:28:16.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.577+0000 7f54690d3700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54640ce640 con 0x7f546406e170 2023-10-16T13:28:16.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.577+0000 7f5462d9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f546415b070 0x7f546415b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:16.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.577+0000 7f5462d9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f546415b070 0x7f546415b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49726/0 (socket says 172.21.15.16:49726) 2023-10-16T13:28:16.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.577+0000 7f5462d9d700 1 -- 172.21.15.16:0/685481256 learned_addr learned my addr 172.21.15.16:0/685481256 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:16.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.577+0000 7f546359e700 1 --2- 172.21.15.16:0/685481256 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f546406e170 0x7f54640cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:16.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.577+0000 7f5462d9d700 1 -- 172.21.15.16:0/685481256 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f546406e170 msgr2=0x7f54640cbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:28:16.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.577+0000 7f5462d9d700 1 --2- 172.21.15.16:0/685481256 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f546406e170 0x7f54640cbd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:16.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.577+0000 7f5462d9d700 1 -- 172.21.15.16:0/685481256 >> v1:172.21.15.16:6789/0 conn(0x7f54640cb880 legacy=0x7f546406da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:16.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.577+0000 7f5462d9d700 1 -- 172.21.15.16:0/685481256 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54640ce920 con 0x7f546415b070 2023-10-16T13:28:16.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.577+0000 7f5461d9b700 1 -- 172.21.15.16:0/685481256 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 637309307 0 0) 0x7f54640cef70 con 0x7f54640cb880 2023-10-16T13:28:16.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.577+0000 7f5462d9d700 1 --2- 172.21.15.16:0/685481256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f546415b070 0x7f546415b450 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 crypto rx=0x7f54580098f0 tx=0x7f5458009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8af978dcd4f860fb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:16.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.577+0000 7f5461d9b700 1 -- 172.21.15.16:0/685481256 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f545800b8a0 con 0x7f546415b070 2023-10-16T13:28:16.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.578+0000 7f5461d9b700 1 -- 172.21.15.16:0/685481256 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f545800ba00 con 0x7f546415b070 2023-10-16T13:28:16.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.578+0000 7f5461d9b700 1 -- 172.21.15.16:0/685481256 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f545800bd90 con 0x7f546415b070 2023-10-16T13:28:16.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.578+0000 7f54690d3700 1 -- 172.21.15.16:0/685481256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f546415b070 msgr2=0x7f546415b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:16.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.578+0000 7f54690d3700 1 --2- 172.21.15.16:0/685481256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f546415b070 0x7f546415b450 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 crypto rx=0x7f54580098f0 tx=0x7f5458009c00 comp rx=0 tx=0).stop 2023-10-16T13:28:16.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.578+0000 7f54690d3700 1 -- 172.21.15.16:0/685481256 shutdown_connections 2023-10-16T13:28:16.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.578+0000 7f54690d3700 1 --2- 172.21.15.16:0/685481256 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f546406e170 0x7f54640cbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:16.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.578+0000 7f54690d3700 1 --2- 172.21.15.16:0/685481256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f546415b070 0x7f546415b450 unknown :-1 s=CLOSED pgs=2068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:16.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.578+0000 7f54690d3700 1 -- 172.21.15.16:0/685481256 >> 172.21.15.16:0/685481256 conn(0x7f54640b1f40 msgr2=0x7f54640b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:16.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.578+0000 7f54690d3700 1 -- 172.21.15.16:0/685481256 shutdown_connections 2023-10-16T13:28:16.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.578+0000 7f54690d3700 1 -- 172.21.15.16:0/685481256 wait complete. 2023-10-16T13:28:16.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.580+0000 7f54690d3700 1 Processor -- start 2023-10-16T13:28:16.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.580+0000 7f54690d3700 1 -- start start 2023-10-16T13:28:16.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.580+0000 7f54690d3700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f546406e170 0x7f54640b1070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:16.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.580+0000 7f54690d3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54640cb880 0x7f54640ca560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:16.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.580+0000 7f54690d3700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f546415b070 0x7f54640caaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:16.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.580+0000 7f54690d3700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5464158210 con 0x7f54640cb880 2023-10-16T13:28:16.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.580+0000 7f54690d3700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5464154b20 con 0x7f546415b070 2023-10-16T13:28:16.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.580+0000 7f54690d3700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54640b5f60 con 0x7f546406e170 2023-10-16T13:28:16.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.580+0000 7f546259c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54640cb880 0x7f54640ca560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:16.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.580+0000 7f546259c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54640cb880 0x7f54640ca560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49734/0 (socket says 172.21.15.16:49734) 2023-10-16T13:28:16.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.580+0000 7f546259c700 1 -- 172.21.15.16:0/1021486817 learned_addr learned my addr 172.21.15.16:0/1021486817 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:16.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.581+0000 7f5462d9d700 1 --2- 172.21.15.16:0/1021486817 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f546406e170 0x7f54640b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:16.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.581+0000 7f546359e700 1 --2- 172.21.15.16:0/1021486817 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f546415b070 0x7f54640caaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:16.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.581+0000 7f546259c700 1 -- 172.21.15.16:0/1021486817 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f546406e170 msgr2=0x7f54640b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:16.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.581+0000 7f546259c700 1 --2- 172.21.15.16:0/1021486817 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f546406e170 0x7f54640b1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:16.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.581+0000 7f546259c700 1 -- 172.21.15.16:0/1021486817 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f546415b070 msgr2=0x7f54640caaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:16.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.581+0000 7f546259c700 1 --2- 172.21.15.16:0/1021486817 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f546415b070 0x7f54640caaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:16.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.581+0000 7f546259c700 1 -- 172.21.15.16:0/1021486817 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5458009550 con 0x7f54640cb880 2023-10-16T13:28:16.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.581+0000 7f546259c700 1 --2- 172.21.15.16:0/1021486817 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54640cb880 0x7f54640ca560 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 crypto rx=0x7f54540011e0 tx=0x7f545400cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:16.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.581+0000 7f544b7fe700 1 -- 172.21.15.16:0/1021486817 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54540054e0 con 0x7f54640cb880 2023-10-16T13:28:16.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.581+0000 7f544b7fe700 1 -- 172.21.15.16:0/1021486817 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5454003680 con 0x7f54640cb880 2023-10-16T13:28:16.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.581+0000 7f54690d3700 1 -- 172.21.15.16:0/1021486817 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54640b6240 con 0x7f54640cb880 2023-10-16T13:28:16.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.581+0000 7f544b7fe700 1 -- 172.21.15.16:0/1021486817 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5454003a30 con 0x7f54640cb880 2023-10-16T13:28:16.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.581+0000 7f54690d3700 1 -- 172.21.15.16:0/1021486817 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54640b3cc0 con 0x7f54640cb880 2023-10-16T13:28:16.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.582+0000 7f54690d3700 1 -- 172.21.15.16:0/1021486817 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f546404ebb0 con 0x7f54640cb880 2023-10-16T13:28:16.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.583+0000 7f544b7fe700 1 -- 172.21.15.16:0/1021486817 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 107) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5454003b90 con 0x7f54640cb880 2023-10-16T13:28:16.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.583+0000 7f544b7fe700 1 --2- 172.21.15.16:0/1021486817 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f545006b020 0x7f545006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:16.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.583+0000 7f544b7fe700 1 -- 172.21.15.16:0/1021486817 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 5277+0+0 (secure 0 0 0) 0x7f5454088af0 con 0x7f54640cb880 2023-10-16T13:28:16.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.583+0000 7f5462d9d700 1 --2- 172.21.15.16:0/1021486817 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f545006b020 0x7f545006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:16.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.584+0000 7f5462d9d700 1 --2- 172.21.15.16:0/1021486817 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f545006b020 0x7f545006d4e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5458014040 tx=0x7f5458008030 comp rx=0 tx=0).ready entity=mgr.10047 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:16.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.588+0000 7f544b7fe700 1 -- 172.21.15.16:0/1021486817 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5454053c40 con 0x7f54640cb880 2023-10-16T13:28:16.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.795+0000 7f54690d3700 1 -- 172.21.15.16:0/1021486817 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5464154e40 con 0x7f54640cb880 2023-10-16T13:28:16.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.795+0000 7f544b7fe700 1 -- 172.21.15.16:0/1021486817 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v186) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5454017070 con 0x7f54640cb880 2023-10-16T13:28:16.798 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T13:28:16.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.797+0000 7f54690d3700 1 -- 172.21.15.16:0/1021486817 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f545006b020 msgr2=0x7f545006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:16.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.797+0000 7f54690d3700 1 --2- 172.21.15.16:0/1021486817 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f545006b020 0x7f545006d4e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5458014040 tx=0x7f5458008030 comp rx=0 tx=0).stop 2023-10-16T13:28:16.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.797+0000 7f54690d3700 1 -- 172.21.15.16:0/1021486817 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54640cb880 msgr2=0x7f54640ca560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:16.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.797+0000 7f54690d3700 1 --2- 172.21.15.16:0/1021486817 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54640cb880 0x7f54640ca560 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 crypto rx=0x7f54540011e0 tx=0x7f545400cae0 comp rx=0 tx=0).stop 2023-10-16T13:28:16.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.797+0000 7f54690d3700 1 -- 172.21.15.16:0/1021486817 shutdown_connections 2023-10-16T13:28:16.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.797+0000 7f54690d3700 1 --2- 172.21.15.16:0/1021486817 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f545006b020 0x7f545006d4e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:16.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.797+0000 7f54690d3700 1 --2- 172.21.15.16:0/1021486817 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f546415b070 0x7f54640caaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:16.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.797+0000 7f54690d3700 1 --2- 172.21.15.16:0/1021486817 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f546406e170 0x7f54640b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:16.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.797+0000 7f54690d3700 1 --2- 172.21.15.16:0/1021486817 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54640cb880 0x7f54640ca560 unknown :-1 s=CLOSED pgs=2069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:16.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.797+0000 7f54690d3700 1 -- 172.21.15.16:0/1021486817 >> 172.21.15.16:0/1021486817 conn(0x7f54640b1f40 msgr2=0x7f546406da60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:16.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.797+0000 7f54690d3700 1 -- 172.21.15.16:0/1021486817 shutdown_connections 2023-10-16T13:28:16.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.797+0000 7f54690d3700 1 -- 172.21.15.16:0/1021486817 wait complete. 2023-10-16T13:28:16.812 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-16T13:28:16.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.935+0000 7f02be4af700 1 Processor -- start 2023-10-16T13:28:16.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.935+0000 7f02be4af700 1 -- start start 2023-10-16T13:28:16.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.936+0000 7f02be4af700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f02b815b280 0x7f02b81613e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:16.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.936+0000 7f02be4af700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02b8161c30 con 0x7f02b806a7e0 2023-10-16T13:28:16.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.936+0000 7f02be4af700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02b8161f10 con 0x7f02b80cc8a0 2023-10-16T13:28:16.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.936+0000 7f02be4af700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02b81621f0 con 0x7f02b815b280 2023-10-16T13:28:16.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.936+0000 7f02bca4c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f02b806a7e0 0x7f02b80d02b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:39020/0 (socket says 172.21.15.16:39020) 2023-10-16T13:28:16.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.936+0000 7f02bca4c700 1 -- 172.21.15.16:0/355946518 learned_addr learned my addr 172.21.15.16:0/355946518 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:28:16.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.936+0000 7f02b7fff700 1 --2- 172.21.15.16:0/355946518 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f02b815b280 0x7f02b81613e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:16.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.936+0000 7f02b6ffd700 1 -- 172.21.15.16:0/355946518 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2856837407 0 0) 0x7f02b8161c30 con 0x7f02b806a7e0 2023-10-16T13:28:16.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.936+0000 7f02b6ffd700 1 -- 172.21.15.16:0/355946518 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02a40035b0 con 0x7f02b806a7e0 2023-10-16T13:28:16.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.936+0000 7f02b6ffd700 1 -- 172.21.15.16:0/355946518 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3043009881 0 0) 0x7f02a40035b0 con 0x7f02b806a7e0 2023-10-16T13:28:16.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.936+0000 7f02b6ffd700 1 -- 172.21.15.16:0/355946518 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f02b815b280 msgr2=0x7f02b81613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:28:16.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.936+0000 7f02b6ffd700 1 --2- 172.21.15.16:0/355946518 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f02b815b280 0x7f02b81613e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:16.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.936+0000 7f02b6ffd700 1 -- 172.21.15.16:0/355946518 >> v1:172.21.15.49:6789/0 conn(0x7f02b80cc8a0 legacy=0x7f02b815e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:16.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.936+0000 7f02b6ffd700 1 -- 172.21.15.16:0/355946518 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02b815df00 con 0x7f02b806a7e0 2023-10-16T13:28:16.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.936+0000 7f02b6ffd700 1 -- 172.21.15.16:0/355946518 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f02b00019f0 con 0x7f02b806a7e0 2023-10-16T13:28:16.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.937+0000 7f02b6ffd700 1 -- 172.21.15.16:0/355946518 >> v1:172.21.15.16:6789/0 conn(0x7f02b806a7e0 legacy=0x7f02b80d02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:16.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.937+0000 7f02b6ffd700 1 --2- 172.21.15.16:0/355946518 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f02b815b280 0x7f02a4003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:16.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.937+0000 7f02b6ffd700 1 --2- 172.21.15.16:0/355946518 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02a4003f70 0x7f02a4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:16.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.937+0000 7f02b6ffd700 1 --2- 172.21.15.16:0/355946518 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f02a4004d90 0x7f02a4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:16.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.937+0000 7f02b6ffd700 1 -- 172.21.15.16:0/355946518 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b81621f0 con 0x7f02a4003f70 2023-10-16T13:28:16.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.937+0000 7f02b6ffd700 1 -- 172.21.15.16:0/355946518 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02a400ba70 con 0x7f02a4004d90 2023-10-16T13:28:16.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.937+0000 7f02b6ffd700 1 -- 172.21.15.16:0/355946518 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02a400bc30 con 0x7f02b815b280 2023-10-16T13:28:16.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.937+0000 7f02b77fe700 1 --2- 172.21.15.16:0/355946518 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02a4003f70 0x7f02a4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:16.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.937+0000 7f02b7fff700 1 --2- 172.21.15.16:0/355946518 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f02b815b280 0x7f02a4003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:16.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.937+0000 7f02bca4c700 1 --2- 172.21.15.16:0/355946518 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f02a4004d90 0x7f02a4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:16.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.937+0000 7f02b77fe700 1 -- 172.21.15.16:0/355946518 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f02b815b280 msgr2=0x7f02a4003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:28:16.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.937+0000 7f02b77fe700 1 --2- 172.21.15.16:0/355946518 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f02b815b280 0x7f02a4003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:16.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.937+0000 7f02b77fe700 1 -- 172.21.15.16:0/355946518 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f02a4004d90 msgr2=0x7f02a4005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:28:16.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.937+0000 7f02b77fe700 1 --2- 172.21.15.16:0/355946518 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f02a4004d90 0x7f02a4005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:16.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.937+0000 7f02b77fe700 1 -- 172.21.15.16:0/355946518 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02a40035b0 con 0x7f02a4003f70 2023-10-16T13:28:16.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.937+0000 7f02b77fe700 1 --2- 172.21.15.16:0/355946518 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02a4003f70 0x7f02a4004850 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 crypto rx=0x7f02a00018d0 tx=0x7f02a4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec2aa04e945e4693 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:16.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.937+0000 7f02b6ffd700 1 -- 172.21.15.16:0/355946518 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a0012620 con 0x7f02a4003f70 2023-10-16T13:28:16.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.937+0000 7f02b6ffd700 1 -- 172.21.15.16:0/355946518 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02a0012780 con 0x7f02a4003f70 2023-10-16T13:28:16.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.937+0000 7f02b6ffd700 1 -- 172.21.15.16:0/355946518 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a0012b30 con 0x7f02a4003f70 2023-10-16T13:28:16.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.938+0000 7f02be4af700 1 -- 172.21.15.16:0/355946518 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02a4003f70 msgr2=0x7f02a4004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:16.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.938+0000 7f02be4af700 1 --2- 172.21.15.16:0/355946518 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02a4003f70 0x7f02a4004850 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 crypto rx=0x7f02a00018d0 tx=0x7f02a4005880 comp rx=0 tx=0).stop 2023-10-16T13:28:16.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.938+0000 7f02bca4c700 1 -- 172.21.15.16:0/355946518 reap_dead start 2023-10-16T13:28:16.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.938+0000 7f02be4af700 1 -- 172.21.15.16:0/355946518 shutdown_connections 2023-10-16T13:28:16.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.938+0000 7f02be4af700 1 -- 172.21.15.16:0/355946518 >> 172.21.15.16:0/355946518 conn(0x7f02b80b1fd0 msgr2=0x7f02b80b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:16.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.938+0000 7f02be4af700 1 -- 172.21.15.16:0/355946518 shutdown_connections 2023-10-16T13:28:16.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.938+0000 7f02be4af700 1 -- 172.21.15.16:0/355946518 wait complete. 2023-10-16T13:28:16.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.939+0000 7f02be4af700 1 Processor -- start 2023-10-16T13:28:16.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.939+0000 7f02be4af700 1 -- start start 2023-10-16T13:28:16.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.940+0000 7f02be4af700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f02a4003f70 0x7f02b80c67d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:16.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.940+0000 7f02be4af700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f02b815f330 0x7f02b80ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:16.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.940+0000 7f02be4af700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02b8071e00 0x7f02b8072290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:16.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.940+0000 7f02be4af700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b80ad730 con 0x7f02b8071e00 2023-10-16T13:28:16.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.940+0000 7f02be4af700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b80727d0 con 0x7f02b815f330 2023-10-16T13:28:16.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.940+0000 7f02be4af700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02b80c6d10 con 0x7f02a4003f70 2023-10-16T13:28:16.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.940+0000 7f02bca4c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02b8071e00 0x7f02b8072290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:16.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.940+0000 7f02bca4c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02b8071e00 0x7f02b8072290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49758/0 (socket says 172.21.15.16:49758) 2023-10-16T13:28:16.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.940+0000 7f02bca4c700 1 -- 172.21.15.16:0/4045769561 learned_addr learned my addr 172.21.15.16:0/4045769561 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:16.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.940+0000 7f02b7fff700 1 --2- 172.21.15.16:0/4045769561 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f02a4003f70 0x7f02b80c67d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:16.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.940+0000 7f02b77fe700 1 --2- 172.21.15.16:0/4045769561 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f02b815f330 0x7f02b80ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:16.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.940+0000 7f02bca4c700 1 -- 172.21.15.16:0/4045769561 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f02a4003f70 msgr2=0x7f02b80c67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:16.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.940+0000 7f02bca4c700 1 --2- 172.21.15.16:0/4045769561 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f02a4003f70 0x7f02b80c67d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:16.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.940+0000 7f02bca4c700 1 -- 172.21.15.16:0/4045769561 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f02b815f330 msgr2=0x7f02b80ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:16.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.940+0000 7f02bca4c700 1 --2- 172.21.15.16:0/4045769561 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f02b815f330 0x7f02b80ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:16.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.940+0000 7f02bca4c700 1 -- 172.21.15.16:0/4045769561 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02b80af1c0 con 0x7f02b8071e00 2023-10-16T13:28:16.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.940+0000 7f02b7fff700 1 --2- 172.21.15.16:0/4045769561 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f02a4003f70 0x7f02b80c67d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:28:16.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.941+0000 7f02bca4c700 1 --2- 172.21.15.16:0/4045769561 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02b8071e00 0x7f02b8072290 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 crypto rx=0x7f02b80d1540 tx=0x7f02b000a640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:16.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.941+0000 7f02b4ff9700 1 -- 172.21.15.16:0/4045769561 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02b000adf0 con 0x7f02b8071e00 2023-10-16T13:28:16.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.941+0000 7f02b4ff9700 1 -- 172.21.15.16:0/4045769561 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02b00133f0 con 0x7f02b8071e00 2023-10-16T13:28:16.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.941+0000 7f02be4af700 1 -- 172.21.15.16:0/4045769561 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02b80af650 con 0x7f02b8071e00 2023-10-16T13:28:16.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.941+0000 7f02b4ff9700 1 -- 172.21.15.16:0/4045769561 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02b00137a0 con 0x7f02b8071e00 2023-10-16T13:28:16.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.941+0000 7f02be4af700 1 -- 172.21.15.16:0/4045769561 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02b80aeb70 con 0x7f02b8071e00 2023-10-16T13:28:16.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.942+0000 7f02be4af700 1 -- 172.21.15.16:0/4045769561 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02b80af350 con 0x7f02b8071e00 2023-10-16T13:28:16.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.942+0000 7f02b4ff9700 1 -- 172.21.15.16:0/4045769561 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 107) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f02b0013900 con 0x7f02b8071e00 2023-10-16T13:28:16.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.943+0000 7f02b4ff9700 1 --2- 172.21.15.16:0/4045769561 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f029806b020 0x7f029806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:16.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.943+0000 7f02b4ff9700 1 -- 172.21.15.16:0/4045769561 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 5277+0+0 (secure 0 0 0) 0x7f02b0089f80 con 0x7f02b8071e00 2023-10-16T13:28:16.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.943+0000 7f02b7fff700 1 --2- 172.21.15.16:0/4045769561 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f029806b020 0x7f029806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:16.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.943+0000 7f02b7fff700 1 --2- 172.21.15.16:0/4045769561 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f029806b020 0x7f029806d4e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f02a8009c60 tx=0x7f02a8008040 comp rx=0 tx=0).ready entity=mgr.10047 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:16.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:16.947+0000 7f02b4ff9700 1 -- 172.21.15.16:0/4045769561 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f02b00550d0 con 0x7f02b8071e00 2023-10-16T13:28:17.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:17.156+0000 7f02be4af700 1 -- 172.21.15.16:0/4045769561 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f02b804ebb0 con 0x7f02b8071e00 2023-10-16T13:28:17.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:17.156+0000 7f02b4ff9700 1 -- 172.21.15.16:0/4045769561 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v186) v1 ==== 84+0+65 (secure 0 0 0) 0x7f02b00586f0 con 0x7f02b8071e00 2023-10-16T13:28:17.159 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:28:17.159 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2023-10-16T13:28:17.159 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:28:17.159 INFO:teuthology.orchestra.run.smithi016.stdout: "poolnum": 1, 2023-10-16T13:28:17.160 INFO:teuthology.orchestra.run.smithi016.stdout: "poolname": ".mgr" 2023-10-16T13:28:17.160 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:28:17.160 INFO:teuthology.orchestra.run.smithi016.stdout:] 2023-10-16T13:28:17.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:17.158+0000 7f02be4af700 1 -- 172.21.15.16:0/4045769561 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f029806b020 msgr2=0x7f029806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:17.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:17.158+0000 7f02be4af700 1 --2- 172.21.15.16:0/4045769561 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f029806b020 0x7f029806d4e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f02a8009c60 tx=0x7f02a8008040 comp rx=0 tx=0).stop 2023-10-16T13:28:17.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:17.158+0000 7f02be4af700 1 -- 172.21.15.16:0/4045769561 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02b8071e00 msgr2=0x7f02b8072290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:17.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:17.158+0000 7f02be4af700 1 --2- 172.21.15.16:0/4045769561 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02b8071e00 0x7f02b8072290 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 crypto rx=0x7f02b80d1540 tx=0x7f02b000a640 comp rx=0 tx=0).stop 2023-10-16T13:28:17.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:17.158+0000 7f02be4af700 1 -- 172.21.15.16:0/4045769561 shutdown_connections 2023-10-16T13:28:17.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:17.158+0000 7f02be4af700 1 --2- 172.21.15.16:0/4045769561 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02b8071e00 0x7f02b8072290 unknown :-1 s=CLOSED pgs=2072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:17.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:17.158+0000 7f02be4af700 1 --2- 172.21.15.16:0/4045769561 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f02a4003f70 0x7f02b80c67d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:17.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:17.158+0000 7f02be4af700 1 --2- 172.21.15.16:0/4045769561 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f029806b020 0x7f029806d4e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:17.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:17.158+0000 7f02be4af700 1 --2- 172.21.15.16:0/4045769561 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f02b815f330 0x7f02b80ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:17.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:17.158+0000 7f02be4af700 1 -- 172.21.15.16:0/4045769561 >> 172.21.15.16:0/4045769561 conn(0x7f02b80b1fd0 msgr2=0x7f02b8161150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:17.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:17.158+0000 7f02be4af700 1 -- 172.21.15.16:0/4045769561 shutdown_connections 2023-10-16T13:28:17.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:17.158+0000 7f02be4af700 1 -- 172.21.15.16:0/4045769561 wait complete. 2023-10-16T13:28:17.173 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-16T13:28:17.173 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-16T13:28:17.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:17.300+0000 7f7f304d8700 1 Processor -- start 2023-10-16T13:28:17.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:17.300+0000 7f7f304d8700 1 -- start start 2023-10-16T13:28:17.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:17.301+0000 7f7f304d8700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7f28157b30 0x7f7f280cefc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:17.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:17.301+0000 7f7f304d8700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7f28157f10 0x7f7f280cf500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:17.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:17.301+0000 7f7f304d8700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f28142fc0 con 0x7f7f28161610 2023-10-16T13:28:17.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:17.301+0000 7f7f304d8700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f28162160 con 0x7f7f28157b30 2023-10-16T13:28:17.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:17.301+0000 7f7f304d8700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f2806d900 con 0x7f7f28157f10 2023-10-16T13:28:17.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:17.301+0000 7f7f2e274700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7f28157b30 0x7f7f280cefc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:17.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:17.301+0000 7f7f2e274700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7f28157b30 0x7f7f280cefc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49762/0 (socket says 172.21.15.16:49762) 2023-10-16T13:28:17.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:17.301+0000 7f7f2e274700 1 -- 172.21.15.16:0/766318047 learned_addr learned my addr 172.21.15.16:0/766318047 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:17.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:17.302+0000 7f7f2e274700 1 -- 172.21.15.16:0/766318047 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7f28157f10 msgr2=0x7f7f280cf500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:28:17.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:17.302+0000 7f7f2e274700 1 --2- 172.21.15.16:0/766318047 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7f28157f10 0x7f7f280cf500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:17.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:17.302+0000 7f7f2e274700 1 -- 172.21.15.16:0/766318047 >> v1:172.21.15.16:6789/0 conn(0x7f7f28161610 legacy=0x7f7f28161a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:17.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:17.302+0000 7f7f2d272700 1 -- 172.21.15.16:0/766318047 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 294072517 0 0) 0x7f7f28142fc0 con 0x7f7f28161610 2023-10-16T13:28:17.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:17.302+0000 7f7f2e274700 1 -- 172.21.15.16:0/766318047 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f28143210 con 0x7f7f28157b30 2023-10-16T13:28:17.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:17.302+0000 7f7f2e274700 1 --2- 172.21.15.16:0/766318047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f28157b30 0x7f7f280cefc0 secure :-1 s=READY pgs=2074 cs=0 l=1 rev1=1 crypto rx=0x7f7f180098f0 tx=0x7f7f18009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33aaaf5ec71506c2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:17.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:17.302+0000 7f7f2d272700 1 -- 172.21.15.16:0/766318047 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f1800b8a0 con 0x7f7f28157b30 2023-10-16T13:28:17.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:17.302+0000 7f7f2d272700 1 -- 172.21.15.16:0/766318047 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f1800ba00 con 0x7f7f28157b30 2023-10-16T13:28:17.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:17.302+0000 7f7f2d272700 1 -- 172.21.15.16:0/766318047 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f1800bd90 con 0x7f7f28157b30 2023-10-16T13:28:17.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:17.303+0000 7f7f304d8700 1 -- 172.21.15.16:0/766318047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f28157b30 msgr2=0x7f7f280cefc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:17.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:17.303+0000 7f7f304d8700 1 --2- 172.21.15.16:0/766318047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f28157b30 0x7f7f280cefc0 secure :-1 s=READY pgs=2074 cs=0 l=1 rev1=1 crypto rx=0x7f7f180098f0 tx=0x7f7f18009c00 comp rx=0 tx=0).stop 2023-10-16T13:28:17.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:17.303+0000 7f7f304d8700 1 -- 172.21.15.16:0/766318047 shutdown_connections 2023-10-16T13:28:17.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:17.303+0000 7f7f304d8700 1 --2- 172.21.15.16:0/766318047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f28157b30 0x7f7f280cefc0 unknown :-1 s=CLOSED pgs=2074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:17.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:17.303+0000 7f7f304d8700 1 --2- 172.21.15.16:0/766318047 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7f28157f10 0x7f7f280cf500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:17.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:17.303+0000 7f7f304d8700 1 -- 172.21.15.16:0/766318047 >> 172.21.15.16:0/766318047 conn(0x7f7f2806a7e0 msgr2=0x7f7f2806abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:17.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:17.303+0000 7f7f304d8700 1 -- 172.21.15.16:0/766318047 shutdown_connections 2023-10-16T13:28:17.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:17.303+0000 7f7f304d8700 1 -- 172.21.15.16:0/766318047 wait complete. 2023-10-16T13:28:17.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:17.304+0000 7f7f304d8700 1 Processor -- start 2023-10-16T13:28:17.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:17.304+0000 7f7f304d8700 1 -- start start 2023-10-16T13:28:17.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:17.305+0000 7f7f304d8700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7f28157b30 0x7f7f280b0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:17.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:17.305+0000 7f7f304d8700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7f28157f10 0x7f7f28148f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:17.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:17.305+0000 7f7f304d8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f28161610 0x7f7f281f6680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:17.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:17.305+0000 7f7f304d8700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f281f7600 con 0x7f7f28161610 2023-10-16T13:28:17.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:17.305+0000 7f7f304d8700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f28148b70 con 0x7f7f28157f10 2023-10-16T13:28:17.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:17.305+0000 7f7f304d8700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f280c78d0 con 0x7f7f28157b30 2023-10-16T13:28:17.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:17.306+0000 7f7f2ea75700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f28161610 0x7f7f281f6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:17.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:17.306+0000 7f7f2ea75700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f28161610 0x7f7f281f6680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49772/0 (socket says 172.21.15.16:49772) 2023-10-16T13:28:17.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:17.306+0000 7f7f2ea75700 1 -- 172.21.15.16:0/1064389696 learned_addr learned my addr 172.21.15.16:0/1064389696 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:17.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:17.306+0000 7f7f2e274700 1 --2- 172.21.15.16:0/1064389696 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7f28157b30 0x7f7f280b0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:17.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:17.306+0000 7f7f2da73700 1 --2- 172.21.15.16:0/1064389696 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7f28157f10 0x7f7f28148f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:17.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:17.306+0000 7f7f2ea75700 1 -- 172.21.15.16:0/1064389696 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7f28157b30 msgr2=0x7f7f280b0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:17.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:17.306+0000 7f7f2ea75700 1 --2- 172.21.15.16:0/1064389696 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7f28157b30 0x7f7f280b0650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:17.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:17.306+0000 7f7f2ea75700 1 -- 172.21.15.16:0/1064389696 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7f28157f10 msgr2=0x7f7f28148f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:17.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:17.306+0000 7f7f2ea75700 1 --2- 172.21.15.16:0/1064389696 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7f28157f10 0x7f7f28148f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:17.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:17.306+0000 7f7f2ea75700 1 -- 172.21.15.16:0/1064389696 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f18009550 con 0x7f7f28161610 2023-10-16T13:28:17.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:17.306+0000 7f7f2e274700 1 --2- 172.21.15.16:0/1064389696 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7f28157b30 0x7f7f280b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:28:17.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:17.306+0000 7f7f2da73700 1 --2- 172.21.15.16:0/1064389696 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7f28157f10 0x7f7f28148f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:28:17.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:17.306+0000 7f7f2ea75700 1 --2- 172.21.15.16:0/1064389696 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f28161610 0x7f7f281f6680 secure :-1 s=READY pgs=2075 cs=0 l=1 rev1=1 crypto rx=0x7f7f20001560 tx=0x7f7f2000cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:17.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:17.306+0000 7f7f1effd700 1 -- 172.21.15.16:0/1064389696 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f20003e30 con 0x7f7f28161610 2023-10-16T13:28:17.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:17.306+0000 7f7f1effd700 1 -- 172.21.15.16:0/1064389696 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f20005070 con 0x7f7f28161610 2023-10-16T13:28:17.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:17.306+0000 7f7f304d8700 1 -- 172.21.15.16:0/1064389696 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f280b0b90 con 0x7f7f28161610 2023-10-16T13:28:17.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:17.306+0000 7f7f1effd700 1 -- 172.21.15.16:0/1064389696 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f20005420 con 0x7f7f28161610 2023-10-16T13:28:17.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:17.306+0000 7f7f304d8700 1 -- 172.21.15.16:0/1064389696 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f281f7320 con 0x7f7f28161610 2023-10-16T13:28:17.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:17.307+0000 7f7f304d8700 1 -- 172.21.15.16:0/1064389696 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f28148cb0 con 0x7f7f28161610 2023-10-16T13:28:17.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:17.308+0000 7f7f1effd700 1 -- 172.21.15.16:0/1064389696 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 107) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7f20005580 con 0x7f7f28161610 2023-10-16T13:28:17.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:17.308+0000 7f7f1effd700 1 --2- 172.21.15.16:0/1064389696 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f7f0c06b020 0x7f7f0c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:17.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:17.308+0000 7f7f1effd700 1 -- 172.21.15.16:0/1064389696 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 5277+0+0 (secure 0 0 0) 0x7f7f20088c90 con 0x7f7f28161610 2023-10-16T13:28:17.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:17.308+0000 7f7f2e274700 1 --2- 172.21.15.16:0/1064389696 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f7f0c06b020 0x7f7f0c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:17.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:17.309+0000 7f7f2e274700 1 --2- 172.21.15.16:0/1064389696 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f7f0c06b020 0x7f7f0c06d4e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7f18014040 tx=0x7f7f18008030 comp rx=0 tx=0).ready entity=mgr.10047 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:17.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:17.313+0000 7f7f1effd700 1 -- 172.21.15.16:0/1064389696 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7f28148cb0 con 0x7f7f28161610 2023-10-16T13:28:17.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:17.515+0000 7f7f304d8700 1 -- 172.21.15.16:0/1064389696 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f7f28148cb0 con 0x7f7f28161610 2023-10-16T13:28:18.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.081+0000 7f7f1effd700 1 -- 172.21.15.16:0/1064389696 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v187) v1 ==== 138+0+0 (secure 0 0 0) 0x7f7f28148cb0 con 0x7f7f28161610 2023-10-16T13:28:18.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.086+0000 7f7f304d8700 1 -- 172.21.15.16:0/1064389696 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f7f0c06b020 msgr2=0x7f7f0c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:18.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.086+0000 7f7f304d8700 1 --2- 172.21.15.16:0/1064389696 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f7f0c06b020 0x7f7f0c06d4e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7f18014040 tx=0x7f7f18008030 comp rx=0 tx=0).stop 2023-10-16T13:28:18.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.086+0000 7f7f304d8700 1 -- 172.21.15.16:0/1064389696 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f28161610 msgr2=0x7f7f281f6680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:18.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.086+0000 7f7f304d8700 1 --2- 172.21.15.16:0/1064389696 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f28161610 0x7f7f281f6680 secure :-1 s=READY pgs=2075 cs=0 l=1 rev1=1 crypto rx=0x7f7f20001560 tx=0x7f7f2000cc00 comp rx=0 tx=0).stop 2023-10-16T13:28:18.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.089+0000 7f7f304d8700 1 -- 172.21.15.16:0/1064389696 shutdown_connections 2023-10-16T13:28:18.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.089+0000 7f7f304d8700 1 --2- 172.21.15.16:0/1064389696 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f28161610 0x7f7f281f6680 unknown :-1 s=CLOSED pgs=2075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:18.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.089+0000 7f7f304d8700 1 --2- 172.21.15.16:0/1064389696 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7f28157b30 0x7f7f280b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:18.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.089+0000 7f7f304d8700 1 --2- 172.21.15.16:0/1064389696 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f7f0c06b020 0x7f7f0c06d4e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:18.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.089+0000 7f7f304d8700 1 --2- 172.21.15.16:0/1064389696 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7f28157f10 0x7f7f28148f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:18.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.089+0000 7f7f304d8700 1 -- 172.21.15.16:0/1064389696 >> 172.21.15.16:0/1064389696 conn(0x7f7f2806a7e0 msgr2=0x7f7f280b2200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:18.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.091+0000 7f7f304d8700 1 -- 172.21.15.16:0/1064389696 shutdown_connections 2023-10-16T13:28:18.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.091+0000 7f7f304d8700 1 -- 172.21.15.16:0/1064389696 wait complete. 2023-10-16T13:28:18.095 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' created 2023-10-16T13:28:18.109 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-16T13:28:18.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.245+0000 7f958feb0700 1 Processor -- start 2023-10-16T13:28:18.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.246+0000 7f958feb0700 1 -- start start 2023-10-16T13:28:18.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.246+0000 7f958feb0700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f9588072b30 0x7f9588071a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:18.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.246+0000 7f958feb0700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f95880caa40 0x7f95880cae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:18.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.246+0000 7f958feb0700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95880cb3d0 con 0x7f958815bb40 2023-10-16T13:28:18.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.247+0000 7f958feb0700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f958806ea20 con 0x7f9588072b30 2023-10-16T13:28:18.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.247+0000 7f958feb0700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f958815c040 con 0x7f95880caa40 2023-10-16T13:28:18.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.247+0000 7f958e44d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f95880caa40 0x7f95880cae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:18.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.247+0000 7f958dc4c700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f9588072b30 0x7f9588071a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:18.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.247+0000 7f958e44d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f95880caa40 0x7f95880cae90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:38038/0 (socket says 172.21.15.16:38038) 2023-10-16T13:28:18.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.247+0000 7f958dc4c700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f9588072b30 0x7f9588071a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:48838/0 (socket says 172.21.15.16:48838) 2023-10-16T13:28:18.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.247+0000 7f958e44d700 1 -- 172.21.15.16:0/3170132004 learned_addr learned my addr 172.21.15.16:0/3170132004 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:18.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.247+0000 7f958d44b700 1 --1- 172.21.15.16:0/3170132004 >> v1:172.21.15.49:6789/0 conn(0x7f958815bb40 0x7f958806e310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:46232/0 (socket says 172.21.15.16:46232) 2023-10-16T13:28:18.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.247+0000 7f958e44d700 1 -- 172.21.15.16:0/3170132004 >> v2:172.21.15.49:3300/0 conn(0x7f9588072b30 msgr2=0x7f9588071a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:28:18.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.247+0000 7f958e44d700 1 --2- 172.21.15.16:0/3170132004 >> v2:172.21.15.49:3300/0 conn(0x7f9588072b30 0x7f9588071a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:18.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.247+0000 7f958e44d700 1 -- 172.21.15.16:0/3170132004 >> v1:172.21.15.49:6789/0 conn(0x7f958815bb40 legacy=0x7f958806e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:18.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.247+0000 7f958e44d700 1 -- 172.21.15.16:0/3170132004 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f958815c320 con 0x7f95880caa40 2023-10-16T13:28:18.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.247+0000 7f958dc4c700 1 --2- 172.21.15.16:0/3170132004 >> v2:172.21.15.49:3300/0 conn(0x7f9588072b30 0x7f9588071a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:28:18.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.247+0000 7f958e44d700 1 --2- 172.21.15.16:0/3170132004 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f95880caa40 0x7f95880cae90 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f9580009e50 tx=0x7f958000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ed306b0257059c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:18.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.248+0000 7f958cc4a700 1 -- 172.21.15.16:0/3170132004 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f958000e070 con 0x7f95880caa40 2023-10-16T13:28:18.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.248+0000 7f958cc4a700 1 -- 172.21.15.16:0/3170132004 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f958000b910 con 0x7f95880caa40 2023-10-16T13:28:18.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.248+0000 7f958cc4a700 1 -- 172.21.15.16:0/3170132004 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f958000bcc0 con 0x7f95880caa40 2023-10-16T13:28:18.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.248+0000 7f958feb0700 1 -- 172.21.15.16:0/3170132004 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f95880caa40 msgr2=0x7f95880cae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:18.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.248+0000 7f958feb0700 1 --2- 172.21.15.16:0/3170132004 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f95880caa40 0x7f95880cae90 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f9580009e50 tx=0x7f958000b330 comp rx=0 tx=0).stop 2023-10-16T13:28:18.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.248+0000 7f958feb0700 1 -- 172.21.15.16:0/3170132004 shutdown_connections 2023-10-16T13:28:18.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.248+0000 7f958feb0700 1 --2- 172.21.15.16:0/3170132004 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f95880caa40 0x7f95880cae90 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:18.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.248+0000 7f958feb0700 1 --2- 172.21.15.16:0/3170132004 >> v2:172.21.15.49:3300/0 conn(0x7f9588072b30 0x7f9588071a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:18.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.248+0000 7f958feb0700 1 -- 172.21.15.16:0/3170132004 >> 172.21.15.16:0/3170132004 conn(0x7f95880b1f40 msgr2=0x7f95880b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:18.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.248+0000 7f958feb0700 1 -- 172.21.15.16:0/3170132004 shutdown_connections 2023-10-16T13:28:18.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.248+0000 7f958feb0700 1 -- 172.21.15.16:0/3170132004 wait complete. 2023-10-16T13:28:18.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.249+0000 7f958feb0700 1 Processor -- start 2023-10-16T13:28:18.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.249+0000 7f958feb0700 1 -- start start 2023-10-16T13:28:18.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.250+0000 7f958feb0700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9588072b30 0x7f9588157d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:18.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.250+0000 7f958feb0700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f95880caa40 0x7f9588158240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:18.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.250+0000 7f958feb0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f958815bb40 0x7f95880c9a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:18.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.250+0000 7f958feb0700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95881527d0 con 0x7f958815bb40 2023-10-16T13:28:18.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.250+0000 7f958feb0700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95880b3be0 con 0x7f95880caa40 2023-10-16T13:28:18.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.250+0000 7f958feb0700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95880b3ec0 con 0x7f9588072b30 2023-10-16T13:28:18.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.250+0000 7f958e44d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f958815bb40 0x7f95880c9a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:18.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.250+0000 7f958e44d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f958815bb40 0x7f95880c9a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49786/0 (socket says 172.21.15.16:49786) 2023-10-16T13:28:18.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.250+0000 7f958e44d700 1 -- 172.21.15.16:0/3709033131 learned_addr learned my addr 172.21.15.16:0/3709033131 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:18.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.251+0000 7f958dc4c700 1 --2- 172.21.15.16:0/3709033131 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9588072b30 0x7f9588157d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:18.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.251+0000 7f958d44b700 1 --2- 172.21.15.16:0/3709033131 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f95880caa40 0x7f9588158240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:18.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.251+0000 7f958e44d700 1 -- 172.21.15.16:0/3709033131 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9588072b30 msgr2=0x7f9588157d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:18.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.251+0000 7f958e44d700 1 --2- 172.21.15.16:0/3709033131 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9588072b30 0x7f9588157d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:18.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.251+0000 7f958e44d700 1 -- 172.21.15.16:0/3709033131 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f95880caa40 msgr2=0x7f9588158240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:18.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.251+0000 7f958e44d700 1 --2- 172.21.15.16:0/3709033131 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f95880caa40 0x7f9588158240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:18.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.251+0000 7f958e44d700 1 -- 172.21.15.16:0/3709033131 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95780094a0 con 0x7f958815bb40 2023-10-16T13:28:18.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.251+0000 7f958d44b700 1 --2- 172.21.15.16:0/3709033131 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f95880caa40 0x7f9588158240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:28:18.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.251+0000 7f958e44d700 1 --2- 172.21.15.16:0/3709033131 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f958815bb40 0x7f95880c9a30 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 crypto rx=0x7f958806eb60 tx=0x7f9580004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:18.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.251+0000 7f957e7fc700 1 -- 172.21.15.16:0/3709033131 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9580004960 con 0x7f958815bb40 2023-10-16T13:28:18.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.251+0000 7f957e7fc700 1 -- 172.21.15.16:0/3709033131 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95800099b0 con 0x7f958815bb40 2023-10-16T13:28:18.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.251+0000 7f958feb0700 1 -- 172.21.15.16:0/3709033131 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9580009b30 con 0x7f958815bb40 2023-10-16T13:28:18.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.251+0000 7f957e7fc700 1 -- 172.21.15.16:0/3709033131 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f958001b670 con 0x7f958815bb40 2023-10-16T13:28:18.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.251+0000 7f958feb0700 1 -- 172.21.15.16:0/3709033131 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95880b4480 con 0x7f958815bb40 2023-10-16T13:28:18.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.252+0000 7f958feb0700 1 -- 172.21.15.16:0/3709033131 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f958804ebb0 con 0x7f958815bb40 2023-10-16T13:28:18.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.253+0000 7f957e7fc700 1 -- 172.21.15.16:0/3709033131 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 107) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f958001b7d0 con 0x7f958815bb40 2023-10-16T13:28:18.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.253+0000 7f957e7fc700 1 --2- 172.21.15.16:0/3709033131 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f956c06b020 0x7f956c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:18.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.253+0000 7f957e7fc700 1 -- 172.21.15.16:0/3709033131 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 5647+0+0 (secure 0 0 0) 0x7f958008b680 con 0x7f958815bb40 2023-10-16T13:28:18.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.253+0000 7f958dc4c700 1 --2- 172.21.15.16:0/3709033131 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f956c06b020 0x7f956c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:18.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.254+0000 7f958dc4c700 1 --2- 172.21.15.16:0/3709033131 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f956c06b020 0x7f956c06d4e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9578009c60 tx=0x7f9578008040 comp rx=0 tx=0).ready entity=mgr.10047 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:18.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.257+0000 7f957e7fc700 1 -- 172.21.15.16:0/3709033131 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9580056540 con 0x7f958815bb40 2023-10-16T13:28:18.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:18.459+0000 7f958feb0700 1 -- 172.21.15.16:0/3709033131 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f95880b2d40 con 0x7f958815bb40 2023-10-16T13:28:19.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:19.083+0000 7f957e7fc700 1 -- 172.21.15.16:0/3709033131 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v188) v1 ==== 170+0+0 (secure 0 0 0) 0x7f9580059b60 con 0x7f958815bb40 2023-10-16T13:28:19.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:19.090+0000 7f958feb0700 1 -- 172.21.15.16:0/3709033131 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f956c06b020 msgr2=0x7f956c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:19.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:19.090+0000 7f958feb0700 1 --2- 172.21.15.16:0/3709033131 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f956c06b020 0x7f956c06d4e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9578009c60 tx=0x7f9578008040 comp rx=0 tx=0).stop 2023-10-16T13:28:19.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:19.090+0000 7f958feb0700 1 -- 172.21.15.16:0/3709033131 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f958815bb40 msgr2=0x7f95880c9a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:19.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:19.090+0000 7f958feb0700 1 --2- 172.21.15.16:0/3709033131 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f958815bb40 0x7f95880c9a30 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 crypto rx=0x7f958806eb60 tx=0x7f9580004ad0 comp rx=0 tx=0).stop 2023-10-16T13:28:19.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:19.091+0000 7f958feb0700 1 -- 172.21.15.16:0/3709033131 shutdown_connections 2023-10-16T13:28:19.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:19.091+0000 7f958feb0700 1 --2- 172.21.15.16:0/3709033131 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f958815bb40 0x7f95880c9a30 unknown :-1 s=CLOSED pgs=2076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:19.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:19.091+0000 7f958feb0700 1 --2- 172.21.15.16:0/3709033131 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9588072b30 0x7f9588157d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:19.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:19.091+0000 7f958feb0700 1 --2- 172.21.15.16:0/3709033131 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f956c06b020 0x7f956c06d4e0 secure :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9578009c60 tx=0x7f9578008040 comp rx=0 tx=0).stop 2023-10-16T13:28:19.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:19.091+0000 7f958feb0700 1 --2- 172.21.15.16:0/3709033131 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f95880caa40 0x7f9588158240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:19.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:19.091+0000 7f958feb0700 1 -- 172.21.15.16:0/3709033131 >> 172.21.15.16:0/3709033131 conn(0x7f95880b1f40 msgr2=0x7f95880d0380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:19.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:19.093+0000 7f958feb0700 1 -- 172.21.15.16:0/3709033131 shutdown_connections 2023-10-16T13:28:19.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:19.105+0000 7f958feb0700 1 -- 172.21.15.16:0/3709033131 wait complete. 2023-10-16T13:28:19.111 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' created 2023-10-16T13:28:19.125 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-16T13:28:19.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:19.255+0000 7f909cc07700 1 Processor -- start 2023-10-16T13:28:19.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:19.255+0000 7f909cc07700 1 -- start start 2023-10-16T13:28:19.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:19.256+0000 7f909cc07700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f909806a7e0 0x7f909806ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:19.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:19.256+0000 7f909cc07700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f909815ecb0 con 0x7f909806e570 2023-10-16T13:28:19.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:19.256+0000 7f909cc07700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f909815b6a0 con 0x7f909815b2c0 2023-10-16T13:28:19.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:19.256+0000 7f909cc07700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f909806b190 con 0x7f909806a7e0 2023-10-16T13:28:19.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:19.256+0000 7f9095d9b700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f909806e570 0x7f909815e5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:39066/0 (socket says 172.21.15.16:39066) 2023-10-16T13:28:19.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:19.256+0000 7f9095d9b700 1 -- 172.21.15.16:0/2423272329 learned_addr learned my addr 172.21.15.16:0/2423272329 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:28:19.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:19.256+0000 7f9096d9d700 1 --2- 172.21.15.16:0/2423272329 >> v2:172.21.15.16:3300/0 conn(0x7f909806a7e0 0x7f909806ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:19.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:19.256+0000 7f9096d9d700 1 -- 172.21.15.16:0/2423272329 >> v1:172.21.15.49:6789/0 conn(0x7f909815b2c0 legacy=0x7f9098161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:19.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:19.256+0000 7f9096d9d700 1 -- 172.21.15.16:0/2423272329 >> v1:172.21.15.16:6789/0 conn(0x7f909806e570 legacy=0x7f909815e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:19.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:19.256+0000 7f9096d9d700 1 -- 172.21.15.16:0/2423272329 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9098161b20 con 0x7f909806a7e0 2023-10-16T13:28:19.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:19.257+0000 7f9096d9d700 1 --2- 172.21.15.16:0/2423272329 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f909806a7e0 0x7f909806ac50 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 crypto rx=0x7f9090009d90 tx=0x7f909000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28f1b8b78765f8d7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:19.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:19.257+0000 7f909559a700 1 -- 172.21.15.16:0/2423272329 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f909000e070 con 0x7f909806a7e0 2023-10-16T13:28:19.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:19.257+0000 7f909559a700 1 -- 172.21.15.16:0/2423272329 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f909000bbd0 con 0x7f909806a7e0 2023-10-16T13:28:19.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:19.257+0000 7f909559a700 1 -- 172.21.15.16:0/2423272329 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9090012430 con 0x7f909806a7e0 2023-10-16T13:28:19.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:19.257+0000 7f909cc07700 1 -- 172.21.15.16:0/2423272329 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f909806a7e0 msgr2=0x7f909806ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:19.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:19.257+0000 7f909cc07700 1 --2- 172.21.15.16:0/2423272329 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f909806a7e0 0x7f909806ac50 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 crypto rx=0x7f9090009d90 tx=0x7f909000b330 comp rx=0 tx=0).stop 2023-10-16T13:28:19.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:19.257+0000 7f909cc07700 1 -- 172.21.15.16:0/2423272329 shutdown_connections 2023-10-16T13:28:19.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:19.257+0000 7f909cc07700 1 --2- 172.21.15.16:0/2423272329 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f909806a7e0 0x7f909806ac50 unknown :-1 s=CLOSED pgs=2078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:19.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:19.257+0000 7f909cc07700 1 -- 172.21.15.16:0/2423272329 >> 172.21.15.16:0/2423272329 conn(0x7f90980b1f40 msgr2=0x7f90980b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:19.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:19.257+0000 7f909cc07700 1 -- 172.21.15.16:0/2423272329 shutdown_connections 2023-10-16T13:28:19.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:19.257+0000 7f909cc07700 1 -- 172.21.15.16:0/2423272329 wait complete. 2023-10-16T13:28:19.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:19.259+0000 7f909cc07700 1 Processor -- start 2023-10-16T13:28:19.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:19.259+0000 7f909cc07700 1 -- start start 2023-10-16T13:28:19.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:19.259+0000 7f909cc07700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f909806a7e0 0x7f90980c3b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:19.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:19.259+0000 7f909cc07700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f909806e570 0x7f909814b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:19.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:19.259+0000 7f909cc07700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f909815b2c0 0x7f909814bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:19.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:19.259+0000 7f909cc07700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90980b3f60 con 0x7f909815b2c0 2023-10-16T13:28:19.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:19.259+0000 7f909cc07700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90980c66d0 con 0x7f909806a7e0 2023-10-16T13:28:19.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:19.259+0000 7f909cc07700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90980c69b0 con 0x7f909806e570 2023-10-16T13:28:19.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:19.259+0000 7f9096d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f909815b2c0 0x7f909814bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:19.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:19.260+0000 7f9096d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f909815b2c0 0x7f909814bd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49800/0 (socket says 172.21.15.16:49800) 2023-10-16T13:28:19.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:19.260+0000 7f9096d9d700 1 -- 172.21.15.16:0/1307162605 learned_addr learned my addr 172.21.15.16:0/1307162605 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:19.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:19.260+0000 7f9095d9b700 1 --2- 172.21.15.16:0/1307162605 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f909806e570 0x7f909814b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:19.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:19.260+0000 7f909659c700 1 --2- 172.21.15.16:0/1307162605 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f909806a7e0 0x7f90980c3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:19.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:19.260+0000 7f9096d9d700 1 -- 172.21.15.16:0/1307162605 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f909806e570 msgr2=0x7f909814b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:19.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:19.260+0000 7f9096d9d700 1 --2- 172.21.15.16:0/1307162605 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f909806e570 0x7f909814b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:19.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:19.260+0000 7f9096d9d700 1 -- 172.21.15.16:0/1307162605 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f909806a7e0 msgr2=0x7f90980c3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:19.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:19.260+0000 7f9096d9d700 1 --2- 172.21.15.16:0/1307162605 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f909806a7e0 0x7f90980c3b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:19.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:19.260+0000 7f9096d9d700 1 -- 172.21.15.16:0/1307162605 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9090009a70 con 0x7f909815b2c0 2023-10-16T13:28:19.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:19.260+0000 7f9096d9d700 1 --2- 172.21.15.16:0/1307162605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f909815b2c0 0x7f909814bd10 secure :-1 s=READY pgs=2079 cs=0 l=1 rev1=1 crypto rx=0x7f9090014040 tx=0x7f9090004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:19.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:19.260+0000 7f9086ffd700 1 -- 172.21.15.16:0/1307162605 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90900098f0 con 0x7f909815b2c0 2023-10-16T13:28:19.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:19.260+0000 7f9086ffd700 1 -- 172.21.15.16:0/1307162605 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9090020070 con 0x7f909815b2c0 2023-10-16T13:28:19.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:19.260+0000 7f9086ffd700 1 -- 172.21.15.16:0/1307162605 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9090004280 con 0x7f909815b2c0 2023-10-16T13:28:19.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:19.260+0000 7f909cc07700 1 -- 172.21.15.16:0/1307162605 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90980c6c90 con 0x7f909815b2c0 2023-10-16T13:28:19.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:19.260+0000 7f909cc07700 1 -- 172.21.15.16:0/1307162605 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90980c62c0 con 0x7f909815b2c0 2023-10-16T13:28:19.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:19.261+0000 7f909cc07700 1 -- 172.21.15.16:0/1307162605 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90980c6480 con 0x7f909815b2c0 2023-10-16T13:28:19.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:19.262+0000 7f9086ffd700 1 -- 172.21.15.16:0/1307162605 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 107) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f90900043e0 con 0x7f909815b2c0 2023-10-16T13:28:19.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:19.262+0000 7f9086ffd700 1 --2- 172.21.15.16:0/1307162605 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f907806b020 0x7f907806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:19.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:19.262+0000 7f9086ffd700 1 -- 172.21.15.16:0/1307162605 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f9090022070 con 0x7f909815b2c0 2023-10-16T13:28:19.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:19.262+0000 7f909659c700 1 --2- 172.21.15.16:0/1307162605 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f907806b020 0x7f907806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:19.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:19.263+0000 7f909659c700 1 --2- 172.21.15.16:0/1307162605 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f907806b020 0x7f907806d4e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f908c000f60 tx=0x7f908c009040 comp rx=0 tx=0).ready entity=mgr.10047 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:19.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:19.267+0000 7f9086ffd700 1 -- 172.21.15.16:0/1307162605 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9090056630 con 0x7f909815b2c0 2023-10-16T13:28:19.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:19.522+0000 7f909cc07700 1 -- 172.21.15.16:0/1307162605 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f90980b4180 con 0x7f909815b2c0 2023-10-16T13:28:20.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.098+0000 7f9086ffd700 1 -- 172.21.15.16:0/1307162605 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 23 and data pool 24 v91) v1 ==== 174+0+0 (secure 0 0 0) 0x7f9090059c50 con 0x7f909815b2c0 2023-10-16T13:28:20.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.110+0000 7f909cc07700 1 -- 172.21.15.16:0/1307162605 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f907806b020 msgr2=0x7f907806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:20.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.110+0000 7f909cc07700 1 --2- 172.21.15.16:0/1307162605 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f907806b020 0x7f907806d4e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f908c000f60 tx=0x7f908c009040 comp rx=0 tx=0).stop 2023-10-16T13:28:20.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.110+0000 7f909cc07700 1 -- 172.21.15.16:0/1307162605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f909815b2c0 msgr2=0x7f909814bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:20.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.110+0000 7f909cc07700 1 --2- 172.21.15.16:0/1307162605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f909815b2c0 0x7f909814bd10 secure :-1 s=READY pgs=2079 cs=0 l=1 rev1=1 crypto rx=0x7f9090014040 tx=0x7f9090004ad0 comp rx=0 tx=0).stop 2023-10-16T13:28:20.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.110+0000 7f909cc07700 1 -- 172.21.15.16:0/1307162605 shutdown_connections 2023-10-16T13:28:20.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.110+0000 7f909cc07700 1 --2- 172.21.15.16:0/1307162605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f909815b2c0 0x7f909814bd10 unknown :-1 s=CLOSED pgs=2079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:20.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.110+0000 7f909cc07700 1 --2- 172.21.15.16:0/1307162605 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f907806b020 0x7f907806d4e0 secure :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f908c000f60 tx=0x7f908c009040 comp rx=0 tx=0).stop 2023-10-16T13:28:20.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.110+0000 7f909cc07700 1 --2- 172.21.15.16:0/1307162605 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f909806a7e0 0x7f90980c3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:20.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.110+0000 7f909cc07700 1 --2- 172.21.15.16:0/1307162605 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f909806e570 0x7f909814b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:20.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.110+0000 7f909cc07700 1 -- 172.21.15.16:0/1307162605 >> 172.21.15.16:0/1307162605 conn(0x7f90980b1f40 msgr2=0x7f90981612d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:20.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.112+0000 7f909cc07700 1 -- 172.21.15.16:0/1307162605 shutdown_connections 2023-10-16T13:28:20.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.112+0000 7f909cc07700 1 -- 172.21.15.16:0/1307162605 wait complete. 2023-10-16T13:28:20.124 INFO:teuthology.orchestra.run.smithi016.stderr:new fs with metadata pool 23 and data pool 24 2023-10-16T13:28:20.138 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:28:20.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.267+0000 7f585272e700 1 Processor -- start 2023-10-16T13:28:20.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.267+0000 7f585272e700 1 -- start start 2023-10-16T13:28:20.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.268+0000 7f585272e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f584c0b3690 0x7f584c06bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:20.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.268+0000 7f585272e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f584c06c210 0x7f584c0cc8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:20.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.268+0000 7f585272e700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f584c06ccc0 con 0x7f584c15da40 2023-10-16T13:28:20.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.268+0000 7f585272e700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f584c0cc4d0 con 0x7f584c06c210 2023-10-16T13:28:20.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.268+0000 7f585272e700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f584c0cbb70 con 0x7f584c0b3690 2023-10-16T13:28:20.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.268+0000 7f5850ccb700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f584c06c210 0x7f584c0cc8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:20.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.268+0000 7f5850ccb700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f584c06c210 0x7f584c0cc8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49820/0 (socket says 172.21.15.16:49820) 2023-10-16T13:28:20.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.268+0000 7f5850ccb700 1 -- 172.21.15.16:0/1328201717 learned_addr learned my addr 172.21.15.16:0/1328201717 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:20.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.268+0000 7f584bfff700 1 --1- 172.21.15.16:0/1328201717 >> v1:172.21.15.16:6789/0 conn(0x7f584c15da40 0x7f584c0b2f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:39090/0 (socket says 172.21.15.16:39090) 2023-10-16T13:28:20.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.268+0000 7f584b7fe700 1 --2- 172.21.15.16:0/1328201717 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f584c0b3690 0x7f584c06bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:20.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.269+0000 7f5850ccb700 1 -- 172.21.15.16:0/1328201717 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f584c0b3690 msgr2=0x7f584c06bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:28:20.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.269+0000 7f5850ccb700 1 --2- 172.21.15.16:0/1328201717 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f584c0b3690 0x7f584c06bcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:20.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.269+0000 7f5850ccb700 1 -- 172.21.15.16:0/1328201717 >> v1:172.21.15.16:6789/0 conn(0x7f584c15da40 legacy=0x7f584c0b2f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:20.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.269+0000 7f5850ccb700 1 -- 172.21.15.16:0/1328201717 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f584c06c9e0 con 0x7f584c06c210 2023-10-16T13:28:20.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.269+0000 7f584affd700 1 -- 172.21.15.16:0/1328201717 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 74848986 0 0) 0x7f584c06ccc0 con 0x7f584c15da40 2023-10-16T13:28:20.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.269+0000 7f5850ccb700 1 --2- 172.21.15.16:0/1328201717 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f584c06c210 0x7f584c0cc8d0 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 crypto rx=0x7f5844009e50 tx=0x7f584400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3a40a6cbec33de8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:20.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.269+0000 7f584affd700 1 -- 172.21.15.16:0/1328201717 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f584400e070 con 0x7f584c06c210 2023-10-16T13:28:20.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.269+0000 7f584affd700 1 -- 172.21.15.16:0/1328201717 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f584400b910 con 0x7f584c06c210 2023-10-16T13:28:20.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.269+0000 7f584affd700 1 -- 172.21.15.16:0/1328201717 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f584400bcc0 con 0x7f584c06c210 2023-10-16T13:28:20.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.269+0000 7f585272e700 1 -- 172.21.15.16:0/1328201717 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f584c06c210 msgr2=0x7f584c0cc8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:20.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.269+0000 7f585272e700 1 --2- 172.21.15.16:0/1328201717 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f584c06c210 0x7f584c0cc8d0 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 crypto rx=0x7f5844009e50 tx=0x7f584400b330 comp rx=0 tx=0).stop 2023-10-16T13:28:20.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.270+0000 7f585272e700 1 -- 172.21.15.16:0/1328201717 shutdown_connections 2023-10-16T13:28:20.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.270+0000 7f585272e700 1 --2- 172.21.15.16:0/1328201717 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f584c06c210 0x7f584c0cc8d0 unknown :-1 s=CLOSED pgs=2081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:20.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.270+0000 7f585272e700 1 --2- 172.21.15.16:0/1328201717 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f584c0b3690 0x7f584c06bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:20.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.270+0000 7f585272e700 1 -- 172.21.15.16:0/1328201717 >> 172.21.15.16:0/1328201717 conn(0x7f584c0cdf80 msgr2=0x7f584c0ce390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:20.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.270+0000 7f585272e700 1 -- 172.21.15.16:0/1328201717 shutdown_connections 2023-10-16T13:28:20.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.270+0000 7f585272e700 1 -- 172.21.15.16:0/1328201717 wait complete. 2023-10-16T13:28:20.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.271+0000 7f585272e700 1 Processor -- start 2023-10-16T13:28:20.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.271+0000 7f585272e700 1 -- start start 2023-10-16T13:28:20.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.272+0000 7f585272e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f584c06c210 0x7f584c1f2b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:20.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.272+0000 7f585272e700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f584c0b3690 0x7f584c1565d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:20.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.272+0000 7f585272e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f584c15da40 0x7f584c153840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:20.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.272+0000 7f585272e700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f584c0ca740 con 0x7f584c15da40 2023-10-16T13:28:20.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.272+0000 7f585272e700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f584c0c9b30 con 0x7f584c0b3690 2023-10-16T13:28:20.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.272+0000 7f585272e700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f584c0c9e10 con 0x7f584c06c210 2023-10-16T13:28:20.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.272+0000 7f5850ccb700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f584c15da40 0x7f584c153840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:20.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.272+0000 7f5850ccb700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f584c15da40 0x7f584c153840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49826/0 (socket says 172.21.15.16:49826) 2023-10-16T13:28:20.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.272+0000 7f5850ccb700 1 -- 172.21.15.16:0/3201709482 learned_addr learned my addr 172.21.15.16:0/3201709482 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:20.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.272+0000 7f584bfff700 1 --2- 172.21.15.16:0/3201709482 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f584c06c210 0x7f584c1f2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:20.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.272+0000 7f584b7fe700 1 --2- 172.21.15.16:0/3201709482 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f584c0b3690 0x7f584c1565d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:20.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.272+0000 7f5850ccb700 1 -- 172.21.15.16:0/3201709482 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f584c06c210 msgr2=0x7f584c1f2b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:20.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.272+0000 7f5850ccb700 1 --2- 172.21.15.16:0/3201709482 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f584c06c210 0x7f584c1f2b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:20.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.272+0000 7f5850ccb700 1 -- 172.21.15.16:0/3201709482 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f584c0b3690 msgr2=0x7f584c1565d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:20.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.272+0000 7f5850ccb700 1 --2- 172.21.15.16:0/3201709482 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f584c0b3690 0x7f584c1565d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:20.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.272+0000 7f5850ccb700 1 -- 172.21.15.16:0/3201709482 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5844009b30 con 0x7f584c15da40 2023-10-16T13:28:20.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.272+0000 7f5850ccb700 1 --2- 172.21.15.16:0/3201709482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f584c15da40 0x7f584c153840 secure :-1 s=READY pgs=2082 cs=0 l=1 rev1=1 crypto rx=0x7f584400b680 tx=0x7f5844004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:20.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.272+0000 7f5848ff9700 1 -- 172.21.15.16:0/3201709482 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58440048b0 con 0x7f584c15da40 2023-10-16T13:28:20.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.273+0000 7f5848ff9700 1 -- 172.21.15.16:0/3201709482 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58440099b0 con 0x7f584c15da40 2023-10-16T13:28:20.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.273+0000 7f585272e700 1 -- 172.21.15.16:0/3201709482 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f584c0ca0f0 con 0x7f584c15da40 2023-10-16T13:28:20.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.273+0000 7f5848ff9700 1 -- 172.21.15.16:0/3201709482 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f584401b670 con 0x7f584c15da40 2023-10-16T13:28:20.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.273+0000 7f585272e700 1 -- 172.21.15.16:0/3201709482 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f584c1f3060 con 0x7f584c15da40 2023-10-16T13:28:20.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.274+0000 7f58427fc700 1 -- 172.21.15.16:0/3201709482 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58240052f0 con 0x7f584c15da40 2023-10-16T13:28:20.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.274+0000 7f5848ff9700 1 -- 172.21.15.16:0/3201709482 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 107) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f584401b7d0 con 0x7f584c15da40 2023-10-16T13:28:20.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.274+0000 7f5848ff9700 1 --2- 172.21.15.16:0/3201709482 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f582c06b020 0x7f582c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:20.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.275+0000 7f5848ff9700 1 -- 172.21.15.16:0/3201709482 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f584408aed0 con 0x7f584c15da40 2023-10-16T13:28:20.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.275+0000 7f584bfff700 1 --2- 172.21.15.16:0/3201709482 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f582c06b020 0x7f582c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:20.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.275+0000 7f584bfff700 1 --2- 172.21.15.16:0/3201709482 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f582c06b020 0x7f582c06d4e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5834007fd0 tx=0x7f5834007de0 comp rx=0 tx=0).ready entity=mgr.10047 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:20.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.279+0000 7f5848ff9700 1 -- 172.21.15.16:0/3201709482 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5844055c40 con 0x7f584c15da40 2023-10-16T13:28:20.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.483+0000 7f58427fc700 1 -- 172.21.15.16:0/3201709482 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5824005870 con 0x7f584c15da40 2023-10-16T13:28:20.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.484+0000 7f5848ff9700 1 -- 172.21.15.16:0/3201709482 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v189) v1 ==== 74+0+16780 (secure 0 0 0) 0x7f584401f090 con 0x7f584c15da40 2023-10-16T13:28:20.487 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:28:20.487 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":189,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:28:19.523314+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2023-10-16T13:28:17.549815+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"189","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2023-10-16T13:28:18.493327+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"189","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:28:20.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.486+0000 7f58427fc700 1 -- 172.21.15.16:0/3201709482 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f582c06b020 msgr2=0x7f582c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:20.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.486+0000 7f58427fc700 1 --2- 172.21.15.16:0/3201709482 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f582c06b020 0x7f582c06d4e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5834007fd0 tx=0x7f5834007de0 comp rx=0 tx=0).stop 2023-10-16T13:28:20.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.486+0000 7f58427fc700 1 -- 172.21.15.16:0/3201709482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f584c15da40 msgr2=0x7f584c153840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:20.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.486+0000 7f58427fc700 1 --2- 172.21.15.16:0/3201709482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f584c15da40 0x7f584c153840 secure :-1 s=READY pgs=2082 cs=0 l=1 rev1=1 crypto rx=0x7f584400b680 tx=0x7f5844004ad0 comp rx=0 tx=0).stop 2023-10-16T13:28:20.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.486+0000 7f58427fc700 1 -- 172.21.15.16:0/3201709482 shutdown_connections 2023-10-16T13:28:20.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.486+0000 7f58427fc700 1 --2- 172.21.15.16:0/3201709482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f584c15da40 0x7f584c153840 unknown :-1 s=CLOSED pgs=2082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:20.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.486+0000 7f58427fc700 1 --2- 172.21.15.16:0/3201709482 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f584c06c210 0x7f584c1f2b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:20.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.486+0000 7f58427fc700 1 --2- 172.21.15.16:0/3201709482 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f582c06b020 0x7f582c06d4e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:20.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.486+0000 7f58427fc700 1 --2- 172.21.15.16:0/3201709482 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f584c0b3690 0x7f584c1565d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:20.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.486+0000 7f58427fc700 1 -- 172.21.15.16:0/3201709482 >> 172.21.15.16:0/3201709482 conn(0x7f584c0cdf80 msgr2=0x7f584c0ce360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:20.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.486+0000 7f58427fc700 1 -- 172.21.15.16:0/3201709482 shutdown_connections 2023-10-16T13:28:20.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.486+0000 7f58427fc700 1 -- 172.21.15.16:0/3201709482 wait complete. 2023-10-16T13:28:20.501 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:28:20.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.624+0000 7f3781848700 1 Processor -- start 2023-10-16T13:28:20.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.624+0000 7f3781848700 1 -- start start 2023-10-16T13:28:20.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.625+0000 7f3781848700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f377c0ce4f0 0x7f377c0cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:20.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.625+0000 7f3781848700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f377c0cc2a0 con 0x7f377c15b630 2023-10-16T13:28:20.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.625+0000 7f3781848700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f377c0cc540 con 0x7f377c15ba10 2023-10-16T13:28:20.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.625+0000 7f3781848700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f377c0cdb20 con 0x7f377c0ce4f0 2023-10-16T13:28:20.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.625+0000 7f377affd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f377c15b630 0x7f377c0cdde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:39104/0 (socket says 172.21.15.16:39104) 2023-10-16T13:28:20.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.625+0000 7f377affd700 1 -- 172.21.15.16:0/2294555398 learned_addr learned my addr 172.21.15.16:0/2294555398 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:28:20.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.625+0000 7f377a7fc700 1 --2- 172.21.15.16:0/2294555398 >> v2:172.21.15.49:3300/0 conn(0x7f377c0ce4f0 0x7f377c0cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:20.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.626+0000 7f3779ffb700 1 -- 172.21.15.16:0/2294555398 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1864703979 0 0) 0x7f377c0cc2a0 con 0x7f377c15b630 2023-10-16T13:28:20.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.626+0000 7f3779ffb700 1 -- 172.21.15.16:0/2294555398 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37680035b0 con 0x7f377c15b630 2023-10-16T13:28:20.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.626+0000 7f377a7fc700 1 -- 172.21.15.16:0/2294555398 >> v1:172.21.15.49:6789/0 conn(0x7f377c15ba10 legacy=0x7f377c0cfca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:20.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.626+0000 7f377a7fc700 1 -- 172.21.15.16:0/2294555398 >> v1:172.21.15.16:6789/0 conn(0x7f377c15b630 legacy=0x7f377c0cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:20.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.626+0000 7f377a7fc700 1 -- 172.21.15.16:0/2294555398 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f377c0cc820 con 0x7f377c0ce4f0 2023-10-16T13:28:20.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.626+0000 7f377a7fc700 1 --2- 172.21.15.16:0/2294555398 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f377c0ce4f0 0x7f377c0cd5e0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f376c009840 tx=0x7f376c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=454b6bf7a00e29b8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:20.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.626+0000 7f3779ffb700 1 -- 172.21.15.16:0/2294555398 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f376c011830 con 0x7f377c0ce4f0 2023-10-16T13:28:20.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.626+0000 7f3779ffb700 1 -- 172.21.15.16:0/2294555398 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f376c011990 con 0x7f377c0ce4f0 2023-10-16T13:28:20.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.626+0000 7f3779ffb700 1 -- 172.21.15.16:0/2294555398 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f376c011d20 con 0x7f377c0ce4f0 2023-10-16T13:28:20.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.627+0000 7f3781848700 1 -- 172.21.15.16:0/2294555398 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f377c0ce4f0 msgr2=0x7f377c0cd5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:20.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.627+0000 7f3781848700 1 --2- 172.21.15.16:0/2294555398 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f377c0ce4f0 0x7f377c0cd5e0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f376c009840 tx=0x7f376c009b50 comp rx=0 tx=0).stop 2023-10-16T13:28:20.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.627+0000 7f3781848700 1 -- 172.21.15.16:0/2294555398 shutdown_connections 2023-10-16T13:28:20.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.627+0000 7f3781848700 1 --2- 172.21.15.16:0/2294555398 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f377c0ce4f0 0x7f377c0cd5e0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:20.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.627+0000 7f3781848700 1 -- 172.21.15.16:0/2294555398 >> 172.21.15.16:0/2294555398 conn(0x7f377c0d14b0 msgr2=0x7f377c0d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:20.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.627+0000 7f3781848700 1 -- 172.21.15.16:0/2294555398 shutdown_connections 2023-10-16T13:28:20.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.627+0000 7f3781848700 1 -- 172.21.15.16:0/2294555398 wait complete. 2023-10-16T13:28:20.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.628+0000 7f3781848700 1 Processor -- start 2023-10-16T13:28:20.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.628+0000 7f3781848700 1 -- start start 2023-10-16T13:28:20.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.629+0000 7f3781848700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f377c0ce4f0 0x7f377c1f0920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:20.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.629+0000 7f3781848700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f377c15b630 0x7f377c0b5e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:20.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.629+0000 7f3781848700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f377c15ba10 0x7f377c0b6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:20.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.629+0000 7f3781848700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f377c1f32d0 con 0x7f377c0ce4f0 2023-10-16T13:28:20.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.629+0000 7f3781848700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f377c1f34e0 con 0x7f377c15b630 2023-10-16T13:28:20.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.629+0000 7f3781848700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f377c1f3780 con 0x7f377c15ba10 2023-10-16T13:28:20.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.629+0000 7f377affd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f377c0ce4f0 0x7f377c1f0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:20.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.629+0000 7f377affd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f377c0ce4f0 0x7f377c1f0920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49836/0 (socket says 172.21.15.16:49836) 2023-10-16T13:28:20.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.629+0000 7f377affd700 1 -- 172.21.15.16:0/612112322 learned_addr learned my addr 172.21.15.16:0/612112322 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:20.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.629+0000 7f377a7fc700 1 --2- 172.21.15.16:0/612112322 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f377c15b630 0x7f377c0b5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:20.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.629+0000 7f377affd700 1 -- 172.21.15.16:0/612112322 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f377c15ba10 msgr2=0x7f377c0b6350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:20.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.629+0000 7f377b7fe700 1 --2- 172.21.15.16:0/612112322 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f377c15ba10 0x7f377c0b6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:20.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.629+0000 7f377affd700 1 --2- 172.21.15.16:0/612112322 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f377c15ba10 0x7f377c0b6350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:20.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.629+0000 7f377affd700 1 -- 172.21.15.16:0/612112322 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f377c15b630 msgr2=0x7f377c0b5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:20.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.629+0000 7f377affd700 1 --2- 172.21.15.16:0/612112322 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f377c15b630 0x7f377c0b5e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:20.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.629+0000 7f377affd700 1 -- 172.21.15.16:0/612112322 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f376c0094a0 con 0x7f377c0ce4f0 2023-10-16T13:28:20.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.629+0000 7f377a7fc700 1 --2- 172.21.15.16:0/612112322 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f377c15b630 0x7f377c0b5e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:28:20.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.630+0000 7f377affd700 1 --2- 172.21.15.16:0/612112322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f377c0ce4f0 0x7f377c1f0920 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 crypto rx=0x7f3764001260 tx=0x7f376400ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:20.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.630+0000 7f3773fff700 1 -- 172.21.15.16:0/612112322 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37640074e0 con 0x7f377c0ce4f0 2023-10-16T13:28:20.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.630+0000 7f3773fff700 1 -- 172.21.15.16:0/612112322 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3764007640 con 0x7f377c0ce4f0 2023-10-16T13:28:20.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.630+0000 7f3781848700 1 -- 172.21.15.16:0/612112322 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f377c1f3a60 con 0x7f377c0ce4f0 2023-10-16T13:28:20.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.630+0000 7f3773fff700 1 -- 172.21.15.16:0/612112322 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3764005a50 con 0x7f377c0ce4f0 2023-10-16T13:28:20.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.630+0000 7f3781848700 1 -- 172.21.15.16:0/612112322 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f377c1f4020 con 0x7f377c0ce4f0 2023-10-16T13:28:20.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.631+0000 7f3781848700 1 -- 172.21.15.16:0/612112322 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f377c06b8f0 con 0x7f377c0ce4f0 2023-10-16T13:28:20.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.631+0000 7f3773fff700 1 -- 172.21.15.16:0/612112322 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 107) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3764005bb0 con 0x7f377c0ce4f0 2023-10-16T13:28:20.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.632+0000 7f3773fff700 1 --2- 172.21.15.16:0/612112322 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f375c06aff0 0x7f375c06d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:20.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.632+0000 7f3773fff700 1 -- 172.21.15.16:0/612112322 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f3764088f40 con 0x7f377c0ce4f0 2023-10-16T13:28:20.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.632+0000 7f377a7fc700 1 --2- 172.21.15.16:0/612112322 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f375c06aff0 0x7f375c06d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:20.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.632+0000 7f377a7fc700 1 --2- 172.21.15.16:0/612112322 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f375c06aff0 0x7f375c06d4b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f377c0b41a0 tx=0x7f376c008030 comp rx=0 tx=0).ready entity=mgr.10047 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:20.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.636+0000 7f3773fff700 1 -- 172.21.15.16:0/612112322 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3764057720 con 0x7f377c0ce4f0 2023-10-16T13:28:20.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.838+0000 7f3781848700 1 -- 172.21.15.16:0/612112322 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f377c1f4210 con 0x7f377c0ce4f0 2023-10-16T13:28:20.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.840+0000 7f3773fff700 1 -- 172.21.15.16:0/612112322 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v189) v1 ==== 74+0+16780 (secure 0 0 0) 0x7f377c1f4210 con 0x7f377c0ce4f0 2023-10-16T13:28:20.842 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:28:20.843 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":189,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:28:19.523314+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2023-10-16T13:28:17.549815+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"189","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2023-10-16T13:28:18.493327+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"189","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:28:20.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.842+0000 7f3781848700 1 -- 172.21.15.16:0/612112322 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f375c06aff0 msgr2=0x7f375c06d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:20.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.842+0000 7f3781848700 1 --2- 172.21.15.16:0/612112322 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f375c06aff0 0x7f375c06d4b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f377c0b41a0 tx=0x7f376c008030 comp rx=0 tx=0).stop 2023-10-16T13:28:20.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.842+0000 7f3781848700 1 -- 172.21.15.16:0/612112322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f377c0ce4f0 msgr2=0x7f377c1f0920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:20.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.842+0000 7f3781848700 1 --2- 172.21.15.16:0/612112322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f377c0ce4f0 0x7f377c1f0920 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 crypto rx=0x7f3764001260 tx=0x7f376400ca90 comp rx=0 tx=0).stop 2023-10-16T13:28:20.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.842+0000 7f3781848700 1 -- 172.21.15.16:0/612112322 shutdown_connections 2023-10-16T13:28:20.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.842+0000 7f3781848700 1 --2- 172.21.15.16:0/612112322 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f377c15ba10 0x7f377c0b6350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:20.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.842+0000 7f3781848700 1 --2- 172.21.15.16:0/612112322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f377c0ce4f0 0x7f377c1f0920 unknown :-1 s=CLOSED pgs=2084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:20.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.842+0000 7f3781848700 1 --2- 172.21.15.16:0/612112322 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f375c06aff0 0x7f375c06d4b0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:20.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.842+0000 7f3781848700 1 --2- 172.21.15.16:0/612112322 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f377c15b630 0x7f377c0b5e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:20.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.842+0000 7f3781848700 1 -- 172.21.15.16:0/612112322 >> 172.21.15.16:0/612112322 conn(0x7f377c0d14b0 msgr2=0x7f377c0d1890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:20.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.842+0000 7f3781848700 1 -- 172.21.15.16:0/612112322 shutdown_connections 2023-10-16T13:28:20.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.842+0000 7f3781848700 1 -- 172.21.15.16:0/612112322 wait complete. 2023-10-16T13:28:20.857 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-16T13:28:20.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.983+0000 7f9816f0f700 1 Processor -- start 2023-10-16T13:28:20.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.983+0000 7f9816f0f700 1 -- start start 2023-10-16T13:28:20.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.984+0000 7f9816f0f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f981015b2c0 0x7f9810161340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:20.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.984+0000 7f9816f0f700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f981006a7e0 0x7f981006ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:20.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.984+0000 7f9816f0f700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f981015ecb0 con 0x7f981006e570 2023-10-16T13:28:20.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.984+0000 7f9816f0f700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f981006b190 con 0x7f981015b2c0 2023-10-16T13:28:20.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.984+0000 7f9816f0f700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9810161880 con 0x7f981006a7e0 2023-10-16T13:28:20.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.984+0000 7f9814cab700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f981015b2c0 0x7f9810161340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:20.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.984+0000 7f9814cab700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f981015b2c0 0x7f9810161340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49844/0 (socket says 172.21.15.16:49844) 2023-10-16T13:28:20.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.984+0000 7f9814cab700 1 -- 172.21.15.16:0/2763940334 learned_addr learned my addr 172.21.15.16:0/2763940334 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:20.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.984+0000 7f980ffff700 1 --1- 172.21.15.16:0/2763940334 >> v1:172.21.15.16:6789/0 conn(0x7f981006e570 0x7f981015e5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:39118/0 (socket says 172.21.15.16:39118) 2023-10-16T13:28:20.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.984+0000 7f98154ac700 1 --2- 172.21.15.16:0/2763940334 >> v2:172.21.15.49:3300/0 conn(0x7f981006a7e0 0x7f981006ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:20.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.984+0000 7f9814cab700 1 -- 172.21.15.16:0/2763940334 >> v2:172.21.15.49:3300/0 conn(0x7f981006a7e0 msgr2=0x7f981006ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:28:20.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.984+0000 7f9814cab700 1 --2- 172.21.15.16:0/2763940334 >> v2:172.21.15.49:3300/0 conn(0x7f981006a7e0 0x7f981006ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:20.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.984+0000 7f9814cab700 1 -- 172.21.15.16:0/2763940334 >> v1:172.21.15.16:6789/0 conn(0x7f981006e570 legacy=0x7f981015e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:20.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.984+0000 7f9814cab700 1 -- 172.21.15.16:0/2763940334 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f981015b6a0 con 0x7f981015b2c0 2023-10-16T13:28:20.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.984+0000 7f98154ac700 1 --2- 172.21.15.16:0/2763940334 >> v2:172.21.15.49:3300/0 conn(0x7f981006a7e0 0x7f981006ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:28:20.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.984+0000 7f9814cab700 1 --2- 172.21.15.16:0/2763940334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f981015b2c0 0x7f9810161340 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 crypto rx=0x7f981006b880 tx=0x7f9804009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7805260653f414bc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:20.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.984+0000 7f980f7fe700 1 -- 172.21.15.16:0/2763940334 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f980400b870 con 0x7f981015b2c0 2023-10-16T13:28:20.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.985+0000 7f980f7fe700 1 -- 172.21.15.16:0/2763940334 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f980400b9d0 con 0x7f981015b2c0 2023-10-16T13:28:20.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.985+0000 7f980f7fe700 1 -- 172.21.15.16:0/2763940334 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f980400bd60 con 0x7f981015b2c0 2023-10-16T13:28:20.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.985+0000 7f9816f0f700 1 -- 172.21.15.16:0/2763940334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f981015b2c0 msgr2=0x7f9810161340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:20.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.985+0000 7f9816f0f700 1 --2- 172.21.15.16:0/2763940334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f981015b2c0 0x7f9810161340 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 crypto rx=0x7f981006b880 tx=0x7f9804009bd0 comp rx=0 tx=0).stop 2023-10-16T13:28:20.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.985+0000 7f9816f0f700 1 -- 172.21.15.16:0/2763940334 shutdown_connections 2023-10-16T13:28:20.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.985+0000 7f9816f0f700 1 --2- 172.21.15.16:0/2763940334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f981015b2c0 0x7f9810161340 unknown :-1 s=CLOSED pgs=2086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:20.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.985+0000 7f9816f0f700 1 --2- 172.21.15.16:0/2763940334 >> v2:172.21.15.49:3300/0 conn(0x7f981006a7e0 0x7f981006ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:20.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.985+0000 7f9816f0f700 1 -- 172.21.15.16:0/2763940334 >> 172.21.15.16:0/2763940334 conn(0x7f98100b1f40 msgr2=0x7f98100b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:20.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.985+0000 7f9816f0f700 1 -- 172.21.15.16:0/2763940334 shutdown_connections 2023-10-16T13:28:20.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.985+0000 7f9816f0f700 1 -- 172.21.15.16:0/2763940334 wait complete. 2023-10-16T13:28:20.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.987+0000 7f9816f0f700 1 Processor -- start 2023-10-16T13:28:20.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.987+0000 7f9816f0f700 1 -- start start 2023-10-16T13:28:20.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.988+0000 7f9816f0f700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f981006a7e0 0x7f98100c6570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:20.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.988+0000 7f9816f0f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f981006e570 0x7f9810160b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:20.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.988+0000 7f9816f0f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f981015b2c0 0x7f98100c9d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:20.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.988+0000 7f9816f0f700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98100ca2d0 con 0x7f981006e570 2023-10-16T13:28:20.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.988+0000 7f9816f0f700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98100c6c70 con 0x7f981006a7e0 2023-10-16T13:28:20.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.988+0000 7f9816f0f700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98100c9010 con 0x7f981015b2c0 2023-10-16T13:28:20.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.988+0000 7f980ffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f981006e570 0x7f9810160b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:20.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.988+0000 7f980ffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f981006e570 0x7f9810160b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49852/0 (socket says 172.21.15.16:49852) 2023-10-16T13:28:20.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.988+0000 7f980ffff700 1 -- 172.21.15.16:0/2718366239 learned_addr learned my addr 172.21.15.16:0/2718366239 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:20.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.988+0000 7f9814cab700 1 --2- 172.21.15.16:0/2718366239 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f981006a7e0 0x7f98100c6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:20.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.988+0000 7f98154ac700 1 --2- 172.21.15.16:0/2718366239 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f981015b2c0 0x7f98100c9d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:20.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.988+0000 7f980ffff700 1 -- 172.21.15.16:0/2718366239 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f981015b2c0 msgr2=0x7f98100c9d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:20.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.988+0000 7f980ffff700 1 --2- 172.21.15.16:0/2718366239 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f981015b2c0 0x7f98100c9d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:20.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.988+0000 7f980ffff700 1 -- 172.21.15.16:0/2718366239 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f981006a7e0 msgr2=0x7f98100c6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:20.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.988+0000 7f980ffff700 1 --2- 172.21.15.16:0/2718366239 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f981006a7e0 0x7f98100c6570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:20.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.988+0000 7f980ffff700 1 -- 172.21.15.16:0/2718366239 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9804009550 con 0x7f981006e570 2023-10-16T13:28:20.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.988+0000 7f9814cab700 1 --2- 172.21.15.16:0/2718366239 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f981006a7e0 0x7f98100c6570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:28:20.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.988+0000 7f980ffff700 1 --2- 172.21.15.16:0/2718366239 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f981006e570 0x7f9810160b20 secure :-1 s=READY pgs=2087 cs=0 l=1 rev1=1 crypto rx=0x7f97fc00c500 tx=0x7f97fc00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:20.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.989+0000 7f980d7fa700 1 -- 172.21.15.16:0/2718366239 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97fc005cb0 con 0x7f981006e570 2023-10-16T13:28:20.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.989+0000 7f980d7fa700 1 -- 172.21.15.16:0/2718366239 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97fc005e10 con 0x7f981006e570 2023-10-16T13:28:20.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.989+0000 7f9816f0f700 1 -- 172.21.15.16:0/2718366239 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98100c92b0 con 0x7f981006e570 2023-10-16T13:28:20.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.989+0000 7f980d7fa700 1 -- 172.21.15.16:0/2718366239 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97fc0072d0 con 0x7f981006e570 2023-10-16T13:28:20.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.989+0000 7f9816f0f700 1 -- 172.21.15.16:0/2718366239 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98100c9870 con 0x7f981006e570 2023-10-16T13:28:20.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.990+0000 7f9816f0f700 1 -- 172.21.15.16:0/2718366239 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98101620c0 con 0x7f981006e570 2023-10-16T13:28:20.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.990+0000 7f980d7fa700 1 -- 172.21.15.16:0/2718366239 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 107) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f97fc007430 con 0x7f981006e570 2023-10-16T13:28:20.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.991+0000 7f980d7fa700 1 --2- 172.21.15.16:0/2718366239 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f97f406b020 0x7f97f406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:21.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.991+0000 7f980d7fa700 1 -- 172.21.15.16:0/2718366239 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f97fc088ba0 con 0x7f981006e570 2023-10-16T13:28:21.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.991+0000 7f9814cab700 1 --2- 172.21.15.16:0/2718366239 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f97f406b020 0x7f97f406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:21.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.991+0000 7f9814cab700 1 --2- 172.21.15.16:0/2718366239 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f97f406b020 0x7f97f406d4e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9804009130 tx=0x7f9804008030 comp rx=0 tx=0).ready entity=mgr.10047 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:21.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:20.995+0000 7f980d7fa700 1 -- 172.21.15.16:0/2718366239 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f98101620c0 con 0x7f981006e570 2023-10-16T13:28:21.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:21.254+0000 7f9816f0f700 1 -- 172.21.15.16:0/2718366239 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f98101620c0 con 0x7f981006e570 2023-10-16T13:28:22.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.109+0000 7f980d7fa700 1 -- 172.21.15.16:0/2718366239 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v94) v1 ==== 118+0+0 (secure 0 0 0) 0x7f98101620c0 con 0x7f981006e570 2023-10-16T13:28:22.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.113+0000 7f9816f0f700 1 -- 172.21.15.16:0/2718366239 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f97f406b020 msgr2=0x7f97f406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:22.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.113+0000 7f9816f0f700 1 --2- 172.21.15.16:0/2718366239 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f97f406b020 0x7f97f406d4e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9804009130 tx=0x7f9804008030 comp rx=0 tx=0).stop 2023-10-16T13:28:22.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.113+0000 7f9816f0f700 1 -- 172.21.15.16:0/2718366239 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f981006e570 msgr2=0x7f9810160b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:22.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.113+0000 7f9816f0f700 1 --2- 172.21.15.16:0/2718366239 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f981006e570 0x7f9810160b20 secure :-1 s=READY pgs=2087 cs=0 l=1 rev1=1 crypto rx=0x7f97fc00c500 tx=0x7f97fc00c8c0 comp rx=0 tx=0).stop 2023-10-16T13:28:22.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.113+0000 7f9816f0f700 1 -- 172.21.15.16:0/2718366239 shutdown_connections 2023-10-16T13:28:22.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.113+0000 7f9816f0f700 1 --2- 172.21.15.16:0/2718366239 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f981015b2c0 0x7f98100c9d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:22.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.113+0000 7f9816f0f700 1 --2- 172.21.15.16:0/2718366239 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f97f406b020 0x7f97f406d4e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:22.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.113+0000 7f9816f0f700 1 --2- 172.21.15.16:0/2718366239 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f981006a7e0 0x7f98100c6570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:22.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.113+0000 7f9816f0f700 1 --2- 172.21.15.16:0/2718366239 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f981006e570 0x7f9810160b20 unknown :-1 s=CLOSED pgs=2087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:22.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.113+0000 7f9816f0f700 1 -- 172.21.15.16:0/2718366239 >> 172.21.15.16:0/2718366239 conn(0x7f98100b1f40 msgr2=0x7f981015cd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:22.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.113+0000 7f9816f0f700 1 -- 172.21.15.16:0/2718366239 shutdown_connections 2023-10-16T13:28:22.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.113+0000 7f9816f0f700 1 -- 172.21.15.16:0/2718366239 wait complete. 2023-10-16T13:28:22.130 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-16T13:28:22.131 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-16T13:28:22.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.263+0000 7f66fd9dd700 1 Processor -- start 2023-10-16T13:28:22.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.263+0000 7f66fd9dd700 1 -- start start 2023-10-16T13:28:22.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.264+0000 7f66fd9dd700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f66f815b070 0x7f66f815b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:22.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.264+0000 7f66fd9dd700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f66f80cb880 0x7f66f80cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:22.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.264+0000 7f66fd9dd700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66f80cef70 con 0x7f66f806da60 2023-10-16T13:28:22.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.264+0000 7f66fd9dd700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66f80ce360 con 0x7f66f80cb880 2023-10-16T13:28:22.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.264+0000 7f66fd9dd700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66f80ce640 con 0x7f66f815b070 2023-10-16T13:28:22.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.265+0000 7f66f67fc700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f66f80cb880 0x7f66f80cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:22.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.265+0000 7f66f6ffd700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f66f815b070 0x7f66f815b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:22.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.265+0000 7f66f77fe700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7f66f806da60 0x7f66f80cf400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:46316/0 (socket says 172.21.15.16:46316) 2023-10-16T13:28:22.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.265+0000 7f66f77fe700 1 -- 172.21.15.16:0/2066179017 learned_addr learned my addr 172.21.15.16:0/2066179017 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:28:22.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.265+0000 7f66f67fc700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f66f80cb880 0x7f66f80cbd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:48922/0 (socket says 172.21.15.16:48922) 2023-10-16T13:28:22.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.265+0000 7f66f6ffd700 1 --2- 172.21.15.16:0/2066179017 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f66f815b070 0x7f66f815b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:38116/0 (socket says 172.21.15.16:38116) 2023-10-16T13:28:22.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.265+0000 7f66f6ffd700 1 -- 172.21.15.16:0/2066179017 >> v2:172.21.15.49:3300/0 conn(0x7f66f80cb880 msgr2=0x7f66f80cbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:28:22.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.265+0000 7f66f6ffd700 1 --2- 172.21.15.16:0/2066179017 >> v2:172.21.15.49:3300/0 conn(0x7f66f80cb880 0x7f66f80cbd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:22.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.265+0000 7f66f6ffd700 1 -- 172.21.15.16:0/2066179017 >> v1:172.21.15.49:6789/0 conn(0x7f66f806da60 legacy=0x7f66f80cf400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:22.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.265+0000 7f66f6ffd700 1 -- 172.21.15.16:0/2066179017 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66f80ce920 con 0x7f66f815b070 2023-10-16T13:28:22.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.265+0000 7f66f67fc700 1 --2- 172.21.15.16:0/2066179017 >> v2:172.21.15.49:3300/0 conn(0x7f66f80cb880 0x7f66f80cbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:28:22.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.265+0000 7f66f6ffd700 1 --2- 172.21.15.16:0/2066179017 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f66f815b070 0x7f66f815b450 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f66ec009840 tx=0x7f66ec009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd734d9d64123d7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:22.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.266+0000 7f66f5ffb700 1 -- 172.21.15.16:0/2066179017 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66ec011830 con 0x7f66f815b070 2023-10-16T13:28:22.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.266+0000 7f66f5ffb700 1 -- 172.21.15.16:0/2066179017 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66ec011990 con 0x7f66f815b070 2023-10-16T13:28:22.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.266+0000 7f66f5ffb700 1 -- 172.21.15.16:0/2066179017 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66ec011d20 con 0x7f66f815b070 2023-10-16T13:28:22.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.266+0000 7f66fd9dd700 1 -- 172.21.15.16:0/2066179017 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f66f815b070 msgr2=0x7f66f815b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:22.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.266+0000 7f66fd9dd700 1 --2- 172.21.15.16:0/2066179017 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f66f815b070 0x7f66f815b450 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f66ec009840 tx=0x7f66ec009b50 comp rx=0 tx=0).stop 2023-10-16T13:28:22.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.266+0000 7f66fd9dd700 1 -- 172.21.15.16:0/2066179017 shutdown_connections 2023-10-16T13:28:22.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.266+0000 7f66fd9dd700 1 --2- 172.21.15.16:0/2066179017 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f66f815b070 0x7f66f815b450 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:22.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.266+0000 7f66fd9dd700 1 --2- 172.21.15.16:0/2066179017 >> v2:172.21.15.49:3300/0 conn(0x7f66f80cb880 0x7f66f80cbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:22.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.266+0000 7f66fd9dd700 1 -- 172.21.15.16:0/2066179017 >> 172.21.15.16:0/2066179017 conn(0x7f66f80b1f40 msgr2=0x7f66f80b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:22.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.266+0000 7f66fd9dd700 1 -- 172.21.15.16:0/2066179017 shutdown_connections 2023-10-16T13:28:22.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.266+0000 7f66fd9dd700 1 -- 172.21.15.16:0/2066179017 wait complete. 2023-10-16T13:28:22.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.267+0000 7f66fd9dd700 1 Processor -- start 2023-10-16T13:28:22.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.268+0000 7f66fd9dd700 1 -- start start 2023-10-16T13:28:22.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.268+0000 7f66fd9dd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66f806da60 0x7f66f80b58d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:22.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.268+0000 7f66fd9dd700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f66f80cb880 0x7f66f8152d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:22.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.268+0000 7f66fd9dd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f66f815b070 0x7f66f80c68b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:22.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.268+0000 7f66fd9dd700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66f8154100 con 0x7f66f806da60 2023-10-16T13:28:22.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.268+0000 7f66fd9dd700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66f81537d0 con 0x7f66f815b070 2023-10-16T13:28:22.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.268+0000 7f66fd9dd700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66f8153ab0 con 0x7f66f80cb880 2023-10-16T13:28:22.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.268+0000 7f66f6ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66f806da60 0x7f66f80b58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:22.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.268+0000 7f66f6ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66f806da60 0x7f66f80b58d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49864/0 (socket says 172.21.15.16:49864) 2023-10-16T13:28:22.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.268+0000 7f66f6ffd700 1 -- 172.21.15.16:0/2784961636 learned_addr learned my addr 172.21.15.16:0/2784961636 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:22.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.269+0000 7f66f77fe700 1 --2- 172.21.15.16:0/2784961636 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f66f815b070 0x7f66f80c68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:22.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.269+0000 7f66f67fc700 1 --2- 172.21.15.16:0/2784961636 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f66f80cb880 0x7f66f8152d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:22.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.269+0000 7f66f6ffd700 1 -- 172.21.15.16:0/2784961636 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f66f80cb880 msgr2=0x7f66f8152d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:22.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.269+0000 7f66f6ffd700 1 --2- 172.21.15.16:0/2784961636 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f66f80cb880 0x7f66f8152d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:22.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.269+0000 7f66f6ffd700 1 -- 172.21.15.16:0/2784961636 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f66f815b070 msgr2=0x7f66f80c68b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:22.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.269+0000 7f66f6ffd700 1 --2- 172.21.15.16:0/2784961636 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f66f815b070 0x7f66f80c68b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:22.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.269+0000 7f66f6ffd700 1 -- 172.21.15.16:0/2784961636 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66e80094a0 con 0x7f66f806da60 2023-10-16T13:28:22.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.269+0000 7f66f77fe700 1 --2- 172.21.15.16:0/2784961636 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f66f815b070 0x7f66f80c68b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:28:22.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.269+0000 7f66f6ffd700 1 --2- 172.21.15.16:0/2784961636 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66f806da60 0x7f66f80b58d0 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 crypto rx=0x7f66ec009fd0 tx=0x7f66ec004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:22.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.269+0000 7f66dffff700 1 -- 172.21.15.16:0/2784961636 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66ec0036a0 con 0x7f66f806da60 2023-10-16T13:28:22.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.269+0000 7f66dffff700 1 -- 172.21.15.16:0/2784961636 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66ec01e070 con 0x7f66f806da60 2023-10-16T13:28:22.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.269+0000 7f66fd9dd700 1 -- 172.21.15.16:0/2784961636 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66ec0094a0 con 0x7f66f806da60 2023-10-16T13:28:22.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.269+0000 7f66dffff700 1 -- 172.21.15.16:0/2784961636 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66ec003a60 con 0x7f66f806da60 2023-10-16T13:28:22.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.269+0000 7f66fd9dd700 1 -- 172.21.15.16:0/2784961636 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66f80b5e10 con 0x7f66f806da60 2023-10-16T13:28:22.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.271+0000 7f66fd9dd700 1 -- 172.21.15.16:0/2784961636 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66f80b6000 con 0x7f66f806da60 2023-10-16T13:28:22.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.271+0000 7f66dffff700 1 -- 172.21.15.16:0/2784961636 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 107) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f66ec003bc0 con 0x7f66f806da60 2023-10-16T13:28:22.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.271+0000 7f66dffff700 1 --2- 172.21.15.16:0/2784961636 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f66e406b020 0x7f66e406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:22.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.271+0000 7f66f67fc700 1 --2- 172.21.15.16:0/2784961636 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f66e406b020 0x7f66e406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:22.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.271+0000 7f66dffff700 1 -- 172.21.15.16:0/2784961636 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f66ec08a880 con 0x7f66f806da60 2023-10-16T13:28:22.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.272+0000 7f66f67fc700 1 --2- 172.21.15.16:0/2784961636 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f66e406b020 0x7f66e406d4e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f66f80cf870 tx=0x7f66e8008040 comp rx=0 tx=0).ready entity=mgr.10047 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:22.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.276+0000 7f66dffff700 1 -- 172.21.15.16:0/2784961636 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f66f80b6000 con 0x7f66f806da60 2023-10-16T13:28:22.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:22.527+0000 7f66fd9dd700 1 -- 172.21.15.16:0/2784961636 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f66f80b6000 con 0x7f66f806da60 2023-10-16T13:28:23.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.110+0000 7f66dffff700 1 -- 172.21.15.16:0/2784961636 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v95) v1 ==== 122+0+0 (secure 0 0 0) 0x7f66f80b6000 con 0x7f66f806da60 2023-10-16T13:28:23.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.113+0000 7f66fd9dd700 1 -- 172.21.15.16:0/2784961636 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f66e406b020 msgr2=0x7f66e406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:23.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.113+0000 7f66fd9dd700 1 --2- 172.21.15.16:0/2784961636 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f66e406b020 0x7f66e406d4e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f66f80cf870 tx=0x7f66e8008040 comp rx=0 tx=0).stop 2023-10-16T13:28:23.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.113+0000 7f66fd9dd700 1 -- 172.21.15.16:0/2784961636 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66f806da60 msgr2=0x7f66f80b58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:23.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.113+0000 7f66fd9dd700 1 --2- 172.21.15.16:0/2784961636 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66f806da60 0x7f66f80b58d0 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 crypto rx=0x7f66ec009fd0 tx=0x7f66ec004750 comp rx=0 tx=0).stop 2023-10-16T13:28:23.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.113+0000 7f66fd9dd700 1 -- 172.21.15.16:0/2784961636 shutdown_connections 2023-10-16T13:28:23.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.113+0000 7f66fd9dd700 1 --2- 172.21.15.16:0/2784961636 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f66e406b020 0x7f66e406d4e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:23.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.113+0000 7f66fd9dd700 1 --2- 172.21.15.16:0/2784961636 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f66f815b070 0x7f66f80c68b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:23.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.113+0000 7f66fd9dd700 1 --2- 172.21.15.16:0/2784961636 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66f806da60 0x7f66f80b58d0 unknown :-1 s=CLOSED pgs=2088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:23.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.113+0000 7f66fd9dd700 1 --2- 172.21.15.16:0/2784961636 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f66f80cb880 0x7f66f8152d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:23.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.113+0000 7f66fd9dd700 1 -- 172.21.15.16:0/2784961636 >> 172.21.15.16:0/2784961636 conn(0x7f66f80b1f40 msgr2=0x7f66f80cbd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:23.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.114+0000 7f66fd9dd700 1 -- 172.21.15.16:0/2784961636 shutdown_connections 2023-10-16T13:28:23.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.114+0000 7f66fd9dd700 1 -- 172.21.15.16:0/2784961636 wait complete. 2023-10-16T13:28:23.131 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-16T13:28:23.132 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:28:23.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.261+0000 7fe2b3ecc700 1 Processor -- start 2023-10-16T13:28:23.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.261+0000 7fe2b3ecc700 1 -- start start 2023-10-16T13:28:23.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.262+0000 7fe2b3ecc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe2ac15a980 0x7fe2ac06f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:23.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.262+0000 7fe2b3ecc700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe2ac0725d0 0x7fe2ac06f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:23.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.262+0000 7fe2b3ecc700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2ac0cf110 con 0x7fe2ac15a5a0 2023-10-16T13:28:23.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.262+0000 7fe2b3ecc700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2ac0cfc90 con 0x7fe2ac15a980 2023-10-16T13:28:23.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.262+0000 7fe2b3ecc700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2ac0cf360 con 0x7fe2ac0725d0 2023-10-16T13:28:23.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.262+0000 7fe2b1467700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe2ac15a980 0x7fe2ac06f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:23.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.262+0000 7fe2b1467700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe2ac15a980 0x7fe2ac06f370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49874/0 (socket says 172.21.15.16:49874) 2023-10-16T13:28:23.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.262+0000 7fe2b1467700 1 -- 172.21.15.16:0/2426645771 learned_addr learned my addr 172.21.15.16:0/2426645771 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:23.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.263+0000 7fe2b2469700 1 --2- 172.21.15.16:0/2426645771 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe2ac0725d0 0x7fe2ac06f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:23.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.263+0000 7fe2b1467700 1 -- 172.21.15.16:0/2426645771 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe2ac0725d0 msgr2=0x7fe2ac06f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:28:23.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.263+0000 7fe2b1467700 1 --2- 172.21.15.16:0/2426645771 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe2ac0725d0 0x7fe2ac06f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:23.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.263+0000 7fe2b1467700 1 -- 172.21.15.16:0/2426645771 >> v1:172.21.15.16:6789/0 conn(0x7fe2ac15a5a0 legacy=0x7fe2ac071c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:23.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.263+0000 7fe2b1467700 1 -- 172.21.15.16:0/2426645771 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2ac0cf640 con 0x7fe2ac15a980 2023-10-16T13:28:23.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.263+0000 7fe2b1467700 1 --2- 172.21.15.16:0/2426645771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2ac15a980 0x7fe2ac06f370 secure :-1 s=READY pgs=2090 cs=0 l=1 rev1=1 crypto rx=0x7fe29c0098f0 tx=0x7fe29c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dea6a9b5a508a57c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:23.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.263+0000 7fe2b0c66700 1 -- 172.21.15.16:0/2426645771 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe29c00b8a0 con 0x7fe2ac15a980 2023-10-16T13:28:23.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.263+0000 7fe2b0c66700 1 -- 172.21.15.16:0/2426645771 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe29c00ba00 con 0x7fe2ac15a980 2023-10-16T13:28:23.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.263+0000 7fe2b0c66700 1 -- 172.21.15.16:0/2426645771 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe29c00bd90 con 0x7fe2ac15a980 2023-10-16T13:28:23.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.263+0000 7fe2b3ecc700 1 -- 172.21.15.16:0/2426645771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2ac15a980 msgr2=0x7fe2ac06f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:23.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.263+0000 7fe2b3ecc700 1 --2- 172.21.15.16:0/2426645771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2ac15a980 0x7fe2ac06f370 secure :-1 s=READY pgs=2090 cs=0 l=1 rev1=1 crypto rx=0x7fe29c0098f0 tx=0x7fe29c009c00 comp rx=0 tx=0).stop 2023-10-16T13:28:23.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.263+0000 7fe2b3ecc700 1 -- 172.21.15.16:0/2426645771 shutdown_connections 2023-10-16T13:28:23.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.263+0000 7fe2b3ecc700 1 --2- 172.21.15.16:0/2426645771 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe2ac0725d0 0x7fe2ac06f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:23.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.263+0000 7fe2b3ecc700 1 --2- 172.21.15.16:0/2426645771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2ac15a980 0x7fe2ac06f370 unknown :-1 s=CLOSED pgs=2090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:23.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.263+0000 7fe2b3ecc700 1 -- 172.21.15.16:0/2426645771 >> 172.21.15.16:0/2426645771 conn(0x7fe2ac0b2040 msgr2=0x7fe2ac0b2420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:23.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.264+0000 7fe2b3ecc700 1 -- 172.21.15.16:0/2426645771 shutdown_connections 2023-10-16T13:28:23.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.264+0000 7fe2b3ecc700 1 -- 172.21.15.16:0/2426645771 wait complete. 2023-10-16T13:28:23.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.265+0000 7fe2b3ecc700 1 Processor -- start 2023-10-16T13:28:23.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.265+0000 7fe2b3ecc700 1 -- start start 2023-10-16T13:28:23.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.265+0000 7fe2b3ecc700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe2ac0725d0 0x7fe2ac158720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:23.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.266+0000 7fe2b3ecc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2ac15a5a0 0x7fe2ac0b4890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:23.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.266+0000 7fe2b3ecc700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe2ac15a980 0x7fe2ac0c95b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:23.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.266+0000 7fe2b3ecc700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2ac158c60 con 0x7fe2ac15a5a0 2023-10-16T13:28:23.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.266+0000 7fe2b3ecc700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2ac0ae040 con 0x7fe2ac0725d0 2023-10-16T13:28:23.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.266+0000 7fe2b3ecc700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2ac0ae2e0 con 0x7fe2ac15a980 2023-10-16T13:28:23.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.266+0000 7fe2b1467700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2ac15a5a0 0x7fe2ac0b4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:23.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.266+0000 7fe2b1467700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2ac15a5a0 0x7fe2ac0b4890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49880/0 (socket says 172.21.15.16:49880) 2023-10-16T13:28:23.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.266+0000 7fe2b1467700 1 -- 172.21.15.16:0/2444822460 learned_addr learned my addr 172.21.15.16:0/2444822460 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:23.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.266+0000 7fe2b1c68700 1 --2- 172.21.15.16:0/2444822460 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe2ac0725d0 0x7fe2ac158720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:23.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.266+0000 7fe2b2469700 1 --2- 172.21.15.16:0/2444822460 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe2ac15a980 0x7fe2ac0c95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:23.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.266+0000 7fe2b1467700 1 -- 172.21.15.16:0/2444822460 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe2ac15a980 msgr2=0x7fe2ac0c95b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:23.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.266+0000 7fe2b1467700 1 --2- 172.21.15.16:0/2444822460 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe2ac15a980 0x7fe2ac0c95b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:23.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.266+0000 7fe2b1467700 1 -- 172.21.15.16:0/2444822460 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe2ac0725d0 msgr2=0x7fe2ac158720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:23.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.266+0000 7fe2b1467700 1 --2- 172.21.15.16:0/2444822460 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe2ac0725d0 0x7fe2ac158720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:23.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.266+0000 7fe2b1467700 1 -- 172.21.15.16:0/2444822460 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe29c009550 con 0x7fe2ac15a5a0 2023-10-16T13:28:23.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.266+0000 7fe2b2469700 1 --2- 172.21.15.16:0/2444822460 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe2ac15a980 0x7fe2ac0c95b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:28:23.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.266+0000 7fe2b1467700 1 --2- 172.21.15.16:0/2444822460 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2ac15a5a0 0x7fe2ac0b4890 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 crypto rx=0x7fe29c009c40 tx=0x7fe29c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:23.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.266+0000 7fe2a27fc700 1 -- 172.21.15.16:0/2444822460 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe29c004590 con 0x7fe2ac15a5a0 2023-10-16T13:28:23.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.266+0000 7fe2a27fc700 1 -- 172.21.15.16:0/2444822460 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe29c01e070 con 0x7fe2ac15a5a0 2023-10-16T13:28:23.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.266+0000 7fe2b3ecc700 1 -- 172.21.15.16:0/2444822460 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2ac0ae5c0 con 0x7fe2ac15a5a0 2023-10-16T13:28:23.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.266+0000 7fe2a27fc700 1 -- 172.21.15.16:0/2444822460 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe29c0038e0 con 0x7fe2ac15a5a0 2023-10-16T13:28:23.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.266+0000 7fe2b3ecc700 1 -- 172.21.15.16:0/2444822460 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2ac0aeb80 con 0x7fe2ac15a5a0 2023-10-16T13:28:23.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.267+0000 7fe2b3ecc700 1 -- 172.21.15.16:0/2444822460 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2ac04ebb0 con 0x7fe2ac15a5a0 2023-10-16T13:28:23.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.268+0000 7fe2a27fc700 1 -- 172.21.15.16:0/2444822460 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 107) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe29c003a40 con 0x7fe2ac15a5a0 2023-10-16T13:28:23.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.268+0000 7fe2a27fc700 1 --2- 172.21.15.16:0/2444822460 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7fe29006b020 0x7fe29006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:23.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.268+0000 7fe2a27fc700 1 -- 172.21.15.16:0/2444822460 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 6129+0+0 (secure 0 0 0) 0x7fe29c020070 con 0x7fe2ac15a5a0 2023-10-16T13:28:23.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.268+0000 7fe2b1c68700 1 --2- 172.21.15.16:0/2444822460 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7fe29006b020 0x7fe29006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:23.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.269+0000 7fe2b1c68700 1 --2- 172.21.15.16:0/2444822460 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7fe29006b020 0x7fe29006d4e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe2a8007fd0 tx=0x7fe2a8009040 comp rx=0 tx=0).ready entity=mgr.10047 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:23.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.273+0000 7fe2a27fc700 1 -- 172.21.15.16:0/2444822460 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe29c055bc0 con 0x7fe2ac15a5a0 2023-10-16T13:28:23.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.524+0000 7fe2b3ecc700 1 -- 172.21.15.16:0/2444822460 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe2ac0b2c70 con 0x7fe2ac15a5a0 2023-10-16T13:28:23.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.525+0000 7fe2a27fc700 1 -- 172.21.15.16:0/2444822460 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+4920 (secure 0 0 0) 0x7fe29c015090 con 0x7fe2ac15a5a0 2023-10-16T13:28:23.527 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:28:23.528 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":95,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8306,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2415675492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":2415675492},{"type":"v1","addr":"172.21.15.16:6837","nonce":2415675492}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":90},{"gid":8315,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/4031561298","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":4031561298},{"type":"v1","addr":"172.21.15.49:6834","nonce":4031561298}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":90},{"gid":8318,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/2853801536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":2853801536},{"type":"v1","addr":"172.21.15.49:6835","nonce":2853801536}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":90}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:28:19.523721+0000","modified":"2023-10-16T13:28:22.528826+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10296":{"gid":10296,"name":"c","rank":0,"incarnation":91,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6836/3244401013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":3244401013},{"type":"v1","addr":"172.21.15.16:6836","nonce":3244401013}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2023-10-16T13:28:23.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.527+0000 7fe2b3ecc700 1 -- 172.21.15.16:0/2444822460 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7fe29006b020 msgr2=0x7fe29006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:23.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.527+0000 7fe2b3ecc700 1 --2- 172.21.15.16:0/2444822460 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7fe29006b020 0x7fe29006d4e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe2a8007fd0 tx=0x7fe2a8009040 comp rx=0 tx=0).stop 2023-10-16T13:28:23.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.527+0000 7fe2b3ecc700 1 -- 172.21.15.16:0/2444822460 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2ac15a5a0 msgr2=0x7fe2ac0b4890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:23.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.527+0000 7fe2b3ecc700 1 --2- 172.21.15.16:0/2444822460 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2ac15a5a0 0x7fe2ac0b4890 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 crypto rx=0x7fe29c009c40 tx=0x7fe29c004750 comp rx=0 tx=0).stop 2023-10-16T13:28:23.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.527+0000 7fe2b3ecc700 1 -- 172.21.15.16:0/2444822460 shutdown_connections 2023-10-16T13:28:23.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.527+0000 7fe2b3ecc700 1 --2- 172.21.15.16:0/2444822460 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe2ac15a980 0x7fe2ac0c95b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:23.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.527+0000 7fe2b3ecc700 1 --2- 172.21.15.16:0/2444822460 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7fe29006b020 0x7fe29006d4e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:23.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.527+0000 7fe2b3ecc700 1 --2- 172.21.15.16:0/2444822460 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe2ac0725d0 0x7fe2ac158720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:23.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.527+0000 7fe2b3ecc700 1 --2- 172.21.15.16:0/2444822460 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2ac15a5a0 0x7fe2ac0b4890 unknown :-1 s=CLOSED pgs=2091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:23.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.527+0000 7fe2b3ecc700 1 -- 172.21.15.16:0/2444822460 >> 172.21.15.16:0/2444822460 conn(0x7fe2ac0b2040 msgr2=0x7fe2ac159b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:23.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.527+0000 7fe2b3ecc700 1 -- 172.21.15.16:0/2444822460 shutdown_connections 2023-10-16T13:28:23.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.527+0000 7fe2b3ecc700 1 -- 172.21.15.16:0/2444822460 wait complete. 2023-10-16T13:28:23.532 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 95 2023-10-16T13:28:23.542 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:28:23.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.664+0000 7f621e128700 1 Processor -- start 2023-10-16T13:28:23.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.665+0000 7f621e128700 1 -- start start 2023-10-16T13:28:23.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.665+0000 7f621e128700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f621806aef0 0x7f62180cb980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:23.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.665+0000 7f621e128700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62180cedb0 con 0x7f62180ce160 2023-10-16T13:28:23.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.665+0000 7f621e128700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62180d12c0 con 0x7f621815db40 2023-10-16T13:28:23.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.665+0000 7f621e128700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f621815df20 con 0x7f621806aef0 2023-10-16T13:28:23.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.665+0000 7f6216ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f621806aef0 0x7f62180cb980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:23.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.666+0000 7f6216ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f621806aef0 0x7f62180cb980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49888/0 (socket says 172.21.15.16:49888) 2023-10-16T13:28:23.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.666+0000 7f6216ffd700 1 -- 172.21.15.16:0/3020954338 learned_addr learned my addr 172.21.15.16:0/3020954338 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:23.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.666+0000 7f6217fff700 1 --1- 172.21.15.16:0/3020954338 >> v1:172.21.15.16:6789/0 conn(0x7f62180ce160 0x7f62180ce5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:39164/0 (socket says 172.21.15.16:39164) 2023-10-16T13:28:23.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.666+0000 7f6216ffd700 1 -- 172.21.15.16:0/3020954338 >> v1:172.21.15.49:6789/0 conn(0x7f621815db40 legacy=0x7f621806a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:23.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.666+0000 7f6216ffd700 1 -- 172.21.15.16:0/3020954338 >> v1:172.21.15.16:6789/0 conn(0x7f62180ce160 legacy=0x7f62180ce5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:23.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.666+0000 7f6216ffd700 1 -- 172.21.15.16:0/3020954338 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62180cabc0 con 0x7f621806aef0 2023-10-16T13:28:23.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.666+0000 7f6216ffd700 1 --2- 172.21.15.16:0/3020954338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f621806aef0 0x7f62180cb980 secure :-1 s=READY pgs=2093 cs=0 l=1 rev1=1 crypto rx=0x7f620c0098f0 tx=0x7f620c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e585717221a3de85 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:23.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.666+0000 7f62167fc700 1 -- 172.21.15.16:0/3020954338 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f620c00b8a0 con 0x7f621806aef0 2023-10-16T13:28:23.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.666+0000 7f62167fc700 1 -- 172.21.15.16:0/3020954338 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f620c00ba00 con 0x7f621806aef0 2023-10-16T13:28:23.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.666+0000 7f62167fc700 1 -- 172.21.15.16:0/3020954338 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f620c00bd90 con 0x7f621806aef0 2023-10-16T13:28:23.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.667+0000 7f621e128700 1 -- 172.21.15.16:0/3020954338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f621806aef0 msgr2=0x7f62180cb980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:23.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.667+0000 7f621e128700 1 --2- 172.21.15.16:0/3020954338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f621806aef0 0x7f62180cb980 secure :-1 s=READY pgs=2093 cs=0 l=1 rev1=1 crypto rx=0x7f620c0098f0 tx=0x7f620c009c00 comp rx=0 tx=0).stop 2023-10-16T13:28:23.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.667+0000 7f621e128700 1 -- 172.21.15.16:0/3020954338 shutdown_connections 2023-10-16T13:28:23.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.667+0000 7f621e128700 1 --2- 172.21.15.16:0/3020954338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f621806aef0 0x7f62180cb980 unknown :-1 s=CLOSED pgs=2093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:23.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.667+0000 7f621e128700 1 -- 172.21.15.16:0/3020954338 >> 172.21.15.16:0/3020954338 conn(0x7f62180b1fd0 msgr2=0x7f62180b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:23.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.667+0000 7f621e128700 1 -- 172.21.15.16:0/3020954338 shutdown_connections 2023-10-16T13:28:23.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.667+0000 7f621e128700 1 -- 172.21.15.16:0/3020954338 wait complete. 2023-10-16T13:28:23.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.668+0000 7f621e128700 1 Processor -- start 2023-10-16T13:28:23.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.668+0000 7f621e128700 1 -- start start 2023-10-16T13:28:23.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.669+0000 7f621e128700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f621806aef0 0x7f62181586a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:23.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.669+0000 7f621e128700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f62180ce160 0x7f62181f3a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:23.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.669+0000 7f621e128700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f621815db40 0x7f62181604c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:23.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.669+0000 7f621e128700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6218158be0 con 0x7f621806aef0 2023-10-16T13:28:23.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.669+0000 7f621e128700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f621815ea10 con 0x7f62180ce160 2023-10-16T13:28:23.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.669+0000 7f621e128700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62181f3f40 con 0x7f621815db40 2023-10-16T13:28:23.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.669+0000 7f62177fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f621806aef0 0x7f62181586a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:23.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.669+0000 7f62177fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f621806aef0 0x7f62181586a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49892/0 (socket says 172.21.15.16:49892) 2023-10-16T13:28:23.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.669+0000 7f62177fe700 1 -- 172.21.15.16:0/3937861058 learned_addr learned my addr 172.21.15.16:0/3937861058 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:23.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.669+0000 7f6217fff700 1 --2- 172.21.15.16:0/3937861058 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f621815db40 0x7f62181604c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:23.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.669+0000 7f62177fe700 1 -- 172.21.15.16:0/3937861058 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f621815db40 msgr2=0x7f62181604c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:23.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.669+0000 7f62177fe700 1 --2- 172.21.15.16:0/3937861058 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f621815db40 0x7f62181604c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:23.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.669+0000 7f62177fe700 1 -- 172.21.15.16:0/3937861058 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f62180ce160 msgr2=0x7f62181f3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:23.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.669+0000 7f62177fe700 1 --2- 172.21.15.16:0/3937861058 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f62180ce160 0x7f62181f3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:23.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.669+0000 7f62177fe700 1 -- 172.21.15.16:0/3937861058 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f620c009550 con 0x7f621806aef0 2023-10-16T13:28:23.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.669+0000 7f62177fe700 1 --2- 172.21.15.16:0/3937861058 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f621806aef0 0x7f62181586a0 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 crypto rx=0x7f620800c4e0 tx=0x7f620800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:23.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.670+0000 7f61fffff700 1 -- 172.21.15.16:0/3937861058 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62080052d0 con 0x7f621806aef0 2023-10-16T13:28:23.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.670+0000 7f61fffff700 1 -- 172.21.15.16:0/3937861058 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6208005430 con 0x7f621806aef0 2023-10-16T13:28:23.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.670+0000 7f621e128700 1 -- 172.21.15.16:0/3937861058 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62180afbf0 con 0x7f621806aef0 2023-10-16T13:28:23.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.670+0000 7f61fffff700 1 -- 172.21.15.16:0/3937861058 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62080038c0 con 0x7f621806aef0 2023-10-16T13:28:23.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.670+0000 7f621e128700 1 -- 172.21.15.16:0/3937861058 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62180c8690 con 0x7f621806aef0 2023-10-16T13:28:23.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.670+0000 7f621e128700 1 -- 172.21.15.16:0/3937861058 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f621804ebb0 con 0x7f621806aef0 2023-10-16T13:28:23.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.671+0000 7f61fffff700 1 -- 172.21.15.16:0/3937861058 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 107) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6208003a20 con 0x7f621806aef0 2023-10-16T13:28:23.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.672+0000 7f61fffff700 1 --2- 172.21.15.16:0/3937861058 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f620406b020 0x7f620406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:23.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.672+0000 7f6216ffd700 1 --2- 172.21.15.16:0/3937861058 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f620406b020 0x7f620406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:23.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.672+0000 7f61fffff700 1 -- 172.21.15.16:0/3937861058 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f6208088df0 con 0x7f621806aef0 2023-10-16T13:28:23.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.672+0000 7f6216ffd700 1 --2- 172.21.15.16:0/3937861058 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f620406b020 0x7f620406d4e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f620c014040 tx=0x7f620c010000 comp rx=0 tx=0).ready entity=mgr.10047 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:23.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.676+0000 7f61fffff700 1 -- 172.21.15.16:0/3937861058 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6208053bd0 con 0x7f621806aef0 2023-10-16T13:28:23.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.881+0000 7f621e128700 1 -- 172.21.15.16:0/3937861058 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f62180cbc60 con 0x7f621806aef0 2023-10-16T13:28:23.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.882+0000 7f61fffff700 1 -- 172.21.15.16:0/3937861058 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v190) v1 ==== 74+0+16767 (secure 0 0 0) 0x7f6208017090 con 0x7f621806aef0 2023-10-16T13:28:23.885 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:28:23.885 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":190,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:28:21.086257+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2023-10-16T13:28:17.549815+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"189","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2023-10-16T13:28:18.493327+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"190","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:28:23.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.884+0000 7f621e128700 1 -- 172.21.15.16:0/3937861058 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f620406b020 msgr2=0x7f620406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:23.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.884+0000 7f621e128700 1 --2- 172.21.15.16:0/3937861058 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f620406b020 0x7f620406d4e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f620c014040 tx=0x7f620c010000 comp rx=0 tx=0).stop 2023-10-16T13:28:23.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.884+0000 7f621e128700 1 -- 172.21.15.16:0/3937861058 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f621806aef0 msgr2=0x7f62181586a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:23.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.884+0000 7f621e128700 1 --2- 172.21.15.16:0/3937861058 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f621806aef0 0x7f62181586a0 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 crypto rx=0x7f620800c4e0 tx=0x7f620800c8a0 comp rx=0 tx=0).stop 2023-10-16T13:28:23.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.884+0000 7f621e128700 1 -- 172.21.15.16:0/3937861058 shutdown_connections 2023-10-16T13:28:23.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.884+0000 7f621e128700 1 --2- 172.21.15.16:0/3937861058 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f621815db40 0x7f62181604c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:23.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.884+0000 7f621e128700 1 --2- 172.21.15.16:0/3937861058 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f621806aef0 0x7f62181586a0 unknown :-1 s=CLOSED pgs=2094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:23.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.884+0000 7f621e128700 1 --2- 172.21.15.16:0/3937861058 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f620406b020 0x7f620406d4e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:23.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.884+0000 7f621e128700 1 --2- 172.21.15.16:0/3937861058 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f62180ce160 0x7f62181f3a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:23.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.884+0000 7f621e128700 1 -- 172.21.15.16:0/3937861058 >> 172.21.15.16:0/3937861058 conn(0x7f62180b1fd0 msgr2=0x7f62180cd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:23.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.884+0000 7f621e128700 1 -- 172.21.15.16:0/3937861058 shutdown_connections 2023-10-16T13:28:23.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:23.885+0000 7f621e128700 1 -- 172.21.15.16:0/3937861058 wait complete. 2023-10-16T13:28:23.900 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-16T13:28:23.900 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T13:28:24.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.023+0000 7f520995e700 1 Processor -- start 2023-10-16T13:28:24.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.023+0000 7f520995e700 1 -- start start 2023-10-16T13:28:24.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.024+0000 7f520995e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f520415d8d0 0x7f52040d1c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:24.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.024+0000 7f520995e700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f52040d0750 0x7f520406c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:24.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.024+0000 7f520995e700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f520406ba00 con 0x7f52040d41c0 2023-10-16T13:28:24.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.024+0000 7f520995e700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52040d2150 con 0x7f520415d8d0 2023-10-16T13:28:24.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.024+0000 7f520995e700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52040d26d0 con 0x7f52040d0750 2023-10-16T13:28:24.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.024+0000 7f5202ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f520415d8d0 0x7f52040d1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:24.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.024+0000 7f5202ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f520415d8d0 0x7f52040d1c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49900/0 (socket says 172.21.15.16:49900) 2023-10-16T13:28:24.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.024+0000 7f5202ffd700 1 -- 172.21.15.16:0/4142368587 learned_addr learned my addr 172.21.15.16:0/4142368587 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:24.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.024+0000 7f52037fe700 1 --2- 172.21.15.16:0/4142368587 >> v2:172.21.15.49:3300/0 conn(0x7f52040d0750 0x7f520406c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:24.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.024+0000 7f5202ffd700 1 -- 172.21.15.16:0/4142368587 >> v2:172.21.15.49:3300/0 conn(0x7f52040d0750 msgr2=0x7f520406c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:28:24.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.024+0000 7f5202ffd700 1 --2- 172.21.15.16:0/4142368587 >> v2:172.21.15.49:3300/0 conn(0x7f52040d0750 0x7f520406c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:24.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.024+0000 7f5202ffd700 1 -- 172.21.15.16:0/4142368587 >> v1:172.21.15.16:6789/0 conn(0x7f52040d41c0 legacy=0x7f52040d0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:24.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.024+0000 7f5202ffd700 1 -- 172.21.15.16:0/4142368587 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52040d23f0 con 0x7f520415d8d0 2023-10-16T13:28:24.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.024+0000 7f5202ffd700 1 --2- 172.21.15.16:0/4142368587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f520415d8d0 0x7f52040d1c10 secure :-1 s=READY pgs=2096 cs=0 l=1 rev1=1 crypto rx=0x7f52040d4640 tx=0x7f51f4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6c0817a35ea9a18 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:24.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.024+0000 7f5201ffb700 1 -- 172.21.15.16:0/4142368587 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f400b870 con 0x7f520415d8d0 2023-10-16T13:28:24.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.024+0000 7f5201ffb700 1 -- 172.21.15.16:0/4142368587 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51f400b9d0 con 0x7f520415d8d0 2023-10-16T13:28:24.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.025+0000 7f5201ffb700 1 -- 172.21.15.16:0/4142368587 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f400bd60 con 0x7f520415d8d0 2023-10-16T13:28:24.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.025+0000 7f520995e700 1 -- 172.21.15.16:0/4142368587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f520415d8d0 msgr2=0x7f52040d1c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:24.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.025+0000 7f520995e700 1 --2- 172.21.15.16:0/4142368587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f520415d8d0 0x7f52040d1c10 secure :-1 s=READY pgs=2096 cs=0 l=1 rev1=1 crypto rx=0x7f52040d4640 tx=0x7f51f4009bd0 comp rx=0 tx=0).stop 2023-10-16T13:28:24.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.025+0000 7f520995e700 1 -- 172.21.15.16:0/4142368587 shutdown_connections 2023-10-16T13:28:24.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.025+0000 7f520995e700 1 --2- 172.21.15.16:0/4142368587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f520415d8d0 0x7f52040d1c10 unknown :-1 s=CLOSED pgs=2096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:24.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.025+0000 7f520995e700 1 --2- 172.21.15.16:0/4142368587 >> v2:172.21.15.49:3300/0 conn(0x7f52040d0750 0x7f520406c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:24.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.025+0000 7f520995e700 1 -- 172.21.15.16:0/4142368587 >> 172.21.15.16:0/4142368587 conn(0x7f52040d3630 msgr2=0x7f52040d3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:24.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.025+0000 7f520995e700 1 -- 172.21.15.16:0/4142368587 shutdown_connections 2023-10-16T13:28:24.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.025+0000 7f520995e700 1 -- 172.21.15.16:0/4142368587 wait complete. 2023-10-16T13:28:24.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.026+0000 7f520995e700 1 Processor -- start 2023-10-16T13:28:24.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.026+0000 7f520995e700 1 -- start start 2023-10-16T13:28:24.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.027+0000 7f520995e700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f52040d0750 0x7f52040b0c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:24.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.027+0000 7f520995e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f52040d41c0 0x7f52041f68f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:24.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.027+0000 7f520995e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f520415d8d0 0x7f52041f6e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:24.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.027+0000 7f520995e700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52040c8d80 con 0x7f520415d8d0 2023-10-16T13:28:24.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.027+0000 7f520995e700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52040c8190 con 0x7f52040d0750 2023-10-16T13:28:24.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.027+0000 7f520995e700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52041f7370 con 0x7f52040d41c0 2023-10-16T13:28:24.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.027+0000 7f52037fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f520415d8d0 0x7f52041f6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:24.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.027+0000 7f52037fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f520415d8d0 0x7f52041f6e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49908/0 (socket says 172.21.15.16:49908) 2023-10-16T13:28:24.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.027+0000 7f52037fe700 1 -- 172.21.15.16:0/3310307979 learned_addr learned my addr 172.21.15.16:0/3310307979 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:24.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.028+0000 7f52027fc700 1 --2- 172.21.15.16:0/3310307979 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f52040d41c0 0x7f52041f68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:24.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.028+0000 7f5202ffd700 1 --2- 172.21.15.16:0/3310307979 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f52040d0750 0x7f52040b0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:24.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.028+0000 7f52037fe700 1 -- 172.21.15.16:0/3310307979 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f52040d41c0 msgr2=0x7f52041f68f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:24.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.028+0000 7f52037fe700 1 --2- 172.21.15.16:0/3310307979 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f52040d41c0 0x7f52041f68f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:24.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.028+0000 7f52037fe700 1 -- 172.21.15.16:0/3310307979 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f52040d0750 msgr2=0x7f52040b0c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:24.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.028+0000 7f52037fe700 1 --2- 172.21.15.16:0/3310307979 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f52040d0750 0x7f52040b0c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:24.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.028+0000 7f52037fe700 1 -- 172.21.15.16:0/3310307979 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51f4009550 con 0x7f520415d8d0 2023-10-16T13:28:24.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.028+0000 7f52027fc700 1 --2- 172.21.15.16:0/3310307979 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f52040d41c0 0x7f52041f68f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:28:24.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.028+0000 7f5202ffd700 1 --2- 172.21.15.16:0/3310307979 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f52040d0750 0x7f52040b0c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:28:24.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.028+0000 7f52037fe700 1 --2- 172.21.15.16:0/3310307979 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f520415d8d0 0x7f52041f6e30 secure :-1 s=READY pgs=2097 cs=0 l=1 rev1=1 crypto rx=0x7f51fc007e30 tx=0x7f51fc00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:24.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.028+0000 7f51ebfff700 1 -- 172.21.15.16:0/3310307979 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51fc00eea0 con 0x7f520415d8d0 2023-10-16T13:28:24.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.028+0000 7f51ebfff700 1 -- 172.21.15.16:0/3310307979 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51fc014420 con 0x7f520415d8d0 2023-10-16T13:28:24.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.028+0000 7f520995e700 1 -- 172.21.15.16:0/3310307979 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52040c83e0 con 0x7f520415d8d0 2023-10-16T13:28:24.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.028+0000 7f51ebfff700 1 -- 172.21.15.16:0/3310307979 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51fc0147d0 con 0x7f520415d8d0 2023-10-16T13:28:24.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.028+0000 7f520995e700 1 -- 172.21.15.16:0/3310307979 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52040c89a0 con 0x7f520415d8d0 2023-10-16T13:28:24.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.029+0000 7f520995e700 1 -- 172.21.15.16:0/3310307979 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52040c8ec0 con 0x7f520415d8d0 2023-10-16T13:28:24.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.030+0000 7f51ebfff700 1 -- 172.21.15.16:0/3310307979 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 107) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f51fc014930 con 0x7f520415d8d0 2023-10-16T13:28:24.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.030+0000 7f51ebfff700 1 --2- 172.21.15.16:0/3310307979 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f51f006b020 0x7f51f006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:24.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.030+0000 7f51ebfff700 1 -- 172.21.15.16:0/3310307979 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f51fc08a030 con 0x7f520415d8d0 2023-10-16T13:28:24.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.030+0000 7f5202ffd700 1 --2- 172.21.15.16:0/3310307979 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f51f006b020 0x7f51f006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:24.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.031+0000 7f5202ffd700 1 --2- 172.21.15.16:0/3310307979 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f51f006b020 0x7f51f006d4e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f51f4009130 tx=0x7f51f4008030 comp rx=0 tx=0).ready entity=mgr.10047 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:24.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.035+0000 7f51ebfff700 1 -- 172.21.15.16:0/3310307979 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f52040c8ec0 con 0x7f520415d8d0 2023-10-16T13:28:24.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.237+0000 7f520995e700 1 -- 172.21.15.16:0/3310307979 --> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f520406bf80 con 0x7f51f006b020 2023-10-16T13:28:24.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.246+0000 7f51ebfff700 1 -- 172.21.15.16:0/3310307979 <== mgr.10047 v2:172.21.15.16:6832/4211329181 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289887 (secure 0 0 0) 0x7f51f401f070 con 0x7f51f006b020 2023-10-16T13:28:24.253 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:28:24.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.250+0000 7f520995e700 1 -- 172.21.15.16:0/3310307979 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f51f006b020 msgr2=0x7f51f006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:24.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.250+0000 7f520995e700 1 --2- 172.21.15.16:0/3310307979 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f51f006b020 0x7f51f006d4e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f51f4009130 tx=0x7f51f4008030 comp rx=0 tx=0).stop 2023-10-16T13:28:24.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.250+0000 7f520995e700 1 -- 172.21.15.16:0/3310307979 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f520415d8d0 msgr2=0x7f52041f6e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:24.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.250+0000 7f520995e700 1 --2- 172.21.15.16:0/3310307979 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f520415d8d0 0x7f52041f6e30 secure :-1 s=READY pgs=2097 cs=0 l=1 rev1=1 crypto rx=0x7f51fc007e30 tx=0x7f51fc00e3c0 comp rx=0 tx=0).stop 2023-10-16T13:28:24.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.250+0000 7f520995e700 1 -- 172.21.15.16:0/3310307979 shutdown_connections 2023-10-16T13:28:24.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.250+0000 7f520995e700 1 --2- 172.21.15.16:0/3310307979 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f520415d8d0 0x7f52041f6e30 unknown :-1 s=CLOSED pgs=2097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:24.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.250+0000 7f520995e700 1 --2- 172.21.15.16:0/3310307979 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f51f006b020 0x7f51f006d4e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:24.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.250+0000 7f520995e700 1 --2- 172.21.15.16:0/3310307979 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f52040d0750 0x7f52040b0c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:24.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.250+0000 7f520995e700 1 --2- 172.21.15.16:0/3310307979 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f52040d41c0 0x7f52041f68f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:24.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.250+0000 7f520995e700 1 -- 172.21.15.16:0/3310307979 >> 172.21.15.16:0/3310307979 conn(0x7f52040d3630 msgr2=0x7f520415e3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:24.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.250+0000 7f520995e700 1 -- 172.21.15.16:0/3310307979 shutdown_connections 2023-10-16T13:28:24.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.250+0000 7f520995e700 1 -- 172.21.15.16:0/3310307979 wait complete. 2023-10-16T13:28:24.258 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-10-16T13:28:24.265 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2023-10-16T13:28:23.239677+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1652,"num_read_kb":3396,"num_write":523,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":331,"ondisk_log_size":331,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":358012,"kb_used_data":19720,"kb_used_omap":12,"kb_used_meta":338163,"kb_avail":754616708,"statfs":{"total":773094113280,"available":772727508992,"internally_reserved":0,"allocated":20193280,"data_stored":10834444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12740,"internal_metadata":346279484},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.156362"},"pg_stats":[{"pgid":"24.3f","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.216121+0000","last_change":"2023-10-16T13:28:20.102101+0000","last_active":"2023-10-16T13:28:21.216121+0000","last_peered":"2023-10-16T13:28:21.216121+0000","last_clean":"2023-10-16T13:28:21.216121+0000","last_became_active":"2023-10-16T13:28:20.101891+0000","last_became_peered":"2023-10-16T13:28:20.101891+0000","last_unstale":"2023-10-16T13:28:21.216121+0000","last_undegraded":"2023-10-16T13:28:21.216121+0000","last_fullsized":"2023-10-16T13:28:21.216121+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:36:09.581917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3e","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.216435+0000","last_change":"2023-10-16T13:28:20.108645+0000","last_active":"2023-10-16T13:28:21.216435+0000","last_peered":"2023-10-16T13:28:21.216435+0000","last_clean":"2023-10-16T13:28:21.216435+0000","last_became_active":"2023-10-16T13:28:20.108103+0000","last_became_peered":"2023-10-16T13:28:20.108103+0000","last_unstale":"2023-10-16T13:28:21.216435+0000","last_undegraded":"2023-10-16T13:28:21.216435+0000","last_fullsized":"2023-10-16T13:28:21.216435+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:53:08.194107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3d","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.215578+0000","last_change":"2023-10-16T13:28:20.127594+0000","last_active":"2023-10-16T13:28:21.215578+0000","last_peered":"2023-10-16T13:28:21.215578+0000","last_clean":"2023-10-16T13:28:21.215578+0000","last_became_active":"2023-10-16T13:28:20.127125+0000","last_became_peered":"2023-10-16T13:28:20.127125+0000","last_unstale":"2023-10-16T13:28:21.215578+0000","last_undegraded":"2023-10-16T13:28:21.215578+0000","last_fullsized":"2023-10-16T13:28:21.215578+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:28:41.639458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3c","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.215907+0000","last_change":"2023-10-16T13:28:20.116020+0000","last_active":"2023-10-16T13:28:21.215907+0000","last_peered":"2023-10-16T13:28:21.215907+0000","last_clean":"2023-10-16T13:28:21.215907+0000","last_became_active":"2023-10-16T13:28:20.115108+0000","last_became_peered":"2023-10-16T13:28:20.115108+0000","last_unstale":"2023-10-16T13:28:21.215907+0000","last_undegraded":"2023-10-16T13:28:21.215907+0000","last_fullsized":"2023-10-16T13:28:21.215907+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:08:52.920478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3b","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.090318+0000","last_change":"2023-10-16T13:28:20.100002+0000","last_active":"2023-10-16T13:28:21.090318+0000","last_peered":"2023-10-16T13:28:21.090318+0000","last_clean":"2023-10-16T13:28:21.090318+0000","last_became_active":"2023-10-16T13:28:20.099467+0000","last_became_peered":"2023-10-16T13:28:20.099467+0000","last_unstale":"2023-10-16T13:28:21.090318+0000","last_undegraded":"2023-10-16T13:28:21.090318+0000","last_fullsized":"2023-10-16T13:28:21.090318+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:48:02.876298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.3a","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.216082+0000","last_change":"2023-10-16T13:28:20.109342+0000","last_active":"2023-10-16T13:28:21.216082+0000","last_peered":"2023-10-16T13:28:21.216082+0000","last_clean":"2023-10-16T13:28:21.216082+0000","last_became_active":"2023-10-16T13:28:20.109039+0000","last_became_peered":"2023-10-16T13:28:20.109039+0000","last_unstale":"2023-10-16T13:28:21.216082+0000","last_undegraded":"2023-10-16T13:28:21.216082+0000","last_fullsized":"2023-10-16T13:28:21.216082+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:41:45.307869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.39","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.214203+0000","last_change":"2023-10-16T13:28:20.116945+0000","last_active":"2023-10-16T13:28:21.214203+0000","last_peered":"2023-10-16T13:28:21.214203+0000","last_clean":"2023-10-16T13:28:21.214203+0000","last_became_active":"2023-10-16T13:28:20.116488+0000","last_became_peered":"2023-10-16T13:28:20.116488+0000","last_unstale":"2023-10-16T13:28:21.214203+0000","last_undegraded":"2023-10-16T13:28:21.214203+0000","last_fullsized":"2023-10-16T13:28:21.214203+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:19:33.393496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.38","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.214045+0000","last_change":"2023-10-16T13:28:20.099310+0000","last_active":"2023-10-16T13:28:21.214045+0000","last_peered":"2023-10-16T13:28:21.214045+0000","last_clean":"2023-10-16T13:28:21.214045+0000","last_became_active":"2023-10-16T13:28:20.098957+0000","last_became_peered":"2023-10-16T13:28:20.098957+0000","last_unstale":"2023-10-16T13:28:21.214045+0000","last_undegraded":"2023-10-16T13:28:21.214045+0000","last_fullsized":"2023-10-16T13:28:21.214045+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:19:50.557706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.37","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.214821+0000","last_change":"2023-10-16T13:28:20.106002+0000","last_active":"2023-10-16T13:28:21.214821+0000","last_peered":"2023-10-16T13:28:21.214821+0000","last_clean":"2023-10-16T13:28:21.214821+0000","last_became_active":"2023-10-16T13:28:20.105687+0000","last_became_peered":"2023-10-16T13:28:20.105687+0000","last_unstale":"2023-10-16T13:28:21.214821+0000","last_undegraded":"2023-10-16T13:28:21.214821+0000","last_fullsized":"2023-10-16T13:28:21.214821+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:59:28.632287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.36","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.214810+0000","last_change":"2023-10-16T13:28:20.104608+0000","last_active":"2023-10-16T13:28:21.214810+0000","last_peered":"2023-10-16T13:28:21.214810+0000","last_clean":"2023-10-16T13:28:21.214810+0000","last_became_active":"2023-10-16T13:28:20.104279+0000","last_became_peered":"2023-10-16T13:28:20.104279+0000","last_unstale":"2023-10-16T13:28:21.214810+0000","last_undegraded":"2023-10-16T13:28:21.214810+0000","last_fullsized":"2023-10-16T13:28:21.214810+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:57:06.927477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.35","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.216432+0000","last_change":"2023-10-16T13:28:20.099232+0000","last_active":"2023-10-16T13:28:21.216432+0000","last_peered":"2023-10-16T13:28:21.216432+0000","last_clean":"2023-10-16T13:28:21.216432+0000","last_became_active":"2023-10-16T13:28:20.098914+0000","last_became_peered":"2023-10-16T13:28:20.098914+0000","last_unstale":"2023-10-16T13:28:21.216432+0000","last_undegraded":"2023-10-16T13:28:21.216432+0000","last_fullsized":"2023-10-16T13:28:21.216432+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:40:40.178349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.34","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.216605+0000","last_change":"2023-10-16T13:28:20.109397+0000","last_active":"2023-10-16T13:28:21.216605+0000","last_peered":"2023-10-16T13:28:21.216605+0000","last_clean":"2023-10-16T13:28:21.216605+0000","last_became_active":"2023-10-16T13:28:20.109081+0000","last_became_peered":"2023-10-16T13:28:20.109081+0000","last_unstale":"2023-10-16T13:28:21.216605+0000","last_undegraded":"2023-10-16T13:28:21.216605+0000","last_fullsized":"2023-10-16T13:28:21.216605+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:08:46.302886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.33","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.090473+0000","last_change":"2023-10-16T13:28:20.114290+0000","last_active":"2023-10-16T13:28:21.090473+0000","last_peered":"2023-10-16T13:28:21.090473+0000","last_clean":"2023-10-16T13:28:21.090473+0000","last_became_active":"2023-10-16T13:28:20.112316+0000","last_became_peered":"2023-10-16T13:28:20.112316+0000","last_unstale":"2023-10-16T13:28:21.090473+0000","last_undegraded":"2023-10-16T13:28:21.090473+0000","last_fullsized":"2023-10-16T13:28:21.090473+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:32:58.447860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.32","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.089718+0000","last_change":"2023-10-16T13:28:20.129334+0000","last_active":"2023-10-16T13:28:21.089718+0000","last_peered":"2023-10-16T13:28:21.089718+0000","last_clean":"2023-10-16T13:28:21.089718+0000","last_became_active":"2023-10-16T13:28:20.129040+0000","last_became_peered":"2023-10-16T13:28:20.129040+0000","last_unstale":"2023-10-16T13:28:21.089718+0000","last_undegraded":"2023-10-16T13:28:21.089718+0000","last_fullsized":"2023-10-16T13:28:21.089718+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:19:15.031376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.31","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.215450+0000","last_change":"2023-10-16T13:28:20.098947+0000","last_active":"2023-10-16T13:28:21.215450+0000","last_peered":"2023-10-16T13:28:21.215450+0000","last_clean":"2023-10-16T13:28:21.215450+0000","last_became_active":"2023-10-16T13:28:20.098618+0000","last_became_peered":"2023-10-16T13:28:20.098618+0000","last_unstale":"2023-10-16T13:28:21.215450+0000","last_undegraded":"2023-10-16T13:28:21.215450+0000","last_fullsized":"2023-10-16T13:28:21.215450+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:15:12.164129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.30","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.216379+0000","last_change":"2023-10-16T13:28:20.115722+0000","last_active":"2023-10-16T13:28:21.216379+0000","last_peered":"2023-10-16T13:28:21.216379+0000","last_clean":"2023-10-16T13:28:21.216379+0000","last_became_active":"2023-10-16T13:28:20.115432+0000","last_became_peered":"2023-10-16T13:28:20.115432+0000","last_unstale":"2023-10-16T13:28:21.216379+0000","last_undegraded":"2023-10-16T13:28:21.216379+0000","last_fullsized":"2023-10-16T13:28:21.216379+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:59:18.613450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.2f","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.214920+0000","last_change":"2023-10-16T13:28:20.093755+0000","last_active":"2023-10-16T13:28:21.214920+0000","last_peered":"2023-10-16T13:28:21.214920+0000","last_clean":"2023-10-16T13:28:21.214920+0000","last_became_active":"2023-10-16T13:28:20.093452+0000","last_became_peered":"2023-10-16T13:28:20.093452+0000","last_unstale":"2023-10-16T13:28:21.214920+0000","last_undegraded":"2023-10-16T13:28:21.214920+0000","last_fullsized":"2023-10-16T13:28:21.214920+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:49:00.696789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2e","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.215869+0000","last_change":"2023-10-16T13:28:20.099180+0000","last_active":"2023-10-16T13:28:21.215869+0000","last_peered":"2023-10-16T13:28:21.215869+0000","last_clean":"2023-10-16T13:28:21.215869+0000","last_became_active":"2023-10-16T13:28:20.098418+0000","last_became_peered":"2023-10-16T13:28:20.098418+0000","last_unstale":"2023-10-16T13:28:21.215869+0000","last_undegraded":"2023-10-16T13:28:21.215869+0000","last_fullsized":"2023-10-16T13:28:21.215869+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:27:04.199827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.2d","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.216280+0000","last_change":"2023-10-16T13:28:20.099591+0000","last_active":"2023-10-16T13:28:21.216280+0000","last_peered":"2023-10-16T13:28:21.216280+0000","last_clean":"2023-10-16T13:28:21.216280+0000","last_became_active":"2023-10-16T13:28:20.098426+0000","last_became_peered":"2023-10-16T13:28:20.098426+0000","last_unstale":"2023-10-16T13:28:21.216280+0000","last_undegraded":"2023-10-16T13:28:21.216280+0000","last_fullsized":"2023-10-16T13:28:21.216280+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:17:51.974122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.2c","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.214206+0000","last_change":"2023-10-16T13:28:20.117441+0000","last_active":"2023-10-16T13:28:21.214206+0000","last_peered":"2023-10-16T13:28:21.214206+0000","last_clean":"2023-10-16T13:28:21.214206+0000","last_became_active":"2023-10-16T13:28:20.117148+0000","last_became_peered":"2023-10-16T13:28:20.117148+0000","last_unstale":"2023-10-16T13:28:21.214206+0000","last_undegraded":"2023-10-16T13:28:21.214206+0000","last_fullsized":"2023-10-16T13:28:21.214206+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:41:35.647131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.2b","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.090613+0000","last_change":"2023-10-16T13:28:20.113501+0000","last_active":"2023-10-16T13:28:21.090613+0000","last_peered":"2023-10-16T13:28:21.090613+0000","last_clean":"2023-10-16T13:28:21.090613+0000","last_became_active":"2023-10-16T13:28:20.113147+0000","last_became_peered":"2023-10-16T13:28:20.113147+0000","last_unstale":"2023-10-16T13:28:21.090613+0000","last_undegraded":"2023-10-16T13:28:21.090613+0000","last_fullsized":"2023-10-16T13:28:21.090613+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:06:01.800100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.2a","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.216227+0000","last_change":"2023-10-16T13:28:20.111456+0000","last_active":"2023-10-16T13:28:21.216227+0000","last_peered":"2023-10-16T13:28:21.216227+0000","last_clean":"2023-10-16T13:28:21.216227+0000","last_became_active":"2023-10-16T13:28:20.108931+0000","last_became_peered":"2023-10-16T13:28:20.108931+0000","last_unstale":"2023-10-16T13:28:21.216227+0000","last_undegraded":"2023-10-16T13:28:21.216227+0000","last_fullsized":"2023-10-16T13:28:21.216227+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:25:48.347351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.29","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.089557+0000","last_change":"2023-10-16T13:28:20.102883+0000","last_active":"2023-10-16T13:28:21.089557+0000","last_peered":"2023-10-16T13:28:21.089557+0000","last_clean":"2023-10-16T13:28:21.089557+0000","last_became_active":"2023-10-16T13:28:20.102560+0000","last_became_peered":"2023-10-16T13:28:20.102560+0000","last_unstale":"2023-10-16T13:28:21.089557+0000","last_undegraded":"2023-10-16T13:28:21.089557+0000","last_fullsized":"2023-10-16T13:28:21.089557+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:32:59.618770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.28","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.216550+0000","last_change":"2023-10-16T13:28:20.106077+0000","last_active":"2023-10-16T13:28:21.216550+0000","last_peered":"2023-10-16T13:28:21.216550+0000","last_clean":"2023-10-16T13:28:21.216550+0000","last_became_active":"2023-10-16T13:28:20.105768+0000","last_became_peered":"2023-10-16T13:28:20.105768+0000","last_unstale":"2023-10-16T13:28:21.216550+0000","last_undegraded":"2023-10-16T13:28:21.216550+0000","last_fullsized":"2023-10-16T13:28:21.216550+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:17:03.172142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.27","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.090678+0000","last_change":"2023-10-16T13:28:20.103292+0000","last_active":"2023-10-16T13:28:21.090678+0000","last_peered":"2023-10-16T13:28:21.090678+0000","last_clean":"2023-10-16T13:28:21.090678+0000","last_became_active":"2023-10-16T13:28:20.103011+0000","last_became_peered":"2023-10-16T13:28:20.103011+0000","last_unstale":"2023-10-16T13:28:21.090678+0000","last_undegraded":"2023-10-16T13:28:21.090678+0000","last_fullsized":"2023-10-16T13:28:21.090678+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:22:31.339408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.26","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.214324+0000","last_change":"2023-10-16T13:28:20.116791+0000","last_active":"2023-10-16T13:28:21.214324+0000","last_peered":"2023-10-16T13:28:21.214324+0000","last_clean":"2023-10-16T13:28:21.214324+0000","last_became_active":"2023-10-16T13:28:20.107199+0000","last_became_peered":"2023-10-16T13:28:20.107199+0000","last_unstale":"2023-10-16T13:28:21.214324+0000","last_undegraded":"2023-10-16T13:28:21.214324+0000","last_fullsized":"2023-10-16T13:28:21.214324+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:06:05.629013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.25","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.216230+0000","last_change":"2023-10-16T13:28:20.112444+0000","last_active":"2023-10-16T13:28:21.216230+0000","last_peered":"2023-10-16T13:28:21.216230+0000","last_clean":"2023-10-16T13:28:21.216230+0000","last_became_active":"2023-10-16T13:28:20.112149+0000","last_became_peered":"2023-10-16T13:28:20.112149+0000","last_unstale":"2023-10-16T13:28:21.216230+0000","last_undegraded":"2023-10-16T13:28:21.216230+0000","last_fullsized":"2023-10-16T13:28:21.216230+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:42:51.321596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.24","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.090302+0000","last_change":"2023-10-16T13:28:20.111638+0000","last_active":"2023-10-16T13:28:21.090302+0000","last_peered":"2023-10-16T13:28:21.090302+0000","last_clean":"2023-10-16T13:28:21.090302+0000","last_became_active":"2023-10-16T13:28:20.111162+0000","last_became_peered":"2023-10-16T13:28:20.111162+0000","last_unstale":"2023-10-16T13:28:21.090302+0000","last_undegraded":"2023-10-16T13:28:21.090302+0000","last_fullsized":"2023-10-16T13:28:21.090302+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:40:37.979224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.23","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.215586+0000","last_change":"2023-10-16T13:28:20.107895+0000","last_active":"2023-10-16T13:28:21.215586+0000","last_peered":"2023-10-16T13:28:21.215586+0000","last_clean":"2023-10-16T13:28:21.215586+0000","last_became_active":"2023-10-16T13:28:20.106225+0000","last_became_peered":"2023-10-16T13:28:20.106225+0000","last_unstale":"2023-10-16T13:28:21.215586+0000","last_undegraded":"2023-10-16T13:28:21.215586+0000","last_fullsized":"2023-10-16T13:28:21.215586+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:40:08.759737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.22","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.215453+0000","last_change":"2023-10-16T13:28:20.099102+0000","last_active":"2023-10-16T13:28:21.215453+0000","last_peered":"2023-10-16T13:28:21.215453+0000","last_clean":"2023-10-16T13:28:21.215453+0000","last_became_active":"2023-10-16T13:28:20.098824+0000","last_became_peered":"2023-10-16T13:28:20.098824+0000","last_unstale":"2023-10-16T13:28:21.215453+0000","last_undegraded":"2023-10-16T13:28:21.215453+0000","last_fullsized":"2023-10-16T13:28:21.215453+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:22:51.779612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.21","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.215714+0000","last_change":"2023-10-16T13:28:20.094545+0000","last_active":"2023-10-16T13:28:21.215714+0000","last_peered":"2023-10-16T13:28:21.215714+0000","last_clean":"2023-10-16T13:28:21.215714+0000","last_became_active":"2023-10-16T13:28:20.094132+0000","last_became_peered":"2023-10-16T13:28:20.094132+0000","last_unstale":"2023-10-16T13:28:21.215714+0000","last_undegraded":"2023-10-16T13:28:21.215714+0000","last_fullsized":"2023-10-16T13:28:21.215714+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:07:30.602878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.20","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.215377+0000","last_change":"2023-10-16T13:28:20.105432+0000","last_active":"2023-10-16T13:28:21.215377+0000","last_peered":"2023-10-16T13:28:21.215377+0000","last_clean":"2023-10-16T13:28:21.215377+0000","last_became_active":"2023-10-16T13:28:20.105119+0000","last_became_peered":"2023-10-16T13:28:20.105119+0000","last_unstale":"2023-10-16T13:28:21.215377+0000","last_undegraded":"2023-10-16T13:28:21.215377+0000","last_fullsized":"2023-10-16T13:28:21.215377+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:17:43.138771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.10","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.215338+0000","last_change":"2023-10-16T13:28:20.112431+0000","last_active":"2023-10-16T13:28:21.215338+0000","last_peered":"2023-10-16T13:28:21.215338+0000","last_clean":"2023-10-16T13:28:21.215338+0000","last_became_active":"2023-10-16T13:28:20.112069+0000","last_became_peered":"2023-10-16T13:28:20.112069+0000","last_unstale":"2023-10-16T13:28:21.215338+0000","last_undegraded":"2023-10-16T13:28:21.215338+0000","last_fullsized":"2023-10-16T13:28:21.215338+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:53:47.291050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"189'4","reported_seq":16,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.216279+0000","last_change":"2023-10-16T13:28:19.104524+0000","last_active":"2023-10-16T13:28:21.216279+0000","last_peered":"2023-10-16T13:28:21.216279+0000","last_clean":"2023-10-16T13:28:21.216279+0000","last_became_active":"2023-10-16T13:28:19.104244+0000","last_became_peered":"2023-10-16T13:28:19.104244+0000","last_unstale":"2023-10-16T13:28:21.216279+0000","last_undegraded":"2023-10-16T13:28:21.216279+0000","last_fullsized":"2023-10-16T13:28:21.216279+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_clean_scrub_stamp":"2023-10-16T13:28:18.078605+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:35:49.979057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.090535+0000","last_change":"2023-10-16T13:28:20.101142+0000","last_active":"2023-10-16T13:28:21.090535+0000","last_peered":"2023-10-16T13:28:21.090535+0000","last_clean":"2023-10-16T13:28:21.090535+0000","last_became_active":"2023-10-16T13:28:20.100040+0000","last_became_peered":"2023-10-16T13:28:20.100040+0000","last_unstale":"2023-10-16T13:28:21.090535+0000","last_undegraded":"2023-10-16T13:28:21.090535+0000","last_fullsized":"2023-10-16T13:28:21.090535+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:22:21.987373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"189'1","reported_seq":13,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.090661+0000","last_change":"2023-10-16T13:28:19.105174+0000","last_active":"2023-10-16T13:28:21.090661+0000","last_peered":"2023-10-16T13:28:21.090661+0000","last_clean":"2023-10-16T13:28:21.090661+0000","last_became_active":"2023-10-16T13:28:19.104911+0000","last_became_peered":"2023-10-16T13:28:19.104911+0000","last_unstale":"2023-10-16T13:28:21.090661+0000","last_undegraded":"2023-10-16T13:28:21.090661+0000","last_fullsized":"2023-10-16T13:28:21.090661+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_clean_scrub_stamp":"2023-10-16T13:28:18.078605+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:41:44.303899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.089829+0000","last_change":"2023-10-16T13:28:20.129907+0000","last_active":"2023-10-16T13:28:21.089829+0000","last_peered":"2023-10-16T13:28:21.089829+0000","last_clean":"2023-10-16T13:28:21.089829+0000","last_became_active":"2023-10-16T13:28:20.129420+0000","last_became_peered":"2023-10-16T13:28:20.129420+0000","last_unstale":"2023-10-16T13:28:21.089829+0000","last_undegraded":"2023-10-16T13:28:21.089829+0000","last_fullsized":"2023-10-16T13:28:21.089829+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:53:08.194107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.1d","version":"189'2","reported_seq":14,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.216502+0000","last_change":"2023-10-16T13:28:19.105224+0000","last_active":"2023-10-16T13:28:21.216502+0000","last_peered":"2023-10-16T13:28:21.216502+0000","last_clean":"2023-10-16T13:28:21.216502+0000","last_became_active":"2023-10-16T13:28:19.104943+0000","last_became_peered":"2023-10-16T13:28:19.104943+0000","last_unstale":"2023-10-16T13:28:21.216502+0000","last_undegraded":"2023-10-16T13:28:21.216502+0000","last_fullsized":"2023-10-16T13:28:21.216502+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_clean_scrub_stamp":"2023-10-16T13:28:18.078605+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:30:52.074608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.13","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.215739+0000","last_change":"2023-10-16T13:28:20.112381+0000","last_active":"2023-10-16T13:28:21.215739+0000","last_peered":"2023-10-16T13:28:21.215739+0000","last_clean":"2023-10-16T13:28:21.215739+0000","last_became_active":"2023-10-16T13:28:20.112023+0000","last_became_peered":"2023-10-16T13:28:20.112023+0000","last_unstale":"2023-10-16T13:28:21.215739+0000","last_undegraded":"2023-10-16T13:28:21.215739+0000","last_fullsized":"2023-10-16T13:28:21.215739+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:51:19.846445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1c","version":"189'2","reported_seq":14,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.214302+0000","last_change":"2023-10-16T13:28:19.100575+0000","last_active":"2023-10-16T13:28:21.214302+0000","last_peered":"2023-10-16T13:28:21.214302+0000","last_clean":"2023-10-16T13:28:21.214302+0000","last_became_active":"2023-10-16T13:28:19.098362+0000","last_became_peered":"2023-10-16T13:28:19.098362+0000","last_unstale":"2023-10-16T13:28:21.214302+0000","last_undegraded":"2023-10-16T13:28:21.214302+0000","last_fullsized":"2023-10-16T13:28:21.214302+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_clean_scrub_stamp":"2023-10-16T13:28:18.078605+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:07:44.579395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.14","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.214514+0000","last_change":"2023-10-16T13:28:20.098841+0000","last_active":"2023-10-16T13:28:21.214514+0000","last_peered":"2023-10-16T13:28:21.214514+0000","last_clean":"2023-10-16T13:28:21.214514+0000","last_became_active":"2023-10-16T13:28:20.098478+0000","last_became_peered":"2023-10-16T13:28:20.098478+0000","last_unstale":"2023-10-16T13:28:21.214514+0000","last_undegraded":"2023-10-16T13:28:21.214514+0000","last_fullsized":"2023-10-16T13:28:21.214514+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:55:38.929310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1b","version":"189'1","reported_seq":13,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.214634+0000","last_change":"2023-10-16T13:28:19.110059+0000","last_active":"2023-10-16T13:28:21.214634+0000","last_peered":"2023-10-16T13:28:21.214634+0000","last_clean":"2023-10-16T13:28:21.214634+0000","last_became_active":"2023-10-16T13:28:19.109771+0000","last_became_peered":"2023-10-16T13:28:19.109771+0000","last_unstale":"2023-10-16T13:28:21.214634+0000","last_undegraded":"2023-10-16T13:28:21.214634+0000","last_fullsized":"2023-10-16T13:28:21.214634+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_clean_scrub_stamp":"2023-10-16T13:28:18.078605+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:52:28.628298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.15","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.214564+0000","last_change":"2023-10-16T13:28:20.109881+0000","last_active":"2023-10-16T13:28:21.214564+0000","last_peered":"2023-10-16T13:28:21.214564+0000","last_clean":"2023-10-16T13:28:21.214564+0000","last_became_active":"2023-10-16T13:28:20.109597+0000","last_became_peered":"2023-10-16T13:28:20.109597+0000","last_unstale":"2023-10-16T13:28:21.214564+0000","last_undegraded":"2023-10-16T13:28:21.214564+0000","last_fullsized":"2023-10-16T13:28:21.214564+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:28:42.998553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.1a","version":"189'1","reported_seq":13,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.089846+0000","last_change":"2023-10-16T13:28:19.106310+0000","last_active":"2023-10-16T13:28:21.089846+0000","last_peered":"2023-10-16T13:28:21.089846+0000","last_clean":"2023-10-16T13:28:21.089846+0000","last_became_active":"2023-10-16T13:28:19.105968+0000","last_became_peered":"2023-10-16T13:28:19.105968+0000","last_unstale":"2023-10-16T13:28:21.089846+0000","last_undegraded":"2023-10-16T13:28:21.089846+0000","last_fullsized":"2023-10-16T13:28:21.089846+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_clean_scrub_stamp":"2023-10-16T13:28:18.078605+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:42:50.317626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.214462+0000","last_change":"2023-10-16T13:28:20.109833+0000","last_active":"2023-10-16T13:28:21.214462+0000","last_peered":"2023-10-16T13:28:21.214462+0000","last_clean":"2023-10-16T13:28:21.214462+0000","last_became_active":"2023-10-16T13:28:20.104089+0000","last_became_peered":"2023-10-16T13:28:20.104089+0000","last_unstale":"2023-10-16T13:28:21.214462+0000","last_undegraded":"2023-10-16T13:28:21.214462+0000","last_fullsized":"2023-10-16T13:28:21.214462+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:31:25.372564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":12,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.213943+0000","last_change":"2023-10-16T13:28:19.107783+0000","last_active":"2023-10-16T13:28:21.213943+0000","last_peered":"2023-10-16T13:28:21.213943+0000","last_clean":"2023-10-16T13:28:21.213943+0000","last_became_active":"2023-10-16T13:28:19.107466+0000","last_became_peered":"2023-10-16T13:28:19.107466+0000","last_unstale":"2023-10-16T13:28:21.213943+0000","last_undegraded":"2023-10-16T13:28:21.213943+0000","last_fullsized":"2023-10-16T13:28:21.213943+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_clean_scrub_stamp":"2023-10-16T13:28:18.078605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:00:29.699489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.216546+0000","last_change":"2023-10-16T13:28:20.103276+0000","last_active":"2023-10-16T13:28:21.216546+0000","last_peered":"2023-10-16T13:28:21.216546+0000","last_clean":"2023-10-16T13:28:21.216546+0000","last_became_active":"2023-10-16T13:28:20.102935+0000","last_became_peered":"2023-10-16T13:28:20.102935+0000","last_unstale":"2023-10-16T13:28:21.216546+0000","last_undegraded":"2023-10-16T13:28:21.216546+0000","last_fullsized":"2023-10-16T13:28:21.216546+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:19:49.628069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":12,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.214042+0000","last_change":"2023-10-16T13:28:19.103446+0000","last_active":"2023-10-16T13:28:21.214042+0000","last_peered":"2023-10-16T13:28:21.214042+0000","last_clean":"2023-10-16T13:28:21.214042+0000","last_became_active":"2023-10-16T13:28:19.102980+0000","last_became_peered":"2023-10-16T13:28:19.102980+0000","last_unstale":"2023-10-16T13:28:21.214042+0000","last_undegraded":"2023-10-16T13:28:21.214042+0000","last_fullsized":"2023-10-16T13:28:21.214042+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_clean_scrub_stamp":"2023-10-16T13:28:18.078605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:10:13.208037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.215167+0000","last_change":"2023-10-16T13:28:20.112844+0000","last_active":"2023-10-16T13:28:21.215167+0000","last_peered":"2023-10-16T13:28:21.215167+0000","last_clean":"2023-10-16T13:28:21.215167+0000","last_became_active":"2023-10-16T13:28:20.112558+0000","last_became_peered":"2023-10-16T13:28:20.112558+0000","last_unstale":"2023-10-16T13:28:21.215167+0000","last_undegraded":"2023-10-16T13:28:21.215167+0000","last_fullsized":"2023-10-16T13:28:21.215167+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:40:14.995519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":12,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.090729+0000","last_change":"2023-10-16T13:28:19.101765+0000","last_active":"2023-10-16T13:28:21.090729+0000","last_peered":"2023-10-16T13:28:21.090729+0000","last_clean":"2023-10-16T13:28:21.090729+0000","last_became_active":"2023-10-16T13:28:19.101518+0000","last_became_peered":"2023-10-16T13:28:19.101518+0000","last_unstale":"2023-10-16T13:28:21.090729+0000","last_undegraded":"2023-10-16T13:28:21.090729+0000","last_fullsized":"2023-10-16T13:28:21.090729+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_clean_scrub_stamp":"2023-10-16T13:28:18.078605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:25:47.343382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.090923+0000","last_change":"2023-10-16T13:28:20.115429+0000","last_active":"2023-10-16T13:28:21.090923+0000","last_peered":"2023-10-16T13:28:21.090923+0000","last_clean":"2023-10-16T13:28:21.090923+0000","last_became_active":"2023-10-16T13:28:20.115090+0000","last_became_peered":"2023-10-16T13:28:20.115090+0000","last_unstale":"2023-10-16T13:28:21.090923+0000","last_undegraded":"2023-10-16T13:28:21.090923+0000","last_fullsized":"2023-10-16T13:28:21.090923+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:34:43.083147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.15","version":"189'3","reported_seq":15,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.215683+0000","last_change":"2023-10-16T13:28:19.098263+0000","last_active":"2023-10-16T13:28:21.215683+0000","last_peered":"2023-10-16T13:28:21.215683+0000","last_clean":"2023-10-16T13:28:21.215683+0000","last_became_active":"2023-10-16T13:28:19.097824+0000","last_became_peered":"2023-10-16T13:28:19.097824+0000","last_unstale":"2023-10-16T13:28:21.215683+0000","last_undegraded":"2023-10-16T13:28:21.215683+0000","last_fullsized":"2023-10-16T13:28:21.215683+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_clean_scrub_stamp":"2023-10-16T13:28:18.078605+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:23:12.689030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1b","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.214673+0000","last_change":"2023-10-16T13:28:20.127366+0000","last_active":"2023-10-16T13:28:21.214673+0000","last_peered":"2023-10-16T13:28:21.214673+0000","last_clean":"2023-10-16T13:28:21.214673+0000","last_became_active":"2023-10-16T13:28:20.127030+0000","last_became_peered":"2023-10-16T13:28:20.127030+0000","last_unstale":"2023-10-16T13:28:21.214673+0000","last_undegraded":"2023-10-16T13:28:21.214673+0000","last_fullsized":"2023-10-16T13:28:21.214673+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:01:21.020025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"189'6","reported_seq":18,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.090546+0000","last_change":"2023-10-16T13:28:19.099350+0000","last_active":"2023-10-16T13:28:21.090546+0000","last_peered":"2023-10-16T13:28:21.090546+0000","last_clean":"2023-10-16T13:28:21.090546+0000","last_became_active":"2023-10-16T13:28:19.099093+0000","last_became_peered":"2023-10-16T13:28:19.099093+0000","last_unstale":"2023-10-16T13:28:21.090546+0000","last_undegraded":"2023-10-16T13:28:21.090546+0000","last_fullsized":"2023-10-16T13:28:21.090546+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_clean_scrub_stamp":"2023-10-16T13:28:18.078605+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:49:33.267714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1c","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.215238+0000","last_change":"2023-10-16T13:28:20.110489+0000","last_active":"2023-10-16T13:28:21.215238+0000","last_peered":"2023-10-16T13:28:21.215238+0000","last_clean":"2023-10-16T13:28:21.215238+0000","last_became_active":"2023-10-16T13:28:20.110171+0000","last_became_peered":"2023-10-16T13:28:20.110171+0000","last_unstale":"2023-10-16T13:28:21.215238+0000","last_undegraded":"2023-10-16T13:28:21.215238+0000","last_fullsized":"2023-10-16T13:28:21.215238+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:25:31.239161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.13","version":"189'2","reported_seq":14,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.090371+0000","last_change":"2023-10-16T13:28:19.100376+0000","last_active":"2023-10-16T13:28:21.090371+0000","last_peered":"2023-10-16T13:28:21.090371+0000","last_clean":"2023-10-16T13:28:21.090371+0000","last_became_active":"2023-10-16T13:28:19.100135+0000","last_became_peered":"2023-10-16T13:28:19.100135+0000","last_unstale":"2023-10-16T13:28:21.090371+0000","last_undegraded":"2023-10-16T13:28:21.090371+0000","last_fullsized":"2023-10-16T13:28:21.090371+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_clean_scrub_stamp":"2023-10-16T13:28:18.078605+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:40:13.991550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1d","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.215020+0000","last_change":"2023-10-16T13:28:20.098726+0000","last_active":"2023-10-16T13:28:21.215020+0000","last_peered":"2023-10-16T13:28:21.215020+0000","last_clean":"2023-10-16T13:28:21.215020+0000","last_became_active":"2023-10-16T13:28:20.098176+0000","last_became_peered":"2023-10-16T13:28:20.098176+0000","last_unstale":"2023-10-16T13:28:21.215020+0000","last_undegraded":"2023-10-16T13:28:21.215020+0000","last_fullsized":"2023-10-16T13:28:21.215020+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:30:04.900461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":12,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.090162+0000","last_change":"2023-10-16T13:28:19.102894+0000","last_active":"2023-10-16T13:28:21.090162+0000","last_peered":"2023-10-16T13:28:21.090162+0000","last_clean":"2023-10-16T13:28:21.090162+0000","last_became_active":"2023-10-16T13:28:19.102601+0000","last_became_peered":"2023-10-16T13:28:19.102601+0000","last_unstale":"2023-10-16T13:28:21.090162+0000","last_undegraded":"2023-10-16T13:28:21.090162+0000","last_fullsized":"2023-10-16T13:28:21.090162+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_clean_scrub_stamp":"2023-10-16T13:28:18.078605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:24:16.739829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1e","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.215821+0000","last_change":"2023-10-16T13:28:20.106915+0000","last_active":"2023-10-16T13:28:21.215821+0000","last_peered":"2023-10-16T13:28:21.215821+0000","last_clean":"2023-10-16T13:28:21.215821+0000","last_became_active":"2023-10-16T13:28:20.106673+0000","last_became_peered":"2023-10-16T13:28:20.106673+0000","last_unstale":"2023-10-16T13:28:21.215821+0000","last_undegraded":"2023-10-16T13:28:21.215821+0000","last_fullsized":"2023-10-16T13:28:21.215821+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:57:53.529437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":12,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.215549+0000","last_change":"2023-10-16T13:28:19.105790+0000","last_active":"2023-10-16T13:28:21.215549+0000","last_peered":"2023-10-16T13:28:21.215549+0000","last_clean":"2023-10-16T13:28:21.215549+0000","last_became_active":"2023-10-16T13:28:19.105411+0000","last_became_peered":"2023-10-16T13:28:19.105411+0000","last_unstale":"2023-10-16T13:28:21.215549+0000","last_undegraded":"2023-10-16T13:28:21.215549+0000","last_fullsized":"2023-10-16T13:28:21.215549+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_clean_scrub_stamp":"2023-10-16T13:28:18.078605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:59:27.628317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1f","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.214931+0000","last_change":"2023-10-16T13:28:20.116267+0000","last_active":"2023-10-16T13:28:21.214931+0000","last_peered":"2023-10-16T13:28:21.214931+0000","last_clean":"2023-10-16T13:28:21.214931+0000","last_became_active":"2023-10-16T13:28:20.115967+0000","last_became_peered":"2023-10-16T13:28:20.115967+0000","last_unstale":"2023-10-16T13:28:21.214931+0000","last_undegraded":"2023-10-16T13:28:21.214931+0000","last_fullsized":"2023-10-16T13:28:21.214931+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:28:35.228943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.10","version":"189'8","reported_seq":20,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.214282+0000","last_change":"2023-10-16T13:28:19.099553+0000","last_active":"2023-10-16T13:28:21.214282+0000","last_peered":"2023-10-16T13:28:21.214282+0000","last_clean":"2023-10-16T13:28:21.214282+0000","last_became_active":"2023-10-16T13:28:19.099272+0000","last_became_peered":"2023-10-16T13:28:19.099272+0000","last_unstale":"2023-10-16T13:28:21.214282+0000","last_undegraded":"2023-10-16T13:28:21.214282+0000","last_fullsized":"2023-10-16T13:28:21.214282+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_clean_scrub_stamp":"2023-10-16T13:28:18.078605+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:14:10.839754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.214186+0000","last_change":"2023-10-16T13:28:20.128281+0000","last_active":"2023-10-16T13:28:21.214186+0000","last_peered":"2023-10-16T13:28:21.214186+0000","last_clean":"2023-10-16T13:28:21.214186+0000","last_became_active":"2023-10-16T13:28:20.127986+0000","last_became_peered":"2023-10-16T13:28:20.127986+0000","last_unstale":"2023-10-16T13:28:21.214186+0000","last_undegraded":"2023-10-16T13:28:21.214186+0000","last_fullsized":"2023-10-16T13:28:21.214186+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:19:07.823278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.f","version":"189'1","reported_seq":13,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.090766+0000","last_change":"2023-10-16T13:28:19.100121+0000","last_active":"2023-10-16T13:28:21.090766+0000","last_peered":"2023-10-16T13:28:21.090766+0000","last_clean":"2023-10-16T13:28:21.090766+0000","last_became_active":"2023-10-16T13:28:19.099767+0000","last_became_peered":"2023-10-16T13:28:19.099767+0000","last_unstale":"2023-10-16T13:28:21.090766+0000","last_undegraded":"2023-10-16T13:28:21.090766+0000","last_fullsized":"2023-10-16T13:28:21.090766+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_clean_scrub_stamp":"2023-10-16T13:28:18.078605+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:32:58.614800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.215996+0000","last_change":"2023-10-16T13:28:20.096710+0000","last_active":"2023-10-16T13:28:21.215996+0000","last_peered":"2023-10-16T13:28:21.215996+0000","last_clean":"2023-10-16T13:28:21.215996+0000","last_became_active":"2023-10-16T13:28:20.096293+0000","last_became_peered":"2023-10-16T13:28:20.096293+0000","last_unstale":"2023-10-16T13:28:21.215996+0000","last_undegraded":"2023-10-16T13:28:21.215996+0000","last_fullsized":"2023-10-16T13:28:21.215996+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:53:31.172370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.e","version":"189'2","reported_seq":14,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.214634+0000","last_change":"2023-10-16T13:28:19.108810+0000","last_active":"2023-10-16T13:28:21.214634+0000","last_peered":"2023-10-16T13:28:21.214634+0000","last_clean":"2023-10-16T13:28:21.214634+0000","last_became_active":"2023-10-16T13:28:19.108522+0000","last_became_peered":"2023-10-16T13:28:19.108522+0000","last_unstale":"2023-10-16T13:28:21.214634+0000","last_undegraded":"2023-10-16T13:28:21.214634+0000","last_fullsized":"2023-10-16T13:28:21.214634+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_clean_scrub_stamp":"2023-10-16T13:28:18.078605+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:58:39.810683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.215228+0000","last_change":"2023-10-16T13:28:20.102972+0000","last_active":"2023-10-16T13:28:21.215228+0000","last_peered":"2023-10-16T13:28:21.215228+0000","last_clean":"2023-10-16T13:28:21.215228+0000","last_became_active":"2023-10-16T13:28:20.102654+0000","last_became_peered":"2023-10-16T13:28:20.102654+0000","last_unstale":"2023-10-16T13:28:21.215228+0000","last_undegraded":"2023-10-16T13:28:21.215228+0000","last_fullsized":"2023-10-16T13:28:21.215228+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:15:57.533753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.d","version":"189'1","reported_seq":13,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.090485+0000","last_change":"2023-10-16T13:28:19.101127+0000","last_active":"2023-10-16T13:28:21.090485+0000","last_peered":"2023-10-16T13:28:21.090485+0000","last_clean":"2023-10-16T13:28:21.090485+0000","last_became_active":"2023-10-16T13:28:19.100857+0000","last_became_peered":"2023-10-16T13:28:19.100857+0000","last_unstale":"2023-10-16T13:28:21.090485+0000","last_undegraded":"2023-10-16T13:28:21.090485+0000","last_fullsized":"2023-10-16T13:28:21.090485+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_clean_scrub_stamp":"2023-10-16T13:28:18.078605+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:34:58.602698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.215586+0000","last_change":"2023-10-16T13:28:20.098940+0000","last_active":"2023-10-16T13:28:21.215586+0000","last_peered":"2023-10-16T13:28:21.215586+0000","last_clean":"2023-10-16T13:28:21.215586+0000","last_became_active":"2023-10-16T13:28:20.098644+0000","last_became_peered":"2023-10-16T13:28:20.098644+0000","last_unstale":"2023-10-16T13:28:21.215586+0000","last_undegraded":"2023-10-16T13:28:21.215586+0000","last_fullsized":"2023-10-16T13:28:21.215586+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:20:39.551231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":12,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.214665+0000","last_change":"2023-10-16T13:28:19.109251+0000","last_active":"2023-10-16T13:28:21.214665+0000","last_peered":"2023-10-16T13:28:21.214665+0000","last_clean":"2023-10-16T13:28:21.214665+0000","last_became_active":"2023-10-16T13:28:19.108876+0000","last_became_peered":"2023-10-16T13:28:19.108876+0000","last_unstale":"2023-10-16T13:28:21.214665+0000","last_undegraded":"2023-10-16T13:28:21.214665+0000","last_fullsized":"2023-10-16T13:28:21.214665+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_clean_scrub_stamp":"2023-10-16T13:28:18.078605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:01:27.654076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.090442+0000","last_change":"2023-10-16T13:28:20.111460+0000","last_active":"2023-10-16T13:28:21.090442+0000","last_peered":"2023-10-16T13:28:21.090442+0000","last_clean":"2023-10-16T13:28:21.090442+0000","last_became_active":"2023-10-16T13:28:20.110770+0000","last_became_peered":"2023-10-16T13:28:20.110770+0000","last_unstale":"2023-10-16T13:28:21.090442+0000","last_undegraded":"2023-10-16T13:28:21.090442+0000","last_fullsized":"2023-10-16T13:28:21.090442+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:36:27.353238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.b","version":"189'2","reported_seq":14,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.090419+0000","last_change":"2023-10-16T13:28:19.105265+0000","last_active":"2023-10-16T13:28:21.090419+0000","last_peered":"2023-10-16T13:28:21.090419+0000","last_clean":"2023-10-16T13:28:21.090419+0000","last_became_active":"2023-10-16T13:28:19.104959+0000","last_became_peered":"2023-10-16T13:28:19.104959+0000","last_unstale":"2023-10-16T13:28:21.090419+0000","last_undegraded":"2023-10-16T13:28:21.090419+0000","last_fullsized":"2023-10-16T13:28:21.090419+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_clean_scrub_stamp":"2023-10-16T13:28:18.078605+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:53:46.287080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.090302+0000","last_change":"2023-10-16T13:28:20.104597+0000","last_active":"2023-10-16T13:28:21.090302+0000","last_peered":"2023-10-16T13:28:21.090302+0000","last_clean":"2023-10-16T13:28:21.090302+0000","last_became_active":"2023-10-16T13:28:20.102370+0000","last_became_peered":"2023-10-16T13:28:20.102370+0000","last_unstale":"2023-10-16T13:28:21.090302+0000","last_undegraded":"2023-10-16T13:28:21.090302+0000","last_fullsized":"2023-10-16T13:28:21.090302+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:08:46.302886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":12,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.215334+0000","last_change":"2023-10-16T13:28:19.104160+0000","last_active":"2023-10-16T13:28:21.215334+0000","last_peered":"2023-10-16T13:28:21.215334+0000","last_clean":"2023-10-16T13:28:21.215334+0000","last_became_active":"2023-10-16T13:28:19.103823+0000","last_became_peered":"2023-10-16T13:28:19.103823+0000","last_unstale":"2023-10-16T13:28:21.215334+0000","last_undegraded":"2023-10-16T13:28:21.215334+0000","last_fullsized":"2023-10-16T13:28:21.215334+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_clean_scrub_stamp":"2023-10-16T13:28:18.078605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:48:59.692819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.090623+0000","last_change":"2023-10-16T13:28:20.101156+0000","last_active":"2023-10-16T13:28:21.090623+0000","last_peered":"2023-10-16T13:28:21.090623+0000","last_clean":"2023-10-16T13:28:21.090623+0000","last_became_active":"2023-10-16T13:28:20.100880+0000","last_became_peered":"2023-10-16T13:28:20.100880+0000","last_unstale":"2023-10-16T13:28:21.090623+0000","last_undegraded":"2023-10-16T13:28:21.090623+0000","last_fullsized":"2023-10-16T13:28:21.090623+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:24:17.743799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":12,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.213959+0000","last_change":"2023-10-16T13:28:19.104614+0000","last_active":"2023-10-16T13:28:21.213959+0000","last_peered":"2023-10-16T13:28:21.213959+0000","last_clean":"2023-10-16T13:28:21.213959+0000","last_became_active":"2023-10-16T13:28:19.104290+0000","last_became_peered":"2023-10-16T13:28:19.104290+0000","last_unstale":"2023-10-16T13:28:21.213959+0000","last_undegraded":"2023-10-16T13:28:21.213959+0000","last_fullsized":"2023-10-16T13:28:21.213959+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_clean_scrub_stamp":"2023-10-16T13:28:18.078605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:11:41.561063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.215993+0000","last_change":"2023-10-16T13:28:20.127626+0000","last_active":"2023-10-16T13:28:21.215993+0000","last_peered":"2023-10-16T13:28:21.215993+0000","last_clean":"2023-10-16T13:28:21.215993+0000","last_became_active":"2023-10-16T13:28:20.127377+0000","last_became_peered":"2023-10-16T13:28:20.127377+0000","last_unstale":"2023-10-16T13:28:21.215993+0000","last_undegraded":"2023-10-16T13:28:21.215993+0000","last_fullsized":"2023-10-16T13:28:21.215993+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:34:59.606668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":12,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.215445+0000","last_change":"2023-10-16T13:28:19.102225+0000","last_active":"2023-10-16T13:28:21.215445+0000","last_peered":"2023-10-16T13:28:21.215445+0000","last_clean":"2023-10-16T13:28:21.215445+0000","last_became_active":"2023-10-16T13:28:19.101887+0000","last_became_peered":"2023-10-16T13:28:19.101887+0000","last_unstale":"2023-10-16T13:28:21.215445+0000","last_undegraded":"2023-10-16T13:28:21.215445+0000","last_fullsized":"2023-10-16T13:28:21.215445+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_clean_scrub_stamp":"2023-10-16T13:28:18.078605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:11:00.906437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.215167+0000","last_change":"2023-10-16T13:28:20.093505+0000","last_active":"2023-10-16T13:28:21.215167+0000","last_peered":"2023-10-16T13:28:21.215167+0000","last_clean":"2023-10-16T13:28:21.215167+0000","last_became_active":"2023-10-16T13:28:20.093190+0000","last_became_peered":"2023-10-16T13:28:20.093190+0000","last_unstale":"2023-10-16T13:28:21.215167+0000","last_undegraded":"2023-10-16T13:28:21.215167+0000","last_fullsized":"2023-10-16T13:28:21.215167+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:26:02.101104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.7","version":"189'2","reported_seq":14,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.090733+0000","last_change":"2023-10-16T13:28:19.105641+0000","last_active":"2023-10-16T13:28:21.090733+0000","last_peered":"2023-10-16T13:28:21.090733+0000","last_clean":"2023-10-16T13:28:21.090733+0000","last_became_active":"2023-10-16T13:28:19.105360+0000","last_became_peered":"2023-10-16T13:28:19.105360+0000","last_unstale":"2023-10-16T13:28:21.090733+0000","last_undegraded":"2023-10-16T13:28:21.090733+0000","last_fullsized":"2023-10-16T13:28:21.090733+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_clean_scrub_stamp":"2023-10-16T13:28:18.078605+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:11:00.906437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.214349+0000","last_change":"2023-10-16T13:28:20.116732+0000","last_active":"2023-10-16T13:28:21.214349+0000","last_peered":"2023-10-16T13:28:21.214349+0000","last_clean":"2023-10-16T13:28:21.214349+0000","last_became_active":"2023-10-16T13:28:20.116106+0000","last_became_peered":"2023-10-16T13:28:20.116106+0000","last_unstale":"2023-10-16T13:28:21.214349+0000","last_undegraded":"2023-10-16T13:28:21.214349+0000","last_fullsized":"2023-10-16T13:28:21.214349+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:04:42.792375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.6","version":"0'0","reported_seq":12,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.090626+0000","last_change":"2023-10-16T13:28:19.104697+0000","last_active":"2023-10-16T13:28:21.090626+0000","last_peered":"2023-10-16T13:28:21.090626+0000","last_clean":"2023-10-16T13:28:21.090626+0000","last_became_active":"2023-10-16T13:28:19.104014+0000","last_became_peered":"2023-10-16T13:28:19.104014+0000","last_unstale":"2023-10-16T13:28:21.090626+0000","last_undegraded":"2023-10-16T13:28:21.090626+0000","last_fullsized":"2023-10-16T13:28:21.090626+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_clean_scrub_stamp":"2023-10-16T13:28:18.078605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:08:45.298916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.214046+0000","last_change":"2023-10-16T13:28:20.128832+0000","last_active":"2023-10-16T13:28:21.214046+0000","last_peered":"2023-10-16T13:28:21.214046+0000","last_clean":"2023-10-16T13:28:21.214046+0000","last_became_active":"2023-10-16T13:28:20.128413+0000","last_became_peered":"2023-10-16T13:28:20.128413+0000","last_unstale":"2023-10-16T13:28:21.214046+0000","last_undegraded":"2023-10-16T13:28:21.214046+0000","last_fullsized":"2023-10-16T13:28:21.214046+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:17:30.158648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":12,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.090840+0000","last_change":"2023-10-16T13:28:19.098941+0000","last_active":"2023-10-16T13:28:21.090840+0000","last_peered":"2023-10-16T13:28:21.090840+0000","last_clean":"2023-10-16T13:28:21.090840+0000","last_became_active":"2023-10-16T13:28:19.098663+0000","last_became_peered":"2023-10-16T13:28:19.098663+0000","last_unstale":"2023-10-16T13:28:21.090840+0000","last_undegraded":"2023-10-16T13:28:21.090840+0000","last_fullsized":"2023-10-16T13:28:21.090840+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_clean_scrub_stamp":"2023-10-16T13:28:18.078605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:48:01.872328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.b","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.214535+0000","last_change":"2023-10-16T13:28:20.127422+0000","last_active":"2023-10-16T13:28:21.214535+0000","last_peered":"2023-10-16T13:28:21.214535+0000","last_clean":"2023-10-16T13:28:21.214535+0000","last_became_active":"2023-10-16T13:28:20.127075+0000","last_became_peered":"2023-10-16T13:28:20.127075+0000","last_unstale":"2023-10-16T13:28:21.214535+0000","last_undegraded":"2023-10-16T13:28:21.214535+0000","last_fullsized":"2023-10-16T13:28:21.214535+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:04:05.766711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":12,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.214389+0000","last_change":"2023-10-16T13:28:19.103735+0000","last_active":"2023-10-16T13:28:21.214389+0000","last_peered":"2023-10-16T13:28:21.214389+0000","last_clean":"2023-10-16T13:28:21.214389+0000","last_became_active":"2023-10-16T13:28:19.102055+0000","last_became_peered":"2023-10-16T13:28:19.102055+0000","last_unstale":"2023-10-16T13:28:21.214389+0000","last_undegraded":"2023-10-16T13:28:21.214389+0000","last_fullsized":"2023-10-16T13:28:21.214389+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_clean_scrub_stamp":"2023-10-16T13:28:18.078605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:40:30.583418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.216039+0000","last_change":"2023-10-16T13:28:20.103042+0000","last_active":"2023-10-16T13:28:21.216039+0000","last_peered":"2023-10-16T13:28:21.216039+0000","last_clean":"2023-10-16T13:28:21.216039+0000","last_became_active":"2023-10-16T13:28:20.100745+0000","last_became_peered":"2023-10-16T13:28:20.100745+0000","last_unstale":"2023-10-16T13:28:21.216039+0000","last_undegraded":"2023-10-16T13:28:21.216039+0000","last_fullsized":"2023-10-16T13:28:21.216039+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:55:53.825888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":12,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.216277+0000","last_change":"2023-10-16T13:28:19.106585+0000","last_active":"2023-10-16T13:28:21.216277+0000","last_peered":"2023-10-16T13:28:21.216277+0000","last_clean":"2023-10-16T13:28:21.216277+0000","last_became_active":"2023-10-16T13:28:19.106272+0000","last_became_peered":"2023-10-16T13:28:19.106272+0000","last_unstale":"2023-10-16T13:28:21.216277+0000","last_undegraded":"2023-10-16T13:28:21.216277+0000","last_fullsized":"2023-10-16T13:28:21.216277+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_clean_scrub_stamp":"2023-10-16T13:28:18.078605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:55:36.675550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.d","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.216274+0000","last_change":"2023-10-16T13:28:20.112242+0000","last_active":"2023-10-16T13:28:21.216274+0000","last_peered":"2023-10-16T13:28:21.216274+0000","last_clean":"2023-10-16T13:28:21.216274+0000","last_became_active":"2023-10-16T13:28:20.111971+0000","last_became_peered":"2023-10-16T13:28:20.111971+0000","last_unstale":"2023-10-16T13:28:21.216274+0000","last_undegraded":"2023-10-16T13:28:21.216274+0000","last_fullsized":"2023-10-16T13:28:21.216274+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:32:59.618770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":12,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.090242+0000","last_change":"2023-10-16T13:28:19.091686+0000","last_active":"2023-10-16T13:28:21.090242+0000","last_peered":"2023-10-16T13:28:21.090242+0000","last_clean":"2023-10-16T13:28:21.090242+0000","last_became_active":"2023-10-16T13:28:19.091391+0000","last_became_peered":"2023-10-16T13:28:19.091391+0000","last_unstale":"2023-10-16T13:28:21.090242+0000","last_undegraded":"2023-10-16T13:28:21.090242+0000","last_fullsized":"2023-10-16T13:28:21.090242+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_clean_scrub_stamp":"2023-10-16T13:28:18.078605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:29:26.913110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.e","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.090418+0000","last_change":"2023-10-16T13:28:20.111062+0000","last_active":"2023-10-16T13:28:21.090418+0000","last_peered":"2023-10-16T13:28:21.090418+0000","last_clean":"2023-10-16T13:28:21.090418+0000","last_became_active":"2023-10-16T13:28:20.110750+0000","last_became_peered":"2023-10-16T13:28:20.110750+0000","last_unstale":"2023-10-16T13:28:21.090418+0000","last_undegraded":"2023-10-16T13:28:21.090418+0000","last_fullsized":"2023-10-16T13:28:21.090418+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:43:10.467090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":12,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.215817+0000","last_change":"2023-10-16T13:28:19.105211+0000","last_active":"2023-10-16T13:28:21.215817+0000","last_peered":"2023-10-16T13:28:21.215817+0000","last_clean":"2023-10-16T13:28:21.215817+0000","last_became_active":"2023-10-16T13:28:19.104879+0000","last_became_peered":"2023-10-16T13:28:19.104879+0000","last_unstale":"2023-10-16T13:28:21.215817+0000","last_undegraded":"2023-10-16T13:28:21.215817+0000","last_fullsized":"2023-10-16T13:28:21.215817+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_clean_scrub_stamp":"2023-10-16T13:28:18.078605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:30:03.896491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.f","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.214642+0000","last_change":"2023-10-16T13:28:20.118722+0000","last_active":"2023-10-16T13:28:21.214642+0000","last_peered":"2023-10-16T13:28:21.214642+0000","last_clean":"2023-10-16T13:28:21.214642+0000","last_became_active":"2023-10-16T13:28:20.117253+0000","last_became_peered":"2023-10-16T13:28:20.117253+0000","last_unstale":"2023-10-16T13:28:21.214642+0000","last_undegraded":"2023-10-16T13:28:21.214642+0000","last_fullsized":"2023-10-16T13:28:21.214642+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:33:47.029032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":12,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.089405+0000","last_change":"2023-10-16T13:28:19.106788+0000","last_active":"2023-10-16T13:28:21.089405+0000","last_peered":"2023-10-16T13:28:21.089405+0000","last_clean":"2023-10-16T13:28:21.089405+0000","last_became_active":"2023-10-16T13:28:19.105953+0000","last_became_peered":"2023-10-16T13:28:19.105953+0000","last_unstale":"2023-10-16T13:28:21.089405+0000","last_undegraded":"2023-10-16T13:28:21.089405+0000","last_fullsized":"2023-10-16T13:28:21.089405+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_clean_scrub_stamp":"2023-10-16T13:28:18.078605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:17:50.970152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.216324+0000","last_change":"2023-10-16T13:28:20.105567+0000","last_active":"2023-10-16T13:28:21.216324+0000","last_peered":"2023-10-16T13:28:21.216324+0000","last_clean":"2023-10-16T13:28:21.216324+0000","last_became_active":"2023-10-16T13:28:20.105296+0000","last_became_peered":"2023-10-16T13:28:20.105296+0000","last_unstale":"2023-10-16T13:28:21.216324+0000","last_undegraded":"2023-10-16T13:28:21.216324+0000","last_fullsized":"2023-10-16T13:28:21.216324+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:19:15.031376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":12,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.090518+0000","last_change":"2023-10-16T13:28:19.104937+0000","last_active":"2023-10-16T13:28:21.090518+0000","last_peered":"2023-10-16T13:28:21.090518+0000","last_clean":"2023-10-16T13:28:21.090518+0000","last_became_active":"2023-10-16T13:28:19.104398+0000","last_became_peered":"2023-10-16T13:28:19.104398+0000","last_unstale":"2023-10-16T13:28:21.090518+0000","last_undegraded":"2023-10-16T13:28:21.090518+0000","last_fullsized":"2023-10-16T13:28:21.090518+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_clean_scrub_stamp":"2023-10-16T13:28:18.078605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:19:14.027406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"180'293","reported_seq":969,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-10-16T13:28:20.101683+0000","last_change":"2023-10-16T12:10:34.746820+0000","last_active":"2023-10-16T13:28:20.101683+0000","last_peered":"2023-10-16T13:28:20.101683+0000","last_clean":"2023-10-16T13:28:20.101683+0000","last_became_active":"2023-10-16T12:10:34.745574+0000","last_became_peered":"2023-10-16T12:10:34.745574+0000","last_unstale":"2023-10-16T13:28:20.101683+0000","last_undegraded":"2023-10-16T13:28:20.101683+0000","last_fullsized":"2023-10-16T13:28:20.101683+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_clean_scrub_stamp":"2023-10-16T12:10:33.577111+0000","objects_scrubbed":0,"log_size":293,"log_dups_size":0,"ondisk_log_size":293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:14.461986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1652,"num_read_kb":3396,"num_write":479,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":24,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":23,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1652,"num_read_kb":3396,"num_write":479,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":293,"ondisk_log_size":293,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575773,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52120,"kb_used_data":2892,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319720,"statfs":{"total":96636764160,"available":96583393280,"internally_reserved":0,"allocated":2961408,"data_stored":1796479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":50395595},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:28:07 2023","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.66400000000000003,"15min":0.64000000000000001},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":3.218,"5min":3.218,"15min":3.218},"last":0.53800000000000003},{"interface":"front","average":{"1min":1.0229999999999999,"5min":0.71399999999999997,"15min":0.65600000000000003},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":9.1270000000000007,"5min":9.1270000000000007,"15min":9.1270000000000007},"last":0.78200000000000003}]},{"osd":1,"last update":"Mon Oct 16 13:28:07 2023","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":0.71999999999999997,"15min":0.66100000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":8.7539999999999996,"5min":8.7539999999999996,"15min":8.7539999999999996},"last":0.69099999999999995},{"interface":"front","average":{"1min":1.0609999999999999,"5min":0.72799999999999998,"15min":0.66100000000000003},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.36899999999999999},"max":{"1min":9.173,"5min":9.173,"15min":9.173},"last":0.66000000000000003}]},{"osd":2,"last update":"Mon Oct 16 13:28:07 2023","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.74099999999999999,"15min":0.70699999999999996},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":6.5869999999999997,"5min":6.5869999999999997,"15min":6.5869999999999997},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.84899999999999998,"5min":0.72899999999999998,"15min":0.70999999999999996},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":3.0640000000000001,"5min":3.0640000000000001,"15min":3.0640000000000001},"last":0.79700000000000004}]},{"osd":3,"last update":"Mon Oct 16 13:28:07 2023","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.67200000000000004,"15min":0.65400000000000003},"min":{"1min":0.41399999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":3.1299999999999999,"5min":3.1299999999999999,"15min":3.1299999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":1.0569999999999999,"5min":0.73699999999999999,"15min":0.68400000000000005},"min":{"1min":0.441,"5min":0.441,"15min":0.40899999999999997},"max":{"1min":8.9920000000000009,"5min":8.9920000000000009,"15min":8.9920000000000009},"last":0.73599999999999999}]},{"osd":4,"last update":"Mon Oct 16 13:28:07 2023","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":0.59199999999999997,"15min":0.55000000000000004},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.313},"max":{"1min":6.3769999999999998,"5min":6.3769999999999998,"15min":6.3769999999999998},"last":0.433},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.66500000000000004,"15min":0.63600000000000001},"min":{"1min":0.35299999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":3.1779999999999999,"5min":3.1779999999999999,"15min":3.1779999999999999},"last":0.63400000000000001}]},{"osd":5,"last update":"Mon Oct 16 13:28:07 2023","interfaces":[{"interface":"back","average":{"1min":1.0940000000000001,"5min":0.65400000000000003,"15min":0.57899999999999996},"min":{"1min":0.36799999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":8.9209999999999994,"5min":8.9209999999999994,"15min":8.9209999999999994},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.99399999999999999,"5min":0.66400000000000003,"15min":0.621},"min":{"1min":0.40100000000000002,"5min":0.377,"15min":0.376},"max":{"1min":6.7560000000000002,"5min":6.7560000000000002,"15min":6.7560000000000002},"last":0.60799999999999998}]},{"osd":6,"last update":"Mon Oct 16 13:28:07 2023","interfaces":[{"interface":"back","average":{"1min":1.0940000000000001,"5min":0.72599999999999998,"15min":0.66800000000000004},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.38500000000000001},"max":{"1min":8.8279999999999994,"5min":8.8279999999999994,"15min":8.8279999999999994},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.67400000000000004,"15min":0.65700000000000003},"min":{"1min":0.41699999999999998,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":3.2599999999999998,"5min":3.2599999999999998,"15min":3.2599999999999998},"last":0.84199999999999997}]}]},{"osd":6,"up_from":13,"seq":55834575771,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41812,"kb_used_data":2312,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330028,"statfs":{"total":96636764160,"available":96593948672,"internally_reserved":0,"allocated":2367488,"data_stored":1206111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:28:07 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.67000000000000004,"15min":0.67300000000000004},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.29899999999999999},"max":{"1min":0.80600000000000005,"5min":0.90800000000000003,"15min":0.91400000000000003},"last":0.89100000000000001},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.65800000000000003,"15min":0.67700000000000005},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.34699999999999998},"max":{"1min":0.67100000000000004,"5min":0.85999999999999999,"15min":0.93100000000000005},"last":0.80000000000000004}]},{"osd":1,"last update":"Mon Oct 16 13:28:07 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.61799999999999999,"15min":0.624},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.72299999999999998,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.61599999999999999,"15min":0.63200000000000001},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.34899999999999998},"max":{"1min":0.72999999999999998,"5min":0.88200000000000001,"15min":0.95599999999999996},"last":0.70899999999999996}]},{"osd":2,"last update":"Mon Oct 16 13:28:07 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.63200000000000001,"15min":0.63500000000000001},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.33500000000000002},"max":{"1min":0.69399999999999995,"5min":0.93899999999999995,"15min":0.94199999999999995},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.61099999999999999,"15min":0.63200000000000001},"min":{"1min":0.374,"5min":0.374,"15min":0.33100000000000002},"max":{"1min":0.74199999999999999,"5min":0.83099999999999996,"15min":0.93100000000000005},"last":0.85999999999999999}]},{"osd":3,"last update":"Mon Oct 16 13:28:07 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.67000000000000004,"15min":0.68100000000000005},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.374},"max":{"1min":0.78000000000000003,"5min":1.288,"15min":1.288},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.73099999999999998,"15min":0.72399999999999998},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.32300000000000001},"max":{"1min":0.76300000000000001,"5min":1.351,"15min":1.351},"last":0.84199999999999997}]},{"osd":4,"last update":"Mon Oct 16 13:28:07 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.51700000000000002},"min":{"1min":0.378,"5min":0.34100000000000003,"15min":0.32000000000000001},"max":{"1min":0.69199999999999995,"5min":0.85599999999999998,"15min":0.86299999999999999},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61699999999999999,"15min":0.60999999999999999},"min":{"1min":0.46100000000000002,"5min":0.35199999999999998,"15min":0.32000000000000001},"max":{"1min":0.79600000000000004,"5min":0.89600000000000002,"15min":0.90100000000000002},"last":0.65600000000000003}]},{"osd":5,"last update":"Mon Oct 16 13:28:07 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58199999999999996,"15min":0.57299999999999995},"min":{"1min":0.40999999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.72999999999999998,"5min":0.85899999999999999,"15min":0.91800000000000004},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.65200000000000002,"15min":0.63400000000000001},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.32700000000000001},"max":{"1min":0.85099999999999998,"5min":0.91000000000000003,"15min":0.93799999999999994},"last":0.78200000000000003}]},{"osd":7,"last update":"Mon Oct 16 13:28:07 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.58799999999999997,"15min":0.57499999999999996},"min":{"1min":0.40600000000000003,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.72899999999999998,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.63700000000000001,"15min":0.628},"min":{"1min":0.45900000000000002,"5min":0.39300000000000002,"15min":0.378},"max":{"1min":0.75600000000000001,"5min":0.92000000000000004,"15min":0.98899999999999999},"last":0.68200000000000005}]}]},{"osd":1,"up_from":13,"seq":55834575771,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41824,"kb_used_data":2324,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330016,"statfs":{"total":96636764160,"available":96593936384,"internally_reserved":0,"allocated":2379776,"data_stored":1207723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":40434121},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:28:09 2023","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.40999999999999998,"15min":0.41299999999999998},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.245},"max":{"1min":0.59799999999999998,"5min":0.60399999999999998,"15min":0.76600000000000001},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.497,"15min":0.49399999999999999},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":1.46,"5min":1.46,"15min":1.46},"last":0.36299999999999999}]},{"osd":2,"last update":"Mon Oct 16 13:28:09 2023","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.45200000000000001,"15min":0.45200000000000001},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.48799999999999999,"15min":0.49299999999999999},"min":{"1min":0.34599999999999997,"5min":0.32300000000000001,"15min":0.29299999999999998},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.39300000000000002}]},{"osd":3,"last update":"Mon Oct 16 13:28:09 2023","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.435,"15min":0.42799999999999999},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29199999999999998},"max":{"1min":1.5009999999999999,"5min":1.5009999999999999,"15min":1.5009999999999999},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.48899999999999999,"15min":0.48599999999999999},"min":{"1min":0.34200000000000003,"5min":0.311,"15min":0.31},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.377}]},{"osd":4,"last update":"Mon Oct 16 13:28:09 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.66000000000000003,"15min":0.66500000000000004},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.542,"5min":1.542,"15min":1.542},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.68400000000000005,"15min":0.68100000000000005},"min":{"1min":0.435,"5min":0.435,"15min":0.42099999999999999},"max":{"1min":0.95099999999999996,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.76400000000000001}]},{"osd":5,"last update":"Mon Oct 16 13:28:09 2023","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.73199999999999998,"15min":0.72299999999999998},"min":{"1min":0.502,"5min":0.502,"15min":0.44400000000000001},"max":{"1min":3.2610000000000001,"5min":3.2610000000000001,"15min":3.2610000000000001},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.72799999999999998,"15min":0.73099999999999998},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.579,"5min":1.579,"15min":1.579},"last":0.77900000000000003}]},{"osd":6,"last update":"Mon Oct 16 13:28:09 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.68700000000000006,"15min":0.68799999999999994},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.45000000000000001},"max":{"1min":0.96499999999999997,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.72599999999999998,"15min":0.71899999999999997},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.432},"max":{"1min":3.3130000000000002,"5min":3.3130000000000002,"15min":3.3130000000000002},"last":0.71399999999999997}]},{"osd":7,"last update":"Mon Oct 16 13:28:09 2023","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.73999999999999999,"15min":0.73899999999999999},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.44},"max":{"1min":1.9430000000000001,"5min":1.9430000000000001,"15min":1.9430000000000001},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.77200000000000002,"15min":0.75900000000000001},"min":{"1min":0.505,"5min":0.505,"15min":0.498},"max":{"1min":1.7549999999999999,"5min":1.7549999999999999,"15min":1.7549999999999999},"last":0.72599999999999998}]}]},{"osd":0,"up_from":13,"seq":55834575772,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42408,"kb_used_data":2900,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329432,"statfs":{"total":96636764160,"available":96593338368,"internally_reserved":0,"allocated":2969600,"data_stored":1796583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 13:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.40200000000000002,"15min":0.41099999999999998},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.317},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.438,"15min":0.44700000000000001},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.52600000000000002,"5min":0.745,"15min":0.75800000000000001},"last":0.38}]},{"osd":2,"last update":"Mon Oct 16 13:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.44700000000000001,"15min":0.45600000000000002},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.52200000000000002,"5min":0.70799999999999996,"15min":0.73399999999999999},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.46999999999999997,"15min":0.48199999999999998},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.75700000000000001},"last":0.42799999999999999}]},{"osd":3,"last update":"Mon Oct 16 13:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.443,"15min":0.44900000000000001},"min":{"1min":0.28000000000000003,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.50900000000000001,"15min":0.52500000000000002},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.82099999999999995},"last":0.40600000000000003}]},{"osd":4,"last update":"Mon Oct 16 13:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.69499999999999995,"15min":0.68600000000000005},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.40200000000000002},"max":{"1min":0.79900000000000004,"5min":1.0229999999999999,"15min":1.0369999999999999},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.68000000000000005,"15min":0.68500000000000005},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.39000000000000001},"max":{"1min":0.77100000000000002,"5min":1.0429999999999999,"15min":1.0669999999999999},"last":0.59899999999999998}]},{"osd":5,"last update":"Mon Oct 16 13:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.67800000000000005,"15min":0.68300000000000005},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.84499999999999997,"5min":1.0049999999999999,"15min":1.034},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.71399999999999997,"15min":0.71599999999999997},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.46700000000000003},"max":{"1min":0.84399999999999997,"5min":1.0129999999999999,"15min":1.109},"last":0.58399999999999996}]},{"osd":6,"last update":"Mon Oct 16 13:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.72099999999999997,"15min":0.72899999999999998},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.88900000000000001,"5min":1.0649999999999999,"15min":1.109},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.62,"5min":0.71299999999999997,"15min":0.72799999999999998},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.44500000000000001},"max":{"1min":0.83099999999999996,"5min":1.03,"15min":1.0720000000000001},"last":0.72999999999999998}]},{"osd":7,"last update":"Mon Oct 16 13:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.73299999999999998,"15min":0.73699999999999999},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.93000000000000005,"5min":0.99199999999999999,"15min":1.0880000000000001},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.72599999999999998,"15min":0.72999999999999998},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.92000000000000004,"5min":0.996,"15min":1.0920000000000001},"last":0.77400000000000002}]}]},{"osd":2,"up_from":13,"seq":55834575772,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51544,"kb_used_data":2312,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320296,"statfs":{"total":96636764160,"available":96583983104,"internally_reserved":0,"allocated":2367488,"data_stored":1206111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.39800000000000002,"15min":0.41199999999999998},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.56200000000000006,"5min":0.65600000000000003,"15min":0.77800000000000002},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.5,"5min":0.496,"15min":0.48899999999999999},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.339,"5min":1.339,"15min":1.339},"last":0.46700000000000003}]},{"osd":1,"last update":"Mon Oct 16 13:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.52500000000000002,"15min":0.52000000000000002},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32400000000000001},"max":{"1min":1.3819999999999999,"5min":1.3819999999999999,"15min":1.3819999999999999},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.45800000000000002,"15min":0.47099999999999997},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.60799999999999998,"5min":0.78200000000000003,"15min":0.79000000000000004},"last":0.379}]},{"osd":3,"last update":"Mon Oct 16 13:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.42999999999999999,"15min":0.44900000000000001},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.65700000000000003,"5min":0.66200000000000003,"15min":0.81200000000000006},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55700000000000005,"15min":0.56000000000000005},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.4179999999999999,"5min":1.4179999999999999,"15min":1.4179999999999999},"last":0.503}]},{"osd":4,"last update":"Mon Oct 16 13:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.70099999999999996,"15min":0.69999999999999996},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.92900000000000005,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.70199999999999996,"15min":0.70799999999999996},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.95099999999999996,"5min":0.996,"15min":1.0109999999999999},"last":0.70899999999999996}]},{"osd":5,"last update":"Mon Oct 16 13:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.71099999999999997,"15min":0.70599999999999996},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.45,"5min":1.45,"15min":1.45},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.66700000000000004,"15min":0.68100000000000005},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.80700000000000005,"5min":0.96899999999999997,"15min":1.0740000000000001},"last":0.81799999999999995}]},{"osd":6,"last update":"Mon Oct 16 13:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.65100000000000002,"15min":0.66200000000000003},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.90000000000000002,"5min":1.046,"15min":1.345},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.73599999999999999,"15min":0.73299999999999998},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.4830000000000001,"5min":1.4830000000000001,"15min":1.4830000000000001},"last":0.64300000000000002}]},{"osd":7,"last update":"Mon Oct 16 13:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.70699999999999996,"15min":0.71999999999999997},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.81899999999999995,"5min":1.0620000000000001,"15min":1.115},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.74299999999999999,"15min":0.73799999999999999},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.5149999999999999,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":0.60199999999999998}]}]},{"osd":3,"up_from":13,"seq":55834575771,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47400,"kb_used_data":2332,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324440,"statfs":{"total":96636764160,"available":96588226560,"internally_reserved":0,"allocated":2387968,"data_stored":1208327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":46135752},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.40899999999999997,"15min":0.40600000000000003},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.60199999999999998,"5min":0.629,"15min":0.73099999999999998},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.377,"5min":0.42699999999999999,"15min":0.433},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.628,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.40899999999999997}]},{"osd":1,"last update":"Mon Oct 16 13:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.44500000000000001,"15min":0.434},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.70699999999999996,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.437,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.29299999999999998},"max":{"1min":0.66300000000000003,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.43099999999999999}]},{"osd":2,"last update":"Mon Oct 16 13:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.48999999999999999,"15min":0.495},"min":{"1min":0.31900000000000001,"5min":0.311,"15min":0.311},"max":{"1min":0.56899999999999995,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.52100000000000002,"15min":0.52600000000000002},"min":{"1min":0.32600000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.89800000000000002},"last":0.48699999999999999}]},{"osd":4,"last update":"Mon Oct 16 13:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.65300000000000002,"15min":0.67100000000000004},"min":{"1min":0.47599999999999998,"5min":0.46600000000000003,"15min":0.41799999999999998},"max":{"1min":0.82299999999999995,"5min":1.0640000000000001,"15min":1.1060000000000001},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.67900000000000005,"15min":0.68899999999999995},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.78100000000000003,"5min":1.008,"15min":1.0189999999999999},"last":0.84299999999999997}]},{"osd":5,"last update":"Mon Oct 16 13:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.69099999999999995,"15min":0.70999999999999996},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89000000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":1.079},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.69099999999999995,"15min":0.69799999999999995},"min":{"1min":0.42699999999999999,"5min":0.42499999999999999,"15min":0.40200000000000002},"max":{"1min":0.90500000000000003,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.93100000000000005}]},{"osd":6,"last update":"Mon Oct 16 13:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.70399999999999996,"15min":0.70699999999999996},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.42799999999999999},"max":{"1min":0.92900000000000005,"5min":1.0469999999999999,"15min":1.0800000000000001},"last":0.89700000000000002},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.72999999999999998,"15min":0.74099999999999999},"min":{"1min":0.502,"5min":0.502,"15min":0.46999999999999997},"max":{"1min":0.94299999999999995,"5min":1.073,"15min":1.1220000000000001},"last":1.119}]},{"osd":7,"last update":"Mon Oct 16 13:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.71699999999999997,"15min":0.72499999999999998},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.86299999999999999,"5min":1.099,"15min":1.099},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.754,"15min":0.753},"min":{"1min":0.49099999999999999,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.998,"5min":1.0609999999999999,"15min":1.131},"last":0.95799999999999996}]}]},{"osd":4,"up_from":13,"seq":55834575770,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29356,"kb_used_data":2328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342484,"statfs":{"total":96636764160,"available":96606703616,"internally_reserved":0,"allocated":2383872,"data_stored":1206875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:28:01 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.621,"15min":0.62},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.628,"15min":0.63500000000000001},"min":{"1min":0.40000000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.76400000000000001,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.53800000000000003}]},{"osd":1,"last update":"Mon Oct 16 13:28:01 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.63700000000000001,"15min":0.64500000000000002},"min":{"1min":0.39600000000000002,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.86699999999999999,"5min":0.878,"15min":0.93799999999999994},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.63500000000000001,"15min":0.63700000000000001},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.85099999999999998,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.40400000000000003}]},{"osd":2,"last update":"Mon Oct 16 13:28:01 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.63600000000000001,"15min":0.65100000000000002},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.83099999999999996,"5min":0.90000000000000002,"15min":1.091},"last":0.5},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.66300000000000003,"15min":0.65700000000000003},"min":{"1min":0.46500000000000002,"5min":0.38,"15min":0.38},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.51500000000000001}]},{"osd":3,"last update":"Mon Oct 16 13:28:01 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.63400000000000001,"15min":0.64600000000000002},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.82799999999999996,"5min":0.89200000000000002,"15min":0.95399999999999996},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.68000000000000005,"15min":0.67700000000000005},"min":{"1min":0.48299999999999998,"5min":0.41599999999999998,"15min":0.41499999999999998},"max":{"1min":0.80900000000000005,"5min":0.86799999999999999,"15min":0.98899999999999999},"last":0.59299999999999997}]},{"osd":5,"last update":"Mon Oct 16 13:28:01 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.51600000000000001,"15min":0.501},"min":{"1min":0.39600000000000002,"5min":0.33600000000000002,"15min":0.309},"max":{"1min":0.89200000000000002,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54300000000000004,"15min":0.54000000000000004},"min":{"1min":0.35799999999999998,"5min":0.31,"15min":0.31},"max":{"1min":0.755,"5min":0.93000000000000005,"15min":0.97099999999999997},"last":0.57999999999999996}]},{"osd":6,"last update":"Mon Oct 16 13:28:01 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55100000000000005,"15min":0.56200000000000006},"min":{"1min":0.377,"5min":0.34499999999999997,"15min":0.32800000000000001},"max":{"1min":0.78900000000000003,"5min":0.90900000000000003,"15min":1.046},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.45300000000000001,"5min":0.433,"15min":0.34999999999999998},"max":{"1min":0.96999999999999997,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.60599999999999998}]},{"osd":7,"last update":"Mon Oct 16 13:28:01 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.41999999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.88800000000000001,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.65300000000000002,"15min":0.65000000000000002},"min":{"1min":0.441,"5min":0.39500000000000002,"15min":0.38400000000000001},"max":{"1min":0.92400000000000004,"5min":0.94899999999999995,"15min":1.1359999999999999},"last":0.63}]}]},{"osd":5,"up_from":13,"seq":55834575771,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51548,"kb_used_data":2320,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320292,"statfs":{"total":96636764160,"available":96583979008,"internally_reserved":0,"allocated":2375680,"data_stored":1206235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:27:50 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.624,"15min":0.61799999999999999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.76600000000000001,"5min":0.87,"15min":0.87},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.63,"15min":0.64400000000000002},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.79900000000000004,"5min":0.89100000000000001,"15min":0.94899999999999995},"last":0.66100000000000003}]},{"osd":1,"last update":"Mon Oct 16 13:27:50 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.64400000000000002,"15min":0.64800000000000002},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.72899999999999998,"5min":0.872,"15min":0.93899999999999995},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.64000000000000001,"15min":0.65100000000000002},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.77400000000000002,"5min":0.90200000000000002,"15min":0.93300000000000005},"last":0.69099999999999995}]},{"osd":2,"last update":"Mon Oct 16 13:27:50 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.66000000000000003,"15min":0.65200000000000002},"min":{"1min":0.45300000000000001,"5min":0.441,"15min":0.39400000000000002},"max":{"1min":0.79200000000000004,"5min":1.091,"15min":1.091},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.624,"5min":0.64000000000000001,"15min":0.64600000000000002},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.35799999999999998},"max":{"1min":0.80900000000000005,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.73899999999999999}]},{"osd":3,"last update":"Mon Oct 16 13:27:50 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.0069999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.68899999999999995,"15min":0.68500000000000005},"min":{"1min":0.44900000000000001,"5min":0.42599999999999999,"15min":0.41099999999999998},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":0.873}]},{"osd":4,"last update":"Mon Oct 16 13:27:50 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.52700000000000002,"15min":0.52200000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":1.028},{"interface":"front","average":{"1min":0.621,"5min":0.60199999999999998,"15min":0.58599999999999997},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":1.0740000000000001}]},{"osd":6,"last update":"Mon Oct 16 13:27:50 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.59499999999999997,"15min":0.57299999999999995},"min":{"1min":0.443,"5min":0.39800000000000002,"15min":0.34399999999999997},"max":{"1min":1.095,"5min":1.095,"15min":1.095},"last":0.98199999999999998},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.63,"15min":0.61699999999999999},"min":{"1min":0.371,"5min":0.371,"15min":0.312},"max":{"1min":0.90300000000000002,"5min":1.006,"15min":1.006},"last":0.90900000000000003}]},{"osd":7,"last update":"Mon Oct 16 13:27:50 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.67600000000000005,"15min":0.64800000000000002},"min":{"1min":0.47099999999999997,"5min":0.40600000000000003,"15min":0.34100000000000003},"max":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.95699999999999996},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.69599999999999995,"15min":0.67500000000000004},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.39700000000000002},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":1.042}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T13:28:24.271 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T13:28:24.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.428+0000 7f485b70f700 1 Processor -- start 2023-10-16T13:28:24.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.428+0000 7f485b70f700 1 -- start start 2023-10-16T13:28:24.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.429+0000 7f485b70f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f485406a7e0 0x7f485406ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:24.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.429+0000 7f485b70f700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f485406b150 con 0x7f485415b280 2023-10-16T13:28:24.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.429+0000 7f485b70f700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4854161c30 con 0x7f48540cc8a0 2023-10-16T13:28:24.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.429+0000 7f485b70f700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4854161f10 con 0x7f485406a7e0 2023-10-16T13:28:24.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.429+0000 7f4859cac700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f485406a7e0 0x7f485406ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:24.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.429+0000 7f4859cac700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f485406a7e0 0x7f485406ac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49916/0 (socket says 172.21.15.16:49916) 2023-10-16T13:28:24.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.429+0000 7f4859cac700 1 -- 172.21.15.16:0/1407810771 learned_addr learned my addr 172.21.15.16:0/1407810771 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:24.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.429+0000 7f4859cac700 1 -- 172.21.15.16:0/1407810771 >> v1:172.21.15.49:6789/0 conn(0x7f48540cc8a0 legacy=0x7f485415e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:28:24.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.429+0000 7f4859cac700 1 -- 172.21.15.16:0/1407810771 >> v1:172.21.15.16:6789/0 conn(0x7f485415b280 legacy=0x7f48541613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:24.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.429+0000 7f4859cac700 1 -- 172.21.15.16:0/1407810771 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48541621f0 con 0x7f485406a7e0 2023-10-16T13:28:24.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.429+0000 7f4859cac700 1 --2- 172.21.15.16:0/1407810771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f485406a7e0 0x7f485406ac10 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 crypto rx=0x7f484c009e50 tx=0x7f484c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdcae6f030d6a2e4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:24.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.430+0000 7f484bfff700 1 -- 172.21.15.16:0/1407810771 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f484c00e070 con 0x7f485406a7e0 2023-10-16T13:28:24.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.430+0000 7f484bfff700 1 -- 172.21.15.16:0/1407810771 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f484c003a20 con 0x7f485406a7e0 2023-10-16T13:28:24.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.430+0000 7f484bfff700 1 -- 172.21.15.16:0/1407810771 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f484c003dd0 con 0x7f485406a7e0 2023-10-16T13:28:24.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.430+0000 7f485b70f700 1 -- 172.21.15.16:0/1407810771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f485406a7e0 msgr2=0x7f485406ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:24.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.430+0000 7f485b70f700 1 --2- 172.21.15.16:0/1407810771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f485406a7e0 0x7f485406ac10 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 crypto rx=0x7f484c009e50 tx=0x7f484c00b330 comp rx=0 tx=0).stop 2023-10-16T13:28:24.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.430+0000 7f485b70f700 1 -- 172.21.15.16:0/1407810771 shutdown_connections 2023-10-16T13:28:24.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.430+0000 7f485b70f700 1 --2- 172.21.15.16:0/1407810771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f485406a7e0 0x7f485406ac10 unknown :-1 s=CLOSED pgs=2099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:24.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.430+0000 7f485b70f700 1 -- 172.21.15.16:0/1407810771 >> 172.21.15.16:0/1407810771 conn(0x7f48540b1fd0 msgr2=0x7f48540b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:24.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.430+0000 7f485b70f700 1 -- 172.21.15.16:0/1407810771 shutdown_connections 2023-10-16T13:28:24.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.430+0000 7f485b70f700 1 -- 172.21.15.16:0/1407810771 wait complete. 2023-10-16T13:28:24.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.431+0000 7f485b70f700 1 Processor -- start 2023-10-16T13:28:24.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.432+0000 7f485b70f700 1 -- start start 2023-10-16T13:28:24.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.432+0000 7f485b70f700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f485406a7e0 0x7f48540c4a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:24.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.432+0000 7f485b70f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f48540cc8a0 0x7f48540c6770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:24.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.432+0000 7f485b70f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f485415b280 0x7f485415f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:24.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.432+0000 7f485b70f700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48540c4f90 con 0x7f485415b280 2023-10-16T13:28:24.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.432+0000 7f485b70f700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48541572f0 con 0x7f485406a7e0 2023-10-16T13:28:24.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.432+0000 7f485b70f700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48541f6330 con 0x7f48540cc8a0 2023-10-16T13:28:24.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.432+0000 7f4859cac700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f485415b280 0x7f485415f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:24.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.432+0000 7f4859cac700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f485415b280 0x7f485415f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49922/0 (socket says 172.21.15.16:49922) 2023-10-16T13:28:24.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.432+0000 7f4859cac700 1 -- 172.21.15.16:0/3650469873 learned_addr learned my addr 172.21.15.16:0/3650469873 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:24.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.433+0000 7f4858caa700 1 --2- 172.21.15.16:0/3650469873 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f48540cc8a0 0x7f48540c6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:24.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.433+0000 7f48594ab700 1 --2- 172.21.15.16:0/3650469873 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f485406a7e0 0x7f48540c4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:24.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.433+0000 7f4859cac700 1 -- 172.21.15.16:0/3650469873 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f48540cc8a0 msgr2=0x7f48540c6770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:24.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.433+0000 7f4859cac700 1 --2- 172.21.15.16:0/3650469873 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f48540cc8a0 0x7f48540c6770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:24.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.433+0000 7f4859cac700 1 -- 172.21.15.16:0/3650469873 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f485406a7e0 msgr2=0x7f48540c4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:24.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.433+0000 7f4859cac700 1 --2- 172.21.15.16:0/3650469873 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f485406a7e0 0x7f48540c4a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:24.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.433+0000 7f4859cac700 1 -- 172.21.15.16:0/3650469873 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f484c009b30 con 0x7f485415b280 2023-10-16T13:28:24.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.433+0000 7f4859cac700 1 --2- 172.21.15.16:0/3650469873 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f485415b280 0x7f485415f880 secure :-1 s=READY pgs=2100 cs=0 l=1 rev1=1 crypto rx=0x7f484c00b730 tx=0x7f484c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:24.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.433+0000 7f4849ffb700 1 -- 172.21.15.16:0/3650469873 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f484c004440 con 0x7f485415b280 2023-10-16T13:28:24.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.433+0000 7f4849ffb700 1 -- 172.21.15.16:0/3650469873 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f484c0045a0 con 0x7f485415b280 2023-10-16T13:28:24.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.433+0000 7f485b70f700 1 -- 172.21.15.16:0/3650469873 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48541f6610 con 0x7f485415b280 2023-10-16T13:28:24.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.433+0000 7f4849ffb700 1 -- 172.21.15.16:0/3650469873 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f484c004930 con 0x7f485415b280 2023-10-16T13:28:24.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.433+0000 7f485b70f700 1 -- 172.21.15.16:0/3650469873 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48541f6bd0 con 0x7f485415b280 2023-10-16T13:28:24.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.434+0000 7f485b70f700 1 -- 172.21.15.16:0/3650469873 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48541574f0 con 0x7f485415b280 2023-10-16T13:28:24.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.435+0000 7f4849ffb700 1 -- 172.21.15.16:0/3650469873 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 107) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f484c019030 con 0x7f485415b280 2023-10-16T13:28:24.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.435+0000 7f4849ffb700 1 --2- 172.21.15.16:0/3650469873 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f483806b020 0x7f483806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:24.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.435+0000 7f4849ffb700 1 -- 172.21.15.16:0/3650469873 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f484c08b070 con 0x7f485415b280 2023-10-16T13:28:24.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.435+0000 7f48594ab700 1 --2- 172.21.15.16:0/3650469873 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f483806b020 0x7f483806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:24.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.435+0000 7f48594ab700 1 --2- 172.21.15.16:0/3650469873 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f483806b020 0x7f483806d4e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4850001620 tx=0x7f4850009040 comp rx=0 tx=0).ready entity=mgr.10047 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:24.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.440+0000 7f4849ffb700 1 -- 172.21.15.16:0/3650469873 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f48541574f0 con 0x7f485415b280 2023-10-16T13:28:24.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.639+0000 7f485b70f700 1 -- 172.21.15.16:0/3650469873 --> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f48541f6d60 con 0x7f483806b020 2023-10-16T13:28:24.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.647+0000 7f4849ffb700 1 -- 172.21.15.16:0/3650469873 <== mgr.10047 v2:172.21.15.16:6832/4211329181 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289887 (secure 0 0 0) 0x7f48541f6d60 con 0x7f483806b020 2023-10-16T13:28:24.653 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:28:24.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.650+0000 7f485b70f700 1 -- 172.21.15.16:0/3650469873 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f483806b020 msgr2=0x7f483806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:24.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.650+0000 7f485b70f700 1 --2- 172.21.15.16:0/3650469873 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f483806b020 0x7f483806d4e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4850001620 tx=0x7f4850009040 comp rx=0 tx=0).stop 2023-10-16T13:28:24.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.650+0000 7f485b70f700 1 -- 172.21.15.16:0/3650469873 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f485415b280 msgr2=0x7f485415f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:24.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.650+0000 7f485b70f700 1 --2- 172.21.15.16:0/3650469873 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f485415b280 0x7f485415f880 secure :-1 s=READY pgs=2100 cs=0 l=1 rev1=1 crypto rx=0x7f484c00b730 tx=0x7f484c004ad0 comp rx=0 tx=0).stop 2023-10-16T13:28:24.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.651+0000 7f485b70f700 1 -- 172.21.15.16:0/3650469873 shutdown_connections 2023-10-16T13:28:24.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.651+0000 7f485b70f700 1 --2- 172.21.15.16:0/3650469873 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f485415b280 0x7f485415f880 unknown :-1 s=CLOSED pgs=2100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:24.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.651+0000 7f485b70f700 1 --2- 172.21.15.16:0/3650469873 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f483806b020 0x7f483806d4e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:24.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.651+0000 7f485b70f700 1 --2- 172.21.15.16:0/3650469873 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f485406a7e0 0x7f48540c4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:24.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.651+0000 7f485b70f700 1 --2- 172.21.15.16:0/3650469873 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f48540cc8a0 0x7f48540c6770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:24.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.651+0000 7f485b70f700 1 -- 172.21.15.16:0/3650469873 >> 172.21.15.16:0/3650469873 conn(0x7f48540b1fd0 msgr2=0x7f485415a570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:24.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.651+0000 7f485b70f700 1 -- 172.21.15.16:0/3650469873 shutdown_connections 2023-10-16T13:28:24.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.651+0000 7f485b70f700 1 -- 172.21.15.16:0/3650469873 wait complete. 2023-10-16T13:28:24.659 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-10-16T13:28:24.665 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2023-10-16T13:28:23.239677+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1652,"num_read_kb":3396,"num_write":523,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":331,"ondisk_log_size":331,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":358012,"kb_used_data":19720,"kb_used_omap":12,"kb_used_meta":338163,"kb_avail":754616708,"statfs":{"total":773094113280,"available":772727508992,"internally_reserved":0,"allocated":20193280,"data_stored":10834444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12740,"internal_metadata":346279484},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.156362"},"pg_stats":[{"pgid":"24.3f","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.216121+0000","last_change":"2023-10-16T13:28:20.102101+0000","last_active":"2023-10-16T13:28:21.216121+0000","last_peered":"2023-10-16T13:28:21.216121+0000","last_clean":"2023-10-16T13:28:21.216121+0000","last_became_active":"2023-10-16T13:28:20.101891+0000","last_became_peered":"2023-10-16T13:28:20.101891+0000","last_unstale":"2023-10-16T13:28:21.216121+0000","last_undegraded":"2023-10-16T13:28:21.216121+0000","last_fullsized":"2023-10-16T13:28:21.216121+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:36:09.581917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3e","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.216435+0000","last_change":"2023-10-16T13:28:20.108645+0000","last_active":"2023-10-16T13:28:21.216435+0000","last_peered":"2023-10-16T13:28:21.216435+0000","last_clean":"2023-10-16T13:28:21.216435+0000","last_became_active":"2023-10-16T13:28:20.108103+0000","last_became_peered":"2023-10-16T13:28:20.108103+0000","last_unstale":"2023-10-16T13:28:21.216435+0000","last_undegraded":"2023-10-16T13:28:21.216435+0000","last_fullsized":"2023-10-16T13:28:21.216435+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:53:08.194107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3d","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.215578+0000","last_change":"2023-10-16T13:28:20.127594+0000","last_active":"2023-10-16T13:28:21.215578+0000","last_peered":"2023-10-16T13:28:21.215578+0000","last_clean":"2023-10-16T13:28:21.215578+0000","last_became_active":"2023-10-16T13:28:20.127125+0000","last_became_peered":"2023-10-16T13:28:20.127125+0000","last_unstale":"2023-10-16T13:28:21.215578+0000","last_undegraded":"2023-10-16T13:28:21.215578+0000","last_fullsized":"2023-10-16T13:28:21.215578+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:28:41.639458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3c","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.215907+0000","last_change":"2023-10-16T13:28:20.116020+0000","last_active":"2023-10-16T13:28:21.215907+0000","last_peered":"2023-10-16T13:28:21.215907+0000","last_clean":"2023-10-16T13:28:21.215907+0000","last_became_active":"2023-10-16T13:28:20.115108+0000","last_became_peered":"2023-10-16T13:28:20.115108+0000","last_unstale":"2023-10-16T13:28:21.215907+0000","last_undegraded":"2023-10-16T13:28:21.215907+0000","last_fullsized":"2023-10-16T13:28:21.215907+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:08:52.920478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3b","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.090318+0000","last_change":"2023-10-16T13:28:20.100002+0000","last_active":"2023-10-16T13:28:21.090318+0000","last_peered":"2023-10-16T13:28:21.090318+0000","last_clean":"2023-10-16T13:28:21.090318+0000","last_became_active":"2023-10-16T13:28:20.099467+0000","last_became_peered":"2023-10-16T13:28:20.099467+0000","last_unstale":"2023-10-16T13:28:21.090318+0000","last_undegraded":"2023-10-16T13:28:21.090318+0000","last_fullsized":"2023-10-16T13:28:21.090318+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:48:02.876298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.3a","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.216082+0000","last_change":"2023-10-16T13:28:20.109342+0000","last_active":"2023-10-16T13:28:21.216082+0000","last_peered":"2023-10-16T13:28:21.216082+0000","last_clean":"2023-10-16T13:28:21.216082+0000","last_became_active":"2023-10-16T13:28:20.109039+0000","last_became_peered":"2023-10-16T13:28:20.109039+0000","last_unstale":"2023-10-16T13:28:21.216082+0000","last_undegraded":"2023-10-16T13:28:21.216082+0000","last_fullsized":"2023-10-16T13:28:21.216082+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:41:45.307869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.39","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.214203+0000","last_change":"2023-10-16T13:28:20.116945+0000","last_active":"2023-10-16T13:28:21.214203+0000","last_peered":"2023-10-16T13:28:21.214203+0000","last_clean":"2023-10-16T13:28:21.214203+0000","last_became_active":"2023-10-16T13:28:20.116488+0000","last_became_peered":"2023-10-16T13:28:20.116488+0000","last_unstale":"2023-10-16T13:28:21.214203+0000","last_undegraded":"2023-10-16T13:28:21.214203+0000","last_fullsized":"2023-10-16T13:28:21.214203+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:19:33.393496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.38","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.214045+0000","last_change":"2023-10-16T13:28:20.099310+0000","last_active":"2023-10-16T13:28:21.214045+0000","last_peered":"2023-10-16T13:28:21.214045+0000","last_clean":"2023-10-16T13:28:21.214045+0000","last_became_active":"2023-10-16T13:28:20.098957+0000","last_became_peered":"2023-10-16T13:28:20.098957+0000","last_unstale":"2023-10-16T13:28:21.214045+0000","last_undegraded":"2023-10-16T13:28:21.214045+0000","last_fullsized":"2023-10-16T13:28:21.214045+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:19:50.557706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.37","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.214821+0000","last_change":"2023-10-16T13:28:20.106002+0000","last_active":"2023-10-16T13:28:21.214821+0000","last_peered":"2023-10-16T13:28:21.214821+0000","last_clean":"2023-10-16T13:28:21.214821+0000","last_became_active":"2023-10-16T13:28:20.105687+0000","last_became_peered":"2023-10-16T13:28:20.105687+0000","last_unstale":"2023-10-16T13:28:21.214821+0000","last_undegraded":"2023-10-16T13:28:21.214821+0000","last_fullsized":"2023-10-16T13:28:21.214821+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:59:28.632287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.36","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.214810+0000","last_change":"2023-10-16T13:28:20.104608+0000","last_active":"2023-10-16T13:28:21.214810+0000","last_peered":"2023-10-16T13:28:21.214810+0000","last_clean":"2023-10-16T13:28:21.214810+0000","last_became_active":"2023-10-16T13:28:20.104279+0000","last_became_peered":"2023-10-16T13:28:20.104279+0000","last_unstale":"2023-10-16T13:28:21.214810+0000","last_undegraded":"2023-10-16T13:28:21.214810+0000","last_fullsized":"2023-10-16T13:28:21.214810+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:57:06.927477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.35","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.216432+0000","last_change":"2023-10-16T13:28:20.099232+0000","last_active":"2023-10-16T13:28:21.216432+0000","last_peered":"2023-10-16T13:28:21.216432+0000","last_clean":"2023-10-16T13:28:21.216432+0000","last_became_active":"2023-10-16T13:28:20.098914+0000","last_became_peered":"2023-10-16T13:28:20.098914+0000","last_unstale":"2023-10-16T13:28:21.216432+0000","last_undegraded":"2023-10-16T13:28:21.216432+0000","last_fullsized":"2023-10-16T13:28:21.216432+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:40:40.178349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.34","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.216605+0000","last_change":"2023-10-16T13:28:20.109397+0000","last_active":"2023-10-16T13:28:21.216605+0000","last_peered":"2023-10-16T13:28:21.216605+0000","last_clean":"2023-10-16T13:28:21.216605+0000","last_became_active":"2023-10-16T13:28:20.109081+0000","last_became_peered":"2023-10-16T13:28:20.109081+0000","last_unstale":"2023-10-16T13:28:21.216605+0000","last_undegraded":"2023-10-16T13:28:21.216605+0000","last_fullsized":"2023-10-16T13:28:21.216605+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:08:46.302886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.33","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.090473+0000","last_change":"2023-10-16T13:28:20.114290+0000","last_active":"2023-10-16T13:28:21.090473+0000","last_peered":"2023-10-16T13:28:21.090473+0000","last_clean":"2023-10-16T13:28:21.090473+0000","last_became_active":"2023-10-16T13:28:20.112316+0000","last_became_peered":"2023-10-16T13:28:20.112316+0000","last_unstale":"2023-10-16T13:28:21.090473+0000","last_undegraded":"2023-10-16T13:28:21.090473+0000","last_fullsized":"2023-10-16T13:28:21.090473+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:32:58.447860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.32","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.089718+0000","last_change":"2023-10-16T13:28:20.129334+0000","last_active":"2023-10-16T13:28:21.089718+0000","last_peered":"2023-10-16T13:28:21.089718+0000","last_clean":"2023-10-16T13:28:21.089718+0000","last_became_active":"2023-10-16T13:28:20.129040+0000","last_became_peered":"2023-10-16T13:28:20.129040+0000","last_unstale":"2023-10-16T13:28:21.089718+0000","last_undegraded":"2023-10-16T13:28:21.089718+0000","last_fullsized":"2023-10-16T13:28:21.089718+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:19:15.031376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.31","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.215450+0000","last_change":"2023-10-16T13:28:20.098947+0000","last_active":"2023-10-16T13:28:21.215450+0000","last_peered":"2023-10-16T13:28:21.215450+0000","last_clean":"2023-10-16T13:28:21.215450+0000","last_became_active":"2023-10-16T13:28:20.098618+0000","last_became_peered":"2023-10-16T13:28:20.098618+0000","last_unstale":"2023-10-16T13:28:21.215450+0000","last_undegraded":"2023-10-16T13:28:21.215450+0000","last_fullsized":"2023-10-16T13:28:21.215450+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:15:12.164129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.30","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.216379+0000","last_change":"2023-10-16T13:28:20.115722+0000","last_active":"2023-10-16T13:28:21.216379+0000","last_peered":"2023-10-16T13:28:21.216379+0000","last_clean":"2023-10-16T13:28:21.216379+0000","last_became_active":"2023-10-16T13:28:20.115432+0000","last_became_peered":"2023-10-16T13:28:20.115432+0000","last_unstale":"2023-10-16T13:28:21.216379+0000","last_undegraded":"2023-10-16T13:28:21.216379+0000","last_fullsized":"2023-10-16T13:28:21.216379+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:59:18.613450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.2f","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.214920+0000","last_change":"2023-10-16T13:28:20.093755+0000","last_active":"2023-10-16T13:28:21.214920+0000","last_peered":"2023-10-16T13:28:21.214920+0000","last_clean":"2023-10-16T13:28:21.214920+0000","last_became_active":"2023-10-16T13:28:20.093452+0000","last_became_peered":"2023-10-16T13:28:20.093452+0000","last_unstale":"2023-10-16T13:28:21.214920+0000","last_undegraded":"2023-10-16T13:28:21.214920+0000","last_fullsized":"2023-10-16T13:28:21.214920+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:49:00.696789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2e","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.215869+0000","last_change":"2023-10-16T13:28:20.099180+0000","last_active":"2023-10-16T13:28:21.215869+0000","last_peered":"2023-10-16T13:28:21.215869+0000","last_clean":"2023-10-16T13:28:21.215869+0000","last_became_active":"2023-10-16T13:28:20.098418+0000","last_became_peered":"2023-10-16T13:28:20.098418+0000","last_unstale":"2023-10-16T13:28:21.215869+0000","last_undegraded":"2023-10-16T13:28:21.215869+0000","last_fullsized":"2023-10-16T13:28:21.215869+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:27:04.199827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.2d","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.216280+0000","last_change":"2023-10-16T13:28:20.099591+0000","last_active":"2023-10-16T13:28:21.216280+0000","last_peered":"2023-10-16T13:28:21.216280+0000","last_clean":"2023-10-16T13:28:21.216280+0000","last_became_active":"2023-10-16T13:28:20.098426+0000","last_became_peered":"2023-10-16T13:28:20.098426+0000","last_unstale":"2023-10-16T13:28:21.216280+0000","last_undegraded":"2023-10-16T13:28:21.216280+0000","last_fullsized":"2023-10-16T13:28:21.216280+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:17:51.974122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.2c","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.214206+0000","last_change":"2023-10-16T13:28:20.117441+0000","last_active":"2023-10-16T13:28:21.214206+0000","last_peered":"2023-10-16T13:28:21.214206+0000","last_clean":"2023-10-16T13:28:21.214206+0000","last_became_active":"2023-10-16T13:28:20.117148+0000","last_became_peered":"2023-10-16T13:28:20.117148+0000","last_unstale":"2023-10-16T13:28:21.214206+0000","last_undegraded":"2023-10-16T13:28:21.214206+0000","last_fullsized":"2023-10-16T13:28:21.214206+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:41:35.647131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.2b","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.090613+0000","last_change":"2023-10-16T13:28:20.113501+0000","last_active":"2023-10-16T13:28:21.090613+0000","last_peered":"2023-10-16T13:28:21.090613+0000","last_clean":"2023-10-16T13:28:21.090613+0000","last_became_active":"2023-10-16T13:28:20.113147+0000","last_became_peered":"2023-10-16T13:28:20.113147+0000","last_unstale":"2023-10-16T13:28:21.090613+0000","last_undegraded":"2023-10-16T13:28:21.090613+0000","last_fullsized":"2023-10-16T13:28:21.090613+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:06:01.800100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.2a","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.216227+0000","last_change":"2023-10-16T13:28:20.111456+0000","last_active":"2023-10-16T13:28:21.216227+0000","last_peered":"2023-10-16T13:28:21.216227+0000","last_clean":"2023-10-16T13:28:21.216227+0000","last_became_active":"2023-10-16T13:28:20.108931+0000","last_became_peered":"2023-10-16T13:28:20.108931+0000","last_unstale":"2023-10-16T13:28:21.216227+0000","last_undegraded":"2023-10-16T13:28:21.216227+0000","last_fullsized":"2023-10-16T13:28:21.216227+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:25:48.347351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.29","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.089557+0000","last_change":"2023-10-16T13:28:20.102883+0000","last_active":"2023-10-16T13:28:21.089557+0000","last_peered":"2023-10-16T13:28:21.089557+0000","last_clean":"2023-10-16T13:28:21.089557+0000","last_became_active":"2023-10-16T13:28:20.102560+0000","last_became_peered":"2023-10-16T13:28:20.102560+0000","last_unstale":"2023-10-16T13:28:21.089557+0000","last_undegraded":"2023-10-16T13:28:21.089557+0000","last_fullsized":"2023-10-16T13:28:21.089557+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:32:59.618770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.28","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.216550+0000","last_change":"2023-10-16T13:28:20.106077+0000","last_active":"2023-10-16T13:28:21.216550+0000","last_peered":"2023-10-16T13:28:21.216550+0000","last_clean":"2023-10-16T13:28:21.216550+0000","last_became_active":"2023-10-16T13:28:20.105768+0000","last_became_peered":"2023-10-16T13:28:20.105768+0000","last_unstale":"2023-10-16T13:28:21.216550+0000","last_undegraded":"2023-10-16T13:28:21.216550+0000","last_fullsized":"2023-10-16T13:28:21.216550+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:17:03.172142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.27","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.090678+0000","last_change":"2023-10-16T13:28:20.103292+0000","last_active":"2023-10-16T13:28:21.090678+0000","last_peered":"2023-10-16T13:28:21.090678+0000","last_clean":"2023-10-16T13:28:21.090678+0000","last_became_active":"2023-10-16T13:28:20.103011+0000","last_became_peered":"2023-10-16T13:28:20.103011+0000","last_unstale":"2023-10-16T13:28:21.090678+0000","last_undegraded":"2023-10-16T13:28:21.090678+0000","last_fullsized":"2023-10-16T13:28:21.090678+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:22:31.339408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.26","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.214324+0000","last_change":"2023-10-16T13:28:20.116791+0000","last_active":"2023-10-16T13:28:21.214324+0000","last_peered":"2023-10-16T13:28:21.214324+0000","last_clean":"2023-10-16T13:28:21.214324+0000","last_became_active":"2023-10-16T13:28:20.107199+0000","last_became_peered":"2023-10-16T13:28:20.107199+0000","last_unstale":"2023-10-16T13:28:21.214324+0000","last_undegraded":"2023-10-16T13:28:21.214324+0000","last_fullsized":"2023-10-16T13:28:21.214324+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:06:05.629013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.25","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.216230+0000","last_change":"2023-10-16T13:28:20.112444+0000","last_active":"2023-10-16T13:28:21.216230+0000","last_peered":"2023-10-16T13:28:21.216230+0000","last_clean":"2023-10-16T13:28:21.216230+0000","last_became_active":"2023-10-16T13:28:20.112149+0000","last_became_peered":"2023-10-16T13:28:20.112149+0000","last_unstale":"2023-10-16T13:28:21.216230+0000","last_undegraded":"2023-10-16T13:28:21.216230+0000","last_fullsized":"2023-10-16T13:28:21.216230+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:42:51.321596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.24","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.090302+0000","last_change":"2023-10-16T13:28:20.111638+0000","last_active":"2023-10-16T13:28:21.090302+0000","last_peered":"2023-10-16T13:28:21.090302+0000","last_clean":"2023-10-16T13:28:21.090302+0000","last_became_active":"2023-10-16T13:28:20.111162+0000","last_became_peered":"2023-10-16T13:28:20.111162+0000","last_unstale":"2023-10-16T13:28:21.090302+0000","last_undegraded":"2023-10-16T13:28:21.090302+0000","last_fullsized":"2023-10-16T13:28:21.090302+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:40:37.979224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.23","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.215586+0000","last_change":"2023-10-16T13:28:20.107895+0000","last_active":"2023-10-16T13:28:21.215586+0000","last_peered":"2023-10-16T13:28:21.215586+0000","last_clean":"2023-10-16T13:28:21.215586+0000","last_became_active":"2023-10-16T13:28:20.106225+0000","last_became_peered":"2023-10-16T13:28:20.106225+0000","last_unstale":"2023-10-16T13:28:21.215586+0000","last_undegraded":"2023-10-16T13:28:21.215586+0000","last_fullsized":"2023-10-16T13:28:21.215586+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:40:08.759737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.22","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.215453+0000","last_change":"2023-10-16T13:28:20.099102+0000","last_active":"2023-10-16T13:28:21.215453+0000","last_peered":"2023-10-16T13:28:21.215453+0000","last_clean":"2023-10-16T13:28:21.215453+0000","last_became_active":"2023-10-16T13:28:20.098824+0000","last_became_peered":"2023-10-16T13:28:20.098824+0000","last_unstale":"2023-10-16T13:28:21.215453+0000","last_undegraded":"2023-10-16T13:28:21.215453+0000","last_fullsized":"2023-10-16T13:28:21.215453+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:22:51.779612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.21","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.215714+0000","last_change":"2023-10-16T13:28:20.094545+0000","last_active":"2023-10-16T13:28:21.215714+0000","last_peered":"2023-10-16T13:28:21.215714+0000","last_clean":"2023-10-16T13:28:21.215714+0000","last_became_active":"2023-10-16T13:28:20.094132+0000","last_became_peered":"2023-10-16T13:28:20.094132+0000","last_unstale":"2023-10-16T13:28:21.215714+0000","last_undegraded":"2023-10-16T13:28:21.215714+0000","last_fullsized":"2023-10-16T13:28:21.215714+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:07:30.602878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.20","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.215377+0000","last_change":"2023-10-16T13:28:20.105432+0000","last_active":"2023-10-16T13:28:21.215377+0000","last_peered":"2023-10-16T13:28:21.215377+0000","last_clean":"2023-10-16T13:28:21.215377+0000","last_became_active":"2023-10-16T13:28:20.105119+0000","last_became_peered":"2023-10-16T13:28:20.105119+0000","last_unstale":"2023-10-16T13:28:21.215377+0000","last_undegraded":"2023-10-16T13:28:21.215377+0000","last_fullsized":"2023-10-16T13:28:21.215377+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:17:43.138771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.10","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.215338+0000","last_change":"2023-10-16T13:28:20.112431+0000","last_active":"2023-10-16T13:28:21.215338+0000","last_peered":"2023-10-16T13:28:21.215338+0000","last_clean":"2023-10-16T13:28:21.215338+0000","last_became_active":"2023-10-16T13:28:20.112069+0000","last_became_peered":"2023-10-16T13:28:20.112069+0000","last_unstale":"2023-10-16T13:28:21.215338+0000","last_undegraded":"2023-10-16T13:28:21.215338+0000","last_fullsized":"2023-10-16T13:28:21.215338+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:53:47.291050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"189'4","reported_seq":16,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.216279+0000","last_change":"2023-10-16T13:28:19.104524+0000","last_active":"2023-10-16T13:28:21.216279+0000","last_peered":"2023-10-16T13:28:21.216279+0000","last_clean":"2023-10-16T13:28:21.216279+0000","last_became_active":"2023-10-16T13:28:19.104244+0000","last_became_peered":"2023-10-16T13:28:19.104244+0000","last_unstale":"2023-10-16T13:28:21.216279+0000","last_undegraded":"2023-10-16T13:28:21.216279+0000","last_fullsized":"2023-10-16T13:28:21.216279+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_clean_scrub_stamp":"2023-10-16T13:28:18.078605+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:35:49.979057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.090535+0000","last_change":"2023-10-16T13:28:20.101142+0000","last_active":"2023-10-16T13:28:21.090535+0000","last_peered":"2023-10-16T13:28:21.090535+0000","last_clean":"2023-10-16T13:28:21.090535+0000","last_became_active":"2023-10-16T13:28:20.100040+0000","last_became_peered":"2023-10-16T13:28:20.100040+0000","last_unstale":"2023-10-16T13:28:21.090535+0000","last_undegraded":"2023-10-16T13:28:21.090535+0000","last_fullsized":"2023-10-16T13:28:21.090535+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:22:21.987373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"189'1","reported_seq":13,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.090661+0000","last_change":"2023-10-16T13:28:19.105174+0000","last_active":"2023-10-16T13:28:21.090661+0000","last_peered":"2023-10-16T13:28:21.090661+0000","last_clean":"2023-10-16T13:28:21.090661+0000","last_became_active":"2023-10-16T13:28:19.104911+0000","last_became_peered":"2023-10-16T13:28:19.104911+0000","last_unstale":"2023-10-16T13:28:21.090661+0000","last_undegraded":"2023-10-16T13:28:21.090661+0000","last_fullsized":"2023-10-16T13:28:21.090661+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_clean_scrub_stamp":"2023-10-16T13:28:18.078605+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:41:44.303899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.089829+0000","last_change":"2023-10-16T13:28:20.129907+0000","last_active":"2023-10-16T13:28:21.089829+0000","last_peered":"2023-10-16T13:28:21.089829+0000","last_clean":"2023-10-16T13:28:21.089829+0000","last_became_active":"2023-10-16T13:28:20.129420+0000","last_became_peered":"2023-10-16T13:28:20.129420+0000","last_unstale":"2023-10-16T13:28:21.089829+0000","last_undegraded":"2023-10-16T13:28:21.089829+0000","last_fullsized":"2023-10-16T13:28:21.089829+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:53:08.194107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.1d","version":"189'2","reported_seq":14,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.216502+0000","last_change":"2023-10-16T13:28:19.105224+0000","last_active":"2023-10-16T13:28:21.216502+0000","last_peered":"2023-10-16T13:28:21.216502+0000","last_clean":"2023-10-16T13:28:21.216502+0000","last_became_active":"2023-10-16T13:28:19.104943+0000","last_became_peered":"2023-10-16T13:28:19.104943+0000","last_unstale":"2023-10-16T13:28:21.216502+0000","last_undegraded":"2023-10-16T13:28:21.216502+0000","last_fullsized":"2023-10-16T13:28:21.216502+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_clean_scrub_stamp":"2023-10-16T13:28:18.078605+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:30:52.074608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.13","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.215739+0000","last_change":"2023-10-16T13:28:20.112381+0000","last_active":"2023-10-16T13:28:21.215739+0000","last_peered":"2023-10-16T13:28:21.215739+0000","last_clean":"2023-10-16T13:28:21.215739+0000","last_became_active":"2023-10-16T13:28:20.112023+0000","last_became_peered":"2023-10-16T13:28:20.112023+0000","last_unstale":"2023-10-16T13:28:21.215739+0000","last_undegraded":"2023-10-16T13:28:21.215739+0000","last_fullsized":"2023-10-16T13:28:21.215739+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:51:19.846445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1c","version":"189'2","reported_seq":14,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.214302+0000","last_change":"2023-10-16T13:28:19.100575+0000","last_active":"2023-10-16T13:28:21.214302+0000","last_peered":"2023-10-16T13:28:21.214302+0000","last_clean":"2023-10-16T13:28:21.214302+0000","last_became_active":"2023-10-16T13:28:19.098362+0000","last_became_peered":"2023-10-16T13:28:19.098362+0000","last_unstale":"2023-10-16T13:28:21.214302+0000","last_undegraded":"2023-10-16T13:28:21.214302+0000","last_fullsized":"2023-10-16T13:28:21.214302+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_clean_scrub_stamp":"2023-10-16T13:28:18.078605+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:07:44.579395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.14","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.214514+0000","last_change":"2023-10-16T13:28:20.098841+0000","last_active":"2023-10-16T13:28:21.214514+0000","last_peered":"2023-10-16T13:28:21.214514+0000","last_clean":"2023-10-16T13:28:21.214514+0000","last_became_active":"2023-10-16T13:28:20.098478+0000","last_became_peered":"2023-10-16T13:28:20.098478+0000","last_unstale":"2023-10-16T13:28:21.214514+0000","last_undegraded":"2023-10-16T13:28:21.214514+0000","last_fullsized":"2023-10-16T13:28:21.214514+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:55:38.929310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1b","version":"189'1","reported_seq":13,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.214634+0000","last_change":"2023-10-16T13:28:19.110059+0000","last_active":"2023-10-16T13:28:21.214634+0000","last_peered":"2023-10-16T13:28:21.214634+0000","last_clean":"2023-10-16T13:28:21.214634+0000","last_became_active":"2023-10-16T13:28:19.109771+0000","last_became_peered":"2023-10-16T13:28:19.109771+0000","last_unstale":"2023-10-16T13:28:21.214634+0000","last_undegraded":"2023-10-16T13:28:21.214634+0000","last_fullsized":"2023-10-16T13:28:21.214634+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_clean_scrub_stamp":"2023-10-16T13:28:18.078605+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:52:28.628298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.15","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.214564+0000","last_change":"2023-10-16T13:28:20.109881+0000","last_active":"2023-10-16T13:28:21.214564+0000","last_peered":"2023-10-16T13:28:21.214564+0000","last_clean":"2023-10-16T13:28:21.214564+0000","last_became_active":"2023-10-16T13:28:20.109597+0000","last_became_peered":"2023-10-16T13:28:20.109597+0000","last_unstale":"2023-10-16T13:28:21.214564+0000","last_undegraded":"2023-10-16T13:28:21.214564+0000","last_fullsized":"2023-10-16T13:28:21.214564+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:28:42.998553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.1a","version":"189'1","reported_seq":13,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.089846+0000","last_change":"2023-10-16T13:28:19.106310+0000","last_active":"2023-10-16T13:28:21.089846+0000","last_peered":"2023-10-16T13:28:21.089846+0000","last_clean":"2023-10-16T13:28:21.089846+0000","last_became_active":"2023-10-16T13:28:19.105968+0000","last_became_peered":"2023-10-16T13:28:19.105968+0000","last_unstale":"2023-10-16T13:28:21.089846+0000","last_undegraded":"2023-10-16T13:28:21.089846+0000","last_fullsized":"2023-10-16T13:28:21.089846+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_clean_scrub_stamp":"2023-10-16T13:28:18.078605+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:42:50.317626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.214462+0000","last_change":"2023-10-16T13:28:20.109833+0000","last_active":"2023-10-16T13:28:21.214462+0000","last_peered":"2023-10-16T13:28:21.214462+0000","last_clean":"2023-10-16T13:28:21.214462+0000","last_became_active":"2023-10-16T13:28:20.104089+0000","last_became_peered":"2023-10-16T13:28:20.104089+0000","last_unstale":"2023-10-16T13:28:21.214462+0000","last_undegraded":"2023-10-16T13:28:21.214462+0000","last_fullsized":"2023-10-16T13:28:21.214462+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:31:25.372564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":12,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.213943+0000","last_change":"2023-10-16T13:28:19.107783+0000","last_active":"2023-10-16T13:28:21.213943+0000","last_peered":"2023-10-16T13:28:21.213943+0000","last_clean":"2023-10-16T13:28:21.213943+0000","last_became_active":"2023-10-16T13:28:19.107466+0000","last_became_peered":"2023-10-16T13:28:19.107466+0000","last_unstale":"2023-10-16T13:28:21.213943+0000","last_undegraded":"2023-10-16T13:28:21.213943+0000","last_fullsized":"2023-10-16T13:28:21.213943+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_clean_scrub_stamp":"2023-10-16T13:28:18.078605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:00:29.699489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.216546+0000","last_change":"2023-10-16T13:28:20.103276+0000","last_active":"2023-10-16T13:28:21.216546+0000","last_peered":"2023-10-16T13:28:21.216546+0000","last_clean":"2023-10-16T13:28:21.216546+0000","last_became_active":"2023-10-16T13:28:20.102935+0000","last_became_peered":"2023-10-16T13:28:20.102935+0000","last_unstale":"2023-10-16T13:28:21.216546+0000","last_undegraded":"2023-10-16T13:28:21.216546+0000","last_fullsized":"2023-10-16T13:28:21.216546+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:19:49.628069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":12,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.214042+0000","last_change":"2023-10-16T13:28:19.103446+0000","last_active":"2023-10-16T13:28:21.214042+0000","last_peered":"2023-10-16T13:28:21.214042+0000","last_clean":"2023-10-16T13:28:21.214042+0000","last_became_active":"2023-10-16T13:28:19.102980+0000","last_became_peered":"2023-10-16T13:28:19.102980+0000","last_unstale":"2023-10-16T13:28:21.214042+0000","last_undegraded":"2023-10-16T13:28:21.214042+0000","last_fullsized":"2023-10-16T13:28:21.214042+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_clean_scrub_stamp":"2023-10-16T13:28:18.078605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:10:13.208037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.215167+0000","last_change":"2023-10-16T13:28:20.112844+0000","last_active":"2023-10-16T13:28:21.215167+0000","last_peered":"2023-10-16T13:28:21.215167+0000","last_clean":"2023-10-16T13:28:21.215167+0000","last_became_active":"2023-10-16T13:28:20.112558+0000","last_became_peered":"2023-10-16T13:28:20.112558+0000","last_unstale":"2023-10-16T13:28:21.215167+0000","last_undegraded":"2023-10-16T13:28:21.215167+0000","last_fullsized":"2023-10-16T13:28:21.215167+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:40:14.995519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":12,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.090729+0000","last_change":"2023-10-16T13:28:19.101765+0000","last_active":"2023-10-16T13:28:21.090729+0000","last_peered":"2023-10-16T13:28:21.090729+0000","last_clean":"2023-10-16T13:28:21.090729+0000","last_became_active":"2023-10-16T13:28:19.101518+0000","last_became_peered":"2023-10-16T13:28:19.101518+0000","last_unstale":"2023-10-16T13:28:21.090729+0000","last_undegraded":"2023-10-16T13:28:21.090729+0000","last_fullsized":"2023-10-16T13:28:21.090729+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_clean_scrub_stamp":"2023-10-16T13:28:18.078605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:25:47.343382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.090923+0000","last_change":"2023-10-16T13:28:20.115429+0000","last_active":"2023-10-16T13:28:21.090923+0000","last_peered":"2023-10-16T13:28:21.090923+0000","last_clean":"2023-10-16T13:28:21.090923+0000","last_became_active":"2023-10-16T13:28:20.115090+0000","last_became_peered":"2023-10-16T13:28:20.115090+0000","last_unstale":"2023-10-16T13:28:21.090923+0000","last_undegraded":"2023-10-16T13:28:21.090923+0000","last_fullsized":"2023-10-16T13:28:21.090923+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:34:43.083147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.15","version":"189'3","reported_seq":15,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.215683+0000","last_change":"2023-10-16T13:28:19.098263+0000","last_active":"2023-10-16T13:28:21.215683+0000","last_peered":"2023-10-16T13:28:21.215683+0000","last_clean":"2023-10-16T13:28:21.215683+0000","last_became_active":"2023-10-16T13:28:19.097824+0000","last_became_peered":"2023-10-16T13:28:19.097824+0000","last_unstale":"2023-10-16T13:28:21.215683+0000","last_undegraded":"2023-10-16T13:28:21.215683+0000","last_fullsized":"2023-10-16T13:28:21.215683+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_clean_scrub_stamp":"2023-10-16T13:28:18.078605+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:23:12.689030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1b","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.214673+0000","last_change":"2023-10-16T13:28:20.127366+0000","last_active":"2023-10-16T13:28:21.214673+0000","last_peered":"2023-10-16T13:28:21.214673+0000","last_clean":"2023-10-16T13:28:21.214673+0000","last_became_active":"2023-10-16T13:28:20.127030+0000","last_became_peered":"2023-10-16T13:28:20.127030+0000","last_unstale":"2023-10-16T13:28:21.214673+0000","last_undegraded":"2023-10-16T13:28:21.214673+0000","last_fullsized":"2023-10-16T13:28:21.214673+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:01:21.020025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"189'6","reported_seq":18,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.090546+0000","last_change":"2023-10-16T13:28:19.099350+0000","last_active":"2023-10-16T13:28:21.090546+0000","last_peered":"2023-10-16T13:28:21.090546+0000","last_clean":"2023-10-16T13:28:21.090546+0000","last_became_active":"2023-10-16T13:28:19.099093+0000","last_became_peered":"2023-10-16T13:28:19.099093+0000","last_unstale":"2023-10-16T13:28:21.090546+0000","last_undegraded":"2023-10-16T13:28:21.090546+0000","last_fullsized":"2023-10-16T13:28:21.090546+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_clean_scrub_stamp":"2023-10-16T13:28:18.078605+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:49:33.267714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1c","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.215238+0000","last_change":"2023-10-16T13:28:20.110489+0000","last_active":"2023-10-16T13:28:21.215238+0000","last_peered":"2023-10-16T13:28:21.215238+0000","last_clean":"2023-10-16T13:28:21.215238+0000","last_became_active":"2023-10-16T13:28:20.110171+0000","last_became_peered":"2023-10-16T13:28:20.110171+0000","last_unstale":"2023-10-16T13:28:21.215238+0000","last_undegraded":"2023-10-16T13:28:21.215238+0000","last_fullsized":"2023-10-16T13:28:21.215238+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:25:31.239161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.13","version":"189'2","reported_seq":14,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.090371+0000","last_change":"2023-10-16T13:28:19.100376+0000","last_active":"2023-10-16T13:28:21.090371+0000","last_peered":"2023-10-16T13:28:21.090371+0000","last_clean":"2023-10-16T13:28:21.090371+0000","last_became_active":"2023-10-16T13:28:19.100135+0000","last_became_peered":"2023-10-16T13:28:19.100135+0000","last_unstale":"2023-10-16T13:28:21.090371+0000","last_undegraded":"2023-10-16T13:28:21.090371+0000","last_fullsized":"2023-10-16T13:28:21.090371+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_clean_scrub_stamp":"2023-10-16T13:28:18.078605+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:40:13.991550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1d","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.215020+0000","last_change":"2023-10-16T13:28:20.098726+0000","last_active":"2023-10-16T13:28:21.215020+0000","last_peered":"2023-10-16T13:28:21.215020+0000","last_clean":"2023-10-16T13:28:21.215020+0000","last_became_active":"2023-10-16T13:28:20.098176+0000","last_became_peered":"2023-10-16T13:28:20.098176+0000","last_unstale":"2023-10-16T13:28:21.215020+0000","last_undegraded":"2023-10-16T13:28:21.215020+0000","last_fullsized":"2023-10-16T13:28:21.215020+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:30:04.900461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":12,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.090162+0000","last_change":"2023-10-16T13:28:19.102894+0000","last_active":"2023-10-16T13:28:21.090162+0000","last_peered":"2023-10-16T13:28:21.090162+0000","last_clean":"2023-10-16T13:28:21.090162+0000","last_became_active":"2023-10-16T13:28:19.102601+0000","last_became_peered":"2023-10-16T13:28:19.102601+0000","last_unstale":"2023-10-16T13:28:21.090162+0000","last_undegraded":"2023-10-16T13:28:21.090162+0000","last_fullsized":"2023-10-16T13:28:21.090162+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_clean_scrub_stamp":"2023-10-16T13:28:18.078605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:24:16.739829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1e","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.215821+0000","last_change":"2023-10-16T13:28:20.106915+0000","last_active":"2023-10-16T13:28:21.215821+0000","last_peered":"2023-10-16T13:28:21.215821+0000","last_clean":"2023-10-16T13:28:21.215821+0000","last_became_active":"2023-10-16T13:28:20.106673+0000","last_became_peered":"2023-10-16T13:28:20.106673+0000","last_unstale":"2023-10-16T13:28:21.215821+0000","last_undegraded":"2023-10-16T13:28:21.215821+0000","last_fullsized":"2023-10-16T13:28:21.215821+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:57:53.529437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":12,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.215549+0000","last_change":"2023-10-16T13:28:19.105790+0000","last_active":"2023-10-16T13:28:21.215549+0000","last_peered":"2023-10-16T13:28:21.215549+0000","last_clean":"2023-10-16T13:28:21.215549+0000","last_became_active":"2023-10-16T13:28:19.105411+0000","last_became_peered":"2023-10-16T13:28:19.105411+0000","last_unstale":"2023-10-16T13:28:21.215549+0000","last_undegraded":"2023-10-16T13:28:21.215549+0000","last_fullsized":"2023-10-16T13:28:21.215549+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_clean_scrub_stamp":"2023-10-16T13:28:18.078605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:59:27.628317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1f","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.214931+0000","last_change":"2023-10-16T13:28:20.116267+0000","last_active":"2023-10-16T13:28:21.214931+0000","last_peered":"2023-10-16T13:28:21.214931+0000","last_clean":"2023-10-16T13:28:21.214931+0000","last_became_active":"2023-10-16T13:28:20.115967+0000","last_became_peered":"2023-10-16T13:28:20.115967+0000","last_unstale":"2023-10-16T13:28:21.214931+0000","last_undegraded":"2023-10-16T13:28:21.214931+0000","last_fullsized":"2023-10-16T13:28:21.214931+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:28:35.228943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.10","version":"189'8","reported_seq":20,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.214282+0000","last_change":"2023-10-16T13:28:19.099553+0000","last_active":"2023-10-16T13:28:21.214282+0000","last_peered":"2023-10-16T13:28:21.214282+0000","last_clean":"2023-10-16T13:28:21.214282+0000","last_became_active":"2023-10-16T13:28:19.099272+0000","last_became_peered":"2023-10-16T13:28:19.099272+0000","last_unstale":"2023-10-16T13:28:21.214282+0000","last_undegraded":"2023-10-16T13:28:21.214282+0000","last_fullsized":"2023-10-16T13:28:21.214282+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_clean_scrub_stamp":"2023-10-16T13:28:18.078605+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:14:10.839754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.214186+0000","last_change":"2023-10-16T13:28:20.128281+0000","last_active":"2023-10-16T13:28:21.214186+0000","last_peered":"2023-10-16T13:28:21.214186+0000","last_clean":"2023-10-16T13:28:21.214186+0000","last_became_active":"2023-10-16T13:28:20.127986+0000","last_became_peered":"2023-10-16T13:28:20.127986+0000","last_unstale":"2023-10-16T13:28:21.214186+0000","last_undegraded":"2023-10-16T13:28:21.214186+0000","last_fullsized":"2023-10-16T13:28:21.214186+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:19:07.823278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.f","version":"189'1","reported_seq":13,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.090766+0000","last_change":"2023-10-16T13:28:19.100121+0000","last_active":"2023-10-16T13:28:21.090766+0000","last_peered":"2023-10-16T13:28:21.090766+0000","last_clean":"2023-10-16T13:28:21.090766+0000","last_became_active":"2023-10-16T13:28:19.099767+0000","last_became_peered":"2023-10-16T13:28:19.099767+0000","last_unstale":"2023-10-16T13:28:21.090766+0000","last_undegraded":"2023-10-16T13:28:21.090766+0000","last_fullsized":"2023-10-16T13:28:21.090766+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_clean_scrub_stamp":"2023-10-16T13:28:18.078605+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:32:58.614800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.215996+0000","last_change":"2023-10-16T13:28:20.096710+0000","last_active":"2023-10-16T13:28:21.215996+0000","last_peered":"2023-10-16T13:28:21.215996+0000","last_clean":"2023-10-16T13:28:21.215996+0000","last_became_active":"2023-10-16T13:28:20.096293+0000","last_became_peered":"2023-10-16T13:28:20.096293+0000","last_unstale":"2023-10-16T13:28:21.215996+0000","last_undegraded":"2023-10-16T13:28:21.215996+0000","last_fullsized":"2023-10-16T13:28:21.215996+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:53:31.172370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.e","version":"189'2","reported_seq":14,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.214634+0000","last_change":"2023-10-16T13:28:19.108810+0000","last_active":"2023-10-16T13:28:21.214634+0000","last_peered":"2023-10-16T13:28:21.214634+0000","last_clean":"2023-10-16T13:28:21.214634+0000","last_became_active":"2023-10-16T13:28:19.108522+0000","last_became_peered":"2023-10-16T13:28:19.108522+0000","last_unstale":"2023-10-16T13:28:21.214634+0000","last_undegraded":"2023-10-16T13:28:21.214634+0000","last_fullsized":"2023-10-16T13:28:21.214634+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_clean_scrub_stamp":"2023-10-16T13:28:18.078605+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:58:39.810683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.215228+0000","last_change":"2023-10-16T13:28:20.102972+0000","last_active":"2023-10-16T13:28:21.215228+0000","last_peered":"2023-10-16T13:28:21.215228+0000","last_clean":"2023-10-16T13:28:21.215228+0000","last_became_active":"2023-10-16T13:28:20.102654+0000","last_became_peered":"2023-10-16T13:28:20.102654+0000","last_unstale":"2023-10-16T13:28:21.215228+0000","last_undegraded":"2023-10-16T13:28:21.215228+0000","last_fullsized":"2023-10-16T13:28:21.215228+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:15:57.533753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.d","version":"189'1","reported_seq":13,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.090485+0000","last_change":"2023-10-16T13:28:19.101127+0000","last_active":"2023-10-16T13:28:21.090485+0000","last_peered":"2023-10-16T13:28:21.090485+0000","last_clean":"2023-10-16T13:28:21.090485+0000","last_became_active":"2023-10-16T13:28:19.100857+0000","last_became_peered":"2023-10-16T13:28:19.100857+0000","last_unstale":"2023-10-16T13:28:21.090485+0000","last_undegraded":"2023-10-16T13:28:21.090485+0000","last_fullsized":"2023-10-16T13:28:21.090485+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_clean_scrub_stamp":"2023-10-16T13:28:18.078605+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:34:58.602698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.215586+0000","last_change":"2023-10-16T13:28:20.098940+0000","last_active":"2023-10-16T13:28:21.215586+0000","last_peered":"2023-10-16T13:28:21.215586+0000","last_clean":"2023-10-16T13:28:21.215586+0000","last_became_active":"2023-10-16T13:28:20.098644+0000","last_became_peered":"2023-10-16T13:28:20.098644+0000","last_unstale":"2023-10-16T13:28:21.215586+0000","last_undegraded":"2023-10-16T13:28:21.215586+0000","last_fullsized":"2023-10-16T13:28:21.215586+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:20:39.551231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":12,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.214665+0000","last_change":"2023-10-16T13:28:19.109251+0000","last_active":"2023-10-16T13:28:21.214665+0000","last_peered":"2023-10-16T13:28:21.214665+0000","last_clean":"2023-10-16T13:28:21.214665+0000","last_became_active":"2023-10-16T13:28:19.108876+0000","last_became_peered":"2023-10-16T13:28:19.108876+0000","last_unstale":"2023-10-16T13:28:21.214665+0000","last_undegraded":"2023-10-16T13:28:21.214665+0000","last_fullsized":"2023-10-16T13:28:21.214665+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_clean_scrub_stamp":"2023-10-16T13:28:18.078605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:01:27.654076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.090442+0000","last_change":"2023-10-16T13:28:20.111460+0000","last_active":"2023-10-16T13:28:21.090442+0000","last_peered":"2023-10-16T13:28:21.090442+0000","last_clean":"2023-10-16T13:28:21.090442+0000","last_became_active":"2023-10-16T13:28:20.110770+0000","last_became_peered":"2023-10-16T13:28:20.110770+0000","last_unstale":"2023-10-16T13:28:21.090442+0000","last_undegraded":"2023-10-16T13:28:21.090442+0000","last_fullsized":"2023-10-16T13:28:21.090442+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:36:27.353238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.b","version":"189'2","reported_seq":14,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.090419+0000","last_change":"2023-10-16T13:28:19.105265+0000","last_active":"2023-10-16T13:28:21.090419+0000","last_peered":"2023-10-16T13:28:21.090419+0000","last_clean":"2023-10-16T13:28:21.090419+0000","last_became_active":"2023-10-16T13:28:19.104959+0000","last_became_peered":"2023-10-16T13:28:19.104959+0000","last_unstale":"2023-10-16T13:28:21.090419+0000","last_undegraded":"2023-10-16T13:28:21.090419+0000","last_fullsized":"2023-10-16T13:28:21.090419+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_clean_scrub_stamp":"2023-10-16T13:28:18.078605+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:53:46.287080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.090302+0000","last_change":"2023-10-16T13:28:20.104597+0000","last_active":"2023-10-16T13:28:21.090302+0000","last_peered":"2023-10-16T13:28:21.090302+0000","last_clean":"2023-10-16T13:28:21.090302+0000","last_became_active":"2023-10-16T13:28:20.102370+0000","last_became_peered":"2023-10-16T13:28:20.102370+0000","last_unstale":"2023-10-16T13:28:21.090302+0000","last_undegraded":"2023-10-16T13:28:21.090302+0000","last_fullsized":"2023-10-16T13:28:21.090302+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:08:46.302886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":12,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.215334+0000","last_change":"2023-10-16T13:28:19.104160+0000","last_active":"2023-10-16T13:28:21.215334+0000","last_peered":"2023-10-16T13:28:21.215334+0000","last_clean":"2023-10-16T13:28:21.215334+0000","last_became_active":"2023-10-16T13:28:19.103823+0000","last_became_peered":"2023-10-16T13:28:19.103823+0000","last_unstale":"2023-10-16T13:28:21.215334+0000","last_undegraded":"2023-10-16T13:28:21.215334+0000","last_fullsized":"2023-10-16T13:28:21.215334+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_clean_scrub_stamp":"2023-10-16T13:28:18.078605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:48:59.692819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.090623+0000","last_change":"2023-10-16T13:28:20.101156+0000","last_active":"2023-10-16T13:28:21.090623+0000","last_peered":"2023-10-16T13:28:21.090623+0000","last_clean":"2023-10-16T13:28:21.090623+0000","last_became_active":"2023-10-16T13:28:20.100880+0000","last_became_peered":"2023-10-16T13:28:20.100880+0000","last_unstale":"2023-10-16T13:28:21.090623+0000","last_undegraded":"2023-10-16T13:28:21.090623+0000","last_fullsized":"2023-10-16T13:28:21.090623+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:24:17.743799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":12,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.213959+0000","last_change":"2023-10-16T13:28:19.104614+0000","last_active":"2023-10-16T13:28:21.213959+0000","last_peered":"2023-10-16T13:28:21.213959+0000","last_clean":"2023-10-16T13:28:21.213959+0000","last_became_active":"2023-10-16T13:28:19.104290+0000","last_became_peered":"2023-10-16T13:28:19.104290+0000","last_unstale":"2023-10-16T13:28:21.213959+0000","last_undegraded":"2023-10-16T13:28:21.213959+0000","last_fullsized":"2023-10-16T13:28:21.213959+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_clean_scrub_stamp":"2023-10-16T13:28:18.078605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:11:41.561063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.215993+0000","last_change":"2023-10-16T13:28:20.127626+0000","last_active":"2023-10-16T13:28:21.215993+0000","last_peered":"2023-10-16T13:28:21.215993+0000","last_clean":"2023-10-16T13:28:21.215993+0000","last_became_active":"2023-10-16T13:28:20.127377+0000","last_became_peered":"2023-10-16T13:28:20.127377+0000","last_unstale":"2023-10-16T13:28:21.215993+0000","last_undegraded":"2023-10-16T13:28:21.215993+0000","last_fullsized":"2023-10-16T13:28:21.215993+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:34:59.606668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":12,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.215445+0000","last_change":"2023-10-16T13:28:19.102225+0000","last_active":"2023-10-16T13:28:21.215445+0000","last_peered":"2023-10-16T13:28:21.215445+0000","last_clean":"2023-10-16T13:28:21.215445+0000","last_became_active":"2023-10-16T13:28:19.101887+0000","last_became_peered":"2023-10-16T13:28:19.101887+0000","last_unstale":"2023-10-16T13:28:21.215445+0000","last_undegraded":"2023-10-16T13:28:21.215445+0000","last_fullsized":"2023-10-16T13:28:21.215445+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_clean_scrub_stamp":"2023-10-16T13:28:18.078605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:11:00.906437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.215167+0000","last_change":"2023-10-16T13:28:20.093505+0000","last_active":"2023-10-16T13:28:21.215167+0000","last_peered":"2023-10-16T13:28:21.215167+0000","last_clean":"2023-10-16T13:28:21.215167+0000","last_became_active":"2023-10-16T13:28:20.093190+0000","last_became_peered":"2023-10-16T13:28:20.093190+0000","last_unstale":"2023-10-16T13:28:21.215167+0000","last_undegraded":"2023-10-16T13:28:21.215167+0000","last_fullsized":"2023-10-16T13:28:21.215167+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:26:02.101104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.7","version":"189'2","reported_seq":14,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.090733+0000","last_change":"2023-10-16T13:28:19.105641+0000","last_active":"2023-10-16T13:28:21.090733+0000","last_peered":"2023-10-16T13:28:21.090733+0000","last_clean":"2023-10-16T13:28:21.090733+0000","last_became_active":"2023-10-16T13:28:19.105360+0000","last_became_peered":"2023-10-16T13:28:19.105360+0000","last_unstale":"2023-10-16T13:28:21.090733+0000","last_undegraded":"2023-10-16T13:28:21.090733+0000","last_fullsized":"2023-10-16T13:28:21.090733+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_clean_scrub_stamp":"2023-10-16T13:28:18.078605+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:11:00.906437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.214349+0000","last_change":"2023-10-16T13:28:20.116732+0000","last_active":"2023-10-16T13:28:21.214349+0000","last_peered":"2023-10-16T13:28:21.214349+0000","last_clean":"2023-10-16T13:28:21.214349+0000","last_became_active":"2023-10-16T13:28:20.116106+0000","last_became_peered":"2023-10-16T13:28:20.116106+0000","last_unstale":"2023-10-16T13:28:21.214349+0000","last_undegraded":"2023-10-16T13:28:21.214349+0000","last_fullsized":"2023-10-16T13:28:21.214349+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:04:42.792375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.6","version":"0'0","reported_seq":12,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.090626+0000","last_change":"2023-10-16T13:28:19.104697+0000","last_active":"2023-10-16T13:28:21.090626+0000","last_peered":"2023-10-16T13:28:21.090626+0000","last_clean":"2023-10-16T13:28:21.090626+0000","last_became_active":"2023-10-16T13:28:19.104014+0000","last_became_peered":"2023-10-16T13:28:19.104014+0000","last_unstale":"2023-10-16T13:28:21.090626+0000","last_undegraded":"2023-10-16T13:28:21.090626+0000","last_fullsized":"2023-10-16T13:28:21.090626+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_clean_scrub_stamp":"2023-10-16T13:28:18.078605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:08:45.298916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.214046+0000","last_change":"2023-10-16T13:28:20.128832+0000","last_active":"2023-10-16T13:28:21.214046+0000","last_peered":"2023-10-16T13:28:21.214046+0000","last_clean":"2023-10-16T13:28:21.214046+0000","last_became_active":"2023-10-16T13:28:20.128413+0000","last_became_peered":"2023-10-16T13:28:20.128413+0000","last_unstale":"2023-10-16T13:28:21.214046+0000","last_undegraded":"2023-10-16T13:28:21.214046+0000","last_fullsized":"2023-10-16T13:28:21.214046+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:17:30.158648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":12,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.090840+0000","last_change":"2023-10-16T13:28:19.098941+0000","last_active":"2023-10-16T13:28:21.090840+0000","last_peered":"2023-10-16T13:28:21.090840+0000","last_clean":"2023-10-16T13:28:21.090840+0000","last_became_active":"2023-10-16T13:28:19.098663+0000","last_became_peered":"2023-10-16T13:28:19.098663+0000","last_unstale":"2023-10-16T13:28:21.090840+0000","last_undegraded":"2023-10-16T13:28:21.090840+0000","last_fullsized":"2023-10-16T13:28:21.090840+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_clean_scrub_stamp":"2023-10-16T13:28:18.078605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:48:01.872328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.b","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.214535+0000","last_change":"2023-10-16T13:28:20.127422+0000","last_active":"2023-10-16T13:28:21.214535+0000","last_peered":"2023-10-16T13:28:21.214535+0000","last_clean":"2023-10-16T13:28:21.214535+0000","last_became_active":"2023-10-16T13:28:20.127075+0000","last_became_peered":"2023-10-16T13:28:20.127075+0000","last_unstale":"2023-10-16T13:28:21.214535+0000","last_undegraded":"2023-10-16T13:28:21.214535+0000","last_fullsized":"2023-10-16T13:28:21.214535+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:04:05.766711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":12,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.214389+0000","last_change":"2023-10-16T13:28:19.103735+0000","last_active":"2023-10-16T13:28:21.214389+0000","last_peered":"2023-10-16T13:28:21.214389+0000","last_clean":"2023-10-16T13:28:21.214389+0000","last_became_active":"2023-10-16T13:28:19.102055+0000","last_became_peered":"2023-10-16T13:28:19.102055+0000","last_unstale":"2023-10-16T13:28:21.214389+0000","last_undegraded":"2023-10-16T13:28:21.214389+0000","last_fullsized":"2023-10-16T13:28:21.214389+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_clean_scrub_stamp":"2023-10-16T13:28:18.078605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:40:30.583418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.216039+0000","last_change":"2023-10-16T13:28:20.103042+0000","last_active":"2023-10-16T13:28:21.216039+0000","last_peered":"2023-10-16T13:28:21.216039+0000","last_clean":"2023-10-16T13:28:21.216039+0000","last_became_active":"2023-10-16T13:28:20.100745+0000","last_became_peered":"2023-10-16T13:28:20.100745+0000","last_unstale":"2023-10-16T13:28:21.216039+0000","last_undegraded":"2023-10-16T13:28:21.216039+0000","last_fullsized":"2023-10-16T13:28:21.216039+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:55:53.825888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":12,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.216277+0000","last_change":"2023-10-16T13:28:19.106585+0000","last_active":"2023-10-16T13:28:21.216277+0000","last_peered":"2023-10-16T13:28:21.216277+0000","last_clean":"2023-10-16T13:28:21.216277+0000","last_became_active":"2023-10-16T13:28:19.106272+0000","last_became_peered":"2023-10-16T13:28:19.106272+0000","last_unstale":"2023-10-16T13:28:21.216277+0000","last_undegraded":"2023-10-16T13:28:21.216277+0000","last_fullsized":"2023-10-16T13:28:21.216277+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_clean_scrub_stamp":"2023-10-16T13:28:18.078605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:55:36.675550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.d","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.216274+0000","last_change":"2023-10-16T13:28:20.112242+0000","last_active":"2023-10-16T13:28:21.216274+0000","last_peered":"2023-10-16T13:28:21.216274+0000","last_clean":"2023-10-16T13:28:21.216274+0000","last_became_active":"2023-10-16T13:28:20.111971+0000","last_became_peered":"2023-10-16T13:28:20.111971+0000","last_unstale":"2023-10-16T13:28:21.216274+0000","last_undegraded":"2023-10-16T13:28:21.216274+0000","last_fullsized":"2023-10-16T13:28:21.216274+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:32:59.618770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":12,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.090242+0000","last_change":"2023-10-16T13:28:19.091686+0000","last_active":"2023-10-16T13:28:21.090242+0000","last_peered":"2023-10-16T13:28:21.090242+0000","last_clean":"2023-10-16T13:28:21.090242+0000","last_became_active":"2023-10-16T13:28:19.091391+0000","last_became_peered":"2023-10-16T13:28:19.091391+0000","last_unstale":"2023-10-16T13:28:21.090242+0000","last_undegraded":"2023-10-16T13:28:21.090242+0000","last_fullsized":"2023-10-16T13:28:21.090242+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_clean_scrub_stamp":"2023-10-16T13:28:18.078605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:29:26.913110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.e","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.090418+0000","last_change":"2023-10-16T13:28:20.111062+0000","last_active":"2023-10-16T13:28:21.090418+0000","last_peered":"2023-10-16T13:28:21.090418+0000","last_clean":"2023-10-16T13:28:21.090418+0000","last_became_active":"2023-10-16T13:28:20.110750+0000","last_became_peered":"2023-10-16T13:28:20.110750+0000","last_unstale":"2023-10-16T13:28:21.090418+0000","last_undegraded":"2023-10-16T13:28:21.090418+0000","last_fullsized":"2023-10-16T13:28:21.090418+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:43:10.467090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":12,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.215817+0000","last_change":"2023-10-16T13:28:19.105211+0000","last_active":"2023-10-16T13:28:21.215817+0000","last_peered":"2023-10-16T13:28:21.215817+0000","last_clean":"2023-10-16T13:28:21.215817+0000","last_became_active":"2023-10-16T13:28:19.104879+0000","last_became_peered":"2023-10-16T13:28:19.104879+0000","last_unstale":"2023-10-16T13:28:21.215817+0000","last_undegraded":"2023-10-16T13:28:21.215817+0000","last_fullsized":"2023-10-16T13:28:21.215817+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_clean_scrub_stamp":"2023-10-16T13:28:18.078605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:30:03.896491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.f","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.214642+0000","last_change":"2023-10-16T13:28:20.118722+0000","last_active":"2023-10-16T13:28:21.214642+0000","last_peered":"2023-10-16T13:28:21.214642+0000","last_clean":"2023-10-16T13:28:21.214642+0000","last_became_active":"2023-10-16T13:28:20.117253+0000","last_became_peered":"2023-10-16T13:28:20.117253+0000","last_unstale":"2023-10-16T13:28:21.214642+0000","last_undegraded":"2023-10-16T13:28:21.214642+0000","last_fullsized":"2023-10-16T13:28:21.214642+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:33:47.029032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":12,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.089405+0000","last_change":"2023-10-16T13:28:19.106788+0000","last_active":"2023-10-16T13:28:21.089405+0000","last_peered":"2023-10-16T13:28:21.089405+0000","last_clean":"2023-10-16T13:28:21.089405+0000","last_became_active":"2023-10-16T13:28:19.105953+0000","last_became_peered":"2023-10-16T13:28:19.105953+0000","last_unstale":"2023-10-16T13:28:21.089405+0000","last_undegraded":"2023-10-16T13:28:21.089405+0000","last_fullsized":"2023-10-16T13:28:21.089405+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_clean_scrub_stamp":"2023-10-16T13:28:18.078605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:17:50.970152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":11,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.216324+0000","last_change":"2023-10-16T13:28:20.105567+0000","last_active":"2023-10-16T13:28:21.216324+0000","last_peered":"2023-10-16T13:28:21.216324+0000","last_clean":"2023-10-16T13:28:21.216324+0000","last_became_active":"2023-10-16T13:28:20.105296+0000","last_became_peered":"2023-10-16T13:28:20.105296+0000","last_unstale":"2023-10-16T13:28:21.216324+0000","last_undegraded":"2023-10-16T13:28:21.216324+0000","last_fullsized":"2023-10-16T13:28:21.216324+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":188,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:19.082575+0000","last_clean_scrub_stamp":"2023-10-16T13:28:19.082575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:19:15.031376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":12,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-16T13:28:21.090518+0000","last_change":"2023-10-16T13:28:19.104937+0000","last_active":"2023-10-16T13:28:21.090518+0000","last_peered":"2023-10-16T13:28:21.090518+0000","last_clean":"2023-10-16T13:28:21.090518+0000","last_became_active":"2023-10-16T13:28:19.104398+0000","last_became_peered":"2023-10-16T13:28:19.104398+0000","last_unstale":"2023-10-16T13:28:21.090518+0000","last_undegraded":"2023-10-16T13:28:21.090518+0000","last_fullsized":"2023-10-16T13:28:21.090518+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:28:18.078605+0000","last_clean_scrub_stamp":"2023-10-16T13:28:18.078605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:19:14.027406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"180'293","reported_seq":969,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-10-16T13:28:20.101683+0000","last_change":"2023-10-16T12:10:34.746820+0000","last_active":"2023-10-16T13:28:20.101683+0000","last_peered":"2023-10-16T13:28:20.101683+0000","last_clean":"2023-10-16T13:28:20.101683+0000","last_became_active":"2023-10-16T12:10:34.745574+0000","last_became_peered":"2023-10-16T12:10:34.745574+0000","last_unstale":"2023-10-16T13:28:20.101683+0000","last_undegraded":"2023-10-16T13:28:20.101683+0000","last_fullsized":"2023-10-16T13:28:20.101683+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_clean_scrub_stamp":"2023-10-16T12:10:33.577111+0000","objects_scrubbed":0,"log_size":293,"log_dups_size":0,"ondisk_log_size":293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:14.461986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1652,"num_read_kb":3396,"num_write":479,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":24,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":23,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1652,"num_read_kb":3396,"num_write":479,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":293,"ondisk_log_size":293,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575773,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52120,"kb_used_data":2892,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319720,"statfs":{"total":96636764160,"available":96583393280,"internally_reserved":0,"allocated":2961408,"data_stored":1796479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":50395595},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:28:07 2023","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.66400000000000003,"15min":0.64000000000000001},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":3.218,"5min":3.218,"15min":3.218},"last":0.53800000000000003},{"interface":"front","average":{"1min":1.0229999999999999,"5min":0.71399999999999997,"15min":0.65600000000000003},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":9.1270000000000007,"5min":9.1270000000000007,"15min":9.1270000000000007},"last":0.78200000000000003}]},{"osd":1,"last update":"Mon Oct 16 13:28:07 2023","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":0.71999999999999997,"15min":0.66100000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":8.7539999999999996,"5min":8.7539999999999996,"15min":8.7539999999999996},"last":0.69099999999999995},{"interface":"front","average":{"1min":1.0609999999999999,"5min":0.72799999999999998,"15min":0.66100000000000003},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.36899999999999999},"max":{"1min":9.173,"5min":9.173,"15min":9.173},"last":0.66000000000000003}]},{"osd":2,"last update":"Mon Oct 16 13:28:07 2023","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.74099999999999999,"15min":0.70699999999999996},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":6.5869999999999997,"5min":6.5869999999999997,"15min":6.5869999999999997},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.84899999999999998,"5min":0.72899999999999998,"15min":0.70999999999999996},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":3.0640000000000001,"5min":3.0640000000000001,"15min":3.0640000000000001},"last":0.79700000000000004}]},{"osd":3,"last update":"Mon Oct 16 13:28:07 2023","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.67200000000000004,"15min":0.65400000000000003},"min":{"1min":0.41399999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":3.1299999999999999,"5min":3.1299999999999999,"15min":3.1299999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":1.0569999999999999,"5min":0.73699999999999999,"15min":0.68400000000000005},"min":{"1min":0.441,"5min":0.441,"15min":0.40899999999999997},"max":{"1min":8.9920000000000009,"5min":8.9920000000000009,"15min":8.9920000000000009},"last":0.73599999999999999}]},{"osd":4,"last update":"Mon Oct 16 13:28:07 2023","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":0.59199999999999997,"15min":0.55000000000000004},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.313},"max":{"1min":6.3769999999999998,"5min":6.3769999999999998,"15min":6.3769999999999998},"last":0.433},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.66500000000000004,"15min":0.63600000000000001},"min":{"1min":0.35299999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":3.1779999999999999,"5min":3.1779999999999999,"15min":3.1779999999999999},"last":0.63400000000000001}]},{"osd":5,"last update":"Mon Oct 16 13:28:07 2023","interfaces":[{"interface":"back","average":{"1min":1.0940000000000001,"5min":0.65400000000000003,"15min":0.57899999999999996},"min":{"1min":0.36799999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":8.9209999999999994,"5min":8.9209999999999994,"15min":8.9209999999999994},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.99399999999999999,"5min":0.66400000000000003,"15min":0.621},"min":{"1min":0.40100000000000002,"5min":0.377,"15min":0.376},"max":{"1min":6.7560000000000002,"5min":6.7560000000000002,"15min":6.7560000000000002},"last":0.60799999999999998}]},{"osd":6,"last update":"Mon Oct 16 13:28:07 2023","interfaces":[{"interface":"back","average":{"1min":1.0940000000000001,"5min":0.72599999999999998,"15min":0.66800000000000004},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.38500000000000001},"max":{"1min":8.8279999999999994,"5min":8.8279999999999994,"15min":8.8279999999999994},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.67400000000000004,"15min":0.65700000000000003},"min":{"1min":0.41699999999999998,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":3.2599999999999998,"5min":3.2599999999999998,"15min":3.2599999999999998},"last":0.84199999999999997}]}]},{"osd":6,"up_from":13,"seq":55834575771,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41812,"kb_used_data":2312,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330028,"statfs":{"total":96636764160,"available":96593948672,"internally_reserved":0,"allocated":2367488,"data_stored":1206111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:28:07 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.67000000000000004,"15min":0.67300000000000004},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.29899999999999999},"max":{"1min":0.80600000000000005,"5min":0.90800000000000003,"15min":0.91400000000000003},"last":0.89100000000000001},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.65800000000000003,"15min":0.67700000000000005},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.34699999999999998},"max":{"1min":0.67100000000000004,"5min":0.85999999999999999,"15min":0.93100000000000005},"last":0.80000000000000004}]},{"osd":1,"last update":"Mon Oct 16 13:28:07 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.61799999999999999,"15min":0.624},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.72299999999999998,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.61599999999999999,"15min":0.63200000000000001},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.34899999999999998},"max":{"1min":0.72999999999999998,"5min":0.88200000000000001,"15min":0.95599999999999996},"last":0.70899999999999996}]},{"osd":2,"last update":"Mon Oct 16 13:28:07 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.63200000000000001,"15min":0.63500000000000001},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.33500000000000002},"max":{"1min":0.69399999999999995,"5min":0.93899999999999995,"15min":0.94199999999999995},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.61099999999999999,"15min":0.63200000000000001},"min":{"1min":0.374,"5min":0.374,"15min":0.33100000000000002},"max":{"1min":0.74199999999999999,"5min":0.83099999999999996,"15min":0.93100000000000005},"last":0.85999999999999999}]},{"osd":3,"last update":"Mon Oct 16 13:28:07 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.67000000000000004,"15min":0.68100000000000005},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.374},"max":{"1min":0.78000000000000003,"5min":1.288,"15min":1.288},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.73099999999999998,"15min":0.72399999999999998},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.32300000000000001},"max":{"1min":0.76300000000000001,"5min":1.351,"15min":1.351},"last":0.84199999999999997}]},{"osd":4,"last update":"Mon Oct 16 13:28:07 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.51700000000000002},"min":{"1min":0.378,"5min":0.34100000000000003,"15min":0.32000000000000001},"max":{"1min":0.69199999999999995,"5min":0.85599999999999998,"15min":0.86299999999999999},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61699999999999999,"15min":0.60999999999999999},"min":{"1min":0.46100000000000002,"5min":0.35199999999999998,"15min":0.32000000000000001},"max":{"1min":0.79600000000000004,"5min":0.89600000000000002,"15min":0.90100000000000002},"last":0.65600000000000003}]},{"osd":5,"last update":"Mon Oct 16 13:28:07 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58199999999999996,"15min":0.57299999999999995},"min":{"1min":0.40999999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.72999999999999998,"5min":0.85899999999999999,"15min":0.91800000000000004},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.65200000000000002,"15min":0.63400000000000001},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.32700000000000001},"max":{"1min":0.85099999999999998,"5min":0.91000000000000003,"15min":0.93799999999999994},"last":0.78200000000000003}]},{"osd":7,"last update":"Mon Oct 16 13:28:07 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.58799999999999997,"15min":0.57499999999999996},"min":{"1min":0.40600000000000003,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.72899999999999998,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.63700000000000001,"15min":0.628},"min":{"1min":0.45900000000000002,"5min":0.39300000000000002,"15min":0.378},"max":{"1min":0.75600000000000001,"5min":0.92000000000000004,"15min":0.98899999999999999},"last":0.68200000000000005}]}]},{"osd":1,"up_from":13,"seq":55834575771,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41824,"kb_used_data":2324,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330016,"statfs":{"total":96636764160,"available":96593936384,"internally_reserved":0,"allocated":2379776,"data_stored":1207723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":40434121},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:28:09 2023","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.40999999999999998,"15min":0.41299999999999998},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.245},"max":{"1min":0.59799999999999998,"5min":0.60399999999999998,"15min":0.76600000000000001},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.497,"15min":0.49399999999999999},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":1.46,"5min":1.46,"15min":1.46},"last":0.36299999999999999}]},{"osd":2,"last update":"Mon Oct 16 13:28:09 2023","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.45200000000000001,"15min":0.45200000000000001},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.48799999999999999,"15min":0.49299999999999999},"min":{"1min":0.34599999999999997,"5min":0.32300000000000001,"15min":0.29299999999999998},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.39300000000000002}]},{"osd":3,"last update":"Mon Oct 16 13:28:09 2023","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.435,"15min":0.42799999999999999},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29199999999999998},"max":{"1min":1.5009999999999999,"5min":1.5009999999999999,"15min":1.5009999999999999},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.48899999999999999,"15min":0.48599999999999999},"min":{"1min":0.34200000000000003,"5min":0.311,"15min":0.31},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.377}]},{"osd":4,"last update":"Mon Oct 16 13:28:09 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.66000000000000003,"15min":0.66500000000000004},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.542,"5min":1.542,"15min":1.542},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.68400000000000005,"15min":0.68100000000000005},"min":{"1min":0.435,"5min":0.435,"15min":0.42099999999999999},"max":{"1min":0.95099999999999996,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.76400000000000001}]},{"osd":5,"last update":"Mon Oct 16 13:28:09 2023","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.73199999999999998,"15min":0.72299999999999998},"min":{"1min":0.502,"5min":0.502,"15min":0.44400000000000001},"max":{"1min":3.2610000000000001,"5min":3.2610000000000001,"15min":3.2610000000000001},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.72799999999999998,"15min":0.73099999999999998},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.579,"5min":1.579,"15min":1.579},"last":0.77900000000000003}]},{"osd":6,"last update":"Mon Oct 16 13:28:09 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.68700000000000006,"15min":0.68799999999999994},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.45000000000000001},"max":{"1min":0.96499999999999997,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.72599999999999998,"15min":0.71899999999999997},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.432},"max":{"1min":3.3130000000000002,"5min":3.3130000000000002,"15min":3.3130000000000002},"last":0.71399999999999997}]},{"osd":7,"last update":"Mon Oct 16 13:28:09 2023","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.73999999999999999,"15min":0.73899999999999999},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.44},"max":{"1min":1.9430000000000001,"5min":1.9430000000000001,"15min":1.9430000000000001},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.77200000000000002,"15min":0.75900000000000001},"min":{"1min":0.505,"5min":0.505,"15min":0.498},"max":{"1min":1.7549999999999999,"5min":1.7549999999999999,"15min":1.7549999999999999},"last":0.72599999999999998}]}]},{"osd":0,"up_from":13,"seq":55834575772,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42408,"kb_used_data":2900,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329432,"statfs":{"total":96636764160,"available":96593338368,"internally_reserved":0,"allocated":2969600,"data_stored":1796583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 13:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.40200000000000002,"15min":0.41099999999999998},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.317},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.438,"15min":0.44700000000000001},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.52600000000000002,"5min":0.745,"15min":0.75800000000000001},"last":0.38}]},{"osd":2,"last update":"Mon Oct 16 13:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.44700000000000001,"15min":0.45600000000000002},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.52200000000000002,"5min":0.70799999999999996,"15min":0.73399999999999999},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.46999999999999997,"15min":0.48199999999999998},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.75700000000000001},"last":0.42799999999999999}]},{"osd":3,"last update":"Mon Oct 16 13:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.443,"15min":0.44900000000000001},"min":{"1min":0.28000000000000003,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.50900000000000001,"15min":0.52500000000000002},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.82099999999999995},"last":0.40600000000000003}]},{"osd":4,"last update":"Mon Oct 16 13:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.69499999999999995,"15min":0.68600000000000005},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.40200000000000002},"max":{"1min":0.79900000000000004,"5min":1.0229999999999999,"15min":1.0369999999999999},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.68000000000000005,"15min":0.68500000000000005},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.39000000000000001},"max":{"1min":0.77100000000000002,"5min":1.0429999999999999,"15min":1.0669999999999999},"last":0.59899999999999998}]},{"osd":5,"last update":"Mon Oct 16 13:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.67800000000000005,"15min":0.68300000000000005},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.84499999999999997,"5min":1.0049999999999999,"15min":1.034},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.71399999999999997,"15min":0.71599999999999997},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.46700000000000003},"max":{"1min":0.84399999999999997,"5min":1.0129999999999999,"15min":1.109},"last":0.58399999999999996}]},{"osd":6,"last update":"Mon Oct 16 13:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.72099999999999997,"15min":0.72899999999999998},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.88900000000000001,"5min":1.0649999999999999,"15min":1.109},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.62,"5min":0.71299999999999997,"15min":0.72799999999999998},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.44500000000000001},"max":{"1min":0.83099999999999996,"5min":1.03,"15min":1.0720000000000001},"last":0.72999999999999998}]},{"osd":7,"last update":"Mon Oct 16 13:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.73299999999999998,"15min":0.73699999999999999},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.93000000000000005,"5min":0.99199999999999999,"15min":1.0880000000000001},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.72599999999999998,"15min":0.72999999999999998},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.92000000000000004,"5min":0.996,"15min":1.0920000000000001},"last":0.77400000000000002}]}]},{"osd":2,"up_from":13,"seq":55834575772,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51544,"kb_used_data":2312,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320296,"statfs":{"total":96636764160,"available":96583983104,"internally_reserved":0,"allocated":2367488,"data_stored":1206111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.39800000000000002,"15min":0.41199999999999998},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.56200000000000006,"5min":0.65600000000000003,"15min":0.77800000000000002},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.5,"5min":0.496,"15min":0.48899999999999999},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.339,"5min":1.339,"15min":1.339},"last":0.46700000000000003}]},{"osd":1,"last update":"Mon Oct 16 13:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.52500000000000002,"15min":0.52000000000000002},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32400000000000001},"max":{"1min":1.3819999999999999,"5min":1.3819999999999999,"15min":1.3819999999999999},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.45800000000000002,"15min":0.47099999999999997},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.60799999999999998,"5min":0.78200000000000003,"15min":0.79000000000000004},"last":0.379}]},{"osd":3,"last update":"Mon Oct 16 13:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.42999999999999999,"15min":0.44900000000000001},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.65700000000000003,"5min":0.66200000000000003,"15min":0.81200000000000006},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55700000000000005,"15min":0.56000000000000005},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.4179999999999999,"5min":1.4179999999999999,"15min":1.4179999999999999},"last":0.503}]},{"osd":4,"last update":"Mon Oct 16 13:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.70099999999999996,"15min":0.69999999999999996},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.92900000000000005,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.70199999999999996,"15min":0.70799999999999996},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.95099999999999996,"5min":0.996,"15min":1.0109999999999999},"last":0.70899999999999996}]},{"osd":5,"last update":"Mon Oct 16 13:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.71099999999999997,"15min":0.70599999999999996},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.45,"5min":1.45,"15min":1.45},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.66700000000000004,"15min":0.68100000000000005},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.80700000000000005,"5min":0.96899999999999997,"15min":1.0740000000000001},"last":0.81799999999999995}]},{"osd":6,"last update":"Mon Oct 16 13:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.65100000000000002,"15min":0.66200000000000003},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.90000000000000002,"5min":1.046,"15min":1.345},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.73599999999999999,"15min":0.73299999999999998},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.4830000000000001,"5min":1.4830000000000001,"15min":1.4830000000000001},"last":0.64300000000000002}]},{"osd":7,"last update":"Mon Oct 16 13:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.70699999999999996,"15min":0.71999999999999997},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.81899999999999995,"5min":1.0620000000000001,"15min":1.115},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.74299999999999999,"15min":0.73799999999999999},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.5149999999999999,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":0.60199999999999998}]}]},{"osd":3,"up_from":13,"seq":55834575771,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47400,"kb_used_data":2332,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324440,"statfs":{"total":96636764160,"available":96588226560,"internally_reserved":0,"allocated":2387968,"data_stored":1208327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":46135752},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.40899999999999997,"15min":0.40600000000000003},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.60199999999999998,"5min":0.629,"15min":0.73099999999999998},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.377,"5min":0.42699999999999999,"15min":0.433},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.628,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.40899999999999997}]},{"osd":1,"last update":"Mon Oct 16 13:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.44500000000000001,"15min":0.434},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.70699999999999996,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.437,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.29299999999999998},"max":{"1min":0.66300000000000003,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.43099999999999999}]},{"osd":2,"last update":"Mon Oct 16 13:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.48999999999999999,"15min":0.495},"min":{"1min":0.31900000000000001,"5min":0.311,"15min":0.311},"max":{"1min":0.56899999999999995,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.52100000000000002,"15min":0.52600000000000002},"min":{"1min":0.32600000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.89800000000000002},"last":0.48699999999999999}]},{"osd":4,"last update":"Mon Oct 16 13:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.65300000000000002,"15min":0.67100000000000004},"min":{"1min":0.47599999999999998,"5min":0.46600000000000003,"15min":0.41799999999999998},"max":{"1min":0.82299999999999995,"5min":1.0640000000000001,"15min":1.1060000000000001},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.67900000000000005,"15min":0.68899999999999995},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.78100000000000003,"5min":1.008,"15min":1.0189999999999999},"last":0.84299999999999997}]},{"osd":5,"last update":"Mon Oct 16 13:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.69099999999999995,"15min":0.70999999999999996},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89000000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":1.079},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.69099999999999995,"15min":0.69799999999999995},"min":{"1min":0.42699999999999999,"5min":0.42499999999999999,"15min":0.40200000000000002},"max":{"1min":0.90500000000000003,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.93100000000000005}]},{"osd":6,"last update":"Mon Oct 16 13:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.70399999999999996,"15min":0.70699999999999996},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.42799999999999999},"max":{"1min":0.92900000000000005,"5min":1.0469999999999999,"15min":1.0800000000000001},"last":0.89700000000000002},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.72999999999999998,"15min":0.74099999999999999},"min":{"1min":0.502,"5min":0.502,"15min":0.46999999999999997},"max":{"1min":0.94299999999999995,"5min":1.073,"15min":1.1220000000000001},"last":1.119}]},{"osd":7,"last update":"Mon Oct 16 13:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.71699999999999997,"15min":0.72499999999999998},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.86299999999999999,"5min":1.099,"15min":1.099},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.754,"15min":0.753},"min":{"1min":0.49099999999999999,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.998,"5min":1.0609999999999999,"15min":1.131},"last":0.95799999999999996}]}]},{"osd":4,"up_from":13,"seq":55834575770,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29356,"kb_used_data":2328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342484,"statfs":{"total":96636764160,"available":96606703616,"internally_reserved":0,"allocated":2383872,"data_stored":1206875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:28:01 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.621,"15min":0.62},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.628,"15min":0.63500000000000001},"min":{"1min":0.40000000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.76400000000000001,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.53800000000000003}]},{"osd":1,"last update":"Mon Oct 16 13:28:01 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.63700000000000001,"15min":0.64500000000000002},"min":{"1min":0.39600000000000002,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.86699999999999999,"5min":0.878,"15min":0.93799999999999994},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.63500000000000001,"15min":0.63700000000000001},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.85099999999999998,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.40400000000000003}]},{"osd":2,"last update":"Mon Oct 16 13:28:01 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.63600000000000001,"15min":0.65100000000000002},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.83099999999999996,"5min":0.90000000000000002,"15min":1.091},"last":0.5},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.66300000000000003,"15min":0.65700000000000003},"min":{"1min":0.46500000000000002,"5min":0.38,"15min":0.38},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.51500000000000001}]},{"osd":3,"last update":"Mon Oct 16 13:28:01 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.63400000000000001,"15min":0.64600000000000002},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.82799999999999996,"5min":0.89200000000000002,"15min":0.95399999999999996},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.68000000000000005,"15min":0.67700000000000005},"min":{"1min":0.48299999999999998,"5min":0.41599999999999998,"15min":0.41499999999999998},"max":{"1min":0.80900000000000005,"5min":0.86799999999999999,"15min":0.98899999999999999},"last":0.59299999999999997}]},{"osd":5,"last update":"Mon Oct 16 13:28:01 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.51600000000000001,"15min":0.501},"min":{"1min":0.39600000000000002,"5min":0.33600000000000002,"15min":0.309},"max":{"1min":0.89200000000000002,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54300000000000004,"15min":0.54000000000000004},"min":{"1min":0.35799999999999998,"5min":0.31,"15min":0.31},"max":{"1min":0.755,"5min":0.93000000000000005,"15min":0.97099999999999997},"last":0.57999999999999996}]},{"osd":6,"last update":"Mon Oct 16 13:28:01 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55100000000000005,"15min":0.56200000000000006},"min":{"1min":0.377,"5min":0.34499999999999997,"15min":0.32800000000000001},"max":{"1min":0.78900000000000003,"5min":0.90900000000000003,"15min":1.046},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.45300000000000001,"5min":0.433,"15min":0.34999999999999998},"max":{"1min":0.96999999999999997,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.60599999999999998}]},{"osd":7,"last update":"Mon Oct 16 13:28:01 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.41999999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.88800000000000001,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.65300000000000002,"15min":0.65000000000000002},"min":{"1min":0.441,"5min":0.39500000000000002,"15min":0.38400000000000001},"max":{"1min":0.92400000000000004,"5min":0.94899999999999995,"15min":1.1359999999999999},"last":0.63}]}]},{"osd":5,"up_from":13,"seq":55834575771,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51548,"kb_used_data":2320,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320292,"statfs":{"total":96636764160,"available":96583979008,"internally_reserved":0,"allocated":2375680,"data_stored":1206235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:27:50 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.624,"15min":0.61799999999999999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.76600000000000001,"5min":0.87,"15min":0.87},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.63,"15min":0.64400000000000002},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.79900000000000004,"5min":0.89100000000000001,"15min":0.94899999999999995},"last":0.66100000000000003}]},{"osd":1,"last update":"Mon Oct 16 13:27:50 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.64400000000000002,"15min":0.64800000000000002},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.72899999999999998,"5min":0.872,"15min":0.93899999999999995},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.64000000000000001,"15min":0.65100000000000002},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.77400000000000002,"5min":0.90200000000000002,"15min":0.93300000000000005},"last":0.69099999999999995}]},{"osd":2,"last update":"Mon Oct 16 13:27:50 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.66000000000000003,"15min":0.65200000000000002},"min":{"1min":0.45300000000000001,"5min":0.441,"15min":0.39400000000000002},"max":{"1min":0.79200000000000004,"5min":1.091,"15min":1.091},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.624,"5min":0.64000000000000001,"15min":0.64600000000000002},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.35799999999999998},"max":{"1min":0.80900000000000005,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.73899999999999999}]},{"osd":3,"last update":"Mon Oct 16 13:27:50 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.0069999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.68899999999999995,"15min":0.68500000000000005},"min":{"1min":0.44900000000000001,"5min":0.42599999999999999,"15min":0.41099999999999998},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":0.873}]},{"osd":4,"last update":"Mon Oct 16 13:27:50 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.52700000000000002,"15min":0.52200000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":1.028},{"interface":"front","average":{"1min":0.621,"5min":0.60199999999999998,"15min":0.58599999999999997},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":1.0740000000000001}]},{"osd":6,"last update":"Mon Oct 16 13:27:50 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.59499999999999997,"15min":0.57299999999999995},"min":{"1min":0.443,"5min":0.39800000000000002,"15min":0.34399999999999997},"max":{"1min":1.095,"5min":1.095,"15min":1.095},"last":0.98199999999999998},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.63,"15min":0.61699999999999999},"min":{"1min":0.371,"5min":0.371,"15min":0.312},"max":{"1min":0.90300000000000002,"5min":1.006,"15min":1.006},"last":0.90900000000000003}]},{"osd":7,"last update":"Mon Oct 16 13:27:50 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.67600000000000005,"15min":0.64800000000000002},"min":{"1min":0.47099999999999997,"5min":0.40600000000000003,"15min":0.34100000000000003},"max":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.95699999999999996},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.69599999999999995,"15min":0.67500000000000004},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.39700000000000002},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":1.042}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T13:28:24.671 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-16T13:28:24.672 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-16T13:28:24.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.835+0000 7fe7ddfa1700 1 Processor -- start 2023-10-16T13:28:24.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.835+0000 7fe7ddfa1700 1 -- start start 2023-10-16T13:28:24.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.836+0000 7fe7ddfa1700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fe7d815ec80 0x7fe7d80cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:24.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.836+0000 7fe7ddfa1700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7d80b2440 con 0x7fe7d815f060 2023-10-16T13:28:24.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.836+0000 7fe7ddfa1700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7d80d00e0 con 0x7fe7d80d2110 2023-10-16T13:28:24.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.836+0000 7fe7ddfa1700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7d80b2c50 con 0x7fe7d815ec80 2023-10-16T13:28:24.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.836+0000 7fe7d6ffd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe7d815f060 0x7fe7d80b4440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:39202/0 (socket says 172.21.15.16:39202) 2023-10-16T13:28:24.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.836+0000 7fe7d6ffd700 1 -- 172.21.15.16:0/1530352047 learned_addr learned my addr 172.21.15.16:0/1530352047 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:28:24.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.836+0000 7fe7d77fe700 1 --2- 172.21.15.16:0/1530352047 >> v2:172.21.15.49:3300/0 conn(0x7fe7d815ec80 0x7fe7d80cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:24.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.836+0000 7fe7d67fc700 1 -- 172.21.15.16:0/1530352047 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1082384592 0 0) 0x7fe7d80b2440 con 0x7fe7d815f060 2023-10-16T13:28:24.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.836+0000 7fe7d67fc700 1 -- 172.21.15.16:0/1530352047 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7c40035b0 con 0x7fe7d815f060 2023-10-16T13:28:24.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.837+0000 7fe7d67fc700 1 -- 172.21.15.16:0/1530352047 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 90822363 0 0) 0x7fe7c40035b0 con 0x7fe7d815f060 2023-10-16T13:28:24.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.837+0000 7fe7d67fc700 1 -- 172.21.15.16:0/1530352047 >> v2:172.21.15.49:3300/0 conn(0x7fe7d815ec80 msgr2=0x7fe7d80cdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:28:24.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.837+0000 7fe7d67fc700 1 --2- 172.21.15.16:0/1530352047 >> v2:172.21.15.49:3300/0 conn(0x7fe7d815ec80 0x7fe7d80cdec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:24.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.837+0000 7fe7d67fc700 1 -- 172.21.15.16:0/1530352047 >> v1:172.21.15.49:6789/0 conn(0x7fe7d80d2110 legacy=0x7fe7d80b4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:24.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.837+0000 7fe7d67fc700 1 -- 172.21.15.16:0/1530352047 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7d80cef60 con 0x7fe7d815f060 2023-10-16T13:28:24.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.837+0000 7fe7d67fc700 1 -- 172.21.15.16:0/1530352047 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7fe7c8001670 con 0x7fe7d815f060 2023-10-16T13:28:24.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.837+0000 7fe7d67fc700 1 -- 172.21.15.16:0/1530352047 >> v1:172.21.15.16:6789/0 conn(0x7fe7d815f060 legacy=0x7fe7d80b4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:24.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.837+0000 7fe7d67fc700 1 --2- 172.21.15.16:0/1530352047 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7c4003a00 0x7fe7c4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:24.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.837+0000 7fe7d67fc700 1 --2- 172.21.15.16:0/1530352047 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7c4004850 0x7fe7c4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:24.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.837+0000 7fe7d67fc700 1 --2- 172.21.15.16:0/1530352047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7c4005210 0x7fe7c400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:24.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.837+0000 7fe7d67fc700 1 -- 172.21.15.16:0/1530352047 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d80b2c50 con 0x7fe7c4005210 2023-10-16T13:28:24.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.837+0000 7fe7d67fc700 1 -- 172.21.15.16:0/1530352047 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7c400c0b0 con 0x7fe7c4003a00 2023-10-16T13:28:24.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.837+0000 7fe7d67fc700 1 -- 172.21.15.16:0/1530352047 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7c400c270 con 0x7fe7c4004850 2023-10-16T13:28:24.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.837+0000 7fe7d7fff700 1 --2- 172.21.15.16:0/1530352047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7c4005210 0x7fe7c400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:24.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.837+0000 7fe7d77fe700 1 --2- 172.21.15.16:0/1530352047 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7c4003a00 0x7fe7c4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:24.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.837+0000 7fe7d6ffd700 1 --2- 172.21.15.16:0/1530352047 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7c4004850 0x7fe7c4004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:24.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.837+0000 7fe7d7fff700 1 -- 172.21.15.16:0/1530352047 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7c4004850 msgr2=0x7fe7c4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:28:24.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.837+0000 7fe7d7fff700 1 --2- 172.21.15.16:0/1530352047 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7c4004850 0x7fe7c4004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:24.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.837+0000 7fe7d7fff700 1 -- 172.21.15.16:0/1530352047 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7c4003a00 msgr2=0x7fe7c4003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:28:24.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.837+0000 7fe7d7fff700 1 --2- 172.21.15.16:0/1530352047 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7c4003a00 0x7fe7c4003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:24.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.837+0000 7fe7d7fff700 1 -- 172.21.15.16:0/1530352047 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7c40035b0 con 0x7fe7c4005210 2023-10-16T13:28:24.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.837+0000 7fe7d7fff700 1 -- 172.21.15.16:0/1530352047 reap_dead start 2023-10-16T13:28:24.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.837+0000 7fe7d7fff700 1 --2- 172.21.15.16:0/1530352047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7c4005210 0x7fe7c400ba10 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 crypto rx=0x7fe7d000c190 tx=0x7fe7c4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6110557a6a77cf12 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:24.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.838+0000 7fe7d67fc700 1 -- 172.21.15.16:0/1530352047 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7d0001db0 con 0x7fe7c4005210 2023-10-16T13:28:24.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.838+0000 7fe7d67fc700 1 -- 172.21.15.16:0/1530352047 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7d000ccf0 con 0x7fe7c4005210 2023-10-16T13:28:24.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.838+0000 7fe7d67fc700 1 -- 172.21.15.16:0/1530352047 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7d0003c40 con 0x7fe7c4005210 2023-10-16T13:28:24.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.838+0000 7fe7ddfa1700 1 -- 172.21.15.16:0/1530352047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7c4005210 msgr2=0x7fe7c400ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:24.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.838+0000 7fe7ddfa1700 1 --2- 172.21.15.16:0/1530352047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7c4005210 0x7fe7c400ba10 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 crypto rx=0x7fe7d000c190 tx=0x7fe7c4005880 comp rx=0 tx=0).stop 2023-10-16T13:28:24.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.838+0000 7fe7ddfa1700 1 -- 172.21.15.16:0/1530352047 shutdown_connections 2023-10-16T13:28:24.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.838+0000 7fe7ddfa1700 1 --2- 172.21.15.16:0/1530352047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7c4005210 0x7fe7c400ba10 unknown :-1 s=CLOSED pgs=2102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:24.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.838+0000 7fe7ddfa1700 1 -- 172.21.15.16:0/1530352047 >> 172.21.15.16:0/1530352047 conn(0x7fe7d80d15f0 msgr2=0x7fe7d80d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:24.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.838+0000 7fe7ddfa1700 1 -- 172.21.15.16:0/1530352047 shutdown_connections 2023-10-16T13:28:24.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.838+0000 7fe7ddfa1700 1 -- 172.21.15.16:0/1530352047 wait complete. 2023-10-16T13:28:24.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.840+0000 7fe7ddfa1700 1 Processor -- start 2023-10-16T13:28:24.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.840+0000 7fe7ddfa1700 1 -- start start 2023-10-16T13:28:24.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.841+0000 7fe7ddfa1700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7c4005210 0x7fe7d81f0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:24.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.841+0000 7fe7ddfa1700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7d81f21b0 0x7fe7d81f3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:24.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.841+0000 7fe7ddfa1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7d80b41c0 0x7fe7d80b4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:24.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.841+0000 7fe7ddfa1700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d80b4b80 con 0x7fe7d80b41c0 2023-10-16T13:28:24.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.841+0000 7fe7ddfa1700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d81f18b0 con 0x7fe7d81f21b0 2023-10-16T13:28:24.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.841+0000 7fe7ddfa1700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d81f1b90 con 0x7fe7c4005210 2023-10-16T13:28:24.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.841+0000 7fe7d7fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7d80b41c0 0x7fe7d80b4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:24.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.841+0000 7fe7d7fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7d80b41c0 0x7fe7d80b4640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49942/0 (socket says 172.21.15.16:49942) 2023-10-16T13:28:24.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.841+0000 7fe7d7fff700 1 -- 172.21.15.16:0/24978894 learned_addr learned my addr 172.21.15.16:0/24978894 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:24.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.841+0000 7fe7d6ffd700 1 --2- 172.21.15.16:0/24978894 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7d81f21b0 0x7fe7d81f3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:24.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.841+0000 7fe7d77fe700 1 --2- 172.21.15.16:0/24978894 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7c4005210 0x7fe7d81f0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:24.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.841+0000 7fe7d7fff700 1 -- 172.21.15.16:0/24978894 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7c4005210 msgr2=0x7fe7d81f0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:24.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.841+0000 7fe7d7fff700 1 --2- 172.21.15.16:0/24978894 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7c4005210 0x7fe7d81f0fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:24.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.841+0000 7fe7d7fff700 1 -- 172.21.15.16:0/24978894 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7d81f21b0 msgr2=0x7fe7d81f3cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:24.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.841+0000 7fe7d7fff700 1 --2- 172.21.15.16:0/24978894 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7d81f21b0 0x7fe7d81f3cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:24.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.841+0000 7fe7d7fff700 1 -- 172.21.15.16:0/24978894 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7d81f1e70 con 0x7fe7d80b41c0 2023-10-16T13:28:24.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.842+0000 7fe7d7fff700 1 --2- 172.21.15.16:0/24978894 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7d80b41c0 0x7fe7d80b4640 secure :-1 s=READY pgs=2103 cs=0 l=1 rev1=1 crypto rx=0x7fe7d0005040 tx=0x7fe7d00014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:24.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.842+0000 7fe7cffff700 1 -- 172.21.15.16:0/24978894 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7d00052d0 con 0x7fe7d80b41c0 2023-10-16T13:28:24.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.842+0000 7fe7cffff700 1 -- 172.21.15.16:0/24978894 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7d0005430 con 0x7fe7d80b41c0 2023-10-16T13:28:24.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.842+0000 7fe7ddfa1700 1 -- 172.21.15.16:0/24978894 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7d81563c0 con 0x7fe7d80b41c0 2023-10-16T13:28:24.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.842+0000 7fe7cffff700 1 -- 172.21.15.16:0/24978894 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7d00057a0 con 0x7fe7d80b41c0 2023-10-16T13:28:24.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.842+0000 7fe7ddfa1700 1 -- 172.21.15.16:0/24978894 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7d81f1500 con 0x7fe7d80b41c0 2023-10-16T13:28:24.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.843+0000 7fe7ddfa1700 1 -- 172.21.15.16:0/24978894 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7d81f2000 con 0x7fe7d80b41c0 2023-10-16T13:28:24.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.843+0000 7fe7cffff700 1 -- 172.21.15.16:0/24978894 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 107) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe7d001d480 con 0x7fe7d80b41c0 2023-10-16T13:28:24.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.844+0000 7fe7cffff700 1 --2- 172.21.15.16:0/24978894 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7fe7b806b020 0x7fe7b806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:24.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.844+0000 7fe7d77fe700 1 --2- 172.21.15.16:0/24978894 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7fe7b806b020 0x7fe7b806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:24.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.844+0000 7fe7cffff700 1 -- 172.21.15.16:0/24978894 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 6129+0+0 (secure 0 0 0) 0x7fe7d008a360 con 0x7fe7d80b41c0 2023-10-16T13:28:24.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.844+0000 7fe7d77fe700 1 --2- 172.21.15.16:0/24978894 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7fe7b806b020 0x7fe7b806d4e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe7c0009be0 tx=0x7fe7c0009210 comp rx=0 tx=0).ready entity=mgr.10047 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:24.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:24.848+0000 7fe7cffff700 1 -- 172.21.15.16:0/24978894 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe7d0055350 con 0x7fe7d80b41c0 2023-10-16T13:28:25.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.132+0000 7fe7ddfa1700 1 -- 172.21.15.16:0/24978894 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fe7d8156200 con 0x7fe7d80b41c0 2023-10-16T13:28:25.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.134+0000 7fe7cffff700 1 -- 172.21.15.16:0/24978894 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v48) v1 ==== 190+0+147 (secure 0 0 0) 0x7fe7d0058970 con 0x7fe7d80b41c0 2023-10-16T13:28:25.137 INFO:teuthology.orchestra.run.smithi016.stdout:[client.0] 2023-10-16T13:28:25.137 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAPKC1l2l6PFxAA9QyXtBAHdGAXpwH0JfknuQ== 2023-10-16T13:28:25.137 INFO:teuthology.orchestra.run.smithi016.stdout: caps mds = "allow" 2023-10-16T13:28:25.137 INFO:teuthology.orchestra.run.smithi016.stdout: caps mon = "allow r" 2023-10-16T13:28:25.138 INFO:teuthology.orchestra.run.smithi016.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-16T13:28:25.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.137+0000 7fe7ddfa1700 1 -- 172.21.15.16:0/24978894 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7fe7b806b020 msgr2=0x7fe7b806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:25.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.137+0000 7fe7ddfa1700 1 --2- 172.21.15.16:0/24978894 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7fe7b806b020 0x7fe7b806d4e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe7c0009be0 tx=0x7fe7c0009210 comp rx=0 tx=0).stop 2023-10-16T13:28:25.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.137+0000 7fe7ddfa1700 1 -- 172.21.15.16:0/24978894 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7d80b41c0 msgr2=0x7fe7d80b4640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:25.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.137+0000 7fe7ddfa1700 1 --2- 172.21.15.16:0/24978894 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7d80b41c0 0x7fe7d80b4640 secure :-1 s=READY pgs=2103 cs=0 l=1 rev1=1 crypto rx=0x7fe7d0005040 tx=0x7fe7d00014a0 comp rx=0 tx=0).stop 2023-10-16T13:28:25.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.137+0000 7fe7ddfa1700 1 -- 172.21.15.16:0/24978894 shutdown_connections 2023-10-16T13:28:25.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.137+0000 7fe7ddfa1700 1 --2- 172.21.15.16:0/24978894 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7d80b41c0 0x7fe7d80b4640 unknown :-1 s=CLOSED pgs=2103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:25.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.137+0000 7fe7ddfa1700 1 --2- 172.21.15.16:0/24978894 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7c4005210 0x7fe7d81f0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:25.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.137+0000 7fe7ddfa1700 1 --2- 172.21.15.16:0/24978894 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7fe7b806b020 0x7fe7b806d4e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:25.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.137+0000 7fe7ddfa1700 1 --2- 172.21.15.16:0/24978894 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7d81f21b0 0x7fe7d81f3cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:25.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.137+0000 7fe7ddfa1700 1 -- 172.21.15.16:0/24978894 >> 172.21.15.16:0/24978894 conn(0x7fe7d80d15f0 msgr2=0x7fe7d806f510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:25.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.137+0000 7fe7ddfa1700 1 -- 172.21.15.16:0/24978894 shutdown_connections 2023-10-16T13:28:25.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.137+0000 7fe7ddfa1700 1 -- 172.21.15.16:0/24978894 wait complete. 2023-10-16T13:28:25.143 INFO:teuthology.orchestra.run.smithi016.stderr:updated caps for client.0 2023-10-16T13:28:25.152 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:28:25.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.274+0000 7f12a21a8700 1 Processor -- start 2023-10-16T13:28:25.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.274+0000 7f12a21a8700 1 -- start start 2023-10-16T13:28:25.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.275+0000 7f12a21a8700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f129c0d0ae0 0x7f129c0cef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:25.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.275+0000 7f12a21a8700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f129c072a50 con 0x7f129c0ce010 2023-10-16T13:28:25.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.275+0000 7f12a21a8700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f129c0cba30 con 0x7f129c15a5c0 2023-10-16T13:28:25.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.275+0000 7f12a21a8700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f129c0cf440 con 0x7f129c0d0ae0 2023-10-16T13:28:25.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.275+0000 7f129bfff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f129c0ce010 0x7f129c0cc1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:39224/0 (socket says 172.21.15.16:39224) 2023-10-16T13:28:25.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.275+0000 7f129bfff700 1 -- 172.21.15.16:0/1867708366 learned_addr learned my addr 172.21.15.16:0/1867708366 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:28:25.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.276+0000 7f129affd700 1 --2- 172.21.15.16:0/1867708366 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f129c0d0ae0 0x7f129c0cef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:25.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.276+0000 7f129a7fc700 1 -- 172.21.15.16:0/1867708366 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1419295299 0 0) 0x7f129c072a50 con 0x7f129c0ce010 2023-10-16T13:28:25.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.276+0000 7f129a7fc700 1 -- 172.21.15.16:0/1867708366 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12840035b0 con 0x7f129c0ce010 2023-10-16T13:28:25.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.276+0000 7f129a7fc700 1 -- 172.21.15.16:0/1867708366 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4023819730 0 0) 0x7f12840035b0 con 0x7f129c0ce010 2023-10-16T13:28:25.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.276+0000 7f129a7fc700 1 -- 172.21.15.16:0/1867708366 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f129c0d0ae0 msgr2=0x7f129c0cef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:28:25.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.276+0000 7f129a7fc700 1 --2- 172.21.15.16:0/1867708366 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f129c0d0ae0 0x7f129c0cef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:25.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.276+0000 7f129a7fc700 1 -- 172.21.15.16:0/1867708366 >> v1:172.21.15.49:6789/0 conn(0x7f129c15a5c0 legacy=0x7f129c0d03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:25.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.276+0000 7f129a7fc700 1 -- 172.21.15.16:0/1867708366 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f129c0cf630 con 0x7f129c0ce010 2023-10-16T13:28:25.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.276+0000 7f129a7fc700 1 -- 172.21.15.16:0/1867708366 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f12940019f0 con 0x7f129c0ce010 2023-10-16T13:28:25.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.276+0000 7f129a7fc700 1 -- 172.21.15.16:0/1867708366 >> v1:172.21.15.16:6789/0 conn(0x7f129c0ce010 legacy=0x7f129c0cc1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:25.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.276+0000 7f129a7fc700 1 --2- 172.21.15.16:0/1867708366 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1284003a30 0x7f1284003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:25.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.276+0000 7f129a7fc700 1 --2- 172.21.15.16:0/1867708366 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f129c0d0ae0 0x7f1284004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:25.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.276+0000 7f129a7fc700 1 --2- 172.21.15.16:0/1867708366 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1284004d90 0x7f1284005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:25.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.276+0000 7f129a7fc700 1 -- 172.21.15.16:0/1867708366 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f128400bac0 con 0x7f1284003a30 2023-10-16T13:28:25.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.276+0000 7f129a7fc700 1 -- 172.21.15.16:0/1867708366 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f128400bc00 con 0x7f1284004d90 2023-10-16T13:28:25.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.276+0000 7f129a7fc700 1 -- 172.21.15.16:0/1867708366 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f128400dfd0 con 0x7f129c0d0ae0 2023-10-16T13:28:25.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.276+0000 7f129b7fe700 1 --2- 172.21.15.16:0/1867708366 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1284003a30 0x7f1284003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:25.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.276+0000 7f129affd700 1 --2- 172.21.15.16:0/1867708366 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f129c0d0ae0 0x7f1284004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:25.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.276+0000 7f129bfff700 1 --2- 172.21.15.16:0/1867708366 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1284004d90 0x7f1284005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:25.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.276+0000 7f129b7fe700 1 -- 172.21.15.16:0/1867708366 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f129c0d0ae0 msgr2=0x7f1284004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:28:25.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.276+0000 7f129b7fe700 1 --2- 172.21.15.16:0/1867708366 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f129c0d0ae0 0x7f1284004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:25.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.276+0000 7f129b7fe700 1 -- 172.21.15.16:0/1867708366 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1284004d90 msgr2=0x7f1284005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:28:25.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.276+0000 7f129b7fe700 1 --2- 172.21.15.16:0/1867708366 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1284004d90 0x7f1284005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:25.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.276+0000 7f129b7fe700 1 -- 172.21.15.16:0/1867708366 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12840035b0 con 0x7f1284003a30 2023-10-16T13:28:25.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.277+0000 7f129b7fe700 1 --2- 172.21.15.16:0/1867708366 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1284003a30 0x7f1284003e50 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 crypto rx=0x7f1290007fd0 tx=0x7f1284005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db888ea9567d6b2c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:25.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.277+0000 7f129a7fc700 1 -- 172.21.15.16:0/1867708366 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12900052d0 con 0x7f1284003a30 2023-10-16T13:28:25.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.277+0000 7f129a7fc700 1 -- 172.21.15.16:0/1867708366 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1290005470 con 0x7f1284003a30 2023-10-16T13:28:25.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.277+0000 7f129a7fc700 1 -- 172.21.15.16:0/1867708366 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12900038c0 con 0x7f1284003a30 2023-10-16T13:28:25.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.277+0000 7f12a21a8700 1 -- 172.21.15.16:0/1867708366 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1284003a30 msgr2=0x7f1284003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:25.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.277+0000 7f12a21a8700 1 --2- 172.21.15.16:0/1867708366 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1284003a30 0x7f1284003e50 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 crypto rx=0x7f1290007fd0 tx=0x7f1284005880 comp rx=0 tx=0).stop 2023-10-16T13:28:25.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.277+0000 7f129bfff700 1 -- 172.21.15.16:0/1867708366 reap_dead start 2023-10-16T13:28:25.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.277+0000 7f12a21a8700 1 -- 172.21.15.16:0/1867708366 shutdown_connections 2023-10-16T13:28:25.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.277+0000 7f12a21a8700 1 -- 172.21.15.16:0/1867708366 >> 172.21.15.16:0/1867708366 conn(0x7f129c0b1fd0 msgr2=0x7f129c0b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:25.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.277+0000 7f12a21a8700 1 -- 172.21.15.16:0/1867708366 shutdown_connections 2023-10-16T13:28:25.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.277+0000 7f12a21a8700 1 -- 172.21.15.16:0/1867708366 wait complete. 2023-10-16T13:28:25.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.279+0000 7f12a21a8700 1 Processor -- start 2023-10-16T13:28:25.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.279+0000 7f12a21a8700 1 -- start start 2023-10-16T13:28:25.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.279+0000 7f12a21a8700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1284003a30 0x7f129c0c4a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:25.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.280+0000 7f12a21a8700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f129c155f10 0x7f129c1566c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:25.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.280+0000 7f12a21a8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f129c156c00 0x7f129c1594d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:25.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.280+0000 7f12a21a8700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f129c0b3e00 con 0x7f129c156c00 2023-10-16T13:28:25.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.280+0000 7f12a21a8700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f129c0b3ad0 con 0x7f129c155f10 2023-10-16T13:28:25.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.280+0000 7f12a21a8700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f129c0af960 con 0x7f1284003a30 2023-10-16T13:28:25.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.280+0000 7f129bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f129c156c00 0x7f129c1594d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:25.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.280+0000 7f129bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f129c156c00 0x7f129c1594d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49962/0 (socket says 172.21.15.16:49962) 2023-10-16T13:28:25.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.280+0000 7f129bfff700 1 -- 172.21.15.16:0/2987091948 learned_addr learned my addr 172.21.15.16:0/2987091948 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:25.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.280+0000 7f129b7fe700 1 --2- 172.21.15.16:0/2987091948 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1284003a30 0x7f129c0c4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:25.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.280+0000 7f129bfff700 1 -- 172.21.15.16:0/2987091948 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1284003a30 msgr2=0x7f129c0c4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:25.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.280+0000 7f129bfff700 1 --2- 172.21.15.16:0/2987091948 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1284003a30 0x7f129c0c4a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:25.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.280+0000 7f129bfff700 1 -- 172.21.15.16:0/2987091948 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f129c155f10 msgr2=0x7f129c1566c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:25.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.280+0000 7f129bfff700 1 --2- 172.21.15.16:0/2987091948 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f129c155f10 0x7f129c1566c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:25.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.280+0000 7f129bfff700 1 -- 172.21.15.16:0/2987091948 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f129c0b3160 con 0x7f129c156c00 2023-10-16T13:28:25.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.280+0000 7f129bfff700 1 --2- 172.21.15.16:0/2987091948 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f129c156c00 0x7f129c1594d0 secure :-1 s=READY pgs=2106 cs=0 l=1 rev1=1 crypto rx=0x7f1294000f80 tx=0x7f129400a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:25.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.280+0000 7f1283fff700 1 -- 172.21.15.16:0/2987091948 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f129400ad00 con 0x7f129c156c00 2023-10-16T13:28:25.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.280+0000 7f1283fff700 1 -- 172.21.15.16:0/2987091948 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f129400ae80 con 0x7f129c156c00 2023-10-16T13:28:25.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.280+0000 7f12a21a8700 1 -- 172.21.15.16:0/2987091948 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f129c0b3440 con 0x7f129c156c00 2023-10-16T13:28:25.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.280+0000 7f1283fff700 1 -- 172.21.15.16:0/2987091948 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1294013630 con 0x7f129c156c00 2023-10-16T13:28:25.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.280+0000 7f12a21a8700 1 -- 172.21.15.16:0/2987091948 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f129c156390 con 0x7f129c156c00 2023-10-16T13:28:25.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.281+0000 7f12a21a8700 1 -- 172.21.15.16:0/2987091948 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f129c04ebb0 con 0x7f129c156c00 2023-10-16T13:28:25.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.282+0000 7f1283fff700 1 -- 172.21.15.16:0/2987091948 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 107) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1294013790 con 0x7f129c156c00 2023-10-16T13:28:25.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.282+0000 7f1283fff700 1 --2- 172.21.15.16:0/2987091948 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f128806b020 0x7f128806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:25.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.282+0000 7f1283fff700 1 -- 172.21.15.16:0/2987091948 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f129408a4e0 con 0x7f129c156c00 2023-10-16T13:28:25.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.282+0000 7f129b7fe700 1 --2- 172.21.15.16:0/2987091948 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f128806b020 0x7f128806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:25.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.283+0000 7f129b7fe700 1 --2- 172.21.15.16:0/2987091948 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f128806b020 0x7f128806d4e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1290007fd0 tx=0x7f1290009030 comp rx=0 tx=0).ready entity=mgr.10047 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:25.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.287+0000 7f1283fff700 1 -- 172.21.15.16:0/2987091948 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f12940552c0 con 0x7f129c156c00 2023-10-16T13:28:25.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.536+0000 7f12a21a8700 1 -- 172.21.15.16:0/2987091948 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f129c15dba0 con 0x7f129c156c00 2023-10-16T13:28:25.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.536+0000 7f1283fff700 1 -- 172.21.15.16:0/2987091948 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+4920 (secure 0 0 0) 0x7f12940588e0 con 0x7f129c156c00 2023-10-16T13:28:25.539 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:28:25.539 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":95,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8306,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2415675492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":2415675492},{"type":"v1","addr":"172.21.15.16:6837","nonce":2415675492}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":90},{"gid":8315,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/4031561298","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":4031561298},{"type":"v1","addr":"172.21.15.49:6834","nonce":4031561298}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":90},{"gid":8318,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/2853801536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":2853801536},{"type":"v1","addr":"172.21.15.49:6835","nonce":2853801536}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":90}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:28:19.523721+0000","modified":"2023-10-16T13:28:22.528826+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10296":{"gid":10296,"name":"c","rank":0,"incarnation":91,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6836/3244401013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":3244401013},{"type":"v1","addr":"172.21.15.16:6836","nonce":3244401013}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2023-10-16T13:28:25.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.538+0000 7f12a21a8700 1 -- 172.21.15.16:0/2987091948 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f128806b020 msgr2=0x7f128806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:25.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.538+0000 7f12a21a8700 1 --2- 172.21.15.16:0/2987091948 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f128806b020 0x7f128806d4e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1290007fd0 tx=0x7f1290009030 comp rx=0 tx=0).stop 2023-10-16T13:28:25.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.538+0000 7f12a21a8700 1 -- 172.21.15.16:0/2987091948 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f129c156c00 msgr2=0x7f129c1594d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:25.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.538+0000 7f12a21a8700 1 --2- 172.21.15.16:0/2987091948 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f129c156c00 0x7f129c1594d0 secure :-1 s=READY pgs=2106 cs=0 l=1 rev1=1 crypto rx=0x7f1294000f80 tx=0x7f129400a5f0 comp rx=0 tx=0).stop 2023-10-16T13:28:25.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.538+0000 7f12a21a8700 1 -- 172.21.15.16:0/2987091948 shutdown_connections 2023-10-16T13:28:25.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.538+0000 7f12a21a8700 1 --2- 172.21.15.16:0/2987091948 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f129c156c00 0x7f129c1594d0 unknown :-1 s=CLOSED pgs=2106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:25.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.538+0000 7f12a21a8700 1 --2- 172.21.15.16:0/2987091948 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1284003a30 0x7f129c0c4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:25.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.538+0000 7f12a21a8700 1 --2- 172.21.15.16:0/2987091948 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f128806b020 0x7f128806d4e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:25.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.538+0000 7f12a21a8700 1 --2- 172.21.15.16:0/2987091948 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f129c155f10 0x7f129c1566c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:25.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.538+0000 7f12a21a8700 1 -- 172.21.15.16:0/2987091948 >> 172.21.15.16:0/2987091948 conn(0x7f129c0b1fd0 msgr2=0x7f129c0d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:25.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.539+0000 7f12a21a8700 1 -- 172.21.15.16:0/2987091948 shutdown_connections 2023-10-16T13:28:25.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:25.539+0000 7f12a21a8700 1 -- 172.21.15.16:0/2987091948 wait complete. 2023-10-16T13:28:25.543 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 95 2023-10-16T13:28:25.611 INFO:tasks.ceph.mds.c.smithi016.stderr:2023-10-16T13:28:25.610+0000 7f8a357c3700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T13:28:25.956 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 95, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T13:28:19.523721+0000', 'modified': '2023-10-16T13:28:22.528826+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10296}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10296': {'gid': 10296, 'name': 'c', 'rank': 0, 'incarnation': 91, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.16:6836/3244401013', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6834', 'nonce': 3244401013}, {'type': 'v1', 'addr': '172.21.15.16:6836', 'nonce': 3244401013}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [24], 'metadata_pool': 23, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-16T13:28:25.956 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-16T13:28:25.957 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-10-16T13:28:26.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.083+0000 7efd9d18a700 1 Processor -- start 2023-10-16T13:28:26.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.083+0000 7efd9d18a700 1 -- start start 2023-10-16T13:28:26.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.084+0000 7efd9d18a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efd98160eb0 0x7efd9806d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:26.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.084+0000 7efd9d18a700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd9806c150 con 0x7efd980cab70 2023-10-16T13:28:26.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.084+0000 7efd9d18a700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd9806be70 con 0x7efd9806b900 2023-10-16T13:28:26.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.084+0000 7efd9d18a700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd980af370 con 0x7efd98160eb0 2023-10-16T13:28:26.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.084+0000 7efd9659c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7efd980cab70 0x7efd9806b1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:39240/0 (socket says 172.21.15.16:39240) 2023-10-16T13:28:26.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.084+0000 7efd9659c700 1 -- 172.21.15.16:0/307436417 learned_addr learned my addr 172.21.15.16:0/307436417 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:28:26.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.084+0000 7efd96d9d700 1 --2- 172.21.15.16:0/307436417 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efd98160eb0 0x7efd9806d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:26.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.084+0000 7efd95d9b700 1 -- 172.21.15.16:0/307436417 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 61945355 0 0) 0x7efd9806c150 con 0x7efd980cab70 2023-10-16T13:28:26.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.085+0000 7efd95d9b700 1 -- 172.21.15.16:0/307436417 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd800035b0 con 0x7efd980cab70 2023-10-16T13:28:26.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.085+0000 7efd95d9b700 1 -- 172.21.15.16:0/307436417 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3610285558 0 0) 0x7efd9806be70 con 0x7efd9806b900 2023-10-16T13:28:26.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.085+0000 7efd95d9b700 1 -- 172.21.15.16:0/307436417 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd9806c150 con 0x7efd9806b900 2023-10-16T13:28:26.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.085+0000 7efd95d9b700 1 -- 172.21.15.16:0/307436417 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2173902304 0 0) 0x7efd800035b0 con 0x7efd980cab70 2023-10-16T13:28:26.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.085+0000 7efd96d9d700 1 -- 172.21.15.16:0/307436417 >> v1:172.21.15.49:6789/0 conn(0x7efd9806b900 legacy=0x7efd980ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:26.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.085+0000 7efd96d9d700 1 -- 172.21.15.16:0/307436417 >> v1:172.21.15.16:6789/0 conn(0x7efd980cab70 legacy=0x7efd9806b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:26.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.085+0000 7efd96d9d700 1 -- 172.21.15.16:0/307436417 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd980cdf10 con 0x7efd98160eb0 2023-10-16T13:28:26.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.085+0000 7efd96d9d700 1 --2- 172.21.15.16:0/307436417 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efd98160eb0 0x7efd9806d8f0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7efd8c009870 tx=0x7efd8c009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb96933b46e1d5e7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:26.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.085+0000 7efd95d9b700 1 -- 172.21.15.16:0/307436417 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd88001670 con 0x7efd98160eb0 2023-10-16T13:28:26.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.085+0000 7efd95d9b700 1 -- 172.21.15.16:0/307436417 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd8c004030 con 0x7efd98160eb0 2023-10-16T13:28:26.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.085+0000 7efd95d9b700 1 -- 172.21.15.16:0/307436417 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd8c0043a0 con 0x7efd98160eb0 2023-10-16T13:28:26.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.086+0000 7efd9d18a700 1 -- 172.21.15.16:0/307436417 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efd98160eb0 msgr2=0x7efd9806d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:26.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.086+0000 7efd9d18a700 1 --2- 172.21.15.16:0/307436417 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efd98160eb0 0x7efd9806d8f0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7efd8c009870 tx=0x7efd8c009b80 comp rx=0 tx=0).stop 2023-10-16T13:28:26.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.086+0000 7efd9d18a700 1 -- 172.21.15.16:0/307436417 shutdown_connections 2023-10-16T13:28:26.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.086+0000 7efd9d18a700 1 --2- 172.21.15.16:0/307436417 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efd98160eb0 0x7efd9806d8f0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:26.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.086+0000 7efd9d18a700 1 -- 172.21.15.16:0/307436417 >> 172.21.15.16:0/307436417 conn(0x7efd980cc5b0 msgr2=0x7efd980cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:26.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.086+0000 7efd9d18a700 1 -- 172.21.15.16:0/307436417 shutdown_connections 2023-10-16T13:28:26.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.086+0000 7efd9d18a700 1 -- 172.21.15.16:0/307436417 wait complete. 2023-10-16T13:28:26.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.087+0000 7efd9d18a700 1 Processor -- start 2023-10-16T13:28:26.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.087+0000 7efd9d18a700 1 -- start start 2023-10-16T13:28:26.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.088+0000 7efd9d18a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd9806b900 0x7efd980b4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:26.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.088+0000 7efd9d18a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efd980cab70 0x7efd98157380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:26.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.088+0000 7efd9d18a700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efd98160eb0 0x7efd98155fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:26.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.088+0000 7efd9d18a700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd980b50d0 con 0x7efd9806b900 2023-10-16T13:28:26.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.088+0000 7efd9d18a700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd98156500 con 0x7efd98160eb0 2023-10-16T13:28:26.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.088+0000 7efd9d18a700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd980c73f0 con 0x7efd980cab70 2023-10-16T13:28:26.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.088+0000 7efd96d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd9806b900 0x7efd980b4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:26.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.088+0000 7efd96d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd9806b900 0x7efd980b4810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49970/0 (socket says 172.21.15.16:49970) 2023-10-16T13:28:26.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.088+0000 7efd96d9d700 1 -- 172.21.15.16:0/3029156572 learned_addr learned my addr 172.21.15.16:0/3029156572 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:26.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.088+0000 7efd96d9d700 1 -- 172.21.15.16:0/3029156572 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efd980cab70 msgr2=0x7efd98157380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:26.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.088+0000 7efd96d9d700 1 --2- 172.21.15.16:0/3029156572 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efd980cab70 0x7efd98157380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:26.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.088+0000 7efd96d9d700 1 -- 172.21.15.16:0/3029156572 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efd98160eb0 msgr2=0x7efd98155fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:26.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.088+0000 7efd96d9d700 1 --2- 172.21.15.16:0/3029156572 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efd98160eb0 0x7efd98155fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:26.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.089+0000 7efd96d9d700 1 -- 172.21.15.16:0/3029156572 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd8c0094a0 con 0x7efd9806b900 2023-10-16T13:28:26.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.089+0000 7efd96d9d700 1 --2- 172.21.15.16:0/3029156572 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd9806b900 0x7efd980b4810 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x7efd98072670 tx=0x7efd8c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:26.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.089+0000 7efd7f7fe700 1 -- 172.21.15.16:0/3029156572 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd8c003e40 con 0x7efd9806b900 2023-10-16T13:28:26.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.089+0000 7efd7f7fe700 1 -- 172.21.15.16:0/3029156572 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd8c01f070 con 0x7efd9806b900 2023-10-16T13:28:26.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.089+0000 7efd9d18a700 1 -- 172.21.15.16:0/3029156572 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd980b4df0 con 0x7efd9806b900 2023-10-16T13:28:26.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.089+0000 7efd7f7fe700 1 -- 172.21.15.16:0/3029156572 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd8c011870 con 0x7efd9806b900 2023-10-16T13:28:26.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.089+0000 7efd9d18a700 1 -- 172.21.15.16:0/3029156572 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd980c7080 con 0x7efd9806b900 2023-10-16T13:28:26.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.090+0000 7efd7f7fe700 1 -- 172.21.15.16:0/3029156572 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 107) v1 ==== 89331+0+0 (secure 0 0 0) 0x7efd8c0119d0 con 0x7efd9806b900 2023-10-16T13:28:26.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.091+0000 7efd7f7fe700 1 --2- 172.21.15.16:0/3029156572 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7efd8406b020 0x7efd8406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:26.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.091+0000 7efd7f7fe700 1 -- 172.21.15.16:0/3029156572 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 6129+0+0 (secure 0 0 0) 0x7efd8c021070 con 0x7efd9806b900 2023-10-16T13:28:26.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.091+0000 7efd9659c700 1 --2- 172.21.15.16:0/3029156572 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7efd8406b020 0x7efd8406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:26.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.091+0000 7efd9659c700 1 --2- 172.21.15.16:0/3029156572 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7efd8406b020 0x7efd8406d4e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7efd980ae910 tx=0x7efd88001230 comp rx=0 tx=0).ready entity=mgr.10047 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:26.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.093+0000 7efda9374b80 1 Processor -- start 2023-10-16T13:28:26.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.093+0000 7efda9374b80 1 -- start start 2023-10-16T13:28:26.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.094+0000 7efda9374b80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557ca5e27220 0x557ca5e2ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:26.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.094+0000 7efda9374b80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x557ca5e2c630 0x557ca5e2efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:26.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.094+0000 7efda9374b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557ca5e02010 0x557ca5e026c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:26.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.094+0000 7efda9374b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ca5d78c20 con 0x557ca5e02010 2023-10-16T13:28:26.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.094+0000 7efda9374b80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ca5d5dc10 con 0x557ca5e27220 2023-10-16T13:28:26.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.094+0000 7efda9374b80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x557ca5d5b530 con 0x557ca5e2c630 2023-10-16T13:28:26.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.094+0000 7efd9759e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557ca5e02010 0x557ca5e026c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:26.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.094+0000 7efd9759e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557ca5e02010 0x557ca5e026c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49982/0 (socket says 172.21.15.16:49982) 2023-10-16T13:28:26.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.094+0000 7efd9759e700 1 -- 172.21.15.16:0/331053048 learned_addr learned my addr 172.21.15.16:0/331053048 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:26.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.094+0000 7efd96d9d700 1 --2- 172.21.15.16:0/331053048 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557ca5e27220 0x557ca5e2ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:26.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.094+0000 7efd9659c700 1 --2- 172.21.15.16:0/331053048 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x557ca5e2c630 0x557ca5e2efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:26.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.094+0000 7efd9759e700 1 -- 172.21.15.16:0/331053048 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x557ca5e2c630 msgr2=0x557ca5e2efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:28:26.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.094+0000 7efd9759e700 1 --2- 172.21.15.16:0/331053048 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x557ca5e2c630 0x557ca5e2efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:26.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.094+0000 7efd9759e700 1 -- 172.21.15.16:0/331053048 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557ca5e27220 msgr2=0x557ca5e2ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:28:26.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.094+0000 7efd9759e700 1 --2- 172.21.15.16:0/331053048 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557ca5e27220 0x557ca5e2ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:26.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.094+0000 7efd9759e700 1 -- 172.21.15.16:0/331053048 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557ca5d143a0 con 0x557ca5e02010 2023-10-16T13:28:26.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.094+0000 7efd96d9d700 1 --2- 172.21.15.16:0/331053048 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557ca5e27220 0x557ca5e2ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:28:26.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.094+0000 7efd9759e700 1 --2- 172.21.15.16:0/331053048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557ca5e02010 0x557ca5e026c0 secure :-1 s=READY pgs=2109 cs=0 l=1 rev1=1 crypto rx=0x7efd90001fd0 tx=0x7efd9000c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44c3ddb41b127f2a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:26.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.094+0000 7efd7cff9700 1 -- 172.21.15.16:0/331053048 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd9000ce60 con 0x557ca5e02010 2023-10-16T13:28:26.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.094+0000 7efd7cff9700 1 -- 172.21.15.16:0/331053048 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd90003a00 con 0x557ca5e02010 2023-10-16T13:28:26.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.094+0000 7efd7cff9700 1 -- 172.21.15.16:0/331053048 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd90003db0 con 0x557ca5e02010 2023-10-16T13:28:26.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.095+0000 7efda9374b80 1 -- 172.21.15.16:0/331053048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557ca5e02010 msgr2=0x557ca5e026c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:26.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.095+0000 7efda9374b80 1 --2- 172.21.15.16:0/331053048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557ca5e02010 0x557ca5e026c0 secure :-1 s=READY pgs=2109 cs=0 l=1 rev1=1 crypto rx=0x7efd90001fd0 tx=0x7efd9000c3c0 comp rx=0 tx=0).stop 2023-10-16T13:28:26.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.095+0000 7efda9374b80 1 -- 172.21.15.16:0/331053048 shutdown_connections 2023-10-16T13:28:26.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.095+0000 7efda9374b80 1 --2- 172.21.15.16:0/331053048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557ca5e02010 0x557ca5e026c0 unknown :-1 s=CLOSED pgs=2109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:26.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.095+0000 7efda9374b80 1 --2- 172.21.15.16:0/331053048 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557ca5e27220 0x557ca5e2ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:26.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.095+0000 7efda9374b80 1 --2- 172.21.15.16:0/331053048 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x557ca5e2c630 0x557ca5e2efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:26.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.095+0000 7efda9374b80 1 -- 172.21.15.16:0/331053048 >> 172.21.15.16:0/331053048 conn(0x557ca5d31d80 msgr2=0x557ca5e4cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:26.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.095+0000 7efda9374b80 1 -- 172.21.15.16:0/331053048 shutdown_connections 2023-10-16T13:28:26.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.095+0000 7efda9374b80 1 -- 172.21.15.16:0/331053048 wait complete. 2023-10-16T13:28:26.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.096+0000 7efda9374b80 1 -- start start 2023-10-16T13:28:26.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.096+0000 7efda9374b80 1 Processor -- start 2023-10-16T13:28:26.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.097+0000 7efda9374b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557ca5e02010 0x557ca5e44030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:26.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.097+0000 7efda9374b80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557ca5e27220 0x557ca5e308c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:26.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.097+0000 7efda9374b80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x557ca5e2c630 0x557ca5e2a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:26.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.097+0000 7efda9374b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ca5d56e20 con 0x557ca5e02010 2023-10-16T13:28:26.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.097+0000 7efda9374b80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ca5cf36c0 con 0x557ca5e27220 2023-10-16T13:28:26.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.097+0000 7efda9374b80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x557ca5cf3360 con 0x557ca5e2c630 2023-10-16T13:28:26.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.097+0000 7efda9374b80 10 client.0 _ll_register_callbacks cb 0x557ca5b507b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T13:28:26.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.097+0000 7efd96d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557ca5e02010 0x557ca5e44030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:26.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.097+0000 7efd96d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557ca5e02010 0x557ca5e44030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49984/0 (socket says 172.21.15.16:49984) 2023-10-16T13:28:26.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.097+0000 7efd96d9d700 1 -- 172.21.15.16:0/2667351821 learned_addr learned my addr 172.21.15.16:0/2667351821 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:26.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.097+0000 7efd9659c700 1 --2- 172.21.15.16:0/2667351821 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557ca5e27220 0x557ca5e308c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:26.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.097+0000 7efd9759e700 1 --2- 172.21.15.16:0/2667351821 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x557ca5e2c630 0x557ca5e2a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:26.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.097+0000 7efd96d9d700 1 -- 172.21.15.16:0/2667351821 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x557ca5e2c630 msgr2=0x557ca5e2a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:28:26.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.097+0000 7efd96d9d700 1 --2- 172.21.15.16:0/2667351821 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x557ca5e2c630 0x557ca5e2a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:26.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.097+0000 7efd96d9d700 1 -- 172.21.15.16:0/2667351821 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557ca5e27220 msgr2=0x557ca5e308c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:28:26.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.097+0000 7efd96d9d700 1 --2- 172.21.15.16:0/2667351821 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557ca5e27220 0x557ca5e308c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:26.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.097+0000 7efd96d9d700 1 -- 172.21.15.16:0/2667351821 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd9806c150 con 0x557ca5e02010 2023-10-16T13:28:26.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.097+0000 7efd9659c700 1 --2- 172.21.15.16:0/2667351821 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557ca5e27220 0x557ca5e308c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:28:26.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.097+0000 7efd96d9d700 1 --2- 172.21.15.16:0/2667351821 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557ca5e02010 0x557ca5e44030 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 crypto rx=0x7efd8c059d30 tx=0x7efd8c05a570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e65fbb53948b9f14 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:26.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.097+0000 7efd75ffb700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2023-10-16T13:28:26.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.097+0000 7efd75ffb700 1 -- 172.21.15.16:0/2667351821 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd8c0620a0 con 0x557ca5e02010 2023-10-16T13:28:26.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.097+0000 7efd75ffb700 1 -- 172.21.15.16:0/2667351821 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(190..190 src has 1..190) v4 ==== 6129+0+0 (secure 0 0 0) 0x7efd8c011c80 con 0x557ca5e02010 2023-10-16T13:28:26.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.097+0000 7efda9374b80 1 -- 172.21.15.16:0/2667351821 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557ca5df75b0 con 0x557ca5e02010 2023-10-16T13:28:26.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.098+0000 7efd75ffb700 1 -- 172.21.15.16:0/2667351821 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=95) v2 ==== 24+0+0 (secure 0 0 0) 0x7efd8c011e60 con 0x557ca5e02010 2023-10-16T13:28:26.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.098+0000 7efda9374b80 10 client.10443 fetch_fsmap learned FSMap version 95 2023-10-16T13:28:26.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.098+0000 7efda9374b80 1 -- 172.21.15.16:0/2667351821 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=95,monmap=2+}) v3 -- 0x557ca5b81d90 con 0x557ca5e02010 2023-10-16T13:28:26.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.098+0000 7efd75ffb700 1 -- 172.21.15.16:0/2667351821 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x557ca5df75b0 con 0x557ca5e02010 2023-10-16T13:28:26.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.098+0000 7efd75ffb700 1 -- 172.21.15.16:0/2667351821 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 95) v1 ==== 3294+0+0 (secure 0 0 0) 0x7efd8c05a000 con 0x557ca5e02010 2023-10-16T13:28:26.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.098+0000 7efda9374b80 10 client.10443 fetch_fsmap finished waiting for FSMap version 95 2023-10-16T13:28:26.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.098+0000 7efda9374b80 10 client.10443 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-16T13:28:26.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.098+0000 7efda9374b80 20 client.10443 populate_metadata read hostname 'smithi016' 2023-10-16T13:28:26.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.098+0000 7efda9374b80 1 --2- 172.21.15.16:0/2667351821 >> [v2:172.21.15.16:6834/3244401013,v1:172.21.15.16:6836/3244401013] conn(0x557ca5e31b20 0x557ca5e27600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:26.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.098+0000 7efda9374b80 4 client.10443 mds_command: new command op to 10296 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T13:28:26.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.098+0000 7efda9374b80 1 -- 172.21.15.16:0/2667351821 --> [v2:172.21.15.16:6834/3244401013,v1:172.21.15.16:6836/3244401013] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557ca5d4fe00 con 0x557ca5e31b20 2023-10-16T13:28:26.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.098+0000 7efd9659c700 1 --2- 172.21.15.16:0/2667351821 >> [v2:172.21.15.16:6834/3244401013,v1:172.21.15.16:6836/3244401013] conn(0x557ca5e31b20 0x557ca5e27600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:26.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.098+0000 7efd75ffb700 10 client.10443 ms_handle_connect on v2:172.21.15.16:6834/3244401013 2023-10-16T13:28:26.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.098+0000 7efd9659c700 1 --2- 172.21.15.16:0/2667351821 >> [v2:172.21.15.16:6834/3244401013,v1:172.21.15.16:6836/3244401013] conn(0x557ca5e31b20 0x557ca5e27600 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=291009fc93bfc5e8 server_cookie=7bc765ccd0bd1b78 in_seq=0 out_seq=0 2023-10-16T13:28:26.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.099+0000 7efd75ffb700 1 -- 172.21.15.16:0/2667351821 <== mds.0 v2:172.21.15.16:6834/3244401013 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x557ca5d4fe00 con 0x557ca5e31b20 2023-10-16T13:28:26.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.099+0000 7efd75ffb700 10 client.10443 handle_command_reply: tid=0 2023-10-16T13:28:26.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.099+0000 7efda9374b80 1 client.10443 shutdown 2023-10-16T13:28:26.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.099+0000 7efda9374b80 1 -- 172.21.15.16:0/2667351821 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557ca5e02010 msgr2=0x557ca5e44030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:26.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.099+0000 7efda9374b80 1 --2- 172.21.15.16:0/2667351821 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557ca5e02010 0x557ca5e44030 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 crypto rx=0x7efd8c059d30 tx=0x7efd8c05a570 comp rx=0 tx=0).stop 2023-10-16T13:28:26.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.099+0000 7efda9374b80 1 -- 172.21.15.16:0/2667351821 shutdown_connections 2023-10-16T13:28:26.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.099+0000 7efda9374b80 1 --2- 172.21.15.16:0/2667351821 >> [v2:172.21.15.16:6834/3244401013,v1:172.21.15.16:6836/3244401013] conn(0x557ca5e31b20 0x557ca5e27600 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:26.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.099+0000 7efda9374b80 1 --2- 172.21.15.16:0/2667351821 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x557ca5e2c630 0x557ca5e2a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:26.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.099+0000 7efd75ffb700 0 client.10443 ms_handle_reset on v2:172.21.15.16:6834/3244401013 2023-10-16T13:28:26.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.099+0000 7efda9374b80 1 --2- 172.21.15.16:0/2667351821 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557ca5e02010 0x557ca5e44030 unknown :-1 s=CLOSED pgs=2110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:26.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.099+0000 7efda9374b80 1 --2- 172.21.15.16:0/2667351821 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557ca5e27220 0x557ca5e308c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:26.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.099+0000 7efda9374b80 1 -- 172.21.15.16:0/2667351821 >> 172.21.15.16:0/2667351821 conn(0x557ca5d31d80 msgr2=0x557ca5df9640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:26.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.099+0000 7efda9374b80 1 -- 172.21.15.16:0/2667351821 shutdown_connections 2023-10-16T13:28:26.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.099+0000 7efda9374b80 1 -- 172.21.15.16:0/2667351821 wait complete. 2023-10-16T13:28:26.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.099+0000 7efda9374b80 20 client.10443 trim_cache size 0 max 16384 2023-10-16T13:28:26.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.117+0000 7efda9374b80 1 Processor -- start 2023-10-16T13:28:26.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.117+0000 7efda9374b80 1 -- start start 2023-10-16T13:28:26.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.118+0000 7efda9374b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557ca5e02010 0x557ca5e2b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:26.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.118+0000 7efda9374b80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x557ca5e27220 0x557ca5e2bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:26.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.118+0000 7efda9374b80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557ca5e2c630 0x557ca5e03690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:26.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.118+0000 7efda9374b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ca5bd78d0 con 0x557ca5e02010 2023-10-16T13:28:26.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.118+0000 7efda9374b80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ca5e310e0 con 0x557ca5e2c630 2023-10-16T13:28:26.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.118+0000 7efda9374b80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x557ca5e04ef0 con 0x557ca5e27220 2023-10-16T13:28:26.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.118+0000 7efd96d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557ca5e02010 0x557ca5e2b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:26.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.118+0000 7efd96d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557ca5e02010 0x557ca5e2b750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49992/0 (socket says 172.21.15.16:49992) 2023-10-16T13:28:26.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.118+0000 7efd96d9d700 1 -- 172.21.15.16:0/1723473859 learned_addr learned my addr 172.21.15.16:0/1723473859 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:26.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.118+0000 7efd9759e700 1 --2- 172.21.15.16:0/1723473859 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557ca5e2c630 0x557ca5e03690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:26.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.118+0000 7efd9659c700 1 --2- 172.21.15.16:0/1723473859 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x557ca5e27220 0x557ca5e2bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:26.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.119+0000 7efd96d9d700 1 -- 172.21.15.16:0/1723473859 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x557ca5e27220 msgr2=0x557ca5e2bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:28:26.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.119+0000 7efd96d9d700 1 --2- 172.21.15.16:0/1723473859 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x557ca5e27220 0x557ca5e2bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:26.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.119+0000 7efd96d9d700 1 -- 172.21.15.16:0/1723473859 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557ca5e2c630 msgr2=0x557ca5e03690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:28:26.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.119+0000 7efd96d9d700 1 --2- 172.21.15.16:0/1723473859 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557ca5e2c630 0x557ca5e03690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:26.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.119+0000 7efd96d9d700 1 -- 172.21.15.16:0/1723473859 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd980c8700 con 0x557ca5e02010 2023-10-16T13:28:26.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.119+0000 7efd9759e700 1 --2- 172.21.15.16:0/1723473859 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557ca5e2c630 0x557ca5e03690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:28:26.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.119+0000 7efd96d9d700 1 --2- 172.21.15.16:0/1723473859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557ca5e02010 0x557ca5e2b750 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 crypto rx=0x7efd8c061b70 tx=0x7efd8c060000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92db6ebebd1c2265 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:26.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.119+0000 7efd7d7fa700 1 -- 172.21.15.16:0/1723473859 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd8c059b70 con 0x557ca5e02010 2023-10-16T13:28:26.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.119+0000 7efd7d7fa700 1 -- 172.21.15.16:0/1723473859 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd8c05d070 con 0x557ca5e02010 2023-10-16T13:28:26.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.119+0000 7efd7d7fa700 1 -- 172.21.15.16:0/1723473859 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd8c0620a0 con 0x557ca5e02010 2023-10-16T13:28:26.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.120+0000 7efda9374b80 1 -- 172.21.15.16:0/1723473859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557ca5e02010 msgr2=0x557ca5e2b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:26.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.120+0000 7efda9374b80 1 --2- 172.21.15.16:0/1723473859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557ca5e02010 0x557ca5e2b750 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 crypto rx=0x7efd8c061b70 tx=0x7efd8c060000 comp rx=0 tx=0).stop 2023-10-16T13:28:26.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.120+0000 7efda9374b80 1 -- 172.21.15.16:0/1723473859 shutdown_connections 2023-10-16T13:28:26.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.120+0000 7efda9374b80 1 --2- 172.21.15.16:0/1723473859 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557ca5e2c630 0x557ca5e03690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:26.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.120+0000 7efda9374b80 1 --2- 172.21.15.16:0/1723473859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557ca5e02010 0x557ca5e2b750 unknown :-1 s=CLOSED pgs=2111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:26.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.120+0000 7efda9374b80 1 --2- 172.21.15.16:0/1723473859 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x557ca5e27220 0x557ca5e2bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:26.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.120+0000 7efda9374b80 1 -- 172.21.15.16:0/1723473859 >> 172.21.15.16:0/1723473859 conn(0x557ca5d31d80 msgr2=0x557ca5e4d110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:26.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.120+0000 7efda9374b80 1 -- 172.21.15.16:0/1723473859 shutdown_connections 2023-10-16T13:28:26.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.120+0000 7efda9374b80 1 -- 172.21.15.16:0/1723473859 wait complete. 2023-10-16T13:28:26.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.120+0000 7efda9374b80 1 -- start start 2023-10-16T13:28:26.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.120+0000 7efda9374b80 1 Processor -- start 2023-10-16T13:28:26.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.121+0000 7efda9374b80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557ca5e02010 0x557ca5e47ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:26.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.121+0000 7efda9374b80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x557ca5e27220 0x557ca5e48010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:26.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.121+0000 7efda9374b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557ca5e2c630 0x557ca5e43cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:26.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.121+0000 7efda9374b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ca5e44b90 con 0x557ca5e2c630 2023-10-16T13:28:26.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.121+0000 7efda9374b80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ca5ea7090 con 0x557ca5e02010 2023-10-16T13:28:26.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.121+0000 7efda9374b80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x557ca5e48be0 con 0x557ca5e27220 2023-10-16T13:28:26.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.121+0000 7efda9374b80 10 client.0 _ll_register_callbacks cb 0x7efd8c010380 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T13:28:26.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.121+0000 7efd9759e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557ca5e2c630 0x557ca5e43cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:26.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.121+0000 7efd9759e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557ca5e2c630 0x557ca5e43cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50002/0 (socket says 172.21.15.16:50002) 2023-10-16T13:28:26.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.121+0000 7efd9759e700 1 -- 172.21.15.16:0/609559132 learned_addr learned my addr 172.21.15.16:0/609559132 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:26.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.121+0000 7efd96d9d700 1 --2- 172.21.15.16:0/609559132 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557ca5e02010 0x557ca5e47ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:26.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.121+0000 7efd9659c700 1 --2- 172.21.15.16:0/609559132 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x557ca5e27220 0x557ca5e48010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:26.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.122+0000 7efd9759e700 1 -- 172.21.15.16:0/609559132 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x557ca5e27220 msgr2=0x557ca5e48010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:28:26.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.122+0000 7efd9759e700 1 --2- 172.21.15.16:0/609559132 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x557ca5e27220 0x557ca5e48010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:26.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.122+0000 7efd9759e700 1 -- 172.21.15.16:0/609559132 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557ca5e02010 msgr2=0x557ca5e47ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:28:26.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.122+0000 7efd9759e700 1 --2- 172.21.15.16:0/609559132 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557ca5e02010 0x557ca5e47ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:26.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.122+0000 7efd9759e700 1 -- 172.21.15.16:0/609559132 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557ca5b81d90 con 0x557ca5e2c630 2023-10-16T13:28:26.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.122+0000 7efd9759e700 1 --2- 172.21.15.16:0/609559132 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557ca5e2c630 0x557ca5e43cf0 secure :-1 s=READY pgs=2112 cs=0 l=1 rev1=1 crypto rx=0x7efd90015040 tx=0x7efd90001360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b6faed1d4bebef8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:26.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.122+0000 7efd75ffb700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2023-10-16T13:28:26.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.122+0000 7efd75ffb700 1 -- 172.21.15.16:0/609559132 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd90003d70 con 0x557ca5e2c630 2023-10-16T13:28:26.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.122+0000 7efd75ffb700 1 -- 172.21.15.16:0/609559132 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(190..190 src has 1..190) v4 ==== 6129+0+0 (secure 0 0 0) 0x7efd90028ca0 con 0x557ca5e2c630 2023-10-16T13:28:26.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.122+0000 7efda9374b80 1 -- 172.21.15.16:0/609559132 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efd90003a00 con 0x557ca5e2c630 2023-10-16T13:28:26.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.122+0000 7efd75ffb700 1 -- 172.21.15.16:0/609559132 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=95) v2 ==== 24+0+0 (secure 0 0 0) 0x7efd90028e60 con 0x557ca5e2c630 2023-10-16T13:28:26.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.122+0000 7efda9374b80 10 client.10449 fetch_fsmap learned FSMap version 95 2023-10-16T13:28:26.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.122+0000 7efda9374b80 1 -- 172.21.15.16:0/609559132 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=95,monmap=2+}) v3 -- 0x557ca5b6ac10 con 0x557ca5e2c630 2023-10-16T13:28:26.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.122+0000 7efd75ffb700 1 -- 172.21.15.16:0/609559132 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd90003a00 con 0x557ca5e2c630 2023-10-16T13:28:26.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.122+0000 7efd75ffb700 1 -- 172.21.15.16:0/609559132 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 95) v1 ==== 3294+0+0 (secure 0 0 0) 0x7efd90005070 con 0x557ca5e2c630 2023-10-16T13:28:26.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.122+0000 7efda9374b80 10 client.10449 fetch_fsmap finished waiting for FSMap version 95 2023-10-16T13:28:26.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.123+0000 7efda9374b80 10 client.10449 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-16T13:28:26.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.123+0000 7efda9374b80 20 client.10449 populate_metadata read hostname 'smithi016' 2023-10-16T13:28:26.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.123+0000 7efda9374b80 1 --2- 172.21.15.16:0/609559132 >> [v2:172.21.15.16:6834/3244401013,v1:172.21.15.16:6836/3244401013] conn(0x557ca5e44230 0x557ca5e44610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:26.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.123+0000 7efda9374b80 4 client.10449 mds_command: new command op to 10296 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T13:28:26.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.123+0000 7efda9374b80 1 -- 172.21.15.16:0/609559132 --> [v2:172.21.15.16:6834/3244401013,v1:172.21.15.16:6836/3244401013] -- command(tid 0: {"prefix": "status"}) v1 -- 0x557ca5d4fb20 con 0x557ca5e44230 2023-10-16T13:28:26.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.123+0000 7efd96d9d700 1 --2- 172.21.15.16:0/609559132 >> [v2:172.21.15.16:6834/3244401013,v1:172.21.15.16:6836/3244401013] conn(0x557ca5e44230 0x557ca5e44610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:26.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.123+0000 7efd75ffb700 10 client.10449 ms_handle_connect on v2:172.21.15.16:6834/3244401013 2023-10-16T13:28:26.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.123+0000 7efd96d9d700 1 --2- 172.21.15.16:0/609559132 >> [v2:172.21.15.16:6834/3244401013,v1:172.21.15.16:6836/3244401013] conn(0x557ca5e44230 0x557ca5e44610 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9677067666b8e7a3 server_cookie=5948617658cd31ef in_seq=0 out_seq=0 2023-10-16T13:28:26.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.123+0000 7efd75ffb700 1 -- 172.21.15.16:0/609559132 <== mds.0 v2:172.21.15.16:6834/3244401013 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+328 (crc 0 0 0) 0x557ca5d4fb20 con 0x557ca5e44230 2023-10-16T13:28:26.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.123+0000 7efd75ffb700 10 client.10449 handle_command_reply: tid=0 2023-10-16T13:28:26.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.123+0000 7efda9374b80 1 client.10449 shutdown 2023-10-16T13:28:26.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.123+0000 7efda9374b80 1 -- 172.21.15.16:0/609559132 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557ca5e2c630 msgr2=0x557ca5e43cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:26.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.123+0000 7efda9374b80 1 --2- 172.21.15.16:0/609559132 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557ca5e2c630 0x557ca5e43cf0 secure :-1 s=READY pgs=2112 cs=0 l=1 rev1=1 crypto rx=0x7efd90015040 tx=0x7efd90001360 comp rx=0 tx=0).stop 2023-10-16T13:28:26.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.123+0000 7efda9374b80 1 -- 172.21.15.16:0/609559132 shutdown_connections 2023-10-16T13:28:26.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.123+0000 7efda9374b80 1 --2- 172.21.15.16:0/609559132 >> [v2:172.21.15.16:6834/3244401013,v1:172.21.15.16:6836/3244401013] conn(0x557ca5e44230 0x557ca5e44610 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:26.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.123+0000 7efda9374b80 1 --2- 172.21.15.16:0/609559132 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557ca5e2c630 0x557ca5e43cf0 unknown :-1 s=CLOSED pgs=2112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:26.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.123+0000 7efd75ffb700 0 client.10449 ms_handle_reset on v2:172.21.15.16:6834/3244401013 2023-10-16T13:28:26.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.123+0000 7efda9374b80 1 --2- 172.21.15.16:0/609559132 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557ca5e02010 0x557ca5e47ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:26.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.123+0000 7efda9374b80 1 --2- 172.21.15.16:0/609559132 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x557ca5e27220 0x557ca5e48010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:26.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.123+0000 7efda9374b80 1 -- 172.21.15.16:0/609559132 >> 172.21.15.16:0/609559132 conn(0x557ca5d31d80 msgr2=0x557ca5e023f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:26.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.123+0000 7efda9374b80 1 -- 172.21.15.16:0/609559132 shutdown_connections 2023-10-16T13:28:26.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.123+0000 7efda9374b80 1 -- 172.21.15.16:0/609559132 wait complete. 2023-10-16T13:28:26.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.124+0000 7efda9374b80 20 client.10449 trim_cache size 0 max 16384 2023-10-16T13:28:26.142 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-10-16T13:28:26.142 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "5b28dfa1-b21f-474a-9b2e-9fb77af58254", 2023-10-16T13:28:26.142 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 0, 2023-10-16T13:28:26.143 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 10296, 2023-10-16T13:28:26.143 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2023-10-16T13:28:26.143 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2023-10-16T13:28:26.143 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2023-10-16T13:28:26.143 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 6.0248092389999996, 2023-10-16T13:28:26.143 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 95, 2023-10-16T13:28:26.144 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 190, 2023-10-16T13:28:26.144 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 190, 2023-10-16T13:28:26.144 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 11.390109073 2023-10-16T13:28:26.144 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-10-16T13:28:26.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.124+0000 7efd9d18a700 1 -- 172.21.15.16:0/3029156572 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7efd8406b020 msgr2=0x7efd8406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:26.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.124+0000 7efd9d18a700 1 --2- 172.21.15.16:0/3029156572 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7efd8406b020 0x7efd8406d4e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7efd980ae910 tx=0x7efd88001230 comp rx=0 tx=0).stop 2023-10-16T13:28:26.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.124+0000 7efd9d18a700 1 -- 172.21.15.16:0/3029156572 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd9806b900 msgr2=0x7efd980b4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:26.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.124+0000 7efd9d18a700 1 --2- 172.21.15.16:0/3029156572 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd9806b900 0x7efd980b4810 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x7efd98072670 tx=0x7efd8c004750 comp rx=0 tx=0).stop 2023-10-16T13:28:26.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.124+0000 7efd9d18a700 1 -- 172.21.15.16:0/3029156572 shutdown_connections 2023-10-16T13:28:26.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.124+0000 7efd9d18a700 1 --2- 172.21.15.16:0/3029156572 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7efd8406b020 0x7efd8406d4e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:26.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.124+0000 7efd9d18a700 1 --2- 172.21.15.16:0/3029156572 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efd98160eb0 0x7efd98155fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:26.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.124+0000 7efd9d18a700 1 --2- 172.21.15.16:0/3029156572 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd9806b900 0x7efd980b4810 unknown :-1 s=CLOSED pgs=2108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:26.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.124+0000 7efd9d18a700 1 --2- 172.21.15.16:0/3029156572 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efd980cab70 0x7efd98157380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:26.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.124+0000 7efd9d18a700 1 -- 172.21.15.16:0/3029156572 >> 172.21.15.16:0/3029156572 conn(0x7efd980cc5b0 msgr2=0x7efd9806a7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:26.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.124+0000 7efd9d18a700 1 -- 172.21.15.16:0/3029156572 shutdown_connections 2023-10-16T13:28:26.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.124+0000 7efd9d18a700 1 -- 172.21.15.16:0/3029156572 wait complete. 2023-10-16T13:28:26.148 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T13:28:26.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.271+0000 7f8ad0e13700 1 Processor -- start 2023-10-16T13:28:26.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.271+0000 7f8ad0e13700 1 -- start start 2023-10-16T13:28:26.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.272+0000 7f8ad0e13700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8acc160eb0 0x7f8acc06d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:26.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.272+0000 7f8ad0e13700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f8acc0cab70 0x7f8acc0caff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:26.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.272+0000 7f8ad0e13700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8acc06c150 con 0x7f8acc06b1f0 2023-10-16T13:28:26.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.272+0000 7f8ad0e13700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8acc0af370 con 0x7f8acc160eb0 2023-10-16T13:28:26.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.272+0000 7f8ad0e13700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8acc0cb530 con 0x7f8acc0cab70 2023-10-16T13:28:26.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.272+0000 7f8aca59c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8acc160eb0 0x7f8acc06d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:26.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.272+0000 7f8acad9d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f8acc06b1f0 0x7f8acc06b640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:39284/0 (socket says 172.21.15.16:39284) 2023-10-16T13:28:26.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.272+0000 7f8acad9d700 1 -- 172.21.15.16:0/991185162 learned_addr learned my addr 172.21.15.16:0/991185162 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:28:26.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.273+0000 7f8ac9d9b700 1 --2- 172.21.15.16:0/991185162 >> v2:172.21.15.49:3300/0 conn(0x7f8acc0cab70 0x7f8acc0caff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:26.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.273+0000 7f8aca59c700 1 -- 172.21.15.16:0/991185162 >> v2:172.21.15.49:3300/0 conn(0x7f8acc0cab70 msgr2=0x7f8acc0caff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:28:26.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.273+0000 7f8aca59c700 1 --2- 172.21.15.16:0/991185162 >> v2:172.21.15.49:3300/0 conn(0x7f8acc0cab70 0x7f8acc0caff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:26.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.273+0000 7f8aca59c700 1 -- 172.21.15.16:0/991185162 >> v1:172.21.15.16:6789/0 conn(0x7f8acc06b1f0 legacy=0x7f8acc06b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:26.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.273+0000 7f8ac959a700 1 -- 172.21.15.16:0/991185162 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2218561806 0 0) 0x7f8acc06c150 con 0x7f8acc06b1f0 2023-10-16T13:28:26.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.273+0000 7f8aca59c700 1 -- 172.21.15.16:0/991185162 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8acc06be70 con 0x7f8acc160eb0 2023-10-16T13:28:26.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.273+0000 7f8aca59c700 1 --2- 172.21.15.16:0/991185162 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8acc160eb0 0x7f8acc06d8f0 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 crypto rx=0x7f8acc0ca3c0 tx=0x7f8ab4009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c511a7f900135a4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:26.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.273+0000 7f8ac959a700 1 -- 172.21.15.16:0/991185162 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ab4011830 con 0x7f8acc160eb0 2023-10-16T13:28:26.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.273+0000 7f8ac959a700 1 -- 172.21.15.16:0/991185162 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ab4011990 con 0x7f8acc160eb0 2023-10-16T13:28:26.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.273+0000 7f8ac959a700 1 -- 172.21.15.16:0/991185162 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ab4011d20 con 0x7f8acc160eb0 2023-10-16T13:28:26.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.274+0000 7f8ad0e13700 1 -- 172.21.15.16:0/991185162 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8acc160eb0 msgr2=0x7f8acc06d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:26.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.274+0000 7f8ad0e13700 1 --2- 172.21.15.16:0/991185162 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8acc160eb0 0x7f8acc06d8f0 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 crypto rx=0x7f8acc0ca3c0 tx=0x7f8ab4009b20 comp rx=0 tx=0).stop 2023-10-16T13:28:26.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.274+0000 7f8ad0e13700 1 -- 172.21.15.16:0/991185162 shutdown_connections 2023-10-16T13:28:26.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.274+0000 7f8ad0e13700 1 --2- 172.21.15.16:0/991185162 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8acc160eb0 0x7f8acc06d8f0 unknown :-1 s=CLOSED pgs=2114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:26.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.274+0000 7f8ad0e13700 1 --2- 172.21.15.16:0/991185162 >> v2:172.21.15.49:3300/0 conn(0x7f8acc0cab70 0x7f8acc0caff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:26.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.274+0000 7f8ad0e13700 1 -- 172.21.15.16:0/991185162 >> 172.21.15.16:0/991185162 conn(0x7f8acc0cc5b0 msgr2=0x7f8acc0cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:26.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.274+0000 7f8ad0e13700 1 -- 172.21.15.16:0/991185162 shutdown_connections 2023-10-16T13:28:26.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.274+0000 7f8ad0e13700 1 -- 172.21.15.16:0/991185162 wait complete. 2023-10-16T13:28:26.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.275+0000 7f8ad0e13700 1 Processor -- start 2023-10-16T13:28:26.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.275+0000 7f8ad0e13700 1 -- start start 2023-10-16T13:28:26.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.276+0000 7f8ad0e13700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8acc06b1f0 0x7f8acc1f59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:26.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.276+0000 7f8ad0e13700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8acc0cab70 0x7f8acc0c8dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:26.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.276+0000 7f8ad0e13700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8acc160eb0 0x7f8acc0c9310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:26.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.276+0000 7f8ad0e13700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8acc0b2790 con 0x7f8acc0cab70 2023-10-16T13:28:26.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.276+0000 7f8ad0e13700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8acc0b4430 con 0x7f8acc06b1f0 2023-10-16T13:28:26.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.276+0000 7f8ad0e13700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8acc0b4710 con 0x7f8acc160eb0 2023-10-16T13:28:26.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.276+0000 7f8ac9d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8acc0cab70 0x7f8acc0c8dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:26.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.277+0000 7f8ac9d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8acc0cab70 0x7f8acc0c8dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50014/0 (socket says 172.21.15.16:50014) 2023-10-16T13:28:26.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.277+0000 7f8ac9d9b700 1 -- 172.21.15.16:0/4101521103 learned_addr learned my addr 172.21.15.16:0/4101521103 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:26.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.277+0000 7f8aca59c700 1 --2- 172.21.15.16:0/4101521103 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8acc06b1f0 0x7f8acc1f59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:26.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.277+0000 7f8acad9d700 1 --2- 172.21.15.16:0/4101521103 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8acc160eb0 0x7f8acc0c9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:26.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.277+0000 7f8ac9d9b700 1 -- 172.21.15.16:0/4101521103 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8acc160eb0 msgr2=0x7f8acc0c9310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:26.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.277+0000 7f8ac9d9b700 1 --2- 172.21.15.16:0/4101521103 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8acc160eb0 0x7f8acc0c9310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:26.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.277+0000 7f8ac9d9b700 1 -- 172.21.15.16:0/4101521103 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8acc06b1f0 msgr2=0x7f8acc1f59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:26.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.277+0000 7f8ac9d9b700 1 --2- 172.21.15.16:0/4101521103 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8acc06b1f0 0x7f8acc1f59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:26.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.277+0000 7f8ac9d9b700 1 -- 172.21.15.16:0/4101521103 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ab40094a0 con 0x7f8acc0cab70 2023-10-16T13:28:26.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.277+0000 7f8acad9d700 1 --2- 172.21.15.16:0/4101521103 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8acc160eb0 0x7f8acc0c9310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:28:26.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.277+0000 7f8ac9d9b700 1 --2- 172.21.15.16:0/4101521103 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8acc0cab70 0x7f8acc0c8dd0 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 crypto rx=0x7f8abc00d800 tx=0x7f8abc00db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:26.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.277+0000 7f8ac2ffd700 1 -- 172.21.15.16:0/4101521103 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8abc0125f0 con 0x7f8acc0cab70 2023-10-16T13:28:26.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.277+0000 7f8ac2ffd700 1 -- 172.21.15.16:0/4101521103 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8abc012750 con 0x7f8acc0cab70 2023-10-16T13:28:26.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.277+0000 7f8ad0e13700 1 -- 172.21.15.16:0/4101521103 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8acc0b49f0 con 0x7f8acc0cab70 2023-10-16T13:28:26.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.277+0000 7f8ac2ffd700 1 -- 172.21.15.16:0/4101521103 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8abc012b00 con 0x7f8acc0cab70 2023-10-16T13:28:26.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.277+0000 7f8ad0e13700 1 -- 172.21.15.16:0/4101521103 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8acc1f5ee0 con 0x7f8acc0cab70 2023-10-16T13:28:26.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.278+0000 7f8ad0e13700 1 -- 172.21.15.16:0/4101521103 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8acc0b4f10 con 0x7f8acc0cab70 2023-10-16T13:28:26.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.279+0000 7f8ac2ffd700 1 -- 172.21.15.16:0/4101521103 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 107) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8abc012c60 con 0x7f8acc0cab70 2023-10-16T13:28:26.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.279+0000 7f8ac2ffd700 1 --2- 172.21.15.16:0/4101521103 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f8aac06b020 0x7f8aac06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:26.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.279+0000 7f8ac2ffd700 1 -- 172.21.15.16:0/4101521103 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f8abc089bd0 con 0x7f8acc0cab70 2023-10-16T13:28:26.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.279+0000 7f8aca59c700 1 --2- 172.21.15.16:0/4101521103 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f8aac06b020 0x7f8aac06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:26.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.280+0000 7f8aca59c700 1 --2- 172.21.15.16:0/4101521103 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f8aac06b020 0x7f8aac06d4e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8ab40090b0 tx=0x7f8ab4008030 comp rx=0 tx=0).ready entity=mgr.10047 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:26.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.284+0000 7f8ac2ffd700 1 -- 172.21.15.16:0/4101521103 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8abc0549d0 con 0x7f8acc0cab70 2023-10-16T13:28:26.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.534+0000 7f8ad0e13700 1 -- 172.21.15.16:0/4101521103 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f8acc04ebb0 con 0x7f8acc0cab70 2023-10-16T13:28:26.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.534+0000 7f8ac2ffd700 1 -- 172.21.15.16:0/4101521103 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v95) v1 ==== 53+0+73 (secure 0 0 0) 0x7f8abc057ff0 con 0x7f8acc0cab70 2023-10-16T13:28:26.537 INFO:teuthology.orchestra.run.smithi016.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-16T13:28:26.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.536+0000 7f8ad0e13700 1 -- 172.21.15.16:0/4101521103 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f8aac06b020 msgr2=0x7f8aac06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:26.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.536+0000 7f8ad0e13700 1 --2- 172.21.15.16:0/4101521103 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f8aac06b020 0x7f8aac06d4e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8ab40090b0 tx=0x7f8ab4008030 comp rx=0 tx=0).stop 2023-10-16T13:28:26.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.536+0000 7f8ad0e13700 1 -- 172.21.15.16:0/4101521103 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8acc0cab70 msgr2=0x7f8acc0c8dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:26.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.536+0000 7f8ad0e13700 1 --2- 172.21.15.16:0/4101521103 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8acc0cab70 0x7f8acc0c8dd0 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 crypto rx=0x7f8abc00d800 tx=0x7f8abc00db10 comp rx=0 tx=0).stop 2023-10-16T13:28:26.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.536+0000 7f8ad0e13700 1 -- 172.21.15.16:0/4101521103 shutdown_connections 2023-10-16T13:28:26.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.536+0000 7f8ad0e13700 1 --2- 172.21.15.16:0/4101521103 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8acc160eb0 0x7f8acc0c9310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:26.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.536+0000 7f8ad0e13700 1 --2- 172.21.15.16:0/4101521103 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f8aac06b020 0x7f8aac06d4e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:26.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.536+0000 7f8ad0e13700 1 --2- 172.21.15.16:0/4101521103 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8acc06b1f0 0x7f8acc1f59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:26.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.536+0000 7f8ad0e13700 1 --2- 172.21.15.16:0/4101521103 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8acc0cab70 0x7f8acc0c8dd0 unknown :-1 s=CLOSED pgs=2115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:26.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.536+0000 7f8ad0e13700 1 -- 172.21.15.16:0/4101521103 >> 172.21.15.16:0/4101521103 conn(0x7f8acc0cc5b0 msgr2=0x7f8acc160050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:26.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.536+0000 7f8ad0e13700 1 -- 172.21.15.16:0/4101521103 shutdown_connections 2023-10-16T13:28:26.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:26.536+0000 7f8ad0e13700 1 -- 172.21.15.16:0/4101521103 wait complete. 2023-10-16T13:28:26.551 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-16T13:28:26.551 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T13:28:26.552 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi016.front.sepia.ceph.com 2023-10-16T13:28:26.552 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-16T13:28:26.552 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-16T13:28:26.553 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-16T13:28:26.553 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-16T13:28:26.553 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T13:28:26.554 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T13:28:26.554 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:28:26.554 DEBUG:teuthology.orchestra.run.smithi016:> ip addr 2023-10-16T13:28:26.572 INFO:teuthology.orchestra.run.smithi016.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T13:28:26.572 INFO:teuthology.orchestra.run.smithi016.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T13:28:26.572 INFO:teuthology.orchestra.run.smithi016.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T13:28:26.573 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T13:28:26.573 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 ::1/128 scope host 2023-10-16T13:28:26.573 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T13:28:26.573 INFO:teuthology.orchestra.run.smithi016.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T13:28:26.574 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e0 brd ff:ff:ff:ff:ff:ff 2023-10-16T13:28:26.574 INFO:teuthology.orchestra.run.smithi016.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T13:28:26.574 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e1 brd ff:ff:ff:ff:ff:ff 2023-10-16T13:28:26.574 INFO:teuthology.orchestra.run.smithi016.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T13:28:26.574 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:64 brd ff:ff:ff:ff:ff:ff 2023-10-16T13:28:26.575 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.21.15.16/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-16T13:28:26.575 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft 37630sec preferred_lft 37630sec 2023-10-16T13:28:26.575 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::ec4:7aff:febd:1564/64 scope link 2023-10-16T13:28:26.575 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T13:28:26.575 INFO:teuthology.orchestra.run.smithi016.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T13:28:26.575 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:65 brd ff:ff:ff:ff:ff:ff 2023-10-16T13:28:26.576 INFO:teuthology.orchestra.run.smithi016.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T13:28:26.576 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 36:cf:ea:25:db:f7 brd ff:ff:ff:ff:ff:ff 2023-10-16T13:28:26.576 INFO:teuthology.orchestra.run.smithi016.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-16T13:28:26.576 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T13:28:26.576 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::cfc:c5ff:fe52:a786/64 scope link 2023-10-16T13:28:26.577 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T13:28:26.577 INFO:teuthology.orchestra.run.smithi016.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T13:28:26.577 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 6e:62:9d:a4:43:23 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T13:28:26.577 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::6c62:9dff:fea4:4323/64 scope link 2023-10-16T13:28:26.577 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T13:28:26.577 INFO:teuthology.orchestra.run.smithi016.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T13:28:26.578 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 36:cf:ea:25:db:f7 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-16T13:28:26.578 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::34cf:eaff:fe25:dbf7/64 scope link 2023-10-16T13:28:26.578 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T13:28:26.578 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:28:26.579 DEBUG:teuthology.orchestra.run.smithi016:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T13:28:26.638 INFO:teuthology.orchestra.run.smithi016.stdout:1 2023-10-16T13:28:26.640 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:28:26.640 DEBUG:teuthology.orchestra.run.smithi016:> route 2023-10-16T13:28:26.699 INFO:teuthology.orchestra.run.smithi016.stdout:Kernel IP routing table 2023-10-16T13:28:26.700 INFO:teuthology.orchestra.run.smithi016.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T13:28:26.700 INFO:teuthology.orchestra.run.smithi016.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-16T13:28:26.700 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-16T13:28:26.700 INFO:teuthology.orchestra.run.smithi016.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T13:28:26.701 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:28:26.702 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec bash -c ' 2023-10-16T13:28:26.702 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2023-10-16T13:28:26.702 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-16T13:28:26.703 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-16T13:28:26.703 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-16T13:28:26.703 DEBUG:teuthology.orchestra.run.smithi016:> ') 2023-10-16T13:28:26.808 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:28:26.809 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2023-10-16T13:28:26.826 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-16T13:28:26.826 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-16T13:28:26.826 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T13:28:26.827 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:28:26.827 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set brx.0 up 2023-10-16T13:28:26.893 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T13:28:26.894 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-16T13:28:26.950 INFO:teuthology.orchestra.run.smithi016.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-16T13:28:26.950 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T13:28:26.951 DEBUG:teuthology.orchestra.run.smithi016:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-16T13:28:27.006 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-16T13:28:27.007 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:28:27.007 DEBUG:teuthology.orchestra.run.smithi016:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T13:28:27.094 INFO:teuthology.orchestra.run.smithi016.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T13:28:27.094 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T13:28:27.095 INFO:teuthology.orchestra.run.smithi016.stdout:invalid new device string format 2023-10-16T13:28:27.104 INFO:teuthology.orchestra.run.smithi016.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T13:28:27.104 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T13:28:27.104 INFO:teuthology.orchestra.run.smithi016.stdout:invalid new device string format 2023-10-16T13:28:27.104 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: resolved to: "172.21.15.16:3300,172.21.15.49:3300,172.21.15.49:3301" 2023-10-16T13:28:27.104 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T13:28:27.177 INFO:teuthology.orchestra.run.smithi016.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T13:28:27.178 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T13:28:27.178 INFO:teuthology.orchestra.run.smithi016.stdout:invalid new device string format 2023-10-16T13:28:27.178 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: resolved to: "172.21.15.16:3300,172.21.15.49:3300,172.21.15.49:3301" 2023-10-16T13:28:27.178 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T13:28:27.179 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: resolved to: "172.21.15.16:6789,172.21.15.49:6789,172.21.15.49:6790" 2023-10-16T13:28:27.179 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.16:6789,172.21.15.49:6789,172.21.15.49:6790:/ 2023-10-16T13:28:27.179 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=5b28dfa1-b21f-474a-9b2e-9fb77af58254" will pass to kernel 2023-10-16T13:28:27.180 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-16T13:28:27.180 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:28:27.181 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-16T13:28:27.208 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:28:27.209 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T13:28:27.290 INFO:teuthology.orchestra.run.smithi016.stdout:client8402 2023-10-16T13:28:27.291 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T13:28:27.291 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:28:27.347 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-16T13:28:27.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.512+0000 7f4c83de6700 1 Processor -- start 2023-10-16T13:28:27.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.512+0000 7f4c83de6700 1 -- start start 2023-10-16T13:28:27.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.513+0000 7f4c83de6700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f4c7c0ce4f0 0x7f4c7c0d0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:27.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.513+0000 7f4c83de6700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4c7c15ba10 0x7f4c7c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:27.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.513+0000 7f4c83de6700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c7c0b0970 con 0x7f4c7c15b630 2023-10-16T13:28:27.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.513+0000 7f4c83de6700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c7c0cd690 con 0x7f4c7c0ce4f0 2023-10-16T13:28:27.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.513+0000 7f4c83de6700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c7c0cd970 con 0x7f4c7c15ba10 2023-10-16T13:28:27.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.513+0000 7f4c81b82700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f4c7c15b630 0x7f4c7c0cdde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:39298/0 (socket says 172.21.15.16:39298) 2023-10-16T13:28:27.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.513+0000 7f4c81b82700 1 -- 172.21.15.16:0/2784083266 learned_addr learned my addr 172.21.15.16:0/2784083266 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:28:27.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.513+0000 7f4c82383700 1 --2- 172.21.15.16:0/2784083266 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4c7c15ba10 0x7f4c7c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:27.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.513+0000 7f4c81381700 1 --2- 172.21.15.16:0/2784083266 >> v2:172.21.15.49:3300/0 conn(0x7f4c7c0ce4f0 0x7f4c7c0d0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:27.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.513+0000 7f4c80b80700 1 -- 172.21.15.16:0/2784083266 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1914975249 0 0) 0x7f4c7c0b0970 con 0x7f4c7c15b630 2023-10-16T13:28:27.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.513+0000 7f4c80b80700 1 -- 172.21.15.16:0/2784083266 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c680035b0 con 0x7f4c7c15b630 2023-10-16T13:28:27.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.513+0000 7f4c80b80700 1 -- 172.21.15.16:0/2784083266 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4068843038 0 0) 0x7f4c680035b0 con 0x7f4c7c15b630 2023-10-16T13:28:27.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.513+0000 7f4c80b80700 1 -- 172.21.15.16:0/2784083266 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4c7c15ba10 msgr2=0x7f4c7c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:28:27.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.513+0000 7f4c80b80700 1 --2- 172.21.15.16:0/2784083266 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4c7c15ba10 0x7f4c7c0cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:27.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.513+0000 7f4c80b80700 1 -- 172.21.15.16:0/2784083266 >> v2:172.21.15.49:3300/0 conn(0x7f4c7c0ce4f0 msgr2=0x7f4c7c0d0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:28:27.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.513+0000 7f4c80b80700 1 --2- 172.21.15.16:0/2784083266 >> v2:172.21.15.49:3300/0 conn(0x7f4c7c0ce4f0 0x7f4c7c0d0380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:27.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.513+0000 7f4c80b80700 1 -- 172.21.15.16:0/2784083266 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c7c0b0c30 con 0x7f4c7c15b630 2023-10-16T13:28:27.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.513+0000 7f4c80b80700 1 -- 172.21.15.16:0/2784083266 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f4c78001670 con 0x7f4c7c15b630 2023-10-16T13:28:27.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.513+0000 7f4c80b80700 1 -- 172.21.15.16:0/2784083266 >> v1:172.21.15.16:6789/0 conn(0x7f4c7c15b630 legacy=0x7f4c7c0cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:27.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.513+0000 7f4c80b80700 1 --2- 172.21.15.16:0/2784083266 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4c7c15ba10 0x7f4c68003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:27.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.513+0000 7f4c80b80700 1 --2- 172.21.15.16:0/2784083266 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4c68003f80 0x7f4c68004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:27.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.513+0000 7f4c80b80700 1 --2- 172.21.15.16:0/2784083266 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c68004d90 0x7f4c68005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:27.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.513+0000 7f4c80b80700 1 -- 172.21.15.16:0/2784083266 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c7c0cd690 con 0x7f4c68004d90 2023-10-16T13:28:27.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.513+0000 7f4c80b80700 1 -- 172.21.15.16:0/2784083266 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c7c0cd970 con 0x7f4c68003f80 2023-10-16T13:28:27.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.513+0000 7f4c80b80700 1 -- 172.21.15.16:0/2784083266 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c6800ba10 con 0x7f4c7c15ba10 2023-10-16T13:28:27.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.514+0000 7f4c82383700 1 --2- 172.21.15.16:0/2784083266 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c68004d90 0x7f4c68005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:27.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.514+0000 7f4c81b82700 1 --2- 172.21.15.16:0/2784083266 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4c7c15ba10 0x7f4c68003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:27.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.514+0000 7f4c81381700 1 --2- 172.21.15.16:0/2784083266 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4c68003f80 0x7f4c68004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:27.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.514+0000 7f4c82383700 1 -- 172.21.15.16:0/2784083266 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4c7c15ba10 msgr2=0x7f4c68003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:28:27.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.514+0000 7f4c82383700 1 --2- 172.21.15.16:0/2784083266 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4c7c15ba10 0x7f4c68003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:27.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.514+0000 7f4c82383700 1 -- 172.21.15.16:0/2784083266 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4c68003f80 msgr2=0x7f4c68004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:28:27.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.514+0000 7f4c82383700 1 --2- 172.21.15.16:0/2784083266 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4c68003f80 0x7f4c68004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:27.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.514+0000 7f4c82383700 1 -- 172.21.15.16:0/2784083266 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c680035b0 con 0x7f4c68004d90 2023-10-16T13:28:27.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.514+0000 7f4c82383700 1 --2- 172.21.15.16:0/2784083266 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c68004d90 0x7f4c68005210 secure :-1 s=READY pgs=2117 cs=0 l=1 rev1=1 crypto rx=0x7f4c74009fc0 tx=0x7f4c68005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22af5e91c416716f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:27.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.514+0000 7f4c80b80700 1 -- 172.21.15.16:0/2784083266 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c7400ebd0 con 0x7f4c68004d90 2023-10-16T13:28:27.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.514+0000 7f4c80b80700 1 -- 172.21.15.16:0/2784083266 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c7400f040 con 0x7f4c68004d90 2023-10-16T13:28:27.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.514+0000 7f4c80b80700 1 -- 172.21.15.16:0/2784083266 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c74014480 con 0x7f4c68004d90 2023-10-16T13:28:27.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.515+0000 7f4c83de6700 1 -- 172.21.15.16:0/2784083266 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c68004d90 msgr2=0x7f4c68005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:27.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.515+0000 7f4c83de6700 1 --2- 172.21.15.16:0/2784083266 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c68004d90 0x7f4c68005210 secure :-1 s=READY pgs=2117 cs=0 l=1 rev1=1 crypto rx=0x7f4c74009fc0 tx=0x7f4c68005880 comp rx=0 tx=0).stop 2023-10-16T13:28:27.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.515+0000 7f4c82383700 1 -- 172.21.15.16:0/2784083266 reap_dead start 2023-10-16T13:28:27.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.515+0000 7f4c83de6700 1 -- 172.21.15.16:0/2784083266 shutdown_connections 2023-10-16T13:28:27.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.515+0000 7f4c83de6700 1 -- 172.21.15.16:0/2784083266 >> 172.21.15.16:0/2784083266 conn(0x7f4c7c0d14b0 msgr2=0x7f4c7c0d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:27.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.515+0000 7f4c83de6700 1 -- 172.21.15.16:0/2784083266 shutdown_connections 2023-10-16T13:28:27.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.515+0000 7f4c83de6700 1 -- 172.21.15.16:0/2784083266 wait complete. 2023-10-16T13:28:27.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.516+0000 7f4c83de6700 1 Processor -- start 2023-10-16T13:28:27.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.516+0000 7f4c83de6700 1 -- start start 2023-10-16T13:28:27.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.517+0000 7f4c83de6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c68004d90 0x7f4c7c15e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:27.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.517+0000 7f4c83de6700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4c7c1514b0 0x7f4c7c0d1b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:27.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.517+0000 7f4c83de6700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4c7c14fea0 0x7f4c7c150320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:27.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.517+0000 7f4c83de6700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c7c15bfe0 con 0x7f4c68004d90 2023-10-16T13:28:27.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.517+0000 7f4c83de6700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c7c15c2c0 con 0x7f4c7c14fea0 2023-10-16T13:28:27.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.517+0000 7f4c83de6700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c7c150860 con 0x7f4c7c1514b0 2023-10-16T13:28:27.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.517+0000 7f4c81b82700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c68004d90 0x7f4c7c15e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:27.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.517+0000 7f4c81b82700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c68004d90 0x7f4c7c15e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50036/0 (socket says 172.21.15.16:50036) 2023-10-16T13:28:27.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.517+0000 7f4c81b82700 1 -- 172.21.15.16:0/774728158 learned_addr learned my addr 172.21.15.16:0/774728158 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:27.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.517+0000 7f4c81381700 1 --2- 172.21.15.16:0/774728158 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4c7c1514b0 0x7f4c7c0d1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:27.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.517+0000 7f4c82383700 1 --2- 172.21.15.16:0/774728158 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4c7c14fea0 0x7f4c7c150320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:27.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.517+0000 7f4c81b82700 1 -- 172.21.15.16:0/774728158 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4c7c1514b0 msgr2=0x7f4c7c0d1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:27.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.517+0000 7f4c81b82700 1 --2- 172.21.15.16:0/774728158 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4c7c1514b0 0x7f4c7c0d1b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:27.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.517+0000 7f4c81b82700 1 -- 172.21.15.16:0/774728158 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4c7c14fea0 msgr2=0x7f4c7c150320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:27.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.517+0000 7f4c81b82700 1 --2- 172.21.15.16:0/774728158 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4c7c14fea0 0x7f4c7c150320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:27.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.517+0000 7f4c81b82700 1 -- 172.21.15.16:0/774728158 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c7c1538b0 con 0x7f4c68004d90 2023-10-16T13:28:27.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.517+0000 7f4c81b82700 1 --2- 172.21.15.16:0/774728158 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c68004d90 0x7f4c7c15e700 secure :-1 s=READY pgs=2118 cs=0 l=1 rev1=1 crypto rx=0x7f4c78007340 tx=0x7f4c78001760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:27.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.517+0000 7f4c727fc700 1 -- 172.21.15.16:0/774728158 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c7800f6c0 con 0x7f4c68004d90 2023-10-16T13:28:27.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.517+0000 7f4c727fc700 1 -- 172.21.15.16:0/774728158 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c7800f820 con 0x7f4c68004d90 2023-10-16T13:28:27.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.517+0000 7f4c83de6700 1 -- 172.21.15.16:0/774728158 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c7c153b90 con 0x7f4c68004d90 2023-10-16T13:28:27.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.517+0000 7f4c727fc700 1 -- 172.21.15.16:0/774728158 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c7800fbd0 con 0x7f4c68004d90 2023-10-16T13:28:27.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.518+0000 7f4c83de6700 1 -- 172.21.15.16:0/774728158 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c7c15ec40 con 0x7f4c68004d90 2023-10-16T13:28:27.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.518+0000 7f4c83de6700 1 -- 172.21.15.16:0/774728158 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c7c1540b0 con 0x7f4c68004d90 2023-10-16T13:28:27.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.519+0000 7f4c727fc700 1 -- 172.21.15.16:0/774728158 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 107) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4c7801d360 con 0x7f4c68004d90 2023-10-16T13:28:27.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.519+0000 7f4c727fc700 1 --2- 172.21.15.16:0/774728158 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f4c6006b020 0x7f4c6006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:27.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.520+0000 7f4c81381700 1 --2- 172.21.15.16:0/774728158 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f4c6006b020 0x7f4c6006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:27.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.520+0000 7f4c727fc700 1 -- 172.21.15.16:0/774728158 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f4c78089260 con 0x7f4c68004d90 2023-10-16T13:28:27.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.520+0000 7f4c81381700 1 --2- 172.21.15.16:0/774728158 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f4c6006b020 0x7f4c6006d4e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4c7c1f4ad0 tx=0x7f4c6c009210 comp rx=0 tx=0).ready entity=mgr.10047 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:27.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.524+0000 7f4c727fc700 1 -- 172.21.15.16:0/774728158 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4c78054050 con 0x7f4c68004d90 2023-10-16T13:28:27.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.758+0000 7f4c83de6700 1 -- 172.21.15.16:0/774728158 --> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f4c7c0b0f10 con 0x7f4c6006b020 2023-10-16T13:28:27.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.759+0000 7f4c727fc700 1 -- 172.21.15.16:0/774728158 <== mgr.10047 v2:172.21.15.16:6832/4211329181 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f4c6c019070 con 0x7f4c6006b020 2023-10-16T13:28:27.762 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2023-10-16T13:28:27.762 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:28:27.762 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cephfs" 2023-10-16T13:28:27.763 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:28:27.763 INFO:teuthology.orchestra.run.smithi016.stdout:] 2023-10-16T13:28:27.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.761+0000 7f4c83de6700 1 -- 172.21.15.16:0/774728158 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f4c6006b020 msgr2=0x7f4c6006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:27.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.761+0000 7f4c83de6700 1 --2- 172.21.15.16:0/774728158 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f4c6006b020 0x7f4c6006d4e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4c7c1f4ad0 tx=0x7f4c6c009210 comp rx=0 tx=0).stop 2023-10-16T13:28:27.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.761+0000 7f4c83de6700 1 -- 172.21.15.16:0/774728158 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c68004d90 msgr2=0x7f4c7c15e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:27.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.761+0000 7f4c83de6700 1 --2- 172.21.15.16:0/774728158 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c68004d90 0x7f4c7c15e700 secure :-1 s=READY pgs=2118 cs=0 l=1 rev1=1 crypto rx=0x7f4c78007340 tx=0x7f4c78001760 comp rx=0 tx=0).stop 2023-10-16T13:28:27.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.762+0000 7f4c83de6700 1 -- 172.21.15.16:0/774728158 shutdown_connections 2023-10-16T13:28:27.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.762+0000 7f4c83de6700 1 --2- 172.21.15.16:0/774728158 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f4c6006b020 0x7f4c6006d4e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:27.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.762+0000 7f4c83de6700 1 --2- 172.21.15.16:0/774728158 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4c7c14fea0 0x7f4c7c150320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:27.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.762+0000 7f4c83de6700 1 --2- 172.21.15.16:0/774728158 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c68004d90 0x7f4c7c15e700 unknown :-1 s=CLOSED pgs=2118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:27.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.762+0000 7f4c83de6700 1 --2- 172.21.15.16:0/774728158 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4c7c1514b0 0x7f4c7c0d1b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:27.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.762+0000 7f4c83de6700 1 -- 172.21.15.16:0/774728158 >> 172.21.15.16:0/774728158 conn(0x7f4c7c0d14b0 msgr2=0x7f4c7c0c7dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:27.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.762+0000 7f4c83de6700 1 -- 172.21.15.16:0/774728158 shutdown_connections 2023-10-16T13:28:27.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.762+0000 7f4c83de6700 1 -- 172.21.15.16:0/774728158 wait complete. 2023-10-16T13:28:27.776 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-16T13:28:27.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.901+0000 7f79ffe7e700 1 Processor -- start 2023-10-16T13:28:27.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.901+0000 7f79ffe7e700 1 -- start start 2023-10-16T13:28:27.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.902+0000 7f79ffe7e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f79f80d2110 0x7f79f80ae1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:27.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.902+0000 7f79ffe7e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f79f806ca10 0x7f79f80ae720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:27.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.902+0000 7f79ffe7e700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79f80b20f0 con 0x7f79f80ce6a0 2023-10-16T13:28:27.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.902+0000 7f79ffe7e700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79f80d03d0 con 0x7f79f806ca10 2023-10-16T13:28:27.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.902+0000 7f79ffe7e700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79f80b2340 con 0x7f79f80d2110 2023-10-16T13:28:27.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.902+0000 7f79fe41b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f79f806ca10 0x7f79f80ae720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:27.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.902+0000 7f79fe41b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f79f806ca10 0x7f79f80ae720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50048/0 (socket says 172.21.15.16:50048) 2023-10-16T13:28:27.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.902+0000 7f79fe41b700 1 -- 172.21.15.16:0/2415685157 learned_addr learned my addr 172.21.15.16:0/2415685157 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:27.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.903+0000 7f79fd419700 1 --2- 172.21.15.16:0/2415685157 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f79f80d2110 0x7f79f80ae1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:27.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.903+0000 7f79fe41b700 1 -- 172.21.15.16:0/2415685157 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f79f80d2110 msgr2=0x7f79f80ae1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:28:27.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.903+0000 7f79fe41b700 1 --2- 172.21.15.16:0/2415685157 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f79f80d2110 0x7f79f80ae1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:27.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.903+0000 7f79fe41b700 1 -- 172.21.15.16:0/2415685157 >> v1:172.21.15.16:6789/0 conn(0x7f79f80ce6a0 legacy=0x7f79f806c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:27.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.903+0000 7f79fe41b700 1 -- 172.21.15.16:0/2415685157 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79f806bb00 con 0x7f79f806ca10 2023-10-16T13:28:27.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.903+0000 7f79fd419700 1 --2- 172.21.15.16:0/2415685157 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f79f80d2110 0x7f79f80ae1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:28:27.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.903+0000 7f79fe41b700 1 --2- 172.21.15.16:0/2415685157 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79f806ca10 0x7f79f80ae720 secure :-1 s=READY pgs=2120 cs=0 l=1 rev1=1 crypto rx=0x7f79f0009e50 tx=0x7f79f000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65abee49dea42e47 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:27.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.903+0000 7f79fcc18700 1 -- 172.21.15.16:0/2415685157 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79f000e070 con 0x7f79f806ca10 2023-10-16T13:28:27.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.903+0000 7f79fcc18700 1 -- 172.21.15.16:0/2415685157 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79f000b9c0 con 0x7f79f806ca10 2023-10-16T13:28:27.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.903+0000 7f79fcc18700 1 -- 172.21.15.16:0/2415685157 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79f000bd70 con 0x7f79f806ca10 2023-10-16T13:28:27.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.903+0000 7f79ffe7e700 1 -- 172.21.15.16:0/2415685157 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79f806ca10 msgr2=0x7f79f80ae720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:27.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.903+0000 7f79ffe7e700 1 --2- 172.21.15.16:0/2415685157 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79f806ca10 0x7f79f80ae720 secure :-1 s=READY pgs=2120 cs=0 l=1 rev1=1 crypto rx=0x7f79f0009e50 tx=0x7f79f000b330 comp rx=0 tx=0).stop 2023-10-16T13:28:27.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.903+0000 7f79ffe7e700 1 -- 172.21.15.16:0/2415685157 shutdown_connections 2023-10-16T13:28:27.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.903+0000 7f79ffe7e700 1 --2- 172.21.15.16:0/2415685157 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79f806ca10 0x7f79f80ae720 unknown :-1 s=CLOSED pgs=2120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:27.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.903+0000 7f79ffe7e700 1 --2- 172.21.15.16:0/2415685157 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f79f80d2110 0x7f79f80ae1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:27.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.903+0000 7f79ffe7e700 1 -- 172.21.15.16:0/2415685157 >> 172.21.15.16:0/2415685157 conn(0x7f79f80d15f0 msgr2=0x7f79f80d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:27.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.904+0000 7f79ffe7e700 1 -- 172.21.15.16:0/2415685157 shutdown_connections 2023-10-16T13:28:27.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.904+0000 7f79ffe7e700 1 -- 172.21.15.16:0/2415685157 wait complete. 2023-10-16T13:28:27.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.905+0000 7f79ffe7e700 1 Processor -- start 2023-10-16T13:28:27.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.905+0000 7f79ffe7e700 1 -- start start 2023-10-16T13:28:27.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.906+0000 7f79ffe7e700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f79f806ca10 0x7f79f81f6de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:27.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.906+0000 7f79ffe7e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79f80ce6a0 0x7f79f80b4630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:27.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.906+0000 7f79ffe7e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f79f80d2110 0x7f79f80af4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:27.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.906+0000 7f79ffe7e700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79f80b4d30 con 0x7f79f80ce6a0 2023-10-16T13:28:27.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.906+0000 7f79ffe7e700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79f80b5010 con 0x7f79f806ca10 2023-10-16T13:28:27.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.906+0000 7f79ffe7e700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79f80b52f0 con 0x7f79f80d2110 2023-10-16T13:28:27.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.906+0000 7f79fd419700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79f80ce6a0 0x7f79f80b4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:27.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.906+0000 7f79fd419700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79f80ce6a0 0x7f79f80b4630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50052/0 (socket says 172.21.15.16:50052) 2023-10-16T13:28:27.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.906+0000 7f79fd419700 1 -- 172.21.15.16:0/1769157098 learned_addr learned my addr 172.21.15.16:0/1769157098 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:27.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.906+0000 7f79fe41b700 1 --2- 172.21.15.16:0/1769157098 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f79f80d2110 0x7f79f80af4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:27.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.906+0000 7f79fdc1a700 1 --2- 172.21.15.16:0/1769157098 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f79f806ca10 0x7f79f81f6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:27.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.906+0000 7f79fd419700 1 -- 172.21.15.16:0/1769157098 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f79f80d2110 msgr2=0x7f79f80af4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:27.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.906+0000 7f79fd419700 1 --2- 172.21.15.16:0/1769157098 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f79f80d2110 0x7f79f80af4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:27.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.906+0000 7f79fd419700 1 -- 172.21.15.16:0/1769157098 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f79f806ca10 msgr2=0x7f79f81f6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:27.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.906+0000 7f79fd419700 1 --2- 172.21.15.16:0/1769157098 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f79f806ca10 0x7f79f81f6de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:27.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.906+0000 7f79fd419700 1 -- 172.21.15.16:0/1769157098 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79f0009b30 con 0x7f79f80ce6a0 2023-10-16T13:28:27.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.906+0000 7f79fe41b700 1 --2- 172.21.15.16:0/1769157098 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f79f80d2110 0x7f79f80af4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:28:27.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.906+0000 7f79fd419700 1 --2- 172.21.15.16:0/1769157098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79f80ce6a0 0x7f79f80b4630 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 crypto rx=0x7f79f400c7f0 tx=0x7f79f400cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:27.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.906+0000 7f79ee7fc700 1 -- 172.21.15.16:0/1769157098 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79f4011800 con 0x7f79f80ce6a0 2023-10-16T13:28:27.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.906+0000 7f79ee7fc700 1 -- 172.21.15.16:0/1769157098 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79f4011960 con 0x7f79f80ce6a0 2023-10-16T13:28:27.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.906+0000 7f79ffe7e700 1 -- 172.21.15.16:0/1769157098 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79f80b55d0 con 0x7f79f80ce6a0 2023-10-16T13:28:27.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.906+0000 7f79ee7fc700 1 -- 172.21.15.16:0/1769157098 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79f4011d10 con 0x7f79f80ce6a0 2023-10-16T13:28:27.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.906+0000 7f79ffe7e700 1 -- 172.21.15.16:0/1769157098 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79f80af9e0 con 0x7f79f80ce6a0 2023-10-16T13:28:27.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.907+0000 7f79ffe7e700 1 -- 172.21.15.16:0/1769157098 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79f80c6950 con 0x7f79f80ce6a0 2023-10-16T13:28:27.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.908+0000 7f79ee7fc700 1 -- 172.21.15.16:0/1769157098 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 107) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f79f4018360 con 0x7f79f80ce6a0 2023-10-16T13:28:27.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.908+0000 7f79ee7fc700 1 --2- 172.21.15.16:0/1769157098 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f79dc06b020 0x7f79dc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:27.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.908+0000 7f79ee7fc700 1 -- 172.21.15.16:0/1769157098 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f79f4089920 con 0x7f79f80ce6a0 2023-10-16T13:28:27.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.908+0000 7f79fdc1a700 1 --2- 172.21.15.16:0/1769157098 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f79dc06b020 0x7f79dc06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:27.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.909+0000 7f79fdc1a700 1 --2- 172.21.15.16:0/1769157098 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f79dc06b020 0x7f79dc06d4e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f79e8000f60 tx=0x7f79e8009040 comp rx=0 tx=0).ready entity=mgr.10047 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:27.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:27.913+0000 7f79ee7fc700 1 -- 172.21.15.16:0/1769157098 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f79f4054690 con 0x7f79f80ce6a0 2023-10-16T13:28:28.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:28.181+0000 7f79ffe7e700 1 -- 172.21.15.16:0/1769157098 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f79f804ebb0 con 0x7f79f80ce6a0 2023-10-16T13:28:29.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.147+0000 7f79ee7fc700 1 -- 172.21.15.16:0/1769157098 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v108) v1 ==== 92+0+0 (secure 0 0 0) 0x7f79f401f0a0 con 0x7f79f80ce6a0 2023-10-16T13:28:29.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.151+0000 7f79ffe7e700 1 -- 172.21.15.16:0/1769157098 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f79dc06b020 msgr2=0x7f79dc06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:29.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.151+0000 7f79ffe7e700 1 --2- 172.21.15.16:0/1769157098 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f79dc06b020 0x7f79dc06d4e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f79e8000f60 tx=0x7f79e8009040 comp rx=0 tx=0).stop 2023-10-16T13:28:29.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.151+0000 7f79ffe7e700 1 -- 172.21.15.16:0/1769157098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79f80ce6a0 msgr2=0x7f79f80b4630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:29.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.151+0000 7f79ffe7e700 1 --2- 172.21.15.16:0/1769157098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79f80ce6a0 0x7f79f80b4630 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 crypto rx=0x7f79f400c7f0 tx=0x7f79f400cbb0 comp rx=0 tx=0).stop 2023-10-16T13:28:29.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.152+0000 7f79ffe7e700 1 -- 172.21.15.16:0/1769157098 shutdown_connections 2023-10-16T13:28:29.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.152+0000 7f79ffe7e700 1 --2- 172.21.15.16:0/1769157098 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f79f80d2110 0x7f79f80af4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:29.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.152+0000 7f79ffe7e700 1 --2- 172.21.15.16:0/1769157098 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f79dc06b020 0x7f79dc06d4e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:29.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.152+0000 7f79ffe7e700 1 --2- 172.21.15.16:0/1769157098 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f79f806ca10 0x7f79f81f6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:29.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.152+0000 7f79ffe7e700 1 --2- 172.21.15.16:0/1769157098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79f80ce6a0 0x7f79f80b4630 unknown :-1 s=CLOSED pgs=2121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:29.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.152+0000 7f79ffe7e700 1 -- 172.21.15.16:0/1769157098 >> 172.21.15.16:0/1769157098 conn(0x7f79f80d15f0 msgr2=0x7f79f80b0f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:29.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.153+0000 7f79ffe7e700 1 -- 172.21.15.16:0/1769157098 shutdown_connections 2023-10-16T13:28:29.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.153+0000 7f79ffe7e700 1 -- 172.21.15.16:0/1769157098 wait complete. 2023-10-16T13:28:29.171 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-16T13:28:29.302 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:28:29.299+0000 7f9f8a5b7200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T13:28:29.310 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:28:29.308+0000 7f6d18dfa200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T13:28:29.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.308+0000 7f5886b1e700 1 Processor -- start 2023-10-16T13:28:29.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.308+0000 7f5886b1e700 1 -- start start 2023-10-16T13:28:29.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.309+0000 7f5886b1e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f58800af210 0x7f58800ce180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:29.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.309+0000 7f5886b1e700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f588006aa10 con 0x7f58800ce6c0 2023-10-16T13:28:29.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.309+0000 7f5886b1e700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58800ceed0 con 0x7f5880160ee0 2023-10-16T13:28:29.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.309+0000 7f5886b1e700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58800cf770 con 0x7f58800af210 2023-10-16T13:28:29.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.309+0000 7f587ffff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f58800af210 0x7f58800ce180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:29.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.309+0000 7f587ffff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f58800af210 0x7f58800ce180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50086/0 (socket says 172.21.15.16:50086) 2023-10-16T13:28:29.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.309+0000 7f58850bb700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f58800ce6c0 0x7f588006aea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:39362/0 (socket says 172.21.15.16:39362) 2023-10-16T13:28:29.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.309+0000 7f587ffff700 1 -- 172.21.15.16:0/2929986818 learned_addr learned my addr 172.21.15.16:0/2929986818 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:29.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.309+0000 7f587ffff700 1 -- 172.21.15.16:0/2929986818 >> v1:172.21.15.49:6789/0 conn(0x7f5880160ee0 legacy=0x7f58800aeb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:29.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.309+0000 7f587ffff700 1 -- 172.21.15.16:0/2929986818 >> v1:172.21.15.16:6789/0 conn(0x7f58800ce6c0 legacy=0x7f588006aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:29.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.309+0000 7f587ffff700 1 -- 172.21.15.16:0/2929986818 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58800cf1b0 con 0x7f58800af210 2023-10-16T13:28:29.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.310+0000 7f587ffff700 1 --2- 172.21.15.16:0/2929986818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58800af210 0x7f58800ce180 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 crypto rx=0x7f586c0098f0 tx=0x7f586c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f88cd5b16a136ae5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:29.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.310+0000 7f587f7fe700 1 -- 172.21.15.16:0/2929986818 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f586c00b8a0 con 0x7f58800af210 2023-10-16T13:28:29.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.310+0000 7f587f7fe700 1 -- 172.21.15.16:0/2929986818 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f586c00ba00 con 0x7f58800af210 2023-10-16T13:28:29.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.310+0000 7f587f7fe700 1 -- 172.21.15.16:0/2929986818 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f586c00bd90 con 0x7f58800af210 2023-10-16T13:28:29.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.310+0000 7f5886b1e700 1 -- 172.21.15.16:0/2929986818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58800af210 msgr2=0x7f58800ce180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:29.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.310+0000 7f5886b1e700 1 --2- 172.21.15.16:0/2929986818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58800af210 0x7f58800ce180 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 crypto rx=0x7f586c0098f0 tx=0x7f586c009c00 comp rx=0 tx=0).stop 2023-10-16T13:28:29.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.310+0000 7f5886b1e700 1 -- 172.21.15.16:0/2929986818 shutdown_connections 2023-10-16T13:28:29.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.310+0000 7f5886b1e700 1 --2- 172.21.15.16:0/2929986818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58800af210 0x7f58800ce180 unknown :-1 s=CLOSED pgs=2127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:29.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.310+0000 7f5886b1e700 1 -- 172.21.15.16:0/2929986818 >> 172.21.15.16:0/2929986818 conn(0x7f58800cc5e0 msgr2=0x7f58800cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:29.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.310+0000 7f5886b1e700 1 -- 172.21.15.16:0/2929986818 shutdown_connections 2023-10-16T13:28:29.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.310+0000 7f5886b1e700 1 -- 172.21.15.16:0/2929986818 wait complete. 2023-10-16T13:28:29.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.312+0000 7f5886b1e700 1 Processor -- start 2023-10-16T13:28:29.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.312+0000 7f5886b1e700 1 -- start start 2023-10-16T13:28:29.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.312+0000 7f5886b1e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58800af210 0x7f58801f7310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:29.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.312+0000 7f5886b1e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f58800ce6c0 0x7f58801f3310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:29.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.313+0000 7f5886b1e700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5880160ee0 0x7f588014d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:29.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.313+0000 7f5886b1e700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58801f7850 con 0x7f58800af210 2023-10-16T13:28:29.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.313+0000 7f5886b1e700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58801f3850 con 0x7f5880160ee0 2023-10-16T13:28:29.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.313+0000 7f5886b1e700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58801f3a80 con 0x7f58800ce6c0 2023-10-16T13:28:29.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.313+0000 7f58848ba700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58800af210 0x7f58801f7310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:29.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.313+0000 7f58848ba700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58800af210 0x7f58801f7310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50090/0 (socket says 172.21.15.16:50090) 2023-10-16T13:28:29.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.313+0000 7f58848ba700 1 -- 172.21.15.16:0/661447200 learned_addr learned my addr 172.21.15.16:0/661447200 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:29.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.313+0000 7f58850bb700 1 --2- 172.21.15.16:0/661447200 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5880160ee0 0x7f588014d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:29.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.313+0000 7f587ffff700 1 --2- 172.21.15.16:0/661447200 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f58800ce6c0 0x7f58801f3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:29.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.313+0000 7f58848ba700 1 -- 172.21.15.16:0/661447200 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f58800ce6c0 msgr2=0x7f58801f3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:29.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.313+0000 7f58848ba700 1 --2- 172.21.15.16:0/661447200 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f58800ce6c0 0x7f58801f3310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:29.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.313+0000 7f58848ba700 1 -- 172.21.15.16:0/661447200 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5880160ee0 msgr2=0x7f588014d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:29.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.313+0000 7f58848ba700 1 --2- 172.21.15.16:0/661447200 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5880160ee0 0x7f588014d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:29.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.313+0000 7f58848ba700 1 -- 172.21.15.16:0/661447200 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f586c009550 con 0x7f58800af210 2023-10-16T13:28:29.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.313+0000 7f58848ba700 1 --2- 172.21.15.16:0/661447200 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58800af210 0x7f58801f7310 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 crypto rx=0x7f587400c490 tx=0x7f587400c7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:29.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.313+0000 7f587d7fa700 1 -- 172.21.15.16:0/661447200 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f587400cea0 con 0x7f58800af210 2023-10-16T13:28:29.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.313+0000 7f587d7fa700 1 -- 172.21.15.16:0/661447200 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58740052f0 con 0x7f58800af210 2023-10-16T13:28:29.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.313+0000 7f5886b1e700 1 -- 172.21.15.16:0/661447200 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58801f3d60 con 0x7f58800af210 2023-10-16T13:28:29.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.313+0000 7f587d7fa700 1 -- 172.21.15.16:0/661447200 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5874006860 con 0x7f58800af210 2023-10-16T13:28:29.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.313+0000 7f5886b1e700 1 -- 172.21.15.16:0/661447200 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f588014cfb0 con 0x7f58800af210 2023-10-16T13:28:29.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.314+0000 7f5886b1e700 1 -- 172.21.15.16:0/661447200 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58801f4280 con 0x7f58800af210 2023-10-16T13:28:29.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.315+0000 7f587d7fa700 1 -- 172.21.15.16:0/661447200 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 108) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f5874003680 con 0x7f58800af210 2023-10-16T13:28:29.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.315+0000 7f587d7fa700 1 --2- 172.21.15.16:0/661447200 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f586406b070 0x7f586406d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:29.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.315+0000 7f587ffff700 1 -- 172.21.15.16:0/661447200 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f586406b070 msgr2=0x7f586406d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/4211329181 2023-10-16T13:28:29.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.315+0000 7f587d7fa700 1 -- 172.21.15.16:0/661447200 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f5874088b20 con 0x7f58800af210 2023-10-16T13:28:29.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.315+0000 7f587ffff700 1 --2- 172.21.15.16:0/661447200 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f586406b070 0x7f586406d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:28:29.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.320+0000 7f587d7fa700 1 -- 172.21.15.16:0/661447200 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5874053910 con 0x7f58800af210 2023-10-16T13:28:29.371 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:28:29.368+0000 7f9f8a5b7200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T13:28:29.382 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:28:29.380+0000 7f6d18dfa200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T13:28:29.446 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:28:29.443+0000 7f9f8a5b7200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T13:28:29.458 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:28:29.456+0000 7f6d18dfa200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T13:28:29.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.516+0000 7f587ffff700 1 -- 172.21.15.16:0/661447200 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f586406b070 msgr2=0x7f586406d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/4211329181 2023-10-16T13:28:29.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.516+0000 7f587ffff700 1 --2- 172.21.15.16:0/661447200 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f586406b070 0x7f586406d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:28:29.527 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:28:29.525+0000 7f9f8a5b7200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T13:28:29.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.532+0000 7f5886b1e700 1 -- 172.21.15.16:0/661447200 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f588004ebb0 con 0x7f58800af210 2023-10-16T13:28:29.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.535+0000 7f587d7fa700 1 -- 172.21.15.16:0/661447200 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v65)=0 v65) v1 ==== 138+0+0 (secure 0 0 0) 0x7f5874056f30 con 0x7f58800af210 2023-10-16T13:28:29.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.538+0000 7f5886b1e700 1 -- 172.21.15.16:0/661447200 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f586406b070 msgr2=0x7f586406d530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:28:29.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.538+0000 7f5886b1e700 1 --2- 172.21.15.16:0/661447200 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f586406b070 0x7f586406d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:29.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.538+0000 7f5886b1e700 1 -- 172.21.15.16:0/661447200 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58800af210 msgr2=0x7f58801f7310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:29.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.538+0000 7f5886b1e700 1 --2- 172.21.15.16:0/661447200 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58800af210 0x7f58801f7310 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 crypto rx=0x7f587400c490 tx=0x7f587400c7a0 comp rx=0 tx=0).stop 2023-10-16T13:28:29.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.538+0000 7f5886b1e700 1 -- 172.21.15.16:0/661447200 shutdown_connections 2023-10-16T13:28:29.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.538+0000 7f5886b1e700 1 --2- 172.21.15.16:0/661447200 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f586406b070 0x7f586406d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:29.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.538+0000 7f5886b1e700 1 --2- 172.21.15.16:0/661447200 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5880160ee0 0x7f588014d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:29.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.538+0000 7f5886b1e700 1 --2- 172.21.15.16:0/661447200 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58800af210 0x7f58801f7310 unknown :-1 s=CLOSED pgs=2128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:29.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.538+0000 7f5886b1e700 1 --2- 172.21.15.16:0/661447200 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f58800ce6c0 0x7f58801f3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:29.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.538+0000 7f5886b1e700 1 -- 172.21.15.16:0/661447200 >> 172.21.15.16:0/661447200 conn(0x7f58800cc5e0 msgr2=0x7f58800cad70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:29.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.538+0000 7f5886b1e700 1 -- 172.21.15.16:0/661447200 shutdown_connections 2023-10-16T13:28:29.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.538+0000 7f5886b1e700 1 -- 172.21.15.16:0/661447200 wait complete. 2023-10-16T13:28:29.543 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:28:29.540+0000 7f6d18dfa200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T13:28:29.553 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-16T13:28:29.676 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:28:29.673+0000 7f9f8a5b7200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T13:28:29.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.683+0000 7f00aab18700 1 Processor -- start 2023-10-16T13:28:29.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.683+0000 7f00aab18700 1 -- start start 2023-10-16T13:28:29.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.684+0000 7f00aab18700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f00a415da10 0x7f00a40d1520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:29.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.684+0000 7f00aab18700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f00a40d1d20 0x7f00a40d21b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:29.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.684+0000 7f00aab18700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00a40d4540 con 0x7f00a40d4060 2023-10-16T13:28:29.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.684+0000 7f00aab18700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00a40cf9b0 con 0x7f00a415da10 2023-10-16T13:28:29.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.684+0000 7f00aab18700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00a40d26f0 con 0x7f00a40d1d20 2023-10-16T13:28:29.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.684+0000 7f00a88b4700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f00a415da10 0x7f00a40d1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:29.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.684+0000 7f00a88b4700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f00a415da10 0x7f00a40d1520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50134/0 (socket says 172.21.15.16:50134) 2023-10-16T13:28:29.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.684+0000 7f00a88b4700 1 -- 172.21.15.16:0/4090737309 learned_addr learned my addr 172.21.15.16:0/4090737309 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:29.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.684+0000 7f00a90b5700 1 --2- 172.21.15.16:0/4090737309 >> v2:172.21.15.49:3300/0 conn(0x7f00a40d1d20 0x7f00a40d21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:29.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.684+0000 7f00a88b4700 1 -- 172.21.15.16:0/4090737309 >> v2:172.21.15.49:3300/0 conn(0x7f00a40d1d20 msgr2=0x7f00a40d21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:28:29.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.684+0000 7f00a88b4700 1 --2- 172.21.15.16:0/4090737309 >> v2:172.21.15.49:3300/0 conn(0x7f00a40d1d20 0x7f00a40d21b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:29.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.684+0000 7f00a88b4700 1 -- 172.21.15.16:0/4090737309 >> v1:172.21.15.49:6789/0 conn(0x7f00a40d4060 legacy=0x7f00a40cf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:29.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.684+0000 7f00a88b4700 1 -- 172.21.15.16:0/4090737309 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00a40ce7e0 con 0x7f00a415da10 2023-10-16T13:28:29.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.684+0000 7f00a90b5700 1 --2- 172.21.15.16:0/4090737309 >> v2:172.21.15.49:3300/0 conn(0x7f00a40d1d20 0x7f00a40d21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:28:29.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.684+0000 7f00a88b4700 1 --2- 172.21.15.16:0/4090737309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00a415da10 0x7f00a40d1520 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 crypto rx=0x7f00a40d32a0 tx=0x7f0098009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d0ca8d8e010392d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:29.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.684+0000 7f00a37fe700 1 -- 172.21.15.16:0/4090737309 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0098005e30 con 0x7f00a415da10 2023-10-16T13:28:29.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.684+0000 7f00a37fe700 1 -- 172.21.15.16:0/4090737309 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0098004030 con 0x7f00a415da10 2023-10-16T13:28:29.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.684+0000 7f00a37fe700 1 -- 172.21.15.16:0/4090737309 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00980043e0 con 0x7f00a415da10 2023-10-16T13:28:29.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.685+0000 7f00aab18700 1 -- 172.21.15.16:0/4090737309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00a415da10 msgr2=0x7f00a40d1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:29.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.685+0000 7f00aab18700 1 --2- 172.21.15.16:0/4090737309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00a415da10 0x7f00a40d1520 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 crypto rx=0x7f00a40d32a0 tx=0x7f0098009bd0 comp rx=0 tx=0).stop 2023-10-16T13:28:29.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.685+0000 7f00aab18700 1 -- 172.21.15.16:0/4090737309 shutdown_connections 2023-10-16T13:28:29.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.685+0000 7f00aab18700 1 --2- 172.21.15.16:0/4090737309 >> v2:172.21.15.49:3300/0 conn(0x7f00a40d1d20 0x7f00a40d21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:29.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.685+0000 7f00aab18700 1 --2- 172.21.15.16:0/4090737309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00a415da10 0x7f00a40d1520 unknown :-1 s=CLOSED pgs=2129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:29.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.685+0000 7f00aab18700 1 -- 172.21.15.16:0/4090737309 >> 172.21.15.16:0/4090737309 conn(0x7f00a409c9f0 msgr2=0x7f00a40d3740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:29.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.685+0000 7f00aab18700 1 -- 172.21.15.16:0/4090737309 shutdown_connections 2023-10-16T13:28:29.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.685+0000 7f00aab18700 1 -- 172.21.15.16:0/4090737309 wait complete. 2023-10-16T13:28:29.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.686+0000 7f00aab18700 1 Processor -- start 2023-10-16T13:28:29.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.686+0000 7f00aab18700 1 -- start start 2023-10-16T13:28:29.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.687+0000 7f00aab18700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f00a40d1d20 0x7f00a415c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:29.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.687+0000 7f00aab18700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00a40d4060 0x7f00a415bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:29.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.687+0000 7f00aab18700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f00a415da10 0x7f00a40cb950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:29.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.687+0000 7f00aab18700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00a40cb4c0 con 0x7f00a40d4060 2023-10-16T13:28:29.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.687+0000 7f00aab18700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00a40ca8b0 con 0x7f00a40d1d20 2023-10-16T13:28:29.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.687+0000 7f00aab18700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00a40cab90 con 0x7f00a415da10 2023-10-16T13:28:29.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.687+0000 7f00a3fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00a40d4060 0x7f00a415bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:29.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.687+0000 7f00a88b4700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f00a40d1d20 0x7f00a415c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:29.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.687+0000 7f00a3fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00a40d4060 0x7f00a415bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50142/0 (socket says 172.21.15.16:50142) 2023-10-16T13:28:29.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.687+0000 7f00a3fff700 1 -- 172.21.15.16:0/772593558 learned_addr learned my addr 172.21.15.16:0/772593558 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:29.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.687+0000 7f00a90b5700 1 --2- 172.21.15.16:0/772593558 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f00a415da10 0x7f00a40cb950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:29.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.688+0000 7f00a88b4700 1 -- 172.21.15.16:0/772593558 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f00a415da10 msgr2=0x7f00a40cb950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:29.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.688+0000 7f00a88b4700 1 --2- 172.21.15.16:0/772593558 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f00a415da10 0x7f00a40cb950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:29.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.688+0000 7f00a88b4700 1 -- 172.21.15.16:0/772593558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00a40d4060 msgr2=0x7f00a415bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:29.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.688+0000 7f00a88b4700 1 --2- 172.21.15.16:0/772593558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00a40d4060 0x7f00a415bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:29.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.688+0000 7f00a88b4700 1 -- 172.21.15.16:0/772593558 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0098009500 con 0x7f00a40d1d20 2023-10-16T13:28:29.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.688+0000 7f00a3fff700 1 --2- 172.21.15.16:0/772593558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00a40d4060 0x7f00a415bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:28:29.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.688+0000 7f00a88b4700 1 --2- 172.21.15.16:0/772593558 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f00a40d1d20 0x7f00a415c480 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f0098009fd0 tx=0x7f0098004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:29.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.688+0000 7f00a17fa700 1 -- 172.21.15.16:0/772593558 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0098003c70 con 0x7f00a40d1d20 2023-10-16T13:28:29.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.688+0000 7f00a17fa700 1 -- 172.21.15.16:0/772593558 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0098003df0 con 0x7f00a40d1d20 2023-10-16T13:28:29.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.688+0000 7f00a17fa700 1 -- 172.21.15.16:0/772593558 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0098011660 con 0x7f00a40d1d20 2023-10-16T13:28:29.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.688+0000 7f00aab18700 1 -- 172.21.15.16:0/772593558 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00a40cae70 con 0x7f00a40d1d20 2023-10-16T13:28:29.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.688+0000 7f00aab18700 1 -- 172.21.15.16:0/772593558 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00a415c1f0 con 0x7f00a40d1d20 2023-10-16T13:28:29.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.689+0000 7f00aab18700 1 -- 172.21.15.16:0/772593558 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00a40cb7a0 con 0x7f00a40d1d20 2023-10-16T13:28:29.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.690+0000 7f00a17fa700 1 -- 172.21.15.16:0/772593558 <== mon.1 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 108) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f00980117c0 con 0x7f00a40d1d20 2023-10-16T13:28:29.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.690+0000 7f00a17fa700 1 --2- 172.21.15.16:0/772593558 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f008806b070 0x7f008806d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:29.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.690+0000 7f00a3fff700 1 -- 172.21.15.16:0/772593558 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f008806b070 msgr2=0x7f008806d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/4211329181 2023-10-16T13:28:29.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.690+0000 7f00a17fa700 1 -- 172.21.15.16:0/772593558 <== mon.1 v2:172.21.15.49:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f009808a9e0 con 0x7f00a40d1d20 2023-10-16T13:28:29.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.690+0000 7f00a3fff700 1 --2- 172.21.15.16:0/772593558 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f008806b070 0x7f008806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:28:29.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.695+0000 7f00a17fa700 1 -- 172.21.15.16:0/772593558 <== mon.1 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0098055860 con 0x7f00a40d1d20 2023-10-16T13:28:29.703 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:28:29.696+0000 7f6d18dfa200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T13:28:29.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.891+0000 7f00a3fff700 1 -- 172.21.15.16:0/772593558 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f008806b070 msgr2=0x7f008806d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/4211329181 2023-10-16T13:28:29.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.891+0000 7f00a3fff700 1 --2- 172.21.15.16:0/772593558 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f008806b070 0x7f008806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:28:29.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.901+0000 7f00aab18700 1 -- 172.21.15.16:0/772593558 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f00a404ebb0 con 0x7f00a40d1d20 2023-10-16T13:28:29.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.905+0000 7f00a17fa700 1 -- 172.21.15.16:0/772593558 <== mon.1 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v66)=0 v66) v1 ==== 133+0+0 (secure 0 0 0) 0x7f0098058e80 con 0x7f00a40d1d20 2023-10-16T13:28:29.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.907+0000 7f00aab18700 1 -- 172.21.15.16:0/772593558 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f008806b070 msgr2=0x7f008806d530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:28:29.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.907+0000 7f00aab18700 1 --2- 172.21.15.16:0/772593558 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f008806b070 0x7f008806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:29.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.907+0000 7f00aab18700 1 -- 172.21.15.16:0/772593558 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f00a40d1d20 msgr2=0x7f00a415c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:29.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.907+0000 7f00aab18700 1 --2- 172.21.15.16:0/772593558 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f00a40d1d20 0x7f00a415c480 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f0098009fd0 tx=0x7f0098004750 comp rx=0 tx=0).stop 2023-10-16T13:28:29.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.908+0000 7f00aab18700 1 -- 172.21.15.16:0/772593558 shutdown_connections 2023-10-16T13:28:29.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.908+0000 7f00aab18700 1 --2- 172.21.15.16:0/772593558 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f00a415da10 0x7f00a40cb950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:29.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.908+0000 7f00aab18700 1 --2- 172.21.15.16:0/772593558 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f008806b070 0x7f008806d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:29.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.908+0000 7f00aab18700 1 --2- 172.21.15.16:0/772593558 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f00a40d1d20 0x7f00a415c480 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:29.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.908+0000 7f00aab18700 1 --2- 172.21.15.16:0/772593558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00a40d4060 0x7f00a415bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:29.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.908+0000 7f00aab18700 1 -- 172.21.15.16:0/772593558 >> 172.21.15.16:0/772593558 conn(0x7f00a409c9f0 msgr2=0x7f00a40cf1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:29.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.908+0000 7f00aab18700 1 -- 172.21.15.16:0/772593558 shutdown_connections 2023-10-16T13:28:29.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:29.908+0000 7f00aab18700 1 -- 172.21.15.16:0/772593558 wait complete. 2023-10-16T13:28:29.923 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:28:29.923 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-10-16T13:28:29.945 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:28:29.946 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/test_restart) 2023-10-16T13:28:30.004 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/test_restart 1m 2023-10-16T13:28:30.016 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:28:30.014+0000 7f9f8a5b7200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T13:28:30.051 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:28:30.049+0000 7f6d18dfa200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T13:28:30.137 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:28:30.134+0000 7f9f8a5b7200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T13:28:30.175 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:28:30.173+0000 7f6d18dfa200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T13:28:30.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:30.175+0000 7f6659d68700 1 Processor -- start 2023-10-16T13:28:30.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:30.175+0000 7f6659d68700 1 -- start start 2023-10-16T13:28:30.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:30.176+0000 7f6659d68700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f665415da10 0x7f66540d1520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:30.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:30.176+0000 7f6659d68700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f66540d4060 0x7f66540cf2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:30.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:30.176+0000 7f6659d68700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66540d4540 con 0x7f66540d1d20 2023-10-16T13:28:30.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:30.176+0000 7f6659d68700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66540ce7e0 con 0x7f66540d4060 2023-10-16T13:28:30.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:30.176+0000 7f6659d68700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66540ceac0 con 0x7f665415da10 2023-10-16T13:28:30.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:30.176+0000 7f6653fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f66540d1d20 0x7f66540d2130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:39462/0 (socket says 172.21.15.16:39462) 2023-10-16T13:28:30.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:30.176+0000 7f6653fff700 1 -- 172.21.15.16:0/2764411473 learned_addr learned my addr 172.21.15.16:0/2764411473 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:28:30.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:30.176+0000 7f6652ffd700 1 --2- 172.21.15.16:0/2764411473 >> v2:172.21.15.49:3300/0 conn(0x7f66540d4060 0x7f66540cf2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:30.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:30.176+0000 7f66537fe700 1 --2- 172.21.15.16:0/2764411473 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f665415da10 0x7f66540d1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:30.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:30.177+0000 7f66527fc700 1 -- 172.21.15.16:0/2764411473 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3480887873 0 0) 0x7f66540d4540 con 0x7f66540d1d20 2023-10-16T13:28:30.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:30.177+0000 7f66527fc700 1 -- 172.21.15.16:0/2764411473 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66400035b0 con 0x7f66540d1d20 2023-10-16T13:28:30.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:30.177+0000 7f66537fe700 1 -- 172.21.15.16:0/2764411473 >> v2:172.21.15.49:3300/0 conn(0x7f66540d4060 msgr2=0x7f66540cf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:28:30.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:30.177+0000 7f66537fe700 1 --2- 172.21.15.16:0/2764411473 >> v2:172.21.15.49:3300/0 conn(0x7f66540d4060 0x7f66540cf2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:30.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:30.177+0000 7f66537fe700 1 -- 172.21.15.16:0/2764411473 >> v1:172.21.15.16:6789/0 conn(0x7f66540d1d20 legacy=0x7f66540d2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:30.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:30.177+0000 7f66537fe700 1 -- 172.21.15.16:0/2764411473 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66540ceda0 con 0x7f665415da10 2023-10-16T13:28:30.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:30.177+0000 7f6652ffd700 1 --2- 172.21.15.16:0/2764411473 >> v2:172.21.15.49:3300/0 conn(0x7f66540d4060 0x7f66540cf2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:28:30.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:30.177+0000 7f66537fe700 1 --2- 172.21.15.16:0/2764411473 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f665415da10 0x7f66540d1520 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f664c009820 tx=0x7f664c009b30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9ed83f8493421d5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:30.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:30.177+0000 7f66527fc700 1 -- 172.21.15.16:0/2764411473 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f664c005e30 con 0x7f665415da10 2023-10-16T13:28:30.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:30.177+0000 7f66527fc700 1 -- 172.21.15.16:0/2764411473 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f664c006030 con 0x7f665415da10 2023-10-16T13:28:30.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:30.177+0000 7f66527fc700 1 -- 172.21.15.16:0/2764411473 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f664c0063c0 con 0x7f665415da10 2023-10-16T13:28:30.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:30.178+0000 7f6659d68700 1 -- 172.21.15.16:0/2764411473 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f665415da10 msgr2=0x7f66540d1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:30.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:30.178+0000 7f6659d68700 1 --2- 172.21.15.16:0/2764411473 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f665415da10 0x7f66540d1520 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f664c009820 tx=0x7f664c009b30 comp rx=0 tx=0).stop 2023-10-16T13:28:30.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:30.178+0000 7f6659d68700 1 -- 172.21.15.16:0/2764411473 shutdown_connections 2023-10-16T13:28:30.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:30.178+0000 7f6659d68700 1 --2- 172.21.15.16:0/2764411473 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f665415da10 0x7f66540d1520 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:30.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:30.178+0000 7f6659d68700 1 --2- 172.21.15.16:0/2764411473 >> v2:172.21.15.49:3300/0 conn(0x7f66540d4060 0x7f66540cf2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:30.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:30.178+0000 7f6659d68700 1 -- 172.21.15.16:0/2764411473 >> 172.21.15.16:0/2764411473 conn(0x7f665409c9f0 msgr2=0x7f66540d3740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:30.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:30.178+0000 7f6659d68700 1 -- 172.21.15.16:0/2764411473 shutdown_connections 2023-10-16T13:28:30.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:30.178+0000 7f6659d68700 1 -- 172.21.15.16:0/2764411473 wait complete. 2023-10-16T13:28:30.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:30.179+0000 7f6659d68700 1 Processor -- start 2023-10-16T13:28:30.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:30.179+0000 7f6659d68700 1 -- start start 2023-10-16T13:28:30.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:30.180+0000 7f6659d68700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66540d1d20 0x7f66541f2810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:30.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:30.180+0000 7f6659d68700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f66540d4060 0x7f66540b3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:30.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:30.180+0000 7f6659d68700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f665415da10 0x7f6654160050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:30.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:30.180+0000 7f6659d68700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66540b2f40 con 0x7f66540d1d20 2023-10-16T13:28:30.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:30.180+0000 7f6659d68700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66540b3220 con 0x7f66540d4060 2023-10-16T13:28:30.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:30.180+0000 7f6659d68700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66540b3500 con 0x7f665415da10 2023-10-16T13:28:30.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:30.180+0000 7f66537fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66540d1d20 0x7f66541f2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:30.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:30.180+0000 7f66537fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66540d1d20 0x7f66541f2810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50190/0 (socket says 172.21.15.16:50190) 2023-10-16T13:28:30.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:30.180+0000 7f66537fe700 1 -- 172.21.15.16:0/862711969 learned_addr learned my addr 172.21.15.16:0/862711969 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:30.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:30.180+0000 7f6652ffd700 1 --2- 172.21.15.16:0/862711969 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f66540d4060 0x7f66540b3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:30.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:30.180+0000 7f6653fff700 1 --2- 172.21.15.16:0/862711969 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f665415da10 0x7f6654160050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:30.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:30.181+0000 7f66537fe700 1 -- 172.21.15.16:0/862711969 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f665415da10 msgr2=0x7f6654160050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:30.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:30.181+0000 7f66537fe700 1 --2- 172.21.15.16:0/862711969 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f665415da10 0x7f6654160050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:30.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:30.181+0000 7f66537fe700 1 -- 172.21.15.16:0/862711969 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f66540d4060 msgr2=0x7f66540b3fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:30.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:30.181+0000 7f66537fe700 1 --2- 172.21.15.16:0/862711969 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f66540d4060 0x7f66540b3fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:30.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:30.181+0000 7f66537fe700 1 -- 172.21.15.16:0/862711969 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f663c0094a0 con 0x7f66540d1d20 2023-10-16T13:28:30.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:30.181+0000 7f6653fff700 1 --2- 172.21.15.16:0/862711969 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f665415da10 0x7f6654160050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:28:30.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:30.181+0000 7f6652ffd700 1 --2- 172.21.15.16:0/862711969 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f66540d4060 0x7f66540b3fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:28:30.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:30.181+0000 7f66537fe700 1 --2- 172.21.15.16:0/862711969 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66540d1d20 0x7f66541f2810 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 crypto rx=0x7f664c000c00 tx=0x7f664c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:30.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:30.181+0000 7f664bfff700 1 -- 172.21.15.16:0/862711969 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f664c0061a0 con 0x7f66540d1d20 2023-10-16T13:28:30.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:30.181+0000 7f664bfff700 1 -- 172.21.15.16:0/862711969 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f664c020070 con 0x7f66540d1d20 2023-10-16T13:28:30.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:30.181+0000 7f6659d68700 1 -- 172.21.15.16:0/862711969 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f664c009430 con 0x7f66540d1d20 2023-10-16T13:28:30.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:30.181+0000 7f664bfff700 1 -- 172.21.15.16:0/862711969 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f664c006560 con 0x7f66540d1d20 2023-10-16T13:28:30.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:30.181+0000 7f6659d68700 1 -- 172.21.15.16:0/862711969 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66540b3ac0 con 0x7f66540d1d20 2023-10-16T13:28:30.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:30.182+0000 7f6659d68700 1 -- 172.21.15.16:0/862711969 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f665404ebb0 con 0x7f66540d1d20 2023-10-16T13:28:30.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:30.183+0000 7f664bfff700 1 -- 172.21.15.16:0/862711969 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 108) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f664c022030 con 0x7f66540d1d20 2023-10-16T13:28:30.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:30.183+0000 7f664bfff700 1 --2- 172.21.15.16:0/862711969 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f663406b070 0x7f663406d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:30.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:30.183+0000 7f6652ffd700 1 -- 172.21.15.16:0/862711969 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f663406b070 msgr2=0x7f663406d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/4211329181 2023-10-16T13:28:30.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:30.183+0000 7f664bfff700 1 -- 172.21.15.16:0/862711969 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f664c08ac40 con 0x7f66540d1d20 2023-10-16T13:28:30.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:30.183+0000 7f6652ffd700 1 --2- 172.21.15.16:0/862711969 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f663406b070 0x7f663406d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:28:30.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:30.192+0000 7f664bfff700 1 -- 172.21.15.16:0/862711969 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f664c055ac0 con 0x7f66540d1d20 2023-10-16T13:28:30.214 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:28:30.212+0000 7f9f8a5b7200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T13:28:30.258 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:28:30.255+0000 7f6d18dfa200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T13:28:30.299 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:28:30.296+0000 7f9f8a5b7200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T13:28:30.343 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:28:30.341+0000 7f6d18dfa200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T13:28:30.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:30.384+0000 7f6652ffd700 1 -- 172.21.15.16:0/862711969 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f663406b070 msgr2=0x7f663406d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/4211329181 2023-10-16T13:28:30.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:30.384+0000 7f6652ffd700 1 --2- 172.21.15.16:0/862711969 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f663406b070 0x7f663406d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:28:30.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:30.422+0000 7f6659d68700 1 -- 172.21.15.16:0/862711969 --> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/test_restart", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f665415ddf0 con 0x7f663406b070 2023-10-16T13:28:30.614 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:28:30.617 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:28:30.687 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:28:30.685+0000 7f9f8a5b7200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T13:28:30.732 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:28:30.730+0000 7f6d18dfa200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T13:28:30.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:30.785+0000 7f6652ffd700 1 -- 172.21.15.16:0/862711969 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f663406b070 msgr2=0x7f663406d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/4211329181 2023-10-16T13:28:30.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:30.785+0000 7f6652ffd700 1 --2- 172.21.15.16:0/862711969 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f663406b070 0x7f663406d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-16T13:28:31.294 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:28:31.292+0000 7f9f8a5b7200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T13:28:31.343 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:28:31.341+0000 7f6d18dfa200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T13:28:31.367 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:28:31.364+0000 7f9f8a5b7200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T13:28:31.417 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:28:31.414+0000 7f6d18dfa200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T13:28:31.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:31.586+0000 7f6652ffd700 1 -- 172.21.15.16:0/862711969 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f663406b070 msgr2=0x7f663406d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/4211329181 2023-10-16T13:28:31.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:31.586+0000 7f6652ffd700 1 --2- 172.21.15.16:0/862711969 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f663406b070 0x7f663406d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-16T13:28:32.031 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:28:32.028+0000 7f9f8a5b7200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:28:32.087 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:28:32.085+0000 7f6d18dfa200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:28:32.114 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:28:32.111+0000 7f9f8a5b7200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T13:28:32.170 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:28:32.168+0000 7f6d18dfa200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T13:28:32.264 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:28:32.262+0000 7f9f8a5b7200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:28:32.323 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:28:32.321+0000 7f6d18dfa200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:28:32.502 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:28:32.499+0000 7f9f8a5b7200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T13:28:32.564 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:28:32.561+0000 7f6d18dfa200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T13:28:32.579 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:28:32.577+0000 7f9f8a5b7200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T13:28:32.642 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:28:32.640+0000 7f6d18dfa200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T13:28:32.855 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:28:32.852+0000 7f9f8a5b7200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T13:28:32.922 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:28:32.920+0000 7f6d18dfa200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T13:28:33.086 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:28:33.084+0000 7f9f8a5b7200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T13:28:33.158 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:28:33.156+0000 7f6d18dfa200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T13:28:33.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:33.187+0000 7f6652ffd700 1 -- 172.21.15.16:0/862711969 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f663406b070 msgr2=0x7f663406d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/4211329181 2023-10-16T13:28:33.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:33.187+0000 7f6652ffd700 1 --2- 172.21.15.16:0/862711969 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f663406b070 0x7f663406d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-10-16T13:28:33.242 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:28:33.239+0000 7f9f8a5b7200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T13:28:33.317 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:28:33.315+0000 7f6d18dfa200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T13:28:33.361 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:28:33.359+0000 7f9f8a5b7200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T13:28:33.438 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:28:33.436+0000 7f6d18dfa200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T13:28:33.522 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:28:33.520+0000 7f9f8a5b7200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T13:28:33.603 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:28:33.601+0000 7f6d18dfa200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T13:28:33.794 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:28:33.791+0000 7f9f8a5b7200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T13:28:33.884 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:28:33.881+0000 7f6d18dfa200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T13:28:33.961 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:28:33.959+0000 7f9f8a5b7200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T13:28:34.054 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:28:34.052+0000 7f6d18dfa200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T13:28:34.517 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:28:34.514+0000 7f9f8a5b7200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T13:28:34.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:34.583+0000 7f664bfff700 1 -- 172.21.15.16:0/862711969 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 109) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f664c055e90 con 0x7f66540d1d20 2023-10-16T13:28:34.620 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:28:34.618+0000 7f6d18dfa200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T13:28:34.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:34.637+0000 7f664bfff700 1 -- 172.21.15.16:0/862711969 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mgrmap(e 110) v1 ==== 88957+0+0 (secure 0 0 0) 0x7f664c088f80 con 0x7f66540d1d20 2023-10-16T13:28:34.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:34.637+0000 7f664bfff700 1 -- 172.21.15.16:0/862711969 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f663406b070 msgr2=0x7f663406d530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:28:34.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:34.637+0000 7f664bfff700 1 --2- 172.21.15.16:0/862711969 >> [v2:172.21.15.16:6832/4211329181,v1:172.21.15.16:6833/4211329181] conn(0x7f663406b070 0x7f663406d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:34.775 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:28:34.772+0000 7f6c8bdd5700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:28:34.775 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:28:34.772+0000 7f6c8bdd5700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:28:34.775 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:28:34.772+0000 7f6c8bdd5700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:28:34.776 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:28:34.772+0000 7f6c8bdd5700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:28:34.776 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:28:34.772+0000 7f6c8bdd5700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:28:34.776 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:28:34.772+0000 7f6c8e5da700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:28:34.776 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:28:34.772+0000 7f6c8e5da700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:28:34.776 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:28:34.772+0000 7f6c8e5da700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:28:34.776 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:28:34.772+0000 7f6c8e5da700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:28:34.777 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:28:34.772+0000 7f6c8e5da700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:28:35.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.641+0000 7f664bfff700 1 -- 172.21.15.16:0/862711969 <== mon.0 v2:172.21.15.16:3300/0 9 ==== mgrmap(e 111) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f664c08a990 con 0x7f66540d1d20 2023-10-16T13:28:35.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.642+0000 7f664bfff700 1 --2- 172.21.15.16:0/862711969 >> [v2:172.21.15.16:6832/3952564978,v1:172.21.15.16:6833/3952564978] conn(0x7f6634027a00 0x7f6634027de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:35.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.642+0000 7f664bfff700 1 -- 172.21.15.16:0/862711969 --> [v2:172.21.15.16:6832/3952564978,v1:172.21.15.16:6833/3952564978] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/test_restart", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f6634075590 con 0x7f6634027a00 2023-10-16T13:28:35.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.644+0000 7f6652ffd700 1 --2- 172.21.15.16:0/862711969 >> [v2:172.21.15.16:6832/3952564978,v1:172.21.15.16:6833/3952564978] conn(0x7f6634027a00 0x7f6634027de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:35.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.647+0000 7f6652ffd700 1 --2- 172.21.15.16:0/862711969 >> [v2:172.21.15.16:6832/3952564978,v1:172.21.15.16:6833/3952564978] conn(0x7f6634027a00 0x7f6634027de0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f6654159cb0 tx=0x7f663c009210 comp rx=0 tx=0).ready entity=mgr.10485 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:35.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.706+0000 7f664bfff700 1 -- 172.21.15.16:0/862711969 <== mgr.10485 v2:172.21.15.16:6832/3952564978 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f6634075590 con 0x7f6634027a00 2023-10-16T13:28:35.709 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule set for path /snap_test_dir1/test_restart 2023-10-16T13:28:35.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.708+0000 7f6659d68700 1 -- 172.21.15.16:0/862711969 >> [v2:172.21.15.16:6832/3952564978,v1:172.21.15.16:6833/3952564978] conn(0x7f6634027a00 msgr2=0x7f6634027de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:35.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.708+0000 7f6659d68700 1 --2- 172.21.15.16:0/862711969 >> [v2:172.21.15.16:6832/3952564978,v1:172.21.15.16:6833/3952564978] conn(0x7f6634027a00 0x7f6634027de0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f6654159cb0 tx=0x7f663c009210 comp rx=0 tx=0).stop 2023-10-16T13:28:35.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.708+0000 7f6659d68700 1 -- 172.21.15.16:0/862711969 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66540d1d20 msgr2=0x7f66541f2810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:35.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.708+0000 7f6659d68700 1 --2- 172.21.15.16:0/862711969 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66540d1d20 0x7f66541f2810 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 crypto rx=0x7f664c000c00 tx=0x7f664c004750 comp rx=0 tx=0).stop 2023-10-16T13:28:35.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.708+0000 7f6653fff700 1 -- 172.21.15.16:0/862711969 reap_dead start 2023-10-16T13:28:35.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.709+0000 7f6659d68700 1 -- 172.21.15.16:0/862711969 shutdown_connections 2023-10-16T13:28:35.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.709+0000 7f6659d68700 1 -- 172.21.15.16:0/862711969 >> 172.21.15.16:0/862711969 conn(0x7f665409c9f0 msgr2=0x7f665415dff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:35.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.709+0000 7f6659d68700 1 -- 172.21.15.16:0/862711969 shutdown_connections 2023-10-16T13:28:35.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.709+0000 7f6659d68700 1 -- 172.21.15.16:0/862711969 wait complete. 2023-10-16T13:28:35.725 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/test_restart 2023-10-16T13:28:35.725 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/test_restart 1m 2023-10-16T13:28:35.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.853+0000 7f3650bab700 1 Processor -- start 2023-10-16T13:28:35.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.853+0000 7f3650bab700 1 -- start start 2023-10-16T13:28:35.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.854+0000 7f3650bab700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f364c06c300 0x7f364c06c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:35.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.854+0000 7f3650bab700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f364c0b20f0 con 0x7f364c0b2620 2023-10-16T13:28:35.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.854+0000 7f3650bab700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f364c0d0720 con 0x7f364c0d2110 2023-10-16T13:28:35.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.854+0000 7f3650bab700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f364c0af1f0 con 0x7f364c06c300 2023-10-16T13:28:35.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.854+0000 7f364ad9d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f364c0b2620 0x7f364c0b1660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:39658/0 (socket says 172.21.15.16:39658) 2023-10-16T13:28:35.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.854+0000 7f364ad9d700 1 -- 172.21.15.16:0/1702756144 learned_addr learned my addr 172.21.15.16:0/1702756144 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:28:35.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.855+0000 7f364a59c700 1 --2- 172.21.15.16:0/1702756144 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f364c06c300 0x7f364c06c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:35.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.855+0000 7f364959a700 1 -- 172.21.15.16:0/1702756144 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 726167188 0 0) 0x7f364c0b20f0 con 0x7f364c0b2620 2023-10-16T13:28:35.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.855+0000 7f364959a700 1 -- 172.21.15.16:0/1702756144 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36380035b0 con 0x7f364c0b2620 2023-10-16T13:28:35.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.855+0000 7f364959a700 1 -- 172.21.15.16:0/1702756144 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2649314529 0 0) 0x7f364c0d0720 con 0x7f364c0d2110 2023-10-16T13:28:35.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.855+0000 7f364959a700 1 -- 172.21.15.16:0/1702756144 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f364c0b20f0 con 0x7f364c0d2110 2023-10-16T13:28:35.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.855+0000 7f364959a700 1 -- 172.21.15.16:0/1702756144 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2732224001 0 0) 0x7f36380035b0 con 0x7f364c0b2620 2023-10-16T13:28:35.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.855+0000 7f364959a700 1 -- 172.21.15.16:0/1702756144 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f364c06c300 msgr2=0x7f364c06c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:28:35.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.855+0000 7f364959a700 1 --2- 172.21.15.16:0/1702756144 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f364c06c300 0x7f364c06c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:35.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.855+0000 7f364959a700 1 -- 172.21.15.16:0/1702756144 >> v1:172.21.15.49:6789/0 conn(0x7f364c0d2110 legacy=0x7f364c0b0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:35.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.855+0000 7f364959a700 1 -- 172.21.15.16:0/1702756144 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f364c1614c0 con 0x7f364c0b2620 2023-10-16T13:28:35.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.855+0000 7f364959a700 1 -- 172.21.15.16:0/1702756144 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f36440019f0 con 0x7f364c0b2620 2023-10-16T13:28:35.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.855+0000 7f364a59c700 1 --2- 172.21.15.16:0/1702756144 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f364c06c300 0x7f364c06c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:28:35.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.855+0000 7f364959a700 1 -- 172.21.15.16:0/1702756144 >> v1:172.21.15.16:6789/0 conn(0x7f364c0b2620 legacy=0x7f364c0b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:35.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.855+0000 7f364959a700 1 --2- 172.21.15.16:0/1702756144 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3638003bc0 0x7f3638003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:35.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.855+0000 7f364959a700 1 --2- 172.21.15.16:0/1702756144 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f364c06c300 0x7f3638004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:35.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.855+0000 7f364959a700 1 --2- 172.21.15.16:0/1702756144 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3638004d90 0x7f3638005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:35.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.855+0000 7f364959a700 1 -- 172.21.15.16:0/1702756144 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f364c0af1f0 con 0x7f3638004d90 2023-10-16T13:28:35.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.855+0000 7f364959a700 1 -- 172.21.15.16:0/1702756144 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f363800bac0 con 0x7f3638003bc0 2023-10-16T13:28:35.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.855+0000 7f364959a700 1 -- 172.21.15.16:0/1702756144 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f363800bc00 con 0x7f364c06c300 2023-10-16T13:28:35.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.855+0000 7f364ad9d700 1 --2- 172.21.15.16:0/1702756144 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3638004d90 0x7f3638005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:35.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.855+0000 7f364a59c700 1 --2- 172.21.15.16:0/1702756144 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3638003bc0 0x7f3638003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:35.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.855+0000 7f3649d9b700 1 --2- 172.21.15.16:0/1702756144 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f364c06c300 0x7f3638004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:35.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.855+0000 7f364ad9d700 1 -- 172.21.15.16:0/1702756144 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f364c06c300 msgr2=0x7f3638004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:28:35.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.856+0000 7f364ad9d700 1 --2- 172.21.15.16:0/1702756144 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f364c06c300 0x7f3638004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:35.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.856+0000 7f364ad9d700 1 -- 172.21.15.16:0/1702756144 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3638003bc0 msgr2=0x7f3638003fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:28:35.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.856+0000 7f364ad9d700 1 --2- 172.21.15.16:0/1702756144 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3638003bc0 0x7f3638003fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:35.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.856+0000 7f364ad9d700 1 -- 172.21.15.16:0/1702756144 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36340094a0 con 0x7f3638004d90 2023-10-16T13:28:35.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.856+0000 7f364ad9d700 1 --2- 172.21.15.16:0/1702756144 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3638004d90 0x7f3638005210 secure :-1 s=READY pgs=2144 cs=0 l=1 rev1=1 crypto rx=0x7f364400a7d0 tx=0x7f3638005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d164b4b10fa2849 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:35.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.856+0000 7f364959a700 1 -- 172.21.15.16:0/1702756144 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3644012600 con 0x7f3638004d90 2023-10-16T13:28:35.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.856+0000 7f364959a700 1 -- 172.21.15.16:0/1702756144 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3644012780 con 0x7f3638004d90 2023-10-16T13:28:35.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.856+0000 7f364959a700 1 -- 172.21.15.16:0/1702756144 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3644012b30 con 0x7f3638004d90 2023-10-16T13:28:35.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.856+0000 7f3650bab700 1 -- 172.21.15.16:0/1702756144 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3638004d90 msgr2=0x7f3638005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:35.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.856+0000 7f3650bab700 1 --2- 172.21.15.16:0/1702756144 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3638004d90 0x7f3638005210 secure :-1 s=READY pgs=2144 cs=0 l=1 rev1=1 crypto rx=0x7f364400a7d0 tx=0x7f3638005880 comp rx=0 tx=0).stop 2023-10-16T13:28:35.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.856+0000 7f364ad9d700 1 -- 172.21.15.16:0/1702756144 reap_dead start 2023-10-16T13:28:35.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.856+0000 7f3650bab700 1 -- 172.21.15.16:0/1702756144 shutdown_connections 2023-10-16T13:28:35.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.856+0000 7f3650bab700 1 -- 172.21.15.16:0/1702756144 >> 172.21.15.16:0/1702756144 conn(0x7f364c0d15f0 msgr2=0x7f364c0d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:35.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.856+0000 7f3650bab700 1 -- 172.21.15.16:0/1702756144 shutdown_connections 2023-10-16T13:28:35.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.856+0000 7f3650bab700 1 -- 172.21.15.16:0/1702756144 wait complete. 2023-10-16T13:28:35.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.858+0000 7f3650bab700 1 Processor -- start 2023-10-16T13:28:35.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.858+0000 7f3650bab700 1 -- start start 2023-10-16T13:28:35.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.859+0000 7f3650bab700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3638004d90 0x7f364c1f3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:35.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.859+0000 7f3650bab700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f364c0c5120 0x7f364c09b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:35.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.859+0000 7f3650bab700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f364c0b1180 0x7f364c1f2790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:35.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.859+0000 7f3650bab700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f364c1f4300 con 0x7f3638004d90 2023-10-16T13:28:35.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.859+0000 7f3650bab700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f364c0c7280 con 0x7f364c0b1180 2023-10-16T13:28:35.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.859+0000 7f3650bab700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f364c0c55b0 con 0x7f364c0c5120 2023-10-16T13:28:35.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.859+0000 7f364a59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3638004d90 0x7f364c1f3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:35.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.859+0000 7f364a59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3638004d90 0x7f364c1f3dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50396/0 (socket says 172.21.15.16:50396) 2023-10-16T13:28:35.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.859+0000 7f364a59c700 1 -- 172.21.15.16:0/634542240 learned_addr learned my addr 172.21.15.16:0/634542240 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:28:35.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.859+0000 7f3649d9b700 1 --2- 172.21.15.16:0/634542240 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f364c0c5120 0x7f364c09b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:35.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.859+0000 7f364ad9d700 1 --2- 172.21.15.16:0/634542240 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f364c0b1180 0x7f364c1f2790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:35.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.859+0000 7f364a59c700 1 -- 172.21.15.16:0/634542240 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f364c0c5120 msgr2=0x7f364c09b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:35.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.859+0000 7f364a59c700 1 --2- 172.21.15.16:0/634542240 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f364c0c5120 0x7f364c09b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:35.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.859+0000 7f364a59c700 1 -- 172.21.15.16:0/634542240 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f364c0b1180 msgr2=0x7f364c1f2790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:35.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.859+0000 7f364a59c700 1 --2- 172.21.15.16:0/634542240 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f364c0b1180 0x7f364c1f2790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:35.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.859+0000 7f364a59c700 1 -- 172.21.15.16:0/634542240 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f364c0c5890 con 0x7f3638004d90 2023-10-16T13:28:35.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.859+0000 7f364a59c700 1 --2- 172.21.15.16:0/634542240 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3638004d90 0x7f364c1f3dc0 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 crypto rx=0x7f3634011cd0 tx=0x7f363400f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:35.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.859+0000 7f3642ffd700 1 -- 172.21.15.16:0/634542240 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3634011e70 con 0x7f3638004d90 2023-10-16T13:28:35.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.859+0000 7f3642ffd700 1 -- 172.21.15.16:0/634542240 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f363400fd40 con 0x7f3638004d90 2023-10-16T13:28:35.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.859+0000 7f3650bab700 1 -- 172.21.15.16:0/634542240 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f364c0c5b70 con 0x7f3638004d90 2023-10-16T13:28:35.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.859+0000 7f3642ffd700 1 -- 172.21.15.16:0/634542240 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3634004240 con 0x7f3638004d90 2023-10-16T13:28:35.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.859+0000 7f3650bab700 1 -- 172.21.15.16:0/634542240 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f364c09ed10 con 0x7f3638004d90 2023-10-16T13:28:35.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.860+0000 7f3650bab700 1 -- 172.21.15.16:0/634542240 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f364c0c7560 con 0x7f3638004d90 2023-10-16T13:28:35.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.861+0000 7f3642ffd700 1 -- 172.21.15.16:0/634542240 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 111) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f36340043a0 con 0x7f3638004d90 2023-10-16T13:28:35.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.861+0000 7f3642ffd700 1 --2- 172.21.15.16:0/634542240 >> [v2:172.21.15.16:6832/3952564978,v1:172.21.15.16:6833/3952564978] conn(0x7f362c06ada0 0x7f362c06d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:28:35.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.862+0000 7f3649d9b700 1 --2- 172.21.15.16:0/634542240 >> [v2:172.21.15.16:6832/3952564978,v1:172.21.15.16:6833/3952564978] conn(0x7f362c06ada0 0x7f362c06d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:28:35.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.862+0000 7f3642ffd700 1 -- 172.21.15.16:0/634542240 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 6473+0+0 (secure 0 0 0) 0x7f3634089c40 con 0x7f3638004d90 2023-10-16T13:28:35.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.862+0000 7f3649d9b700 1 --2- 172.21.15.16:0/634542240 >> [v2:172.21.15.16:6832/3952564978,v1:172.21.15.16:6833/3952564978] conn(0x7f362c06ada0 0x7f362c06d260 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f364c09e7b0 tx=0x7f363c009040 comp rx=0 tx=0).ready entity=mgr.10485 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:28:35.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:35.866+0000 7f3642ffd700 1 -- 172.21.15.16:0/634542240 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f363408f020 con 0x7f3638004d90 2023-10-16T13:28:36.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:36.101+0000 7f3650bab700 1 -- 172.21.15.16:0/634542240 --> [v2:172.21.15.16:6832/3952564978,v1:172.21.15.16:6833/3952564978] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/test_restart", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f364c0ce980 con 0x7f362c06ada0 2023-10-16T13:28:36.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:36.144+0000 7f3642ffd700 1 -- 172.21.15.16:0/634542240 <== mgr.10485 v2:172.21.15.16:6832/3952564978 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+57 (secure 0 0 0) 0x7f363c0052b0 con 0x7f362c06ada0 2023-10-16T13:28:36.147 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule activated for path /snap_test_dir1/test_restart 2023-10-16T13:28:36.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:36.146+0000 7f3650bab700 1 -- 172.21.15.16:0/634542240 >> [v2:172.21.15.16:6832/3952564978,v1:172.21.15.16:6833/3952564978] conn(0x7f362c06ada0 msgr2=0x7f362c06d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:36.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:36.146+0000 7f3650bab700 1 --2- 172.21.15.16:0/634542240 >> [v2:172.21.15.16:6832/3952564978,v1:172.21.15.16:6833/3952564978] conn(0x7f362c06ada0 0x7f362c06d260 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f364c09e7b0 tx=0x7f363c009040 comp rx=0 tx=0).stop 2023-10-16T13:28:36.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:36.146+0000 7f3650bab700 1 -- 172.21.15.16:0/634542240 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3638004d90 msgr2=0x7f364c1f3dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:28:36.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:36.146+0000 7f3650bab700 1 --2- 172.21.15.16:0/634542240 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3638004d90 0x7f364c1f3dc0 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 crypto rx=0x7f3634011cd0 tx=0x7f363400f520 comp rx=0 tx=0).stop 2023-10-16T13:28:36.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:36.146+0000 7f3650bab700 1 -- 172.21.15.16:0/634542240 shutdown_connections 2023-10-16T13:28:36.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:36.146+0000 7f3650bab700 1 --2- 172.21.15.16:0/634542240 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f364c0b1180 0x7f364c1f2790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:36.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:36.146+0000 7f3650bab700 1 --2- 172.21.15.16:0/634542240 >> [v2:172.21.15.16:6832/3952564978,v1:172.21.15.16:6833/3952564978] conn(0x7f362c06ada0 0x7f362c06d260 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:36.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:36.146+0000 7f3650bab700 1 --2- 172.21.15.16:0/634542240 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3638004d90 0x7f364c1f3dc0 unknown :-1 s=CLOSED pgs=2145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:36.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:36.146+0000 7f3650bab700 1 --2- 172.21.15.16:0/634542240 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f364c0c5120 0x7f364c09b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:28:36.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:36.146+0000 7f3650bab700 1 -- 172.21.15.16:0/634542240 >> 172.21.15.16:0/634542240 conn(0x7f364c0d15f0 msgr2=0x7f364c0af6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:28:36.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:36.146+0000 7f3650bab700 1 -- 172.21.15.16:0/634542240 shutdown_connections 2023-10-16T13:28:36.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:28:36.146+0000 7f3650bab700 1 -- 172.21.15.16:0/634542240 wait complete. 2023-10-16T13:28:36.161 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/test_restart 2023-10-16T13:29:00.156 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:29:00.154+0000 7f6c7dd79700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:29:00.157 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:29:00.154+0000 7f6c7dd79700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:29:00.157 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:29:00.154+0000 7f6c7dd79700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:29:00.157 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:29:00.154+0000 7f6c7dd79700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:29:00.158 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:29:00.154+0000 7f6c7dd79700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:29:00.682 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:29:00.685 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:29:30.758 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:29:30.761 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:30:00.838 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:30:00.841 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:30:30.918 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:30:30.923 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:31:00.982 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:31:00.985 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:31:31.064 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:31:31.067 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:32:01.145 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:32:01.148 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:32:31.233 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:32:31.237 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:33:01.306 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:33:01.309 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:33:31.421 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:33:31.424 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:34:01.502 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:34:01.506 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:34:31.574 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:34:31.578 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:35:01.647 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:35:01.651 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:35:31.720 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:35:31.723 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:36:01.796 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:36:01.799 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:36:31.901 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:36:31.905 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:37:01.981 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:37:01.984 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:37:32.060 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:37:32.063 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:38:02.124 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:38:02.127 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:38:32.197 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:38:32.200 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:39:02.267 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:39:02.271 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:39:16.162 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:39:16.163 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/test_restart/.snap) 2023-10-16T13:39:16.187 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_29_00_UTC 2023-10-16T13:39:16.187 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_30_00_UTC 2023-10-16T13:39:16.187 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_31_00_UTC 2023-10-16T13:39:16.187 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_32_00_UTC 2023-10-16T13:39:16.188 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_33_00_UTC 2023-10-16T13:39:16.188 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_34_00_UTC 2023-10-16T13:39:16.188 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_35_00_UTC 2023-10-16T13:39:16.188 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_36_00_UTC 2023-10-16T13:39:16.188 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_37_00_UTC 2023-10-16T13:39:16.189 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_38_00_UTC 2023-10-16T13:39:16.189 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_39_00_UTC 2023-10-16T13:39:16.189 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-16-13_29_00_UTC', 'scheduled-2023-10-16-13_30_00_UTC', 'scheduled-2023-10-16-13_31_00_UTC', 'scheduled-2023-10-16-13_32_00_UTC', 'scheduled-2023-10-16-13_33_00_UTC', 'scheduled-2023-10-16-13_34_00_UTC', 'scheduled-2023-10-16-13_35_00_UTC', 'scheduled-2023-10-16-13_36_00_UTC', 'scheduled-2023-10-16-13_37_00_UTC', 'scheduled-2023-10-16-13_38_00_UTC', 'scheduled-2023-10-16-13_39_00_UTC'] 2023-10-16T13:39:16.189 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/test_restart 2023-10-16T13:39:16.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.365+0000 7f31235a6700 1 Processor -- start 2023-10-16T13:39:16.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.365+0000 7f31235a6700 1 -- start start 2023-10-16T13:39:16.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.366+0000 7f31235a6700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f311c0cb880 0x7f311c0cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:39:16.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.366+0000 7f31235a6700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f311c0cef70 con 0x7f311c06da60 2023-10-16T13:39:16.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.366+0000 7f31235a6700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f311c0ce360 con 0x7f311c15b070 2023-10-16T13:39:16.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.366+0000 7f31235a6700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f311c0ce640 con 0x7f311c0cb880 2023-10-16T13:39:16.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.366+0000 7f3121b43700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f311c06da60 0x7f311c0cf400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:39714/0 (socket says 172.21.15.16:39714) 2023-10-16T13:39:16.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.366+0000 7f3121b43700 1 -- 172.21.15.16:0/3005148094 learned_addr learned my addr 172.21.15.16:0/3005148094 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:39:16.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.367+0000 7f3120b41700 1 --2- 172.21.15.16:0/3005148094 >> v2:172.21.15.49:3300/0 conn(0x7f311c0cb880 0x7f311c0cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:39:16.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.367+0000 7f3113fff700 1 -- 172.21.15.16:0/3005148094 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 552774236 0 0) 0x7f311c0cef70 con 0x7f311c06da60 2023-10-16T13:39:16.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.367+0000 7f3113fff700 1 -- 172.21.15.16:0/3005148094 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31080035b0 con 0x7f311c06da60 2023-10-16T13:39:16.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.367+0000 7f3113fff700 1 -- 172.21.15.16:0/3005148094 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2755548336 0 0) 0x7f31080035b0 con 0x7f311c06da60 2023-10-16T13:39:16.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.367+0000 7f3113fff700 1 -- 172.21.15.16:0/3005148094 >> v2:172.21.15.49:3300/0 conn(0x7f311c0cb880 msgr2=0x7f311c0cbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:39:16.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.367+0000 7f3113fff700 1 --2- 172.21.15.16:0/3005148094 >> v2:172.21.15.49:3300/0 conn(0x7f311c0cb880 0x7f311c0cbd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:39:16.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.367+0000 7f3113fff700 1 -- 172.21.15.16:0/3005148094 >> v1:172.21.15.49:6789/0 conn(0x7f311c15b070 legacy=0x7f311c15b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:39:16.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.367+0000 7f3113fff700 1 -- 172.21.15.16:0/3005148094 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f311c0ce920 con 0x7f311c06da60 2023-10-16T13:39:16.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.367+0000 7f3113fff700 1 -- 172.21.15.16:0/3005148094 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f31140019f0 con 0x7f311c06da60 2023-10-16T13:39:16.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.367+0000 7f3113fff700 1 -- 172.21.15.16:0/3005148094 >> v1:172.21.15.16:6789/0 conn(0x7f311c06da60 legacy=0x7f311c0cf400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:39:16.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.367+0000 7f3113fff700 1 --2- 172.21.15.16:0/3005148094 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3108003a00 0x7f3108003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:39:16.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.367+0000 7f3113fff700 1 --2- 172.21.15.16:0/3005148094 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3108004400 0x7f31080048d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:39:16.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.367+0000 7f3113fff700 1 --2- 172.21.15.16:0/3005148094 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3108005170 0x7f310800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:39:16.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.367+0000 7f3113fff700 1 -- 172.21.15.16:0/3005148094 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f311c0ce640 con 0x7f3108005170 2023-10-16T13:39:16.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.367+0000 7f3113fff700 1 -- 172.21.15.16:0/3005148094 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f310800c0b0 con 0x7f3108003a00 2023-10-16T13:39:16.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.367+0000 7f3113fff700 1 -- 172.21.15.16:0/3005148094 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f310800c270 con 0x7f3108004400 2023-10-16T13:39:16.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.367+0000 7f3121b43700 1 --2- 172.21.15.16:0/3005148094 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3108005170 0x7f310800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:39:16.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.367+0000 7f3121342700 1 --2- 172.21.15.16:0/3005148094 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3108003a00 0x7f3108003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:39:16.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.367+0000 7f3120b41700 1 --2- 172.21.15.16:0/3005148094 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3108004400 0x7f31080048d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:39:16.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.367+0000 7f3121b43700 1 -- 172.21.15.16:0/3005148094 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3108004400 msgr2=0x7f31080048d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:39:16.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.367+0000 7f3121b43700 1 --2- 172.21.15.16:0/3005148094 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3108004400 0x7f31080048d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:39:16.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.367+0000 7f3121b43700 1 -- 172.21.15.16:0/3005148094 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3108003a00 msgr2=0x7f3108003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:39:16.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.367+0000 7f3121b43700 1 --2- 172.21.15.16:0/3005148094 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3108003a00 0x7f3108003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:39:16.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.367+0000 7f3121b43700 1 -- 172.21.15.16:0/3005148094 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31080035b0 con 0x7f3108005170 2023-10-16T13:39:16.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.367+0000 7f3121b43700 1 -- 172.21.15.16:0/3005148094 reap_dead start 2023-10-16T13:39:16.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.368+0000 7f3121b43700 1 --2- 172.21.15.16:0/3005148094 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3108005170 0x7f310800ba10 secure :-1 s=READY pgs=2151 cs=0 l=1 rev1=1 crypto rx=0x7f311400e7f0 tx=0x7f3108005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=409bdfb8f08b5db3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:39:16.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.368+0000 7f3113fff700 1 -- 172.21.15.16:0/3005148094 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3114010690 con 0x7f3108005170 2023-10-16T13:39:16.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.368+0000 7f3113fff700 1 -- 172.21.15.16:0/3005148094 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31140107f0 con 0x7f3108005170 2023-10-16T13:39:16.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.368+0000 7f3113fff700 1 -- 172.21.15.16:0/3005148094 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3114010ba0 con 0x7f3108005170 2023-10-16T13:39:16.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.368+0000 7f31235a6700 1 -- 172.21.15.16:0/3005148094 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3108005170 msgr2=0x7f310800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:39:16.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.368+0000 7f31235a6700 1 --2- 172.21.15.16:0/3005148094 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3108005170 0x7f310800ba10 secure :-1 s=READY pgs=2151 cs=0 l=1 rev1=1 crypto rx=0x7f311400e7f0 tx=0x7f3108005880 comp rx=0 tx=0).stop 2023-10-16T13:39:16.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.368+0000 7f31235a6700 1 -- 172.21.15.16:0/3005148094 shutdown_connections 2023-10-16T13:39:16.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.368+0000 7f31235a6700 1 --2- 172.21.15.16:0/3005148094 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3108005170 0x7f310800ba10 unknown :-1 s=CLOSED pgs=2151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:39:16.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.368+0000 7f31235a6700 1 -- 172.21.15.16:0/3005148094 >> 172.21.15.16:0/3005148094 conn(0x7f311c0b1f40 msgr2=0x7f311c0b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:39:16.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.368+0000 7f31235a6700 1 -- 172.21.15.16:0/3005148094 shutdown_connections 2023-10-16T13:39:16.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.368+0000 7f31235a6700 1 -- 172.21.15.16:0/3005148094 wait complete. 2023-10-16T13:39:16.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.370+0000 7f31235a6700 1 Processor -- start 2023-10-16T13:39:16.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.370+0000 7f31235a6700 1 -- start start 2023-10-16T13:39:16.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.370+0000 7f31235a6700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3108005170 0x7f311c151980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:39:16.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.370+0000 7f31235a6700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f311c156990 0x7f311c156e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:39:16.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.370+0000 7f31235a6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f311c155200 0x7f311c1556b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:39:16.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.370+0000 7f31235a6700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f311c157350 con 0x7f311c155200 2023-10-16T13:39:16.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.370+0000 7f31235a6700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f311c0b3940 con 0x7f311c156990 2023-10-16T13:39:16.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.370+0000 7f31235a6700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f311c0b0550 con 0x7f3108005170 2023-10-16T13:39:16.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.371+0000 7f3121b43700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f311c155200 0x7f311c1556b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:39:16.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.371+0000 7f3121b43700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f311c155200 0x7f311c1556b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50452/0 (socket says 172.21.15.16:50452) 2023-10-16T13:39:16.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.371+0000 7f3121b43700 1 -- 172.21.15.16:0/1548281701 learned_addr learned my addr 172.21.15.16:0/1548281701 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:39:16.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.371+0000 7f3121342700 1 --2- 172.21.15.16:0/1548281701 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3108005170 0x7f311c151980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:39:16.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.371+0000 7f3120b41700 1 --2- 172.21.15.16:0/1548281701 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f311c156990 0x7f311c156e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:39:16.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.371+0000 7f3121b43700 1 -- 172.21.15.16:0/1548281701 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3108005170 msgr2=0x7f311c151980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:39:16.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.371+0000 7f3121b43700 1 --2- 172.21.15.16:0/1548281701 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3108005170 0x7f311c151980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:39:16.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.371+0000 7f3121b43700 1 -- 172.21.15.16:0/1548281701 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f311c156990 msgr2=0x7f311c156e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:39:16.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.371+0000 7f3121b43700 1 --2- 172.21.15.16:0/1548281701 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f311c156990 0x7f311c156e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:39:16.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.371+0000 7f3121b43700 1 -- 172.21.15.16:0/1548281701 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f311c151ec0 con 0x7f311c155200 2023-10-16T13:39:16.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.371+0000 7f3121342700 1 --2- 172.21.15.16:0/1548281701 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3108005170 0x7f311c151980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:39:16.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.371+0000 7f3120b41700 1 --2- 172.21.15.16:0/1548281701 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f311c156990 0x7f311c156e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:39:16.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.371+0000 7f3121b43700 1 --2- 172.21.15.16:0/1548281701 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f311c155200 0x7f311c1556b0 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 crypto rx=0x7f311400ede0 tx=0x7f3114001d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:39:16.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.371+0000 7f3111ffb700 1 -- 172.21.15.16:0/1548281701 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31140019b0 con 0x7f311c155200 2023-10-16T13:39:16.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.371+0000 7f3111ffb700 1 -- 172.21.15.16:0/1548281701 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3114021070 con 0x7f311c155200 2023-10-16T13:39:16.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.371+0000 7f31235a6700 1 -- 172.21.15.16:0/1548281701 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f311c155f70 con 0x7f311c155200 2023-10-16T13:39:16.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.371+0000 7f3111ffb700 1 -- 172.21.15.16:0/1548281701 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3114011690 con 0x7f311c155200 2023-10-16T13:39:16.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.371+0000 7f31235a6700 1 -- 172.21.15.16:0/1548281701 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f311c0b01e0 con 0x7f311c155200 2023-10-16T13:39:16.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.372+0000 7f31235a6700 1 -- 172.21.15.16:0/1548281701 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f311c0b06e0 con 0x7f311c155200 2023-10-16T13:39:16.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.373+0000 7f3111ffb700 1 -- 172.21.15.16:0/1548281701 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 115) v1 ==== 89360+0+0 (secure 0 0 0) 0x7f31140117f0 con 0x7f311c155200 2023-10-16T13:39:16.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.373+0000 7f3111ffb700 1 --2- 172.21.15.16:0/1548281701 >> [v2:172.21.15.16:6832/3952564978,v1:172.21.15.16:6833/3952564978] conn(0x7f310006b070 0x7f310006d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:39:16.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.373+0000 7f3111ffb700 1 -- 172.21.15.16:0/1548281701 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 6473+0+0 (secure 0 0 0) 0x7f311408daf0 con 0x7f311c155200 2023-10-16T13:39:16.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.373+0000 7f3121342700 1 --2- 172.21.15.16:0/1548281701 >> [v2:172.21.15.16:6832/3952564978,v1:172.21.15.16:6833/3952564978] conn(0x7f310006b070 0x7f310006d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:39:16.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.374+0000 7f3121342700 1 --2- 172.21.15.16:0/1548281701 >> [v2:172.21.15.16:6832/3952564978,v1:172.21.15.16:6833/3952564978] conn(0x7f310006b070 0x7f310006d530 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f311c158190 tx=0x7f3118009040 comp rx=0 tx=0).ready entity=mgr.10485 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:39:16.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.378+0000 7f3111ffb700 1 -- 172.21.15.16:0/1548281701 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f31140587a0 con 0x7f311c155200 2023-10-16T13:39:16.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.607+0000 7f31235a6700 1 -- 172.21.15.16:0/1548281701 --> [v2:172.21.15.16:6832/3952564978,v1:172.21.15.16:6833/3952564978] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/test_restart", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f311c0728d0 con 0x7f310006b070 2023-10-16T13:39:16.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.613+0000 7f3111ffb700 1 -- 172.21.15.16:0/1548281701 <== mgr.10485 v2:172.21.15.16:6832/3952564978 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+375 (secure 0 0 0) 0x7f3118005280 con 0x7f310006b070 2023-10-16T13:39:16.615 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:39:16.616 INFO:teuthology.orchestra.run.smithi016.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T13:28:35", "first": "2023-10-16T13:29:00", "last": "2023-10-16T13:39:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": true}] 2023-10-16T13:39:16.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.615+0000 7f31235a6700 1 -- 172.21.15.16:0/1548281701 >> [v2:172.21.15.16:6832/3952564978,v1:172.21.15.16:6833/3952564978] conn(0x7f310006b070 msgr2=0x7f310006d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:39:16.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.615+0000 7f31235a6700 1 --2- 172.21.15.16:0/1548281701 >> [v2:172.21.15.16:6832/3952564978,v1:172.21.15.16:6833/3952564978] conn(0x7f310006b070 0x7f310006d530 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f311c158190 tx=0x7f3118009040 comp rx=0 tx=0).stop 2023-10-16T13:39:16.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.615+0000 7f31235a6700 1 -- 172.21.15.16:0/1548281701 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f311c155200 msgr2=0x7f311c1556b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:39:16.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.615+0000 7f31235a6700 1 --2- 172.21.15.16:0/1548281701 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f311c155200 0x7f311c1556b0 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 crypto rx=0x7f311400ede0 tx=0x7f3114001d00 comp rx=0 tx=0).stop 2023-10-16T13:39:16.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.615+0000 7f31235a6700 1 -- 172.21.15.16:0/1548281701 shutdown_connections 2023-10-16T13:39:16.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.615+0000 7f31235a6700 1 --2- 172.21.15.16:0/1548281701 >> [v2:172.21.15.16:6832/3952564978,v1:172.21.15.16:6833/3952564978] conn(0x7f310006b070 0x7f310006d530 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:39:16.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.615+0000 7f31235a6700 1 --2- 172.21.15.16:0/1548281701 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f311c155200 0x7f311c1556b0 unknown :-1 s=CLOSED pgs=2152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:39:16.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.615+0000 7f31235a6700 1 --2- 172.21.15.16:0/1548281701 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3108005170 0x7f311c151980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:39:16.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.615+0000 7f31235a6700 1 --2- 172.21.15.16:0/1548281701 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f311c156990 0x7f311c156e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:39:16.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.615+0000 7f31235a6700 1 -- 172.21.15.16:0/1548281701 >> 172.21.15.16:0/1548281701 conn(0x7f311c0b1f40 msgr2=0x7f311c153740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:39:16.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.615+0000 7f31235a6700 1 -- 172.21.15.16:0/1548281701 shutdown_connections 2023-10-16T13:39:16.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.615+0000 7f31235a6700 1 -- 172.21.15.16:0/1548281701 wait complete. 2023-10-16T13:39:16.629 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T13:28:35", "first": "2023-10-16T13:29:00", "last": "2023-10-16T13:39:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": true}] 2023-10-16T13:39:16.630 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/test_restart', 'rel_path': '/snap_test_dir1/test_restart', 'schedule': '1m', 'retention': {}, 'start': '2023-10-16T00:00:00', 'created': '2023-10-16T13:28:35', 'first': '2023-10-16T13:29:00', 'last': '2023-10-16T13:39:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': True} 2023-10-16T13:39:16.630 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-10-16T13:39:16.630 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-10-16T13:39:16.631 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-10-16T13:39:16.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.753+0000 7f296367d700 1 Processor -- start 2023-10-16T13:39:16.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.753+0000 7f296367d700 1 -- start start 2023-10-16T13:39:16.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.753+0000 7f296367d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f295c160d90 0x7f295c06a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:39:16.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.754+0000 7f296367d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f295c06da60 0x7f295c06ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:39:16.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.754+0000 7f296367d700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f295c0ced00 con 0x7f295c06e450 2023-10-16T13:39:16.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.754+0000 7f296367d700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f295c162180 con 0x7f295c160d90 2023-10-16T13:39:16.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.754+0000 7f296367d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f295c0caa40 con 0x7f295c06da60 2023-10-16T13:39:16.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.754+0000 7f2961419700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f295c160d90 0x7f295c06a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:39:16.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.754+0000 7f2961419700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f295c160d90 0x7f295c06a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50456/0 (socket says 172.21.15.16:50456) 2023-10-16T13:39:16.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.754+0000 7f2961419700 1 -- 172.21.15.16:0/2877743439 learned_addr learned my addr 172.21.15.16:0/2877743439 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:39:16.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.754+0000 7f2961c1a700 1 --2- 172.21.15.16:0/2877743439 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f295c06da60 0x7f295c06ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:39:16.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.754+0000 7f2961419700 1 -- 172.21.15.16:0/2877743439 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f295c06da60 msgr2=0x7f295c06ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:39:16.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.754+0000 7f2961419700 1 --2- 172.21.15.16:0/2877743439 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f295c06da60 0x7f295c06ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:39:16.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.754+0000 7f2961419700 1 -- 172.21.15.16:0/2877743439 >> v1:172.21.15.49:6789/0 conn(0x7f295c06e450 legacy=0x7f295c0cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:39:16.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.754+0000 7f2961419700 1 -- 172.21.15.16:0/2877743439 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f295c161570 con 0x7f295c160d90 2023-10-16T13:39:16.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.754+0000 7f2961419700 1 --2- 172.21.15.16:0/2877743439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f295c160d90 0x7f295c06a7e0 secure :-1 s=READY pgs=2153 cs=0 l=1 rev1=1 crypto rx=0x7f294c0098f0 tx=0x7f294c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90a5ff8bc1c2e263 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:39:16.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.754+0000 7f2953fff700 1 -- 172.21.15.16:0/2877743439 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f294c011830 con 0x7f295c160d90 2023-10-16T13:39:16.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.754+0000 7f2953fff700 1 -- 172.21.15.16:0/2877743439 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f294c011990 con 0x7f295c160d90 2023-10-16T13:39:16.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.754+0000 7f2953fff700 1 -- 172.21.15.16:0/2877743439 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f294c011d20 con 0x7f295c160d90 2023-10-16T13:39:16.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.755+0000 7f296367d700 1 -- 172.21.15.16:0/2877743439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f295c160d90 msgr2=0x7f295c06a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:39:16.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.755+0000 7f296367d700 1 --2- 172.21.15.16:0/2877743439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f295c160d90 0x7f295c06a7e0 secure :-1 s=READY pgs=2153 cs=0 l=1 rev1=1 crypto rx=0x7f294c0098f0 tx=0x7f294c009c00 comp rx=0 tx=0).stop 2023-10-16T13:39:16.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.755+0000 7f296367d700 1 -- 172.21.15.16:0/2877743439 shutdown_connections 2023-10-16T13:39:16.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.755+0000 7f296367d700 1 --2- 172.21.15.16:0/2877743439 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f295c06da60 0x7f295c06ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:39:16.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.755+0000 7f296367d700 1 --2- 172.21.15.16:0/2877743439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f295c160d90 0x7f295c06a7e0 unknown :-1 s=CLOSED pgs=2153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:39:16.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.755+0000 7f296367d700 1 -- 172.21.15.16:0/2877743439 >> 172.21.15.16:0/2877743439 conn(0x7f295c0b1f40 msgr2=0x7f295c0b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:39:16.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.755+0000 7f296367d700 1 -- 172.21.15.16:0/2877743439 shutdown_connections 2023-10-16T13:39:16.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.755+0000 7f296367d700 1 -- 172.21.15.16:0/2877743439 wait complete. 2023-10-16T13:39:16.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.756+0000 7f296367d700 1 Processor -- start 2023-10-16T13:39:16.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.756+0000 7f296367d700 1 -- start start 2023-10-16T13:39:16.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.757+0000 7f296367d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f295c06da60 0x7f295c1541a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:39:16.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.757+0000 7f296367d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f295c06e450 0x7f295c151930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:39:16.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.757+0000 7f296367d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f295c160d90 0x7f295c0c4250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:39:16.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.757+0000 7f296367d700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f295c0c4ea0 con 0x7f295c06da60 2023-10-16T13:39:16.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.757+0000 7f296367d700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f295c0c4bc0 con 0x7f295c160d90 2023-10-16T13:39:16.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.757+0000 7f296367d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f295c1536c0 con 0x7f295c06e450 2023-10-16T13:39:16.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.757+0000 7f2961419700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f295c06da60 0x7f295c1541a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:39:16.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.757+0000 7f2961419700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f295c06da60 0x7f295c1541a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50462/0 (socket says 172.21.15.16:50462) 2023-10-16T13:39:16.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.757+0000 7f2961419700 1 -- 172.21.15.16:0/2476954854 learned_addr learned my addr 172.21.15.16:0/2476954854 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:39:16.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.757+0000 7f2961c1a700 1 --2- 172.21.15.16:0/2476954854 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f295c160d90 0x7f295c0c4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:39:16.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.757+0000 7f2960c18700 1 --2- 172.21.15.16:0/2476954854 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f295c06e450 0x7f295c151930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:39:16.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.757+0000 7f2961419700 1 -- 172.21.15.16:0/2476954854 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f295c06e450 msgr2=0x7f295c151930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:39:16.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.757+0000 7f2961419700 1 --2- 172.21.15.16:0/2476954854 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f295c06e450 0x7f295c151930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:39:16.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.757+0000 7f2961419700 1 -- 172.21.15.16:0/2476954854 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f295c160d90 msgr2=0x7f295c0c4250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:39:16.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.757+0000 7f2961419700 1 --2- 172.21.15.16:0/2476954854 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f295c160d90 0x7f295c0c4250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:39:16.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.757+0000 7f2961419700 1 -- 172.21.15.16:0/2476954854 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f294c009550 con 0x7f295c06da60 2023-10-16T13:39:16.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.758+0000 7f2961419700 1 --2- 172.21.15.16:0/2476954854 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f295c06da60 0x7f295c1541a0 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 crypto rx=0x7f294c000c00 tx=0x7f294c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:39:16.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.758+0000 7f2951ffb700 1 -- 172.21.15.16:0/2476954854 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f294c004400 con 0x7f295c06da60 2023-10-16T13:39:16.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.758+0000 7f2951ffb700 1 -- 172.21.15.16:0/2476954854 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f294c01f070 con 0x7f295c06da60 2023-10-16T13:39:16.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.758+0000 7f296367d700 1 -- 172.21.15.16:0/2476954854 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f295c1539a0 con 0x7f295c06da60 2023-10-16T13:39:16.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.758+0000 7f2951ffb700 1 -- 172.21.15.16:0/2476954854 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f294c0038e0 con 0x7f295c06da60 2023-10-16T13:39:16.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.758+0000 7f296367d700 1 -- 172.21.15.16:0/2476954854 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f295c0c4790 con 0x7f295c06da60 2023-10-16T13:39:16.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.759+0000 7f296367d700 1 -- 172.21.15.16:0/2476954854 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f295c0c4980 con 0x7f295c06da60 2023-10-16T13:39:16.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.759+0000 7f2951ffb700 1 -- 172.21.15.16:0/2476954854 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 115) v1 ==== 89360+0+0 (secure 0 0 0) 0x7f294c021030 con 0x7f295c06da60 2023-10-16T13:39:16.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.760+0000 7f2951ffb700 1 --2- 172.21.15.16:0/2476954854 >> [v2:172.21.15.16:6832/3952564978,v1:172.21.15.16:6833/3952564978] conn(0x7f294006b070 0x7f294006d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:39:16.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.760+0000 7f2951ffb700 1 -- 172.21.15.16:0/2476954854 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 6473+0+0 (secure 0 0 0) 0x7f294c08ae30 con 0x7f295c06da60 2023-10-16T13:39:16.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.760+0000 7f2960c18700 1 --2- 172.21.15.16:0/2476954854 >> [v2:172.21.15.16:6832/3952564978,v1:172.21.15.16:6833/3952564978] conn(0x7f294006b070 0x7f294006d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:39:16.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.760+0000 7f2960c18700 1 --2- 172.21.15.16:0/2476954854 >> [v2:172.21.15.16:6832/3952564978,v1:172.21.15.16:6833/3952564978] conn(0x7f294006b070 0x7f294006d530 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f295c06b610 tx=0x7f2958009040 comp rx=0 tx=0).ready entity=mgr.10485 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:39:16.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:16.764+0000 7f2951ffb700 1 -- 172.21.15.16:0/2476954854 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f295c0c4980 con 0x7f295c06da60 2023-10-16T13:39:17.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:17.028+0000 7f296367d700 1 -- 172.21.15.16:0/2476954854 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f295c153ec0 con 0x7f295c06da60 2023-10-16T13:39:17.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:17.031+0000 7f2951ffb700 1 -- 172.21.15.16:0/2476954854 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v115) v1 ==== 74+0+171237 (secure 0 0 0) 0x7f295c153ec0 con 0x7f295c06da60 2023-10-16T13:39:17.036 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:39:17.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:17.034+0000 7f296367d700 1 -- 172.21.15.16:0/2476954854 >> [v2:172.21.15.16:6832/3952564978,v1:172.21.15.16:6833/3952564978] conn(0x7f294006b070 msgr2=0x7f294006d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:39:17.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:17.034+0000 7f296367d700 1 --2- 172.21.15.16:0/2476954854 >> [v2:172.21.15.16:6832/3952564978,v1:172.21.15.16:6833/3952564978] conn(0x7f294006b070 0x7f294006d530 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f295c06b610 tx=0x7f2958009040 comp rx=0 tx=0).stop 2023-10-16T13:39:17.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:17.034+0000 7f296367d700 1 -- 172.21.15.16:0/2476954854 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f295c06da60 msgr2=0x7f295c1541a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:39:17.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:17.034+0000 7f296367d700 1 --2- 172.21.15.16:0/2476954854 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f295c06da60 0x7f295c1541a0 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 crypto rx=0x7f294c000c00 tx=0x7f294c004750 comp rx=0 tx=0).stop 2023-10-16T13:39:17.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:17.034+0000 7f296367d700 1 -- 172.21.15.16:0/2476954854 shutdown_connections 2023-10-16T13:39:17.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:17.034+0000 7f296367d700 1 --2- 172.21.15.16:0/2476954854 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f295c160d90 0x7f295c0c4250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:39:17.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:17.034+0000 7f296367d700 1 --2- 172.21.15.16:0/2476954854 >> [v2:172.21.15.16:6832/3952564978,v1:172.21.15.16:6833/3952564978] conn(0x7f294006b070 0x7f294006d530 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:39:17.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:17.034+0000 7f296367d700 1 --2- 172.21.15.16:0/2476954854 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f295c06da60 0x7f295c1541a0 unknown :-1 s=CLOSED pgs=2154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:39:17.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:17.034+0000 7f296367d700 1 --2- 172.21.15.16:0/2476954854 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f295c06e450 0x7f295c151930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:39:17.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:17.034+0000 7f296367d700 1 -- 172.21.15.16:0/2476954854 >> 172.21.15.16:0/2476954854 conn(0x7f295c0b1f40 msgr2=0x7f295c0cc140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:39:17.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:17.034+0000 7f296367d700 1 -- 172.21.15.16:0/2476954854 shutdown_connections 2023-10-16T13:39:17.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:39:17.034+0000 7f296367d700 1 -- 172.21.15.16:0/2476954854 wait complete. 2023-10-16T13:39:17.048 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":115,"active_gid":10485,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3952564978},{"type":"v1","addr":"172.21.15.16:6833","nonce":3952564978}]},"active_addr":"172.21.15.16:6833/3952564978","active_change":"2023-10-16T13:28:34.632126+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":9664,"name":"x","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful","snap_schedule"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":191,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":1641955134}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":857902049}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":2554840790}]},{"name":"snap_schedule","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":360056771}]},{"name":"snap_schedule","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":1548506671}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":3520685377}]}]} 2023-10-16T13:39:17.053 DEBUG:tasks.cephfs.test_snap_schedules:restarting active mgr: y 2023-10-16T13:39:17.053 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-10-16T13:39:17.053 INFO:tasks.ceph.mgr.y:Stopping old one... 2023-10-16T13:39:17.054 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-10-16T13:39:17.054 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T13:39:17.083 INFO:tasks.ceph.mgr.y:Stopped 2023-10-16T13:39:17.083 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2023-10-16T13:39:17.125 INFO:tasks.ceph.mgr.y:Started 2023-10-16T13:39:17.306 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:39:17.303+0000 7f30807b8200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T13:39:17.374 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:39:17.372+0000 7f30807b8200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T13:39:17.449 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:39:17.447+0000 7f30807b8200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T13:39:17.530 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:39:17.528+0000 7f30807b8200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T13:39:17.677 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:39:17.675+0000 7f30807b8200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T13:39:18.017 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:39:18.015+0000 7f30807b8200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T13:39:18.139 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:39:18.136+0000 7f30807b8200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T13:39:18.215 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:39:18.213+0000 7f30807b8200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T13:39:18.299 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:39:18.297+0000 7f30807b8200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T13:39:18.687 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:39:18.684+0000 7f30807b8200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T13:39:19.293 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:39:19.291+0000 7f30807b8200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T13:39:19.366 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:39:19.364+0000 7f30807b8200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T13:39:20.030 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:39:20.027+0000 7f30807b8200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:39:20.112 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:39:20.110+0000 7f30807b8200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T13:39:20.265 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:39:20.263+0000 7f30807b8200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:39:20.506 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:39:20.503+0000 7f30807b8200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T13:39:20.584 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:39:20.582+0000 7f30807b8200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T13:39:20.862 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:39:20.859+0000 7f30807b8200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T13:39:21.096 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:39:21.092+0000 7f30807b8200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T13:39:21.254 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:39:21.251+0000 7f30807b8200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T13:39:21.374 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:39:21.371+0000 7f30807b8200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T13:39:21.539 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:39:21.536+0000 7f30807b8200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T13:39:21.814 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:39:21.810+0000 7f30807b8200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T13:39:21.984 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:39:21.981+0000 7f30807b8200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T13:39:22.547 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:39:22.543+0000 7f30807b8200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T13:39:22.713 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:39:22.710+0000 7f2ff2f92700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:39:22.713 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:39:22.710+0000 7f2ff2f92700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:39:22.714 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:39:22.710+0000 7f2ff2f92700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:39:22.714 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:39:22.710+0000 7f2ff2f92700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:39:22.714 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:39:22.710+0000 7f2ff2f92700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:39:22.715 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:39:22.710+0000 7f2ff4f96700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:39:22.715 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:39:22.710+0000 7f2ff4f96700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:39:22.715 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:39:22.710+0000 7f2ff4f96700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:39:22.715 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:39:22.710+0000 7f2ff4f96700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:39:22.716 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:39:22.710+0000 7f2ff4f96700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:39:32.338 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:39:32.341 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:40:00.710 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:40:00.707+0000 7f2ffdfa8700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:40:00.710 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:40:00.707+0000 7f2ffdfa8700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:40:00.711 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:40:00.707+0000 7f2ffdfa8700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:40:00.711 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:40:00.707+0000 7f2ffdfa8700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:40:00.711 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:40:00.707+0000 7f2ffdfa8700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:40:02.397 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:40:02.400 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:40:32.465 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:40:32.468 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:41:02.534 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:41:02.538 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:41:32.599 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:41:32.603 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:42:02.679 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:42:02.683 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:42:32.751 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:42:32.754 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:43:02.824 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:43:02.827 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:43:32.895 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:43:32.898 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:44:02.966 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:44:02.969 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:44:17.127 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/test_restart 1m 2023-10-16T13:44:17.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.267+0000 7f716f8eb700 1 Processor -- start 2023-10-16T13:44:17.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.267+0000 7f716f8eb700 1 -- start start 2023-10-16T13:44:17.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.268+0000 7f716f8eb700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f71680cbdd0 0x7f71680713e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:17.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.268+0000 7f716f8eb700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7168071920 0x7f71680cd3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:17.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.268+0000 7f716f8eb700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71680cb940 con 0x7f7168159e40 2023-10-16T13:44:17.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.268+0000 7f716f8eb700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71680ceb60 con 0x7f71680cbdd0 2023-10-16T13:44:17.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.268+0000 7f716f8eb700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71680ce880 con 0x7f7168071920 2023-10-16T13:44:17.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.268+0000 7f716ce86700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f71680cbdd0 0x7f71680713e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:17.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.268+0000 7f716ce86700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f71680cbdd0 0x7f71680713e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50830/0 (socket says 172.21.15.16:50830) 2023-10-16T13:44:17.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.268+0000 7f716ce86700 1 -- 172.21.15.16:0/911157289 learned_addr learned my addr 172.21.15.16:0/911157289 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:17.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.268+0000 7f716ce86700 1 -- 172.21.15.16:0/911157289 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7168071920 msgr2=0x7f71680cd3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:17.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.268+0000 7f716ce86700 1 --2- 172.21.15.16:0/911157289 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7168071920 0x7f71680cd3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:17.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.268+0000 7f716ce86700 1 -- 172.21.15.16:0/911157289 >> v1:172.21.15.16:6789/0 conn(0x7f7168159e40 legacy=0x7f7168070cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:17.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.268+0000 7f716ce86700 1 -- 172.21.15.16:0/911157289 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71680cb2f0 con 0x7f71680cbdd0 2023-10-16T13:44:17.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.269+0000 7f716ce86700 1 --2- 172.21.15.16:0/911157289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71680cbdd0 0x7f71680713e0 secure :-1 s=READY pgs=2173 cs=0 l=1 rev1=1 crypto rx=0x7f716400bb90 tx=0x7f716400bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7f6879554a77b9b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:17.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.269+0000 7f715ffff700 1 -- 172.21.15.16:0/911157289 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f716400dd50 con 0x7f71680cbdd0 2023-10-16T13:44:17.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.269+0000 7f715ffff700 1 -- 172.21.15.16:0/911157289 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7164010470 con 0x7f71680cbdd0 2023-10-16T13:44:17.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.269+0000 7f715ffff700 1 -- 172.21.15.16:0/911157289 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7164010800 con 0x7f71680cbdd0 2023-10-16T13:44:17.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.269+0000 7f716f8eb700 1 -- 172.21.15.16:0/911157289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71680cbdd0 msgr2=0x7f71680713e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:17.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.269+0000 7f716f8eb700 1 --2- 172.21.15.16:0/911157289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71680cbdd0 0x7f71680713e0 secure :-1 s=READY pgs=2173 cs=0 l=1 rev1=1 crypto rx=0x7f716400bb90 tx=0x7f716400bea0 comp rx=0 tx=0).stop 2023-10-16T13:44:17.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.269+0000 7f716f8eb700 1 -- 172.21.15.16:0/911157289 shutdown_connections 2023-10-16T13:44:17.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.269+0000 7f716f8eb700 1 --2- 172.21.15.16:0/911157289 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7168071920 0x7f71680cd3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:17.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.269+0000 7f716f8eb700 1 --2- 172.21.15.16:0/911157289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71680cbdd0 0x7f71680713e0 unknown :-1 s=CLOSED pgs=2173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:17.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.269+0000 7f716f8eb700 1 -- 172.21.15.16:0/911157289 >> 172.21.15.16:0/911157289 conn(0x7f716806cdb0 msgr2=0x7f716806d1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:17.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.270+0000 7f716f8eb700 1 -- 172.21.15.16:0/911157289 shutdown_connections 2023-10-16T13:44:17.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.270+0000 7f716f8eb700 1 -- 172.21.15.16:0/911157289 wait complete. 2023-10-16T13:44:17.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.271+0000 7f716f8eb700 1 Processor -- start 2023-10-16T13:44:17.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.271+0000 7f716f8eb700 1 -- start start 2023-10-16T13:44:17.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.271+0000 7f716f8eb700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71680cbdd0 0x7f7168152860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:17.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.271+0000 7f716f8eb700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7168159e40 0x7f71680c7ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:17.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.271+0000 7f716f8eb700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f71680c80e0 0x7f71680c3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:17.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.271+0000 7f716f8eb700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71680c8590 con 0x7f71680cbdd0 2023-10-16T13:44:17.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.271+0000 7f716f8eb700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f716815acd0 con 0x7f7168159e40 2023-10-16T13:44:17.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.271+0000 7f716f8eb700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71680c4d60 con 0x7f71680c80e0 2023-10-16T13:44:17.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.272+0000 7f716d687700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71680cbdd0 0x7f7168152860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:17.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.272+0000 7f716d687700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71680cbdd0 0x7f7168152860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50834/0 (socket says 172.21.15.16:50834) 2023-10-16T13:44:17.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.272+0000 7f716d687700 1 -- 172.21.15.16:0/3416900268 learned_addr learned my addr 172.21.15.16:0/3416900268 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:17.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.272+0000 7f716de88700 1 --2- 172.21.15.16:0/3416900268 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f71680c80e0 0x7f71680c3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:17.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.272+0000 7f716ce86700 1 --2- 172.21.15.16:0/3416900268 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7168159e40 0x7f71680c7ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:17.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.272+0000 7f716d687700 1 -- 172.21.15.16:0/3416900268 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f71680c80e0 msgr2=0x7f71680c3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:17.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.272+0000 7f716d687700 1 --2- 172.21.15.16:0/3416900268 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f71680c80e0 0x7f71680c3eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:17.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.272+0000 7f716d687700 1 -- 172.21.15.16:0/3416900268 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7168159e40 msgr2=0x7f71680c7ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:17.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.272+0000 7f716d687700 1 --2- 172.21.15.16:0/3416900268 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7168159e40 0x7f71680c7ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:17.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.272+0000 7f716d687700 1 -- 172.21.15.16:0/3416900268 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f716400b7f0 con 0x7f71680cbdd0 2023-10-16T13:44:17.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.272+0000 7f716de88700 1 --2- 172.21.15.16:0/3416900268 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f71680c80e0 0x7f71680c3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:44:17.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.272+0000 7f716ce86700 1 --2- 172.21.15.16:0/3416900268 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7168159e40 0x7f71680c7ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:44:17.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.272+0000 7f716d687700 1 --2- 172.21.15.16:0/3416900268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71680cbdd0 0x7f7168152860 secure :-1 s=READY pgs=2174 cs=0 l=1 rev1=1 crypto rx=0x7f715800c450 tx=0x7f715800c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:17.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.272+0000 7f715dffb700 1 -- 172.21.15.16:0/3416900268 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71580052d0 con 0x7f71680cbdd0 2023-10-16T13:44:17.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.272+0000 7f715dffb700 1 -- 172.21.15.16:0/3416900268 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7158005430 con 0x7f71680cbdd0 2023-10-16T13:44:17.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.272+0000 7f716f8eb700 1 -- 172.21.15.16:0/3416900268 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71680c5040 con 0x7f71680cbdd0 2023-10-16T13:44:17.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.272+0000 7f715dffb700 1 -- 172.21.15.16:0/3416900268 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71580038c0 con 0x7f71680cbdd0 2023-10-16T13:44:17.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.272+0000 7f716f8eb700 1 -- 172.21.15.16:0/3416900268 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71680c5320 con 0x7f71680cbdd0 2023-10-16T13:44:17.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.273+0000 7f716f8eb700 1 -- 172.21.15.16:0/3416900268 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f716815ab30 con 0x7f71680cbdd0 2023-10-16T13:44:17.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.274+0000 7f715dffb700 1 -- 172.21.15.16:0/3416900268 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 121) v1 ==== 89360+0+0 (secure 0 0 0) 0x7f7158003a20 con 0x7f71680cbdd0 2023-10-16T13:44:17.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.274+0000 7f715dffb700 1 --2- 172.21.15.16:0/3416900268 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f714c06b070 0x7f714c06d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:17.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.274+0000 7f715dffb700 1 -- 172.21.15.16:0/3416900268 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 6817+0+0 (secure 0 0 0) 0x7f7158088f30 con 0x7f71680cbdd0 2023-10-16T13:44:17.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.274+0000 7f716ce86700 1 --2- 172.21.15.16:0/3416900268 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f714c06b070 0x7f714c06d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:17.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.275+0000 7f716ce86700 1 --2- 172.21.15.16:0/3416900268 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f714c06b070 0x7f714c06d530 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f71680cc230 tx=0x7f7164013000 comp rx=0 tx=0).ready entity=mgr.10578 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:17.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.279+0000 7f715dffb700 1 -- 172.21.15.16:0/3416900268 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f716815ab30 con 0x7f71680cbdd0 2023-10-16T13:44:17.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.513+0000 7f716f8eb700 1 -- 172.21.15.16:0/3416900268 --> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/test_restart", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f71680aed90 con 0x7f714c06b070 2023-10-16T13:44:17.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.563+0000 7f715dffb700 1 -- 172.21.15.16:0/3416900268 <== mgr.10578 v2:172.21.15.16:6832/371380443 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f7164020040 con 0x7f714c06b070 2023-10-16T13:44:17.566 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule deactivated for path /snap_test_dir1/test_restart 2023-10-16T13:44:17.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.566+0000 7f716f8eb700 1 -- 172.21.15.16:0/3416900268 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f714c06b070 msgr2=0x7f714c06d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:17.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.566+0000 7f716f8eb700 1 --2- 172.21.15.16:0/3416900268 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f714c06b070 0x7f714c06d530 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f71680cc230 tx=0x7f7164013000 comp rx=0 tx=0).stop 2023-10-16T13:44:17.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.566+0000 7f716f8eb700 1 -- 172.21.15.16:0/3416900268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71680cbdd0 msgr2=0x7f7168152860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:17.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.566+0000 7f716f8eb700 1 --2- 172.21.15.16:0/3416900268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71680cbdd0 0x7f7168152860 secure :-1 s=READY pgs=2174 cs=0 l=1 rev1=1 crypto rx=0x7f715800c450 tx=0x7f715800c810 comp rx=0 tx=0).stop 2023-10-16T13:44:17.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.566+0000 7f716f8eb700 1 -- 172.21.15.16:0/3416900268 shutdown_connections 2023-10-16T13:44:17.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.566+0000 7f716f8eb700 1 --2- 172.21.15.16:0/3416900268 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f714c06b070 0x7f714c06d530 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:17.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.566+0000 7f716f8eb700 1 --2- 172.21.15.16:0/3416900268 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f71680c80e0 0x7f71680c3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:17.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.566+0000 7f716f8eb700 1 --2- 172.21.15.16:0/3416900268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71680cbdd0 0x7f7168152860 unknown :-1 s=CLOSED pgs=2174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:17.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.566+0000 7f716f8eb700 1 --2- 172.21.15.16:0/3416900268 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7168159e40 0x7f71680c7ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:17.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.566+0000 7f716f8eb700 1 -- 172.21.15.16:0/3416900268 >> 172.21.15.16:0/3416900268 conn(0x7f716806cdb0 msgr2=0x7f71680cdf10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:17.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.566+0000 7f716f8eb700 1 -- 172.21.15.16:0/3416900268 shutdown_connections 2023-10-16T13:44:17.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.566+0000 7f716f8eb700 1 -- 172.21.15.16:0/3416900268 wait complete. 2023-10-16T13:44:17.580 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/test_restart 2023-10-16T13:44:17.580 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:44:17.581 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/test_restart/.snap) 2023-10-16T13:44:17.600 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_29_00_UTC 2023-10-16T13:44:17.601 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_30_00_UTC 2023-10-16T13:44:17.601 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_31_00_UTC 2023-10-16T13:44:17.602 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_32_00_UTC 2023-10-16T13:44:17.602 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_33_00_UTC 2023-10-16T13:44:17.602 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_34_00_UTC 2023-10-16T13:44:17.603 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_35_00_UTC 2023-10-16T13:44:17.603 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_36_00_UTC 2023-10-16T13:44:17.604 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_37_00_UTC 2023-10-16T13:44:17.604 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_38_00_UTC 2023-10-16T13:44:17.604 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_39_00_UTC 2023-10-16T13:44:17.605 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_40_00_UTC 2023-10-16T13:44:17.605 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_41_00_UTC 2023-10-16T13:44:17.605 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_42_00_UTC 2023-10-16T13:44:17.606 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_43_00_UTC 2023-10-16T13:44:17.606 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_44_00_UTC 2023-10-16T13:44:17.607 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-16-13_29_00_UTC', 'scheduled-2023-10-16-13_30_00_UTC', 'scheduled-2023-10-16-13_31_00_UTC', 'scheduled-2023-10-16-13_32_00_UTC', 'scheduled-2023-10-16-13_33_00_UTC', 'scheduled-2023-10-16-13_34_00_UTC', 'scheduled-2023-10-16-13_35_00_UTC', 'scheduled-2023-10-16-13_36_00_UTC', 'scheduled-2023-10-16-13_37_00_UTC', 'scheduled-2023-10-16-13_38_00_UTC', 'scheduled-2023-10-16-13_39_00_UTC', 'scheduled-2023-10-16-13_40_00_UTC', 'scheduled-2023-10-16-13_41_00_UTC', 'scheduled-2023-10-16-13_42_00_UTC', 'scheduled-2023-10-16-13_43_00_UTC', 'scheduled-2023-10-16-13_44_00_UTC'] 2023-10-16T13:44:17.607 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/test_restart 2023-10-16T13:44:17.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.767+0000 7f8bd9d10700 1 Processor -- start 2023-10-16T13:44:17.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.767+0000 7f8bd9d10700 1 -- start start 2023-10-16T13:44:17.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.768+0000 7f8bd9d10700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8bd415b070 0x7f8bd415b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:17.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.768+0000 7f8bd9d10700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8bd40cb880 0x7f8bd40cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:17.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.768+0000 7f8bd9d10700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f8bd406da60 0x7f8bd406def0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:17.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.768+0000 7f8bd9d10700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bd406e430 con 0x7f8bd415b070 2023-10-16T13:44:17.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.768+0000 7f8bd9d10700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bd40cef70 con 0x7f8bd406da60 2023-10-16T13:44:17.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.768+0000 7f8bd9d10700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bd40ce360 con 0x7f8bd40cb880 2023-10-16T13:44:17.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.768+0000 7f8bd37fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8bd415b070 0x7f8bd415b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:17.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.768+0000 7f8bd37fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8bd415b070 0x7f8bd415b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50842/0 (socket says 172.21.15.16:50842) 2023-10-16T13:44:17.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.768+0000 7f8bd37fe700 1 -- 172.21.15.16:0/530720130 learned_addr learned my addr 172.21.15.16:0/530720130 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:17.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.768+0000 7f8bd2ffd700 1 --2- 172.21.15.16:0/530720130 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8bd40cb880 0x7f8bd40cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:17.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.768+0000 7f8bd3fff700 1 --2- 172.21.15.16:0/530720130 >> v2:172.21.15.49:3300/0 conn(0x7f8bd406da60 0x7f8bd406def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:17.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.768+0000 7f8bd37fe700 1 -- 172.21.15.16:0/530720130 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8bd40cb880 msgr2=0x7f8bd40cbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:17.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.768+0000 7f8bd37fe700 1 --2- 172.21.15.16:0/530720130 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8bd40cb880 0x7f8bd40cbd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:17.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.768+0000 7f8bd37fe700 1 -- 172.21.15.16:0/530720130 >> v2:172.21.15.49:3300/0 conn(0x7f8bd406da60 msgr2=0x7f8bd406def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:17.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.768+0000 7f8bd37fe700 1 --2- 172.21.15.16:0/530720130 >> v2:172.21.15.49:3300/0 conn(0x7f8bd406da60 0x7f8bd406def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:17.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.768+0000 7f8bd37fe700 1 -- 172.21.15.16:0/530720130 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bd40ce640 con 0x7f8bd415b070 2023-10-16T13:44:17.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.768+0000 7f8bd3fff700 1 --2- 172.21.15.16:0/530720130 >> v2:172.21.15.49:3300/0 conn(0x7f8bd406da60 0x7f8bd406def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:44:17.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.769+0000 7f8bd37fe700 1 --2- 172.21.15.16:0/530720130 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8bd415b070 0x7f8bd415b450 secure :-1 s=READY pgs=2175 cs=0 l=1 rev1=1 crypto rx=0x7f8bd4072fc0 tx=0x7f8bbc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=138e8ed71a4da4dc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:17.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.769+0000 7f8bd27fc700 1 -- 172.21.15.16:0/530720130 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bbc00f8a0 con 0x7f8bd415b070 2023-10-16T13:44:17.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.769+0000 7f8bd27fc700 1 -- 172.21.15.16:0/530720130 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bbc00fa00 con 0x7f8bd415b070 2023-10-16T13:44:17.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.769+0000 7f8bd27fc700 1 -- 172.21.15.16:0/530720130 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bbc00fd90 con 0x7f8bd415b070 2023-10-16T13:44:17.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.769+0000 7f8bd9d10700 1 -- 172.21.15.16:0/530720130 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8bd415b070 msgr2=0x7f8bd415b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:17.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.769+0000 7f8bd9d10700 1 --2- 172.21.15.16:0/530720130 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8bd415b070 0x7f8bd415b450 secure :-1 s=READY pgs=2175 cs=0 l=1 rev1=1 crypto rx=0x7f8bd4072fc0 tx=0x7f8bbc009bd0 comp rx=0 tx=0).stop 2023-10-16T13:44:17.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.769+0000 7f8bd9d10700 1 -- 172.21.15.16:0/530720130 shutdown_connections 2023-10-16T13:44:17.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.769+0000 7f8bd9d10700 1 --2- 172.21.15.16:0/530720130 >> v2:172.21.15.49:3300/0 conn(0x7f8bd406da60 0x7f8bd406def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:17.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.769+0000 7f8bd9d10700 1 --2- 172.21.15.16:0/530720130 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8bd415b070 0x7f8bd415b450 unknown :-1 s=CLOSED pgs=2175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:17.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.769+0000 7f8bd9d10700 1 --2- 172.21.15.16:0/530720130 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8bd40cb880 0x7f8bd40cbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:17.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.769+0000 7f8bd9d10700 1 -- 172.21.15.16:0/530720130 >> 172.21.15.16:0/530720130 conn(0x7f8bd40b1f40 msgr2=0x7f8bd40b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:17.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.769+0000 7f8bd9d10700 1 -- 172.21.15.16:0/530720130 shutdown_connections 2023-10-16T13:44:17.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.769+0000 7f8bd9d10700 1 -- 172.21.15.16:0/530720130 wait complete. 2023-10-16T13:44:17.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.771+0000 7f8bd9d10700 1 Processor -- start 2023-10-16T13:44:17.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.771+0000 7f8bd9d10700 1 -- start start 2023-10-16T13:44:17.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.771+0000 7f8bd9d10700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8bd406da60 0x7f8bd40aecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:17.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.771+0000 7f8bd9d10700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8bd40cb880 0x7f8bd40acf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:17.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.772+0000 7f8bd9d10700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8bd415b070 0x7f8bd40ccb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:17.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.772+0000 7f8bd9d10700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bd414a860 con 0x7f8bd40cb880 2023-10-16T13:44:17.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.772+0000 7f8bd9d10700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bd4149be0 con 0x7f8bd415b070 2023-10-16T13:44:17.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.772+0000 7f8bd9d10700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bd4149ec0 con 0x7f8bd406da60 2023-10-16T13:44:17.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.772+0000 7f8bd2ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8bd40cb880 0x7f8bd40acf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:17.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.772+0000 7f8bd2ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8bd40cb880 0x7f8bd40acf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50850/0 (socket says 172.21.15.16:50850) 2023-10-16T13:44:17.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.772+0000 7f8bd2ffd700 1 -- 172.21.15.16:0/2149997783 learned_addr learned my addr 172.21.15.16:0/2149997783 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:17.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.772+0000 7f8bd37fe700 1 --2- 172.21.15.16:0/2149997783 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8bd406da60 0x7f8bd40aecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:17.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.772+0000 7f8bd3fff700 1 --2- 172.21.15.16:0/2149997783 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8bd415b070 0x7f8bd40ccb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:17.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.772+0000 7f8bd2ffd700 1 -- 172.21.15.16:0/2149997783 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8bd406da60 msgr2=0x7f8bd40aecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:17.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.772+0000 7f8bd2ffd700 1 --2- 172.21.15.16:0/2149997783 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8bd406da60 0x7f8bd40aecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:17.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.772+0000 7f8bd2ffd700 1 -- 172.21.15.16:0/2149997783 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8bd415b070 msgr2=0x7f8bd40ccb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:17.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.772+0000 7f8bd2ffd700 1 --2- 172.21.15.16:0/2149997783 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8bd415b070 0x7f8bd40ccb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:17.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.772+0000 7f8bd2ffd700 1 -- 172.21.15.16:0/2149997783 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bbc009550 con 0x7f8bd40cb880 2023-10-16T13:44:17.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.772+0000 7f8bd3fff700 1 --2- 172.21.15.16:0/2149997783 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8bd415b070 0x7f8bd40ccb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:44:17.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.772+0000 7f8bd2ffd700 1 --2- 172.21.15.16:0/2149997783 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8bd40cb880 0x7f8bd40acf00 secure :-1 s=READY pgs=2176 cs=0 l=1 rev1=1 crypto rx=0x7f8bc400e810 tx=0x7f8bc400ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:17.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.772+0000 7f8bcbfff700 1 -- 172.21.15.16:0/2149997783 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bc4013610 con 0x7f8bd40cb880 2023-10-16T13:44:17.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.772+0000 7f8bcbfff700 1 -- 172.21.15.16:0/2149997783 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bc4013770 con 0x7f8bd40cb880 2023-10-16T13:44:17.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.772+0000 7f8bd9d10700 1 -- 172.21.15.16:0/2149997783 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bd414a1a0 con 0x7f8bd40cb880 2023-10-16T13:44:17.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.772+0000 7f8bcbfff700 1 -- 172.21.15.16:0/2149997783 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bc4013b20 con 0x7f8bd40cb880 2023-10-16T13:44:17.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.772+0000 7f8bd9d10700 1 -- 172.21.15.16:0/2149997783 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bd40af230 con 0x7f8bd40cb880 2023-10-16T13:44:17.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.773+0000 7f8bd9d10700 1 -- 172.21.15.16:0/2149997783 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bd414a9a0 con 0x7f8bd40cb880 2023-10-16T13:44:17.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.774+0000 7f8bcbfff700 1 -- 172.21.15.16:0/2149997783 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 121) v1 ==== 89360+0+0 (secure 0 0 0) 0x7f8bc4013c80 con 0x7f8bd40cb880 2023-10-16T13:44:17.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.774+0000 7f8bcbfff700 1 --2- 172.21.15.16:0/2149997783 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f8bb406b070 0x7f8bb406d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:17.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.774+0000 7f8bcbfff700 1 -- 172.21.15.16:0/2149997783 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 6817+0+0 (secure 0 0 0) 0x7f8bc40899c0 con 0x7f8bd40cb880 2023-10-16T13:44:17.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.774+0000 7f8bd37fe700 1 --2- 172.21.15.16:0/2149997783 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f8bb406b070 0x7f8bb406d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:17.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.775+0000 7f8bd37fe700 1 --2- 172.21.15.16:0/2149997783 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f8bb406b070 0x7f8bb406d530 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f8bbc000c00 tx=0x7f8bbc008030 comp rx=0 tx=0).ready entity=mgr.10578 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:17.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:17.779+0000 7f8bcbfff700 1 -- 172.21.15.16:0/2149997783 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8bd414a9a0 con 0x7f8bd40cb880 2023-10-16T13:44:18.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.009+0000 7f8bd9d10700 1 -- 172.21.15.16:0/2149997783 --> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/test_restart", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8bd40cec00 con 0x7f8bb406b070 2023-10-16T13:44:18.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.015+0000 7f8bcbfff700 1 -- 172.21.15.16:0/2149997783 <== mgr.10578 v2:172.21.15.16:6832/371380443 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+376 (secure 0 0 0) 0x7f8bbc00f880 con 0x7f8bb406b070 2023-10-16T13:44:18.018 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:44:18.018 INFO:teuthology.orchestra.run.smithi016.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T13:28:35", "first": "2023-10-16T13:29:00", "last": "2023-10-16T13:44:00", "last_pruned": null, "created_count": 16, "pruned_count": 0, "active": false}] 2023-10-16T13:44:18.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.017+0000 7f8bd9d10700 1 -- 172.21.15.16:0/2149997783 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f8bb406b070 msgr2=0x7f8bb406d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:18.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.017+0000 7f8bd9d10700 1 --2- 172.21.15.16:0/2149997783 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f8bb406b070 0x7f8bb406d530 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f8bbc000c00 tx=0x7f8bbc008030 comp rx=0 tx=0).stop 2023-10-16T13:44:18.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.017+0000 7f8bd9d10700 1 -- 172.21.15.16:0/2149997783 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8bd40cb880 msgr2=0x7f8bd40acf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:18.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.017+0000 7f8bd9d10700 1 --2- 172.21.15.16:0/2149997783 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8bd40cb880 0x7f8bd40acf00 secure :-1 s=READY pgs=2176 cs=0 l=1 rev1=1 crypto rx=0x7f8bc400e810 tx=0x7f8bc400ebd0 comp rx=0 tx=0).stop 2023-10-16T13:44:18.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.017+0000 7f8bd9d10700 1 -- 172.21.15.16:0/2149997783 shutdown_connections 2023-10-16T13:44:18.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.017+0000 7f8bd9d10700 1 --2- 172.21.15.16:0/2149997783 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8bd415b070 0x7f8bd40ccb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:18.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.017+0000 7f8bd9d10700 1 --2- 172.21.15.16:0/2149997783 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f8bb406b070 0x7f8bb406d530 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:18.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.017+0000 7f8bd9d10700 1 --2- 172.21.15.16:0/2149997783 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8bd406da60 0x7f8bd40aecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:18.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.017+0000 7f8bd9d10700 1 --2- 172.21.15.16:0/2149997783 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8bd40cb880 0x7f8bd40acf00 unknown :-1 s=CLOSED pgs=2176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:18.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.017+0000 7f8bd9d10700 1 -- 172.21.15.16:0/2149997783 >> 172.21.15.16:0/2149997783 conn(0x7f8bd40b1f40 msgr2=0x7f8bd40d0e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:18.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.017+0000 7f8bd9d10700 1 -- 172.21.15.16:0/2149997783 shutdown_connections 2023-10-16T13:44:18.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.017+0000 7f8bd9d10700 1 -- 172.21.15.16:0/2149997783 wait complete. 2023-10-16T13:44:18.031 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1m", "retention": {}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T13:28:35", "first": "2023-10-16T13:29:00", "last": "2023-10-16T13:44:00", "last_pruned": null, "created_count": 16, "pruned_count": 0, "active": false}] 2023-10-16T13:44:18.032 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/test_restart', 'rel_path': '/snap_test_dir1/test_restart', 'schedule': '1m', 'retention': {}, 'start': '2023-10-16T00:00:00', 'created': '2023-10-16T13:28:35', 'first': '2023-10-16T13:29:00', 'last': '2023-10-16T13:44:00', 'last_pruned': None, 'created_count': 16, 'pruned_count': 0, 'active': False} 2023-10-16T13:44:18.032 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 16 2023-10-16T13:44:18.032 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 16 2023-10-16T13:44:18.033 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/test_restart 1m 2023-10-16T13:44:18.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.159+0000 7f1ac145e700 1 Processor -- start 2023-10-16T13:44:18.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.159+0000 7f1ac145e700 1 -- start start 2023-10-16T13:44:18.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.159+0000 7f1ac145e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1abc15d8d0 0x7f1abc0b2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:18.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.159+0000 7f1ac145e700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f1abc0b8fa0 0x7f1abc0b6720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:18.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.159+0000 7f1ac145e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1abc0b6c60 0x7f1abc0ce350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:18.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.160+0000 7f1ac145e700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1abc0d0bb0 con 0x7f1abc15d8d0 2023-10-16T13:44:18.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.160+0000 7f1ac145e700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1abc15dcb0 con 0x7f1abc0b8fa0 2023-10-16T13:44:18.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.160+0000 7f1ac145e700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1abc0b2f30 con 0x7f1abc0b6c60 2023-10-16T13:44:18.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.160+0000 7f1abaffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1abc15d8d0 0x7f1abc0b2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:18.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.160+0000 7f1abaffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1abc15d8d0 0x7f1abc0b2830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50856/0 (socket says 172.21.15.16:50856) 2023-10-16T13:44:18.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.160+0000 7f1abaffd700 1 -- 172.21.15.16:0/51558692 learned_addr learned my addr 172.21.15.16:0/51558692 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:18.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.160+0000 7f1aba7fc700 1 --2- 172.21.15.16:0/51558692 >> v2:172.21.15.49:3300/0 conn(0x7f1abc0b8fa0 0x7f1abc0b6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:18.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.160+0000 7f1abb7fe700 1 --2- 172.21.15.16:0/51558692 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1abc0b6c60 0x7f1abc0ce350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:18.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.160+0000 7f1abaffd700 1 -- 172.21.15.16:0/51558692 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1abc0b6c60 msgr2=0x7f1abc0ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:18.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.160+0000 7f1abaffd700 1 --2- 172.21.15.16:0/51558692 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1abc0b6c60 0x7f1abc0ce350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:18.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.160+0000 7f1abaffd700 1 -- 172.21.15.16:0/51558692 >> v2:172.21.15.49:3300/0 conn(0x7f1abc0b8fa0 msgr2=0x7f1abc0b6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:18.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.160+0000 7f1abaffd700 1 --2- 172.21.15.16:0/51558692 >> v2:172.21.15.49:3300/0 conn(0x7f1abc0b8fa0 0x7f1abc0b6720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:18.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.160+0000 7f1abaffd700 1 -- 172.21.15.16:0/51558692 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1abc0d1c40 con 0x7f1abc15d8d0 2023-10-16T13:44:18.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.160+0000 7f1aba7fc700 1 --2- 172.21.15.16:0/51558692 >> v2:172.21.15.49:3300/0 conn(0x7f1abc0b8fa0 0x7f1abc0b6720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:44:18.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.160+0000 7f1abaffd700 1 --2- 172.21.15.16:0/51558692 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1abc15d8d0 0x7f1abc0b2830 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 crypto rx=0x7f1abc0cdc10 tx=0x7f1ab0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83a351411fcc125c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:18.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.160+0000 7f1ab9ffb700 1 -- 172.21.15.16:0/51558692 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab000b870 con 0x7f1abc15d8d0 2023-10-16T13:44:18.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.160+0000 7f1ab9ffb700 1 -- 172.21.15.16:0/51558692 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ab000b9d0 con 0x7f1abc15d8d0 2023-10-16T13:44:18.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.160+0000 7f1ab9ffb700 1 -- 172.21.15.16:0/51558692 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab000bd60 con 0x7f1abc15d8d0 2023-10-16T13:44:18.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.161+0000 7f1ac145e700 1 -- 172.21.15.16:0/51558692 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1abc15d8d0 msgr2=0x7f1abc0b2830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:18.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.161+0000 7f1ac145e700 1 --2- 172.21.15.16:0/51558692 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1abc15d8d0 0x7f1abc0b2830 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 crypto rx=0x7f1abc0cdc10 tx=0x7f1ab0009bd0 comp rx=0 tx=0).stop 2023-10-16T13:44:18.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.161+0000 7f1ac145e700 1 -- 172.21.15.16:0/51558692 shutdown_connections 2023-10-16T13:44:18.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.161+0000 7f1ac145e700 1 --2- 172.21.15.16:0/51558692 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1abc0b6c60 0x7f1abc0ce350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:18.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.161+0000 7f1ac145e700 1 --2- 172.21.15.16:0/51558692 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1abc15d8d0 0x7f1abc0b2830 unknown :-1 s=CLOSED pgs=2177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:18.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.161+0000 7f1ac145e700 1 --2- 172.21.15.16:0/51558692 >> v2:172.21.15.49:3300/0 conn(0x7f1abc0b8fa0 0x7f1abc0b6720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:18.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.161+0000 7f1ac145e700 1 -- 172.21.15.16:0/51558692 >> 172.21.15.16:0/51558692 conn(0x7f1abc072330 msgr2=0x7f1abc072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:18.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.161+0000 7f1ac145e700 1 -- 172.21.15.16:0/51558692 shutdown_connections 2023-10-16T13:44:18.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.161+0000 7f1ac145e700 1 -- 172.21.15.16:0/51558692 wait complete. 2023-10-16T13:44:18.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.163+0000 7f1ac145e700 1 Processor -- start 2023-10-16T13:44:18.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.163+0000 7f1ac145e700 1 -- start start 2023-10-16T13:44:18.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.164+0000 7f1ac145e700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1abc0b6c60 0x7f1abc0cb6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:18.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.164+0000 7f1ac145e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1abc0b8fa0 0x7f1abc0c9630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:18.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.164+0000 7f1ac145e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1abc15d8d0 0x7f1abc0c9b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:18.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.164+0000 7f1ac145e700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1abc0c91a0 con 0x7f1abc0b8fa0 2023-10-16T13:44:18.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.164+0000 7f1ac145e700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1abc0cc180 con 0x7f1abc0b6c60 2023-10-16T13:44:18.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.164+0000 7f1ac145e700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1abc0c8870 con 0x7f1abc15d8d0 2023-10-16T13:44:18.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.164+0000 7f1aba7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1abc0b8fa0 0x7f1abc0c9630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:18.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.164+0000 7f1aba7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1abc0b8fa0 0x7f1abc0c9630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50864/0 (socket says 172.21.15.16:50864) 2023-10-16T13:44:18.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.164+0000 7f1aba7fc700 1 -- 172.21.15.16:0/3322119732 learned_addr learned my addr 172.21.15.16:0/3322119732 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:18.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.164+0000 7f1abaffd700 1 --2- 172.21.15.16:0/3322119732 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1abc0b6c60 0x7f1abc0cb6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:18.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.164+0000 7f1abb7fe700 1 --2- 172.21.15.16:0/3322119732 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1abc15d8d0 0x7f1abc0c9b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:18.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.164+0000 7f1aba7fc700 1 -- 172.21.15.16:0/3322119732 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1abc15d8d0 msgr2=0x7f1abc0c9b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:18.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.164+0000 7f1aba7fc700 1 --2- 172.21.15.16:0/3322119732 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1abc15d8d0 0x7f1abc0c9b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:18.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.164+0000 7f1aba7fc700 1 -- 172.21.15.16:0/3322119732 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1abc0b6c60 msgr2=0x7f1abc0cb6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:18.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.164+0000 7f1aba7fc700 1 --2- 172.21.15.16:0/3322119732 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1abc0b6c60 0x7f1abc0cb6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:18.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.164+0000 7f1aba7fc700 1 -- 172.21.15.16:0/3322119732 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ab0009550 con 0x7f1abc0b8fa0 2023-10-16T13:44:18.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.164+0000 7f1aba7fc700 1 --2- 172.21.15.16:0/3322119732 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1abc0b8fa0 0x7f1abc0c9630 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 crypto rx=0x7f1aac00e810 tx=0x7f1aac00eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:18.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.164+0000 7f1aa37fe700 1 -- 172.21.15.16:0/3322119732 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aac00c020 con 0x7f1abc0b8fa0 2023-10-16T13:44:18.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.165+0000 7f1aa37fe700 1 -- 172.21.15.16:0/3322119732 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1aac004250 con 0x7f1abc0b8fa0 2023-10-16T13:44:18.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.165+0000 7f1ac145e700 1 -- 172.21.15.16:0/3322119732 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1abc0c8b50 con 0x7f1abc0b8fa0 2023-10-16T13:44:18.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.165+0000 7f1aa37fe700 1 -- 172.21.15.16:0/3322119732 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aac0036a0 con 0x7f1abc0b8fa0 2023-10-16T13:44:18.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.165+0000 7f1ac145e700 1 -- 172.21.15.16:0/3322119732 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1abc1f72c0 con 0x7f1abc0b8fa0 2023-10-16T13:44:18.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.166+0000 7f1ac145e700 1 -- 172.21.15.16:0/3322119732 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1abc04ebb0 con 0x7f1abc0b8fa0 2023-10-16T13:44:18.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.166+0000 7f1aa37fe700 1 -- 172.21.15.16:0/3322119732 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 121) v1 ==== 89360+0+0 (secure 0 0 0) 0x7f1aac003800 con 0x7f1abc0b8fa0 2023-10-16T13:44:18.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.166+0000 7f1aa37fe700 1 --2- 172.21.15.16:0/3322119732 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f1aa806b070 0x7f1aa806d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:18.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.166+0000 7f1aa37fe700 1 -- 172.21.15.16:0/3322119732 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 6817+0+0 (secure 0 0 0) 0x7f1aac089640 con 0x7f1abc0b8fa0 2023-10-16T13:44:18.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.167+0000 7f1abaffd700 1 --2- 172.21.15.16:0/3322119732 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f1aa806b070 0x7f1aa806d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:18.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.167+0000 7f1abaffd700 1 --2- 172.21.15.16:0/3322119732 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f1aa806b070 0x7f1aa806d530 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1ab0014040 tx=0x7f1ab0008030 comp rx=0 tx=0).ready entity=mgr.10578 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:18.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.171+0000 7f1aa37fe700 1 -- 172.21.15.16:0/3322119732 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1aac054100 con 0x7f1abc0b8fa0 2023-10-16T13:44:18.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.404+0000 7f1ac145e700 1 -- 172.21.15.16:0/3322119732 --> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/test_restart", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f1abc0d21c0 con 0x7f1aa806b070 2023-10-16T13:44:18.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.458+0000 7f1aa37fe700 1 -- 172.21.15.16:0/3322119732 <== mgr.10578 v2:172.21.15.16:6832/371380443 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+55 (secure 0 0 0) 0x7f1ab000bb40 con 0x7f1aa806b070 2023-10-16T13:44:18.461 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule removed for path /snap_test_dir1/test_restart 2023-10-16T13:44:18.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.460+0000 7f1ac145e700 1 -- 172.21.15.16:0/3322119732 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f1aa806b070 msgr2=0x7f1aa806d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:18.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.460+0000 7f1ac145e700 1 --2- 172.21.15.16:0/3322119732 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f1aa806b070 0x7f1aa806d530 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1ab0014040 tx=0x7f1ab0008030 comp rx=0 tx=0).stop 2023-10-16T13:44:18.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.460+0000 7f1ac145e700 1 -- 172.21.15.16:0/3322119732 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1abc0b8fa0 msgr2=0x7f1abc0c9630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:18.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.460+0000 7f1ac145e700 1 --2- 172.21.15.16:0/3322119732 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1abc0b8fa0 0x7f1abc0c9630 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 crypto rx=0x7f1aac00e810 tx=0x7f1aac00eb20 comp rx=0 tx=0).stop 2023-10-16T13:44:18.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.460+0000 7f1ac145e700 1 -- 172.21.15.16:0/3322119732 shutdown_connections 2023-10-16T13:44:18.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.460+0000 7f1ac145e700 1 --2- 172.21.15.16:0/3322119732 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f1aa806b070 0x7f1aa806d530 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:18.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.460+0000 7f1ac145e700 1 --2- 172.21.15.16:0/3322119732 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1abc15d8d0 0x7f1abc0c9b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:18.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.460+0000 7f1ac145e700 1 --2- 172.21.15.16:0/3322119732 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1abc0b6c60 0x7f1abc0cb6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:18.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.460+0000 7f1ac145e700 1 --2- 172.21.15.16:0/3322119732 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1abc0b8fa0 0x7f1abc0c9630 unknown :-1 s=CLOSED pgs=2178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:18.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.460+0000 7f1ac145e700 1 -- 172.21.15.16:0/3322119732 >> 172.21.15.16:0/3322119732 conn(0x7f1abc072330 msgr2=0x7f1abc15cca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:18.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.460+0000 7f1ac145e700 1 -- 172.21.15.16:0/3322119732 shutdown_connections 2023-10-16T13:44:18.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:18.460+0000 7f1ac145e700 1 -- 172.21.15.16:0/3322119732 wait complete. 2023-10-16T13:44:18.475 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/test_restart 2023-10-16T13:44:18.475 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:44:18.476 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/test_restart/.snap) 2023-10-16T13:44:18.494 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_29_00_UTC 2023-10-16T13:44:18.494 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_30_00_UTC 2023-10-16T13:44:18.494 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_31_00_UTC 2023-10-16T13:44:18.495 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_32_00_UTC 2023-10-16T13:44:18.495 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_33_00_UTC 2023-10-16T13:44:18.495 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_34_00_UTC 2023-10-16T13:44:18.495 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_35_00_UTC 2023-10-16T13:44:18.495 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_36_00_UTC 2023-10-16T13:44:18.496 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_37_00_UTC 2023-10-16T13:44:18.496 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_38_00_UTC 2023-10-16T13:44:18.496 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_39_00_UTC 2023-10-16T13:44:18.496 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_40_00_UTC 2023-10-16T13:44:18.496 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_41_00_UTC 2023-10-16T13:44:18.497 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_42_00_UTC 2023-10-16T13:44:18.497 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_43_00_UTC 2023-10-16T13:44:18.497 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_44_00_UTC 2023-10-16T13:44:18.497 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-13_29_00_UTC 2023-10-16T13:44:18.498 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:44:18.498 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-13_29_00_UTC) 2023-10-16T13:44:18.557 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-13_30_00_UTC 2023-10-16T13:44:18.557 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:44:18.558 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-13_30_00_UTC) 2023-10-16T13:44:18.619 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-13_31_00_UTC 2023-10-16T13:44:18.619 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:44:18.620 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-13_31_00_UTC) 2023-10-16T13:44:18.680 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-13_32_00_UTC 2023-10-16T13:44:18.681 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:44:18.681 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-13_32_00_UTC) 2023-10-16T13:44:18.743 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-13_33_00_UTC 2023-10-16T13:44:18.743 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:44:18.744 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-13_33_00_UTC) 2023-10-16T13:44:18.805 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-13_34_00_UTC 2023-10-16T13:44:18.805 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:44:18.806 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-13_34_00_UTC) 2023-10-16T13:44:18.867 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-13_35_00_UTC 2023-10-16T13:44:18.867 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:44:18.868 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-13_35_00_UTC) 2023-10-16T13:44:18.928 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-13_36_00_UTC 2023-10-16T13:44:18.929 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:44:18.930 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-13_36_00_UTC) 2023-10-16T13:44:18.991 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-13_37_00_UTC 2023-10-16T13:44:18.991 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:44:18.992 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-13_37_00_UTC) 2023-10-16T13:44:19.053 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-13_38_00_UTC 2023-10-16T13:44:19.053 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:44:19.054 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-13_38_00_UTC) 2023-10-16T13:44:19.115 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-13_39_00_UTC 2023-10-16T13:44:19.115 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:44:19.115 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-13_39_00_UTC) 2023-10-16T13:44:19.176 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-13_40_00_UTC 2023-10-16T13:44:19.177 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:44:19.177 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-13_40_00_UTC) 2023-10-16T13:44:19.238 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-13_41_00_UTC 2023-10-16T13:44:19.239 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:44:19.239 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-13_41_00_UTC) 2023-10-16T13:44:19.300 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-13_42_00_UTC 2023-10-16T13:44:19.300 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:44:19.301 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-13_42_00_UTC) 2023-10-16T13:44:19.361 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-13_43_00_UTC 2023-10-16T13:44:19.361 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:44:19.362 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-13_43_00_UTC) 2023-10-16T13:44:19.422 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-13_44_00_UTC 2023-10-16T13:44:19.423 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:44:19.423 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-16-13_44_00_UTC) 2023-10-16T13:44:19.483 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:44:19.484 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart) 2023-10-16T13:44:19.543 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-16T13:44:19.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:19.708+0000 7f4ae07b9700 1 Processor -- start 2023-10-16T13:44:19.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:19.708+0000 7f4ae07b9700 1 -- start start 2023-10-16T13:44:19.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:19.709+0000 7f4ae07b9700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4ad815b2c0 0x7f4ad8161340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:19.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:19.709+0000 7f4ae07b9700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4ad806e570 0x7f4ad815e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:19.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:19.709+0000 7f4ae07b9700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ad8161880 con 0x7f4ad806a7e0 2023-10-16T13:44:19.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:19.709+0000 7f4ae07b9700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ad815b6a0 con 0x7f4ad806e570 2023-10-16T13:44:19.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:19.709+0000 7f4ae07b9700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad8161b20 con 0x7f4ad815b2c0 2023-10-16T13:44:19.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:19.709+0000 7f4addd54700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4ad806e570 0x7f4ad815e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:19.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:19.709+0000 7f4addd54700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4ad806e570 0x7f4ad815e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50870/0 (socket says 172.21.15.16:50870) 2023-10-16T13:44:19.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:19.709+0000 7f4addd54700 1 -- 172.21.15.16:0/3270417370 learned_addr learned my addr 172.21.15.16:0/3270417370 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:19.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:19.709+0000 7f4aded56700 1 --1- 172.21.15.16:0/3270417370 >> v1:172.21.15.16:6789/0 conn(0x7f4ad806a7e0 0x7f4ad8071a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:40148/0 (socket says 172.21.15.16:40148) 2023-10-16T13:44:19.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:19.709+0000 7f4ade555700 1 --2- 172.21.15.16:0/3270417370 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4ad815b2c0 0x7f4ad8161340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:19.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:19.709+0000 7f4addd54700 1 -- 172.21.15.16:0/3270417370 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4ad815b2c0 msgr2=0x7f4ad8161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:19.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:19.709+0000 7f4addd54700 1 --2- 172.21.15.16:0/3270417370 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4ad815b2c0 0x7f4ad8161340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:19.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:19.709+0000 7f4addd54700 1 -- 172.21.15.16:0/3270417370 >> v1:172.21.15.16:6789/0 conn(0x7f4ad806a7e0 legacy=0x7f4ad8071a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:19.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:19.709+0000 7f4addd54700 1 -- 172.21.15.16:0/3270417370 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ad8161e00 con 0x7f4ad806e570 2023-10-16T13:44:19.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:19.709+0000 7f4ade555700 1 --2- 172.21.15.16:0/3270417370 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4ad815b2c0 0x7f4ad8161340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:44:19.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:19.710+0000 7f4addd54700 1 --2- 172.21.15.16:0/3270417370 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ad806e570 0x7f4ad815e5a0 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 crypto rx=0x7f4ad40098f0 tx=0x7f4ad4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2550a1560bde4e6f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:19.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:19.710+0000 7f4add553700 1 -- 172.21.15.16:0/3270417370 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ad400b8a0 con 0x7f4ad806e570 2023-10-16T13:44:19.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:19.710+0000 7f4add553700 1 -- 172.21.15.16:0/3270417370 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ad400ba00 con 0x7f4ad806e570 2023-10-16T13:44:19.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:19.710+0000 7f4add553700 1 -- 172.21.15.16:0/3270417370 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ad400bd90 con 0x7f4ad806e570 2023-10-16T13:44:19.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:19.710+0000 7f4ae07b9700 1 -- 172.21.15.16:0/3270417370 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ad806e570 msgr2=0x7f4ad815e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:19.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:19.710+0000 7f4ae07b9700 1 --2- 172.21.15.16:0/3270417370 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ad806e570 0x7f4ad815e5a0 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 crypto rx=0x7f4ad40098f0 tx=0x7f4ad4009c00 comp rx=0 tx=0).stop 2023-10-16T13:44:19.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:19.710+0000 7f4ae07b9700 1 -- 172.21.15.16:0/3270417370 shutdown_connections 2023-10-16T13:44:19.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:19.710+0000 7f4ae07b9700 1 --2- 172.21.15.16:0/3270417370 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4ad815b2c0 0x7f4ad8161340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:19.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:19.710+0000 7f4ae07b9700 1 --2- 172.21.15.16:0/3270417370 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ad806e570 0x7f4ad815e5a0 unknown :-1 s=CLOSED pgs=2180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:19.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:19.710+0000 7f4ae07b9700 1 -- 172.21.15.16:0/3270417370 >> 172.21.15.16:0/3270417370 conn(0x7f4ad80b1f40 msgr2=0x7f4ad80b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:19.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:19.710+0000 7f4ae07b9700 1 -- 172.21.15.16:0/3270417370 shutdown_connections 2023-10-16T13:44:19.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:19.710+0000 7f4ae07b9700 1 -- 172.21.15.16:0/3270417370 wait complete. 2023-10-16T13:44:19.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:19.712+0000 7f4ae07b9700 1 Processor -- start 2023-10-16T13:44:19.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:19.712+0000 7f4ae07b9700 1 -- start start 2023-10-16T13:44:19.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:19.712+0000 7f4ae07b9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ad806a7e0 0x7f4ad8159730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:19.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:19.712+0000 7f4ae07b9700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4ad806e570 0x7f4ad8159c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:19.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:19.712+0000 7f4ae07b9700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4ad815b2c0 0x7f4ad81f6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:19.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:19.712+0000 7f4ae07b9700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad81f3620 con 0x7f4ad806a7e0 2023-10-16T13:44:19.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:19.712+0000 7f4ae07b9700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad80c8360 con 0x7f4ad815b2c0 2023-10-16T13:44:19.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:19.712+0000 7f4ae07b9700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad81f2f60 con 0x7f4ad806e570 2023-10-16T13:44:19.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:19.712+0000 7f4ade555700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ad806a7e0 0x7f4ad8159730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:19.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:19.713+0000 7f4ade555700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ad806a7e0 0x7f4ad8159730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50876/0 (socket says 172.21.15.16:50876) 2023-10-16T13:44:19.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:19.713+0000 7f4ade555700 1 -- 172.21.15.16:0/3229427420 learned_addr learned my addr 172.21.15.16:0/3229427420 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:19.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:19.713+0000 7f4addd54700 1 --2- 172.21.15.16:0/3229427420 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4ad806e570 0x7f4ad8159c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:19.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:19.713+0000 7f4aded56700 1 --2- 172.21.15.16:0/3229427420 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4ad815b2c0 0x7f4ad81f6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:19.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:19.713+0000 7f4ade555700 1 -- 172.21.15.16:0/3229427420 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4ad806e570 msgr2=0x7f4ad8159c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:19.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:19.713+0000 7f4ade555700 1 --2- 172.21.15.16:0/3229427420 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4ad806e570 0x7f4ad8159c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:19.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:19.713+0000 7f4ade555700 1 -- 172.21.15.16:0/3229427420 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4ad815b2c0 msgr2=0x7f4ad81f6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:19.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:19.713+0000 7f4ade555700 1 --2- 172.21.15.16:0/3229427420 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4ad815b2c0 0x7f4ad81f6240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:19.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:19.713+0000 7f4ade555700 1 -- 172.21.15.16:0/3229427420 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ad4009550 con 0x7f4ad806a7e0 2023-10-16T13:44:19.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:19.713+0000 7f4ade555700 1 --2- 172.21.15.16:0/3229427420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ad806a7e0 0x7f4ad8159730 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 crypto rx=0x7f4ac800c750 tx=0x7f4ac800ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:19.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:19.715+0000 7f4aceffd700 1 -- 172.21.15.16:0/3229427420 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ac8011800 con 0x7f4ad806a7e0 2023-10-16T13:44:19.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:19.715+0000 7f4aceffd700 1 -- 172.21.15.16:0/3229427420 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ac8011960 con 0x7f4ad806a7e0 2023-10-16T13:44:19.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:19.715+0000 7f4ae07b9700 1 -- 172.21.15.16:0/3229427420 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ad81f3240 con 0x7f4ad806a7e0 2023-10-16T13:44:19.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:19.715+0000 7f4aceffd700 1 -- 172.21.15.16:0/3229427420 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ac8011cd0 con 0x7f4ad806a7e0 2023-10-16T13:44:19.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:19.715+0000 7f4ae07b9700 1 -- 172.21.15.16:0/3229427420 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ad8157a90 con 0x7f4ad806a7e0 2023-10-16T13:44:19.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:19.716+0000 7f4ae07b9700 1 -- 172.21.15.16:0/3229427420 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ad804ebb0 con 0x7f4ad806a7e0 2023-10-16T13:44:19.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:19.717+0000 7f4aceffd700 1 -- 172.21.15.16:0/3229427420 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 121) v1 ==== 89360+0+0 (secure 0 0 0) 0x7f4ac8021350 con 0x7f4ad806a7e0 2023-10-16T13:44:19.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:19.717+0000 7f4aceffd700 1 --2- 172.21.15.16:0/3229427420 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f4abc06b070 0x7f4abc06d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:19.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:19.717+0000 7f4aceffd700 1 -- 172.21.15.16:0/3229427420 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 6817+0+0 (secure 0 0 0) 0x7f4ac8089bb0 con 0x7f4ad806a7e0 2023-10-16T13:44:19.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:19.717+0000 7f4addd54700 1 --2- 172.21.15.16:0/3229427420 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f4abc06b070 0x7f4abc06d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:19.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:19.718+0000 7f4addd54700 1 --2- 172.21.15.16:0/3229427420 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f4abc06b070 0x7f4abc06d530 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4ad4009c40 tx=0x7f4ad4008030 comp rx=0 tx=0).ready entity=mgr.10578 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:19.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:19.722+0000 7f4aceffd700 1 -- 172.21.15.16:0/3229427420 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4ac80545e0 con 0x7f4ad806a7e0 2023-10-16T13:44:19.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:19.989+0000 7f4ae07b9700 1 -- 172.21.15.16:0/3229427420 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f4ad81f33d0 con 0x7f4ad806a7e0 2023-10-16T13:44:20.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:20.719+0000 7f4aceffd700 1 -- 172.21.15.16:0/3229427420 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v122) v1 ==== 93+0+0 (secure 0 0 0) 0x7f4ac801e0a0 con 0x7f4ad806a7e0 2023-10-16T13:44:20.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:20.727+0000 7f4aceffd700 1 -- 172.21.15.16:0/3229427420 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mgrmap(e 122) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f4ac8087c70 con 0x7f4ad806a7e0 2023-10-16T13:44:20.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:20.729+0000 7f4ae07b9700 1 -- 172.21.15.16:0/3229427420 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f4abc06b070 msgr2=0x7f4abc06d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:20.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:20.729+0000 7f4ae07b9700 1 --2- 172.21.15.16:0/3229427420 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f4abc06b070 0x7f4abc06d530 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4ad4009c40 tx=0x7f4ad4008030 comp rx=0 tx=0).stop 2023-10-16T13:44:20.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:20.729+0000 7f4ae07b9700 1 -- 172.21.15.16:0/3229427420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ad806a7e0 msgr2=0x7f4ad8159730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:20.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:20.729+0000 7f4ae07b9700 1 --2- 172.21.15.16:0/3229427420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ad806a7e0 0x7f4ad8159730 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 crypto rx=0x7f4ac800c750 tx=0x7f4ac800ca60 comp rx=0 tx=0).stop 2023-10-16T13:44:20.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:20.729+0000 7f4ae07b9700 1 -- 172.21.15.16:0/3229427420 shutdown_connections 2023-10-16T13:44:20.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:20.729+0000 7f4ae07b9700 1 --2- 172.21.15.16:0/3229427420 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4ad815b2c0 0x7f4ad81f6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:20.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:20.729+0000 7f4ae07b9700 1 --2- 172.21.15.16:0/3229427420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ad806a7e0 0x7f4ad8159730 unknown :-1 s=CLOSED pgs=2181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:20.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:20.729+0000 7f4ae07b9700 1 --2- 172.21.15.16:0/3229427420 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f4abc06b070 0x7f4abc06d530 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:20.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:20.729+0000 7f4ae07b9700 1 --2- 172.21.15.16:0/3229427420 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4ad806e570 0x7f4ad8159c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:20.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:20.729+0000 7f4ae07b9700 1 -- 172.21.15.16:0/3229427420 >> 172.21.15.16:0/3229427420 conn(0x7f4ad80b1f40 msgr2=0x7f4ad80cb290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:20.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:20.731+0000 7f4ae07b9700 1 -- 172.21.15.16:0/3229427420 shutdown_connections 2023-10-16T13:44:20.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:20.731+0000 7f4ae07b9700 1 -- 172.21.15.16:0/3229427420 wait complete. 2023-10-16T13:44:20.758 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T13:44:20.758 DEBUG:teuthology.orchestra.run.smithi016:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T13:44:20.807 DEBUG:teuthology.orchestra.run.smithi049:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T13:44:20.850 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T13:44:20.850 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T13:44:20.851 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:44:20.872 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T13:44:20.872 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:44:20.903 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:44:20.902+0000 7ffb56192200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T13:44:20.913 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:44:20.910+0000 7fe4965ec200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T13:44:20.930 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:44:20.973 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:44:20.971+0000 7ffb56192200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T13:44:20.983 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:44:20.981+0000 7fe4965ec200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T13:44:21.047 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:44:21.046+0000 7ffb56192200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T13:44:21.060 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:44:21.058+0000 7fe4965ec200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T13:44:21.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.095+0000 7fd9283f3700 1 Processor -- start 2023-10-16T13:44:21.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.095+0000 7fd9283f3700 1 -- start start 2023-10-16T13:44:21.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.096+0000 7fd9283f3700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd92015d900 0x7fd9200d2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:21.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.096+0000 7fd9283f3700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd9200d2f70 0x7fd9200cdc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:21.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.096+0000 7fd9283f3700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9200ce1a0 con 0x7fd9200ce9f0 2023-10-16T13:44:21.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.096+0000 7fd9283f3700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9200cee70 con 0x7fd9200d2f70 2023-10-16T13:44:21.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.096+0000 7fd9283f3700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9200ce400 con 0x7fd92015d900 2023-10-16T13:44:21.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.096+0000 7fd926990700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd9200d2f70 0x7fd9200cdc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:21.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.096+0000 7fd926990700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd9200d2f70 0x7fd9200cdc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50928/0 (socket says 172.21.15.16:50928) 2023-10-16T13:44:21.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.096+0000 7fd926990700 1 -- 172.21.15.16:0/175313974 learned_addr learned my addr 172.21.15.16:0/175313974 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:21.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.096+0000 7fd92618f700 1 --2- 172.21.15.16:0/175313974 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd92015d900 0x7fd9200d2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:21.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.097+0000 7fd926990700 1 -- 172.21.15.16:0/175313974 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd92015d900 msgr2=0x7fd9200d2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:21.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.097+0000 7fd926990700 1 --2- 172.21.15.16:0/175313974 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd92015d900 0x7fd9200d2040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:21.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.097+0000 7fd926990700 1 -- 172.21.15.16:0/175313974 >> v1:172.21.15.49:6789/0 conn(0x7fd9200ce9f0 legacy=0x7fd9200d2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:21.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.097+0000 7fd926990700 1 -- 172.21.15.16:0/175313974 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd92015dce0 con 0x7fd9200d2f70 2023-10-16T13:44:21.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.097+0000 7fd92518d700 1 -- 172.21.15.16:0/175313974 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3696808114 0 0) 0x7fd9200ce1a0 con 0x7fd9200ce9f0 2023-10-16T13:44:21.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.097+0000 7fd92618f700 1 --2- 172.21.15.16:0/175313974 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd92015d900 0x7fd9200d2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:44:21.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.097+0000 7fd926990700 1 --2- 172.21.15.16:0/175313974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9200d2f70 0x7fd9200cdc60 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 crypto rx=0x7fd918009e50 tx=0x7fd91800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9aca3c3c3b4ac94f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:21.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.097+0000 7fd92518d700 1 -- 172.21.15.16:0/175313974 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd91800e070 con 0x7fd9200d2f70 2023-10-16T13:44:21.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.097+0000 7fd92518d700 1 -- 172.21.15.16:0/175313974 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd91800b910 con 0x7fd9200d2f70 2023-10-16T13:44:21.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.097+0000 7fd92518d700 1 -- 172.21.15.16:0/175313974 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd91800bcc0 con 0x7fd9200d2f70 2023-10-16T13:44:21.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.098+0000 7fd9283f3700 1 -- 172.21.15.16:0/175313974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9200d2f70 msgr2=0x7fd9200cdc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:21.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.098+0000 7fd9283f3700 1 --2- 172.21.15.16:0/175313974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9200d2f70 0x7fd9200cdc60 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 crypto rx=0x7fd918009e50 tx=0x7fd91800b330 comp rx=0 tx=0).stop 2023-10-16T13:44:21.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.098+0000 7fd9283f3700 1 -- 172.21.15.16:0/175313974 shutdown_connections 2023-10-16T13:44:21.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.098+0000 7fd9283f3700 1 --2- 172.21.15.16:0/175313974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9200d2f70 0x7fd9200cdc60 unknown :-1 s=CLOSED pgs=2184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:21.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.098+0000 7fd9283f3700 1 --2- 172.21.15.16:0/175313974 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd92015d900 0x7fd9200d2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:21.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.098+0000 7fd9283f3700 1 -- 172.21.15.16:0/175313974 >> 172.21.15.16:0/175313974 conn(0x7fd920072330 msgr2=0x7fd920072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:21.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.098+0000 7fd9283f3700 1 -- 172.21.15.16:0/175313974 shutdown_connections 2023-10-16T13:44:21.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.098+0000 7fd9283f3700 1 -- 172.21.15.16:0/175313974 wait complete. 2023-10-16T13:44:21.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.099+0000 7fd9283f3700 1 Processor -- start 2023-10-16T13:44:21.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.099+0000 7fd9283f3700 1 -- start start 2023-10-16T13:44:21.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.100+0000 7fd9283f3700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd9200ce9f0 0x7fd9201451c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:21.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.100+0000 7fd9283f3700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd9200d2f70 0x7fd9201f26e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:21.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.100+0000 7fd9283f3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd92015d900 0x7fd92015b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:21.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.100+0000 7fd9283f3700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9200b47b0 con 0x7fd92015d900 2023-10-16T13:44:21.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.100+0000 7fd9283f3700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd920145700 con 0x7fd9200d2f70 2023-10-16T13:44:21.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.100+0000 7fd9283f3700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9201f67c0 con 0x7fd9200ce9f0 2023-10-16T13:44:21.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.100+0000 7fd92618f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd9200ce9f0 0x7fd9201451c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:21.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.100+0000 7fd926990700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd92015d900 0x7fd92015b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:21.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.100+0000 7fd92598e700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd9200d2f70 0x7fd9201f26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:21.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.100+0000 7fd92618f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd9200ce9f0 0x7fd9201451c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:39188/0 (socket says 172.21.15.16:39188) 2023-10-16T13:44:21.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.100+0000 7fd92618f700 1 -- 172.21.15.16:0/2263354053 learned_addr learned my addr 172.21.15.16:0/2263354053 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:21.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.100+0000 7fd926990700 1 -- 172.21.15.16:0/2263354053 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd9200ce9f0 msgr2=0x7fd9201451c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:21.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.100+0000 7fd926990700 1 --2- 172.21.15.16:0/2263354053 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd9200ce9f0 0x7fd9201451c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:21.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.100+0000 7fd926990700 1 -- 172.21.15.16:0/2263354053 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd9200d2f70 msgr2=0x7fd9201f26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:21.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.100+0000 7fd926990700 1 --2- 172.21.15.16:0/2263354053 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd9200d2f70 0x7fd9201f26e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:21.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.100+0000 7fd926990700 1 -- 172.21.15.16:0/2263354053 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd91c0094a0 con 0x7fd92015d900 2023-10-16T13:44:21.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.100+0000 7fd92598e700 1 --2- 172.21.15.16:0/2263354053 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd9200d2f70 0x7fd9201f26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:44:21.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.100+0000 7fd92618f700 1 --2- 172.21.15.16:0/2263354053 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd9200ce9f0 0x7fd9201451c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:44:21.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.101+0000 7fd926990700 1 --2- 172.21.15.16:0/2263354053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd92015d900 0x7fd92015b170 secure :-1 s=READY pgs=2185 cs=0 l=1 rev1=1 crypto rx=0x7fd918015000 tx=0x7fd918004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:21.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.101+0000 7fd916ffd700 1 -- 172.21.15.16:0/2263354053 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd918004880 con 0x7fd92015d900 2023-10-16T13:44:21.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.101+0000 7fd916ffd700 1 -- 172.21.15.16:0/2263354053 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9180092e0 con 0x7fd92015d900 2023-10-16T13:44:21.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.101+0000 7fd9283f3700 1 -- 172.21.15.16:0/2263354053 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd918009b30 con 0x7fd92015d900 2023-10-16T13:44:21.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.101+0000 7fd916ffd700 1 -- 172.21.15.16:0/2263354053 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd91801c680 con 0x7fd92015d900 2023-10-16T13:44:21.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.101+0000 7fd9283f3700 1 -- 172.21.15.16:0/2263354053 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9201f6d80 con 0x7fd92015d900 2023-10-16T13:44:21.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.102+0000 7fd9283f3700 1 -- 172.21.15.16:0/2263354053 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9200b4940 con 0x7fd92015d900 2023-10-16T13:44:21.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.102+0000 7fd916ffd700 1 -- 172.21.15.16:0/2263354053 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 122) v1 ==== 89343+0+0 (secure 0 0 0) 0x7fd91801c7e0 con 0x7fd92015d900 2023-10-16T13:44:21.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.103+0000 7fd916ffd700 1 --2- 172.21.15.16:0/2263354053 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7fd90406b020 0x7fd90406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:21.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.103+0000 7fd92618f700 1 -- 172.21.15.16:0/2263354053 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7fd90406b020 msgr2=0x7fd90406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/371380443 2023-10-16T13:44:21.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.103+0000 7fd916ffd700 1 -- 172.21.15.16:0/2263354053 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 6905+0+0 (secure 0 0 0) 0x7fd91808b700 con 0x7fd92015d900 2023-10-16T13:44:21.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.103+0000 7fd92618f700 1 --2- 172.21.15.16:0/2263354053 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7fd90406b020 0x7fd90406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:44:21.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.107+0000 7fd916ffd700 1 -- 172.21.15.16:0/2263354053 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd918056430 con 0x7fd92015d900 2023-10-16T13:44:21.128 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:44:21.127+0000 7ffb56192200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T13:44:21.145 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:44:21.143+0000 7fe4965ec200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T13:44:21.288 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:44:21.287+0000 7ffb56192200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T13:44:21.302 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:44:21.300+0000 7fe4965ec200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T13:44:21.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.304+0000 7fd92618f700 1 -- 172.21.15.16:0/2263354053 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7fd90406b020 msgr2=0x7fd90406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/371380443 2023-10-16T13:44:21.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.304+0000 7fd92618f700 1 --2- 172.21.15.16:0/2263354053 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7fd90406b020 0x7fd90406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:44:21.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.339+0000 7fd9283f3700 1 -- 172.21.15.16:0/2263354053 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd9200b4940 con 0x7fd92015d900 2023-10-16T13:44:21.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.341+0000 7fd916ffd700 1 -- 172.21.15.16:0/2263354053 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v193) v1 ==== 74+0+17931 (secure 0 0 0) 0x7fd9200b4940 con 0x7fd92015d900 2023-10-16T13:44:21.343 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:44:21.344 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":193,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:44:20.715063+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2023-10-16T13:28:17.549815+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"189","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2023-10-16T13:28:18.493327+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"193","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":32,"snap_epoch":193,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6833/3952564978":"2023-10-17T13:39:22.557973+0000","172.21.15.16:6832/3952564978":"2023-10-17T13:39:22.557973+0000","172.21.15.16:0/2554840790":"2023-10-17T13:39:22.557973+0000","172.21.15.16:0/360056771":"2023-10-17T13:39:22.557973+0000","172.21.15.16:0/3520685377":"2023-10-17T13:39:22.557973+0000","172.21.15.16:0/1571009468":"2023-10-17T13:28:34.631949+0000","172.21.15.16:0/678119374":"2023-10-17T13:28:34.631949+0000","172.21.15.16:0/3920247727":"2023-10-17T13:28:34.631949+0000","172.21.15.16:0/857902049":"2023-10-17T13:39:22.557973+0000","172.21.15.16:6833/4211329181":"2023-10-17T13:28:34.631949+0000","172.21.15.16:6832/4211329181":"2023-10-17T13:28:34.631949+0000","172.21.15.16:0/1548506671":"2023-10-17T13:39:22.557973+0000","172.21.15.16:0/2852193608":"2023-10-17T13:28:34.631949+0000","172.21.15.16:0/1641955134":"2023-10-17T13:39:22.557973+0000","172.21.15.16:0/471630266":"2023-10-17T13:28:34.631949+0000","172.21.15.16:0/3153082432":"2023-10-17T13:28:34.631949+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":15},{"begin":18,"length":15}]}],"new_removed_snaps":[{"pool":24,"snaps":[{"begin":2,"length":15},{"begin":18,"length":15}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:44:21.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.343+0000 7fd9283f3700 1 -- 172.21.15.16:0/2263354053 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7fd90406b020 msgr2=0x7fd90406d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:44:21.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.343+0000 7fd9283f3700 1 --2- 172.21.15.16:0/2263354053 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7fd90406b020 0x7fd90406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:21.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.343+0000 7fd9283f3700 1 -- 172.21.15.16:0/2263354053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd92015d900 msgr2=0x7fd92015b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:21.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.343+0000 7fd9283f3700 1 --2- 172.21.15.16:0/2263354053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd92015d900 0x7fd92015b170 secure :-1 s=READY pgs=2185 cs=0 l=1 rev1=1 crypto rx=0x7fd918015000 tx=0x7fd918004ad0 comp rx=0 tx=0).stop 2023-10-16T13:44:21.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.343+0000 7fd9283f3700 1 -- 172.21.15.16:0/2263354053 shutdown_connections 2023-10-16T13:44:21.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.343+0000 7fd9283f3700 1 --2- 172.21.15.16:0/2263354053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd92015d900 0x7fd92015b170 unknown :-1 s=CLOSED pgs=2185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:21.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.343+0000 7fd9283f3700 1 --2- 172.21.15.16:0/2263354053 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7fd90406b020 0x7fd90406d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:21.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.343+0000 7fd9283f3700 1 --2- 172.21.15.16:0/2263354053 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd9200ce9f0 0x7fd9201451c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:21.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.343+0000 7fd9283f3700 1 --2- 172.21.15.16:0/2263354053 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd9200d2f70 0x7fd9201f26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:21.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.343+0000 7fd9283f3700 1 -- 172.21.15.16:0/2263354053 >> 172.21.15.16:0/2263354053 conn(0x7fd920072330 msgr2=0x7fd920158980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:21.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.343+0000 7fd9283f3700 1 -- 172.21.15.16:0/2263354053 shutdown_connections 2023-10-16T13:44:21.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.343+0000 7fd9283f3700 1 -- 172.21.15.16:0/2263354053 wait complete. 2023-10-16T13:44:21.358 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 23, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T13:28:17.549815+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '189', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 24, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T13:28:18.493327+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '193', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 32, 'snap_epoch': 193, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T13:44:21.359 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T13:44:21.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.532+0000 7f7aa8e10700 1 Processor -- start 2023-10-16T13:44:21.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.533+0000 7f7aa8e10700 1 -- start start 2023-10-16T13:44:21.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.533+0000 7f7aa8e10700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f7aa415ec80 0x7f7aa40cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:21.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.533+0000 7f7aa8e10700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7aa415f060 0x7f7aa40ce400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:21.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.533+0000 7f7aa8e10700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7aa40d2110 0x7f7aa40b4440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:21.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.533+0000 7f7aa8e10700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7aa40b2c50 con 0x7f7aa415f060 2023-10-16T13:44:21.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.533+0000 7f7aa8e10700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7aa40b2440 con 0x7f7aa415ec80 2023-10-16T13:44:21.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.533+0000 7f7aa8e10700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa40d00e0 con 0x7f7aa40d2110 2023-10-16T13:44:21.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.534+0000 7f7a9bfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7aa415f060 0x7f7aa40ce400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:21.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.534+0000 7f7a9bfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7aa415f060 0x7f7aa40ce400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50960/0 (socket says 172.21.15.16:50960) 2023-10-16T13:44:21.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.534+0000 7f7a9bfff700 1 -- 172.21.15.16:0/566246930 learned_addr learned my addr 172.21.15.16:0/566246930 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:21.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.534+0000 7f7aa2d9d700 1 --2- 172.21.15.16:0/566246930 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7aa40d2110 0x7f7aa40b4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:21.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.534+0000 7f7aa259c700 1 --2- 172.21.15.16:0/566246930 >> v2:172.21.15.49:3300/0 conn(0x7f7aa415ec80 0x7f7aa40cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:21.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.534+0000 7f7a9bfff700 1 -- 172.21.15.16:0/566246930 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7aa40d2110 msgr2=0x7f7aa40b4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:21.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.534+0000 7f7a9bfff700 1 --2- 172.21.15.16:0/566246930 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7aa40d2110 0x7f7aa40b4440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:21.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.534+0000 7f7a9bfff700 1 -- 172.21.15.16:0/566246930 >> v2:172.21.15.49:3300/0 conn(0x7f7aa415ec80 msgr2=0x7f7aa40cdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:21.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.534+0000 7f7a9bfff700 1 --2- 172.21.15.16:0/566246930 >> v2:172.21.15.49:3300/0 conn(0x7f7aa415ec80 0x7f7aa40cdec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:21.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.534+0000 7f7a9bfff700 1 -- 172.21.15.16:0/566246930 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aa40cef60 con 0x7f7aa415f060 2023-10-16T13:44:21.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.534+0000 7f7a9bfff700 1 --2- 172.21.15.16:0/566246930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7aa415f060 0x7f7aa40ce400 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 crypto rx=0x7f7aa4067b20 tx=0x7f7a8c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=828fdf90dcffe2bc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:21.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.534+0000 7f7aa1d9b700 1 -- 172.21.15.16:0/566246930 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a8c011830 con 0x7f7aa415f060 2023-10-16T13:44:21.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.534+0000 7f7aa1d9b700 1 -- 172.21.15.16:0/566246930 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a8c011990 con 0x7f7aa415f060 2023-10-16T13:44:21.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.534+0000 7f7aa1d9b700 1 -- 172.21.15.16:0/566246930 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a8c011d20 con 0x7f7aa415f060 2023-10-16T13:44:21.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.535+0000 7f7aa8e10700 1 -- 172.21.15.16:0/566246930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7aa415f060 msgr2=0x7f7aa40ce400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:21.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.535+0000 7f7aa8e10700 1 --2- 172.21.15.16:0/566246930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7aa415f060 0x7f7aa40ce400 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 crypto rx=0x7f7aa4067b20 tx=0x7f7a8c009bd0 comp rx=0 tx=0).stop 2023-10-16T13:44:21.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.535+0000 7f7aa8e10700 1 -- 172.21.15.16:0/566246930 shutdown_connections 2023-10-16T13:44:21.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.535+0000 7f7aa8e10700 1 --2- 172.21.15.16:0/566246930 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7aa40d2110 0x7f7aa40b4440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:21.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.535+0000 7f7aa8e10700 1 --2- 172.21.15.16:0/566246930 >> v2:172.21.15.49:3300/0 conn(0x7f7aa415ec80 0x7f7aa40cdec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:21.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.535+0000 7f7aa8e10700 1 --2- 172.21.15.16:0/566246930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7aa415f060 0x7f7aa40ce400 unknown :-1 s=CLOSED pgs=2186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:21.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.535+0000 7f7aa8e10700 1 -- 172.21.15.16:0/566246930 >> 172.21.15.16:0/566246930 conn(0x7f7aa40d15f0 msgr2=0x7f7aa40d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:21.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.535+0000 7f7aa8e10700 1 -- 172.21.15.16:0/566246930 shutdown_connections 2023-10-16T13:44:21.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.535+0000 7f7aa8e10700 1 -- 172.21.15.16:0/566246930 wait complete. 2023-10-16T13:44:21.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.536+0000 7f7aa8e10700 1 Processor -- start 2023-10-16T13:44:21.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.536+0000 7f7aa8e10700 1 -- start start 2023-10-16T13:44:21.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.537+0000 7f7aa8e10700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7aa40d2110 0x7f7aa4150b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:21.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.537+0000 7f7aa8e10700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7aa415ec80 0x7f7aa40c6830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:21.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.537+0000 7f7aa8e10700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7aa415f060 0x7f7aa40c6d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:21.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.537+0000 7f7aa8e10700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa41e4710 con 0x7f7aa40d2110 2023-10-16T13:44:21.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.537+0000 7f7aa8e10700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa41510a0 con 0x7f7aa415f060 2023-10-16T13:44:21.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.537+0000 7f7aa8e10700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa40c7660 con 0x7f7aa415ec80 2023-10-16T13:44:21.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.537+0000 7f7aa259c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7aa40d2110 0x7f7aa4150b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:21.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.537+0000 7f7aa259c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7aa40d2110 0x7f7aa4150b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50964/0 (socket says 172.21.15.16:50964) 2023-10-16T13:44:21.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.537+0000 7f7aa259c700 1 -- 172.21.15.16:0/1445127394 learned_addr learned my addr 172.21.15.16:0/1445127394 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:21.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.537+0000 7f7a9bfff700 1 --2- 172.21.15.16:0/1445127394 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7aa415ec80 0x7f7aa40c6830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:21.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.537+0000 7f7aa2d9d700 1 --2- 172.21.15.16:0/1445127394 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7aa415f060 0x7f7aa40c6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:21.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.538+0000 7f7aa259c700 1 -- 172.21.15.16:0/1445127394 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7aa415ec80 msgr2=0x7f7aa40c6830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:21.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.538+0000 7f7aa259c700 1 --2- 172.21.15.16:0/1445127394 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7aa415ec80 0x7f7aa40c6830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:21.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.538+0000 7f7aa259c700 1 -- 172.21.15.16:0/1445127394 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7aa415f060 msgr2=0x7f7aa40c6d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:21.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.538+0000 7f7aa259c700 1 --2- 172.21.15.16:0/1445127394 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7aa415f060 0x7f7aa40c6d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:21.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.538+0000 7f7aa259c700 1 -- 172.21.15.16:0/1445127394 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a8c009550 con 0x7f7aa40d2110 2023-10-16T13:44:21.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.538+0000 7f7a9bfff700 1 --2- 172.21.15.16:0/1445127394 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7aa415ec80 0x7f7aa40c6830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:44:21.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.538+0000 7f7aa259c700 1 --2- 172.21.15.16:0/1445127394 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7aa40d2110 0x7f7aa4150b60 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 crypto rx=0x7f7a9400e9b0 tx=0x7f7a9400ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:21.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.538+0000 7f7a9affd700 1 -- 172.21.15.16:0/1445127394 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a94009710 con 0x7f7aa40d2110 2023-10-16T13:44:21.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.538+0000 7f7a9affd700 1 -- 172.21.15.16:0/1445127394 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a9400ee50 con 0x7f7aa40d2110 2023-10-16T13:44:21.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.538+0000 7f7aa8e10700 1 -- 172.21.15.16:0/1445127394 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7aa40c7940 con 0x7f7aa40d2110 2023-10-16T13:44:21.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.538+0000 7f7a9affd700 1 -- 172.21.15.16:0/1445127394 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a9400ca50 con 0x7f7aa40d2110 2023-10-16T13:44:21.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.538+0000 7f7aa8e10700 1 -- 172.21.15.16:0/1445127394 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7aa40c72b0 con 0x7f7aa40d2110 2023-10-16T13:44:21.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.539+0000 7f7aa8e10700 1 -- 172.21.15.16:0/1445127394 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7aa41e49f0 con 0x7f7aa40d2110 2023-10-16T13:44:21.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.540+0000 7f7a9affd700 1 -- 172.21.15.16:0/1445127394 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 122) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f7a9400cbb0 con 0x7f7aa40d2110 2023-10-16T13:44:21.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.540+0000 7f7a9affd700 1 --2- 172.21.15.16:0/1445127394 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f7a8406b020 0x7f7a8406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:21.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.540+0000 7f7a9bfff700 1 -- 172.21.15.16:0/1445127394 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f7a8406b020 msgr2=0x7f7a8406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/371380443 2023-10-16T13:44:21.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.540+0000 7f7a9affd700 1 -- 172.21.15.16:0/1445127394 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 6905+0+0 (secure 0 0 0) 0x7f7a9408c070 con 0x7f7aa40d2110 2023-10-16T13:44:21.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.540+0000 7f7a9bfff700 1 --2- 172.21.15.16:0/1445127394 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f7a8406b020 0x7f7a8406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:44:21.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.545+0000 7f7a9affd700 1 -- 172.21.15.16:0/1445127394 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7a94054760 con 0x7f7aa40d2110 2023-10-16T13:44:21.668 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:44:21.667+0000 7ffb56192200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T13:44:21.700 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:44:21.698+0000 7fe4965ec200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T13:44:21.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.741+0000 7f7a9bfff700 1 -- 172.21.15.16:0/1445127394 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f7a8406b020 msgr2=0x7f7a8406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/371380443 2023-10-16T13:44:21.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.741+0000 7f7a9bfff700 1 --2- 172.21.15.16:0/1445127394 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f7a8406b020 0x7f7a8406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:44:21.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.781+0000 7f7aa8e10700 1 -- 172.21.15.16:0/1445127394 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7aa404ebb0 con 0x7f7aa40d2110 2023-10-16T13:44:21.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.781+0000 7f7a9affd700 1 -- 172.21.15.16:0/1445127394 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v194) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7a94021020 con 0x7f7aa40d2110 2023-10-16T13:44:21.784 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T13:44:21.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.783+0000 7f7aa8e10700 1 -- 172.21.15.16:0/1445127394 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f7a8406b020 msgr2=0x7f7a8406d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:44:21.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.783+0000 7f7aa8e10700 1 --2- 172.21.15.16:0/1445127394 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f7a8406b020 0x7f7a8406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:21.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.783+0000 7f7aa8e10700 1 -- 172.21.15.16:0/1445127394 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7aa40d2110 msgr2=0x7f7aa4150b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:21.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.783+0000 7f7aa8e10700 1 --2- 172.21.15.16:0/1445127394 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7aa40d2110 0x7f7aa4150b60 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 crypto rx=0x7f7a9400e9b0 tx=0x7f7a9400ecc0 comp rx=0 tx=0).stop 2023-10-16T13:44:21.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.784+0000 7f7aa8e10700 1 -- 172.21.15.16:0/1445127394 shutdown_connections 2023-10-16T13:44:21.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.784+0000 7f7aa8e10700 1 --2- 172.21.15.16:0/1445127394 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7aa415f060 0x7f7aa40c6d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:21.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.784+0000 7f7aa8e10700 1 --2- 172.21.15.16:0/1445127394 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7aa40d2110 0x7f7aa4150b60 unknown :-1 s=CLOSED pgs=2187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:21.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.784+0000 7f7aa8e10700 1 --2- 172.21.15.16:0/1445127394 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f7a8406b020 0x7f7a8406d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:21.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.784+0000 7f7aa8e10700 1 --2- 172.21.15.16:0/1445127394 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7aa415ec80 0x7f7aa40c6830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:21.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.784+0000 7f7aa8e10700 1 -- 172.21.15.16:0/1445127394 >> 172.21.15.16:0/1445127394 conn(0x7f7aa40d15f0 msgr2=0x7f7aa40b46f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:21.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.784+0000 7f7aa8e10700 1 -- 172.21.15.16:0/1445127394 shutdown_connections 2023-10-16T13:44:21.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.784+0000 7f7aa8e10700 1 -- 172.21.15.16:0/1445127394 wait complete. 2023-10-16T13:44:21.792 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:44:21.790+0000 7ffb56192200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T13:44:21.800 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T13:44:21.829 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:44:21.827+0000 7fe4965ec200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T13:44:21.869 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:44:21.867+0000 7ffb56192200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T13:44:21.914 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:44:21.912+0000 7fe4965ec200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T13:44:21.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.933+0000 7fae150f5700 1 Processor -- start 2023-10-16T13:44:21.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.933+0000 7fae150f5700 1 -- start start 2023-10-16T13:44:21.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.933+0000 7fae150f5700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fae1015b2c0 0x7fae10161340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:21.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.933+0000 7fae150f5700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fae1006e570 0x7fae1015e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:21.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.933+0000 7fae150f5700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fae1006a7e0 0x7fae1006ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:21.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.933+0000 7fae150f5700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae1006b1b0 con 0x7fae1015b2c0 2023-10-16T13:44:21.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.933+0000 7fae150f5700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae10161880 con 0x7fae1006e570 2023-10-16T13:44:21.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.933+0000 7fae150f5700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae1015b6a0 con 0x7fae1006a7e0 2023-10-16T13:44:21.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.934+0000 7fae0ed9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fae1015b2c0 0x7fae10161340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:21.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.934+0000 7fae0ed9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fae1015b2c0 0x7fae10161340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50974/0 (socket says 172.21.15.16:50974) 2023-10-16T13:44:21.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.934+0000 7fae0ed9d700 1 -- 172.21.15.16:0/812515835 learned_addr learned my addr 172.21.15.16:0/812515835 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:21.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.934+0000 7fae0e59c700 1 --2- 172.21.15.16:0/812515835 >> v2:172.21.15.49:3300/0 conn(0x7fae1006e570 0x7fae1015e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:21.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.934+0000 7fae0f59e700 1 --2- 172.21.15.16:0/812515835 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fae1006a7e0 0x7fae1006ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:21.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.934+0000 7fae0ed9d700 1 -- 172.21.15.16:0/812515835 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fae1006a7e0 msgr2=0x7fae1006ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:21.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.934+0000 7fae0ed9d700 1 --2- 172.21.15.16:0/812515835 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fae1006a7e0 0x7fae1006ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:21.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.934+0000 7fae0ed9d700 1 -- 172.21.15.16:0/812515835 >> v2:172.21.15.49:3300/0 conn(0x7fae1006e570 msgr2=0x7fae1015e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:21.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.934+0000 7fae0ed9d700 1 --2- 172.21.15.16:0/812515835 >> v2:172.21.15.49:3300/0 conn(0x7fae1006e570 0x7fae1015e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:21.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.934+0000 7fae0ed9d700 1 -- 172.21.15.16:0/812515835 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae10161b20 con 0x7fae1015b2c0 2023-10-16T13:44:21.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.934+0000 7fae0ed9d700 1 --2- 172.21.15.16:0/812515835 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae1015b2c0 0x7fae10161340 secure :-1 s=READY pgs=2188 cs=0 l=1 rev1=1 crypto rx=0x7fae10070500 tx=0x7fae00009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=320c2608aefa0e97 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:21.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.934+0000 7fae0dd9b700 1 -- 172.21.15.16:0/812515835 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae00011830 con 0x7fae1015b2c0 2023-10-16T13:44:21.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.934+0000 7fae0dd9b700 1 -- 172.21.15.16:0/812515835 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae00011990 con 0x7fae1015b2c0 2023-10-16T13:44:21.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.934+0000 7fae0dd9b700 1 -- 172.21.15.16:0/812515835 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae00011d20 con 0x7fae1015b2c0 2023-10-16T13:44:21.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.935+0000 7fae150f5700 1 -- 172.21.15.16:0/812515835 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae1015b2c0 msgr2=0x7fae10161340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:21.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.935+0000 7fae150f5700 1 --2- 172.21.15.16:0/812515835 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae1015b2c0 0x7fae10161340 secure :-1 s=READY pgs=2188 cs=0 l=1 rev1=1 crypto rx=0x7fae10070500 tx=0x7fae00009bd0 comp rx=0 tx=0).stop 2023-10-16T13:44:21.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.935+0000 7fae150f5700 1 -- 172.21.15.16:0/812515835 shutdown_connections 2023-10-16T13:44:21.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.935+0000 7fae150f5700 1 --2- 172.21.15.16:0/812515835 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fae1006a7e0 0x7fae1006ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:21.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.935+0000 7fae150f5700 1 --2- 172.21.15.16:0/812515835 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae1015b2c0 0x7fae10161340 unknown :-1 s=CLOSED pgs=2188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:21.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.935+0000 7fae150f5700 1 --2- 172.21.15.16:0/812515835 >> v2:172.21.15.49:3300/0 conn(0x7fae1006e570 0x7fae1015e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:21.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.935+0000 7fae150f5700 1 -- 172.21.15.16:0/812515835 >> 172.21.15.16:0/812515835 conn(0x7fae100b1f40 msgr2=0x7fae100b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:21.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.935+0000 7fae150f5700 1 -- 172.21.15.16:0/812515835 shutdown_connections 2023-10-16T13:44:21.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.935+0000 7fae150f5700 1 -- 172.21.15.16:0/812515835 wait complete. 2023-10-16T13:44:21.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.937+0000 7fae150f5700 1 Processor -- start 2023-10-16T13:44:21.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.937+0000 7fae150f5700 1 -- start start 2023-10-16T13:44:21.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.939+0000 7fae150f5700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fae1006a7e0 0x7fae1014f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:21.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.939+0000 7fae150f5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae1006e570 0x7fae100b44f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:21.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.939+0000 7fae150f5700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fae1015b2c0 0x7fae100cbba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:21.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.939+0000 7fae150f5700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae1014fe80 con 0x7fae1006e570 2023-10-16T13:44:21.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.939+0000 7fae150f5700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae100c7f80 con 0x7fae1015b2c0 2023-10-16T13:44:21.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.939+0000 7fae150f5700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae100b4d40 con 0x7fae1006a7e0 2023-10-16T13:44:21.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.939+0000 7fae0e59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae1006e570 0x7fae100b44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:21.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.939+0000 7fae0ed9d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fae1006a7e0 0x7fae1014f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:21.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.939+0000 7fae0e59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae1006e570 0x7fae100b44f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50982/0 (socket says 172.21.15.16:50982) 2023-10-16T13:44:21.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.939+0000 7fae0e59c700 1 -- 172.21.15.16:0/2157156357 learned_addr learned my addr 172.21.15.16:0/2157156357 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:21.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.939+0000 7fae0f59e700 1 --2- 172.21.15.16:0/2157156357 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fae1015b2c0 0x7fae100cbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:21.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.939+0000 7fae0e59c700 1 -- 172.21.15.16:0/2157156357 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fae1006a7e0 msgr2=0x7fae1014f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:21.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.939+0000 7fae0e59c700 1 --2- 172.21.15.16:0/2157156357 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fae1006a7e0 0x7fae1014f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:21.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.939+0000 7fae0e59c700 1 -- 172.21.15.16:0/2157156357 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fae1015b2c0 msgr2=0x7fae100cbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:21.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.939+0000 7fae0e59c700 1 --2- 172.21.15.16:0/2157156357 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fae1015b2c0 0x7fae100cbba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:21.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.939+0000 7fae0e59c700 1 -- 172.21.15.16:0/2157156357 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae00009550 con 0x7fae1006e570 2023-10-16T13:44:21.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.939+0000 7fae0ed9d700 1 --2- 172.21.15.16:0/2157156357 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fae1006a7e0 0x7fae1014f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:44:21.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.939+0000 7fae0f59e700 1 --2- 172.21.15.16:0/2157156357 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fae1015b2c0 0x7fae100cbba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:44:21.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.939+0000 7fae0e59c700 1 --2- 172.21.15.16:0/2157156357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae1006e570 0x7fae100b44f0 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 crypto rx=0x7fae0400e870 tx=0x7fae0400eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:21.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.939+0000 7fadf77fe700 1 -- 172.21.15.16:0/2157156357 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae04009590 con 0x7fae1006e570 2023-10-16T13:44:21.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.939+0000 7fadf77fe700 1 -- 172.21.15.16:0/2157156357 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae04013680 con 0x7fae1006e570 2023-10-16T13:44:21.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.939+0000 7fae150f5700 1 -- 172.21.15.16:0/2157156357 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae100b5020 con 0x7fae1006e570 2023-10-16T13:44:21.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.939+0000 7fadf77fe700 1 -- 172.21.15.16:0/2157156357 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae040139f0 con 0x7fae1006e570 2023-10-16T13:44:21.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.940+0000 7fae150f5700 1 -- 172.21.15.16:0/2157156357 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae100b55e0 con 0x7fae1006e570 2023-10-16T13:44:21.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.941+0000 7fae150f5700 1 -- 172.21.15.16:0/2157156357 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae1004ebb0 con 0x7fae1006e570 2023-10-16T13:44:21.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.941+0000 7fadf77fe700 1 -- 172.21.15.16:0/2157156357 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 122) v1 ==== 89343+0+0 (secure 0 0 0) 0x7fae04013b50 con 0x7fae1006e570 2023-10-16T13:44:21.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.941+0000 7fadf77fe700 1 --2- 172.21.15.16:0/2157156357 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7fadfc06b020 0x7fadfc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:21.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.941+0000 7fae0ed9d700 1 -- 172.21.15.16:0/2157156357 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7fadfc06b020 msgr2=0x7fadfc06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/371380443 2023-10-16T13:44:21.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.942+0000 7fadf77fe700 1 -- 172.21.15.16:0/2157156357 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(194..194 src has 1..194) v4 ==== 6889+0+0 (secure 0 0 0) 0x7fae040898d0 con 0x7fae1006e570 2023-10-16T13:44:21.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.942+0000 7fae0ed9d700 1 --2- 172.21.15.16:0/2157156357 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7fadfc06b020 0x7fadfc06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:44:21.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:21.946+0000 7fadf77fe700 1 -- 172.21.15.16:0/2157156357 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fae04054460 con 0x7fae1006e570 2023-10-16T13:44:21.957 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:44:21.955+0000 7ffb56192200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T13:44:22.002 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:44:22.000+0000 7fe4965ec200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T13:44:22.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.143+0000 7fae0ed9d700 1 -- 172.21.15.16:0/2157156357 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7fadfc06b020 msgr2=0x7fadfc06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/371380443 2023-10-16T13:44:22.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.143+0000 7fae0ed9d700 1 --2- 172.21.15.16:0/2157156357 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7fadfc06b020 0x7fadfc06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:44:22.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.151+0000 7fae150f5700 1 -- 172.21.15.16:0/2157156357 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fae10160fd0 con 0x7fae1006e570 2023-10-16T13:44:22.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.151+0000 7fadf77fe700 1 -- 172.21.15.16:0/2157156357 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v194) v1 ==== 104+0+11 (secure 0 0 0) 0x7fae04057a80 con 0x7fae1006e570 2023-10-16T13:44:22.154 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-10-16T13:44:22.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.154+0000 7fae150f5700 1 -- 172.21.15.16:0/2157156357 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7fadfc06b020 msgr2=0x7fadfc06d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:44:22.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.154+0000 7fae150f5700 1 --2- 172.21.15.16:0/2157156357 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7fadfc06b020 0x7fadfc06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:22.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.154+0000 7fae150f5700 1 -- 172.21.15.16:0/2157156357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae1006e570 msgr2=0x7fae100b44f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:22.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.154+0000 7fae150f5700 1 --2- 172.21.15.16:0/2157156357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae1006e570 0x7fae100b44f0 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 crypto rx=0x7fae0400e870 tx=0x7fae0400eb80 comp rx=0 tx=0).stop 2023-10-16T13:44:22.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.154+0000 7fae150f5700 1 -- 172.21.15.16:0/2157156357 shutdown_connections 2023-10-16T13:44:22.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.154+0000 7fae150f5700 1 --2- 172.21.15.16:0/2157156357 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fae1015b2c0 0x7fae100cbba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:22.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.154+0000 7fae150f5700 1 --2- 172.21.15.16:0/2157156357 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7fadfc06b020 0x7fadfc06d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:22.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.154+0000 7fae150f5700 1 --2- 172.21.15.16:0/2157156357 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fae1006a7e0 0x7fae1014f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:22.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.154+0000 7fae150f5700 1 --2- 172.21.15.16:0/2157156357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae1006e570 0x7fae100b44f0 unknown :-1 s=CLOSED pgs=2189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:22.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.154+0000 7fae150f5700 1 -- 172.21.15.16:0/2157156357 >> 172.21.15.16:0/2157156357 conn(0x7fae100b1f40 msgr2=0x7fae1015d790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:22.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.154+0000 7fae150f5700 1 -- 172.21.15.16:0/2157156357 shutdown_connections 2023-10-16T13:44:22.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.154+0000 7fae150f5700 1 -- 172.21.15.16:0/2157156357 wait complete. 2023-10-16T13:44:22.169 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T13:44:22.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.336+0000 7fb0f04e1700 1 Processor -- start 2023-10-16T13:44:22.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.336+0000 7fb0f04e1700 1 -- start start 2023-10-16T13:44:22.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.337+0000 7fb0f04e1700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb0e815b2c0 0x7fb0e8161340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:22.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.337+0000 7fb0f04e1700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fb0e806a7e0 0x7fb0e806ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:22.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.337+0000 7fb0f04e1700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0e815ecb0 con 0x7fb0e806e570 2023-10-16T13:44:22.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.337+0000 7fb0f04e1700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0e806b190 con 0x7fb0e815b2c0 2023-10-16T13:44:22.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.337+0000 7fb0f04e1700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0e8161880 con 0x7fb0e806a7e0 2023-10-16T13:44:22.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.337+0000 7fb0ee27d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb0e815b2c0 0x7fb0e8161340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:22.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.338+0000 7fb0ee27d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb0e815b2c0 0x7fb0e8161340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51008/0 (socket says 172.21.15.16:51008) 2023-10-16T13:44:22.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.338+0000 7fb0eda7c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb0e806e570 0x7fb0e815e5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:40282/0 (socket says 172.21.15.16:40282) 2023-10-16T13:44:22.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.338+0000 7fb0ee27d700 1 -- 172.21.15.16:0/1154212943 learned_addr learned my addr 172.21.15.16:0/1154212943 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:22.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.338+0000 7fb0eea7e700 1 --2- 172.21.15.16:0/1154212943 >> v2:172.21.15.49:3300/0 conn(0x7fb0e806a7e0 0x7fb0e806ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:22.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.338+0000 7fb0ee27d700 1 -- 172.21.15.16:0/1154212943 >> v2:172.21.15.49:3300/0 conn(0x7fb0e806a7e0 msgr2=0x7fb0e806ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:22.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.338+0000 7fb0ee27d700 1 --2- 172.21.15.16:0/1154212943 >> v2:172.21.15.49:3300/0 conn(0x7fb0e806a7e0 0x7fb0e806ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:22.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.338+0000 7fb0ee27d700 1 -- 172.21.15.16:0/1154212943 >> v1:172.21.15.16:6789/0 conn(0x7fb0e806e570 legacy=0x7fb0e815e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:22.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.338+0000 7fb0ee27d700 1 -- 172.21.15.16:0/1154212943 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0e815b6a0 con 0x7fb0e815b2c0 2023-10-16T13:44:22.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.338+0000 7fb0ed27b700 1 -- 172.21.15.16:0/1154212943 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1807845988 0 0) 0x7fb0e815ecb0 con 0x7fb0e806e570 2023-10-16T13:44:22.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.338+0000 7fb0ee27d700 1 --2- 172.21.15.16:0/1154212943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0e815b2c0 0x7fb0e8161340 secure :-1 s=READY pgs=2191 cs=0 l=1 rev1=1 crypto rx=0x7fb0e806b880 tx=0x7fb0e4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42fe9397a31e7b9c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:22.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.338+0000 7fb0ed27b700 1 -- 172.21.15.16:0/1154212943 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0e4011830 con 0x7fb0e815b2c0 2023-10-16T13:44:22.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.338+0000 7fb0ed27b700 1 -- 172.21.15.16:0/1154212943 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0e4011990 con 0x7fb0e815b2c0 2023-10-16T13:44:22.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.338+0000 7fb0ed27b700 1 -- 172.21.15.16:0/1154212943 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0e4011d20 con 0x7fb0e815b2c0 2023-10-16T13:44:22.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.339+0000 7fb0f04e1700 1 -- 172.21.15.16:0/1154212943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0e815b2c0 msgr2=0x7fb0e8161340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:22.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.339+0000 7fb0f04e1700 1 --2- 172.21.15.16:0/1154212943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0e815b2c0 0x7fb0e8161340 secure :-1 s=READY pgs=2191 cs=0 l=1 rev1=1 crypto rx=0x7fb0e806b880 tx=0x7fb0e4009bd0 comp rx=0 tx=0).stop 2023-10-16T13:44:22.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.339+0000 7fb0f04e1700 1 -- 172.21.15.16:0/1154212943 shutdown_connections 2023-10-16T13:44:22.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.339+0000 7fb0f04e1700 1 --2- 172.21.15.16:0/1154212943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0e815b2c0 0x7fb0e8161340 unknown :-1 s=CLOSED pgs=2191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:22.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.339+0000 7fb0f04e1700 1 --2- 172.21.15.16:0/1154212943 >> v2:172.21.15.49:3300/0 conn(0x7fb0e806a7e0 0x7fb0e806ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:22.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.339+0000 7fb0f04e1700 1 -- 172.21.15.16:0/1154212943 >> 172.21.15.16:0/1154212943 conn(0x7fb0e80b1f40 msgr2=0x7fb0e80b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:22.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.339+0000 7fb0f04e1700 1 -- 172.21.15.16:0/1154212943 shutdown_connections 2023-10-16T13:44:22.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.339+0000 7fb0f04e1700 1 -- 172.21.15.16:0/1154212943 wait complete. 2023-10-16T13:44:22.348 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:44:22.344+0000 7ffb56192200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T13:44:22.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.341+0000 7fb0f04e1700 1 Processor -- start 2023-10-16T13:44:22.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.341+0000 7fb0f04e1700 1 -- start start 2023-10-16T13:44:22.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.342+0000 7fb0f04e1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0e806a7e0 0x7fb0e8157970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:22.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.342+0000 7fb0f04e1700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb0e806e570 0x7fb0e80cb530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:22.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.342+0000 7fb0f04e1700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb0e815b2c0 0x7fb0e80cba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:22.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.342+0000 7fb0f04e1700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e8157eb0 con 0x7fb0e806a7e0 2023-10-16T13:44:22.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.342+0000 7fb0f04e1700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e81f4170 con 0x7fb0e815b2c0 2023-10-16T13:44:22.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.342+0000 7fb0f04e1700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e81f4710 con 0x7fb0e806e570 2023-10-16T13:44:22.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.342+0000 7fb0ee27d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0e806a7e0 0x7fb0e8157970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:22.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.342+0000 7fb0ee27d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0e806a7e0 0x7fb0e8157970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51014/0 (socket says 172.21.15.16:51014) 2023-10-16T13:44:22.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.342+0000 7fb0ee27d700 1 -- 172.21.15.16:0/4078427738 learned_addr learned my addr 172.21.15.16:0/4078427738 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:22.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.342+0000 7fb0eda7c700 1 --2- 172.21.15.16:0/4078427738 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb0e806e570 0x7fb0e80cb530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:22.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.342+0000 7fb0eea7e700 1 --2- 172.21.15.16:0/4078427738 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb0e815b2c0 0x7fb0e80cba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:22.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.342+0000 7fb0ee27d700 1 -- 172.21.15.16:0/4078427738 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb0e806e570 msgr2=0x7fb0e80cb530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:22.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.342+0000 7fb0ee27d700 1 --2- 172.21.15.16:0/4078427738 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb0e806e570 0x7fb0e80cb530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:22.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.342+0000 7fb0ee27d700 1 -- 172.21.15.16:0/4078427738 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb0e815b2c0 msgr2=0x7fb0e80cba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:22.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.342+0000 7fb0ee27d700 1 --2- 172.21.15.16:0/4078427738 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb0e815b2c0 0x7fb0e80cba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:22.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.342+0000 7fb0ee27d700 1 -- 172.21.15.16:0/4078427738 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0e4009550 con 0x7fb0e806a7e0 2023-10-16T13:44:22.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.342+0000 7fb0ee27d700 1 --2- 172.21.15.16:0/4078427738 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0e806a7e0 0x7fb0e8157970 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 crypto rx=0x7fb0e4009c10 tx=0x7fb0e4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:22.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.342+0000 7fb0deffd700 1 -- 172.21.15.16:0/4078427738 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0e4004590 con 0x7fb0e806a7e0 2023-10-16T13:44:22.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.342+0000 7fb0deffd700 1 -- 172.21.15.16:0/4078427738 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0e401e070 con 0x7fb0e806a7e0 2023-10-16T13:44:22.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.342+0000 7fb0f04e1700 1 -- 172.21.15.16:0/4078427738 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0e81f49f0 con 0x7fb0e806a7e0 2023-10-16T13:44:22.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.342+0000 7fb0deffd700 1 -- 172.21.15.16:0/4078427738 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0e40038c0 con 0x7fb0e806a7e0 2023-10-16T13:44:22.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.343+0000 7fb0f04e1700 1 -- 172.21.15.16:0/4078427738 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0e81f33d0 con 0x7fb0e806a7e0 2023-10-16T13:44:22.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.344+0000 7fb0f04e1700 1 -- 172.21.15.16:0/4078427738 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0e81f4f10 con 0x7fb0e806a7e0 2023-10-16T13:44:22.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.344+0000 7fb0deffd700 1 -- 172.21.15.16:0/4078427738 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 122) v1 ==== 89343+0+0 (secure 0 0 0) 0x7fb0e4003a20 con 0x7fb0e806a7e0 2023-10-16T13:44:22.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.344+0000 7fb0deffd700 1 --2- 172.21.15.16:0/4078427738 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7fb0cc06b020 0x7fb0cc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:22.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.345+0000 7fb0eda7c700 1 -- 172.21.15.16:0/4078427738 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7fb0cc06b020 msgr2=0x7fb0cc06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/371380443 2023-10-16T13:44:22.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.345+0000 7fb0deffd700 1 -- 172.21.15.16:0/4078427738 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(194..194 src has 1..194) v4 ==== 6889+0+0 (secure 0 0 0) 0x7fb0e408aa20 con 0x7fb0e806a7e0 2023-10-16T13:44:22.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.345+0000 7fb0eda7c700 1 --2- 172.21.15.16:0/4078427738 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7fb0cc06b020 0x7fb0cc06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:44:22.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.349+0000 7fb0deffd700 1 -- 172.21.15.16:0/4078427738 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb0e81f4f10 con 0x7fb0e806a7e0 2023-10-16T13:44:22.400 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:44:22.397+0000 7fe4965ec200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T13:44:22.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.546+0000 7fb0eda7c700 1 -- 172.21.15.16:0/4078427738 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7fb0cc06b020 msgr2=0x7fb0cc06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/371380443 2023-10-16T13:44:22.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.546+0000 7fb0eda7c700 1 --2- 172.21.15.16:0/4078427738 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7fb0cc06b020 0x7fb0cc06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:44:22.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.555+0000 7fb0f04e1700 1 -- 172.21.15.16:0/4078427738 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb0e81f4f10 con 0x7fb0e806a7e0 2023-10-16T13:44:22.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.555+0000 7fb0deffd700 1 -- 172.21.15.16:0/4078427738 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v194) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb0e81f4f10 con 0x7fb0e806a7e0 2023-10-16T13:44:22.558 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-10-16T13:44:22.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.557+0000 7fb0f04e1700 1 -- 172.21.15.16:0/4078427738 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7fb0cc06b020 msgr2=0x7fb0cc06d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:44:22.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.557+0000 7fb0f04e1700 1 --2- 172.21.15.16:0/4078427738 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7fb0cc06b020 0x7fb0cc06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:22.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.557+0000 7fb0f04e1700 1 -- 172.21.15.16:0/4078427738 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0e806a7e0 msgr2=0x7fb0e8157970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:22.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.557+0000 7fb0f04e1700 1 --2- 172.21.15.16:0/4078427738 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0e806a7e0 0x7fb0e8157970 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 crypto rx=0x7fb0e4009c10 tx=0x7fb0e4004750 comp rx=0 tx=0).stop 2023-10-16T13:44:22.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.557+0000 7fb0f04e1700 1 -- 172.21.15.16:0/4078427738 shutdown_connections 2023-10-16T13:44:22.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.557+0000 7fb0f04e1700 1 --2- 172.21.15.16:0/4078427738 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb0e815b2c0 0x7fb0e80cba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:22.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.557+0000 7fb0f04e1700 1 --2- 172.21.15.16:0/4078427738 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0e806a7e0 0x7fb0e8157970 unknown :-1 s=CLOSED pgs=2192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:22.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.557+0000 7fb0f04e1700 1 --2- 172.21.15.16:0/4078427738 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7fb0cc06b020 0x7fb0cc06d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:22.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.557+0000 7fb0f04e1700 1 --2- 172.21.15.16:0/4078427738 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb0e806e570 0x7fb0e80cb530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:22.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.557+0000 7fb0f04e1700 1 -- 172.21.15.16:0/4078427738 >> 172.21.15.16:0/4078427738 conn(0x7fb0e80b1f40 msgr2=0x7fb0e80caa20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:22.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.557+0000 7fb0f04e1700 1 -- 172.21.15.16:0/4078427738 shutdown_connections 2023-10-16T13:44:22.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.557+0000 7fb0f04e1700 1 -- 172.21.15.16:0/4078427738 wait complete. 2023-10-16T13:44:22.573 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T13:44:22.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.697+0000 7fd97f6de700 1 Processor -- start 2023-10-16T13:44:22.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.698+0000 7fd97f6de700 1 -- start start 2023-10-16T13:44:22.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.698+0000 7fd97f6de700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd9780cdfa0 0x7fd9780b2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:22.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.698+0000 7fd97f6de700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fd9780d0bb0 0x7fd9780b2980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:22.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.698+0000 7fd97f6de700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9780d0fe0 con 0x7fd9780ce380 2023-10-16T13:44:22.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.698+0000 7fd97f6de700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9780d1210 con 0x7fd9780d0bb0 2023-10-16T13:44:22.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.698+0000 7fd97f6de700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9780cbc30 con 0x7fd9780cdfa0 2023-10-16T13:44:22.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.699+0000 7fd97cc79700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd9780ce380 0x7fd9780d04a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:40298/0 (socket says 172.21.15.16:40298) 2023-10-16T13:44:22.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.699+0000 7fd97cc79700 1 -- 172.21.15.16:0/3999715797 learned_addr learned my addr 172.21.15.16:0/3999715797 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:44:22.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.699+0000 7fd97d47a700 1 --2- 172.21.15.16:0/3999715797 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd9780cdfa0 0x7fd9780b2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:22.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.699+0000 7fd97dc7b700 1 --2- 172.21.15.16:0/3999715797 >> v2:172.21.15.49:3300/0 conn(0x7fd9780d0bb0 0x7fd9780b2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:22.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.699+0000 7fd96ffff700 1 -- 172.21.15.16:0/3999715797 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2831244375 0 0) 0x7fd9780d0fe0 con 0x7fd9780ce380 2023-10-16T13:44:22.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.699+0000 7fd96ffff700 1 -- 172.21.15.16:0/3999715797 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9640035b0 con 0x7fd9780ce380 2023-10-16T13:44:22.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.699+0000 7fd97d47a700 1 -- 172.21.15.16:0/3999715797 >> v2:172.21.15.49:3300/0 conn(0x7fd9780d0bb0 msgr2=0x7fd9780b2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:22.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.699+0000 7fd97d47a700 1 --2- 172.21.15.16:0/3999715797 >> v2:172.21.15.49:3300/0 conn(0x7fd9780d0bb0 0x7fd9780b2980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:22.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.699+0000 7fd97d47a700 1 -- 172.21.15.16:0/3999715797 >> v1:172.21.15.16:6789/0 conn(0x7fd9780ce380 legacy=0x7fd9780d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:22.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.699+0000 7fd97d47a700 1 -- 172.21.15.16:0/3999715797 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9780cbf10 con 0x7fd9780cdfa0 2023-10-16T13:44:22.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.699+0000 7fd97d47a700 1 --2- 172.21.15.16:0/3999715797 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd9780cdfa0 0x7fd9780b2440 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fd974009840 tx=0x7fd974009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3b879a8599729bf server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:22.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.699+0000 7fd96ffff700 1 -- 172.21.15.16:0/3999715797 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd974011830 con 0x7fd9780cdfa0 2023-10-16T13:44:22.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.699+0000 7fd96ffff700 1 -- 172.21.15.16:0/3999715797 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd974011990 con 0x7fd9780cdfa0 2023-10-16T13:44:22.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.699+0000 7fd96ffff700 1 -- 172.21.15.16:0/3999715797 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd974011d00 con 0x7fd9780cdfa0 2023-10-16T13:44:22.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.700+0000 7fd97f6de700 1 -- 172.21.15.16:0/3999715797 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd9780cdfa0 msgr2=0x7fd9780b2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:22.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.700+0000 7fd97f6de700 1 --2- 172.21.15.16:0/3999715797 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd9780cdfa0 0x7fd9780b2440 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fd974009840 tx=0x7fd974009b50 comp rx=0 tx=0).stop 2023-10-16T13:44:22.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.700+0000 7fd97f6de700 1 -- 172.21.15.16:0/3999715797 shutdown_connections 2023-10-16T13:44:22.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.700+0000 7fd97f6de700 1 --2- 172.21.15.16:0/3999715797 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd9780cdfa0 0x7fd9780b2440 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:22.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.700+0000 7fd97f6de700 1 --2- 172.21.15.16:0/3999715797 >> v2:172.21.15.49:3300/0 conn(0x7fd9780d0bb0 0x7fd9780b2980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:22.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.700+0000 7fd97f6de700 1 -- 172.21.15.16:0/3999715797 >> 172.21.15.16:0/3999715797 conn(0x7fd9780d14b0 msgr2=0x7fd9780d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:22.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.700+0000 7fd97f6de700 1 -- 172.21.15.16:0/3999715797 shutdown_connections 2023-10-16T13:44:22.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.700+0000 7fd97f6de700 1 -- 172.21.15.16:0/3999715797 wait complete. 2023-10-16T13:44:22.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.701+0000 7fd97f6de700 1 Processor -- start 2023-10-16T13:44:22.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.701+0000 7fd97f6de700 1 -- start start 2023-10-16T13:44:22.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.702+0000 7fd97f6de700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd9780cdfa0 0x7fd978159ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:22.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.702+0000 7fd97f6de700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9780ce380 0x7fd9780c8fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:22.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.702+0000 7fd97f6de700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd9780d0bb0 0x7fd9780c94e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:22.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.702+0000 7fd97f6de700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9781552c0 con 0x7fd9780ce380 2023-10-16T13:44:22.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.702+0000 7fd97f6de700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9780c9f20 con 0x7fd9780cdfa0 2023-10-16T13:44:22.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.702+0000 7fd97f6de700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd97815a430 con 0x7fd9780d0bb0 2023-10-16T13:44:22.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.702+0000 7fd97cc79700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9780ce380 0x7fd9780c8fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:22.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.702+0000 7fd97cc79700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9780ce380 0x7fd9780c8fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51032/0 (socket says 172.21.15.16:51032) 2023-10-16T13:44:22.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.702+0000 7fd97cc79700 1 -- 172.21.15.16:0/314969896 learned_addr learned my addr 172.21.15.16:0/314969896 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:22.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.702+0000 7fd97dc7b700 1 --2- 172.21.15.16:0/314969896 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd9780d0bb0 0x7fd9780c94e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:22.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.702+0000 7fd97d47a700 1 --2- 172.21.15.16:0/314969896 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd9780cdfa0 0x7fd978159ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:22.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.703+0000 7fd97dc7b700 1 -- 172.21.15.16:0/314969896 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd9780cdfa0 msgr2=0x7fd978159ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:22.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.703+0000 7fd97dc7b700 1 --2- 172.21.15.16:0/314969896 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd9780cdfa0 0x7fd978159ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:22.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.703+0000 7fd97dc7b700 1 -- 172.21.15.16:0/314969896 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9780ce380 msgr2=0x7fd9780c8fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:22.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.703+0000 7fd97dc7b700 1 --2- 172.21.15.16:0/314969896 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9780ce380 0x7fd9780c8fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:22.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.703+0000 7fd97dc7b700 1 -- 172.21.15.16:0/314969896 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9740094a0 con 0x7fd9780d0bb0 2023-10-16T13:44:22.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.703+0000 7fd97dc7b700 1 --2- 172.21.15.16:0/314969896 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd9780d0bb0 0x7fd9780c94e0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd97000eba0 tx=0x7fd97000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:22.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.703+0000 7fd96dffb700 1 -- 172.21.15.16:0/314969896 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd970009a80 con 0x7fd9780d0bb0 2023-10-16T13:44:22.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.703+0000 7fd96dffb700 1 -- 172.21.15.16:0/314969896 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd97000f040 con 0x7fd9780d0bb0 2023-10-16T13:44:22.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.703+0000 7fd97f6de700 1 -- 172.21.15.16:0/314969896 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9780b5310 con 0x7fd9780d0bb0 2023-10-16T13:44:22.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.703+0000 7fd96dffb700 1 -- 172.21.15.16:0/314969896 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd97000cc40 con 0x7fd9780d0bb0 2023-10-16T13:44:22.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.703+0000 7fd97f6de700 1 -- 172.21.15.16:0/314969896 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9780b58d0 con 0x7fd9780d0bb0 2023-10-16T13:44:22.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.704+0000 7fd97f6de700 1 -- 172.21.15.16:0/314969896 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd97804ebb0 con 0x7fd9780d0bb0 2023-10-16T13:44:22.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.705+0000 7fd96dffb700 1 -- 172.21.15.16:0/314969896 <== mon.2 v2:172.21.15.49:3301/0 4 ==== mgrmap(e 122) v1 ==== 89343+0+0 (secure 0 0 0) 0x7fd970014480 con 0x7fd9780d0bb0 2023-10-16T13:44:22.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.705+0000 7fd96dffb700 1 --2- 172.21.15.16:0/314969896 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7fd95c06b020 0x7fd95c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:22.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.705+0000 7fd97d47a700 1 -- 172.21.15.16:0/314969896 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7fd95c06b020 msgr2=0x7fd95c06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/371380443 2023-10-16T13:44:22.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.705+0000 7fd96dffb700 1 -- 172.21.15.16:0/314969896 <== mon.2 v2:172.21.15.49:3301/0 5 ==== osd_map(194..194 src has 1..194) v4 ==== 6889+0+0 (secure 0 0 0) 0x7fd97008a680 con 0x7fd9780d0bb0 2023-10-16T13:44:22.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.705+0000 7fd97d47a700 1 --2- 172.21.15.16:0/314969896 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7fd95c06b020 0x7fd95c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:44:22.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.714+0000 7fd96dffb700 1 -- 172.21.15.16:0/314969896 <== mon.2 v2:172.21.15.49:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd970055060 con 0x7fd9780d0bb0 2023-10-16T13:44:22.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.906+0000 7fd97d47a700 1 -- 172.21.15.16:0/314969896 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7fd95c06b020 msgr2=0x7fd95c06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/371380443 2023-10-16T13:44:22.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.906+0000 7fd97d47a700 1 --2- 172.21.15.16:0/314969896 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7fd95c06b020 0x7fd95c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:44:22.958 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:44:22.956+0000 7ffb56192200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T13:44:22.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.986+0000 7fd97f6de700 1 -- 172.21.15.16:0/314969896 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fd9780b6e40 con 0x7fd9780d0bb0 2023-10-16T13:44:22.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.987+0000 7fd96dffb700 1 -- 172.21.15.16:0/314969896 <== mon.2 v2:172.21.15.49:3301/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v194) v1 ==== 81+0+936 (secure 0 0 0) 0x7fd970058680 con 0x7fd9780d0bb0 2023-10-16T13:44:22.990 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6833/3952564978 2023-10-17T13:39:22.557973+0000 2023-10-16T13:44:22.990 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6832/3952564978 2023-10-17T13:39:22.557973+0000 2023-10-16T13:44:22.990 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2554840790 2023-10-17T13:39:22.557973+0000 2023-10-16T13:44:22.990 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/360056771 2023-10-17T13:39:22.557973+0000 2023-10-16T13:44:22.990 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3520685377 2023-10-17T13:39:22.557973+0000 2023-10-16T13:44:22.991 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/1571009468 2023-10-17T13:28:34.631949+0000 2023-10-16T13:44:22.991 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/678119374 2023-10-17T13:28:34.631949+0000 2023-10-16T13:44:22.991 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3920247727 2023-10-17T13:28:34.631949+0000 2023-10-16T13:44:22.991 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/857902049 2023-10-17T13:39:22.557973+0000 2023-10-16T13:44:22.991 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6833/4211329181 2023-10-17T13:28:34.631949+0000 2023-10-16T13:44:22.991 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6832/4211329181 2023-10-17T13:28:34.631949+0000 2023-10-16T13:44:22.992 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/1548506671 2023-10-17T13:39:22.557973+0000 2023-10-16T13:44:22.992 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2852193608 2023-10-17T13:28:34.631949+0000 2023-10-16T13:44:22.992 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/1641955134 2023-10-17T13:39:22.557973+0000 2023-10-16T13:44:22.992 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/471630266 2023-10-17T13:28:34.631949+0000 2023-10-16T13:44:22.992 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3153082432 2023-10-17T13:28:34.631949+0000 2023-10-16T13:44:22.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.990+0000 7fd97f6de700 1 -- 172.21.15.16:0/314969896 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7fd95c06b020 msgr2=0x7fd95c06d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:44:22.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.990+0000 7fd97f6de700 1 --2- 172.21.15.16:0/314969896 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7fd95c06b020 0x7fd95c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:22.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.990+0000 7fd97f6de700 1 -- 172.21.15.16:0/314969896 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd9780d0bb0 msgr2=0x7fd9780c94e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:22.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.990+0000 7fd97f6de700 1 --2- 172.21.15.16:0/314969896 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd9780d0bb0 0x7fd9780c94e0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd97000eba0 tx=0x7fd97000eeb0 comp rx=0 tx=0).stop 2023-10-16T13:44:22.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.990+0000 7fd97f6de700 1 -- 172.21.15.16:0/314969896 shutdown_connections 2023-10-16T13:44:22.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.990+0000 7fd97f6de700 1 --2- 172.21.15.16:0/314969896 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7fd95c06b020 0x7fd95c06d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:22.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.990+0000 7fd97f6de700 1 --2- 172.21.15.16:0/314969896 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd9780d0bb0 0x7fd9780c94e0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:22.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.990+0000 7fd97f6de700 1 --2- 172.21.15.16:0/314969896 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd9780cdfa0 0x7fd978159ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:22.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.990+0000 7fd97f6de700 1 --2- 172.21.15.16:0/314969896 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9780ce380 0x7fd9780c8fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:22.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.990+0000 7fd97f6de700 1 -- 172.21.15.16:0/314969896 >> 172.21.15.16:0/314969896 conn(0x7fd9780d14b0 msgr2=0x7fd9780d0330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:22.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.990+0000 7fd97f6de700 1 -- 172.21.15.16:0/314969896 shutdown_connections 2023-10-16T13:44:22.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:22.990+0000 7fd97f6de700 1 -- 172.21.15.16:0/314969896 wait complete. 2023-10-16T13:44:22.996 INFO:teuthology.orchestra.run.smithi016.stderr:listed 16 entries 2023-10-16T13:44:23.005 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-16T13:44:23.006 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:44:23.006 DEBUG:teuthology.orchestra.run.smithi016:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-16T13:44:23.017 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:44:23.014+0000 7fe4965ec200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T13:44:23.030 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:44:23.029+0000 7ffb56192200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T13:44:23.049 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2023-10-16T13:44:23.049 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:44:23.049 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-16T13:44:23.097 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:44:23.094+0000 7fe4965ec200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T13:44:23.106 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T13:44:23.106 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:44:23.124 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284204k,nr_inodes=4071051,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=753 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260540k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T13:44:23.125 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T13:44:23.125 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:44:23.164 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16287364k,nr_inodes=4071841,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20251 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3261172k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T13:44:23.165 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T13:44:23.165 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:44:23.183 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284204k,nr_inodes=4071051,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=753 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260540k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T13:44:23.183 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:44:23.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.331+0000 7f2a05b9f700 1 Processor -- start 2023-10-16T13:44:23.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.331+0000 7f2a05b9f700 1 -- start start 2023-10-16T13:44:23.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.332+0000 7f2a05b9f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2a000cf4a0 0x7f2a000726e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:23.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.332+0000 7f2a05b9f700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f2a000cf880 0x7f2a000b52f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:23.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.332+0000 7f2a05b9f700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a0015bc80 con 0x7f2a000b5830 2023-10-16T13:44:23.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.332+0000 7f2a05b9f700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a00072c20 con 0x7f2a000cf4a0 2023-10-16T13:44:23.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.332+0000 7f2a05b9f700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a0015b070 con 0x7f2a000cf880 2023-10-16T13:44:23.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.332+0000 7f29ff7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2a000cf4a0 0x7f2a000726e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:23.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.332+0000 7f29ff7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2a000cf4a0 0x7f2a000726e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51040/0 (socket says 172.21.15.16:51040) 2023-10-16T13:44:23.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.332+0000 7f29fffff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2a000b5830 0x7f2a0015a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:40318/0 (socket says 172.21.15.16:40318) 2023-10-16T13:44:23.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.332+0000 7f29ff7fe700 1 -- 172.21.15.16:0/2650419960 learned_addr learned my addr 172.21.15.16:0/2650419960 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:23.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.332+0000 7f29feffd700 1 --2- 172.21.15.16:0/2650419960 >> v2:172.21.15.49:3300/0 conn(0x7f2a000cf880 0x7f2a000b52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:23.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.333+0000 7f29ff7fe700 1 -- 172.21.15.16:0/2650419960 >> v2:172.21.15.49:3300/0 conn(0x7f2a000cf880 msgr2=0x7f2a000b52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:23.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.333+0000 7f29ff7fe700 1 --2- 172.21.15.16:0/2650419960 >> v2:172.21.15.49:3300/0 conn(0x7f2a000cf880 0x7f2a000b52f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:23.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.333+0000 7f29ff7fe700 1 -- 172.21.15.16:0/2650419960 >> v1:172.21.15.16:6789/0 conn(0x7f2a000b5830 legacy=0x7f2a0015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:23.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.333+0000 7f29ff7fe700 1 -- 172.21.15.16:0/2650419960 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a0015b350 con 0x7f2a000cf4a0 2023-10-16T13:44:23.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.333+0000 7f29fe7fc700 1 -- 172.21.15.16:0/2650419960 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2204691793 0 0) 0x7f2a0015bc80 con 0x7f2a000b5830 2023-10-16T13:44:23.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.333+0000 7f29ff7fe700 1 --2- 172.21.15.16:0/2650419960 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a000cf4a0 0x7f2a000726e0 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x7f2a00071da0 tx=0x7f29f0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cde40480323a30ae server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:23.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.333+0000 7f29fe7fc700 1 -- 172.21.15.16:0/2650419960 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f0011830 con 0x7f2a000cf4a0 2023-10-16T13:44:23.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.333+0000 7f29fe7fc700 1 -- 172.21.15.16:0/2650419960 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29f0011990 con 0x7f2a000cf4a0 2023-10-16T13:44:23.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.333+0000 7f29fe7fc700 1 -- 172.21.15.16:0/2650419960 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f0011d20 con 0x7f2a000cf4a0 2023-10-16T13:44:23.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.334+0000 7f2a05b9f700 1 -- 172.21.15.16:0/2650419960 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a000cf4a0 msgr2=0x7f2a000726e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:23.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.334+0000 7f2a05b9f700 1 --2- 172.21.15.16:0/2650419960 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a000cf4a0 0x7f2a000726e0 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x7f2a00071da0 tx=0x7f29f0009bd0 comp rx=0 tx=0).stop 2023-10-16T13:44:23.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.334+0000 7f2a05b9f700 1 -- 172.21.15.16:0/2650419960 shutdown_connections 2023-10-16T13:44:23.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.334+0000 7f2a05b9f700 1 --2- 172.21.15.16:0/2650419960 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a000cf4a0 0x7f2a000726e0 unknown :-1 s=CLOSED pgs=2195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:23.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.334+0000 7f2a05b9f700 1 --2- 172.21.15.16:0/2650419960 >> v2:172.21.15.49:3300/0 conn(0x7f2a000cf880 0x7f2a000b52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:23.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.334+0000 7f2a05b9f700 1 -- 172.21.15.16:0/2650419960 >> 172.21.15.16:0/2650419960 conn(0x7f2a000b1fd0 msgr2=0x7f2a000b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:23.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.334+0000 7f2a05b9f700 1 -- 172.21.15.16:0/2650419960 shutdown_connections 2023-10-16T13:44:23.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.334+0000 7f2a05b9f700 1 -- 172.21.15.16:0/2650419960 wait complete. 2023-10-16T13:44:23.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.335+0000 7f2a05b9f700 1 Processor -- start 2023-10-16T13:44:23.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.335+0000 7f2a05b9f700 1 -- start start 2023-10-16T13:44:23.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.336+0000 7f2a05b9f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a000b5830 0x7f2a000ce260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:23.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.336+0000 7f2a05b9f700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2a000cf4a0 0x7f2a000ca010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:23.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.336+0000 7f2a05b9f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2a000cf880 0x7f2a000ca550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:23.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.336+0000 7f2a05b9f700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a00151de0 con 0x7f2a000b5830 2023-10-16T13:44:23.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.336+0000 7f2a05b9f700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a000c7700 con 0x7f2a000cf4a0 2023-10-16T13:44:23.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.336+0000 7f2a05b9f700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a000c79e0 con 0x7f2a000cf880 2023-10-16T13:44:23.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.336+0000 7f29ff7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a000b5830 0x7f2a000ce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:23.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.336+0000 7f29ff7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a000b5830 0x7f2a000ce260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51046/0 (socket says 172.21.15.16:51046) 2023-10-16T13:44:23.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.336+0000 7f29ff7fe700 1 -- 172.21.15.16:0/2827669966 learned_addr learned my addr 172.21.15.16:0/2827669966 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:23.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.336+0000 7f29feffd700 1 --2- 172.21.15.16:0/2827669966 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2a000cf4a0 0x7f2a000ca010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:23.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.336+0000 7f29fffff700 1 --2- 172.21.15.16:0/2827669966 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2a000cf880 0x7f2a000ca550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:23.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.336+0000 7f29ff7fe700 1 -- 172.21.15.16:0/2827669966 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2a000cf880 msgr2=0x7f2a000ca550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:23.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.336+0000 7f29ff7fe700 1 --2- 172.21.15.16:0/2827669966 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2a000cf880 0x7f2a000ca550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:23.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.336+0000 7f29ff7fe700 1 -- 172.21.15.16:0/2827669966 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2a000cf4a0 msgr2=0x7f2a000ca010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:23.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.336+0000 7f29ff7fe700 1 --2- 172.21.15.16:0/2827669966 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2a000cf4a0 0x7f2a000ca010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:23.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.336+0000 7f29ff7fe700 1 -- 172.21.15.16:0/2827669966 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29f0009550 con 0x7f2a000b5830 2023-10-16T13:44:23.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.336+0000 7f29fffff700 1 --2- 172.21.15.16:0/2827669966 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2a000cf880 0x7f2a000ca550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:44:23.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.337+0000 7f29ff7fe700 1 --2- 172.21.15.16:0/2827669966 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a000b5830 0x7f2a000ce260 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 crypto rx=0x7f29f0009c10 tx=0x7f29f0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:23.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.337+0000 7f29e7fff700 1 -- 172.21.15.16:0/2827669966 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f00036a0 con 0x7f2a000b5830 2023-10-16T13:44:23.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.337+0000 7f29e7fff700 1 -- 172.21.15.16:0/2827669966 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29f0003800 con 0x7f2a000b5830 2023-10-16T13:44:23.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.337+0000 7f2a05b9f700 1 -- 172.21.15.16:0/2827669966 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a000c7cc0 con 0x7f2a000b5830 2023-10-16T13:44:23.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.337+0000 7f29e7fff700 1 -- 172.21.15.16:0/2827669966 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f0003b70 con 0x7f2a000b5830 2023-10-16T13:44:23.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.337+0000 7f2a05b9f700 1 -- 172.21.15.16:0/2827669966 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a000c47d0 con 0x7f2a000b5830 2023-10-16T13:44:23.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.338+0000 7f2a05b9f700 1 -- 172.21.15.16:0/2827669966 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a000c4960 con 0x7f2a000b5830 2023-10-16T13:44:23.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.338+0000 7f29e7fff700 1 -- 172.21.15.16:0/2827669966 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 122) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f29f0003cd0 con 0x7f2a000b5830 2023-10-16T13:44:23.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.339+0000 7f29e7fff700 1 --2- 172.21.15.16:0/2827669966 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f29ec06b020 0x7f29ec06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:23.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.339+0000 7f29feffd700 1 -- 172.21.15.16:0/2827669966 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f29ec06b020 msgr2=0x7f29ec06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/371380443 2023-10-16T13:44:23.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.339+0000 7f29e7fff700 1 -- 172.21.15.16:0/2827669966 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(194..194 src has 1..194) v4 ==== 6889+0+0 (secure 0 0 0) 0x7f29f0021030 con 0x7f2a000b5830 2023-10-16T13:44:23.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.339+0000 7f29feffd700 1 --2- 172.21.15.16:0/2827669966 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f29ec06b020 0x7f29ec06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:44:23.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.344+0000 7f29e7fff700 1 -- 172.21.15.16:0/2827669966 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2a000c4960 con 0x7f2a000b5830 2023-10-16T13:44:23.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.540+0000 7f29feffd700 1 -- 172.21.15.16:0/2827669966 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f29ec06b020 msgr2=0x7f29ec06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/371380443 2023-10-16T13:44:23.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.540+0000 7f29feffd700 1 --2- 172.21.15.16:0/2827669966 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f29ec06b020 0x7f29ec06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:44:23.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.598+0000 7f2a05b9f700 1 -- 172.21.15.16:0/2827669966 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2a000c4960 con 0x7f2a000b5830 2023-10-16T13:44:23.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.598+0000 7f29e7fff700 1 -- 172.21.15.16:0/2827669966 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+4920 (secure 0 0 0) 0x7f2a000c4960 con 0x7f2a000b5830 2023-10-16T13:44:23.601 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:44:23.601 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":95,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8306,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2415675492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":2415675492},{"type":"v1","addr":"172.21.15.16:6837","nonce":2415675492}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":90},{"gid":8315,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/4031561298","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":4031561298},{"type":"v1","addr":"172.21.15.49:6834","nonce":4031561298}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":90},{"gid":8318,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/2853801536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":2853801536},{"type":"v1","addr":"172.21.15.49:6835","nonce":2853801536}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":90}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:28:19.523721+0000","modified":"2023-10-16T13:28:22.528826+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10296":{"gid":10296,"name":"c","rank":0,"incarnation":91,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6836/3244401013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":3244401013},{"type":"v1","addr":"172.21.15.16:6836","nonce":3244401013}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2023-10-16T13:44:23.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.600+0000 7f2a05b9f700 1 -- 172.21.15.16:0/2827669966 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f29ec06b020 msgr2=0x7f29ec06d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:44:23.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.600+0000 7f2a05b9f700 1 --2- 172.21.15.16:0/2827669966 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f29ec06b020 0x7f29ec06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:23.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.600+0000 7f2a05b9f700 1 -- 172.21.15.16:0/2827669966 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a000b5830 msgr2=0x7f2a000ce260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:23.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.600+0000 7f2a05b9f700 1 --2- 172.21.15.16:0/2827669966 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a000b5830 0x7f2a000ce260 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 crypto rx=0x7f29f0009c10 tx=0x7f29f0004750 comp rx=0 tx=0).stop 2023-10-16T13:44:23.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.600+0000 7f2a05b9f700 1 -- 172.21.15.16:0/2827669966 shutdown_connections 2023-10-16T13:44:23.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.600+0000 7f2a05b9f700 1 --2- 172.21.15.16:0/2827669966 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f29ec06b020 0x7f29ec06d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:23.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.600+0000 7f2a05b9f700 1 --2- 172.21.15.16:0/2827669966 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2a000cf880 0x7f2a000ca550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:23.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.600+0000 7f2a05b9f700 1 --2- 172.21.15.16:0/2827669966 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a000b5830 0x7f2a000ce260 unknown :-1 s=CLOSED pgs=2196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:23.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.600+0000 7f2a05b9f700 1 --2- 172.21.15.16:0/2827669966 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2a000cf4a0 0x7f2a000ca010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:23.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.600+0000 7f2a05b9f700 1 -- 172.21.15.16:0/2827669966 >> 172.21.15.16:0/2827669966 conn(0x7f2a000b1fd0 msgr2=0x7f2a0015a890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:23.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.601+0000 7f2a05b9f700 1 -- 172.21.15.16:0/2827669966 shutdown_connections 2023-10-16T13:44:23.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.601+0000 7f2a05b9f700 1 -- 172.21.15.16:0/2827669966 wait complete. 2023-10-16T13:44:23.606 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 95 2023-10-16T13:44:23.616 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:44:23.710 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:44:23.708+0000 7ffb56192200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:44:23.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.744+0000 7f76ddea8700 1 Processor -- start 2023-10-16T13:44:23.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.744+0000 7f76ddea8700 1 -- start start 2023-10-16T13:44:23.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.744+0000 7f76ddea8700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f76d80ce380 0x7f76d80d0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:23.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.745+0000 7f76ddea8700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76d80d1210 con 0x7f76d80cdfa0 2023-10-16T13:44:23.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.745+0000 7f76ddea8700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76d80cbc30 con 0x7f76d80b2b50 2023-10-16T13:44:23.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.745+0000 7f76ddea8700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76d80cbf10 con 0x7f76d80ce380 2023-10-16T13:44:23.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.745+0000 7f76d7fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f76d80ce380 0x7f76d80d0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:23.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.745+0000 7f76d77fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f76d80cdfa0 0x7f76d80b2440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:40330/0 (socket says 172.21.15.16:40330) 2023-10-16T13:44:23.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.745+0000 7f76d77fe700 1 -- 172.21.15.16:0/1229987887 learned_addr learned my addr 172.21.15.16:0/1229987887 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:44:23.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.745+0000 7f76d7fff700 1 -- 172.21.15.16:0/1229987887 >> v1:172.21.15.49:6789/0 conn(0x7f76d80b2b50 legacy=0x7f76d80d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:23.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.745+0000 7f76d7fff700 1 -- 172.21.15.16:0/1229987887 >> v1:172.21.15.16:6789/0 conn(0x7f76d80cdfa0 legacy=0x7f76d80b2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:23.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.745+0000 7f76d67fc700 1 -- 172.21.15.16:0/1229987887 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 149989818 0 0) 0x7f76d80d1210 con 0x7f76d80cdfa0 2023-10-16T13:44:23.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.745+0000 7f76d7fff700 1 -- 172.21.15.16:0/1229987887 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76d80cc1f0 con 0x7f76d80ce380 2023-10-16T13:44:23.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.745+0000 7f76d7fff700 1 --2- 172.21.15.16:0/1229987887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76d80ce380 0x7f76d80d0bb0 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 crypto rx=0x7f76d0009d90 tx=0x7f76d000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43631864302184a0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:23.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.745+0000 7f76d67fc700 1 -- 172.21.15.16:0/1229987887 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76d000e070 con 0x7f76d80ce380 2023-10-16T13:44:23.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.745+0000 7f76d67fc700 1 -- 172.21.15.16:0/1229987887 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76d000bbd0 con 0x7f76d80ce380 2023-10-16T13:44:23.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.745+0000 7f76d67fc700 1 -- 172.21.15.16:0/1229987887 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76d0012430 con 0x7f76d80ce380 2023-10-16T13:44:23.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.746+0000 7f76ddea8700 1 -- 172.21.15.16:0/1229987887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76d80ce380 msgr2=0x7f76d80d0bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:23.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.746+0000 7f76ddea8700 1 --2- 172.21.15.16:0/1229987887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76d80ce380 0x7f76d80d0bb0 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 crypto rx=0x7f76d0009d90 tx=0x7f76d000b330 comp rx=0 tx=0).stop 2023-10-16T13:44:23.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.746+0000 7f76ddea8700 1 -- 172.21.15.16:0/1229987887 shutdown_connections 2023-10-16T13:44:23.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.746+0000 7f76ddea8700 1 --2- 172.21.15.16:0/1229987887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76d80ce380 0x7f76d80d0bb0 unknown :-1 s=CLOSED pgs=2198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:23.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.746+0000 7f76ddea8700 1 -- 172.21.15.16:0/1229987887 >> 172.21.15.16:0/1229987887 conn(0x7f76d80d14b0 msgr2=0x7f76d80d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:23.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.746+0000 7f76ddea8700 1 -- 172.21.15.16:0/1229987887 shutdown_connections 2023-10-16T13:44:23.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.746+0000 7f76ddea8700 1 -- 172.21.15.16:0/1229987887 wait complete. 2023-10-16T13:44:23.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.747+0000 7f76ddea8700 1 Processor -- start 2023-10-16T13:44:23.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.748+0000 7f76ddea8700 1 -- start start 2023-10-16T13:44:23.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.748+0000 7f76ddea8700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f76d80b2b50 0x7f76d8155b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:23.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.748+0000 7f76ddea8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76d80cdfa0 0x7f76d80c6540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:23.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.748+0000 7f76ddea8700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f76d80ce380 0x7f76d80c6a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:23.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.748+0000 7f76ddea8700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76d809efc0 con 0x7f76d80cdfa0 2023-10-16T13:44:23.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.748+0000 7f76ddea8700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76d809ece0 con 0x7f76d80b2b50 2023-10-16T13:44:23.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.748+0000 7f76ddea8700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76d80c86b0 con 0x7f76d80ce380 2023-10-16T13:44:23.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.748+0000 7f76d6ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76d80cdfa0 0x7f76d80c6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:23.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.748+0000 7f76d6ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76d80cdfa0 0x7f76d80c6540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51064/0 (socket says 172.21.15.16:51064) 2023-10-16T13:44:23.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.748+0000 7f76d6ffd700 1 -- 172.21.15.16:0/4073814712 learned_addr learned my addr 172.21.15.16:0/4073814712 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:23.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.748+0000 7f76d77fe700 1 --2- 172.21.15.16:0/4073814712 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f76d80b2b50 0x7f76d8155b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:23.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.748+0000 7f76d7fff700 1 --2- 172.21.15.16:0/4073814712 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f76d80ce380 0x7f76d80c6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:23.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.749+0000 7f76d6ffd700 1 -- 172.21.15.16:0/4073814712 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f76d80ce380 msgr2=0x7f76d80c6a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:23.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.749+0000 7f76d6ffd700 1 --2- 172.21.15.16:0/4073814712 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f76d80ce380 0x7f76d80c6a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:23.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.749+0000 7f76d6ffd700 1 -- 172.21.15.16:0/4073814712 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f76d80b2b50 msgr2=0x7f76d8155b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:23.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.749+0000 7f76d6ffd700 1 --2- 172.21.15.16:0/4073814712 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f76d80b2b50 0x7f76d8155b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:23.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.749+0000 7f76d6ffd700 1 -- 172.21.15.16:0/4073814712 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76d0009a70 con 0x7f76d80cdfa0 2023-10-16T13:44:23.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.749+0000 7f76d6ffd700 1 --2- 172.21.15.16:0/4073814712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76d80cdfa0 0x7f76d80c6540 secure :-1 s=READY pgs=2199 cs=0 l=1 rev1=1 crypto rx=0x7f76cc00c4e0 tx=0x7f76cc00c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:23.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.749+0000 7f76bffff700 1 -- 172.21.15.16:0/4073814712 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76cc0052d0 con 0x7f76d80cdfa0 2023-10-16T13:44:23.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.749+0000 7f76bffff700 1 -- 172.21.15.16:0/4073814712 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76cc005430 con 0x7f76d80cdfa0 2023-10-16T13:44:23.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.749+0000 7f76ddea8700 1 -- 172.21.15.16:0/4073814712 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76d80c8990 con 0x7f76d80cdfa0 2023-10-16T13:44:23.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.749+0000 7f76bffff700 1 -- 172.21.15.16:0/4073814712 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76cc0038c0 con 0x7f76d80cdfa0 2023-10-16T13:44:23.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.749+0000 7f76ddea8700 1 -- 172.21.15.16:0/4073814712 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76d8156060 con 0x7f76d80cdfa0 2023-10-16T13:44:23.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.750+0000 7f76ddea8700 1 -- 172.21.15.16:0/4073814712 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76d80c8eb0 con 0x7f76d80cdfa0 2023-10-16T13:44:23.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.751+0000 7f76bffff700 1 -- 172.21.15.16:0/4073814712 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 122) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f76cc003a20 con 0x7f76d80cdfa0 2023-10-16T13:44:23.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.751+0000 7f76bffff700 1 --2- 172.21.15.16:0/4073814712 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f76c406b020 0x7f76c406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:23.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.751+0000 7f76d77fe700 1 -- 172.21.15.16:0/4073814712 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f76c406b020 msgr2=0x7f76c406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/371380443 2023-10-16T13:44:23.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.751+0000 7f76bffff700 1 -- 172.21.15.16:0/4073814712 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(194..194 src has 1..194) v4 ==== 6889+0+0 (secure 0 0 0) 0x7f76cc0891b0 con 0x7f76d80cdfa0 2023-10-16T13:44:23.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.751+0000 7f76d77fe700 1 --2- 172.21.15.16:0/4073814712 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f76c406b020 0x7f76c406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:44:23.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.755+0000 7f76bffff700 1 -- 172.21.15.16:0/4073814712 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f76cc053d20 con 0x7f76d80cdfa0 2023-10-16T13:44:23.790 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:44:23.787+0000 7fe4965ec200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:44:23.794 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:44:23.792+0000 7ffb56192200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T13:44:23.873 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:44:23.871+0000 7fe4965ec200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T13:44:23.945 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:44:23.944+0000 7ffb56192200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:44:23.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.952+0000 7f76d77fe700 1 -- 172.21.15.16:0/4073814712 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f76c406b020 msgr2=0x7f76c406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/371380443 2023-10-16T13:44:23.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.952+0000 7f76d77fe700 1 --2- 172.21.15.16:0/4073814712 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f76c406b020 0x7f76c406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:44:23.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.962+0000 7f76ddea8700 1 -- 172.21.15.16:0/4073814712 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f76d80c8eb0 con 0x7f76d80cdfa0 2023-10-16T13:44:23.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.963+0000 7f76bffff700 1 -- 172.21.15.16:0/4073814712 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v194) v1 ==== 74+0+17905 (secure 0 0 0) 0x7f76d80c8eb0 con 0x7f76d80cdfa0 2023-10-16T13:44:23.966 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:44:23.966 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":194,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:44:21.720034+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2023-10-16T13:28:17.549815+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"189","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2023-10-16T13:28:18.493327+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"194","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":33,"snap_epoch":194,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6833/3952564978":"2023-10-17T13:39:22.557973+0000","172.21.15.16:6832/3952564978":"2023-10-17T13:39:22.557973+0000","172.21.15.16:0/2554840790":"2023-10-17T13:39:22.557973+0000","172.21.15.16:0/360056771":"2023-10-17T13:39:22.557973+0000","172.21.15.16:0/3520685377":"2023-10-17T13:39:22.557973+0000","172.21.15.16:0/1571009468":"2023-10-17T13:28:34.631949+0000","172.21.15.16:0/678119374":"2023-10-17T13:28:34.631949+0000","172.21.15.16:0/3920247727":"2023-10-17T13:28:34.631949+0000","172.21.15.16:0/857902049":"2023-10-17T13:39:22.557973+0000","172.21.15.16:6833/4211329181":"2023-10-17T13:28:34.631949+0000","172.21.15.16:6832/4211329181":"2023-10-17T13:28:34.631949+0000","172.21.15.16:0/1548506671":"2023-10-17T13:39:22.557973+0000","172.21.15.16:0/2852193608":"2023-10-17T13:28:34.631949+0000","172.21.15.16:0/1641955134":"2023-10-17T13:39:22.557973+0000","172.21.15.16:0/471630266":"2023-10-17T13:28:34.631949+0000","172.21.15.16:0/3153082432":"2023-10-17T13:28:34.631949+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[{"pool":24,"snaps":[{"begin":17,"length":1},{"begin":33,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:44:23.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.965+0000 7f76ddea8700 1 -- 172.21.15.16:0/4073814712 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f76c406b020 msgr2=0x7f76c406d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:44:23.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.965+0000 7f76ddea8700 1 --2- 172.21.15.16:0/4073814712 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f76c406b020 0x7f76c406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:23.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.965+0000 7f76ddea8700 1 -- 172.21.15.16:0/4073814712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76d80cdfa0 msgr2=0x7f76d80c6540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:23.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.965+0000 7f76ddea8700 1 --2- 172.21.15.16:0/4073814712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76d80cdfa0 0x7f76d80c6540 secure :-1 s=READY pgs=2199 cs=0 l=1 rev1=1 crypto rx=0x7f76cc00c4e0 tx=0x7f76cc00c7f0 comp rx=0 tx=0).stop 2023-10-16T13:44:23.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.965+0000 7f76ddea8700 1 -- 172.21.15.16:0/4073814712 shutdown_connections 2023-10-16T13:44:23.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.965+0000 7f76ddea8700 1 --2- 172.21.15.16:0/4073814712 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f76c406b020 0x7f76c406d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:23.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.965+0000 7f76ddea8700 1 --2- 172.21.15.16:0/4073814712 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f76d80ce380 0x7f76d80c6a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:23.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.965+0000 7f76ddea8700 1 --2- 172.21.15.16:0/4073814712 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f76d80b2b50 0x7f76d8155b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:23.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.965+0000 7f76ddea8700 1 --2- 172.21.15.16:0/4073814712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76d80cdfa0 0x7f76d80c6540 unknown :-1 s=CLOSED pgs=2199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:23.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.965+0000 7f76ddea8700 1 -- 172.21.15.16:0/4073814712 >> 172.21.15.16:0/4073814712 conn(0x7f76d80d14b0 msgr2=0x7f76d80d1990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:23.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.966+0000 7f76ddea8700 1 -- 172.21.15.16:0/4073814712 shutdown_connections 2023-10-16T13:44:23.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:23.966+0000 7f76ddea8700 1 -- 172.21.15.16:0/4073814712 wait complete. 2023-10-16T13:44:23.981 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 23, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T13:28:17.549815+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '189', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 24, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T13:28:18.493327+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '194', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 33, 'snap_epoch': 194, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T13:44:23.981 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T13:44:24.030 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:44:24.027+0000 7fe4965ec200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:44:24.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.108+0000 7f5dda24f700 1 Processor -- start 2023-10-16T13:44:24.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.108+0000 7f5dda24f700 1 -- start start 2023-10-16T13:44:24.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.110+0000 7f5dda24f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5dd40cb880 0x7f5dd40cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:24.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.110+0000 7f5dda24f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5dd406da60 0x7f5dd406def0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:24.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.110+0000 7f5dda24f700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dd40cef70 con 0x7f5dd415b070 2023-10-16T13:44:24.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.110+0000 7f5dda24f700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dd406e430 con 0x7f5dd406da60 2023-10-16T13:44:24.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.110+0000 7f5dda24f700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd40ce360 con 0x7f5dd40cb880 2023-10-16T13:44:24.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.110+0000 7f5dd3fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5dd406da60 0x7f5dd406def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:24.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.110+0000 7f5dd3fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5dd406da60 0x7f5dd406def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51090/0 (socket says 172.21.15.16:51090) 2023-10-16T13:44:24.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.110+0000 7f5dd3fff700 1 -- 172.21.15.16:0/3565495917 learned_addr learned my addr 172.21.15.16:0/3565495917 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:24.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.110+0000 7f5dd2ffd700 1 --2- 172.21.15.16:0/3565495917 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5dd40cb880 0x7f5dd40cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:24.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.110+0000 7f5dd3fff700 1 -- 172.21.15.16:0/3565495917 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5dd40cb880 msgr2=0x7f5dd40cbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:24.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.110+0000 7f5dd3fff700 1 --2- 172.21.15.16:0/3565495917 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5dd40cb880 0x7f5dd40cbd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:24.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.110+0000 7f5dd3fff700 1 -- 172.21.15.16:0/3565495917 >> v1:172.21.15.49:6789/0 conn(0x7f5dd415b070 legacy=0x7f5dd415b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:24.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.110+0000 7f5dd3fff700 1 -- 172.21.15.16:0/3565495917 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dd40ce640 con 0x7f5dd406da60 2023-10-16T13:44:24.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.110+0000 7f5dd3fff700 1 --2- 172.21.15.16:0/3565495917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5dd406da60 0x7f5dd406def0 secure :-1 s=READY pgs=2200 cs=0 l=1 rev1=1 crypto rx=0x7f5dcc00d150 tx=0x7f5dcc00d460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d144483c6edc6d3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:24.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.110+0000 7f5dd27fc700 1 -- 172.21.15.16:0/3565495917 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dcc010070 con 0x7f5dd406da60 2023-10-16T13:44:24.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.110+0000 7f5dd27fc700 1 -- 172.21.15.16:0/3565495917 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dcc00dd10 con 0x7f5dd406da60 2023-10-16T13:44:24.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.110+0000 7f5dd27fc700 1 -- 172.21.15.16:0/3565495917 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dcc00b310 con 0x7f5dd406da60 2023-10-16T13:44:24.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.111+0000 7f5dda24f700 1 -- 172.21.15.16:0/3565495917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5dd406da60 msgr2=0x7f5dd406def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:24.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.111+0000 7f5dda24f700 1 --2- 172.21.15.16:0/3565495917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5dd406da60 0x7f5dd406def0 secure :-1 s=READY pgs=2200 cs=0 l=1 rev1=1 crypto rx=0x7f5dcc00d150 tx=0x7f5dcc00d460 comp rx=0 tx=0).stop 2023-10-16T13:44:24.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.111+0000 7f5dda24f700 1 -- 172.21.15.16:0/3565495917 shutdown_connections 2023-10-16T13:44:24.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.111+0000 7f5dda24f700 1 --2- 172.21.15.16:0/3565495917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5dd406da60 0x7f5dd406def0 unknown :-1 s=CLOSED pgs=2200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:24.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.111+0000 7f5dda24f700 1 --2- 172.21.15.16:0/3565495917 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5dd40cb880 0x7f5dd40cbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:24.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.111+0000 7f5dda24f700 1 -- 172.21.15.16:0/3565495917 >> 172.21.15.16:0/3565495917 conn(0x7f5dd40b1f40 msgr2=0x7f5dd40b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:24.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.111+0000 7f5dda24f700 1 -- 172.21.15.16:0/3565495917 shutdown_connections 2023-10-16T13:44:24.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.111+0000 7f5dda24f700 1 -- 172.21.15.16:0/3565495917 wait complete. 2023-10-16T13:44:24.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.112+0000 7f5dda24f700 1 Processor -- start 2023-10-16T13:44:24.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.113+0000 7f5dda24f700 1 -- start start 2023-10-16T13:44:24.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.113+0000 7f5dda24f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5dd406da60 0x7f5dd415be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:24.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.113+0000 7f5dda24f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5dd40cb880 0x7f5dd41f2130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:24.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.113+0000 7f5dda24f700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5dd415b070 0x7f5dd41f2670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:24.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.113+0000 7f5dda24f700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd4155c90 con 0x7f5dd406da60 2023-10-16T13:44:24.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.113+0000 7f5dda24f700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd41556a0 con 0x7f5dd415b070 2023-10-16T13:44:24.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.113+0000 7f5dda24f700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd415c920 con 0x7f5dd40cb880 2023-10-16T13:44:24.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.113+0000 7f5dd37fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5dd406da60 0x7f5dd415be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:24.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.114+0000 7f5dd37fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5dd406da60 0x7f5dd415be60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51092/0 (socket says 172.21.15.16:51092) 2023-10-16T13:44:24.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.114+0000 7f5dd37fe700 1 -- 172.21.15.16:0/227971054 learned_addr learned my addr 172.21.15.16:0/227971054 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:24.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.114+0000 7f5dd3fff700 1 --2- 172.21.15.16:0/227971054 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5dd415b070 0x7f5dd41f2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:24.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.114+0000 7f5dd2ffd700 1 --2- 172.21.15.16:0/227971054 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5dd40cb880 0x7f5dd41f2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:24.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.114+0000 7f5dd37fe700 1 -- 172.21.15.16:0/227971054 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5dd40cb880 msgr2=0x7f5dd41f2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:24.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.114+0000 7f5dd37fe700 1 --2- 172.21.15.16:0/227971054 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5dd40cb880 0x7f5dd41f2130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:24.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.114+0000 7f5dd37fe700 1 -- 172.21.15.16:0/227971054 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5dd415b070 msgr2=0x7f5dd41f2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:24.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.114+0000 7f5dd37fe700 1 --2- 172.21.15.16:0/227971054 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5dd415b070 0x7f5dd41f2670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:24.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.114+0000 7f5dd37fe700 1 -- 172.21.15.16:0/227971054 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dcc00bd40 con 0x7f5dd406da60 2023-10-16T13:44:24.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.114+0000 7f5dd37fe700 1 --2- 172.21.15.16:0/227971054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5dd406da60 0x7f5dd415be60 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 crypto rx=0x7f5dc400c450 tx=0x7f5dc400c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:24.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.114+0000 7f5dbbfff700 1 -- 172.21.15.16:0/227971054 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dc40052d0 con 0x7f5dd406da60 2023-10-16T13:44:24.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.114+0000 7f5dbbfff700 1 -- 172.21.15.16:0/227971054 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dc4005430 con 0x7f5dd406da60 2023-10-16T13:44:24.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.114+0000 7f5dda24f700 1 -- 172.21.15.16:0/227971054 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dd415cc00 con 0x7f5dd406da60 2023-10-16T13:44:24.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.114+0000 7f5dbbfff700 1 -- 172.21.15.16:0/227971054 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dc40038c0 con 0x7f5dd406da60 2023-10-16T13:44:24.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.114+0000 7f5dda24f700 1 -- 172.21.15.16:0/227971054 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dd40b5f40 con 0x7f5dd406da60 2023-10-16T13:44:24.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.115+0000 7f5dda24f700 1 -- 172.21.15.16:0/227971054 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dd415ce20 con 0x7f5dd406da60 2023-10-16T13:44:24.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.116+0000 7f5dbbfff700 1 -- 172.21.15.16:0/227971054 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 122) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f5dc4003a20 con 0x7f5dd406da60 2023-10-16T13:44:24.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.116+0000 7f5dbbfff700 1 --2- 172.21.15.16:0/227971054 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f5dc006b020 0x7f5dc006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:24.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.116+0000 7f5dd2ffd700 1 -- 172.21.15.16:0/227971054 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f5dc006b020 msgr2=0x7f5dc006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/371380443 2023-10-16T13:44:24.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.116+0000 7f5dbbfff700 1 -- 172.21.15.16:0/227971054 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(194..194 src has 1..194) v4 ==== 6889+0+0 (secure 0 0 0) 0x7f5dc4088f90 con 0x7f5dd406da60 2023-10-16T13:44:24.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.117+0000 7f5dd2ffd700 1 --2- 172.21.15.16:0/227971054 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f5dc006b020 0x7f5dc006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:44:24.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.121+0000 7f5dbbfff700 1 -- 172.21.15.16:0/227971054 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5dc40bbfe0 con 0x7f5dd406da60 2023-10-16T13:44:24.185 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:44:24.184+0000 7ffb56192200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T13:44:24.262 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:44:24.261+0000 7ffb56192200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T13:44:24.279 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:44:24.276+0000 7fe4965ec200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T13:44:24.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.318+0000 7f5dd2ffd700 1 -- 172.21.15.16:0/227971054 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f5dc006b020 msgr2=0x7f5dc006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/371380443 2023-10-16T13:44:24.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.318+0000 7f5dd2ffd700 1 --2- 172.21.15.16:0/227971054 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f5dc006b020 0x7f5dc006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:44:24.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.324+0000 7f5dda24f700 1 -- 172.21.15.16:0/227971054 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5dd404ebb0 con 0x7f5dd406da60 2023-10-16T13:44:24.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.324+0000 7f5dbbfff700 1 -- 172.21.15.16:0/227971054 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v194) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5dc4053a10 con 0x7f5dd406da60 2023-10-16T13:44:24.327 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T13:44:24.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.326+0000 7f5dda24f700 1 -- 172.21.15.16:0/227971054 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f5dc006b020 msgr2=0x7f5dc006d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:44:24.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.326+0000 7f5dda24f700 1 --2- 172.21.15.16:0/227971054 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f5dc006b020 0x7f5dc006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:24.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.326+0000 7f5dda24f700 1 -- 172.21.15.16:0/227971054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5dd406da60 msgr2=0x7f5dd415be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:24.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.326+0000 7f5dda24f700 1 --2- 172.21.15.16:0/227971054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5dd406da60 0x7f5dd415be60 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 crypto rx=0x7f5dc400c450 tx=0x7f5dc400c760 comp rx=0 tx=0).stop 2023-10-16T13:44:24.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.326+0000 7f5dda24f700 1 -- 172.21.15.16:0/227971054 shutdown_connections 2023-10-16T13:44:24.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.326+0000 7f5dda24f700 1 --2- 172.21.15.16:0/227971054 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5dd415b070 0x7f5dd41f2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:24.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.326+0000 7f5dda24f700 1 --2- 172.21.15.16:0/227971054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5dd406da60 0x7f5dd415be60 unknown :-1 s=CLOSED pgs=2201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:24.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.326+0000 7f5dda24f700 1 --2- 172.21.15.16:0/227971054 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f5dc006b020 0x7f5dc006d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:24.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.326+0000 7f5dda24f700 1 --2- 172.21.15.16:0/227971054 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5dd40cb880 0x7f5dd41f2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:24.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.326+0000 7f5dda24f700 1 -- 172.21.15.16:0/227971054 >> 172.21.15.16:0/227971054 conn(0x7f5dd40b1f40 msgr2=0x7f5dd40cca40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:24.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.327+0000 7f5dda24f700 1 -- 172.21.15.16:0/227971054 shutdown_connections 2023-10-16T13:44:24.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.327+0000 7f5dda24f700 1 -- 172.21.15.16:0/227971054 wait complete. 2023-10-16T13:44:24.342 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T13:44:24.360 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:44:24.358+0000 7fe4965ec200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T13:44:24.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.472+0000 7f1276f6f700 1 Processor -- start 2023-10-16T13:44:24.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.472+0000 7f1276f6f700 1 -- start start 2023-10-16T13:44:24.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.474+0000 7f1276f6f700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f12700ce5d0 0x7f12700cc190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:24.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.474+0000 7f1276f6f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f127015f060 0x7f12700cc6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:24.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.474+0000 7f1276f6f700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12700b2440 con 0x7f127015ec80 2023-10-16T13:44:24.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.474+0000 7f1276f6f700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12700b2c50 con 0x7f12700ce5d0 2023-10-16T13:44:24.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.474+0000 7f1276f6f700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12700d00e0 con 0x7f127015f060 2023-10-16T13:44:24.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.474+0000 7f127550c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f127015f060 0x7f12700cc6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:24.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.474+0000 7f126ffff700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f12700ce5d0 0x7f12700cc190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:24.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.474+0000 7f1274d0b700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7f127015ec80 0x7f12700cdec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:47558/0 (socket says 172.21.15.16:47558) 2023-10-16T13:44:24.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.474+0000 7f1274d0b700 1 -- 172.21.15.16:0/3702059771 learned_addr learned my addr 172.21.15.16:0/3702059771 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:44:24.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.474+0000 7f126f7fe700 1 -- 172.21.15.16:0/3702059771 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1894934299 0 0) 0x7f12700b2440 con 0x7f127015ec80 2023-10-16T13:44:24.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.474+0000 7f126f7fe700 1 -- 172.21.15.16:0/3702059771 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12600035b0 con 0x7f127015ec80 2023-10-16T13:44:24.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.474+0000 7f126ffff700 1 -- 172.21.15.16:0/3702059771 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f127015f060 msgr2=0x7f12700cc6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:24.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.474+0000 7f126ffff700 1 --2- 172.21.15.16:0/3702059771 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f127015f060 0x7f12700cc6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:24.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.474+0000 7f126ffff700 1 -- 172.21.15.16:0/3702059771 >> v1:172.21.15.49:6789/0 conn(0x7f127015ec80 legacy=0x7f12700cdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:24.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.474+0000 7f126ffff700 1 -- 172.21.15.16:0/3702059771 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12700cef60 con 0x7f12700ce5d0 2023-10-16T13:44:24.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.474+0000 7f127550c700 1 --2- 172.21.15.16:0/3702059771 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f127015f060 0x7f12700cc6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:44:24.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.475+0000 7f126ffff700 1 --2- 172.21.15.16:0/3702059771 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f12700ce5d0 0x7f12700cc190 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f125c009840 tx=0x7f125c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a05f94bf44c3fe7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:24.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.475+0000 7f126f7fe700 1 -- 172.21.15.16:0/3702059771 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f125c011830 con 0x7f12700ce5d0 2023-10-16T13:44:24.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.475+0000 7f126f7fe700 1 -- 172.21.15.16:0/3702059771 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f125c011990 con 0x7f12700ce5d0 2023-10-16T13:44:24.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.475+0000 7f126f7fe700 1 -- 172.21.15.16:0/3702059771 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f125c011d20 con 0x7f12700ce5d0 2023-10-16T13:44:24.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.476+0000 7f1276f6f700 1 -- 172.21.15.16:0/3702059771 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f12700ce5d0 msgr2=0x7f12700cc190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:24.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.476+0000 7f1276f6f700 1 --2- 172.21.15.16:0/3702059771 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f12700ce5d0 0x7f12700cc190 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f125c009840 tx=0x7f125c009b50 comp rx=0 tx=0).stop 2023-10-16T13:44:24.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.476+0000 7f1276f6f700 1 -- 172.21.15.16:0/3702059771 shutdown_connections 2023-10-16T13:44:24.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.476+0000 7f1276f6f700 1 --2- 172.21.15.16:0/3702059771 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f127015f060 0x7f12700cc6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:24.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.476+0000 7f1276f6f700 1 --2- 172.21.15.16:0/3702059771 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f12700ce5d0 0x7f12700cc190 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:24.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.476+0000 7f1276f6f700 1 -- 172.21.15.16:0/3702059771 >> 172.21.15.16:0/3702059771 conn(0x7f12700d15f0 msgr2=0x7f12700d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:24.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.476+0000 7f1276f6f700 1 -- 172.21.15.16:0/3702059771 shutdown_connections 2023-10-16T13:44:24.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.476+0000 7f1276f6f700 1 -- 172.21.15.16:0/3702059771 wait complete. 2023-10-16T13:44:24.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.478+0000 7f1276f6f700 1 Processor -- start 2023-10-16T13:44:24.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.478+0000 7f1276f6f700 1 -- start start 2023-10-16T13:44:24.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.479+0000 7f1276f6f700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f12700ce5d0 0x7f1270152b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:24.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.479+0000 7f1276f6f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f127015ec80 0x7f12700ca240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:24.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.479+0000 7f1276f6f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f127015f060 0x7f127006b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:24.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.479+0000 7f1276f6f700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12700cad00 con 0x7f127015f060 2023-10-16T13:44:24.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.479+0000 7f1276f6f700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12700cafe0 con 0x7f12700ce5d0 2023-10-16T13:44:24.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.479+0000 7f1276f6f700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12700cb2c0 con 0x7f127015ec80 2023-10-16T13:44:24.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.479+0000 7f1274d0b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f12700ce5d0 0x7f1270152b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:24.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.479+0000 7f127550c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f127015f060 0x7f127006b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:24.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.479+0000 7f126ffff700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f127015ec80 0x7f12700ca240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:24.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.479+0000 7f1274d0b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f12700ce5d0 0x7f1270152b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:50170/0 (socket says 172.21.15.16:50170) 2023-10-16T13:44:24.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.479+0000 7f126ffff700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f127015ec80 0x7f12700ca240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:39368/0 (socket says 172.21.15.16:39368) 2023-10-16T13:44:24.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.479+0000 7f1274d0b700 1 -- 172.21.15.16:0/3833064956 learned_addr learned my addr 172.21.15.16:0/3833064956 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:24.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.479+0000 7f127550c700 1 -- 172.21.15.16:0/3833064956 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f127015ec80 msgr2=0x7f12700ca240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:24.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.479+0000 7f127550c700 1 --2- 172.21.15.16:0/3833064956 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f127015ec80 0x7f12700ca240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:24.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.479+0000 7f127550c700 1 -- 172.21.15.16:0/3833064956 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f12700ce5d0 msgr2=0x7f1270152b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:24.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.479+0000 7f127550c700 1 --2- 172.21.15.16:0/3833064956 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f12700ce5d0 0x7f1270152b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:24.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.479+0000 7f127550c700 1 -- 172.21.15.16:0/3833064956 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1268009a70 con 0x7f127015f060 2023-10-16T13:44:24.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.479+0000 7f126ffff700 1 --2- 172.21.15.16:0/3833064956 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f127015ec80 0x7f12700ca240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:44:24.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.480+0000 7f127550c700 1 --2- 172.21.15.16:0/3833064956 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f127015f060 0x7f127006b8f0 secure :-1 s=READY pgs=2202 cs=0 l=1 rev1=1 crypto rx=0x7f12700cbf20 tx=0x7f126800ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:24.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.480+0000 7f126d7fa700 1 -- 172.21.15.16:0/3833064956 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f126800cc20 con 0x7f127015f060 2023-10-16T13:44:24.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.480+0000 7f1276f6f700 1 -- 172.21.15.16:0/3833064956 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f125c0094a0 con 0x7f127015f060 2023-10-16T13:44:24.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.480+0000 7f126d7fa700 1 -- 172.21.15.16:0/3833064956 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f126800f040 con 0x7f127015f060 2023-10-16T13:44:24.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.480+0000 7f126d7fa700 1 -- 172.21.15.16:0/3833064956 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1268009e70 con 0x7f127015f060 2023-10-16T13:44:24.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.480+0000 7f1276f6f700 1 -- 172.21.15.16:0/3833064956 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12700cb5a0 con 0x7f127015f060 2023-10-16T13:44:24.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.481+0000 7f1276f6f700 1 -- 172.21.15.16:0/3833064956 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12700c53b0 con 0x7f127015f060 2023-10-16T13:44:24.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.482+0000 7f126d7fa700 1 -- 172.21.15.16:0/3833064956 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 122) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f1268014480 con 0x7f127015f060 2023-10-16T13:44:24.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.482+0000 7f126d7fa700 1 --2- 172.21.15.16:0/3833064956 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f125406b020 0x7f125406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:24.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.482+0000 7f1274d0b700 1 -- 172.21.15.16:0/3833064956 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f125406b020 msgr2=0x7f125406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/371380443 2023-10-16T13:44:24.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.482+0000 7f126d7fa700 1 -- 172.21.15.16:0/3833064956 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(194..194 src has 1..194) v4 ==== 6889+0+0 (secure 0 0 0) 0x7f126808a7c0 con 0x7f127015f060 2023-10-16T13:44:24.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.482+0000 7f1274d0b700 1 --2- 172.21.15.16:0/3833064956 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f125406b020 0x7f125406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:44:24.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.487+0000 7f126d7fa700 1 -- 172.21.15.16:0/3833064956 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f12680551c0 con 0x7f127015f060 2023-10-16T13:44:24.541 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:44:24.539+0000 7ffb56192200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T13:44:24.650 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:44:24.648+0000 7fe4965ec200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T13:44:24.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.683+0000 7f1274d0b700 1 -- 172.21.15.16:0/3833064956 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f125406b020 msgr2=0x7f125406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/371380443 2023-10-16T13:44:24.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.683+0000 7f1274d0b700 1 --2- 172.21.15.16:0/3833064956 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f125406b020 0x7f125406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:44:24.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.704+0000 7f1276f6f700 1 -- 172.21.15.16:0/3833064956 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f12700c67c0 con 0x7f127015f060 2023-10-16T13:44:24.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.704+0000 7f126d7fa700 1 -- 172.21.15.16:0/3833064956 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v194) v1 ==== 104+0+11 (secure 0 0 0) 0x7f12700c67c0 con 0x7f127015f060 2023-10-16T13:44:24.707 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-10-16T13:44:24.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.706+0000 7f1276f6f700 1 -- 172.21.15.16:0/3833064956 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f125406b020 msgr2=0x7f125406d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:44:24.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.706+0000 7f1276f6f700 1 --2- 172.21.15.16:0/3833064956 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f125406b020 0x7f125406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:24.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.706+0000 7f1276f6f700 1 -- 172.21.15.16:0/3833064956 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f127015f060 msgr2=0x7f127006b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:24.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.706+0000 7f1276f6f700 1 --2- 172.21.15.16:0/3833064956 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f127015f060 0x7f127006b8f0 secure :-1 s=READY pgs=2202 cs=0 l=1 rev1=1 crypto rx=0x7f12700cbf20 tx=0x7f126800ef90 comp rx=0 tx=0).stop 2023-10-16T13:44:24.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.706+0000 7f1276f6f700 1 -- 172.21.15.16:0/3833064956 shutdown_connections 2023-10-16T13:44:24.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.706+0000 7f1276f6f700 1 --2- 172.21.15.16:0/3833064956 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f127015f060 0x7f127006b8f0 unknown :-1 s=CLOSED pgs=2202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:24.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.707+0000 7f1276f6f700 1 --2- 172.21.15.16:0/3833064956 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f12700ce5d0 0x7f1270152b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:24.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.707+0000 7f1276f6f700 1 --2- 172.21.15.16:0/3833064956 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f125406b020 0x7f125406d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:24.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.707+0000 7f1276f6f700 1 --2- 172.21.15.16:0/3833064956 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f127015ec80 0x7f12700ca240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:24.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.707+0000 7f1276f6f700 1 -- 172.21.15.16:0/3833064956 >> 172.21.15.16:0/3833064956 conn(0x7f12700d15f0 msgr2=0x7f1270070340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:24.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.707+0000 7f1276f6f700 1 -- 172.21.15.16:0/3833064956 shutdown_connections 2023-10-16T13:44:24.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.707+0000 7f1276f6f700 1 -- 172.21.15.16:0/3833064956 wait complete. 2023-10-16T13:44:24.722 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T13:44:24.775 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:44:24.773+0000 7ffb56192200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T13:44:24.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.889+0000 7ff430a34700 1 Processor -- start 2023-10-16T13:44:24.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.889+0000 7ff430a34700 1 -- start start 2023-10-16T13:44:24.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.890+0000 7ff430a34700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7ff42c06e570 0x7ff42c15e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:24.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.890+0000 7ff430a34700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff42c15b6a0 con 0x7ff42c06a7e0 2023-10-16T13:44:24.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.890+0000 7ff430a34700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff42c161b20 con 0x7ff42c15b2c0 2023-10-16T13:44:24.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.890+0000 7ff430a34700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff42c161e00 con 0x7ff42c06e570 2023-10-16T13:44:24.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.891+0000 7ff42ad9d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff42c06a7e0 0x7ff42c071a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:40396/0 (socket says 172.21.15.16:40396) 2023-10-16T13:44:24.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.891+0000 7ff42ad9d700 1 -- 172.21.15.16:0/1688824958 learned_addr learned my addr 172.21.15.16:0/1688824958 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:44:24.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.891+0000 7ff429d9b700 1 --2- 172.21.15.16:0/1688824958 >> v2:172.21.15.49:3300/0 conn(0x7ff42c06e570 0x7ff42c15e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:24.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.891+0000 7ff42959a700 1 -- 172.21.15.16:0/1688824958 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4057809538 0 0) 0x7ff42c15b6a0 con 0x7ff42c06a7e0 2023-10-16T13:44:24.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.891+0000 7ff42959a700 1 -- 172.21.15.16:0/1688824958 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4140035b0 con 0x7ff42c06a7e0 2023-10-16T13:44:24.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.891+0000 7ff42959a700 1 -- 172.21.15.16:0/1688824958 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1644266982 0 0) 0x7ff42c161b20 con 0x7ff42c15b2c0 2023-10-16T13:44:24.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.891+0000 7ff42959a700 1 -- 172.21.15.16:0/1688824958 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff42c15b6a0 con 0x7ff42c15b2c0 2023-10-16T13:44:24.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.891+0000 7ff429d9b700 1 -- 172.21.15.16:0/1688824958 >> v1:172.21.15.49:6789/0 conn(0x7ff42c15b2c0 legacy=0x7ff42c161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:24.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.891+0000 7ff429d9b700 1 -- 172.21.15.16:0/1688824958 >> v1:172.21.15.16:6789/0 conn(0x7ff42c06a7e0 legacy=0x7ff42c071a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:24.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.891+0000 7ff429d9b700 1 -- 172.21.15.16:0/1688824958 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff42c1620e0 con 0x7ff42c06e570 2023-10-16T13:44:24.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.891+0000 7ff429d9b700 1 --2- 172.21.15.16:0/1688824958 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff42c06e570 0x7ff42c15e5a0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7ff420009840 tx=0x7ff420009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b1207641cfe8b7f5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:24.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.892+0000 7ff42959a700 1 -- 172.21.15.16:0/1688824958 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff420011830 con 0x7ff42c06e570 2023-10-16T13:44:24.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.892+0000 7ff42959a700 1 -- 172.21.15.16:0/1688824958 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff420011990 con 0x7ff42c06e570 2023-10-16T13:44:24.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.892+0000 7ff42959a700 1 -- 172.21.15.16:0/1688824958 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff420011d20 con 0x7ff42c06e570 2023-10-16T13:44:24.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.892+0000 7ff430a34700 1 -- 172.21.15.16:0/1688824958 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff42c06e570 msgr2=0x7ff42c15e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:24.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.892+0000 7ff430a34700 1 --2- 172.21.15.16:0/1688824958 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff42c06e570 0x7ff42c15e5a0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7ff420009840 tx=0x7ff420009b50 comp rx=0 tx=0).stop 2023-10-16T13:44:24.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.892+0000 7ff430a34700 1 -- 172.21.15.16:0/1688824958 shutdown_connections 2023-10-16T13:44:24.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.892+0000 7ff430a34700 1 --2- 172.21.15.16:0/1688824958 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff42c06e570 0x7ff42c15e5a0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:24.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.892+0000 7ff430a34700 1 -- 172.21.15.16:0/1688824958 >> 172.21.15.16:0/1688824958 conn(0x7ff42c0b1f40 msgr2=0x7ff42c0b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:24.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.892+0000 7ff430a34700 1 -- 172.21.15.16:0/1688824958 shutdown_connections 2023-10-16T13:44:24.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.892+0000 7ff430a34700 1 -- 172.21.15.16:0/1688824958 wait complete. 2023-10-16T13:44:24.900 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:44:24.893+0000 7fe4965ec200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T13:44:24.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.894+0000 7ff430a34700 1 Processor -- start 2023-10-16T13:44:24.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.894+0000 7ff430a34700 1 -- start start 2023-10-16T13:44:24.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.894+0000 7ff430a34700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff42c06a7e0 0x7ff42c14d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:24.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.894+0000 7ff430a34700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff42c06e570 0x7ff42c0c74c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:24.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.894+0000 7ff430a34700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff42c15b2c0 0x7ff42c159300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:24.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.894+0000 7ff430a34700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff42c1f4700 con 0x7ff42c06e570 2023-10-16T13:44:24.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.894+0000 7ff430a34700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff42c1f49e0 con 0x7ff42c06a7e0 2023-10-16T13:44:24.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.894+0000 7ff430a34700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff42c1f4cc0 con 0x7ff42c15b2c0 2023-10-16T13:44:24.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.895+0000 7ff429d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff42c06e570 0x7ff42c0c74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:24.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.895+0000 7ff429d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff42c06e570 0x7ff42c0c74c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51126/0 (socket says 172.21.15.16:51126) 2023-10-16T13:44:24.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.895+0000 7ff429d9b700 1 -- 172.21.15.16:0/2119505216 learned_addr learned my addr 172.21.15.16:0/2119505216 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:24.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.895+0000 7ff42a59c700 1 --2- 172.21.15.16:0/2119505216 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff42c06a7e0 0x7ff42c14d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:24.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.895+0000 7ff42ad9d700 1 --2- 172.21.15.16:0/2119505216 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff42c15b2c0 0x7ff42c159300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:24.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.895+0000 7ff429d9b700 1 -- 172.21.15.16:0/2119505216 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff42c15b2c0 msgr2=0x7ff42c159300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:24.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.895+0000 7ff429d9b700 1 --2- 172.21.15.16:0/2119505216 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff42c15b2c0 0x7ff42c159300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:24.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.895+0000 7ff429d9b700 1 -- 172.21.15.16:0/2119505216 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff42c06a7e0 msgr2=0x7ff42c14d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:24.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.895+0000 7ff429d9b700 1 --2- 172.21.15.16:0/2119505216 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff42c06a7e0 0x7ff42c14d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:24.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.895+0000 7ff429d9b700 1 -- 172.21.15.16:0/2119505216 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4200094a0 con 0x7ff42c06e570 2023-10-16T13:44:24.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.895+0000 7ff429d9b700 1 --2- 172.21.15.16:0/2119505216 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff42c06e570 0x7ff42c0c74c0 secure :-1 s=READY pgs=2204 cs=0 l=1 rev1=1 crypto rx=0x7ff4200090b0 tx=0x7ff420004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:24.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.895+0000 7ff41affd700 1 -- 172.21.15.16:0/2119505216 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff42001a6e0 con 0x7ff42c06e570 2023-10-16T13:44:24.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.895+0000 7ff41affd700 1 -- 172.21.15.16:0/2119505216 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff42001a840 con 0x7ff42c06e570 2023-10-16T13:44:24.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.895+0000 7ff430a34700 1 -- 172.21.15.16:0/2119505216 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff42c1f4fa0 con 0x7ff42c06e570 2023-10-16T13:44:24.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.895+0000 7ff41affd700 1 -- 172.21.15.16:0/2119505216 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff42001abb0 con 0x7ff42c06e570 2023-10-16T13:44:24.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.895+0000 7ff430a34700 1 -- 172.21.15.16:0/2119505216 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff42c0b2f10 con 0x7ff42c06e570 2023-10-16T13:44:24.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.896+0000 7ff430a34700 1 -- 172.21.15.16:0/2119505216 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff42c1f54c0 con 0x7ff42c06e570 2023-10-16T13:44:24.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.897+0000 7ff41affd700 1 -- 172.21.15.16:0/2119505216 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 122) v1 ==== 89343+0+0 (secure 0 0 0) 0x7ff420022070 con 0x7ff42c06e570 2023-10-16T13:44:24.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.897+0000 7ff41affd700 1 --2- 172.21.15.16:0/2119505216 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7ff40c06b020 0x7ff40c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:24.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.897+0000 7ff42a59c700 1 -- 172.21.15.16:0/2119505216 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7ff40c06b020 msgr2=0x7ff40c06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/371380443 2023-10-16T13:44:24.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.897+0000 7ff41affd700 1 -- 172.21.15.16:0/2119505216 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(194..194 src has 1..194) v4 ==== 6889+0+0 (secure 0 0 0) 0x7ff42008acb0 con 0x7ff42c06e570 2023-10-16T13:44:24.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.897+0000 7ff42a59c700 1 --2- 172.21.15.16:0/2119505216 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7ff40c06b020 0x7ff40c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:44:24.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:24.902+0000 7ff41affd700 1 -- 172.21.15.16:0/2119505216 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff420055960 con 0x7ff42c06e570 2023-10-16T13:44:24.932 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:44:24.930+0000 7ffb56192200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T13:44:25.051 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:44:25.050+0000 7ffb56192200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T13:44:25.057 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:44:25.055+0000 7fe4965ec200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T13:44:25.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.098+0000 7ff42a59c700 1 -- 172.21.15.16:0/2119505216 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7ff40c06b020 msgr2=0x7ff40c06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/371380443 2023-10-16T13:44:25.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.098+0000 7ff42a59c700 1 --2- 172.21.15.16:0/2119505216 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7ff40c06b020 0x7ff40c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:44:25.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.107+0000 7ff430a34700 1 -- 172.21.15.16:0/2119505216 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff42c04ebb0 con 0x7ff42c06e570 2023-10-16T13:44:25.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.107+0000 7ff41affd700 1 -- 172.21.15.16:0/2119505216 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v194) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff420029020 con 0x7ff42c06e570 2023-10-16T13:44:25.110 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-10-16T13:44:25.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.109+0000 7ff430a34700 1 -- 172.21.15.16:0/2119505216 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7ff40c06b020 msgr2=0x7ff40c06d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:44:25.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.109+0000 7ff430a34700 1 --2- 172.21.15.16:0/2119505216 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7ff40c06b020 0x7ff40c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:25.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.109+0000 7ff430a34700 1 -- 172.21.15.16:0/2119505216 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff42c06e570 msgr2=0x7ff42c0c74c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:25.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.109+0000 7ff430a34700 1 --2- 172.21.15.16:0/2119505216 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff42c06e570 0x7ff42c0c74c0 secure :-1 s=READY pgs=2204 cs=0 l=1 rev1=1 crypto rx=0x7ff4200090b0 tx=0x7ff420004750 comp rx=0 tx=0).stop 2023-10-16T13:44:25.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.109+0000 7ff430a34700 1 -- 172.21.15.16:0/2119505216 shutdown_connections 2023-10-16T13:44:25.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.109+0000 7ff430a34700 1 --2- 172.21.15.16:0/2119505216 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7ff40c06b020 0x7ff40c06d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:25.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.109+0000 7ff430a34700 1 --2- 172.21.15.16:0/2119505216 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff42c15b2c0 0x7ff42c159300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:25.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.110+0000 7ff430a34700 1 --2- 172.21.15.16:0/2119505216 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff42c06a7e0 0x7ff42c14d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:25.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.110+0000 7ff430a34700 1 --2- 172.21.15.16:0/2119505216 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff42c06e570 0x7ff42c0c74c0 unknown :-1 s=CLOSED pgs=2204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:25.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.110+0000 7ff430a34700 1 -- 172.21.15.16:0/2119505216 >> 172.21.15.16:0/2119505216 conn(0x7ff42c0b1f40 msgr2=0x7ff42c0cc100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:25.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.110+0000 7ff430a34700 1 -- 172.21.15.16:0/2119505216 shutdown_connections 2023-10-16T13:44:25.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.110+0000 7ff430a34700 1 -- 172.21.15.16:0/2119505216 wait complete. 2023-10-16T13:44:25.124 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:44:25.185 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:44:25.182+0000 7fe4965ec200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T13:44:25.213 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:44:25.212+0000 7ffb56192200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T13:44:25.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.250+0000 7f5bb8512700 1 Processor -- start 2023-10-16T13:44:25.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.250+0000 7f5bb8512700 1 -- start start 2023-10-16T13:44:25.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.251+0000 7f5bb8512700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f5bb00ce4f0 0x7f5bb00d0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:25.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.251+0000 7f5bb8512700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5bb015ba10 0x7f5bb00cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:25.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.252+0000 7f5bb8512700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bb00b0970 con 0x7f5bb015b630 2023-10-16T13:44:25.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.252+0000 7f5bb8512700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bb00cd690 con 0x7f5bb00ce4f0 2023-10-16T13:44:25.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.252+0000 7f5bb8512700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb00cd970 con 0x7f5bb015ba10 2023-10-16T13:44:25.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.252+0000 7f5bb62ae700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f5bb015b630 0x7f5bb00cdde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:40408/0 (socket says 172.21.15.16:40408) 2023-10-16T13:44:25.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.252+0000 7f5bb62ae700 1 -- 172.21.15.16:0/3627427432 learned_addr learned my addr 172.21.15.16:0/3627427432 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:44:25.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.252+0000 7f5bb5aad700 1 --2- 172.21.15.16:0/3627427432 >> v2:172.21.15.49:3300/0 conn(0x7f5bb00ce4f0 0x7f5bb00d0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:25.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.252+0000 7f5bb6aaf700 1 --2- 172.21.15.16:0/3627427432 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5bb015ba10 0x7f5bb00cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:25.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.252+0000 7f5bb52ac700 1 -- 172.21.15.16:0/3627427432 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1461441474 0 0) 0x7f5bb00b0970 con 0x7f5bb015b630 2023-10-16T13:44:25.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.252+0000 7f5bb52ac700 1 -- 172.21.15.16:0/3627427432 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b9c0035b0 con 0x7f5bb015b630 2023-10-16T13:44:25.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.252+0000 7f5bb6aaf700 1 -- 172.21.15.16:0/3627427432 >> v2:172.21.15.49:3300/0 conn(0x7f5bb00ce4f0 msgr2=0x7f5bb00d0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:25.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.252+0000 7f5bb6aaf700 1 --2- 172.21.15.16:0/3627427432 >> v2:172.21.15.49:3300/0 conn(0x7f5bb00ce4f0 0x7f5bb00d0380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:25.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.252+0000 7f5bb6aaf700 1 -- 172.21.15.16:0/3627427432 >> v1:172.21.15.16:6789/0 conn(0x7f5bb015b630 legacy=0x7f5bb00cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:25.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.252+0000 7f5bb6aaf700 1 -- 172.21.15.16:0/3627427432 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bb00b0c30 con 0x7f5bb015ba10 2023-10-16T13:44:25.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.252+0000 7f5bb6aaf700 1 --2- 172.21.15.16:0/3627427432 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5bb015ba10 0x7f5bb00cf290 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f5ba8009d90 tx=0x7f5ba800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=21b520f7ee3ad3ea server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:25.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.253+0000 7f5bb52ac700 1 -- 172.21.15.16:0/3627427432 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ba800e070 con 0x7f5bb015ba10 2023-10-16T13:44:25.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.253+0000 7f5bb52ac700 1 -- 172.21.15.16:0/3627427432 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ba800bb20 con 0x7f5bb015ba10 2023-10-16T13:44:25.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.253+0000 7f5bb52ac700 1 -- 172.21.15.16:0/3627427432 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ba8012430 con 0x7f5bb015ba10 2023-10-16T13:44:25.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.253+0000 7f5bb8512700 1 -- 172.21.15.16:0/3627427432 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5bb015ba10 msgr2=0x7f5bb00cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:25.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.253+0000 7f5bb8512700 1 --2- 172.21.15.16:0/3627427432 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5bb015ba10 0x7f5bb00cf290 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f5ba8009d90 tx=0x7f5ba800b330 comp rx=0 tx=0).stop 2023-10-16T13:44:25.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.254+0000 7f5bb8512700 1 -- 172.21.15.16:0/3627427432 shutdown_connections 2023-10-16T13:44:25.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.254+0000 7f5bb8512700 1 --2- 172.21.15.16:0/3627427432 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5bb015ba10 0x7f5bb00cf290 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:25.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.254+0000 7f5bb8512700 1 --2- 172.21.15.16:0/3627427432 >> v2:172.21.15.49:3300/0 conn(0x7f5bb00ce4f0 0x7f5bb00d0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:25.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.254+0000 7f5bb8512700 1 -- 172.21.15.16:0/3627427432 >> 172.21.15.16:0/3627427432 conn(0x7f5bb00d14b0 msgr2=0x7f5bb00d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:25.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.254+0000 7f5bb8512700 1 -- 172.21.15.16:0/3627427432 shutdown_connections 2023-10-16T13:44:25.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.254+0000 7f5bb8512700 1 -- 172.21.15.16:0/3627427432 wait complete. 2023-10-16T13:44:25.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.256+0000 7f5bb8512700 1 Processor -- start 2023-10-16T13:44:25.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.256+0000 7f5bb8512700 1 -- start start 2023-10-16T13:44:25.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.257+0000 7f5bb8512700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5bb00ce4f0 0x7f5bb015d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:25.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.257+0000 7f5bb8512700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5bb015b630 0x7f5bb01f1780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:25.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.257+0000 7f5bb8512700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5bb015ba10 0x7f5bb00c8580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:25.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.257+0000 7f5bb8512700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb00c9b80 con 0x7f5bb00ce4f0 2023-10-16T13:44:25.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.257+0000 7f5bb8512700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb01f1cc0 con 0x7f5bb015b630 2023-10-16T13:44:25.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.257+0000 7f5bb8512700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb00c9dd0 con 0x7f5bb015ba10 2023-10-16T13:44:25.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.257+0000 7f5bb62ae700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5bb00ce4f0 0x7f5bb015d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:25.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.257+0000 7f5bb62ae700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5bb00ce4f0 0x7f5bb015d7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51140/0 (socket says 172.21.15.16:51140) 2023-10-16T13:44:25.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.257+0000 7f5bb62ae700 1 -- 172.21.15.16:0/2017364954 learned_addr learned my addr 172.21.15.16:0/2017364954 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:25.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.257+0000 7f5bb6aaf700 1 --2- 172.21.15.16:0/2017364954 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5bb015ba10 0x7f5bb00c8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:25.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.257+0000 7f5bb5aad700 1 --2- 172.21.15.16:0/2017364954 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5bb015b630 0x7f5bb01f1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:25.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.257+0000 7f5bb62ae700 1 -- 172.21.15.16:0/2017364954 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5bb015ba10 msgr2=0x7f5bb00c8580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:25.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.257+0000 7f5bb62ae700 1 --2- 172.21.15.16:0/2017364954 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5bb015ba10 0x7f5bb00c8580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:25.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.257+0000 7f5bb62ae700 1 -- 172.21.15.16:0/2017364954 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5bb015b630 msgr2=0x7f5bb01f1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:25.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.257+0000 7f5bb62ae700 1 --2- 172.21.15.16:0/2017364954 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5bb015b630 0x7f5bb01f1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:25.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.257+0000 7f5bb62ae700 1 -- 172.21.15.16:0/2017364954 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ba8009a70 con 0x7f5bb00ce4f0 2023-10-16T13:44:25.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.257+0000 7f5bb6aaf700 1 --2- 172.21.15.16:0/2017364954 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5bb015ba10 0x7f5bb00c8580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:44:25.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.258+0000 7f5bb62ae700 1 --2- 172.21.15.16:0/2017364954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5bb00ce4f0 0x7f5bb015d7d0 secure :-1 s=READY pgs=2206 cs=0 l=1 rev1=1 crypto rx=0x7f5bac00c730 tx=0x7f5bac00ca40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:25.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.258+0000 7f5ba6ffd700 1 -- 172.21.15.16:0/2017364954 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bac005d70 con 0x7f5bb00ce4f0 2023-10-16T13:44:25.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.258+0000 7f5ba6ffd700 1 -- 172.21.15.16:0/2017364954 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bac0070d0 con 0x7f5bb00ce4f0 2023-10-16T13:44:25.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.258+0000 7f5ba6ffd700 1 -- 172.21.15.16:0/2017364954 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bac007440 con 0x7f5bb00ce4f0 2023-10-16T13:44:25.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.258+0000 7f5bb8512700 1 -- 172.21.15.16:0/2017364954 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bb00ca0b0 con 0x7f5bb00ce4f0 2023-10-16T13:44:25.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.258+0000 7f5bb8512700 1 -- 172.21.15.16:0/2017364954 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bb00ca670 con 0x7f5bb00ce4f0 2023-10-16T13:44:25.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.259+0000 7f5bb8512700 1 -- 172.21.15.16:0/2017364954 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bb004ebb0 con 0x7f5bb00ce4f0 2023-10-16T13:44:25.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.259+0000 7f5ba6ffd700 1 -- 172.21.15.16:0/2017364954 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 122) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f5bac0075a0 con 0x7f5bb00ce4f0 2023-10-16T13:44:25.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.260+0000 7f5ba6ffd700 1 --2- 172.21.15.16:0/2017364954 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f5b9406b020 0x7f5b9406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:25.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.260+0000 7f5bb5aad700 1 -- 172.21.15.16:0/2017364954 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f5b9406b020 msgr2=0x7f5b9406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/371380443 2023-10-16T13:44:25.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.260+0000 7f5ba6ffd700 1 -- 172.21.15.16:0/2017364954 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(194..194 src has 1..194) v4 ==== 6889+0+0 (secure 0 0 0) 0x7f5bac088e70 con 0x7f5bb00ce4f0 2023-10-16T13:44:25.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.260+0000 7f5bb5aad700 1 --2- 172.21.15.16:0/2017364954 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f5b9406b020 0x7f5b9406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:44:25.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.264+0000 7f5ba6ffd700 1 -- 172.21.15.16:0/2017364954 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5bac053ab0 con 0x7f5bb00ce4f0 2023-10-16T13:44:25.357 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:44:25.354+0000 7fe4965ec200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T13:44:25.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.461+0000 7f5bb5aad700 1 -- 172.21.15.16:0/2017364954 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f5b9406b020 msgr2=0x7f5b9406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/371380443 2023-10-16T13:44:25.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.461+0000 7f5bb5aad700 1 --2- 172.21.15.16:0/2017364954 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f5b9406b020 0x7f5b9406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:44:25.486 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:44:25.484+0000 7ffb56192200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T13:44:25.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.517+0000 7f5bb8512700 1 -- 172.21.15.16:0/2017364954 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5bb006b5d0 con 0x7f5bb00ce4f0 2023-10-16T13:44:25.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.518+0000 7f5ba6ffd700 1 -- 172.21.15.16:0/2017364954 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+4920 (secure 0 0 0) 0x7f5bac017090 con 0x7f5bb00ce4f0 2023-10-16T13:44:25.520 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:44:25.521 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":95,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8306,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2415675492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":2415675492},{"type":"v1","addr":"172.21.15.16:6837","nonce":2415675492}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":90},{"gid":8315,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/4031561298","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":4031561298},{"type":"v1","addr":"172.21.15.49:6834","nonce":4031561298}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":90},{"gid":8318,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/2853801536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":2853801536},{"type":"v1","addr":"172.21.15.49:6835","nonce":2853801536}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":90}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:28:19.523721+0000","modified":"2023-10-16T13:28:22.528826+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10296":{"gid":10296,"name":"c","rank":0,"incarnation":91,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6836/3244401013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":3244401013},{"type":"v1","addr":"172.21.15.16:6836","nonce":3244401013}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2023-10-16T13:44:25.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.520+0000 7f5bb8512700 1 -- 172.21.15.16:0/2017364954 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f5b9406b020 msgr2=0x7f5b9406d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:44:25.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.520+0000 7f5bb8512700 1 --2- 172.21.15.16:0/2017364954 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f5b9406b020 0x7f5b9406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:25.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.520+0000 7f5bb8512700 1 -- 172.21.15.16:0/2017364954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5bb00ce4f0 msgr2=0x7f5bb015d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:25.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.520+0000 7f5bb8512700 1 --2- 172.21.15.16:0/2017364954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5bb00ce4f0 0x7f5bb015d7d0 secure :-1 s=READY pgs=2206 cs=0 l=1 rev1=1 crypto rx=0x7f5bac00c730 tx=0x7f5bac00ca40 comp rx=0 tx=0).stop 2023-10-16T13:44:25.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.520+0000 7f5bb8512700 1 -- 172.21.15.16:0/2017364954 shutdown_connections 2023-10-16T13:44:25.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.520+0000 7f5bb8512700 1 --2- 172.21.15.16:0/2017364954 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f5b9406b020 0x7f5b9406d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:25.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.520+0000 7f5bb8512700 1 --2- 172.21.15.16:0/2017364954 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5bb015ba10 0x7f5bb00c8580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:25.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.520+0000 7f5bb8512700 1 --2- 172.21.15.16:0/2017364954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5bb00ce4f0 0x7f5bb015d7d0 unknown :-1 s=CLOSED pgs=2206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:25.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.520+0000 7f5bb8512700 1 --2- 172.21.15.16:0/2017364954 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5bb015b630 0x7f5bb01f1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:25.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.520+0000 7f5bb8512700 1 -- 172.21.15.16:0/2017364954 >> 172.21.15.16:0/2017364954 conn(0x7f5bb00d14b0 msgr2=0x7f5bb00d1890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:25.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.520+0000 7f5bb8512700 1 -- 172.21.15.16:0/2017364954 shutdown_connections 2023-10-16T13:44:25.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.520+0000 7f5bb8512700 1 -- 172.21.15.16:0/2017364954 wait complete. 2023-10-16T13:44:25.525 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 95 2023-10-16T13:44:25.538 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:44:25.637 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:44:25.635+0000 7fe4965ec200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T13:44:25.654 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:44:25.651+0000 7ffb56192200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T13:44:25.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.662+0000 7f7d4d345700 1 Processor -- start 2023-10-16T13:44:25.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.663+0000 7f7d4d345700 1 -- start start 2023-10-16T13:44:25.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.664+0000 7f7d4d345700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f7d480ce4f0 0x7f7d480d0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:25.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.664+0000 7f7d4d345700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7d4815ba10 0x7f7d480cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:25.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.664+0000 7f7d4d345700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d480b0970 con 0x7f7d4815b630 2023-10-16T13:44:25.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.664+0000 7f7d4d345700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d480cd690 con 0x7f7d4815ba10 2023-10-16T13:44:25.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.664+0000 7f7d4d345700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d480cd970 con 0x7f7d480ce4f0 2023-10-16T13:44:25.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.664+0000 7f7d477fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7d4815ba10 0x7f7d480cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:25.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.664+0000 7f7d477fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7d4815ba10 0x7f7d480cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51154/0 (socket says 172.21.15.16:51154) 2023-10-16T13:44:25.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.664+0000 7f7d477fe700 1 -- 172.21.15.16:0/2473154956 learned_addr learned my addr 172.21.15.16:0/2473154956 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:25.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.664+0000 7f7d467fc700 1 --2- 172.21.15.16:0/2473154956 >> v2:172.21.15.49:3300/0 conn(0x7f7d480ce4f0 0x7f7d480d0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:25.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.664+0000 7f7d477fe700 1 -- 172.21.15.16:0/2473154956 >> v2:172.21.15.49:3300/0 conn(0x7f7d480ce4f0 msgr2=0x7f7d480d0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:25.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.664+0000 7f7d477fe700 1 --2- 172.21.15.16:0/2473154956 >> v2:172.21.15.49:3300/0 conn(0x7f7d480ce4f0 0x7f7d480d0380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:25.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.664+0000 7f7d45ffb700 1 -- 172.21.15.16:0/2473154956 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1498652730 0 0) 0x7f7d480b0970 con 0x7f7d4815b630 2023-10-16T13:44:25.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.664+0000 7f7d477fe700 1 -- 172.21.15.16:0/2473154956 >> v1:172.21.15.49:6789/0 conn(0x7f7d4815b630 legacy=0x7f7d480cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:25.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.664+0000 7f7d477fe700 1 -- 172.21.15.16:0/2473154956 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d480b0c30 con 0x7f7d4815ba10 2023-10-16T13:44:25.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.665+0000 7f7d477fe700 1 --2- 172.21.15.16:0/2473154956 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d4815ba10 0x7f7d480cf290 secure :-1 s=READY pgs=2207 cs=0 l=1 rev1=1 crypto rx=0x7f7d480d2460 tx=0x7f7d4000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23eeeda426ac4590 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:25.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.665+0000 7f7d45ffb700 1 -- 172.21.15.16:0/2473154956 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d4000e070 con 0x7f7d4815ba10 2023-10-16T13:44:25.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.665+0000 7f7d45ffb700 1 -- 172.21.15.16:0/2473154956 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d4000b910 con 0x7f7d4815ba10 2023-10-16T13:44:25.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.665+0000 7f7d45ffb700 1 -- 172.21.15.16:0/2473154956 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d4000bcc0 con 0x7f7d4815ba10 2023-10-16T13:44:25.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.665+0000 7f7d4d345700 1 -- 172.21.15.16:0/2473154956 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d4815ba10 msgr2=0x7f7d480cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:25.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.665+0000 7f7d4d345700 1 --2- 172.21.15.16:0/2473154956 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d4815ba10 0x7f7d480cf290 secure :-1 s=READY pgs=2207 cs=0 l=1 rev1=1 crypto rx=0x7f7d480d2460 tx=0x7f7d4000b330 comp rx=0 tx=0).stop 2023-10-16T13:44:25.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.665+0000 7f7d4d345700 1 -- 172.21.15.16:0/2473154956 shutdown_connections 2023-10-16T13:44:25.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.665+0000 7f7d4d345700 1 --2- 172.21.15.16:0/2473154956 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d4815ba10 0x7f7d480cf290 unknown :-1 s=CLOSED pgs=2207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:25.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.665+0000 7f7d4d345700 1 --2- 172.21.15.16:0/2473154956 >> v2:172.21.15.49:3300/0 conn(0x7f7d480ce4f0 0x7f7d480d0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:25.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.665+0000 7f7d4d345700 1 -- 172.21.15.16:0/2473154956 >> 172.21.15.16:0/2473154956 conn(0x7f7d480d14b0 msgr2=0x7f7d480d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:25.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.665+0000 7f7d4d345700 1 -- 172.21.15.16:0/2473154956 shutdown_connections 2023-10-16T13:44:25.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.665+0000 7f7d4d345700 1 -- 172.21.15.16:0/2473154956 wait complete. 2023-10-16T13:44:25.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.667+0000 7f7d4d345700 1 Processor -- start 2023-10-16T13:44:25.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.667+0000 7f7d4d345700 1 -- start start 2023-10-16T13:44:25.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.668+0000 7f7d4d345700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7d480ce4f0 0x7f7d48150ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:25.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.668+0000 7f7d4d345700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7d4815b630 0x7f7d4815e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:25.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.668+0000 7f7d4d345700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d4815ba10 0x7f7d480b4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:25.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.668+0000 7f7d4d345700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d48150ff0 con 0x7f7d4815ba10 2023-10-16T13:44:25.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.668+0000 7f7d4d345700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4815c910 con 0x7f7d480ce4f0 2023-10-16T13:44:25.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.668+0000 7f7d4d345700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4815d770 con 0x7f7d4815b630 2023-10-16T13:44:25.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.668+0000 7f7d477fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d4815ba10 0x7f7d480b4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:25.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.668+0000 7f7d477fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d4815ba10 0x7f7d480b4300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51160/0 (socket says 172.21.15.16:51160) 2023-10-16T13:44:25.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.668+0000 7f7d46ffd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7d480ce4f0 0x7f7d48150ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:25.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.668+0000 7f7d477fe700 1 -- 172.21.15.16:0/2313668761 learned_addr learned my addr 172.21.15.16:0/2313668761 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:25.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.668+0000 7f7d467fc700 1 --2- 172.21.15.16:0/2313668761 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7d4815b630 0x7f7d4815e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:25.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.668+0000 7f7d477fe700 1 -- 172.21.15.16:0/2313668761 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7d4815b630 msgr2=0x7f7d4815e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:25.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.668+0000 7f7d477fe700 1 --2- 172.21.15.16:0/2313668761 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7d4815b630 0x7f7d4815e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:25.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.668+0000 7f7d477fe700 1 -- 172.21.15.16:0/2313668761 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7d480ce4f0 msgr2=0x7f7d48150ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:25.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.668+0000 7f7d477fe700 1 --2- 172.21.15.16:0/2313668761 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7d480ce4f0 0x7f7d48150ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:25.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.668+0000 7f7d477fe700 1 -- 172.21.15.16:0/2313668761 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d40009b30 con 0x7f7d4815ba10 2023-10-16T13:44:25.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.668+0000 7f7d467fc700 1 --2- 172.21.15.16:0/2313668761 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7d4815b630 0x7f7d4815e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:44:25.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.668+0000 7f7d477fe700 1 --2- 172.21.15.16:0/2313668761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d4815ba10 0x7f7d480b4300 secure :-1 s=READY pgs=2208 cs=0 l=1 rev1=1 crypto rx=0x7f7d48072680 tx=0x7f7d40004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:25.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.669+0000 7f7d2f7fe700 1 -- 172.21.15.16:0/2313668761 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d40004840 con 0x7f7d4815ba10 2023-10-16T13:44:25.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.669+0000 7f7d2f7fe700 1 -- 172.21.15.16:0/2313668761 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d400099b0 con 0x7f7d4815ba10 2023-10-16T13:44:25.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.669+0000 7f7d4d345700 1 -- 172.21.15.16:0/2313668761 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d4815cb60 con 0x7f7d4815ba10 2023-10-16T13:44:25.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.669+0000 7f7d2f7fe700 1 -- 172.21.15.16:0/2313668761 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d4001b670 con 0x7f7d4815ba10 2023-10-16T13:44:25.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.669+0000 7f7d4d345700 1 -- 172.21.15.16:0/2313668761 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d4815d120 con 0x7f7d4815ba10 2023-10-16T13:44:25.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.670+0000 7f7d4d345700 1 -- 172.21.15.16:0/2313668761 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d4815d8b0 con 0x7f7d4815ba10 2023-10-16T13:44:25.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.670+0000 7f7d2f7fe700 1 -- 172.21.15.16:0/2313668761 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 122) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f7d4001b7d0 con 0x7f7d4815ba10 2023-10-16T13:44:25.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.671+0000 7f7d2f7fe700 1 --2- 172.21.15.16:0/2313668761 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f7d3406b020 0x7f7d3406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:25.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.671+0000 7f7d46ffd700 1 -- 172.21.15.16:0/2313668761 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f7d3406b020 msgr2=0x7f7d3406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/371380443 2023-10-16T13:44:25.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.671+0000 7f7d2f7fe700 1 -- 172.21.15.16:0/2313668761 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(194..194 src has 1..194) v4 ==== 6889+0+0 (secure 0 0 0) 0x7f7d4008b490 con 0x7f7d4815ba10 2023-10-16T13:44:25.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.671+0000 7f7d46ffd700 1 --2- 172.21.15.16:0/2313668761 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f7d3406b020 0x7f7d3406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:44:25.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.679+0000 7f7d2f7fe700 1 -- 172.21.15.16:0/2313668761 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7d4815d8b0 con 0x7f7d4815ba10 2023-10-16T13:44:25.816 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:44:25.814+0000 7fe4965ec200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T13:44:25.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.872+0000 7f7d46ffd700 1 -- 172.21.15.16:0/2313668761 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f7d3406b020 msgr2=0x7f7d3406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/371380443 2023-10-16T13:44:25.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.872+0000 7f7d46ffd700 1 --2- 172.21.15.16:0/2313668761 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f7d3406b020 0x7f7d3406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:44:25.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.888+0000 7f7d4d345700 1 -- 172.21.15.16:0/2313668761 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7d4815d8b0 con 0x7f7d4815ba10 2023-10-16T13:44:25.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.890+0000 7f7d2f7fe700 1 -- 172.21.15.16:0/2313668761 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v194) v1 ==== 74+0+17905 (secure 0 0 0) 0x7f7d4815d8b0 con 0x7f7d4815ba10 2023-10-16T13:44:25.892 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:44:25.893 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":194,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:44:21.720034+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2023-10-16T13:28:17.549815+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"189","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2023-10-16T13:28:18.493327+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"194","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":33,"snap_epoch":194,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6833/3952564978":"2023-10-17T13:39:22.557973+0000","172.21.15.16:6832/3952564978":"2023-10-17T13:39:22.557973+0000","172.21.15.16:0/2554840790":"2023-10-17T13:39:22.557973+0000","172.21.15.16:0/360056771":"2023-10-17T13:39:22.557973+0000","172.21.15.16:0/3520685377":"2023-10-17T13:39:22.557973+0000","172.21.15.16:0/1571009468":"2023-10-17T13:28:34.631949+0000","172.21.15.16:0/678119374":"2023-10-17T13:28:34.631949+0000","172.21.15.16:0/3920247727":"2023-10-17T13:28:34.631949+0000","172.21.15.16:0/857902049":"2023-10-17T13:39:22.557973+0000","172.21.15.16:6833/4211329181":"2023-10-17T13:28:34.631949+0000","172.21.15.16:6832/4211329181":"2023-10-17T13:28:34.631949+0000","172.21.15.16:0/1548506671":"2023-10-17T13:39:22.557973+0000","172.21.15.16:0/2852193608":"2023-10-17T13:28:34.631949+0000","172.21.15.16:0/1641955134":"2023-10-17T13:39:22.557973+0000","172.21.15.16:0/471630266":"2023-10-17T13:28:34.631949+0000","172.21.15.16:0/3153082432":"2023-10-17T13:28:34.631949+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[{"pool":24,"snaps":[{"begin":17,"length":1},{"begin":33,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:44:25.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.892+0000 7f7d4d345700 1 -- 172.21.15.16:0/2313668761 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f7d3406b020 msgr2=0x7f7d3406d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:44:25.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.892+0000 7f7d4d345700 1 --2- 172.21.15.16:0/2313668761 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f7d3406b020 0x7f7d3406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:25.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.892+0000 7f7d4d345700 1 -- 172.21.15.16:0/2313668761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d4815ba10 msgr2=0x7f7d480b4300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:25.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.892+0000 7f7d4d345700 1 --2- 172.21.15.16:0/2313668761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d4815ba10 0x7f7d480b4300 secure :-1 s=READY pgs=2208 cs=0 l=1 rev1=1 crypto rx=0x7f7d48072680 tx=0x7f7d40004ad0 comp rx=0 tx=0).stop 2023-10-16T13:44:25.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.892+0000 7f7d4d345700 1 -- 172.21.15.16:0/2313668761 shutdown_connections 2023-10-16T13:44:25.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.892+0000 7f7d4d345700 1 --2- 172.21.15.16:0/2313668761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d4815ba10 0x7f7d480b4300 unknown :-1 s=CLOSED pgs=2208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:25.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.892+0000 7f7d4d345700 1 --2- 172.21.15.16:0/2313668761 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7d480ce4f0 0x7f7d48150ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:25.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.892+0000 7f7d4d345700 1 --2- 172.21.15.16:0/2313668761 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f7d3406b020 0x7f7d3406d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:25.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.892+0000 7f7d4d345700 1 --2- 172.21.15.16:0/2313668761 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7d4815b630 0x7f7d4815e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:25.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.892+0000 7f7d4d345700 1 -- 172.21.15.16:0/2313668761 >> 172.21.15.16:0/2313668761 conn(0x7f7d480d14b0 msgr2=0x7f7d4815a940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:25.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.892+0000 7f7d4d345700 1 -- 172.21.15.16:0/2313668761 shutdown_connections 2023-10-16T13:44:25.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:25.892+0000 7f7d4d345700 1 -- 172.21.15.16:0/2313668761 wait complete. 2023-10-16T13:44:25.907 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-16T13:44:25.908 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:44:26.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.031+0000 7f69367a4700 1 Processor -- start 2023-10-16T13:44:26.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.031+0000 7f69367a4700 1 -- start start 2023-10-16T13:44:26.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.032+0000 7f69367a4700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f69300ce4f0 0x7f69300d0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:26.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.032+0000 7f69367a4700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f693015ba10 0x7f69300cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:26.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.032+0000 7f69367a4700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69300b0970 con 0x7f693015b630 2023-10-16T13:44:26.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.032+0000 7f69367a4700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69300cd690 con 0x7f693015ba10 2023-10-16T13:44:26.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.032+0000 7f69367a4700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69300cd970 con 0x7f69300ce4f0 2023-10-16T13:44:26.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.032+0000 7f692ffff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f693015b630 0x7f69300cdde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:40440/0 (socket says 172.21.15.16:40440) 2023-10-16T13:44:26.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.032+0000 7f692ffff700 1 -- 172.21.15.16:0/3222933599 learned_addr learned my addr 172.21.15.16:0/3222933599 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:44:26.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.032+0000 7f6934d41700 1 --2- 172.21.15.16:0/3222933599 >> v2:172.21.15.49:3300/0 conn(0x7f693015ba10 0x7f69300cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:26.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.032+0000 7f692f7fe700 1 --2- 172.21.15.16:0/3222933599 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f69300ce4f0 0x7f69300d0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:26.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.032+0000 7f692effd700 1 -- 172.21.15.16:0/3222933599 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 227624294 0 0) 0x7f69300b0970 con 0x7f693015b630 2023-10-16T13:44:26.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.032+0000 7f692effd700 1 -- 172.21.15.16:0/3222933599 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69180035b0 con 0x7f693015b630 2023-10-16T13:44:26.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.032+0000 7f6934d41700 1 -- 172.21.15.16:0/3222933599 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f69300ce4f0 msgr2=0x7f69300d0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:26.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.032+0000 7f6934d41700 1 --2- 172.21.15.16:0/3222933599 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f69300ce4f0 0x7f69300d0380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:26.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.032+0000 7f6934d41700 1 -- 172.21.15.16:0/3222933599 >> v1:172.21.15.16:6789/0 conn(0x7f693015b630 legacy=0x7f69300cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:26.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.032+0000 7f692effd700 1 -- 172.21.15.16:0/3222933599 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3112509404 0 0) 0x7f69180035b0 con 0x7f693015b630 2023-10-16T13:44:26.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.032+0000 7f6934d41700 1 -- 172.21.15.16:0/3222933599 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69300b0c30 con 0x7f693015ba10 2023-10-16T13:44:26.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.032+0000 7f692f7fe700 1 --2- 172.21.15.16:0/3222933599 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f69300ce4f0 0x7f69300d0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:44:26.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.033+0000 7f6934d41700 1 --2- 172.21.15.16:0/3222933599 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f693015ba10 0x7f69300cf290 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f6928009d90 tx=0x7f692800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b1eb9dbefff664 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:26.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.033+0000 7f692effd700 1 -- 172.21.15.16:0/3222933599 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f692800e070 con 0x7f693015ba10 2023-10-16T13:44:26.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.033+0000 7f692effd700 1 -- 172.21.15.16:0/3222933599 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f692800bb20 con 0x7f693015ba10 2023-10-16T13:44:26.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.033+0000 7f692effd700 1 -- 172.21.15.16:0/3222933599 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f692800be90 con 0x7f693015ba10 2023-10-16T13:44:26.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.033+0000 7f69367a4700 1 -- 172.21.15.16:0/3222933599 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f693015ba10 msgr2=0x7f69300cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:26.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.033+0000 7f69367a4700 1 --2- 172.21.15.16:0/3222933599 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f693015ba10 0x7f69300cf290 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f6928009d90 tx=0x7f692800b330 comp rx=0 tx=0).stop 2023-10-16T13:44:26.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.033+0000 7f69367a4700 1 -- 172.21.15.16:0/3222933599 shutdown_connections 2023-10-16T13:44:26.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.033+0000 7f69367a4700 1 --2- 172.21.15.16:0/3222933599 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f693015ba10 0x7f69300cf290 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:26.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.033+0000 7f69367a4700 1 --2- 172.21.15.16:0/3222933599 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f69300ce4f0 0x7f69300d0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:26.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.033+0000 7f69367a4700 1 -- 172.21.15.16:0/3222933599 >> 172.21.15.16:0/3222933599 conn(0x7f69300d14b0 msgr2=0x7f69300d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:26.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.033+0000 7f69367a4700 1 -- 172.21.15.16:0/3222933599 shutdown_connections 2023-10-16T13:44:26.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.033+0000 7f69367a4700 1 -- 172.21.15.16:0/3222933599 wait complete. 2023-10-16T13:44:26.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.035+0000 7f69367a4700 1 Processor -- start 2023-10-16T13:44:26.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.035+0000 7f69367a4700 1 -- start start 2023-10-16T13:44:26.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.035+0000 7f69367a4700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f69300ce4f0 0x7f693015e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:26.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.035+0000 7f69367a4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f693015b630 0x7f69301f0a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:26.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.035+0000 7f69367a4700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f693015ba10 0x7f69300b1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:26.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.035+0000 7f69367a4700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f693015ed40 con 0x7f693015b630 2023-10-16T13:44:26.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.035+0000 7f69367a4700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f693015cd70 con 0x7f693015ba10 2023-10-16T13:44:26.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.035+0000 7f69367a4700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f693015d050 con 0x7f69300ce4f0 2023-10-16T13:44:26.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.036+0000 7f692f7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f693015b630 0x7f69301f0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:26.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.036+0000 7f692f7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f693015b630 0x7f69301f0a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51174/0 (socket says 172.21.15.16:51174) 2023-10-16T13:44:26.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.036+0000 7f692f7fe700 1 -- 172.21.15.16:0/4191804527 learned_addr learned my addr 172.21.15.16:0/4191804527 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:26.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.036+0000 7f692ffff700 1 --2- 172.21.15.16:0/4191804527 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f69300ce4f0 0x7f693015e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:26.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.036+0000 7f6934d41700 1 --2- 172.21.15.16:0/4191804527 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f693015ba10 0x7f69300b1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:26.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.036+0000 7f692f7fe700 1 -- 172.21.15.16:0/4191804527 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f69300ce4f0 msgr2=0x7f693015e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:26.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.036+0000 7f692f7fe700 1 --2- 172.21.15.16:0/4191804527 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f69300ce4f0 0x7f693015e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:26.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.036+0000 7f692f7fe700 1 -- 172.21.15.16:0/4191804527 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f693015ba10 msgr2=0x7f69300b1840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:26.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.036+0000 7f692f7fe700 1 --2- 172.21.15.16:0/4191804527 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f693015ba10 0x7f69300b1840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:26.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.036+0000 7f692f7fe700 1 -- 172.21.15.16:0/4191804527 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69240094a0 con 0x7f693015b630 2023-10-16T13:44:26.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.036+0000 7f692f7fe700 1 --2- 172.21.15.16:0/4191804527 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f693015b630 0x7f69301f0a20 secure :-1 s=READY pgs=2210 cs=0 l=1 rev1=1 crypto rx=0x7f692400ebe0 tx=0x7f692400eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:26.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.036+0000 7f692cff9700 1 -- 172.21.15.16:0/4191804527 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6924009760 con 0x7f693015b630 2023-10-16T13:44:26.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.036+0000 7f692cff9700 1 -- 172.21.15.16:0/4191804527 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f692400c850 con 0x7f693015b630 2023-10-16T13:44:26.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.036+0000 7f69367a4700 1 -- 172.21.15.16:0/4191804527 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6928009a70 con 0x7f693015b630 2023-10-16T13:44:26.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.036+0000 7f692cff9700 1 -- 172.21.15.16:0/4191804527 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f692400cbe0 con 0x7f693015b630 2023-10-16T13:44:26.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.036+0000 7f69367a4700 1 -- 172.21.15.16:0/4191804527 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f693015d610 con 0x7f693015b630 2023-10-16T13:44:26.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.037+0000 7f69367a4700 1 -- 172.21.15.16:0/4191804527 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f693004ebb0 con 0x7f693015b630 2023-10-16T13:44:26.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.038+0000 7f692cff9700 1 -- 172.21.15.16:0/4191804527 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 122) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f692400cd40 con 0x7f693015b630 2023-10-16T13:44:26.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.038+0000 7f692cff9700 1 --2- 172.21.15.16:0/4191804527 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f691c06b050 0x7f691c06d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:26.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.038+0000 7f692ffff700 1 -- 172.21.15.16:0/4191804527 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f691c06b050 msgr2=0x7f691c06d510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/371380443 2023-10-16T13:44:26.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.038+0000 7f692cff9700 1 -- 172.21.15.16:0/4191804527 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(194..194 src has 1..194) v4 ==== 6889+0+0 (secure 0 0 0) 0x7f6924089b60 con 0x7f693015b630 2023-10-16T13:44:26.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.038+0000 7f692ffff700 1 --2- 172.21.15.16:0/4191804527 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f691c06b050 0x7f691c06d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:44:26.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.043+0000 7f692cff9700 1 -- 172.21.15.16:0/4191804527 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6924054660 con 0x7f693015b630 2023-10-16T13:44:26.208 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:44:26.205+0000 7ffb56192200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T13:44:26.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.239+0000 7f692ffff700 1 -- 172.21.15.16:0/4191804527 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f691c06b050 msgr2=0x7f691c06d510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/371380443 2023-10-16T13:44:26.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.239+0000 7f692ffff700 1 --2- 172.21.15.16:0/4191804527 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f691c06b050 0x7f691c06d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:44:26.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.292+0000 7f69367a4700 1 -- 172.21.15.16:0/4191804527 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6930154ef0 con 0x7f693015b630 2023-10-16T13:44:26.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.293+0000 7f692cff9700 1 -- 172.21.15.16:0/4191804527 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+4920 (secure 0 0 0) 0x7f6924021090 con 0x7f693015b630 2023-10-16T13:44:26.295 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:44:26.295 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":95,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8306,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2415675492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":2415675492},{"type":"v1","addr":"172.21.15.16:6837","nonce":2415675492}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":90},{"gid":8315,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/4031561298","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":4031561298},{"type":"v1","addr":"172.21.15.49:6834","nonce":4031561298}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":90},{"gid":8318,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/2853801536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":2853801536},{"type":"v1","addr":"172.21.15.49:6835","nonce":2853801536}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":90}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:28:19.523721+0000","modified":"2023-10-16T13:28:22.528826+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10296":{"gid":10296,"name":"c","rank":0,"incarnation":91,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6836/3244401013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":3244401013},{"type":"v1","addr":"172.21.15.16:6836","nonce":3244401013}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2023-10-16T13:44:26.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.295+0000 7f69367a4700 1 -- 172.21.15.16:0/4191804527 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f691c06b050 msgr2=0x7f691c06d510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:44:26.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.295+0000 7f69367a4700 1 --2- 172.21.15.16:0/4191804527 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f691c06b050 0x7f691c06d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:26.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.295+0000 7f69367a4700 1 -- 172.21.15.16:0/4191804527 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f693015b630 msgr2=0x7f69301f0a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:26.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.295+0000 7f69367a4700 1 --2- 172.21.15.16:0/4191804527 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f693015b630 0x7f69301f0a20 secure :-1 s=READY pgs=2210 cs=0 l=1 rev1=1 crypto rx=0x7f692400ebe0 tx=0x7f692400eef0 comp rx=0 tx=0).stop 2023-10-16T13:44:26.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.295+0000 7f69367a4700 1 -- 172.21.15.16:0/4191804527 shutdown_connections 2023-10-16T13:44:26.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.295+0000 7f69367a4700 1 --2- 172.21.15.16:0/4191804527 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f693015ba10 0x7f69300b1840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:26.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.295+0000 7f69367a4700 1 --2- 172.21.15.16:0/4191804527 >> [v2:172.21.15.16:6832/371380443,v1:172.21.15.16:6833/371380443] conn(0x7f691c06b050 0x7f691c06d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:26.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.295+0000 7f69367a4700 1 --2- 172.21.15.16:0/4191804527 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f69300ce4f0 0x7f693015e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:26.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.295+0000 7f69367a4700 1 --2- 172.21.15.16:0/4191804527 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f693015b630 0x7f69301f0a20 unknown :-1 s=CLOSED pgs=2210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:26.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.295+0000 7f69367a4700 1 -- 172.21.15.16:0/4191804527 >> 172.21.15.16:0/4191804527 conn(0x7f69300d14b0 msgr2=0x7f693006f910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:26.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.295+0000 7f69367a4700 1 -- 172.21.15.16:0/4191804527 shutdown_connections 2023-10-16T13:44:26.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.295+0000 7f69367a4700 1 -- 172.21.15.16:0/4191804527 wait complete. 2023-10-16T13:44:26.300 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 95 2023-10-16T13:44:26.310 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:44:26.397 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:44:26.395+0000 7fe4965ec200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T13:44:26.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.444+0000 7fa8e2a3f700 1 Processor -- start 2023-10-16T13:44:26.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.444+0000 7fa8e2a3f700 1 -- start start 2023-10-16T13:44:26.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.445+0000 7fa8e2a3f700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fa8dc159c90 0x7fa8dc15d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:26.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.445+0000 7fa8e2a3f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa8dc0cb190 0x7fa8dc15dee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:26.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.445+0000 7fa8e2a3f700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8dc06b1f0 con 0x7fa8dc15a070 2023-10-16T13:44:26.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.445+0000 7fa8e2a3f700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8dc06b4b0 con 0x7fa8dc159c90 2023-10-16T13:44:26.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.445+0000 7fa8e2a3f700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8dc06b790 con 0x7fa8dc0cb190 2023-10-16T13:44:26.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.445+0000 7fa8db7fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa8dc15a070 0x7fa8dc0ae520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:40476/0 (socket says 172.21.15.16:40476) 2023-10-16T13:44:26.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.445+0000 7fa8db7fe700 1 -- 172.21.15.16:0/863695437 learned_addr learned my addr 172.21.15.16:0/863695437 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:44:26.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.446+0000 7fa8e0fdc700 1 --2- 172.21.15.16:0/863695437 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa8dc0cb190 0x7fa8dc15dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:26.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.446+0000 7fa8dbfff700 1 --2- 172.21.15.16:0/863695437 >> v2:172.21.15.49:3300/0 conn(0x7fa8dc159c90 0x7fa8dc15d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:26.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.446+0000 7fa8daffd700 1 -- 172.21.15.16:0/863695437 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1811378916 0 0) 0x7fa8dc06b1f0 con 0x7fa8dc15a070 2023-10-16T13:44:26.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.446+0000 7fa8daffd700 1 -- 172.21.15.16:0/863695437 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8c40035b0 con 0x7fa8dc15a070 2023-10-16T13:44:26.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.446+0000 7fa8e0fdc700 1 -- 172.21.15.16:0/863695437 >> v2:172.21.15.49:3300/0 conn(0x7fa8dc159c90 msgr2=0x7fa8dc15d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:26.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.446+0000 7fa8e0fdc700 1 --2- 172.21.15.16:0/863695437 >> v2:172.21.15.49:3300/0 conn(0x7fa8dc159c90 0x7fa8dc15d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:26.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.446+0000 7fa8e0fdc700 1 -- 172.21.15.16:0/863695437 >> v1:172.21.15.16:6789/0 conn(0x7fa8dc15a070 legacy=0x7fa8dc0ae520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:26.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.446+0000 7fa8e0fdc700 1 -- 172.21.15.16:0/863695437 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8dc06ba70 con 0x7fa8dc0cb190 2023-10-16T13:44:26.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.446+0000 7fa8dbfff700 1 --2- 172.21.15.16:0/863695437 >> v2:172.21.15.49:3300/0 conn(0x7fa8dc159c90 0x7fa8dc15d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:44:26.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.446+0000 7fa8e0fdc700 1 --2- 172.21.15.16:0/863695437 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa8dc0cb190 0x7fa8dc15dee0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fa8d4009d90 tx=0x7fa8d400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3b1203794b4caf server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:26.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.446+0000 7fa8daffd700 1 -- 172.21.15.16:0/863695437 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8d400e070 con 0x7fa8dc0cb190 2023-10-16T13:44:26.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.446+0000 7fa8daffd700 1 -- 172.21.15.16:0/863695437 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8d400bb20 con 0x7fa8dc0cb190 2023-10-16T13:44:26.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.446+0000 7fa8daffd700 1 -- 172.21.15.16:0/863695437 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8d4012430 con 0x7fa8dc0cb190 2023-10-16T13:44:26.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.447+0000 7fa8e2a3f700 1 -- 172.21.15.16:0/863695437 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa8dc0cb190 msgr2=0x7fa8dc15dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:26.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.447+0000 7fa8e2a3f700 1 --2- 172.21.15.16:0/863695437 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa8dc0cb190 0x7fa8dc15dee0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fa8d4009d90 tx=0x7fa8d400b330 comp rx=0 tx=0).stop 2023-10-16T13:44:26.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.447+0000 7fa8e2a3f700 1 -- 172.21.15.16:0/863695437 shutdown_connections 2023-10-16T13:44:26.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.447+0000 7fa8e2a3f700 1 --2- 172.21.15.16:0/863695437 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa8dc0cb190 0x7fa8dc15dee0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:26.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.447+0000 7fa8e2a3f700 1 --2- 172.21.15.16:0/863695437 >> v2:172.21.15.49:3300/0 conn(0x7fa8dc159c90 0x7fa8dc15d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:26.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.447+0000 7fa8e2a3f700 1 -- 172.21.15.16:0/863695437 >> 172.21.15.16:0/863695437 conn(0x7fa8dc0cc5b0 msgr2=0x7fa8dc0cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:26.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.447+0000 7fa8e2a3f700 1 -- 172.21.15.16:0/863695437 shutdown_connections 2023-10-16T13:44:26.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.447+0000 7fa8e2a3f700 1 -- 172.21.15.16:0/863695437 wait complete. 2023-10-16T13:44:26.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.449+0000 7fa8e2a3f700 1 Processor -- start 2023-10-16T13:44:26.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.449+0000 7fa8e2a3f700 1 -- start start 2023-10-16T13:44:26.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.450+0000 7fa8e2a3f700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa8dc0cb190 0x7fa8dc1f5c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:26.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.450+0000 7fa8e2a3f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8dc159c90 0x7fa8dc15b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:26.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.450+0000 7fa8e2a3f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa8dc15a070 0x7fa8dc15bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:26.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.450+0000 7fa8e2a3f700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8dc0b3650 con 0x7fa8dc159c90 2023-10-16T13:44:26.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.450+0000 7fa8e2a3f700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8dc0b3930 con 0x7fa8dc0cb190 2023-10-16T13:44:26.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.450+0000 7fa8e2a3f700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8dc0b3c10 con 0x7fa8dc15a070 2023-10-16T13:44:26.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.450+0000 7fa8db7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8dc159c90 0x7fa8dc15b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:26.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.450+0000 7fa8dbfff700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa8dc0cb190 0x7fa8dc1f5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:26.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.450+0000 7fa8db7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8dc159c90 0x7fa8dc15b9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51210/0 (socket says 172.21.15.16:51210) 2023-10-16T13:44:26.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.450+0000 7fa8db7fe700 1 -- 172.21.15.16:0/327098539 learned_addr learned my addr 172.21.15.16:0/327098539 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:26.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.450+0000 7fa8e0fdc700 1 --2- 172.21.15.16:0/327098539 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa8dc15a070 0x7fa8dc15bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:26.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.450+0000 7fa8db7fe700 1 -- 172.21.15.16:0/327098539 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa8dc15a070 msgr2=0x7fa8dc15bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:26.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.450+0000 7fa8db7fe700 1 --2- 172.21.15.16:0/327098539 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa8dc15a070 0x7fa8dc15bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:26.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.450+0000 7fa8db7fe700 1 -- 172.21.15.16:0/327098539 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa8dc0cb190 msgr2=0x7fa8dc1f5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:26.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.450+0000 7fa8db7fe700 1 --2- 172.21.15.16:0/327098539 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa8dc0cb190 0x7fa8dc1f5c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:26.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.450+0000 7fa8db7fe700 1 -- 172.21.15.16:0/327098539 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8cc0094a0 con 0x7fa8dc159c90 2023-10-16T13:44:26.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.450+0000 7fa8dbfff700 1 --2- 172.21.15.16:0/327098539 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa8dc0cb190 0x7fa8dc1f5c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:44:26.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.451+0000 7fa8db7fe700 1 --2- 172.21.15.16:0/327098539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8dc159c90 0x7fa8dc15b9f0 secure :-1 s=READY pgs=2213 cs=0 l=1 rev1=1 crypto rx=0x7fa8d000c4d0 tx=0x7fa8d000c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:26.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.451+0000 7fa8d8ff9700 1 -- 172.21.15.16:0/327098539 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8d0005d70 con 0x7fa8dc159c90 2023-10-16T13:44:26.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.451+0000 7fa8e2a3f700 1 -- 172.21.15.16:0/327098539 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8d4009a70 con 0x7fa8dc159c90 2023-10-16T13:44:26.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.451+0000 7fa8e2a3f700 1 -- 172.21.15.16:0/327098539 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8dc0adaa0 con 0x7fa8dc159c90 2023-10-16T13:44:26.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.451+0000 7fa8d8ff9700 1 -- 172.21.15.16:0/327098539 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8d000ce80 con 0x7fa8dc159c90 2023-10-16T13:44:26.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.451+0000 7fa8d8ff9700 1 -- 172.21.15.16:0/327098539 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8d00072d0 con 0x7fa8dc159c90 2023-10-16T13:44:26.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.452+0000 7fa8e2a3f700 1 -- 172.21.15.16:0/327098539 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8dc04ebb0 con 0x7fa8dc159c90 2023-10-16T13:44:26.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.452+0000 7fa8d8ff9700 1 -- 172.21.15.16:0/327098539 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 123) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fa8d0007430 con 0x7fa8dc159c90 2023-10-16T13:44:26.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.453+0000 7fa8d8ff9700 1 -- 172.21.15.16:0/327098539 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 7189+0+0 (secure 0 0 0) 0x7fa8d0088920 con 0x7fa8dc159c90 2023-10-16T13:44:26.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.458+0000 7fa8d8ff9700 1 -- 172.21.15.16:0/327098539 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa8d0053880 con 0x7fa8dc159c90 2023-10-16T13:44:26.487 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:44:26.484+0000 7fe411dd8700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:44:26.487 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:44:26.484+0000 7fe411dd8700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:44:26.488 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:44:26.484+0000 7fe411dd8700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:44:26.488 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:44:26.484+0000 7fe411dd8700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:44:26.488 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:44:26.484+0000 7fe411dd8700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:44:26.489 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:44:26.485+0000 7fe40edd2700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:44:26.489 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:44:26.485+0000 7fe40edd2700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:44:26.489 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:44:26.485+0000 7fe40edd2700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:44:26.489 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:44:26.485+0000 7fe40edd2700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:44:26.490 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:44:26.485+0000 7fe40edd2700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:44:26.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.718+0000 7fa8e2a3f700 1 -- 172.21.15.16:0/327098539 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa8dc0ce410 con 0x7fa8dc159c90 2023-10-16T13:44:26.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.719+0000 7fa8d8ff9700 1 -- 172.21.15.16:0/327098539 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+4920 (secure 0 0 0) 0x7fa8d0056ea0 con 0x7fa8dc159c90 2023-10-16T13:44:26.721 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:44:26.722 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":95,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8306,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2415675492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":2415675492},{"type":"v1","addr":"172.21.15.16:6837","nonce":2415675492}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":90},{"gid":8315,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/4031561298","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":4031561298},{"type":"v1","addr":"172.21.15.49:6834","nonce":4031561298}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":90},{"gid":8318,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/2853801536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":2853801536},{"type":"v1","addr":"172.21.15.49:6835","nonce":2853801536}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":90}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:28:19.523721+0000","modified":"2023-10-16T13:28:22.528826+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10296":{"gid":10296,"name":"c","rank":0,"incarnation":91,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6836/3244401013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":3244401013},{"type":"v1","addr":"172.21.15.16:6836","nonce":3244401013}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2023-10-16T13:44:26.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.721+0000 7fa8e2a3f700 1 -- 172.21.15.16:0/327098539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8dc159c90 msgr2=0x7fa8dc15b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:26.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.721+0000 7fa8e2a3f700 1 --2- 172.21.15.16:0/327098539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8dc159c90 0x7fa8dc15b9f0 secure :-1 s=READY pgs=2213 cs=0 l=1 rev1=1 crypto rx=0x7fa8d000c4d0 tx=0x7fa8d000c7e0 comp rx=0 tx=0).stop 2023-10-16T13:44:26.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.721+0000 7fa8e2a3f700 1 -- 172.21.15.16:0/327098539 shutdown_connections 2023-10-16T13:44:26.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.721+0000 7fa8e2a3f700 1 --2- 172.21.15.16:0/327098539 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa8dc15a070 0x7fa8dc15bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:26.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.721+0000 7fa8e2a3f700 1 --2- 172.21.15.16:0/327098539 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa8dc0cb190 0x7fa8dc1f5c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:26.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.721+0000 7fa8e2a3f700 1 --2- 172.21.15.16:0/327098539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8dc159c90 0x7fa8dc15b9f0 unknown :-1 s=CLOSED pgs=2213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:26.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.721+0000 7fa8e2a3f700 1 -- 172.21.15.16:0/327098539 >> 172.21.15.16:0/327098539 conn(0x7fa8dc0cc5b0 msgr2=0x7fa8dc06a7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:26.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.721+0000 7fa8e2a3f700 1 -- 172.21.15.16:0/327098539 shutdown_connections 2023-10-16T13:44:26.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.721+0000 7fa8e2a3f700 1 -- 172.21.15.16:0/327098539 wait complete. 2023-10-16T13:44:26.725 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 95 2023-10-16T13:44:26.736 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:44:26.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.858+0000 7f488fa63700 1 Processor -- start 2023-10-16T13:44:26.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.858+0000 7f488fa63700 1 -- start start 2023-10-16T13:44:26.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.859+0000 7f488fa63700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f488815e0b0 0x7f48880aec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:26.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.859+0000 7f488fa63700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f488806b1f0 con 0x7f48880cb190 2023-10-16T13:44:26.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.859+0000 7f488fa63700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f488806b4b0 con 0x7f4888159c90 2023-10-16T13:44:26.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.859+0000 7f488fa63700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f488806b790 con 0x7f488815e0b0 2023-10-16T13:44:26.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.859+0000 7f488cffe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f48880cb190 0x7f48880ae520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:40536/0 (socket says 172.21.15.16:40536) 2023-10-16T13:44:26.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.859+0000 7f488cffe700 1 -- 172.21.15.16:0/1507874127 learned_addr learned my addr 172.21.15.16:0/1507874127 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:44:26.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.860+0000 7f488e000700 1 --2- 172.21.15.16:0/1507874127 >> v2:172.21.15.49:3300/0 conn(0x7f488815e0b0 0x7f48880aec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:26.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.860+0000 7f487bfff700 1 -- 172.21.15.16:0/1507874127 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3991477203 0 0) 0x7f488806b1f0 con 0x7f48880cb190 2023-10-16T13:44:26.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.860+0000 7f487bfff700 1 -- 172.21.15.16:0/1507874127 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48740046c0 con 0x7f48880cb190 2023-10-16T13:44:26.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.860+0000 7f488e000700 1 -- 172.21.15.16:0/1507874127 >> v1:172.21.15.49:6789/0 conn(0x7f4888159c90 legacy=0x7f488815d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:26.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.860+0000 7f487bfff700 1 -- 172.21.15.16:0/1507874127 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3629511845 0 0) 0x7f488806b4b0 con 0x7f4888159c90 2023-10-16T13:44:26.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.860+0000 7f488e000700 1 -- 172.21.15.16:0/1507874127 >> v1:172.21.15.16:6789/0 conn(0x7f48880cb190 legacy=0x7f48880ae520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:26.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.860+0000 7f488e000700 1 -- 172.21.15.16:0/1507874127 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f488806ba70 con 0x7f488815e0b0 2023-10-16T13:44:26.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.860+0000 7f488e000700 1 --2- 172.21.15.16:0/1507874127 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f488815e0b0 0x7f48880aec30 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f4880009d90 tx=0x7f488000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b826bf150f03697 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:26.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.860+0000 7f487bfff700 1 -- 172.21.15.16:0/1507874127 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f488000e070 con 0x7f488815e0b0 2023-10-16T13:44:26.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.860+0000 7f487bfff700 1 -- 172.21.15.16:0/1507874127 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f488000bb20 con 0x7f488815e0b0 2023-10-16T13:44:26.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.860+0000 7f487bfff700 1 -- 172.21.15.16:0/1507874127 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f488000bed0 con 0x7f488815e0b0 2023-10-16T13:44:26.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.861+0000 7f488fa63700 1 -- 172.21.15.16:0/1507874127 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f488815e0b0 msgr2=0x7f48880aec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:26.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.861+0000 7f488fa63700 1 --2- 172.21.15.16:0/1507874127 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f488815e0b0 0x7f48880aec30 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f4880009d90 tx=0x7f488000b330 comp rx=0 tx=0).stop 2023-10-16T13:44:26.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.861+0000 7f488fa63700 1 -- 172.21.15.16:0/1507874127 shutdown_connections 2023-10-16T13:44:26.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.861+0000 7f488fa63700 1 --2- 172.21.15.16:0/1507874127 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f488815e0b0 0x7f48880aec30 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:26.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.861+0000 7f488fa63700 1 -- 172.21.15.16:0/1507874127 >> 172.21.15.16:0/1507874127 conn(0x7f48880cc5b0 msgr2=0x7f48880cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:26.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.861+0000 7f488fa63700 1 -- 172.21.15.16:0/1507874127 shutdown_connections 2023-10-16T13:44:26.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.861+0000 7f488fa63700 1 -- 172.21.15.16:0/1507874127 wait complete. 2023-10-16T13:44:26.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.862+0000 7f488fa63700 1 Processor -- start 2023-10-16T13:44:26.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.862+0000 7f488fa63700 1 -- start start 2023-10-16T13:44:26.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.863+0000 7f488fa63700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48880cb190 0x7f4888155cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:26.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.863+0000 7f488fa63700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4888159c90 0x7f488814e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:26.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.863+0000 7f488fa63700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f488815e0b0 0x7f488806a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:26.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.863+0000 7f488fa63700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f488814f840 con 0x7f48880cb190 2023-10-16T13:44:26.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.863+0000 7f488fa63700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4888156390 con 0x7f4888159c90 2023-10-16T13:44:26.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.863+0000 7f488fa63700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f488814ecf0 con 0x7f488815e0b0 2023-10-16T13:44:26.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.863+0000 7f488d7ff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48880cb190 0x7f4888155cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:26.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.863+0000 7f488d7ff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48880cb190 0x7f4888155cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51266/0 (socket says 172.21.15.16:51266) 2023-10-16T13:44:26.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.863+0000 7f488d7ff700 1 -- 172.21.15.16:0/2462383773 learned_addr learned my addr 172.21.15.16:0/2462383773 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:26.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.863+0000 7f488cffe700 1 --2- 172.21.15.16:0/2462383773 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4888159c90 0x7f488814e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:26.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.863+0000 7f488e000700 1 --2- 172.21.15.16:0/2462383773 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f488815e0b0 0x7f488806a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:26.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.864+0000 7f488d7ff700 1 -- 172.21.15.16:0/2462383773 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f488815e0b0 msgr2=0x7f488806a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:26.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.864+0000 7f488d7ff700 1 --2- 172.21.15.16:0/2462383773 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f488815e0b0 0x7f488806a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:26.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.864+0000 7f488d7ff700 1 -- 172.21.15.16:0/2462383773 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4888159c90 msgr2=0x7f488814e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:26.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.864+0000 7f488d7ff700 1 --2- 172.21.15.16:0/2462383773 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4888159c90 0x7f488814e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:26.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.864+0000 7f488d7ff700 1 -- 172.21.15.16:0/2462383773 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4880009a70 con 0x7f48880cb190 2023-10-16T13:44:26.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.864+0000 7f488d7ff700 1 --2- 172.21.15.16:0/2462383773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48880cb190 0x7f4888155cb0 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 crypto rx=0x7f48840011e0 tx=0x7f488400ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:26.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.864+0000 7f4879ffb700 1 -- 172.21.15.16:0/2462383773 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48840054e0 con 0x7f48880cb190 2023-10-16T13:44:26.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.864+0000 7f4879ffb700 1 -- 172.21.15.16:0/2462383773 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4884003680 con 0x7f48880cb190 2023-10-16T13:44:26.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.864+0000 7f4879ffb700 1 -- 172.21.15.16:0/2462383773 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4884003a30 con 0x7f48880cb190 2023-10-16T13:44:26.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.864+0000 7f488fa63700 1 -- 172.21.15.16:0/2462383773 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48880af2f0 con 0x7f48880cb190 2023-10-16T13:44:26.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.864+0000 7f488fa63700 1 -- 172.21.15.16:0/2462383773 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f488815c690 con 0x7f48880cb190 2023-10-16T13:44:26.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.865+0000 7f488fa63700 1 -- 172.21.15.16:0/2462383773 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f488804ebb0 con 0x7f48880cb190 2023-10-16T13:44:26.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.865+0000 7f4879ffb700 1 -- 172.21.15.16:0/2462383773 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 123) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f4884003b90 con 0x7f48880cb190 2023-10-16T13:44:26.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.866+0000 7f4879ffb700 1 -- 172.21.15.16:0/2462383773 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 7189+0+0 (secure 0 0 0) 0x7f4884088a80 con 0x7f48880cb190 2023-10-16T13:44:26.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:26.870+0000 7f4879ffb700 1 -- 172.21.15.16:0/2462383773 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f48840539b0 con 0x7f48880cb190 2023-10-16T13:44:27.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.076+0000 7f488fa63700 1 -- 172.21.15.16:0/2462383773 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f48880ce410 con 0x7f48880cb190 2023-10-16T13:44:27.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.078+0000 7f4879ffb700 1 -- 172.21.15.16:0/2462383773 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v195) v1 ==== 74+0+18334 (secure 0 0 0) 0x7f4884017090 con 0x7f48880cb190 2023-10-16T13:44:27.080 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:44:27.081 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":195,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:44:26.409015+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2023-10-16T13:28:17.549815+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"189","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2023-10-16T13:28:18.493327+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"194","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":33,"snap_epoch":194,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/241231349":"2023-10-17T13:44:26.408978+0000","172.21.15.16:6833/371380443":"2023-10-17T13:44:26.408978+0000","172.21.15.16:6832/371380443":"2023-10-17T13:44:26.408978+0000","172.21.15.16:0/1071548593":"2023-10-17T13:44:26.408978+0000","172.21.15.16:0/1451116395":"2023-10-17T13:44:26.408978+0000","172.21.15.16:0/1245782052":"2023-10-17T13:44:26.408978+0000","172.21.15.16:6833/3952564978":"2023-10-17T13:39:22.557973+0000","172.21.15.16:6832/3952564978":"2023-10-17T13:39:22.557973+0000","172.21.15.16:0/2554840790":"2023-10-17T13:39:22.557973+0000","172.21.15.16:0/360056771":"2023-10-17T13:39:22.557973+0000","172.21.15.16:0/3520685377":"2023-10-17T13:39:22.557973+0000","172.21.15.16:0/1571009468":"2023-10-17T13:28:34.631949+0000","172.21.15.16:0/810487160":"2023-10-17T13:44:26.408978+0000","172.21.15.16:0/3374696233":"2023-10-17T13:44:26.408978+0000","172.21.15.16:0/678119374":"2023-10-17T13:28:34.631949+0000","172.21.15.16:0/3920247727":"2023-10-17T13:28:34.631949+0000","172.21.15.16:0/857902049":"2023-10-17T13:39:22.557973+0000","172.21.15.16:6833/4211329181":"2023-10-17T13:28:34.631949+0000","172.21.15.16:6832/4211329181":"2023-10-17T13:28:34.631949+0000","172.21.15.16:0/1548506671":"2023-10-17T13:39:22.557973+0000","172.21.15.16:0/2852193608":"2023-10-17T13:28:34.631949+0000","172.21.15.16:0/1641955134":"2023-10-17T13:39:22.557973+0000","172.21.15.16:0/471630266":"2023-10-17T13:28:34.631949+0000","172.21.15.16:0/3153082432":"2023-10-17T13:28:34.631949+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:44:27.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.080+0000 7f488fa63700 1 -- 172.21.15.16:0/2462383773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48880cb190 msgr2=0x7f4888155cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:27.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.080+0000 7f488fa63700 1 --2- 172.21.15.16:0/2462383773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48880cb190 0x7f4888155cb0 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 crypto rx=0x7f48840011e0 tx=0x7f488400ca30 comp rx=0 tx=0).stop 2023-10-16T13:44:27.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.080+0000 7f488fa63700 1 -- 172.21.15.16:0/2462383773 shutdown_connections 2023-10-16T13:44:27.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.080+0000 7f488fa63700 1 --2- 172.21.15.16:0/2462383773 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f488815e0b0 0x7f488806a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:27.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.080+0000 7f488fa63700 1 --2- 172.21.15.16:0/2462383773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48880cb190 0x7f4888155cb0 unknown :-1 s=CLOSED pgs=2219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:27.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.080+0000 7f488fa63700 1 --2- 172.21.15.16:0/2462383773 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4888159c90 0x7f488814e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:27.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.080+0000 7f488fa63700 1 -- 172.21.15.16:0/2462383773 >> 172.21.15.16:0/2462383773 conn(0x7f48880cc5b0 msgr2=0x7f48880c2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:27.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.080+0000 7f488fa63700 1 -- 172.21.15.16:0/2462383773 shutdown_connections 2023-10-16T13:44:27.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.080+0000 7f488fa63700 1 -- 172.21.15.16:0/2462383773 wait complete. 2023-10-16T13:44:27.095 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-16T13:44:27.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.219+0000 7f0f75874700 1 Processor -- start 2023-10-16T13:44:27.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.219+0000 7f0f75874700 1 -- start start 2023-10-16T13:44:27.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.220+0000 7f0f75874700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0f7015d750 0x7f0f700cf920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:27.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.220+0000 7f0f75874700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f700d0070 con 0x7f0f7015db30 2023-10-16T13:44:27.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.220+0000 7f0f75874700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f700d0350 con 0x7f0f70161490 2023-10-16T13:44:27.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.220+0000 7f0f75874700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f700cfe60 con 0x7f0f7015d750 2023-10-16T13:44:27.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.220+0000 7f0f6effd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0f7015d750 0x7f0f700cf920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:27.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.220+0000 7f0f6e7fc700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0f7015db30 0x7f0f7006c960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:40546/0 (socket says 172.21.15.16:40546) 2023-10-16T13:44:27.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.220+0000 7f0f6e7fc700 1 -- 172.21.15.16:0/1962545896 learned_addr learned my addr 172.21.15.16:0/1962545896 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:44:27.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.221+0000 7f0f6effd700 1 -- 172.21.15.16:0/1962545896 >> v1:172.21.15.49:6789/0 conn(0x7f0f70161490 legacy=0x7f0f70161870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:27.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.221+0000 7f0f6effd700 1 -- 172.21.15.16:0/1962545896 >> v1:172.21.15.16:6789/0 conn(0x7f0f7015db30 legacy=0x7f0f7006c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:27.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.221+0000 7f0f6effd700 1 -- 172.21.15.16:0/1962545896 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f700d0630 con 0x7f0f7015d750 2023-10-16T13:44:27.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.221+0000 7f0f6dffb700 1 -- 172.21.15.16:0/1962545896 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3182156628 0 0) 0x7f0f700d0070 con 0x7f0f7015db30 2023-10-16T13:44:27.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.221+0000 7f0f6effd700 1 --2- 172.21.15.16:0/1962545896 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f7015d750 0x7f0f700cf920 secure :-1 s=READY pgs=2221 cs=0 l=1 rev1=1 crypto rx=0x7f0f64009820 tx=0x7f0f64009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24abf67a18d6562e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:27.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.221+0000 7f0f6dffb700 1 -- 172.21.15.16:0/1962545896 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f64005e30 con 0x7f0f7015d750 2023-10-16T13:44:27.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.221+0000 7f0f6dffb700 1 -- 172.21.15.16:0/1962545896 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f64004030 con 0x7f0f7015d750 2023-10-16T13:44:27.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.221+0000 7f0f6dffb700 1 -- 172.21.15.16:0/1962545896 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f640043c0 con 0x7f0f7015d750 2023-10-16T13:44:27.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.222+0000 7f0f75874700 1 -- 172.21.15.16:0/1962545896 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f7015d750 msgr2=0x7f0f700cf920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:27.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.222+0000 7f0f75874700 1 --2- 172.21.15.16:0/1962545896 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f7015d750 0x7f0f700cf920 secure :-1 s=READY pgs=2221 cs=0 l=1 rev1=1 crypto rx=0x7f0f64009820 tx=0x7f0f64009b30 comp rx=0 tx=0).stop 2023-10-16T13:44:27.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.222+0000 7f0f75874700 1 -- 172.21.15.16:0/1962545896 shutdown_connections 2023-10-16T13:44:27.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.222+0000 7f0f75874700 1 --2- 172.21.15.16:0/1962545896 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f7015d750 0x7f0f700cf920 unknown :-1 s=CLOSED pgs=2221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:27.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.222+0000 7f0f75874700 1 -- 172.21.15.16:0/1962545896 >> 172.21.15.16:0/1962545896 conn(0x7f0f700d14b0 msgr2=0x7f0f700d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:27.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.222+0000 7f0f75874700 1 -- 172.21.15.16:0/1962545896 shutdown_connections 2023-10-16T13:44:27.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.222+0000 7f0f75874700 1 -- 172.21.15.16:0/1962545896 wait complete. 2023-10-16T13:44:27.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.223+0000 7f0f75874700 1 Processor -- start 2023-10-16T13:44:27.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.223+0000 7f0f75874700 1 -- start start 2023-10-16T13:44:27.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.224+0000 7f0f75874700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f7015d750 0x7f0f700c7c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:27.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.224+0000 7f0f75874700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f7015db30 0x7f0f7009eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:27.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.224+0000 7f0f75874700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f70161490 0x7f0f700b4d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:27.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.224+0000 7f0f75874700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f700c6cb0 con 0x7f0f7015db30 2023-10-16T13:44:27.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.224+0000 7f0f75874700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f700c77f0 con 0x7f0f70161490 2023-10-16T13:44:27.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.224+0000 7f0f75874700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f700c6ec0 con 0x7f0f7015d750 2023-10-16T13:44:27.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.224+0000 7f0f6e7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f7015db30 0x7f0f7009eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:27.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.224+0000 7f0f6e7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f7015db30 0x7f0f7009eb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51278/0 (socket says 172.21.15.16:51278) 2023-10-16T13:44:27.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.224+0000 7f0f6e7fc700 1 -- 172.21.15.16:0/1724008909 learned_addr learned my addr 172.21.15.16:0/1724008909 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:27.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.224+0000 7f0f6effd700 1 --2- 172.21.15.16:0/1724008909 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f7015d750 0x7f0f700c7c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:27.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.224+0000 7f0f6f7fe700 1 --2- 172.21.15.16:0/1724008909 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f70161490 0x7f0f700b4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:27.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.224+0000 7f0f6e7fc700 1 -- 172.21.15.16:0/1724008909 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f7015d750 msgr2=0x7f0f700c7c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:27.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.224+0000 7f0f6e7fc700 1 --2- 172.21.15.16:0/1724008909 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f7015d750 0x7f0f700c7c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:27.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.224+0000 7f0f6e7fc700 1 -- 172.21.15.16:0/1724008909 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f70161490 msgr2=0x7f0f700b4d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:27.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.224+0000 7f0f6e7fc700 1 --2- 172.21.15.16:0/1724008909 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f70161490 0x7f0f700b4d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:27.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.224+0000 7f0f6e7fc700 1 -- 172.21.15.16:0/1724008909 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f64009430 con 0x7f0f7015db30 2023-10-16T13:44:27.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.224+0000 7f0f6f7fe700 1 --2- 172.21.15.16:0/1724008909 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f70161490 0x7f0f700b4d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:44:27.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.224+0000 7f0f6e7fc700 1 --2- 172.21.15.16:0/1724008909 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f7015db30 0x7f0f7009eb10 secure :-1 s=READY pgs=2222 cs=0 l=1 rev1=1 crypto rx=0x7f0f6000c540 tx=0x7f0f6000c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:27.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.225+0000 7f0f57fff700 1 -- 172.21.15.16:0/1724008909 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f60004c80 con 0x7f0f7015db30 2023-10-16T13:44:27.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.225+0000 7f0f57fff700 1 -- 172.21.15.16:0/1724008909 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f60004de0 con 0x7f0f7015db30 2023-10-16T13:44:27.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.225+0000 7f0f75874700 1 -- 172.21.15.16:0/1724008909 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f700c71a0 con 0x7f0f7015db30 2023-10-16T13:44:27.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.225+0000 7f0f57fff700 1 -- 172.21.15.16:0/1724008909 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f60007310 con 0x7f0f7015db30 2023-10-16T13:44:27.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.225+0000 7f0f75874700 1 -- 172.21.15.16:0/1724008909 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f7009f050 con 0x7f0f7015db30 2023-10-16T13:44:27.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.226+0000 7f0f75874700 1 -- 172.21.15.16:0/1724008909 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f7004ebb0 con 0x7f0f7015db30 2023-10-16T13:44:27.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.226+0000 7f0f57fff700 1 -- 172.21.15.16:0/1724008909 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 123) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f0f60007470 con 0x7f0f7015db30 2023-10-16T13:44:27.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.227+0000 7f0f57fff700 1 -- 172.21.15.16:0/1724008909 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 7189+0+0 (secure 0 0 0) 0x7f0f600893d0 con 0x7f0f7015db30 2023-10-16T13:44:27.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.231+0000 7f0f57fff700 1 -- 172.21.15.16:0/1724008909 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0f60054300 con 0x7f0f7015db30 2023-10-16T13:44:27.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.417+0000 7f0f57fff700 1 -- 172.21.15.16:0/1724008909 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 124) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f0f60053db0 con 0x7f0f7015db30 2023-10-16T13:44:27.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.418+0000 7f0f57fff700 1 --2- 172.21.15.16:0/1724008909 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f0f5c071060 0x7f0f5c073450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:27.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.419+0000 7f0f6effd700 1 --2- 172.21.15.16:0/1724008909 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f0f5c071060 0x7f0f5c073450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:27.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.419+0000 7f0f6effd700 1 --2- 172.21.15.16:0/1724008909 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f0f5c071060 0x7f0f5c073450 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f0f64014040 tx=0x7f0f64008070 comp rx=0 tx=0).ready entity=mgr.10665 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:27.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.488+0000 7f0f75874700 1 -- 172.21.15.16:0/1724008909 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f0f700c7930 con 0x7f0f7015db30 2023-10-16T13:44:27.491 INFO:tasks.ceph.mon.a.smithi016.stderr:2023-10-16T13:44:27.489+0000 7f9fc8535700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-16T13:44:27.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.490+0000 7f0f57fff700 1 -- 172.21.15.16:0/1724008909 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v96) v1 ==== 161+0+0 (secure 0 0 0) 0x7f0f60073ec0 con 0x7f0f7015db30 2023-10-16T13:44:27.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.494+0000 7f0f75874700 1 -- 172.21.15.16:0/1724008909 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f0f5c071060 msgr2=0x7f0f5c073450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:27.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.494+0000 7f0f75874700 1 --2- 172.21.15.16:0/1724008909 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f0f5c071060 0x7f0f5c073450 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f0f64014040 tx=0x7f0f64008070 comp rx=0 tx=0).stop 2023-10-16T13:44:27.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.494+0000 7f0f75874700 1 -- 172.21.15.16:0/1724008909 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f7015db30 msgr2=0x7f0f7009eb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:27.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.494+0000 7f0f75874700 1 --2- 172.21.15.16:0/1724008909 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f7015db30 0x7f0f7009eb10 secure :-1 s=READY pgs=2222 cs=0 l=1 rev1=1 crypto rx=0x7f0f6000c540 tx=0x7f0f6000c900 comp rx=0 tx=0).stop 2023-10-16T13:44:27.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.494+0000 7f0f75874700 1 -- 172.21.15.16:0/1724008909 shutdown_connections 2023-10-16T13:44:27.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.494+0000 7f0f75874700 1 --2- 172.21.15.16:0/1724008909 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f0f5c071060 0x7f0f5c073450 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:27.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.494+0000 7f0f75874700 1 --2- 172.21.15.16:0/1724008909 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f70161490 0x7f0f700b4d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:27.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.494+0000 7f0f75874700 1 --2- 172.21.15.16:0/1724008909 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f7015d750 0x7f0f700c7c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:27.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.494+0000 7f0f75874700 1 --2- 172.21.15.16:0/1724008909 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f7015db30 0x7f0f7009eb10 unknown :-1 s=CLOSED pgs=2222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:27.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.494+0000 7f0f75874700 1 -- 172.21.15.16:0/1724008909 >> 172.21.15.16:0/1724008909 conn(0x7f0f700d14b0 msgr2=0x7f0f7006f6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:27.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.494+0000 7f0f75874700 1 -- 172.21.15.16:0/1724008909 shutdown_connections 2023-10-16T13:44:27.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.494+0000 7f0f75874700 1 -- 172.21.15.16:0/1724008909 wait complete. 2023-10-16T13:44:27.500 INFO:teuthology.orchestra.run.smithi016.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-16T13:44:27.510 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-16T13:44:27.533 INFO:tasks.ceph.mds.c.smithi016.stdout:starting mds.c at 2023-10-16T13:44:27.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.637+0000 7f08e6e04700 1 Processor -- start 2023-10-16T13:44:27.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.637+0000 7f08e6e04700 1 -- start start 2023-10-16T13:44:27.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.638+0000 7f08e6e04700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f08e0160fe0 0x7f08e00ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:27.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.638+0000 7f08e6e04700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f08e006da30 0x7f08e006a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:27.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.638+0000 7f08e6e04700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08e01613c0 con 0x7f08e006ad20 2023-10-16T13:44:27.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.638+0000 7f08e6e04700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08e00cf680 con 0x7f08e0160fe0 2023-10-16T13:44:27.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.638+0000 7f08e6e04700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08e006b170 con 0x7f08e006da30 2023-10-16T13:44:27.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.638+0000 7f08e4ba0700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f08e0160fe0 0x7f08e00ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:27.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.638+0000 7f08e53a1700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f08e006ad20 0x7f08e0071a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:40596/0 (socket says 172.21.15.16:40596) 2023-10-16T13:44:27.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.638+0000 7f08e53a1700 1 -- 172.21.15.16:0/87603978 learned_addr learned my addr 172.21.15.16:0/87603978 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:44:27.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.638+0000 7f08dffff700 1 --2- 172.21.15.16:0/87603978 >> v2:172.21.15.49:3300/0 conn(0x7f08e006da30 0x7f08e006a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:27.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.638+0000 7f08e4ba0700 1 -- 172.21.15.16:0/87603978 >> v2:172.21.15.49:3300/0 conn(0x7f08e006da30 msgr2=0x7f08e006a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:27.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.638+0000 7f08e4ba0700 1 --2- 172.21.15.16:0/87603978 >> v2:172.21.15.49:3300/0 conn(0x7f08e006da30 0x7f08e006a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:27.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.638+0000 7f08e4ba0700 1 -- 172.21.15.16:0/87603978 >> v1:172.21.15.16:6789/0 conn(0x7f08e006ad20 legacy=0x7f08e0071a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:27.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.638+0000 7f08e4ba0700 1 -- 172.21.15.16:0/87603978 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08e00adb00 con 0x7f08e0160fe0 2023-10-16T13:44:27.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.638+0000 7f08df7fe700 1 -- 172.21.15.16:0/87603978 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1908985881 0 0) 0x7f08d8001490 con 0x7f08e006ad20 2023-10-16T13:44:27.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.638+0000 7f08dffff700 1 --2- 172.21.15.16:0/87603978 >> v2:172.21.15.49:3300/0 conn(0x7f08e006da30 0x7f08e006a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:44:27.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.638+0000 7f08e4ba0700 1 --2- 172.21.15.16:0/87603978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08e0160fe0 0x7f08e00ac8d0 secure :-1 s=READY pgs=2227 cs=0 l=1 rev1=1 crypto rx=0x7f08e00cf330 tx=0x7f08d4009b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc910f7c7cdea239 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:27.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.638+0000 7f08df7fe700 1 -- 172.21.15.16:0/87603978 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d4005e30 con 0x7f08e0160fe0 2023-10-16T13:44:27.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.639+0000 7f08df7fe700 1 -- 172.21.15.16:0/87603978 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08d4006030 con 0x7f08e0160fe0 2023-10-16T13:44:27.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.639+0000 7f08df7fe700 1 -- 172.21.15.16:0/87603978 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d40063c0 con 0x7f08e0160fe0 2023-10-16T13:44:27.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.639+0000 7f08e6e04700 1 -- 172.21.15.16:0/87603978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08e0160fe0 msgr2=0x7f08e00ac8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:27.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.639+0000 7f08e6e04700 1 --2- 172.21.15.16:0/87603978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08e0160fe0 0x7f08e00ac8d0 secure :-1 s=READY pgs=2227 cs=0 l=1 rev1=1 crypto rx=0x7f08e00cf330 tx=0x7f08d4009b00 comp rx=0 tx=0).stop 2023-10-16T13:44:27.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.639+0000 7f08e6e04700 1 -- 172.21.15.16:0/87603978 shutdown_connections 2023-10-16T13:44:27.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.639+0000 7f08e6e04700 1 --2- 172.21.15.16:0/87603978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08e0160fe0 0x7f08e00ac8d0 unknown :-1 s=CLOSED pgs=2227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:27.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.639+0000 7f08e6e04700 1 --2- 172.21.15.16:0/87603978 >> v2:172.21.15.49:3300/0 conn(0x7f08e006da30 0x7f08e006a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:27.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.639+0000 7f08e6e04700 1 -- 172.21.15.16:0/87603978 >> 172.21.15.16:0/87603978 conn(0x7f08e00b1ed0 msgr2=0x7f08e00b22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:27.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.639+0000 7f08e6e04700 1 -- 172.21.15.16:0/87603978 shutdown_connections 2023-10-16T13:44:27.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.639+0000 7f08e6e04700 1 -- 172.21.15.16:0/87603978 wait complete. 2023-10-16T13:44:27.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.640+0000 7f08e6e04700 1 Processor -- start 2023-10-16T13:44:27.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.640+0000 7f08e6e04700 1 -- start start 2023-10-16T13:44:27.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.641+0000 7f08e6e04700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08e006ad20 0x7f08e00b5bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:27.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.641+0000 7f08e6e04700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f08e006da30 0x7f08e01e6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:27.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.641+0000 7f08e6e04700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f08e0160fe0 0x7f08e01e67b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:27.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.641+0000 7f08e6e04700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08e01e7e60 con 0x7f08e006ad20 2023-10-16T13:44:27.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.641+0000 7f08e6e04700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08e00b6100 con 0x7f08e006da30 2023-10-16T13:44:27.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.641+0000 7f08e6e04700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08e01e7320 con 0x7f08e0160fe0 2023-10-16T13:44:27.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.641+0000 7f08e4ba0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08e006ad20 0x7f08e00b5bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:27.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.641+0000 7f08e4ba0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08e006ad20 0x7f08e00b5bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51324/0 (socket says 172.21.15.16:51324) 2023-10-16T13:44:27.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.641+0000 7f08e4ba0700 1 -- 172.21.15.16:0/524479926 learned_addr learned my addr 172.21.15.16:0/524479926 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:27.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.642+0000 7f08e53a1700 1 --2- 172.21.15.16:0/524479926 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f08e0160fe0 0x7f08e01e67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:27.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.642+0000 7f08e4ba0700 1 -- 172.21.15.16:0/524479926 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f08e0160fe0 msgr2=0x7f08e01e67b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:27.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.642+0000 7f08e4ba0700 1 --2- 172.21.15.16:0/524479926 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f08e0160fe0 0x7f08e01e67b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:27.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.642+0000 7f08e4ba0700 1 -- 172.21.15.16:0/524479926 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f08e006da30 msgr2=0x7f08e01e6270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:27.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.642+0000 7f08dffff700 1 --2- 172.21.15.16:0/524479926 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f08e006da30 0x7f08e01e6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:27.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.642+0000 7f08e4ba0700 1 --2- 172.21.15.16:0/524479926 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f08e006da30 0x7f08e01e6270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:27.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.642+0000 7f08e4ba0700 1 -- 172.21.15.16:0/524479926 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08d4009430 con 0x7f08e006ad20 2023-10-16T13:44:27.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.642+0000 7f08dffff700 1 --2- 172.21.15.16:0/524479926 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f08e006da30 0x7f08e01e6270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:44:27.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.642+0000 7f08e4ba0700 1 --2- 172.21.15.16:0/524479926 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08e006ad20 0x7f08e00b5bc0 secure :-1 s=READY pgs=2228 cs=0 l=1 rev1=1 crypto rx=0x7f08d4000c00 tx=0x7f08d4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:27.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.642+0000 7f08dd7fa700 1 -- 172.21.15.16:0/524479926 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d4004240 con 0x7f08e006ad20 2023-10-16T13:44:27.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.642+0000 7f08dd7fa700 1 -- 172.21.15.16:0/524479926 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08d40043c0 con 0x7f08e006ad20 2023-10-16T13:44:27.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.642+0000 7f08e6e04700 1 -- 172.21.15.16:0/524479926 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08e01e75c0 con 0x7f08e006ad20 2023-10-16T13:44:27.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.642+0000 7f08dd7fa700 1 -- 172.21.15.16:0/524479926 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d40036a0 con 0x7f08e006ad20 2023-10-16T13:44:27.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.642+0000 7f08e6e04700 1 -- 172.21.15.16:0/524479926 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08e01e7b80 con 0x7f08e006ad20 2023-10-16T13:44:27.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.643+0000 7f08e6e04700 1 -- 172.21.15.16:0/524479926 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08e004ebb0 con 0x7f08e006ad20 2023-10-16T13:44:27.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.643+0000 7f08dd7fa700 1 -- 172.21.15.16:0/524479926 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 124) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f08d4022070 con 0x7f08e006ad20 2023-10-16T13:44:27.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.644+0000 7f08dd7fa700 1 --2- 172.21.15.16:0/524479926 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f08c406ad50 0x7f08c406d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:27.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.644+0000 7f08dd7fa700 1 -- 172.21.15.16:0/524479926 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 7275+0+0 (secure 0 0 0) 0x7f08d408b140 con 0x7f08e006ad20 2023-10-16T13:44:27.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.644+0000 7f08dffff700 1 --2- 172.21.15.16:0/524479926 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f08c406ad50 0x7f08c406d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:27.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.644+0000 7f08dffff700 1 --2- 172.21.15.16:0/524479926 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f08c406ad50 0x7f08c406d210 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f08e0154020 tx=0x7f08cc008040 comp rx=0 tx=0).ready entity=mgr.10665 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:27.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.649+0000 7f08dd7fa700 1 -- 172.21.15.16:0/524479926 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f08d4055f10 con 0x7f08e006ad20 2023-10-16T13:44:27.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:27.899+0000 7f08e6e04700 1 -- 172.21.15.16:0/524479926 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f08e00c6fb0 con 0x7f08e006ad20 2023-10-16T13:44:28.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:28.491+0000 7f08dd7fa700 1 -- 172.21.15.16:0/524479926 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v97) v1 ==== 104+0+0 (secure 0 0 0) 0x7f08d402a0a0 con 0x7f08e006ad20 2023-10-16T13:44:28.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:28.494+0000 7f08e6e04700 1 -- 172.21.15.16:0/524479926 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f08c406ad50 msgr2=0x7f08c406d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:28.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:28.494+0000 7f08e6e04700 1 --2- 172.21.15.16:0/524479926 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f08c406ad50 0x7f08c406d210 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f08e0154020 tx=0x7f08cc008040 comp rx=0 tx=0).stop 2023-10-16T13:44:28.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:28.494+0000 7f08e6e04700 1 -- 172.21.15.16:0/524479926 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08e006ad20 msgr2=0x7f08e00b5bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:28.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:28.494+0000 7f08e6e04700 1 --2- 172.21.15.16:0/524479926 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08e006ad20 0x7f08e00b5bc0 secure :-1 s=READY pgs=2228 cs=0 l=1 rev1=1 crypto rx=0x7f08d4000c00 tx=0x7f08d4004750 comp rx=0 tx=0).stop 2023-10-16T13:44:28.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:28.494+0000 7f08e6e04700 1 -- 172.21.15.16:0/524479926 shutdown_connections 2023-10-16T13:44:28.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:28.494+0000 7f08e6e04700 1 --2- 172.21.15.16:0/524479926 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f08c406ad50 0x7f08c406d210 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:28.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:28.494+0000 7f08e6e04700 1 --2- 172.21.15.16:0/524479926 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f08e0160fe0 0x7f08e01e67b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:28.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:28.494+0000 7f08e6e04700 1 --2- 172.21.15.16:0/524479926 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08e006ad20 0x7f08e00b5bc0 unknown :-1 s=CLOSED pgs=2228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:28.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:28.494+0000 7f08e6e04700 1 --2- 172.21.15.16:0/524479926 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f08e006da30 0x7f08e01e6270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:28.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:28.494+0000 7f08e6e04700 1 -- 172.21.15.16:0/524479926 >> 172.21.15.16:0/524479926 conn(0x7f08e00b1ed0 msgr2=0x7f08e00b22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:28.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:28.494+0000 7f08e6e04700 1 -- 172.21.15.16:0/524479926 shutdown_connections 2023-10-16T13:44:28.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:28.494+0000 7f08e6e04700 1 -- 172.21.15.16:0/524479926 wait complete. 2023-10-16T13:44:28.511 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-16T13:44:28.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:28.681+0000 7f1209538700 1 Processor -- start 2023-10-16T13:44:28.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:28.681+0000 7f1209538700 1 -- start start 2023-10-16T13:44:28.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:28.682+0000 7f1209538700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f12040ce260 0x7f12040ce6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:28.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:28.682+0000 7f1209538700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12040b91b0 con 0x7f12040d1740 2023-10-16T13:44:28.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:28.682+0000 7f1209538700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12040b7d00 con 0x7f120415d7a0 2023-10-16T13:44:28.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:28.682+0000 7f1209538700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12040b8980 con 0x7f12040ce260 2023-10-16T13:44:28.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:28.682+0000 7f12027fc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f12040ce260 0x7f12040ce6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:28.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:28.682+0000 7f12027fc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f12040ce260 0x7f12040ce6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51354/0 (socket says 172.21.15.16:51354) 2023-10-16T13:44:28.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:28.682+0000 7f12027fc700 1 -- 172.21.15.16:0/1218645772 learned_addr learned my addr 172.21.15.16:0/1218645772 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:28.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:28.682+0000 7f12027fc700 1 -- 172.21.15.16:0/1218645772 >> v1:172.21.15.49:6789/0 conn(0x7f120415d7a0 legacy=0x7f12040b7240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:28.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:28.682+0000 7f12027fc700 1 -- 172.21.15.16:0/1218645772 >> v1:172.21.15.16:6789/0 conn(0x7f12040d1740 legacy=0x7f12040d1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:28.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:28.682+0000 7f12027fc700 1 -- 172.21.15.16:0/1218645772 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12040b7fe0 con 0x7f12040ce260 2023-10-16T13:44:28.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:28.682+0000 7f12027fc700 1 --2- 172.21.15.16:0/1218645772 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12040ce260 0x7f12040ce6e0 secure :-1 s=READY pgs=2232 cs=0 l=1 rev1=1 crypto rx=0x7f11f40098f0 tx=0x7f11f4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89dddcbae217d804 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:28.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:28.683+0000 7f1201ffb700 1 -- 172.21.15.16:0/1218645772 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f400b8a0 con 0x7f12040ce260 2023-10-16T13:44:28.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:28.683+0000 7f1201ffb700 1 -- 172.21.15.16:0/1218645772 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11f400ba00 con 0x7f12040ce260 2023-10-16T13:44:28.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:28.683+0000 7f1201ffb700 1 -- 172.21.15.16:0/1218645772 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f400bd90 con 0x7f12040ce260 2023-10-16T13:44:28.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:28.683+0000 7f1209538700 1 -- 172.21.15.16:0/1218645772 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12040ce260 msgr2=0x7f12040ce6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:28.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:28.683+0000 7f1209538700 1 --2- 172.21.15.16:0/1218645772 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12040ce260 0x7f12040ce6e0 secure :-1 s=READY pgs=2232 cs=0 l=1 rev1=1 crypto rx=0x7f11f40098f0 tx=0x7f11f4009c00 comp rx=0 tx=0).stop 2023-10-16T13:44:28.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:28.683+0000 7f1209538700 1 -- 172.21.15.16:0/1218645772 shutdown_connections 2023-10-16T13:44:28.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:28.683+0000 7f1209538700 1 --2- 172.21.15.16:0/1218645772 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12040ce260 0x7f12040ce6e0 unknown :-1 s=CLOSED pgs=2232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:28.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:28.683+0000 7f1209538700 1 -- 172.21.15.16:0/1218645772 >> 172.21.15.16:0/1218645772 conn(0x7f1204072330 msgr2=0x7f1204072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:28.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:28.683+0000 7f1209538700 1 -- 172.21.15.16:0/1218645772 shutdown_connections 2023-10-16T13:44:28.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:28.683+0000 7f1209538700 1 -- 172.21.15.16:0/1218645772 wait complete. 2023-10-16T13:44:28.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:28.685+0000 7f1209538700 1 Processor -- start 2023-10-16T13:44:28.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:28.685+0000 7f1209538700 1 -- start start 2023-10-16T13:44:28.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:28.685+0000 7f1209538700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f12040ce260 0x7f12040d3b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:28.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:28.685+0000 7f1209538700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f12040d1740 0x7f12041f2560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:28.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:28.685+0000 7f1209538700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f120415d7a0 0x7f120415fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:28.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:28.685+0000 7f1209538700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12041524f0 con 0x7f120415d7a0 2023-10-16T13:44:28.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:28.685+0000 7f1209538700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12040c9b70 con 0x7f12040d1740 2023-10-16T13:44:28.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:28.685+0000 7f1209538700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12040c9e50 con 0x7f12040ce260 2023-10-16T13:44:28.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:28.685+0000 7f12037fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f120415d7a0 0x7f120415fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:28.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:28.685+0000 7f12037fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f120415d7a0 0x7f120415fc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51362/0 (socket says 172.21.15.16:51362) 2023-10-16T13:44:28.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:28.685+0000 7f12037fe700 1 -- 172.21.15.16:0/3613064710 learned_addr learned my addr 172.21.15.16:0/3613064710 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:28.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:28.686+0000 7f12027fc700 1 --2- 172.21.15.16:0/3613064710 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f12040d1740 0x7f12041f2560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:28.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:28.686+0000 7f12037fe700 1 -- 172.21.15.16:0/3613064710 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f12040ce260 msgr2=0x7f12040d3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:28.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:28.686+0000 7f1202ffd700 1 --2- 172.21.15.16:0/3613064710 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f12040ce260 0x7f12040d3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:28.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:28.686+0000 7f12037fe700 1 --2- 172.21.15.16:0/3613064710 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f12040ce260 0x7f12040d3b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:28.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:28.686+0000 7f12037fe700 1 -- 172.21.15.16:0/3613064710 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f12040d1740 msgr2=0x7f12041f2560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:28.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:28.686+0000 7f12037fe700 1 --2- 172.21.15.16:0/3613064710 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f12040d1740 0x7f12041f2560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:28.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:28.686+0000 7f12037fe700 1 -- 172.21.15.16:0/3613064710 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11f4009550 con 0x7f120415d7a0 2023-10-16T13:44:28.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:28.686+0000 7f12027fc700 1 --2- 172.21.15.16:0/3613064710 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f12040d1740 0x7f12041f2560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:44:28.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:28.686+0000 7f1202ffd700 1 --2- 172.21.15.16:0/3613064710 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f12040ce260 0x7f12040d3b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:44:28.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:28.686+0000 7f12037fe700 1 --2- 172.21.15.16:0/3613064710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f120415d7a0 0x7f120415fc80 secure :-1 s=READY pgs=2233 cs=0 l=1 rev1=1 crypto rx=0x7f11fc00cc50 tx=0x7f11fc00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:28.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:28.686+0000 7f11eb7fe700 1 -- 172.21.15.16:0/3613064710 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11fc00ad80 con 0x7f120415d7a0 2023-10-16T13:44:28.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:28.686+0000 7f11eb7fe700 1 -- 172.21.15.16:0/3613064710 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11fc00ce50 con 0x7f120415d7a0 2023-10-16T13:44:28.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:28.686+0000 7f1209538700 1 -- 172.21.15.16:0/3613064710 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12040c6f90 con 0x7f120415d7a0 2023-10-16T13:44:28.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:28.686+0000 7f11eb7fe700 1 -- 172.21.15.16:0/3613064710 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11fc003c40 con 0x7f120415d7a0 2023-10-16T13:44:28.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:28.686+0000 7f1209538700 1 -- 172.21.15.16:0/3613064710 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12040c7550 con 0x7f120415d7a0 2023-10-16T13:44:28.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:28.687+0000 7f1209538700 1 -- 172.21.15.16:0/3613064710 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f120404ec20 con 0x7f120415d7a0 2023-10-16T13:44:28.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:28.688+0000 7f11eb7fe700 1 -- 172.21.15.16:0/3613064710 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 124) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f11fc0050d0 con 0x7f120415d7a0 2023-10-16T13:44:28.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:28.688+0000 7f11eb7fe700 1 --2- 172.21.15.16:0/3613064710 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f11f006ad50 0x7f11f006d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:28.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:28.688+0000 7f11eb7fe700 1 -- 172.21.15.16:0/3613064710 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 7275+0+0 (secure 0 0 0) 0x7f11fc089960 con 0x7f120415d7a0 2023-10-16T13:44:28.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:28.688+0000 7f1202ffd700 1 --2- 172.21.15.16:0/3613064710 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f11f006ad50 0x7f11f006d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:28.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:28.689+0000 7f1202ffd700 1 --2- 172.21.15.16:0/3613064710 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f11f006ad50 0x7f11f006d210 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f11f8000f60 tx=0x7f11f8009040 comp rx=0 tx=0).ready entity=mgr.10665 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:28.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:28.693+0000 7f11eb7fe700 1 -- 172.21.15.16:0/3613064710 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f11fc054730 con 0x7f120415d7a0 2023-10-16T13:44:28.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:28.896+0000 7f1209538700 1 -- 172.21.15.16:0/3613064710 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f1204160af0 con 0x7f120415d7a0 2023-10-16T13:44:29.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.424+0000 7f11eb7fe700 1 -- 172.21.15.16:0/3613064710 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 125) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f11fc05e070 con 0x7f120415d7a0 2023-10-16T13:44:29.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.734+0000 7f11eb7fe700 1 -- 172.21.15.16:0/3613064710 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v197) v1 ==== 181+0+0 (secure 0 0 0) 0x7f11fc0731d0 con 0x7f120415d7a0 2023-10-16T13:44:29.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.738+0000 7f1209538700 1 -- 172.21.15.16:0/3613064710 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f11f006ad50 msgr2=0x7f11f006d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:29.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.738+0000 7f1209538700 1 --2- 172.21.15.16:0/3613064710 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f11f006ad50 0x7f11f006d210 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f11f8000f60 tx=0x7f11f8009040 comp rx=0 tx=0).stop 2023-10-16T13:44:29.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.738+0000 7f1209538700 1 -- 172.21.15.16:0/3613064710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f120415d7a0 msgr2=0x7f120415fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:29.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.738+0000 7f1209538700 1 --2- 172.21.15.16:0/3613064710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f120415d7a0 0x7f120415fc80 secure :-1 s=READY pgs=2233 cs=0 l=1 rev1=1 crypto rx=0x7f11fc00cc50 tx=0x7f11fc00a340 comp rx=0 tx=0).stop 2023-10-16T13:44:29.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.738+0000 7f1209538700 1 -- 172.21.15.16:0/3613064710 shutdown_connections 2023-10-16T13:44:29.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.738+0000 7f1209538700 1 --2- 172.21.15.16:0/3613064710 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f11f006ad50 0x7f11f006d210 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:29.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.738+0000 7f1209538700 1 --2- 172.21.15.16:0/3613064710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f120415d7a0 0x7f120415fc80 unknown :-1 s=CLOSED pgs=2233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:29.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.738+0000 7f1209538700 1 --2- 172.21.15.16:0/3613064710 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f12040ce260 0x7f12040d3b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:29.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.738+0000 7f1209538700 1 --2- 172.21.15.16:0/3613064710 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f12040d1740 0x7f12041f2560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:29.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.738+0000 7f1209538700 1 -- 172.21.15.16:0/3613064710 >> 172.21.15.16:0/3613064710 conn(0x7f1204072330 msgr2=0x7f120415ee00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:29.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.738+0000 7f1209538700 1 -- 172.21.15.16:0/3613064710 shutdown_connections 2023-10-16T13:44:29.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.738+0000 7f1209538700 1 -- 172.21.15.16:0/3613064710 wait complete. 2023-10-16T13:44:29.743 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' removed 2023-10-16T13:44:29.757 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-16T13:44:29.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.891+0000 7f286d592700 1 Processor -- start 2023-10-16T13:44:29.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.891+0000 7f286d592700 1 -- start start 2023-10-16T13:44:29.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.891+0000 7f286d592700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f28680cb1f0 0x7f2868071fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:29.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.892+0000 7f286d592700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2868072640 con 0x7f2868151680 2023-10-16T13:44:29.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.892+0000 7f286d592700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2868072920 con 0x7f286806ccd0 2023-10-16T13:44:29.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.892+0000 7f286d592700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28680c97d0 con 0x7f28680cb1f0 2023-10-16T13:44:29.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.892+0000 7f2866ffd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2868151680 0x7f286806c5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:40640/0 (socket says 172.21.15.16:40640) 2023-10-16T13:44:29.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.892+0000 7f2866ffd700 1 -- 172.21.15.16:0/172453007 learned_addr learned my addr 172.21.15.16:0/172453007 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:44:29.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.892+0000 7f28677fe700 1 --2- 172.21.15.16:0/172453007 >> v2:172.21.15.49:3300/0 conn(0x7f28680cb1f0 0x7f2868071fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:29.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.892+0000 7f2865ffb700 1 -- 172.21.15.16:0/172453007 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2025723799 0 0) 0x7f2868072640 con 0x7f2868151680 2023-10-16T13:44:29.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.892+0000 7f2865ffb700 1 -- 172.21.15.16:0/172453007 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28500035b0 con 0x7f2868151680 2023-10-16T13:44:29.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.892+0000 7f2865ffb700 1 -- 172.21.15.16:0/172453007 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1690033383 0 0) 0x7f28500035b0 con 0x7f2868151680 2023-10-16T13:44:29.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.892+0000 7f2865ffb700 1 -- 172.21.15.16:0/172453007 >> v2:172.21.15.49:3300/0 conn(0x7f28680cb1f0 msgr2=0x7f2868071fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:29.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.892+0000 7f2865ffb700 1 --2- 172.21.15.16:0/172453007 >> v2:172.21.15.49:3300/0 conn(0x7f28680cb1f0 0x7f2868071fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:29.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.892+0000 7f2865ffb700 1 -- 172.21.15.16:0/172453007 >> v1:172.21.15.49:6789/0 conn(0x7f286806ccd0 legacy=0x7f28680718d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:29.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.892+0000 7f2865ffb700 1 -- 172.21.15.16:0/172453007 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28680c99e0 con 0x7f2868151680 2023-10-16T13:44:29.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.892+0000 7f2865ffb700 1 -- 172.21.15.16:0/172453007 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f2858001670 con 0x7f2868151680 2023-10-16T13:44:29.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.892+0000 7f2865ffb700 1 -- 172.21.15.16:0/172453007 >> v1:172.21.15.16:6789/0 conn(0x7f2868151680 legacy=0x7f286806c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:29.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.892+0000 7f2865ffb700 1 --2- 172.21.15.16:0/172453007 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2850003a00 0x7f2850003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:29.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.892+0000 7f2865ffb700 1 --2- 172.21.15.16:0/172453007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2850004850 0x7f2850004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:29.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.892+0000 7f2865ffb700 1 --2- 172.21.15.16:0/172453007 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2850005210 0x7f285000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:29.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.892+0000 7f2865ffb700 1 -- 172.21.15.16:0/172453007 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28680c97d0 con 0x7f2850004850 2023-10-16T13:44:29.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.892+0000 7f2865ffb700 1 -- 172.21.15.16:0/172453007 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f285000c1b0 con 0x7f2850003a00 2023-10-16T13:44:29.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.892+0000 7f2865ffb700 1 -- 172.21.15.16:0/172453007 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f285000c370 con 0x7f2850005210 2023-10-16T13:44:29.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.892+0000 7f28677fe700 1 --2- 172.21.15.16:0/172453007 >> v2:172.21.15.49:3300/0 conn(0x7f28680cb1f0 0x7f2868071fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:44:29.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.892+0000 7f28667fc700 1 --2- 172.21.15.16:0/172453007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2850004850 0x7f2850004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:29.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.893+0000 7f2866ffd700 1 --2- 172.21.15.16:0/172453007 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2850003a00 0x7f2850003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:29.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.893+0000 7f28677fe700 1 --2- 172.21.15.16:0/172453007 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2850005210 0x7f285000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:29.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.893+0000 7f28667fc700 1 -- 172.21.15.16:0/172453007 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2850005210 msgr2=0x7f285000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:29.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.893+0000 7f28667fc700 1 --2- 172.21.15.16:0/172453007 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2850005210 0x7f285000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:29.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.893+0000 7f28667fc700 1 -- 172.21.15.16:0/172453007 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2850003a00 msgr2=0x7f2850003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:29.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.893+0000 7f28667fc700 1 --2- 172.21.15.16:0/172453007 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2850003a00 0x7f2850003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:29.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.893+0000 7f28667fc700 1 -- 172.21.15.16:0/172453007 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28500035b0 con 0x7f2850004850 2023-10-16T13:44:29.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.893+0000 7f28677fe700 1 -- 172.21.15.16:0/172453007 reap_dead start 2023-10-16T13:44:29.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.893+0000 7f28667fc700 1 --2- 172.21.15.16:0/172453007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2850004850 0x7f2850004cd0 secure :-1 s=READY pgs=2235 cs=0 l=1 rev1=1 crypto rx=0x7f285c0017c0 tx=0x7f2850005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11cc27ab9d508dfb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:29.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.893+0000 7f2865ffb700 1 -- 172.21.15.16:0/172453007 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f285c0052d0 con 0x7f2850004850 2023-10-16T13:44:29.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.893+0000 7f2865ffb700 1 -- 172.21.15.16:0/172453007 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f285c005430 con 0x7f2850004850 2023-10-16T13:44:29.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.893+0000 7f2865ffb700 1 -- 172.21.15.16:0/172453007 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f285c0038c0 con 0x7f2850004850 2023-10-16T13:44:29.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.894+0000 7f286d592700 1 -- 172.21.15.16:0/172453007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2850004850 msgr2=0x7f2850004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:29.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.894+0000 7f286d592700 1 --2- 172.21.15.16:0/172453007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2850004850 0x7f2850004cd0 secure :-1 s=READY pgs=2235 cs=0 l=1 rev1=1 crypto rx=0x7f285c0017c0 tx=0x7f2850005880 comp rx=0 tx=0).stop 2023-10-16T13:44:29.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.894+0000 7f286d592700 1 -- 172.21.15.16:0/172453007 shutdown_connections 2023-10-16T13:44:29.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.894+0000 7f286d592700 1 --2- 172.21.15.16:0/172453007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2850004850 0x7f2850004cd0 unknown :-1 s=CLOSED pgs=2235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:29.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.894+0000 7f286d592700 1 -- 172.21.15.16:0/172453007 >> 172.21.15.16:0/172453007 conn(0x7f286806ae40 msgr2=0x7f28680b0cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:29.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.894+0000 7f286d592700 1 -- 172.21.15.16:0/172453007 shutdown_connections 2023-10-16T13:44:29.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.894+0000 7f286d592700 1 -- 172.21.15.16:0/172453007 wait complete. 2023-10-16T13:44:29.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.895+0000 7f286d592700 1 Processor -- start 2023-10-16T13:44:29.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.895+0000 7f286d592700 1 -- start start 2023-10-16T13:44:29.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.896+0000 7f286d592700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2850004850 0x7f286815d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:29.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.896+0000 7f286d592700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f28680c5260 0x7f2868143110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:29.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.896+0000 7f286d592700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f28680b0040 0x7f28680b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:29.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.896+0000 7f286d592700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28680b0a00 con 0x7f2850004850 2023-10-16T13:44:29.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.896+0000 7f286d592700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28680c6250 con 0x7f28680c5260 2023-10-16T13:44:29.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.896+0000 7f286d592700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28680c6810 con 0x7f28680b0040 2023-10-16T13:44:29.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.896+0000 7f2866ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2850004850 0x7f286815d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:29.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.896+0000 7f2866ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2850004850 0x7f286815d420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51378/0 (socket says 172.21.15.16:51378) 2023-10-16T13:44:29.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.896+0000 7f2866ffd700 1 -- 172.21.15.16:0/251877119 learned_addr learned my addr 172.21.15.16:0/251877119 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:29.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.896+0000 7f28667fc700 1 --2- 172.21.15.16:0/251877119 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f28680c5260 0x7f2868143110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:29.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.896+0000 7f28677fe700 1 --2- 172.21.15.16:0/251877119 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f28680b0040 0x7f28680b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:29.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.896+0000 7f2866ffd700 1 -- 172.21.15.16:0/251877119 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f28680b0040 msgr2=0x7f28680b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:29.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.896+0000 7f2866ffd700 1 --2- 172.21.15.16:0/251877119 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f28680b0040 0x7f28680b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:29.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.896+0000 7f2866ffd700 1 -- 172.21.15.16:0/251877119 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f28680c5260 msgr2=0x7f2868143110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:29.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.896+0000 7f2866ffd700 1 --2- 172.21.15.16:0/251877119 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f28680c5260 0x7f2868143110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:29.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.896+0000 7f2866ffd700 1 -- 172.21.15.16:0/251877119 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f286815dab0 con 0x7f2850004850 2023-10-16T13:44:29.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.896+0000 7f2866ffd700 1 --2- 172.21.15.16:0/251877119 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2850004850 0x7f286815d420 secure :-1 s=READY pgs=2236 cs=0 l=1 rev1=1 crypto rx=0x7f2858007320 tx=0x7f2858007630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:29.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.896+0000 7f284f7fe700 1 -- 172.21.15.16:0/251877119 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f285800f5e0 con 0x7f2850004850 2023-10-16T13:44:29.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.896+0000 7f284f7fe700 1 -- 172.21.15.16:0/251877119 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f285800f740 con 0x7f2850004850 2023-10-16T13:44:29.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.897+0000 7f286d592700 1 -- 172.21.15.16:0/251877119 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f286815dd90 con 0x7f2850004850 2023-10-16T13:44:29.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.897+0000 7f284f7fe700 1 -- 172.21.15.16:0/251877119 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f285800faf0 con 0x7f2850004850 2023-10-16T13:44:29.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.897+0000 7f286d592700 1 -- 172.21.15.16:0/251877119 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28680c56e0 con 0x7f2850004850 2023-10-16T13:44:29.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.898+0000 7f286d592700 1 -- 172.21.15.16:0/251877119 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28680c58a0 con 0x7f2850004850 2023-10-16T13:44:29.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.898+0000 7f284f7fe700 1 -- 172.21.15.16:0/251877119 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 125) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f285800fc50 con 0x7f2850004850 2023-10-16T13:44:29.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.898+0000 7f284f7fe700 1 --2- 172.21.15.16:0/251877119 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f285406af90 0x7f285406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:29.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.899+0000 7f284f7fe700 1 -- 172.21.15.16:0/251877119 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6837+0+0 (secure 0 0 0) 0x7f2858089450 con 0x7f2850004850 2023-10-16T13:44:29.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.899+0000 7f28667fc700 1 --2- 172.21.15.16:0/251877119 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f285406af90 0x7f285406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:29.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.899+0000 7f28667fc700 1 --2- 172.21.15.16:0/251877119 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f285406af90 0x7f285406d450 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f285c015040 tx=0x7f285c009030 comp rx=0 tx=0).ready entity=mgr.10665 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:29.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:29.903+0000 7f284f7fe700 1 -- 172.21.15.16:0/251877119 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f28580540a0 con 0x7f2850004850 2023-10-16T13:44:30.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:30.106+0000 7f286d592700 1 -- 172.21.15.16:0/251877119 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f286804ec20 con 0x7f2850004850 2023-10-16T13:44:30.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:30.744+0000 7f284f7fe700 1 -- 172.21.15.16:0/251877119 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v198) v1 ==== 169+0+0 (secure 0 0 0) 0x7f28580576c0 con 0x7f2850004850 2023-10-16T13:44:30.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:30.753+0000 7f286d592700 1 -- 172.21.15.16:0/251877119 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f285406af90 msgr2=0x7f285406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:30.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:30.753+0000 7f286d592700 1 --2- 172.21.15.16:0/251877119 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f285406af90 0x7f285406d450 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f285c015040 tx=0x7f285c009030 comp rx=0 tx=0).stop 2023-10-16T13:44:30.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:30.753+0000 7f286d592700 1 -- 172.21.15.16:0/251877119 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2850004850 msgr2=0x7f286815d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:30.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:30.753+0000 7f286d592700 1 --2- 172.21.15.16:0/251877119 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2850004850 0x7f286815d420 secure :-1 s=READY pgs=2236 cs=0 l=1 rev1=1 crypto rx=0x7f2858007320 tx=0x7f2858007630 comp rx=0 tx=0).stop 2023-10-16T13:44:30.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:30.756+0000 7f286d592700 1 -- 172.21.15.16:0/251877119 shutdown_connections 2023-10-16T13:44:30.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:30.756+0000 7f286d592700 1 --2- 172.21.15.16:0/251877119 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f285406af90 0x7f285406d450 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:30.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:30.756+0000 7f286d592700 1 --2- 172.21.15.16:0/251877119 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f28680b0040 0x7f28680b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:30.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:30.756+0000 7f286d592700 1 --2- 172.21.15.16:0/251877119 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2850004850 0x7f286815d420 secure :-1 s=CLOSED pgs=2236 cs=0 l=1 rev1=1 crypto rx=0x7f2858007320 tx=0x7f2858007630 comp rx=0 tx=0).stop 2023-10-16T13:44:30.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:30.756+0000 7f286d592700 1 --2- 172.21.15.16:0/251877119 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f28680c5260 0x7f2868143110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:30.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:30.756+0000 7f286d592700 1 -- 172.21.15.16:0/251877119 >> 172.21.15.16:0/251877119 conn(0x7f286806ae40 msgr2=0x7f28680acb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:30.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:30.757+0000 7f286d592700 1 -- 172.21.15.16:0/251877119 shutdown_connections 2023-10-16T13:44:30.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:30.757+0000 7f286d592700 1 -- 172.21.15.16:0/251877119 wait complete. 2023-10-16T13:44:30.765 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' removed 2023-10-16T13:44:30.779 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-16T13:44:30.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:30.910+0000 7efc9dded700 1 Processor -- start 2023-10-16T13:44:30.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:30.910+0000 7efc9dded700 1 -- start start 2023-10-16T13:44:30.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:30.911+0000 7efc9dded700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7efc9806e570 0x7efc9815e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:30.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:30.911+0000 7efc9dded700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efc9806a7e0 0x7efc9806ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:30.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:30.911+0000 7efc9dded700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc9815b6a0 con 0x7efc9815b2c0 2023-10-16T13:44:30.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:30.911+0000 7efc9dded700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc9806b1b0 con 0x7efc9806e570 2023-10-16T13:44:30.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:30.911+0000 7efc9dded700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc98161b20 con 0x7efc9806a7e0 2023-10-16T13:44:30.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:30.911+0000 7efc96ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7efc9806e570 0x7efc9815e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:30.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:30.911+0000 7efc96ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7efc9806e570 0x7efc9815e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51388/0 (socket says 172.21.15.16:51388) 2023-10-16T13:44:30.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:30.911+0000 7efc96ffd700 1 -- 172.21.15.16:0/1116128384 learned_addr learned my addr 172.21.15.16:0/1116128384 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:30.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:30.911+0000 7efc97fff700 1 --2- 172.21.15.16:0/1116128384 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efc9806a7e0 0x7efc9806ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:30.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:30.912+0000 7efc96ffd700 1 -- 172.21.15.16:0/1116128384 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efc9806a7e0 msgr2=0x7efc9806ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:30.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:30.912+0000 7efc96ffd700 1 --2- 172.21.15.16:0/1116128384 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efc9806a7e0 0x7efc9806ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:30.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:30.912+0000 7efc96ffd700 1 -- 172.21.15.16:0/1116128384 >> v1:172.21.15.49:6789/0 conn(0x7efc9815b2c0 legacy=0x7efc98161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:30.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:30.912+0000 7efc96ffd700 1 -- 172.21.15.16:0/1116128384 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc98161e00 con 0x7efc9806e570 2023-10-16T13:44:30.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:30.912+0000 7efc96ffd700 1 --2- 172.21.15.16:0/1116128384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc9806e570 0x7efc9815e5a0 secure :-1 s=READY pgs=2237 cs=0 l=1 rev1=1 crypto rx=0x7efc8c0098f0 tx=0x7efc8c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fefd688ae1c330df server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:30.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:30.912+0000 7efc967fc700 1 -- 172.21.15.16:0/1116128384 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc8c00b8a0 con 0x7efc9806e570 2023-10-16T13:44:30.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:30.912+0000 7efc967fc700 1 -- 172.21.15.16:0/1116128384 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc8c00ba00 con 0x7efc9806e570 2023-10-16T13:44:30.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:30.912+0000 7efc967fc700 1 -- 172.21.15.16:0/1116128384 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc8c00bd90 con 0x7efc9806e570 2023-10-16T13:44:30.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:30.912+0000 7efc9dded700 1 -- 172.21.15.16:0/1116128384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc9806e570 msgr2=0x7efc9815e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:30.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:30.912+0000 7efc9dded700 1 --2- 172.21.15.16:0/1116128384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc9806e570 0x7efc9815e5a0 secure :-1 s=READY pgs=2237 cs=0 l=1 rev1=1 crypto rx=0x7efc8c0098f0 tx=0x7efc8c009c00 comp rx=0 tx=0).stop 2023-10-16T13:44:30.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:30.912+0000 7efc9dded700 1 -- 172.21.15.16:0/1116128384 shutdown_connections 2023-10-16T13:44:30.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:30.912+0000 7efc9dded700 1 --2- 172.21.15.16:0/1116128384 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efc9806a7e0 0x7efc9806ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:30.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:30.913+0000 7efc9dded700 1 --2- 172.21.15.16:0/1116128384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc9806e570 0x7efc9815e5a0 unknown :-1 s=CLOSED pgs=2237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:30.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:30.913+0000 7efc9dded700 1 -- 172.21.15.16:0/1116128384 >> 172.21.15.16:0/1116128384 conn(0x7efc980b1f40 msgr2=0x7efc980b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:30.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:30.913+0000 7efc9dded700 1 -- 172.21.15.16:0/1116128384 shutdown_connections 2023-10-16T13:44:30.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:30.913+0000 7efc9dded700 1 -- 172.21.15.16:0/1116128384 wait complete. 2023-10-16T13:44:30.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:30.914+0000 7efc9dded700 1 Processor -- start 2023-10-16T13:44:30.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:30.914+0000 7efc9dded700 1 -- start start 2023-10-16T13:44:30.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:30.915+0000 7efc9dded700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efc9806a7e0 0x7efc981587f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:30.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:30.915+0000 7efc9dded700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc9806e570 0x7efc981557a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:30.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:30.915+0000 7efc9dded700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efc9815b2c0 0x7efc98155ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:30.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:30.915+0000 7efc9dded700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc980ca390 con 0x7efc9806e570 2023-10-16T13:44:30.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:30.915+0000 7efc9dded700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc980b5c40 con 0x7efc9806a7e0 2023-10-16T13:44:30.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:30.915+0000 7efc9dded700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc980c74c0 con 0x7efc9815b2c0 2023-10-16T13:44:30.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:30.915+0000 7efc96ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc9806e570 0x7efc981557a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:30.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:30.915+0000 7efc96ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc9806e570 0x7efc981557a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51392/0 (socket says 172.21.15.16:51392) 2023-10-16T13:44:30.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:30.915+0000 7efc96ffd700 1 -- 172.21.15.16:0/3633823932 learned_addr learned my addr 172.21.15.16:0/3633823932 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:30.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:30.916+0000 7efc977fe700 1 --2- 172.21.15.16:0/3633823932 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efc9806a7e0 0x7efc981587f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:30.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:30.916+0000 7efc97fff700 1 --2- 172.21.15.16:0/3633823932 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efc9815b2c0 0x7efc98155ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:30.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:30.916+0000 7efc96ffd700 1 -- 172.21.15.16:0/3633823932 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efc9815b2c0 msgr2=0x7efc98155ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:30.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:30.916+0000 7efc96ffd700 1 --2- 172.21.15.16:0/3633823932 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efc9815b2c0 0x7efc98155ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:30.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:30.916+0000 7efc96ffd700 1 -- 172.21.15.16:0/3633823932 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efc9806a7e0 msgr2=0x7efc981587f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:30.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:30.916+0000 7efc96ffd700 1 --2- 172.21.15.16:0/3633823932 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efc9806a7e0 0x7efc981587f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:30.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:30.916+0000 7efc96ffd700 1 -- 172.21.15.16:0/3633823932 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc8c009550 con 0x7efc9806e570 2023-10-16T13:44:30.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:30.916+0000 7efc977fe700 1 --2- 172.21.15.16:0/3633823932 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efc9806a7e0 0x7efc981587f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:44:30.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:30.916+0000 7efc97fff700 1 --2- 172.21.15.16:0/3633823932 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efc9815b2c0 0x7efc98155ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:44:30.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:30.916+0000 7efc96ffd700 1 --2- 172.21.15.16:0/3633823932 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc9806e570 0x7efc981557a0 secure :-1 s=READY pgs=2238 cs=0 l=1 rev1=1 crypto rx=0x7efc8c000c00 tx=0x7efc8c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:30.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:30.916+0000 7efc7ffff700 1 -- 172.21.15.16:0/3633823932 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc8c004480 con 0x7efc9806e570 2023-10-16T13:44:30.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:30.916+0000 7efc7ffff700 1 -- 172.21.15.16:0/3633823932 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc8c01f070 con 0x7efc9806e570 2023-10-16T13:44:30.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:30.916+0000 7efc7ffff700 1 -- 172.21.15.16:0/3633823932 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc8c0038c0 con 0x7efc9806e570 2023-10-16T13:44:30.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:30.916+0000 7efc9dded700 1 -- 172.21.15.16:0/3633823932 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc980c76d0 con 0x7efc9806e570 2023-10-16T13:44:30.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:30.916+0000 7efc9dded700 1 -- 172.21.15.16:0/3633823932 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc980c7c90 con 0x7efc9806e570 2023-10-16T13:44:30.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:30.917+0000 7efc9dded700 1 -- 172.21.15.16:0/3633823932 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc9804ebb0 con 0x7efc9806e570 2023-10-16T13:44:30.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:30.918+0000 7efc7ffff700 1 -- 172.21.15.16:0/3633823932 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 126) v1 ==== 89331+0+0 (secure 0 0 0) 0x7efc8c021030 con 0x7efc9806e570 2023-10-16T13:44:30.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:30.918+0000 7efc7ffff700 1 --2- 172.21.15.16:0/3633823932 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7efc8406b020 0x7efc8406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:30.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:30.918+0000 7efc7ffff700 1 -- 172.21.15.16:0/3633823932 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 6423+0+0 (secure 0 0 0) 0x7efc8c08af50 con 0x7efc9806e570 2023-10-16T13:44:30.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:30.918+0000 7efc977fe700 1 --2- 172.21.15.16:0/3633823932 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7efc8406b020 0x7efc8406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:30.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:30.919+0000 7efc977fe700 1 --2- 172.21.15.16:0/3633823932 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7efc8406b020 0x7efc8406d4e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7efc88000f60 tx=0x7efc88009040 comp rx=0 tx=0).ready entity=mgr.10665 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:30.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:30.923+0000 7efc7ffff700 1 -- 172.21.15.16:0/3633823932 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7efc8c055cb0 con 0x7efc9806e570 2023-10-16T13:44:31.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.124+0000 7efc9dded700 1 -- 172.21.15.16:0/3633823932 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7efc980b6370 con 0x7efc9806e570 2023-10-16T13:44:31.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.128+0000 7efc7ffff700 1 -- 172.21.15.16:0/3633823932 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v67) v1 ==== 115+0+0 (secure 0 0 0) 0x7efc8c0592d0 con 0x7efc9806e570 2023-10-16T13:44:31.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.130+0000 7efc9dded700 1 -- 172.21.15.16:0/3633823932 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7efc8406b020 msgr2=0x7efc8406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:31.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.130+0000 7efc9dded700 1 --2- 172.21.15.16:0/3633823932 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7efc8406b020 0x7efc8406d4e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7efc88000f60 tx=0x7efc88009040 comp rx=0 tx=0).stop 2023-10-16T13:44:31.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.130+0000 7efc9dded700 1 -- 172.21.15.16:0/3633823932 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc9806e570 msgr2=0x7efc981557a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:31.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.130+0000 7efc9dded700 1 --2- 172.21.15.16:0/3633823932 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc9806e570 0x7efc981557a0 secure :-1 s=READY pgs=2238 cs=0 l=1 rev1=1 crypto rx=0x7efc8c000c00 tx=0x7efc8c004750 comp rx=0 tx=0).stop 2023-10-16T13:44:31.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.130+0000 7efc9dded700 1 -- 172.21.15.16:0/3633823932 shutdown_connections 2023-10-16T13:44:31.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.130+0000 7efc9dded700 1 --2- 172.21.15.16:0/3633823932 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7efc8406b020 0x7efc8406d4e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:31.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.130+0000 7efc9dded700 1 --2- 172.21.15.16:0/3633823932 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efc9815b2c0 0x7efc98155ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:31.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.130+0000 7efc9dded700 1 --2- 172.21.15.16:0/3633823932 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efc9806a7e0 0x7efc981587f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:31.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.130+0000 7efc9dded700 1 --2- 172.21.15.16:0/3633823932 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc9806e570 0x7efc981557a0 secure :-1 s=CLOSED pgs=2238 cs=0 l=1 rev1=1 crypto rx=0x7efc8c000c00 tx=0x7efc8c004750 comp rx=0 tx=0).stop 2023-10-16T13:44:31.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.130+0000 7efc9dded700 1 -- 172.21.15.16:0/3633823932 >> 172.21.15.16:0/3633823932 conn(0x7efc980b1f40 msgr2=0x7efc9815baf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:31.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.130+0000 7efc9dded700 1 -- 172.21.15.16:0/3633823932 shutdown_connections 2023-10-16T13:44:31.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.130+0000 7efc9dded700 1 -- 172.21.15.16:0/3633823932 wait complete. 2023-10-16T13:44:31.146 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-16T13:44:31.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.275+0000 7f86ab189700 1 Processor -- start 2023-10-16T13:44:31.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.275+0000 7f86ab189700 1 -- start start 2023-10-16T13:44:31.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.275+0000 7f86ab189700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f86a415d8d0 0x7f86a40b2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:31.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.275+0000 7f86ab189700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f86a40b8fa0 0x7f86a40b6720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:31.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.275+0000 7f86ab189700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86a40b2f30 con 0x7f86a40b6c60 2023-10-16T13:44:31.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.275+0000 7f86ab189700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86a40d0bb0 con 0x7f86a40b8fa0 2023-10-16T13:44:31.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.275+0000 7f86ab189700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86a415dcb0 con 0x7f86a415d8d0 2023-10-16T13:44:31.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.276+0000 7f86a3fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f86a40b8fa0 0x7f86a40b6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:31.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.276+0000 7f86a3fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f86a40b8fa0 0x7f86a40b6720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51402/0 (socket says 172.21.15.16:51402) 2023-10-16T13:44:31.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.276+0000 7f86a3fff700 1 -- 172.21.15.16:0/2547957093 learned_addr learned my addr 172.21.15.16:0/2547957093 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:31.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.276+0000 7f86a8f25700 1 --2- 172.21.15.16:0/2547957093 >> v2:172.21.15.49:3300/0 conn(0x7f86a415d8d0 0x7f86a40b2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:31.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.276+0000 7f86a3fff700 1 -- 172.21.15.16:0/2547957093 >> v2:172.21.15.49:3300/0 conn(0x7f86a415d8d0 msgr2=0x7f86a40b2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:31.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.276+0000 7f86a3fff700 1 --2- 172.21.15.16:0/2547957093 >> v2:172.21.15.49:3300/0 conn(0x7f86a415d8d0 0x7f86a40b2830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:31.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.276+0000 7f86a3fff700 1 -- 172.21.15.16:0/2547957093 >> v1:172.21.15.16:6789/0 conn(0x7f86a40b6c60 legacy=0x7f86a40ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:31.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.276+0000 7f86a3fff700 1 -- 172.21.15.16:0/2547957093 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86a40d1c40 con 0x7f86a40b8fa0 2023-10-16T13:44:31.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.276+0000 7f86a8f25700 1 --2- 172.21.15.16:0/2547957093 >> v2:172.21.15.49:3300/0 conn(0x7f86a415d8d0 0x7f86a40b2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:44:31.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.276+0000 7f86a3fff700 1 --2- 172.21.15.16:0/2547957093 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86a40b8fa0 0x7f86a40b6720 secure :-1 s=READY pgs=2240 cs=0 l=1 rev1=1 crypto rx=0x7f86a40d0dd0 tx=0x7f868c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6e32dbcfe15a802 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:31.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.276+0000 7f86a37fe700 1 -- 172.21.15.16:0/2547957093 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f868c00b870 con 0x7f86a40b8fa0 2023-10-16T13:44:31.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.276+0000 7f86a37fe700 1 -- 172.21.15.16:0/2547957093 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f868c00b9d0 con 0x7f86a40b8fa0 2023-10-16T13:44:31.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.276+0000 7f86a37fe700 1 -- 172.21.15.16:0/2547957093 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f868c00bd60 con 0x7f86a40b8fa0 2023-10-16T13:44:31.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.277+0000 7f86ab189700 1 -- 172.21.15.16:0/2547957093 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86a40b8fa0 msgr2=0x7f86a40b6720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:31.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.277+0000 7f86ab189700 1 --2- 172.21.15.16:0/2547957093 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86a40b8fa0 0x7f86a40b6720 secure :-1 s=READY pgs=2240 cs=0 l=1 rev1=1 crypto rx=0x7f86a40d0dd0 tx=0x7f868c009bd0 comp rx=0 tx=0).stop 2023-10-16T13:44:31.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.277+0000 7f86ab189700 1 -- 172.21.15.16:0/2547957093 shutdown_connections 2023-10-16T13:44:31.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.277+0000 7f86ab189700 1 --2- 172.21.15.16:0/2547957093 >> v2:172.21.15.49:3300/0 conn(0x7f86a415d8d0 0x7f86a40b2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:31.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.277+0000 7f86ab189700 1 --2- 172.21.15.16:0/2547957093 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86a40b8fa0 0x7f86a40b6720 unknown :-1 s=CLOSED pgs=2240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:31.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.277+0000 7f86ab189700 1 -- 172.21.15.16:0/2547957093 >> 172.21.15.16:0/2547957093 conn(0x7f86a4072330 msgr2=0x7f86a4072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:31.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.277+0000 7f86ab189700 1 -- 172.21.15.16:0/2547957093 shutdown_connections 2023-10-16T13:44:31.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.277+0000 7f86ab189700 1 -- 172.21.15.16:0/2547957093 wait complete. 2023-10-16T13:44:31.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.278+0000 7f86ab189700 1 Processor -- start 2023-10-16T13:44:31.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.278+0000 7f86ab189700 1 -- start start 2023-10-16T13:44:31.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.279+0000 7f86ab189700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f86a40b6c60 0x7f86a40c9300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:31.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.279+0000 7f86ab189700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86a40b8fa0 0x7f86a4149ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:31.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.279+0000 7f86ab189700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f86a415d8d0 0x7f86a414a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:31.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.279+0000 7f86ab189700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86a4148200 con 0x7f86a40b8fa0 2023-10-16T13:44:31.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.279+0000 7f86ab189700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86a4147580 con 0x7f86a40b6c60 2023-10-16T13:44:31.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.279+0000 7f86ab189700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86a4147860 con 0x7f86a415d8d0 2023-10-16T13:44:31.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.279+0000 7f86a3fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86a40b8fa0 0x7f86a4149ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:31.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.279+0000 7f86a3fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86a40b8fa0 0x7f86a4149ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51408/0 (socket says 172.21.15.16:51408) 2023-10-16T13:44:31.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.279+0000 7f86a3fff700 1 -- 172.21.15.16:0/1897788643 learned_addr learned my addr 172.21.15.16:0/1897788643 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:31.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.279+0000 7f86a8f25700 1 --2- 172.21.15.16:0/1897788643 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f86a40b6c60 0x7f86a40c9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:31.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.279+0000 7f86a9726700 1 --2- 172.21.15.16:0/1897788643 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f86a415d8d0 0x7f86a414a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:31.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.279+0000 7f86a3fff700 1 -- 172.21.15.16:0/1897788643 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f86a415d8d0 msgr2=0x7f86a414a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:31.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.279+0000 7f86a3fff700 1 --2- 172.21.15.16:0/1897788643 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f86a415d8d0 0x7f86a414a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:31.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.279+0000 7f86a3fff700 1 -- 172.21.15.16:0/1897788643 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f86a40b6c60 msgr2=0x7f86a40c9300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:31.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.279+0000 7f86a3fff700 1 --2- 172.21.15.16:0/1897788643 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f86a40b6c60 0x7f86a40c9300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:31.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.279+0000 7f86a3fff700 1 -- 172.21.15.16:0/1897788643 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f868c009550 con 0x7f86a40b8fa0 2023-10-16T13:44:31.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.279+0000 7f86a9726700 1 --2- 172.21.15.16:0/1897788643 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f86a415d8d0 0x7f86a414a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:44:31.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.280+0000 7f86a3fff700 1 --2- 172.21.15.16:0/1897788643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86a40b8fa0 0x7f86a4149ad0 secure :-1 s=READY pgs=2241 cs=0 l=1 rev1=1 crypto rx=0x7f868c014040 tx=0x7f868c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:31.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.280+0000 7f86a17fa700 1 -- 172.21.15.16:0/1897788643 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f868c004520 con 0x7f86a40b8fa0 2023-10-16T13:44:31.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.280+0000 7f86a17fa700 1 -- 172.21.15.16:0/1897788643 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f868c01e070 con 0x7f86a40b8fa0 2023-10-16T13:44:31.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.280+0000 7f86ab189700 1 -- 172.21.15.16:0/1897788643 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86a4147b40 con 0x7f86a40b8fa0 2023-10-16T13:44:31.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.280+0000 7f86a17fa700 1 -- 172.21.15.16:0/1897788643 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f868c0038c0 con 0x7f86a40b8fa0 2023-10-16T13:44:31.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.280+0000 7f86ab189700 1 -- 172.21.15.16:0/1897788643 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86a415b6f0 con 0x7f86a40b8fa0 2023-10-16T13:44:31.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.281+0000 7f86ab189700 1 -- 172.21.15.16:0/1897788643 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86a404ebb0 con 0x7f86a40b8fa0 2023-10-16T13:44:31.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.281+0000 7f86a17fa700 1 -- 172.21.15.16:0/1897788643 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 126) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f868c003a20 con 0x7f86a40b8fa0 2023-10-16T13:44:31.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.282+0000 7f86a17fa700 1 --2- 172.21.15.16:0/1897788643 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f868406b020 0x7f868406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:31.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.282+0000 7f86a17fa700 1 -- 172.21.15.16:0/1897788643 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 6423+0+0 (secure 0 0 0) 0x7f868c020070 con 0x7f86a40b8fa0 2023-10-16T13:44:31.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.282+0000 7f86a8f25700 1 --2- 172.21.15.16:0/1897788643 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f868406b020 0x7f868406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:31.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.282+0000 7f86a8f25700 1 --2- 172.21.15.16:0/1897788643 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f868406b020 0x7f868406d4e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8694006fd0 tx=0x7f8694008040 comp rx=0 tx=0).ready entity=mgr.10665 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:31.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.286+0000 7f86a17fa700 1 -- 172.21.15.16:0/1897788643 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f868c055d80 con 0x7f86a40b8fa0 2023-10-16T13:44:31.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.491+0000 7f86ab189700 1 -- 172.21.15.16:0/1897788643 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f86a40c9b50 con 0x7f86a40b8fa0 2023-10-16T13:44:31.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.494+0000 7f86a17fa700 1 -- 172.21.15.16:0/1897788643 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v68) v1 ==== 104+0+0 (secure 0 0 0) 0x7f868c0593a0 con 0x7f86a40b8fa0 2023-10-16T13:44:31.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.497+0000 7f86ab189700 1 -- 172.21.15.16:0/1897788643 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f868406b020 msgr2=0x7f868406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:31.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.497+0000 7f86ab189700 1 --2- 172.21.15.16:0/1897788643 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f868406b020 0x7f868406d4e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8694006fd0 tx=0x7f8694008040 comp rx=0 tx=0).stop 2023-10-16T13:44:31.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.497+0000 7f86ab189700 1 -- 172.21.15.16:0/1897788643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86a40b8fa0 msgr2=0x7f86a4149ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:31.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.497+0000 7f86ab189700 1 --2- 172.21.15.16:0/1897788643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86a40b8fa0 0x7f86a4149ad0 secure :-1 s=READY pgs=2241 cs=0 l=1 rev1=1 crypto rx=0x7f868c014040 tx=0x7f868c004750 comp rx=0 tx=0).stop 2023-10-16T13:44:31.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.497+0000 7f86ab189700 1 -- 172.21.15.16:0/1897788643 shutdown_connections 2023-10-16T13:44:31.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.497+0000 7f86ab189700 1 --2- 172.21.15.16:0/1897788643 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f868406b020 0x7f868406d4e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:31.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.497+0000 7f86ab189700 1 --2- 172.21.15.16:0/1897788643 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f86a415d8d0 0x7f86a414a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:31.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.497+0000 7f86ab189700 1 --2- 172.21.15.16:0/1897788643 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f86a40b6c60 0x7f86a40c9300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:31.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.497+0000 7f86ab189700 1 --2- 172.21.15.16:0/1897788643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86a40b8fa0 0x7f86a4149ad0 unknown :-1 s=CLOSED pgs=2241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:31.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.497+0000 7f86ab189700 1 -- 172.21.15.16:0/1897788643 >> 172.21.15.16:0/1897788643 conn(0x7f86a4072330 msgr2=0x7f86a40cff00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:31.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.497+0000 7f86ab189700 1 -- 172.21.15.16:0/1897788643 shutdown_connections 2023-10-16T13:44:31.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.497+0000 7f86ab189700 1 -- 172.21.15.16:0/1897788643 wait complete. 2023-10-16T13:44:31.511 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-16T13:44:31.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.636+0000 7f0aeb575700 1 Processor -- start 2023-10-16T13:44:31.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.636+0000 7f0aeb575700 1 -- start start 2023-10-16T13:44:31.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.637+0000 7f0aeb575700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0ae40cda60 0x7f0ae406d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:31.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.637+0000 7f0aeb575700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f0ae406dea0 0x7f0ae40ce850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:31.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.637+0000 7f0aeb575700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ae406e2f0 con 0x7f0ae415b790 2023-10-16T13:44:31.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.637+0000 7f0aeb575700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ae40ce0e0 con 0x7f0ae406dea0 2023-10-16T13:44:31.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.637+0000 7f0aeb575700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ae406a7e0 con 0x7f0ae40cda60 2023-10-16T13:44:31.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.637+0000 7f0ae8b10700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0ae40cda60 0x7f0ae406d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:31.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.637+0000 7f0ae9b12700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f0ae406dea0 0x7f0ae40ce850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:31.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.637+0000 7f0ae9311700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7f0ae415b790 0x7f0ae415bb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:47870/0 (socket says 172.21.15.16:47870) 2023-10-16T13:44:31.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.637+0000 7f0ae9b12700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f0ae406dea0 0x7f0ae40ce850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:50480/0 (socket says 172.21.15.16:50480) 2023-10-16T13:44:31.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.637+0000 7f0ae9311700 1 -- 172.21.15.16:0/3503528461 learned_addr learned my addr 172.21.15.16:0/3503528461 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:44:31.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.637+0000 7f0ae8b10700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0ae40cda60 0x7f0ae406d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:39674/0 (socket says 172.21.15.16:39674) 2023-10-16T13:44:31.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.638+0000 7f0ae8b10700 1 -- 172.21.15.16:0/3503528461 >> v2:172.21.15.49:3300/0 conn(0x7f0ae406dea0 msgr2=0x7f0ae40ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:31.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.638+0000 7f0ae8b10700 1 --2- 172.21.15.16:0/3503528461 >> v2:172.21.15.49:3300/0 conn(0x7f0ae406dea0 0x7f0ae40ce850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:31.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.638+0000 7f0ae8b10700 1 -- 172.21.15.16:0/3503528461 >> v1:172.21.15.49:6789/0 conn(0x7f0ae415b790 legacy=0x7f0ae415bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:31.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.638+0000 7f0ae8b10700 1 -- 172.21.15.16:0/3503528461 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ae415c410 con 0x7f0ae40cda60 2023-10-16T13:44:31.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.638+0000 7f0ae9b12700 1 --2- 172.21.15.16:0/3503528461 >> v2:172.21.15.49:3300/0 conn(0x7f0ae406dea0 0x7f0ae40ce850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:44:31.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.638+0000 7f0ae8b10700 1 --2- 172.21.15.16:0/3503528461 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0ae40cda60 0x7f0ae406d960 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f0ae0009840 tx=0x7f0ae0009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d003f4002727516 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:31.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.638+0000 7f0adbfff700 1 -- 172.21.15.16:0/3503528461 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ae0011830 con 0x7f0ae40cda60 2023-10-16T13:44:31.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.638+0000 7f0adbfff700 1 -- 172.21.15.16:0/3503528461 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ae0011990 con 0x7f0ae40cda60 2023-10-16T13:44:31.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.638+0000 7f0adbfff700 1 -- 172.21.15.16:0/3503528461 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ae0011d20 con 0x7f0ae40cda60 2023-10-16T13:44:31.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.639+0000 7f0aeb575700 1 -- 172.21.15.16:0/3503528461 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0ae40cda60 msgr2=0x7f0ae406d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:31.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.639+0000 7f0aeb575700 1 --2- 172.21.15.16:0/3503528461 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0ae40cda60 0x7f0ae406d960 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f0ae0009840 tx=0x7f0ae0009b50 comp rx=0 tx=0).stop 2023-10-16T13:44:31.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.639+0000 7f0aeb575700 1 -- 172.21.15.16:0/3503528461 shutdown_connections 2023-10-16T13:44:31.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.639+0000 7f0aeb575700 1 --2- 172.21.15.16:0/3503528461 >> v2:172.21.15.49:3300/0 conn(0x7f0ae406dea0 0x7f0ae40ce850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:31.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.639+0000 7f0aeb575700 1 --2- 172.21.15.16:0/3503528461 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0ae40cda60 0x7f0ae406d960 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:31.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.639+0000 7f0aeb575700 1 -- 172.21.15.16:0/3503528461 >> 172.21.15.16:0/3503528461 conn(0x7f0ae40cc5e0 msgr2=0x7f0ae40cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:31.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.639+0000 7f0aeb575700 1 -- 172.21.15.16:0/3503528461 shutdown_connections 2023-10-16T13:44:31.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.639+0000 7f0aeb575700 1 -- 172.21.15.16:0/3503528461 wait complete. 2023-10-16T13:44:31.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.640+0000 7f0aeb575700 1 Processor -- start 2023-10-16T13:44:31.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.640+0000 7f0aeb575700 1 -- start start 2023-10-16T13:44:31.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.641+0000 7f0aeb575700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0ae406dea0 0x7f0ae40addd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:31.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.641+0000 7f0aeb575700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0ae40cda60 0x7f0ae4153dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:31.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.641+0000 7f0aeb575700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ae415b790 0x7f0ae41f66d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:31.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.641+0000 7f0aeb575700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ae4154300 con 0x7f0ae415b790 2023-10-16T13:44:31.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.641+0000 7f0aeb575700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ae4154510 con 0x7f0ae406dea0 2023-10-16T13:44:31.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.641+0000 7f0aeb575700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ae41547f0 con 0x7f0ae40cda60 2023-10-16T13:44:31.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.641+0000 7f0ae9b12700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ae415b790 0x7f0ae41f66d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:31.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.641+0000 7f0ae9b12700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ae415b790 0x7f0ae41f66d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51424/0 (socket says 172.21.15.16:51424) 2023-10-16T13:44:31.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.641+0000 7f0ae9b12700 1 -- 172.21.15.16:0/653128483 learned_addr learned my addr 172.21.15.16:0/653128483 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:31.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.641+0000 7f0ae8b10700 1 --2- 172.21.15.16:0/653128483 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0ae40cda60 0x7f0ae4153dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:31.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.641+0000 7f0ae9311700 1 --2- 172.21.15.16:0/653128483 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0ae406dea0 0x7f0ae40addd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:31.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.641+0000 7f0ae9b12700 1 -- 172.21.15.16:0/653128483 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0ae40cda60 msgr2=0x7f0ae4153dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:31.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.641+0000 7f0ae9b12700 1 --2- 172.21.15.16:0/653128483 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0ae40cda60 0x7f0ae4153dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:31.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.641+0000 7f0ae9b12700 1 -- 172.21.15.16:0/653128483 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0ae406dea0 msgr2=0x7f0ae40addd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:31.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.641+0000 7f0ae9b12700 1 --2- 172.21.15.16:0/653128483 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0ae406dea0 0x7f0ae40addd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:31.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.641+0000 7f0ae9b12700 1 -- 172.21.15.16:0/653128483 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0adc009a70 con 0x7f0ae415b790 2023-10-16T13:44:31.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.641+0000 7f0ae9b12700 1 --2- 172.21.15.16:0/653128483 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ae415b790 0x7f0ae41f66d0 secure :-1 s=READY pgs=2242 cs=0 l=1 rev1=1 crypto rx=0x7f0adc00efc0 tx=0x7f0adc00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:31.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.642+0000 7f0ad9ffb700 1 -- 172.21.15.16:0/653128483 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0adc00ca90 con 0x7f0ae415b790 2023-10-16T13:44:31.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.642+0000 7f0aeb575700 1 -- 172.21.15.16:0/653128483 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ae00094a0 con 0x7f0ae415b790 2023-10-16T13:44:31.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.642+0000 7f0ad9ffb700 1 -- 172.21.15.16:0/653128483 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0adc00f040 con 0x7f0ae415b790 2023-10-16T13:44:31.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.642+0000 7f0ad9ffb700 1 -- 172.21.15.16:0/653128483 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0adc00ce10 con 0x7f0ae415b790 2023-10-16T13:44:31.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.642+0000 7f0aeb575700 1 -- 172.21.15.16:0/653128483 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ae4154db0 con 0x7f0ae415b790 2023-10-16T13:44:31.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.643+0000 7f0aeb575700 1 -- 172.21.15.16:0/653128483 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ae404ebb0 con 0x7f0ae415b790 2023-10-16T13:44:31.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.643+0000 7f0ad9ffb700 1 -- 172.21.15.16:0/653128483 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 126) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0adc014480 con 0x7f0ae415b790 2023-10-16T13:44:31.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.644+0000 7f0ad9ffb700 1 --2- 172.21.15.16:0/653128483 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f0ac806b020 0x7f0ac806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:31.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.644+0000 7f0ad9ffb700 1 -- 172.21.15.16:0/653128483 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 6423+0+0 (secure 0 0 0) 0x7f0adc08a200 con 0x7f0ae415b790 2023-10-16T13:44:31.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.644+0000 7f0ae9311700 1 --2- 172.21.15.16:0/653128483 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f0ac806b020 0x7f0ac806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:31.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.644+0000 7f0ae9311700 1 --2- 172.21.15.16:0/653128483 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f0ac806b020 0x7f0ac806d4e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0ad4005910 tx=0x7f0ad4009040 comp rx=0 tx=0).ready entity=mgr.10665 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:31.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.648+0000 7f0ad9ffb700 1 -- 172.21.15.16:0/653128483 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0adc054d40 con 0x7f0ae415b790 2023-10-16T13:44:31.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.857+0000 7f0aeb575700 1 -- 172.21.15.16:0/653128483 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f0ae406b330 con 0x7f0ae415b790 2023-10-16T13:44:31.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.860+0000 7f0ad9ffb700 1 -- 172.21.15.16:0/653128483 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v69) v1 ==== 120+0+0 (secure 0 0 0) 0x7f0adc009d70 con 0x7f0ae415b790 2023-10-16T13:44:31.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.863+0000 7f0aeb575700 1 -- 172.21.15.16:0/653128483 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f0ac806b020 msgr2=0x7f0ac806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:31.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.863+0000 7f0aeb575700 1 --2- 172.21.15.16:0/653128483 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f0ac806b020 0x7f0ac806d4e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0ad4005910 tx=0x7f0ad4009040 comp rx=0 tx=0).stop 2023-10-16T13:44:31.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.863+0000 7f0aeb575700 1 -- 172.21.15.16:0/653128483 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ae415b790 msgr2=0x7f0ae41f66d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:31.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.863+0000 7f0aeb575700 1 --2- 172.21.15.16:0/653128483 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ae415b790 0x7f0ae41f66d0 secure :-1 s=READY pgs=2242 cs=0 l=1 rev1=1 crypto rx=0x7f0adc00efc0 tx=0x7f0adc00c3a0 comp rx=0 tx=0).stop 2023-10-16T13:44:31.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.863+0000 7f0aeb575700 1 -- 172.21.15.16:0/653128483 shutdown_connections 2023-10-16T13:44:31.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.863+0000 7f0aeb575700 1 --2- 172.21.15.16:0/653128483 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f0ac806b020 0x7f0ac806d4e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:31.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.863+0000 7f0aeb575700 1 --2- 172.21.15.16:0/653128483 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ae415b790 0x7f0ae41f66d0 unknown :-1 s=CLOSED pgs=2242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:31.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.863+0000 7f0aeb575700 1 --2- 172.21.15.16:0/653128483 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0ae406dea0 0x7f0ae40addd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:31.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.863+0000 7f0aeb575700 1 --2- 172.21.15.16:0/653128483 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0ae40cda60 0x7f0ae4153dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:31.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.863+0000 7f0aeb575700 1 -- 172.21.15.16:0/653128483 >> 172.21.15.16:0/653128483 conn(0x7f0ae40cc5e0 msgr2=0x7f0ae406b850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:31.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.863+0000 7f0aeb575700 1 -- 172.21.15.16:0/653128483 shutdown_connections 2023-10-16T13:44:31.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:31.863+0000 7f0aeb575700 1 -- 172.21.15.16:0/653128483 wait complete. 2023-10-16T13:44:31.879 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart' 2023-10-16T13:44:32.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.002+0000 7f758a3a1700 1 Processor -- start 2023-10-16T13:44:32.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.002+0000 7f758a3a1700 1 -- start start 2023-10-16T13:44:32.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.003+0000 7f758a3a1700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f758415b630 0x7f75840cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:32.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.003+0000 7f758a3a1700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f75840ce4f0 0x7f75840cfca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:32.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.003+0000 7f758a3a1700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75840cc2a0 con 0x7f758415ba10 2023-10-16T13:44:32.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.003+0000 7f758a3a1700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75840cdb20 con 0x7f75840ce4f0 2023-10-16T13:44:32.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.003+0000 7f758a3a1700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75840cc540 con 0x7f758415b630 2023-10-16T13:44:32.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.003+0000 7f758893e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f75840ce4f0 0x7f75840cfca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:32.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.003+0000 7f758893e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f75840ce4f0 0x7f75840cfca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51430/0 (socket says 172.21.15.16:51430) 2023-10-16T13:44:32.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.003+0000 7f758893e700 1 -- 172.21.15.16:0/720997860 learned_addr learned my addr 172.21.15.16:0/720997860 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:32.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.003+0000 7f7583fff700 1 --2- 172.21.15.16:0/720997860 >> v2:172.21.15.49:3300/0 conn(0x7f758415b630 0x7f75840cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:32.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.003+0000 7f758893e700 1 -- 172.21.15.16:0/720997860 >> v2:172.21.15.49:3300/0 conn(0x7f758415b630 msgr2=0x7f75840cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:32.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.003+0000 7f758893e700 1 --2- 172.21.15.16:0/720997860 >> v2:172.21.15.49:3300/0 conn(0x7f758415b630 0x7f75840cd5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:32.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.003+0000 7f758893e700 1 -- 172.21.15.16:0/720997860 >> v1:172.21.15.49:6789/0 conn(0x7f758415ba10 legacy=0x7f75840cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:32.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.003+0000 7f758893e700 1 -- 172.21.15.16:0/720997860 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75840cc820 con 0x7f75840ce4f0 2023-10-16T13:44:32.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.003+0000 7f758893e700 1 --2- 172.21.15.16:0/720997860 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75840ce4f0 0x7f75840cfca0 secure :-1 s=READY pgs=2243 cs=0 l=1 rev1=1 crypto rx=0x7f75840cefb0 tx=0x7f757c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c39cf68ddd8d068a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:32.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.004+0000 7f7582ffd700 1 -- 172.21.15.16:0/720997860 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f757c00e070 con 0x7f75840ce4f0 2023-10-16T13:44:32.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.004+0000 7f7582ffd700 1 -- 172.21.15.16:0/720997860 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f757c003a20 con 0x7f75840ce4f0 2023-10-16T13:44:32.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.004+0000 7f7582ffd700 1 -- 172.21.15.16:0/720997860 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f757c003dd0 con 0x7f75840ce4f0 2023-10-16T13:44:32.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.004+0000 7f758a3a1700 1 -- 172.21.15.16:0/720997860 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75840ce4f0 msgr2=0x7f75840cfca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:32.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.004+0000 7f758a3a1700 1 --2- 172.21.15.16:0/720997860 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75840ce4f0 0x7f75840cfca0 secure :-1 s=READY pgs=2243 cs=0 l=1 rev1=1 crypto rx=0x7f75840cefb0 tx=0x7f757c00b330 comp rx=0 tx=0).stop 2023-10-16T13:44:32.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.004+0000 7f758a3a1700 1 -- 172.21.15.16:0/720997860 shutdown_connections 2023-10-16T13:44:32.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.004+0000 7f758a3a1700 1 --2- 172.21.15.16:0/720997860 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75840ce4f0 0x7f75840cfca0 unknown :-1 s=CLOSED pgs=2243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:32.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.004+0000 7f758a3a1700 1 --2- 172.21.15.16:0/720997860 >> v2:172.21.15.49:3300/0 conn(0x7f758415b630 0x7f75840cd5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:32.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.004+0000 7f758a3a1700 1 -- 172.21.15.16:0/720997860 >> 172.21.15.16:0/720997860 conn(0x7f75840d14b0 msgr2=0x7f75840d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:32.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.004+0000 7f758a3a1700 1 -- 172.21.15.16:0/720997860 shutdown_connections 2023-10-16T13:44:32.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.004+0000 7f758a3a1700 1 -- 172.21.15.16:0/720997860 wait complete. 2023-10-16T13:44:32.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.005+0000 7f758a3a1700 1 Processor -- start 2023-10-16T13:44:32.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.006+0000 7f758a3a1700 1 -- start start 2023-10-16T13:44:32.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.006+0000 7f758a3a1700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f75840ce4f0 0x7f758406d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:32.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.006+0000 7f758a3a1700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f758415b630 0x7f758406f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:32.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.006+0000 7f758a3a1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f758415ba10 0x7f758406f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:32.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.006+0000 7f758a3a1700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75840c5910 con 0x7f758415ba10 2023-10-16T13:44:32.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.006+0000 7f758a3a1700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f758406d010 con 0x7f75840ce4f0 2023-10-16T13:44:32.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.006+0000 7f758a3a1700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75840c8580 con 0x7f758415b630 2023-10-16T13:44:32.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.007+0000 7f758893e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f758415ba10 0x7f758406f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:32.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.007+0000 7f758893e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f758415ba10 0x7f758406f8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51436/0 (socket says 172.21.15.16:51436) 2023-10-16T13:44:32.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.007+0000 7f758893e700 1 -- 172.21.15.16:0/3712472997 learned_addr learned my addr 172.21.15.16:0/3712472997 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:32.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.007+0000 7f7583fff700 1 --2- 172.21.15.16:0/3712472997 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f75840ce4f0 0x7f758406d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:32.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.007+0000 7f75837fe700 1 --2- 172.21.15.16:0/3712472997 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f758415b630 0x7f758406f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:32.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.007+0000 7f758893e700 1 -- 172.21.15.16:0/3712472997 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f758415b630 msgr2=0x7f758406f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:32.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.007+0000 7f758893e700 1 --2- 172.21.15.16:0/3712472997 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f758415b630 0x7f758406f380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:32.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.007+0000 7f758893e700 1 -- 172.21.15.16:0/3712472997 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f75840ce4f0 msgr2=0x7f758406d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:32.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.007+0000 7f758893e700 1 --2- 172.21.15.16:0/3712472997 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f75840ce4f0 0x7f758406d510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:32.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.007+0000 7f758893e700 1 -- 172.21.15.16:0/3712472997 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f757c009b30 con 0x7f758415ba10 2023-10-16T13:44:32.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.007+0000 7f758893e700 1 --2- 172.21.15.16:0/3712472997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f758415ba10 0x7f758406f8c0 secure :-1 s=READY pgs=2244 cs=0 l=1 rev1=1 crypto rx=0x7f757c00b370 tx=0x7f757c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:32.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.007+0000 7f7580ff9700 1 -- 172.21.15.16:0/3712472997 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f757c004440 con 0x7f758415ba10 2023-10-16T13:44:32.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.007+0000 7f7580ff9700 1 -- 172.21.15.16:0/3712472997 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f757c0045a0 con 0x7f758415ba10 2023-10-16T13:44:32.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.007+0000 7f758a3a1700 1 -- 172.21.15.16:0/3712472997 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75840c8860 con 0x7f758415ba10 2023-10-16T13:44:32.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.007+0000 7f7580ff9700 1 -- 172.21.15.16:0/3712472997 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f757c004950 con 0x7f758415ba10 2023-10-16T13:44:32.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.007+0000 7f758a3a1700 1 -- 172.21.15.16:0/3712472997 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75840c8e20 con 0x7f758415ba10 2023-10-16T13:44:32.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.008+0000 7f758a3a1700 1 -- 172.21.15.16:0/3712472997 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f758404ebb0 con 0x7f758415ba10 2023-10-16T13:44:32.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.009+0000 7f7580ff9700 1 -- 172.21.15.16:0/3712472997 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 126) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f757c012430 con 0x7f758415ba10 2023-10-16T13:44:32.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.009+0000 7f7580ff9700 1 --2- 172.21.15.16:0/3712472997 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f756406b020 0x7f756406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:32.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.009+0000 7f7580ff9700 1 -- 172.21.15.16:0/3712472997 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 6423+0+0 (secure 0 0 0) 0x7f757c08b000 con 0x7f758415ba10 2023-10-16T13:44:32.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.009+0000 7f7583fff700 1 --2- 172.21.15.16:0/3712472997 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f756406b020 0x7f756406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:32.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.010+0000 7f7583fff700 1 --2- 172.21.15.16:0/3712472997 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f756406b020 0x7f756406d4e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7574006fd0 tx=0x7f7574008040 comp rx=0 tx=0).ready entity=mgr.10665 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:32.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.014+0000 7f7580ff9700 1 -- 172.21.15.16:0/3712472997 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f757c055bd0 con 0x7f758415ba10 2023-10-16T13:44:32.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.298+0000 7f758a3a1700 1 -- 172.21.15.16:0/3712472997 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]} v 0) v1 -- 0x7f758406d2f0 con 0x7f758415ba10 2023-10-16T13:44:32.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.747+0000 7f7580ff9700 1 -- 172.21.15.16:0/3712472997 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]}]=0 v3047) v1 ==== 165+0+0 (secure 0 0 0) 0x7f757c0591f0 con 0x7f758415ba10 2023-10-16T13:44:32.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.751+0000 7f758a3a1700 1 -- 172.21.15.16:0/3712472997 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f756406b020 msgr2=0x7f756406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:32.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.751+0000 7f758a3a1700 1 --2- 172.21.15.16:0/3712472997 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f756406b020 0x7f756406d4e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7574006fd0 tx=0x7f7574008040 comp rx=0 tx=0).stop 2023-10-16T13:44:32.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.751+0000 7f758a3a1700 1 -- 172.21.15.16:0/3712472997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f758415ba10 msgr2=0x7f758406f8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:32.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.751+0000 7f758a3a1700 1 --2- 172.21.15.16:0/3712472997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f758415ba10 0x7f758406f8c0 secure :-1 s=READY pgs=2244 cs=0 l=1 rev1=1 crypto rx=0x7f757c00b370 tx=0x7f757c004ad0 comp rx=0 tx=0).stop 2023-10-16T13:44:32.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.751+0000 7f758a3a1700 1 -- 172.21.15.16:0/3712472997 shutdown_connections 2023-10-16T13:44:32.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.751+0000 7f758a3a1700 1 --2- 172.21.15.16:0/3712472997 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f756406b020 0x7f756406d4e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:32.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.751+0000 7f758a3a1700 1 --2- 172.21.15.16:0/3712472997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f758415ba10 0x7f758406f8c0 unknown :-1 s=CLOSED pgs=2244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:32.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.751+0000 7f758a3a1700 1 --2- 172.21.15.16:0/3712472997 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f75840ce4f0 0x7f758406d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:32.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.751+0000 7f758a3a1700 1 --2- 172.21.15.16:0/3712472997 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f758415b630 0x7f758406f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:32.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.751+0000 7f758a3a1700 1 -- 172.21.15.16:0/3712472997 >> 172.21.15.16:0/3712472997 conn(0x7f75840d14b0 msgr2=0x7f758415aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:32.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.751+0000 7f758a3a1700 1 -- 172.21.15.16:0/3712472997 shutdown_connections 2023-10-16T13:44:32.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.751+0000 7f758a3a1700 1 -- 172.21.15.16:0/3712472997 wait complete. 2023-10-16T13:44:32.769 INFO:tasks.cephfs_test_runner:Test that snap schedule is resumed after mgr restart ... ok 2023-10-16T13:44:32.769 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_with_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test scheduled snapshots along with rentention policy 2023-10-16T13:44:32.769 INFO:tasks.cephfs_test_runner:test_snap_schedule_with_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-16T13:44:32.770 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:44:32.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.900+0000 7f14507b4700 1 Processor -- start 2023-10-16T13:44:32.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.900+0000 7f14507b4700 1 -- start start 2023-10-16T13:44:32.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.901+0000 7f14507b4700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f144815a5c0 0x7f14480cef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:32.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.901+0000 7f14507b4700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f14480d0ae0 0x7f14480cc1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:32.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.901+0000 7f14507b4700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1448072a50 con 0x7f14480ce010 2023-10-16T13:44:32.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.901+0000 7f14507b4700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14480cf440 con 0x7f14480d0ae0 2023-10-16T13:44:32.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.901+0000 7f14507b4700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14480cba30 con 0x7f144815a5c0 2023-10-16T13:44:32.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.901+0000 7f144e550700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f144815a5c0 0x7f14480cef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:32.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.901+0000 7f144ed51700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f14480d0ae0 0x7f14480cc1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:32.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.901+0000 7f144dd4f700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7f14480ce010 0x7f14480d03d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:47896/0 (socket says 172.21.15.16:47896) 2023-10-16T13:44:32.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.901+0000 7f144dd4f700 1 -- 172.21.15.16:0/4253250897 learned_addr learned my addr 172.21.15.16:0/4253250897 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:44:32.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.901+0000 7f144ed51700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f14480d0ae0 0x7f14480cc1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:50506/0 (socket says 172.21.15.16:50506) 2023-10-16T13:44:32.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.901+0000 7f144e550700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f144815a5c0 0x7f14480cef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:39698/0 (socket says 172.21.15.16:39698) 2023-10-16T13:44:32.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.902+0000 7f144e550700 1 -- 172.21.15.16:0/4253250897 >> v2:172.21.15.49:3300/0 conn(0x7f14480d0ae0 msgr2=0x7f14480cc1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:32.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.902+0000 7f144e550700 1 --2- 172.21.15.16:0/4253250897 >> v2:172.21.15.49:3300/0 conn(0x7f14480d0ae0 0x7f14480cc1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:32.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.902+0000 7f144e550700 1 -- 172.21.15.16:0/4253250897 >> v1:172.21.15.49:6789/0 conn(0x7f14480ce010 legacy=0x7f14480d03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:32.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.902+0000 7f144e550700 1 -- 172.21.15.16:0/4253250897 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14480cf630 con 0x7f144815a5c0 2023-10-16T13:44:32.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.902+0000 7f144ed51700 1 --2- 172.21.15.16:0/4253250897 >> v2:172.21.15.49:3300/0 conn(0x7f14480d0ae0 0x7f14480cc1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:44:32.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.902+0000 7f144e550700 1 --2- 172.21.15.16:0/4253250897 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f144815a5c0 0x7f14480cef00 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f1438009840 tx=0x7f1438009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b5dea1a57f395a9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:32.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.902+0000 7f144d54e700 1 -- 172.21.15.16:0/4253250897 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1438011830 con 0x7f144815a5c0 2023-10-16T13:44:32.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.902+0000 7f144d54e700 1 -- 172.21.15.16:0/4253250897 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1438011990 con 0x7f144815a5c0 2023-10-16T13:44:32.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.902+0000 7f144d54e700 1 -- 172.21.15.16:0/4253250897 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1438011d20 con 0x7f144815a5c0 2023-10-16T13:44:32.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.903+0000 7f14507b4700 1 -- 172.21.15.16:0/4253250897 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f144815a5c0 msgr2=0x7f14480cef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:32.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.903+0000 7f14507b4700 1 --2- 172.21.15.16:0/4253250897 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f144815a5c0 0x7f14480cef00 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f1438009840 tx=0x7f1438009b50 comp rx=0 tx=0).stop 2023-10-16T13:44:32.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.903+0000 7f14507b4700 1 -- 172.21.15.16:0/4253250897 shutdown_connections 2023-10-16T13:44:32.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.903+0000 7f14507b4700 1 --2- 172.21.15.16:0/4253250897 >> v2:172.21.15.49:3300/0 conn(0x7f14480d0ae0 0x7f14480cc1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:32.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.903+0000 7f14507b4700 1 --2- 172.21.15.16:0/4253250897 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f144815a5c0 0x7f14480cef00 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:32.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.903+0000 7f14507b4700 1 -- 172.21.15.16:0/4253250897 >> 172.21.15.16:0/4253250897 conn(0x7f14480b1fd0 msgr2=0x7f14480b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:32.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.903+0000 7f14507b4700 1 -- 172.21.15.16:0/4253250897 shutdown_connections 2023-10-16T13:44:32.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.903+0000 7f14507b4700 1 -- 172.21.15.16:0/4253250897 wait complete. 2023-10-16T13:44:32.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.904+0000 7f14507b4700 1 Processor -- start 2023-10-16T13:44:32.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.904+0000 7f14507b4700 1 -- start start 2023-10-16T13:44:32.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.905+0000 7f14507b4700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f14480ce010 0x7f14480a84c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:32.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.905+0000 7f14507b4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14480d0ae0 0x7f14480ae880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:32.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.905+0000 7f14507b4700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f144815a5c0 0x7f14480b57f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:32.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.905+0000 7f14507b4700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f144814c510 con 0x7f14480d0ae0 2023-10-16T13:44:32.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.905+0000 7f14507b4700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14480a8a00 con 0x7f144815a5c0 2023-10-16T13:44:32.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.905+0000 7f14507b4700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f144814bbe0 con 0x7f14480ce010 2023-10-16T13:44:32.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.905+0000 7f144dd4f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14480d0ae0 0x7f14480ae880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:32.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.905+0000 7f144dd4f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14480d0ae0 0x7f14480ae880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51448/0 (socket says 172.21.15.16:51448) 2023-10-16T13:44:32.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.905+0000 7f144dd4f700 1 -- 172.21.15.16:0/246449152 learned_addr learned my addr 172.21.15.16:0/246449152 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:32.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.905+0000 7f144e550700 1 --2- 172.21.15.16:0/246449152 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f14480ce010 0x7f14480a84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:32.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.905+0000 7f144ed51700 1 --2- 172.21.15.16:0/246449152 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f144815a5c0 0x7f14480b57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:32.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.906+0000 7f144dd4f700 1 -- 172.21.15.16:0/246449152 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f14480ce010 msgr2=0x7f14480a84c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:32.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.906+0000 7f144dd4f700 1 --2- 172.21.15.16:0/246449152 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f14480ce010 0x7f14480a84c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:32.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.906+0000 7f144dd4f700 1 -- 172.21.15.16:0/246449152 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f144815a5c0 msgr2=0x7f14480b57f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:32.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.906+0000 7f144dd4f700 1 --2- 172.21.15.16:0/246449152 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f144815a5c0 0x7f14480b57f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:32.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.906+0000 7f144dd4f700 1 -- 172.21.15.16:0/246449152 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f144000bc80 con 0x7f14480d0ae0 2023-10-16T13:44:32.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.906+0000 7f144dd4f700 1 --2- 172.21.15.16:0/246449152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14480d0ae0 0x7f14480ae880 secure :-1 s=READY pgs=2245 cs=0 l=1 rev1=1 crypto rx=0x7f144400c480 tx=0x7f144400c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:32.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.906+0000 7f143effd700 1 -- 172.21.15.16:0/246449152 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1444005d70 con 0x7f14480d0ae0 2023-10-16T13:44:32.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.906+0000 7f143effd700 1 -- 172.21.15.16:0/246449152 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f144400ce90 con 0x7f14480d0ae0 2023-10-16T13:44:32.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.906+0000 7f14507b4700 1 -- 172.21.15.16:0/246449152 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14380094a0 con 0x7f14480d0ae0 2023-10-16T13:44:32.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.906+0000 7f143effd700 1 -- 172.21.15.16:0/246449152 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14440072d0 con 0x7f14480d0ae0 2023-10-16T13:44:32.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.906+0000 7f14507b4700 1 -- 172.21.15.16:0/246449152 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f144814c1a0 con 0x7f14480d0ae0 2023-10-16T13:44:32.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.907+0000 7f14507b4700 1 -- 172.21.15.16:0/246449152 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f144804ebb0 con 0x7f14480d0ae0 2023-10-16T13:44:32.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.907+0000 7f143effd700 1 -- 172.21.15.16:0/246449152 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 126) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1444007430 con 0x7f14480d0ae0 2023-10-16T13:44:32.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.908+0000 7f143effd700 1 --2- 172.21.15.16:0/246449152 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f142c06b020 0x7f142c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:32.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.908+0000 7f143effd700 1 -- 172.21.15.16:0/246449152 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 6423+0+0 (secure 0 0 0) 0x7f1444088ba0 con 0x7f14480d0ae0 2023-10-16T13:44:32.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.908+0000 7f144e550700 1 --2- 172.21.15.16:0/246449152 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f142c06b020 0x7f142c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:32.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.908+0000 7f144e550700 1 --2- 172.21.15.16:0/246449152 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f142c06b020 0x7f142c06d4e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f1438000c00 tx=0x7f143801d040 comp rx=0 tx=0).ready entity=mgr.10665 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:32.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:32.913+0000 7f143effd700 1 -- 172.21.15.16:0/246449152 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f14440539b0 con 0x7f14480d0ae0 2023-10-16T13:44:33.038 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:44:33.041 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:44:33.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.117+0000 7f14507b4700 1 -- 172.21.15.16:0/246449152 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f144815df20 con 0x7f14480d0ae0 2023-10-16T13:44:33.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.118+0000 7f143effd700 1 -- 172.21.15.16:0/246449152 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v198) v1 ==== 74+0+14796 (secure 0 0 0) 0x7f1444017090 con 0x7f14480d0ae0 2023-10-16T13:44:33.121 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:44:33.121 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":198,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:44:30.741025+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6834/3244401013":"2023-10-17T13:44:27.489969+0000","172.21.15.16:0/241231349":"2023-10-17T13:44:26.408978+0000","172.21.15.16:6833/371380443":"2023-10-17T13:44:26.408978+0000","172.21.15.16:6832/371380443":"2023-10-17T13:44:26.408978+0000","172.21.15.16:0/1071548593":"2023-10-17T13:44:26.408978+0000","172.21.15.16:0/1451116395":"2023-10-17T13:44:26.408978+0000","172.21.15.16:0/1245782052":"2023-10-17T13:44:26.408978+0000","172.21.15.16:6833/3952564978":"2023-10-17T13:39:22.557973+0000","172.21.15.16:6832/3952564978":"2023-10-17T13:39:22.557973+0000","172.21.15.16:0/2554840790":"2023-10-17T13:39:22.557973+0000","172.21.15.16:0/360056771":"2023-10-17T13:39:22.557973+0000","172.21.15.16:6836/3244401013":"2023-10-17T13:44:27.489969+0000","172.21.15.16:0/3520685377":"2023-10-17T13:39:22.557973+0000","172.21.15.16:0/1571009468":"2023-10-17T13:28:34.631949+0000","172.21.15.16:0/810487160":"2023-10-17T13:44:26.408978+0000","172.21.15.16:0/3374696233":"2023-10-17T13:44:26.408978+0000","172.21.15.16:0/678119374":"2023-10-17T13:28:34.631949+0000","172.21.15.16:0/3920247727":"2023-10-17T13:28:34.631949+0000","172.21.15.16:0/857902049":"2023-10-17T13:39:22.557973+0000","172.21.15.16:6833/4211329181":"2023-10-17T13:28:34.631949+0000","172.21.15.16:6832/4211329181":"2023-10-17T13:28:34.631949+0000","172.21.15.16:0/1548506671":"2023-10-17T13:39:22.557973+0000","172.21.15.16:0/2852193608":"2023-10-17T13:28:34.631949+0000","172.21.15.16:0/1641955134":"2023-10-17T13:39:22.557973+0000","172.21.15.16:0/471630266":"2023-10-17T13:28:34.631949+0000","172.21.15.16:0/3153082432":"2023-10-17T13:28:34.631949+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:44:33.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.120+0000 7f14507b4700 1 -- 172.21.15.16:0/246449152 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f142c06b020 msgr2=0x7f142c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:33.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.120+0000 7f14507b4700 1 --2- 172.21.15.16:0/246449152 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f142c06b020 0x7f142c06d4e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f1438000c00 tx=0x7f143801d040 comp rx=0 tx=0).stop 2023-10-16T13:44:33.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.120+0000 7f14507b4700 1 -- 172.21.15.16:0/246449152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14480d0ae0 msgr2=0x7f14480ae880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:33.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.120+0000 7f14507b4700 1 --2- 172.21.15.16:0/246449152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14480d0ae0 0x7f14480ae880 secure :-1 s=READY pgs=2245 cs=0 l=1 rev1=1 crypto rx=0x7f144400c480 tx=0x7f144400c790 comp rx=0 tx=0).stop 2023-10-16T13:44:33.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.120+0000 7f14507b4700 1 -- 172.21.15.16:0/246449152 shutdown_connections 2023-10-16T13:44:33.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.120+0000 7f14507b4700 1 --2- 172.21.15.16:0/246449152 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f142c06b020 0x7f142c06d4e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:33.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.120+0000 7f14507b4700 1 --2- 172.21.15.16:0/246449152 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f144815a5c0 0x7f14480b57f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:33.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.120+0000 7f14507b4700 1 --2- 172.21.15.16:0/246449152 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f14480ce010 0x7f14480a84c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:33.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.120+0000 7f14507b4700 1 --2- 172.21.15.16:0/246449152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14480d0ae0 0x7f14480ae880 unknown :-1 s=CLOSED pgs=2245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:33.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.120+0000 7f14507b4700 1 -- 172.21.15.16:0/246449152 >> 172.21.15.16:0/246449152 conn(0x7f14480b1fd0 msgr2=0x7f144806b160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:33.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.120+0000 7f14507b4700 1 -- 172.21.15.16:0/246449152 shutdown_connections 2023-10-16T13:44:33.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.120+0000 7f14507b4700 1 -- 172.21.15.16:0/246449152 wait complete. 2023-10-16T13:44:33.135 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T13:44:33.136 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T13:44:33.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.301+0000 7f64b1504700 1 Processor -- start 2023-10-16T13:44:33.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.301+0000 7f64b1504700 1 -- start start 2023-10-16T13:44:33.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.302+0000 7f64b1504700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f64ac15da70 0x7f64ac0d14b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:33.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.302+0000 7f64b1504700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f64ac15de50 0x7f64ac0cffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:33.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.302+0000 7f64b1504700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64ac0d2730 con 0x7f64ac0d04e0 2023-10-16T13:44:33.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.302+0000 7f64b1504700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64ac0d08f0 con 0x7f64ac15de50 2023-10-16T13:44:33.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.302+0000 7f64b1504700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64ac0d43f0 con 0x7f64ac15da70 2023-10-16T13:44:33.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.302+0000 7f64a27fc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f64ac15de50 0x7f64ac0cffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:33.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.302+0000 7f64a27fc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f64ac15de50 0x7f64ac0cffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51456/0 (socket says 172.21.15.16:51456) 2023-10-16T13:44:33.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.302+0000 7f64a27fc700 1 -- 172.21.15.16:0/3441113408 learned_addr learned my addr 172.21.15.16:0/3441113408 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:33.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.302+0000 7f64aaffd700 1 --2- 172.21.15.16:0/3441113408 >> v2:172.21.15.49:3300/0 conn(0x7f64ac15da70 0x7f64ac0d14b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:33.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.302+0000 7f64a27fc700 1 -- 172.21.15.16:0/3441113408 >> v2:172.21.15.49:3300/0 conn(0x7f64ac15da70 msgr2=0x7f64ac0d14b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:33.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.302+0000 7f64a27fc700 1 --2- 172.21.15.16:0/3441113408 >> v2:172.21.15.49:3300/0 conn(0x7f64ac15da70 0x7f64ac0d14b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:33.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.302+0000 7f64a27fc700 1 -- 172.21.15.16:0/3441113408 >> v1:172.21.15.49:6789/0 conn(0x7f64ac0d04e0 legacy=0x7f64ac1615b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:33.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.302+0000 7f64a27fc700 1 -- 172.21.15.16:0/3441113408 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64ac0d1b20 con 0x7f64ac15de50 2023-10-16T13:44:33.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.302+0000 7f64aaffd700 1 --2- 172.21.15.16:0/3441113408 >> v2:172.21.15.49:3300/0 conn(0x7f64ac15da70 0x7f64ac0d14b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:44:33.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.303+0000 7f64a27fc700 1 --2- 172.21.15.16:0/3441113408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64ac15de50 0x7f64ac0cffa0 secure :-1 s=READY pgs=2246 cs=0 l=1 rev1=1 crypto rx=0x7f64ac08b470 tx=0x7f6494009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=104fd48f10d8f0b8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:33.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.303+0000 7f64aa7fc700 1 -- 172.21.15.16:0/3441113408 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f649400b870 con 0x7f64ac15de50 2023-10-16T13:44:33.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.303+0000 7f64aa7fc700 1 -- 172.21.15.16:0/3441113408 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f649400b9d0 con 0x7f64ac15de50 2023-10-16T13:44:33.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.303+0000 7f64aa7fc700 1 -- 172.21.15.16:0/3441113408 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f649400bd60 con 0x7f64ac15de50 2023-10-16T13:44:33.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.303+0000 7f64b1504700 1 -- 172.21.15.16:0/3441113408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64ac15de50 msgr2=0x7f64ac0cffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:33.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.303+0000 7f64b1504700 1 --2- 172.21.15.16:0/3441113408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64ac15de50 0x7f64ac0cffa0 secure :-1 s=READY pgs=2246 cs=0 l=1 rev1=1 crypto rx=0x7f64ac08b470 tx=0x7f6494009bd0 comp rx=0 tx=0).stop 2023-10-16T13:44:33.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.303+0000 7f64b1504700 1 -- 172.21.15.16:0/3441113408 shutdown_connections 2023-10-16T13:44:33.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.303+0000 7f64b1504700 1 --2- 172.21.15.16:0/3441113408 >> v2:172.21.15.49:3300/0 conn(0x7f64ac15da70 0x7f64ac0d14b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:33.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.303+0000 7f64b1504700 1 --2- 172.21.15.16:0/3441113408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64ac15de50 0x7f64ac0cffa0 unknown :-1 s=CLOSED pgs=2246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:33.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.303+0000 7f64b1504700 1 -- 172.21.15.16:0/3441113408 >> 172.21.15.16:0/3441113408 conn(0x7f64ac0d36d0 msgr2=0x7f64ac0d3ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:33.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.303+0000 7f64b1504700 1 -- 172.21.15.16:0/3441113408 shutdown_connections 2023-10-16T13:44:33.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.303+0000 7f64b1504700 1 -- 172.21.15.16:0/3441113408 wait complete. 2023-10-16T13:44:33.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.305+0000 7f64b1504700 1 Processor -- start 2023-10-16T13:44:33.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.305+0000 7f64b1504700 1 -- start start 2023-10-16T13:44:33.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.305+0000 7f64b1504700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f64ac0d04e0 0x7f64ac154390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:33.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.305+0000 7f64b1504700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64ac15da70 0x7f64ac1563c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:33.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.305+0000 7f64b1504700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f64ac15de50 0x7f64ac0cc640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:33.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.305+0000 7f64b1504700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64ac0c7d40 con 0x7f64ac15da70 2023-10-16T13:44:33.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.306+0000 7f64b1504700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64ac0c7ed0 con 0x7f64ac0d04e0 2023-10-16T13:44:33.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.306+0000 7f64b1504700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64ac156900 con 0x7f64ac15de50 2023-10-16T13:44:33.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.306+0000 7f64a27fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64ac15da70 0x7f64ac1563c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:33.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.306+0000 7f64a27fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64ac15da70 0x7f64ac1563c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51462/0 (socket says 172.21.15.16:51462) 2023-10-16T13:44:33.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.306+0000 7f64a27fc700 1 -- 172.21.15.16:0/3877143445 learned_addr learned my addr 172.21.15.16:0/3877143445 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:33.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.306+0000 7f64aaffd700 1 --2- 172.21.15.16:0/3877143445 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f64ac0d04e0 0x7f64ac154390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:33.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.306+0000 7f64ab7fe700 1 --2- 172.21.15.16:0/3877143445 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f64ac15de50 0x7f64ac0cc640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:33.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.306+0000 7f64a27fc700 1 -- 172.21.15.16:0/3877143445 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f64ac15de50 msgr2=0x7f64ac0cc640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:33.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.306+0000 7f64a27fc700 1 --2- 172.21.15.16:0/3877143445 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f64ac15de50 0x7f64ac0cc640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:33.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.306+0000 7f64a27fc700 1 -- 172.21.15.16:0/3877143445 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f64ac0d04e0 msgr2=0x7f64ac154390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:33.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.306+0000 7f64a27fc700 1 --2- 172.21.15.16:0/3877143445 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f64ac0d04e0 0x7f64ac154390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:33.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.306+0000 7f64a27fc700 1 -- 172.21.15.16:0/3877143445 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6494009550 con 0x7f64ac15da70 2023-10-16T13:44:33.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.306+0000 7f64aaffd700 1 --2- 172.21.15.16:0/3877143445 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f64ac0d04e0 0x7f64ac154390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:44:33.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.306+0000 7f64a27fc700 1 --2- 172.21.15.16:0/3877143445 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64ac15da70 0x7f64ac1563c0 secure :-1 s=READY pgs=2247 cs=0 l=1 rev1=1 crypto rx=0x7f64ac08b610 tx=0x7f6494004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:33.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.306+0000 7f64a3fff700 1 -- 172.21.15.16:0/3877143445 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6494004540 con 0x7f64ac15da70 2023-10-16T13:44:33.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.306+0000 7f64a3fff700 1 -- 172.21.15.16:0/3877143445 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64940036a0 con 0x7f64ac15da70 2023-10-16T13:44:33.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.306+0000 7f64a3fff700 1 -- 172.21.15.16:0/3877143445 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6494003a30 con 0x7f64ac15da70 2023-10-16T13:44:33.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.306+0000 7f64b1504700 1 -- 172.21.15.16:0/3877143445 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64ac154a20 con 0x7f64ac15da70 2023-10-16T13:44:33.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.306+0000 7f64b1504700 1 -- 172.21.15.16:0/3877143445 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64ac154db0 con 0x7f64ac15da70 2023-10-16T13:44:33.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.307+0000 7f64a17fa700 1 -- 172.21.15.16:0/3877143445 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64840052f0 con 0x7f64ac15da70 2023-10-16T13:44:33.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.308+0000 7f64a3fff700 1 -- 172.21.15.16:0/3877143445 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 126) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6494021070 con 0x7f64ac15da70 2023-10-16T13:44:33.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.308+0000 7f64a3fff700 1 --2- 172.21.15.16:0/3877143445 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f648c06b020 0x7f648c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:33.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.308+0000 7f64a3fff700 1 -- 172.21.15.16:0/3877143445 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 6423+0+0 (secure 0 0 0) 0x7f649408b0c0 con 0x7f64ac15da70 2023-10-16T13:44:33.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.308+0000 7f64aaffd700 1 --2- 172.21.15.16:0/3877143445 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f648c06b020 0x7f648c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:33.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.309+0000 7f64aaffd700 1 --2- 172.21.15.16:0/3877143445 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f648c06b020 0x7f648c06d4e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f649c0099e0 tx=0x7f649c008040 comp rx=0 tx=0).ready entity=mgr.10665 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:33.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.313+0000 7f64a3fff700 1 -- 172.21.15.16:0/3877143445 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6494055ea0 con 0x7f64ac15da70 2023-10-16T13:44:33.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.516+0000 7f64a17fa700 1 -- 172.21.15.16:0/3877143445 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6484005b20 con 0x7f64ac15da70 2023-10-16T13:44:33.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.517+0000 7f64a3fff700 1 -- 172.21.15.16:0/3877143445 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v198) v1 ==== 93+0+10 (secure 0 0 0) 0x7f64940150a0 con 0x7f64ac15da70 2023-10-16T13:44:33.520 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T13:44:33.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.519+0000 7f64a17fa700 1 -- 172.21.15.16:0/3877143445 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f648c06b020 msgr2=0x7f648c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:33.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.519+0000 7f64a17fa700 1 --2- 172.21.15.16:0/3877143445 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f648c06b020 0x7f648c06d4e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f649c0099e0 tx=0x7f649c008040 comp rx=0 tx=0).stop 2023-10-16T13:44:33.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.519+0000 7f64a17fa700 1 -- 172.21.15.16:0/3877143445 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64ac15da70 msgr2=0x7f64ac1563c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:33.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.519+0000 7f64a17fa700 1 --2- 172.21.15.16:0/3877143445 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64ac15da70 0x7f64ac1563c0 secure :-1 s=READY pgs=2247 cs=0 l=1 rev1=1 crypto rx=0x7f64ac08b610 tx=0x7f6494004750 comp rx=0 tx=0).stop 2023-10-16T13:44:33.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.519+0000 7f64a17fa700 1 -- 172.21.15.16:0/3877143445 shutdown_connections 2023-10-16T13:44:33.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.519+0000 7f64a17fa700 1 --2- 172.21.15.16:0/3877143445 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f648c06b020 0x7f648c06d4e0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:33.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.519+0000 7f64a17fa700 1 --2- 172.21.15.16:0/3877143445 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f64ac15de50 0x7f64ac0cc640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:33.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.519+0000 7f64a17fa700 1 --2- 172.21.15.16:0/3877143445 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f64ac0d04e0 0x7f64ac154390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:33.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.519+0000 7f64a17fa700 1 --2- 172.21.15.16:0/3877143445 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64ac15da70 0x7f64ac1563c0 unknown :-1 s=CLOSED pgs=2247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:33.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.519+0000 7f64a17fa700 1 -- 172.21.15.16:0/3877143445 >> 172.21.15.16:0/3877143445 conn(0x7f64ac0d36d0 msgr2=0x7f64ac0b1940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:33.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.519+0000 7f64a17fa700 1 -- 172.21.15.16:0/3877143445 shutdown_connections 2023-10-16T13:44:33.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.519+0000 7f64a17fa700 1 -- 172.21.15.16:0/3877143445 wait complete. 2023-10-16T13:44:33.534 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention' 2023-10-16T13:44:33.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.658+0000 7faaef295700 1 Processor -- start 2023-10-16T13:44:33.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.658+0000 7faaef295700 1 -- start start 2023-10-16T13:44:33.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.659+0000 7faaef295700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faae815db40 0x7faae80cb980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:33.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.659+0000 7faaef295700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7faae806a7e0 0x7faae806ac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:33.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.659+0000 7faaef295700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faae80cedb0 con 0x7faae80ce160 2023-10-16T13:44:33.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.659+0000 7faaef295700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faae806b1a0 con 0x7faae806a7e0 2023-10-16T13:44:33.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.659+0000 7faaef295700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faae815df20 con 0x7faae815db40 2023-10-16T13:44:33.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.659+0000 7faaec830700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7faae806a7e0 0x7faae806ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:33.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.659+0000 7faaed031700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faae815db40 0x7faae80cb980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:33.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.659+0000 7faaed832700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7faae80ce160 0x7faae80ce5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:47928/0 (socket says 172.21.15.16:47928) 2023-10-16T13:44:33.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.659+0000 7faaec830700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7faae806a7e0 0x7faae806ac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:50530/0 (socket says 172.21.15.16:50530) 2023-10-16T13:44:33.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.659+0000 7faaed832700 1 -- 172.21.15.16:0/1059275535 learned_addr learned my addr 172.21.15.16:0/1059275535 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:44:33.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.660+0000 7faaec830700 1 -- 172.21.15.16:0/1059275535 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faae815db40 msgr2=0x7faae80cb980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:33.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.660+0000 7faaec830700 1 --2- 172.21.15.16:0/1059275535 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faae815db40 0x7faae80cb980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:33.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.660+0000 7faaec830700 1 -- 172.21.15.16:0/1059275535 >> v1:172.21.15.49:6789/0 conn(0x7faae80ce160 legacy=0x7faae80ce5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:33.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.660+0000 7faadffff700 1 -- 172.21.15.16:0/1059275535 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2513784620 0 0) 0x7faae80cedb0 con 0x7faae80ce160 2023-10-16T13:44:33.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.660+0000 7faaec830700 1 -- 172.21.15.16:0/1059275535 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faae80d12c0 con 0x7faae806a7e0 2023-10-16T13:44:33.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.660+0000 7faaec830700 1 --2- 172.21.15.16:0/1059275535 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faae806a7e0 0x7faae806ac60 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7faae4009840 tx=0x7faae4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=48ec512c01da4c1e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:33.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.660+0000 7faadffff700 1 -- 172.21.15.16:0/1059275535 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faae4011830 con 0x7faae806a7e0 2023-10-16T13:44:33.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.660+0000 7faadffff700 1 -- 172.21.15.16:0/1059275535 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faae4011990 con 0x7faae806a7e0 2023-10-16T13:44:33.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.660+0000 7faadffff700 1 -- 172.21.15.16:0/1059275535 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faae4011d20 con 0x7faae806a7e0 2023-10-16T13:44:33.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.661+0000 7faaef295700 1 -- 172.21.15.16:0/1059275535 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faae806a7e0 msgr2=0x7faae806ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:33.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.661+0000 7faaef295700 1 --2- 172.21.15.16:0/1059275535 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faae806a7e0 0x7faae806ac60 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7faae4009840 tx=0x7faae4009b50 comp rx=0 tx=0).stop 2023-10-16T13:44:33.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.661+0000 7faaef295700 1 -- 172.21.15.16:0/1059275535 shutdown_connections 2023-10-16T13:44:33.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.661+0000 7faaef295700 1 --2- 172.21.15.16:0/1059275535 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faae815db40 0x7faae80cb980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:33.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.661+0000 7faaef295700 1 --2- 172.21.15.16:0/1059275535 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faae806a7e0 0x7faae806ac60 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:33.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.661+0000 7faaef295700 1 -- 172.21.15.16:0/1059275535 >> 172.21.15.16:0/1059275535 conn(0x7faae80b1fd0 msgr2=0x7faae80b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:33.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.661+0000 7faaef295700 1 -- 172.21.15.16:0/1059275535 shutdown_connections 2023-10-16T13:44:33.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.661+0000 7faaef295700 1 -- 172.21.15.16:0/1059275535 wait complete. 2023-10-16T13:44:33.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.662+0000 7faaef295700 1 Processor -- start 2023-10-16T13:44:33.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.662+0000 7faaef295700 1 -- start start 2023-10-16T13:44:33.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.663+0000 7faaef295700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faae806a7e0 0x7faae81f25f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:33.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.663+0000 7faaef295700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faae80ce160 0x7faae815edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:33.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.663+0000 7faaef295700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faae815db40 0x7faae80c6430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:33.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.663+0000 7faaef295700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faae8160220 con 0x7faae815db40 2023-10-16T13:44:33.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.663+0000 7faaef295700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faae815f6a0 con 0x7faae806a7e0 2023-10-16T13:44:33.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.663+0000 7faaef295700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faae815f980 con 0x7faae80ce160 2023-10-16T13:44:33.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.663+0000 7faaed832700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faae815db40 0x7faae80c6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:33.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.663+0000 7faaed832700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faae815db40 0x7faae80c6430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51478/0 (socket says 172.21.15.16:51478) 2023-10-16T13:44:33.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.663+0000 7faaed832700 1 -- 172.21.15.16:0/775108412 learned_addr learned my addr 172.21.15.16:0/775108412 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:33.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.663+0000 7faaed031700 1 --2- 172.21.15.16:0/775108412 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faae806a7e0 0x7faae81f25f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:33.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.663+0000 7faaec830700 1 --2- 172.21.15.16:0/775108412 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faae80ce160 0x7faae815edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:33.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.663+0000 7faaed832700 1 -- 172.21.15.16:0/775108412 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faae80ce160 msgr2=0x7faae815edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:33.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.663+0000 7faaed832700 1 --2- 172.21.15.16:0/775108412 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faae80ce160 0x7faae815edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:33.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.663+0000 7faaed832700 1 -- 172.21.15.16:0/775108412 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faae806a7e0 msgr2=0x7faae81f25f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:33.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.663+0000 7faaed832700 1 --2- 172.21.15.16:0/775108412 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faae806a7e0 0x7faae81f25f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:33.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.663+0000 7faaed832700 1 -- 172.21.15.16:0/775108412 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faae40094a0 con 0x7faae815db40 2023-10-16T13:44:33.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.663+0000 7faaec830700 1 --2- 172.21.15.16:0/775108412 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faae80ce160 0x7faae815edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:44:33.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.664+0000 7faaed832700 1 --2- 172.21.15.16:0/775108412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faae815db40 0x7faae80c6430 secure :-1 s=READY pgs=2248 cs=0 l=1 rev1=1 crypto rx=0x7faae000caf0 tx=0x7faae000ce00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:33.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.664+0000 7faaddffb700 1 -- 172.21.15.16:0/775108412 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faae000aaa0 con 0x7faae815db40 2023-10-16T13:44:33.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.664+0000 7faaddffb700 1 -- 172.21.15.16:0/775108412 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faae000ac20 con 0x7faae815db40 2023-10-16T13:44:33.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.664+0000 7faaef295700 1 -- 172.21.15.16:0/775108412 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faae815fc60 con 0x7faae815db40 2023-10-16T13:44:33.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.664+0000 7faaddffb700 1 -- 172.21.15.16:0/775108412 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faae00070d0 con 0x7faae815db40 2023-10-16T13:44:33.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.664+0000 7faaef295700 1 -- 172.21.15.16:0/775108412 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faae81f2b30 con 0x7faae815db40 2023-10-16T13:44:33.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.665+0000 7faaef295700 1 -- 172.21.15.16:0/775108412 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faae804ebb0 con 0x7faae815db40 2023-10-16T13:44:33.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.665+0000 7faaddffb700 1 -- 172.21.15.16:0/775108412 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 126) v1 ==== 89331+0+0 (secure 0 0 0) 0x7faae0007230 con 0x7faae815db40 2023-10-16T13:44:33.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.666+0000 7faaddffb700 1 --2- 172.21.15.16:0/775108412 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7faacc06b020 0x7faacc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:33.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.666+0000 7faaddffb700 1 -- 172.21.15.16:0/775108412 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 6423+0+0 (secure 0 0 0) 0x7faae00896b0 con 0x7faae815db40 2023-10-16T13:44:33.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.666+0000 7faaed031700 1 --2- 172.21.15.16:0/775108412 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7faacc06b020 0x7faacc06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:33.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.666+0000 7faaed031700 1 --2- 172.21.15.16:0/775108412 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7faacc06b020 0x7faacc06d4e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7faad8009be0 tx=0x7faad8009210 comp rx=0 tx=0).ready entity=mgr.10665 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:33.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.670+0000 7faaddffb700 1 -- 172.21.15.16:0/775108412 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7faae0054410 con 0x7faae815db40 2023-10-16T13:44:33.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:33.959+0000 7faaef295700 1 -- 172.21.15.16:0/775108412 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]} v 0) v1 -- 0x7faae80cbca0 con 0x7faae815db40 2023-10-16T13:44:34.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:34.753+0000 7faaddffb700 1 -- 172.21.15.16:0/775108412 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]}]=0 v3049) v1 ==== 166+0+0 (secure 0 0 0) 0x7faae0057a30 con 0x7faae815db40 2023-10-16T13:44:34.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:34.757+0000 7faaef295700 1 -- 172.21.15.16:0/775108412 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7faacc06b020 msgr2=0x7faacc06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:34.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:34.757+0000 7faaef295700 1 --2- 172.21.15.16:0/775108412 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7faacc06b020 0x7faacc06d4e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7faad8009be0 tx=0x7faad8009210 comp rx=0 tx=0).stop 2023-10-16T13:44:34.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:34.757+0000 7faaef295700 1 -- 172.21.15.16:0/775108412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faae815db40 msgr2=0x7faae80c6430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:34.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:34.757+0000 7faaef295700 1 --2- 172.21.15.16:0/775108412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faae815db40 0x7faae80c6430 secure :-1 s=READY pgs=2248 cs=0 l=1 rev1=1 crypto rx=0x7faae000caf0 tx=0x7faae000ce00 comp rx=0 tx=0).stop 2023-10-16T13:44:34.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:34.757+0000 7faaef295700 1 -- 172.21.15.16:0/775108412 shutdown_connections 2023-10-16T13:44:34.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:34.757+0000 7faaef295700 1 --2- 172.21.15.16:0/775108412 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7faacc06b020 0x7faacc06d4e0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:34.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:34.757+0000 7faaef295700 1 --2- 172.21.15.16:0/775108412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faae815db40 0x7faae80c6430 unknown :-1 s=CLOSED pgs=2248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:34.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:34.757+0000 7faaef295700 1 --2- 172.21.15.16:0/775108412 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faae806a7e0 0x7faae81f25f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:34.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:34.757+0000 7faaef295700 1 --2- 172.21.15.16:0/775108412 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faae80ce160 0x7faae815edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:34.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:34.757+0000 7faaef295700 1 -- 172.21.15.16:0/775108412 >> 172.21.15.16:0/775108412 conn(0x7faae80b1fd0 msgr2=0x7faae80cd280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:34.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:34.757+0000 7faaef295700 1 -- 172.21.15.16:0/775108412 shutdown_connections 2023-10-16T13:44:34.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:34.757+0000 7faaef295700 1 -- 172.21.15.16:0/775108412 wait complete. 2023-10-16T13:44:34.774 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-16T13:44:34.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:34.907+0000 7f3223824700 1 Processor -- start 2023-10-16T13:44:34.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:34.908+0000 7f3223824700 1 -- start start 2023-10-16T13:44:34.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:34.908+0000 7f3223824700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f321c15b790 0x7f321c0cda60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:34.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:34.908+0000 7f3223824700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f321c06e070 0x7f321c0ce850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:34.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:34.908+0000 7f3223824700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f321c0ce0e0 con 0x7f321c15bb70 2023-10-16T13:44:34.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:34.908+0000 7f3223824700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f321c06a7e0 con 0x7f321c06e070 2023-10-16T13:44:34.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:34.908+0000 7f3223824700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f321c15c410 con 0x7f321c15b790 2023-10-16T13:44:34.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:34.909+0000 7f3221dc1700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f321c06e070 0x7f321c0ce850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:34.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:34.909+0000 7f3221dc1700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f321c06e070 0x7f321c0ce850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51486/0 (socket says 172.21.15.16:51486) 2023-10-16T13:44:34.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:34.909+0000 7f3221dc1700 1 -- 172.21.15.16:0/998326287 learned_addr learned my addr 172.21.15.16:0/998326287 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:34.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:34.909+0000 7f32215c0700 1 --2- 172.21.15.16:0/998326287 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f321c15b790 0x7f321c0cda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:34.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:34.909+0000 7f3221dc1700 1 -- 172.21.15.16:0/998326287 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f321c15b790 msgr2=0x7f321c0cda60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:34.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:34.909+0000 7f3221dc1700 1 --2- 172.21.15.16:0/998326287 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f321c15b790 0x7f321c0cda60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:34.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:34.909+0000 7f3221dc1700 1 -- 172.21.15.16:0/998326287 >> v1:172.21.15.16:6789/0 conn(0x7f321c15bb70 legacy=0x7f321c06d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:34.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:34.909+0000 7f3221dc1700 1 -- 172.21.15.16:0/998326287 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f321c15c6f0 con 0x7f321c06e070 2023-10-16T13:44:34.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:34.909+0000 7f3221dc1700 1 --2- 172.21.15.16:0/998326287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f321c06e070 0x7f321c0ce850 secure :-1 s=READY pgs=2250 cs=0 l=1 rev1=1 crypto rx=0x7f3214009e50 tx=0x7f321400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=333c74123edfbcb4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:34.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:34.909+0000 7f3213fff700 1 -- 172.21.15.16:0/998326287 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f321400e070 con 0x7f321c06e070 2023-10-16T13:44:34.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:34.909+0000 7f3213fff700 1 -- 172.21.15.16:0/998326287 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f321400b910 con 0x7f321c06e070 2023-10-16T13:44:34.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:34.909+0000 7f3213fff700 1 -- 172.21.15.16:0/998326287 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f321400bcc0 con 0x7f321c06e070 2023-10-16T13:44:34.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:34.910+0000 7f3223824700 1 -- 172.21.15.16:0/998326287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f321c06e070 msgr2=0x7f321c0ce850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:34.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:34.910+0000 7f3223824700 1 --2- 172.21.15.16:0/998326287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f321c06e070 0x7f321c0ce850 secure :-1 s=READY pgs=2250 cs=0 l=1 rev1=1 crypto rx=0x7f3214009e50 tx=0x7f321400b330 comp rx=0 tx=0).stop 2023-10-16T13:44:34.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:34.910+0000 7f3223824700 1 -- 172.21.15.16:0/998326287 shutdown_connections 2023-10-16T13:44:34.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:34.910+0000 7f3223824700 1 --2- 172.21.15.16:0/998326287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f321c06e070 0x7f321c0ce850 unknown :-1 s=CLOSED pgs=2250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:34.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:34.910+0000 7f3223824700 1 --2- 172.21.15.16:0/998326287 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f321c15b790 0x7f321c0cda60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:34.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:34.910+0000 7f3223824700 1 -- 172.21.15.16:0/998326287 >> 172.21.15.16:0/998326287 conn(0x7f321c0cc5e0 msgr2=0x7f321c0cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:34.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:34.910+0000 7f3223824700 1 -- 172.21.15.16:0/998326287 shutdown_connections 2023-10-16T13:44:34.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:34.910+0000 7f3223824700 1 -- 172.21.15.16:0/998326287 wait complete. 2023-10-16T13:44:34.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:34.912+0000 7f3223824700 1 Processor -- start 2023-10-16T13:44:34.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:34.912+0000 7f3223824700 1 -- start start 2023-10-16T13:44:34.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:34.913+0000 7f3223824700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f321c06e070 0x7f321c0c9760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:34.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:34.913+0000 7f3223824700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f321c15b790 0x7f321c153080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:34.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:34.913+0000 7f3223824700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f321c15bb70 0x7f321c0c6050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:34.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:34.913+0000 7f3223824700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f321c15a6b0 con 0x7f321c15b790 2023-10-16T13:44:34.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:34.913+0000 7f3223824700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f321c15aad0 con 0x7f321c06e070 2023-10-16T13:44:34.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:34.913+0000 7f3223824700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f321c1535c0 con 0x7f321c15bb70 2023-10-16T13:44:34.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:34.913+0000 7f3220dbf700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f321c15b790 0x7f321c153080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:34.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:34.913+0000 7f3220dbf700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f321c15b790 0x7f321c153080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51490/0 (socket says 172.21.15.16:51490) 2023-10-16T13:44:34.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:34.913+0000 7f3220dbf700 1 -- 172.21.15.16:0/452634319 learned_addr learned my addr 172.21.15.16:0/452634319 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:34.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:34.913+0000 7f32215c0700 1 --2- 172.21.15.16:0/452634319 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f321c06e070 0x7f321c0c9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:34.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:34.913+0000 7f3221dc1700 1 --2- 172.21.15.16:0/452634319 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f321c15bb70 0x7f321c0c6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:34.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:34.913+0000 7f3220dbf700 1 -- 172.21.15.16:0/452634319 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f321c15bb70 msgr2=0x7f321c0c6050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:34.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:34.913+0000 7f3220dbf700 1 --2- 172.21.15.16:0/452634319 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f321c15bb70 0x7f321c0c6050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:34.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:34.913+0000 7f3220dbf700 1 -- 172.21.15.16:0/452634319 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f321c06e070 msgr2=0x7f321c0c9760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:34.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:34.913+0000 7f3220dbf700 1 --2- 172.21.15.16:0/452634319 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f321c06e070 0x7f321c0c9760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:34.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:34.913+0000 7f3220dbf700 1 -- 172.21.15.16:0/452634319 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3214009b30 con 0x7f321c15b790 2023-10-16T13:44:34.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:34.913+0000 7f3220dbf700 1 --2- 172.21.15.16:0/452634319 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f321c15b790 0x7f321c153080 secure :-1 s=READY pgs=2251 cs=0 l=1 rev1=1 crypto rx=0x7f320c00c450 tx=0x7f320c00c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:34.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:34.913+0000 7f3211ffb700 1 -- 172.21.15.16:0/452634319 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f320c0052d0 con 0x7f321c15b790 2023-10-16T13:44:34.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:34.913+0000 7f3211ffb700 1 -- 172.21.15.16:0/452634319 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f320c005430 con 0x7f321c15b790 2023-10-16T13:44:34.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:34.913+0000 7f3223824700 1 -- 172.21.15.16:0/452634319 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f321c0c6590 con 0x7f321c15b790 2023-10-16T13:44:34.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:34.913+0000 7f3211ffb700 1 -- 172.21.15.16:0/452634319 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f320c0038c0 con 0x7f321c15b790 2023-10-16T13:44:34.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:34.914+0000 7f3223824700 1 -- 172.21.15.16:0/452634319 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f321c159d10 con 0x7f321c15b790 2023-10-16T13:44:34.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:34.914+0000 7f3223824700 1 -- 172.21.15.16:0/452634319 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f321c066f10 con 0x7f321c15b790 2023-10-16T13:44:34.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:34.915+0000 7f3211ffb700 1 -- 172.21.15.16:0/452634319 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 126) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f320c003a20 con 0x7f321c15b790 2023-10-16T13:44:34.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:34.915+0000 7f3211ffb700 1 --2- 172.21.15.16:0/452634319 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f320006b020 0x7f320006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:34.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:34.915+0000 7f3211ffb700 1 -- 172.21.15.16:0/452634319 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 6423+0+0 (secure 0 0 0) 0x7f320c088cb0 con 0x7f321c15b790 2023-10-16T13:44:34.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:34.915+0000 7f32215c0700 1 --2- 172.21.15.16:0/452634319 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f320006b020 0x7f320006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:34.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:34.916+0000 7f32215c0700 1 --2- 172.21.15.16:0/452634319 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f320006b020 0x7f320006d4e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f321c157f40 tx=0x7f3218009210 comp rx=0 tx=0).ready entity=mgr.10665 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:34.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:34.920+0000 7f3211ffb700 1 -- 172.21.15.16:0/452634319 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f320c053a10 con 0x7f321c15b790 2023-10-16T13:44:35.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.127+0000 7f3223824700 1 -- 172.21.15.16:0/452634319 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f321c15a7f0 con 0x7f321c15b790 2023-10-16T13:44:35.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.130+0000 7f3211ffb700 1 -- 172.21.15.16:0/452634319 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v70)=0 v70) v1 ==== 122+0+0 (secure 0 0 0) 0x7f321c15a7f0 con 0x7f321c15b790 2023-10-16T13:44:35.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.133+0000 7f3223824700 1 -- 172.21.15.16:0/452634319 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f320006b020 msgr2=0x7f320006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:35.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.133+0000 7f3223824700 1 --2- 172.21.15.16:0/452634319 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f320006b020 0x7f320006d4e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f321c157f40 tx=0x7f3218009210 comp rx=0 tx=0).stop 2023-10-16T13:44:35.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.133+0000 7f3223824700 1 -- 172.21.15.16:0/452634319 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f321c15b790 msgr2=0x7f321c153080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:35.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.133+0000 7f3223824700 1 --2- 172.21.15.16:0/452634319 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f321c15b790 0x7f321c153080 secure :-1 s=READY pgs=2251 cs=0 l=1 rev1=1 crypto rx=0x7f320c00c450 tx=0x7f320c00c760 comp rx=0 tx=0).stop 2023-10-16T13:44:35.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.133+0000 7f3223824700 1 -- 172.21.15.16:0/452634319 shutdown_connections 2023-10-16T13:44:35.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.133+0000 7f3223824700 1 --2- 172.21.15.16:0/452634319 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f320006b020 0x7f320006d4e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:35.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.133+0000 7f3223824700 1 --2- 172.21.15.16:0/452634319 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f321c15bb70 0x7f321c0c6050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:35.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.133+0000 7f3223824700 1 --2- 172.21.15.16:0/452634319 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f321c06e070 0x7f321c0c9760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:35.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.133+0000 7f3223824700 1 --2- 172.21.15.16:0/452634319 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f321c15b790 0x7f321c153080 unknown :-1 s=CLOSED pgs=2251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:35.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.133+0000 7f3223824700 1 -- 172.21.15.16:0/452634319 >> 172.21.15.16:0/452634319 conn(0x7f321c0cc5e0 msgr2=0x7f321c157220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:35.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.133+0000 7f3223824700 1 -- 172.21.15.16:0/452634319 shutdown_connections 2023-10-16T13:44:35.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.133+0000 7f3223824700 1 -- 172.21.15.16:0/452634319 wait complete. 2023-10-16T13:44:35.147 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T13:44:35.148 DEBUG:teuthology.orchestra.run.smithi016:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T13:44:35.194 DEBUG:teuthology.orchestra.run.smithi049:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T13:44:35.241 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T13:44:35.241 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T13:44:35.242 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:44:35.259 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16287364k,nr_inodes=4071841,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20251 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3261172k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T13:44:35.260 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T13:44:35.260 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:44:35.280 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284204k,nr_inodes=4071051,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=753 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260540k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T13:44:35.280 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T13:44:35.281 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:44:35.317 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16287364k,nr_inodes=4071841,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20251 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3261172k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T13:44:35.318 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T13:44:35.318 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:44:35.339 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284204k,nr_inodes=4071051,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=753 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260540k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T13:44:35.340 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-16T13:44:35.340 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:44:35.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.480+0000 7f14dd556700 1 Processor -- start 2023-10-16T13:44:35.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.480+0000 7f14dd556700 1 -- start start 2023-10-16T13:44:35.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.481+0000 7f14dd556700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f14d80d0750 0x7f14d8168450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:35.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.481+0000 7f14dd556700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14d806ba00 con 0x7f14d815d8d0 2023-10-16T13:44:35.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.481+0000 7f14dd556700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14d80d26d0 con 0x7f14d80d41c0 2023-10-16T13:44:35.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.481+0000 7f14dd556700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14d80d23f0 con 0x7f14d80d0750 2023-10-16T13:44:35.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.481+0000 7f14d77fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f14d80d0750 0x7f14d8168450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:35.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.481+0000 7f14d77fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f14d80d0750 0x7f14d8168450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51498/0 (socket says 172.21.15.16:51498) 2023-10-16T13:44:35.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.481+0000 7f14d77fe700 1 -- 172.21.15.16:0/4024700191 learned_addr learned my addr 172.21.15.16:0/4024700191 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:35.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.481+0000 7f14d77fe700 1 -- 172.21.15.16:0/4024700191 >> v1:172.21.15.49:6789/0 conn(0x7f14d80d41c0 legacy=0x7f14d80d0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:35.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.481+0000 7f14d77fe700 1 -- 172.21.15.16:0/4024700191 >> v1:172.21.15.16:6789/0 conn(0x7f14d815d8d0 legacy=0x7f14d80d1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:35.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.481+0000 7f14d77fe700 1 -- 172.21.15.16:0/4024700191 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14d806bca0 con 0x7f14d80d0750 2023-10-16T13:44:35.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.481+0000 7f14d77fe700 1 --2- 172.21.15.16:0/4024700191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14d80d0750 0x7f14d8168450 secure :-1 s=READY pgs=2253 cs=0 l=1 rev1=1 crypto rx=0x7f14d0009e50 tx=0x7f14d000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1cb0bfd6bd74567 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:35.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.481+0000 7f14d5ffb700 1 -- 172.21.15.16:0/4024700191 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d000e070 con 0x7f14d80d0750 2023-10-16T13:44:35.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.482+0000 7f14d5ffb700 1 -- 172.21.15.16:0/4024700191 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14d000b9c0 con 0x7f14d80d0750 2023-10-16T13:44:35.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.482+0000 7f14d5ffb700 1 -- 172.21.15.16:0/4024700191 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d000bd70 con 0x7f14d80d0750 2023-10-16T13:44:35.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.482+0000 7f14dd556700 1 -- 172.21.15.16:0/4024700191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14d80d0750 msgr2=0x7f14d8168450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:35.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.482+0000 7f14dd556700 1 --2- 172.21.15.16:0/4024700191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14d80d0750 0x7f14d8168450 secure :-1 s=READY pgs=2253 cs=0 l=1 rev1=1 crypto rx=0x7f14d0009e50 tx=0x7f14d000b330 comp rx=0 tx=0).stop 2023-10-16T13:44:35.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.482+0000 7f14dd556700 1 -- 172.21.15.16:0/4024700191 shutdown_connections 2023-10-16T13:44:35.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.482+0000 7f14dd556700 1 --2- 172.21.15.16:0/4024700191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14d80d0750 0x7f14d8168450 unknown :-1 s=CLOSED pgs=2253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:35.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.482+0000 7f14dd556700 1 -- 172.21.15.16:0/4024700191 >> 172.21.15.16:0/4024700191 conn(0x7f14d80d3630 msgr2=0x7f14d80d3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:35.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.482+0000 7f14dd556700 1 -- 172.21.15.16:0/4024700191 shutdown_connections 2023-10-16T13:44:35.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.482+0000 7f14dd556700 1 -- 172.21.15.16:0/4024700191 wait complete. 2023-10-16T13:44:35.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.483+0000 7f14dd556700 1 Processor -- start 2023-10-16T13:44:35.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.483+0000 7f14dd556700 1 -- start start 2023-10-16T13:44:35.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.484+0000 7f14dd556700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f14d80d0750 0x7f14d80a0c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:35.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.484+0000 7f14dd556700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14d80d41c0 0x7f14d80b0470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:35.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.484+0000 7f14dd556700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f14d815d8d0 0x7f14d81516a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:35.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.484+0000 7f14dd556700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14d809f990 con 0x7f14d80d41c0 2023-10-16T13:44:35.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.484+0000 7f14dd556700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14d809fbf0 con 0x7f14d80d0750 2023-10-16T13:44:35.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.484+0000 7f14dd556700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14d809fed0 con 0x7f14d815d8d0 2023-10-16T13:44:35.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.484+0000 7f14d67fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14d80d41c0 0x7f14d80b0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:35.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.484+0000 7f14d67fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14d80d41c0 0x7f14d80b0470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51502/0 (socket says 172.21.15.16:51502) 2023-10-16T13:44:35.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.484+0000 7f14d67fc700 1 -- 172.21.15.16:0/102555509 learned_addr learned my addr 172.21.15.16:0/102555509 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:35.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.485+0000 7f14d6ffd700 1 --2- 172.21.15.16:0/102555509 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f14d80d0750 0x7f14d80a0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:35.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.485+0000 7f14d77fe700 1 --2- 172.21.15.16:0/102555509 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f14d815d8d0 0x7f14d81516a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:35.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.485+0000 7f14d67fc700 1 -- 172.21.15.16:0/102555509 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f14d815d8d0 msgr2=0x7f14d81516a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:35.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.485+0000 7f14d67fc700 1 --2- 172.21.15.16:0/102555509 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f14d815d8d0 0x7f14d81516a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:35.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.485+0000 7f14d67fc700 1 -- 172.21.15.16:0/102555509 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f14d80d0750 msgr2=0x7f14d80a0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:35.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.485+0000 7f14d67fc700 1 --2- 172.21.15.16:0/102555509 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f14d80d0750 0x7f14d80a0c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:35.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.485+0000 7f14d67fc700 1 -- 172.21.15.16:0/102555509 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14d0009b30 con 0x7f14d80d41c0 2023-10-16T13:44:35.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.485+0000 7f14d6ffd700 1 --2- 172.21.15.16:0/102555509 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f14d80d0750 0x7f14d80a0c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:44:35.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.485+0000 7f14d67fc700 1 --2- 172.21.15.16:0/102555509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14d80d41c0 0x7f14d80b0470 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 crypto rx=0x7f14c800c4e0 tx=0x7f14c800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:35.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.485+0000 7f14bf7fe700 1 -- 172.21.15.16:0/102555509 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14c80052d0 con 0x7f14d80d41c0 2023-10-16T13:44:35.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.485+0000 7f14bf7fe700 1 -- 172.21.15.16:0/102555509 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14c8005430 con 0x7f14d80d41c0 2023-10-16T13:44:35.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.485+0000 7f14bf7fe700 1 -- 172.21.15.16:0/102555509 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14c80038c0 con 0x7f14d80d41c0 2023-10-16T13:44:35.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.485+0000 7f14dd556700 1 -- 172.21.15.16:0/102555509 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14d80a01b0 con 0x7f14d80d41c0 2023-10-16T13:44:35.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.485+0000 7f14dd556700 1 -- 172.21.15.16:0/102555509 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14d80a0770 con 0x7f14d80d41c0 2023-10-16T13:44:35.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.486+0000 7f14dd556700 1 -- 172.21.15.16:0/102555509 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14d80a0990 con 0x7f14d80d41c0 2023-10-16T13:44:35.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.487+0000 7f14bf7fe700 1 -- 172.21.15.16:0/102555509 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 126) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f14c8003a20 con 0x7f14d80d41c0 2023-10-16T13:44:35.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.487+0000 7f14bf7fe700 1 --2- 172.21.15.16:0/102555509 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f14c406b020 0x7f14c406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:35.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.487+0000 7f14bf7fe700 1 -- 172.21.15.16:0/102555509 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 6423+0+0 (secure 0 0 0) 0x7f14c8088f00 con 0x7f14d80d41c0 2023-10-16T13:44:35.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.487+0000 7f14d6ffd700 1 --2- 172.21.15.16:0/102555509 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f14c406b020 0x7f14c406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:35.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.487+0000 7f14d6ffd700 1 --2- 172.21.15.16:0/102555509 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f14c406b020 0x7f14c406d4e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f14cc000f60 tx=0x7f14cc009040 comp rx=0 tx=0).ready entity=mgr.10665 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:35.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.492+0000 7f14bf7fe700 1 -- 172.21.15.16:0/102555509 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f14c8053bd0 con 0x7f14d80d41c0 2023-10-16T13:44:35.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.746+0000 7f14dd556700 1 -- 172.21.15.16:0/102555509 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f14d804ebb0 con 0x7f14d80d41c0 2023-10-16T13:44:35.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.746+0000 7f14bf7fe700 1 -- 172.21.15.16:0/102555509 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+3700 (secure 0 0 0) 0x7f14c8017090 con 0x7f14d80d41c0 2023-10-16T13:44:35.749 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:44:35.749 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":97,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8306,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2415675492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":2415675492},{"type":"v1","addr":"172.21.15.16:6837","nonce":2415675492}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":90},{"gid":8315,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/4031561298","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":4031561298},{"type":"v1","addr":"172.21.15.49:6834","nonce":4031561298}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":90},{"gid":8318,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/2853801536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":2853801536},{"type":"v1","addr":"172.21.15.49:6835","nonce":2853801536}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":90},{"gid":10806,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/3221347971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":3221347971},{"type":"v1","addr":"172.21.15.16:6836","nonce":3221347971}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97}],"filesystems":[]} 2023-10-16T13:44:35.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.749+0000 7f14dd556700 1 -- 172.21.15.16:0/102555509 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f14c406b020 msgr2=0x7f14c406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:35.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.749+0000 7f14dd556700 1 --2- 172.21.15.16:0/102555509 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f14c406b020 0x7f14c406d4e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f14cc000f60 tx=0x7f14cc009040 comp rx=0 tx=0).stop 2023-10-16T13:44:35.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.749+0000 7f14dd556700 1 -- 172.21.15.16:0/102555509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14d80d41c0 msgr2=0x7f14d80b0470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:35.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.749+0000 7f14dd556700 1 --2- 172.21.15.16:0/102555509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14d80d41c0 0x7f14d80b0470 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 crypto rx=0x7f14c800c4e0 tx=0x7f14c800c8a0 comp rx=0 tx=0).stop 2023-10-16T13:44:35.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.749+0000 7f14dd556700 1 -- 172.21.15.16:0/102555509 shutdown_connections 2023-10-16T13:44:35.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.749+0000 7f14dd556700 1 --2- 172.21.15.16:0/102555509 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f14c406b020 0x7f14c406d4e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:35.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.749+0000 7f14dd556700 1 --2- 172.21.15.16:0/102555509 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f14d815d8d0 0x7f14d81516a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:35.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.749+0000 7f14dd556700 1 --2- 172.21.15.16:0/102555509 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f14d80d0750 0x7f14d80a0c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:35.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.749+0000 7f14dd556700 1 --2- 172.21.15.16:0/102555509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14d80d41c0 0x7f14d80b0470 unknown :-1 s=CLOSED pgs=2254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:35.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.749+0000 7f14dd556700 1 -- 172.21.15.16:0/102555509 >> 172.21.15.16:0/102555509 conn(0x7f14d80d3630 msgr2=0x7f14d80d1c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:35.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.749+0000 7f14dd556700 1 -- 172.21.15.16:0/102555509 shutdown_connections 2023-10-16T13:44:35.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.749+0000 7f14dd556700 1 -- 172.21.15.16:0/102555509 wait complete. 2023-10-16T13:44:35.754 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 97 2023-10-16T13:44:35.766 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-16T13:44:35.767 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-16T13:44:35.767 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-16T13:44:35.767 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T13:44:35.768 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-16T13:44:35.768 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-16T13:44:35.768 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-16T13:44:35.769 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T13:44:35.769 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-16T13:44:35.769 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-16T13:44:35.770 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-16T13:44:35.770 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T13:44:35.770 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-16T13:44:35.770 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-16T13:44:35.771 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-16T13:44:35.771 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T13:44:35.776 INFO:tasks.ceph.mds.a:Stopped 2023-10-16T13:44:35.776 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-16T13:44:35.777 INFO:tasks.ceph.mds.c:Stopped 2023-10-16T13:44:35.778 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-16T13:44:35.784 INFO:tasks.ceph.mds.b:Stopped 2023-10-16T13:44:35.785 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-16T13:44:35.786 INFO:tasks.ceph.mds.d:Stopped 2023-10-16T13:44:35.786 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-16T13:44:35.818 INFO:tasks.ceph.mds.a:Started 2023-10-16T13:44:35.818 DEBUG:teuthology.parallel:result is None 2023-10-16T13:44:35.819 INFO:tasks.ceph.mds.c:Started 2023-10-16T13:44:35.819 DEBUG:teuthology.parallel:result is None 2023-10-16T13:44:35.827 INFO:tasks.ceph.mds.b:Started 2023-10-16T13:44:35.827 DEBUG:teuthology.parallel:result is None 2023-10-16T13:44:35.828 INFO:tasks.ceph.mds.d:Started 2023-10-16T13:44:35.828 DEBUG:teuthology.parallel:result is None 2023-10-16T13:44:35.828 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-16T13:44:35.901 INFO:tasks.ceph.mds.a.smithi016.stdout:starting mds.a at 2023-10-16T13:44:35.901 INFO:tasks.ceph.mds.c.smithi016.stdout:starting mds.c at 2023-10-16T13:44:35.915 INFO:tasks.ceph.mds.d.smithi049.stdout:starting mds.d at 2023-10-16T13:44:35.916 INFO:tasks.ceph.mds.b.smithi049.stdout:starting mds.b at 2023-10-16T13:44:35.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.960+0000 7f356e7c4700 1 Processor -- start 2023-10-16T13:44:35.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.960+0000 7f356e7c4700 1 -- start start 2023-10-16T13:44:35.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.961+0000 7f356e7c4700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f356815d8d0 0x7f35680d1c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:35.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.961+0000 7f356e7c4700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f356806ba00 con 0x7f35680d41c0 2023-10-16T13:44:35.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.961+0000 7f356e7c4700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35680d2150 con 0x7f35680d0750 2023-10-16T13:44:35.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.961+0000 7f356e7c4700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35680d26d0 con 0x7f356815d8d0 2023-10-16T13:44:35.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.961+0000 7f35677fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f35680d41c0 0x7f35680d0040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:40806/0 (socket says 172.21.15.16:40806) 2023-10-16T13:44:35.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.961+0000 7f35677fe700 1 -- 172.21.15.16:0/2478531558 learned_addr learned my addr 172.21.15.16:0/2478531558 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:44:35.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.961+0000 7f3567fff700 1 --2- 172.21.15.16:0/2478531558 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f356815d8d0 0x7f35680d1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:35.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.961+0000 7f3566ffd700 1 -- 172.21.15.16:0/2478531558 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2970654097 0 0) 0x7f356806ba00 con 0x7f35680d41c0 2023-10-16T13:44:35.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.961+0000 7f3566ffd700 1 -- 172.21.15.16:0/2478531558 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35540035b0 con 0x7f35680d41c0 2023-10-16T13:44:35.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.962+0000 7f3566ffd700 1 -- 172.21.15.16:0/2478531558 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3245670180 0 0) 0x7f35680d2150 con 0x7f35680d0750 2023-10-16T13:44:35.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.962+0000 7f3566ffd700 1 -- 172.21.15.16:0/2478531558 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f356806ba00 con 0x7f35680d0750 2023-10-16T13:44:35.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.962+0000 7f3567fff700 1 -- 172.21.15.16:0/2478531558 >> v1:172.21.15.49:6789/0 conn(0x7f35680d0750 legacy=0x7f356806c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:35.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.962+0000 7f3567fff700 1 -- 172.21.15.16:0/2478531558 >> v1:172.21.15.16:6789/0 conn(0x7f35680d41c0 legacy=0x7f35680d0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:35.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.962+0000 7f3567fff700 1 -- 172.21.15.16:0/2478531558 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35680d23f0 con 0x7f356815d8d0 2023-10-16T13:44:35.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.962+0000 7f3567fff700 1 --2- 172.21.15.16:0/2478531558 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f356815d8d0 0x7f35680d1c10 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f3558009870 tx=0x7f3558009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e59eeea0b44217 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:35.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.962+0000 7f3566ffd700 1 -- 172.21.15.16:0/2478531558 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3558011830 con 0x7f356815d8d0 2023-10-16T13:44:35.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.962+0000 7f3566ffd700 1 -- 172.21.15.16:0/2478531558 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3558011990 con 0x7f356815d8d0 2023-10-16T13:44:35.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.962+0000 7f3566ffd700 1 -- 172.21.15.16:0/2478531558 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3558011d00 con 0x7f356815d8d0 2023-10-16T13:44:35.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.962+0000 7f356e7c4700 1 -- 172.21.15.16:0/2478531558 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f356815d8d0 msgr2=0x7f35680d1c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:35.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.962+0000 7f356e7c4700 1 --2- 172.21.15.16:0/2478531558 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f356815d8d0 0x7f35680d1c10 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f3558009870 tx=0x7f3558009b80 comp rx=0 tx=0).stop 2023-10-16T13:44:35.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.963+0000 7f356e7c4700 1 -- 172.21.15.16:0/2478531558 shutdown_connections 2023-10-16T13:44:35.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.963+0000 7f356e7c4700 1 --2- 172.21.15.16:0/2478531558 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f356815d8d0 0x7f35680d1c10 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:35.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.963+0000 7f356e7c4700 1 -- 172.21.15.16:0/2478531558 >> 172.21.15.16:0/2478531558 conn(0x7f35680d3630 msgr2=0x7f35680d3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:35.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.963+0000 7f356e7c4700 1 -- 172.21.15.16:0/2478531558 shutdown_connections 2023-10-16T13:44:35.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.963+0000 7f356e7c4700 1 -- 172.21.15.16:0/2478531558 wait complete. 2023-10-16T13:44:35.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.964+0000 7f356e7c4700 1 Processor -- start 2023-10-16T13:44:35.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.964+0000 7f356e7c4700 1 -- start start 2023-10-16T13:44:35.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.965+0000 7f356e7c4700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f35680d0750 0x7f35681e7870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:35.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.965+0000 7f356e7c4700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f35680d41c0 0x7f356815a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:35.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.965+0000 7f356e7c4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f356815d8d0 0x7f35681f2e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:35.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.965+0000 7f356e7c4700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35680c7d60 con 0x7f356815d8d0 2023-10-16T13:44:35.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.965+0000 7f356e7c4700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35680c8040 con 0x7f35680d0750 2023-10-16T13:44:35.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.965+0000 7f356e7c4700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35680c8320 con 0x7f35680d41c0 2023-10-16T13:44:35.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.965+0000 7f356cd61700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f356815d8d0 0x7f35681f2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:35.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.965+0000 7f3567fff700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f35680d0750 0x7f35681e7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:35.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.965+0000 7f356cd61700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f356815d8d0 0x7f35681f2e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51540/0 (socket says 172.21.15.16:51540) 2023-10-16T13:44:35.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.965+0000 7f356cd61700 1 -- 172.21.15.16:0/303845620 learned_addr learned my addr 172.21.15.16:0/303845620 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:35.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.965+0000 7f35677fe700 1 --2- 172.21.15.16:0/303845620 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f35680d41c0 0x7f356815a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:35.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.965+0000 7f356cd61700 1 -- 172.21.15.16:0/303845620 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f35680d41c0 msgr2=0x7f356815a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:35.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.965+0000 7f356cd61700 1 --2- 172.21.15.16:0/303845620 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f35680d41c0 0x7f356815a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:35.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.965+0000 7f356cd61700 1 -- 172.21.15.16:0/303845620 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f35680d0750 msgr2=0x7f35681e7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:35.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.965+0000 7f356cd61700 1 --2- 172.21.15.16:0/303845620 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f35680d0750 0x7f35681e7870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:35.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.965+0000 7f356cd61700 1 -- 172.21.15.16:0/303845620 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35580094a0 con 0x7f356815d8d0 2023-10-16T13:44:35.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.965+0000 7f3567fff700 1 --2- 172.21.15.16:0/303845620 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f35680d0750 0x7f35681e7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:44:35.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.965+0000 7f356cd61700 1 --2- 172.21.15.16:0/303845620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f356815d8d0 0x7f35681f2e50 secure :-1 s=READY pgs=2261 cs=0 l=1 rev1=1 crypto rx=0x7f356000cc20 tx=0x7f356000cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:35.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.965+0000 7f3564ff9700 1 -- 172.21.15.16:0/303845620 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3560003c00 con 0x7f356815d8d0 2023-10-16T13:44:35.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.966+0000 7f3564ff9700 1 -- 172.21.15.16:0/303845620 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3560003d60 con 0x7f356815d8d0 2023-10-16T13:44:35.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.966+0000 7f356e7c4700 1 -- 172.21.15.16:0/303845620 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35680c8600 con 0x7f356815d8d0 2023-10-16T13:44:35.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.966+0000 7f3564ff9700 1 -- 172.21.15.16:0/303845620 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f356000aae0 con 0x7f356815d8d0 2023-10-16T13:44:35.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.966+0000 7f356e7c4700 1 -- 172.21.15.16:0/303845620 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35681e7db0 con 0x7f356815d8d0 2023-10-16T13:44:35.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.967+0000 7f356e7c4700 1 -- 172.21.15.16:0/303845620 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35680c8b20 con 0x7f356815d8d0 2023-10-16T13:44:35.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.967+0000 7f3564ff9700 1 -- 172.21.15.16:0/303845620 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 126) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f356000ac40 con 0x7f356815d8d0 2023-10-16T13:44:35.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.968+0000 7f3564ff9700 1 --2- 172.21.15.16:0/303845620 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f354806b020 0x7f354806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:35.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.968+0000 7f3564ff9700 1 -- 172.21.15.16:0/303845620 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 6423+0+0 (secure 0 0 0) 0x7f356008a300 con 0x7f356815d8d0 2023-10-16T13:44:35.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.968+0000 7f3567fff700 1 --2- 172.21.15.16:0/303845620 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f354806b020 0x7f354806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:35.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.968+0000 7f3567fff700 1 --2- 172.21.15.16:0/303845620 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f354806b020 0x7f354806d4e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f35580090b0 tx=0x7f355801c040 comp rx=0 tx=0).ready entity=mgr.10665 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:35.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:35.972+0000 7f3564ff9700 1 -- 172.21.15.16:0/303845620 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f35600550b0 con 0x7f356815d8d0 2023-10-16T13:44:36.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:36.191+0000 7f356e7c4700 1 -- 172.21.15.16:0/303845620 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f35680c8b20 con 0x7f356815d8d0 2023-10-16T13:44:36.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:36.764+0000 7f3564ff9700 1 -- 172.21.15.16:0/303845620 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v200) v1 ==== 97+0+0 (secure 0 0 0) 0x7f35680c8b20 con 0x7f356815d8d0 2023-10-16T13:44:36.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:36.767+0000 7f356e7c4700 1 -- 172.21.15.16:0/303845620 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f354806b020 msgr2=0x7f354806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:36.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:36.767+0000 7f356e7c4700 1 --2- 172.21.15.16:0/303845620 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f354806b020 0x7f354806d4e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f35580090b0 tx=0x7f355801c040 comp rx=0 tx=0).stop 2023-10-16T13:44:36.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:36.767+0000 7f356e7c4700 1 -- 172.21.15.16:0/303845620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f356815d8d0 msgr2=0x7f35681f2e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:36.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:36.767+0000 7f356e7c4700 1 --2- 172.21.15.16:0/303845620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f356815d8d0 0x7f35681f2e50 secure :-1 s=READY pgs=2261 cs=0 l=1 rev1=1 crypto rx=0x7f356000cc20 tx=0x7f356000cf30 comp rx=0 tx=0).stop 2023-10-16T13:44:36.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:36.768+0000 7f356e7c4700 1 -- 172.21.15.16:0/303845620 shutdown_connections 2023-10-16T13:44:36.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:36.768+0000 7f356e7c4700 1 --2- 172.21.15.16:0/303845620 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f354806b020 0x7f354806d4e0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:36.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:36.768+0000 7f356e7c4700 1 --2- 172.21.15.16:0/303845620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f356815d8d0 0x7f35681f2e50 unknown :-1 s=CLOSED pgs=2261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:36.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:36.768+0000 7f356e7c4700 1 --2- 172.21.15.16:0/303845620 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f35680d0750 0x7f35681e7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:36.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:36.768+0000 7f356e7c4700 1 --2- 172.21.15.16:0/303845620 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f35680d41c0 0x7f356815a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:36.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:36.768+0000 7f356e7c4700 1 -- 172.21.15.16:0/303845620 >> 172.21.15.16:0/303845620 conn(0x7f35680d3630 msgr2=0x7f356806c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:36.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:36.768+0000 7f356e7c4700 1 -- 172.21.15.16:0/303845620 shutdown_connections 2023-10-16T13:44:36.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:36.768+0000 7f356e7c4700 1 -- 172.21.15.16:0/303845620 wait complete. 2023-10-16T13:44:36.774 INFO:teuthology.orchestra.run.smithi016.stderr: removed all blocklist entries 2023-10-16T13:44:36.789 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-16T13:44:36.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:36.918+0000 7f22795ed700 1 Processor -- start 2023-10-16T13:44:36.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:36.918+0000 7f22795ed700 1 -- start start 2023-10-16T13:44:36.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:36.919+0000 7f22795ed700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f22740d41c0 0x7f22740d0040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:36.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:36.919+0000 7f22795ed700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f22740d0580 0x7f227406c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:36.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:36.919+0000 7f22795ed700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f227406ba00 con 0x7f227415d8d0 2023-10-16T13:44:36.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:36.919+0000 7f22795ed700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22740d0a10 con 0x7f22740d41c0 2023-10-16T13:44:36.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:36.919+0000 7f22795ed700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22740d26d0 con 0x7f22740d0580 2023-10-16T13:44:36.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:36.919+0000 7f226a7fc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f22740d41c0 0x7f22740d0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:36.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:36.919+0000 7f226a7fc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f22740d41c0 0x7f22740d0040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51548/0 (socket says 172.21.15.16:51548) 2023-10-16T13:44:36.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:36.919+0000 7f226a7fc700 1 -- 172.21.15.16:0/2931811823 learned_addr learned my addr 172.21.15.16:0/2931811823 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:36.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:36.919+0000 7f22737fe700 1 --2- 172.21.15.16:0/2931811823 >> v2:172.21.15.49:3300/0 conn(0x7f22740d0580 0x7f227406c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:36.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:36.919+0000 7f226a7fc700 1 -- 172.21.15.16:0/2931811823 >> v2:172.21.15.49:3300/0 conn(0x7f22740d0580 msgr2=0x7f227406c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:36.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:36.919+0000 7f226a7fc700 1 --2- 172.21.15.16:0/2931811823 >> v2:172.21.15.49:3300/0 conn(0x7f22740d0580 0x7f227406c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:36.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:36.919+0000 7f226a7fc700 1 -- 172.21.15.16:0/2931811823 >> v1:172.21.15.49:6789/0 conn(0x7f227415d8d0 legacy=0x7f22740d1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:36.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:36.919+0000 7f226a7fc700 1 -- 172.21.15.16:0/2931811823 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22740d23f0 con 0x7f22740d41c0 2023-10-16T13:44:36.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:36.919+0000 7f22737fe700 1 --2- 172.21.15.16:0/2931811823 >> v2:172.21.15.49:3300/0 conn(0x7f22740d0580 0x7f227406c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:44:36.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:36.919+0000 7f226a7fc700 1 --2- 172.21.15.16:0/2931811823 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22740d41c0 0x7f22740d0040 secure :-1 s=READY pgs=2262 cs=0 l=1 rev1=1 crypto rx=0x7f227406b2d0 tx=0x7f225c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83dd2c354b90391f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:36.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:36.920+0000 7f22727fc700 1 -- 172.21.15.16:0/2931811823 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f225c004230 con 0x7f22740d41c0 2023-10-16T13:44:36.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:36.920+0000 7f22727fc700 1 -- 172.21.15.16:0/2931811823 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f225c004390 con 0x7f22740d41c0 2023-10-16T13:44:36.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:36.920+0000 7f22727fc700 1 -- 172.21.15.16:0/2931811823 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f225c0036a0 con 0x7f22740d41c0 2023-10-16T13:44:36.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:36.920+0000 7f22795ed700 1 -- 172.21.15.16:0/2931811823 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22740d41c0 msgr2=0x7f22740d0040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:36.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:36.920+0000 7f22795ed700 1 --2- 172.21.15.16:0/2931811823 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22740d41c0 0x7f22740d0040 secure :-1 s=READY pgs=2262 cs=0 l=1 rev1=1 crypto rx=0x7f227406b2d0 tx=0x7f225c009bd0 comp rx=0 tx=0).stop 2023-10-16T13:44:36.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:36.920+0000 7f22795ed700 1 -- 172.21.15.16:0/2931811823 shutdown_connections 2023-10-16T13:44:36.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:36.920+0000 7f22795ed700 1 --2- 172.21.15.16:0/2931811823 >> v2:172.21.15.49:3300/0 conn(0x7f22740d0580 0x7f227406c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:36.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:36.920+0000 7f22795ed700 1 --2- 172.21.15.16:0/2931811823 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22740d41c0 0x7f22740d0040 unknown :-1 s=CLOSED pgs=2262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:36.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:36.920+0000 7f22795ed700 1 -- 172.21.15.16:0/2931811823 >> 172.21.15.16:0/2931811823 conn(0x7f22740d3630 msgr2=0x7f22740d3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:36.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:36.920+0000 7f22795ed700 1 -- 172.21.15.16:0/2931811823 shutdown_connections 2023-10-16T13:44:36.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:36.920+0000 7f22795ed700 1 -- 172.21.15.16:0/2931811823 wait complete. 2023-10-16T13:44:36.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:36.921+0000 7f22795ed700 1 Processor -- start 2023-10-16T13:44:36.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:36.922+0000 7f22795ed700 1 -- start start 2023-10-16T13:44:36.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:36.922+0000 7f22795ed700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f22740d0580 0x7f227415aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:36.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:36.922+0000 7f22795ed700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22740d41c0 0x7f22741f2a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:36.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:36.922+0000 7f22795ed700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f227415d8d0 0x7f22740c7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:36.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:36.922+0000 7f22795ed700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22740c7090 con 0x7f22740d41c0 2023-10-16T13:44:36.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:36.922+0000 7f22795ed700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22740c81e0 con 0x7f22740d0580 2023-10-16T13:44:36.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:36.922+0000 7f22795ed700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22740c84c0 con 0x7f227415d8d0 2023-10-16T13:44:36.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:36.922+0000 7f226a7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22740d41c0 0x7f22741f2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:36.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:36.922+0000 7f226a7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22740d41c0 0x7f22741f2a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51554/0 (socket says 172.21.15.16:51554) 2023-10-16T13:44:36.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:36.922+0000 7f226a7fc700 1 -- 172.21.15.16:0/3863415207 learned_addr learned my addr 172.21.15.16:0/3863415207 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:36.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:36.923+0000 7f2272ffd700 1 --2- 172.21.15.16:0/3863415207 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f22740d0580 0x7f227415aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:36.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:36.923+0000 7f22737fe700 1 --2- 172.21.15.16:0/3863415207 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f227415d8d0 0x7f22740c7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:36.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:36.923+0000 7f226a7fc700 1 -- 172.21.15.16:0/3863415207 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f227415d8d0 msgr2=0x7f22740c7ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:36.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:36.923+0000 7f226a7fc700 1 --2- 172.21.15.16:0/3863415207 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f227415d8d0 0x7f22740c7ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:36.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:36.923+0000 7f226a7fc700 1 -- 172.21.15.16:0/3863415207 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f22740d0580 msgr2=0x7f227415aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:36.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:36.923+0000 7f226a7fc700 1 --2- 172.21.15.16:0/3863415207 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f22740d0580 0x7f227415aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:36.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:36.923+0000 7f226a7fc700 1 -- 172.21.15.16:0/3863415207 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f225c009550 con 0x7f22740d41c0 2023-10-16T13:44:36.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:36.923+0000 7f226a7fc700 1 --2- 172.21.15.16:0/3863415207 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22740d41c0 0x7f22741f2a00 secure :-1 s=READY pgs=2263 cs=0 l=1 rev1=1 crypto rx=0x7f227408db90 tx=0x7f225c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:36.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:36.923+0000 7f226bfff700 1 -- 172.21.15.16:0/3863415207 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f225c003d80 con 0x7f22740d41c0 2023-10-16T13:44:36.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:36.923+0000 7f226bfff700 1 -- 172.21.15.16:0/3863415207 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f225c01f070 con 0x7f22740d41c0 2023-10-16T13:44:36.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:36.923+0000 7f22795ed700 1 -- 172.21.15.16:0/3863415207 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22740cbdf0 con 0x7f22740d41c0 2023-10-16T13:44:36.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:36.923+0000 7f226bfff700 1 -- 172.21.15.16:0/3863415207 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f225c011870 con 0x7f22740d41c0 2023-10-16T13:44:36.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:36.923+0000 7f22795ed700 1 -- 172.21.15.16:0/3863415207 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22740cc3b0 con 0x7f22740d41c0 2023-10-16T13:44:36.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:36.924+0000 7f22795ed700 1 -- 172.21.15.16:0/3863415207 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f227404ebb0 con 0x7f22740d41c0 2023-10-16T13:44:36.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:36.925+0000 7f226bfff700 1 -- 172.21.15.16:0/3863415207 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 126) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f225c021030 con 0x7f22740d41c0 2023-10-16T13:44:36.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:36.925+0000 7f226bfff700 1 --2- 172.21.15.16:0/3863415207 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f225406b020 0x7f225406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:36.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:36.925+0000 7f2272ffd700 1 --2- 172.21.15.16:0/3863415207 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f225406b020 0x7f225406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:36.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:36.925+0000 7f226bfff700 1 -- 172.21.15.16:0/3863415207 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 5305+0+0 (secure 0 0 0) 0x7f225c08adf0 con 0x7f22740d41c0 2023-10-16T13:44:36.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:36.926+0000 7f2272ffd700 1 --2- 172.21.15.16:0/3863415207 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f225406b020 0x7f225406d4e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f22740c7470 tx=0x7f2264008040 comp rx=0 tx=0).ready entity=mgr.10665 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:36.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:36.930+0000 7f226bfff700 1 -- 172.21.15.16:0/3863415207 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f225c056050 con 0x7f22740d41c0 2023-10-16T13:44:37.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.216+0000 7f22795ed700 1 -- 172.21.15.16:0/3863415207 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f22740b1d70 con 0x7f22740d41c0 2023-10-16T13:44:37.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.217+0000 7f226bfff700 1 -- 172.21.15.16:0/3863415207 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v53) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f225c059670 con 0x7f22740d41c0 2023-10-16T13:44:37.221 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:44:37.221 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-10-16T13:44:37.221 INFO:teuthology.orchestra.run.smithi016.stdout: "auth_dump": [ 2023-10-16T13:44:37.221 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:44:37.222 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.a", 2023-10-16T13:44:37.222 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOKC1l/rBtOhAA/ob5WUvpz8Glp4E+xPZOYA==", 2023-10-16T13:44:37.222 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:44:37.222 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T13:44:37.222 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-10-16T13:44:37.223 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-10-16T13:44:37.223 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:44:37.223 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:44:37.223 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:44:37.223 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:44:37.223 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.b", 2023-10-16T13:44:37.224 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1lKAVsCxAAvhXuLY6i1+Iz1yq5BVPMNQ==", 2023-10-16T13:44:37.224 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:44:37.224 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T13:44:37.224 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-10-16T13:44:37.224 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-10-16T13:44:37.224 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:44:37.225 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:44:37.225 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:44:37.225 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:44:37.225 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.c", 2023-10-16T13:44:37.225 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1lvBowBhAA3waJNYhdgs8PGCYqYNYUog==", 2023-10-16T13:44:37.225 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:44:37.226 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T13:44:37.226 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-10-16T13:44:37.226 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-10-16T13:44:37.226 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:44:37.226 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:44:37.226 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:44:37.227 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:44:37.227 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.d", 2023-10-16T13:44:37.227 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1lS/fnEhAAfNYgQ5BW86JMjYm/bBfnaQ==", 2023-10-16T13:44:37.227 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:44:37.227 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T13:44:37.228 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-10-16T13:44:37.228 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-10-16T13:44:37.228 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:44:37.228 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:44:37.228 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:44:37.228 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:44:37.229 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.0", 2023-10-16T13:44:37.229 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAQKC1lbxllOhAAZeLdp2GMl6HqQPaQjKlNqw==", 2023-10-16T13:44:37.229 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:44:37.229 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T13:44:37.229 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T13:44:37.229 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:44:37.230 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:44:37.230 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:44:37.230 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:44:37.230 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.1", 2023-10-16T13:44:37.230 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQATKC1lnU6KOhAAHxvlbmzBptf1keGgL+Rbkg==", 2023-10-16T13:44:37.231 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:44:37.231 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T13:44:37.231 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T13:44:37.231 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:44:37.231 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:44:37.231 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:44:37.232 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:44:37.232 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.2", 2023-10-16T13:44:37.232 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAXKC1l9jFyARAAr/NAwtWhBMut9sAAoDEGWg==", 2023-10-16T13:44:37.232 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:44:37.232 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T13:44:37.232 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T13:44:37.233 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:44:37.233 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:44:37.233 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:44:37.233 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:44:37.233 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.3", 2023-10-16T13:44:37.234 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAaKC1lbw/lARAAGJ/9DNb5/7kLFSq7vrLtXg==", 2023-10-16T13:44:37.234 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:44:37.234 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T13:44:37.234 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T13:44:37.234 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:44:37.235 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:44:37.235 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:44:37.235 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:44:37.235 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.4", 2023-10-16T13:44:37.235 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAeKC1lZb7jERAAYpvb1xcSli9CTq7L8ChLOA==", 2023-10-16T13:44:37.235 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:44:37.236 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T13:44:37.236 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T13:44:37.236 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:44:37.236 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:44:37.236 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:44:37.236 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:44:37.237 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.5", 2023-10-16T13:44:37.237 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAhKC1l/4HuEBAAqfnRqjknij9KES/fqiyIBQ==", 2023-10-16T13:44:37.237 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:44:37.237 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T13:44:37.237 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T13:44:37.237 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:44:37.238 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:44:37.238 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:44:37.238 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:44:37.238 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.6", 2023-10-16T13:44:37.238 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAkKC1lzoUkEBAA+Q82OiBMNag8+R316WQtUw==", 2023-10-16T13:44:37.239 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:44:37.239 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T13:44:37.239 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T13:44:37.239 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:44:37.239 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:44:37.240 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:44:37.240 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:44:37.240 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.7", 2023-10-16T13:44:37.240 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAnKC1ljUu4DxAAc+G4anJANC2GhLJ8wb6gYg==", 2023-10-16T13:44:37.240 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:44:37.240 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T13:44:37.241 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T13:44:37.241 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:44:37.241 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:44:37.241 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:44:37.241 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:44:37.242 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.0", 2023-10-16T13:44:37.242 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1l2l6PFxAA9QyXtBAHdGAXpwH0JfknuQ==", 2023-10-16T13:44:37.242 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:44:37.242 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T13:44:37.242 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow r", 2023-10-16T13:44:37.242 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-16T13:44:37.243 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:44:37.243 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:44:37.243 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:44:37.243 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.1", 2023-10-16T13:44:37.243 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1lAcHFIxAAcNt2bmv2R6j5L3DYkdw7eQ==", 2023-10-16T13:44:37.244 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:44:37.244 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T13:44:37.244 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-10-16T13:44:37.244 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-10-16T13:44:37.244 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-10-16T13:44:37.244 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:44:37.245 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:44:37.245 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:44:37.245 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.2", 2023-10-16T13:44:37.245 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1lA6wnGxAAdtaqYhtIJu8qENMrD8xvVA==", 2023-10-16T13:44:37.245 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:44:37.245 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T13:44:37.246 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-10-16T13:44:37.246 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-10-16T13:44:37.246 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-10-16T13:44:37.246 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:44:37.246 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:44:37.247 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:44:37.247 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.3", 2023-10-16T13:44:37.247 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1l/ddAJxAA3fNKKn9XkXas1BlDN+ggjw==", 2023-10-16T13:44:37.247 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:44:37.247 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T13:44:37.247 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-10-16T13:44:37.248 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-10-16T13:44:37.248 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-10-16T13:44:37.248 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:44:37.248 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:44:37.248 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:44:37.248 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.admin", 2023-10-16T13:44:37.249 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOKC1ld5jUHBAAc6esnCce3dSEZn3wVjhIlQ==", 2023-10-16T13:44:37.249 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:44:37.249 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2023-10-16T13:44:37.249 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-10-16T13:44:37.249 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-10-16T13:44:37.250 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:44:37.250 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:44:37.250 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:44:37.250 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:44:37.250 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mgr.x", 2023-10-16T13:44:37.251 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOKC1l6b3vNhAAmj5xbfid2sZM2KfHmAf1Wg==", 2023-10-16T13:44:37.251 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:44:37.251 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2023-10-16T13:44:37.251 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile mgr", 2023-10-16T13:44:37.251 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:44:37.251 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:44:37.252 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:44:37.252 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:44:37.252 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mgr.y", 2023-10-16T13:44:37.252 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOKC1lLz2nMhAA8NqFYZ5nDKkzPBMrkwsVuQ==", 2023-10-16T13:44:37.252 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:44:37.253 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2023-10-16T13:44:37.253 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile mgr", 2023-10-16T13:44:37.253 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:44:37.253 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:44:37.253 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:44:37.254 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-10-16T13:44:37.254 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-10-16T13:44:37.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.219+0000 7f22795ed700 1 -- 172.21.15.16:0/3863415207 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f225406b020 msgr2=0x7f225406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:37.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.219+0000 7f22795ed700 1 --2- 172.21.15.16:0/3863415207 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f225406b020 0x7f225406d4e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f22740c7470 tx=0x7f2264008040 comp rx=0 tx=0).stop 2023-10-16T13:44:37.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.219+0000 7f22795ed700 1 -- 172.21.15.16:0/3863415207 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22740d41c0 msgr2=0x7f22741f2a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:37.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.219+0000 7f22795ed700 1 --2- 172.21.15.16:0/3863415207 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22740d41c0 0x7f22741f2a00 secure :-1 s=READY pgs=2263 cs=0 l=1 rev1=1 crypto rx=0x7f227408db90 tx=0x7f225c004750 comp rx=0 tx=0).stop 2023-10-16T13:44:37.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.219+0000 7f22795ed700 1 -- 172.21.15.16:0/3863415207 shutdown_connections 2023-10-16T13:44:37.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.219+0000 7f22795ed700 1 --2- 172.21.15.16:0/3863415207 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f225406b020 0x7f225406d4e0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:37.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.219+0000 7f22795ed700 1 --2- 172.21.15.16:0/3863415207 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f227415d8d0 0x7f22740c7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:37.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.219+0000 7f22795ed700 1 --2- 172.21.15.16:0/3863415207 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f22740d0580 0x7f227415aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:37.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.219+0000 7f22795ed700 1 --2- 172.21.15.16:0/3863415207 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22740d41c0 0x7f22741f2a00 unknown :-1 s=CLOSED pgs=2263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:37.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.219+0000 7f22795ed700 1 -- 172.21.15.16:0/3863415207 >> 172.21.15.16:0/3863415207 conn(0x7f22740d3630 msgr2=0x7f227415cae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:37.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.219+0000 7f22795ed700 1 -- 172.21.15.16:0/3863415207 shutdown_connections 2023-10-16T13:44:37.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.219+0000 7f22795ed700 1 -- 172.21.15.16:0/3863415207 wait complete. 2023-10-16T13:44:37.258 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:44:37.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.381+0000 7fdcd22ec700 1 Processor -- start 2023-10-16T13:44:37.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.381+0000 7fdcd22ec700 1 -- start start 2023-10-16T13:44:37.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.382+0000 7fdcd22ec700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdccc06b7f0 0x7fdccc06bc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:37.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.382+0000 7fdcd22ec700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fdccc06d940 0x7fdccc06ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:37.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.382+0000 7fdcd22ec700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdccc0c3490 con 0x7fdccc15b9f0 2023-10-16T13:44:37.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.382+0000 7fdcd22ec700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdccc06c1b0 con 0x7fdccc06d940 2023-10-16T13:44:37.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.382+0000 7fdcd22ec700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdccc06e300 con 0x7fdccc06b7f0 2023-10-16T13:44:37.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.382+0000 7fdccbfff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fdccc15b9f0 0x7fdccc0b29a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:40834/0 (socket says 172.21.15.16:40834) 2023-10-16T13:44:37.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.382+0000 7fdccbfff700 1 -- 172.21.15.16:0/4140790257 learned_addr learned my addr 172.21.15.16:0/4140790257 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:44:37.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.382+0000 7fdcd0889700 1 --2- 172.21.15.16:0/4140790257 >> v2:172.21.15.49:3300/0 conn(0x7fdccc06d940 0x7fdccc06ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:37.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.382+0000 7fdccb7fe700 1 --2- 172.21.15.16:0/4140790257 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdccc06b7f0 0x7fdccc06bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:37.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.382+0000 7fdccaffd700 1 -- 172.21.15.16:0/4140790257 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2688763070 0 0) 0x7fdccc0c3490 con 0x7fdccc15b9f0 2023-10-16T13:44:37.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.382+0000 7fdccaffd700 1 -- 172.21.15.16:0/4140790257 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcb40035b0 con 0x7fdccc15b9f0 2023-10-16T13:44:37.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.382+0000 7fdccaffd700 1 -- 172.21.15.16:0/4140790257 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2085512163 0 0) 0x7fdcb40035b0 con 0x7fdccc15b9f0 2023-10-16T13:44:37.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.382+0000 7fdccaffd700 1 -- 172.21.15.16:0/4140790257 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdccc06b7f0 msgr2=0x7fdccc06bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:37.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.382+0000 7fdccaffd700 1 --2- 172.21.15.16:0/4140790257 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdccc06b7f0 0x7fdccc06bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:37.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.382+0000 7fdccaffd700 1 -- 172.21.15.16:0/4140790257 >> v2:172.21.15.49:3300/0 conn(0x7fdccc06d940 msgr2=0x7fdccc06ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:37.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.382+0000 7fdccaffd700 1 --2- 172.21.15.16:0/4140790257 >> v2:172.21.15.49:3300/0 conn(0x7fdccc06d940 0x7fdccc06ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:37.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.382+0000 7fdccaffd700 1 -- 172.21.15.16:0/4140790257 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdccc0cdad0 con 0x7fdccc15b9f0 2023-10-16T13:44:37.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.382+0000 7fdccaffd700 1 -- 172.21.15.16:0/4140790257 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7fdcbc001670 con 0x7fdccc15b9f0 2023-10-16T13:44:37.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.382+0000 7fdccaffd700 1 -- 172.21.15.16:0/4140790257 >> v1:172.21.15.16:6789/0 conn(0x7fdccc15b9f0 legacy=0x7fdccc0b29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:37.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.382+0000 7fdccaffd700 1 --2- 172.21.15.16:0/4140790257 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdccc06b7f0 0x7fdcb4003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:37.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.382+0000 7fdccaffd700 1 --2- 172.21.15.16:0/4140790257 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdcb4003f80 0x7fdcb4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:37.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.382+0000 7fdccaffd700 1 --2- 172.21.15.16:0/4140790257 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdcb4004d90 0x7fdcb4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:37.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.382+0000 7fdccaffd700 1 -- 172.21.15.16:0/4140790257 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdccc06c1b0 con 0x7fdcb4003f80 2023-10-16T13:44:37.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.382+0000 7fdccaffd700 1 -- 172.21.15.16:0/4140790257 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdccc06e300 con 0x7fdcb4004d90 2023-10-16T13:44:37.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.382+0000 7fdccaffd700 1 -- 172.21.15.16:0/4140790257 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb400ba10 con 0x7fdccc06b7f0 2023-10-16T13:44:37.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.382+0000 7fdcd0889700 1 --2- 172.21.15.16:0/4140790257 >> v2:172.21.15.49:3300/0 conn(0x7fdccc06d940 0x7fdccc06ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:44:37.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.383+0000 7fdccb7fe700 1 --2- 172.21.15.16:0/4140790257 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdcb4003f80 0x7fdcb4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:37.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.383+0000 7fdccbfff700 1 --2- 172.21.15.16:0/4140790257 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdccc06b7f0 0x7fdcb4003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:37.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.383+0000 7fdcd0889700 1 --2- 172.21.15.16:0/4140790257 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdcb4004d90 0x7fdcb4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:37.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.383+0000 7fdccb7fe700 1 -- 172.21.15.16:0/4140790257 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdccc06b7f0 msgr2=0x7fdcb4003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:37.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.383+0000 7fdccb7fe700 1 --2- 172.21.15.16:0/4140790257 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdccc06b7f0 0x7fdcb4003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:37.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.383+0000 7fdccb7fe700 1 -- 172.21.15.16:0/4140790257 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdcb4004d90 msgr2=0x7fdcb4005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:37.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.383+0000 7fdccb7fe700 1 --2- 172.21.15.16:0/4140790257 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdcb4004d90 0x7fdcb4005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:37.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.383+0000 7fdccb7fe700 1 -- 172.21.15.16:0/4140790257 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcb40035b0 con 0x7fdcb4003f80 2023-10-16T13:44:37.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.383+0000 7fdcd0889700 1 --2- 172.21.15.16:0/4140790257 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdcb4004d90 0x7fdcb4005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:44:37.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.383+0000 7fdccb7fe700 1 --2- 172.21.15.16:0/4140790257 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdcb4003f80 0x7fdcb4004850 secure :-1 s=READY pgs=2265 cs=0 l=1 rev1=1 crypto rx=0x7fdcc0009b30 tx=0x7fdcb4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c387fc781b83bbbe server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:37.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.383+0000 7fdccaffd700 1 -- 172.21.15.16:0/4140790257 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc000e660 con 0x7fdcb4003f80 2023-10-16T13:44:37.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.383+0000 7fdccaffd700 1 -- 172.21.15.16:0/4140790257 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcc000e7c0 con 0x7fdcb4003f80 2023-10-16T13:44:37.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.383+0000 7fdccaffd700 1 -- 172.21.15.16:0/4140790257 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc000eb30 con 0x7fdcb4003f80 2023-10-16T13:44:37.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.384+0000 7fdcd22ec700 1 -- 172.21.15.16:0/4140790257 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdcb4003f80 msgr2=0x7fdcb4004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:37.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.384+0000 7fdcd22ec700 1 --2- 172.21.15.16:0/4140790257 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdcb4003f80 0x7fdcb4004850 secure :-1 s=READY pgs=2265 cs=0 l=1 rev1=1 crypto rx=0x7fdcc0009b30 tx=0x7fdcb4005880 comp rx=0 tx=0).stop 2023-10-16T13:44:37.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.384+0000 7fdcd0889700 1 -- 172.21.15.16:0/4140790257 reap_dead start 2023-10-16T13:44:37.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.384+0000 7fdcd22ec700 1 -- 172.21.15.16:0/4140790257 shutdown_connections 2023-10-16T13:44:37.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.384+0000 7fdcd22ec700 1 -- 172.21.15.16:0/4140790257 >> 172.21.15.16:0/4140790257 conn(0x7fdccc0cc5b0 msgr2=0x7fdccc0cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:37.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.384+0000 7fdcd22ec700 1 -- 172.21.15.16:0/4140790257 shutdown_connections 2023-10-16T13:44:37.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.384+0000 7fdcd22ec700 1 -- 172.21.15.16:0/4140790257 wait complete. 2023-10-16T13:44:37.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.385+0000 7fdcd22ec700 1 Processor -- start 2023-10-16T13:44:37.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.385+0000 7fdcd22ec700 1 -- start start 2023-10-16T13:44:37.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.386+0000 7fdcd22ec700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdcb4003f80 0x7fdccc0ca180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:37.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.386+0000 7fdcd22ec700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdccc0cb5d0 0x7fdccc15d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:37.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.386+0000 7fdcd22ec700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdccc0c40e0 0x7fdccc0c72c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:37.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.386+0000 7fdcd22ec700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdccc0cb270 con 0x7fdccc0cb5d0 2023-10-16T13:44:37.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.386+0000 7fdcd22ec700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdccc0c9360 con 0x7fdcb4003f80 2023-10-16T13:44:37.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.386+0000 7fdcd22ec700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdccc0ca9d0 con 0x7fdccc0c40e0 2023-10-16T13:44:37.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.386+0000 7fdccb7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdccc0cb5d0 0x7fdccc15d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:37.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.386+0000 7fdccb7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdccc0cb5d0 0x7fdccc15d630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51574/0 (socket says 172.21.15.16:51574) 2023-10-16T13:44:37.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.386+0000 7fdccb7fe700 1 -- 172.21.15.16:0/33765918 learned_addr learned my addr 172.21.15.16:0/33765918 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:37.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.386+0000 7fdccbfff700 1 --2- 172.21.15.16:0/33765918 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdcb4003f80 0x7fdccc0ca180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:37.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.386+0000 7fdcd0889700 1 --2- 172.21.15.16:0/33765918 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdccc0c40e0 0x7fdccc0c72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:37.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.386+0000 7fdccb7fe700 1 -- 172.21.15.16:0/33765918 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdccc0c40e0 msgr2=0x7fdccc0c72c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:37.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.386+0000 7fdccb7fe700 1 --2- 172.21.15.16:0/33765918 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdccc0c40e0 0x7fdccc0c72c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:37.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.386+0000 7fdccb7fe700 1 -- 172.21.15.16:0/33765918 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdcb4003f80 msgr2=0x7fdccc0ca180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:37.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.386+0000 7fdccb7fe700 1 --2- 172.21.15.16:0/33765918 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdcb4003f80 0x7fdccc0ca180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:37.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.386+0000 7fdccb7fe700 1 -- 172.21.15.16:0/33765918 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdccc0cacb0 con 0x7fdccc0cb5d0 2023-10-16T13:44:37.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.386+0000 7fdccb7fe700 1 --2- 172.21.15.16:0/33765918 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdccc0cb5d0 0x7fdccc15d630 secure :-1 s=READY pgs=2266 cs=0 l=1 rev1=1 crypto rx=0x7fdcb40046f0 tx=0x7fdcc00047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:37.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.387+0000 7fdcc8ff9700 1 -- 172.21.15.16:0/33765918 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc0021070 con 0x7fdccc0cb5d0 2023-10-16T13:44:37.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.387+0000 7fdcc8ff9700 1 -- 172.21.15.16:0/33765918 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcc0013820 con 0x7fdccc0cb5d0 2023-10-16T13:44:37.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.387+0000 7fdcd22ec700 1 -- 172.21.15.16:0/33765918 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdccc0caf90 con 0x7fdccc0cb5d0 2023-10-16T13:44:37.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.387+0000 7fdcc8ff9700 1 -- 172.21.15.16:0/33765918 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc000e660 con 0x7fdccc0cb5d0 2023-10-16T13:44:37.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.387+0000 7fdcd22ec700 1 -- 172.21.15.16:0/33765918 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdccc0ca6c0 con 0x7fdccc0cb5d0 2023-10-16T13:44:37.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.388+0000 7fdcd22ec700 1 -- 172.21.15.16:0/33765918 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdccc0c6be0 con 0x7fdccc0cb5d0 2023-10-16T13:44:37.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.388+0000 7fdcc8ff9700 1 -- 172.21.15.16:0/33765918 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 126) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fdcc0025070 con 0x7fdccc0cb5d0 2023-10-16T13:44:37.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.388+0000 7fdcc8ff9700 1 --2- 172.21.15.16:0/33765918 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7fdcb806b020 0x7fdcb806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:37.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.389+0000 7fdcc8ff9700 1 -- 172.21.15.16:0/33765918 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 5305+0+0 (secure 0 0 0) 0x7fdcc008a8e0 con 0x7fdccc0cb5d0 2023-10-16T13:44:37.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.389+0000 7fdccbfff700 1 --2- 172.21.15.16:0/33765918 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7fdcb806b020 0x7fdcb806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:37.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.389+0000 7fdccbfff700 1 --2- 172.21.15.16:0/33765918 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7fdcb806b020 0x7fdcb806d4e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fdcbc000c00 tx=0x7fdcbc007980 comp rx=0 tx=0).ready entity=mgr.10665 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:37.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.393+0000 7fdcc8ff9700 1 -- 172.21.15.16:0/33765918 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdcc0055bc0 con 0x7fdccc0cb5d0 2023-10-16T13:44:37.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.598+0000 7fdcd22ec700 1 -- 172.21.15.16:0/33765918 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdccc04ebb0 con 0x7fdccc0cb5d0 2023-10-16T13:44:37.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.598+0000 7fdcc8ff9700 1 -- 172.21.15.16:0/33765918 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v202) v1 ==== 74+0+13169 (secure 0 0 0) 0x7fdcc00591e0 con 0x7fdccc0cb5d0 2023-10-16T13:44:37.602 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:44:37.602 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":202,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:44:36.774497+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:44:37.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.600+0000 7fdcd22ec700 1 -- 172.21.15.16:0/33765918 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7fdcb806b020 msgr2=0x7fdcb806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:37.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.600+0000 7fdcd22ec700 1 --2- 172.21.15.16:0/33765918 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7fdcb806b020 0x7fdcb806d4e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fdcbc000c00 tx=0x7fdcbc007980 comp rx=0 tx=0).stop 2023-10-16T13:44:37.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.601+0000 7fdcd22ec700 1 -- 172.21.15.16:0/33765918 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdccc0cb5d0 msgr2=0x7fdccc15d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:37.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.601+0000 7fdcd22ec700 1 --2- 172.21.15.16:0/33765918 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdccc0cb5d0 0x7fdccc15d630 secure :-1 s=READY pgs=2266 cs=0 l=1 rev1=1 crypto rx=0x7fdcb40046f0 tx=0x7fdcc00047c0 comp rx=0 tx=0).stop 2023-10-16T13:44:37.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.601+0000 7fdcd22ec700 1 -- 172.21.15.16:0/33765918 shutdown_connections 2023-10-16T13:44:37.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.601+0000 7fdcd22ec700 1 --2- 172.21.15.16:0/33765918 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7fdcb806b020 0x7fdcb806d4e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:37.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.601+0000 7fdcd22ec700 1 --2- 172.21.15.16:0/33765918 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdccc0c40e0 0x7fdccc0c72c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:37.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.601+0000 7fdcd22ec700 1 --2- 172.21.15.16:0/33765918 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdcb4003f80 0x7fdccc0ca180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:37.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.601+0000 7fdcd22ec700 1 --2- 172.21.15.16:0/33765918 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdccc0cb5d0 0x7fdccc15d630 unknown :-1 s=CLOSED pgs=2266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:37.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.601+0000 7fdcd22ec700 1 -- 172.21.15.16:0/33765918 >> 172.21.15.16:0/33765918 conn(0x7fdccc0cc5b0 msgr2=0x7fdccc159500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:37.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.601+0000 7fdcd22ec700 1 -- 172.21.15.16:0/33765918 shutdown_connections 2023-10-16T13:44:37.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.601+0000 7fdcd22ec700 1 -- 172.21.15.16:0/33765918 wait complete. 2023-10-16T13:44:37.617 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T13:44:37.617 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T13:44:37.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.741+0000 7f079a7fb700 1 Processor -- start 2023-10-16T13:44:37.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.742+0000 7f079a7fb700 1 -- start start 2023-10-16T13:44:37.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.742+0000 7f079a7fb700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f07940cde90 0x7f079415ea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:37.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.742+0000 7f079a7fb700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f07940cb6f0 0x7f079415efa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:37.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.742+0000 7f079a7fb700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07940cee50 con 0x7f07940ce270 2023-10-16T13:44:37.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.742+0000 7f079a7fb700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07940c5020 con 0x7f07940cde90 2023-10-16T13:44:37.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.743+0000 7f079a7fb700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07940d2340 con 0x7f07940cb6f0 2023-10-16T13:44:37.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.743+0000 7f0798d98700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f07940ce270 0x7f07940cf2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:40858/0 (socket says 172.21.15.16:40858) 2023-10-16T13:44:37.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.743+0000 7f0798d98700 1 -- 172.21.15.16:0/483358284 learned_addr learned my addr 172.21.15.16:0/483358284 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:44:37.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.743+0000 7f0793fff700 1 --2- 172.21.15.16:0/483358284 >> v2:172.21.15.49:3300/0 conn(0x7f07940cde90 0x7f079415ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:37.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.743+0000 7f07937fe700 1 --2- 172.21.15.16:0/483358284 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f07940cb6f0 0x7f079415efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:37.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.743+0000 7f0792ffd700 1 -- 172.21.15.16:0/483358284 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2074546817 0 0) 0x7f07940cee50 con 0x7f07940ce270 2023-10-16T13:44:37.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.743+0000 7f0792ffd700 1 -- 172.21.15.16:0/483358284 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f077c0035b0 con 0x7f07940ce270 2023-10-16T13:44:37.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.743+0000 7f0793fff700 1 -- 172.21.15.16:0/483358284 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f07940cb6f0 msgr2=0x7f079415efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:37.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.743+0000 7f0793fff700 1 --2- 172.21.15.16:0/483358284 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f07940cb6f0 0x7f079415efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:37.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.743+0000 7f0793fff700 1 -- 172.21.15.16:0/483358284 >> v1:172.21.15.16:6789/0 conn(0x7f07940ce270 legacy=0x7f07940cf2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:37.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.743+0000 7f0792ffd700 1 -- 172.21.15.16:0/483358284 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3554329332 0 0) 0x7f077c0035b0 con 0x7f07940ce270 2023-10-16T13:44:37.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.743+0000 7f0793fff700 1 -- 172.21.15.16:0/483358284 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07940ccdd0 con 0x7f07940cde90 2023-10-16T13:44:37.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.743+0000 7f07937fe700 1 --2- 172.21.15.16:0/483358284 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f07940cb6f0 0x7f079415efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:44:37.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.743+0000 7f0793fff700 1 --2- 172.21.15.16:0/483358284 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f07940cde90 0x7f079415ea60 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f0784009840 tx=0x7f0784009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=66683600edc3d094 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:37.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.744+0000 7f0792ffd700 1 -- 172.21.15.16:0/483358284 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f078c0019f0 con 0x7f07940cde90 2023-10-16T13:44:37.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.744+0000 7f0792ffd700 1 -- 172.21.15.16:0/483358284 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0784004030 con 0x7f07940cde90 2023-10-16T13:44:37.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.744+0000 7f0792ffd700 1 -- 172.21.15.16:0/483358284 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07840043c0 con 0x7f07940cde90 2023-10-16T13:44:37.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.744+0000 7f079a7fb700 1 -- 172.21.15.16:0/483358284 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f07940cde90 msgr2=0x7f079415ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:37.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.744+0000 7f079a7fb700 1 --2- 172.21.15.16:0/483358284 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f07940cde90 0x7f079415ea60 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f0784009840 tx=0x7f0784009b50 comp rx=0 tx=0).stop 2023-10-16T13:44:37.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.744+0000 7f079a7fb700 1 -- 172.21.15.16:0/483358284 shutdown_connections 2023-10-16T13:44:37.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.744+0000 7f079a7fb700 1 --2- 172.21.15.16:0/483358284 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f07940cde90 0x7f079415ea60 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:37.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.744+0000 7f079a7fb700 1 --2- 172.21.15.16:0/483358284 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f07940cb6f0 0x7f079415efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:37.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.744+0000 7f079a7fb700 1 -- 172.21.15.16:0/483358284 >> 172.21.15.16:0/483358284 conn(0x7f07940d14d0 msgr2=0x7f07940d18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:37.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.744+0000 7f079a7fb700 1 -- 172.21.15.16:0/483358284 shutdown_connections 2023-10-16T13:44:37.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.744+0000 7f079a7fb700 1 -- 172.21.15.16:0/483358284 wait complete. 2023-10-16T13:44:37.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.745+0000 7f079a7fb700 1 Processor -- start 2023-10-16T13:44:37.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.746+0000 7f079a7fb700 1 -- start start 2023-10-16T13:44:37.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.746+0000 7f079a7fb700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f07940cb6f0 0x7f0794153310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:37.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.746+0000 7f079a7fb700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07940cde90 0x7f07940b40f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:37.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.746+0000 7f079a7fb700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f07940ce270 0x7f07940b5b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:37.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.746+0000 7f079a7fb700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07940b56e0 con 0x7f07940cde90 2023-10-16T13:44:37.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.746+0000 7f079a7fb700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07941f0710 con 0x7f07940cb6f0 2023-10-16T13:44:37.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.746+0000 7f079a7fb700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07940b4a60 con 0x7f07940ce270 2023-10-16T13:44:37.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.747+0000 7f07937fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07940cde90 0x7f07940b40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:37.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.747+0000 7f07937fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07940cde90 0x7f07940b40f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51588/0 (socket says 172.21.15.16:51588) 2023-10-16T13:44:37.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.747+0000 7f07937fe700 1 -- 172.21.15.16:0/1349094582 learned_addr learned my addr 172.21.15.16:0/1349094582 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:37.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.747+0000 7f0793fff700 1 --2- 172.21.15.16:0/1349094582 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f07940cb6f0 0x7f0794153310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:37.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.747+0000 7f0798d98700 1 --2- 172.21.15.16:0/1349094582 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f07940ce270 0x7f07940b5b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:37.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.747+0000 7f07937fe700 1 -- 172.21.15.16:0/1349094582 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f07940ce270 msgr2=0x7f07940b5b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:37.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.747+0000 7f07937fe700 1 --2- 172.21.15.16:0/1349094582 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f07940ce270 0x7f07940b5b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:37.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.747+0000 7f07937fe700 1 -- 172.21.15.16:0/1349094582 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f07940cb6f0 msgr2=0x7f0794153310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:37.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.747+0000 7f07937fe700 1 --2- 172.21.15.16:0/1349094582 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f07940cb6f0 0x7f0794153310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:37.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.747+0000 7f07937fe700 1 -- 172.21.15.16:0/1349094582 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07880094a0 con 0x7f07940cde90 2023-10-16T13:44:37.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.747+0000 7f07937fe700 1 --2- 172.21.15.16:0/1349094582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07940cde90 0x7f07940b40f0 secure :-1 s=READY pgs=2268 cs=0 l=1 rev1=1 crypto rx=0x7f078800e9e0 tx=0x7f078800ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:37.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.747+0000 7f0790ff9700 1 -- 172.21.15.16:0/1349094582 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f078800ee80 con 0x7f07940cde90 2023-10-16T13:44:37.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.747+0000 7f079a7fb700 1 -- 172.21.15.16:0/1349094582 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07840094a0 con 0x7f07940cde90 2023-10-16T13:44:37.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.747+0000 7f079a7fb700 1 -- 172.21.15.16:0/1349094582 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07940b5020 con 0x7f07940cde90 2023-10-16T13:44:37.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.747+0000 7f0790ff9700 1 -- 172.21.15.16:0/1349094582 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f078800c7f0 con 0x7f07940cde90 2023-10-16T13:44:37.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.747+0000 7f0790ff9700 1 -- 172.21.15.16:0/1349094582 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f078800cb60 con 0x7f07940cde90 2023-10-16T13:44:37.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.748+0000 7f079a7fb700 1 -- 172.21.15.16:0/1349094582 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07740052f0 con 0x7f07940cde90 2023-10-16T13:44:37.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.749+0000 7f0790ff9700 1 -- 172.21.15.16:0/1349094582 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 126) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f078800ccc0 con 0x7f07940cde90 2023-10-16T13:44:37.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.749+0000 7f0790ff9700 1 --2- 172.21.15.16:0/1349094582 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f078006b020 0x7f078006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:37.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.749+0000 7f0790ff9700 1 -- 172.21.15.16:0/1349094582 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 5305+0+0 (secure 0 0 0) 0x7f0788089990 con 0x7f07940cde90 2023-10-16T13:44:37.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.749+0000 7f0793fff700 1 --2- 172.21.15.16:0/1349094582 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f078006b020 0x7f078006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:37.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.750+0000 7f0793fff700 1 --2- 172.21.15.16:0/1349094582 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f078006b020 0x7f078006d4e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f07840090b0 tx=0x7f0784008030 comp rx=0 tx=0).ready entity=mgr.10665 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:37.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.754+0000 7f0790ff9700 1 -- 172.21.15.16:0/1349094582 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f07880549b0 con 0x7f07940cde90 2023-10-16T13:44:37.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.961+0000 7f079a7fb700 1 -- 172.21.15.16:0/1349094582 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0774005b20 con 0x7f07940cde90 2023-10-16T13:44:37.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.962+0000 7f0790ff9700 1 -- 172.21.15.16:0/1349094582 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v202) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0788057fd0 con 0x7f07940cde90 2023-10-16T13:44:37.966 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T13:44:37.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.964+0000 7f079a7fb700 1 -- 172.21.15.16:0/1349094582 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f078006b020 msgr2=0x7f078006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:37.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.964+0000 7f079a7fb700 1 --2- 172.21.15.16:0/1349094582 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f078006b020 0x7f078006d4e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f07840090b0 tx=0x7f0784008030 comp rx=0 tx=0).stop 2023-10-16T13:44:37.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.964+0000 7f079a7fb700 1 -- 172.21.15.16:0/1349094582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07940cde90 msgr2=0x7f07940b40f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:37.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.964+0000 7f079a7fb700 1 --2- 172.21.15.16:0/1349094582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07940cde90 0x7f07940b40f0 secure :-1 s=READY pgs=2268 cs=0 l=1 rev1=1 crypto rx=0x7f078800e9e0 tx=0x7f078800ecf0 comp rx=0 tx=0).stop 2023-10-16T13:44:37.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.964+0000 7f079a7fb700 1 -- 172.21.15.16:0/1349094582 shutdown_connections 2023-10-16T13:44:37.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.964+0000 7f079a7fb700 1 --2- 172.21.15.16:0/1349094582 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f078006b020 0x7f078006d4e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:37.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.964+0000 7f079a7fb700 1 --2- 172.21.15.16:0/1349094582 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f07940ce270 0x7f07940b5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:37.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.964+0000 7f079a7fb700 1 --2- 172.21.15.16:0/1349094582 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f07940cb6f0 0x7f0794153310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:37.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.964+0000 7f079a7fb700 1 --2- 172.21.15.16:0/1349094582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07940cde90 0x7f07940b40f0 unknown :-1 s=CLOSED pgs=2268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:37.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.964+0000 7f079a7fb700 1 -- 172.21.15.16:0/1349094582 >> 172.21.15.16:0/1349094582 conn(0x7f07940d14d0 msgr2=0x7f07940cff50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:37.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.964+0000 7f079a7fb700 1 -- 172.21.15.16:0/1349094582 shutdown_connections 2023-10-16T13:44:37.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:37.964+0000 7f079a7fb700 1 -- 172.21.15.16:0/1349094582 wait complete. 2023-10-16T13:44:37.980 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-16T13:44:38.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.102+0000 7f2bea9f3700 1 Processor -- start 2023-10-16T13:44:38.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.102+0000 7f2bea9f3700 1 -- start start 2023-10-16T13:44:38.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.103+0000 7f2bea9f3700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f2be415b9f0 0x7f2be40b29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:38.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.103+0000 7f2bea9f3700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2be406d940 0x7f2be406ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:38.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.103+0000 7f2bea9f3700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2be40c3490 con 0x7f2be40ceb70 2023-10-16T13:44:38.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.103+0000 7f2bea9f3700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2be406e300 con 0x7f2be415b9f0 2023-10-16T13:44:38.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.103+0000 7f2bea9f3700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2be40cf700 con 0x7f2be406d940 2023-10-16T13:44:38.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.103+0000 7f2be8f90700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2be40ceb70 0x7f2be40ceff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:40872/0 (socket says 172.21.15.16:40872) 2023-10-16T13:44:38.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.103+0000 7f2be8f90700 1 -- 172.21.15.16:0/487895631 learned_addr learned my addr 172.21.15.16:0/487895631 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:44:38.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.103+0000 7f2be3fff700 1 --2- 172.21.15.16:0/487895631 >> v2:172.21.15.49:3300/0 conn(0x7f2be415b9f0 0x7f2be40b29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:38.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.103+0000 7f2be2ffd700 1 -- 172.21.15.16:0/487895631 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2519229958 0 0) 0x7f2be40c3490 con 0x7f2be40ceb70 2023-10-16T13:44:38.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.103+0000 7f2be2ffd700 1 -- 172.21.15.16:0/487895631 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bd00035b0 con 0x7f2be40ceb70 2023-10-16T13:44:38.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.103+0000 7f2be37fe700 1 --2- 172.21.15.16:0/487895631 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2be406d940 0x7f2be406ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:38.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.103+0000 7f2be2ffd700 1 -- 172.21.15.16:0/487895631 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2427186894 0 0) 0x7f2bd00035b0 con 0x7f2be40ceb70 2023-10-16T13:44:38.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.103+0000 7f2be2ffd700 1 -- 172.21.15.16:0/487895631 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2be406d940 msgr2=0x7f2be406ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:38.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.103+0000 7f2be2ffd700 1 --2- 172.21.15.16:0/487895631 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2be406d940 0x7f2be406ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:38.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.103+0000 7f2be2ffd700 1 -- 172.21.15.16:0/487895631 >> v2:172.21.15.49:3300/0 conn(0x7f2be415b9f0 msgr2=0x7f2be40b29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:38.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.103+0000 7f2be2ffd700 1 --2- 172.21.15.16:0/487895631 >> v2:172.21.15.49:3300/0 conn(0x7f2be415b9f0 0x7f2be40b29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:38.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.103+0000 7f2be2ffd700 1 -- 172.21.15.16:0/487895631 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2be40cdad0 con 0x7f2be40ceb70 2023-10-16T13:44:38.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.103+0000 7f2be2ffd700 1 -- 172.21.15.16:0/487895631 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f2bdc0019f0 con 0x7f2be40ceb70 2023-10-16T13:44:38.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.103+0000 7f2be2ffd700 1 -- 172.21.15.16:0/487895631 >> v1:172.21.15.16:6789/0 conn(0x7f2be40ceb70 legacy=0x7f2be40ceff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:38.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.104+0000 7f2be2ffd700 1 --2- 172.21.15.16:0/487895631 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2bd0003a00 0x7f2bd0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:38.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.104+0000 7f2be2ffd700 1 --2- 172.21.15.16:0/487895631 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2be406d940 0x7f2bd0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:38.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.104+0000 7f2be2ffd700 1 --2- 172.21.15.16:0/487895631 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2bd0004d90 0x7f2bd0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:38.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.104+0000 7f2be2ffd700 1 -- 172.21.15.16:0/487895631 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2be406e300 con 0x7f2bd0004d90 2023-10-16T13:44:38.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.104+0000 7f2be2ffd700 1 -- 172.21.15.16:0/487895631 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2be40cf700 con 0x7f2bd0003a00 2023-10-16T13:44:38.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.104+0000 7f2be2ffd700 1 -- 172.21.15.16:0/487895631 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd000ba10 con 0x7f2be406d940 2023-10-16T13:44:38.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.104+0000 7f2be8f90700 1 --2- 172.21.15.16:0/487895631 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2bd0004d90 0x7f2bd0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:38.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.104+0000 7f2be3fff700 1 --2- 172.21.15.16:0/487895631 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2bd0003a00 0x7f2bd0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:38.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.104+0000 7f2be37fe700 1 --2- 172.21.15.16:0/487895631 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2be406d940 0x7f2bd0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:38.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.104+0000 7f2be8f90700 1 -- 172.21.15.16:0/487895631 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2be406d940 msgr2=0x7f2bd0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:38.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.104+0000 7f2be8f90700 1 --2- 172.21.15.16:0/487895631 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2be406d940 0x7f2bd0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:38.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.104+0000 7f2be8f90700 1 -- 172.21.15.16:0/487895631 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2bd0003a00 msgr2=0x7f2bd0003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:38.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.104+0000 7f2be8f90700 1 --2- 172.21.15.16:0/487895631 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2bd0003a00 0x7f2bd0003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:38.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.104+0000 7f2be8f90700 1 -- 172.21.15.16:0/487895631 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bd00035b0 con 0x7f2bd0004d90 2023-10-16T13:44:38.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.104+0000 7f2be37fe700 1 --2- 172.21.15.16:0/487895631 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2be406d940 0x7f2bd0004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:44:38.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.104+0000 7f2be8f90700 1 --2- 172.21.15.16:0/487895631 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2bd0004d90 0x7f2bd0005210 secure :-1 s=READY pgs=2270 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc00a860 tx=0x7f2bd0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a7f464b89c3b3a0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:38.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.104+0000 7f2be2ffd700 1 -- 172.21.15.16:0/487895631 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bdc012600 con 0x7f2bd0004d90 2023-10-16T13:44:38.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.104+0000 7f2be2ffd700 1 -- 172.21.15.16:0/487895631 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bdc012760 con 0x7f2bd0004d90 2023-10-16T13:44:38.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.104+0000 7f2be2ffd700 1 -- 172.21.15.16:0/487895631 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bdc012b10 con 0x7f2bd0004d90 2023-10-16T13:44:38.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.105+0000 7f2bea9f3700 1 -- 172.21.15.16:0/487895631 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2bd0004d90 msgr2=0x7f2bd0005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:38.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.105+0000 7f2bea9f3700 1 --2- 172.21.15.16:0/487895631 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2bd0004d90 0x7f2bd0005210 secure :-1 s=READY pgs=2270 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc00a860 tx=0x7f2bd0005880 comp rx=0 tx=0).stop 2023-10-16T13:44:38.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.105+0000 7f2be8f90700 1 -- 172.21.15.16:0/487895631 reap_dead start 2023-10-16T13:44:38.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.105+0000 7f2bea9f3700 1 -- 172.21.15.16:0/487895631 shutdown_connections 2023-10-16T13:44:38.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.105+0000 7f2bea9f3700 1 -- 172.21.15.16:0/487895631 >> 172.21.15.16:0/487895631 conn(0x7f2be40cc5b0 msgr2=0x7f2be40cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:38.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.105+0000 7f2bea9f3700 1 -- 172.21.15.16:0/487895631 shutdown_connections 2023-10-16T13:44:38.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.105+0000 7f2bea9f3700 1 -- 172.21.15.16:0/487895631 wait complete. 2023-10-16T13:44:38.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.106+0000 7f2bea9f3700 1 Processor -- start 2023-10-16T13:44:38.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.106+0000 7f2bea9f3700 1 -- start start 2023-10-16T13:44:38.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.107+0000 7f2bea9f3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2be414d480 0x7f2be40b2960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:38.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.107+0000 7f2bea9f3700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2be40b4210 0x7f2be415c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:38.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.107+0000 7f2bea9f3700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2be40b4690 0x7f2be415c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:38.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.107+0000 7f2bea9f3700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2be40c36d0 con 0x7f2be414d480 2023-10-16T13:44:38.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.107+0000 7f2bea9f3700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2be414d860 con 0x7f2be40b4690 2023-10-16T13:44:38.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.107+0000 7f2bea9f3700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2be40b4b20 con 0x7f2be40b4210 2023-10-16T13:44:38.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.107+0000 7f2be3fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2be414d480 0x7f2be40b2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:38.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.107+0000 7f2be3fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2be414d480 0x7f2be40b2960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51606/0 (socket says 172.21.15.16:51606) 2023-10-16T13:44:38.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.107+0000 7f2be3fff700 1 -- 172.21.15.16:0/1463032147 learned_addr learned my addr 172.21.15.16:0/1463032147 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:38.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.107+0000 7f2be3fff700 1 -- 172.21.15.16:0/1463032147 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2be40b4210 msgr2=0x7f2be415c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:38.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.107+0000 7f2be3fff700 1 --2- 172.21.15.16:0/1463032147 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2be40b4210 0x7f2be415c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:38.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.107+0000 7f2be3fff700 1 -- 172.21.15.16:0/1463032147 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2be40b4690 msgr2=0x7f2be415c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:38.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.107+0000 7f2be3fff700 1 --2- 172.21.15.16:0/1463032147 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2be40b4690 0x7f2be415c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:38.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.107+0000 7f2be3fff700 1 -- 172.21.15.16:0/1463032147 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2be40b4e00 con 0x7f2be414d480 2023-10-16T13:44:38.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.108+0000 7f2be3fff700 1 --2- 172.21.15.16:0/1463032147 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2be414d480 0x7f2be40b2960 secure :-1 s=READY pgs=2271 cs=0 l=1 rev1=1 crypto rx=0x7f2bd4011aa0 tx=0x7f2bd4011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:38.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.108+0000 7f2be0ff9700 1 -- 172.21.15.16:0/1463032147 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bd400f900 con 0x7f2be414d480 2023-10-16T13:44:38.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.108+0000 7f2be0ff9700 1 -- 172.21.15.16:0/1463032147 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bd400fa80 con 0x7f2be414d480 2023-10-16T13:44:38.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.108+0000 7f2bea9f3700 1 -- 172.21.15.16:0/1463032147 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2be40b50e0 con 0x7f2be414d480 2023-10-16T13:44:38.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.108+0000 7f2be0ff9700 1 -- 172.21.15.16:0/1463032147 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bd400fe30 con 0x7f2be414d480 2023-10-16T13:44:38.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.108+0000 7f2bea9f3700 1 -- 172.21.15.16:0/1463032147 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2be40ca180 con 0x7f2be414d480 2023-10-16T13:44:38.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.109+0000 7f2bea9f3700 1 -- 172.21.15.16:0/1463032147 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2be404ebb0 con 0x7f2be414d480 2023-10-16T13:44:38.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.109+0000 7f2be0ff9700 1 -- 172.21.15.16:0/1463032147 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 126) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2bd4014440 con 0x7f2be414d480 2023-10-16T13:44:38.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.110+0000 7f2be0ff9700 1 --2- 172.21.15.16:0/1463032147 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f2bc406b020 0x7f2bc406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:38.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.110+0000 7f2be0ff9700 1 -- 172.21.15.16:0/1463032147 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 5305+0+0 (secure 0 0 0) 0x7f2bd4089ca0 con 0x7f2be414d480 2023-10-16T13:44:38.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.110+0000 7f2be37fe700 1 --2- 172.21.15.16:0/1463032147 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f2bc406b020 0x7f2bc406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:38.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.110+0000 7f2be37fe700 1 --2- 172.21.15.16:0/1463032147 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f2bc406b020 0x7f2bc406d4e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2be40cb340 tx=0x7f2bcc008040 comp rx=0 tx=0).ready entity=mgr.10665 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:38.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.114+0000 7f2be0ff9700 1 -- 172.21.15.16:0/1463032147 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2bd4054f20 con 0x7f2be414d480 2023-10-16T13:44:38.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.321+0000 7f2bea9f3700 1 -- 172.21.15.16:0/1463032147 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f2be40b5ab0 con 0x7f2be414d480 2023-10-16T13:44:38.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.321+0000 7f2be0ff9700 1 -- 172.21.15.16:0/1463032147 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v202) v1 ==== 84+0+65 (secure 0 0 0) 0x7f2bd4058540 con 0x7f2be414d480 2023-10-16T13:44:38.325 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:44:38.325 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2023-10-16T13:44:38.326 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:44:38.326 INFO:teuthology.orchestra.run.smithi016.stdout: "poolnum": 1, 2023-10-16T13:44:38.326 INFO:teuthology.orchestra.run.smithi016.stdout: "poolname": ".mgr" 2023-10-16T13:44:38.326 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:44:38.326 INFO:teuthology.orchestra.run.smithi016.stdout:] 2023-10-16T13:44:38.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.324+0000 7f2bea9f3700 1 -- 172.21.15.16:0/1463032147 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f2bc406b020 msgr2=0x7f2bc406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:38.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.324+0000 7f2bea9f3700 1 --2- 172.21.15.16:0/1463032147 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f2bc406b020 0x7f2bc406d4e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2be40cb340 tx=0x7f2bcc008040 comp rx=0 tx=0).stop 2023-10-16T13:44:38.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.324+0000 7f2bea9f3700 1 -- 172.21.15.16:0/1463032147 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2be414d480 msgr2=0x7f2be40b2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:38.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.324+0000 7f2bea9f3700 1 --2- 172.21.15.16:0/1463032147 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2be414d480 0x7f2be40b2960 secure :-1 s=READY pgs=2271 cs=0 l=1 rev1=1 crypto rx=0x7f2bd4011aa0 tx=0x7f2bd4011e60 comp rx=0 tx=0).stop 2023-10-16T13:44:38.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.324+0000 7f2bea9f3700 1 -- 172.21.15.16:0/1463032147 shutdown_connections 2023-10-16T13:44:38.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.324+0000 7f2bea9f3700 1 --2- 172.21.15.16:0/1463032147 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f2bc406b020 0x7f2bc406d4e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:38.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.324+0000 7f2bea9f3700 1 --2- 172.21.15.16:0/1463032147 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2be40b4690 0x7f2be415c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:38.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.324+0000 7f2bea9f3700 1 --2- 172.21.15.16:0/1463032147 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2be414d480 0x7f2be40b2960 unknown :-1 s=CLOSED pgs=2271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:38.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.324+0000 7f2bea9f3700 1 --2- 172.21.15.16:0/1463032147 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2be40b4210 0x7f2be415c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:38.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.324+0000 7f2bea9f3700 1 -- 172.21.15.16:0/1463032147 >> 172.21.15.16:0/1463032147 conn(0x7f2be40cc5b0 msgr2=0x7f2be415e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:38.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.324+0000 7f2bea9f3700 1 -- 172.21.15.16:0/1463032147 shutdown_connections 2023-10-16T13:44:38.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.324+0000 7f2bea9f3700 1 -- 172.21.15.16:0/1463032147 wait complete. 2023-10-16T13:44:38.339 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-16T13:44:38.340 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-16T13:44:38.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.466+0000 7effc0365700 1 Processor -- start 2023-10-16T13:44:38.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.466+0000 7effc0365700 1 -- start start 2023-10-16T13:44:38.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.467+0000 7effc0365700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7effb815a3d0 0x7effb806d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:38.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.467+0000 7effc0365700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7effb815e0b0 0x7effb80ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:38.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.467+0000 7effc0365700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7effb80cf430 0x7effb80cdeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:38.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.467+0000 7effc0365700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effb815f0c0 con 0x7effb80cf430 2023-10-16T13:44:38.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.467+0000 7effc0365700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effb8072990 con 0x7effb815e0b0 2023-10-16T13:44:38.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.467+0000 7effc0365700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effb806de50 con 0x7effb815a3d0 2023-10-16T13:44:38.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.467+0000 7effbe902700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7effb80cf430 0x7effb80cdeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:38.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.467+0000 7effbe902700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7effb80cf430 0x7effb80cdeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51618/0 (socket says 172.21.15.16:51618) 2023-10-16T13:44:38.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.467+0000 7effbe902700 1 -- 172.21.15.16:0/760023450 learned_addr learned my addr 172.21.15.16:0/760023450 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:38.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.467+0000 7effbe101700 1 --2- 172.21.15.16:0/760023450 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7effb815a3d0 0x7effb806d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:38.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.467+0000 7effbd900700 1 --2- 172.21.15.16:0/760023450 >> v2:172.21.15.49:3300/0 conn(0x7effb815e0b0 0x7effb80ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:38.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.467+0000 7effbe902700 1 -- 172.21.15.16:0/760023450 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7effb815a3d0 msgr2=0x7effb806d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:38.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.467+0000 7effbe902700 1 --2- 172.21.15.16:0/760023450 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7effb815a3d0 0x7effb806d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:38.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.467+0000 7effbe902700 1 -- 172.21.15.16:0/760023450 >> v2:172.21.15.49:3300/0 conn(0x7effb815e0b0 msgr2=0x7effb80ceef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:38.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.467+0000 7effbe902700 1 --2- 172.21.15.16:0/760023450 >> v2:172.21.15.49:3300/0 conn(0x7effb815e0b0 0x7effb80ceef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:38.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.467+0000 7effbe902700 1 -- 172.21.15.16:0/760023450 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effb80cb870 con 0x7effb80cf430 2023-10-16T13:44:38.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.467+0000 7effbe902700 1 --2- 172.21.15.16:0/760023450 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effb80cf430 0x7effb80cdeb0 secure :-1 s=READY pgs=2272 cs=0 l=1 rev1=1 crypto rx=0x7effb80cde60 tx=0x7effb000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de4dfd95ab5283e1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:38.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.468+0000 7effbd0ff700 1 -- 172.21.15.16:0/760023450 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb000e070 con 0x7effb80cf430 2023-10-16T13:44:38.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.468+0000 7effbd0ff700 1 -- 172.21.15.16:0/760023450 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effb000b910 con 0x7effb80cf430 2023-10-16T13:44:38.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.468+0000 7effbd0ff700 1 -- 172.21.15.16:0/760023450 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb000bcc0 con 0x7effb80cf430 2023-10-16T13:44:38.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.468+0000 7effc0365700 1 -- 172.21.15.16:0/760023450 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effb80cf430 msgr2=0x7effb80cdeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:38.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.468+0000 7effc0365700 1 --2- 172.21.15.16:0/760023450 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effb80cf430 0x7effb80cdeb0 secure :-1 s=READY pgs=2272 cs=0 l=1 rev1=1 crypto rx=0x7effb80cde60 tx=0x7effb000b330 comp rx=0 tx=0).stop 2023-10-16T13:44:38.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.468+0000 7effc0365700 1 -- 172.21.15.16:0/760023450 shutdown_connections 2023-10-16T13:44:38.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.468+0000 7effc0365700 1 --2- 172.21.15.16:0/760023450 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effb80cf430 0x7effb80cdeb0 unknown :-1 s=CLOSED pgs=2272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:38.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.468+0000 7effc0365700 1 --2- 172.21.15.16:0/760023450 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7effb815a3d0 0x7effb806d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:38.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.468+0000 7effc0365700 1 --2- 172.21.15.16:0/760023450 >> v2:172.21.15.49:3300/0 conn(0x7effb815e0b0 0x7effb80ceef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:38.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.468+0000 7effc0365700 1 -- 172.21.15.16:0/760023450 >> 172.21.15.16:0/760023450 conn(0x7effb806f430 msgr2=0x7effb806f810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:38.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.468+0000 7effc0365700 1 -- 172.21.15.16:0/760023450 shutdown_connections 2023-10-16T13:44:38.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.468+0000 7effc0365700 1 -- 172.21.15.16:0/760023450 wait complete. 2023-10-16T13:44:38.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.469+0000 7effc0365700 1 Processor -- start 2023-10-16T13:44:38.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.470+0000 7effc0365700 1 -- start start 2023-10-16T13:44:38.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.470+0000 7effc0365700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7effb80cf430 0x7effb80c9080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:38.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.470+0000 7effc0365700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7effb815a3d0 0x7effb81ef8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:38.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.470+0000 7effc0365700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effb815e0b0 0x7effb81efe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:38.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.470+0000 7effc0365700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effb815c970 con 0x7effb815e0b0 2023-10-16T13:44:38.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.470+0000 7effc0365700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effb80c8dd0 con 0x7effb80cf430 2023-10-16T13:44:38.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.470+0000 7effc0365700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effb80c8250 con 0x7effb815a3d0 2023-10-16T13:44:38.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.470+0000 7effbe902700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effb815e0b0 0x7effb81efe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:38.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.470+0000 7effbe902700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effb815e0b0 0x7effb81efe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51624/0 (socket says 172.21.15.16:51624) 2023-10-16T13:44:38.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.470+0000 7effbe902700 1 -- 172.21.15.16:0/600004344 learned_addr learned my addr 172.21.15.16:0/600004344 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:38.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.471+0000 7effbe902700 1 -- 172.21.15.16:0/600004344 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7effb815a3d0 msgr2=0x7effb81ef8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:38.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.471+0000 7effbe902700 1 --2- 172.21.15.16:0/600004344 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7effb815a3d0 0x7effb81ef8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:38.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.471+0000 7effbe902700 1 -- 172.21.15.16:0/600004344 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7effb80cf430 msgr2=0x7effb80c9080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:38.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.471+0000 7effbe902700 1 --2- 172.21.15.16:0/600004344 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7effb80cf430 0x7effb80c9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:38.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.471+0000 7effbe902700 1 -- 172.21.15.16:0/600004344 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effb0009b30 con 0x7effb815e0b0 2023-10-16T13:44:38.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.471+0000 7effbe902700 1 --2- 172.21.15.16:0/600004344 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effb815e0b0 0x7effb81efe10 secure :-1 s=READY pgs=2273 cs=0 l=1 rev1=1 crypto rx=0x7effb8067440 tx=0x7effb0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:38.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.471+0000 7effaeffd700 1 -- 172.21.15.16:0/600004344 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb0004450 con 0x7effb815e0b0 2023-10-16T13:44:38.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.471+0000 7effaeffd700 1 -- 172.21.15.16:0/600004344 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effb00045b0 con 0x7effb815e0b0 2023-10-16T13:44:38.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.471+0000 7effaeffd700 1 -- 172.21.15.16:0/600004344 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb0004960 con 0x7effb815e0b0 2023-10-16T13:44:38.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.471+0000 7effc0365700 1 -- 172.21.15.16:0/600004344 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effb80c8530 con 0x7effb815e0b0 2023-10-16T13:44:38.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.471+0000 7effc0365700 1 -- 172.21.15.16:0/600004344 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effb80c8af0 con 0x7effb815e0b0 2023-10-16T13:44:38.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.472+0000 7effc0365700 1 -- 172.21.15.16:0/600004344 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effb804ec20 con 0x7effb815e0b0 2023-10-16T13:44:38.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.473+0000 7effaeffd700 1 -- 172.21.15.16:0/600004344 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 126) v1 ==== 89331+0+0 (secure 0 0 0) 0x7effb001a4c0 con 0x7effb815e0b0 2023-10-16T13:44:38.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.473+0000 7effaeffd700 1 --2- 172.21.15.16:0/600004344 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7eff9c06b020 0x7eff9c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:38.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.473+0000 7effaeffd700 1 -- 172.21.15.16:0/600004344 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 5305+0+0 (secure 0 0 0) 0x7effb008c250 con 0x7effb815e0b0 2023-10-16T13:44:38.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.473+0000 7effbe101700 1 --2- 172.21.15.16:0/600004344 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7eff9c06b020 0x7eff9c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:38.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.473+0000 7effbe101700 1 --2- 172.21.15.16:0/600004344 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7eff9c06b020 0x7eff9c06d4e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7effb80c6c90 tx=0x7effb4008040 comp rx=0 tx=0).ready entity=mgr.10665 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:38.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.477+0000 7effaeffd700 1 -- 172.21.15.16:0/600004344 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7effb005ad90 con 0x7effb815e0b0 2023-10-16T13:44:38.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.681+0000 7effc0365700 1 -- 172.21.15.16:0/600004344 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7effb80b6260 con 0x7effb815e0b0 2023-10-16T13:44:38.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.769+0000 7effaeffd700 1 -- 172.21.15.16:0/600004344 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v203) v1 ==== 138+0+0 (secure 0 0 0) 0x7effb005a920 con 0x7effb815e0b0 2023-10-16T13:44:38.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.771+0000 7effc0365700 1 -- 172.21.15.16:0/600004344 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7eff9c06b020 msgr2=0x7eff9c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:38.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.771+0000 7effc0365700 1 --2- 172.21.15.16:0/600004344 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7eff9c06b020 0x7eff9c06d4e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7effb80c6c90 tx=0x7effb4008040 comp rx=0 tx=0).stop 2023-10-16T13:44:38.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.771+0000 7effc0365700 1 -- 172.21.15.16:0/600004344 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effb815e0b0 msgr2=0x7effb81efe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:38.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.771+0000 7effc0365700 1 --2- 172.21.15.16:0/600004344 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effb815e0b0 0x7effb81efe10 secure :-1 s=READY pgs=2273 cs=0 l=1 rev1=1 crypto rx=0x7effb8067440 tx=0x7effb0004ad0 comp rx=0 tx=0).stop 2023-10-16T13:44:38.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.772+0000 7effc0365700 1 -- 172.21.15.16:0/600004344 shutdown_connections 2023-10-16T13:44:38.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.772+0000 7effc0365700 1 --2- 172.21.15.16:0/600004344 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7eff9c06b020 0x7eff9c06d4e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:38.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.772+0000 7effc0365700 1 --2- 172.21.15.16:0/600004344 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effb815e0b0 0x7effb81efe10 unknown :-1 s=CLOSED pgs=2273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:38.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.772+0000 7effc0365700 1 --2- 172.21.15.16:0/600004344 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7effb80cf430 0x7effb80c9080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:38.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.772+0000 7effc0365700 1 --2- 172.21.15.16:0/600004344 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7effb815a3d0 0x7effb81ef8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:38.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.772+0000 7effc0365700 1 -- 172.21.15.16:0/600004344 >> 172.21.15.16:0/600004344 conn(0x7effb806f430 msgr2=0x7effb8157830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:38.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.772+0000 7effc0365700 1 -- 172.21.15.16:0/600004344 shutdown_connections 2023-10-16T13:44:38.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.772+0000 7effc0365700 1 -- 172.21.15.16:0/600004344 wait complete. 2023-10-16T13:44:38.778 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' created 2023-10-16T13:44:38.791 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-16T13:44:38.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.920+0000 7faf492b1700 1 Processor -- start 2023-10-16T13:44:38.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.920+0000 7faf492b1700 1 -- start start 2023-10-16T13:44:38.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.920+0000 7faf492b1700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faf4415d8d0 0x7faf440b2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:38.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.921+0000 7faf492b1700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf440b2f30 con 0x7faf440b8fa0 2023-10-16T13:44:38.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.921+0000 7faf492b1700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf440d1c40 con 0x7faf440ce350 2023-10-16T13:44:38.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.921+0000 7faf492b1700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf440d0bb0 con 0x7faf4415d8d0 2023-10-16T13:44:38.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.921+0000 7faf427fc700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7faf440b8fa0 0x7faf440b6720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:40902/0 (socket says 172.21.15.16:40902) 2023-10-16T13:44:38.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.921+0000 7faf427fc700 1 -- 172.21.15.16:0/3375956930 learned_addr learned my addr 172.21.15.16:0/3375956930 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:44:38.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.921+0000 7faf41ffb700 1 -- 172.21.15.16:0/3375956930 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4079064833 0 0) 0x7faf440b2f30 con 0x7faf440b8fa0 2023-10-16T13:44:38.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.921+0000 7faf41ffb700 1 -- 172.21.15.16:0/3375956930 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf2c0035b0 con 0x7faf440b8fa0 2023-10-16T13:44:38.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.921+0000 7faf42ffd700 1 --2- 172.21.15.16:0/3375956930 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faf4415d8d0 0x7faf440b2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:38.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.921+0000 7faf41ffb700 1 -- 172.21.15.16:0/3375956930 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2054341824 0 0) 0x7faf2c0035b0 con 0x7faf440b8fa0 2023-10-16T13:44:38.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.921+0000 7faf41ffb700 1 -- 172.21.15.16:0/3375956930 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faf4415d8d0 msgr2=0x7faf440b2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:38.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.921+0000 7faf41ffb700 1 --2- 172.21.15.16:0/3375956930 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faf4415d8d0 0x7faf440b2830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:38.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.921+0000 7faf41ffb700 1 -- 172.21.15.16:0/3375956930 >> v1:172.21.15.49:6789/0 conn(0x7faf440ce350 legacy=0x7faf440d29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:38.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.921+0000 7faf41ffb700 1 -- 172.21.15.16:0/3375956930 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf440d1ee0 con 0x7faf440b8fa0 2023-10-16T13:44:38.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.921+0000 7faf41ffb700 1 -- 172.21.15.16:0/3375956930 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7faf34001670 con 0x7faf440b8fa0 2023-10-16T13:44:38.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.921+0000 7faf41ffb700 1 -- 172.21.15.16:0/3375956930 >> v1:172.21.15.16:6789/0 conn(0x7faf440b8fa0 legacy=0x7faf440b6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:38.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.921+0000 7faf41ffb700 1 --2- 172.21.15.16:0/3375956930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf2c003a30 0x7faf2c003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:38.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.921+0000 7faf41ffb700 1 --2- 172.21.15.16:0/3375956930 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faf4415d8d0 0x7faf2c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:38.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.921+0000 7faf41ffb700 1 --2- 172.21.15.16:0/3375956930 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faf2c004d90 0x7faf2c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:38.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.921+0000 7faf41ffb700 1 -- 172.21.15.16:0/3375956930 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf440d0bb0 con 0x7faf2c003a30 2023-10-16T13:44:38.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.921+0000 7faf41ffb700 1 -- 172.21.15.16:0/3375956930 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf2c00bbd0 con 0x7faf2c004d90 2023-10-16T13:44:38.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.921+0000 7faf41ffb700 1 -- 172.21.15.16:0/3375956930 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf2c00bd90 con 0x7faf4415d8d0 2023-10-16T13:44:38.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.921+0000 7faf42ffd700 1 --2- 172.21.15.16:0/3375956930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf2c003a30 0x7faf2c003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:38.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.922+0000 7faf427fc700 1 --2- 172.21.15.16:0/3375956930 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faf4415d8d0 0x7faf2c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:38.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.922+0000 7faf437fe700 1 --2- 172.21.15.16:0/3375956930 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faf2c004d90 0x7faf2c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:38.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.922+0000 7faf42ffd700 1 -- 172.21.15.16:0/3375956930 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faf4415d8d0 msgr2=0x7faf2c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:38.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.922+0000 7faf42ffd700 1 --2- 172.21.15.16:0/3375956930 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faf4415d8d0 0x7faf2c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:38.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.922+0000 7faf42ffd700 1 -- 172.21.15.16:0/3375956930 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faf2c004d90 msgr2=0x7faf2c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:38.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.922+0000 7faf42ffd700 1 --2- 172.21.15.16:0/3375956930 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faf2c004d90 0x7faf2c005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:38.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.922+0000 7faf42ffd700 1 -- 172.21.15.16:0/3375956930 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf2c0035b0 con 0x7faf2c003a30 2023-10-16T13:44:38.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.922+0000 7faf437fe700 1 --2- 172.21.15.16:0/3375956930 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faf2c004d90 0x7faf2c005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:44:38.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.922+0000 7faf42ffd700 1 --2- 172.21.15.16:0/3375956930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf2c003a30 0x7faf2c003e50 secure :-1 s=READY pgs=2275 cs=0 l=1 rev1=1 crypto rx=0x7faf38009b60 tx=0x7faf2c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a35f23bee8e7ef2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:38.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.922+0000 7faf41ffb700 1 -- 172.21.15.16:0/3375956930 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf3800e6c0 con 0x7faf2c003a30 2023-10-16T13:44:38.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.922+0000 7faf41ffb700 1 -- 172.21.15.16:0/3375956930 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf3800e820 con 0x7faf2c003a30 2023-10-16T13:44:38.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.922+0000 7faf41ffb700 1 -- 172.21.15.16:0/3375956930 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf3800eb90 con 0x7faf2c003a30 2023-10-16T13:44:38.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.923+0000 7faf492b1700 1 -- 172.21.15.16:0/3375956930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf2c003a30 msgr2=0x7faf2c003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:38.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.923+0000 7faf492b1700 1 --2- 172.21.15.16:0/3375956930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf2c003a30 0x7faf2c003e50 secure :-1 s=READY pgs=2275 cs=0 l=1 rev1=1 crypto rx=0x7faf38009b60 tx=0x7faf2c005880 comp rx=0 tx=0).stop 2023-10-16T13:44:38.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.923+0000 7faf437fe700 1 -- 172.21.15.16:0/3375956930 reap_dead start 2023-10-16T13:44:38.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.923+0000 7faf492b1700 1 -- 172.21.15.16:0/3375956930 shutdown_connections 2023-10-16T13:44:38.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.923+0000 7faf492b1700 1 -- 172.21.15.16:0/3375956930 >> 172.21.15.16:0/3375956930 conn(0x7faf44072330 msgr2=0x7faf44072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:38.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.923+0000 7faf492b1700 1 -- 172.21.15.16:0/3375956930 shutdown_connections 2023-10-16T13:44:38.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.923+0000 7faf492b1700 1 -- 172.21.15.16:0/3375956930 wait complete. 2023-10-16T13:44:38.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.924+0000 7faf492b1700 1 Processor -- start 2023-10-16T13:44:38.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.924+0000 7faf492b1700 1 -- start start 2023-10-16T13:44:38.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.925+0000 7faf492b1700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faf2c003a30 0x7faf440c9af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:38.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.925+0000 7faf492b1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf440ca700 0x7faf440cbd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:38.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.925+0000 7faf492b1700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faf440ccc60 0x7faf4414f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:38.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.925+0000 7faf492b1700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf440ca030 con 0x7faf440ca700 2023-10-16T13:44:38.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.925+0000 7faf492b1700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf440cb8d0 con 0x7faf2c003a30 2023-10-16T13:44:38.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.925+0000 7faf492b1700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf440cacc0 con 0x7faf440ccc60 2023-10-16T13:44:38.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.925+0000 7faf427fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf440ca700 0x7faf440cbd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:38.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.925+0000 7faf427fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf440ca700 0x7faf440cbd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51642/0 (socket says 172.21.15.16:51642) 2023-10-16T13:44:38.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.925+0000 7faf427fc700 1 -- 172.21.15.16:0/1641456322 learned_addr learned my addr 172.21.15.16:0/1641456322 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:38.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.925+0000 7faf42ffd700 1 --2- 172.21.15.16:0/1641456322 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faf2c003a30 0x7faf440c9af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:38.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.925+0000 7faf437fe700 1 --2- 172.21.15.16:0/1641456322 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faf440ccc60 0x7faf4414f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:38.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.925+0000 7faf427fc700 1 -- 172.21.15.16:0/1641456322 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faf440ccc60 msgr2=0x7faf4414f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:38.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.925+0000 7faf427fc700 1 --2- 172.21.15.16:0/1641456322 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faf440ccc60 0x7faf4414f330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:38.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.925+0000 7faf427fc700 1 -- 172.21.15.16:0/1641456322 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faf2c003a30 msgr2=0x7faf440c9af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:38.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.925+0000 7faf427fc700 1 --2- 172.21.15.16:0/1641456322 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faf2c003a30 0x7faf440c9af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:38.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.925+0000 7faf427fc700 1 -- 172.21.15.16:0/1641456322 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf440cafa0 con 0x7faf440ca700 2023-10-16T13:44:38.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.925+0000 7faf427fc700 1 --2- 172.21.15.16:0/1641456322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf440ca700 0x7faf440cbd60 secure :-1 s=READY pgs=2276 cs=0 l=1 rev1=1 crypto rx=0x7faf34007290 tx=0x7faf340075a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:38.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.926+0000 7faf2b7fe700 1 -- 172.21.15.16:0/1641456322 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf34011580 con 0x7faf440ca700 2023-10-16T13:44:38.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.926+0000 7faf2b7fe700 1 -- 172.21.15.16:0/1641456322 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf340116e0 con 0x7faf440ca700 2023-10-16T13:44:38.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.926+0000 7faf492b1700 1 -- 172.21.15.16:0/1641456322 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf440cb280 con 0x7faf440ca700 2023-10-16T13:44:38.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.926+0000 7faf2b7fe700 1 -- 172.21.15.16:0/1641456322 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf34011a90 con 0x7faf440ca700 2023-10-16T13:44:38.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.926+0000 7faf492b1700 1 -- 172.21.15.16:0/1641456322 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf440cc2a0 con 0x7faf440ca700 2023-10-16T13:44:38.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.927+0000 7faf492b1700 1 -- 172.21.15.16:0/1641456322 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf440cba80 con 0x7faf440ca700 2023-10-16T13:44:38.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.927+0000 7faf2b7fe700 1 -- 172.21.15.16:0/1641456322 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 126) v1 ==== 89331+0+0 (secure 0 0 0) 0x7faf34011bf0 con 0x7faf440ca700 2023-10-16T13:44:38.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.928+0000 7faf2b7fe700 1 --2- 172.21.15.16:0/1641456322 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7faf3006b050 0x7faf3006d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:38.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.928+0000 7faf2b7fe700 1 -- 172.21.15.16:0/1641456322 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(203..203 src has 1..203) v4 ==== 5675+0+0 (secure 0 0 0) 0x7faf340896e0 con 0x7faf440ca700 2023-10-16T13:44:38.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.928+0000 7faf42ffd700 1 --2- 172.21.15.16:0/1641456322 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7faf3006b050 0x7faf3006d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:38.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.928+0000 7faf42ffd700 1 --2- 172.21.15.16:0/1641456322 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7faf3006b050 0x7faf3006d510 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7faf380090b0 tx=0x7faf38008030 comp rx=0 tx=0).ready entity=mgr.10665 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:38.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:38.933+0000 7faf2b7fe700 1 -- 172.21.15.16:0/1641456322 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7faf440cba80 con 0x7faf440ca700 2023-10-16T13:44:39.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:39.136+0000 7faf492b1700 1 -- 172.21.15.16:0/1641456322 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7faf440cba40 con 0x7faf440ca700 2023-10-16T13:44:39.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:39.771+0000 7faf2b7fe700 1 -- 172.21.15.16:0/1641456322 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v204) v1 ==== 170+0+0 (secure 0 0 0) 0x7faf3405e030 con 0x7faf440ca700 2023-10-16T13:44:39.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:39.776+0000 7faf492b1700 1 -- 172.21.15.16:0/1641456322 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7faf3006b050 msgr2=0x7faf3006d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:39.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:39.776+0000 7faf492b1700 1 --2- 172.21.15.16:0/1641456322 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7faf3006b050 0x7faf3006d510 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7faf380090b0 tx=0x7faf38008030 comp rx=0 tx=0).stop 2023-10-16T13:44:39.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:39.776+0000 7faf492b1700 1 -- 172.21.15.16:0/1641456322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf440ca700 msgr2=0x7faf440cbd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:39.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:39.776+0000 7faf492b1700 1 --2- 172.21.15.16:0/1641456322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf440ca700 0x7faf440cbd60 secure :-1 s=READY pgs=2276 cs=0 l=1 rev1=1 crypto rx=0x7faf34007290 tx=0x7faf340075a0 comp rx=0 tx=0).stop 2023-10-16T13:44:39.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:39.777+0000 7faf492b1700 1 -- 172.21.15.16:0/1641456322 shutdown_connections 2023-10-16T13:44:39.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:39.777+0000 7faf492b1700 1 --2- 172.21.15.16:0/1641456322 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7faf3006b050 0x7faf3006d510 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:39.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:39.777+0000 7faf492b1700 1 --2- 172.21.15.16:0/1641456322 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faf440ccc60 0x7faf4414f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:39.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:39.777+0000 7faf492b1700 1 --2- 172.21.15.16:0/1641456322 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faf2c003a30 0x7faf440c9af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:39.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:39.777+0000 7faf492b1700 1 --2- 172.21.15.16:0/1641456322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf440ca700 0x7faf440cbd60 unknown :-1 s=CLOSED pgs=2276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:39.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:39.777+0000 7faf492b1700 1 -- 172.21.15.16:0/1641456322 >> 172.21.15.16:0/1641456322 conn(0x7faf44072330 msgr2=0x7faf440cff00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:39.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:39.778+0000 7faf492b1700 1 -- 172.21.15.16:0/1641456322 shutdown_connections 2023-10-16T13:44:39.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:39.780+0000 7faf492b1700 1 -- 172.21.15.16:0/1641456322 wait complete. 2023-10-16T13:44:39.795 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' created 2023-10-16T13:44:39.810 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-16T13:44:39.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:39.940+0000 7fc163453700 1 Processor -- start 2023-10-16T13:44:39.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:39.940+0000 7fc163453700 1 -- start start 2023-10-16T13:44:39.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:39.941+0000 7fc163453700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fc15c06aef0 0x7fc15c0ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:39.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:39.941+0000 7fc163453700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc15c06da30 0x7fc15c0adf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:39.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:39.941+0000 7fc163453700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc15c1613c0 con 0x7fc15c160fe0 2023-10-16T13:44:39.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:39.941+0000 7fc163453700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc15c0cf680 con 0x7fc15c06da30 2023-10-16T13:44:39.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:39.941+0000 7fc163453700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc15c0adb00 con 0x7fc15c06aef0 2023-10-16T13:44:39.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:39.941+0000 7fc1619f0700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc15c06da30 0x7fc15c0adf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:39.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:39.941+0000 7fc1619f0700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc15c06da30 0x7fc15c0adf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51650/0 (socket says 172.21.15.16:51650) 2023-10-16T13:44:39.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:39.941+0000 7fc1619f0700 1 -- 172.21.15.16:0/3265829121 learned_addr learned my addr 172.21.15.16:0/3265829121 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:39.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:39.941+0000 7fc1609ee700 1 --2- 172.21.15.16:0/3265829121 >> v2:172.21.15.49:3300/0 conn(0x7fc15c06aef0 0x7fc15c0ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:39.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:39.942+0000 7fc1619f0700 1 -- 172.21.15.16:0/3265829121 >> v2:172.21.15.49:3300/0 conn(0x7fc15c06aef0 msgr2=0x7fc15c0ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:39.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:39.942+0000 7fc1619f0700 1 --2- 172.21.15.16:0/3265829121 >> v2:172.21.15.49:3300/0 conn(0x7fc15c06aef0 0x7fc15c0ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:39.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:39.942+0000 7fc1619f0700 1 -- 172.21.15.16:0/3265829121 >> v1:172.21.15.49:6789/0 conn(0x7fc15c160fe0 legacy=0x7fc15c06a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:39.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:39.942+0000 7fc1619f0700 1 -- 172.21.15.16:0/3265829121 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc15c0acef0 con 0x7fc15c06da30 2023-10-16T13:44:39.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:39.942+0000 7fc1619f0700 1 --2- 172.21.15.16:0/3265829121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc15c06da30 0x7fc15c0adf90 secure :-1 s=READY pgs=2277 cs=0 l=1 rev1=1 crypto rx=0x7fc15c0cf330 tx=0x7fc15400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e1cf25a6464bafa server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:39.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:39.942+0000 7fc153fff700 1 -- 172.21.15.16:0/3265829121 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc15400e070 con 0x7fc15c06da30 2023-10-16T13:44:39.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:39.942+0000 7fc153fff700 1 -- 172.21.15.16:0/3265829121 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc15400b9c0 con 0x7fc15c06da30 2023-10-16T13:44:39.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:39.942+0000 7fc153fff700 1 -- 172.21.15.16:0/3265829121 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc15400bd70 con 0x7fc15c06da30 2023-10-16T13:44:39.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:39.942+0000 7fc163453700 1 -- 172.21.15.16:0/3265829121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc15c06da30 msgr2=0x7fc15c0adf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:39.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:39.942+0000 7fc163453700 1 --2- 172.21.15.16:0/3265829121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc15c06da30 0x7fc15c0adf90 secure :-1 s=READY pgs=2277 cs=0 l=1 rev1=1 crypto rx=0x7fc15c0cf330 tx=0x7fc15400b330 comp rx=0 tx=0).stop 2023-10-16T13:44:39.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:39.943+0000 7fc163453700 1 -- 172.21.15.16:0/3265829121 shutdown_connections 2023-10-16T13:44:39.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:39.943+0000 7fc163453700 1 --2- 172.21.15.16:0/3265829121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc15c06da30 0x7fc15c0adf90 unknown :-1 s=CLOSED pgs=2277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:39.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:39.943+0000 7fc163453700 1 --2- 172.21.15.16:0/3265829121 >> v2:172.21.15.49:3300/0 conn(0x7fc15c06aef0 0x7fc15c0ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:39.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:39.943+0000 7fc163453700 1 -- 172.21.15.16:0/3265829121 >> 172.21.15.16:0/3265829121 conn(0x7fc15c0b1ed0 msgr2=0x7fc15c0b22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:39.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:39.943+0000 7fc163453700 1 -- 172.21.15.16:0/3265829121 shutdown_connections 2023-10-16T13:44:39.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:39.943+0000 7fc163453700 1 -- 172.21.15.16:0/3265829121 wait complete. 2023-10-16T13:44:39.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:39.944+0000 7fc163453700 1 Processor -- start 2023-10-16T13:44:39.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:39.944+0000 7fc163453700 1 -- start start 2023-10-16T13:44:39.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:39.945+0000 7fc163453700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc15c06aef0 0x7fc15c1538a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:39.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:39.945+0000 7fc163453700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc15c06da30 0x7fc15c0b43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:39.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:39.945+0000 7fc163453700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc15c160fe0 0x7fc15c0b55e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:39.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:39.945+0000 7fc163453700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc15c153de0 con 0x7fc15c160fe0 2023-10-16T13:44:39.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:39.945+0000 7fc163453700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc15c0b3340 con 0x7fc15c06aef0 2023-10-16T13:44:39.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:39.945+0000 7fc163453700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc15c0b3620 con 0x7fc15c06da30 2023-10-16T13:44:39.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:39.945+0000 7fc1619f0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc15c160fe0 0x7fc15c0b55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:39.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:39.945+0000 7fc1619f0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc15c160fe0 0x7fc15c0b55e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51658/0 (socket says 172.21.15.16:51658) 2023-10-16T13:44:39.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:39.945+0000 7fc1619f0700 1 -- 172.21.15.16:0/2019302911 learned_addr learned my addr 172.21.15.16:0/2019302911 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:39.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:39.945+0000 7fc1611ef700 1 --2- 172.21.15.16:0/2019302911 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc15c06aef0 0x7fc15c1538a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:39.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:39.945+0000 7fc1609ee700 1 --2- 172.21.15.16:0/2019302911 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc15c06da30 0x7fc15c0b43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:39.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:39.945+0000 7fc1619f0700 1 -- 172.21.15.16:0/2019302911 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc15c06da30 msgr2=0x7fc15c0b43e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:39.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:39.945+0000 7fc1619f0700 1 --2- 172.21.15.16:0/2019302911 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc15c06da30 0x7fc15c0b43e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:39.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:39.945+0000 7fc1619f0700 1 -- 172.21.15.16:0/2019302911 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc15c06aef0 msgr2=0x7fc15c1538a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:39.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:39.945+0000 7fc1619f0700 1 --2- 172.21.15.16:0/2019302911 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc15c06aef0 0x7fc15c1538a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:39.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:39.945+0000 7fc1619f0700 1 -- 172.21.15.16:0/2019302911 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc154009b30 con 0x7fc15c160fe0 2023-10-16T13:44:39.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:39.945+0000 7fc1619f0700 1 --2- 172.21.15.16:0/2019302911 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc15c160fe0 0x7fc15c0b55e0 secure :-1 s=READY pgs=2278 cs=0 l=1 rev1=1 crypto rx=0x7fc15c072fd0 tx=0x7fc154004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:39.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:39.945+0000 7fc151ffb700 1 -- 172.21.15.16:0/2019302911 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc154004880 con 0x7fc15c160fe0 2023-10-16T13:44:39.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:39.946+0000 7fc151ffb700 1 -- 172.21.15.16:0/2019302911 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1540099b0 con 0x7fc15c160fe0 2023-10-16T13:44:39.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:39.946+0000 7fc163453700 1 -- 172.21.15.16:0/2019302911 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc15c0b3900 con 0x7fc15c160fe0 2023-10-16T13:44:39.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:39.946+0000 7fc151ffb700 1 -- 172.21.15.16:0/2019302911 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc15401c680 con 0x7fc15c160fe0 2023-10-16T13:44:39.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:39.946+0000 7fc163453700 1 -- 172.21.15.16:0/2019302911 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc15c0b3ec0 con 0x7fc15c160fe0 2023-10-16T13:44:39.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:39.947+0000 7fc163453700 1 -- 172.21.15.16:0/2019302911 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc15c04ebb0 con 0x7fc15c160fe0 2023-10-16T13:44:39.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:39.948+0000 7fc151ffb700 1 -- 172.21.15.16:0/2019302911 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 126) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc15401c7e0 con 0x7fc15c160fe0 2023-10-16T13:44:39.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:39.948+0000 7fc151ffb700 1 --2- 172.21.15.16:0/2019302911 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7fc14006b020 0x7fc14006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:39.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:39.948+0000 7fc151ffb700 1 -- 172.21.15.16:0/2019302911 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(204..204 src has 1..204) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fc15408b9b0 con 0x7fc15c160fe0 2023-10-16T13:44:39.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:39.948+0000 7fc1611ef700 1 --2- 172.21.15.16:0/2019302911 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7fc14006b020 0x7fc14006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:39.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:39.948+0000 7fc1611ef700 1 --2- 172.21.15.16:0/2019302911 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7fc14006b020 0x7fc14006d4e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc15c0c5380 tx=0x7fc14c009040 comp rx=0 tx=0).ready entity=mgr.10665 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:39.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:39.952+0000 7fc151ffb700 1 -- 172.21.15.16:0/2019302911 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc154056850 con 0x7fc15c160fe0 2023-10-16T13:44:40.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:40.205+0000 7fc163453700 1 -- 172.21.15.16:0/2019302911 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fc15c0b4080 con 0x7fc15c160fe0 2023-10-16T13:44:40.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:40.820+0000 7fc151ffb700 1 -- 172.21.15.16:0/2019302911 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 25 and data pool 26 v100) v1 ==== 174+0+0 (secure 0 0 0) 0x7fc154059e70 con 0x7fc15c160fe0 2023-10-16T13:44:40.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:40.828+0000 7fc163453700 1 -- 172.21.15.16:0/2019302911 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7fc14006b020 msgr2=0x7fc14006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:40.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:40.828+0000 7fc163453700 1 --2- 172.21.15.16:0/2019302911 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7fc14006b020 0x7fc14006d4e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc15c0c5380 tx=0x7fc14c009040 comp rx=0 tx=0).stop 2023-10-16T13:44:40.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:40.828+0000 7fc163453700 1 -- 172.21.15.16:0/2019302911 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc15c160fe0 msgr2=0x7fc15c0b55e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:40.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:40.828+0000 7fc163453700 1 --2- 172.21.15.16:0/2019302911 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc15c160fe0 0x7fc15c0b55e0 secure :-1 s=READY pgs=2278 cs=0 l=1 rev1=1 crypto rx=0x7fc15c072fd0 tx=0x7fc154004ad0 comp rx=0 tx=0).stop 2023-10-16T13:44:40.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:40.828+0000 7fc163453700 1 -- 172.21.15.16:0/2019302911 shutdown_connections 2023-10-16T13:44:40.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:40.828+0000 7fc163453700 1 --2- 172.21.15.16:0/2019302911 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7fc14006b020 0x7fc14006d4e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:40.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:40.828+0000 7fc163453700 1 --2- 172.21.15.16:0/2019302911 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc15c160fe0 0x7fc15c0b55e0 unknown :-1 s=CLOSED pgs=2278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:40.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:40.828+0000 7fc163453700 1 --2- 172.21.15.16:0/2019302911 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc15c06aef0 0x7fc15c1538a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:40.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:40.828+0000 7fc163453700 1 --2- 172.21.15.16:0/2019302911 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc15c06da30 0x7fc15c0b43e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:40.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:40.828+0000 7fc163453700 1 -- 172.21.15.16:0/2019302911 >> 172.21.15.16:0/2019302911 conn(0x7fc15c0b1ed0 msgr2=0x7fc15c15c930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:40.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:40.829+0000 7fc163453700 1 -- 172.21.15.16:0/2019302911 shutdown_connections 2023-10-16T13:44:40.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:40.829+0000 7fc163453700 1 -- 172.21.15.16:0/2019302911 wait complete. 2023-10-16T13:44:40.836 INFO:teuthology.orchestra.run.smithi016.stderr:new fs with metadata pool 25 and data pool 26 2023-10-16T13:44:40.852 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:44:40.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:40.981+0000 7faa2a4c0700 1 Processor -- start 2023-10-16T13:44:40.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:40.981+0000 7faa2a4c0700 1 -- start start 2023-10-16T13:44:40.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:40.982+0000 7faa2a4c0700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7faa240ce4f0 0x7faa240d0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:40.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:40.982+0000 7faa2a4c0700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa240b0970 con 0x7faa2415b630 2023-10-16T13:44:40.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:40.982+0000 7faa2a4c0700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa240cd690 con 0x7faa2415ba10 2023-10-16T13:44:40.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:40.982+0000 7faa2a4c0700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa240cd970 con 0x7faa240ce4f0 2023-10-16T13:44:40.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:40.982+0000 7faa237fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7faa240ce4f0 0x7faa240d0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:40.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:40.982+0000 7faa237fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7faa240ce4f0 0x7faa240d0380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51674/0 (socket says 172.21.15.16:51674) 2023-10-16T13:44:40.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:40.982+0000 7faa23fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7faa2415b630 0x7faa240cdde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:40946/0 (socket says 172.21.15.16:40946) 2023-10-16T13:44:40.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:40.982+0000 7faa237fe700 1 -- 172.21.15.16:0/3981924438 learned_addr learned my addr 172.21.15.16:0/3981924438 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:40.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:40.982+0000 7faa237fe700 1 -- 172.21.15.16:0/3981924438 >> v1:172.21.15.49:6789/0 conn(0x7faa2415ba10 legacy=0x7faa240b1430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:40.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:40.982+0000 7faa237fe700 1 -- 172.21.15.16:0/3981924438 >> v1:172.21.15.16:6789/0 conn(0x7faa2415b630 legacy=0x7faa240cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:40.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:40.982+0000 7faa22ffd700 1 -- 172.21.15.16:0/3981924438 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2840426469 0 0) 0x7faa240b0970 con 0x7faa2415b630 2023-10-16T13:44:40.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:40.982+0000 7faa237fe700 1 -- 172.21.15.16:0/3981924438 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa240b0c30 con 0x7faa240ce4f0 2023-10-16T13:44:40.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:40.982+0000 7faa237fe700 1 --2- 172.21.15.16:0/3981924438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa240ce4f0 0x7faa240d0380 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 crypto rx=0x7faa140098f0 tx=0x7faa14009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f3b0c98be8450ea server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:40.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:40.983+0000 7faa22ffd700 1 -- 172.21.15.16:0/3981924438 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa14004230 con 0x7faa240ce4f0 2023-10-16T13:44:40.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:40.983+0000 7faa22ffd700 1 -- 172.21.15.16:0/3981924438 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa14004390 con 0x7faa240ce4f0 2023-10-16T13:44:40.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:40.983+0000 7faa22ffd700 1 -- 172.21.15.16:0/3981924438 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa140036a0 con 0x7faa240ce4f0 2023-10-16T13:44:40.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:40.983+0000 7faa2a4c0700 1 -- 172.21.15.16:0/3981924438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa240ce4f0 msgr2=0x7faa240d0380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:40.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:40.983+0000 7faa2a4c0700 1 --2- 172.21.15.16:0/3981924438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa240ce4f0 0x7faa240d0380 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 crypto rx=0x7faa140098f0 tx=0x7faa14009c00 comp rx=0 tx=0).stop 2023-10-16T13:44:40.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:40.983+0000 7faa2a4c0700 1 -- 172.21.15.16:0/3981924438 shutdown_connections 2023-10-16T13:44:40.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:40.983+0000 7faa2a4c0700 1 --2- 172.21.15.16:0/3981924438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa240ce4f0 0x7faa240d0380 unknown :-1 s=CLOSED pgs=2280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:40.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:40.983+0000 7faa2a4c0700 1 -- 172.21.15.16:0/3981924438 >> 172.21.15.16:0/3981924438 conn(0x7faa240d14b0 msgr2=0x7faa240d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:40.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:40.983+0000 7faa2a4c0700 1 -- 172.21.15.16:0/3981924438 shutdown_connections 2023-10-16T13:44:40.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:40.983+0000 7faa2a4c0700 1 -- 172.21.15.16:0/3981924438 wait complete. 2023-10-16T13:44:40.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:40.984+0000 7faa2a4c0700 1 Processor -- start 2023-10-16T13:44:40.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:40.985+0000 7faa2a4c0700 1 -- start start 2023-10-16T13:44:40.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:40.985+0000 7faa2a4c0700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faa240ce4f0 0x7faa240b6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:40.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:40.985+0000 7faa2a4c0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa2415b630 0x7faa240c4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:40.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:40.985+0000 7faa2a4c0700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faa2415ba10 0x7faa240c5520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:40.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:40.985+0000 7faa2a4c0700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa240b6200 con 0x7faa2415b630 2023-10-16T13:44:40.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:40.985+0000 7faa2a4c0700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa240b6760 con 0x7faa240ce4f0 2023-10-16T13:44:40.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:40.985+0000 7faa2a4c0700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa240c5df0 con 0x7faa2415ba10 2023-10-16T13:44:40.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:40.985+0000 7faa237fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa2415b630 0x7faa240c4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:40.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:40.986+0000 7faa237fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa2415b630 0x7faa240c4fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51680/0 (socket says 172.21.15.16:51680) 2023-10-16T13:44:40.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:40.986+0000 7faa237fe700 1 -- 172.21.15.16:0/1807282425 learned_addr learned my addr 172.21.15.16:0/1807282425 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:40.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:40.986+0000 7faa23fff700 1 --2- 172.21.15.16:0/1807282425 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faa240ce4f0 0x7faa240b6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:40.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:40.986+0000 7faa237fe700 1 -- 172.21.15.16:0/1807282425 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faa2415ba10 msgr2=0x7faa240c5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:40.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:40.986+0000 7faa28a5d700 1 --2- 172.21.15.16:0/1807282425 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faa2415ba10 0x7faa240c5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:40.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:40.986+0000 7faa237fe700 1 --2- 172.21.15.16:0/1807282425 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faa2415ba10 0x7faa240c5520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:40.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:40.986+0000 7faa237fe700 1 -- 172.21.15.16:0/1807282425 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faa240ce4f0 msgr2=0x7faa240b6bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:40.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:40.986+0000 7faa237fe700 1 --2- 172.21.15.16:0/1807282425 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faa240ce4f0 0x7faa240b6bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:40.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:40.986+0000 7faa237fe700 1 -- 172.21.15.16:0/1807282425 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa14009550 con 0x7faa2415b630 2023-10-16T13:44:40.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:40.986+0000 7faa23fff700 1 --2- 172.21.15.16:0/1807282425 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faa240ce4f0 0x7faa240b6bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:44:40.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:40.986+0000 7faa237fe700 1 --2- 172.21.15.16:0/1807282425 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa2415b630 0x7faa240c4fe0 secure :-1 s=READY pgs=2281 cs=0 l=1 rev1=1 crypto rx=0x7faa24072680 tx=0x7faa14004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:40.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:40.986+0000 7faa20ff9700 1 -- 172.21.15.16:0/1807282425 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa14003dc0 con 0x7faa2415b630 2023-10-16T13:44:40.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:40.986+0000 7faa20ff9700 1 -- 172.21.15.16:0/1807282425 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa1401f070 con 0x7faa2415b630 2023-10-16T13:44:40.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:40.986+0000 7faa2a4c0700 1 -- 172.21.15.16:0/1807282425 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa240c60d0 con 0x7faa2415b630 2023-10-16T13:44:40.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:40.986+0000 7faa20ff9700 1 -- 172.21.15.16:0/1807282425 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa1400b8a0 con 0x7faa2415b630 2023-10-16T13:44:40.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:40.986+0000 7faa2a4c0700 1 -- 172.21.15.16:0/1807282425 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa24150c20 con 0x7faa2415b630 2023-10-16T13:44:40.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:40.987+0000 7faa2a4c0700 1 -- 172.21.15.16:0/1807282425 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa240c6310 con 0x7faa2415b630 2023-10-16T13:44:41.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:40.988+0000 7faa20ff9700 1 -- 172.21.15.16:0/1807282425 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 126) v1 ==== 89331+0+0 (secure 0 0 0) 0x7faa14021030 con 0x7faa2415b630 2023-10-16T13:44:41.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:40.988+0000 7faa20ff9700 1 --2- 172.21.15.16:0/1807282425 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7faa1006b020 0x7faa1006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:41.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:40.988+0000 7faa20ff9700 1 -- 172.21.15.16:0/1807282425 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 6157+0+0 (secure 0 0 0) 0x7faa1408b090 con 0x7faa2415b630 2023-10-16T13:44:41.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:40.988+0000 7faa23fff700 1 --2- 172.21.15.16:0/1807282425 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7faa1006b020 0x7faa1006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:41.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:40.989+0000 7faa23fff700 1 --2- 172.21.15.16:0/1807282425 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7faa1006b020 0x7faa1006d4e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7faa18000f60 tx=0x7faa18009040 comp rx=0 tx=0).ready entity=mgr.10665 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:41.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:40.993+0000 7faa20ff9700 1 -- 172.21.15.16:0/1807282425 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7faa240c6310 con 0x7faa2415b630 2023-10-16T13:44:41.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.197+0000 7faa2a4c0700 1 -- 172.21.15.16:0/1807282425 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faa240b68d0 con 0x7faa2415b630 2023-10-16T13:44:41.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.198+0000 7faa20ff9700 1 -- 172.21.15.16:0/1807282425 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v205) v1 ==== 74+0+16830 (secure 0 0 0) 0x7faa14055e70 con 0x7faa2415b630 2023-10-16T13:44:41.202 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:44:41.202 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":205,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:44:40.207870+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2023-10-16T13:44:38.716950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"205","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_data","create_time":"2023-10-16T13:44:39.172499+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"205","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":204,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":204,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":204,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":204,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":204,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":204,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":204,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":204,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:44:41.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.200+0000 7faa2a4c0700 1 -- 172.21.15.16:0/1807282425 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7faa1006b020 msgr2=0x7faa1006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:41.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.200+0000 7faa2a4c0700 1 --2- 172.21.15.16:0/1807282425 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7faa1006b020 0x7faa1006d4e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7faa18000f60 tx=0x7faa18009040 comp rx=0 tx=0).stop 2023-10-16T13:44:41.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.200+0000 7faa2a4c0700 1 -- 172.21.15.16:0/1807282425 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa2415b630 msgr2=0x7faa240c4fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:41.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.200+0000 7faa2a4c0700 1 --2- 172.21.15.16:0/1807282425 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa2415b630 0x7faa240c4fe0 secure :-1 s=READY pgs=2281 cs=0 l=1 rev1=1 crypto rx=0x7faa24072680 tx=0x7faa14004750 comp rx=0 tx=0).stop 2023-10-16T13:44:41.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.200+0000 7faa2a4c0700 1 -- 172.21.15.16:0/1807282425 shutdown_connections 2023-10-16T13:44:41.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.200+0000 7faa2a4c0700 1 --2- 172.21.15.16:0/1807282425 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7faa1006b020 0x7faa1006d4e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:41.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.200+0000 7faa2a4c0700 1 --2- 172.21.15.16:0/1807282425 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faa2415ba10 0x7faa240c5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:41.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.200+0000 7faa2a4c0700 1 --2- 172.21.15.16:0/1807282425 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faa240ce4f0 0x7faa240b6bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:41.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.200+0000 7faa2a4c0700 1 --2- 172.21.15.16:0/1807282425 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa2415b630 0x7faa240c4fe0 unknown :-1 s=CLOSED pgs=2281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:41.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.200+0000 7faa2a4c0700 1 -- 172.21.15.16:0/1807282425 >> 172.21.15.16:0/1807282425 conn(0x7faa240d14b0 msgr2=0x7faa240b1430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:41.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.201+0000 7faa2a4c0700 1 -- 172.21.15.16:0/1807282425 shutdown_connections 2023-10-16T13:44:41.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.201+0000 7faa2a4c0700 1 -- 172.21.15.16:0/1807282425 wait complete. 2023-10-16T13:44:41.217 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:44:41.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.338+0000 7f965207f700 1 Processor -- start 2023-10-16T13:44:41.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.338+0000 7f965207f700 1 -- start start 2023-10-16T13:44:41.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.339+0000 7f965207f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f964c0cc8a0 0x7f964c15e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:41.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.339+0000 7f965207f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f964c06a7e0 0x7f964c06ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:41.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.339+0000 7f965207f700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f964c06b170 con 0x7f964c15b280 2023-10-16T13:44:41.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.339+0000 7f965207f700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f964c161c30 con 0x7f964c06a7e0 2023-10-16T13:44:41.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.339+0000 7f965207f700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f964c161f10 con 0x7f964c0cc8a0 2023-10-16T13:44:41.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.339+0000 7f964bfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f964c06a7e0 0x7f964c06ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:41.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.339+0000 7f964bfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f964c06a7e0 0x7f964c06ac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51690/0 (socket says 172.21.15.16:51690) 2023-10-16T13:44:41.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.339+0000 7f964bfff700 1 -- 172.21.15.16:0/2178501512 learned_addr learned my addr 172.21.15.16:0/2178501512 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:41.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.340+0000 7f964affd700 1 --2- 172.21.15.16:0/2178501512 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f964c0cc8a0 0x7f964c15e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:41.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.340+0000 7f964bfff700 1 -- 172.21.15.16:0/2178501512 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f964c0cc8a0 msgr2=0x7f964c15e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:41.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.340+0000 7f964bfff700 1 --2- 172.21.15.16:0/2178501512 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f964c0cc8a0 0x7f964c15e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:41.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.340+0000 7f964bfff700 1 -- 172.21.15.16:0/2178501512 >> v1:172.21.15.49:6789/0 conn(0x7f964c15b280 legacy=0x7f964c1613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:41.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.340+0000 7f964bfff700 1 -- 172.21.15.16:0/2178501512 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f964c1621f0 con 0x7f964c06a7e0 2023-10-16T13:44:41.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.340+0000 7f964bfff700 1 --2- 172.21.15.16:0/2178501512 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f964c06a7e0 0x7f964c06ac30 secure :-1 s=READY pgs=2282 cs=0 l=1 rev1=1 crypto rx=0x7f9644009e50 tx=0x7f964400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=562aae9e3fb2f44d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:41.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.340+0000 7f964a7fc700 1 -- 172.21.15.16:0/2178501512 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f964400e070 con 0x7f964c06a7e0 2023-10-16T13:44:41.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.340+0000 7f964a7fc700 1 -- 172.21.15.16:0/2178501512 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f964400b9c0 con 0x7f964c06a7e0 2023-10-16T13:44:41.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.340+0000 7f964a7fc700 1 -- 172.21.15.16:0/2178501512 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f964400bd70 con 0x7f964c06a7e0 2023-10-16T13:44:41.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.340+0000 7f965207f700 1 -- 172.21.15.16:0/2178501512 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f964c06a7e0 msgr2=0x7f964c06ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:41.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.340+0000 7f965207f700 1 --2- 172.21.15.16:0/2178501512 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f964c06a7e0 0x7f964c06ac30 secure :-1 s=READY pgs=2282 cs=0 l=1 rev1=1 crypto rx=0x7f9644009e50 tx=0x7f964400b330 comp rx=0 tx=0).stop 2023-10-16T13:44:41.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.340+0000 7f965207f700 1 -- 172.21.15.16:0/2178501512 shutdown_connections 2023-10-16T13:44:41.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.340+0000 7f965207f700 1 --2- 172.21.15.16:0/2178501512 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f964c06a7e0 0x7f964c06ac30 unknown :-1 s=CLOSED pgs=2282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:41.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.340+0000 7f965207f700 1 --2- 172.21.15.16:0/2178501512 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f964c0cc8a0 0x7f964c15e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:41.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.341+0000 7f965207f700 1 -- 172.21.15.16:0/2178501512 >> 172.21.15.16:0/2178501512 conn(0x7f964c0b1fd0 msgr2=0x7f964c0b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:41.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.341+0000 7f965207f700 1 -- 172.21.15.16:0/2178501512 shutdown_connections 2023-10-16T13:44:41.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.341+0000 7f965207f700 1 -- 172.21.15.16:0/2178501512 wait complete. 2023-10-16T13:44:41.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.342+0000 7f965207f700 1 Processor -- start 2023-10-16T13:44:41.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.342+0000 7f965207f700 1 -- start start 2023-10-16T13:44:41.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.343+0000 7f965207f700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f964c06a7e0 0x7f964c0c4d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:41.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.343+0000 7f965207f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f964c0cc8a0 0x7f964c0c64f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:41.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.343+0000 7f965207f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f964c15b280 0x7f964c1f2a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:41.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.343+0000 7f965207f700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f964c0c6e20 con 0x7f964c15b280 2023-10-16T13:44:41.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.343+0000 7f965207f700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f964c158160 con 0x7f964c06a7e0 2023-10-16T13:44:41.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.343+0000 7f965207f700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f964c158440 con 0x7f964c0cc8a0 2023-10-16T13:44:41.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.343+0000 7f964bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f964c15b280 0x7f964c1f2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:41.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.343+0000 7f964bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f964c15b280 0x7f964c1f2a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51696/0 (socket says 172.21.15.16:51696) 2023-10-16T13:44:41.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.343+0000 7f964bfff700 1 -- 172.21.15.16:0/2885133797 learned_addr learned my addr 172.21.15.16:0/2885133797 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:41.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.343+0000 7f964b7fe700 1 --2- 172.21.15.16:0/2885133797 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f964c06a7e0 0x7f964c0c4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:41.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.343+0000 7f964affd700 1 --2- 172.21.15.16:0/2885133797 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f964c0cc8a0 0x7f964c0c64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:41.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.343+0000 7f964bfff700 1 -- 172.21.15.16:0/2885133797 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f964c0cc8a0 msgr2=0x7f964c0c64f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:41.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.343+0000 7f964bfff700 1 --2- 172.21.15.16:0/2885133797 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f964c0cc8a0 0x7f964c0c64f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:41.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.343+0000 7f964bfff700 1 -- 172.21.15.16:0/2885133797 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f964c06a7e0 msgr2=0x7f964c0c4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:41.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.343+0000 7f964bfff700 1 --2- 172.21.15.16:0/2885133797 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f964c06a7e0 0x7f964c0c4d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:41.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.343+0000 7f964bfff700 1 -- 172.21.15.16:0/2885133797 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9644009b30 con 0x7f964c15b280 2023-10-16T13:44:41.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.343+0000 7f964bfff700 1 --2- 172.21.15.16:0/2885133797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f964c15b280 0x7f964c1f2a60 secure :-1 s=READY pgs=2283 cs=0 l=1 rev1=1 crypto rx=0x7f964c0d1540 tx=0x7f9644004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:41.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.343+0000 7f9643fff700 1 -- 172.21.15.16:0/2885133797 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9644004800 con 0x7f964c15b280 2023-10-16T13:44:41.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.343+0000 7f9643fff700 1 -- 172.21.15.16:0/2885133797 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9644004960 con 0x7f964c15b280 2023-10-16T13:44:41.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.343+0000 7f965207f700 1 -- 172.21.15.16:0/2885133797 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f964c158720 con 0x7f964c15b280 2023-10-16T13:44:41.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.343+0000 7f9643fff700 1 -- 172.21.15.16:0/2885133797 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f964401b670 con 0x7f964c15b280 2023-10-16T13:44:41.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.344+0000 7f965207f700 1 -- 172.21.15.16:0/2885133797 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f964c158f20 con 0x7f964c15b280 2023-10-16T13:44:41.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.345+0000 7f965207f700 1 -- 172.21.15.16:0/2885133797 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f964c04ebb0 con 0x7f964c15b280 2023-10-16T13:44:41.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.345+0000 7f9643fff700 1 -- 172.21.15.16:0/2885133797 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 126) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f964401b7d0 con 0x7f964c15b280 2023-10-16T13:44:41.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.345+0000 7f9643fff700 1 --2- 172.21.15.16:0/2885133797 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f962c06b020 0x7f962c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:41.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.345+0000 7f9643fff700 1 -- 172.21.15.16:0/2885133797 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f964408b2b0 con 0x7f964c15b280 2023-10-16T13:44:41.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.345+0000 7f964b7fe700 1 --2- 172.21.15.16:0/2885133797 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f962c06b020 0x7f962c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:41.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.346+0000 7f964b7fe700 1 --2- 172.21.15.16:0/2885133797 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f962c06b020 0x7f962c06d4e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9634007fd0 tx=0x7f9634009040 comp rx=0 tx=0).ready entity=mgr.10665 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:41.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.350+0000 7f9643fff700 1 -- 172.21.15.16:0/2885133797 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9644055f70 con 0x7f964c15b280 2023-10-16T13:44:41.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.556+0000 7f965207f700 1 -- 172.21.15.16:0/2885133797 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f964c158be0 con 0x7f964c15b280 2023-10-16T13:44:41.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.557+0000 7f9643fff700 1 -- 172.21.15.16:0/2885133797 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v205) v1 ==== 74+0+16830 (secure 0 0 0) 0x7f9644059590 con 0x7f964c15b280 2023-10-16T13:44:41.561 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:44:41.561 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":205,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:44:40.207870+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2023-10-16T13:44:38.716950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"205","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_data","create_time":"2023-10-16T13:44:39.172499+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"205","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":204,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":204,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":204,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":204,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":204,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":204,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":204,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":204,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:44:41.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.559+0000 7f965207f700 1 -- 172.21.15.16:0/2885133797 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f962c06b020 msgr2=0x7f962c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:41.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.559+0000 7f965207f700 1 --2- 172.21.15.16:0/2885133797 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f962c06b020 0x7f962c06d4e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9634007fd0 tx=0x7f9634009040 comp rx=0 tx=0).stop 2023-10-16T13:44:41.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.559+0000 7f965207f700 1 -- 172.21.15.16:0/2885133797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f964c15b280 msgr2=0x7f964c1f2a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:41.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.559+0000 7f965207f700 1 --2- 172.21.15.16:0/2885133797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f964c15b280 0x7f964c1f2a60 secure :-1 s=READY pgs=2283 cs=0 l=1 rev1=1 crypto rx=0x7f964c0d1540 tx=0x7f9644004ad0 comp rx=0 tx=0).stop 2023-10-16T13:44:41.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.559+0000 7f965207f700 1 -- 172.21.15.16:0/2885133797 shutdown_connections 2023-10-16T13:44:41.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.559+0000 7f965207f700 1 --2- 172.21.15.16:0/2885133797 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f962c06b020 0x7f962c06d4e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:41.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.559+0000 7f965207f700 1 --2- 172.21.15.16:0/2885133797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f964c15b280 0x7f964c1f2a60 unknown :-1 s=CLOSED pgs=2283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:41.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.559+0000 7f965207f700 1 --2- 172.21.15.16:0/2885133797 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f964c06a7e0 0x7f964c0c4d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:41.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.559+0000 7f965207f700 1 --2- 172.21.15.16:0/2885133797 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f964c0cc8a0 0x7f964c0c64f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:41.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.559+0000 7f965207f700 1 -- 172.21.15.16:0/2885133797 >> 172.21.15.16:0/2885133797 conn(0x7f964c0b1fd0 msgr2=0x7f964c0d0c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:41.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.559+0000 7f965207f700 1 -- 172.21.15.16:0/2885133797 shutdown_connections 2023-10-16T13:44:41.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.560+0000 7f965207f700 1 -- 172.21.15.16:0/2885133797 wait complete. 2023-10-16T13:44:41.576 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-16T13:44:41.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.701+0000 7fedf6780700 1 Processor -- start 2023-10-16T13:44:41.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.701+0000 7fedf6780700 1 -- start start 2023-10-16T13:44:41.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.702+0000 7fedf6780700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fedf00cf4c0 0x7fedf00ca1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:41.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.702+0000 7fedf6780700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedf015dda0 con 0x7fedf0159de0 2023-10-16T13:44:41.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.702+0000 7fedf6780700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedf015e080 con 0x7fedf006bad0 2023-10-16T13:44:41.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.702+0000 7fedf6780700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedf015ea20 con 0x7fedf00cf4c0 2023-10-16T13:44:41.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.702+0000 7fedeffff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fedf0159de0 0x7fedf015a1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:40974/0 (socket says 172.21.15.16:40974) 2023-10-16T13:44:41.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.702+0000 7fedeffff700 1 -- 172.21.15.16:0/332112911 learned_addr learned my addr 172.21.15.16:0/332112911 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:44:41.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.702+0000 7fedef7fe700 1 --2- 172.21.15.16:0/332112911 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fedf00cf4c0 0x7fedf00ca1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:41.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.702+0000 7fedeeffd700 1 -- 172.21.15.16:0/332112911 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3992385207 0 0) 0x7fedf015dda0 con 0x7fedf0159de0 2023-10-16T13:44:41.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.702+0000 7fedeeffd700 1 -- 172.21.15.16:0/332112911 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feddc0034d0 con 0x7fedf0159de0 2023-10-16T13:44:41.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.702+0000 7fedeeffd700 1 -- 172.21.15.16:0/332112911 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1508229414 0 0) 0x7feddc0034d0 con 0x7fedf0159de0 2023-10-16T13:44:41.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.702+0000 7fedeeffd700 1 -- 172.21.15.16:0/332112911 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fedf00cf4c0 msgr2=0x7fedf00ca1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:41.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.702+0000 7fedeeffd700 1 --2- 172.21.15.16:0/332112911 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fedf00cf4c0 0x7fedf00ca1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:41.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.702+0000 7fedeeffd700 1 -- 172.21.15.16:0/332112911 >> v1:172.21.15.49:6789/0 conn(0x7fedf006bad0 legacy=0x7fedf00ca6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:41.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.703+0000 7fedeeffd700 1 -- 172.21.15.16:0/332112911 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedf015e360 con 0x7fedf0159de0 2023-10-16T13:44:41.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.703+0000 7fedeeffd700 1 -- 172.21.15.16:0/332112911 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7fede0001670 con 0x7fedf0159de0 2023-10-16T13:44:41.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.703+0000 7fedeeffd700 1 -- 172.21.15.16:0/332112911 >> v1:172.21.15.16:6789/0 conn(0x7fedf0159de0 legacy=0x7fedf015a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:41.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.703+0000 7fedeeffd700 1 --2- 172.21.15.16:0/332112911 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fedf00cf4c0 0x7feddc003b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:41.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.703+0000 7fedeeffd700 1 --2- 172.21.15.16:0/332112911 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feddc004060 0x7feddc00a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:41.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.703+0000 7fedeeffd700 1 --2- 172.21.15.16:0/332112911 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feddc00afc0 0x7feddc00d430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:41.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.703+0000 7fedeeffd700 1 -- 172.21.15.16:0/332112911 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedf015ea20 con 0x7feddc004060 2023-10-16T13:44:41.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.703+0000 7fedeeffd700 1 -- 172.21.15.16:0/332112911 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feddc00dd10 con 0x7feddc00afc0 2023-10-16T13:44:41.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.703+0000 7fedeeffd700 1 -- 172.21.15.16:0/332112911 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feddc00ded0 con 0x7fedf00cf4c0 2023-10-16T13:44:41.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.703+0000 7fedef7fe700 1 --2- 172.21.15.16:0/332112911 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feddc004060 0x7feddc00a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:41.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.703+0000 7fedeffff700 1 --2- 172.21.15.16:0/332112911 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fedf00cf4c0 0x7feddc003b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:41.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.703+0000 7fedf4d1d700 1 --2- 172.21.15.16:0/332112911 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feddc00afc0 0x7feddc00d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:41.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.703+0000 7fedef7fe700 1 -- 172.21.15.16:0/332112911 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fedf00cf4c0 msgr2=0x7feddc003b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:41.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.703+0000 7fedef7fe700 1 --2- 172.21.15.16:0/332112911 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fedf00cf4c0 0x7feddc003b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:41.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.703+0000 7fedef7fe700 1 -- 172.21.15.16:0/332112911 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feddc00afc0 msgr2=0x7feddc00d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:41.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.703+0000 7fedef7fe700 1 --2- 172.21.15.16:0/332112911 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feddc00afc0 0x7feddc00d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:41.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.703+0000 7fedef7fe700 1 -- 172.21.15.16:0/332112911 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feddc0034d0 con 0x7feddc004060 2023-10-16T13:44:41.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.703+0000 7fedf4d1d700 1 --2- 172.21.15.16:0/332112911 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feddc00afc0 0x7feddc00d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:44:41.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.703+0000 7fedef7fe700 1 --2- 172.21.15.16:0/332112911 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feddc004060 0x7feddc00a8f0 secure :-1 s=READY pgs=2285 cs=0 l=1 rev1=1 crypto rx=0x7fedd8009b60 tx=0x7feddc003780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d4559d19302dccb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:41.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.703+0000 7fedeeffd700 1 -- 172.21.15.16:0/332112911 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd800e6c0 con 0x7feddc004060 2023-10-16T13:44:41.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.703+0000 7fedeeffd700 1 -- 172.21.15.16:0/332112911 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedd800e820 con 0x7feddc004060 2023-10-16T13:44:41.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.703+0000 7fedeeffd700 1 -- 172.21.15.16:0/332112911 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd800eb90 con 0x7feddc004060 2023-10-16T13:44:41.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.704+0000 7fedf6780700 1 -- 172.21.15.16:0/332112911 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feddc004060 msgr2=0x7feddc00a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:41.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.704+0000 7fedf6780700 1 --2- 172.21.15.16:0/332112911 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feddc004060 0x7feddc00a8f0 secure :-1 s=READY pgs=2285 cs=0 l=1 rev1=1 crypto rx=0x7fedd8009b60 tx=0x7feddc003780 comp rx=0 tx=0).stop 2023-10-16T13:44:41.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.704+0000 7fedf4d1d700 1 -- 172.21.15.16:0/332112911 reap_dead start 2023-10-16T13:44:41.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.704+0000 7fedf6780700 1 -- 172.21.15.16:0/332112911 shutdown_connections 2023-10-16T13:44:41.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.704+0000 7fedf6780700 1 -- 172.21.15.16:0/332112911 >> 172.21.15.16:0/332112911 conn(0x7fedf00cc5e0 msgr2=0x7fedf00cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:41.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.704+0000 7fedf6780700 1 -- 172.21.15.16:0/332112911 shutdown_connections 2023-10-16T13:44:41.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.704+0000 7fedf6780700 1 -- 172.21.15.16:0/332112911 wait complete. 2023-10-16T13:44:41.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.705+0000 7fedf6780700 1 Processor -- start 2023-10-16T13:44:41.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.705+0000 7fedf6780700 1 -- start start 2023-10-16T13:44:41.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.706+0000 7fedf6780700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feddc004060 0x7fedf01557a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:41.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.706+0000 7fedf6780700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fedf015c6d0 0x7fedf00cb170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:41.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.706+0000 7fedf6780700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fedf00cb6b0 0x7fedf01537a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:41.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.706+0000 7fedf6780700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedf00cbb40 con 0x7feddc004060 2023-10-16T13:44:41.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.706+0000 7fedf6780700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedf015cf20 con 0x7fedf015c6d0 2023-10-16T13:44:41.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.706+0000 7fedf6780700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedf0155ce0 con 0x7fedf00cb6b0 2023-10-16T13:44:41.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.706+0000 7fedeffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feddc004060 0x7fedf01557a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:41.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.706+0000 7fedeffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feddc004060 0x7fedf01557a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51712/0 (socket says 172.21.15.16:51712) 2023-10-16T13:44:41.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.706+0000 7fedeffff700 1 -- 172.21.15.16:0/3313161154 learned_addr learned my addr 172.21.15.16:0/3313161154 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:41.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.706+0000 7fedeffff700 1 -- 172.21.15.16:0/3313161154 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fedf00cb6b0 msgr2=0x7fedf01537a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:41.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.706+0000 7fedeffff700 1 --2- 172.21.15.16:0/3313161154 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fedf00cb6b0 0x7fedf01537a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:41.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.706+0000 7fedeffff700 1 -- 172.21.15.16:0/3313161154 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fedf015c6d0 msgr2=0x7fedf00cb170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:41.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.706+0000 7fedef7fe700 1 --2- 172.21.15.16:0/3313161154 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fedf015c6d0 0x7fedf00cb170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:41.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.706+0000 7fedeffff700 1 --2- 172.21.15.16:0/3313161154 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fedf015c6d0 0x7fedf00cb170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:41.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.706+0000 7fedeffff700 1 -- 172.21.15.16:0/3313161154 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedf0154a20 con 0x7feddc004060 2023-10-16T13:44:41.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.707+0000 7fedeffff700 1 --2- 172.21.15.16:0/3313161154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feddc004060 0x7fedf01557a0 secure :-1 s=READY pgs=2286 cs=0 l=1 rev1=1 crypto rx=0x7fede00072c0 tx=0x7fede0007680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:41.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.707+0000 7fedecff9700 1 -- 172.21.15.16:0/3313161154 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fede00077a0 con 0x7feddc004060 2023-10-16T13:44:41.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.707+0000 7fedecff9700 1 -- 172.21.15.16:0/3313161154 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fede000f770 con 0x7feddc004060 2023-10-16T13:44:41.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.707+0000 7fedf6780700 1 -- 172.21.15.16:0/3313161154 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedf0154cc0 con 0x7feddc004060 2023-10-16T13:44:41.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.707+0000 7fedecff9700 1 -- 172.21.15.16:0/3313161154 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fede000fb20 con 0x7feddc004060 2023-10-16T13:44:41.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.707+0000 7fedf6780700 1 -- 172.21.15.16:0/3313161154 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedf0155280 con 0x7feddc004060 2023-10-16T13:44:41.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.708+0000 7fedf6780700 1 -- 172.21.15.16:0/3313161154 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedf004ebb0 con 0x7feddc004060 2023-10-16T13:44:41.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.708+0000 7fedecff9700 1 -- 172.21.15.16:0/3313161154 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 126) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fede000fc80 con 0x7feddc004060 2023-10-16T13:44:41.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.709+0000 7fedecff9700 1 --2- 172.21.15.16:0/3313161154 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7fedd006b020 0x7fedd006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:41.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.709+0000 7fedecff9700 1 -- 172.21.15.16:0/3313161154 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 6157+0+0 (secure 0 0 0) 0x7fede0089900 con 0x7feddc004060 2023-10-16T13:44:41.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.709+0000 7fedef7fe700 1 --2- 172.21.15.16:0/3313161154 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7fedd006b020 0x7fedd006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:41.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.709+0000 7fedef7fe700 1 --2- 172.21.15.16:0/3313161154 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7fedd006b020 0x7fedd006d4e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fedd8000c00 tx=0x7fedd8008030 comp rx=0 tx=0).ready entity=mgr.10665 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:41.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.713+0000 7fedecff9700 1 -- 172.21.15.16:0/3313161154 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fede0054640 con 0x7feddc004060 2023-10-16T13:44:41.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:41.972+0000 7fedf6780700 1 -- 172.21.15.16:0/3313161154 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fedf00ce3d0 con 0x7feddc004060 2023-10-16T13:44:42.769 INFO:tasks.ceph.mds.a.smithi016.stderr:2023-10-16T13:44:42.767+0000 7f933b7c2700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T13:44:42.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:42.830+0000 7fedecff9700 1 -- 172.21.15.16:0/3313161154 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v103) v1 ==== 118+0+0 (secure 0 0 0) 0x7fede0057c60 con 0x7feddc004060 2023-10-16T13:44:42.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:42.833+0000 7fedf6780700 1 -- 172.21.15.16:0/3313161154 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7fedd006b020 msgr2=0x7fedd006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:42.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:42.833+0000 7fedf6780700 1 --2- 172.21.15.16:0/3313161154 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7fedd006b020 0x7fedd006d4e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fedd8000c00 tx=0x7fedd8008030 comp rx=0 tx=0).stop 2023-10-16T13:44:42.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:42.833+0000 7fedf6780700 1 -- 172.21.15.16:0/3313161154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feddc004060 msgr2=0x7fedf01557a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:42.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:42.833+0000 7fedf6780700 1 --2- 172.21.15.16:0/3313161154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feddc004060 0x7fedf01557a0 secure :-1 s=READY pgs=2286 cs=0 l=1 rev1=1 crypto rx=0x7fede00072c0 tx=0x7fede0007680 comp rx=0 tx=0).stop 2023-10-16T13:44:42.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:42.833+0000 7fedf6780700 1 -- 172.21.15.16:0/3313161154 shutdown_connections 2023-10-16T13:44:42.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:42.833+0000 7fedf6780700 1 --2- 172.21.15.16:0/3313161154 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7fedd006b020 0x7fedd006d4e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:42.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:42.833+0000 7fedf6780700 1 --2- 172.21.15.16:0/3313161154 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fedf00cb6b0 0x7fedf01537a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:42.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:42.833+0000 7fedf6780700 1 --2- 172.21.15.16:0/3313161154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feddc004060 0x7fedf01557a0 unknown :-1 s=CLOSED pgs=2286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:42.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:42.833+0000 7fedf6780700 1 --2- 172.21.15.16:0/3313161154 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fedf015c6d0 0x7fedf00cb170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:42.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:42.833+0000 7fedf6780700 1 -- 172.21.15.16:0/3313161154 >> 172.21.15.16:0/3313161154 conn(0x7fedf00cc5e0 msgr2=0x7fedf00cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:42.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:42.833+0000 7fedf6780700 1 -- 172.21.15.16:0/3313161154 shutdown_connections 2023-10-16T13:44:42.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:42.833+0000 7fedf6780700 1 -- 172.21.15.16:0/3313161154 wait complete. 2023-10-16T13:44:42.852 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-16T13:44:42.852 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-16T13:44:42.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:42.985+0000 7f5c99a52700 1 Processor -- start 2023-10-16T13:44:42.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:42.985+0000 7f5c99a52700 1 -- start start 2023-10-16T13:44:42.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:42.986+0000 7f5c99a52700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5c94151640 0x7f5c940cf580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:42.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:42.986+0000 7f5c99a52700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5c940ce2c0 0x7f5c9415e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:42.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:42.986+0000 7f5c99a52700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f5c9415e620 0x7f5c9415aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:42.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:42.986+0000 7f5c99a52700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c9415ea90 con 0x7f5c94151640 2023-10-16T13:44:42.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:42.986+0000 7f5c99a52700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c9415bcb0 con 0x7f5c9415e620 2023-10-16T13:44:42.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:42.986+0000 7f5c99a52700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c940b3370 con 0x7f5c940ce2c0 2023-10-16T13:44:42.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:42.986+0000 7f5c92ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5c94151640 0x7f5c940cf580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:42.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:42.986+0000 7f5c92ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5c94151640 0x7f5c940cf580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51720/0 (socket says 172.21.15.16:51720) 2023-10-16T13:44:42.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:42.986+0000 7f5c92ffd700 1 -- 172.21.15.16:0/1047387495 learned_addr learned my addr 172.21.15.16:0/1047387495 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:42.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:42.986+0000 7f5c927fc700 1 --2- 172.21.15.16:0/1047387495 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5c940ce2c0 0x7f5c9415e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:42.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:42.986+0000 7f5c937fe700 1 --2- 172.21.15.16:0/1047387495 >> v2:172.21.15.49:3300/0 conn(0x7f5c9415e620 0x7f5c9415aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:42.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:42.987+0000 7f5c92ffd700 1 -- 172.21.15.16:0/1047387495 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5c940ce2c0 msgr2=0x7f5c9415e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:42.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:42.987+0000 7f5c92ffd700 1 --2- 172.21.15.16:0/1047387495 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5c940ce2c0 0x7f5c9415e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:42.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:42.987+0000 7f5c92ffd700 1 -- 172.21.15.16:0/1047387495 >> v2:172.21.15.49:3300/0 conn(0x7f5c9415e620 msgr2=0x7f5c9415aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:42.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:42.987+0000 7f5c92ffd700 1 --2- 172.21.15.16:0/1047387495 >> v2:172.21.15.49:3300/0 conn(0x7f5c9415e620 0x7f5c9415aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:42.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:42.987+0000 7f5c92ffd700 1 -- 172.21.15.16:0/1047387495 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c94151a20 con 0x7f5c94151640 2023-10-16T13:44:42.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:42.987+0000 7f5c92ffd700 1 --2- 172.21.15.16:0/1047387495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c94151640 0x7f5c940cf580 secure :-1 s=READY pgs=2287 cs=0 l=1 rev1=1 crypto rx=0x7f5c9415bc00 tx=0x7f5c84009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=203e6da7c4290d4a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:42.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:42.987+0000 7f5c91ffb700 1 -- 172.21.15.16:0/1047387495 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c84011830 con 0x7f5c94151640 2023-10-16T13:44:42.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:42.987+0000 7f5c91ffb700 1 -- 172.21.15.16:0/1047387495 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c84011990 con 0x7f5c94151640 2023-10-16T13:44:42.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:42.987+0000 7f5c91ffb700 1 -- 172.21.15.16:0/1047387495 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c84011d20 con 0x7f5c94151640 2023-10-16T13:44:42.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:42.988+0000 7f5c99a52700 1 -- 172.21.15.16:0/1047387495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c94151640 msgr2=0x7f5c940cf580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:42.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:42.988+0000 7f5c99a52700 1 --2- 172.21.15.16:0/1047387495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c94151640 0x7f5c940cf580 secure :-1 s=READY pgs=2287 cs=0 l=1 rev1=1 crypto rx=0x7f5c9415bc00 tx=0x7f5c84009bd0 comp rx=0 tx=0).stop 2023-10-16T13:44:42.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:42.988+0000 7f5c99a52700 1 -- 172.21.15.16:0/1047387495 shutdown_connections 2023-10-16T13:44:42.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:42.988+0000 7f5c99a52700 1 --2- 172.21.15.16:0/1047387495 >> v2:172.21.15.49:3300/0 conn(0x7f5c9415e620 0x7f5c9415aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:42.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:42.988+0000 7f5c99a52700 1 --2- 172.21.15.16:0/1047387495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c94151640 0x7f5c940cf580 unknown :-1 s=CLOSED pgs=2287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:42.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:42.988+0000 7f5c99a52700 1 --2- 172.21.15.16:0/1047387495 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5c940ce2c0 0x7f5c9415e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:42.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:42.988+0000 7f5c99a52700 1 -- 172.21.15.16:0/1047387495 >> 172.21.15.16:0/1047387495 conn(0x7f5c940cfd50 msgr2=0x7f5c940d0130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:42.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:42.988+0000 7f5c99a52700 1 -- 172.21.15.16:0/1047387495 shutdown_connections 2023-10-16T13:44:42.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:42.988+0000 7f5c99a52700 1 -- 172.21.15.16:0/1047387495 wait complete. 2023-10-16T13:44:42.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:42.989+0000 7f5c99a52700 1 Processor -- start 2023-10-16T13:44:42.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:42.989+0000 7f5c99a52700 1 -- start start 2023-10-16T13:44:43.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:42.990+0000 7f5c99a52700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c940ce2c0 0x7f5c9415abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:43.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:42.990+0000 7f5c99a52700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5c94151640 0x7f5c941451c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:43.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:42.990+0000 7f5c99a52700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5c9415e620 0x7f5c94145700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:43.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:42.990+0000 7f5c99a52700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c940ca040 con 0x7f5c940ce2c0 2023-10-16T13:44:43.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:42.990+0000 7f5c99a52700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c94151fc0 con 0x7f5c94151640 2023-10-16T13:44:43.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:42.990+0000 7f5c99a52700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c941f4a90 con 0x7f5c9415e620 2023-10-16T13:44:43.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:42.990+0000 7f5c92ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c940ce2c0 0x7f5c9415abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:43.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:42.990+0000 7f5c92ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c940ce2c0 0x7f5c9415abf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51726/0 (socket says 172.21.15.16:51726) 2023-10-16T13:44:43.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:42.990+0000 7f5c92ffd700 1 -- 172.21.15.16:0/2911360314 learned_addr learned my addr 172.21.15.16:0/2911360314 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:43.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:42.990+0000 7f5c927fc700 1 --2- 172.21.15.16:0/2911360314 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5c94151640 0x7f5c941451c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:43.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:42.990+0000 7f5c937fe700 1 --2- 172.21.15.16:0/2911360314 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5c9415e620 0x7f5c94145700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:43.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:42.990+0000 7f5c92ffd700 1 -- 172.21.15.16:0/2911360314 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5c9415e620 msgr2=0x7f5c94145700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:43.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:42.990+0000 7f5c92ffd700 1 --2- 172.21.15.16:0/2911360314 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5c9415e620 0x7f5c94145700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:43.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:42.990+0000 7f5c92ffd700 1 -- 172.21.15.16:0/2911360314 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5c94151640 msgr2=0x7f5c941451c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:43.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:42.990+0000 7f5c92ffd700 1 --2- 172.21.15.16:0/2911360314 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5c94151640 0x7f5c941451c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:43.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:42.990+0000 7f5c92ffd700 1 -- 172.21.15.16:0/2911360314 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c84009550 con 0x7f5c940ce2c0 2023-10-16T13:44:43.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:42.990+0000 7f5c92ffd700 1 --2- 172.21.15.16:0/2911360314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c940ce2c0 0x7f5c9415abf0 secure :-1 s=READY pgs=2288 cs=0 l=1 rev1=1 crypto rx=0x7f5c84000c00 tx=0x7f5c84004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:43.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:42.991+0000 7f5c7bfff700 1 -- 172.21.15.16:0/2911360314 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c84004240 con 0x7f5c940ce2c0 2023-10-16T13:44:43.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:42.991+0000 7f5c7bfff700 1 -- 172.21.15.16:0/2911360314 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c8401f070 con 0x7f5c940ce2c0 2023-10-16T13:44:43.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:42.991+0000 7f5c7bfff700 1 -- 172.21.15.16:0/2911360314 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c840045e0 con 0x7f5c940ce2c0 2023-10-16T13:44:43.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:42.991+0000 7f5c99a52700 1 -- 172.21.15.16:0/2911360314 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c941f4d70 con 0x7f5c940ce2c0 2023-10-16T13:44:43.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:42.991+0000 7f5c99a52700 1 -- 172.21.15.16:0/2911360314 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c940c9c60 con 0x7f5c940ce2c0 2023-10-16T13:44:43.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:42.992+0000 7f5c99a52700 1 -- 172.21.15.16:0/2911360314 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c940ca180 con 0x7f5c940ce2c0 2023-10-16T13:44:43.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:42.992+0000 7f5c7bfff700 1 -- 172.21.15.16:0/2911360314 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 126) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5c84021030 con 0x7f5c940ce2c0 2023-10-16T13:44:43.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:42.993+0000 7f5c7bfff700 1 --2- 172.21.15.16:0/2911360314 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f5c8006b020 0x7f5c8006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:43.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:42.993+0000 7f5c7bfff700 1 -- 172.21.15.16:0/2911360314 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(206..206 src has 1..206) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f5c8408a9c0 con 0x7f5c940ce2c0 2023-10-16T13:44:43.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:42.993+0000 7f5c927fc700 1 --2- 172.21.15.16:0/2911360314 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f5c8006b020 0x7f5c8006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:43.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:42.993+0000 7f5c927fc700 1 --2- 172.21.15.16:0/2911360314 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f5c8006b020 0x7f5c8006d4e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5c88009ac0 tx=0x7f5c88009210 comp rx=0 tx=0).ready entity=mgr.10665 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:43.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:42.997+0000 7f5c7bfff700 1 -- 172.21.15.16:0/2911360314 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5c940ca180 con 0x7f5c940ce2c0 2023-10-16T13:44:43.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:43.253+0000 7f5c99a52700 1 -- 172.21.15.16:0/2911360314 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f5c940ca180 con 0x7f5c940ce2c0 2023-10-16T13:44:43.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:43.832+0000 7f5c7bfff700 1 -- 172.21.15.16:0/2911360314 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v104) v1 ==== 122+0+0 (secure 0 0 0) 0x7f5c940ca180 con 0x7f5c940ce2c0 2023-10-16T13:44:43.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:43.836+0000 7f5c99a52700 1 -- 172.21.15.16:0/2911360314 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f5c8006b020 msgr2=0x7f5c8006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:43.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:43.836+0000 7f5c99a52700 1 --2- 172.21.15.16:0/2911360314 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f5c8006b020 0x7f5c8006d4e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5c88009ac0 tx=0x7f5c88009210 comp rx=0 tx=0).stop 2023-10-16T13:44:43.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:43.836+0000 7f5c99a52700 1 -- 172.21.15.16:0/2911360314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c940ce2c0 msgr2=0x7f5c9415abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:43.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:43.836+0000 7f5c99a52700 1 --2- 172.21.15.16:0/2911360314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c940ce2c0 0x7f5c9415abf0 secure :-1 s=READY pgs=2288 cs=0 l=1 rev1=1 crypto rx=0x7f5c84000c00 tx=0x7f5c84004750 comp rx=0 tx=0).stop 2023-10-16T13:44:43.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:43.836+0000 7f5c99a52700 1 -- 172.21.15.16:0/2911360314 shutdown_connections 2023-10-16T13:44:43.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:43.836+0000 7f5c99a52700 1 --2- 172.21.15.16:0/2911360314 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f5c8006b020 0x7f5c8006d4e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:43.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:43.836+0000 7f5c99a52700 1 --2- 172.21.15.16:0/2911360314 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5c9415e620 0x7f5c94145700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:43.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:43.836+0000 7f5c99a52700 1 --2- 172.21.15.16:0/2911360314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c940ce2c0 0x7f5c9415abf0 unknown :-1 s=CLOSED pgs=2288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:43.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:43.836+0000 7f5c99a52700 1 --2- 172.21.15.16:0/2911360314 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5c94151640 0x7f5c941451c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:43.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:43.836+0000 7f5c99a52700 1 -- 172.21.15.16:0/2911360314 >> 172.21.15.16:0/2911360314 conn(0x7f5c940cfd50 msgr2=0x7f5c940b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:43.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:43.836+0000 7f5c99a52700 1 -- 172.21.15.16:0/2911360314 shutdown_connections 2023-10-16T13:44:43.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:43.836+0000 7f5c99a52700 1 -- 172.21.15.16:0/2911360314 wait complete. 2023-10-16T13:44:43.855 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-16T13:44:43.856 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:44:43.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:43.985+0000 7f47e1acf700 1 Processor -- start 2023-10-16T13:44:43.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:43.985+0000 7f47e1acf700 1 -- start start 2023-10-16T13:44:43.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:43.986+0000 7f47e1acf700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f47dc0cc8a0 0x7f47dc15e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:43.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:43.986+0000 7f47e1acf700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f47dc06a7e0 0x7f47dc06ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:43.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:43.986+0000 7f47e1acf700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47dc06b150 con 0x7f47dc15b280 2023-10-16T13:44:43.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:43.986+0000 7f47e1acf700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47dc161c30 con 0x7f47dc0cc8a0 2023-10-16T13:44:43.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:43.986+0000 7f47e1acf700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47dc161f10 con 0x7f47dc06a7e0 2023-10-16T13:44:43.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:43.986+0000 7f47daffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f47dc0cc8a0 0x7f47dc15e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:43.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:43.986+0000 7f47db7fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f47dc15b280 0x7f47dc1613e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41008/0 (socket says 172.21.15.16:41008) 2023-10-16T13:44:43.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:43.986+0000 7f47daffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f47dc0cc8a0 0x7f47dc15e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51736/0 (socket says 172.21.15.16:51736) 2023-10-16T13:44:43.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:43.986+0000 7f47db7fe700 1 -- 172.21.15.16:0/2104750691 learned_addr learned my addr 172.21.15.16:0/2104750691 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:44:43.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:43.986+0000 7f47dbfff700 1 --2- 172.21.15.16:0/2104750691 >> v2:172.21.15.49:3300/0 conn(0x7f47dc06a7e0 0x7f47dc06ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:43.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:43.986+0000 7f47daffd700 1 -- 172.21.15.16:0/2104750691 >> v2:172.21.15.49:3300/0 conn(0x7f47dc06a7e0 msgr2=0x7f47dc06ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:43.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:43.986+0000 7f47daffd700 1 --2- 172.21.15.16:0/2104750691 >> v2:172.21.15.49:3300/0 conn(0x7f47dc06a7e0 0x7f47dc06ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:43.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:43.986+0000 7f47daffd700 1 -- 172.21.15.16:0/2104750691 >> v1:172.21.15.16:6789/0 conn(0x7f47dc15b280 legacy=0x7f47dc1613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:43.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:43.986+0000 7f47daffd700 1 -- 172.21.15.16:0/2104750691 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47dc1621f0 con 0x7f47dc0cc8a0 2023-10-16T13:44:43.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:43.986+0000 7f47da7fc700 1 -- 172.21.15.16:0/2104750691 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 596355623 0 0) 0x7f47dc06b150 con 0x7f47dc15b280 2023-10-16T13:44:43.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:43.986+0000 7f47daffd700 1 --2- 172.21.15.16:0/2104750691 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47dc0cc8a0 0x7f47dc15e700 secure :-1 s=READY pgs=2290 cs=0 l=1 rev1=1 crypto rx=0x7f47dc0cbed0 tx=0x7f47cc009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99aad01ed3e5415 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:43.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:43.986+0000 7f47da7fc700 1 -- 172.21.15.16:0/2104750691 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47cc011830 con 0x7f47dc0cc8a0 2023-10-16T13:44:43.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:43.986+0000 7f47da7fc700 1 -- 172.21.15.16:0/2104750691 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47cc011990 con 0x7f47dc0cc8a0 2023-10-16T13:44:43.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:43.986+0000 7f47da7fc700 1 -- 172.21.15.16:0/2104750691 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47cc011d20 con 0x7f47dc0cc8a0 2023-10-16T13:44:43.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:43.987+0000 7f47e1acf700 1 -- 172.21.15.16:0/2104750691 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47dc0cc8a0 msgr2=0x7f47dc15e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:43.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:43.987+0000 7f47e1acf700 1 --2- 172.21.15.16:0/2104750691 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47dc0cc8a0 0x7f47dc15e700 secure :-1 s=READY pgs=2290 cs=0 l=1 rev1=1 crypto rx=0x7f47dc0cbed0 tx=0x7f47cc009b20 comp rx=0 tx=0).stop 2023-10-16T13:44:43.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:43.987+0000 7f47e1acf700 1 -- 172.21.15.16:0/2104750691 shutdown_connections 2023-10-16T13:44:43.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:43.987+0000 7f47e1acf700 1 --2- 172.21.15.16:0/2104750691 >> v2:172.21.15.49:3300/0 conn(0x7f47dc06a7e0 0x7f47dc06ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:43.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:43.987+0000 7f47e1acf700 1 --2- 172.21.15.16:0/2104750691 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47dc0cc8a0 0x7f47dc15e700 unknown :-1 s=CLOSED pgs=2290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:43.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:43.987+0000 7f47e1acf700 1 -- 172.21.15.16:0/2104750691 >> 172.21.15.16:0/2104750691 conn(0x7f47dc0b1fd0 msgr2=0x7f47dc0b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:43.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:43.987+0000 7f47e1acf700 1 -- 172.21.15.16:0/2104750691 shutdown_connections 2023-10-16T13:44:43.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:43.987+0000 7f47e1acf700 1 -- 172.21.15.16:0/2104750691 wait complete. 2023-10-16T13:44:43.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:43.988+0000 7f47e1acf700 1 Processor -- start 2023-10-16T13:44:43.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:43.988+0000 7f47e1acf700 1 -- start start 2023-10-16T13:44:43.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:43.989+0000 7f47e1acf700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47dc06a7e0 0x7f47dc156b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:43.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:43.989+0000 7f47e1acf700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f47dc15b280 0x7f47dc15d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:43.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:43.989+0000 7f47e1acf700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f47dc15c3c0 0x7f47dc0cce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:43.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:43.989+0000 7f47e1acf700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47dc0c6160 con 0x7f47dc06a7e0 2023-10-16T13:44:43.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:43.989+0000 7f47e1acf700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47dc0c5550 con 0x7f47dc15b280 2023-10-16T13:44:43.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:43.989+0000 7f47e1acf700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47dc0c5830 con 0x7f47dc15c3c0 2023-10-16T13:44:43.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:43.989+0000 7f47db7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47dc06a7e0 0x7f47dc156b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:44.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:43.989+0000 7f47db7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47dc06a7e0 0x7f47dc156b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51740/0 (socket says 172.21.15.16:51740) 2023-10-16T13:44:44.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:43.989+0000 7f47db7fe700 1 -- 172.21.15.16:0/3081208069 learned_addr learned my addr 172.21.15.16:0/3081208069 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:44.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:43.989+0000 7f47daffd700 1 --2- 172.21.15.16:0/3081208069 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f47dc15b280 0x7f47dc15d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:44.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:43.989+0000 7f47dbfff700 1 --2- 172.21.15.16:0/3081208069 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f47dc15c3c0 0x7f47dc0cce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:44.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:43.989+0000 7f47db7fe700 1 -- 172.21.15.16:0/3081208069 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f47dc15c3c0 msgr2=0x7f47dc0cce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:44.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:43.989+0000 7f47db7fe700 1 --2- 172.21.15.16:0/3081208069 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f47dc15c3c0 0x7f47dc0cce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:44.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:43.989+0000 7f47db7fe700 1 -- 172.21.15.16:0/3081208069 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f47dc15b280 msgr2=0x7f47dc15d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:44.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:43.989+0000 7f47db7fe700 1 --2- 172.21.15.16:0/3081208069 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f47dc15b280 0x7f47dc15d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:44.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:43.990+0000 7f47db7fe700 1 -- 172.21.15.16:0/3081208069 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47cc0094a0 con 0x7f47dc06a7e0 2023-10-16T13:44:44.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:43.990+0000 7f47db7fe700 1 --2- 172.21.15.16:0/3081208069 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47dc06a7e0 0x7f47dc156b60 secure :-1 s=READY pgs=2291 cs=0 l=1 rev1=1 crypto rx=0x7f47d000c550 tx=0x7f47d000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:44.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:43.990+0000 7f47c3fff700 1 -- 172.21.15.16:0/3081208069 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47d0004c80 con 0x7f47dc06a7e0 2023-10-16T13:44:44.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:43.990+0000 7f47c3fff700 1 -- 172.21.15.16:0/3081208069 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47d0004e00 con 0x7f47dc06a7e0 2023-10-16T13:44:44.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:43.990+0000 7f47e1acf700 1 -- 172.21.15.16:0/3081208069 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47dc0c5b10 con 0x7f47dc06a7e0 2023-10-16T13:44:44.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:43.990+0000 7f47c3fff700 1 -- 172.21.15.16:0/3081208069 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47d0007310 con 0x7f47dc06a7e0 2023-10-16T13:44:44.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:43.990+0000 7f47e1acf700 1 -- 172.21.15.16:0/3081208069 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47dc15d740 con 0x7f47dc06a7e0 2023-10-16T13:44:44.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:43.991+0000 7f47e1acf700 1 -- 172.21.15.16:0/3081208069 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47dc04ebb0 con 0x7f47dc06a7e0 2023-10-16T13:44:44.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:43.991+0000 7f47c3fff700 1 -- 172.21.15.16:0/3081208069 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 126) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f47d0007470 con 0x7f47dc06a7e0 2023-10-16T13:44:44.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:43.992+0000 7f47c3fff700 1 --2- 172.21.15.16:0/3081208069 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f47c806b020 0x7f47c806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:44.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:43.992+0000 7f47c3fff700 1 -- 172.21.15.16:0/3081208069 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(206..206 src has 1..206) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f47d0089680 con 0x7f47dc06a7e0 2023-10-16T13:44:44.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:43.992+0000 7f47daffd700 1 --2- 172.21.15.16:0/3081208069 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f47c806b020 0x7f47c806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:44.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:43.992+0000 7f47daffd700 1 --2- 172.21.15.16:0/3081208069 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f47c806b020 0x7f47c806d4e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f47dc0cbed0 tx=0x7f47cc008030 comp rx=0 tx=0).ready entity=mgr.10665 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:44.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:43.996+0000 7f47c3fff700 1 -- 172.21.15.16:0/3081208069 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f47d00543e0 con 0x7f47dc06a7e0 2023-10-16T13:44:44.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.243+0000 7f47e1acf700 1 -- 172.21.15.16:0/3081208069 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f47dc0c64f0 con 0x7f47dc06a7e0 2023-10-16T13:44:44.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.244+0000 7f47c3fff700 1 -- 172.21.15.16:0/3081208069 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+4925 (secure 0 0 0) 0x7f47d0057a00 con 0x7f47dc06a7e0 2023-10-16T13:44:44.248 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:44:44.248 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":104,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8729,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/3614602930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":3614602930},{"type":"v1","addr":"172.21.15.49:6835","nonce":3614602930}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":99},{"gid":10045,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/3427940027","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":3427940027},{"type":"v1","addr":"172.21.15.49:6834","nonce":3427940027}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":99},{"gid":10896,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/4279896386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":4279896386},{"type":"v1","addr":"172.21.15.16:6836","nonce":4279896386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:44:40.208202+0000","modified":"2023-10-16T13:44:43.255450+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10899},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10899":{"gid":10899,"name":"a","rank":0,"incarnation":100,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/4233835071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":4233835071},{"type":"v1","addr":"172.21.15.16:6837","nonce":4233835071}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2023-10-16T13:44:44.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.246+0000 7f47e1acf700 1 -- 172.21.15.16:0/3081208069 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f47c806b020 msgr2=0x7f47c806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:44.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.246+0000 7f47e1acf700 1 --2- 172.21.15.16:0/3081208069 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f47c806b020 0x7f47c806d4e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f47dc0cbed0 tx=0x7f47cc008030 comp rx=0 tx=0).stop 2023-10-16T13:44:44.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.246+0000 7f47e1acf700 1 -- 172.21.15.16:0/3081208069 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47dc06a7e0 msgr2=0x7f47dc156b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:44.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.246+0000 7f47e1acf700 1 --2- 172.21.15.16:0/3081208069 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47dc06a7e0 0x7f47dc156b60 secure :-1 s=READY pgs=2291 cs=0 l=1 rev1=1 crypto rx=0x7f47d000c550 tx=0x7f47d000c910 comp rx=0 tx=0).stop 2023-10-16T13:44:44.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.246+0000 7f47e1acf700 1 -- 172.21.15.16:0/3081208069 shutdown_connections 2023-10-16T13:44:44.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.246+0000 7f47e1acf700 1 --2- 172.21.15.16:0/3081208069 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f47c806b020 0x7f47c806d4e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:44.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.246+0000 7f47e1acf700 1 --2- 172.21.15.16:0/3081208069 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f47dc15c3c0 0x7f47dc0cce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:44.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.246+0000 7f47e1acf700 1 --2- 172.21.15.16:0/3081208069 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47dc06a7e0 0x7f47dc156b60 unknown :-1 s=CLOSED pgs=2291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:44.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.246+0000 7f47e1acf700 1 --2- 172.21.15.16:0/3081208069 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f47dc15b280 0x7f47dc15d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:44.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.246+0000 7f47e1acf700 1 -- 172.21.15.16:0/3081208069 >> 172.21.15.16:0/3081208069 conn(0x7f47dc0b1fd0 msgr2=0x7f47dc15a900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:44.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.246+0000 7f47e1acf700 1 -- 172.21.15.16:0/3081208069 shutdown_connections 2023-10-16T13:44:44.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.246+0000 7f47e1acf700 1 -- 172.21.15.16:0/3081208069 wait complete. 2023-10-16T13:44:44.252 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 104 2023-10-16T13:44:44.262 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:44:44.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.384+0000 7fb2adea3700 1 Processor -- start 2023-10-16T13:44:44.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.384+0000 7fb2adea3700 1 -- start start 2023-10-16T13:44:44.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.385+0000 7fb2adea3700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fb2a815b280 0x7fb2a81613e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:44.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.385+0000 7fb2adea3700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb2a806a7e0 0x7fb2a806ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:44.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.385+0000 7fb2adea3700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2a8161c30 con 0x7fb2a80cc8a0 2023-10-16T13:44:44.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.385+0000 7fb2adea3700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2a806b190 con 0x7fb2a815b280 2023-10-16T13:44:44.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.385+0000 7fb2adea3700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2a8161f10 con 0x7fb2a806a7e0 2023-10-16T13:44:44.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.385+0000 7fb2a7fff700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb2a806a7e0 0x7fb2a806ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:44.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.385+0000 7fb2a77fe700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fb2a815b280 0x7fb2a81613e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:44.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.385+0000 7fb2a7fff700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb2a806a7e0 0x7fb2a806ac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:40010/0 (socket says 172.21.15.16:40010) 2023-10-16T13:44:44.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.385+0000 7fb2a7fff700 1 -- 172.21.15.16:0/4162872182 learned_addr learned my addr 172.21.15.16:0/4162872182 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:44.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.385+0000 7fb2a6ffd700 1 --1- 172.21.15.16:0/4162872182 >> v1:172.21.15.49:6789/0 conn(0x7fb2a80cc8a0 0x7fb2a815e700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:48204/0 (socket says 172.21.15.16:48204) 2023-10-16T13:44:44.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.385+0000 7fb2a7fff700 1 -- 172.21.15.16:0/4162872182 >> v2:172.21.15.49:3300/0 conn(0x7fb2a815b280 msgr2=0x7fb2a81613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:44.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.385+0000 7fb2a7fff700 1 --2- 172.21.15.16:0/4162872182 >> v2:172.21.15.49:3300/0 conn(0x7fb2a815b280 0x7fb2a81613e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:44.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.385+0000 7fb2a7fff700 1 -- 172.21.15.16:0/4162872182 >> v1:172.21.15.49:6789/0 conn(0x7fb2a80cc8a0 legacy=0x7fb2a815e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:44.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.385+0000 7fb2a7fff700 1 -- 172.21.15.16:0/4162872182 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2a81621f0 con 0x7fb2a806a7e0 2023-10-16T13:44:44.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.385+0000 7fb2a77fe700 1 --2- 172.21.15.16:0/4162872182 >> v2:172.21.15.49:3300/0 conn(0x7fb2a815b280 0x7fb2a81613e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:44:44.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.386+0000 7fb2a7fff700 1 --2- 172.21.15.16:0/4162872182 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb2a806a7e0 0x7fb2a806ac50 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fb2a0009e50 tx=0x7fb2a000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab93fe93d4100ff8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:44.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.386+0000 7fb2a67fc700 1 -- 172.21.15.16:0/4162872182 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2a000e070 con 0x7fb2a806a7e0 2023-10-16T13:44:44.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.386+0000 7fb2a67fc700 1 -- 172.21.15.16:0/4162872182 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2a000b910 con 0x7fb2a806a7e0 2023-10-16T13:44:44.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.386+0000 7fb2a67fc700 1 -- 172.21.15.16:0/4162872182 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2a000bcc0 con 0x7fb2a806a7e0 2023-10-16T13:44:44.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.386+0000 7fb2adea3700 1 -- 172.21.15.16:0/4162872182 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb2a806a7e0 msgr2=0x7fb2a806ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:44.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.386+0000 7fb2adea3700 1 --2- 172.21.15.16:0/4162872182 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb2a806a7e0 0x7fb2a806ac50 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fb2a0009e50 tx=0x7fb2a000b330 comp rx=0 tx=0).stop 2023-10-16T13:44:44.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.386+0000 7fb2adea3700 1 -- 172.21.15.16:0/4162872182 shutdown_connections 2023-10-16T13:44:44.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.386+0000 7fb2adea3700 1 --2- 172.21.15.16:0/4162872182 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb2a806a7e0 0x7fb2a806ac50 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:44.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.386+0000 7fb2adea3700 1 --2- 172.21.15.16:0/4162872182 >> v2:172.21.15.49:3300/0 conn(0x7fb2a815b280 0x7fb2a81613e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:44.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.386+0000 7fb2adea3700 1 -- 172.21.15.16:0/4162872182 >> 172.21.15.16:0/4162872182 conn(0x7fb2a80b1fd0 msgr2=0x7fb2a80b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:44.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.386+0000 7fb2adea3700 1 -- 172.21.15.16:0/4162872182 shutdown_connections 2023-10-16T13:44:44.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.387+0000 7fb2adea3700 1 -- 172.21.15.16:0/4162872182 wait complete. 2023-10-16T13:44:44.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.388+0000 7fb2adea3700 1 Processor -- start 2023-10-16T13:44:44.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.388+0000 7fb2adea3700 1 -- start start 2023-10-16T13:44:44.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.389+0000 7fb2adea3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2a806a7e0 0x7fb2a8158bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:44.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.389+0000 7fb2adea3700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb2a80cc8a0 0x7fb2a8071de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:44.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.389+0000 7fb2adea3700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb2a815b280 0x7fb2a8072320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:44.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.389+0000 7fb2adea3700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2a80c5c00 con 0x7fb2a806a7e0 2023-10-16T13:44:44.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.389+0000 7fb2adea3700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2a81f6410 con 0x7fb2a815b280 2023-10-16T13:44:44.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.389+0000 7fb2adea3700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2a8159130 con 0x7fb2a80cc8a0 2023-10-16T13:44:44.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.389+0000 7fb2a77fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2a806a7e0 0x7fb2a8158bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:44.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.389+0000 7fb2a77fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2a806a7e0 0x7fb2a8158bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51754/0 (socket says 172.21.15.16:51754) 2023-10-16T13:44:44.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.389+0000 7fb2a77fe700 1 -- 172.21.15.16:0/5951042 learned_addr learned my addr 172.21.15.16:0/5951042 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:44.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.389+0000 7fb2a6ffd700 1 --2- 172.21.15.16:0/5951042 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb2a80cc8a0 0x7fb2a8071de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:44.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.389+0000 7fb2a7fff700 1 --2- 172.21.15.16:0/5951042 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb2a815b280 0x7fb2a8072320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:44.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.389+0000 7fb2a77fe700 1 -- 172.21.15.16:0/5951042 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb2a80cc8a0 msgr2=0x7fb2a8071de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:44.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.389+0000 7fb2a77fe700 1 --2- 172.21.15.16:0/5951042 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb2a80cc8a0 0x7fb2a8071de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:44.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.389+0000 7fb2a77fe700 1 -- 172.21.15.16:0/5951042 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb2a815b280 msgr2=0x7fb2a8072320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:44.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.389+0000 7fb2a77fe700 1 --2- 172.21.15.16:0/5951042 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb2a815b280 0x7fb2a8072320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:44.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.389+0000 7fb2a77fe700 1 -- 172.21.15.16:0/5951042 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2980094a0 con 0x7fb2a806a7e0 2023-10-16T13:44:44.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.389+0000 7fb2a7fff700 1 --2- 172.21.15.16:0/5951042 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb2a815b280 0x7fb2a8072320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:44:44.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.389+0000 7fb2a77fe700 1 --2- 172.21.15.16:0/5951042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2a806a7e0 0x7fb2a8158bf0 secure :-1 s=READY pgs=2292 cs=0 l=1 rev1=1 crypto rx=0x7fb29800ebe0 tx=0x7fb298009e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:44.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.389+0000 7fb28ffff700 1 -- 172.21.15.16:0/5951042 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb29800c8b0 con 0x7fb2a806a7e0 2023-10-16T13:44:44.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.389+0000 7fb28ffff700 1 -- 172.21.15.16:0/5951042 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb29800ca10 con 0x7fb2a806a7e0 2023-10-16T13:44:44.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.390+0000 7fb2adea3700 1 -- 172.21.15.16:0/5951042 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2a0009b30 con 0x7fb2a806a7e0 2023-10-16T13:44:44.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.390+0000 7fb28ffff700 1 -- 172.21.15.16:0/5951042 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb29800cda0 con 0x7fb2a806a7e0 2023-10-16T13:44:44.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.390+0000 7fb2adea3700 1 -- 172.21.15.16:0/5951042 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2a81f6660 con 0x7fb2a806a7e0 2023-10-16T13:44:44.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.391+0000 7fb2adea3700 1 -- 172.21.15.16:0/5951042 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2a81f6f90 con 0x7fb2a806a7e0 2023-10-16T13:44:44.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.391+0000 7fb28ffff700 1 -- 172.21.15.16:0/5951042 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 126) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb298013470 con 0x7fb2a806a7e0 2023-10-16T13:44:44.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.391+0000 7fb28ffff700 1 --2- 172.21.15.16:0/5951042 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7fb29406b020 0x7fb29406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:44.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.392+0000 7fb2a6ffd700 1 --2- 172.21.15.16:0/5951042 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7fb29406b020 0x7fb29406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:44.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.392+0000 7fb28ffff700 1 -- 172.21.15.16:0/5951042 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(206..206 src has 1..206) v4 ==== 6157+0+0 (secure 0 0 0) 0x7fb298089ad0 con 0x7fb2a806a7e0 2023-10-16T13:44:44.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.392+0000 7fb2a6ffd700 1 --2- 172.21.15.16:0/5951042 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7fb29406b020 0x7fb29406d4e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb2a80b1180 tx=0x7fb29c007d10 comp rx=0 tx=0).ready entity=mgr.10665 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:44.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.396+0000 7fb28ffff700 1 -- 172.21.15.16:0/5951042 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb298054810 con 0x7fb2a806a7e0 2023-10-16T13:44:44.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.600+0000 7fb2adea3700 1 -- 172.21.15.16:0/5951042 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb2a804ebb0 con 0x7fb2a806a7e0 2023-10-16T13:44:44.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.602+0000 7fb28ffff700 1 -- 172.21.15.16:0/5951042 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v206) v1 ==== 74+0+16817 (secure 0 0 0) 0x7fb298017090 con 0x7fb2a806a7e0 2023-10-16T13:44:44.605 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:44:44.606 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":206,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:44:41.798857+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2023-10-16T13:44:38.716950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"205","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_data","create_time":"2023-10-16T13:44:39.172499+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"206","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":204,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":204,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":204,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":204,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":204,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":204,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":204,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":204,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:44:44.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.604+0000 7fb2adea3700 1 -- 172.21.15.16:0/5951042 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7fb29406b020 msgr2=0x7fb29406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:44.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.604+0000 7fb2adea3700 1 --2- 172.21.15.16:0/5951042 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7fb29406b020 0x7fb29406d4e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb2a80b1180 tx=0x7fb29c007d10 comp rx=0 tx=0).stop 2023-10-16T13:44:44.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.604+0000 7fb2adea3700 1 -- 172.21.15.16:0/5951042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2a806a7e0 msgr2=0x7fb2a8158bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:44.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.604+0000 7fb2adea3700 1 --2- 172.21.15.16:0/5951042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2a806a7e0 0x7fb2a8158bf0 secure :-1 s=READY pgs=2292 cs=0 l=1 rev1=1 crypto rx=0x7fb29800ebe0 tx=0x7fb298009e20 comp rx=0 tx=0).stop 2023-10-16T13:44:44.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.604+0000 7fb2adea3700 1 -- 172.21.15.16:0/5951042 shutdown_connections 2023-10-16T13:44:44.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.604+0000 7fb2adea3700 1 --2- 172.21.15.16:0/5951042 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7fb29406b020 0x7fb29406d4e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:44.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.604+0000 7fb2adea3700 1 --2- 172.21.15.16:0/5951042 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb2a815b280 0x7fb2a8072320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:44.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.604+0000 7fb2adea3700 1 --2- 172.21.15.16:0/5951042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2a806a7e0 0x7fb2a8158bf0 unknown :-1 s=CLOSED pgs=2292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:44.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.604+0000 7fb2adea3700 1 --2- 172.21.15.16:0/5951042 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb2a80cc8a0 0x7fb2a8071de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:44.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.604+0000 7fb2adea3700 1 -- 172.21.15.16:0/5951042 >> 172.21.15.16:0/5951042 conn(0x7fb2a80b1fd0 msgr2=0x7fb2a815a670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:44.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.604+0000 7fb2adea3700 1 -- 172.21.15.16:0/5951042 shutdown_connections 2023-10-16T13:44:44.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.604+0000 7fb2adea3700 1 -- 172.21.15.16:0/5951042 wait complete. 2023-10-16T13:44:44.620 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-16T13:44:44.621 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T13:44:44.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.742+0000 7fbc077af700 1 Processor -- start 2023-10-16T13:44:44.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.742+0000 7fbc077af700 1 -- start start 2023-10-16T13:44:44.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.743+0000 7fbc077af700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fbc0015b280 0x7fbc001613e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:44.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.743+0000 7fbc077af700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbc0006a7e0 0x7fbc0006ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:44.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.743+0000 7fbc077af700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc0006b150 con 0x7fbc000cc8a0 2023-10-16T13:44:44.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.743+0000 7fbc077af700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc00161c30 con 0x7fbc0006a7e0 2023-10-16T13:44:44.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.743+0000 7fbc077af700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc00161f10 con 0x7fbc0015b280 2023-10-16T13:44:44.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.743+0000 7fbc05d4c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbc0006a7e0 0x7fbc0006ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:44.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.743+0000 7fbc05d4c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbc0006a7e0 0x7fbc0006ac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51766/0 (socket says 172.21.15.16:51766) 2023-10-16T13:44:44.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.743+0000 7fbc04d4a700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fbc000cc8a0 0x7fbc0015e700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41036/0 (socket says 172.21.15.16:41036) 2023-10-16T13:44:44.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.743+0000 7fbc05d4c700 1 -- 172.21.15.16:0/1500736581 learned_addr learned my addr 172.21.15.16:0/1500736581 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:44.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.744+0000 7fbc0554b700 1 --2- 172.21.15.16:0/1500736581 >> v2:172.21.15.49:3300/0 conn(0x7fbc0015b280 0x7fbc001613e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:44.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.744+0000 7fbc05d4c700 1 -- 172.21.15.16:0/1500736581 >> v2:172.21.15.49:3300/0 conn(0x7fbc0015b280 msgr2=0x7fbc001613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:44.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.744+0000 7fbc05d4c700 1 --2- 172.21.15.16:0/1500736581 >> v2:172.21.15.49:3300/0 conn(0x7fbc0015b280 0x7fbc001613e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:44.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.744+0000 7fbc05d4c700 1 -- 172.21.15.16:0/1500736581 >> v1:172.21.15.16:6789/0 conn(0x7fbc000cc8a0 legacy=0x7fbc0015e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:44.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.744+0000 7fbc05d4c700 1 -- 172.21.15.16:0/1500736581 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc001621f0 con 0x7fbc0006a7e0 2023-10-16T13:44:44.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.744+0000 7fbc05d4c700 1 --2- 172.21.15.16:0/1500736581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc0006a7e0 0x7fbc0006ac10 secure :-1 s=READY pgs=2294 cs=0 l=1 rev1=1 crypto rx=0x7fbc0006b8a0 tx=0x7fbbf800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6b8a7e0af4b2099 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:44.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.744+0000 7fbbf7fff700 1 -- 172.21.15.16:0/1500736581 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbf800e070 con 0x7fbc0006a7e0 2023-10-16T13:44:44.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.744+0000 7fbbf7fff700 1 -- 172.21.15.16:0/1500736581 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbf8003a20 con 0x7fbc0006a7e0 2023-10-16T13:44:44.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.744+0000 7fbbf7fff700 1 -- 172.21.15.16:0/1500736581 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbf8003dd0 con 0x7fbc0006a7e0 2023-10-16T13:44:44.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.745+0000 7fbc077af700 1 -- 172.21.15.16:0/1500736581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc0006a7e0 msgr2=0x7fbc0006ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:44.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.745+0000 7fbc077af700 1 --2- 172.21.15.16:0/1500736581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc0006a7e0 0x7fbc0006ac10 secure :-1 s=READY pgs=2294 cs=0 l=1 rev1=1 crypto rx=0x7fbc0006b8a0 tx=0x7fbbf800b330 comp rx=0 tx=0).stop 2023-10-16T13:44:44.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.745+0000 7fbc077af700 1 -- 172.21.15.16:0/1500736581 shutdown_connections 2023-10-16T13:44:44.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.745+0000 7fbc077af700 1 --2- 172.21.15.16:0/1500736581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc0006a7e0 0x7fbc0006ac10 unknown :-1 s=CLOSED pgs=2294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:44.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.745+0000 7fbc077af700 1 --2- 172.21.15.16:0/1500736581 >> v2:172.21.15.49:3300/0 conn(0x7fbc0015b280 0x7fbc001613e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:44.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.745+0000 7fbc077af700 1 -- 172.21.15.16:0/1500736581 >> 172.21.15.16:0/1500736581 conn(0x7fbc000b1fd0 msgr2=0x7fbc000b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:44.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.745+0000 7fbc077af700 1 -- 172.21.15.16:0/1500736581 shutdown_connections 2023-10-16T13:44:44.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.745+0000 7fbc077af700 1 -- 172.21.15.16:0/1500736581 wait complete. 2023-10-16T13:44:44.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.746+0000 7fbc077af700 1 Processor -- start 2023-10-16T13:44:44.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.746+0000 7fbc077af700 1 -- start start 2023-10-16T13:44:44.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.747+0000 7fbc077af700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc0006a7e0 0x7fbc001f27b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:44.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.747+0000 7fbc077af700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbc000cc8a0 0x7fbc000cb7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:44.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.747+0000 7fbc077af700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbc0015b280 0x7fbc000cbce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:44.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.747+0000 7fbc077af700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc0014b0a0 con 0x7fbc0006a7e0 2023-10-16T13:44:44.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.747+0000 7fbc077af700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc001f7290 con 0x7fbc0015b280 2023-10-16T13:44:44.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.747+0000 7fbc077af700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc0014b2f0 con 0x7fbc000cc8a0 2023-10-16T13:44:44.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.747+0000 7fbc0554b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc0006a7e0 0x7fbc001f27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:44.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.747+0000 7fbc0554b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc0006a7e0 0x7fbc001f27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51768/0 (socket says 172.21.15.16:51768) 2023-10-16T13:44:44.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.747+0000 7fbc0554b700 1 -- 172.21.15.16:0/2277991080 learned_addr learned my addr 172.21.15.16:0/2277991080 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:44.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.747+0000 7fbc05d4c700 1 --2- 172.21.15.16:0/2277991080 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbc0015b280 0x7fbc000cbce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:44.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.747+0000 7fbc04d4a700 1 --2- 172.21.15.16:0/2277991080 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbc000cc8a0 0x7fbc000cb7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:44.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.747+0000 7fbc0554b700 1 -- 172.21.15.16:0/2277991080 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbc000cc8a0 msgr2=0x7fbc000cb7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:44.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.747+0000 7fbc0554b700 1 --2- 172.21.15.16:0/2277991080 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbc000cc8a0 0x7fbc000cb7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:44.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.747+0000 7fbc0554b700 1 -- 172.21.15.16:0/2277991080 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbc0015b280 msgr2=0x7fbc000cbce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:44.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.747+0000 7fbc0554b700 1 --2- 172.21.15.16:0/2277991080 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbc0015b280 0x7fbc000cbce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:44.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.747+0000 7fbc0554b700 1 -- 172.21.15.16:0/2277991080 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbf8009b30 con 0x7fbc0006a7e0 2023-10-16T13:44:44.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.747+0000 7fbc05d4c700 1 --2- 172.21.15.16:0/2277991080 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbc0015b280 0x7fbc000cbce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:44:44.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.747+0000 7fbc0554b700 1 --2- 172.21.15.16:0/2277991080 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc0006a7e0 0x7fbc001f27b0 secure :-1 s=READY pgs=2295 cs=0 l=1 rev1=1 crypto rx=0x7fbbf000e810 tx=0x7fbbf000ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:44.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.747+0000 7fbbf5ffb700 1 -- 172.21.15.16:0/2277991080 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbf0013610 con 0x7fbc0006a7e0 2023-10-16T13:44:44.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.747+0000 7fbbf5ffb700 1 -- 172.21.15.16:0/2277991080 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbf0013770 con 0x7fbc0006a7e0 2023-10-16T13:44:44.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.747+0000 7fbc077af700 1 -- 172.21.15.16:0/2277991080 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc0014b5d0 con 0x7fbc0006a7e0 2023-10-16T13:44:44.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.748+0000 7fbbf5ffb700 1 -- 172.21.15.16:0/2277991080 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbf0013b20 con 0x7fbc0006a7e0 2023-10-16T13:44:44.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.748+0000 7fbc077af700 1 -- 172.21.15.16:0/2277991080 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc001f2cf0 con 0x7fbc0006a7e0 2023-10-16T13:44:44.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.749+0000 7fbc077af700 1 -- 172.21.15.16:0/2277991080 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc0004ebb0 con 0x7fbc0006a7e0 2023-10-16T13:44:44.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.749+0000 7fbbf5ffb700 1 -- 172.21.15.16:0/2277991080 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 126) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fbbf0013c80 con 0x7fbc0006a7e0 2023-10-16T13:44:44.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.750+0000 7fbbf5ffb700 1 --2- 172.21.15.16:0/2277991080 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7fbbe406b020 0x7fbbe406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:44.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.750+0000 7fbbf5ffb700 1 -- 172.21.15.16:0/2277991080 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(206..206 src has 1..206) v4 ==== 6157+0+0 (secure 0 0 0) 0x7fbbf0089760 con 0x7fbc0006a7e0 2023-10-16T13:44:44.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.750+0000 7fbc04d4a700 1 --2- 172.21.15.16:0/2277991080 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7fbbe406b020 0x7fbbe406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:44.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.750+0000 7fbc04d4a700 1 --2- 172.21.15.16:0/2277991080 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7fbbe406b020 0x7fbbe406d4e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fbc0014eb10 tx=0x7fbbfc009040 comp rx=0 tx=0).ready entity=mgr.10665 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:44.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.754+0000 7fbbf5ffb700 1 -- 172.21.15.16:0/2277991080 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbbf0054440 con 0x7fbc0006a7e0 2023-10-16T13:44:44.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.954+0000 7fbc077af700 1 -- 172.21.15.16:0/2277991080 --> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbc000ad490 con 0x7fbbe406b020 2023-10-16T13:44:44.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.963+0000 7fbbf5ffb700 1 -- 172.21.15.16:0/2277991080 <== mgr.10665 v2:172.21.15.16:6832/1268439414 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289891 (secure 0 0 0) 0x7fbc000ad490 con 0x7fbbe406b020 2023-10-16T13:44:44.970 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:44:44.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.967+0000 7fbc077af700 1 -- 172.21.15.16:0/2277991080 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7fbbe406b020 msgr2=0x7fbbe406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:44.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.967+0000 7fbc077af700 1 --2- 172.21.15.16:0/2277991080 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7fbbe406b020 0x7fbbe406d4e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fbc0014eb10 tx=0x7fbbfc009040 comp rx=0 tx=0).stop 2023-10-16T13:44:44.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.967+0000 7fbc077af700 1 -- 172.21.15.16:0/2277991080 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc0006a7e0 msgr2=0x7fbc001f27b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:44.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.967+0000 7fbc077af700 1 --2- 172.21.15.16:0/2277991080 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc0006a7e0 0x7fbc001f27b0 secure :-1 s=READY pgs=2295 cs=0 l=1 rev1=1 crypto rx=0x7fbbf000e810 tx=0x7fbbf000ebd0 comp rx=0 tx=0).stop 2023-10-16T13:44:44.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.967+0000 7fbc077af700 1 -- 172.21.15.16:0/2277991080 shutdown_connections 2023-10-16T13:44:44.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.967+0000 7fbc077af700 1 --2- 172.21.15.16:0/2277991080 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7fbbe406b020 0x7fbbe406d4e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:44.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.967+0000 7fbc077af700 1 --2- 172.21.15.16:0/2277991080 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbc0015b280 0x7fbc000cbce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:44.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.967+0000 7fbc077af700 1 --2- 172.21.15.16:0/2277991080 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc0006a7e0 0x7fbc001f27b0 unknown :-1 s=CLOSED pgs=2295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:44.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.967+0000 7fbc077af700 1 --2- 172.21.15.16:0/2277991080 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbc000cc8a0 0x7fbc000cb7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:44.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.967+0000 7fbc077af700 1 -- 172.21.15.16:0/2277991080 >> 172.21.15.16:0/2277991080 conn(0x7fbc000b1fd0 msgr2=0x7fbc0015d670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:44.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.967+0000 7fbc077af700 1 -- 172.21.15.16:0/2277991080 shutdown_connections 2023-10-16T13:44:44.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:44.967+0000 7fbc077af700 1 -- 172.21.15.16:0/2277991080 wait complete. 2023-10-16T13:44:44.975 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-10-16T13:44:44.983 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2023-10-16T13:44:44.426110+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1940,"num_read_kb":4014,"num_write":583,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":364,"ondisk_log_size":364,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":136,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":383248,"kb_used_data":21344,"kb_used_omap":12,"kb_used_meta":361779,"kb_avail":754591472,"statfs":{"total":773094113280,"available":772701667328,"internally_reserved":0,"allocated":21856256,"data_stored":11729884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12740,"internal_metadata":370462268},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.654429"},"pg_stats":[{"pgid":"26.3f","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.264372+0000","last_change":"2023-10-16T13:44:40.826902+0000","last_active":"2023-10-16T13:44:42.264372+0000","last_peered":"2023-10-16T13:44:42.264372+0000","last_clean":"2023-10-16T13:44:42.264372+0000","last_became_active":"2023-10-16T13:44:40.826586+0000","last_became_peered":"2023-10-16T13:44:40.826586+0000","last_unstale":"2023-10-16T13:44:42.264372+0000","last_undegraded":"2023-10-16T13:44:42.264372+0000","last_fullsized":"2023-10-16T13:44:42.264372+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:47:01.341720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.3e","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.264405+0000","last_change":"2023-10-16T13:44:40.816625+0000","last_active":"2023-10-16T13:44:42.264405+0000","last_peered":"2023-10-16T13:44:42.264405+0000","last_clean":"2023-10-16T13:44:42.264405+0000","last_became_active":"2023-10-16T13:44:40.815628+0000","last_became_peered":"2023-10-16T13:44:40.815628+0000","last_unstale":"2023-10-16T13:44:42.264405+0000","last_undegraded":"2023-10-16T13:44:42.264405+0000","last_fullsized":"2023-10-16T13:44:42.264405+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:30.538620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.3d","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.265348+0000","last_change":"2023-10-16T13:44:40.821185+0000","last_active":"2023-10-16T13:44:42.265348+0000","last_peered":"2023-10-16T13:44:42.265348+0000","last_clean":"2023-10-16T13:44:42.265348+0000","last_became_active":"2023-10-16T13:44:40.820881+0000","last_became_peered":"2023-10-16T13:44:40.820881+0000","last_unstale":"2023-10-16T13:44:42.265348+0000","last_undegraded":"2023-10-16T13:44:42.265348+0000","last_fullsized":"2023-10-16T13:44:42.265348+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:13:11.470834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.3c","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.263941+0000","last_change":"2023-10-16T13:44:40.820272+0000","last_active":"2023-10-16T13:44:42.263941+0000","last_peered":"2023-10-16T13:44:42.263941+0000","last_clean":"2023-10-16T13:44:42.263941+0000","last_became_active":"2023-10-16T13:44:40.820011+0000","last_became_peered":"2023-10-16T13:44:40.820011+0000","last_unstale":"2023-10-16T13:44:42.263941+0000","last_undegraded":"2023-10-16T13:44:42.263941+0000","last_fullsized":"2023-10-16T13:44:42.263941+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:03:54.739025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.3b","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.803290+0000","last_change":"2023-10-16T13:44:40.810785+0000","last_active":"2023-10-16T13:44:41.803290+0000","last_peered":"2023-10-16T13:44:41.803290+0000","last_clean":"2023-10-16T13:44:41.803290+0000","last_became_active":"2023-10-16T13:44:40.810486+0000","last_became_peered":"2023-10-16T13:44:40.810486+0000","last_unstale":"2023-10-16T13:44:41.803290+0000","last_undegraded":"2023-10-16T13:44:41.803290+0000","last_fullsized":"2023-10-16T13:44:41.803290+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:55:25.050877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.3a","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.803359+0000","last_change":"2023-10-16T13:44:40.819529+0000","last_active":"2023-10-16T13:44:41.803359+0000","last_peered":"2023-10-16T13:44:41.803359+0000","last_clean":"2023-10-16T13:44:41.803359+0000","last_became_active":"2023-10-16T13:44:40.819182+0000","last_became_peered":"2023-10-16T13:44:40.819182+0000","last_unstale":"2023-10-16T13:44:41.803359+0000","last_undegraded":"2023-10-16T13:44:41.803359+0000","last_fullsized":"2023-10-16T13:44:41.803359+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:25:15.818591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.39","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.801818+0000","last_change":"2023-10-16T13:44:40.826478+0000","last_active":"2023-10-16T13:44:41.801818+0000","last_peered":"2023-10-16T13:44:41.801818+0000","last_clean":"2023-10-16T13:44:41.801818+0000","last_became_active":"2023-10-16T13:44:40.826181+0000","last_became_peered":"2023-10-16T13:44:40.826181+0000","last_unstale":"2023-10-16T13:44:41.801818+0000","last_undegraded":"2023-10-16T13:44:41.801818+0000","last_fullsized":"2023-10-16T13:44:41.801818+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:29:14.006949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.38","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.899132+0000","last_change":"2023-10-16T13:44:40.825799+0000","last_active":"2023-10-16T13:44:41.899132+0000","last_peered":"2023-10-16T13:44:41.899132+0000","last_clean":"2023-10-16T13:44:41.899132+0000","last_became_active":"2023-10-16T13:44:40.825491+0000","last_became_peered":"2023-10-16T13:44:40.825491+0000","last_unstale":"2023-10-16T13:44:41.899132+0000","last_undegraded":"2023-10-16T13:44:41.899132+0000","last_fullsized":"2023-10-16T13:44:41.899132+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:38:47.872141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.37","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.264456+0000","last_change":"2023-10-16T13:44:40.814315+0000","last_active":"2023-10-16T13:44:42.264456+0000","last_peered":"2023-10-16T13:44:42.264456+0000","last_clean":"2023-10-16T13:44:42.264456+0000","last_became_active":"2023-10-16T13:44:40.814017+0000","last_became_peered":"2023-10-16T13:44:40.814017+0000","last_unstale":"2023-10-16T13:44:42.264456+0000","last_undegraded":"2023-10-16T13:44:42.264456+0000","last_fullsized":"2023-10-16T13:44:42.264456+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:39:57.915119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.36","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.803512+0000","last_change":"2023-10-16T13:44:40.819560+0000","last_active":"2023-10-16T13:44:41.803512+0000","last_peered":"2023-10-16T13:44:41.803512+0000","last_clean":"2023-10-16T13:44:41.803512+0000","last_became_active":"2023-10-16T13:44:40.819156+0000","last_became_peered":"2023-10-16T13:44:40.819156+0000","last_unstale":"2023-10-16T13:44:41.803512+0000","last_undegraded":"2023-10-16T13:44:41.803512+0000","last_fullsized":"2023-10-16T13:44:41.803512+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:41:39.966432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.35","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.264874+0000","last_change":"2023-10-16T13:44:40.817394+0000","last_active":"2023-10-16T13:44:42.264874+0000","last_peered":"2023-10-16T13:44:42.264874+0000","last_clean":"2023-10-16T13:44:42.264874+0000","last_became_active":"2023-10-16T13:44:40.817131+0000","last_became_peered":"2023-10-16T13:44:40.817131+0000","last_unstale":"2023-10-16T13:44:42.264874+0000","last_undegraded":"2023-10-16T13:44:42.264874+0000","last_fullsized":"2023-10-16T13:44:42.264874+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:09:03.969733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.34","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.264473+0000","last_change":"2023-10-16T13:44:40.829617+0000","last_active":"2023-10-16T13:44:42.264473+0000","last_peered":"2023-10-16T13:44:42.264473+0000","last_clean":"2023-10-16T13:44:42.264473+0000","last_became_active":"2023-10-16T13:44:40.829156+0000","last_became_peered":"2023-10-16T13:44:40.829156+0000","last_unstale":"2023-10-16T13:44:42.264473+0000","last_undegraded":"2023-10-16T13:44:42.264473+0000","last_fullsized":"2023-10-16T13:44:42.264473+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:42:50.455035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.33","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.265085+0000","last_change":"2023-10-16T13:44:40.813904+0000","last_active":"2023-10-16T13:44:42.265085+0000","last_peered":"2023-10-16T13:44:42.265085+0000","last_clean":"2023-10-16T13:44:42.265085+0000","last_became_active":"2023-10-16T13:44:40.813589+0000","last_became_peered":"2023-10-16T13:44:40.813589+0000","last_unstale":"2023-10-16T13:44:42.265085+0000","last_undegraded":"2023-10-16T13:44:42.265085+0000","last_fullsized":"2023-10-16T13:44:42.265085+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:25:14.054951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.32","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.801681+0000","last_change":"2023-10-16T13:44:40.813471+0000","last_active":"2023-10-16T13:44:41.801681+0000","last_peered":"2023-10-16T13:44:41.801681+0000","last_clean":"2023-10-16T13:44:41.801681+0000","last_became_active":"2023-10-16T13:44:40.812676+0000","last_became_peered":"2023-10-16T13:44:40.812676+0000","last_unstale":"2023-10-16T13:44:41.801681+0000","last_undegraded":"2023-10-16T13:44:41.801681+0000","last_fullsized":"2023-10-16T13:44:41.801681+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:31:53.429470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.31","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.263699+0000","last_change":"2023-10-16T13:44:40.816134+0000","last_active":"2023-10-16T13:44:42.263699+0000","last_peered":"2023-10-16T13:44:42.263699+0000","last_clean":"2023-10-16T13:44:42.263699+0000","last_became_active":"2023-10-16T13:44:40.815573+0000","last_became_peered":"2023-10-16T13:44:40.815573+0000","last_unstale":"2023-10-16T13:44:42.263699+0000","last_undegraded":"2023-10-16T13:44:42.263699+0000","last_fullsized":"2023-10-16T13:44:42.263699+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:09:03.969733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.30","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.803164+0000","last_change":"2023-10-16T13:44:40.816249+0000","last_active":"2023-10-16T13:44:41.803164+0000","last_peered":"2023-10-16T13:44:41.803164+0000","last_clean":"2023-10-16T13:44:41.803164+0000","last_became_active":"2023-10-16T13:44:40.815919+0000","last_became_peered":"2023-10-16T13:44:40.815919+0000","last_unstale":"2023-10-16T13:44:41.803164+0000","last_undegraded":"2023-10-16T13:44:41.803164+0000","last_fullsized":"2023-10-16T13:44:41.803164+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:41:29.011926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.2f","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.803460+0000","last_change":"2023-10-16T13:44:40.826410+0000","last_active":"2023-10-16T13:44:41.803460+0000","last_peered":"2023-10-16T13:44:41.803460+0000","last_clean":"2023-10-16T13:44:41.803460+0000","last_became_active":"2023-10-16T13:44:40.826089+0000","last_became_peered":"2023-10-16T13:44:40.826089+0000","last_unstale":"2023-10-16T13:44:41.803460+0000","last_undegraded":"2023-10-16T13:44:41.803460+0000","last_fullsized":"2023-10-16T13:44:41.803460+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:39:08.726953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.2e","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.264313+0000","last_change":"2023-10-16T13:44:40.826108+0000","last_active":"2023-10-16T13:44:42.264313+0000","last_peered":"2023-10-16T13:44:42.264313+0000","last_clean":"2023-10-16T13:44:42.264313+0000","last_became_active":"2023-10-16T13:44:40.825804+0000","last_became_peered":"2023-10-16T13:44:40.825804+0000","last_unstale":"2023-10-16T13:44:42.264313+0000","last_undegraded":"2023-10-16T13:44:42.264313+0000","last_fullsized":"2023-10-16T13:44:42.264313+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:21:56.832533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.2d","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.264552+0000","last_change":"2023-10-16T13:44:40.810061+0000","last_active":"2023-10-16T13:44:42.264552+0000","last_peered":"2023-10-16T13:44:42.264552+0000","last_clean":"2023-10-16T13:44:42.264552+0000","last_became_active":"2023-10-16T13:44:40.809672+0000","last_became_peered":"2023-10-16T13:44:40.809672+0000","last_unstale":"2023-10-16T13:44:42.264552+0000","last_undegraded":"2023-10-16T13:44:42.264552+0000","last_fullsized":"2023-10-16T13:44:42.264552+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:07:54.767396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.2c","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.803628+0000","last_change":"2023-10-16T13:44:40.829539+0000","last_active":"2023-10-16T13:44:41.803628+0000","last_peered":"2023-10-16T13:44:41.803628+0000","last_clean":"2023-10-16T13:44:41.803628+0000","last_became_active":"2023-10-16T13:44:40.829219+0000","last_became_peered":"2023-10-16T13:44:40.829219+0000","last_unstale":"2023-10-16T13:44:41.803628+0000","last_undegraded":"2023-10-16T13:44:41.803628+0000","last_fullsized":"2023-10-16T13:44:41.803628+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:43:50.582740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.2b","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.264122+0000","last_change":"2023-10-16T13:44:40.814959+0000","last_active":"2023-10-16T13:44:42.264122+0000","last_peered":"2023-10-16T13:44:42.264122+0000","last_clean":"2023-10-16T13:44:42.264122+0000","last_became_active":"2023-10-16T13:44:40.814620+0000","last_became_peered":"2023-10-16T13:44:40.814620+0000","last_unstale":"2023-10-16T13:44:42.264122+0000","last_undegraded":"2023-10-16T13:44:42.264122+0000","last_fullsized":"2023-10-16T13:44:42.264122+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:36:06.385172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.2a","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.263873+0000","last_change":"2023-10-16T13:44:40.817464+0000","last_active":"2023-10-16T13:44:42.263873+0000","last_peered":"2023-10-16T13:44:42.263873+0000","last_clean":"2023-10-16T13:44:42.263873+0000","last_became_active":"2023-10-16T13:44:40.817178+0000","last_became_peered":"2023-10-16T13:44:40.817178+0000","last_unstale":"2023-10-16T13:44:42.263873+0000","last_undegraded":"2023-10-16T13:44:42.263873+0000","last_fullsized":"2023-10-16T13:44:42.263873+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:32:26.254719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.29","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.899353+0000","last_change":"2023-10-16T13:44:40.825857+0000","last_active":"2023-10-16T13:44:41.899353+0000","last_peered":"2023-10-16T13:44:41.899353+0000","last_clean":"2023-10-16T13:44:41.899353+0000","last_became_active":"2023-10-16T13:44:40.825553+0000","last_became_peered":"2023-10-16T13:44:40.825553+0000","last_unstale":"2023-10-16T13:44:41.899353+0000","last_undegraded":"2023-10-16T13:44:41.899353+0000","last_fullsized":"2023-10-16T13:44:41.899353+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:08:27.304252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.28","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.264436+0000","last_change":"2023-10-16T13:44:40.828378+0000","last_active":"2023-10-16T13:44:42.264436+0000","last_peered":"2023-10-16T13:44:42.264436+0000","last_clean":"2023-10-16T13:44:42.264436+0000","last_became_active":"2023-10-16T13:44:40.828088+0000","last_became_peered":"2023-10-16T13:44:40.828088+0000","last_unstale":"2023-10-16T13:44:42.264436+0000","last_undegraded":"2023-10-16T13:44:42.264436+0000","last_fullsized":"2023-10-16T13:44:42.264436+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:34:24.198222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.27","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.801791+0000","last_change":"2023-10-16T13:44:40.817056+0000","last_active":"2023-10-16T13:44:41.801791+0000","last_peered":"2023-10-16T13:44:41.801791+0000","last_clean":"2023-10-16T13:44:41.801791+0000","last_became_active":"2023-10-16T13:44:40.816727+0000","last_became_peered":"2023-10-16T13:44:40.816727+0000","last_unstale":"2023-10-16T13:44:41.801791+0000","last_undegraded":"2023-10-16T13:44:41.801791+0000","last_fullsized":"2023-10-16T13:44:41.801791+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:57:05.396702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.26","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.265199+0000","last_change":"2023-10-16T13:44:40.814830+0000","last_active":"2023-10-16T13:44:42.265199+0000","last_peered":"2023-10-16T13:44:42.265199+0000","last_clean":"2023-10-16T13:44:42.265199+0000","last_became_active":"2023-10-16T13:44:40.814089+0000","last_became_peered":"2023-10-16T13:44:40.814089+0000","last_unstale":"2023-10-16T13:44:42.265199+0000","last_undegraded":"2023-10-16T13:44:42.265199+0000","last_fullsized":"2023-10-16T13:44:42.265199+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:42:27.071178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.25","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.899335+0000","last_change":"2023-10-16T13:44:40.825948+0000","last_active":"2023-10-16T13:44:41.899335+0000","last_peered":"2023-10-16T13:44:41.899335+0000","last_clean":"2023-10-16T13:44:41.899335+0000","last_became_active":"2023-10-16T13:44:40.825659+0000","last_became_peered":"2023-10-16T13:44:40.825659+0000","last_unstale":"2023-10-16T13:44:41.899335+0000","last_undegraded":"2023-10-16T13:44:41.899335+0000","last_fullsized":"2023-10-16T13:44:41.899335+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:43:26.778586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.24","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.803323+0000","last_change":"2023-10-16T13:44:40.815262+0000","last_active":"2023-10-16T13:44:41.803323+0000","last_peered":"2023-10-16T13:44:41.803323+0000","last_clean":"2023-10-16T13:44:41.803323+0000","last_became_active":"2023-10-16T13:44:40.814894+0000","last_became_peered":"2023-10-16T13:44:40.814894+0000","last_unstale":"2023-10-16T13:44:41.803323+0000","last_undegraded":"2023-10-16T13:44:41.803323+0000","last_fullsized":"2023-10-16T13:44:41.803323+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:30:04.291099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.23","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.265002+0000","last_change":"2023-10-16T13:44:40.819830+0000","last_active":"2023-10-16T13:44:42.265002+0000","last_peered":"2023-10-16T13:44:42.265002+0000","last_clean":"2023-10-16T13:44:42.265002+0000","last_became_active":"2023-10-16T13:44:40.819473+0000","last_became_peered":"2023-10-16T13:44:40.819473+0000","last_unstale":"2023-10-16T13:44:42.265002+0000","last_undegraded":"2023-10-16T13:44:42.265002+0000","last_fullsized":"2023-10-16T13:44:42.265002+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:26:13.931633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.22","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.264999+0000","last_change":"2023-10-16T13:44:40.819766+0000","last_active":"2023-10-16T13:44:42.264999+0000","last_peered":"2023-10-16T13:44:42.264999+0000","last_clean":"2023-10-16T13:44:42.264999+0000","last_became_active":"2023-10-16T13:44:40.819334+0000","last_became_peered":"2023-10-16T13:44:40.819334+0000","last_unstale":"2023-10-16T13:44:42.264999+0000","last_undegraded":"2023-10-16T13:44:42.264999+0000","last_fullsized":"2023-10-16T13:44:42.264999+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:32:26.254719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.21","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.801936+0000","last_change":"2023-10-16T13:44:40.817822+0000","last_active":"2023-10-16T13:44:41.801936+0000","last_peered":"2023-10-16T13:44:41.801936+0000","last_clean":"2023-10-16T13:44:41.801936+0000","last_became_active":"2023-10-16T13:44:40.817470+0000","last_became_peered":"2023-10-16T13:44:40.817470+0000","last_unstale":"2023-10-16T13:44:41.801936+0000","last_undegraded":"2023-10-16T13:44:41.801936+0000","last_fullsized":"2023-10-16T13:44:41.801936+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:36:24.516163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.20","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.803479+0000","last_change":"2023-10-16T13:44:40.808877+0000","last_active":"2023-10-16T13:44:41.803479+0000","last_peered":"2023-10-16T13:44:41.803479+0000","last_clean":"2023-10-16T13:44:41.803479+0000","last_became_active":"2023-10-16T13:44:40.808545+0000","last_became_peered":"2023-10-16T13:44:40.808545+0000","last_unstale":"2023-10-16T13:44:41.803479+0000","last_undegraded":"2023-10-16T13:44:41.803479+0000","last_fullsized":"2023-10-16T13:44:41.803479+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:22:10.355097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1c","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.899457+0000","last_change":"2023-10-16T13:44:40.829316+0000","last_active":"2023-10-16T13:44:41.899457+0000","last_peered":"2023-10-16T13:44:41.899457+0000","last_clean":"2023-10-16T13:44:41.899457+0000","last_became_active":"2023-10-16T13:44:40.828906+0000","last_became_peered":"2023-10-16T13:44:40.828906+0000","last_unstale":"2023-10-16T13:44:41.899457+0000","last_undegraded":"2023-10-16T13:44:41.899457+0000","last_fullsized":"2023-10-16T13:44:41.899457+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:59:10.429999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.1f","version":"205'4","reported_seq":16,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.264191+0000","last_change":"2023-10-16T13:44:39.789990+0000","last_active":"2023-10-16T13:44:42.264191+0000","last_peered":"2023-10-16T13:44:42.264191+0000","last_clean":"2023-10-16T13:44:42.264191+0000","last_became_active":"2023-10-16T13:44:39.789670+0000","last_became_peered":"2023-10-16T13:44:39.789670+0000","last_unstale":"2023-10-16T13:44:42.264191+0000","last_undegraded":"2023-10-16T13:44:42.264191+0000","last_fullsized":"2023-10-16T13:44:42.264191+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_clean_scrub_stamp":"2023-10-16T13:44:38.766670+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:58:57.993734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1d","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.802302+0000","last_change":"2023-10-16T13:44:40.826125+0000","last_active":"2023-10-16T13:44:41.802302+0000","last_peered":"2023-10-16T13:44:41.802302+0000","last_clean":"2023-10-16T13:44:41.802302+0000","last_became_active":"2023-10-16T13:44:40.825778+0000","last_became_peered":"2023-10-16T13:44:40.825778+0000","last_unstale":"2023-10-16T13:44:41.802302+0000","last_undegraded":"2023-10-16T13:44:41.802302+0000","last_fullsized":"2023-10-16T13:44:41.802302+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:37:59.510239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1e","version":"205'1","reported_seq":13,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.803607+0000","last_change":"2023-10-16T13:44:39.798962+0000","last_active":"2023-10-16T13:44:41.803607+0000","last_peered":"2023-10-16T13:44:41.803607+0000","last_clean":"2023-10-16T13:44:41.803607+0000","last_became_active":"2023-10-16T13:44:39.798670+0000","last_became_peered":"2023-10-16T13:44:39.798670+0000","last_unstale":"2023-10-16T13:44:41.803607+0000","last_undegraded":"2023-10-16T13:44:41.803607+0000","last_fullsized":"2023-10-16T13:44:41.803607+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_clean_scrub_stamp":"2023-10-16T13:44:38.766670+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:33:06.483394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1e","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.265504+0000","last_change":"2023-10-16T13:44:40.826910+0000","last_active":"2023-10-16T13:44:42.265504+0000","last_peered":"2023-10-16T13:44:42.265504+0000","last_clean":"2023-10-16T13:44:42.265504+0000","last_became_active":"2023-10-16T13:44:40.826597+0000","last_became_peered":"2023-10-16T13:44:40.826597+0000","last_unstale":"2023-10-16T13:44:42.265504+0000","last_undegraded":"2023-10-16T13:44:42.265504+0000","last_fullsized":"2023-10-16T13:44:42.265504+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:36:06.385172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1d","version":"205'2","reported_seq":14,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.264898+0000","last_change":"2023-10-16T13:44:39.791516+0000","last_active":"2023-10-16T13:44:42.264898+0000","last_peered":"2023-10-16T13:44:42.264898+0000","last_clean":"2023-10-16T13:44:42.264898+0000","last_became_active":"2023-10-16T13:44:39.791205+0000","last_became_peered":"2023-10-16T13:44:39.791205+0000","last_unstale":"2023-10-16T13:44:42.264898+0000","last_undegraded":"2023-10-16T13:44:42.264898+0000","last_fullsized":"2023-10-16T13:44:42.264898+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_clean_scrub_stamp":"2023-10-16T13:44:38.766670+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:27:15.853209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1f","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.265487+0000","last_change":"2023-10-16T13:44:40.815054+0000","last_active":"2023-10-16T13:44:42.265487+0000","last_peered":"2023-10-16T13:44:42.265487+0000","last_clean":"2023-10-16T13:44:42.265487+0000","last_became_active":"2023-10-16T13:44:40.814521+0000","last_became_peered":"2023-10-16T13:44:40.814521+0000","last_unstale":"2023-10-16T13:44:42.265487+0000","last_undegraded":"2023-10-16T13:44:42.265487+0000","last_fullsized":"2023-10-16T13:44:42.265487+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:07:54.767396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1c","version":"205'2","reported_seq":14,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.264263+0000","last_change":"2023-10-16T13:44:39.794105+0000","last_active":"2023-10-16T13:44:42.264263+0000","last_peered":"2023-10-16T13:44:42.264263+0000","last_clean":"2023-10-16T13:44:42.264263+0000","last_became_active":"2023-10-16T13:44:39.793773+0000","last_became_peered":"2023-10-16T13:44:39.793773+0000","last_unstale":"2023-10-16T13:44:42.264263+0000","last_undegraded":"2023-10-16T13:44:42.264263+0000","last_fullsized":"2023-10-16T13:44:42.264263+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_clean_scrub_stamp":"2023-10-16T13:44:38.766670+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:55:37.119423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.264063+0000","last_change":"2023-10-16T13:44:40.809999+0000","last_active":"2023-10-16T13:44:42.264063+0000","last_peered":"2023-10-16T13:44:42.264063+0000","last_clean":"2023-10-16T13:44:42.264063+0000","last_became_active":"2023-10-16T13:44:40.809679+0000","last_became_peered":"2023-10-16T13:44:40.809679+0000","last_unstale":"2023-10-16T13:44:42.264063+0000","last_undegraded":"2023-10-16T13:44:42.264063+0000","last_fullsized":"2023-10-16T13:44:42.264063+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:25:11.898623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.1b","version":"205'1","reported_seq":13,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.803404+0000","last_change":"2023-10-16T13:44:39.799020+0000","last_active":"2023-10-16T13:44:41.803404+0000","last_peered":"2023-10-16T13:44:41.803404+0000","last_clean":"2023-10-16T13:44:41.803404+0000","last_became_active":"2023-10-16T13:44:39.798711+0000","last_became_peered":"2023-10-16T13:44:39.798711+0000","last_unstale":"2023-10-16T13:44:41.803404+0000","last_undegraded":"2023-10-16T13:44:41.803404+0000","last_fullsized":"2023-10-16T13:44:41.803404+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_clean_scrub_stamp":"2023-10-16T13:44:38.766670+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:38:24.443535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.263938+0000","last_change":"2023-10-16T13:44:40.815920+0000","last_active":"2023-10-16T13:44:42.263938+0000","last_peered":"2023-10-16T13:44:42.263938+0000","last_clean":"2023-10-16T13:44:42.263938+0000","last_became_active":"2023-10-16T13:44:40.815144+0000","last_became_peered":"2023-10-16T13:44:40.815144+0000","last_unstale":"2023-10-16T13:44:42.263938+0000","last_undegraded":"2023-10-16T13:44:42.263938+0000","last_fullsized":"2023-10-16T13:44:42.263938+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:26:13.931633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1a","version":"205'1","reported_seq":13,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.263983+0000","last_change":"2023-10-16T13:44:39.790201+0000","last_active":"2023-10-16T13:44:42.263983+0000","last_peered":"2023-10-16T13:44:42.263983+0000","last_clean":"2023-10-16T13:44:42.263983+0000","last_became_active":"2023-10-16T13:44:39.789508+0000","last_became_peered":"2023-10-16T13:44:39.789508+0000","last_unstale":"2023-10-16T13:44:42.263983+0000","last_undegraded":"2023-10-16T13:44:42.263983+0000","last_fullsized":"2023-10-16T13:44:42.263983+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_clean_scrub_stamp":"2023-10-16T13:44:38.766670+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:56:14.442023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1a","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.264020+0000","last_change":"2023-10-16T13:44:40.818039+0000","last_active":"2023-10-16T13:44:42.264020+0000","last_peered":"2023-10-16T13:44:42.264020+0000","last_clean":"2023-10-16T13:44:42.264020+0000","last_became_active":"2023-10-16T13:44:40.817737+0000","last_became_peered":"2023-10-16T13:44:40.817737+0000","last_unstale":"2023-10-16T13:44:42.264020+0000","last_undegraded":"2023-10-16T13:44:42.264020+0000","last_fullsized":"2023-10-16T13:44:42.264020+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:42:27.071178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":12,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.264704+0000","last_change":"2023-10-16T13:44:39.793328+0000","last_active":"2023-10-16T13:44:42.264704+0000","last_peered":"2023-10-16T13:44:42.264704+0000","last_clean":"2023-10-16T13:44:42.264704+0000","last_became_active":"2023-10-16T13:44:39.793035+0000","last_became_peered":"2023-10-16T13:44:39.793035+0000","last_unstale":"2023-10-16T13:44:42.264704+0000","last_undegraded":"2023-10-16T13:44:42.264704+0000","last_fullsized":"2023-10-16T13:44:42.264704+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_clean_scrub_stamp":"2023-10-16T13:44:38.766670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:00:05.037176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.14","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.899566+0000","last_change":"2023-10-16T13:44:40.829589+0000","last_active":"2023-10-16T13:44:41.899566+0000","last_peered":"2023-10-16T13:44:41.899566+0000","last_clean":"2023-10-16T13:44:41.899566+0000","last_became_active":"2023-10-16T13:44:40.828946+0000","last_became_peered":"2023-10-16T13:44:40.828946+0000","last_unstale":"2023-10-16T13:44:41.899566+0000","last_undegraded":"2023-10-16T13:44:41.899566+0000","last_fullsized":"2023-10-16T13:44:41.899566+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:38:25.447770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":12,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.803510+0000","last_change":"2023-10-16T13:44:39.787201+0000","last_active":"2023-10-16T13:44:41.803510+0000","last_peered":"2023-10-16T13:44:41.803510+0000","last_clean":"2023-10-16T13:44:41.803510+0000","last_became_active":"2023-10-16T13:44:39.786906+0000","last_became_peered":"2023-10-16T13:44:39.786906+0000","last_unstale":"2023-10-16T13:44:41.803510+0000","last_undegraded":"2023-10-16T13:44:41.803510+0000","last_fullsized":"2023-10-16T13:44:41.803510+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_clean_scrub_stamp":"2023-10-16T13:44:38.766670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:21:01.530053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.15","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.803576+0000","last_change":"2023-10-16T13:44:40.820881+0000","last_active":"2023-10-16T13:44:41.803576+0000","last_peered":"2023-10-16T13:44:41.803576+0000","last_clean":"2023-10-16T13:44:41.803576+0000","last_became_active":"2023-10-16T13:44:40.820547+0000","last_became_peered":"2023-10-16T13:44:40.820547+0000","last_unstale":"2023-10-16T13:44:41.803576+0000","last_undegraded":"2023-10-16T13:44:41.803576+0000","last_fullsized":"2023-10-16T13:44:41.803576+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:36:29.582161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":12,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.264384+0000","last_change":"2023-10-16T13:44:39.798951+0000","last_active":"2023-10-16T13:44:42.264384+0000","last_peered":"2023-10-16T13:44:42.264384+0000","last_clean":"2023-10-16T13:44:42.264384+0000","last_became_active":"2023-10-16T13:44:39.798595+0000","last_became_peered":"2023-10-16T13:44:39.798595+0000","last_unstale":"2023-10-16T13:44:42.264384+0000","last_undegraded":"2023-10-16T13:44:42.264384+0000","last_fullsized":"2023-10-16T13:44:42.264384+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_clean_scrub_stamp":"2023-10-16T13:44:38.766670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:21:12.180441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.264367+0000","last_change":"2023-10-16T13:44:40.818265+0000","last_active":"2023-10-16T13:44:42.264367+0000","last_peered":"2023-10-16T13:44:42.264367+0000","last_clean":"2023-10-16T13:44:42.264367+0000","last_became_active":"2023-10-16T13:44:40.817914+0000","last_became_peered":"2023-10-16T13:44:40.817914+0000","last_unstale":"2023-10-16T13:44:42.264367+0000","last_undegraded":"2023-10-16T13:44:42.264367+0000","last_fullsized":"2023-10-16T13:44:42.264367+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:29:27.837805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.15","version":"205'3","reported_seq":15,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.263970+0000","last_change":"2023-10-16T13:44:39.788873+0000","last_active":"2023-10-16T13:44:42.263970+0000","last_peered":"2023-10-16T13:44:42.263970+0000","last_clean":"2023-10-16T13:44:42.263970+0000","last_became_active":"2023-10-16T13:44:39.788569+0000","last_became_peered":"2023-10-16T13:44:39.788569+0000","last_unstale":"2023-10-16T13:44:42.263970+0000","last_undegraded":"2023-10-16T13:44:42.263970+0000","last_fullsized":"2023-10-16T13:44:42.263970+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_clean_scrub_stamp":"2023-10-16T13:44:38.766670+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:39:22.547238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.264530+0000","last_change":"2023-10-16T13:44:40.816037+0000","last_active":"2023-10-16T13:44:42.264530+0000","last_peered":"2023-10-16T13:44:42.264530+0000","last_clean":"2023-10-16T13:44:42.264530+0000","last_became_active":"2023-10-16T13:44:40.815712+0000","last_became_peered":"2023-10-16T13:44:40.815712+0000","last_unstale":"2023-10-16T13:44:42.264530+0000","last_undegraded":"2023-10-16T13:44:42.264530+0000","last_fullsized":"2023-10-16T13:44:42.264530+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:41:34.307413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.14","version":"205'6","reported_seq":18,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.264320+0000","last_change":"2023-10-16T13:44:39.792836+0000","last_active":"2023-10-16T13:44:42.264320+0000","last_peered":"2023-10-16T13:44:42.264320+0000","last_clean":"2023-10-16T13:44:42.264320+0000","last_became_active":"2023-10-16T13:44:39.792546+0000","last_became_peered":"2023-10-16T13:44:39.792546+0000","last_unstale":"2023-10-16T13:44:42.264320+0000","last_undegraded":"2023-10-16T13:44:42.264320+0000","last_fullsized":"2023-10-16T13:44:42.264320+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_clean_scrub_stamp":"2023-10-16T13:44:38.766670+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:11:58.401310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.10","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.801661+0000","last_change":"2023-10-16T13:44:40.826862+0000","last_active":"2023-10-16T13:44:41.801661+0000","last_peered":"2023-10-16T13:44:41.801661+0000","last_clean":"2023-10-16T13:44:41.801661+0000","last_became_active":"2023-10-16T13:44:40.825709+0000","last_became_peered":"2023-10-16T13:44:40.825709+0000","last_unstale":"2023-10-16T13:44:41.801661+0000","last_undegraded":"2023-10-16T13:44:41.801661+0000","last_fullsized":"2023-10-16T13:44:41.801661+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:06:12.484962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.13","version":"205'2","reported_seq":14,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.264516+0000","last_change":"2023-10-16T13:44:39.797595+0000","last_active":"2023-10-16T13:44:42.264516+0000","last_peered":"2023-10-16T13:44:42.264516+0000","last_clean":"2023-10-16T13:44:42.264516+0000","last_became_active":"2023-10-16T13:44:39.797344+0000","last_became_peered":"2023-10-16T13:44:39.797344+0000","last_unstale":"2023-10-16T13:44:42.264516+0000","last_undegraded":"2023-10-16T13:44:42.264516+0000","last_fullsized":"2023-10-16T13:44:42.264516+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_clean_scrub_stamp":"2023-10-16T13:44:38.766670+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:45:36.847018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.803113+0000","last_change":"2023-10-16T13:44:40.814628+0000","last_active":"2023-10-16T13:44:41.803113+0000","last_peered":"2023-10-16T13:44:41.803113+0000","last_clean":"2023-10-16T13:44:41.803113+0000","last_became_active":"2023-10-16T13:44:40.814339+0000","last_became_peered":"2023-10-16T13:44:40.814339+0000","last_unstale":"2023-10-16T13:44:41.803113+0000","last_undegraded":"2023-10-16T13:44:41.803113+0000","last_fullsized":"2023-10-16T13:44:41.803113+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:31:52.400499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":12,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.264226+0000","last_change":"2023-10-16T13:44:39.798272+0000","last_active":"2023-10-16T13:44:42.264226+0000","last_peered":"2023-10-16T13:44:42.264226+0000","last_clean":"2023-10-16T13:44:42.264226+0000","last_became_active":"2023-10-16T13:44:39.797982+0000","last_became_peered":"2023-10-16T13:44:39.797982+0000","last_unstale":"2023-10-16T13:44:42.264226+0000","last_undegraded":"2023-10-16T13:44:42.264226+0000","last_fullsized":"2023-10-16T13:44:42.264226+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_clean_scrub_stamp":"2023-10-16T13:44:38.766670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:27:15.853209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.264378+0000","last_change":"2023-10-16T13:44:40.822642+0000","last_active":"2023-10-16T13:44:42.264378+0000","last_peered":"2023-10-16T13:44:42.264378+0000","last_clean":"2023-10-16T13:44:42.264378+0000","last_became_active":"2023-10-16T13:44:40.822332+0000","last_became_peered":"2023-10-16T13:44:40.822332+0000","last_unstale":"2023-10-16T13:44:42.264378+0000","last_undegraded":"2023-10-16T13:44:42.264378+0000","last_fullsized":"2023-10-16T13:44:42.264378+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:13:11.470834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":12,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.263769+0000","last_change":"2023-10-16T13:44:39.791877+0000","last_active":"2023-10-16T13:44:42.263769+0000","last_peered":"2023-10-16T13:44:42.263769+0000","last_clean":"2023-10-16T13:44:42.263769+0000","last_became_active":"2023-10-16T13:44:39.791524+0000","last_became_peered":"2023-10-16T13:44:39.791524+0000","last_unstale":"2023-10-16T13:44:42.263769+0000","last_undegraded":"2023-10-16T13:44:42.263769+0000","last_fullsized":"2023-10-16T13:44:42.263769+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_clean_scrub_stamp":"2023-10-16T13:44:38.766670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:41:33.303177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.13","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.264385+0000","last_change":"2023-10-16T13:44:40.815297+0000","last_active":"2023-10-16T13:44:42.264385+0000","last_peered":"2023-10-16T13:44:42.264385+0000","last_clean":"2023-10-16T13:44:42.264385+0000","last_became_active":"2023-10-16T13:44:40.814931+0000","last_became_peered":"2023-10-16T13:44:40.814931+0000","last_unstale":"2023-10-16T13:44:42.264385+0000","last_undegraded":"2023-10-16T13:44:42.264385+0000","last_fullsized":"2023-10-16T13:44:42.264385+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:15:16.571866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.10","version":"205'8","reported_seq":20,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.264976+0000","last_change":"2023-10-16T13:44:39.790827+0000","last_active":"2023-10-16T13:44:42.264976+0000","last_peered":"2023-10-16T13:44:42.264976+0000","last_clean":"2023-10-16T13:44:42.264976+0000","last_became_active":"2023-10-16T13:44:39.790495+0000","last_became_peered":"2023-10-16T13:44:39.790495+0000","last_unstale":"2023-10-16T13:44:42.264976+0000","last_undegraded":"2023-10-16T13:44:42.264976+0000","last_fullsized":"2023-10-16T13:44:42.264976+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_clean_scrub_stamp":"2023-10-16T13:44:38.766670+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:55:38.777672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.802001+0000","last_change":"2023-10-16T13:44:40.813572+0000","last_active":"2023-10-16T13:44:41.802001+0000","last_peered":"2023-10-16T13:44:41.802001+0000","last_clean":"2023-10-16T13:44:41.802001+0000","last_became_active":"2023-10-16T13:44:40.813022+0000","last_became_peered":"2023-10-16T13:44:40.813022+0000","last_unstale":"2023-10-16T13:44:41.802001+0000","last_undegraded":"2023-10-16T13:44:41.802001+0000","last_fullsized":"2023-10-16T13:44:41.802001+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:49:43.725008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.f","version":"205'1","reported_seq":13,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.265266+0000","last_change":"2023-10-16T13:44:39.791453+0000","last_active":"2023-10-16T13:44:42.265266+0000","last_peered":"2023-10-16T13:44:42.265266+0000","last_clean":"2023-10-16T13:44:42.265266+0000","last_became_active":"2023-10-16T13:44:39.791134+0000","last_became_peered":"2023-10-16T13:44:39.791134+0000","last_unstale":"2023-10-16T13:44:42.265266+0000","last_undegraded":"2023-10-16T13:44:42.265266+0000","last_fullsized":"2023-10-16T13:44:42.265266+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_clean_scrub_stamp":"2023-10-16T13:44:38.766670+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:56:52.601574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.d","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.803535+0000","last_change":"2023-10-16T13:44:40.808177+0000","last_active":"2023-10-16T13:44:41.803535+0000","last_peered":"2023-10-16T13:44:41.803535+0000","last_clean":"2023-10-16T13:44:41.803535+0000","last_became_active":"2023-10-16T13:44:40.807851+0000","last_became_peered":"2023-10-16T13:44:40.807851+0000","last_unstale":"2023-10-16T13:44:41.803535+0000","last_undegraded":"2023-10-16T13:44:41.803535+0000","last_fullsized":"2023-10-16T13:44:41.803535+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:33:52.507520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.e","version":"205'2","reported_seq":14,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.264052+0000","last_change":"2023-10-16T13:44:39.792006+0000","last_active":"2023-10-16T13:44:42.264052+0000","last_peered":"2023-10-16T13:44:42.264052+0000","last_clean":"2023-10-16T13:44:42.264052+0000","last_became_active":"2023-10-16T13:44:39.791137+0000","last_became_peered":"2023-10-16T13:44:39.791137+0000","last_unstale":"2023-10-16T13:44:42.264052+0000","last_undegraded":"2023-10-16T13:44:42.264052+0000","last_fullsized":"2023-10-16T13:44:42.264052+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_clean_scrub_stamp":"2023-10-16T13:44:38.766670+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:19:16.883382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.e","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.265446+0000","last_change":"2023-10-16T13:44:40.808405+0000","last_active":"2023-10-16T13:44:42.265446+0000","last_peered":"2023-10-16T13:44:42.265446+0000","last_clean":"2023-10-16T13:44:42.265446+0000","last_became_active":"2023-10-16T13:44:40.808071+0000","last_became_peered":"2023-10-16T13:44:40.808071+0000","last_unstale":"2023-10-16T13:44:42.265446+0000","last_undegraded":"2023-10-16T13:44:42.265446+0000","last_fullsized":"2023-10-16T13:44:42.265446+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:17:19.764750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.d","version":"205'1","reported_seq":13,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.265344+0000","last_change":"2023-10-16T13:44:39.799230+0000","last_active":"2023-10-16T13:44:42.265344+0000","last_peered":"2023-10-16T13:44:42.265344+0000","last_clean":"2023-10-16T13:44:42.265344+0000","last_became_active":"2023-10-16T13:44:39.798885+0000","last_became_peered":"2023-10-16T13:44:39.798885+0000","last_unstale":"2023-10-16T13:44:42.265344+0000","last_undegraded":"2023-10-16T13:44:42.265344+0000","last_fullsized":"2023-10-16T13:44:42.265344+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_clean_scrub_stamp":"2023-10-16T13:44:38.766670+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:21:55.828297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.f","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.264603+0000","last_change":"2023-10-16T13:44:40.821116+0000","last_active":"2023-10-16T13:44:42.264603+0000","last_peered":"2023-10-16T13:44:42.264603+0000","last_clean":"2023-10-16T13:44:42.264603+0000","last_became_active":"2023-10-16T13:44:40.820753+0000","last_became_peered":"2023-10-16T13:44:40.820753+0000","last_unstale":"2023-10-16T13:44:42.264603+0000","last_undegraded":"2023-10-16T13:44:42.264603+0000","last_fullsized":"2023-10-16T13:44:42.264603+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:56:15.446259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":12,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.265619+0000","last_change":"2023-10-16T13:44:39.794797+0000","last_active":"2023-10-16T13:44:42.265619+0000","last_peered":"2023-10-16T13:44:42.265619+0000","last_clean":"2023-10-16T13:44:42.265619+0000","last_became_active":"2023-10-16T13:44:39.794494+0000","last_became_peered":"2023-10-16T13:44:39.794494+0000","last_unstale":"2023-10-16T13:44:42.265619+0000","last_undegraded":"2023-10-16T13:44:42.265619+0000","last_fullsized":"2023-10-16T13:44:42.265619+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_clean_scrub_stamp":"2023-10-16T13:44:38.766670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:21:12.180441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.263572+0000","last_change":"2023-10-16T13:44:40.809568+0000","last_active":"2023-10-16T13:44:42.263572+0000","last_peered":"2023-10-16T13:44:42.263572+0000","last_clean":"2023-10-16T13:44:42.263572+0000","last_became_active":"2023-10-16T13:44:40.809079+0000","last_became_peered":"2023-10-16T13:44:40.809079+0000","last_unstale":"2023-10-16T13:44:42.263572+0000","last_undegraded":"2023-10-16T13:44:42.263572+0000","last_fullsized":"2023-10-16T13:44:42.263572+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:56:53.605810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.b","version":"205'2","reported_seq":14,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.264616+0000","last_change":"2023-10-16T13:44:39.796244+0000","last_active":"2023-10-16T13:44:42.264616+0000","last_peered":"2023-10-16T13:44:42.264616+0000","last_clean":"2023-10-16T13:44:42.264616+0000","last_became_active":"2023-10-16T13:44:39.795941+0000","last_became_peered":"2023-10-16T13:44:39.795941+0000","last_unstale":"2023-10-16T13:44:42.264616+0000","last_undegraded":"2023-10-16T13:44:42.264616+0000","last_fullsized":"2023-10-16T13:44:42.264616+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_clean_scrub_stamp":"2023-10-16T13:44:38.766670+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:37:09.220924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.264368+0000","last_change":"2023-10-16T13:44:40.819695+0000","last_active":"2023-10-16T13:44:42.264368+0000","last_peered":"2023-10-16T13:44:42.264368+0000","last_clean":"2023-10-16T13:44:42.264368+0000","last_became_active":"2023-10-16T13:44:40.819397+0000","last_became_peered":"2023-10-16T13:44:40.819397+0000","last_unstale":"2023-10-16T13:44:42.264368+0000","last_undegraded":"2023-10-16T13:44:42.264368+0000","last_fullsized":"2023-10-16T13:44:42.264368+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:38:47.872141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":12,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.264028+0000","last_change":"2023-10-16T13:44:39.790396+0000","last_active":"2023-10-16T13:44:42.264028+0000","last_peered":"2023-10-16T13:44:42.264028+0000","last_clean":"2023-10-16T13:44:42.264028+0000","last_became_active":"2023-10-16T13:44:39.789911+0000","last_became_peered":"2023-10-16T13:44:39.789911+0000","last_unstale":"2023-10-16T13:44:42.264028+0000","last_undegraded":"2023-10-16T13:44:42.264028+0000","last_fullsized":"2023-10-16T13:44:42.264028+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_clean_scrub_stamp":"2023-10-16T13:44:38.766670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:28:24.113684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.263861+0000","last_change":"2023-10-16T13:44:40.819184+0000","last_active":"2023-10-16T13:44:42.263861+0000","last_peered":"2023-10-16T13:44:42.263861+0000","last_clean":"2023-10-16T13:44:42.263861+0000","last_became_active":"2023-10-16T13:44:40.818881+0000","last_became_peered":"2023-10-16T13:44:40.818881+0000","last_unstale":"2023-10-16T13:44:42.263861+0000","last_undegraded":"2023-10-16T13:44:42.263861+0000","last_fullsized":"2023-10-16T13:44:42.263861+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:05:44.245583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":12,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.801698+0000","last_change":"2023-10-16T13:44:39.795496+0000","last_active":"2023-10-16T13:44:41.801698+0000","last_peered":"2023-10-16T13:44:41.801698+0000","last_clean":"2023-10-16T13:44:41.801698+0000","last_became_active":"2023-10-16T13:44:39.795210+0000","last_became_peered":"2023-10-16T13:44:39.795210+0000","last_unstale":"2023-10-16T13:44:41.801698+0000","last_undegraded":"2023-10-16T13:44:41.801698+0000","last_fullsized":"2023-10-16T13:44:41.801698+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_clean_scrub_stamp":"2023-10-16T13:44:38.766670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:59:20.223638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.b","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.803322+0000","last_change":"2023-10-16T13:44:40.810076+0000","last_active":"2023-10-16T13:44:41.803322+0000","last_peered":"2023-10-16T13:44:41.803322+0000","last_clean":"2023-10-16T13:44:41.803322+0000","last_became_active":"2023-10-16T13:44:40.809585+0000","last_became_peered":"2023-10-16T13:44:40.809585+0000","last_unstale":"2023-10-16T13:44:41.803322+0000","last_undegraded":"2023-10-16T13:44:41.803322+0000","last_fullsized":"2023-10-16T13:44:41.803322+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:40:19.863783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":12,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.265496+0000","last_change":"2023-10-16T13:44:39.788432+0000","last_active":"2023-10-16T13:44:42.265496+0000","last_peered":"2023-10-16T13:44:42.265496+0000","last_clean":"2023-10-16T13:44:42.265496+0000","last_became_active":"2023-10-16T13:44:39.787791+0000","last_became_peered":"2023-10-16T13:44:39.787791+0000","last_unstale":"2023-10-16T13:44:42.265496+0000","last_undegraded":"2023-10-16T13:44:42.265496+0000","last_fullsized":"2023-10-16T13:44:42.265496+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_clean_scrub_stamp":"2023-10-16T13:44:38.766670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:12:45.268611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.802147+0000","last_change":"2023-10-16T13:44:40.813865+0000","last_active":"2023-10-16T13:44:41.802147+0000","last_peered":"2023-10-16T13:44:41.802147+0000","last_clean":"2023-10-16T13:44:41.802147+0000","last_became_active":"2023-10-16T13:44:40.813532+0000","last_became_peered":"2023-10-16T13:44:40.813532+0000","last_unstale":"2023-10-16T13:44:41.802147+0000","last_undegraded":"2023-10-16T13:44:41.802147+0000","last_fullsized":"2023-10-16T13:44:41.802147+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:32:39.343118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.7","version":"205'2","reported_seq":14,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.264210+0000","last_change":"2023-10-16T13:44:39.789533+0000","last_active":"2023-10-16T13:44:42.264210+0000","last_peered":"2023-10-16T13:44:42.264210+0000","last_clean":"2023-10-16T13:44:42.264210+0000","last_became_active":"2023-10-16T13:44:39.789225+0000","last_became_peered":"2023-10-16T13:44:39.789225+0000","last_unstale":"2023-10-16T13:44:42.264210+0000","last_undegraded":"2023-10-16T13:44:42.264210+0000","last_fullsized":"2023-10-16T13:44:42.264210+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_clean_scrub_stamp":"2023-10-16T13:44:38.766670+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:07:10.730051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.264118+0000","last_change":"2023-10-16T13:44:40.813440+0000","last_active":"2023-10-16T13:44:42.264118+0000","last_peered":"2023-10-16T13:44:42.264118+0000","last_clean":"2023-10-16T13:44:42.264118+0000","last_became_active":"2023-10-16T13:44:40.813147+0000","last_became_peered":"2023-10-16T13:44:40.813147+0000","last_unstale":"2023-10-16T13:44:42.264118+0000","last_undegraded":"2023-10-16T13:44:42.264118+0000","last_fullsized":"2023-10-16T13:44:42.264118+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:17:19.764750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.6","version":"0'0","reported_seq":12,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.802399+0000","last_change":"2023-10-16T13:44:39.795267+0000","last_active":"2023-10-16T13:44:41.802399+0000","last_peered":"2023-10-16T13:44:41.802399+0000","last_clean":"2023-10-16T13:44:41.802399+0000","last_became_active":"2023-10-16T13:44:39.794950+0000","last_became_peered":"2023-10-16T13:44:39.794950+0000","last_unstale":"2023-10-16T13:44:41.802399+0000","last_undegraded":"2023-10-16T13:44:41.802399+0000","last_fullsized":"2023-10-16T13:44:41.802399+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_clean_scrub_stamp":"2023-10-16T13:44:38.766670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:58:00.704123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.6","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.899473+0000","last_change":"2023-10-16T13:44:40.827414+0000","last_active":"2023-10-16T13:44:41.899473+0000","last_peered":"2023-10-16T13:44:41.899473+0000","last_clean":"2023-10-16T13:44:41.899473+0000","last_became_active":"2023-10-16T13:44:40.826996+0000","last_became_peered":"2023-10-16T13:44:40.826996+0000","last_unstale":"2023-10-16T13:44:41.899473+0000","last_undegraded":"2023-10-16T13:44:41.899473+0000","last_fullsized":"2023-10-16T13:44:41.899473+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:33:07.487629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":12,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.803591+0000","last_change":"2023-10-16T13:44:39.799361+0000","last_active":"2023-10-16T13:44:41.803591+0000","last_peered":"2023-10-16T13:44:41.803591+0000","last_clean":"2023-10-16T13:44:41.803591+0000","last_became_active":"2023-10-16T13:44:39.799109+0000","last_became_peered":"2023-10-16T13:44:39.799109+0000","last_unstale":"2023-10-16T13:44:41.803591+0000","last_undegraded":"2023-10-16T13:44:41.803591+0000","last_fullsized":"2023-10-16T13:44:41.803591+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_clean_scrub_stamp":"2023-10-16T13:44:38.766670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:03:57.432396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.7","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.802590+0000","last_change":"2023-10-16T13:44:40.813784+0000","last_active":"2023-10-16T13:44:41.802590+0000","last_peered":"2023-10-16T13:44:41.802590+0000","last_clean":"2023-10-16T13:44:41.802590+0000","last_became_active":"2023-10-16T13:44:40.813441+0000","last_became_peered":"2023-10-16T13:44:40.813441+0000","last_unstale":"2023-10-16T13:44:41.802590+0000","last_undegraded":"2023-10-16T13:44:41.802590+0000","last_fullsized":"2023-10-16T13:44:41.802590+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:23:06.700497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":12,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.803430+0000","last_change":"2023-10-16T13:44:39.796004+0000","last_active":"2023-10-16T13:44:41.803430+0000","last_peered":"2023-10-16T13:44:41.803430+0000","last_clean":"2023-10-16T13:44:41.803430+0000","last_became_active":"2023-10-16T13:44:39.795702+0000","last_became_peered":"2023-10-16T13:44:39.795702+0000","last_unstale":"2023-10-16T13:44:41.803430+0000","last_undegraded":"2023-10-16T13:44:41.803430+0000","last_fullsized":"2023-10-16T13:44:41.803430+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_clean_scrub_stamp":"2023-10-16T13:44:38.766670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:34:04.095991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.263936+0000","last_change":"2023-10-16T13:44:40.822780+0000","last_active":"2023-10-16T13:44:42.263936+0000","last_peered":"2023-10-16T13:44:42.263936+0000","last_clean":"2023-10-16T13:44:42.263936+0000","last_became_active":"2023-10-16T13:44:40.822388+0000","last_became_peered":"2023-10-16T13:44:40.822388+0000","last_unstale":"2023-10-16T13:44:42.263936+0000","last_undegraded":"2023-10-16T13:44:42.263936+0000","last_fullsized":"2023-10-16T13:44:42.263936+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:29:27.837805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":12,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.899182+0000","last_change":"2023-10-16T13:44:39.794644+0000","last_active":"2023-10-16T13:44:41.899182+0000","last_peered":"2023-10-16T13:44:41.899182+0000","last_clean":"2023-10-16T13:44:41.899182+0000","last_became_active":"2023-10-16T13:44:39.794272+0000","last_became_peered":"2023-10-16T13:44:39.794272+0000","last_unstale":"2023-10-16T13:44:41.899182+0000","last_undegraded":"2023-10-16T13:44:41.899182+0000","last_fullsized":"2023-10-16T13:44:41.899182+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_clean_scrub_stamp":"2023-10-16T13:44:38.766670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:07:10.730051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.264583+0000","last_change":"2023-10-16T13:44:40.815288+0000","last_active":"2023-10-16T13:44:42.264583+0000","last_peered":"2023-10-16T13:44:42.264583+0000","last_clean":"2023-10-16T13:44:42.264583+0000","last_became_active":"2023-10-16T13:44:40.814982+0000","last_became_peered":"2023-10-16T13:44:40.814982+0000","last_unstale":"2023-10-16T13:44:42.264583+0000","last_undegraded":"2023-10-16T13:44:42.264583+0000","last_fullsized":"2023-10-16T13:44:42.264583+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:33:19.386232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":12,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.899057+0000","last_change":"2023-10-16T13:44:39.799090+0000","last_active":"2023-10-16T13:44:41.899057+0000","last_peered":"2023-10-16T13:44:41.899057+0000","last_clean":"2023-10-16T13:44:41.899057+0000","last_became_active":"2023-10-16T13:44:39.798655+0000","last_became_peered":"2023-10-16T13:44:39.798655+0000","last_unstale":"2023-10-16T13:44:41.899057+0000","last_undegraded":"2023-10-16T13:44:41.899057+0000","last_fullsized":"2023-10-16T13:44:41.899057+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_clean_scrub_stamp":"2023-10-16T13:44:38.766670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:39:56.910883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.264176+0000","last_change":"2023-10-16T13:44:40.821533+0000","last_active":"2023-10-16T13:44:42.264176+0000","last_peered":"2023-10-16T13:44:42.264176+0000","last_clean":"2023-10-16T13:44:42.264176+0000","last_became_active":"2023-10-16T13:44:40.820856+0000","last_became_peered":"2023-10-16T13:44:40.820856+0000","last_unstale":"2023-10-16T13:44:42.264176+0000","last_undegraded":"2023-10-16T13:44:42.264176+0000","last_fullsized":"2023-10-16T13:44:42.264176+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:53:07.234210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":12,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.801888+0000","last_change":"2023-10-16T13:44:39.791606+0000","last_active":"2023-10-16T13:44:41.801888+0000","last_peered":"2023-10-16T13:44:41.801888+0000","last_clean":"2023-10-16T13:44:41.801888+0000","last_became_active":"2023-10-16T13:44:39.791258+0000","last_became_peered":"2023-10-16T13:44:39.791258+0000","last_unstale":"2023-10-16T13:44:41.801888+0000","last_undegraded":"2023-10-16T13:44:41.801888+0000","last_fullsized":"2023-10-16T13:44:41.801888+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_clean_scrub_stamp":"2023-10-16T13:44:38.766670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:29:27.824587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.899210+0000","last_change":"2023-10-16T13:44:40.830044+0000","last_active":"2023-10-16T13:44:41.899210+0000","last_peered":"2023-10-16T13:44:41.899210+0000","last_clean":"2023-10-16T13:44:41.899210+0000","last_became_active":"2023-10-16T13:44:40.829735+0000","last_became_peered":"2023-10-16T13:44:40.829735+0000","last_unstale":"2023-10-16T13:44:41.899210+0000","last_undegraded":"2023-10-16T13:44:41.899210+0000","last_fullsized":"2023-10-16T13:44:41.899210+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:42:50.455035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":12,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.265571+0000","last_change":"2023-10-16T13:44:39.791696+0000","last_active":"2023-10-16T13:44:42.265571+0000","last_peered":"2023-10-16T13:44:42.265571+0000","last_clean":"2023-10-16T13:44:42.265571+0000","last_became_active":"2023-10-16T13:44:39.791371+0000","last_became_peered":"2023-10-16T13:44:39.791371+0000","last_unstale":"2023-10-16T13:44:42.265571+0000","last_undegraded":"2023-10-16T13:44:42.265571+0000","last_fullsized":"2023-10-16T13:44:42.265571+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_clean_scrub_stamp":"2023-10-16T13:44:38.766670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:24:22.879467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.1b","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.265483+0000","last_change":"2023-10-16T13:44:40.823583+0000","last_active":"2023-10-16T13:44:42.265483+0000","last_peered":"2023-10-16T13:44:42.265483+0000","last_clean":"2023-10-16T13:44:42.265483+0000","last_became_active":"2023-10-16T13:44:40.823308+0000","last_became_peered":"2023-10-16T13:44:40.823308+0000","last_unstale":"2023-10-16T13:44:42.265483+0000","last_undegraded":"2023-10-16T13:44:42.265483+0000","last_fullsized":"2023-10-16T13:44:42.265483+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:15:16.571866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":12,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.263814+0000","last_change":"2023-10-16T13:44:39.787527+0000","last_active":"2023-10-16T13:44:42.263814+0000","last_peered":"2023-10-16T13:44:42.263814+0000","last_clean":"2023-10-16T13:44:42.263814+0000","last_became_active":"2023-10-16T13:44:39.787221+0000","last_became_peered":"2023-10-16T13:44:39.787221+0000","last_unstale":"2023-10-16T13:44:42.263814+0000","last_undegraded":"2023-10-16T13:44:42.263814+0000","last_fullsized":"2023-10-16T13:44:42.263814+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_clean_scrub_stamp":"2023-10-16T13:44:38.766670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:39:56.910883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"196'326","reported_seq":1107,"reported_epoch":205,"state":"active+clean","last_fresh":"2023-10-16T13:44:40.821237+0000","last_change":"2023-10-16T12:10:34.746820+0000","last_active":"2023-10-16T13:44:40.821237+0000","last_peered":"2023-10-16T13:44:40.821237+0000","last_clean":"2023-10-16T13:44:40.821237+0000","last_became_active":"2023-10-16T12:10:34.745574+0000","last_became_peered":"2023-10-16T12:10:34.745574+0000","last_unstale":"2023-10-16T13:44:40.821237+0000","last_undegraded":"2023-10-16T13:44:40.821237+0000","last_fullsized":"2023-10-16T13:44:40.821237+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_clean_scrub_stamp":"2023-10-16T12:10:33.577111+0000","objects_scrubbed":0,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:14.461986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1940,"num_read_kb":4014,"num_write":539,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":26,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":25,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1940,"num_read_kb":4014,"num_write":539,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":326,"ondisk_log_size":326,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575967,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56480,"kb_used_data":3092,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315360,"statfs":{"total":96636764160,"available":96578928640,"internally_reserved":0,"allocated":3166208,"data_stored":1906722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54655435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:43:42 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.63400000000000001,"15min":0.64200000000000002},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.33000000000000002},"max":{"1min":0.87,"5min":0.90500000000000003,"15min":0.96199999999999997},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.61499999999999999,"15min":0.63600000000000001},"min":{"1min":0.51500000000000001,"5min":0.41099999999999998,"15min":0.379},"max":{"1min":0.71999999999999997,"5min":0.82999999999999996,"15min":0.91200000000000003},"last":1.1879999999999999}]},{"osd":1,"last update":"Mon Oct 16 13:43:42 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63200000000000001,"15min":0.64900000000000002},"min":{"1min":0.45000000000000001,"5min":0.39600000000000002,"15min":0.375},"max":{"1min":0.80400000000000005,"5min":0.88700000000000001,"15min":0.95899999999999996},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.64500000000000002,"15min":0.64300000000000002},"min":{"1min":0.55600000000000005,"5min":0.42099999999999999,"15min":0.40400000000000003},"max":{"1min":0.77600000000000002,"5min":0.83599999999999997,"15min":1.022},"last":0.44400000000000001}]},{"osd":2,"last update":"Mon Oct 16 13:43:42 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.69499999999999995,"15min":0.69799999999999995},"min":{"1min":0.57699999999999996,"5min":0.51800000000000002,"15min":0.44900000000000001},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.93500000000000005},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.70199999999999996,"15min":0.70699999999999996},"min":{"1min":0.56799999999999995,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.84499999999999997,"5min":0.92800000000000005,"15min":0.94199999999999995},"last":0.50900000000000001}]},{"osd":3,"last update":"Mon Oct 16 13:43:42 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.66300000000000003,"15min":0.66800000000000004},"min":{"1min":0.53600000000000003,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.82299999999999995,"5min":0.89200000000000002,"15min":0.91300000000000003},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.65700000000000003,"15min":0.66500000000000004},"min":{"1min":0.53500000000000003,"5min":0.439,"15min":0.40100000000000002},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.98299999999999998},"last":1.24}]},{"osd":4,"last update":"Mon Oct 16 13:43:42 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.51300000000000001,"15min":0.52800000000000002},"min":{"1min":0.38100000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.66900000000000004,"5min":0.77900000000000003,"15min":0.94599999999999995},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.624,"5min":0.60199999999999998,"15min":0.623},"min":{"1min":0.49199999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.92900000000000005},"last":0.59099999999999997}]},{"osd":5,"last update":"Mon Oct 16 13:43:42 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.53100000000000003,"15min":0.54300000000000004},"min":{"1min":0.42599999999999999,"5min":0.35999999999999999,"15min":0.32500000000000001},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.90000000000000002},"last":1.2829999999999999},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.57499999999999996,"15min":0.59099999999999997},"min":{"1min":0.45500000000000002,"5min":0.38400000000000001,"15min":0.35099999999999998},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.65400000000000003}]},{"osd":6,"last update":"Mon Oct 16 13:43:42 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.59399999999999997,"15min":0.621},"min":{"1min":0.46300000000000002,"5min":0.40500000000000003,"15min":0.378},"max":{"1min":0.874,"5min":0.874,"15min":0.95899999999999996},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.59999999999999998,"15min":0.61899999999999999},"min":{"1min":0.44800000000000001,"5min":0.38700000000000001,"15min":0.35199999999999998},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.97499999999999998},"last":0.63900000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575965,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42016,"kb_used_data":2516,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329824,"statfs":{"total":96636764160,"available":96593739776,"internally_reserved":0,"allocated":2576384,"data_stored":1316424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:44:33 2023","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.67600000000000005,"15min":0.67900000000000005},"min":{"1min":0.39000000000000001,"5min":0.375,"15min":0.28699999999999998},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.93600000000000005},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.66500000000000004,"15min":0.67300000000000004},"min":{"1min":0.52200000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.85499999999999998,"5min":0.88600000000000001,"15min":0.93799999999999994},"last":0.61499999999999999}]},{"osd":1,"last update":"Mon Oct 16 13:44:33 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.629,"15min":0.63800000000000001},"min":{"1min":0.41399999999999998,"5min":0.36599999999999999,"15min":0.33100000000000002},"max":{"1min":0.81399999999999995,"5min":0.85999999999999999,"15min":0.90900000000000003},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.61199999999999999,"15min":0.625},"min":{"1min":0.437,"5min":0.38500000000000001,"15min":0.38300000000000001},"max":{"1min":0.88500000000000001,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.55300000000000005}]},{"osd":2,"last update":"Mon Oct 16 13:44:33 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.622,"15min":0.63700000000000001},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.378},"max":{"1min":0.878,"5min":0.878,"15min":0.879},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.624,"15min":0.63500000000000001},"min":{"1min":0.45400000000000001,"5min":0.38700000000000001,"15min":0.379},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.91000000000000003},"last":0.69799999999999995}]},{"osd":3,"last update":"Mon Oct 16 13:44:33 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66800000000000004,"15min":0.67200000000000004},"min":{"1min":0.496,"5min":0.39900000000000002,"15min":0.38200000000000001},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.998},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.71399999999999997,"15min":0.72599999999999998},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.42299999999999999},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96899999999999997},"last":0.72099999999999997}]},{"osd":4,"last update":"Mon Oct 16 13:44:33 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.52600000000000002,"15min":0.52700000000000002},"min":{"1min":0.439,"5min":0.32600000000000001,"15min":0.315},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.95799999999999996},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.624,"15min":0.621},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.32900000000000001},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":1.0309999999999999},"last":0.75800000000000001}]},{"osd":5,"last update":"Mon Oct 16 13:44:33 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.59999999999999998,"15min":0.59199999999999997},"min":{"1min":0.46999999999999997,"5min":0.39400000000000002,"15min":0.34899999999999998},"max":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.67000000000000004,"15min":0.66100000000000003},"min":{"1min":0.40600000000000003,"5min":0.371,"15min":0.30599999999999999},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.77500000000000002}]},{"osd":7,"last update":"Mon Oct 16 13:44:33 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.61199999999999999,"15min":0.60499999999999998},"min":{"1min":0.41699999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.081,"5min":1.081,"15min":1.081},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.66200000000000003,"15min":0.65100000000000002},"min":{"1min":0.443,"5min":0.379,"15min":0.375},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.79700000000000004}]}]},{"osd":1,"up_from":13,"seq":55834575965,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47580,"kb_used_data":2512,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324260,"statfs":{"total":96636764160,"available":96588042240,"internally_reserved":0,"allocated":2572288,"data_stored":1316354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":46135753},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:43:51 2023","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.41099999999999998,"15min":0.41199999999999998},"min":{"1min":0.28599999999999998,"5min":0.251,"15min":0.251},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.71999999999999997},"last":0.436},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.496,"15min":0.498},"min":{"1min":0.42299999999999999,"5min":0.318,"15min":0.28799999999999998},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.77200000000000002},"last":0.45500000000000002}]},{"osd":2,"last update":"Mon Oct 16 13:43:51 2023","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.45900000000000002,"15min":0.45700000000000002},"min":{"1min":0.29099999999999998,"5min":0.28799999999999998,"15min":0.26600000000000001},"max":{"1min":0.63300000000000001,"5min":0.67200000000000004,"15min":0.751},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.48699999999999999,"15min":0.495},"min":{"1min":0.317,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.66600000000000004,"5min":0.68700000000000006,"15min":0.71399999999999997},"last":0.498}]},{"osd":3,"last update":"Mon Oct 16 13:43:51 2023","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.435,"15min":0.42899999999999999},"min":{"1min":0.34300000000000003,"5min":0.315,"15min":0.28799999999999998},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.77500000000000002},"last":0.373},{"interface":"front","average":{"1min":0.502,"5min":0.47999999999999998,"15min":0.48299999999999998},"min":{"1min":0.32400000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.39400000000000002}]},{"osd":4,"last update":"Mon Oct 16 13:43:51 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.67600000000000005,"15min":0.67000000000000004},"min":{"1min":0.48799999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.94499999999999995,"5min":0.95899999999999996,"15min":1.0069999999999999},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.67600000000000005,"15min":0.67500000000000004},"min":{"1min":0.46899999999999997,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.91300000000000003,"5min":0.92800000000000005,"15min":0.997},"last":0.72799999999999998}]},{"osd":5,"last update":"Mon Oct 16 13:43:51 2023","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.71799999999999997,"15min":0.71799999999999997},"min":{"1min":0.54200000000000004,"5min":0.53200000000000003,"15min":0.41199999999999998},"max":{"1min":0.93300000000000005,"5min":0.97499999999999998,"15min":1.0760000000000001},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.746,"15min":0.73499999999999999},"min":{"1min":0.55900000000000005,"5min":0.48999999999999999,"15min":0.46200000000000002},"max":{"1min":1.028,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.76000000000000001}]},{"osd":6,"last update":"Mon Oct 16 13:43:51 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.69199999999999995,"15min":0.68700000000000006},"min":{"1min":0.56299999999999994,"5min":0.432,"15min":0.42099999999999999},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.751,"5min":0.71699999999999997,"15min":0.71499999999999997},"min":{"1min":0.57999999999999996,"5min":0.47799999999999998,"15min":0.46899999999999997},"max":{"1min":0.998,"5min":1.03,"15min":1.083},"last":0.71599999999999997}]},{"osd":7,"last update":"Mon Oct 16 13:43:51 2023","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.755,"15min":0.747},"min":{"1min":0.55500000000000005,"5min":0.497,"15min":0.47399999999999998},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0660000000000001},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.76000000000000001,"15min":0.753},"min":{"1min":0.54700000000000004,"5min":0.505,"15min":0.44600000000000001},"max":{"1min":1.042,"5min":1.042,"15min":1.0580000000000001},"last":0.752}]}]},{"osd":0,"up_from":13,"seq":55834575966,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30132,"kb_used_data":3108,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341708,"statfs":{"total":96636764160,"available":96605908992,"internally_reserved":0,"allocated":3182592,"data_stored":1907450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 13:44:39 2023","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.41599999999999998,"15min":0.42499999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.51600000000000001,"5min":0.66400000000000003,"15min":0.71399999999999997},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.438,"15min":0.44600000000000001},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.25900000000000001},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.82199999999999995},"last":0.30099999999999999}]},{"osd":2,"last update":"Mon Oct 16 13:44:39 2023","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.44800000000000001,"15min":0.45900000000000002},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.56699999999999995,"5min":0.64200000000000002,"15min":0.75700000000000001},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.435,"5min":0.48299999999999998,"15min":0.48999999999999999},"min":{"1min":0.33700000000000002,"5min":0.33100000000000002,"15min":0.32600000000000001},"max":{"1min":0.60099999999999998,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.34699999999999998}]},{"osd":3,"last update":"Mon Oct 16 13:44:39 2023","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.44700000000000001,"15min":0.45700000000000002},"min":{"1min":0.29899999999999999,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.58299999999999996,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.373},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.52400000000000002,"15min":0.52900000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.69399999999999995,"5min":0.82999999999999996,"15min":0.86399999999999999},"last":0.53000000000000003}]},{"osd":4,"last update":"Mon Oct 16 13:44:39 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.67400000000000004,"15min":0.68700000000000006},"min":{"1min":0.48199999999999998,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.87,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.625,"5min":0.66900000000000004,"15min":0.69399999999999995},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40200000000000002},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":1.0049999999999999},"last":0.54800000000000004}]},{"osd":5,"last update":"Mon Oct 16 13:44:39 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.68200000000000005,"15min":0.69399999999999995},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.84199999999999997,"5min":0.95999999999999996,"15min":1.0249999999999999},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.70899999999999996,"15min":0.73099999999999998},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.92100000000000004,"5min":0.998,"15min":1.0720000000000001},"last":0.59099999999999997}]},{"osd":6,"last update":"Mon Oct 16 13:44:39 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.72399999999999998,"15min":0.73699999999999999},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.879,"5min":1.103,"15min":1.103},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.71299999999999997,"15min":0.72799999999999998},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.93899999999999995,"5min":1.0409999999999999,"15min":1.05},"last":0.42999999999999999}]},{"osd":7,"last update":"Mon Oct 16 13:44:39 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.73599999999999999,"15min":0.75},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.91100000000000003,"5min":0.98299999999999998,"15min":1.056},"last":0.498},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.72399999999999998,"15min":0.73799999999999999},"min":{"1min":0.49299999999999999,"5min":0.48699999999999999,"15min":0.45400000000000001},"max":{"1min":0.89500000000000002,"5min":1.087,"15min":1.087},"last":0.51500000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575966,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51764,"kb_used_data":2528,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320076,"statfs":{"total":96636764160,"available":96583757824,"internally_reserved":0,"allocated":2588672,"data_stored":1323684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:44:35 2023","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.42999999999999999,"15min":0.41499999999999998},"min":{"1min":0.27700000000000002,"5min":0.251,"15min":0.24099999999999999},"max":{"1min":0.61499999999999999,"5min":0.71699999999999997,"15min":0.77500000000000002},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.495,"15min":0.48499999999999999},"min":{"1min":0.36799999999999999,"5min":0.35999999999999999,"15min":0.29599999999999999},"max":{"1min":0.77500000000000002,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.32400000000000001}]},{"osd":1,"last update":"Mon Oct 16 13:44:35 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.51900000000000002,"15min":0.50700000000000001},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.69999999999999996,"5min":0.73199999999999998,"15min":0.81399999999999995},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.503,"15min":0.48199999999999998},"min":{"1min":0.33700000000000002,"5min":0.29899999999999999,"15min":0.27700000000000002},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.35399999999999998}]},{"osd":3,"last update":"Mon Oct 16 13:44:35 2023","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.45600000000000002,"15min":0.44600000000000001},"min":{"1min":0.31,"5min":0.29099999999999998,"15min":0.28399999999999997},"max":{"1min":0.58499999999999996,"5min":0.70099999999999996,"15min":0.86699999999999999},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.56999999999999995,"15min":0.55200000000000005},"min":{"1min":0.40500000000000003,"5min":0.371,"15min":0.309},"max":{"1min":0.74199999999999999,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.39100000000000001}]},{"osd":4,"last update":"Mon Oct 16 13:44:35 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.748,"15min":0.72899999999999998},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.91300000000000003,"5min":1.3600000000000001,"15min":1.3600000000000001},"last":0.625},{"interface":"front","average":{"1min":0.751,"5min":0.75700000000000001,"15min":0.73399999999999999},"min":{"1min":0.47799999999999998,"5min":0.45700000000000002,"15min":0.44900000000000001},"max":{"1min":1.3340000000000001,"5min":1.403,"15min":1.403},"last":0.61399999999999999}]},{"osd":5,"last update":"Mon Oct 16 13:44:35 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.72599999999999998,"15min":0.70099999999999996},"min":{"1min":0.505,"5min":0.505,"15min":0.441},"max":{"1min":0.87,"5min":1.486,"15min":1.486},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.73799999999999999,"15min":0.71199999999999997},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":1.3660000000000001,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":0.66500000000000004}]},{"osd":6,"last update":"Mon Oct 16 13:44:35 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.70699999999999996,"15min":0.67700000000000005},"min":{"1min":0.50700000000000001,"5min":0.48999999999999999,"15min":0.46800000000000003},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.755,"15min":0.73599999999999999},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.88600000000000001,"5min":1.4379999999999999,"15min":1.4379999999999999},"last":0.64800000000000002}]},{"osd":7,"last update":"Mon Oct 16 13:44:35 2023","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.76300000000000001,"15min":0.73199999999999998},"min":{"1min":0.53400000000000003,"5min":0.52000000000000002,"15min":0.47899999999999998},"max":{"1min":1.4370000000000001,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.76400000000000001,"15min":0.748},"min":{"1min":0.54600000000000004,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.94499999999999995,"5min":1.377,"15min":1.377},"last":0.56399999999999995}]}]},{"osd":3,"up_from":13,"seq":55834575965,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51760,"kb_used_data":2532,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320080,"statfs":{"total":96636764160,"available":96583761920,"internally_reserved":0,"allocated":2592768,"data_stored":1323262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50395592},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:44:38 2023","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.40899999999999997,"15min":0.41599999999999998},"min":{"1min":0.25,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.58399999999999996,"5min":0.64200000000000002,"15min":0.77700000000000002},"last":0.436},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.42999999999999999,"15min":0.435},"min":{"1min":0.27400000000000002,"5min":0.252,"15min":0.252},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.90100000000000002},"last":0.504}]},{"osd":1,"last update":"Mon Oct 16 13:44:38 2023","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.439,"15min":0.441},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.66500000000000004,"5min":0.70499999999999996,"15min":0.76300000000000001},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.47499999999999998,"15min":0.48399999999999999},"min":{"1min":0.36899999999999999,"5min":0.253,"15min":0.253},"max":{"1min":0.58899999999999997,"5min":0.71499999999999997,"15min":0.77800000000000002},"last":0.58299999999999996}]},{"osd":2,"last update":"Mon Oct 16 13:44:38 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.501,"15min":0.5},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":1.167,"5min":1.167,"15min":1.167},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.52400000000000002,"15min":0.53000000000000003},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.65500000000000003,"5min":0.76800000000000002,"15min":0.78700000000000003},"last":0.67600000000000005}]},{"osd":4,"last update":"Mon Oct 16 13:44:38 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.66300000000000003,"15min":0.67400000000000004},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.85999999999999999,"5min":1.137,"15min":1.137},"last":0.98499999999999999},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.70499999999999996,"15min":0.70099999999999996},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46100000000000002},"max":{"1min":1.2110000000000001,"5min":1.2909999999999999,"15min":1.2909999999999999},"last":0.72899999999999998}]},{"osd":5,"last update":"Mon Oct 16 13:44:38 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.71299999999999997,"15min":0.71599999999999997},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.374},"max":{"1min":0.91300000000000003,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.71399999999999997,"15min":0.70499999999999996},"min":{"1min":0.46100000000000002,"5min":0.432,"15min":0.432},"max":{"1min":0.81200000000000006,"5min":1.173,"15min":1.173},"last":1.0549999999999999}]},{"osd":6,"last update":"Mon Oct 16 13:44:38 2023","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.755,"15min":0.72899999999999998},"min":{"1min":0.51000000000000001,"5min":0.47099999999999997,"15min":0.41799999999999998},"max":{"1min":5.0999999999999996,"5min":5.0999999999999996,"15min":5.0999999999999996},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.73499999999999999,"15min":0.73799999999999999},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.871,"5min":1.2290000000000001,"15min":1.2290000000000001},"last":1.0309999999999999}]},{"osd":7,"last update":"Mon Oct 16 13:44:38 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.73399999999999999,"15min":0.73999999999999999},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.85699999999999998,"5min":1.2569999999999999,"15min":1.2569999999999999},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.75900000000000001,"15min":0.749},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.43099999999999999},"max":{"1min":1.262,"5min":1.262,"15min":1.262},"last":0.95999999999999996}]}]},{"osd":4,"up_from":13,"seq":55834575964,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51760,"kb_used_data":2528,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320080,"statfs":{"total":96636764160,"available":96583761920,"internally_reserved":0,"allocated":2588672,"data_stored":1318000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.628,"15min":0.627},"min":{"1min":0.37,"5min":0.36299999999999999,"15min":0.33800000000000002},"max":{"1min":0.72999999999999998,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.64400000000000002,"15min":0.63800000000000001},"min":{"1min":0.46400000000000002,"5min":0.41499999999999998,"15min":0.35499999999999998},"max":{"1min":0.81499999999999995,"5min":0.878,"15min":0.95599999999999996},"last":0.57199999999999995}]},{"osd":1,"last update":"Mon Oct 16 13:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.65500000000000003,"15min":0.66000000000000003},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.375},"max":{"1min":0.83099999999999996,"5min":0.85099999999999998,"15min":0.89300000000000002},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.64600000000000002,"15min":0.64200000000000002},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.79200000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.63800000000000001}]},{"osd":2,"last update":"Mon Oct 16 13:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.67300000000000004,"15min":0.65600000000000003},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.39600000000000002},"max":{"1min":0.88500000000000001,"5min":0.89400000000000002,"15min":0.90600000000000003},"last":0.622},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66000000000000003},"min":{"1min":0.40799999999999997,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.92600000000000005},"last":0.73099999999999998}]},{"osd":3,"last update":"Mon Oct 16 13:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.39500000000000002},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.99199999999999999},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.68700000000000006,"15min":0.68000000000000005},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.83699999999999997,"5min":0.90700000000000003,"15min":0.97399999999999998},"last":0.67300000000000004}]},{"osd":5,"last update":"Mon Oct 16 13:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.37,"5min":0.34999999999999998,"15min":0.30599999999999999},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.91700000000000004},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.55700000000000005,"15min":0.56599999999999995},"min":{"1min":0.377,"5min":0.33800000000000002,"15min":0.33600000000000002},"max":{"1min":0.80500000000000005,"5min":0.82299999999999995,"15min":0.94899999999999995},"last":0.77000000000000002}]},{"osd":6,"last update":"Mon Oct 16 13:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.59299999999999997,"15min":0.59099999999999997},"min":{"1min":0.434,"5min":0.34499999999999997,"15min":0.33200000000000002},"max":{"1min":0.877,"5min":0.877,"15min":0.94299999999999995},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.68400000000000005,"15min":0.68799999999999994},"min":{"1min":0.51300000000000001,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.95299999999999996},"last":0.85699999999999998}]},{"osd":7,"last update":"Mon Oct 16 13:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.626,"15min":0.627},"min":{"1min":0.44600000000000001,"5min":0.39900000000000002,"15min":0.35799999999999998},"max":{"1min":0.85799999999999998,"5min":0.97499999999999998,"15min":0.98599999999999999},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.66900000000000004,"15min":0.67600000000000005},"min":{"1min":0.52900000000000003,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":1.026},"last":0.754}]}]},{"osd":5,"up_from":13,"seq":55834575965,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51756,"kb_used_data":2528,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320084,"statfs":{"total":96636764160,"available":96583766016,"internally_reserved":0,"allocated":2588672,"data_stored":1317988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:44:22 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.63800000000000001,"15min":0.622},"min":{"1min":0.51800000000000002,"5min":0.30499999999999999,"15min":0.26800000000000002},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.64200000000000002,"15min":0.64600000000000002},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.29699999999999999},"max":{"1min":0.80900000000000005,"5min":0.93700000000000006,"15min":0.96099999999999997},"last":0.56100000000000005}]},{"osd":1,"last update":"Mon Oct 16 13:44:22 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.67000000000000004,"15min":0.66100000000000003},"min":{"1min":0.42499999999999999,"5min":0.35899999999999999,"15min":0.29499999999999998},"max":{"1min":0.83599999999999997,"5min":0.873,"15min":0.89500000000000002},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66000000000000003,"15min":0.64900000000000002},"min":{"1min":0.44700000000000001,"5min":0.40799999999999997,"15min":0.38400000000000001},"max":{"1min":0.79500000000000004,"5min":0.871,"15min":0.94999999999999996},"last":0.51400000000000001}]},{"osd":2,"last update":"Mon Oct 16 13:44:22 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.67800000000000005,"15min":0.66200000000000003},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.38800000000000001},"max":{"1min":0.91000000000000003,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.68000000000000005,"15min":0.65300000000000002},"min":{"1min":0.52500000000000002,"5min":0.502,"15min":0.40200000000000002},"max":{"1min":0.871,"5min":0.89400000000000002,"15min":0.93500000000000005},"last":0.59299999999999997}]},{"osd":3,"last update":"Mon Oct 16 13:44:22 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.66900000000000004},"min":{"1min":0.46300000000000002,"5min":0.46000000000000002,"15min":0.45000000000000001},"max":{"1min":0.88900000000000001,"5min":0.90100000000000002,"15min":0.94999999999999996},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.70899999999999996,"15min":0.69499999999999995},"min":{"1min":0.59599999999999997,"5min":0.51700000000000002,"15min":0.46700000000000003},"max":{"1min":0.876,"5min":0.876,"15min":0.96999999999999997},"last":0.70499999999999996}]},{"osd":4,"last update":"Mon Oct 16 13:44:22 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.56000000000000005,"15min":0.53200000000000003},"min":{"1min":0.372,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.77900000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.60699999999999998,"15min":0.58899999999999997},"min":{"1min":0.39500000000000002,"5min":0.38800000000000001,"15min":0.34699999999999998},"max":{"1min":0.748,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.79000000000000004}]},{"osd":6,"last update":"Mon Oct 16 13:44:22 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.59499999999999997,"15min":0.57799999999999996},"min":{"1min":0.41399999999999998,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.76300000000000001,"5min":0.88900000000000001,"15min":0.93200000000000005},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.63600000000000001,"15min":0.61699999999999999},"min":{"1min":0.44900000000000001,"5min":0.42299999999999999,"15min":0.39000000000000001},"max":{"1min":0.82599999999999996,"5min":0.876,"15min":0.89600000000000002},"last":0.746}]},{"osd":7,"last update":"Mon Oct 16 13:44:22 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.64200000000000002,"15min":0.628},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.76700000000000002,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.70099999999999996,"15min":0.68000000000000005},"min":{"1min":0.46200000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.84999999999999998,"5min":0.98299999999999998,"15min":1.0229999999999999},"last":0.82499999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T13:44:44.989 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T13:44:45.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.145+0000 7f3e34422700 1 Processor -- start 2023-10-16T13:44:45.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.145+0000 7f3e34422700 1 -- start start 2023-10-16T13:44:45.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.146+0000 7f3e34422700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3e2c0cb190 0x7f3e2c0ae520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:45.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.146+0000 7f3e34422700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3e2c15e0b0 0x7f3e2c0aea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:45.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.146+0000 7f3e34422700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e2c06b1f0 con 0x7f3e2c159c90 2023-10-16T13:44:45.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.146+0000 7f3e34422700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e2c06b4b0 con 0x7f3e2c15e0b0 2023-10-16T13:44:45.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.146+0000 7f3e34422700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e2c06b790 con 0x7f3e2c0cb190 2023-10-16T13:44:45.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.146+0000 7f3e321be700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f3e2c159c90 0x7f3e2c15d9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41050/0 (socket says 172.21.15.16:41050) 2023-10-16T13:44:45.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.146+0000 7f3e329bf700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3e2c15e0b0 0x7f3e2c0aea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:45.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.146+0000 7f3e321be700 1 -- 172.21.15.16:0/690804524 learned_addr learned my addr 172.21.15.16:0/690804524 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:44:45.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.146+0000 7f3e319bd700 1 --2- 172.21.15.16:0/690804524 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3e2c0cb190 0x7f3e2c0ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:45.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.146+0000 7f3e329bf700 1 -- 172.21.15.16:0/690804524 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3e2c0cb190 msgr2=0x7f3e2c0ae520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:45.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.146+0000 7f3e329bf700 1 --2- 172.21.15.16:0/690804524 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3e2c0cb190 0x7f3e2c0ae520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:45.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.146+0000 7f3e329bf700 1 -- 172.21.15.16:0/690804524 >> v1:172.21.15.16:6789/0 conn(0x7f3e2c159c90 legacy=0x7f3e2c15d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:45.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.147+0000 7f3e329bf700 1 -- 172.21.15.16:0/690804524 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e2c06ba70 con 0x7f3e2c15e0b0 2023-10-16T13:44:45.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.146+0000 7f3e311bc700 1 -- 172.21.15.16:0/690804524 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4067126315 0 0) 0x7f3e2c06b1f0 con 0x7f3e2c159c90 2023-10-16T13:44:45.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.147+0000 7f3e319bd700 1 --2- 172.21.15.16:0/690804524 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3e2c0cb190 0x7f3e2c0ae520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:44:45.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.147+0000 7f3e329bf700 1 --2- 172.21.15.16:0/690804524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e2c15e0b0 0x7f3e2c0aea60 secure :-1 s=READY pgs=2297 cs=0 l=1 rev1=1 crypto rx=0x7f3e2400aea0 tx=0x7f3e2400c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc3c149e47ee31a7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:45.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.147+0000 7f3e311bc700 1 -- 172.21.15.16:0/690804524 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e2400f070 con 0x7f3e2c15e0b0 2023-10-16T13:44:45.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.147+0000 7f3e311bc700 1 -- 172.21.15.16:0/690804524 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e2400a2e0 con 0x7f3e2c15e0b0 2023-10-16T13:44:45.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.147+0000 7f3e311bc700 1 -- 172.21.15.16:0/690804524 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e2400cc00 con 0x7f3e2c15e0b0 2023-10-16T13:44:45.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.147+0000 7f3e34422700 1 -- 172.21.15.16:0/690804524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e2c15e0b0 msgr2=0x7f3e2c0aea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:45.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.147+0000 7f3e34422700 1 --2- 172.21.15.16:0/690804524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e2c15e0b0 0x7f3e2c0aea60 secure :-1 s=READY pgs=2297 cs=0 l=1 rev1=1 crypto rx=0x7f3e2400aea0 tx=0x7f3e2400c330 comp rx=0 tx=0).stop 2023-10-16T13:44:45.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.147+0000 7f3e34422700 1 -- 172.21.15.16:0/690804524 shutdown_connections 2023-10-16T13:44:45.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.147+0000 7f3e34422700 1 --2- 172.21.15.16:0/690804524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e2c15e0b0 0x7f3e2c0aea60 unknown :-1 s=CLOSED pgs=2297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:45.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.147+0000 7f3e34422700 1 --2- 172.21.15.16:0/690804524 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3e2c0cb190 0x7f3e2c0ae520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:45.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.147+0000 7f3e34422700 1 -- 172.21.15.16:0/690804524 >> 172.21.15.16:0/690804524 conn(0x7f3e2c0cc5b0 msgr2=0x7f3e2c0cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:45.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.148+0000 7f3e34422700 1 -- 172.21.15.16:0/690804524 shutdown_connections 2023-10-16T13:44:45.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.148+0000 7f3e34422700 1 -- 172.21.15.16:0/690804524 wait complete. 2023-10-16T13:44:45.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.149+0000 7f3e34422700 1 Processor -- start 2023-10-16T13:44:45.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.149+0000 7f3e34422700 1 -- start start 2023-10-16T13:44:45.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.149+0000 7f3e34422700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3e2c0cb190 0x7f3e2c0c4630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:45.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.149+0000 7f3e34422700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3e2c159c90 0x7f3e2c14aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:45.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.149+0000 7f3e34422700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e2c15e0b0 0x7f3e2c06e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:45.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.150+0000 7f3e34422700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e2c0c5fb0 con 0x7f3e2c15e0b0 2023-10-16T13:44:45.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.150+0000 7f3e34422700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e2c0c3660 con 0x7f3e2c159c90 2023-10-16T13:44:45.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.150+0000 7f3e34422700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e2c0c41a0 con 0x7f3e2c0cb190 2023-10-16T13:44:45.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.150+0000 7f3e329bf700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e2c15e0b0 0x7f3e2c06e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:45.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.150+0000 7f3e329bf700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e2c15e0b0 0x7f3e2c06e0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51786/0 (socket says 172.21.15.16:51786) 2023-10-16T13:44:45.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.150+0000 7f3e329bf700 1 -- 172.21.15.16:0/852926029 learned_addr learned my addr 172.21.15.16:0/852926029 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:45.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.150+0000 7f3e321be700 1 --2- 172.21.15.16:0/852926029 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3e2c0cb190 0x7f3e2c0c4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:45.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.150+0000 7f3e319bd700 1 --2- 172.21.15.16:0/852926029 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3e2c159c90 0x7f3e2c14aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:45.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.150+0000 7f3e329bf700 1 -- 172.21.15.16:0/852926029 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3e2c0cb190 msgr2=0x7f3e2c0c4630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:45.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.150+0000 7f3e329bf700 1 --2- 172.21.15.16:0/852926029 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3e2c0cb190 0x7f3e2c0c4630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:45.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.150+0000 7f3e329bf700 1 -- 172.21.15.16:0/852926029 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3e2c159c90 msgr2=0x7f3e2c14aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:45.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.150+0000 7f3e329bf700 1 --2- 172.21.15.16:0/852926029 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3e2c159c90 0x7f3e2c14aef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:45.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.150+0000 7f3e329bf700 1 -- 172.21.15.16:0/852926029 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e2400ab80 con 0x7f3e2c15e0b0 2023-10-16T13:44:45.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.150+0000 7f3e329bf700 1 --2- 172.21.15.16:0/852926029 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e2c15e0b0 0x7f3e2c06e0a0 secure :-1 s=READY pgs=2298 cs=0 l=1 rev1=1 crypto rx=0x7f3e2400c730 tx=0x7f3e24004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:45.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.150+0000 7f3e22ffd700 1 -- 172.21.15.16:0/852926029 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e240045d0 con 0x7f3e2c15e0b0 2023-10-16T13:44:45.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.150+0000 7f3e22ffd700 1 -- 172.21.15.16:0/852926029 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e24004730 con 0x7f3e2c15e0b0 2023-10-16T13:44:45.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.150+0000 7f3e34422700 1 -- 172.21.15.16:0/852926029 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e2c14b430 con 0x7f3e2c15e0b0 2023-10-16T13:44:45.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.150+0000 7f3e22ffd700 1 -- 172.21.15.16:0/852926029 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e24008440 con 0x7f3e2c15e0b0 2023-10-16T13:44:45.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.150+0000 7f3e34422700 1 -- 172.21.15.16:0/852926029 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e2c0c3b50 con 0x7f3e2c15e0b0 2023-10-16T13:44:45.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.151+0000 7f3e34422700 1 -- 172.21.15.16:0/852926029 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e2c0c4310 con 0x7f3e2c15e0b0 2023-10-16T13:44:45.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.152+0000 7f3e22ffd700 1 -- 172.21.15.16:0/852926029 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 126) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3e240085c0 con 0x7f3e2c15e0b0 2023-10-16T13:44:45.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.152+0000 7f3e22ffd700 1 --2- 172.21.15.16:0/852926029 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f3e1006b020 0x7f3e1006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:45.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.152+0000 7f3e22ffd700 1 -- 172.21.15.16:0/852926029 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(206..206 src has 1..206) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f3e2408b1a0 con 0x7f3e2c15e0b0 2023-10-16T13:44:45.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.152+0000 7f3e321be700 1 --2- 172.21.15.16:0/852926029 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f3e1006b020 0x7f3e1006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:45.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.153+0000 7f3e321be700 1 --2- 172.21.15.16:0/852926029 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f3e1006b020 0x7f3e1006d4e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c000f60 tx=0x7f3e1c007be0 comp rx=0 tx=0).ready entity=mgr.10665 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:45.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.157+0000 7f3e22ffd700 1 -- 172.21.15.16:0/852926029 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3e24056020 con 0x7f3e2c15e0b0 2023-10-16T13:44:45.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.357+0000 7f3e34422700 1 -- 172.21.15.16:0/852926029 --> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3e2c06bd50 con 0x7f3e1006b020 2023-10-16T13:44:45.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.365+0000 7f3e22ffd700 1 -- 172.21.15.16:0/852926029 <== mgr.10665 v2:172.21.15.16:6832/1268439414 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289891 (secure 0 0 0) 0x7f3e1c0012b0 con 0x7f3e1006b020 2023-10-16T13:44:45.372 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:44:45.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.369+0000 7f3e34422700 1 -- 172.21.15.16:0/852926029 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f3e1006b020 msgr2=0x7f3e1006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:45.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.369+0000 7f3e34422700 1 --2- 172.21.15.16:0/852926029 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f3e1006b020 0x7f3e1006d4e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c000f60 tx=0x7f3e1c007be0 comp rx=0 tx=0).stop 2023-10-16T13:44:45.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.369+0000 7f3e34422700 1 -- 172.21.15.16:0/852926029 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e2c15e0b0 msgr2=0x7f3e2c06e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:45.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.369+0000 7f3e34422700 1 --2- 172.21.15.16:0/852926029 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e2c15e0b0 0x7f3e2c06e0a0 secure :-1 s=READY pgs=2298 cs=0 l=1 rev1=1 crypto rx=0x7f3e2400c730 tx=0x7f3e24004ad0 comp rx=0 tx=0).stop 2023-10-16T13:44:45.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.369+0000 7f3e34422700 1 -- 172.21.15.16:0/852926029 shutdown_connections 2023-10-16T13:44:45.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.369+0000 7f3e34422700 1 --2- 172.21.15.16:0/852926029 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f3e1006b020 0x7f3e1006d4e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:45.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.369+0000 7f3e34422700 1 --2- 172.21.15.16:0/852926029 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e2c15e0b0 0x7f3e2c06e0a0 unknown :-1 s=CLOSED pgs=2298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:45.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.369+0000 7f3e34422700 1 --2- 172.21.15.16:0/852926029 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3e2c0cb190 0x7f3e2c0c4630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:45.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.369+0000 7f3e34422700 1 --2- 172.21.15.16:0/852926029 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3e2c159c90 0x7f3e2c14aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:45.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.369+0000 7f3e34422700 1 -- 172.21.15.16:0/852926029 >> 172.21.15.16:0/852926029 conn(0x7f3e2c0cc5b0 msgr2=0x7f3e2c0b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:45.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.369+0000 7f3e34422700 1 -- 172.21.15.16:0/852926029 shutdown_connections 2023-10-16T13:44:45.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.369+0000 7f3e34422700 1 -- 172.21.15.16:0/852926029 wait complete. 2023-10-16T13:44:45.376 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-10-16T13:44:45.386 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2023-10-16T13:44:44.426110+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1940,"num_read_kb":4014,"num_write":583,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":364,"ondisk_log_size":364,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":136,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":383248,"kb_used_data":21344,"kb_used_omap":12,"kb_used_meta":361779,"kb_avail":754591472,"statfs":{"total":773094113280,"available":772701667328,"internally_reserved":0,"allocated":21856256,"data_stored":11729884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12740,"internal_metadata":370462268},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.654429"},"pg_stats":[{"pgid":"26.3f","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.264372+0000","last_change":"2023-10-16T13:44:40.826902+0000","last_active":"2023-10-16T13:44:42.264372+0000","last_peered":"2023-10-16T13:44:42.264372+0000","last_clean":"2023-10-16T13:44:42.264372+0000","last_became_active":"2023-10-16T13:44:40.826586+0000","last_became_peered":"2023-10-16T13:44:40.826586+0000","last_unstale":"2023-10-16T13:44:42.264372+0000","last_undegraded":"2023-10-16T13:44:42.264372+0000","last_fullsized":"2023-10-16T13:44:42.264372+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:47:01.341720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.3e","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.264405+0000","last_change":"2023-10-16T13:44:40.816625+0000","last_active":"2023-10-16T13:44:42.264405+0000","last_peered":"2023-10-16T13:44:42.264405+0000","last_clean":"2023-10-16T13:44:42.264405+0000","last_became_active":"2023-10-16T13:44:40.815628+0000","last_became_peered":"2023-10-16T13:44:40.815628+0000","last_unstale":"2023-10-16T13:44:42.264405+0000","last_undegraded":"2023-10-16T13:44:42.264405+0000","last_fullsized":"2023-10-16T13:44:42.264405+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:30.538620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.3d","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.265348+0000","last_change":"2023-10-16T13:44:40.821185+0000","last_active":"2023-10-16T13:44:42.265348+0000","last_peered":"2023-10-16T13:44:42.265348+0000","last_clean":"2023-10-16T13:44:42.265348+0000","last_became_active":"2023-10-16T13:44:40.820881+0000","last_became_peered":"2023-10-16T13:44:40.820881+0000","last_unstale":"2023-10-16T13:44:42.265348+0000","last_undegraded":"2023-10-16T13:44:42.265348+0000","last_fullsized":"2023-10-16T13:44:42.265348+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:13:11.470834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.3c","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.263941+0000","last_change":"2023-10-16T13:44:40.820272+0000","last_active":"2023-10-16T13:44:42.263941+0000","last_peered":"2023-10-16T13:44:42.263941+0000","last_clean":"2023-10-16T13:44:42.263941+0000","last_became_active":"2023-10-16T13:44:40.820011+0000","last_became_peered":"2023-10-16T13:44:40.820011+0000","last_unstale":"2023-10-16T13:44:42.263941+0000","last_undegraded":"2023-10-16T13:44:42.263941+0000","last_fullsized":"2023-10-16T13:44:42.263941+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:03:54.739025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.3b","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.803290+0000","last_change":"2023-10-16T13:44:40.810785+0000","last_active":"2023-10-16T13:44:41.803290+0000","last_peered":"2023-10-16T13:44:41.803290+0000","last_clean":"2023-10-16T13:44:41.803290+0000","last_became_active":"2023-10-16T13:44:40.810486+0000","last_became_peered":"2023-10-16T13:44:40.810486+0000","last_unstale":"2023-10-16T13:44:41.803290+0000","last_undegraded":"2023-10-16T13:44:41.803290+0000","last_fullsized":"2023-10-16T13:44:41.803290+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:55:25.050877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.3a","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.803359+0000","last_change":"2023-10-16T13:44:40.819529+0000","last_active":"2023-10-16T13:44:41.803359+0000","last_peered":"2023-10-16T13:44:41.803359+0000","last_clean":"2023-10-16T13:44:41.803359+0000","last_became_active":"2023-10-16T13:44:40.819182+0000","last_became_peered":"2023-10-16T13:44:40.819182+0000","last_unstale":"2023-10-16T13:44:41.803359+0000","last_undegraded":"2023-10-16T13:44:41.803359+0000","last_fullsized":"2023-10-16T13:44:41.803359+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:25:15.818591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.39","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.801818+0000","last_change":"2023-10-16T13:44:40.826478+0000","last_active":"2023-10-16T13:44:41.801818+0000","last_peered":"2023-10-16T13:44:41.801818+0000","last_clean":"2023-10-16T13:44:41.801818+0000","last_became_active":"2023-10-16T13:44:40.826181+0000","last_became_peered":"2023-10-16T13:44:40.826181+0000","last_unstale":"2023-10-16T13:44:41.801818+0000","last_undegraded":"2023-10-16T13:44:41.801818+0000","last_fullsized":"2023-10-16T13:44:41.801818+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:29:14.006949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.38","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.899132+0000","last_change":"2023-10-16T13:44:40.825799+0000","last_active":"2023-10-16T13:44:41.899132+0000","last_peered":"2023-10-16T13:44:41.899132+0000","last_clean":"2023-10-16T13:44:41.899132+0000","last_became_active":"2023-10-16T13:44:40.825491+0000","last_became_peered":"2023-10-16T13:44:40.825491+0000","last_unstale":"2023-10-16T13:44:41.899132+0000","last_undegraded":"2023-10-16T13:44:41.899132+0000","last_fullsized":"2023-10-16T13:44:41.899132+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:38:47.872141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.37","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.264456+0000","last_change":"2023-10-16T13:44:40.814315+0000","last_active":"2023-10-16T13:44:42.264456+0000","last_peered":"2023-10-16T13:44:42.264456+0000","last_clean":"2023-10-16T13:44:42.264456+0000","last_became_active":"2023-10-16T13:44:40.814017+0000","last_became_peered":"2023-10-16T13:44:40.814017+0000","last_unstale":"2023-10-16T13:44:42.264456+0000","last_undegraded":"2023-10-16T13:44:42.264456+0000","last_fullsized":"2023-10-16T13:44:42.264456+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:39:57.915119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.36","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.803512+0000","last_change":"2023-10-16T13:44:40.819560+0000","last_active":"2023-10-16T13:44:41.803512+0000","last_peered":"2023-10-16T13:44:41.803512+0000","last_clean":"2023-10-16T13:44:41.803512+0000","last_became_active":"2023-10-16T13:44:40.819156+0000","last_became_peered":"2023-10-16T13:44:40.819156+0000","last_unstale":"2023-10-16T13:44:41.803512+0000","last_undegraded":"2023-10-16T13:44:41.803512+0000","last_fullsized":"2023-10-16T13:44:41.803512+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:41:39.966432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.35","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.264874+0000","last_change":"2023-10-16T13:44:40.817394+0000","last_active":"2023-10-16T13:44:42.264874+0000","last_peered":"2023-10-16T13:44:42.264874+0000","last_clean":"2023-10-16T13:44:42.264874+0000","last_became_active":"2023-10-16T13:44:40.817131+0000","last_became_peered":"2023-10-16T13:44:40.817131+0000","last_unstale":"2023-10-16T13:44:42.264874+0000","last_undegraded":"2023-10-16T13:44:42.264874+0000","last_fullsized":"2023-10-16T13:44:42.264874+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:09:03.969733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.34","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.264473+0000","last_change":"2023-10-16T13:44:40.829617+0000","last_active":"2023-10-16T13:44:42.264473+0000","last_peered":"2023-10-16T13:44:42.264473+0000","last_clean":"2023-10-16T13:44:42.264473+0000","last_became_active":"2023-10-16T13:44:40.829156+0000","last_became_peered":"2023-10-16T13:44:40.829156+0000","last_unstale":"2023-10-16T13:44:42.264473+0000","last_undegraded":"2023-10-16T13:44:42.264473+0000","last_fullsized":"2023-10-16T13:44:42.264473+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:42:50.455035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.33","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.265085+0000","last_change":"2023-10-16T13:44:40.813904+0000","last_active":"2023-10-16T13:44:42.265085+0000","last_peered":"2023-10-16T13:44:42.265085+0000","last_clean":"2023-10-16T13:44:42.265085+0000","last_became_active":"2023-10-16T13:44:40.813589+0000","last_became_peered":"2023-10-16T13:44:40.813589+0000","last_unstale":"2023-10-16T13:44:42.265085+0000","last_undegraded":"2023-10-16T13:44:42.265085+0000","last_fullsized":"2023-10-16T13:44:42.265085+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:25:14.054951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.32","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.801681+0000","last_change":"2023-10-16T13:44:40.813471+0000","last_active":"2023-10-16T13:44:41.801681+0000","last_peered":"2023-10-16T13:44:41.801681+0000","last_clean":"2023-10-16T13:44:41.801681+0000","last_became_active":"2023-10-16T13:44:40.812676+0000","last_became_peered":"2023-10-16T13:44:40.812676+0000","last_unstale":"2023-10-16T13:44:41.801681+0000","last_undegraded":"2023-10-16T13:44:41.801681+0000","last_fullsized":"2023-10-16T13:44:41.801681+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:31:53.429470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.31","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.263699+0000","last_change":"2023-10-16T13:44:40.816134+0000","last_active":"2023-10-16T13:44:42.263699+0000","last_peered":"2023-10-16T13:44:42.263699+0000","last_clean":"2023-10-16T13:44:42.263699+0000","last_became_active":"2023-10-16T13:44:40.815573+0000","last_became_peered":"2023-10-16T13:44:40.815573+0000","last_unstale":"2023-10-16T13:44:42.263699+0000","last_undegraded":"2023-10-16T13:44:42.263699+0000","last_fullsized":"2023-10-16T13:44:42.263699+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:09:03.969733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.30","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.803164+0000","last_change":"2023-10-16T13:44:40.816249+0000","last_active":"2023-10-16T13:44:41.803164+0000","last_peered":"2023-10-16T13:44:41.803164+0000","last_clean":"2023-10-16T13:44:41.803164+0000","last_became_active":"2023-10-16T13:44:40.815919+0000","last_became_peered":"2023-10-16T13:44:40.815919+0000","last_unstale":"2023-10-16T13:44:41.803164+0000","last_undegraded":"2023-10-16T13:44:41.803164+0000","last_fullsized":"2023-10-16T13:44:41.803164+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:41:29.011926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.2f","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.803460+0000","last_change":"2023-10-16T13:44:40.826410+0000","last_active":"2023-10-16T13:44:41.803460+0000","last_peered":"2023-10-16T13:44:41.803460+0000","last_clean":"2023-10-16T13:44:41.803460+0000","last_became_active":"2023-10-16T13:44:40.826089+0000","last_became_peered":"2023-10-16T13:44:40.826089+0000","last_unstale":"2023-10-16T13:44:41.803460+0000","last_undegraded":"2023-10-16T13:44:41.803460+0000","last_fullsized":"2023-10-16T13:44:41.803460+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:39:08.726953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.2e","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.264313+0000","last_change":"2023-10-16T13:44:40.826108+0000","last_active":"2023-10-16T13:44:42.264313+0000","last_peered":"2023-10-16T13:44:42.264313+0000","last_clean":"2023-10-16T13:44:42.264313+0000","last_became_active":"2023-10-16T13:44:40.825804+0000","last_became_peered":"2023-10-16T13:44:40.825804+0000","last_unstale":"2023-10-16T13:44:42.264313+0000","last_undegraded":"2023-10-16T13:44:42.264313+0000","last_fullsized":"2023-10-16T13:44:42.264313+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:21:56.832533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.2d","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.264552+0000","last_change":"2023-10-16T13:44:40.810061+0000","last_active":"2023-10-16T13:44:42.264552+0000","last_peered":"2023-10-16T13:44:42.264552+0000","last_clean":"2023-10-16T13:44:42.264552+0000","last_became_active":"2023-10-16T13:44:40.809672+0000","last_became_peered":"2023-10-16T13:44:40.809672+0000","last_unstale":"2023-10-16T13:44:42.264552+0000","last_undegraded":"2023-10-16T13:44:42.264552+0000","last_fullsized":"2023-10-16T13:44:42.264552+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:07:54.767396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.2c","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.803628+0000","last_change":"2023-10-16T13:44:40.829539+0000","last_active":"2023-10-16T13:44:41.803628+0000","last_peered":"2023-10-16T13:44:41.803628+0000","last_clean":"2023-10-16T13:44:41.803628+0000","last_became_active":"2023-10-16T13:44:40.829219+0000","last_became_peered":"2023-10-16T13:44:40.829219+0000","last_unstale":"2023-10-16T13:44:41.803628+0000","last_undegraded":"2023-10-16T13:44:41.803628+0000","last_fullsized":"2023-10-16T13:44:41.803628+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:43:50.582740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.2b","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.264122+0000","last_change":"2023-10-16T13:44:40.814959+0000","last_active":"2023-10-16T13:44:42.264122+0000","last_peered":"2023-10-16T13:44:42.264122+0000","last_clean":"2023-10-16T13:44:42.264122+0000","last_became_active":"2023-10-16T13:44:40.814620+0000","last_became_peered":"2023-10-16T13:44:40.814620+0000","last_unstale":"2023-10-16T13:44:42.264122+0000","last_undegraded":"2023-10-16T13:44:42.264122+0000","last_fullsized":"2023-10-16T13:44:42.264122+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:36:06.385172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.2a","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.263873+0000","last_change":"2023-10-16T13:44:40.817464+0000","last_active":"2023-10-16T13:44:42.263873+0000","last_peered":"2023-10-16T13:44:42.263873+0000","last_clean":"2023-10-16T13:44:42.263873+0000","last_became_active":"2023-10-16T13:44:40.817178+0000","last_became_peered":"2023-10-16T13:44:40.817178+0000","last_unstale":"2023-10-16T13:44:42.263873+0000","last_undegraded":"2023-10-16T13:44:42.263873+0000","last_fullsized":"2023-10-16T13:44:42.263873+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:32:26.254719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.29","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.899353+0000","last_change":"2023-10-16T13:44:40.825857+0000","last_active":"2023-10-16T13:44:41.899353+0000","last_peered":"2023-10-16T13:44:41.899353+0000","last_clean":"2023-10-16T13:44:41.899353+0000","last_became_active":"2023-10-16T13:44:40.825553+0000","last_became_peered":"2023-10-16T13:44:40.825553+0000","last_unstale":"2023-10-16T13:44:41.899353+0000","last_undegraded":"2023-10-16T13:44:41.899353+0000","last_fullsized":"2023-10-16T13:44:41.899353+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:08:27.304252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.28","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.264436+0000","last_change":"2023-10-16T13:44:40.828378+0000","last_active":"2023-10-16T13:44:42.264436+0000","last_peered":"2023-10-16T13:44:42.264436+0000","last_clean":"2023-10-16T13:44:42.264436+0000","last_became_active":"2023-10-16T13:44:40.828088+0000","last_became_peered":"2023-10-16T13:44:40.828088+0000","last_unstale":"2023-10-16T13:44:42.264436+0000","last_undegraded":"2023-10-16T13:44:42.264436+0000","last_fullsized":"2023-10-16T13:44:42.264436+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:34:24.198222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.27","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.801791+0000","last_change":"2023-10-16T13:44:40.817056+0000","last_active":"2023-10-16T13:44:41.801791+0000","last_peered":"2023-10-16T13:44:41.801791+0000","last_clean":"2023-10-16T13:44:41.801791+0000","last_became_active":"2023-10-16T13:44:40.816727+0000","last_became_peered":"2023-10-16T13:44:40.816727+0000","last_unstale":"2023-10-16T13:44:41.801791+0000","last_undegraded":"2023-10-16T13:44:41.801791+0000","last_fullsized":"2023-10-16T13:44:41.801791+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:57:05.396702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.26","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.265199+0000","last_change":"2023-10-16T13:44:40.814830+0000","last_active":"2023-10-16T13:44:42.265199+0000","last_peered":"2023-10-16T13:44:42.265199+0000","last_clean":"2023-10-16T13:44:42.265199+0000","last_became_active":"2023-10-16T13:44:40.814089+0000","last_became_peered":"2023-10-16T13:44:40.814089+0000","last_unstale":"2023-10-16T13:44:42.265199+0000","last_undegraded":"2023-10-16T13:44:42.265199+0000","last_fullsized":"2023-10-16T13:44:42.265199+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:42:27.071178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.25","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.899335+0000","last_change":"2023-10-16T13:44:40.825948+0000","last_active":"2023-10-16T13:44:41.899335+0000","last_peered":"2023-10-16T13:44:41.899335+0000","last_clean":"2023-10-16T13:44:41.899335+0000","last_became_active":"2023-10-16T13:44:40.825659+0000","last_became_peered":"2023-10-16T13:44:40.825659+0000","last_unstale":"2023-10-16T13:44:41.899335+0000","last_undegraded":"2023-10-16T13:44:41.899335+0000","last_fullsized":"2023-10-16T13:44:41.899335+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:43:26.778586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.24","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.803323+0000","last_change":"2023-10-16T13:44:40.815262+0000","last_active":"2023-10-16T13:44:41.803323+0000","last_peered":"2023-10-16T13:44:41.803323+0000","last_clean":"2023-10-16T13:44:41.803323+0000","last_became_active":"2023-10-16T13:44:40.814894+0000","last_became_peered":"2023-10-16T13:44:40.814894+0000","last_unstale":"2023-10-16T13:44:41.803323+0000","last_undegraded":"2023-10-16T13:44:41.803323+0000","last_fullsized":"2023-10-16T13:44:41.803323+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:30:04.291099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.23","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.265002+0000","last_change":"2023-10-16T13:44:40.819830+0000","last_active":"2023-10-16T13:44:42.265002+0000","last_peered":"2023-10-16T13:44:42.265002+0000","last_clean":"2023-10-16T13:44:42.265002+0000","last_became_active":"2023-10-16T13:44:40.819473+0000","last_became_peered":"2023-10-16T13:44:40.819473+0000","last_unstale":"2023-10-16T13:44:42.265002+0000","last_undegraded":"2023-10-16T13:44:42.265002+0000","last_fullsized":"2023-10-16T13:44:42.265002+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:26:13.931633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.22","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.264999+0000","last_change":"2023-10-16T13:44:40.819766+0000","last_active":"2023-10-16T13:44:42.264999+0000","last_peered":"2023-10-16T13:44:42.264999+0000","last_clean":"2023-10-16T13:44:42.264999+0000","last_became_active":"2023-10-16T13:44:40.819334+0000","last_became_peered":"2023-10-16T13:44:40.819334+0000","last_unstale":"2023-10-16T13:44:42.264999+0000","last_undegraded":"2023-10-16T13:44:42.264999+0000","last_fullsized":"2023-10-16T13:44:42.264999+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:32:26.254719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.21","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.801936+0000","last_change":"2023-10-16T13:44:40.817822+0000","last_active":"2023-10-16T13:44:41.801936+0000","last_peered":"2023-10-16T13:44:41.801936+0000","last_clean":"2023-10-16T13:44:41.801936+0000","last_became_active":"2023-10-16T13:44:40.817470+0000","last_became_peered":"2023-10-16T13:44:40.817470+0000","last_unstale":"2023-10-16T13:44:41.801936+0000","last_undegraded":"2023-10-16T13:44:41.801936+0000","last_fullsized":"2023-10-16T13:44:41.801936+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:36:24.516163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.20","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.803479+0000","last_change":"2023-10-16T13:44:40.808877+0000","last_active":"2023-10-16T13:44:41.803479+0000","last_peered":"2023-10-16T13:44:41.803479+0000","last_clean":"2023-10-16T13:44:41.803479+0000","last_became_active":"2023-10-16T13:44:40.808545+0000","last_became_peered":"2023-10-16T13:44:40.808545+0000","last_unstale":"2023-10-16T13:44:41.803479+0000","last_undegraded":"2023-10-16T13:44:41.803479+0000","last_fullsized":"2023-10-16T13:44:41.803479+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:22:10.355097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1c","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.899457+0000","last_change":"2023-10-16T13:44:40.829316+0000","last_active":"2023-10-16T13:44:41.899457+0000","last_peered":"2023-10-16T13:44:41.899457+0000","last_clean":"2023-10-16T13:44:41.899457+0000","last_became_active":"2023-10-16T13:44:40.828906+0000","last_became_peered":"2023-10-16T13:44:40.828906+0000","last_unstale":"2023-10-16T13:44:41.899457+0000","last_undegraded":"2023-10-16T13:44:41.899457+0000","last_fullsized":"2023-10-16T13:44:41.899457+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:59:10.429999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.1f","version":"205'4","reported_seq":16,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.264191+0000","last_change":"2023-10-16T13:44:39.789990+0000","last_active":"2023-10-16T13:44:42.264191+0000","last_peered":"2023-10-16T13:44:42.264191+0000","last_clean":"2023-10-16T13:44:42.264191+0000","last_became_active":"2023-10-16T13:44:39.789670+0000","last_became_peered":"2023-10-16T13:44:39.789670+0000","last_unstale":"2023-10-16T13:44:42.264191+0000","last_undegraded":"2023-10-16T13:44:42.264191+0000","last_fullsized":"2023-10-16T13:44:42.264191+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_clean_scrub_stamp":"2023-10-16T13:44:38.766670+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:58:57.993734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1d","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.802302+0000","last_change":"2023-10-16T13:44:40.826125+0000","last_active":"2023-10-16T13:44:41.802302+0000","last_peered":"2023-10-16T13:44:41.802302+0000","last_clean":"2023-10-16T13:44:41.802302+0000","last_became_active":"2023-10-16T13:44:40.825778+0000","last_became_peered":"2023-10-16T13:44:40.825778+0000","last_unstale":"2023-10-16T13:44:41.802302+0000","last_undegraded":"2023-10-16T13:44:41.802302+0000","last_fullsized":"2023-10-16T13:44:41.802302+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:37:59.510239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1e","version":"205'1","reported_seq":13,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.803607+0000","last_change":"2023-10-16T13:44:39.798962+0000","last_active":"2023-10-16T13:44:41.803607+0000","last_peered":"2023-10-16T13:44:41.803607+0000","last_clean":"2023-10-16T13:44:41.803607+0000","last_became_active":"2023-10-16T13:44:39.798670+0000","last_became_peered":"2023-10-16T13:44:39.798670+0000","last_unstale":"2023-10-16T13:44:41.803607+0000","last_undegraded":"2023-10-16T13:44:41.803607+0000","last_fullsized":"2023-10-16T13:44:41.803607+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_clean_scrub_stamp":"2023-10-16T13:44:38.766670+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:33:06.483394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1e","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.265504+0000","last_change":"2023-10-16T13:44:40.826910+0000","last_active":"2023-10-16T13:44:42.265504+0000","last_peered":"2023-10-16T13:44:42.265504+0000","last_clean":"2023-10-16T13:44:42.265504+0000","last_became_active":"2023-10-16T13:44:40.826597+0000","last_became_peered":"2023-10-16T13:44:40.826597+0000","last_unstale":"2023-10-16T13:44:42.265504+0000","last_undegraded":"2023-10-16T13:44:42.265504+0000","last_fullsized":"2023-10-16T13:44:42.265504+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:36:06.385172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1d","version":"205'2","reported_seq":14,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.264898+0000","last_change":"2023-10-16T13:44:39.791516+0000","last_active":"2023-10-16T13:44:42.264898+0000","last_peered":"2023-10-16T13:44:42.264898+0000","last_clean":"2023-10-16T13:44:42.264898+0000","last_became_active":"2023-10-16T13:44:39.791205+0000","last_became_peered":"2023-10-16T13:44:39.791205+0000","last_unstale":"2023-10-16T13:44:42.264898+0000","last_undegraded":"2023-10-16T13:44:42.264898+0000","last_fullsized":"2023-10-16T13:44:42.264898+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_clean_scrub_stamp":"2023-10-16T13:44:38.766670+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:27:15.853209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1f","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.265487+0000","last_change":"2023-10-16T13:44:40.815054+0000","last_active":"2023-10-16T13:44:42.265487+0000","last_peered":"2023-10-16T13:44:42.265487+0000","last_clean":"2023-10-16T13:44:42.265487+0000","last_became_active":"2023-10-16T13:44:40.814521+0000","last_became_peered":"2023-10-16T13:44:40.814521+0000","last_unstale":"2023-10-16T13:44:42.265487+0000","last_undegraded":"2023-10-16T13:44:42.265487+0000","last_fullsized":"2023-10-16T13:44:42.265487+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:07:54.767396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1c","version":"205'2","reported_seq":14,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.264263+0000","last_change":"2023-10-16T13:44:39.794105+0000","last_active":"2023-10-16T13:44:42.264263+0000","last_peered":"2023-10-16T13:44:42.264263+0000","last_clean":"2023-10-16T13:44:42.264263+0000","last_became_active":"2023-10-16T13:44:39.793773+0000","last_became_peered":"2023-10-16T13:44:39.793773+0000","last_unstale":"2023-10-16T13:44:42.264263+0000","last_undegraded":"2023-10-16T13:44:42.264263+0000","last_fullsized":"2023-10-16T13:44:42.264263+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_clean_scrub_stamp":"2023-10-16T13:44:38.766670+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:55:37.119423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.264063+0000","last_change":"2023-10-16T13:44:40.809999+0000","last_active":"2023-10-16T13:44:42.264063+0000","last_peered":"2023-10-16T13:44:42.264063+0000","last_clean":"2023-10-16T13:44:42.264063+0000","last_became_active":"2023-10-16T13:44:40.809679+0000","last_became_peered":"2023-10-16T13:44:40.809679+0000","last_unstale":"2023-10-16T13:44:42.264063+0000","last_undegraded":"2023-10-16T13:44:42.264063+0000","last_fullsized":"2023-10-16T13:44:42.264063+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:25:11.898623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.1b","version":"205'1","reported_seq":13,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.803404+0000","last_change":"2023-10-16T13:44:39.799020+0000","last_active":"2023-10-16T13:44:41.803404+0000","last_peered":"2023-10-16T13:44:41.803404+0000","last_clean":"2023-10-16T13:44:41.803404+0000","last_became_active":"2023-10-16T13:44:39.798711+0000","last_became_peered":"2023-10-16T13:44:39.798711+0000","last_unstale":"2023-10-16T13:44:41.803404+0000","last_undegraded":"2023-10-16T13:44:41.803404+0000","last_fullsized":"2023-10-16T13:44:41.803404+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_clean_scrub_stamp":"2023-10-16T13:44:38.766670+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:38:24.443535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.263938+0000","last_change":"2023-10-16T13:44:40.815920+0000","last_active":"2023-10-16T13:44:42.263938+0000","last_peered":"2023-10-16T13:44:42.263938+0000","last_clean":"2023-10-16T13:44:42.263938+0000","last_became_active":"2023-10-16T13:44:40.815144+0000","last_became_peered":"2023-10-16T13:44:40.815144+0000","last_unstale":"2023-10-16T13:44:42.263938+0000","last_undegraded":"2023-10-16T13:44:42.263938+0000","last_fullsized":"2023-10-16T13:44:42.263938+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:26:13.931633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1a","version":"205'1","reported_seq":13,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.263983+0000","last_change":"2023-10-16T13:44:39.790201+0000","last_active":"2023-10-16T13:44:42.263983+0000","last_peered":"2023-10-16T13:44:42.263983+0000","last_clean":"2023-10-16T13:44:42.263983+0000","last_became_active":"2023-10-16T13:44:39.789508+0000","last_became_peered":"2023-10-16T13:44:39.789508+0000","last_unstale":"2023-10-16T13:44:42.263983+0000","last_undegraded":"2023-10-16T13:44:42.263983+0000","last_fullsized":"2023-10-16T13:44:42.263983+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_clean_scrub_stamp":"2023-10-16T13:44:38.766670+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:56:14.442023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1a","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.264020+0000","last_change":"2023-10-16T13:44:40.818039+0000","last_active":"2023-10-16T13:44:42.264020+0000","last_peered":"2023-10-16T13:44:42.264020+0000","last_clean":"2023-10-16T13:44:42.264020+0000","last_became_active":"2023-10-16T13:44:40.817737+0000","last_became_peered":"2023-10-16T13:44:40.817737+0000","last_unstale":"2023-10-16T13:44:42.264020+0000","last_undegraded":"2023-10-16T13:44:42.264020+0000","last_fullsized":"2023-10-16T13:44:42.264020+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:42:27.071178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":12,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.264704+0000","last_change":"2023-10-16T13:44:39.793328+0000","last_active":"2023-10-16T13:44:42.264704+0000","last_peered":"2023-10-16T13:44:42.264704+0000","last_clean":"2023-10-16T13:44:42.264704+0000","last_became_active":"2023-10-16T13:44:39.793035+0000","last_became_peered":"2023-10-16T13:44:39.793035+0000","last_unstale":"2023-10-16T13:44:42.264704+0000","last_undegraded":"2023-10-16T13:44:42.264704+0000","last_fullsized":"2023-10-16T13:44:42.264704+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_clean_scrub_stamp":"2023-10-16T13:44:38.766670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:00:05.037176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.14","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.899566+0000","last_change":"2023-10-16T13:44:40.829589+0000","last_active":"2023-10-16T13:44:41.899566+0000","last_peered":"2023-10-16T13:44:41.899566+0000","last_clean":"2023-10-16T13:44:41.899566+0000","last_became_active":"2023-10-16T13:44:40.828946+0000","last_became_peered":"2023-10-16T13:44:40.828946+0000","last_unstale":"2023-10-16T13:44:41.899566+0000","last_undegraded":"2023-10-16T13:44:41.899566+0000","last_fullsized":"2023-10-16T13:44:41.899566+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:38:25.447770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":12,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.803510+0000","last_change":"2023-10-16T13:44:39.787201+0000","last_active":"2023-10-16T13:44:41.803510+0000","last_peered":"2023-10-16T13:44:41.803510+0000","last_clean":"2023-10-16T13:44:41.803510+0000","last_became_active":"2023-10-16T13:44:39.786906+0000","last_became_peered":"2023-10-16T13:44:39.786906+0000","last_unstale":"2023-10-16T13:44:41.803510+0000","last_undegraded":"2023-10-16T13:44:41.803510+0000","last_fullsized":"2023-10-16T13:44:41.803510+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_clean_scrub_stamp":"2023-10-16T13:44:38.766670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:21:01.530053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.15","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.803576+0000","last_change":"2023-10-16T13:44:40.820881+0000","last_active":"2023-10-16T13:44:41.803576+0000","last_peered":"2023-10-16T13:44:41.803576+0000","last_clean":"2023-10-16T13:44:41.803576+0000","last_became_active":"2023-10-16T13:44:40.820547+0000","last_became_peered":"2023-10-16T13:44:40.820547+0000","last_unstale":"2023-10-16T13:44:41.803576+0000","last_undegraded":"2023-10-16T13:44:41.803576+0000","last_fullsized":"2023-10-16T13:44:41.803576+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:36:29.582161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":12,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.264384+0000","last_change":"2023-10-16T13:44:39.798951+0000","last_active":"2023-10-16T13:44:42.264384+0000","last_peered":"2023-10-16T13:44:42.264384+0000","last_clean":"2023-10-16T13:44:42.264384+0000","last_became_active":"2023-10-16T13:44:39.798595+0000","last_became_peered":"2023-10-16T13:44:39.798595+0000","last_unstale":"2023-10-16T13:44:42.264384+0000","last_undegraded":"2023-10-16T13:44:42.264384+0000","last_fullsized":"2023-10-16T13:44:42.264384+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_clean_scrub_stamp":"2023-10-16T13:44:38.766670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:21:12.180441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.264367+0000","last_change":"2023-10-16T13:44:40.818265+0000","last_active":"2023-10-16T13:44:42.264367+0000","last_peered":"2023-10-16T13:44:42.264367+0000","last_clean":"2023-10-16T13:44:42.264367+0000","last_became_active":"2023-10-16T13:44:40.817914+0000","last_became_peered":"2023-10-16T13:44:40.817914+0000","last_unstale":"2023-10-16T13:44:42.264367+0000","last_undegraded":"2023-10-16T13:44:42.264367+0000","last_fullsized":"2023-10-16T13:44:42.264367+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:29:27.837805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.15","version":"205'3","reported_seq":15,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.263970+0000","last_change":"2023-10-16T13:44:39.788873+0000","last_active":"2023-10-16T13:44:42.263970+0000","last_peered":"2023-10-16T13:44:42.263970+0000","last_clean":"2023-10-16T13:44:42.263970+0000","last_became_active":"2023-10-16T13:44:39.788569+0000","last_became_peered":"2023-10-16T13:44:39.788569+0000","last_unstale":"2023-10-16T13:44:42.263970+0000","last_undegraded":"2023-10-16T13:44:42.263970+0000","last_fullsized":"2023-10-16T13:44:42.263970+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_clean_scrub_stamp":"2023-10-16T13:44:38.766670+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:39:22.547238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.264530+0000","last_change":"2023-10-16T13:44:40.816037+0000","last_active":"2023-10-16T13:44:42.264530+0000","last_peered":"2023-10-16T13:44:42.264530+0000","last_clean":"2023-10-16T13:44:42.264530+0000","last_became_active":"2023-10-16T13:44:40.815712+0000","last_became_peered":"2023-10-16T13:44:40.815712+0000","last_unstale":"2023-10-16T13:44:42.264530+0000","last_undegraded":"2023-10-16T13:44:42.264530+0000","last_fullsized":"2023-10-16T13:44:42.264530+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:41:34.307413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.14","version":"205'6","reported_seq":18,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.264320+0000","last_change":"2023-10-16T13:44:39.792836+0000","last_active":"2023-10-16T13:44:42.264320+0000","last_peered":"2023-10-16T13:44:42.264320+0000","last_clean":"2023-10-16T13:44:42.264320+0000","last_became_active":"2023-10-16T13:44:39.792546+0000","last_became_peered":"2023-10-16T13:44:39.792546+0000","last_unstale":"2023-10-16T13:44:42.264320+0000","last_undegraded":"2023-10-16T13:44:42.264320+0000","last_fullsized":"2023-10-16T13:44:42.264320+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_clean_scrub_stamp":"2023-10-16T13:44:38.766670+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:11:58.401310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.10","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.801661+0000","last_change":"2023-10-16T13:44:40.826862+0000","last_active":"2023-10-16T13:44:41.801661+0000","last_peered":"2023-10-16T13:44:41.801661+0000","last_clean":"2023-10-16T13:44:41.801661+0000","last_became_active":"2023-10-16T13:44:40.825709+0000","last_became_peered":"2023-10-16T13:44:40.825709+0000","last_unstale":"2023-10-16T13:44:41.801661+0000","last_undegraded":"2023-10-16T13:44:41.801661+0000","last_fullsized":"2023-10-16T13:44:41.801661+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:06:12.484962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.13","version":"205'2","reported_seq":14,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.264516+0000","last_change":"2023-10-16T13:44:39.797595+0000","last_active":"2023-10-16T13:44:42.264516+0000","last_peered":"2023-10-16T13:44:42.264516+0000","last_clean":"2023-10-16T13:44:42.264516+0000","last_became_active":"2023-10-16T13:44:39.797344+0000","last_became_peered":"2023-10-16T13:44:39.797344+0000","last_unstale":"2023-10-16T13:44:42.264516+0000","last_undegraded":"2023-10-16T13:44:42.264516+0000","last_fullsized":"2023-10-16T13:44:42.264516+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_clean_scrub_stamp":"2023-10-16T13:44:38.766670+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:45:36.847018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.803113+0000","last_change":"2023-10-16T13:44:40.814628+0000","last_active":"2023-10-16T13:44:41.803113+0000","last_peered":"2023-10-16T13:44:41.803113+0000","last_clean":"2023-10-16T13:44:41.803113+0000","last_became_active":"2023-10-16T13:44:40.814339+0000","last_became_peered":"2023-10-16T13:44:40.814339+0000","last_unstale":"2023-10-16T13:44:41.803113+0000","last_undegraded":"2023-10-16T13:44:41.803113+0000","last_fullsized":"2023-10-16T13:44:41.803113+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:31:52.400499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":12,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.264226+0000","last_change":"2023-10-16T13:44:39.798272+0000","last_active":"2023-10-16T13:44:42.264226+0000","last_peered":"2023-10-16T13:44:42.264226+0000","last_clean":"2023-10-16T13:44:42.264226+0000","last_became_active":"2023-10-16T13:44:39.797982+0000","last_became_peered":"2023-10-16T13:44:39.797982+0000","last_unstale":"2023-10-16T13:44:42.264226+0000","last_undegraded":"2023-10-16T13:44:42.264226+0000","last_fullsized":"2023-10-16T13:44:42.264226+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_clean_scrub_stamp":"2023-10-16T13:44:38.766670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:27:15.853209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.264378+0000","last_change":"2023-10-16T13:44:40.822642+0000","last_active":"2023-10-16T13:44:42.264378+0000","last_peered":"2023-10-16T13:44:42.264378+0000","last_clean":"2023-10-16T13:44:42.264378+0000","last_became_active":"2023-10-16T13:44:40.822332+0000","last_became_peered":"2023-10-16T13:44:40.822332+0000","last_unstale":"2023-10-16T13:44:42.264378+0000","last_undegraded":"2023-10-16T13:44:42.264378+0000","last_fullsized":"2023-10-16T13:44:42.264378+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:13:11.470834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":12,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.263769+0000","last_change":"2023-10-16T13:44:39.791877+0000","last_active":"2023-10-16T13:44:42.263769+0000","last_peered":"2023-10-16T13:44:42.263769+0000","last_clean":"2023-10-16T13:44:42.263769+0000","last_became_active":"2023-10-16T13:44:39.791524+0000","last_became_peered":"2023-10-16T13:44:39.791524+0000","last_unstale":"2023-10-16T13:44:42.263769+0000","last_undegraded":"2023-10-16T13:44:42.263769+0000","last_fullsized":"2023-10-16T13:44:42.263769+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_clean_scrub_stamp":"2023-10-16T13:44:38.766670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:41:33.303177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.13","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.264385+0000","last_change":"2023-10-16T13:44:40.815297+0000","last_active":"2023-10-16T13:44:42.264385+0000","last_peered":"2023-10-16T13:44:42.264385+0000","last_clean":"2023-10-16T13:44:42.264385+0000","last_became_active":"2023-10-16T13:44:40.814931+0000","last_became_peered":"2023-10-16T13:44:40.814931+0000","last_unstale":"2023-10-16T13:44:42.264385+0000","last_undegraded":"2023-10-16T13:44:42.264385+0000","last_fullsized":"2023-10-16T13:44:42.264385+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:15:16.571866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.10","version":"205'8","reported_seq":20,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.264976+0000","last_change":"2023-10-16T13:44:39.790827+0000","last_active":"2023-10-16T13:44:42.264976+0000","last_peered":"2023-10-16T13:44:42.264976+0000","last_clean":"2023-10-16T13:44:42.264976+0000","last_became_active":"2023-10-16T13:44:39.790495+0000","last_became_peered":"2023-10-16T13:44:39.790495+0000","last_unstale":"2023-10-16T13:44:42.264976+0000","last_undegraded":"2023-10-16T13:44:42.264976+0000","last_fullsized":"2023-10-16T13:44:42.264976+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_clean_scrub_stamp":"2023-10-16T13:44:38.766670+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:55:38.777672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.802001+0000","last_change":"2023-10-16T13:44:40.813572+0000","last_active":"2023-10-16T13:44:41.802001+0000","last_peered":"2023-10-16T13:44:41.802001+0000","last_clean":"2023-10-16T13:44:41.802001+0000","last_became_active":"2023-10-16T13:44:40.813022+0000","last_became_peered":"2023-10-16T13:44:40.813022+0000","last_unstale":"2023-10-16T13:44:41.802001+0000","last_undegraded":"2023-10-16T13:44:41.802001+0000","last_fullsized":"2023-10-16T13:44:41.802001+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:49:43.725008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.f","version":"205'1","reported_seq":13,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.265266+0000","last_change":"2023-10-16T13:44:39.791453+0000","last_active":"2023-10-16T13:44:42.265266+0000","last_peered":"2023-10-16T13:44:42.265266+0000","last_clean":"2023-10-16T13:44:42.265266+0000","last_became_active":"2023-10-16T13:44:39.791134+0000","last_became_peered":"2023-10-16T13:44:39.791134+0000","last_unstale":"2023-10-16T13:44:42.265266+0000","last_undegraded":"2023-10-16T13:44:42.265266+0000","last_fullsized":"2023-10-16T13:44:42.265266+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_clean_scrub_stamp":"2023-10-16T13:44:38.766670+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:56:52.601574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.d","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.803535+0000","last_change":"2023-10-16T13:44:40.808177+0000","last_active":"2023-10-16T13:44:41.803535+0000","last_peered":"2023-10-16T13:44:41.803535+0000","last_clean":"2023-10-16T13:44:41.803535+0000","last_became_active":"2023-10-16T13:44:40.807851+0000","last_became_peered":"2023-10-16T13:44:40.807851+0000","last_unstale":"2023-10-16T13:44:41.803535+0000","last_undegraded":"2023-10-16T13:44:41.803535+0000","last_fullsized":"2023-10-16T13:44:41.803535+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:33:52.507520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.e","version":"205'2","reported_seq":14,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.264052+0000","last_change":"2023-10-16T13:44:39.792006+0000","last_active":"2023-10-16T13:44:42.264052+0000","last_peered":"2023-10-16T13:44:42.264052+0000","last_clean":"2023-10-16T13:44:42.264052+0000","last_became_active":"2023-10-16T13:44:39.791137+0000","last_became_peered":"2023-10-16T13:44:39.791137+0000","last_unstale":"2023-10-16T13:44:42.264052+0000","last_undegraded":"2023-10-16T13:44:42.264052+0000","last_fullsized":"2023-10-16T13:44:42.264052+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_clean_scrub_stamp":"2023-10-16T13:44:38.766670+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:19:16.883382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.e","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.265446+0000","last_change":"2023-10-16T13:44:40.808405+0000","last_active":"2023-10-16T13:44:42.265446+0000","last_peered":"2023-10-16T13:44:42.265446+0000","last_clean":"2023-10-16T13:44:42.265446+0000","last_became_active":"2023-10-16T13:44:40.808071+0000","last_became_peered":"2023-10-16T13:44:40.808071+0000","last_unstale":"2023-10-16T13:44:42.265446+0000","last_undegraded":"2023-10-16T13:44:42.265446+0000","last_fullsized":"2023-10-16T13:44:42.265446+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:17:19.764750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.d","version":"205'1","reported_seq":13,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.265344+0000","last_change":"2023-10-16T13:44:39.799230+0000","last_active":"2023-10-16T13:44:42.265344+0000","last_peered":"2023-10-16T13:44:42.265344+0000","last_clean":"2023-10-16T13:44:42.265344+0000","last_became_active":"2023-10-16T13:44:39.798885+0000","last_became_peered":"2023-10-16T13:44:39.798885+0000","last_unstale":"2023-10-16T13:44:42.265344+0000","last_undegraded":"2023-10-16T13:44:42.265344+0000","last_fullsized":"2023-10-16T13:44:42.265344+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_clean_scrub_stamp":"2023-10-16T13:44:38.766670+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:21:55.828297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.f","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.264603+0000","last_change":"2023-10-16T13:44:40.821116+0000","last_active":"2023-10-16T13:44:42.264603+0000","last_peered":"2023-10-16T13:44:42.264603+0000","last_clean":"2023-10-16T13:44:42.264603+0000","last_became_active":"2023-10-16T13:44:40.820753+0000","last_became_peered":"2023-10-16T13:44:40.820753+0000","last_unstale":"2023-10-16T13:44:42.264603+0000","last_undegraded":"2023-10-16T13:44:42.264603+0000","last_fullsized":"2023-10-16T13:44:42.264603+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:56:15.446259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":12,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.265619+0000","last_change":"2023-10-16T13:44:39.794797+0000","last_active":"2023-10-16T13:44:42.265619+0000","last_peered":"2023-10-16T13:44:42.265619+0000","last_clean":"2023-10-16T13:44:42.265619+0000","last_became_active":"2023-10-16T13:44:39.794494+0000","last_became_peered":"2023-10-16T13:44:39.794494+0000","last_unstale":"2023-10-16T13:44:42.265619+0000","last_undegraded":"2023-10-16T13:44:42.265619+0000","last_fullsized":"2023-10-16T13:44:42.265619+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_clean_scrub_stamp":"2023-10-16T13:44:38.766670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:21:12.180441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.263572+0000","last_change":"2023-10-16T13:44:40.809568+0000","last_active":"2023-10-16T13:44:42.263572+0000","last_peered":"2023-10-16T13:44:42.263572+0000","last_clean":"2023-10-16T13:44:42.263572+0000","last_became_active":"2023-10-16T13:44:40.809079+0000","last_became_peered":"2023-10-16T13:44:40.809079+0000","last_unstale":"2023-10-16T13:44:42.263572+0000","last_undegraded":"2023-10-16T13:44:42.263572+0000","last_fullsized":"2023-10-16T13:44:42.263572+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:56:53.605810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.b","version":"205'2","reported_seq":14,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.264616+0000","last_change":"2023-10-16T13:44:39.796244+0000","last_active":"2023-10-16T13:44:42.264616+0000","last_peered":"2023-10-16T13:44:42.264616+0000","last_clean":"2023-10-16T13:44:42.264616+0000","last_became_active":"2023-10-16T13:44:39.795941+0000","last_became_peered":"2023-10-16T13:44:39.795941+0000","last_unstale":"2023-10-16T13:44:42.264616+0000","last_undegraded":"2023-10-16T13:44:42.264616+0000","last_fullsized":"2023-10-16T13:44:42.264616+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_clean_scrub_stamp":"2023-10-16T13:44:38.766670+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:37:09.220924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.264368+0000","last_change":"2023-10-16T13:44:40.819695+0000","last_active":"2023-10-16T13:44:42.264368+0000","last_peered":"2023-10-16T13:44:42.264368+0000","last_clean":"2023-10-16T13:44:42.264368+0000","last_became_active":"2023-10-16T13:44:40.819397+0000","last_became_peered":"2023-10-16T13:44:40.819397+0000","last_unstale":"2023-10-16T13:44:42.264368+0000","last_undegraded":"2023-10-16T13:44:42.264368+0000","last_fullsized":"2023-10-16T13:44:42.264368+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:38:47.872141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":12,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.264028+0000","last_change":"2023-10-16T13:44:39.790396+0000","last_active":"2023-10-16T13:44:42.264028+0000","last_peered":"2023-10-16T13:44:42.264028+0000","last_clean":"2023-10-16T13:44:42.264028+0000","last_became_active":"2023-10-16T13:44:39.789911+0000","last_became_peered":"2023-10-16T13:44:39.789911+0000","last_unstale":"2023-10-16T13:44:42.264028+0000","last_undegraded":"2023-10-16T13:44:42.264028+0000","last_fullsized":"2023-10-16T13:44:42.264028+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_clean_scrub_stamp":"2023-10-16T13:44:38.766670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:28:24.113684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.263861+0000","last_change":"2023-10-16T13:44:40.819184+0000","last_active":"2023-10-16T13:44:42.263861+0000","last_peered":"2023-10-16T13:44:42.263861+0000","last_clean":"2023-10-16T13:44:42.263861+0000","last_became_active":"2023-10-16T13:44:40.818881+0000","last_became_peered":"2023-10-16T13:44:40.818881+0000","last_unstale":"2023-10-16T13:44:42.263861+0000","last_undegraded":"2023-10-16T13:44:42.263861+0000","last_fullsized":"2023-10-16T13:44:42.263861+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:05:44.245583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":12,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.801698+0000","last_change":"2023-10-16T13:44:39.795496+0000","last_active":"2023-10-16T13:44:41.801698+0000","last_peered":"2023-10-16T13:44:41.801698+0000","last_clean":"2023-10-16T13:44:41.801698+0000","last_became_active":"2023-10-16T13:44:39.795210+0000","last_became_peered":"2023-10-16T13:44:39.795210+0000","last_unstale":"2023-10-16T13:44:41.801698+0000","last_undegraded":"2023-10-16T13:44:41.801698+0000","last_fullsized":"2023-10-16T13:44:41.801698+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_clean_scrub_stamp":"2023-10-16T13:44:38.766670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:59:20.223638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.b","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.803322+0000","last_change":"2023-10-16T13:44:40.810076+0000","last_active":"2023-10-16T13:44:41.803322+0000","last_peered":"2023-10-16T13:44:41.803322+0000","last_clean":"2023-10-16T13:44:41.803322+0000","last_became_active":"2023-10-16T13:44:40.809585+0000","last_became_peered":"2023-10-16T13:44:40.809585+0000","last_unstale":"2023-10-16T13:44:41.803322+0000","last_undegraded":"2023-10-16T13:44:41.803322+0000","last_fullsized":"2023-10-16T13:44:41.803322+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:40:19.863783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":12,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.265496+0000","last_change":"2023-10-16T13:44:39.788432+0000","last_active":"2023-10-16T13:44:42.265496+0000","last_peered":"2023-10-16T13:44:42.265496+0000","last_clean":"2023-10-16T13:44:42.265496+0000","last_became_active":"2023-10-16T13:44:39.787791+0000","last_became_peered":"2023-10-16T13:44:39.787791+0000","last_unstale":"2023-10-16T13:44:42.265496+0000","last_undegraded":"2023-10-16T13:44:42.265496+0000","last_fullsized":"2023-10-16T13:44:42.265496+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_clean_scrub_stamp":"2023-10-16T13:44:38.766670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:12:45.268611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.802147+0000","last_change":"2023-10-16T13:44:40.813865+0000","last_active":"2023-10-16T13:44:41.802147+0000","last_peered":"2023-10-16T13:44:41.802147+0000","last_clean":"2023-10-16T13:44:41.802147+0000","last_became_active":"2023-10-16T13:44:40.813532+0000","last_became_peered":"2023-10-16T13:44:40.813532+0000","last_unstale":"2023-10-16T13:44:41.802147+0000","last_undegraded":"2023-10-16T13:44:41.802147+0000","last_fullsized":"2023-10-16T13:44:41.802147+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:32:39.343118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.7","version":"205'2","reported_seq":14,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.264210+0000","last_change":"2023-10-16T13:44:39.789533+0000","last_active":"2023-10-16T13:44:42.264210+0000","last_peered":"2023-10-16T13:44:42.264210+0000","last_clean":"2023-10-16T13:44:42.264210+0000","last_became_active":"2023-10-16T13:44:39.789225+0000","last_became_peered":"2023-10-16T13:44:39.789225+0000","last_unstale":"2023-10-16T13:44:42.264210+0000","last_undegraded":"2023-10-16T13:44:42.264210+0000","last_fullsized":"2023-10-16T13:44:42.264210+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_clean_scrub_stamp":"2023-10-16T13:44:38.766670+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:07:10.730051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.264118+0000","last_change":"2023-10-16T13:44:40.813440+0000","last_active":"2023-10-16T13:44:42.264118+0000","last_peered":"2023-10-16T13:44:42.264118+0000","last_clean":"2023-10-16T13:44:42.264118+0000","last_became_active":"2023-10-16T13:44:40.813147+0000","last_became_peered":"2023-10-16T13:44:40.813147+0000","last_unstale":"2023-10-16T13:44:42.264118+0000","last_undegraded":"2023-10-16T13:44:42.264118+0000","last_fullsized":"2023-10-16T13:44:42.264118+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:17:19.764750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.6","version":"0'0","reported_seq":12,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.802399+0000","last_change":"2023-10-16T13:44:39.795267+0000","last_active":"2023-10-16T13:44:41.802399+0000","last_peered":"2023-10-16T13:44:41.802399+0000","last_clean":"2023-10-16T13:44:41.802399+0000","last_became_active":"2023-10-16T13:44:39.794950+0000","last_became_peered":"2023-10-16T13:44:39.794950+0000","last_unstale":"2023-10-16T13:44:41.802399+0000","last_undegraded":"2023-10-16T13:44:41.802399+0000","last_fullsized":"2023-10-16T13:44:41.802399+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_clean_scrub_stamp":"2023-10-16T13:44:38.766670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:58:00.704123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.6","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.899473+0000","last_change":"2023-10-16T13:44:40.827414+0000","last_active":"2023-10-16T13:44:41.899473+0000","last_peered":"2023-10-16T13:44:41.899473+0000","last_clean":"2023-10-16T13:44:41.899473+0000","last_became_active":"2023-10-16T13:44:40.826996+0000","last_became_peered":"2023-10-16T13:44:40.826996+0000","last_unstale":"2023-10-16T13:44:41.899473+0000","last_undegraded":"2023-10-16T13:44:41.899473+0000","last_fullsized":"2023-10-16T13:44:41.899473+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:33:07.487629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":12,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.803591+0000","last_change":"2023-10-16T13:44:39.799361+0000","last_active":"2023-10-16T13:44:41.803591+0000","last_peered":"2023-10-16T13:44:41.803591+0000","last_clean":"2023-10-16T13:44:41.803591+0000","last_became_active":"2023-10-16T13:44:39.799109+0000","last_became_peered":"2023-10-16T13:44:39.799109+0000","last_unstale":"2023-10-16T13:44:41.803591+0000","last_undegraded":"2023-10-16T13:44:41.803591+0000","last_fullsized":"2023-10-16T13:44:41.803591+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_clean_scrub_stamp":"2023-10-16T13:44:38.766670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:03:57.432396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.7","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.802590+0000","last_change":"2023-10-16T13:44:40.813784+0000","last_active":"2023-10-16T13:44:41.802590+0000","last_peered":"2023-10-16T13:44:41.802590+0000","last_clean":"2023-10-16T13:44:41.802590+0000","last_became_active":"2023-10-16T13:44:40.813441+0000","last_became_peered":"2023-10-16T13:44:40.813441+0000","last_unstale":"2023-10-16T13:44:41.802590+0000","last_undegraded":"2023-10-16T13:44:41.802590+0000","last_fullsized":"2023-10-16T13:44:41.802590+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:23:06.700497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":12,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.803430+0000","last_change":"2023-10-16T13:44:39.796004+0000","last_active":"2023-10-16T13:44:41.803430+0000","last_peered":"2023-10-16T13:44:41.803430+0000","last_clean":"2023-10-16T13:44:41.803430+0000","last_became_active":"2023-10-16T13:44:39.795702+0000","last_became_peered":"2023-10-16T13:44:39.795702+0000","last_unstale":"2023-10-16T13:44:41.803430+0000","last_undegraded":"2023-10-16T13:44:41.803430+0000","last_fullsized":"2023-10-16T13:44:41.803430+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_clean_scrub_stamp":"2023-10-16T13:44:38.766670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:34:04.095991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.263936+0000","last_change":"2023-10-16T13:44:40.822780+0000","last_active":"2023-10-16T13:44:42.263936+0000","last_peered":"2023-10-16T13:44:42.263936+0000","last_clean":"2023-10-16T13:44:42.263936+0000","last_became_active":"2023-10-16T13:44:40.822388+0000","last_became_peered":"2023-10-16T13:44:40.822388+0000","last_unstale":"2023-10-16T13:44:42.263936+0000","last_undegraded":"2023-10-16T13:44:42.263936+0000","last_fullsized":"2023-10-16T13:44:42.263936+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:29:27.837805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":12,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.899182+0000","last_change":"2023-10-16T13:44:39.794644+0000","last_active":"2023-10-16T13:44:41.899182+0000","last_peered":"2023-10-16T13:44:41.899182+0000","last_clean":"2023-10-16T13:44:41.899182+0000","last_became_active":"2023-10-16T13:44:39.794272+0000","last_became_peered":"2023-10-16T13:44:39.794272+0000","last_unstale":"2023-10-16T13:44:41.899182+0000","last_undegraded":"2023-10-16T13:44:41.899182+0000","last_fullsized":"2023-10-16T13:44:41.899182+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_clean_scrub_stamp":"2023-10-16T13:44:38.766670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:07:10.730051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.264583+0000","last_change":"2023-10-16T13:44:40.815288+0000","last_active":"2023-10-16T13:44:42.264583+0000","last_peered":"2023-10-16T13:44:42.264583+0000","last_clean":"2023-10-16T13:44:42.264583+0000","last_became_active":"2023-10-16T13:44:40.814982+0000","last_became_peered":"2023-10-16T13:44:40.814982+0000","last_unstale":"2023-10-16T13:44:42.264583+0000","last_undegraded":"2023-10-16T13:44:42.264583+0000","last_fullsized":"2023-10-16T13:44:42.264583+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:33:19.386232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":12,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.899057+0000","last_change":"2023-10-16T13:44:39.799090+0000","last_active":"2023-10-16T13:44:41.899057+0000","last_peered":"2023-10-16T13:44:41.899057+0000","last_clean":"2023-10-16T13:44:41.899057+0000","last_became_active":"2023-10-16T13:44:39.798655+0000","last_became_peered":"2023-10-16T13:44:39.798655+0000","last_unstale":"2023-10-16T13:44:41.899057+0000","last_undegraded":"2023-10-16T13:44:41.899057+0000","last_fullsized":"2023-10-16T13:44:41.899057+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_clean_scrub_stamp":"2023-10-16T13:44:38.766670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:39:56.910883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.264176+0000","last_change":"2023-10-16T13:44:40.821533+0000","last_active":"2023-10-16T13:44:42.264176+0000","last_peered":"2023-10-16T13:44:42.264176+0000","last_clean":"2023-10-16T13:44:42.264176+0000","last_became_active":"2023-10-16T13:44:40.820856+0000","last_became_peered":"2023-10-16T13:44:40.820856+0000","last_unstale":"2023-10-16T13:44:42.264176+0000","last_undegraded":"2023-10-16T13:44:42.264176+0000","last_fullsized":"2023-10-16T13:44:42.264176+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:53:07.234210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":12,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.801888+0000","last_change":"2023-10-16T13:44:39.791606+0000","last_active":"2023-10-16T13:44:41.801888+0000","last_peered":"2023-10-16T13:44:41.801888+0000","last_clean":"2023-10-16T13:44:41.801888+0000","last_became_active":"2023-10-16T13:44:39.791258+0000","last_became_peered":"2023-10-16T13:44:39.791258+0000","last_unstale":"2023-10-16T13:44:41.801888+0000","last_undegraded":"2023-10-16T13:44:41.801888+0000","last_fullsized":"2023-10-16T13:44:41.801888+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_clean_scrub_stamp":"2023-10-16T13:44:38.766670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:29:27.824587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:41.899210+0000","last_change":"2023-10-16T13:44:40.830044+0000","last_active":"2023-10-16T13:44:41.899210+0000","last_peered":"2023-10-16T13:44:41.899210+0000","last_clean":"2023-10-16T13:44:41.899210+0000","last_became_active":"2023-10-16T13:44:40.829735+0000","last_became_peered":"2023-10-16T13:44:40.829735+0000","last_unstale":"2023-10-16T13:44:41.899210+0000","last_undegraded":"2023-10-16T13:44:41.899210+0000","last_fullsized":"2023-10-16T13:44:41.899210+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:42:50.455035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":12,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.265571+0000","last_change":"2023-10-16T13:44:39.791696+0000","last_active":"2023-10-16T13:44:42.265571+0000","last_peered":"2023-10-16T13:44:42.265571+0000","last_clean":"2023-10-16T13:44:42.265571+0000","last_became_active":"2023-10-16T13:44:39.791371+0000","last_became_peered":"2023-10-16T13:44:39.791371+0000","last_unstale":"2023-10-16T13:44:42.265571+0000","last_undegraded":"2023-10-16T13:44:42.265571+0000","last_fullsized":"2023-10-16T13:44:42.265571+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_clean_scrub_stamp":"2023-10-16T13:44:38.766670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:24:22.879467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.1b","version":"0'0","reported_seq":11,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.265483+0000","last_change":"2023-10-16T13:44:40.823583+0000","last_active":"2023-10-16T13:44:42.265483+0000","last_peered":"2023-10-16T13:44:42.265483+0000","last_clean":"2023-10-16T13:44:42.265483+0000","last_became_active":"2023-10-16T13:44:40.823308+0000","last_became_peered":"2023-10-16T13:44:40.823308+0000","last_unstale":"2023-10-16T13:44:42.265483+0000","last_undegraded":"2023-10-16T13:44:42.265483+0000","last_fullsized":"2023-10-16T13:44:42.265483+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:39.770906+0000","last_clean_scrub_stamp":"2023-10-16T13:44:39.770906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:15:16.571866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":12,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-16T13:44:42.263814+0000","last_change":"2023-10-16T13:44:39.787527+0000","last_active":"2023-10-16T13:44:42.263814+0000","last_peered":"2023-10-16T13:44:42.263814+0000","last_clean":"2023-10-16T13:44:42.263814+0000","last_became_active":"2023-10-16T13:44:39.787221+0000","last_became_peered":"2023-10-16T13:44:39.787221+0000","last_unstale":"2023-10-16T13:44:42.263814+0000","last_undegraded":"2023-10-16T13:44:42.263814+0000","last_fullsized":"2023-10-16T13:44:42.263814+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:44:38.766670+0000","last_clean_scrub_stamp":"2023-10-16T13:44:38.766670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:39:56.910883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"196'326","reported_seq":1107,"reported_epoch":205,"state":"active+clean","last_fresh":"2023-10-16T13:44:40.821237+0000","last_change":"2023-10-16T12:10:34.746820+0000","last_active":"2023-10-16T13:44:40.821237+0000","last_peered":"2023-10-16T13:44:40.821237+0000","last_clean":"2023-10-16T13:44:40.821237+0000","last_became_active":"2023-10-16T12:10:34.745574+0000","last_became_peered":"2023-10-16T12:10:34.745574+0000","last_unstale":"2023-10-16T13:44:40.821237+0000","last_undegraded":"2023-10-16T13:44:40.821237+0000","last_fullsized":"2023-10-16T13:44:40.821237+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_clean_scrub_stamp":"2023-10-16T12:10:33.577111+0000","objects_scrubbed":0,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:14.461986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1940,"num_read_kb":4014,"num_write":539,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":26,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":25,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1940,"num_read_kb":4014,"num_write":539,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":326,"ondisk_log_size":326,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575967,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56480,"kb_used_data":3092,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315360,"statfs":{"total":96636764160,"available":96578928640,"internally_reserved":0,"allocated":3166208,"data_stored":1906722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54655435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:43:42 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.63400000000000001,"15min":0.64200000000000002},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.33000000000000002},"max":{"1min":0.87,"5min":0.90500000000000003,"15min":0.96199999999999997},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.61499999999999999,"15min":0.63600000000000001},"min":{"1min":0.51500000000000001,"5min":0.41099999999999998,"15min":0.379},"max":{"1min":0.71999999999999997,"5min":0.82999999999999996,"15min":0.91200000000000003},"last":1.1879999999999999}]},{"osd":1,"last update":"Mon Oct 16 13:43:42 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63200000000000001,"15min":0.64900000000000002},"min":{"1min":0.45000000000000001,"5min":0.39600000000000002,"15min":0.375},"max":{"1min":0.80400000000000005,"5min":0.88700000000000001,"15min":0.95899999999999996},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.64500000000000002,"15min":0.64300000000000002},"min":{"1min":0.55600000000000005,"5min":0.42099999999999999,"15min":0.40400000000000003},"max":{"1min":0.77600000000000002,"5min":0.83599999999999997,"15min":1.022},"last":0.44400000000000001}]},{"osd":2,"last update":"Mon Oct 16 13:43:42 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.69499999999999995,"15min":0.69799999999999995},"min":{"1min":0.57699999999999996,"5min":0.51800000000000002,"15min":0.44900000000000001},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.93500000000000005},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.70199999999999996,"15min":0.70699999999999996},"min":{"1min":0.56799999999999995,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.84499999999999997,"5min":0.92800000000000005,"15min":0.94199999999999995},"last":0.50900000000000001}]},{"osd":3,"last update":"Mon Oct 16 13:43:42 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.66300000000000003,"15min":0.66800000000000004},"min":{"1min":0.53600000000000003,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.82299999999999995,"5min":0.89200000000000002,"15min":0.91300000000000003},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.65700000000000003,"15min":0.66500000000000004},"min":{"1min":0.53500000000000003,"5min":0.439,"15min":0.40100000000000002},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.98299999999999998},"last":1.24}]},{"osd":4,"last update":"Mon Oct 16 13:43:42 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.51300000000000001,"15min":0.52800000000000002},"min":{"1min":0.38100000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.66900000000000004,"5min":0.77900000000000003,"15min":0.94599999999999995},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.624,"5min":0.60199999999999998,"15min":0.623},"min":{"1min":0.49199999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.92900000000000005},"last":0.59099999999999997}]},{"osd":5,"last update":"Mon Oct 16 13:43:42 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.53100000000000003,"15min":0.54300000000000004},"min":{"1min":0.42599999999999999,"5min":0.35999999999999999,"15min":0.32500000000000001},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.90000000000000002},"last":1.2829999999999999},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.57499999999999996,"15min":0.59099999999999997},"min":{"1min":0.45500000000000002,"5min":0.38400000000000001,"15min":0.35099999999999998},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.65400000000000003}]},{"osd":6,"last update":"Mon Oct 16 13:43:42 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.59399999999999997,"15min":0.621},"min":{"1min":0.46300000000000002,"5min":0.40500000000000003,"15min":0.378},"max":{"1min":0.874,"5min":0.874,"15min":0.95899999999999996},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.59999999999999998,"15min":0.61899999999999999},"min":{"1min":0.44800000000000001,"5min":0.38700000000000001,"15min":0.35199999999999998},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.97499999999999998},"last":0.63900000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575965,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42016,"kb_used_data":2516,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329824,"statfs":{"total":96636764160,"available":96593739776,"internally_reserved":0,"allocated":2576384,"data_stored":1316424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:44:33 2023","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.67600000000000005,"15min":0.67900000000000005},"min":{"1min":0.39000000000000001,"5min":0.375,"15min":0.28699999999999998},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.93600000000000005},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.66500000000000004,"15min":0.67300000000000004},"min":{"1min":0.52200000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.85499999999999998,"5min":0.88600000000000001,"15min":0.93799999999999994},"last":0.61499999999999999}]},{"osd":1,"last update":"Mon Oct 16 13:44:33 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.629,"15min":0.63800000000000001},"min":{"1min":0.41399999999999998,"5min":0.36599999999999999,"15min":0.33100000000000002},"max":{"1min":0.81399999999999995,"5min":0.85999999999999999,"15min":0.90900000000000003},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.61199999999999999,"15min":0.625},"min":{"1min":0.437,"5min":0.38500000000000001,"15min":0.38300000000000001},"max":{"1min":0.88500000000000001,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.55300000000000005}]},{"osd":2,"last update":"Mon Oct 16 13:44:33 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.622,"15min":0.63700000000000001},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.378},"max":{"1min":0.878,"5min":0.878,"15min":0.879},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.624,"15min":0.63500000000000001},"min":{"1min":0.45400000000000001,"5min":0.38700000000000001,"15min":0.379},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.91000000000000003},"last":0.69799999999999995}]},{"osd":3,"last update":"Mon Oct 16 13:44:33 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66800000000000004,"15min":0.67200000000000004},"min":{"1min":0.496,"5min":0.39900000000000002,"15min":0.38200000000000001},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.998},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.71399999999999997,"15min":0.72599999999999998},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.42299999999999999},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96899999999999997},"last":0.72099999999999997}]},{"osd":4,"last update":"Mon Oct 16 13:44:33 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.52600000000000002,"15min":0.52700000000000002},"min":{"1min":0.439,"5min":0.32600000000000001,"15min":0.315},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.95799999999999996},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.624,"15min":0.621},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.32900000000000001},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":1.0309999999999999},"last":0.75800000000000001}]},{"osd":5,"last update":"Mon Oct 16 13:44:33 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.59999999999999998,"15min":0.59199999999999997},"min":{"1min":0.46999999999999997,"5min":0.39400000000000002,"15min":0.34899999999999998},"max":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.67000000000000004,"15min":0.66100000000000003},"min":{"1min":0.40600000000000003,"5min":0.371,"15min":0.30599999999999999},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.77500000000000002}]},{"osd":7,"last update":"Mon Oct 16 13:44:33 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.61199999999999999,"15min":0.60499999999999998},"min":{"1min":0.41699999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.081,"5min":1.081,"15min":1.081},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.66200000000000003,"15min":0.65100000000000002},"min":{"1min":0.443,"5min":0.379,"15min":0.375},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.79700000000000004}]}]},{"osd":1,"up_from":13,"seq":55834575965,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47580,"kb_used_data":2512,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324260,"statfs":{"total":96636764160,"available":96588042240,"internally_reserved":0,"allocated":2572288,"data_stored":1316354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":46135753},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:43:51 2023","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.41099999999999998,"15min":0.41199999999999998},"min":{"1min":0.28599999999999998,"5min":0.251,"15min":0.251},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.71999999999999997},"last":0.436},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.496,"15min":0.498},"min":{"1min":0.42299999999999999,"5min":0.318,"15min":0.28799999999999998},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.77200000000000002},"last":0.45500000000000002}]},{"osd":2,"last update":"Mon Oct 16 13:43:51 2023","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.45900000000000002,"15min":0.45700000000000002},"min":{"1min":0.29099999999999998,"5min":0.28799999999999998,"15min":0.26600000000000001},"max":{"1min":0.63300000000000001,"5min":0.67200000000000004,"15min":0.751},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.48699999999999999,"15min":0.495},"min":{"1min":0.317,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.66600000000000004,"5min":0.68700000000000006,"15min":0.71399999999999997},"last":0.498}]},{"osd":3,"last update":"Mon Oct 16 13:43:51 2023","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.435,"15min":0.42899999999999999},"min":{"1min":0.34300000000000003,"5min":0.315,"15min":0.28799999999999998},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.77500000000000002},"last":0.373},{"interface":"front","average":{"1min":0.502,"5min":0.47999999999999998,"15min":0.48299999999999998},"min":{"1min":0.32400000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.39400000000000002}]},{"osd":4,"last update":"Mon Oct 16 13:43:51 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.67600000000000005,"15min":0.67000000000000004},"min":{"1min":0.48799999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.94499999999999995,"5min":0.95899999999999996,"15min":1.0069999999999999},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.67600000000000005,"15min":0.67500000000000004},"min":{"1min":0.46899999999999997,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.91300000000000003,"5min":0.92800000000000005,"15min":0.997},"last":0.72799999999999998}]},{"osd":5,"last update":"Mon Oct 16 13:43:51 2023","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.71799999999999997,"15min":0.71799999999999997},"min":{"1min":0.54200000000000004,"5min":0.53200000000000003,"15min":0.41199999999999998},"max":{"1min":0.93300000000000005,"5min":0.97499999999999998,"15min":1.0760000000000001},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.746,"15min":0.73499999999999999},"min":{"1min":0.55900000000000005,"5min":0.48999999999999999,"15min":0.46200000000000002},"max":{"1min":1.028,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.76000000000000001}]},{"osd":6,"last update":"Mon Oct 16 13:43:51 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.69199999999999995,"15min":0.68700000000000006},"min":{"1min":0.56299999999999994,"5min":0.432,"15min":0.42099999999999999},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.751,"5min":0.71699999999999997,"15min":0.71499999999999997},"min":{"1min":0.57999999999999996,"5min":0.47799999999999998,"15min":0.46899999999999997},"max":{"1min":0.998,"5min":1.03,"15min":1.083},"last":0.71599999999999997}]},{"osd":7,"last update":"Mon Oct 16 13:43:51 2023","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.755,"15min":0.747},"min":{"1min":0.55500000000000005,"5min":0.497,"15min":0.47399999999999998},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0660000000000001},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.76000000000000001,"15min":0.753},"min":{"1min":0.54700000000000004,"5min":0.505,"15min":0.44600000000000001},"max":{"1min":1.042,"5min":1.042,"15min":1.0580000000000001},"last":0.752}]}]},{"osd":0,"up_from":13,"seq":55834575966,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30132,"kb_used_data":3108,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341708,"statfs":{"total":96636764160,"available":96605908992,"internally_reserved":0,"allocated":3182592,"data_stored":1907450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 13:44:39 2023","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.41599999999999998,"15min":0.42499999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.51600000000000001,"5min":0.66400000000000003,"15min":0.71399999999999997},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.438,"15min":0.44600000000000001},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.25900000000000001},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.82199999999999995},"last":0.30099999999999999}]},{"osd":2,"last update":"Mon Oct 16 13:44:39 2023","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.44800000000000001,"15min":0.45900000000000002},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.56699999999999995,"5min":0.64200000000000002,"15min":0.75700000000000001},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.435,"5min":0.48299999999999998,"15min":0.48999999999999999},"min":{"1min":0.33700000000000002,"5min":0.33100000000000002,"15min":0.32600000000000001},"max":{"1min":0.60099999999999998,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.34699999999999998}]},{"osd":3,"last update":"Mon Oct 16 13:44:39 2023","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.44700000000000001,"15min":0.45700000000000002},"min":{"1min":0.29899999999999999,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.58299999999999996,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.373},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.52400000000000002,"15min":0.52900000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.69399999999999995,"5min":0.82999999999999996,"15min":0.86399999999999999},"last":0.53000000000000003}]},{"osd":4,"last update":"Mon Oct 16 13:44:39 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.67400000000000004,"15min":0.68700000000000006},"min":{"1min":0.48199999999999998,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.87,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.625,"5min":0.66900000000000004,"15min":0.69399999999999995},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40200000000000002},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":1.0049999999999999},"last":0.54800000000000004}]},{"osd":5,"last update":"Mon Oct 16 13:44:39 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.68200000000000005,"15min":0.69399999999999995},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.84199999999999997,"5min":0.95999999999999996,"15min":1.0249999999999999},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.70899999999999996,"15min":0.73099999999999998},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.92100000000000004,"5min":0.998,"15min":1.0720000000000001},"last":0.59099999999999997}]},{"osd":6,"last update":"Mon Oct 16 13:44:39 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.72399999999999998,"15min":0.73699999999999999},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.879,"5min":1.103,"15min":1.103},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.71299999999999997,"15min":0.72799999999999998},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.93899999999999995,"5min":1.0409999999999999,"15min":1.05},"last":0.42999999999999999}]},{"osd":7,"last update":"Mon Oct 16 13:44:39 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.73599999999999999,"15min":0.75},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.91100000000000003,"5min":0.98299999999999998,"15min":1.056},"last":0.498},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.72399999999999998,"15min":0.73799999999999999},"min":{"1min":0.49299999999999999,"5min":0.48699999999999999,"15min":0.45400000000000001},"max":{"1min":0.89500000000000002,"5min":1.087,"15min":1.087},"last":0.51500000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575966,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51764,"kb_used_data":2528,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320076,"statfs":{"total":96636764160,"available":96583757824,"internally_reserved":0,"allocated":2588672,"data_stored":1323684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:44:35 2023","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.42999999999999999,"15min":0.41499999999999998},"min":{"1min":0.27700000000000002,"5min":0.251,"15min":0.24099999999999999},"max":{"1min":0.61499999999999999,"5min":0.71699999999999997,"15min":0.77500000000000002},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.495,"15min":0.48499999999999999},"min":{"1min":0.36799999999999999,"5min":0.35999999999999999,"15min":0.29599999999999999},"max":{"1min":0.77500000000000002,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.32400000000000001}]},{"osd":1,"last update":"Mon Oct 16 13:44:35 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.51900000000000002,"15min":0.50700000000000001},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.69999999999999996,"5min":0.73199999999999998,"15min":0.81399999999999995},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.503,"15min":0.48199999999999998},"min":{"1min":0.33700000000000002,"5min":0.29899999999999999,"15min":0.27700000000000002},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.35399999999999998}]},{"osd":3,"last update":"Mon Oct 16 13:44:35 2023","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.45600000000000002,"15min":0.44600000000000001},"min":{"1min":0.31,"5min":0.29099999999999998,"15min":0.28399999999999997},"max":{"1min":0.58499999999999996,"5min":0.70099999999999996,"15min":0.86699999999999999},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.56999999999999995,"15min":0.55200000000000005},"min":{"1min":0.40500000000000003,"5min":0.371,"15min":0.309},"max":{"1min":0.74199999999999999,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.39100000000000001}]},{"osd":4,"last update":"Mon Oct 16 13:44:35 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.748,"15min":0.72899999999999998},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.91300000000000003,"5min":1.3600000000000001,"15min":1.3600000000000001},"last":0.625},{"interface":"front","average":{"1min":0.751,"5min":0.75700000000000001,"15min":0.73399999999999999},"min":{"1min":0.47799999999999998,"5min":0.45700000000000002,"15min":0.44900000000000001},"max":{"1min":1.3340000000000001,"5min":1.403,"15min":1.403},"last":0.61399999999999999}]},{"osd":5,"last update":"Mon Oct 16 13:44:35 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.72599999999999998,"15min":0.70099999999999996},"min":{"1min":0.505,"5min":0.505,"15min":0.441},"max":{"1min":0.87,"5min":1.486,"15min":1.486},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.73799999999999999,"15min":0.71199999999999997},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":1.3660000000000001,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":0.66500000000000004}]},{"osd":6,"last update":"Mon Oct 16 13:44:35 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.70699999999999996,"15min":0.67700000000000005},"min":{"1min":0.50700000000000001,"5min":0.48999999999999999,"15min":0.46800000000000003},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.755,"15min":0.73599999999999999},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.88600000000000001,"5min":1.4379999999999999,"15min":1.4379999999999999},"last":0.64800000000000002}]},{"osd":7,"last update":"Mon Oct 16 13:44:35 2023","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.76300000000000001,"15min":0.73199999999999998},"min":{"1min":0.53400000000000003,"5min":0.52000000000000002,"15min":0.47899999999999998},"max":{"1min":1.4370000000000001,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.76400000000000001,"15min":0.748},"min":{"1min":0.54600000000000004,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.94499999999999995,"5min":1.377,"15min":1.377},"last":0.56399999999999995}]}]},{"osd":3,"up_from":13,"seq":55834575965,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51760,"kb_used_data":2532,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320080,"statfs":{"total":96636764160,"available":96583761920,"internally_reserved":0,"allocated":2592768,"data_stored":1323262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50395592},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:44:38 2023","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.40899999999999997,"15min":0.41599999999999998},"min":{"1min":0.25,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.58399999999999996,"5min":0.64200000000000002,"15min":0.77700000000000002},"last":0.436},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.42999999999999999,"15min":0.435},"min":{"1min":0.27400000000000002,"5min":0.252,"15min":0.252},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.90100000000000002},"last":0.504}]},{"osd":1,"last update":"Mon Oct 16 13:44:38 2023","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.439,"15min":0.441},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.66500000000000004,"5min":0.70499999999999996,"15min":0.76300000000000001},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.47499999999999998,"15min":0.48399999999999999},"min":{"1min":0.36899999999999999,"5min":0.253,"15min":0.253},"max":{"1min":0.58899999999999997,"5min":0.71499999999999997,"15min":0.77800000000000002},"last":0.58299999999999996}]},{"osd":2,"last update":"Mon Oct 16 13:44:38 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.501,"15min":0.5},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":1.167,"5min":1.167,"15min":1.167},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.52400000000000002,"15min":0.53000000000000003},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.65500000000000003,"5min":0.76800000000000002,"15min":0.78700000000000003},"last":0.67600000000000005}]},{"osd":4,"last update":"Mon Oct 16 13:44:38 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.66300000000000003,"15min":0.67400000000000004},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.85999999999999999,"5min":1.137,"15min":1.137},"last":0.98499999999999999},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.70499999999999996,"15min":0.70099999999999996},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46100000000000002},"max":{"1min":1.2110000000000001,"5min":1.2909999999999999,"15min":1.2909999999999999},"last":0.72899999999999998}]},{"osd":5,"last update":"Mon Oct 16 13:44:38 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.71299999999999997,"15min":0.71599999999999997},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.374},"max":{"1min":0.91300000000000003,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.71399999999999997,"15min":0.70499999999999996},"min":{"1min":0.46100000000000002,"5min":0.432,"15min":0.432},"max":{"1min":0.81200000000000006,"5min":1.173,"15min":1.173},"last":1.0549999999999999}]},{"osd":6,"last update":"Mon Oct 16 13:44:38 2023","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.755,"15min":0.72899999999999998},"min":{"1min":0.51000000000000001,"5min":0.47099999999999997,"15min":0.41799999999999998},"max":{"1min":5.0999999999999996,"5min":5.0999999999999996,"15min":5.0999999999999996},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.73499999999999999,"15min":0.73799999999999999},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.871,"5min":1.2290000000000001,"15min":1.2290000000000001},"last":1.0309999999999999}]},{"osd":7,"last update":"Mon Oct 16 13:44:38 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.73399999999999999,"15min":0.73999999999999999},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.85699999999999998,"5min":1.2569999999999999,"15min":1.2569999999999999},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.75900000000000001,"15min":0.749},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.43099999999999999},"max":{"1min":1.262,"5min":1.262,"15min":1.262},"last":0.95999999999999996}]}]},{"osd":4,"up_from":13,"seq":55834575964,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51760,"kb_used_data":2528,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320080,"statfs":{"total":96636764160,"available":96583761920,"internally_reserved":0,"allocated":2588672,"data_stored":1318000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.628,"15min":0.627},"min":{"1min":0.37,"5min":0.36299999999999999,"15min":0.33800000000000002},"max":{"1min":0.72999999999999998,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.64400000000000002,"15min":0.63800000000000001},"min":{"1min":0.46400000000000002,"5min":0.41499999999999998,"15min":0.35499999999999998},"max":{"1min":0.81499999999999995,"5min":0.878,"15min":0.95599999999999996},"last":0.57199999999999995}]},{"osd":1,"last update":"Mon Oct 16 13:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.65500000000000003,"15min":0.66000000000000003},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.375},"max":{"1min":0.83099999999999996,"5min":0.85099999999999998,"15min":0.89300000000000002},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.64600000000000002,"15min":0.64200000000000002},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.79200000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.63800000000000001}]},{"osd":2,"last update":"Mon Oct 16 13:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.67300000000000004,"15min":0.65600000000000003},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.39600000000000002},"max":{"1min":0.88500000000000001,"5min":0.89400000000000002,"15min":0.90600000000000003},"last":0.622},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66000000000000003},"min":{"1min":0.40799999999999997,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.92600000000000005},"last":0.73099999999999998}]},{"osd":3,"last update":"Mon Oct 16 13:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.39500000000000002},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.99199999999999999},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.68700000000000006,"15min":0.68000000000000005},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.83699999999999997,"5min":0.90700000000000003,"15min":0.97399999999999998},"last":0.67300000000000004}]},{"osd":5,"last update":"Mon Oct 16 13:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.37,"5min":0.34999999999999998,"15min":0.30599999999999999},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.91700000000000004},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.55700000000000005,"15min":0.56599999999999995},"min":{"1min":0.377,"5min":0.33800000000000002,"15min":0.33600000000000002},"max":{"1min":0.80500000000000005,"5min":0.82299999999999995,"15min":0.94899999999999995},"last":0.77000000000000002}]},{"osd":6,"last update":"Mon Oct 16 13:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.59299999999999997,"15min":0.59099999999999997},"min":{"1min":0.434,"5min":0.34499999999999997,"15min":0.33200000000000002},"max":{"1min":0.877,"5min":0.877,"15min":0.94299999999999995},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.68400000000000005,"15min":0.68799999999999994},"min":{"1min":0.51300000000000001,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.95299999999999996},"last":0.85699999999999998}]},{"osd":7,"last update":"Mon Oct 16 13:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.626,"15min":0.627},"min":{"1min":0.44600000000000001,"5min":0.39900000000000002,"15min":0.35799999999999998},"max":{"1min":0.85799999999999998,"5min":0.97499999999999998,"15min":0.98599999999999999},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.66900000000000004,"15min":0.67600000000000005},"min":{"1min":0.52900000000000003,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":1.026},"last":0.754}]}]},{"osd":5,"up_from":13,"seq":55834575965,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51756,"kb_used_data":2528,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320084,"statfs":{"total":96636764160,"available":96583766016,"internally_reserved":0,"allocated":2588672,"data_stored":1317988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:44:22 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.63800000000000001,"15min":0.622},"min":{"1min":0.51800000000000002,"5min":0.30499999999999999,"15min":0.26800000000000002},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.64200000000000002,"15min":0.64600000000000002},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.29699999999999999},"max":{"1min":0.80900000000000005,"5min":0.93700000000000006,"15min":0.96099999999999997},"last":0.56100000000000005}]},{"osd":1,"last update":"Mon Oct 16 13:44:22 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.67000000000000004,"15min":0.66100000000000003},"min":{"1min":0.42499999999999999,"5min":0.35899999999999999,"15min":0.29499999999999998},"max":{"1min":0.83599999999999997,"5min":0.873,"15min":0.89500000000000002},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66000000000000003,"15min":0.64900000000000002},"min":{"1min":0.44700000000000001,"5min":0.40799999999999997,"15min":0.38400000000000001},"max":{"1min":0.79500000000000004,"5min":0.871,"15min":0.94999999999999996},"last":0.51400000000000001}]},{"osd":2,"last update":"Mon Oct 16 13:44:22 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.67800000000000005,"15min":0.66200000000000003},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.38800000000000001},"max":{"1min":0.91000000000000003,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.68000000000000005,"15min":0.65300000000000002},"min":{"1min":0.52500000000000002,"5min":0.502,"15min":0.40200000000000002},"max":{"1min":0.871,"5min":0.89400000000000002,"15min":0.93500000000000005},"last":0.59299999999999997}]},{"osd":3,"last update":"Mon Oct 16 13:44:22 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.66900000000000004},"min":{"1min":0.46300000000000002,"5min":0.46000000000000002,"15min":0.45000000000000001},"max":{"1min":0.88900000000000001,"5min":0.90100000000000002,"15min":0.94999999999999996},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.70899999999999996,"15min":0.69499999999999995},"min":{"1min":0.59599999999999997,"5min":0.51700000000000002,"15min":0.46700000000000003},"max":{"1min":0.876,"5min":0.876,"15min":0.96999999999999997},"last":0.70499999999999996}]},{"osd":4,"last update":"Mon Oct 16 13:44:22 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.56000000000000005,"15min":0.53200000000000003},"min":{"1min":0.372,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.77900000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.60699999999999998,"15min":0.58899999999999997},"min":{"1min":0.39500000000000002,"5min":0.38800000000000001,"15min":0.34699999999999998},"max":{"1min":0.748,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.79000000000000004}]},{"osd":6,"last update":"Mon Oct 16 13:44:22 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.59499999999999997,"15min":0.57799999999999996},"min":{"1min":0.41399999999999998,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.76300000000000001,"5min":0.88900000000000001,"15min":0.93200000000000005},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.63600000000000001,"15min":0.61699999999999999},"min":{"1min":0.44900000000000001,"5min":0.42299999999999999,"15min":0.39000000000000001},"max":{"1min":0.82599999999999996,"5min":0.876,"15min":0.89600000000000002},"last":0.746}]},{"osd":7,"last update":"Mon Oct 16 13:44:22 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.64200000000000002,"15min":0.628},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.76700000000000002,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.70099999999999996,"15min":0.68000000000000005},"min":{"1min":0.46200000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.84999999999999998,"5min":0.98299999999999998,"15min":1.0229999999999999},"last":0.82499999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T13:44:45.392 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-16T13:44:45.392 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-16T13:44:45.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.555+0000 7f787c9c6700 1 Processor -- start 2023-10-16T13:44:45.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.555+0000 7f787c9c6700 1 -- start start 2023-10-16T13:44:45.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.556+0000 7f787c9c6700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f787815b2c0 0x7f7878161340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:45.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.556+0000 7f787c9c6700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f787806a7e0 0x7f787806ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:45.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.556+0000 7f787c9c6700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f787815ecb0 con 0x7f787806e570 2023-10-16T13:44:45.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.556+0000 7f787c9c6700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f787806b1b0 con 0x7f787806a7e0 2023-10-16T13:44:45.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.556+0000 7f787c9c6700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7878161880 con 0x7f787815b2c0 2023-10-16T13:44:45.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.557+0000 7f7876d9d700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f787806a7e0 0x7f787806ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:45.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.557+0000 7f787659c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f787815b2c0 0x7f7878161340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:45.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.557+0000 7f7876d9d700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f787806a7e0 0x7f787806ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:50854/0 (socket says 172.21.15.16:50854) 2023-10-16T13:44:45.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.557+0000 7f787659c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f787815b2c0 0x7f7878161340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:40048/0 (socket says 172.21.15.16:40048) 2023-10-16T13:44:45.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.557+0000 7f7876d9d700 1 -- 172.21.15.16:0/2832691466 learned_addr learned my addr 172.21.15.16:0/2832691466 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:45.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.557+0000 7f7875d9b700 1 --1- 172.21.15.16:0/2832691466 >> v1:172.21.15.49:6789/0 conn(0x7f787806e570 0x7f787815e5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:48246/0 (socket says 172.21.15.16:48246) 2023-10-16T13:44:45.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.557+0000 7f7876d9d700 1 -- 172.21.15.16:0/2832691466 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f787815b2c0 msgr2=0x7f7878161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:45.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.557+0000 7f7876d9d700 1 --2- 172.21.15.16:0/2832691466 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f787815b2c0 0x7f7878161340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:45.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.557+0000 7f7876d9d700 1 -- 172.21.15.16:0/2832691466 >> v1:172.21.15.49:6789/0 conn(0x7f787806e570 legacy=0x7f787815e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:45.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.557+0000 7f7876d9d700 1 -- 172.21.15.16:0/2832691466 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f787815b6a0 con 0x7f787806a7e0 2023-10-16T13:44:45.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.557+0000 7f7876d9d700 1 --2- 172.21.15.16:0/2832691466 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f787806a7e0 0x7f787806ac70 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f7870009e50 tx=0x7f787000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed2e99a3681a55f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:45.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.557+0000 7f787559a700 1 -- 172.21.15.16:0/2832691466 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f787000e070 con 0x7f787806a7e0 2023-10-16T13:44:45.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.557+0000 7f787559a700 1 -- 172.21.15.16:0/2832691466 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f787000b910 con 0x7f787806a7e0 2023-10-16T13:44:45.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.557+0000 7f787559a700 1 -- 172.21.15.16:0/2832691466 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f787000bcc0 con 0x7f787806a7e0 2023-10-16T13:44:45.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.558+0000 7f787c9c6700 1 -- 172.21.15.16:0/2832691466 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f787806a7e0 msgr2=0x7f787806ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:45.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.558+0000 7f787c9c6700 1 --2- 172.21.15.16:0/2832691466 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f787806a7e0 0x7f787806ac70 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f7870009e50 tx=0x7f787000b330 comp rx=0 tx=0).stop 2023-10-16T13:44:45.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.558+0000 7f787c9c6700 1 -- 172.21.15.16:0/2832691466 shutdown_connections 2023-10-16T13:44:45.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.558+0000 7f787c9c6700 1 --2- 172.21.15.16:0/2832691466 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f787806a7e0 0x7f787806ac70 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:45.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.558+0000 7f787c9c6700 1 --2- 172.21.15.16:0/2832691466 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f787815b2c0 0x7f7878161340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:45.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.558+0000 7f787c9c6700 1 -- 172.21.15.16:0/2832691466 >> 172.21.15.16:0/2832691466 conn(0x7f78780b1f40 msgr2=0x7f78780b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:45.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.558+0000 7f787c9c6700 1 -- 172.21.15.16:0/2832691466 shutdown_connections 2023-10-16T13:44:45.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.558+0000 7f787c9c6700 1 -- 172.21.15.16:0/2832691466 wait complete. 2023-10-16T13:44:45.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.559+0000 7f787c9c6700 1 Processor -- start 2023-10-16T13:44:45.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.559+0000 7f787c9c6700 1 -- start start 2023-10-16T13:44:45.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.560+0000 7f787c9c6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f787806a7e0 0x7f787815ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:45.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.560+0000 7f787c9c6700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f787806e570 0x7f7878160500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:45.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.560+0000 7f787c9c6700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f787815b2c0 0x7f7878159220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:45.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.560+0000 7f787c9c6700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78780c8680 con 0x7f787806a7e0 2023-10-16T13:44:45.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.560+0000 7f787c9c6700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78780b34d0 con 0x7f787806e570 2023-10-16T13:44:45.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.560+0000 7f787c9c6700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78780b2f10 con 0x7f787815b2c0 2023-10-16T13:44:45.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.560+0000 7f787659c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f787806a7e0 0x7f787815ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:45.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.560+0000 7f787659c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f787806a7e0 0x7f787815ffc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51794/0 (socket says 172.21.15.16:51794) 2023-10-16T13:44:45.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.560+0000 7f787659c700 1 -- 172.21.15.16:0/3246933385 learned_addr learned my addr 172.21.15.16:0/3246933385 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:45.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.560+0000 7f7875d9b700 1 --2- 172.21.15.16:0/3246933385 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f787806e570 0x7f7878160500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:45.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.560+0000 7f7876d9d700 1 --2- 172.21.15.16:0/3246933385 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f787815b2c0 0x7f7878159220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:45.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.561+0000 7f787659c700 1 -- 172.21.15.16:0/3246933385 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f787815b2c0 msgr2=0x7f7878159220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:45.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.561+0000 7f787659c700 1 --2- 172.21.15.16:0/3246933385 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f787815b2c0 0x7f7878159220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:45.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.561+0000 7f787659c700 1 -- 172.21.15.16:0/3246933385 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f787806e570 msgr2=0x7f7878160500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:45.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.561+0000 7f787659c700 1 --2- 172.21.15.16:0/3246933385 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f787806e570 0x7f7878160500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:45.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.561+0000 7f787659c700 1 -- 172.21.15.16:0/3246933385 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7870009b30 con 0x7f787806a7e0 2023-10-16T13:44:45.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.561+0000 7f787659c700 1 --2- 172.21.15.16:0/3246933385 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f787806a7e0 0x7f787815ffc0 secure :-1 s=READY pgs=2299 cs=0 l=1 rev1=1 crypto rx=0x7f786000e9b0 tx=0x7f786000ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:45.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.561+0000 7f786effd700 1 -- 172.21.15.16:0/3246933385 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7860009710 con 0x7f787806a7e0 2023-10-16T13:44:45.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.561+0000 7f786effd700 1 -- 172.21.15.16:0/3246933385 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f786000ee50 con 0x7f787806a7e0 2023-10-16T13:44:45.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.561+0000 7f787c9c6700 1 -- 172.21.15.16:0/3246933385 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78780b31f0 con 0x7f787806a7e0 2023-10-16T13:44:45.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.561+0000 7f786effd700 1 -- 172.21.15.16:0/3246933385 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f786000ca50 con 0x7f787806a7e0 2023-10-16T13:44:45.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.561+0000 7f787c9c6700 1 -- 172.21.15.16:0/3246933385 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78780c8310 con 0x7f787806a7e0 2023-10-16T13:44:45.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.562+0000 7f787c9c6700 1 -- 172.21.15.16:0/3246933385 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f787804ebb0 con 0x7f787806a7e0 2023-10-16T13:44:45.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.563+0000 7f786effd700 1 -- 172.21.15.16:0/3246933385 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 126) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f786000cbb0 con 0x7f787806a7e0 2023-10-16T13:44:45.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.563+0000 7f786effd700 1 --2- 172.21.15.16:0/3246933385 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f785806b020 0x7f785806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:45.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.563+0000 7f786effd700 1 -- 172.21.15.16:0/3246933385 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(206..206 src has 1..206) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f78600898f0 con 0x7f787806a7e0 2023-10-16T13:44:45.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.563+0000 7f7875d9b700 1 --2- 172.21.15.16:0/3246933385 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f785806b020 0x7f785806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:45.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.563+0000 7f7875d9b700 1 --2- 172.21.15.16:0/3246933385 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f785806b020 0x7f785806d4e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f78780b0fd0 tx=0x7f7868007ed0 comp rx=0 tx=0).ready entity=mgr.10665 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:45.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.568+0000 7f786effd700 1 -- 172.21.15.16:0/3246933385 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f78600546d0 con 0x7f787806a7e0 2023-10-16T13:44:45.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.857+0000 7f787c9c6700 1 -- 172.21.15.16:0/3246933385 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f78780b6370 con 0x7f787806a7e0 2023-10-16T13:44:45.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.858+0000 7f786effd700 1 -- 172.21.15.16:0/3246933385 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v54) v1 ==== 190+0+147 (secure 0 0 0) 0x7f78600210e0 con 0x7f787806a7e0 2023-10-16T13:44:45.862 INFO:teuthology.orchestra.run.smithi016.stdout:[client.0] 2023-10-16T13:44:45.862 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAPKC1l2l6PFxAA9QyXtBAHdGAXpwH0JfknuQ== 2023-10-16T13:44:45.862 INFO:teuthology.orchestra.run.smithi016.stdout: caps mds = "allow" 2023-10-16T13:44:45.863 INFO:teuthology.orchestra.run.smithi016.stdout: caps mon = "allow r" 2023-10-16T13:44:45.863 INFO:teuthology.orchestra.run.smithi016.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-16T13:44:45.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.860+0000 7f787c9c6700 1 -- 172.21.15.16:0/3246933385 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f785806b020 msgr2=0x7f785806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:45.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.860+0000 7f787c9c6700 1 --2- 172.21.15.16:0/3246933385 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f785806b020 0x7f785806d4e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f78780b0fd0 tx=0x7f7868007ed0 comp rx=0 tx=0).stop 2023-10-16T13:44:45.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.860+0000 7f787c9c6700 1 -- 172.21.15.16:0/3246933385 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f787806a7e0 msgr2=0x7f787815ffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:45.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.860+0000 7f787c9c6700 1 --2- 172.21.15.16:0/3246933385 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f787806a7e0 0x7f787815ffc0 secure :-1 s=READY pgs=2299 cs=0 l=1 rev1=1 crypto rx=0x7f786000e9b0 tx=0x7f786000ecc0 comp rx=0 tx=0).stop 2023-10-16T13:44:45.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.861+0000 7f787c9c6700 1 -- 172.21.15.16:0/3246933385 shutdown_connections 2023-10-16T13:44:45.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.861+0000 7f787c9c6700 1 --2- 172.21.15.16:0/3246933385 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f785806b020 0x7f785806d4e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:45.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.861+0000 7f787c9c6700 1 --2- 172.21.15.16:0/3246933385 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f787815b2c0 0x7f7878159220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:45.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.861+0000 7f787c9c6700 1 --2- 172.21.15.16:0/3246933385 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f787806a7e0 0x7f787815ffc0 unknown :-1 s=CLOSED pgs=2299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:45.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.861+0000 7f787c9c6700 1 --2- 172.21.15.16:0/3246933385 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f787806e570 0x7f7878160500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:45.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.861+0000 7f787c9c6700 1 -- 172.21.15.16:0/3246933385 >> 172.21.15.16:0/3246933385 conn(0x7f78780b1f40 msgr2=0x7f78780cbdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:45.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.861+0000 7f787c9c6700 1 -- 172.21.15.16:0/3246933385 shutdown_connections 2023-10-16T13:44:45.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.861+0000 7f787c9c6700 1 -- 172.21.15.16:0/3246933385 wait complete. 2023-10-16T13:44:45.867 INFO:teuthology.orchestra.run.smithi016.stderr:updated caps for client.0 2023-10-16T13:44:45.876 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:44:46.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.998+0000 7f2de5b92700 1 Processor -- start 2023-10-16T13:44:46.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.998+0000 7f2de5b92700 1 -- start start 2023-10-16T13:44:46.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.999+0000 7f2de5b92700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2de015b280 0x7f2de01613e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:46.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.999+0000 7f2de5b92700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f2de00cc8a0 0x7f2de015e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:46.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.999+0000 7f2de5b92700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2de0161c30 con 0x7f2de006a7e0 2023-10-16T13:44:46.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.999+0000 7f2de5b92700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2de0161f10 con 0x7f2de015b280 2023-10-16T13:44:46.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.999+0000 7f2de5b92700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2de01621f0 con 0x7f2de00cc8a0 2023-10-16T13:44:46.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.999+0000 7f2ddf7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2de015b280 0x7f2de01613e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:46.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.999+0000 7f2ddffff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2de006a7e0 0x7f2de00d02b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41082/0 (socket says 172.21.15.16:41082) 2023-10-16T13:44:46.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.999+0000 7f2ddffff700 1 -- 172.21.15.16:0/2720312381 learned_addr learned my addr 172.21.15.16:0/2720312381 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:44:46.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.999+0000 7f2ddeffd700 1 --2- 172.21.15.16:0/2720312381 >> v2:172.21.15.49:3300/0 conn(0x7f2de00cc8a0 0x7f2de015e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:46.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.999+0000 7f2ddf7fe700 1 -- 172.21.15.16:0/2720312381 >> v2:172.21.15.49:3300/0 conn(0x7f2de00cc8a0 msgr2=0x7f2de015e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:46.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.999+0000 7f2ddf7fe700 1 --2- 172.21.15.16:0/2720312381 >> v2:172.21.15.49:3300/0 conn(0x7f2de00cc8a0 0x7f2de015e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:46.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.999+0000 7f2ddf7fe700 1 -- 172.21.15.16:0/2720312381 >> v1:172.21.15.16:6789/0 conn(0x7f2de006a7e0 legacy=0x7f2de00d02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:46.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.999+0000 7f2ddf7fe700 1 -- 172.21.15.16:0/2720312381 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2de015df00 con 0x7f2de015b280 2023-10-16T13:44:46.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.999+0000 7f2dde7fc700 1 -- 172.21.15.16:0/2720312381 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1737314563 0 0) 0x7f2de0161c30 con 0x7f2de006a7e0 2023-10-16T13:44:46.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.999+0000 7f2ddf7fe700 1 --2- 172.21.15.16:0/2720312381 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2de015b280 0x7f2de01613e0 secure :-1 s=READY pgs=2301 cs=0 l=1 rev1=1 crypto rx=0x7f2de0071970 tx=0x7f2dd4009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edc19f144e633f71 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:46.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.999+0000 7f2dde7fc700 1 -- 172.21.15.16:0/2720312381 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd4011830 con 0x7f2de015b280 2023-10-16T13:44:46.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.999+0000 7f2dde7fc700 1 -- 172.21.15.16:0/2720312381 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dd4011990 con 0x7f2de015b280 2023-10-16T13:44:46.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:45.999+0000 7f2dde7fc700 1 -- 172.21.15.16:0/2720312381 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd4011d20 con 0x7f2de015b280 2023-10-16T13:44:46.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.000+0000 7f2de5b92700 1 -- 172.21.15.16:0/2720312381 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2de015b280 msgr2=0x7f2de01613e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:46.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.000+0000 7f2de5b92700 1 --2- 172.21.15.16:0/2720312381 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2de015b280 0x7f2de01613e0 secure :-1 s=READY pgs=2301 cs=0 l=1 rev1=1 crypto rx=0x7f2de0071970 tx=0x7f2dd4009b20 comp rx=0 tx=0).stop 2023-10-16T13:44:46.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.000+0000 7f2de5b92700 1 -- 172.21.15.16:0/2720312381 shutdown_connections 2023-10-16T13:44:46.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.000+0000 7f2de5b92700 1 --2- 172.21.15.16:0/2720312381 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2de015b280 0x7f2de01613e0 unknown :-1 s=CLOSED pgs=2301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:46.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.000+0000 7f2de5b92700 1 --2- 172.21.15.16:0/2720312381 >> v2:172.21.15.49:3300/0 conn(0x7f2de00cc8a0 0x7f2de015e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:46.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.000+0000 7f2de5b92700 1 -- 172.21.15.16:0/2720312381 >> 172.21.15.16:0/2720312381 conn(0x7f2de00b1fd0 msgr2=0x7f2de00b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:46.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.000+0000 7f2de5b92700 1 -- 172.21.15.16:0/2720312381 shutdown_connections 2023-10-16T13:44:46.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.000+0000 7f2de5b92700 1 -- 172.21.15.16:0/2720312381 wait complete. 2023-10-16T13:44:46.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.001+0000 7f2de5b92700 1 Processor -- start 2023-10-16T13:44:46.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.001+0000 7f2de5b92700 1 -- start start 2023-10-16T13:44:46.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.002+0000 7f2de5b92700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2de006a7e0 0x7f2de015f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:46.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.002+0000 7f2de5b92700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2de00cc8a0 0x7f2de01f2890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:46.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.002+0000 7f2de5b92700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2de015b280 0x7f2de0159520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:46.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.002+0000 7f2de5b92700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2de015f580 con 0x7f2de00cc8a0 2023-10-16T13:44:46.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.002+0000 7f2de5b92700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2de00c97a0 con 0x7f2de006a7e0 2023-10-16T13:44:46.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.002+0000 7f2de5b92700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2de00c8b90 con 0x7f2de015b280 2023-10-16T13:44:46.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.002+0000 7f2ddeffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2de00cc8a0 0x7f2de01f2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:46.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.002+0000 7f2ddeffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2de00cc8a0 0x7f2de01f2890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51812/0 (socket says 172.21.15.16:51812) 2023-10-16T13:44:46.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.002+0000 7f2ddeffd700 1 -- 172.21.15.16:0/3447639705 learned_addr learned my addr 172.21.15.16:0/3447639705 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:46.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.002+0000 7f2ddf7fe700 1 --2- 172.21.15.16:0/3447639705 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2de006a7e0 0x7f2de015f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:46.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.003+0000 7f2ddffff700 1 --2- 172.21.15.16:0/3447639705 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2de015b280 0x7f2de0159520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:46.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.003+0000 7f2ddeffd700 1 -- 172.21.15.16:0/3447639705 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2de015b280 msgr2=0x7f2de0159520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:46.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.003+0000 7f2ddeffd700 1 --2- 172.21.15.16:0/3447639705 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2de015b280 0x7f2de0159520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:46.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.003+0000 7f2ddeffd700 1 -- 172.21.15.16:0/3447639705 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2de006a7e0 msgr2=0x7f2de015f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:46.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.003+0000 7f2ddeffd700 1 --2- 172.21.15.16:0/3447639705 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2de006a7e0 0x7f2de015f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:46.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.003+0000 7f2ddeffd700 1 -- 172.21.15.16:0/3447639705 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dd40094a0 con 0x7f2de00cc8a0 2023-10-16T13:44:46.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.003+0000 7f2ddffff700 1 --2- 172.21.15.16:0/3447639705 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2de015b280 0x7f2de0159520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:44:46.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.003+0000 7f2ddeffd700 1 --2- 172.21.15.16:0/3447639705 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2de00cc8a0 0x7f2de01f2890 secure :-1 s=READY pgs=2302 cs=0 l=1 rev1=1 crypto rx=0x7f2dd000e7e0 tx=0x7f2dd000eba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:46.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.003+0000 7f2dc7fff700 1 -- 172.21.15.16:0/3447639705 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd0013610 con 0x7f2de00cc8a0 2023-10-16T13:44:46.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.003+0000 7f2dc7fff700 1 -- 172.21.15.16:0/3447639705 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dd0013770 con 0x7f2de00cc8a0 2023-10-16T13:44:46.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.003+0000 7f2dc7fff700 1 -- 172.21.15.16:0/3447639705 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd0013ae0 con 0x7f2de00cc8a0 2023-10-16T13:44:46.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.003+0000 7f2de5b92700 1 -- 172.21.15.16:0/3447639705 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2de00c8e70 con 0x7f2de00cc8a0 2023-10-16T13:44:46.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.003+0000 7f2de5b92700 1 -- 172.21.15.16:0/3447639705 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2de00c9430 con 0x7f2de00cc8a0 2023-10-16T13:44:46.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.004+0000 7f2de5b92700 1 -- 172.21.15.16:0/3447639705 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2de00c9950 con 0x7f2de00cc8a0 2023-10-16T13:44:46.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.005+0000 7f2dc7fff700 1 -- 172.21.15.16:0/3447639705 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 126) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2dd0013c40 con 0x7f2de00cc8a0 2023-10-16T13:44:46.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.005+0000 7f2dc7fff700 1 --2- 172.21.15.16:0/3447639705 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f2dcc06b020 0x7f2dcc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:46.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.005+0000 7f2dc7fff700 1 -- 172.21.15.16:0/3447639705 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(206..206 src has 1..206) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f2dd0089960 con 0x7f2de00cc8a0 2023-10-16T13:44:46.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.005+0000 7f2ddf7fe700 1 --2- 172.21.15.16:0/3447639705 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f2dcc06b020 0x7f2dcc06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:46.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.006+0000 7f2ddf7fe700 1 --2- 172.21.15.16:0/3447639705 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f2dcc06b020 0x7f2dcc06d4e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2dd40090b0 tx=0x7f2dd4008030 comp rx=0 tx=0).ready entity=mgr.10665 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:46.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.009+0000 7f2dc7fff700 1 -- 172.21.15.16:0/3447639705 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2dd00546a0 con 0x7f2de00cc8a0 2023-10-16T13:44:46.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.256+0000 7f2de5b92700 1 -- 172.21.15.16:0/3447639705 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2de00c9910 con 0x7f2de00cc8a0 2023-10-16T13:44:46.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.257+0000 7f2dc7fff700 1 -- 172.21.15.16:0/3447639705 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+4925 (secure 0 0 0) 0x7f2dd0057cc0 con 0x7f2de00cc8a0 2023-10-16T13:44:46.261 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:44:46.261 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":104,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8729,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/3614602930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":3614602930},{"type":"v1","addr":"172.21.15.49:6835","nonce":3614602930}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":99},{"gid":10045,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/3427940027","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":3427940027},{"type":"v1","addr":"172.21.15.49:6834","nonce":3427940027}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":99},{"gid":10896,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/4279896386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":4279896386},{"type":"v1","addr":"172.21.15.16:6836","nonce":4279896386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:44:40.208202+0000","modified":"2023-10-16T13:44:43.255450+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10899},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10899":{"gid":10899,"name":"a","rank":0,"incarnation":100,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/4233835071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":4233835071},{"type":"v1","addr":"172.21.15.16:6837","nonce":4233835071}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2023-10-16T13:44:46.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.259+0000 7f2de5b92700 1 -- 172.21.15.16:0/3447639705 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f2dcc06b020 msgr2=0x7f2dcc06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:46.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.259+0000 7f2de5b92700 1 --2- 172.21.15.16:0/3447639705 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f2dcc06b020 0x7f2dcc06d4e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2dd40090b0 tx=0x7f2dd4008030 comp rx=0 tx=0).stop 2023-10-16T13:44:46.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.259+0000 7f2de5b92700 1 -- 172.21.15.16:0/3447639705 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2de00cc8a0 msgr2=0x7f2de01f2890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:46.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.259+0000 7f2de5b92700 1 --2- 172.21.15.16:0/3447639705 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2de00cc8a0 0x7f2de01f2890 secure :-1 s=READY pgs=2302 cs=0 l=1 rev1=1 crypto rx=0x7f2dd000e7e0 tx=0x7f2dd000eba0 comp rx=0 tx=0).stop 2023-10-16T13:44:46.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.259+0000 7f2de5b92700 1 -- 172.21.15.16:0/3447639705 shutdown_connections 2023-10-16T13:44:46.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.259+0000 7f2de5b92700 1 --2- 172.21.15.16:0/3447639705 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f2dcc06b020 0x7f2dcc06d4e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:46.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.259+0000 7f2de5b92700 1 --2- 172.21.15.16:0/3447639705 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2de015b280 0x7f2de0159520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:46.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.259+0000 7f2de5b92700 1 --2- 172.21.15.16:0/3447639705 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2de006a7e0 0x7f2de015f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:46.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.259+0000 7f2de5b92700 1 --2- 172.21.15.16:0/3447639705 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2de00cc8a0 0x7f2de01f2890 unknown :-1 s=CLOSED pgs=2302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:46.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.259+0000 7f2de5b92700 1 -- 172.21.15.16:0/3447639705 >> 172.21.15.16:0/3447639705 conn(0x7f2de00b1fd0 msgr2=0x7f2de00caa40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:46.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.259+0000 7f2de5b92700 1 -- 172.21.15.16:0/3447639705 shutdown_connections 2023-10-16T13:44:46.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.259+0000 7f2de5b92700 1 -- 172.21.15.16:0/3447639705 wait complete. 2023-10-16T13:44:46.265 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 104 2023-10-16T13:44:46.677 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 104, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T13:44:40.208202+0000', 'modified': '2023-10-16T13:44:43.255450+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10899}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10899': {'gid': 10899, 'name': 'a', 'rank': 0, 'incarnation': 100, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.16:6837/4233835071', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6835', 'nonce': 4233835071}, {'type': 'v1', 'addr': '172.21.15.16:6837', 'nonce': 4233835071}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [26], 'metadata_pool': 25, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-16T13:44:46.678 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-16T13:44:46.678 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-10-16T13:44:46.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.810+0000 7f5acf167700 1 Processor -- start 2023-10-16T13:44:46.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.810+0000 7f5acf167700 1 -- start start 2023-10-16T13:44:46.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.811+0000 7f5acf167700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5ac80d0ae0 0x7f5ac80cef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:46.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.811+0000 7f5acf167700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ac8072a50 con 0x7f5ac80ce010 2023-10-16T13:44:46.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.811+0000 7f5acf167700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ac80cba30 con 0x7f5ac815a5c0 2023-10-16T13:44:46.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.811+0000 7f5acf167700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac80cf440 con 0x7f5ac80d0ae0 2023-10-16T13:44:46.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.811+0000 7f5acd704700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f5ac80ce010 0x7f5ac80cc1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41094/0 (socket says 172.21.15.16:41094) 2023-10-16T13:44:46.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.811+0000 7f5acd704700 1 -- 172.21.15.16:0/547400237 learned_addr learned my addr 172.21.15.16:0/547400237 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:44:46.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.812+0000 7f5ac7fff700 1 --2- 172.21.15.16:0/547400237 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5ac80d0ae0 0x7f5ac80cef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:46.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.812+0000 7f5ac77fe700 1 -- 172.21.15.16:0/547400237 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 118412955 0 0) 0x7f5ac8072a50 con 0x7f5ac80ce010 2023-10-16T13:44:46.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.812+0000 7f5ac77fe700 1 -- 172.21.15.16:0/547400237 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ab80035b0 con 0x7f5ac80ce010 2023-10-16T13:44:46.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.812+0000 7f5ac77fe700 1 -- 172.21.15.16:0/547400237 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 603489297 0 0) 0x7f5ac80cba30 con 0x7f5ac815a5c0 2023-10-16T13:44:46.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.812+0000 7f5ac77fe700 1 -- 172.21.15.16:0/547400237 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ac8072a50 con 0x7f5ac815a5c0 2023-10-16T13:44:46.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.812+0000 7f5ac77fe700 1 -- 172.21.15.16:0/547400237 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 826111819 0 0) 0x7f5ab80035b0 con 0x7f5ac80ce010 2023-10-16T13:44:46.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.812+0000 7f5ac77fe700 1 -- 172.21.15.16:0/547400237 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5ac80d0ae0 msgr2=0x7f5ac80cef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:46.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.812+0000 7f5ac77fe700 1 --2- 172.21.15.16:0/547400237 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5ac80d0ae0 0x7f5ac80cef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:46.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.812+0000 7f5ac77fe700 1 -- 172.21.15.16:0/547400237 >> v1:172.21.15.49:6789/0 conn(0x7f5ac815a5c0 legacy=0x7f5ac80d03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:46.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.812+0000 7f5ac77fe700 1 -- 172.21.15.16:0/547400237 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ac80cf630 con 0x7f5ac80ce010 2023-10-16T13:44:46.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.812+0000 7f5ac77fe700 1 -- 172.21.15.16:0/547400237 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f5ac00019f0 con 0x7f5ac80ce010 2023-10-16T13:44:46.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.812+0000 7f5ac77fe700 1 -- 172.21.15.16:0/547400237 >> v1:172.21.15.16:6789/0 conn(0x7f5ac80ce010 legacy=0x7f5ac80cc1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:46.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.812+0000 7f5ac77fe700 1 --2- 172.21.15.16:0/547400237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ab8003bc0 0x7f5ab8003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:46.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.812+0000 7f5ac77fe700 1 --2- 172.21.15.16:0/547400237 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5ac80d0ae0 0x7f5ab8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:46.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.812+0000 7f5ac77fe700 1 --2- 172.21.15.16:0/547400237 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5ab8004d90 0x7f5ab8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:46.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.812+0000 7f5ac77fe700 1 -- 172.21.15.16:0/547400237 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab800bac0 con 0x7f5ab8003bc0 2023-10-16T13:44:46.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.812+0000 7f5ac77fe700 1 -- 172.21.15.16:0/547400237 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab800bc00 con 0x7f5ab8004d90 2023-10-16T13:44:46.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.812+0000 7f5ac77fe700 1 -- 172.21.15.16:0/547400237 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab800bdc0 con 0x7f5ac80d0ae0 2023-10-16T13:44:46.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.812+0000 7f5accf03700 1 --2- 172.21.15.16:0/547400237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ab8003bc0 0x7f5ab8003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:46.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.812+0000 7f5ac7fff700 1 --2- 172.21.15.16:0/547400237 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5ac80d0ae0 0x7f5ab8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:46.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.812+0000 7f5acd704700 1 --2- 172.21.15.16:0/547400237 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5ab8004d90 0x7f5ab8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:46.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.812+0000 7f5accf03700 1 -- 172.21.15.16:0/547400237 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5ac80d0ae0 msgr2=0x7f5ab8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:46.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.812+0000 7f5accf03700 1 --2- 172.21.15.16:0/547400237 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5ac80d0ae0 0x7f5ab8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:46.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.812+0000 7f5accf03700 1 -- 172.21.15.16:0/547400237 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5ab8004d90 msgr2=0x7f5ab8005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:46.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.812+0000 7f5accf03700 1 --2- 172.21.15.16:0/547400237 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5ab8004d90 0x7f5ab8005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:46.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.813+0000 7f5accf03700 1 -- 172.21.15.16:0/547400237 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ab80035b0 con 0x7f5ab8003bc0 2023-10-16T13:44:46.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.813+0000 7f5acd704700 1 --2- 172.21.15.16:0/547400237 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5ab8004d90 0x7f5ab8005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:44:46.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.813+0000 7f5accf03700 1 --2- 172.21.15.16:0/547400237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ab8003bc0 0x7f5ab8003fc0 secure :-1 s=READY pgs=2304 cs=0 l=1 rev1=1 crypto rx=0x7f5abc001940 tx=0x7f5ab8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a8c43e5d34c6a7f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:46.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.813+0000 7f5ac77fe700 1 -- 172.21.15.16:0/547400237 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5abc001430 con 0x7f5ab8003bc0 2023-10-16T13:44:46.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.813+0000 7f5ac77fe700 1 -- 172.21.15.16:0/547400237 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5abc001d70 con 0x7f5ab8003bc0 2023-10-16T13:44:46.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.813+0000 7f5ac77fe700 1 -- 172.21.15.16:0/547400237 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5abc005510 con 0x7f5ab8003bc0 2023-10-16T13:44:46.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.813+0000 7f5acf167700 1 -- 172.21.15.16:0/547400237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ab8003bc0 msgr2=0x7f5ab8003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:46.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.813+0000 7f5acf167700 1 --2- 172.21.15.16:0/547400237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ab8003bc0 0x7f5ab8003fc0 secure :-1 s=READY pgs=2304 cs=0 l=1 rev1=1 crypto rx=0x7f5abc001940 tx=0x7f5ab8005880 comp rx=0 tx=0).stop 2023-10-16T13:44:46.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.813+0000 7f5acd704700 1 -- 172.21.15.16:0/547400237 reap_dead start 2023-10-16T13:44:46.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.813+0000 7f5acf167700 1 -- 172.21.15.16:0/547400237 shutdown_connections 2023-10-16T13:44:46.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.813+0000 7f5acf167700 1 -- 172.21.15.16:0/547400237 >> 172.21.15.16:0/547400237 conn(0x7f5ac80b1fd0 msgr2=0x7f5ac80b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:46.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.813+0000 7f5acf167700 1 -- 172.21.15.16:0/547400237 shutdown_connections 2023-10-16T13:44:46.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.813+0000 7f5acf167700 1 -- 172.21.15.16:0/547400237 wait complete. 2023-10-16T13:44:46.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.815+0000 7f5acf167700 1 Processor -- start 2023-10-16T13:44:46.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.815+0000 7f5acf167700 1 -- start start 2023-10-16T13:44:46.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.816+0000 7f5acf167700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5ab8003bc0 0x7f5ac815d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:46.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.816+0000 7f5acf167700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5ac81f2640 0x7f5ac81566f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:46.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.816+0000 7f5acf167700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ac80ca1d0 0x7f5ac81594d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:46.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.816+0000 7f5acf167700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac81f1c30 con 0x7f5ac80ca1d0 2023-10-16T13:44:46.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.816+0000 7f5acf167700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac815d770 con 0x7f5ac81f2640 2023-10-16T13:44:46.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.816+0000 7f5acf167700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac815da50 con 0x7f5ab8003bc0 2023-10-16T13:44:46.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.816+0000 7f5acd704700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ac80ca1d0 0x7f5ac81594d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:46.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.816+0000 7f5acd704700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ac80ca1d0 0x7f5ac81594d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51834/0 (socket says 172.21.15.16:51834) 2023-10-16T13:44:46.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.816+0000 7f5acd704700 1 -- 172.21.15.16:0/340988831 learned_addr learned my addr 172.21.15.16:0/340988831 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:46.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.816+0000 7f5ac7fff700 1 --2- 172.21.15.16:0/340988831 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5ac81f2640 0x7f5ac81566f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:46.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.816+0000 7f5accf03700 1 --2- 172.21.15.16:0/340988831 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5ab8003bc0 0x7f5ac815d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:46.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.816+0000 7f5acd704700 1 -- 172.21.15.16:0/340988831 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5ab8003bc0 msgr2=0x7f5ac815d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:46.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.816+0000 7f5acd704700 1 --2- 172.21.15.16:0/340988831 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5ab8003bc0 0x7f5ac815d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:46.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.816+0000 7f5acd704700 1 -- 172.21.15.16:0/340988831 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5ac81f2640 msgr2=0x7f5ac81566f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:46.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.816+0000 7f5acd704700 1 --2- 172.21.15.16:0/340988831 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5ac81f2640 0x7f5ac81566f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:46.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.816+0000 7f5acd704700 1 -- 172.21.15.16:0/340988831 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ac81f1e40 con 0x7f5ac80ca1d0 2023-10-16T13:44:46.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.816+0000 7f5accf03700 1 --2- 172.21.15.16:0/340988831 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5ab8003bc0 0x7f5ac815d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:44:46.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.816+0000 7f5acd704700 1 --2- 172.21.15.16:0/340988831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ac80ca1d0 0x7f5ac81594d0 secure :-1 s=READY pgs=2305 cs=0 l=1 rev1=1 crypto rx=0x7f5ac0001330 tx=0x7f5ac000e440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:46.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.816+0000 7f5ac57fa700 1 -- 172.21.15.16:0/340988831 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ac000ec50 con 0x7f5ac80ca1d0 2023-10-16T13:44:46.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.816+0000 7f5ac57fa700 1 -- 172.21.15.16:0/340988831 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ac000edb0 con 0x7f5ac80ca1d0 2023-10-16T13:44:46.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.816+0000 7f5acf167700 1 -- 172.21.15.16:0/340988831 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ac81f2120 con 0x7f5ac80ca1d0 2023-10-16T13:44:46.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.816+0000 7f5ac57fa700 1 -- 172.21.15.16:0/340988831 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ac0013660 con 0x7f5ac80ca1d0 2023-10-16T13:44:46.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.816+0000 7f5acf167700 1 -- 172.21.15.16:0/340988831 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ac8156c30 con 0x7f5ac80ca1d0 2023-10-16T13:44:46.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.818+0000 7f5ac57fa700 1 -- 172.21.15.16:0/340988831 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 126) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5ac00137c0 con 0x7f5ac80ca1d0 2023-10-16T13:44:46.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.819+0000 7f5ac57fa700 1 --2- 172.21.15.16:0/340988831 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f5aac06b020 0x7f5aac06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:46.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.819+0000 7f5ac57fa700 1 -- 172.21.15.16:0/340988831 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(206..206 src has 1..206) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f5ac0089fc0 con 0x7f5ac80ca1d0 2023-10-16T13:44:46.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.819+0000 7f5accf03700 1 --2- 172.21.15.16:0/340988831 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f5aac06b020 0x7f5aac06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:46.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.819+0000 7f5accf03700 1 --2- 172.21.15.16:0/340988831 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f5aac06b020 0x7f5aac06d4e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5abc005040 tx=0x7f5abc00a000 comp rx=0 tx=0).ready entity=mgr.10665 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:46.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.820+0000 7f5adb351b80 1 Processor -- start 2023-10-16T13:44:46.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.821+0000 7f5adb351b80 1 -- start start 2023-10-16T13:44:46.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.821+0000 7f5adb351b80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x565082b3b220 0x565082b42a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:46.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.821+0000 7f5adb351b80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x565082b40630 0x565082b42fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:46.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.821+0000 7f5adb351b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x565082b16010 0x565082b166c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:46.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.821+0000 7f5adb351b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x565082a8cc20 con 0x565082b16010 2023-10-16T13:44:46.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.821+0000 7f5adb351b80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x565082a71c10 con 0x565082b3b220 2023-10-16T13:44:46.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.821+0000 7f5adb351b80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x565082a6f530 con 0x565082b40630 2023-10-16T13:44:46.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.821+0000 7f5acd704700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x565082b16010 0x565082b166c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:46.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.821+0000 7f5acd704700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x565082b16010 0x565082b166c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51842/0 (socket says 172.21.15.16:51842) 2023-10-16T13:44:46.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.821+0000 7f5acd704700 1 -- 172.21.15.16:0/411345792 learned_addr learned my addr 172.21.15.16:0/411345792 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:46.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.822+0000 7f5accf03700 1 --2- 172.21.15.16:0/411345792 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x565082b3b220 0x565082b42a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:46.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.822+0000 7f5acd704700 1 -- 172.21.15.16:0/411345792 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x565082b40630 msgr2=0x565082b42fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:46.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.822+0000 7f5acd704700 1 --2- 172.21.15.16:0/411345792 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x565082b40630 0x565082b42fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:46.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.822+0000 7f5acd704700 1 -- 172.21.15.16:0/411345792 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x565082b3b220 msgr2=0x565082b42a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:46.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.822+0000 7f5acd704700 1 --2- 172.21.15.16:0/411345792 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x565082b3b220 0x565082b42a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:46.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.822+0000 7f5acd704700 1 -- 172.21.15.16:0/411345792 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x565082a283a0 con 0x565082b16010 2023-10-16T13:44:46.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.822+0000 7f5acd704700 1 --2- 172.21.15.16:0/411345792 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x565082b16010 0x565082b166c0 secure :-1 s=READY pgs=2306 cs=0 l=1 rev1=1 crypto rx=0x7f5ac000a040 tx=0x7f5ac0058650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b16063201898ca62 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:46.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.822+0000 7f5ab27fc700 1 -- 172.21.15.16:0/411345792 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ac0070070 con 0x565082b16010 2023-10-16T13:44:46.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.822+0000 7f5ab27fc700 1 -- 172.21.15.16:0/411345792 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ac0060ce0 con 0x565082b16010 2023-10-16T13:44:46.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.822+0000 7f5ab27fc700 1 -- 172.21.15.16:0/411345792 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ac00112e0 con 0x565082b16010 2023-10-16T13:44:46.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.823+0000 7f5adb351b80 1 -- 172.21.15.16:0/411345792 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x565082b16010 msgr2=0x565082b166c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:46.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.823+0000 7f5adb351b80 1 --2- 172.21.15.16:0/411345792 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x565082b16010 0x565082b166c0 secure :-1 s=READY pgs=2306 cs=0 l=1 rev1=1 crypto rx=0x7f5ac000a040 tx=0x7f5ac0058650 comp rx=0 tx=0).stop 2023-10-16T13:44:46.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.823+0000 7f5adb351b80 1 -- 172.21.15.16:0/411345792 shutdown_connections 2023-10-16T13:44:46.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.823+0000 7f5adb351b80 1 --2- 172.21.15.16:0/411345792 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x565082b16010 0x565082b166c0 unknown :-1 s=CLOSED pgs=2306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:46.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.823+0000 7f5adb351b80 1 --2- 172.21.15.16:0/411345792 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x565082b3b220 0x565082b42a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:46.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.823+0000 7f5adb351b80 1 --2- 172.21.15.16:0/411345792 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x565082b40630 0x565082b42fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:46.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.823+0000 7f5adb351b80 1 -- 172.21.15.16:0/411345792 >> 172.21.15.16:0/411345792 conn(0x565082a45d80 msgr2=0x565082b60ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:46.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.823+0000 7f5adb351b80 1 -- 172.21.15.16:0/411345792 shutdown_connections 2023-10-16T13:44:46.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.823+0000 7f5adb351b80 1 -- 172.21.15.16:0/411345792 wait complete. 2023-10-16T13:44:46.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.823+0000 7f5adb351b80 1 -- start start 2023-10-16T13:44:46.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.823+0000 7f5adb351b80 1 Processor -- start 2023-10-16T13:44:46.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.824+0000 7f5adb351b80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x565082b16010 0x565082b46760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:46.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.824+0000 7f5adb351b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x565082b3b220 0x565082b3b720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:46.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.824+0000 7f5adb351b80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x565082b40630 0x565082b0cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:46.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.824+0000 7f5adb351b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x565082a6ae20 con 0x565082b3b220 2023-10-16T13:44:46.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.824+0000 7f5adb351b80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x565082a076c0 con 0x565082b16010 2023-10-16T13:44:46.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.824+0000 7f5adb351b80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x565082a07360 con 0x565082b40630 2023-10-16T13:44:46.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.824+0000 7f5adb351b80 10 client.0 _ll_register_callbacks cb 0x5650828647b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T13:44:46.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.824+0000 7f5ac7fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x565082b3b220 0x565082b3b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:46.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.824+0000 7f5ac7fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x565082b3b220 0x565082b3b720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51846/0 (socket says 172.21.15.16:51846) 2023-10-16T13:44:46.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.824+0000 7f5ac7fff700 1 -- 172.21.15.16:0/4074384695 learned_addr learned my addr 172.21.15.16:0/4074384695 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:46.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.825+0000 7f5accf03700 1 --2- 172.21.15.16:0/4074384695 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x565082b16010 0x565082b46760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:46.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.825+0000 7f5acd704700 1 --2- 172.21.15.16:0/4074384695 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x565082b40630 0x565082b0cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:46.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.825+0000 7f5ac7fff700 1 -- 172.21.15.16:0/4074384695 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x565082b40630 msgr2=0x565082b0cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:46.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.825+0000 7f5ac7fff700 1 --2- 172.21.15.16:0/4074384695 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x565082b40630 0x565082b0cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:46.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.825+0000 7f5ac7fff700 1 -- 172.21.15.16:0/4074384695 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x565082b16010 msgr2=0x565082b46760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:46.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.825+0000 7f5ac7fff700 1 --2- 172.21.15.16:0/4074384695 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x565082b16010 0x565082b46760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:46.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.825+0000 7f5ac7fff700 1 -- 172.21.15.16:0/4074384695 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ac8156c30 con 0x565082b3b220 2023-10-16T13:44:46.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.825+0000 7f5accf03700 1 --2- 172.21.15.16:0/4074384695 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x565082b16010 0x565082b46760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:44:46.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.825+0000 7f5ac7fff700 1 --2- 172.21.15.16:0/4074384695 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x565082b3b220 0x565082b3b720 secure :-1 s=READY pgs=2307 cs=0 l=1 rev1=1 crypto rx=0x7f5ab4014600 tx=0x7f5ab40149c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99b54f20b5e92bc0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:46.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.825+0000 7f5aa77fe700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2023-10-16T13:44:46.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.825+0000 7f5aa77fe700 1 -- 172.21.15.16:0/4074384695 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ab4016700 con 0x565082b3b220 2023-10-16T13:44:46.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.825+0000 7f5aa77fe700 1 -- 172.21.15.16:0/4074384695 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(206..206 src has 1..206) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f5ab401a070 con 0x565082b3b220 2023-10-16T13:44:46.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.825+0000 7f5adb351b80 1 -- 172.21.15.16:0/4074384695 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x565082b0b5b0 con 0x565082b3b220 2023-10-16T13:44:46.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.825+0000 7f5aa77fe700 1 -- 172.21.15.16:0/4074384695 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=104) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5ab4016860 con 0x565082b3b220 2023-10-16T13:44:46.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.825+0000 7f5adb351b80 10 client.11043 fetch_fsmap learned FSMap version 104 2023-10-16T13:44:46.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.825+0000 7f5adb351b80 1 -- 172.21.15.16:0/4074384695 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=104,monmap=2+}) v3 -- 0x565082895d90 con 0x565082b3b220 2023-10-16T13:44:46.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.825+0000 7f5aa77fe700 1 -- 172.21.15.16:0/4074384695 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x565082b0b5b0 con 0x565082b3b220 2023-10-16T13:44:46.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.825+0000 7f5aa77fe700 1 -- 172.21.15.16:0/4074384695 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 104) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f5ab4011030 con 0x565082b3b220 2023-10-16T13:44:46.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.825+0000 7f5adb351b80 10 client.11043 fetch_fsmap finished waiting for FSMap version 104 2023-10-16T13:44:46.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.825+0000 7f5adb351b80 10 client.11043 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-16T13:44:46.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.825+0000 7f5adb351b80 20 client.11043 populate_metadata read hostname 'smithi016' 2023-10-16T13:44:46.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.825+0000 7f5adb351b80 1 --2- 172.21.15.16:0/4074384695 >> [v2:172.21.15.16:6835/4233835071,v1:172.21.15.16:6837/4233835071] conn(0x565082b3eb20 0x565082b3ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:46.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.826+0000 7f5adb351b80 4 client.11043 mds_command: new command op to 10899 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T13:44:46.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.826+0000 7f5adb351b80 1 -- 172.21.15.16:0/4074384695 --> [v2:172.21.15.16:6835/4233835071,v1:172.21.15.16:6837/4233835071] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x565082a63e00 con 0x565082b3eb20 2023-10-16T13:44:46.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.826+0000 7f5accf03700 1 --2- 172.21.15.16:0/4074384695 >> [v2:172.21.15.16:6835/4233835071,v1:172.21.15.16:6837/4233835071] conn(0x565082b3eb20 0x565082b3ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:46.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.826+0000 7f5aa77fe700 10 client.11043 ms_handle_connect on v2:172.21.15.16:6835/4233835071 2023-10-16T13:44:46.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.826+0000 7f5accf03700 1 --2- 172.21.15.16:0/4074384695 >> [v2:172.21.15.16:6835/4233835071,v1:172.21.15.16:6837/4233835071] conn(0x565082b3eb20 0x565082b3ef00 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=82c51f0cabe0b0cb server_cookie=8f2a9ebe318f7b51 in_seq=0 out_seq=0 2023-10-16T13:44:46.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.827+0000 7f5aa77fe700 1 -- 172.21.15.16:0/4074384695 <== mds.0 v2:172.21.15.16:6835/4233835071 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x565082a63e00 con 0x565082b3eb20 2023-10-16T13:44:46.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.827+0000 7f5aa77fe700 10 client.11043 handle_command_reply: tid=0 2023-10-16T13:44:46.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.827+0000 7f5adb351b80 1 client.11043 shutdown 2023-10-16T13:44:46.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.827+0000 7f5adb351b80 1 -- 172.21.15.16:0/4074384695 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x565082b3b220 msgr2=0x565082b3b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:46.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.827+0000 7f5adb351b80 1 --2- 172.21.15.16:0/4074384695 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x565082b3b220 0x565082b3b720 secure :-1 s=READY pgs=2307 cs=0 l=1 rev1=1 crypto rx=0x7f5ab4014600 tx=0x7f5ab40149c0 comp rx=0 tx=0).stop 2023-10-16T13:44:46.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.827+0000 7f5adb351b80 1 -- 172.21.15.16:0/4074384695 shutdown_connections 2023-10-16T13:44:46.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.827+0000 7f5adb351b80 1 --2- 172.21.15.16:0/4074384695 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x565082b40630 0x565082b0cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:46.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.827+0000 7f5adb351b80 1 --2- 172.21.15.16:0/4074384695 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x565082b16010 0x565082b46760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:46.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.827+0000 7f5adb351b80 1 --2- 172.21.15.16:0/4074384695 >> [v2:172.21.15.16:6835/4233835071,v1:172.21.15.16:6837/4233835071] conn(0x565082b3eb20 0x565082b3ef00 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:46.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.827+0000 7f5adb351b80 1 --2- 172.21.15.16:0/4074384695 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x565082b3b220 0x565082b3b720 unknown :-1 s=CLOSED pgs=2307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:46.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.827+0000 7f5aa77fe700 0 client.11043 ms_handle_reset on v2:172.21.15.16:6835/4233835071 2023-10-16T13:44:46.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.827+0000 7f5adb351b80 1 -- 172.21.15.16:0/4074384695 >> 172.21.15.16:0/4074384695 conn(0x565082a45d80 msgr2=0x565082b179f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:46.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.827+0000 7f5adb351b80 1 -- 172.21.15.16:0/4074384695 shutdown_connections 2023-10-16T13:44:46.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.827+0000 7f5adb351b80 1 -- 172.21.15.16:0/4074384695 wait complete. 2023-10-16T13:44:46.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.827+0000 7f5adb351b80 20 client.11043 trim_cache size 0 max 16384 2023-10-16T13:44:46.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.845+0000 7f5adb351b80 1 Processor -- start 2023-10-16T13:44:46.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.845+0000 7f5adb351b80 1 -- start start 2023-10-16T13:44:46.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.846+0000 7f5adb351b80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x565082b16010 0x565082b57610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:46.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.846+0000 7f5adb351b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x565082b3b220 0x565082b34690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:46.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.846+0000 7f5adb351b80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x565082b40630 0x565082b34bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:46.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.846+0000 7f5adb351b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650828eb8d0 con 0x565082b3b220 2023-10-16T13:44:46.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.846+0000 7f5adb351b80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x565082aee3c0 con 0x565082b40630 2023-10-16T13:44:46.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.846+0000 7f5adb351b80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x565082b189f0 con 0x565082b16010 2023-10-16T13:44:46.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.846+0000 7f5ac7fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x565082b3b220 0x565082b34690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:46.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.846+0000 7f5ac7fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x565082b3b220 0x565082b34690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51854/0 (socket says 172.21.15.16:51854) 2023-10-16T13:44:46.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.846+0000 7f5ac7fff700 1 -- 172.21.15.16:0/4177223395 learned_addr learned my addr 172.21.15.16:0/4177223395 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:46.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.846+0000 7f5acd704700 1 --2- 172.21.15.16:0/4177223395 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x565082b40630 0x565082b34bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:46.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.847+0000 7f5accf03700 1 --2- 172.21.15.16:0/4177223395 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x565082b16010 0x565082b57610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:46.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.847+0000 7f5ac7fff700 1 -- 172.21.15.16:0/4177223395 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x565082b16010 msgr2=0x565082b57610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:46.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.847+0000 7f5ac7fff700 1 --2- 172.21.15.16:0/4177223395 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x565082b16010 0x565082b57610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:46.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.847+0000 7f5ac7fff700 1 -- 172.21.15.16:0/4177223395 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x565082b40630 msgr2=0x565082b34bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:46.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.847+0000 7f5ac7fff700 1 --2- 172.21.15.16:0/4177223395 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x565082b40630 0x565082b34bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:46.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.847+0000 7f5ac7fff700 1 -- 172.21.15.16:0/4177223395 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56508287ec10 con 0x565082b3b220 2023-10-16T13:44:46.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.847+0000 7f5ac7fff700 1 --2- 172.21.15.16:0/4177223395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x565082b3b220 0x565082b34690 secure :-1 s=READY pgs=2308 cs=0 l=1 rev1=1 crypto rx=0x7f5ab4001eb0 tx=0x7f5ab4019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=758dce098b6e2c12 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:46.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.847+0000 7f5ab2ffd700 1 -- 172.21.15.16:0/4177223395 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ab4016900 con 0x565082b3b220 2023-10-16T13:44:46.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.847+0000 7f5ab2ffd700 1 -- 172.21.15.16:0/4177223395 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ab4016c50 con 0x565082b3b220 2023-10-16T13:44:46.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.847+0000 7f5ab2ffd700 1 -- 172.21.15.16:0/4177223395 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ab401f480 con 0x565082b3b220 2023-10-16T13:44:46.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.848+0000 7f5adb351b80 1 -- 172.21.15.16:0/4177223395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x565082b3b220 msgr2=0x565082b34690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:46.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.848+0000 7f5adb351b80 1 --2- 172.21.15.16:0/4177223395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x565082b3b220 0x565082b34690 secure :-1 s=READY pgs=2308 cs=0 l=1 rev1=1 crypto rx=0x7f5ab4001eb0 tx=0x7f5ab4019000 comp rx=0 tx=0).stop 2023-10-16T13:44:46.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.848+0000 7f5adb351b80 1 -- 172.21.15.16:0/4177223395 shutdown_connections 2023-10-16T13:44:46.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.848+0000 7f5adb351b80 1 --2- 172.21.15.16:0/4177223395 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x565082b40630 0x565082b34bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:46.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.848+0000 7f5adb351b80 1 --2- 172.21.15.16:0/4177223395 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x565082b16010 0x565082b57610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:46.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.848+0000 7f5adb351b80 1 --2- 172.21.15.16:0/4177223395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x565082b3b220 0x565082b34690 unknown :-1 s=CLOSED pgs=2308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:46.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.848+0000 7f5adb351b80 1 -- 172.21.15.16:0/4177223395 >> 172.21.15.16:0/4177223395 conn(0x565082a45d80 msgr2=0x565082b808c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:46.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.848+0000 7f5adb351b80 1 -- 172.21.15.16:0/4177223395 shutdown_connections 2023-10-16T13:44:46.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.848+0000 7f5adb351b80 1 -- 172.21.15.16:0/4177223395 wait complete. 2023-10-16T13:44:46.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.848+0000 7f5adb351b80 1 -- start start 2023-10-16T13:44:46.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.849+0000 7f5adb351b80 1 Processor -- start 2023-10-16T13:44:46.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.849+0000 7f5adb351b80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x565082b16010 0x565082b163f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:46.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.849+0000 7f5adb351b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x565082b3b220 0x565082b16930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:46.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.849+0000 7f5adb351b80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x565082b40630 0x565082b34f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:46.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.849+0000 7f5adb351b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x565082b5e350 con 0x565082b3b220 2023-10-16T13:44:46.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.849+0000 7f5adb351b80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x565082b34b30 con 0x565082b16010 2023-10-16T13:44:46.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.849+0000 7f5adb351b80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x565082b6f780 con 0x565082b40630 2023-10-16T13:44:46.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.849+0000 7f5adb351b80 10 client.0 _ll_register_callbacks cb 0x7f5ab4009dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T13:44:46.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.849+0000 7f5ac7fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x565082b3b220 0x565082b16930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:46.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.849+0000 7f5ac7fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x565082b3b220 0x565082b16930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51860/0 (socket says 172.21.15.16:51860) 2023-10-16T13:44:46.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.849+0000 7f5ac7fff700 1 -- 172.21.15.16:0/4183184343 learned_addr learned my addr 172.21.15.16:0/4183184343 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:46.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.849+0000 7f5accf03700 1 --2- 172.21.15.16:0/4183184343 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x565082b16010 0x565082b163f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:46.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.850+0000 7f5acd704700 1 --2- 172.21.15.16:0/4183184343 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x565082b40630 0x565082b34f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:46.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.850+0000 7f5ac7fff700 1 -- 172.21.15.16:0/4183184343 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x565082b40630 msgr2=0x565082b34f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:46.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.850+0000 7f5ac7fff700 1 --2- 172.21.15.16:0/4183184343 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x565082b40630 0x565082b34f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:46.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.850+0000 7f5ac7fff700 1 -- 172.21.15.16:0/4183184343 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x565082b16010 msgr2=0x565082b163f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:46.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.850+0000 7f5ac7fff700 1 --2- 172.21.15.16:0/4183184343 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x565082b16010 0x565082b163f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:46.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.850+0000 7f5ac7fff700 1 -- 172.21.15.16:0/4183184343 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x565082895d90 con 0x565082b3b220 2023-10-16T13:44:46.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.850+0000 7f5ac7fff700 1 --2- 172.21.15.16:0/4183184343 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x565082b3b220 0x565082b16930 secure :-1 s=READY pgs=2309 cs=0 l=1 rev1=1 crypto rx=0x7f5ab4009340 tx=0x7f5ab4009210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d64481fefe567cfc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:46.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.850+0000 7f5aa77fe700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2023-10-16T13:44:46.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.850+0000 7f5aa77fe700 1 -- 172.21.15.16:0/4183184343 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ab401f890 con 0x565082b3b220 2023-10-16T13:44:46.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.850+0000 7f5adb351b80 1 -- 172.21.15.16:0/4183184343 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5ac0060ce0 con 0x565082b3b220 2023-10-16T13:44:46.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.850+0000 7f5aa77fe700 1 -- 172.21.15.16:0/4183184343 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(206..206 src has 1..206) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f5ab4000e40 con 0x565082b3b220 2023-10-16T13:44:46.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.850+0000 7f5aa77fe700 1 -- 172.21.15.16:0/4183184343 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=104) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5ab401f9f0 con 0x565082b3b220 2023-10-16T13:44:46.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.850+0000 7f5adb351b80 10 client.11049 fetch_fsmap learned FSMap version 104 2023-10-16T13:44:46.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.850+0000 7f5adb351b80 1 -- 172.21.15.16:0/4183184343 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=104,monmap=2+}) v3 -- 0x5650828bc200 con 0x565082b3b220 2023-10-16T13:44:46.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.850+0000 7f5aa77fe700 1 -- 172.21.15.16:0/4183184343 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ac0060ce0 con 0x565082b3b220 2023-10-16T13:44:46.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.850+0000 7f5aa77fe700 1 -- 172.21.15.16:0/4183184343 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 104) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f5ab4011050 con 0x565082b3b220 2023-10-16T13:44:46.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.851+0000 7f5adb351b80 10 client.11049 fetch_fsmap finished waiting for FSMap version 104 2023-10-16T13:44:46.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.851+0000 7f5adb351b80 10 client.11049 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-16T13:44:46.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.851+0000 7f5adb351b80 20 client.11049 populate_metadata read hostname 'smithi016' 2023-10-16T13:44:46.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.851+0000 7f5adb351b80 1 --2- 172.21.15.16:0/4183184343 >> [v2:172.21.15.16:6835/4233835071,v1:172.21.15.16:6837/4233835071] conn(0x565082b12d20 0x565082b354c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:46.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.851+0000 7f5adb351b80 4 client.11049 mds_command: new command op to 10899 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T13:44:46.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.851+0000 7f5adb351b80 1 -- 172.21.15.16:0/4183184343 --> [v2:172.21.15.16:6835/4233835071,v1:172.21.15.16:6837/4233835071] -- command(tid 0: {"prefix": "status"}) v1 -- 0x565082a63b20 con 0x565082b12d20 2023-10-16T13:44:46.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.851+0000 7f5accf03700 1 --2- 172.21.15.16:0/4183184343 >> [v2:172.21.15.16:6835/4233835071,v1:172.21.15.16:6837/4233835071] conn(0x565082b12d20 0x565082b354c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:46.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.851+0000 7f5aa77fe700 10 client.11049 ms_handle_connect on v2:172.21.15.16:6835/4233835071 2023-10-16T13:44:46.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.851+0000 7f5accf03700 1 --2- 172.21.15.16:0/4183184343 >> [v2:172.21.15.16:6835/4233835071,v1:172.21.15.16:6837/4233835071] conn(0x565082b12d20 0x565082b354c0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d76040c682ff0fc server_cookie=764a7b56cea6edbb in_seq=0 out_seq=0 2023-10-16T13:44:46.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.851+0000 7f5aa77fe700 1 -- 172.21.15.16:0/4183184343 <== mds.0 v2:172.21.15.16:6835/4233835071 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+335 (crc 0 0 0) 0x565082a63b20 con 0x565082b12d20 2023-10-16T13:44:46.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.851+0000 7f5aa77fe700 10 client.11049 handle_command_reply: tid=0 2023-10-16T13:44:46.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.851+0000 7f5adb351b80 1 client.11049 shutdown 2023-10-16T13:44:46.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.851+0000 7f5adb351b80 1 -- 172.21.15.16:0/4183184343 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x565082b3b220 msgr2=0x565082b16930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:46.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.851+0000 7f5adb351b80 1 --2- 172.21.15.16:0/4183184343 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x565082b3b220 0x565082b16930 secure :-1 s=READY pgs=2309 cs=0 l=1 rev1=1 crypto rx=0x7f5ab4009340 tx=0x7f5ab4009210 comp rx=0 tx=0).stop 2023-10-16T13:44:46.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.851+0000 7f5adb351b80 1 -- 172.21.15.16:0/4183184343 shutdown_connections 2023-10-16T13:44:46.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.851+0000 7f5adb351b80 1 --2- 172.21.15.16:0/4183184343 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x565082b40630 0x565082b34f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:46.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.851+0000 7f5adb351b80 1 --2- 172.21.15.16:0/4183184343 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x565082b16010 0x565082b163f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:46.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.851+0000 7f5adb351b80 1 --2- 172.21.15.16:0/4183184343 >> [v2:172.21.15.16:6835/4233835071,v1:172.21.15.16:6837/4233835071] conn(0x565082b12d20 0x565082b354c0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:46.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.851+0000 7f5adb351b80 1 --2- 172.21.15.16:0/4183184343 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x565082b3b220 0x565082b16930 unknown :-1 s=CLOSED pgs=2309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:46.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.852+0000 7f5aa77fe700 0 client.11049 ms_handle_reset on v2:172.21.15.16:6835/4233835071 2023-10-16T13:44:46.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.852+0000 7f5adb351b80 1 -- 172.21.15.16:0/4183184343 >> 172.21.15.16:0/4183184343 conn(0x565082a45d80 msgr2=0x565082b57610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:46.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.852+0000 7f5adb351b80 1 -- 172.21.15.16:0/4183184343 shutdown_connections 2023-10-16T13:44:46.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.852+0000 7f5adb351b80 1 -- 172.21.15.16:0/4183184343 wait complete. 2023-10-16T13:44:46.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.852+0000 7f5adb351b80 20 client.11049 trim_cache size 0 max 16384 2023-10-16T13:44:46.880 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-10-16T13:44:46.880 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "5b28dfa1-b21f-474a-9b2e-9fb77af58254", 2023-10-16T13:44:46.880 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 0, 2023-10-16T13:44:46.880 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 10899, 2023-10-16T13:44:46.880 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2023-10-16T13:44:46.881 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2023-10-16T13:44:46.881 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2023-10-16T13:44:46.881 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 6.0305344029999999, 2023-10-16T13:44:46.881 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 104, 2023-10-16T13:44:46.881 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 206, 2023-10-16T13:44:46.882 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 206, 2023-10-16T13:44:46.882 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 10.952180694000001 2023-10-16T13:44:46.882 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-10-16T13:44:46.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.852+0000 7f5acf167700 1 -- 172.21.15.16:0/340988831 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f5aac06b020 msgr2=0x7f5aac06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:46.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.852+0000 7f5acf167700 1 --2- 172.21.15.16:0/340988831 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f5aac06b020 0x7f5aac06d4e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5abc005040 tx=0x7f5abc00a000 comp rx=0 tx=0).stop 2023-10-16T13:44:46.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.852+0000 7f5acf167700 1 -- 172.21.15.16:0/340988831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ac80ca1d0 msgr2=0x7f5ac81594d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:46.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.852+0000 7f5acf167700 1 --2- 172.21.15.16:0/340988831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ac80ca1d0 0x7f5ac81594d0 secure :-1 s=READY pgs=2305 cs=0 l=1 rev1=1 crypto rx=0x7f5ac0001330 tx=0x7f5ac000e440 comp rx=0 tx=0).stop 2023-10-16T13:44:46.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.852+0000 7f5acf167700 1 -- 172.21.15.16:0/340988831 shutdown_connections 2023-10-16T13:44:46.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.852+0000 7f5acf167700 1 --2- 172.21.15.16:0/340988831 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f5aac06b020 0x7f5aac06d4e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:46.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.852+0000 7f5acf167700 1 --2- 172.21.15.16:0/340988831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ac80ca1d0 0x7f5ac81594d0 unknown :-1 s=CLOSED pgs=2305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:46.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.852+0000 7f5acf167700 1 --2- 172.21.15.16:0/340988831 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5ab8003bc0 0x7f5ac815d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:46.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.852+0000 7f5acf167700 1 --2- 172.21.15.16:0/340988831 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5ac81f2640 0x7f5ac81566f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:46.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.852+0000 7f5acf167700 1 -- 172.21.15.16:0/340988831 >> 172.21.15.16:0/340988831 conn(0x7f5ac80b1fd0 msgr2=0x7f5ac80d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:46.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.852+0000 7f5acf167700 1 -- 172.21.15.16:0/340988831 shutdown_connections 2023-10-16T13:44:46.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:46.852+0000 7f5acf167700 1 -- 172.21.15.16:0/340988831 wait complete. 2023-10-16T13:44:46.886 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T13:44:47.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.009+0000 7f0d82766700 1 Processor -- start 2023-10-16T13:44:47.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.009+0000 7f0d82766700 1 -- start start 2023-10-16T13:44:47.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.009+0000 7f0d82766700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f0d7c06b900 0x7f0d7c06d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:47.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.009+0000 7f0d82766700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d7c06c150 con 0x7f0d7c160eb0 2023-10-16T13:44:47.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.009+0000 7f0d82766700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d7c06be70 con 0x7f0d7c0cb280 2023-10-16T13:44:47.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.009+0000 7f0d82766700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d7c0af370 con 0x7f0d7c06b900 2023-10-16T13:44:47.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.009+0000 7f0d7bfff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0d7c160eb0 0x7f0d7c0cab70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41140/0 (socket says 172.21.15.16:41140) 2023-10-16T13:44:47.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.009+0000 7f0d7bfff700 1 -- 172.21.15.16:0/3998346804 learned_addr learned my addr 172.21.15.16:0/3998346804 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:44:47.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.009+0000 7f0d80d03700 1 --2- 172.21.15.16:0/3998346804 >> v2:172.21.15.49:3300/0 conn(0x7f0d7c06b900 0x7f0d7c06d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:47.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.009+0000 7f0d7affd700 1 -- 172.21.15.16:0/3998346804 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1825947765 0 0) 0x7f0d7c06c150 con 0x7f0d7c160eb0 2023-10-16T13:44:47.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.009+0000 7f0d7affd700 1 -- 172.21.15.16:0/3998346804 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d640035b0 con 0x7f0d7c160eb0 2023-10-16T13:44:47.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.009+0000 7f0d7affd700 1 -- 172.21.15.16:0/3998346804 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2513479184 0 0) 0x7f0d640035b0 con 0x7f0d7c160eb0 2023-10-16T13:44:47.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.009+0000 7f0d7affd700 1 -- 172.21.15.16:0/3998346804 >> v2:172.21.15.49:3300/0 conn(0x7f0d7c06b900 msgr2=0x7f0d7c06d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:47.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.009+0000 7f0d7affd700 1 --2- 172.21.15.16:0/3998346804 >> v2:172.21.15.49:3300/0 conn(0x7f0d7c06b900 0x7f0d7c06d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:47.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.009+0000 7f0d7affd700 1 -- 172.21.15.16:0/3998346804 >> v1:172.21.15.49:6789/0 conn(0x7f0d7c0cb280 legacy=0x7f0d7c06b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:47.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.009+0000 7f0d7affd700 1 -- 172.21.15.16:0/3998346804 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d7c0cdf10 con 0x7f0d7c160eb0 2023-10-16T13:44:47.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.009+0000 7f0d7affd700 1 -- 172.21.15.16:0/3998346804 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f0d6c001670 con 0x7f0d7c160eb0 2023-10-16T13:44:47.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.009+0000 7f0d7affd700 1 -- 172.21.15.16:0/3998346804 >> v1:172.21.15.16:6789/0 conn(0x7f0d7c160eb0 legacy=0x7f0d7c0cab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:47.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.009+0000 7f0d7affd700 1 --2- 172.21.15.16:0/3998346804 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d64003a00 0x7f0d64003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:47.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.009+0000 7f0d7affd700 1 --2- 172.21.15.16:0/3998346804 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0d64004850 0x7f0d64004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:47.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.009+0000 7f0d7affd700 1 --2- 172.21.15.16:0/3998346804 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0d64005210 0x7f0d6400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:47.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.009+0000 7f0d7affd700 1 -- 172.21.15.16:0/3998346804 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d6400c090 con 0x7f0d64003a00 2023-10-16T13:44:47.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.009+0000 7f0d7affd700 1 -- 172.21.15.16:0/3998346804 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d6400c200 con 0x7f0d64004850 2023-10-16T13:44:47.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.009+0000 7f0d7affd700 1 -- 172.21.15.16:0/3998346804 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d6400c3c0 con 0x7f0d64005210 2023-10-16T13:44:47.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.009+0000 7f0d7bfff700 1 --2- 172.21.15.16:0/3998346804 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d64003a00 0x7f0d64003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:47.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.009+0000 7f0d7b7fe700 1 --2- 172.21.15.16:0/3998346804 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0d64004850 0x7f0d64004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:47.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.010+0000 7f0d80d03700 1 --2- 172.21.15.16:0/3998346804 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0d64005210 0x7f0d6400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:47.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.010+0000 7f0d7bfff700 1 -- 172.21.15.16:0/3998346804 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0d64005210 msgr2=0x7f0d6400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:47.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.010+0000 7f0d7bfff700 1 --2- 172.21.15.16:0/3998346804 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0d64005210 0x7f0d6400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:47.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.010+0000 7f0d7bfff700 1 -- 172.21.15.16:0/3998346804 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0d64004850 msgr2=0x7f0d64004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:47.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.010+0000 7f0d7bfff700 1 --2- 172.21.15.16:0/3998346804 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0d64004850 0x7f0d64004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:47.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.010+0000 7f0d7bfff700 1 -- 172.21.15.16:0/3998346804 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d640035b0 con 0x7f0d64003a00 2023-10-16T13:44:47.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.010+0000 7f0d80d03700 1 -- 172.21.15.16:0/3998346804 reap_dead start 2023-10-16T13:44:47.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.010+0000 7f0d7bfff700 1 --2- 172.21.15.16:0/3998346804 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d64003a00 0x7f0d64003e20 secure :-1 s=READY pgs=2311 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c00c240 tx=0x7f0d64005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be8392f1663cdeeb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:47.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.010+0000 7f0d7affd700 1 -- 172.21.15.16:0/3998346804 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d6c007c10 con 0x7f0d64003a00 2023-10-16T13:44:47.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.010+0000 7f0d7affd700 1 -- 172.21.15.16:0/3998346804 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d6c00ce00 con 0x7f0d64003a00 2023-10-16T13:44:47.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.010+0000 7f0d7affd700 1 -- 172.21.15.16:0/3998346804 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d6c005ba0 con 0x7f0d64003a00 2023-10-16T13:44:47.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.010+0000 7f0d82766700 1 -- 172.21.15.16:0/3998346804 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d64003a00 msgr2=0x7f0d64003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:47.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.010+0000 7f0d82766700 1 --2- 172.21.15.16:0/3998346804 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d64003a00 0x7f0d64003e20 secure :-1 s=READY pgs=2311 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c00c240 tx=0x7f0d64005880 comp rx=0 tx=0).stop 2023-10-16T13:44:47.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.011+0000 7f0d82766700 1 -- 172.21.15.16:0/3998346804 shutdown_connections 2023-10-16T13:44:47.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.011+0000 7f0d82766700 1 --2- 172.21.15.16:0/3998346804 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d64003a00 0x7f0d64003e20 unknown :-1 s=CLOSED pgs=2311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:47.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.011+0000 7f0d82766700 1 -- 172.21.15.16:0/3998346804 >> 172.21.15.16:0/3998346804 conn(0x7f0d7c0cc5b0 msgr2=0x7f0d7c0cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:47.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.011+0000 7f0d82766700 1 -- 172.21.15.16:0/3998346804 shutdown_connections 2023-10-16T13:44:47.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.011+0000 7f0d82766700 1 -- 172.21.15.16:0/3998346804 wait complete. 2023-10-16T13:44:47.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.012+0000 7f0d82766700 1 Processor -- start 2023-10-16T13:44:47.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.012+0000 7f0d82766700 1 -- start start 2023-10-16T13:44:47.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.013+0000 7f0d82766700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d64003a00 0x7f0d7c0b34f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:47.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.013+0000 7f0d82766700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0d7c0b5790 0x7f0d7c1f62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:47.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.013+0000 7f0d82766700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0d7c1f6820 0x7f0d7c0c8a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:47.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.013+0000 7f0d82766700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d7c1f6cb0 con 0x7f0d64003a00 2023-10-16T13:44:47.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.013+0000 7f0d82766700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d7c0c4340 con 0x7f0d7c1f6820 2023-10-16T13:44:47.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.013+0000 7f0d82766700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d7c0c3770 con 0x7f0d7c0b5790 2023-10-16T13:44:47.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.013+0000 7f0d7bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d64003a00 0x7f0d7c0b34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:47.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.013+0000 7f0d7bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d64003a00 0x7f0d7c0b34f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51878/0 (socket says 172.21.15.16:51878) 2023-10-16T13:44:47.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.013+0000 7f0d7bfff700 1 -- 172.21.15.16:0/2128902501 learned_addr learned my addr 172.21.15.16:0/2128902501 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:47.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.013+0000 7f0d7b7fe700 1 --2- 172.21.15.16:0/2128902501 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0d7c0b5790 0x7f0d7c1f62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:47.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.013+0000 7f0d80d03700 1 --2- 172.21.15.16:0/2128902501 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0d7c1f6820 0x7f0d7c0c8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:47.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.013+0000 7f0d7bfff700 1 -- 172.21.15.16:0/2128902501 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0d7c0b5790 msgr2=0x7f0d7c1f62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:47.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.013+0000 7f0d7bfff700 1 --2- 172.21.15.16:0/2128902501 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0d7c0b5790 0x7f0d7c1f62e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:47.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.013+0000 7f0d7bfff700 1 -- 172.21.15.16:0/2128902501 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0d7c1f6820 msgr2=0x7f0d7c0c8a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:47.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.013+0000 7f0d7bfff700 1 --2- 172.21.15.16:0/2128902501 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0d7c1f6820 0x7f0d7c0c8a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:47.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.013+0000 7f0d7bfff700 1 -- 172.21.15.16:0/2128902501 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d7c0c3a10 con 0x7f0d64003a00 2023-10-16T13:44:47.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.013+0000 7f0d7bfff700 1 --2- 172.21.15.16:0/2128902501 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d64003a00 0x7f0d7c0b34f0 secure :-1 s=READY pgs=2312 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c000c00 tx=0x7f0d6c001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:47.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.013+0000 7f0d78ff9700 1 -- 172.21.15.16:0/2128902501 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d6c004c80 con 0x7f0d64003a00 2023-10-16T13:44:47.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.013+0000 7f0d78ff9700 1 -- 172.21.15.16:0/2128902501 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d6c004de0 con 0x7f0d64003a00 2023-10-16T13:44:47.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.014+0000 7f0d82766700 1 -- 172.21.15.16:0/2128902501 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d7c0c3cf0 con 0x7f0d64003a00 2023-10-16T13:44:47.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.014+0000 7f0d78ff9700 1 -- 172.21.15.16:0/2128902501 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d6c007310 con 0x7f0d64003a00 2023-10-16T13:44:47.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.014+0000 7f0d82766700 1 -- 172.21.15.16:0/2128902501 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d7c0b3a30 con 0x7f0d64003a00 2023-10-16T13:44:47.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.014+0000 7f0d82766700 1 -- 172.21.15.16:0/2128902501 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d7c04ebb0 con 0x7f0d64003a00 2023-10-16T13:44:47.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.015+0000 7f0d78ff9700 1 -- 172.21.15.16:0/2128902501 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 126) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0d6c007470 con 0x7f0d64003a00 2023-10-16T13:44:47.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.015+0000 7f0d78ff9700 1 --2- 172.21.15.16:0/2128902501 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f0d6806b020 0x7f0d6806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:47.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.016+0000 7f0d7b7fe700 1 --2- 172.21.15.16:0/2128902501 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f0d6806b020 0x7f0d6806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:47.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.016+0000 7f0d78ff9700 1 -- 172.21.15.16:0/2128902501 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(206..206 src has 1..206) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f0d6c08a120 con 0x7f0d64003a00 2023-10-16T13:44:47.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.016+0000 7f0d7b7fe700 1 --2- 172.21.15.16:0/2128902501 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f0d6806b020 0x7f0d6806d4e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0d7c0c7da0 tx=0x7f0d70009040 comp rx=0 tx=0).ready entity=mgr.10665 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:47.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.020+0000 7f0d78ff9700 1 -- 172.21.15.16:0/2128902501 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0d6c054f90 con 0x7f0d64003a00 2023-10-16T13:44:47.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.270+0000 7f0d82766700 1 -- 172.21.15.16:0/2128902501 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f0d7c0aeb00 con 0x7f0d64003a00 2023-10-16T13:44:47.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.270+0000 7f0d78ff9700 1 -- 172.21.15.16:0/2128902501 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v104) v1 ==== 53+0+73 (secure 0 0 0) 0x7f0d6c017060 con 0x7f0d64003a00 2023-10-16T13:44:47.273 INFO:teuthology.orchestra.run.smithi016.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-16T13:44:47.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.273+0000 7f0d82766700 1 -- 172.21.15.16:0/2128902501 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f0d6806b020 msgr2=0x7f0d6806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:47.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.273+0000 7f0d82766700 1 --2- 172.21.15.16:0/2128902501 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f0d6806b020 0x7f0d6806d4e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0d7c0c7da0 tx=0x7f0d70009040 comp rx=0 tx=0).stop 2023-10-16T13:44:47.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.273+0000 7f0d82766700 1 -- 172.21.15.16:0/2128902501 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d64003a00 msgr2=0x7f0d7c0b34f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:47.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.273+0000 7f0d82766700 1 --2- 172.21.15.16:0/2128902501 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d64003a00 0x7f0d7c0b34f0 secure :-1 s=READY pgs=2312 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c000c00 tx=0x7f0d6c001a00 comp rx=0 tx=0).stop 2023-10-16T13:44:47.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.273+0000 7f0d82766700 1 -- 172.21.15.16:0/2128902501 shutdown_connections 2023-10-16T13:44:47.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.273+0000 7f0d82766700 1 --2- 172.21.15.16:0/2128902501 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f0d6806b020 0x7f0d6806d4e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:47.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.273+0000 7f0d82766700 1 --2- 172.21.15.16:0/2128902501 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0d7c1f6820 0x7f0d7c0c8a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:47.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.273+0000 7f0d82766700 1 --2- 172.21.15.16:0/2128902501 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d64003a00 0x7f0d7c0b34f0 unknown :-1 s=CLOSED pgs=2312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:47.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.273+0000 7f0d82766700 1 --2- 172.21.15.16:0/2128902501 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0d7c0b5790 0x7f0d7c1f62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:47.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.273+0000 7f0d82766700 1 -- 172.21.15.16:0/2128902501 >> 172.21.15.16:0/2128902501 conn(0x7f0d7c0cc5b0 msgr2=0x7f0d7c06ae10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:47.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.273+0000 7f0d82766700 1 -- 172.21.15.16:0/2128902501 shutdown_connections 2023-10-16T13:44:47.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:47.273+0000 7f0d82766700 1 -- 172.21.15.16:0/2128902501 wait complete. 2023-10-16T13:44:47.287 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-16T13:44:47.288 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T13:44:47.288 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi016.front.sepia.ceph.com 2023-10-16T13:44:47.288 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-16T13:44:47.289 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-16T13:44:47.289 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-16T13:44:47.289 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-16T13:44:47.290 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T13:44:47.290 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T13:44:47.291 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:44:47.291 DEBUG:teuthology.orchestra.run.smithi016:> ip addr 2023-10-16T13:44:47.309 INFO:teuthology.orchestra.run.smithi016.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T13:44:47.309 INFO:teuthology.orchestra.run.smithi016.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T13:44:47.310 INFO:teuthology.orchestra.run.smithi016.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T13:44:47.310 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T13:44:47.310 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 ::1/128 scope host 2023-10-16T13:44:47.310 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T13:44:47.310 INFO:teuthology.orchestra.run.smithi016.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T13:44:47.311 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e0 brd ff:ff:ff:ff:ff:ff 2023-10-16T13:44:47.311 INFO:teuthology.orchestra.run.smithi016.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T13:44:47.311 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e1 brd ff:ff:ff:ff:ff:ff 2023-10-16T13:44:47.311 INFO:teuthology.orchestra.run.smithi016.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T13:44:47.311 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:64 brd ff:ff:ff:ff:ff:ff 2023-10-16T13:44:47.312 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.21.15.16/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-16T13:44:47.312 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft 36649sec preferred_lft 36649sec 2023-10-16T13:44:47.312 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::ec4:7aff:febd:1564/64 scope link 2023-10-16T13:44:47.312 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T13:44:47.312 INFO:teuthology.orchestra.run.smithi016.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T13:44:47.312 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:65 brd ff:ff:ff:ff:ff:ff 2023-10-16T13:44:47.313 INFO:teuthology.orchestra.run.smithi016.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T13:44:47.313 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 36:cf:ea:25:db:f7 brd ff:ff:ff:ff:ff:ff 2023-10-16T13:44:47.313 INFO:teuthology.orchestra.run.smithi016.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-16T13:44:47.313 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T13:44:47.313 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::cfc:c5ff:fe52:a786/64 scope link 2023-10-16T13:44:47.313 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T13:44:47.314 INFO:teuthology.orchestra.run.smithi016.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T13:44:47.314 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 6e:62:9d:a4:43:23 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T13:44:47.314 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::6c62:9dff:fea4:4323/64 scope link 2023-10-16T13:44:47.314 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T13:44:47.314 INFO:teuthology.orchestra.run.smithi016.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T13:44:47.315 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 36:cf:ea:25:db:f7 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-16T13:44:47.315 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::34cf:eaff:fe25:dbf7/64 scope link 2023-10-16T13:44:47.315 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T13:44:47.315 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:44:47.316 DEBUG:teuthology.orchestra.run.smithi016:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T13:44:47.375 INFO:teuthology.orchestra.run.smithi016.stdout:1 2023-10-16T13:44:47.376 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:44:47.377 DEBUG:teuthology.orchestra.run.smithi016:> route 2023-10-16T13:44:47.438 INFO:teuthology.orchestra.run.smithi016.stdout:Kernel IP routing table 2023-10-16T13:44:47.438 INFO:teuthology.orchestra.run.smithi016.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T13:44:47.439 INFO:teuthology.orchestra.run.smithi016.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-16T13:44:47.439 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-16T13:44:47.439 INFO:teuthology.orchestra.run.smithi016.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T13:44:47.439 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:44:47.440 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec bash -c ' 2023-10-16T13:44:47.440 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2023-10-16T13:44:47.441 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-16T13:44:47.441 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-16T13:44:47.441 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-16T13:44:47.442 DEBUG:teuthology.orchestra.run.smithi016:> ') 2023-10-16T13:44:47.545 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:44:47.545 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2023-10-16T13:44:47.563 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-16T13:44:47.563 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-16T13:44:47.563 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T13:44:47.564 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:44:47.564 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set brx.0 up 2023-10-16T13:44:47.630 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T13:44:47.631 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-16T13:44:47.687 INFO:teuthology.orchestra.run.smithi016.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-16T13:44:47.688 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T13:44:47.688 DEBUG:teuthology.orchestra.run.smithi016:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-16T13:44:47.744 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-16T13:44:47.744 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:44:47.744 DEBUG:teuthology.orchestra.run.smithi016:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T13:44:47.831 INFO:teuthology.orchestra.run.smithi016.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T13:44:47.831 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T13:44:47.832 INFO:teuthology.orchestra.run.smithi016.stdout:invalid new device string format 2023-10-16T13:44:47.840 INFO:teuthology.orchestra.run.smithi016.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T13:44:47.840 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T13:44:47.841 INFO:teuthology.orchestra.run.smithi016.stdout:invalid new device string format 2023-10-16T13:44:47.841 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: resolved to: "172.21.15.16:3300,172.21.15.49:3300,172.21.15.49:3301" 2023-10-16T13:44:47.841 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T13:44:47.879 INFO:teuthology.orchestra.run.smithi016.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T13:44:47.880 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T13:44:47.880 INFO:teuthology.orchestra.run.smithi016.stdout:invalid new device string format 2023-10-16T13:44:47.880 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: resolved to: "172.21.15.16:3300,172.21.15.49:3300,172.21.15.49:3301" 2023-10-16T13:44:47.880 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T13:44:47.881 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: resolved to: "172.21.15.16:6789,172.21.15.49:6789,172.21.15.49:6790" 2023-10-16T13:44:47.881 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.16:6789,172.21.15.49:6789,172.21.15.49:6790:/ 2023-10-16T13:44:47.881 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=5b28dfa1-b21f-474a-9b2e-9fb77af58254" will pass to kernel 2023-10-16T13:44:47.886 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-16T13:44:47.886 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:44:47.887 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-16T13:44:47.921 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:44:47.921 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T13:44:48.004 INFO:teuthology.orchestra.run.smithi016.stdout:client11058 2023-10-16T13:44:48.004 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T13:44:48.005 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:44:48.061 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-16T13:44:48.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.224+0000 7fdf13933700 1 Processor -- start 2023-10-16T13:44:48.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.225+0000 7fdf13933700 1 -- start start 2023-10-16T13:44:48.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.225+0000 7fdf13933700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fdf0c0cc8a0 0x7fdf0c15e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:48.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.226+0000 7fdf13933700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf0c161c30 con 0x7fdf0c15b280 2023-10-16T13:44:48.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.226+0000 7fdf13933700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf0c161f10 con 0x7fdf0c06a7e0 2023-10-16T13:44:48.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.226+0000 7fdf13933700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf0c1621f0 con 0x7fdf0c0cc8a0 2023-10-16T13:44:48.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.226+0000 7fdf116cf700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fdf0c15b280 0x7fdf0c1613e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41164/0 (socket says 172.21.15.16:41164) 2023-10-16T13:44:48.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.226+0000 7fdf116cf700 1 -- 172.21.15.16:0/1045563989 learned_addr learned my addr 172.21.15.16:0/1045563989 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:44:48.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.226+0000 7fdf10ece700 1 --2- 172.21.15.16:0/1045563989 >> v2:172.21.15.49:3300/0 conn(0x7fdf0c0cc8a0 0x7fdf0c15e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:48.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.226+0000 7fdefffff700 1 -- 172.21.15.16:0/1045563989 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3323683870 0 0) 0x7fdf0c161c30 con 0x7fdf0c15b280 2023-10-16T13:44:48.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.226+0000 7fdefffff700 1 -- 172.21.15.16:0/1045563989 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdef80035b0 con 0x7fdf0c15b280 2023-10-16T13:44:48.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.226+0000 7fdf10ece700 1 -- 172.21.15.16:0/1045563989 >> v1:172.21.15.49:6789/0 conn(0x7fdf0c06a7e0 legacy=0x7fdf0c0d02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:48.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.226+0000 7fdf10ece700 1 -- 172.21.15.16:0/1045563989 >> v1:172.21.15.16:6789/0 conn(0x7fdf0c15b280 legacy=0x7fdf0c1613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:48.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.226+0000 7fdf10ece700 1 -- 172.21.15.16:0/1045563989 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf0c15df00 con 0x7fdf0c0cc8a0 2023-10-16T13:44:48.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.226+0000 7fdf10ece700 1 --2- 172.21.15.16:0/1045563989 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdf0c0cc8a0 0x7fdf0c15e700 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fdf00009840 tx=0x7fdf00009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd71327a6f90ba8b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:48.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.227+0000 7fdefffff700 1 -- 172.21.15.16:0/1045563989 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf00011830 con 0x7fdf0c0cc8a0 2023-10-16T13:44:48.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.227+0000 7fdefffff700 1 -- 172.21.15.16:0/1045563989 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf00011990 con 0x7fdf0c0cc8a0 2023-10-16T13:44:48.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.227+0000 7fdefffff700 1 -- 172.21.15.16:0/1045563989 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf00011d20 con 0x7fdf0c0cc8a0 2023-10-16T13:44:48.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.227+0000 7fdf13933700 1 -- 172.21.15.16:0/1045563989 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdf0c0cc8a0 msgr2=0x7fdf0c15e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:48.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.227+0000 7fdf13933700 1 --2- 172.21.15.16:0/1045563989 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdf0c0cc8a0 0x7fdf0c15e700 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fdf00009840 tx=0x7fdf00009b50 comp rx=0 tx=0).stop 2023-10-16T13:44:48.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.227+0000 7fdf13933700 1 -- 172.21.15.16:0/1045563989 shutdown_connections 2023-10-16T13:44:48.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.227+0000 7fdf13933700 1 --2- 172.21.15.16:0/1045563989 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdf0c0cc8a0 0x7fdf0c15e700 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:48.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.227+0000 7fdf13933700 1 -- 172.21.15.16:0/1045563989 >> 172.21.15.16:0/1045563989 conn(0x7fdf0c0b1fd0 msgr2=0x7fdf0c0b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:48.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.227+0000 7fdf13933700 1 -- 172.21.15.16:0/1045563989 shutdown_connections 2023-10-16T13:44:48.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.227+0000 7fdf13933700 1 -- 172.21.15.16:0/1045563989 wait complete. 2023-10-16T13:44:48.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.229+0000 7fdf13933700 1 Processor -- start 2023-10-16T13:44:48.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.229+0000 7fdf13933700 1 -- start start 2023-10-16T13:44:48.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.230+0000 7fdf13933700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdf0c06a7e0 0x7fdf0c1f59e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:48.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.230+0000 7fdf13933700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf0c0cc8a0 0x7fdf0c0c6730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:48.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.230+0000 7fdf13933700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdf0c15b280 0x7fdf0c15c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:48.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.230+0000 7fdf13933700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf0c0c5cb0 con 0x7fdf0c0cc8a0 2023-10-16T13:44:48.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.230+0000 7fdf13933700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf0c0ad630 con 0x7fdf0c06a7e0 2023-10-16T13:44:48.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.230+0000 7fdf13933700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf0c0ad910 con 0x7fdf0c15b280 2023-10-16T13:44:48.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.230+0000 7fdf10ece700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf0c0cc8a0 0x7fdf0c0c6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:48.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.230+0000 7fdf10ece700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf0c0cc8a0 0x7fdf0c0c6730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51898/0 (socket says 172.21.15.16:51898) 2023-10-16T13:44:48.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.230+0000 7fdf10ece700 1 -- 172.21.15.16:0/3113398919 learned_addr learned my addr 172.21.15.16:0/3113398919 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:48.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.230+0000 7fdf116cf700 1 --2- 172.21.15.16:0/3113398919 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdf0c06a7e0 0x7fdf0c1f59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:48.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.230+0000 7fdf11ed0700 1 --2- 172.21.15.16:0/3113398919 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdf0c15b280 0x7fdf0c15c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:48.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.230+0000 7fdf10ece700 1 -- 172.21.15.16:0/3113398919 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdf0c15b280 msgr2=0x7fdf0c15c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:48.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.230+0000 7fdf10ece700 1 --2- 172.21.15.16:0/3113398919 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdf0c15b280 0x7fdf0c15c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:48.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.230+0000 7fdf10ece700 1 -- 172.21.15.16:0/3113398919 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdf0c06a7e0 msgr2=0x7fdf0c1f59e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:48.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.230+0000 7fdf10ece700 1 --2- 172.21.15.16:0/3113398919 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdf0c06a7e0 0x7fdf0c1f59e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:48.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.230+0000 7fdf10ece700 1 -- 172.21.15.16:0/3113398919 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf000094a0 con 0x7fdf0c0cc8a0 2023-10-16T13:44:48.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.230+0000 7fdf10ece700 1 --2- 172.21.15.16:0/3113398919 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf0c0cc8a0 0x7fdf0c0c6730 secure :-1 s=READY pgs=2315 cs=0 l=1 rev1=1 crypto rx=0x7fdf00009fd0 tx=0x7fdf00004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:48.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.230+0000 7fdefdffb700 1 -- 172.21.15.16:0/3113398919 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf000036a0 con 0x7fdf0c0cc8a0 2023-10-16T13:44:48.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.230+0000 7fdefdffb700 1 -- 172.21.15.16:0/3113398919 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf0001f070 con 0x7fdf0c0cc8a0 2023-10-16T13:44:48.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.230+0000 7fdefdffb700 1 -- 172.21.15.16:0/3113398919 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf00003a60 con 0x7fdf0c0cc8a0 2023-10-16T13:44:48.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.231+0000 7fdf13933700 1 -- 172.21.15.16:0/3113398919 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf0c0adbf0 con 0x7fdf0c0cc8a0 2023-10-16T13:44:48.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.231+0000 7fdf13933700 1 -- 172.21.15.16:0/3113398919 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf0c0ae1b0 con 0x7fdf0c0cc8a0 2023-10-16T13:44:48.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.232+0000 7fdf13933700 1 -- 172.21.15.16:0/3113398919 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf0c0ae390 con 0x7fdf0c0cc8a0 2023-10-16T13:44:48.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.232+0000 7fdefdffb700 1 -- 172.21.15.16:0/3113398919 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 126) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fdf00003bc0 con 0x7fdf0c0cc8a0 2023-10-16T13:44:48.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.233+0000 7fdefdffb700 1 --2- 172.21.15.16:0/3113398919 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7fdef006b020 0x7fdef006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:48.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.233+0000 7fdefdffb700 1 -- 172.21.15.16:0/3113398919 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(206..206 src has 1..206) v4 ==== 6157+0+0 (secure 0 0 0) 0x7fdf00021070 con 0x7fdf0c0cc8a0 2023-10-16T13:44:48.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.233+0000 7fdf116cf700 1 --2- 172.21.15.16:0/3113398919 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7fdef006b020 0x7fdef006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:48.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.233+0000 7fdf116cf700 1 --2- 172.21.15.16:0/3113398919 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7fdef006b020 0x7fdef006d4e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fdf08007fd0 tx=0x7fdf08007dc0 comp rx=0 tx=0).ready entity=mgr.10665 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:48.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.238+0000 7fdefdffb700 1 -- 172.21.15.16:0/3113398919 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdf00055f20 con 0x7fdf0c0cc8a0 2023-10-16T13:44:48.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.479+0000 7fdf13933700 1 -- 172.21.15.16:0/3113398919 --> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fdf0c15ec40 con 0x7fdef006b020 2023-10-16T13:44:48.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.480+0000 7fdefdffb700 1 -- 172.21.15.16:0/3113398919 <== mgr.10665 v2:172.21.15.16:6832/1268439414 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fdf080058e0 con 0x7fdef006b020 2023-10-16T13:44:48.483 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2023-10-16T13:44:48.483 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:44:48.483 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cephfs" 2023-10-16T13:44:48.483 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:44:48.484 INFO:teuthology.orchestra.run.smithi016.stdout:] 2023-10-16T13:44:48.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.482+0000 7fdf13933700 1 -- 172.21.15.16:0/3113398919 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7fdef006b020 msgr2=0x7fdef006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:48.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.482+0000 7fdf13933700 1 --2- 172.21.15.16:0/3113398919 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7fdef006b020 0x7fdef006d4e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fdf08007fd0 tx=0x7fdf08007dc0 comp rx=0 tx=0).stop 2023-10-16T13:44:48.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.482+0000 7fdf13933700 1 -- 172.21.15.16:0/3113398919 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf0c0cc8a0 msgr2=0x7fdf0c0c6730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:48.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.482+0000 7fdf13933700 1 --2- 172.21.15.16:0/3113398919 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf0c0cc8a0 0x7fdf0c0c6730 secure :-1 s=READY pgs=2315 cs=0 l=1 rev1=1 crypto rx=0x7fdf00009fd0 tx=0x7fdf00004750 comp rx=0 tx=0).stop 2023-10-16T13:44:48.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.483+0000 7fdf13933700 1 -- 172.21.15.16:0/3113398919 shutdown_connections 2023-10-16T13:44:48.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.483+0000 7fdf13933700 1 --2- 172.21.15.16:0/3113398919 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7fdef006b020 0x7fdef006d4e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:48.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.483+0000 7fdf13933700 1 --2- 172.21.15.16:0/3113398919 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdf0c15b280 0x7fdf0c15c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:48.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.483+0000 7fdf13933700 1 --2- 172.21.15.16:0/3113398919 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdf0c06a7e0 0x7fdf0c1f59e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:48.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.483+0000 7fdf13933700 1 --2- 172.21.15.16:0/3113398919 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf0c0cc8a0 0x7fdf0c0c6730 unknown :-1 s=CLOSED pgs=2315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:48.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.483+0000 7fdf13933700 1 -- 172.21.15.16:0/3113398919 >> 172.21.15.16:0/3113398919 conn(0x7fdf0c0b1fd0 msgr2=0x7fdf0c0cd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:48.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.483+0000 7fdf13933700 1 -- 172.21.15.16:0/3113398919 shutdown_connections 2023-10-16T13:44:48.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.483+0000 7fdf13933700 1 -- 172.21.15.16:0/3113398919 wait complete. 2023-10-16T13:44:48.498 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-16T13:44:48.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.623+0000 7f4b50bf9700 1 Processor -- start 2023-10-16T13:44:48.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.623+0000 7f4b50bf9700 1 -- start start 2023-10-16T13:44:48.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.624+0000 7f4b50bf9700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4b4c15b070 0x7f4b4c15b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:48.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.624+0000 7f4b50bf9700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4b4c06e170 0x7f4b4c0cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:48.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.624+0000 7f4b50bf9700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b4c0cef70 con 0x7f4b4c0cb880 2023-10-16T13:44:48.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.624+0000 7f4b50bf9700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b4c0ce360 con 0x7f4b4c06e170 2023-10-16T13:44:48.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.624+0000 7f4b50bf9700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b4c0ce640 con 0x7f4b4c15b070 2023-10-16T13:44:48.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.624+0000 7f4b49d9b700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f4b4c0cb880 0x7f4b4c06da60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41178/0 (socket says 172.21.15.16:41178) 2023-10-16T13:44:48.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.624+0000 7f4b4ad9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4b4c06e170 0x7f4b4c0cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:48.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.624+0000 7f4b49d9b700 1 -- 172.21.15.16:0/4104413164 learned_addr learned my addr 172.21.15.16:0/4104413164 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:44:48.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.625+0000 7f4b4a59c700 1 --2- 172.21.15.16:0/4104413164 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4b4c15b070 0x7f4b4c15b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:48.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.625+0000 7f4b4ad9d700 1 -- 172.21.15.16:0/4104413164 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4b4c15b070 msgr2=0x7f4b4c15b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:48.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.625+0000 7f4b4ad9d700 1 --2- 172.21.15.16:0/4104413164 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4b4c15b070 0x7f4b4c15b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:48.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.625+0000 7f4b4ad9d700 1 -- 172.21.15.16:0/4104413164 >> v1:172.21.15.16:6789/0 conn(0x7f4b4c0cb880 legacy=0x7f4b4c06da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:48.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.625+0000 7f4b4ad9d700 1 -- 172.21.15.16:0/4104413164 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b4c0ce920 con 0x7f4b4c06e170 2023-10-16T13:44:48.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.625+0000 7f4b4ad9d700 1 --2- 172.21.15.16:0/4104413164 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b4c06e170 0x7f4b4c0cbd00 secure :-1 s=READY pgs=2317 cs=0 l=1 rev1=1 crypto rx=0x7f4b44009d90 tx=0x7f4b4400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad3bea2e3c29fa2e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:48.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.625+0000 7f4b4959a700 1 -- 172.21.15.16:0/4104413164 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b4400e070 con 0x7f4b4c06e170 2023-10-16T13:44:48.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.625+0000 7f4b4959a700 1 -- 172.21.15.16:0/4104413164 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b4400bbd0 con 0x7f4b4c06e170 2023-10-16T13:44:48.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.625+0000 7f4b4959a700 1 -- 172.21.15.16:0/4104413164 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b44012430 con 0x7f4b4c06e170 2023-10-16T13:44:48.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.626+0000 7f4b50bf9700 1 -- 172.21.15.16:0/4104413164 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b4c06e170 msgr2=0x7f4b4c0cbd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:48.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.626+0000 7f4b50bf9700 1 --2- 172.21.15.16:0/4104413164 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b4c06e170 0x7f4b4c0cbd00 secure :-1 s=READY pgs=2317 cs=0 l=1 rev1=1 crypto rx=0x7f4b44009d90 tx=0x7f4b4400b330 comp rx=0 tx=0).stop 2023-10-16T13:44:48.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.626+0000 7f4b50bf9700 1 -- 172.21.15.16:0/4104413164 shutdown_connections 2023-10-16T13:44:48.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.626+0000 7f4b50bf9700 1 --2- 172.21.15.16:0/4104413164 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b4c06e170 0x7f4b4c0cbd00 unknown :-1 s=CLOSED pgs=2317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:48.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.626+0000 7f4b50bf9700 1 --2- 172.21.15.16:0/4104413164 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4b4c15b070 0x7f4b4c15b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:48.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.626+0000 7f4b50bf9700 1 -- 172.21.15.16:0/4104413164 >> 172.21.15.16:0/4104413164 conn(0x7f4b4c0b1f40 msgr2=0x7f4b4c0b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:48.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.626+0000 7f4b50bf9700 1 -- 172.21.15.16:0/4104413164 shutdown_connections 2023-10-16T13:44:48.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.626+0000 7f4b50bf9700 1 -- 172.21.15.16:0/4104413164 wait complete. 2023-10-16T13:44:48.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.627+0000 7f4b50bf9700 1 Processor -- start 2023-10-16T13:44:48.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.627+0000 7f4b50bf9700 1 -- start start 2023-10-16T13:44:48.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.628+0000 7f4b50bf9700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4b4c06e170 0x7f4b4c154270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:48.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.628+0000 7f4b50bf9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b4c0cb880 0x7f4b4c0c6000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:48.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.628+0000 7f4b50bf9700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4b4c15b070 0x7f4b4c1532f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:48.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.628+0000 7f4b50bf9700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b4c142fd0 con 0x7f4b4c0cb880 2023-10-16T13:44:48.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.628+0000 7f4b50bf9700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b4c155f10 con 0x7f4b4c15b070 2023-10-16T13:44:48.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.628+0000 7f4b50bf9700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b4c1561f0 con 0x7f4b4c06e170 2023-10-16T13:44:48.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.628+0000 7f4b49d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b4c0cb880 0x7f4b4c0c6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:48.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.628+0000 7f4b49d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b4c0cb880 0x7f4b4c0c6000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51912/0 (socket says 172.21.15.16:51912) 2023-10-16T13:44:48.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.628+0000 7f4b49d9b700 1 -- 172.21.15.16:0/4115692149 learned_addr learned my addr 172.21.15.16:0/4115692149 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:48.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.628+0000 7f4b4ad9d700 1 --2- 172.21.15.16:0/4115692149 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4b4c15b070 0x7f4b4c1532f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:48.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.628+0000 7f4b4a59c700 1 --2- 172.21.15.16:0/4115692149 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4b4c06e170 0x7f4b4c154270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:48.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.628+0000 7f4b49d9b700 1 -- 172.21.15.16:0/4115692149 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4b4c06e170 msgr2=0x7f4b4c154270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:48.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.628+0000 7f4b49d9b700 1 --2- 172.21.15.16:0/4115692149 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4b4c06e170 0x7f4b4c154270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:48.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.628+0000 7f4b49d9b700 1 -- 172.21.15.16:0/4115692149 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4b4c15b070 msgr2=0x7f4b4c1532f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:48.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.628+0000 7f4b49d9b700 1 --2- 172.21.15.16:0/4115692149 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4b4c15b070 0x7f4b4c1532f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:48.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.628+0000 7f4b49d9b700 1 -- 172.21.15.16:0/4115692149 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b44009a70 con 0x7f4b4c0cb880 2023-10-16T13:44:48.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.629+0000 7f4b49d9b700 1 --2- 172.21.15.16:0/4115692149 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b4c0cb880 0x7f4b4c0c6000 secure :-1 s=READY pgs=2318 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c00c520 tx=0x7f4b3c00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:48.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.629+0000 7f4b3affd700 1 -- 172.21.15.16:0/4115692149 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b3c005d70 con 0x7f4b4c0cb880 2023-10-16T13:44:48.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.629+0000 7f4b3affd700 1 -- 172.21.15.16:0/4115692149 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b3c0070d0 con 0x7f4b4c0cb880 2023-10-16T13:44:48.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.629+0000 7f4b50bf9700 1 -- 172.21.15.16:0/4115692149 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b4c1564d0 con 0x7f4b4c0cb880 2023-10-16T13:44:48.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.629+0000 7f4b3affd700 1 -- 172.21.15.16:0/4115692149 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b3c007440 con 0x7f4b4c0cb880 2023-10-16T13:44:48.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.629+0000 7f4b50bf9700 1 -- 172.21.15.16:0/4115692149 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b4c155ba0 con 0x7f4b4c0cb880 2023-10-16T13:44:48.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.630+0000 7f4b50bf9700 1 -- 172.21.15.16:0/4115692149 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b4c143460 con 0x7f4b4c0cb880 2023-10-16T13:44:48.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.630+0000 7f4b3affd700 1 -- 172.21.15.16:0/4115692149 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 126) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4b3c0075a0 con 0x7f4b4c0cb880 2023-10-16T13:44:48.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.631+0000 7f4b3affd700 1 --2- 172.21.15.16:0/4115692149 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f4b2c06b020 0x7f4b2c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:48.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.631+0000 7f4b3affd700 1 -- 172.21.15.16:0/4115692149 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(206..206 src has 1..206) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f4b3c088b70 con 0x7f4b4c0cb880 2023-10-16T13:44:48.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.631+0000 7f4b4a59c700 1 --2- 172.21.15.16:0/4115692149 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f4b2c06b020 0x7f4b2c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:48.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.631+0000 7f4b4a59c700 1 --2- 172.21.15.16:0/4115692149 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f4b2c06b020 0x7f4b2c06d4e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4b4c153050 tx=0x7f4b40008040 comp rx=0 tx=0).ready entity=mgr.10665 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:48.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.636+0000 7f4b3affd700 1 -- 172.21.15.16:0/4115692149 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4b4c143460 con 0x7f4b4c0cb880 2023-10-16T13:44:48.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:48.907+0000 7f4b50bf9700 1 -- 172.21.15.16:0/4115692149 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f4b4c143460 con 0x7f4b4c0cb880 2023-10-16T13:44:49.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:49.874+0000 7f4b3affd700 1 -- 172.21.15.16:0/4115692149 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v127) v1 ==== 92+0+0 (secure 0 0 0) 0x7f4b4c143460 con 0x7f4b4c0cb880 2023-10-16T13:44:49.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:49.879+0000 7f4b3affd700 1 -- 172.21.15.16:0/4115692149 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mgrmap(e 127) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f4b3c086ed0 con 0x7f4b4c0cb880 2023-10-16T13:44:49.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:49.880+0000 7f4b50bf9700 1 -- 172.21.15.16:0/4115692149 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f4b2c06b020 msgr2=0x7f4b2c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:49.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:49.880+0000 7f4b50bf9700 1 --2- 172.21.15.16:0/4115692149 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f4b2c06b020 0x7f4b2c06d4e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4b4c153050 tx=0x7f4b40008040 comp rx=0 tx=0).stop 2023-10-16T13:44:49.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:49.880+0000 7f4b50bf9700 1 -- 172.21.15.16:0/4115692149 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b4c0cb880 msgr2=0x7f4b4c0c6000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:49.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:49.880+0000 7f4b50bf9700 1 --2- 172.21.15.16:0/4115692149 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b4c0cb880 0x7f4b4c0c6000 secure :-1 s=READY pgs=2318 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c00c520 tx=0x7f4b3c00c830 comp rx=0 tx=0).stop 2023-10-16T13:44:49.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:49.880+0000 7f4b50bf9700 1 -- 172.21.15.16:0/4115692149 shutdown_connections 2023-10-16T13:44:49.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:49.880+0000 7f4b50bf9700 1 --2- 172.21.15.16:0/4115692149 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f4b2c06b020 0x7f4b2c06d4e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:49.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:49.880+0000 7f4b50bf9700 1 --2- 172.21.15.16:0/4115692149 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4b4c15b070 0x7f4b4c1532f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:49.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:49.880+0000 7f4b50bf9700 1 --2- 172.21.15.16:0/4115692149 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4b4c06e170 0x7f4b4c154270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:49.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:49.880+0000 7f4b50bf9700 1 --2- 172.21.15.16:0/4115692149 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b4c0cb880 0x7f4b4c0c6000 unknown :-1 s=CLOSED pgs=2318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:49.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:49.880+0000 7f4b50bf9700 1 -- 172.21.15.16:0/4115692149 >> 172.21.15.16:0/4115692149 conn(0x7f4b4c0b1f40 msgr2=0x7f4b4c0cf300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:49.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:49.880+0000 7f4b50bf9700 1 -- 172.21.15.16:0/4115692149 shutdown_connections 2023-10-16T13:44:49.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:49.881+0000 7f4b50bf9700 1 -- 172.21.15.16:0/4115692149 wait complete. 2023-10-16T13:44:49.899 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-16T13:44:50.029 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:44:50.026+0000 7f7ff1695200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T13:44:50.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.035+0000 7fe946f23700 1 Processor -- start 2023-10-16T13:44:50.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.035+0000 7fe946f23700 1 -- start start 2023-10-16T13:44:50.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.036+0000 7fe946f23700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe94015da10 0x7fe9400d1520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:50.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.036+0000 7fe946f23700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fe9400d1d20 0x7fe9400d21b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:50.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.036+0000 7fe946f23700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9400d4540 con 0x7fe9400d4060 2023-10-16T13:44:50.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.036+0000 7fe946f23700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9400cf9b0 con 0x7fe94015da10 2023-10-16T13:44:50.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.036+0000 7fe946f23700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9400d26f0 con 0x7fe9400d1d20 2023-10-16T13:44:50.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.036+0000 7fe944cbf700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe94015da10 0x7fe9400d1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:50.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.036+0000 7fe944cbf700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe94015da10 0x7fe9400d1520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51944/0 (socket says 172.21.15.16:51944) 2023-10-16T13:44:50.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.036+0000 7fe944cbf700 1 -- 172.21.15.16:0/2579826033 learned_addr learned my addr 172.21.15.16:0/2579826033 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:50.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.036+0000 7fe9454c0700 1 --2- 172.21.15.16:0/2579826033 >> v2:172.21.15.49:3300/0 conn(0x7fe9400d1d20 0x7fe9400d21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:50.041 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:44:50.036+0000 7f188d506200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T13:44:50.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.036+0000 7fe944cbf700 1 -- 172.21.15.16:0/2579826033 >> v2:172.21.15.49:3300/0 conn(0x7fe9400d1d20 msgr2=0x7fe9400d21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:50.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.036+0000 7fe944cbf700 1 --2- 172.21.15.16:0/2579826033 >> v2:172.21.15.49:3300/0 conn(0x7fe9400d1d20 0x7fe9400d21b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:50.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.036+0000 7fe944cbf700 1 -- 172.21.15.16:0/2579826033 >> v1:172.21.15.49:6789/0 conn(0x7fe9400d4060 legacy=0x7fe9400cf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:50.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.036+0000 7fe944cbf700 1 -- 172.21.15.16:0/2579826033 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9400ce7e0 con 0x7fe94015da10 2023-10-16T13:44:50.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.037+0000 7fe944cbf700 1 --2- 172.21.15.16:0/2579826033 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe94015da10 0x7fe9400d1520 secure :-1 s=READY pgs=2322 cs=0 l=1 rev1=1 crypto rx=0x7fe9400d32a0 tx=0x7fe934009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91215cfe863d7d7a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:50.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.037+0000 7fe93f7fe700 1 -- 172.21.15.16:0/2579826033 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe934005e30 con 0x7fe94015da10 2023-10-16T13:44:50.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.037+0000 7fe93f7fe700 1 -- 172.21.15.16:0/2579826033 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe934004030 con 0x7fe94015da10 2023-10-16T13:44:50.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.037+0000 7fe93f7fe700 1 -- 172.21.15.16:0/2579826033 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9340043e0 con 0x7fe94015da10 2023-10-16T13:44:50.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.037+0000 7fe946f23700 1 -- 172.21.15.16:0/2579826033 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe94015da10 msgr2=0x7fe9400d1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:50.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.037+0000 7fe946f23700 1 --2- 172.21.15.16:0/2579826033 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe94015da10 0x7fe9400d1520 secure :-1 s=READY pgs=2322 cs=0 l=1 rev1=1 crypto rx=0x7fe9400d32a0 tx=0x7fe934009bd0 comp rx=0 tx=0).stop 2023-10-16T13:44:50.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.037+0000 7fe946f23700 1 -- 172.21.15.16:0/2579826033 shutdown_connections 2023-10-16T13:44:50.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.037+0000 7fe946f23700 1 --2- 172.21.15.16:0/2579826033 >> v2:172.21.15.49:3300/0 conn(0x7fe9400d1d20 0x7fe9400d21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:50.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.037+0000 7fe946f23700 1 --2- 172.21.15.16:0/2579826033 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe94015da10 0x7fe9400d1520 unknown :-1 s=CLOSED pgs=2322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:50.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.037+0000 7fe946f23700 1 -- 172.21.15.16:0/2579826033 >> 172.21.15.16:0/2579826033 conn(0x7fe94009c9f0 msgr2=0x7fe9400d3740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:50.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.038+0000 7fe946f23700 1 -- 172.21.15.16:0/2579826033 shutdown_connections 2023-10-16T13:44:50.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.038+0000 7fe946f23700 1 -- 172.21.15.16:0/2579826033 wait complete. 2023-10-16T13:44:50.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.039+0000 7fe946f23700 1 Processor -- start 2023-10-16T13:44:50.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.039+0000 7fe946f23700 1 -- start start 2023-10-16T13:44:50.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.040+0000 7fe946f23700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9400d1d20 0x7fe9400ccb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:50.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.040+0000 7fe946f23700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe9400d4060 0x7fe9401f3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:50.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.040+0000 7fe946f23700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe94015da10 0x7fe94014a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:50.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.040+0000 7fe946f23700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9400b8290 con 0x7fe9400d1d20 2023-10-16T13:44:50.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.040+0000 7fe946f23700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9400b7680 con 0x7fe94015da10 2023-10-16T13:44:50.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.040+0000 7fe946f23700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9400b7960 con 0x7fe9400d4060 2023-10-16T13:44:50.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.040+0000 7fe944cbf700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9400d1d20 0x7fe9400ccb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:50.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.040+0000 7fe944cbf700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9400d1d20 0x7fe9400ccb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51950/0 (socket says 172.21.15.16:51950) 2023-10-16T13:44:50.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.040+0000 7fe944cbf700 1 -- 172.21.15.16:0/678704058 learned_addr learned my addr 172.21.15.16:0/678704058 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:50.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.040+0000 7fe9454c0700 1 --2- 172.21.15.16:0/678704058 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe94015da10 0x7fe94014a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:50.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.040+0000 7fe93ffff700 1 --2- 172.21.15.16:0/678704058 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe9400d4060 0x7fe9401f3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:50.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.040+0000 7fe944cbf700 1 -- 172.21.15.16:0/678704058 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe9400d4060 msgr2=0x7fe9401f3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:50.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.040+0000 7fe944cbf700 1 --2- 172.21.15.16:0/678704058 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe9400d4060 0x7fe9401f3600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:50.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.040+0000 7fe944cbf700 1 -- 172.21.15.16:0/678704058 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe94015da10 msgr2=0x7fe94014a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:50.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.040+0000 7fe944cbf700 1 --2- 172.21.15.16:0/678704058 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe94015da10 0x7fe94014a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:50.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.040+0000 7fe944cbf700 1 -- 172.21.15.16:0/678704058 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe934009500 con 0x7fe9400d1d20 2023-10-16T13:44:50.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.040+0000 7fe944cbf700 1 --2- 172.21.15.16:0/678704058 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9400d1d20 0x7fe9400ccb20 secure :-1 s=READY pgs=2323 cs=0 l=1 rev1=1 crypto rx=0x7fe934009c10 tx=0x7fe934004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:50.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.041+0000 7fe93d7fa700 1 -- 172.21.15.16:0/678704058 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe934003c30 con 0x7fe9400d1d20 2023-10-16T13:44:50.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.041+0000 7fe93d7fa700 1 -- 172.21.15.16:0/678704058 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe934003d90 con 0x7fe9400d1d20 2023-10-16T13:44:50.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.041+0000 7fe946f23700 1 -- 172.21.15.16:0/678704058 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9400b7c40 con 0x7fe9400d1d20 2023-10-16T13:44:50.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.041+0000 7fe93d7fa700 1 -- 172.21.15.16:0/678704058 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe93401c640 con 0x7fe9400d1d20 2023-10-16T13:44:50.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.041+0000 7fe946f23700 1 -- 172.21.15.16:0/678704058 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe940158340 con 0x7fe9400d1d20 2023-10-16T13:44:50.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.042+0000 7fe946f23700 1 -- 172.21.15.16:0/678704058 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe94004ebb0 con 0x7fe9400d1d20 2023-10-16T13:44:50.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.042+0000 7fe93d7fa700 1 -- 172.21.15.16:0/678704058 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 127) v1 ==== 89348+0+0 (secure 0 0 0) 0x7fe93401c7a0 con 0x7fe9400d1d20 2023-10-16T13:44:50.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.043+0000 7fe93d7fa700 1 --2- 172.21.15.16:0/678704058 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7fe92406b070 0x7fe92406d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:50.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.043+0000 7fe93ffff700 1 -- 172.21.15.16:0/678704058 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7fe92406b070 msgr2=0x7fe92406d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1268439414 2023-10-16T13:44:50.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.043+0000 7fe93d7fa700 1 -- 172.21.15.16:0/678704058 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(206..206 src has 1..206) v4 ==== 6157+0+0 (secure 0 0 0) 0x7fe93408a8d0 con 0x7fe9400d1d20 2023-10-16T13:44:50.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.043+0000 7fe93ffff700 1 --2- 172.21.15.16:0/678704058 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7fe92406b070 0x7fe92406d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:44:50.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.051+0000 7fe93d7fa700 1 -- 172.21.15.16:0/678704058 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe9340556c0 con 0x7fe9400d1d20 2023-10-16T13:44:50.098 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:44:50.095+0000 7f7ff1695200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T13:44:50.113 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:44:50.111+0000 7f188d506200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T13:44:50.172 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:44:50.170+0000 7f7ff1695200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T13:44:50.188 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:44:50.186+0000 7f188d506200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T13:44:50.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.244+0000 7fe93ffff700 1 -- 172.21.15.16:0/678704058 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7fe92406b070 msgr2=0x7fe92406d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1268439414 2023-10-16T13:44:50.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.244+0000 7fe93ffff700 1 --2- 172.21.15.16:0/678704058 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7fe92406b070 0x7fe92406d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:44:50.253 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:44:50.251+0000 7f7ff1695200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T13:44:50.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.257+0000 7fe946f23700 1 -- 172.21.15.16:0/678704058 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7fe9400d1b00 con 0x7fe9400d1d20 2023-10-16T13:44:50.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.260+0000 7fe93d7fa700 1 -- 172.21.15.16:0/678704058 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v71)=0 v71) v1 ==== 138+0+0 (secure 0 0 0) 0x7fe934058ce0 con 0x7fe9400d1d20 2023-10-16T13:44:50.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.263+0000 7fe946f23700 1 -- 172.21.15.16:0/678704058 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7fe92406b070 msgr2=0x7fe92406d530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:44:50.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.263+0000 7fe946f23700 1 --2- 172.21.15.16:0/678704058 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7fe92406b070 0x7fe92406d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:50.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.263+0000 7fe946f23700 1 -- 172.21.15.16:0/678704058 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9400d1d20 msgr2=0x7fe9400ccb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:50.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.263+0000 7fe946f23700 1 --2- 172.21.15.16:0/678704058 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9400d1d20 0x7fe9400ccb20 secure :-1 s=READY pgs=2323 cs=0 l=1 rev1=1 crypto rx=0x7fe934009c10 tx=0x7fe934004750 comp rx=0 tx=0).stop 2023-10-16T13:44:50.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.263+0000 7fe946f23700 1 -- 172.21.15.16:0/678704058 shutdown_connections 2023-10-16T13:44:50.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.263+0000 7fe946f23700 1 --2- 172.21.15.16:0/678704058 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7fe92406b070 0x7fe92406d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:50.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.263+0000 7fe946f23700 1 --2- 172.21.15.16:0/678704058 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe94015da10 0x7fe94014a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:50.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.263+0000 7fe946f23700 1 --2- 172.21.15.16:0/678704058 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9400d1d20 0x7fe9400ccb20 unknown :-1 s=CLOSED pgs=2323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:50.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.263+0000 7fe946f23700 1 --2- 172.21.15.16:0/678704058 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe9400d4060 0x7fe9401f3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:50.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.263+0000 7fe946f23700 1 -- 172.21.15.16:0/678704058 >> 172.21.15.16:0/678704058 conn(0x7fe94009c9f0 msgr2=0x7fe9400cf1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:50.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.263+0000 7fe946f23700 1 -- 172.21.15.16:0/678704058 shutdown_connections 2023-10-16T13:44:50.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.263+0000 7fe946f23700 1 -- 172.21.15.16:0/678704058 wait complete. 2023-10-16T13:44:50.270 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:44:50.268+0000 7f188d506200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T13:44:50.278 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-16T13:44:50.401 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:44:50.399+0000 7f7ff1695200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T13:44:50.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.406+0000 7fd12ab1d700 1 Processor -- start 2023-10-16T13:44:50.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.406+0000 7fd12ab1d700 1 -- start start 2023-10-16T13:44:50.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.407+0000 7fd12ab1d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd12415ec80 0x7fd1240cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:50.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.407+0000 7fd12ab1d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd1240d2110 0x7fd1240cd1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:50.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.407+0000 7fd12ab1d700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1240b2440 con 0x7fd12415f060 2023-10-16T13:44:50.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.407+0000 7fd12ab1d700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1240b2c50 con 0x7fd12415ec80 2023-10-16T13:44:50.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.407+0000 7fd12ab1d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1240d00e0 con 0x7fd1240d2110 2023-10-16T13:44:50.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.407+0000 7fd1288b9700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd12415ec80 0x7fd1240cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:50.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.407+0000 7fd123fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd12415f060 0x7fd1240b4440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41252/0 (socket says 172.21.15.16:41252) 2023-10-16T13:44:50.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.407+0000 7fd123fff700 1 -- 172.21.15.16:0/4188641945 learned_addr learned my addr 172.21.15.16:0/4188641945 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:44:50.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.407+0000 7fd1290ba700 1 --2- 172.21.15.16:0/4188641945 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd1240d2110 0x7fd1240cd1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:50.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.407+0000 7fd1288b9700 1 -- 172.21.15.16:0/4188641945 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd1240d2110 msgr2=0x7fd1240cd1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:50.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.407+0000 7fd1288b9700 1 --2- 172.21.15.16:0/4188641945 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd1240d2110 0x7fd1240cd1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:50.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.407+0000 7fd1288b9700 1 -- 172.21.15.16:0/4188641945 >> v1:172.21.15.16:6789/0 conn(0x7fd12415f060 legacy=0x7fd1240b4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:50.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.407+0000 7fd1237fe700 1 -- 172.21.15.16:0/4188641945 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 160761992 0 0) 0x7fd1240b2440 con 0x7fd12415f060 2023-10-16T13:44:50.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.407+0000 7fd1288b9700 1 -- 172.21.15.16:0/4188641945 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1240cef60 con 0x7fd12415ec80 2023-10-16T13:44:50.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.408+0000 7fd1288b9700 1 --2- 172.21.15.16:0/4188641945 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd12415ec80 0x7fd1240cdec0 secure :-1 s=READY pgs=2325 cs=0 l=1 rev1=1 crypto rx=0x7fd118009840 tx=0x7fd118009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc7b4d8db45735a3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:50.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.408+0000 7fd1237fe700 1 -- 172.21.15.16:0/4188641945 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd118011830 con 0x7fd12415ec80 2023-10-16T13:44:50.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.408+0000 7fd1237fe700 1 -- 172.21.15.16:0/4188641945 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd118011990 con 0x7fd12415ec80 2023-10-16T13:44:50.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.408+0000 7fd1237fe700 1 -- 172.21.15.16:0/4188641945 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd118011d20 con 0x7fd12415ec80 2023-10-16T13:44:50.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.408+0000 7fd12ab1d700 1 -- 172.21.15.16:0/4188641945 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd12415ec80 msgr2=0x7fd1240cdec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:50.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.408+0000 7fd12ab1d700 1 --2- 172.21.15.16:0/4188641945 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd12415ec80 0x7fd1240cdec0 secure :-1 s=READY pgs=2325 cs=0 l=1 rev1=1 crypto rx=0x7fd118009840 tx=0x7fd118009b50 comp rx=0 tx=0).stop 2023-10-16T13:44:50.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.408+0000 7fd12ab1d700 1 -- 172.21.15.16:0/4188641945 shutdown_connections 2023-10-16T13:44:50.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.408+0000 7fd12ab1d700 1 --2- 172.21.15.16:0/4188641945 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd1240d2110 0x7fd1240cd1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:50.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.408+0000 7fd12ab1d700 1 --2- 172.21.15.16:0/4188641945 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd12415ec80 0x7fd1240cdec0 unknown :-1 s=CLOSED pgs=2325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:50.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.408+0000 7fd12ab1d700 1 -- 172.21.15.16:0/4188641945 >> 172.21.15.16:0/4188641945 conn(0x7fd1240d15f0 msgr2=0x7fd1240d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:50.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.408+0000 7fd12ab1d700 1 -- 172.21.15.16:0/4188641945 shutdown_connections 2023-10-16T13:44:50.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.408+0000 7fd12ab1d700 1 -- 172.21.15.16:0/4188641945 wait complete. 2023-10-16T13:44:50.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.410+0000 7fd12ab1d700 1 Processor -- start 2023-10-16T13:44:50.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.410+0000 7fd12ab1d700 1 -- start start 2023-10-16T13:44:50.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.410+0000 7fd12ab1d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1240d2110 0x7fd124150370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:50.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.410+0000 7fd12ab1d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd12415ec80 0x7fd12406c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:50.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.410+0000 7fd12ab1d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd12415f060 0x7fd1240c94e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:50.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.411+0000 7fd12ab1d700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd12406d2a0 con 0x7fd1240d2110 2023-10-16T13:44:50.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.411+0000 7fd12ab1d700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd12415aef0 con 0x7fd12415f060 2023-10-16T13:44:50.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.411+0000 7fd12ab1d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd12406cce0 con 0x7fd12415ec80 2023-10-16T13:44:50.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.411+0000 7fd1288b9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1240d2110 0x7fd124150370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:50.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.411+0000 7fd1288b9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1240d2110 0x7fd124150370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51984/0 (socket says 172.21.15.16:51984) 2023-10-16T13:44:50.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.411+0000 7fd1288b9700 1 -- 172.21.15.16:0/2409730900 learned_addr learned my addr 172.21.15.16:0/2409730900 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:50.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.411+0000 7fd123fff700 1 --2- 172.21.15.16:0/2409730900 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd12415ec80 0x7fd12406c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:50.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.411+0000 7fd1290ba700 1 --2- 172.21.15.16:0/2409730900 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd12415f060 0x7fd1240c94e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:50.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.411+0000 7fd1288b9700 1 -- 172.21.15.16:0/2409730900 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd12415ec80 msgr2=0x7fd12406c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:50.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.411+0000 7fd1288b9700 1 --2- 172.21.15.16:0/2409730900 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd12415ec80 0x7fd12406c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:50.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.411+0000 7fd1288b9700 1 -- 172.21.15.16:0/2409730900 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd12415f060 msgr2=0x7fd1240c94e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:50.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.411+0000 7fd1288b9700 1 --2- 172.21.15.16:0/2409730900 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd12415f060 0x7fd1240c94e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:50.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.411+0000 7fd1288b9700 1 -- 172.21.15.16:0/2409730900 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1180094a0 con 0x7fd1240d2110 2023-10-16T13:44:50.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.411+0000 7fd1288b9700 1 --2- 172.21.15.16:0/2409730900 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1240d2110 0x7fd124150370 secure :-1 s=READY pgs=2326 cs=0 l=1 rev1=1 crypto rx=0x7fd118009fd0 tx=0x7fd118004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:50.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.411+0000 7fd1217fa700 1 -- 172.21.15.16:0/2409730900 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1180036a0 con 0x7fd1240d2110 2023-10-16T13:44:50.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.411+0000 7fd1217fa700 1 -- 172.21.15.16:0/2409730900 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd11801f070 con 0x7fd1240d2110 2023-10-16T13:44:50.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.411+0000 7fd1217fa700 1 -- 172.21.15.16:0/2409730900 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd118003a60 con 0x7fd1240d2110 2023-10-16T13:44:50.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.411+0000 7fd12ab1d700 1 -- 172.21.15.16:0/2409730900 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd12406cfc0 con 0x7fd1240d2110 2023-10-16T13:44:50.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.411+0000 7fd12ab1d700 1 -- 172.21.15.16:0/2409730900 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1241508b0 con 0x7fd1240d2110 2023-10-16T13:44:50.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.412+0000 7fd12ab1d700 1 -- 172.21.15.16:0/2409730900 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd12415a9e0 con 0x7fd1240d2110 2023-10-16T13:44:50.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.413+0000 7fd1217fa700 1 -- 172.21.15.16:0/2409730900 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 127) v1 ==== 89348+0+0 (secure 0 0 0) 0x7fd118003bc0 con 0x7fd1240d2110 2023-10-16T13:44:50.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.413+0000 7fd1217fa700 1 --2- 172.21.15.16:0/2409730900 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7fd10806b070 0x7fd10806d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:50.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.413+0000 7fd123fff700 1 -- 172.21.15.16:0/2409730900 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7fd10806b070 msgr2=0x7fd10806d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1268439414 2023-10-16T13:44:50.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.414+0000 7fd1217fa700 1 -- 172.21.15.16:0/2409730900 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(206..206 src has 1..206) v4 ==== 6157+0+0 (secure 0 0 0) 0x7fd118021070 con 0x7fd1240d2110 2023-10-16T13:44:50.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.414+0000 7fd123fff700 1 --2- 172.21.15.16:0/2409730900 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7fd10806b070 0x7fd10806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:44:50.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.418+0000 7fd1217fa700 1 -- 172.21.15.16:0/2409730900 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd1180bdfe0 con 0x7fd1240d2110 2023-10-16T13:44:50.425 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:44:50.421+0000 7f188d506200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T13:44:50.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.615+0000 7fd123fff700 1 -- 172.21.15.16:0/2409730900 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7fd10806b070 msgr2=0x7fd10806d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1268439414 2023-10-16T13:44:50.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.615+0000 7fd123fff700 1 --2- 172.21.15.16:0/2409730900 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7fd10806b070 0x7fd10806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:44:50.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.626+0000 7fd12ab1d700 1 -- 172.21.15.16:0/2409730900 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fd1240b47a0 con 0x7fd1240d2110 2023-10-16T13:44:50.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.629+0000 7fd1217fa700 1 -- 172.21.15.16:0/2409730900 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v72)=0 v72) v1 ==== 133+0+0 (secure 0 0 0) 0x7fd12415a9e0 con 0x7fd1240d2110 2023-10-16T13:44:50.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.631+0000 7fd12ab1d700 1 -- 172.21.15.16:0/2409730900 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7fd10806b070 msgr2=0x7fd10806d530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:44:50.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.631+0000 7fd12ab1d700 1 --2- 172.21.15.16:0/2409730900 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7fd10806b070 0x7fd10806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:50.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.631+0000 7fd12ab1d700 1 -- 172.21.15.16:0/2409730900 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1240d2110 msgr2=0x7fd124150370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:50.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.631+0000 7fd12ab1d700 1 --2- 172.21.15.16:0/2409730900 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1240d2110 0x7fd124150370 secure :-1 s=READY pgs=2326 cs=0 l=1 rev1=1 crypto rx=0x7fd118009fd0 tx=0x7fd118004750 comp rx=0 tx=0).stop 2023-10-16T13:44:50.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.631+0000 7fd12ab1d700 1 -- 172.21.15.16:0/2409730900 shutdown_connections 2023-10-16T13:44:50.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.631+0000 7fd12ab1d700 1 --2- 172.21.15.16:0/2409730900 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7fd10806b070 0x7fd10806d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:50.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.631+0000 7fd12ab1d700 1 --2- 172.21.15.16:0/2409730900 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd12415f060 0x7fd1240c94e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:50.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.631+0000 7fd12ab1d700 1 --2- 172.21.15.16:0/2409730900 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1240d2110 0x7fd124150370 unknown :-1 s=CLOSED pgs=2326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:50.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.631+0000 7fd12ab1d700 1 --2- 172.21.15.16:0/2409730900 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd12415ec80 0x7fd12406c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:50.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.631+0000 7fd12ab1d700 1 -- 172.21.15.16:0/2409730900 >> 172.21.15.16:0/2409730900 conn(0x7fd1240d15f0 msgr2=0x7fd1240d19d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:50.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.631+0000 7fd12ab1d700 1 -- 172.21.15.16:0/2409730900 shutdown_connections 2023-10-16T13:44:50.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.631+0000 7fd12ab1d700 1 -- 172.21.15.16:0/2409730900 wait complete. 2023-10-16T13:44:50.646 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:44:50.647 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-10-16T13:44:50.669 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1m 2023-10-16T13:44:50.745 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:44:50.742+0000 7f7ff1695200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T13:44:50.771 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:44:50.768+0000 7f188d506200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T13:44:50.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.837+0000 7f2fc5696700 1 Processor -- start 2023-10-16T13:44:50.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.837+0000 7f2fc5696700 1 -- start start 2023-10-16T13:44:50.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.838+0000 7f2fc5696700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f2fc015b2c0 0x7f2fc0161340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:50.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.838+0000 7f2fc5696700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2fc006e570 0x7f2fc015e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:50.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.838+0000 7f2fc5696700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fc0161880 con 0x7f2fc006a7e0 2023-10-16T13:44:50.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.838+0000 7f2fc5696700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fc015b6a0 con 0x7f2fc015b2c0 2023-10-16T13:44:50.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.838+0000 7f2fc5696700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc0161b20 con 0x7f2fc006e570 2023-10-16T13:44:50.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.838+0000 7f2fbe7fc700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2fc006e570 0x7f2fc015e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:50.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.838+0000 7f2fbeffd700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f2fc015b2c0 0x7f2fc0161340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:50.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.838+0000 7f2fbf7fe700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7f2fc006a7e0 0x7f2fc0071a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:48470/0 (socket says 172.21.15.16:48470) 2023-10-16T13:44:50.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.838+0000 7f2fbe7fc700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2fc006e570 0x7f2fc015e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:40270/0 (socket says 172.21.15.16:40270) 2023-10-16T13:44:50.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.838+0000 7f2fbeffd700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f2fc015b2c0 0x7f2fc0161340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:51072/0 (socket says 172.21.15.16:51072) 2023-10-16T13:44:50.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.838+0000 7f2fbe7fc700 1 -- 172.21.15.16:0/2072287633 learned_addr learned my addr 172.21.15.16:0/2072287633 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:50.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.838+0000 7f2fbe7fc700 1 -- 172.21.15.16:0/2072287633 >> v2:172.21.15.49:3300/0 conn(0x7f2fc015b2c0 msgr2=0x7f2fc0161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:50.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.838+0000 7f2fbe7fc700 1 --2- 172.21.15.16:0/2072287633 >> v2:172.21.15.49:3300/0 conn(0x7f2fc015b2c0 0x7f2fc0161340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:50.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.838+0000 7f2fbe7fc700 1 -- 172.21.15.16:0/2072287633 >> v1:172.21.15.49:6789/0 conn(0x7f2fc006a7e0 legacy=0x7f2fc0071a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:50.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.838+0000 7f2fbe7fc700 1 -- 172.21.15.16:0/2072287633 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc0161e00 con 0x7f2fc006e570 2023-10-16T13:44:50.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.838+0000 7f2fbeffd700 1 --2- 172.21.15.16:0/2072287633 >> v2:172.21.15.49:3300/0 conn(0x7f2fc015b2c0 0x7f2fc0161340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:44:50.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.838+0000 7f2fbe7fc700 1 --2- 172.21.15.16:0/2072287633 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2fc006e570 0x7f2fc015e5a0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f2fb40098f0 tx=0x7f2fb4009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c016cf7832c5b636 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:50.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.839+0000 7f2fbdffb700 1 -- 172.21.15.16:0/2072287633 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb4011830 con 0x7f2fc006e570 2023-10-16T13:44:50.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.839+0000 7f2fbdffb700 1 -- 172.21.15.16:0/2072287633 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fb4011990 con 0x7f2fc006e570 2023-10-16T13:44:50.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.839+0000 7f2fbdffb700 1 -- 172.21.15.16:0/2072287633 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb4011d20 con 0x7f2fc006e570 2023-10-16T13:44:50.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.840+0000 7f2fc5696700 1 -- 172.21.15.16:0/2072287633 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2fc006e570 msgr2=0x7f2fc015e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:50.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.840+0000 7f2fc5696700 1 --2- 172.21.15.16:0/2072287633 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2fc006e570 0x7f2fc015e5a0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f2fb40098f0 tx=0x7f2fb4009c00 comp rx=0 tx=0).stop 2023-10-16T13:44:50.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.840+0000 7f2fc5696700 1 -- 172.21.15.16:0/2072287633 shutdown_connections 2023-10-16T13:44:50.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.840+0000 7f2fc5696700 1 --2- 172.21.15.16:0/2072287633 >> v2:172.21.15.49:3300/0 conn(0x7f2fc015b2c0 0x7f2fc0161340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:50.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.840+0000 7f2fc5696700 1 --2- 172.21.15.16:0/2072287633 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2fc006e570 0x7f2fc015e5a0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:50.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.840+0000 7f2fc5696700 1 -- 172.21.15.16:0/2072287633 >> 172.21.15.16:0/2072287633 conn(0x7f2fc00b1f40 msgr2=0x7f2fc00b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:50.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.840+0000 7f2fc5696700 1 -- 172.21.15.16:0/2072287633 shutdown_connections 2023-10-16T13:44:50.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.840+0000 7f2fc5696700 1 -- 172.21.15.16:0/2072287633 wait complete. 2023-10-16T13:44:50.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.842+0000 7f2fc5696700 1 Processor -- start 2023-10-16T13:44:50.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.842+0000 7f2fc5696700 1 -- start start 2023-10-16T13:44:50.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.843+0000 7f2fc5696700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2fc006a7e0 0x7f2fc014c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:50.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.843+0000 7f2fc5696700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2fc006e570 0x7f2fc00c8510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:50.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.843+0000 7f2fc5696700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2fc015b2c0 0x7f2fc00b5950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:50.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.843+0000 7f2fc5696700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc00c9900 con 0x7f2fc015b2c0 2023-10-16T13:44:50.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.843+0000 7f2fc5696700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc00c8a50 con 0x7f2fc006a7e0 2023-10-16T13:44:50.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.843+0000 7f2fc5696700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc014a590 con 0x7f2fc006e570 2023-10-16T13:44:50.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.843+0000 7f2fbf7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2fc015b2c0 0x7f2fc00b5950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:50.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.843+0000 7f2fbeffd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2fc006a7e0 0x7f2fc014c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:50.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.843+0000 7f2fbf7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2fc015b2c0 0x7f2fc00b5950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52020/0 (socket says 172.21.15.16:52020) 2023-10-16T13:44:50.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.843+0000 7f2fbf7fe700 1 -- 172.21.15.16:0/3776374669 learned_addr learned my addr 172.21.15.16:0/3776374669 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:50.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.843+0000 7f2fbe7fc700 1 --2- 172.21.15.16:0/3776374669 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2fc006e570 0x7f2fc00c8510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:50.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.844+0000 7f2fbf7fe700 1 -- 172.21.15.16:0/3776374669 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2fc006e570 msgr2=0x7f2fc00c8510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:50.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.844+0000 7f2fbf7fe700 1 --2- 172.21.15.16:0/3776374669 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2fc006e570 0x7f2fc00c8510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:50.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.844+0000 7f2fbf7fe700 1 -- 172.21.15.16:0/3776374669 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2fc006a7e0 msgr2=0x7f2fc014c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:50.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.844+0000 7f2fbf7fe700 1 --2- 172.21.15.16:0/3776374669 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2fc006a7e0 0x7f2fc014c1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:50.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.844+0000 7f2fbf7fe700 1 -- 172.21.15.16:0/3776374669 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fb00094a0 con 0x7f2fc015b2c0 2023-10-16T13:44:50.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.844+0000 7f2fbe7fc700 1 --2- 172.21.15.16:0/3776374669 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2fc006e570 0x7f2fc00c8510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:44:50.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.844+0000 7f2fbeffd700 1 --2- 172.21.15.16:0/3776374669 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2fc006a7e0 0x7f2fc014c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:44:50.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.844+0000 7f2fbf7fe700 1 --2- 172.21.15.16:0/3776374669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2fc015b2c0 0x7f2fc00b5950 secure :-1 s=READY pgs=2327 cs=0 l=1 rev1=1 crypto rx=0x7f2fb800cbf0 tx=0x7f2fb800cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:50.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.844+0000 7f2fa77fe700 1 -- 172.21.15.16:0/3776374669 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb800aa60 con 0x7f2fc015b2c0 2023-10-16T13:44:50.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.844+0000 7f2fa77fe700 1 -- 172.21.15.16:0/3776374669 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fb800abc0 con 0x7f2fc015b2c0 2023-10-16T13:44:50.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.844+0000 7f2fc5696700 1 -- 172.21.15.16:0/3776374669 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fb4009550 con 0x7f2fc015b2c0 2023-10-16T13:44:50.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.844+0000 7f2fa77fe700 1 -- 172.21.15.16:0/3776374669 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb8007070 con 0x7f2fc015b2c0 2023-10-16T13:44:50.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.844+0000 7f2fc5696700 1 -- 172.21.15.16:0/3776374669 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fc00c8f60 con 0x7f2fc015b2c0 2023-10-16T13:44:50.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.846+0000 7f2fa77fe700 1 -- 172.21.15.16:0/3776374669 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 127) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f2fb80071d0 con 0x7f2fc015b2c0 2023-10-16T13:44:50.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.846+0000 7f2fc5696700 1 -- 172.21.15.16:0/3776374669 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fc00c9a40 con 0x7f2fc015b2c0 2023-10-16T13:44:50.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.846+0000 7f2fa77fe700 1 --2- 172.21.15.16:0/3776374669 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f2fac06b070 0x7f2fac06d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:50.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.846+0000 7f2fbeffd700 1 -- 172.21.15.16:0/3776374669 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f2fac06b070 msgr2=0x7f2fac06d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1268439414 2023-10-16T13:44:50.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.846+0000 7f2fa77fe700 1 -- 172.21.15.16:0/3776374669 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(206..206 src has 1..206) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f2fb8089410 con 0x7f2fc015b2c0 2023-10-16T13:44:50.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.846+0000 7f2fbeffd700 1 --2- 172.21.15.16:0/3776374669 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f2fac06b070 0x7f2fac06d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:44:50.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:50.851+0000 7f2fa77fe700 1 -- 172.21.15.16:0/3776374669 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2fc00c9a40 con 0x7f2fc015b2c0 2023-10-16T13:44:50.864 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:44:50.862+0000 7f7ff1695200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T13:44:50.895 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:44:50.893+0000 7f188d506200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T13:44:50.941 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:44:50.939+0000 7f7ff1695200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T13:44:50.972 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:44:50.970+0000 7f188d506200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T13:44:51.025 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:44:51.022+0000 7f7ff1695200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T13:44:51.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:51.047+0000 7f2fbeffd700 1 -- 172.21.15.16:0/3776374669 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f2fac06b070 msgr2=0x7f2fac06d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1268439414 2023-10-16T13:44:51.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:51.047+0000 7f2fbeffd700 1 --2- 172.21.15.16:0/3776374669 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f2fac06b070 0x7f2fac06d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:44:51.057 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:44:51.055+0000 7f188d506200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T13:44:51.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:51.084+0000 7f2fc5696700 1 -- 172.21.15.16:0/3776374669 --> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f2fc015e080 con 0x7f2fac06b070 2023-10-16T13:44:51.411 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:44:51.408+0000 7f7ff1695200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T13:44:51.445 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:44:51.443+0000 7f188d506200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T13:44:51.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:51.448+0000 7f2fbeffd700 1 -- 172.21.15.16:0/3776374669 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f2fac06b070 msgr2=0x7f2fac06d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1268439414 2023-10-16T13:44:51.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:51.448+0000 7f2fbeffd700 1 --2- 172.21.15.16:0/3776374669 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f2fac06b070 0x7f2fac06d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-16T13:44:52.017 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:44:52.014+0000 7f7ff1695200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T13:44:52.054 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:44:52.052+0000 7f188d506200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T13:44:52.089 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:44:52.087+0000 7f7ff1695200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T13:44:52.127 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:44:52.125+0000 7f188d506200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T13:44:52.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:52.249+0000 7f2fbeffd700 1 -- 172.21.15.16:0/3776374669 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f2fac06b070 msgr2=0x7f2fac06d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1268439414 2023-10-16T13:44:52.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:52.249+0000 7f2fbeffd700 1 --2- 172.21.15.16:0/3776374669 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f2fac06b070 0x7f2fac06d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-16T13:44:52.752 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:44:52.749+0000 7f7ff1695200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:44:52.794 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:44:52.792+0000 7f188d506200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:44:52.833 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:44:52.830+0000 7f7ff1695200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T13:44:52.876 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:44:52.874+0000 7f188d506200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T13:44:52.983 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:44:52.981+0000 7f7ff1695200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:44:53.028 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:44:53.026+0000 7f188d506200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:44:53.219 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:44:53.217+0000 7f7ff1695200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T13:44:53.270 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:44:53.268+0000 7f188d506200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T13:44:53.297 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:44:53.294+0000 7f7ff1695200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T13:44:53.348 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:44:53.346+0000 7f188d506200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T13:44:53.573 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:44:53.570+0000 7f7ff1695200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T13:44:53.627 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:44:53.625+0000 7f188d506200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T13:44:53.805 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:44:53.802+0000 7f7ff1695200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T13:44:53.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:53.850+0000 7f2fbeffd700 1 -- 172.21.15.16:0/3776374669 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f2fac06b070 msgr2=0x7f2fac06d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1268439414 2023-10-16T13:44:53.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:53.850+0000 7f2fbeffd700 1 --2- 172.21.15.16:0/3776374669 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f2fac06b070 0x7f2fac06d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-10-16T13:44:53.861 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:44:53.859+0000 7f188d506200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T13:44:53.960 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:44:53.957+0000 7f7ff1695200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T13:44:54.019 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:44:54.017+0000 7f188d506200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T13:44:54.078 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:44:54.075+0000 7f7ff1695200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T13:44:54.141 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:44:54.138+0000 7f188d506200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T13:44:54.239 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:44:54.236+0000 7f7ff1695200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T13:44:54.306 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:44:54.304+0000 7f188d506200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T13:44:54.510 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:44:54.507+0000 7f7ff1695200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T13:44:54.581 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:44:54.579+0000 7f188d506200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T13:44:54.676 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:44:54.674+0000 7f7ff1695200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T13:44:54.751 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:44:54.749+0000 7f188d506200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T13:44:55.228 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:44:55.226+0000 7f7ff1695200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T13:44:55.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:55.296+0000 7f2fa77fe700 1 -- 172.21.15.16:0/3776374669 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 128) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f2fb8054700 con 0x7f2fc015b2c0 2023-10-16T13:44:55.315 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:44:55.312+0000 7f188d506200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T13:44:55.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:55.332+0000 7f2fa77fe700 1 -- 172.21.15.16:0/3776374669 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mgrmap(e 129) v1 ==== 88957+0+0 (secure 0 0 0) 0x7f2fb8087740 con 0x7f2fc015b2c0 2023-10-16T13:44:55.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:55.332+0000 7f2fa77fe700 1 -- 172.21.15.16:0/3776374669 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f2fac06b070 msgr2=0x7f2fac06d530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:44:55.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:55.332+0000 7f2fa77fe700 1 --2- 172.21.15.16:0/3776374669 >> [v2:172.21.15.16:6832/1268439414,v1:172.21.15.16:6833/1268439414] conn(0x7f2fac06b070 0x7f2fac06d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:55.476 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:44:55.473+0000 7f18034e7700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:44:55.476 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:44:55.473+0000 7f18034e7700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:44:55.476 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:44:55.473+0000 7f18034e7700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:44:55.477 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:44:55.473+0000 7f18034e7700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:44:55.477 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:44:55.473+0000 7f18034e7700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:44:55.477 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:44:55.473+0000 7f18004e1700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:44:55.477 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:44:55.473+0000 7f18004e1700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:44:55.477 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:44:55.473+0000 7f18004e1700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:44:55.477 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:44:55.473+0000 7f18004e1700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:44:55.478 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:44:55.473+0000 7f18004e1700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:44:56.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.336+0000 7f2fa77fe700 1 -- 172.21.15.16:0/3776374669 <== mon.0 v2:172.21.15.16:3300/0 9 ==== mgrmap(e 130) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f2fb8054fd0 con 0x7f2fc015b2c0 2023-10-16T13:44:56.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.337+0000 7f2fa77fe700 1 --2- 172.21.15.16:0/3776374669 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f2fac01d250 0x7f2fac075610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:56.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.337+0000 7f2fa77fe700 1 -- 172.21.15.16:0/3776374669 --> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f2fac01d630 con 0x7f2fac01d250 2023-10-16T13:44:56.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.339+0000 7f2fbeffd700 1 --2- 172.21.15.16:0/3776374669 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f2fac01d250 0x7f2fac075610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:56.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.341+0000 7f2fbeffd700 1 --2- 172.21.15.16:0/3776374669 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f2fac01d250 0x7f2fac075610 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f2fb0009c60 tx=0x7f2fb0009210 comp rx=0 tx=0).ready entity=mgr.11088 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:56.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.401+0000 7f2fa77fe700 1 -- 172.21.15.16:0/3776374669 <== mgr.11088 v2:172.21.15.16:6832/3220139196 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f2fac01d630 con 0x7f2fac01d250 2023-10-16T13:44:56.403 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule set for path snap_test_dir1 2023-10-16T13:44:56.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.403+0000 7f2fc5696700 1 -- 172.21.15.16:0/3776374669 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f2fac01d250 msgr2=0x7f2fac075610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:56.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.403+0000 7f2fc5696700 1 --2- 172.21.15.16:0/3776374669 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f2fac01d250 0x7f2fac075610 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f2fb0009c60 tx=0x7f2fb0009210 comp rx=0 tx=0).stop 2023-10-16T13:44:56.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.403+0000 7f2fc5696700 1 -- 172.21.15.16:0/3776374669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2fc015b2c0 msgr2=0x7f2fc00b5950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:56.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.403+0000 7f2fc5696700 1 --2- 172.21.15.16:0/3776374669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2fc015b2c0 0x7f2fc00b5950 secure :-1 s=READY pgs=2327 cs=0 l=1 rev1=1 crypto rx=0x7f2fb800cbf0 tx=0x7f2fb800cf00 comp rx=0 tx=0).stop 2023-10-16T13:44:56.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.403+0000 7f2fbf7fe700 1 -- 172.21.15.16:0/3776374669 reap_dead start 2023-10-16T13:44:56.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.403+0000 7f2fc5696700 1 -- 172.21.15.16:0/3776374669 shutdown_connections 2023-10-16T13:44:56.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.403+0000 7f2fc5696700 1 -- 172.21.15.16:0/3776374669 >> 172.21.15.16:0/3776374669 conn(0x7f2fc00b1f40 msgr2=0x7f2fc00cc150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:56.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.403+0000 7f2fc5696700 1 -- 172.21.15.16:0/3776374669 shutdown_connections 2023-10-16T13:44:56.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.404+0000 7f2fc5696700 1 -- 172.21.15.16:0/3776374669 wait complete. 2023-10-16T13:44:56.419 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2023-10-16T13:44:56.420 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention add snap_test_dir1 1m 2023-10-16T13:44:56.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.549+0000 7f64e6017700 1 Processor -- start 2023-10-16T13:44:56.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.549+0000 7f64e6017700 1 -- start start 2023-10-16T13:44:56.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.550+0000 7f64e6017700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f64e015a3d0 0x7f64e006d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:56.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.550+0000 7f64e6017700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f64e015e0b0 0x7f64e00ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:56.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.550+0000 7f64e6017700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64e00cb870 con 0x7f64e00cf430 2023-10-16T13:44:56.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.550+0000 7f64e6017700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64e015f0c0 con 0x7f64e015a3d0 2023-10-16T13:44:56.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.550+0000 7f64e6017700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64e0072990 con 0x7f64e015e0b0 2023-10-16T13:44:56.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.550+0000 7f64df7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f64e015a3d0 0x7f64e006d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:56.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.550+0000 7f64df7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f64e015a3d0 0x7f64e006d910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52188/0 (socket says 172.21.15.16:52188) 2023-10-16T13:44:56.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.550+0000 7f64df7fe700 1 -- 172.21.15.16:0/2292780490 learned_addr learned my addr 172.21.15.16:0/2292780490 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:56.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.550+0000 7f64deffd700 1 --2- 172.21.15.16:0/2292780490 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f64e015e0b0 0x7f64e00ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:56.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.550+0000 7f64df7fe700 1 -- 172.21.15.16:0/2292780490 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f64e015e0b0 msgr2=0x7f64e00ceef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:56.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.550+0000 7f64df7fe700 1 --2- 172.21.15.16:0/2292780490 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f64e015e0b0 0x7f64e00ceef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:56.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.550+0000 7f64df7fe700 1 -- 172.21.15.16:0/2292780490 >> v1:172.21.15.16:6789/0 conn(0x7f64e00cf430 legacy=0x7f64e00cdeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:56.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.550+0000 7f64df7fe700 1 -- 172.21.15.16:0/2292780490 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64e006e040 con 0x7f64e015a3d0 2023-10-16T13:44:56.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.551+0000 7f64df7fe700 1 --2- 172.21.15.16:0/2292780490 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64e015a3d0 0x7f64e006d910 secure :-1 s=READY pgs=2339 cs=0 l=1 rev1=1 crypto rx=0x7f64c80098f0 tx=0x7f64c8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0b1f24e8271eecf server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:56.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.551+0000 7f64de7fc700 1 -- 172.21.15.16:0/2292780490 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64c800b8a0 con 0x7f64e015a3d0 2023-10-16T13:44:56.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.551+0000 7f64de7fc700 1 -- 172.21.15.16:0/2292780490 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64c800ba00 con 0x7f64e015a3d0 2023-10-16T13:44:56.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.551+0000 7f64de7fc700 1 -- 172.21.15.16:0/2292780490 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64c800bd90 con 0x7f64e015a3d0 2023-10-16T13:44:56.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.551+0000 7f64e6017700 1 -- 172.21.15.16:0/2292780490 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64e015a3d0 msgr2=0x7f64e006d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:56.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.551+0000 7f64e6017700 1 --2- 172.21.15.16:0/2292780490 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64e015a3d0 0x7f64e006d910 secure :-1 s=READY pgs=2339 cs=0 l=1 rev1=1 crypto rx=0x7f64c80098f0 tx=0x7f64c8009c00 comp rx=0 tx=0).stop 2023-10-16T13:44:56.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.551+0000 7f64e6017700 1 -- 172.21.15.16:0/2292780490 shutdown_connections 2023-10-16T13:44:56.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.551+0000 7f64e6017700 1 --2- 172.21.15.16:0/2292780490 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64e015a3d0 0x7f64e006d910 unknown :-1 s=CLOSED pgs=2339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:56.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.551+0000 7f64e6017700 1 --2- 172.21.15.16:0/2292780490 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f64e015e0b0 0x7f64e00ceef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:56.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.551+0000 7f64e6017700 1 -- 172.21.15.16:0/2292780490 >> 172.21.15.16:0/2292780490 conn(0x7f64e006f430 msgr2=0x7f64e006f810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:56.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.551+0000 7f64e6017700 1 -- 172.21.15.16:0/2292780490 shutdown_connections 2023-10-16T13:44:56.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.551+0000 7f64e6017700 1 -- 172.21.15.16:0/2292780490 wait complete. 2023-10-16T13:44:56.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.553+0000 7f64e6017700 1 Processor -- start 2023-10-16T13:44:56.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.553+0000 7f64e6017700 1 -- start start 2023-10-16T13:44:56.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.553+0000 7f64e6017700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64e00cf430 0x7f64e0146e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:56.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.553+0000 7f64e6017700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f64e015a3d0 0x7f64e00cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:56.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.553+0000 7f64e6017700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f64e015e0b0 0x7f64e00cc240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:56.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.553+0000 7f64e6017700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64e01475f0 con 0x7f64e00cf430 2023-10-16T13:44:56.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.553+0000 7f64e6017700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64e01478d0 con 0x7f64e015a3d0 2023-10-16T13:44:56.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.553+0000 7f64e6017700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64e0147340 con 0x7f64e015e0b0 2023-10-16T13:44:56.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.554+0000 7f64df7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64e00cf430 0x7f64e0146e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:56.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.554+0000 7f64df7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64e00cf430 0x7f64e0146e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52194/0 (socket says 172.21.15.16:52194) 2023-10-16T13:44:56.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.554+0000 7f64df7fe700 1 -- 172.21.15.16:0/3842664446 learned_addr learned my addr 172.21.15.16:0/3842664446 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:56.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.554+0000 7f64deffd700 1 --2- 172.21.15.16:0/3842664446 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f64e015a3d0 0x7f64e00cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:56.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.554+0000 7f64dffff700 1 --2- 172.21.15.16:0/3842664446 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f64e015e0b0 0x7f64e00cc240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:56.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.554+0000 7f64df7fe700 1 -- 172.21.15.16:0/3842664446 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f64e015e0b0 msgr2=0x7f64e00cc240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:56.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.554+0000 7f64df7fe700 1 --2- 172.21.15.16:0/3842664446 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f64e015e0b0 0x7f64e00cc240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:56.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.554+0000 7f64df7fe700 1 -- 172.21.15.16:0/3842664446 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f64e015a3d0 msgr2=0x7f64e00cbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:56.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.554+0000 7f64df7fe700 1 --2- 172.21.15.16:0/3842664446 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f64e015a3d0 0x7f64e00cbd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:56.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.554+0000 7f64df7fe700 1 -- 172.21.15.16:0/3842664446 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64c8009550 con 0x7f64e00cf430 2023-10-16T13:44:56.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.554+0000 7f64dffff700 1 --2- 172.21.15.16:0/3842664446 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f64e015e0b0 0x7f64e00cc240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:44:56.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.554+0000 7f64deffd700 1 --2- 172.21.15.16:0/3842664446 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f64e015a3d0 0x7f64e00cbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:44:56.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.554+0000 7f64df7fe700 1 --2- 172.21.15.16:0/3842664446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64e00cf430 0x7f64e0146e00 secure :-1 s=READY pgs=2340 cs=0 l=1 rev1=1 crypto rx=0x7f64c8000c00 tx=0x7f64c8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:56.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.554+0000 7f64d7fff700 1 -- 172.21.15.16:0/3842664446 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64c8004480 con 0x7f64e00cf430 2023-10-16T13:44:56.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.554+0000 7f64d7fff700 1 -- 172.21.15.16:0/3842664446 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64c801f070 con 0x7f64e00cf430 2023-10-16T13:44:56.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.554+0000 7f64d7fff700 1 -- 172.21.15.16:0/3842664446 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64c80038c0 con 0x7f64e00cf430 2023-10-16T13:44:56.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.554+0000 7f64e6017700 1 -- 172.21.15.16:0/3842664446 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64e015d690 con 0x7f64e00cf430 2023-10-16T13:44:56.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.554+0000 7f64e6017700 1 -- 172.21.15.16:0/3842664446 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64e015dc50 con 0x7f64e00cf430 2023-10-16T13:44:56.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.555+0000 7f64e6017700 1 -- 172.21.15.16:0/3842664446 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64e004ec20 con 0x7f64e00cf430 2023-10-16T13:44:56.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.556+0000 7f64d7fff700 1 -- 172.21.15.16:0/3842664446 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 130) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f64c8021030 con 0x7f64e00cf430 2023-10-16T13:44:56.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.556+0000 7f64d7fff700 1 --2- 172.21.15.16:0/3842664446 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f64c006ada0 0x7f64c006d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:56.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.556+0000 7f64d7fff700 1 -- 172.21.15.16:0/3842664446 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 6501+0+0 (secure 0 0 0) 0x7f64c808ab00 con 0x7f64e00cf430 2023-10-16T13:44:56.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.556+0000 7f64deffd700 1 --2- 172.21.15.16:0/3842664446 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f64c006ada0 0x7f64c006d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:56.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.557+0000 7f64deffd700 1 --2- 172.21.15.16:0/3842664446 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f64c006ada0 0x7f64c006d260 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f64e01f4410 tx=0x7f64d0008040 comp rx=0 tx=0).ready entity=mgr.11088 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:56.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.561+0000 7f64d7fff700 1 -- 172.21.15.16:0/3842664446 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f64c8055cf0 con 0x7f64e00cf430 2023-10-16T13:44:56.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.794+0000 7f64e6017700 1 -- 172.21.15.16:0/3842664446 --> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "path": "snap_test_dir1", "retention_spec_or_period": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f64e006e600 con 0x7f64c006ada0 2023-10-16T13:44:56.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.846+0000 7f64d7fff700 1 -- 172.21.15.16:0/3842664446 <== mgr.11088 v2:172.21.15.16:6832/3220139196 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+39 (secure 0 0 0) 0x7f64d0015440 con 0x7f64c006ada0 2023-10-16T13:44:56.848 INFO:teuthology.orchestra.run.smithi016.stdout:Retention added to path snap_test_dir1 2023-10-16T13:44:56.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.848+0000 7f64e6017700 1 -- 172.21.15.16:0/3842664446 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f64c006ada0 msgr2=0x7f64c006d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:56.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.848+0000 7f64e6017700 1 --2- 172.21.15.16:0/3842664446 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f64c006ada0 0x7f64c006d260 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f64e01f4410 tx=0x7f64d0008040 comp rx=0 tx=0).stop 2023-10-16T13:44:56.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.848+0000 7f64e6017700 1 -- 172.21.15.16:0/3842664446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64e00cf430 msgr2=0x7f64e0146e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:56.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.848+0000 7f64e6017700 1 --2- 172.21.15.16:0/3842664446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64e00cf430 0x7f64e0146e00 secure :-1 s=READY pgs=2340 cs=0 l=1 rev1=1 crypto rx=0x7f64c8000c00 tx=0x7f64c8004750 comp rx=0 tx=0).stop 2023-10-16T13:44:56.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.848+0000 7f64e6017700 1 -- 172.21.15.16:0/3842664446 shutdown_connections 2023-10-16T13:44:56.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.848+0000 7f64e6017700 1 --2- 172.21.15.16:0/3842664446 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f64e015e0b0 0x7f64e00cc240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:56.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.848+0000 7f64e6017700 1 --2- 172.21.15.16:0/3842664446 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f64c006ada0 0x7f64c006d260 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:56.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.848+0000 7f64e6017700 1 --2- 172.21.15.16:0/3842664446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64e00cf430 0x7f64e0146e00 unknown :-1 s=CLOSED pgs=2340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:56.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.848+0000 7f64e6017700 1 --2- 172.21.15.16:0/3842664446 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f64e015a3d0 0x7f64e00cbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:56.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.848+0000 7f64e6017700 1 -- 172.21.15.16:0/3842664446 >> 172.21.15.16:0/3842664446 conn(0x7f64e006f430 msgr2=0x7f64e00ce870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:56.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.848+0000 7f64e6017700 1 -- 172.21.15.16:0/3842664446 shutdown_connections 2023-10-16T13:44:56.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.848+0000 7f64e6017700 1 -- 172.21.15.16:0/3842664446 wait complete. 2023-10-16T13:44:56.863 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention added to path snap_test_dir1 2023-10-16T13:44:56.863 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2023-10-16-13_45_00 in ~4s... 2023-10-16T13:44:56.863 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m'], expected_retention: [{'m': 1}] 2023-10-16T13:44:56.864 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json snap_test_dir1 2023-10-16T13:44:56.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.988+0000 7f3b40396700 1 Processor -- start 2023-10-16T13:44:56.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.988+0000 7f3b40396700 1 -- start start 2023-10-16T13:44:56.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.989+0000 7f3b40396700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f3b380d2110 0x7f3b380ae1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:56.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.989+0000 7f3b40396700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3b3806ca10 0x7f3b380ae720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:56.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.989+0000 7f3b40396700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b380b20f0 con 0x7f3b380ce6a0 2023-10-16T13:44:56.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.989+0000 7f3b40396700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b380d03d0 con 0x7f3b3806ca10 2023-10-16T13:44:56.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.989+0000 7f3b40396700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b380b2340 con 0x7f3b380d2110 2023-10-16T13:44:56.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.989+0000 7f3b3e933700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3b3806ca10 0x7f3b380ae720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:56.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.989+0000 7f3b3e933700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3b3806ca10 0x7f3b380ae720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52206/0 (socket says 172.21.15.16:52206) 2023-10-16T13:44:56.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.989+0000 7f3b3e933700 1 -- 172.21.15.16:0/1078739975 learned_addr learned my addr 172.21.15.16:0/1078739975 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:56.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.989+0000 7f3b3d931700 1 --2- 172.21.15.16:0/1078739975 >> v2:172.21.15.49:3300/0 conn(0x7f3b380d2110 0x7f3b380ae1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:56.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.989+0000 7f3b3e933700 1 -- 172.21.15.16:0/1078739975 >> v2:172.21.15.49:3300/0 conn(0x7f3b380d2110 msgr2=0x7f3b380ae1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:56.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.989+0000 7f3b3e933700 1 --2- 172.21.15.16:0/1078739975 >> v2:172.21.15.49:3300/0 conn(0x7f3b380d2110 0x7f3b380ae1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:56.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.989+0000 7f3b3e933700 1 -- 172.21.15.16:0/1078739975 >> v1:172.21.15.49:6789/0 conn(0x7f3b380ce6a0 legacy=0x7f3b3806c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:44:56.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.989+0000 7f3b3e933700 1 -- 172.21.15.16:0/1078739975 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b3806bb00 con 0x7f3b3806ca10 2023-10-16T13:44:56.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.990+0000 7f3b3e933700 1 --2- 172.21.15.16:0/1078739975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b3806ca10 0x7f3b380ae720 secure :-1 s=READY pgs=2341 cs=0 l=1 rev1=1 crypto rx=0x7f3b380cf0f0 tx=0x7f3b3000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d4d702bd8cab97e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:56.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.990+0000 7f3b3d130700 1 -- 172.21.15.16:0/1078739975 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b3000e070 con 0x7f3b3806ca10 2023-10-16T13:44:56.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.990+0000 7f3b3d130700 1 -- 172.21.15.16:0/1078739975 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b30003a20 con 0x7f3b3806ca10 2023-10-16T13:44:56.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.990+0000 7f3b3d130700 1 -- 172.21.15.16:0/1078739975 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b30003dd0 con 0x7f3b3806ca10 2023-10-16T13:44:56.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.990+0000 7f3b40396700 1 -- 172.21.15.16:0/1078739975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b3806ca10 msgr2=0x7f3b380ae720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:56.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.990+0000 7f3b40396700 1 --2- 172.21.15.16:0/1078739975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b3806ca10 0x7f3b380ae720 secure :-1 s=READY pgs=2341 cs=0 l=1 rev1=1 crypto rx=0x7f3b380cf0f0 tx=0x7f3b3000b330 comp rx=0 tx=0).stop 2023-10-16T13:44:56.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.990+0000 7f3b40396700 1 -- 172.21.15.16:0/1078739975 shutdown_connections 2023-10-16T13:44:56.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.990+0000 7f3b40396700 1 --2- 172.21.15.16:0/1078739975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b3806ca10 0x7f3b380ae720 unknown :-1 s=CLOSED pgs=2341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:56.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.990+0000 7f3b40396700 1 --2- 172.21.15.16:0/1078739975 >> v2:172.21.15.49:3300/0 conn(0x7f3b380d2110 0x7f3b380ae1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:56.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.990+0000 7f3b40396700 1 -- 172.21.15.16:0/1078739975 >> 172.21.15.16:0/1078739975 conn(0x7f3b380d15f0 msgr2=0x7f3b380d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:56.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.990+0000 7f3b40396700 1 -- 172.21.15.16:0/1078739975 shutdown_connections 2023-10-16T13:44:56.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.990+0000 7f3b40396700 1 -- 172.21.15.16:0/1078739975 wait complete. 2023-10-16T13:44:57.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.992+0000 7f3b40396700 1 Processor -- start 2023-10-16T13:44:57.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.992+0000 7f3b40396700 1 -- start start 2023-10-16T13:44:57.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.992+0000 7f3b40396700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3b3806ca10 0x7f3b3815bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:57.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.992+0000 7f3b40396700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b380ce6a0 0x7f3b380ccb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:57.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.992+0000 7f3b40396700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3b380d2110 0x7f3b380b51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:57.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.992+0000 7f3b40396700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b380c6240 con 0x7f3b380ce6a0 2023-10-16T13:44:57.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.992+0000 7f3b40396700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b3815c0a0 con 0x7f3b3806ca10 2023-10-16T13:44:57.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.992+0000 7f3b40396700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b380c6490 con 0x7f3b380d2110 2023-10-16T13:44:57.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.993+0000 7f3b3d931700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b380ce6a0 0x7f3b380ccb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:57.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.993+0000 7f3b3d931700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b380ce6a0 0x7f3b380ccb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52210/0 (socket says 172.21.15.16:52210) 2023-10-16T13:44:57.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.993+0000 7f3b3d931700 1 -- 172.21.15.16:0/1232847159 learned_addr learned my addr 172.21.15.16:0/1232847159 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:44:57.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.993+0000 7f3b3e132700 1 --2- 172.21.15.16:0/1232847159 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3b3806ca10 0x7f3b3815bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:57.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.993+0000 7f3b3e933700 1 --2- 172.21.15.16:0/1232847159 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3b380d2110 0x7f3b380b51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:57.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.993+0000 7f3b3d931700 1 -- 172.21.15.16:0/1232847159 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3b380d2110 msgr2=0x7f3b380b51c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:57.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.993+0000 7f3b3d931700 1 --2- 172.21.15.16:0/1232847159 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3b380d2110 0x7f3b380b51c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:57.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.993+0000 7f3b3d931700 1 -- 172.21.15.16:0/1232847159 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3b3806ca10 msgr2=0x7f3b3815bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:57.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.993+0000 7f3b3d931700 1 --2- 172.21.15.16:0/1232847159 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3b3806ca10 0x7f3b3815bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:57.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.993+0000 7f3b3d931700 1 -- 172.21.15.16:0/1232847159 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b30009b30 con 0x7f3b380ce6a0 2023-10-16T13:44:57.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.993+0000 7f3b3d931700 1 --2- 172.21.15.16:0/1232847159 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b380ce6a0 0x7f3b380ccb90 secure :-1 s=READY pgs=2342 cs=0 l=1 rev1=1 crypto rx=0x7f3b3400d800 tx=0x7f3b3400db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:57.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.993+0000 7f3b2effd700 1 -- 172.21.15.16:0/1232847159 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b340125f0 con 0x7f3b380ce6a0 2023-10-16T13:44:57.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.993+0000 7f3b2effd700 1 -- 172.21.15.16:0/1232847159 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b34012750 con 0x7f3b380ce6a0 2023-10-16T13:44:57.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.993+0000 7f3b40396700 1 -- 172.21.15.16:0/1232847159 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b380c6770 con 0x7f3b380ce6a0 2023-10-16T13:44:57.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.993+0000 7f3b2effd700 1 -- 172.21.15.16:0/1232847159 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b34012ae0 con 0x7f3b380ce6a0 2023-10-16T13:44:57.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.993+0000 7f3b40396700 1 -- 172.21.15.16:0/1232847159 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b380c6d30 con 0x7f3b380ce6a0 2023-10-16T13:44:57.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.994+0000 7f3b40396700 1 -- 172.21.15.16:0/1232847159 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b3804ebb0 con 0x7f3b380ce6a0 2023-10-16T13:44:57.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.995+0000 7f3b2effd700 1 -- 172.21.15.16:0/1232847159 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 130) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f3b34012c40 con 0x7f3b380ce6a0 2023-10-16T13:44:57.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.995+0000 7f3b2effd700 1 --2- 172.21.15.16:0/1232847159 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f3b1c06add0 0x7f3b1c06d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:44:57.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.995+0000 7f3b2effd700 1 -- 172.21.15.16:0/1232847159 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 6501+0+0 (secure 0 0 0) 0x7f3b340896a0 con 0x7f3b380ce6a0 2023-10-16T13:44:57.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.995+0000 7f3b3e132700 1 --2- 172.21.15.16:0/1232847159 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f3b1c06add0 0x7f3b1c06d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:44:57.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:56.996+0000 7f3b3e132700 1 --2- 172.21.15.16:0/1232847159 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f3b1c06add0 0x7f3b1c06d290 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f3b28007fd0 tx=0x7f3b28007ec0 comp rx=0 tx=0).ready entity=mgr.11088 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:44:57.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:57.000+0000 7f3b2effd700 1 -- 172.21.15.16:0/1232847159 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3b340546f0 con 0x7f3b380ce6a0 2023-10-16T13:44:57.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:57.228+0000 7f3b40396700 1 -- 172.21.15.16:0/1232847159 --> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3b380d0690 con 0x7f3b1c06add0 2023-10-16T13:44:57.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:57.237+0000 7f3b2effd700 1 -- 172.21.15.16:0/1232847159 <== mgr.11088 v2:172.21.15.16:6832/3220139196 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+72 (secure 0 0 0) 0x7f3b28004d00 con 0x7f3b1c06add0 2023-10-16T13:44:57.240 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:44:57.240 INFO:teuthology.orchestra.run.smithi016.stdout:{"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{"m": 1}]} 2023-10-16T13:44:57.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:57.239+0000 7f3b40396700 1 -- 172.21.15.16:0/1232847159 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f3b1c06add0 msgr2=0x7f3b1c06d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:57.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:57.239+0000 7f3b40396700 1 --2- 172.21.15.16:0/1232847159 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f3b1c06add0 0x7f3b1c06d290 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f3b28007fd0 tx=0x7f3b28007ec0 comp rx=0 tx=0).stop 2023-10-16T13:44:57.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:57.239+0000 7f3b40396700 1 -- 172.21.15.16:0/1232847159 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b380ce6a0 msgr2=0x7f3b380ccb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:44:57.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:57.239+0000 7f3b40396700 1 --2- 172.21.15.16:0/1232847159 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b380ce6a0 0x7f3b380ccb90 secure :-1 s=READY pgs=2342 cs=0 l=1 rev1=1 crypto rx=0x7f3b3400d800 tx=0x7f3b3400db10 comp rx=0 tx=0).stop 2023-10-16T13:44:57.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:57.239+0000 7f3b40396700 1 -- 172.21.15.16:0/1232847159 shutdown_connections 2023-10-16T13:44:57.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:57.239+0000 7f3b40396700 1 --2- 172.21.15.16:0/1232847159 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3b380d2110 0x7f3b380b51c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:57.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:57.239+0000 7f3b40396700 1 --2- 172.21.15.16:0/1232847159 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3b3806ca10 0x7f3b3815bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:57.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:57.239+0000 7f3b40396700 1 --2- 172.21.15.16:0/1232847159 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f3b1c06add0 0x7f3b1c06d290 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:57.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:57.239+0000 7f3b40396700 1 --2- 172.21.15.16:0/1232847159 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b380ce6a0 0x7f3b380ccb90 unknown :-1 s=CLOSED pgs=2342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:44:57.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:57.239+0000 7f3b40396700 1 -- 172.21.15.16:0/1232847159 >> 172.21.15.16:0/1232847159 conn(0x7f3b380d15f0 msgr2=0x7f3b380cd510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:44:57.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:57.240+0000 7f3b40396700 1 -- 172.21.15.16:0/1232847159 shutdown_connections 2023-10-16T13:44:57.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:44:57.240+0000 7f3b40396700 1 -- 172.21.15.16:0/1232847159 wait complete. 2023-10-16T13:44:57.254 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{"m": 1}]} 2023-10-16T13:44:57.257 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1m'], 'retention': [{'m': 1}]} 2023-10-16T13:44:57.257 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:44:57.258 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T13:44:57.276 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T13:44:57.276 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T13:44:57.277 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T13:44:58.278 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:44:58.279 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T13:44:58.301 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T13:44:58.301 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T13:44:58.301 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T13:44:59.302 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:44:59.303 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T13:44:59.325 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T13:44:59.325 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T13:44:59.326 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T13:45:00.326 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:45:00.327 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T13:45:00.350 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-16T13:45:00.351 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T13:45:00.351 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T13:45:00.859 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:45:00.856+0000 7f17f24c5700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:45:00.859 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:45:00.856+0000 7f17f24c5700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:45:00.859 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:45:00.856+0000 7f17f24c5700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:45:00.860 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:45:00.857+0000 7f17f24c5700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:45:00.860 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:45:00.857+0000 7f17f24c5700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:45:01.352 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:45:01.353 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T13:45:01.376 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_45_00_UTC 2023-10-16T13:45:01.377 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-13_45_00_UTC'} 2023-10-16T13:45:01.377 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2023-10-16-13_45_00_UTC'} 2023-10-16T13:45:01.378 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T13:45:01.378 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2023-10-16-13_45_00_UTC'] 2023-10-16T13:45:01.379 DEBUG:tasks.cephfs.test_snap_schedules:exec=1697463896.8633978, now = 1697463901.3792741, timo = 4 2023-10-16T13:45:02.380 DEBUG:tasks.cephfs.test_snap_schedules:expecting removal of snap snap_test_dir1/.snap/scheduled-2023-10-16-13_45_00 in ~64s... 2023-10-16T13:45:02.381 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:45:02.381 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T13:45:02.402 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_45_00_UTC 2023-10-16T13:45:02.403 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-13_45_00_UTC'} 2023-10-16T13:45:02.403 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T13:45:02.403 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T13:45:03.102 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:45:03.104 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:45:03.406 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:45:03.407 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T13:45:03.430 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_45_00_UTC 2023-10-16T13:45:03.430 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-13_45_00_UTC'} 2023-10-16T13:45:03.431 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T13:45:03.431 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T13:45:04.432 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:45:04.433 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T13:45:04.455 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_45_00_UTC 2023-10-16T13:45:04.456 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-13_45_00_UTC'} 2023-10-16T13:45:04.456 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T13:45:04.456 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T13:45:05.457 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:45:05.459 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T13:45:05.480 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_45_00_UTC 2023-10-16T13:45:05.481 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-13_45_00_UTC'} 2023-10-16T13:45:05.482 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T13:45:05.482 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T13:45:06.483 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:45:06.483 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T13:45:06.505 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_45_00_UTC 2023-10-16T13:45:06.506 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-13_45_00_UTC'} 2023-10-16T13:45:06.507 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T13:45:06.507 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T13:45:07.508 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:45:07.509 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T13:45:07.531 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_45_00_UTC 2023-10-16T13:45:07.532 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-13_45_00_UTC'} 2023-10-16T13:45:07.532 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T13:45:07.532 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T13:45:08.533 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:45:08.533 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T13:45:08.556 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_45_00_UTC 2023-10-16T13:45:08.556 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-13_45_00_UTC'} 2023-10-16T13:45:08.557 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T13:45:08.557 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T13:45:09.558 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:45:09.559 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T13:45:09.582 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_45_00_UTC 2023-10-16T13:45:09.582 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-13_45_00_UTC'} 2023-10-16T13:45:09.582 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T13:45:09.583 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T13:45:10.584 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:45:10.585 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T13:45:10.607 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_45_00_UTC 2023-10-16T13:45:10.608 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-13_45_00_UTC'} 2023-10-16T13:45:10.608 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T13:45:10.608 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T13:45:11.609 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:45:11.610 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T13:45:11.631 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_45_00_UTC 2023-10-16T13:45:11.632 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-13_45_00_UTC'} 2023-10-16T13:45:11.633 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T13:45:11.633 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T13:45:12.634 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:45:12.635 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T13:45:12.657 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_45_00_UTC 2023-10-16T13:45:12.658 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-13_45_00_UTC'} 2023-10-16T13:45:12.658 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T13:45:12.658 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T13:45:13.660 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:45:13.660 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T13:45:13.683 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_45_00_UTC 2023-10-16T13:45:13.683 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-13_45_00_UTC'} 2023-10-16T13:45:13.684 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T13:45:13.684 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T13:45:14.685 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:45:14.686 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T13:45:14.707 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_45_00_UTC 2023-10-16T13:45:14.708 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-13_45_00_UTC'} 2023-10-16T13:45:14.708 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T13:45:14.708 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T13:45:15.709 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:45:15.710 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T13:45:15.732 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_45_00_UTC 2023-10-16T13:45:15.733 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-13_45_00_UTC'} 2023-10-16T13:45:15.733 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T13:45:15.733 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T13:45:16.734 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:45:16.735 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T13:45:16.757 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_45_00_UTC 2023-10-16T13:45:16.758 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-13_45_00_UTC'} 2023-10-16T13:45:16.758 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T13:45:16.759 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T13:45:17.759 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:45:17.760 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T13:45:17.782 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_45_00_UTC 2023-10-16T13:45:17.782 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-13_45_00_UTC'} 2023-10-16T13:45:17.783 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T13:45:17.783 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T13:45:18.784 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:45:18.785 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T13:45:18.807 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_45_00_UTC 2023-10-16T13:45:18.808 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-13_45_00_UTC'} 2023-10-16T13:45:18.808 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T13:45:18.808 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T13:45:19.810 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:45:19.811 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T13:45:19.833 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_45_00_UTC 2023-10-16T13:45:19.833 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-13_45_00_UTC'} 2023-10-16T13:45:19.833 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T13:45:19.834 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T13:45:20.835 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:45:20.836 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T13:45:20.857 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_45_00_UTC 2023-10-16T13:45:20.858 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-13_45_00_UTC'} 2023-10-16T13:45:20.858 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T13:45:20.859 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T13:45:21.860 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:45:21.860 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T13:45:21.882 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_45_00_UTC 2023-10-16T13:45:21.883 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-13_45_00_UTC'} 2023-10-16T13:45:21.883 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T13:45:21.884 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T13:45:22.884 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:45:22.885 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T13:45:22.907 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_45_00_UTC 2023-10-16T13:45:22.908 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-13_45_00_UTC'} 2023-10-16T13:45:22.908 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T13:45:22.909 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T13:45:23.909 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:45:23.910 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T13:45:23.933 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_45_00_UTC 2023-10-16T13:45:23.933 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-13_45_00_UTC'} 2023-10-16T13:45:23.934 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T13:45:23.934 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T13:45:24.935 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:45:24.936 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T13:45:24.959 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_45_00_UTC 2023-10-16T13:45:24.960 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-13_45_00_UTC'} 2023-10-16T13:45:24.960 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T13:45:24.960 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T13:45:25.962 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:45:25.962 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T13:45:25.984 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_45_00_UTC 2023-10-16T13:45:25.985 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-13_45_00_UTC'} 2023-10-16T13:45:25.985 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T13:45:25.985 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T13:45:26.986 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:45:26.987 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T13:45:27.008 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_45_00_UTC 2023-10-16T13:45:27.009 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-13_45_00_UTC'} 2023-10-16T13:45:27.010 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T13:45:27.010 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T13:45:28.011 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:45:28.012 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T13:45:28.033 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_45_00_UTC 2023-10-16T13:45:28.034 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-13_45_00_UTC'} 2023-10-16T13:45:28.034 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T13:45:28.035 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T13:45:29.036 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:45:29.037 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T13:45:29.059 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_45_00_UTC 2023-10-16T13:45:29.059 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-13_45_00_UTC'} 2023-10-16T13:45:29.060 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T13:45:29.060 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T13:45:30.061 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:45:30.061 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T13:45:30.083 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_45_00_UTC 2023-10-16T13:45:30.084 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-13_45_00_UTC'} 2023-10-16T13:45:30.084 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T13:45:30.084 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T13:45:31.085 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:45:31.085 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T13:45:31.107 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_45_00_UTC 2023-10-16T13:45:31.108 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-13_45_00_UTC'} 2023-10-16T13:45:31.108 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T13:45:31.109 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T13:45:32.110 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:45:32.111 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T13:45:32.132 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_45_00_UTC 2023-10-16T13:45:32.133 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-13_45_00_UTC'} 2023-10-16T13:45:32.134 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T13:45:32.134 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T13:45:33.135 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:45:33.136 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T13:45:33.158 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_45_00_UTC 2023-10-16T13:45:33.159 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-13_45_00_UTC'} 2023-10-16T13:45:33.159 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T13:45:33.159 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T13:45:33.169 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:45:33.202 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:45:34.160 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:45:34.161 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T13:45:34.183 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_45_00_UTC 2023-10-16T13:45:34.184 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-13_45_00_UTC'} 2023-10-16T13:45:34.184 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T13:45:34.184 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T13:45:35.186 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:45:35.187 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T13:45:35.208 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_45_00_UTC 2023-10-16T13:45:35.209 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-13_45_00_UTC'} 2023-10-16T13:45:35.209 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T13:45:35.210 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T13:45:36.211 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:45:36.211 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T13:45:36.233 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_45_00_UTC 2023-10-16T13:45:36.234 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-13_45_00_UTC'} 2023-10-16T13:45:36.234 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T13:45:36.235 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T13:45:37.236 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:45:37.237 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T13:45:37.259 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_45_00_UTC 2023-10-16T13:45:37.260 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-13_45_00_UTC'} 2023-10-16T13:45:37.260 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T13:45:37.260 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T13:45:38.261 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:45:38.262 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T13:45:38.283 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_45_00_UTC 2023-10-16T13:45:38.284 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-13_45_00_UTC'} 2023-10-16T13:45:38.285 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T13:45:38.285 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T13:45:39.286 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:45:39.287 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T13:45:39.308 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_45_00_UTC 2023-10-16T13:45:39.309 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-13_45_00_UTC'} 2023-10-16T13:45:39.309 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T13:45:39.310 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T13:45:40.311 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:45:40.311 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T13:45:40.333 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_45_00_UTC 2023-10-16T13:45:40.334 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-13_45_00_UTC'} 2023-10-16T13:45:40.334 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T13:45:40.334 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T13:45:41.335 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:45:41.336 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T13:45:41.358 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_45_00_UTC 2023-10-16T13:45:41.359 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-13_45_00_UTC'} 2023-10-16T13:45:41.359 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T13:45:41.360 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T13:45:42.360 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:45:42.361 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T13:45:42.383 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_45_00_UTC 2023-10-16T13:45:42.384 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-13_45_00_UTC'} 2023-10-16T13:45:42.384 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T13:45:42.384 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T13:45:43.386 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:45:43.386 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T13:45:43.408 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_45_00_UTC 2023-10-16T13:45:43.409 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-13_45_00_UTC'} 2023-10-16T13:45:43.410 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T13:45:43.410 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T13:45:44.411 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:45:44.412 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T13:45:44.435 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_45_00_UTC 2023-10-16T13:45:44.435 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-13_45_00_UTC'} 2023-10-16T13:45:44.436 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T13:45:44.436 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T13:45:45.437 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:45:45.438 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T13:45:45.459 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_45_00_UTC 2023-10-16T13:45:45.460 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-13_45_00_UTC'} 2023-10-16T13:45:45.461 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T13:45:45.461 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T13:45:46.462 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:45:46.463 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T13:45:46.485 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_45_00_UTC 2023-10-16T13:45:46.486 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-13_45_00_UTC'} 2023-10-16T13:45:46.486 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T13:45:46.487 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T13:45:47.488 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:45:47.489 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T13:45:47.511 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_45_00_UTC 2023-10-16T13:45:47.512 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-13_45_00_UTC'} 2023-10-16T13:45:47.512 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T13:45:47.512 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T13:45:48.513 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:45:48.514 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T13:45:48.536 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_45_00_UTC 2023-10-16T13:45:48.537 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-13_45_00_UTC'} 2023-10-16T13:45:48.537 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T13:45:48.537 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T13:45:49.538 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:45:49.539 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T13:45:49.561 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_45_00_UTC 2023-10-16T13:45:49.561 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-13_45_00_UTC'} 2023-10-16T13:45:49.562 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T13:45:49.562 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T13:45:50.563 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:45:50.563 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T13:45:50.585 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_45_00_UTC 2023-10-16T13:45:50.586 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-13_45_00_UTC'} 2023-10-16T13:45:50.586 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T13:45:50.587 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T13:45:51.588 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:45:51.588 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T13:45:51.610 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_45_00_UTC 2023-10-16T13:45:51.611 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-13_45_00_UTC'} 2023-10-16T13:45:51.611 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T13:45:51.611 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T13:45:52.612 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:45:52.613 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T13:45:52.636 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_45_00_UTC 2023-10-16T13:45:52.636 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-13_45_00_UTC'} 2023-10-16T13:45:52.636 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T13:45:52.637 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T13:45:53.638 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:45:53.638 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T13:45:53.660 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_45_00_UTC 2023-10-16T13:45:53.661 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-13_45_00_UTC'} 2023-10-16T13:45:53.661 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T13:45:53.662 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T13:45:54.663 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:45:54.664 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T13:45:54.685 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_45_00_UTC 2023-10-16T13:45:54.686 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-13_45_00_UTC'} 2023-10-16T13:45:54.687 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T13:45:54.687 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T13:45:55.688 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:45:55.689 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T13:45:55.710 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_45_00_UTC 2023-10-16T13:45:55.711 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-13_45_00_UTC'} 2023-10-16T13:45:55.711 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T13:45:55.712 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T13:45:56.713 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:45:56.714 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T13:45:56.736 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_45_00_UTC 2023-10-16T13:45:56.737 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-13_45_00_UTC'} 2023-10-16T13:45:56.737 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T13:45:56.738 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T13:45:57.739 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:45:57.740 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T13:45:57.762 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_45_00_UTC 2023-10-16T13:45:57.762 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-13_45_00_UTC'} 2023-10-16T13:45:57.763 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T13:45:57.763 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T13:45:58.764 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:45:58.765 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T13:45:58.787 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_45_00_UTC 2023-10-16T13:45:58.787 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-13_45_00_UTC'} 2023-10-16T13:45:58.788 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T13:45:58.788 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T13:45:59.789 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:45:59.789 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T13:45:59.811 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_45_00_UTC 2023-10-16T13:45:59.812 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-13_45_00_UTC'} 2023-10-16T13:45:59.813 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T13:45:59.813 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T13:46:00.814 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:46:00.815 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T13:46:00.837 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_45_00_UTC 2023-10-16T13:46:00.838 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-13_45_00_UTC'} 2023-10-16T13:46:00.838 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-16T13:46:00.839 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-16T13:46:01.839 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:46:01.840 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T13:46:01.862 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_46_00_UTC 2023-10-16T13:46:01.863 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-16-13_46_00_UTC'} 2023-10-16T13:46:01.863 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2023-10-16-13_46_00_UTC'} 2023-10-16T13:46:01.863 INFO:tasks.cephfs.test_snap_schedules:removed: {'scheduled-2023-10-16-13_45_00_UTC'} 2023-10-16T13:46:01.864 DEBUG:tasks.cephfs.test_snap_schedules:snapshots removed=['scheduled-2023-10-16-13_45_00_UTC'] 2023-10-16T13:46:01.864 DEBUG:tasks.cephfs.test_snap_schedules:exec=1697463896.8633978, now = 1697463961.8643198, timo = 64 2023-10-16T13:46:02.865 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2023-10-16T13:46:03.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.004+0000 7f3b9b1e7700 1 Processor -- start 2023-10-16T13:46:03.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.004+0000 7f3b9b1e7700 1 -- start start 2023-10-16T13:46:03.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.005+0000 7f3b9b1e7700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f3b9406a7e0 0x7f3b9406ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:03.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.005+0000 7f3b9b1e7700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b9415ecb0 con 0x7f3b9415b2c0 2023-10-16T13:46:03.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.005+0000 7f3b9b1e7700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b9415b6a0 con 0x7f3b9406e570 2023-10-16T13:46:03.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.005+0000 7f3b9b1e7700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b9406b190 con 0x7f3b9406a7e0 2023-10-16T13:46:03.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.005+0000 7f3b98f83700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f3b9415b2c0 0x7f3b94161340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41522/0 (socket says 172.21.15.16:41522) 2023-10-16T13:46:03.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.005+0000 7f3b98f83700 1 -- 172.21.15.16:0/1014486425 learned_addr learned my addr 172.21.15.16:0/1014486425 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:46:03.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.005+0000 7f3b99784700 1 --2- 172.21.15.16:0/1014486425 >> v2:172.21.15.49:3300/0 conn(0x7f3b9406a7e0 0x7f3b9406ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:03.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.006+0000 7f3b937fe700 1 -- 172.21.15.16:0/1014486425 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2728221154 0 0) 0x7f3b9415ecb0 con 0x7f3b9415b2c0 2023-10-16T13:46:03.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.006+0000 7f3b937fe700 1 -- 172.21.15.16:0/1014486425 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b840035b0 con 0x7f3b9415b2c0 2023-10-16T13:46:03.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.006+0000 7f3b937fe700 1 -- 172.21.15.16:0/1014486425 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1218306062 0 0) 0x7f3b840035b0 con 0x7f3b9415b2c0 2023-10-16T13:46:03.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.006+0000 7f3b937fe700 1 -- 172.21.15.16:0/1014486425 >> v2:172.21.15.49:3300/0 conn(0x7f3b9406a7e0 msgr2=0x7f3b9406ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:03.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.006+0000 7f3b937fe700 1 --2- 172.21.15.16:0/1014486425 >> v2:172.21.15.49:3300/0 conn(0x7f3b9406a7e0 0x7f3b9406ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:03.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.006+0000 7f3b937fe700 1 -- 172.21.15.16:0/1014486425 >> v1:172.21.15.49:6789/0 conn(0x7f3b9406e570 legacy=0x7f3b9415e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:03.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.006+0000 7f3b937fe700 1 -- 172.21.15.16:0/1014486425 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b94161b20 con 0x7f3b9415b2c0 2023-10-16T13:46:03.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.006+0000 7f3b937fe700 1 -- 172.21.15.16:0/1014486425 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f3b88001670 con 0x7f3b9415b2c0 2023-10-16T13:46:03.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.006+0000 7f3b937fe700 1 -- 172.21.15.16:0/1014486425 >> v1:172.21.15.16:6789/0 conn(0x7f3b9415b2c0 legacy=0x7f3b94161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:03.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.006+0000 7f3b937fe700 1 --2- 172.21.15.16:0/1014486425 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3b84003a00 0x7f3b84003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:03.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.006+0000 7f3b937fe700 1 --2- 172.21.15.16:0/1014486425 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b84004850 0x7f3b84004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:03.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.006+0000 7f3b937fe700 1 --2- 172.21.15.16:0/1014486425 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3b84005210 0x7f3b8400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:03.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.006+0000 7f3b937fe700 1 -- 172.21.15.16:0/1014486425 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b9406b190 con 0x7f3b84004850 2023-10-16T13:46:03.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.006+0000 7f3b937fe700 1 -- 172.21.15.16:0/1014486425 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b8400c1b0 con 0x7f3b84003a00 2023-10-16T13:46:03.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.006+0000 7f3b937fe700 1 -- 172.21.15.16:0/1014486425 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b8400c370 con 0x7f3b84005210 2023-10-16T13:46:03.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.006+0000 7f3b93fff700 1 --2- 172.21.15.16:0/1014486425 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b84004850 0x7f3b84004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:03.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.006+0000 7f3b98f83700 1 --2- 172.21.15.16:0/1014486425 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3b84003a00 0x7f3b84003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:03.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.006+0000 7f3b99784700 1 --2- 172.21.15.16:0/1014486425 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3b84005210 0x7f3b8400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:03.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.006+0000 7f3b93fff700 1 -- 172.21.15.16:0/1014486425 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3b84005210 msgr2=0x7f3b8400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:03.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.006+0000 7f3b93fff700 1 --2- 172.21.15.16:0/1014486425 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3b84005210 0x7f3b8400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:03.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.006+0000 7f3b93fff700 1 -- 172.21.15.16:0/1014486425 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3b84003a00 msgr2=0x7f3b84003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:03.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.006+0000 7f3b93fff700 1 --2- 172.21.15.16:0/1014486425 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3b84003a00 0x7f3b84003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:03.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.006+0000 7f3b93fff700 1 -- 172.21.15.16:0/1014486425 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b840035b0 con 0x7f3b84004850 2023-10-16T13:46:03.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.006+0000 7f3b99784700 1 -- 172.21.15.16:0/1014486425 reap_dead start 2023-10-16T13:46:03.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.007+0000 7f3b93fff700 1 --2- 172.21.15.16:0/1014486425 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b84004850 0x7f3b84004cd0 secure :-1 s=READY pgs=2348 cs=0 l=1 rev1=1 crypto rx=0x7f3b800017c0 tx=0x7f3b84005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dc35d72534b5d40 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:03.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.007+0000 7f3b937fe700 1 -- 172.21.15.16:0/1014486425 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b800052d0 con 0x7f3b84004850 2023-10-16T13:46:03.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.007+0000 7f3b937fe700 1 -- 172.21.15.16:0/1014486425 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b80005450 con 0x7f3b84004850 2023-10-16T13:46:03.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.007+0000 7f3b937fe700 1 -- 172.21.15.16:0/1014486425 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b800038c0 con 0x7f3b84004850 2023-10-16T13:46:03.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.007+0000 7f3b9b1e7700 1 -- 172.21.15.16:0/1014486425 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b84004850 msgr2=0x7f3b84004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:03.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.007+0000 7f3b9b1e7700 1 --2- 172.21.15.16:0/1014486425 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b84004850 0x7f3b84004cd0 secure :-1 s=READY pgs=2348 cs=0 l=1 rev1=1 crypto rx=0x7f3b800017c0 tx=0x7f3b84005880 comp rx=0 tx=0).stop 2023-10-16T13:46:03.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.007+0000 7f3b9b1e7700 1 -- 172.21.15.16:0/1014486425 shutdown_connections 2023-10-16T13:46:03.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.007+0000 7f3b9b1e7700 1 --2- 172.21.15.16:0/1014486425 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b84004850 0x7f3b84004cd0 unknown :-1 s=CLOSED pgs=2348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:03.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.007+0000 7f3b9b1e7700 1 -- 172.21.15.16:0/1014486425 >> 172.21.15.16:0/1014486425 conn(0x7f3b940b1f40 msgr2=0x7f3b940b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:03.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.007+0000 7f3b9b1e7700 1 -- 172.21.15.16:0/1014486425 shutdown_connections 2023-10-16T13:46:03.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.007+0000 7f3b9b1e7700 1 -- 172.21.15.16:0/1014486425 wait complete. 2023-10-16T13:46:03.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.009+0000 7f3b9b1e7700 1 Processor -- start 2023-10-16T13:46:03.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.009+0000 7f3b9b1e7700 1 -- start start 2023-10-16T13:46:03.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.009+0000 7f3b9b1e7700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3b84004850 0x7f3b940c9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:03.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.010+0000 7f3b9b1e7700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3b940c6b10 0x7f3b940c3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:03.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.010+0000 7f3b9b1e7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b940c6f90 0x7f3b940c4100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:03.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.010+0000 7f3b9b1e7700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b941492f0 con 0x7f3b940c6f90 2023-10-16T13:46:03.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.010+0000 7f3b9b1e7700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b940c5db0 con 0x7f3b940c6b10 2023-10-16T13:46:03.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.010+0000 7f3b9b1e7700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b941f4050 con 0x7f3b84004850 2023-10-16T13:46:03.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.010+0000 7f3b99784700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b940c6f90 0x7f3b940c4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:03.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.010+0000 7f3b99784700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b940c6f90 0x7f3b940c4100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52264/0 (socket says 172.21.15.16:52264) 2023-10-16T13:46:03.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.010+0000 7f3b99784700 1 -- 172.21.15.16:0/3201856199 learned_addr learned my addr 172.21.15.16:0/3201856199 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:03.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.010+0000 7f3b93fff700 1 --2- 172.21.15.16:0/3201856199 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3b940c6b10 0x7f3b940c3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:03.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.010+0000 7f3b98f83700 1 --2- 172.21.15.16:0/3201856199 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3b84004850 0x7f3b940c9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:03.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.010+0000 7f3b99784700 1 -- 172.21.15.16:0/3201856199 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3b84004850 msgr2=0x7f3b940c9aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:03.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.010+0000 7f3b99784700 1 --2- 172.21.15.16:0/3201856199 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3b84004850 0x7f3b940c9aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:03.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.010+0000 7f3b99784700 1 -- 172.21.15.16:0/3201856199 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3b940c6b10 msgr2=0x7f3b940c3bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:03.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.010+0000 7f3b99784700 1 --2- 172.21.15.16:0/3201856199 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3b940c6b10 0x7f3b940c3bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:03.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.010+0000 7f3b99784700 1 -- 172.21.15.16:0/3201856199 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b940c6030 con 0x7f3b940c6f90 2023-10-16T13:46:03.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.010+0000 7f3b98f83700 1 --2- 172.21.15.16:0/3201856199 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3b84004850 0x7f3b940c9aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:46:03.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.010+0000 7f3b99784700 1 --2- 172.21.15.16:0/3201856199 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b940c6f90 0x7f3b940c4100 secure :-1 s=READY pgs=2349 cs=0 l=1 rev1=1 crypto rx=0x7f3b9406b430 tx=0x7f3b8c011f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:03.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.010+0000 7f3b917fa700 1 -- 172.21.15.16:0/3201856199 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b8c00e040 con 0x7f3b940c6f90 2023-10-16T13:46:03.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.010+0000 7f3b917fa700 1 -- 172.21.15.16:0/3201856199 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b8c00f040 con 0x7f3b940c6f90 2023-10-16T13:46:03.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.010+0000 7f3b9b1e7700 1 -- 172.21.15.16:0/3201856199 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b940c6310 con 0x7f3b940c6f90 2023-10-16T13:46:03.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.010+0000 7f3b917fa700 1 -- 172.21.15.16:0/3201856199 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b8c00fd50 con 0x7f3b940c6f90 2023-10-16T13:46:03.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.010+0000 7f3b9b1e7700 1 -- 172.21.15.16:0/3201856199 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b940c8510 con 0x7f3b940c6f90 2023-10-16T13:46:03.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.011+0000 7f3b9b1e7700 1 -- 172.21.15.16:0/3201856199 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b940c6830 con 0x7f3b940c6f90 2023-10-16T13:46:03.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.012+0000 7f3b917fa700 1 -- 172.21.15.16:0/3201856199 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 133) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f3b8c015430 con 0x7f3b940c6f90 2023-10-16T13:46:03.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.012+0000 7f3b917fa700 1 --2- 172.21.15.16:0/3201856199 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f3b7806b190 0x7f3b7806d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:03.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.012+0000 7f3b917fa700 1 -- 172.21.15.16:0/3201856199 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 6588+0+0 (secure 0 0 0) 0x7f3b8c08a580 con 0x7f3b940c6f90 2023-10-16T13:46:03.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.012+0000 7f3b98f83700 1 --2- 172.21.15.16:0/3201856199 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f3b7806b190 0x7f3b7806d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:03.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.013+0000 7f3b98f83700 1 --2- 172.21.15.16:0/3201856199 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f3b7806b190 0x7f3b7806d650 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3b9404fec0 tx=0x7f3b88009040 comp rx=0 tx=0).ready entity=mgr.11088 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:03.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.017+0000 7f3b917fa700 1 -- 172.21.15.16:0/3201856199 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3b940c6830 con 0x7f3b940c6f90 2023-10-16T13:46:03.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.249+0000 7f3b9b1e7700 1 -- 172.21.15.16:0/3201856199 --> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f3b941620e0 con 0x7f3b7806b190 2023-10-16T13:46:03.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.311+0000 7f3b917fa700 1 -- 172.21.15.16:0/3201856199 <== mgr.11088 v2:172.21.15.16:6832/3220139196 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f3b880015d0 con 0x7f3b7806b190 2023-10-16T13:46:03.314 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule removed for path snap_test_dir1 2023-10-16T13:46:03.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.314+0000 7f3b9b1e7700 1 -- 172.21.15.16:0/3201856199 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f3b7806b190 msgr2=0x7f3b7806d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:03.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.314+0000 7f3b9b1e7700 1 --2- 172.21.15.16:0/3201856199 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f3b7806b190 0x7f3b7806d650 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3b9404fec0 tx=0x7f3b88009040 comp rx=0 tx=0).stop 2023-10-16T13:46:03.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.314+0000 7f3b9b1e7700 1 -- 172.21.15.16:0/3201856199 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b940c6f90 msgr2=0x7f3b940c4100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:03.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.314+0000 7f3b9b1e7700 1 --2- 172.21.15.16:0/3201856199 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b940c6f90 0x7f3b940c4100 secure :-1 s=READY pgs=2349 cs=0 l=1 rev1=1 crypto rx=0x7f3b9406b430 tx=0x7f3b8c011f80 comp rx=0 tx=0).stop 2023-10-16T13:46:03.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.314+0000 7f3b9b1e7700 1 -- 172.21.15.16:0/3201856199 shutdown_connections 2023-10-16T13:46:03.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.314+0000 7f3b9b1e7700 1 --2- 172.21.15.16:0/3201856199 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f3b7806b190 0x7f3b7806d650 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:03.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.314+0000 7f3b9b1e7700 1 --2- 172.21.15.16:0/3201856199 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b940c6f90 0x7f3b940c4100 unknown :-1 s=CLOSED pgs=2349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:03.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.314+0000 7f3b9b1e7700 1 --2- 172.21.15.16:0/3201856199 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3b84004850 0x7f3b940c9aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:03.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.314+0000 7f3b9b1e7700 1 --2- 172.21.15.16:0/3201856199 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3b940c6b10 0x7f3b940c3bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:03.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.314+0000 7f3b9b1e7700 1 -- 172.21.15.16:0/3201856199 >> 172.21.15.16:0/3201856199 conn(0x7f3b940b1f40 msgr2=0x7f3b940cc230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:03.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.314+0000 7f3b9b1e7700 1 -- 172.21.15.16:0/3201856199 shutdown_connections 2023-10-16T13:46:03.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.314+0000 7f3b9b1e7700 1 -- 172.21.15.16:0/3201856199 wait complete. 2023-10-16T13:46:03.328 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2023-10-16T13:46:03.329 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:46:03.329 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T13:46:03.347 INFO:teuthology.orchestra.run.smithi016.stdout:scheduled-2023-10-16-13_46_00_UTC 2023-10-16T13:46:03.347 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-16-13_46_00_UTC 2023-10-16T13:46:03.348 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:46:03.348 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-16-13_46_00_UTC) 2023-10-16T13:46:03.396 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:46:03.409 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:46:03.410 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2023-10-16T13:46:03.451 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:46:03.470 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-16T13:46:03.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.663+0000 7f0ac0c6a700 1 Processor -- start 2023-10-16T13:46:03.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.663+0000 7f0ac0c6a700 1 -- start start 2023-10-16T13:46:03.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.664+0000 7f0ac0c6a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0abc15ec80 0x7f0abc0cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:03.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.664+0000 7f0ac0c6a700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0abc0b2440 con 0x7f0abc15f060 2023-10-16T13:46:03.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.664+0000 7f0ac0c6a700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0abc0d00e0 con 0x7f0abc0d2110 2023-10-16T13:46:03.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.664+0000 7f0ac0c6a700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0abc0b2c50 con 0x7f0abc15ec80 2023-10-16T13:46:03.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.664+0000 7f0ab9d9b700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0abc15f060 0x7f0abc0b4440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41544/0 (socket says 172.21.15.16:41544) 2023-10-16T13:46:03.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.664+0000 7f0ab9d9b700 1 -- 172.21.15.16:0/1670229701 learned_addr learned my addr 172.21.15.16:0/1670229701 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:46:03.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.664+0000 7f0aba59c700 1 --2- 172.21.15.16:0/1670229701 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0abc15ec80 0x7f0abc0cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:03.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.664+0000 7f0ab959a700 1 -- 172.21.15.16:0/1670229701 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3726762051 0 0) 0x7f0abc0b2440 con 0x7f0abc15f060 2023-10-16T13:46:03.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.664+0000 7f0ab959a700 1 -- 172.21.15.16:0/1670229701 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0aa40035b0 con 0x7f0abc15f060 2023-10-16T13:46:03.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.664+0000 7f0ab959a700 1 -- 172.21.15.16:0/1670229701 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 35108573 0 0) 0x7f0aa40035b0 con 0x7f0abc15f060 2023-10-16T13:46:03.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.664+0000 7f0ab959a700 1 -- 172.21.15.16:0/1670229701 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0abc15ec80 msgr2=0x7f0abc0cdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:03.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.664+0000 7f0ab959a700 1 --2- 172.21.15.16:0/1670229701 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0abc15ec80 0x7f0abc0cdec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:03.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.664+0000 7f0ab959a700 1 -- 172.21.15.16:0/1670229701 >> v1:172.21.15.49:6789/0 conn(0x7f0abc0d2110 legacy=0x7f0abc0b4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:03.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.664+0000 7f0ab959a700 1 -- 172.21.15.16:0/1670229701 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0abc0cef60 con 0x7f0abc15f060 2023-10-16T13:46:03.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.664+0000 7f0ab959a700 1 -- 172.21.15.16:0/1670229701 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f0ab0001670 con 0x7f0abc15f060 2023-10-16T13:46:03.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.665+0000 7f0ab959a700 1 -- 172.21.15.16:0/1670229701 >> v1:172.21.15.16:6789/0 conn(0x7f0abc15f060 legacy=0x7f0abc0b4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:03.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.665+0000 7f0ab959a700 1 --2- 172.21.15.16:0/1670229701 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0abc15ec80 0x7f0aa4003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:03.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.665+0000 7f0ab959a700 1 --2- 172.21.15.16:0/1670229701 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0aa4003fb0 0x7f0aa4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:03.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.665+0000 7f0ab959a700 1 --2- 172.21.15.16:0/1670229701 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aa4004d90 0x7f0aa4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:03.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.665+0000 7f0ab959a700 1 -- 172.21.15.16:0/1670229701 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0abc0b2c50 con 0x7f0aa4004d90 2023-10-16T13:46:03.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.665+0000 7f0ab959a700 1 -- 172.21.15.16:0/1670229701 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa400baf0 con 0x7f0aa4003fb0 2023-10-16T13:46:03.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.665+0000 7f0ab959a700 1 -- 172.21.15.16:0/1670229701 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa400bcb0 con 0x7f0abc15ec80 2023-10-16T13:46:03.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.665+0000 7f0abad9d700 1 --2- 172.21.15.16:0/1670229701 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aa4004d90 0x7f0aa4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:03.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.665+0000 7f0aba59c700 1 --2- 172.21.15.16:0/1670229701 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0abc15ec80 0x7f0aa4003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:03.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.665+0000 7f0ab9d9b700 1 --2- 172.21.15.16:0/1670229701 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0aa4003fb0 0x7f0aa4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:03.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.665+0000 7f0abad9d700 1 -- 172.21.15.16:0/1670229701 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0abc15ec80 msgr2=0x7f0aa4003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:03.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.665+0000 7f0abad9d700 1 --2- 172.21.15.16:0/1670229701 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0abc15ec80 0x7f0aa4003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:03.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.665+0000 7f0abad9d700 1 -- 172.21.15.16:0/1670229701 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0aa4003fb0 msgr2=0x7f0aa4004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:03.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.665+0000 7f0abad9d700 1 --2- 172.21.15.16:0/1670229701 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0aa4003fb0 0x7f0aa4004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:03.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.665+0000 7f0abad9d700 1 -- 172.21.15.16:0/1670229701 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0aa40035b0 con 0x7f0aa4004d90 2023-10-16T13:46:03.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.665+0000 7f0abad9d700 1 --2- 172.21.15.16:0/1670229701 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aa4004d90 0x7f0aa4005210 secure :-1 s=READY pgs=2351 cs=0 l=1 rev1=1 crypto rx=0x7f0ab400c240 tx=0x7f0aa4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fe62290fc3799e1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:03.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.665+0000 7f0ab959a700 1 -- 172.21.15.16:0/1670229701 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ab4001df0 con 0x7f0aa4004d90 2023-10-16T13:46:03.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.665+0000 7f0ab959a700 1 -- 172.21.15.16:0/1670229701 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ab400cda0 con 0x7f0aa4004d90 2023-10-16T13:46:03.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.665+0000 7f0ab959a700 1 -- 172.21.15.16:0/1670229701 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ab4003c40 con 0x7f0aa4004d90 2023-10-16T13:46:03.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.666+0000 7f0ac0c6a700 1 -- 172.21.15.16:0/1670229701 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aa4004d90 msgr2=0x7f0aa4005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:03.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.666+0000 7f0ac0c6a700 1 --2- 172.21.15.16:0/1670229701 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aa4004d90 0x7f0aa4005210 secure :-1 s=READY pgs=2351 cs=0 l=1 rev1=1 crypto rx=0x7f0ab400c240 tx=0x7f0aa4005880 comp rx=0 tx=0).stop 2023-10-16T13:46:03.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.666+0000 7f0abad9d700 1 -- 172.21.15.16:0/1670229701 reap_dead start 2023-10-16T13:46:03.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.666+0000 7f0ac0c6a700 1 -- 172.21.15.16:0/1670229701 shutdown_connections 2023-10-16T13:46:03.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.666+0000 7f0ac0c6a700 1 -- 172.21.15.16:0/1670229701 >> 172.21.15.16:0/1670229701 conn(0x7f0abc0d15f0 msgr2=0x7f0abc0d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:03.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.666+0000 7f0ac0c6a700 1 -- 172.21.15.16:0/1670229701 shutdown_connections 2023-10-16T13:46:03.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.666+0000 7f0ac0c6a700 1 -- 172.21.15.16:0/1670229701 wait complete. 2023-10-16T13:46:03.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.667+0000 7f0ac0c6a700 1 Processor -- start 2023-10-16T13:46:03.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.667+0000 7f0ac0c6a700 1 -- start start 2023-10-16T13:46:03.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.668+0000 7f0ac0c6a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aa4004d90 0x7f0abc0c9190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:03.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.668+0000 7f0ac0c6a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0abc158ed0 0x7f0abc0b6040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:03.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.668+0000 7f0ac0c6a700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0abc0b6580 0x7f0abc0ca6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:03.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.668+0000 7f0ac0c6a700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0abc0b6a00 con 0x7f0aa4004d90 2023-10-16T13:46:03.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.668+0000 7f0ac0c6a700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0abc0c9f50 con 0x7f0abc0b6580 2023-10-16T13:46:03.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.668+0000 7f0ac0c6a700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0abc0ca230 con 0x7f0abc158ed0 2023-10-16T13:46:03.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.668+0000 7f0aba59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aa4004d90 0x7f0abc0c9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:03.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.668+0000 7f0aba59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aa4004d90 0x7f0abc0c9190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52280/0 (socket says 172.21.15.16:52280) 2023-10-16T13:46:03.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.668+0000 7f0aba59c700 1 -- 172.21.15.16:0/401111480 learned_addr learned my addr 172.21.15.16:0/401111480 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:03.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.668+0000 7f0ab9d9b700 1 --2- 172.21.15.16:0/401111480 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0abc158ed0 0x7f0abc0b6040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:03.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.669+0000 7f0aba59c700 1 -- 172.21.15.16:0/401111480 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0abc158ed0 msgr2=0x7f0abc0b6040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:03.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.669+0000 7f0aba59c700 1 --2- 172.21.15.16:0/401111480 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0abc158ed0 0x7f0abc0b6040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:03.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.669+0000 7f0aba59c700 1 -- 172.21.15.16:0/401111480 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0abc0b6580 msgr2=0x7f0abc0ca6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:03.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.669+0000 7f0abad9d700 1 --2- 172.21.15.16:0/401111480 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0abc0b6580 0x7f0abc0ca6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:03.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.669+0000 7f0aba59c700 1 --2- 172.21.15.16:0/401111480 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0abc0b6580 0x7f0abc0ca6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:03.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.669+0000 7f0aba59c700 1 -- 172.21.15.16:0/401111480 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0abc159720 con 0x7f0aa4004d90 2023-10-16T13:46:03.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.669+0000 7f0ab9d9b700 1 --2- 172.21.15.16:0/401111480 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0abc158ed0 0x7f0abc0b6040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:46:03.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.669+0000 7f0aba59c700 1 --2- 172.21.15.16:0/401111480 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aa4004d90 0x7f0abc0c9190 secure :-1 s=READY pgs=2352 cs=0 l=1 rev1=1 crypto rx=0x7f0aac011ad0 tx=0x7f0aac011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:03.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.669+0000 7f0aaaffd700 1 -- 172.21.15.16:0/401111480 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aac00f9a0 con 0x7f0aa4004d90 2023-10-16T13:46:03.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.669+0000 7f0aaaffd700 1 -- 172.21.15.16:0/401111480 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0aac00fb00 con 0x7f0aa4004d90 2023-10-16T13:46:03.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.669+0000 7f0ac0c6a700 1 -- 172.21.15.16:0/401111480 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0abc159a00 con 0x7f0aa4004d90 2023-10-16T13:46:03.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.669+0000 7f0aaaffd700 1 -- 172.21.15.16:0/401111480 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aac00feb0 con 0x7f0aa4004d90 2023-10-16T13:46:03.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.669+0000 7f0ac0c6a700 1 -- 172.21.15.16:0/401111480 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0abc159350 con 0x7f0aa4004d90 2023-10-16T13:46:03.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.670+0000 7f0ac0c6a700 1 -- 172.21.15.16:0/401111480 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0abc04ebb0 con 0x7f0aa4004d90 2023-10-16T13:46:03.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.670+0000 7f0aaaffd700 1 -- 172.21.15.16:0/401111480 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 133) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f0aac014440 con 0x7f0aa4004d90 2023-10-16T13:46:03.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.671+0000 7f0aaaffd700 1 --2- 172.21.15.16:0/401111480 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f0a9c06b190 0x7f0a9c06d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:03.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.671+0000 7f0aaaffd700 1 -- 172.21.15.16:0/401111480 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 6588+0+0 (secure 0 0 0) 0x7f0aac08a500 con 0x7f0aa4004d90 2023-10-16T13:46:03.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.671+0000 7f0ab9d9b700 1 --2- 172.21.15.16:0/401111480 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f0a9c06b190 0x7f0a9c06d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:03.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.671+0000 7f0ab9d9b700 1 --2- 172.21.15.16:0/401111480 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f0a9c06b190 0x7f0a9c06d650 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0abc0b7570 tx=0x7f0ab0009040 comp rx=0 tx=0).ready entity=mgr.11088 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:03.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.676+0000 7f0aaaffd700 1 -- 172.21.15.16:0/401111480 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0aac054fa0 con 0x7f0aa4004d90 2023-10-16T13:46:03.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:03.938+0000 7f0ac0c6a700 1 -- 172.21.15.16:0/401111480 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f0abc0ccf60 con 0x7f0aa4004d90 2023-10-16T13:46:04.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:04.493+0000 7f0aaaffd700 1 -- 172.21.15.16:0/401111480 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v134) v1 ==== 93+0+0 (secure 0 0 0) 0x7f0aac0585c0 con 0x7f0aa4004d90 2023-10-16T13:46:04.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:04.497+0000 7f0ac0c6a700 1 -- 172.21.15.16:0/401111480 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f0a9c06b190 msgr2=0x7f0a9c06d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:04.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:04.497+0000 7f0ac0c6a700 1 --2- 172.21.15.16:0/401111480 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f0a9c06b190 0x7f0a9c06d650 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0abc0b7570 tx=0x7f0ab0009040 comp rx=0 tx=0).stop 2023-10-16T13:46:04.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:04.497+0000 7f0ac0c6a700 1 -- 172.21.15.16:0/401111480 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aa4004d90 msgr2=0x7f0abc0c9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:04.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:04.497+0000 7f0ac0c6a700 1 --2- 172.21.15.16:0/401111480 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aa4004d90 0x7f0abc0c9190 secure :-1 s=READY pgs=2352 cs=0 l=1 rev1=1 crypto rx=0x7f0aac011ad0 tx=0x7f0aac011e90 comp rx=0 tx=0).stop 2023-10-16T13:46:04.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:04.497+0000 7f0ac0c6a700 1 -- 172.21.15.16:0/401111480 shutdown_connections 2023-10-16T13:46:04.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:04.497+0000 7f0ac0c6a700 1 --2- 172.21.15.16:0/401111480 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0abc0b6580 0x7f0abc0ca6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:04.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:04.497+0000 7f0ac0c6a700 1 --2- 172.21.15.16:0/401111480 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f0a9c06b190 0x7f0a9c06d650 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:04.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:04.497+0000 7f0ac0c6a700 1 --2- 172.21.15.16:0/401111480 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aa4004d90 0x7f0abc0c9190 secure :-1 s=CLOSED pgs=2352 cs=0 l=1 rev1=1 crypto rx=0x7f0aac011ad0 tx=0x7f0aac011e90 comp rx=0 tx=0).stop 2023-10-16T13:46:04.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:04.497+0000 7f0ac0c6a700 1 --2- 172.21.15.16:0/401111480 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0abc158ed0 0x7f0abc0b6040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:04.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:04.497+0000 7f0ac0c6a700 1 -- 172.21.15.16:0/401111480 >> 172.21.15.16:0/401111480 conn(0x7f0abc0d15f0 msgr2=0x7f0abc0cd480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:04.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:04.499+0000 7f0ac0c6a700 1 -- 172.21.15.16:0/401111480 shutdown_connections 2023-10-16T13:46:04.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:04.499+0000 7f0ac0c6a700 1 -- 172.21.15.16:0/401111480 wait complete. 2023-10-16T13:46:04.517 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T13:46:04.518 DEBUG:teuthology.orchestra.run.smithi016:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T13:46:04.562 DEBUG:teuthology.orchestra.run.smithi049:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T13:46:04.606 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T13:46:04.607 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T13:46:04.607 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:46:04.625 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T13:46:04.626 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:46:04.651 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:46:04.649+0000 7f3e59442200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T13:46:04.656 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:04.654+0000 7f9d0ecc4200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T13:46:04.673 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:46:04.720 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:46:04.718+0000 7f3e59442200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T13:46:04.726 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:04.723+0000 7f9d0ecc4200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T13:46:04.795 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:46:04.793+0000 7f3e59442200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T13:46:04.803 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:04.801+0000 7f9d0ecc4200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T13:46:04.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:04.839+0000 7f113d412700 1 Processor -- start 2023-10-16T13:46:04.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:04.839+0000 7f113d412700 1 -- start start 2023-10-16T13:46:04.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:04.840+0000 7f113d412700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f11380ce4f0 0x7f11380d0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:04.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:04.840+0000 7f113d412700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11380b0970 con 0x7f113815b630 2023-10-16T13:46:04.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:04.840+0000 7f113d412700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11380cd690 con 0x7f113815ba10 2023-10-16T13:46:04.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:04.840+0000 7f113d412700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11380cd970 con 0x7f11380ce4f0 2023-10-16T13:46:04.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:04.840+0000 7f11367fc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f11380ce4f0 0x7f11380d0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:04.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:04.840+0000 7f11367fc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f11380ce4f0 0x7f11380d0380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52350/0 (socket says 172.21.15.16:52350) 2023-10-16T13:46:04.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:04.840+0000 7f11367fc700 1 -- 172.21.15.16:0/3154766327 learned_addr learned my addr 172.21.15.16:0/3154766327 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:04.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:04.840+0000 7f1136ffd700 1 --1- 172.21.15.16:0/3154766327 >> v1:172.21.15.16:6789/0 conn(0x7f113815b630 0x7f11380cdde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41622/0 (socket says 172.21.15.16:41622) 2023-10-16T13:46:04.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:04.841+0000 7f11367fc700 1 -- 172.21.15.16:0/3154766327 >> v1:172.21.15.49:6789/0 conn(0x7f113815ba10 legacy=0x7f11380b1430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:04.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:04.841+0000 7f11367fc700 1 -- 172.21.15.16:0/3154766327 >> v1:172.21.15.16:6789/0 conn(0x7f113815b630 legacy=0x7f11380cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:04.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:04.841+0000 7f11367fc700 1 -- 172.21.15.16:0/3154766327 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11380b0c30 con 0x7f11380ce4f0 2023-10-16T13:46:04.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:04.841+0000 7f1135ffb700 1 -- 172.21.15.16:0/3154766327 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3267117398 0 0) 0x7f11380b0970 con 0x7f113815b630 2023-10-16T13:46:04.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:04.841+0000 7f11367fc700 1 --2- 172.21.15.16:0/3154766327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11380ce4f0 0x7f11380d0380 secure :-1 s=READY pgs=2358 cs=0 l=1 rev1=1 crypto rx=0x7f112c0098f0 tx=0x7f112c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea8767dd954e213b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:04.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:04.841+0000 7f1135ffb700 1 -- 172.21.15.16:0/3154766327 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f112c00b8a0 con 0x7f11380ce4f0 2023-10-16T13:46:04.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:04.841+0000 7f1135ffb700 1 -- 172.21.15.16:0/3154766327 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f112c00ba00 con 0x7f11380ce4f0 2023-10-16T13:46:04.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:04.841+0000 7f1135ffb700 1 -- 172.21.15.16:0/3154766327 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f112c00bd90 con 0x7f11380ce4f0 2023-10-16T13:46:04.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:04.842+0000 7f113d412700 1 -- 172.21.15.16:0/3154766327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11380ce4f0 msgr2=0x7f11380d0380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:04.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:04.842+0000 7f113d412700 1 --2- 172.21.15.16:0/3154766327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11380ce4f0 0x7f11380d0380 secure :-1 s=READY pgs=2358 cs=0 l=1 rev1=1 crypto rx=0x7f112c0098f0 tx=0x7f112c009c00 comp rx=0 tx=0).stop 2023-10-16T13:46:04.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:04.842+0000 7f113d412700 1 -- 172.21.15.16:0/3154766327 shutdown_connections 2023-10-16T13:46:04.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:04.842+0000 7f113d412700 1 --2- 172.21.15.16:0/3154766327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11380ce4f0 0x7f11380d0380 unknown :-1 s=CLOSED pgs=2358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:04.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:04.842+0000 7f113d412700 1 -- 172.21.15.16:0/3154766327 >> 172.21.15.16:0/3154766327 conn(0x7f11380d14b0 msgr2=0x7f11380d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:04.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:04.842+0000 7f113d412700 1 -- 172.21.15.16:0/3154766327 shutdown_connections 2023-10-16T13:46:04.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:04.842+0000 7f113d412700 1 -- 172.21.15.16:0/3154766327 wait complete. 2023-10-16T13:46:04.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:04.843+0000 7f113d412700 1 Processor -- start 2023-10-16T13:46:04.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:04.843+0000 7f113d412700 1 -- start start 2023-10-16T13:46:04.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:04.844+0000 7f113d412700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f11380ce4f0 0x7f11380b47a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:04.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:04.844+0000 7f113d412700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f113815b630 0x7f113815d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:04.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:04.844+0000 7f113d412700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f113815ba10 0x7f11381f2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:04.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:04.844+0000 7f113d412700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f113815ced0 con 0x7f113815ba10 2023-10-16T13:46:04.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:04.844+0000 7f113d412700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11380b4ce0 con 0x7f113815b630 2023-10-16T13:46:04.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:04.844+0000 7f113d412700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11381f38a0 con 0x7f11380ce4f0 2023-10-16T13:46:04.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:04.844+0000 7f11377fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f113815ba10 0x7f11381f2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:04.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:04.844+0000 7f11367fc700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f113815b630 0x7f113815d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:04.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:04.844+0000 7f11367fc700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f113815b630 0x7f113815d8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:51418/0 (socket says 172.21.15.16:51418) 2023-10-16T13:46:04.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:04.844+0000 7f11367fc700 1 -- 172.21.15.16:0/3497814322 learned_addr learned my addr 172.21.15.16:0/3497814322 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:04.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:04.844+0000 7f1136ffd700 1 --2- 172.21.15.16:0/3497814322 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f11380ce4f0 0x7f11380b47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:04.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:04.844+0000 7f11367fc700 1 -- 172.21.15.16:0/3497814322 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f11380ce4f0 msgr2=0x7f11380b47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:04.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:04.844+0000 7f11367fc700 1 --2- 172.21.15.16:0/3497814322 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f11380ce4f0 0x7f11380b47a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:04.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:04.844+0000 7f11367fc700 1 -- 172.21.15.16:0/3497814322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f113815ba10 msgr2=0x7f11381f2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:04.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:04.844+0000 7f11367fc700 1 --2- 172.21.15.16:0/3497814322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f113815ba10 0x7f11381f2e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:04.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:04.844+0000 7f11367fc700 1 -- 172.21.15.16:0/3497814322 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f112c009550 con 0x7f113815b630 2023-10-16T13:46:04.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:04.844+0000 7f11377fe700 1 --2- 172.21.15.16:0/3497814322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f113815ba10 0x7f11381f2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:46:04.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:04.845+0000 7f11367fc700 1 --2- 172.21.15.16:0/3497814322 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f113815b630 0x7f113815d8e0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f1138072680 tx=0x7f112c004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:04.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:04.845+0000 7f111f7fe700 1 -- 172.21.15.16:0/3497814322 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f112c01a900 con 0x7f113815b630 2023-10-16T13:46:04.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:04.845+0000 7f111f7fe700 1 -- 172.21.15.16:0/3497814322 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f112c01f070 con 0x7f113815b630 2023-10-16T13:46:04.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:04.845+0000 7f111f7fe700 1 -- 172.21.15.16:0/3497814322 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f112c01ac60 con 0x7f113815b630 2023-10-16T13:46:04.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:04.845+0000 7f113d412700 1 -- 172.21.15.16:0/3497814322 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11381f3b80 con 0x7f113815b630 2023-10-16T13:46:04.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:04.845+0000 7f113d412700 1 -- 172.21.15.16:0/3497814322 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f113815d3c0 con 0x7f113815b630 2023-10-16T13:46:04.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:04.846+0000 7f113d412700 1 -- 172.21.15.16:0/3497814322 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f113804ebb0 con 0x7f113815b630 2023-10-16T13:46:04.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:04.846+0000 7f111f7fe700 1 -- 172.21.15.16:0/3497814322 <== mon.1 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 134) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f112c01b430 con 0x7f113815b630 2023-10-16T13:46:04.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:04.847+0000 7f111f7fe700 1 --2- 172.21.15.16:0/3497814322 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f112406b140 0x7f112406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:04.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:04.847+0000 7f1136ffd700 1 -- 172.21.15.16:0/3497814322 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f112406b140 msgr2=0x7f112406d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3220139196 2023-10-16T13:46:04.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:04.847+0000 7f111f7fe700 1 -- 172.21.15.16:0/3497814322 <== mon.1 v2:172.21.15.49:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 6588+0+0 (secure 0 0 0) 0x7f112c021070 con 0x7f113815b630 2023-10-16T13:46:04.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:04.847+0000 7f1136ffd700 1 --2- 172.21.15.16:0/3497814322 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f112406b140 0x7f112406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:46:04.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:04.852+0000 7f111f7fe700 1 -- 172.21.15.16:0/3497814322 <== mon.1 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f112c0560d0 con 0x7f113815b630 2023-10-16T13:46:04.877 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:46:04.875+0000 7f3e59442200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T13:46:04.888 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:04.886+0000 7f9d0ecc4200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T13:46:05.026 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:46:05.024+0000 7f3e59442200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T13:46:05.038 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:05.035+0000 7f9d0ecc4200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T13:46:05.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.048+0000 7f1136ffd700 1 -- 172.21.15.16:0/3497814322 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f112406b140 msgr2=0x7f112406d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3220139196 2023-10-16T13:46:05.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.048+0000 7f1136ffd700 1 --2- 172.21.15.16:0/3497814322 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f112406b140 0x7f112406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:46:05.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.057+0000 7f113d412700 1 -- 172.21.15.16:0/3497814322 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f11381f3d10 con 0x7f113815b630 2023-10-16T13:46:05.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.059+0000 7f111f7fe700 1 -- 172.21.15.16:0/3497814322 <== mon.1 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v210) v1 ==== 74+0+17519 (secure 0 0 0) 0x7f112c0596f0 con 0x7f113815b630 2023-10-16T13:46:05.061 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:46:05.062 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":210,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:45:57.425666+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2023-10-16T13:44:38.716950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"205","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_data","create_time":"2023-10-16T13:44:39.172499+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"206","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":204,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":204,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":204,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":204,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":204,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":204,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"26.5","mappings":[{"from":1,"to":0}]},{"pgid":"26.21","mappings":[{"from":1,"to":0}]},{"pgid":"26.24","mappings":[{"from":3,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/3477190137":"2023-10-17T13:44:55.326924+0000","172.21.15.16:0/1906189720":"2023-10-17T13:44:55.326924+0000","172.21.15.16:6833/1268439414":"2023-10-17T13:44:55.326924+0000","172.21.15.16:6832/1268439414":"2023-10-17T13:44:55.326924+0000","172.21.15.16:0/3697646183":"2023-10-17T13:44:55.326924+0000","172.21.15.16:0/4067982926":"2023-10-17T13:44:55.326924+0000","172.21.15.16:0/3408848304":"2023-10-17T13:44:55.326924+0000","172.21.15.16:0/2782518306":"2023-10-17T13:44:55.326924+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:46:05.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.061+0000 7f113d412700 1 -- 172.21.15.16:0/3497814322 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f112406b140 msgr2=0x7f112406d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:46:05.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.061+0000 7f113d412700 1 --2- 172.21.15.16:0/3497814322 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f112406b140 0x7f112406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:05.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.061+0000 7f113d412700 1 -- 172.21.15.16:0/3497814322 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f113815b630 msgr2=0x7f113815d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:05.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.061+0000 7f113d412700 1 --2- 172.21.15.16:0/3497814322 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f113815b630 0x7f113815d8e0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f1138072680 tx=0x7f112c004750 comp rx=0 tx=0).stop 2023-10-16T13:46:05.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.061+0000 7f113d412700 1 -- 172.21.15.16:0/3497814322 shutdown_connections 2023-10-16T13:46:05.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.061+0000 7f113d412700 1 --2- 172.21.15.16:0/3497814322 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f112406b140 0x7f112406d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:05.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.061+0000 7f113d412700 1 --2- 172.21.15.16:0/3497814322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f113815ba10 0x7f11381f2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:05.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.061+0000 7f113d412700 1 --2- 172.21.15.16:0/3497814322 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f11380ce4f0 0x7f11380b47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:05.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.061+0000 7f113d412700 1 --2- 172.21.15.16:0/3497814322 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f113815b630 0x7f113815d8e0 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:05.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.061+0000 7f113d412700 1 -- 172.21.15.16:0/3497814322 >> 172.21.15.16:0/3497814322 conn(0x7f11380d14b0 msgr2=0x7f113806fc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:05.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.061+0000 7f113d412700 1 -- 172.21.15.16:0/3497814322 shutdown_connections 2023-10-16T13:46:05.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.061+0000 7f113d412700 1 -- 172.21.15.16:0/3497814322 wait complete. 2023-10-16T13:46:05.077 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 25, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T13:44:38.716950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '205', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 26, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T13:44:39.172499+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '206', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T13:46:05.077 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T13:46:05.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.244+0000 7f0a78851700 1 Processor -- start 2023-10-16T13:46:05.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.244+0000 7f0a78851700 1 -- start start 2023-10-16T13:46:05.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.246+0000 7f0a78851700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f0a7415ec80 0x7f0a740cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:05.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.246+0000 7f0a78851700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0a7415f060 0x7f0a740ce400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:05.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.246+0000 7f0a78851700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a740b2440 con 0x7f0a740d2110 2023-10-16T13:46:05.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.246+0000 7f0a78851700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a740b2c50 con 0x7f0a7415ec80 2023-10-16T13:46:05.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.246+0000 7f0a78851700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a740d00e0 con 0x7f0a7415f060 2023-10-16T13:46:05.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.246+0000 7f0a71d9b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0a7415f060 0x7f0a740ce400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:05.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.246+0000 7f0a7259c700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f0a7415ec80 0x7f0a740cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:05.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.246+0000 7f0a71d9b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0a7415f060 0x7f0a740ce400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:40658/0 (socket says 172.21.15.16:40658) 2023-10-16T13:46:05.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.246+0000 7f0a71d9b700 1 -- 172.21.15.16:0/2050524780 learned_addr learned my addr 172.21.15.16:0/2050524780 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:05.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.247+0000 7f0a71d9b700 1 -- 172.21.15.16:0/2050524780 >> v2:172.21.15.49:3300/0 conn(0x7f0a7415ec80 msgr2=0x7f0a740cdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:05.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.247+0000 7f0a71d9b700 1 --2- 172.21.15.16:0/2050524780 >> v2:172.21.15.49:3300/0 conn(0x7f0a7415ec80 0x7f0a740cdec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:05.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.247+0000 7f0a71d9b700 1 -- 172.21.15.16:0/2050524780 >> v1:172.21.15.49:6789/0 conn(0x7f0a740d2110 legacy=0x7f0a740b4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:05.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.247+0000 7f0a71d9b700 1 -- 172.21.15.16:0/2050524780 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a740cef60 con 0x7f0a7415f060 2023-10-16T13:46:05.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.247+0000 7f0a7259c700 1 --2- 172.21.15.16:0/2050524780 >> v2:172.21.15.49:3300/0 conn(0x7f0a7415ec80 0x7f0a740cdec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:46:05.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.247+0000 7f0a71d9b700 1 --2- 172.21.15.16:0/2050524780 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0a7415f060 0x7f0a740ce400 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f0a680098f0 tx=0x7f0a68009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f93dd7a0313b47b1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:05.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.247+0000 7f0a7159a700 1 -- 172.21.15.16:0/2050524780 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a68011830 con 0x7f0a7415f060 2023-10-16T13:46:05.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.247+0000 7f0a7159a700 1 -- 172.21.15.16:0/2050524780 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a68011990 con 0x7f0a7415f060 2023-10-16T13:46:05.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.247+0000 7f0a7159a700 1 -- 172.21.15.16:0/2050524780 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a68011d20 con 0x7f0a7415f060 2023-10-16T13:46:05.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.248+0000 7f0a78851700 1 -- 172.21.15.16:0/2050524780 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0a7415f060 msgr2=0x7f0a740ce400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:05.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.248+0000 7f0a78851700 1 --2- 172.21.15.16:0/2050524780 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0a7415f060 0x7f0a740ce400 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f0a680098f0 tx=0x7f0a68009c00 comp rx=0 tx=0).stop 2023-10-16T13:46:05.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.248+0000 7f0a78851700 1 -- 172.21.15.16:0/2050524780 shutdown_connections 2023-10-16T13:46:05.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.248+0000 7f0a78851700 1 --2- 172.21.15.16:0/2050524780 >> v2:172.21.15.49:3300/0 conn(0x7f0a7415ec80 0x7f0a740cdec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:05.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.248+0000 7f0a78851700 1 --2- 172.21.15.16:0/2050524780 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0a7415f060 0x7f0a740ce400 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:05.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.248+0000 7f0a78851700 1 -- 172.21.15.16:0/2050524780 >> 172.21.15.16:0/2050524780 conn(0x7f0a740d15f0 msgr2=0x7f0a740d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:05.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.248+0000 7f0a78851700 1 -- 172.21.15.16:0/2050524780 shutdown_connections 2023-10-16T13:46:05.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.248+0000 7f0a78851700 1 -- 172.21.15.16:0/2050524780 wait complete. 2023-10-16T13:46:05.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.250+0000 7f0a78851700 1 Processor -- start 2023-10-16T13:46:05.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.250+0000 7f0a78851700 1 -- start start 2023-10-16T13:46:05.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.251+0000 7f0a78851700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0a740d2110 0x7f0a7415ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:05.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.251+0000 7f0a78851700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a7415ec80 0x7f0a740c5c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:05.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.251+0000 7f0a78851700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0a7415f060 0x7f0a740b65e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:05.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.251+0000 7f0a78851700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a740c97a0 con 0x7f0a7415ec80 2023-10-16T13:46:05.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.251+0000 7f0a78851700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a740c91e0 con 0x7f0a740d2110 2023-10-16T13:46:05.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.251+0000 7f0a78851700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a740c94c0 con 0x7f0a7415f060 2023-10-16T13:46:05.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.252+0000 7f0a71d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a7415ec80 0x7f0a740c5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:05.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.252+0000 7f0a7259c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0a740d2110 0x7f0a7415ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:05.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.252+0000 7f0a71d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a7415ec80 0x7f0a740c5c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52406/0 (socket says 172.21.15.16:52406) 2023-10-16T13:46:05.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.252+0000 7f0a7259c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0a740d2110 0x7f0a7415ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:51466/0 (socket says 172.21.15.16:51466) 2023-10-16T13:46:05.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.252+0000 7f0a71d9b700 1 -- 172.21.15.16:0/3183268958 learned_addr learned my addr 172.21.15.16:0/3183268958 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:05.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.252+0000 7f0a72d9d700 1 --2- 172.21.15.16:0/3183268958 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0a7415f060 0x7f0a740b65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:05.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.252+0000 7f0a71d9b700 1 -- 172.21.15.16:0/3183268958 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0a7415f060 msgr2=0x7f0a740b65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:05.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.252+0000 7f0a71d9b700 1 --2- 172.21.15.16:0/3183268958 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0a7415f060 0x7f0a740b65e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:05.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.252+0000 7f0a71d9b700 1 -- 172.21.15.16:0/3183268958 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0a740d2110 msgr2=0x7f0a7415ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:05.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.252+0000 7f0a71d9b700 1 --2- 172.21.15.16:0/3183268958 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0a740d2110 0x7f0a7415ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:05.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.252+0000 7f0a71d9b700 1 -- 172.21.15.16:0/3183268958 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a640094a0 con 0x7f0a7415ec80 2023-10-16T13:46:05.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.252+0000 7f0a71d9b700 1 --2- 172.21.15.16:0/3183268958 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a7415ec80 0x7f0a740c5c40 secure :-1 s=READY pgs=2359 cs=0 l=1 rev1=1 crypto rx=0x7f0a68009c40 tx=0x7f0a68004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:05.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.252+0000 7f0a62ffd700 1 -- 172.21.15.16:0/3183268958 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a68004510 con 0x7f0a7415ec80 2023-10-16T13:46:05.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.252+0000 7f0a62ffd700 1 -- 172.21.15.16:0/3183268958 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a6801e070 con 0x7f0a7415ec80 2023-10-16T13:46:05.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.252+0000 7f0a62ffd700 1 -- 172.21.15.16:0/3183268958 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a680038e0 con 0x7f0a7415ec80 2023-10-16T13:46:05.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.252+0000 7f0a78851700 1 -- 172.21.15.16:0/3183268958 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a68009550 con 0x7f0a7415ec80 2023-10-16T13:46:05.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.252+0000 7f0a78851700 1 -- 172.21.15.16:0/3183268958 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a740c6920 con 0x7f0a7415ec80 2023-10-16T13:46:05.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.254+0000 7f0a78851700 1 -- 172.21.15.16:0/3183268958 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a740c64a0 con 0x7f0a7415ec80 2023-10-16T13:46:05.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.254+0000 7f0a62ffd700 1 -- 172.21.15.16:0/3183268958 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 134) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f0a68003a40 con 0x7f0a7415ec80 2023-10-16T13:46:05.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.254+0000 7f0a62ffd700 1 --2- 172.21.15.16:0/3183268958 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f0a5406b140 0x7f0a5406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:05.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.255+0000 7f0a7259c700 1 -- 172.21.15.16:0/3183268958 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f0a5406b140 msgr2=0x7f0a5406d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3220139196 2023-10-16T13:46:05.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.255+0000 7f0a62ffd700 1 -- 172.21.15.16:0/3183268958 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 6588+0+0 (secure 0 0 0) 0x7f0a6808aaf0 con 0x7f0a7415ec80 2023-10-16T13:46:05.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.255+0000 7f0a7259c700 1 --2- 172.21.15.16:0/3183268958 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f0a5406b140 0x7f0a5406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:46:05.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.259+0000 7f0a62ffd700 1 -- 172.21.15.16:0/3183268958 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0a740c64a0 con 0x7f0a7415ec80 2023-10-16T13:46:05.367 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:46:05.366+0000 7f3e59442200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T13:46:05.390 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:05.388+0000 7f9d0ecc4200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T13:46:05.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.456+0000 7f0a7259c700 1 -- 172.21.15.16:0/3183268958 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f0a5406b140 msgr2=0x7f0a5406d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3220139196 2023-10-16T13:46:05.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.456+0000 7f0a7259c700 1 --2- 172.21.15.16:0/3183268958 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f0a5406b140 0x7f0a5406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:46:05.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.464+0000 7f0a78851700 1 -- 172.21.15.16:0/3183268958 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0a740cfd50 con 0x7f0a7415ec80 2023-10-16T13:46:05.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.465+0000 7f0a62ffd700 1 -- 172.21.15.16:0/3183268958 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v210) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0a740cfd50 con 0x7f0a7415ec80 2023-10-16T13:46:05.467 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T13:46:05.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.467+0000 7f0a78851700 1 -- 172.21.15.16:0/3183268958 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f0a5406b140 msgr2=0x7f0a5406d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:46:05.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.467+0000 7f0a78851700 1 --2- 172.21.15.16:0/3183268958 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f0a5406b140 0x7f0a5406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:05.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.467+0000 7f0a78851700 1 -- 172.21.15.16:0/3183268958 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a7415ec80 msgr2=0x7f0a740c5c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:05.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.467+0000 7f0a78851700 1 --2- 172.21.15.16:0/3183268958 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a7415ec80 0x7f0a740c5c40 secure :-1 s=READY pgs=2359 cs=0 l=1 rev1=1 crypto rx=0x7f0a68009c40 tx=0x7f0a68004750 comp rx=0 tx=0).stop 2023-10-16T13:46:05.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.467+0000 7f0a78851700 1 -- 172.21.15.16:0/3183268958 shutdown_connections 2023-10-16T13:46:05.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.467+0000 7f0a78851700 1 --2- 172.21.15.16:0/3183268958 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0a7415f060 0x7f0a740b65e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:05.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.467+0000 7f0a78851700 1 --2- 172.21.15.16:0/3183268958 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0a740d2110 0x7f0a7415ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:05.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.467+0000 7f0a78851700 1 --2- 172.21.15.16:0/3183268958 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f0a5406b140 0x7f0a5406d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:05.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.467+0000 7f0a78851700 1 --2- 172.21.15.16:0/3183268958 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a7415ec80 0x7f0a740c5c40 unknown :-1 s=CLOSED pgs=2359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:05.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.467+0000 7f0a78851700 1 -- 172.21.15.16:0/3183268958 >> 172.21.15.16:0/3183268958 conn(0x7f0a740d15f0 msgr2=0x7f0a740d19d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:05.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.467+0000 7f0a78851700 1 -- 172.21.15.16:0/3183268958 shutdown_connections 2023-10-16T13:46:05.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.467+0000 7f0a78851700 1 -- 172.21.15.16:0/3183268958 wait complete. 2023-10-16T13:46:05.483 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T13:46:05.487 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:46:05.486+0000 7f3e59442200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T13:46:05.515 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:05.513+0000 7f9d0ecc4200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T13:46:05.566 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:46:05.564+0000 7f3e59442200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T13:46:05.594 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:05.592+0000 7f9d0ecc4200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T13:46:05.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.612+0000 7fe03d624700 1 Processor -- start 2023-10-16T13:46:05.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.612+0000 7fe03d624700 1 -- start start 2023-10-16T13:46:05.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.612+0000 7fe03d624700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe0380ce6a0 0x7fe03806c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:05.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.612+0000 7fe03d624700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fe0380d2110 0x7fe03806c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:05.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.612+0000 7fe03d624700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0380b20f0 con 0x7fe0380b2620 2023-10-16T13:46:05.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.612+0000 7fe03d624700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0380af1f0 con 0x7fe0380ce6a0 2023-10-16T13:46:05.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.612+0000 7fe03d624700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0380d03d0 con 0x7fe0380d2110 2023-10-16T13:46:05.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.613+0000 7fe036ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe0380ce6a0 0x7fe03806c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:05.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.613+0000 7fe036ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe0380ce6a0 0x7fe03806c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52414/0 (socket says 172.21.15.16:52414) 2023-10-16T13:46:05.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.613+0000 7fe036ffd700 1 -- 172.21.15.16:0/3870207914 learned_addr learned my addr 172.21.15.16:0/3870207914 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:05.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.613+0000 7fe0367fc700 1 --2- 172.21.15.16:0/3870207914 >> v2:172.21.15.49:3300/0 conn(0x7fe0380d2110 0x7fe03806c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:05.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.613+0000 7fe036ffd700 1 -- 172.21.15.16:0/3870207914 >> v2:172.21.15.49:3300/0 conn(0x7fe0380d2110 msgr2=0x7fe03806c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:05.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.613+0000 7fe036ffd700 1 --2- 172.21.15.16:0/3870207914 >> v2:172.21.15.49:3300/0 conn(0x7fe0380d2110 0x7fe03806c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:05.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.613+0000 7fe036ffd700 1 -- 172.21.15.16:0/3870207914 >> v1:172.21.15.49:6789/0 conn(0x7fe0380b2620 legacy=0x7fe0380b0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:05.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.613+0000 7fe036ffd700 1 -- 172.21.15.16:0/3870207914 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0380b2340 con 0x7fe0380ce6a0 2023-10-16T13:46:05.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.613+0000 7fe036ffd700 1 --2- 172.21.15.16:0/3870207914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0380ce6a0 0x7fe03806c300 secure :-1 s=READY pgs=2360 cs=0 l=1 rev1=1 crypto rx=0x7fe0380cf0f0 tx=0x7fe028009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b967f465ca74c7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:05.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.613+0000 7fe035ffb700 1 -- 172.21.15.16:0/3870207914 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe028011830 con 0x7fe0380ce6a0 2023-10-16T13:46:05.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.613+0000 7fe035ffb700 1 -- 172.21.15.16:0/3870207914 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe028011990 con 0x7fe0380ce6a0 2023-10-16T13:46:05.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.613+0000 7fe035ffb700 1 -- 172.21.15.16:0/3870207914 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe028011d20 con 0x7fe0380ce6a0 2023-10-16T13:46:05.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.614+0000 7fe03d624700 1 -- 172.21.15.16:0/3870207914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0380ce6a0 msgr2=0x7fe03806c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:05.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.614+0000 7fe03d624700 1 --2- 172.21.15.16:0/3870207914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0380ce6a0 0x7fe03806c300 secure :-1 s=READY pgs=2360 cs=0 l=1 rev1=1 crypto rx=0x7fe0380cf0f0 tx=0x7fe028009bd0 comp rx=0 tx=0).stop 2023-10-16T13:46:05.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.614+0000 7fe03d624700 1 -- 172.21.15.16:0/3870207914 shutdown_connections 2023-10-16T13:46:05.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.614+0000 7fe03d624700 1 --2- 172.21.15.16:0/3870207914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0380ce6a0 0x7fe03806c300 unknown :-1 s=CLOSED pgs=2360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:05.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.614+0000 7fe03d624700 1 --2- 172.21.15.16:0/3870207914 >> v2:172.21.15.49:3300/0 conn(0x7fe0380d2110 0x7fe03806c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:05.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.614+0000 7fe03d624700 1 -- 172.21.15.16:0/3870207914 >> 172.21.15.16:0/3870207914 conn(0x7fe0380d15f0 msgr2=0x7fe0380d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:05.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.614+0000 7fe03d624700 1 -- 172.21.15.16:0/3870207914 shutdown_connections 2023-10-16T13:46:05.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.614+0000 7fe03d624700 1 -- 172.21.15.16:0/3870207914 wait complete. 2023-10-16T13:46:05.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.616+0000 7fe03d624700 1 Processor -- start 2023-10-16T13:46:05.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.616+0000 7fe03d624700 1 -- start start 2023-10-16T13:46:05.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.617+0000 7fe03d624700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe0380b2620 0x7fe0381f5c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:05.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.617+0000 7fe03d624700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe0380ce6a0 0x7fe0380c5b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:05.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.617+0000 7fe03d624700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0380d2110 0x7fe0380700e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:05.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.617+0000 7fe03d624700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0380b6d90 con 0x7fe0380d2110 2023-10-16T13:46:05.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.617+0000 7fe03d624700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0380b4170 con 0x7fe0380ce6a0 2023-10-16T13:46:05.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.617+0000 7fe03d624700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0380b4fd0 con 0x7fe0380b2620 2023-10-16T13:46:05.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.618+0000 7fe0377fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0380d2110 0x7fe0380700e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:05.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.618+0000 7fe0377fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0380d2110 0x7fe0380700e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52424/0 (socket says 172.21.15.16:52424) 2023-10-16T13:46:05.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.618+0000 7fe0377fe700 1 -- 172.21.15.16:0/2713741219 learned_addr learned my addr 172.21.15.16:0/2713741219 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:05.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.618+0000 7fe0367fc700 1 --2- 172.21.15.16:0/2713741219 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe0380ce6a0 0x7fe0380c5b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:05.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.618+0000 7fe036ffd700 1 --2- 172.21.15.16:0/2713741219 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe0380b2620 0x7fe0381f5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:05.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.618+0000 7fe0377fe700 1 -- 172.21.15.16:0/2713741219 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe0380b2620 msgr2=0x7fe0381f5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:05.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.618+0000 7fe0377fe700 1 --2- 172.21.15.16:0/2713741219 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe0380b2620 0x7fe0381f5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:05.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.618+0000 7fe0377fe700 1 -- 172.21.15.16:0/2713741219 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe0380ce6a0 msgr2=0x7fe0380c5b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:05.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.618+0000 7fe0377fe700 1 --2- 172.21.15.16:0/2713741219 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe0380ce6a0 0x7fe0380c5b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:05.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.618+0000 7fe0377fe700 1 -- 172.21.15.16:0/2713741219 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe028009550 con 0x7fe0380d2110 2023-10-16T13:46:05.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.618+0000 7fe0377fe700 1 --2- 172.21.15.16:0/2713741219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0380d2110 0x7fe0380700e0 secure :-1 s=READY pgs=2361 cs=0 l=1 rev1=1 crypto rx=0x7fe03000cc80 tx=0x7fe03000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:05.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.618+0000 7fe01f7fe700 1 -- 172.21.15.16:0/2713741219 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe030003c00 con 0x7fe0380d2110 2023-10-16T13:46:05.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.618+0000 7fe01f7fe700 1 -- 172.21.15.16:0/2713741219 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe030003d60 con 0x7fe0380d2110 2023-10-16T13:46:05.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.618+0000 7fe03d624700 1 -- 172.21.15.16:0/2713741219 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0380c7db0 con 0x7fe0380d2110 2023-10-16T13:46:05.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.618+0000 7fe01f7fe700 1 -- 172.21.15.16:0/2713741219 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe03000aaa0 con 0x7fe0380d2110 2023-10-16T13:46:05.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.618+0000 7fe03d624700 1 -- 172.21.15.16:0/2713741219 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0380b46a0 con 0x7fe0380d2110 2023-10-16T13:46:05.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.620+0000 7fe03d624700 1 -- 172.21.15.16:0/2713741219 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe03804ebb0 con 0x7fe0380d2110 2023-10-16T13:46:05.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.620+0000 7fe01f7fe700 1 -- 172.21.15.16:0/2713741219 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 134) v1 ==== 89445+0+0 (secure 0 0 0) 0x7fe03000ac00 con 0x7fe0380d2110 2023-10-16T13:46:05.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.620+0000 7fe01f7fe700 1 --2- 172.21.15.16:0/2713741219 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7fe02406b140 0x7fe02406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:05.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.621+0000 7fe036ffd700 1 -- 172.21.15.16:0/2713741219 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7fe02406b140 msgr2=0x7fe02406d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3220139196 2023-10-16T13:46:05.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.621+0000 7fe01f7fe700 1 -- 172.21.15.16:0/2713741219 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 6588+0+0 (secure 0 0 0) 0x7fe030089d60 con 0x7fe0380d2110 2023-10-16T13:46:05.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.621+0000 7fe036ffd700 1 --2- 172.21.15.16:0/2713741219 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7fe02406b140 0x7fe02406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:46:05.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.625+0000 7fe01f7fe700 1 -- 172.21.15.16:0/2713741219 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe030054740 con 0x7fe0380d2110 2023-10-16T13:46:05.651 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:46:05.649+0000 7f3e59442200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T13:46:05.694 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:05.692+0000 7f9d0ecc4200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T13:46:05.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.822+0000 7fe036ffd700 1 -- 172.21.15.16:0/2713741219 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7fe02406b140 msgr2=0x7fe02406d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3220139196 2023-10-16T13:46:05.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.822+0000 7fe036ffd700 1 --2- 172.21.15.16:0/2713741219 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7fe02406b140 0x7fe02406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:46:05.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.843+0000 7fe03d624700 1 -- 172.21.15.16:0/2713741219 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe0380b5110 con 0x7fe0380d2110 2023-10-16T13:46:05.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.844+0000 7fe01f7fe700 1 -- 172.21.15.16:0/2713741219 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v210) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe030057d60 con 0x7fe0380d2110 2023-10-16T13:46:05.846 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-10-16T13:46:05.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.846+0000 7fe03d624700 1 -- 172.21.15.16:0/2713741219 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7fe02406b140 msgr2=0x7fe02406d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:46:05.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.846+0000 7fe03d624700 1 --2- 172.21.15.16:0/2713741219 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7fe02406b140 0x7fe02406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:05.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.846+0000 7fe03d624700 1 -- 172.21.15.16:0/2713741219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0380d2110 msgr2=0x7fe0380700e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:05.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.846+0000 7fe03d624700 1 --2- 172.21.15.16:0/2713741219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0380d2110 0x7fe0380700e0 secure :-1 s=READY pgs=2361 cs=0 l=1 rev1=1 crypto rx=0x7fe03000cc80 tx=0x7fe03000cf90 comp rx=0 tx=0).stop 2023-10-16T13:46:05.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.846+0000 7fe03d624700 1 -- 172.21.15.16:0/2713741219 shutdown_connections 2023-10-16T13:46:05.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.846+0000 7fe03d624700 1 --2- 172.21.15.16:0/2713741219 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7fe02406b140 0x7fe02406d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:05.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.846+0000 7fe03d624700 1 --2- 172.21.15.16:0/2713741219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0380d2110 0x7fe0380700e0 unknown :-1 s=CLOSED pgs=2361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:05.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.846+0000 7fe03d624700 1 --2- 172.21.15.16:0/2713741219 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe0380b2620 0x7fe0381f5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:05.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.846+0000 7fe03d624700 1 --2- 172.21.15.16:0/2713741219 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe0380ce6a0 0x7fe0380c5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:05.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.846+0000 7fe03d624700 1 -- 172.21.15.16:0/2713741219 >> 172.21.15.16:0/2713741219 conn(0x7fe0380d15f0 msgr2=0x7fe0380cd0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:05.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.846+0000 7fe03d624700 1 -- 172.21.15.16:0/2713741219 shutdown_connections 2023-10-16T13:46:05.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:05.846+0000 7fe03d624700 1 -- 172.21.15.16:0/2713741219 wait complete. 2023-10-16T13:46:05.862 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T13:46:06.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.029+0000 7f879d3e1700 1 Processor -- start 2023-10-16T13:46:06.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.029+0000 7f879d3e1700 1 -- start start 2023-10-16T13:46:06.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.030+0000 7f879d3e1700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f87980ce5d0 0x7f87980cc190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:06.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.030+0000 7f879d3e1700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f879815f060 0x7f87980cc6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:06.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.030+0000 7f879d3e1700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87980b2440 con 0x7f879815ec80 2023-10-16T13:46:06.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.030+0000 7f879d3e1700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87980b2c50 con 0x7f879815f060 2023-10-16T13:46:06.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.030+0000 7f879d3e1700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87980d00e0 con 0x7f87980ce5d0 2023-10-16T13:46:06.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.030+0000 7f87977fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f879815f060 0x7f87980cc6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:06.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.030+0000 7f87977fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f879815f060 0x7f87980cc6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52450/0 (socket says 172.21.15.16:52450) 2023-10-16T13:46:06.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.030+0000 7f87977fe700 1 -- 172.21.15.16:0/1785102630 learned_addr learned my addr 172.21.15.16:0/1785102630 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:06.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.030+0000 7f87967fc700 1 --2- 172.21.15.16:0/1785102630 >> v2:172.21.15.49:3300/0 conn(0x7f87980ce5d0 0x7f87980cc190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:06.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.031+0000 7f87977fe700 1 -- 172.21.15.16:0/1785102630 >> v2:172.21.15.49:3300/0 conn(0x7f87980ce5d0 msgr2=0x7f87980cc190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:06.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.031+0000 7f87977fe700 1 --2- 172.21.15.16:0/1785102630 >> v2:172.21.15.49:3300/0 conn(0x7f87980ce5d0 0x7f87980cc190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:06.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.031+0000 7f87977fe700 1 -- 172.21.15.16:0/1785102630 >> v1:172.21.15.49:6789/0 conn(0x7f879815ec80 legacy=0x7f87980cdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:06.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.031+0000 7f87977fe700 1 -- 172.21.15.16:0/1785102630 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87980cef60 con 0x7f879815f060 2023-10-16T13:46:06.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.031+0000 7f87967fc700 1 --2- 172.21.15.16:0/1785102630 >> v2:172.21.15.49:3300/0 conn(0x7f87980ce5d0 0x7f87980cc190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:46:06.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.031+0000 7f87977fe700 1 --2- 172.21.15.16:0/1785102630 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f879815f060 0x7f87980cc6d0 secure :-1 s=READY pgs=2362 cs=0 l=1 rev1=1 crypto rx=0x7f8790009e50 tx=0x7f879000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd632adafd9b43c2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:06.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.031+0000 7f8795ffb700 1 -- 172.21.15.16:0/1785102630 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f879000e070 con 0x7f879815f060 2023-10-16T13:46:06.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.031+0000 7f8795ffb700 1 -- 172.21.15.16:0/1785102630 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f879000b8d0 con 0x7f879815f060 2023-10-16T13:46:06.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.031+0000 7f8795ffb700 1 -- 172.21.15.16:0/1785102630 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f879000bc80 con 0x7f879815f060 2023-10-16T13:46:06.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.032+0000 7f879d3e1700 1 -- 172.21.15.16:0/1785102630 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f879815f060 msgr2=0x7f87980cc6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:06.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.032+0000 7f879d3e1700 1 --2- 172.21.15.16:0/1785102630 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f879815f060 0x7f87980cc6d0 secure :-1 s=READY pgs=2362 cs=0 l=1 rev1=1 crypto rx=0x7f8790009e50 tx=0x7f879000b330 comp rx=0 tx=0).stop 2023-10-16T13:46:06.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.032+0000 7f879d3e1700 1 -- 172.21.15.16:0/1785102630 shutdown_connections 2023-10-16T13:46:06.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.032+0000 7f879d3e1700 1 --2- 172.21.15.16:0/1785102630 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f879815f060 0x7f87980cc6d0 unknown :-1 s=CLOSED pgs=2362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:06.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.032+0000 7f879d3e1700 1 --2- 172.21.15.16:0/1785102630 >> v2:172.21.15.49:3300/0 conn(0x7f87980ce5d0 0x7f87980cc190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:06.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.032+0000 7f879d3e1700 1 -- 172.21.15.16:0/1785102630 >> 172.21.15.16:0/1785102630 conn(0x7f87980d15f0 msgr2=0x7f87980d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:06.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.032+0000 7f879d3e1700 1 -- 172.21.15.16:0/1785102630 shutdown_connections 2023-10-16T13:46:06.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.032+0000 7f879d3e1700 1 -- 172.21.15.16:0/1785102630 wait complete. 2023-10-16T13:46:06.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.033+0000 7f879d3e1700 1 Processor -- start 2023-10-16T13:46:06.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.033+0000 7f879d3e1700 1 -- start start 2023-10-16T13:46:06.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.034+0000 7f879d3e1700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f87980ce5d0 0x7f8798151b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:06.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.034+0000 7f879d3e1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f879815ec80 0x7f87980c86c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:06.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.034+0000 7f879d3e1700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f879815f060 0x7f87981e5750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:06.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.034+0000 7f879d3e1700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87980c9e00 con 0x7f879815ec80 2023-10-16T13:46:06.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.034+0000 7f879d3e1700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87981520d0 con 0x7f879815f060 2023-10-16T13:46:06.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.034+0000 7f879d3e1700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87980c9180 con 0x7f87980ce5d0 2023-10-16T13:46:06.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.034+0000 7f87967fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f879815ec80 0x7f87980c86c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:06.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.034+0000 7f87967fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f879815ec80 0x7f87980c86c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52454/0 (socket says 172.21.15.16:52454) 2023-10-16T13:46:06.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.034+0000 7f87967fc700 1 -- 172.21.15.16:0/2917827389 learned_addr learned my addr 172.21.15.16:0/2917827389 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:06.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.034+0000 7f87977fe700 1 --2- 172.21.15.16:0/2917827389 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f879815f060 0x7f87981e5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:06.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.034+0000 7f8796ffd700 1 --2- 172.21.15.16:0/2917827389 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f87980ce5d0 0x7f8798151b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:06.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.034+0000 7f87967fc700 1 -- 172.21.15.16:0/2917827389 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f87980ce5d0 msgr2=0x7f8798151b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:06.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.034+0000 7f87967fc700 1 --2- 172.21.15.16:0/2917827389 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f87980ce5d0 0x7f8798151b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:06.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.034+0000 7f87967fc700 1 -- 172.21.15.16:0/2917827389 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f879815f060 msgr2=0x7f87981e5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:06.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.034+0000 7f87967fc700 1 --2- 172.21.15.16:0/2917827389 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f879815f060 0x7f87981e5750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:06.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.035+0000 7f87967fc700 1 -- 172.21.15.16:0/2917827389 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f878c0094a0 con 0x7f879815ec80 2023-10-16T13:46:06.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.035+0000 7f87967fc700 1 --2- 172.21.15.16:0/2917827389 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f879815ec80 0x7f87980c86c0 secure :-1 s=READY pgs=2363 cs=0 l=1 rev1=1 crypto rx=0x7f878c00ebe0 tx=0x7f878c00eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:06.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.035+0000 7f877f7fe700 1 -- 172.21.15.16:0/2917827389 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f878c00c850 con 0x7f879815ec80 2023-10-16T13:46:06.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.035+0000 7f877f7fe700 1 -- 172.21.15.16:0/2917827389 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f878c00c9b0 con 0x7f879815ec80 2023-10-16T13:46:06.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.035+0000 7f879d3e1700 1 -- 172.21.15.16:0/2917827389 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8790009b30 con 0x7f879815ec80 2023-10-16T13:46:06.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.035+0000 7f877f7fe700 1 -- 172.21.15.16:0/2917827389 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f878c00cd40 con 0x7f879815ec80 2023-10-16T13:46:06.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.035+0000 7f879d3e1700 1 -- 172.21.15.16:0/2917827389 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87980c9740 con 0x7f879815ec80 2023-10-16T13:46:06.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.036+0000 7f879d3e1700 1 -- 172.21.15.16:0/2917827389 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87980c9f40 con 0x7f879815ec80 2023-10-16T13:46:06.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.037+0000 7f877f7fe700 1 -- 172.21.15.16:0/2917827389 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 134) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f878c013470 con 0x7f879815ec80 2023-10-16T13:46:06.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.037+0000 7f877f7fe700 1 --2- 172.21.15.16:0/2917827389 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f878406b140 0x7f878406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:06.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.037+0000 7f8796ffd700 1 -- 172.21.15.16:0/2917827389 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f878406b140 msgr2=0x7f878406d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3220139196 2023-10-16T13:46:06.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.037+0000 7f877f7fe700 1 -- 172.21.15.16:0/2917827389 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 6588+0+0 (secure 0 0 0) 0x7f878c089df0 con 0x7f879815ec80 2023-10-16T13:46:06.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.037+0000 7f8796ffd700 1 --2- 172.21.15.16:0/2917827389 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f878406b140 0x7f878406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:46:06.049 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:46:06.039+0000 7f3e59442200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T13:46:06.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.042+0000 7f877f7fe700 1 -- 172.21.15.16:0/2917827389 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f87980c9f40 con 0x7f879815ec80 2023-10-16T13:46:06.093 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:06.091+0000 7f9d0ecc4200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T13:46:06.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.238+0000 7f8796ffd700 1 -- 172.21.15.16:0/2917827389 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f878406b140 msgr2=0x7f878406d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3220139196 2023-10-16T13:46:06.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.238+0000 7f8796ffd700 1 --2- 172.21.15.16:0/2917827389 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f878406b140 0x7f878406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:46:06.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.245+0000 7f879d3e1700 1 -- 172.21.15.16:0/2917827389 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f87980c9f40 con 0x7f879815ec80 2023-10-16T13:46:06.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.245+0000 7f877f7fe700 1 -- 172.21.15.16:0/2917827389 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v210) v1 ==== 100+0+11 (secure 0 0 0) 0x7f87980c9f40 con 0x7f879815ec80 2023-10-16T13:46:06.248 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-10-16T13:46:06.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.247+0000 7f879d3e1700 1 -- 172.21.15.16:0/2917827389 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f878406b140 msgr2=0x7f878406d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:46:06.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.247+0000 7f879d3e1700 1 --2- 172.21.15.16:0/2917827389 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f878406b140 0x7f878406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:06.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.248+0000 7f879d3e1700 1 -- 172.21.15.16:0/2917827389 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f879815ec80 msgr2=0x7f87980c86c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:06.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.248+0000 7f879d3e1700 1 --2- 172.21.15.16:0/2917827389 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f879815ec80 0x7f87980c86c0 secure :-1 s=READY pgs=2363 cs=0 l=1 rev1=1 crypto rx=0x7f878c00ebe0 tx=0x7f878c00eef0 comp rx=0 tx=0).stop 2023-10-16T13:46:06.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.248+0000 7f879d3e1700 1 -- 172.21.15.16:0/2917827389 shutdown_connections 2023-10-16T13:46:06.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.248+0000 7f879d3e1700 1 --2- 172.21.15.16:0/2917827389 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f879815f060 0x7f87981e5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:06.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.248+0000 7f879d3e1700 1 --2- 172.21.15.16:0/2917827389 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f87980ce5d0 0x7f8798151b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:06.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.248+0000 7f879d3e1700 1 --2- 172.21.15.16:0/2917827389 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f878406b140 0x7f878406d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:06.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.248+0000 7f879d3e1700 1 --2- 172.21.15.16:0/2917827389 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f879815ec80 0x7f87980c86c0 unknown :-1 s=CLOSED pgs=2363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:06.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.248+0000 7f879d3e1700 1 -- 172.21.15.16:0/2917827389 >> 172.21.15.16:0/2917827389 conn(0x7f87980d15f0 msgr2=0x7f8798153a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:06.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.248+0000 7f879d3e1700 1 -- 172.21.15.16:0/2917827389 shutdown_connections 2023-10-16T13:46:06.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.248+0000 7f879d3e1700 1 -- 172.21.15.16:0/2917827389 wait complete. 2023-10-16T13:46:06.263 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T13:46:06.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.389+0000 7f355d5fc700 1 Processor -- start 2023-10-16T13:46:06.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.389+0000 7f355d5fc700 1 -- start start 2023-10-16T13:46:06.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.390+0000 7f355d5fc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f355815db40 0x7f35580cb980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:06.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.390+0000 7f355d5fc700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f35580ce870 0x7f355806ac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:06.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.390+0000 7f355d5fc700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35580cedb0 con 0x7f355806a7e0 2023-10-16T13:46:06.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.390+0000 7f355d5fc700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f355806b1a0 con 0x7f355815db40 2023-10-16T13:46:06.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.390+0000 7f355d5fc700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f355815df20 con 0x7f35580ce870 2023-10-16T13:46:06.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.390+0000 7f3556ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f355815db40 0x7f35580cb980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:06.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.390+0000 7f3556ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f355815db40 0x7f35580cb980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52462/0 (socket says 172.21.15.16:52462) 2023-10-16T13:46:06.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.390+0000 7f3556ffd700 1 -- 172.21.15.16:0/1942548432 learned_addr learned my addr 172.21.15.16:0/1942548432 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:06.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.390+0000 7f35577fe700 1 --2- 172.21.15.16:0/1942548432 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f35580ce870 0x7f355806ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:06.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.390+0000 7f3556ffd700 1 -- 172.21.15.16:0/1942548432 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f35580ce870 msgr2=0x7f355806ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:06.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.390+0000 7f3556ffd700 1 --2- 172.21.15.16:0/1942548432 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f35580ce870 0x7f355806ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:06.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.390+0000 7f3556ffd700 1 -- 172.21.15.16:0/1942548432 >> v1:172.21.15.49:6789/0 conn(0x7f355806a7e0 legacy=0x7f35580ce160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:06.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.390+0000 7f3556ffd700 1 -- 172.21.15.16:0/1942548432 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35580d12c0 con 0x7f355815db40 2023-10-16T13:46:06.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.390+0000 7f3556ffd700 1 --2- 172.21.15.16:0/1942548432 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f355815db40 0x7f35580cb980 secure :-1 s=READY pgs=2364 cs=0 l=1 rev1=1 crypto rx=0x7f35480098f0 tx=0x7f3548009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bb9f0086995831d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:06.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.390+0000 7f3555ffb700 1 -- 172.21.15.16:0/1942548432 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3548011830 con 0x7f355815db40 2023-10-16T13:46:06.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.391+0000 7f3555ffb700 1 -- 172.21.15.16:0/1942548432 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3548011990 con 0x7f355815db40 2023-10-16T13:46:06.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.391+0000 7f3555ffb700 1 -- 172.21.15.16:0/1942548432 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3548011d20 con 0x7f355815db40 2023-10-16T13:46:06.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.391+0000 7f355d5fc700 1 -- 172.21.15.16:0/1942548432 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f355815db40 msgr2=0x7f35580cb980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:06.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.391+0000 7f355d5fc700 1 --2- 172.21.15.16:0/1942548432 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f355815db40 0x7f35580cb980 secure :-1 s=READY pgs=2364 cs=0 l=1 rev1=1 crypto rx=0x7f35480098f0 tx=0x7f3548009c00 comp rx=0 tx=0).stop 2023-10-16T13:46:06.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.391+0000 7f355d5fc700 1 -- 172.21.15.16:0/1942548432 shutdown_connections 2023-10-16T13:46:06.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.391+0000 7f355d5fc700 1 --2- 172.21.15.16:0/1942548432 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f35580ce870 0x7f355806ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:06.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.391+0000 7f355d5fc700 1 --2- 172.21.15.16:0/1942548432 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f355815db40 0x7f35580cb980 unknown :-1 s=CLOSED pgs=2364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:06.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.391+0000 7f355d5fc700 1 -- 172.21.15.16:0/1942548432 >> 172.21.15.16:0/1942548432 conn(0x7f35580b1fd0 msgr2=0x7f35580b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:06.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.391+0000 7f355d5fc700 1 -- 172.21.15.16:0/1942548432 shutdown_connections 2023-10-16T13:46:06.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.391+0000 7f355d5fc700 1 -- 172.21.15.16:0/1942548432 wait complete. 2023-10-16T13:46:06.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.393+0000 7f355d5fc700 1 Processor -- start 2023-10-16T13:46:06.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.393+0000 7f355d5fc700 1 -- start start 2023-10-16T13:46:06.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.393+0000 7f355d5fc700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f355806a7e0 0x7f35580c93b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:06.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.393+0000 7f355d5fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35580ce870 0x7f35580c98f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:06.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.393+0000 7f355d5fc700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f355815db40 0x7f35580c7510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:06.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.393+0000 7f355d5fc700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35580c8d00 con 0x7f35580ce870 2023-10-16T13:46:06.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.393+0000 7f355d5fc700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3558153f90 con 0x7f355815db40 2023-10-16T13:46:06.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.393+0000 7f355d5fc700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35580c80f0 con 0x7f355806a7e0 2023-10-16T13:46:06.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.394+0000 7f35567fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35580ce870 0x7f35580c98f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:06.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.394+0000 7f35567fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35580ce870 0x7f35580c98f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52470/0 (socket says 172.21.15.16:52470) 2023-10-16T13:46:06.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.394+0000 7f35567fc700 1 -- 172.21.15.16:0/2710858995 learned_addr learned my addr 172.21.15.16:0/2710858995 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:06.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.394+0000 7f35577fe700 1 --2- 172.21.15.16:0/2710858995 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f355815db40 0x7f35580c7510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:06.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.394+0000 7f3556ffd700 1 --2- 172.21.15.16:0/2710858995 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f355806a7e0 0x7f35580c93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:06.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.394+0000 7f35567fc700 1 -- 172.21.15.16:0/2710858995 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f355806a7e0 msgr2=0x7f35580c93b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:06.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.394+0000 7f35567fc700 1 --2- 172.21.15.16:0/2710858995 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f355806a7e0 0x7f35580c93b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:06.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.394+0000 7f35567fc700 1 -- 172.21.15.16:0/2710858995 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f355815db40 msgr2=0x7f35580c7510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:06.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.394+0000 7f35567fc700 1 --2- 172.21.15.16:0/2710858995 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f355815db40 0x7f35580c7510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:06.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.394+0000 7f35567fc700 1 -- 172.21.15.16:0/2710858995 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3548009550 con 0x7f35580ce870 2023-10-16T13:46:06.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.394+0000 7f35567fc700 1 --2- 172.21.15.16:0/2710858995 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35580ce870 0x7f35580c98f0 secure :-1 s=READY pgs=2365 cs=0 l=1 rev1=1 crypto rx=0x7f354c00c450 tx=0x7f354c00c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:06.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.394+0000 7f353f7fe700 1 -- 172.21.15.16:0/2710858995 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f354c0052d0 con 0x7f35580ce870 2023-10-16T13:46:06.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.394+0000 7f353f7fe700 1 -- 172.21.15.16:0/2710858995 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f354c005430 con 0x7f35580ce870 2023-10-16T13:46:06.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.394+0000 7f355d5fc700 1 -- 172.21.15.16:0/2710858995 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35580c83d0 con 0x7f35580ce870 2023-10-16T13:46:06.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.394+0000 7f353f7fe700 1 -- 172.21.15.16:0/2710858995 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f354c0038c0 con 0x7f35580ce870 2023-10-16T13:46:06.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.394+0000 7f355d5fc700 1 -- 172.21.15.16:0/2710858995 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35580c8990 con 0x7f35580ce870 2023-10-16T13:46:06.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.396+0000 7f355d5fc700 1 -- 172.21.15.16:0/2710858995 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35580c8b20 con 0x7f35580ce870 2023-10-16T13:46:06.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.396+0000 7f353f7fe700 1 -- 172.21.15.16:0/2710858995 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 134) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f354c003a20 con 0x7f35580ce870 2023-10-16T13:46:06.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.396+0000 7f353f7fe700 1 --2- 172.21.15.16:0/2710858995 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f354406b140 0x7f354406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:06.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.396+0000 7f3556ffd700 1 -- 172.21.15.16:0/2710858995 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f354406b140 msgr2=0x7f354406d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3220139196 2023-10-16T13:46:06.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.396+0000 7f353f7fe700 1 -- 172.21.15.16:0/2710858995 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 6588+0+0 (secure 0 0 0) 0x7f354c088fd0 con 0x7f35580ce870 2023-10-16T13:46:06.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.396+0000 7f3556ffd700 1 --2- 172.21.15.16:0/2710858995 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f354406b140 0x7f354406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:46:06.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.401+0000 7f353f7fe700 1 -- 172.21.15.16:0/2710858995 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f354c053a80 con 0x7f35580ce870 2023-10-16T13:46:06.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.597+0000 7f3556ffd700 1 -- 172.21.15.16:0/2710858995 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f354406b140 msgr2=0x7f354406d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3220139196 2023-10-16T13:46:06.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.597+0000 7f3556ffd700 1 --2- 172.21.15.16:0/2710858995 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f354406b140 0x7f354406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:46:06.653 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:46:06.651+0000 7f3e59442200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T13:46:06.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.699+0000 7f355d5fc700 1 -- 172.21.15.16:0/2710858995 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f355804ebb0 con 0x7f35580ce870 2023-10-16T13:46:06.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.699+0000 7f353f7fe700 1 -- 172.21.15.16:0/2710858995 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v211) v1 ==== 80+0+470 (secure 0 0 0) 0x7f354c0171f0 con 0x7f35580ce870 2023-10-16T13:46:06.702 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3477190137 2023-10-17T13:44:55.326924+0000 2023-10-16T13:46:06.702 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/1906189720 2023-10-17T13:44:55.326924+0000 2023-10-16T13:46:06.703 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6833/1268439414 2023-10-17T13:44:55.326924+0000 2023-10-16T13:46:06.703 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6832/1268439414 2023-10-17T13:44:55.326924+0000 2023-10-16T13:46:06.703 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3697646183 2023-10-17T13:44:55.326924+0000 2023-10-16T13:46:06.703 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/4067982926 2023-10-17T13:44:55.326924+0000 2023-10-16T13:46:06.703 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3408848304 2023-10-17T13:44:55.326924+0000 2023-10-16T13:46:06.703 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2782518306 2023-10-17T13:44:55.326924+0000 2023-10-16T13:46:06.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.702+0000 7f355d5fc700 1 -- 172.21.15.16:0/2710858995 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f354406b140 msgr2=0x7f354406d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:46:06.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.702+0000 7f355d5fc700 1 --2- 172.21.15.16:0/2710858995 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f354406b140 0x7f354406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:06.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.702+0000 7f355d5fc700 1 -- 172.21.15.16:0/2710858995 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35580ce870 msgr2=0x7f35580c98f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:06.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.702+0000 7f355d5fc700 1 --2- 172.21.15.16:0/2710858995 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35580ce870 0x7f35580c98f0 secure :-1 s=READY pgs=2365 cs=0 l=1 rev1=1 crypto rx=0x7f354c00c450 tx=0x7f354c00c760 comp rx=0 tx=0).stop 2023-10-16T13:46:06.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.702+0000 7f355d5fc700 1 -- 172.21.15.16:0/2710858995 shutdown_connections 2023-10-16T13:46:06.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.702+0000 7f355d5fc700 1 --2- 172.21.15.16:0/2710858995 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f355815db40 0x7f35580c7510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:06.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.702+0000 7f355d5fc700 1 --2- 172.21.15.16:0/2710858995 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f355806a7e0 0x7f35580c93b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:06.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.702+0000 7f355d5fc700 1 --2- 172.21.15.16:0/2710858995 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f354406b140 0x7f354406d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:06.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.702+0000 7f355d5fc700 1 --2- 172.21.15.16:0/2710858995 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35580ce870 0x7f35580c98f0 unknown :-1 s=CLOSED pgs=2365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:06.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.702+0000 7f355d5fc700 1 -- 172.21.15.16:0/2710858995 >> 172.21.15.16:0/2710858995 conn(0x7f35580b1fd0 msgr2=0x7f355815bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:06.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.702+0000 7f355d5fc700 1 -- 172.21.15.16:0/2710858995 shutdown_connections 2023-10-16T13:46:06.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:06.702+0000 7f355d5fc700 1 -- 172.21.15.16:0/2710858995 wait complete. 2023-10-16T13:46:06.707 INFO:teuthology.orchestra.run.smithi016.stderr:listed 8 entries 2023-10-16T13:46:06.718 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-16T13:46:06.719 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:46:06.719 DEBUG:teuthology.orchestra.run.smithi016:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-16T13:46:06.727 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:46:06.726+0000 7f3e59442200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T13:46:06.734 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:06.732+0000 7f9d0ecc4200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T13:46:06.758 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2023-10-16T13:46:06.758 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:46:06.759 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-16T13:46:06.812 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:06.810+0000 7f9d0ecc4200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T13:46:06.815 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T13:46:06.815 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:46:06.833 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284204k,nr_inodes=4071051,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=753 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260540k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T13:46:06.834 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T13:46:06.834 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:46:06.873 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16287364k,nr_inodes=4071841,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20251 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3261172k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T13:46:06.873 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T13:46:06.874 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:46:06.892 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284204k,nr_inodes=4071051,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=753 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260540k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T13:46:06.892 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:46:07.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.037+0000 7f4eaf767700 1 Processor -- start 2023-10-16T13:46:07.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.037+0000 7f4eaf767700 1 -- start start 2023-10-16T13:46:07.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.038+0000 7f4eaf767700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4ea815d900 0x7f4ea80d2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:07.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.038+0000 7f4eaf767700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f4ea80d2f70 0x7f4ea80cdc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:07.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.038+0000 7f4eaf767700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ea80ce1a0 con 0x7f4ea80ce9f0 2023-10-16T13:46:07.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.038+0000 7f4eaf767700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ea80cee70 con 0x7f4ea815d900 2023-10-16T13:46:07.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.038+0000 7f4eaf767700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ea80ce400 con 0x7f4ea80d2f70 2023-10-16T13:46:07.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.038+0000 7f4ead503700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4ea815d900 0x7f4ea80d2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:07.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.038+0000 7f4ead503700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4ea815d900 0x7f4ea80d2040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52478/0 (socket says 172.21.15.16:52478) 2023-10-16T13:46:07.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.038+0000 7f4ead503700 1 -- 172.21.15.16:0/3977556866 learned_addr learned my addr 172.21.15.16:0/3977556866 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:07.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.038+0000 7f4eadd04700 1 --2- 172.21.15.16:0/3977556866 >> v2:172.21.15.49:3300/0 conn(0x7f4ea80d2f70 0x7f4ea80cdc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:07.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.039+0000 7f4ead503700 1 -- 172.21.15.16:0/3977556866 >> v2:172.21.15.49:3300/0 conn(0x7f4ea80d2f70 msgr2=0x7f4ea80cdc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:07.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.039+0000 7f4ead503700 1 --2- 172.21.15.16:0/3977556866 >> v2:172.21.15.49:3300/0 conn(0x7f4ea80d2f70 0x7f4ea80cdc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:07.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.039+0000 7f4ead503700 1 -- 172.21.15.16:0/3977556866 >> v1:172.21.15.49:6789/0 conn(0x7f4ea80ce9f0 legacy=0x7f4ea80d2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:07.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.039+0000 7f4ead503700 1 -- 172.21.15.16:0/3977556866 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ea815dce0 con 0x7f4ea815d900 2023-10-16T13:46:07.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.039+0000 7f4ead503700 1 --2- 172.21.15.16:0/3977556866 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ea815d900 0x7f4ea80d2040 secure :-1 s=READY pgs=2366 cs=0 l=1 rev1=1 crypto rx=0x7f4ea80b30b0 tx=0x7f4ea4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c40a66e9fdbfc41 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:07.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.039+0000 7f4e9ffff700 1 -- 172.21.15.16:0/3977556866 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea4011830 con 0x7f4ea815d900 2023-10-16T13:46:07.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.039+0000 7f4e9ffff700 1 -- 172.21.15.16:0/3977556866 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ea4011990 con 0x7f4ea815d900 2023-10-16T13:46:07.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.039+0000 7f4e9ffff700 1 -- 172.21.15.16:0/3977556866 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea4011d20 con 0x7f4ea815d900 2023-10-16T13:46:07.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.040+0000 7f4eaf767700 1 -- 172.21.15.16:0/3977556866 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ea815d900 msgr2=0x7f4ea80d2040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:07.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.040+0000 7f4eaf767700 1 --2- 172.21.15.16:0/3977556866 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ea815d900 0x7f4ea80d2040 secure :-1 s=READY pgs=2366 cs=0 l=1 rev1=1 crypto rx=0x7f4ea80b30b0 tx=0x7f4ea4009bd0 comp rx=0 tx=0).stop 2023-10-16T13:46:07.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.040+0000 7f4eaf767700 1 -- 172.21.15.16:0/3977556866 shutdown_connections 2023-10-16T13:46:07.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.040+0000 7f4eaf767700 1 --2- 172.21.15.16:0/3977556866 >> v2:172.21.15.49:3300/0 conn(0x7f4ea80d2f70 0x7f4ea80cdc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:07.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.040+0000 7f4eaf767700 1 --2- 172.21.15.16:0/3977556866 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ea815d900 0x7f4ea80d2040 unknown :-1 s=CLOSED pgs=2366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:07.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.040+0000 7f4eaf767700 1 -- 172.21.15.16:0/3977556866 >> 172.21.15.16:0/3977556866 conn(0x7f4ea8072330 msgr2=0x7f4ea8072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:07.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.040+0000 7f4eaf767700 1 -- 172.21.15.16:0/3977556866 shutdown_connections 2023-10-16T13:46:07.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.040+0000 7f4eaf767700 1 -- 172.21.15.16:0/3977556866 wait complete. 2023-10-16T13:46:07.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.042+0000 7f4eaf767700 1 Processor -- start 2023-10-16T13:46:07.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.043+0000 7f4eaf767700 1 -- start start 2023-10-16T13:46:07.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.044+0000 7f4eaf767700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4ea80ce9f0 0x7f4ea80b7130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:07.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.044+0000 7f4eaf767700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ea80d2f70 0x7f4ea814ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:07.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.044+0000 7f4eaf767700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4ea815d900 0x7f4ea80cb910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:07.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.044+0000 7f4eaf767700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea81f85c0 con 0x7f4ea80d2f70 2023-10-16T13:46:07.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.044+0000 7f4eaf767700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea80b8250 con 0x7f4ea815d900 2023-10-16T13:46:07.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.044+0000 7f4eaf767700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea80b7670 con 0x7f4ea80ce9f0 2023-10-16T13:46:07.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.044+0000 7f4ead503700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4ea80ce9f0 0x7f4ea80b7130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:07.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.044+0000 7f4eacd02700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ea80d2f70 0x7f4ea814ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:07.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.044+0000 7f4ead503700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4ea80ce9f0 0x7f4ea80b7130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:40742/0 (socket says 172.21.15.16:40742) 2023-10-16T13:46:07.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.044+0000 7f4ead503700 1 -- 172.21.15.16:0/2343184629 learned_addr learned my addr 172.21.15.16:0/2343184629 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:07.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.044+0000 7f4eadd04700 1 --2- 172.21.15.16:0/2343184629 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4ea815d900 0x7f4ea80cb910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:07.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.044+0000 7f4ead503700 1 -- 172.21.15.16:0/2343184629 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4ea815d900 msgr2=0x7f4ea80cb910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:07.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.044+0000 7f4ead503700 1 --2- 172.21.15.16:0/2343184629 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4ea815d900 0x7f4ea80cb910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:07.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.044+0000 7f4ead503700 1 -- 172.21.15.16:0/2343184629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ea80d2f70 msgr2=0x7f4ea814ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:07.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.044+0000 7f4ead503700 1 --2- 172.21.15.16:0/2343184629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ea80d2f70 0x7f4ea814ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:07.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.044+0000 7f4ead503700 1 -- 172.21.15.16:0/2343184629 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ea4009550 con 0x7f4ea80ce9f0 2023-10-16T13:46:07.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.044+0000 7f4eacd02700 1 --2- 172.21.15.16:0/2343184629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ea80d2f70 0x7f4ea814ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:46:07.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.045+0000 7f4ead503700 1 --2- 172.21.15.16:0/2343184629 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4ea80ce9f0 0x7f4ea80b7130 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f4ea4009c10 tx=0x7f4ea4004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:07.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.045+0000 7f4e9dffb700 1 -- 172.21.15.16:0/2343184629 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea4004380 con 0x7f4ea80ce9f0 2023-10-16T13:46:07.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.045+0000 7f4e9dffb700 1 -- 172.21.15.16:0/2343184629 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ea401f070 con 0x7f4ea80ce9f0 2023-10-16T13:46:07.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.045+0000 7f4eaf767700 1 -- 172.21.15.16:0/2343184629 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ea80b9270 con 0x7f4ea80ce9f0 2023-10-16T13:46:07.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.045+0000 7f4e9dffb700 1 -- 172.21.15.16:0/2343184629 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea40036a0 con 0x7f4ea80ce9f0 2023-10-16T13:46:07.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.046+0000 7f4eaf767700 1 -- 172.21.15.16:0/2343184629 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ea80b3460 con 0x7f4ea80ce9f0 2023-10-16T13:46:07.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.047+0000 7f4eaf767700 1 -- 172.21.15.16:0/2343184629 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ea80d3450 con 0x7f4ea80ce9f0 2023-10-16T13:46:07.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.048+0000 7f4e9dffb700 1 -- 172.21.15.16:0/2343184629 <== mon.2 v2:172.21.15.49:3301/0 4 ==== mgrmap(e 134) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f4ea4021030 con 0x7f4ea80ce9f0 2023-10-16T13:46:07.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.049+0000 7f4e9dffb700 1 --2- 172.21.15.16:0/2343184629 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f4e8c06b190 0x7f4e8c06d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:07.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.049+0000 7f4eacd02700 1 -- 172.21.15.16:0/2343184629 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f4e8c06b190 msgr2=0x7f4e8c06d650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3220139196 2023-10-16T13:46:07.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.049+0000 7f4e9dffb700 1 -- 172.21.15.16:0/2343184629 <== mon.2 v2:172.21.15.49:3301/0 5 ==== osd_map(211..211 src has 1..211) v4 ==== 6676+0+0 (secure 0 0 0) 0x7f4ea408b710 con 0x7f4ea80ce9f0 2023-10-16T13:46:07.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.049+0000 7f4eacd02700 1 --2- 172.21.15.16:0/2343184629 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f4e8c06b190 0x7f4e8c06d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:46:07.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.052+0000 7f4e9dffb700 1 -- 172.21.15.16:0/2343184629 <== mon.2 v2:172.21.15.49:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4ea4056190 con 0x7f4ea80ce9f0 2023-10-16T13:46:07.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.250+0000 7f4eacd02700 1 -- 172.21.15.16:0/2343184629 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f4e8c06b190 msgr2=0x7f4e8c06d650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3220139196 2023-10-16T13:46:07.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.250+0000 7f4eacd02700 1 --2- 172.21.15.16:0/2343184629 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f4e8c06b190 0x7f4e8c06d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:46:07.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.310+0000 7f4eaf767700 1 -- 172.21.15.16:0/2343184629 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4ea804ebb0 con 0x7f4ea80ce9f0 2023-10-16T13:46:07.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.311+0000 7f4e9dffb700 1 -- 172.21.15.16:0/2343184629 <== mon.2 v2:172.21.15.49:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+4925 (secure 0 0 0) 0x7f4ea4015020 con 0x7f4ea80ce9f0 2023-10-16T13:46:07.313 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:46:07.313 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":104,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8729,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/3614602930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":3614602930},{"type":"v1","addr":"172.21.15.49:6835","nonce":3614602930}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":99},{"gid":10045,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/3427940027","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":3427940027},{"type":"v1","addr":"172.21.15.49:6834","nonce":3427940027}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":99},{"gid":10896,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/4279896386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":4279896386},{"type":"v1","addr":"172.21.15.16:6836","nonce":4279896386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:44:40.208202+0000","modified":"2023-10-16T13:44:43.255450+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10899},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10899":{"gid":10899,"name":"a","rank":0,"incarnation":100,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/4233835071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":4233835071},{"type":"v1","addr":"172.21.15.16:6837","nonce":4233835071}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2023-10-16T13:46:07.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.313+0000 7f4eaf767700 1 -- 172.21.15.16:0/2343184629 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f4e8c06b190 msgr2=0x7f4e8c06d650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:46:07.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.313+0000 7f4eaf767700 1 --2- 172.21.15.16:0/2343184629 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f4e8c06b190 0x7f4e8c06d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:07.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.313+0000 7f4eaf767700 1 -- 172.21.15.16:0/2343184629 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4ea80ce9f0 msgr2=0x7f4ea80b7130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:07.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.313+0000 7f4eaf767700 1 --2- 172.21.15.16:0/2343184629 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4ea80ce9f0 0x7f4ea80b7130 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f4ea4009c10 tx=0x7f4ea4004750 comp rx=0 tx=0).stop 2023-10-16T13:46:07.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.313+0000 7f4eaf767700 1 -- 172.21.15.16:0/2343184629 shutdown_connections 2023-10-16T13:46:07.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.313+0000 7f4eaf767700 1 --2- 172.21.15.16:0/2343184629 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4ea815d900 0x7f4ea80cb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:07.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.313+0000 7f4eaf767700 1 --2- 172.21.15.16:0/2343184629 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4ea80ce9f0 0x7f4ea80b7130 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:07.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.313+0000 7f4eaf767700 1 --2- 172.21.15.16:0/2343184629 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f4e8c06b190 0x7f4e8c06d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:07.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.313+0000 7f4eaf767700 1 --2- 172.21.15.16:0/2343184629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ea80d2f70 0x7f4ea814ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:07.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.313+0000 7f4eaf767700 1 -- 172.21.15.16:0/2343184629 >> 172.21.15.16:0/2343184629 conn(0x7f4ea8072330 msgr2=0x7f4ea80ccb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:07.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.313+0000 7f4eaf767700 1 -- 172.21.15.16:0/2343184629 shutdown_connections 2023-10-16T13:46:07.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.313+0000 7f4eaf767700 1 -- 172.21.15.16:0/2343184629 wait complete. 2023-10-16T13:46:07.319 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 104 2023-10-16T13:46:07.329 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:46:07.403 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:46:07.402+0000 7f3e59442200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:46:07.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.455+0000 7fe0b4a9e700 1 Processor -- start 2023-10-16T13:46:07.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.455+0000 7fe0b4a9e700 1 -- start start 2023-10-16T13:46:07.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.456+0000 7fe0b4a9e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe0b00cf880 0x7fe0b00b52f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:07.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.456+0000 7fe0b4a9e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe0b00b5830 0x7fe0b015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:07.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.456+0000 7fe0b4a9e700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0b015bc80 con 0x7fe0b00cf4a0 2023-10-16T13:46:07.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.456+0000 7fe0b4a9e700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0b015b070 con 0x7fe0b00cf880 2023-10-16T13:46:07.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.456+0000 7fe0b4a9e700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0b015b350 con 0x7fe0b00b5830 2023-10-16T13:46:07.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.456+0000 7fe0add9b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe0b00cf880 0x7fe0b00b52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:07.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.456+0000 7fe0add9b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe0b00cf880 0x7fe0b00b52f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52496/0 (socket says 172.21.15.16:52496) 2023-10-16T13:46:07.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.456+0000 7fe0add9b700 1 -- 172.21.15.16:0/419270760 learned_addr learned my addr 172.21.15.16:0/419270760 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:07.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.457+0000 7fe0aed9d700 1 --2- 172.21.15.16:0/419270760 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe0b00b5830 0x7fe0b015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:07.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.457+0000 7fe0add9b700 1 -- 172.21.15.16:0/419270760 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe0b00b5830 msgr2=0x7fe0b015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:07.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.457+0000 7fe0add9b700 1 --2- 172.21.15.16:0/419270760 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe0b00b5830 0x7fe0b015a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:07.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.457+0000 7fe0add9b700 1 -- 172.21.15.16:0/419270760 >> v1:172.21.15.49:6789/0 conn(0x7fe0b00cf4a0 legacy=0x7fe0b00726e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:07.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.457+0000 7fe0ad59a700 1 -- 172.21.15.16:0/419270760 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1087200920 0 0) 0x7fe0b015bc80 con 0x7fe0b00cf4a0 2023-10-16T13:46:07.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.457+0000 7fe0add9b700 1 -- 172.21.15.16:0/419270760 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0b015b630 con 0x7fe0b00cf880 2023-10-16T13:46:07.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.457+0000 7fe0add9b700 1 --2- 172.21.15.16:0/419270760 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0b00cf880 0x7fe0b00b52f0 secure :-1 s=READY pgs=2367 cs=0 l=1 rev1=1 crypto rx=0x7fe0a400bb90 tx=0x7fe0a400bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51ffc6029643deee server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:07.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.457+0000 7fe0ad59a700 1 -- 172.21.15.16:0/419270760 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0a400dd30 con 0x7fe0b00cf880 2023-10-16T13:46:07.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.457+0000 7fe0ad59a700 1 -- 172.21.15.16:0/419270760 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0a400de90 con 0x7fe0b00cf880 2023-10-16T13:46:07.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.457+0000 7fe0ad59a700 1 -- 172.21.15.16:0/419270760 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0a4010690 con 0x7fe0b00cf880 2023-10-16T13:46:07.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.458+0000 7fe0b4a9e700 1 -- 172.21.15.16:0/419270760 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0b00cf880 msgr2=0x7fe0b00b52f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:07.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.458+0000 7fe0b4a9e700 1 --2- 172.21.15.16:0/419270760 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0b00cf880 0x7fe0b00b52f0 secure :-1 s=READY pgs=2367 cs=0 l=1 rev1=1 crypto rx=0x7fe0a400bb90 tx=0x7fe0a400bea0 comp rx=0 tx=0).stop 2023-10-16T13:46:07.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.458+0000 7fe0b4a9e700 1 -- 172.21.15.16:0/419270760 shutdown_connections 2023-10-16T13:46:07.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.458+0000 7fe0b4a9e700 1 --2- 172.21.15.16:0/419270760 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe0b00b5830 0x7fe0b015a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:07.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.458+0000 7fe0b4a9e700 1 --2- 172.21.15.16:0/419270760 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0b00cf880 0x7fe0b00b52f0 unknown :-1 s=CLOSED pgs=2367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:07.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.458+0000 7fe0b4a9e700 1 -- 172.21.15.16:0/419270760 >> 172.21.15.16:0/419270760 conn(0x7fe0b00b1fd0 msgr2=0x7fe0b00b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:07.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.458+0000 7fe0b4a9e700 1 -- 172.21.15.16:0/419270760 shutdown_connections 2023-10-16T13:46:07.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.458+0000 7fe0b4a9e700 1 -- 172.21.15.16:0/419270760 wait complete. 2023-10-16T13:46:07.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.459+0000 7fe0b4a9e700 1 Processor -- start 2023-10-16T13:46:07.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.459+0000 7fe0b4a9e700 1 -- start start 2023-10-16T13:46:07.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.460+0000 7fe0b4a9e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe0b00b5830 0x7fe0b01589d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:07.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.460+0000 7fe0b4a9e700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe0b00cf4a0 0x7fe0b00ae2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:07.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.460+0000 7fe0b4a9e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0b00cf880 0x7fe0b00ae810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:07.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.460+0000 7fe0b4a9e700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0b01f4050 con 0x7fe0b00cf880 2023-10-16T13:46:07.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.460+0000 7fe0b4a9e700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0b01f3110 con 0x7fe0b00cf4a0 2023-10-16T13:46:07.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.460+0000 7fe0b4a9e700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0b01f33d0 con 0x7fe0b00b5830 2023-10-16T13:46:07.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.460+0000 7fe0aed9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0b00cf880 0x7fe0b00ae810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:07.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.460+0000 7fe0aed9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0b00cf880 0x7fe0b00ae810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52504/0 (socket says 172.21.15.16:52504) 2023-10-16T13:46:07.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.460+0000 7fe0ae59c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe0b00b5830 0x7fe0b01589d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:07.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.460+0000 7fe0aed9d700 1 -- 172.21.15.16:0/1866301670 learned_addr learned my addr 172.21.15.16:0/1866301670 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:07.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.460+0000 7fe0add9b700 1 --2- 172.21.15.16:0/1866301670 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe0b00cf4a0 0x7fe0b00ae2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:07.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.461+0000 7fe0aed9d700 1 -- 172.21.15.16:0/1866301670 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe0b00b5830 msgr2=0x7fe0b01589d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:07.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.461+0000 7fe0aed9d700 1 --2- 172.21.15.16:0/1866301670 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe0b00b5830 0x7fe0b01589d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:07.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.461+0000 7fe0aed9d700 1 -- 172.21.15.16:0/1866301670 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe0b00cf4a0 msgr2=0x7fe0b00ae2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:07.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.461+0000 7fe0aed9d700 1 --2- 172.21.15.16:0/1866301670 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe0b00cf4a0 0x7fe0b00ae2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:07.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.461+0000 7fe0aed9d700 1 -- 172.21.15.16:0/1866301670 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0a400b7f0 con 0x7fe0b00cf880 2023-10-16T13:46:07.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.461+0000 7fe0aed9d700 1 --2- 172.21.15.16:0/1866301670 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0b00cf880 0x7fe0b00ae810 secure :-1 s=READY pgs=2368 cs=0 l=1 rev1=1 crypto rx=0x7fe0a800efc0 tx=0x7fe0a800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:07.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.461+0000 7fe09effd700 1 -- 172.21.15.16:0/1866301670 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0a8009ea0 con 0x7fe0b00cf880 2023-10-16T13:46:07.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.461+0000 7fe09effd700 1 -- 172.21.15.16:0/1866301670 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0a800f040 con 0x7fe0b00cf880 2023-10-16T13:46:07.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.461+0000 7fe0b4a9e700 1 -- 172.21.15.16:0/1866301670 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0b01f36b0 con 0x7fe0b00cf880 2023-10-16T13:46:07.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.461+0000 7fe09effd700 1 -- 172.21.15.16:0/1866301670 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0a800ced0 con 0x7fe0b00cf880 2023-10-16T13:46:07.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.461+0000 7fe0b4a9e700 1 -- 172.21.15.16:0/1866301670 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0b01f3c70 con 0x7fe0b00cf880 2023-10-16T13:46:07.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.462+0000 7fe0b4a9e700 1 -- 172.21.15.16:0/1866301670 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0b004ebb0 con 0x7fe0b00cf880 2023-10-16T13:46:07.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.463+0000 7fe09effd700 1 -- 172.21.15.16:0/1866301670 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 134) v1 ==== 89445+0+0 (secure 0 0 0) 0x7fe0a8014480 con 0x7fe0b00cf880 2023-10-16T13:46:07.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.463+0000 7fe09effd700 1 --2- 172.21.15.16:0/1866301670 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7fe09006b140 0x7fe09006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:07.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.463+0000 7fe0ae59c700 1 -- 172.21.15.16:0/1866301670 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7fe09006b140 msgr2=0x7fe09006d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3220139196 2023-10-16T13:46:07.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.463+0000 7fe09effd700 1 -- 172.21.15.16:0/1866301670 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(211..211 src has 1..211) v4 ==== 6676+0+0 (secure 0 0 0) 0x7fe0a808ae50 con 0x7fe0b00cf880 2023-10-16T13:46:07.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.463+0000 7fe0ae59c700 1 --2- 172.21.15.16:0/1866301670 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7fe09006b140 0x7fe09006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:46:07.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.471+0000 7fe09effd700 1 -- 172.21.15.16:0/1866301670 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe0a80556c0 con 0x7fe0b00cf880 2023-10-16T13:46:07.486 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:46:07.484+0000 7f3e59442200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T13:46:07.510 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:07.508+0000 7f9d0ecc4200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:46:07.596 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:07.593+0000 7f9d0ecc4200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T13:46:07.644 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:46:07.642+0000 7f3e59442200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:46:07.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.664+0000 7fe0ae59c700 1 -- 172.21.15.16:0/1866301670 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7fe09006b140 msgr2=0x7fe09006d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3220139196 2023-10-16T13:46:07.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.664+0000 7fe0ae59c700 1 --2- 172.21.15.16:0/1866301670 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7fe09006b140 0x7fe09006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:46:07.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.683+0000 7fe0b4a9e700 1 -- 172.21.15.16:0/1866301670 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe0b0159fe0 con 0x7fe0b00cf880 2023-10-16T13:46:07.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.684+0000 7fe09effd700 1 -- 172.21.15.16:0/1866301670 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v212) v1 ==== 74+0+17654 (secure 0 0 0) 0x7fe0a8018020 con 0x7fe0b00cf880 2023-10-16T13:46:07.687 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:46:07.687 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":212,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:46:07.496780+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2023-10-16T13:44:38.716950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"205","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_data","create_time":"2023-10-16T13:44:39.172499+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"212","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":212,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":204,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":204,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":204,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":204,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":204,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":204,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"26.5","mappings":[{"from":1,"to":0}]},{"pgid":"26.21","mappings":[{"from":1,"to":0}]},{"pgid":"26.24","mappings":[{"from":3,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/3477190137":"2023-10-17T13:44:55.326924+0000","172.21.15.16:0/1906189720":"2023-10-17T13:44:55.326924+0000","172.21.15.16:6833/1268439414":"2023-10-17T13:44:55.326924+0000","172.21.15.16:6832/1268439414":"2023-10-17T13:44:55.326924+0000","172.21.15.16:0/3697646183":"2023-10-17T13:44:55.326924+0000","172.21.15.16:0/4067982926":"2023-10-17T13:44:55.326924+0000","172.21.15.16:0/3408848304":"2023-10-17T13:44:55.326924+0000","172.21.15.16:0/2782518306":"2023-10-17T13:44:55.326924+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[{"pool":26,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:46:07.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.687+0000 7fe0b4a9e700 1 -- 172.21.15.16:0/1866301670 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7fe09006b140 msgr2=0x7fe09006d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:46:07.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.687+0000 7fe0b4a9e700 1 --2- 172.21.15.16:0/1866301670 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7fe09006b140 0x7fe09006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:07.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.687+0000 7fe0b4a9e700 1 -- 172.21.15.16:0/1866301670 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0b00cf880 msgr2=0x7fe0b00ae810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:07.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.687+0000 7fe0b4a9e700 1 --2- 172.21.15.16:0/1866301670 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0b00cf880 0x7fe0b00ae810 secure :-1 s=READY pgs=2368 cs=0 l=1 rev1=1 crypto rx=0x7fe0a800efc0 tx=0x7fe0a800c3a0 comp rx=0 tx=0).stop 2023-10-16T13:46:07.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.687+0000 7fe0b4a9e700 1 -- 172.21.15.16:0/1866301670 shutdown_connections 2023-10-16T13:46:07.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.687+0000 7fe0b4a9e700 1 --2- 172.21.15.16:0/1866301670 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7fe09006b140 0x7fe09006d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:07.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.687+0000 7fe0b4a9e700 1 --2- 172.21.15.16:0/1866301670 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0b00cf880 0x7fe0b00ae810 unknown :-1 s=CLOSED pgs=2368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:07.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.687+0000 7fe0b4a9e700 1 --2- 172.21.15.16:0/1866301670 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe0b00b5830 0x7fe0b01589d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:07.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.687+0000 7fe0b4a9e700 1 --2- 172.21.15.16:0/1866301670 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe0b00cf4a0 0x7fe0b00ae2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:07.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.688+0000 7fe0b4a9e700 1 -- 172.21.15.16:0/1866301670 >> 172.21.15.16:0/1866301670 conn(0x7fe0b00b1fd0 msgr2=0x7fe0b01598d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:07.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.688+0000 7fe0b4a9e700 1 -- 172.21.15.16:0/1866301670 shutdown_connections 2023-10-16T13:46:07.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.688+0000 7fe0b4a9e700 1 -- 172.21.15.16:0/1866301670 wait complete. 2023-10-16T13:46:07.707 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 25, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T13:44:38.716950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '205', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 26, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T13:44:39.172499+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '212', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 5, 'snap_epoch': 212, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T13:46:07.708 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T13:46:07.760 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:07.758+0000 7f9d0ecc4200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:46:07.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.837+0000 7f6cf4f90700 1 Processor -- start 2023-10-16T13:46:07.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.837+0000 7f6cf4f90700 1 -- start start 2023-10-16T13:46:07.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.838+0000 7f6cf4f90700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f6cf0072b30 0x7f6cf0071a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:07.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.838+0000 7f6cf4f90700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6cf015bb40 0x7f6cf006e310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:07.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.838+0000 7f6cf4f90700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6cf00caa40 0x7f6cf00cae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:07.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.838+0000 7f6cf4f90700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cf00cb3b0 con 0x7f6cf00caa40 2023-10-16T13:46:07.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.838+0000 7f6cf4f90700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cf015c040 con 0x7f6cf0072b30 2023-10-16T13:46:07.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.838+0000 7f6cf4f90700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf015c320 con 0x7f6cf015bb40 2023-10-16T13:46:07.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.838+0000 7f6ceed9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6cf00caa40 0x7f6cf00cae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:07.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.838+0000 7f6cee59c700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f6cf0072b30 0x7f6cf0071a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:07.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.838+0000 7f6cedd9b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6cf015bb40 0x7f6cf006e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:07.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.839+0000 7f6ceed9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6cf00caa40 0x7f6cf00cae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52540/0 (socket says 172.21.15.16:52540) 2023-10-16T13:46:07.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.839+0000 7f6cee59c700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f6cf0072b30 0x7f6cf0071a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:51598/0 (socket says 172.21.15.16:51598) 2023-10-16T13:46:07.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.839+0000 7f6ceed9d700 1 -- 172.21.15.16:0/2695243447 learned_addr learned my addr 172.21.15.16:0/2695243447 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:07.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.839+0000 7f6ceed9d700 1 -- 172.21.15.16:0/2695243447 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6cf015bb40 msgr2=0x7f6cf006e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:07.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.839+0000 7f6ceed9d700 1 --2- 172.21.15.16:0/2695243447 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6cf015bb40 0x7f6cf006e310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:07.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.839+0000 7f6ceed9d700 1 -- 172.21.15.16:0/2695243447 >> v2:172.21.15.49:3300/0 conn(0x7f6cf0072b30 msgr2=0x7f6cf0071a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:07.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.839+0000 7f6ceed9d700 1 --2- 172.21.15.16:0/2695243447 >> v2:172.21.15.49:3300/0 conn(0x7f6cf0072b30 0x7f6cf0071a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:07.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.839+0000 7f6ceed9d700 1 -- 172.21.15.16:0/2695243447 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cf015c600 con 0x7f6cf00caa40 2023-10-16T13:46:07.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.839+0000 7f6cedd9b700 1 --2- 172.21.15.16:0/2695243447 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6cf015bb40 0x7f6cf006e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:46:07.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.839+0000 7f6cee59c700 1 --2- 172.21.15.16:0/2695243447 >> v2:172.21.15.49:3300/0 conn(0x7f6cf0072b30 0x7f6cf0071a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:46:07.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.839+0000 7f6ceed9d700 1 --2- 172.21.15.16:0/2695243447 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cf00caa40 0x7f6cf00cae70 secure :-1 s=READY pgs=2369 cs=0 l=1 rev1=1 crypto rx=0x7f6cf006b780 tx=0x7f6ce800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23786f02e19f05e7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:07.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.839+0000 7f6ced59a700 1 -- 172.21.15.16:0/2695243447 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce800e070 con 0x7f6cf00caa40 2023-10-16T13:46:07.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.839+0000 7f6ced59a700 1 -- 172.21.15.16:0/2695243447 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ce800b910 con 0x7f6cf00caa40 2023-10-16T13:46:07.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.839+0000 7f6ced59a700 1 -- 172.21.15.16:0/2695243447 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce800bcc0 con 0x7f6cf00caa40 2023-10-16T13:46:07.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.840+0000 7f6cf4f90700 1 -- 172.21.15.16:0/2695243447 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cf00caa40 msgr2=0x7f6cf00cae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:07.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.840+0000 7f6cf4f90700 1 --2- 172.21.15.16:0/2695243447 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cf00caa40 0x7f6cf00cae70 secure :-1 s=READY pgs=2369 cs=0 l=1 rev1=1 crypto rx=0x7f6cf006b780 tx=0x7f6ce800b330 comp rx=0 tx=0).stop 2023-10-16T13:46:07.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.840+0000 7f6cf4f90700 1 -- 172.21.15.16:0/2695243447 shutdown_connections 2023-10-16T13:46:07.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.840+0000 7f6cf4f90700 1 --2- 172.21.15.16:0/2695243447 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cf00caa40 0x7f6cf00cae70 unknown :-1 s=CLOSED pgs=2369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:07.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.840+0000 7f6cf4f90700 1 --2- 172.21.15.16:0/2695243447 >> v2:172.21.15.49:3300/0 conn(0x7f6cf0072b30 0x7f6cf0071a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:07.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.840+0000 7f6cf4f90700 1 --2- 172.21.15.16:0/2695243447 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6cf015bb40 0x7f6cf006e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:07.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.840+0000 7f6cf4f90700 1 -- 172.21.15.16:0/2695243447 >> 172.21.15.16:0/2695243447 conn(0x7f6cf00b1f40 msgr2=0x7f6cf00b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:07.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.840+0000 7f6cf4f90700 1 -- 172.21.15.16:0/2695243447 shutdown_connections 2023-10-16T13:46:07.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.840+0000 7f6cf4f90700 1 -- 172.21.15.16:0/2695243447 wait complete. 2023-10-16T13:46:07.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.842+0000 7f6cf4f90700 1 Processor -- start 2023-10-16T13:46:07.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.842+0000 7f6cf4f90700 1 -- start start 2023-10-16T13:46:07.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.842+0000 7f6cf4f90700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6cf0072b30 0x7f6cf0152940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:07.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.842+0000 7f6cf4f90700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6cf00caa40 0x7f6cf00c81e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:07.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.842+0000 7f6cf4f90700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cf015bb40 0x7f6cf00c4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:07.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.842+0000 7f6cf4f90700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf015f850 con 0x7f6cf015bb40 2023-10-16T13:46:07.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.842+0000 7f6cf4f90700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf015e7b0 con 0x7f6cf00caa40 2023-10-16T13:46:07.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.842+0000 7f6cf4f90700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf015ea90 con 0x7f6cf0072b30 2023-10-16T13:46:07.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.842+0000 7f6ceed9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cf015bb40 0x7f6cf00c4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:07.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.842+0000 7f6cee59c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6cf0072b30 0x7f6cf0152940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:07.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.843+0000 7f6cedd9b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6cf00caa40 0x7f6cf00c81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:07.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.843+0000 7f6cee59c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6cf0072b30 0x7f6cf0152940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:40800/0 (socket says 172.21.15.16:40800) 2023-10-16T13:46:07.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.843+0000 7f6cedd9b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6cf00caa40 0x7f6cf00c81e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:51606/0 (socket says 172.21.15.16:51606) 2023-10-16T13:46:07.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.843+0000 7f6cee59c700 1 -- 172.21.15.16:0/2257505466 learned_addr learned my addr 172.21.15.16:0/2257505466 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:07.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.843+0000 7f6ceed9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cf015bb40 0x7f6cf00c4f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52546/0 (socket says 172.21.15.16:52546) 2023-10-16T13:46:07.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.843+0000 7f6cee59c700 1 -- 172.21.15.16:0/2257505466 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6cf00caa40 msgr2=0x7f6cf00c81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:07.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.843+0000 7f6cee59c700 1 --2- 172.21.15.16:0/2257505466 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6cf00caa40 0x7f6cf00c81e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:07.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.843+0000 7f6cee59c700 1 -- 172.21.15.16:0/2257505466 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cf015bb40 msgr2=0x7f6cf00c4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:07.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.843+0000 7f6cee59c700 1 --2- 172.21.15.16:0/2257505466 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cf015bb40 0x7f6cf00c4f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:07.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.843+0000 7f6cee59c700 1 -- 172.21.15.16:0/2257505466 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ce00094a0 con 0x7f6cf0072b30 2023-10-16T13:46:07.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.843+0000 7f6ceed9d700 1 --2- 172.21.15.16:0/2257505466 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cf015bb40 0x7f6cf00c4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:46:07.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.843+0000 7f6cee59c700 1 --2- 172.21.15.16:0/2257505466 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6cf0072b30 0x7f6cf0152940 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f6ce000ebe0 tx=0x7f6ce000eef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:07.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.843+0000 7f6cdeffd700 1 -- 172.21.15.16:0/2257505466 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce0009760 con 0x7f6cf0072b30 2023-10-16T13:46:07.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.843+0000 7f6cdeffd700 1 -- 172.21.15.16:0/2257505466 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ce000c850 con 0x7f6cf0072b30 2023-10-16T13:46:07.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.843+0000 7f6cf4f90700 1 -- 172.21.15.16:0/2257505466 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ce40094a0 con 0x7f6cf0072b30 2023-10-16T13:46:07.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.843+0000 7f6cdeffd700 1 -- 172.21.15.16:0/2257505466 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce000cbe0 con 0x7f6cf0072b30 2023-10-16T13:46:07.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.843+0000 7f6cf4f90700 1 -- 172.21.15.16:0/2257505466 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ce8009b30 con 0x7f6cf0072b30 2023-10-16T13:46:07.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.844+0000 7f6cf4f90700 1 -- 172.21.15.16:0/2257505466 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cf015f6a0 con 0x7f6cf0072b30 2023-10-16T13:46:07.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.845+0000 7f6cdeffd700 1 -- 172.21.15.16:0/2257505466 <== mon.2 v2:172.21.15.49:3301/0 4 ==== mgrmap(e 134) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f6ce000cd40 con 0x7f6cf0072b30 2023-10-16T13:46:07.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.845+0000 7f6cdeffd700 1 --2- 172.21.15.16:0/2257505466 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f6cd006b140 0x7f6cd006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:07.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.845+0000 7f6cedd9b700 1 -- 172.21.15.16:0/2257505466 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f6cd006b140 msgr2=0x7f6cd006d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3220139196 2023-10-16T13:46:07.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.845+0000 7f6cdeffd700 1 -- 172.21.15.16:0/2257505466 <== mon.2 v2:172.21.15.49:3301/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6660+0+0 (secure 0 0 0) 0x7f6ce0089e90 con 0x7f6cf0072b30 2023-10-16T13:46:07.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.845+0000 7f6cedd9b700 1 --2- 172.21.15.16:0/2257505466 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f6cd006b140 0x7f6cd006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:46:07.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:07.850+0000 7f6cdeffd700 1 -- 172.21.15.16:0/2257505466 <== mon.2 v2:172.21.15.49:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6ce0054780 con 0x7f6cf0072b30 2023-10-16T13:46:07.885 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:46:07.884+0000 7f3e59442200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T13:46:07.963 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:46:07.962+0000 7f3e59442200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T13:46:08.008 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:08.005+0000 7f9d0ecc4200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T13:46:08.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.046+0000 7f6cedd9b700 1 -- 172.21.15.16:0/2257505466 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f6cd006b140 msgr2=0x7f6cd006d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3220139196 2023-10-16T13:46:08.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.046+0000 7f6cedd9b700 1 --2- 172.21.15.16:0/2257505466 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f6cd006b140 0x7f6cd006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:46:08.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.054+0000 7f6cf4f90700 1 -- 172.21.15.16:0/2257505466 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6cf00b63a0 con 0x7f6cf0072b30 2023-10-16T13:46:08.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.055+0000 7f6cdeffd700 1 -- 172.21.15.16:0/2257505466 <== mon.2 v2:172.21.15.49:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v212) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6ce00210a0 con 0x7f6cf0072b30 2023-10-16T13:46:08.058 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T13:46:08.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.059+0000 7f6cf4f90700 1 -- 172.21.15.16:0/2257505466 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f6cd006b140 msgr2=0x7f6cd006d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:46:08.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.059+0000 7f6cf4f90700 1 --2- 172.21.15.16:0/2257505466 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f6cd006b140 0x7f6cd006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:08.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.059+0000 7f6cf4f90700 1 -- 172.21.15.16:0/2257505466 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6cf0072b30 msgr2=0x7f6cf0152940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:08.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.059+0000 7f6cf4f90700 1 --2- 172.21.15.16:0/2257505466 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6cf0072b30 0x7f6cf0152940 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f6ce000ebe0 tx=0x7f6ce000eef0 comp rx=0 tx=0).stop 2023-10-16T13:46:08.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.059+0000 7f6cf4f90700 1 -- 172.21.15.16:0/2257505466 shutdown_connections 2023-10-16T13:46:08.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.059+0000 7f6cf4f90700 1 --2- 172.21.15.16:0/2257505466 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f6cd006b140 0x7f6cd006d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:08.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.059+0000 7f6cf4f90700 1 --2- 172.21.15.16:0/2257505466 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cf015bb40 0x7f6cf00c4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:08.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.059+0000 7f6cf4f90700 1 --2- 172.21.15.16:0/2257505466 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6cf0072b30 0x7f6cf0152940 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:08.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.059+0000 7f6cf4f90700 1 --2- 172.21.15.16:0/2257505466 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6cf00caa40 0x7f6cf00c81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:08.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.059+0000 7f6cf4f90700 1 -- 172.21.15.16:0/2257505466 >> 172.21.15.16:0/2257505466 conn(0x7f6cf00b1f40 msgr2=0x7f6cf006a970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:08.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.059+0000 7f6cf4f90700 1 -- 172.21.15.16:0/2257505466 shutdown_connections 2023-10-16T13:46:08.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.059+0000 7f6cf4f90700 1 -- 172.21.15.16:0/2257505466 wait complete. 2023-10-16T13:46:08.076 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T13:46:08.095 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:08.093+0000 7f9d0ecc4200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T13:46:08.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.211+0000 7f818fce7700 1 Processor -- start 2023-10-16T13:46:08.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.212+0000 7f818fce7700 1 -- start start 2023-10-16T13:46:08.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.212+0000 7f818fce7700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f818806e170 0x7f81880cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:08.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.212+0000 7f818fce7700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81880cef70 con 0x7f818815b070 2023-10-16T13:46:08.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.212+0000 7f818fce7700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81880ce360 con 0x7f81880cb880 2023-10-16T13:46:08.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.212+0000 7f818fce7700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81880ce640 con 0x7f818806e170 2023-10-16T13:46:08.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.213+0000 7f818da83700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f818815b070 0x7f818815b450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41826/0 (socket says 172.21.15.16:41826) 2023-10-16T13:46:08.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.213+0000 7f818da83700 1 -- 172.21.15.16:0/2023002848 learned_addr learned my addr 172.21.15.16:0/2023002848 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:46:08.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.213+0000 7f818e284700 1 --2- 172.21.15.16:0/2023002848 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f818806e170 0x7f81880cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:08.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.213+0000 7f818ca81700 1 -- 172.21.15.16:0/2023002848 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2566500674 0 0) 0x7f81880cef70 con 0x7f818815b070 2023-10-16T13:46:08.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.213+0000 7f818ca81700 1 -- 172.21.15.16:0/2023002848 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81740035b0 con 0x7f818815b070 2023-10-16T13:46:08.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.213+0000 7f818ca81700 1 -- 172.21.15.16:0/2023002848 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3589512429 0 0) 0x7f81880ce360 con 0x7f81880cb880 2023-10-16T13:46:08.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.213+0000 7f818ca81700 1 -- 172.21.15.16:0/2023002848 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81880cef70 con 0x7f81880cb880 2023-10-16T13:46:08.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.213+0000 7f818e284700 1 -- 172.21.15.16:0/2023002848 >> v1:172.21.15.49:6789/0 conn(0x7f81880cb880 legacy=0x7f818806da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:08.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.213+0000 7f818e284700 1 -- 172.21.15.16:0/2023002848 >> v1:172.21.15.16:6789/0 conn(0x7f818815b070 legacy=0x7f818815b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:08.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.213+0000 7f818e284700 1 -- 172.21.15.16:0/2023002848 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81880ce920 con 0x7f818806e170 2023-10-16T13:46:08.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.213+0000 7f818e284700 1 --2- 172.21.15.16:0/2023002848 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f818806e170 0x7f81880cbd00 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f8180009dc0 tx=0x7f818000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=214647d31512616c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:08.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.213+0000 7f818ca81700 1 -- 172.21.15.16:0/2023002848 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f818000e070 con 0x7f818806e170 2023-10-16T13:46:08.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.213+0000 7f818ca81700 1 -- 172.21.15.16:0/2023002848 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f818000bb20 con 0x7f818806e170 2023-10-16T13:46:08.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.213+0000 7f818ca81700 1 -- 172.21.15.16:0/2023002848 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8180012430 con 0x7f818806e170 2023-10-16T13:46:08.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.214+0000 7f818fce7700 1 -- 172.21.15.16:0/2023002848 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f818806e170 msgr2=0x7f81880cbd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:08.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.214+0000 7f818fce7700 1 --2- 172.21.15.16:0/2023002848 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f818806e170 0x7f81880cbd00 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f8180009dc0 tx=0x7f818000b330 comp rx=0 tx=0).stop 2023-10-16T13:46:08.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.214+0000 7f818fce7700 1 -- 172.21.15.16:0/2023002848 shutdown_connections 2023-10-16T13:46:08.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.214+0000 7f818fce7700 1 --2- 172.21.15.16:0/2023002848 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f818806e170 0x7f81880cbd00 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:08.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.214+0000 7f818fce7700 1 -- 172.21.15.16:0/2023002848 >> 172.21.15.16:0/2023002848 conn(0x7f81880b1f40 msgr2=0x7f81880b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:08.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.214+0000 7f818fce7700 1 -- 172.21.15.16:0/2023002848 shutdown_connections 2023-10-16T13:46:08.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.214+0000 7f818fce7700 1 -- 172.21.15.16:0/2023002848 wait complete. 2023-10-16T13:46:08.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.216+0000 7f818fce7700 1 Processor -- start 2023-10-16T13:46:08.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.216+0000 7f818fce7700 1 -- start start 2023-10-16T13:46:08.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.216+0000 7f818fce7700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f818806e170 0x7f8188153810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:08.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.216+0000 7f818fce7700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f81880cb880 0x7f81880b3f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:08.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.216+0000 7f818fce7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f818815b070 0x7f81881f6340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:08.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.216+0000 7f818fce7700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81881f6d20 con 0x7f818815b070 2023-10-16T13:46:08.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.216+0000 7f818fce7700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81881f7000 con 0x7f818806e170 2023-10-16T13:46:08.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.216+0000 7f818fce7700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81880b3700 con 0x7f81880cb880 2023-10-16T13:46:08.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.217+0000 7f818e284700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f818815b070 0x7f81881f6340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:08.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.217+0000 7f818da83700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f818806e170 0x7f8188153810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:08.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.217+0000 7f818d282700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f81880cb880 0x7f81880b3f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:08.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.217+0000 7f818da83700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f818806e170 0x7f8188153810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:51620/0 (socket says 172.21.15.16:51620) 2023-10-16T13:46:08.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.217+0000 7f818d282700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f81880cb880 0x7f81880b3f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:40818/0 (socket says 172.21.15.16:40818) 2023-10-16T13:46:08.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.217+0000 7f818da83700 1 -- 172.21.15.16:0/2691439509 learned_addr learned my addr 172.21.15.16:0/2691439509 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:08.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.217+0000 7f818da83700 1 -- 172.21.15.16:0/2691439509 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f81880cb880 msgr2=0x7f81880b3f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:08.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.217+0000 7f818da83700 1 --2- 172.21.15.16:0/2691439509 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f81880cb880 0x7f81880b3f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:08.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.217+0000 7f818da83700 1 -- 172.21.15.16:0/2691439509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f818815b070 msgr2=0x7f81881f6340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:08.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.217+0000 7f818da83700 1 --2- 172.21.15.16:0/2691439509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f818815b070 0x7f81881f6340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:08.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.217+0000 7f818da83700 1 -- 172.21.15.16:0/2691439509 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8180009a70 con 0x7f818806e170 2023-10-16T13:46:08.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.217+0000 7f818e284700 1 --2- 172.21.15.16:0/2691439509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f818815b070 0x7f81881f6340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:46:08.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.217+0000 7f818da83700 1 --2- 172.21.15.16:0/2691439509 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f818806e170 0x7f8188153810 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f8178001eb0 tx=0x7f817800c7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:08.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.217+0000 7f817e7fc700 1 -- 172.21.15.16:0/2691439509 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8178005d70 con 0x7f818806e170 2023-10-16T13:46:08.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.217+0000 7f817e7fc700 1 -- 172.21.15.16:0/2691439509 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81780070d0 con 0x7f818806e170 2023-10-16T13:46:08.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.217+0000 7f818fce7700 1 -- 172.21.15.16:0/2691439509 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81880b39e0 con 0x7f818806e170 2023-10-16T13:46:08.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.217+0000 7f817e7fc700 1 -- 172.21.15.16:0/2691439509 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8178007440 con 0x7f818806e170 2023-10-16T13:46:08.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.217+0000 7f818fce7700 1 -- 172.21.15.16:0/2691439509 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81880b5e20 con 0x7f818806e170 2023-10-16T13:46:08.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.218+0000 7f818fce7700 1 -- 172.21.15.16:0/2691439509 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81880b3560 con 0x7f818806e170 2023-10-16T13:46:08.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.219+0000 7f817e7fc700 1 -- 172.21.15.16:0/2691439509 <== mon.1 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 134) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f81780075a0 con 0x7f818806e170 2023-10-16T13:46:08.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.219+0000 7f817e7fc700 1 --2- 172.21.15.16:0/2691439509 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f816c06b140 0x7f816c06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:08.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.219+0000 7f818d282700 1 -- 172.21.15.16:0/2691439509 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f816c06b140 msgr2=0x7f816c06d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3220139196 2023-10-16T13:46:08.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.219+0000 7f817e7fc700 1 -- 172.21.15.16:0/2691439509 <== mon.1 v2:172.21.15.49:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6660+0+0 (secure 0 0 0) 0x7f8178089030 con 0x7f818806e170 2023-10-16T13:46:08.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.219+0000 7f818d282700 1 --2- 172.21.15.16:0/2691439509 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f816c06b140 0x7f816c06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:46:08.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.224+0000 7f817e7fc700 1 -- 172.21.15.16:0/2691439509 <== mon.1 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f81880b3560 con 0x7f818806e170 2023-10-16T13:46:08.242 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:46:08.241+0000 7f3e59442200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T13:46:08.382 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:08.380+0000 7f9d0ecc4200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T13:46:08.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.420+0000 7f818d282700 1 -- 172.21.15.16:0/2691439509 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f816c06b140 msgr2=0x7f816c06d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3220139196 2023-10-16T13:46:08.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.420+0000 7f818d282700 1 --2- 172.21.15.16:0/2691439509 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f816c06b140 0x7f816c06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:46:08.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.428+0000 7f818fce7700 1 -- 172.21.15.16:0/2691439509 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f81880b3c20 con 0x7f818806e170 2023-10-16T13:46:08.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.428+0000 7f817e7fc700 1 -- 172.21.15.16:0/2691439509 <== mon.1 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v212) v1 ==== 104+0+11 (secure 0 0 0) 0x7f81880b3c20 con 0x7f818806e170 2023-10-16T13:46:08.431 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-10-16T13:46:08.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.431+0000 7f818fce7700 1 -- 172.21.15.16:0/2691439509 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f816c06b140 msgr2=0x7f816c06d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:46:08.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.431+0000 7f818fce7700 1 --2- 172.21.15.16:0/2691439509 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f816c06b140 0x7f816c06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:08.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.431+0000 7f818fce7700 1 -- 172.21.15.16:0/2691439509 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f818806e170 msgr2=0x7f8188153810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:08.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.431+0000 7f818fce7700 1 --2- 172.21.15.16:0/2691439509 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f818806e170 0x7f8188153810 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f8178001eb0 tx=0x7f817800c7b0 comp rx=0 tx=0).stop 2023-10-16T13:46:08.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.431+0000 7f818fce7700 1 -- 172.21.15.16:0/2691439509 shutdown_connections 2023-10-16T13:46:08.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.431+0000 7f818fce7700 1 --2- 172.21.15.16:0/2691439509 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f816c06b140 0x7f816c06d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:08.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.431+0000 7f818fce7700 1 --2- 172.21.15.16:0/2691439509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f818815b070 0x7f81881f6340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:08.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.431+0000 7f818fce7700 1 --2- 172.21.15.16:0/2691439509 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f818806e170 0x7f8188153810 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:08.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.431+0000 7f818fce7700 1 --2- 172.21.15.16:0/2691439509 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f81880cb880 0x7f81880b3f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:08.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.431+0000 7f818fce7700 1 -- 172.21.15.16:0/2691439509 >> 172.21.15.16:0/2691439509 conn(0x7f81880b1f40 msgr2=0x7f818815d460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:08.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.431+0000 7f818fce7700 1 -- 172.21.15.16:0/2691439509 shutdown_connections 2023-10-16T13:46:08.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.431+0000 7f818fce7700 1 -- 172.21.15.16:0/2691439509 wait complete. 2023-10-16T13:46:08.446 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T13:46:08.477 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:46:08.475+0000 7f3e59442200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T13:46:08.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.615+0000 7f66ab2f9700 1 Processor -- start 2023-10-16T13:46:08.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.615+0000 7f66ab2f9700 1 -- start start 2023-10-16T13:46:08.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.616+0000 7f66ab2f9700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f66a406a7e0 0x7f66a406ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:08.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.616+0000 7f66ab2f9700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66a415ecb0 con 0x7f66a406e570 2023-10-16T13:46:08.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.616+0000 7f66ab2f9700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66a415b6a0 con 0x7f66a415b2c0 2023-10-16T13:46:08.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.616+0000 7f66ab2f9700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66a406b190 con 0x7f66a406a7e0 2023-10-16T13:46:08.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.616+0000 7f66a8894700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f66a406e570 0x7f66a415e5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41842/0 (socket says 172.21.15.16:41842) 2023-10-16T13:46:08.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.616+0000 7f66a8894700 1 -- 172.21.15.16:0/3955698112 learned_addr learned my addr 172.21.15.16:0/3955698112 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:46:08.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.616+0000 7f66a9896700 1 --2- 172.21.15.16:0/3955698112 >> v2:172.21.15.49:3300/0 conn(0x7f66a406a7e0 0x7f66a406ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:08.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.616+0000 7f669bfff700 1 -- 172.21.15.16:0/3955698112 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 804791947 0 0) 0x7f66a415ecb0 con 0x7f66a406e570 2023-10-16T13:46:08.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.617+0000 7f669bfff700 1 -- 172.21.15.16:0/3955698112 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66900035b0 con 0x7f66a406e570 2023-10-16T13:46:08.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.617+0000 7f66a9896700 1 -- 172.21.15.16:0/3955698112 >> v1:172.21.15.49:6789/0 conn(0x7f66a415b2c0 legacy=0x7f66a4161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:08.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.617+0000 7f66a9896700 1 -- 172.21.15.16:0/3955698112 >> v1:172.21.15.16:6789/0 conn(0x7f66a406e570 legacy=0x7f66a415e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:08.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.617+0000 7f66a9896700 1 -- 172.21.15.16:0/3955698112 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66a4161b20 con 0x7f66a406a7e0 2023-10-16T13:46:08.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.617+0000 7f66a9896700 1 --2- 172.21.15.16:0/3955698112 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f66a406a7e0 0x7f66a406ac50 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f669c009d90 tx=0x7f669c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dbaa3a251af97127 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:08.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.617+0000 7f669bfff700 1 -- 172.21.15.16:0/3955698112 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f669c00e070 con 0x7f66a406a7e0 2023-10-16T13:46:08.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.617+0000 7f669bfff700 1 -- 172.21.15.16:0/3955698112 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f669c00bb20 con 0x7f66a406a7e0 2023-10-16T13:46:08.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.617+0000 7f669bfff700 1 -- 172.21.15.16:0/3955698112 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f669c012430 con 0x7f66a406a7e0 2023-10-16T13:46:08.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.618+0000 7f66ab2f9700 1 -- 172.21.15.16:0/3955698112 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f66a406a7e0 msgr2=0x7f66a406ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:08.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.618+0000 7f66ab2f9700 1 --2- 172.21.15.16:0/3955698112 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f66a406a7e0 0x7f66a406ac50 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f669c009d90 tx=0x7f669c00b330 comp rx=0 tx=0).stop 2023-10-16T13:46:08.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.618+0000 7f66ab2f9700 1 -- 172.21.15.16:0/3955698112 shutdown_connections 2023-10-16T13:46:08.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.618+0000 7f66ab2f9700 1 --2- 172.21.15.16:0/3955698112 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f66a406a7e0 0x7f66a406ac50 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:08.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.618+0000 7f66ab2f9700 1 -- 172.21.15.16:0/3955698112 >> 172.21.15.16:0/3955698112 conn(0x7f66a40b1f40 msgr2=0x7f66a40b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:08.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.618+0000 7f66ab2f9700 1 -- 172.21.15.16:0/3955698112 shutdown_connections 2023-10-16T13:46:08.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.618+0000 7f66ab2f9700 1 -- 172.21.15.16:0/3955698112 wait complete. 2023-10-16T13:46:08.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.619+0000 7f66ab2f9700 1 Processor -- start 2023-10-16T13:46:08.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.619+0000 7f66ab2f9700 1 -- start start 2023-10-16T13:46:08.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.620+0000 7f66ab2f9700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f66a406a7e0 0x7f66a40c6b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:08.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.620+0000 7f66ab2f9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66a406e570 0x7f66a40b4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:08.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.620+0000 7f66ab2f9700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f66a415b2c0 0x7f66a4155a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:08.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.620+0000 7f66ab2f9700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66a40c7090 con 0x7f66a406e570 2023-10-16T13:46:08.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.620+0000 7f66ab2f9700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66a40cd5c0 con 0x7f66a415b2c0 2023-10-16T13:46:08.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.620+0000 7f66ab2f9700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66a40c5050 con 0x7f66a406a7e0 2023-10-16T13:46:08.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.620+0000 7f66a8894700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66a406e570 0x7f66a40b4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:08.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.620+0000 7f66a9896700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f66a415b2c0 0x7f66a4155a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:08.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.620+0000 7f66a8894700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66a406e570 0x7f66a40b4770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52576/0 (socket says 172.21.15.16:52576) 2023-10-16T13:46:08.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.620+0000 7f66a8894700 1 -- 172.21.15.16:0/862479573 learned_addr learned my addr 172.21.15.16:0/862479573 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:08.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.620+0000 7f66a9896700 1 -- 172.21.15.16:0/862479573 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f66a406a7e0 msgr2=0x7f66a40c6b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:08.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.620+0000 7f66a9896700 1 --2- 172.21.15.16:0/862479573 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f66a406a7e0 0x7f66a40c6b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:08.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.620+0000 7f66a9896700 1 -- 172.21.15.16:0/862479573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66a406e570 msgr2=0x7f66a40b4770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:08.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.620+0000 7f66a9896700 1 --2- 172.21.15.16:0/862479573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66a406e570 0x7f66a40b4770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:08.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.620+0000 7f66a9896700 1 -- 172.21.15.16:0/862479573 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f669c009a70 con 0x7f66a415b2c0 2023-10-16T13:46:08.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.620+0000 7f66a8894700 1 --2- 172.21.15.16:0/862479573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66a406e570 0x7f66a40b4770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:46:08.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.621+0000 7f66a9896700 1 --2- 172.21.15.16:0/862479573 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f66a415b2c0 0x7f66a4155a00 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f669c016000 tx=0x7f669c004ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:08.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.621+0000 7f6699ffb700 1 -- 172.21.15.16:0/862479573 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f669c012430 con 0x7f66a415b2c0 2023-10-16T13:46:08.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.621+0000 7f6699ffb700 1 -- 172.21.15.16:0/862479573 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f669c020070 con 0x7f66a415b2c0 2023-10-16T13:46:08.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.621+0000 7f6699ffb700 1 -- 172.21.15.16:0/862479573 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f669c004280 con 0x7f66a415b2c0 2023-10-16T13:46:08.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.621+0000 7f66ab2f9700 1 -- 172.21.15.16:0/862479573 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66a40c5330 con 0x7f66a415b2c0 2023-10-16T13:46:08.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.621+0000 7f66ab2f9700 1 -- 172.21.15.16:0/862479573 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66a40c58f0 con 0x7f66a415b2c0 2023-10-16T13:46:08.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.622+0000 7f66ab2f9700 1 -- 172.21.15.16:0/862479573 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66a404ebb0 con 0x7f66a415b2c0 2023-10-16T13:46:08.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.623+0000 7f6699ffb700 1 -- 172.21.15.16:0/862479573 <== mon.1 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 134) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f669c0043e0 con 0x7f66a415b2c0 2023-10-16T13:46:08.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.623+0000 7f6699ffb700 1 --2- 172.21.15.16:0/862479573 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f668806b140 0x7f668806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:08.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.623+0000 7f66a9095700 1 -- 172.21.15.16:0/862479573 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f668806b140 msgr2=0x7f668806d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3220139196 2023-10-16T13:46:08.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.623+0000 7f6699ffb700 1 -- 172.21.15.16:0/862479573 <== mon.1 v2:172.21.15.49:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6660+0+0 (secure 0 0 0) 0x7f669c022070 con 0x7f66a415b2c0 2023-10-16T13:46:08.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.623+0000 7f66a9095700 1 --2- 172.21.15.16:0/862479573 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f668806b140 0x7f668806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:46:08.633 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:08.626+0000 7f9d0ecc4200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T13:46:08.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.627+0000 7f6699ffb700 1 -- 172.21.15.16:0/862479573 <== mon.1 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f669c056400 con 0x7f66a415b2c0 2023-10-16T13:46:08.634 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:46:08.633+0000 7f3e59442200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T13:46:08.755 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:46:08.753+0000 7f3e59442200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T13:46:08.790 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:08.788+0000 7f9d0ecc4200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T13:46:08.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.824+0000 7f66a9095700 1 -- 172.21.15.16:0/862479573 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f668806b140 msgr2=0x7f668806d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3220139196 2023-10-16T13:46:08.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.824+0000 7f66a9095700 1 --2- 172.21.15.16:0/862479573 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f668806b140 0x7f668806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:46:08.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.836+0000 7f66ab2f9700 1 -- 172.21.15.16:0/862479573 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f66a40b6370 con 0x7f66a415b2c0 2023-10-16T13:46:08.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.837+0000 7f6699ffb700 1 -- 172.21.15.16:0/862479573 <== mon.1 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v212) v1 ==== 100+0+11 (secure 0 0 0) 0x7f669c059a20 con 0x7f66a415b2c0 2023-10-16T13:46:08.839 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-10-16T13:46:08.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.839+0000 7f66ab2f9700 1 -- 172.21.15.16:0/862479573 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f668806b140 msgr2=0x7f668806d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:46:08.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.839+0000 7f66ab2f9700 1 --2- 172.21.15.16:0/862479573 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f668806b140 0x7f668806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:08.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.839+0000 7f66ab2f9700 1 -- 172.21.15.16:0/862479573 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f66a415b2c0 msgr2=0x7f66a4155a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:08.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.839+0000 7f66ab2f9700 1 --2- 172.21.15.16:0/862479573 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f66a415b2c0 0x7f66a4155a00 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f669c016000 tx=0x7f669c004ad0 comp rx=0 tx=0).stop 2023-10-16T13:46:08.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.839+0000 7f66ab2f9700 1 -- 172.21.15.16:0/862479573 shutdown_connections 2023-10-16T13:46:08.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.839+0000 7f66ab2f9700 1 --2- 172.21.15.16:0/862479573 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f66a415b2c0 0x7f66a4155a00 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:08.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.839+0000 7f66ab2f9700 1 --2- 172.21.15.16:0/862479573 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f66a406a7e0 0x7f66a40c6b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:08.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.839+0000 7f66ab2f9700 1 --2- 172.21.15.16:0/862479573 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f668806b140 0x7f668806d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:08.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.839+0000 7f66ab2f9700 1 --2- 172.21.15.16:0/862479573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66a406e570 0x7f66a40b4770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:08.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.839+0000 7f66ab2f9700 1 -- 172.21.15.16:0/862479573 >> 172.21.15.16:0/862479573 conn(0x7f66a40b1f40 msgr2=0x7f66a4160f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:08.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.839+0000 7f66ab2f9700 1 -- 172.21.15.16:0/862479573 shutdown_connections 2023-10-16T13:46:08.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.839+0000 7f66ab2f9700 1 -- 172.21.15.16:0/862479573 wait complete. 2023-10-16T13:46:08.854 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:46:08.917 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:08.915+0000 7f9d0ecc4200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T13:46:08.920 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:46:08.918+0000 7f3e59442200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T13:46:08.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.978+0000 7f9394bb7700 1 Processor -- start 2023-10-16T13:46:08.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.978+0000 7f9394bb7700 1 -- start start 2023-10-16T13:46:08.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.979+0000 7f9394bb7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f93900cdfa0 0x7f93900b2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:08.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.979+0000 7f9394bb7700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f93900ce380 0x7f93900b2980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:08.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.979+0000 7f9394bb7700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93900d0fc0 con 0x7f93900d04a0 2023-10-16T13:46:08.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.979+0000 7f9394bb7700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93900d1210 con 0x7f93900cdfa0 2023-10-16T13:46:08.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.979+0000 7f9394bb7700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93900cbc30 con 0x7f93900ce380 2023-10-16T13:46:08.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.980+0000 7f938e59c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f93900cdfa0 0x7f93900b2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:08.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.980+0000 7f938ed9d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f93900d04a0 0x7f93900d08b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41862/0 (socket says 172.21.15.16:41862) 2023-10-16T13:46:08.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.980+0000 7f938ed9d700 1 -- 172.21.15.16:0/3797021998 learned_addr learned my addr 172.21.15.16:0/3797021998 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:46:08.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.980+0000 7f938dd9b700 1 --2- 172.21.15.16:0/3797021998 >> v2:172.21.15.49:3300/0 conn(0x7f93900ce380 0x7f93900b2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:08.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.980+0000 7f938e59c700 1 -- 172.21.15.16:0/3797021998 >> v2:172.21.15.49:3300/0 conn(0x7f93900ce380 msgr2=0x7f93900b2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:08.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.980+0000 7f938e59c700 1 --2- 172.21.15.16:0/3797021998 >> v2:172.21.15.49:3300/0 conn(0x7f93900ce380 0x7f93900b2980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:08.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.980+0000 7f938e59c700 1 -- 172.21.15.16:0/3797021998 >> v1:172.21.15.16:6789/0 conn(0x7f93900d04a0 legacy=0x7f93900d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:08.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.980+0000 7f938d59a700 1 -- 172.21.15.16:0/3797021998 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3230364504 0 0) 0x7f93900d0fc0 con 0x7f93900d04a0 2023-10-16T13:46:08.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.980+0000 7f938e59c700 1 -- 172.21.15.16:0/3797021998 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93900cbf10 con 0x7f93900cdfa0 2023-10-16T13:46:08.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.980+0000 7f938e59c700 1 --2- 172.21.15.16:0/3797021998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93900cdfa0 0x7f93900b2440 secure :-1 s=READY pgs=2373 cs=0 l=1 rev1=1 crypto rx=0x7f93900671e0 tx=0x7f9380009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8436f69e584d08d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:08.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.980+0000 7f938d59a700 1 -- 172.21.15.16:0/3797021998 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9380011830 con 0x7f93900cdfa0 2023-10-16T13:46:08.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.980+0000 7f938d59a700 1 -- 172.21.15.16:0/3797021998 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9380011990 con 0x7f93900cdfa0 2023-10-16T13:46:08.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.980+0000 7f938d59a700 1 -- 172.21.15.16:0/3797021998 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9380011d20 con 0x7f93900cdfa0 2023-10-16T13:46:08.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.981+0000 7f9394bb7700 1 -- 172.21.15.16:0/3797021998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93900cdfa0 msgr2=0x7f93900b2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:08.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.981+0000 7f9394bb7700 1 --2- 172.21.15.16:0/3797021998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93900cdfa0 0x7f93900b2440 secure :-1 s=READY pgs=2373 cs=0 l=1 rev1=1 crypto rx=0x7f93900671e0 tx=0x7f9380009b20 comp rx=0 tx=0).stop 2023-10-16T13:46:08.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.981+0000 7f9394bb7700 1 -- 172.21.15.16:0/3797021998 shutdown_connections 2023-10-16T13:46:08.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.981+0000 7f9394bb7700 1 --2- 172.21.15.16:0/3797021998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93900cdfa0 0x7f93900b2440 unknown :-1 s=CLOSED pgs=2373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:08.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.981+0000 7f9394bb7700 1 --2- 172.21.15.16:0/3797021998 >> v2:172.21.15.49:3300/0 conn(0x7f93900ce380 0x7f93900b2980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:08.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.981+0000 7f9394bb7700 1 -- 172.21.15.16:0/3797021998 >> 172.21.15.16:0/3797021998 conn(0x7f93900d14b0 msgr2=0x7f93900d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:08.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.981+0000 7f9394bb7700 1 -- 172.21.15.16:0/3797021998 shutdown_connections 2023-10-16T13:46:08.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.981+0000 7f9394bb7700 1 -- 172.21.15.16:0/3797021998 wait complete. 2023-10-16T13:46:08.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.982+0000 7f9394bb7700 1 Processor -- start 2023-10-16T13:46:08.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.982+0000 7f9394bb7700 1 -- start start 2023-10-16T13:46:08.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.983+0000 7f9394bb7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93900cdfa0 0x7f93900c73f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:08.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.983+0000 7f9394bb7700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f93900ce380 0x7f9390155b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:08.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.983+0000 7f9394bb7700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f93900d04a0 0x7f93900c9090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:08.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.983+0000 7f9394bb7700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f939015ce40 con 0x7f93900cdfa0 2023-10-16T13:46:08.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.983+0000 7f9394bb7700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f939015e9e0 con 0x7f93900ce380 2023-10-16T13:46:08.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.983+0000 7f9394bb7700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93901f6e60 con 0x7f93900d04a0 2023-10-16T13:46:08.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.983+0000 7f938e59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93900cdfa0 0x7f93900c73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:08.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.983+0000 7f938e59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93900cdfa0 0x7f93900c73f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52590/0 (socket says 172.21.15.16:52590) 2023-10-16T13:46:08.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.983+0000 7f938e59c700 1 -- 172.21.15.16:0/4013132115 learned_addr learned my addr 172.21.15.16:0/4013132115 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:08.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.983+0000 7f938dd9b700 1 --2- 172.21.15.16:0/4013132115 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f93900ce380 0x7f9390155b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:08.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.983+0000 7f938ed9d700 1 --2- 172.21.15.16:0/4013132115 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f93900d04a0 0x7f93900c9090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:08.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.984+0000 7f938e59c700 1 -- 172.21.15.16:0/4013132115 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f93900d04a0 msgr2=0x7f93900c9090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:08.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.984+0000 7f938e59c700 1 --2- 172.21.15.16:0/4013132115 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f93900d04a0 0x7f93900c9090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:08.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.984+0000 7f938e59c700 1 -- 172.21.15.16:0/4013132115 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f93900ce380 msgr2=0x7f9390155b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:08.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.984+0000 7f938e59c700 1 --2- 172.21.15.16:0/4013132115 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f93900ce380 0x7f9390155b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:08.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.984+0000 7f938e59c700 1 -- 172.21.15.16:0/4013132115 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93800094a0 con 0x7f93900cdfa0 2023-10-16T13:46:08.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.984+0000 7f938e59c700 1 --2- 172.21.15.16:0/4013132115 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93900cdfa0 0x7f93900c73f0 secure :-1 s=READY pgs=2374 cs=0 l=1 rev1=1 crypto rx=0x7f9380009b60 tx=0x7f9380004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:08.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.984+0000 7f937effd700 1 -- 172.21.15.16:0/4013132115 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9380004550 con 0x7f93900cdfa0 2023-10-16T13:46:08.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.984+0000 7f937effd700 1 -- 172.21.15.16:0/4013132115 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f938001f070 con 0x7f93900cdfa0 2023-10-16T13:46:08.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.984+0000 7f9394bb7700 1 -- 172.21.15.16:0/4013132115 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93901f7140 con 0x7f93900cdfa0 2023-10-16T13:46:08.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.984+0000 7f937effd700 1 -- 172.21.15.16:0/4013132115 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93800038e0 con 0x7f93900cdfa0 2023-10-16T13:46:08.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.984+0000 7f9394bb7700 1 -- 172.21.15.16:0/4013132115 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93901f7700 con 0x7f93900cdfa0 2023-10-16T13:46:08.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.985+0000 7f9394bb7700 1 -- 172.21.15.16:0/4013132115 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9390066970 con 0x7f93900cdfa0 2023-10-16T13:46:08.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.986+0000 7f937effd700 1 -- 172.21.15.16:0/4013132115 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 134) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f9380003a40 con 0x7f93900cdfa0 2023-10-16T13:46:08.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.986+0000 7f937effd700 1 --2- 172.21.15.16:0/4013132115 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f937006b140 0x7f937006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:08.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.986+0000 7f938dd9b700 1 -- 172.21.15.16:0/4013132115 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f937006b140 msgr2=0x7f937006d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3220139196 2023-10-16T13:46:08.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.986+0000 7f937effd700 1 -- 172.21.15.16:0/4013132115 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6660+0+0 (secure 0 0 0) 0x7f9380021070 con 0x7f93900cdfa0 2023-10-16T13:46:08.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.986+0000 7f938dd9b700 1 --2- 172.21.15.16:0/4013132115 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f937006b140 0x7f937006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:46:08.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:08.991+0000 7f937effd700 1 -- 172.21.15.16:0/4013132115 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9380055fc0 con 0x7f93900cdfa0 2023-10-16T13:46:09.101 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:09.099+0000 7f9d0ecc4200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T13:46:09.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.187+0000 7f938dd9b700 1 -- 172.21.15.16:0/4013132115 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f937006b140 msgr2=0x7f937006d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3220139196 2023-10-16T13:46:09.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.187+0000 7f938dd9b700 1 --2- 172.21.15.16:0/4013132115 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f937006b140 0x7f937006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:46:09.193 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:46:09.191+0000 7f3e59442200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T13:46:09.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.255+0000 7f9394bb7700 1 -- 172.21.15.16:0/4013132115 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f939006cb70 con 0x7f93900cdfa0 2023-10-16T13:46:09.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.255+0000 7f937effd700 1 -- 172.21.15.16:0/4013132115 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+4925 (secure 0 0 0) 0x7f93800595e0 con 0x7f93900cdfa0 2023-10-16T13:46:09.258 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:46:09.259 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":104,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8729,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/3614602930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":3614602930},{"type":"v1","addr":"172.21.15.49:6835","nonce":3614602930}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":99},{"gid":10045,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/3427940027","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":3427940027},{"type":"v1","addr":"172.21.15.49:6834","nonce":3427940027}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":99},{"gid":10896,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/4279896386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":4279896386},{"type":"v1","addr":"172.21.15.16:6836","nonce":4279896386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:44:40.208202+0000","modified":"2023-10-16T13:44:43.255450+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10899},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10899":{"gid":10899,"name":"a","rank":0,"incarnation":100,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/4233835071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":4233835071},{"type":"v1","addr":"172.21.15.16:6837","nonce":4233835071}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2023-10-16T13:46:09.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.258+0000 7f9394bb7700 1 -- 172.21.15.16:0/4013132115 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f937006b140 msgr2=0x7f937006d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:46:09.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.258+0000 7f9394bb7700 1 --2- 172.21.15.16:0/4013132115 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f937006b140 0x7f937006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:09.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.258+0000 7f9394bb7700 1 -- 172.21.15.16:0/4013132115 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93900cdfa0 msgr2=0x7f93900c73f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:09.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.258+0000 7f9394bb7700 1 --2- 172.21.15.16:0/4013132115 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93900cdfa0 0x7f93900c73f0 secure :-1 s=READY pgs=2374 cs=0 l=1 rev1=1 crypto rx=0x7f9380009b60 tx=0x7f9380004750 comp rx=0 tx=0).stop 2023-10-16T13:46:09.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.258+0000 7f9394bb7700 1 -- 172.21.15.16:0/4013132115 shutdown_connections 2023-10-16T13:46:09.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.258+0000 7f9394bb7700 1 --2- 172.21.15.16:0/4013132115 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f93900d04a0 0x7f93900c9090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:09.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.258+0000 7f9394bb7700 1 --2- 172.21.15.16:0/4013132115 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f937006b140 0x7f937006d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:09.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.258+0000 7f9394bb7700 1 --2- 172.21.15.16:0/4013132115 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93900cdfa0 0x7f93900c73f0 unknown :-1 s=CLOSED pgs=2374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:09.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.258+0000 7f9394bb7700 1 --2- 172.21.15.16:0/4013132115 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f93900ce380 0x7f9390155b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:09.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.258+0000 7f9394bb7700 1 -- 172.21.15.16:0/4013132115 >> 172.21.15.16:0/4013132115 conn(0x7f93900d14b0 msgr2=0x7f93900b36d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:09.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.258+0000 7f9394bb7700 1 -- 172.21.15.16:0/4013132115 shutdown_connections 2023-10-16T13:46:09.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.258+0000 7f9394bb7700 1 -- 172.21.15.16:0/4013132115 wait complete. 2023-10-16T13:46:09.263 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 104 2023-10-16T13:46:09.273 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:46:09.362 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:46:09.360+0000 7f3e59442200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T13:46:09.381 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:09.379+0000 7f9d0ecc4200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T13:46:09.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.397+0000 7fddc08db700 1 Processor -- start 2023-10-16T13:46:09.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.397+0000 7fddc08db700 1 -- start start 2023-10-16T13:46:09.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.398+0000 7fddc08db700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fddbc0cdfa0 0x7fddbc0b2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:09.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.398+0000 7fddc08db700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fddbc0d0bb0 0x7fddbc0b2980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:09.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.398+0000 7fddc08db700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddbc0d0fe0 con 0x7fddbc0ce380 2023-10-16T13:46:09.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.398+0000 7fddc08db700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddbc0d1210 con 0x7fddbc0d0bb0 2023-10-16T13:46:09.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.398+0000 7fddc08db700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddbc0cbc30 con 0x7fddbc0cdfa0 2023-10-16T13:46:09.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.398+0000 7fddba59c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fddbc0cdfa0 0x7fddbc0b2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:09.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.398+0000 7fddbad9d700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fddbc0d0bb0 0x7fddbc0b2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:09.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.398+0000 7fddba59c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fddbc0cdfa0 0x7fddbc0b2440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:40858/0 (socket says 172.21.15.16:40858) 2023-10-16T13:46:09.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.398+0000 7fddbad9d700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fddbc0d0bb0 0x7fddbc0b2980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:51666/0 (socket says 172.21.15.16:51666) 2023-10-16T13:46:09.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.398+0000 7fddba59c700 1 -- 172.21.15.16:0/195535264 learned_addr learned my addr 172.21.15.16:0/195535264 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:09.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.398+0000 7fddb9d9b700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7fddbc0ce380 0x7fddbc0d04a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:49056/0 (socket says 172.21.15.16:49056) 2023-10-16T13:46:09.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.399+0000 7fddbad9d700 1 -- 172.21.15.16:0/195535264 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fddbc0cdfa0 msgr2=0x7fddbc0b2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:09.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.399+0000 7fddbad9d700 1 --2- 172.21.15.16:0/195535264 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fddbc0cdfa0 0x7fddbc0b2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:09.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.399+0000 7fddbad9d700 1 -- 172.21.15.16:0/195535264 >> v1:172.21.15.49:6789/0 conn(0x7fddbc0ce380 legacy=0x7fddbc0d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:09.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.399+0000 7fddbad9d700 1 -- 172.21.15.16:0/195535264 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddbc0cbf10 con 0x7fddbc0d0bb0 2023-10-16T13:46:09.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.399+0000 7fddba59c700 1 --2- 172.21.15.16:0/195535264 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fddbc0cdfa0 0x7fddbc0b2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:46:09.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.399+0000 7fddbad9d700 1 --2- 172.21.15.16:0/195535264 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fddbc0d0bb0 0x7fddbc0b2980 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fddb4009d90 tx=0x7fddb400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=841753f59eb73880 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:09.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.399+0000 7fddb959a700 1 -- 172.21.15.16:0/195535264 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddb400e070 con 0x7fddbc0d0bb0 2023-10-16T13:46:09.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.399+0000 7fddb959a700 1 -- 172.21.15.16:0/195535264 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddb400bb20 con 0x7fddbc0d0bb0 2023-10-16T13:46:09.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.399+0000 7fddb959a700 1 -- 172.21.15.16:0/195535264 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddb4012430 con 0x7fddbc0d0bb0 2023-10-16T13:46:09.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.400+0000 7fddc08db700 1 -- 172.21.15.16:0/195535264 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fddbc0d0bb0 msgr2=0x7fddbc0b2980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:09.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.400+0000 7fddc08db700 1 --2- 172.21.15.16:0/195535264 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fddbc0d0bb0 0x7fddbc0b2980 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fddb4009d90 tx=0x7fddb400b330 comp rx=0 tx=0).stop 2023-10-16T13:46:09.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.400+0000 7fddc08db700 1 -- 172.21.15.16:0/195535264 shutdown_connections 2023-10-16T13:46:09.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.400+0000 7fddc08db700 1 --2- 172.21.15.16:0/195535264 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fddbc0d0bb0 0x7fddbc0b2980 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:09.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.400+0000 7fddc08db700 1 --2- 172.21.15.16:0/195535264 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fddbc0cdfa0 0x7fddbc0b2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:09.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.400+0000 7fddc08db700 1 -- 172.21.15.16:0/195535264 >> 172.21.15.16:0/195535264 conn(0x7fddbc0d14b0 msgr2=0x7fddbc0d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:09.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.400+0000 7fddc08db700 1 -- 172.21.15.16:0/195535264 shutdown_connections 2023-10-16T13:46:09.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.400+0000 7fddc08db700 1 -- 172.21.15.16:0/195535264 wait complete. 2023-10-16T13:46:09.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.401+0000 7fddc08db700 1 Processor -- start 2023-10-16T13:46:09.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.401+0000 7fddc08db700 1 -- start start 2023-10-16T13:46:09.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.402+0000 7fddc08db700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddbc0cdfa0 0x7fddbc0700e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:09.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.402+0000 7fddc08db700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fddbc0ce380 0x7fddbc1f2d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:09.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.402+0000 7fddc08db700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fddbc0d0bb0 0x7fddbc0c8e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:09.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.402+0000 7fddc08db700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddbc0c62d0 con 0x7fddbc0cdfa0 2023-10-16T13:46:09.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.402+0000 7fddc08db700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddbc0c65b0 con 0x7fddbc0ce380 2023-10-16T13:46:09.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.402+0000 7fddc08db700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddbc0c6890 con 0x7fddbc0d0bb0 2023-10-16T13:46:09.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.402+0000 7fddba59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddbc0cdfa0 0x7fddbc0700e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:09.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.402+0000 7fddba59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddbc0cdfa0 0x7fddbc0700e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52606/0 (socket says 172.21.15.16:52606) 2023-10-16T13:46:09.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.402+0000 7fddb9d9b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fddbc0ce380 0x7fddbc1f2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:09.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.402+0000 7fddba59c700 1 -- 172.21.15.16:0/858146418 learned_addr learned my addr 172.21.15.16:0/858146418 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:09.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.402+0000 7fddbad9d700 1 --2- 172.21.15.16:0/858146418 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fddbc0d0bb0 0x7fddbc0c8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:09.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.403+0000 7fddb9d9b700 1 -- 172.21.15.16:0/858146418 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fddbc0d0bb0 msgr2=0x7fddbc0c8e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:09.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.403+0000 7fddb9d9b700 1 --2- 172.21.15.16:0/858146418 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fddbc0d0bb0 0x7fddbc0c8e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:09.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.403+0000 7fddb9d9b700 1 -- 172.21.15.16:0/858146418 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddbc0cdfa0 msgr2=0x7fddbc0700e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:09.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.403+0000 7fddb9d9b700 1 --2- 172.21.15.16:0/858146418 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddbc0cdfa0 0x7fddbc0700e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:09.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.403+0000 7fddb9d9b700 1 -- 172.21.15.16:0/858146418 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddac009550 con 0x7fddbc0ce380 2023-10-16T13:46:09.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.403+0000 7fddb9d9b700 1 --2- 172.21.15.16:0/858146418 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fddbc0ce380 0x7fddbc1f2d30 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fddb000c450 tx=0x7fddb000c760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:09.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.403+0000 7fddaaffd700 1 -- 172.21.15.16:0/858146418 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddb0005cb0 con 0x7fddbc0ce380 2023-10-16T13:46:09.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.403+0000 7fddaaffd700 1 -- 172.21.15.16:0/858146418 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddb0005e10 con 0x7fddbc0ce380 2023-10-16T13:46:09.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.403+0000 7fddc08db700 1 -- 172.21.15.16:0/858146418 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddb4009a70 con 0x7fddbc0ce380 2023-10-16T13:46:09.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.403+0000 7fddaaffd700 1 -- 172.21.15.16:0/858146418 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddb00072d0 con 0x7fddbc0ce380 2023-10-16T13:46:09.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.403+0000 7fddc08db700 1 -- 172.21.15.16:0/858146418 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddbc0c6e50 con 0x7fddbc0ce380 2023-10-16T13:46:09.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.404+0000 7fddc08db700 1 -- 172.21.15.16:0/858146418 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddbc04ebb0 con 0x7fddbc0ce380 2023-10-16T13:46:09.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.404+0000 7fddaaffd700 1 -- 172.21.15.16:0/858146418 <== mon.1 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 134) v1 ==== 89445+0+0 (secure 0 0 0) 0x7fddb0007430 con 0x7fddbc0ce380 2023-10-16T13:46:09.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.405+0000 7fddaaffd700 1 --2- 172.21.15.16:0/858146418 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7fdd9c06b140 0x7fdd9c06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:09.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.405+0000 7fddba59c700 1 -- 172.21.15.16:0/858146418 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7fdd9c06b140 msgr2=0x7fdd9c06d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3220139196 2023-10-16T13:46:09.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.405+0000 7fddaaffd700 1 -- 172.21.15.16:0/858146418 <== mon.1 v2:172.21.15.49:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6660+0+0 (secure 0 0 0) 0x7fddb0088ef0 con 0x7fddbc0ce380 2023-10-16T13:46:09.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.405+0000 7fddba59c700 1 --2- 172.21.15.16:0/858146418 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7fdd9c06b140 0x7fdd9c06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:46:09.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.409+0000 7fddaaffd700 1 -- 172.21.15.16:0/858146418 <== mon.1 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fddb00539c0 con 0x7fddbc0ce380 2023-10-16T13:46:09.558 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:09.556+0000 7f9d0ecc4200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T13:46:09.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.606+0000 7fddba59c700 1 -- 172.21.15.16:0/858146418 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7fdd9c06b140 msgr2=0x7fdd9c06d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3220139196 2023-10-16T13:46:09.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.606+0000 7fddba59c700 1 --2- 172.21.15.16:0/858146418 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7fdd9c06b140 0x7fdd9c06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:46:09.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.619+0000 7fddc08db700 1 -- 172.21.15.16:0/858146418 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fddbc0c7010 con 0x7fddbc0ce380 2023-10-16T13:46:09.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.621+0000 7fddaaffd700 1 -- 172.21.15.16:0/858146418 <== mon.1 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v212) v1 ==== 74+0+17654 (secure 0 0 0) 0x7fddb0017020 con 0x7fddbc0ce380 2023-10-16T13:46:09.623 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:46:09.624 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":212,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:46:07.496780+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2023-10-16T13:44:38.716950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"205","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_data","create_time":"2023-10-16T13:44:39.172499+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"212","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":212,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":204,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":204,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":204,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":204,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":204,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":204,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"26.5","mappings":[{"from":1,"to":0}]},{"pgid":"26.21","mappings":[{"from":1,"to":0}]},{"pgid":"26.24","mappings":[{"from":3,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/3477190137":"2023-10-17T13:44:55.326924+0000","172.21.15.16:0/1906189720":"2023-10-17T13:44:55.326924+0000","172.21.15.16:6833/1268439414":"2023-10-17T13:44:55.326924+0000","172.21.15.16:6832/1268439414":"2023-10-17T13:44:55.326924+0000","172.21.15.16:0/3697646183":"2023-10-17T13:44:55.326924+0000","172.21.15.16:0/4067982926":"2023-10-17T13:44:55.326924+0000","172.21.15.16:0/3408848304":"2023-10-17T13:44:55.326924+0000","172.21.15.16:0/2782518306":"2023-10-17T13:44:55.326924+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[{"pool":26,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:46:09.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.623+0000 7fddc08db700 1 -- 172.21.15.16:0/858146418 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7fdd9c06b140 msgr2=0x7fdd9c06d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:46:09.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.623+0000 7fddc08db700 1 --2- 172.21.15.16:0/858146418 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7fdd9c06b140 0x7fdd9c06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:09.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.623+0000 7fddc08db700 1 -- 172.21.15.16:0/858146418 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fddbc0ce380 msgr2=0x7fddbc1f2d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:09.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.623+0000 7fddc08db700 1 --2- 172.21.15.16:0/858146418 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fddbc0ce380 0x7fddbc1f2d30 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fddb000c450 tx=0x7fddb000c760 comp rx=0 tx=0).stop 2023-10-16T13:46:09.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.623+0000 7fddc08db700 1 -- 172.21.15.16:0/858146418 shutdown_connections 2023-10-16T13:46:09.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.623+0000 7fddc08db700 1 --2- 172.21.15.16:0/858146418 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fddbc0d0bb0 0x7fddbc0c8e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:09.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.623+0000 7fddc08db700 1 --2- 172.21.15.16:0/858146418 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7fdd9c06b140 0x7fdd9c06d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:09.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.623+0000 7fddc08db700 1 --2- 172.21.15.16:0/858146418 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddbc0cdfa0 0x7fddbc0700e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:09.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.623+0000 7fddc08db700 1 --2- 172.21.15.16:0/858146418 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fddbc0ce380 0x7fddbc1f2d30 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:09.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.623+0000 7fddc08db700 1 -- 172.21.15.16:0/858146418 >> 172.21.15.16:0/858146418 conn(0x7fddbc0d14b0 msgr2=0x7fddbc0b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:09.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.623+0000 7fddc08db700 1 -- 172.21.15.16:0/858146418 shutdown_connections 2023-10-16T13:46:09.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.623+0000 7fddc08db700 1 -- 172.21.15.16:0/858146418 wait complete. 2023-10-16T13:46:09.639 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-16T13:46:09.639 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:46:09.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.764+0000 7f903c393700 1 Processor -- start 2023-10-16T13:46:09.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.764+0000 7f903c393700 1 -- start start 2023-10-16T13:46:09.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.765+0000 7f903c393700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f903415d900 0x7f90340d2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:09.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.765+0000 7f903c393700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f90340ce9f0 0x7f90340cdc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:09.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.765+0000 7f903c393700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f90340d2860 0x7f90340d2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:09.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.765+0000 7f903c393700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90340d3230 con 0x7f90340d2860 2023-10-16T13:46:09.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.765+0000 7f903c393700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90340ce1a0 con 0x7f903415d900 2023-10-16T13:46:09.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.765+0000 7f903c393700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90340cee70 con 0x7f90340ce9f0 2023-10-16T13:46:09.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.765+0000 7f903a930700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f90340d2860 0x7f90340d2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:09.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.765+0000 7f903a930700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f90340d2860 0x7f90340d2cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52618/0 (socket says 172.21.15.16:52618) 2023-10-16T13:46:09.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.765+0000 7f903a930700 1 -- 172.21.15.16:0/592579861 learned_addr learned my addr 172.21.15.16:0/592579861 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:09.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.765+0000 7f903a12f700 1 --2- 172.21.15.16:0/592579861 >> v2:172.21.15.49:3300/0 conn(0x7f903415d900 0x7f90340d2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:09.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.765+0000 7f903992e700 1 --2- 172.21.15.16:0/592579861 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f90340ce9f0 0x7f90340cdc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:09.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.765+0000 7f903a930700 1 -- 172.21.15.16:0/592579861 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f90340ce9f0 msgr2=0x7f90340cdc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:09.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.765+0000 7f903a930700 1 --2- 172.21.15.16:0/592579861 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f90340ce9f0 0x7f90340cdc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:09.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.765+0000 7f903a930700 1 -- 172.21.15.16:0/592579861 >> v2:172.21.15.49:3300/0 conn(0x7f903415d900 msgr2=0x7f90340d2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:09.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.765+0000 7f903a930700 1 --2- 172.21.15.16:0/592579861 >> v2:172.21.15.49:3300/0 conn(0x7f903415d900 0x7f90340d2040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:09.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.765+0000 7f903a930700 1 -- 172.21.15.16:0/592579861 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90340ce400 con 0x7f90340d2860 2023-10-16T13:46:09.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.765+0000 7f903a930700 1 --2- 172.21.15.16:0/592579861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90340d2860 0x7f90340d2cf0 secure :-1 s=READY pgs=2375 cs=0 l=1 rev1=1 crypto rx=0x7f90340d4710 tx=0x7f902c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaa7356664ab9f5e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:09.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.766+0000 7f903912d700 1 -- 172.21.15.16:0/592579861 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f902c00e070 con 0x7f90340d2860 2023-10-16T13:46:09.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.766+0000 7f903912d700 1 -- 172.21.15.16:0/592579861 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f902c00b910 con 0x7f90340d2860 2023-10-16T13:46:09.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.766+0000 7f903912d700 1 -- 172.21.15.16:0/592579861 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f902c00bcc0 con 0x7f90340d2860 2023-10-16T13:46:09.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.766+0000 7f903c393700 1 -- 172.21.15.16:0/592579861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90340d2860 msgr2=0x7f90340d2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:09.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.766+0000 7f903c393700 1 --2- 172.21.15.16:0/592579861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90340d2860 0x7f90340d2cf0 secure :-1 s=READY pgs=2375 cs=0 l=1 rev1=1 crypto rx=0x7f90340d4710 tx=0x7f902c00b330 comp rx=0 tx=0).stop 2023-10-16T13:46:09.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.766+0000 7f903c393700 1 -- 172.21.15.16:0/592579861 shutdown_connections 2023-10-16T13:46:09.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.766+0000 7f903c393700 1 --2- 172.21.15.16:0/592579861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90340d2860 0x7f90340d2cf0 unknown :-1 s=CLOSED pgs=2375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:09.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.766+0000 7f903c393700 1 --2- 172.21.15.16:0/592579861 >> v2:172.21.15.49:3300/0 conn(0x7f903415d900 0x7f90340d2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:09.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.766+0000 7f903c393700 1 --2- 172.21.15.16:0/592579861 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f90340ce9f0 0x7f90340cdc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:09.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.766+0000 7f903c393700 1 -- 172.21.15.16:0/592579861 >> 172.21.15.16:0/592579861 conn(0x7f9034072330 msgr2=0x7f9034072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:09.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.766+0000 7f903c393700 1 -- 172.21.15.16:0/592579861 shutdown_connections 2023-10-16T13:46:09.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.766+0000 7f903c393700 1 -- 172.21.15.16:0/592579861 wait complete. 2023-10-16T13:46:09.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.768+0000 7f903c393700 1 Processor -- start 2023-10-16T13:46:09.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.768+0000 7f903c393700 1 -- start start 2023-10-16T13:46:09.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.768+0000 7f903c393700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f90340ce9f0 0x7f90341f5c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:09.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.768+0000 7f903c393700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f90340d2860 0x7f9034154410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:09.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.768+0000 7f903c393700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f903415d900 0x7f90340c92f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:09.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.768+0000 7f903c393700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f903415e2d0 con 0x7f903415d900 2023-10-16T13:46:09.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.768+0000 7f903c393700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9034154ed0 con 0x7f90340ce9f0 2023-10-16T13:46:09.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.768+0000 7f903c393700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90341551b0 con 0x7f90340d2860 2023-10-16T13:46:09.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.769+0000 7f903a930700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f903415d900 0x7f90340c92f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:09.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.769+0000 7f903a12f700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f90340ce9f0 0x7f90341f5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:09.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.769+0000 7f903992e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f90340d2860 0x7f9034154410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:09.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.769+0000 7f903a930700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f903415d900 0x7f90340c92f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52624/0 (socket says 172.21.15.16:52624) 2023-10-16T13:46:09.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.769+0000 7f903a930700 1 -- 172.21.15.16:0/1213506834 learned_addr learned my addr 172.21.15.16:0/1213506834 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:09.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.769+0000 7f903a12f700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f90340ce9f0 0x7f90341f5c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:51682/0 (socket says 172.21.15.16:51682) 2023-10-16T13:46:09.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.769+0000 7f903a930700 1 -- 172.21.15.16:0/1213506834 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f90340d2860 msgr2=0x7f9034154410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:09.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.769+0000 7f903a930700 1 --2- 172.21.15.16:0/1213506834 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f90340d2860 0x7f9034154410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:09.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.769+0000 7f903a930700 1 -- 172.21.15.16:0/1213506834 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f90340ce9f0 msgr2=0x7f90341f5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:09.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.769+0000 7f903a930700 1 --2- 172.21.15.16:0/1213506834 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f90340ce9f0 0x7f90341f5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:09.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.769+0000 7f903a930700 1 -- 172.21.15.16:0/1213506834 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f902c009b30 con 0x7f903415d900 2023-10-16T13:46:09.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.769+0000 7f903a930700 1 --2- 172.21.15.16:0/1213506834 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f903415d900 0x7f90340c92f0 secure :-1 s=READY pgs=2376 cs=0 l=1 rev1=1 crypto rx=0x7f90340d0f20 tx=0x7f902c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:09.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.769+0000 7f902affd700 1 -- 172.21.15.16:0/1213506834 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f902c0099b0 con 0x7f903415d900 2023-10-16T13:46:09.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.769+0000 7f902affd700 1 -- 172.21.15.16:0/1213506834 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f902c00ce80 con 0x7f903415d900 2023-10-16T13:46:09.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.769+0000 7f902affd700 1 -- 172.21.15.16:0/1213506834 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f902c004490 con 0x7f903415d900 2023-10-16T13:46:09.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.769+0000 7f903c393700 1 -- 172.21.15.16:0/1213506834 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9034155490 con 0x7f903415d900 2023-10-16T13:46:09.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.769+0000 7f903c393700 1 -- 172.21.15.16:0/1213506834 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9034149cf0 con 0x7f903415d900 2023-10-16T13:46:09.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.770+0000 7f903c393700 1 -- 172.21.15.16:0/1213506834 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f903415e410 con 0x7f903415d900 2023-10-16T13:46:09.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.771+0000 7f902affd700 1 -- 172.21.15.16:0/1213506834 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 134) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f902c0045f0 con 0x7f903415d900 2023-10-16T13:46:09.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.771+0000 7f902affd700 1 --2- 172.21.15.16:0/1213506834 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f901806b140 0x7f901806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:09.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.771+0000 7f903a12f700 1 -- 172.21.15.16:0/1213506834 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f901806b140 msgr2=0x7f901806d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3220139196 2023-10-16T13:46:09.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.771+0000 7f902affd700 1 -- 172.21.15.16:0/1213506834 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6660+0+0 (secure 0 0 0) 0x7f902c08b480 con 0x7f903415d900 2023-10-16T13:46:09.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.771+0000 7f903a12f700 1 --2- 172.21.15.16:0/1213506834 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f901806b140 0x7f901806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:46:09.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.776+0000 7f902affd700 1 -- 172.21.15.16:0/1213506834 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f903415e410 con 0x7f903415d900 2023-10-16T13:46:09.921 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:46:09.918+0000 7f3e59442200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T13:46:09.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.972+0000 7f903a12f700 1 -- 172.21.15.16:0/1213506834 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f901806b140 msgr2=0x7f901806d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3220139196 2023-10-16T13:46:09.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:09.972+0000 7f903a12f700 1 --2- 172.21.15.16:0/1213506834 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f901806b140 0x7f901806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:46:10.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.044+0000 7f903c393700 1 -- 172.21.15.16:0/1213506834 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f903415e410 con 0x7f903415d900 2023-10-16T13:46:10.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.044+0000 7f902affd700 1 -- 172.21.15.16:0/1213506834 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+4925 (secure 0 0 0) 0x7f903415e410 con 0x7f903415d900 2023-10-16T13:46:10.047 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:46:10.047 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":104,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8729,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/3614602930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":3614602930},{"type":"v1","addr":"172.21.15.49:6835","nonce":3614602930}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":99},{"gid":10045,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/3427940027","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":3427940027},{"type":"v1","addr":"172.21.15.49:6834","nonce":3427940027}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":99},{"gid":10896,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/4279896386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":4279896386},{"type":"v1","addr":"172.21.15.16:6836","nonce":4279896386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:44:40.208202+0000","modified":"2023-10-16T13:44:43.255450+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10899},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10899":{"gid":10899,"name":"a","rank":0,"incarnation":100,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/4233835071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":4233835071},{"type":"v1","addr":"172.21.15.16:6837","nonce":4233835071}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2023-10-16T13:46:10.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.048+0000 7f903c393700 1 -- 172.21.15.16:0/1213506834 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f901806b140 msgr2=0x7f901806d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:46:10.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.048+0000 7f903c393700 1 --2- 172.21.15.16:0/1213506834 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f901806b140 0x7f901806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:10.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.048+0000 7f903c393700 1 -- 172.21.15.16:0/1213506834 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f903415d900 msgr2=0x7f90340c92f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:10.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.048+0000 7f903c393700 1 --2- 172.21.15.16:0/1213506834 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f903415d900 0x7f90340c92f0 secure :-1 s=READY pgs=2376 cs=0 l=1 rev1=1 crypto rx=0x7f90340d0f20 tx=0x7f902c004ad0 comp rx=0 tx=0).stop 2023-10-16T13:46:10.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.048+0000 7f903c393700 1 -- 172.21.15.16:0/1213506834 shutdown_connections 2023-10-16T13:46:10.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.048+0000 7f903c393700 1 --2- 172.21.15.16:0/1213506834 >> [v2:172.21.15.16:6832/3220139196,v1:172.21.15.16:6833/3220139196] conn(0x7f901806b140 0x7f901806d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:10.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.048+0000 7f903c393700 1 --2- 172.21.15.16:0/1213506834 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f903415d900 0x7f90340c92f0 unknown :-1 s=CLOSED pgs=2376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:10.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.048+0000 7f903c393700 1 --2- 172.21.15.16:0/1213506834 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f90340ce9f0 0x7f90341f5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:10.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.048+0000 7f903c393700 1 --2- 172.21.15.16:0/1213506834 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f90340d2860 0x7f9034154410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:10.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.048+0000 7f903c393700 1 -- 172.21.15.16:0/1213506834 >> 172.21.15.16:0/1213506834 conn(0x7f9034072330 msgr2=0x7f90340ccb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:10.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.048+0000 7f903c393700 1 -- 172.21.15.16:0/1213506834 shutdown_connections 2023-10-16T13:46:10.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.048+0000 7f903c393700 1 -- 172.21.15.16:0/1213506834 wait complete. 2023-10-16T13:46:10.053 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 104 2023-10-16T13:46:10.063 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:46:10.148 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:10.146+0000 7f9d0ecc4200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T13:46:10.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.192+0000 7fcc4da25700 1 Processor -- start 2023-10-16T13:46:10.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.192+0000 7fcc4da25700 1 -- start start 2023-10-16T13:46:10.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.193+0000 7fcc4da25700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcc480cbed0 0x7fcc4806eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:10.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.193+0000 7fcc4da25700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc4806f0a0 con 0x7fcc4806d350 2023-10-16T13:46:10.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.193+0000 7fcc4da25700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc4806f380 con 0x7fcc4815a870 2023-10-16T13:46:10.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.193+0000 7fcc4da25700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc4815ac50 con 0x7fcc480cbed0 2023-10-16T13:46:10.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.193+0000 7fcc477fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcc480cbed0 0x7fcc4806eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:10.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.193+0000 7fcc467fc700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fcc4806d350 0x7fcc480cde10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41924/0 (socket says 172.21.15.16:41924) 2023-10-16T13:46:10.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.193+0000 7fcc467fc700 1 -- 172.21.15.16:0/2865757277 learned_addr learned my addr 172.21.15.16:0/2865757277 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:46:10.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.194+0000 7fcc477fe700 1 -- 172.21.15.16:0/2865757277 >> v1:172.21.15.49:6789/0 conn(0x7fcc4815a870 legacy=0x7fcc480cb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:10.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.194+0000 7fcc477fe700 1 -- 172.21.15.16:0/2865757277 >> v1:172.21.15.16:6789/0 conn(0x7fcc4806d350 legacy=0x7fcc480cde10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:10.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.194+0000 7fcc477fe700 1 -- 172.21.15.16:0/2865757277 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc4806f660 con 0x7fcc480cbed0 2023-10-16T13:46:10.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.194+0000 7fcc45ffb700 1 -- 172.21.15.16:0/2865757277 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2103092006 0 0) 0x7fcc4806f0a0 con 0x7fcc4806d350 2023-10-16T13:46:10.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.194+0000 7fcc477fe700 1 --2- 172.21.15.16:0/2865757277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc480cbed0 0x7fcc4806eaa0 secure :-1 s=READY pgs=2381 cs=0 l=1 rev1=1 crypto rx=0x7fcc4000bfa0 tx=0x7fcc4000d330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91514f001cf0369a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:10.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.194+0000 7fcc45ffb700 1 -- 172.21.15.16:0/2865757277 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc40010070 con 0x7fcc480cbed0 2023-10-16T13:46:10.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.194+0000 7fcc45ffb700 1 -- 172.21.15.16:0/2865757277 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc4000b2e0 con 0x7fcc480cbed0 2023-10-16T13:46:10.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.194+0000 7fcc45ffb700 1 -- 172.21.15.16:0/2865757277 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc4000dcf0 con 0x7fcc480cbed0 2023-10-16T13:46:10.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.195+0000 7fcc4da25700 1 -- 172.21.15.16:0/2865757277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc480cbed0 msgr2=0x7fcc4806eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:10.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.195+0000 7fcc4da25700 1 --2- 172.21.15.16:0/2865757277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc480cbed0 0x7fcc4806eaa0 secure :-1 s=READY pgs=2381 cs=0 l=1 rev1=1 crypto rx=0x7fcc4000bfa0 tx=0x7fcc4000d330 comp rx=0 tx=0).stop 2023-10-16T13:46:10.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.195+0000 7fcc4da25700 1 -- 172.21.15.16:0/2865757277 shutdown_connections 2023-10-16T13:46:10.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.195+0000 7fcc4da25700 1 --2- 172.21.15.16:0/2865757277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc480cbed0 0x7fcc4806eaa0 unknown :-1 s=CLOSED pgs=2381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:10.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.195+0000 7fcc4da25700 1 -- 172.21.15.16:0/2865757277 >> 172.21.15.16:0/2865757277 conn(0x7fcc480cc5b0 msgr2=0x7fcc480cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:10.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.195+0000 7fcc4da25700 1 -- 172.21.15.16:0/2865757277 shutdown_connections 2023-10-16T13:46:10.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.195+0000 7fcc4da25700 1 -- 172.21.15.16:0/2865757277 wait complete. 2023-10-16T13:46:10.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.196+0000 7fcc4da25700 1 Processor -- start 2023-10-16T13:46:10.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.196+0000 7fcc4da25700 1 -- start start 2023-10-16T13:46:10.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.197+0000 7fcc4da25700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcc4806d350 0x7fcc48145600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:10.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.197+0000 7fcc4da25700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcc480cbed0 0x7fcc4814a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:10.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.197+0000 7fcc4da25700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc4815a870 0x7fcc4814ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:10.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.197+0000 7fcc4da25700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc481570e0 con 0x7fcc4815a870 2023-10-16T13:46:10.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.197+0000 7fcc4da25700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc48157350 con 0x7fcc4806d350 2023-10-16T13:46:10.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.197+0000 7fcc4da25700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc48157630 con 0x7fcc480cbed0 2023-10-16T13:46:10.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.197+0000 7fcc477fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc4815a870 0x7fcc4814ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:10.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.197+0000 7fcc477fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc4815a870 0x7fcc4814ad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52658/0 (socket says 172.21.15.16:52658) 2023-10-16T13:46:10.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.197+0000 7fcc477fe700 1 -- 172.21.15.16:0/3335218668 learned_addr learned my addr 172.21.15.16:0/3335218668 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:10.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.198+0000 7fcc467fc700 1 --2- 172.21.15.16:0/3335218668 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcc480cbed0 0x7fcc4814a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:10.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.198+0000 7fcc46ffd700 1 --2- 172.21.15.16:0/3335218668 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcc4806d350 0x7fcc48145600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:10.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.198+0000 7fcc477fe700 1 -- 172.21.15.16:0/3335218668 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcc480cbed0 msgr2=0x7fcc4814a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:10.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.198+0000 7fcc477fe700 1 --2- 172.21.15.16:0/3335218668 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcc480cbed0 0x7fcc4814a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:10.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.198+0000 7fcc477fe700 1 -- 172.21.15.16:0/3335218668 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcc4806d350 msgr2=0x7fcc48145600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:10.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.198+0000 7fcc477fe700 1 --2- 172.21.15.16:0/3335218668 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcc4806d350 0x7fcc48145600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:10.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.198+0000 7fcc477fe700 1 -- 172.21.15.16:0/3335218668 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc4000bc80 con 0x7fcc4815a870 2023-10-16T13:46:10.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.198+0000 7fcc46ffd700 1 --2- 172.21.15.16:0/3335218668 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcc4806d350 0x7fcc48145600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:46:10.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.198+0000 7fcc467fc700 1 --2- 172.21.15.16:0/3335218668 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcc480cbed0 0x7fcc4814a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:46:10.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.198+0000 7fcc477fe700 1 --2- 172.21.15.16:0/3335218668 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc4815a870 0x7fcc4814ad10 secure :-1 s=READY pgs=2382 cs=0 l=1 rev1=1 crypto rx=0x7fcc480c3630 tx=0x7fcc40004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:10.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.198+0000 7fcc2ffff700 1 -- 172.21.15.16:0/3335218668 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc4000dea0 con 0x7fcc4815a870 2023-10-16T13:46:10.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.198+0000 7fcc2ffff700 1 -- 172.21.15.16:0/3335218668 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc4001e660 con 0x7fcc4815a870 2023-10-16T13:46:10.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.198+0000 7fcc4da25700 1 -- 172.21.15.16:0/3335218668 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc48157910 con 0x7fcc4815a870 2023-10-16T13:46:10.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.198+0000 7fcc2ffff700 1 -- 172.21.15.16:0/3335218668 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc4001ea10 con 0x7fcc4815a870 2023-10-16T13:46:10.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.198+0000 7fcc4da25700 1 -- 172.21.15.16:0/3335218668 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc48145b40 con 0x7fcc4815a870 2023-10-16T13:46:10.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.200+0000 7fcc4da25700 1 -- 172.21.15.16:0/3335218668 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc48157dd0 con 0x7fcc4815a870 2023-10-16T13:46:10.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.200+0000 7fcc2ffff700 1 -- 172.21.15.16:0/3335218668 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 135) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fcc4001eb70 con 0x7fcc4815a870 2023-10-16T13:46:10.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.200+0000 7fcc2ffff700 1 -- 172.21.15.16:0/3335218668 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 7046+0+0 (secure 0 0 0) 0x7fcc4008d510 con 0x7fcc4815a870 2023-10-16T13:46:10.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.206+0000 7fcc2ffff700 1 -- 172.21.15.16:0/3335218668 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcc40058570 con 0x7fcc4815a870 2023-10-16T13:46:10.236 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:10.233+0000 7f9c874aa700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:46:10.236 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:10.233+0000 7f9c874aa700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:46:10.236 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:10.233+0000 7f9c874aa700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:46:10.237 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:10.233+0000 7f9c874aa700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:46:10.237 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:10.233+0000 7f9c874aa700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:46:10.237 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:10.233+0000 7f9c8a4b0700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:46:10.237 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:10.233+0000 7f9c8a4b0700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:46:10.237 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:10.233+0000 7f9c8a4b0700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:46:10.237 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:10.233+0000 7f9c8a4b0700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:46:10.238 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:10.233+0000 7f9c8a4b0700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:46:10.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.461+0000 7fcc4da25700 1 -- 172.21.15.16:0/3335218668 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcc4804ebb0 con 0x7fcc4815a870 2023-10-16T13:46:10.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.462+0000 7fcc2ffff700 1 -- 172.21.15.16:0/3335218668 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+4925 (secure 0 0 0) 0x7fcc4005bb90 con 0x7fcc4815a870 2023-10-16T13:46:10.464 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:46:10.465 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":104,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8729,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/3614602930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":3614602930},{"type":"v1","addr":"172.21.15.49:6835","nonce":3614602930}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":99},{"gid":10045,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/3427940027","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":3427940027},{"type":"v1","addr":"172.21.15.49:6834","nonce":3427940027}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":99},{"gid":10896,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/4279896386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":4279896386},{"type":"v1","addr":"172.21.15.16:6836","nonce":4279896386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:44:40.208202+0000","modified":"2023-10-16T13:44:43.255450+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10899},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10899":{"gid":10899,"name":"a","rank":0,"incarnation":100,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/4233835071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":4233835071},{"type":"v1","addr":"172.21.15.16:6837","nonce":4233835071}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2023-10-16T13:46:10.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.464+0000 7fcc4da25700 1 -- 172.21.15.16:0/3335218668 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc4815a870 msgr2=0x7fcc4814ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:10.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.464+0000 7fcc4da25700 1 --2- 172.21.15.16:0/3335218668 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc4815a870 0x7fcc4814ad10 secure :-1 s=READY pgs=2382 cs=0 l=1 rev1=1 crypto rx=0x7fcc480c3630 tx=0x7fcc40004ad0 comp rx=0 tx=0).stop 2023-10-16T13:46:10.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.464+0000 7fcc4da25700 1 -- 172.21.15.16:0/3335218668 shutdown_connections 2023-10-16T13:46:10.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.464+0000 7fcc4da25700 1 --2- 172.21.15.16:0/3335218668 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc4815a870 0x7fcc4814ad10 unknown :-1 s=CLOSED pgs=2382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:10.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.464+0000 7fcc4da25700 1 --2- 172.21.15.16:0/3335218668 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcc4806d350 0x7fcc48145600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:10.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.464+0000 7fcc4da25700 1 --2- 172.21.15.16:0/3335218668 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcc480cbed0 0x7fcc4814a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:10.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.464+0000 7fcc4da25700 1 -- 172.21.15.16:0/3335218668 >> 172.21.15.16:0/3335218668 conn(0x7fcc480cc5b0 msgr2=0x7fcc4814c240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:10.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.464+0000 7fcc4da25700 1 -- 172.21.15.16:0/3335218668 shutdown_connections 2023-10-16T13:46:10.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.464+0000 7fcc4da25700 1 -- 172.21.15.16:0/3335218668 wait complete. 2023-10-16T13:46:10.468 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 104 2023-10-16T13:46:10.478 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:46:10.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.602+0000 7f38906bc700 1 Processor -- start 2023-10-16T13:46:10.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.602+0000 7f38906bc700 1 -- start start 2023-10-16T13:46:10.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.603+0000 7f38906bc700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f38880ce380 0x7f38880d0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:10.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.603+0000 7f38906bc700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38880d1210 con 0x7f38880b2b50 2023-10-16T13:46:10.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.603+0000 7f38906bc700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38880cbc30 con 0x7f38880cdfa0 2023-10-16T13:46:10.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.603+0000 7f38906bc700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38880cbf10 con 0x7f38880ce380 2023-10-16T13:46:10.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.603+0000 7f388dc57700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f38880b2b50 0x7f38880d04a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41988/0 (socket says 172.21.15.16:41988) 2023-10-16T13:46:10.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.603+0000 7f388dc57700 1 -- 172.21.15.16:0/1716490877 learned_addr learned my addr 172.21.15.16:0/1716490877 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:46:10.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.603+0000 7f388ec59700 1 --2- 172.21.15.16:0/1716490877 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f38880ce380 0x7f38880d0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:10.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.603+0000 7f388d456700 1 -- 172.21.15.16:0/1716490877 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 884832350 0 0) 0x7f38880d1210 con 0x7f38880b2b50 2023-10-16T13:46:10.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.603+0000 7f388d456700 1 -- 172.21.15.16:0/1716490877 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38740035b0 con 0x7f38880b2b50 2023-10-16T13:46:10.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.603+0000 7f388d456700 1 -- 172.21.15.16:0/1716490877 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1859713431 0 0) 0x7f38880cbc30 con 0x7f38880cdfa0 2023-10-16T13:46:10.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.603+0000 7f388d456700 1 -- 172.21.15.16:0/1716490877 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38880d1210 con 0x7f38880cdfa0 2023-10-16T13:46:10.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.603+0000 7f388ec59700 1 -- 172.21.15.16:0/1716490877 >> v1:172.21.15.49:6789/0 conn(0x7f38880cdfa0 legacy=0x7f38880b2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:10.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.603+0000 7f388ec59700 1 -- 172.21.15.16:0/1716490877 >> v1:172.21.15.16:6789/0 conn(0x7f38880b2b50 legacy=0x7f38880d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:10.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.603+0000 7f388ec59700 1 -- 172.21.15.16:0/1716490877 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38880cc1f0 con 0x7f38880ce380 2023-10-16T13:46:10.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.604+0000 7f388ec59700 1 --2- 172.21.15.16:0/1716490877 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f38880ce380 0x7f38880d0bb0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f3880009dc0 tx=0x7f388000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9133b1b7ece4e7ac server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:10.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.604+0000 7f388d456700 1 -- 172.21.15.16:0/1716490877 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f388000e070 con 0x7f38880ce380 2023-10-16T13:46:10.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.604+0000 7f388d456700 1 -- 172.21.15.16:0/1716490877 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f388000bb20 con 0x7f38880ce380 2023-10-16T13:46:10.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.604+0000 7f388d456700 1 -- 172.21.15.16:0/1716490877 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3880012430 con 0x7f38880ce380 2023-10-16T13:46:10.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.604+0000 7f38906bc700 1 -- 172.21.15.16:0/1716490877 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f38880ce380 msgr2=0x7f38880d0bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:10.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.604+0000 7f38906bc700 1 --2- 172.21.15.16:0/1716490877 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f38880ce380 0x7f38880d0bb0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f3880009dc0 tx=0x7f388000b330 comp rx=0 tx=0).stop 2023-10-16T13:46:10.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.604+0000 7f38906bc700 1 -- 172.21.15.16:0/1716490877 shutdown_connections 2023-10-16T13:46:10.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.604+0000 7f38906bc700 1 --2- 172.21.15.16:0/1716490877 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f38880ce380 0x7f38880d0bb0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:10.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.604+0000 7f38906bc700 1 -- 172.21.15.16:0/1716490877 >> 172.21.15.16:0/1716490877 conn(0x7f38880d14b0 msgr2=0x7f38880d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:10.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.604+0000 7f38906bc700 1 -- 172.21.15.16:0/1716490877 shutdown_connections 2023-10-16T13:46:10.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.604+0000 7f38906bc700 1 -- 172.21.15.16:0/1716490877 wait complete. 2023-10-16T13:46:10.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.606+0000 7f38906bc700 1 Processor -- start 2023-10-16T13:46:10.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.606+0000 7f38906bc700 1 -- start start 2023-10-16T13:46:10.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.606+0000 7f38906bc700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f38880b2b50 0x7f38880ca6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:10.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.606+0000 7f38906bc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38880cdfa0 0x7f388809d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:10.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.606+0000 7f38906bc700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f38880ce380 0x7f388809b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:10.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.606+0000 7f38906bc700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f388809e890 con 0x7f38880cdfa0 2023-10-16T13:46:10.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.606+0000 7f38906bc700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f388809dc10 con 0x7f38880b2b50 2023-10-16T13:46:10.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.606+0000 7f38906bc700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f388809def0 con 0x7f38880ce380 2023-10-16T13:46:10.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.606+0000 7f388dc57700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38880cdfa0 0x7f388809d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:10.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.606+0000 7f388dc57700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38880cdfa0 0x7f388809d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52720/0 (socket says 172.21.15.16:52720) 2023-10-16T13:46:10.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.606+0000 7f388dc57700 1 -- 172.21.15.16:0/3397202397 learned_addr learned my addr 172.21.15.16:0/3397202397 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:10.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.606+0000 7f388ec59700 1 --2- 172.21.15.16:0/3397202397 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f38880ce380 0x7f388809b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:10.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.606+0000 7f388e458700 1 --2- 172.21.15.16:0/3397202397 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f38880b2b50 0x7f38880ca6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:10.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.607+0000 7f388dc57700 1 -- 172.21.15.16:0/3397202397 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f38880ce380 msgr2=0x7f388809b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:10.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.607+0000 7f388dc57700 1 --2- 172.21.15.16:0/3397202397 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f38880ce380 0x7f388809b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:10.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.607+0000 7f388dc57700 1 -- 172.21.15.16:0/3397202397 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f38880b2b50 msgr2=0x7f38880ca6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:10.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.607+0000 7f388dc57700 1 --2- 172.21.15.16:0/3397202397 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f38880b2b50 0x7f38880ca6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:10.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.607+0000 7f388dc57700 1 -- 172.21.15.16:0/3397202397 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3880009a70 con 0x7f38880cdfa0 2023-10-16T13:46:10.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.607+0000 7f388dc57700 1 --2- 172.21.15.16:0/3397202397 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38880cdfa0 0x7f388809d3c0 secure :-1 s=READY pgs=2391 cs=0 l=1 rev1=1 crypto rx=0x7f387800c700 tx=0x7f387800ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:10.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.607+0000 7f387effd700 1 -- 172.21.15.16:0/3397202397 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3878005d70 con 0x7f38880cdfa0 2023-10-16T13:46:10.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.607+0000 7f387effd700 1 -- 172.21.15.16:0/3397202397 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38780070d0 con 0x7f38880cdfa0 2023-10-16T13:46:10.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.607+0000 7f38906bc700 1 -- 172.21.15.16:0/3397202397 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f388809e1d0 con 0x7f38880cdfa0 2023-10-16T13:46:10.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.607+0000 7f387effd700 1 -- 172.21.15.16:0/3397202397 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3878007460 con 0x7f38880cdfa0 2023-10-16T13:46:10.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.607+0000 7f38906bc700 1 -- 172.21.15.16:0/3397202397 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38880cac00 con 0x7f38880cdfa0 2023-10-16T13:46:10.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.608+0000 7f38906bc700 1 -- 172.21.15.16:0/3397202397 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f388809e9d0 con 0x7f38880cdfa0 2023-10-16T13:46:10.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.608+0000 7f387effd700 1 -- 172.21.15.16:0/3397202397 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 135) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f38780075c0 con 0x7f38880cdfa0 2023-10-16T13:46:10.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.609+0000 7f387effd700 1 -- 172.21.15.16:0/3397202397 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 7046+0+0 (secure 0 0 0) 0x7f38780889c0 con 0x7f38880cdfa0 2023-10-16T13:46:10.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.614+0000 7f387effd700 1 -- 172.21.15.16:0/3397202397 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f388809e9d0 con 0x7f38880cdfa0 2023-10-16T13:46:10.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.817+0000 7f38906bc700 1 -- 172.21.15.16:0/3397202397 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f388809e9d0 con 0x7f38880cdfa0 2023-10-16T13:46:10.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.818+0000 7f387effd700 1 -- 172.21.15.16:0/3397202397 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v213) v1 ==== 74+0+18212 (secure 0 0 0) 0x7f388809e9d0 con 0x7f38880cdfa0 2023-10-16T13:46:10.821 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:46:10.822 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":213,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:46:10.160075+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2023-10-16T13:44:38.716950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"205","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_data","create_time":"2023-10-16T13:44:39.172499+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"212","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":212,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":204,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":204,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":204,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":204,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":204,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":204,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"26.5","mappings":[{"from":1,"to":0}]},{"pgid":"26.21","mappings":[{"from":1,"to":0}]},{"pgid":"26.24","mappings":[{"from":3,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/1458874080":"2023-10-17T13:46:10.160023+0000","172.21.15.16:0/1497663382":"2023-10-17T13:46:10.160023+0000","172.21.15.16:6832/3220139196":"2023-10-17T13:46:10.160023+0000","172.21.15.16:0/1142997361":"2023-10-17T13:46:10.160023+0000","172.21.15.16:0/4155301956":"2023-10-17T13:46:10.160023+0000","172.21.15.16:0/2102874149":"2023-10-17T13:46:10.160023+0000","172.21.15.16:0/1863977242":"2023-10-17T13:46:10.160023+0000","172.21.15.16:0/196217497":"2023-10-17T13:46:10.160023+0000","172.21.15.16:0/3477190137":"2023-10-17T13:44:55.326924+0000","172.21.15.16:0/2374795404":"2023-10-17T13:46:10.160023+0000","172.21.15.16:0/1906189720":"2023-10-17T13:44:55.326924+0000","172.21.15.16:6833/1268439414":"2023-10-17T13:44:55.326924+0000","172.21.15.16:6832/1268439414":"2023-10-17T13:44:55.326924+0000","172.21.15.16:6833/3220139196":"2023-10-17T13:46:10.160023+0000","172.21.15.16:0/3697646183":"2023-10-17T13:44:55.326924+0000","172.21.15.16:0/4067982926":"2023-10-17T13:44:55.326924+0000","172.21.15.16:0/3408848304":"2023-10-17T13:44:55.326924+0000","172.21.15.16:0/2782518306":"2023-10-17T13:44:55.326924+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:46:10.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.821+0000 7f38906bc700 1 -- 172.21.15.16:0/3397202397 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38880cdfa0 msgr2=0x7f388809d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:10.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.821+0000 7f38906bc700 1 --2- 172.21.15.16:0/3397202397 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38880cdfa0 0x7f388809d3c0 secure :-1 s=READY pgs=2391 cs=0 l=1 rev1=1 crypto rx=0x7f387800c700 tx=0x7f387800ca10 comp rx=0 tx=0).stop 2023-10-16T13:46:10.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.821+0000 7f38906bc700 1 -- 172.21.15.16:0/3397202397 shutdown_connections 2023-10-16T13:46:10.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.821+0000 7f38906bc700 1 --2- 172.21.15.16:0/3397202397 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f38880ce380 0x7f388809b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:10.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.821+0000 7f38906bc700 1 --2- 172.21.15.16:0/3397202397 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f38880b2b50 0x7f38880ca6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:10.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.821+0000 7f38906bc700 1 --2- 172.21.15.16:0/3397202397 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38880cdfa0 0x7f388809d3c0 unknown :-1 s=CLOSED pgs=2391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:10.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.821+0000 7f38906bc700 1 -- 172.21.15.16:0/3397202397 >> 172.21.15.16:0/3397202397 conn(0x7f38880d14b0 msgr2=0x7f3888160930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:10.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.821+0000 7f38906bc700 1 -- 172.21.15.16:0/3397202397 shutdown_connections 2023-10-16T13:46:10.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.821+0000 7f38906bc700 1 -- 172.21.15.16:0/3397202397 wait complete. 2023-10-16T13:46:10.835 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-16T13:46:10.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.959+0000 7f80be2ba700 1 Processor -- start 2023-10-16T13:46:10.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.959+0000 7f80be2ba700 1 -- start start 2023-10-16T13:46:10.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.960+0000 7f80be2ba700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f80b815b280 0x7f80b81613e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:10.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.960+0000 7f80be2ba700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f80b80cc8a0 0x7f80b815e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:10.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.960+0000 7f80be2ba700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80b8161c30 con 0x7f80b806a7e0 2023-10-16T13:46:10.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.960+0000 7f80be2ba700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80b8161f10 con 0x7f80b815b280 2023-10-16T13:46:10.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.960+0000 7f80be2ba700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80b81621f0 con 0x7f80b80cc8a0 2023-10-16T13:46:10.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.960+0000 7f80b7fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f80b815b280 0x7f80b81613e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:10.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.960+0000 7f80b7fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f80b815b280 0x7f80b81613e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52724/0 (socket says 172.21.15.16:52724) 2023-10-16T13:46:10.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.960+0000 7f80b7fff700 1 -- 172.21.15.16:0/2472279738 learned_addr learned my addr 172.21.15.16:0/2472279738 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:10.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.960+0000 7f80b77fe700 1 --2- 172.21.15.16:0/2472279738 >> v2:172.21.15.49:3300/0 conn(0x7f80b80cc8a0 0x7f80b815e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:10.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.961+0000 7f80b7fff700 1 -- 172.21.15.16:0/2472279738 >> v2:172.21.15.49:3300/0 conn(0x7f80b80cc8a0 msgr2=0x7f80b815e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:10.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.961+0000 7f80b7fff700 1 --2- 172.21.15.16:0/2472279738 >> v2:172.21.15.49:3300/0 conn(0x7f80b80cc8a0 0x7f80b815e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:10.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.961+0000 7f80b7fff700 1 -- 172.21.15.16:0/2472279738 >> v1:172.21.15.49:6789/0 conn(0x7f80b806a7e0 legacy=0x7f80b80d02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:10.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.961+0000 7f80b7fff700 1 -- 172.21.15.16:0/2472279738 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80b815df00 con 0x7f80b815b280 2023-10-16T13:46:10.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.961+0000 7f80b7fff700 1 --2- 172.21.15.16:0/2472279738 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80b815b280 0x7f80b81613e0 secure :-1 s=READY pgs=2392 cs=0 l=1 rev1=1 crypto rx=0x7f80b8071970 tx=0x7f80a8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14b37653516d4586 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:10.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.961+0000 7f80b6ffd700 1 -- 172.21.15.16:0/2472279738 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80a8004230 con 0x7f80b815b280 2023-10-16T13:46:10.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.961+0000 7f80b6ffd700 1 -- 172.21.15.16:0/2472279738 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80a8004390 con 0x7f80b815b280 2023-10-16T13:46:10.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.961+0000 7f80b6ffd700 1 -- 172.21.15.16:0/2472279738 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80a80036a0 con 0x7f80b815b280 2023-10-16T13:46:10.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.961+0000 7f80be2ba700 1 -- 172.21.15.16:0/2472279738 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80b815b280 msgr2=0x7f80b81613e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:10.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.961+0000 7f80be2ba700 1 --2- 172.21.15.16:0/2472279738 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80b815b280 0x7f80b81613e0 secure :-1 s=READY pgs=2392 cs=0 l=1 rev1=1 crypto rx=0x7f80b8071970 tx=0x7f80a8009bd0 comp rx=0 tx=0).stop 2023-10-16T13:46:10.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.961+0000 7f80be2ba700 1 -- 172.21.15.16:0/2472279738 shutdown_connections 2023-10-16T13:46:10.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.961+0000 7f80be2ba700 1 --2- 172.21.15.16:0/2472279738 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80b815b280 0x7f80b81613e0 unknown :-1 s=CLOSED pgs=2392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:10.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.961+0000 7f80be2ba700 1 --2- 172.21.15.16:0/2472279738 >> v2:172.21.15.49:3300/0 conn(0x7f80b80cc8a0 0x7f80b815e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:10.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.961+0000 7f80be2ba700 1 -- 172.21.15.16:0/2472279738 >> 172.21.15.16:0/2472279738 conn(0x7f80b80b1fd0 msgr2=0x7f80b80b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:10.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.961+0000 7f80be2ba700 1 -- 172.21.15.16:0/2472279738 shutdown_connections 2023-10-16T13:46:10.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.962+0000 7f80be2ba700 1 -- 172.21.15.16:0/2472279738 wait complete. 2023-10-16T13:46:10.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.963+0000 7f80be2ba700 1 Processor -- start 2023-10-16T13:46:10.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.963+0000 7f80be2ba700 1 -- start start 2023-10-16T13:46:10.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.964+0000 7f80be2ba700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f80b806a7e0 0x7f80b8158940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:10.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.964+0000 7f80be2ba700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f80b80cc8a0 0x7f80b806abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:10.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.964+0000 7f80be2ba700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80b815b280 0x7f80b806b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:10.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.964+0000 7f80be2ba700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80b8158e80 con 0x7f80b815b280 2023-10-16T13:46:10.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.964+0000 7f80be2ba700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80b80c3f70 con 0x7f80b806a7e0 2023-10-16T13:46:10.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.964+0000 7f80be2ba700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80b8157a60 con 0x7f80b80cc8a0 2023-10-16T13:46:10.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.964+0000 7f80bc857700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80b815b280 0x7f80b806b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:10.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.964+0000 7f80bc857700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80b815b280 0x7f80b806b100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52734/0 (socket says 172.21.15.16:52734) 2023-10-16T13:46:10.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.964+0000 7f80bc857700 1 -- 172.21.15.16:0/1429365537 learned_addr learned my addr 172.21.15.16:0/1429365537 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:10.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.964+0000 7f80b7fff700 1 --2- 172.21.15.16:0/1429365537 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f80b806a7e0 0x7f80b8158940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:10.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.964+0000 7f80b77fe700 1 --2- 172.21.15.16:0/1429365537 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f80b80cc8a0 0x7f80b806abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:10.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.964+0000 7f80bc857700 1 -- 172.21.15.16:0/1429365537 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f80b80cc8a0 msgr2=0x7f80b806abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:10.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.964+0000 7f80bc857700 1 --2- 172.21.15.16:0/1429365537 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f80b80cc8a0 0x7f80b806abc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:10.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.964+0000 7f80bc857700 1 -- 172.21.15.16:0/1429365537 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f80b806a7e0 msgr2=0x7f80b8158940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:10.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.964+0000 7f80bc857700 1 --2- 172.21.15.16:0/1429365537 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f80b806a7e0 0x7f80b8158940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:10.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.964+0000 7f80bc857700 1 -- 172.21.15.16:0/1429365537 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80a8009550 con 0x7f80b815b280 2023-10-16T13:46:10.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.964+0000 7f80b77fe700 1 --2- 172.21.15.16:0/1429365537 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f80b80cc8a0 0x7f80b806abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:46:10.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.964+0000 7f80b7fff700 1 --2- 172.21.15.16:0/1429365537 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f80b806a7e0 0x7f80b8158940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:46:10.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.964+0000 7f80bc857700 1 --2- 172.21.15.16:0/1429365537 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80b815b280 0x7f80b806b100 secure :-1 s=READY pgs=2393 cs=0 l=1 rev1=1 crypto rx=0x7f80b000b970 tx=0x7f80b000bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:10.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.964+0000 7f80b4ff9700 1 -- 172.21.15.16:0/1429365537 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80b000c9b0 con 0x7f80b815b280 2023-10-16T13:46:10.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.964+0000 7f80b4ff9700 1 -- 172.21.15.16:0/1429365537 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80b000cb10 con 0x7f80b815b280 2023-10-16T13:46:10.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.964+0000 7f80be2ba700 1 -- 172.21.15.16:0/1429365537 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80b80c41c0 con 0x7f80b815b280 2023-10-16T13:46:10.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.964+0000 7f80b4ff9700 1 -- 172.21.15.16:0/1429365537 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80b0014440 con 0x7f80b815b280 2023-10-16T13:46:10.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.964+0000 7f80be2ba700 1 -- 172.21.15.16:0/1429365537 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80b80c4780 con 0x7f80b815b280 2023-10-16T13:46:10.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.965+0000 7f80be2ba700 1 -- 172.21.15.16:0/1429365537 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80b804ebb0 con 0x7f80b815b280 2023-10-16T13:46:10.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.966+0000 7f80b4ff9700 1 -- 172.21.15.16:0/1429365537 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 135) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f80b001d3b0 con 0x7f80b815b280 2023-10-16T13:46:10.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.966+0000 7f80b4ff9700 1 -- 172.21.15.16:0/1429365537 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 7046+0+0 (secure 0 0 0) 0x7f80b00891d0 con 0x7f80b815b280 2023-10-16T13:46:10.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:10.971+0000 7f80b4ff9700 1 -- 172.21.15.16:0/1429365537 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f80b0054190 con 0x7f80b815b280 2023-10-16T13:46:11.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.170+0000 7f80b4ff9700 1 -- 172.21.15.16:0/1429365537 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 136) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f80b0053c40 con 0x7f80b815b280 2023-10-16T13:46:11.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.170+0000 7f80b4ff9700 1 --2- 172.21.15.16:0/1429365537 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f8098071080 0x7f8098073470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:11.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.171+0000 7f80b7fff700 1 --2- 172.21.15.16:0/1429365537 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f8098071080 0x7f8098073470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:11.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.174+0000 7f80b7fff700 1 --2- 172.21.15.16:0/1429365537 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f8098071080 0x7f8098073470 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f80a8009130 tx=0x7f80a8008030 comp rx=0 tx=0).ready entity=mgr.11190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:11.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.226+0000 7f80be2ba700 1 -- 172.21.15.16:0/1429365537 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f80b8072240 con 0x7f80b815b280 2023-10-16T13:46:11.228 INFO:tasks.ceph.mon.a.smithi016.stderr:2023-10-16T13:46:11.226+0000 7f9fc8535700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-16T13:46:11.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.227+0000 7f80b4ff9700 1 -- 172.21.15.16:0/1429365537 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v105) v1 ==== 161+0+0 (secure 0 0 0) 0x7f80b0072a10 con 0x7f80b815b280 2023-10-16T13:46:11.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.231+0000 7f80be2ba700 1 -- 172.21.15.16:0/1429365537 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f8098071080 msgr2=0x7f8098073470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:11.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.231+0000 7f80be2ba700 1 --2- 172.21.15.16:0/1429365537 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f8098071080 0x7f8098073470 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f80a8009130 tx=0x7f80a8008030 comp rx=0 tx=0).stop 2023-10-16T13:46:11.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.231+0000 7f80be2ba700 1 -- 172.21.15.16:0/1429365537 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80b815b280 msgr2=0x7f80b806b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:11.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.231+0000 7f80be2ba700 1 --2- 172.21.15.16:0/1429365537 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80b815b280 0x7f80b806b100 secure :-1 s=READY pgs=2393 cs=0 l=1 rev1=1 crypto rx=0x7f80b000b970 tx=0x7f80b000bd30 comp rx=0 tx=0).stop 2023-10-16T13:46:11.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.231+0000 7f80be2ba700 1 -- 172.21.15.16:0/1429365537 shutdown_connections 2023-10-16T13:46:11.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.231+0000 7f80be2ba700 1 --2- 172.21.15.16:0/1429365537 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f8098071080 0x7f8098073470 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:11.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.231+0000 7f80be2ba700 1 --2- 172.21.15.16:0/1429365537 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80b815b280 0x7f80b806b100 unknown :-1 s=CLOSED pgs=2393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:11.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.231+0000 7f80be2ba700 1 --2- 172.21.15.16:0/1429365537 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f80b806a7e0 0x7f80b8158940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:11.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.231+0000 7f80be2ba700 1 --2- 172.21.15.16:0/1429365537 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f80b80cc8a0 0x7f80b806abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:11.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.231+0000 7f80be2ba700 1 -- 172.21.15.16:0/1429365537 >> 172.21.15.16:0/1429365537 conn(0x7f80b80b1fd0 msgr2=0x7f80b80caa40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:11.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.231+0000 7f80be2ba700 1 -- 172.21.15.16:0/1429365537 shutdown_connections 2023-10-16T13:46:11.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.231+0000 7f80be2ba700 1 -- 172.21.15.16:0/1429365537 wait complete. 2023-10-16T13:46:11.236 INFO:teuthology.orchestra.run.smithi016.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-16T13:46:11.246 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-16T13:46:11.270 INFO:tasks.ceph.mds.a.smithi016.stdout:starting mds.a at 2023-10-16T13:46:11.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.372+0000 7f75c71c6700 1 Processor -- start 2023-10-16T13:46:11.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.372+0000 7f75c71c6700 1 -- start start 2023-10-16T13:46:11.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.373+0000 7f75c71c6700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f75c015d8d0 0x7f75c00b2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:11.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.373+0000 7f75c71c6700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f75c00b8fa0 0x7f75c00b6720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:11.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.373+0000 7f75c71c6700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75c00b2f30 con 0x7f75c00b6c60 2023-10-16T13:46:11.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.373+0000 7f75c71c6700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75c00d0bb0 con 0x7f75c00b8fa0 2023-10-16T13:46:11.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.373+0000 7f75c71c6700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75c015dcb0 con 0x7f75c015d8d0 2023-10-16T13:46:11.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.373+0000 7f75c4f62700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f75c015d8d0 0x7f75c00b2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:11.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.373+0000 7f75bffff700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f75c00b8fa0 0x7f75c00b6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:11.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.373+0000 7f75c5763700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7f75c00b6c60 0x7f75c00ce350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:49232/0 (socket says 172.21.15.16:49232) 2023-10-16T13:46:11.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.373+0000 7f75c4f62700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f75c015d8d0 0x7f75c00b2830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:41030/0 (socket says 172.21.15.16:41030) 2023-10-16T13:46:11.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.373+0000 7f75c5763700 1 -- 172.21.15.16:0/760770355 learned_addr learned my addr 172.21.15.16:0/760770355 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:46:11.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.373+0000 7f75bffff700 1 -- 172.21.15.16:0/760770355 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f75c015d8d0 msgr2=0x7f75c00b2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:11.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.373+0000 7f75bffff700 1 --2- 172.21.15.16:0/760770355 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f75c015d8d0 0x7f75c00b2830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:11.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.373+0000 7f75bffff700 1 -- 172.21.15.16:0/760770355 >> v1:172.21.15.49:6789/0 conn(0x7f75c00b6c60 legacy=0x7f75c00ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:11.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.374+0000 7f75bffff700 1 -- 172.21.15.16:0/760770355 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75c00d1c40 con 0x7f75c00b8fa0 2023-10-16T13:46:11.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.374+0000 7f75c4f62700 1 --2- 172.21.15.16:0/760770355 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f75c015d8d0 0x7f75c00b2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:46:11.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.374+0000 7f75bffff700 1 --2- 172.21.15.16:0/760770355 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f75c00b8fa0 0x7f75c00b6720 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f75ac009840 tx=0x7f75ac009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e4aa7ba856927340 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:11.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.374+0000 7f75bf7fe700 1 -- 172.21.15.16:0/760770355 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75ac011830 con 0x7f75c00b8fa0 2023-10-16T13:46:11.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.374+0000 7f75bf7fe700 1 -- 172.21.15.16:0/760770355 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75ac011990 con 0x7f75c00b8fa0 2023-10-16T13:46:11.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.374+0000 7f75bf7fe700 1 -- 172.21.15.16:0/760770355 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75ac011d20 con 0x7f75c00b8fa0 2023-10-16T13:46:11.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.374+0000 7f75c71c6700 1 -- 172.21.15.16:0/760770355 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f75c00b8fa0 msgr2=0x7f75c00b6720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:11.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.374+0000 7f75c71c6700 1 --2- 172.21.15.16:0/760770355 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f75c00b8fa0 0x7f75c00b6720 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f75ac009840 tx=0x7f75ac009b50 comp rx=0 tx=0).stop 2023-10-16T13:46:11.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.374+0000 7f75c71c6700 1 -- 172.21.15.16:0/760770355 shutdown_connections 2023-10-16T13:46:11.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.374+0000 7f75c71c6700 1 --2- 172.21.15.16:0/760770355 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f75c015d8d0 0x7f75c00b2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:11.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.374+0000 7f75c71c6700 1 --2- 172.21.15.16:0/760770355 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f75c00b8fa0 0x7f75c00b6720 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:11.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.374+0000 7f75c71c6700 1 -- 172.21.15.16:0/760770355 >> 172.21.15.16:0/760770355 conn(0x7f75c0072330 msgr2=0x7f75c0072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:11.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.375+0000 7f75c71c6700 1 -- 172.21.15.16:0/760770355 shutdown_connections 2023-10-16T13:46:11.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.375+0000 7f75c71c6700 1 -- 172.21.15.16:0/760770355 wait complete. 2023-10-16T13:46:11.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.376+0000 7f75c71c6700 1 Processor -- start 2023-10-16T13:46:11.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.376+0000 7f75c71c6700 1 -- start start 2023-10-16T13:46:11.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.377+0000 7f75c71c6700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f75c00b6c60 0x7f75c015de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:11.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.377+0000 7f75c71c6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75c00b8fa0 0x7f75c015e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:11.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.377+0000 7f75c71c6700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f75c015d8d0 0x7f75c00cb9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:11.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.377+0000 7f75c71c6700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c00ccfe0 con 0x7f75c00b8fa0 2023-10-16T13:46:11.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.377+0000 7f75c71c6700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c01f3340 con 0x7f75c015d8d0 2023-10-16T13:46:11.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.377+0000 7f75c71c6700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75c015a500 con 0x7f75c00b6c60 2023-10-16T13:46:11.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.377+0000 7f75bffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75c00b8fa0 0x7f75c015e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:11.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.377+0000 7f75bffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75c00b8fa0 0x7f75c015e360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52780/0 (socket says 172.21.15.16:52780) 2023-10-16T13:46:11.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.377+0000 7f75bffff700 1 -- 172.21.15.16:0/1348040558 learned_addr learned my addr 172.21.15.16:0/1348040558 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:11.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.378+0000 7f75c5763700 1 --2- 172.21.15.16:0/1348040558 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f75c015d8d0 0x7f75c00cb9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:11.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.378+0000 7f75c4f62700 1 --2- 172.21.15.16:0/1348040558 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f75c00b6c60 0x7f75c015de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:11.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.378+0000 7f75bffff700 1 -- 172.21.15.16:0/1348040558 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f75c00b6c60 msgr2=0x7f75c015de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:11.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.378+0000 7f75bffff700 1 --2- 172.21.15.16:0/1348040558 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f75c00b6c60 0x7f75c015de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:11.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.378+0000 7f75bffff700 1 -- 172.21.15.16:0/1348040558 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f75c015d8d0 msgr2=0x7f75c00cb9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:11.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.378+0000 7f75bffff700 1 --2- 172.21.15.16:0/1348040558 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f75c015d8d0 0x7f75c00cb9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:11.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.378+0000 7f75bffff700 1 -- 172.21.15.16:0/1348040558 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75b40094a0 con 0x7f75c00b8fa0 2023-10-16T13:46:11.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.378+0000 7f75c4f62700 1 --2- 172.21.15.16:0/1348040558 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f75c00b6c60 0x7f75c015de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:46:11.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.378+0000 7f75bffff700 1 --2- 172.21.15.16:0/1348040558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75c00b8fa0 0x7f75c015e360 secure :-1 s=READY pgs=2396 cs=0 l=1 rev1=1 crypto rx=0x7f75ac009fd0 tx=0x7f75ac004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:11.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.378+0000 7f75bd7fa700 1 -- 172.21.15.16:0/1348040558 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75ac0036a0 con 0x7f75c00b8fa0 2023-10-16T13:46:11.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.378+0000 7f75bd7fa700 1 -- 172.21.15.16:0/1348040558 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75ac01e070 con 0x7f75c00b8fa0 2023-10-16T13:46:11.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.378+0000 7f75c71c6700 1 -- 172.21.15.16:0/1348040558 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75ac0094a0 con 0x7f75c00b8fa0 2023-10-16T13:46:11.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.378+0000 7f75bd7fa700 1 -- 172.21.15.16:0/1348040558 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75ac003a60 con 0x7f75c00b8fa0 2023-10-16T13:46:11.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.378+0000 7f75c71c6700 1 -- 172.21.15.16:0/1348040558 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75c01f2ff0 con 0x7f75c00b8fa0 2023-10-16T13:46:11.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.379+0000 7f75c71c6700 1 -- 172.21.15.16:0/1348040558 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75c01f3480 con 0x7f75c00b8fa0 2023-10-16T13:46:11.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.380+0000 7f75bd7fa700 1 -- 172.21.15.16:0/1348040558 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 136) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f75ac003bc0 con 0x7f75c00b8fa0 2023-10-16T13:46:11.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.380+0000 7f75bd7fa700 1 --2- 172.21.15.16:0/1348040558 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f75a406ad50 0x7f75a406d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:11.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.380+0000 7f75c4f62700 1 --2- 172.21.15.16:0/1348040558 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f75a406ad50 0x7f75a406d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:11.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.380+0000 7f75bd7fa700 1 -- 172.21.15.16:0/1348040558 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 7132+0+0 (secure 0 0 0) 0x7f75ac08a6a0 con 0x7f75c00b8fa0 2023-10-16T13:46:11.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.381+0000 7f75c4f62700 1 --2- 172.21.15.16:0/1348040558 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f75a406ad50 0x7f75a406d210 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f75b4009dc0 tx=0x7f75b4008040 comp rx=0 tx=0).ready entity=mgr.11190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:11.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.385+0000 7f75bd7fa700 1 -- 172.21.15.16:0/1348040558 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f75c01f3480 con 0x7f75c00b8fa0 2023-10-16T13:46:11.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:11.639+0000 7f75c71c6700 1 -- 172.21.15.16:0/1348040558 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f75c01f3480 con 0x7f75c00b8fa0 2023-10-16T13:46:12.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.229+0000 7f75bd7fa700 1 -- 172.21.15.16:0/1348040558 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v106) v1 ==== 104+0+0 (secure 0 0 0) 0x7f75c01f3480 con 0x7f75c00b8fa0 2023-10-16T13:46:12.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.232+0000 7f75c71c6700 1 -- 172.21.15.16:0/1348040558 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f75a406ad50 msgr2=0x7f75a406d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:12.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.232+0000 7f75c71c6700 1 --2- 172.21.15.16:0/1348040558 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f75a406ad50 0x7f75a406d210 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f75b4009dc0 tx=0x7f75b4008040 comp rx=0 tx=0).stop 2023-10-16T13:46:12.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.232+0000 7f75c71c6700 1 -- 172.21.15.16:0/1348040558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75c00b8fa0 msgr2=0x7f75c015e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:12.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.232+0000 7f75c71c6700 1 --2- 172.21.15.16:0/1348040558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75c00b8fa0 0x7f75c015e360 secure :-1 s=READY pgs=2396 cs=0 l=1 rev1=1 crypto rx=0x7f75ac009fd0 tx=0x7f75ac004750 comp rx=0 tx=0).stop 2023-10-16T13:46:12.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.233+0000 7f75c71c6700 1 -- 172.21.15.16:0/1348040558 shutdown_connections 2023-10-16T13:46:12.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.233+0000 7f75c71c6700 1 --2- 172.21.15.16:0/1348040558 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f75c015d8d0 0x7f75c00cb9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:12.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.233+0000 7f75c71c6700 1 --2- 172.21.15.16:0/1348040558 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f75c00b6c60 0x7f75c015de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:12.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.233+0000 7f75c71c6700 1 --2- 172.21.15.16:0/1348040558 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f75a406ad50 0x7f75a406d210 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:12.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.233+0000 7f75c71c6700 1 --2- 172.21.15.16:0/1348040558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75c00b8fa0 0x7f75c015e360 unknown :-1 s=CLOSED pgs=2396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:12.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.233+0000 7f75c71c6700 1 -- 172.21.15.16:0/1348040558 >> 172.21.15.16:0/1348040558 conn(0x7f75c0072330 msgr2=0x7f75c00b22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:12.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.233+0000 7f75c71c6700 1 -- 172.21.15.16:0/1348040558 shutdown_connections 2023-10-16T13:46:12.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.233+0000 7f75c71c6700 1 -- 172.21.15.16:0/1348040558 wait complete. 2023-10-16T13:46:12.253 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-16T13:46:12.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.430+0000 7fd6acb59700 1 Processor -- start 2023-10-16T13:46:12.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.430+0000 7fd6acb59700 1 -- start start 2023-10-16T13:46:12.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.431+0000 7fd6acb59700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd6a8160e60 0x7fd6a815e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:12.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.431+0000 7fd6acb59700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fd6a806d980 0x7fd6a815efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:12.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.431+0000 7fd6acb59700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6a80cdeb0 con 0x7fd6a806a7e0 2023-10-16T13:46:12.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.431+0000 7fd6acb59700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6a81613e0 con 0x7fd6a806d980 2023-10-16T13:46:12.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.431+0000 7fd6acb59700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a81582d0 con 0x7fd6a8160e60 2023-10-16T13:46:12.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.431+0000 7fd6a6d9d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd6a806a7e0 0x7fd6a806abc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42082/0 (socket says 172.21.15.16:42082) 2023-10-16T13:46:12.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.431+0000 7fd6a6d9d700 1 -- 172.21.15.16:0/3937550300 learned_addr learned my addr 172.21.15.16:0/3937550300 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:46:12.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.431+0000 7fd6a659c700 1 --2- 172.21.15.16:0/3937550300 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd6a8160e60 0x7fd6a815e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:12.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.431+0000 7fd6a5d9b700 1 --2- 172.21.15.16:0/3937550300 >> v2:172.21.15.49:3300/0 conn(0x7fd6a806d980 0x7fd6a815efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:12.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.431+0000 7fd6a559a700 1 -- 172.21.15.16:0/3937550300 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2892670677 0 0) 0x7fd6a80cdeb0 con 0x7fd6a806a7e0 2023-10-16T13:46:12.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.431+0000 7fd6a559a700 1 -- 172.21.15.16:0/3937550300 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6900035b0 con 0x7fd6a806a7e0 2023-10-16T13:46:12.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.432+0000 7fd6a559a700 1 -- 172.21.15.16:0/3937550300 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3630946619 0 0) 0x7fd6900035b0 con 0x7fd6a806a7e0 2023-10-16T13:46:12.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.432+0000 7fd6a5d9b700 1 -- 172.21.15.16:0/3937550300 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd6a8160e60 msgr2=0x7fd6a815e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:12.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.432+0000 7fd6a5d9b700 1 --2- 172.21.15.16:0/3937550300 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd6a8160e60 0x7fd6a815e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:12.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.432+0000 7fd6a5d9b700 1 -- 172.21.15.16:0/3937550300 >> v1:172.21.15.16:6789/0 conn(0x7fd6a806a7e0 legacy=0x7fd6a806abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:12.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.432+0000 7fd6a5d9b700 1 -- 172.21.15.16:0/3937550300 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6a815edb0 con 0x7fd6a806d980 2023-10-16T13:46:12.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.432+0000 7fd6a659c700 1 --2- 172.21.15.16:0/3937550300 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd6a8160e60 0x7fd6a815e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:46:12.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.432+0000 7fd6a5d9b700 1 --2- 172.21.15.16:0/3937550300 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd6a806d980 0x7fd6a815efb0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fd698009840 tx=0x7fd698009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bcb919284563cdfb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:12.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.432+0000 7fd6a559a700 1 -- 172.21.15.16:0/3937550300 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a00019f0 con 0x7fd6a806d980 2023-10-16T13:46:12.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.432+0000 7fd6a559a700 1 -- 172.21.15.16:0/3937550300 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd698004030 con 0x7fd6a806d980 2023-10-16T13:46:12.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.432+0000 7fd6a559a700 1 -- 172.21.15.16:0/3937550300 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6980043a0 con 0x7fd6a806d980 2023-10-16T13:46:12.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.433+0000 7fd6acb59700 1 -- 172.21.15.16:0/3937550300 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd6a806d980 msgr2=0x7fd6a815efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:12.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.433+0000 7fd6acb59700 1 --2- 172.21.15.16:0/3937550300 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd6a806d980 0x7fd6a815efb0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fd698009840 tx=0x7fd698009b50 comp rx=0 tx=0).stop 2023-10-16T13:46:12.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.433+0000 7fd6acb59700 1 -- 172.21.15.16:0/3937550300 shutdown_connections 2023-10-16T13:46:12.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.433+0000 7fd6acb59700 1 --2- 172.21.15.16:0/3937550300 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd6a8160e60 0x7fd6a815e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:12.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.433+0000 7fd6acb59700 1 --2- 172.21.15.16:0/3937550300 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd6a806d980 0x7fd6a815efb0 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:12.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.433+0000 7fd6acb59700 1 -- 172.21.15.16:0/3937550300 >> 172.21.15.16:0/3937550300 conn(0x7fd6a806f430 msgr2=0x7fd6a806f810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:12.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.433+0000 7fd6acb59700 1 -- 172.21.15.16:0/3937550300 shutdown_connections 2023-10-16T13:46:12.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.433+0000 7fd6acb59700 1 -- 172.21.15.16:0/3937550300 wait complete. 2023-10-16T13:46:12.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.434+0000 7fd6acb59700 1 Processor -- start 2023-10-16T13:46:12.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.434+0000 7fd6acb59700 1 -- start start 2023-10-16T13:46:12.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.435+0000 7fd6acb59700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd6a806a7e0 0x7fd6a80acae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:12.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.435+0000 7fd6acb59700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd6a806d980 0x7fd6a80af070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:12.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.435+0000 7fd6acb59700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6a8160e60 0x7fd6a81511c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:12.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.435+0000 7fd6acb59700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a8155c80 con 0x7fd6a8160e60 2023-10-16T13:46:12.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.435+0000 7fd6acb59700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a8155f60 con 0x7fd6a806d980 2023-10-16T13:46:12.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.435+0000 7fd6acb59700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a8156240 con 0x7fd6a806a7e0 2023-10-16T13:46:12.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.435+0000 7fd6a6d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6a8160e60 0x7fd6a81511c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:12.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.435+0000 7fd6a6d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6a8160e60 0x7fd6a81511c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52812/0 (socket says 172.21.15.16:52812) 2023-10-16T13:46:12.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.435+0000 7fd6a6d9d700 1 -- 172.21.15.16:0/2149555096 learned_addr learned my addr 172.21.15.16:0/2149555096 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:12.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.435+0000 7fd6a5d9b700 1 --2- 172.21.15.16:0/2149555096 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd6a806d980 0x7fd6a80af070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:12.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.435+0000 7fd6a659c700 1 --2- 172.21.15.16:0/2149555096 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd6a806a7e0 0x7fd6a80acae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:12.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.435+0000 7fd6a6d9d700 1 -- 172.21.15.16:0/2149555096 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd6a806a7e0 msgr2=0x7fd6a80acae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:12.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.435+0000 7fd6a6d9d700 1 --2- 172.21.15.16:0/2149555096 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd6a806a7e0 0x7fd6a80acae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:12.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.435+0000 7fd6a6d9d700 1 -- 172.21.15.16:0/2149555096 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd6a806d980 msgr2=0x7fd6a80af070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:12.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.435+0000 7fd6a6d9d700 1 --2- 172.21.15.16:0/2149555096 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd6a806d980 0x7fd6a80af070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:12.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.435+0000 7fd6a6d9d700 1 -- 172.21.15.16:0/2149555096 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd69c0094a0 con 0x7fd6a8160e60 2023-10-16T13:46:12.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.435+0000 7fd6a5d9b700 1 --2- 172.21.15.16:0/2149555096 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd6a806d980 0x7fd6a80af070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:46:12.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.435+0000 7fd6a6d9d700 1 --2- 172.21.15.16:0/2149555096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6a8160e60 0x7fd6a81511c0 secure :-1 s=READY pgs=2400 cs=0 l=1 rev1=1 crypto rx=0x7fd6a000cbe0 tx=0x7fd6a000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:12.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.436+0000 7fd696ffd700 1 -- 172.21.15.16:0/2149555096 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a000ab80 con 0x7fd6a8160e60 2023-10-16T13:46:12.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.436+0000 7fd696ffd700 1 -- 172.21.15.16:0/2149555096 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6a000ad00 con 0x7fd6a8160e60 2023-10-16T13:46:12.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.436+0000 7fd6acb59700 1 -- 172.21.15.16:0/2149555096 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6980094a0 con 0x7fd6a8160e60 2023-10-16T13:46:12.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.436+0000 7fd696ffd700 1 -- 172.21.15.16:0/2149555096 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a0007310 con 0x7fd6a8160e60 2023-10-16T13:46:12.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.436+0000 7fd6acb59700 1 -- 172.21.15.16:0/2149555096 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6a814f700 con 0x7fd6a8160e60 2023-10-16T13:46:12.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.437+0000 7fd6acb59700 1 -- 172.21.15.16:0/2149555096 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6a804ec20 con 0x7fd6a8160e60 2023-10-16T13:46:12.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.437+0000 7fd696ffd700 1 -- 172.21.15.16:0/2149555096 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 136) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fd6a0007470 con 0x7fd6a8160e60 2023-10-16T13:46:12.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.437+0000 7fd696ffd700 1 --2- 172.21.15.16:0/2149555096 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fd68806ad50 0x7fd68806d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:12.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.438+0000 7fd696ffd700 1 -- 172.21.15.16:0/2149555096 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 7132+0+0 (secure 0 0 0) 0x7fd6a00893c0 con 0x7fd6a8160e60 2023-10-16T13:46:12.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.438+0000 7fd6a659c700 1 --2- 172.21.15.16:0/2149555096 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fd68806ad50 0x7fd68806d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:12.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.438+0000 7fd6a659c700 1 --2- 172.21.15.16:0/2149555096 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fd68806ad50 0x7fd68806d210 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd6a81567c0 tx=0x7fd69c008040 comp rx=0 tx=0).ready entity=mgr.11190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:12.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.442+0000 7fd696ffd700 1 -- 172.21.15.16:0/2149555096 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd6a00540d0 con 0x7fd6a8160e60 2023-10-16T13:46:12.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:12.645+0000 7fd6acb59700 1 -- 172.21.15.16:0/2149555096 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fd6a80b6260 con 0x7fd6a8160e60 2023-10-16T13:46:13.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:13.177+0000 7fd696ffd700 1 -- 172.21.15.16:0/2149555096 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 137) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fd6a00544a0 con 0x7fd6a8160e60 2023-10-16T13:46:13.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:13.513+0000 7fd696ffd700 1 -- 172.21.15.16:0/2149555096 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v216) v1 ==== 181+0+0 (secure 0 0 0) 0x7fd6a00853e0 con 0x7fd6a8160e60 2023-10-16T13:46:13.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:13.517+0000 7fd6acb59700 1 -- 172.21.15.16:0/2149555096 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fd68806ad50 msgr2=0x7fd68806d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:13.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:13.517+0000 7fd6acb59700 1 --2- 172.21.15.16:0/2149555096 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fd68806ad50 0x7fd68806d210 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd6a81567c0 tx=0x7fd69c008040 comp rx=0 tx=0).stop 2023-10-16T13:46:13.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:13.517+0000 7fd6acb59700 1 -- 172.21.15.16:0/2149555096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6a8160e60 msgr2=0x7fd6a81511c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:13.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:13.517+0000 7fd6acb59700 1 --2- 172.21.15.16:0/2149555096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6a8160e60 0x7fd6a81511c0 secure :-1 s=READY pgs=2400 cs=0 l=1 rev1=1 crypto rx=0x7fd6a000cbe0 tx=0x7fd6a000a060 comp rx=0 tx=0).stop 2023-10-16T13:46:13.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:13.517+0000 7fd6acb59700 1 -- 172.21.15.16:0/2149555096 shutdown_connections 2023-10-16T13:46:13.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:13.517+0000 7fd6acb59700 1 --2- 172.21.15.16:0/2149555096 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fd68806ad50 0x7fd68806d210 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd6a81567c0 tx=0x7fd69c008040 comp rx=0 tx=0).stop 2023-10-16T13:46:13.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:13.517+0000 7fd6acb59700 1 --2- 172.21.15.16:0/2149555096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6a8160e60 0x7fd6a81511c0 unknown :-1 s=CLOSED pgs=2400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:13.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:13.517+0000 7fd6acb59700 1 --2- 172.21.15.16:0/2149555096 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd6a806a7e0 0x7fd6a80acae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:13.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:13.517+0000 7fd6acb59700 1 --2- 172.21.15.16:0/2149555096 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd6a806d980 0x7fd6a80af070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:13.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:13.517+0000 7fd6acb59700 1 -- 172.21.15.16:0/2149555096 >> 172.21.15.16:0/2149555096 conn(0x7fd6a806f430 msgr2=0x7fd6a815e640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:13.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:13.517+0000 7fd6acb59700 1 -- 172.21.15.16:0/2149555096 shutdown_connections 2023-10-16T13:46:13.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:13.517+0000 7fd6acb59700 1 -- 172.21.15.16:0/2149555096 wait complete. 2023-10-16T13:46:13.522 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' removed 2023-10-16T13:46:13.537 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-16T13:46:13.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:13.670+0000 7f9062a1b700 1 Processor -- start 2023-10-16T13:46:13.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:13.670+0000 7f9062a1b700 1 -- start start 2023-10-16T13:46:13.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:13.671+0000 7f9062a1b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f905c06aef0 0x7f905c0cdef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:13.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:13.671+0000 7f9062a1b700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f905c0ceb90 con 0x7f905c15d920 2023-10-16T13:46:13.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:13.671+0000 7f9062a1b700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f905c0ce5d0 con 0x7f905c0726b0 2023-10-16T13:46:13.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:13.671+0000 7f9062a1b700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f905c15dd00 con 0x7f905c06aef0 2023-10-16T13:46:13.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:13.671+0000 7f905b7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f905c06aef0 0x7f905c0cdef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:13.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:13.671+0000 7f905bfff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f905c15d920 0x7f905c06a7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42092/0 (socket says 172.21.15.16:42092) 2023-10-16T13:46:13.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:13.671+0000 7f905bfff700 1 -- 172.21.15.16:0/4003465538 learned_addr learned my addr 172.21.15.16:0/4003465538 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:46:13.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:13.671+0000 7f905b7fe700 1 -- 172.21.15.16:0/4003465538 >> v1:172.21.15.49:6789/0 conn(0x7f905c0726b0 legacy=0x7f905c0cb310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:13.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:13.671+0000 7f905b7fe700 1 -- 172.21.15.16:0/4003465538 >> v1:172.21.15.16:6789/0 conn(0x7f905c15d920 legacy=0x7f905c06a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:13.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:13.671+0000 7f905b7fe700 1 -- 172.21.15.16:0/4003465538 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f905c0ce8b0 con 0x7f905c06aef0 2023-10-16T13:46:13.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:13.671+0000 7f905affd700 1 -- 172.21.15.16:0/4003465538 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1788939537 0 0) 0x7f905c0ceb90 con 0x7f905c15d920 2023-10-16T13:46:13.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:13.671+0000 7f905b7fe700 1 --2- 172.21.15.16:0/4003465538 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f905c06aef0 0x7f905c0cdef0 secure :-1 s=READY pgs=2402 cs=0 l=1 rev1=1 crypto rx=0x7f9044009840 tx=0x7f9044009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b117becc57828f75 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:13.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:13.672+0000 7f905affd700 1 -- 172.21.15.16:0/4003465538 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9044004230 con 0x7f905c06aef0 2023-10-16T13:46:13.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:13.672+0000 7f905affd700 1 -- 172.21.15.16:0/4003465538 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9044004390 con 0x7f905c06aef0 2023-10-16T13:46:13.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:13.672+0000 7f905affd700 1 -- 172.21.15.16:0/4003465538 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90440036a0 con 0x7f905c06aef0 2023-10-16T13:46:13.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:13.672+0000 7f9062a1b700 1 -- 172.21.15.16:0/4003465538 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f905c06aef0 msgr2=0x7f905c0cdef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:13.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:13.672+0000 7f9062a1b700 1 --2- 172.21.15.16:0/4003465538 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f905c06aef0 0x7f905c0cdef0 secure :-1 s=READY pgs=2402 cs=0 l=1 rev1=1 crypto rx=0x7f9044009840 tx=0x7f9044009b50 comp rx=0 tx=0).stop 2023-10-16T13:46:13.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:13.672+0000 7f9062a1b700 1 -- 172.21.15.16:0/4003465538 shutdown_connections 2023-10-16T13:46:13.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:13.672+0000 7f9062a1b700 1 --2- 172.21.15.16:0/4003465538 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f905c06aef0 0x7f905c0cdef0 unknown :-1 s=CLOSED pgs=2402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:13.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:13.672+0000 7f9062a1b700 1 -- 172.21.15.16:0/4003465538 >> 172.21.15.16:0/4003465538 conn(0x7f905c0b1ed0 msgr2=0x7f905c0b22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:13.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:13.672+0000 7f9062a1b700 1 -- 172.21.15.16:0/4003465538 shutdown_connections 2023-10-16T13:46:13.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:13.672+0000 7f9062a1b700 1 -- 172.21.15.16:0/4003465538 wait complete. 2023-10-16T13:46:13.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:13.674+0000 7f9062a1b700 1 Processor -- start 2023-10-16T13:46:13.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:13.674+0000 7f9062a1b700 1 -- start start 2023-10-16T13:46:13.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:13.675+0000 7f9062a1b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f905c06aef0 0x7f905c1483f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:13.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:13.675+0000 7f9062a1b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f905c0726b0 0x7f905c0c7e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:13.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:13.675+0000 7f9062a1b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f905c15d920 0x7f905c0c8340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:13.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:13.675+0000 7f9062a1b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f905c157060 con 0x7f905c15d920 2023-10-16T13:46:13.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:13.675+0000 7f9062a1b700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f905c148f40 con 0x7f905c06aef0 2023-10-16T13:46:13.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:13.675+0000 7f9062a1b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f905c0c8970 con 0x7f905c0726b0 2023-10-16T13:46:13.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:13.675+0000 7f9060fb8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f905c15d920 0x7f905c0c8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:13.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:13.675+0000 7f9060fb8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f905c15d920 0x7f905c0c8340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52828/0 (socket says 172.21.15.16:52828) 2023-10-16T13:46:13.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:13.675+0000 7f9060fb8700 1 -- 172.21.15.16:0/1486830634 learned_addr learned my addr 172.21.15.16:0/1486830634 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:13.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:13.675+0000 7f905bfff700 1 --2- 172.21.15.16:0/1486830634 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f905c06aef0 0x7f905c1483f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:13.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:13.675+0000 7f905b7fe700 1 --2- 172.21.15.16:0/1486830634 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f905c0726b0 0x7f905c0c7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:13.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:13.675+0000 7f9060fb8700 1 -- 172.21.15.16:0/1486830634 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f905c0726b0 msgr2=0x7f905c0c7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:13.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:13.675+0000 7f9060fb8700 1 --2- 172.21.15.16:0/1486830634 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f905c0726b0 0x7f905c0c7e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:13.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:13.675+0000 7f9060fb8700 1 -- 172.21.15.16:0/1486830634 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f905c06aef0 msgr2=0x7f905c1483f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:13.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:13.675+0000 7f9060fb8700 1 --2- 172.21.15.16:0/1486830634 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f905c06aef0 0x7f905c1483f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:13.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:13.675+0000 7f9060fb8700 1 -- 172.21.15.16:0/1486830634 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90440094a0 con 0x7f905c15d920 2023-10-16T13:46:13.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:13.675+0000 7f905b7fe700 1 --2- 172.21.15.16:0/1486830634 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f905c0726b0 0x7f905c0c7e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:46:13.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:13.676+0000 7f9060fb8700 1 --2- 172.21.15.16:0/1486830634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f905c15d920 0x7f905c0c8340 secure :-1 s=READY pgs=2403 cs=0 l=1 rev1=1 crypto rx=0x7f905400cc10 tx=0x7f905400cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:13.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:13.676+0000 7f9058ff9700 1 -- 172.21.15.16:0/1486830634 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f905400aab0 con 0x7f905c15d920 2023-10-16T13:46:13.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:13.676+0000 7f9058ff9700 1 -- 172.21.15.16:0/1486830634 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f905400ac10 con 0x7f905c15d920 2023-10-16T13:46:13.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:13.676+0000 7f9062a1b700 1 -- 172.21.15.16:0/1486830634 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f905c0c8c50 con 0x7f905c15d920 2023-10-16T13:46:13.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:13.676+0000 7f9058ff9700 1 -- 172.21.15.16:0/1486830634 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9054012420 con 0x7f905c15d920 2023-10-16T13:46:13.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:13.676+0000 7f9062a1b700 1 -- 172.21.15.16:0/1486830634 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f905c157590 con 0x7f905c15d920 2023-10-16T13:46:13.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:13.677+0000 7f9062a1b700 1 -- 172.21.15.16:0/1486830634 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f905c04ec20 con 0x7f905c15d920 2023-10-16T13:46:13.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:13.677+0000 7f9058ff9700 1 -- 172.21.15.16:0/1486830634 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 137) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f9054012580 con 0x7f905c15d920 2023-10-16T13:46:13.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:13.678+0000 7f9058ff9700 1 --2- 172.21.15.16:0/1486830634 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f903c06af90 0x7f903c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:13.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:13.678+0000 7f9058ff9700 1 -- 172.21.15.16:0/1486830634 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(216..216 src has 1..216) v4 ==== 6666+0+0 (secure 0 0 0) 0x7f9054089c00 con 0x7f905c15d920 2023-10-16T13:46:13.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:13.678+0000 7f905bfff700 1 --2- 172.21.15.16:0/1486830634 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f903c06af90 0x7f903c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:13.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:13.678+0000 7f905bfff700 1 --2- 172.21.15.16:0/1486830634 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f903c06af90 0x7f903c06d450 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f904c000f60 tx=0x7f904c007e70 comp rx=0 tx=0).ready entity=mgr.11190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:13.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:13.682+0000 7f9058ff9700 1 -- 172.21.15.16:0/1486830634 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f905408f020 con 0x7f905c15d920 2023-10-16T13:46:13.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:13.885+0000 7f9062a1b700 1 -- 172.21.15.16:0/1486830634 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f905c157780 con 0x7f905c15d920 2023-10-16T13:46:14.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.524+0000 7f9058ff9700 1 -- 172.21.15.16:0/1486830634 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v217) v1 ==== 169+0+0 (secure 0 0 0) 0x7f9054054990 con 0x7f905c15d920 2023-10-16T13:46:14.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.532+0000 7f9062a1b700 1 -- 172.21.15.16:0/1486830634 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f903c06af90 msgr2=0x7f903c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:14.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.532+0000 7f9062a1b700 1 --2- 172.21.15.16:0/1486830634 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f903c06af90 0x7f903c06d450 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f904c000f60 tx=0x7f904c007e70 comp rx=0 tx=0).stop 2023-10-16T13:46:14.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.532+0000 7f9062a1b700 1 -- 172.21.15.16:0/1486830634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f905c15d920 msgr2=0x7f905c0c8340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:14.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.532+0000 7f9062a1b700 1 --2- 172.21.15.16:0/1486830634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f905c15d920 0x7f905c0c8340 secure :-1 s=READY pgs=2403 cs=0 l=1 rev1=1 crypto rx=0x7f905400cc10 tx=0x7f905400cf20 comp rx=0 tx=0).stop 2023-10-16T13:46:14.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.535+0000 7f9062a1b700 1 -- 172.21.15.16:0/1486830634 shutdown_connections 2023-10-16T13:46:14.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.535+0000 7f9062a1b700 1 --2- 172.21.15.16:0/1486830634 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f903c06af90 0x7f903c06d450 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:14.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.535+0000 7f9062a1b700 1 --2- 172.21.15.16:0/1486830634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f905c15d920 0x7f905c0c8340 unknown :-1 s=CLOSED pgs=2403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:14.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.535+0000 7f9062a1b700 1 --2- 172.21.15.16:0/1486830634 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f905c06aef0 0x7f905c1483f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:14.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.535+0000 7f9062a1b700 1 --2- 172.21.15.16:0/1486830634 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f905c0726b0 0x7f905c0c7e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:14.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.535+0000 7f9062a1b700 1 -- 172.21.15.16:0/1486830634 >> 172.21.15.16:0/1486830634 conn(0x7f905c0b1ed0 msgr2=0x7f905c0ae8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:14.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.536+0000 7f9062a1b700 1 -- 172.21.15.16:0/1486830634 shutdown_connections 2023-10-16T13:46:14.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.536+0000 7f9062a1b700 1 -- 172.21.15.16:0/1486830634 wait complete. 2023-10-16T13:46:14.541 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' removed 2023-10-16T13:46:14.556 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-16T13:46:14.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.687+0000 7fadf8f55700 1 Processor -- start 2023-10-16T13:46:14.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.687+0000 7fadf8f55700 1 -- start start 2023-10-16T13:46:14.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.688+0000 7fadf8f55700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fadf40ce1a0 0x7fadf40ce620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:14.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.688+0000 7fadf8f55700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadf40ceef0 con 0x7fadf406ac30 2023-10-16T13:46:14.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.688+0000 7fadf8f55700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadf40cf1d0 con 0x7fadf4161370 2023-10-16T13:46:14.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.688+0000 7fadf8f55700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadf40af300 con 0x7fadf40ce1a0 2023-10-16T13:46:14.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.688+0000 7fadf1d9b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fadf40ce1a0 0x7fadf40ce620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:14.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.688+0000 7fadf2d9d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fadf406ac30 0x7fadf406b060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42110/0 (socket says 172.21.15.16:42110) 2023-10-16T13:46:14.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.688+0000 7fadf2d9d700 1 -- 172.21.15.16:0/17031037 learned_addr learned my addr 172.21.15.16:0/17031037 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:46:14.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.688+0000 7fadf1d9b700 1 -- 172.21.15.16:0/17031037 >> v1:172.21.15.49:6789/0 conn(0x7fadf4161370 legacy=0x7fadf4161750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:14.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.688+0000 7fadf1d9b700 1 -- 172.21.15.16:0/17031037 >> v1:172.21.15.16:6789/0 conn(0x7fadf406ac30 legacy=0x7fadf406b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:14.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.689+0000 7fadf1d9b700 1 -- 172.21.15.16:0/17031037 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadf40cf4b0 con 0x7fadf40ce1a0 2023-10-16T13:46:14.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.688+0000 7fadf159a700 1 -- 172.21.15.16:0/17031037 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2791890757 0 0) 0x7fadf40ceef0 con 0x7fadf406ac30 2023-10-16T13:46:14.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.689+0000 7fadf1d9b700 1 --2- 172.21.15.16:0/17031037 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fadf40ce1a0 0x7fadf40ce620 secure :-1 s=READY pgs=2405 cs=0 l=1 rev1=1 crypto rx=0x7faddc009840 tx=0x7faddc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23d2484bf5c87203 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:14.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.689+0000 7fadf159a700 1 -- 172.21.15.16:0/17031037 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faddc004230 con 0x7fadf40ce1a0 2023-10-16T13:46:14.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.689+0000 7fadf159a700 1 -- 172.21.15.16:0/17031037 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faddc004390 con 0x7fadf40ce1a0 2023-10-16T13:46:14.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.689+0000 7fadf159a700 1 -- 172.21.15.16:0/17031037 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faddc0036a0 con 0x7fadf40ce1a0 2023-10-16T13:46:14.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.689+0000 7fadf8f55700 1 -- 172.21.15.16:0/17031037 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fadf40ce1a0 msgr2=0x7fadf40ce620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:14.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.689+0000 7fadf8f55700 1 --2- 172.21.15.16:0/17031037 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fadf40ce1a0 0x7fadf40ce620 secure :-1 s=READY pgs=2405 cs=0 l=1 rev1=1 crypto rx=0x7faddc009840 tx=0x7faddc009b50 comp rx=0 tx=0).stop 2023-10-16T13:46:14.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.689+0000 7fadf8f55700 1 -- 172.21.15.16:0/17031037 shutdown_connections 2023-10-16T13:46:14.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.689+0000 7fadf8f55700 1 --2- 172.21.15.16:0/17031037 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fadf40ce1a0 0x7fadf40ce620 unknown :-1 s=CLOSED pgs=2405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:14.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.689+0000 7fadf8f55700 1 -- 172.21.15.16:0/17031037 >> 172.21.15.16:0/17031037 conn(0x7fadf40cc5c0 msgr2=0x7fadf40cc9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:14.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.689+0000 7fadf8f55700 1 -- 172.21.15.16:0/17031037 shutdown_connections 2023-10-16T13:46:14.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.689+0000 7fadf8f55700 1 -- 172.21.15.16:0/17031037 wait complete. 2023-10-16T13:46:14.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.691+0000 7fadf8f55700 1 Processor -- start 2023-10-16T13:46:14.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.691+0000 7fadf8f55700 1 -- start start 2023-10-16T13:46:14.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.692+0000 7fadf8f55700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fadf406ac30 0x7fadf4157bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:14.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.692+0000 7fadf8f55700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fadf40ce1a0 0x7fadf4149100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:14.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.692+0000 7fadf8f55700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fadf4161370 0x7fadf4158370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:14.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.692+0000 7fadf8f55700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadf41f7810 con 0x7fadf4161370 2023-10-16T13:46:14.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.692+0000 7fadf8f55700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadf415ef50 con 0x7fadf40ce1a0 2023-10-16T13:46:14.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.692+0000 7fadf8f55700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadf415f230 con 0x7fadf406ac30 2023-10-16T13:46:14.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.692+0000 7fadf2d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fadf4161370 0x7fadf4158370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:14.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.692+0000 7fadf2d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fadf4161370 0x7fadf4158370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52842/0 (socket says 172.21.15.16:52842) 2023-10-16T13:46:14.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.692+0000 7fadf2d9d700 1 -- 172.21.15.16:0/809239162 learned_addr learned my addr 172.21.15.16:0/809239162 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:14.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.692+0000 7fadf1d9b700 1 --2- 172.21.15.16:0/809239162 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fadf40ce1a0 0x7fadf4149100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:14.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.692+0000 7fadf259c700 1 --2- 172.21.15.16:0/809239162 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fadf406ac30 0x7fadf4157bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:14.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.692+0000 7fadf2d9d700 1 -- 172.21.15.16:0/809239162 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fadf406ac30 msgr2=0x7fadf4157bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:14.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.692+0000 7fadf2d9d700 1 --2- 172.21.15.16:0/809239162 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fadf406ac30 0x7fadf4157bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:14.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.692+0000 7fadf2d9d700 1 -- 172.21.15.16:0/809239162 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fadf40ce1a0 msgr2=0x7fadf4149100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:14.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.692+0000 7fadf2d9d700 1 --2- 172.21.15.16:0/809239162 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fadf40ce1a0 0x7fadf4149100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:14.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.692+0000 7fadf2d9d700 1 -- 172.21.15.16:0/809239162 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faddc0094a0 con 0x7fadf4161370 2023-10-16T13:46:14.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.692+0000 7fadf259c700 1 --2- 172.21.15.16:0/809239162 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fadf406ac30 0x7fadf4157bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:46:14.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.692+0000 7fadf2d9d700 1 --2- 172.21.15.16:0/809239162 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fadf4161370 0x7fadf4158370 secure :-1 s=READY pgs=2406 cs=0 l=1 rev1=1 crypto rx=0x7fadec00cba0 tx=0x7fadec00ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:14.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.692+0000 7fadeaffd700 1 -- 172.21.15.16:0/809239162 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadec00aad0 con 0x7fadf4161370 2023-10-16T13:46:14.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.692+0000 7fadeaffd700 1 -- 172.21.15.16:0/809239162 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadec00ac50 con 0x7fadf4161370 2023-10-16T13:46:14.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.692+0000 7fadf8f55700 1 -- 172.21.15.16:0/809239162 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadf415f510 con 0x7fadf4161370 2023-10-16T13:46:14.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.692+0000 7fadeaffd700 1 -- 172.21.15.16:0/809239162 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadec0070d0 con 0x7fadf4161370 2023-10-16T13:46:14.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.693+0000 7fadf8f55700 1 -- 172.21.15.16:0/809239162 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadf415fad0 con 0x7fadf4161370 2023-10-16T13:46:14.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.693+0000 7fadf8f55700 1 -- 172.21.15.16:0/809239162 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadf415fcf0 con 0x7fadf4161370 2023-10-16T13:46:14.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.694+0000 7fadeaffd700 1 -- 172.21.15.16:0/809239162 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 138) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fadec007230 con 0x7fadf4161370 2023-10-16T13:46:14.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.694+0000 7fadeaffd700 1 --2- 172.21.15.16:0/809239162 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fadd406b020 0x7fadd406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:14.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.694+0000 7fadeaffd700 1 -- 172.21.15.16:0/809239162 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6165+0+0 (secure 0 0 0) 0x7fadec0896e0 con 0x7fadf4161370 2023-10-16T13:46:14.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.694+0000 7fadf259c700 1 --2- 172.21.15.16:0/809239162 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fadd406b020 0x7fadd406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:14.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.695+0000 7fadf259c700 1 --2- 172.21.15.16:0/809239162 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fadd406b020 0x7fadd406d4e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fade4009930 tx=0x7fade4008040 comp rx=0 tx=0).ready entity=mgr.11190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:14.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.699+0000 7fadeaffd700 1 -- 172.21.15.16:0/809239162 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fadec0544b0 con 0x7fadf4161370 2023-10-16T13:46:14.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.906+0000 7fadf8f55700 1 -- 172.21.15.16:0/809239162 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7fadf404ebb0 con 0x7fadf4161370 2023-10-16T13:46:14.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.909+0000 7fadeaffd700 1 -- 172.21.15.16:0/809239162 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v73) v1 ==== 115+0+0 (secure 0 0 0) 0x7fadec087e60 con 0x7fadf4161370 2023-10-16T13:46:14.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.911+0000 7fadf8f55700 1 -- 172.21.15.16:0/809239162 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fadd406b020 msgr2=0x7fadd406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:14.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.911+0000 7fadf8f55700 1 --2- 172.21.15.16:0/809239162 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fadd406b020 0x7fadd406d4e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fade4009930 tx=0x7fade4008040 comp rx=0 tx=0).stop 2023-10-16T13:46:14.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.911+0000 7fadf8f55700 1 -- 172.21.15.16:0/809239162 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fadf4161370 msgr2=0x7fadf4158370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:14.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.911+0000 7fadf8f55700 1 --2- 172.21.15.16:0/809239162 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fadf4161370 0x7fadf4158370 secure :-1 s=READY pgs=2406 cs=0 l=1 rev1=1 crypto rx=0x7fadec00cba0 tx=0x7fadec00ceb0 comp rx=0 tx=0).stop 2023-10-16T13:46:14.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.911+0000 7fadf8f55700 1 -- 172.21.15.16:0/809239162 shutdown_connections 2023-10-16T13:46:14.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.911+0000 7fadf8f55700 1 --2- 172.21.15.16:0/809239162 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fadd406b020 0x7fadd406d4e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:14.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.911+0000 7fadf8f55700 1 --2- 172.21.15.16:0/809239162 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fadf4161370 0x7fadf4158370 unknown :-1 s=CLOSED pgs=2406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:14.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.911+0000 7fadf8f55700 1 --2- 172.21.15.16:0/809239162 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fadf406ac30 0x7fadf4157bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:14.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.911+0000 7fadf8f55700 1 --2- 172.21.15.16:0/809239162 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fadf40ce1a0 0x7fadf4149100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:14.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.911+0000 7fadf8f55700 1 -- 172.21.15.16:0/809239162 >> 172.21.15.16:0/809239162 conn(0x7fadf40cc5c0 msgr2=0x7fadf40afa70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:14.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.911+0000 7fadf8f55700 1 -- 172.21.15.16:0/809239162 shutdown_connections 2023-10-16T13:46:14.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:14.911+0000 7fadf8f55700 1 -- 172.21.15.16:0/809239162 wait complete. 2023-10-16T13:46:14.927 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-16T13:46:15.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.051+0000 7fbf8754b700 1 Processor -- start 2023-10-16T13:46:15.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.051+0000 7fbf8754b700 1 -- start start 2023-10-16T13:46:15.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.052+0000 7fbf8754b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbf8006e570 0x7fbf8015e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:15.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.052+0000 7fbf8754b700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf8015b6a0 con 0x7fbf8006a7e0 2023-10-16T13:46:15.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.052+0000 7fbf8754b700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf80161b20 con 0x7fbf8015b2c0 2023-10-16T13:46:15.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.052+0000 7fbf8754b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf80161e00 con 0x7fbf8006e570 2023-10-16T13:46:15.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.052+0000 7fbf85ae8700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fbf8006a7e0 0x7fbf80071a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42124/0 (socket says 172.21.15.16:42124) 2023-10-16T13:46:15.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.052+0000 7fbf85ae8700 1 -- 172.21.15.16:0/1641523089 learned_addr learned my addr 172.21.15.16:0/1641523089 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:46:15.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.053+0000 7fbf77fff700 1 -- 172.21.15.16:0/1641523089 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1170102784 0 0) 0x7fbf8015b6a0 con 0x7fbf8006a7e0 2023-10-16T13:46:15.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.053+0000 7fbf84ae6700 1 --2- 172.21.15.16:0/1641523089 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbf8006e570 0x7fbf8015e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:15.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.053+0000 7fbf77fff700 1 -- 172.21.15.16:0/1641523089 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf6c0035b0 con 0x7fbf8006a7e0 2023-10-16T13:46:15.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.053+0000 7fbf77fff700 1 -- 172.21.15.16:0/1641523089 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4261906122 0 0) 0x7fbf6c0035b0 con 0x7fbf8006a7e0 2023-10-16T13:46:15.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.053+0000 7fbf77fff700 1 -- 172.21.15.16:0/1641523089 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbf8006e570 msgr2=0x7fbf8015e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:15.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.053+0000 7fbf77fff700 1 --2- 172.21.15.16:0/1641523089 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbf8006e570 0x7fbf8015e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:15.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.053+0000 7fbf77fff700 1 -- 172.21.15.16:0/1641523089 >> v1:172.21.15.49:6789/0 conn(0x7fbf8015b2c0 legacy=0x7fbf80161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:15.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.053+0000 7fbf77fff700 1 -- 172.21.15.16:0/1641523089 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf801620e0 con 0x7fbf8006a7e0 2023-10-16T13:46:15.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.053+0000 7fbf77fff700 1 -- 172.21.15.16:0/1641523089 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7fbf780019f0 con 0x7fbf8006a7e0 2023-10-16T13:46:15.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.053+0000 7fbf77fff700 1 -- 172.21.15.16:0/1641523089 >> v1:172.21.15.16:6789/0 conn(0x7fbf8006a7e0 legacy=0x7fbf80071a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:15.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.053+0000 7fbf77fff700 1 --2- 172.21.15.16:0/1641523089 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf6c003a30 0x7fbf6c003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:15.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.053+0000 7fbf77fff700 1 --2- 172.21.15.16:0/1641523089 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbf6c004850 0x7fbf6c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:15.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.053+0000 7fbf77fff700 1 --2- 172.21.15.16:0/1641523089 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbf8006e570 0x7fbf6c005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:15.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.053+0000 7fbf77fff700 1 -- 172.21.15.16:0/1641523089 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf80161e00 con 0x7fbf6c003a30 2023-10-16T13:46:15.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.053+0000 7fbf77fff700 1 -- 172.21.15.16:0/1641523089 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf6c00bb40 con 0x7fbf6c004850 2023-10-16T13:46:15.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.053+0000 7fbf77fff700 1 -- 172.21.15.16:0/1641523089 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf6c00bd00 con 0x7fbf8006e570 2023-10-16T13:46:15.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.053+0000 7fbf852e7700 1 --2- 172.21.15.16:0/1641523089 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf6c003a30 0x7fbf6c003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:15.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.053+0000 7fbf84ae6700 1 --2- 172.21.15.16:0/1641523089 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbf6c004850 0x7fbf6c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:15.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.053+0000 7fbf85ae8700 1 --2- 172.21.15.16:0/1641523089 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbf8006e570 0x7fbf6c005250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:15.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.053+0000 7fbf852e7700 1 -- 172.21.15.16:0/1641523089 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbf8006e570 msgr2=0x7fbf6c005250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:15.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.053+0000 7fbf852e7700 1 --2- 172.21.15.16:0/1641523089 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbf8006e570 0x7fbf6c005250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:15.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.053+0000 7fbf852e7700 1 -- 172.21.15.16:0/1641523089 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbf6c004850 msgr2=0x7fbf6c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:15.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.053+0000 7fbf852e7700 1 --2- 172.21.15.16:0/1641523089 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbf6c004850 0x7fbf6c004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:15.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.053+0000 7fbf852e7700 1 -- 172.21.15.16:0/1641523089 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf6c0035b0 con 0x7fbf6c003a30 2023-10-16T13:46:15.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.054+0000 7fbf852e7700 1 --2- 172.21.15.16:0/1641523089 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf6c003a30 0x7fbf6c003e50 secure :-1 s=READY pgs=2408 cs=0 l=1 rev1=1 crypto rx=0x7fbf700017c0 tx=0x7fbf6c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43ed66351dedf698 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:15.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.054+0000 7fbf77fff700 1 -- 172.21.15.16:0/1641523089 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf700052d0 con 0x7fbf6c003a30 2023-10-16T13:46:15.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.054+0000 7fbf77fff700 1 -- 172.21.15.16:0/1641523089 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf70005430 con 0x7fbf6c003a30 2023-10-16T13:46:15.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.054+0000 7fbf77fff700 1 -- 172.21.15.16:0/1641523089 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf700038c0 con 0x7fbf6c003a30 2023-10-16T13:46:15.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.054+0000 7fbf8754b700 1 -- 172.21.15.16:0/1641523089 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf6c003a30 msgr2=0x7fbf6c003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:15.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.054+0000 7fbf8754b700 1 --2- 172.21.15.16:0/1641523089 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf6c003a30 0x7fbf6c003e50 secure :-1 s=READY pgs=2408 cs=0 l=1 rev1=1 crypto rx=0x7fbf700017c0 tx=0x7fbf6c005880 comp rx=0 tx=0).stop 2023-10-16T13:46:15.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.054+0000 7fbf85ae8700 1 -- 172.21.15.16:0/1641523089 reap_dead start 2023-10-16T13:46:15.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.054+0000 7fbf8754b700 1 -- 172.21.15.16:0/1641523089 shutdown_connections 2023-10-16T13:46:15.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.054+0000 7fbf8754b700 1 -- 172.21.15.16:0/1641523089 >> 172.21.15.16:0/1641523089 conn(0x7fbf800b1f40 msgr2=0x7fbf800b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:15.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.054+0000 7fbf8754b700 1 -- 172.21.15.16:0/1641523089 shutdown_connections 2023-10-16T13:46:15.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.054+0000 7fbf8754b700 1 -- 172.21.15.16:0/1641523089 wait complete. 2023-10-16T13:46:15.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.056+0000 7fbf8754b700 1 Processor -- start 2023-10-16T13:46:15.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.056+0000 7fbf8754b700 1 -- start start 2023-10-16T13:46:15.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.056+0000 7fbf8754b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbf6c003a30 0x7fbf80158750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:15.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.056+0000 7fbf8754b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf800c4640 0x7fbf800c90f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:15.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.056+0000 7fbf8754b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbf800c9630 0x7fbf800c74c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:15.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.056+0000 7fbf8754b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf800c9ab0 con 0x7fbf800c4640 2023-10-16T13:46:15.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.056+0000 7fbf8754b700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf8015eb30 con 0x7fbf6c003a30 2023-10-16T13:46:15.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.056+0000 7fbf8754b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf800c81b0 con 0x7fbf800c9630 2023-10-16T13:46:15.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.056+0000 7fbf84ae6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf800c4640 0x7fbf800c90f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:15.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.056+0000 7fbf84ae6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf800c4640 0x7fbf800c90f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52858/0 (socket says 172.21.15.16:52858) 2023-10-16T13:46:15.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.056+0000 7fbf84ae6700 1 -- 172.21.15.16:0/2620684627 learned_addr learned my addr 172.21.15.16:0/2620684627 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:15.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.057+0000 7fbf84ae6700 1 -- 172.21.15.16:0/2620684627 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbf800c9630 msgr2=0x7fbf800c74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:15.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.057+0000 7fbf84ae6700 1 --2- 172.21.15.16:0/2620684627 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbf800c9630 0x7fbf800c74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:15.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.057+0000 7fbf84ae6700 1 -- 172.21.15.16:0/2620684627 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbf6c003a30 msgr2=0x7fbf80158750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:15.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.057+0000 7fbf84ae6700 1 --2- 172.21.15.16:0/2620684627 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbf6c003a30 0x7fbf80158750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:15.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.057+0000 7fbf84ae6700 1 -- 172.21.15.16:0/2620684627 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf800c8490 con 0x7fbf800c4640 2023-10-16T13:46:15.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.057+0000 7fbf84ae6700 1 --2- 172.21.15.16:0/2620684627 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf800c4640 0x7fbf800c90f0 secure :-1 s=READY pgs=2409 cs=0 l=1 rev1=1 crypto rx=0x7fbf7c011aa0 tx=0x7fbf7c011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:15.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.057+0000 7fbf75ffb700 1 -- 172.21.15.16:0/2620684627 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf7c00f980 con 0x7fbf800c4640 2023-10-16T13:46:15.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.057+0000 7fbf75ffb700 1 -- 172.21.15.16:0/2620684627 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf7c00fae0 con 0x7fbf800c4640 2023-10-16T13:46:15.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.057+0000 7fbf8754b700 1 -- 172.21.15.16:0/2620684627 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf8015e4e0 con 0x7fbf800c4640 2023-10-16T13:46:15.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.057+0000 7fbf75ffb700 1 -- 172.21.15.16:0/2620684627 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf7c00fe90 con 0x7fbf800c4640 2023-10-16T13:46:15.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.057+0000 7fbf8754b700 1 -- 172.21.15.16:0/2620684627 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf80158c90 con 0x7fbf800c4640 2023-10-16T13:46:15.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.058+0000 7fbf8754b700 1 -- 172.21.15.16:0/2620684627 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf8004ebb0 con 0x7fbf800c4640 2023-10-16T13:46:15.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.059+0000 7fbf75ffb700 1 -- 172.21.15.16:0/2620684627 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 138) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fbf7c014440 con 0x7fbf800c4640 2023-10-16T13:46:15.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.059+0000 7fbf75ffb700 1 --2- 172.21.15.16:0/2620684627 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fbf6406b020 0x7fbf6406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:15.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.059+0000 7fbf75ffb700 1 -- 172.21.15.16:0/2620684627 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6165+0+0 (secure 0 0 0) 0x7fbf7c08a0d0 con 0x7fbf800c4640 2023-10-16T13:46:15.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.059+0000 7fbf852e7700 1 --2- 172.21.15.16:0/2620684627 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fbf6406b020 0x7fbf6406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:15.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.060+0000 7fbf852e7700 1 --2- 172.21.15.16:0/2620684627 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fbf6406b020 0x7fbf6406d4e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fbf70000c00 tx=0x7fbf70011000 comp rx=0 tx=0).ready entity=mgr.11190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:15.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.064+0000 7fbf75ffb700 1 -- 172.21.15.16:0/2620684627 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbf7c054f60 con 0x7fbf800c4640 2023-10-16T13:46:15.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.269+0000 7fbf8754b700 1 -- 172.21.15.16:0/2620684627 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fbf8015ec70 con 0x7fbf800c4640 2023-10-16T13:46:15.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.272+0000 7fbf75ffb700 1 -- 172.21.15.16:0/2620684627 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v74) v1 ==== 104+0+0 (secure 0 0 0) 0x7fbf8015ec70 con 0x7fbf800c4640 2023-10-16T13:46:15.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.275+0000 7fbf8754b700 1 -- 172.21.15.16:0/2620684627 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fbf6406b020 msgr2=0x7fbf6406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:15.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.275+0000 7fbf8754b700 1 --2- 172.21.15.16:0/2620684627 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fbf6406b020 0x7fbf6406d4e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fbf70000c00 tx=0x7fbf70011000 comp rx=0 tx=0).stop 2023-10-16T13:46:15.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.275+0000 7fbf8754b700 1 -- 172.21.15.16:0/2620684627 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf800c4640 msgr2=0x7fbf800c90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:15.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.275+0000 7fbf8754b700 1 --2- 172.21.15.16:0/2620684627 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf800c4640 0x7fbf800c90f0 secure :-1 s=READY pgs=2409 cs=0 l=1 rev1=1 crypto rx=0x7fbf7c011aa0 tx=0x7fbf7c011e60 comp rx=0 tx=0).stop 2023-10-16T13:46:15.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.275+0000 7fbf8754b700 1 -- 172.21.15.16:0/2620684627 shutdown_connections 2023-10-16T13:46:15.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.275+0000 7fbf8754b700 1 --2- 172.21.15.16:0/2620684627 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbf800c9630 0x7fbf800c74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:15.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.275+0000 7fbf8754b700 1 --2- 172.21.15.16:0/2620684627 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbf6c003a30 0x7fbf80158750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:15.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.275+0000 7fbf8754b700 1 --2- 172.21.15.16:0/2620684627 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fbf6406b020 0x7fbf6406d4e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:15.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.275+0000 7fbf8754b700 1 --2- 172.21.15.16:0/2620684627 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf800c4640 0x7fbf800c90f0 unknown :-1 s=CLOSED pgs=2409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:15.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.275+0000 7fbf8754b700 1 -- 172.21.15.16:0/2620684627 >> 172.21.15.16:0/2620684627 conn(0x7fbf800b1f40 msgr2=0x7fbf80071970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:15.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.275+0000 7fbf8754b700 1 -- 172.21.15.16:0/2620684627 shutdown_connections 2023-10-16T13:46:15.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.275+0000 7fbf8754b700 1 -- 172.21.15.16:0/2620684627 wait complete. 2023-10-16T13:46:15.289 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-16T13:46:15.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.414+0000 7f2fe5ea7700 1 Processor -- start 2023-10-16T13:46:15.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.414+0000 7f2fe5ea7700 1 -- start start 2023-10-16T13:46:15.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.415+0000 7f2fe5ea7700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2fe00cb880 0x7f2fe00cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:15.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.415+0000 7f2fe5ea7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2fe006da60 0x7f2fe006def0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:15.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.415+0000 7f2fe5ea7700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fe00cef70 con 0x7f2fe015b070 2023-10-16T13:46:15.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.415+0000 7f2fe5ea7700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fe006e430 con 0x7f2fe006da60 2023-10-16T13:46:15.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.415+0000 7f2fe5ea7700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe00ce360 con 0x7f2fe00cb880 2023-10-16T13:46:15.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.415+0000 7f2fdffff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2fe006da60 0x7f2fe006def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:15.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.415+0000 7f2fdffff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2fe006da60 0x7f2fe006def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52870/0 (socket says 172.21.15.16:52870) 2023-10-16T13:46:15.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.415+0000 7f2fdffff700 1 -- 172.21.15.16:0/1118716287 learned_addr learned my addr 172.21.15.16:0/1118716287 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:15.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.415+0000 7f2fdeffd700 1 --2- 172.21.15.16:0/1118716287 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2fe00cb880 0x7f2fe00cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:15.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.415+0000 7f2fdffff700 1 -- 172.21.15.16:0/1118716287 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2fe00cb880 msgr2=0x7f2fe00cbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:15.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.415+0000 7f2fdffff700 1 --2- 172.21.15.16:0/1118716287 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2fe00cb880 0x7f2fe00cbd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:15.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.415+0000 7f2fdffff700 1 -- 172.21.15.16:0/1118716287 >> v1:172.21.15.49:6789/0 conn(0x7f2fe015b070 legacy=0x7f2fe015b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:15.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.416+0000 7f2fdffff700 1 -- 172.21.15.16:0/1118716287 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fe00ce640 con 0x7f2fe006da60 2023-10-16T13:46:15.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.416+0000 7f2fdffff700 1 --2- 172.21.15.16:0/1118716287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2fe006da60 0x7f2fe006def0 secure :-1 s=READY pgs=2410 cs=0 l=1 rev1=1 crypto rx=0x7f2fd8009e50 tx=0x7f2fd800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dd66b831900213e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:15.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.416+0000 7f2fde7fc700 1 -- 172.21.15.16:0/1118716287 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd800e070 con 0x7f2fe006da60 2023-10-16T13:46:15.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.416+0000 7f2fde7fc700 1 -- 172.21.15.16:0/1118716287 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fd800b9c0 con 0x7f2fe006da60 2023-10-16T13:46:15.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.416+0000 7f2fde7fc700 1 -- 172.21.15.16:0/1118716287 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd800bd70 con 0x7f2fe006da60 2023-10-16T13:46:15.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.416+0000 7f2fe5ea7700 1 -- 172.21.15.16:0/1118716287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2fe006da60 msgr2=0x7f2fe006def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:15.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.416+0000 7f2fe5ea7700 1 --2- 172.21.15.16:0/1118716287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2fe006da60 0x7f2fe006def0 secure :-1 s=READY pgs=2410 cs=0 l=1 rev1=1 crypto rx=0x7f2fd8009e50 tx=0x7f2fd800b330 comp rx=0 tx=0).stop 2023-10-16T13:46:15.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.416+0000 7f2fe5ea7700 1 -- 172.21.15.16:0/1118716287 shutdown_connections 2023-10-16T13:46:15.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.416+0000 7f2fe5ea7700 1 --2- 172.21.15.16:0/1118716287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2fe006da60 0x7f2fe006def0 unknown :-1 s=CLOSED pgs=2410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:15.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.416+0000 7f2fe5ea7700 1 --2- 172.21.15.16:0/1118716287 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2fe00cb880 0x7f2fe00cbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:15.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.416+0000 7f2fe5ea7700 1 -- 172.21.15.16:0/1118716287 >> 172.21.15.16:0/1118716287 conn(0x7f2fe00b1f40 msgr2=0x7f2fe00b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:15.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.416+0000 7f2fe5ea7700 1 -- 172.21.15.16:0/1118716287 shutdown_connections 2023-10-16T13:46:15.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.417+0000 7f2fe5ea7700 1 -- 172.21.15.16:0/1118716287 wait complete. 2023-10-16T13:46:15.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.418+0000 7f2fe5ea7700 1 Processor -- start 2023-10-16T13:46:15.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.418+0000 7f2fe5ea7700 1 -- start start 2023-10-16T13:46:15.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.419+0000 7f2fe5ea7700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2fe006da60 0x7f2fe01f0790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:15.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.419+0000 7f2fe5ea7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2fe00cb880 0x7f2fe00ca820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:15.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.419+0000 7f2fe5ea7700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2fe015b070 0x7f2fe00c8c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:15.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.419+0000 7f2fe5ea7700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe00c6840 con 0x7f2fe00cb880 2023-10-16T13:46:15.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.419+0000 7f2fe5ea7700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe00c9780 con 0x7f2fe015b070 2023-10-16T13:46:15.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.419+0000 7f2fe5ea7700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe00c9a60 con 0x7f2fe006da60 2023-10-16T13:46:15.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.419+0000 7f2fdeffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2fe00cb880 0x7f2fe00ca820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:15.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.419+0000 7f2fdeffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2fe00cb880 0x7f2fe00ca820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52874/0 (socket says 172.21.15.16:52874) 2023-10-16T13:46:15.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.419+0000 7f2fdeffd700 1 -- 172.21.15.16:0/2167140121 learned_addr learned my addr 172.21.15.16:0/2167140121 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:15.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.419+0000 7f2fdffff700 1 --2- 172.21.15.16:0/2167140121 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2fe015b070 0x7f2fe00c8c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:15.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.419+0000 7f2fdf7fe700 1 --2- 172.21.15.16:0/2167140121 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2fe006da60 0x7f2fe01f0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:15.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.419+0000 7f2fdeffd700 1 -- 172.21.15.16:0/2167140121 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2fe006da60 msgr2=0x7f2fe01f0790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:15.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.419+0000 7f2fdeffd700 1 --2- 172.21.15.16:0/2167140121 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2fe006da60 0x7f2fe01f0790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:15.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.419+0000 7f2fdeffd700 1 -- 172.21.15.16:0/2167140121 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2fe015b070 msgr2=0x7f2fe00c8c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:15.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.419+0000 7f2fdeffd700 1 --2- 172.21.15.16:0/2167140121 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2fe015b070 0x7f2fe00c8c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:15.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.419+0000 7f2fdeffd700 1 -- 172.21.15.16:0/2167140121 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fd8009b30 con 0x7f2fe00cb880 2023-10-16T13:46:15.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.419+0000 7f2fdeffd700 1 --2- 172.21.15.16:0/2167140121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2fe00cb880 0x7f2fe00ca820 secure :-1 s=READY pgs=2411 cs=0 l=1 rev1=1 crypto rx=0x7f2fd000e810 tx=0x7f2fd000ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:15.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.419+0000 7f2fd7fff700 1 -- 172.21.15.16:0/2167140121 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd0013610 con 0x7f2fe00cb880 2023-10-16T13:46:15.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.419+0000 7f2fd7fff700 1 -- 172.21.15.16:0/2167140121 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fd0013770 con 0x7f2fe00cb880 2023-10-16T13:46:15.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.419+0000 7f2fd7fff700 1 -- 172.21.15.16:0/2167140121 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd0013b20 con 0x7f2fe00cb880 2023-10-16T13:46:15.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.419+0000 7f2fe5ea7700 1 -- 172.21.15.16:0/2167140121 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fe00c9d40 con 0x7f2fe00cb880 2023-10-16T13:46:15.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.419+0000 7f2fe5ea7700 1 -- 172.21.15.16:0/2167140121 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fe00ca300 con 0x7f2fe00cb880 2023-10-16T13:46:15.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.420+0000 7f2fe5ea7700 1 -- 172.21.15.16:0/2167140121 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fe004ebb0 con 0x7f2fe00cb880 2023-10-16T13:46:15.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.421+0000 7f2fd7fff700 1 -- 172.21.15.16:0/2167140121 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 138) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2fd0013c80 con 0x7f2fe00cb880 2023-10-16T13:46:15.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.421+0000 7f2fd7fff700 1 --2- 172.21.15.16:0/2167140121 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f2fc006b020 0x7f2fc006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:15.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.421+0000 7f2fdf7fe700 1 --2- 172.21.15.16:0/2167140121 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f2fc006b020 0x7f2fc006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:15.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.421+0000 7f2fd7fff700 1 -- 172.21.15.16:0/2167140121 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6165+0+0 (secure 0 0 0) 0x7f2fd0089780 con 0x7f2fe00cb880 2023-10-16T13:46:15.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.422+0000 7f2fdf7fe700 1 --2- 172.21.15.16:0/2167140121 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f2fc006b020 0x7f2fc006d4e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f2fc8000f60 tx=0x7f2fc8009040 comp rx=0 tx=0).ready entity=mgr.11190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:15.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.426+0000 7f2fd7fff700 1 -- 172.21.15.16:0/2167140121 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2fd0054440 con 0x7f2fe00cb880 2023-10-16T13:46:15.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.633+0000 7f2fe5ea7700 1 -- 172.21.15.16:0/2167140121 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f2fe00728a0 con 0x7f2fe00cb880 2023-10-16T13:46:15.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.636+0000 7f2fd7fff700 1 -- 172.21.15.16:0/2167140121 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v75) v1 ==== 120+0+0 (secure 0 0 0) 0x7f2fd0057a60 con 0x7f2fe00cb880 2023-10-16T13:46:15.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.638+0000 7f2fe5ea7700 1 -- 172.21.15.16:0/2167140121 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f2fc006b020 msgr2=0x7f2fc006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:15.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.638+0000 7f2fe5ea7700 1 --2- 172.21.15.16:0/2167140121 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f2fc006b020 0x7f2fc006d4e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f2fc8000f60 tx=0x7f2fc8009040 comp rx=0 tx=0).stop 2023-10-16T13:46:15.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.638+0000 7f2fe5ea7700 1 -- 172.21.15.16:0/2167140121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2fe00cb880 msgr2=0x7f2fe00ca820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:15.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.638+0000 7f2fe5ea7700 1 --2- 172.21.15.16:0/2167140121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2fe00cb880 0x7f2fe00ca820 secure :-1 s=READY pgs=2411 cs=0 l=1 rev1=1 crypto rx=0x7f2fd000e810 tx=0x7f2fd000ebd0 comp rx=0 tx=0).stop 2023-10-16T13:46:15.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.638+0000 7f2fe5ea7700 1 -- 172.21.15.16:0/2167140121 shutdown_connections 2023-10-16T13:46:15.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.638+0000 7f2fe5ea7700 1 --2- 172.21.15.16:0/2167140121 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2fe015b070 0x7f2fe00c8c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:15.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.638+0000 7f2fe5ea7700 1 --2- 172.21.15.16:0/2167140121 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2fe006da60 0x7f2fe01f0790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:15.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.638+0000 7f2fe5ea7700 1 --2- 172.21.15.16:0/2167140121 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f2fc006b020 0x7f2fc006d4e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:15.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.638+0000 7f2fe5ea7700 1 --2- 172.21.15.16:0/2167140121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2fe00cb880 0x7f2fe00ca820 unknown :-1 s=CLOSED pgs=2411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:15.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.638+0000 7f2fe5ea7700 1 -- 172.21.15.16:0/2167140121 >> 172.21.15.16:0/2167140121 conn(0x7f2fe00b1f40 msgr2=0x7f2fe00d0380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:15.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.638+0000 7f2fe5ea7700 1 -- 172.21.15.16:0/2167140121 shutdown_connections 2023-10-16T13:46:15.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.638+0000 7f2fe5ea7700 1 -- 172.21.15.16:0/2167140121 wait complete. 2023-10-16T13:46:15.653 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention' 2023-10-16T13:46:15.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.776+0000 7fb95ac3c700 1 Processor -- start 2023-10-16T13:46:15.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.776+0000 7fb95ac3c700 1 -- start start 2023-10-16T13:46:15.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.777+0000 7fb95ac3c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb9540cf4a0 0x7fb9540726e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:15.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.777+0000 7fb95ac3c700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fb9540cf880 0x7fb9540b52f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:15.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.777+0000 7fb95ac3c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb9540b5830 0x7fb95415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:15.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.777+0000 7fb95ac3c700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb95415bc80 con 0x7fb9540cf4a0 2023-10-16T13:46:15.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.777+0000 7fb95ac3c700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb954072c20 con 0x7fb9540cf880 2023-10-16T13:46:15.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.777+0000 7fb95ac3c700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb95415b070 con 0x7fb9540b5830 2023-10-16T13:46:15.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.777+0000 7fb9589d8700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb9540cf4a0 0x7fb9540726e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:15.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.777+0000 7fb9589d8700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb9540cf4a0 0x7fb9540726e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52880/0 (socket says 172.21.15.16:52880) 2023-10-16T13:46:15.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.777+0000 7fb9589d8700 1 -- 172.21.15.16:0/3590388078 learned_addr learned my addr 172.21.15.16:0/3590388078 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:15.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.777+0000 7fb953fff700 1 --2- 172.21.15.16:0/3590388078 >> v2:172.21.15.49:3300/0 conn(0x7fb9540cf880 0x7fb9540b52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:15.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.777+0000 7fb9591d9700 1 --2- 172.21.15.16:0/3590388078 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb9540b5830 0x7fb95415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:15.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.777+0000 7fb9589d8700 1 -- 172.21.15.16:0/3590388078 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb9540b5830 msgr2=0x7fb95415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:15.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.777+0000 7fb9589d8700 1 --2- 172.21.15.16:0/3590388078 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb9540b5830 0x7fb95415a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:15.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.777+0000 7fb9589d8700 1 -- 172.21.15.16:0/3590388078 >> v2:172.21.15.49:3300/0 conn(0x7fb9540cf880 msgr2=0x7fb9540b52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:15.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.777+0000 7fb9589d8700 1 --2- 172.21.15.16:0/3590388078 >> v2:172.21.15.49:3300/0 conn(0x7fb9540cf880 0x7fb9540b52f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:15.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.777+0000 7fb9589d8700 1 -- 172.21.15.16:0/3590388078 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb95415b350 con 0x7fb9540cf4a0 2023-10-16T13:46:15.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.777+0000 7fb953fff700 1 --2- 172.21.15.16:0/3590388078 >> v2:172.21.15.49:3300/0 conn(0x7fb9540cf880 0x7fb9540b52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:46:15.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.778+0000 7fb9589d8700 1 --2- 172.21.15.16:0/3590388078 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9540cf4a0 0x7fb9540726e0 secure :-1 s=READY pgs=2412 cs=0 l=1 rev1=1 crypto rx=0x7fb954071da0 tx=0x7fb948009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38830eeb66858fd7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:15.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.778+0000 7fb9537fe700 1 -- 172.21.15.16:0/3590388078 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb94800b870 con 0x7fb9540cf4a0 2023-10-16T13:46:15.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.778+0000 7fb9537fe700 1 -- 172.21.15.16:0/3590388078 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb94800b9d0 con 0x7fb9540cf4a0 2023-10-16T13:46:15.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.778+0000 7fb9537fe700 1 -- 172.21.15.16:0/3590388078 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb94800bd60 con 0x7fb9540cf4a0 2023-10-16T13:46:15.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.778+0000 7fb95ac3c700 1 -- 172.21.15.16:0/3590388078 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9540cf4a0 msgr2=0x7fb9540726e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:15.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.778+0000 7fb95ac3c700 1 --2- 172.21.15.16:0/3590388078 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9540cf4a0 0x7fb9540726e0 secure :-1 s=READY pgs=2412 cs=0 l=1 rev1=1 crypto rx=0x7fb954071da0 tx=0x7fb948009bd0 comp rx=0 tx=0).stop 2023-10-16T13:46:15.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.778+0000 7fb95ac3c700 1 -- 172.21.15.16:0/3590388078 shutdown_connections 2023-10-16T13:46:15.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.778+0000 7fb95ac3c700 1 --2- 172.21.15.16:0/3590388078 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb9540b5830 0x7fb95415a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:15.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.778+0000 7fb95ac3c700 1 --2- 172.21.15.16:0/3590388078 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9540cf4a0 0x7fb9540726e0 unknown :-1 s=CLOSED pgs=2412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:15.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.778+0000 7fb95ac3c700 1 --2- 172.21.15.16:0/3590388078 >> v2:172.21.15.49:3300/0 conn(0x7fb9540cf880 0x7fb9540b52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:15.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.778+0000 7fb95ac3c700 1 -- 172.21.15.16:0/3590388078 >> 172.21.15.16:0/3590388078 conn(0x7fb9540b1fd0 msgr2=0x7fb9540b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:15.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.778+0000 7fb95ac3c700 1 -- 172.21.15.16:0/3590388078 shutdown_connections 2023-10-16T13:46:15.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.778+0000 7fb95ac3c700 1 -- 172.21.15.16:0/3590388078 wait complete. 2023-10-16T13:46:15.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.780+0000 7fb95ac3c700 1 Processor -- start 2023-10-16T13:46:15.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.780+0000 7fb95ac3c700 1 -- start start 2023-10-16T13:46:15.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.780+0000 7fb95ac3c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb9540b5830 0x7fb9540c9e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:15.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.780+0000 7fb95ac3c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb9540cf4a0 0x7fb9540c5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:15.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.781+0000 7fb95ac3c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9540cf880 0x7fb9540c4510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:15.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.781+0000 7fb95ac3c700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9540c3100 con 0x7fb9540cf880 2023-10-16T13:46:15.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.781+0000 7fb95ac3c700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9540ca380 con 0x7fb9540b5830 2023-10-16T13:46:15.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.781+0000 7fb95ac3c700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9540c8560 con 0x7fb9540cf4a0 2023-10-16T13:46:15.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.781+0000 7fb9591d9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9540cf880 0x7fb9540c4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:15.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.781+0000 7fb9591d9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9540cf880 0x7fb9540c4510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52890/0 (socket says 172.21.15.16:52890) 2023-10-16T13:46:15.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.781+0000 7fb9591d9700 1 -- 172.21.15.16:0/891489662 learned_addr learned my addr 172.21.15.16:0/891489662 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:15.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.781+0000 7fb9589d8700 1 --2- 172.21.15.16:0/891489662 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb9540b5830 0x7fb9540c9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:15.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.781+0000 7fb953fff700 1 --2- 172.21.15.16:0/891489662 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb9540cf4a0 0x7fb9540c5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:15.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.781+0000 7fb9591d9700 1 -- 172.21.15.16:0/891489662 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb9540cf4a0 msgr2=0x7fb9540c5fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:15.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.781+0000 7fb9591d9700 1 --2- 172.21.15.16:0/891489662 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb9540cf4a0 0x7fb9540c5fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:15.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.781+0000 7fb9591d9700 1 -- 172.21.15.16:0/891489662 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb9540b5830 msgr2=0x7fb9540c9e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:15.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.781+0000 7fb9591d9700 1 --2- 172.21.15.16:0/891489662 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb9540b5830 0x7fb9540c9e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:15.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.781+0000 7fb9591d9700 1 -- 172.21.15.16:0/891489662 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb948009550 con 0x7fb9540cf880 2023-10-16T13:46:15.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.781+0000 7fb953fff700 1 --2- 172.21.15.16:0/891489662 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb9540cf4a0 0x7fb9540c5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:46:15.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.781+0000 7fb9591d9700 1 --2- 172.21.15.16:0/891489662 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9540cf880 0x7fb9540c4510 secure :-1 s=READY pgs=2413 cs=0 l=1 rev1=1 crypto rx=0x7fb94c007e60 tx=0x7fb94c00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:15.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.781+0000 7fb9517fa700 1 -- 172.21.15.16:0/891489662 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb94c00ede0 con 0x7fb9540cf880 2023-10-16T13:46:15.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.781+0000 7fb9517fa700 1 -- 172.21.15.16:0/891489662 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb94c014420 con 0x7fb9540cf880 2023-10-16T13:46:15.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.781+0000 7fb95ac3c700 1 -- 172.21.15.16:0/891489662 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9540c64e0 con 0x7fb9540cf880 2023-10-16T13:46:15.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.781+0000 7fb9517fa700 1 -- 172.21.15.16:0/891489662 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb94c0147d0 con 0x7fb9540cf880 2023-10-16T13:46:15.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.781+0000 7fb95ac3c700 1 -- 172.21.15.16:0/891489662 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9540c6a10 con 0x7fb9540cf880 2023-10-16T13:46:15.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.782+0000 7fb95ac3c700 1 -- 172.21.15.16:0/891489662 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb95404ebb0 con 0x7fb9540cf880 2023-10-16T13:46:15.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.783+0000 7fb9517fa700 1 -- 172.21.15.16:0/891489662 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 138) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb94c014930 con 0x7fb9540cf880 2023-10-16T13:46:15.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.783+0000 7fb9517fa700 1 --2- 172.21.15.16:0/891489662 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fb93806b020 0x7fb93806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:15.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.783+0000 7fb9517fa700 1 -- 172.21.15.16:0/891489662 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6165+0+0 (secure 0 0 0) 0x7fb94c08a010 con 0x7fb9540cf880 2023-10-16T13:46:15.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.783+0000 7fb9589d8700 1 --2- 172.21.15.16:0/891489662 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fb93806b020 0x7fb93806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:15.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.784+0000 7fb9589d8700 1 --2- 172.21.15.16:0/891489662 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fb93806b020 0x7fb93806d4e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb948000c00 tx=0x7fb948008030 comp rx=0 tx=0).ready entity=mgr.11190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:15.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:15.788+0000 7fb9517fa700 1 -- 172.21.15.16:0/891489662 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb94c054e70 con 0x7fb9540cf880 2023-10-16T13:46:16.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.080+0000 7fb95ac3c700 1 -- 172.21.15.16:0/891489662 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]} v 0) v1 -- 0x7fb9540b5610 con 0x7fb9540cf880 2023-10-16T13:46:16.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.526+0000 7fb9517fa700 1 -- 172.21.15.16:0/891489662 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]}]=0 v3120) v1 ==== 163+0+0 (secure 0 0 0) 0x7fb94c058490 con 0x7fb9540cf880 2023-10-16T13:46:16.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.529+0000 7fb95ac3c700 1 -- 172.21.15.16:0/891489662 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fb93806b020 msgr2=0x7fb93806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:16.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.529+0000 7fb95ac3c700 1 --2- 172.21.15.16:0/891489662 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fb93806b020 0x7fb93806d4e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb948000c00 tx=0x7fb948008030 comp rx=0 tx=0).stop 2023-10-16T13:46:16.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.529+0000 7fb95ac3c700 1 -- 172.21.15.16:0/891489662 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9540cf880 msgr2=0x7fb9540c4510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:16.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.529+0000 7fb95ac3c700 1 --2- 172.21.15.16:0/891489662 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9540cf880 0x7fb9540c4510 secure :-1 s=READY pgs=2413 cs=0 l=1 rev1=1 crypto rx=0x7fb94c007e60 tx=0x7fb94c00e3c0 comp rx=0 tx=0).stop 2023-10-16T13:46:16.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.529+0000 7fb95ac3c700 1 -- 172.21.15.16:0/891489662 shutdown_connections 2023-10-16T13:46:16.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.529+0000 7fb95ac3c700 1 --2- 172.21.15.16:0/891489662 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fb93806b020 0x7fb93806d4e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:16.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.529+0000 7fb95ac3c700 1 --2- 172.21.15.16:0/891489662 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9540cf880 0x7fb9540c4510 unknown :-1 s=CLOSED pgs=2413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:16.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.529+0000 7fb95ac3c700 1 --2- 172.21.15.16:0/891489662 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb9540b5830 0x7fb9540c9e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:16.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.529+0000 7fb95ac3c700 1 --2- 172.21.15.16:0/891489662 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb9540cf4a0 0x7fb9540c5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:16.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.529+0000 7fb95ac3c700 1 -- 172.21.15.16:0/891489662 >> 172.21.15.16:0/891489662 conn(0x7fb9540b1fd0 msgr2=0x7fb9540b4dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:16.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.529+0000 7fb95ac3c700 1 -- 172.21.15.16:0/891489662 shutdown_connections 2023-10-16T13:46:16.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.529+0000 7fb95ac3c700 1 -- 172.21.15.16:0/891489662 wait complete. 2023-10-16T13:46:16.546 INFO:tasks.cephfs_test_runner:Test scheduled snapshots along with rentention policy ... ok 2023-10-16T13:46:16.547 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_for_non_default_fs (tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument) Test command succes with --fs argument for non-default fs 2023-10-16T13:46:16.547 INFO:tasks.cephfs_test_runner:test_snap_schedule_for_non_default_fs (tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument) 2023-10-16T13:46:16.548 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:46:16.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.677+0000 7f0f42155700 1 Processor -- start 2023-10-16T13:46:16.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.677+0000 7f0f42155700 1 -- start start 2023-10-16T13:46:16.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.678+0000 7f0f42155700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0f3c15a870 0x7f0f3c06eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:16.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.678+0000 7f0f42155700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f0f3c0cbed0 0x7f0f3c0cde10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:16.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.678+0000 7f0f42155700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f3c06f0a0 con 0x7f0f3c06d350 2023-10-16T13:46:16.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.678+0000 7f0f42155700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f3c15ac50 con 0x7f0f3c15a870 2023-10-16T13:46:16.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.678+0000 7f0f42155700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f3c06f380 con 0x7f0f3c0cbed0 2023-10-16T13:46:16.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.678+0000 7f0f3b7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0f3c15a870 0x7f0f3c06eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:16.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.678+0000 7f0f3affd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0f3c06d350 0x7f0f3c0cb7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42168/0 (socket says 172.21.15.16:42168) 2023-10-16T13:46:16.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.678+0000 7f0f3affd700 1 -- 172.21.15.16:0/3944313029 learned_addr learned my addr 172.21.15.16:0/3944313029 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:46:16.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.679+0000 7f0f3bfff700 1 --2- 172.21.15.16:0/3944313029 >> v2:172.21.15.49:3300/0 conn(0x7f0f3c0cbed0 0x7f0f3c0cde10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:16.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.679+0000 7f0f3a7fc700 1 -- 172.21.15.16:0/3944313029 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3263251951 0 0) 0x7f0f3c06f0a0 con 0x7f0f3c06d350 2023-10-16T13:46:16.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.679+0000 7f0f3a7fc700 1 -- 172.21.15.16:0/3944313029 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f280035b0 con 0x7f0f3c06d350 2023-10-16T13:46:16.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.679+0000 7f0f3b7fe700 1 -- 172.21.15.16:0/3944313029 >> v2:172.21.15.49:3300/0 conn(0x7f0f3c0cbed0 msgr2=0x7f0f3c0cde10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:16.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.679+0000 7f0f3b7fe700 1 --2- 172.21.15.16:0/3944313029 >> v2:172.21.15.49:3300/0 conn(0x7f0f3c0cbed0 0x7f0f3c0cde10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:16.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.679+0000 7f0f3b7fe700 1 -- 172.21.15.16:0/3944313029 >> v1:172.21.15.16:6789/0 conn(0x7f0f3c06d350 legacy=0x7f0f3c0cb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:16.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.679+0000 7f0f3b7fe700 1 -- 172.21.15.16:0/3944313029 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f3c06f660 con 0x7f0f3c15a870 2023-10-16T13:46:16.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.679+0000 7f0f3b7fe700 1 --2- 172.21.15.16:0/3944313029 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f3c15a870 0x7f0f3c06eaa0 secure :-1 s=READY pgs=2415 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c066e00 tx=0x7f0f2c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9bfc732b4622e00 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:16.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.679+0000 7f0f3a7fc700 1 -- 172.21.15.16:0/3944313029 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f2c011830 con 0x7f0f3c15a870 2023-10-16T13:46:16.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.679+0000 7f0f3a7fc700 1 -- 172.21.15.16:0/3944313029 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f2c011990 con 0x7f0f3c15a870 2023-10-16T13:46:16.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.679+0000 7f0f3a7fc700 1 -- 172.21.15.16:0/3944313029 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f2c011d20 con 0x7f0f3c15a870 2023-10-16T13:46:16.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.680+0000 7f0f42155700 1 -- 172.21.15.16:0/3944313029 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f3c15a870 msgr2=0x7f0f3c06eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:16.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.680+0000 7f0f42155700 1 --2- 172.21.15.16:0/3944313029 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f3c15a870 0x7f0f3c06eaa0 secure :-1 s=READY pgs=2415 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c066e00 tx=0x7f0f2c009b20 comp rx=0 tx=0).stop 2023-10-16T13:46:16.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.680+0000 7f0f42155700 1 -- 172.21.15.16:0/3944313029 shutdown_connections 2023-10-16T13:46:16.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.680+0000 7f0f42155700 1 --2- 172.21.15.16:0/3944313029 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f3c15a870 0x7f0f3c06eaa0 unknown :-1 s=CLOSED pgs=2415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:16.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.680+0000 7f0f42155700 1 --2- 172.21.15.16:0/3944313029 >> v2:172.21.15.49:3300/0 conn(0x7f0f3c0cbed0 0x7f0f3c0cde10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:16.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.680+0000 7f0f42155700 1 -- 172.21.15.16:0/3944313029 >> 172.21.15.16:0/3944313029 conn(0x7f0f3c0cc5b0 msgr2=0x7f0f3c0cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:16.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.680+0000 7f0f42155700 1 -- 172.21.15.16:0/3944313029 shutdown_connections 2023-10-16T13:46:16.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.680+0000 7f0f42155700 1 -- 172.21.15.16:0/3944313029 wait complete. 2023-10-16T13:46:16.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.681+0000 7f0f42155700 1 Processor -- start 2023-10-16T13:46:16.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.681+0000 7f0f42155700 1 -- start start 2023-10-16T13:46:16.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.682+0000 7f0f42155700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f3c06d350 0x7f0f3c14de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:16.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.682+0000 7f0f42155700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f3c0cbed0 0x7f0f3c165310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:16.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.682+0000 7f0f42155700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f3c15a870 0x7f0f3c1669b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:16.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.682+0000 7f0f42155700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f3c0c47b0 con 0x7f0f3c06d350 2023-10-16T13:46:16.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.682+0000 7f0f42155700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f3c165c10 con 0x7f0f3c15a870 2023-10-16T13:46:16.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.682+0000 7f0f42155700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f3c165ef0 con 0x7f0f3c0cbed0 2023-10-16T13:46:16.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.682+0000 7f0f3b7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f3c06d350 0x7f0f3c14de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:16.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.682+0000 7f0f3b7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f3c06d350 0x7f0f3c14de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52900/0 (socket says 172.21.15.16:52900) 2023-10-16T13:46:16.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.682+0000 7f0f3b7fe700 1 -- 172.21.15.16:0/2619467575 learned_addr learned my addr 172.21.15.16:0/2619467575 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:16.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.682+0000 7f0f3bfff700 1 --2- 172.21.15.16:0/2619467575 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f3c15a870 0x7f0f3c1669b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:16.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.682+0000 7f0f3affd700 1 --2- 172.21.15.16:0/2619467575 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f3c0cbed0 0x7f0f3c165310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:16.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.682+0000 7f0f3b7fe700 1 -- 172.21.15.16:0/2619467575 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f3c0cbed0 msgr2=0x7f0f3c165310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:16.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.682+0000 7f0f3b7fe700 1 --2- 172.21.15.16:0/2619467575 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f3c0cbed0 0x7f0f3c165310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:16.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.682+0000 7f0f3b7fe700 1 -- 172.21.15.16:0/2619467575 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f3c15a870 msgr2=0x7f0f3c1669b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:16.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.682+0000 7f0f3b7fe700 1 --2- 172.21.15.16:0/2619467575 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f3c15a870 0x7f0f3c1669b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:16.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.682+0000 7f0f3b7fe700 1 -- 172.21.15.16:0/2619467575 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f2c0094a0 con 0x7f0f3c06d350 2023-10-16T13:46:16.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.682+0000 7f0f3bfff700 1 --2- 172.21.15.16:0/2619467575 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f3c15a870 0x7f0f3c1669b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:46:16.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.683+0000 7f0f3b7fe700 1 --2- 172.21.15.16:0/2619467575 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f3c06d350 0x7f0f3c14de10 secure :-1 s=READY pgs=2416 cs=0 l=1 rev1=1 crypto rx=0x7f0f2c000c00 tx=0x7f0f2c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:16.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.683+0000 7f0f33fff700 1 -- 172.21.15.16:0/2619467575 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f2c0036a0 con 0x7f0f3c06d350 2023-10-16T13:46:16.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.683+0000 7f0f33fff700 1 -- 172.21.15.16:0/2619467575 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f2c01e070 con 0x7f0f3c06d350 2023-10-16T13:46:16.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.683+0000 7f0f42155700 1 -- 172.21.15.16:0/2619467575 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f3c1661d0 con 0x7f0f3c06d350 2023-10-16T13:46:16.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.683+0000 7f0f33fff700 1 -- 172.21.15.16:0/2619467575 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f2c003a20 con 0x7f0f3c06d350 2023-10-16T13:46:16.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.683+0000 7f0f42155700 1 -- 172.21.15.16:0/2619467575 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f3c1664b0 con 0x7f0f3c06d350 2023-10-16T13:46:16.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.684+0000 7f0f42155700 1 -- 172.21.15.16:0/2619467575 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f3c0c85e0 con 0x7f0f3c06d350 2023-10-16T13:46:16.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.684+0000 7f0f33fff700 1 -- 172.21.15.16:0/2619467575 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 138) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0f2c003b80 con 0x7f0f3c06d350 2023-10-16T13:46:16.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.685+0000 7f0f33fff700 1 --2- 172.21.15.16:0/2619467575 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f0f1c06b020 0x7f0f1c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:16.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.685+0000 7f0f33fff700 1 -- 172.21.15.16:0/2619467575 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6165+0+0 (secure 0 0 0) 0x7f0f2c08a890 con 0x7f0f3c06d350 2023-10-16T13:46:16.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.685+0000 7f0f3affd700 1 --2- 172.21.15.16:0/2619467575 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f0f1c06b020 0x7f0f1c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:16.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.685+0000 7f0f3affd700 1 --2- 172.21.15.16:0/2619467575 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f0f1c06b020 0x7f0f1c06d4e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c0cf360 tx=0x7f0f24009040 comp rx=0 tx=0).ready entity=mgr.11190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:16.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.689+0000 7f0f33fff700 1 -- 172.21.15.16:0/2619467575 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0f3c0c85e0 con 0x7f0f3c06d350 2023-10-16T13:46:16.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.896+0000 7f0f42155700 1 -- 172.21.15.16:0/2619467575 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0f3c0c85e0 con 0x7f0f3c06d350 2023-10-16T13:46:16.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.897+0000 7f0f33fff700 1 -- 172.21.15.16:0/2619467575 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v217) v1 ==== 74+0+14425 (secure 0 0 0) 0x7f0f2c020090 con 0x7f0f3c06d350 2023-10-16T13:46:16.899 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:46:16.899 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":217,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:46:14.520310+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":204,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":204,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":204,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":204,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":204,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":204,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6837/4233835071":"2023-10-17T13:46:11.227222+0000","172.21.15.16:6835/4233835071":"2023-10-17T13:46:11.227222+0000","172.21.15.16:0/1458874080":"2023-10-17T13:46:10.160023+0000","172.21.15.16:0/1497663382":"2023-10-17T13:46:10.160023+0000","172.21.15.16:6832/3220139196":"2023-10-17T13:46:10.160023+0000","172.21.15.16:0/1142997361":"2023-10-17T13:46:10.160023+0000","172.21.15.16:0/4155301956":"2023-10-17T13:46:10.160023+0000","172.21.15.16:0/2102874149":"2023-10-17T13:46:10.160023+0000","172.21.15.16:0/1863977242":"2023-10-17T13:46:10.160023+0000","172.21.15.16:0/196217497":"2023-10-17T13:46:10.160023+0000","172.21.15.16:0/3477190137":"2023-10-17T13:44:55.326924+0000","172.21.15.16:0/2374795404":"2023-10-17T13:46:10.160023+0000","172.21.15.16:0/1906189720":"2023-10-17T13:44:55.326924+0000","172.21.15.16:6833/1268439414":"2023-10-17T13:44:55.326924+0000","172.21.15.16:6832/1268439414":"2023-10-17T13:44:55.326924+0000","172.21.15.16:6833/3220139196":"2023-10-17T13:46:10.160023+0000","172.21.15.16:0/3697646183":"2023-10-17T13:44:55.326924+0000","172.21.15.16:0/4067982926":"2023-10-17T13:44:55.326924+0000","172.21.15.16:0/3408848304":"2023-10-17T13:44:55.326924+0000","172.21.15.16:0/2782518306":"2023-10-17T13:44:55.326924+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:46:16.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.898+0000 7f0f42155700 1 -- 172.21.15.16:0/2619467575 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f0f1c06b020 msgr2=0x7f0f1c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:16.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.899+0000 7f0f42155700 1 --2- 172.21.15.16:0/2619467575 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f0f1c06b020 0x7f0f1c06d4e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c0cf360 tx=0x7f0f24009040 comp rx=0 tx=0).stop 2023-10-16T13:46:16.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.899+0000 7f0f42155700 1 -- 172.21.15.16:0/2619467575 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f3c06d350 msgr2=0x7f0f3c14de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:16.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.899+0000 7f0f42155700 1 --2- 172.21.15.16:0/2619467575 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f3c06d350 0x7f0f3c14de10 secure :-1 s=READY pgs=2416 cs=0 l=1 rev1=1 crypto rx=0x7f0f2c000c00 tx=0x7f0f2c004750 comp rx=0 tx=0).stop 2023-10-16T13:46:16.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.899+0000 7f0f42155700 1 -- 172.21.15.16:0/2619467575 shutdown_connections 2023-10-16T13:46:16.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.899+0000 7f0f42155700 1 --2- 172.21.15.16:0/2619467575 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f3c15a870 0x7f0f3c1669b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:16.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.899+0000 7f0f42155700 1 --2- 172.21.15.16:0/2619467575 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f0f1c06b020 0x7f0f1c06d4e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:16.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.899+0000 7f0f42155700 1 --2- 172.21.15.16:0/2619467575 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f3c06d350 0x7f0f3c14de10 unknown :-1 s=CLOSED pgs=2416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:16.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.899+0000 7f0f42155700 1 --2- 172.21.15.16:0/2619467575 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f3c0cbed0 0x7f0f3c165310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:16.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.899+0000 7f0f42155700 1 -- 172.21.15.16:0/2619467575 >> 172.21.15.16:0/2619467575 conn(0x7f0f3c0cc5b0 msgr2=0x7f0f3c159b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:16.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.899+0000 7f0f42155700 1 -- 172.21.15.16:0/2619467575 shutdown_connections 2023-10-16T13:46:16.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:16.899+0000 7f0f42155700 1 -- 172.21.15.16:0/2619467575 wait complete. 2023-10-16T13:46:16.914 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T13:46:16.914 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T13:46:17.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.040+0000 7fdcdb71a700 1 Processor -- start 2023-10-16T13:46:17.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.040+0000 7fdcdb71a700 1 -- start start 2023-10-16T13:46:17.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.041+0000 7fdcdb71a700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fdcd406aef0 0x7fdcd40ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:17.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.041+0000 7fdcdb71a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdcd406da30 0x7fdcd40adf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:17.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.041+0000 7fdcdb71a700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcd41613c0 con 0x7fdcd4160fe0 2023-10-16T13:46:17.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.041+0000 7fdcdb71a700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcd40cf680 con 0x7fdcd406aef0 2023-10-16T13:46:17.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.041+0000 7fdcdb71a700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd40adb00 con 0x7fdcd406da30 2023-10-16T13:46:17.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.041+0000 7fdcd8cb5700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fdcd406aef0 0x7fdcd40ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:17.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.041+0000 7fdcd9cb7700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdcd406da30 0x7fdcd40adf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:17.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.041+0000 7fdcd94b6700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7fdcd4160fe0 0x7fdcd406a7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:49364/0 (socket says 172.21.15.16:49364) 2023-10-16T13:46:17.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.041+0000 7fdcd9cb7700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdcd406da30 0x7fdcd40adf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:41170/0 (socket says 172.21.15.16:41170) 2023-10-16T13:46:17.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.041+0000 7fdcd94b6700 1 -- 172.21.15.16:0/2887669404 learned_addr learned my addr 172.21.15.16:0/2887669404 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:46:17.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.041+0000 7fdcd8cb5700 1 --2- 172.21.15.16:0/2887669404 >> v2:172.21.15.49:3300/0 conn(0x7fdcd406aef0 0x7fdcd40ac8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:51972/0 (socket says 172.21.15.16:51972) 2023-10-16T13:46:17.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.041+0000 7fdcd9cb7700 1 -- 172.21.15.16:0/2887669404 >> v2:172.21.15.49:3300/0 conn(0x7fdcd406aef0 msgr2=0x7fdcd40ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:17.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.041+0000 7fdcd9cb7700 1 --2- 172.21.15.16:0/2887669404 >> v2:172.21.15.49:3300/0 conn(0x7fdcd406aef0 0x7fdcd40ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:17.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.041+0000 7fdcd9cb7700 1 -- 172.21.15.16:0/2887669404 >> v1:172.21.15.49:6789/0 conn(0x7fdcd4160fe0 legacy=0x7fdcd406a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:17.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.041+0000 7fdcd9cb7700 1 -- 172.21.15.16:0/2887669404 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcd40acef0 con 0x7fdcd406da30 2023-10-16T13:46:17.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.041+0000 7fdcd8cb5700 1 --2- 172.21.15.16:0/2887669404 >> v2:172.21.15.49:3300/0 conn(0x7fdcd406aef0 0x7fdcd40ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:46:17.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.042+0000 7fdcd9cb7700 1 --2- 172.21.15.16:0/2887669404 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdcd406da30 0x7fdcd40adf90 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fdccc009d90 tx=0x7fdccc00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=de2709b9b1467221 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:17.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.042+0000 7fdcc7fff700 1 -- 172.21.15.16:0/2887669404 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdccc00e070 con 0x7fdcd406da30 2023-10-16T13:46:17.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.042+0000 7fdcc7fff700 1 -- 172.21.15.16:0/2887669404 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdccc00bb20 con 0x7fdcd406da30 2023-10-16T13:46:17.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.042+0000 7fdcc7fff700 1 -- 172.21.15.16:0/2887669404 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdccc012430 con 0x7fdcd406da30 2023-10-16T13:46:17.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.042+0000 7fdcdb71a700 1 -- 172.21.15.16:0/2887669404 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdcd406da30 msgr2=0x7fdcd40adf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:17.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.042+0000 7fdcdb71a700 1 --2- 172.21.15.16:0/2887669404 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdcd406da30 0x7fdcd40adf90 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fdccc009d90 tx=0x7fdccc00b330 comp rx=0 tx=0).stop 2023-10-16T13:46:17.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.042+0000 7fdcdb71a700 1 -- 172.21.15.16:0/2887669404 shutdown_connections 2023-10-16T13:46:17.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.042+0000 7fdcdb71a700 1 --2- 172.21.15.16:0/2887669404 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdcd406da30 0x7fdcd40adf90 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:17.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.042+0000 7fdcdb71a700 1 --2- 172.21.15.16:0/2887669404 >> v2:172.21.15.49:3300/0 conn(0x7fdcd406aef0 0x7fdcd40ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:17.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.042+0000 7fdcdb71a700 1 -- 172.21.15.16:0/2887669404 >> 172.21.15.16:0/2887669404 conn(0x7fdcd40b1ed0 msgr2=0x7fdcd40b22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:17.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.042+0000 7fdcdb71a700 1 -- 172.21.15.16:0/2887669404 shutdown_connections 2023-10-16T13:46:17.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.042+0000 7fdcdb71a700 1 -- 172.21.15.16:0/2887669404 wait complete. 2023-10-16T13:46:17.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.044+0000 7fdcdb71a700 1 Processor -- start 2023-10-16T13:46:17.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.044+0000 7fdcdb71a700 1 -- start start 2023-10-16T13:46:17.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.044+0000 7fdcdb71a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdcd406aef0 0x7fdcd40c6790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:17.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.044+0000 7fdcdb71a700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdcd406da30 0x7fdcd4153950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:17.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.044+0000 7fdcdb71a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdcd4160fe0 0x7fdcd40c3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:17.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.044+0000 7fdcdb71a700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd40b40f0 con 0x7fdcd4160fe0 2023-10-16T13:46:17.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.044+0000 7fdcdb71a700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd40b4f50 con 0x7fdcd406da30 2023-10-16T13:46:17.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.044+0000 7fdcdb71a700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd4153e90 con 0x7fdcd406aef0 2023-10-16T13:46:17.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.045+0000 7fdcd9cb7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdcd4160fe0 0x7fdcd40c3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:17.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.045+0000 7fdcd9cb7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdcd4160fe0 0x7fdcd40c3bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52918/0 (socket says 172.21.15.16:52918) 2023-10-16T13:46:17.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.045+0000 7fdcd9cb7700 1 -- 172.21.15.16:0/2293683781 learned_addr learned my addr 172.21.15.16:0/2293683781 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:17.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.045+0000 7fdcd8cb5700 1 --2- 172.21.15.16:0/2293683781 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdcd406da30 0x7fdcd4153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:17.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.045+0000 7fdcd94b6700 1 --2- 172.21.15.16:0/2293683781 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdcd406aef0 0x7fdcd40c6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:17.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.045+0000 7fdcd9cb7700 1 -- 172.21.15.16:0/2293683781 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdcd406aef0 msgr2=0x7fdcd40c6790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:17.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.045+0000 7fdcd9cb7700 1 --2- 172.21.15.16:0/2293683781 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdcd406aef0 0x7fdcd40c6790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:17.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.045+0000 7fdcd9cb7700 1 -- 172.21.15.16:0/2293683781 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdcd406da30 msgr2=0x7fdcd4153950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:17.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.045+0000 7fdcd9cb7700 1 --2- 172.21.15.16:0/2293683781 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdcd406da30 0x7fdcd4153950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:17.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.045+0000 7fdcd9cb7700 1 -- 172.21.15.16:0/2293683781 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcd00094a0 con 0x7fdcd4160fe0 2023-10-16T13:46:17.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.045+0000 7fdcd9cb7700 1 --2- 172.21.15.16:0/2293683781 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdcd4160fe0 0x7fdcd40c3bc0 secure :-1 s=READY pgs=2417 cs=0 l=1 rev1=1 crypto rx=0x7fdcd40adc40 tx=0x7fdccc004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:17.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.045+0000 7fdcc5ffb700 1 -- 172.21.15.16:0/2293683781 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdccc012430 con 0x7fdcd4160fe0 2023-10-16T13:46:17.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.045+0000 7fdcc5ffb700 1 -- 172.21.15.16:0/2293683781 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdccc020070 con 0x7fdcd4160fe0 2023-10-16T13:46:17.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.045+0000 7fdcdb71a700 1 -- 172.21.15.16:0/2293683781 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdccc009a70 con 0x7fdcd4160fe0 2023-10-16T13:46:17.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.045+0000 7fdcc5ffb700 1 -- 172.21.15.16:0/2293683781 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdccc004280 con 0x7fdcd4160fe0 2023-10-16T13:46:17.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.046+0000 7fdcdb71a700 1 -- 172.21.15.16:0/2293683781 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcd40b4620 con 0x7fdcd4160fe0 2023-10-16T13:46:17.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.046+0000 7fdcdb71a700 1 -- 172.21.15.16:0/2293683781 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcd404ebb0 con 0x7fdcd4160fe0 2023-10-16T13:46:17.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.047+0000 7fdcc5ffb700 1 -- 172.21.15.16:0/2293683781 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 138) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fdccc0043e0 con 0x7fdcd4160fe0 2023-10-16T13:46:17.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.048+0000 7fdcc5ffb700 1 --2- 172.21.15.16:0/2293683781 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fdcb806b020 0x7fdcb806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:17.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.048+0000 7fdcc5ffb700 1 -- 172.21.15.16:0/2293683781 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6165+0+0 (secure 0 0 0) 0x7fdccc022070 con 0x7fdcd4160fe0 2023-10-16T13:46:17.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.048+0000 7fdcd94b6700 1 --2- 172.21.15.16:0/2293683781 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fdcb806b020 0x7fdcb806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:17.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.048+0000 7fdcd94b6700 1 --2- 172.21.15.16:0/2293683781 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fdcb806b020 0x7fdcb806d4e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fdcc8005960 tx=0x7fdcc8009040 comp rx=0 tx=0).ready entity=mgr.11190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:17.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.052+0000 7fdcc5ffb700 1 -- 172.21.15.16:0/2293683781 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdccc056530 con 0x7fdcd4160fe0 2023-10-16T13:46:17.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.254+0000 7fdcdb71a700 1 -- 172.21.15.16:0/2293683781 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fdcd40b5090 con 0x7fdcd4160fe0 2023-10-16T13:46:17.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.254+0000 7fdcc5ffb700 1 -- 172.21.15.16:0/2293683781 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v217) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdccc059b50 con 0x7fdcd4160fe0 2023-10-16T13:46:17.257 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T13:46:17.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.256+0000 7fdcdb71a700 1 -- 172.21.15.16:0/2293683781 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fdcb806b020 msgr2=0x7fdcb806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:17.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.256+0000 7fdcdb71a700 1 --2- 172.21.15.16:0/2293683781 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fdcb806b020 0x7fdcb806d4e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fdcc8005960 tx=0x7fdcc8009040 comp rx=0 tx=0).stop 2023-10-16T13:46:17.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.256+0000 7fdcdb71a700 1 -- 172.21.15.16:0/2293683781 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdcd4160fe0 msgr2=0x7fdcd40c3bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:17.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.256+0000 7fdcdb71a700 1 --2- 172.21.15.16:0/2293683781 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdcd4160fe0 0x7fdcd40c3bc0 secure :-1 s=READY pgs=2417 cs=0 l=1 rev1=1 crypto rx=0x7fdcd40adc40 tx=0x7fdccc004ad0 comp rx=0 tx=0).stop 2023-10-16T13:46:17.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.256+0000 7fdcdb71a700 1 -- 172.21.15.16:0/2293683781 shutdown_connections 2023-10-16T13:46:17.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.256+0000 7fdcdb71a700 1 --2- 172.21.15.16:0/2293683781 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fdcb806b020 0x7fdcb806d4e0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:17.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.256+0000 7fdcdb71a700 1 --2- 172.21.15.16:0/2293683781 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdcd4160fe0 0x7fdcd40c3bc0 unknown :-1 s=CLOSED pgs=2417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:17.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.256+0000 7fdcdb71a700 1 --2- 172.21.15.16:0/2293683781 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdcd406aef0 0x7fdcd40c6790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:17.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.256+0000 7fdcdb71a700 1 --2- 172.21.15.16:0/2293683781 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdcd406da30 0x7fdcd4153950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:17.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.256+0000 7fdcdb71a700 1 -- 172.21.15.16:0/2293683781 >> 172.21.15.16:0/2293683781 conn(0x7fdcd40b1ed0 msgr2=0x7fdcd40ade90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:17.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.256+0000 7fdcdb71a700 1 -- 172.21.15.16:0/2293683781 shutdown_connections 2023-10-16T13:46:17.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.257+0000 7fdcdb71a700 1 -- 172.21.15.16:0/2293683781 wait complete. 2023-10-16T13:46:17.271 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs' 2023-10-16T13:46:17.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.395+0000 7f21e402e700 1 Processor -- start 2023-10-16T13:46:17.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.395+0000 7f21e402e700 1 -- start start 2023-10-16T13:46:17.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.396+0000 7f21e402e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f21dc15b860 0x7f21dc06a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:17.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.396+0000 7f21e402e700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f21dc0b2be0 0x7f21dc06ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:17.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.396+0000 7f21e402e700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21dc0d11c0 con 0x7f21dc0cd770 2023-10-16T13:46:17.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.396+0000 7f21e402e700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21dc0d03d0 con 0x7f21dc0b2be0 2023-10-16T13:46:17.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.396+0000 7f21e402e700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21dc0cd360 con 0x7f21dc15b860 2023-10-16T13:46:17.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.396+0000 7f21e1dca700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f21dc15b860 0x7f21dc06a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:17.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.396+0000 7f21e15c9700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f21dc0b2be0 0x7f21dc06ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:17.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.396+0000 7f21e25cb700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7f21dc0cd770 0x7f21dc0719e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:49382/0 (socket says 172.21.15.16:49382) 2023-10-16T13:46:17.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.396+0000 7f21e25cb700 1 -- 172.21.15.16:0/2897376981 learned_addr learned my addr 172.21.15.16:0/2897376981 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:46:17.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.397+0000 7f21e0dc8700 1 -- 172.21.15.16:0/2897376981 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3850341287 0 0) 0x7f21dc0d11c0 con 0x7f21dc0cd770 2023-10-16T13:46:17.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.397+0000 7f21e0dc8700 1 -- 172.21.15.16:0/2897376981 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21c80035b0 con 0x7f21dc0cd770 2023-10-16T13:46:17.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.397+0000 7f21e15c9700 1 -- 172.21.15.16:0/2897376981 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f21dc15b860 msgr2=0x7f21dc06a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:17.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.397+0000 7f21e15c9700 1 --2- 172.21.15.16:0/2897376981 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f21dc15b860 0x7f21dc06a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:17.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.397+0000 7f21e15c9700 1 -- 172.21.15.16:0/2897376981 >> v1:172.21.15.49:6789/0 conn(0x7f21dc0cd770 legacy=0x7f21dc0719e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:17.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.397+0000 7f21e15c9700 1 -- 172.21.15.16:0/2897376981 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21dc0d05b0 con 0x7f21dc0b2be0 2023-10-16T13:46:17.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.397+0000 7f21e1dca700 1 --2- 172.21.15.16:0/2897376981 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f21dc15b860 0x7f21dc06a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:46:17.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.397+0000 7f21e15c9700 1 --2- 172.21.15.16:0/2897376981 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f21dc0b2be0 0x7f21dc06ad20 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f21d8009840 tx=0x7f21d8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db22fb153e9c05fc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:17.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.397+0000 7f21e0dc8700 1 -- 172.21.15.16:0/2897376981 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d8011830 con 0x7f21dc0b2be0 2023-10-16T13:46:17.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.397+0000 7f21e0dc8700 1 -- 172.21.15.16:0/2897376981 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21d8011990 con 0x7f21dc0b2be0 2023-10-16T13:46:17.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.397+0000 7f21e0dc8700 1 -- 172.21.15.16:0/2897376981 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d8011d20 con 0x7f21dc0b2be0 2023-10-16T13:46:17.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.398+0000 7f21e402e700 1 -- 172.21.15.16:0/2897376981 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f21dc0b2be0 msgr2=0x7f21dc06ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:17.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.398+0000 7f21e402e700 1 --2- 172.21.15.16:0/2897376981 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f21dc0b2be0 0x7f21dc06ad20 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f21d8009840 tx=0x7f21d8009b50 comp rx=0 tx=0).stop 2023-10-16T13:46:17.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.398+0000 7f21e402e700 1 -- 172.21.15.16:0/2897376981 shutdown_connections 2023-10-16T13:46:17.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.398+0000 7f21e402e700 1 --2- 172.21.15.16:0/2897376981 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f21dc15b860 0x7f21dc06a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:17.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.398+0000 7f21e402e700 1 --2- 172.21.15.16:0/2897376981 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f21dc0b2be0 0x7f21dc06ad20 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:17.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.398+0000 7f21e402e700 1 -- 172.21.15.16:0/2897376981 >> 172.21.15.16:0/2897376981 conn(0x7f21dc0b1fd0 msgr2=0x7f21dc0b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:17.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.398+0000 7f21e402e700 1 -- 172.21.15.16:0/2897376981 shutdown_connections 2023-10-16T13:46:17.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.398+0000 7f21e402e700 1 -- 172.21.15.16:0/2897376981 wait complete. 2023-10-16T13:46:17.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.399+0000 7f21e402e700 1 Processor -- start 2023-10-16T13:46:17.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.399+0000 7f21e402e700 1 -- start start 2023-10-16T13:46:17.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.400+0000 7f21e402e700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f21dc0b2be0 0x7f21dc1585a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:17.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.400+0000 7f21e402e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f21dc0cd770 0x7f21dc1e44b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:17.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.400+0000 7f21e402e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21dc15b860 0x7f21dc15cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:17.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.400+0000 7f21e402e700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21dc0ca910 con 0x7f21dc15b860 2023-10-16T13:46:17.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.400+0000 7f21e402e700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21dc158ae0 con 0x7f21dc0b2be0 2023-10-16T13:46:17.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.400+0000 7f21e402e700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21dc158fa0 con 0x7f21dc0cd770 2023-10-16T13:46:17.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.400+0000 7f21e25cb700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21dc15b860 0x7f21dc15cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:17.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.400+0000 7f21e1dca700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f21dc0b2be0 0x7f21dc1585a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:17.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.400+0000 7f21e25cb700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21dc15b860 0x7f21dc15cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52932/0 (socket says 172.21.15.16:52932) 2023-10-16T13:46:17.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.400+0000 7f21e25cb700 1 -- 172.21.15.16:0/3701133381 learned_addr learned my addr 172.21.15.16:0/3701133381 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:17.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.400+0000 7f21e1dca700 1 --2- 172.21.15.16:0/3701133381 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f21dc0b2be0 0x7f21dc1585a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:51990/0 (socket says 172.21.15.16:51990) 2023-10-16T13:46:17.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.400+0000 7f21e15c9700 1 --2- 172.21.15.16:0/3701133381 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f21dc0cd770 0x7f21dc1e44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:17.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.400+0000 7f21e25cb700 1 -- 172.21.15.16:0/3701133381 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f21dc0cd770 msgr2=0x7f21dc1e44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:17.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.400+0000 7f21e25cb700 1 --2- 172.21.15.16:0/3701133381 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f21dc0cd770 0x7f21dc1e44b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:17.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.400+0000 7f21e25cb700 1 -- 172.21.15.16:0/3701133381 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f21dc0b2be0 msgr2=0x7f21dc1585a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:17.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.400+0000 7f21e25cb700 1 --2- 172.21.15.16:0/3701133381 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f21dc0b2be0 0x7f21dc1585a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:17.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.400+0000 7f21e25cb700 1 -- 172.21.15.16:0/3701133381 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21cc0094a0 con 0x7f21dc15b860 2023-10-16T13:46:17.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.400+0000 7f21e25cb700 1 --2- 172.21.15.16:0/3701133381 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21dc15b860 0x7f21dc15cf40 secure :-1 s=READY pgs=2418 cs=0 l=1 rev1=1 crypto rx=0x7f21d400caa0 tx=0x7f21d400cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:17.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.401+0000 7f21d27fc700 1 -- 172.21.15.16:0/3701133381 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d400a9e0 con 0x7f21dc15b860 2023-10-16T13:46:17.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.401+0000 7f21d27fc700 1 -- 172.21.15.16:0/3701133381 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21d400ab40 con 0x7f21dc15b860 2023-10-16T13:46:17.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.401+0000 7f21e402e700 1 -- 172.21.15.16:0/3701133381 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21d80094a0 con 0x7f21dc15b860 2023-10-16T13:46:17.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.401+0000 7f21d27fc700 1 -- 172.21.15.16:0/3701133381 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d4012400 con 0x7f21dc15b860 2023-10-16T13:46:17.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.401+0000 7f21e402e700 1 -- 172.21.15.16:0/3701133381 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21dc0ca350 con 0x7f21dc15b860 2023-10-16T13:46:17.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.402+0000 7f21e402e700 1 -- 172.21.15.16:0/3701133381 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21dc0d0e30 con 0x7f21dc15b860 2023-10-16T13:46:17.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.402+0000 7f21d27fc700 1 -- 172.21.15.16:0/3701133381 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 138) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f21d401b3b0 con 0x7f21dc15b860 2023-10-16T13:46:17.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.402+0000 7f21d27fc700 1 --2- 172.21.15.16:0/3701133381 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f21c006b020 0x7f21c006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:17.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.403+0000 7f21d27fc700 1 -- 172.21.15.16:0/3701133381 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6165+0+0 (secure 0 0 0) 0x7f21d4089570 con 0x7f21dc15b860 2023-10-16T13:46:17.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.403+0000 7f21e1dca700 1 --2- 172.21.15.16:0/3701133381 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f21c006b020 0x7f21c006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:17.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.403+0000 7f21e1dca700 1 --2- 172.21.15.16:0/3701133381 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f21c006b020 0x7f21c006d4e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f21cc009d70 tx=0x7f21cc009210 comp rx=0 tx=0).ready entity=mgr.11190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:17.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.407+0000 7f21d27fc700 1 -- 172.21.15.16:0/3701133381 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f21dc0d0e30 con 0x7f21dc15b860 2023-10-16T13:46:17.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:17.697+0000 7f21e402e700 1 -- 172.21.15.16:0/3701133381 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs"]} v 0) v1 -- 0x7f21dc0d0e30 con 0x7f21dc15b860 2023-10-16T13:46:18.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.532+0000 7f21d27fc700 1 -- 172.21.15.16:0/3701133381 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs"]}]=0 v3122) v1 ==== 189+0+0 (secure 0 0 0) 0x7f21dc0d0e30 con 0x7f21dc15b860 2023-10-16T13:46:18.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.535+0000 7f21e402e700 1 -- 172.21.15.16:0/3701133381 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f21c006b020 msgr2=0x7f21c006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:18.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.535+0000 7f21e402e700 1 --2- 172.21.15.16:0/3701133381 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f21c006b020 0x7f21c006d4e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f21cc009d70 tx=0x7f21cc009210 comp rx=0 tx=0).stop 2023-10-16T13:46:18.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.535+0000 7f21e402e700 1 -- 172.21.15.16:0/3701133381 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21dc15b860 msgr2=0x7f21dc15cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:18.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.535+0000 7f21e402e700 1 --2- 172.21.15.16:0/3701133381 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21dc15b860 0x7f21dc15cf40 secure :-1 s=READY pgs=2418 cs=0 l=1 rev1=1 crypto rx=0x7f21d400caa0 tx=0x7f21d400cdb0 comp rx=0 tx=0).stop 2023-10-16T13:46:18.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.535+0000 7f21e402e700 1 -- 172.21.15.16:0/3701133381 shutdown_connections 2023-10-16T13:46:18.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.535+0000 7f21e402e700 1 --2- 172.21.15.16:0/3701133381 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f21c006b020 0x7f21c006d4e0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:18.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.535+0000 7f21e402e700 1 --2- 172.21.15.16:0/3701133381 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21dc15b860 0x7f21dc15cf40 unknown :-1 s=CLOSED pgs=2418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:18.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.535+0000 7f21e402e700 1 --2- 172.21.15.16:0/3701133381 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f21dc0b2be0 0x7f21dc1585a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:18.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.535+0000 7f21e402e700 1 --2- 172.21.15.16:0/3701133381 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f21dc0cd770 0x7f21dc1e44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:18.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.535+0000 7f21e402e700 1 -- 172.21.15.16:0/3701133381 >> 172.21.15.16:0/3701133381 conn(0x7f21dc0b1fd0 msgr2=0x7f21dc1597f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:18.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.535+0000 7f21e402e700 1 -- 172.21.15.16:0/3701133381 shutdown_connections 2023-10-16T13:46:18.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.535+0000 7f21e402e700 1 -- 172.21.15.16:0/3701133381 wait complete. 2023-10-16T13:46:18.552 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-16T13:46:18.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.684+0000 7ffaa0a5b700 1 Processor -- start 2023-10-16T13:46:18.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.684+0000 7ffaa0a5b700 1 -- start start 2023-10-16T13:46:18.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.685+0000 7ffaa0a5b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ffa9c0ce6a0 0x7ffa9c06c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:18.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.685+0000 7ffaa0a5b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ffa9c0d2110 0x7ffa9c06c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:18.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.685+0000 7ffaa0a5b700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa9c0b20f0 con 0x7ffa9c0b2620 2023-10-16T13:46:18.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.685+0000 7ffaa0a5b700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa9c0af1f0 con 0x7ffa9c0d2110 2023-10-16T13:46:18.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.685+0000 7ffaa0a5b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa9c0d03d0 con 0x7ffa9c0ce6a0 2023-10-16T13:46:18.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.685+0000 7ffa99d9b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ffa9c0d2110 0x7ffa9c06c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:18.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.685+0000 7ffa99d9b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ffa9c0d2110 0x7ffa9c06c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52938/0 (socket says 172.21.15.16:52938) 2023-10-16T13:46:18.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.685+0000 7ffa99d9b700 1 -- 172.21.15.16:0/2668003231 learned_addr learned my addr 172.21.15.16:0/2668003231 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:18.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.686+0000 7ffa99d9b700 1 -- 172.21.15.16:0/2668003231 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ffa9c0ce6a0 msgr2=0x7ffa9c06c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:18.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.686+0000 7ffa99d9b700 1 --2- 172.21.15.16:0/2668003231 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ffa9c0ce6a0 0x7ffa9c06c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:18.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.686+0000 7ffa99d9b700 1 -- 172.21.15.16:0/2668003231 >> v1:172.21.15.49:6789/0 conn(0x7ffa9c0b2620 legacy=0x7ffa9c0b0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:18.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.686+0000 7ffa99d9b700 1 -- 172.21.15.16:0/2668003231 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa9c0b2340 con 0x7ffa9c0d2110 2023-10-16T13:46:18.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.686+0000 7ffa99d9b700 1 --2- 172.21.15.16:0/2668003231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa9c0d2110 0x7ffa9c06c840 secure :-1 s=READY pgs=2419 cs=0 l=1 rev1=1 crypto rx=0x7ffa8c0098f0 tx=0x7ffa8c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9ca35760dd2980e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:18.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.686+0000 7ffa9959a700 1 -- 172.21.15.16:0/2668003231 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa8c004230 con 0x7ffa9c0d2110 2023-10-16T13:46:18.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.686+0000 7ffa9959a700 1 -- 172.21.15.16:0/2668003231 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa8c004390 con 0x7ffa9c0d2110 2023-10-16T13:46:18.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.686+0000 7ffa9959a700 1 -- 172.21.15.16:0/2668003231 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa8c0036a0 con 0x7ffa9c0d2110 2023-10-16T13:46:18.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.687+0000 7ffaa0a5b700 1 -- 172.21.15.16:0/2668003231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa9c0d2110 msgr2=0x7ffa9c06c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:18.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.687+0000 7ffaa0a5b700 1 --2- 172.21.15.16:0/2668003231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa9c0d2110 0x7ffa9c06c840 secure :-1 s=READY pgs=2419 cs=0 l=1 rev1=1 crypto rx=0x7ffa8c0098f0 tx=0x7ffa8c009c00 comp rx=0 tx=0).stop 2023-10-16T13:46:18.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.687+0000 7ffaa0a5b700 1 -- 172.21.15.16:0/2668003231 shutdown_connections 2023-10-16T13:46:18.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.687+0000 7ffaa0a5b700 1 --2- 172.21.15.16:0/2668003231 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ffa9c0ce6a0 0x7ffa9c06c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:18.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.687+0000 7ffaa0a5b700 1 --2- 172.21.15.16:0/2668003231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa9c0d2110 0x7ffa9c06c840 unknown :-1 s=CLOSED pgs=2419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:18.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.687+0000 7ffaa0a5b700 1 -- 172.21.15.16:0/2668003231 >> 172.21.15.16:0/2668003231 conn(0x7ffa9c0d15f0 msgr2=0x7ffa9c0d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:18.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.687+0000 7ffaa0a5b700 1 -- 172.21.15.16:0/2668003231 shutdown_connections 2023-10-16T13:46:18.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.687+0000 7ffaa0a5b700 1 -- 172.21.15.16:0/2668003231 wait complete. 2023-10-16T13:46:18.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.688+0000 7ffaa0a5b700 1 Processor -- start 2023-10-16T13:46:18.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.688+0000 7ffaa0a5b700 1 -- start start 2023-10-16T13:46:18.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.689+0000 7ffaa0a5b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ffa9c0b2620 0x7ffa9c0b4170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:18.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.689+0000 7ffaa0a5b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa9c0ce6a0 0x7ffa9c1f06d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:18.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.689+0000 7ffaa0a5b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ffa9c0d2110 0x7ffa9c1f3be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:18.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.689+0000 7ffaa0a5b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa9c0b6df0 con 0x7ffa9c0ce6a0 2023-10-16T13:46:18.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.689+0000 7ffaa0a5b700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa9c1f0c10 con 0x7ffa9c0d2110 2023-10-16T13:46:18.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.689+0000 7ffaa0a5b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa9c1f4120 con 0x7ffa9c0b2620 2023-10-16T13:46:18.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.689+0000 7ffa99d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa9c0ce6a0 0x7ffa9c1f06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:18.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.689+0000 7ffa99d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa9c0ce6a0 0x7ffa9c1f06d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52944/0 (socket says 172.21.15.16:52944) 2023-10-16T13:46:18.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.689+0000 7ffa99d9b700 1 -- 172.21.15.16:0/1548501254 learned_addr learned my addr 172.21.15.16:0/1548501254 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:18.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.689+0000 7ffa99d9b700 1 -- 172.21.15.16:0/1548501254 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ffa9c0b2620 msgr2=0x7ffa9c0b4170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:18.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.689+0000 7ffa99d9b700 1 --2- 172.21.15.16:0/1548501254 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ffa9c0b2620 0x7ffa9c0b4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:18.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.689+0000 7ffa9ad9d700 1 --2- 172.21.15.16:0/1548501254 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ffa9c0d2110 0x7ffa9c1f3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:18.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.689+0000 7ffa99d9b700 1 -- 172.21.15.16:0/1548501254 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ffa9c0d2110 msgr2=0x7ffa9c1f3be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:18.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.689+0000 7ffa99d9b700 1 --2- 172.21.15.16:0/1548501254 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ffa9c0d2110 0x7ffa9c1f3be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:18.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.689+0000 7ffa99d9b700 1 -- 172.21.15.16:0/1548501254 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa8c009550 con 0x7ffa9c0ce6a0 2023-10-16T13:46:18.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.690+0000 7ffa99d9b700 1 --2- 172.21.15.16:0/1548501254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa9c0ce6a0 0x7ffa9c1f06d0 secure :-1 s=READY pgs=2420 cs=0 l=1 rev1=1 crypto rx=0x7ffa8c009130 tx=0x7ffa8c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:18.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.690+0000 7ffa8affd700 1 -- 172.21.15.16:0/1548501254 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa8c00b860 con 0x7ffa9c0ce6a0 2023-10-16T13:46:18.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.690+0000 7ffa8affd700 1 -- 172.21.15.16:0/1548501254 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa8c00b9c0 con 0x7ffa9c0ce6a0 2023-10-16T13:46:18.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.690+0000 7ffaa0a5b700 1 -- 172.21.15.16:0/1548501254 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa9c0b4750 con 0x7ffa9c0ce6a0 2023-10-16T13:46:18.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.690+0000 7ffa8affd700 1 -- 172.21.15.16:0/1548501254 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa8c00bd70 con 0x7ffa9c0ce6a0 2023-10-16T13:46:18.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.690+0000 7ffaa0a5b700 1 -- 172.21.15.16:0/1548501254 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa9c0b4d10 con 0x7ffa9c0ce6a0 2023-10-16T13:46:18.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.691+0000 7ffaa0a5b700 1 -- 172.21.15.16:0/1548501254 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa9c04ebb0 con 0x7ffa9c0ce6a0 2023-10-16T13:46:18.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.691+0000 7ffa8affd700 1 -- 172.21.15.16:0/1548501254 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 138) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ffa8c01b440 con 0x7ffa9c0ce6a0 2023-10-16T13:46:18.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.692+0000 7ffa8affd700 1 --2- 172.21.15.16:0/1548501254 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7ffa7c06b020 0x7ffa7c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:18.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.692+0000 7ffa8affd700 1 -- 172.21.15.16:0/1548501254 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6165+0+0 (secure 0 0 0) 0x7ffa8c08b380 con 0x7ffa9c0ce6a0 2023-10-16T13:46:18.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.692+0000 7ffa9a59c700 1 --2- 172.21.15.16:0/1548501254 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7ffa7c06b020 0x7ffa7c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:18.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.692+0000 7ffa9a59c700 1 --2- 172.21.15.16:0/1548501254 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7ffa7c06b020 0x7ffa7c06d4e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ffa900097b0 tx=0x7ffa90006d20 comp rx=0 tx=0).ready entity=mgr.11190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:18.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.697+0000 7ffa8affd700 1 -- 172.21.15.16:0/1548501254 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ffa8c056140 con 0x7ffa9c0ce6a0 2023-10-16T13:46:18.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.901+0000 7ffaa0a5b700 1 -- 172.21.15.16:0/1548501254 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7ffa9c0afce0 con 0x7ffa9c0ce6a0 2023-10-16T13:46:18.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.904+0000 7ffa8affd700 1 -- 172.21.15.16:0/1548501254 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v76)=0 v76) v1 ==== 122+0+0 (secure 0 0 0) 0x7ffa8c059760 con 0x7ffa9c0ce6a0 2023-10-16T13:46:18.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.906+0000 7ffaa0a5b700 1 -- 172.21.15.16:0/1548501254 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7ffa7c06b020 msgr2=0x7ffa7c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:18.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.906+0000 7ffaa0a5b700 1 --2- 172.21.15.16:0/1548501254 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7ffa7c06b020 0x7ffa7c06d4e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ffa900097b0 tx=0x7ffa90006d20 comp rx=0 tx=0).stop 2023-10-16T13:46:18.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.906+0000 7ffaa0a5b700 1 -- 172.21.15.16:0/1548501254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa9c0ce6a0 msgr2=0x7ffa9c1f06d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:18.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.906+0000 7ffaa0a5b700 1 --2- 172.21.15.16:0/1548501254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa9c0ce6a0 0x7ffa9c1f06d0 secure :-1 s=READY pgs=2420 cs=0 l=1 rev1=1 crypto rx=0x7ffa8c009130 tx=0x7ffa8c004750 comp rx=0 tx=0).stop 2023-10-16T13:46:18.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.906+0000 7ffaa0a5b700 1 -- 172.21.15.16:0/1548501254 shutdown_connections 2023-10-16T13:46:18.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.906+0000 7ffaa0a5b700 1 --2- 172.21.15.16:0/1548501254 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ffa9c0d2110 0x7ffa9c1f3be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:18.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.906+0000 7ffaa0a5b700 1 --2- 172.21.15.16:0/1548501254 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ffa9c0b2620 0x7ffa9c0b4170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:18.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.906+0000 7ffaa0a5b700 1 --2- 172.21.15.16:0/1548501254 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7ffa7c06b020 0x7ffa7c06d4e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:18.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.906+0000 7ffaa0a5b700 1 --2- 172.21.15.16:0/1548501254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa9c0ce6a0 0x7ffa9c1f06d0 unknown :-1 s=CLOSED pgs=2420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:18.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.906+0000 7ffaa0a5b700 1 -- 172.21.15.16:0/1548501254 >> 172.21.15.16:0/1548501254 conn(0x7ffa9c0d15f0 msgr2=0x7ffa9c0d19d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:18.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.906+0000 7ffaa0a5b700 1 -- 172.21.15.16:0/1548501254 shutdown_connections 2023-10-16T13:46:18.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:18.906+0000 7ffaa0a5b700 1 -- 172.21.15.16:0/1548501254 wait complete. 2023-10-16T13:46:18.921 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T13:46:18.922 DEBUG:teuthology.orchestra.run.smithi016:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T13:46:18.961 DEBUG:teuthology.orchestra.run.smithi049:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T13:46:19.012 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T13:46:19.012 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T13:46:19.013 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:46:19.031 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16287364k,nr_inodes=4071841,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20251 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3261172k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T13:46:19.031 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T13:46:19.031 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:46:19.051 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284204k,nr_inodes=4071051,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=753 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260540k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T13:46:19.052 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T13:46:19.052 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:46:19.088 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16287364k,nr_inodes=4071841,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20251 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3261172k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T13:46:19.088 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T13:46:19.088 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:46:19.110 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284204k,nr_inodes=4071051,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=753 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260540k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T13:46:19.111 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-16T13:46:19.111 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:46:19.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.251+0000 7fe1d2ac9700 1 Processor -- start 2023-10-16T13:46:19.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.251+0000 7fe1d2ac9700 1 -- start start 2023-10-16T13:46:19.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.251+0000 7fe1d2ac9700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fe1cc15a5c0 0x7fe1cc0cef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:19.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.251+0000 7fe1d2ac9700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe1cc0d0ae0 0x7fe1cc0cc1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:19.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.251+0000 7fe1d2ac9700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1cc072a50 con 0x7fe1cc0ce010 2023-10-16T13:46:19.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.251+0000 7fe1d2ac9700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1cc0cf440 con 0x7fe1cc15a5c0 2023-10-16T13:46:19.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.252+0000 7fe1d2ac9700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1cc0cba30 con 0x7fe1cc0d0ae0 2023-10-16T13:46:19.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.252+0000 7fe1cbfff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe1cc0ce010 0x7fe1cc0d03d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42226/0 (socket says 172.21.15.16:42226) 2023-10-16T13:46:19.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.252+0000 7fe1cbfff700 1 -- 172.21.15.16:0/2530591134 learned_addr learned my addr 172.21.15.16:0/2530591134 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:46:19.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.252+0000 7fe1d0865700 1 --2- 172.21.15.16:0/2530591134 >> v2:172.21.15.49:3300/0 conn(0x7fe1cc15a5c0 0x7fe1cc0cef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:19.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.252+0000 7fe1d1066700 1 --2- 172.21.15.16:0/2530591134 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe1cc0d0ae0 0x7fe1cc0cc1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:19.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.252+0000 7fe1cb7fe700 1 -- 172.21.15.16:0/2530591134 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1495621034 0 0) 0x7fe1cc072a50 con 0x7fe1cc0ce010 2023-10-16T13:46:19.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.252+0000 7fe1cb7fe700 1 -- 172.21.15.16:0/2530591134 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1bc0035b0 con 0x7fe1cc0ce010 2023-10-16T13:46:19.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.252+0000 7fe1cb7fe700 1 -- 172.21.15.16:0/2530591134 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1867954991 0 0) 0x7fe1bc0035b0 con 0x7fe1cc0ce010 2023-10-16T13:46:19.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.252+0000 7fe1d1066700 1 -- 172.21.15.16:0/2530591134 >> v2:172.21.15.49:3300/0 conn(0x7fe1cc15a5c0 msgr2=0x7fe1cc0cef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:19.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.252+0000 7fe1d1066700 1 --2- 172.21.15.16:0/2530591134 >> v2:172.21.15.49:3300/0 conn(0x7fe1cc15a5c0 0x7fe1cc0cef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:19.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.252+0000 7fe1d1066700 1 -- 172.21.15.16:0/2530591134 >> v1:172.21.15.16:6789/0 conn(0x7fe1cc0ce010 legacy=0x7fe1cc0d03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:19.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.252+0000 7fe1d1066700 1 -- 172.21.15.16:0/2530591134 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1cc0cf630 con 0x7fe1cc0d0ae0 2023-10-16T13:46:19.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.252+0000 7fe1d0865700 1 --2- 172.21.15.16:0/2530591134 >> v2:172.21.15.49:3300/0 conn(0x7fe1cc15a5c0 0x7fe1cc0cef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:46:19.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.252+0000 7fe1d1066700 1 --2- 172.21.15.16:0/2530591134 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe1cc0d0ae0 0x7fe1cc0cc1f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fe1c4009d90 tx=0x7fe1c400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8e996694b7d8023d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:19.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.253+0000 7fe1cb7fe700 1 -- 172.21.15.16:0/2530591134 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1c400e070 con 0x7fe1cc0d0ae0 2023-10-16T13:46:19.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.253+0000 7fe1cb7fe700 1 -- 172.21.15.16:0/2530591134 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1c4003a20 con 0x7fe1cc0d0ae0 2023-10-16T13:46:19.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.253+0000 7fe1cb7fe700 1 -- 172.21.15.16:0/2530591134 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1b8001670 con 0x7fe1cc0d0ae0 2023-10-16T13:46:19.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.253+0000 7fe1d2ac9700 1 -- 172.21.15.16:0/2530591134 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe1cc0d0ae0 msgr2=0x7fe1cc0cc1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:19.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.253+0000 7fe1d2ac9700 1 --2- 172.21.15.16:0/2530591134 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe1cc0d0ae0 0x7fe1cc0cc1f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fe1c4009d90 tx=0x7fe1c400b330 comp rx=0 tx=0).stop 2023-10-16T13:46:19.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.253+0000 7fe1d2ac9700 1 -- 172.21.15.16:0/2530591134 shutdown_connections 2023-10-16T13:46:19.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.253+0000 7fe1d2ac9700 1 --2- 172.21.15.16:0/2530591134 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe1cc0d0ae0 0x7fe1cc0cc1f0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:19.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.253+0000 7fe1d2ac9700 1 --2- 172.21.15.16:0/2530591134 >> v2:172.21.15.49:3300/0 conn(0x7fe1cc15a5c0 0x7fe1cc0cef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:19.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.253+0000 7fe1d2ac9700 1 -- 172.21.15.16:0/2530591134 >> 172.21.15.16:0/2530591134 conn(0x7fe1cc0b1fd0 msgr2=0x7fe1cc0b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:19.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.253+0000 7fe1d2ac9700 1 -- 172.21.15.16:0/2530591134 shutdown_connections 2023-10-16T13:46:19.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.253+0000 7fe1d2ac9700 1 -- 172.21.15.16:0/2530591134 wait complete. 2023-10-16T13:46:19.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.254+0000 7fe1d2ac9700 1 Processor -- start 2023-10-16T13:46:19.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.254+0000 7fe1d2ac9700 1 -- start start 2023-10-16T13:46:19.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.255+0000 7fe1d2ac9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1cc0ce010 0x7fe1cc0cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:19.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.255+0000 7fe1d2ac9700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe1cc0d0ae0 0x7fe1cc1f43a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:19.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.255+0000 7fe1d2ac9700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe1cc15a5c0 0x7fe1cc0b0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:19.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.255+0000 7fe1d2ac9700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1cc0adf00 con 0x7fe1cc0ce010 2023-10-16T13:46:19.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.255+0000 7fe1d2ac9700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1cc1f39c0 con 0x7fe1cc15a5c0 2023-10-16T13:46:19.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.255+0000 7fe1d2ac9700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1cc1f3ca0 con 0x7fe1cc0d0ae0 2023-10-16T13:46:19.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.255+0000 7fe1d0865700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1cc0ce010 0x7fe1cc0cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:19.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.255+0000 7fe1d0865700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1cc0ce010 0x7fe1cc0cd1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52958/0 (socket says 172.21.15.16:52958) 2023-10-16T13:46:19.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.255+0000 7fe1d0865700 1 -- 172.21.15.16:0/3422898428 learned_addr learned my addr 172.21.15.16:0/3422898428 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:19.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.256+0000 7fe1d1066700 1 --2- 172.21.15.16:0/3422898428 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe1cc15a5c0 0x7fe1cc0b0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:19.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.256+0000 7fe1cbfff700 1 --2- 172.21.15.16:0/3422898428 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe1cc0d0ae0 0x7fe1cc1f43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:19.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.256+0000 7fe1d0865700 1 -- 172.21.15.16:0/3422898428 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe1cc0d0ae0 msgr2=0x7fe1cc1f43a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:19.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.256+0000 7fe1d0865700 1 --2- 172.21.15.16:0/3422898428 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe1cc0d0ae0 0x7fe1cc1f43a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:19.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.256+0000 7fe1d0865700 1 -- 172.21.15.16:0/3422898428 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe1cc15a5c0 msgr2=0x7fe1cc0b0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:19.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.256+0000 7fe1d0865700 1 --2- 172.21.15.16:0/3422898428 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe1cc15a5c0 0x7fe1cc0b0ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:19.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.256+0000 7fe1d0865700 1 -- 172.21.15.16:0/3422898428 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1c00094a0 con 0x7fe1cc0ce010 2023-10-16T13:46:19.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.256+0000 7fe1d1066700 1 --2- 172.21.15.16:0/3422898428 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe1cc15a5c0 0x7fe1cc0b0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:46:19.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.256+0000 7fe1cbfff700 1 --2- 172.21.15.16:0/3422898428 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe1cc0d0ae0 0x7fe1cc1f43a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:46:19.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.256+0000 7fe1d0865700 1 --2- 172.21.15.16:0/3422898428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1cc0ce010 0x7fe1cc0cd1e0 secure :-1 s=READY pgs=2422 cs=0 l=1 rev1=1 crypto rx=0x7fe1c000ebe0 tx=0x7fe1c0009ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:19.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.256+0000 7fe1c97fa700 1 -- 172.21.15.16:0/3422898428 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1c000c910 con 0x7fe1cc0ce010 2023-10-16T13:46:19.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.256+0000 7fe1c97fa700 1 -- 172.21.15.16:0/3422898428 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1c000ca70 con 0x7fe1cc0ce010 2023-10-16T13:46:19.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.256+0000 7fe1c97fa700 1 -- 172.21.15.16:0/3422898428 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1c000cde0 con 0x7fe1cc0ce010 2023-10-16T13:46:19.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.256+0000 7fe1d2ac9700 1 -- 172.21.15.16:0/3422898428 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1c4009a70 con 0x7fe1cc0ce010 2023-10-16T13:46:19.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.256+0000 7fe1d2ac9700 1 -- 172.21.15.16:0/3422898428 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1cc1f48e0 con 0x7fe1cc0ce010 2023-10-16T13:46:19.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.257+0000 7fe1d2ac9700 1 -- 172.21.15.16:0/3422898428 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1cc1f41c0 con 0x7fe1cc0ce010 2023-10-16T13:46:19.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.257+0000 7fe1c97fa700 1 -- 172.21.15.16:0/3422898428 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 138) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe1c0013470 con 0x7fe1cc0ce010 2023-10-16T13:46:19.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.258+0000 7fe1c97fa700 1 --2- 172.21.15.16:0/3422898428 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fe1b006b020 0x7fe1b006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:19.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.258+0000 7fe1c97fa700 1 -- 172.21.15.16:0/3422898428 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6165+0+0 (secure 0 0 0) 0x7fe1c0089a40 con 0x7fe1cc0ce010 2023-10-16T13:46:19.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.258+0000 7fe1cbfff700 1 --2- 172.21.15.16:0/3422898428 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fe1b006b020 0x7fe1b006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:19.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.258+0000 7fe1cbfff700 1 --2- 172.21.15.16:0/3422898428 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fe1b006b020 0x7fe1b006d4e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe1cc1e84c0 tx=0x7fe1b8009040 comp rx=0 tx=0).ready entity=mgr.11190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:19.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.262+0000 7fe1c97fa700 1 -- 172.21.15.16:0/3422898428 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe1c008be60 con 0x7fe1cc0ce010 2023-10-16T13:46:19.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.512+0000 7fe1d2ac9700 1 -- 172.21.15.16:0/3422898428 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe1cc0ae040 con 0x7fe1cc0ce010 2023-10-16T13:46:19.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.513+0000 7fe1c97fa700 1 -- 172.21.15.16:0/3422898428 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+3703 (secure 0 0 0) 0x7fe1c0021e90 con 0x7fe1cc0ce010 2023-10-16T13:46:19.516 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:46:19.516 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":106,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8729,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/3614602930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":3614602930},{"type":"v1","addr":"172.21.15.49:6835","nonce":3614602930}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":99},{"gid":9008,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/4274211183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":4274211183},{"type":"v1","addr":"172.21.15.16:6837","nonce":4274211183}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106},{"gid":10045,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/3427940027","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":3427940027},{"type":"v1","addr":"172.21.15.49:6834","nonce":3427940027}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":99},{"gid":10896,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/4279896386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":4279896386},{"type":"v1","addr":"172.21.15.16:6836","nonce":4279896386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98}],"filesystems":[]} 2023-10-16T13:46:19.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.515+0000 7fe1d2ac9700 1 -- 172.21.15.16:0/3422898428 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fe1b006b020 msgr2=0x7fe1b006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:19.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.515+0000 7fe1d2ac9700 1 --2- 172.21.15.16:0/3422898428 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fe1b006b020 0x7fe1b006d4e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe1cc1e84c0 tx=0x7fe1b8009040 comp rx=0 tx=0).stop 2023-10-16T13:46:19.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.515+0000 7fe1d2ac9700 1 -- 172.21.15.16:0/3422898428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1cc0ce010 msgr2=0x7fe1cc0cd1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:19.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.515+0000 7fe1d2ac9700 1 --2- 172.21.15.16:0/3422898428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1cc0ce010 0x7fe1cc0cd1e0 secure :-1 s=READY pgs=2422 cs=0 l=1 rev1=1 crypto rx=0x7fe1c000ebe0 tx=0x7fe1c0009ec0 comp rx=0 tx=0).stop 2023-10-16T13:46:19.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.515+0000 7fe1d2ac9700 1 -- 172.21.15.16:0/3422898428 shutdown_connections 2023-10-16T13:46:19.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.515+0000 7fe1d2ac9700 1 --2- 172.21.15.16:0/3422898428 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe1cc15a5c0 0x7fe1cc0b0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:19.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.515+0000 7fe1d2ac9700 1 --2- 172.21.15.16:0/3422898428 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fe1b006b020 0x7fe1b006d4e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:19.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.515+0000 7fe1d2ac9700 1 --2- 172.21.15.16:0/3422898428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1cc0ce010 0x7fe1cc0cd1e0 unknown :-1 s=CLOSED pgs=2422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:19.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.515+0000 7fe1d2ac9700 1 --2- 172.21.15.16:0/3422898428 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe1cc0d0ae0 0x7fe1cc1f43a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:19.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.515+0000 7fe1d2ac9700 1 -- 172.21.15.16:0/3422898428 >> 172.21.15.16:0/3422898428 conn(0x7fe1cc0b1fd0 msgr2=0x7fe1cc1598f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:19.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.515+0000 7fe1d2ac9700 1 -- 172.21.15.16:0/3422898428 shutdown_connections 2023-10-16T13:46:19.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.515+0000 7fe1d2ac9700 1 -- 172.21.15.16:0/3422898428 wait complete. 2023-10-16T13:46:19.520 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 106 2023-10-16T13:46:19.530 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-16T13:46:19.530 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-16T13:46:19.530 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-16T13:46:19.531 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T13:46:19.531 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-16T13:46:19.531 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-16T13:46:19.531 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-16T13:46:19.532 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T13:46:19.532 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-16T13:46:19.532 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-16T13:46:19.532 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-16T13:46:19.533 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T13:46:19.533 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-16T13:46:19.533 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-16T13:46:19.533 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-16T13:46:19.534 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T13:46:19.540 INFO:tasks.ceph.mds.a:Stopped 2023-10-16T13:46:19.540 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-16T13:46:19.541 INFO:tasks.ceph.mds.c:Stopped 2023-10-16T13:46:19.541 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-16T13:46:19.544 INFO:tasks.ceph.mds.d:Stopped 2023-10-16T13:46:19.545 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-16T13:46:19.546 INFO:tasks.ceph.mds.b:Stopped 2023-10-16T13:46:19.546 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-16T13:46:19.582 INFO:tasks.ceph.mds.a:Started 2023-10-16T13:46:19.583 DEBUG:teuthology.parallel:result is None 2023-10-16T13:46:19.584 INFO:tasks.ceph.mds.c:Started 2023-10-16T13:46:19.584 DEBUG:teuthology.parallel:result is None 2023-10-16T13:46:19.587 INFO:tasks.ceph.mds.d:Started 2023-10-16T13:46:19.587 DEBUG:teuthology.parallel:result is None 2023-10-16T13:46:19.588 INFO:tasks.ceph.mds.b:Started 2023-10-16T13:46:19.588 DEBUG:teuthology.parallel:result is None 2023-10-16T13:46:19.588 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-16T13:46:19.665 INFO:tasks.ceph.mds.c.smithi016.stdout:starting mds.c at 2023-10-16T13:46:19.666 INFO:tasks.ceph.mds.a.smithi016.stdout:starting mds.a at 2023-10-16T13:46:19.675 INFO:tasks.ceph.mds.d.smithi049.stdout:starting mds.d at 2023-10-16T13:46:19.676 INFO:tasks.ceph.mds.b.smithi049.stdout:starting mds.b at 2023-10-16T13:46:19.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.717+0000 7f10f09a8700 1 Processor -- start 2023-10-16T13:46:19.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.717+0000 7f10f09a8700 1 -- start start 2023-10-16T13:46:19.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.719+0000 7f10f09a8700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f10ec0d0ae0 0x7f10ec0cef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:19.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.719+0000 7f10f09a8700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10ec072a50 con 0x7f10ec0ce010 2023-10-16T13:46:19.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.719+0000 7f10f09a8700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10ec0cba30 con 0x7f10ec15a5c0 2023-10-16T13:46:19.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.719+0000 7f10f09a8700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10ec0cf440 con 0x7f10ec0d0ae0 2023-10-16T13:46:19.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.719+0000 7f10e9d9b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f10ec0d0ae0 0x7f10ec0cef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:19.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.719+0000 7f10ead9d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f10ec0ce010 0x7f10ec0cc1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42266/0 (socket says 172.21.15.16:42266) 2023-10-16T13:46:19.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.719+0000 7f10ead9d700 1 -- 172.21.15.16:0/3505905209 learned_addr learned my addr 172.21.15.16:0/3505905209 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:46:19.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.719+0000 7f10e9d9b700 1 -- 172.21.15.16:0/3505905209 >> v1:172.21.15.49:6789/0 conn(0x7f10ec15a5c0 legacy=0x7f10ec0d03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:19.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.719+0000 7f10e9d9b700 1 -- 172.21.15.16:0/3505905209 >> v1:172.21.15.16:6789/0 conn(0x7f10ec0ce010 legacy=0x7f10ec0cc1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:19.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.719+0000 7f10e959a700 1 -- 172.21.15.16:0/3505905209 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3255612312 0 0) 0x7f10ec072a50 con 0x7f10ec0ce010 2023-10-16T13:46:19.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.719+0000 7f10e9d9b700 1 -- 172.21.15.16:0/3505905209 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10ec0cf630 con 0x7f10ec0d0ae0 2023-10-16T13:46:19.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.719+0000 7f10e9d9b700 1 --2- 172.21.15.16:0/3505905209 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10ec0d0ae0 0x7f10ec0cef00 secure :-1 s=READY pgs=2431 cs=0 l=1 rev1=1 crypto rx=0x7f10dc009840 tx=0x7f10dc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10a303a6b9fd9458 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:19.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.720+0000 7f10e959a700 1 -- 172.21.15.16:0/3505905209 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10dc011830 con 0x7f10ec0d0ae0 2023-10-16T13:46:19.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.720+0000 7f10e959a700 1 -- 172.21.15.16:0/3505905209 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10dc011990 con 0x7f10ec0d0ae0 2023-10-16T13:46:19.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.720+0000 7f10e959a700 1 -- 172.21.15.16:0/3505905209 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10dc011d20 con 0x7f10ec0d0ae0 2023-10-16T13:46:19.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.720+0000 7f10f09a8700 1 -- 172.21.15.16:0/3505905209 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10ec0d0ae0 msgr2=0x7f10ec0cef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:19.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.720+0000 7f10f09a8700 1 --2- 172.21.15.16:0/3505905209 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10ec0d0ae0 0x7f10ec0cef00 secure :-1 s=READY pgs=2431 cs=0 l=1 rev1=1 crypto rx=0x7f10dc009840 tx=0x7f10dc009b50 comp rx=0 tx=0).stop 2023-10-16T13:46:19.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.720+0000 7f10f09a8700 1 -- 172.21.15.16:0/3505905209 shutdown_connections 2023-10-16T13:46:19.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.720+0000 7f10f09a8700 1 --2- 172.21.15.16:0/3505905209 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10ec0d0ae0 0x7f10ec0cef00 unknown :-1 s=CLOSED pgs=2431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:19.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.720+0000 7f10f09a8700 1 -- 172.21.15.16:0/3505905209 >> 172.21.15.16:0/3505905209 conn(0x7f10ec0b1fd0 msgr2=0x7f10ec0b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:19.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.720+0000 7f10f09a8700 1 -- 172.21.15.16:0/3505905209 shutdown_connections 2023-10-16T13:46:19.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.720+0000 7f10f09a8700 1 -- 172.21.15.16:0/3505905209 wait complete. 2023-10-16T13:46:19.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.722+0000 7f10f09a8700 1 Processor -- start 2023-10-16T13:46:19.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.722+0000 7f10f09a8700 1 -- start start 2023-10-16T13:46:19.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.723+0000 7f10f09a8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10ec0ce010 0x7f10ec1efaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:19.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.723+0000 7f10f09a8700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f10ec0d0ae0 0x7f10ec0c3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:19.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.723+0000 7f10f09a8700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f10ec15a5c0 0x7f10ec0cc840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:19.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.723+0000 7f10f09a8700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10ec157280 con 0x7f10ec0ce010 2023-10-16T13:46:19.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.723+0000 7f10f09a8700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10ec15b010 con 0x7f10ec0d0ae0 2023-10-16T13:46:19.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.723+0000 7f10f09a8700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10ec0c51f0 con 0x7f10ec15a5c0 2023-10-16T13:46:19.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.723+0000 7f10ea59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10ec0ce010 0x7f10ec1efaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:19.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.723+0000 7f10ea59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10ec0ce010 0x7f10ec1efaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52994/0 (socket says 172.21.15.16:52994) 2023-10-16T13:46:19.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.723+0000 7f10ea59c700 1 -- 172.21.15.16:0/3693740082 learned_addr learned my addr 172.21.15.16:0/3693740082 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:19.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.723+0000 7f10ead9d700 1 --2- 172.21.15.16:0/3693740082 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f10ec15a5c0 0x7f10ec0cc840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:19.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.723+0000 7f10e9d9b700 1 --2- 172.21.15.16:0/3693740082 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f10ec0d0ae0 0x7f10ec0c3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:19.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.723+0000 7f10ea59c700 1 -- 172.21.15.16:0/3693740082 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f10ec15a5c0 msgr2=0x7f10ec0cc840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:19.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.723+0000 7f10ea59c700 1 --2- 172.21.15.16:0/3693740082 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f10ec15a5c0 0x7f10ec0cc840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:19.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.723+0000 7f10ea59c700 1 -- 172.21.15.16:0/3693740082 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f10ec0d0ae0 msgr2=0x7f10ec0c3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:19.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.723+0000 7f10ea59c700 1 --2- 172.21.15.16:0/3693740082 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f10ec0d0ae0 0x7f10ec0c3f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:19.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.723+0000 7f10ea59c700 1 -- 172.21.15.16:0/3693740082 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10dc0094a0 con 0x7f10ec0ce010 2023-10-16T13:46:19.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.724+0000 7f10ea59c700 1 --2- 172.21.15.16:0/3693740082 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10ec0ce010 0x7f10ec1efaf0 secure :-1 s=READY pgs=2432 cs=0 l=1 rev1=1 crypto rx=0x7f10e000c4e0 tx=0x7f10e000c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:19.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.724+0000 7f10daffd700 1 -- 172.21.15.16:0/3693740082 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10e00052d0 con 0x7f10ec0ce010 2023-10-16T13:46:19.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.724+0000 7f10daffd700 1 -- 172.21.15.16:0/3693740082 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10e0005430 con 0x7f10ec0ce010 2023-10-16T13:46:19.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.724+0000 7f10f09a8700 1 -- 172.21.15.16:0/3693740082 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10ec0c4950 con 0x7f10ec0ce010 2023-10-16T13:46:19.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.724+0000 7f10daffd700 1 -- 172.21.15.16:0/3693740082 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10e00038c0 con 0x7f10ec0ce010 2023-10-16T13:46:19.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.724+0000 7f10f09a8700 1 -- 172.21.15.16:0/3693740082 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10ec0c4f10 con 0x7f10ec0ce010 2023-10-16T13:46:19.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.725+0000 7f10f09a8700 1 -- 172.21.15.16:0/3693740082 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10ec04ebb0 con 0x7f10ec0ce010 2023-10-16T13:46:19.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.725+0000 7f10daffd700 1 -- 172.21.15.16:0/3693740082 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 138) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f10e0003a20 con 0x7f10ec0ce010 2023-10-16T13:46:19.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.726+0000 7f10daffd700 1 --2- 172.21.15.16:0/3693740082 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f10cc06b020 0x7f10cc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:19.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.726+0000 7f10daffd700 1 -- 172.21.15.16:0/3693740082 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6165+0+0 (secure 0 0 0) 0x7f10e0088c60 con 0x7f10ec0ce010 2023-10-16T13:46:19.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.726+0000 7f10e9d9b700 1 --2- 172.21.15.16:0/3693740082 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f10cc06b020 0x7f10cc06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:19.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.726+0000 7f10e9d9b700 1 --2- 172.21.15.16:0/3693740082 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f10cc06b020 0x7f10cc06d4e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f10dc0090b0 tx=0x7f10dc008030 comp rx=0 tx=0).ready entity=mgr.11190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:19.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.730+0000 7f10daffd700 1 -- 172.21.15.16:0/3693740082 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f10e0053a30 con 0x7f10ec0ce010 2023-10-16T13:46:19.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:19.949+0000 7f10f09a8700 1 -- 172.21.15.16:0/3693740082 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f10ec0b60f0 con 0x7f10ec0ce010 2023-10-16T13:46:20.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.542+0000 7f10daffd700 1 -- 172.21.15.16:0/3693740082 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v219) v1 ==== 97+0+0 (secure 0 0 0) 0x7f10e0017080 con 0x7f10ec0ce010 2023-10-16T13:46:20.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.546+0000 7f10f09a8700 1 -- 172.21.15.16:0/3693740082 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f10cc06b020 msgr2=0x7f10cc06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:20.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.546+0000 7f10f09a8700 1 --2- 172.21.15.16:0/3693740082 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f10cc06b020 0x7f10cc06d4e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f10dc0090b0 tx=0x7f10dc008030 comp rx=0 tx=0).stop 2023-10-16T13:46:20.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.546+0000 7f10f09a8700 1 -- 172.21.15.16:0/3693740082 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10ec0ce010 msgr2=0x7f10ec1efaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:20.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.546+0000 7f10f09a8700 1 --2- 172.21.15.16:0/3693740082 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10ec0ce010 0x7f10ec1efaf0 secure :-1 s=READY pgs=2432 cs=0 l=1 rev1=1 crypto rx=0x7f10e000c4e0 tx=0x7f10e000c7f0 comp rx=0 tx=0).stop 2023-10-16T13:46:20.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.546+0000 7f10f09a8700 1 -- 172.21.15.16:0/3693740082 shutdown_connections 2023-10-16T13:46:20.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.546+0000 7f10f09a8700 1 --2- 172.21.15.16:0/3693740082 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f10ec15a5c0 0x7f10ec0cc840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:20.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.546+0000 7f10f09a8700 1 --2- 172.21.15.16:0/3693740082 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f10cc06b020 0x7f10cc06d4e0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:20.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.546+0000 7f10f09a8700 1 --2- 172.21.15.16:0/3693740082 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10ec0ce010 0x7f10ec1efaf0 unknown :-1 s=CLOSED pgs=2432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:20.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.546+0000 7f10f09a8700 1 --2- 172.21.15.16:0/3693740082 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f10ec0d0ae0 0x7f10ec0c3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:20.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.546+0000 7f10f09a8700 1 -- 172.21.15.16:0/3693740082 >> 172.21.15.16:0/3693740082 conn(0x7f10ec0b1fd0 msgr2=0x7f10ec0b2670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:20.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.546+0000 7f10f09a8700 1 -- 172.21.15.16:0/3693740082 shutdown_connections 2023-10-16T13:46:20.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.546+0000 7f10f09a8700 1 -- 172.21.15.16:0/3693740082 wait complete. 2023-10-16T13:46:20.551 INFO:teuthology.orchestra.run.smithi016.stderr: removed all blocklist entries 2023-10-16T13:46:20.565 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-16T13:46:20.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.692+0000 7fa365f29700 1 Processor -- start 2023-10-16T13:46:20.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.692+0000 7fa365f29700 1 -- start start 2023-10-16T13:46:20.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.693+0000 7fa365f29700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa36015db40 0x7fa3600cb980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:20.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.693+0000 7fa365f29700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fa3600ce870 0x7fa36006ac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:20.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.693+0000 7fa365f29700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3600cedb0 con 0x7fa36006a7e0 2023-10-16T13:46:20.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.693+0000 7fa365f29700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa36006b1a0 con 0x7fa3600ce870 2023-10-16T13:46:20.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.693+0000 7fa365f29700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa36015df20 con 0x7fa36015db40 2023-10-16T13:46:20.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.693+0000 7fa35f7fe700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa36015db40 0x7fa3600cb980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:20.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.693+0000 7fa35ffff700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fa3600ce870 0x7fa36006ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:20.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.693+0000 7fa35f7fe700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa36015db40 0x7fa3600cb980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:41262/0 (socket says 172.21.15.16:41262) 2023-10-16T13:46:20.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.693+0000 7fa35f7fe700 1 -- 172.21.15.16:0/547301800 learned_addr learned my addr 172.21.15.16:0/547301800 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:20.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.694+0000 7fa35f7fe700 1 -- 172.21.15.16:0/547301800 >> v2:172.21.15.49:3300/0 conn(0x7fa3600ce870 msgr2=0x7fa36006ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:20.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.694+0000 7fa35f7fe700 1 --2- 172.21.15.16:0/547301800 >> v2:172.21.15.49:3300/0 conn(0x7fa3600ce870 0x7fa36006ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:20.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.694+0000 7fa35f7fe700 1 -- 172.21.15.16:0/547301800 >> v1:172.21.15.49:6789/0 conn(0x7fa36006a7e0 legacy=0x7fa3600ce160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:20.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.694+0000 7fa35f7fe700 1 -- 172.21.15.16:0/547301800 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3600d12c0 con 0x7fa36015db40 2023-10-16T13:46:20.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.694+0000 7fa35ffff700 1 --2- 172.21.15.16:0/547301800 >> v2:172.21.15.49:3300/0 conn(0x7fa3600ce870 0x7fa36006ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:46:20.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.694+0000 7fa35f7fe700 1 --2- 172.21.15.16:0/547301800 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa36015db40 0x7fa3600cb980 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fa3480098f0 tx=0x7fa348009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=130dfed82b7487a2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:20.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.694+0000 7fa35e7fc700 1 -- 172.21.15.16:0/547301800 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa348011830 con 0x7fa36015db40 2023-10-16T13:46:20.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.694+0000 7fa35e7fc700 1 -- 172.21.15.16:0/547301800 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa348011990 con 0x7fa36015db40 2023-10-16T13:46:20.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.694+0000 7fa35e7fc700 1 -- 172.21.15.16:0/547301800 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa348011d20 con 0x7fa36015db40 2023-10-16T13:46:20.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.695+0000 7fa365f29700 1 -- 172.21.15.16:0/547301800 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa36015db40 msgr2=0x7fa3600cb980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:20.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.695+0000 7fa365f29700 1 --2- 172.21.15.16:0/547301800 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa36015db40 0x7fa3600cb980 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fa3480098f0 tx=0x7fa348009c00 comp rx=0 tx=0).stop 2023-10-16T13:46:20.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.695+0000 7fa365f29700 1 -- 172.21.15.16:0/547301800 shutdown_connections 2023-10-16T13:46:20.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.695+0000 7fa365f29700 1 --2- 172.21.15.16:0/547301800 >> v2:172.21.15.49:3300/0 conn(0x7fa3600ce870 0x7fa36006ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:20.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.695+0000 7fa365f29700 1 --2- 172.21.15.16:0/547301800 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa36015db40 0x7fa3600cb980 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:20.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.695+0000 7fa365f29700 1 -- 172.21.15.16:0/547301800 >> 172.21.15.16:0/547301800 conn(0x7fa3600b1fd0 msgr2=0x7fa3600b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:20.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.695+0000 7fa365f29700 1 -- 172.21.15.16:0/547301800 shutdown_connections 2023-10-16T13:46:20.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.695+0000 7fa365f29700 1 -- 172.21.15.16:0/547301800 wait complete. 2023-10-16T13:46:20.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.696+0000 7fa365f29700 1 Processor -- start 2023-10-16T13:46:20.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.696+0000 7fa365f29700 1 -- start start 2023-10-16T13:46:20.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.697+0000 7fa365f29700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa36006a7e0 0x7fa3601609e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:20.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.697+0000 7fa365f29700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa3600ce870 0x7fa3600c3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:20.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.697+0000 7fa365f29700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa36015db40 0x7fa3600c44b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:20.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.697+0000 7fa365f29700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3601470d0 con 0x7fa36015db40 2023-10-16T13:46:20.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.697+0000 7fa365f29700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa360158990 con 0x7fa3600ce870 2023-10-16T13:46:20.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.697+0000 7fa365f29700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa360147320 con 0x7fa36006a7e0 2023-10-16T13:46:20.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.697+0000 7fa35ffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa36015db40 0x7fa3600c44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:20.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.697+0000 7fa35ffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa36015db40 0x7fa3600c44b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53014/0 (socket says 172.21.15.16:53014) 2023-10-16T13:46:20.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.697+0000 7fa35ffff700 1 -- 172.21.15.16:0/2943917512 learned_addr learned my addr 172.21.15.16:0/2943917512 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:20.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.697+0000 7fa35f7fe700 1 --2- 172.21.15.16:0/2943917512 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa36006a7e0 0x7fa3601609e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:20.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.697+0000 7fa35effd700 1 --2- 172.21.15.16:0/2943917512 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa3600ce870 0x7fa3600c3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:20.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.697+0000 7fa35ffff700 1 -- 172.21.15.16:0/2943917512 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa36006a7e0 msgr2=0x7fa3601609e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:20.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.697+0000 7fa35ffff700 1 --2- 172.21.15.16:0/2943917512 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa36006a7e0 0x7fa3601609e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:20.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.697+0000 7fa35ffff700 1 -- 172.21.15.16:0/2943917512 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa3600ce870 msgr2=0x7fa3600c3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:20.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.697+0000 7fa35ffff700 1 --2- 172.21.15.16:0/2943917512 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa3600ce870 0x7fa3600c3f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:20.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.697+0000 7fa35ffff700 1 -- 172.21.15.16:0/2943917512 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa358009a70 con 0x7fa36015db40 2023-10-16T13:46:20.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.697+0000 7fa35effd700 1 --2- 172.21.15.16:0/2943917512 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa3600ce870 0x7fa3600c3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:46:20.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.698+0000 7fa35ffff700 1 --2- 172.21.15.16:0/2943917512 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa36015db40 0x7fa3600c44b0 secure :-1 s=READY pgs=2433 cs=0 l=1 rev1=1 crypto rx=0x7fa35800efb0 tx=0x7fa35800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:20.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.698+0000 7fa357fff700 1 -- 172.21.15.16:0/2943917512 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa35800cd40 con 0x7fa36015db40 2023-10-16T13:46:20.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.698+0000 7fa357fff700 1 -- 172.21.15.16:0/2943917512 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa35800f040 con 0x7fa36015db40 2023-10-16T13:46:20.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.698+0000 7fa365f29700 1 -- 172.21.15.16:0/2943917512 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa348009550 con 0x7fa36015db40 2023-10-16T13:46:20.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.698+0000 7fa357fff700 1 -- 172.21.15.16:0/2943917512 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa35800cea0 con 0x7fa36015db40 2023-10-16T13:46:20.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.698+0000 7fa365f29700 1 -- 172.21.15.16:0/2943917512 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3601478e0 con 0x7fa36015db40 2023-10-16T13:46:20.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.699+0000 7fa365f29700 1 -- 172.21.15.16:0/2943917512 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa36004ebb0 con 0x7fa36015db40 2023-10-16T13:46:20.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.700+0000 7fa357fff700 1 -- 172.21.15.16:0/2943917512 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 138) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa3580146a0 con 0x7fa36015db40 2023-10-16T13:46:20.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.700+0000 7fa357fff700 1 --2- 172.21.15.16:0/2943917512 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fa34006b020 0x7fa34006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:20.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.700+0000 7fa357fff700 1 -- 172.21.15.16:0/2943917512 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(221..221 src has 1..221) v4 ==== 5305+0+0 (secure 0 0 0) 0x7fa35808a280 con 0x7fa36015db40 2023-10-16T13:46:20.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.700+0000 7fa35f7fe700 1 --2- 172.21.15.16:0/2943917512 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fa34006b020 0x7fa34006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:20.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.701+0000 7fa35f7fe700 1 --2- 172.21.15.16:0/2943917512 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fa34006b020 0x7fa34006d4e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa348009c40 tx=0x7fa348008030 comp rx=0 tx=0).ready entity=mgr.11190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:20.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.705+0000 7fa357fff700 1 -- 172.21.15.16:0/2943917512 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa3580552a0 con 0x7fa36015db40 2023-10-16T13:46:20.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.992+0000 7fa365f29700 1 -- 172.21.15.16:0/2943917512 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fa360160fb0 con 0x7fa36015db40 2023-10-16T13:46:20.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.992+0000 7fa357fff700 1 -- 172.21.15.16:0/2943917512 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v54) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fa358009e60 con 0x7fa36015db40 2023-10-16T13:46:20.995 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:46:20.995 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-10-16T13:46:20.995 INFO:teuthology.orchestra.run.smithi016.stdout: "auth_dump": [ 2023-10-16T13:46:20.996 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:46:20.996 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.a", 2023-10-16T13:46:20.996 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOKC1l/rBtOhAA/ob5WUvpz8Glp4E+xPZOYA==", 2023-10-16T13:46:20.996 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:46:20.996 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T13:46:20.997 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-10-16T13:46:20.997 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-10-16T13:46:20.997 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:46:20.997 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:46:20.997 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:46:20.997 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:46:20.998 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.b", 2023-10-16T13:46:20.998 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1lKAVsCxAAvhXuLY6i1+Iz1yq5BVPMNQ==", 2023-10-16T13:46:20.998 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:46:20.998 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T13:46:20.998 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-10-16T13:46:20.999 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-10-16T13:46:20.999 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:46:20.999 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:46:20.999 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:46:20.999 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:46:21.000 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.c", 2023-10-16T13:46:21.000 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1lvBowBhAA3waJNYhdgs8PGCYqYNYUog==", 2023-10-16T13:46:21.000 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:46:21.000 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T13:46:21.000 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-10-16T13:46:21.001 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-10-16T13:46:21.001 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:46:21.001 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:46:21.001 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:46:21.001 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:46:21.001 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.d", 2023-10-16T13:46:21.002 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1lS/fnEhAAfNYgQ5BW86JMjYm/bBfnaQ==", 2023-10-16T13:46:21.002 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:46:21.002 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T13:46:21.002 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-10-16T13:46:21.002 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-10-16T13:46:21.003 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:46:21.003 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:46:21.003 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:46:21.003 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:46:21.003 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.0", 2023-10-16T13:46:21.004 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAQKC1lbxllOhAAZeLdp2GMl6HqQPaQjKlNqw==", 2023-10-16T13:46:21.004 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:46:21.004 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T13:46:21.004 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T13:46:21.004 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:46:21.005 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:46:21.005 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:46:21.005 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:46:21.005 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.1", 2023-10-16T13:46:21.005 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQATKC1lnU6KOhAAHxvlbmzBptf1keGgL+Rbkg==", 2023-10-16T13:46:21.005 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:46:21.006 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T13:46:21.006 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T13:46:21.006 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:46:21.006 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:46:21.006 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:46:21.006 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:46:21.007 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.2", 2023-10-16T13:46:21.007 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAXKC1l9jFyARAAr/NAwtWhBMut9sAAoDEGWg==", 2023-10-16T13:46:21.007 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:46:21.007 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T13:46:21.007 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T13:46:21.008 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:46:21.008 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:46:21.008 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:46:21.008 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:46:21.008 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.3", 2023-10-16T13:46:21.009 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAaKC1lbw/lARAAGJ/9DNb5/7kLFSq7vrLtXg==", 2023-10-16T13:46:21.009 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:46:21.009 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T13:46:21.009 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T13:46:21.009 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:46:21.009 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:46:21.010 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:46:21.010 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:46:21.010 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.4", 2023-10-16T13:46:21.010 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAeKC1lZb7jERAAYpvb1xcSli9CTq7L8ChLOA==", 2023-10-16T13:46:21.010 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:46:21.011 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T13:46:21.011 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T13:46:21.011 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:46:21.011 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:46:21.011 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:46:21.012 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:46:21.012 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.5", 2023-10-16T13:46:21.012 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAhKC1l/4HuEBAAqfnRqjknij9KES/fqiyIBQ==", 2023-10-16T13:46:21.012 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:46:21.012 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T13:46:21.013 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T13:46:21.013 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:46:21.013 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:46:21.013 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:46:21.013 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:46:21.013 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.6", 2023-10-16T13:46:21.014 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAkKC1lzoUkEBAA+Q82OiBMNag8+R316WQtUw==", 2023-10-16T13:46:21.014 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:46:21.014 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T13:46:21.014 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T13:46:21.014 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:46:21.015 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:46:21.015 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:46:21.015 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:46:21.015 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.7", 2023-10-16T13:46:21.015 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAnKC1ljUu4DxAAc+G4anJANC2GhLJ8wb6gYg==", 2023-10-16T13:46:21.015 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:46:21.016 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T13:46:21.016 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T13:46:21.016 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:46:21.016 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:46:21.016 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:46:21.016 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:46:21.017 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.0", 2023-10-16T13:46:21.017 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1l2l6PFxAA9QyXtBAHdGAXpwH0JfknuQ==", 2023-10-16T13:46:21.017 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:46:21.017 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T13:46:21.017 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow r", 2023-10-16T13:46:21.018 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-16T13:46:21.018 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:46:21.018 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:46:21.018 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:46:21.018 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.1", 2023-10-16T13:46:21.019 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1lAcHFIxAAcNt2bmv2R6j5L3DYkdw7eQ==", 2023-10-16T13:46:21.019 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:46:21.019 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T13:46:21.019 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-10-16T13:46:21.019 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-10-16T13:46:21.020 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-10-16T13:46:21.020 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:46:21.020 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:46:21.020 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:46:21.020 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.2", 2023-10-16T13:46:21.020 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1lA6wnGxAAdtaqYhtIJu8qENMrD8xvVA==", 2023-10-16T13:46:21.020 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:46:21.021 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T13:46:21.021 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-10-16T13:46:21.021 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-10-16T13:46:21.021 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-10-16T13:46:21.021 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:46:21.022 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:46:21.022 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:46:21.022 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.3", 2023-10-16T13:46:21.022 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1l/ddAJxAA3fNKKn9XkXas1BlDN+ggjw==", 2023-10-16T13:46:21.022 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:46:21.022 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T13:46:21.023 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-10-16T13:46:21.023 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-10-16T13:46:21.023 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-10-16T13:46:21.023 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:46:21.023 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:46:21.024 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:46:21.024 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.admin", 2023-10-16T13:46:21.024 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOKC1ld5jUHBAAc6esnCce3dSEZn3wVjhIlQ==", 2023-10-16T13:46:21.024 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:46:21.024 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2023-10-16T13:46:21.024 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-10-16T13:46:21.024 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-10-16T13:46:21.025 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:46:21.025 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:46:21.025 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:46:21.025 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:46:21.025 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mgr.x", 2023-10-16T13:46:21.025 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOKC1l6b3vNhAAmj5xbfid2sZM2KfHmAf1Wg==", 2023-10-16T13:46:21.026 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:46:21.026 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2023-10-16T13:46:21.026 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile mgr", 2023-10-16T13:46:21.026 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:46:21.026 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:46:21.027 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:46:21.027 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:46:21.027 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mgr.y", 2023-10-16T13:46:21.027 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOKC1lLz2nMhAA8NqFYZ5nDKkzPBMrkwsVuQ==", 2023-10-16T13:46:21.027 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:46:21.027 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2023-10-16T13:46:21.027 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile mgr", 2023-10-16T13:46:21.028 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:46:21.028 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:46:21.028 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:46:21.028 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-10-16T13:46:21.028 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-10-16T13:46:21.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.994+0000 7fa365f29700 1 -- 172.21.15.16:0/2943917512 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fa34006b020 msgr2=0x7fa34006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:21.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.994+0000 7fa365f29700 1 --2- 172.21.15.16:0/2943917512 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fa34006b020 0x7fa34006d4e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa348009c40 tx=0x7fa348008030 comp rx=0 tx=0).stop 2023-10-16T13:46:21.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.994+0000 7fa365f29700 1 -- 172.21.15.16:0/2943917512 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa36015db40 msgr2=0x7fa3600c44b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:21.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.994+0000 7fa365f29700 1 --2- 172.21.15.16:0/2943917512 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa36015db40 0x7fa3600c44b0 secure :-1 s=READY pgs=2433 cs=0 l=1 rev1=1 crypto rx=0x7fa35800efb0 tx=0x7fa35800c3a0 comp rx=0 tx=0).stop 2023-10-16T13:46:21.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.995+0000 7fa365f29700 1 -- 172.21.15.16:0/2943917512 shutdown_connections 2023-10-16T13:46:21.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.995+0000 7fa365f29700 1 --2- 172.21.15.16:0/2943917512 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fa34006b020 0x7fa34006d4e0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:21.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.995+0000 7fa365f29700 1 --2- 172.21.15.16:0/2943917512 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa36015db40 0x7fa3600c44b0 unknown :-1 s=CLOSED pgs=2433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:21.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.995+0000 7fa365f29700 1 --2- 172.21.15.16:0/2943917512 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa36006a7e0 0x7fa3601609e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:21.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.995+0000 7fa365f29700 1 --2- 172.21.15.16:0/2943917512 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa3600ce870 0x7fa3600c3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:21.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.995+0000 7fa365f29700 1 -- 172.21.15.16:0/2943917512 >> 172.21.15.16:0/2943917512 conn(0x7fa3600b1fd0 msgr2=0x7fa36015c470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:21.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.995+0000 7fa365f29700 1 -- 172.21.15.16:0/2943917512 shutdown_connections 2023-10-16T13:46:21.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:20.995+0000 7fa365f29700 1 -- 172.21.15.16:0/2943917512 wait complete. 2023-10-16T13:46:21.032 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:46:21.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.155+0000 7f0fb8bbc700 1 Processor -- start 2023-10-16T13:46:21.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.155+0000 7f0fb8bbc700 1 -- start start 2023-10-16T13:46:21.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.156+0000 7f0fb8bbc700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f0fb415a530 0x7f0fb40b32d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:21.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.156+0000 7f0fb8bbc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0fb4072560 0x7f0fb40cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:21.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.156+0000 7f0fb8bbc700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fb40cba90 con 0x7f0fb40729f0 2023-10-16T13:46:21.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.156+0000 7f0fb8bbc700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fb40cd230 con 0x7f0fb4072560 2023-10-16T13:46:21.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.156+0000 7f0fb8bbc700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fb40cd510 con 0x7f0fb415a530 2023-10-16T13:46:21.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.156+0000 7f0fb1d9b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0fb4072560 0x7f0fb40cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:21.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.156+0000 7f0fb1d9b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0fb4072560 0x7f0fb40cf230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53018/0 (socket says 172.21.15.16:53018) 2023-10-16T13:46:21.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.156+0000 7f0fb1d9b700 1 -- 172.21.15.16:0/3545414382 learned_addr learned my addr 172.21.15.16:0/3545414382 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:21.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.156+0000 7f0fb259c700 1 --2- 172.21.15.16:0/3545414382 >> v2:172.21.15.49:3300/0 conn(0x7f0fb415a530 0x7f0fb40b32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:21.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.156+0000 7f0fb1d9b700 1 -- 172.21.15.16:0/3545414382 >> v2:172.21.15.49:3300/0 conn(0x7f0fb415a530 msgr2=0x7f0fb40b32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:21.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.156+0000 7f0fb1d9b700 1 --2- 172.21.15.16:0/3545414382 >> v2:172.21.15.49:3300/0 conn(0x7f0fb415a530 0x7f0fb40b32d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:21.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.156+0000 7f0fb1d9b700 1 -- 172.21.15.16:0/3545414382 >> v1:172.21.15.49:6789/0 conn(0x7f0fb40729f0 legacy=0x7f0fb40cbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:21.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.156+0000 7f0fb1d9b700 1 -- 172.21.15.16:0/3545414382 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fb40cd7f0 con 0x7f0fb4072560 2023-10-16T13:46:21.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.156+0000 7f0fb1d9b700 1 --2- 172.21.15.16:0/3545414382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fb4072560 0x7f0fb40cf230 secure :-1 s=READY pgs=2434 cs=0 l=1 rev1=1 crypto rx=0x7f0fb406eaa0 tx=0x7f0fa800be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72d26af1dd8e7dfa server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:21.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.157+0000 7f0fb159a700 1 -- 172.21.15.16:0/3545414382 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fa800dd30 con 0x7f0fb4072560 2023-10-16T13:46:21.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.157+0000 7f0fb159a700 1 -- 172.21.15.16:0/3545414382 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fa800de90 con 0x7f0fb4072560 2023-10-16T13:46:21.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.157+0000 7f0fb159a700 1 -- 172.21.15.16:0/3545414382 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fa8010690 con 0x7f0fb4072560 2023-10-16T13:46:21.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.157+0000 7f0fb8bbc700 1 -- 172.21.15.16:0/3545414382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fb4072560 msgr2=0x7f0fb40cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:21.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.157+0000 7f0fb8bbc700 1 --2- 172.21.15.16:0/3545414382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fb4072560 0x7f0fb40cf230 secure :-1 s=READY pgs=2434 cs=0 l=1 rev1=1 crypto rx=0x7f0fb406eaa0 tx=0x7f0fa800be70 comp rx=0 tx=0).stop 2023-10-16T13:46:21.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.157+0000 7f0fb8bbc700 1 -- 172.21.15.16:0/3545414382 shutdown_connections 2023-10-16T13:46:21.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.157+0000 7f0fb8bbc700 1 --2- 172.21.15.16:0/3545414382 >> v2:172.21.15.49:3300/0 conn(0x7f0fb415a530 0x7f0fb40b32d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:21.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.157+0000 7f0fb8bbc700 1 --2- 172.21.15.16:0/3545414382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fb4072560 0x7f0fb40cf230 unknown :-1 s=CLOSED pgs=2434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:21.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.157+0000 7f0fb8bbc700 1 -- 172.21.15.16:0/3545414382 >> 172.21.15.16:0/3545414382 conn(0x7f0fb40b1fd0 msgr2=0x7f0fb40b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:21.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.157+0000 7f0fb8bbc700 1 -- 172.21.15.16:0/3545414382 shutdown_connections 2023-10-16T13:46:21.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.157+0000 7f0fb8bbc700 1 -- 172.21.15.16:0/3545414382 wait complete. 2023-10-16T13:46:21.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.159+0000 7f0fb8bbc700 1 Processor -- start 2023-10-16T13:46:21.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.159+0000 7f0fb8bbc700 1 -- start start 2023-10-16T13:46:21.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.160+0000 7f0fb8bbc700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0fb4072560 0x7f0fb414e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:21.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.160+0000 7f0fb8bbc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fb40729f0 0x7f0fb40ce7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:21.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.160+0000 7f0fb8bbc700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0fb415a530 0x7f0fb40b5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:21.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.160+0000 7f0fb8bbc700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb40cf1a0 con 0x7f0fb40729f0 2023-10-16T13:46:21.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.160+0000 7f0fb8bbc700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb415d920 con 0x7f0fb415a530 2023-10-16T13:46:21.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.160+0000 7f0fb8bbc700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb414ef00 con 0x7f0fb4072560 2023-10-16T13:46:21.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.160+0000 7f0fb1d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fb40729f0 0x7f0fb40ce7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:21.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.160+0000 7f0fb1d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fb40729f0 0x7f0fb40ce7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53026/0 (socket says 172.21.15.16:53026) 2023-10-16T13:46:21.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.160+0000 7f0fb1d9b700 1 -- 172.21.15.16:0/2966044373 learned_addr learned my addr 172.21.15.16:0/2966044373 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:21.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.160+0000 7f0fb2d9d700 1 --2- 172.21.15.16:0/2966044373 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0fb415a530 0x7f0fb40b5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:21.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.160+0000 7f0fb259c700 1 --2- 172.21.15.16:0/2966044373 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0fb4072560 0x7f0fb414e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:21.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.160+0000 7f0fb1d9b700 1 -- 172.21.15.16:0/2966044373 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0fb4072560 msgr2=0x7f0fb414e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:21.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.160+0000 7f0fb1d9b700 1 --2- 172.21.15.16:0/2966044373 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0fb4072560 0x7f0fb414e9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:21.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.160+0000 7f0fb1d9b700 1 -- 172.21.15.16:0/2966044373 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0fb415a530 msgr2=0x7f0fb40b5140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:21.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.160+0000 7f0fb1d9b700 1 --2- 172.21.15.16:0/2966044373 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0fb415a530 0x7f0fb40b5140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:21.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.161+0000 7f0fb1d9b700 1 -- 172.21.15.16:0/2966044373 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fa800b7f0 con 0x7f0fb40729f0 2023-10-16T13:46:21.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.161+0000 7f0fb259c700 1 --2- 172.21.15.16:0/2966044373 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0fb4072560 0x7f0fb414e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:46:21.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.161+0000 7f0fb1d9b700 1 --2- 172.21.15.16:0/2966044373 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fb40729f0 0x7f0fb40ce7c0 secure :-1 s=READY pgs=2435 cs=0 l=1 rev1=1 crypto rx=0x7f0fa800b7c0 tx=0x7f0fa8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:21.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.161+0000 7f0fa2ffd700 1 -- 172.21.15.16:0/2966044373 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fa8003c70 con 0x7f0fb40729f0 2023-10-16T13:46:21.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.161+0000 7f0fa2ffd700 1 -- 172.21.15.16:0/2966044373 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fa8003dd0 con 0x7f0fb40729f0 2023-10-16T13:46:21.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.161+0000 7f0fb8bbc700 1 -- 172.21.15.16:0/2966044373 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fb415cce0 con 0x7f0fb40729f0 2023-10-16T13:46:21.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.161+0000 7f0fa2ffd700 1 -- 172.21.15.16:0/2966044373 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fa80106b0 con 0x7f0fb40729f0 2023-10-16T13:46:21.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.161+0000 7f0fb8bbc700 1 -- 172.21.15.16:0/2966044373 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fb415d260 con 0x7f0fb40729f0 2023-10-16T13:46:21.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.162+0000 7f0fb8bbc700 1 -- 172.21.15.16:0/2966044373 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fb404ebb0 con 0x7f0fb40729f0 2023-10-16T13:46:21.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.162+0000 7f0fa2ffd700 1 -- 172.21.15.16:0/2966044373 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 138) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0fa8023440 con 0x7f0fb40729f0 2023-10-16T13:46:21.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.163+0000 7f0fa2ffd700 1 --2- 172.21.15.16:0/2966044373 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f0f9406b020 0x7f0f9406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:21.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.163+0000 7f0fa2ffd700 1 -- 172.21.15.16:0/2966044373 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(221..221 src has 1..221) v4 ==== 5305+0+0 (secure 0 0 0) 0x7f0fa808af90 con 0x7f0fb40729f0 2023-10-16T13:46:21.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.163+0000 7f0fb259c700 1 --2- 172.21.15.16:0/2966044373 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f0f9406b020 0x7f0f9406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:21.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.163+0000 7f0fb259c700 1 --2- 172.21.15.16:0/2966044373 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f0f9406b020 0x7f0f9406d4e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0fa4007a50 tx=0x7f0fa4008040 comp rx=0 tx=0).ready entity=mgr.11190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:21.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.167+0000 7f0fa2ffd700 1 -- 172.21.15.16:0/2966044373 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0fa8055fa0 con 0x7f0fb40729f0 2023-10-16T13:46:21.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.373+0000 7f0fb8bbc700 1 -- 172.21.15.16:0/2966044373 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0fb40b0af0 con 0x7f0fb40729f0 2023-10-16T13:46:21.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.374+0000 7f0fa2ffd700 1 -- 172.21.15.16:0/2966044373 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v221) v1 ==== 74+0+13169 (secure 0 0 0) 0x7f0fa80595c0 con 0x7f0fb40729f0 2023-10-16T13:46:21.377 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:46:21.377 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":221,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:46:20.550542+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":204,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":204,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":204,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":204,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":204,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":204,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:46:21.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.376+0000 7f0fb8bbc700 1 -- 172.21.15.16:0/2966044373 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f0f9406b020 msgr2=0x7f0f9406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:21.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.376+0000 7f0fb8bbc700 1 --2- 172.21.15.16:0/2966044373 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f0f9406b020 0x7f0f9406d4e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0fa4007a50 tx=0x7f0fa4008040 comp rx=0 tx=0).stop 2023-10-16T13:46:21.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.376+0000 7f0fb8bbc700 1 -- 172.21.15.16:0/2966044373 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fb40729f0 msgr2=0x7f0fb40ce7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:21.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.376+0000 7f0fb8bbc700 1 --2- 172.21.15.16:0/2966044373 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fb40729f0 0x7f0fb40ce7c0 secure :-1 s=READY pgs=2435 cs=0 l=1 rev1=1 crypto rx=0x7f0fa800b7c0 tx=0x7f0fa8004750 comp rx=0 tx=0).stop 2023-10-16T13:46:21.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.376+0000 7f0fb8bbc700 1 -- 172.21.15.16:0/2966044373 shutdown_connections 2023-10-16T13:46:21.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.376+0000 7f0fb8bbc700 1 --2- 172.21.15.16:0/2966044373 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0fb415a530 0x7f0fb40b5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:21.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.376+0000 7f0fb8bbc700 1 --2- 172.21.15.16:0/2966044373 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0fb4072560 0x7f0fb414e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:21.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.376+0000 7f0fb8bbc700 1 --2- 172.21.15.16:0/2966044373 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f0f9406b020 0x7f0f9406d4e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:21.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.376+0000 7f0fb8bbc700 1 --2- 172.21.15.16:0/2966044373 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0fb40729f0 0x7f0fb40ce7c0 unknown :-1 s=CLOSED pgs=2435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:21.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.376+0000 7f0fb8bbc700 1 -- 172.21.15.16:0/2966044373 >> 172.21.15.16:0/2966044373 conn(0x7f0fb40b1fd0 msgr2=0x7f0fb40b1010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:21.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.376+0000 7f0fb8bbc700 1 -- 172.21.15.16:0/2966044373 shutdown_connections 2023-10-16T13:46:21.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.376+0000 7f0fb8bbc700 1 -- 172.21.15.16:0/2966044373 wait complete. 2023-10-16T13:46:21.391 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T13:46:21.392 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T13:46:21.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.517+0000 7f1dfc24b700 1 Processor -- start 2023-10-16T13:46:21.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.517+0000 7f1dfc24b700 1 -- start start 2023-10-16T13:46:21.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.518+0000 7f1dfc24b700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f1df415b2c0 0x7f1df4161340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:21.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.518+0000 7f1dfc24b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1df406a7e0 0x7f1df406ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:21.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.518+0000 7f1dfc24b700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1df415ecb0 con 0x7f1df406e570 2023-10-16T13:46:21.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.518+0000 7f1dfc24b700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1df406b1b0 con 0x7f1df415b2c0 2023-10-16T13:46:21.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.518+0000 7f1dfc24b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1df4161880 con 0x7f1df406a7e0 2023-10-16T13:46:21.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.519+0000 7f1df97e6700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f1df406e570 0x7f1df415e5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42306/0 (socket says 172.21.15.16:42306) 2023-10-16T13:46:21.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.519+0000 7f1df97e6700 1 -- 172.21.15.16:0/2335908975 learned_addr learned my addr 172.21.15.16:0/2335908975 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:46:21.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.519+0000 7f1df9fe7700 1 --2- 172.21.15.16:0/2335908975 >> v2:172.21.15.49:3300/0 conn(0x7f1df415b2c0 0x7f1df4161340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:21.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.519+0000 7f1dfa7e8700 1 --2- 172.21.15.16:0/2335908975 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1df406a7e0 0x7f1df406ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:21.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.519+0000 7f1df8fe5700 1 -- 172.21.15.16:0/2335908975 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 481176947 0 0) 0x7f1df415ecb0 con 0x7f1df406e570 2023-10-16T13:46:21.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.519+0000 7f1df8fe5700 1 -- 172.21.15.16:0/2335908975 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1de00035b0 con 0x7f1df406e570 2023-10-16T13:46:21.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.519+0000 7f1df9fe7700 1 -- 172.21.15.16:0/2335908975 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1df406a7e0 msgr2=0x7f1df406ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:21.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.519+0000 7f1df9fe7700 1 --2- 172.21.15.16:0/2335908975 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1df406a7e0 0x7f1df406ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:21.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.519+0000 7f1df9fe7700 1 -- 172.21.15.16:0/2335908975 >> v1:172.21.15.16:6789/0 conn(0x7f1df406e570 legacy=0x7f1df415e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:21.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.519+0000 7f1df9fe7700 1 -- 172.21.15.16:0/2335908975 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1df415b6a0 con 0x7f1df415b2c0 2023-10-16T13:46:21.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.519+0000 7f1dfa7e8700 1 --2- 172.21.15.16:0/2335908975 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1df406a7e0 0x7f1df406ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:46:21.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.519+0000 7f1df9fe7700 1 --2- 172.21.15.16:0/2335908975 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1df415b2c0 0x7f1df4161340 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f1de4009840 tx=0x7f1de4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4321cad6b291b0f6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:21.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.519+0000 7f1df8fe5700 1 -- 172.21.15.16:0/2335908975 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de4011830 con 0x7f1df415b2c0 2023-10-16T13:46:21.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.519+0000 7f1df8fe5700 1 -- 172.21.15.16:0/2335908975 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1de4011990 con 0x7f1df415b2c0 2023-10-16T13:46:21.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.519+0000 7f1df8fe5700 1 -- 172.21.15.16:0/2335908975 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de4011d20 con 0x7f1df415b2c0 2023-10-16T13:46:21.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.520+0000 7f1dfc24b700 1 -- 172.21.15.16:0/2335908975 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1df415b2c0 msgr2=0x7f1df4161340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:21.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.520+0000 7f1dfc24b700 1 --2- 172.21.15.16:0/2335908975 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1df415b2c0 0x7f1df4161340 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f1de4009840 tx=0x7f1de4009b50 comp rx=0 tx=0).stop 2023-10-16T13:46:21.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.520+0000 7f1dfc24b700 1 -- 172.21.15.16:0/2335908975 shutdown_connections 2023-10-16T13:46:21.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.520+0000 7f1dfc24b700 1 --2- 172.21.15.16:0/2335908975 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1df406a7e0 0x7f1df406ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:21.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.520+0000 7f1dfc24b700 1 --2- 172.21.15.16:0/2335908975 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1df415b2c0 0x7f1df4161340 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:21.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.520+0000 7f1dfc24b700 1 -- 172.21.15.16:0/2335908975 >> 172.21.15.16:0/2335908975 conn(0x7f1df40b1f40 msgr2=0x7f1df40b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:21.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.520+0000 7f1dfc24b700 1 -- 172.21.15.16:0/2335908975 shutdown_connections 2023-10-16T13:46:21.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.520+0000 7f1dfc24b700 1 -- 172.21.15.16:0/2335908975 wait complete. 2023-10-16T13:46:21.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.521+0000 7f1dfc24b700 1 Processor -- start 2023-10-16T13:46:21.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.521+0000 7f1dfc24b700 1 -- start start 2023-10-16T13:46:21.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.522+0000 7f1dfc24b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1df406a7e0 0x7f1df414ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:21.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.522+0000 7f1dfc24b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1df406e570 0x7f1df4160bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:21.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.522+0000 7f1dfc24b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1df415b2c0 0x7f1df40cbef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:21.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.522+0000 7f1dfc24b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1df41f6450 con 0x7f1df406e570 2023-10-16T13:46:21.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.522+0000 7f1dfc24b700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1df41f6710 con 0x7f1df415b2c0 2023-10-16T13:46:21.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.522+0000 7f1dfc24b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1df41f69f0 con 0x7f1df406a7e0 2023-10-16T13:46:21.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.522+0000 7f1df97e6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1df406e570 0x7f1df4160bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:21.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.522+0000 7f1df97e6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1df406e570 0x7f1df4160bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53040/0 (socket says 172.21.15.16:53040) 2023-10-16T13:46:21.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.522+0000 7f1df97e6700 1 -- 172.21.15.16:0/347750990 learned_addr learned my addr 172.21.15.16:0/347750990 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:21.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.522+0000 7f1dfa7e8700 1 --2- 172.21.15.16:0/347750990 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1df415b2c0 0x7f1df40cbef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:21.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.522+0000 7f1df9fe7700 1 --2- 172.21.15.16:0/347750990 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1df406a7e0 0x7f1df414ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:21.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.523+0000 7f1df97e6700 1 -- 172.21.15.16:0/347750990 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1df406a7e0 msgr2=0x7f1df414ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:21.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.523+0000 7f1df97e6700 1 --2- 172.21.15.16:0/347750990 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1df406a7e0 0x7f1df414ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:21.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.523+0000 7f1df97e6700 1 -- 172.21.15.16:0/347750990 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1df415b2c0 msgr2=0x7f1df40cbef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:21.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.523+0000 7f1df97e6700 1 --2- 172.21.15.16:0/347750990 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1df415b2c0 0x7f1df40cbef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:21.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.523+0000 7f1df97e6700 1 -- 172.21.15.16:0/347750990 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dec009a70 con 0x7f1df406e570 2023-10-16T13:46:21.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.523+0000 7f1df9fe7700 1 --2- 172.21.15.16:0/347750990 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1df406a7e0 0x7f1df414ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:46:21.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.523+0000 7f1df97e6700 1 --2- 172.21.15.16:0/347750990 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1df406e570 0x7f1df4160bf0 secure :-1 s=READY pgs=2437 cs=0 l=1 rev1=1 crypto rx=0x7f1df0001eb0 tx=0x7f1df000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:21.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.523+0000 7f1dea7fc700 1 -- 172.21.15.16:0/347750990 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1df0005d70 con 0x7f1df406e570 2023-10-16T13:46:21.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.523+0000 7f1dea7fc700 1 -- 172.21.15.16:0/347750990 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1df00070d0 con 0x7f1df406e570 2023-10-16T13:46:21.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.523+0000 7f1dfc24b700 1 -- 172.21.15.16:0/347750990 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1de40094a0 con 0x7f1df406e570 2023-10-16T13:46:21.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.523+0000 7f1dea7fc700 1 -- 172.21.15.16:0/347750990 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1df0007460 con 0x7f1df406e570 2023-10-16T13:46:21.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.523+0000 7f1dfc24b700 1 -- 172.21.15.16:0/347750990 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1df41f6fb0 con 0x7f1df406e570 2023-10-16T13:46:21.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.524+0000 7f1dfc24b700 1 -- 172.21.15.16:0/347750990 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1df404ebb0 con 0x7f1df406e570 2023-10-16T13:46:21.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.524+0000 7f1dea7fc700 1 -- 172.21.15.16:0/347750990 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 138) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1df00075c0 con 0x7f1df406e570 2023-10-16T13:46:21.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.525+0000 7f1dea7fc700 1 --2- 172.21.15.16:0/347750990 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f1dd806b020 0x7f1dd806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:21.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.525+0000 7f1dea7fc700 1 -- 172.21.15.16:0/347750990 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(221..221 src has 1..221) v4 ==== 5305+0+0 (secure 0 0 0) 0x7f1df0088ad0 con 0x7f1df406e570 2023-10-16T13:46:21.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.525+0000 7f1df9fe7700 1 --2- 172.21.15.16:0/347750990 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f1dd806b020 0x7f1dd806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:21.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.525+0000 7f1df9fe7700 1 --2- 172.21.15.16:0/347750990 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f1dd806b020 0x7f1dd806d4e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1de4000c00 tx=0x7f1de4008030 comp rx=0 tx=0).ready entity=mgr.11190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:21.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.529+0000 7f1dea7fc700 1 -- 172.21.15.16:0/347750990 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1df0053b80 con 0x7f1df406e570 2023-10-16T13:46:21.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.734+0000 7f1dfc24b700 1 -- 172.21.15.16:0/347750990 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1df40b6370 con 0x7f1df406e570 2023-10-16T13:46:21.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.734+0000 7f1dea7fc700 1 -- 172.21.15.16:0/347750990 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v221) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1df00571a0 con 0x7f1df406e570 2023-10-16T13:46:21.737 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T13:46:21.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.736+0000 7f1dfc24b700 1 -- 172.21.15.16:0/347750990 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f1dd806b020 msgr2=0x7f1dd806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:21.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.736+0000 7f1dfc24b700 1 --2- 172.21.15.16:0/347750990 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f1dd806b020 0x7f1dd806d4e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1de4000c00 tx=0x7f1de4008030 comp rx=0 tx=0).stop 2023-10-16T13:46:21.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.736+0000 7f1dfc24b700 1 -- 172.21.15.16:0/347750990 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1df406e570 msgr2=0x7f1df4160bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:21.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.736+0000 7f1dfc24b700 1 --2- 172.21.15.16:0/347750990 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1df406e570 0x7f1df4160bf0 secure :-1 s=READY pgs=2437 cs=0 l=1 rev1=1 crypto rx=0x7f1df0001eb0 tx=0x7f1df000c860 comp rx=0 tx=0).stop 2023-10-16T13:46:21.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.736+0000 7f1dfc24b700 1 -- 172.21.15.16:0/347750990 shutdown_connections 2023-10-16T13:46:21.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.736+0000 7f1dfc24b700 1 --2- 172.21.15.16:0/347750990 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1df415b2c0 0x7f1df40cbef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:21.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.736+0000 7f1dfc24b700 1 --2- 172.21.15.16:0/347750990 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1df406a7e0 0x7f1df414ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:21.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.736+0000 7f1dfc24b700 1 --2- 172.21.15.16:0/347750990 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f1dd806b020 0x7f1dd806d4e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:21.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.736+0000 7f1dfc24b700 1 --2- 172.21.15.16:0/347750990 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1df406e570 0x7f1df4160bf0 unknown :-1 s=CLOSED pgs=2437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:21.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.736+0000 7f1dfc24b700 1 -- 172.21.15.16:0/347750990 >> 172.21.15.16:0/347750990 conn(0x7f1df40b1f40 msgr2=0x7f1df415b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:21.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.736+0000 7f1dfc24b700 1 -- 172.21.15.16:0/347750990 shutdown_connections 2023-10-16T13:46:21.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.736+0000 7f1dfc24b700 1 -- 172.21.15.16:0/347750990 wait complete. 2023-10-16T13:46:21.751 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-16T13:46:21.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.874+0000 7f2afe11e700 1 Processor -- start 2023-10-16T13:46:21.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.874+0000 7f2afe11e700 1 -- start start 2023-10-16T13:46:21.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.875+0000 7f2afe11e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2af815a5c0 0x7f2af80cef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:21.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.875+0000 7f2afe11e700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f2af80ce010 0x7f2af80d03d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:21.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.875+0000 7f2afe11e700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2af8072a50 con 0x7f2af80d0910 2023-10-16T13:46:21.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.875+0000 7f2afe11e700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2af80cf440 con 0x7f2af815a5c0 2023-10-16T13:46:21.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.875+0000 7f2afe11e700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2af80d0da0 con 0x7f2af80ce010 2023-10-16T13:46:21.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.875+0000 7f2af77fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2af815a5c0 0x7f2af80cef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:21.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.875+0000 7f2af77fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2af815a5c0 0x7f2af80cef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53048/0 (socket says 172.21.15.16:53048) 2023-10-16T13:46:21.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.875+0000 7f2af77fe700 1 -- 172.21.15.16:0/2220982944 learned_addr learned my addr 172.21.15.16:0/2220982944 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:21.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.875+0000 7f2af7fff700 1 --1- 172.21.15.16:0/2220982944 >> v1:172.21.15.16:6789/0 conn(0x7f2af80d0910 0x7f2af815e210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42324/0 (socket says 172.21.15.16:42324) 2023-10-16T13:46:21.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.875+0000 7f2af6ffd700 1 --2- 172.21.15.16:0/2220982944 >> v2:172.21.15.49:3300/0 conn(0x7f2af80ce010 0x7f2af80d03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:21.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.875+0000 7f2af77fe700 1 -- 172.21.15.16:0/2220982944 >> v2:172.21.15.49:3300/0 conn(0x7f2af80ce010 msgr2=0x7f2af80d03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:21.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.875+0000 7f2af77fe700 1 --2- 172.21.15.16:0/2220982944 >> v2:172.21.15.49:3300/0 conn(0x7f2af80ce010 0x7f2af80d03d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:21.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.875+0000 7f2af77fe700 1 -- 172.21.15.16:0/2220982944 >> v1:172.21.15.16:6789/0 conn(0x7f2af80d0910 legacy=0x7f2af815e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:21.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.875+0000 7f2af67fc700 1 -- 172.21.15.16:0/2220982944 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4052617681 0 0) 0x7f2af8072a50 con 0x7f2af80d0910 2023-10-16T13:46:21.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.875+0000 7f2af77fe700 1 -- 172.21.15.16:0/2220982944 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2af80cba30 con 0x7f2af815a5c0 2023-10-16T13:46:21.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.875+0000 7f2af77fe700 1 --2- 172.21.15.16:0/2220982944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2af815a5c0 0x7f2af80cef00 secure :-1 s=READY pgs=2439 cs=0 l=1 rev1=1 crypto rx=0x7f2af8067660 tx=0x7f2ae8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17a99ffe23e7d4fe server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:21.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.876+0000 7f2af67fc700 1 -- 172.21.15.16:0/2220982944 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ae800b870 con 0x7f2af815a5c0 2023-10-16T13:46:21.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.876+0000 7f2af67fc700 1 -- 172.21.15.16:0/2220982944 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ae800b9d0 con 0x7f2af815a5c0 2023-10-16T13:46:21.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.876+0000 7f2af67fc700 1 -- 172.21.15.16:0/2220982944 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ae800bd60 con 0x7f2af815a5c0 2023-10-16T13:46:21.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.876+0000 7f2afe11e700 1 -- 172.21.15.16:0/2220982944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2af815a5c0 msgr2=0x7f2af80cef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:21.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.876+0000 7f2afe11e700 1 --2- 172.21.15.16:0/2220982944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2af815a5c0 0x7f2af80cef00 secure :-1 s=READY pgs=2439 cs=0 l=1 rev1=1 crypto rx=0x7f2af8067660 tx=0x7f2ae8009bd0 comp rx=0 tx=0).stop 2023-10-16T13:46:21.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.876+0000 7f2afe11e700 1 -- 172.21.15.16:0/2220982944 shutdown_connections 2023-10-16T13:46:21.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.876+0000 7f2afe11e700 1 --2- 172.21.15.16:0/2220982944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2af815a5c0 0x7f2af80cef00 unknown :-1 s=CLOSED pgs=2439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:21.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.876+0000 7f2afe11e700 1 --2- 172.21.15.16:0/2220982944 >> v2:172.21.15.49:3300/0 conn(0x7f2af80ce010 0x7f2af80d03d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:21.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.876+0000 7f2afe11e700 1 -- 172.21.15.16:0/2220982944 >> 172.21.15.16:0/2220982944 conn(0x7f2af80b1fd0 msgr2=0x7f2af80b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:21.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.876+0000 7f2afe11e700 1 -- 172.21.15.16:0/2220982944 shutdown_connections 2023-10-16T13:46:21.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.876+0000 7f2afe11e700 1 -- 172.21.15.16:0/2220982944 wait complete. 2023-10-16T13:46:21.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.877+0000 7f2afe11e700 1 Processor -- start 2023-10-16T13:46:21.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.878+0000 7f2afe11e700 1 -- start start 2023-10-16T13:46:21.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.878+0000 7f2afe11e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2af80ce010 0x7f2af8159a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:21.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.878+0000 7f2afe11e700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2af80d0910 0x7f2af80c9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:21.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.878+0000 7f2afe11e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2af815a5c0 0x7f2af815e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:21.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.878+0000 7f2afe11e700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2af8164500 con 0x7f2af80ce010 2023-10-16T13:46:21.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.878+0000 7f2afe11e700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2af8163880 con 0x7f2af80d0910 2023-10-16T13:46:21.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.878+0000 7f2afe11e700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2af8163b60 con 0x7f2af815a5c0 2023-10-16T13:46:21.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.879+0000 7f2af77fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2af80ce010 0x7f2af8159a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:21.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.879+0000 7f2af77fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2af80ce010 0x7f2af8159a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53054/0 (socket says 172.21.15.16:53054) 2023-10-16T13:46:21.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.879+0000 7f2af77fe700 1 -- 172.21.15.16:0/1554729797 learned_addr learned my addr 172.21.15.16:0/1554729797 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:21.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.879+0000 7f2af6ffd700 1 --2- 172.21.15.16:0/1554729797 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2af80d0910 0x7f2af80c9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:21.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.879+0000 7f2af7fff700 1 --2- 172.21.15.16:0/1554729797 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2af815a5c0 0x7f2af815e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:21.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.879+0000 7f2af77fe700 1 -- 172.21.15.16:0/1554729797 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2af815a5c0 msgr2=0x7f2af815e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:21.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.879+0000 7f2af77fe700 1 --2- 172.21.15.16:0/1554729797 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2af815a5c0 0x7f2af815e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:21.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.879+0000 7f2af77fe700 1 -- 172.21.15.16:0/1554729797 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2af80d0910 msgr2=0x7f2af80c9fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:21.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.879+0000 7f2af77fe700 1 --2- 172.21.15.16:0/1554729797 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2af80d0910 0x7f2af80c9fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:21.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.879+0000 7f2af77fe700 1 -- 172.21.15.16:0/1554729797 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ae8009550 con 0x7f2af80ce010 2023-10-16T13:46:21.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.879+0000 7f2af6ffd700 1 --2- 172.21.15.16:0/1554729797 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2af80d0910 0x7f2af80c9fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:46:21.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.879+0000 7f2af77fe700 1 --2- 172.21.15.16:0/1554729797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2af80ce010 0x7f2af8159a90 secure :-1 s=READY pgs=2440 cs=0 l=1 rev1=1 crypto rx=0x7f2ae8009c10 tx=0x7f2ae8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:21.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.879+0000 7f2adffff700 1 -- 172.21.15.16:0/1554729797 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ae8004310 con 0x7f2af80ce010 2023-10-16T13:46:21.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.879+0000 7f2adffff700 1 -- 172.21.15.16:0/1554729797 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ae801f070 con 0x7f2af80ce010 2023-10-16T13:46:21.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.879+0000 7f2afe11e700 1 -- 172.21.15.16:0/1554729797 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2af8163e40 con 0x7f2af80ce010 2023-10-16T13:46:21.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.879+0000 7f2adffff700 1 -- 172.21.15.16:0/1554729797 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ae80036a0 con 0x7f2af80ce010 2023-10-16T13:46:21.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.879+0000 7f2afe11e700 1 -- 172.21.15.16:0/1554729797 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2af8165100 con 0x7f2af80ce010 2023-10-16T13:46:21.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.880+0000 7f2afe11e700 1 -- 172.21.15.16:0/1554729797 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2af8165290 con 0x7f2af80ce010 2023-10-16T13:46:21.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.881+0000 7f2adffff700 1 -- 172.21.15.16:0/1554729797 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 138) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2ae8021030 con 0x7f2af80ce010 2023-10-16T13:46:21.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.881+0000 7f2adffff700 1 --2- 172.21.15.16:0/1554729797 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f2ae406b050 0x7f2ae406d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:21.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.881+0000 7f2adffff700 1 -- 172.21.15.16:0/1554729797 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(221..221 src has 1..221) v4 ==== 5305+0+0 (secure 0 0 0) 0x7f2ae808a880 con 0x7f2af80ce010 2023-10-16T13:46:21.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.881+0000 7f2af6ffd700 1 --2- 172.21.15.16:0/1554729797 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f2ae406b050 0x7f2ae406d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:21.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.881+0000 7f2af6ffd700 1 --2- 172.21.15.16:0/1554729797 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f2ae406b050 0x7f2ae406d510 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2af8166630 tx=0x7f2aec008040 comp rx=0 tx=0).ready entity=mgr.11190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:21.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:21.886+0000 7f2adffff700 1 -- 172.21.15.16:0/1554729797 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2ae8055a40 con 0x7f2af80ce010 2023-10-16T13:46:22.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.093+0000 7f2afe11e700 1 -- 172.21.15.16:0/1554729797 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f2af804ebb0 con 0x7f2af80ce010 2023-10-16T13:46:22.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.093+0000 7f2adffff700 1 -- 172.21.15.16:0/1554729797 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v221) v1 ==== 84+0+65 (secure 0 0 0) 0x7f2ae8015020 con 0x7f2af80ce010 2023-10-16T13:46:22.096 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:46:22.097 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2023-10-16T13:46:22.097 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:46:22.097 INFO:teuthology.orchestra.run.smithi016.stdout: "poolnum": 1, 2023-10-16T13:46:22.097 INFO:teuthology.orchestra.run.smithi016.stdout: "poolname": ".mgr" 2023-10-16T13:46:22.097 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:46:22.097 INFO:teuthology.orchestra.run.smithi016.stdout:] 2023-10-16T13:46:22.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.096+0000 7f2afe11e700 1 -- 172.21.15.16:0/1554729797 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f2ae406b050 msgr2=0x7f2ae406d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:22.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.096+0000 7f2afe11e700 1 --2- 172.21.15.16:0/1554729797 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f2ae406b050 0x7f2ae406d510 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2af8166630 tx=0x7f2aec008040 comp rx=0 tx=0).stop 2023-10-16T13:46:22.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.096+0000 7f2afe11e700 1 -- 172.21.15.16:0/1554729797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2af80ce010 msgr2=0x7f2af8159a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:22.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.096+0000 7f2afe11e700 1 --2- 172.21.15.16:0/1554729797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2af80ce010 0x7f2af8159a90 secure :-1 s=READY pgs=2440 cs=0 l=1 rev1=1 crypto rx=0x7f2ae8009c10 tx=0x7f2ae8004750 comp rx=0 tx=0).stop 2023-10-16T13:46:22.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.096+0000 7f2afe11e700 1 -- 172.21.15.16:0/1554729797 shutdown_connections 2023-10-16T13:46:22.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.096+0000 7f2afe11e700 1 --2- 172.21.15.16:0/1554729797 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2af815a5c0 0x7f2af815e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:22.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.096+0000 7f2afe11e700 1 --2- 172.21.15.16:0/1554729797 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f2ae406b050 0x7f2ae406d510 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:22.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.096+0000 7f2afe11e700 1 --2- 172.21.15.16:0/1554729797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2af80ce010 0x7f2af8159a90 unknown :-1 s=CLOSED pgs=2440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:22.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.096+0000 7f2afe11e700 1 --2- 172.21.15.16:0/1554729797 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2af80d0910 0x7f2af80c9fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:22.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.096+0000 7f2afe11e700 1 -- 172.21.15.16:0/1554729797 >> 172.21.15.16:0/1554729797 conn(0x7f2af80b1fd0 msgr2=0x7f2af8165c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:22.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.096+0000 7f2afe11e700 1 -- 172.21.15.16:0/1554729797 shutdown_connections 2023-10-16T13:46:22.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.096+0000 7f2afe11e700 1 -- 172.21.15.16:0/1554729797 wait complete. 2023-10-16T13:46:22.111 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-16T13:46:22.111 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-16T13:46:22.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.237+0000 7ff2d060e700 1 Processor -- start 2023-10-16T13:46:22.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.237+0000 7ff2d060e700 1 -- start start 2023-10-16T13:46:22.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.238+0000 7ff2d060e700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7ff2c815d920 0x7ff2c80cdef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:22.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.238+0000 7ff2d060e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff2c80726b0 0x7ff2c806a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:22.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.238+0000 7ff2d060e700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2c80ceb90 con 0x7ff2c806ad20 2023-10-16T13:46:22.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.238+0000 7ff2d060e700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2c815dd00 con 0x7ff2c815d920 2023-10-16T13:46:22.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.238+0000 7ff2d060e700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c806b1b0 con 0x7ff2c80726b0 2023-10-16T13:46:22.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.239+0000 7ff2ce3aa700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7ff2c815d920 0x7ff2c80cdef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:22.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.239+0000 7ff2cdba9700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff2c80726b0 0x7ff2c806a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:22.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.239+0000 7ff2cebab700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7ff2c806ad20 0x7ff2c80ccd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:49522/0 (socket says 172.21.15.16:49522) 2023-10-16T13:46:22.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.239+0000 7ff2cebab700 1 -- 172.21.15.16:0/2448230917 learned_addr learned my addr 172.21.15.16:0/2448230917 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:46:22.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.239+0000 7ff2cd3a8700 1 -- 172.21.15.16:0/2448230917 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2448104226 0 0) 0x7ff2c80ceb90 con 0x7ff2c806ad20 2023-10-16T13:46:22.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.239+0000 7ff2cd3a8700 1 -- 172.21.15.16:0/2448230917 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2b40035b0 con 0x7ff2c806ad20 2023-10-16T13:46:22.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.239+0000 7ff2ce3aa700 1 -- 172.21.15.16:0/2448230917 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff2c80726b0 msgr2=0x7ff2c806a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:22.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.239+0000 7ff2ce3aa700 1 --2- 172.21.15.16:0/2448230917 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff2c80726b0 0x7ff2c806a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:22.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.239+0000 7ff2ce3aa700 1 -- 172.21.15.16:0/2448230917 >> v1:172.21.15.49:6789/0 conn(0x7ff2c806ad20 legacy=0x7ff2c80ccd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:22.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.239+0000 7ff2ce3aa700 1 -- 172.21.15.16:0/2448230917 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2c80ce5d0 con 0x7ff2c815d920 2023-10-16T13:46:22.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.239+0000 7ff2cdba9700 1 --2- 172.21.15.16:0/2448230917 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff2c80726b0 0x7ff2c806a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:46:22.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.239+0000 7ff2ce3aa700 1 --2- 172.21.15.16:0/2448230917 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff2c815d920 0x7ff2c80cdef0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7ff2b8009840 tx=0x7ff2b8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=811126ce95d33320 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:22.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.240+0000 7ff2cd3a8700 1 -- 172.21.15.16:0/2448230917 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2b8011830 con 0x7ff2c815d920 2023-10-16T13:46:22.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.240+0000 7ff2cd3a8700 1 -- 172.21.15.16:0/2448230917 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2b8011990 con 0x7ff2c815d920 2023-10-16T13:46:22.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.240+0000 7ff2cd3a8700 1 -- 172.21.15.16:0/2448230917 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2b8011d20 con 0x7ff2c815d920 2023-10-16T13:46:22.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.240+0000 7ff2d060e700 1 -- 172.21.15.16:0/2448230917 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff2c815d920 msgr2=0x7ff2c80cdef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:22.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.240+0000 7ff2d060e700 1 --2- 172.21.15.16:0/2448230917 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff2c815d920 0x7ff2c80cdef0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7ff2b8009840 tx=0x7ff2b8009b50 comp rx=0 tx=0).stop 2023-10-16T13:46:22.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.240+0000 7ff2d060e700 1 -- 172.21.15.16:0/2448230917 shutdown_connections 2023-10-16T13:46:22.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.240+0000 7ff2d060e700 1 --2- 172.21.15.16:0/2448230917 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff2c815d920 0x7ff2c80cdef0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:22.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.240+0000 7ff2d060e700 1 --2- 172.21.15.16:0/2448230917 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff2c80726b0 0x7ff2c806a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:22.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.240+0000 7ff2d060e700 1 -- 172.21.15.16:0/2448230917 >> 172.21.15.16:0/2448230917 conn(0x7ff2c80b1ed0 msgr2=0x7ff2c80b22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:22.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.240+0000 7ff2d060e700 1 -- 172.21.15.16:0/2448230917 shutdown_connections 2023-10-16T13:46:22.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.240+0000 7ff2d060e700 1 -- 172.21.15.16:0/2448230917 wait complete. 2023-10-16T13:46:22.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.241+0000 7ff2d060e700 1 Processor -- start 2023-10-16T13:46:22.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.241+0000 7ff2d060e700 1 -- start start 2023-10-16T13:46:22.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.242+0000 7ff2d060e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff2c806ad20 0x7ff2c815ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:22.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.242+0000 7ff2d060e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2c80726b0 0x7ff2c80c4250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:22.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.242+0000 7ff2d060e700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff2c815d920 0x7ff2c80c4790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:22.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.242+0000 7ff2d060e700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c81e6dd0 con 0x7ff2c80726b0 2023-10-16T13:46:22.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.242+0000 7ff2d060e700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c80b17b0 con 0x7ff2c815d920 2023-10-16T13:46:22.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.242+0000 7ff2d060e700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c81e6430 con 0x7ff2c806ad20 2023-10-16T13:46:22.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.242+0000 7ff2cdba9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2c80726b0 0x7ff2c80c4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:22.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.242+0000 7ff2cdba9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2c80726b0 0x7ff2c80c4250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53070/0 (socket says 172.21.15.16:53070) 2023-10-16T13:46:22.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.242+0000 7ff2cdba9700 1 -- 172.21.15.16:0/3394765096 learned_addr learned my addr 172.21.15.16:0/3394765096 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:22.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.242+0000 7ff2cebab700 1 --2- 172.21.15.16:0/3394765096 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff2c815d920 0x7ff2c80c4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:22.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.242+0000 7ff2ce3aa700 1 --2- 172.21.15.16:0/3394765096 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff2c806ad20 0x7ff2c815ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:22.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.243+0000 7ff2cdba9700 1 -- 172.21.15.16:0/3394765096 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff2c806ad20 msgr2=0x7ff2c815ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:22.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.243+0000 7ff2cdba9700 1 --2- 172.21.15.16:0/3394765096 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff2c806ad20 0x7ff2c815ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:22.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.243+0000 7ff2cdba9700 1 -- 172.21.15.16:0/3394765096 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff2c815d920 msgr2=0x7ff2c80c4790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:22.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.243+0000 7ff2cdba9700 1 --2- 172.21.15.16:0/3394765096 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff2c815d920 0x7ff2c80c4790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:22.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.243+0000 7ff2cdba9700 1 -- 172.21.15.16:0/3394765096 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2c40094a0 con 0x7ff2c80726b0 2023-10-16T13:46:22.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.243+0000 7ff2cdba9700 1 --2- 172.21.15.16:0/3394765096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2c80726b0 0x7ff2c80c4250 secure :-1 s=READY pgs=2441 cs=0 l=1 rev1=1 crypto rx=0x7ff2c400ebe0 tx=0x7ff2c400eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:22.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.243+0000 7ff2beffd700 1 -- 172.21.15.16:0/3394765096 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2c400c850 con 0x7ff2c80726b0 2023-10-16T13:46:22.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.243+0000 7ff2beffd700 1 -- 172.21.15.16:0/3394765096 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2c400c9b0 con 0x7ff2c80726b0 2023-10-16T13:46:22.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.243+0000 7ff2beffd700 1 -- 172.21.15.16:0/3394765096 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2c400cd40 con 0x7ff2c80726b0 2023-10-16T13:46:22.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.243+0000 7ff2d060e700 1 -- 172.21.15.16:0/3394765096 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2b80094a0 con 0x7ff2c80726b0 2023-10-16T13:46:22.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.243+0000 7ff2d060e700 1 -- 172.21.15.16:0/3394765096 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2c81e69f0 con 0x7ff2c80726b0 2023-10-16T13:46:22.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.244+0000 7ff2d060e700 1 -- 172.21.15.16:0/3394765096 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2c804ec20 con 0x7ff2c80726b0 2023-10-16T13:46:22.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.244+0000 7ff2beffd700 1 -- 172.21.15.16:0/3394765096 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 138) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff2c4013470 con 0x7ff2c80726b0 2023-10-16T13:46:22.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.245+0000 7ff2beffd700 1 --2- 172.21.15.16:0/3394765096 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7ff2ac06b020 0x7ff2ac06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:22.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.245+0000 7ff2ce3aa700 1 --2- 172.21.15.16:0/3394765096 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7ff2ac06b020 0x7ff2ac06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:22.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.245+0000 7ff2beffd700 1 -- 172.21.15.16:0/3394765096 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(221..221 src has 1..221) v4 ==== 5305+0+0 (secure 0 0 0) 0x7ff2c4089640 con 0x7ff2c80726b0 2023-10-16T13:46:22.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.245+0000 7ff2ce3aa700 1 --2- 172.21.15.16:0/3394765096 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7ff2ac06b020 0x7ff2ac06d4e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff2b8000c00 tx=0x7ff2b801c040 comp rx=0 tx=0).ready entity=mgr.11190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:22.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.249+0000 7ff2beffd700 1 -- 172.21.15.16:0/3394765096 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff2c4054770 con 0x7ff2c80726b0 2023-10-16T13:46:22.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:22.455+0000 7ff2d060e700 1 -- 172.21.15.16:0/3394765096 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7ff2c80ced40 con 0x7ff2c80726b0 2023-10-16T13:46:23.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:23.317+0000 7ff2beffd700 1 -- 172.21.15.16:0/3394765096 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v222) v1 ==== 138+0+0 (secure 0 0 0) 0x7ff2c400cea0 con 0x7ff2c80726b0 2023-10-16T13:46:23.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:23.322+0000 7ff2d060e700 1 -- 172.21.15.16:0/3394765096 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7ff2ac06b020 msgr2=0x7ff2ac06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:23.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:23.322+0000 7ff2d060e700 1 --2- 172.21.15.16:0/3394765096 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7ff2ac06b020 0x7ff2ac06d4e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff2b8000c00 tx=0x7ff2b801c040 comp rx=0 tx=0).stop 2023-10-16T13:46:23.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:23.322+0000 7ff2d060e700 1 -- 172.21.15.16:0/3394765096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2c80726b0 msgr2=0x7ff2c80c4250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:23.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:23.322+0000 7ff2d060e700 1 --2- 172.21.15.16:0/3394765096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2c80726b0 0x7ff2c80c4250 secure :-1 s=READY pgs=2441 cs=0 l=1 rev1=1 crypto rx=0x7ff2c400ebe0 tx=0x7ff2c400eef0 comp rx=0 tx=0).stop 2023-10-16T13:46:23.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:23.322+0000 7ff2d060e700 1 -- 172.21.15.16:0/3394765096 shutdown_connections 2023-10-16T13:46:23.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:23.322+0000 7ff2d060e700 1 --2- 172.21.15.16:0/3394765096 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff2c815d920 0x7ff2c80c4790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:23.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:23.323+0000 7ff2d060e700 1 --2- 172.21.15.16:0/3394765096 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff2c806ad20 0x7ff2c815ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:23.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:23.323+0000 7ff2d060e700 1 --2- 172.21.15.16:0/3394765096 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7ff2ac06b020 0x7ff2ac06d4e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:23.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:23.323+0000 7ff2d060e700 1 --2- 172.21.15.16:0/3394765096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2c80726b0 0x7ff2c80c4250 unknown :-1 s=CLOSED pgs=2441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:23.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:23.323+0000 7ff2d060e700 1 -- 172.21.15.16:0/3394765096 >> 172.21.15.16:0/3394765096 conn(0x7ff2c80b1ed0 msgr2=0x7ff2c815c800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:23.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:23.324+0000 7ff2d060e700 1 -- 172.21.15.16:0/3394765096 shutdown_connections 2023-10-16T13:46:23.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:23.324+0000 7ff2d060e700 1 -- 172.21.15.16:0/3394765096 wait complete. 2023-10-16T13:46:23.328 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' created 2023-10-16T13:46:23.343 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-16T13:46:23.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:23.479+0000 7fcc28f90700 1 Processor -- start 2023-10-16T13:46:23.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:23.479+0000 7fcc28f90700 1 -- start start 2023-10-16T13:46:23.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:23.480+0000 7fcc28f90700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fcc2415b070 0x7fcc2415b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:23.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:23.480+0000 7fcc28f90700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcc2406e170 0x7fcc240cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:23.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:23.480+0000 7fcc28f90700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc240cef70 con 0x7fcc240cb880 2023-10-16T13:46:23.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:23.480+0000 7fcc28f90700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc240ce360 con 0x7fcc2406e170 2023-10-16T13:46:23.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:23.480+0000 7fcc28f90700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc240ce640 con 0x7fcc2415b070 2023-10-16T13:46:23.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:23.480+0000 7fcc22d9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcc2406e170 0x7fcc240cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:23.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:23.480+0000 7fcc21d9b700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fcc240cb880 0x7fcc2406da60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42350/0 (socket says 172.21.15.16:42350) 2023-10-16T13:46:23.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:23.480+0000 7fcc21d9b700 1 -- 172.21.15.16:0/2072491701 learned_addr learned my addr 172.21.15.16:0/2072491701 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:46:23.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:23.480+0000 7fcc2259c700 1 --2- 172.21.15.16:0/2072491701 >> v2:172.21.15.49:3300/0 conn(0x7fcc2415b070 0x7fcc2415b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:23.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:23.481+0000 7fcc22d9d700 1 -- 172.21.15.16:0/2072491701 >> v2:172.21.15.49:3300/0 conn(0x7fcc2415b070 msgr2=0x7fcc2415b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:23.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:23.481+0000 7fcc22d9d700 1 --2- 172.21.15.16:0/2072491701 >> v2:172.21.15.49:3300/0 conn(0x7fcc2415b070 0x7fcc2415b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:23.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:23.481+0000 7fcc22d9d700 1 -- 172.21.15.16:0/2072491701 >> v1:172.21.15.16:6789/0 conn(0x7fcc240cb880 legacy=0x7fcc2406da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:23.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:23.481+0000 7fcc2159a700 1 -- 172.21.15.16:0/2072491701 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1755561828 0 0) 0x7fcc240cef70 con 0x7fcc240cb880 2023-10-16T13:46:23.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:23.481+0000 7fcc22d9d700 1 -- 172.21.15.16:0/2072491701 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc240ce920 con 0x7fcc2406e170 2023-10-16T13:46:23.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:23.481+0000 7fcc22d9d700 1 --2- 172.21.15.16:0/2072491701 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc2406e170 0x7fcc240cbd00 secure :-1 s=READY pgs=2443 cs=0 l=1 rev1=1 crypto rx=0x7fcc240705a0 tx=0x7fcc1c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cfc4c533bfe9316 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:23.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:23.481+0000 7fcc2159a700 1 -- 172.21.15.16:0/2072491701 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc1c00e070 con 0x7fcc2406e170 2023-10-16T13:46:23.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:23.481+0000 7fcc2159a700 1 -- 172.21.15.16:0/2072491701 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc1c00bbd0 con 0x7fcc2406e170 2023-10-16T13:46:23.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:23.481+0000 7fcc2159a700 1 -- 172.21.15.16:0/2072491701 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc1c012430 con 0x7fcc2406e170 2023-10-16T13:46:23.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:23.482+0000 7fcc28f90700 1 -- 172.21.15.16:0/2072491701 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc2406e170 msgr2=0x7fcc240cbd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:23.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:23.482+0000 7fcc28f90700 1 --2- 172.21.15.16:0/2072491701 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc2406e170 0x7fcc240cbd00 secure :-1 s=READY pgs=2443 cs=0 l=1 rev1=1 crypto rx=0x7fcc240705a0 tx=0x7fcc1c00b330 comp rx=0 tx=0).stop 2023-10-16T13:46:23.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:23.482+0000 7fcc28f90700 1 -- 172.21.15.16:0/2072491701 shutdown_connections 2023-10-16T13:46:23.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:23.482+0000 7fcc28f90700 1 --2- 172.21.15.16:0/2072491701 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc2406e170 0x7fcc240cbd00 unknown :-1 s=CLOSED pgs=2443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:23.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:23.482+0000 7fcc28f90700 1 --2- 172.21.15.16:0/2072491701 >> v2:172.21.15.49:3300/0 conn(0x7fcc2415b070 0x7fcc2415b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:23.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:23.482+0000 7fcc28f90700 1 -- 172.21.15.16:0/2072491701 >> 172.21.15.16:0/2072491701 conn(0x7fcc240b1f40 msgr2=0x7fcc240b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:23.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:23.482+0000 7fcc28f90700 1 -- 172.21.15.16:0/2072491701 shutdown_connections 2023-10-16T13:46:23.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:23.482+0000 7fcc28f90700 1 -- 172.21.15.16:0/2072491701 wait complete. 2023-10-16T13:46:23.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:23.483+0000 7fcc28f90700 1 Processor -- start 2023-10-16T13:46:23.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:23.483+0000 7fcc28f90700 1 -- start start 2023-10-16T13:46:23.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:23.484+0000 7fcc28f90700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc2406e170 0x7fcc240b4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:23.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:23.484+0000 7fcc28f90700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcc240cb880 0x7fcc240b4b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:23.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:23.484+0000 7fcc28f90700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcc2415b070 0x7fcc240ad530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:23.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:23.484+0000 7fcc28f90700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc241f3d80 con 0x7fcc2406e170 2023-10-16T13:46:23.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:23.484+0000 7fcc28f90700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc240aca50 con 0x7fcc240cb880 2023-10-16T13:46:23.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:23.484+0000 7fcc28f90700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc240acd30 con 0x7fcc2415b070 2023-10-16T13:46:23.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:23.484+0000 7fcc2259c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc2406e170 0x7fcc240b4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:23.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:23.484+0000 7fcc2259c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc2406e170 0x7fcc240b4610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53082/0 (socket says 172.21.15.16:53082) 2023-10-16T13:46:23.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:23.484+0000 7fcc2259c700 1 -- 172.21.15.16:0/2462213784 learned_addr learned my addr 172.21.15.16:0/2462213784 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:23.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:23.484+0000 7fcc22d9d700 1 --2- 172.21.15.16:0/2462213784 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcc2415b070 0x7fcc240ad530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:23.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:23.484+0000 7fcc21d9b700 1 --2- 172.21.15.16:0/2462213784 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcc240cb880 0x7fcc240b4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:23.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:23.484+0000 7fcc2259c700 1 -- 172.21.15.16:0/2462213784 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcc2415b070 msgr2=0x7fcc240ad530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:23.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:23.484+0000 7fcc2259c700 1 --2- 172.21.15.16:0/2462213784 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcc2415b070 0x7fcc240ad530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:23.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:23.484+0000 7fcc2259c700 1 -- 172.21.15.16:0/2462213784 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcc240cb880 msgr2=0x7fcc240b4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:23.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:23.484+0000 7fcc2259c700 1 --2- 172.21.15.16:0/2462213784 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcc240cb880 0x7fcc240b4b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:23.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:23.484+0000 7fcc2259c700 1 -- 172.21.15.16:0/2462213784 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc1c009a70 con 0x7fcc2406e170 2023-10-16T13:46:23.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:23.484+0000 7fcc2259c700 1 --2- 172.21.15.16:0/2462213784 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc2406e170 0x7fcc240b4610 secure :-1 s=READY pgs=2444 cs=0 l=1 rev1=1 crypto rx=0x7fcc1800d800 tx=0x7fcc1800db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:23.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:23.485+0000 7fcc12ffd700 1 -- 172.21.15.16:0/2462213784 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc180125f0 con 0x7fcc2406e170 2023-10-16T13:46:23.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:23.485+0000 7fcc12ffd700 1 -- 172.21.15.16:0/2462213784 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc18012750 con 0x7fcc2406e170 2023-10-16T13:46:23.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:23.485+0000 7fcc12ffd700 1 -- 172.21.15.16:0/2462213784 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc18012b00 con 0x7fcc2406e170 2023-10-16T13:46:23.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:23.485+0000 7fcc28f90700 1 -- 172.21.15.16:0/2462213784 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc240ad010 con 0x7fcc2406e170 2023-10-16T13:46:23.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:23.485+0000 7fcc28f90700 1 -- 172.21.15.16:0/2462213784 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc240adc50 con 0x7fcc2406e170 2023-10-16T13:46:23.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:23.486+0000 7fcc28f90700 1 -- 172.21.15.16:0/2462213784 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc240ad380 con 0x7fcc2406e170 2023-10-16T13:46:23.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:23.486+0000 7fcc12ffd700 1 -- 172.21.15.16:0/2462213784 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 138) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fcc18012c60 con 0x7fcc2406e170 2023-10-16T13:46:23.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:23.487+0000 7fcc12ffd700 1 --2- 172.21.15.16:0/2462213784 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fcc0406b020 0x7fcc0406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:23.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:23.487+0000 7fcc12ffd700 1 -- 172.21.15.16:0/2462213784 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(222..222 src has 1..222) v4 ==== 5675+0+0 (secure 0 0 0) 0x7fcc1808a200 con 0x7fcc2406e170 2023-10-16T13:46:23.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:23.487+0000 7fcc21d9b700 1 --2- 172.21.15.16:0/2462213784 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fcc0406b020 0x7fcc0406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:23.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:23.487+0000 7fcc21d9b700 1 --2- 172.21.15.16:0/2462213784 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fcc0406b020 0x7fcc0406d4e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fcc2415e3b0 tx=0x7fcc14007de0 comp rx=0 tx=0).ready entity=mgr.11190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:23.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:23.491+0000 7fcc12ffd700 1 -- 172.21.15.16:0/2462213784 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcc180552a0 con 0x7fcc2406e170 2023-10-16T13:46:23.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:23.695+0000 7fcc28f90700 1 -- 172.21.15.16:0/2462213784 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fcc2404ebb0 con 0x7fcc2406e170 2023-10-16T13:46:24.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.320+0000 7fcc12ffd700 1 -- 172.21.15.16:0/2462213784 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v223) v1 ==== 170+0+0 (secure 0 0 0) 0x7fcc180588c0 con 0x7fcc2406e170 2023-10-16T13:46:24.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.325+0000 7fcc28f90700 1 -- 172.21.15.16:0/2462213784 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fcc0406b020 msgr2=0x7fcc0406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:24.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.325+0000 7fcc28f90700 1 --2- 172.21.15.16:0/2462213784 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fcc0406b020 0x7fcc0406d4e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fcc2415e3b0 tx=0x7fcc14007de0 comp rx=0 tx=0).stop 2023-10-16T13:46:24.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.325+0000 7fcc28f90700 1 -- 172.21.15.16:0/2462213784 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc2406e170 msgr2=0x7fcc240b4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:24.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.325+0000 7fcc28f90700 1 --2- 172.21.15.16:0/2462213784 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc2406e170 0x7fcc240b4610 secure :-1 s=READY pgs=2444 cs=0 l=1 rev1=1 crypto rx=0x7fcc1800d800 tx=0x7fcc1800db10 comp rx=0 tx=0).stop 2023-10-16T13:46:24.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.326+0000 7fcc28f90700 1 -- 172.21.15.16:0/2462213784 shutdown_connections 2023-10-16T13:46:24.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.326+0000 7fcc28f90700 1 --2- 172.21.15.16:0/2462213784 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcc2415b070 0x7fcc240ad530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:24.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.326+0000 7fcc28f90700 1 --2- 172.21.15.16:0/2462213784 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fcc0406b020 0x7fcc0406d4e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:24.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.326+0000 7fcc28f90700 1 --2- 172.21.15.16:0/2462213784 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc2406e170 0x7fcc240b4610 unknown :-1 s=CLOSED pgs=2444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:24.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.326+0000 7fcc28f90700 1 --2- 172.21.15.16:0/2462213784 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcc240cb880 0x7fcc240b4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:24.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.326+0000 7fcc28f90700 1 -- 172.21.15.16:0/2462213784 >> 172.21.15.16:0/2462213784 conn(0x7fcc240b1f40 msgr2=0x7fcc240cf400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:24.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.326+0000 7fcc28f90700 1 -- 172.21.15.16:0/2462213784 shutdown_connections 2023-10-16T13:46:24.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.326+0000 7fcc28f90700 1 -- 172.21.15.16:0/2462213784 wait complete. 2023-10-16T13:46:24.334 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' created 2023-10-16T13:46:24.355 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-16T13:46:24.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.486+0000 7f2193904700 1 Processor -- start 2023-10-16T13:46:24.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.487+0000 7f2193904700 1 -- start start 2023-10-16T13:46:24.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.487+0000 7f2193904700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f218c06e170 0x7f218c0cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:24.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.487+0000 7f2193904700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f218c0cef70 con 0x7f218c15b070 2023-10-16T13:46:24.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.487+0000 7f2193904700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f218c0ce360 con 0x7f218c0cb880 2023-10-16T13:46:24.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.487+0000 7f2193904700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f218c0ce640 con 0x7f218c06e170 2023-10-16T13:46:24.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.487+0000 7f21916a0700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f218c15b070 0x7f218c15b450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42364/0 (socket says 172.21.15.16:42364) 2023-10-16T13:46:24.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.487+0000 7f21916a0700 1 -- 172.21.15.16:0/1189571430 learned_addr learned my addr 172.21.15.16:0/1189571430 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:46:24.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.488+0000 7f2191ea1700 1 --2- 172.21.15.16:0/1189571430 >> v2:172.21.15.49:3300/0 conn(0x7f218c06e170 0x7f218c0cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:24.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.488+0000 7f2183fff700 1 -- 172.21.15.16:0/1189571430 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2977379935 0 0) 0x7f218c0cef70 con 0x7f218c15b070 2023-10-16T13:46:24.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.488+0000 7f2183fff700 1 -- 172.21.15.16:0/1189571430 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21780035b0 con 0x7f218c15b070 2023-10-16T13:46:24.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.488+0000 7f2183fff700 1 -- 172.21.15.16:0/1189571430 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3292055991 0 0) 0x7f218c0ce360 con 0x7f218c0cb880 2023-10-16T13:46:24.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.488+0000 7f2183fff700 1 -- 172.21.15.16:0/1189571430 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f218c0cef70 con 0x7f218c0cb880 2023-10-16T13:46:24.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.488+0000 7f2183fff700 1 -- 172.21.15.16:0/1189571430 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3910679120 0 0) 0x7f21780035b0 con 0x7f218c15b070 2023-10-16T13:46:24.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.488+0000 7f2183fff700 1 -- 172.21.15.16:0/1189571430 >> v2:172.21.15.49:3300/0 conn(0x7f218c06e170 msgr2=0x7f218c0cbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:24.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.488+0000 7f2183fff700 1 --2- 172.21.15.16:0/1189571430 >> v2:172.21.15.49:3300/0 conn(0x7f218c06e170 0x7f218c0cbd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:24.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.488+0000 7f2183fff700 1 -- 172.21.15.16:0/1189571430 >> v1:172.21.15.49:6789/0 conn(0x7f218c0cb880 legacy=0x7f218c06da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:24.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.488+0000 7f2183fff700 1 -- 172.21.15.16:0/1189571430 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f218c0ce920 con 0x7f218c15b070 2023-10-16T13:46:24.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.488+0000 7f2183fff700 1 -- 172.21.15.16:0/1189571430 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f217c001670 con 0x7f218c15b070 2023-10-16T13:46:24.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.488+0000 7f2191ea1700 1 --2- 172.21.15.16:0/1189571430 >> v2:172.21.15.49:3300/0 conn(0x7f218c06e170 0x7f218c0cbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:46:24.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.488+0000 7f2183fff700 1 -- 172.21.15.16:0/1189571430 >> v1:172.21.15.16:6789/0 conn(0x7f218c15b070 legacy=0x7f218c15b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:24.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.488+0000 7f2183fff700 1 --2- 172.21.15.16:0/1189571430 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2178003b90 0x7f2178003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:24.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.488+0000 7f2183fff700 1 --2- 172.21.15.16:0/1189571430 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2178004850 0x7f2178004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:24.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.488+0000 7f2183fff700 1 --2- 172.21.15.16:0/1189571430 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2178005210 0x7f217800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:24.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.488+0000 7f2183fff700 1 -- 172.21.15.16:0/1189571430 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f218c0ce640 con 0x7f2178005210 2023-10-16T13:46:24.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.488+0000 7f2183fff700 1 -- 172.21.15.16:0/1189571430 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f217800c140 con 0x7f2178003b90 2023-10-16T13:46:24.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.488+0000 7f2183fff700 1 -- 172.21.15.16:0/1189571430 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f217800c300 con 0x7f2178004850 2023-10-16T13:46:24.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.488+0000 7f2191ea1700 1 --2- 172.21.15.16:0/1189571430 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2178005210 0x7f217800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:24.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.488+0000 7f21916a0700 1 --2- 172.21.15.16:0/1189571430 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2178003b90 0x7f2178003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:24.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.488+0000 7f2190e9f700 1 --2- 172.21.15.16:0/1189571430 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2178004850 0x7f2178004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:24.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.489+0000 7f2191ea1700 1 -- 172.21.15.16:0/1189571430 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2178004850 msgr2=0x7f2178004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:24.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.489+0000 7f2191ea1700 1 --2- 172.21.15.16:0/1189571430 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2178004850 0x7f2178004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:24.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.489+0000 7f2191ea1700 1 -- 172.21.15.16:0/1189571430 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2178003b90 msgr2=0x7f2178003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:24.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.489+0000 7f2191ea1700 1 --2- 172.21.15.16:0/1189571430 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2178003b90 0x7f2178003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:24.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.489+0000 7f2191ea1700 1 -- 172.21.15.16:0/1189571430 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21780035b0 con 0x7f2178005210 2023-10-16T13:46:24.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.489+0000 7f21916a0700 1 --2- 172.21.15.16:0/1189571430 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2178003b90 0x7f2178003f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:46:24.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.489+0000 7f2191ea1700 1 -- 172.21.15.16:0/1189571430 reap_dead start 2023-10-16T13:46:24.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.489+0000 7f2191ea1700 1 --2- 172.21.15.16:0/1189571430 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2178005210 0x7f217800ba10 secure :-1 s=READY pgs=2446 cs=0 l=1 rev1=1 crypto rx=0x7f218c06d6e0 tx=0x7f2178005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aae5a88ceba5190 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:24.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.489+0000 7f2183fff700 1 -- 172.21.15.16:0/1189571430 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f218400ed90 con 0x7f2178005210 2023-10-16T13:46:24.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.489+0000 7f2183fff700 1 -- 172.21.15.16:0/1189571430 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f218400f040 con 0x7f2178005210 2023-10-16T13:46:24.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.489+0000 7f2183fff700 1 -- 172.21.15.16:0/1189571430 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21840146c0 con 0x7f2178005210 2023-10-16T13:46:24.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.490+0000 7f2193904700 1 -- 172.21.15.16:0/1189571430 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2178005210 msgr2=0x7f217800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:24.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.490+0000 7f2193904700 1 --2- 172.21.15.16:0/1189571430 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2178005210 0x7f217800ba10 secure :-1 s=READY pgs=2446 cs=0 l=1 rev1=1 crypto rx=0x7f218c06d6e0 tx=0x7f2178005880 comp rx=0 tx=0).stop 2023-10-16T13:46:24.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.490+0000 7f2193904700 1 -- 172.21.15.16:0/1189571430 shutdown_connections 2023-10-16T13:46:24.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.490+0000 7f2193904700 1 --2- 172.21.15.16:0/1189571430 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2178005210 0x7f217800ba10 unknown :-1 s=CLOSED pgs=2446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:24.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.490+0000 7f2193904700 1 -- 172.21.15.16:0/1189571430 >> 172.21.15.16:0/1189571430 conn(0x7f218c0b1f40 msgr2=0x7f218c0b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:24.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.490+0000 7f2193904700 1 -- 172.21.15.16:0/1189571430 shutdown_connections 2023-10-16T13:46:24.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.490+0000 7f2193904700 1 -- 172.21.15.16:0/1189571430 wait complete. 2023-10-16T13:46:24.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.491+0000 7f2193904700 1 Processor -- start 2023-10-16T13:46:24.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.491+0000 7f2193904700 1 -- start start 2023-10-16T13:46:24.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.492+0000 7f2193904700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2178005210 0x7f218c144550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:24.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.492+0000 7f2193904700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f218c0b2320 0x7f218c0b27a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:24.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.492+0000 7f2193904700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f218c0b5640 0x7f218c0b5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:24.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.492+0000 7f2193904700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f218c0b2ce0 con 0x7f218c0b2320 2023-10-16T13:46:24.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.492+0000 7f2193904700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f218c0b4880 con 0x7f2178005210 2023-10-16T13:46:24.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.492+0000 7f2193904700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f218c0b4b60 con 0x7f218c0b5640 2023-10-16T13:46:24.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.492+0000 7f2190e9f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f218c0b2320 0x7f218c0b27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:24.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.492+0000 7f2190e9f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f218c0b2320 0x7f218c0b27a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53104/0 (socket says 172.21.15.16:53104) 2023-10-16T13:46:24.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.492+0000 7f2190e9f700 1 -- 172.21.15.16:0/178462762 learned_addr learned my addr 172.21.15.16:0/178462762 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:24.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.492+0000 7f21916a0700 1 --2- 172.21.15.16:0/178462762 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2178005210 0x7f218c144550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:24.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.492+0000 7f2191ea1700 1 --2- 172.21.15.16:0/178462762 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f218c0b5640 0x7f218c0b5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:24.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.492+0000 7f2190e9f700 1 -- 172.21.15.16:0/178462762 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f218c0b5640 msgr2=0x7f218c0b5ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:24.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.492+0000 7f2190e9f700 1 --2- 172.21.15.16:0/178462762 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f218c0b5640 0x7f218c0b5ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:24.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.492+0000 7f2190e9f700 1 -- 172.21.15.16:0/178462762 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2178005210 msgr2=0x7f218c144550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:24.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.492+0000 7f2190e9f700 1 --2- 172.21.15.16:0/178462762 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2178005210 0x7f218c144550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:24.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.492+0000 7f2190e9f700 1 -- 172.21.15.16:0/178462762 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f218c0b4e40 con 0x7f218c0b2320 2023-10-16T13:46:24.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.492+0000 7f2190e9f700 1 --2- 172.21.15.16:0/178462762 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f218c0b2320 0x7f218c0b27a0 secure :-1 s=READY pgs=2447 cs=0 l=1 rev1=1 crypto rx=0x7f2188011980 tx=0x7f2188011c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:24.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.492+0000 7f2181ffb700 1 -- 172.21.15.16:0/178462762 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2188005520 con 0x7f218c0b2320 2023-10-16T13:46:24.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.492+0000 7f2181ffb700 1 -- 172.21.15.16:0/178462762 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2188016070 con 0x7f218c0b2320 2023-10-16T13:46:24.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.492+0000 7f2193904700 1 -- 172.21.15.16:0/178462762 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f218c0b5120 con 0x7f218c0b2320 2023-10-16T13:46:24.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.492+0000 7f2181ffb700 1 -- 172.21.15.16:0/178462762 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2188006a80 con 0x7f218c0b2320 2023-10-16T13:46:24.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.493+0000 7f2193904700 1 -- 172.21.15.16:0/178462762 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f218c0b6000 con 0x7f218c0b2320 2023-10-16T13:46:24.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.493+0000 7f2193904700 1 -- 172.21.15.16:0/178462762 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f218c0b5490 con 0x7f218c0b2320 2023-10-16T13:46:24.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.495+0000 7f2181ffb700 1 -- 172.21.15.16:0/178462762 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 138) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2188006be0 con 0x7f218c0b2320 2023-10-16T13:46:24.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.495+0000 7f2181ffb700 1 --2- 172.21.15.16:0/178462762 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f217006b020 0x7f217006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:24.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.495+0000 7f2181ffb700 1 -- 172.21.15.16:0/178462762 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f218808bca0 con 0x7f218c0b2320 2023-10-16T13:46:24.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.495+0000 7f21916a0700 1 --2- 172.21.15.16:0/178462762 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f217006b020 0x7f217006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:24.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.496+0000 7f21916a0700 1 --2- 172.21.15.16:0/178462762 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f217006b020 0x7f217006d4e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f217c007fd0 tx=0x7f217c001e20 comp rx=0 tx=0).ready entity=mgr.11190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:24.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.499+0000 7f2181ffb700 1 -- 172.21.15.16:0/178462762 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2188056bc0 con 0x7f218c0b2320 2023-10-16T13:46:24.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:24.753+0000 7f2193904700 1 -- 172.21.15.16:0/178462762 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f218c04ebb0 con 0x7f218c0b2320 2023-10-16T13:46:25.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.580+0000 7f2181ffb700 1 -- 172.21.15.16:0/178462762 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 27 and data pool 28 v109) v1 ==== 174+0+0 (secure 0 0 0) 0x7f218805a1e0 con 0x7f218c0b2320 2023-10-16T13:46:25.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.583+0000 7f2193904700 1 -- 172.21.15.16:0/178462762 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f217006b020 msgr2=0x7f217006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:25.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.583+0000 7f2193904700 1 --2- 172.21.15.16:0/178462762 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f217006b020 0x7f217006d4e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f217c007fd0 tx=0x7f217c001e20 comp rx=0 tx=0).stop 2023-10-16T13:46:25.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.583+0000 7f2193904700 1 -- 172.21.15.16:0/178462762 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f218c0b2320 msgr2=0x7f218c0b27a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:25.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.583+0000 7f2193904700 1 --2- 172.21.15.16:0/178462762 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f218c0b2320 0x7f218c0b27a0 secure :-1 s=READY pgs=2447 cs=0 l=1 rev1=1 crypto rx=0x7f2188011980 tx=0x7f2188011c90 comp rx=0 tx=0).stop 2023-10-16T13:46:25.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.584+0000 7f2193904700 1 -- 172.21.15.16:0/178462762 shutdown_connections 2023-10-16T13:46:25.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.584+0000 7f2193904700 1 --2- 172.21.15.16:0/178462762 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f218c0b5640 0x7f218c0b5ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:25.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.584+0000 7f2193904700 1 --2- 172.21.15.16:0/178462762 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2178005210 0x7f218c144550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:25.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.584+0000 7f2193904700 1 --2- 172.21.15.16:0/178462762 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f217006b020 0x7f217006d4e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:25.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.584+0000 7f2193904700 1 --2- 172.21.15.16:0/178462762 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f218c0b2320 0x7f218c0b27a0 unknown :-1 s=CLOSED pgs=2447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:25.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.584+0000 7f2193904700 1 -- 172.21.15.16:0/178462762 >> 172.21.15.16:0/178462762 conn(0x7f218c0b1f40 msgr2=0x7f218c0afe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:25.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.584+0000 7f2193904700 1 -- 172.21.15.16:0/178462762 shutdown_connections 2023-10-16T13:46:25.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.584+0000 7f2193904700 1 -- 172.21.15.16:0/178462762 wait complete. 2023-10-16T13:46:25.594 INFO:teuthology.orchestra.run.smithi016.stderr:new fs with metadata pool 27 and data pool 28 2023-10-16T13:46:25.607 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:46:25.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.735+0000 7f6b5b88b700 1 Processor -- start 2023-10-16T13:46:25.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.736+0000 7f6b5b88b700 1 -- start start 2023-10-16T13:46:25.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.736+0000 7f6b5b88b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6b5415a5c0 0x7f6b540cef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:25.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.736+0000 7f6b5b88b700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b54072a50 con 0x7f6b540ce010 2023-10-16T13:46:25.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.736+0000 7f6b5b88b700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b540cba30 con 0x7f6b540d0ae0 2023-10-16T13:46:25.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.736+0000 7f6b5b88b700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b540cf440 con 0x7f6b5415a5c0 2023-10-16T13:46:25.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.736+0000 7f6b59627700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6b5415a5c0 0x7f6b540cef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:25.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.736+0000 7f6b59627700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6b5415a5c0 0x7f6b540cef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53110/0 (socket says 172.21.15.16:53110) 2023-10-16T13:46:25.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.736+0000 7f6b58e26700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f6b540ce010 0x7f6b540d03d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42384/0 (socket says 172.21.15.16:42384) 2023-10-16T13:46:25.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.736+0000 7f6b59627700 1 -- 172.21.15.16:0/2816376593 learned_addr learned my addr 172.21.15.16:0/2816376593 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:25.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.737+0000 7f6b59627700 1 -- 172.21.15.16:0/2816376593 >> v1:172.21.15.49:6789/0 conn(0x7f6b540d0ae0 legacy=0x7f6b540cc1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:25.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.737+0000 7f6b59627700 1 -- 172.21.15.16:0/2816376593 >> v1:172.21.15.16:6789/0 conn(0x7f6b540ce010 legacy=0x7f6b540d03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:25.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.737+0000 7f6b59627700 1 -- 172.21.15.16:0/2816376593 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b540cf630 con 0x7f6b5415a5c0 2023-10-16T13:46:25.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.737+0000 7f6b4bfff700 1 -- 172.21.15.16:0/2816376593 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3812830740 0 0) 0x7f6b54072a50 con 0x7f6b540ce010 2023-10-16T13:46:25.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.737+0000 7f6b59627700 1 --2- 172.21.15.16:0/2816376593 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b5415a5c0 0x7f6b540cef00 secure :-1 s=READY pgs=2449 cs=0 l=1 rev1=1 crypto rx=0x7f6b440098f0 tx=0x7f6b44009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4eb1298b6779bc24 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:25.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.737+0000 7f6b4bfff700 1 -- 172.21.15.16:0/2816376593 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b4400b8a0 con 0x7f6b5415a5c0 2023-10-16T13:46:25.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.737+0000 7f6b4bfff700 1 -- 172.21.15.16:0/2816376593 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b4400ba00 con 0x7f6b5415a5c0 2023-10-16T13:46:25.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.737+0000 7f6b4bfff700 1 -- 172.21.15.16:0/2816376593 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b4400bd90 con 0x7f6b5415a5c0 2023-10-16T13:46:25.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.738+0000 7f6b5b88b700 1 -- 172.21.15.16:0/2816376593 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b5415a5c0 msgr2=0x7f6b540cef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:25.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.738+0000 7f6b5b88b700 1 --2- 172.21.15.16:0/2816376593 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b5415a5c0 0x7f6b540cef00 secure :-1 s=READY pgs=2449 cs=0 l=1 rev1=1 crypto rx=0x7f6b440098f0 tx=0x7f6b44009c00 comp rx=0 tx=0).stop 2023-10-16T13:46:25.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.738+0000 7f6b5b88b700 1 -- 172.21.15.16:0/2816376593 shutdown_connections 2023-10-16T13:46:25.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.738+0000 7f6b5b88b700 1 --2- 172.21.15.16:0/2816376593 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b5415a5c0 0x7f6b540cef00 unknown :-1 s=CLOSED pgs=2449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:25.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.738+0000 7f6b5b88b700 1 -- 172.21.15.16:0/2816376593 >> 172.21.15.16:0/2816376593 conn(0x7f6b540b1fd0 msgr2=0x7f6b540b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:25.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.738+0000 7f6b5b88b700 1 -- 172.21.15.16:0/2816376593 shutdown_connections 2023-10-16T13:46:25.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.738+0000 7f6b5b88b700 1 -- 172.21.15.16:0/2816376593 wait complete. 2023-10-16T13:46:25.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.739+0000 7f6b5b88b700 1 Processor -- start 2023-10-16T13:46:25.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.739+0000 7f6b5b88b700 1 -- start start 2023-10-16T13:46:25.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.740+0000 7f6b5b88b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6b540ce010 0x7f6b540c5690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:25.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.740+0000 7f6b5b88b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b540d0ae0 0x7f6b540c5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:25.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.740+0000 7f6b5b88b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6b5415a5c0 0x7f6b540ad130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:25.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.740+0000 7f6b5b88b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b54157280 con 0x7f6b540d0ae0 2023-10-16T13:46:25.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.740+0000 7f6b5b88b700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b540b4b20 con 0x7f6b540ce010 2023-10-16T13:46:25.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.740+0000 7f6b5b88b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b540b5980 con 0x7f6b5415a5c0 2023-10-16T13:46:25.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.740+0000 7f6b58e26700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b540d0ae0 0x7f6b540c5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:25.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.740+0000 7f6b58e26700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b540d0ae0 0x7f6b540c5bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53118/0 (socket says 172.21.15.16:53118) 2023-10-16T13:46:25.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.740+0000 7f6b58e26700 1 -- 172.21.15.16:0/3444208762 learned_addr learned my addr 172.21.15.16:0/3444208762 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:25.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.740+0000 7f6b59627700 1 --2- 172.21.15.16:0/3444208762 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6b540ce010 0x7f6b540c5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:25.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.740+0000 7f6b59e28700 1 --2- 172.21.15.16:0/3444208762 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6b5415a5c0 0x7f6b540ad130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:25.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.740+0000 7f6b58e26700 1 -- 172.21.15.16:0/3444208762 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6b5415a5c0 msgr2=0x7f6b540ad130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:25.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.740+0000 7f6b58e26700 1 --2- 172.21.15.16:0/3444208762 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6b5415a5c0 0x7f6b540ad130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:25.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.740+0000 7f6b58e26700 1 -- 172.21.15.16:0/3444208762 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6b540ce010 msgr2=0x7f6b540c5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:25.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.740+0000 7f6b58e26700 1 --2- 172.21.15.16:0/3444208762 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6b540ce010 0x7f6b540c5690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:25.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.740+0000 7f6b58e26700 1 -- 172.21.15.16:0/3444208762 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b44009550 con 0x7f6b540d0ae0 2023-10-16T13:46:25.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.740+0000 7f6b58e26700 1 --2- 172.21.15.16:0/3444208762 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b540d0ae0 0x7f6b540c5bd0 secure :-1 s=READY pgs=2450 cs=0 l=1 rev1=1 crypto rx=0x7f6b5000c480 tx=0x7f6b5000c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:25.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.741+0000 7f6b49ffb700 1 -- 172.21.15.16:0/3444208762 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b50004c80 con 0x7f6b540d0ae0 2023-10-16T13:46:25.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.741+0000 7f6b49ffb700 1 -- 172.21.15.16:0/3444208762 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b50004e00 con 0x7f6b540d0ae0 2023-10-16T13:46:25.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.741+0000 7f6b5b88b700 1 -- 172.21.15.16:0/3444208762 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b540b4d70 con 0x7f6b540d0ae0 2023-10-16T13:46:25.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.741+0000 7f6b49ffb700 1 -- 172.21.15.16:0/3444208762 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b50007310 con 0x7f6b540d0ae0 2023-10-16T13:46:25.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.741+0000 7f6b5b88b700 1 -- 172.21.15.16:0/3444208762 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b540b5330 con 0x7f6b540d0ae0 2023-10-16T13:46:25.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.742+0000 7f6b5b88b700 1 -- 172.21.15.16:0/3444208762 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b5404ebb0 con 0x7f6b540d0ae0 2023-10-16T13:46:25.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.742+0000 7f6b49ffb700 1 -- 172.21.15.16:0/3444208762 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 138) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6b50007470 con 0x7f6b540d0ae0 2023-10-16T13:46:25.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.742+0000 7f6b49ffb700 1 --2- 172.21.15.16:0/3444208762 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f6b3806b020 0x7f6b3806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:25.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.743+0000 7f6b49ffb700 1 -- 172.21.15.16:0/3444208762 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(224..224 src has 1..224) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f6b50089550 con 0x7f6b540d0ae0 2023-10-16T13:46:25.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.743+0000 7f6b59627700 1 --2- 172.21.15.16:0/3444208762 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f6b3806b020 0x7f6b3806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:25.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.743+0000 7f6b59627700 1 --2- 172.21.15.16:0/3444208762 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f6b3806b020 0x7f6b3806d4e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6b44009130 tx=0x7f6b44008030 comp rx=0 tx=0).ready entity=mgr.11190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:25.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.747+0000 7f6b49ffb700 1 -- 172.21.15.16:0/3444208762 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6b500543c0 con 0x7f6b540d0ae0 2023-10-16T13:46:25.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.952+0000 7f6b5b88b700 1 -- 172.21.15.16:0/3444208762 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6b540b5ac0 con 0x7f6b540d0ae0 2023-10-16T13:46:25.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.953+0000 7f6b49ffb700 1 -- 172.21.15.16:0/3444208762 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v224) v1 ==== 74+0+16894 (secure 0 0 0) 0x7f6b50007720 con 0x7f6b540d0ae0 2023-10-16T13:46:25.956 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:46:25.956 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":224,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:46:24.754970+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2023-10-16T13:46:22.489653+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"224","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2023-10-16T13:46:23.729509+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"224","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":223,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":223,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":223,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":223,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":223,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":223,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":223,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":223,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:46:25.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.955+0000 7f6b5b88b700 1 -- 172.21.15.16:0/3444208762 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f6b3806b020 msgr2=0x7f6b3806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:25.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.955+0000 7f6b5b88b700 1 --2- 172.21.15.16:0/3444208762 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f6b3806b020 0x7f6b3806d4e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6b44009130 tx=0x7f6b44008030 comp rx=0 tx=0).stop 2023-10-16T13:46:25.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.956+0000 7f6b5b88b700 1 -- 172.21.15.16:0/3444208762 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b540d0ae0 msgr2=0x7f6b540c5bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:25.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.956+0000 7f6b5b88b700 1 --2- 172.21.15.16:0/3444208762 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b540d0ae0 0x7f6b540c5bd0 secure :-1 s=READY pgs=2450 cs=0 l=1 rev1=1 crypto rx=0x7f6b5000c480 tx=0x7f6b5000c840 comp rx=0 tx=0).stop 2023-10-16T13:46:25.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.956+0000 7f6b5b88b700 1 -- 172.21.15.16:0/3444208762 shutdown_connections 2023-10-16T13:46:25.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.956+0000 7f6b5b88b700 1 --2- 172.21.15.16:0/3444208762 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6b5415a5c0 0x7f6b540ad130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:25.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.956+0000 7f6b5b88b700 1 --2- 172.21.15.16:0/3444208762 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6b540ce010 0x7f6b540c5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:25.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.956+0000 7f6b5b88b700 1 --2- 172.21.15.16:0/3444208762 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f6b3806b020 0x7f6b3806d4e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:25.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.956+0000 7f6b5b88b700 1 --2- 172.21.15.16:0/3444208762 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b540d0ae0 0x7f6b540c5bd0 unknown :-1 s=CLOSED pgs=2450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:25.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.956+0000 7f6b5b88b700 1 -- 172.21.15.16:0/3444208762 >> 172.21.15.16:0/3444208762 conn(0x7f6b540b1fd0 msgr2=0x7f6b5415e1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:25.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.956+0000 7f6b5b88b700 1 -- 172.21.15.16:0/3444208762 shutdown_connections 2023-10-16T13:46:25.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:25.956+0000 7f6b5b88b700 1 -- 172.21.15.16:0/3444208762 wait complete. 2023-10-16T13:46:25.971 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:46:26.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.094+0000 7fdaf9790700 1 Processor -- start 2023-10-16T13:46:26.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.095+0000 7fdaf9790700 1 -- start start 2023-10-16T13:46:26.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.095+0000 7fdaf9790700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fdaf40d0750 0x7fdaf4168450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:26.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.095+0000 7fdaf9790700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaf406ba00 con 0x7fdaf415d8d0 2023-10-16T13:46:26.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.095+0000 7fdaf9790700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaf40d26d0 con 0x7fdaf40d41c0 2023-10-16T13:46:26.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.095+0000 7fdaf9790700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdaf40d23f0 con 0x7fdaf40d0750 2023-10-16T13:46:26.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.096+0000 7fdaf2ffd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fdaf415d8d0 0x7fdaf40d1c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42398/0 (socket says 172.21.15.16:42398) 2023-10-16T13:46:26.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.096+0000 7fdaf2ffd700 1 -- 172.21.15.16:0/2425447064 learned_addr learned my addr 172.21.15.16:0/2425447064 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:46:26.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.096+0000 7fdaf37fe700 1 --2- 172.21.15.16:0/2425447064 >> v2:172.21.15.49:3300/0 conn(0x7fdaf40d0750 0x7fdaf4168450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:26.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.096+0000 7fdaf1ffb700 1 -- 172.21.15.16:0/2425447064 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4252694672 0 0) 0x7fdaf406ba00 con 0x7fdaf415d8d0 2023-10-16T13:46:26.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.096+0000 7fdaf1ffb700 1 -- 172.21.15.16:0/2425447064 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdae00035b0 con 0x7fdaf415d8d0 2023-10-16T13:46:26.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.096+0000 7fdaf1ffb700 1 -- 172.21.15.16:0/2425447064 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 468925757 0 0) 0x7fdaf40d26d0 con 0x7fdaf40d41c0 2023-10-16T13:46:26.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.096+0000 7fdaf1ffb700 1 -- 172.21.15.16:0/2425447064 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdaf406ba00 con 0x7fdaf40d41c0 2023-10-16T13:46:26.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.096+0000 7fdaf1ffb700 1 -- 172.21.15.16:0/2425447064 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4171971701 0 0) 0x7fdae00035b0 con 0x7fdaf415d8d0 2023-10-16T13:46:26.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.096+0000 7fdaf1ffb700 1 -- 172.21.15.16:0/2425447064 >> v2:172.21.15.49:3300/0 conn(0x7fdaf40d0750 msgr2=0x7fdaf4168450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:26.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.096+0000 7fdaf1ffb700 1 --2- 172.21.15.16:0/2425447064 >> v2:172.21.15.49:3300/0 conn(0x7fdaf40d0750 0x7fdaf4168450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:26.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.096+0000 7fdaf1ffb700 1 -- 172.21.15.16:0/2425447064 >> v1:172.21.15.49:6789/0 conn(0x7fdaf40d41c0 legacy=0x7fdaf40d0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:26.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.096+0000 7fdaf1ffb700 1 -- 172.21.15.16:0/2425447064 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaf406bca0 con 0x7fdaf415d8d0 2023-10-16T13:46:26.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.096+0000 7fdaf1ffb700 1 -- 172.21.15.16:0/2425447064 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7fdae4001670 con 0x7fdaf415d8d0 2023-10-16T13:46:26.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.096+0000 7fdaf1ffb700 1 -- 172.21.15.16:0/2425447064 >> v1:172.21.15.16:6789/0 conn(0x7fdaf415d8d0 legacy=0x7fdaf40d1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:26.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.096+0000 7fdaf1ffb700 1 --2- 172.21.15.16:0/2425447064 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdae0003b90 0x7fdae0003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:26.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.096+0000 7fdaf1ffb700 1 --2- 172.21.15.16:0/2425447064 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdae0004850 0x7fdae0004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:26.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.096+0000 7fdaf1ffb700 1 --2- 172.21.15.16:0/2425447064 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdae0005210 0x7fdae000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:26.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.096+0000 7fdaf1ffb700 1 -- 172.21.15.16:0/2425447064 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf40d23f0 con 0x7fdae0004850 2023-10-16T13:46:26.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.096+0000 7fdaf1ffb700 1 -- 172.21.15.16:0/2425447064 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdae000c090 con 0x7fdae0003b90 2023-10-16T13:46:26.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.096+0000 7fdaf1ffb700 1 -- 172.21.15.16:0/2425447064 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdae000c250 con 0x7fdae0005210 2023-10-16T13:46:26.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.096+0000 7fdaf27fc700 1 --2- 172.21.15.16:0/2425447064 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdae0004850 0x7fdae0004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:26.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.096+0000 7fdaf2ffd700 1 --2- 172.21.15.16:0/2425447064 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdae0003b90 0x7fdae0003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:26.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.096+0000 7fdaf37fe700 1 --2- 172.21.15.16:0/2425447064 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdae0005210 0x7fdae000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:26.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.097+0000 7fdaf27fc700 1 -- 172.21.15.16:0/2425447064 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdae0005210 msgr2=0x7fdae000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:26.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.097+0000 7fdaf27fc700 1 --2- 172.21.15.16:0/2425447064 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdae0005210 0x7fdae000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:26.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.097+0000 7fdaf27fc700 1 -- 172.21.15.16:0/2425447064 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdae0003b90 msgr2=0x7fdae0003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:26.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.097+0000 7fdaf27fc700 1 --2- 172.21.15.16:0/2425447064 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdae0003b90 0x7fdae0003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:26.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.097+0000 7fdaf27fc700 1 -- 172.21.15.16:0/2425447064 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdae00035b0 con 0x7fdae0004850 2023-10-16T13:46:26.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.097+0000 7fdaf37fe700 1 --2- 172.21.15.16:0/2425447064 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdae0005210 0x7fdae000ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:46:26.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.097+0000 7fdaf37fe700 1 -- 172.21.15.16:0/2425447064 reap_dead start 2023-10-16T13:46:26.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.097+0000 7fdaf27fc700 1 --2- 172.21.15.16:0/2425447064 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdae0004850 0x7fdae0004cd0 secure :-1 s=READY pgs=2452 cs=0 l=1 rev1=1 crypto rx=0x7fdadc001940 tx=0x7fdae0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e239a824e5bc161 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:26.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.097+0000 7fdaf1ffb700 1 -- 172.21.15.16:0/2425447064 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdadc001430 con 0x7fdae0004850 2023-10-16T13:46:26.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.097+0000 7fdaf1ffb700 1 -- 172.21.15.16:0/2425447064 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdadc001d70 con 0x7fdae0004850 2023-10-16T13:46:26.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.097+0000 7fdaf1ffb700 1 -- 172.21.15.16:0/2425447064 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdadc005510 con 0x7fdae0004850 2023-10-16T13:46:26.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.097+0000 7fdaf9790700 1 -- 172.21.15.16:0/2425447064 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdae0004850 msgr2=0x7fdae0004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:26.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.097+0000 7fdaf9790700 1 --2- 172.21.15.16:0/2425447064 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdae0004850 0x7fdae0004cd0 secure :-1 s=READY pgs=2452 cs=0 l=1 rev1=1 crypto rx=0x7fdadc001940 tx=0x7fdae0005880 comp rx=0 tx=0).stop 2023-10-16T13:46:26.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.097+0000 7fdaf9790700 1 -- 172.21.15.16:0/2425447064 shutdown_connections 2023-10-16T13:46:26.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.097+0000 7fdaf9790700 1 --2- 172.21.15.16:0/2425447064 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdae0004850 0x7fdae0004cd0 unknown :-1 s=CLOSED pgs=2452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:26.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.097+0000 7fdaf9790700 1 -- 172.21.15.16:0/2425447064 >> 172.21.15.16:0/2425447064 conn(0x7fdaf40d3630 msgr2=0x7fdaf40d3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:26.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.097+0000 7fdaf9790700 1 -- 172.21.15.16:0/2425447064 shutdown_connections 2023-10-16T13:46:26.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.098+0000 7fdaf9790700 1 -- 172.21.15.16:0/2425447064 wait complete. 2023-10-16T13:46:26.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.099+0000 7fdaf9790700 1 Processor -- start 2023-10-16T13:46:26.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.099+0000 7fdaf9790700 1 -- start start 2023-10-16T13:46:26.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.100+0000 7fdaf9790700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdae0004850 0x7fdaf414eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:26.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.100+0000 7fdaf9790700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdaf41f2bd0 0x7fdaf40c9a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:26.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.100+0000 7fdaf9790700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdaf41f3050 0x7fdaf40c9f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:26.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.100+0000 7fdaf9790700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf41e6df0 con 0x7fdae0004850 2023-10-16T13:46:26.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.100+0000 7fdaf9790700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf414f0e0 con 0x7fdaf41f3050 2023-10-16T13:46:26.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.100+0000 7fdaf9790700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf415af00 con 0x7fdaf41f2bd0 2023-10-16T13:46:26.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.100+0000 7fdaf2ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdae0004850 0x7fdaf414eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:26.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.100+0000 7fdaf2ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdae0004850 0x7fdaf414eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53136/0 (socket says 172.21.15.16:53136) 2023-10-16T13:46:26.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.100+0000 7fdaf2ffd700 1 -- 172.21.15.16:0/3111842148 learned_addr learned my addr 172.21.15.16:0/3111842148 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:26.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.100+0000 7fdaf37fe700 1 --2- 172.21.15.16:0/3111842148 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdaf41f3050 0x7fdaf40c9f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:26.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.100+0000 7fdaf27fc700 1 --2- 172.21.15.16:0/3111842148 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdaf41f2bd0 0x7fdaf40c9a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:26.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.100+0000 7fdaf2ffd700 1 -- 172.21.15.16:0/3111842148 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdaf41f2bd0 msgr2=0x7fdaf40c9a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:26.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.100+0000 7fdaf2ffd700 1 --2- 172.21.15.16:0/3111842148 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdaf41f2bd0 0x7fdaf40c9a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:26.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.100+0000 7fdaf2ffd700 1 -- 172.21.15.16:0/3111842148 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdaf41f3050 msgr2=0x7fdaf40c9f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:26.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.100+0000 7fdaf2ffd700 1 --2- 172.21.15.16:0/3111842148 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdaf41f3050 0x7fdaf40c9f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:26.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.100+0000 7fdaf2ffd700 1 -- 172.21.15.16:0/3111842148 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaf41e61e0 con 0x7fdae0004850 2023-10-16T13:46:26.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.100+0000 7fdaf37fe700 1 --2- 172.21.15.16:0/3111842148 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdaf41f3050 0x7fdaf40c9f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:46:26.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.100+0000 7fdaf27fc700 1 --2- 172.21.15.16:0/3111842148 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdaf41f2bd0 0x7fdaf40c9a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:46:26.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.100+0000 7fdaf2ffd700 1 --2- 172.21.15.16:0/3111842148 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdae0004850 0x7fdaf414eba0 secure :-1 s=READY pgs=2453 cs=0 l=1 rev1=1 crypto rx=0x7fdae4007340 tx=0x7fdae4007650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:26.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.100+0000 7fdaeb7fe700 1 -- 172.21.15.16:0/3111842148 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdae400f8d0 con 0x7fdae0004850 2023-10-16T13:46:26.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.100+0000 7fdaeb7fe700 1 -- 172.21.15.16:0/3111842148 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdae400fa30 con 0x7fdae0004850 2023-10-16T13:46:26.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.100+0000 7fdaf9790700 1 -- 172.21.15.16:0/3111842148 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdaf41e64c0 con 0x7fdae0004850 2023-10-16T13:46:26.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.100+0000 7fdaeb7fe700 1 -- 172.21.15.16:0/3111842148 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdae400fde0 con 0x7fdae0004850 2023-10-16T13:46:26.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.100+0000 7fdaf9790700 1 -- 172.21.15.16:0/3111842148 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdaf41e6a80 con 0x7fdae0004850 2023-10-16T13:46:26.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.101+0000 7fdaf9790700 1 -- 172.21.15.16:0/3111842148 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdaf404ebb0 con 0x7fdae0004850 2023-10-16T13:46:26.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.102+0000 7fdaeb7fe700 1 -- 172.21.15.16:0/3111842148 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 138) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fdae401d460 con 0x7fdae0004850 2023-10-16T13:46:26.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.102+0000 7fdaeb7fe700 1 --2- 172.21.15.16:0/3111842148 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fdad406b020 0x7fdad406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:26.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.102+0000 7fdaeb7fe700 1 -- 172.21.15.16:0/3111842148 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(224..224 src has 1..224) v4 ==== 6157+0+0 (secure 0 0 0) 0x7fdae4089a00 con 0x7fdae0004850 2023-10-16T13:46:26.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.102+0000 7fdaf27fc700 1 --2- 172.21.15.16:0/3111842148 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fdad406b020 0x7fdad406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:26.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.103+0000 7fdaf27fc700 1 --2- 172.21.15.16:0/3111842148 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fdad406b020 0x7fdad406d4e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fdaf40c9810 tx=0x7fdadc00a000 comp rx=0 tx=0).ready entity=mgr.11190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:26.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.107+0000 7fdaeb7fe700 1 -- 172.21.15.16:0/3111842148 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdae4054870 con 0x7fdae0004850 2023-10-16T13:46:26.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.313+0000 7fdaf9790700 1 -- 172.21.15.16:0/3111842148 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdaf41e2ce0 con 0x7fdae0004850 2023-10-16T13:46:26.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.314+0000 7fdaeb7fe700 1 -- 172.21.15.16:0/3111842148 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v224) v1 ==== 74+0+16894 (secure 0 0 0) 0x7fdae4057e90 con 0x7fdae0004850 2023-10-16T13:46:26.316 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:46:26.317 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":224,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:46:24.754970+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2023-10-16T13:46:22.489653+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"224","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2023-10-16T13:46:23.729509+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"224","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":223,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":223,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":223,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":223,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":223,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":223,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":223,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":223,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:46:26.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.316+0000 7fdaf9790700 1 -- 172.21.15.16:0/3111842148 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fdad406b020 msgr2=0x7fdad406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:26.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.316+0000 7fdaf9790700 1 --2- 172.21.15.16:0/3111842148 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fdad406b020 0x7fdad406d4e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fdaf40c9810 tx=0x7fdadc00a000 comp rx=0 tx=0).stop 2023-10-16T13:46:26.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.316+0000 7fdaf9790700 1 -- 172.21.15.16:0/3111842148 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdae0004850 msgr2=0x7fdaf414eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:26.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.316+0000 7fdaf9790700 1 --2- 172.21.15.16:0/3111842148 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdae0004850 0x7fdaf414eba0 secure :-1 s=READY pgs=2453 cs=0 l=1 rev1=1 crypto rx=0x7fdae4007340 tx=0x7fdae4007650 comp rx=0 tx=0).stop 2023-10-16T13:46:26.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.316+0000 7fdaf9790700 1 -- 172.21.15.16:0/3111842148 shutdown_connections 2023-10-16T13:46:26.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.316+0000 7fdaf9790700 1 --2- 172.21.15.16:0/3111842148 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdaf41f3050 0x7fdaf40c9f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:26.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.316+0000 7fdaf9790700 1 --2- 172.21.15.16:0/3111842148 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fdad406b020 0x7fdad406d4e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:26.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.316+0000 7fdaf9790700 1 --2- 172.21.15.16:0/3111842148 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdae0004850 0x7fdaf414eba0 unknown :-1 s=CLOSED pgs=2453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:26.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.316+0000 7fdaf9790700 1 --2- 172.21.15.16:0/3111842148 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdaf41f2bd0 0x7fdaf40c9a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:26.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.316+0000 7fdaf9790700 1 -- 172.21.15.16:0/3111842148 >> 172.21.15.16:0/3111842148 conn(0x7fdaf40d3630 msgr2=0x7fdaf415c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:26.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.316+0000 7fdaf9790700 1 -- 172.21.15.16:0/3111842148 shutdown_connections 2023-10-16T13:46:26.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.316+0000 7fdaf9790700 1 -- 172.21.15.16:0/3111842148 wait complete. 2023-10-16T13:46:26.331 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-16T13:46:26.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.457+0000 7f567a4b8700 1 Processor -- start 2023-10-16T13:46:26.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.457+0000 7f567a4b8700 1 -- start start 2023-10-16T13:46:26.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.458+0000 7f567a4b8700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f56740ce5d0 0x7f56740cc190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:26.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.458+0000 7f567a4b8700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f567415f060 0x7f56740cc6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:26.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.458+0000 7f567a4b8700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56740b2440 con 0x7f567415ec80 2023-10-16T13:46:26.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.458+0000 7f567a4b8700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56740b2c50 con 0x7f567415f060 2023-10-16T13:46:26.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.458+0000 7f567a4b8700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56740d00e0 con 0x7f56740ce5d0 2023-10-16T13:46:26.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.458+0000 7f5673fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f567415ec80 0x7f56740cdec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42418/0 (socket says 172.21.15.16:42418) 2023-10-16T13:46:26.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.458+0000 7f5673fff700 1 -- 172.21.15.16:0/4236359061 learned_addr learned my addr 172.21.15.16:0/4236359061 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:46:26.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.458+0000 7f5678a55700 1 --2- 172.21.15.16:0/4236359061 >> v2:172.21.15.49:3300/0 conn(0x7f567415f060 0x7f56740cc6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:26.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.458+0000 7f56737fe700 1 --2- 172.21.15.16:0/4236359061 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f56740ce5d0 0x7f56740cc190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:26.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.459+0000 7f5672ffd700 1 -- 172.21.15.16:0/4236359061 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4082382186 0 0) 0x7f56740b2440 con 0x7f567415ec80 2023-10-16T13:46:26.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.459+0000 7f5672ffd700 1 -- 172.21.15.16:0/4236359061 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56600035b0 con 0x7f567415ec80 2023-10-16T13:46:26.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.459+0000 7f5678a55700 1 -- 172.21.15.16:0/4236359061 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f56740ce5d0 msgr2=0x7f56740cc190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:26.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.459+0000 7f5678a55700 1 --2- 172.21.15.16:0/4236359061 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f56740ce5d0 0x7f56740cc190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:26.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.459+0000 7f5672ffd700 1 -- 172.21.15.16:0/4236359061 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 255483424 0 0) 0x7f56600035b0 con 0x7f567415ec80 2023-10-16T13:46:26.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.459+0000 7f5678a55700 1 -- 172.21.15.16:0/4236359061 >> v1:172.21.15.16:6789/0 conn(0x7f567415ec80 legacy=0x7f56740cdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:26.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.459+0000 7f5678a55700 1 -- 172.21.15.16:0/4236359061 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56740cef60 con 0x7f567415f060 2023-10-16T13:46:26.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.459+0000 7f56737fe700 1 --2- 172.21.15.16:0/4236359061 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f56740ce5d0 0x7f56740cc190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:46:26.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.459+0000 7f5678a55700 1 --2- 172.21.15.16:0/4236359061 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f567415f060 0x7f56740cc6d0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f566c009d90 tx=0x7f566c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ac7d816b1583e80d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:26.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.459+0000 7f5672ffd700 1 -- 172.21.15.16:0/4236359061 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f566c00e070 con 0x7f567415f060 2023-10-16T13:46:26.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.459+0000 7f5672ffd700 1 -- 172.21.15.16:0/4236359061 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f566c003a20 con 0x7f567415f060 2023-10-16T13:46:26.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.459+0000 7f5672ffd700 1 -- 172.21.15.16:0/4236359061 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5664001670 con 0x7f567415f060 2023-10-16T13:46:26.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.460+0000 7f567a4b8700 1 -- 172.21.15.16:0/4236359061 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f567415f060 msgr2=0x7f56740cc6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:26.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.460+0000 7f567a4b8700 1 --2- 172.21.15.16:0/4236359061 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f567415f060 0x7f56740cc6d0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f566c009d90 tx=0x7f566c00b330 comp rx=0 tx=0).stop 2023-10-16T13:46:26.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.460+0000 7f567a4b8700 1 -- 172.21.15.16:0/4236359061 shutdown_connections 2023-10-16T13:46:26.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.460+0000 7f567a4b8700 1 --2- 172.21.15.16:0/4236359061 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f567415f060 0x7f56740cc6d0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:26.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.460+0000 7f567a4b8700 1 --2- 172.21.15.16:0/4236359061 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f56740ce5d0 0x7f56740cc190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:26.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.460+0000 7f567a4b8700 1 -- 172.21.15.16:0/4236359061 >> 172.21.15.16:0/4236359061 conn(0x7f56740d15f0 msgr2=0x7f56740d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:26.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.460+0000 7f567a4b8700 1 -- 172.21.15.16:0/4236359061 shutdown_connections 2023-10-16T13:46:26.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.460+0000 7f567a4b8700 1 -- 172.21.15.16:0/4236359061 wait complete. 2023-10-16T13:46:26.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.461+0000 7f567a4b8700 1 Processor -- start 2023-10-16T13:46:26.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.461+0000 7f567a4b8700 1 -- start start 2023-10-16T13:46:26.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.462+0000 7f567a4b8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56740ce5d0 0x7f56741f29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:26.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.462+0000 7f567a4b8700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f567415ec80 0x7f56740c7910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:26.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.462+0000 7f567a4b8700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f567415f060 0x7f56740cca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:26.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.462+0000 7f567a4b8700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56741f2f20 con 0x7f56740ce5d0 2023-10-16T13:46:26.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.462+0000 7f567a4b8700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f567406c300 con 0x7f567415ec80 2023-10-16T13:46:26.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.462+0000 7f567a4b8700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56740cc740 con 0x7f567415f060 2023-10-16T13:46:26.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.462+0000 7f5673fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56740ce5d0 0x7f56741f29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:26.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.462+0000 7f5673fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56740ce5d0 0x7f56741f29e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53150/0 (socket says 172.21.15.16:53150) 2023-10-16T13:46:26.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.462+0000 7f5673fff700 1 -- 172.21.15.16:0/4147899392 learned_addr learned my addr 172.21.15.16:0/4147899392 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:26.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.462+0000 7f56737fe700 1 --2- 172.21.15.16:0/4147899392 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f567415ec80 0x7f56740c7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:26.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.462+0000 7f5678a55700 1 --2- 172.21.15.16:0/4147899392 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f567415f060 0x7f56740cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:26.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.462+0000 7f5673fff700 1 -- 172.21.15.16:0/4147899392 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f567415f060 msgr2=0x7f56740cca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:26.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.462+0000 7f5673fff700 1 --2- 172.21.15.16:0/4147899392 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f567415f060 0x7f56740cca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:26.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.462+0000 7f5673fff700 1 -- 172.21.15.16:0/4147899392 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f567415ec80 msgr2=0x7f56740c7910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:26.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.462+0000 7f5673fff700 1 --2- 172.21.15.16:0/4147899392 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f567415ec80 0x7f56740c7910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:26.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.462+0000 7f5673fff700 1 -- 172.21.15.16:0/4147899392 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f565c0094a0 con 0x7f56740ce5d0 2023-10-16T13:46:26.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.463+0000 7f5673fff700 1 --2- 172.21.15.16:0/4147899392 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56740ce5d0 0x7f56741f29e0 secure :-1 s=READY pgs=2455 cs=0 l=1 rev1=1 crypto rx=0x7f566400c550 tx=0x7f566400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:26.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.463+0000 7f5670ff9700 1 -- 172.21.15.16:0/4147899392 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5664004c80 con 0x7f56740ce5d0 2023-10-16T13:46:26.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.463+0000 7f5670ff9700 1 -- 172.21.15.16:0/4147899392 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5664004de0 con 0x7f56740ce5d0 2023-10-16T13:46:26.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.463+0000 7f5670ff9700 1 -- 172.21.15.16:0/4147899392 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56640072f0 con 0x7f56740ce5d0 2023-10-16T13:46:26.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.463+0000 7f567a4b8700 1 -- 172.21.15.16:0/4147899392 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f566c009a70 con 0x7f56740ce5d0 2023-10-16T13:46:26.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.463+0000 7f567a4b8700 1 -- 172.21.15.16:0/4147899392 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f567406c830 con 0x7f56740ce5d0 2023-10-16T13:46:26.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.464+0000 7f567a4b8700 1 -- 172.21.15.16:0/4147899392 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56740678e0 con 0x7f56740ce5d0 2023-10-16T13:46:26.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.464+0000 7f5670ff9700 1 -- 172.21.15.16:0/4147899392 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 138) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5664007450 con 0x7f56740ce5d0 2023-10-16T13:46:26.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.465+0000 7f5670ff9700 1 --2- 172.21.15.16:0/4147899392 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f565406b020 0x7f565406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:26.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.465+0000 7f56737fe700 1 --2- 172.21.15.16:0/4147899392 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f565406b020 0x7f565406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:26.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.465+0000 7f5670ff9700 1 -- 172.21.15.16:0/4147899392 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(224..224 src has 1..224) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f5664089510 con 0x7f56740ce5d0 2023-10-16T13:46:26.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.465+0000 7f56737fe700 1 --2- 172.21.15.16:0/4147899392 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f565406b020 0x7f565406d4e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f567406d810 tx=0x7f565c009210 comp rx=0 tx=0).ready entity=mgr.11190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:26.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.469+0000 7f5670ff9700 1 -- 172.21.15.16:0/4147899392 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5664054380 con 0x7f56740ce5d0 2023-10-16T13:46:26.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:26.737+0000 7f567a4b8700 1 -- 172.21.15.16:0/4147899392 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f567404ebb0 con 0x7f56740ce5d0 2023-10-16T13:46:27.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.586+0000 7f5670ff9700 1 -- 172.21.15.16:0/4147899392 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v112) v1 ==== 118+0+0 (secure 0 0 0) 0x7f5664007700 con 0x7f56740ce5d0 2023-10-16T13:46:27.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.590+0000 7f567a4b8700 1 -- 172.21.15.16:0/4147899392 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f565406b020 msgr2=0x7f565406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:27.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.590+0000 7f567a4b8700 1 --2- 172.21.15.16:0/4147899392 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f565406b020 0x7f565406d4e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f567406d810 tx=0x7f565c009210 comp rx=0 tx=0).stop 2023-10-16T13:46:27.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.590+0000 7f567a4b8700 1 -- 172.21.15.16:0/4147899392 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56740ce5d0 msgr2=0x7f56741f29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:27.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.590+0000 7f567a4b8700 1 --2- 172.21.15.16:0/4147899392 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56740ce5d0 0x7f56741f29e0 secure :-1 s=READY pgs=2455 cs=0 l=1 rev1=1 crypto rx=0x7f566400c550 tx=0x7f566400c860 comp rx=0 tx=0).stop 2023-10-16T13:46:27.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.590+0000 7f567a4b8700 1 -- 172.21.15.16:0/4147899392 shutdown_connections 2023-10-16T13:46:27.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.590+0000 7f567a4b8700 1 --2- 172.21.15.16:0/4147899392 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f567415f060 0x7f56740cca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:27.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.590+0000 7f567a4b8700 1 --2- 172.21.15.16:0/4147899392 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f565406b020 0x7f565406d4e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:27.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.590+0000 7f567a4b8700 1 --2- 172.21.15.16:0/4147899392 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56740ce5d0 0x7f56741f29e0 unknown :-1 s=CLOSED pgs=2455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:27.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.590+0000 7f567a4b8700 1 --2- 172.21.15.16:0/4147899392 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f567415ec80 0x7f56740c7910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:27.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.590+0000 7f567a4b8700 1 -- 172.21.15.16:0/4147899392 >> 172.21.15.16:0/4147899392 conn(0x7f56740d15f0 msgr2=0x7f567415db40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:27.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.590+0000 7f567a4b8700 1 -- 172.21.15.16:0/4147899392 shutdown_connections 2023-10-16T13:46:27.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.590+0000 7f567a4b8700 1 -- 172.21.15.16:0/4147899392 wait complete. 2023-10-16T13:46:27.609 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-16T13:46:27.609 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-16T13:46:27.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.745+0000 7f072092f700 1 Processor -- start 2023-10-16T13:46:27.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.746+0000 7f072092f700 1 -- start start 2023-10-16T13:46:27.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.746+0000 7f072092f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f071c15b2c0 0x7f071c161340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:27.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.746+0000 7f072092f700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f071c06e570 0x7f071c15e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:27.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.747+0000 7f072092f700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f071c161880 con 0x7f071c06a7e0 2023-10-16T13:46:27.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.747+0000 7f072092f700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f071c15b6a0 con 0x7f071c06e570 2023-10-16T13:46:27.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.747+0000 7f072092f700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f071c161b20 con 0x7f071c15b2c0 2023-10-16T13:46:27.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.747+0000 7f071ad9d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f071c06a7e0 0x7f071c071a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42436/0 (socket says 172.21.15.16:42436) 2023-10-16T13:46:27.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.747+0000 7f071ad9d700 1 -- 172.21.15.16:0/3758476345 learned_addr learned my addr 172.21.15.16:0/3758476345 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:46:27.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.747+0000 7f071a59c700 1 --2- 172.21.15.16:0/3758476345 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f071c15b2c0 0x7f071c161340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:27.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.747+0000 7f0719d9b700 1 --2- 172.21.15.16:0/3758476345 >> v2:172.21.15.49:3300/0 conn(0x7f071c06e570 0x7f071c15e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:27.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.747+0000 7f071959a700 1 -- 172.21.15.16:0/3758476345 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3805742732 0 0) 0x7f071c161880 con 0x7f071c06a7e0 2023-10-16T13:46:27.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.747+0000 7f071959a700 1 -- 172.21.15.16:0/3758476345 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07080035b0 con 0x7f071c06a7e0 2023-10-16T13:46:27.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.747+0000 7f071959a700 1 -- 172.21.15.16:0/3758476345 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 277917927 0 0) 0x7f07080035b0 con 0x7f071c06a7e0 2023-10-16T13:46:27.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.747+0000 7f071959a700 1 -- 172.21.15.16:0/3758476345 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f071c15b2c0 msgr2=0x7f071c161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:27.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.747+0000 7f071959a700 1 --2- 172.21.15.16:0/3758476345 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f071c15b2c0 0x7f071c161340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:27.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.747+0000 7f071959a700 1 -- 172.21.15.16:0/3758476345 >> v2:172.21.15.49:3300/0 conn(0x7f071c06e570 msgr2=0x7f071c15e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:27.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.747+0000 7f071959a700 1 --2- 172.21.15.16:0/3758476345 >> v2:172.21.15.49:3300/0 conn(0x7f071c06e570 0x7f071c15e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:27.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.747+0000 7f071959a700 1 -- 172.21.15.16:0/3758476345 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f071c161e00 con 0x7f071c06a7e0 2023-10-16T13:46:27.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.747+0000 7f071959a700 1 -- 172.21.15.16:0/3758476345 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f07140019f0 con 0x7f071c06a7e0 2023-10-16T13:46:27.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.747+0000 7f071959a700 1 -- 172.21.15.16:0/3758476345 >> v1:172.21.15.16:6789/0 conn(0x7f071c06a7e0 legacy=0x7f071c071a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:27.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.747+0000 7f071959a700 1 --2- 172.21.15.16:0/3758476345 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0708003a00 0x7f0708003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:27.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.747+0000 7f071959a700 1 --2- 172.21.15.16:0/3758476345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0708004850 0x7f0708004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:27.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.747+0000 7f071959a700 1 --2- 172.21.15.16:0/3758476345 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f071c15b2c0 0x7f0708005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:27.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.747+0000 7f071959a700 1 -- 172.21.15.16:0/3758476345 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f071c15b6a0 con 0x7f0708004850 2023-10-16T13:46:27.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.747+0000 7f071959a700 1 -- 172.21.15.16:0/3758476345 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f071c161b20 con 0x7f0708003a00 2023-10-16T13:46:27.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.747+0000 7f071959a700 1 -- 172.21.15.16:0/3758476345 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f070800ba10 con 0x7f071c15b2c0 2023-10-16T13:46:27.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.747+0000 7f0719d9b700 1 --2- 172.21.15.16:0/3758476345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0708004850 0x7f0708004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:27.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.748+0000 7f071ad9d700 1 --2- 172.21.15.16:0/3758476345 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f071c15b2c0 0x7f0708005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:27.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.748+0000 7f071a59c700 1 --2- 172.21.15.16:0/3758476345 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0708003a00 0x7f0708003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:27.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.748+0000 7f0719d9b700 1 -- 172.21.15.16:0/3758476345 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f071c15b2c0 msgr2=0x7f0708005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:27.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.748+0000 7f0719d9b700 1 --2- 172.21.15.16:0/3758476345 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f071c15b2c0 0x7f0708005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:27.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.748+0000 7f0719d9b700 1 -- 172.21.15.16:0/3758476345 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0708003a00 msgr2=0x7f0708003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:27.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.748+0000 7f0719d9b700 1 --2- 172.21.15.16:0/3758476345 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0708003a00 0x7f0708003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:27.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.748+0000 7f0719d9b700 1 -- 172.21.15.16:0/3758476345 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07080035b0 con 0x7f0708004850 2023-10-16T13:46:27.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.748+0000 7f071a59c700 1 --2- 172.21.15.16:0/3758476345 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0708003a00 0x7f0708003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:46:27.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.748+0000 7f0719d9b700 1 --2- 172.21.15.16:0/3758476345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0708004850 0x7f0708004cd0 secure :-1 s=READY pgs=2457 cs=0 l=1 rev1=1 crypto rx=0x7f0704009b90 tx=0x7f0708005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8db86ed7580aadb5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:27.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.748+0000 7f071959a700 1 -- 172.21.15.16:0/3758476345 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f070400e6c0 con 0x7f0708004850 2023-10-16T13:46:27.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.748+0000 7f071959a700 1 -- 172.21.15.16:0/3758476345 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f070400e820 con 0x7f0708004850 2023-10-16T13:46:27.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.748+0000 7f071959a700 1 -- 172.21.15.16:0/3758476345 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f070400eb90 con 0x7f0708004850 2023-10-16T13:46:27.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.749+0000 7f072092f700 1 -- 172.21.15.16:0/3758476345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0708004850 msgr2=0x7f0708004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:27.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.749+0000 7f072092f700 1 --2- 172.21.15.16:0/3758476345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0708004850 0x7f0708004cd0 secure :-1 s=READY pgs=2457 cs=0 l=1 rev1=1 crypto rx=0x7f0704009b90 tx=0x7f0708005880 comp rx=0 tx=0).stop 2023-10-16T13:46:27.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.749+0000 7f071ad9d700 1 -- 172.21.15.16:0/3758476345 reap_dead start 2023-10-16T13:46:27.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.749+0000 7f072092f700 1 -- 172.21.15.16:0/3758476345 shutdown_connections 2023-10-16T13:46:27.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.749+0000 7f072092f700 1 -- 172.21.15.16:0/3758476345 >> 172.21.15.16:0/3758476345 conn(0x7f071c0b1f40 msgr2=0x7f071c0b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:27.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.749+0000 7f072092f700 1 -- 172.21.15.16:0/3758476345 shutdown_connections 2023-10-16T13:46:27.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.749+0000 7f072092f700 1 -- 172.21.15.16:0/3758476345 wait complete. 2023-10-16T13:46:27.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.750+0000 7f072092f700 1 Processor -- start 2023-10-16T13:46:27.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.750+0000 7f072092f700 1 -- start start 2023-10-16T13:46:27.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.751+0000 7f072092f700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0708004850 0x7f071c15ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:27.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.751+0000 7f072092f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f071c1f6d40 0x7f071c14f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:27.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.751+0000 7f072092f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f071c0b4610 0x7f071c0b4a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:27.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.751+0000 7f072092f700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f071c0b4fd0 con 0x7f071c1f6d40 2023-10-16T13:46:27.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.751+0000 7f072092f700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f071c0c7bb0 con 0x7f0708004850 2023-10-16T13:46:27.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.751+0000 7f072092f700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f071c0c9550 con 0x7f071c0b4610 2023-10-16T13:46:27.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.751+0000 7f0719d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f071c1f6d40 0x7f071c14f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:27.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.751+0000 7f0719d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f071c1f6d40 0x7f071c14f650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53174/0 (socket says 172.21.15.16:53174) 2023-10-16T13:46:27.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.751+0000 7f0719d9b700 1 -- 172.21.15.16:0/627185558 learned_addr learned my addr 172.21.15.16:0/627185558 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:27.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.751+0000 7f071a59c700 1 --2- 172.21.15.16:0/627185558 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0708004850 0x7f071c15ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:27.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.751+0000 7f071ad9d700 1 --2- 172.21.15.16:0/627185558 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f071c0b4610 0x7f071c0b4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:27.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.751+0000 7f0719d9b700 1 -- 172.21.15.16:0/627185558 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f071c0b4610 msgr2=0x7f071c0b4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:27.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.751+0000 7f0719d9b700 1 --2- 172.21.15.16:0/627185558 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f071c0b4610 0x7f071c0b4a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:27.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.751+0000 7f0719d9b700 1 -- 172.21.15.16:0/627185558 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0708004850 msgr2=0x7f071c15ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:27.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.751+0000 7f0719d9b700 1 --2- 172.21.15.16:0/627185558 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0708004850 0x7f071c15ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:27.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.751+0000 7f0719d9b700 1 -- 172.21.15.16:0/627185558 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f071c0ca0a0 con 0x7f071c1f6d40 2023-10-16T13:46:27.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.752+0000 7f0719d9b700 1 --2- 172.21.15.16:0/627185558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f071c1f6d40 0x7f071c14f650 secure :-1 s=READY pgs=2458 cs=0 l=1 rev1=1 crypto rx=0x7f070400e4b0 tx=0x7f07040047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:27.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.752+0000 7f0712ffd700 1 -- 172.21.15.16:0/627185558 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0704021070 con 0x7f071c1f6d40 2023-10-16T13:46:27.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.752+0000 7f0712ffd700 1 -- 172.21.15.16:0/627185558 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07040045c0 con 0x7f071c1f6d40 2023-10-16T13:46:27.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.752+0000 7f072092f700 1 -- 172.21.15.16:0/627185558 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f071c0c7560 con 0x7f071c1f6d40 2023-10-16T13:46:27.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.752+0000 7f0712ffd700 1 -- 172.21.15.16:0/627185558 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f070400e6c0 con 0x7f071c1f6d40 2023-10-16T13:46:27.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.752+0000 7f072092f700 1 -- 172.21.15.16:0/627185558 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f071c14fb90 con 0x7f071c1f6d40 2023-10-16T13:46:27.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.753+0000 7f072092f700 1 -- 172.21.15.16:0/627185558 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f071c04ebb0 con 0x7f071c1f6d40 2023-10-16T13:46:27.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.753+0000 7f0712ffd700 1 -- 172.21.15.16:0/627185558 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 138) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0704025070 con 0x7f071c1f6d40 2023-10-16T13:46:27.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.754+0000 7f0712ffd700 1 --2- 172.21.15.16:0/627185558 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f06fc06b020 0x7f06fc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:27.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.754+0000 7f0712ffd700 1 -- 172.21.15.16:0/627185558 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(225..225 src has 1..225) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f070408ab40 con 0x7f071c1f6d40 2023-10-16T13:46:27.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.754+0000 7f071a59c700 1 --2- 172.21.15.16:0/627185558 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f06fc06b020 0x7f06fc06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:27.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.754+0000 7f071a59c700 1 --2- 172.21.15.16:0/627185558 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f06fc06b020 0x7f06fc06d4e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f071c0cc670 tx=0x7f070c008040 comp rx=0 tx=0).ready entity=mgr.11190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:27.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:27.759+0000 7f0712ffd700 1 -- 172.21.15.16:0/627185558 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0704055ad0 con 0x7f071c1f6d40 2023-10-16T13:46:28.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:28.009+0000 7f072092f700 1 -- 172.21.15.16:0/627185558 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f071c0b6370 con 0x7f071c1f6d40 2023-10-16T13:46:28.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:28.590+0000 7f0712ffd700 1 -- 172.21.15.16:0/627185558 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v113) v1 ==== 122+0+0 (secure 0 0 0) 0x7f070402c0c0 con 0x7f071c1f6d40 2023-10-16T13:46:28.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:28.593+0000 7f072092f700 1 -- 172.21.15.16:0/627185558 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f06fc06b020 msgr2=0x7f06fc06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:28.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:28.593+0000 7f072092f700 1 --2- 172.21.15.16:0/627185558 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f06fc06b020 0x7f06fc06d4e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f071c0cc670 tx=0x7f070c008040 comp rx=0 tx=0).stop 2023-10-16T13:46:28.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:28.593+0000 7f072092f700 1 -- 172.21.15.16:0/627185558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f071c1f6d40 msgr2=0x7f071c14f650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:28.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:28.593+0000 7f072092f700 1 --2- 172.21.15.16:0/627185558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f071c1f6d40 0x7f071c14f650 secure :-1 s=READY pgs=2458 cs=0 l=1 rev1=1 crypto rx=0x7f070400e4b0 tx=0x7f07040047c0 comp rx=0 tx=0).stop 2023-10-16T13:46:28.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:28.593+0000 7f072092f700 1 -- 172.21.15.16:0/627185558 shutdown_connections 2023-10-16T13:46:28.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:28.593+0000 7f072092f700 1 --2- 172.21.15.16:0/627185558 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f071c0b4610 0x7f071c0b4a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:28.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:28.593+0000 7f072092f700 1 --2- 172.21.15.16:0/627185558 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0708004850 0x7f071c15ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:28.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:28.593+0000 7f072092f700 1 --2- 172.21.15.16:0/627185558 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f06fc06b020 0x7f06fc06d4e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:28.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:28.593+0000 7f072092f700 1 --2- 172.21.15.16:0/627185558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f071c1f6d40 0x7f071c14f650 unknown :-1 s=CLOSED pgs=2458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:28.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:28.593+0000 7f072092f700 1 -- 172.21.15.16:0/627185558 >> 172.21.15.16:0/627185558 conn(0x7f071c0b1f40 msgr2=0x7f071c15be30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:28.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:28.593+0000 7f072092f700 1 -- 172.21.15.16:0/627185558 shutdown_connections 2023-10-16T13:46:28.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:28.593+0000 7f072092f700 1 -- 172.21.15.16:0/627185558 wait complete. 2023-10-16T13:46:28.611 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-16T13:46:28.611 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:46:28.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:28.741+0000 7fe060582700 1 Processor -- start 2023-10-16T13:46:28.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:28.741+0000 7fe060582700 1 -- start start 2023-10-16T13:46:28.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:28.742+0000 7fe060582700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe05806a7e0 0x7fe05806ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:28.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:28.742+0000 7fe060582700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe05806b150 con 0x7fe0580cc8a0 2023-10-16T13:46:28.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:28.742+0000 7fe060582700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe058161c30 con 0x7fe05815b280 2023-10-16T13:46:28.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:28.742+0000 7fe060582700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe058161f10 con 0x7fe05806a7e0 2023-10-16T13:46:28.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:28.742+0000 7fe05eb1f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe05806a7e0 0x7fe05806ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:28.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:28.742+0000 7fe05db1d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe0580cc8a0 0x7fe05815e700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42452/0 (socket says 172.21.15.16:42452) 2023-10-16T13:46:28.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:28.742+0000 7fe05db1d700 1 -- 172.21.15.16:0/1530580126 learned_addr learned my addr 172.21.15.16:0/1530580126 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:46:28.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:28.742+0000 7fe05d31c700 1 -- 172.21.15.16:0/1530580126 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 66468978 0 0) 0x7fe05806b150 con 0x7fe0580cc8a0 2023-10-16T13:46:28.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:28.742+0000 7fe05d31c700 1 -- 172.21.15.16:0/1530580126 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0440035b0 con 0x7fe0580cc8a0 2023-10-16T13:46:28.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:28.742+0000 7fe05eb1f700 1 -- 172.21.15.16:0/1530580126 >> v1:172.21.15.49:6789/0 conn(0x7fe05815b280 legacy=0x7fe0581613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:28.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:28.742+0000 7fe05eb1f700 1 -- 172.21.15.16:0/1530580126 >> v1:172.21.15.16:6789/0 conn(0x7fe0580cc8a0 legacy=0x7fe05815e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:28.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:28.742+0000 7fe05eb1f700 1 -- 172.21.15.16:0/1530580126 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0581621f0 con 0x7fe05806a7e0 2023-10-16T13:46:28.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:28.743+0000 7fe05eb1f700 1 --2- 172.21.15.16:0/1530580126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe05806a7e0 0x7fe05806ac10 secure :-1 s=READY pgs=2460 cs=0 l=1 rev1=1 crypto rx=0x7fe050009d90 tx=0x7fe05000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a131b64d08af698 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:28.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:28.743+0000 7fe05d31c700 1 -- 172.21.15.16:0/1530580126 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe05000e070 con 0x7fe05806a7e0 2023-10-16T13:46:28.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:28.743+0000 7fe05d31c700 1 -- 172.21.15.16:0/1530580126 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe05000bb20 con 0x7fe05806a7e0 2023-10-16T13:46:28.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:28.743+0000 7fe05d31c700 1 -- 172.21.15.16:0/1530580126 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe050012430 con 0x7fe05806a7e0 2023-10-16T13:46:28.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:28.743+0000 7fe060582700 1 -- 172.21.15.16:0/1530580126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe05806a7e0 msgr2=0x7fe05806ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:28.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:28.743+0000 7fe060582700 1 --2- 172.21.15.16:0/1530580126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe05806a7e0 0x7fe05806ac10 secure :-1 s=READY pgs=2460 cs=0 l=1 rev1=1 crypto rx=0x7fe050009d90 tx=0x7fe05000b330 comp rx=0 tx=0).stop 2023-10-16T13:46:28.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:28.743+0000 7fe060582700 1 -- 172.21.15.16:0/1530580126 shutdown_connections 2023-10-16T13:46:28.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:28.743+0000 7fe060582700 1 --2- 172.21.15.16:0/1530580126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe05806a7e0 0x7fe05806ac10 unknown :-1 s=CLOSED pgs=2460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:28.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:28.743+0000 7fe060582700 1 -- 172.21.15.16:0/1530580126 >> 172.21.15.16:0/1530580126 conn(0x7fe0580b1fd0 msgr2=0x7fe0580b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:28.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:28.743+0000 7fe060582700 1 -- 172.21.15.16:0/1530580126 shutdown_connections 2023-10-16T13:46:28.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:28.743+0000 7fe060582700 1 -- 172.21.15.16:0/1530580126 wait complete. 2023-10-16T13:46:28.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:28.745+0000 7fe060582700 1 Processor -- start 2023-10-16T13:46:28.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:28.745+0000 7fe060582700 1 -- start start 2023-10-16T13:46:28.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:28.745+0000 7fe060582700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe05806a7e0 0x7fe0580c83f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:28.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:28.745+0000 7fe060582700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0580cc8a0 0x7fe05815fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:28.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:28.745+0000 7fe060582700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe05815b280 0x7fe0581600a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:28.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:28.745+0000 7fe060582700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe05815da20 con 0x7fe0580cc8a0 2023-10-16T13:46:28.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:28.745+0000 7fe060582700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0581f6100 con 0x7fe05815b280 2023-10-16T13:46:28.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:28.745+0000 7fe060582700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0581f63e0 con 0x7fe05806a7e0 2023-10-16T13:46:28.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:28.745+0000 7fe05db1d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0580cc8a0 0x7fe05815fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:28.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:28.746+0000 7fe05db1d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0580cc8a0 0x7fe05815fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53186/0 (socket says 172.21.15.16:53186) 2023-10-16T13:46:28.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:28.746+0000 7fe05db1d700 1 -- 172.21.15.16:0/1493556275 learned_addr learned my addr 172.21.15.16:0/1493556275 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:28.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:28.746+0000 7fe05eb1f700 1 --2- 172.21.15.16:0/1493556275 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe05815b280 0x7fe0581600a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:28.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:28.746+0000 7fe05e31e700 1 --2- 172.21.15.16:0/1493556275 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe05806a7e0 0x7fe0580c83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:28.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:28.746+0000 7fe05db1d700 1 -- 172.21.15.16:0/1493556275 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe05806a7e0 msgr2=0x7fe0580c83f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:28.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:28.746+0000 7fe05db1d700 1 --2- 172.21.15.16:0/1493556275 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe05806a7e0 0x7fe0580c83f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:28.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:28.746+0000 7fe05db1d700 1 -- 172.21.15.16:0/1493556275 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe05815b280 msgr2=0x7fe0581600a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:28.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:28.746+0000 7fe05db1d700 1 --2- 172.21.15.16:0/1493556275 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe05815b280 0x7fe0581600a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:28.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:28.746+0000 7fe05db1d700 1 -- 172.21.15.16:0/1493556275 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe050009a70 con 0x7fe0580cc8a0 2023-10-16T13:46:28.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:28.746+0000 7fe05eb1f700 1 --2- 172.21.15.16:0/1493556275 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe05815b280 0x7fe0581600a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:46:28.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:28.746+0000 7fe05e31e700 1 --2- 172.21.15.16:0/1493556275 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe05806a7e0 0x7fe0580c83f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:46:28.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:28.746+0000 7fe05db1d700 1 --2- 172.21.15.16:0/1493556275 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0580cc8a0 0x7fe05815fb60 secure :-1 s=READY pgs=2461 cs=0 l=1 rev1=1 crypto rx=0x7fe048005910 tx=0x7fe04800ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:28.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:28.746+0000 7fe04effd700 1 -- 172.21.15.16:0/1493556275 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe048005d70 con 0x7fe0580cc8a0 2023-10-16T13:46:28.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:28.746+0000 7fe04effd700 1 -- 172.21.15.16:0/1493556275 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0480070d0 con 0x7fe0580cc8a0 2023-10-16T13:46:28.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:28.746+0000 7fe060582700 1 -- 172.21.15.16:0/1493556275 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0581f66c0 con 0x7fe0580cc8a0 2023-10-16T13:46:28.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:28.746+0000 7fe04effd700 1 -- 172.21.15.16:0/1493556275 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe048007460 con 0x7fe0580cc8a0 2023-10-16T13:46:28.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:28.746+0000 7fe060582700 1 -- 172.21.15.16:0/1493556275 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe058158630 con 0x7fe0580cc8a0 2023-10-16T13:46:28.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:28.747+0000 7fe060582700 1 -- 172.21.15.16:0/1493556275 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0581f6ec0 con 0x7fe0580cc8a0 2023-10-16T13:46:28.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:28.748+0000 7fe04effd700 1 -- 172.21.15.16:0/1493556275 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 138) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe0480075c0 con 0x7fe0580cc8a0 2023-10-16T13:46:28.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:28.748+0000 7fe04effd700 1 --2- 172.21.15.16:0/1493556275 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fe03c06b020 0x7fe03c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:28.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:28.748+0000 7fe05e31e700 1 --2- 172.21.15.16:0/1493556275 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fe03c06b020 0x7fe03c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:28.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:28.748+0000 7fe04effd700 1 -- 172.21.15.16:0/1493556275 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(225..225 src has 1..225) v4 ==== 6157+0+0 (secure 0 0 0) 0x7fe048088ed0 con 0x7fe0580cc8a0 2023-10-16T13:46:28.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:28.749+0000 7fe05e31e700 1 --2- 172.21.15.16:0/1493556275 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fe03c06b020 0x7fe03c06d4e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe054000f60 tx=0x7fe054009040 comp rx=0 tx=0).ready entity=mgr.11190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:28.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:28.753+0000 7fe04effd700 1 -- 172.21.15.16:0/1493556275 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe0581f6ec0 con 0x7fe0580cc8a0 2023-10-16T13:46:29.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.003+0000 7fe060582700 1 -- 172.21.15.16:0/1493556275 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe0581f6ec0 con 0x7fe0580cc8a0 2023-10-16T13:46:29.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.003+0000 7fe04effd700 1 -- 172.21.15.16:0/1493556275 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) v1 ==== 95+0+4929 (secure 0 0 0) 0x7fe0581f6ec0 con 0x7fe0580cc8a0 2023-10-16T13:46:29.006 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:46:29.006 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":113,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10516,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/4065699432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":4065699432},{"type":"v1","addr":"172.21.15.49:6834","nonce":4065699432}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":108},{"gid":11415,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/1373048456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1373048456},{"type":"v1","addr":"172.21.15.16:6836","nonce":1373048456}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":108},{"gid":11418,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3860406853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":3860406853},{"type":"v1","addr":"172.21.15.16:6837","nonce":3860406853}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":107}],"filesystems":[{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:46:24.755313+0000","modified":"2023-10-16T13:46:28.011141+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11430},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11430":{"gid":11430,"name":"b","rank":0,"incarnation":109,"state":"up:active","state_seq":3,"addr":"172.21.15.49:6835/2905760119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":2905760119},{"type":"v1","addr":"172.21.15.49:6835","nonce":2905760119}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13}]} 2023-10-16T13:46:29.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.005+0000 7fe060582700 1 -- 172.21.15.16:0/1493556275 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fe03c06b020 msgr2=0x7fe03c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:29.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.005+0000 7fe060582700 1 --2- 172.21.15.16:0/1493556275 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fe03c06b020 0x7fe03c06d4e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe054000f60 tx=0x7fe054009040 comp rx=0 tx=0).stop 2023-10-16T13:46:29.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.005+0000 7fe060582700 1 -- 172.21.15.16:0/1493556275 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0580cc8a0 msgr2=0x7fe05815fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:29.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.005+0000 7fe060582700 1 --2- 172.21.15.16:0/1493556275 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0580cc8a0 0x7fe05815fb60 secure :-1 s=READY pgs=2461 cs=0 l=1 rev1=1 crypto rx=0x7fe048005910 tx=0x7fe04800ca90 comp rx=0 tx=0).stop 2023-10-16T13:46:29.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.006+0000 7fe060582700 1 -- 172.21.15.16:0/1493556275 shutdown_connections 2023-10-16T13:46:29.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.006+0000 7fe060582700 1 --2- 172.21.15.16:0/1493556275 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe05815b280 0x7fe0581600a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:29.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.006+0000 7fe060582700 1 --2- 172.21.15.16:0/1493556275 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe05806a7e0 0x7fe0580c83f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:29.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.006+0000 7fe060582700 1 --2- 172.21.15.16:0/1493556275 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fe03c06b020 0x7fe03c06d4e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:29.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.006+0000 7fe060582700 1 --2- 172.21.15.16:0/1493556275 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0580cc8a0 0x7fe05815fb60 unknown :-1 s=CLOSED pgs=2461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:29.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.006+0000 7fe060582700 1 -- 172.21.15.16:0/1493556275 >> 172.21.15.16:0/1493556275 conn(0x7fe0580b1fd0 msgr2=0x7fe05815a670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:29.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.006+0000 7fe060582700 1 -- 172.21.15.16:0/1493556275 shutdown_connections 2023-10-16T13:46:29.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.006+0000 7fe060582700 1 -- 172.21.15.16:0/1493556275 wait complete. 2023-10-16T13:46:29.011 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 113 2023-10-16T13:46:29.020 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:46:29.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.143+0000 7f6518913700 1 Processor -- start 2023-10-16T13:46:29.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.144+0000 7f6518913700 1 -- start start 2023-10-16T13:46:29.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.144+0000 7f6518913700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f65140b2b50 0x7f65140d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:29.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.144+0000 7f6518913700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f65140ce380 0x7f65140d09e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:29.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.144+0000 7f6518913700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65140d1210 con 0x7f65140cdfa0 2023-10-16T13:46:29.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.144+0000 7f6518913700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65140cbc30 con 0x7f65140ce380 2023-10-16T13:46:29.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.144+0000 7f6518913700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65140cbf10 con 0x7f65140b2b50 2023-10-16T13:46:29.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.145+0000 7f6512d9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f65140ce380 0x7f65140d09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:29.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.145+0000 7f6512d9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f65140ce380 0x7f65140d09e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53196/0 (socket says 172.21.15.16:53196) 2023-10-16T13:46:29.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.145+0000 7f651259c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f65140cdfa0 0x7f65140b2440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42466/0 (socket says 172.21.15.16:42466) 2023-10-16T13:46:29.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.145+0000 7f6512d9d700 1 -- 172.21.15.16:0/887807726 learned_addr learned my addr 172.21.15.16:0/887807726 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:29.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.145+0000 7f6511d9b700 1 --2- 172.21.15.16:0/887807726 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f65140b2b50 0x7f65140d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:29.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.145+0000 7f6512d9d700 1 -- 172.21.15.16:0/887807726 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f65140b2b50 msgr2=0x7f65140d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:29.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.145+0000 7f6512d9d700 1 --2- 172.21.15.16:0/887807726 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f65140b2b50 0x7f65140d04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:29.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.145+0000 7f6512d9d700 1 -- 172.21.15.16:0/887807726 >> v1:172.21.15.16:6789/0 conn(0x7f65140cdfa0 legacy=0x7f65140b2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:29.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.145+0000 7f651159a700 1 -- 172.21.15.16:0/887807726 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3940225106 0 0) 0x7f65140d1210 con 0x7f65140cdfa0 2023-10-16T13:46:29.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.145+0000 7f6512d9d700 1 -- 172.21.15.16:0/887807726 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65140cc1f0 con 0x7f65140ce380 2023-10-16T13:46:29.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.145+0000 7f6511d9b700 1 --2- 172.21.15.16:0/887807726 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f65140b2b50 0x7f65140d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:46:29.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.145+0000 7f6512d9d700 1 --2- 172.21.15.16:0/887807726 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65140ce380 0x7f65140d09e0 secure :-1 s=READY pgs=2463 cs=0 l=1 rev1=1 crypto rx=0x7f650c009e50 tx=0x7f650c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=999477c75e381f3f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:29.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.145+0000 7f651159a700 1 -- 172.21.15.16:0/887807726 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f650c00e070 con 0x7f65140ce380 2023-10-16T13:46:29.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.145+0000 7f651159a700 1 -- 172.21.15.16:0/887807726 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f650c00b9c0 con 0x7f65140ce380 2023-10-16T13:46:29.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.145+0000 7f651159a700 1 -- 172.21.15.16:0/887807726 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f650c00bd70 con 0x7f65140ce380 2023-10-16T13:46:29.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.146+0000 7f6518913700 1 -- 172.21.15.16:0/887807726 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65140ce380 msgr2=0x7f65140d09e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:29.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.146+0000 7f6518913700 1 --2- 172.21.15.16:0/887807726 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65140ce380 0x7f65140d09e0 secure :-1 s=READY pgs=2463 cs=0 l=1 rev1=1 crypto rx=0x7f650c009e50 tx=0x7f650c00b330 comp rx=0 tx=0).stop 2023-10-16T13:46:29.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.146+0000 7f6518913700 1 -- 172.21.15.16:0/887807726 shutdown_connections 2023-10-16T13:46:29.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.146+0000 7f6518913700 1 --2- 172.21.15.16:0/887807726 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65140ce380 0x7f65140d09e0 unknown :-1 s=CLOSED pgs=2463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:29.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.146+0000 7f6518913700 1 --2- 172.21.15.16:0/887807726 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f65140b2b50 0x7f65140d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:29.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.146+0000 7f6518913700 1 -- 172.21.15.16:0/887807726 >> 172.21.15.16:0/887807726 conn(0x7f65140d14b0 msgr2=0x7f65140d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:29.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.146+0000 7f6518913700 1 -- 172.21.15.16:0/887807726 shutdown_connections 2023-10-16T13:46:29.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.146+0000 7f6518913700 1 -- 172.21.15.16:0/887807726 wait complete. 2023-10-16T13:46:29.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.147+0000 7f6518913700 1 Processor -- start 2023-10-16T13:46:29.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.147+0000 7f6518913700 1 -- start start 2023-10-16T13:46:29.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.148+0000 7f6518913700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f65140b2b50 0x7f65140b60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:29.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.148+0000 7f6518913700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f65140cdfa0 0x7f65140b4a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:29.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.148+0000 7f6518913700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65140ce380 0x7f65140c5be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:29.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.148+0000 7f6518913700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65140b5c80 con 0x7f65140ce380 2023-10-16T13:46:29.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.148+0000 7f6518913700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65140b5000 con 0x7f65140b2b50 2023-10-16T13:46:29.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.148+0000 7f6518913700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65140b52e0 con 0x7f65140cdfa0 2023-10-16T13:46:29.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.148+0000 7f6512d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65140ce380 0x7f65140c5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:29.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.148+0000 7f6512d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65140ce380 0x7f65140c5be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53202/0 (socket says 172.21.15.16:53202) 2023-10-16T13:46:29.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.148+0000 7f6512d9d700 1 -- 172.21.15.16:0/2257052307 learned_addr learned my addr 172.21.15.16:0/2257052307 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:29.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.148+0000 7f651259c700 1 --2- 172.21.15.16:0/2257052307 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f65140b2b50 0x7f65140b60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:29.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.148+0000 7f6511d9b700 1 --2- 172.21.15.16:0/2257052307 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f65140cdfa0 0x7f65140b4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:29.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.149+0000 7f6512d9d700 1 -- 172.21.15.16:0/2257052307 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f65140cdfa0 msgr2=0x7f65140b4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:29.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.149+0000 7f6512d9d700 1 --2- 172.21.15.16:0/2257052307 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f65140cdfa0 0x7f65140b4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:29.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.149+0000 7f6512d9d700 1 -- 172.21.15.16:0/2257052307 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f65140b2b50 msgr2=0x7f65140b60a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:29.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.149+0000 7f6512d9d700 1 --2- 172.21.15.16:0/2257052307 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f65140b2b50 0x7f65140b60a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:29.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.149+0000 7f6512d9d700 1 -- 172.21.15.16:0/2257052307 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f650c009b30 con 0x7f65140ce380 2023-10-16T13:46:29.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.149+0000 7f6511d9b700 1 --2- 172.21.15.16:0/2257052307 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f65140cdfa0 0x7f65140b4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:46:29.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.149+0000 7f6512d9d700 1 --2- 172.21.15.16:0/2257052307 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65140ce380 0x7f65140c5be0 secure :-1 s=READY pgs=2464 cs=0 l=1 rev1=1 crypto rx=0x7f650c00b730 tx=0x7f650c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:29.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.149+0000 7f650affd700 1 -- 172.21.15.16:0/2257052307 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f650c003ea0 con 0x7f65140ce380 2023-10-16T13:46:29.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.149+0000 7f650affd700 1 -- 172.21.15.16:0/2257052307 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f650c004960 con 0x7f65140ce380 2023-10-16T13:46:29.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.149+0000 7f650affd700 1 -- 172.21.15.16:0/2257052307 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f650c01b670 con 0x7f65140ce380 2023-10-16T13:46:29.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.149+0000 7f6518913700 1 -- 172.21.15.16:0/2257052307 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65140b55c0 con 0x7f65140ce380 2023-10-16T13:46:29.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.149+0000 7f6518913700 1 -- 172.21.15.16:0/2257052307 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65140c6120 con 0x7f65140ce380 2023-10-16T13:46:29.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.150+0000 7f6518913700 1 -- 172.21.15.16:0/2257052307 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f651404ebb0 con 0x7f65140ce380 2023-10-16T13:46:29.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.150+0000 7f650affd700 1 -- 172.21.15.16:0/2257052307 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 138) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f650c01b7d0 con 0x7f65140ce380 2023-10-16T13:46:29.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.151+0000 7f650affd700 1 --2- 172.21.15.16:0/2257052307 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f64f406b020 0x7f64f406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:29.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.151+0000 7f650affd700 1 -- 172.21.15.16:0/2257052307 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(225..225 src has 1..225) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f650c08b040 con 0x7f65140ce380 2023-10-16T13:46:29.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.151+0000 7f651259c700 1 --2- 172.21.15.16:0/2257052307 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f64f406b020 0x7f64f406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:29.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.151+0000 7f651259c700 1 --2- 172.21.15.16:0/2257052307 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f64f406b020 0x7f64f406d4e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f6504007fd0 tx=0x7f6504007de0 comp rx=0 tx=0).ready entity=mgr.11190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:29.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.155+0000 7f650affd700 1 -- 172.21.15.16:0/2257052307 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f650c055da0 con 0x7f65140ce380 2023-10-16T13:46:29.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.357+0000 7f6518913700 1 -- 172.21.15.16:0/2257052307 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f65140cd100 con 0x7f65140ce380 2023-10-16T13:46:29.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.358+0000 7f650affd700 1 -- 172.21.15.16:0/2257052307 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v225) v1 ==== 74+0+16881 (secure 0 0 0) 0x7f650c0593c0 con 0x7f65140ce380 2023-10-16T13:46:29.361 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:46:29.362 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":225,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:46:26.556006+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2023-10-16T13:46:22.489653+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"224","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2023-10-16T13:46:23.729509+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"225","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":223,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":223,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":223,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":223,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":223,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":223,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":223,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":223,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:46:29.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.361+0000 7f6518913700 1 -- 172.21.15.16:0/2257052307 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f64f406b020 msgr2=0x7f64f406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:29.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.361+0000 7f6518913700 1 --2- 172.21.15.16:0/2257052307 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f64f406b020 0x7f64f406d4e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f6504007fd0 tx=0x7f6504007de0 comp rx=0 tx=0).stop 2023-10-16T13:46:29.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.361+0000 7f6518913700 1 -- 172.21.15.16:0/2257052307 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65140ce380 msgr2=0x7f65140c5be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:29.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.361+0000 7f6518913700 1 --2- 172.21.15.16:0/2257052307 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65140ce380 0x7f65140c5be0 secure :-1 s=READY pgs=2464 cs=0 l=1 rev1=1 crypto rx=0x7f650c00b730 tx=0x7f650c004ad0 comp rx=0 tx=0).stop 2023-10-16T13:46:29.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.361+0000 7f6518913700 1 -- 172.21.15.16:0/2257052307 shutdown_connections 2023-10-16T13:46:29.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.361+0000 7f6518913700 1 --2- 172.21.15.16:0/2257052307 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f64f406b020 0x7f64f406d4e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:29.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.361+0000 7f6518913700 1 --2- 172.21.15.16:0/2257052307 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65140ce380 0x7f65140c5be0 unknown :-1 s=CLOSED pgs=2464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:29.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.361+0000 7f6518913700 1 --2- 172.21.15.16:0/2257052307 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f65140b2b50 0x7f65140b60a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:29.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.361+0000 7f6518913700 1 --2- 172.21.15.16:0/2257052307 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f65140cdfa0 0x7f65140b4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:29.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.361+0000 7f6518913700 1 -- 172.21.15.16:0/2257052307 >> 172.21.15.16:0/2257052307 conn(0x7f65140d14b0 msgr2=0x7f651406eba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:29.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.361+0000 7f6518913700 1 -- 172.21.15.16:0/2257052307 shutdown_connections 2023-10-16T13:46:29.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.361+0000 7f6518913700 1 -- 172.21.15.16:0/2257052307 wait complete. 2023-10-16T13:46:29.376 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-16T13:46:29.376 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T13:46:29.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.499+0000 7f8568d56700 1 Processor -- start 2023-10-16T13:46:29.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.499+0000 7f8568d56700 1 -- start start 2023-10-16T13:46:29.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.500+0000 7f8568d56700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f85640cdfa0 0x7f85640b2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:29.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.500+0000 7f8568d56700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85640d0fc0 con 0x7f85640d0bb0 2023-10-16T13:46:29.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.500+0000 7f8568d56700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85640d1210 con 0x7f85640ce380 2023-10-16T13:46:29.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.500+0000 7f8568d56700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85640cbc30 con 0x7f85640cdfa0 2023-10-16T13:46:29.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.500+0000 7f856259c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f85640cdfa0 0x7f85640b2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:29.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.500+0000 7f8562d9d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f85640d0bb0 0x7f856415d120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42480/0 (socket says 172.21.15.16:42480) 2023-10-16T13:46:29.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.500+0000 7f8562d9d700 1 -- 172.21.15.16:0/979950391 learned_addr learned my addr 172.21.15.16:0/979950391 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:46:29.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.500+0000 7f856259c700 1 --2- 172.21.15.16:0/979950391 >> v2:172.21.15.16:3300/0 conn(0x7f85640cdfa0 0x7f85640b2440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53206/0 (socket says 172.21.15.16:53206) 2023-10-16T13:46:29.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.501+0000 7f856259c700 1 -- 172.21.15.16:0/979950391 >> v1:172.21.15.49:6789/0 conn(0x7f85640ce380 legacy=0x7f85640d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:29.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.501+0000 7f856259c700 1 -- 172.21.15.16:0/979950391 >> v1:172.21.15.16:6789/0 conn(0x7f85640d0bb0 legacy=0x7f856415d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:29.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.501+0000 7f856159a700 1 -- 172.21.15.16:0/979950391 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3801962413 0 0) 0x7f85640d0fc0 con 0x7f85640d0bb0 2023-10-16T13:46:29.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.501+0000 7f856259c700 1 -- 172.21.15.16:0/979950391 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85640cbf10 con 0x7f85640cdfa0 2023-10-16T13:46:29.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.501+0000 7f856259c700 1 --2- 172.21.15.16:0/979950391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85640cdfa0 0x7f85640b2440 secure :-1 s=READY pgs=2466 cs=0 l=1 rev1=1 crypto rx=0x7f8558009840 tx=0x7f8558009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68941861834aa433 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:29.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.501+0000 7f856159a700 1 -- 172.21.15.16:0/979950391 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8558011830 con 0x7f85640cdfa0 2023-10-16T13:46:29.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.501+0000 7f856159a700 1 -- 172.21.15.16:0/979950391 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8558011990 con 0x7f85640cdfa0 2023-10-16T13:46:29.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.501+0000 7f856159a700 1 -- 172.21.15.16:0/979950391 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8558011d20 con 0x7f85640cdfa0 2023-10-16T13:46:29.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.501+0000 7f8568d56700 1 -- 172.21.15.16:0/979950391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85640cdfa0 msgr2=0x7f85640b2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:29.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.501+0000 7f8568d56700 1 --2- 172.21.15.16:0/979950391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85640cdfa0 0x7f85640b2440 secure :-1 s=READY pgs=2466 cs=0 l=1 rev1=1 crypto rx=0x7f8558009840 tx=0x7f8558009b50 comp rx=0 tx=0).stop 2023-10-16T13:46:29.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.502+0000 7f8568d56700 1 -- 172.21.15.16:0/979950391 shutdown_connections 2023-10-16T13:46:29.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.502+0000 7f8568d56700 1 --2- 172.21.15.16:0/979950391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85640cdfa0 0x7f85640b2440 unknown :-1 s=CLOSED pgs=2466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:29.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.502+0000 7f8568d56700 1 -- 172.21.15.16:0/979950391 >> 172.21.15.16:0/979950391 conn(0x7f85640d14b0 msgr2=0x7f85640d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:29.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.502+0000 7f8568d56700 1 -- 172.21.15.16:0/979950391 shutdown_connections 2023-10-16T13:46:29.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.502+0000 7f8568d56700 1 -- 172.21.15.16:0/979950391 wait complete. 2023-10-16T13:46:29.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.503+0000 7f8568d56700 1 Processor -- start 2023-10-16T13:46:29.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.503+0000 7f8568d56700 1 -- start start 2023-10-16T13:46:29.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.504+0000 7f8568d56700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f85640cdfa0 0x7f856409d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:29.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.504+0000 7f8568d56700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85640ce380 0x7f85640c7600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:29.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.504+0000 7f8568d56700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f85640d0bb0 0x7f856414ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:29.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.504+0000 7f8568d56700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f856409e460 con 0x7f85640ce380 2023-10-16T13:46:29.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.504+0000 7f8568d56700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f856409d5b0 con 0x7f85640cdfa0 2023-10-16T13:46:29.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.504+0000 7f8568d56700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85640c7b40 con 0x7f85640d0bb0 2023-10-16T13:46:29.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.504+0000 7f8561d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85640ce380 0x7f85640c7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:29.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.504+0000 7f8561d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85640ce380 0x7f85640c7600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53214/0 (socket says 172.21.15.16:53214) 2023-10-16T13:46:29.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.504+0000 7f8561d9b700 1 -- 172.21.15.16:0/3400292243 learned_addr learned my addr 172.21.15.16:0/3400292243 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:29.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.504+0000 7f856259c700 1 --2- 172.21.15.16:0/3400292243 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f85640cdfa0 0x7f856409d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:29.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.504+0000 7f8562d9d700 1 --2- 172.21.15.16:0/3400292243 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f85640d0bb0 0x7f856414ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:29.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.504+0000 7f8561d9b700 1 -- 172.21.15.16:0/3400292243 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f85640d0bb0 msgr2=0x7f856414ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:29.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.504+0000 7f8561d9b700 1 --2- 172.21.15.16:0/3400292243 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f85640d0bb0 0x7f856414ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:29.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.504+0000 7f8561d9b700 1 -- 172.21.15.16:0/3400292243 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f85640cdfa0 msgr2=0x7f856409d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:29.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.504+0000 7f8561d9b700 1 --2- 172.21.15.16:0/3400292243 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f85640cdfa0 0x7f856409d070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:29.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.504+0000 7f8561d9b700 1 -- 172.21.15.16:0/3400292243 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85580094a0 con 0x7f85640ce380 2023-10-16T13:46:29.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.504+0000 7f8561d9b700 1 --2- 172.21.15.16:0/3400292243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85640ce380 0x7f85640c7600 secure :-1 s=READY pgs=2467 cs=0 l=1 rev1=1 crypto rx=0x7f855400c4e0 tx=0x7f855400c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:29.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.504+0000 7f8552ffd700 1 -- 172.21.15.16:0/3400292243 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85540052d0 con 0x7f85640ce380 2023-10-16T13:46:29.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.505+0000 7f8552ffd700 1 -- 172.21.15.16:0/3400292243 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8554005430 con 0x7f85640ce380 2023-10-16T13:46:29.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.505+0000 7f8568d56700 1 -- 172.21.15.16:0/3400292243 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f856414f340 con 0x7f85640ce380 2023-10-16T13:46:29.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.505+0000 7f8552ffd700 1 -- 172.21.15.16:0/3400292243 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85540038c0 con 0x7f85640ce380 2023-10-16T13:46:29.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.505+0000 7f8568d56700 1 -- 172.21.15.16:0/3400292243 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f856409dac0 con 0x7f85640ce380 2023-10-16T13:46:29.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.505+0000 7f8568d56700 1 -- 172.21.15.16:0/3400292243 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f856409e5a0 con 0x7f85640ce380 2023-10-16T13:46:29.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.506+0000 7f8552ffd700 1 -- 172.21.15.16:0/3400292243 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 138) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8554003a20 con 0x7f85640ce380 2023-10-16T13:46:29.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.506+0000 7f8552ffd700 1 --2- 172.21.15.16:0/3400292243 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f854406b020 0x7f854406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:29.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.507+0000 7f8552ffd700 1 -- 172.21.15.16:0/3400292243 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(225..225 src has 1..225) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f8554088cb0 con 0x7f85640ce380 2023-10-16T13:46:29.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.507+0000 7f856259c700 1 --2- 172.21.15.16:0/3400292243 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f854406b020 0x7f854406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:29.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.507+0000 7f856259c700 1 --2- 172.21.15.16:0/3400292243 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f854406b020 0x7f854406d4e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f85580090b0 tx=0x7f855801c040 comp rx=0 tx=0).ready entity=mgr.11190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:29.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.511+0000 7f8552ffd700 1 -- 172.21.15.16:0/3400292243 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f856409e5a0 con 0x7f85640ce380 2023-10-16T13:46:29.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.711+0000 7f8568d56700 1 -- 172.21.15.16:0/3400292243 --> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f85640cc4d0 con 0x7f854406b020 2023-10-16T13:46:29.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.720+0000 7f8552ffd700 1 -- 172.21.15.16:0/3400292243 <== mgr.11190 v2:172.21.15.16:6832/3171422553 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289728 (secure 0 0 0) 0x7f8558011c00 con 0x7f854406b020 2023-10-16T13:46:29.726 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:46:29.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.723+0000 7f8568d56700 1 -- 172.21.15.16:0/3400292243 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f854406b020 msgr2=0x7f854406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:29.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.723+0000 7f8568d56700 1 --2- 172.21.15.16:0/3400292243 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f854406b020 0x7f854406d4e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f85580090b0 tx=0x7f855801c040 comp rx=0 tx=0).stop 2023-10-16T13:46:29.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.723+0000 7f8568d56700 1 -- 172.21.15.16:0/3400292243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85640ce380 msgr2=0x7f85640c7600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:29.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.723+0000 7f8568d56700 1 --2- 172.21.15.16:0/3400292243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85640ce380 0x7f85640c7600 secure :-1 s=READY pgs=2467 cs=0 l=1 rev1=1 crypto rx=0x7f855400c4e0 tx=0x7f855400c7f0 comp rx=0 tx=0).stop 2023-10-16T13:46:29.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.723+0000 7f8568d56700 1 -- 172.21.15.16:0/3400292243 shutdown_connections 2023-10-16T13:46:29.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.723+0000 7f8568d56700 1 --2- 172.21.15.16:0/3400292243 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f85640d0bb0 0x7f856414ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:29.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.723+0000 7f8568d56700 1 --2- 172.21.15.16:0/3400292243 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f85640cdfa0 0x7f856409d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:29.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.723+0000 7f8568d56700 1 --2- 172.21.15.16:0/3400292243 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f854406b020 0x7f854406d4e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:29.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.723+0000 7f8568d56700 1 --2- 172.21.15.16:0/3400292243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f85640ce380 0x7f85640c7600 unknown :-1 s=CLOSED pgs=2467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:29.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.723+0000 7f8568d56700 1 -- 172.21.15.16:0/3400292243 >> 172.21.15.16:0/3400292243 conn(0x7f85640d14b0 msgr2=0x7f85640cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:29.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.723+0000 7f8568d56700 1 -- 172.21.15.16:0/3400292243 shutdown_connections 2023-10-16T13:46:29.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.724+0000 7f8568d56700 1 -- 172.21.15.16:0/3400292243 wait complete. 2023-10-16T13:46:29.731 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-10-16T13:46:29.738 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-10-16T13:46:28.177392+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":623,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":386,"ondisk_log_size":386,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":103,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":385036,"kb_used_data":23112,"kb_used_omap":12,"kb_used_meta":361779,"kb_avail":754589684,"statfs":{"total":773094113280,"available":772699836416,"internally_reserved":0,"allocated":23666688,"data_stored":12716917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12740,"internal_metadata":370462268},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.858249"},"pg_stats":[{"pgid":"28.3f","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.590280+0000","last_change":"2023-10-16T13:46:25.582893+0000","last_active":"2023-10-16T13:46:25.590280+0000","last_peered":"2023-10-16T13:46:25.590280+0000","last_clean":"2023-10-16T13:46:25.590280+0000","last_became_active":"2023-10-16T13:46:25.581980+0000","last_became_peered":"2023-10-16T13:46:25.581980+0000","last_unstale":"2023-10-16T13:46:25.590280+0000","last_undegraded":"2023-10-16T13:46:25.590280+0000","last_fullsized":"2023-10-16T13:46:25.590280+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:21:54.809497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3e","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.579398+0000","last_change":"2023-10-16T13:46:25.570175+0000","last_active":"2023-10-16T13:46:25.579398+0000","last_peered":"2023-10-16T13:46:25.579398+0000","last_clean":"2023-10-16T13:46:25.579398+0000","last_became_active":"2023-10-16T13:46:25.569413+0000","last_became_peered":"2023-10-16T13:46:25.569413+0000","last_unstale":"2023-10-16T13:46:25.579398+0000","last_undegraded":"2023-10-16T13:46:25.579398+0000","last_fullsized":"2023-10-16T13:46:25.579398+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:00:39.897377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.3d","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.576214+0000","last_change":"2023-10-16T13:46:25.575243+0000","last_active":"2023-10-16T13:46:25.576214+0000","last_peered":"2023-10-16T13:46:25.576214+0000","last_clean":"2023-10-16T13:46:25.576214+0000","last_became_active":"2023-10-16T13:46:25.574483+0000","last_became_peered":"2023-10-16T13:46:25.574483+0000","last_unstale":"2023-10-16T13:46:25.576214+0000","last_undegraded":"2023-10-16T13:46:25.576214+0000","last_fullsized":"2023-10-16T13:46:25.576214+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:17:18.357714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.3c","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.583542+0000","last_change":"2023-10-16T13:46:25.578245+0000","last_active":"2023-10-16T13:46:25.583542+0000","last_peered":"2023-10-16T13:46:25.583542+0000","last_clean":"2023-10-16T13:46:25.583542+0000","last_became_active":"2023-10-16T13:46:25.577671+0000","last_became_peered":"2023-10-16T13:46:25.577671+0000","last_unstale":"2023-10-16T13:46:25.583542+0000","last_undegraded":"2023-10-16T13:46:25.583542+0000","last_fullsized":"2023-10-16T13:46:25.583542+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:31:15.200012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.3b","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.572946+0000","last_change":"2023-10-16T13:46:25.570352+0000","last_active":"2023-10-16T13:46:25.572946+0000","last_peered":"2023-10-16T13:46:25.572946+0000","last_clean":"2023-10-16T13:46:25.572946+0000","last_became_active":"2023-10-16T13:46:25.570035+0000","last_became_peered":"2023-10-16T13:46:25.570035+0000","last_unstale":"2023-10-16T13:46:25.572946+0000","last_undegraded":"2023-10-16T13:46:25.572946+0000","last_fullsized":"2023-10-16T13:46:25.572946+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:27:41.282540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.3a","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.571695+0000","last_change":"2023-10-16T13:46:25.568946+0000","last_active":"2023-10-16T13:46:25.571695+0000","last_peered":"2023-10-16T13:46:25.571695+0000","last_clean":"2023-10-16T13:46:25.571695+0000","last_became_active":"2023-10-16T13:46:25.568652+0000","last_became_peered":"2023-10-16T13:46:25.568652+0000","last_unstale":"2023-10-16T13:46:25.571695+0000","last_undegraded":"2023-10-16T13:46:25.571695+0000","last_fullsized":"2023-10-16T13:46:25.571695+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:10:39.170811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.39","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.574900+0000","last_change":"2023-10-16T13:46:25.573056+0000","last_active":"2023-10-16T13:46:25.574900+0000","last_peered":"2023-10-16T13:46:25.574900+0000","last_clean":"2023-10-16T13:46:25.574900+0000","last_became_active":"2023-10-16T13:46:25.572702+0000","last_became_peered":"2023-10-16T13:46:25.572702+0000","last_unstale":"2023-10-16T13:46:25.574900+0000","last_undegraded":"2023-10-16T13:46:25.574900+0000","last_fullsized":"2023-10-16T13:46:25.574900+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:45:49.954882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.38","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.579179+0000","last_change":"2023-10-16T13:46:25.576391+0000","last_active":"2023-10-16T13:46:25.579179+0000","last_peered":"2023-10-16T13:46:25.579179+0000","last_clean":"2023-10-16T13:46:25.579179+0000","last_became_active":"2023-10-16T13:46:25.575906+0000","last_became_peered":"2023-10-16T13:46:25.575906+0000","last_unstale":"2023-10-16T13:46:25.579179+0000","last_undegraded":"2023-10-16T13:46:25.579179+0000","last_fullsized":"2023-10-16T13:46:25.579179+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:57:24.329549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.37","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.583962+0000","last_change":"2023-10-16T13:46:25.583298+0000","last_active":"2023-10-16T13:46:25.583962+0000","last_peered":"2023-10-16T13:46:25.583962+0000","last_clean":"2023-10-16T13:46:25.583962+0000","last_became_active":"2023-10-16T13:46:25.582794+0000","last_became_peered":"2023-10-16T13:46:25.582794+0000","last_unstale":"2023-10-16T13:46:25.583962+0000","last_undegraded":"2023-10-16T13:46:25.583962+0000","last_fullsized":"2023-10-16T13:46:25.583962+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:26:58.168760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.36","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.593239+0000","last_change":"2023-10-16T13:46:25.585253+0000","last_active":"2023-10-16T13:46:25.593239+0000","last_peered":"2023-10-16T13:46:25.593239+0000","last_clean":"2023-10-16T13:46:25.593239+0000","last_became_active":"2023-10-16T13:46:25.584881+0000","last_became_peered":"2023-10-16T13:46:25.584881+0000","last_unstale":"2023-10-16T13:46:25.593239+0000","last_undegraded":"2023-10-16T13:46:25.593239+0000","last_fullsized":"2023-10-16T13:46:25.593239+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:43:07.323286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.35","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.580786+0000","last_change":"2023-10-16T13:46:25.578710+0000","last_active":"2023-10-16T13:46:25.580786+0000","last_peered":"2023-10-16T13:46:25.580786+0000","last_clean":"2023-10-16T13:46:25.580786+0000","last_became_active":"2023-10-16T13:46:25.578250+0000","last_became_peered":"2023-10-16T13:46:25.578250+0000","last_unstale":"2023-10-16T13:46:25.580786+0000","last_undegraded":"2023-10-16T13:46:25.580786+0000","last_fullsized":"2023-10-16T13:46:25.580786+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:47:31.681981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.34","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.576335+0000","last_change":"2023-10-16T13:46:25.575302+0000","last_active":"2023-10-16T13:46:25.576335+0000","last_peered":"2023-10-16T13:46:25.576335+0000","last_clean":"2023-10-16T13:46:25.576335+0000","last_became_active":"2023-10-16T13:46:25.575009+0000","last_became_peered":"2023-10-16T13:46:25.575009+0000","last_unstale":"2023-10-16T13:46:25.576335+0000","last_undegraded":"2023-10-16T13:46:25.576335+0000","last_fullsized":"2023-10-16T13:46:25.576335+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:57:04.481600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.33","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.583336+0000","last_change":"2023-10-16T13:46:25.576474+0000","last_active":"2023-10-16T13:46:25.583336+0000","last_peered":"2023-10-16T13:46:25.583336+0000","last_clean":"2023-10-16T13:46:25.583336+0000","last_became_active":"2023-10-16T13:46:25.576187+0000","last_became_peered":"2023-10-16T13:46:25.576187+0000","last_unstale":"2023-10-16T13:46:25.583336+0000","last_undegraded":"2023-10-16T13:46:25.583336+0000","last_fullsized":"2023-10-16T13:46:25.583336+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:26:58.168760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.32","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.584726+0000","last_change":"2023-10-16T13:46:25.581421+0000","last_active":"2023-10-16T13:46:25.584726+0000","last_peered":"2023-10-16T13:46:25.584726+0000","last_clean":"2023-10-16T13:46:25.584726+0000","last_became_active":"2023-10-16T13:46:25.580936+0000","last_became_peered":"2023-10-16T13:46:25.580936+0000","last_unstale":"2023-10-16T13:46:25.584726+0000","last_undegraded":"2023-10-16T13:46:25.584726+0000","last_fullsized":"2023-10-16T13:46:25.584726+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:22:57.732319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.31","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.580509+0000","last_change":"2023-10-16T13:46:25.577384+0000","last_active":"2023-10-16T13:46:25.580509+0000","last_peered":"2023-10-16T13:46:25.580509+0000","last_clean":"2023-10-16T13:46:25.580509+0000","last_became_active":"2023-10-16T13:46:25.577070+0000","last_became_peered":"2023-10-16T13:46:25.577070+0000","last_unstale":"2023-10-16T13:46:25.580509+0000","last_undegraded":"2023-10-16T13:46:25.580509+0000","last_fullsized":"2023-10-16T13:46:25.580509+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:44:11.618820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.30","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.573597+0000","last_change":"2023-10-16T13:46:25.570144+0000","last_active":"2023-10-16T13:46:25.573597+0000","last_peered":"2023-10-16T13:46:25.573597+0000","last_clean":"2023-10-16T13:46:25.573597+0000","last_became_active":"2023-10-16T13:46:25.569866+0000","last_became_peered":"2023-10-16T13:46:25.569866+0000","last_unstale":"2023-10-16T13:46:25.573597+0000","last_undegraded":"2023-10-16T13:46:25.573597+0000","last_fullsized":"2023-10-16T13:46:25.573597+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:04:47.313972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.2f","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.580110+0000","last_change":"2023-10-16T13:46:25.577983+0000","last_active":"2023-10-16T13:46:25.580110+0000","last_peered":"2023-10-16T13:46:25.580110+0000","last_clean":"2023-10-16T13:46:25.580110+0000","last_became_active":"2023-10-16T13:46:25.577680+0000","last_became_peered":"2023-10-16T13:46:25.577680+0000","last_unstale":"2023-10-16T13:46:25.580110+0000","last_undegraded":"2023-10-16T13:46:25.580110+0000","last_fullsized":"2023-10-16T13:46:25.580110+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:31:15.200012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.2e","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.573094+0000","last_change":"2023-10-16T13:46:25.567084+0000","last_active":"2023-10-16T13:46:25.573094+0000","last_peered":"2023-10-16T13:46:25.573094+0000","last_clean":"2023-10-16T13:46:25.573094+0000","last_became_active":"2023-10-16T13:46:25.566470+0000","last_became_peered":"2023-10-16T13:46:25.566470+0000","last_unstale":"2023-10-16T13:46:25.573094+0000","last_undegraded":"2023-10-16T13:46:25.573094+0000","last_fullsized":"2023-10-16T13:46:25.573094+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:41:30.230501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.2d","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.580018+0000","last_change":"2023-10-16T13:46:25.578104+0000","last_active":"2023-10-16T13:46:25.580018+0000","last_peered":"2023-10-16T13:46:25.580018+0000","last_clean":"2023-10-16T13:46:25.580018+0000","last_became_active":"2023-10-16T13:46:25.577833+0000","last_became_peered":"2023-10-16T13:46:25.577833+0000","last_unstale":"2023-10-16T13:46:25.580018+0000","last_undegraded":"2023-10-16T13:46:25.580018+0000","last_fullsized":"2023-10-16T13:46:25.580018+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:02:44.293532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2c","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.577162+0000","last_change":"2023-10-16T13:46:25.572889+0000","last_active":"2023-10-16T13:46:25.577162+0000","last_peered":"2023-10-16T13:46:25.577162+0000","last_clean":"2023-10-16T13:46:25.577162+0000","last_became_active":"2023-10-16T13:46:25.572611+0000","last_became_peered":"2023-10-16T13:46:25.572611+0000","last_unstale":"2023-10-16T13:46:25.577162+0000","last_undegraded":"2023-10-16T13:46:25.577162+0000","last_fullsized":"2023-10-16T13:46:25.577162+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:36:05.027611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2b","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.589925+0000","last_change":"2023-10-16T13:46:25.584056+0000","last_active":"2023-10-16T13:46:25.589925+0000","last_peered":"2023-10-16T13:46:25.589925+0000","last_clean":"2023-10-16T13:46:25.589925+0000","last_became_active":"2023-10-16T13:46:25.583758+0000","last_became_peered":"2023-10-16T13:46:25.583758+0000","last_unstale":"2023-10-16T13:46:25.589925+0000","last_undegraded":"2023-10-16T13:46:25.589925+0000","last_fullsized":"2023-10-16T13:46:25.589925+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:58:38.153452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2a","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.583764+0000","last_change":"2023-10-16T13:46:25.581238+0000","last_active":"2023-10-16T13:46:25.583764+0000","last_peered":"2023-10-16T13:46:25.583764+0000","last_clean":"2023-10-16T13:46:25.583764+0000","last_became_active":"2023-10-16T13:46:25.580555+0000","last_became_peered":"2023-10-16T13:46:25.580555+0000","last_unstale":"2023-10-16T13:46:25.583764+0000","last_undegraded":"2023-10-16T13:46:25.583764+0000","last_fullsized":"2023-10-16T13:46:25.583764+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:16:22.462066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.29","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.581349+0000","last_change":"2023-10-16T13:46:25.578743+0000","last_active":"2023-10-16T13:46:25.581349+0000","last_peered":"2023-10-16T13:46:25.581349+0000","last_clean":"2023-10-16T13:46:25.581349+0000","last_became_active":"2023-10-16T13:46:25.578447+0000","last_became_peered":"2023-10-16T13:46:25.578447+0000","last_unstale":"2023-10-16T13:46:25.581349+0000","last_undegraded":"2023-10-16T13:46:25.581349+0000","last_fullsized":"2023-10-16T13:46:25.581349+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:25:52.567353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.28","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.584554+0000","last_change":"2023-10-16T13:46:25.583046+0000","last_active":"2023-10-16T13:46:25.584554+0000","last_peered":"2023-10-16T13:46:25.584554+0000","last_clean":"2023-10-16T13:46:25.584554+0000","last_became_active":"2023-10-16T13:46:25.582711+0000","last_became_peered":"2023-10-16T13:46:25.582711+0000","last_unstale":"2023-10-16T13:46:25.584554+0000","last_undegraded":"2023-10-16T13:46:25.584554+0000","last_fullsized":"2023-10-16T13:46:25.584554+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:24:48.472015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.27","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.573342+0000","last_change":"2023-10-16T13:46:25.572081+0000","last_active":"2023-10-16T13:46:25.573342+0000","last_peered":"2023-10-16T13:46:25.573342+0000","last_clean":"2023-10-16T13:46:25.573342+0000","last_became_active":"2023-10-16T13:46:25.568456+0000","last_became_peered":"2023-10-16T13:46:25.568456+0000","last_unstale":"2023-10-16T13:46:25.573342+0000","last_undegraded":"2023-10-16T13:46:25.573342+0000","last_fullsized":"2023-10-16T13:46:25.573342+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:53:48.001931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.26","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.582249+0000","last_change":"2023-10-16T13:46:25.579701+0000","last_active":"2023-10-16T13:46:25.582249+0000","last_peered":"2023-10-16T13:46:25.582249+0000","last_clean":"2023-10-16T13:46:25.582249+0000","last_became_active":"2023-10-16T13:46:25.579378+0000","last_became_peered":"2023-10-16T13:46:25.579378+0000","last_unstale":"2023-10-16T13:46:25.582249+0000","last_undegraded":"2023-10-16T13:46:25.582249+0000","last_fullsized":"2023-10-16T13:46:25.582249+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:06:17.800392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.25","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.580766+0000","last_change":"2023-10-16T13:46:25.578298+0000","last_active":"2023-10-16T13:46:25.580766+0000","last_peered":"2023-10-16T13:46:25.580766+0000","last_clean":"2023-10-16T13:46:25.580766+0000","last_became_active":"2023-10-16T13:46:25.578057+0000","last_became_peered":"2023-10-16T13:46:25.578057+0000","last_unstale":"2023-10-16T13:46:25.580766+0000","last_undegraded":"2023-10-16T13:46:25.580766+0000","last_fullsized":"2023-10-16T13:46:25.580766+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:26:58.168760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.24","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.576314+0000","last_change":"2023-10-16T13:46:25.573728+0000","last_active":"2023-10-16T13:46:25.576314+0000","last_peered":"2023-10-16T13:46:25.576314+0000","last_clean":"2023-10-16T13:46:25.576314+0000","last_became_active":"2023-10-16T13:46:25.573427+0000","last_became_peered":"2023-10-16T13:46:25.573427+0000","last_unstale":"2023-10-16T13:46:25.576314+0000","last_undegraded":"2023-10-16T13:46:25.576314+0000","last_fullsized":"2023-10-16T13:46:25.576314+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:56:16.772087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.23","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.573957+0000","last_change":"2023-10-16T13:46:25.568230+0000","last_active":"2023-10-16T13:46:25.573957+0000","last_peered":"2023-10-16T13:46:25.573957+0000","last_clean":"2023-10-16T13:46:25.573957+0000","last_became_active":"2023-10-16T13:46:25.567927+0000","last_became_peered":"2023-10-16T13:46:25.567927+0000","last_unstale":"2023-10-16T13:46:25.573957+0000","last_undegraded":"2023-10-16T13:46:25.573957+0000","last_fullsized":"2023-10-16T13:46:25.573957+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:44:11.242845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.22","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.571779+0000","last_change":"2023-10-16T13:46:25.569845+0000","last_active":"2023-10-16T13:46:25.571779+0000","last_peered":"2023-10-16T13:46:25.571779+0000","last_clean":"2023-10-16T13:46:25.571779+0000","last_became_active":"2023-10-16T13:46:25.569524+0000","last_became_peered":"2023-10-16T13:46:25.569524+0000","last_unstale":"2023-10-16T13:46:25.571779+0000","last_undegraded":"2023-10-16T13:46:25.571779+0000","last_fullsized":"2023-10-16T13:46:25.571779+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:34:10.802361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.21","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.583525+0000","last_change":"2023-10-16T13:46:25.580726+0000","last_active":"2023-10-16T13:46:25.583525+0000","last_peered":"2023-10-16T13:46:25.583525+0000","last_clean":"2023-10-16T13:46:25.583525+0000","last_became_active":"2023-10-16T13:46:25.580423+0000","last_became_peered":"2023-10-16T13:46:25.580423+0000","last_unstale":"2023-10-16T13:46:25.583525+0000","last_undegraded":"2023-10-16T13:46:25.583525+0000","last_fullsized":"2023-10-16T13:46:25.583525+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:25:08.151545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.20","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.578259+0000","last_change":"2023-10-16T13:46:25.572825+0000","last_active":"2023-10-16T13:46:25.578259+0000","last_peered":"2023-10-16T13:46:25.578259+0000","last_clean":"2023-10-16T13:46:25.578259+0000","last_became_active":"2023-10-16T13:46:25.572507+0000","last_became_peered":"2023-10-16T13:46:25.572507+0000","last_unstale":"2023-10-16T13:46:25.578259+0000","last_undegraded":"2023-10-16T13:46:25.578259+0000","last_fullsized":"2023-10-16T13:46:25.578259+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:44:11.242845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.581295+0000","last_change":"2023-10-16T13:46:25.578414+0000","last_active":"2023-10-16T13:46:25.581295+0000","last_peered":"2023-10-16T13:46:25.581295+0000","last_clean":"2023-10-16T13:46:25.581295+0000","last_became_active":"2023-10-16T13:46:25.578111+0000","last_became_peered":"2023-10-16T13:46:25.578111+0000","last_unstale":"2023-10-16T13:46:25.581295+0000","last_undegraded":"2023-10-16T13:46:25.581295+0000","last_fullsized":"2023-10-16T13:46:25.581295+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:14:53.997941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.1f","version":"224'4","reported_seq":15,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.595125+0000","last_change":"2023-10-16T13:46:24.345142+0000","last_active":"2023-10-16T13:46:25.595125+0000","last_peered":"2023-10-16T13:46:25.595125+0000","last_clean":"2023-10-16T13:46:25.595125+0000","last_became_active":"2023-10-16T13:46:24.344816+0000","last_became_peered":"2023-10-16T13:46:24.344816+0000","last_unstale":"2023-10-16T13:46:25.595125+0000","last_undegraded":"2023-10-16T13:46:25.595125+0000","last_fullsized":"2023-10-16T13:46:25.595125+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:54:50.780055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.579816+0000","last_change":"2023-10-16T13:46:25.577384+0000","last_active":"2023-10-16T13:46:25.579816+0000","last_peered":"2023-10-16T13:46:25.579816+0000","last_clean":"2023-10-16T13:46:25.579816+0000","last_became_active":"2023-10-16T13:46:25.577071+0000","last_became_peered":"2023-10-16T13:46:25.577071+0000","last_unstale":"2023-10-16T13:46:25.579816+0000","last_undegraded":"2023-10-16T13:46:25.579816+0000","last_fullsized":"2023-10-16T13:46:25.579816+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:04:10.854058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1e","version":"224'1","reported_seq":12,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.594198+0000","last_change":"2023-10-16T13:46:24.340000+0000","last_active":"2023-10-16T13:46:25.594198+0000","last_peered":"2023-10-16T13:46:25.594198+0000","last_clean":"2023-10-16T13:46:25.594198+0000","last_became_active":"2023-10-16T13:46:24.339714+0000","last_became_peered":"2023-10-16T13:46:24.339714+0000","last_unstale":"2023-10-16T13:46:25.594198+0000","last_undegraded":"2023-10-16T13:46:25.594198+0000","last_fullsized":"2023-10-16T13:46:25.594198+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:21:04.619534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.579291+0000","last_change":"2023-10-16T13:46:25.576216+0000","last_active":"2023-10-16T13:46:25.579291+0000","last_peered":"2023-10-16T13:46:25.579291+0000","last_clean":"2023-10-16T13:46:25.579291+0000","last_became_active":"2023-10-16T13:46:25.575876+0000","last_became_peered":"2023-10-16T13:46:25.575876+0000","last_unstale":"2023-10-16T13:46:25.579291+0000","last_undegraded":"2023-10-16T13:46:25.579291+0000","last_fullsized":"2023-10-16T13:46:25.579291+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:29:01.405087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1d","version":"224'2","reported_seq":13,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.593539+0000","last_change":"2023-10-16T13:46:24.340608+0000","last_active":"2023-10-16T13:46:25.593539+0000","last_peered":"2023-10-16T13:46:25.593539+0000","last_clean":"2023-10-16T13:46:25.593539+0000","last_became_active":"2023-10-16T13:46:24.340189+0000","last_became_peered":"2023-10-16T13:46:24.340189+0000","last_unstale":"2023-10-16T13:46:25.593539+0000","last_undegraded":"2023-10-16T13:46:25.593539+0000","last_fullsized":"2023-10-16T13:46:25.593539+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:16:00.359557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1b","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.574063+0000","last_change":"2023-10-16T13:46:25.571278+0000","last_active":"2023-10-16T13:46:25.574063+0000","last_peered":"2023-10-16T13:46:25.574063+0000","last_clean":"2023-10-16T13:46:25.574063+0000","last_became_active":"2023-10-16T13:46:25.570849+0000","last_became_peered":"2023-10-16T13:46:25.570849+0000","last_unstale":"2023-10-16T13:46:25.574063+0000","last_undegraded":"2023-10-16T13:46:25.574063+0000","last_fullsized":"2023-10-16T13:46:25.574063+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:03:49.009125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1c","version":"224'2","reported_seq":13,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.596447+0000","last_change":"2023-10-16T13:46:24.340877+0000","last_active":"2023-10-16T13:46:25.596447+0000","last_peered":"2023-10-16T13:46:25.596447+0000","last_clean":"2023-10-16T13:46:25.596447+0000","last_became_active":"2023-10-16T13:46:24.340607+0000","last_became_peered":"2023-10-16T13:46:24.340607+0000","last_unstale":"2023-10-16T13:46:25.596447+0000","last_undegraded":"2023-10-16T13:46:25.596447+0000","last_fullsized":"2023-10-16T13:46:25.596447+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:16:31.012130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.571853+0000","last_change":"2023-10-16T13:46:25.569919+0000","last_active":"2023-10-16T13:46:25.571853+0000","last_peered":"2023-10-16T13:46:25.571853+0000","last_clean":"2023-10-16T13:46:25.571853+0000","last_became_active":"2023-10-16T13:46:25.569619+0000","last_became_peered":"2023-10-16T13:46:25.569619+0000","last_unstale":"2023-10-16T13:46:25.571853+0000","last_undegraded":"2023-10-16T13:46:25.571853+0000","last_fullsized":"2023-10-16T13:46:25.571853+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:27:58.479275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.1b","version":"224'1","reported_seq":12,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.593694+0000","last_change":"2023-10-16T13:46:24.343654+0000","last_active":"2023-10-16T13:46:25.593694+0000","last_peered":"2023-10-16T13:46:25.593694+0000","last_clean":"2023-10-16T13:46:25.593694+0000","last_became_active":"2023-10-16T13:46:24.343330+0000","last_became_peered":"2023-10-16T13:46:24.343330+0000","last_unstale":"2023-10-16T13:46:25.593694+0000","last_undegraded":"2023-10-16T13:46:25.593694+0000","last_fullsized":"2023-10-16T13:46:25.593694+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:47:30.680184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.572282+0000","last_change":"2023-10-16T13:46:25.571656+0000","last_active":"2023-10-16T13:46:25.572282+0000","last_peered":"2023-10-16T13:46:25.572282+0000","last_clean":"2023-10-16T13:46:25.572282+0000","last_became_active":"2023-10-16T13:46:25.571372+0000","last_became_peered":"2023-10-16T13:46:25.571372+0000","last_unstale":"2023-10-16T13:46:25.572282+0000","last_undegraded":"2023-10-16T13:46:25.572282+0000","last_fullsized":"2023-10-16T13:46:25.572282+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:10:48.517374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":11,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.564945+0000","last_change":"2023-10-16T13:46:24.343337+0000","last_active":"2023-10-16T13:46:25.564945+0000","last_peered":"2023-10-16T13:46:25.564945+0000","last_clean":"2023-10-16T13:46:25.564945+0000","last_became_active":"2023-10-16T13:46:24.343043+0000","last_became_peered":"2023-10-16T13:46:24.343043+0000","last_unstale":"2023-10-16T13:46:25.564945+0000","last_undegraded":"2023-10-16T13:46:25.564945+0000","last_fullsized":"2023-10-16T13:46:25.564945+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:16:31.012130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1f","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.573321+0000","last_change":"2023-10-16T13:46:25.571884+0000","last_active":"2023-10-16T13:46:25.573321+0000","last_peered":"2023-10-16T13:46:25.573321+0000","last_clean":"2023-10-16T13:46:25.573321+0000","last_became_active":"2023-10-16T13:46:25.571601+0000","last_became_peered":"2023-10-16T13:46:25.571601+0000","last_unstale":"2023-10-16T13:46:25.573321+0000","last_undegraded":"2023-10-16T13:46:25.573321+0000","last_fullsized":"2023-10-16T13:46:25.573321+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:35:32.750171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":11,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.563849+0000","last_change":"2023-10-16T13:46:24.338318+0000","last_active":"2023-10-16T13:46:25.563849+0000","last_peered":"2023-10-16T13:46:25.563849+0000","last_clean":"2023-10-16T13:46:25.563849+0000","last_became_active":"2023-10-16T13:46:24.337766+0000","last_became_peered":"2023-10-16T13:46:24.337766+0000","last_unstale":"2023-10-16T13:46:25.563849+0000","last_undegraded":"2023-10-16T13:46:25.563849+0000","last_fullsized":"2023-10-16T13:46:25.563849+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:25:07.149748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.575412+0000","last_change":"2023-10-16T13:46:25.574355+0000","last_active":"2023-10-16T13:46:25.575412+0000","last_peered":"2023-10-16T13:46:25.575412+0000","last_clean":"2023-10-16T13:46:25.575412+0000","last_became_active":"2023-10-16T13:46:25.574057+0000","last_became_peered":"2023-10-16T13:46:25.574057+0000","last_unstale":"2023-10-16T13:46:25.575412+0000","last_undegraded":"2023-10-16T13:46:25.575412+0000","last_fullsized":"2023-10-16T13:46:25.575412+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:19:04.312392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":11,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.567482+0000","last_change":"2023-10-16T13:46:24.346329+0000","last_active":"2023-10-16T13:46:25.567482+0000","last_peered":"2023-10-16T13:46:25.567482+0000","last_clean":"2023-10-16T13:46:25.567482+0000","last_became_active":"2023-10-16T13:46:24.346051+0000","last_became_peered":"2023-10-16T13:46:24.346051+0000","last_unstale":"2023-10-16T13:46:25.567482+0000","last_undegraded":"2023-10-16T13:46:25.567482+0000","last_fullsized":"2023-10-16T13:46:25.567482+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:14:11.499564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.573701+0000","last_change":"2023-10-16T13:46:25.572773+0000","last_active":"2023-10-16T13:46:25.573701+0000","last_peered":"2023-10-16T13:46:25.573701+0000","last_clean":"2023-10-16T13:46:25.573701+0000","last_became_active":"2023-10-16T13:46:25.572324+0000","last_became_peered":"2023-10-16T13:46:25.572324+0000","last_unstale":"2023-10-16T13:46:25.573701+0000","last_undegraded":"2023-10-16T13:46:25.573701+0000","last_fullsized":"2023-10-16T13:46:25.573701+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:16:32.013927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":11,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.567680+0000","last_change":"2023-10-16T13:46:24.344174+0000","last_active":"2023-10-16T13:46:25.567680+0000","last_peered":"2023-10-16T13:46:25.567680+0000","last_clean":"2023-10-16T13:46:25.567680+0000","last_became_active":"2023-10-16T13:46:24.343883+0000","last_became_peered":"2023-10-16T13:46:24.343883+0000","last_unstale":"2023-10-16T13:46:25.567680+0000","last_undegraded":"2023-10-16T13:46:25.567680+0000","last_fullsized":"2023-10-16T13:46:25.567680+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:53:47.000133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.580895+0000","last_change":"2023-10-16T13:46:25.580296+0000","last_active":"2023-10-16T13:46:25.580895+0000","last_peered":"2023-10-16T13:46:25.580895+0000","last_clean":"2023-10-16T13:46:25.580895+0000","last_became_active":"2023-10-16T13:46:25.579963+0000","last_became_peered":"2023-10-16T13:46:25.579963+0000","last_unstale":"2023-10-16T13:46:25.580895+0000","last_undegraded":"2023-10-16T13:46:25.580895+0000","last_fullsized":"2023-10-16T13:46:25.580895+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:26:58.870084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.15","version":"224'3","reported_seq":14,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.596696+0000","last_change":"2023-10-16T13:46:24.340273+0000","last_active":"2023-10-16T13:46:25.596696+0000","last_peered":"2023-10-16T13:46:25.596696+0000","last_clean":"2023-10-16T13:46:25.596696+0000","last_became_active":"2023-10-16T13:46:24.339939+0000","last_became_peered":"2023-10-16T13:46:24.339939+0000","last_unstale":"2023-10-16T13:46:25.596696+0000","last_undegraded":"2023-10-16T13:46:25.596696+0000","last_fullsized":"2023-10-16T13:46:25.596696+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:25:07.149748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.13","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.580537+0000","last_change":"2023-10-16T13:46:25.578929+0000","last_active":"2023-10-16T13:46:25.580537+0000","last_peered":"2023-10-16T13:46:25.580537+0000","last_clean":"2023-10-16T13:46:25.580537+0000","last_became_active":"2023-10-16T13:46:25.578431+0000","last_became_peered":"2023-10-16T13:46:25.578431+0000","last_unstale":"2023-10-16T13:46:25.580537+0000","last_undegraded":"2023-10-16T13:46:25.580537+0000","last_fullsized":"2023-10-16T13:46:25.580537+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:16:05.895498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.14","version":"224'6","reported_seq":17,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.596786+0000","last_change":"2023-10-16T13:46:24.347248+0000","last_active":"2023-10-16T13:46:25.596786+0000","last_peered":"2023-10-16T13:46:25.596786+0000","last_clean":"2023-10-16T13:46:25.596786+0000","last_became_active":"2023-10-16T13:46:24.346699+0000","last_became_peered":"2023-10-16T13:46:24.346699+0000","last_unstale":"2023-10-16T13:46:25.596786+0000","last_undegraded":"2023-10-16T13:46:25.596786+0000","last_fullsized":"2023-10-16T13:46:25.596786+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:15:37.760742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.14","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.568407+0000","last_change":"2023-10-16T13:46:25.566526+0000","last_active":"2023-10-16T13:46:25.568407+0000","last_peered":"2023-10-16T13:46:25.568407+0000","last_clean":"2023-10-16T13:46:25.568407+0000","last_became_active":"2023-10-16T13:46:25.566252+0000","last_became_peered":"2023-10-16T13:46:25.566252+0000","last_unstale":"2023-10-16T13:46:25.568407+0000","last_undegraded":"2023-10-16T13:46:25.568407+0000","last_fullsized":"2023-10-16T13:46:25.568407+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:45:30.778861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.13","version":"224'2","reported_seq":13,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.595410+0000","last_change":"2023-10-16T13:46:24.333306+0000","last_active":"2023-10-16T13:46:25.595410+0000","last_peered":"2023-10-16T13:46:25.595410+0000","last_clean":"2023-10-16T13:46:25.595410+0000","last_became_active":"2023-10-16T13:46:24.333042+0000","last_became_peered":"2023-10-16T13:46:24.333042+0000","last_unstale":"2023-10-16T13:46:25.595410+0000","last_undegraded":"2023-10-16T13:46:25.595410+0000","last_fullsized":"2023-10-16T13:46:25.595410+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:44:45.300765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.578222+0000","last_change":"2023-10-16T13:46:25.575462+0000","last_active":"2023-10-16T13:46:25.578222+0000","last_peered":"2023-10-16T13:46:25.578222+0000","last_clean":"2023-10-16T13:46:25.578222+0000","last_became_active":"2023-10-16T13:46:25.574929+0000","last_became_peered":"2023-10-16T13:46:25.574929+0000","last_unstale":"2023-10-16T13:46:25.578222+0000","last_undegraded":"2023-10-16T13:46:25.578222+0000","last_fullsized":"2023-10-16T13:46:25.578222+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:26:58.870084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":11,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.562921+0000","last_change":"2023-10-16T13:46:24.336805+0000","last_active":"2023-10-16T13:46:25.562921+0000","last_peered":"2023-10-16T13:46:25.562921+0000","last_clean":"2023-10-16T13:46:25.562921+0000","last_became_active":"2023-10-16T13:46:24.336473+0000","last_became_peered":"2023-10-16T13:46:24.336473+0000","last_unstale":"2023-10-16T13:46:25.562921+0000","last_undegraded":"2023-10-16T13:46:25.562921+0000","last_fullsized":"2023-10-16T13:46:25.562921+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:44:10.241047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.576844+0000","last_change":"2023-10-16T13:46:25.573821+0000","last_active":"2023-10-16T13:46:25.576844+0000","last_peered":"2023-10-16T13:46:25.576844+0000","last_clean":"2023-10-16T13:46:25.576844+0000","last_became_active":"2023-10-16T13:46:25.572882+0000","last_became_peered":"2023-10-16T13:46:25.572882+0000","last_unstale":"2023-10-16T13:46:25.576844+0000","last_undegraded":"2023-10-16T13:46:25.576844+0000","last_fullsized":"2023-10-16T13:46:25.576844+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:10:39.170811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":11,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.558466+0000","last_change":"2023-10-16T13:46:24.332386+0000","last_active":"2023-10-16T13:46:25.558466+0000","last_peered":"2023-10-16T13:46:25.558466+0000","last_clean":"2023-10-16T13:46:25.558466+0000","last_became_active":"2023-10-16T13:46:24.332013+0000","last_became_peered":"2023-10-16T13:46:24.332013+0000","last_unstale":"2023-10-16T13:46:25.558466+0000","last_undegraded":"2023-10-16T13:46:25.558466+0000","last_fullsized":"2023-10-16T13:46:25.558466+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:04:46.312175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.586582+0000","last_change":"2023-10-16T13:46:25.584003+0000","last_active":"2023-10-16T13:46:25.586582+0000","last_peered":"2023-10-16T13:46:25.586582+0000","last_clean":"2023-10-16T13:46:25.586582+0000","last_became_active":"2023-10-16T13:46:25.583732+0000","last_became_peered":"2023-10-16T13:46:25.583732+0000","last_unstale":"2023-10-16T13:46:25.586582+0000","last_undegraded":"2023-10-16T13:46:25.586582+0000","last_fullsized":"2023-10-16T13:46:25.586582+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:17:18.357714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.10","version":"224'8","reported_seq":19,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.596537+0000","last_change":"2023-10-16T13:46:24.346086+0000","last_active":"2023-10-16T13:46:25.596537+0000","last_peered":"2023-10-16T13:46:25.596537+0000","last_clean":"2023-10-16T13:46:25.596537+0000","last_became_active":"2023-10-16T13:46:24.345786+0000","last_became_peered":"2023-10-16T13:46:24.345786+0000","last_unstale":"2023-10-16T13:46:25.596537+0000","last_undegraded":"2023-10-16T13:46:25.596537+0000","last_fullsized":"2023-10-16T13:46:25.596537+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:39:19.890170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.593434+0000","last_change":"2023-10-16T13:46:25.585651+0000","last_active":"2023-10-16T13:46:25.593434+0000","last_peered":"2023-10-16T13:46:25.593434+0000","last_clean":"2023-10-16T13:46:25.593434+0000","last_became_active":"2023-10-16T13:46:25.585367+0000","last_became_peered":"2023-10-16T13:46:25.585367+0000","last_unstale":"2023-10-16T13:46:25.593434+0000","last_undegraded":"2023-10-16T13:46:25.593434+0000","last_fullsized":"2023-10-16T13:46:25.593434+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:16:03.750659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.f","version":"224'1","reported_seq":12,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.595561+0000","last_change":"2023-10-16T13:46:24.338988+0000","last_active":"2023-10-16T13:46:25.595561+0000","last_peered":"2023-10-16T13:46:25.595561+0000","last_clean":"2023-10-16T13:46:25.595561+0000","last_became_active":"2023-10-16T13:46:24.338477+0000","last_became_peered":"2023-10-16T13:46:24.338477+0000","last_unstale":"2023-10-16T13:46:25.595561+0000","last_undegraded":"2023-10-16T13:46:25.595561+0000","last_fullsized":"2023-10-16T13:46:25.595561+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:47:21.411665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.579844+0000","last_change":"2023-10-16T13:46:25.578233+0000","last_active":"2023-10-16T13:46:25.579844+0000","last_peered":"2023-10-16T13:46:25.579844+0000","last_clean":"2023-10-16T13:46:25.579844+0000","last_became_active":"2023-10-16T13:46:25.577908+0000","last_became_peered":"2023-10-16T13:46:25.577908+0000","last_unstale":"2023-10-16T13:46:25.579844+0000","last_undegraded":"2023-10-16T13:46:25.579844+0000","last_fullsized":"2023-10-16T13:46:25.579844+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:19:05.170819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.e","version":"224'2","reported_seq":13,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.592651+0000","last_change":"2023-10-16T13:46:24.339351+0000","last_active":"2023-10-16T13:46:25.592651+0000","last_peered":"2023-10-16T13:46:25.592651+0000","last_clean":"2023-10-16T13:46:25.592651+0000","last_became_active":"2023-10-16T13:46:24.339062+0000","last_became_peered":"2023-10-16T13:46:24.339062+0000","last_unstale":"2023-10-16T13:46:25.592651+0000","last_undegraded":"2023-10-16T13:46:25.592651+0000","last_fullsized":"2023-10-16T13:46:25.592651+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:14:49.845621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.587605+0000","last_change":"2023-10-16T13:46:25.582651+0000","last_active":"2023-10-16T13:46:25.587605+0000","last_peered":"2023-10-16T13:46:25.587605+0000","last_clean":"2023-10-16T13:46:25.587605+0000","last_became_active":"2023-10-16T13:46:25.582350+0000","last_became_peered":"2023-10-16T13:46:25.582350+0000","last_unstale":"2023-10-16T13:46:25.587605+0000","last_undegraded":"2023-10-16T13:46:25.587605+0000","last_fullsized":"2023-10-16T13:46:25.587605+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:23:29.519783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.d","version":"224'1","reported_seq":12,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.595657+0000","last_change":"2023-10-16T13:46:24.338316+0000","last_active":"2023-10-16T13:46:25.595657+0000","last_peered":"2023-10-16T13:46:25.595657+0000","last_clean":"2023-10-16T13:46:25.595657+0000","last_became_active":"2023-10-16T13:46:24.338002+0000","last_became_peered":"2023-10-16T13:46:24.338002+0000","last_unstale":"2023-10-16T13:46:25.595657+0000","last_undegraded":"2023-10-16T13:46:25.595657+0000","last_fullsized":"2023-10-16T13:46:25.595657+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:04:35.398533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.580910+0000","last_change":"2023-10-16T13:46:25.579683+0000","last_active":"2023-10-16T13:46:25.580910+0000","last_peered":"2023-10-16T13:46:25.580910+0000","last_clean":"2023-10-16T13:46:25.580910+0000","last_became_active":"2023-10-16T13:46:25.579398+0000","last_became_peered":"2023-10-16T13:46:25.579398+0000","last_unstale":"2023-10-16T13:46:25.580910+0000","last_undegraded":"2023-10-16T13:46:25.580910+0000","last_fullsized":"2023-10-16T13:46:25.580910+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:34:10.802361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":11,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.564021+0000","last_change":"2023-10-16T13:46:24.339475+0000","last_active":"2023-10-16T13:46:25.564021+0000","last_peered":"2023-10-16T13:46:25.564021+0000","last_clean":"2023-10-16T13:46:25.564021+0000","last_became_active":"2023-10-16T13:46:24.339105+0000","last_became_peered":"2023-10-16T13:46:24.339105+0000","last_unstale":"2023-10-16T13:46:25.564021+0000","last_undegraded":"2023-10-16T13:46:25.564021+0000","last_fullsized":"2023-10-16T13:46:25.564021+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:13:27.085544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.579261+0000","last_change":"2023-10-16T13:46:25.572423+0000","last_active":"2023-10-16T13:46:25.579261+0000","last_peered":"2023-10-16T13:46:25.579261+0000","last_clean":"2023-10-16T13:46:25.579261+0000","last_became_active":"2023-10-16T13:46:25.569272+0000","last_became_peered":"2023-10-16T13:46:25.569272+0000","last_unstale":"2023-10-16T13:46:25.579261+0000","last_undegraded":"2023-10-16T13:46:25.579261+0000","last_fullsized":"2023-10-16T13:46:25.579261+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:45:10.250543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.b","version":"224'2","reported_seq":13,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.595169+0000","last_change":"2023-10-16T13:46:24.347651+0000","last_active":"2023-10-16T13:46:25.595169+0000","last_peered":"2023-10-16T13:46:25.595169+0000","last_clean":"2023-10-16T13:46:25.595169+0000","last_became_active":"2023-10-16T13:46:24.347370+0000","last_became_peered":"2023-10-16T13:46:24.347370+0000","last_unstale":"2023-10-16T13:46:25.595169+0000","last_undegraded":"2023-10-16T13:46:25.595169+0000","last_fullsized":"2023-10-16T13:46:25.595169+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:18:32.988431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.583510+0000","last_change":"2023-10-16T13:46:25.580894+0000","last_active":"2023-10-16T13:46:25.583510+0000","last_peered":"2023-10-16T13:46:25.583510+0000","last_clean":"2023-10-16T13:46:25.583510+0000","last_became_active":"2023-10-16T13:46:25.580589+0000","last_became_peered":"2023-10-16T13:46:25.580589+0000","last_unstale":"2023-10-16T13:46:25.583510+0000","last_undegraded":"2023-10-16T13:46:25.583510+0000","last_fullsized":"2023-10-16T13:46:25.583510+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:26:58.870084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":11,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.559163+0000","last_change":"2023-10-16T13:46:24.337053+0000","last_active":"2023-10-16T13:46:25.559163+0000","last_peered":"2023-10-16T13:46:25.559163+0000","last_clean":"2023-10-16T13:46:25.559163+0000","last_became_active":"2023-10-16T13:46:24.336736+0000","last_became_peered":"2023-10-16T13:46:24.336736+0000","last_unstale":"2023-10-16T13:46:25.559163+0000","last_undegraded":"2023-10-16T13:46:25.559163+0000","last_fullsized":"2023-10-16T13:46:25.559163+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:36:24.043336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.573223+0000","last_change":"2023-10-16T13:46:25.570388+0000","last_active":"2023-10-16T13:46:25.573223+0000","last_peered":"2023-10-16T13:46:25.573223+0000","last_clean":"2023-10-16T13:46:25.573223+0000","last_became_active":"2023-10-16T13:46:25.570059+0000","last_became_peered":"2023-10-16T13:46:25.570059+0000","last_unstale":"2023-10-16T13:46:25.573223+0000","last_undegraded":"2023-10-16T13:46:25.573223+0000","last_fullsized":"2023-10-16T13:46:25.573223+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:18:48.273122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":11,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.567409+0000","last_change":"2023-10-16T13:46:24.344523+0000","last_active":"2023-10-16T13:46:25.567409+0000","last_peered":"2023-10-16T13:46:25.567409+0000","last_clean":"2023-10-16T13:46:25.567409+0000","last_became_active":"2023-10-16T13:46:24.344238+0000","last_became_peered":"2023-10-16T13:46:24.344238+0000","last_unstale":"2023-10-16T13:46:25.567409+0000","last_undegraded":"2023-10-16T13:46:25.567409+0000","last_fullsized":"2023-10-16T13:46:25.567409+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:37:49.931017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.580131+0000","last_change":"2023-10-16T13:46:25.576886+0000","last_active":"2023-10-16T13:46:25.580131+0000","last_peered":"2023-10-16T13:46:25.580131+0000","last_clean":"2023-10-16T13:46:25.580131+0000","last_became_active":"2023-10-16T13:46:25.576540+0000","last_became_peered":"2023-10-16T13:46:25.576540+0000","last_unstale":"2023-10-16T13:46:25.580131+0000","last_undegraded":"2023-10-16T13:46:25.580131+0000","last_fullsized":"2023-10-16T13:46:25.580131+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:10:48.517374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":11,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.565462+0000","last_change":"2023-10-16T13:46:24.341910+0000","last_active":"2023-10-16T13:46:25.565462+0000","last_peered":"2023-10-16T13:46:25.565462+0000","last_clean":"2023-10-16T13:46:25.565462+0000","last_became_active":"2023-10-16T13:46:24.341633+0000","last_became_peered":"2023-10-16T13:46:24.341633+0000","last_unstale":"2023-10-16T13:46:25.565462+0000","last_undegraded":"2023-10-16T13:46:25.565462+0000","last_fullsized":"2023-10-16T13:46:25.565462+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:00:55.319505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.593421+0000","last_change":"2023-10-16T13:46:25.585631+0000","last_active":"2023-10-16T13:46:25.593421+0000","last_peered":"2023-10-16T13:46:25.593421+0000","last_clean":"2023-10-16T13:46:25.593421+0000","last_became_active":"2023-10-16T13:46:25.585329+0000","last_became_peered":"2023-10-16T13:46:25.585329+0000","last_unstale":"2023-10-16T13:46:25.593421+0000","last_undegraded":"2023-10-16T13:46:25.593421+0000","last_fullsized":"2023-10-16T13:46:25.593421+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:25:56.358129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.7","version":"224'2","reported_seq":13,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.595993+0000","last_change":"2023-10-16T13:46:24.344410+0000","last_active":"2023-10-16T13:46:25.595993+0000","last_peered":"2023-10-16T13:46:25.595993+0000","last_clean":"2023-10-16T13:46:25.595993+0000","last_became_active":"2023-10-16T13:46:24.343665+0000","last_became_peered":"2023-10-16T13:46:24.343665+0000","last_unstale":"2023-10-16T13:46:25.595993+0000","last_undegraded":"2023-10-16T13:46:25.595993+0000","last_fullsized":"2023-10-16T13:46:25.595993+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:21:31.259195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.582757+0000","last_change":"2023-10-16T13:46:25.577247+0000","last_active":"2023-10-16T13:46:25.582757+0000","last_peered":"2023-10-16T13:46:25.582757+0000","last_clean":"2023-10-16T13:46:25.582757+0000","last_became_active":"2023-10-16T13:46:25.576949+0000","last_became_peered":"2023-10-16T13:46:25.576949+0000","last_unstale":"2023-10-16T13:46:25.582757+0000","last_undegraded":"2023-10-16T13:46:25.582757+0000","last_fullsized":"2023-10-16T13:46:25.582757+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:31:15.200012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":11,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.563517+0000","last_change":"2023-10-16T13:46:24.346027+0000","last_active":"2023-10-16T13:46:25.563517+0000","last_peered":"2023-10-16T13:46:25.563517+0000","last_clean":"2023-10-16T13:46:25.563517+0000","last_became_active":"2023-10-16T13:46:24.345748+0000","last_became_peered":"2023-10-16T13:46:24.345748+0000","last_unstale":"2023-10-16T13:46:25.563517+0000","last_undegraded":"2023-10-16T13:46:25.563517+0000","last_fullsized":"2023-10-16T13:46:25.563517+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:23:29.526203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.572900+0000","last_change":"2023-10-16T13:46:25.567174+0000","last_active":"2023-10-16T13:46:25.572900+0000","last_peered":"2023-10-16T13:46:25.572900+0000","last_clean":"2023-10-16T13:46:25.572900+0000","last_became_active":"2023-10-16T13:46:25.566392+0000","last_became_peered":"2023-10-16T13:46:25.566392+0000","last_unstale":"2023-10-16T13:46:25.572900+0000","last_undegraded":"2023-10-16T13:46:25.572900+0000","last_fullsized":"2023-10-16T13:46:25.572900+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:47:31.681981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":11,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.569201+0000","last_change":"2023-10-16T13:46:24.340274+0000","last_active":"2023-10-16T13:46:25.569201+0000","last_peered":"2023-10-16T13:46:25.569201+0000","last_clean":"2023-10-16T13:46:25.569201+0000","last_became_active":"2023-10-16T13:46:24.339988+0000","last_became_peered":"2023-10-16T13:46:24.339988+0000","last_unstale":"2023-10-16T13:46:25.569201+0000","last_undegraded":"2023-10-16T13:46:25.569201+0000","last_fullsized":"2023-10-16T13:46:25.569201+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:48:44.887565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.582283+0000","last_change":"2023-10-16T13:46:25.580242+0000","last_active":"2023-10-16T13:46:25.582283+0000","last_peered":"2023-10-16T13:46:25.582283+0000","last_clean":"2023-10-16T13:46:25.582283+0000","last_became_active":"2023-10-16T13:46:25.579959+0000","last_became_peered":"2023-10-16T13:46:25.579959+0000","last_unstale":"2023-10-16T13:46:25.582283+0000","last_undegraded":"2023-10-16T13:46:25.582283+0000","last_fullsized":"2023-10-16T13:46:25.582283+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:34:14.799734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":11,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.564799+0000","last_change":"2023-10-16T13:46:24.333052+0000","last_active":"2023-10-16T13:46:25.564799+0000","last_peered":"2023-10-16T13:46:25.564799+0000","last_clean":"2023-10-16T13:46:25.564799+0000","last_became_active":"2023-10-16T13:46:24.332751+0000","last_became_peered":"2023-10-16T13:46:24.332751+0000","last_unstale":"2023-10-16T13:46:25.564799+0000","last_undegraded":"2023-10-16T13:46:25.564799+0000","last_fullsized":"2023-10-16T13:46:25.564799+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:29:38.839702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.579220+0000","last_change":"2023-10-16T13:46:25.576269+0000","last_active":"2023-10-16T13:46:25.579220+0000","last_peered":"2023-10-16T13:46:25.579220+0000","last_clean":"2023-10-16T13:46:25.579220+0000","last_became_active":"2023-10-16T13:46:25.575732+0000","last_became_peered":"2023-10-16T13:46:25.575732+0000","last_unstale":"2023-10-16T13:46:25.579220+0000","last_undegraded":"2023-10-16T13:46:25.579220+0000","last_fullsized":"2023-10-16T13:46:25.579220+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:05:54.303051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":11,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.568934+0000","last_change":"2023-10-16T13:46:24.338016+0000","last_active":"2023-10-16T13:46:25.568934+0000","last_peered":"2023-10-16T13:46:25.568934+0000","last_clean":"2023-10-16T13:46:25.568934+0000","last_became_active":"2023-10-16T13:46:24.337687+0000","last_became_peered":"2023-10-16T13:46:24.337687+0000","last_unstale":"2023-10-16T13:46:25.568934+0000","last_undegraded":"2023-10-16T13:46:25.568934+0000","last_fullsized":"2023-10-16T13:46:25.568934+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:27:57.477477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.583491+0000","last_change":"2023-10-16T13:46:25.575797+0000","last_active":"2023-10-16T13:46:25.583491+0000","last_peered":"2023-10-16T13:46:25.583491+0000","last_clean":"2023-10-16T13:46:25.583491+0000","last_became_active":"2023-10-16T13:46:25.575494+0000","last_became_peered":"2023-10-16T13:46:25.575494+0000","last_unstale":"2023-10-16T13:46:25.583491+0000","last_undegraded":"2023-10-16T13:46:25.583491+0000","last_fullsized":"2023-10-16T13:46:25.583491+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:35:32.750171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":11,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.559508+0000","last_change":"2023-10-16T13:46:24.340328+0000","last_active":"2023-10-16T13:46:25.559508+0000","last_peered":"2023-10-16T13:46:25.559508+0000","last_clean":"2023-10-16T13:46:25.559508+0000","last_became_active":"2023-10-16T13:46:24.340042+0000","last_became_peered":"2023-10-16T13:46:24.340042+0000","last_unstale":"2023-10-16T13:46:25.559508+0000","last_undegraded":"2023-10-16T13:46:25.559508+0000","last_fullsized":"2023-10-16T13:46:25.559508+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:15:37.760742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.584250+0000","last_change":"2023-10-16T13:46:25.577347+0000","last_active":"2023-10-16T13:46:25.584250+0000","last_peered":"2023-10-16T13:46:25.584250+0000","last_clean":"2023-10-16T13:46:25.584250+0000","last_became_active":"2023-10-16T13:46:25.576780+0000","last_became_peered":"2023-10-16T13:46:25.576780+0000","last_unstale":"2023-10-16T13:46:25.584250+0000","last_undegraded":"2023-10-16T13:46:25.584250+0000","last_fullsized":"2023-10-16T13:46:25.584250+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:41:30.230501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":11,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.567212+0000","last_change":"2023-10-16T13:46:24.340170+0000","last_active":"2023-10-16T13:46:25.567212+0000","last_peered":"2023-10-16T13:46:25.567212+0000","last_clean":"2023-10-16T13:46:25.567212+0000","last_became_active":"2023-10-16T13:46:24.339859+0000","last_became_peered":"2023-10-16T13:46:24.339859+0000","last_unstale":"2023-10-16T13:46:25.567212+0000","last_undegraded":"2023-10-16T13:46:25.567212+0000","last_fullsized":"2023-10-16T13:46:25.567212+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:25:04.566680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.7","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.580683+0000","last_change":"2023-10-16T13:46:25.577432+0000","last_active":"2023-10-16T13:46:25.580683+0000","last_peered":"2023-10-16T13:46:25.580683+0000","last_clean":"2023-10-16T13:46:25.580683+0000","last_became_active":"2023-10-16T13:46:25.577131+0000","last_became_peered":"2023-10-16T13:46:25.577131+0000","last_unstale":"2023-10-16T13:46:25.580683+0000","last_undegraded":"2023-10-16T13:46:25.580683+0000","last_fullsized":"2023-10-16T13:46:25.580683+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:10:39.170811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":11,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.564110+0000","last_change":"2023-10-16T13:46:24.346138+0000","last_active":"2023-10-16T13:46:25.564110+0000","last_peered":"2023-10-16T13:46:25.564110+0000","last_clean":"2023-10-16T13:46:25.564110+0000","last_became_active":"2023-10-16T13:46:24.345803+0000","last_became_peered":"2023-10-16T13:46:24.345803+0000","last_unstale":"2023-10-16T13:46:25.564110+0000","last_undegraded":"2023-10-16T13:46:25.564110+0000","last_fullsized":"2023-10-16T13:46:25.564110+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:08:14.564787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.581979+0000","last_change":"2023-10-16T13:46:25.579532+0000","last_active":"2023-10-16T13:46:25.581979+0000","last_peered":"2023-10-16T13:46:25.581979+0000","last_clean":"2023-10-16T13:46:25.581979+0000","last_became_active":"2023-10-16T13:46:25.579042+0000","last_became_peered":"2023-10-16T13:46:25.579042+0000","last_unstale":"2023-10-16T13:46:25.581979+0000","last_undegraded":"2023-10-16T13:46:25.581979+0000","last_fullsized":"2023-10-16T13:46:25.581979+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:01:57.893289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1a","version":"224'1","reported_seq":12,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.596111+0000","last_change":"2023-10-16T13:46:24.344199+0000","last_active":"2023-10-16T13:46:25.596111+0000","last_peered":"2023-10-16T13:46:25.596111+0000","last_clean":"2023-10-16T13:46:25.596111+0000","last_became_active":"2023-10-16T13:46:24.343947+0000","last_became_peered":"2023-10-16T13:46:24.343947+0000","last_unstale":"2023-10-16T13:46:25.596111+0000","last_undegraded":"2023-10-16T13:46:25.596111+0000","last_fullsized":"2023-10-16T13:46:25.596111+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:17:17.355917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"214'348","reported_seq":1201,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-16T13:46:24.338304+0000","last_change":"2023-10-16T12:10:34.746820+0000","last_active":"2023-10-16T13:46:24.338304+0000","last_peered":"2023-10-16T13:46:24.338304+0000","last_clean":"2023-10-16T13:46:24.338304+0000","last_became_active":"2023-10-16T12:10:34.745574+0000","last_became_peered":"2023-10-16T12:10:34.745574+0000","last_unstale":"2023-10-16T13:46:24.338304+0000","last_undegraded":"2023-10-16T13:46:24.338304+0000","last_fullsized":"2023-10-16T13:46:24.338304+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_clean_scrub_stamp":"2023-10-16T12:10:33.577111+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:14.461986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":28,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":27,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":348,"ondisk_log_size":348,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575986,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56740,"kb_used_data":3344,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315100,"statfs":{"total":96636764160,"available":96578662400,"internally_reserved":0,"allocated":3424256,"data_stored":2038361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54655435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:45:48 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.61499999999999999,"15min":0.63800000000000001},"min":{"1min":0.35499999999999998,"5min":0.34899999999999998,"15min":0.33000000000000002},"max":{"1min":0.81899999999999995,"5min":0.87,"15min":0.96199999999999997},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.61799999999999999,"15min":0.63300000000000001},"min":{"1min":0.39200000000000002,"5min":0.38900000000000001,"15min":0.379},"max":{"1min":0.83899999999999997,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.53000000000000003}]},{"osd":1,"last update":"Mon Oct 16 13:45:48 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.61399999999999999,"15min":0.64400000000000002},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.79000000000000004,"5min":0.84599999999999997,"15min":0.95899999999999996},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.621,"5min":0.63100000000000001,"15min":0.63800000000000001},"min":{"1min":0.48199999999999998,"5min":0.375,"15min":0.375},"max":{"1min":0.77000000000000002,"5min":0.79100000000000004,"15min":1.022},"last":0.67400000000000004}]},{"osd":2,"last update":"Mon Oct 16 13:45:48 2023","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.69399999999999995,"15min":0.69499999999999995},"min":{"1min":0.48999999999999999,"5min":0.436,"15min":0.436},"max":{"1min":0.83499999999999996,"5min":0.83899999999999997,"15min":0.93500000000000005},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.69599999999999995,"15min":0.70199999999999996},"min":{"1min":0.51000000000000001,"5min":0.47599999999999998,"15min":0.39400000000000002},"max":{"1min":0.871,"5min":0.92800000000000005,"15min":0.93799999999999994},"last":0.70599999999999996}]},{"osd":3,"last update":"Mon Oct 16 13:45:48 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.65000000000000002,"15min":0.66200000000000003},"min":{"1min":0.45200000000000001,"5min":0.438,"15min":0.438},"max":{"1min":0.81499999999999995,"5min":0.89200000000000002,"15min":0.91300000000000003},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.64600000000000002,"15min":0.66300000000000003},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.80400000000000005,"5min":1.24,"15min":1.24},"last":0.623}]},{"osd":4,"last update":"Mon Oct 16 13:45:48 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.51300000000000001,"15min":0.52400000000000002},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.72699999999999998,"5min":0.745,"15min":0.86099999999999999},"last":0.433},{"interface":"front","average":{"1min":0.623,"5min":0.61399999999999999,"15min":0.61899999999999999},"min":{"1min":0.40699999999999997,"5min":0.36799999999999999,"15min":0.33100000000000002},"max":{"1min":0.77700000000000002,"5min":0.90400000000000003,"15min":0.92900000000000005},"last":0.57299999999999995}]},{"osd":5,"last update":"Mon Oct 16 13:45:48 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.53100000000000003,"15min":0.54400000000000004},"min":{"1min":0.41699999999999998,"5min":0.33400000000000002,"15min":0.32500000000000001},"max":{"1min":0.71599999999999997,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.57299999999999995,"15min":0.58799999999999997},"min":{"1min":0.41199999999999998,"5min":0.38400000000000001,"15min":0.35099999999999998},"max":{"1min":0.80100000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.54900000000000004}]},{"osd":6,"last update":"Mon Oct 16 13:45:48 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.59999999999999998,"15min":0.61499999999999999},"min":{"1min":0.47399999999999998,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.79500000000000004,"5min":0.874,"15min":0.95899999999999996},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.61299999999999999,"15min":0.62},"min":{"1min":0.41099999999999998,"5min":0.378,"15min":0.35199999999999998},"max":{"1min":0.81299999999999994,"5min":0.95799999999999996,"15min":0.97499999999999998},"last":0.63900000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575984,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42240,"kb_used_data":2736,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329600,"statfs":{"total":96636764160,"available":96593510400,"internally_reserved":0,"allocated":2801664,"data_stored":1439088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:45:36 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.67200000000000004,"15min":0.67400000000000004},"min":{"1min":0.39900000000000002,"5min":0.375,"15min":0.28699999999999998},"max":{"1min":0.82299999999999995,"5min":0.89400000000000002,"15min":0.93600000000000005},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.42399999999999999,"5min":0.41999999999999998,"15min":0.36299999999999999},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.93799999999999994},"last":0.59099999999999997}]},{"osd":1,"last update":"Mon Oct 16 13:45:36 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.61699999999999999,"15min":0.629},"min":{"1min":0.379,"5min":0.36599999999999999,"15min":0.33100000000000002},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.90900000000000003},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.61099999999999999,"15min":0.621},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.38300000000000001},"max":{"1min":0.77000000000000002,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.65600000000000003}]},{"osd":2,"last update":"Mon Oct 16 13:45:36 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.628,"15min":0.63500000000000001},"min":{"1min":0.45600000000000002,"5min":0.39900000000000002,"15min":0.378},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.61799999999999999,"15min":0.628},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.77600000000000002,"5min":0.89000000000000001,"15min":0.91000000000000003},"last":0.46800000000000003}]},{"osd":3,"last update":"Mon Oct 16 13:45:36 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.65100000000000002,"15min":0.66500000000000004},"min":{"1min":0.45700000000000002,"5min":0.39900000000000002,"15min":0.38200000000000001},"max":{"1min":0.80000000000000004,"5min":0.92400000000000004,"15min":0.95599999999999996},"last":0.622},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.71299999999999997,"15min":0.71999999999999997},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.42299999999999999},"max":{"1min":0.94999999999999996,"5min":0.96099999999999997,"15min":0.96899999999999997},"last":0.71599999999999997}]},{"osd":4,"last update":"Mon Oct 16 13:45:36 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.51500000000000001,"15min":0.52500000000000002},"min":{"1min":0.36899999999999999,"5min":0.32600000000000001,"15min":0.315},"max":{"1min":0.67100000000000004,"5min":0.85599999999999998,"15min":0.95799999999999996},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.623,"15min":0.61899999999999999},"min":{"1min":0.42799999999999999,"5min":0.35499999999999998,"15min":0.32900000000000001},"max":{"1min":0.84199999999999997,"5min":0.98799999999999999,"15min":1.0309999999999999},"last":0.73099999999999998}]},{"osd":5,"last update":"Mon Oct 16 13:45:36 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.59299999999999997,"15min":0.58899999999999997},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.73999999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.66800000000000004,"15min":0.65900000000000003},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.30599999999999999},"max":{"1min":0.84499999999999997,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.75}]},{"osd":7,"last update":"Mon Oct 16 13:45:36 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.61199999999999999,"15min":0.60399999999999998},"min":{"1min":0.375,"5min":0.375,"15min":0.34300000000000003},"max":{"1min":0.81499999999999995,"5min":1.081,"15min":1.081},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.62,"5min":0.65500000000000003,"15min":0.64800000000000002},"min":{"1min":0.45300000000000001,"5min":0.38300000000000001,"15min":0.375},"max":{"1min":0.79700000000000004,"5min":1.099,"15min":1.099},"last":0.79600000000000004}]}]},{"osd":1,"up_from":13,"seq":55834575984,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47824,"kb_used_data":2748,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324016,"statfs":{"total":96636764160,"available":96587792384,"internally_reserved":0,"allocated":2813952,"data_stored":1446311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":46135753},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:45:54 2023","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.40100000000000002,"15min":0.40999999999999998},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.71999999999999997},"last":0.373},{"interface":"front","average":{"1min":0.497,"5min":0.496,"15min":0.499},"min":{"1min":0.34599999999999997,"5min":0.253,"15min":0.253},"max":{"1min":0.68000000000000005,"5min":0.70999999999999996,"15min":0.77200000000000002},"last":0.32000000000000001}]},{"osd":2,"last update":"Mon Oct 16 13:45:54 2023","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.45500000000000002,"15min":0.45800000000000002},"min":{"1min":0.28000000000000003,"5min":0.254,"15min":0.254},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.48199999999999998,"15min":0.49299999999999999},"min":{"1min":0.315,"5min":0.315,"15min":0.30199999999999999},"max":{"1min":0.64200000000000002,"5min":0.66900000000000004,"15min":0.71399999999999997},"last":0.46899999999999997}]},{"osd":3,"last update":"Mon Oct 16 13:45:54 2023","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.438,"15min":0.43099999999999999},"min":{"1min":0.33900000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.57399999999999995,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.47299999999999998,"15min":0.47899999999999998},"min":{"1min":0.26700000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.64800000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.35499999999999998}]},{"osd":4,"last update":"Mon Oct 16 13:45:54 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.66000000000000003,"15min":0.66700000000000004},"min":{"1min":0.52600000000000002,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.86099999999999999,"5min":0.95899999999999996,"15min":1.0069999999999999},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.66200000000000003,"15min":0.67200000000000004},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.747,"5min":0.91300000000000003,"15min":0.997},"last":0.42899999999999999}]},{"osd":5,"last update":"Mon Oct 16 13:45:54 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.69799999999999995,"15min":0.71399999999999997},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.92300000000000004,"5min":0.97499999999999998,"15min":1.0760000000000001},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.73199999999999998,"15min":0.73399999999999999},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.46200000000000002},"max":{"1min":0.90000000000000002,"5min":1.0469999999999999,"15min":1.0660000000000001},"last":0.40999999999999998}]},{"osd":6,"last update":"Mon Oct 16 13:45:54 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.69099999999999995,"15min":0.68500000000000005},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.80100000000000005,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.70799999999999996,"15min":0.71299999999999997},"min":{"1min":0.499,"5min":0.499,"15min":0.47799999999999998},"max":{"1min":0.88100000000000001,"5min":1.03,"15min":1.083},"last":0.60199999999999998}]},{"osd":7,"last update":"Mon Oct 16 13:45:54 2023","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.73899999999999999,"15min":0.745},"min":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.48899999999999999},"max":{"1min":0.91400000000000003,"5min":1.0629999999999999,"15min":1.0660000000000001},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.751,"15min":0.751},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.92700000000000005,"5min":1.042,"15min":1.0580000000000001},"last":0.54100000000000004}]}]},{"osd":0,"up_from":13,"seq":55834575985,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30340,"kb_used_data":3316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341500,"statfs":{"total":96636764160,"available":96605696000,"internally_reserved":0,"allocated":3395584,"data_stored":2029456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 13:45:42 2023","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.40899999999999997,"15min":0.42399999999999999},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.58699999999999997,"5min":0.65800000000000003,"15min":0.71399999999999997},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.42599999999999999,"15min":0.442},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.25900000000000001},"max":{"1min":0.58599999999999997,"5min":0.64600000000000002,"15min":0.82199999999999995},"last":0.29799999999999999}]},{"osd":2,"last update":"Mon Oct 16 13:45:42 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44500000000000001,"15min":0.45800000000000002},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.75700000000000001},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.46800000000000003,"15min":0.48699999999999999},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.83299999999999996},"last":0.34100000000000003}]},{"osd":3,"last update":"Mon Oct 16 13:45:42 2023","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.43099999999999999,"15min":0.45400000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.61799999999999999,"5min":0.63400000000000001,"15min":0.78600000000000003},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.51600000000000001,"15min":0.52500000000000002},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.70399999999999996,"5min":0.82999999999999996,"15min":0.86399999999999999},"last":0.317}]},{"osd":4,"last update":"Mon Oct 16 13:45:42 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.66500000000000004,"15min":0.68600000000000005},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.96999999999999997,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.66300000000000003,"15min":0.69299999999999995},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":1.0049999999999999},"last":0.499}]},{"osd":5,"last update":"Mon Oct 16 13:45:42 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.66700000000000004,"15min":0.69499999999999995},"min":{"1min":0.44500000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.95399999999999996,"5min":0.95999999999999996,"15min":1.0249999999999999},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.70599999999999996,"15min":0.72999999999999998},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.081,"5min":1.081,"15min":1.081},"last":0.56399999999999995}]},{"osd":6,"last update":"Mon Oct 16 13:45:42 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.71599999999999997,"15min":0.73499999999999999},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.034,"5min":1.103,"15min":1.103},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.70199999999999996,"15min":0.72499999999999998},"min":{"1min":0.42999999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.91000000000000003,"5min":1.0409999999999999,"15min":1.05},"last":0.51000000000000001}]},{"osd":7,"last update":"Mon Oct 16 13:45:42 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.71899999999999997,"15min":0.746},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.002,"5min":1.002,"15min":1.056},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.72099999999999997,"15min":0.73699999999999999},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.99199999999999999,"5min":1.087,"15min":1.087},"last":0.55200000000000005}]}]},{"osd":2,"up_from":13,"seq":55834575985,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51968,"kb_used_data":2736,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319872,"statfs":{"total":96636764160,"available":96583548928,"internally_reserved":0,"allocated":2801664,"data_stored":1439088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:45:36 2023","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.42499999999999999,"15min":0.41299999999999998},"min":{"1min":0.26300000000000001,"5min":0.251,"15min":0.24099999999999999},"max":{"1min":0.52200000000000002,"5min":0.71699999999999997,"15min":0.77500000000000002},"last":0.255},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.48599999999999999,"15min":0.48299999999999998},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.63500000000000001,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.35199999999999998}]},{"osd":1,"last update":"Mon Oct 16 13:45:36 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.51500000000000001,"15min":0.50600000000000001},"min":{"1min":0.313,"5min":0.311,"15min":0.311},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.81399999999999995},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.5,"15min":0.47999999999999998},"min":{"1min":0.31900000000000001,"5min":0.29899999999999999,"15min":0.27700000000000002},"max":{"1min":0.57799999999999996,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.29099999999999998}]},{"osd":3,"last update":"Mon Oct 16 13:45:36 2023","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.44900000000000001,"15min":0.44500000000000001},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.60299999999999998,"5min":0.70099999999999996,"15min":0.86699999999999999},"last":0.314},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.56100000000000005,"15min":0.55100000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.309},"max":{"1min":0.74099999999999999,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.27400000000000002}]},{"osd":4,"last update":"Mon Oct 16 13:45:36 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.74099999999999999,"15min":0.72399999999999998},"min":{"1min":0.51600000000000001,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.89400000000000002,"5min":1.3600000000000001,"15min":1.3600000000000001},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.745,"15min":0.72799999999999998},"min":{"1min":0.502,"5min":0.45700000000000002,"15min":0.44900000000000001},"max":{"1min":0.91600000000000004,"5min":1.403,"15min":1.403},"last":0.624}]},{"osd":5,"last update":"Mon Oct 16 13:45:36 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.70599999999999996,"15min":0.69699999999999995},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.441},"max":{"1min":0.79700000000000004,"5min":1.486,"15min":1.486},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.73099999999999998,"15min":0.70799999999999996},"min":{"1min":0.44800000000000001,"5min":0.437,"15min":0.437},"max":{"1min":0.83099999999999996,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":0.51600000000000001}]},{"osd":6,"last update":"Mon Oct 16 13:45:36 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.69399999999999995,"15min":0.67400000000000004},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.98899999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.497},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.74099999999999999,"15min":0.73099999999999998},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.83699999999999997,"5min":1.4379999999999999,"15min":1.4379999999999999},"last":0.56200000000000006}]},{"osd":7,"last update":"Mon Oct 16 13:45:36 2023","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.75700000000000001,"15min":0.72899999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.0509999999999999,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.75,"15min":0.74199999999999999},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.98099999999999998,"5min":1.377,"15min":1.377},"last":0.57999999999999996}]}]},{"osd":3,"up_from":13,"seq":55834575984,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51988,"kb_used_data":2760,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319852,"statfs":{"total":96636764160,"available":96583528448,"internally_reserved":0,"allocated":2826240,"data_stored":1446437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50395592},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:45:39 2023","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.40699999999999997,"15min":0.41399999999999998},"min":{"1min":0.27600000000000002,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.52900000000000003,"5min":0.64200000000000002,"15min":0.72899999999999998},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.42499999999999999,"15min":0.43099999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.54700000000000004,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.35399999999999998}]},{"osd":1,"last update":"Mon Oct 16 13:45:39 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.436,"15min":0.438},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.76300000000000001},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.46999999999999997,"15min":0.48299999999999998},"min":{"1min":0.34300000000000003,"5min":0.253,"15min":0.253},"max":{"1min":0.68700000000000006,"5min":0.71499999999999997,"15min":0.77800000000000002},"last":0.27700000000000002}]},{"osd":2,"last update":"Mon Oct 16 13:45:39 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.49199999999999999,"15min":0.498},"min":{"1min":0.318,"5min":0.316,"15min":0.316},"max":{"1min":0.65600000000000003,"5min":1.167,"15min":1.167},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.51900000000000002,"15min":0.52900000000000003},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.73299999999999998,"5min":0.76500000000000001,"15min":0.78700000000000003},"last":0.38}]},{"osd":4,"last update":"Mon Oct 16 13:45:39 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.66000000000000003,"15min":0.67100000000000004},"min":{"1min":0.434,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.98499999999999999,"5min":1.137,"15min":1.137},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.69499999999999995,"15min":0.69699999999999995},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.0589999999999999,"5min":1.2909999999999999,"15min":1.2909999999999999},"last":0.47899999999999998}]},{"osd":5,"last update":"Mon Oct 16 13:45:39 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.69899999999999995,"15min":0.71299999999999997},"min":{"1min":0.46100000000000002,"5min":0.42899999999999999,"15min":0.374},"max":{"1min":0.89400000000000002,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.71599999999999997,"15min":0.70499999999999996},"min":{"1min":0.47099999999999997,"5min":0.432,"15min":0.432},"max":{"1min":1.0549999999999999,"5min":1.173,"15min":1.173},"last":0.60099999999999998}]},{"osd":6,"last update":"Mon Oct 16 13:45:39 2023","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.747,"15min":0.72499999999999998},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.41799999999999998},"max":{"1min":0.94499999999999995,"5min":5.0999999999999996,"15min":5.0999999999999996},"last":0.443},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.72899999999999998,"15min":0.73699999999999999},"min":{"1min":0.48599999999999999,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.0309999999999999,"5min":1.2290000000000001,"15min":1.2290000000000001},"last":0.49299999999999999}]},{"osd":7,"last update":"Mon Oct 16 13:45:39 2023","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.73399999999999999,"15min":0.73899999999999999},"min":{"1min":0.56599999999999995,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.92300000000000004,"5min":1.2569999999999999,"15min":1.2569999999999999},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.75,"15min":0.748},"min":{"1min":0.48999999999999999,"5min":0.47799999999999998,"15min":0.43099999999999999},"max":{"1min":1.014,"5min":1.262,"15min":1.262},"last":0.51000000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575983,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51968,"kb_used_data":2736,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319872,"statfs":{"total":96636764160,"available":96583548928,"internally_reserved":0,"allocated":2801664,"data_stored":1439088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:45:36 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.61599999999999999,"15min":0.625},"min":{"1min":0.45000000000000001,"5min":0.36299999999999999,"15min":0.33800000000000002},"max":{"1min":0.80600000000000005,"5min":0.83599999999999997,"15min":0.92700000000000005},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.628,"15min":0.63300000000000001},"min":{"1min":0.436,"5min":0.41499999999999998,"15min":0.35499999999999998},"max":{"1min":0.84099999999999997,"5min":0.878,"15min":0.89500000000000002},"last":0.84099999999999997}]},{"osd":1,"last update":"Mon Oct 16 13:45:36 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.64100000000000001,"15min":0.65900000000000003},"min":{"1min":0.48899999999999999,"5min":0.38400000000000001,"15min":0.375},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.877},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.63800000000000001,"15min":0.63700000000000001},"min":{"1min":0.41199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.78000000000000003,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.81100000000000005}]},{"osd":2,"last update":"Mon Oct 16 13:45:36 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.65800000000000003,"15min":0.65300000000000002},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.39600000000000002},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.90600000000000003},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.66200000000000003,"15min":0.66100000000000003},"min":{"1min":0.48599999999999999,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.85999999999999999,"5min":0.89700000000000002,"15min":0.92600000000000005},"last":0.746}]},{"osd":3,"last update":"Mon Oct 16 13:45:36 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.64100000000000001,"15min":0.64900000000000002},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.79100000000000004,"5min":0.86599999999999999,"15min":0.99199999999999999},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.68500000000000005,"15min":0.68300000000000005},"min":{"1min":0.54900000000000004,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.84699999999999998,"5min":0.90700000000000003,"15min":0.93899999999999995},"last":0.76400000000000001}]},{"osd":5,"last update":"Mon Oct 16 13:45:36 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.54400000000000004,"15min":0.52800000000000002},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.30599999999999999},"max":{"1min":0.76500000000000001,"5min":0.81299999999999994,"15min":0.91700000000000004},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.623,"5min":0.57699999999999996,"15min":0.56899999999999995},"min":{"1min":0.39000000000000001,"5min":0.33800000000000002,"15min":0.33600000000000002},"max":{"1min":0.80500000000000005,"5min":0.82299999999999995,"15min":0.94899999999999995},"last":0.625}]},{"osd":6,"last update":"Mon Oct 16 13:45:36 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.60899999999999999,"15min":0.59399999999999997},"min":{"1min":0.38900000000000001,"5min":0.34499999999999997,"15min":0.33200000000000002},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.92700000000000005},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.69599999999999995,"15min":0.68899999999999995},"min":{"1min":0.52800000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.95299999999999996},"last":0.86299999999999999}]},{"osd":7,"last update":"Mon Oct 16 13:45:36 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.64500000000000002,"15min":0.63100000000000001},"min":{"1min":0.44500000000000001,"5min":0.39900000000000002,"15min":0.35799999999999998},"max":{"1min":0.86899999999999999,"5min":0.97499999999999998,"15min":0.98599999999999999},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.69399999999999995,"15min":0.68100000000000005},"min":{"1min":0.53800000000000003,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.91200000000000003,"5min":0.91700000000000004,"15min":1.026},"last":0.82499999999999996}]}]},{"osd":5,"up_from":13,"seq":55834575984,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51968,"kb_used_data":2736,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319872,"statfs":{"total":96636764160,"available":96583548928,"internally_reserved":0,"allocated":2801664,"data_stored":1439088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:46:24 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.61399999999999999,"15min":0.61499999999999999},"min":{"1min":0.314,"5min":0.314,"15min":0.26800000000000002},"max":{"1min":0.878,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63,"15min":0.64200000000000002},"min":{"1min":0.38700000000000001,"5min":0.35899999999999999,"15min":0.29699999999999999},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.91200000000000003}]},{"osd":1,"last update":"Mon Oct 16 13:46:24 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.66000000000000003,"15min":0.66200000000000003},"min":{"1min":0.44600000000000001,"5min":0.34300000000000003,"15min":0.29499999999999998},"max":{"1min":1,"5min":1,"15min":1},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.63800000000000001,"15min":0.64000000000000001},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.57099999999999995}]},{"osd":2,"last update":"Mon Oct 16 13:46:24 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.67000000000000004,"15min":0.65700000000000003},"min":{"1min":0.41699999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.91000000000000003,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.44},{"interface":"front","average":{"1min":0.629,"5min":0.65700000000000003,"15min":0.64900000000000002},"min":{"1min":0.42599999999999999,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.83899999999999997,"5min":0.871,"15min":0.93500000000000005},"last":0.73999999999999999}]},{"osd":3,"last update":"Mon Oct 16 13:46:24 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.67400000000000004,"15min":0.66400000000000003},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.623},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.69899999999999995,"15min":0.69499999999999995},"min":{"1min":0.41399999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.4199999999999999,"5min":1.4199999999999999,"15min":1.4199999999999999},"last":0.67100000000000004}]},{"osd":4,"last update":"Mon Oct 16 13:46:24 2023","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.54400000000000004,"15min":0.53300000000000003},"min":{"1min":0.34100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.82499999999999996,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.59299999999999997,"15min":0.58899999999999997},"min":{"1min":0.34300000000000003,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.85399999999999998,"5min":0.89000000000000001,"15min":0.93400000000000005},"last":0.70799999999999996}]},{"osd":6,"last update":"Mon Oct 16 13:46:24 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.58899999999999997,"15min":0.57999999999999996},"min":{"1min":0.38700000000000001,"5min":0.315,"15min":0.315},"max":{"1min":0.877,"5min":0.88900000000000001,"15min":0.93200000000000005},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.61899999999999999},"min":{"1min":0.372,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89600000000000002},"last":0.64600000000000002}]},{"osd":7,"last update":"Mon Oct 16 13:46:24 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.64800000000000002,"15min":0.63300000000000001},"min":{"1min":0.378,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.69699999999999995,"15min":0.68400000000000005},"min":{"1min":0.46200000000000002,"5min":0.375,"15min":0.375},"max":{"1min":0.94699999999999995,"5min":0.98299999999999998,"15min":1.0229999999999999},"last":0.76200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T13:46:29.745 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T13:46:29.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.901+0000 7f55582d9700 1 Processor -- start 2023-10-16T13:46:29.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.901+0000 7f55582d9700 1 -- start start 2023-10-16T13:46:29.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.902+0000 7f55582d9700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f55500cf880 0x7f55500b52f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:29.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.902+0000 7f55582d9700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f55500b5830 0x7f555015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:29.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.902+0000 7f55582d9700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f555015bc80 con 0x7f55500cf4a0 2023-10-16T13:46:29.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.902+0000 7f55582d9700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f555015b070 con 0x7f55500cf880 2023-10-16T13:46:29.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.902+0000 7f55582d9700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f555015b350 con 0x7f55500b5830 2023-10-16T13:46:29.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.902+0000 7f5555874700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f55500cf880 0x7f55500b52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:29.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.902+0000 7f5555874700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f55500cf880 0x7f55500b52f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53222/0 (socket says 172.21.15.16:53222) 2023-10-16T13:46:29.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.902+0000 7f5555874700 1 -- 172.21.15.16:0/4072203275 learned_addr learned my addr 172.21.15.16:0/4072203275 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:29.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.902+0000 7f5556876700 1 --2- 172.21.15.16:0/4072203275 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f55500b5830 0x7f555015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:29.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.903+0000 7f5555874700 1 -- 172.21.15.16:0/4072203275 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f55500b5830 msgr2=0x7f555015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:29.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.903+0000 7f5555874700 1 --2- 172.21.15.16:0/4072203275 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f55500b5830 0x7f555015a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:29.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.903+0000 7f5555874700 1 -- 172.21.15.16:0/4072203275 >> v1:172.21.15.49:6789/0 conn(0x7f55500cf4a0 legacy=0x7f55500726e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:29.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.903+0000 7f5555874700 1 -- 172.21.15.16:0/4072203275 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f555015b630 con 0x7f55500cf880 2023-10-16T13:46:29.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.903+0000 7f5556876700 1 --2- 172.21.15.16:0/4072203275 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f55500b5830 0x7f555015a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:46:29.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.903+0000 7f5555874700 1 --2- 172.21.15.16:0/4072203275 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55500cf880 0x7f55500b52f0 secure :-1 s=READY pgs=2468 cs=0 l=1 rev1=1 crypto rx=0x7f554c0098f0 tx=0x7f554c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d0dcc6175956553 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:29.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.903+0000 7f5555073700 1 -- 172.21.15.16:0/4072203275 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f554c00b8a0 con 0x7f55500cf880 2023-10-16T13:46:29.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.903+0000 7f5555073700 1 -- 172.21.15.16:0/4072203275 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f554c00ba00 con 0x7f55500cf880 2023-10-16T13:46:29.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.903+0000 7f5555073700 1 -- 172.21.15.16:0/4072203275 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f554c00bd90 con 0x7f55500cf880 2023-10-16T13:46:29.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.903+0000 7f55582d9700 1 -- 172.21.15.16:0/4072203275 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55500cf880 msgr2=0x7f55500b52f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:29.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.903+0000 7f55582d9700 1 --2- 172.21.15.16:0/4072203275 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55500cf880 0x7f55500b52f0 secure :-1 s=READY pgs=2468 cs=0 l=1 rev1=1 crypto rx=0x7f554c0098f0 tx=0x7f554c009c00 comp rx=0 tx=0).stop 2023-10-16T13:46:29.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.904+0000 7f55582d9700 1 -- 172.21.15.16:0/4072203275 shutdown_connections 2023-10-16T13:46:29.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.904+0000 7f55582d9700 1 --2- 172.21.15.16:0/4072203275 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f55500b5830 0x7f555015a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:29.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.904+0000 7f55582d9700 1 --2- 172.21.15.16:0/4072203275 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55500cf880 0x7f55500b52f0 unknown :-1 s=CLOSED pgs=2468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:29.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.904+0000 7f55582d9700 1 -- 172.21.15.16:0/4072203275 >> 172.21.15.16:0/4072203275 conn(0x7f55500b1fd0 msgr2=0x7f55500b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:29.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.904+0000 7f55582d9700 1 -- 172.21.15.16:0/4072203275 shutdown_connections 2023-10-16T13:46:29.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.904+0000 7f55582d9700 1 -- 172.21.15.16:0/4072203275 wait complete. 2023-10-16T13:46:29.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.905+0000 7f55582d9700 1 Processor -- start 2023-10-16T13:46:29.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.905+0000 7f55582d9700 1 -- start start 2023-10-16T13:46:29.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.906+0000 7f55582d9700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f55500b5830 0x7f55501e7060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:29.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.906+0000 7f55582d9700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f55500cf4a0 0x7f5550157180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:29.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.906+0000 7f55582d9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55500cf880 0x7f55501576c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:29.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.906+0000 7f55582d9700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5550158710 con 0x7f55500cf880 2023-10-16T13:46:29.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.906+0000 7f55582d9700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5550157c00 con 0x7f55500cf4a0 2023-10-16T13:46:29.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.906+0000 7f55582d9700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5550157e70 con 0x7f55500b5830 2023-10-16T13:46:29.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.906+0000 7f5556876700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55500cf880 0x7f55501576c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:29.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.906+0000 7f5556876700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55500cf880 0x7f55501576c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53230/0 (socket says 172.21.15.16:53230) 2023-10-16T13:46:29.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.906+0000 7f5556876700 1 -- 172.21.15.16:0/2970460651 learned_addr learned my addr 172.21.15.16:0/2970460651 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:29.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.906+0000 7f5555874700 1 --2- 172.21.15.16:0/2970460651 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f55500cf4a0 0x7f5550157180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:29.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.906+0000 7f5556075700 1 --2- 172.21.15.16:0/2970460651 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f55500b5830 0x7f55501e7060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:29.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.906+0000 7f5556876700 1 -- 172.21.15.16:0/2970460651 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f55500b5830 msgr2=0x7f55501e7060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:29.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.906+0000 7f5556876700 1 --2- 172.21.15.16:0/2970460651 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f55500b5830 0x7f55501e7060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:29.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.906+0000 7f5556876700 1 -- 172.21.15.16:0/2970460651 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f55500cf4a0 msgr2=0x7f5550157180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:29.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.906+0000 7f5556876700 1 --2- 172.21.15.16:0/2970460651 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f55500cf4a0 0x7f5550157180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:29.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.906+0000 7f5556876700 1 -- 172.21.15.16:0/2970460651 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f554c009550 con 0x7f55500cf880 2023-10-16T13:46:29.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.906+0000 7f5556876700 1 --2- 172.21.15.16:0/2970460651 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55500cf880 0x7f55501576c0 secure :-1 s=READY pgs=2469 cs=0 l=1 rev1=1 crypto rx=0x7f554800cf20 tx=0x7f554800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:29.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.906+0000 7f5546ffd700 1 -- 172.21.15.16:0/2970460651 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f554800ae80 con 0x7f55500cf880 2023-10-16T13:46:29.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.906+0000 7f5546ffd700 1 -- 172.21.15.16:0/2970460651 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5548015440 con 0x7f55500cf880 2023-10-16T13:46:29.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.906+0000 7f55582d9700 1 -- 172.21.15.16:0/2970460651 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5550158150 con 0x7f55500cf880 2023-10-16T13:46:29.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.906+0000 7f5546ffd700 1 -- 172.21.15.16:0/2970460651 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55480157d0 con 0x7f55500cf880 2023-10-16T13:46:29.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.906+0000 7f55582d9700 1 -- 172.21.15.16:0/2970460651 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55500cbd00 con 0x7f55500cf880 2023-10-16T13:46:29.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.907+0000 7f55582d9700 1 -- 172.21.15.16:0/2970460651 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55500cbb60 con 0x7f55500cf880 2023-10-16T13:46:29.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.908+0000 7f5546ffd700 1 -- 172.21.15.16:0/2970460651 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 138) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f554800d430 con 0x7f55500cf880 2023-10-16T13:46:29.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.908+0000 7f5546ffd700 1 --2- 172.21.15.16:0/2970460651 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f553406b020 0x7f553406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:29.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.908+0000 7f5546ffd700 1 -- 172.21.15.16:0/2970460651 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(225..225 src has 1..225) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f554808a080 con 0x7f55500cf880 2023-10-16T13:46:29.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.908+0000 7f5556075700 1 --2- 172.21.15.16:0/2970460651 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f553406b020 0x7f553406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:29.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.909+0000 7f5556075700 1 --2- 172.21.15.16:0/2970460651 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f553406b020 0x7f553406d4e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5540000f60 tx=0x7f5540009040 comp rx=0 tx=0).ready entity=mgr.11190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:29.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:29.913+0000 7f5546ffd700 1 -- 172.21.15.16:0/2970460651 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f55480bcfe0 con 0x7f55500cf880 2023-10-16T13:46:30.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.113+0000 7f55582d9700 1 -- 172.21.15.16:0/2970460651 --> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f555015b910 con 0x7f553406b020 2023-10-16T13:46:30.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.121+0000 7f5546ffd700 1 -- 172.21.15.16:0/2970460651 <== mgr.11190 v2:172.21.15.16:6832/3171422553 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289728 (secure 0 0 0) 0x7f5540001490 con 0x7f553406b020 2023-10-16T13:46:30.127 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:46:30.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.124+0000 7f55582d9700 1 -- 172.21.15.16:0/2970460651 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f553406b020 msgr2=0x7f553406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:30.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.124+0000 7f55582d9700 1 --2- 172.21.15.16:0/2970460651 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f553406b020 0x7f553406d4e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5540000f60 tx=0x7f5540009040 comp rx=0 tx=0).stop 2023-10-16T13:46:30.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.124+0000 7f55582d9700 1 -- 172.21.15.16:0/2970460651 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55500cf880 msgr2=0x7f55501576c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:30.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.124+0000 7f55582d9700 1 --2- 172.21.15.16:0/2970460651 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55500cf880 0x7f55501576c0 secure :-1 s=READY pgs=2469 cs=0 l=1 rev1=1 crypto rx=0x7f554800cf20 tx=0x7f554800a340 comp rx=0 tx=0).stop 2023-10-16T13:46:30.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.125+0000 7f55582d9700 1 -- 172.21.15.16:0/2970460651 shutdown_connections 2023-10-16T13:46:30.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.125+0000 7f55582d9700 1 --2- 172.21.15.16:0/2970460651 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f553406b020 0x7f553406d4e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:30.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.125+0000 7f55582d9700 1 --2- 172.21.15.16:0/2970460651 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55500cf880 0x7f55501576c0 unknown :-1 s=CLOSED pgs=2469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:30.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.125+0000 7f55582d9700 1 --2- 172.21.15.16:0/2970460651 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f55500b5830 0x7f55501e7060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:30.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.125+0000 7f55582d9700 1 --2- 172.21.15.16:0/2970460651 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f55500cf4a0 0x7f5550157180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:30.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.125+0000 7f55582d9700 1 -- 172.21.15.16:0/2970460651 >> 172.21.15.16:0/2970460651 conn(0x7f55500b1fd0 msgr2=0x7f55500b1790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:30.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.125+0000 7f55582d9700 1 -- 172.21.15.16:0/2970460651 shutdown_connections 2023-10-16T13:46:30.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.125+0000 7f55582d9700 1 -- 172.21.15.16:0/2970460651 wait complete. 2023-10-16T13:46:30.132 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-10-16T13:46:30.139 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-10-16T13:46:28.177392+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":623,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":386,"ondisk_log_size":386,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":103,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":385036,"kb_used_data":23112,"kb_used_omap":12,"kb_used_meta":361779,"kb_avail":754589684,"statfs":{"total":773094113280,"available":772699836416,"internally_reserved":0,"allocated":23666688,"data_stored":12716917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12740,"internal_metadata":370462268},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.858249"},"pg_stats":[{"pgid":"28.3f","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.590280+0000","last_change":"2023-10-16T13:46:25.582893+0000","last_active":"2023-10-16T13:46:25.590280+0000","last_peered":"2023-10-16T13:46:25.590280+0000","last_clean":"2023-10-16T13:46:25.590280+0000","last_became_active":"2023-10-16T13:46:25.581980+0000","last_became_peered":"2023-10-16T13:46:25.581980+0000","last_unstale":"2023-10-16T13:46:25.590280+0000","last_undegraded":"2023-10-16T13:46:25.590280+0000","last_fullsized":"2023-10-16T13:46:25.590280+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:21:54.809497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3e","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.579398+0000","last_change":"2023-10-16T13:46:25.570175+0000","last_active":"2023-10-16T13:46:25.579398+0000","last_peered":"2023-10-16T13:46:25.579398+0000","last_clean":"2023-10-16T13:46:25.579398+0000","last_became_active":"2023-10-16T13:46:25.569413+0000","last_became_peered":"2023-10-16T13:46:25.569413+0000","last_unstale":"2023-10-16T13:46:25.579398+0000","last_undegraded":"2023-10-16T13:46:25.579398+0000","last_fullsized":"2023-10-16T13:46:25.579398+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:00:39.897377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.3d","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.576214+0000","last_change":"2023-10-16T13:46:25.575243+0000","last_active":"2023-10-16T13:46:25.576214+0000","last_peered":"2023-10-16T13:46:25.576214+0000","last_clean":"2023-10-16T13:46:25.576214+0000","last_became_active":"2023-10-16T13:46:25.574483+0000","last_became_peered":"2023-10-16T13:46:25.574483+0000","last_unstale":"2023-10-16T13:46:25.576214+0000","last_undegraded":"2023-10-16T13:46:25.576214+0000","last_fullsized":"2023-10-16T13:46:25.576214+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:17:18.357714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.3c","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.583542+0000","last_change":"2023-10-16T13:46:25.578245+0000","last_active":"2023-10-16T13:46:25.583542+0000","last_peered":"2023-10-16T13:46:25.583542+0000","last_clean":"2023-10-16T13:46:25.583542+0000","last_became_active":"2023-10-16T13:46:25.577671+0000","last_became_peered":"2023-10-16T13:46:25.577671+0000","last_unstale":"2023-10-16T13:46:25.583542+0000","last_undegraded":"2023-10-16T13:46:25.583542+0000","last_fullsized":"2023-10-16T13:46:25.583542+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:31:15.200012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.3b","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.572946+0000","last_change":"2023-10-16T13:46:25.570352+0000","last_active":"2023-10-16T13:46:25.572946+0000","last_peered":"2023-10-16T13:46:25.572946+0000","last_clean":"2023-10-16T13:46:25.572946+0000","last_became_active":"2023-10-16T13:46:25.570035+0000","last_became_peered":"2023-10-16T13:46:25.570035+0000","last_unstale":"2023-10-16T13:46:25.572946+0000","last_undegraded":"2023-10-16T13:46:25.572946+0000","last_fullsized":"2023-10-16T13:46:25.572946+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:27:41.282540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.3a","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.571695+0000","last_change":"2023-10-16T13:46:25.568946+0000","last_active":"2023-10-16T13:46:25.571695+0000","last_peered":"2023-10-16T13:46:25.571695+0000","last_clean":"2023-10-16T13:46:25.571695+0000","last_became_active":"2023-10-16T13:46:25.568652+0000","last_became_peered":"2023-10-16T13:46:25.568652+0000","last_unstale":"2023-10-16T13:46:25.571695+0000","last_undegraded":"2023-10-16T13:46:25.571695+0000","last_fullsized":"2023-10-16T13:46:25.571695+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:10:39.170811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.39","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.574900+0000","last_change":"2023-10-16T13:46:25.573056+0000","last_active":"2023-10-16T13:46:25.574900+0000","last_peered":"2023-10-16T13:46:25.574900+0000","last_clean":"2023-10-16T13:46:25.574900+0000","last_became_active":"2023-10-16T13:46:25.572702+0000","last_became_peered":"2023-10-16T13:46:25.572702+0000","last_unstale":"2023-10-16T13:46:25.574900+0000","last_undegraded":"2023-10-16T13:46:25.574900+0000","last_fullsized":"2023-10-16T13:46:25.574900+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:45:49.954882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.38","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.579179+0000","last_change":"2023-10-16T13:46:25.576391+0000","last_active":"2023-10-16T13:46:25.579179+0000","last_peered":"2023-10-16T13:46:25.579179+0000","last_clean":"2023-10-16T13:46:25.579179+0000","last_became_active":"2023-10-16T13:46:25.575906+0000","last_became_peered":"2023-10-16T13:46:25.575906+0000","last_unstale":"2023-10-16T13:46:25.579179+0000","last_undegraded":"2023-10-16T13:46:25.579179+0000","last_fullsized":"2023-10-16T13:46:25.579179+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:57:24.329549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.37","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.583962+0000","last_change":"2023-10-16T13:46:25.583298+0000","last_active":"2023-10-16T13:46:25.583962+0000","last_peered":"2023-10-16T13:46:25.583962+0000","last_clean":"2023-10-16T13:46:25.583962+0000","last_became_active":"2023-10-16T13:46:25.582794+0000","last_became_peered":"2023-10-16T13:46:25.582794+0000","last_unstale":"2023-10-16T13:46:25.583962+0000","last_undegraded":"2023-10-16T13:46:25.583962+0000","last_fullsized":"2023-10-16T13:46:25.583962+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:26:58.168760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.36","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.593239+0000","last_change":"2023-10-16T13:46:25.585253+0000","last_active":"2023-10-16T13:46:25.593239+0000","last_peered":"2023-10-16T13:46:25.593239+0000","last_clean":"2023-10-16T13:46:25.593239+0000","last_became_active":"2023-10-16T13:46:25.584881+0000","last_became_peered":"2023-10-16T13:46:25.584881+0000","last_unstale":"2023-10-16T13:46:25.593239+0000","last_undegraded":"2023-10-16T13:46:25.593239+0000","last_fullsized":"2023-10-16T13:46:25.593239+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:43:07.323286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.35","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.580786+0000","last_change":"2023-10-16T13:46:25.578710+0000","last_active":"2023-10-16T13:46:25.580786+0000","last_peered":"2023-10-16T13:46:25.580786+0000","last_clean":"2023-10-16T13:46:25.580786+0000","last_became_active":"2023-10-16T13:46:25.578250+0000","last_became_peered":"2023-10-16T13:46:25.578250+0000","last_unstale":"2023-10-16T13:46:25.580786+0000","last_undegraded":"2023-10-16T13:46:25.580786+0000","last_fullsized":"2023-10-16T13:46:25.580786+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:47:31.681981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.34","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.576335+0000","last_change":"2023-10-16T13:46:25.575302+0000","last_active":"2023-10-16T13:46:25.576335+0000","last_peered":"2023-10-16T13:46:25.576335+0000","last_clean":"2023-10-16T13:46:25.576335+0000","last_became_active":"2023-10-16T13:46:25.575009+0000","last_became_peered":"2023-10-16T13:46:25.575009+0000","last_unstale":"2023-10-16T13:46:25.576335+0000","last_undegraded":"2023-10-16T13:46:25.576335+0000","last_fullsized":"2023-10-16T13:46:25.576335+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:57:04.481600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.33","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.583336+0000","last_change":"2023-10-16T13:46:25.576474+0000","last_active":"2023-10-16T13:46:25.583336+0000","last_peered":"2023-10-16T13:46:25.583336+0000","last_clean":"2023-10-16T13:46:25.583336+0000","last_became_active":"2023-10-16T13:46:25.576187+0000","last_became_peered":"2023-10-16T13:46:25.576187+0000","last_unstale":"2023-10-16T13:46:25.583336+0000","last_undegraded":"2023-10-16T13:46:25.583336+0000","last_fullsized":"2023-10-16T13:46:25.583336+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:26:58.168760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.32","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.584726+0000","last_change":"2023-10-16T13:46:25.581421+0000","last_active":"2023-10-16T13:46:25.584726+0000","last_peered":"2023-10-16T13:46:25.584726+0000","last_clean":"2023-10-16T13:46:25.584726+0000","last_became_active":"2023-10-16T13:46:25.580936+0000","last_became_peered":"2023-10-16T13:46:25.580936+0000","last_unstale":"2023-10-16T13:46:25.584726+0000","last_undegraded":"2023-10-16T13:46:25.584726+0000","last_fullsized":"2023-10-16T13:46:25.584726+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:22:57.732319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.31","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.580509+0000","last_change":"2023-10-16T13:46:25.577384+0000","last_active":"2023-10-16T13:46:25.580509+0000","last_peered":"2023-10-16T13:46:25.580509+0000","last_clean":"2023-10-16T13:46:25.580509+0000","last_became_active":"2023-10-16T13:46:25.577070+0000","last_became_peered":"2023-10-16T13:46:25.577070+0000","last_unstale":"2023-10-16T13:46:25.580509+0000","last_undegraded":"2023-10-16T13:46:25.580509+0000","last_fullsized":"2023-10-16T13:46:25.580509+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:44:11.618820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.30","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.573597+0000","last_change":"2023-10-16T13:46:25.570144+0000","last_active":"2023-10-16T13:46:25.573597+0000","last_peered":"2023-10-16T13:46:25.573597+0000","last_clean":"2023-10-16T13:46:25.573597+0000","last_became_active":"2023-10-16T13:46:25.569866+0000","last_became_peered":"2023-10-16T13:46:25.569866+0000","last_unstale":"2023-10-16T13:46:25.573597+0000","last_undegraded":"2023-10-16T13:46:25.573597+0000","last_fullsized":"2023-10-16T13:46:25.573597+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:04:47.313972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.2f","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.580110+0000","last_change":"2023-10-16T13:46:25.577983+0000","last_active":"2023-10-16T13:46:25.580110+0000","last_peered":"2023-10-16T13:46:25.580110+0000","last_clean":"2023-10-16T13:46:25.580110+0000","last_became_active":"2023-10-16T13:46:25.577680+0000","last_became_peered":"2023-10-16T13:46:25.577680+0000","last_unstale":"2023-10-16T13:46:25.580110+0000","last_undegraded":"2023-10-16T13:46:25.580110+0000","last_fullsized":"2023-10-16T13:46:25.580110+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:31:15.200012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.2e","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.573094+0000","last_change":"2023-10-16T13:46:25.567084+0000","last_active":"2023-10-16T13:46:25.573094+0000","last_peered":"2023-10-16T13:46:25.573094+0000","last_clean":"2023-10-16T13:46:25.573094+0000","last_became_active":"2023-10-16T13:46:25.566470+0000","last_became_peered":"2023-10-16T13:46:25.566470+0000","last_unstale":"2023-10-16T13:46:25.573094+0000","last_undegraded":"2023-10-16T13:46:25.573094+0000","last_fullsized":"2023-10-16T13:46:25.573094+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:41:30.230501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.2d","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.580018+0000","last_change":"2023-10-16T13:46:25.578104+0000","last_active":"2023-10-16T13:46:25.580018+0000","last_peered":"2023-10-16T13:46:25.580018+0000","last_clean":"2023-10-16T13:46:25.580018+0000","last_became_active":"2023-10-16T13:46:25.577833+0000","last_became_peered":"2023-10-16T13:46:25.577833+0000","last_unstale":"2023-10-16T13:46:25.580018+0000","last_undegraded":"2023-10-16T13:46:25.580018+0000","last_fullsized":"2023-10-16T13:46:25.580018+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:02:44.293532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2c","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.577162+0000","last_change":"2023-10-16T13:46:25.572889+0000","last_active":"2023-10-16T13:46:25.577162+0000","last_peered":"2023-10-16T13:46:25.577162+0000","last_clean":"2023-10-16T13:46:25.577162+0000","last_became_active":"2023-10-16T13:46:25.572611+0000","last_became_peered":"2023-10-16T13:46:25.572611+0000","last_unstale":"2023-10-16T13:46:25.577162+0000","last_undegraded":"2023-10-16T13:46:25.577162+0000","last_fullsized":"2023-10-16T13:46:25.577162+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:36:05.027611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2b","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.589925+0000","last_change":"2023-10-16T13:46:25.584056+0000","last_active":"2023-10-16T13:46:25.589925+0000","last_peered":"2023-10-16T13:46:25.589925+0000","last_clean":"2023-10-16T13:46:25.589925+0000","last_became_active":"2023-10-16T13:46:25.583758+0000","last_became_peered":"2023-10-16T13:46:25.583758+0000","last_unstale":"2023-10-16T13:46:25.589925+0000","last_undegraded":"2023-10-16T13:46:25.589925+0000","last_fullsized":"2023-10-16T13:46:25.589925+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:58:38.153452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2a","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.583764+0000","last_change":"2023-10-16T13:46:25.581238+0000","last_active":"2023-10-16T13:46:25.583764+0000","last_peered":"2023-10-16T13:46:25.583764+0000","last_clean":"2023-10-16T13:46:25.583764+0000","last_became_active":"2023-10-16T13:46:25.580555+0000","last_became_peered":"2023-10-16T13:46:25.580555+0000","last_unstale":"2023-10-16T13:46:25.583764+0000","last_undegraded":"2023-10-16T13:46:25.583764+0000","last_fullsized":"2023-10-16T13:46:25.583764+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:16:22.462066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.29","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.581349+0000","last_change":"2023-10-16T13:46:25.578743+0000","last_active":"2023-10-16T13:46:25.581349+0000","last_peered":"2023-10-16T13:46:25.581349+0000","last_clean":"2023-10-16T13:46:25.581349+0000","last_became_active":"2023-10-16T13:46:25.578447+0000","last_became_peered":"2023-10-16T13:46:25.578447+0000","last_unstale":"2023-10-16T13:46:25.581349+0000","last_undegraded":"2023-10-16T13:46:25.581349+0000","last_fullsized":"2023-10-16T13:46:25.581349+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:25:52.567353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.28","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.584554+0000","last_change":"2023-10-16T13:46:25.583046+0000","last_active":"2023-10-16T13:46:25.584554+0000","last_peered":"2023-10-16T13:46:25.584554+0000","last_clean":"2023-10-16T13:46:25.584554+0000","last_became_active":"2023-10-16T13:46:25.582711+0000","last_became_peered":"2023-10-16T13:46:25.582711+0000","last_unstale":"2023-10-16T13:46:25.584554+0000","last_undegraded":"2023-10-16T13:46:25.584554+0000","last_fullsized":"2023-10-16T13:46:25.584554+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:24:48.472015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.27","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.573342+0000","last_change":"2023-10-16T13:46:25.572081+0000","last_active":"2023-10-16T13:46:25.573342+0000","last_peered":"2023-10-16T13:46:25.573342+0000","last_clean":"2023-10-16T13:46:25.573342+0000","last_became_active":"2023-10-16T13:46:25.568456+0000","last_became_peered":"2023-10-16T13:46:25.568456+0000","last_unstale":"2023-10-16T13:46:25.573342+0000","last_undegraded":"2023-10-16T13:46:25.573342+0000","last_fullsized":"2023-10-16T13:46:25.573342+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:53:48.001931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.26","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.582249+0000","last_change":"2023-10-16T13:46:25.579701+0000","last_active":"2023-10-16T13:46:25.582249+0000","last_peered":"2023-10-16T13:46:25.582249+0000","last_clean":"2023-10-16T13:46:25.582249+0000","last_became_active":"2023-10-16T13:46:25.579378+0000","last_became_peered":"2023-10-16T13:46:25.579378+0000","last_unstale":"2023-10-16T13:46:25.582249+0000","last_undegraded":"2023-10-16T13:46:25.582249+0000","last_fullsized":"2023-10-16T13:46:25.582249+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:06:17.800392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.25","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.580766+0000","last_change":"2023-10-16T13:46:25.578298+0000","last_active":"2023-10-16T13:46:25.580766+0000","last_peered":"2023-10-16T13:46:25.580766+0000","last_clean":"2023-10-16T13:46:25.580766+0000","last_became_active":"2023-10-16T13:46:25.578057+0000","last_became_peered":"2023-10-16T13:46:25.578057+0000","last_unstale":"2023-10-16T13:46:25.580766+0000","last_undegraded":"2023-10-16T13:46:25.580766+0000","last_fullsized":"2023-10-16T13:46:25.580766+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:26:58.168760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.24","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.576314+0000","last_change":"2023-10-16T13:46:25.573728+0000","last_active":"2023-10-16T13:46:25.576314+0000","last_peered":"2023-10-16T13:46:25.576314+0000","last_clean":"2023-10-16T13:46:25.576314+0000","last_became_active":"2023-10-16T13:46:25.573427+0000","last_became_peered":"2023-10-16T13:46:25.573427+0000","last_unstale":"2023-10-16T13:46:25.576314+0000","last_undegraded":"2023-10-16T13:46:25.576314+0000","last_fullsized":"2023-10-16T13:46:25.576314+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:56:16.772087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.23","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.573957+0000","last_change":"2023-10-16T13:46:25.568230+0000","last_active":"2023-10-16T13:46:25.573957+0000","last_peered":"2023-10-16T13:46:25.573957+0000","last_clean":"2023-10-16T13:46:25.573957+0000","last_became_active":"2023-10-16T13:46:25.567927+0000","last_became_peered":"2023-10-16T13:46:25.567927+0000","last_unstale":"2023-10-16T13:46:25.573957+0000","last_undegraded":"2023-10-16T13:46:25.573957+0000","last_fullsized":"2023-10-16T13:46:25.573957+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:44:11.242845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.22","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.571779+0000","last_change":"2023-10-16T13:46:25.569845+0000","last_active":"2023-10-16T13:46:25.571779+0000","last_peered":"2023-10-16T13:46:25.571779+0000","last_clean":"2023-10-16T13:46:25.571779+0000","last_became_active":"2023-10-16T13:46:25.569524+0000","last_became_peered":"2023-10-16T13:46:25.569524+0000","last_unstale":"2023-10-16T13:46:25.571779+0000","last_undegraded":"2023-10-16T13:46:25.571779+0000","last_fullsized":"2023-10-16T13:46:25.571779+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:34:10.802361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.21","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.583525+0000","last_change":"2023-10-16T13:46:25.580726+0000","last_active":"2023-10-16T13:46:25.583525+0000","last_peered":"2023-10-16T13:46:25.583525+0000","last_clean":"2023-10-16T13:46:25.583525+0000","last_became_active":"2023-10-16T13:46:25.580423+0000","last_became_peered":"2023-10-16T13:46:25.580423+0000","last_unstale":"2023-10-16T13:46:25.583525+0000","last_undegraded":"2023-10-16T13:46:25.583525+0000","last_fullsized":"2023-10-16T13:46:25.583525+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:25:08.151545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.20","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.578259+0000","last_change":"2023-10-16T13:46:25.572825+0000","last_active":"2023-10-16T13:46:25.578259+0000","last_peered":"2023-10-16T13:46:25.578259+0000","last_clean":"2023-10-16T13:46:25.578259+0000","last_became_active":"2023-10-16T13:46:25.572507+0000","last_became_peered":"2023-10-16T13:46:25.572507+0000","last_unstale":"2023-10-16T13:46:25.578259+0000","last_undegraded":"2023-10-16T13:46:25.578259+0000","last_fullsized":"2023-10-16T13:46:25.578259+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:44:11.242845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.581295+0000","last_change":"2023-10-16T13:46:25.578414+0000","last_active":"2023-10-16T13:46:25.581295+0000","last_peered":"2023-10-16T13:46:25.581295+0000","last_clean":"2023-10-16T13:46:25.581295+0000","last_became_active":"2023-10-16T13:46:25.578111+0000","last_became_peered":"2023-10-16T13:46:25.578111+0000","last_unstale":"2023-10-16T13:46:25.581295+0000","last_undegraded":"2023-10-16T13:46:25.581295+0000","last_fullsized":"2023-10-16T13:46:25.581295+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:14:53.997941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.1f","version":"224'4","reported_seq":15,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.595125+0000","last_change":"2023-10-16T13:46:24.345142+0000","last_active":"2023-10-16T13:46:25.595125+0000","last_peered":"2023-10-16T13:46:25.595125+0000","last_clean":"2023-10-16T13:46:25.595125+0000","last_became_active":"2023-10-16T13:46:24.344816+0000","last_became_peered":"2023-10-16T13:46:24.344816+0000","last_unstale":"2023-10-16T13:46:25.595125+0000","last_undegraded":"2023-10-16T13:46:25.595125+0000","last_fullsized":"2023-10-16T13:46:25.595125+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:54:50.780055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.579816+0000","last_change":"2023-10-16T13:46:25.577384+0000","last_active":"2023-10-16T13:46:25.579816+0000","last_peered":"2023-10-16T13:46:25.579816+0000","last_clean":"2023-10-16T13:46:25.579816+0000","last_became_active":"2023-10-16T13:46:25.577071+0000","last_became_peered":"2023-10-16T13:46:25.577071+0000","last_unstale":"2023-10-16T13:46:25.579816+0000","last_undegraded":"2023-10-16T13:46:25.579816+0000","last_fullsized":"2023-10-16T13:46:25.579816+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:04:10.854058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1e","version":"224'1","reported_seq":12,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.594198+0000","last_change":"2023-10-16T13:46:24.340000+0000","last_active":"2023-10-16T13:46:25.594198+0000","last_peered":"2023-10-16T13:46:25.594198+0000","last_clean":"2023-10-16T13:46:25.594198+0000","last_became_active":"2023-10-16T13:46:24.339714+0000","last_became_peered":"2023-10-16T13:46:24.339714+0000","last_unstale":"2023-10-16T13:46:25.594198+0000","last_undegraded":"2023-10-16T13:46:25.594198+0000","last_fullsized":"2023-10-16T13:46:25.594198+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:21:04.619534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.579291+0000","last_change":"2023-10-16T13:46:25.576216+0000","last_active":"2023-10-16T13:46:25.579291+0000","last_peered":"2023-10-16T13:46:25.579291+0000","last_clean":"2023-10-16T13:46:25.579291+0000","last_became_active":"2023-10-16T13:46:25.575876+0000","last_became_peered":"2023-10-16T13:46:25.575876+0000","last_unstale":"2023-10-16T13:46:25.579291+0000","last_undegraded":"2023-10-16T13:46:25.579291+0000","last_fullsized":"2023-10-16T13:46:25.579291+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:29:01.405087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1d","version":"224'2","reported_seq":13,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.593539+0000","last_change":"2023-10-16T13:46:24.340608+0000","last_active":"2023-10-16T13:46:25.593539+0000","last_peered":"2023-10-16T13:46:25.593539+0000","last_clean":"2023-10-16T13:46:25.593539+0000","last_became_active":"2023-10-16T13:46:24.340189+0000","last_became_peered":"2023-10-16T13:46:24.340189+0000","last_unstale":"2023-10-16T13:46:25.593539+0000","last_undegraded":"2023-10-16T13:46:25.593539+0000","last_fullsized":"2023-10-16T13:46:25.593539+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:16:00.359557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1b","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.574063+0000","last_change":"2023-10-16T13:46:25.571278+0000","last_active":"2023-10-16T13:46:25.574063+0000","last_peered":"2023-10-16T13:46:25.574063+0000","last_clean":"2023-10-16T13:46:25.574063+0000","last_became_active":"2023-10-16T13:46:25.570849+0000","last_became_peered":"2023-10-16T13:46:25.570849+0000","last_unstale":"2023-10-16T13:46:25.574063+0000","last_undegraded":"2023-10-16T13:46:25.574063+0000","last_fullsized":"2023-10-16T13:46:25.574063+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:03:49.009125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1c","version":"224'2","reported_seq":13,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.596447+0000","last_change":"2023-10-16T13:46:24.340877+0000","last_active":"2023-10-16T13:46:25.596447+0000","last_peered":"2023-10-16T13:46:25.596447+0000","last_clean":"2023-10-16T13:46:25.596447+0000","last_became_active":"2023-10-16T13:46:24.340607+0000","last_became_peered":"2023-10-16T13:46:24.340607+0000","last_unstale":"2023-10-16T13:46:25.596447+0000","last_undegraded":"2023-10-16T13:46:25.596447+0000","last_fullsized":"2023-10-16T13:46:25.596447+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:16:31.012130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.571853+0000","last_change":"2023-10-16T13:46:25.569919+0000","last_active":"2023-10-16T13:46:25.571853+0000","last_peered":"2023-10-16T13:46:25.571853+0000","last_clean":"2023-10-16T13:46:25.571853+0000","last_became_active":"2023-10-16T13:46:25.569619+0000","last_became_peered":"2023-10-16T13:46:25.569619+0000","last_unstale":"2023-10-16T13:46:25.571853+0000","last_undegraded":"2023-10-16T13:46:25.571853+0000","last_fullsized":"2023-10-16T13:46:25.571853+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:27:58.479275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.1b","version":"224'1","reported_seq":12,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.593694+0000","last_change":"2023-10-16T13:46:24.343654+0000","last_active":"2023-10-16T13:46:25.593694+0000","last_peered":"2023-10-16T13:46:25.593694+0000","last_clean":"2023-10-16T13:46:25.593694+0000","last_became_active":"2023-10-16T13:46:24.343330+0000","last_became_peered":"2023-10-16T13:46:24.343330+0000","last_unstale":"2023-10-16T13:46:25.593694+0000","last_undegraded":"2023-10-16T13:46:25.593694+0000","last_fullsized":"2023-10-16T13:46:25.593694+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:47:30.680184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.572282+0000","last_change":"2023-10-16T13:46:25.571656+0000","last_active":"2023-10-16T13:46:25.572282+0000","last_peered":"2023-10-16T13:46:25.572282+0000","last_clean":"2023-10-16T13:46:25.572282+0000","last_became_active":"2023-10-16T13:46:25.571372+0000","last_became_peered":"2023-10-16T13:46:25.571372+0000","last_unstale":"2023-10-16T13:46:25.572282+0000","last_undegraded":"2023-10-16T13:46:25.572282+0000","last_fullsized":"2023-10-16T13:46:25.572282+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:10:48.517374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":11,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.564945+0000","last_change":"2023-10-16T13:46:24.343337+0000","last_active":"2023-10-16T13:46:25.564945+0000","last_peered":"2023-10-16T13:46:25.564945+0000","last_clean":"2023-10-16T13:46:25.564945+0000","last_became_active":"2023-10-16T13:46:24.343043+0000","last_became_peered":"2023-10-16T13:46:24.343043+0000","last_unstale":"2023-10-16T13:46:25.564945+0000","last_undegraded":"2023-10-16T13:46:25.564945+0000","last_fullsized":"2023-10-16T13:46:25.564945+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:16:31.012130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1f","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.573321+0000","last_change":"2023-10-16T13:46:25.571884+0000","last_active":"2023-10-16T13:46:25.573321+0000","last_peered":"2023-10-16T13:46:25.573321+0000","last_clean":"2023-10-16T13:46:25.573321+0000","last_became_active":"2023-10-16T13:46:25.571601+0000","last_became_peered":"2023-10-16T13:46:25.571601+0000","last_unstale":"2023-10-16T13:46:25.573321+0000","last_undegraded":"2023-10-16T13:46:25.573321+0000","last_fullsized":"2023-10-16T13:46:25.573321+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:35:32.750171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":11,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.563849+0000","last_change":"2023-10-16T13:46:24.338318+0000","last_active":"2023-10-16T13:46:25.563849+0000","last_peered":"2023-10-16T13:46:25.563849+0000","last_clean":"2023-10-16T13:46:25.563849+0000","last_became_active":"2023-10-16T13:46:24.337766+0000","last_became_peered":"2023-10-16T13:46:24.337766+0000","last_unstale":"2023-10-16T13:46:25.563849+0000","last_undegraded":"2023-10-16T13:46:25.563849+0000","last_fullsized":"2023-10-16T13:46:25.563849+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:25:07.149748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.575412+0000","last_change":"2023-10-16T13:46:25.574355+0000","last_active":"2023-10-16T13:46:25.575412+0000","last_peered":"2023-10-16T13:46:25.575412+0000","last_clean":"2023-10-16T13:46:25.575412+0000","last_became_active":"2023-10-16T13:46:25.574057+0000","last_became_peered":"2023-10-16T13:46:25.574057+0000","last_unstale":"2023-10-16T13:46:25.575412+0000","last_undegraded":"2023-10-16T13:46:25.575412+0000","last_fullsized":"2023-10-16T13:46:25.575412+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:19:04.312392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":11,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.567482+0000","last_change":"2023-10-16T13:46:24.346329+0000","last_active":"2023-10-16T13:46:25.567482+0000","last_peered":"2023-10-16T13:46:25.567482+0000","last_clean":"2023-10-16T13:46:25.567482+0000","last_became_active":"2023-10-16T13:46:24.346051+0000","last_became_peered":"2023-10-16T13:46:24.346051+0000","last_unstale":"2023-10-16T13:46:25.567482+0000","last_undegraded":"2023-10-16T13:46:25.567482+0000","last_fullsized":"2023-10-16T13:46:25.567482+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:14:11.499564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.573701+0000","last_change":"2023-10-16T13:46:25.572773+0000","last_active":"2023-10-16T13:46:25.573701+0000","last_peered":"2023-10-16T13:46:25.573701+0000","last_clean":"2023-10-16T13:46:25.573701+0000","last_became_active":"2023-10-16T13:46:25.572324+0000","last_became_peered":"2023-10-16T13:46:25.572324+0000","last_unstale":"2023-10-16T13:46:25.573701+0000","last_undegraded":"2023-10-16T13:46:25.573701+0000","last_fullsized":"2023-10-16T13:46:25.573701+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:16:32.013927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":11,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.567680+0000","last_change":"2023-10-16T13:46:24.344174+0000","last_active":"2023-10-16T13:46:25.567680+0000","last_peered":"2023-10-16T13:46:25.567680+0000","last_clean":"2023-10-16T13:46:25.567680+0000","last_became_active":"2023-10-16T13:46:24.343883+0000","last_became_peered":"2023-10-16T13:46:24.343883+0000","last_unstale":"2023-10-16T13:46:25.567680+0000","last_undegraded":"2023-10-16T13:46:25.567680+0000","last_fullsized":"2023-10-16T13:46:25.567680+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:53:47.000133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.580895+0000","last_change":"2023-10-16T13:46:25.580296+0000","last_active":"2023-10-16T13:46:25.580895+0000","last_peered":"2023-10-16T13:46:25.580895+0000","last_clean":"2023-10-16T13:46:25.580895+0000","last_became_active":"2023-10-16T13:46:25.579963+0000","last_became_peered":"2023-10-16T13:46:25.579963+0000","last_unstale":"2023-10-16T13:46:25.580895+0000","last_undegraded":"2023-10-16T13:46:25.580895+0000","last_fullsized":"2023-10-16T13:46:25.580895+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:26:58.870084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.15","version":"224'3","reported_seq":14,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.596696+0000","last_change":"2023-10-16T13:46:24.340273+0000","last_active":"2023-10-16T13:46:25.596696+0000","last_peered":"2023-10-16T13:46:25.596696+0000","last_clean":"2023-10-16T13:46:25.596696+0000","last_became_active":"2023-10-16T13:46:24.339939+0000","last_became_peered":"2023-10-16T13:46:24.339939+0000","last_unstale":"2023-10-16T13:46:25.596696+0000","last_undegraded":"2023-10-16T13:46:25.596696+0000","last_fullsized":"2023-10-16T13:46:25.596696+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:25:07.149748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.13","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.580537+0000","last_change":"2023-10-16T13:46:25.578929+0000","last_active":"2023-10-16T13:46:25.580537+0000","last_peered":"2023-10-16T13:46:25.580537+0000","last_clean":"2023-10-16T13:46:25.580537+0000","last_became_active":"2023-10-16T13:46:25.578431+0000","last_became_peered":"2023-10-16T13:46:25.578431+0000","last_unstale":"2023-10-16T13:46:25.580537+0000","last_undegraded":"2023-10-16T13:46:25.580537+0000","last_fullsized":"2023-10-16T13:46:25.580537+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:16:05.895498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.14","version":"224'6","reported_seq":17,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.596786+0000","last_change":"2023-10-16T13:46:24.347248+0000","last_active":"2023-10-16T13:46:25.596786+0000","last_peered":"2023-10-16T13:46:25.596786+0000","last_clean":"2023-10-16T13:46:25.596786+0000","last_became_active":"2023-10-16T13:46:24.346699+0000","last_became_peered":"2023-10-16T13:46:24.346699+0000","last_unstale":"2023-10-16T13:46:25.596786+0000","last_undegraded":"2023-10-16T13:46:25.596786+0000","last_fullsized":"2023-10-16T13:46:25.596786+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:15:37.760742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.14","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.568407+0000","last_change":"2023-10-16T13:46:25.566526+0000","last_active":"2023-10-16T13:46:25.568407+0000","last_peered":"2023-10-16T13:46:25.568407+0000","last_clean":"2023-10-16T13:46:25.568407+0000","last_became_active":"2023-10-16T13:46:25.566252+0000","last_became_peered":"2023-10-16T13:46:25.566252+0000","last_unstale":"2023-10-16T13:46:25.568407+0000","last_undegraded":"2023-10-16T13:46:25.568407+0000","last_fullsized":"2023-10-16T13:46:25.568407+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:45:30.778861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.13","version":"224'2","reported_seq":13,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.595410+0000","last_change":"2023-10-16T13:46:24.333306+0000","last_active":"2023-10-16T13:46:25.595410+0000","last_peered":"2023-10-16T13:46:25.595410+0000","last_clean":"2023-10-16T13:46:25.595410+0000","last_became_active":"2023-10-16T13:46:24.333042+0000","last_became_peered":"2023-10-16T13:46:24.333042+0000","last_unstale":"2023-10-16T13:46:25.595410+0000","last_undegraded":"2023-10-16T13:46:25.595410+0000","last_fullsized":"2023-10-16T13:46:25.595410+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:44:45.300765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.578222+0000","last_change":"2023-10-16T13:46:25.575462+0000","last_active":"2023-10-16T13:46:25.578222+0000","last_peered":"2023-10-16T13:46:25.578222+0000","last_clean":"2023-10-16T13:46:25.578222+0000","last_became_active":"2023-10-16T13:46:25.574929+0000","last_became_peered":"2023-10-16T13:46:25.574929+0000","last_unstale":"2023-10-16T13:46:25.578222+0000","last_undegraded":"2023-10-16T13:46:25.578222+0000","last_fullsized":"2023-10-16T13:46:25.578222+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:26:58.870084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":11,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.562921+0000","last_change":"2023-10-16T13:46:24.336805+0000","last_active":"2023-10-16T13:46:25.562921+0000","last_peered":"2023-10-16T13:46:25.562921+0000","last_clean":"2023-10-16T13:46:25.562921+0000","last_became_active":"2023-10-16T13:46:24.336473+0000","last_became_peered":"2023-10-16T13:46:24.336473+0000","last_unstale":"2023-10-16T13:46:25.562921+0000","last_undegraded":"2023-10-16T13:46:25.562921+0000","last_fullsized":"2023-10-16T13:46:25.562921+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:44:10.241047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.576844+0000","last_change":"2023-10-16T13:46:25.573821+0000","last_active":"2023-10-16T13:46:25.576844+0000","last_peered":"2023-10-16T13:46:25.576844+0000","last_clean":"2023-10-16T13:46:25.576844+0000","last_became_active":"2023-10-16T13:46:25.572882+0000","last_became_peered":"2023-10-16T13:46:25.572882+0000","last_unstale":"2023-10-16T13:46:25.576844+0000","last_undegraded":"2023-10-16T13:46:25.576844+0000","last_fullsized":"2023-10-16T13:46:25.576844+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:10:39.170811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":11,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.558466+0000","last_change":"2023-10-16T13:46:24.332386+0000","last_active":"2023-10-16T13:46:25.558466+0000","last_peered":"2023-10-16T13:46:25.558466+0000","last_clean":"2023-10-16T13:46:25.558466+0000","last_became_active":"2023-10-16T13:46:24.332013+0000","last_became_peered":"2023-10-16T13:46:24.332013+0000","last_unstale":"2023-10-16T13:46:25.558466+0000","last_undegraded":"2023-10-16T13:46:25.558466+0000","last_fullsized":"2023-10-16T13:46:25.558466+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:04:46.312175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.586582+0000","last_change":"2023-10-16T13:46:25.584003+0000","last_active":"2023-10-16T13:46:25.586582+0000","last_peered":"2023-10-16T13:46:25.586582+0000","last_clean":"2023-10-16T13:46:25.586582+0000","last_became_active":"2023-10-16T13:46:25.583732+0000","last_became_peered":"2023-10-16T13:46:25.583732+0000","last_unstale":"2023-10-16T13:46:25.586582+0000","last_undegraded":"2023-10-16T13:46:25.586582+0000","last_fullsized":"2023-10-16T13:46:25.586582+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:17:18.357714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.10","version":"224'8","reported_seq":19,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.596537+0000","last_change":"2023-10-16T13:46:24.346086+0000","last_active":"2023-10-16T13:46:25.596537+0000","last_peered":"2023-10-16T13:46:25.596537+0000","last_clean":"2023-10-16T13:46:25.596537+0000","last_became_active":"2023-10-16T13:46:24.345786+0000","last_became_peered":"2023-10-16T13:46:24.345786+0000","last_unstale":"2023-10-16T13:46:25.596537+0000","last_undegraded":"2023-10-16T13:46:25.596537+0000","last_fullsized":"2023-10-16T13:46:25.596537+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:39:19.890170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.593434+0000","last_change":"2023-10-16T13:46:25.585651+0000","last_active":"2023-10-16T13:46:25.593434+0000","last_peered":"2023-10-16T13:46:25.593434+0000","last_clean":"2023-10-16T13:46:25.593434+0000","last_became_active":"2023-10-16T13:46:25.585367+0000","last_became_peered":"2023-10-16T13:46:25.585367+0000","last_unstale":"2023-10-16T13:46:25.593434+0000","last_undegraded":"2023-10-16T13:46:25.593434+0000","last_fullsized":"2023-10-16T13:46:25.593434+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:16:03.750659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.f","version":"224'1","reported_seq":12,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.595561+0000","last_change":"2023-10-16T13:46:24.338988+0000","last_active":"2023-10-16T13:46:25.595561+0000","last_peered":"2023-10-16T13:46:25.595561+0000","last_clean":"2023-10-16T13:46:25.595561+0000","last_became_active":"2023-10-16T13:46:24.338477+0000","last_became_peered":"2023-10-16T13:46:24.338477+0000","last_unstale":"2023-10-16T13:46:25.595561+0000","last_undegraded":"2023-10-16T13:46:25.595561+0000","last_fullsized":"2023-10-16T13:46:25.595561+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:47:21.411665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.579844+0000","last_change":"2023-10-16T13:46:25.578233+0000","last_active":"2023-10-16T13:46:25.579844+0000","last_peered":"2023-10-16T13:46:25.579844+0000","last_clean":"2023-10-16T13:46:25.579844+0000","last_became_active":"2023-10-16T13:46:25.577908+0000","last_became_peered":"2023-10-16T13:46:25.577908+0000","last_unstale":"2023-10-16T13:46:25.579844+0000","last_undegraded":"2023-10-16T13:46:25.579844+0000","last_fullsized":"2023-10-16T13:46:25.579844+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:19:05.170819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.e","version":"224'2","reported_seq":13,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.592651+0000","last_change":"2023-10-16T13:46:24.339351+0000","last_active":"2023-10-16T13:46:25.592651+0000","last_peered":"2023-10-16T13:46:25.592651+0000","last_clean":"2023-10-16T13:46:25.592651+0000","last_became_active":"2023-10-16T13:46:24.339062+0000","last_became_peered":"2023-10-16T13:46:24.339062+0000","last_unstale":"2023-10-16T13:46:25.592651+0000","last_undegraded":"2023-10-16T13:46:25.592651+0000","last_fullsized":"2023-10-16T13:46:25.592651+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:14:49.845621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.587605+0000","last_change":"2023-10-16T13:46:25.582651+0000","last_active":"2023-10-16T13:46:25.587605+0000","last_peered":"2023-10-16T13:46:25.587605+0000","last_clean":"2023-10-16T13:46:25.587605+0000","last_became_active":"2023-10-16T13:46:25.582350+0000","last_became_peered":"2023-10-16T13:46:25.582350+0000","last_unstale":"2023-10-16T13:46:25.587605+0000","last_undegraded":"2023-10-16T13:46:25.587605+0000","last_fullsized":"2023-10-16T13:46:25.587605+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:23:29.519783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.d","version":"224'1","reported_seq":12,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.595657+0000","last_change":"2023-10-16T13:46:24.338316+0000","last_active":"2023-10-16T13:46:25.595657+0000","last_peered":"2023-10-16T13:46:25.595657+0000","last_clean":"2023-10-16T13:46:25.595657+0000","last_became_active":"2023-10-16T13:46:24.338002+0000","last_became_peered":"2023-10-16T13:46:24.338002+0000","last_unstale":"2023-10-16T13:46:25.595657+0000","last_undegraded":"2023-10-16T13:46:25.595657+0000","last_fullsized":"2023-10-16T13:46:25.595657+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:04:35.398533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.580910+0000","last_change":"2023-10-16T13:46:25.579683+0000","last_active":"2023-10-16T13:46:25.580910+0000","last_peered":"2023-10-16T13:46:25.580910+0000","last_clean":"2023-10-16T13:46:25.580910+0000","last_became_active":"2023-10-16T13:46:25.579398+0000","last_became_peered":"2023-10-16T13:46:25.579398+0000","last_unstale":"2023-10-16T13:46:25.580910+0000","last_undegraded":"2023-10-16T13:46:25.580910+0000","last_fullsized":"2023-10-16T13:46:25.580910+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:34:10.802361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":11,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.564021+0000","last_change":"2023-10-16T13:46:24.339475+0000","last_active":"2023-10-16T13:46:25.564021+0000","last_peered":"2023-10-16T13:46:25.564021+0000","last_clean":"2023-10-16T13:46:25.564021+0000","last_became_active":"2023-10-16T13:46:24.339105+0000","last_became_peered":"2023-10-16T13:46:24.339105+0000","last_unstale":"2023-10-16T13:46:25.564021+0000","last_undegraded":"2023-10-16T13:46:25.564021+0000","last_fullsized":"2023-10-16T13:46:25.564021+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:13:27.085544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.579261+0000","last_change":"2023-10-16T13:46:25.572423+0000","last_active":"2023-10-16T13:46:25.579261+0000","last_peered":"2023-10-16T13:46:25.579261+0000","last_clean":"2023-10-16T13:46:25.579261+0000","last_became_active":"2023-10-16T13:46:25.569272+0000","last_became_peered":"2023-10-16T13:46:25.569272+0000","last_unstale":"2023-10-16T13:46:25.579261+0000","last_undegraded":"2023-10-16T13:46:25.579261+0000","last_fullsized":"2023-10-16T13:46:25.579261+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:45:10.250543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.b","version":"224'2","reported_seq":13,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.595169+0000","last_change":"2023-10-16T13:46:24.347651+0000","last_active":"2023-10-16T13:46:25.595169+0000","last_peered":"2023-10-16T13:46:25.595169+0000","last_clean":"2023-10-16T13:46:25.595169+0000","last_became_active":"2023-10-16T13:46:24.347370+0000","last_became_peered":"2023-10-16T13:46:24.347370+0000","last_unstale":"2023-10-16T13:46:25.595169+0000","last_undegraded":"2023-10-16T13:46:25.595169+0000","last_fullsized":"2023-10-16T13:46:25.595169+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:18:32.988431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.583510+0000","last_change":"2023-10-16T13:46:25.580894+0000","last_active":"2023-10-16T13:46:25.583510+0000","last_peered":"2023-10-16T13:46:25.583510+0000","last_clean":"2023-10-16T13:46:25.583510+0000","last_became_active":"2023-10-16T13:46:25.580589+0000","last_became_peered":"2023-10-16T13:46:25.580589+0000","last_unstale":"2023-10-16T13:46:25.583510+0000","last_undegraded":"2023-10-16T13:46:25.583510+0000","last_fullsized":"2023-10-16T13:46:25.583510+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:26:58.870084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":11,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.559163+0000","last_change":"2023-10-16T13:46:24.337053+0000","last_active":"2023-10-16T13:46:25.559163+0000","last_peered":"2023-10-16T13:46:25.559163+0000","last_clean":"2023-10-16T13:46:25.559163+0000","last_became_active":"2023-10-16T13:46:24.336736+0000","last_became_peered":"2023-10-16T13:46:24.336736+0000","last_unstale":"2023-10-16T13:46:25.559163+0000","last_undegraded":"2023-10-16T13:46:25.559163+0000","last_fullsized":"2023-10-16T13:46:25.559163+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:36:24.043336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.573223+0000","last_change":"2023-10-16T13:46:25.570388+0000","last_active":"2023-10-16T13:46:25.573223+0000","last_peered":"2023-10-16T13:46:25.573223+0000","last_clean":"2023-10-16T13:46:25.573223+0000","last_became_active":"2023-10-16T13:46:25.570059+0000","last_became_peered":"2023-10-16T13:46:25.570059+0000","last_unstale":"2023-10-16T13:46:25.573223+0000","last_undegraded":"2023-10-16T13:46:25.573223+0000","last_fullsized":"2023-10-16T13:46:25.573223+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:18:48.273122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":11,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.567409+0000","last_change":"2023-10-16T13:46:24.344523+0000","last_active":"2023-10-16T13:46:25.567409+0000","last_peered":"2023-10-16T13:46:25.567409+0000","last_clean":"2023-10-16T13:46:25.567409+0000","last_became_active":"2023-10-16T13:46:24.344238+0000","last_became_peered":"2023-10-16T13:46:24.344238+0000","last_unstale":"2023-10-16T13:46:25.567409+0000","last_undegraded":"2023-10-16T13:46:25.567409+0000","last_fullsized":"2023-10-16T13:46:25.567409+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:37:49.931017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.580131+0000","last_change":"2023-10-16T13:46:25.576886+0000","last_active":"2023-10-16T13:46:25.580131+0000","last_peered":"2023-10-16T13:46:25.580131+0000","last_clean":"2023-10-16T13:46:25.580131+0000","last_became_active":"2023-10-16T13:46:25.576540+0000","last_became_peered":"2023-10-16T13:46:25.576540+0000","last_unstale":"2023-10-16T13:46:25.580131+0000","last_undegraded":"2023-10-16T13:46:25.580131+0000","last_fullsized":"2023-10-16T13:46:25.580131+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:10:48.517374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":11,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.565462+0000","last_change":"2023-10-16T13:46:24.341910+0000","last_active":"2023-10-16T13:46:25.565462+0000","last_peered":"2023-10-16T13:46:25.565462+0000","last_clean":"2023-10-16T13:46:25.565462+0000","last_became_active":"2023-10-16T13:46:24.341633+0000","last_became_peered":"2023-10-16T13:46:24.341633+0000","last_unstale":"2023-10-16T13:46:25.565462+0000","last_undegraded":"2023-10-16T13:46:25.565462+0000","last_fullsized":"2023-10-16T13:46:25.565462+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:00:55.319505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.593421+0000","last_change":"2023-10-16T13:46:25.585631+0000","last_active":"2023-10-16T13:46:25.593421+0000","last_peered":"2023-10-16T13:46:25.593421+0000","last_clean":"2023-10-16T13:46:25.593421+0000","last_became_active":"2023-10-16T13:46:25.585329+0000","last_became_peered":"2023-10-16T13:46:25.585329+0000","last_unstale":"2023-10-16T13:46:25.593421+0000","last_undegraded":"2023-10-16T13:46:25.593421+0000","last_fullsized":"2023-10-16T13:46:25.593421+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:25:56.358129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.7","version":"224'2","reported_seq":13,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.595993+0000","last_change":"2023-10-16T13:46:24.344410+0000","last_active":"2023-10-16T13:46:25.595993+0000","last_peered":"2023-10-16T13:46:25.595993+0000","last_clean":"2023-10-16T13:46:25.595993+0000","last_became_active":"2023-10-16T13:46:24.343665+0000","last_became_peered":"2023-10-16T13:46:24.343665+0000","last_unstale":"2023-10-16T13:46:25.595993+0000","last_undegraded":"2023-10-16T13:46:25.595993+0000","last_fullsized":"2023-10-16T13:46:25.595993+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:21:31.259195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.582757+0000","last_change":"2023-10-16T13:46:25.577247+0000","last_active":"2023-10-16T13:46:25.582757+0000","last_peered":"2023-10-16T13:46:25.582757+0000","last_clean":"2023-10-16T13:46:25.582757+0000","last_became_active":"2023-10-16T13:46:25.576949+0000","last_became_peered":"2023-10-16T13:46:25.576949+0000","last_unstale":"2023-10-16T13:46:25.582757+0000","last_undegraded":"2023-10-16T13:46:25.582757+0000","last_fullsized":"2023-10-16T13:46:25.582757+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:31:15.200012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":11,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.563517+0000","last_change":"2023-10-16T13:46:24.346027+0000","last_active":"2023-10-16T13:46:25.563517+0000","last_peered":"2023-10-16T13:46:25.563517+0000","last_clean":"2023-10-16T13:46:25.563517+0000","last_became_active":"2023-10-16T13:46:24.345748+0000","last_became_peered":"2023-10-16T13:46:24.345748+0000","last_unstale":"2023-10-16T13:46:25.563517+0000","last_undegraded":"2023-10-16T13:46:25.563517+0000","last_fullsized":"2023-10-16T13:46:25.563517+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:23:29.526203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.572900+0000","last_change":"2023-10-16T13:46:25.567174+0000","last_active":"2023-10-16T13:46:25.572900+0000","last_peered":"2023-10-16T13:46:25.572900+0000","last_clean":"2023-10-16T13:46:25.572900+0000","last_became_active":"2023-10-16T13:46:25.566392+0000","last_became_peered":"2023-10-16T13:46:25.566392+0000","last_unstale":"2023-10-16T13:46:25.572900+0000","last_undegraded":"2023-10-16T13:46:25.572900+0000","last_fullsized":"2023-10-16T13:46:25.572900+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:47:31.681981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":11,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.569201+0000","last_change":"2023-10-16T13:46:24.340274+0000","last_active":"2023-10-16T13:46:25.569201+0000","last_peered":"2023-10-16T13:46:25.569201+0000","last_clean":"2023-10-16T13:46:25.569201+0000","last_became_active":"2023-10-16T13:46:24.339988+0000","last_became_peered":"2023-10-16T13:46:24.339988+0000","last_unstale":"2023-10-16T13:46:25.569201+0000","last_undegraded":"2023-10-16T13:46:25.569201+0000","last_fullsized":"2023-10-16T13:46:25.569201+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:48:44.887565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.582283+0000","last_change":"2023-10-16T13:46:25.580242+0000","last_active":"2023-10-16T13:46:25.582283+0000","last_peered":"2023-10-16T13:46:25.582283+0000","last_clean":"2023-10-16T13:46:25.582283+0000","last_became_active":"2023-10-16T13:46:25.579959+0000","last_became_peered":"2023-10-16T13:46:25.579959+0000","last_unstale":"2023-10-16T13:46:25.582283+0000","last_undegraded":"2023-10-16T13:46:25.582283+0000","last_fullsized":"2023-10-16T13:46:25.582283+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:34:14.799734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":11,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.564799+0000","last_change":"2023-10-16T13:46:24.333052+0000","last_active":"2023-10-16T13:46:25.564799+0000","last_peered":"2023-10-16T13:46:25.564799+0000","last_clean":"2023-10-16T13:46:25.564799+0000","last_became_active":"2023-10-16T13:46:24.332751+0000","last_became_peered":"2023-10-16T13:46:24.332751+0000","last_unstale":"2023-10-16T13:46:25.564799+0000","last_undegraded":"2023-10-16T13:46:25.564799+0000","last_fullsized":"2023-10-16T13:46:25.564799+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:29:38.839702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.579220+0000","last_change":"2023-10-16T13:46:25.576269+0000","last_active":"2023-10-16T13:46:25.579220+0000","last_peered":"2023-10-16T13:46:25.579220+0000","last_clean":"2023-10-16T13:46:25.579220+0000","last_became_active":"2023-10-16T13:46:25.575732+0000","last_became_peered":"2023-10-16T13:46:25.575732+0000","last_unstale":"2023-10-16T13:46:25.579220+0000","last_undegraded":"2023-10-16T13:46:25.579220+0000","last_fullsized":"2023-10-16T13:46:25.579220+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:05:54.303051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":11,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.568934+0000","last_change":"2023-10-16T13:46:24.338016+0000","last_active":"2023-10-16T13:46:25.568934+0000","last_peered":"2023-10-16T13:46:25.568934+0000","last_clean":"2023-10-16T13:46:25.568934+0000","last_became_active":"2023-10-16T13:46:24.337687+0000","last_became_peered":"2023-10-16T13:46:24.337687+0000","last_unstale":"2023-10-16T13:46:25.568934+0000","last_undegraded":"2023-10-16T13:46:25.568934+0000","last_fullsized":"2023-10-16T13:46:25.568934+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:27:57.477477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.583491+0000","last_change":"2023-10-16T13:46:25.575797+0000","last_active":"2023-10-16T13:46:25.583491+0000","last_peered":"2023-10-16T13:46:25.583491+0000","last_clean":"2023-10-16T13:46:25.583491+0000","last_became_active":"2023-10-16T13:46:25.575494+0000","last_became_peered":"2023-10-16T13:46:25.575494+0000","last_unstale":"2023-10-16T13:46:25.583491+0000","last_undegraded":"2023-10-16T13:46:25.583491+0000","last_fullsized":"2023-10-16T13:46:25.583491+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:35:32.750171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":11,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.559508+0000","last_change":"2023-10-16T13:46:24.340328+0000","last_active":"2023-10-16T13:46:25.559508+0000","last_peered":"2023-10-16T13:46:25.559508+0000","last_clean":"2023-10-16T13:46:25.559508+0000","last_became_active":"2023-10-16T13:46:24.340042+0000","last_became_peered":"2023-10-16T13:46:24.340042+0000","last_unstale":"2023-10-16T13:46:25.559508+0000","last_undegraded":"2023-10-16T13:46:25.559508+0000","last_fullsized":"2023-10-16T13:46:25.559508+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:15:37.760742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.584250+0000","last_change":"2023-10-16T13:46:25.577347+0000","last_active":"2023-10-16T13:46:25.584250+0000","last_peered":"2023-10-16T13:46:25.584250+0000","last_clean":"2023-10-16T13:46:25.584250+0000","last_became_active":"2023-10-16T13:46:25.576780+0000","last_became_peered":"2023-10-16T13:46:25.576780+0000","last_unstale":"2023-10-16T13:46:25.584250+0000","last_undegraded":"2023-10-16T13:46:25.584250+0000","last_fullsized":"2023-10-16T13:46:25.584250+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:41:30.230501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":11,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.567212+0000","last_change":"2023-10-16T13:46:24.340170+0000","last_active":"2023-10-16T13:46:25.567212+0000","last_peered":"2023-10-16T13:46:25.567212+0000","last_clean":"2023-10-16T13:46:25.567212+0000","last_became_active":"2023-10-16T13:46:24.339859+0000","last_became_peered":"2023-10-16T13:46:24.339859+0000","last_unstale":"2023-10-16T13:46:25.567212+0000","last_undegraded":"2023-10-16T13:46:25.567212+0000","last_fullsized":"2023-10-16T13:46:25.567212+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:25:04.566680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.7","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.580683+0000","last_change":"2023-10-16T13:46:25.577432+0000","last_active":"2023-10-16T13:46:25.580683+0000","last_peered":"2023-10-16T13:46:25.580683+0000","last_clean":"2023-10-16T13:46:25.580683+0000","last_became_active":"2023-10-16T13:46:25.577131+0000","last_became_peered":"2023-10-16T13:46:25.577131+0000","last_unstale":"2023-10-16T13:46:25.580683+0000","last_undegraded":"2023-10-16T13:46:25.580683+0000","last_fullsized":"2023-10-16T13:46:25.580683+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:10:39.170811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":11,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.564110+0000","last_change":"2023-10-16T13:46:24.346138+0000","last_active":"2023-10-16T13:46:25.564110+0000","last_peered":"2023-10-16T13:46:25.564110+0000","last_clean":"2023-10-16T13:46:25.564110+0000","last_became_active":"2023-10-16T13:46:24.345803+0000","last_became_peered":"2023-10-16T13:46:24.345803+0000","last_unstale":"2023-10-16T13:46:25.564110+0000","last_undegraded":"2023-10-16T13:46:25.564110+0000","last_fullsized":"2023-10-16T13:46:25.564110+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:08:14.564787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":9,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.581979+0000","last_change":"2023-10-16T13:46:25.579532+0000","last_active":"2023-10-16T13:46:25.581979+0000","last_peered":"2023-10-16T13:46:25.581979+0000","last_clean":"2023-10-16T13:46:25.581979+0000","last_became_active":"2023-10-16T13:46:25.579042+0000","last_became_peered":"2023-10-16T13:46:25.579042+0000","last_unstale":"2023-10-16T13:46:25.581979+0000","last_undegraded":"2023-10-16T13:46:25.581979+0000","last_fullsized":"2023-10-16T13:46:25.581979+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:01:57.893289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1a","version":"224'1","reported_seq":12,"reported_epoch":224,"state":"active+clean","last_fresh":"2023-10-16T13:46:25.596111+0000","last_change":"2023-10-16T13:46:24.344199+0000","last_active":"2023-10-16T13:46:25.596111+0000","last_peered":"2023-10-16T13:46:25.596111+0000","last_clean":"2023-10-16T13:46:25.596111+0000","last_became_active":"2023-10-16T13:46:24.343947+0000","last_became_peered":"2023-10-16T13:46:24.343947+0000","last_unstale":"2023-10-16T13:46:25.596111+0000","last_undegraded":"2023-10-16T13:46:25.596111+0000","last_fullsized":"2023-10-16T13:46:25.596111+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:17:17.355917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"214'348","reported_seq":1201,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-16T13:46:24.338304+0000","last_change":"2023-10-16T12:10:34.746820+0000","last_active":"2023-10-16T13:46:24.338304+0000","last_peered":"2023-10-16T13:46:24.338304+0000","last_clean":"2023-10-16T13:46:24.338304+0000","last_became_active":"2023-10-16T12:10:34.745574+0000","last_became_peered":"2023-10-16T12:10:34.745574+0000","last_unstale":"2023-10-16T13:46:24.338304+0000","last_undegraded":"2023-10-16T13:46:24.338304+0000","last_fullsized":"2023-10-16T13:46:24.338304+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_clean_scrub_stamp":"2023-10-16T12:10:33.577111+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:14.461986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":28,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":27,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":348,"ondisk_log_size":348,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575986,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56740,"kb_used_data":3344,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315100,"statfs":{"total":96636764160,"available":96578662400,"internally_reserved":0,"allocated":3424256,"data_stored":2038361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54655435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:45:48 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.61499999999999999,"15min":0.63800000000000001},"min":{"1min":0.35499999999999998,"5min":0.34899999999999998,"15min":0.33000000000000002},"max":{"1min":0.81899999999999995,"5min":0.87,"15min":0.96199999999999997},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.61799999999999999,"15min":0.63300000000000001},"min":{"1min":0.39200000000000002,"5min":0.38900000000000001,"15min":0.379},"max":{"1min":0.83899999999999997,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.53000000000000003}]},{"osd":1,"last update":"Mon Oct 16 13:45:48 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.61399999999999999,"15min":0.64400000000000002},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.79000000000000004,"5min":0.84599999999999997,"15min":0.95899999999999996},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.621,"5min":0.63100000000000001,"15min":0.63800000000000001},"min":{"1min":0.48199999999999998,"5min":0.375,"15min":0.375},"max":{"1min":0.77000000000000002,"5min":0.79100000000000004,"15min":1.022},"last":0.67400000000000004}]},{"osd":2,"last update":"Mon Oct 16 13:45:48 2023","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.69399999999999995,"15min":0.69499999999999995},"min":{"1min":0.48999999999999999,"5min":0.436,"15min":0.436},"max":{"1min":0.83499999999999996,"5min":0.83899999999999997,"15min":0.93500000000000005},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.69599999999999995,"15min":0.70199999999999996},"min":{"1min":0.51000000000000001,"5min":0.47599999999999998,"15min":0.39400000000000002},"max":{"1min":0.871,"5min":0.92800000000000005,"15min":0.93799999999999994},"last":0.70599999999999996}]},{"osd":3,"last update":"Mon Oct 16 13:45:48 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.65000000000000002,"15min":0.66200000000000003},"min":{"1min":0.45200000000000001,"5min":0.438,"15min":0.438},"max":{"1min":0.81499999999999995,"5min":0.89200000000000002,"15min":0.91300000000000003},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.64600000000000002,"15min":0.66300000000000003},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.80400000000000005,"5min":1.24,"15min":1.24},"last":0.623}]},{"osd":4,"last update":"Mon Oct 16 13:45:48 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.51300000000000001,"15min":0.52400000000000002},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.72699999999999998,"5min":0.745,"15min":0.86099999999999999},"last":0.433},{"interface":"front","average":{"1min":0.623,"5min":0.61399999999999999,"15min":0.61899999999999999},"min":{"1min":0.40699999999999997,"5min":0.36799999999999999,"15min":0.33100000000000002},"max":{"1min":0.77700000000000002,"5min":0.90400000000000003,"15min":0.92900000000000005},"last":0.57299999999999995}]},{"osd":5,"last update":"Mon Oct 16 13:45:48 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.53100000000000003,"15min":0.54400000000000004},"min":{"1min":0.41699999999999998,"5min":0.33400000000000002,"15min":0.32500000000000001},"max":{"1min":0.71599999999999997,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.57299999999999995,"15min":0.58799999999999997},"min":{"1min":0.41199999999999998,"5min":0.38400000000000001,"15min":0.35099999999999998},"max":{"1min":0.80100000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.54900000000000004}]},{"osd":6,"last update":"Mon Oct 16 13:45:48 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.59999999999999998,"15min":0.61499999999999999},"min":{"1min":0.47399999999999998,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.79500000000000004,"5min":0.874,"15min":0.95899999999999996},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.61299999999999999,"15min":0.62},"min":{"1min":0.41099999999999998,"5min":0.378,"15min":0.35199999999999998},"max":{"1min":0.81299999999999994,"5min":0.95799999999999996,"15min":0.97499999999999998},"last":0.63900000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575984,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42240,"kb_used_data":2736,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329600,"statfs":{"total":96636764160,"available":96593510400,"internally_reserved":0,"allocated":2801664,"data_stored":1439088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:45:36 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.67200000000000004,"15min":0.67400000000000004},"min":{"1min":0.39900000000000002,"5min":0.375,"15min":0.28699999999999998},"max":{"1min":0.82299999999999995,"5min":0.89400000000000002,"15min":0.93600000000000005},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.42399999999999999,"5min":0.41999999999999998,"15min":0.36299999999999999},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.93799999999999994},"last":0.59099999999999997}]},{"osd":1,"last update":"Mon Oct 16 13:45:36 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.61699999999999999,"15min":0.629},"min":{"1min":0.379,"5min":0.36599999999999999,"15min":0.33100000000000002},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.90900000000000003},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.61099999999999999,"15min":0.621},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.38300000000000001},"max":{"1min":0.77000000000000002,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.65600000000000003}]},{"osd":2,"last update":"Mon Oct 16 13:45:36 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.628,"15min":0.63500000000000001},"min":{"1min":0.45600000000000002,"5min":0.39900000000000002,"15min":0.378},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.61799999999999999,"15min":0.628},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.77600000000000002,"5min":0.89000000000000001,"15min":0.91000000000000003},"last":0.46800000000000003}]},{"osd":3,"last update":"Mon Oct 16 13:45:36 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.65100000000000002,"15min":0.66500000000000004},"min":{"1min":0.45700000000000002,"5min":0.39900000000000002,"15min":0.38200000000000001},"max":{"1min":0.80000000000000004,"5min":0.92400000000000004,"15min":0.95599999999999996},"last":0.622},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.71299999999999997,"15min":0.71999999999999997},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.42299999999999999},"max":{"1min":0.94999999999999996,"5min":0.96099999999999997,"15min":0.96899999999999997},"last":0.71599999999999997}]},{"osd":4,"last update":"Mon Oct 16 13:45:36 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.51500000000000001,"15min":0.52500000000000002},"min":{"1min":0.36899999999999999,"5min":0.32600000000000001,"15min":0.315},"max":{"1min":0.67100000000000004,"5min":0.85599999999999998,"15min":0.95799999999999996},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.623,"15min":0.61899999999999999},"min":{"1min":0.42799999999999999,"5min":0.35499999999999998,"15min":0.32900000000000001},"max":{"1min":0.84199999999999997,"5min":0.98799999999999999,"15min":1.0309999999999999},"last":0.73099999999999998}]},{"osd":5,"last update":"Mon Oct 16 13:45:36 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.59299999999999997,"15min":0.58899999999999997},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.73999999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.66800000000000004,"15min":0.65900000000000003},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.30599999999999999},"max":{"1min":0.84499999999999997,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.75}]},{"osd":7,"last update":"Mon Oct 16 13:45:36 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.61199999999999999,"15min":0.60399999999999998},"min":{"1min":0.375,"5min":0.375,"15min":0.34300000000000003},"max":{"1min":0.81499999999999995,"5min":1.081,"15min":1.081},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.62,"5min":0.65500000000000003,"15min":0.64800000000000002},"min":{"1min":0.45300000000000001,"5min":0.38300000000000001,"15min":0.375},"max":{"1min":0.79700000000000004,"5min":1.099,"15min":1.099},"last":0.79600000000000004}]}]},{"osd":1,"up_from":13,"seq":55834575984,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47824,"kb_used_data":2748,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324016,"statfs":{"total":96636764160,"available":96587792384,"internally_reserved":0,"allocated":2813952,"data_stored":1446311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":46135753},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:45:54 2023","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.40100000000000002,"15min":0.40999999999999998},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.71999999999999997},"last":0.373},{"interface":"front","average":{"1min":0.497,"5min":0.496,"15min":0.499},"min":{"1min":0.34599999999999997,"5min":0.253,"15min":0.253},"max":{"1min":0.68000000000000005,"5min":0.70999999999999996,"15min":0.77200000000000002},"last":0.32000000000000001}]},{"osd":2,"last update":"Mon Oct 16 13:45:54 2023","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.45500000000000002,"15min":0.45800000000000002},"min":{"1min":0.28000000000000003,"5min":0.254,"15min":0.254},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.48199999999999998,"15min":0.49299999999999999},"min":{"1min":0.315,"5min":0.315,"15min":0.30199999999999999},"max":{"1min":0.64200000000000002,"5min":0.66900000000000004,"15min":0.71399999999999997},"last":0.46899999999999997}]},{"osd":3,"last update":"Mon Oct 16 13:45:54 2023","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.438,"15min":0.43099999999999999},"min":{"1min":0.33900000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.57399999999999995,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.47299999999999998,"15min":0.47899999999999998},"min":{"1min":0.26700000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.64800000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.35499999999999998}]},{"osd":4,"last update":"Mon Oct 16 13:45:54 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.66000000000000003,"15min":0.66700000000000004},"min":{"1min":0.52600000000000002,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.86099999999999999,"5min":0.95899999999999996,"15min":1.0069999999999999},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.66200000000000003,"15min":0.67200000000000004},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.747,"5min":0.91300000000000003,"15min":0.997},"last":0.42899999999999999}]},{"osd":5,"last update":"Mon Oct 16 13:45:54 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.69799999999999995,"15min":0.71399999999999997},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.92300000000000004,"5min":0.97499999999999998,"15min":1.0760000000000001},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.73199999999999998,"15min":0.73399999999999999},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.46200000000000002},"max":{"1min":0.90000000000000002,"5min":1.0469999999999999,"15min":1.0660000000000001},"last":0.40999999999999998}]},{"osd":6,"last update":"Mon Oct 16 13:45:54 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.69099999999999995,"15min":0.68500000000000005},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.80100000000000005,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.70799999999999996,"15min":0.71299999999999997},"min":{"1min":0.499,"5min":0.499,"15min":0.47799999999999998},"max":{"1min":0.88100000000000001,"5min":1.03,"15min":1.083},"last":0.60199999999999998}]},{"osd":7,"last update":"Mon Oct 16 13:45:54 2023","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.73899999999999999,"15min":0.745},"min":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.48899999999999999},"max":{"1min":0.91400000000000003,"5min":1.0629999999999999,"15min":1.0660000000000001},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.751,"15min":0.751},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.92700000000000005,"5min":1.042,"15min":1.0580000000000001},"last":0.54100000000000004}]}]},{"osd":0,"up_from":13,"seq":55834575985,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30340,"kb_used_data":3316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341500,"statfs":{"total":96636764160,"available":96605696000,"internally_reserved":0,"allocated":3395584,"data_stored":2029456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 13:45:42 2023","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.40899999999999997,"15min":0.42399999999999999},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.58699999999999997,"5min":0.65800000000000003,"15min":0.71399999999999997},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.42599999999999999,"15min":0.442},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.25900000000000001},"max":{"1min":0.58599999999999997,"5min":0.64600000000000002,"15min":0.82199999999999995},"last":0.29799999999999999}]},{"osd":2,"last update":"Mon Oct 16 13:45:42 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44500000000000001,"15min":0.45800000000000002},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.75700000000000001},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.46800000000000003,"15min":0.48699999999999999},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.83299999999999996},"last":0.34100000000000003}]},{"osd":3,"last update":"Mon Oct 16 13:45:42 2023","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.43099999999999999,"15min":0.45400000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.61799999999999999,"5min":0.63400000000000001,"15min":0.78600000000000003},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.51600000000000001,"15min":0.52500000000000002},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.70399999999999996,"5min":0.82999999999999996,"15min":0.86399999999999999},"last":0.317}]},{"osd":4,"last update":"Mon Oct 16 13:45:42 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.66500000000000004,"15min":0.68600000000000005},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.96999999999999997,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.66300000000000003,"15min":0.69299999999999995},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":1.0049999999999999},"last":0.499}]},{"osd":5,"last update":"Mon Oct 16 13:45:42 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.66700000000000004,"15min":0.69499999999999995},"min":{"1min":0.44500000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.95399999999999996,"5min":0.95999999999999996,"15min":1.0249999999999999},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.70599999999999996,"15min":0.72999999999999998},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.081,"5min":1.081,"15min":1.081},"last":0.56399999999999995}]},{"osd":6,"last update":"Mon Oct 16 13:45:42 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.71599999999999997,"15min":0.73499999999999999},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.034,"5min":1.103,"15min":1.103},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.70199999999999996,"15min":0.72499999999999998},"min":{"1min":0.42999999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.91000000000000003,"5min":1.0409999999999999,"15min":1.05},"last":0.51000000000000001}]},{"osd":7,"last update":"Mon Oct 16 13:45:42 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.71899999999999997,"15min":0.746},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.002,"5min":1.002,"15min":1.056},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.72099999999999997,"15min":0.73699999999999999},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.99199999999999999,"5min":1.087,"15min":1.087},"last":0.55200000000000005}]}]},{"osd":2,"up_from":13,"seq":55834575985,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51968,"kb_used_data":2736,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319872,"statfs":{"total":96636764160,"available":96583548928,"internally_reserved":0,"allocated":2801664,"data_stored":1439088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:45:36 2023","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.42499999999999999,"15min":0.41299999999999998},"min":{"1min":0.26300000000000001,"5min":0.251,"15min":0.24099999999999999},"max":{"1min":0.52200000000000002,"5min":0.71699999999999997,"15min":0.77500000000000002},"last":0.255},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.48599999999999999,"15min":0.48299999999999998},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.63500000000000001,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.35199999999999998}]},{"osd":1,"last update":"Mon Oct 16 13:45:36 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.51500000000000001,"15min":0.50600000000000001},"min":{"1min":0.313,"5min":0.311,"15min":0.311},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.81399999999999995},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.5,"15min":0.47999999999999998},"min":{"1min":0.31900000000000001,"5min":0.29899999999999999,"15min":0.27700000000000002},"max":{"1min":0.57799999999999996,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.29099999999999998}]},{"osd":3,"last update":"Mon Oct 16 13:45:36 2023","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.44900000000000001,"15min":0.44500000000000001},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.60299999999999998,"5min":0.70099999999999996,"15min":0.86699999999999999},"last":0.314},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.56100000000000005,"15min":0.55100000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.309},"max":{"1min":0.74099999999999999,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.27400000000000002}]},{"osd":4,"last update":"Mon Oct 16 13:45:36 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.74099999999999999,"15min":0.72399999999999998},"min":{"1min":0.51600000000000001,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.89400000000000002,"5min":1.3600000000000001,"15min":1.3600000000000001},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.745,"15min":0.72799999999999998},"min":{"1min":0.502,"5min":0.45700000000000002,"15min":0.44900000000000001},"max":{"1min":0.91600000000000004,"5min":1.403,"15min":1.403},"last":0.624}]},{"osd":5,"last update":"Mon Oct 16 13:45:36 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.70599999999999996,"15min":0.69699999999999995},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.441},"max":{"1min":0.79700000000000004,"5min":1.486,"15min":1.486},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.73099999999999998,"15min":0.70799999999999996},"min":{"1min":0.44800000000000001,"5min":0.437,"15min":0.437},"max":{"1min":0.83099999999999996,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":0.51600000000000001}]},{"osd":6,"last update":"Mon Oct 16 13:45:36 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.69399999999999995,"15min":0.67400000000000004},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.98899999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.497},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.74099999999999999,"15min":0.73099999999999998},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.83699999999999997,"5min":1.4379999999999999,"15min":1.4379999999999999},"last":0.56200000000000006}]},{"osd":7,"last update":"Mon Oct 16 13:45:36 2023","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.75700000000000001,"15min":0.72899999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.0509999999999999,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.75,"15min":0.74199999999999999},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.98099999999999998,"5min":1.377,"15min":1.377},"last":0.57999999999999996}]}]},{"osd":3,"up_from":13,"seq":55834575984,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51988,"kb_used_data":2760,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319852,"statfs":{"total":96636764160,"available":96583528448,"internally_reserved":0,"allocated":2826240,"data_stored":1446437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50395592},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:45:39 2023","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.40699999999999997,"15min":0.41399999999999998},"min":{"1min":0.27600000000000002,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.52900000000000003,"5min":0.64200000000000002,"15min":0.72899999999999998},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.42499999999999999,"15min":0.43099999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.54700000000000004,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.35399999999999998}]},{"osd":1,"last update":"Mon Oct 16 13:45:39 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.436,"15min":0.438},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.76300000000000001},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.46999999999999997,"15min":0.48299999999999998},"min":{"1min":0.34300000000000003,"5min":0.253,"15min":0.253},"max":{"1min":0.68700000000000006,"5min":0.71499999999999997,"15min":0.77800000000000002},"last":0.27700000000000002}]},{"osd":2,"last update":"Mon Oct 16 13:45:39 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.49199999999999999,"15min":0.498},"min":{"1min":0.318,"5min":0.316,"15min":0.316},"max":{"1min":0.65600000000000003,"5min":1.167,"15min":1.167},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.51900000000000002,"15min":0.52900000000000003},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.73299999999999998,"5min":0.76500000000000001,"15min":0.78700000000000003},"last":0.38}]},{"osd":4,"last update":"Mon Oct 16 13:45:39 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.66000000000000003,"15min":0.67100000000000004},"min":{"1min":0.434,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.98499999999999999,"5min":1.137,"15min":1.137},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.69499999999999995,"15min":0.69699999999999995},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.0589999999999999,"5min":1.2909999999999999,"15min":1.2909999999999999},"last":0.47899999999999998}]},{"osd":5,"last update":"Mon Oct 16 13:45:39 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.69899999999999995,"15min":0.71299999999999997},"min":{"1min":0.46100000000000002,"5min":0.42899999999999999,"15min":0.374},"max":{"1min":0.89400000000000002,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.71599999999999997,"15min":0.70499999999999996},"min":{"1min":0.47099999999999997,"5min":0.432,"15min":0.432},"max":{"1min":1.0549999999999999,"5min":1.173,"15min":1.173},"last":0.60099999999999998}]},{"osd":6,"last update":"Mon Oct 16 13:45:39 2023","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.747,"15min":0.72499999999999998},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.41799999999999998},"max":{"1min":0.94499999999999995,"5min":5.0999999999999996,"15min":5.0999999999999996},"last":0.443},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.72899999999999998,"15min":0.73699999999999999},"min":{"1min":0.48599999999999999,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.0309999999999999,"5min":1.2290000000000001,"15min":1.2290000000000001},"last":0.49299999999999999}]},{"osd":7,"last update":"Mon Oct 16 13:45:39 2023","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.73399999999999999,"15min":0.73899999999999999},"min":{"1min":0.56599999999999995,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.92300000000000004,"5min":1.2569999999999999,"15min":1.2569999999999999},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.75,"15min":0.748},"min":{"1min":0.48999999999999999,"5min":0.47799999999999998,"15min":0.43099999999999999},"max":{"1min":1.014,"5min":1.262,"15min":1.262},"last":0.51000000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575983,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51968,"kb_used_data":2736,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319872,"statfs":{"total":96636764160,"available":96583548928,"internally_reserved":0,"allocated":2801664,"data_stored":1439088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:45:36 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.61599999999999999,"15min":0.625},"min":{"1min":0.45000000000000001,"5min":0.36299999999999999,"15min":0.33800000000000002},"max":{"1min":0.80600000000000005,"5min":0.83599999999999997,"15min":0.92700000000000005},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.628,"15min":0.63300000000000001},"min":{"1min":0.436,"5min":0.41499999999999998,"15min":0.35499999999999998},"max":{"1min":0.84099999999999997,"5min":0.878,"15min":0.89500000000000002},"last":0.84099999999999997}]},{"osd":1,"last update":"Mon Oct 16 13:45:36 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.64100000000000001,"15min":0.65900000000000003},"min":{"1min":0.48899999999999999,"5min":0.38400000000000001,"15min":0.375},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.877},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.63800000000000001,"15min":0.63700000000000001},"min":{"1min":0.41199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.78000000000000003,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.81100000000000005}]},{"osd":2,"last update":"Mon Oct 16 13:45:36 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.65800000000000003,"15min":0.65300000000000002},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.39600000000000002},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.90600000000000003},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.66200000000000003,"15min":0.66100000000000003},"min":{"1min":0.48599999999999999,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.85999999999999999,"5min":0.89700000000000002,"15min":0.92600000000000005},"last":0.746}]},{"osd":3,"last update":"Mon Oct 16 13:45:36 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.64100000000000001,"15min":0.64900000000000002},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.79100000000000004,"5min":0.86599999999999999,"15min":0.99199999999999999},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.68500000000000005,"15min":0.68300000000000005},"min":{"1min":0.54900000000000004,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.84699999999999998,"5min":0.90700000000000003,"15min":0.93899999999999995},"last":0.76400000000000001}]},{"osd":5,"last update":"Mon Oct 16 13:45:36 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.54400000000000004,"15min":0.52800000000000002},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.30599999999999999},"max":{"1min":0.76500000000000001,"5min":0.81299999999999994,"15min":0.91700000000000004},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.623,"5min":0.57699999999999996,"15min":0.56899999999999995},"min":{"1min":0.39000000000000001,"5min":0.33800000000000002,"15min":0.33600000000000002},"max":{"1min":0.80500000000000005,"5min":0.82299999999999995,"15min":0.94899999999999995},"last":0.625}]},{"osd":6,"last update":"Mon Oct 16 13:45:36 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.60899999999999999,"15min":0.59399999999999997},"min":{"1min":0.38900000000000001,"5min":0.34499999999999997,"15min":0.33200000000000002},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.92700000000000005},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.69599999999999995,"15min":0.68899999999999995},"min":{"1min":0.52800000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.95299999999999996},"last":0.86299999999999999}]},{"osd":7,"last update":"Mon Oct 16 13:45:36 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.64500000000000002,"15min":0.63100000000000001},"min":{"1min":0.44500000000000001,"5min":0.39900000000000002,"15min":0.35799999999999998},"max":{"1min":0.86899999999999999,"5min":0.97499999999999998,"15min":0.98599999999999999},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.69399999999999995,"15min":0.68100000000000005},"min":{"1min":0.53800000000000003,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.91200000000000003,"5min":0.91700000000000004,"15min":1.026},"last":0.82499999999999996}]}]},{"osd":5,"up_from":13,"seq":55834575984,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51968,"kb_used_data":2736,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319872,"statfs":{"total":96636764160,"available":96583548928,"internally_reserved":0,"allocated":2801664,"data_stored":1439088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:46:24 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.61399999999999999,"15min":0.61499999999999999},"min":{"1min":0.314,"5min":0.314,"15min":0.26800000000000002},"max":{"1min":0.878,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63,"15min":0.64200000000000002},"min":{"1min":0.38700000000000001,"5min":0.35899999999999999,"15min":0.29699999999999999},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.91200000000000003}]},{"osd":1,"last update":"Mon Oct 16 13:46:24 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.66000000000000003,"15min":0.66200000000000003},"min":{"1min":0.44600000000000001,"5min":0.34300000000000003,"15min":0.29499999999999998},"max":{"1min":1,"5min":1,"15min":1},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.63800000000000001,"15min":0.64000000000000001},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.57099999999999995}]},{"osd":2,"last update":"Mon Oct 16 13:46:24 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.67000000000000004,"15min":0.65700000000000003},"min":{"1min":0.41699999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.91000000000000003,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.44},{"interface":"front","average":{"1min":0.629,"5min":0.65700000000000003,"15min":0.64900000000000002},"min":{"1min":0.42599999999999999,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.83899999999999997,"5min":0.871,"15min":0.93500000000000005},"last":0.73999999999999999}]},{"osd":3,"last update":"Mon Oct 16 13:46:24 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.67400000000000004,"15min":0.66400000000000003},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.623},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.69899999999999995,"15min":0.69499999999999995},"min":{"1min":0.41399999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.4199999999999999,"5min":1.4199999999999999,"15min":1.4199999999999999},"last":0.67100000000000004}]},{"osd":4,"last update":"Mon Oct 16 13:46:24 2023","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.54400000000000004,"15min":0.53300000000000003},"min":{"1min":0.34100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.82499999999999996,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.59299999999999997,"15min":0.58899999999999997},"min":{"1min":0.34300000000000003,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.85399999999999998,"5min":0.89000000000000001,"15min":0.93400000000000005},"last":0.70799999999999996}]},{"osd":6,"last update":"Mon Oct 16 13:46:24 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.58899999999999997,"15min":0.57999999999999996},"min":{"1min":0.38700000000000001,"5min":0.315,"15min":0.315},"max":{"1min":0.877,"5min":0.88900000000000001,"15min":0.93200000000000005},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.61899999999999999},"min":{"1min":0.372,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89600000000000002},"last":0.64600000000000002}]},{"osd":7,"last update":"Mon Oct 16 13:46:24 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.64800000000000002,"15min":0.63300000000000001},"min":{"1min":0.378,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.69699999999999995,"15min":0.68400000000000005},"min":{"1min":0.46200000000000002,"5min":0.375,"15min":0.375},"max":{"1min":0.94699999999999995,"5min":0.98299999999999998,"15min":1.0229999999999999},"last":0.76200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T13:46:30.146 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-16T13:46:30.146 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-16T13:46:30.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.310+0000 7ff57fbb8700 1 Processor -- start 2023-10-16T13:46:30.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.310+0000 7ff57fbb8700 1 -- start start 2023-10-16T13:46:30.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.311+0000 7ff57fbb8700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff57815b2c0 0x7ff578161340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:30.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.311+0000 7ff57fbb8700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff57806e570 0x7ff57815e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:30.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.311+0000 7ff57fbb8700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff578161880 con 0x7ff57806a7e0 2023-10-16T13:46:30.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.311+0000 7ff57fbb8700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff57815b6a0 con 0x7ff57815b2c0 2023-10-16T13:46:30.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.311+0000 7ff57fbb8700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff578161b20 con 0x7ff57806e570 2023-10-16T13:46:30.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.311+0000 7ff57d954700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff57815b2c0 0x7ff578161340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:30.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.311+0000 7ff57d954700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff57815b2c0 0x7ff578161340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53234/0 (socket says 172.21.15.16:53234) 2023-10-16T13:46:30.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.311+0000 7ff57d954700 1 -- 172.21.15.16:0/4136305100 learned_addr learned my addr 172.21.15.16:0/4136305100 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:30.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.312+0000 7ff57d153700 1 --2- 172.21.15.16:0/4136305100 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff57806e570 0x7ff57815e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:30.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.312+0000 7ff57d954700 1 -- 172.21.15.16:0/4136305100 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff57806e570 msgr2=0x7ff57815e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:30.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.312+0000 7ff57d954700 1 --2- 172.21.15.16:0/4136305100 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff57806e570 0x7ff57815e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:30.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.312+0000 7ff57d954700 1 -- 172.21.15.16:0/4136305100 >> v1:172.21.15.49:6789/0 conn(0x7ff57806a7e0 legacy=0x7ff578071a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:30.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.312+0000 7ff57d954700 1 -- 172.21.15.16:0/4136305100 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff578161e00 con 0x7ff57815b2c0 2023-10-16T13:46:30.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.312+0000 7ff57d153700 1 --2- 172.21.15.16:0/4136305100 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff57806e570 0x7ff57815e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:46:30.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.312+0000 7ff57d954700 1 --2- 172.21.15.16:0/4136305100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff57815b2c0 0x7ff578161340 secure :-1 s=READY pgs=2470 cs=0 l=1 rev1=1 crypto rx=0x7ff5680098f0 tx=0x7ff568009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=126495d449bf6172 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:30.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.312+0000 7ff57c952700 1 -- 172.21.15.16:0/4136305100 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff568004230 con 0x7ff57815b2c0 2023-10-16T13:46:30.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.312+0000 7ff57c952700 1 -- 172.21.15.16:0/4136305100 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff568004390 con 0x7ff57815b2c0 2023-10-16T13:46:30.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.312+0000 7ff57c952700 1 -- 172.21.15.16:0/4136305100 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5680036a0 con 0x7ff57815b2c0 2023-10-16T13:46:30.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.313+0000 7ff57fbb8700 1 -- 172.21.15.16:0/4136305100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff57815b2c0 msgr2=0x7ff578161340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:30.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.313+0000 7ff57fbb8700 1 --2- 172.21.15.16:0/4136305100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff57815b2c0 0x7ff578161340 secure :-1 s=READY pgs=2470 cs=0 l=1 rev1=1 crypto rx=0x7ff5680098f0 tx=0x7ff568009c00 comp rx=0 tx=0).stop 2023-10-16T13:46:30.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.313+0000 7ff57fbb8700 1 -- 172.21.15.16:0/4136305100 shutdown_connections 2023-10-16T13:46:30.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.313+0000 7ff57fbb8700 1 --2- 172.21.15.16:0/4136305100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff57815b2c0 0x7ff578161340 unknown :-1 s=CLOSED pgs=2470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:30.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.313+0000 7ff57fbb8700 1 --2- 172.21.15.16:0/4136305100 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff57806e570 0x7ff57815e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:30.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.313+0000 7ff57fbb8700 1 -- 172.21.15.16:0/4136305100 >> 172.21.15.16:0/4136305100 conn(0x7ff5780b1f40 msgr2=0x7ff5780b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:30.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.313+0000 7ff57fbb8700 1 -- 172.21.15.16:0/4136305100 shutdown_connections 2023-10-16T13:46:30.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.313+0000 7ff57fbb8700 1 -- 172.21.15.16:0/4136305100 wait complete. 2023-10-16T13:46:30.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.314+0000 7ff57fbb8700 1 Processor -- start 2023-10-16T13:46:30.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.314+0000 7ff57fbb8700 1 -- start start 2023-10-16T13:46:30.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.315+0000 7ff57fbb8700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff57806a7e0 0x7ff57815d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:30.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.315+0000 7ff57fbb8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff57806e570 0x7ff5781f2c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:30.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.315+0000 7ff57fbb8700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff57815b2c0 0x7ff5781f3190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:30.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.315+0000 7ff57fbb8700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff578149de0 con 0x7ff57806e570 2023-10-16T13:46:30.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.315+0000 7ff57fbb8700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5780ca7c0 con 0x7ff57815b2c0 2023-10-16T13:46:30.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.315+0000 7ff57fbb8700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5780ca330 con 0x7ff57806a7e0 2023-10-16T13:46:30.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.315+0000 7ff57d153700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff57806e570 0x7ff5781f2c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:30.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.315+0000 7ff57d153700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff57806e570 0x7ff5781f2c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53242/0 (socket says 172.21.15.16:53242) 2023-10-16T13:46:30.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.315+0000 7ff57d153700 1 -- 172.21.15.16:0/1059185917 learned_addr learned my addr 172.21.15.16:0/1059185917 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:30.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.315+0000 7ff57d954700 1 --2- 172.21.15.16:0/1059185917 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff57806a7e0 0x7ff57815d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:30.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.315+0000 7ff57e155700 1 --2- 172.21.15.16:0/1059185917 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff57815b2c0 0x7ff5781f3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:30.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.315+0000 7ff57d153700 1 -- 172.21.15.16:0/1059185917 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff57806a7e0 msgr2=0x7ff57815d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:30.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.315+0000 7ff57d153700 1 --2- 172.21.15.16:0/1059185917 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff57806a7e0 0x7ff57815d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:30.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.315+0000 7ff57d153700 1 -- 172.21.15.16:0/1059185917 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff57815b2c0 msgr2=0x7ff5781f3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:30.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.315+0000 7ff57d153700 1 --2- 172.21.15.16:0/1059185917 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff57815b2c0 0x7ff5781f3190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:30.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.315+0000 7ff57d153700 1 -- 172.21.15.16:0/1059185917 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff568009550 con 0x7ff57806e570 2023-10-16T13:46:30.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.315+0000 7ff57d954700 1 --2- 172.21.15.16:0/1059185917 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff57806a7e0 0x7ff57815d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:46:30.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.315+0000 7ff57d153700 1 --2- 172.21.15.16:0/1059185917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff57806e570 0x7ff5781f2c50 secure :-1 s=READY pgs=2471 cs=0 l=1 rev1=1 crypto rx=0x7ff57400c780 tx=0x7ff57400cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:30.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.315+0000 7ff56e7fc700 1 -- 172.21.15.16:0/1059185917 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff574011800 con 0x7ff57806e570 2023-10-16T13:46:30.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.316+0000 7ff56e7fc700 1 -- 172.21.15.16:0/1059185917 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff574011960 con 0x7ff57806e570 2023-10-16T13:46:30.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.316+0000 7ff57fbb8700 1 -- 172.21.15.16:0/1059185917 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5781609a0 con 0x7ff57806e570 2023-10-16T13:46:30.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.316+0000 7ff56e7fc700 1 -- 172.21.15.16:0/1059185917 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff574011cd0 con 0x7ff57806e570 2023-10-16T13:46:30.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.316+0000 7ff57fbb8700 1 -- 172.21.15.16:0/1059185917 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5780c7b80 con 0x7ff57806e570 2023-10-16T13:46:30.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.316+0000 7ff57fbb8700 1 -- 172.21.15.16:0/1059185917 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff57804ebb0 con 0x7ff57806e570 2023-10-16T13:46:30.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.317+0000 7ff56e7fc700 1 -- 172.21.15.16:0/1059185917 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 138) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff574021350 con 0x7ff57806e570 2023-10-16T13:46:30.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.317+0000 7ff56e7fc700 1 --2- 172.21.15.16:0/1059185917 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7ff55c06b020 0x7ff55c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:30.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.317+0000 7ff56e7fc700 1 -- 172.21.15.16:0/1059185917 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(225..225 src has 1..225) v4 ==== 6157+0+0 (secure 0 0 0) 0x7ff5740899b0 con 0x7ff57806e570 2023-10-16T13:46:30.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.317+0000 7ff57d954700 1 --2- 172.21.15.16:0/1059185917 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7ff55c06b020 0x7ff55c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:30.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.318+0000 7ff57d954700 1 --2- 172.21.15.16:0/1059185917 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7ff55c06b020 0x7ff55c06d4e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff568009130 tx=0x7ff568008030 comp rx=0 tx=0).ready entity=mgr.11190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:30.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.322+0000 7ff56e7fc700 1 -- 172.21.15.16:0/1059185917 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff574054690 con 0x7ff57806e570 2023-10-16T13:46:30.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.608+0000 7ff57fbb8700 1 -- 172.21.15.16:0/1059185917 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7ff5780ca470 con 0x7ff57806e570 2023-10-16T13:46:30.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.610+0000 7ff56e7fc700 1 -- 172.21.15.16:0/1059185917 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v56) v1 ==== 190+0+147 (secure 0 0 0) 0x7ff57401e0b0 con 0x7ff57806e570 2023-10-16T13:46:30.613 INFO:teuthology.orchestra.run.smithi016.stdout:[client.0] 2023-10-16T13:46:30.613 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAPKC1l2l6PFxAA9QyXtBAHdGAXpwH0JfknuQ== 2023-10-16T13:46:30.613 INFO:teuthology.orchestra.run.smithi016.stdout: caps mds = "allow" 2023-10-16T13:46:30.614 INFO:teuthology.orchestra.run.smithi016.stdout: caps mon = "allow r" 2023-10-16T13:46:30.614 INFO:teuthology.orchestra.run.smithi016.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-16T13:46:30.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.612+0000 7ff57fbb8700 1 -- 172.21.15.16:0/1059185917 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7ff55c06b020 msgr2=0x7ff55c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:30.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.612+0000 7ff57fbb8700 1 --2- 172.21.15.16:0/1059185917 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7ff55c06b020 0x7ff55c06d4e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff568009130 tx=0x7ff568008030 comp rx=0 tx=0).stop 2023-10-16T13:46:30.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.612+0000 7ff57fbb8700 1 -- 172.21.15.16:0/1059185917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff57806e570 msgr2=0x7ff5781f2c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:30.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.612+0000 7ff57fbb8700 1 --2- 172.21.15.16:0/1059185917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff57806e570 0x7ff5781f2c50 secure :-1 s=READY pgs=2471 cs=0 l=1 rev1=1 crypto rx=0x7ff57400c780 tx=0x7ff57400cb40 comp rx=0 tx=0).stop 2023-10-16T13:46:30.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.612+0000 7ff57fbb8700 1 -- 172.21.15.16:0/1059185917 shutdown_connections 2023-10-16T13:46:30.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.612+0000 7ff57fbb8700 1 --2- 172.21.15.16:0/1059185917 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff57815b2c0 0x7ff5781f3190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:30.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.612+0000 7ff57fbb8700 1 --2- 172.21.15.16:0/1059185917 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff57806a7e0 0x7ff57815d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:30.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.612+0000 7ff57fbb8700 1 --2- 172.21.15.16:0/1059185917 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7ff55c06b020 0x7ff55c06d4e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:30.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.612+0000 7ff57fbb8700 1 --2- 172.21.15.16:0/1059185917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff57806e570 0x7ff5781f2c50 unknown :-1 s=CLOSED pgs=2471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:30.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.612+0000 7ff57fbb8700 1 -- 172.21.15.16:0/1059185917 >> 172.21.15.16:0/1059185917 conn(0x7ff5780b1f40 msgr2=0x7ff5780cb6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:30.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.613+0000 7ff57fbb8700 1 -- 172.21.15.16:0/1059185917 shutdown_connections 2023-10-16T13:46:30.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.613+0000 7ff57fbb8700 1 -- 172.21.15.16:0/1059185917 wait complete. 2023-10-16T13:46:30.618 INFO:teuthology.orchestra.run.smithi016.stderr:updated caps for client.0 2023-10-16T13:46:30.627 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:46:30.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.750+0000 7f0e95c0d700 1 Processor -- start 2023-10-16T13:46:30.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.750+0000 7f0e95c0d700 1 -- start start 2023-10-16T13:46:30.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.751+0000 7f0e95c0d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0e900cf4a0 0x7f0e900726e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:30.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.751+0000 7f0e95c0d700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f0e900b5a00 0x7f0e9015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:30.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.751+0000 7f0e95c0d700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e9015bc80 con 0x7f0e900cf880 2023-10-16T13:46:30.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.751+0000 7f0e95c0d700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e900b5e70 con 0x7f0e900cf4a0 2023-10-16T13:46:30.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.751+0000 7f0e95c0d700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e90072c20 con 0x7f0e900b5a00 2023-10-16T13:46:30.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.751+0000 7f0e8f7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0e900cf4a0 0x7f0e900726e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:30.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.751+0000 7f0e8f7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0e900cf4a0 0x7f0e900726e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53248/0 (socket says 172.21.15.16:53248) 2023-10-16T13:46:30.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.751+0000 7f0e8f7fe700 1 -- 172.21.15.16:0/3415505215 learned_addr learned my addr 172.21.15.16:0/3415505215 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:30.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.751+0000 7f0e8f7fe700 1 -- 172.21.15.16:0/3415505215 >> v2:172.21.15.49:3300/0 conn(0x7f0e900b5a00 msgr2=0x7f0e9015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:30.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.751+0000 7f0e8ffff700 1 --2- 172.21.15.16:0/3415505215 >> v2:172.21.15.49:3300/0 conn(0x7f0e900b5a00 0x7f0e9015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:30.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.751+0000 7f0e8f7fe700 1 --2- 172.21.15.16:0/3415505215 >> v2:172.21.15.49:3300/0 conn(0x7f0e900b5a00 0x7f0e9015a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:30.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.751+0000 7f0e8f7fe700 1 -- 172.21.15.16:0/3415505215 >> v1:172.21.15.49:6789/0 conn(0x7f0e900cf880 legacy=0x7f0e900b52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:30.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.751+0000 7f0e8f7fe700 1 -- 172.21.15.16:0/3415505215 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e9015b070 con 0x7f0e900cf4a0 2023-10-16T13:46:30.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.751+0000 7f0e8f7fe700 1 --2- 172.21.15.16:0/3415505215 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e900cf4a0 0x7f0e900726e0 secure :-1 s=READY pgs=2472 cs=0 l=1 rev1=1 crypto rx=0x7f0e900cdeb0 tx=0x7f0e80009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea944c86320a737d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:30.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.752+0000 7f0e8e7fc700 1 -- 172.21.15.16:0/3415505215 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e80004230 con 0x7f0e900cf4a0 2023-10-16T13:46:30.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.752+0000 7f0e8e7fc700 1 -- 172.21.15.16:0/3415505215 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e80004390 con 0x7f0e900cf4a0 2023-10-16T13:46:30.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.752+0000 7f0e8e7fc700 1 -- 172.21.15.16:0/3415505215 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e800036a0 con 0x7f0e900cf4a0 2023-10-16T13:46:30.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.752+0000 7f0e95c0d700 1 -- 172.21.15.16:0/3415505215 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e900cf4a0 msgr2=0x7f0e900726e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:30.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.752+0000 7f0e95c0d700 1 --2- 172.21.15.16:0/3415505215 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e900cf4a0 0x7f0e900726e0 secure :-1 s=READY pgs=2472 cs=0 l=1 rev1=1 crypto rx=0x7f0e900cdeb0 tx=0x7f0e80009bd0 comp rx=0 tx=0).stop 2023-10-16T13:46:30.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.752+0000 7f0e95c0d700 1 -- 172.21.15.16:0/3415505215 shutdown_connections 2023-10-16T13:46:30.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.752+0000 7f0e95c0d700 1 --2- 172.21.15.16:0/3415505215 >> v2:172.21.15.49:3300/0 conn(0x7f0e900b5a00 0x7f0e9015a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:30.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.752+0000 7f0e95c0d700 1 --2- 172.21.15.16:0/3415505215 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e900cf4a0 0x7f0e900726e0 unknown :-1 s=CLOSED pgs=2472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:30.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.752+0000 7f0e95c0d700 1 -- 172.21.15.16:0/3415505215 >> 172.21.15.16:0/3415505215 conn(0x7f0e900b1fd0 msgr2=0x7f0e900b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:30.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.752+0000 7f0e95c0d700 1 -- 172.21.15.16:0/3415505215 shutdown_connections 2023-10-16T13:46:30.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.752+0000 7f0e95c0d700 1 -- 172.21.15.16:0/3415505215 wait complete. 2023-10-16T13:46:30.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.753+0000 7f0e95c0d700 1 Processor -- start 2023-10-16T13:46:30.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.754+0000 7f0e95c0d700 1 -- start start 2023-10-16T13:46:30.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.754+0000 7f0e95c0d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e900b5a00 0x7f0e901473a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:30.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.754+0000 7f0e95c0d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0e900cf4a0 0x7f0e901e7060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:30.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.754+0000 7f0e95c0d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0e900cf880 0x7f0e900c9910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:30.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.754+0000 7f0e95c0d700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e900c68f0 con 0x7f0e900b5a00 2023-10-16T13:46:30.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.754+0000 7f0e95c0d700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e901478e0 con 0x7f0e900cf4a0 2023-10-16T13:46:30.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.754+0000 7f0e95c0d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e900c5c70 con 0x7f0e900cf880 2023-10-16T13:46:30.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.754+0000 7f0e8f7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e900b5a00 0x7f0e901473a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:30.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.754+0000 7f0e8f7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e900b5a00 0x7f0e901473a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53254/0 (socket says 172.21.15.16:53254) 2023-10-16T13:46:30.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.754+0000 7f0e8f7fe700 1 -- 172.21.15.16:0/2499897250 learned_addr learned my addr 172.21.15.16:0/2499897250 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:30.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.755+0000 7f0e8effd700 1 --2- 172.21.15.16:0/2499897250 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0e900cf4a0 0x7f0e901e7060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:30.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.755+0000 7f0e8ffff700 1 --2- 172.21.15.16:0/2499897250 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0e900cf880 0x7f0e900c9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:30.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.755+0000 7f0e8f7fe700 1 -- 172.21.15.16:0/2499897250 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0e900cf880 msgr2=0x7f0e900c9910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:30.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.755+0000 7f0e8f7fe700 1 --2- 172.21.15.16:0/2499897250 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0e900cf880 0x7f0e900c9910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:30.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.755+0000 7f0e8f7fe700 1 -- 172.21.15.16:0/2499897250 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0e900cf4a0 msgr2=0x7f0e901e7060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:30.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.755+0000 7f0e8f7fe700 1 --2- 172.21.15.16:0/2499897250 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0e900cf4a0 0x7f0e901e7060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:30.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.755+0000 7f0e8f7fe700 1 -- 172.21.15.16:0/2499897250 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e80009550 con 0x7f0e900b5a00 2023-10-16T13:46:30.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.755+0000 7f0e8f7fe700 1 --2- 172.21.15.16:0/2499897250 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e900b5a00 0x7f0e901473a0 secure :-1 s=READY pgs=2473 cs=0 l=1 rev1=1 crypto rx=0x7f0e900cdfc0 tx=0x7f0e80004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:30.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.755+0000 7f0e77fff700 1 -- 172.21.15.16:0/2499897250 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e80003d80 con 0x7f0e900b5a00 2023-10-16T13:46:30.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.755+0000 7f0e77fff700 1 -- 172.21.15.16:0/2499897250 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e8001f070 con 0x7f0e900b5a00 2023-10-16T13:46:30.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.755+0000 7f0e95c0d700 1 -- 172.21.15.16:0/2499897250 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e900c5f50 con 0x7f0e900b5a00 2023-10-16T13:46:30.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.755+0000 7f0e77fff700 1 -- 172.21.15.16:0/2499897250 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e80011870 con 0x7f0e900b5a00 2023-10-16T13:46:30.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.755+0000 7f0e95c0d700 1 -- 172.21.15.16:0/2499897250 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e900c6510 con 0x7f0e900b5a00 2023-10-16T13:46:30.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.756+0000 7f0e95c0d700 1 -- 172.21.15.16:0/2499897250 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e9004ebb0 con 0x7f0e900b5a00 2023-10-16T13:46:30.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.757+0000 7f0e77fff700 1 -- 172.21.15.16:0/2499897250 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 138) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0e80021030 con 0x7f0e900b5a00 2023-10-16T13:46:30.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.757+0000 7f0e77fff700 1 --2- 172.21.15.16:0/2499897250 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f0e7c06b020 0x7f0e7c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:30.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.757+0000 7f0e77fff700 1 -- 172.21.15.16:0/2499897250 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(225..225 src has 1..225) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f0e8008af80 con 0x7f0e900b5a00 2023-10-16T13:46:30.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.757+0000 7f0e8effd700 1 --2- 172.21.15.16:0/2499897250 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f0e7c06b020 0x7f0e7c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:30.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.758+0000 7f0e8effd700 1 --2- 172.21.15.16:0/2499897250 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f0e7c06b020 0x7f0e7c06d4e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0e90159880 tx=0x7f0e84008040 comp rx=0 tx=0).ready entity=mgr.11190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:30.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:30.762+0000 7f0e77fff700 1 -- 172.21.15.16:0/2499897250 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0e80055df0 con 0x7f0e900b5a00 2023-10-16T13:46:31.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.012+0000 7f0e95c0d700 1 -- 172.21.15.16:0/2499897250 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0e900afad0 con 0x7f0e900b5a00 2023-10-16T13:46:31.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.013+0000 7f0e77fff700 1 -- 172.21.15.16:0/2499897250 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) v1 ==== 95+0+4929 (secure 0 0 0) 0x7f0e80059410 con 0x7f0e900b5a00 2023-10-16T13:46:31.016 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:46:31.016 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":113,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10516,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/4065699432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":4065699432},{"type":"v1","addr":"172.21.15.49:6834","nonce":4065699432}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":108},{"gid":11415,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/1373048456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1373048456},{"type":"v1","addr":"172.21.15.16:6836","nonce":1373048456}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":108},{"gid":11418,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3860406853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":3860406853},{"type":"v1","addr":"172.21.15.16:6837","nonce":3860406853}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":107}],"filesystems":[{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:46:24.755313+0000","modified":"2023-10-16T13:46:28.011141+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11430},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11430":{"gid":11430,"name":"b","rank":0,"incarnation":109,"state":"up:active","state_seq":3,"addr":"172.21.15.49:6835/2905760119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":2905760119},{"type":"v1","addr":"172.21.15.49:6835","nonce":2905760119}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13}]} 2023-10-16T13:46:31.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.015+0000 7f0e95c0d700 1 -- 172.21.15.16:0/2499897250 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f0e7c06b020 msgr2=0x7f0e7c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:31.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.015+0000 7f0e95c0d700 1 --2- 172.21.15.16:0/2499897250 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f0e7c06b020 0x7f0e7c06d4e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0e90159880 tx=0x7f0e84008040 comp rx=0 tx=0).stop 2023-10-16T13:46:31.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.015+0000 7f0e95c0d700 1 -- 172.21.15.16:0/2499897250 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e900b5a00 msgr2=0x7f0e901473a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:31.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.015+0000 7f0e95c0d700 1 --2- 172.21.15.16:0/2499897250 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e900b5a00 0x7f0e901473a0 secure :-1 s=READY pgs=2473 cs=0 l=1 rev1=1 crypto rx=0x7f0e900cdfc0 tx=0x7f0e80004750 comp rx=0 tx=0).stop 2023-10-16T13:46:31.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.015+0000 7f0e95c0d700 1 -- 172.21.15.16:0/2499897250 shutdown_connections 2023-10-16T13:46:31.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.015+0000 7f0e95c0d700 1 --2- 172.21.15.16:0/2499897250 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0e900cf880 0x7f0e900c9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:31.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.015+0000 7f0e95c0d700 1 --2- 172.21.15.16:0/2499897250 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f0e7c06b020 0x7f0e7c06d4e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:31.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.015+0000 7f0e95c0d700 1 --2- 172.21.15.16:0/2499897250 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e900b5a00 0x7f0e901473a0 unknown :-1 s=CLOSED pgs=2473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:31.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.015+0000 7f0e95c0d700 1 --2- 172.21.15.16:0/2499897250 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0e900cf4a0 0x7f0e901e7060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:31.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.015+0000 7f0e95c0d700 1 -- 172.21.15.16:0/2499897250 >> 172.21.15.16:0/2499897250 conn(0x7f0e900b1fd0 msgr2=0x7f0e900cdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:31.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.015+0000 7f0e95c0d700 1 -- 172.21.15.16:0/2499897250 shutdown_connections 2023-10-16T13:46:31.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.015+0000 7f0e95c0d700 1 -- 172.21.15.16:0/2499897250 wait complete. 2023-10-16T13:46:31.021 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 113 2023-10-16T13:46:31.432 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 113, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T13:46:24.755313+0000', 'modified': '2023-10-16T13:46:28.011141+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 11430}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_11430': {'gid': 11430, 'name': 'b', 'rank': 0, 'incarnation': 109, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.49:6835/2905760119', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6833', 'nonce': 2905760119}, {'type': 'v1', 'addr': '172.21.15.49:6835', 'nonce': 2905760119}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-16T13:46:31.432 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-16T13:46:31.433 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-10-16T13:46:31.553 INFO:tasks.ceph.mds.b.smithi049.stderr:2023-10-16T13:46:31.550+0000 7fa1c43aa700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T13:46:31.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.565+0000 7fb954e2f700 1 Processor -- start 2023-10-16T13:46:31.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.565+0000 7fb954e2f700 1 -- start start 2023-10-16T13:46:31.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.566+0000 7fb954e2f700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fb95015b9f0 0x7fb9500b29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:31.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.566+0000 7fb954e2f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb95006d940 0x7fb95006ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:31.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.566+0000 7fb954e2f700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9500c3490 con 0x7fb9500ceb70 2023-10-16T13:46:31.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.566+0000 7fb954e2f700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb95006e300 con 0x7fb95006d940 2023-10-16T13:46:31.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.566+0000 7fb954e2f700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9500cf730 con 0x7fb95015b9f0 2023-10-16T13:46:31.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.566+0000 7fb94dd9b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb95006d940 0x7fb95006ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:31.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.566+0000 7fb94ed9d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb9500ceb70 0x7fb9500cf020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42540/0 (socket says 172.21.15.16:42540) 2023-10-16T13:46:31.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.566+0000 7fb94ed9d700 1 -- 172.21.15.16:0/3304713749 learned_addr learned my addr 172.21.15.16:0/3304713749 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:46:31.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.567+0000 7fb94e59c700 1 --2- 172.21.15.16:0/3304713749 >> v2:172.21.15.49:3300/0 conn(0x7fb95015b9f0 0x7fb9500b29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:31.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.567+0000 7fb94d59a700 1 -- 172.21.15.16:0/3304713749 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 810294702 0 0) 0x7fb9500c3490 con 0x7fb9500ceb70 2023-10-16T13:46:31.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.567+0000 7fb94d59a700 1 -- 172.21.15.16:0/3304713749 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb93c0035b0 con 0x7fb9500ceb70 2023-10-16T13:46:31.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.567+0000 7fb94dd9b700 1 -- 172.21.15.16:0/3304713749 >> v2:172.21.15.49:3300/0 conn(0x7fb95015b9f0 msgr2=0x7fb9500b29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:31.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.567+0000 7fb94dd9b700 1 --2- 172.21.15.16:0/3304713749 >> v2:172.21.15.49:3300/0 conn(0x7fb95015b9f0 0x7fb9500b29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:31.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.567+0000 7fb94dd9b700 1 -- 172.21.15.16:0/3304713749 >> v1:172.21.15.16:6789/0 conn(0x7fb9500ceb70 legacy=0x7fb9500cf020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:31.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.567+0000 7fb94dd9b700 1 -- 172.21.15.16:0/3304713749 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9500cdad0 con 0x7fb95006d940 2023-10-16T13:46:31.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.567+0000 7fb94dd9b700 1 --2- 172.21.15.16:0/3304713749 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb95006d940 0x7fb95006ddc0 secure :-1 s=READY pgs=2475 cs=0 l=1 rev1=1 crypto rx=0x7fb9500af5e0 tx=0x7fb938009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a94fabefb605fc97 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:31.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.567+0000 7fb94d59a700 1 -- 172.21.15.16:0/3304713749 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb938011830 con 0x7fb95006d940 2023-10-16T13:46:31.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.567+0000 7fb94d59a700 1 -- 172.21.15.16:0/3304713749 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb938011990 con 0x7fb95006d940 2023-10-16T13:46:31.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.567+0000 7fb94d59a700 1 -- 172.21.15.16:0/3304713749 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb938011d20 con 0x7fb95006d940 2023-10-16T13:46:31.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.568+0000 7fb954e2f700 1 -- 172.21.15.16:0/3304713749 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb95006d940 msgr2=0x7fb95006ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:31.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.568+0000 7fb954e2f700 1 --2- 172.21.15.16:0/3304713749 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb95006d940 0x7fb95006ddc0 secure :-1 s=READY pgs=2475 cs=0 l=1 rev1=1 crypto rx=0x7fb9500af5e0 tx=0x7fb938009b20 comp rx=0 tx=0).stop 2023-10-16T13:46:31.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.568+0000 7fb954e2f700 1 -- 172.21.15.16:0/3304713749 shutdown_connections 2023-10-16T13:46:31.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.568+0000 7fb954e2f700 1 --2- 172.21.15.16:0/3304713749 >> v2:172.21.15.49:3300/0 conn(0x7fb95015b9f0 0x7fb9500b29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:31.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.568+0000 7fb954e2f700 1 --2- 172.21.15.16:0/3304713749 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb95006d940 0x7fb95006ddc0 unknown :-1 s=CLOSED pgs=2475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:31.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.568+0000 7fb954e2f700 1 -- 172.21.15.16:0/3304713749 >> 172.21.15.16:0/3304713749 conn(0x7fb9500cc5b0 msgr2=0x7fb9500cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:31.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.568+0000 7fb954e2f700 1 -- 172.21.15.16:0/3304713749 shutdown_connections 2023-10-16T13:46:31.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.568+0000 7fb954e2f700 1 -- 172.21.15.16:0/3304713749 wait complete. 2023-10-16T13:46:31.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.570+0000 7fb954e2f700 1 Processor -- start 2023-10-16T13:46:31.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.570+0000 7fb954e2f700 1 -- start start 2023-10-16T13:46:31.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.571+0000 7fb954e2f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb95006d940 0x7fb9500b0f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:31.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.571+0000 7fb954e2f700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb9500ceb70 0x7fb95014d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:31.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.571+0000 7fb954e2f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb9500c6230 0x7fb950158e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:31.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.571+0000 7fb954e2f700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9500b14a0 con 0x7fb95006d940 2023-10-16T13:46:31.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.571+0000 7fb954e2f700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb95015d130 con 0x7fb9500ceb70 2023-10-16T13:46:31.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.571+0000 7fb954e2f700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9501f46c0 con 0x7fb9500c6230 2023-10-16T13:46:31.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.571+0000 7fb94e59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb95006d940 0x7fb9500b0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:31.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.571+0000 7fb94e59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb95006d940 0x7fb9500b0f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53268/0 (socket says 172.21.15.16:53268) 2023-10-16T13:46:31.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.571+0000 7fb94e59c700 1 -- 172.21.15.16:0/3835972547 learned_addr learned my addr 172.21.15.16:0/3835972547 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:31.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.571+0000 7fb94dd9b700 1 --2- 172.21.15.16:0/3835972547 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb9500ceb70 0x7fb95014d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:31.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.571+0000 7fb94ed9d700 1 --2- 172.21.15.16:0/3835972547 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb9500c6230 0x7fb950158e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:31.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.571+0000 7fb94e59c700 1 -- 172.21.15.16:0/3835972547 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb9500c6230 msgr2=0x7fb950158e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:31.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.571+0000 7fb94e59c700 1 --2- 172.21.15.16:0/3835972547 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb9500c6230 0x7fb950158e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:31.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.571+0000 7fb94e59c700 1 -- 172.21.15.16:0/3835972547 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb9500ceb70 msgr2=0x7fb95014d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:31.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.571+0000 7fb94e59c700 1 --2- 172.21.15.16:0/3835972547 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb9500ceb70 0x7fb95014d130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:31.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.571+0000 7fb94e59c700 1 -- 172.21.15.16:0/3835972547 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9380094a0 con 0x7fb95006d940 2023-10-16T13:46:31.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.571+0000 7fb94ed9d700 1 --2- 172.21.15.16:0/3835972547 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb9500c6230 0x7fb950158e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:46:31.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.571+0000 7fb94e59c700 1 --2- 172.21.15.16:0/3835972547 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb95006d940 0x7fb9500b0f60 secure :-1 s=READY pgs=2476 cs=0 l=1 rev1=1 crypto rx=0x7fb94000e810 tx=0x7fb94000ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:31.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.571+0000 7fb946ffd700 1 -- 172.21.15.16:0/3835972547 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb940013610 con 0x7fb95006d940 2023-10-16T13:46:31.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.571+0000 7fb946ffd700 1 -- 172.21.15.16:0/3835972547 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb940013770 con 0x7fb95006d940 2023-10-16T13:46:31.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.571+0000 7fb954e2f700 1 -- 172.21.15.16:0/3835972547 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9501f49a0 con 0x7fb95006d940 2023-10-16T13:46:31.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.571+0000 7fb946ffd700 1 -- 172.21.15.16:0/3835972547 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb940013b20 con 0x7fb95006d940 2023-10-16T13:46:31.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.571+0000 7fb954e2f700 1 -- 172.21.15.16:0/3835972547 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9501f4f60 con 0x7fb95006d940 2023-10-16T13:46:31.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.573+0000 7fb946ffd700 1 -- 172.21.15.16:0/3835972547 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 138) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb940013c80 con 0x7fb95006d940 2023-10-16T13:46:31.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.573+0000 7fb946ffd700 1 --2- 172.21.15.16:0/3835972547 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fb93006b020 0x7fb93006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:31.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.573+0000 7fb946ffd700 1 -- 172.21.15.16:0/3835972547 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(225..225 src has 1..225) v4 ==== 6157+0+0 (secure 0 0 0) 0x7fb940089720 con 0x7fb95006d940 2023-10-16T13:46:31.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.573+0000 7fb94dd9b700 1 --2- 172.21.15.16:0/3835972547 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fb93006b020 0x7fb93006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:31.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.574+0000 7fb94dd9b700 1 --2- 172.21.15.16:0/3835972547 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fb93006b020 0x7fb93006d4e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb9500af5e0 tx=0x7fb93801c040 comp rx=0 tx=0).ready entity=mgr.11190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:31.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.575+0000 7fb961019b80 1 Processor -- start 2023-10-16T13:46:31.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.575+0000 7fb961019b80 1 -- start start 2023-10-16T13:46:31.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.576+0000 7fb961019b80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x561110071b40 0x561110071f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:31.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.576+0000 7fb961019b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561110098f30 0x561110071280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:31.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.576+0000 7fb961019b80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56111006cf00 0x561110098300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:31.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.576+0000 7fb961019b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x56110ffe5c20 con 0x561110098f30 2023-10-16T13:46:31.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.576+0000 7fb961019b80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x56110ffcac10 con 0x56111006cf00 2023-10-16T13:46:31.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.576+0000 7fb961019b80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x56110ffc8530 con 0x561110071b40 2023-10-16T13:46:31.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.576+0000 7fb94dd9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561110098f30 0x561110071280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:31.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.576+0000 7fb94dd9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561110098f30 0x561110071280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53278/0 (socket says 172.21.15.16:53278) 2023-10-16T13:46:31.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.576+0000 7fb94dd9b700 1 -- 172.21.15.16:0/3962401182 learned_addr learned my addr 172.21.15.16:0/3962401182 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:31.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.576+0000 7fb94ed9d700 1 --2- 172.21.15.16:0/3962401182 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56111006cf00 0x561110098300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:31.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.576+0000 7fb94e59c700 1 --2- 172.21.15.16:0/3962401182 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x561110071b40 0x561110071f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:31.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.577+0000 7fb94dd9b700 1 -- 172.21.15.16:0/3962401182 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x561110071b40 msgr2=0x561110071f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:31.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.577+0000 7fb94dd9b700 1 --2- 172.21.15.16:0/3962401182 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x561110071b40 0x561110071f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:31.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.577+0000 7fb94dd9b700 1 -- 172.21.15.16:0/3962401182 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56111006cf00 msgr2=0x561110098300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:31.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.577+0000 7fb94dd9b700 1 --2- 172.21.15.16:0/3962401182 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56111006cf00 0x561110098300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:31.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.577+0000 7fb94dd9b700 1 -- 172.21.15.16:0/3962401182 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56110ff813a0 con 0x561110098f30 2023-10-16T13:46:31.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.577+0000 7fb94ed9d700 1 --2- 172.21.15.16:0/3962401182 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56111006cf00 0x561110098300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:46:31.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.577+0000 7fb94e59c700 1 --2- 172.21.15.16:0/3962401182 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x561110071b40 0x561110071f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:46:31.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.577+0000 7fb94dd9b700 1 --2- 172.21.15.16:0/3962401182 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561110098f30 0x561110071280 secure :-1 s=READY pgs=2477 cs=0 l=1 rev1=1 crypto rx=0x7fb9500725b0 tx=0x7fb938010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ada285de5ba2f8e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:31.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.577+0000 7fb92ffff700 1 -- 172.21.15.16:0/3962401182 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9380036a0 con 0x561110098f30 2023-10-16T13:46:31.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.577+0000 7fb92ffff700 1 -- 172.21.15.16:0/3962401182 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb938003800 con 0x561110098f30 2023-10-16T13:46:31.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.577+0000 7fb92ffff700 1 -- 172.21.15.16:0/3962401182 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb938003b70 con 0x561110098f30 2023-10-16T13:46:31.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.578+0000 7fb961019b80 1 -- 172.21.15.16:0/3962401182 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561110098f30 msgr2=0x561110071280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:31.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.578+0000 7fb961019b80 1 --2- 172.21.15.16:0/3962401182 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561110098f30 0x561110071280 secure :-1 s=READY pgs=2477 cs=0 l=1 rev1=1 crypto rx=0x7fb9500725b0 tx=0x7fb938010000 comp rx=0 tx=0).stop 2023-10-16T13:46:31.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.578+0000 7fb961019b80 1 -- 172.21.15.16:0/3962401182 shutdown_connections 2023-10-16T13:46:31.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.578+0000 7fb961019b80 1 --2- 172.21.15.16:0/3962401182 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56111006cf00 0x561110098300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:31.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.578+0000 7fb961019b80 1 --2- 172.21.15.16:0/3962401182 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x561110071b40 0x561110071f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:31.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.578+0000 7fb961019b80 1 --2- 172.21.15.16:0/3962401182 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561110098f30 0x561110071280 unknown :-1 s=CLOSED pgs=2477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:31.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.578+0000 7fb961019b80 1 -- 172.21.15.16:0/3962401182 >> 172.21.15.16:0/3962401182 conn(0x56110ff9ed80 msgr2=0x5611100b9ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:31.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.578+0000 7fb961019b80 1 -- 172.21.15.16:0/3962401182 shutdown_connections 2023-10-16T13:46:31.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.578+0000 7fb961019b80 1 -- 172.21.15.16:0/3962401182 wait complete. 2023-10-16T13:46:31.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.578+0000 7fb961019b80 1 -- start start 2023-10-16T13:46:31.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.579+0000 7fb961019b80 1 Processor -- start 2023-10-16T13:46:31.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.579+0000 7fb961019b80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56111006cf00 0x5611100b0750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:31.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.579+0000 7fb961019b80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x561110071b40 0x5611100919e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:31.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.579+0000 7fb961019b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561110098f30 0x5611100cba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:31.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.579+0000 7fb961019b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x56110ffc3e20 con 0x561110098f30 2023-10-16T13:46:31.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.580+0000 7fb961019b80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x56110ff606c0 con 0x56111006cf00 2023-10-16T13:46:31.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.580+0000 7fb961019b80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x56110ff60360 con 0x561110071b40 2023-10-16T13:46:31.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.580+0000 7fb961019b80 10 client.0 _ll_register_callbacks cb 0x56110fdbd7b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T13:46:31.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.580+0000 7fb94ed9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561110098f30 0x5611100cba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:31.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.580+0000 7fb94ed9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561110098f30 0x5611100cba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53286/0 (socket says 172.21.15.16:53286) 2023-10-16T13:46:31.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.580+0000 7fb94ed9d700 1 -- 172.21.15.16:0/981422923 learned_addr learned my addr 172.21.15.16:0/981422923 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:31.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.580+0000 7fb94e59c700 1 --2- 172.21.15.16:0/981422923 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56111006cf00 0x5611100b0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:31.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.580+0000 7fb94dd9b700 1 --2- 172.21.15.16:0/981422923 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x561110071b40 0x5611100919e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:31.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.580+0000 7fb94ed9d700 1 -- 172.21.15.16:0/981422923 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x561110071b40 msgr2=0x5611100919e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:31.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.580+0000 7fb94ed9d700 1 --2- 172.21.15.16:0/981422923 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x561110071b40 0x5611100919e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:31.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.580+0000 7fb94ed9d700 1 -- 172.21.15.16:0/981422923 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56111006cf00 msgr2=0x5611100b0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:31.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.580+0000 7fb94ed9d700 1 --2- 172.21.15.16:0/981422923 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56111006cf00 0x5611100b0750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:31.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.580+0000 7fb94ed9d700 1 -- 172.21.15.16:0/981422923 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9500cdad0 con 0x561110098f30 2023-10-16T13:46:31.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.580+0000 7fb94e59c700 1 --2- 172.21.15.16:0/981422923 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56111006cf00 0x5611100b0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:46:31.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.580+0000 7fb94ed9d700 1 --2- 172.21.15.16:0/981422923 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561110098f30 0x5611100cba90 secure :-1 s=READY pgs=2478 cs=0 l=1 rev1=1 crypto rx=0x7fb94800f7b0 tx=0x7fb94800fb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62cfd7b1eb75db68 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:31.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.580+0000 7fb92d7fa700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2023-10-16T13:46:31.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.580+0000 7fb92d7fa700 1 -- 172.21.15.16:0/981422923 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9480072e0 con 0x561110098f30 2023-10-16T13:46:31.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.580+0000 7fb92d7fa700 1 -- 172.21.15.16:0/981422923 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(225..225 src has 1..225) v4 ==== 6157+0+0 (secure 0 0 0) 0x7fb948007440 con 0x561110098f30 2023-10-16T13:46:31.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.580+0000 7fb961019b80 1 -- 172.21.15.16:0/981422923 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5611100645b0 con 0x561110098f30 2023-10-16T13:46:31.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.580+0000 7fb92d7fa700 1 -- 172.21.15.16:0/981422923 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=113) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb948001bb0 con 0x561110098f30 2023-10-16T13:46:31.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.580+0000 7fb961019b80 10 client.11568 fetch_fsmap learned FSMap version 113 2023-10-16T13:46:31.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.580+0000 7fb961019b80 1 -- 172.21.15.16:0/981422923 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=113,monmap=2+}) v3 -- 0x56110fdeed90 con 0x561110098f30 2023-10-16T13:46:31.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.581+0000 7fb92d7fa700 1 -- 172.21.15.16:0/981422923 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5611100645b0 con 0x561110098f30 2023-10-16T13:46:31.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.581+0000 7fb92d7fa700 1 -- 172.21.15.16:0/981422923 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 113) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fb948012070 con 0x561110098f30 2023-10-16T13:46:31.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.581+0000 7fb961019b80 10 client.11568 fetch_fsmap finished waiting for FSMap version 113 2023-10-16T13:46:31.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.581+0000 7fb961019b80 10 client.11568 resolve_mds: resolved name 'b' to daemon mds.b 2023-10-16T13:46:31.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.581+0000 7fb961019b80 20 client.11568 populate_metadata read hostname 'smithi016' 2023-10-16T13:46:31.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.581+0000 7fb961019b80 1 --2- 172.21.15.16:0/981422923 >> [v2:172.21.15.49:6833/2905760119,v1:172.21.15.49:6835/2905760119] conn(0x561110098400 0x5611100ca620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:31.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.581+0000 7fb961019b80 4 client.11568 mds_command: new command op to 11430 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T13:46:31.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.581+0000 7fb961019b80 1 -- 172.21.15.16:0/981422923 --> [v2:172.21.15.49:6833/2905760119,v1:172.21.15.49:6835/2905760119] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56110ffbce00 con 0x561110098400 2023-10-16T13:46:31.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.581+0000 7fb94e59c700 1 --2- 172.21.15.16:0/981422923 >> [v2:172.21.15.49:6833/2905760119,v1:172.21.15.49:6835/2905760119] conn(0x561110098400 0x5611100ca620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:31.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.581+0000 7fb92d7fa700 10 client.11568 ms_handle_connect on v2:172.21.15.49:6833/2905760119 2023-10-16T13:46:31.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.581+0000 7fb94e59c700 1 --2- 172.21.15.16:0/981422923 >> [v2:172.21.15.49:6833/2905760119,v1:172.21.15.49:6835/2905760119] conn(0x561110098400 0x5611100ca620 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=23ba8a17c896993d server_cookie=ce97e5737255d24a in_seq=0 out_seq=0 2023-10-16T13:46:31.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.582+0000 7fb92d7fa700 1 -- 172.21.15.16:0/981422923 <== mds.0 v2:172.21.15.49:6833/2905760119 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x56110ffbce00 con 0x561110098400 2023-10-16T13:46:31.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.582+0000 7fb92d7fa700 10 client.11568 handle_command_reply: tid=0 2023-10-16T13:46:31.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.582+0000 7fb961019b80 1 client.11568 shutdown 2023-10-16T13:46:31.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.582+0000 7fb961019b80 1 -- 172.21.15.16:0/981422923 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561110098f30 msgr2=0x5611100cba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:31.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.582+0000 7fb961019b80 1 --2- 172.21.15.16:0/981422923 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561110098f30 0x5611100cba90 secure :-1 s=READY pgs=2478 cs=0 l=1 rev1=1 crypto rx=0x7fb94800f7b0 tx=0x7fb94800fb70 comp rx=0 tx=0).stop 2023-10-16T13:46:31.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.583+0000 7fb961019b80 1 -- 172.21.15.16:0/981422923 shutdown_connections 2023-10-16T13:46:31.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.583+0000 7fb961019b80 1 --2- 172.21.15.16:0/981422923 >> [v2:172.21.15.49:6833/2905760119,v1:172.21.15.49:6835/2905760119] conn(0x561110098400 0x5611100ca620 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:31.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.583+0000 7fb961019b80 1 --2- 172.21.15.16:0/981422923 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561110098f30 0x5611100cba90 unknown :-1 s=CLOSED pgs=2478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:31.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.583+0000 7fb92d7fa700 0 client.11568 ms_handle_reset on v2:172.21.15.49:6833/2905760119 2023-10-16T13:46:31.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.583+0000 7fb961019b80 1 --2- 172.21.15.16:0/981422923 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56111006cf00 0x5611100b0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:31.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.583+0000 7fb961019b80 1 --2- 172.21.15.16:0/981422923 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x561110071b40 0x5611100919e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:31.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.583+0000 7fb961019b80 1 -- 172.21.15.16:0/981422923 >> 172.21.15.16:0/981422923 conn(0x56110ff9ed80 msgr2=0x56111006bea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:31.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.583+0000 7fb961019b80 1 -- 172.21.15.16:0/981422923 shutdown_connections 2023-10-16T13:46:31.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.583+0000 7fb961019b80 1 -- 172.21.15.16:0/981422923 wait complete. 2023-10-16T13:46:31.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.583+0000 7fb961019b80 20 client.11568 trim_cache size 0 max 16384 2023-10-16T13:46:31.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.601+0000 7fb961019b80 1 Processor -- start 2023-10-16T13:46:31.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.601+0000 7fb961019b80 1 -- start start 2023-10-16T13:46:31.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.602+0000 7fb961019b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56111006cf00 0x56111008fe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:31.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.602+0000 7fb961019b80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x561110071b40 0x561110090390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:31.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.602+0000 7fb961019b80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x561110098f30 0x5611100908d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:31.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.602+0000 7fb961019b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x56110fe448d0 con 0x56111006cf00 2023-10-16T13:46:31.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.602+0000 7fb961019b80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x56111006ccb0 con 0x561110071b40 2023-10-16T13:46:31.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.602+0000 7fb961019b80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x561110071f20 con 0x561110098f30 2023-10-16T13:46:31.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.602+0000 7fb94e59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56111006cf00 0x56111008fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:31.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.602+0000 7fb94e59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56111006cf00 0x56111008fe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53290/0 (socket says 172.21.15.16:53290) 2023-10-16T13:46:31.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.602+0000 7fb94e59c700 1 -- 172.21.15.16:0/3074757729 learned_addr learned my addr 172.21.15.16:0/3074757729 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:31.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.602+0000 7fb94dd9b700 1 --2- 172.21.15.16:0/3074757729 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x561110071b40 0x561110090390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:31.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.602+0000 7fb94ed9d700 1 --2- 172.21.15.16:0/3074757729 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x561110098f30 0x5611100908d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:31.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.602+0000 7fb94e59c700 1 -- 172.21.15.16:0/3074757729 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x561110098f30 msgr2=0x5611100908d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:31.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.602+0000 7fb94e59c700 1 --2- 172.21.15.16:0/3074757729 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x561110098f30 0x5611100908d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:31.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.602+0000 7fb94e59c700 1 -- 172.21.15.16:0/3074757729 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x561110071b40 msgr2=0x561110090390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:31.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.602+0000 7fb94e59c700 1 --2- 172.21.15.16:0/3074757729 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x561110071b40 0x561110090390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:31.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.602+0000 7fb94e59c700 1 -- 172.21.15.16:0/3074757729 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56111006dee0 con 0x56111006cf00 2023-10-16T13:46:31.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.602+0000 7fb94ed9d700 1 --2- 172.21.15.16:0/3074757729 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x561110098f30 0x5611100908d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:46:31.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.602+0000 7fb94e59c700 1 --2- 172.21.15.16:0/3074757729 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56111006cf00 0x56111008fe50 secure :-1 s=READY pgs=2479 cs=0 l=1 rev1=1 crypto rx=0x7fb9400545c0 tx=0x7fb94000e760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35ba7827f363edf9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:31.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.602+0000 7fb944ff9700 1 -- 172.21.15.16:0/3074757729 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9400580c0 con 0x56111006cf00 2023-10-16T13:46:31.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.602+0000 7fb944ff9700 1 -- 172.21.15.16:0/3074757729 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb94005c210 con 0x56111006cf00 2023-10-16T13:46:31.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.602+0000 7fb944ff9700 1 -- 172.21.15.16:0/3074757729 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9400603c0 con 0x56111006cf00 2023-10-16T13:46:31.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.603+0000 7fb961019b80 1 -- 172.21.15.16:0/3074757729 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56111006cf00 msgr2=0x56111008fe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:31.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.603+0000 7fb961019b80 1 --2- 172.21.15.16:0/3074757729 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56111006cf00 0x56111008fe50 secure :-1 s=READY pgs=2479 cs=0 l=1 rev1=1 crypto rx=0x7fb9400545c0 tx=0x7fb94000e760 comp rx=0 tx=0).stop 2023-10-16T13:46:31.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.603+0000 7fb961019b80 1 -- 172.21.15.16:0/3074757729 shutdown_connections 2023-10-16T13:46:31.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.603+0000 7fb961019b80 1 --2- 172.21.15.16:0/3074757729 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x561110098f30 0x5611100908d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:31.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.603+0000 7fb961019b80 1 --2- 172.21.15.16:0/3074757729 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56111006cf00 0x56111008fe50 unknown :-1 s=CLOSED pgs=2479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:31.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.603+0000 7fb961019b80 1 --2- 172.21.15.16:0/3074757729 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x561110071b40 0x561110090390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:31.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.603+0000 7fb961019b80 1 -- 172.21.15.16:0/3074757729 >> 172.21.15.16:0/3074757729 conn(0x56110ff9ed80 msgr2=0x56111008f240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:31.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.603+0000 7fb961019b80 1 -- 172.21.15.16:0/3074757729 shutdown_connections 2023-10-16T13:46:31.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.603+0000 7fb961019b80 1 -- 172.21.15.16:0/3074757729 wait complete. 2023-10-16T13:46:31.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.604+0000 7fb961019b80 1 -- start start 2023-10-16T13:46:31.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.604+0000 7fb961019b80 1 Processor -- start 2023-10-16T13:46:31.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.605+0000 7fb961019b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56111006cf00 0x561110117a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:31.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.605+0000 7fb961019b80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x561110071b40 0x56111006c180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:31.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.605+0000 7fb961019b80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x561110098f30 0x56111006c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:31.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.605+0000 7fb961019b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x56111009a730 con 0x56111006cf00 2023-10-16T13:46:31.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.605+0000 7fb961019b80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611100adeb0 con 0x561110071b40 2023-10-16T13:46:31.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.605+0000 7fb961019b80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x561110117fc0 con 0x561110098f30 2023-10-16T13:46:31.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.605+0000 7fb961019b80 10 client.0 _ll_register_callbacks cb 0x7fb9480020d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T13:46:31.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.605+0000 7fb94e59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56111006cf00 0x561110117a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:31.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.605+0000 7fb94e59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56111006cf00 0x561110117a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53296/0 (socket says 172.21.15.16:53296) 2023-10-16T13:46:31.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.605+0000 7fb94e59c700 1 -- 172.21.15.16:0/485111613 learned_addr learned my addr 172.21.15.16:0/485111613 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:31.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.605+0000 7fb94dd9b700 1 --2- 172.21.15.16:0/485111613 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x561110071b40 0x56111006c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:31.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.605+0000 7fb94ed9d700 1 --2- 172.21.15.16:0/485111613 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x561110098f30 0x56111006c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:31.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.605+0000 7fb94e59c700 1 -- 172.21.15.16:0/485111613 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x561110098f30 msgr2=0x56111006c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:31.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.605+0000 7fb94e59c700 1 --2- 172.21.15.16:0/485111613 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x561110098f30 0x56111006c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:31.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.605+0000 7fb94e59c700 1 -- 172.21.15.16:0/485111613 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x561110071b40 msgr2=0x56111006c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:31.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.605+0000 7fb94e59c700 1 --2- 172.21.15.16:0/485111613 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x561110071b40 0x56111006c180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:31.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.605+0000 7fb94e59c700 1 -- 172.21.15.16:0/485111613 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9501f4f60 con 0x56111006cf00 2023-10-16T13:46:31.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.605+0000 7fb94e59c700 1 --2- 172.21.15.16:0/485111613 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56111006cf00 0x561110117a80 secure :-1 s=READY pgs=2480 cs=0 l=1 rev1=1 crypto rx=0x7fb9400545c0 tx=0x7fb948002180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cba74e137b69366 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:31.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.605+0000 7fb92d7fa700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2023-10-16T13:46:31.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.605+0000 7fb92d7fa700 1 -- 172.21.15.16:0/485111613 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9400603c0 con 0x56111006cf00 2023-10-16T13:46:31.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.606+0000 7fb92d7fa700 1 -- 172.21.15.16:0/485111613 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(225..225 src has 1..225) v4 ==== 6157+0+0 (secure 0 0 0) 0x7fb9400580c0 con 0x56111006cf00 2023-10-16T13:46:31.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.606+0000 7fb961019b80 1 -- 172.21.15.16:0/485111613 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb938003800 con 0x56111006cf00 2023-10-16T13:46:31.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.606+0000 7fb92d7fa700 1 -- 172.21.15.16:0/485111613 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=113) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb94005e0b0 con 0x56111006cf00 2023-10-16T13:46:31.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.606+0000 7fb961019b80 10 client.11574 fetch_fsmap learned FSMap version 113 2023-10-16T13:46:31.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.606+0000 7fb961019b80 1 -- 172.21.15.16:0/485111613 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=113,monmap=2+}) v3 -- 0x56110fdd7c10 con 0x56111006cf00 2023-10-16T13:46:31.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.606+0000 7fb92d7fa700 1 -- 172.21.15.16:0/485111613 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb938003800 con 0x56111006cf00 2023-10-16T13:46:31.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.606+0000 7fb92d7fa700 1 -- 172.21.15.16:0/485111613 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 113) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fb94006fbe0 con 0x56111006cf00 2023-10-16T13:46:31.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.606+0000 7fb961019b80 10 client.11574 fetch_fsmap finished waiting for FSMap version 113 2023-10-16T13:46:31.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.606+0000 7fb961019b80 10 client.11574 resolve_mds: resolved name 'b' to daemon mds.b 2023-10-16T13:46:31.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.606+0000 7fb961019b80 20 client.11574 populate_metadata read hostname 'smithi016' 2023-10-16T13:46:31.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.606+0000 7fb961019b80 1 --2- 172.21.15.16:0/485111613 >> [v2:172.21.15.49:6833/2905760119,v1:172.21.15.49:6835/2905760119] conn(0x5611100b6760 0x5611100b6b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:31.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.606+0000 7fb961019b80 4 client.11574 mds_command: new command op to 11430 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T13:46:31.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.606+0000 7fb961019b80 1 -- 172.21.15.16:0/485111613 --> [v2:172.21.15.49:6833/2905760119,v1:172.21.15.49:6835/2905760119] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56110ffbcb20 con 0x5611100b6760 2023-10-16T13:46:31.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.606+0000 7fb94dd9b700 1 --2- 172.21.15.16:0/485111613 >> [v2:172.21.15.49:6833/2905760119,v1:172.21.15.49:6835/2905760119] conn(0x5611100b6760 0x5611100b6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:31.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.607+0000 7fb92d7fa700 10 client.11574 ms_handle_connect on v2:172.21.15.49:6833/2905760119 2023-10-16T13:46:31.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.607+0000 7fb94dd9b700 1 --2- 172.21.15.16:0/485111613 >> [v2:172.21.15.49:6833/2905760119,v1:172.21.15.49:6835/2905760119] conn(0x5611100b6760 0x5611100b6b40 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=533fbe7a2ee6d941 server_cookie=822c5960529046d1 in_seq=0 out_seq=0 2023-10-16T13:46:31.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.607+0000 7fb92d7fa700 1 -- 172.21.15.16:0/485111613 <== mds.0 v2:172.21.15.49:6833/2905760119 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+329 (crc 0 0 0) 0x56110ffbcb20 con 0x5611100b6760 2023-10-16T13:46:31.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.607+0000 7fb92d7fa700 10 client.11574 handle_command_reply: tid=0 2023-10-16T13:46:31.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.607+0000 7fb961019b80 1 client.11574 shutdown 2023-10-16T13:46:31.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.607+0000 7fb961019b80 1 -- 172.21.15.16:0/485111613 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56111006cf00 msgr2=0x561110117a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:31.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.607+0000 7fb961019b80 1 --2- 172.21.15.16:0/485111613 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56111006cf00 0x561110117a80 secure :-1 s=READY pgs=2480 cs=0 l=1 rev1=1 crypto rx=0x7fb9400545c0 tx=0x7fb948002180 comp rx=0 tx=0).stop 2023-10-16T13:46:31.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.607+0000 7fb961019b80 1 -- 172.21.15.16:0/485111613 shutdown_connections 2023-10-16T13:46:31.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.607+0000 7fb961019b80 1 --2- 172.21.15.16:0/485111613 >> [v2:172.21.15.49:6833/2905760119,v1:172.21.15.49:6835/2905760119] conn(0x5611100b6760 0x5611100b6b40 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:31.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.607+0000 7fb961019b80 1 --2- 172.21.15.16:0/485111613 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x561110098f30 0x56111006c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:31.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.607+0000 7fb961019b80 1 --2- 172.21.15.16:0/485111613 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56111006cf00 0x561110117a80 unknown :-1 s=CLOSED pgs=2480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:31.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.607+0000 7fb961019b80 1 --2- 172.21.15.16:0/485111613 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x561110071b40 0x56111006c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:31.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.607+0000 7fb92d7fa700 0 client.11574 ms_handle_reset on v2:172.21.15.49:6833/2905760119 2023-10-16T13:46:31.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.607+0000 7fb961019b80 1 -- 172.21.15.16:0/485111613 >> 172.21.15.16:0/485111613 conn(0x56110ff9ed80 msgr2=0x56111009ee30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:31.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.607+0000 7fb961019b80 1 -- 172.21.15.16:0/485111613 shutdown_connections 2023-10-16T13:46:31.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.607+0000 7fb961019b80 1 -- 172.21.15.16:0/485111613 wait complete. 2023-10-16T13:46:31.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.607+0000 7fb961019b80 20 client.11574 trim_cache size 0 max 16384 2023-10-16T13:46:31.629 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-10-16T13:46:31.629 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "5b28dfa1-b21f-474a-9b2e-9fb77af58254", 2023-10-16T13:46:31.629 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 0, 2023-10-16T13:46:31.629 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 11430, 2023-10-16T13:46:31.629 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2023-10-16T13:46:31.630 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2023-10-16T13:46:31.630 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2023-10-16T13:46:31.630 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 6.0263042880000004, 2023-10-16T13:46:31.630 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 113, 2023-10-16T13:46:31.630 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 225, 2023-10-16T13:46:31.631 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 225, 2023-10-16T13:46:31.631 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 11.931490988 2023-10-16T13:46:31.631 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-10-16T13:46:31.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.608+0000 7fb954e2f700 1 -- 172.21.15.16:0/3835972547 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fb93006b020 msgr2=0x7fb93006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:31.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.608+0000 7fb954e2f700 1 --2- 172.21.15.16:0/3835972547 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fb93006b020 0x7fb93006d4e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb9500af5e0 tx=0x7fb93801c040 comp rx=0 tx=0).stop 2023-10-16T13:46:31.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.608+0000 7fb954e2f700 1 -- 172.21.15.16:0/3835972547 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb95006d940 msgr2=0x7fb9500b0f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:31.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.608+0000 7fb954e2f700 1 --2- 172.21.15.16:0/3835972547 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb95006d940 0x7fb9500b0f60 secure :-1 s=READY pgs=2476 cs=0 l=1 rev1=1 crypto rx=0x7fb94000e810 tx=0x7fb94000ebd0 comp rx=0 tx=0).stop 2023-10-16T13:46:31.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.608+0000 7fb954e2f700 1 -- 172.21.15.16:0/3835972547 shutdown_connections 2023-10-16T13:46:31.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.608+0000 7fb954e2f700 1 --2- 172.21.15.16:0/3835972547 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb9500c6230 0x7fb950158e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:31.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.608+0000 7fb954e2f700 1 --2- 172.21.15.16:0/3835972547 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fb93006b020 0x7fb93006d4e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:31.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.608+0000 7fb954e2f700 1 --2- 172.21.15.16:0/3835972547 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb95006d940 0x7fb9500b0f60 unknown :-1 s=CLOSED pgs=2476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:31.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.608+0000 7fb954e2f700 1 --2- 172.21.15.16:0/3835972547 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb9500ceb70 0x7fb95014d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:31.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.608+0000 7fb954e2f700 1 -- 172.21.15.16:0/3835972547 >> 172.21.15.16:0/3835972547 conn(0x7fb9500cc5b0 msgr2=0x7fb9500cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:31.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.608+0000 7fb954e2f700 1 -- 172.21.15.16:0/3835972547 shutdown_connections 2023-10-16T13:46:31.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.608+0000 7fb954e2f700 1 -- 172.21.15.16:0/3835972547 wait complete. 2023-10-16T13:46:31.635 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T13:46:31.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.758+0000 7f35f7fcb700 1 Processor -- start 2023-10-16T13:46:31.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.758+0000 7f35f7fcb700 1 -- start start 2023-10-16T13:46:31.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.759+0000 7f35f7fcb700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f35f00cf880 0x7f35f00b52f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:31.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.759+0000 7f35f7fcb700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f35f00b5830 0x7f35f015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:31.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.759+0000 7f35f7fcb700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35f015bc80 con 0x7f35f00cf4a0 2023-10-16T13:46:31.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.759+0000 7f35f7fcb700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35f015b070 con 0x7f35f00b5830 2023-10-16T13:46:31.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.759+0000 7f35f7fcb700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35f015b350 con 0x7f35f00cf880 2023-10-16T13:46:31.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.759+0000 7f35f5d67700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f35f00cf4a0 0x7f35f00726e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42578/0 (socket says 172.21.15.16:42578) 2023-10-16T13:46:31.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.759+0000 7f35f5d67700 1 -- 172.21.15.16:0/490316513 learned_addr learned my addr 172.21.15.16:0/490316513 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:46:31.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.760+0000 7f35f6568700 1 --2- 172.21.15.16:0/490316513 >> v2:172.21.15.49:3300/0 conn(0x7f35f00b5830 0x7f35f015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:31.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.760+0000 7f35f5566700 1 --2- 172.21.15.16:0/490316513 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f35f00cf880 0x7f35f00b52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:31.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.760+0000 7f35f4d65700 1 -- 172.21.15.16:0/490316513 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2769829330 0 0) 0x7f35f015bc80 con 0x7f35f00cf4a0 2023-10-16T13:46:31.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.760+0000 7f35f4d65700 1 -- 172.21.15.16:0/490316513 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35dc0035b0 con 0x7f35f00cf4a0 2023-10-16T13:46:31.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.760+0000 7f35f6568700 1 -- 172.21.15.16:0/490316513 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f35f00cf880 msgr2=0x7f35f00b52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:31.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.760+0000 7f35f6568700 1 --2- 172.21.15.16:0/490316513 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f35f00cf880 0x7f35f00b52f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:31.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.760+0000 7f35f4d65700 1 -- 172.21.15.16:0/490316513 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 531447674 0 0) 0x7f35dc0035b0 con 0x7f35f00cf4a0 2023-10-16T13:46:31.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.760+0000 7f35f6568700 1 -- 172.21.15.16:0/490316513 >> v1:172.21.15.16:6789/0 conn(0x7f35f00cf4a0 legacy=0x7f35f00726e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:31.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.760+0000 7f35f6568700 1 -- 172.21.15.16:0/490316513 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35f015b630 con 0x7f35f00b5830 2023-10-16T13:46:31.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.760+0000 7f35f6568700 1 --2- 172.21.15.16:0/490316513 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f35f00b5830 0x7f35f015a090 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f35e8009d90 tx=0x7f35e800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cc66382e644f6b18 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:31.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.760+0000 7f35f4d65700 1 -- 172.21.15.16:0/490316513 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35e800e070 con 0x7f35f00b5830 2023-10-16T13:46:31.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.760+0000 7f35f4d65700 1 -- 172.21.15.16:0/490316513 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35e8003a20 con 0x7f35f00b5830 2023-10-16T13:46:31.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.760+0000 7f35f4d65700 1 -- 172.21.15.16:0/490316513 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35ec001670 con 0x7f35f00b5830 2023-10-16T13:46:31.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.761+0000 7f35f7fcb700 1 -- 172.21.15.16:0/490316513 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f35f00b5830 msgr2=0x7f35f015a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:31.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.761+0000 7f35f7fcb700 1 --2- 172.21.15.16:0/490316513 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f35f00b5830 0x7f35f015a090 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f35e8009d90 tx=0x7f35e800b330 comp rx=0 tx=0).stop 2023-10-16T13:46:31.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.761+0000 7f35f7fcb700 1 -- 172.21.15.16:0/490316513 shutdown_connections 2023-10-16T13:46:31.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.761+0000 7f35f7fcb700 1 --2- 172.21.15.16:0/490316513 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f35f00b5830 0x7f35f015a090 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:31.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.761+0000 7f35f7fcb700 1 --2- 172.21.15.16:0/490316513 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f35f00cf880 0x7f35f00b52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:31.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.761+0000 7f35f7fcb700 1 -- 172.21.15.16:0/490316513 >> 172.21.15.16:0/490316513 conn(0x7f35f00b1fd0 msgr2=0x7f35f00b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:31.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.761+0000 7f35f7fcb700 1 -- 172.21.15.16:0/490316513 shutdown_connections 2023-10-16T13:46:31.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.761+0000 7f35f7fcb700 1 -- 172.21.15.16:0/490316513 wait complete. 2023-10-16T13:46:31.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.762+0000 7f35f7fcb700 1 Processor -- start 2023-10-16T13:46:31.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.762+0000 7f35f7fcb700 1 -- start start 2023-10-16T13:46:31.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.763+0000 7f35f7fcb700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f35f00b5830 0x7f35f006aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:31.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.763+0000 7f35f7fcb700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f35f00cf4a0 0x7f35f00c3f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:31.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.763+0000 7f35f7fcb700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35f00cf880 0x7f35f01480b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:31.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.763+0000 7f35f7fcb700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35f00c5bb0 con 0x7f35f00cf880 2023-10-16T13:46:31.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.763+0000 7f35f7fcb700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35f00c6e80 con 0x7f35f00cf4a0 2023-10-16T13:46:31.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.763+0000 7f35f7fcb700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35f00c5de0 con 0x7f35f00b5830 2023-10-16T13:46:31.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.763+0000 7f35f6568700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35f00cf880 0x7f35f01480b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:31.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.763+0000 7f35f6568700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35f00cf880 0x7f35f01480b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53314/0 (socket says 172.21.15.16:53314) 2023-10-16T13:46:31.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.763+0000 7f35f6568700 1 -- 172.21.15.16:0/2730954010 learned_addr learned my addr 172.21.15.16:0/2730954010 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:31.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.763+0000 7f35f5566700 1 --2- 172.21.15.16:0/2730954010 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f35f00cf4a0 0x7f35f00c3f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:31.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.763+0000 7f35f5d67700 1 --2- 172.21.15.16:0/2730954010 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f35f00b5830 0x7f35f006aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:31.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.763+0000 7f35f6568700 1 -- 172.21.15.16:0/2730954010 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f35f00b5830 msgr2=0x7f35f006aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:31.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.764+0000 7f35f6568700 1 --2- 172.21.15.16:0/2730954010 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f35f00b5830 0x7f35f006aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:31.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.764+0000 7f35f6568700 1 -- 172.21.15.16:0/2730954010 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f35f00cf4a0 msgr2=0x7f35f00c3f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:31.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.764+0000 7f35f6568700 1 --2- 172.21.15.16:0/2730954010 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f35f00cf4a0 0x7f35f00c3f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:31.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.764+0000 7f35f6568700 1 -- 172.21.15.16:0/2730954010 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35e8009a70 con 0x7f35f00cf880 2023-10-16T13:46:31.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.764+0000 7f35f6568700 1 --2- 172.21.15.16:0/2730954010 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35f00cf880 0x7f35f01480b0 secure :-1 s=READY pgs=2482 cs=0 l=1 rev1=1 crypto rx=0x7f35f00cdfc0 tx=0x7f35e8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:31.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.764+0000 7f35e27fc700 1 -- 172.21.15.16:0/2730954010 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35e80046b0 con 0x7f35f00cf880 2023-10-16T13:46:31.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.764+0000 7f35e27fc700 1 -- 172.21.15.16:0/2730954010 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35e8004850 con 0x7f35f00cf880 2023-10-16T13:46:31.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.764+0000 7f35f7fcb700 1 -- 172.21.15.16:0/2730954010 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35f00c60c0 con 0x7f35f00cf880 2023-10-16T13:46:31.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.764+0000 7f35e27fc700 1 -- 172.21.15.16:0/2730954010 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35e801c680 con 0x7f35f00cf880 2023-10-16T13:46:31.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.764+0000 7f35f7fcb700 1 -- 172.21.15.16:0/2730954010 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35f00c6680 con 0x7f35f00cf880 2023-10-16T13:46:31.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.765+0000 7f35f7fcb700 1 -- 172.21.15.16:0/2730954010 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35f00c6ba0 con 0x7f35f00cf880 2023-10-16T13:46:31.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.765+0000 7f35e27fc700 1 -- 172.21.15.16:0/2730954010 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 138) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f35e801c7e0 con 0x7f35f00cf880 2023-10-16T13:46:31.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.766+0000 7f35e27fc700 1 --2- 172.21.15.16:0/2730954010 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f35d406b020 0x7f35d406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:31.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.766+0000 7f35e27fc700 1 -- 172.21.15.16:0/2730954010 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(225..225 src has 1..225) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f35e808b870 con 0x7f35f00cf880 2023-10-16T13:46:31.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.766+0000 7f35f5d67700 1 --2- 172.21.15.16:0/2730954010 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f35d406b020 0x7f35d406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:31.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.766+0000 7f35f5d67700 1 --2- 172.21.15.16:0/2730954010 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f35d406b020 0x7f35d406d4e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f35ec0011e0 tx=0x7f35ec009040 comp rx=0 tx=0).ready entity=mgr.11190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:31.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:31.770+0000 7f35e27fc700 1 -- 172.21.15.16:0/2730954010 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f35f00c6ba0 con 0x7f35f00cf880 2023-10-16T13:46:32.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.021+0000 7f35f7fcb700 1 -- 172.21.15.16:0/2730954010 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f35f00c6ba0 con 0x7f35f00cf880 2023-10-16T13:46:32.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.021+0000 7f35e27fc700 1 -- 172.21.15.16:0/2730954010 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v113) v1 ==== 53+0+73 (secure 0 0 0) 0x7f35f00c6ba0 con 0x7f35f00cf880 2023-10-16T13:46:32.025 INFO:teuthology.orchestra.run.smithi016.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-16T13:46:32.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.024+0000 7f35f7fcb700 1 -- 172.21.15.16:0/2730954010 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f35d406b020 msgr2=0x7f35d406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:32.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.024+0000 7f35f7fcb700 1 --2- 172.21.15.16:0/2730954010 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f35d406b020 0x7f35d406d4e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f35ec0011e0 tx=0x7f35ec009040 comp rx=0 tx=0).stop 2023-10-16T13:46:32.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.024+0000 7f35f7fcb700 1 -- 172.21.15.16:0/2730954010 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35f00cf880 msgr2=0x7f35f01480b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:32.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.024+0000 7f35f7fcb700 1 --2- 172.21.15.16:0/2730954010 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35f00cf880 0x7f35f01480b0 secure :-1 s=READY pgs=2482 cs=0 l=1 rev1=1 crypto rx=0x7f35f00cdfc0 tx=0x7f35e8004ad0 comp rx=0 tx=0).stop 2023-10-16T13:46:32.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.024+0000 7f35f7fcb700 1 -- 172.21.15.16:0/2730954010 shutdown_connections 2023-10-16T13:46:32.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.024+0000 7f35f7fcb700 1 --2- 172.21.15.16:0/2730954010 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f35d406b020 0x7f35d406d4e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:32.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.024+0000 7f35f7fcb700 1 --2- 172.21.15.16:0/2730954010 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35f00cf880 0x7f35f01480b0 unknown :-1 s=CLOSED pgs=2482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:32.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.024+0000 7f35f7fcb700 1 --2- 172.21.15.16:0/2730954010 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f35f00b5830 0x7f35f006aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:32.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.024+0000 7f35f7fcb700 1 --2- 172.21.15.16:0/2730954010 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f35f00cf4a0 0x7f35f00c3f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:32.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.024+0000 7f35f7fcb700 1 -- 172.21.15.16:0/2730954010 >> 172.21.15.16:0/2730954010 conn(0x7f35f00b1fd0 msgr2=0x7f35f015a470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:32.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.024+0000 7f35f7fcb700 1 -- 172.21.15.16:0/2730954010 shutdown_connections 2023-10-16T13:46:32.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.024+0000 7f35f7fcb700 1 -- 172.21.15.16:0/2730954010 wait complete. 2023-10-16T13:46:32.038 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-16T13:46:32.039 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T13:46:32.039 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi016.front.sepia.ceph.com 2023-10-16T13:46:32.039 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-16T13:46:32.040 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-16T13:46:32.040 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-16T13:46:32.040 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-16T13:46:32.041 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T13:46:32.041 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T13:46:32.042 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:46:32.042 DEBUG:teuthology.orchestra.run.smithi016:> ip addr 2023-10-16T13:46:32.060 INFO:teuthology.orchestra.run.smithi016.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T13:46:32.060 INFO:teuthology.orchestra.run.smithi016.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T13:46:32.060 INFO:teuthology.orchestra.run.smithi016.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T13:46:32.061 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T13:46:32.061 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 ::1/128 scope host 2023-10-16T13:46:32.061 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T13:46:32.061 INFO:teuthology.orchestra.run.smithi016.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T13:46:32.061 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e0 brd ff:ff:ff:ff:ff:ff 2023-10-16T13:46:32.062 INFO:teuthology.orchestra.run.smithi016.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T13:46:32.062 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e1 brd ff:ff:ff:ff:ff:ff 2023-10-16T13:46:32.062 INFO:teuthology.orchestra.run.smithi016.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T13:46:32.062 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:64 brd ff:ff:ff:ff:ff:ff 2023-10-16T13:46:32.062 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.21.15.16/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-16T13:46:32.063 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft 36544sec preferred_lft 36544sec 2023-10-16T13:46:32.063 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::ec4:7aff:febd:1564/64 scope link 2023-10-16T13:46:32.063 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T13:46:32.063 INFO:teuthology.orchestra.run.smithi016.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T13:46:32.063 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:65 brd ff:ff:ff:ff:ff:ff 2023-10-16T13:46:32.064 INFO:teuthology.orchestra.run.smithi016.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T13:46:32.064 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 36:cf:ea:25:db:f7 brd ff:ff:ff:ff:ff:ff 2023-10-16T13:46:32.064 INFO:teuthology.orchestra.run.smithi016.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-16T13:46:32.064 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T13:46:32.064 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::cfc:c5ff:fe52:a786/64 scope link 2023-10-16T13:46:32.064 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T13:46:32.065 INFO:teuthology.orchestra.run.smithi016.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T13:46:32.065 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 6e:62:9d:a4:43:23 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T13:46:32.065 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::6c62:9dff:fea4:4323/64 scope link 2023-10-16T13:46:32.065 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T13:46:32.065 INFO:teuthology.orchestra.run.smithi016.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T13:46:32.066 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 36:cf:ea:25:db:f7 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-16T13:46:32.066 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::34cf:eaff:fe25:dbf7/64 scope link 2023-10-16T13:46:32.066 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T13:46:32.066 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:46:32.067 DEBUG:teuthology.orchestra.run.smithi016:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T13:46:32.126 INFO:teuthology.orchestra.run.smithi016.stdout:1 2023-10-16T13:46:32.127 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:46:32.127 DEBUG:teuthology.orchestra.run.smithi016:> route 2023-10-16T13:46:32.185 INFO:teuthology.orchestra.run.smithi016.stdout:Kernel IP routing table 2023-10-16T13:46:32.185 INFO:teuthology.orchestra.run.smithi016.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T13:46:32.186 INFO:teuthology.orchestra.run.smithi016.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-16T13:46:32.186 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-16T13:46:32.186 INFO:teuthology.orchestra.run.smithi016.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T13:46:32.186 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:46:32.187 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec bash -c ' 2023-10-16T13:46:32.187 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2023-10-16T13:46:32.187 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-16T13:46:32.188 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-16T13:46:32.188 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-16T13:46:32.188 DEBUG:teuthology.orchestra.run.smithi016:> ') 2023-10-16T13:46:32.292 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:46:32.292 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2023-10-16T13:46:32.309 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-16T13:46:32.310 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-16T13:46:32.310 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T13:46:32.310 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:46:32.311 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set brx.0 up 2023-10-16T13:46:32.378 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T13:46:32.378 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-16T13:46:32.435 INFO:teuthology.orchestra.run.smithi016.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-16T13:46:32.435 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T13:46:32.436 DEBUG:teuthology.orchestra.run.smithi016:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-16T13:46:32.491 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-16T13:46:32.492 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:46:32.492 DEBUG:teuthology.orchestra.run.smithi016:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T13:46:32.578 INFO:teuthology.orchestra.run.smithi016.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T13:46:32.578 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T13:46:32.578 INFO:teuthology.orchestra.run.smithi016.stdout:invalid new device string format 2023-10-16T13:46:32.587 INFO:teuthology.orchestra.run.smithi016.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T13:46:32.587 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T13:46:32.588 INFO:teuthology.orchestra.run.smithi016.stdout:invalid new device string format 2023-10-16T13:46:32.588 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: resolved to: "172.21.15.16:3300,172.21.15.49:3300,172.21.15.49:3301" 2023-10-16T13:46:32.588 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T13:46:32.661 INFO:teuthology.orchestra.run.smithi016.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T13:46:32.661 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T13:46:32.661 INFO:teuthology.orchestra.run.smithi016.stdout:invalid new device string format 2023-10-16T13:46:32.661 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: resolved to: "172.21.15.16:3300,172.21.15.49:3300,172.21.15.49:3301" 2023-10-16T13:46:32.662 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T13:46:32.662 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: resolved to: "172.21.15.16:6789,172.21.15.49:6789,172.21.15.49:6790" 2023-10-16T13:46:32.662 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.16:6789,172.21.15.49:6789,172.21.15.49:6790:/ 2023-10-16T13:46:32.662 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=5b28dfa1-b21f-474a-9b2e-9fb77af58254" will pass to kernel 2023-10-16T13:46:32.663 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-16T13:46:32.664 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:46:32.664 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-16T13:46:32.692 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:46:32.693 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T13:46:32.781 INFO:teuthology.orchestra.run.smithi016.stdout:client9155 2023-10-16T13:46:32.782 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T13:46:32.782 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:46:32.799 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2023-10-16T13:46:32.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.966+0000 7fc1245d8700 1 Processor -- start 2023-10-16T13:46:32.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.967+0000 7fc1245d8700 1 -- start start 2023-10-16T13:46:32.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.967+0000 7fc1245d8700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc11c157b30 0x7fc11c0cefc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:32.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.967+0000 7fc1245d8700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fc11c161d20 0x7fc11c0cf500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:32.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.967+0000 7fc1245d8700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc11c142fc0 con 0x7fc11c157f10 2023-10-16T13:46:32.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.967+0000 7fc1245d8700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc11c06d900 con 0x7fc11c161d20 2023-10-16T13:46:32.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.967+0000 7fc1245d8700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc11c143210 con 0x7fc11c157b30 2023-10-16T13:46:32.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.968+0000 7fc121b73700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc11c157f10 0x7fc11c161610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42596/0 (socket says 172.21.15.16:42596) 2023-10-16T13:46:32.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.968+0000 7fc121b73700 1 -- 172.21.15.16:0/2357146403 learned_addr learned my addr 172.21.15.16:0/2357146403 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:46:32.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.968+0000 7fc122b75700 1 --2- 172.21.15.16:0/2357146403 >> v2:172.21.15.49:3300/0 conn(0x7fc11c161d20 0x7fc11c0cf500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:32.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.968+0000 7fc122374700 1 --2- 172.21.15.16:0/2357146403 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc11c157b30 0x7fc11c0cefc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:32.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.968+0000 7fc121372700 1 -- 172.21.15.16:0/2357146403 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1529738046 0 0) 0x7fc11c142fc0 con 0x7fc11c157f10 2023-10-16T13:46:32.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.968+0000 7fc121372700 1 -- 172.21.15.16:0/2357146403 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1080035b0 con 0x7fc11c157f10 2023-10-16T13:46:32.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.968+0000 7fc121372700 1 -- 172.21.15.16:0/2357146403 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1594226011 0 0) 0x7fc1080035b0 con 0x7fc11c157f10 2023-10-16T13:46:32.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.968+0000 7fc121372700 1 -- 172.21.15.16:0/2357146403 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc11c157b30 msgr2=0x7fc11c0cefc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:32.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.968+0000 7fc121372700 1 --2- 172.21.15.16:0/2357146403 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc11c157b30 0x7fc11c0cefc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:32.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.968+0000 7fc121372700 1 -- 172.21.15.16:0/2357146403 >> v2:172.21.15.49:3300/0 conn(0x7fc11c161d20 msgr2=0x7fc11c0cf500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:32.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.968+0000 7fc121372700 1 --2- 172.21.15.16:0/2357146403 >> v2:172.21.15.49:3300/0 conn(0x7fc11c161d20 0x7fc11c0cf500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:32.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.968+0000 7fc121372700 1 -- 172.21.15.16:0/2357146403 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc11c1434f0 con 0x7fc11c157f10 2023-10-16T13:46:32.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.968+0000 7fc121372700 1 -- 172.21.15.16:0/2357146403 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7fc118001670 con 0x7fc11c157f10 2023-10-16T13:46:32.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.968+0000 7fc121372700 1 -- 172.21.15.16:0/2357146403 >> v1:172.21.15.16:6789/0 conn(0x7fc11c157f10 legacy=0x7fc11c161610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:32.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.968+0000 7fc121372700 1 --2- 172.21.15.16:0/2357146403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc108003a00 0x7fc108003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:32.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.968+0000 7fc121372700 1 --2- 172.21.15.16:0/2357146403 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc108004850 0x7fc108004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:32.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.968+0000 7fc121372700 1 --2- 172.21.15.16:0/2357146403 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc11c157b30 0x7fc108005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:32.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.968+0000 7fc121372700 1 -- 172.21.15.16:0/2357146403 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc11c06d900 con 0x7fc108003a00 2023-10-16T13:46:32.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.968+0000 7fc121372700 1 -- 172.21.15.16:0/2357146403 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc11c143210 con 0x7fc108004850 2023-10-16T13:46:32.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.968+0000 7fc121372700 1 -- 172.21.15.16:0/2357146403 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc10800ba10 con 0x7fc11c157b30 2023-10-16T13:46:32.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.968+0000 7fc122374700 1 --2- 172.21.15.16:0/2357146403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc108003a00 0x7fc108003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:32.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.968+0000 7fc121b73700 1 --2- 172.21.15.16:0/2357146403 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc108004850 0x7fc108004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:32.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.969+0000 7fc122b75700 1 --2- 172.21.15.16:0/2357146403 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc11c157b30 0x7fc108005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:32.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.969+0000 7fc122374700 1 -- 172.21.15.16:0/2357146403 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc11c157b30 msgr2=0x7fc108005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:32.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.969+0000 7fc122374700 1 --2- 172.21.15.16:0/2357146403 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc11c157b30 0x7fc108005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:32.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.969+0000 7fc122374700 1 -- 172.21.15.16:0/2357146403 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc108004850 msgr2=0x7fc108004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:32.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.969+0000 7fc122374700 1 --2- 172.21.15.16:0/2357146403 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc108004850 0x7fc108004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:32.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.969+0000 7fc122374700 1 -- 172.21.15.16:0/2357146403 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1080035b0 con 0x7fc108003a00 2023-10-16T13:46:32.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.969+0000 7fc122b75700 1 --2- 172.21.15.16:0/2357146403 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc11c157b30 0x7fc108005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:46:32.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.969+0000 7fc122374700 1 --2- 172.21.15.16:0/2357146403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc108003a00 0x7fc108003e20 secure :-1 s=READY pgs=2484 cs=0 l=1 rev1=1 crypto rx=0x7fc10c009b60 tx=0x7fc108005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86941688ae58b28b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:32.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.969+0000 7fc121372700 1 -- 172.21.15.16:0/2357146403 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc10c00e660 con 0x7fc108003a00 2023-10-16T13:46:32.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.969+0000 7fc121372700 1 -- 172.21.15.16:0/2357146403 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc10c00e7c0 con 0x7fc108003a00 2023-10-16T13:46:32.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.969+0000 7fc121372700 1 -- 172.21.15.16:0/2357146403 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc10c00eb30 con 0x7fc108003a00 2023-10-16T13:46:32.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.969+0000 7fc1245d8700 1 -- 172.21.15.16:0/2357146403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc108003a00 msgr2=0x7fc108003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:32.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.969+0000 7fc1245d8700 1 --2- 172.21.15.16:0/2357146403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc108003a00 0x7fc108003e20 secure :-1 s=READY pgs=2484 cs=0 l=1 rev1=1 crypto rx=0x7fc10c009b60 tx=0x7fc108005880 comp rx=0 tx=0).stop 2023-10-16T13:46:32.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.969+0000 7fc122b75700 1 -- 172.21.15.16:0/2357146403 reap_dead start 2023-10-16T13:46:32.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.970+0000 7fc1245d8700 1 -- 172.21.15.16:0/2357146403 shutdown_connections 2023-10-16T13:46:32.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.970+0000 7fc1245d8700 1 -- 172.21.15.16:0/2357146403 >> 172.21.15.16:0/2357146403 conn(0x7fc11c06a7e0 msgr2=0x7fc11c06abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:32.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.970+0000 7fc1245d8700 1 -- 172.21.15.16:0/2357146403 shutdown_connections 2023-10-16T13:46:32.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.970+0000 7fc1245d8700 1 -- 172.21.15.16:0/2357146403 wait complete. 2023-10-16T13:46:32.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.971+0000 7fc1245d8700 1 Processor -- start 2023-10-16T13:46:32.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.971+0000 7fc1245d8700 1 -- start start 2023-10-16T13:46:32.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.972+0000 7fc1245d8700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc108003a00 0x7fc11c0c7750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:32.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.972+0000 7fc1245d8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc11c0ad550 0x7fc11c0ad9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:32.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.972+0000 7fc1245d8700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc11c0ae400 0x7fc11c0ae880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:32.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.972+0000 7fc1245d8700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc11c0b0140 con 0x7fc11c0ad550 2023-10-16T13:46:32.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.972+0000 7fc1245d8700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc11c0af1a0 con 0x7fc108003a00 2023-10-16T13:46:32.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.972+0000 7fc1245d8700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc11c0aeec0 con 0x7fc11c0ae400 2023-10-16T13:46:32.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.972+0000 7fc121b73700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc11c0ad550 0x7fc11c0ad9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:32.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.972+0000 7fc121b73700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc11c0ad550 0x7fc11c0ad9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53336/0 (socket says 172.21.15.16:53336) 2023-10-16T13:46:32.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.972+0000 7fc121b73700 1 -- 172.21.15.16:0/2121757001 learned_addr learned my addr 172.21.15.16:0/2121757001 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:32.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.972+0000 7fc122374700 1 --2- 172.21.15.16:0/2121757001 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc108003a00 0x7fc11c0c7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:32.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.972+0000 7fc122b75700 1 --2- 172.21.15.16:0/2121757001 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc11c0ae400 0x7fc11c0ae880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:32.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.972+0000 7fc121b73700 1 -- 172.21.15.16:0/2121757001 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc11c0ae400 msgr2=0x7fc11c0ae880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:32.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.972+0000 7fc121b73700 1 --2- 172.21.15.16:0/2121757001 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc11c0ae400 0x7fc11c0ae880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:32.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.972+0000 7fc121b73700 1 -- 172.21.15.16:0/2121757001 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc108003a00 msgr2=0x7fc11c0c7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:32.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.972+0000 7fc121b73700 1 --2- 172.21.15.16:0/2121757001 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc108003a00 0x7fc11c0c7750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:32.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.972+0000 7fc121b73700 1 -- 172.21.15.16:0/2121757001 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc11c0aca70 con 0x7fc11c0ad550 2023-10-16T13:46:32.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.972+0000 7fc122b75700 1 --2- 172.21.15.16:0/2121757001 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc11c0ae400 0x7fc11c0ae880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:46:32.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.972+0000 7fc121b73700 1 --2- 172.21.15.16:0/2121757001 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc11c0ad550 0x7fc11c0ad9d0 secure :-1 s=READY pgs=2485 cs=0 l=1 rev1=1 crypto rx=0x7fc118007310 tx=0x7fc118007620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:32.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.972+0000 7fc112ffd700 1 -- 172.21.15.16:0/2121757001 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc11800f6c0 con 0x7fc11c0ad550 2023-10-16T13:46:32.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.972+0000 7fc112ffd700 1 -- 172.21.15.16:0/2121757001 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc11800f840 con 0x7fc11c0ad550 2023-10-16T13:46:32.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.972+0000 7fc1245d8700 1 -- 172.21.15.16:0/2121757001 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc11c0acd50 con 0x7fc11c0ad550 2023-10-16T13:46:32.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.972+0000 7fc112ffd700 1 -- 172.21.15.16:0/2121757001 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc11800fbf0 con 0x7fc11c0ad550 2023-10-16T13:46:32.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.972+0000 7fc1245d8700 1 -- 172.21.15.16:0/2121757001 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc11c0c7c90 con 0x7fc11c0ad550 2023-10-16T13:46:32.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.973+0000 7fc1245d8700 1 -- 172.21.15.16:0/2121757001 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc11c0ac8d0 con 0x7fc11c0ad550 2023-10-16T13:46:32.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.974+0000 7fc112ffd700 1 -- 172.21.15.16:0/2121757001 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 138) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc11800fd50 con 0x7fc11c0ad550 2023-10-16T13:46:32.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.974+0000 7fc112ffd700 1 --2- 172.21.15.16:0/2121757001 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fc10006b020 0x7fc10006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:32.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.974+0000 7fc112ffd700 1 -- 172.21.15.16:0/2121757001 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(225..225 src has 1..225) v4 ==== 6157+0+0 (secure 0 0 0) 0x7fc1180898d0 con 0x7fc11c0ad550 2023-10-16T13:46:32.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.974+0000 7fc122374700 1 --2- 172.21.15.16:0/2121757001 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fc10006b020 0x7fc10006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:32.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.975+0000 7fc122374700 1 --2- 172.21.15.16:0/2121757001 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fc10006b020 0x7fc10006d4e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc10c0090b0 tx=0x7fc10c01f040 comp rx=0 tx=0).ready entity=mgr.11190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:32.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:32.979+0000 7fc112ffd700 1 -- 172.21.15.16:0/2121757001 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc11c0ac8d0 con 0x7fc11c0ad550 2023-10-16T13:46:33.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.230+0000 7fc1245d8700 1 -- 172.21.15.16:0/2121757001 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fc11c0cefa0 con 0x7fc11c0ad550 2023-10-16T13:46:33.509 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:46:33.512 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:46:33.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.610+0000 7fc112ffd700 1 -- 172.21.15.16:0/2121757001 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v114) v1 ==== 136+0+0 (secure 0 0 0) 0x7fc11c0cefa0 con 0x7fc11c0ad550 2023-10-16T13:46:33.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.612+0000 7fc1245d8700 1 -- 172.21.15.16:0/2121757001 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fc10006b020 msgr2=0x7fc10006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:33.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.612+0000 7fc1245d8700 1 --2- 172.21.15.16:0/2121757001 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fc10006b020 0x7fc10006d4e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc10c0090b0 tx=0x7fc10c01f040 comp rx=0 tx=0).stop 2023-10-16T13:46:33.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.612+0000 7fc1245d8700 1 -- 172.21.15.16:0/2121757001 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc11c0ad550 msgr2=0x7fc11c0ad9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:33.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.612+0000 7fc1245d8700 1 --2- 172.21.15.16:0/2121757001 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc11c0ad550 0x7fc11c0ad9d0 secure :-1 s=READY pgs=2485 cs=0 l=1 rev1=1 crypto rx=0x7fc118007310 tx=0x7fc118007620 comp rx=0 tx=0).stop 2023-10-16T13:46:33.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.612+0000 7fc1245d8700 1 -- 172.21.15.16:0/2121757001 shutdown_connections 2023-10-16T13:46:33.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.612+0000 7fc1245d8700 1 --2- 172.21.15.16:0/2121757001 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc11c0ae400 0x7fc11c0ae880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:33.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.612+0000 7fc1245d8700 1 --2- 172.21.15.16:0/2121757001 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc108003a00 0x7fc11c0c7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:33.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.612+0000 7fc1245d8700 1 --2- 172.21.15.16:0/2121757001 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fc10006b020 0x7fc10006d4e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:33.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.612+0000 7fc1245d8700 1 --2- 172.21.15.16:0/2121757001 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc11c0ad550 0x7fc11c0ad9d0 unknown :-1 s=CLOSED pgs=2485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:33.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.612+0000 7fc1245d8700 1 -- 172.21.15.16:0/2121757001 >> 172.21.15.16:0/2121757001 conn(0x7fc11c06a7e0 msgr2=0x7fc11c157170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:33.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.612+0000 7fc1245d8700 1 -- 172.21.15.16:0/2121757001 shutdown_connections 2023-10-16T13:46:33.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.612+0000 7fc1245d8700 1 -- 172.21.15.16:0/2121757001 wait complete. 2023-10-16T13:46:33.628 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:46:33.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.751+0000 7fdac835d700 1 Processor -- start 2023-10-16T13:46:33.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.751+0000 7fdac835d700 1 -- start start 2023-10-16T13:46:33.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.752+0000 7fdac835d700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fdac006aef0 0x7fdac00b3480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:33.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.752+0000 7fdac835d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdac00b2be0 0x7fdac00b39c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:33.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.752+0000 7fdac835d700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdac00d11c0 con 0x7fdac015b860 2023-10-16T13:46:33.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.752+0000 7fdac835d700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdac00d03d0 con 0x7fdac006aef0 2023-10-16T13:46:33.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.752+0000 7fdac835d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdac00cd360 con 0x7fdac00b2be0 2023-10-16T13:46:33.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.752+0000 7fdac60f9700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fdac015b860 0x7fdac006a7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42616/0 (socket says 172.21.15.16:42616) 2023-10-16T13:46:33.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.752+0000 7fdac60f9700 1 -- 172.21.15.16:0/3022727050 learned_addr learned my addr 172.21.15.16:0/3022727050 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:46:33.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.752+0000 7fdac68fa700 1 --2- 172.21.15.16:0/3022727050 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdac00b2be0 0x7fdac00b39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:33.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.752+0000 7fdac58f8700 1 --2- 172.21.15.16:0/3022727050 >> v2:172.21.15.49:3300/0 conn(0x7fdac006aef0 0x7fdac00b3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:33.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.752+0000 7fdac50f7700 1 -- 172.21.15.16:0/3022727050 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3748897963 0 0) 0x7fdac00d11c0 con 0x7fdac015b860 2023-10-16T13:46:33.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.752+0000 7fdac50f7700 1 -- 172.21.15.16:0/3022727050 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdaac0035b0 con 0x7fdac015b860 2023-10-16T13:46:33.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.752+0000 7fdac50f7700 1 -- 172.21.15.16:0/3022727050 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3735702460 0 0) 0x7fdaac0035b0 con 0x7fdac015b860 2023-10-16T13:46:33.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.752+0000 7fdac50f7700 1 -- 172.21.15.16:0/3022727050 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdac00b2be0 msgr2=0x7fdac00b39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:33.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.752+0000 7fdac50f7700 1 --2- 172.21.15.16:0/3022727050 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdac00b2be0 0x7fdac00b39c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:33.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.752+0000 7fdac50f7700 1 -- 172.21.15.16:0/3022727050 >> v2:172.21.15.49:3300/0 conn(0x7fdac006aef0 msgr2=0x7fdac00b3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:33.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.752+0000 7fdac50f7700 1 --2- 172.21.15.16:0/3022727050 >> v2:172.21.15.49:3300/0 conn(0x7fdac006aef0 0x7fdac00b3480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:33.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.752+0000 7fdac50f7700 1 -- 172.21.15.16:0/3022727050 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdac00d05b0 con 0x7fdac015b860 2023-10-16T13:46:33.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.752+0000 7fdac50f7700 1 -- 172.21.15.16:0/3022727050 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7fdab4001670 con 0x7fdac015b860 2023-10-16T13:46:33.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.752+0000 7fdac50f7700 1 -- 172.21.15.16:0/3022727050 >> v1:172.21.15.16:6789/0 conn(0x7fdac015b860 legacy=0x7fdac006a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:33.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.752+0000 7fdac50f7700 1 --2- 172.21.15.16:0/3022727050 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdaac003a00 0x7fdaac003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:33.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.752+0000 7fdac50f7700 1 --2- 172.21.15.16:0/3022727050 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdac00b2be0 0x7fdaac004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:33.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.752+0000 7fdac50f7700 1 --2- 172.21.15.16:0/3022727050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdaac004d90 0x7fdaac005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:33.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.752+0000 7fdac50f7700 1 -- 172.21.15.16:0/3022727050 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdac00d03d0 con 0x7fdaac004d90 2023-10-16T13:46:33.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.752+0000 7fdac50f7700 1 -- 172.21.15.16:0/3022727050 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdac00cd360 con 0x7fdaac003a00 2023-10-16T13:46:33.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.752+0000 7fdac50f7700 1 -- 172.21.15.16:0/3022727050 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdaac00ba10 con 0x7fdac00b2be0 2023-10-16T13:46:33.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.753+0000 7fdac68fa700 1 --2- 172.21.15.16:0/3022727050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdaac004d90 0x7fdaac005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:33.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.753+0000 7fdac58f8700 1 --2- 172.21.15.16:0/3022727050 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdac00b2be0 0x7fdaac004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:33.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.753+0000 7fdac60f9700 1 --2- 172.21.15.16:0/3022727050 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdaac003a00 0x7fdaac003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:33.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.753+0000 7fdac68fa700 1 -- 172.21.15.16:0/3022727050 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdac00b2be0 msgr2=0x7fdaac004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:33.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.753+0000 7fdac68fa700 1 --2- 172.21.15.16:0/3022727050 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdac00b2be0 0x7fdaac004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:33.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.753+0000 7fdac68fa700 1 -- 172.21.15.16:0/3022727050 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdaac003a00 msgr2=0x7fdaac003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:33.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.753+0000 7fdac68fa700 1 --2- 172.21.15.16:0/3022727050 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdaac003a00 0x7fdaac003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:33.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.753+0000 7fdac68fa700 1 -- 172.21.15.16:0/3022727050 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaac0035b0 con 0x7fdaac004d90 2023-10-16T13:46:33.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.753+0000 7fdac68fa700 1 --2- 172.21.15.16:0/3022727050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdaac004d90 0x7fdaac005210 secure :-1 s=READY pgs=2487 cs=0 l=1 rev1=1 crypto rx=0x7fdab8009fc0 tx=0x7fdaac005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=997ad04536162d91 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:33.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.753+0000 7fdac50f7700 1 -- 172.21.15.16:0/3022727050 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdab800ebd0 con 0x7fdaac004d90 2023-10-16T13:46:33.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.753+0000 7fdac50f7700 1 -- 172.21.15.16:0/3022727050 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdab800f040 con 0x7fdaac004d90 2023-10-16T13:46:33.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.753+0000 7fdac50f7700 1 -- 172.21.15.16:0/3022727050 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdab8014480 con 0x7fdaac004d90 2023-10-16T13:46:33.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.754+0000 7fdac835d700 1 -- 172.21.15.16:0/3022727050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdaac004d90 msgr2=0x7fdaac005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:33.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.754+0000 7fdac835d700 1 --2- 172.21.15.16:0/3022727050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdaac004d90 0x7fdaac005210 secure :-1 s=READY pgs=2487 cs=0 l=1 rev1=1 crypto rx=0x7fdab8009fc0 tx=0x7fdaac005880 comp rx=0 tx=0).stop 2023-10-16T13:46:33.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.754+0000 7fdac68fa700 1 -- 172.21.15.16:0/3022727050 reap_dead start 2023-10-16T13:46:33.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.754+0000 7fdac835d700 1 -- 172.21.15.16:0/3022727050 shutdown_connections 2023-10-16T13:46:33.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.754+0000 7fdac835d700 1 -- 172.21.15.16:0/3022727050 >> 172.21.15.16:0/3022727050 conn(0x7fdac00b1fd0 msgr2=0x7fdac00b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:33.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.754+0000 7fdac835d700 1 -- 172.21.15.16:0/3022727050 shutdown_connections 2023-10-16T13:46:33.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.754+0000 7fdac835d700 1 -- 172.21.15.16:0/3022727050 wait complete. 2023-10-16T13:46:33.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.755+0000 7fdac835d700 1 Processor -- start 2023-10-16T13:46:33.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.755+0000 7fdac835d700 1 -- start start 2023-10-16T13:46:33.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.756+0000 7fdac835d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdaac004d90 0x7fdac00b3df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:33.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.756+0000 7fdac835d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdac014b110 0x7fdac014b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:33.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.756+0000 7fdac835d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdac0148430 0x7fdac01488c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:33.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.756+0000 7fdac835d700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdac0149080 con 0x7fdac0148430 2023-10-16T13:46:33.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.756+0000 7fdac835d700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdac0148e00 con 0x7fdaac004d90 2023-10-16T13:46:33.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.756+0000 7fdac835d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdac014bbd0 con 0x7fdac014b110 2023-10-16T13:46:33.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.756+0000 7fdac68fa700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdac0148430 0x7fdac01488c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:33.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.756+0000 7fdac68fa700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdac0148430 0x7fdac01488c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53358/0 (socket says 172.21.15.16:53358) 2023-10-16T13:46:33.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.756+0000 7fdac68fa700 1 -- 172.21.15.16:0/549919764 learned_addr learned my addr 172.21.15.16:0/549919764 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:33.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.756+0000 7fdac60f9700 1 --2- 172.21.15.16:0/549919764 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdaac004d90 0x7fdac00b3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:33.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.756+0000 7fdac58f8700 1 --2- 172.21.15.16:0/549919764 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdac014b110 0x7fdac014b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:33.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.756+0000 7fdac68fa700 1 -- 172.21.15.16:0/549919764 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdac014b110 msgr2=0x7fdac014b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:33.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.756+0000 7fdac68fa700 1 --2- 172.21.15.16:0/549919764 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdac014b110 0x7fdac014b590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:33.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.756+0000 7fdac68fa700 1 -- 172.21.15.16:0/549919764 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdaac004d90 msgr2=0x7fdac00b3df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:33.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.756+0000 7fdac68fa700 1 --2- 172.21.15.16:0/549919764 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdaac004d90 0x7fdac00b3df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:33.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.756+0000 7fdac68fa700 1 -- 172.21.15.16:0/549919764 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdac014beb0 con 0x7fdac0148430 2023-10-16T13:46:33.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.756+0000 7fdac68fa700 1 --2- 172.21.15.16:0/549919764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdac0148430 0x7fdac01488c0 secure :-1 s=READY pgs=2488 cs=0 l=1 rev1=1 crypto rx=0x7fdaac004710 tx=0x7fdab8009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:33.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.757+0000 7fdab2ffd700 1 -- 172.21.15.16:0/549919764 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdab8007000 con 0x7fdac0148430 2023-10-16T13:46:33.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.757+0000 7fdab2ffd700 1 -- 172.21.15.16:0/549919764 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdab8022070 con 0x7fdac0148430 2023-10-16T13:46:33.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.757+0000 7fdac835d700 1 -- 172.21.15.16:0/549919764 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdac0149290 con 0x7fdac0148430 2023-10-16T13:46:33.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.757+0000 7fdab2ffd700 1 -- 172.21.15.16:0/549919764 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdab8003a20 con 0x7fdac0148430 2023-10-16T13:46:33.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.757+0000 7fdac835d700 1 -- 172.21.15.16:0/549919764 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdac015e010 con 0x7fdac0148430 2023-10-16T13:46:33.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.757+0000 7fdac835d700 1 -- 172.21.15.16:0/549919764 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdac01497b0 con 0x7fdac0148430 2023-10-16T13:46:33.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.758+0000 7fdab2ffd700 1 -- 172.21.15.16:0/549919764 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 138) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fdab8003b80 con 0x7fdac0148430 2023-10-16T13:46:33.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.759+0000 7fdab2ffd700 1 --2- 172.21.15.16:0/549919764 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fdaa406b020 0x7fdaa406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:33.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.759+0000 7fdab2ffd700 1 -- 172.21.15.16:0/549919764 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(225..225 src has 1..225) v4 ==== 6157+0+0 (secure 0 0 0) 0x7fdab8024070 con 0x7fdac0148430 2023-10-16T13:46:33.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.759+0000 7fdac60f9700 1 --2- 172.21.15.16:0/549919764 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fdaa406b020 0x7fdaa406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:33.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.759+0000 7fdac60f9700 1 --2- 172.21.15.16:0/549919764 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fdaa406b020 0x7fdaa406d4e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fdab4000c00 tx=0x7fdab4007980 comp rx=0 tx=0).ready entity=mgr.11190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:33.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.763+0000 7fdab2ffd700 1 -- 172.21.15.16:0/549919764 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdac01497b0 con 0x7fdac0148430 2023-10-16T13:46:33.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.965+0000 7fdac835d700 1 -- 172.21.15.16:0/549919764 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdac01497b0 con 0x7fdac0148430 2023-10-16T13:46:33.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.966+0000 7fdab2ffd700 1 -- 172.21.15.16:0/549919764 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v225) v1 ==== 74+0+16881 (secure 0 0 0) 0x7fdac01497b0 con 0x7fdac0148430 2023-10-16T13:46:33.969 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:46:33.969 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":225,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:46:26.556006+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2023-10-16T13:46:22.489653+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"224","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2023-10-16T13:46:23.729509+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"225","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":223,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":223,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":223,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":223,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":223,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":223,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":223,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":223,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:46:33.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.968+0000 7fdac835d700 1 -- 172.21.15.16:0/549919764 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fdaa406b020 msgr2=0x7fdaa406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:33.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.968+0000 7fdac835d700 1 --2- 172.21.15.16:0/549919764 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fdaa406b020 0x7fdaa406d4e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fdab4000c00 tx=0x7fdab4007980 comp rx=0 tx=0).stop 2023-10-16T13:46:33.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.968+0000 7fdac835d700 1 -- 172.21.15.16:0/549919764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdac0148430 msgr2=0x7fdac01488c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:33.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.968+0000 7fdac835d700 1 --2- 172.21.15.16:0/549919764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdac0148430 0x7fdac01488c0 secure :-1 s=READY pgs=2488 cs=0 l=1 rev1=1 crypto rx=0x7fdaac004710 tx=0x7fdab8009750 comp rx=0 tx=0).stop 2023-10-16T13:46:33.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.968+0000 7fdac835d700 1 -- 172.21.15.16:0/549919764 shutdown_connections 2023-10-16T13:46:33.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.968+0000 7fdac835d700 1 --2- 172.21.15.16:0/549919764 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fdaa406b020 0x7fdaa406d4e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:33.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.968+0000 7fdac835d700 1 --2- 172.21.15.16:0/549919764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdac0148430 0x7fdac01488c0 unknown :-1 s=CLOSED pgs=2488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:33.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.968+0000 7fdac835d700 1 --2- 172.21.15.16:0/549919764 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdaac004d90 0x7fdac00b3df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:33.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.968+0000 7fdac835d700 1 --2- 172.21.15.16:0/549919764 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdac014b110 0x7fdac014b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:33.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.968+0000 7fdac835d700 1 -- 172.21.15.16:0/549919764 >> 172.21.15.16:0/549919764 conn(0x7fdac00b1fd0 msgr2=0x7fdac00b5d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:33.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.968+0000 7fdac835d700 1 -- 172.21.15.16:0/549919764 shutdown_connections 2023-10-16T13:46:33.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:33.968+0000 7fdac835d700 1 -- 172.21.15.16:0/549919764 wait complete. 2023-10-16T13:46:33.983 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 27, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T13:46:22.489653+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '224', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 28, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T13:46:23.729509+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '225', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T13:46:33.984 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T13:46:34.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.110+0000 7f10f676f700 1 Processor -- start 2023-10-16T13:46:34.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.110+0000 7f10f676f700 1 -- start start 2023-10-16T13:46:34.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.111+0000 7f10f676f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f10f00cb880 0x7f10f00cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:34.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.111+0000 7f10f676f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f10f006da60 0x7f10f006def0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:34.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.111+0000 7f10f676f700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10f00cef70 con 0x7f10f015b070 2023-10-16T13:46:34.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.111+0000 7f10f676f700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10f006e430 con 0x7f10f00cb880 2023-10-16T13:46:34.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.111+0000 7f10f676f700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10f00ce360 con 0x7f10f006da60 2023-10-16T13:46:34.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.111+0000 7f10ef7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f10f00cb880 0x7f10f00cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:34.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.111+0000 7f10effff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f10f015b070 0x7f10f015b450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42636/0 (socket says 172.21.15.16:42636) 2023-10-16T13:46:34.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.111+0000 7f10ef7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f10f00cb880 0x7f10f00cbd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53364/0 (socket says 172.21.15.16:53364) 2023-10-16T13:46:34.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.111+0000 7f10effff700 1 -- 172.21.15.16:0/3659325906 learned_addr learned my addr 172.21.15.16:0/3659325906 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:46:34.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.111+0000 7f10f4d0c700 1 --2- 172.21.15.16:0/3659325906 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f10f006da60 0x7f10f006def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:34.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.111+0000 7f10ef7fe700 1 -- 172.21.15.16:0/3659325906 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f10f006da60 msgr2=0x7f10f006def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:34.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.111+0000 7f10ef7fe700 1 --2- 172.21.15.16:0/3659325906 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f10f006da60 0x7f10f006def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:34.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.111+0000 7f10ef7fe700 1 -- 172.21.15.16:0/3659325906 >> v1:172.21.15.16:6789/0 conn(0x7f10f015b070 legacy=0x7f10f015b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:34.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.111+0000 7f10ef7fe700 1 -- 172.21.15.16:0/3659325906 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10f00ce640 con 0x7f10f00cb880 2023-10-16T13:46:34.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.111+0000 7f10eeffd700 1 -- 172.21.15.16:0/3659325906 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 946596098 0 0) 0x7f10f00cef70 con 0x7f10f015b070 2023-10-16T13:46:34.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.111+0000 7f10ef7fe700 1 --2- 172.21.15.16:0/3659325906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10f00cb880 0x7f10f00cbd00 secure :-1 s=READY pgs=2490 cs=0 l=1 rev1=1 crypto rx=0x7f10d8009840 tx=0x7f10d8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f31930dda94d4405 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:34.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.112+0000 7f10eeffd700 1 -- 172.21.15.16:0/3659325906 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10d8011830 con 0x7f10f00cb880 2023-10-16T13:46:34.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.112+0000 7f10eeffd700 1 -- 172.21.15.16:0/3659325906 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10d8011990 con 0x7f10f00cb880 2023-10-16T13:46:34.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.112+0000 7f10eeffd700 1 -- 172.21.15.16:0/3659325906 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10d8011d20 con 0x7f10f00cb880 2023-10-16T13:46:34.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.112+0000 7f10f676f700 1 -- 172.21.15.16:0/3659325906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10f00cb880 msgr2=0x7f10f00cbd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:34.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.112+0000 7f10f676f700 1 --2- 172.21.15.16:0/3659325906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10f00cb880 0x7f10f00cbd00 secure :-1 s=READY pgs=2490 cs=0 l=1 rev1=1 crypto rx=0x7f10d8009840 tx=0x7f10d8009b50 comp rx=0 tx=0).stop 2023-10-16T13:46:34.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.112+0000 7f10f676f700 1 -- 172.21.15.16:0/3659325906 shutdown_connections 2023-10-16T13:46:34.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.112+0000 7f10f676f700 1 --2- 172.21.15.16:0/3659325906 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f10f006da60 0x7f10f006def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:34.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.112+0000 7f10f676f700 1 --2- 172.21.15.16:0/3659325906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10f00cb880 0x7f10f00cbd00 unknown :-1 s=CLOSED pgs=2490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:34.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.112+0000 7f10f676f700 1 -- 172.21.15.16:0/3659325906 >> 172.21.15.16:0/3659325906 conn(0x7f10f00b1f40 msgr2=0x7f10f00b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:34.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.112+0000 7f10f676f700 1 -- 172.21.15.16:0/3659325906 shutdown_connections 2023-10-16T13:46:34.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.112+0000 7f10f676f700 1 -- 172.21.15.16:0/3659325906 wait complete. 2023-10-16T13:46:34.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.113+0000 7f10f676f700 1 Processor -- start 2023-10-16T13:46:34.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.114+0000 7f10f676f700 1 -- start start 2023-10-16T13:46:34.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.114+0000 7f10f676f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10f006da60 0x7f10f0151910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:34.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.114+0000 7f10f676f700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f10f00cb880 0x7f10f0155ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:34.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.114+0000 7f10f676f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f10f015b070 0x7f10f0152380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:34.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.114+0000 7f10f676f700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10f0157050 con 0x7f10f006da60 2023-10-16T13:46:34.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.114+0000 7f10f676f700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10f0157330 con 0x7f10f00cb880 2023-10-16T13:46:34.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.114+0000 7f10f676f700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10f0151e50 con 0x7f10f015b070 2023-10-16T13:46:34.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.114+0000 7f10effff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10f006da60 0x7f10f0151910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:34.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.114+0000 7f10effff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10f006da60 0x7f10f0151910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53368/0 (socket says 172.21.15.16:53368) 2023-10-16T13:46:34.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.114+0000 7f10effff700 1 -- 172.21.15.16:0/2046435765 learned_addr learned my addr 172.21.15.16:0/2046435765 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:34.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.115+0000 7f10effff700 1 -- 172.21.15.16:0/2046435765 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f10f015b070 msgr2=0x7f10f0152380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:34.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.115+0000 7f10effff700 1 --2- 172.21.15.16:0/2046435765 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f10f015b070 0x7f10f0152380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:34.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.115+0000 7f10effff700 1 -- 172.21.15.16:0/2046435765 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f10f00cb880 msgr2=0x7f10f0155ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:34.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.115+0000 7f10ef7fe700 1 --2- 172.21.15.16:0/2046435765 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f10f00cb880 0x7f10f0155ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:34.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.115+0000 7f10effff700 1 --2- 172.21.15.16:0/2046435765 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f10f00cb880 0x7f10f0155ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:34.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.115+0000 7f10effff700 1 -- 172.21.15.16:0/2046435765 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10d80094a0 con 0x7f10f006da60 2023-10-16T13:46:34.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.115+0000 7f10ef7fe700 1 --2- 172.21.15.16:0/2046435765 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f10f00cb880 0x7f10f0155ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:46:34.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.115+0000 7f10effff700 1 --2- 172.21.15.16:0/2046435765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10f006da60 0x7f10f0151910 secure :-1 s=READY pgs=2491 cs=0 l=1 rev1=1 crypto rx=0x7f10e000c550 tx=0x7f10e000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:34.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.115+0000 7f10ecff9700 1 -- 172.21.15.16:0/2046435765 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10e0004c80 con 0x7f10f006da60 2023-10-16T13:46:34.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.115+0000 7f10ecff9700 1 -- 172.21.15.16:0/2046435765 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10e0004e00 con 0x7f10f006da60 2023-10-16T13:46:34.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.115+0000 7f10ecff9700 1 -- 172.21.15.16:0/2046435765 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10e0007310 con 0x7f10f006da60 2023-10-16T13:46:34.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.115+0000 7f10f676f700 1 -- 172.21.15.16:0/2046435765 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10f01563d0 con 0x7f10f006da60 2023-10-16T13:46:34.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.115+0000 7f10f676f700 1 -- 172.21.15.16:0/2046435765 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10f0156990 con 0x7f10f006da60 2023-10-16T13:46:34.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.116+0000 7f10f676f700 1 -- 172.21.15.16:0/2046435765 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10f004ebb0 con 0x7f10f006da60 2023-10-16T13:46:34.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.117+0000 7f10ecff9700 1 -- 172.21.15.16:0/2046435765 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 138) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f10e0007470 con 0x7f10f006da60 2023-10-16T13:46:34.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.117+0000 7f10ecff9700 1 --2- 172.21.15.16:0/2046435765 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f10d006b020 0x7f10d006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:34.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.117+0000 7f10ecff9700 1 -- 172.21.15.16:0/2046435765 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(225..225 src has 1..225) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f10e0089560 con 0x7f10f006da60 2023-10-16T13:46:34.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.117+0000 7f10ef7fe700 1 --2- 172.21.15.16:0/2046435765 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f10d006b020 0x7f10d006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:34.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.117+0000 7f10ef7fe700 1 --2- 172.21.15.16:0/2046435765 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f10d006b020 0x7f10d006d4e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f10f0155030 tx=0x7f10d8008030 comp rx=0 tx=0).ready entity=mgr.11190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:34.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.121+0000 7f10ecff9700 1 -- 172.21.15.16:0/2046435765 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f10e00543d0 con 0x7f10f006da60 2023-10-16T13:46:34.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.322+0000 7f10f676f700 1 -- 172.21.15.16:0/2046435765 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f10f00c68c0 con 0x7f10f006da60 2023-10-16T13:46:34.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.323+0000 7f10ecff9700 1 -- 172.21.15.16:0/2046435765 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v225) v1 ==== 93+0+10 (secure 0 0 0) 0x7f10e00579f0 con 0x7f10f006da60 2023-10-16T13:46:34.326 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T13:46:34.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.325+0000 7f10f676f700 1 -- 172.21.15.16:0/2046435765 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f10d006b020 msgr2=0x7f10d006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:34.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.325+0000 7f10f676f700 1 --2- 172.21.15.16:0/2046435765 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f10d006b020 0x7f10d006d4e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f10f0155030 tx=0x7f10d8008030 comp rx=0 tx=0).stop 2023-10-16T13:46:34.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.325+0000 7f10f676f700 1 -- 172.21.15.16:0/2046435765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10f006da60 msgr2=0x7f10f0151910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:34.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.325+0000 7f10f676f700 1 --2- 172.21.15.16:0/2046435765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10f006da60 0x7f10f0151910 secure :-1 s=READY pgs=2491 cs=0 l=1 rev1=1 crypto rx=0x7f10e000c550 tx=0x7f10e000c910 comp rx=0 tx=0).stop 2023-10-16T13:46:34.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.325+0000 7f10f676f700 1 -- 172.21.15.16:0/2046435765 shutdown_connections 2023-10-16T13:46:34.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.325+0000 7f10f676f700 1 --2- 172.21.15.16:0/2046435765 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f10f015b070 0x7f10f0152380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:34.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.325+0000 7f10f676f700 1 --2- 172.21.15.16:0/2046435765 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f10d006b020 0x7f10d006d4e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:34.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.325+0000 7f10f676f700 1 --2- 172.21.15.16:0/2046435765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10f006da60 0x7f10f0151910 unknown :-1 s=CLOSED pgs=2491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:34.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.325+0000 7f10f676f700 1 --2- 172.21.15.16:0/2046435765 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f10f00cb880 0x7f10f0155ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:34.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.325+0000 7f10f676f700 1 -- 172.21.15.16:0/2046435765 >> 172.21.15.16:0/2046435765 conn(0x7f10f00b1f40 msgr2=0x7f10f00ccd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:34.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.325+0000 7f10f676f700 1 -- 172.21.15.16:0/2046435765 shutdown_connections 2023-10-16T13:46:34.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.325+0000 7f10f676f700 1 -- 172.21.15.16:0/2046435765 wait complete. 2023-10-16T13:46:34.340 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T13:46:34.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.465+0000 7f50bdf4f700 1 Processor -- start 2023-10-16T13:46:34.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.465+0000 7f50bdf4f700 1 -- start start 2023-10-16T13:46:34.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.466+0000 7f50bdf4f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f50b80cda60 0x7f50b806d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:34.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.466+0000 7f50bdf4f700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f50b806dea0 0x7f50b80ce850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:34.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.466+0000 7f50bdf4f700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50b80ce0e0 con 0x7f50b815b790 2023-10-16T13:46:34.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.466+0000 7f50bdf4f700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50b806e2b0 con 0x7f50b80cda60 2023-10-16T13:46:34.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.466+0000 7f50bdf4f700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50b806a7e0 con 0x7f50b806dea0 2023-10-16T13:46:34.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.466+0000 7f50b6ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f50b80cda60 0x7f50b806d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:34.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.466+0000 7f50b6ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f50b80cda60 0x7f50b806d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53376/0 (socket says 172.21.15.16:53376) 2023-10-16T13:46:34.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.466+0000 7f50b6ffd700 1 -- 172.21.15.16:0/3279779828 learned_addr learned my addr 172.21.15.16:0/3279779828 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:34.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.467+0000 7f50b7fff700 1 --2- 172.21.15.16:0/3279779828 >> v2:172.21.15.49:3300/0 conn(0x7f50b806dea0 0x7f50b80ce850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:34.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.467+0000 7f50b6ffd700 1 -- 172.21.15.16:0/3279779828 >> v2:172.21.15.49:3300/0 conn(0x7f50b806dea0 msgr2=0x7f50b80ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:34.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.467+0000 7f50b6ffd700 1 --2- 172.21.15.16:0/3279779828 >> v2:172.21.15.49:3300/0 conn(0x7f50b806dea0 0x7f50b80ce850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:34.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.467+0000 7f50b6ffd700 1 -- 172.21.15.16:0/3279779828 >> v1:172.21.15.49:6789/0 conn(0x7f50b815b790 legacy=0x7f50b815bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:34.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.467+0000 7f50b6ffd700 1 -- 172.21.15.16:0/3279779828 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50b815c410 con 0x7f50b80cda60 2023-10-16T13:46:34.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.467+0000 7f50b6ffd700 1 --2- 172.21.15.16:0/3279779828 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50b80cda60 0x7f50b806d960 secure :-1 s=READY pgs=2492 cs=0 l=1 rev1=1 crypto rx=0x7f50b80af630 tx=0x7f50a0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35c83e8983e32ab6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:34.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.467+0000 7f50b67fc700 1 -- 172.21.15.16:0/3279779828 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50a0011830 con 0x7f50b80cda60 2023-10-16T13:46:34.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.467+0000 7f50b67fc700 1 -- 172.21.15.16:0/3279779828 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50a0011990 con 0x7f50b80cda60 2023-10-16T13:46:34.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.467+0000 7f50b67fc700 1 -- 172.21.15.16:0/3279779828 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50a0011d20 con 0x7f50b80cda60 2023-10-16T13:46:34.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.468+0000 7f50bdf4f700 1 -- 172.21.15.16:0/3279779828 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50b80cda60 msgr2=0x7f50b806d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:34.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.468+0000 7f50bdf4f700 1 --2- 172.21.15.16:0/3279779828 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50b80cda60 0x7f50b806d960 secure :-1 s=READY pgs=2492 cs=0 l=1 rev1=1 crypto rx=0x7f50b80af630 tx=0x7f50a0009bd0 comp rx=0 tx=0).stop 2023-10-16T13:46:34.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.468+0000 7f50bdf4f700 1 -- 172.21.15.16:0/3279779828 shutdown_connections 2023-10-16T13:46:34.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.468+0000 7f50bdf4f700 1 --2- 172.21.15.16:0/3279779828 >> v2:172.21.15.49:3300/0 conn(0x7f50b806dea0 0x7f50b80ce850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:34.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.468+0000 7f50bdf4f700 1 --2- 172.21.15.16:0/3279779828 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50b80cda60 0x7f50b806d960 unknown :-1 s=CLOSED pgs=2492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:34.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.468+0000 7f50bdf4f700 1 -- 172.21.15.16:0/3279779828 >> 172.21.15.16:0/3279779828 conn(0x7f50b80cc5e0 msgr2=0x7f50b80cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:34.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.468+0000 7f50bdf4f700 1 -- 172.21.15.16:0/3279779828 shutdown_connections 2023-10-16T13:46:34.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.468+0000 7f50bdf4f700 1 -- 172.21.15.16:0/3279779828 wait complete. 2023-10-16T13:46:34.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.469+0000 7f50bdf4f700 1 Processor -- start 2023-10-16T13:46:34.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.469+0000 7f50bdf4f700 1 -- start start 2023-10-16T13:46:34.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.470+0000 7f50bdf4f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50b806dea0 0x7f50b8147b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:34.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.470+0000 7f50bdf4f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f50b80cda60 0x7f50b80b29b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:34.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.470+0000 7f50bdf4f700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f50b815b790 0x7f50b80c3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:34.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.470+0000 7f50bdf4f700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50b8155800 con 0x7f50b806dea0 2023-10-16T13:46:34.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.470+0000 7f50bdf4f700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50b81f0ca0 con 0x7f50b815b790 2023-10-16T13:46:34.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.470+0000 7f50bdf4f700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50b80b3c10 con 0x7f50b80cda60 2023-10-16T13:46:34.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.470+0000 7f50b77fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50b806dea0 0x7f50b8147b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:34.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.470+0000 7f50b77fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50b806dea0 0x7f50b8147b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53380/0 (socket says 172.21.15.16:53380) 2023-10-16T13:46:34.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.470+0000 7f50b77fe700 1 -- 172.21.15.16:0/806323003 learned_addr learned my addr 172.21.15.16:0/806323003 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:34.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.470+0000 7f50b6ffd700 1 --2- 172.21.15.16:0/806323003 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f50b80cda60 0x7f50b80b29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:34.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.470+0000 7f50b7fff700 1 --2- 172.21.15.16:0/806323003 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f50b815b790 0x7f50b80c3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:34.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.470+0000 7f50b77fe700 1 -- 172.21.15.16:0/806323003 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f50b80cda60 msgr2=0x7f50b80b29b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:34.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.470+0000 7f50b77fe700 1 --2- 172.21.15.16:0/806323003 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f50b80cda60 0x7f50b80b29b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:34.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.470+0000 7f50b77fe700 1 -- 172.21.15.16:0/806323003 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f50b815b790 msgr2=0x7f50b80c3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:34.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.470+0000 7f50b77fe700 1 --2- 172.21.15.16:0/806323003 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f50b815b790 0x7f50b80c3eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:34.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.470+0000 7f50b77fe700 1 -- 172.21.15.16:0/806323003 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50a0009550 con 0x7f50b806dea0 2023-10-16T13:46:34.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.470+0000 7f50b77fe700 1 --2- 172.21.15.16:0/806323003 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50b806dea0 0x7f50b8147b30 secure :-1 s=READY pgs=2493 cs=0 l=1 rev1=1 crypto rx=0x7f50a800c4e0 tx=0x7f50a800c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:34.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.471+0000 7f50affff700 1 -- 172.21.15.16:0/806323003 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50a80052d0 con 0x7f50b806dea0 2023-10-16T13:46:34.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.471+0000 7f50affff700 1 -- 172.21.15.16:0/806323003 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50a8005430 con 0x7f50b806dea0 2023-10-16T13:46:34.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.471+0000 7f50bdf4f700 1 -- 172.21.15.16:0/806323003 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50b80b3ef0 con 0x7f50b806dea0 2023-10-16T13:46:34.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.471+0000 7f50affff700 1 -- 172.21.15.16:0/806323003 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50a80038c0 con 0x7f50b806dea0 2023-10-16T13:46:34.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.471+0000 7f50bdf4f700 1 -- 172.21.15.16:0/806323003 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50b80b44b0 con 0x7f50b806dea0 2023-10-16T13:46:34.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.472+0000 7f50bdf4f700 1 -- 172.21.15.16:0/806323003 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50b804ebb0 con 0x7f50b806dea0 2023-10-16T13:46:34.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.472+0000 7f50affff700 1 -- 172.21.15.16:0/806323003 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 138) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f50a8003a20 con 0x7f50b806dea0 2023-10-16T13:46:34.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.473+0000 7f50affff700 1 --2- 172.21.15.16:0/806323003 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f509806b020 0x7f509806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:34.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.473+0000 7f50affff700 1 -- 172.21.15.16:0/806323003 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(225..225 src has 1..225) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f50a8088ca0 con 0x7f50b806dea0 2023-10-16T13:46:34.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.473+0000 7f50b6ffd700 1 --2- 172.21.15.16:0/806323003 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f509806b020 0x7f509806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:34.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.473+0000 7f50b6ffd700 1 --2- 172.21.15.16:0/806323003 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f509806b020 0x7f509806d4e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f50a0009c10 tx=0x7f50a0008030 comp rx=0 tx=0).ready entity=mgr.11190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:34.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.477+0000 7f50affff700 1 -- 172.21.15.16:0/806323003 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f50a8053a80 con 0x7f50b806dea0 2023-10-16T13:46:34.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.686+0000 7f50bdf4f700 1 -- 172.21.15.16:0/806323003 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f50b806bd30 con 0x7f50b806dea0 2023-10-16T13:46:34.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.686+0000 7f50affff700 1 -- 172.21.15.16:0/806323003 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v225) v1 ==== 104+0+11 (secure 0 0 0) 0x7f50a8017020 con 0x7f50b806dea0 2023-10-16T13:46:34.689 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-10-16T13:46:34.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.689+0000 7f50bdf4f700 1 -- 172.21.15.16:0/806323003 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f509806b020 msgr2=0x7f509806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:34.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.689+0000 7f50bdf4f700 1 --2- 172.21.15.16:0/806323003 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f509806b020 0x7f509806d4e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f50a0009c10 tx=0x7f50a0008030 comp rx=0 tx=0).stop 2023-10-16T13:46:34.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.689+0000 7f50bdf4f700 1 -- 172.21.15.16:0/806323003 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50b806dea0 msgr2=0x7f50b8147b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:34.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.689+0000 7f50bdf4f700 1 --2- 172.21.15.16:0/806323003 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50b806dea0 0x7f50b8147b30 secure :-1 s=READY pgs=2493 cs=0 l=1 rev1=1 crypto rx=0x7f50a800c4e0 tx=0x7f50a800c7f0 comp rx=0 tx=0).stop 2023-10-16T13:46:34.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.689+0000 7f50bdf4f700 1 -- 172.21.15.16:0/806323003 shutdown_connections 2023-10-16T13:46:34.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.689+0000 7f50bdf4f700 1 --2- 172.21.15.16:0/806323003 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f50b815b790 0x7f50b80c3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:34.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.689+0000 7f50bdf4f700 1 --2- 172.21.15.16:0/806323003 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f509806b020 0x7f509806d4e0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:34.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.689+0000 7f50bdf4f700 1 --2- 172.21.15.16:0/806323003 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50b806dea0 0x7f50b8147b30 unknown :-1 s=CLOSED pgs=2493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:34.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.689+0000 7f50bdf4f700 1 --2- 172.21.15.16:0/806323003 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f50b80cda60 0x7f50b80b29b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:34.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.689+0000 7f50bdf4f700 1 -- 172.21.15.16:0/806323003 >> 172.21.15.16:0/806323003 conn(0x7f50b80cc5e0 msgr2=0x7f50b80cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:34.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.689+0000 7f50bdf4f700 1 -- 172.21.15.16:0/806323003 shutdown_connections 2023-10-16T13:46:34.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.689+0000 7f50bdf4f700 1 -- 172.21.15.16:0/806323003 wait complete. 2023-10-16T13:46:34.704 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T13:46:34.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.830+0000 7f2b249f9700 1 Processor -- start 2023-10-16T13:46:34.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.830+0000 7f2b249f9700 1 -- start start 2023-10-16T13:46:34.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.831+0000 7f2b249f9700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2b20160ee0 0x7f2b200aeb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:34.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.831+0000 7f2b249f9700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b200af4a0 con 0x7f2b200ce890 2023-10-16T13:46:34.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.831+0000 7f2b249f9700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b2006aa10 con 0x7f2b200af040 2023-10-16T13:46:34.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.831+0000 7f2b249f9700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b200cf770 con 0x7f2b20160ee0 2023-10-16T13:46:34.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.831+0000 7f2b1ed9d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2b200ce890 0x7f2b2006aea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42664/0 (socket says 172.21.15.16:42664) 2023-10-16T13:46:34.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.831+0000 7f2b1ed9d700 1 -- 172.21.15.16:0/3896067972 learned_addr learned my addr 172.21.15.16:0/3896067972 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:46:34.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.831+0000 7f2b1e59c700 1 --2- 172.21.15.16:0/3896067972 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2b20160ee0 0x7f2b200aeb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:34.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.831+0000 7f2b1d59a700 1 -- 172.21.15.16:0/3896067972 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3194230316 0 0) 0x7f2b200af4a0 con 0x7f2b200ce890 2023-10-16T13:46:34.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.831+0000 7f2b1d59a700 1 -- 172.21.15.16:0/3896067972 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b080035b0 con 0x7f2b200ce890 2023-10-16T13:46:34.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.832+0000 7f2b1d59a700 1 -- 172.21.15.16:0/3896067972 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3940999217 0 0) 0x7f2b080035b0 con 0x7f2b200ce890 2023-10-16T13:46:34.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.832+0000 7f2b1d59a700 1 -- 172.21.15.16:0/3896067972 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2b20160ee0 msgr2=0x7f2b200aeb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:34.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.832+0000 7f2b1d59a700 1 --2- 172.21.15.16:0/3896067972 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2b20160ee0 0x7f2b200aeb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:34.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.832+0000 7f2b1d59a700 1 -- 172.21.15.16:0/3896067972 >> v1:172.21.15.49:6789/0 conn(0x7f2b200af040 legacy=0x7f2b200ce180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:34.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.832+0000 7f2b1d59a700 1 -- 172.21.15.16:0/3896067972 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b200ceed0 con 0x7f2b200ce890 2023-10-16T13:46:34.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.832+0000 7f2b1d59a700 1 -- 172.21.15.16:0/3896067972 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f2b180019f0 con 0x7f2b200ce890 2023-10-16T13:46:34.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.832+0000 7f2b1d59a700 1 -- 172.21.15.16:0/3896067972 >> v1:172.21.15.16:6789/0 conn(0x7f2b200ce890 legacy=0x7f2b2006aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:34.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.832+0000 7f2b1d59a700 1 --2- 172.21.15.16:0/3896067972 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2b08003a30 0x7f2b08003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:34.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.832+0000 7f2b1d59a700 1 --2- 172.21.15.16:0/3896067972 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b08004850 0x7f2b08004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:34.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.832+0000 7f2b1d59a700 1 --2- 172.21.15.16:0/3896067972 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2b20160ee0 0x7f2b08005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:34.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.832+0000 7f2b1d59a700 1 -- 172.21.15.16:0/3896067972 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b200cf770 con 0x7f2b08004850 2023-10-16T13:46:34.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.832+0000 7f2b1d59a700 1 -- 172.21.15.16:0/3896067972 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b0800bac0 con 0x7f2b08003a30 2023-10-16T13:46:34.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.832+0000 7f2b1d59a700 1 -- 172.21.15.16:0/3896067972 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b0800bc80 con 0x7f2b20160ee0 2023-10-16T13:46:34.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.832+0000 7f2b1dd9b700 1 --2- 172.21.15.16:0/3896067972 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b08004850 0x7f2b08004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:34.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.832+0000 7f2b1e59c700 1 --2- 172.21.15.16:0/3896067972 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2b08003a30 0x7f2b08003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:34.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.832+0000 7f2b1ed9d700 1 --2- 172.21.15.16:0/3896067972 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2b20160ee0 0x7f2b08005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:34.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.832+0000 7f2b1dd9b700 1 -- 172.21.15.16:0/3896067972 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2b20160ee0 msgr2=0x7f2b08005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:34.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.832+0000 7f2b1dd9b700 1 --2- 172.21.15.16:0/3896067972 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2b20160ee0 0x7f2b08005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:34.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.832+0000 7f2b1dd9b700 1 -- 172.21.15.16:0/3896067972 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2b08003a30 msgr2=0x7f2b08003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:34.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.832+0000 7f2b1dd9b700 1 --2- 172.21.15.16:0/3896067972 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2b08003a30 0x7f2b08003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:34.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.832+0000 7f2b1dd9b700 1 -- 172.21.15.16:0/3896067972 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b080035b0 con 0x7f2b08004850 2023-10-16T13:46:34.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.832+0000 7f2b1e59c700 1 --2- 172.21.15.16:0/3896067972 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2b08003a30 0x7f2b08003e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:46:34.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.832+0000 7f2b1ed9d700 1 --2- 172.21.15.16:0/3896067972 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2b20160ee0 0x7f2b08005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:46:34.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.832+0000 7f2b1dd9b700 1 --2- 172.21.15.16:0/3896067972 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b08004850 0x7f2b08004cd0 secure :-1 s=READY pgs=2495 cs=0 l=1 rev1=1 crypto rx=0x7f2b10007fd0 tx=0x7f2b08005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ce9e96c6e125af3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:34.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.832+0000 7f2b1d59a700 1 -- 172.21.15.16:0/3896067972 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b100052d0 con 0x7f2b08004850 2023-10-16T13:46:34.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.833+0000 7f2b1d59a700 1 -- 172.21.15.16:0/3896067972 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b10005470 con 0x7f2b08004850 2023-10-16T13:46:34.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.833+0000 7f2b1d59a700 1 -- 172.21.15.16:0/3896067972 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b100038c0 con 0x7f2b08004850 2023-10-16T13:46:34.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.833+0000 7f2b249f9700 1 -- 172.21.15.16:0/3896067972 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b08004850 msgr2=0x7f2b08004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:34.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.833+0000 7f2b249f9700 1 --2- 172.21.15.16:0/3896067972 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b08004850 0x7f2b08004cd0 secure :-1 s=READY pgs=2495 cs=0 l=1 rev1=1 crypto rx=0x7f2b10007fd0 tx=0x7f2b08005880 comp rx=0 tx=0).stop 2023-10-16T13:46:34.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.833+0000 7f2b1ed9d700 1 -- 172.21.15.16:0/3896067972 reap_dead start 2023-10-16T13:46:34.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.833+0000 7f2b249f9700 1 -- 172.21.15.16:0/3896067972 shutdown_connections 2023-10-16T13:46:34.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.833+0000 7f2b249f9700 1 -- 172.21.15.16:0/3896067972 >> 172.21.15.16:0/3896067972 conn(0x7f2b200cc5e0 msgr2=0x7f2b200cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:34.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.833+0000 7f2b249f9700 1 -- 172.21.15.16:0/3896067972 shutdown_connections 2023-10-16T13:46:34.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.833+0000 7f2b249f9700 1 -- 172.21.15.16:0/3896067972 wait complete. 2023-10-16T13:46:34.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.834+0000 7f2b249f9700 1 Processor -- start 2023-10-16T13:46:34.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.835+0000 7f2b249f9700 1 -- start start 2023-10-16T13:46:34.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.835+0000 7f2b249f9700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2b08004850 0x7f2b20158300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:34.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.835+0000 7f2b249f9700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2b20157920 0x7f2b20155ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:34.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.835+0000 7f2b249f9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b200ad920 0x7f2b200addd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:34.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.835+0000 7f2b249f9700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b200ae310 con 0x7f2b200ad920 2023-10-16T13:46:34.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.835+0000 7f2b249f9700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b20157490 con 0x7f2b08004850 2023-10-16T13:46:34.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.835+0000 7f2b249f9700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b201589e0 con 0x7f2b20157920 2023-10-16T13:46:34.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.835+0000 7f2b1ed9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b200ad920 0x7f2b200addd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:34.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.835+0000 7f2b1ed9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b200ad920 0x7f2b200addd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53404/0 (socket says 172.21.15.16:53404) 2023-10-16T13:46:34.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.835+0000 7f2b1ed9d700 1 -- 172.21.15.16:0/2645629247 learned_addr learned my addr 172.21.15.16:0/2645629247 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:34.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.836+0000 7f2b1e59c700 1 --2- 172.21.15.16:0/2645629247 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2b08004850 0x7f2b20158300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:34.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.836+0000 7f2b1dd9b700 1 --2- 172.21.15.16:0/2645629247 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2b20157920 0x7f2b20155ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:34.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.836+0000 7f2b1ed9d700 1 -- 172.21.15.16:0/2645629247 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2b20157920 msgr2=0x7f2b20155ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:34.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.836+0000 7f2b1ed9d700 1 --2- 172.21.15.16:0/2645629247 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2b20157920 0x7f2b20155ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:34.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.836+0000 7f2b1ed9d700 1 -- 172.21.15.16:0/2645629247 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2b08004850 msgr2=0x7f2b20158300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:34.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.836+0000 7f2b1ed9d700 1 --2- 172.21.15.16:0/2645629247 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2b08004850 0x7f2b20158300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:34.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.836+0000 7f2b1ed9d700 1 -- 172.21.15.16:0/2645629247 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b20156b60 con 0x7f2b200ad920 2023-10-16T13:46:34.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.836+0000 7f2b1ed9d700 1 --2- 172.21.15.16:0/2645629247 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b200ad920 0x7f2b200addd0 secure :-1 s=READY pgs=2496 cs=0 l=1 rev1=1 crypto rx=0x7f2b200725b0 tx=0x7f2b18007c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:34.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.836+0000 7f2b0effd700 1 -- 172.21.15.16:0/2645629247 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b1800edc0 con 0x7f2b200ad920 2023-10-16T13:46:34.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.836+0000 7f2b0effd700 1 -- 172.21.15.16:0/2645629247 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b18013420 con 0x7f2b200ad920 2023-10-16T13:46:34.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.836+0000 7f2b0effd700 1 -- 172.21.15.16:0/2645629247 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b180137d0 con 0x7f2b200ad920 2023-10-16T13:46:34.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.836+0000 7f2b249f9700 1 -- 172.21.15.16:0/2645629247 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b20156e40 con 0x7f2b200ad920 2023-10-16T13:46:34.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.836+0000 7f2b249f9700 1 -- 172.21.15.16:0/2645629247 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b201561e0 con 0x7f2b200ad920 2023-10-16T13:46:34.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.837+0000 7f2b249f9700 1 -- 172.21.15.16:0/2645629247 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b2004ebb0 con 0x7f2b200ad920 2023-10-16T13:46:34.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.838+0000 7f2b0effd700 1 -- 172.21.15.16:0/2645629247 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 138) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2b18013930 con 0x7f2b200ad920 2023-10-16T13:46:34.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.838+0000 7f2b0effd700 1 --2- 172.21.15.16:0/2645629247 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f2b0006b020 0x7f2b0006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:34.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.838+0000 7f2b0effd700 1 -- 172.21.15.16:0/2645629247 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(225..225 src has 1..225) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f2b18089fb0 con 0x7f2b200ad920 2023-10-16T13:46:34.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.838+0000 7f2b1e59c700 1 --2- 172.21.15.16:0/2645629247 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f2b0006b020 0x7f2b0006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:34.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.839+0000 7f2b1e59c700 1 --2- 172.21.15.16:0/2645629247 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f2b0006b020 0x7f2b0006d4e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f2b14009b30 tx=0x7f2b14008040 comp rx=0 tx=0).ready entity=mgr.11190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:34.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:34.843+0000 7f2b0effd700 1 -- 172.21.15.16:0/2645629247 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2b180bcfe0 con 0x7f2b200ad920 2023-10-16T13:46:35.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.042+0000 7f2b249f9700 1 -- 172.21.15.16:0/2645629247 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f2b201575d0 con 0x7f2b200ad920 2023-10-16T13:46:35.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.043+0000 7f2b0effd700 1 -- 172.21.15.16:0/2645629247 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v225) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2b18054e20 con 0x7f2b200ad920 2023-10-16T13:46:35.046 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-10-16T13:46:35.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.045+0000 7f2b249f9700 1 -- 172.21.15.16:0/2645629247 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f2b0006b020 msgr2=0x7f2b0006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:35.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.045+0000 7f2b249f9700 1 --2- 172.21.15.16:0/2645629247 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f2b0006b020 0x7f2b0006d4e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f2b14009b30 tx=0x7f2b14008040 comp rx=0 tx=0).stop 2023-10-16T13:46:35.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.045+0000 7f2b249f9700 1 -- 172.21.15.16:0/2645629247 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b200ad920 msgr2=0x7f2b200addd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:35.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.045+0000 7f2b249f9700 1 --2- 172.21.15.16:0/2645629247 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b200ad920 0x7f2b200addd0 secure :-1 s=READY pgs=2496 cs=0 l=1 rev1=1 crypto rx=0x7f2b200725b0 tx=0x7f2b18007c90 comp rx=0 tx=0).stop 2023-10-16T13:46:35.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.045+0000 7f2b249f9700 1 -- 172.21.15.16:0/2645629247 shutdown_connections 2023-10-16T13:46:35.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.045+0000 7f2b249f9700 1 --2- 172.21.15.16:0/2645629247 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f2b0006b020 0x7f2b0006d4e0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:35.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.045+0000 7f2b249f9700 1 --2- 172.21.15.16:0/2645629247 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b200ad920 0x7f2b200addd0 unknown :-1 s=CLOSED pgs=2496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:35.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.045+0000 7f2b249f9700 1 --2- 172.21.15.16:0/2645629247 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2b08004850 0x7f2b20158300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:35.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.045+0000 7f2b249f9700 1 --2- 172.21.15.16:0/2645629247 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2b20157920 0x7f2b20155ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:35.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.045+0000 7f2b249f9700 1 -- 172.21.15.16:0/2645629247 >> 172.21.15.16:0/2645629247 conn(0x7f2b200cc5e0 msgr2=0x7f2b200cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:35.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.045+0000 7f2b249f9700 1 -- 172.21.15.16:0/2645629247 shutdown_connections 2023-10-16T13:46:35.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.045+0000 7f2b249f9700 1 -- 172.21.15.16:0/2645629247 wait complete. 2023-10-16T13:46:35.060 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-16T13:46:35.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.182+0000 7fedddae2700 1 Processor -- start 2023-10-16T13:46:35.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.182+0000 7fedddae2700 1 -- start start 2023-10-16T13:46:35.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.183+0000 7fedddae2700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fedd815db40 0x7fedd80cb980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:35.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.183+0000 7fedddae2700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fedd806a7e0 0x7fedd806ac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:35.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.183+0000 7fedddae2700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fedd80ce160 0x7fedd80ce5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:35.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.183+0000 7fedddae2700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedd806b1a0 con 0x7fedd815db40 2023-10-16T13:46:35.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.183+0000 7fedddae2700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedd815df20 con 0x7fedd806a7e0 2023-10-16T13:46:35.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.183+0000 7fedddae2700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedd80cedb0 con 0x7fedd80ce160 2023-10-16T13:46:35.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.183+0000 7fedd77fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fedd815db40 0x7fedd80cb980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:35.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.183+0000 7fedd77fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fedd815db40 0x7fedd80cb980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53408/0 (socket says 172.21.15.16:53408) 2023-10-16T13:46:35.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.183+0000 7fedd77fe700 1 -- 172.21.15.16:0/2697374727 learned_addr learned my addr 172.21.15.16:0/2697374727 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:35.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.183+0000 7fedd6ffd700 1 --2- 172.21.15.16:0/2697374727 >> v2:172.21.15.49:3300/0 conn(0x7fedd806a7e0 0x7fedd806ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:35.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.183+0000 7fedd7fff700 1 --2- 172.21.15.16:0/2697374727 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fedd80ce160 0x7fedd80ce5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:35.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.183+0000 7fedd77fe700 1 -- 172.21.15.16:0/2697374727 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fedd80ce160 msgr2=0x7fedd80ce5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:35.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.183+0000 7fedd77fe700 1 --2- 172.21.15.16:0/2697374727 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fedd80ce160 0x7fedd80ce5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:35.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.183+0000 7fedd77fe700 1 -- 172.21.15.16:0/2697374727 >> v2:172.21.15.49:3300/0 conn(0x7fedd806a7e0 msgr2=0x7fedd806ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:35.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.183+0000 7fedd77fe700 1 --2- 172.21.15.16:0/2697374727 >> v2:172.21.15.49:3300/0 conn(0x7fedd806a7e0 0x7fedd806ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:35.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.183+0000 7fedd77fe700 1 -- 172.21.15.16:0/2697374727 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedd80d12c0 con 0x7fedd815db40 2023-10-16T13:46:35.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.183+0000 7fedd7fff700 1 --2- 172.21.15.16:0/2697374727 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fedd80ce160 0x7fedd80ce5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:46:35.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.183+0000 7fedd77fe700 1 --2- 172.21.15.16:0/2697374727 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedd815db40 0x7fedd80cb980 secure :-1 s=READY pgs=2497 cs=0 l=1 rev1=1 crypto rx=0x7fedd8067660 tx=0x7fedc8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2605ba80ec189697 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:35.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.184+0000 7fedd67fc700 1 -- 172.21.15.16:0/2697374727 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedc8011830 con 0x7fedd815db40 2023-10-16T13:46:35.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.184+0000 7fedd67fc700 1 -- 172.21.15.16:0/2697374727 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedc8011990 con 0x7fedd815db40 2023-10-16T13:46:35.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.184+0000 7fedd67fc700 1 -- 172.21.15.16:0/2697374727 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedc8011d20 con 0x7fedd815db40 2023-10-16T13:46:35.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.184+0000 7fedddae2700 1 -- 172.21.15.16:0/2697374727 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedd815db40 msgr2=0x7fedd80cb980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:35.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.184+0000 7fedddae2700 1 --2- 172.21.15.16:0/2697374727 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedd815db40 0x7fedd80cb980 secure :-1 s=READY pgs=2497 cs=0 l=1 rev1=1 crypto rx=0x7fedd8067660 tx=0x7fedc8009bd0 comp rx=0 tx=0).stop 2023-10-16T13:46:35.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.184+0000 7fedddae2700 1 -- 172.21.15.16:0/2697374727 shutdown_connections 2023-10-16T13:46:35.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.184+0000 7fedddae2700 1 --2- 172.21.15.16:0/2697374727 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fedd80ce160 0x7fedd80ce5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:35.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.184+0000 7fedddae2700 1 --2- 172.21.15.16:0/2697374727 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedd815db40 0x7fedd80cb980 unknown :-1 s=CLOSED pgs=2497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:35.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.184+0000 7fedddae2700 1 --2- 172.21.15.16:0/2697374727 >> v2:172.21.15.49:3300/0 conn(0x7fedd806a7e0 0x7fedd806ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:35.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.184+0000 7fedddae2700 1 -- 172.21.15.16:0/2697374727 >> 172.21.15.16:0/2697374727 conn(0x7fedd80b1fd0 msgr2=0x7fedd80b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:35.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.184+0000 7fedddae2700 1 -- 172.21.15.16:0/2697374727 shutdown_connections 2023-10-16T13:46:35.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.184+0000 7fedddae2700 1 -- 172.21.15.16:0/2697374727 wait complete. 2023-10-16T13:46:35.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.185+0000 7fedddae2700 1 Processor -- start 2023-10-16T13:46:35.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.186+0000 7fedddae2700 1 -- start start 2023-10-16T13:46:35.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.186+0000 7fedddae2700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fedd806a7e0 0x7fedd80c7510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:35.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.186+0000 7fedddae2700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fedd80ce160 0x7fedd8158bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:35.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.186+0000 7fedddae2700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedd815db40 0x7fedd80ae2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:35.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.186+0000 7fedddae2700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedd80c9ba0 con 0x7fedd815db40 2023-10-16T13:46:35.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.186+0000 7fedddae2700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedd80ae830 con 0x7fedd80ce160 2023-10-16T13:46:35.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.186+0000 7fedddae2700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedd80c7b10 con 0x7fedd806a7e0 2023-10-16T13:46:35.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.186+0000 7fedd7fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedd815db40 0x7fedd80ae2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:35.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.186+0000 7fedd7fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedd815db40 0x7fedd80ae2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53418/0 (socket says 172.21.15.16:53418) 2023-10-16T13:46:35.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.186+0000 7fedd7fff700 1 -- 172.21.15.16:0/877802894 learned_addr learned my addr 172.21.15.16:0/877802894 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:35.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.186+0000 7fedd6ffd700 1 --2- 172.21.15.16:0/877802894 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fedd80ce160 0x7fedd8158bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:35.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.186+0000 7fedd77fe700 1 --2- 172.21.15.16:0/877802894 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fedd806a7e0 0x7fedd80c7510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:35.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.187+0000 7fedd7fff700 1 -- 172.21.15.16:0/877802894 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fedd806a7e0 msgr2=0x7fedd80c7510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:35.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.187+0000 7fedd7fff700 1 --2- 172.21.15.16:0/877802894 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fedd806a7e0 0x7fedd80c7510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:35.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.187+0000 7fedd7fff700 1 -- 172.21.15.16:0/877802894 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fedd80ce160 msgr2=0x7fedd8158bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:35.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.187+0000 7fedd7fff700 1 --2- 172.21.15.16:0/877802894 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fedd80ce160 0x7fedd8158bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:35.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.187+0000 7fedd7fff700 1 -- 172.21.15.16:0/877802894 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedc8009550 con 0x7fedd815db40 2023-10-16T13:46:35.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.187+0000 7fedd7fff700 1 --2- 172.21.15.16:0/877802894 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedd815db40 0x7fedd80ae2f0 secure :-1 s=READY pgs=2498 cs=0 l=1 rev1=1 crypto rx=0x7fedd0007d40 tx=0x7fedd000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:35.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.187+0000 7fedbffff700 1 -- 172.21.15.16:0/877802894 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd000dd20 con 0x7fedd815db40 2023-10-16T13:46:35.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.187+0000 7fedbffff700 1 -- 172.21.15.16:0/877802894 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedd000f040 con 0x7fedd815db40 2023-10-16T13:46:35.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.187+0000 7fedddae2700 1 -- 172.21.15.16:0/877802894 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedd80c7df0 con 0x7fedd815db40 2023-10-16T13:46:35.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.187+0000 7fedbffff700 1 -- 172.21.15.16:0/877802894 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd0003c60 con 0x7fedd815db40 2023-10-16T13:46:35.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.187+0000 7fedddae2700 1 -- 172.21.15.16:0/877802894 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedd80c83b0 con 0x7fedd815db40 2023-10-16T13:46:35.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.188+0000 7fedddae2700 1 -- 172.21.15.16:0/877802894 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedd804ebb0 con 0x7fedd815db40 2023-10-16T13:46:35.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.189+0000 7fedbffff700 1 -- 172.21.15.16:0/877802894 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 138) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fedd0004030 con 0x7fedd815db40 2023-10-16T13:46:35.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.189+0000 7fedbffff700 1 --2- 172.21.15.16:0/877802894 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fedc406b020 0x7fedc406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:35.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.189+0000 7fedbffff700 1 -- 172.21.15.16:0/877802894 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(225..225 src has 1..225) v4 ==== 6157+0+0 (secure 0 0 0) 0x7fedd0089900 con 0x7fedd815db40 2023-10-16T13:46:35.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.189+0000 7fedd77fe700 1 --2- 172.21.15.16:0/877802894 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fedc406b020 0x7fedc406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:35.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.189+0000 7fedd77fe700 1 --2- 172.21.15.16:0/877802894 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fedc406b020 0x7fedc406d4e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fedc8009c10 tx=0x7fedc8008030 comp rx=0 tx=0).ready entity=mgr.11190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:35.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.194+0000 7fedbffff700 1 -- 172.21.15.16:0/877802894 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fedd0054660 con 0x7fedd815db40 2023-10-16T13:46:35.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.403+0000 7fedddae2700 1 -- 172.21.15.16:0/877802894 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fedd80cb150 con 0x7fedd815db40 2023-10-16T13:46:35.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.403+0000 7fedbffff700 1 -- 172.21.15.16:0/877802894 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v225) v1 ==== 84+0+209 (secure 0 0 0) 0x7fedd001a090 con 0x7fedd815db40 2023-10-16T13:46:35.407 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:46:35.407 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2023-10-16T13:46:35.407 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:46:35.407 INFO:teuthology.orchestra.run.smithi016.stdout: "poolnum": 1, 2023-10-16T13:46:35.407 INFO:teuthology.orchestra.run.smithi016.stdout: "poolname": ".mgr" 2023-10-16T13:46:35.408 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:46:35.408 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:46:35.408 INFO:teuthology.orchestra.run.smithi016.stdout: "poolnum": 27, 2023-10-16T13:46:35.408 INFO:teuthology.orchestra.run.smithi016.stdout: "poolname": "cephfs_metadata" 2023-10-16T13:46:35.408 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:46:35.409 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:46:35.409 INFO:teuthology.orchestra.run.smithi016.stdout: "poolnum": 28, 2023-10-16T13:46:35.409 INFO:teuthology.orchestra.run.smithi016.stdout: "poolname": "cephfs_data" 2023-10-16T13:46:35.409 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:46:35.409 INFO:teuthology.orchestra.run.smithi016.stdout:] 2023-10-16T13:46:35.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.406+0000 7fedddae2700 1 -- 172.21.15.16:0/877802894 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fedc406b020 msgr2=0x7fedc406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:35.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.406+0000 7fedddae2700 1 --2- 172.21.15.16:0/877802894 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fedc406b020 0x7fedc406d4e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fedc8009c10 tx=0x7fedc8008030 comp rx=0 tx=0).stop 2023-10-16T13:46:35.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.406+0000 7fedddae2700 1 -- 172.21.15.16:0/877802894 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedd815db40 msgr2=0x7fedd80ae2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:35.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.406+0000 7fedddae2700 1 --2- 172.21.15.16:0/877802894 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedd815db40 0x7fedd80ae2f0 secure :-1 s=READY pgs=2498 cs=0 l=1 rev1=1 crypto rx=0x7fedd0007d40 tx=0x7fedd000d3b0 comp rx=0 tx=0).stop 2023-10-16T13:46:35.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.406+0000 7fedddae2700 1 -- 172.21.15.16:0/877802894 shutdown_connections 2023-10-16T13:46:35.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.406+0000 7fedddae2700 1 --2- 172.21.15.16:0/877802894 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fedc406b020 0x7fedc406d4e0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:35.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.406+0000 7fedddae2700 1 --2- 172.21.15.16:0/877802894 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedd815db40 0x7fedd80ae2f0 unknown :-1 s=CLOSED pgs=2498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:35.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.406+0000 7fedddae2700 1 --2- 172.21.15.16:0/877802894 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fedd806a7e0 0x7fedd80c7510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:35.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.406+0000 7fedddae2700 1 --2- 172.21.15.16:0/877802894 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fedd80ce160 0x7fedd8158bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:35.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.406+0000 7fedddae2700 1 -- 172.21.15.16:0/877802894 >> 172.21.15.16:0/877802894 conn(0x7fedd80b1fd0 msgr2=0x7fedd80b0320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:35.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.406+0000 7fedddae2700 1 -- 172.21.15.16:0/877802894 shutdown_connections 2023-10-16T13:46:35.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.406+0000 7fedddae2700 1 -- 172.21.15.16:0/877802894 wait complete. 2023-10-16T13:46:35.421 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'backup_fs' 2023-10-16T13:46:35.421 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create backup_fs_metadata --pg_num_min 64 2023-10-16T13:46:35.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.547+0000 7fba584c8700 1 Processor -- start 2023-10-16T13:46:35.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.548+0000 7fba584c8700 1 -- start start 2023-10-16T13:46:35.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.548+0000 7fba584c8700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fba5006d980 0x7fba5015efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:35.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.549+0000 7fba584c8700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba500cdeb0 con 0x7fba50160e60 2023-10-16T13:46:35.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.549+0000 7fba584c8700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba501613e0 con 0x7fba5006a7e0 2023-10-16T13:46:35.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.549+0000 7fba584c8700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba501582d0 con 0x7fba5006d980 2023-10-16T13:46:35.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.549+0000 7fba56264700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fba50160e60 0x7fba5015e870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42696/0 (socket says 172.21.15.16:42696) 2023-10-16T13:46:35.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.549+0000 7fba56264700 1 -- 172.21.15.16:0/2010316598 learned_addr learned my addr 172.21.15.16:0/2010316598 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:46:35.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.549+0000 7fba55a63700 1 --2- 172.21.15.16:0/2010316598 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fba5006d980 0x7fba5015efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:35.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.549+0000 7fba55262700 1 -- 172.21.15.16:0/2010316598 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3624492184 0 0) 0x7fba500cdeb0 con 0x7fba50160e60 2023-10-16T13:46:35.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.549+0000 7fba55262700 1 -- 172.21.15.16:0/2010316598 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba3c0035b0 con 0x7fba50160e60 2023-10-16T13:46:35.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.549+0000 7fba55262700 1 -- 172.21.15.16:0/2010316598 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3069314464 0 0) 0x7fba3c0035b0 con 0x7fba50160e60 2023-10-16T13:46:35.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.549+0000 7fba55262700 1 -- 172.21.15.16:0/2010316598 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fba5006d980 msgr2=0x7fba5015efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:35.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.549+0000 7fba55262700 1 --2- 172.21.15.16:0/2010316598 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fba5006d980 0x7fba5015efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:35.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.549+0000 7fba55262700 1 -- 172.21.15.16:0/2010316598 >> v1:172.21.15.49:6789/0 conn(0x7fba5006a7e0 legacy=0x7fba50161870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:35.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.549+0000 7fba55262700 1 -- 172.21.15.16:0/2010316598 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba500ce100 con 0x7fba50160e60 2023-10-16T13:46:35.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.549+0000 7fba55262700 1 -- 172.21.15.16:0/2010316598 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7fba4c001670 con 0x7fba50160e60 2023-10-16T13:46:35.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.549+0000 7fba55262700 1 -- 172.21.15.16:0/2010316598 >> v1:172.21.15.16:6789/0 conn(0x7fba50160e60 legacy=0x7fba5015e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:35.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.549+0000 7fba55262700 1 --2- 172.21.15.16:0/2010316598 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba3c003a30 0x7fba3c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:35.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.549+0000 7fba55262700 1 --2- 172.21.15.16:0/2010316598 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fba3c0043e0 0x7fba3c0048b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:35.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.549+0000 7fba55262700 1 --2- 172.21.15.16:0/2010316598 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fba5006d980 0x7fba3c005110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:35.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.549+0000 7fba55262700 1 -- 172.21.15.16:0/2010316598 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba501582d0 con 0x7fba3c003a30 2023-10-16T13:46:35.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.549+0000 7fba55262700 1 -- 172.21.15.16:0/2010316598 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba3c00ba40 con 0x7fba3c0043e0 2023-10-16T13:46:35.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.549+0000 7fba55262700 1 -- 172.21.15.16:0/2010316598 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba3c00bc00 con 0x7fba5006d980 2023-10-16T13:46:35.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.549+0000 7fba56264700 1 --2- 172.21.15.16:0/2010316598 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba3c003a30 0x7fba3c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:35.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.550+0000 7fba55a63700 1 --2- 172.21.15.16:0/2010316598 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fba3c0043e0 0x7fba3c0048b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:35.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.550+0000 7fba56a65700 1 --2- 172.21.15.16:0/2010316598 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fba5006d980 0x7fba3c005110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:35.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.550+0000 7fba56264700 1 -- 172.21.15.16:0/2010316598 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fba5006d980 msgr2=0x7fba3c005110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:35.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.550+0000 7fba56264700 1 --2- 172.21.15.16:0/2010316598 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fba5006d980 0x7fba3c005110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:35.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.550+0000 7fba56264700 1 -- 172.21.15.16:0/2010316598 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fba3c0043e0 msgr2=0x7fba3c0048b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:35.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.550+0000 7fba56264700 1 --2- 172.21.15.16:0/2010316598 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fba3c0043e0 0x7fba3c0048b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:35.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.550+0000 7fba56264700 1 -- 172.21.15.16:0/2010316598 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba3c0035b0 con 0x7fba3c003a30 2023-10-16T13:46:35.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.550+0000 7fba56264700 1 --2- 172.21.15.16:0/2010316598 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba3c003a30 0x7fba3c003e10 secure :-1 s=READY pgs=2500 cs=0 l=1 rev1=1 crypto rx=0x7fba4c00c2f0 tx=0x7fba3c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee8473cd398f270d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:35.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.550+0000 7fba55262700 1 -- 172.21.15.16:0/2010316598 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba4c00ce30 con 0x7fba3c003a30 2023-10-16T13:46:35.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.550+0000 7fba55262700 1 -- 172.21.15.16:0/2010316598 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba4c005960 con 0x7fba3c003a30 2023-10-16T13:46:35.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.550+0000 7fba55262700 1 -- 172.21.15.16:0/2010316598 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba4c005d10 con 0x7fba3c003a30 2023-10-16T13:46:35.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.551+0000 7fba584c8700 1 -- 172.21.15.16:0/2010316598 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba3c003a30 msgr2=0x7fba3c003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:35.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.551+0000 7fba584c8700 1 --2- 172.21.15.16:0/2010316598 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba3c003a30 0x7fba3c003e10 secure :-1 s=READY pgs=2500 cs=0 l=1 rev1=1 crypto rx=0x7fba4c00c2f0 tx=0x7fba3c005880 comp rx=0 tx=0).stop 2023-10-16T13:46:35.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.551+0000 7fba56a65700 1 -- 172.21.15.16:0/2010316598 reap_dead start 2023-10-16T13:46:35.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.551+0000 7fba584c8700 1 -- 172.21.15.16:0/2010316598 shutdown_connections 2023-10-16T13:46:35.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.551+0000 7fba584c8700 1 -- 172.21.15.16:0/2010316598 >> 172.21.15.16:0/2010316598 conn(0x7fba5006f430 msgr2=0x7fba5006f810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:35.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.551+0000 7fba584c8700 1 -- 172.21.15.16:0/2010316598 shutdown_connections 2023-10-16T13:46:35.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.551+0000 7fba584c8700 1 -- 172.21.15.16:0/2010316598 wait complete. 2023-10-16T13:46:35.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.552+0000 7fba584c8700 1 Processor -- start 2023-10-16T13:46:35.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.552+0000 7fba584c8700 1 -- start start 2023-10-16T13:46:35.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.553+0000 7fba584c8700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fba3c003a30 0x7fba500aefd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:35.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.553+0000 7fba584c8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba500b4640 0x7fba5014d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:35.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.553+0000 7fba584c8700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fba50155cc0 0x7fba50156170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:35.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.553+0000 7fba584c8700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba500af510 con 0x7fba500b4640 2023-10-16T13:46:35.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.553+0000 7fba584c8700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba5014d610 con 0x7fba50155cc0 2023-10-16T13:46:35.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.553+0000 7fba584c8700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba501566b0 con 0x7fba3c003a30 2023-10-16T13:46:35.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.553+0000 7fba55a63700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba500b4640 0x7fba5014d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:35.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.553+0000 7fba55a63700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba500b4640 0x7fba5014d0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53436/0 (socket says 172.21.15.16:53436) 2023-10-16T13:46:35.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.553+0000 7fba55a63700 1 -- 172.21.15.16:0/407506227 learned_addr learned my addr 172.21.15.16:0/407506227 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:35.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.554+0000 7fba56264700 1 --2- 172.21.15.16:0/407506227 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fba3c003a30 0x7fba500aefd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:35.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.554+0000 7fba56a65700 1 --2- 172.21.15.16:0/407506227 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fba50155cc0 0x7fba50156170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:35.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.554+0000 7fba55a63700 1 -- 172.21.15.16:0/407506227 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fba3c003a30 msgr2=0x7fba500aefd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:35.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.554+0000 7fba55a63700 1 --2- 172.21.15.16:0/407506227 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fba3c003a30 0x7fba500aefd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:35.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.554+0000 7fba55a63700 1 -- 172.21.15.16:0/407506227 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fba50155cc0 msgr2=0x7fba50156170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:35.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.554+0000 7fba55a63700 1 --2- 172.21.15.16:0/407506227 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fba50155cc0 0x7fba50156170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:35.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.554+0000 7fba55a63700 1 -- 172.21.15.16:0/407506227 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba500b4db0 con 0x7fba500b4640 2023-10-16T13:46:35.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.554+0000 7fba55a63700 1 --2- 172.21.15.16:0/407506227 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba500b4640 0x7fba5014d0d0 secure :-1 s=READY pgs=2501 cs=0 l=1 rev1=1 crypto rx=0x7fba440119d0 tx=0x7fba44011ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:35.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.554+0000 7fba42ffd700 1 -- 172.21.15.16:0/407506227 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba44011e70 con 0x7fba500b4640 2023-10-16T13:46:35.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.554+0000 7fba42ffd700 1 -- 172.21.15.16:0/407506227 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba4400f940 con 0x7fba500b4640 2023-10-16T13:46:35.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.554+0000 7fba584c8700 1 -- 172.21.15.16:0/407506227 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba500b5090 con 0x7fba500b4640 2023-10-16T13:46:35.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.554+0000 7fba42ffd700 1 -- 172.21.15.16:0/407506227 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba4400fcd0 con 0x7fba500b4640 2023-10-16T13:46:35.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.554+0000 7fba584c8700 1 -- 172.21.15.16:0/407506227 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba500b5650 con 0x7fba500b4640 2023-10-16T13:46:35.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.555+0000 7fba584c8700 1 -- 172.21.15.16:0/407506227 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba5004ec20 con 0x7fba500b4640 2023-10-16T13:46:35.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.556+0000 7fba42ffd700 1 -- 172.21.15.16:0/407506227 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 138) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fba44014440 con 0x7fba500b4640 2023-10-16T13:46:35.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.556+0000 7fba42ffd700 1 --2- 172.21.15.16:0/407506227 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fba3406b020 0x7fba3406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:35.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.556+0000 7fba42ffd700 1 -- 172.21.15.16:0/407506227 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(225..225 src has 1..225) v4 ==== 6157+0+0 (secure 0 0 0) 0x7fba44089e10 con 0x7fba500b4640 2023-10-16T13:46:35.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.556+0000 7fba56264700 1 --2- 172.21.15.16:0/407506227 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fba3406b020 0x7fba3406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:35.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.557+0000 7fba56264700 1 --2- 172.21.15.16:0/407506227 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fba3406b020 0x7fba3406d4e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fba501f6600 tx=0x7fba4c007000 comp rx=0 tx=0).ready entity=mgr.11190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:35.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.561+0000 7fba42ffd700 1 -- 172.21.15.16:0/407506227 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fba44054bf0 con 0x7fba500b4640 2023-10-16T13:46:35.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:35.766+0000 7fba584c8700 1 -- 172.21.15.16:0/407506227 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "backup_fs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fba500c6d70 con 0x7fba500b4640 2023-10-16T13:46:36.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:36.621+0000 7fba42ffd700 1 -- 172.21.15.16:0/407506227 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "backup_fs_metadata", "pg_num_min": 64}]=0 pool 'backup_fs_metadata' created v226) v1 ==== 144+0+0 (secure 0 0 0) 0x7fba44058210 con 0x7fba500b4640 2023-10-16T13:46:36.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:36.625+0000 7fba584c8700 1 -- 172.21.15.16:0/407506227 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fba3406b020 msgr2=0x7fba3406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:36.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:36.625+0000 7fba584c8700 1 --2- 172.21.15.16:0/407506227 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fba3406b020 0x7fba3406d4e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fba501f6600 tx=0x7fba4c007000 comp rx=0 tx=0).stop 2023-10-16T13:46:36.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:36.625+0000 7fba584c8700 1 -- 172.21.15.16:0/407506227 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba500b4640 msgr2=0x7fba5014d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:36.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:36.625+0000 7fba584c8700 1 --2- 172.21.15.16:0/407506227 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba500b4640 0x7fba5014d0d0 secure :-1 s=READY pgs=2501 cs=0 l=1 rev1=1 crypto rx=0x7fba440119d0 tx=0x7fba44011ce0 comp rx=0 tx=0).stop 2023-10-16T13:46:36.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:36.625+0000 7fba584c8700 1 -- 172.21.15.16:0/407506227 shutdown_connections 2023-10-16T13:46:36.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:36.625+0000 7fba584c8700 1 --2- 172.21.15.16:0/407506227 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fba50155cc0 0x7fba50156170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:36.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:36.625+0000 7fba584c8700 1 --2- 172.21.15.16:0/407506227 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fba3c003a30 0x7fba500aefd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:36.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:36.625+0000 7fba584c8700 1 --2- 172.21.15.16:0/407506227 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fba3406b020 0x7fba3406d4e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:36.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:36.625+0000 7fba584c8700 1 --2- 172.21.15.16:0/407506227 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba500b4640 0x7fba5014d0d0 unknown :-1 s=CLOSED pgs=2501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:36.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:36.625+0000 7fba584c8700 1 -- 172.21.15.16:0/407506227 >> 172.21.15.16:0/407506227 conn(0x7fba5006f430 msgr2=0x7fba50157480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:36.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:36.626+0000 7fba584c8700 1 -- 172.21.15.16:0/407506227 shutdown_connections 2023-10-16T13:46:36.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:36.627+0000 7fba584c8700 1 -- 172.21.15.16:0/407506227 wait complete. 2023-10-16T13:46:36.631 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'backup_fs_metadata' created 2023-10-16T13:46:36.647 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create backup_fs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-16T13:46:36.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:36.783+0000 7f8f7751b700 1 Processor -- start 2023-10-16T13:46:36.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:36.783+0000 7f8f7751b700 1 -- start start 2023-10-16T13:46:36.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:36.784+0000 7f8f7751b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8f700d2110 0x7f8f700ae1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:36.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:36.784+0000 7f8f7751b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8f7006ca10 0x7f8f700ae720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:36.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:36.784+0000 7f8f7751b700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f700b20f0 con 0x7f8f700ce6a0 2023-10-16T13:46:36.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:36.784+0000 7f8f7751b700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f700d03d0 con 0x7f8f700d2110 2023-10-16T13:46:36.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:36.784+0000 7f8f7751b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f700b2340 con 0x7f8f7006ca10 2023-10-16T13:46:36.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:36.784+0000 7f8f74ab6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8f700d2110 0x7f8f700ae1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:36.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:36.784+0000 7f8f74ab6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8f700d2110 0x7f8f700ae1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53444/0 (socket says 172.21.15.16:53444) 2023-10-16T13:46:36.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:36.784+0000 7f8f752b7700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f8f700ce6a0 0x7f8f7006c300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42716/0 (socket says 172.21.15.16:42716) 2023-10-16T13:46:36.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:36.784+0000 7f8f74ab6700 1 -- 172.21.15.16:0/712588618 learned_addr learned my addr 172.21.15.16:0/712588618 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:36.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:36.784+0000 7f8f75ab8700 1 --2- 172.21.15.16:0/712588618 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8f7006ca10 0x7f8f700ae720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:36.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:36.784+0000 7f8f74ab6700 1 -- 172.21.15.16:0/712588618 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8f7006ca10 msgr2=0x7f8f700ae720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:36.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:36.784+0000 7f8f74ab6700 1 --2- 172.21.15.16:0/712588618 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8f7006ca10 0x7f8f700ae720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:36.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:36.784+0000 7f8f74ab6700 1 -- 172.21.15.16:0/712588618 >> v1:172.21.15.16:6789/0 conn(0x7f8f700ce6a0 legacy=0x7f8f7006c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:36.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:36.784+0000 7f8f67fff700 1 -- 172.21.15.16:0/712588618 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 986571676 0 0) 0x7f8f700b20f0 con 0x7f8f700ce6a0 2023-10-16T13:46:36.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:36.784+0000 7f8f74ab6700 1 -- 172.21.15.16:0/712588618 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f7006bb00 con 0x7f8f700d2110 2023-10-16T13:46:36.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:36.785+0000 7f8f74ab6700 1 --2- 172.21.15.16:0/712588618 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f700d2110 0x7f8f700ae1e0 secure :-1 s=READY pgs=2503 cs=0 l=1 rev1=1 crypto rx=0x7f8f6c0098f0 tx=0x7f8f6c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37a0436516d0e68d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:36.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:36.785+0000 7f8f67fff700 1 -- 172.21.15.16:0/712588618 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f6c00b8a0 con 0x7f8f700d2110 2023-10-16T13:46:36.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:36.785+0000 7f8f67fff700 1 -- 172.21.15.16:0/712588618 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f6c00ba00 con 0x7f8f700d2110 2023-10-16T13:46:36.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:36.785+0000 7f8f67fff700 1 -- 172.21.15.16:0/712588618 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f6c00bd90 con 0x7f8f700d2110 2023-10-16T13:46:36.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:36.785+0000 7f8f7751b700 1 -- 172.21.15.16:0/712588618 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f700d2110 msgr2=0x7f8f700ae1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:36.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:36.785+0000 7f8f7751b700 1 --2- 172.21.15.16:0/712588618 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f700d2110 0x7f8f700ae1e0 secure :-1 s=READY pgs=2503 cs=0 l=1 rev1=1 crypto rx=0x7f8f6c0098f0 tx=0x7f8f6c009c00 comp rx=0 tx=0).stop 2023-10-16T13:46:36.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:36.785+0000 7f8f7751b700 1 -- 172.21.15.16:0/712588618 shutdown_connections 2023-10-16T13:46:36.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:36.785+0000 7f8f7751b700 1 --2- 172.21.15.16:0/712588618 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8f7006ca10 0x7f8f700ae720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:36.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:36.785+0000 7f8f7751b700 1 --2- 172.21.15.16:0/712588618 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f700d2110 0x7f8f700ae1e0 unknown :-1 s=CLOSED pgs=2503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:36.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:36.785+0000 7f8f7751b700 1 -- 172.21.15.16:0/712588618 >> 172.21.15.16:0/712588618 conn(0x7f8f700d15f0 msgr2=0x7f8f700d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:36.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:36.785+0000 7f8f7751b700 1 -- 172.21.15.16:0/712588618 shutdown_connections 2023-10-16T13:46:36.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:36.785+0000 7f8f7751b700 1 -- 172.21.15.16:0/712588618 wait complete. 2023-10-16T13:46:36.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:36.787+0000 7f8f7751b700 1 Processor -- start 2023-10-16T13:46:36.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:36.787+0000 7f8f7751b700 1 -- start start 2023-10-16T13:46:36.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:36.787+0000 7f8f7751b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f7006ca10 0x7f8f7015d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:36.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:36.787+0000 7f8f7751b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8f700ce6a0 0x7f8f700ae450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:36.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:36.787+0000 7f8f7751b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8f700d2110 0x7f8f700c9e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:36.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:36.787+0000 7f8f7751b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f700b4e70 con 0x7f8f7006ca10 2023-10-16T13:46:36.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:36.787+0000 7f8f7751b700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f700b42a0 con 0x7f8f700d2110 2023-10-16T13:46:36.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:36.787+0000 7f8f7751b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f700b4540 con 0x7f8f700ce6a0 2023-10-16T13:46:36.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:36.787+0000 7f8f752b7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f7006ca10 0x7f8f7015d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:36.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:36.787+0000 7f8f752b7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f7006ca10 0x7f8f7015d790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53448/0 (socket says 172.21.15.16:53448) 2023-10-16T13:46:36.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:36.787+0000 7f8f752b7700 1 -- 172.21.15.16:0/1153905084 learned_addr learned my addr 172.21.15.16:0/1153905084 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:36.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:36.788+0000 7f8f74ab6700 1 --2- 172.21.15.16:0/1153905084 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8f700ce6a0 0x7f8f700ae450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:36.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:36.788+0000 7f8f75ab8700 1 --2- 172.21.15.16:0/1153905084 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8f700d2110 0x7f8f700c9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:36.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:36.788+0000 7f8f752b7700 1 -- 172.21.15.16:0/1153905084 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8f700ce6a0 msgr2=0x7f8f700ae450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:36.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:36.788+0000 7f8f752b7700 1 --2- 172.21.15.16:0/1153905084 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8f700ce6a0 0x7f8f700ae450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:36.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:36.788+0000 7f8f752b7700 1 -- 172.21.15.16:0/1153905084 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8f700d2110 msgr2=0x7f8f700c9e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:36.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:36.788+0000 7f8f752b7700 1 --2- 172.21.15.16:0/1153905084 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8f700d2110 0x7f8f700c9e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:36.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:36.788+0000 7f8f752b7700 1 -- 172.21.15.16:0/1153905084 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f6c009550 con 0x7f8f7006ca10 2023-10-16T13:46:36.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:36.788+0000 7f8f74ab6700 1 --2- 172.21.15.16:0/1153905084 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8f700ce6a0 0x7f8f700ae450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:46:36.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:36.788+0000 7f8f752b7700 1 --2- 172.21.15.16:0/1153905084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f7006ca10 0x7f8f7015d790 secure :-1 s=READY pgs=2504 cs=0 l=1 rev1=1 crypto rx=0x7f8f600011e0 tx=0x7f8f6000cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:36.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:36.788+0000 7f8f65ffb700 1 -- 172.21.15.16:0/1153905084 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f60004c80 con 0x7f8f7006ca10 2023-10-16T13:46:36.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:36.788+0000 7f8f65ffb700 1 -- 172.21.15.16:0/1153905084 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f60004e00 con 0x7f8f7006ca10 2023-10-16T13:46:36.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:36.788+0000 7f8f7751b700 1 -- 172.21.15.16:0/1153905084 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f700b4820 con 0x7f8f7006ca10 2023-10-16T13:46:36.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:36.788+0000 7f8f65ffb700 1 -- 172.21.15.16:0/1153905084 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f60007310 con 0x7f8f7006ca10 2023-10-16T13:46:36.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:36.788+0000 7f8f7751b700 1 -- 172.21.15.16:0/1153905084 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f7015dcd0 con 0x7f8f7006ca10 2023-10-16T13:46:36.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:36.789+0000 7f8f7751b700 1 -- 172.21.15.16:0/1153905084 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f7004ebb0 con 0x7f8f7006ca10 2023-10-16T13:46:36.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:36.790+0000 7f8f65ffb700 1 -- 172.21.15.16:0/1153905084 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 138) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8f60007470 con 0x7f8f7006ca10 2023-10-16T13:46:36.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:36.790+0000 7f8f65ffb700 1 --2- 172.21.15.16:0/1153905084 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f8f5406b020 0x7f8f5406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:36.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:36.790+0000 7f8f65ffb700 1 -- 172.21.15.16:0/1153905084 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(226..226 src has 1..226) v4 ==== 6530+0+0 (secure 0 0 0) 0x7f8f60089880 con 0x7f8f7006ca10 2023-10-16T13:46:36.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:36.791+0000 7f8f74ab6700 1 --2- 172.21.15.16:0/1153905084 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f8f5406b020 0x7f8f5406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:36.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:36.791+0000 7f8f74ab6700 1 --2- 172.21.15.16:0/1153905084 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f8f5406b020 0x7f8f5406d4e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8f6c0098f0 tx=0x7f8f6c008030 comp rx=0 tx=0).ready entity=mgr.11190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:36.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:36.795+0000 7f8f65ffb700 1 -- 172.21.15.16:0/1153905084 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8f600544d0 con 0x7f8f7006ca10 2023-10-16T13:46:37.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.000+0000 7f8f7751b700 1 -- 172.21.15.16:0/1153905084 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "backup_fs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f8f700b1e60 con 0x7f8f7006ca10 2023-10-16T13:46:37.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.623+0000 7f8f65ffb700 1 -- 172.21.15.16:0/1153905084 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "backup_fs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'backup_fs_data' created v227) v1 ==== 176+0+0 (secure 0 0 0) 0x7f8f60057af0 con 0x7f8f7006ca10 2023-10-16T13:46:37.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.628+0000 7f8f7751b700 1 -- 172.21.15.16:0/1153905084 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f8f5406b020 msgr2=0x7f8f5406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:37.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.628+0000 7f8f7751b700 1 --2- 172.21.15.16:0/1153905084 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f8f5406b020 0x7f8f5406d4e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8f6c0098f0 tx=0x7f8f6c008030 comp rx=0 tx=0).stop 2023-10-16T13:46:37.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.628+0000 7f8f7751b700 1 -- 172.21.15.16:0/1153905084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f7006ca10 msgr2=0x7f8f7015d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:37.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.628+0000 7f8f7751b700 1 --2- 172.21.15.16:0/1153905084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f7006ca10 0x7f8f7015d790 secure :-1 s=READY pgs=2504 cs=0 l=1 rev1=1 crypto rx=0x7f8f600011e0 tx=0x7f8f6000cae0 comp rx=0 tx=0).stop 2023-10-16T13:46:37.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.629+0000 7f8f7751b700 1 -- 172.21.15.16:0/1153905084 shutdown_connections 2023-10-16T13:46:37.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.629+0000 7f8f7751b700 1 --2- 172.21.15.16:0/1153905084 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8f700d2110 0x7f8f700c9e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:37.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.629+0000 7f8f7751b700 1 --2- 172.21.15.16:0/1153905084 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f8f5406b020 0x7f8f5406d4e0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:37.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.629+0000 7f8f7751b700 1 --2- 172.21.15.16:0/1153905084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f7006ca10 0x7f8f7015d790 unknown :-1 s=CLOSED pgs=2504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:37.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.629+0000 7f8f7751b700 1 --2- 172.21.15.16:0/1153905084 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8f700ce6a0 0x7f8f700ae450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:37.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.629+0000 7f8f7751b700 1 -- 172.21.15.16:0/1153905084 >> 172.21.15.16:0/1153905084 conn(0x7f8f700d15f0 msgr2=0x7f8f700cf880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:37.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.630+0000 7f8f7751b700 1 -- 172.21.15.16:0/1153905084 shutdown_connections 2023-10-16T13:46:37.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.632+0000 7f8f7751b700 1 -- 172.21.15.16:0/1153905084 wait complete. 2023-10-16T13:46:37.641 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'backup_fs_data' created 2023-10-16T13:46:37.658 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new backup_fs backup_fs_metadata backup_fs_data 2023-10-16T13:46:37.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.788+0000 7fa0fdf26700 1 Processor -- start 2023-10-16T13:46:37.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.788+0000 7fa0fdf26700 1 -- start start 2023-10-16T13:46:37.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.789+0000 7fa0fdf26700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa0f815b2c0 0x7fa0f8161340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:37.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.789+0000 7fa0fdf26700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fa0f806a7e0 0x7fa0f806ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:37.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.789+0000 7fa0fdf26700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0f815ecb0 con 0x7fa0f806e570 2023-10-16T13:46:37.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.789+0000 7fa0fdf26700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0f806b1b0 con 0x7fa0f806a7e0 2023-10-16T13:46:37.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.789+0000 7fa0fdf26700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f8161880 con 0x7fa0f815b2c0 2023-10-16T13:46:37.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.789+0000 7fa0f6ffd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa0f806e570 0x7fa0f815e5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42730/0 (socket says 172.21.15.16:42730) 2023-10-16T13:46:37.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.789+0000 7fa0f6ffd700 1 -- 172.21.15.16:0/466659159 learned_addr learned my addr 172.21.15.16:0/466659159 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:46:37.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.789+0000 7fa0f7fff700 1 --2- 172.21.15.16:0/466659159 >> v2:172.21.15.49:3300/0 conn(0x7fa0f806a7e0 0x7fa0f806ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:37.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.789+0000 7fa0f77fe700 1 --2- 172.21.15.16:0/466659159 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa0f815b2c0 0x7fa0f8161340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:37.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.789+0000 7fa0f67fc700 1 -- 172.21.15.16:0/466659159 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 715798138 0 0) 0x7fa0f815ecb0 con 0x7fa0f806e570 2023-10-16T13:46:37.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.789+0000 7fa0f67fc700 1 -- 172.21.15.16:0/466659159 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0e00035b0 con 0x7fa0f806e570 2023-10-16T13:46:37.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.789+0000 7fa0f67fc700 1 -- 172.21.15.16:0/466659159 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 804091437 0 0) 0x7fa0e00035b0 con 0x7fa0f806e570 2023-10-16T13:46:37.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.789+0000 7fa0f67fc700 1 -- 172.21.15.16:0/466659159 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa0f815b2c0 msgr2=0x7fa0f8161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:37.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.789+0000 7fa0f67fc700 1 --2- 172.21.15.16:0/466659159 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa0f815b2c0 0x7fa0f8161340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:37.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.789+0000 7fa0f67fc700 1 -- 172.21.15.16:0/466659159 >> v2:172.21.15.49:3300/0 conn(0x7fa0f806a7e0 msgr2=0x7fa0f806ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:37.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.789+0000 7fa0f67fc700 1 --2- 172.21.15.16:0/466659159 >> v2:172.21.15.49:3300/0 conn(0x7fa0f806a7e0 0x7fa0f806ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:37.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.789+0000 7fa0f67fc700 1 -- 172.21.15.16:0/466659159 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0f815b6a0 con 0x7fa0f806e570 2023-10-16T13:46:37.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.789+0000 7fa0f67fc700 1 -- 172.21.15.16:0/466659159 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7fa0ec001670 con 0x7fa0f806e570 2023-10-16T13:46:37.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.789+0000 7fa0f67fc700 1 -- 172.21.15.16:0/466659159 >> v1:172.21.15.16:6789/0 conn(0x7fa0f806e570 legacy=0x7fa0f815e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:37.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.789+0000 7fa0f67fc700 1 --2- 172.21.15.16:0/466659159 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa0e0003a00 0x7fa0e0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:37.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.789+0000 7fa0f67fc700 1 --2- 172.21.15.16:0/466659159 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0e00043f0 0x7fa0e00048d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:37.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.789+0000 7fa0f67fc700 1 --2- 172.21.15.16:0/466659159 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa0e0004fe0 0x7fa0e000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:37.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.789+0000 7fa0f67fc700 1 -- 172.21.15.16:0/466659159 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f806b1b0 con 0x7fa0e00043f0 2023-10-16T13:46:37.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.789+0000 7fa0f67fc700 1 -- 172.21.15.16:0/466659159 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f8161880 con 0x7fa0e0004fe0 2023-10-16T13:46:37.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.789+0000 7fa0f67fc700 1 -- 172.21.15.16:0/466659159 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e000bf80 con 0x7fa0e0003a00 2023-10-16T13:46:37.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.790+0000 7fa0f77fe700 1 --2- 172.21.15.16:0/466659159 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa0f815b2c0 0x7fa0f8161340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:46:37.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.790+0000 7fa0f7fff700 1 --2- 172.21.15.16:0/466659159 >> v2:172.21.15.49:3300/0 conn(0x7fa0f806a7e0 0x7fa0f806ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:46:37.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.790+0000 7fa0f6ffd700 1 --2- 172.21.15.16:0/466659159 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0e00043f0 0x7fa0e00048d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:37.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.790+0000 7fa0f7fff700 1 --2- 172.21.15.16:0/466659159 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa0e0004fe0 0x7fa0e000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:37.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.790+0000 7fa0f77fe700 1 --2- 172.21.15.16:0/466659159 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa0e0003a00 0x7fa0e0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:37.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.790+0000 7fa0f6ffd700 1 -- 172.21.15.16:0/466659159 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa0e0003a00 msgr2=0x7fa0e0003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:37.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.790+0000 7fa0f6ffd700 1 --2- 172.21.15.16:0/466659159 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa0e0003a00 0x7fa0e0003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:37.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.790+0000 7fa0f6ffd700 1 -- 172.21.15.16:0/466659159 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa0e0004fe0 msgr2=0x7fa0e000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:37.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.790+0000 7fa0f6ffd700 1 --2- 172.21.15.16:0/466659159 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa0e0004fe0 0x7fa0e000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:37.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.790+0000 7fa0f6ffd700 1 -- 172.21.15.16:0/466659159 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0e00035b0 con 0x7fa0e00043f0 2023-10-16T13:46:37.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.790+0000 7fa0f77fe700 1 --2- 172.21.15.16:0/466659159 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa0e0003a00 0x7fa0e0003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:46:37.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.790+0000 7fa0f6ffd700 1 --2- 172.21.15.16:0/466659159 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0e00043f0 0x7fa0e00048d0 secure :-1 s=READY pgs=2506 cs=0 l=1 rev1=1 crypto rx=0x7fa0ec00c320 tx=0x7fa0e0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a11ec65d527faf server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:37.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.790+0000 7fa0f67fc700 1 -- 172.21.15.16:0/466659159 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0ec00ce00 con 0x7fa0e00043f0 2023-10-16T13:46:37.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.790+0000 7fa0f67fc700 1 -- 172.21.15.16:0/466659159 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0ec005960 con 0x7fa0e00043f0 2023-10-16T13:46:37.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.790+0000 7fa0f67fc700 1 -- 172.21.15.16:0/466659159 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0ec005d10 con 0x7fa0e00043f0 2023-10-16T13:46:37.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.791+0000 7fa0fdf26700 1 -- 172.21.15.16:0/466659159 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0e00043f0 msgr2=0x7fa0e00048d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:37.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.791+0000 7fa0fdf26700 1 --2- 172.21.15.16:0/466659159 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0e00043f0 0x7fa0e00048d0 secure :-1 s=READY pgs=2506 cs=0 l=1 rev1=1 crypto rx=0x7fa0ec00c320 tx=0x7fa0e0005880 comp rx=0 tx=0).stop 2023-10-16T13:46:37.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.791+0000 7fa0f7fff700 1 -- 172.21.15.16:0/466659159 reap_dead start 2023-10-16T13:46:37.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.791+0000 7fa0fdf26700 1 -- 172.21.15.16:0/466659159 shutdown_connections 2023-10-16T13:46:37.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.791+0000 7fa0fdf26700 1 -- 172.21.15.16:0/466659159 >> 172.21.15.16:0/466659159 conn(0x7fa0f80b1f40 msgr2=0x7fa0f80b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:37.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.791+0000 7fa0fdf26700 1 -- 172.21.15.16:0/466659159 shutdown_connections 2023-10-16T13:46:37.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.791+0000 7fa0fdf26700 1 -- 172.21.15.16:0/466659159 wait complete. 2023-10-16T13:46:37.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.792+0000 7fa0fdf26700 1 Processor -- start 2023-10-16T13:46:37.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.792+0000 7fa0fdf26700 1 -- start start 2023-10-16T13:46:37.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.793+0000 7fa0fdf26700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0e00043f0 0x7fa0f80cce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:37.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.793+0000 7fa0fdf26700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa0f8157bb0 0x7fa0f80cd340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:37.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.793+0000 7fa0fdf26700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa0f80c9670 0x7fa0f80c9b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:37.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.793+0000 7fa0fdf26700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f80b60c0 con 0x7fa0e00043f0 2023-10-16T13:46:37.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.793+0000 7fa0fdf26700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f81f37d0 con 0x7fa0f80c9670 2023-10-16T13:46:37.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.793+0000 7fa0fdf26700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f80ca380 con 0x7fa0f8157bb0 2023-10-16T13:46:37.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.793+0000 7fa0f77fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0e00043f0 0x7fa0f80cce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:37.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.793+0000 7fa0f77fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0e00043f0 0x7fa0f80cce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53468/0 (socket says 172.21.15.16:53468) 2023-10-16T13:46:37.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.793+0000 7fa0f77fe700 1 -- 172.21.15.16:0/3772941523 learned_addr learned my addr 172.21.15.16:0/3772941523 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:37.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.793+0000 7fa0f6ffd700 1 --2- 172.21.15.16:0/3772941523 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa0f8157bb0 0x7fa0f80cd340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:37.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.793+0000 7fa0f7fff700 1 --2- 172.21.15.16:0/3772941523 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa0f80c9670 0x7fa0f80c9b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:37.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.794+0000 7fa0f77fe700 1 -- 172.21.15.16:0/3772941523 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa0f8157bb0 msgr2=0x7fa0f80cd340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:37.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.794+0000 7fa0f77fe700 1 --2- 172.21.15.16:0/3772941523 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa0f8157bb0 0x7fa0f80cd340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:37.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.794+0000 7fa0f77fe700 1 -- 172.21.15.16:0/3772941523 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa0f80c9670 msgr2=0x7fa0f80c9b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:37.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.794+0000 7fa0f77fe700 1 --2- 172.21.15.16:0/3772941523 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa0f80c9670 0x7fa0f80c9b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:37.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.794+0000 7fa0f77fe700 1 -- 172.21.15.16:0/3772941523 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0f81f3180 con 0x7fa0e00043f0 2023-10-16T13:46:37.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.794+0000 7fa0f7fff700 1 --2- 172.21.15.16:0/3772941523 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa0f80c9670 0x7fa0f80c9b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:46:37.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.794+0000 7fa0f77fe700 1 --2- 172.21.15.16:0/3772941523 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0e00043f0 0x7fa0f80cce00 secure :-1 s=READY pgs=2507 cs=0 l=1 rev1=1 crypto rx=0x7fa0e80108e0 tx=0x7fa0e8010ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:37.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.794+0000 7fa0dffff700 1 -- 172.21.15.16:0/3772941523 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0e8010ee0 con 0x7fa0e00043f0 2023-10-16T13:46:37.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.794+0000 7fa0dffff700 1 -- 172.21.15.16:0/3772941523 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0e800e8d0 con 0x7fa0e00043f0 2023-10-16T13:46:37.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.794+0000 7fa0fdf26700 1 -- 172.21.15.16:0/3772941523 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0f81f3460 con 0x7fa0e00043f0 2023-10-16T13:46:37.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.794+0000 7fa0dffff700 1 -- 172.21.15.16:0/3772941523 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0e800ec40 con 0x7fa0e00043f0 2023-10-16T13:46:37.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.794+0000 7fa0fdf26700 1 -- 172.21.15.16:0/3772941523 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0f81f2e30 con 0x7fa0e00043f0 2023-10-16T13:46:37.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.795+0000 7fa0fdf26700 1 -- 172.21.15.16:0/3772941523 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0f81f3ab0 con 0x7fa0e00043f0 2023-10-16T13:46:37.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.796+0000 7fa0dffff700 1 -- 172.21.15.16:0/3772941523 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 138) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa0e8016460 con 0x7fa0e00043f0 2023-10-16T13:46:37.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.796+0000 7fa0dffff700 1 --2- 172.21.15.16:0/3772941523 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fa0e406b020 0x7fa0e406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:37.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.797+0000 7fa0dffff700 1 -- 172.21.15.16:0/3772941523 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 6915+0+0 (secure 0 0 0) 0x7fa0e808a6f0 con 0x7fa0e00043f0 2023-10-16T13:46:37.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.797+0000 7fa0f6ffd700 1 --2- 172.21.15.16:0/3772941523 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fa0e406b020 0x7fa0e406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:37.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.797+0000 7fa0f6ffd700 1 --2- 172.21.15.16:0/3772941523 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fa0e406b020 0x7fa0e406d4e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa0e00047d0 tx=0x7fa0ec00a000 comp rx=0 tx=0).ready entity=mgr.11190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:37.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:37.801+0000 7fa0dffff700 1 -- 172.21.15.16:0/3772941523 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa0e80552b0 con 0x7fa0e00043f0 2023-10-16T13:46:38.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:38.051+0000 7fa0fdf26700 1 -- 172.21.15.16:0/3772941523 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "backup_fs", "metadata": "backup_fs_metadata", "data": "backup_fs_data"} v 0) v1 -- 0x7fa0f804ebb0 con 0x7fa0e00043f0 2023-10-16T13:46:38.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:38.647+0000 7fa0dffff700 1 -- 172.21.15.16:0/3772941523 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "backup_fs", "metadata": "backup_fs_metadata", "data": "backup_fs_data"}]=0 new fs with metadata pool 29 and data pool 30 v115) v1 ==== 183+0+0 (secure 0 0 0) 0x7fa0e80588d0 con 0x7fa0e00043f0 2023-10-16T13:46:38.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:38.671+0000 7fa0fdf26700 1 -- 172.21.15.16:0/3772941523 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fa0e406b020 msgr2=0x7fa0e406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:38.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:38.671+0000 7fa0fdf26700 1 --2- 172.21.15.16:0/3772941523 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fa0e406b020 0x7fa0e406d4e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa0e00047d0 tx=0x7fa0ec00a000 comp rx=0 tx=0).stop 2023-10-16T13:46:38.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:38.671+0000 7fa0fdf26700 1 -- 172.21.15.16:0/3772941523 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0e00043f0 msgr2=0x7fa0f80cce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:38.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:38.671+0000 7fa0fdf26700 1 --2- 172.21.15.16:0/3772941523 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0e00043f0 0x7fa0f80cce00 secure :-1 s=READY pgs=2507 cs=0 l=1 rev1=1 crypto rx=0x7fa0e80108e0 tx=0x7fa0e8010ca0 comp rx=0 tx=0).stop 2023-10-16T13:46:38.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:38.671+0000 7fa0fdf26700 1 -- 172.21.15.16:0/3772941523 shutdown_connections 2023-10-16T13:46:38.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:38.671+0000 7fa0fdf26700 1 --2- 172.21.15.16:0/3772941523 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa0f80c9670 0x7fa0f80c9b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:38.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:38.671+0000 7fa0fdf26700 1 --2- 172.21.15.16:0/3772941523 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fa0e406b020 0x7fa0e406d4e0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:38.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:38.671+0000 7fa0fdf26700 1 --2- 172.21.15.16:0/3772941523 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0e00043f0 0x7fa0f80cce00 unknown :-1 s=CLOSED pgs=2507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:38.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:38.671+0000 7fa0fdf26700 1 --2- 172.21.15.16:0/3772941523 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa0f8157bb0 0x7fa0f80cd340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:38.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:38.671+0000 7fa0fdf26700 1 -- 172.21.15.16:0/3772941523 >> 172.21.15.16:0/3772941523 conn(0x7fa0f80b1f40 msgr2=0x7fa0f80cc2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:38.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:38.671+0000 7fa0fdf26700 1 -- 172.21.15.16:0/3772941523 shutdown_connections 2023-10-16T13:46:38.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:38.672+0000 7fa0fdf26700 1 -- 172.21.15.16:0/3772941523 wait complete. 2023-10-16T13:46:38.677 INFO:teuthology.orchestra.run.smithi016.stderr:new fs with metadata pool 29 and data pool 30 2023-10-16T13:46:38.691 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:46:38.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:38.820+0000 7fac3eb9d700 1 Processor -- start 2023-10-16T13:46:38.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:38.820+0000 7fac3eb9d700 1 -- start start 2023-10-16T13:46:38.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:38.821+0000 7fac3eb9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fac380b3ba0 0x7fac380b2da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:38.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:38.821+0000 7fac3eb9d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fac3815a6e0 0x7fac38070860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:38.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:38.821+0000 7fac3eb9d700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac380b4060 con 0x7fac3815a300 2023-10-16T13:46:38.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:38.821+0000 7fac3eb9d700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac380b22c0 con 0x7fac380b3ba0 2023-10-16T13:46:38.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:38.821+0000 7fac3eb9d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac380b25a0 con 0x7fac3815a6e0 2023-10-16T13:46:38.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:38.821+0000 7fac37fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fac380b3ba0 0x7fac380b2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:38.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:38.821+0000 7fac37fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fac380b3ba0 0x7fac380b2da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53490/0 (socket says 172.21.15.16:53490) 2023-10-16T13:46:38.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:38.821+0000 7fac37fff700 1 -- 172.21.15.16:0/268109694 learned_addr learned my addr 172.21.15.16:0/268109694 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:38.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:38.821+0000 7fac3d13a700 1 --2- 172.21.15.16:0/268109694 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fac3815a6e0 0x7fac38070860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:38.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:38.821+0000 7fac37fff700 1 -- 172.21.15.16:0/268109694 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fac3815a6e0 msgr2=0x7fac38070860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:38.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:38.821+0000 7fac37fff700 1 --2- 172.21.15.16:0/268109694 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fac3815a6e0 0x7fac38070860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:38.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:38.821+0000 7fac37fff700 1 -- 172.21.15.16:0/268109694 >> v1:172.21.15.49:6789/0 conn(0x7fac3815a300 legacy=0x7fac3815ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:38.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:38.821+0000 7fac37fff700 1 -- 172.21.15.16:0/268109694 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac380b2880 con 0x7fac380b3ba0 2023-10-16T13:46:38.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:38.821+0000 7fac37fff700 1 --2- 172.21.15.16:0/268109694 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac380b3ba0 0x7fac380b2da0 secure :-1 s=READY pgs=2508 cs=0 l=1 rev1=1 crypto rx=0x7fac2c0098f0 tx=0x7fac2c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62beaea69e7f5d29 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:38.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:38.821+0000 7fac377fe700 1 -- 172.21.15.16:0/268109694 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac2c00b8a0 con 0x7fac380b3ba0 2023-10-16T13:46:38.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:38.821+0000 7fac377fe700 1 -- 172.21.15.16:0/268109694 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac2c00ba00 con 0x7fac380b3ba0 2023-10-16T13:46:38.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:38.821+0000 7fac377fe700 1 -- 172.21.15.16:0/268109694 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac2c00bd90 con 0x7fac380b3ba0 2023-10-16T13:46:38.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:38.822+0000 7fac3eb9d700 1 -- 172.21.15.16:0/268109694 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac380b3ba0 msgr2=0x7fac380b2da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:38.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:38.822+0000 7fac3eb9d700 1 --2- 172.21.15.16:0/268109694 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac380b3ba0 0x7fac380b2da0 secure :-1 s=READY pgs=2508 cs=0 l=1 rev1=1 crypto rx=0x7fac2c0098f0 tx=0x7fac2c009c00 comp rx=0 tx=0).stop 2023-10-16T13:46:38.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:38.822+0000 7fac3eb9d700 1 -- 172.21.15.16:0/268109694 shutdown_connections 2023-10-16T13:46:38.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:38.822+0000 7fac3eb9d700 1 --2- 172.21.15.16:0/268109694 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fac3815a6e0 0x7fac38070860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:38.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:38.822+0000 7fac3eb9d700 1 --2- 172.21.15.16:0/268109694 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac380b3ba0 0x7fac380b2da0 unknown :-1 s=CLOSED pgs=2508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:38.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:38.822+0000 7fac3eb9d700 1 -- 172.21.15.16:0/268109694 >> 172.21.15.16:0/268109694 conn(0x7fac380b4cd0 msgr2=0x7fac3806a7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:38.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:38.822+0000 7fac3eb9d700 1 -- 172.21.15.16:0/268109694 shutdown_connections 2023-10-16T13:46:38.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:38.822+0000 7fac3eb9d700 1 -- 172.21.15.16:0/268109694 wait complete. 2023-10-16T13:46:38.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:38.823+0000 7fac3eb9d700 1 Processor -- start 2023-10-16T13:46:38.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:38.823+0000 7fac3eb9d700 1 -- start start 2023-10-16T13:46:38.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:38.824+0000 7fac3eb9d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fac380b3ba0 0x7fac38157e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:38.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:38.824+0000 7fac3eb9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac3815a300 0x7fac381504c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:38.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:38.824+0000 7fac3eb9d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fac3815a6e0 0x7fac380a84b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:38.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:38.824+0000 7fac3eb9d700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac38157ac0 con 0x7fac3815a300 2023-10-16T13:46:38.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:38.824+0000 7fac3eb9d700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac38158340 con 0x7fac380b3ba0 2023-10-16T13:46:38.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:38.824+0000 7fac3eb9d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac380ae6f0 con 0x7fac3815a6e0 2023-10-16T13:46:38.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:38.824+0000 7fac37fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac3815a300 0x7fac381504c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:38.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:38.824+0000 7fac37fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac3815a300 0x7fac381504c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53496/0 (socket says 172.21.15.16:53496) 2023-10-16T13:46:38.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:38.824+0000 7fac37fff700 1 -- 172.21.15.16:0/4118946379 learned_addr learned my addr 172.21.15.16:0/4118946379 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:38.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:38.825+0000 7fac3c939700 1 --2- 172.21.15.16:0/4118946379 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fac380b3ba0 0x7fac38157e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:38.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:38.825+0000 7fac3d13a700 1 --2- 172.21.15.16:0/4118946379 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fac3815a6e0 0x7fac380a84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:38.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:38.825+0000 7fac37fff700 1 -- 172.21.15.16:0/4118946379 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fac3815a6e0 msgr2=0x7fac380a84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:38.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:38.825+0000 7fac37fff700 1 --2- 172.21.15.16:0/4118946379 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fac3815a6e0 0x7fac380a84b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:38.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:38.825+0000 7fac37fff700 1 -- 172.21.15.16:0/4118946379 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fac380b3ba0 msgr2=0x7fac38157e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:38.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:38.825+0000 7fac37fff700 1 --2- 172.21.15.16:0/4118946379 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fac380b3ba0 0x7fac38157e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:38.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:38.825+0000 7fac37fff700 1 -- 172.21.15.16:0/4118946379 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac2c009550 con 0x7fac3815a300 2023-10-16T13:46:38.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:38.825+0000 7fac3c939700 1 --2- 172.21.15.16:0/4118946379 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fac380b3ba0 0x7fac38157e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:46:38.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:38.825+0000 7fac37fff700 1 --2- 172.21.15.16:0/4118946379 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac3815a300 0x7fac381504c0 secure :-1 s=READY pgs=2509 cs=0 l=1 rev1=1 crypto rx=0x7fac2c000c00 tx=0x7fac2c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:38.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:38.825+0000 7fac357fa700 1 -- 172.21.15.16:0/4118946379 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac2c0036a0 con 0x7fac3815a300 2023-10-16T13:46:38.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:38.825+0000 7fac357fa700 1 -- 172.21.15.16:0/4118946379 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac2c003800 con 0x7fac3815a300 2023-10-16T13:46:38.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:38.825+0000 7fac3eb9d700 1 -- 172.21.15.16:0/4118946379 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac38150a00 con 0x7fac3815a300 2023-10-16T13:46:38.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:38.825+0000 7fac357fa700 1 -- 172.21.15.16:0/4118946379 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac2c003b90 con 0x7fac3815a300 2023-10-16T13:46:38.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:38.825+0000 7fac3eb9d700 1 -- 172.21.15.16:0/4118946379 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac38158850 con 0x7fac3815a300 2023-10-16T13:46:38.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:38.826+0000 7fac3eb9d700 1 -- 172.21.15.16:0/4118946379 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac3804ebb0 con 0x7fac3815a300 2023-10-16T13:46:38.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:38.826+0000 7fac357fa700 1 -- 172.21.15.16:0/4118946379 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 138) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fac2c003cf0 con 0x7fac3815a300 2023-10-16T13:46:38.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:38.827+0000 7fac357fa700 1 --2- 172.21.15.16:0/4118946379 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fac2406b020 0x7fac2406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:38.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:38.827+0000 7fac3c939700 1 --2- 172.21.15.16:0/4118946379 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fac2406b020 0x7fac2406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:38.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:38.827+0000 7fac357fa700 1 -- 172.21.15.16:0/4118946379 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 7021+0+0 (secure 0 0 0) 0x7fac2c08cc30 con 0x7fac3815a300 2023-10-16T13:46:38.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:38.827+0000 7fac3c939700 1 --2- 172.21.15.16:0/4118946379 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fac2406b020 0x7fac2406d4e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fac28000f60 tx=0x7fac28009040 comp rx=0 tx=0).ready entity=mgr.11190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:38.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:38.832+0000 7fac357fa700 1 -- 172.21.15.16:0/4118946379 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fac2c057740 con 0x7fac3815a300 2023-10-16T13:46:39.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.036+0000 7fac3eb9d700 1 -- 172.21.15.16:0/4118946379 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fac380b2f40 con 0x7fac3815a300 2023-10-16T13:46:39.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.037+0000 7fac357fa700 1 -- 172.21.15.16:0/4118946379 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v228) v1 ==== 74+0+20550 (secure 0 0 0) 0x7fac2c05ad60 con 0x7fac3815a300 2023-10-16T13:46:39.040 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:46:39.041 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":228,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:46:38.053152+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2023-10-16T13:46:22.489653+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"224","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2023-10-16T13:46:23.729509+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"225","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T13:46:35.800444+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"228","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"backup_fs_data","create_time":"2023-10-16T13:46:37.035323+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"228","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:46:39.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.039+0000 7fac3eb9d700 1 -- 172.21.15.16:0/4118946379 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fac2406b020 msgr2=0x7fac2406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:39.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.039+0000 7fac3eb9d700 1 --2- 172.21.15.16:0/4118946379 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fac2406b020 0x7fac2406d4e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fac28000f60 tx=0x7fac28009040 comp rx=0 tx=0).stop 2023-10-16T13:46:39.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.039+0000 7fac3eb9d700 1 -- 172.21.15.16:0/4118946379 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac3815a300 msgr2=0x7fac381504c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:39.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.039+0000 7fac3eb9d700 1 --2- 172.21.15.16:0/4118946379 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac3815a300 0x7fac381504c0 secure :-1 s=READY pgs=2509 cs=0 l=1 rev1=1 crypto rx=0x7fac2c000c00 tx=0x7fac2c004750 comp rx=0 tx=0).stop 2023-10-16T13:46:39.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.039+0000 7fac3eb9d700 1 -- 172.21.15.16:0/4118946379 shutdown_connections 2023-10-16T13:46:39.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.039+0000 7fac3eb9d700 1 --2- 172.21.15.16:0/4118946379 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fac3815a6e0 0x7fac380a84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:39.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.039+0000 7fac3eb9d700 1 --2- 172.21.15.16:0/4118946379 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fac380b3ba0 0x7fac38157e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:39.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.039+0000 7fac3eb9d700 1 --2- 172.21.15.16:0/4118946379 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fac2406b020 0x7fac2406d4e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:39.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.039+0000 7fac3eb9d700 1 --2- 172.21.15.16:0/4118946379 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac3815a300 0x7fac381504c0 unknown :-1 s=CLOSED pgs=2509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:39.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.039+0000 7fac3eb9d700 1 -- 172.21.15.16:0/4118946379 >> 172.21.15.16:0/4118946379 conn(0x7fac380b4cd0 msgr2=0x7fac380afc40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:39.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.039+0000 7fac3eb9d700 1 -- 172.21.15.16:0/4118946379 shutdown_connections 2023-10-16T13:46:39.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.039+0000 7fac3eb9d700 1 -- 172.21.15.16:0/4118946379 wait complete. 2023-10-16T13:46:39.055 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:46:39.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.177+0000 7efcf56dc700 1 Processor -- start 2023-10-16T13:46:39.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.177+0000 7efcf56dc700 1 -- start start 2023-10-16T13:46:39.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.178+0000 7efcf56dc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7efcf015d8d0 0x7efcf00d1c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:39.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.178+0000 7efcf56dc700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7efcf00d0750 0x7efcf006c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:39.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.178+0000 7efcf56dc700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcf006ba00 con 0x7efcf00d41c0 2023-10-16T13:46:39.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.178+0000 7efcf56dc700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcf00d2150 con 0x7efcf015d8d0 2023-10-16T13:46:39.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.178+0000 7efcf56dc700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcf00d26d0 con 0x7efcf00d0750 2023-10-16T13:46:39.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.178+0000 7efceeffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7efcf015d8d0 0x7efcf00d1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:39.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.178+0000 7efceeffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7efcf015d8d0 0x7efcf00d1c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53502/0 (socket says 172.21.15.16:53502) 2023-10-16T13:46:39.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.178+0000 7efceeffd700 1 -- 172.21.15.16:0/3647653251 learned_addr learned my addr 172.21.15.16:0/3647653251 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:39.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.178+0000 7efcef7fe700 1 --2- 172.21.15.16:0/3647653251 >> v2:172.21.15.49:3300/0 conn(0x7efcf00d0750 0x7efcf006c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:39.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.178+0000 7efceeffd700 1 -- 172.21.15.16:0/3647653251 >> v2:172.21.15.49:3300/0 conn(0x7efcf00d0750 msgr2=0x7efcf006c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:39.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.178+0000 7efceeffd700 1 --2- 172.21.15.16:0/3647653251 >> v2:172.21.15.49:3300/0 conn(0x7efcf00d0750 0x7efcf006c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:39.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.178+0000 7efceeffd700 1 -- 172.21.15.16:0/3647653251 >> v1:172.21.15.49:6789/0 conn(0x7efcf00d41c0 legacy=0x7efcf00d0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:39.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.178+0000 7efceeffd700 1 -- 172.21.15.16:0/3647653251 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcf00d23f0 con 0x7efcf015d8d0 2023-10-16T13:46:39.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.178+0000 7efceeffd700 1 --2- 172.21.15.16:0/3647653251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efcf015d8d0 0x7efcf00d1c10 secure :-1 s=READY pgs=2510 cs=0 l=1 rev1=1 crypto rx=0x7efcf00d4640 tx=0x7efce0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b28d846407d0294 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:39.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.178+0000 7efcedffb700 1 -- 172.21.15.16:0/3647653251 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efce0004230 con 0x7efcf015d8d0 2023-10-16T13:46:39.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.178+0000 7efcedffb700 1 -- 172.21.15.16:0/3647653251 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efce0004390 con 0x7efcf015d8d0 2023-10-16T13:46:39.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.178+0000 7efcedffb700 1 -- 172.21.15.16:0/3647653251 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efce00036a0 con 0x7efcf015d8d0 2023-10-16T13:46:39.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.179+0000 7efcf56dc700 1 -- 172.21.15.16:0/3647653251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efcf015d8d0 msgr2=0x7efcf00d1c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:39.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.179+0000 7efcf56dc700 1 --2- 172.21.15.16:0/3647653251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efcf015d8d0 0x7efcf00d1c10 secure :-1 s=READY pgs=2510 cs=0 l=1 rev1=1 crypto rx=0x7efcf00d4640 tx=0x7efce0009bd0 comp rx=0 tx=0).stop 2023-10-16T13:46:39.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.179+0000 7efcf56dc700 1 -- 172.21.15.16:0/3647653251 shutdown_connections 2023-10-16T13:46:39.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.179+0000 7efcf56dc700 1 --2- 172.21.15.16:0/3647653251 >> v2:172.21.15.49:3300/0 conn(0x7efcf00d0750 0x7efcf006c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:39.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.179+0000 7efcf56dc700 1 --2- 172.21.15.16:0/3647653251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efcf015d8d0 0x7efcf00d1c10 unknown :-1 s=CLOSED pgs=2510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:39.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.179+0000 7efcf56dc700 1 -- 172.21.15.16:0/3647653251 >> 172.21.15.16:0/3647653251 conn(0x7efcf00d3630 msgr2=0x7efcf00d3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:39.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.179+0000 7efcf56dc700 1 -- 172.21.15.16:0/3647653251 shutdown_connections 2023-10-16T13:46:39.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.179+0000 7efcf56dc700 1 -- 172.21.15.16:0/3647653251 wait complete. 2023-10-16T13:46:39.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.180+0000 7efcf56dc700 1 Processor -- start 2023-10-16T13:46:39.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.180+0000 7efcf56dc700 1 -- start start 2023-10-16T13:46:39.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.181+0000 7efcf56dc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efcf00d0750 0x7efcf00a0100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:39.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.181+0000 7efcf56dc700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efcf00d41c0 0x7efcf00a0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:39.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.181+0000 7efcf56dc700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efcf015d8d0 0x7efcf009de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:39.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.181+0000 7efcf56dc700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcf014ff70 con 0x7efcf00d0750 2023-10-16T13:46:39.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.181+0000 7efcf56dc700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcf00a1150 con 0x7efcf00d41c0 2023-10-16T13:46:39.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.181+0000 7efcf56dc700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcf00a0e70 con 0x7efcf015d8d0 2023-10-16T13:46:39.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.181+0000 7efceeffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efcf00d0750 0x7efcf00a0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:39.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.181+0000 7efceeffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efcf00d0750 0x7efcf00a0100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53506/0 (socket says 172.21.15.16:53506) 2023-10-16T13:46:39.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.181+0000 7efceeffd700 1 -- 172.21.15.16:0/3529465780 learned_addr learned my addr 172.21.15.16:0/3529465780 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:39.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.181+0000 7efcee7fc700 1 --2- 172.21.15.16:0/3529465780 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efcf00d41c0 0x7efcf00a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:39.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.181+0000 7efcef7fe700 1 --2- 172.21.15.16:0/3529465780 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efcf015d8d0 0x7efcf009de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:39.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.182+0000 7efceeffd700 1 -- 172.21.15.16:0/3529465780 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efcf015d8d0 msgr2=0x7efcf009de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:39.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.182+0000 7efceeffd700 1 --2- 172.21.15.16:0/3529465780 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efcf015d8d0 0x7efcf009de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:39.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.182+0000 7efceeffd700 1 -- 172.21.15.16:0/3529465780 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efcf00d41c0 msgr2=0x7efcf00a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:39.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.182+0000 7efceeffd700 1 --2- 172.21.15.16:0/3529465780 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efcf00d41c0 0x7efcf00a0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:39.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.182+0000 7efceeffd700 1 -- 172.21.15.16:0/3529465780 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efce0009550 con 0x7efcf00d0750 2023-10-16T13:46:39.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.182+0000 7efceeffd700 1 --2- 172.21.15.16:0/3529465780 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efcf00d0750 0x7efcf00a0100 secure :-1 s=READY pgs=2511 cs=0 l=1 rev1=1 crypto rx=0x7efce0009c10 tx=0x7efce0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:39.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.182+0000 7efcd77fe700 1 -- 172.21.15.16:0/3529465780 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efce0003e20 con 0x7efcf00d0750 2023-10-16T13:46:39.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.182+0000 7efcd77fe700 1 -- 172.21.15.16:0/3529465780 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efce0011630 con 0x7efcf00d0750 2023-10-16T13:46:39.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.182+0000 7efcd77fe700 1 -- 172.21.15.16:0/3529465780 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efce00119e0 con 0x7efcf00d0750 2023-10-16T13:46:39.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.182+0000 7efcf56dc700 1 -- 172.21.15.16:0/3529465780 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcf009d3b0 con 0x7efcf00d0750 2023-10-16T13:46:39.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.182+0000 7efcf56dc700 1 -- 172.21.15.16:0/3529465780 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcf009d970 con 0x7efcf00d0750 2023-10-16T13:46:39.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.183+0000 7efcf56dc700 1 -- 172.21.15.16:0/3529465780 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcf004ebb0 con 0x7efcf00d0750 2023-10-16T13:46:39.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.184+0000 7efcd77fe700 1 -- 172.21.15.16:0/3529465780 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 138) v1 ==== 89331+0+0 (secure 0 0 0) 0x7efce0011b40 con 0x7efcf00d0750 2023-10-16T13:46:39.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.184+0000 7efcd77fe700 1 --2- 172.21.15.16:0/3529465780 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7efcdc06b020 0x7efcdc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:39.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.184+0000 7efcd77fe700 1 -- 172.21.15.16:0/3529465780 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 7021+0+0 (secure 0 0 0) 0x7efce008b110 con 0x7efcf00d0750 2023-10-16T13:46:39.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.184+0000 7efcee7fc700 1 --2- 172.21.15.16:0/3529465780 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7efcdc06b020 0x7efcdc06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:39.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.184+0000 7efcee7fc700 1 --2- 172.21.15.16:0/3529465780 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7efcdc06b020 0x7efcdc06d4e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7efcf00d1d30 tx=0x7efce4007ec0 comp rx=0 tx=0).ready entity=mgr.11190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:39.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.189+0000 7efcd77fe700 1 -- 172.21.15.16:0/3529465780 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7efce0055ba0 con 0x7efcf00d0750 2023-10-16T13:46:39.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.392+0000 7efcf56dc700 1 -- 172.21.15.16:0/3529465780 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efcf006c5f0 con 0x7efcf00d0750 2023-10-16T13:46:39.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.393+0000 7efcd77fe700 1 -- 172.21.15.16:0/3529465780 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v228) v1 ==== 74+0+20550 (secure 0 0 0) 0x7efce00591c0 con 0x7efcf00d0750 2023-10-16T13:46:39.396 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:46:39.397 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":228,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:46:38.053152+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2023-10-16T13:46:22.489653+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"224","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2023-10-16T13:46:23.729509+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"225","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T13:46:35.800444+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"228","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"backup_fs_data","create_time":"2023-10-16T13:46:37.035323+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"228","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:46:39.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.396+0000 7efcf56dc700 1 -- 172.21.15.16:0/3529465780 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7efcdc06b020 msgr2=0x7efcdc06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:39.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.396+0000 7efcf56dc700 1 --2- 172.21.15.16:0/3529465780 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7efcdc06b020 0x7efcdc06d4e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7efcf00d1d30 tx=0x7efce4007ec0 comp rx=0 tx=0).stop 2023-10-16T13:46:39.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.396+0000 7efcf56dc700 1 -- 172.21.15.16:0/3529465780 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efcf00d0750 msgr2=0x7efcf00a0100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:39.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.396+0000 7efcf56dc700 1 --2- 172.21.15.16:0/3529465780 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efcf00d0750 0x7efcf00a0100 secure :-1 s=READY pgs=2511 cs=0 l=1 rev1=1 crypto rx=0x7efce0009c10 tx=0x7efce0004750 comp rx=0 tx=0).stop 2023-10-16T13:46:39.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.396+0000 7efcf56dc700 1 -- 172.21.15.16:0/3529465780 shutdown_connections 2023-10-16T13:46:39.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.396+0000 7efcf56dc700 1 --2- 172.21.15.16:0/3529465780 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efcf015d8d0 0x7efcf009de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:39.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.396+0000 7efcf56dc700 1 --2- 172.21.15.16:0/3529465780 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7efcdc06b020 0x7efcdc06d4e0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:39.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.396+0000 7efcf56dc700 1 --2- 172.21.15.16:0/3529465780 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efcf00d0750 0x7efcf00a0100 unknown :-1 s=CLOSED pgs=2511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:39.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.396+0000 7efcf56dc700 1 --2- 172.21.15.16:0/3529465780 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efcf00d41c0 0x7efcf00a0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:39.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.396+0000 7efcf56dc700 1 -- 172.21.15.16:0/3529465780 >> 172.21.15.16:0/3529465780 conn(0x7efcf00d3630 msgr2=0x7efcf00d11e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:39.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.396+0000 7efcf56dc700 1 -- 172.21.15.16:0/3529465780 shutdown_connections 2023-10-16T13:46:39.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.396+0000 7efcf56dc700 1 -- 172.21.15.16:0/3529465780 wait complete. 2023-10-16T13:46:39.411 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set backup_fs standby_count_wanted 0 2023-10-16T13:46:39.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.537+0000 7f6a76cdd700 1 Processor -- start 2023-10-16T13:46:39.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.537+0000 7f6a76cdd700 1 -- start start 2023-10-16T13:46:39.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.538+0000 7f6a76cdd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6a70072b30 0x7f6a70071a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:39.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.538+0000 7f6a76cdd700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6a7015bb40 0x7f6a7006e310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:39.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.538+0000 7f6a76cdd700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f6a700caa40 0x7f6a700cae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:39.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.538+0000 7f6a76cdd700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a700cb3b0 con 0x7f6a70072b30 2023-10-16T13:46:39.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.538+0000 7f6a76cdd700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a7015c040 con 0x7f6a700caa40 2023-10-16T13:46:39.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.538+0000 7f6a76cdd700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a7015c320 con 0x7f6a7015bb40 2023-10-16T13:46:39.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.538+0000 7f6a74a79700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6a70072b30 0x7f6a70071a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:39.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.538+0000 7f6a74a79700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6a70072b30 0x7f6a70071a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53514/0 (socket says 172.21.15.16:53514) 2023-10-16T13:46:39.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.538+0000 7f6a74a79700 1 -- 172.21.15.16:0/2695007370 learned_addr learned my addr 172.21.15.16:0/2695007370 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:39.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.538+0000 7f6a6ffff700 1 --2- 172.21.15.16:0/2695007370 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6a7015bb40 0x7f6a7006e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:39.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.538+0000 7f6a7527a700 1 --2- 172.21.15.16:0/2695007370 >> v2:172.21.15.49:3300/0 conn(0x7f6a700caa40 0x7f6a700cae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:39.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.538+0000 7f6a74a79700 1 -- 172.21.15.16:0/2695007370 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6a7015bb40 msgr2=0x7f6a7006e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:39.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.538+0000 7f6a74a79700 1 --2- 172.21.15.16:0/2695007370 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6a7015bb40 0x7f6a7006e310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:39.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.538+0000 7f6a74a79700 1 -- 172.21.15.16:0/2695007370 >> v2:172.21.15.49:3300/0 conn(0x7f6a700caa40 msgr2=0x7f6a700cae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:39.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.538+0000 7f6a74a79700 1 --2- 172.21.15.16:0/2695007370 >> v2:172.21.15.49:3300/0 conn(0x7f6a700caa40 0x7f6a700cae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:39.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.538+0000 7f6a74a79700 1 -- 172.21.15.16:0/2695007370 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a7015c600 con 0x7f6a70072b30 2023-10-16T13:46:39.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.538+0000 7f6a6ffff700 1 --2- 172.21.15.16:0/2695007370 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6a7015bb40 0x7f6a7006e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:46:39.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.539+0000 7f6a74a79700 1 --2- 172.21.15.16:0/2695007370 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a70072b30 0x7f6a70071a70 secure :-1 s=READY pgs=2512 cs=0 l=1 rev1=1 crypto rx=0x7f6a70071a00 tx=0x7f6a64009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9287d76a5e822622 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:39.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.539+0000 7f6a6f7fe700 1 -- 172.21.15.16:0/2695007370 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a6400b870 con 0x7f6a70072b30 2023-10-16T13:46:39.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.539+0000 7f6a6f7fe700 1 -- 172.21.15.16:0/2695007370 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a6400b9d0 con 0x7f6a70072b30 2023-10-16T13:46:39.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.539+0000 7f6a6f7fe700 1 -- 172.21.15.16:0/2695007370 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a6400bd60 con 0x7f6a70072b30 2023-10-16T13:46:39.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.539+0000 7f6a76cdd700 1 -- 172.21.15.16:0/2695007370 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a70072b30 msgr2=0x7f6a70071a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:39.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.539+0000 7f6a76cdd700 1 --2- 172.21.15.16:0/2695007370 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a70072b30 0x7f6a70071a70 secure :-1 s=READY pgs=2512 cs=0 l=1 rev1=1 crypto rx=0x7f6a70071a00 tx=0x7f6a64009bd0 comp rx=0 tx=0).stop 2023-10-16T13:46:39.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.539+0000 7f6a76cdd700 1 -- 172.21.15.16:0/2695007370 shutdown_connections 2023-10-16T13:46:39.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.539+0000 7f6a76cdd700 1 --2- 172.21.15.16:0/2695007370 >> v2:172.21.15.49:3300/0 conn(0x7f6a700caa40 0x7f6a700cae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:39.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.539+0000 7f6a76cdd700 1 --2- 172.21.15.16:0/2695007370 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a70072b30 0x7f6a70071a70 unknown :-1 s=CLOSED pgs=2512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:39.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.539+0000 7f6a76cdd700 1 --2- 172.21.15.16:0/2695007370 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6a7015bb40 0x7f6a7006e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:39.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.539+0000 7f6a76cdd700 1 -- 172.21.15.16:0/2695007370 >> 172.21.15.16:0/2695007370 conn(0x7f6a700b1f40 msgr2=0x7f6a700b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:39.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.539+0000 7f6a76cdd700 1 -- 172.21.15.16:0/2695007370 shutdown_connections 2023-10-16T13:46:39.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.539+0000 7f6a76cdd700 1 -- 172.21.15.16:0/2695007370 wait complete. 2023-10-16T13:46:39.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.541+0000 7f6a76cdd700 1 Processor -- start 2023-10-16T13:46:39.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.541+0000 7f6a76cdd700 1 -- start start 2023-10-16T13:46:39.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.542+0000 7f6a76cdd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a70072b30 0x7f6a701586f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:39.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.542+0000 7f6a76cdd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6a700caa40 0x7f6a70155290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:39.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.542+0000 7f6a76cdd700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6a7015bb40 0x7f6a700b5860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:39.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.542+0000 7f6a76cdd700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a701582d0 con 0x7f6a70072b30 2023-10-16T13:46:39.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.542+0000 7f6a76cdd700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a70157650 con 0x7f6a700caa40 2023-10-16T13:46:39.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.542+0000 7f6a76cdd700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a70157930 con 0x7f6a7015bb40 2023-10-16T13:46:39.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.542+0000 7f6a74a79700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a70072b30 0x7f6a701586f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:39.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.542+0000 7f6a74a79700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a70072b30 0x7f6a701586f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53520/0 (socket says 172.21.15.16:53520) 2023-10-16T13:46:39.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.542+0000 7f6a74a79700 1 -- 172.21.15.16:0/1846885675 learned_addr learned my addr 172.21.15.16:0/1846885675 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:39.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.542+0000 7f6a7527a700 1 --2- 172.21.15.16:0/1846885675 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6a7015bb40 0x7f6a700b5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:39.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.542+0000 7f6a6ffff700 1 --2- 172.21.15.16:0/1846885675 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6a700caa40 0x7f6a70155290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:39.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.542+0000 7f6a74a79700 1 -- 172.21.15.16:0/1846885675 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6a7015bb40 msgr2=0x7f6a700b5860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:39.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.542+0000 7f6a74a79700 1 --2- 172.21.15.16:0/1846885675 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6a7015bb40 0x7f6a700b5860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:39.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.542+0000 7f6a74a79700 1 -- 172.21.15.16:0/1846885675 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6a700caa40 msgr2=0x7f6a70155290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:39.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.542+0000 7f6a74a79700 1 --2- 172.21.15.16:0/1846885675 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6a700caa40 0x7f6a70155290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:39.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.542+0000 7f6a74a79700 1 -- 172.21.15.16:0/1846885675 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a64009550 con 0x7f6a70072b30 2023-10-16T13:46:39.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.542+0000 7f6a7527a700 1 --2- 172.21.15.16:0/1846885675 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6a7015bb40 0x7f6a700b5860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:46:39.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.542+0000 7f6a6ffff700 1 --2- 172.21.15.16:0/1846885675 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6a700caa40 0x7f6a70155290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:46:39.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.542+0000 7f6a74a79700 1 --2- 172.21.15.16:0/1846885675 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a70072b30 0x7f6a701586f0 secure :-1 s=READY pgs=2513 cs=0 l=1 rev1=1 crypto rx=0x7f6a64014040 tx=0x7f6a64004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:39.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.542+0000 7f6a6d7fa700 1 -- 172.21.15.16:0/1846885675 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a64004520 con 0x7f6a70072b30 2023-10-16T13:46:39.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.542+0000 7f6a6d7fa700 1 -- 172.21.15.16:0/1846885675 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a6401e070 con 0x7f6a70072b30 2023-10-16T13:46:39.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.542+0000 7f6a76cdd700 1 -- 172.21.15.16:0/1846885675 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a70157c10 con 0x7f6a70072b30 2023-10-16T13:46:39.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.542+0000 7f6a6d7fa700 1 -- 172.21.15.16:0/1846885675 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a640038e0 con 0x7f6a70072b30 2023-10-16T13:46:39.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.542+0000 7f6a76cdd700 1 -- 172.21.15.16:0/1846885675 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a700b5da0 con 0x7f6a70072b30 2023-10-16T13:46:39.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.543+0000 7f6a76cdd700 1 -- 172.21.15.16:0/1846885675 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a70158410 con 0x7f6a70072b30 2023-10-16T13:46:39.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.544+0000 7f6a6d7fa700 1 -- 172.21.15.16:0/1846885675 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 138) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6a64003a40 con 0x7f6a70072b30 2023-10-16T13:46:39.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.544+0000 7f6a6d7fa700 1 --2- 172.21.15.16:0/1846885675 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f6a5406b020 0x7f6a5406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:39.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.544+0000 7f6a6d7fa700 1 -- 172.21.15.16:0/1846885675 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 7021+0+0 (secure 0 0 0) 0x7f6a64020070 con 0x7f6a70072b30 2023-10-16T13:46:39.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.544+0000 7f6a6ffff700 1 --2- 172.21.15.16:0/1846885675 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f6a5406b020 0x7f6a5406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:39.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.545+0000 7f6a6ffff700 1 --2- 172.21.15.16:0/1846885675 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f6a5406b020 0x7f6a5406d4e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6a701f6570 tx=0x7f6a5c008040 comp rx=0 tx=0).ready entity=mgr.11190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:39.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.549+0000 7f6a6d7fa700 1 -- 172.21.15.16:0/1846885675 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6a70158410 con 0x7f6a70072b30 2023-10-16T13:46:39.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:39.804+0000 7f6a76cdd700 1 -- 172.21.15.16:0/1846885675 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "backup_fs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f6a7004ebb0 con 0x7f6a70072b30 2023-10-16T13:46:40.549 INFO:tasks.ceph.mds.a.smithi016.stderr:2023-10-16T13:46:40.547+0000 7f9a762b4700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T13:46:40.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:40.656+0000 7f6a6d7fa700 1 -- 172.21.15.16:0/1846885675 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "backup_fs", "var": "standby_count_wanted", "val": "0"}]=0 v118) v1 ==== 121+0+0 (secure 0 0 0) 0x7f6a64055dc0 con 0x7f6a70072b30 2023-10-16T13:46:40.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:40.658+0000 7f6a76cdd700 1 -- 172.21.15.16:0/1846885675 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f6a5406b020 msgr2=0x7f6a5406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:40.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:40.658+0000 7f6a76cdd700 1 --2- 172.21.15.16:0/1846885675 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f6a5406b020 0x7f6a5406d4e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6a701f6570 tx=0x7f6a5c008040 comp rx=0 tx=0).stop 2023-10-16T13:46:40.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:40.659+0000 7f6a76cdd700 1 -- 172.21.15.16:0/1846885675 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a70072b30 msgr2=0x7f6a701586f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:40.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:40.659+0000 7f6a76cdd700 1 --2- 172.21.15.16:0/1846885675 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a70072b30 0x7f6a701586f0 secure :-1 s=READY pgs=2513 cs=0 l=1 rev1=1 crypto rx=0x7f6a64014040 tx=0x7f6a64004750 comp rx=0 tx=0).stop 2023-10-16T13:46:40.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:40.659+0000 7f6a76cdd700 1 -- 172.21.15.16:0/1846885675 shutdown_connections 2023-10-16T13:46:40.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:40.659+0000 7f6a76cdd700 1 --2- 172.21.15.16:0/1846885675 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6a7015bb40 0x7f6a700b5860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:40.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:40.659+0000 7f6a76cdd700 1 --2- 172.21.15.16:0/1846885675 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f6a5406b020 0x7f6a5406d4e0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:40.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:40.659+0000 7f6a76cdd700 1 --2- 172.21.15.16:0/1846885675 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a70072b30 0x7f6a701586f0 unknown :-1 s=CLOSED pgs=2513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:40.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:40.659+0000 7f6a76cdd700 1 --2- 172.21.15.16:0/1846885675 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6a700caa40 0x7f6a70155290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:40.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:40.659+0000 7f6a76cdd700 1 -- 172.21.15.16:0/1846885675 >> 172.21.15.16:0/1846885675 conn(0x7f6a700b1f40 msgr2=0x7f6a70159740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:40.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:40.659+0000 7f6a76cdd700 1 -- 172.21.15.16:0/1846885675 shutdown_connections 2023-10-16T13:46:40.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:40.659+0000 7f6a76cdd700 1 -- 172.21.15.16:0/1846885675 wait complete. 2023-10-16T13:46:40.676 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-16T13:46:40.677 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set backup_fs allow_standby_replay false 2023-10-16T13:46:40.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:40.809+0000 7fd4bdf27700 1 Processor -- start 2023-10-16T13:46:40.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:40.809+0000 7fd4bdf27700 1 -- start start 2023-10-16T13:46:40.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:40.810+0000 7fd4bdf27700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd4b8160ee0 0x7fd4b80aeb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:40.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:40.810+0000 7fd4bdf27700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd4b80af040 0x7fd4b80ce180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:40.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:40.810+0000 7fd4bdf27700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fd4b80ce6c0 0x7fd4b81615b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:40.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:40.810+0000 7fd4bdf27700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4b80cf770 con 0x7fd4b8160ee0 2023-10-16T13:46:40.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:40.810+0000 7fd4bdf27700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4b80af4a0 con 0x7fd4b80ce6c0 2023-10-16T13:46:40.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:40.810+0000 7fd4bdf27700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b806aa10 con 0x7fd4b80af040 2023-10-16T13:46:40.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:40.810+0000 7fd4b77fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd4b8160ee0 0x7fd4b80aeb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:40.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:40.810+0000 7fd4b77fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd4b8160ee0 0x7fd4b80aeb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53528/0 (socket says 172.21.15.16:53528) 2023-10-16T13:46:40.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:40.810+0000 7fd4b77fe700 1 -- 172.21.15.16:0/223100161 learned_addr learned my addr 172.21.15.16:0/223100161 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:40.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:40.811+0000 7fd4b7fff700 1 --2- 172.21.15.16:0/223100161 >> v2:172.21.15.49:3300/0 conn(0x7fd4b80ce6c0 0x7fd4b81615b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:40.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:40.811+0000 7fd4b6ffd700 1 --2- 172.21.15.16:0/223100161 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd4b80af040 0x7fd4b80ce180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:40.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:40.811+0000 7fd4b77fe700 1 -- 172.21.15.16:0/223100161 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd4b80af040 msgr2=0x7fd4b80ce180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:40.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:40.811+0000 7fd4b77fe700 1 --2- 172.21.15.16:0/223100161 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd4b80af040 0x7fd4b80ce180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:40.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:40.811+0000 7fd4b77fe700 1 -- 172.21.15.16:0/223100161 >> v2:172.21.15.49:3300/0 conn(0x7fd4b80ce6c0 msgr2=0x7fd4b81615b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:40.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:40.811+0000 7fd4b77fe700 1 --2- 172.21.15.16:0/223100161 >> v2:172.21.15.49:3300/0 conn(0x7fd4b80ce6c0 0x7fd4b81615b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:40.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:40.811+0000 7fd4b77fe700 1 -- 172.21.15.16:0/223100161 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4b80ceed0 con 0x7fd4b8160ee0 2023-10-16T13:46:40.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:40.811+0000 7fd4b77fe700 1 --2- 172.21.15.16:0/223100161 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4b8160ee0 0x7fd4b80aeb00 secure :-1 s=READY pgs=2514 cs=0 l=1 rev1=1 crypto rx=0x7fd4a80098f0 tx=0x7fd4a8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=319c9241f61ac344 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:40.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:40.811+0000 7fd4b67fc700 1 -- 172.21.15.16:0/223100161 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4a800b860 con 0x7fd4b8160ee0 2023-10-16T13:46:40.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:40.811+0000 7fd4b67fc700 1 -- 172.21.15.16:0/223100161 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4a800b9c0 con 0x7fd4b8160ee0 2023-10-16T13:46:40.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:40.811+0000 7fd4b67fc700 1 -- 172.21.15.16:0/223100161 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4a800bd50 con 0x7fd4b8160ee0 2023-10-16T13:46:40.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:40.812+0000 7fd4bdf27700 1 -- 172.21.15.16:0/223100161 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4b8160ee0 msgr2=0x7fd4b80aeb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:40.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:40.812+0000 7fd4bdf27700 1 --2- 172.21.15.16:0/223100161 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4b8160ee0 0x7fd4b80aeb00 secure :-1 s=READY pgs=2514 cs=0 l=1 rev1=1 crypto rx=0x7fd4a80098f0 tx=0x7fd4a8009c00 comp rx=0 tx=0).stop 2023-10-16T13:46:40.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:40.812+0000 7fd4bdf27700 1 -- 172.21.15.16:0/223100161 shutdown_connections 2023-10-16T13:46:40.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:40.812+0000 7fd4bdf27700 1 --2- 172.21.15.16:0/223100161 >> v2:172.21.15.49:3300/0 conn(0x7fd4b80ce6c0 0x7fd4b81615b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:40.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:40.812+0000 7fd4bdf27700 1 --2- 172.21.15.16:0/223100161 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4b8160ee0 0x7fd4b80aeb00 unknown :-1 s=CLOSED pgs=2514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:40.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:40.812+0000 7fd4bdf27700 1 --2- 172.21.15.16:0/223100161 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd4b80af040 0x7fd4b80ce180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:40.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:40.812+0000 7fd4bdf27700 1 -- 172.21.15.16:0/223100161 >> 172.21.15.16:0/223100161 conn(0x7fd4b80cc5e0 msgr2=0x7fd4b80cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:40.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:40.812+0000 7fd4bdf27700 1 -- 172.21.15.16:0/223100161 shutdown_connections 2023-10-16T13:46:40.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:40.812+0000 7fd4bdf27700 1 -- 172.21.15.16:0/223100161 wait complete. 2023-10-16T13:46:40.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:40.813+0000 7fd4bdf27700 1 Processor -- start 2023-10-16T13:46:40.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:40.813+0000 7fd4bdf27700 1 -- start start 2023-10-16T13:46:40.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:40.814+0000 7fd4bdf27700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd4b80af040 0x7fd4b80c6c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:40.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:40.814+0000 7fd4bdf27700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd4b80ce6c0 0x7fd4b81f2860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:40.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:40.814+0000 7fd4bdf27700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4b8160ee0 0x7fd4b81f6680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:40.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:40.814+0000 7fd4bdf27700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b80c7170 con 0x7fd4b8160ee0 2023-10-16T13:46:40.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:40.814+0000 7fd4bdf27700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b80c90f0 con 0x7fd4b80af040 2023-10-16T13:46:40.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:40.814+0000 7fd4bdf27700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b80c93d0 con 0x7fd4b80ce6c0 2023-10-16T13:46:40.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:40.814+0000 7fd4b7fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4b8160ee0 0x7fd4b81f6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:40.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:40.814+0000 7fd4b7fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4b8160ee0 0x7fd4b81f6680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53536/0 (socket says 172.21.15.16:53536) 2023-10-16T13:46:40.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:40.814+0000 7fd4b7fff700 1 -- 172.21.15.16:0/1390012578 learned_addr learned my addr 172.21.15.16:0/1390012578 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:40.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:40.814+0000 7fd4b77fe700 1 --2- 172.21.15.16:0/1390012578 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd4b80af040 0x7fd4b80c6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:40.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:40.814+0000 7fd4b6ffd700 1 --2- 172.21.15.16:0/1390012578 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd4b80ce6c0 0x7fd4b81f2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:40.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:40.814+0000 7fd4b7fff700 1 -- 172.21.15.16:0/1390012578 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd4b80ce6c0 msgr2=0x7fd4b81f2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:40.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:40.814+0000 7fd4b7fff700 1 --2- 172.21.15.16:0/1390012578 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd4b80ce6c0 0x7fd4b81f2860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:40.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:40.814+0000 7fd4b7fff700 1 -- 172.21.15.16:0/1390012578 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd4b80af040 msgr2=0x7fd4b80c6c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:40.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:40.814+0000 7fd4b7fff700 1 --2- 172.21.15.16:0/1390012578 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd4b80af040 0x7fd4b80c6c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:40.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:40.814+0000 7fd4b7fff700 1 -- 172.21.15.16:0/1390012578 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4a8009550 con 0x7fd4b8160ee0 2023-10-16T13:46:40.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:40.814+0000 7fd4b7fff700 1 --2- 172.21.15.16:0/1390012578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4b8160ee0 0x7fd4b81f6680 secure :-1 s=READY pgs=2515 cs=0 l=1 rev1=1 crypto rx=0x7fd4b0007e60 tx=0x7fd4b000e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:40.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:40.815+0000 7fd4affff700 1 -- 172.21.15.16:0/1390012578 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4b000ed30 con 0x7fd4b8160ee0 2023-10-16T13:46:40.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:40.815+0000 7fd4affff700 1 -- 172.21.15.16:0/1390012578 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4b0014420 con 0x7fd4b8160ee0 2023-10-16T13:46:40.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:40.815+0000 7fd4bdf27700 1 -- 172.21.15.16:0/1390012578 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4b80c96b0 con 0x7fd4b8160ee0 2023-10-16T13:46:40.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:40.815+0000 7fd4affff700 1 -- 172.21.15.16:0/1390012578 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4b00147d0 con 0x7fd4b8160ee0 2023-10-16T13:46:40.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:40.815+0000 7fd4bdf27700 1 -- 172.21.15.16:0/1390012578 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4b80aca70 con 0x7fd4b8160ee0 2023-10-16T13:46:40.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:40.816+0000 7fd4bdf27700 1 -- 172.21.15.16:0/1390012578 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4b80ac8d0 con 0x7fd4b8160ee0 2023-10-16T13:46:40.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:40.816+0000 7fd4affff700 1 -- 172.21.15.16:0/1390012578 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 138) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd4b0014930 con 0x7fd4b8160ee0 2023-10-16T13:46:40.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:40.817+0000 7fd4affff700 1 --2- 172.21.15.16:0/1390012578 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fd49806b020 0x7fd49806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:40.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:40.817+0000 7fd4affff700 1 -- 172.21.15.16:0/1390012578 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 7021+0+0 (secure 0 0 0) 0x7fd4b008a290 con 0x7fd4b8160ee0 2023-10-16T13:46:40.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:40.817+0000 7fd4b77fe700 1 --2- 172.21.15.16:0/1390012578 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fd49806b020 0x7fd49806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:40.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:40.817+0000 7fd4b77fe700 1 --2- 172.21.15.16:0/1390012578 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fd49806b020 0x7fd49806d4e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd4a8009c40 tx=0x7fd4a8008030 comp rx=0 tx=0).ready entity=mgr.11190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:40.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:40.821+0000 7fd4affff700 1 -- 172.21.15.16:0/1390012578 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd4b80ac8d0 con 0x7fd4b8160ee0 2023-10-16T13:46:41.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.075+0000 7fd4bdf27700 1 -- 172.21.15.16:0/1390012578 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "backup_fs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fd4b80ac8d0 con 0x7fd4b8160ee0 2023-10-16T13:46:41.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.661+0000 7fd4affff700 1 -- 172.21.15.16:0/1390012578 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "backup_fs", "var": "allow_standby_replay", "val": "false"}]=0 v119) v1 ==== 125+0+0 (secure 0 0 0) 0x7fd4b80ac8d0 con 0x7fd4b8160ee0 2023-10-16T13:46:41.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.664+0000 7fd4bdf27700 1 -- 172.21.15.16:0/1390012578 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fd49806b020 msgr2=0x7fd49806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:41.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.664+0000 7fd4bdf27700 1 --2- 172.21.15.16:0/1390012578 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fd49806b020 0x7fd49806d4e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd4a8009c40 tx=0x7fd4a8008030 comp rx=0 tx=0).stop 2023-10-16T13:46:41.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.664+0000 7fd4bdf27700 1 -- 172.21.15.16:0/1390012578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4b8160ee0 msgr2=0x7fd4b81f6680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:41.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.664+0000 7fd4bdf27700 1 --2- 172.21.15.16:0/1390012578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4b8160ee0 0x7fd4b81f6680 secure :-1 s=READY pgs=2515 cs=0 l=1 rev1=1 crypto rx=0x7fd4b0007e60 tx=0x7fd4b000e3c0 comp rx=0 tx=0).stop 2023-10-16T13:46:41.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.664+0000 7fd4bdf27700 1 -- 172.21.15.16:0/1390012578 shutdown_connections 2023-10-16T13:46:41.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.664+0000 7fd4bdf27700 1 --2- 172.21.15.16:0/1390012578 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fd49806b020 0x7fd49806d4e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:41.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.664+0000 7fd4bdf27700 1 --2- 172.21.15.16:0/1390012578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4b8160ee0 0x7fd4b81f6680 unknown :-1 s=CLOSED pgs=2515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:41.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.664+0000 7fd4bdf27700 1 --2- 172.21.15.16:0/1390012578 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd4b80af040 0x7fd4b80c6c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:41.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.664+0000 7fd4bdf27700 1 --2- 172.21.15.16:0/1390012578 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd4b80ce6c0 0x7fd4b81f2860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:41.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.664+0000 7fd4bdf27700 1 -- 172.21.15.16:0/1390012578 >> 172.21.15.16:0/1390012578 conn(0x7fd4b80cc5e0 msgr2=0x7fd4b806aea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:41.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.664+0000 7fd4bdf27700 1 -- 172.21.15.16:0/1390012578 shutdown_connections 2023-10-16T13:46:41.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.664+0000 7fd4bdf27700 1 -- 172.21.15.16:0/1390012578 wait complete. 2023-10-16T13:46:41.682 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'backup_fs' 2023-10-16T13:46:41.682 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:46:41.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.812+0000 7fbb9e8de700 1 Processor -- start 2023-10-16T13:46:41.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.812+0000 7fbb9e8de700 1 -- start start 2023-10-16T13:46:41.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.813+0000 7fbb9e8de700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbb9815d750 0x7fbb980cf920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:41.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.813+0000 7fbb9e8de700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbb9815db30 0x7fbb9806c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:41.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.813+0000 7fbb9e8de700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fbb98161490 0x7fbb98161920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:41.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.813+0000 7fbb9e8de700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb98161e60 con 0x7fbb9815db30 2023-10-16T13:46:41.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.813+0000 7fbb9e8de700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb980cfe60 con 0x7fbb98161490 2023-10-16T13:46:41.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.813+0000 7fbb9e8de700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb980d0070 con 0x7fbb9815d750 2023-10-16T13:46:41.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.813+0000 7fbb977fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbb9815db30 0x7fbb9806c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:41.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.813+0000 7fbb977fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbb9815db30 0x7fbb9806c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53542/0 (socket says 172.21.15.16:53542) 2023-10-16T13:46:41.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.813+0000 7fbb977fe700 1 -- 172.21.15.16:0/2796045121 learned_addr learned my addr 172.21.15.16:0/2796045121 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:41.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.814+0000 7fbb97fff700 1 --2- 172.21.15.16:0/2796045121 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbb9815d750 0x7fbb980cf920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:41.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.814+0000 7fbb9ce7b700 1 --2- 172.21.15.16:0/2796045121 >> v2:172.21.15.49:3300/0 conn(0x7fbb98161490 0x7fbb98161920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:41.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.814+0000 7fbb977fe700 1 -- 172.21.15.16:0/2796045121 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbb9815d750 msgr2=0x7fbb980cf920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:41.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.814+0000 7fbb977fe700 1 --2- 172.21.15.16:0/2796045121 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbb9815d750 0x7fbb980cf920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:41.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.814+0000 7fbb977fe700 1 -- 172.21.15.16:0/2796045121 >> v2:172.21.15.49:3300/0 conn(0x7fbb98161490 msgr2=0x7fbb98161920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:41.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.814+0000 7fbb977fe700 1 --2- 172.21.15.16:0/2796045121 >> v2:172.21.15.49:3300/0 conn(0x7fbb98161490 0x7fbb98161920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:41.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.814+0000 7fbb977fe700 1 -- 172.21.15.16:0/2796045121 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb980d0350 con 0x7fbb9815db30 2023-10-16T13:46:41.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.814+0000 7fbb9ce7b700 1 --2- 172.21.15.16:0/2796045121 >> v2:172.21.15.49:3300/0 conn(0x7fbb98161490 0x7fbb98161920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:46:41.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.814+0000 7fbb97fff700 1 --2- 172.21.15.16:0/2796045121 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbb9815d750 0x7fbb980cf920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:46:41.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.814+0000 7fbb977fe700 1 --2- 172.21.15.16:0/2796045121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb9815db30 0x7fbb9806c960 secure :-1 s=READY pgs=2516 cs=0 l=1 rev1=1 crypto rx=0x7fbb980cdad0 tx=0x7fbb8c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a49beee7605eb1e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:41.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.814+0000 7fbb96ffd700 1 -- 172.21.15.16:0/2796045121 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb8c00b870 con 0x7fbb9815db30 2023-10-16T13:46:41.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.814+0000 7fbb96ffd700 1 -- 172.21.15.16:0/2796045121 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb8c00b9d0 con 0x7fbb9815db30 2023-10-16T13:46:41.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.814+0000 7fbb96ffd700 1 -- 172.21.15.16:0/2796045121 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb8c00bd60 con 0x7fbb9815db30 2023-10-16T13:46:41.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.815+0000 7fbb9e8de700 1 -- 172.21.15.16:0/2796045121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb9815db30 msgr2=0x7fbb9806c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:41.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.815+0000 7fbb9e8de700 1 --2- 172.21.15.16:0/2796045121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb9815db30 0x7fbb9806c960 secure :-1 s=READY pgs=2516 cs=0 l=1 rev1=1 crypto rx=0x7fbb980cdad0 tx=0x7fbb8c009bd0 comp rx=0 tx=0).stop 2023-10-16T13:46:41.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.815+0000 7fbb9e8de700 1 -- 172.21.15.16:0/2796045121 shutdown_connections 2023-10-16T13:46:41.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.815+0000 7fbb9e8de700 1 --2- 172.21.15.16:0/2796045121 >> v2:172.21.15.49:3300/0 conn(0x7fbb98161490 0x7fbb98161920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:41.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.815+0000 7fbb9e8de700 1 --2- 172.21.15.16:0/2796045121 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbb9815d750 0x7fbb980cf920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:41.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.815+0000 7fbb9e8de700 1 --2- 172.21.15.16:0/2796045121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb9815db30 0x7fbb9806c960 unknown :-1 s=CLOSED pgs=2516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:41.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.815+0000 7fbb9e8de700 1 -- 172.21.15.16:0/2796045121 >> 172.21.15.16:0/2796045121 conn(0x7fbb980d14b0 msgr2=0x7fbb980d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:41.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.815+0000 7fbb9e8de700 1 -- 172.21.15.16:0/2796045121 shutdown_connections 2023-10-16T13:46:41.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.815+0000 7fbb9e8de700 1 -- 172.21.15.16:0/2796045121 wait complete. 2023-10-16T13:46:41.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.816+0000 7fbb9e8de700 1 Processor -- start 2023-10-16T13:46:41.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.816+0000 7fbb9e8de700 1 -- start start 2023-10-16T13:46:41.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.817+0000 7fbb9e8de700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb9815d750 0x7fbb98150a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:41.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.817+0000 7fbb9e8de700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbb9815db30 0x7fbb9809ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:41.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.817+0000 7fbb9e8de700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbb98161490 0x7fbb980c6f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:41.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.817+0000 7fbb9e8de700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb9809f1a0 con 0x7fbb9815d750 2023-10-16T13:46:41.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.817+0000 7fbb9e8de700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb9809e620 con 0x7fbb9815db30 2023-10-16T13:46:41.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.817+0000 7fbb9e8de700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb9809e900 con 0x7fbb98161490 2023-10-16T13:46:41.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.817+0000 7fbb97fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb9815d750 0x7fbb98150a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:41.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.817+0000 7fbb97fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb9815d750 0x7fbb98150a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53550/0 (socket says 172.21.15.16:53550) 2023-10-16T13:46:41.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.817+0000 7fbb97fff700 1 -- 172.21.15.16:0/4180251212 learned_addr learned my addr 172.21.15.16:0/4180251212 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:41.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.817+0000 7fbb977fe700 1 --2- 172.21.15.16:0/4180251212 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbb9815db30 0x7fbb9809ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:41.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.817+0000 7fbb9ce7b700 1 --2- 172.21.15.16:0/4180251212 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbb98161490 0x7fbb980c6f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:41.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.817+0000 7fbb97fff700 1 -- 172.21.15.16:0/4180251212 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbb98161490 msgr2=0x7fbb980c6f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:41.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.817+0000 7fbb97fff700 1 --2- 172.21.15.16:0/4180251212 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbb98161490 0x7fbb980c6f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:41.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.817+0000 7fbb97fff700 1 -- 172.21.15.16:0/4180251212 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbb9815db30 msgr2=0x7fbb9809ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:41.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.817+0000 7fbb97fff700 1 --2- 172.21.15.16:0/4180251212 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbb9815db30 0x7fbb9809ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:41.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.817+0000 7fbb97fff700 1 -- 172.21.15.16:0/4180251212 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb8c009550 con 0x7fbb9815d750 2023-10-16T13:46:41.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.817+0000 7fbb9ce7b700 1 --2- 172.21.15.16:0/4180251212 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbb98161490 0x7fbb980c6f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:46:41.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.818+0000 7fbb97fff700 1 --2- 172.21.15.16:0/4180251212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb9815d750 0x7fbb98150a40 secure :-1 s=READY pgs=2517 cs=0 l=1 rev1=1 crypto rx=0x7fbb8800e810 tx=0x7fbb8800eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:41.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.818+0000 7fbb94ff9700 1 -- 172.21.15.16:0/4180251212 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb8800c070 con 0x7fbb9815d750 2023-10-16T13:46:41.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.818+0000 7fbb94ff9700 1 -- 172.21.15.16:0/4180251212 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb88005c20 con 0x7fbb9815d750 2023-10-16T13:46:41.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.818+0000 7fbb9e8de700 1 -- 172.21.15.16:0/4180251212 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb9809ebe0 con 0x7fbb9815d750 2023-10-16T13:46:41.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.818+0000 7fbb94ff9700 1 -- 172.21.15.16:0/4180251212 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb88004030 con 0x7fbb9815d750 2023-10-16T13:46:41.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.818+0000 7fbb9e8de700 1 -- 172.21.15.16:0/4180251212 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb9809e190 con 0x7fbb9815d750 2023-10-16T13:46:41.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.819+0000 7fbb9e8de700 1 -- 172.21.15.16:0/4180251212 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb9809e380 con 0x7fbb9815d750 2023-10-16T13:46:41.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.819+0000 7fbb94ff9700 1 -- 172.21.15.16:0/4180251212 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 138) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fbb88004190 con 0x7fbb9815d750 2023-10-16T13:46:41.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.820+0000 7fbb94ff9700 1 --2- 172.21.15.16:0/4180251212 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fbb8406b020 0x7fbb8406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:41.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.820+0000 7fbb94ff9700 1 -- 172.21.15.16:0/4180251212 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 7021+0+0 (secure 0 0 0) 0x7fbb88089550 con 0x7fbb9815d750 2023-10-16T13:46:41.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.820+0000 7fbb977fe700 1 --2- 172.21.15.16:0/4180251212 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fbb8406b020 0x7fbb8406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:41.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.820+0000 7fbb977fe700 1 --2- 172.21.15.16:0/4180251212 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fbb8406b020 0x7fbb8406d4e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fbb8c000c00 tx=0x7fbb8c008030 comp rx=0 tx=0).ready entity=mgr.11190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:41.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:41.824+0000 7fbb94ff9700 1 -- 172.21.15.16:0/4180251212 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbb88053eb0 con 0x7fbb9815d750 2023-10-16T13:46:42.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.073+0000 7fbb9e8de700 1 -- 172.21.15.16:0/4180251212 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbb9804ebb0 con 0x7fbb9815d750 2023-10-16T13:46:42.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.074+0000 7fbb94ff9700 1 -- 172.21.15.16:0/4180251212 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) v1 ==== 95+0+6154 (secure 0 0 0) 0x7fbb880574d0 con 0x7fbb9815d750 2023-10-16T13:46:42.078 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:46:42.078 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":119,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10516,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/4065699432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":4065699432},{"type":"v1","addr":"172.21.15.49:6834","nonce":4065699432}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":108},{"gid":11415,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/1373048456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1373048456},{"type":"v1","addr":"172.21.15.16:6836","nonce":1373048456}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":108}],"filesystems":[{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:46:24.755313+0000","modified":"2023-10-16T13:46:28.011141+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11430},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11430":{"gid":11430,"name":"b","rank":0,"incarnation":109,"state":"up:active","state_seq":3,"addr":"172.21.15.49:6835/2905760119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":2905760119},{"type":"v1","addr":"172.21.15.49:6835","nonce":2905760119}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13},{"mdsmap":{"epoch":119,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:46:38.053490+0000","modified":"2023-10-16T13:46:41.076680+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11418},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11418":{"gid":11418,"name":"a","rank":0,"incarnation":115,"state":"up:active","state_seq":6,"addr":"172.21.15.16:6837/3860406853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":3860406853},{"type":"v1","addr":"172.21.15.16:6837","nonce":3860406853}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2023-10-16T13:46:42.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.076+0000 7fbb9e8de700 1 -- 172.21.15.16:0/4180251212 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fbb8406b020 msgr2=0x7fbb8406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:42.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.076+0000 7fbb9e8de700 1 --2- 172.21.15.16:0/4180251212 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fbb8406b020 0x7fbb8406d4e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fbb8c000c00 tx=0x7fbb8c008030 comp rx=0 tx=0).stop 2023-10-16T13:46:42.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.076+0000 7fbb9e8de700 1 -- 172.21.15.16:0/4180251212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb9815d750 msgr2=0x7fbb98150a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:42.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.076+0000 7fbb9e8de700 1 --2- 172.21.15.16:0/4180251212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb9815d750 0x7fbb98150a40 secure :-1 s=READY pgs=2517 cs=0 l=1 rev1=1 crypto rx=0x7fbb8800e810 tx=0x7fbb8800eb20 comp rx=0 tx=0).stop 2023-10-16T13:46:42.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.076+0000 7fbb9e8de700 1 -- 172.21.15.16:0/4180251212 shutdown_connections 2023-10-16T13:46:42.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.076+0000 7fbb9e8de700 1 --2- 172.21.15.16:0/4180251212 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbb98161490 0x7fbb980c6f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:42.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.076+0000 7fbb9e8de700 1 --2- 172.21.15.16:0/4180251212 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7fbb8406b020 0x7fbb8406d4e0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:42.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.076+0000 7fbb9e8de700 1 --2- 172.21.15.16:0/4180251212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb9815d750 0x7fbb98150a40 unknown :-1 s=CLOSED pgs=2517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:42.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.076+0000 7fbb9e8de700 1 --2- 172.21.15.16:0/4180251212 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbb9815db30 0x7fbb9809ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:42.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.076+0000 7fbb9e8de700 1 -- 172.21.15.16:0/4180251212 >> 172.21.15.16:0/4180251212 conn(0x7fbb980d14b0 msgr2=0x7fbb9815c6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:42.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.076+0000 7fbb9e8de700 1 -- 172.21.15.16:0/4180251212 shutdown_connections 2023-10-16T13:46:42.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.076+0000 7fbb9e8de700 1 -- 172.21.15.16:0/4180251212 wait complete. 2023-10-16T13:46:42.083 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 119 2023-10-16T13:46:42.092 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:46:42.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.214+0000 7f95fdfd4700 1 Processor -- start 2023-10-16T13:46:42.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.214+0000 7f95fdfd4700 1 -- start start 2023-10-16T13:46:42.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.214+0000 7f95fdfd4700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f95f815b860 0x7f95f806a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:42.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.214+0000 7f95fdfd4700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f95f80b4490 0x7f95f806ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:42.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.214+0000 7f95fdfd4700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95f80d11c0 con 0x7f95f80b2be0 2023-10-16T13:46:42.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.214+0000 7f95fdfd4700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95f80d03d0 con 0x7f95f815b860 2023-10-16T13:46:42.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.214+0000 7f95fdfd4700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95f80cd360 con 0x7f95f80b4490 2023-10-16T13:46:42.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.215+0000 7f95f6ffd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f95f80b2be0 0x7f95f80719e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42832/0 (socket says 172.21.15.16:42832) 2023-10-16T13:46:42.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.215+0000 7f95f6ffd700 1 -- 172.21.15.16:0/3764563274 learned_addr learned my addr 172.21.15.16:0/3764563274 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:46:42.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.215+0000 7f95f7fff700 1 --2- 172.21.15.16:0/3764563274 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f95f80b4490 0x7f95f806ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:42.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.215+0000 7f95f77fe700 1 --2- 172.21.15.16:0/3764563274 >> v2:172.21.15.49:3300/0 conn(0x7f95f815b860 0x7f95f806a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:42.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.215+0000 7f95f67fc700 1 -- 172.21.15.16:0/3764563274 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3221249371 0 0) 0x7f95f80d11c0 con 0x7f95f80b2be0 2023-10-16T13:46:42.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.215+0000 7f95f67fc700 1 -- 172.21.15.16:0/3764563274 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95e00035b0 con 0x7f95f80b2be0 2023-10-16T13:46:42.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.215+0000 7f95f67fc700 1 -- 172.21.15.16:0/3764563274 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3509305735 0 0) 0x7f95e00035b0 con 0x7f95f80b2be0 2023-10-16T13:46:42.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.215+0000 7f95f77fe700 1 -- 172.21.15.16:0/3764563274 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f95f80b4490 msgr2=0x7f95f806ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:42.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.215+0000 7f95f77fe700 1 --2- 172.21.15.16:0/3764563274 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f95f80b4490 0x7f95f806ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:42.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.215+0000 7f95f77fe700 1 -- 172.21.15.16:0/3764563274 >> v1:172.21.15.16:6789/0 conn(0x7f95f80b2be0 legacy=0x7f95f80719e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:42.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.215+0000 7f95f77fe700 1 -- 172.21.15.16:0/3764563274 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95f80d05b0 con 0x7f95f815b860 2023-10-16T13:46:42.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.215+0000 7f95f7fff700 1 --2- 172.21.15.16:0/3764563274 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f95f80b4490 0x7f95f806ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:46:42.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.215+0000 7f95f77fe700 1 --2- 172.21.15.16:0/3764563274 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f95f815b860 0x7f95f806a7e0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f95e8009840 tx=0x7f95e8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6b0bf02d0d5c2a0e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:42.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.216+0000 7f95f67fc700 1 -- 172.21.15.16:0/3764563274 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95ec001670 con 0x7f95f815b860 2023-10-16T13:46:42.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.216+0000 7f95f67fc700 1 -- 172.21.15.16:0/3764563274 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95e8004030 con 0x7f95f815b860 2023-10-16T13:46:42.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.216+0000 7f95f67fc700 1 -- 172.21.15.16:0/3764563274 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95e80043c0 con 0x7f95f815b860 2023-10-16T13:46:42.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.216+0000 7f95fdfd4700 1 -- 172.21.15.16:0/3764563274 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f95f815b860 msgr2=0x7f95f806a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:42.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.216+0000 7f95fdfd4700 1 --2- 172.21.15.16:0/3764563274 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f95f815b860 0x7f95f806a7e0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f95e8009840 tx=0x7f95e8009b50 comp rx=0 tx=0).stop 2023-10-16T13:46:42.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.216+0000 7f95fdfd4700 1 -- 172.21.15.16:0/3764563274 shutdown_connections 2023-10-16T13:46:42.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.216+0000 7f95fdfd4700 1 --2- 172.21.15.16:0/3764563274 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f95f80b4490 0x7f95f806ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:42.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.216+0000 7f95fdfd4700 1 --2- 172.21.15.16:0/3764563274 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f95f815b860 0x7f95f806a7e0 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:42.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.216+0000 7f95fdfd4700 1 -- 172.21.15.16:0/3764563274 >> 172.21.15.16:0/3764563274 conn(0x7f95f80b1fd0 msgr2=0x7f95f80b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:42.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.216+0000 7f95fdfd4700 1 -- 172.21.15.16:0/3764563274 shutdown_connections 2023-10-16T13:46:42.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.216+0000 7f95fdfd4700 1 -- 172.21.15.16:0/3764563274 wait complete. 2023-10-16T13:46:42.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.217+0000 7f95fdfd4700 1 Processor -- start 2023-10-16T13:46:42.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.218+0000 7f95fdfd4700 1 -- start start 2023-10-16T13:46:42.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.218+0000 7f95fdfd4700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f95f80b2be0 0x7f95f815c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:42.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.218+0000 7f95fdfd4700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f95f80b4490 0x7f95f8151a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:42.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.218+0000 7f95fdfd4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95f815b860 0x7f95f80cf590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:42.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.218+0000 7f95fdfd4700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95f80cf020 con 0x7f95f815b860 2023-10-16T13:46:42.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.218+0000 7f95fdfd4700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95f815cc00 con 0x7f95f80b4490 2023-10-16T13:46:42.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.218+0000 7f95fdfd4700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95f80ce4a0 con 0x7f95f80b2be0 2023-10-16T13:46:42.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.218+0000 7f95f7fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95f815b860 0x7f95f80cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:42.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.218+0000 7f95f7fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95f815b860 0x7f95f80cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53566/0 (socket says 172.21.15.16:53566) 2023-10-16T13:46:42.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.218+0000 7f95f7fff700 1 -- 172.21.15.16:0/3188313138 learned_addr learned my addr 172.21.15.16:0/3188313138 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:42.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.219+0000 7f95f77fe700 1 --2- 172.21.15.16:0/3188313138 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f95f80b2be0 0x7f95f815c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:42.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.219+0000 7f95f6ffd700 1 --2- 172.21.15.16:0/3188313138 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f95f80b4490 0x7f95f8151a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:42.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.219+0000 7f95f7fff700 1 -- 172.21.15.16:0/3188313138 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f95f80b2be0 msgr2=0x7f95f815c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:42.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.219+0000 7f95f7fff700 1 --2- 172.21.15.16:0/3188313138 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f95f80b2be0 0x7f95f815c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:42.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.219+0000 7f95f7fff700 1 -- 172.21.15.16:0/3188313138 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f95f80b4490 msgr2=0x7f95f8151a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:42.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.219+0000 7f95f7fff700 1 --2- 172.21.15.16:0/3188313138 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f95f80b4490 0x7f95f8151a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:42.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.219+0000 7f95f7fff700 1 -- 172.21.15.16:0/3188313138 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95f0009a70 con 0x7f95f815b860 2023-10-16T13:46:42.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.219+0000 7f95f7fff700 1 --2- 172.21.15.16:0/3188313138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95f815b860 0x7f95f80cf590 secure :-1 s=READY pgs=2519 cs=0 l=1 rev1=1 crypto rx=0x7f95f000ef90 tx=0x7f95f000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:42.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.219+0000 7f95dffff700 1 -- 172.21.15.16:0/3188313138 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95f0009cd0 con 0x7f95f815b860 2023-10-16T13:46:42.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.219+0000 7f95dffff700 1 -- 172.21.15.16:0/3188313138 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95f000f040 con 0x7f95f815b860 2023-10-16T13:46:42.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.219+0000 7f95dffff700 1 -- 172.21.15.16:0/3188313138 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95f0014480 con 0x7f95f815b860 2023-10-16T13:46:42.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.219+0000 7f95fdfd4700 1 -- 172.21.15.16:0/3188313138 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95e80094a0 con 0x7f95f815b860 2023-10-16T13:46:42.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.219+0000 7f95fdfd4700 1 -- 172.21.15.16:0/3188313138 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95f80cea60 con 0x7f95f815b860 2023-10-16T13:46:42.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.220+0000 7f95fdfd4700 1 -- 172.21.15.16:0/3188313138 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95f804ebb0 con 0x7f95f815b860 2023-10-16T13:46:42.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.221+0000 7f95dffff700 1 -- 172.21.15.16:0/3188313138 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 138) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f95f00145e0 con 0x7f95f815b860 2023-10-16T13:46:42.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.221+0000 7f95dffff700 1 --2- 172.21.15.16:0/3188313138 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f95e406b020 0x7f95e406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:42.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.221+0000 7f95dffff700 1 -- 172.21.15.16:0/3188313138 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 7021+0+0 (secure 0 0 0) 0x7f95f008aae0 con 0x7f95f815b860 2023-10-16T13:46:42.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.221+0000 7f95f77fe700 1 --2- 172.21.15.16:0/3188313138 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f95e406b020 0x7f95e406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:42.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.222+0000 7f95f77fe700 1 --2- 172.21.15.16:0/3188313138 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f95e406b020 0x7f95e406d4e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f95e80090b0 tx=0x7f95e8008030 comp rx=0 tx=0).ready entity=mgr.11190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:42.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.226+0000 7f95dffff700 1 -- 172.21.15.16:0/3188313138 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f95f0055450 con 0x7f95f815b860 2023-10-16T13:46:42.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.429+0000 7f95fdfd4700 1 -- 172.21.15.16:0/3188313138 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f95f80b5bf0 con 0x7f95f815b860 2023-10-16T13:46:42.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.430+0000 7f95dffff700 1 -- 172.21.15.16:0/3188313138 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v229) v1 ==== 74+0+20537 (secure 0 0 0) 0x7f95f0058a70 con 0x7f95f815b860 2023-10-16T13:46:42.434 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:46:42.434 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":229,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:46:39.646680+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2023-10-16T13:46:22.489653+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"224","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2023-10-16T13:46:23.729509+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"225","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T13:46:35.800444+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"228","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"backup_fs_data","create_time":"2023-10-16T13:46:37.035323+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"229","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:46:42.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.433+0000 7f95fdfd4700 1 -- 172.21.15.16:0/3188313138 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f95e406b020 msgr2=0x7f95e406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:42.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.433+0000 7f95fdfd4700 1 --2- 172.21.15.16:0/3188313138 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f95e406b020 0x7f95e406d4e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f95e80090b0 tx=0x7f95e8008030 comp rx=0 tx=0).stop 2023-10-16T13:46:42.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.433+0000 7f95fdfd4700 1 -- 172.21.15.16:0/3188313138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95f815b860 msgr2=0x7f95f80cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:42.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.433+0000 7f95fdfd4700 1 --2- 172.21.15.16:0/3188313138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95f815b860 0x7f95f80cf590 secure :-1 s=READY pgs=2519 cs=0 l=1 rev1=1 crypto rx=0x7f95f000ef90 tx=0x7f95f000c3a0 comp rx=0 tx=0).stop 2023-10-16T13:46:42.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.433+0000 7f95fdfd4700 1 -- 172.21.15.16:0/3188313138 shutdown_connections 2023-10-16T13:46:42.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.433+0000 7f95fdfd4700 1 --2- 172.21.15.16:0/3188313138 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f95e406b020 0x7f95e406d4e0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:42.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.433+0000 7f95fdfd4700 1 --2- 172.21.15.16:0/3188313138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95f815b860 0x7f95f80cf590 unknown :-1 s=CLOSED pgs=2519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:42.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.433+0000 7f95fdfd4700 1 --2- 172.21.15.16:0/3188313138 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f95f80b2be0 0x7f95f815c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:42.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.433+0000 7f95fdfd4700 1 --2- 172.21.15.16:0/3188313138 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f95f80b4490 0x7f95f8151a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:42.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.433+0000 7f95fdfd4700 1 -- 172.21.15.16:0/3188313138 >> 172.21.15.16:0/3188313138 conn(0x7f95f80b1fd0 msgr2=0x7f95f815a8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:42.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.433+0000 7f95fdfd4700 1 -- 172.21.15.16:0/3188313138 shutdown_connections 2023-10-16T13:46:42.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.433+0000 7f95fdfd4700 1 -- 172.21.15.16:0/3188313138 wait complete. 2023-10-16T13:46:42.449 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-16T13:46:42.449 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T13:46:42.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.570+0000 7f9aa4612700 1 Processor -- start 2023-10-16T13:46:42.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.570+0000 7f9aa4612700 1 -- start start 2023-10-16T13:46:42.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.571+0000 7f9aa4612700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f9a9c15da40 0x7f9a9c0b2f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:42.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.571+0000 7f9aa4612700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9a9c0b34c0 0x7f9a9c06bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:42.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.571+0000 7f9aa4612700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a9c06ccc0 con 0x7f9a9c06c210 2023-10-16T13:46:42.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.571+0000 7f9aa4612700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a9c0cc4d0 con 0x7f9a9c0b34c0 2023-10-16T13:46:42.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.571+0000 7f9aa4612700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a9c0b3950 con 0x7f9a9c15da40 2023-10-16T13:46:42.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.571+0000 7f9aa1bad700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9a9c0b34c0 0x7f9a9c06bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:42.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.572+0000 7f9aa1bad700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9a9c0b34c0 0x7f9a9c06bcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53570/0 (socket says 172.21.15.16:53570) 2023-10-16T13:46:42.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.572+0000 7f9aa1bad700 1 -- 172.21.15.16:0/3436053767 learned_addr learned my addr 172.21.15.16:0/3436053767 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:42.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.572+0000 7f9aa2baf700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f9a9c06c210 0x7f9a9c070530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42848/0 (socket says 172.21.15.16:42848) 2023-10-16T13:46:42.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.572+0000 7f9aa23ae700 1 --2- 172.21.15.16:0/3436053767 >> v2:172.21.15.49:3300/0 conn(0x7f9a9c15da40 0x7f9a9c0b2f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:42.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.572+0000 7f9aa1bad700 1 -- 172.21.15.16:0/3436053767 >> v2:172.21.15.49:3300/0 conn(0x7f9a9c15da40 msgr2=0x7f9a9c0b2f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:42.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.572+0000 7f9aa1bad700 1 --2- 172.21.15.16:0/3436053767 >> v2:172.21.15.49:3300/0 conn(0x7f9a9c15da40 0x7f9a9c0b2f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:42.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.572+0000 7f9aa1bad700 1 -- 172.21.15.16:0/3436053767 >> v1:172.21.15.16:6789/0 conn(0x7f9a9c06c210 legacy=0x7f9a9c070530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:42.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.572+0000 7f9aa1bad700 1 -- 172.21.15.16:0/3436053767 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a9c0cbb70 con 0x7f9a9c0b34c0 2023-10-16T13:46:42.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.572+0000 7f9aa13ac700 1 -- 172.21.15.16:0/3436053767 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3259243710 0 0) 0x7f9a9c06ccc0 con 0x7f9a9c06c210 2023-10-16T13:46:42.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.572+0000 7f9aa1bad700 1 --2- 172.21.15.16:0/3436053767 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a9c0b34c0 0x7f9a9c06bcd0 secure :-1 s=READY pgs=2521 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c071540 tx=0x7f9a90009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a490dd1ab3072f27 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:42.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.572+0000 7f9aa13ac700 1 -- 172.21.15.16:0/3436053767 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a90011830 con 0x7f9a9c0b34c0 2023-10-16T13:46:42.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.572+0000 7f9aa13ac700 1 -- 172.21.15.16:0/3436053767 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a90011990 con 0x7f9a9c0b34c0 2023-10-16T13:46:42.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.572+0000 7f9aa13ac700 1 -- 172.21.15.16:0/3436053767 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a90011d20 con 0x7f9a9c0b34c0 2023-10-16T13:46:42.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.573+0000 7f9aa4612700 1 -- 172.21.15.16:0/3436053767 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a9c0b34c0 msgr2=0x7f9a9c06bcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:42.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.573+0000 7f9aa4612700 1 --2- 172.21.15.16:0/3436053767 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a9c0b34c0 0x7f9a9c06bcd0 secure :-1 s=READY pgs=2521 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c071540 tx=0x7f9a90009bd0 comp rx=0 tx=0).stop 2023-10-16T13:46:42.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.573+0000 7f9aa4612700 1 -- 172.21.15.16:0/3436053767 shutdown_connections 2023-10-16T13:46:42.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.573+0000 7f9aa4612700 1 --2- 172.21.15.16:0/3436053767 >> v2:172.21.15.49:3300/0 conn(0x7f9a9c15da40 0x7f9a9c0b2f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:42.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.573+0000 7f9aa4612700 1 --2- 172.21.15.16:0/3436053767 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a9c0b34c0 0x7f9a9c06bcd0 unknown :-1 s=CLOSED pgs=2521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:42.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.573+0000 7f9aa4612700 1 -- 172.21.15.16:0/3436053767 >> 172.21.15.16:0/3436053767 conn(0x7f9a9c0cdf80 msgr2=0x7f9a9c0ce390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:42.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.573+0000 7f9aa4612700 1 -- 172.21.15.16:0/3436053767 shutdown_connections 2023-10-16T13:46:42.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.573+0000 7f9aa4612700 1 -- 172.21.15.16:0/3436053767 wait complete. 2023-10-16T13:46:42.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.574+0000 7f9aa4612700 1 Processor -- start 2023-10-16T13:46:42.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.575+0000 7f9aa4612700 1 -- start start 2023-10-16T13:46:42.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.575+0000 7f9aa4612700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9a9c06c210 0x7f9a9c1f7140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:42.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.575+0000 7f9aa4612700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9a9c0b34c0 0x7f9a9c0ca5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:42.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.575+0000 7f9aa4612700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a9c15da40 0x7f9a9c1494d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:42.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.575+0000 7f9aa4612700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a9c1f7680 con 0x7f9a9c15da40 2023-10-16T13:46:42.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.576+0000 7f9aa4612700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a9c0cab20 con 0x7f9a9c0b34c0 2023-10-16T13:46:42.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.576+0000 7f9aa4612700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a9c0cad80 con 0x7f9a9c06c210 2023-10-16T13:46:42.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.576+0000 7f9aa2baf700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a9c15da40 0x7f9a9c1494d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:42.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.576+0000 7f9aa2baf700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a9c15da40 0x7f9a9c1494d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53580/0 (socket says 172.21.15.16:53580) 2023-10-16T13:46:42.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.576+0000 7f9aa2baf700 1 -- 172.21.15.16:0/916281161 learned_addr learned my addr 172.21.15.16:0/916281161 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:42.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.576+0000 7f9aa23ae700 1 --2- 172.21.15.16:0/916281161 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9a9c06c210 0x7f9a9c1f7140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:42.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.576+0000 7f9aa1bad700 1 --2- 172.21.15.16:0/916281161 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9a9c0b34c0 0x7f9a9c0ca5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:42.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.576+0000 7f9aa2baf700 1 -- 172.21.15.16:0/916281161 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9a9c06c210 msgr2=0x7f9a9c1f7140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:42.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.576+0000 7f9aa2baf700 1 --2- 172.21.15.16:0/916281161 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9a9c06c210 0x7f9a9c1f7140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:42.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.576+0000 7f9aa2baf700 1 -- 172.21.15.16:0/916281161 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9a9c0b34c0 msgr2=0x7f9a9c0ca5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:42.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.576+0000 7f9aa2baf700 1 --2- 172.21.15.16:0/916281161 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9a9c0b34c0 0x7f9a9c0ca5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:42.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.576+0000 7f9aa2baf700 1 -- 172.21.15.16:0/916281161 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a90009550 con 0x7f9a9c15da40 2023-10-16T13:46:42.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.576+0000 7f9aa2baf700 1 --2- 172.21.15.16:0/916281161 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a9c15da40 0x7f9a9c1494d0 secure :-1 s=READY pgs=2522 cs=0 l=1 rev1=1 crypto rx=0x7f9a94002230 tx=0x7f9a9400cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:42.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.576+0000 7f9a8effd700 1 -- 172.21.15.16:0/916281161 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a9400aac0 con 0x7f9a9c15da40 2023-10-16T13:46:42.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.576+0000 7f9a8effd700 1 -- 172.21.15.16:0/916281161 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a9400ac40 con 0x7f9a9c15da40 2023-10-16T13:46:42.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.576+0000 7f9aa4612700 1 -- 172.21.15.16:0/916281161 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a9c0cb060 con 0x7f9a9c15da40 2023-10-16T13:46:42.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.576+0000 7f9a8effd700 1 -- 172.21.15.16:0/916281161 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a940070d0 con 0x7f9a9c15da40 2023-10-16T13:46:42.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.576+0000 7f9aa4612700 1 -- 172.21.15.16:0/916281161 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a9c0cb620 con 0x7f9a9c15da40 2023-10-16T13:46:42.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.577+0000 7f9aa4612700 1 -- 172.21.15.16:0/916281161 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a9c0d0320 con 0x7f9a9c15da40 2023-10-16T13:46:42.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.578+0000 7f9a8effd700 1 -- 172.21.15.16:0/916281161 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 138) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9a94007250 con 0x7f9a9c15da40 2023-10-16T13:46:42.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.578+0000 7f9a8effd700 1 --2- 172.21.15.16:0/916281161 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f9a8006b020 0x7f9a8006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:42.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.579+0000 7f9aa23ae700 1 --2- 172.21.15.16:0/916281161 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f9a8006b020 0x7f9a8006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:42.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.579+0000 7f9a8effd700 1 -- 172.21.15.16:0/916281161 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 7021+0+0 (secure 0 0 0) 0x7f9a94089a30 con 0x7f9a9c15da40 2023-10-16T13:46:42.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.579+0000 7f9aa23ae700 1 --2- 172.21.15.16:0/916281161 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f9a8006b020 0x7f9a8006d4e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9a98009b40 tx=0x7f9a98009210 comp rx=0 tx=0).ready entity=mgr.11190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:42.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.583+0000 7f9a8effd700 1 -- 172.21.15.16:0/916281161 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9a940544b0 con 0x7f9a9c15da40 2023-10-16T13:46:42.786 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.783+0000 7f9aa4612700 1 -- 172.21.15.16:0/916281161 --> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9a9c0cbe10 con 0x7f9a8006b020 2023-10-16T13:46:42.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.798+0000 7f9a8effd700 1 -- 172.21.15.16:0/916281161 <== mgr.11190 v2:172.21.15.16:6832/3171422553 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+528752 (secure 0 0 0) 0x7f9a98011810 con 0x7f9a8006b020 2023-10-16T13:46:42.807 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:46:42.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.802+0000 7f9aa4612700 1 -- 172.21.15.16:0/916281161 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f9a8006b020 msgr2=0x7f9a8006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:42.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.802+0000 7f9aa4612700 1 --2- 172.21.15.16:0/916281161 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f9a8006b020 0x7f9a8006d4e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9a98009b40 tx=0x7f9a98009210 comp rx=0 tx=0).stop 2023-10-16T13:46:42.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.802+0000 7f9aa4612700 1 -- 172.21.15.16:0/916281161 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a9c15da40 msgr2=0x7f9a9c1494d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:42.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.802+0000 7f9aa4612700 1 --2- 172.21.15.16:0/916281161 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a9c15da40 0x7f9a9c1494d0 secure :-1 s=READY pgs=2522 cs=0 l=1 rev1=1 crypto rx=0x7f9a94002230 tx=0x7f9a9400cf50 comp rx=0 tx=0).stop 2023-10-16T13:46:42.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.802+0000 7f9aa4612700 1 -- 172.21.15.16:0/916281161 shutdown_connections 2023-10-16T13:46:42.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.802+0000 7f9aa4612700 1 --2- 172.21.15.16:0/916281161 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f9a8006b020 0x7f9a8006d4e0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:42.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.802+0000 7f9aa4612700 1 --2- 172.21.15.16:0/916281161 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a9c15da40 0x7f9a9c1494d0 unknown :-1 s=CLOSED pgs=2522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:42.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.802+0000 7f9aa4612700 1 --2- 172.21.15.16:0/916281161 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9a9c06c210 0x7f9a9c1f7140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:42.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.802+0000 7f9aa4612700 1 --2- 172.21.15.16:0/916281161 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9a9c0b34c0 0x7f9a9c0ca5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:42.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.802+0000 7f9aa4612700 1 -- 172.21.15.16:0/916281161 >> 172.21.15.16:0/916281161 conn(0x7f9a9c0cdf80 msgr2=0x7f9a9c160e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:42.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.802+0000 7f9aa4612700 1 -- 172.21.15.16:0/916281161 shutdown_connections 2023-10-16T13:46:42.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.802+0000 7f9aa4612700 1 -- 172.21.15.16:0/916281161 wait complete. 2023-10-16T13:46:42.813 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-10-16T13:46:42.818 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":35,"stamp":"2023-10-16T13:46:42.180406+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596435,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":671,"num_write_kb":2446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":428,"ondisk_log_size":428,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":378,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":389812,"kb_used_data":23740,"kb_used_omap":12,"kb_used_meta":365939,"kb_avail":754584908,"statfs":{"total":773094113280,"available":772694945792,"internally_reserved":0,"allocated":24309760,"data_stored":13053047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12740,"internal_metadata":374722108},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.557370"},"pg_stats":[{"pgid":"1.0","version":"214'348","reported_seq":1206,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.645999+0000","last_change":"2023-10-16T12:10:34.746820+0000","last_active":"2023-10-16T13:46:38.645999+0000","last_peered":"2023-10-16T13:46:38.645999+0000","last_clean":"2023-10-16T13:46:38.645999+0000","last_became_active":"2023-10-16T12:10:34.745574+0000","last_became_peered":"2023-10-16T12:10:34.745574+0000","last_unstale":"2023-10-16T13:46:38.645999+0000","last_undegraded":"2023-10-16T13:46:38.645999+0000","last_fullsized":"2023-10-16T13:46:38.645999+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_clean_scrub_stamp":"2023-10-16T12:10:33.577111+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:14.461986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1a","version":"224'1","reported_seq":17,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.647029+0000","last_change":"2023-10-16T13:46:24.344199+0000","last_active":"2023-10-16T13:46:38.647029+0000","last_peered":"2023-10-16T13:46:38.647029+0000","last_clean":"2023-10-16T13:46:38.647029+0000","last_became_active":"2023-10-16T13:46:24.343947+0000","last_became_peered":"2023-10-16T13:46:24.343947+0000","last_unstale":"2023-10-16T13:46:38.647029+0000","last_undegraded":"2023-10-16T13:46:38.647029+0000","last_fullsized":"2023-10-16T13:46:38.647029+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:17:17.355917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.646904+0000","last_change":"2023-10-16T13:46:25.579532+0000","last_active":"2023-10-16T13:46:38.646904+0000","last_peered":"2023-10-16T13:46:38.646904+0000","last_clean":"2023-10-16T13:46:38.646904+0000","last_became_active":"2023-10-16T13:46:25.579042+0000","last_became_peered":"2023-10-16T13:46:25.579042+0000","last_unstale":"2023-10-16T13:46:38.646904+0000","last_undegraded":"2023-10-16T13:46:38.646904+0000","last_fullsized":"2023-10-16T13:46:38.646904+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:44:22.609007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.1c","version":"228'2","reported_seq":14,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.655435+0000","last_change":"2023-10-16T13:46:37.647219+0000","last_active":"2023-10-16T13:46:39.655435+0000","last_peered":"2023-10-16T13:46:39.655435+0000","last_clean":"2023-10-16T13:46:39.655435+0000","last_became_active":"2023-10-16T13:46:37.646935+0000","last_became_peered":"2023-10-16T13:46:37.646935+0000","last_unstale":"2023-10-16T13:46:39.655435+0000","last_undegraded":"2023-10-16T13:46:39.655435+0000","last_fullsized":"2023-10-16T13:46:39.655435+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_clean_scrub_stamp":"2023-10-16T13:46:36.617142+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:25:57.100673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.1f","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.581696+0000","last_change":"2023-10-16T13:46:38.674756+0000","last_active":"2023-10-16T13:46:40.581696+0000","last_peered":"2023-10-16T13:46:40.581696+0000","last_clean":"2023-10-16T13:46:40.581696+0000","last_became_active":"2023-10-16T13:46:38.674430+0000","last_became_peered":"2023-10-16T13:46:38.674430+0000","last_unstale":"2023-10-16T13:46:40.581696+0000","last_undegraded":"2023-10-16T13:46:40.581696+0000","last_fullsized":"2023-10-16T13:46:40.581696+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:23:02.303147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.641316+0000","last_change":"2023-10-16T13:46:24.346138+0000","last_active":"2023-10-16T13:46:38.641316+0000","last_peered":"2023-10-16T13:46:38.641316+0000","last_clean":"2023-10-16T13:46:38.641316+0000","last_became_active":"2023-10-16T13:46:24.345803+0000","last_became_peered":"2023-10-16T13:46:24.345803+0000","last_unstale":"2023-10-16T13:46:38.641316+0000","last_undegraded":"2023-10-16T13:46:38.641316+0000","last_fullsized":"2023-10-16T13:46:38.641316+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:08:14.564787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.7","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.655005+0000","last_change":"2023-10-16T13:46:25.577432+0000","last_active":"2023-10-16T13:46:38.655005+0000","last_peered":"2023-10-16T13:46:38.655005+0000","last_clean":"2023-10-16T13:46:38.655005+0000","last_became_active":"2023-10-16T13:46:25.577131+0000","last_became_peered":"2023-10-16T13:46:25.577131+0000","last_unstale":"2023-10-16T13:46:38.655005+0000","last_undegraded":"2023-10-16T13:46:38.655005+0000","last_fullsized":"2023-10-16T13:46:38.655005+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:51:29.125394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.6","version":"229'1","reported_seq":13,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.672409+0000","last_change":"2023-10-16T13:46:37.643485+0000","last_active":"2023-10-16T13:46:39.672409+0000","last_peered":"2023-10-16T13:46:39.672409+0000","last_clean":"2023-10-16T13:46:39.672409+0000","last_became_active":"2023-10-16T13:46:37.643166+0000","last_became_peered":"2023-10-16T13:46:37.643166+0000","last_unstale":"2023-10-16T13:46:39.672409+0000","last_undegraded":"2023-10-16T13:46:39.672409+0000","last_fullsized":"2023-10-16T13:46:39.672409+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_clean_scrub_stamp":"2023-10-16T13:46:36.617142+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:25:57.100673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.654859+0000","last_change":"2023-10-16T13:46:38.665257+0000","last_active":"2023-10-16T13:46:39.654859+0000","last_peered":"2023-10-16T13:46:39.654859+0000","last_clean":"2023-10-16T13:46:39.654859+0000","last_became_active":"2023-10-16T13:46:38.660312+0000","last_became_peered":"2023-10-16T13:46:38.660312+0000","last_unstale":"2023-10-16T13:46:39.654859+0000","last_undegraded":"2023-10-16T13:46:39.654859+0000","last_fullsized":"2023-10-16T13:46:39.654859+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:13:45.897327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.646596+0000","last_change":"2023-10-16T13:46:24.340170+0000","last_active":"2023-10-16T13:46:38.646596+0000","last_peered":"2023-10-16T13:46:38.646596+0000","last_clean":"2023-10-16T13:46:38.646596+0000","last_became_active":"2023-10-16T13:46:24.339859+0000","last_became_peered":"2023-10-16T13:46:24.339859+0000","last_unstale":"2023-10-16T13:46:38.646596+0000","last_undegraded":"2023-10-16T13:46:38.646596+0000","last_fullsized":"2023-10-16T13:46:38.646596+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:25:04.566680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.652713+0000","last_change":"2023-10-16T13:46:25.577347+0000","last_active":"2023-10-16T13:46:38.652713+0000","last_peered":"2023-10-16T13:46:38.652713+0000","last_clean":"2023-10-16T13:46:38.652713+0000","last_became_active":"2023-10-16T13:46:25.576780+0000","last_became_peered":"2023-10-16T13:46:25.576780+0000","last_unstale":"2023-10-16T13:46:38.652713+0000","last_undegraded":"2023-10-16T13:46:38.652713+0000","last_fullsized":"2023-10-16T13:46:38.652713+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:16:23.997738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.7","version":"228'2","reported_seq":14,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.098616+0000","last_change":"2023-10-16T13:46:37.650949+0000","last_active":"2023-10-16T13:46:40.098616+0000","last_peered":"2023-10-16T13:46:40.098616+0000","last_clean":"2023-10-16T13:46:40.098616+0000","last_became_active":"2023-10-16T13:46:37.650684+0000","last_became_peered":"2023-10-16T13:46:37.650684+0000","last_unstale":"2023-10-16T13:46:40.098616+0000","last_undegraded":"2023-10-16T13:46:40.098616+0000","last_fullsized":"2023-10-16T13:46:40.098616+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_clean_scrub_stamp":"2023-10-16T13:46:36.617142+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:58:21.317117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.653690+0000","last_change":"2023-10-16T13:46:38.661197+0000","last_active":"2023-10-16T13:46:39.653690+0000","last_peered":"2023-10-16T13:46:39.653690+0000","last_clean":"2023-10-16T13:46:39.653690+0000","last_became_active":"2023-10-16T13:46:38.660903+0000","last_became_peered":"2023-10-16T13:46:38.660903+0000","last_unstale":"2023-10-16T13:46:39.653690+0000","last_undegraded":"2023-10-16T13:46:39.653690+0000","last_fullsized":"2023-10-16T13:46:39.653690+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:47:08.704282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.647136+0000","last_change":"2023-10-16T13:46:24.340328+0000","last_active":"2023-10-16T13:46:38.647136+0000","last_peered":"2023-10-16T13:46:38.647136+0000","last_clean":"2023-10-16T13:46:38.647136+0000","last_became_active":"2023-10-16T13:46:24.340042+0000","last_became_peered":"2023-10-16T13:46:24.340042+0000","last_unstale":"2023-10-16T13:46:38.647136+0000","last_undegraded":"2023-10-16T13:46:38.647136+0000","last_fullsized":"2023-10-16T13:46:38.647136+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:15:37.760742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.659388+0000","last_change":"2023-10-16T13:46:25.575797+0000","last_active":"2023-10-16T13:46:38.659388+0000","last_peered":"2023-10-16T13:46:38.659388+0000","last_clean":"2023-10-16T13:46:38.659388+0000","last_became_active":"2023-10-16T13:46:25.575494+0000","last_became_peered":"2023-10-16T13:46:25.575494+0000","last_unstale":"2023-10-16T13:46:38.659388+0000","last_undegraded":"2023-10-16T13:46:38.659388+0000","last_fullsized":"2023-10-16T13:46:38.659388+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:11:22.607258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.4","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.098577+0000","last_change":"2023-10-16T13:46:37.647232+0000","last_active":"2023-10-16T13:46:40.098577+0000","last_peered":"2023-10-16T13:46:40.098577+0000","last_clean":"2023-10-16T13:46:40.098577+0000","last_became_active":"2023-10-16T13:46:37.646761+0000","last_became_peered":"2023-10-16T13:46:37.646761+0000","last_unstale":"2023-10-16T13:46:40.098577+0000","last_undegraded":"2023-10-16T13:46:40.098577+0000","last_fullsized":"2023-10-16T13:46:40.098577+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_clean_scrub_stamp":"2023-10-16T13:46:36.617142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:46:37.318465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.7","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.867313+0000","last_change":"2023-10-16T13:46:38.662825+0000","last_active":"2023-10-16T13:46:40.867313+0000","last_peered":"2023-10-16T13:46:40.867313+0000","last_clean":"2023-10-16T13:46:40.867313+0000","last_became_active":"2023-10-16T13:46:38.662454+0000","last_became_peered":"2023-10-16T13:46:38.662454+0000","last_unstale":"2023-10-16T13:46:40.867313+0000","last_undegraded":"2023-10-16T13:46:40.867313+0000","last_fullsized":"2023-10-16T13:46:40.867313+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:09:09.585691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.646549+0000","last_change":"2023-10-16T13:46:24.338016+0000","last_active":"2023-10-16T13:46:38.646549+0000","last_peered":"2023-10-16T13:46:38.646549+0000","last_clean":"2023-10-16T13:46:38.646549+0000","last_became_active":"2023-10-16T13:46:24.337687+0000","last_became_peered":"2023-10-16T13:46:24.337687+0000","last_unstale":"2023-10-16T13:46:38.646549+0000","last_undegraded":"2023-10-16T13:46:38.646549+0000","last_fullsized":"2023-10-16T13:46:38.646549+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:27:57.477477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.649679+0000","last_change":"2023-10-16T13:46:25.576269+0000","last_active":"2023-10-16T13:46:38.649679+0000","last_peered":"2023-10-16T13:46:38.649679+0000","last_clean":"2023-10-16T13:46:38.649679+0000","last_became_active":"2023-10-16T13:46:25.575732+0000","last_became_peered":"2023-10-16T13:46:25.575732+0000","last_unstale":"2023-10-16T13:46:38.649679+0000","last_undegraded":"2023-10-16T13:46:38.649679+0000","last_fullsized":"2023-10-16T13:46:38.649679+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:36:08.745864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.5","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.671360+0000","last_change":"2023-10-16T13:46:37.638258+0000","last_active":"2023-10-16T13:46:39.671360+0000","last_peered":"2023-10-16T13:46:39.671360+0000","last_clean":"2023-10-16T13:46:39.671360+0000","last_became_active":"2023-10-16T13:46:37.637824+0000","last_became_peered":"2023-10-16T13:46:37.637824+0000","last_unstale":"2023-10-16T13:46:39.671360+0000","last_undegraded":"2023-10-16T13:46:39.671360+0000","last_fullsized":"2023-10-16T13:46:39.671360+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_clean_scrub_stamp":"2023-10-16T13:46:36.617142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:45:43.077455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.6","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.930230+0000","last_change":"2023-10-16T13:46:38.664836+0000","last_active":"2023-10-16T13:46:39.930230+0000","last_peered":"2023-10-16T13:46:39.930230+0000","last_clean":"2023-10-16T13:46:39.930230+0000","last_became_active":"2023-10-16T13:46:38.664267+0000","last_became_peered":"2023-10-16T13:46:38.664267+0000","last_unstale":"2023-10-16T13:46:39.930230+0000","last_undegraded":"2023-10-16T13:46:39.930230+0000","last_fullsized":"2023-10-16T13:46:39.930230+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:38:51.424578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.649873+0000","last_change":"2023-10-16T13:46:24.333052+0000","last_active":"2023-10-16T13:46:38.649873+0000","last_peered":"2023-10-16T13:46:38.649873+0000","last_clean":"2023-10-16T13:46:38.649873+0000","last_became_active":"2023-10-16T13:46:24.332751+0000","last_became_peered":"2023-10-16T13:46:24.332751+0000","last_unstale":"2023-10-16T13:46:38.649873+0000","last_undegraded":"2023-10-16T13:46:38.649873+0000","last_fullsized":"2023-10-16T13:46:38.649873+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:29:38.839702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.651547+0000","last_change":"2023-10-16T13:46:25.580242+0000","last_active":"2023-10-16T13:46:38.651547+0000","last_peered":"2023-10-16T13:46:38.651547+0000","last_clean":"2023-10-16T13:46:38.651547+0000","last_became_active":"2023-10-16T13:46:25.579959+0000","last_became_peered":"2023-10-16T13:46:25.579959+0000","last_unstale":"2023-10-16T13:46:38.651547+0000","last_undegraded":"2023-10-16T13:46:38.651547+0000","last_fullsized":"2023-10-16T13:46:38.651547+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:52:21.431610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.2","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.673033+0000","last_change":"2023-10-16T13:46:37.634801+0000","last_active":"2023-10-16T13:46:39.673033+0000","last_peered":"2023-10-16T13:46:39.673033+0000","last_clean":"2023-10-16T13:46:39.673033+0000","last_became_active":"2023-10-16T13:46:37.634481+0000","last_became_peered":"2023-10-16T13:46:37.634481+0000","last_unstale":"2023-10-16T13:46:39.673033+0000","last_undegraded":"2023-10-16T13:46:39.673033+0000","last_fullsized":"2023-10-16T13:46:39.673033+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_clean_scrub_stamp":"2023-10-16T13:46:36.617142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:10:24.150487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.580964+0000","last_change":"2023-10-16T13:46:38.661425+0000","last_active":"2023-10-16T13:46:40.580964+0000","last_peered":"2023-10-16T13:46:40.580964+0000","last_clean":"2023-10-16T13:46:40.580964+0000","last_became_active":"2023-10-16T13:46:38.660952+0000","last_became_peered":"2023-10-16T13:46:38.660952+0000","last_unstale":"2023-10-16T13:46:40.580964+0000","last_undegraded":"2023-10-16T13:46:40.580964+0000","last_fullsized":"2023-10-16T13:46:40.580964+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:08:49.765693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.643091+0000","last_change":"2023-10-16T13:46:24.340274+0000","last_active":"2023-10-16T13:46:38.643091+0000","last_peered":"2023-10-16T13:46:38.643091+0000","last_clean":"2023-10-16T13:46:38.643091+0000","last_became_active":"2023-10-16T13:46:24.339988+0000","last_became_peered":"2023-10-16T13:46:24.339988+0000","last_unstale":"2023-10-16T13:46:38.643091+0000","last_undegraded":"2023-10-16T13:46:38.643091+0000","last_fullsized":"2023-10-16T13:46:38.643091+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:48:44.887565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.646919+0000","last_change":"2023-10-16T13:46:25.567174+0000","last_active":"2023-10-16T13:46:38.646919+0000","last_peered":"2023-10-16T13:46:38.646919+0000","last_clean":"2023-10-16T13:46:38.646919+0000","last_became_active":"2023-10-16T13:46:25.566392+0000","last_became_peered":"2023-10-16T13:46:25.566392+0000","last_unstale":"2023-10-16T13:46:38.646919+0000","last_undegraded":"2023-10-16T13:46:38.646919+0000","last_fullsized":"2023-10-16T13:46:38.646919+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:16:23.997738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.3","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.098057+0000","last_change":"2023-10-16T13:46:37.647560+0000","last_active":"2023-10-16T13:46:40.098057+0000","last_peered":"2023-10-16T13:46:40.098057+0000","last_clean":"2023-10-16T13:46:40.098057+0000","last_became_active":"2023-10-16T13:46:37.646929+0000","last_became_peered":"2023-10-16T13:46:37.646929+0000","last_unstale":"2023-10-16T13:46:40.098057+0000","last_undegraded":"2023-10-16T13:46:40.098057+0000","last_fullsized":"2023-10-16T13:46:40.098057+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_clean_scrub_stamp":"2023-10-16T13:46:36.617142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:41:05.573188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.580688+0000","last_change":"2023-10-16T13:46:38.673556+0000","last_active":"2023-10-16T13:46:40.580688+0000","last_peered":"2023-10-16T13:46:40.580688+0000","last_clean":"2023-10-16T13:46:40.580688+0000","last_became_active":"2023-10-16T13:46:38.673246+0000","last_became_peered":"2023-10-16T13:46:38.673246+0000","last_unstale":"2023-10-16T13:46:40.580688+0000","last_undegraded":"2023-10-16T13:46:40.580688+0000","last_fullsized":"2023-10-16T13:46:40.580688+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:08:18.683720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.6","version":"225'1","reported_seq":17,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.647962+0000","last_change":"2023-10-16T13:46:24.346027+0000","last_active":"2023-10-16T13:46:38.647962+0000","last_peered":"2023-10-16T13:46:38.647962+0000","last_clean":"2023-10-16T13:46:38.647962+0000","last_became_active":"2023-10-16T13:46:24.345748+0000","last_became_peered":"2023-10-16T13:46:24.345748+0000","last_unstale":"2023-10-16T13:46:38.647962+0000","last_undegraded":"2023-10-16T13:46:38.647962+0000","last_fullsized":"2023-10-16T13:46:38.647962+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:23:29.526203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.651456+0000","last_change":"2023-10-16T13:46:25.577247+0000","last_active":"2023-10-16T13:46:38.651456+0000","last_peered":"2023-10-16T13:46:38.651456+0000","last_clean":"2023-10-16T13:46:38.651456+0000","last_became_active":"2023-10-16T13:46:25.576949+0000","last_became_peered":"2023-10-16T13:46:25.576949+0000","last_unstale":"2023-10-16T13:46:38.651456+0000","last_undegraded":"2023-10-16T13:46:38.651456+0000","last_fullsized":"2023-10-16T13:46:38.651456+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:35:32.750171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.0","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.097475+0000","last_change":"2023-10-16T13:46:37.647928+0000","last_active":"2023-10-16T13:46:40.097475+0000","last_peered":"2023-10-16T13:46:40.097475+0000","last_clean":"2023-10-16T13:46:40.097475+0000","last_became_active":"2023-10-16T13:46:37.647416+0000","last_became_peered":"2023-10-16T13:46:37.647416+0000","last_unstale":"2023-10-16T13:46:40.097475+0000","last_undegraded":"2023-10-16T13:46:40.097475+0000","last_fullsized":"2023-10-16T13:46:40.097475+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_clean_scrub_stamp":"2023-10-16T13:46:36.617142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:24:07.201333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.097848+0000","last_change":"2023-10-16T13:46:38.669803+0000","last_active":"2023-10-16T13:46:40.097848+0000","last_peered":"2023-10-16T13:46:40.097848+0000","last_clean":"2023-10-16T13:46:40.097848+0000","last_became_active":"2023-10-16T13:46:38.661903+0000","last_became_peered":"2023-10-16T13:46:38.661903+0000","last_unstale":"2023-10-16T13:46:40.097848+0000","last_undegraded":"2023-10-16T13:46:40.097848+0000","last_fullsized":"2023-10-16T13:46:40.097848+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:24:46.861701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.7","version":"224'2","reported_seq":18,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.656794+0000","last_change":"2023-10-16T13:46:24.344410+0000","last_active":"2023-10-16T13:46:38.656794+0000","last_peered":"2023-10-16T13:46:38.656794+0000","last_clean":"2023-10-16T13:46:38.656794+0000","last_became_active":"2023-10-16T13:46:24.343665+0000","last_became_peered":"2023-10-16T13:46:24.343665+0000","last_unstale":"2023-10-16T13:46:38.656794+0000","last_undegraded":"2023-10-16T13:46:38.656794+0000","last_fullsized":"2023-10-16T13:46:38.656794+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:21:31.259195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.647642+0000","last_change":"2023-10-16T13:46:25.585631+0000","last_active":"2023-10-16T13:46:38.647642+0000","last_peered":"2023-10-16T13:46:38.647642+0000","last_clean":"2023-10-16T13:46:38.647642+0000","last_became_active":"2023-10-16T13:46:25.585329+0000","last_became_peered":"2023-10-16T13:46:25.585329+0000","last_unstale":"2023-10-16T13:46:38.647642+0000","last_undegraded":"2023-10-16T13:46:38.647642+0000","last_fullsized":"2023-10-16T13:46:38.647642+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:37:50.932814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.654405+0000","last_change":"2023-10-16T13:46:37.646177+0000","last_active":"2023-10-16T13:46:39.654405+0000","last_peered":"2023-10-16T13:46:39.654405+0000","last_clean":"2023-10-16T13:46:39.654405+0000","last_became_active":"2023-10-16T13:46:37.645682+0000","last_became_peered":"2023-10-16T13:46:37.645682+0000","last_unstale":"2023-10-16T13:46:39.654405+0000","last_undegraded":"2023-10-16T13:46:39.654405+0000","last_fullsized":"2023-10-16T13:46:39.654405+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_clean_scrub_stamp":"2023-10-16T13:46:36.617142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:14:57.521447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.866695+0000","last_change":"2023-10-16T13:46:38.644591+0000","last_active":"2023-10-16T13:46:40.866695+0000","last_peered":"2023-10-16T13:46:40.866695+0000","last_clean":"2023-10-16T13:46:40.866695+0000","last_became_active":"2023-10-16T13:46:38.644285+0000","last_became_peered":"2023-10-16T13:46:38.644285+0000","last_unstale":"2023-10-16T13:46:40.866695+0000","last_undegraded":"2023-10-16T13:46:40.866695+0000","last_fullsized":"2023-10-16T13:46:40.866695+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:41:21.402877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.649147+0000","last_change":"2023-10-16T13:46:24.341910+0000","last_active":"2023-10-16T13:46:38.649147+0000","last_peered":"2023-10-16T13:46:38.649147+0000","last_clean":"2023-10-16T13:46:38.649147+0000","last_became_active":"2023-10-16T13:46:24.341633+0000","last_became_peered":"2023-10-16T13:46:24.341633+0000","last_unstale":"2023-10-16T13:46:38.649147+0000","last_undegraded":"2023-10-16T13:46:38.649147+0000","last_fullsized":"2023-10-16T13:46:38.649147+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:00:55.319505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.648996+0000","last_change":"2023-10-16T13:46:25.576886+0000","last_active":"2023-10-16T13:46:38.648996+0000","last_peered":"2023-10-16T13:46:38.648996+0000","last_clean":"2023-10-16T13:46:38.648996+0000","last_became_active":"2023-10-16T13:46:25.576540+0000","last_became_peered":"2023-10-16T13:46:25.576540+0000","last_unstale":"2023-10-16T13:46:38.648996+0000","last_undegraded":"2023-10-16T13:46:38.648996+0000","last_fullsized":"2023-10-16T13:46:38.648996+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:00:43.545612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.e","version":"228'2","reported_seq":14,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.867130+0000","last_change":"2023-10-16T13:46:37.642438+0000","last_active":"2023-10-16T13:46:40.867130+0000","last_peered":"2023-10-16T13:46:40.867130+0000","last_clean":"2023-10-16T13:46:40.867130+0000","last_became_active":"2023-10-16T13:46:37.642152+0000","last_became_peered":"2023-10-16T13:46:37.642152+0000","last_unstale":"2023-10-16T13:46:40.867130+0000","last_undegraded":"2023-10-16T13:46:40.867130+0000","last_fullsized":"2023-10-16T13:46:40.867130+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_clean_scrub_stamp":"2023-10-16T13:46:36.617142+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:48:58.187956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.d","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.654809+0000","last_change":"2023-10-16T13:46:38.660005+0000","last_active":"2023-10-16T13:46:39.654809+0000","last_peered":"2023-10-16T13:46:39.654809+0000","last_clean":"2023-10-16T13:46:39.654809+0000","last_became_active":"2023-10-16T13:46:38.659645+0000","last_became_peered":"2023-10-16T13:46:38.659645+0000","last_unstale":"2023-10-16T13:46:39.654809+0000","last_undegraded":"2023-10-16T13:46:39.654809+0000","last_fullsized":"2023-10-16T13:46:39.654809+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:49:06.438194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.641824+0000","last_change":"2023-10-16T13:46:24.344523+0000","last_active":"2023-10-16T13:46:38.641824+0000","last_peered":"2023-10-16T13:46:38.641824+0000","last_clean":"2023-10-16T13:46:38.641824+0000","last_became_active":"2023-10-16T13:46:24.344238+0000","last_became_peered":"2023-10-16T13:46:24.344238+0000","last_unstale":"2023-10-16T13:46:38.641824+0000","last_undegraded":"2023-10-16T13:46:38.641824+0000","last_fullsized":"2023-10-16T13:46:38.641824+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:37:49.931017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.648399+0000","last_change":"2023-10-16T13:46:25.570388+0000","last_active":"2023-10-16T13:46:38.648399+0000","last_peered":"2023-10-16T13:46:38.648399+0000","last_clean":"2023-10-16T13:46:38.648399+0000","last_became_active":"2023-10-16T13:46:25.570059+0000","last_became_peered":"2023-10-16T13:46:25.570059+0000","last_unstale":"2023-10-16T13:46:38.648399+0000","last_undegraded":"2023-10-16T13:46:38.648399+0000","last_fullsized":"2023-10-16T13:46:38.648399+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:11:22.607258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.f","version":"228'1","reported_seq":13,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.581684+0000","last_change":"2023-10-16T13:46:37.648761+0000","last_active":"2023-10-16T13:46:40.581684+0000","last_peered":"2023-10-16T13:46:40.581684+0000","last_clean":"2023-10-16T13:46:40.581684+0000","last_became_active":"2023-10-16T13:46:37.648542+0000","last_became_peered":"2023-10-16T13:46:37.648542+0000","last_unstale":"2023-10-16T13:46:40.581684+0000","last_undegraded":"2023-10-16T13:46:40.581684+0000","last_fullsized":"2023-10-16T13:46:40.581684+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_clean_scrub_stamp":"2023-10-16T13:46:36.617142+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:10:10.956373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.653711+0000","last_change":"2023-10-16T13:46:38.674462+0000","last_active":"2023-10-16T13:46:39.653711+0000","last_peered":"2023-10-16T13:46:39.653711+0000","last_clean":"2023-10-16T13:46:39.653711+0000","last_became_active":"2023-10-16T13:46:38.674196+0000","last_became_peered":"2023-10-16T13:46:38.674196+0000","last_unstale":"2023-10-16T13:46:39.653711+0000","last_undegraded":"2023-10-16T13:46:39.653711+0000","last_fullsized":"2023-10-16T13:46:39.653711+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:14:58.526615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.657803+0000","last_change":"2023-10-16T13:46:24.337053+0000","last_active":"2023-10-16T13:46:38.657803+0000","last_peered":"2023-10-16T13:46:38.657803+0000","last_clean":"2023-10-16T13:46:38.657803+0000","last_became_active":"2023-10-16T13:46:24.336736+0000","last_became_peered":"2023-10-16T13:46:24.336736+0000","last_unstale":"2023-10-16T13:46:38.657803+0000","last_undegraded":"2023-10-16T13:46:38.657803+0000","last_fullsized":"2023-10-16T13:46:38.657803+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:36:24.043336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.647324+0000","last_change":"2023-10-16T13:46:25.580894+0000","last_active":"2023-10-16T13:46:38.647324+0000","last_peered":"2023-10-16T13:46:38.647324+0000","last_clean":"2023-10-16T13:46:38.647324+0000","last_became_active":"2023-10-16T13:46:25.580589+0000","last_became_peered":"2023-10-16T13:46:25.580589+0000","last_unstale":"2023-10-16T13:46:38.647324+0000","last_undegraded":"2023-10-16T13:46:38.647324+0000","last_fullsized":"2023-10-16T13:46:38.647324+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:22:31.745564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.c","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.098212+0000","last_change":"2023-10-16T13:46:37.653174+0000","last_active":"2023-10-16T13:46:40.098212+0000","last_peered":"2023-10-16T13:46:40.098212+0000","last_clean":"2023-10-16T13:46:40.098212+0000","last_became_active":"2023-10-16T13:46:37.652887+0000","last_became_peered":"2023-10-16T13:46:37.652887+0000","last_unstale":"2023-10-16T13:46:40.098212+0000","last_undegraded":"2023-10-16T13:46:40.098212+0000","last_fullsized":"2023-10-16T13:46:40.098212+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_clean_scrub_stamp":"2023-10-16T13:46:36.617142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:17:36.285215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.f","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.098986+0000","last_change":"2023-10-16T13:46:38.662558+0000","last_active":"2023-10-16T13:46:40.098986+0000","last_peered":"2023-10-16T13:46:40.098986+0000","last_clean":"2023-10-16T13:46:40.098986+0000","last_became_active":"2023-10-16T13:46:38.661847+0000","last_became_peered":"2023-10-16T13:46:38.661847+0000","last_unstale":"2023-10-16T13:46:40.098986+0000","last_undegraded":"2023-10-16T13:46:40.098986+0000","last_fullsized":"2023-10-16T13:46:40.098986+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:25:18.872239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.b","version":"224'2","reported_seq":18,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.640887+0000","last_change":"2023-10-16T13:46:24.347651+0000","last_active":"2023-10-16T13:46:38.640887+0000","last_peered":"2023-10-16T13:46:38.640887+0000","last_clean":"2023-10-16T13:46:38.640887+0000","last_became_active":"2023-10-16T13:46:24.347370+0000","last_became_peered":"2023-10-16T13:46:24.347370+0000","last_unstale":"2023-10-16T13:46:38.640887+0000","last_undegraded":"2023-10-16T13:46:38.640887+0000","last_fullsized":"2023-10-16T13:46:38.640887+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:18:32.988431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.654660+0000","last_change":"2023-10-16T13:46:25.572423+0000","last_active":"2023-10-16T13:46:38.654660+0000","last_peered":"2023-10-16T13:46:38.654660+0000","last_clean":"2023-10-16T13:46:38.654660+0000","last_became_active":"2023-10-16T13:46:25.569272+0000","last_became_peered":"2023-10-16T13:46:25.569272+0000","last_unstale":"2023-10-16T13:46:38.654660+0000","last_undegraded":"2023-10-16T13:46:38.654660+0000","last_fullsized":"2023-10-16T13:46:38.654660+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:14:45.222853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.d","version":"228'1","reported_seq":13,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.098201+0000","last_change":"2023-10-16T13:46:37.648875+0000","last_active":"2023-10-16T13:46:40.098201+0000","last_peered":"2023-10-16T13:46:40.098201+0000","last_clean":"2023-10-16T13:46:40.098201+0000","last_became_active":"2023-10-16T13:46:37.648583+0000","last_became_peered":"2023-10-16T13:46:37.648583+0000","last_unstale":"2023-10-16T13:46:40.098201+0000","last_undegraded":"2023-10-16T13:46:40.098201+0000","last_fullsized":"2023-10-16T13:46:40.098201+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_clean_scrub_stamp":"2023-10-16T13:46:36.617142+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:38:26.428397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.e","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.581775+0000","last_change":"2023-10-16T13:46:38.661756+0000","last_active":"2023-10-16T13:46:40.581775+0000","last_peered":"2023-10-16T13:46:40.581775+0000","last_clean":"2023-10-16T13:46:40.581775+0000","last_became_active":"2023-10-16T13:46:38.661029+0000","last_became_peered":"2023-10-16T13:46:38.661029+0000","last_unstale":"2023-10-16T13:46:40.581775+0000","last_undegraded":"2023-10-16T13:46:40.581775+0000","last_fullsized":"2023-10-16T13:46:40.581775+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:18:52.510533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.641078+0000","last_change":"2023-10-16T13:46:24.339475+0000","last_active":"2023-10-16T13:46:38.641078+0000","last_peered":"2023-10-16T13:46:38.641078+0000","last_clean":"2023-10-16T13:46:38.641078+0000","last_became_active":"2023-10-16T13:46:24.339105+0000","last_became_peered":"2023-10-16T13:46:24.339105+0000","last_unstale":"2023-10-16T13:46:38.641078+0000","last_undegraded":"2023-10-16T13:46:38.641078+0000","last_fullsized":"2023-10-16T13:46:38.641078+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:13:27.085544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.646755+0000","last_change":"2023-10-16T13:46:25.579683+0000","last_active":"2023-10-16T13:46:38.646755+0000","last_peered":"2023-10-16T13:46:38.646755+0000","last_clean":"2023-10-16T13:46:38.646755+0000","last_became_active":"2023-10-16T13:46:25.579398+0000","last_became_peered":"2023-10-16T13:46:25.579398+0000","last_unstale":"2023-10-16T13:46:38.646755+0000","last_undegraded":"2023-10-16T13:46:38.646755+0000","last_fullsized":"2023-10-16T13:46:38.646755+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:30:09.665561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.a","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.670691+0000","last_change":"2023-10-16T13:46:37.653618+0000","last_active":"2023-10-16T13:46:39.670691+0000","last_peered":"2023-10-16T13:46:39.670691+0000","last_clean":"2023-10-16T13:46:39.670691+0000","last_became_active":"2023-10-16T13:46:37.653296+0000","last_became_peered":"2023-10-16T13:46:37.653296+0000","last_unstale":"2023-10-16T13:46:39.670691+0000","last_undegraded":"2023-10-16T13:46:39.670691+0000","last_fullsized":"2023-10-16T13:46:39.670691+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_clean_scrub_stamp":"2023-10-16T13:46:36.617142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:45:22.549136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.654700+0000","last_change":"2023-10-16T13:46:38.665894+0000","last_active":"2023-10-16T13:46:39.654700+0000","last_peered":"2023-10-16T13:46:39.654700+0000","last_clean":"2023-10-16T13:46:39.654700+0000","last_became_active":"2023-10-16T13:46:38.665612+0000","last_became_peered":"2023-10-16T13:46:38.665612+0000","last_unstale":"2023-10-16T13:46:39.654700+0000","last_undegraded":"2023-10-16T13:46:39.654700+0000","last_fullsized":"2023-10-16T13:46:39.654700+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:44:17.896114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.d","version":"224'1","reported_seq":17,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.658880+0000","last_change":"2023-10-16T13:46:24.338316+0000","last_active":"2023-10-16T13:46:38.658880+0000","last_peered":"2023-10-16T13:46:38.658880+0000","last_clean":"2023-10-16T13:46:38.658880+0000","last_became_active":"2023-10-16T13:46:24.338002+0000","last_became_peered":"2023-10-16T13:46:24.338002+0000","last_unstale":"2023-10-16T13:46:38.658880+0000","last_undegraded":"2023-10-16T13:46:38.658880+0000","last_fullsized":"2023-10-16T13:46:38.658880+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:04:35.398533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.653092+0000","last_change":"2023-10-16T13:46:25.582651+0000","last_active":"2023-10-16T13:46:38.653092+0000","last_peered":"2023-10-16T13:46:38.653092+0000","last_clean":"2023-10-16T13:46:38.653092+0000","last_became_active":"2023-10-16T13:46:25.582350+0000","last_became_peered":"2023-10-16T13:46:25.582350+0000","last_unstale":"2023-10-16T13:46:38.653092+0000","last_undegraded":"2023-10-16T13:46:38.653092+0000","last_fullsized":"2023-10-16T13:46:38.653092+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:25:08.151545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.b","version":"228'2","reported_seq":14,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.098382+0000","last_change":"2023-10-16T13:46:37.647961+0000","last_active":"2023-10-16T13:46:40.098382+0000","last_peered":"2023-10-16T13:46:40.098382+0000","last_clean":"2023-10-16T13:46:40.098382+0000","last_became_active":"2023-10-16T13:46:37.647472+0000","last_became_peered":"2023-10-16T13:46:37.647472+0000","last_unstale":"2023-10-16T13:46:40.098382+0000","last_undegraded":"2023-10-16T13:46:40.098382+0000","last_fullsized":"2023-10-16T13:46:40.098382+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_clean_scrub_stamp":"2023-10-16T13:46:36.617142+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:43:36.812667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.097345+0000","last_change":"2023-10-16T13:46:38.662342+0000","last_active":"2023-10-16T13:46:40.097345+0000","last_peered":"2023-10-16T13:46:40.097345+0000","last_clean":"2023-10-16T13:46:40.097345+0000","last_became_active":"2023-10-16T13:46:38.662039+0000","last_became_peered":"2023-10-16T13:46:38.662039+0000","last_unstale":"2023-10-16T13:46:40.097345+0000","last_undegraded":"2023-10-16T13:46:40.097345+0000","last_fullsized":"2023-10-16T13:46:40.097345+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:31:01.184857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.e","version":"224'2","reported_seq":18,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.650881+0000","last_change":"2023-10-16T13:46:24.339351+0000","last_active":"2023-10-16T13:46:38.650881+0000","last_peered":"2023-10-16T13:46:38.650881+0000","last_clean":"2023-10-16T13:46:38.650881+0000","last_became_active":"2023-10-16T13:46:24.339062+0000","last_became_peered":"2023-10-16T13:46:24.339062+0000","last_unstale":"2023-10-16T13:46:38.650881+0000","last_undegraded":"2023-10-16T13:46:38.650881+0000","last_fullsized":"2023-10-16T13:46:38.650881+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:14:49.845621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.649074+0000","last_change":"2023-10-16T13:46:25.578233+0000","last_active":"2023-10-16T13:46:38.649074+0000","last_peered":"2023-10-16T13:46:38.649074+0000","last_clean":"2023-10-16T13:46:38.649074+0000","last_became_active":"2023-10-16T13:46:25.577908+0000","last_became_peered":"2023-10-16T13:46:25.577908+0000","last_unstale":"2023-10-16T13:46:38.649074+0000","last_undegraded":"2023-10-16T13:46:38.649074+0000","last_fullsized":"2023-10-16T13:46:38.649074+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:15:34.577378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.8","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.866463+0000","last_change":"2023-10-16T13:46:37.644185+0000","last_active":"2023-10-16T13:46:40.866463+0000","last_peered":"2023-10-16T13:46:40.866463+0000","last_clean":"2023-10-16T13:46:40.866463+0000","last_became_active":"2023-10-16T13:46:37.643958+0000","last_became_peered":"2023-10-16T13:46:37.643958+0000","last_unstale":"2023-10-16T13:46:40.866463+0000","last_undegraded":"2023-10-16T13:46:40.866463+0000","last_fullsized":"2023-10-16T13:46:40.866463+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_clean_scrub_stamp":"2023-10-16T13:46:36.617142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:11:00.815968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.b","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.581061+0000","last_change":"2023-10-16T13:46:38.673958+0000","last_active":"2023-10-16T13:46:40.581061+0000","last_peered":"2023-10-16T13:46:40.581061+0000","last_clean":"2023-10-16T13:46:40.581061+0000","last_became_active":"2023-10-16T13:46:38.673696+0000","last_became_peered":"2023-10-16T13:46:38.673696+0000","last_unstale":"2023-10-16T13:46:40.581061+0000","last_undegraded":"2023-10-16T13:46:40.581061+0000","last_fullsized":"2023-10-16T13:46:40.581061+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:47:10.439209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.f","version":"224'1","reported_seq":17,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.647273+0000","last_change":"2023-10-16T13:46:24.338988+0000","last_active":"2023-10-16T13:46:38.647273+0000","last_peered":"2023-10-16T13:46:38.647273+0000","last_clean":"2023-10-16T13:46:38.647273+0000","last_became_active":"2023-10-16T13:46:24.338477+0000","last_became_peered":"2023-10-16T13:46:24.338477+0000","last_unstale":"2023-10-16T13:46:38.647273+0000","last_undegraded":"2023-10-16T13:46:38.647273+0000","last_fullsized":"2023-10-16T13:46:38.647273+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:47:21.411665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.647426+0000","last_change":"2023-10-16T13:46:25.585651+0000","last_active":"2023-10-16T13:46:38.647426+0000","last_peered":"2023-10-16T13:46:38.647426+0000","last_clean":"2023-10-16T13:46:38.647426+0000","last_became_active":"2023-10-16T13:46:25.585367+0000","last_became_peered":"2023-10-16T13:46:25.585367+0000","last_unstale":"2023-10-16T13:46:38.647426+0000","last_undegraded":"2023-10-16T13:46:38.647426+0000","last_fullsized":"2023-10-16T13:46:38.647426+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:19:04.312392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.9","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.670556+0000","last_change":"2023-10-16T13:46:37.639229+0000","last_active":"2023-10-16T13:46:39.670556+0000","last_peered":"2023-10-16T13:46:39.670556+0000","last_clean":"2023-10-16T13:46:39.670556+0000","last_became_active":"2023-10-16T13:46:37.638965+0000","last_became_peered":"2023-10-16T13:46:37.638965+0000","last_unstale":"2023-10-16T13:46:39.670556+0000","last_undegraded":"2023-10-16T13:46:39.670556+0000","last_fullsized":"2023-10-16T13:46:39.670556+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_clean_scrub_stamp":"2023-10-16T13:46:36.617142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:00:52.195970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.653547+0000","last_change":"2023-10-16T13:46:38.674113+0000","last_active":"2023-10-16T13:46:39.653547+0000","last_peered":"2023-10-16T13:46:39.653547+0000","last_clean":"2023-10-16T13:46:39.653547+0000","last_became_active":"2023-10-16T13:46:38.673756+0000","last_became_peered":"2023-10-16T13:46:38.673756+0000","last_unstale":"2023-10-16T13:46:39.653547+0000","last_undegraded":"2023-10-16T13:46:39.653547+0000","last_fullsized":"2023-10-16T13:46:39.653547+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:25:58.105841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.10","version":"224'8","reported_seq":24,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.640658+0000","last_change":"2023-10-16T13:46:24.346086+0000","last_active":"2023-10-16T13:46:38.640658+0000","last_peered":"2023-10-16T13:46:38.640658+0000","last_clean":"2023-10-16T13:46:38.640658+0000","last_became_active":"2023-10-16T13:46:24.345786+0000","last_became_peered":"2023-10-16T13:46:24.345786+0000","last_unstale":"2023-10-16T13:46:38.640658+0000","last_undegraded":"2023-10-16T13:46:38.640658+0000","last_fullsized":"2023-10-16T13:46:38.640658+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:39:19.890170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.647346+0000","last_change":"2023-10-16T13:46:25.584003+0000","last_active":"2023-10-16T13:46:38.647346+0000","last_peered":"2023-10-16T13:46:38.647346+0000","last_clean":"2023-10-16T13:46:38.647346+0000","last_became_active":"2023-10-16T13:46:25.583732+0000","last_became_peered":"2023-10-16T13:46:25.583732+0000","last_unstale":"2023-10-16T13:46:38.647346+0000","last_undegraded":"2023-10-16T13:46:38.647346+0000","last_fullsized":"2023-10-16T13:46:38.647346+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:08:20.004685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.16","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.673697+0000","last_change":"2023-10-16T13:46:37.649721+0000","last_active":"2023-10-16T13:46:39.673697+0000","last_peered":"2023-10-16T13:46:39.673697+0000","last_clean":"2023-10-16T13:46:39.673697+0000","last_became_active":"2023-10-16T13:46:37.649410+0000","last_became_peered":"2023-10-16T13:46:37.649410+0000","last_unstale":"2023-10-16T13:46:39.673697+0000","last_undegraded":"2023-10-16T13:46:39.673697+0000","last_fullsized":"2023-10-16T13:46:39.673697+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_clean_scrub_stamp":"2023-10-16T13:46:36.617142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:35:04.333865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.15","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.671014+0000","last_change":"2023-10-16T13:46:38.653350+0000","last_active":"2023-10-16T13:46:39.671014+0000","last_peered":"2023-10-16T13:46:39.671014+0000","last_clean":"2023-10-16T13:46:39.671014+0000","last_became_active":"2023-10-16T13:46:38.652538+0000","last_became_peered":"2023-10-16T13:46:38.652538+0000","last_unstale":"2023-10-16T13:46:39.671014+0000","last_undegraded":"2023-10-16T13:46:39.671014+0000","last_fullsized":"2023-10-16T13:46:39.671014+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:56:11.960239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.646167+0000","last_change":"2023-10-16T13:46:24.332386+0000","last_active":"2023-10-16T13:46:38.646167+0000","last_peered":"2023-10-16T13:46:38.646167+0000","last_clean":"2023-10-16T13:46:38.646167+0000","last_became_active":"2023-10-16T13:46:24.332013+0000","last_became_peered":"2023-10-16T13:46:24.332013+0000","last_unstale":"2023-10-16T13:46:38.646167+0000","last_undegraded":"2023-10-16T13:46:38.646167+0000","last_fullsized":"2023-10-16T13:46:38.646167+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:04:46.312175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.648293+0000","last_change":"2023-10-16T13:46:25.573821+0000","last_active":"2023-10-16T13:46:38.648293+0000","last_peered":"2023-10-16T13:46:38.648293+0000","last_clean":"2023-10-16T13:46:38.648293+0000","last_became_active":"2023-10-16T13:46:25.572882+0000","last_became_peered":"2023-10-16T13:46:25.572882+0000","last_unstale":"2023-10-16T13:46:38.648293+0000","last_undegraded":"2023-10-16T13:46:38.648293+0000","last_fullsized":"2023-10-16T13:46:38.648293+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:57:13.048327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.17","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.655504+0000","last_change":"2023-10-16T13:46:37.648194+0000","last_active":"2023-10-16T13:46:39.655504+0000","last_peered":"2023-10-16T13:46:39.655504+0000","last_clean":"2023-10-16T13:46:39.655504+0000","last_became_active":"2023-10-16T13:46:37.647909+0000","last_became_peered":"2023-10-16T13:46:37.647909+0000","last_unstale":"2023-10-16T13:46:39.655504+0000","last_undegraded":"2023-10-16T13:46:39.655504+0000","last_fullsized":"2023-10-16T13:46:39.655504+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_clean_scrub_stamp":"2023-10-16T13:46:36.617142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:47:07.699114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.14","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.670727+0000","last_change":"2023-10-16T13:46:38.656732+0000","last_active":"2023-10-16T13:46:39.670727+0000","last_peered":"2023-10-16T13:46:39.670727+0000","last_clean":"2023-10-16T13:46:39.670727+0000","last_became_active":"2023-10-16T13:46:38.656387+0000","last_became_peered":"2023-10-16T13:46:38.656387+0000","last_unstale":"2023-10-16T13:46:39.670727+0000","last_undegraded":"2023-10-16T13:46:39.670727+0000","last_fullsized":"2023-10-16T13:46:39.670727+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:36:17.135906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.652378+0000","last_change":"2023-10-16T13:46:24.336805+0000","last_active":"2023-10-16T13:46:38.652378+0000","last_peered":"2023-10-16T13:46:38.652378+0000","last_clean":"2023-10-16T13:46:38.652378+0000","last_became_active":"2023-10-16T13:46:24.336473+0000","last_became_peered":"2023-10-16T13:46:24.336473+0000","last_unstale":"2023-10-16T13:46:38.652378+0000","last_undegraded":"2023-10-16T13:46:38.652378+0000","last_fullsized":"2023-10-16T13:46:38.652378+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:44:10.241047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.652982+0000","last_change":"2023-10-16T13:46:25.575462+0000","last_active":"2023-10-16T13:46:38.652982+0000","last_peered":"2023-10-16T13:46:38.652982+0000","last_clean":"2023-10-16T13:46:38.652982+0000","last_became_active":"2023-10-16T13:46:25.574929+0000","last_became_peered":"2023-10-16T13:46:25.574929+0000","last_unstale":"2023-10-16T13:46:38.652982+0000","last_undegraded":"2023-10-16T13:46:38.652982+0000","last_fullsized":"2023-10-16T13:46:38.652982+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:39:18.796503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.14","version":"228'6","reported_seq":18,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.930310+0000","last_change":"2023-10-16T13:46:37.652614+0000","last_active":"2023-10-16T13:46:39.930310+0000","last_peered":"2023-10-16T13:46:39.930310+0000","last_clean":"2023-10-16T13:46:39.930310+0000","last_became_active":"2023-10-16T13:46:37.652318+0000","last_became_peered":"2023-10-16T13:46:37.652318+0000","last_unstale":"2023-10-16T13:46:39.930310+0000","last_undegraded":"2023-10-16T13:46:39.930310+0000","last_fullsized":"2023-10-16T13:46:39.930310+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_clean_scrub_stamp":"2023-10-16T13:46:36.617142+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:22:44.044158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.673586+0000","last_change":"2023-10-16T13:46:38.659187+0000","last_active":"2023-10-16T13:46:39.673586+0000","last_peered":"2023-10-16T13:46:39.673586+0000","last_clean":"2023-10-16T13:46:39.673586+0000","last_became_active":"2023-10-16T13:46:38.658898+0000","last_became_peered":"2023-10-16T13:46:38.658898+0000","last_unstale":"2023-10-16T13:46:39.673586+0000","last_undegraded":"2023-10-16T13:46:39.673586+0000","last_fullsized":"2023-10-16T13:46:39.673586+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:11:38.752824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.13","version":"224'2","reported_seq":18,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.649133+0000","last_change":"2023-10-16T13:46:24.333306+0000","last_active":"2023-10-16T13:46:38.649133+0000","last_peered":"2023-10-16T13:46:38.649133+0000","last_clean":"2023-10-16T13:46:38.649133+0000","last_became_active":"2023-10-16T13:46:24.333042+0000","last_became_peered":"2023-10-16T13:46:24.333042+0000","last_unstale":"2023-10-16T13:46:38.649133+0000","last_undegraded":"2023-10-16T13:46:38.649133+0000","last_fullsized":"2023-10-16T13:46:38.649133+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:44:45.300765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.14","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.654564+0000","last_change":"2023-10-16T13:46:25.566526+0000","last_active":"2023-10-16T13:46:38.654564+0000","last_peered":"2023-10-16T13:46:38.654564+0000","last_clean":"2023-10-16T13:46:38.654564+0000","last_became_active":"2023-10-16T13:46:25.566252+0000","last_became_peered":"2023-10-16T13:46:25.566252+0000","last_unstale":"2023-10-16T13:46:38.654564+0000","last_undegraded":"2023-10-16T13:46:38.654564+0000","last_fullsized":"2023-10-16T13:46:38.654564+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:36:03.832144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.15","version":"228'3","reported_seq":15,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.655290+0000","last_change":"2023-10-16T13:46:37.646597+0000","last_active":"2023-10-16T13:46:39.655290+0000","last_peered":"2023-10-16T13:46:39.655290+0000","last_clean":"2023-10-16T13:46:39.655290+0000","last_became_active":"2023-10-16T13:46:37.646321+0000","last_became_peered":"2023-10-16T13:46:37.646321+0000","last_unstale":"2023-10-16T13:46:39.655290+0000","last_undegraded":"2023-10-16T13:46:39.655290+0000","last_fullsized":"2023-10-16T13:46:39.655290+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_clean_scrub_stamp":"2023-10-16T13:46:36.617142+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:21:03.012606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.581891+0000","last_change":"2023-10-16T13:46:38.661774+0000","last_active":"2023-10-16T13:46:40.581891+0000","last_peered":"2023-10-16T13:46:40.581891+0000","last_clean":"2023-10-16T13:46:40.581891+0000","last_became_active":"2023-10-16T13:46:38.661440+0000","last_became_peered":"2023-10-16T13:46:38.661440+0000","last_unstale":"2023-10-16T13:46:40.581891+0000","last_undegraded":"2023-10-16T13:46:40.581891+0000","last_fullsized":"2023-10-16T13:46:40.581891+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:55:30.798202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.14","version":"225'8","reported_seq":24,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.647310+0000","last_change":"2023-10-16T13:46:24.347248+0000","last_active":"2023-10-16T13:46:38.647310+0000","last_peered":"2023-10-16T13:46:38.647310+0000","last_clean":"2023-10-16T13:46:38.647310+0000","last_became_active":"2023-10-16T13:46:24.346699+0000","last_became_peered":"2023-10-16T13:46:24.346699+0000","last_unstale":"2023-10-16T13:46:38.647310+0000","last_undegraded":"2023-10-16T13:46:38.647310+0000","last_fullsized":"2023-10-16T13:46:38.647310+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:15:37.760742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2859,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.13","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.641114+0000","last_change":"2023-10-16T13:46:25.578929+0000","last_active":"2023-10-16T13:46:38.641114+0000","last_peered":"2023-10-16T13:46:38.641114+0000","last_clean":"2023-10-16T13:46:38.641114+0000","last_became_active":"2023-10-16T13:46:25.578431+0000","last_became_peered":"2023-10-16T13:46:25.578431+0000","last_unstale":"2023-10-16T13:46:38.641114+0000","last_undegraded":"2023-10-16T13:46:38.641114+0000","last_fullsized":"2023-10-16T13:46:38.641114+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:23:30.528000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.12","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.653684+0000","last_change":"2023-10-16T13:46:37.644333+0000","last_active":"2023-10-16T13:46:39.653684+0000","last_peered":"2023-10-16T13:46:39.653684+0000","last_clean":"2023-10-16T13:46:39.653684+0000","last_became_active":"2023-10-16T13:46:37.644076+0000","last_became_peered":"2023-10-16T13:46:37.644076+0000","last_unstale":"2023-10-16T13:46:39.653684+0000","last_undegraded":"2023-10-16T13:46:39.653684+0000","last_fullsized":"2023-10-16T13:46:39.653684+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_clean_scrub_stamp":"2023-10-16T13:46:36.617142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:45:22.549136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.097472+0000","last_change":"2023-10-16T13:46:38.662644+0000","last_active":"2023-10-16T13:46:40.097472+0000","last_peered":"2023-10-16T13:46:40.097472+0000","last_clean":"2023-10-16T13:46:40.097472+0000","last_became_active":"2023-10-16T13:46:38.662350+0000","last_became_peered":"2023-10-16T13:46:38.662350+0000","last_unstale":"2023-10-16T13:46:40.097472+0000","last_undegraded":"2023-10-16T13:46:40.097472+0000","last_fullsized":"2023-10-16T13:46:40.097472+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:04:52.014766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.15","version":"224'3","reported_seq":19,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.647904+0000","last_change":"2023-10-16T13:46:24.340273+0000","last_active":"2023-10-16T13:46:38.647904+0000","last_peered":"2023-10-16T13:46:38.647904+0000","last_clean":"2023-10-16T13:46:38.647904+0000","last_became_active":"2023-10-16T13:46:24.339939+0000","last_became_peered":"2023-10-16T13:46:24.339939+0000","last_unstale":"2023-10-16T13:46:38.647904+0000","last_undegraded":"2023-10-16T13:46:38.647904+0000","last_fullsized":"2023-10-16T13:46:38.647904+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:25:07.149748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.646802+0000","last_change":"2023-10-16T13:46:25.580296+0000","last_active":"2023-10-16T13:46:38.646802+0000","last_peered":"2023-10-16T13:46:38.646802+0000","last_clean":"2023-10-16T13:46:38.646802+0000","last_became_active":"2023-10-16T13:46:25.579963+0000","last_became_peered":"2023-10-16T13:46:25.579963+0000","last_unstale":"2023-10-16T13:46:38.646802+0000","last_undegraded":"2023-10-16T13:46:38.646802+0000","last_fullsized":"2023-10-16T13:46:38.646802+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:39:18.796503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.13","version":"228'2","reported_seq":14,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.097706+0000","last_change":"2023-10-16T13:46:37.647772+0000","last_active":"2023-10-16T13:46:40.097706+0000","last_peered":"2023-10-16T13:46:40.097706+0000","last_clean":"2023-10-16T13:46:40.097706+0000","last_became_active":"2023-10-16T13:46:37.647248+0000","last_became_peered":"2023-10-16T13:46:37.647248+0000","last_unstale":"2023-10-16T13:46:40.097706+0000","last_undegraded":"2023-10-16T13:46:40.097706+0000","last_fullsized":"2023-10-16T13:46:40.097706+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_clean_scrub_stamp":"2023-10-16T13:46:36.617142+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:27:12.664826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.10","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.653438+0000","last_change":"2023-10-16T13:46:38.658182+0000","last_active":"2023-10-16T13:46:39.653438+0000","last_peered":"2023-10-16T13:46:39.653438+0000","last_clean":"2023-10-16T13:46:39.653438+0000","last_became_active":"2023-10-16T13:46:38.657901+0000","last_became_peered":"2023-10-16T13:46:38.657901+0000","last_unstale":"2023-10-16T13:46:39.653438+0000","last_undegraded":"2023-10-16T13:46:39.653438+0000","last_fullsized":"2023-10-16T13:46:39.653438+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:45:44.082623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.651460+0000","last_change":"2023-10-16T13:46:24.344174+0000","last_active":"2023-10-16T13:46:38.651460+0000","last_peered":"2023-10-16T13:46:38.651460+0000","last_clean":"2023-10-16T13:46:38.651460+0000","last_became_active":"2023-10-16T13:46:24.343883+0000","last_became_peered":"2023-10-16T13:46:24.343883+0000","last_unstale":"2023-10-16T13:46:38.651460+0000","last_undegraded":"2023-10-16T13:46:38.651460+0000","last_fullsized":"2023-10-16T13:46:38.651460+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:53:47.000133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.651365+0000","last_change":"2023-10-16T13:46:25.572773+0000","last_active":"2023-10-16T13:46:38.651365+0000","last_peered":"2023-10-16T13:46:38.651365+0000","last_clean":"2023-10-16T13:46:38.651365+0000","last_became_active":"2023-10-16T13:46:25.572324+0000","last_became_peered":"2023-10-16T13:46:25.572324+0000","last_unstale":"2023-10-16T13:46:38.651365+0000","last_undegraded":"2023-10-16T13:46:38.651365+0000","last_fullsized":"2023-10-16T13:46:38.651365+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:41:30.230501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.10","version":"228'8","reported_seq":20,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.097389+0000","last_change":"2023-10-16T13:46:37.647724+0000","last_active":"2023-10-16T13:46:40.097389+0000","last_peered":"2023-10-16T13:46:40.097389+0000","last_clean":"2023-10-16T13:46:40.097389+0000","last_became_active":"2023-10-16T13:46:37.647053+0000","last_became_peered":"2023-10-16T13:46:37.647053+0000","last_unstale":"2023-10-16T13:46:40.097389+0000","last_undegraded":"2023-10-16T13:46:40.097389+0000","last_fullsized":"2023-10-16T13:46:40.097389+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_clean_scrub_stamp":"2023-10-16T13:46:36.617142+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:07:12.646343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.13","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.670843+0000","last_change":"2023-10-16T13:46:38.666851+0000","last_active":"2023-10-16T13:46:39.670843+0000","last_peered":"2023-10-16T13:46:39.670843+0000","last_clean":"2023-10-16T13:46:39.670843+0000","last_became_active":"2023-10-16T13:46:38.666538+0000","last_became_peered":"2023-10-16T13:46:38.666538+0000","last_unstale":"2023-10-16T13:46:39.670843+0000","last_undegraded":"2023-10-16T13:46:39.670843+0000","last_fullsized":"2023-10-16T13:46:39.670843+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:21:04.017774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.642294+0000","last_change":"2023-10-16T13:46:24.346329+0000","last_active":"2023-10-16T13:46:38.642294+0000","last_peered":"2023-10-16T13:46:38.642294+0000","last_clean":"2023-10-16T13:46:38.642294+0000","last_became_active":"2023-10-16T13:46:24.346051+0000","last_became_peered":"2023-10-16T13:46:24.346051+0000","last_unstale":"2023-10-16T13:46:38.642294+0000","last_undegraded":"2023-10-16T13:46:38.642294+0000","last_fullsized":"2023-10-16T13:46:38.642294+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:14:11.499564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.641955+0000","last_change":"2023-10-16T13:46:25.574355+0000","last_active":"2023-10-16T13:46:38.641955+0000","last_peered":"2023-10-16T13:46:38.641955+0000","last_clean":"2023-10-16T13:46:38.641955+0000","last_became_active":"2023-10-16T13:46:25.574057+0000","last_became_peered":"2023-10-16T13:46:25.574057+0000","last_unstale":"2023-10-16T13:46:38.641955+0000","last_undegraded":"2023-10-16T13:46:38.641955+0000","last_fullsized":"2023-10-16T13:46:38.641955+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:41:42.462761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.11","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.929520+0000","last_change":"2023-10-16T13:46:37.648607+0000","last_active":"2023-10-16T13:46:39.929520+0000","last_peered":"2023-10-16T13:46:39.929520+0000","last_clean":"2023-10-16T13:46:39.929520+0000","last_became_active":"2023-10-16T13:46:37.648317+0000","last_became_peered":"2023-10-16T13:46:37.648317+0000","last_unstale":"2023-10-16T13:46:39.929520+0000","last_undegraded":"2023-10-16T13:46:39.929520+0000","last_fullsized":"2023-10-16T13:46:39.929520+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_clean_scrub_stamp":"2023-10-16T13:46:36.617142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:47:13.553188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.580948+0000","last_change":"2023-10-16T13:46:38.661614+0000","last_active":"2023-10-16T13:46:40.580948+0000","last_peered":"2023-10-16T13:46:40.580948+0000","last_clean":"2023-10-16T13:46:40.580948+0000","last_became_active":"2023-10-16T13:46:38.661188+0000","last_became_peered":"2023-10-16T13:46:38.661188+0000","last_unstale":"2023-10-16T13:46:40.580948+0000","last_undegraded":"2023-10-16T13:46:40.580948+0000","last_fullsized":"2023-10-16T13:46:40.580948+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:36:32.753888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.653302+0000","last_change":"2023-10-16T13:46:24.338318+0000","last_active":"2023-10-16T13:46:38.653302+0000","last_peered":"2023-10-16T13:46:38.653302+0000","last_clean":"2023-10-16T13:46:38.653302+0000","last_became_active":"2023-10-16T13:46:24.337766+0000","last_became_peered":"2023-10-16T13:46:24.337766+0000","last_unstale":"2023-10-16T13:46:38.653302+0000","last_undegraded":"2023-10-16T13:46:38.653302+0000","last_fullsized":"2023-10-16T13:46:38.653302+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:25:07.149748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1f","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.656659+0000","last_change":"2023-10-16T13:46:25.571884+0000","last_active":"2023-10-16T13:46:38.656659+0000","last_peered":"2023-10-16T13:46:38.656659+0000","last_clean":"2023-10-16T13:46:38.656659+0000","last_became_active":"2023-10-16T13:46:25.571601+0000","last_became_peered":"2023-10-16T13:46:25.571601+0000","last_unstale":"2023-10-16T13:46:38.656659+0000","last_undegraded":"2023-10-16T13:46:38.656659+0000","last_fullsized":"2023-10-16T13:46:38.656659+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:32:41.007775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.1e","version":"228'1","reported_seq":13,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.098663+0000","last_change":"2023-10-16T13:46:37.644119+0000","last_active":"2023-10-16T13:46:40.098663+0000","last_peered":"2023-10-16T13:46:40.098663+0000","last_clean":"2023-10-16T13:46:40.098663+0000","last_became_active":"2023-10-16T13:46:37.643578+0000","last_became_peered":"2023-10-16T13:46:37.643578+0000","last_unstale":"2023-10-16T13:46:40.098663+0000","last_undegraded":"2023-10-16T13:46:40.098663+0000","last_fullsized":"2023-10-16T13:46:40.098663+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_clean_scrub_stamp":"2023-10-16T13:46:36.617142+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:45:47.428975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1d","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.581448+0000","last_change":"2023-10-16T13:46:38.660503+0000","last_active":"2023-10-16T13:46:40.581448+0000","last_peered":"2023-10-16T13:46:40.581448+0000","last_clean":"2023-10-16T13:46:40.581448+0000","last_became_active":"2023-10-16T13:46:38.660181+0000","last_became_peered":"2023-10-16T13:46:38.660181+0000","last_unstale":"2023-10-16T13:46:40.581448+0000","last_undegraded":"2023-10-16T13:46:40.581448+0000","last_fullsized":"2023-10-16T13:46:40.581448+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:20:33.487282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.651830+0000","last_change":"2023-10-16T13:46:24.343337+0000","last_active":"2023-10-16T13:46:38.651830+0000","last_peered":"2023-10-16T13:46:38.651830+0000","last_clean":"2023-10-16T13:46:38.651830+0000","last_became_active":"2023-10-16T13:46:24.343043+0000","last_became_peered":"2023-10-16T13:46:24.343043+0000","last_unstale":"2023-10-16T13:46:38.651830+0000","last_undegraded":"2023-10-16T13:46:38.651830+0000","last_fullsized":"2023-10-16T13:46:38.651830+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:16:31.012130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.643026+0000","last_change":"2023-10-16T13:46:25.571656+0000","last_active":"2023-10-16T13:46:38.643026+0000","last_peered":"2023-10-16T13:46:38.643026+0000","last_clean":"2023-10-16T13:46:38.643026+0000","last_became_active":"2023-10-16T13:46:25.571372+0000","last_became_peered":"2023-10-16T13:46:25.571372+0000","last_unstale":"2023-10-16T13:46:38.643026+0000","last_undegraded":"2023-10-16T13:46:38.643026+0000","last_fullsized":"2023-10-16T13:46:38.643026+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:57:59.993900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1f","version":"228'4","reported_seq":16,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.867403+0000","last_change":"2023-10-16T13:46:37.644759+0000","last_active":"2023-10-16T13:46:40.867403+0000","last_peered":"2023-10-16T13:46:40.867403+0000","last_clean":"2023-10-16T13:46:40.867403+0000","last_became_active":"2023-10-16T13:46:37.644543+0000","last_became_peered":"2023-10-16T13:46:37.644543+0000","last_unstale":"2023-10-16T13:46:40.867403+0000","last_undegraded":"2023-10-16T13:46:40.867403+0000","last_fullsized":"2023-10-16T13:46:40.867403+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_clean_scrub_stamp":"2023-10-16T13:46:36.617142+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:55:04.080446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.1c","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.581302+0000","last_change":"2023-10-16T13:46:38.660367+0000","last_active":"2023-10-16T13:46:40.581302+0000","last_peered":"2023-10-16T13:46:40.581302+0000","last_clean":"2023-10-16T13:46:40.581302+0000","last_became_active":"2023-10-16T13:46:38.660070+0000","last_became_peered":"2023-10-16T13:46:38.660070+0000","last_unstale":"2023-10-16T13:46:40.581302+0000","last_undegraded":"2023-10-16T13:46:40.581302+0000","last_fullsized":"2023-10-16T13:46:40.581302+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:58:29.279269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.1b","version":"224'1","reported_seq":17,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.659232+0000","last_change":"2023-10-16T13:46:24.343654+0000","last_active":"2023-10-16T13:46:38.659232+0000","last_peered":"2023-10-16T13:46:38.659232+0000","last_clean":"2023-10-16T13:46:38.659232+0000","last_became_active":"2023-10-16T13:46:24.343330+0000","last_became_peered":"2023-10-16T13:46:24.343330+0000","last_unstale":"2023-10-16T13:46:38.659232+0000","last_undegraded":"2023-10-16T13:46:38.659232+0000","last_fullsized":"2023-10-16T13:46:38.659232+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:47:30.680184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.642611+0000","last_change":"2023-10-16T13:46:25.569919+0000","last_active":"2023-10-16T13:46:38.642611+0000","last_peered":"2023-10-16T13:46:38.642611+0000","last_clean":"2023-10-16T13:46:38.642611+0000","last_became_active":"2023-10-16T13:46:25.569619+0000","last_became_peered":"2023-10-16T13:46:25.569619+0000","last_unstale":"2023-10-16T13:46:38.642611+0000","last_undegraded":"2023-10-16T13:46:38.642611+0000","last_fullsized":"2023-10-16T13:46:38.642611+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:43:18.855055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1d","version":"228'2","reported_seq":14,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.929824+0000","last_change":"2023-10-16T13:46:37.651911+0000","last_active":"2023-10-16T13:46:39.929824+0000","last_peered":"2023-10-16T13:46:39.929824+0000","last_clean":"2023-10-16T13:46:39.929824+0000","last_became_active":"2023-10-16T13:46:37.651675+0000","last_became_peered":"2023-10-16T13:46:37.651675+0000","last_unstale":"2023-10-16T13:46:39.929824+0000","last_undegraded":"2023-10-16T13:46:39.929824+0000","last_fullsized":"2023-10-16T13:46:39.929824+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_clean_scrub_stamp":"2023-10-16T13:46:36.617142+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:21:44.559586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1e","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.098551+0000","last_change":"2023-10-16T13:46:38.673439+0000","last_active":"2023-10-16T13:46:40.098551+0000","last_peered":"2023-10-16T13:46:40.098551+0000","last_clean":"2023-10-16T13:46:40.098551+0000","last_became_active":"2023-10-16T13:46:38.669858+0000","last_became_peered":"2023-10-16T13:46:38.669858+0000","last_unstale":"2023-10-16T13:46:40.098551+0000","last_undegraded":"2023-10-16T13:46:40.098551+0000","last_fullsized":"2023-10-16T13:46:40.098551+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:04:24.157820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1c","version":"224'2","reported_seq":18,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.647491+0000","last_change":"2023-10-16T13:46:24.340877+0000","last_active":"2023-10-16T13:46:38.647491+0000","last_peered":"2023-10-16T13:46:38.647491+0000","last_clean":"2023-10-16T13:46:38.647491+0000","last_became_active":"2023-10-16T13:46:24.340607+0000","last_became_peered":"2023-10-16T13:46:24.340607+0000","last_unstale":"2023-10-16T13:46:38.647491+0000","last_undegraded":"2023-10-16T13:46:38.647491+0000","last_fullsized":"2023-10-16T13:46:38.647491+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:16:31.012130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1b","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.641051+0000","last_change":"2023-10-16T13:46:25.571278+0000","last_active":"2023-10-16T13:46:38.641051+0000","last_peered":"2023-10-16T13:46:38.641051+0000","last_clean":"2023-10-16T13:46:38.641051+0000","last_became_active":"2023-10-16T13:46:25.570849+0000","last_became_peered":"2023-10-16T13:46:25.570849+0000","last_unstale":"2023-10-16T13:46:38.641051+0000","last_undegraded":"2023-10-16T13:46:38.641051+0000","last_fullsized":"2023-10-16T13:46:38.641051+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:25:56.358129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.1a","version":"228'1","reported_seq":13,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.654473+0000","last_change":"2023-10-16T13:46:37.650793+0000","last_active":"2023-10-16T13:46:39.654473+0000","last_peered":"2023-10-16T13:46:39.654473+0000","last_clean":"2023-10-16T13:46:39.654473+0000","last_became_active":"2023-10-16T13:46:37.650486+0000","last_became_peered":"2023-10-16T13:46:37.650486+0000","last_unstale":"2023-10-16T13:46:39.654473+0000","last_undegraded":"2023-10-16T13:46:39.654473+0000","last_fullsized":"2023-10-16T13:46:39.654473+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_clean_scrub_stamp":"2023-10-16T13:46:36.617142+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:36:16.130738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.097423+0000","last_change":"2023-10-16T13:46:38.661608+0000","last_active":"2023-10-16T13:46:40.097423+0000","last_peered":"2023-10-16T13:46:40.097423+0000","last_clean":"2023-10-16T13:46:40.097423+0000","last_became_active":"2023-10-16T13:46:38.661312+0000","last_became_peered":"2023-10-16T13:46:38.661312+0000","last_unstale":"2023-10-16T13:46:40.097423+0000","last_undegraded":"2023-10-16T13:46:40.097423+0000","last_fullsized":"2023-10-16T13:46:40.097423+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:01:08.982727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1d","version":"224'2","reported_seq":18,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.658778+0000","last_change":"2023-10-16T13:46:24.340608+0000","last_active":"2023-10-16T13:46:38.658778+0000","last_peered":"2023-10-16T13:46:38.658778+0000","last_clean":"2023-10-16T13:46:38.658778+0000","last_became_active":"2023-10-16T13:46:24.340189+0000","last_became_peered":"2023-10-16T13:46:24.340189+0000","last_unstale":"2023-10-16T13:46:38.658778+0000","last_undegraded":"2023-10-16T13:46:38.658778+0000","last_fullsized":"2023-10-16T13:46:38.658778+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:16:00.359557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.647581+0000","last_change":"2023-10-16T13:46:25.576216+0000","last_active":"2023-10-16T13:46:38.647581+0000","last_peered":"2023-10-16T13:46:38.647581+0000","last_clean":"2023-10-16T13:46:38.647581+0000","last_became_active":"2023-10-16T13:46:25.575876+0000","last_became_peered":"2023-10-16T13:46:25.575876+0000","last_unstale":"2023-10-16T13:46:38.647581+0000","last_undegraded":"2023-10-16T13:46:38.647581+0000","last_fullsized":"2023-10-16T13:46:38.647581+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:41:08.099115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1b","version":"228'1","reported_seq":13,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.670711+0000","last_change":"2023-10-16T13:46:37.650980+0000","last_active":"2023-10-16T13:46:39.670711+0000","last_peered":"2023-10-16T13:46:39.670711+0000","last_clean":"2023-10-16T13:46:39.670711+0000","last_became_active":"2023-10-16T13:46:37.650655+0000","last_became_peered":"2023-10-16T13:46:37.650655+0000","last_unstale":"2023-10-16T13:46:39.670711+0000","last_undegraded":"2023-10-16T13:46:39.670711+0000","last_fullsized":"2023-10-16T13:46:39.670711+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_clean_scrub_stamp":"2023-10-16T13:46:36.617142+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:16:13.659948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.580536+0000","last_change":"2023-10-16T13:46:38.660238+0000","last_active":"2023-10-16T13:46:40.580536+0000","last_peered":"2023-10-16T13:46:40.580536+0000","last_clean":"2023-10-16T13:46:40.580536+0000","last_became_active":"2023-10-16T13:46:38.659923+0000","last_became_peered":"2023-10-16T13:46:38.659923+0000","last_unstale":"2023-10-16T13:46:40.580536+0000","last_undegraded":"2023-10-16T13:46:40.580536+0000","last_fullsized":"2023-10-16T13:46:40.580536+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:57:12.270376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.1e","version":"224'1","reported_seq":17,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.650772+0000","last_change":"2023-10-16T13:46:24.340000+0000","last_active":"2023-10-16T13:46:38.650772+0000","last_peered":"2023-10-16T13:46:38.650772+0000","last_clean":"2023-10-16T13:46:38.650772+0000","last_became_active":"2023-10-16T13:46:24.339714+0000","last_became_peered":"2023-10-16T13:46:24.339714+0000","last_unstale":"2023-10-16T13:46:38.650772+0000","last_undegraded":"2023-10-16T13:46:38.650772+0000","last_fullsized":"2023-10-16T13:46:38.650772+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:21:04.619534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.647146+0000","last_change":"2023-10-16T13:46:25.577384+0000","last_active":"2023-10-16T13:46:38.647146+0000","last_peered":"2023-10-16T13:46:38.647146+0000","last_clean":"2023-10-16T13:46:38.647146+0000","last_became_active":"2023-10-16T13:46:25.577071+0000","last_became_peered":"2023-10-16T13:46:25.577071+0000","last_unstale":"2023-10-16T13:46:38.647146+0000","last_undegraded":"2023-10-16T13:46:38.647146+0000","last_fullsized":"2023-10-16T13:46:38.647146+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:44:11.242845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.18","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.580667+0000","last_change":"2023-10-16T13:46:37.643158+0000","last_active":"2023-10-16T13:46:40.580667+0000","last_peered":"2023-10-16T13:46:40.580667+0000","last_clean":"2023-10-16T13:46:40.580667+0000","last_became_active":"2023-10-16T13:46:37.640753+0000","last_became_peered":"2023-10-16T13:46:37.640753+0000","last_unstale":"2023-10-16T13:46:40.580667+0000","last_undegraded":"2023-10-16T13:46:40.580667+0000","last_fullsized":"2023-10-16T13:46:40.580667+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_clean_scrub_stamp":"2023-10-16T13:46:36.617142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:38:43.180849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.1b","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.866964+0000","last_change":"2023-10-16T13:46:38.655353+0000","last_active":"2023-10-16T13:46:40.866964+0000","last_peered":"2023-10-16T13:46:40.866964+0000","last_clean":"2023-10-16T13:46:40.866964+0000","last_became_active":"2023-10-16T13:46:38.655006+0000","last_became_peered":"2023-10-16T13:46:38.655006+0000","last_unstale":"2023-10-16T13:46:40.866964+0000","last_undegraded":"2023-10-16T13:46:40.866964+0000","last_fullsized":"2023-10-16T13:46:40.866964+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:00:56.849374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1f","version":"224'4","reported_seq":20,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.643243+0000","last_change":"2023-10-16T13:46:24.345142+0000","last_active":"2023-10-16T13:46:38.643243+0000","last_peered":"2023-10-16T13:46:38.643243+0000","last_clean":"2023-10-16T13:46:38.643243+0000","last_became_active":"2023-10-16T13:46:24.344816+0000","last_became_peered":"2023-10-16T13:46:24.344816+0000","last_unstale":"2023-10-16T13:46:38.643243+0000","last_undegraded":"2023-10-16T13:46:38.643243+0000","last_fullsized":"2023-10-16T13:46:38.643243+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:54:50.780055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.648377+0000","last_change":"2023-10-16T13:46:25.578414+0000","last_active":"2023-10-16T13:46:38.648377+0000","last_peered":"2023-10-16T13:46:38.648377+0000","last_clean":"2023-10-16T13:46:38.648377+0000","last_became_active":"2023-10-16T13:46:25.578111+0000","last_became_peered":"2023-10-16T13:46:25.578111+0000","last_unstale":"2023-10-16T13:46:38.648377+0000","last_undegraded":"2023-10-16T13:46:38.648377+0000","last_fullsized":"2023-10-16T13:46:38.648377+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:25:44.802079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.19","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.672877+0000","last_change":"2023-10-16T13:46:37.646754+0000","last_active":"2023-10-16T13:46:39.672877+0000","last_peered":"2023-10-16T13:46:39.672877+0000","last_clean":"2023-10-16T13:46:39.672877+0000","last_became_active":"2023-10-16T13:46:37.646395+0000","last_became_peered":"2023-10-16T13:46:37.646395+0000","last_unstale":"2023-10-16T13:46:39.672877+0000","last_undegraded":"2023-10-16T13:46:39.672877+0000","last_fullsized":"2023-10-16T13:46:39.672877+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_clean_scrub_stamp":"2023-10-16T13:46:36.617142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:40:44.718376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1a","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.670847+0000","last_change":"2023-10-16T13:46:38.656584+0000","last_active":"2023-10-16T13:46:39.670847+0000","last_peered":"2023-10-16T13:46:39.670847+0000","last_clean":"2023-10-16T13:46:39.670847+0000","last_became_active":"2023-10-16T13:46:38.656068+0000","last_became_peered":"2023-10-16T13:46:38.656068+0000","last_unstale":"2023-10-16T13:46:39.670847+0000","last_undegraded":"2023-10-16T13:46:39.670847+0000","last_fullsized":"2023-10-16T13:46:39.670847+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:22:22.209998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.20","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.645880+0000","last_change":"2023-10-16T13:46:25.572825+0000","last_active":"2023-10-16T13:46:38.645880+0000","last_peered":"2023-10-16T13:46:38.645880+0000","last_clean":"2023-10-16T13:46:38.645880+0000","last_became_active":"2023-10-16T13:46:25.572507+0000","last_became_peered":"2023-10-16T13:46:25.572507+0000","last_unstale":"2023-10-16T13:46:38.645880+0000","last_undegraded":"2023-10-16T13:46:38.645880+0000","last_fullsized":"2023-10-16T13:46:38.645880+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:51:29.125394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.22","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.929819+0000","last_change":"2023-10-16T13:46:38.673677+0000","last_active":"2023-10-16T13:46:39.929819+0000","last_peered":"2023-10-16T13:46:39.929819+0000","last_clean":"2023-10-16T13:46:39.929819+0000","last_became_active":"2023-10-16T13:46:38.673317+0000","last_became_peered":"2023-10-16T13:46:38.673317+0000","last_unstale":"2023-10-16T13:46:39.929819+0000","last_undegraded":"2023-10-16T13:46:39.929819+0000","last_fullsized":"2023-10-16T13:46:39.929819+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:28:07.580655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.21","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.651614+0000","last_change":"2023-10-16T13:46:25.580726+0000","last_active":"2023-10-16T13:46:38.651614+0000","last_peered":"2023-10-16T13:46:38.651614+0000","last_clean":"2023-10-16T13:46:38.651614+0000","last_became_active":"2023-10-16T13:46:25.580423+0000","last_became_peered":"2023-10-16T13:46:25.580423+0000","last_unstale":"2023-10-16T13:46:38.651614+0000","last_undegraded":"2023-10-16T13:46:38.651614+0000","last_fullsized":"2023-10-16T13:46:38.651614+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:21:32.260993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.23","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.654095+0000","last_change":"2023-10-16T13:46:38.657677+0000","last_active":"2023-10-16T13:46:39.654095+0000","last_peered":"2023-10-16T13:46:39.654095+0000","last_clean":"2023-10-16T13:46:39.654095+0000","last_became_active":"2023-10-16T13:46:38.657347+0000","last_became_peered":"2023-10-16T13:46:38.657347+0000","last_unstale":"2023-10-16T13:46:39.654095+0000","last_undegraded":"2023-10-16T13:46:39.654095+0000","last_fullsized":"2023-10-16T13:46:39.654095+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:21:04.017774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.22","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.642763+0000","last_change":"2023-10-16T13:46:25.569845+0000","last_active":"2023-10-16T13:46:38.642763+0000","last_peered":"2023-10-16T13:46:38.642763+0000","last_clean":"2023-10-16T13:46:38.642763+0000","last_became_active":"2023-10-16T13:46:25.569524+0000","last_became_peered":"2023-10-16T13:46:25.569524+0000","last_unstale":"2023-10-16T13:46:38.642763+0000","last_undegraded":"2023-10-16T13:46:38.642763+0000","last_fullsized":"2023-10-16T13:46:38.642763+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:16:15.086261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.20","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.929777+0000","last_change":"2023-10-16T13:46:38.672106+0000","last_active":"2023-10-16T13:46:39.929777+0000","last_peered":"2023-10-16T13:46:39.929777+0000","last_clean":"2023-10-16T13:46:39.929777+0000","last_became_active":"2023-10-16T13:46:38.671793+0000","last_became_peered":"2023-10-16T13:46:38.671793+0000","last_unstale":"2023-10-16T13:46:39.929777+0000","last_undegraded":"2023-10-16T13:46:39.929777+0000","last_fullsized":"2023-10-16T13:46:39.929777+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:32:54.311537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.23","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.646695+0000","last_change":"2023-10-16T13:46:25.568230+0000","last_active":"2023-10-16T13:46:38.646695+0000","last_peered":"2023-10-16T13:46:38.646695+0000","last_clean":"2023-10-16T13:46:38.646695+0000","last_became_active":"2023-10-16T13:46:25.567927+0000","last_became_peered":"2023-10-16T13:46:25.567927+0000","last_unstale":"2023-10-16T13:46:38.646695+0000","last_undegraded":"2023-10-16T13:46:38.646695+0000","last_fullsized":"2023-10-16T13:46:38.646695+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:51:29.125394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.21","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.866626+0000","last_change":"2023-10-16T13:46:38.663451+0000","last_active":"2023-10-16T13:46:40.866626+0000","last_peered":"2023-10-16T13:46:40.866626+0000","last_clean":"2023-10-16T13:46:40.866626+0000","last_became_active":"2023-10-16T13:46:38.662499+0000","last_became_peered":"2023-10-16T13:46:38.662499+0000","last_unstale":"2023-10-16T13:46:40.866626+0000","last_undegraded":"2023-10-16T13:46:40.866626+0000","last_fullsized":"2023-10-16T13:46:40.866626+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:43:32.158816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.26","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.581740+0000","last_change":"2023-10-16T13:46:38.676278+0000","last_active":"2023-10-16T13:46:40.581740+0000","last_peered":"2023-10-16T13:46:40.581740+0000","last_clean":"2023-10-16T13:46:40.581740+0000","last_became_active":"2023-10-16T13:46:38.675992+0000","last_became_peered":"2023-10-16T13:46:38.675992+0000","last_unstale":"2023-10-16T13:46:40.581740+0000","last_undegraded":"2023-10-16T13:46:40.581740+0000","last_fullsized":"2023-10-16T13:46:40.581740+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:47:41.022437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.24","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.640853+0000","last_change":"2023-10-16T13:46:25.573728+0000","last_active":"2023-10-16T13:46:38.640853+0000","last_peered":"2023-10-16T13:46:38.640853+0000","last_clean":"2023-10-16T13:46:38.640853+0000","last_became_active":"2023-10-16T13:46:25.573427+0000","last_became_peered":"2023-10-16T13:46:25.573427+0000","last_unstale":"2023-10-16T13:46:38.640853+0000","last_undegraded":"2023-10-16T13:46:38.640853+0000","last_fullsized":"2023-10-16T13:46:38.640853+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:02:44.293532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.27","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.655525+0000","last_change":"2023-10-16T13:46:38.666370+0000","last_active":"2023-10-16T13:46:39.655525+0000","last_peered":"2023-10-16T13:46:39.655525+0000","last_clean":"2023-10-16T13:46:39.655525+0000","last_became_active":"2023-10-16T13:46:38.662756+0000","last_became_peered":"2023-10-16T13:46:38.662756+0000","last_unstale":"2023-10-16T13:46:39.655525+0000","last_undegraded":"2023-10-16T13:46:39.655525+0000","last_fullsized":"2023-10-16T13:46:39.655525+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:38:44.186017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.25","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.653088+0000","last_change":"2023-10-16T13:46:25.578298+0000","last_active":"2023-10-16T13:46:38.653088+0000","last_peered":"2023-10-16T13:46:38.653088+0000","last_clean":"2023-10-16T13:46:38.653088+0000","last_became_active":"2023-10-16T13:46:25.578057+0000","last_became_peered":"2023-10-16T13:46:25.578057+0000","last_unstale":"2023-10-16T13:46:38.653088+0000","last_undegraded":"2023-10-16T13:46:38.653088+0000","last_fullsized":"2023-10-16T13:46:38.653088+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:08:20.004685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.26","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.650659+0000","last_change":"2023-10-16T13:46:25.579701+0000","last_active":"2023-10-16T13:46:38.650659+0000","last_peered":"2023-10-16T13:46:38.650659+0000","last_clean":"2023-10-16T13:46:38.650659+0000","last_became_active":"2023-10-16T13:46:25.579378+0000","last_became_peered":"2023-10-16T13:46:25.579378+0000","last_unstale":"2023-10-16T13:46:38.650659+0000","last_undegraded":"2023-10-16T13:46:38.650659+0000","last_fullsized":"2023-10-16T13:46:38.650659+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:16:32.013927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.24","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.098110+0000","last_change":"2023-10-16T13:46:38.673084+0000","last_active":"2023-10-16T13:46:40.098110+0000","last_peered":"2023-10-16T13:46:40.098110+0000","last_clean":"2023-10-16T13:46:40.098110+0000","last_became_active":"2023-10-16T13:46:38.670381+0000","last_became_peered":"2023-10-16T13:46:38.670381+0000","last_unstale":"2023-10-16T13:46:40.098110+0000","last_undegraded":"2023-10-16T13:46:40.098110+0000","last_fullsized":"2023-10-16T13:46:40.098110+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:29:53.145261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.25","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.673681+0000","last_change":"2023-10-16T13:46:38.655995+0000","last_active":"2023-10-16T13:46:39.673681+0000","last_peered":"2023-10-16T13:46:39.673681+0000","last_clean":"2023-10-16T13:46:39.673681+0000","last_became_active":"2023-10-16T13:46:38.655560+0000","last_became_peered":"2023-10-16T13:46:38.655560+0000","last_unstale":"2023-10-16T13:46:39.673681+0000","last_undegraded":"2023-10-16T13:46:39.673681+0000","last_fullsized":"2023-10-16T13:46:39.673681+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:23:00.385693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.27","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.656567+0000","last_change":"2023-10-16T13:46:25.572081+0000","last_active":"2023-10-16T13:46:38.656567+0000","last_peered":"2023-10-16T13:46:38.656567+0000","last_clean":"2023-10-16T13:46:38.656567+0000","last_became_active":"2023-10-16T13:46:25.568456+0000","last_became_peered":"2023-10-16T13:46:25.568456+0000","last_unstale":"2023-10-16T13:46:38.656567+0000","last_undegraded":"2023-10-16T13:46:38.656567+0000","last_fullsized":"2023-10-16T13:46:38.656567+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:26:56.181979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.2a","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.670583+0000","last_change":"2023-10-16T13:46:38.656382+0000","last_active":"2023-10-16T13:46:39.670583+0000","last_peered":"2023-10-16T13:46:39.670583+0000","last_clean":"2023-10-16T13:46:39.670583+0000","last_became_active":"2023-10-16T13:46:38.655682+0000","last_became_peered":"2023-10-16T13:46:38.655682+0000","last_unstale":"2023-10-16T13:46:39.670583+0000","last_undegraded":"2023-10-16T13:46:39.670583+0000","last_fullsized":"2023-10-16T13:46:39.670583+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:14:58.526615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.28","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.640479+0000","last_change":"2023-10-16T13:46:25.583046+0000","last_active":"2023-10-16T13:46:38.640479+0000","last_peered":"2023-10-16T13:46:38.640479+0000","last_clean":"2023-10-16T13:46:38.640479+0000","last_became_active":"2023-10-16T13:46:25.582711+0000","last_became_peered":"2023-10-16T13:46:25.582711+0000","last_unstale":"2023-10-16T13:46:38.640479+0000","last_undegraded":"2023-10-16T13:46:38.640479+0000","last_fullsized":"2023-10-16T13:46:38.640479+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:23:43.446191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.2b","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.581119+0000","last_change":"2023-10-16T13:46:38.663737+0000","last_active":"2023-10-16T13:46:40.581119+0000","last_peered":"2023-10-16T13:46:40.581119+0000","last_clean":"2023-10-16T13:46:40.581119+0000","last_became_active":"2023-10-16T13:46:38.663429+0000","last_became_peered":"2023-10-16T13:46:38.663429+0000","last_unstale":"2023-10-16T13:46:40.581119+0000","last_undegraded":"2023-10-16T13:46:40.581119+0000","last_fullsized":"2023-10-16T13:46:40.581119+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:10:59.149373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.29","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.658035+0000","last_change":"2023-10-16T13:46:25.578743+0000","last_active":"2023-10-16T13:46:38.658035+0000","last_peered":"2023-10-16T13:46:38.658035+0000","last_clean":"2023-10-16T13:46:38.658035+0000","last_became_active":"2023-10-16T13:46:25.578447+0000","last_became_peered":"2023-10-16T13:46:25.578447+0000","last_unstale":"2023-10-16T13:46:38.658035+0000","last_undegraded":"2023-10-16T13:46:38.658035+0000","last_fullsized":"2023-10-16T13:46:38.658035+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:46:55.400521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.28","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.654324+0000","last_change":"2023-10-16T13:46:38.665539+0000","last_active":"2023-10-16T13:46:39.654324+0000","last_peered":"2023-10-16T13:46:39.654324+0000","last_clean":"2023-10-16T13:46:39.654324+0000","last_became_active":"2023-10-16T13:46:38.662338+0000","last_became_peered":"2023-10-16T13:46:38.662338+0000","last_unstale":"2023-10-16T13:46:39.654324+0000","last_undegraded":"2023-10-16T13:46:39.654324+0000","last_fullsized":"2023-10-16T13:46:39.654324+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:08:41.090480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.2a","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.650013+0000","last_change":"2023-10-16T13:46:25.581238+0000","last_active":"2023-10-16T13:46:38.650013+0000","last_peered":"2023-10-16T13:46:38.650013+0000","last_clean":"2023-10-16T13:46:38.650013+0000","last_became_active":"2023-10-16T13:46:25.580555+0000","last_became_peered":"2023-10-16T13:46:25.580555+0000","last_unstale":"2023-10-16T13:46:38.650013+0000","last_undegraded":"2023-10-16T13:46:38.650013+0000","last_fullsized":"2023-10-16T13:46:38.650013+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:54:51.781852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.29","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.866840+0000","last_change":"2023-10-16T13:46:38.663142+0000","last_active":"2023-10-16T13:46:40.866840+0000","last_peered":"2023-10-16T13:46:40.866840+0000","last_clean":"2023-10-16T13:46:40.866840+0000","last_became_active":"2023-10-16T13:46:38.662849+0000","last_became_peered":"2023-10-16T13:46:38.662849+0000","last_unstale":"2023-10-16T13:46:40.866840+0000","last_undegraded":"2023-10-16T13:46:40.866840+0000","last_fullsized":"2023-10-16T13:46:40.866840+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:58:13.297662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.2b","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.647028+0000","last_change":"2023-10-16T13:46:25.584056+0000","last_active":"2023-10-16T13:46:38.647028+0000","last_peered":"2023-10-16T13:46:38.647028+0000","last_clean":"2023-10-16T13:46:38.647028+0000","last_became_active":"2023-10-16T13:46:25.583758+0000","last_became_peered":"2023-10-16T13:46:25.583758+0000","last_unstale":"2023-10-16T13:46:38.647028+0000","last_undegraded":"2023-10-16T13:46:38.647028+0000","last_fullsized":"2023-10-16T13:46:38.647028+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:48:45.889362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2e","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.581648+0000","last_change":"2023-10-16T13:46:38.670081+0000","last_active":"2023-10-16T13:46:40.581648+0000","last_peered":"2023-10-16T13:46:40.581648+0000","last_clean":"2023-10-16T13:46:40.581648+0000","last_became_active":"2023-10-16T13:46:38.665403+0000","last_became_peered":"2023-10-16T13:46:38.665403+0000","last_unstale":"2023-10-16T13:46:40.581648+0000","last_undegraded":"2023-10-16T13:46:40.581648+0000","last_fullsized":"2023-10-16T13:46:40.581648+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:20:35.300682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.2c","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.649945+0000","last_change":"2023-10-16T13:46:25.572889+0000","last_active":"2023-10-16T13:46:38.649945+0000","last_peered":"2023-10-16T13:46:38.649945+0000","last_clean":"2023-10-16T13:46:38.649945+0000","last_became_active":"2023-10-16T13:46:25.572611+0000","last_became_peered":"2023-10-16T13:46:25.572611+0000","last_unstale":"2023-10-16T13:46:38.649945+0000","last_undegraded":"2023-10-16T13:46:38.649945+0000","last_fullsized":"2023-10-16T13:46:38.649945+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:57:59.993900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2f","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.098523+0000","last_change":"2023-10-16T13:46:38.669880+0000","last_active":"2023-10-16T13:46:40.098523+0000","last_peered":"2023-10-16T13:46:40.098523+0000","last_clean":"2023-10-16T13:46:40.098523+0000","last_became_active":"2023-10-16T13:46:38.662277+0000","last_became_peered":"2023-10-16T13:46:38.662277+0000","last_unstale":"2023-10-16T13:46:40.098523+0000","last_undegraded":"2023-10-16T13:46:40.098523+0000","last_fullsized":"2023-10-16T13:46:40.098523+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:01:09.625064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2d","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.648084+0000","last_change":"2023-10-16T13:46:25.578104+0000","last_active":"2023-10-16T13:46:38.648084+0000","last_peered":"2023-10-16T13:46:38.648084+0000","last_clean":"2023-10-16T13:46:38.648084+0000","last_became_active":"2023-10-16T13:46:25.577833+0000","last_became_peered":"2023-10-16T13:46:25.577833+0000","last_unstale":"2023-10-16T13:46:38.648084+0000","last_undegraded":"2023-10-16T13:46:38.648084+0000","last_fullsized":"2023-10-16T13:46:38.648084+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:43:18.855055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2c","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.930376+0000","last_change":"2023-10-16T13:46:38.660311+0000","last_active":"2023-10-16T13:46:39.930376+0000","last_peered":"2023-10-16T13:46:39.930376+0000","last_clean":"2023-10-16T13:46:39.930376+0000","last_became_active":"2023-10-16T13:46:38.660007+0000","last_became_peered":"2023-10-16T13:46:38.660007+0000","last_unstale":"2023-10-16T13:46:39.930376+0000","last_undegraded":"2023-10-16T13:46:39.930376+0000","last_fullsized":"2023-10-16T13:46:39.930376+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:44:59.606324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.2e","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.648186+0000","last_change":"2023-10-16T13:46:25.567084+0000","last_active":"2023-10-16T13:46:38.648186+0000","last_peered":"2023-10-16T13:46:38.648186+0000","last_clean":"2023-10-16T13:46:38.648186+0000","last_became_active":"2023-10-16T13:46:25.566470+0000","last_became_peered":"2023-10-16T13:46:25.566470+0000","last_unstale":"2023-10-16T13:46:38.648186+0000","last_undegraded":"2023-10-16T13:46:38.648186+0000","last_fullsized":"2023-10-16T13:46:38.648186+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:47:01.254594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.2d","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.655515+0000","last_change":"2023-10-16T13:46:38.660973+0000","last_active":"2023-10-16T13:46:39.655515+0000","last_peered":"2023-10-16T13:46:39.655515+0000","last_clean":"2023-10-16T13:46:39.655515+0000","last_became_active":"2023-10-16T13:46:38.660638+0000","last_became_peered":"2023-10-16T13:46:38.660638+0000","last_unstale":"2023-10-16T13:46:39.655515+0000","last_undegraded":"2023-10-16T13:46:39.655515+0000","last_fullsized":"2023-10-16T13:46:39.655515+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:21:02.259177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.2f","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.656469+0000","last_change":"2023-10-16T13:46:25.577983+0000","last_active":"2023-10-16T13:46:38.656469+0000","last_peered":"2023-10-16T13:46:38.656469+0000","last_clean":"2023-10-16T13:46:38.656469+0000","last_became_active":"2023-10-16T13:46:25.577680+0000","last_became_peered":"2023-10-16T13:46:25.577680+0000","last_unstale":"2023-10-16T13:46:38.656469+0000","last_undegraded":"2023-10-16T13:46:38.656469+0000","last_fullsized":"2023-10-16T13:46:38.656469+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:04:14.478904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.32","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.097945+0000","last_change":"2023-10-16T13:46:38.671012+0000","last_active":"2023-10-16T13:46:40.097945+0000","last_peered":"2023-10-16T13:46:40.097945+0000","last_clean":"2023-10-16T13:46:40.097945+0000","last_became_active":"2023-10-16T13:46:38.663063+0000","last_became_peered":"2023-10-16T13:46:38.663063+0000","last_unstale":"2023-10-16T13:46:40.097945+0000","last_undegraded":"2023-10-16T13:46:40.097945+0000","last_fullsized":"2023-10-16T13:46:40.097945+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:26:39.652821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.30","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.653196+0000","last_change":"2023-10-16T13:46:25.570144+0000","last_active":"2023-10-16T13:46:38.653196+0000","last_peered":"2023-10-16T13:46:38.653196+0000","last_clean":"2023-10-16T13:46:38.653196+0000","last_became_active":"2023-10-16T13:46:25.569866+0000","last_became_peered":"2023-10-16T13:46:25.569866+0000","last_unstale":"2023-10-16T13:46:38.653196+0000","last_undegraded":"2023-10-16T13:46:38.653196+0000","last_fullsized":"2023-10-16T13:46:38.653196+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:53:48.001931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.33","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.929922+0000","last_change":"2023-10-16T13:46:38.656671+0000","last_active":"2023-10-16T13:46:39.929922+0000","last_peered":"2023-10-16T13:46:39.929922+0000","last_clean":"2023-10-16T13:46:39.929922+0000","last_became_active":"2023-10-16T13:46:38.656361+0000","last_became_peered":"2023-10-16T13:46:38.656361+0000","last_unstale":"2023-10-16T13:46:39.929922+0000","last_undegraded":"2023-10-16T13:46:39.929922+0000","last_fullsized":"2023-10-16T13:46:39.929922+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:26:36.610920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.31","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.641759+0000","last_change":"2023-10-16T13:46:25.577384+0000","last_active":"2023-10-16T13:46:38.641759+0000","last_peered":"2023-10-16T13:46:38.641759+0000","last_clean":"2023-10-16T13:46:38.641759+0000","last_became_active":"2023-10-16T13:46:25.577070+0000","last_became_peered":"2023-10-16T13:46:25.577070+0000","last_unstale":"2023-10-16T13:46:38.641759+0000","last_undegraded":"2023-10-16T13:46:38.641759+0000","last_fullsized":"2023-10-16T13:46:38.641759+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:36:08.745864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.30","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.929480+0000","last_change":"2023-10-16T13:46:38.655085+0000","last_active":"2023-10-16T13:46:39.929480+0000","last_peered":"2023-10-16T13:46:39.929480+0000","last_clean":"2023-10-16T13:46:39.929480+0000","last_became_active":"2023-10-16T13:46:38.654776+0000","last_became_peered":"2023-10-16T13:46:38.654776+0000","last_unstale":"2023-10-16T13:46:39.929480+0000","last_undegraded":"2023-10-16T13:46:39.929480+0000","last_fullsized":"2023-10-16T13:46:39.929480+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:27:09.485741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.32","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.653492+0000","last_change":"2023-10-16T13:46:25.581421+0000","last_active":"2023-10-16T13:46:38.653492+0000","last_peered":"2023-10-16T13:46:38.653492+0000","last_clean":"2023-10-16T13:46:38.653492+0000","last_became_active":"2023-10-16T13:46:25.580936+0000","last_became_peered":"2023-10-16T13:46:25.580936+0000","last_unstale":"2023-10-16T13:46:38.653492+0000","last_undegraded":"2023-10-16T13:46:38.653492+0000","last_fullsized":"2023-10-16T13:46:38.653492+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:41:42.462761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.31","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.670511+0000","last_change":"2023-10-16T13:46:38.668886+0000","last_active":"2023-10-16T13:46:39.670511+0000","last_peered":"2023-10-16T13:46:39.670511+0000","last_clean":"2023-10-16T13:46:39.670511+0000","last_became_active":"2023-10-16T13:46:38.668431+0000","last_became_peered":"2023-10-16T13:46:38.668431+0000","last_unstale":"2023-10-16T13:46:39.670511+0000","last_undegraded":"2023-10-16T13:46:39.670511+0000","last_fullsized":"2023-10-16T13:46:39.670511+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:47:08.704282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.33","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.659118+0000","last_change":"2023-10-16T13:46:25.576474+0000","last_active":"2023-10-16T13:46:38.659118+0000","last_peered":"2023-10-16T13:46:38.659118+0000","last_clean":"2023-10-16T13:46:38.659118+0000","last_became_active":"2023-10-16T13:46:25.576187+0000","last_became_peered":"2023-10-16T13:46:25.576187+0000","last_unstale":"2023-10-16T13:46:38.659118+0000","last_undegraded":"2023-10-16T13:46:38.659118+0000","last_fullsized":"2023-10-16T13:46:38.659118+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:39:18.796503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.36","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.581479+0000","last_change":"2023-10-16T13:46:38.664098+0000","last_active":"2023-10-16T13:46:40.581479+0000","last_peered":"2023-10-16T13:46:40.581479+0000","last_clean":"2023-10-16T13:46:40.581479+0000","last_became_active":"2023-10-16T13:46:38.663906+0000","last_became_peered":"2023-10-16T13:46:38.663906+0000","last_unstale":"2023-10-16T13:46:40.581479+0000","last_undegraded":"2023-10-16T13:46:40.581479+0000","last_fullsized":"2023-10-16T13:46:40.581479+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:20:32.747440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.34","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.641276+0000","last_change":"2023-10-16T13:46:25.575302+0000","last_active":"2023-10-16T13:46:38.641276+0000","last_peered":"2023-10-16T13:46:38.641276+0000","last_clean":"2023-10-16T13:46:38.641276+0000","last_became_active":"2023-10-16T13:46:25.575009+0000","last_became_peered":"2023-10-16T13:46:25.575009+0000","last_unstale":"2023-10-16T13:46:38.641276+0000","last_undegraded":"2023-10-16T13:46:38.641276+0000","last_fullsized":"2023-10-16T13:46:38.641276+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:16:03.750659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.37","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.867309+0000","last_change":"2023-10-16T13:46:38.663230+0000","last_active":"2023-10-16T13:46:40.867309+0000","last_peered":"2023-10-16T13:46:40.867309+0000","last_clean":"2023-10-16T13:46:40.867309+0000","last_became_active":"2023-10-16T13:46:38.662861+0000","last_became_peered":"2023-10-16T13:46:38.662861+0000","last_unstale":"2023-10-16T13:46:40.867309+0000","last_undegraded":"2023-10-16T13:46:40.867309+0000","last_fullsized":"2023-10-16T13:46:40.867309+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:30:22.969323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.35","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.653449+0000","last_change":"2023-10-16T13:46:25.578710+0000","last_active":"2023-10-16T13:46:38.653449+0000","last_peered":"2023-10-16T13:46:38.653449+0000","last_clean":"2023-10-16T13:46:38.653449+0000","last_became_active":"2023-10-16T13:46:25.578250+0000","last_became_peered":"2023-10-16T13:46:25.578250+0000","last_unstale":"2023-10-16T13:46:38.653449+0000","last_undegraded":"2023-10-16T13:46:38.653449+0000","last_fullsized":"2023-10-16T13:46:38.653449+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:18:48.273122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.34","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.098636+0000","last_change":"2023-10-16T13:46:38.663016+0000","last_active":"2023-10-16T13:46:40.098636+0000","last_peered":"2023-10-16T13:46:40.098636+0000","last_clean":"2023-10-16T13:46:40.098636+0000","last_became_active":"2023-10-16T13:46:38.662722+0000","last_became_peered":"2023-10-16T13:46:38.662722+0000","last_unstale":"2023-10-16T13:46:40.098636+0000","last_undegraded":"2023-10-16T13:46:40.098636+0000","last_fullsized":"2023-10-16T13:46:40.098636+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:19:18.474581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.36","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.646533+0000","last_change":"2023-10-16T13:46:25.585253+0000","last_active":"2023-10-16T13:46:38.646533+0000","last_peered":"2023-10-16T13:46:38.646533+0000","last_clean":"2023-10-16T13:46:38.646533+0000","last_became_active":"2023-10-16T13:46:25.584881+0000","last_became_peered":"2023-10-16T13:46:25.584881+0000","last_unstale":"2023-10-16T13:46:38.646533+0000","last_undegraded":"2023-10-16T13:46:38.646533+0000","last_fullsized":"2023-10-16T13:46:38.646533+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:16:15.086261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.35","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.673378+0000","last_change":"2023-10-16T13:46:38.661044+0000","last_active":"2023-10-16T13:46:39.673378+0000","last_peered":"2023-10-16T13:46:39.673378+0000","last_clean":"2023-10-16T13:46:39.673378+0000","last_became_active":"2023-10-16T13:46:38.660733+0000","last_became_peered":"2023-10-16T13:46:38.660733+0000","last_unstale":"2023-10-16T13:46:39.673378+0000","last_undegraded":"2023-10-16T13:46:39.673378+0000","last_fullsized":"2023-10-16T13:46:39.673378+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:12:08.292404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.37","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.656374+0000","last_change":"2023-10-16T13:46:25.583298+0000","last_active":"2023-10-16T13:46:38.656374+0000","last_peered":"2023-10-16T13:46:38.656374+0000","last_clean":"2023-10-16T13:46:38.656374+0000","last_became_active":"2023-10-16T13:46:25.582794+0000","last_became_peered":"2023-10-16T13:46:25.582794+0000","last_unstale":"2023-10-16T13:46:38.656374+0000","last_undegraded":"2023-10-16T13:46:38.656374+0000","last_fullsized":"2023-10-16T13:46:38.656374+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:22:31.745564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.3a","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.930318+0000","last_change":"2023-10-16T13:46:38.657295+0000","last_active":"2023-10-16T13:46:39.930318+0000","last_peered":"2023-10-16T13:46:39.930318+0000","last_clean":"2023-10-16T13:46:39.930318+0000","last_became_active":"2023-10-16T13:46:38.656993+0000","last_became_peered":"2023-10-16T13:46:38.656993+0000","last_unstale":"2023-10-16T13:46:39.930318+0000","last_undegraded":"2023-10-16T13:46:39.930318+0000","last_fullsized":"2023-10-16T13:46:39.930318+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:26:05.871114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.38","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.647306+0000","last_change":"2023-10-16T13:46:25.576391+0000","last_active":"2023-10-16T13:46:38.647306+0000","last_peered":"2023-10-16T13:46:38.647306+0000","last_clean":"2023-10-16T13:46:38.647306+0000","last_became_active":"2023-10-16T13:46:25.575906+0000","last_became_peered":"2023-10-16T13:46:25.575906+0000","last_unstale":"2023-10-16T13:46:38.647306+0000","last_undegraded":"2023-10-16T13:46:38.647306+0000","last_fullsized":"2023-10-16T13:46:38.647306+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:44:11.618820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3b","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.655127+0000","last_change":"2023-10-16T13:46:38.668670+0000","last_active":"2023-10-16T13:46:39.655127+0000","last_peered":"2023-10-16T13:46:39.655127+0000","last_clean":"2023-10-16T13:46:39.655127+0000","last_became_active":"2023-10-16T13:46:38.667596+0000","last_became_peered":"2023-10-16T13:46:38.667596+0000","last_unstale":"2023-10-16T13:46:39.655127+0000","last_undegraded":"2023-10-16T13:46:39.655127+0000","last_fullsized":"2023-10-16T13:46:39.655127+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:44:54.832181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.39","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.641719+0000","last_change":"2023-10-16T13:46:25.573056+0000","last_active":"2023-10-16T13:46:38.641719+0000","last_peered":"2023-10-16T13:46:38.641719+0000","last_clean":"2023-10-16T13:46:38.641719+0000","last_became_active":"2023-10-16T13:46:25.572702+0000","last_became_peered":"2023-10-16T13:46:25.572702+0000","last_unstale":"2023-10-16T13:46:38.641719+0000","last_undegraded":"2023-10-16T13:46:38.641719+0000","last_fullsized":"2023-10-16T13:46:38.641719+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:05:54.303051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.38","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.929717+0000","last_change":"2023-10-16T13:46:38.657491+0000","last_active":"2023-10-16T13:46:39.929717+0000","last_peered":"2023-10-16T13:46:39.929717+0000","last_clean":"2023-10-16T13:46:39.929717+0000","last_became_active":"2023-10-16T13:46:38.656959+0000","last_became_peered":"2023-10-16T13:46:38.656959+0000","last_unstale":"2023-10-16T13:46:39.929717+0000","last_undegraded":"2023-10-16T13:46:39.929717+0000","last_fullsized":"2023-10-16T13:46:39.929717+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:57:26.352089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3a","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.651707+0000","last_change":"2023-10-16T13:46:25.568946+0000","last_active":"2023-10-16T13:46:38.651707+0000","last_peered":"2023-10-16T13:46:38.651707+0000","last_clean":"2023-10-16T13:46:38.651707+0000","last_became_active":"2023-10-16T13:46:25.568652+0000","last_became_peered":"2023-10-16T13:46:25.568652+0000","last_unstale":"2023-10-16T13:46:38.651707+0000","last_undegraded":"2023-10-16T13:46:38.651707+0000","last_fullsized":"2023-10-16T13:46:38.651707+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:18:48.273122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.39","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.580943+0000","last_change":"2023-10-16T13:46:38.661321+0000","last_active":"2023-10-16T13:46:40.580943+0000","last_peered":"2023-10-16T13:46:40.580943+0000","last_clean":"2023-10-16T13:46:40.580943+0000","last_became_active":"2023-10-16T13:46:38.660892+0000","last_became_peered":"2023-10-16T13:46:38.660892+0000","last_unstale":"2023-10-16T13:46:40.580943+0000","last_undegraded":"2023-10-16T13:46:40.580943+0000","last_fullsized":"2023-10-16T13:46:40.580943+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:18:49.534924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.3b","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.641227+0000","last_change":"2023-10-16T13:46:25.570352+0000","last_active":"2023-10-16T13:46:38.641227+0000","last_peered":"2023-10-16T13:46:38.641227+0000","last_clean":"2023-10-16T13:46:38.641227+0000","last_became_active":"2023-10-16T13:46:25.570035+0000","last_became_peered":"2023-10-16T13:46:25.570035+0000","last_unstale":"2023-10-16T13:46:38.641227+0000","last_undegraded":"2023-10-16T13:46:38.641227+0000","last_fullsized":"2023-10-16T13:46:38.641227+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:43:07.323286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.3e","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.654292+0000","last_change":"2023-10-16T13:46:38.667376+0000","last_active":"2023-10-16T13:46:39.654292+0000","last_peered":"2023-10-16T13:46:39.654292+0000","last_clean":"2023-10-16T13:46:39.654292+0000","last_became_active":"2023-10-16T13:46:38.667037+0000","last_became_peered":"2023-10-16T13:46:38.667037+0000","last_unstale":"2023-10-16T13:46:39.654292+0000","last_undegraded":"2023-10-16T13:46:39.654292+0000","last_fullsized":"2023-10-16T13:46:39.654292+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:36:17.135906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.3c","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.647005+0000","last_change":"2023-10-16T13:46:25.578245+0000","last_active":"2023-10-16T13:46:38.647005+0000","last_peered":"2023-10-16T13:46:38.647005+0000","last_clean":"2023-10-16T13:46:38.647005+0000","last_became_active":"2023-10-16T13:46:25.577671+0000","last_became_peered":"2023-10-16T13:46:25.577671+0000","last_unstale":"2023-10-16T13:46:38.647005+0000","last_undegraded":"2023-10-16T13:46:38.647005+0000","last_fullsized":"2023-10-16T13:46:38.647005+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:47:01.254594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.3f","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.655304+0000","last_change":"2023-10-16T13:46:38.660190+0000","last_active":"2023-10-16T13:46:39.655304+0000","last_peered":"2023-10-16T13:46:39.655304+0000","last_clean":"2023-10-16T13:46:39.655304+0000","last_became_active":"2023-10-16T13:46:38.659872+0000","last_became_peered":"2023-10-16T13:46:38.659872+0000","last_unstale":"2023-10-16T13:46:39.655304+0000","last_undegraded":"2023-10-16T13:46:39.655304+0000","last_fullsized":"2023-10-16T13:46:39.655304+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:19:06.117364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.3d","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.653204+0000","last_change":"2023-10-16T13:46:25.575243+0000","last_active":"2023-10-16T13:46:38.653204+0000","last_peered":"2023-10-16T13:46:38.653204+0000","last_clean":"2023-10-16T13:46:38.653204+0000","last_became_active":"2023-10-16T13:46:25.574483+0000","last_became_peered":"2023-10-16T13:46:25.574483+0000","last_unstale":"2023-10-16T13:46:38.653204+0000","last_undegraded":"2023-10-16T13:46:38.653204+0000","last_fullsized":"2023-10-16T13:46:38.653204+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:21:32.260993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.3c","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.673308+0000","last_change":"2023-10-16T13:46:38.654365+0000","last_active":"2023-10-16T13:46:39.673308+0000","last_peered":"2023-10-16T13:46:39.673308+0000","last_clean":"2023-10-16T13:46:39.673308+0000","last_became_active":"2023-10-16T13:46:38.654040+0000","last_became_peered":"2023-10-16T13:46:38.654040+0000","last_unstale":"2023-10-16T13:46:39.673308+0000","last_undegraded":"2023-10-16T13:46:39.673308+0000","last_fullsized":"2023-10-16T13:46:39.673308+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:40:23.299174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.3e","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.657356+0000","last_change":"2023-10-16T13:46:25.570175+0000","last_active":"2023-10-16T13:46:38.657356+0000","last_peered":"2023-10-16T13:46:38.657356+0000","last_clean":"2023-10-16T13:46:38.657356+0000","last_became_active":"2023-10-16T13:46:25.569413+0000","last_became_peered":"2023-10-16T13:46:25.569413+0000","last_unstale":"2023-10-16T13:46:38.657356+0000","last_undegraded":"2023-10-16T13:46:38.657356+0000","last_fullsized":"2023-10-16T13:46:38.657356+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:20:50.714013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.3d","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.581412+0000","last_change":"2023-10-16T13:46:38.667427+0000","last_active":"2023-10-16T13:46:40.581412+0000","last_peered":"2023-10-16T13:46:40.581412+0000","last_clean":"2023-10-16T13:46:40.581412+0000","last_became_active":"2023-10-16T13:46:38.667091+0000","last_became_peered":"2023-10-16T13:46:38.667091+0000","last_unstale":"2023-10-16T13:46:40.581412+0000","last_undegraded":"2023-10-16T13:46:40.581412+0000","last_fullsized":"2023-10-16T13:46:40.581412+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:22:55.026877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.3f","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.647146+0000","last_change":"2023-10-16T13:46:25.582893+0000","last_active":"2023-10-16T13:46:38.647146+0000","last_peered":"2023-10-16T13:46:38.647146+0000","last_clean":"2023-10-16T13:46:38.647146+0000","last_became_active":"2023-10-16T13:46:25.581980+0000","last_became_peered":"2023-10-16T13:46:25.581980+0000","last_unstale":"2023-10-16T13:46:38.647146+0000","last_undegraded":"2023-10-16T13:46:38.647146+0000","last_fullsized":"2023-10-16T13:46:38.647146+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:04:47.313972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":30,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":28,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":27,"num_pg":32,"stat_sum":{"num_bytes":3657,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":29,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":348,"ondisk_log_size":348,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575989,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56804,"kb_used_data":3412,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315036,"statfs":{"total":96636764160,"available":96578596864,"internally_reserved":0,"allocated":3493888,"data_stored":2077443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54655435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:45:48 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.61499999999999999,"15min":0.63800000000000001},"min":{"1min":0.35499999999999998,"5min":0.34899999999999998,"15min":0.33000000000000002},"max":{"1min":0.81899999999999995,"5min":0.87,"15min":0.96199999999999997},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.61799999999999999,"15min":0.63300000000000001},"min":{"1min":0.39200000000000002,"5min":0.38900000000000001,"15min":0.379},"max":{"1min":0.83899999999999997,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.72099999999999997}]},{"osd":1,"last update":"Mon Oct 16 13:45:48 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.61399999999999999,"15min":0.64400000000000002},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.79000000000000004,"5min":0.84599999999999997,"15min":0.95899999999999996},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.621,"5min":0.63100000000000001,"15min":0.63800000000000001},"min":{"1min":0.48199999999999998,"5min":0.375,"15min":0.375},"max":{"1min":0.77000000000000002,"5min":0.79100000000000004,"15min":1.022},"last":0.64800000000000002}]},{"osd":2,"last update":"Mon Oct 16 13:45:48 2023","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.69399999999999995,"15min":0.69499999999999995},"min":{"1min":0.48999999999999999,"5min":0.436,"15min":0.436},"max":{"1min":0.83499999999999996,"5min":0.83899999999999997,"15min":0.93500000000000005},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.69599999999999995,"15min":0.70199999999999996},"min":{"1min":0.51000000000000001,"5min":0.47599999999999998,"15min":0.39400000000000002},"max":{"1min":0.871,"5min":0.92800000000000005,"15min":0.93799999999999994},"last":0.83699999999999997}]},{"osd":3,"last update":"Mon Oct 16 13:45:48 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.65000000000000002,"15min":0.66200000000000003},"min":{"1min":0.45200000000000001,"5min":0.438,"15min":0.438},"max":{"1min":0.81499999999999995,"5min":0.89200000000000002,"15min":0.91300000000000003},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.64600000000000002,"15min":0.66300000000000003},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.80400000000000005,"5min":1.24,"15min":1.24},"last":0.80500000000000005}]},{"osd":4,"last update":"Mon Oct 16 13:45:48 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.51300000000000001,"15min":0.52400000000000002},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.72699999999999998,"5min":0.745,"15min":0.86099999999999999},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.623,"5min":0.61399999999999999,"15min":0.61899999999999999},"min":{"1min":0.40699999999999997,"5min":0.36799999999999999,"15min":0.33100000000000002},"max":{"1min":0.77700000000000002,"5min":0.90400000000000003,"15min":0.92900000000000005},"last":0.88500000000000001}]},{"osd":5,"last update":"Mon Oct 16 13:45:48 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.53100000000000003,"15min":0.54400000000000004},"min":{"1min":0.41699999999999998,"5min":0.33400000000000002,"15min":0.32500000000000001},"max":{"1min":0.71599999999999997,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.57299999999999995,"15min":0.58799999999999997},"min":{"1min":0.41199999999999998,"5min":0.38400000000000001,"15min":0.35099999999999998},"max":{"1min":0.80100000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.79200000000000004}]},{"osd":6,"last update":"Mon Oct 16 13:45:48 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.59999999999999998,"15min":0.61499999999999999},"min":{"1min":0.47399999999999998,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.79500000000000004,"5min":0.874,"15min":0.95899999999999996},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.61299999999999999,"15min":0.62},"min":{"1min":0.41099999999999998,"5min":0.378,"15min":0.35199999999999998},"max":{"1min":0.81299999999999994,"5min":0.95799999999999996,"15min":0.97499999999999998},"last":0.92100000000000004}]}]},{"osd":6,"up_from":13,"seq":55834575987,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42348,"kb_used_data":2848,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329492,"statfs":{"total":96636764160,"available":96593399808,"internally_reserved":0,"allocated":2916352,"data_stored":1488653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:46:36 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.65300000000000002,"15min":0.66800000000000004},"min":{"1min":0.41499999999999998,"5min":0.375,"15min":0.28699999999999998},"max":{"1min":0.82599999999999996,"5min":0.89400000000000002,"15min":0.92300000000000004},"last":0.95199999999999996},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.64500000000000002,"15min":0.66400000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.80400000000000005,"5min":0.91400000000000003,"15min":0.93799999999999994},"last":0.81100000000000005}]},{"osd":1,"last update":"Mon Oct 16 13:46:36 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.58599999999999997,"15min":0.624},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.33100000000000002},"max":{"1min":0.70299999999999996,"5min":0.86899999999999999,"15min":0.90900000000000003},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.63,"5min":0.61399999999999999,"15min":0.62},"min":{"1min":0.41699999999999998,"5min":0.39800000000000002,"15min":0.38300000000000001},"max":{"1min":0.85799999999999998,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.90800000000000003}]},{"osd":2,"last update":"Mon Oct 16 13:46:36 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.622,"15min":0.63300000000000001},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.85799999999999998,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.59999999999999998,"15min":0.624},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.749,"5min":0.89000000000000001,"15min":0.91000000000000003},"last":0.76400000000000001}]},{"osd":3,"last update":"Mon Oct 16 13:46:36 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63900000000000001,"15min":0.66300000000000003},"min":{"1min":0.439,"5min":0.39900000000000002,"15min":0.38200000000000001},"max":{"1min":0.874,"5min":0.92400000000000004,"15min":0.95599999999999996},"last":0.753},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.69099999999999995,"15min":0.71599999999999997},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.42299999999999999},"max":{"1min":0.85099999999999998,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.85199999999999998}]},{"osd":4,"last update":"Mon Oct 16 13:46:36 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.51800000000000002,"15min":0.52700000000000002},"min":{"1min":0.39000000000000001,"5min":0.35999999999999999,"15min":0.315},"max":{"1min":0.78000000000000003,"5min":0.85599999999999998,"15min":0.95799999999999996},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61699999999999999,"15min":0.62},"min":{"1min":0.375,"5min":0.35499999999999998,"15min":0.32900000000000001},"max":{"1min":0.83699999999999997,"5min":0.98799999999999999,"15min":1.0309999999999999},"last":0.622}]},{"osd":5,"last update":"Mon Oct 16 13:46:36 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.59799999999999998,"15min":0.59399999999999997},"min":{"1min":0.39800000000000002,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.81100000000000005,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.66500000000000004,"15min":0.66000000000000003},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.308},"max":{"1min":0.85899999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.64200000000000002}]},{"osd":7,"last update":"Mon Oct 16 13:46:36 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.60899999999999999,"15min":0.60499999999999998},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.80000000000000004,"5min":1.081,"15min":1.081},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65200000000000002},"min":{"1min":0.42799999999999999,"5min":0.41899999999999998,"15min":0.375},"max":{"1min":0.91500000000000004,"5min":1.099,"15min":1.099},"last":0.56599999999999995}]}]},{"osd":1,"up_from":13,"seq":55834575987,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47880,"kb_used_data":2808,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323960,"statfs":{"total":96636764160,"available":96587735040,"internally_reserved":0,"allocated":2875392,"data_stored":1483474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":46135753},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:45:54 2023","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.40100000000000002,"15min":0.40999999999999998},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.71999999999999997},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.497,"5min":0.496,"15min":0.499},"min":{"1min":0.34599999999999997,"5min":0.253,"15min":0.253},"max":{"1min":0.68000000000000005,"5min":0.70999999999999996,"15min":0.77200000000000002},"last":0.41699999999999998}]},{"osd":2,"last update":"Mon Oct 16 13:45:54 2023","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.45500000000000002,"15min":0.45800000000000002},"min":{"1min":0.28000000000000003,"5min":0.254,"15min":0.254},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.48199999999999998,"15min":0.49299999999999999},"min":{"1min":0.315,"5min":0.315,"15min":0.30199999999999999},"max":{"1min":0.64200000000000002,"5min":0.66900000000000004,"15min":0.71399999999999997},"last":0.34799999999999998}]},{"osd":3,"last update":"Mon Oct 16 13:45:54 2023","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.438,"15min":0.43099999999999999},"min":{"1min":0.33900000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.57399999999999995,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.47299999999999998,"15min":0.47899999999999998},"min":{"1min":0.26700000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.64800000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.442}]},{"osd":4,"last update":"Mon Oct 16 13:45:54 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.66000000000000003,"15min":0.66700000000000004},"min":{"1min":0.52600000000000002,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.86099999999999999,"5min":0.95899999999999996,"15min":1.0069999999999999},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.66200000000000003,"15min":0.67200000000000004},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.747,"5min":0.91300000000000003,"15min":0.997},"last":0.60399999999999998}]},{"osd":5,"last update":"Mon Oct 16 13:45:54 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.69799999999999995,"15min":0.71399999999999997},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.92300000000000004,"5min":0.97499999999999998,"15min":1.0760000000000001},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.73199999999999998,"15min":0.73399999999999999},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.46200000000000002},"max":{"1min":0.90000000000000002,"5min":1.0469999999999999,"15min":1.0660000000000001},"last":0.621}]},{"osd":6,"last update":"Mon Oct 16 13:45:54 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.69099999999999995,"15min":0.68500000000000005},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.80100000000000005,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.70799999999999996,"15min":0.71299999999999997},"min":{"1min":0.499,"5min":0.499,"15min":0.47799999999999998},"max":{"1min":0.88100000000000001,"5min":1.03,"15min":1.083},"last":0.58499999999999996}]},{"osd":7,"last update":"Mon Oct 16 13:45:54 2023","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.73899999999999999,"15min":0.745},"min":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.48899999999999999},"max":{"1min":0.91400000000000003,"5min":1.0629999999999999,"15min":1.0660000000000001},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.751,"15min":0.751},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.92700000000000005,"5min":1.042,"15min":1.0580000000000001},"last":0.67300000000000004}]}]},{"osd":0,"up_from":13,"seq":55834575988,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30412,"kb_used_data":3392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341428,"statfs":{"total":96636764160,"available":96605622272,"internally_reserved":0,"allocated":3473408,"data_stored":2073932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 13:45:42 2023","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.40899999999999997,"15min":0.42399999999999999},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.58699999999999997,"5min":0.65800000000000003,"15min":0.71399999999999997},"last":0.31},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.42599999999999999,"15min":0.442},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.25900000000000001},"max":{"1min":0.58599999999999997,"5min":0.64600000000000002,"15min":0.82199999999999995},"last":0.36099999999999999}]},{"osd":2,"last update":"Mon Oct 16 13:45:42 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44500000000000001,"15min":0.45800000000000002},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.75700000000000001},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.46800000000000003,"15min":0.48699999999999999},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.83299999999999996},"last":0.41499999999999998}]},{"osd":3,"last update":"Mon Oct 16 13:45:42 2023","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.43099999999999999,"15min":0.45400000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.61799999999999999,"5min":0.63400000000000001,"15min":0.78600000000000003},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.51600000000000001,"15min":0.52500000000000002},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.70399999999999996,"5min":0.82999999999999996,"15min":0.86399999999999999},"last":0.44700000000000001}]},{"osd":4,"last update":"Mon Oct 16 13:45:42 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.66500000000000004,"15min":0.68600000000000005},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.96999999999999997,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.66300000000000003,"15min":0.69299999999999995},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":1.0049999999999999},"last":0.67700000000000005}]},{"osd":5,"last update":"Mon Oct 16 13:45:42 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.66700000000000004,"15min":0.69499999999999995},"min":{"1min":0.44500000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.95399999999999996,"5min":0.95999999999999996,"15min":1.0249999999999999},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.70599999999999996,"15min":0.72999999999999998},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.081,"5min":1.081,"15min":1.081},"last":0.55400000000000005}]},{"osd":6,"last update":"Mon Oct 16 13:45:42 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.71599999999999997,"15min":0.73499999999999999},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.034,"5min":1.103,"15min":1.103},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.70199999999999996,"15min":0.72499999999999998},"min":{"1min":0.42999999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.91000000000000003,"5min":1.0409999999999999,"15min":1.05},"last":0.71099999999999997}]},{"osd":7,"last update":"Mon Oct 16 13:45:42 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.71899999999999997,"15min":0.746},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.002,"5min":1.002,"15min":1.056},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.72099999999999997,"15min":0.73699999999999999},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.99199999999999999,"5min":1.087,"15min":1.087},"last":0.75900000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575988,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52040,"kb_used_data":2808,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319800,"statfs":{"total":96636764160,"available":96583475200,"internally_reserved":0,"allocated":2875392,"data_stored":1476565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.41499999999999998,"15min":0.40899999999999997},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.623,"5min":0.71699999999999997,"15min":0.77500000000000002},"last":0.316},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.47699999999999998,"15min":0.47999999999999998},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.28699999999999998}]},{"osd":1,"last update":"Mon Oct 16 13:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.502,"15min":0.5},"min":{"1min":0.317,"5min":0.311,"15min":0.311},"max":{"1min":0.66800000000000004,"5min":0.73199999999999998,"15min":0.81399999999999995},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.48999999999999999,"15min":0.47699999999999998},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.27700000000000002},"max":{"1min":0.69999999999999996,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.42399999999999999}]},{"osd":3,"last update":"Mon Oct 16 13:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.443,"15min":0.44},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.59699999999999998,"5min":0.70099999999999996,"15min":0.72599999999999998},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.54300000000000004,"15min":0.54400000000000004},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.82699999999999996,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.35599999999999998}]},{"osd":4,"last update":"Mon Oct 16 13:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.72699999999999998,"15min":0.71799999999999997},"min":{"1min":0.51500000000000001,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.89000000000000001,"5min":1.3600000000000001,"15min":1.3600000000000001},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.72599999999999998,"15min":0.71899999999999997},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.82899999999999996,"5min":1.403,"15min":1.403},"last":0.47999999999999998}]},{"osd":5,"last update":"Mon Oct 16 13:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.68600000000000005,"15min":0.68899999999999995},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.81100000000000005,"5min":1.486,"15min":1.486},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.71199999999999997,"15min":0.69999999999999996},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.90200000000000002,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":0.51600000000000001}]},{"osd":6,"last update":"Mon Oct 16 13:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.67600000000000005,"15min":0.66600000000000004},"min":{"1min":0.44500000000000001,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.80200000000000005,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.72599999999999998,"15min":0.72199999999999998},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.90600000000000003,"5min":1.4379999999999999,"15min":1.4379999999999999},"last":0.504}]},{"osd":7,"last update":"Mon Oct 16 13:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.73899999999999999,"15min":0.72099999999999997},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.94099999999999995,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.72999999999999998,"15min":0.73099999999999998},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.91500000000000004,"5min":1.377,"15min":1.377},"last":0.44600000000000001}]}]},{"osd":3,"up_from":13,"seq":55834575987,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56212,"kb_used_data":2824,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315628,"statfs":{"total":96636764160,"available":96579203072,"internally_reserved":0,"allocated":2891776,"data_stored":1483622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.40300000000000002,"15min":0.40999999999999998},"min":{"1min":0.23499999999999999,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.57199999999999995,"5min":0.64200000000000002,"15min":0.72899999999999998},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.41499999999999998,"15min":0.42799999999999999},"min":{"1min":0.27300000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.60099999999999998,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.376}]},{"osd":1,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.42699999999999999,"15min":0.434},"min":{"1min":0.26700000000000002,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.55900000000000005,"5min":0.68200000000000005,"15min":0.76300000000000001},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.435,"5min":0.45900000000000002,"15min":0.47999999999999998},"min":{"1min":0.27700000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.65500000000000003,"5min":0.70699999999999996,"15min":0.77800000000000002},"last":0.42899999999999999}]},{"osd":2,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.47999999999999998,"15min":0.49299999999999999},"min":{"1min":0.33800000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.621,"5min":1.167,"15min":1.167},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.501,"15min":0.52300000000000002},"min":{"1min":0.312,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.625,"5min":0.73299999999999998,"15min":0.78700000000000003},"last":0.54000000000000004}]},{"osd":4,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.65100000000000002,"15min":0.66500000000000004},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.91200000000000003,"5min":0.998,"15min":1.137},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.68899999999999995,"15min":0.69399999999999995},"min":{"1min":0.441,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.087,"5min":1.2110000000000001,"15min":1.2909999999999999},"last":0.65500000000000003}]},{"osd":5,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.67700000000000005,"15min":0.70499999999999996},"min":{"1min":0.45500000000000002,"5min":0.42899999999999999,"15min":0.374},"max":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.2729999999999999},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.69299999999999995,"15min":0.69999999999999996},"min":{"1min":0.45600000000000002,"5min":0.432,"15min":0.432},"max":{"1min":0.98799999999999999,"5min":1.0549999999999999,"15min":1.173},"last":0.68000000000000005}]},{"osd":6,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.72899999999999998,"15min":0.71899999999999997},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.41799999999999998},"max":{"1min":0.99099999999999999,"5min":5.0999999999999996,"15min":5.0999999999999996},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.70999999999999996,"15min":0.73399999999999999},"min":{"1min":0.49199999999999999,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.0029999999999999,"5min":1.0309999999999999,"15min":1.2290000000000001},"last":0.72899999999999998}]},{"osd":7,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.71899999999999997,"15min":0.73399999999999999},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.048,"5min":1.048,"15min":1.2569999999999999},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.73299999999999998,"15min":0.746},"min":{"1min":0.501,"5min":0.47799999999999998,"15min":0.43099999999999999},"max":{"1min":1.0369999999999999,"5min":1.262,"15min":1.262},"last":0.70799999999999996}]}]},{"osd":4,"up_from":13,"seq":55834575986,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52052,"kb_used_data":2816,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319788,"statfs":{"total":96636764160,"available":96583462912,"internally_reserved":0,"allocated":2883584,"data_stored":1484146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.60699999999999998,"15min":0.625},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.33800000000000002},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.503},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.623,"15min":0.63100000000000001},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.84099999999999997,"5min":0.878,"15min":0.89500000000000002},"last":0.56399999999999995}]},{"osd":1,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.63,"15min":0.65600000000000003},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.628,"15min":0.63700000000000001},"min":{"1min":0.41099999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.622}]},{"osd":2,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.65200000000000002,"15min":0.65300000000000002},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.83999999999999997,"5min":0.89900000000000002,"15min":0.90600000000000003},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.65600000000000003,"15min":0.65800000000000003},"min":{"1min":0.39600000000000002,"5min":0.39400000000000002,"15min":0.38400000000000001},"max":{"1min":0.88200000000000001,"5min":0.89700000000000002,"15min":0.92600000000000005},"last":0.746}]},{"osd":3,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.63200000000000001,"15min":0.64500000000000002},"min":{"1min":0.40400000000000003,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.78100000000000003,"5min":0.86599999999999999,"15min":0.99199999999999999},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.67400000000000004,"15min":0.68200000000000005},"min":{"1min":0.44800000000000001,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.79100000000000004,"5min":0.872,"15min":0.93899999999999995},"last":0.72699999999999998}]},{"osd":5,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55100000000000005,"15min":0.53000000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.57999999999999996,"15min":0.56899999999999995},"min":{"1min":0.36099999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.82199999999999995,"5min":0.82299999999999995,"15min":0.94899999999999995},"last":0.77300000000000002}]},{"osd":6,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.625,"15min":0.59799999999999998},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.33200000000000002},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92700000000000005},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.747,"5min":0.71399999999999997,"15min":0.69399999999999995},"min":{"1min":0.52600000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.129,"5min":1.129,"15min":1.129},"last":0.875}]},{"osd":7,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.66000000000000003,"15min":0.63500000000000001},"min":{"1min":0.44400000000000001,"5min":0.41999999999999998,"15min":0.35799999999999998},"max":{"1min":0.97099999999999997,"5min":0.97499999999999998,"15min":0.98599999999999999},"last":0.91100000000000003},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.70799999999999996,"15min":0.68400000000000005},"min":{"1min":0.44,"5min":0.433,"15min":0.38200000000000001},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":1.026},"last":0.80500000000000005}]}]},{"osd":5,"up_from":13,"seq":55834575987,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52064,"kb_used_data":2832,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319776,"statfs":{"total":96636764160,"available":96583450624,"internally_reserved":0,"allocated":2899968,"data_stored":1485212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:46:24 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.61399999999999999,"15min":0.61499999999999999},"min":{"1min":0.314,"5min":0.314,"15min":0.26800000000000002},"max":{"1min":0.878,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63,"15min":0.64200000000000002},"min":{"1min":0.38700000000000001,"5min":0.35899999999999999,"15min":0.29699999999999999},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.67500000000000004}]},{"osd":1,"last update":"Mon Oct 16 13:46:24 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.66000000000000003,"15min":0.66200000000000003},"min":{"1min":0.44600000000000001,"5min":0.34300000000000003,"15min":0.29499999999999998},"max":{"1min":1,"5min":1,"15min":1},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.63800000000000001,"15min":0.64000000000000001},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.622}]},{"osd":2,"last update":"Mon Oct 16 13:46:24 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.67000000000000004,"15min":0.65700000000000003},"min":{"1min":0.41699999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.91000000000000003,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.629,"5min":0.65700000000000003,"15min":0.64900000000000002},"min":{"1min":0.42599999999999999,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.83899999999999997,"5min":0.871,"15min":0.93500000000000005},"last":0.73499999999999999}]},{"osd":3,"last update":"Mon Oct 16 13:46:24 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.67400000000000004,"15min":0.66400000000000003},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.85799999999999998},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.69899999999999995,"15min":0.69499999999999995},"min":{"1min":0.41399999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.4199999999999999,"5min":1.4199999999999999,"15min":1.4199999999999999},"last":0.82799999999999996}]},{"osd":4,"last update":"Mon Oct 16 13:46:24 2023","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.54400000000000004,"15min":0.53300000000000003},"min":{"1min":0.34100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.82499999999999996,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.89700000000000002},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.59299999999999997,"15min":0.58899999999999997},"min":{"1min":0.34300000000000003,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.85399999999999998,"5min":0.89000000000000001,"15min":0.93400000000000005},"last":0.99399999999999999}]},{"osd":6,"last update":"Mon Oct 16 13:46:24 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.58899999999999997,"15min":0.57999999999999996},"min":{"1min":0.38700000000000001,"5min":0.315,"15min":0.315},"max":{"1min":0.877,"5min":0.88900000000000001,"15min":0.93200000000000005},"last":0.92000000000000004},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.61899999999999999},"min":{"1min":0.372,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89600000000000002},"last":0.95799999999999996}]},{"osd":7,"last update":"Mon Oct 16 13:46:24 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.64800000000000002,"15min":0.63300000000000001},"min":{"1min":0.378,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.93899999999999995},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.69699999999999995,"15min":0.68400000000000005},"min":{"1min":0.46200000000000002,"5min":0.375,"15min":0.375},"max":{"1min":0.94699999999999995,"5min":0.98299999999999998,"15min":1.0229999999999999},"last":0.97299999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T13:46:42.829 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T13:46:42.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.980+0000 7f79fbc2b700 1 Processor -- start 2023-10-16T13:46:42.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.980+0000 7f79fbc2b700 1 -- start start 2023-10-16T13:46:42.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.981+0000 7f79fbc2b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f79f415b630 0x7f79f40cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:42.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.981+0000 7f79fbc2b700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f79f40ce4f0 0x7f79f40cfca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:42.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.981+0000 7f79fbc2b700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79f40cc2a0 con 0x7f79f415ba10 2023-10-16T13:46:42.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.981+0000 7f79fbc2b700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79f40cdb20 con 0x7f79f415b630 2023-10-16T13:46:42.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.981+0000 7f79fbc2b700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79f40cc540 con 0x7f79f40ce4f0 2023-10-16T13:46:42.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.981+0000 7f79f99c7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f79f415b630 0x7f79f40cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:42.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.981+0000 7f79f91c6700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f79f415ba10 0x7f79f40cdde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42858/0 (socket says 172.21.15.16:42858) 2023-10-16T13:46:42.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.981+0000 7f79f91c6700 1 -- 172.21.15.16:0/2284516441 learned_addr learned my addr 172.21.15.16:0/2284516441 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:46:42.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.982+0000 7f79f99c7700 1 -- 172.21.15.16:0/2284516441 >> v2:172.21.15.49:3300/0 conn(0x7f79f40ce4f0 msgr2=0x7f79f40cfca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:42.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.982+0000 7f79fa1c8700 1 --2- 172.21.15.16:0/2284516441 >> v2:172.21.15.49:3300/0 conn(0x7f79f40ce4f0 0x7f79f40cfca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:42.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.982+0000 7f79f89c5700 1 -- 172.21.15.16:0/2284516441 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 121175630 0 0) 0x7f79f40cc2a0 con 0x7f79f415ba10 2023-10-16T13:46:42.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.982+0000 7f79f99c7700 1 --2- 172.21.15.16:0/2284516441 >> v2:172.21.15.49:3300/0 conn(0x7f79f40ce4f0 0x7f79f40cfca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:42.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.982+0000 7f79f99c7700 1 -- 172.21.15.16:0/2284516441 >> v1:172.21.15.16:6789/0 conn(0x7f79f415ba10 legacy=0x7f79f40cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:42.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.982+0000 7f79f99c7700 1 -- 172.21.15.16:0/2284516441 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79f40cc820 con 0x7f79f415b630 2023-10-16T13:46:42.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.982+0000 7f79fa1c8700 1 --2- 172.21.15.16:0/2284516441 >> v2:172.21.15.49:3300/0 conn(0x7f79f40ce4f0 0x7f79f40cfca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:46:42.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.982+0000 7f79f99c7700 1 --2- 172.21.15.16:0/2284516441 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79f415b630 0x7f79f40cd5e0 secure :-1 s=READY pgs=2524 cs=0 l=1 rev1=1 crypto rx=0x7f79f40cba90 tx=0x7f79e4009b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a69336277b6abe7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:42.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.982+0000 7f79f89c5700 1 -- 172.21.15.16:0/2284516441 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79e4005e30 con 0x7f79f415b630 2023-10-16T13:46:42.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.982+0000 7f79f89c5700 1 -- 172.21.15.16:0/2284516441 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79e4006030 con 0x7f79f415b630 2023-10-16T13:46:42.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.982+0000 7f79f89c5700 1 -- 172.21.15.16:0/2284516441 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79e40063c0 con 0x7f79f415b630 2023-10-16T13:46:42.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.982+0000 7f79fbc2b700 1 -- 172.21.15.16:0/2284516441 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79f415b630 msgr2=0x7f79f40cd5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:42.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.982+0000 7f79fbc2b700 1 --2- 172.21.15.16:0/2284516441 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79f415b630 0x7f79f40cd5e0 secure :-1 s=READY pgs=2524 cs=0 l=1 rev1=1 crypto rx=0x7f79f40cba90 tx=0x7f79e4009b00 comp rx=0 tx=0).stop 2023-10-16T13:46:42.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.983+0000 7f79fbc2b700 1 -- 172.21.15.16:0/2284516441 shutdown_connections 2023-10-16T13:46:42.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.983+0000 7f79fbc2b700 1 --2- 172.21.15.16:0/2284516441 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79f415b630 0x7f79f40cd5e0 unknown :-1 s=CLOSED pgs=2524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:42.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.983+0000 7f79fbc2b700 1 --2- 172.21.15.16:0/2284516441 >> v2:172.21.15.49:3300/0 conn(0x7f79f40ce4f0 0x7f79f40cfca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:42.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.983+0000 7f79fbc2b700 1 -- 172.21.15.16:0/2284516441 >> 172.21.15.16:0/2284516441 conn(0x7f79f40d14b0 msgr2=0x7f79f40d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:42.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.983+0000 7f79fbc2b700 1 -- 172.21.15.16:0/2284516441 shutdown_connections 2023-10-16T13:46:42.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.983+0000 7f79fbc2b700 1 -- 172.21.15.16:0/2284516441 wait complete. 2023-10-16T13:46:42.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.984+0000 7f79fbc2b700 1 Processor -- start 2023-10-16T13:46:42.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.984+0000 7f79fbc2b700 1 -- start start 2023-10-16T13:46:42.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.985+0000 7f79fbc2b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f79f40ce4f0 0x7f79f4150d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:42.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.985+0000 7f79fbc2b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f79f415b630 0x7f79f40c8580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:42.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.985+0000 7f79fbc2b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79f415ba10 0x7f79f41f4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:42.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.985+0000 7f79fbc2b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79f41512a0 con 0x7f79f415ba10 2023-10-16T13:46:42.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.985+0000 7f79fbc2b700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79f40c7de0 con 0x7f79f40ce4f0 2023-10-16T13:46:42.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.985+0000 7f79fbc2b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79f40c7820 con 0x7f79f415b630 2023-10-16T13:46:42.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.985+0000 7f79fa1c8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79f415ba10 0x7f79f41f4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:42.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.985+0000 7f79fa1c8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79f415ba10 0x7f79f41f4010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53592/0 (socket says 172.21.15.16:53592) 2023-10-16T13:46:42.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.985+0000 7f79fa1c8700 1 -- 172.21.15.16:0/322116265 learned_addr learned my addr 172.21.15.16:0/322116265 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:42.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.985+0000 7f79f99c7700 1 --2- 172.21.15.16:0/322116265 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f79f40ce4f0 0x7f79f4150d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:42.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.985+0000 7f79f91c6700 1 --2- 172.21.15.16:0/322116265 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f79f415b630 0x7f79f40c8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:42.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.985+0000 7f79fa1c8700 1 -- 172.21.15.16:0/322116265 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f79f415b630 msgr2=0x7f79f40c8580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:42.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.985+0000 7f79fa1c8700 1 --2- 172.21.15.16:0/322116265 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f79f415b630 0x7f79f40c8580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:42.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.985+0000 7f79fa1c8700 1 -- 172.21.15.16:0/322116265 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f79f40ce4f0 msgr2=0x7f79f4150d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:42.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.985+0000 7f79fa1c8700 1 --2- 172.21.15.16:0/322116265 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f79f40ce4f0 0x7f79f4150d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:42.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.985+0000 7f79fa1c8700 1 -- 172.21.15.16:0/322116265 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79e4009430 con 0x7f79f415ba10 2023-10-16T13:46:42.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.985+0000 7f79f99c7700 1 --2- 172.21.15.16:0/322116265 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f79f40ce4f0 0x7f79f4150d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:46:42.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.985+0000 7f79fa1c8700 1 --2- 172.21.15.16:0/322116265 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79f415ba10 0x7f79f41f4010 secure :-1 s=READY pgs=2525 cs=0 l=1 rev1=1 crypto rx=0x7f79ec00cd50 tx=0x7f79ec00a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:42.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.985+0000 7f79ea7fc700 1 -- 172.21.15.16:0/322116265 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79ec00cea0 con 0x7f79f415ba10 2023-10-16T13:46:42.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.985+0000 7f79ea7fc700 1 -- 172.21.15.16:0/322116265 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79ec00f480 con 0x7f79f415ba10 2023-10-16T13:46:42.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.985+0000 7f79fbc2b700 1 -- 172.21.15.16:0/322116265 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79f40c7b00 con 0x7f79f415ba10 2023-10-16T13:46:42.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.985+0000 7f79ea7fc700 1 -- 172.21.15.16:0/322116265 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79ec00f810 con 0x7f79f415ba10 2023-10-16T13:46:42.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.986+0000 7f79fbc2b700 1 -- 172.21.15.16:0/322116265 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79f40cb010 con 0x7f79f415ba10 2023-10-16T13:46:42.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.986+0000 7f79fbc2b700 1 -- 172.21.15.16:0/322116265 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79f404ebb0 con 0x7f79f415ba10 2023-10-16T13:46:42.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.987+0000 7f79ea7fc700 1 -- 172.21.15.16:0/322116265 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 138) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f79ec00b040 con 0x7f79f415ba10 2023-10-16T13:46:42.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.987+0000 7f79ea7fc700 1 --2- 172.21.15.16:0/322116265 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f79d806b020 0x7f79d806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:42.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.987+0000 7f79ea7fc700 1 -- 172.21.15.16:0/322116265 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 7021+0+0 (secure 0 0 0) 0x7f79ec08a480 con 0x7f79f415ba10 2023-10-16T13:46:42.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.987+0000 7f79f99c7700 1 --2- 172.21.15.16:0/322116265 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f79d806b020 0x7f79d806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:42.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.988+0000 7f79f99c7700 1 --2- 172.21.15.16:0/322116265 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f79d806b020 0x7f79d806d4e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f79e4000c00 tx=0x7f79e4008070 comp rx=0 tx=0).ready entity=mgr.11190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:43.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:42.992+0000 7f79ea7fc700 1 -- 172.21.15.16:0/322116265 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f79ec054e80 con 0x7f79f415ba10 2023-10-16T13:46:43.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.195+0000 7f79fbc2b700 1 -- 172.21.15.16:0/322116265 --> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f79f40ccde0 con 0x7f79d806b020 2023-10-16T13:46:43.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.209+0000 7f79ea7fc700 1 -- 172.21.15.16:0/322116265 <== mgr.11190 v2:172.21.15.16:6832/3171422553 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+528752 (secure 0 0 0) 0x7f79e4006250 con 0x7f79d806b020 2023-10-16T13:46:43.218 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:46:43.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.213+0000 7f79fbc2b700 1 -- 172.21.15.16:0/322116265 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f79d806b020 msgr2=0x7f79d806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:43.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.213+0000 7f79fbc2b700 1 --2- 172.21.15.16:0/322116265 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f79d806b020 0x7f79d806d4e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f79e4000c00 tx=0x7f79e4008070 comp rx=0 tx=0).stop 2023-10-16T13:46:43.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.213+0000 7f79fbc2b700 1 -- 172.21.15.16:0/322116265 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79f415ba10 msgr2=0x7f79f41f4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:43.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.213+0000 7f79fbc2b700 1 --2- 172.21.15.16:0/322116265 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79f415ba10 0x7f79f41f4010 secure :-1 s=READY pgs=2525 cs=0 l=1 rev1=1 crypto rx=0x7f79ec00cd50 tx=0x7f79ec00a560 comp rx=0 tx=0).stop 2023-10-16T13:46:43.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.213+0000 7f79fbc2b700 1 -- 172.21.15.16:0/322116265 shutdown_connections 2023-10-16T13:46:43.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.213+0000 7f79fbc2b700 1 --2- 172.21.15.16:0/322116265 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f79d806b020 0x7f79d806d4e0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:43.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.213+0000 7f79fbc2b700 1 --2- 172.21.15.16:0/322116265 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79f415ba10 0x7f79f41f4010 unknown :-1 s=CLOSED pgs=2525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:43.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.213+0000 7f79fbc2b700 1 --2- 172.21.15.16:0/322116265 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f79f40ce4f0 0x7f79f4150d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:43.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.213+0000 7f79fbc2b700 1 --2- 172.21.15.16:0/322116265 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f79f415b630 0x7f79f40c8580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:43.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.213+0000 7f79fbc2b700 1 -- 172.21.15.16:0/322116265 >> 172.21.15.16:0/322116265 conn(0x7f79f40d14b0 msgr2=0x7f79f415aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:43.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.213+0000 7f79fbc2b700 1 -- 172.21.15.16:0/322116265 shutdown_connections 2023-10-16T13:46:43.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.213+0000 7f79fbc2b700 1 -- 172.21.15.16:0/322116265 wait complete. 2023-10-16T13:46:43.225 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-10-16T13:46:43.230 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":35,"stamp":"2023-10-16T13:46:42.180406+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596435,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":671,"num_write_kb":2446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":428,"ondisk_log_size":428,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":378,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":389812,"kb_used_data":23740,"kb_used_omap":12,"kb_used_meta":365939,"kb_avail":754584908,"statfs":{"total":773094113280,"available":772694945792,"internally_reserved":0,"allocated":24309760,"data_stored":13053047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12740,"internal_metadata":374722108},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.557370"},"pg_stats":[{"pgid":"1.0","version":"214'348","reported_seq":1206,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.645999+0000","last_change":"2023-10-16T12:10:34.746820+0000","last_active":"2023-10-16T13:46:38.645999+0000","last_peered":"2023-10-16T13:46:38.645999+0000","last_clean":"2023-10-16T13:46:38.645999+0000","last_became_active":"2023-10-16T12:10:34.745574+0000","last_became_peered":"2023-10-16T12:10:34.745574+0000","last_unstale":"2023-10-16T13:46:38.645999+0000","last_undegraded":"2023-10-16T13:46:38.645999+0000","last_fullsized":"2023-10-16T13:46:38.645999+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_clean_scrub_stamp":"2023-10-16T12:10:33.577111+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:14.461986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1a","version":"224'1","reported_seq":17,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.647029+0000","last_change":"2023-10-16T13:46:24.344199+0000","last_active":"2023-10-16T13:46:38.647029+0000","last_peered":"2023-10-16T13:46:38.647029+0000","last_clean":"2023-10-16T13:46:38.647029+0000","last_became_active":"2023-10-16T13:46:24.343947+0000","last_became_peered":"2023-10-16T13:46:24.343947+0000","last_unstale":"2023-10-16T13:46:38.647029+0000","last_undegraded":"2023-10-16T13:46:38.647029+0000","last_fullsized":"2023-10-16T13:46:38.647029+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:17:17.355917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.646904+0000","last_change":"2023-10-16T13:46:25.579532+0000","last_active":"2023-10-16T13:46:38.646904+0000","last_peered":"2023-10-16T13:46:38.646904+0000","last_clean":"2023-10-16T13:46:38.646904+0000","last_became_active":"2023-10-16T13:46:25.579042+0000","last_became_peered":"2023-10-16T13:46:25.579042+0000","last_unstale":"2023-10-16T13:46:38.646904+0000","last_undegraded":"2023-10-16T13:46:38.646904+0000","last_fullsized":"2023-10-16T13:46:38.646904+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:44:22.609007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.1c","version":"228'2","reported_seq":14,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.655435+0000","last_change":"2023-10-16T13:46:37.647219+0000","last_active":"2023-10-16T13:46:39.655435+0000","last_peered":"2023-10-16T13:46:39.655435+0000","last_clean":"2023-10-16T13:46:39.655435+0000","last_became_active":"2023-10-16T13:46:37.646935+0000","last_became_peered":"2023-10-16T13:46:37.646935+0000","last_unstale":"2023-10-16T13:46:39.655435+0000","last_undegraded":"2023-10-16T13:46:39.655435+0000","last_fullsized":"2023-10-16T13:46:39.655435+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_clean_scrub_stamp":"2023-10-16T13:46:36.617142+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:25:57.100673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.1f","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.581696+0000","last_change":"2023-10-16T13:46:38.674756+0000","last_active":"2023-10-16T13:46:40.581696+0000","last_peered":"2023-10-16T13:46:40.581696+0000","last_clean":"2023-10-16T13:46:40.581696+0000","last_became_active":"2023-10-16T13:46:38.674430+0000","last_became_peered":"2023-10-16T13:46:38.674430+0000","last_unstale":"2023-10-16T13:46:40.581696+0000","last_undegraded":"2023-10-16T13:46:40.581696+0000","last_fullsized":"2023-10-16T13:46:40.581696+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:23:02.303147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.641316+0000","last_change":"2023-10-16T13:46:24.346138+0000","last_active":"2023-10-16T13:46:38.641316+0000","last_peered":"2023-10-16T13:46:38.641316+0000","last_clean":"2023-10-16T13:46:38.641316+0000","last_became_active":"2023-10-16T13:46:24.345803+0000","last_became_peered":"2023-10-16T13:46:24.345803+0000","last_unstale":"2023-10-16T13:46:38.641316+0000","last_undegraded":"2023-10-16T13:46:38.641316+0000","last_fullsized":"2023-10-16T13:46:38.641316+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:08:14.564787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.7","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.655005+0000","last_change":"2023-10-16T13:46:25.577432+0000","last_active":"2023-10-16T13:46:38.655005+0000","last_peered":"2023-10-16T13:46:38.655005+0000","last_clean":"2023-10-16T13:46:38.655005+0000","last_became_active":"2023-10-16T13:46:25.577131+0000","last_became_peered":"2023-10-16T13:46:25.577131+0000","last_unstale":"2023-10-16T13:46:38.655005+0000","last_undegraded":"2023-10-16T13:46:38.655005+0000","last_fullsized":"2023-10-16T13:46:38.655005+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:51:29.125394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.6","version":"229'1","reported_seq":13,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.672409+0000","last_change":"2023-10-16T13:46:37.643485+0000","last_active":"2023-10-16T13:46:39.672409+0000","last_peered":"2023-10-16T13:46:39.672409+0000","last_clean":"2023-10-16T13:46:39.672409+0000","last_became_active":"2023-10-16T13:46:37.643166+0000","last_became_peered":"2023-10-16T13:46:37.643166+0000","last_unstale":"2023-10-16T13:46:39.672409+0000","last_undegraded":"2023-10-16T13:46:39.672409+0000","last_fullsized":"2023-10-16T13:46:39.672409+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_clean_scrub_stamp":"2023-10-16T13:46:36.617142+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:25:57.100673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.654859+0000","last_change":"2023-10-16T13:46:38.665257+0000","last_active":"2023-10-16T13:46:39.654859+0000","last_peered":"2023-10-16T13:46:39.654859+0000","last_clean":"2023-10-16T13:46:39.654859+0000","last_became_active":"2023-10-16T13:46:38.660312+0000","last_became_peered":"2023-10-16T13:46:38.660312+0000","last_unstale":"2023-10-16T13:46:39.654859+0000","last_undegraded":"2023-10-16T13:46:39.654859+0000","last_fullsized":"2023-10-16T13:46:39.654859+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:13:45.897327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.646596+0000","last_change":"2023-10-16T13:46:24.340170+0000","last_active":"2023-10-16T13:46:38.646596+0000","last_peered":"2023-10-16T13:46:38.646596+0000","last_clean":"2023-10-16T13:46:38.646596+0000","last_became_active":"2023-10-16T13:46:24.339859+0000","last_became_peered":"2023-10-16T13:46:24.339859+0000","last_unstale":"2023-10-16T13:46:38.646596+0000","last_undegraded":"2023-10-16T13:46:38.646596+0000","last_fullsized":"2023-10-16T13:46:38.646596+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:25:04.566680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.652713+0000","last_change":"2023-10-16T13:46:25.577347+0000","last_active":"2023-10-16T13:46:38.652713+0000","last_peered":"2023-10-16T13:46:38.652713+0000","last_clean":"2023-10-16T13:46:38.652713+0000","last_became_active":"2023-10-16T13:46:25.576780+0000","last_became_peered":"2023-10-16T13:46:25.576780+0000","last_unstale":"2023-10-16T13:46:38.652713+0000","last_undegraded":"2023-10-16T13:46:38.652713+0000","last_fullsized":"2023-10-16T13:46:38.652713+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:16:23.997738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.7","version":"228'2","reported_seq":14,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.098616+0000","last_change":"2023-10-16T13:46:37.650949+0000","last_active":"2023-10-16T13:46:40.098616+0000","last_peered":"2023-10-16T13:46:40.098616+0000","last_clean":"2023-10-16T13:46:40.098616+0000","last_became_active":"2023-10-16T13:46:37.650684+0000","last_became_peered":"2023-10-16T13:46:37.650684+0000","last_unstale":"2023-10-16T13:46:40.098616+0000","last_undegraded":"2023-10-16T13:46:40.098616+0000","last_fullsized":"2023-10-16T13:46:40.098616+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_clean_scrub_stamp":"2023-10-16T13:46:36.617142+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:58:21.317117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.653690+0000","last_change":"2023-10-16T13:46:38.661197+0000","last_active":"2023-10-16T13:46:39.653690+0000","last_peered":"2023-10-16T13:46:39.653690+0000","last_clean":"2023-10-16T13:46:39.653690+0000","last_became_active":"2023-10-16T13:46:38.660903+0000","last_became_peered":"2023-10-16T13:46:38.660903+0000","last_unstale":"2023-10-16T13:46:39.653690+0000","last_undegraded":"2023-10-16T13:46:39.653690+0000","last_fullsized":"2023-10-16T13:46:39.653690+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:47:08.704282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.647136+0000","last_change":"2023-10-16T13:46:24.340328+0000","last_active":"2023-10-16T13:46:38.647136+0000","last_peered":"2023-10-16T13:46:38.647136+0000","last_clean":"2023-10-16T13:46:38.647136+0000","last_became_active":"2023-10-16T13:46:24.340042+0000","last_became_peered":"2023-10-16T13:46:24.340042+0000","last_unstale":"2023-10-16T13:46:38.647136+0000","last_undegraded":"2023-10-16T13:46:38.647136+0000","last_fullsized":"2023-10-16T13:46:38.647136+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:15:37.760742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.659388+0000","last_change":"2023-10-16T13:46:25.575797+0000","last_active":"2023-10-16T13:46:38.659388+0000","last_peered":"2023-10-16T13:46:38.659388+0000","last_clean":"2023-10-16T13:46:38.659388+0000","last_became_active":"2023-10-16T13:46:25.575494+0000","last_became_peered":"2023-10-16T13:46:25.575494+0000","last_unstale":"2023-10-16T13:46:38.659388+0000","last_undegraded":"2023-10-16T13:46:38.659388+0000","last_fullsized":"2023-10-16T13:46:38.659388+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:11:22.607258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.4","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.098577+0000","last_change":"2023-10-16T13:46:37.647232+0000","last_active":"2023-10-16T13:46:40.098577+0000","last_peered":"2023-10-16T13:46:40.098577+0000","last_clean":"2023-10-16T13:46:40.098577+0000","last_became_active":"2023-10-16T13:46:37.646761+0000","last_became_peered":"2023-10-16T13:46:37.646761+0000","last_unstale":"2023-10-16T13:46:40.098577+0000","last_undegraded":"2023-10-16T13:46:40.098577+0000","last_fullsized":"2023-10-16T13:46:40.098577+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_clean_scrub_stamp":"2023-10-16T13:46:36.617142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:46:37.318465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.7","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.867313+0000","last_change":"2023-10-16T13:46:38.662825+0000","last_active":"2023-10-16T13:46:40.867313+0000","last_peered":"2023-10-16T13:46:40.867313+0000","last_clean":"2023-10-16T13:46:40.867313+0000","last_became_active":"2023-10-16T13:46:38.662454+0000","last_became_peered":"2023-10-16T13:46:38.662454+0000","last_unstale":"2023-10-16T13:46:40.867313+0000","last_undegraded":"2023-10-16T13:46:40.867313+0000","last_fullsized":"2023-10-16T13:46:40.867313+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:09:09.585691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.646549+0000","last_change":"2023-10-16T13:46:24.338016+0000","last_active":"2023-10-16T13:46:38.646549+0000","last_peered":"2023-10-16T13:46:38.646549+0000","last_clean":"2023-10-16T13:46:38.646549+0000","last_became_active":"2023-10-16T13:46:24.337687+0000","last_became_peered":"2023-10-16T13:46:24.337687+0000","last_unstale":"2023-10-16T13:46:38.646549+0000","last_undegraded":"2023-10-16T13:46:38.646549+0000","last_fullsized":"2023-10-16T13:46:38.646549+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:27:57.477477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.649679+0000","last_change":"2023-10-16T13:46:25.576269+0000","last_active":"2023-10-16T13:46:38.649679+0000","last_peered":"2023-10-16T13:46:38.649679+0000","last_clean":"2023-10-16T13:46:38.649679+0000","last_became_active":"2023-10-16T13:46:25.575732+0000","last_became_peered":"2023-10-16T13:46:25.575732+0000","last_unstale":"2023-10-16T13:46:38.649679+0000","last_undegraded":"2023-10-16T13:46:38.649679+0000","last_fullsized":"2023-10-16T13:46:38.649679+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:36:08.745864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.5","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.671360+0000","last_change":"2023-10-16T13:46:37.638258+0000","last_active":"2023-10-16T13:46:39.671360+0000","last_peered":"2023-10-16T13:46:39.671360+0000","last_clean":"2023-10-16T13:46:39.671360+0000","last_became_active":"2023-10-16T13:46:37.637824+0000","last_became_peered":"2023-10-16T13:46:37.637824+0000","last_unstale":"2023-10-16T13:46:39.671360+0000","last_undegraded":"2023-10-16T13:46:39.671360+0000","last_fullsized":"2023-10-16T13:46:39.671360+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_clean_scrub_stamp":"2023-10-16T13:46:36.617142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:45:43.077455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.6","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.930230+0000","last_change":"2023-10-16T13:46:38.664836+0000","last_active":"2023-10-16T13:46:39.930230+0000","last_peered":"2023-10-16T13:46:39.930230+0000","last_clean":"2023-10-16T13:46:39.930230+0000","last_became_active":"2023-10-16T13:46:38.664267+0000","last_became_peered":"2023-10-16T13:46:38.664267+0000","last_unstale":"2023-10-16T13:46:39.930230+0000","last_undegraded":"2023-10-16T13:46:39.930230+0000","last_fullsized":"2023-10-16T13:46:39.930230+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:38:51.424578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.649873+0000","last_change":"2023-10-16T13:46:24.333052+0000","last_active":"2023-10-16T13:46:38.649873+0000","last_peered":"2023-10-16T13:46:38.649873+0000","last_clean":"2023-10-16T13:46:38.649873+0000","last_became_active":"2023-10-16T13:46:24.332751+0000","last_became_peered":"2023-10-16T13:46:24.332751+0000","last_unstale":"2023-10-16T13:46:38.649873+0000","last_undegraded":"2023-10-16T13:46:38.649873+0000","last_fullsized":"2023-10-16T13:46:38.649873+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:29:38.839702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.651547+0000","last_change":"2023-10-16T13:46:25.580242+0000","last_active":"2023-10-16T13:46:38.651547+0000","last_peered":"2023-10-16T13:46:38.651547+0000","last_clean":"2023-10-16T13:46:38.651547+0000","last_became_active":"2023-10-16T13:46:25.579959+0000","last_became_peered":"2023-10-16T13:46:25.579959+0000","last_unstale":"2023-10-16T13:46:38.651547+0000","last_undegraded":"2023-10-16T13:46:38.651547+0000","last_fullsized":"2023-10-16T13:46:38.651547+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:52:21.431610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.2","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.673033+0000","last_change":"2023-10-16T13:46:37.634801+0000","last_active":"2023-10-16T13:46:39.673033+0000","last_peered":"2023-10-16T13:46:39.673033+0000","last_clean":"2023-10-16T13:46:39.673033+0000","last_became_active":"2023-10-16T13:46:37.634481+0000","last_became_peered":"2023-10-16T13:46:37.634481+0000","last_unstale":"2023-10-16T13:46:39.673033+0000","last_undegraded":"2023-10-16T13:46:39.673033+0000","last_fullsized":"2023-10-16T13:46:39.673033+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_clean_scrub_stamp":"2023-10-16T13:46:36.617142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:10:24.150487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.580964+0000","last_change":"2023-10-16T13:46:38.661425+0000","last_active":"2023-10-16T13:46:40.580964+0000","last_peered":"2023-10-16T13:46:40.580964+0000","last_clean":"2023-10-16T13:46:40.580964+0000","last_became_active":"2023-10-16T13:46:38.660952+0000","last_became_peered":"2023-10-16T13:46:38.660952+0000","last_unstale":"2023-10-16T13:46:40.580964+0000","last_undegraded":"2023-10-16T13:46:40.580964+0000","last_fullsized":"2023-10-16T13:46:40.580964+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:08:49.765693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.643091+0000","last_change":"2023-10-16T13:46:24.340274+0000","last_active":"2023-10-16T13:46:38.643091+0000","last_peered":"2023-10-16T13:46:38.643091+0000","last_clean":"2023-10-16T13:46:38.643091+0000","last_became_active":"2023-10-16T13:46:24.339988+0000","last_became_peered":"2023-10-16T13:46:24.339988+0000","last_unstale":"2023-10-16T13:46:38.643091+0000","last_undegraded":"2023-10-16T13:46:38.643091+0000","last_fullsized":"2023-10-16T13:46:38.643091+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:48:44.887565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.646919+0000","last_change":"2023-10-16T13:46:25.567174+0000","last_active":"2023-10-16T13:46:38.646919+0000","last_peered":"2023-10-16T13:46:38.646919+0000","last_clean":"2023-10-16T13:46:38.646919+0000","last_became_active":"2023-10-16T13:46:25.566392+0000","last_became_peered":"2023-10-16T13:46:25.566392+0000","last_unstale":"2023-10-16T13:46:38.646919+0000","last_undegraded":"2023-10-16T13:46:38.646919+0000","last_fullsized":"2023-10-16T13:46:38.646919+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:16:23.997738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.3","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.098057+0000","last_change":"2023-10-16T13:46:37.647560+0000","last_active":"2023-10-16T13:46:40.098057+0000","last_peered":"2023-10-16T13:46:40.098057+0000","last_clean":"2023-10-16T13:46:40.098057+0000","last_became_active":"2023-10-16T13:46:37.646929+0000","last_became_peered":"2023-10-16T13:46:37.646929+0000","last_unstale":"2023-10-16T13:46:40.098057+0000","last_undegraded":"2023-10-16T13:46:40.098057+0000","last_fullsized":"2023-10-16T13:46:40.098057+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_clean_scrub_stamp":"2023-10-16T13:46:36.617142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:41:05.573188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.580688+0000","last_change":"2023-10-16T13:46:38.673556+0000","last_active":"2023-10-16T13:46:40.580688+0000","last_peered":"2023-10-16T13:46:40.580688+0000","last_clean":"2023-10-16T13:46:40.580688+0000","last_became_active":"2023-10-16T13:46:38.673246+0000","last_became_peered":"2023-10-16T13:46:38.673246+0000","last_unstale":"2023-10-16T13:46:40.580688+0000","last_undegraded":"2023-10-16T13:46:40.580688+0000","last_fullsized":"2023-10-16T13:46:40.580688+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:08:18.683720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.6","version":"225'1","reported_seq":17,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.647962+0000","last_change":"2023-10-16T13:46:24.346027+0000","last_active":"2023-10-16T13:46:38.647962+0000","last_peered":"2023-10-16T13:46:38.647962+0000","last_clean":"2023-10-16T13:46:38.647962+0000","last_became_active":"2023-10-16T13:46:24.345748+0000","last_became_peered":"2023-10-16T13:46:24.345748+0000","last_unstale":"2023-10-16T13:46:38.647962+0000","last_undegraded":"2023-10-16T13:46:38.647962+0000","last_fullsized":"2023-10-16T13:46:38.647962+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:23:29.526203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.651456+0000","last_change":"2023-10-16T13:46:25.577247+0000","last_active":"2023-10-16T13:46:38.651456+0000","last_peered":"2023-10-16T13:46:38.651456+0000","last_clean":"2023-10-16T13:46:38.651456+0000","last_became_active":"2023-10-16T13:46:25.576949+0000","last_became_peered":"2023-10-16T13:46:25.576949+0000","last_unstale":"2023-10-16T13:46:38.651456+0000","last_undegraded":"2023-10-16T13:46:38.651456+0000","last_fullsized":"2023-10-16T13:46:38.651456+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:35:32.750171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.0","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.097475+0000","last_change":"2023-10-16T13:46:37.647928+0000","last_active":"2023-10-16T13:46:40.097475+0000","last_peered":"2023-10-16T13:46:40.097475+0000","last_clean":"2023-10-16T13:46:40.097475+0000","last_became_active":"2023-10-16T13:46:37.647416+0000","last_became_peered":"2023-10-16T13:46:37.647416+0000","last_unstale":"2023-10-16T13:46:40.097475+0000","last_undegraded":"2023-10-16T13:46:40.097475+0000","last_fullsized":"2023-10-16T13:46:40.097475+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_clean_scrub_stamp":"2023-10-16T13:46:36.617142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:24:07.201333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.097848+0000","last_change":"2023-10-16T13:46:38.669803+0000","last_active":"2023-10-16T13:46:40.097848+0000","last_peered":"2023-10-16T13:46:40.097848+0000","last_clean":"2023-10-16T13:46:40.097848+0000","last_became_active":"2023-10-16T13:46:38.661903+0000","last_became_peered":"2023-10-16T13:46:38.661903+0000","last_unstale":"2023-10-16T13:46:40.097848+0000","last_undegraded":"2023-10-16T13:46:40.097848+0000","last_fullsized":"2023-10-16T13:46:40.097848+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:24:46.861701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.7","version":"224'2","reported_seq":18,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.656794+0000","last_change":"2023-10-16T13:46:24.344410+0000","last_active":"2023-10-16T13:46:38.656794+0000","last_peered":"2023-10-16T13:46:38.656794+0000","last_clean":"2023-10-16T13:46:38.656794+0000","last_became_active":"2023-10-16T13:46:24.343665+0000","last_became_peered":"2023-10-16T13:46:24.343665+0000","last_unstale":"2023-10-16T13:46:38.656794+0000","last_undegraded":"2023-10-16T13:46:38.656794+0000","last_fullsized":"2023-10-16T13:46:38.656794+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:21:31.259195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.647642+0000","last_change":"2023-10-16T13:46:25.585631+0000","last_active":"2023-10-16T13:46:38.647642+0000","last_peered":"2023-10-16T13:46:38.647642+0000","last_clean":"2023-10-16T13:46:38.647642+0000","last_became_active":"2023-10-16T13:46:25.585329+0000","last_became_peered":"2023-10-16T13:46:25.585329+0000","last_unstale":"2023-10-16T13:46:38.647642+0000","last_undegraded":"2023-10-16T13:46:38.647642+0000","last_fullsized":"2023-10-16T13:46:38.647642+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:37:50.932814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.654405+0000","last_change":"2023-10-16T13:46:37.646177+0000","last_active":"2023-10-16T13:46:39.654405+0000","last_peered":"2023-10-16T13:46:39.654405+0000","last_clean":"2023-10-16T13:46:39.654405+0000","last_became_active":"2023-10-16T13:46:37.645682+0000","last_became_peered":"2023-10-16T13:46:37.645682+0000","last_unstale":"2023-10-16T13:46:39.654405+0000","last_undegraded":"2023-10-16T13:46:39.654405+0000","last_fullsized":"2023-10-16T13:46:39.654405+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_clean_scrub_stamp":"2023-10-16T13:46:36.617142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:14:57.521447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.866695+0000","last_change":"2023-10-16T13:46:38.644591+0000","last_active":"2023-10-16T13:46:40.866695+0000","last_peered":"2023-10-16T13:46:40.866695+0000","last_clean":"2023-10-16T13:46:40.866695+0000","last_became_active":"2023-10-16T13:46:38.644285+0000","last_became_peered":"2023-10-16T13:46:38.644285+0000","last_unstale":"2023-10-16T13:46:40.866695+0000","last_undegraded":"2023-10-16T13:46:40.866695+0000","last_fullsized":"2023-10-16T13:46:40.866695+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:41:21.402877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.649147+0000","last_change":"2023-10-16T13:46:24.341910+0000","last_active":"2023-10-16T13:46:38.649147+0000","last_peered":"2023-10-16T13:46:38.649147+0000","last_clean":"2023-10-16T13:46:38.649147+0000","last_became_active":"2023-10-16T13:46:24.341633+0000","last_became_peered":"2023-10-16T13:46:24.341633+0000","last_unstale":"2023-10-16T13:46:38.649147+0000","last_undegraded":"2023-10-16T13:46:38.649147+0000","last_fullsized":"2023-10-16T13:46:38.649147+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:00:55.319505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.648996+0000","last_change":"2023-10-16T13:46:25.576886+0000","last_active":"2023-10-16T13:46:38.648996+0000","last_peered":"2023-10-16T13:46:38.648996+0000","last_clean":"2023-10-16T13:46:38.648996+0000","last_became_active":"2023-10-16T13:46:25.576540+0000","last_became_peered":"2023-10-16T13:46:25.576540+0000","last_unstale":"2023-10-16T13:46:38.648996+0000","last_undegraded":"2023-10-16T13:46:38.648996+0000","last_fullsized":"2023-10-16T13:46:38.648996+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:00:43.545612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.e","version":"228'2","reported_seq":14,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.867130+0000","last_change":"2023-10-16T13:46:37.642438+0000","last_active":"2023-10-16T13:46:40.867130+0000","last_peered":"2023-10-16T13:46:40.867130+0000","last_clean":"2023-10-16T13:46:40.867130+0000","last_became_active":"2023-10-16T13:46:37.642152+0000","last_became_peered":"2023-10-16T13:46:37.642152+0000","last_unstale":"2023-10-16T13:46:40.867130+0000","last_undegraded":"2023-10-16T13:46:40.867130+0000","last_fullsized":"2023-10-16T13:46:40.867130+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_clean_scrub_stamp":"2023-10-16T13:46:36.617142+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:48:58.187956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.d","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.654809+0000","last_change":"2023-10-16T13:46:38.660005+0000","last_active":"2023-10-16T13:46:39.654809+0000","last_peered":"2023-10-16T13:46:39.654809+0000","last_clean":"2023-10-16T13:46:39.654809+0000","last_became_active":"2023-10-16T13:46:38.659645+0000","last_became_peered":"2023-10-16T13:46:38.659645+0000","last_unstale":"2023-10-16T13:46:39.654809+0000","last_undegraded":"2023-10-16T13:46:39.654809+0000","last_fullsized":"2023-10-16T13:46:39.654809+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:49:06.438194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.641824+0000","last_change":"2023-10-16T13:46:24.344523+0000","last_active":"2023-10-16T13:46:38.641824+0000","last_peered":"2023-10-16T13:46:38.641824+0000","last_clean":"2023-10-16T13:46:38.641824+0000","last_became_active":"2023-10-16T13:46:24.344238+0000","last_became_peered":"2023-10-16T13:46:24.344238+0000","last_unstale":"2023-10-16T13:46:38.641824+0000","last_undegraded":"2023-10-16T13:46:38.641824+0000","last_fullsized":"2023-10-16T13:46:38.641824+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:37:49.931017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.648399+0000","last_change":"2023-10-16T13:46:25.570388+0000","last_active":"2023-10-16T13:46:38.648399+0000","last_peered":"2023-10-16T13:46:38.648399+0000","last_clean":"2023-10-16T13:46:38.648399+0000","last_became_active":"2023-10-16T13:46:25.570059+0000","last_became_peered":"2023-10-16T13:46:25.570059+0000","last_unstale":"2023-10-16T13:46:38.648399+0000","last_undegraded":"2023-10-16T13:46:38.648399+0000","last_fullsized":"2023-10-16T13:46:38.648399+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:11:22.607258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.f","version":"228'1","reported_seq":13,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.581684+0000","last_change":"2023-10-16T13:46:37.648761+0000","last_active":"2023-10-16T13:46:40.581684+0000","last_peered":"2023-10-16T13:46:40.581684+0000","last_clean":"2023-10-16T13:46:40.581684+0000","last_became_active":"2023-10-16T13:46:37.648542+0000","last_became_peered":"2023-10-16T13:46:37.648542+0000","last_unstale":"2023-10-16T13:46:40.581684+0000","last_undegraded":"2023-10-16T13:46:40.581684+0000","last_fullsized":"2023-10-16T13:46:40.581684+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_clean_scrub_stamp":"2023-10-16T13:46:36.617142+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:10:10.956373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.653711+0000","last_change":"2023-10-16T13:46:38.674462+0000","last_active":"2023-10-16T13:46:39.653711+0000","last_peered":"2023-10-16T13:46:39.653711+0000","last_clean":"2023-10-16T13:46:39.653711+0000","last_became_active":"2023-10-16T13:46:38.674196+0000","last_became_peered":"2023-10-16T13:46:38.674196+0000","last_unstale":"2023-10-16T13:46:39.653711+0000","last_undegraded":"2023-10-16T13:46:39.653711+0000","last_fullsized":"2023-10-16T13:46:39.653711+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:14:58.526615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.657803+0000","last_change":"2023-10-16T13:46:24.337053+0000","last_active":"2023-10-16T13:46:38.657803+0000","last_peered":"2023-10-16T13:46:38.657803+0000","last_clean":"2023-10-16T13:46:38.657803+0000","last_became_active":"2023-10-16T13:46:24.336736+0000","last_became_peered":"2023-10-16T13:46:24.336736+0000","last_unstale":"2023-10-16T13:46:38.657803+0000","last_undegraded":"2023-10-16T13:46:38.657803+0000","last_fullsized":"2023-10-16T13:46:38.657803+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:36:24.043336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.647324+0000","last_change":"2023-10-16T13:46:25.580894+0000","last_active":"2023-10-16T13:46:38.647324+0000","last_peered":"2023-10-16T13:46:38.647324+0000","last_clean":"2023-10-16T13:46:38.647324+0000","last_became_active":"2023-10-16T13:46:25.580589+0000","last_became_peered":"2023-10-16T13:46:25.580589+0000","last_unstale":"2023-10-16T13:46:38.647324+0000","last_undegraded":"2023-10-16T13:46:38.647324+0000","last_fullsized":"2023-10-16T13:46:38.647324+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:22:31.745564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.c","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.098212+0000","last_change":"2023-10-16T13:46:37.653174+0000","last_active":"2023-10-16T13:46:40.098212+0000","last_peered":"2023-10-16T13:46:40.098212+0000","last_clean":"2023-10-16T13:46:40.098212+0000","last_became_active":"2023-10-16T13:46:37.652887+0000","last_became_peered":"2023-10-16T13:46:37.652887+0000","last_unstale":"2023-10-16T13:46:40.098212+0000","last_undegraded":"2023-10-16T13:46:40.098212+0000","last_fullsized":"2023-10-16T13:46:40.098212+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_clean_scrub_stamp":"2023-10-16T13:46:36.617142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:17:36.285215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.f","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.098986+0000","last_change":"2023-10-16T13:46:38.662558+0000","last_active":"2023-10-16T13:46:40.098986+0000","last_peered":"2023-10-16T13:46:40.098986+0000","last_clean":"2023-10-16T13:46:40.098986+0000","last_became_active":"2023-10-16T13:46:38.661847+0000","last_became_peered":"2023-10-16T13:46:38.661847+0000","last_unstale":"2023-10-16T13:46:40.098986+0000","last_undegraded":"2023-10-16T13:46:40.098986+0000","last_fullsized":"2023-10-16T13:46:40.098986+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:25:18.872239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.b","version":"224'2","reported_seq":18,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.640887+0000","last_change":"2023-10-16T13:46:24.347651+0000","last_active":"2023-10-16T13:46:38.640887+0000","last_peered":"2023-10-16T13:46:38.640887+0000","last_clean":"2023-10-16T13:46:38.640887+0000","last_became_active":"2023-10-16T13:46:24.347370+0000","last_became_peered":"2023-10-16T13:46:24.347370+0000","last_unstale":"2023-10-16T13:46:38.640887+0000","last_undegraded":"2023-10-16T13:46:38.640887+0000","last_fullsized":"2023-10-16T13:46:38.640887+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:18:32.988431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.654660+0000","last_change":"2023-10-16T13:46:25.572423+0000","last_active":"2023-10-16T13:46:38.654660+0000","last_peered":"2023-10-16T13:46:38.654660+0000","last_clean":"2023-10-16T13:46:38.654660+0000","last_became_active":"2023-10-16T13:46:25.569272+0000","last_became_peered":"2023-10-16T13:46:25.569272+0000","last_unstale":"2023-10-16T13:46:38.654660+0000","last_undegraded":"2023-10-16T13:46:38.654660+0000","last_fullsized":"2023-10-16T13:46:38.654660+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:14:45.222853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.d","version":"228'1","reported_seq":13,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.098201+0000","last_change":"2023-10-16T13:46:37.648875+0000","last_active":"2023-10-16T13:46:40.098201+0000","last_peered":"2023-10-16T13:46:40.098201+0000","last_clean":"2023-10-16T13:46:40.098201+0000","last_became_active":"2023-10-16T13:46:37.648583+0000","last_became_peered":"2023-10-16T13:46:37.648583+0000","last_unstale":"2023-10-16T13:46:40.098201+0000","last_undegraded":"2023-10-16T13:46:40.098201+0000","last_fullsized":"2023-10-16T13:46:40.098201+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_clean_scrub_stamp":"2023-10-16T13:46:36.617142+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:38:26.428397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.e","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.581775+0000","last_change":"2023-10-16T13:46:38.661756+0000","last_active":"2023-10-16T13:46:40.581775+0000","last_peered":"2023-10-16T13:46:40.581775+0000","last_clean":"2023-10-16T13:46:40.581775+0000","last_became_active":"2023-10-16T13:46:38.661029+0000","last_became_peered":"2023-10-16T13:46:38.661029+0000","last_unstale":"2023-10-16T13:46:40.581775+0000","last_undegraded":"2023-10-16T13:46:40.581775+0000","last_fullsized":"2023-10-16T13:46:40.581775+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:18:52.510533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.641078+0000","last_change":"2023-10-16T13:46:24.339475+0000","last_active":"2023-10-16T13:46:38.641078+0000","last_peered":"2023-10-16T13:46:38.641078+0000","last_clean":"2023-10-16T13:46:38.641078+0000","last_became_active":"2023-10-16T13:46:24.339105+0000","last_became_peered":"2023-10-16T13:46:24.339105+0000","last_unstale":"2023-10-16T13:46:38.641078+0000","last_undegraded":"2023-10-16T13:46:38.641078+0000","last_fullsized":"2023-10-16T13:46:38.641078+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:13:27.085544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.646755+0000","last_change":"2023-10-16T13:46:25.579683+0000","last_active":"2023-10-16T13:46:38.646755+0000","last_peered":"2023-10-16T13:46:38.646755+0000","last_clean":"2023-10-16T13:46:38.646755+0000","last_became_active":"2023-10-16T13:46:25.579398+0000","last_became_peered":"2023-10-16T13:46:25.579398+0000","last_unstale":"2023-10-16T13:46:38.646755+0000","last_undegraded":"2023-10-16T13:46:38.646755+0000","last_fullsized":"2023-10-16T13:46:38.646755+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:30:09.665561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.a","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.670691+0000","last_change":"2023-10-16T13:46:37.653618+0000","last_active":"2023-10-16T13:46:39.670691+0000","last_peered":"2023-10-16T13:46:39.670691+0000","last_clean":"2023-10-16T13:46:39.670691+0000","last_became_active":"2023-10-16T13:46:37.653296+0000","last_became_peered":"2023-10-16T13:46:37.653296+0000","last_unstale":"2023-10-16T13:46:39.670691+0000","last_undegraded":"2023-10-16T13:46:39.670691+0000","last_fullsized":"2023-10-16T13:46:39.670691+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_clean_scrub_stamp":"2023-10-16T13:46:36.617142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:45:22.549136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.654700+0000","last_change":"2023-10-16T13:46:38.665894+0000","last_active":"2023-10-16T13:46:39.654700+0000","last_peered":"2023-10-16T13:46:39.654700+0000","last_clean":"2023-10-16T13:46:39.654700+0000","last_became_active":"2023-10-16T13:46:38.665612+0000","last_became_peered":"2023-10-16T13:46:38.665612+0000","last_unstale":"2023-10-16T13:46:39.654700+0000","last_undegraded":"2023-10-16T13:46:39.654700+0000","last_fullsized":"2023-10-16T13:46:39.654700+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:44:17.896114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.d","version":"224'1","reported_seq":17,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.658880+0000","last_change":"2023-10-16T13:46:24.338316+0000","last_active":"2023-10-16T13:46:38.658880+0000","last_peered":"2023-10-16T13:46:38.658880+0000","last_clean":"2023-10-16T13:46:38.658880+0000","last_became_active":"2023-10-16T13:46:24.338002+0000","last_became_peered":"2023-10-16T13:46:24.338002+0000","last_unstale":"2023-10-16T13:46:38.658880+0000","last_undegraded":"2023-10-16T13:46:38.658880+0000","last_fullsized":"2023-10-16T13:46:38.658880+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:04:35.398533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.653092+0000","last_change":"2023-10-16T13:46:25.582651+0000","last_active":"2023-10-16T13:46:38.653092+0000","last_peered":"2023-10-16T13:46:38.653092+0000","last_clean":"2023-10-16T13:46:38.653092+0000","last_became_active":"2023-10-16T13:46:25.582350+0000","last_became_peered":"2023-10-16T13:46:25.582350+0000","last_unstale":"2023-10-16T13:46:38.653092+0000","last_undegraded":"2023-10-16T13:46:38.653092+0000","last_fullsized":"2023-10-16T13:46:38.653092+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:25:08.151545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.b","version":"228'2","reported_seq":14,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.098382+0000","last_change":"2023-10-16T13:46:37.647961+0000","last_active":"2023-10-16T13:46:40.098382+0000","last_peered":"2023-10-16T13:46:40.098382+0000","last_clean":"2023-10-16T13:46:40.098382+0000","last_became_active":"2023-10-16T13:46:37.647472+0000","last_became_peered":"2023-10-16T13:46:37.647472+0000","last_unstale":"2023-10-16T13:46:40.098382+0000","last_undegraded":"2023-10-16T13:46:40.098382+0000","last_fullsized":"2023-10-16T13:46:40.098382+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_clean_scrub_stamp":"2023-10-16T13:46:36.617142+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:43:36.812667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.097345+0000","last_change":"2023-10-16T13:46:38.662342+0000","last_active":"2023-10-16T13:46:40.097345+0000","last_peered":"2023-10-16T13:46:40.097345+0000","last_clean":"2023-10-16T13:46:40.097345+0000","last_became_active":"2023-10-16T13:46:38.662039+0000","last_became_peered":"2023-10-16T13:46:38.662039+0000","last_unstale":"2023-10-16T13:46:40.097345+0000","last_undegraded":"2023-10-16T13:46:40.097345+0000","last_fullsized":"2023-10-16T13:46:40.097345+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:31:01.184857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.e","version":"224'2","reported_seq":18,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.650881+0000","last_change":"2023-10-16T13:46:24.339351+0000","last_active":"2023-10-16T13:46:38.650881+0000","last_peered":"2023-10-16T13:46:38.650881+0000","last_clean":"2023-10-16T13:46:38.650881+0000","last_became_active":"2023-10-16T13:46:24.339062+0000","last_became_peered":"2023-10-16T13:46:24.339062+0000","last_unstale":"2023-10-16T13:46:38.650881+0000","last_undegraded":"2023-10-16T13:46:38.650881+0000","last_fullsized":"2023-10-16T13:46:38.650881+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:14:49.845621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.649074+0000","last_change":"2023-10-16T13:46:25.578233+0000","last_active":"2023-10-16T13:46:38.649074+0000","last_peered":"2023-10-16T13:46:38.649074+0000","last_clean":"2023-10-16T13:46:38.649074+0000","last_became_active":"2023-10-16T13:46:25.577908+0000","last_became_peered":"2023-10-16T13:46:25.577908+0000","last_unstale":"2023-10-16T13:46:38.649074+0000","last_undegraded":"2023-10-16T13:46:38.649074+0000","last_fullsized":"2023-10-16T13:46:38.649074+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:15:34.577378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.8","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.866463+0000","last_change":"2023-10-16T13:46:37.644185+0000","last_active":"2023-10-16T13:46:40.866463+0000","last_peered":"2023-10-16T13:46:40.866463+0000","last_clean":"2023-10-16T13:46:40.866463+0000","last_became_active":"2023-10-16T13:46:37.643958+0000","last_became_peered":"2023-10-16T13:46:37.643958+0000","last_unstale":"2023-10-16T13:46:40.866463+0000","last_undegraded":"2023-10-16T13:46:40.866463+0000","last_fullsized":"2023-10-16T13:46:40.866463+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_clean_scrub_stamp":"2023-10-16T13:46:36.617142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:11:00.815968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.b","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.581061+0000","last_change":"2023-10-16T13:46:38.673958+0000","last_active":"2023-10-16T13:46:40.581061+0000","last_peered":"2023-10-16T13:46:40.581061+0000","last_clean":"2023-10-16T13:46:40.581061+0000","last_became_active":"2023-10-16T13:46:38.673696+0000","last_became_peered":"2023-10-16T13:46:38.673696+0000","last_unstale":"2023-10-16T13:46:40.581061+0000","last_undegraded":"2023-10-16T13:46:40.581061+0000","last_fullsized":"2023-10-16T13:46:40.581061+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:47:10.439209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.f","version":"224'1","reported_seq":17,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.647273+0000","last_change":"2023-10-16T13:46:24.338988+0000","last_active":"2023-10-16T13:46:38.647273+0000","last_peered":"2023-10-16T13:46:38.647273+0000","last_clean":"2023-10-16T13:46:38.647273+0000","last_became_active":"2023-10-16T13:46:24.338477+0000","last_became_peered":"2023-10-16T13:46:24.338477+0000","last_unstale":"2023-10-16T13:46:38.647273+0000","last_undegraded":"2023-10-16T13:46:38.647273+0000","last_fullsized":"2023-10-16T13:46:38.647273+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:47:21.411665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.647426+0000","last_change":"2023-10-16T13:46:25.585651+0000","last_active":"2023-10-16T13:46:38.647426+0000","last_peered":"2023-10-16T13:46:38.647426+0000","last_clean":"2023-10-16T13:46:38.647426+0000","last_became_active":"2023-10-16T13:46:25.585367+0000","last_became_peered":"2023-10-16T13:46:25.585367+0000","last_unstale":"2023-10-16T13:46:38.647426+0000","last_undegraded":"2023-10-16T13:46:38.647426+0000","last_fullsized":"2023-10-16T13:46:38.647426+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:19:04.312392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.9","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.670556+0000","last_change":"2023-10-16T13:46:37.639229+0000","last_active":"2023-10-16T13:46:39.670556+0000","last_peered":"2023-10-16T13:46:39.670556+0000","last_clean":"2023-10-16T13:46:39.670556+0000","last_became_active":"2023-10-16T13:46:37.638965+0000","last_became_peered":"2023-10-16T13:46:37.638965+0000","last_unstale":"2023-10-16T13:46:39.670556+0000","last_undegraded":"2023-10-16T13:46:39.670556+0000","last_fullsized":"2023-10-16T13:46:39.670556+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_clean_scrub_stamp":"2023-10-16T13:46:36.617142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:00:52.195970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.653547+0000","last_change":"2023-10-16T13:46:38.674113+0000","last_active":"2023-10-16T13:46:39.653547+0000","last_peered":"2023-10-16T13:46:39.653547+0000","last_clean":"2023-10-16T13:46:39.653547+0000","last_became_active":"2023-10-16T13:46:38.673756+0000","last_became_peered":"2023-10-16T13:46:38.673756+0000","last_unstale":"2023-10-16T13:46:39.653547+0000","last_undegraded":"2023-10-16T13:46:39.653547+0000","last_fullsized":"2023-10-16T13:46:39.653547+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:25:58.105841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.10","version":"224'8","reported_seq":24,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.640658+0000","last_change":"2023-10-16T13:46:24.346086+0000","last_active":"2023-10-16T13:46:38.640658+0000","last_peered":"2023-10-16T13:46:38.640658+0000","last_clean":"2023-10-16T13:46:38.640658+0000","last_became_active":"2023-10-16T13:46:24.345786+0000","last_became_peered":"2023-10-16T13:46:24.345786+0000","last_unstale":"2023-10-16T13:46:38.640658+0000","last_undegraded":"2023-10-16T13:46:38.640658+0000","last_fullsized":"2023-10-16T13:46:38.640658+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:39:19.890170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.647346+0000","last_change":"2023-10-16T13:46:25.584003+0000","last_active":"2023-10-16T13:46:38.647346+0000","last_peered":"2023-10-16T13:46:38.647346+0000","last_clean":"2023-10-16T13:46:38.647346+0000","last_became_active":"2023-10-16T13:46:25.583732+0000","last_became_peered":"2023-10-16T13:46:25.583732+0000","last_unstale":"2023-10-16T13:46:38.647346+0000","last_undegraded":"2023-10-16T13:46:38.647346+0000","last_fullsized":"2023-10-16T13:46:38.647346+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:08:20.004685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.16","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.673697+0000","last_change":"2023-10-16T13:46:37.649721+0000","last_active":"2023-10-16T13:46:39.673697+0000","last_peered":"2023-10-16T13:46:39.673697+0000","last_clean":"2023-10-16T13:46:39.673697+0000","last_became_active":"2023-10-16T13:46:37.649410+0000","last_became_peered":"2023-10-16T13:46:37.649410+0000","last_unstale":"2023-10-16T13:46:39.673697+0000","last_undegraded":"2023-10-16T13:46:39.673697+0000","last_fullsized":"2023-10-16T13:46:39.673697+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_clean_scrub_stamp":"2023-10-16T13:46:36.617142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:35:04.333865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.15","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.671014+0000","last_change":"2023-10-16T13:46:38.653350+0000","last_active":"2023-10-16T13:46:39.671014+0000","last_peered":"2023-10-16T13:46:39.671014+0000","last_clean":"2023-10-16T13:46:39.671014+0000","last_became_active":"2023-10-16T13:46:38.652538+0000","last_became_peered":"2023-10-16T13:46:38.652538+0000","last_unstale":"2023-10-16T13:46:39.671014+0000","last_undegraded":"2023-10-16T13:46:39.671014+0000","last_fullsized":"2023-10-16T13:46:39.671014+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:56:11.960239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.646167+0000","last_change":"2023-10-16T13:46:24.332386+0000","last_active":"2023-10-16T13:46:38.646167+0000","last_peered":"2023-10-16T13:46:38.646167+0000","last_clean":"2023-10-16T13:46:38.646167+0000","last_became_active":"2023-10-16T13:46:24.332013+0000","last_became_peered":"2023-10-16T13:46:24.332013+0000","last_unstale":"2023-10-16T13:46:38.646167+0000","last_undegraded":"2023-10-16T13:46:38.646167+0000","last_fullsized":"2023-10-16T13:46:38.646167+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:04:46.312175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.648293+0000","last_change":"2023-10-16T13:46:25.573821+0000","last_active":"2023-10-16T13:46:38.648293+0000","last_peered":"2023-10-16T13:46:38.648293+0000","last_clean":"2023-10-16T13:46:38.648293+0000","last_became_active":"2023-10-16T13:46:25.572882+0000","last_became_peered":"2023-10-16T13:46:25.572882+0000","last_unstale":"2023-10-16T13:46:38.648293+0000","last_undegraded":"2023-10-16T13:46:38.648293+0000","last_fullsized":"2023-10-16T13:46:38.648293+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:57:13.048327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.17","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.655504+0000","last_change":"2023-10-16T13:46:37.648194+0000","last_active":"2023-10-16T13:46:39.655504+0000","last_peered":"2023-10-16T13:46:39.655504+0000","last_clean":"2023-10-16T13:46:39.655504+0000","last_became_active":"2023-10-16T13:46:37.647909+0000","last_became_peered":"2023-10-16T13:46:37.647909+0000","last_unstale":"2023-10-16T13:46:39.655504+0000","last_undegraded":"2023-10-16T13:46:39.655504+0000","last_fullsized":"2023-10-16T13:46:39.655504+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_clean_scrub_stamp":"2023-10-16T13:46:36.617142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:47:07.699114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.14","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.670727+0000","last_change":"2023-10-16T13:46:38.656732+0000","last_active":"2023-10-16T13:46:39.670727+0000","last_peered":"2023-10-16T13:46:39.670727+0000","last_clean":"2023-10-16T13:46:39.670727+0000","last_became_active":"2023-10-16T13:46:38.656387+0000","last_became_peered":"2023-10-16T13:46:38.656387+0000","last_unstale":"2023-10-16T13:46:39.670727+0000","last_undegraded":"2023-10-16T13:46:39.670727+0000","last_fullsized":"2023-10-16T13:46:39.670727+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:36:17.135906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.652378+0000","last_change":"2023-10-16T13:46:24.336805+0000","last_active":"2023-10-16T13:46:38.652378+0000","last_peered":"2023-10-16T13:46:38.652378+0000","last_clean":"2023-10-16T13:46:38.652378+0000","last_became_active":"2023-10-16T13:46:24.336473+0000","last_became_peered":"2023-10-16T13:46:24.336473+0000","last_unstale":"2023-10-16T13:46:38.652378+0000","last_undegraded":"2023-10-16T13:46:38.652378+0000","last_fullsized":"2023-10-16T13:46:38.652378+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:44:10.241047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.652982+0000","last_change":"2023-10-16T13:46:25.575462+0000","last_active":"2023-10-16T13:46:38.652982+0000","last_peered":"2023-10-16T13:46:38.652982+0000","last_clean":"2023-10-16T13:46:38.652982+0000","last_became_active":"2023-10-16T13:46:25.574929+0000","last_became_peered":"2023-10-16T13:46:25.574929+0000","last_unstale":"2023-10-16T13:46:38.652982+0000","last_undegraded":"2023-10-16T13:46:38.652982+0000","last_fullsized":"2023-10-16T13:46:38.652982+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:39:18.796503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.14","version":"228'6","reported_seq":18,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.930310+0000","last_change":"2023-10-16T13:46:37.652614+0000","last_active":"2023-10-16T13:46:39.930310+0000","last_peered":"2023-10-16T13:46:39.930310+0000","last_clean":"2023-10-16T13:46:39.930310+0000","last_became_active":"2023-10-16T13:46:37.652318+0000","last_became_peered":"2023-10-16T13:46:37.652318+0000","last_unstale":"2023-10-16T13:46:39.930310+0000","last_undegraded":"2023-10-16T13:46:39.930310+0000","last_fullsized":"2023-10-16T13:46:39.930310+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_clean_scrub_stamp":"2023-10-16T13:46:36.617142+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:22:44.044158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.673586+0000","last_change":"2023-10-16T13:46:38.659187+0000","last_active":"2023-10-16T13:46:39.673586+0000","last_peered":"2023-10-16T13:46:39.673586+0000","last_clean":"2023-10-16T13:46:39.673586+0000","last_became_active":"2023-10-16T13:46:38.658898+0000","last_became_peered":"2023-10-16T13:46:38.658898+0000","last_unstale":"2023-10-16T13:46:39.673586+0000","last_undegraded":"2023-10-16T13:46:39.673586+0000","last_fullsized":"2023-10-16T13:46:39.673586+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:11:38.752824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.13","version":"224'2","reported_seq":18,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.649133+0000","last_change":"2023-10-16T13:46:24.333306+0000","last_active":"2023-10-16T13:46:38.649133+0000","last_peered":"2023-10-16T13:46:38.649133+0000","last_clean":"2023-10-16T13:46:38.649133+0000","last_became_active":"2023-10-16T13:46:24.333042+0000","last_became_peered":"2023-10-16T13:46:24.333042+0000","last_unstale":"2023-10-16T13:46:38.649133+0000","last_undegraded":"2023-10-16T13:46:38.649133+0000","last_fullsized":"2023-10-16T13:46:38.649133+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:44:45.300765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.14","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.654564+0000","last_change":"2023-10-16T13:46:25.566526+0000","last_active":"2023-10-16T13:46:38.654564+0000","last_peered":"2023-10-16T13:46:38.654564+0000","last_clean":"2023-10-16T13:46:38.654564+0000","last_became_active":"2023-10-16T13:46:25.566252+0000","last_became_peered":"2023-10-16T13:46:25.566252+0000","last_unstale":"2023-10-16T13:46:38.654564+0000","last_undegraded":"2023-10-16T13:46:38.654564+0000","last_fullsized":"2023-10-16T13:46:38.654564+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:36:03.832144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.15","version":"228'3","reported_seq":15,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.655290+0000","last_change":"2023-10-16T13:46:37.646597+0000","last_active":"2023-10-16T13:46:39.655290+0000","last_peered":"2023-10-16T13:46:39.655290+0000","last_clean":"2023-10-16T13:46:39.655290+0000","last_became_active":"2023-10-16T13:46:37.646321+0000","last_became_peered":"2023-10-16T13:46:37.646321+0000","last_unstale":"2023-10-16T13:46:39.655290+0000","last_undegraded":"2023-10-16T13:46:39.655290+0000","last_fullsized":"2023-10-16T13:46:39.655290+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_clean_scrub_stamp":"2023-10-16T13:46:36.617142+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:21:03.012606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.581891+0000","last_change":"2023-10-16T13:46:38.661774+0000","last_active":"2023-10-16T13:46:40.581891+0000","last_peered":"2023-10-16T13:46:40.581891+0000","last_clean":"2023-10-16T13:46:40.581891+0000","last_became_active":"2023-10-16T13:46:38.661440+0000","last_became_peered":"2023-10-16T13:46:38.661440+0000","last_unstale":"2023-10-16T13:46:40.581891+0000","last_undegraded":"2023-10-16T13:46:40.581891+0000","last_fullsized":"2023-10-16T13:46:40.581891+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:55:30.798202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.14","version":"225'8","reported_seq":24,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.647310+0000","last_change":"2023-10-16T13:46:24.347248+0000","last_active":"2023-10-16T13:46:38.647310+0000","last_peered":"2023-10-16T13:46:38.647310+0000","last_clean":"2023-10-16T13:46:38.647310+0000","last_became_active":"2023-10-16T13:46:24.346699+0000","last_became_peered":"2023-10-16T13:46:24.346699+0000","last_unstale":"2023-10-16T13:46:38.647310+0000","last_undegraded":"2023-10-16T13:46:38.647310+0000","last_fullsized":"2023-10-16T13:46:38.647310+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:15:37.760742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2859,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.13","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.641114+0000","last_change":"2023-10-16T13:46:25.578929+0000","last_active":"2023-10-16T13:46:38.641114+0000","last_peered":"2023-10-16T13:46:38.641114+0000","last_clean":"2023-10-16T13:46:38.641114+0000","last_became_active":"2023-10-16T13:46:25.578431+0000","last_became_peered":"2023-10-16T13:46:25.578431+0000","last_unstale":"2023-10-16T13:46:38.641114+0000","last_undegraded":"2023-10-16T13:46:38.641114+0000","last_fullsized":"2023-10-16T13:46:38.641114+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:23:30.528000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.12","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.653684+0000","last_change":"2023-10-16T13:46:37.644333+0000","last_active":"2023-10-16T13:46:39.653684+0000","last_peered":"2023-10-16T13:46:39.653684+0000","last_clean":"2023-10-16T13:46:39.653684+0000","last_became_active":"2023-10-16T13:46:37.644076+0000","last_became_peered":"2023-10-16T13:46:37.644076+0000","last_unstale":"2023-10-16T13:46:39.653684+0000","last_undegraded":"2023-10-16T13:46:39.653684+0000","last_fullsized":"2023-10-16T13:46:39.653684+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_clean_scrub_stamp":"2023-10-16T13:46:36.617142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:45:22.549136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.097472+0000","last_change":"2023-10-16T13:46:38.662644+0000","last_active":"2023-10-16T13:46:40.097472+0000","last_peered":"2023-10-16T13:46:40.097472+0000","last_clean":"2023-10-16T13:46:40.097472+0000","last_became_active":"2023-10-16T13:46:38.662350+0000","last_became_peered":"2023-10-16T13:46:38.662350+0000","last_unstale":"2023-10-16T13:46:40.097472+0000","last_undegraded":"2023-10-16T13:46:40.097472+0000","last_fullsized":"2023-10-16T13:46:40.097472+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:04:52.014766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.15","version":"224'3","reported_seq":19,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.647904+0000","last_change":"2023-10-16T13:46:24.340273+0000","last_active":"2023-10-16T13:46:38.647904+0000","last_peered":"2023-10-16T13:46:38.647904+0000","last_clean":"2023-10-16T13:46:38.647904+0000","last_became_active":"2023-10-16T13:46:24.339939+0000","last_became_peered":"2023-10-16T13:46:24.339939+0000","last_unstale":"2023-10-16T13:46:38.647904+0000","last_undegraded":"2023-10-16T13:46:38.647904+0000","last_fullsized":"2023-10-16T13:46:38.647904+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:25:07.149748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.646802+0000","last_change":"2023-10-16T13:46:25.580296+0000","last_active":"2023-10-16T13:46:38.646802+0000","last_peered":"2023-10-16T13:46:38.646802+0000","last_clean":"2023-10-16T13:46:38.646802+0000","last_became_active":"2023-10-16T13:46:25.579963+0000","last_became_peered":"2023-10-16T13:46:25.579963+0000","last_unstale":"2023-10-16T13:46:38.646802+0000","last_undegraded":"2023-10-16T13:46:38.646802+0000","last_fullsized":"2023-10-16T13:46:38.646802+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:39:18.796503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.13","version":"228'2","reported_seq":14,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.097706+0000","last_change":"2023-10-16T13:46:37.647772+0000","last_active":"2023-10-16T13:46:40.097706+0000","last_peered":"2023-10-16T13:46:40.097706+0000","last_clean":"2023-10-16T13:46:40.097706+0000","last_became_active":"2023-10-16T13:46:37.647248+0000","last_became_peered":"2023-10-16T13:46:37.647248+0000","last_unstale":"2023-10-16T13:46:40.097706+0000","last_undegraded":"2023-10-16T13:46:40.097706+0000","last_fullsized":"2023-10-16T13:46:40.097706+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_clean_scrub_stamp":"2023-10-16T13:46:36.617142+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:27:12.664826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.10","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.653438+0000","last_change":"2023-10-16T13:46:38.658182+0000","last_active":"2023-10-16T13:46:39.653438+0000","last_peered":"2023-10-16T13:46:39.653438+0000","last_clean":"2023-10-16T13:46:39.653438+0000","last_became_active":"2023-10-16T13:46:38.657901+0000","last_became_peered":"2023-10-16T13:46:38.657901+0000","last_unstale":"2023-10-16T13:46:39.653438+0000","last_undegraded":"2023-10-16T13:46:39.653438+0000","last_fullsized":"2023-10-16T13:46:39.653438+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:45:44.082623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.651460+0000","last_change":"2023-10-16T13:46:24.344174+0000","last_active":"2023-10-16T13:46:38.651460+0000","last_peered":"2023-10-16T13:46:38.651460+0000","last_clean":"2023-10-16T13:46:38.651460+0000","last_became_active":"2023-10-16T13:46:24.343883+0000","last_became_peered":"2023-10-16T13:46:24.343883+0000","last_unstale":"2023-10-16T13:46:38.651460+0000","last_undegraded":"2023-10-16T13:46:38.651460+0000","last_fullsized":"2023-10-16T13:46:38.651460+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:53:47.000133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.651365+0000","last_change":"2023-10-16T13:46:25.572773+0000","last_active":"2023-10-16T13:46:38.651365+0000","last_peered":"2023-10-16T13:46:38.651365+0000","last_clean":"2023-10-16T13:46:38.651365+0000","last_became_active":"2023-10-16T13:46:25.572324+0000","last_became_peered":"2023-10-16T13:46:25.572324+0000","last_unstale":"2023-10-16T13:46:38.651365+0000","last_undegraded":"2023-10-16T13:46:38.651365+0000","last_fullsized":"2023-10-16T13:46:38.651365+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:41:30.230501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.10","version":"228'8","reported_seq":20,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.097389+0000","last_change":"2023-10-16T13:46:37.647724+0000","last_active":"2023-10-16T13:46:40.097389+0000","last_peered":"2023-10-16T13:46:40.097389+0000","last_clean":"2023-10-16T13:46:40.097389+0000","last_became_active":"2023-10-16T13:46:37.647053+0000","last_became_peered":"2023-10-16T13:46:37.647053+0000","last_unstale":"2023-10-16T13:46:40.097389+0000","last_undegraded":"2023-10-16T13:46:40.097389+0000","last_fullsized":"2023-10-16T13:46:40.097389+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_clean_scrub_stamp":"2023-10-16T13:46:36.617142+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:07:12.646343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.13","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.670843+0000","last_change":"2023-10-16T13:46:38.666851+0000","last_active":"2023-10-16T13:46:39.670843+0000","last_peered":"2023-10-16T13:46:39.670843+0000","last_clean":"2023-10-16T13:46:39.670843+0000","last_became_active":"2023-10-16T13:46:38.666538+0000","last_became_peered":"2023-10-16T13:46:38.666538+0000","last_unstale":"2023-10-16T13:46:39.670843+0000","last_undegraded":"2023-10-16T13:46:39.670843+0000","last_fullsized":"2023-10-16T13:46:39.670843+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:21:04.017774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.642294+0000","last_change":"2023-10-16T13:46:24.346329+0000","last_active":"2023-10-16T13:46:38.642294+0000","last_peered":"2023-10-16T13:46:38.642294+0000","last_clean":"2023-10-16T13:46:38.642294+0000","last_became_active":"2023-10-16T13:46:24.346051+0000","last_became_peered":"2023-10-16T13:46:24.346051+0000","last_unstale":"2023-10-16T13:46:38.642294+0000","last_undegraded":"2023-10-16T13:46:38.642294+0000","last_fullsized":"2023-10-16T13:46:38.642294+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:14:11.499564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.641955+0000","last_change":"2023-10-16T13:46:25.574355+0000","last_active":"2023-10-16T13:46:38.641955+0000","last_peered":"2023-10-16T13:46:38.641955+0000","last_clean":"2023-10-16T13:46:38.641955+0000","last_became_active":"2023-10-16T13:46:25.574057+0000","last_became_peered":"2023-10-16T13:46:25.574057+0000","last_unstale":"2023-10-16T13:46:38.641955+0000","last_undegraded":"2023-10-16T13:46:38.641955+0000","last_fullsized":"2023-10-16T13:46:38.641955+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:41:42.462761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.11","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.929520+0000","last_change":"2023-10-16T13:46:37.648607+0000","last_active":"2023-10-16T13:46:39.929520+0000","last_peered":"2023-10-16T13:46:39.929520+0000","last_clean":"2023-10-16T13:46:39.929520+0000","last_became_active":"2023-10-16T13:46:37.648317+0000","last_became_peered":"2023-10-16T13:46:37.648317+0000","last_unstale":"2023-10-16T13:46:39.929520+0000","last_undegraded":"2023-10-16T13:46:39.929520+0000","last_fullsized":"2023-10-16T13:46:39.929520+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_clean_scrub_stamp":"2023-10-16T13:46:36.617142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:47:13.553188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.580948+0000","last_change":"2023-10-16T13:46:38.661614+0000","last_active":"2023-10-16T13:46:40.580948+0000","last_peered":"2023-10-16T13:46:40.580948+0000","last_clean":"2023-10-16T13:46:40.580948+0000","last_became_active":"2023-10-16T13:46:38.661188+0000","last_became_peered":"2023-10-16T13:46:38.661188+0000","last_unstale":"2023-10-16T13:46:40.580948+0000","last_undegraded":"2023-10-16T13:46:40.580948+0000","last_fullsized":"2023-10-16T13:46:40.580948+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:36:32.753888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.653302+0000","last_change":"2023-10-16T13:46:24.338318+0000","last_active":"2023-10-16T13:46:38.653302+0000","last_peered":"2023-10-16T13:46:38.653302+0000","last_clean":"2023-10-16T13:46:38.653302+0000","last_became_active":"2023-10-16T13:46:24.337766+0000","last_became_peered":"2023-10-16T13:46:24.337766+0000","last_unstale":"2023-10-16T13:46:38.653302+0000","last_undegraded":"2023-10-16T13:46:38.653302+0000","last_fullsized":"2023-10-16T13:46:38.653302+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:25:07.149748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1f","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.656659+0000","last_change":"2023-10-16T13:46:25.571884+0000","last_active":"2023-10-16T13:46:38.656659+0000","last_peered":"2023-10-16T13:46:38.656659+0000","last_clean":"2023-10-16T13:46:38.656659+0000","last_became_active":"2023-10-16T13:46:25.571601+0000","last_became_peered":"2023-10-16T13:46:25.571601+0000","last_unstale":"2023-10-16T13:46:38.656659+0000","last_undegraded":"2023-10-16T13:46:38.656659+0000","last_fullsized":"2023-10-16T13:46:38.656659+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:32:41.007775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.1e","version":"228'1","reported_seq":13,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.098663+0000","last_change":"2023-10-16T13:46:37.644119+0000","last_active":"2023-10-16T13:46:40.098663+0000","last_peered":"2023-10-16T13:46:40.098663+0000","last_clean":"2023-10-16T13:46:40.098663+0000","last_became_active":"2023-10-16T13:46:37.643578+0000","last_became_peered":"2023-10-16T13:46:37.643578+0000","last_unstale":"2023-10-16T13:46:40.098663+0000","last_undegraded":"2023-10-16T13:46:40.098663+0000","last_fullsized":"2023-10-16T13:46:40.098663+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_clean_scrub_stamp":"2023-10-16T13:46:36.617142+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:45:47.428975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1d","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.581448+0000","last_change":"2023-10-16T13:46:38.660503+0000","last_active":"2023-10-16T13:46:40.581448+0000","last_peered":"2023-10-16T13:46:40.581448+0000","last_clean":"2023-10-16T13:46:40.581448+0000","last_became_active":"2023-10-16T13:46:38.660181+0000","last_became_peered":"2023-10-16T13:46:38.660181+0000","last_unstale":"2023-10-16T13:46:40.581448+0000","last_undegraded":"2023-10-16T13:46:40.581448+0000","last_fullsized":"2023-10-16T13:46:40.581448+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:20:33.487282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":16,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.651830+0000","last_change":"2023-10-16T13:46:24.343337+0000","last_active":"2023-10-16T13:46:38.651830+0000","last_peered":"2023-10-16T13:46:38.651830+0000","last_clean":"2023-10-16T13:46:38.651830+0000","last_became_active":"2023-10-16T13:46:24.343043+0000","last_became_peered":"2023-10-16T13:46:24.343043+0000","last_unstale":"2023-10-16T13:46:38.651830+0000","last_undegraded":"2023-10-16T13:46:38.651830+0000","last_fullsized":"2023-10-16T13:46:38.651830+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:16:31.012130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.643026+0000","last_change":"2023-10-16T13:46:25.571656+0000","last_active":"2023-10-16T13:46:38.643026+0000","last_peered":"2023-10-16T13:46:38.643026+0000","last_clean":"2023-10-16T13:46:38.643026+0000","last_became_active":"2023-10-16T13:46:25.571372+0000","last_became_peered":"2023-10-16T13:46:25.571372+0000","last_unstale":"2023-10-16T13:46:38.643026+0000","last_undegraded":"2023-10-16T13:46:38.643026+0000","last_fullsized":"2023-10-16T13:46:38.643026+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:57:59.993900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1f","version":"228'4","reported_seq":16,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.867403+0000","last_change":"2023-10-16T13:46:37.644759+0000","last_active":"2023-10-16T13:46:40.867403+0000","last_peered":"2023-10-16T13:46:40.867403+0000","last_clean":"2023-10-16T13:46:40.867403+0000","last_became_active":"2023-10-16T13:46:37.644543+0000","last_became_peered":"2023-10-16T13:46:37.644543+0000","last_unstale":"2023-10-16T13:46:40.867403+0000","last_undegraded":"2023-10-16T13:46:40.867403+0000","last_fullsized":"2023-10-16T13:46:40.867403+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_clean_scrub_stamp":"2023-10-16T13:46:36.617142+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:55:04.080446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.1c","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.581302+0000","last_change":"2023-10-16T13:46:38.660367+0000","last_active":"2023-10-16T13:46:40.581302+0000","last_peered":"2023-10-16T13:46:40.581302+0000","last_clean":"2023-10-16T13:46:40.581302+0000","last_became_active":"2023-10-16T13:46:38.660070+0000","last_became_peered":"2023-10-16T13:46:38.660070+0000","last_unstale":"2023-10-16T13:46:40.581302+0000","last_undegraded":"2023-10-16T13:46:40.581302+0000","last_fullsized":"2023-10-16T13:46:40.581302+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:58:29.279269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.1b","version":"224'1","reported_seq":17,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.659232+0000","last_change":"2023-10-16T13:46:24.343654+0000","last_active":"2023-10-16T13:46:38.659232+0000","last_peered":"2023-10-16T13:46:38.659232+0000","last_clean":"2023-10-16T13:46:38.659232+0000","last_became_active":"2023-10-16T13:46:24.343330+0000","last_became_peered":"2023-10-16T13:46:24.343330+0000","last_unstale":"2023-10-16T13:46:38.659232+0000","last_undegraded":"2023-10-16T13:46:38.659232+0000","last_fullsized":"2023-10-16T13:46:38.659232+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:47:30.680184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.642611+0000","last_change":"2023-10-16T13:46:25.569919+0000","last_active":"2023-10-16T13:46:38.642611+0000","last_peered":"2023-10-16T13:46:38.642611+0000","last_clean":"2023-10-16T13:46:38.642611+0000","last_became_active":"2023-10-16T13:46:25.569619+0000","last_became_peered":"2023-10-16T13:46:25.569619+0000","last_unstale":"2023-10-16T13:46:38.642611+0000","last_undegraded":"2023-10-16T13:46:38.642611+0000","last_fullsized":"2023-10-16T13:46:38.642611+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:43:18.855055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1d","version":"228'2","reported_seq":14,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.929824+0000","last_change":"2023-10-16T13:46:37.651911+0000","last_active":"2023-10-16T13:46:39.929824+0000","last_peered":"2023-10-16T13:46:39.929824+0000","last_clean":"2023-10-16T13:46:39.929824+0000","last_became_active":"2023-10-16T13:46:37.651675+0000","last_became_peered":"2023-10-16T13:46:37.651675+0000","last_unstale":"2023-10-16T13:46:39.929824+0000","last_undegraded":"2023-10-16T13:46:39.929824+0000","last_fullsized":"2023-10-16T13:46:39.929824+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_clean_scrub_stamp":"2023-10-16T13:46:36.617142+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:21:44.559586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1e","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.098551+0000","last_change":"2023-10-16T13:46:38.673439+0000","last_active":"2023-10-16T13:46:40.098551+0000","last_peered":"2023-10-16T13:46:40.098551+0000","last_clean":"2023-10-16T13:46:40.098551+0000","last_became_active":"2023-10-16T13:46:38.669858+0000","last_became_peered":"2023-10-16T13:46:38.669858+0000","last_unstale":"2023-10-16T13:46:40.098551+0000","last_undegraded":"2023-10-16T13:46:40.098551+0000","last_fullsized":"2023-10-16T13:46:40.098551+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:04:24.157820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1c","version":"224'2","reported_seq":18,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.647491+0000","last_change":"2023-10-16T13:46:24.340877+0000","last_active":"2023-10-16T13:46:38.647491+0000","last_peered":"2023-10-16T13:46:38.647491+0000","last_clean":"2023-10-16T13:46:38.647491+0000","last_became_active":"2023-10-16T13:46:24.340607+0000","last_became_peered":"2023-10-16T13:46:24.340607+0000","last_unstale":"2023-10-16T13:46:38.647491+0000","last_undegraded":"2023-10-16T13:46:38.647491+0000","last_fullsized":"2023-10-16T13:46:38.647491+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:16:31.012130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1b","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.641051+0000","last_change":"2023-10-16T13:46:25.571278+0000","last_active":"2023-10-16T13:46:38.641051+0000","last_peered":"2023-10-16T13:46:38.641051+0000","last_clean":"2023-10-16T13:46:38.641051+0000","last_became_active":"2023-10-16T13:46:25.570849+0000","last_became_peered":"2023-10-16T13:46:25.570849+0000","last_unstale":"2023-10-16T13:46:38.641051+0000","last_undegraded":"2023-10-16T13:46:38.641051+0000","last_fullsized":"2023-10-16T13:46:38.641051+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:25:56.358129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.1a","version":"228'1","reported_seq":13,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.654473+0000","last_change":"2023-10-16T13:46:37.650793+0000","last_active":"2023-10-16T13:46:39.654473+0000","last_peered":"2023-10-16T13:46:39.654473+0000","last_clean":"2023-10-16T13:46:39.654473+0000","last_became_active":"2023-10-16T13:46:37.650486+0000","last_became_peered":"2023-10-16T13:46:37.650486+0000","last_unstale":"2023-10-16T13:46:39.654473+0000","last_undegraded":"2023-10-16T13:46:39.654473+0000","last_fullsized":"2023-10-16T13:46:39.654473+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_clean_scrub_stamp":"2023-10-16T13:46:36.617142+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:36:16.130738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.097423+0000","last_change":"2023-10-16T13:46:38.661608+0000","last_active":"2023-10-16T13:46:40.097423+0000","last_peered":"2023-10-16T13:46:40.097423+0000","last_clean":"2023-10-16T13:46:40.097423+0000","last_became_active":"2023-10-16T13:46:38.661312+0000","last_became_peered":"2023-10-16T13:46:38.661312+0000","last_unstale":"2023-10-16T13:46:40.097423+0000","last_undegraded":"2023-10-16T13:46:40.097423+0000","last_fullsized":"2023-10-16T13:46:40.097423+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:01:08.982727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1d","version":"224'2","reported_seq":18,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.658778+0000","last_change":"2023-10-16T13:46:24.340608+0000","last_active":"2023-10-16T13:46:38.658778+0000","last_peered":"2023-10-16T13:46:38.658778+0000","last_clean":"2023-10-16T13:46:38.658778+0000","last_became_active":"2023-10-16T13:46:24.340189+0000","last_became_peered":"2023-10-16T13:46:24.340189+0000","last_unstale":"2023-10-16T13:46:38.658778+0000","last_undegraded":"2023-10-16T13:46:38.658778+0000","last_fullsized":"2023-10-16T13:46:38.658778+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:16:00.359557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.647581+0000","last_change":"2023-10-16T13:46:25.576216+0000","last_active":"2023-10-16T13:46:38.647581+0000","last_peered":"2023-10-16T13:46:38.647581+0000","last_clean":"2023-10-16T13:46:38.647581+0000","last_became_active":"2023-10-16T13:46:25.575876+0000","last_became_peered":"2023-10-16T13:46:25.575876+0000","last_unstale":"2023-10-16T13:46:38.647581+0000","last_undegraded":"2023-10-16T13:46:38.647581+0000","last_fullsized":"2023-10-16T13:46:38.647581+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:41:08.099115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1b","version":"228'1","reported_seq":13,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.670711+0000","last_change":"2023-10-16T13:46:37.650980+0000","last_active":"2023-10-16T13:46:39.670711+0000","last_peered":"2023-10-16T13:46:39.670711+0000","last_clean":"2023-10-16T13:46:39.670711+0000","last_became_active":"2023-10-16T13:46:37.650655+0000","last_became_peered":"2023-10-16T13:46:37.650655+0000","last_unstale":"2023-10-16T13:46:39.670711+0000","last_undegraded":"2023-10-16T13:46:39.670711+0000","last_fullsized":"2023-10-16T13:46:39.670711+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_clean_scrub_stamp":"2023-10-16T13:46:36.617142+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:16:13.659948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.580536+0000","last_change":"2023-10-16T13:46:38.660238+0000","last_active":"2023-10-16T13:46:40.580536+0000","last_peered":"2023-10-16T13:46:40.580536+0000","last_clean":"2023-10-16T13:46:40.580536+0000","last_became_active":"2023-10-16T13:46:38.659923+0000","last_became_peered":"2023-10-16T13:46:38.659923+0000","last_unstale":"2023-10-16T13:46:40.580536+0000","last_undegraded":"2023-10-16T13:46:40.580536+0000","last_fullsized":"2023-10-16T13:46:40.580536+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:57:12.270376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.1e","version":"224'1","reported_seq":17,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.650772+0000","last_change":"2023-10-16T13:46:24.340000+0000","last_active":"2023-10-16T13:46:38.650772+0000","last_peered":"2023-10-16T13:46:38.650772+0000","last_clean":"2023-10-16T13:46:38.650772+0000","last_became_active":"2023-10-16T13:46:24.339714+0000","last_became_peered":"2023-10-16T13:46:24.339714+0000","last_unstale":"2023-10-16T13:46:38.650772+0000","last_undegraded":"2023-10-16T13:46:38.650772+0000","last_fullsized":"2023-10-16T13:46:38.650772+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:21:04.619534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.647146+0000","last_change":"2023-10-16T13:46:25.577384+0000","last_active":"2023-10-16T13:46:38.647146+0000","last_peered":"2023-10-16T13:46:38.647146+0000","last_clean":"2023-10-16T13:46:38.647146+0000","last_became_active":"2023-10-16T13:46:25.577071+0000","last_became_peered":"2023-10-16T13:46:25.577071+0000","last_unstale":"2023-10-16T13:46:38.647146+0000","last_undegraded":"2023-10-16T13:46:38.647146+0000","last_fullsized":"2023-10-16T13:46:38.647146+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:44:11.242845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.18","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.580667+0000","last_change":"2023-10-16T13:46:37.643158+0000","last_active":"2023-10-16T13:46:40.580667+0000","last_peered":"2023-10-16T13:46:40.580667+0000","last_clean":"2023-10-16T13:46:40.580667+0000","last_became_active":"2023-10-16T13:46:37.640753+0000","last_became_peered":"2023-10-16T13:46:37.640753+0000","last_unstale":"2023-10-16T13:46:40.580667+0000","last_undegraded":"2023-10-16T13:46:40.580667+0000","last_fullsized":"2023-10-16T13:46:40.580667+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_clean_scrub_stamp":"2023-10-16T13:46:36.617142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:38:43.180849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.1b","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.866964+0000","last_change":"2023-10-16T13:46:38.655353+0000","last_active":"2023-10-16T13:46:40.866964+0000","last_peered":"2023-10-16T13:46:40.866964+0000","last_clean":"2023-10-16T13:46:40.866964+0000","last_became_active":"2023-10-16T13:46:38.655006+0000","last_became_peered":"2023-10-16T13:46:38.655006+0000","last_unstale":"2023-10-16T13:46:40.866964+0000","last_undegraded":"2023-10-16T13:46:40.866964+0000","last_fullsized":"2023-10-16T13:46:40.866964+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:00:56.849374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1f","version":"224'4","reported_seq":20,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.643243+0000","last_change":"2023-10-16T13:46:24.345142+0000","last_active":"2023-10-16T13:46:38.643243+0000","last_peered":"2023-10-16T13:46:38.643243+0000","last_clean":"2023-10-16T13:46:38.643243+0000","last_became_active":"2023-10-16T13:46:24.344816+0000","last_became_peered":"2023-10-16T13:46:24.344816+0000","last_unstale":"2023-10-16T13:46:38.643243+0000","last_undegraded":"2023-10-16T13:46:38.643243+0000","last_fullsized":"2023-10-16T13:46:38.643243+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":222,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:23.316751+0000","last_clean_scrub_stamp":"2023-10-16T13:46:23.316751+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:54:50.780055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.648377+0000","last_change":"2023-10-16T13:46:25.578414+0000","last_active":"2023-10-16T13:46:38.648377+0000","last_peered":"2023-10-16T13:46:38.648377+0000","last_clean":"2023-10-16T13:46:38.648377+0000","last_became_active":"2023-10-16T13:46:25.578111+0000","last_became_peered":"2023-10-16T13:46:25.578111+0000","last_unstale":"2023-10-16T13:46:38.648377+0000","last_undegraded":"2023-10-16T13:46:38.648377+0000","last_fullsized":"2023-10-16T13:46:38.648377+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:25:44.802079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.19","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.672877+0000","last_change":"2023-10-16T13:46:37.646754+0000","last_active":"2023-10-16T13:46:39.672877+0000","last_peered":"2023-10-16T13:46:39.672877+0000","last_clean":"2023-10-16T13:46:39.672877+0000","last_became_active":"2023-10-16T13:46:37.646395+0000","last_became_peered":"2023-10-16T13:46:37.646395+0000","last_unstale":"2023-10-16T13:46:39.672877+0000","last_undegraded":"2023-10-16T13:46:39.672877+0000","last_fullsized":"2023-10-16T13:46:39.672877+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:36.617142+0000","last_clean_scrub_stamp":"2023-10-16T13:46:36.617142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:40:44.718376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1a","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.670847+0000","last_change":"2023-10-16T13:46:38.656584+0000","last_active":"2023-10-16T13:46:39.670847+0000","last_peered":"2023-10-16T13:46:39.670847+0000","last_clean":"2023-10-16T13:46:39.670847+0000","last_became_active":"2023-10-16T13:46:38.656068+0000","last_became_peered":"2023-10-16T13:46:38.656068+0000","last_unstale":"2023-10-16T13:46:39.670847+0000","last_undegraded":"2023-10-16T13:46:39.670847+0000","last_fullsized":"2023-10-16T13:46:39.670847+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:22:22.209998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.20","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.645880+0000","last_change":"2023-10-16T13:46:25.572825+0000","last_active":"2023-10-16T13:46:38.645880+0000","last_peered":"2023-10-16T13:46:38.645880+0000","last_clean":"2023-10-16T13:46:38.645880+0000","last_became_active":"2023-10-16T13:46:25.572507+0000","last_became_peered":"2023-10-16T13:46:25.572507+0000","last_unstale":"2023-10-16T13:46:38.645880+0000","last_undegraded":"2023-10-16T13:46:38.645880+0000","last_fullsized":"2023-10-16T13:46:38.645880+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:51:29.125394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.22","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.929819+0000","last_change":"2023-10-16T13:46:38.673677+0000","last_active":"2023-10-16T13:46:39.929819+0000","last_peered":"2023-10-16T13:46:39.929819+0000","last_clean":"2023-10-16T13:46:39.929819+0000","last_became_active":"2023-10-16T13:46:38.673317+0000","last_became_peered":"2023-10-16T13:46:38.673317+0000","last_unstale":"2023-10-16T13:46:39.929819+0000","last_undegraded":"2023-10-16T13:46:39.929819+0000","last_fullsized":"2023-10-16T13:46:39.929819+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:28:07.580655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.21","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.651614+0000","last_change":"2023-10-16T13:46:25.580726+0000","last_active":"2023-10-16T13:46:38.651614+0000","last_peered":"2023-10-16T13:46:38.651614+0000","last_clean":"2023-10-16T13:46:38.651614+0000","last_became_active":"2023-10-16T13:46:25.580423+0000","last_became_peered":"2023-10-16T13:46:25.580423+0000","last_unstale":"2023-10-16T13:46:38.651614+0000","last_undegraded":"2023-10-16T13:46:38.651614+0000","last_fullsized":"2023-10-16T13:46:38.651614+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:21:32.260993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.23","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.654095+0000","last_change":"2023-10-16T13:46:38.657677+0000","last_active":"2023-10-16T13:46:39.654095+0000","last_peered":"2023-10-16T13:46:39.654095+0000","last_clean":"2023-10-16T13:46:39.654095+0000","last_became_active":"2023-10-16T13:46:38.657347+0000","last_became_peered":"2023-10-16T13:46:38.657347+0000","last_unstale":"2023-10-16T13:46:39.654095+0000","last_undegraded":"2023-10-16T13:46:39.654095+0000","last_fullsized":"2023-10-16T13:46:39.654095+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:21:04.017774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.22","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.642763+0000","last_change":"2023-10-16T13:46:25.569845+0000","last_active":"2023-10-16T13:46:38.642763+0000","last_peered":"2023-10-16T13:46:38.642763+0000","last_clean":"2023-10-16T13:46:38.642763+0000","last_became_active":"2023-10-16T13:46:25.569524+0000","last_became_peered":"2023-10-16T13:46:25.569524+0000","last_unstale":"2023-10-16T13:46:38.642763+0000","last_undegraded":"2023-10-16T13:46:38.642763+0000","last_fullsized":"2023-10-16T13:46:38.642763+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:16:15.086261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.20","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.929777+0000","last_change":"2023-10-16T13:46:38.672106+0000","last_active":"2023-10-16T13:46:39.929777+0000","last_peered":"2023-10-16T13:46:39.929777+0000","last_clean":"2023-10-16T13:46:39.929777+0000","last_became_active":"2023-10-16T13:46:38.671793+0000","last_became_peered":"2023-10-16T13:46:38.671793+0000","last_unstale":"2023-10-16T13:46:39.929777+0000","last_undegraded":"2023-10-16T13:46:39.929777+0000","last_fullsized":"2023-10-16T13:46:39.929777+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:32:54.311537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.23","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.646695+0000","last_change":"2023-10-16T13:46:25.568230+0000","last_active":"2023-10-16T13:46:38.646695+0000","last_peered":"2023-10-16T13:46:38.646695+0000","last_clean":"2023-10-16T13:46:38.646695+0000","last_became_active":"2023-10-16T13:46:25.567927+0000","last_became_peered":"2023-10-16T13:46:25.567927+0000","last_unstale":"2023-10-16T13:46:38.646695+0000","last_undegraded":"2023-10-16T13:46:38.646695+0000","last_fullsized":"2023-10-16T13:46:38.646695+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:51:29.125394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.21","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.866626+0000","last_change":"2023-10-16T13:46:38.663451+0000","last_active":"2023-10-16T13:46:40.866626+0000","last_peered":"2023-10-16T13:46:40.866626+0000","last_clean":"2023-10-16T13:46:40.866626+0000","last_became_active":"2023-10-16T13:46:38.662499+0000","last_became_peered":"2023-10-16T13:46:38.662499+0000","last_unstale":"2023-10-16T13:46:40.866626+0000","last_undegraded":"2023-10-16T13:46:40.866626+0000","last_fullsized":"2023-10-16T13:46:40.866626+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:43:32.158816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.26","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.581740+0000","last_change":"2023-10-16T13:46:38.676278+0000","last_active":"2023-10-16T13:46:40.581740+0000","last_peered":"2023-10-16T13:46:40.581740+0000","last_clean":"2023-10-16T13:46:40.581740+0000","last_became_active":"2023-10-16T13:46:38.675992+0000","last_became_peered":"2023-10-16T13:46:38.675992+0000","last_unstale":"2023-10-16T13:46:40.581740+0000","last_undegraded":"2023-10-16T13:46:40.581740+0000","last_fullsized":"2023-10-16T13:46:40.581740+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:47:41.022437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.24","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.640853+0000","last_change":"2023-10-16T13:46:25.573728+0000","last_active":"2023-10-16T13:46:38.640853+0000","last_peered":"2023-10-16T13:46:38.640853+0000","last_clean":"2023-10-16T13:46:38.640853+0000","last_became_active":"2023-10-16T13:46:25.573427+0000","last_became_peered":"2023-10-16T13:46:25.573427+0000","last_unstale":"2023-10-16T13:46:38.640853+0000","last_undegraded":"2023-10-16T13:46:38.640853+0000","last_fullsized":"2023-10-16T13:46:38.640853+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:02:44.293532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.27","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.655525+0000","last_change":"2023-10-16T13:46:38.666370+0000","last_active":"2023-10-16T13:46:39.655525+0000","last_peered":"2023-10-16T13:46:39.655525+0000","last_clean":"2023-10-16T13:46:39.655525+0000","last_became_active":"2023-10-16T13:46:38.662756+0000","last_became_peered":"2023-10-16T13:46:38.662756+0000","last_unstale":"2023-10-16T13:46:39.655525+0000","last_undegraded":"2023-10-16T13:46:39.655525+0000","last_fullsized":"2023-10-16T13:46:39.655525+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:38:44.186017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.25","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.653088+0000","last_change":"2023-10-16T13:46:25.578298+0000","last_active":"2023-10-16T13:46:38.653088+0000","last_peered":"2023-10-16T13:46:38.653088+0000","last_clean":"2023-10-16T13:46:38.653088+0000","last_became_active":"2023-10-16T13:46:25.578057+0000","last_became_peered":"2023-10-16T13:46:25.578057+0000","last_unstale":"2023-10-16T13:46:38.653088+0000","last_undegraded":"2023-10-16T13:46:38.653088+0000","last_fullsized":"2023-10-16T13:46:38.653088+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:08:20.004685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.26","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.650659+0000","last_change":"2023-10-16T13:46:25.579701+0000","last_active":"2023-10-16T13:46:38.650659+0000","last_peered":"2023-10-16T13:46:38.650659+0000","last_clean":"2023-10-16T13:46:38.650659+0000","last_became_active":"2023-10-16T13:46:25.579378+0000","last_became_peered":"2023-10-16T13:46:25.579378+0000","last_unstale":"2023-10-16T13:46:38.650659+0000","last_undegraded":"2023-10-16T13:46:38.650659+0000","last_fullsized":"2023-10-16T13:46:38.650659+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:16:32.013927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.24","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.098110+0000","last_change":"2023-10-16T13:46:38.673084+0000","last_active":"2023-10-16T13:46:40.098110+0000","last_peered":"2023-10-16T13:46:40.098110+0000","last_clean":"2023-10-16T13:46:40.098110+0000","last_became_active":"2023-10-16T13:46:38.670381+0000","last_became_peered":"2023-10-16T13:46:38.670381+0000","last_unstale":"2023-10-16T13:46:40.098110+0000","last_undegraded":"2023-10-16T13:46:40.098110+0000","last_fullsized":"2023-10-16T13:46:40.098110+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:29:53.145261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.25","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.673681+0000","last_change":"2023-10-16T13:46:38.655995+0000","last_active":"2023-10-16T13:46:39.673681+0000","last_peered":"2023-10-16T13:46:39.673681+0000","last_clean":"2023-10-16T13:46:39.673681+0000","last_became_active":"2023-10-16T13:46:38.655560+0000","last_became_peered":"2023-10-16T13:46:38.655560+0000","last_unstale":"2023-10-16T13:46:39.673681+0000","last_undegraded":"2023-10-16T13:46:39.673681+0000","last_fullsized":"2023-10-16T13:46:39.673681+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:23:00.385693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.27","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.656567+0000","last_change":"2023-10-16T13:46:25.572081+0000","last_active":"2023-10-16T13:46:38.656567+0000","last_peered":"2023-10-16T13:46:38.656567+0000","last_clean":"2023-10-16T13:46:38.656567+0000","last_became_active":"2023-10-16T13:46:25.568456+0000","last_became_peered":"2023-10-16T13:46:25.568456+0000","last_unstale":"2023-10-16T13:46:38.656567+0000","last_undegraded":"2023-10-16T13:46:38.656567+0000","last_fullsized":"2023-10-16T13:46:38.656567+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:26:56.181979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.2a","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.670583+0000","last_change":"2023-10-16T13:46:38.656382+0000","last_active":"2023-10-16T13:46:39.670583+0000","last_peered":"2023-10-16T13:46:39.670583+0000","last_clean":"2023-10-16T13:46:39.670583+0000","last_became_active":"2023-10-16T13:46:38.655682+0000","last_became_peered":"2023-10-16T13:46:38.655682+0000","last_unstale":"2023-10-16T13:46:39.670583+0000","last_undegraded":"2023-10-16T13:46:39.670583+0000","last_fullsized":"2023-10-16T13:46:39.670583+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:14:58.526615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.28","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.640479+0000","last_change":"2023-10-16T13:46:25.583046+0000","last_active":"2023-10-16T13:46:38.640479+0000","last_peered":"2023-10-16T13:46:38.640479+0000","last_clean":"2023-10-16T13:46:38.640479+0000","last_became_active":"2023-10-16T13:46:25.582711+0000","last_became_peered":"2023-10-16T13:46:25.582711+0000","last_unstale":"2023-10-16T13:46:38.640479+0000","last_undegraded":"2023-10-16T13:46:38.640479+0000","last_fullsized":"2023-10-16T13:46:38.640479+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:23:43.446191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.2b","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.581119+0000","last_change":"2023-10-16T13:46:38.663737+0000","last_active":"2023-10-16T13:46:40.581119+0000","last_peered":"2023-10-16T13:46:40.581119+0000","last_clean":"2023-10-16T13:46:40.581119+0000","last_became_active":"2023-10-16T13:46:38.663429+0000","last_became_peered":"2023-10-16T13:46:38.663429+0000","last_unstale":"2023-10-16T13:46:40.581119+0000","last_undegraded":"2023-10-16T13:46:40.581119+0000","last_fullsized":"2023-10-16T13:46:40.581119+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:10:59.149373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.29","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.658035+0000","last_change":"2023-10-16T13:46:25.578743+0000","last_active":"2023-10-16T13:46:38.658035+0000","last_peered":"2023-10-16T13:46:38.658035+0000","last_clean":"2023-10-16T13:46:38.658035+0000","last_became_active":"2023-10-16T13:46:25.578447+0000","last_became_peered":"2023-10-16T13:46:25.578447+0000","last_unstale":"2023-10-16T13:46:38.658035+0000","last_undegraded":"2023-10-16T13:46:38.658035+0000","last_fullsized":"2023-10-16T13:46:38.658035+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:46:55.400521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.28","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.654324+0000","last_change":"2023-10-16T13:46:38.665539+0000","last_active":"2023-10-16T13:46:39.654324+0000","last_peered":"2023-10-16T13:46:39.654324+0000","last_clean":"2023-10-16T13:46:39.654324+0000","last_became_active":"2023-10-16T13:46:38.662338+0000","last_became_peered":"2023-10-16T13:46:38.662338+0000","last_unstale":"2023-10-16T13:46:39.654324+0000","last_undegraded":"2023-10-16T13:46:39.654324+0000","last_fullsized":"2023-10-16T13:46:39.654324+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:08:41.090480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.2a","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.650013+0000","last_change":"2023-10-16T13:46:25.581238+0000","last_active":"2023-10-16T13:46:38.650013+0000","last_peered":"2023-10-16T13:46:38.650013+0000","last_clean":"2023-10-16T13:46:38.650013+0000","last_became_active":"2023-10-16T13:46:25.580555+0000","last_became_peered":"2023-10-16T13:46:25.580555+0000","last_unstale":"2023-10-16T13:46:38.650013+0000","last_undegraded":"2023-10-16T13:46:38.650013+0000","last_fullsized":"2023-10-16T13:46:38.650013+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:54:51.781852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.29","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.866840+0000","last_change":"2023-10-16T13:46:38.663142+0000","last_active":"2023-10-16T13:46:40.866840+0000","last_peered":"2023-10-16T13:46:40.866840+0000","last_clean":"2023-10-16T13:46:40.866840+0000","last_became_active":"2023-10-16T13:46:38.662849+0000","last_became_peered":"2023-10-16T13:46:38.662849+0000","last_unstale":"2023-10-16T13:46:40.866840+0000","last_undegraded":"2023-10-16T13:46:40.866840+0000","last_fullsized":"2023-10-16T13:46:40.866840+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:58:13.297662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.2b","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.647028+0000","last_change":"2023-10-16T13:46:25.584056+0000","last_active":"2023-10-16T13:46:38.647028+0000","last_peered":"2023-10-16T13:46:38.647028+0000","last_clean":"2023-10-16T13:46:38.647028+0000","last_became_active":"2023-10-16T13:46:25.583758+0000","last_became_peered":"2023-10-16T13:46:25.583758+0000","last_unstale":"2023-10-16T13:46:38.647028+0000","last_undegraded":"2023-10-16T13:46:38.647028+0000","last_fullsized":"2023-10-16T13:46:38.647028+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:48:45.889362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2e","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.581648+0000","last_change":"2023-10-16T13:46:38.670081+0000","last_active":"2023-10-16T13:46:40.581648+0000","last_peered":"2023-10-16T13:46:40.581648+0000","last_clean":"2023-10-16T13:46:40.581648+0000","last_became_active":"2023-10-16T13:46:38.665403+0000","last_became_peered":"2023-10-16T13:46:38.665403+0000","last_unstale":"2023-10-16T13:46:40.581648+0000","last_undegraded":"2023-10-16T13:46:40.581648+0000","last_fullsized":"2023-10-16T13:46:40.581648+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:20:35.300682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.2c","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.649945+0000","last_change":"2023-10-16T13:46:25.572889+0000","last_active":"2023-10-16T13:46:38.649945+0000","last_peered":"2023-10-16T13:46:38.649945+0000","last_clean":"2023-10-16T13:46:38.649945+0000","last_became_active":"2023-10-16T13:46:25.572611+0000","last_became_peered":"2023-10-16T13:46:25.572611+0000","last_unstale":"2023-10-16T13:46:38.649945+0000","last_undegraded":"2023-10-16T13:46:38.649945+0000","last_fullsized":"2023-10-16T13:46:38.649945+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:57:59.993900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2f","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.098523+0000","last_change":"2023-10-16T13:46:38.669880+0000","last_active":"2023-10-16T13:46:40.098523+0000","last_peered":"2023-10-16T13:46:40.098523+0000","last_clean":"2023-10-16T13:46:40.098523+0000","last_became_active":"2023-10-16T13:46:38.662277+0000","last_became_peered":"2023-10-16T13:46:38.662277+0000","last_unstale":"2023-10-16T13:46:40.098523+0000","last_undegraded":"2023-10-16T13:46:40.098523+0000","last_fullsized":"2023-10-16T13:46:40.098523+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:01:09.625064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2d","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.648084+0000","last_change":"2023-10-16T13:46:25.578104+0000","last_active":"2023-10-16T13:46:38.648084+0000","last_peered":"2023-10-16T13:46:38.648084+0000","last_clean":"2023-10-16T13:46:38.648084+0000","last_became_active":"2023-10-16T13:46:25.577833+0000","last_became_peered":"2023-10-16T13:46:25.577833+0000","last_unstale":"2023-10-16T13:46:38.648084+0000","last_undegraded":"2023-10-16T13:46:38.648084+0000","last_fullsized":"2023-10-16T13:46:38.648084+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:43:18.855055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2c","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.930376+0000","last_change":"2023-10-16T13:46:38.660311+0000","last_active":"2023-10-16T13:46:39.930376+0000","last_peered":"2023-10-16T13:46:39.930376+0000","last_clean":"2023-10-16T13:46:39.930376+0000","last_became_active":"2023-10-16T13:46:38.660007+0000","last_became_peered":"2023-10-16T13:46:38.660007+0000","last_unstale":"2023-10-16T13:46:39.930376+0000","last_undegraded":"2023-10-16T13:46:39.930376+0000","last_fullsized":"2023-10-16T13:46:39.930376+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:44:59.606324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.2e","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.648186+0000","last_change":"2023-10-16T13:46:25.567084+0000","last_active":"2023-10-16T13:46:38.648186+0000","last_peered":"2023-10-16T13:46:38.648186+0000","last_clean":"2023-10-16T13:46:38.648186+0000","last_became_active":"2023-10-16T13:46:25.566470+0000","last_became_peered":"2023-10-16T13:46:25.566470+0000","last_unstale":"2023-10-16T13:46:38.648186+0000","last_undegraded":"2023-10-16T13:46:38.648186+0000","last_fullsized":"2023-10-16T13:46:38.648186+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:47:01.254594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.2d","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.655515+0000","last_change":"2023-10-16T13:46:38.660973+0000","last_active":"2023-10-16T13:46:39.655515+0000","last_peered":"2023-10-16T13:46:39.655515+0000","last_clean":"2023-10-16T13:46:39.655515+0000","last_became_active":"2023-10-16T13:46:38.660638+0000","last_became_peered":"2023-10-16T13:46:38.660638+0000","last_unstale":"2023-10-16T13:46:39.655515+0000","last_undegraded":"2023-10-16T13:46:39.655515+0000","last_fullsized":"2023-10-16T13:46:39.655515+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:21:02.259177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.2f","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.656469+0000","last_change":"2023-10-16T13:46:25.577983+0000","last_active":"2023-10-16T13:46:38.656469+0000","last_peered":"2023-10-16T13:46:38.656469+0000","last_clean":"2023-10-16T13:46:38.656469+0000","last_became_active":"2023-10-16T13:46:25.577680+0000","last_became_peered":"2023-10-16T13:46:25.577680+0000","last_unstale":"2023-10-16T13:46:38.656469+0000","last_undegraded":"2023-10-16T13:46:38.656469+0000","last_fullsized":"2023-10-16T13:46:38.656469+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:04:14.478904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.32","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.097945+0000","last_change":"2023-10-16T13:46:38.671012+0000","last_active":"2023-10-16T13:46:40.097945+0000","last_peered":"2023-10-16T13:46:40.097945+0000","last_clean":"2023-10-16T13:46:40.097945+0000","last_became_active":"2023-10-16T13:46:38.663063+0000","last_became_peered":"2023-10-16T13:46:38.663063+0000","last_unstale":"2023-10-16T13:46:40.097945+0000","last_undegraded":"2023-10-16T13:46:40.097945+0000","last_fullsized":"2023-10-16T13:46:40.097945+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:26:39.652821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.30","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.653196+0000","last_change":"2023-10-16T13:46:25.570144+0000","last_active":"2023-10-16T13:46:38.653196+0000","last_peered":"2023-10-16T13:46:38.653196+0000","last_clean":"2023-10-16T13:46:38.653196+0000","last_became_active":"2023-10-16T13:46:25.569866+0000","last_became_peered":"2023-10-16T13:46:25.569866+0000","last_unstale":"2023-10-16T13:46:38.653196+0000","last_undegraded":"2023-10-16T13:46:38.653196+0000","last_fullsized":"2023-10-16T13:46:38.653196+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:53:48.001931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.33","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.929922+0000","last_change":"2023-10-16T13:46:38.656671+0000","last_active":"2023-10-16T13:46:39.929922+0000","last_peered":"2023-10-16T13:46:39.929922+0000","last_clean":"2023-10-16T13:46:39.929922+0000","last_became_active":"2023-10-16T13:46:38.656361+0000","last_became_peered":"2023-10-16T13:46:38.656361+0000","last_unstale":"2023-10-16T13:46:39.929922+0000","last_undegraded":"2023-10-16T13:46:39.929922+0000","last_fullsized":"2023-10-16T13:46:39.929922+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:26:36.610920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.31","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.641759+0000","last_change":"2023-10-16T13:46:25.577384+0000","last_active":"2023-10-16T13:46:38.641759+0000","last_peered":"2023-10-16T13:46:38.641759+0000","last_clean":"2023-10-16T13:46:38.641759+0000","last_became_active":"2023-10-16T13:46:25.577070+0000","last_became_peered":"2023-10-16T13:46:25.577070+0000","last_unstale":"2023-10-16T13:46:38.641759+0000","last_undegraded":"2023-10-16T13:46:38.641759+0000","last_fullsized":"2023-10-16T13:46:38.641759+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:36:08.745864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.30","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.929480+0000","last_change":"2023-10-16T13:46:38.655085+0000","last_active":"2023-10-16T13:46:39.929480+0000","last_peered":"2023-10-16T13:46:39.929480+0000","last_clean":"2023-10-16T13:46:39.929480+0000","last_became_active":"2023-10-16T13:46:38.654776+0000","last_became_peered":"2023-10-16T13:46:38.654776+0000","last_unstale":"2023-10-16T13:46:39.929480+0000","last_undegraded":"2023-10-16T13:46:39.929480+0000","last_fullsized":"2023-10-16T13:46:39.929480+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:27:09.485741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.32","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.653492+0000","last_change":"2023-10-16T13:46:25.581421+0000","last_active":"2023-10-16T13:46:38.653492+0000","last_peered":"2023-10-16T13:46:38.653492+0000","last_clean":"2023-10-16T13:46:38.653492+0000","last_became_active":"2023-10-16T13:46:25.580936+0000","last_became_peered":"2023-10-16T13:46:25.580936+0000","last_unstale":"2023-10-16T13:46:38.653492+0000","last_undegraded":"2023-10-16T13:46:38.653492+0000","last_fullsized":"2023-10-16T13:46:38.653492+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:41:42.462761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.31","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.670511+0000","last_change":"2023-10-16T13:46:38.668886+0000","last_active":"2023-10-16T13:46:39.670511+0000","last_peered":"2023-10-16T13:46:39.670511+0000","last_clean":"2023-10-16T13:46:39.670511+0000","last_became_active":"2023-10-16T13:46:38.668431+0000","last_became_peered":"2023-10-16T13:46:38.668431+0000","last_unstale":"2023-10-16T13:46:39.670511+0000","last_undegraded":"2023-10-16T13:46:39.670511+0000","last_fullsized":"2023-10-16T13:46:39.670511+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:47:08.704282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.33","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.659118+0000","last_change":"2023-10-16T13:46:25.576474+0000","last_active":"2023-10-16T13:46:38.659118+0000","last_peered":"2023-10-16T13:46:38.659118+0000","last_clean":"2023-10-16T13:46:38.659118+0000","last_became_active":"2023-10-16T13:46:25.576187+0000","last_became_peered":"2023-10-16T13:46:25.576187+0000","last_unstale":"2023-10-16T13:46:38.659118+0000","last_undegraded":"2023-10-16T13:46:38.659118+0000","last_fullsized":"2023-10-16T13:46:38.659118+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:39:18.796503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.36","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.581479+0000","last_change":"2023-10-16T13:46:38.664098+0000","last_active":"2023-10-16T13:46:40.581479+0000","last_peered":"2023-10-16T13:46:40.581479+0000","last_clean":"2023-10-16T13:46:40.581479+0000","last_became_active":"2023-10-16T13:46:38.663906+0000","last_became_peered":"2023-10-16T13:46:38.663906+0000","last_unstale":"2023-10-16T13:46:40.581479+0000","last_undegraded":"2023-10-16T13:46:40.581479+0000","last_fullsized":"2023-10-16T13:46:40.581479+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:20:32.747440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.34","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.641276+0000","last_change":"2023-10-16T13:46:25.575302+0000","last_active":"2023-10-16T13:46:38.641276+0000","last_peered":"2023-10-16T13:46:38.641276+0000","last_clean":"2023-10-16T13:46:38.641276+0000","last_became_active":"2023-10-16T13:46:25.575009+0000","last_became_peered":"2023-10-16T13:46:25.575009+0000","last_unstale":"2023-10-16T13:46:38.641276+0000","last_undegraded":"2023-10-16T13:46:38.641276+0000","last_fullsized":"2023-10-16T13:46:38.641276+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:16:03.750659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.37","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.867309+0000","last_change":"2023-10-16T13:46:38.663230+0000","last_active":"2023-10-16T13:46:40.867309+0000","last_peered":"2023-10-16T13:46:40.867309+0000","last_clean":"2023-10-16T13:46:40.867309+0000","last_became_active":"2023-10-16T13:46:38.662861+0000","last_became_peered":"2023-10-16T13:46:38.662861+0000","last_unstale":"2023-10-16T13:46:40.867309+0000","last_undegraded":"2023-10-16T13:46:40.867309+0000","last_fullsized":"2023-10-16T13:46:40.867309+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:30:22.969323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.35","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.653449+0000","last_change":"2023-10-16T13:46:25.578710+0000","last_active":"2023-10-16T13:46:38.653449+0000","last_peered":"2023-10-16T13:46:38.653449+0000","last_clean":"2023-10-16T13:46:38.653449+0000","last_became_active":"2023-10-16T13:46:25.578250+0000","last_became_peered":"2023-10-16T13:46:25.578250+0000","last_unstale":"2023-10-16T13:46:38.653449+0000","last_undegraded":"2023-10-16T13:46:38.653449+0000","last_fullsized":"2023-10-16T13:46:38.653449+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:18:48.273122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.34","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.098636+0000","last_change":"2023-10-16T13:46:38.663016+0000","last_active":"2023-10-16T13:46:40.098636+0000","last_peered":"2023-10-16T13:46:40.098636+0000","last_clean":"2023-10-16T13:46:40.098636+0000","last_became_active":"2023-10-16T13:46:38.662722+0000","last_became_peered":"2023-10-16T13:46:38.662722+0000","last_unstale":"2023-10-16T13:46:40.098636+0000","last_undegraded":"2023-10-16T13:46:40.098636+0000","last_fullsized":"2023-10-16T13:46:40.098636+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:19:18.474581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.36","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.646533+0000","last_change":"2023-10-16T13:46:25.585253+0000","last_active":"2023-10-16T13:46:38.646533+0000","last_peered":"2023-10-16T13:46:38.646533+0000","last_clean":"2023-10-16T13:46:38.646533+0000","last_became_active":"2023-10-16T13:46:25.584881+0000","last_became_peered":"2023-10-16T13:46:25.584881+0000","last_unstale":"2023-10-16T13:46:38.646533+0000","last_undegraded":"2023-10-16T13:46:38.646533+0000","last_fullsized":"2023-10-16T13:46:38.646533+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:16:15.086261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.35","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.673378+0000","last_change":"2023-10-16T13:46:38.661044+0000","last_active":"2023-10-16T13:46:39.673378+0000","last_peered":"2023-10-16T13:46:39.673378+0000","last_clean":"2023-10-16T13:46:39.673378+0000","last_became_active":"2023-10-16T13:46:38.660733+0000","last_became_peered":"2023-10-16T13:46:38.660733+0000","last_unstale":"2023-10-16T13:46:39.673378+0000","last_undegraded":"2023-10-16T13:46:39.673378+0000","last_fullsized":"2023-10-16T13:46:39.673378+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:12:08.292404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.37","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.656374+0000","last_change":"2023-10-16T13:46:25.583298+0000","last_active":"2023-10-16T13:46:38.656374+0000","last_peered":"2023-10-16T13:46:38.656374+0000","last_clean":"2023-10-16T13:46:38.656374+0000","last_became_active":"2023-10-16T13:46:25.582794+0000","last_became_peered":"2023-10-16T13:46:25.582794+0000","last_unstale":"2023-10-16T13:46:38.656374+0000","last_undegraded":"2023-10-16T13:46:38.656374+0000","last_fullsized":"2023-10-16T13:46:38.656374+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:22:31.745564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.3a","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.930318+0000","last_change":"2023-10-16T13:46:38.657295+0000","last_active":"2023-10-16T13:46:39.930318+0000","last_peered":"2023-10-16T13:46:39.930318+0000","last_clean":"2023-10-16T13:46:39.930318+0000","last_became_active":"2023-10-16T13:46:38.656993+0000","last_became_peered":"2023-10-16T13:46:38.656993+0000","last_unstale":"2023-10-16T13:46:39.930318+0000","last_undegraded":"2023-10-16T13:46:39.930318+0000","last_fullsized":"2023-10-16T13:46:39.930318+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:26:05.871114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.38","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.647306+0000","last_change":"2023-10-16T13:46:25.576391+0000","last_active":"2023-10-16T13:46:38.647306+0000","last_peered":"2023-10-16T13:46:38.647306+0000","last_clean":"2023-10-16T13:46:38.647306+0000","last_became_active":"2023-10-16T13:46:25.575906+0000","last_became_peered":"2023-10-16T13:46:25.575906+0000","last_unstale":"2023-10-16T13:46:38.647306+0000","last_undegraded":"2023-10-16T13:46:38.647306+0000","last_fullsized":"2023-10-16T13:46:38.647306+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:44:11.618820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3b","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.655127+0000","last_change":"2023-10-16T13:46:38.668670+0000","last_active":"2023-10-16T13:46:39.655127+0000","last_peered":"2023-10-16T13:46:39.655127+0000","last_clean":"2023-10-16T13:46:39.655127+0000","last_became_active":"2023-10-16T13:46:38.667596+0000","last_became_peered":"2023-10-16T13:46:38.667596+0000","last_unstale":"2023-10-16T13:46:39.655127+0000","last_undegraded":"2023-10-16T13:46:39.655127+0000","last_fullsized":"2023-10-16T13:46:39.655127+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:44:54.832181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.39","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.641719+0000","last_change":"2023-10-16T13:46:25.573056+0000","last_active":"2023-10-16T13:46:38.641719+0000","last_peered":"2023-10-16T13:46:38.641719+0000","last_clean":"2023-10-16T13:46:38.641719+0000","last_became_active":"2023-10-16T13:46:25.572702+0000","last_became_peered":"2023-10-16T13:46:25.572702+0000","last_unstale":"2023-10-16T13:46:38.641719+0000","last_undegraded":"2023-10-16T13:46:38.641719+0000","last_fullsized":"2023-10-16T13:46:38.641719+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:05:54.303051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.38","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.929717+0000","last_change":"2023-10-16T13:46:38.657491+0000","last_active":"2023-10-16T13:46:39.929717+0000","last_peered":"2023-10-16T13:46:39.929717+0000","last_clean":"2023-10-16T13:46:39.929717+0000","last_became_active":"2023-10-16T13:46:38.656959+0000","last_became_peered":"2023-10-16T13:46:38.656959+0000","last_unstale":"2023-10-16T13:46:39.929717+0000","last_undegraded":"2023-10-16T13:46:39.929717+0000","last_fullsized":"2023-10-16T13:46:39.929717+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:57:26.352089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3a","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.651707+0000","last_change":"2023-10-16T13:46:25.568946+0000","last_active":"2023-10-16T13:46:38.651707+0000","last_peered":"2023-10-16T13:46:38.651707+0000","last_clean":"2023-10-16T13:46:38.651707+0000","last_became_active":"2023-10-16T13:46:25.568652+0000","last_became_peered":"2023-10-16T13:46:25.568652+0000","last_unstale":"2023-10-16T13:46:38.651707+0000","last_undegraded":"2023-10-16T13:46:38.651707+0000","last_fullsized":"2023-10-16T13:46:38.651707+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:18:48.273122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.39","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.580943+0000","last_change":"2023-10-16T13:46:38.661321+0000","last_active":"2023-10-16T13:46:40.580943+0000","last_peered":"2023-10-16T13:46:40.580943+0000","last_clean":"2023-10-16T13:46:40.580943+0000","last_became_active":"2023-10-16T13:46:38.660892+0000","last_became_peered":"2023-10-16T13:46:38.660892+0000","last_unstale":"2023-10-16T13:46:40.580943+0000","last_undegraded":"2023-10-16T13:46:40.580943+0000","last_fullsized":"2023-10-16T13:46:40.580943+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:18:49.534924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.3b","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.641227+0000","last_change":"2023-10-16T13:46:25.570352+0000","last_active":"2023-10-16T13:46:38.641227+0000","last_peered":"2023-10-16T13:46:38.641227+0000","last_clean":"2023-10-16T13:46:38.641227+0000","last_became_active":"2023-10-16T13:46:25.570035+0000","last_became_peered":"2023-10-16T13:46:25.570035+0000","last_unstale":"2023-10-16T13:46:38.641227+0000","last_undegraded":"2023-10-16T13:46:38.641227+0000","last_fullsized":"2023-10-16T13:46:38.641227+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:43:07.323286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.3e","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.654292+0000","last_change":"2023-10-16T13:46:38.667376+0000","last_active":"2023-10-16T13:46:39.654292+0000","last_peered":"2023-10-16T13:46:39.654292+0000","last_clean":"2023-10-16T13:46:39.654292+0000","last_became_active":"2023-10-16T13:46:38.667037+0000","last_became_peered":"2023-10-16T13:46:38.667037+0000","last_unstale":"2023-10-16T13:46:39.654292+0000","last_undegraded":"2023-10-16T13:46:39.654292+0000","last_fullsized":"2023-10-16T13:46:39.654292+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:36:17.135906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.3c","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.647005+0000","last_change":"2023-10-16T13:46:25.578245+0000","last_active":"2023-10-16T13:46:38.647005+0000","last_peered":"2023-10-16T13:46:38.647005+0000","last_clean":"2023-10-16T13:46:38.647005+0000","last_became_active":"2023-10-16T13:46:25.577671+0000","last_became_peered":"2023-10-16T13:46:25.577671+0000","last_unstale":"2023-10-16T13:46:38.647005+0000","last_undegraded":"2023-10-16T13:46:38.647005+0000","last_fullsized":"2023-10-16T13:46:38.647005+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:47:01.254594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.3f","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.655304+0000","last_change":"2023-10-16T13:46:38.660190+0000","last_active":"2023-10-16T13:46:39.655304+0000","last_peered":"2023-10-16T13:46:39.655304+0000","last_clean":"2023-10-16T13:46:39.655304+0000","last_became_active":"2023-10-16T13:46:38.659872+0000","last_became_peered":"2023-10-16T13:46:38.659872+0000","last_unstale":"2023-10-16T13:46:39.655304+0000","last_undegraded":"2023-10-16T13:46:39.655304+0000","last_fullsized":"2023-10-16T13:46:39.655304+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:19:06.117364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.3d","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.653204+0000","last_change":"2023-10-16T13:46:25.575243+0000","last_active":"2023-10-16T13:46:38.653204+0000","last_peered":"2023-10-16T13:46:38.653204+0000","last_clean":"2023-10-16T13:46:38.653204+0000","last_became_active":"2023-10-16T13:46:25.574483+0000","last_became_peered":"2023-10-16T13:46:25.574483+0000","last_unstale":"2023-10-16T13:46:38.653204+0000","last_undegraded":"2023-10-16T13:46:38.653204+0000","last_fullsized":"2023-10-16T13:46:38.653204+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:21:32.260993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.3c","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:39.673308+0000","last_change":"2023-10-16T13:46:38.654365+0000","last_active":"2023-10-16T13:46:39.673308+0000","last_peered":"2023-10-16T13:46:39.673308+0000","last_clean":"2023-10-16T13:46:39.673308+0000","last_became_active":"2023-10-16T13:46:38.654040+0000","last_became_peered":"2023-10-16T13:46:38.654040+0000","last_unstale":"2023-10-16T13:46:39.673308+0000","last_undegraded":"2023-10-16T13:46:39.673308+0000","last_fullsized":"2023-10-16T13:46:39.673308+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:40:23.299174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.3e","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.657356+0000","last_change":"2023-10-16T13:46:25.570175+0000","last_active":"2023-10-16T13:46:38.657356+0000","last_peered":"2023-10-16T13:46:38.657356+0000","last_clean":"2023-10-16T13:46:38.657356+0000","last_became_active":"2023-10-16T13:46:25.569413+0000","last_became_peered":"2023-10-16T13:46:25.569413+0000","last_unstale":"2023-10-16T13:46:38.657356+0000","last_undegraded":"2023-10-16T13:46:38.657356+0000","last_fullsized":"2023-10-16T13:46:38.657356+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:20:50.714013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.3d","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-16T13:46:40.581412+0000","last_change":"2023-10-16T13:46:38.667427+0000","last_active":"2023-10-16T13:46:40.581412+0000","last_peered":"2023-10-16T13:46:40.581412+0000","last_clean":"2023-10-16T13:46:40.581412+0000","last_became_active":"2023-10-16T13:46:38.667091+0000","last_became_peered":"2023-10-16T13:46:38.667091+0000","last_unstale":"2023-10-16T13:46:40.581412+0000","last_undegraded":"2023-10-16T13:46:40.581412+0000","last_fullsized":"2023-10-16T13:46:40.581412+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:37.622310+0000","last_clean_scrub_stamp":"2023-10-16T13:46:37.622310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:22:55.026877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.3f","version":"0'0","reported_seq":15,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-16T13:46:38.647146+0000","last_change":"2023-10-16T13:46:25.582893+0000","last_active":"2023-10-16T13:46:38.647146+0000","last_peered":"2023-10-16T13:46:38.647146+0000","last_clean":"2023-10-16T13:46:38.647146+0000","last_became_active":"2023-10-16T13:46:25.581980+0000","last_became_peered":"2023-10-16T13:46:25.581980+0000","last_unstale":"2023-10-16T13:46:38.647146+0000","last_undegraded":"2023-10-16T13:46:38.647146+0000","last_fullsized":"2023-10-16T13:46:38.647146+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:46:24.318548+0000","last_clean_scrub_stamp":"2023-10-16T13:46:24.318548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:04:47.313972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":30,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":28,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":27,"num_pg":32,"stat_sum":{"num_bytes":3657,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":29,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":348,"ondisk_log_size":348,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575989,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56804,"kb_used_data":3412,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315036,"statfs":{"total":96636764160,"available":96578596864,"internally_reserved":0,"allocated":3493888,"data_stored":2077443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54655435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:45:48 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.61499999999999999,"15min":0.63800000000000001},"min":{"1min":0.35499999999999998,"5min":0.34899999999999998,"15min":0.33000000000000002},"max":{"1min":0.81899999999999995,"5min":0.87,"15min":0.96199999999999997},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.61799999999999999,"15min":0.63300000000000001},"min":{"1min":0.39200000000000002,"5min":0.38900000000000001,"15min":0.379},"max":{"1min":0.83899999999999997,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.72099999999999997}]},{"osd":1,"last update":"Mon Oct 16 13:45:48 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.61399999999999999,"15min":0.64400000000000002},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.79000000000000004,"5min":0.84599999999999997,"15min":0.95899999999999996},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.621,"5min":0.63100000000000001,"15min":0.63800000000000001},"min":{"1min":0.48199999999999998,"5min":0.375,"15min":0.375},"max":{"1min":0.77000000000000002,"5min":0.79100000000000004,"15min":1.022},"last":0.64800000000000002}]},{"osd":2,"last update":"Mon Oct 16 13:45:48 2023","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.69399999999999995,"15min":0.69499999999999995},"min":{"1min":0.48999999999999999,"5min":0.436,"15min":0.436},"max":{"1min":0.83499999999999996,"5min":0.83899999999999997,"15min":0.93500000000000005},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.69599999999999995,"15min":0.70199999999999996},"min":{"1min":0.51000000000000001,"5min":0.47599999999999998,"15min":0.39400000000000002},"max":{"1min":0.871,"5min":0.92800000000000005,"15min":0.93799999999999994},"last":0.83699999999999997}]},{"osd":3,"last update":"Mon Oct 16 13:45:48 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.65000000000000002,"15min":0.66200000000000003},"min":{"1min":0.45200000000000001,"5min":0.438,"15min":0.438},"max":{"1min":0.81499999999999995,"5min":0.89200000000000002,"15min":0.91300000000000003},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.64600000000000002,"15min":0.66300000000000003},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.80400000000000005,"5min":1.24,"15min":1.24},"last":0.80500000000000005}]},{"osd":4,"last update":"Mon Oct 16 13:45:48 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.51300000000000001,"15min":0.52400000000000002},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.72699999999999998,"5min":0.745,"15min":0.86099999999999999},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.623,"5min":0.61399999999999999,"15min":0.61899999999999999},"min":{"1min":0.40699999999999997,"5min":0.36799999999999999,"15min":0.33100000000000002},"max":{"1min":0.77700000000000002,"5min":0.90400000000000003,"15min":0.92900000000000005},"last":0.88500000000000001}]},{"osd":5,"last update":"Mon Oct 16 13:45:48 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.53100000000000003,"15min":0.54400000000000004},"min":{"1min":0.41699999999999998,"5min":0.33400000000000002,"15min":0.32500000000000001},"max":{"1min":0.71599999999999997,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.57299999999999995,"15min":0.58799999999999997},"min":{"1min":0.41199999999999998,"5min":0.38400000000000001,"15min":0.35099999999999998},"max":{"1min":0.80100000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.79200000000000004}]},{"osd":6,"last update":"Mon Oct 16 13:45:48 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.59999999999999998,"15min":0.61499999999999999},"min":{"1min":0.47399999999999998,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.79500000000000004,"5min":0.874,"15min":0.95899999999999996},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.61299999999999999,"15min":0.62},"min":{"1min":0.41099999999999998,"5min":0.378,"15min":0.35199999999999998},"max":{"1min":0.81299999999999994,"5min":0.95799999999999996,"15min":0.97499999999999998},"last":0.92100000000000004}]}]},{"osd":6,"up_from":13,"seq":55834575987,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42348,"kb_used_data":2848,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329492,"statfs":{"total":96636764160,"available":96593399808,"internally_reserved":0,"allocated":2916352,"data_stored":1488653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:46:36 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.65300000000000002,"15min":0.66800000000000004},"min":{"1min":0.41499999999999998,"5min":0.375,"15min":0.28699999999999998},"max":{"1min":0.82599999999999996,"5min":0.89400000000000002,"15min":0.92300000000000004},"last":0.95199999999999996},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.64500000000000002,"15min":0.66400000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.80400000000000005,"5min":0.91400000000000003,"15min":0.93799999999999994},"last":0.81100000000000005}]},{"osd":1,"last update":"Mon Oct 16 13:46:36 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.58599999999999997,"15min":0.624},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.33100000000000002},"max":{"1min":0.70299999999999996,"5min":0.86899999999999999,"15min":0.90900000000000003},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.63,"5min":0.61399999999999999,"15min":0.62},"min":{"1min":0.41699999999999998,"5min":0.39800000000000002,"15min":0.38300000000000001},"max":{"1min":0.85799999999999998,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.90800000000000003}]},{"osd":2,"last update":"Mon Oct 16 13:46:36 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.622,"15min":0.63300000000000001},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.85799999999999998,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.59999999999999998,"15min":0.624},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.749,"5min":0.89000000000000001,"15min":0.91000000000000003},"last":0.76400000000000001}]},{"osd":3,"last update":"Mon Oct 16 13:46:36 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63900000000000001,"15min":0.66300000000000003},"min":{"1min":0.439,"5min":0.39900000000000002,"15min":0.38200000000000001},"max":{"1min":0.874,"5min":0.92400000000000004,"15min":0.95599999999999996},"last":0.753},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.69099999999999995,"15min":0.71599999999999997},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.42299999999999999},"max":{"1min":0.85099999999999998,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.85199999999999998}]},{"osd":4,"last update":"Mon Oct 16 13:46:36 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.51800000000000002,"15min":0.52700000000000002},"min":{"1min":0.39000000000000001,"5min":0.35999999999999999,"15min":0.315},"max":{"1min":0.78000000000000003,"5min":0.85599999999999998,"15min":0.95799999999999996},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61699999999999999,"15min":0.62},"min":{"1min":0.375,"5min":0.35499999999999998,"15min":0.32900000000000001},"max":{"1min":0.83699999999999997,"5min":0.98799999999999999,"15min":1.0309999999999999},"last":0.622}]},{"osd":5,"last update":"Mon Oct 16 13:46:36 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.59799999999999998,"15min":0.59399999999999997},"min":{"1min":0.39800000000000002,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.81100000000000005,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.66500000000000004,"15min":0.66000000000000003},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.308},"max":{"1min":0.85899999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.64200000000000002}]},{"osd":7,"last update":"Mon Oct 16 13:46:36 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.60899999999999999,"15min":0.60499999999999998},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.80000000000000004,"5min":1.081,"15min":1.081},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65200000000000002},"min":{"1min":0.42799999999999999,"5min":0.41899999999999998,"15min":0.375},"max":{"1min":0.91500000000000004,"5min":1.099,"15min":1.099},"last":0.56599999999999995}]}]},{"osd":1,"up_from":13,"seq":55834575987,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47880,"kb_used_data":2808,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323960,"statfs":{"total":96636764160,"available":96587735040,"internally_reserved":0,"allocated":2875392,"data_stored":1483474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":46135753},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:45:54 2023","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.40100000000000002,"15min":0.40999999999999998},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.71999999999999997},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.497,"5min":0.496,"15min":0.499},"min":{"1min":0.34599999999999997,"5min":0.253,"15min":0.253},"max":{"1min":0.68000000000000005,"5min":0.70999999999999996,"15min":0.77200000000000002},"last":0.41699999999999998}]},{"osd":2,"last update":"Mon Oct 16 13:45:54 2023","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.45500000000000002,"15min":0.45800000000000002},"min":{"1min":0.28000000000000003,"5min":0.254,"15min":0.254},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.48199999999999998,"15min":0.49299999999999999},"min":{"1min":0.315,"5min":0.315,"15min":0.30199999999999999},"max":{"1min":0.64200000000000002,"5min":0.66900000000000004,"15min":0.71399999999999997},"last":0.34799999999999998}]},{"osd":3,"last update":"Mon Oct 16 13:45:54 2023","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.438,"15min":0.43099999999999999},"min":{"1min":0.33900000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.57399999999999995,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.47299999999999998,"15min":0.47899999999999998},"min":{"1min":0.26700000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.64800000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.442}]},{"osd":4,"last update":"Mon Oct 16 13:45:54 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.66000000000000003,"15min":0.66700000000000004},"min":{"1min":0.52600000000000002,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.86099999999999999,"5min":0.95899999999999996,"15min":1.0069999999999999},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.66200000000000003,"15min":0.67200000000000004},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.747,"5min":0.91300000000000003,"15min":0.997},"last":0.60399999999999998}]},{"osd":5,"last update":"Mon Oct 16 13:45:54 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.69799999999999995,"15min":0.71399999999999997},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.92300000000000004,"5min":0.97499999999999998,"15min":1.0760000000000001},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.73199999999999998,"15min":0.73399999999999999},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.46200000000000002},"max":{"1min":0.90000000000000002,"5min":1.0469999999999999,"15min":1.0660000000000001},"last":0.621}]},{"osd":6,"last update":"Mon Oct 16 13:45:54 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.69099999999999995,"15min":0.68500000000000005},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.80100000000000005,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.70799999999999996,"15min":0.71299999999999997},"min":{"1min":0.499,"5min":0.499,"15min":0.47799999999999998},"max":{"1min":0.88100000000000001,"5min":1.03,"15min":1.083},"last":0.58499999999999996}]},{"osd":7,"last update":"Mon Oct 16 13:45:54 2023","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.73899999999999999,"15min":0.745},"min":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.48899999999999999},"max":{"1min":0.91400000000000003,"5min":1.0629999999999999,"15min":1.0660000000000001},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.751,"15min":0.751},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.92700000000000005,"5min":1.042,"15min":1.0580000000000001},"last":0.67300000000000004}]}]},{"osd":0,"up_from":13,"seq":55834575988,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30412,"kb_used_data":3392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341428,"statfs":{"total":96636764160,"available":96605622272,"internally_reserved":0,"allocated":3473408,"data_stored":2073932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 13:45:42 2023","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.40899999999999997,"15min":0.42399999999999999},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.58699999999999997,"5min":0.65800000000000003,"15min":0.71399999999999997},"last":0.31},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.42599999999999999,"15min":0.442},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.25900000000000001},"max":{"1min":0.58599999999999997,"5min":0.64600000000000002,"15min":0.82199999999999995},"last":0.36099999999999999}]},{"osd":2,"last update":"Mon Oct 16 13:45:42 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44500000000000001,"15min":0.45800000000000002},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.75700000000000001},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.46800000000000003,"15min":0.48699999999999999},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.83299999999999996},"last":0.41499999999999998}]},{"osd":3,"last update":"Mon Oct 16 13:45:42 2023","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.43099999999999999,"15min":0.45400000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.61799999999999999,"5min":0.63400000000000001,"15min":0.78600000000000003},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.51600000000000001,"15min":0.52500000000000002},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.70399999999999996,"5min":0.82999999999999996,"15min":0.86399999999999999},"last":0.44700000000000001}]},{"osd":4,"last update":"Mon Oct 16 13:45:42 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.66500000000000004,"15min":0.68600000000000005},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.96999999999999997,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.66300000000000003,"15min":0.69299999999999995},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":1.0049999999999999},"last":0.67700000000000005}]},{"osd":5,"last update":"Mon Oct 16 13:45:42 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.66700000000000004,"15min":0.69499999999999995},"min":{"1min":0.44500000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.95399999999999996,"5min":0.95999999999999996,"15min":1.0249999999999999},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.70599999999999996,"15min":0.72999999999999998},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.081,"5min":1.081,"15min":1.081},"last":0.55400000000000005}]},{"osd":6,"last update":"Mon Oct 16 13:45:42 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.71599999999999997,"15min":0.73499999999999999},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.034,"5min":1.103,"15min":1.103},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.70199999999999996,"15min":0.72499999999999998},"min":{"1min":0.42999999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.91000000000000003,"5min":1.0409999999999999,"15min":1.05},"last":0.71099999999999997}]},{"osd":7,"last update":"Mon Oct 16 13:45:42 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.71899999999999997,"15min":0.746},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.002,"5min":1.002,"15min":1.056},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.72099999999999997,"15min":0.73699999999999999},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.99199999999999999,"5min":1.087,"15min":1.087},"last":0.75900000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575988,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52040,"kb_used_data":2808,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319800,"statfs":{"total":96636764160,"available":96583475200,"internally_reserved":0,"allocated":2875392,"data_stored":1476565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.41499999999999998,"15min":0.40899999999999997},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.623,"5min":0.71699999999999997,"15min":0.77500000000000002},"last":0.316},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.47699999999999998,"15min":0.47999999999999998},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.28699999999999998}]},{"osd":1,"last update":"Mon Oct 16 13:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.502,"15min":0.5},"min":{"1min":0.317,"5min":0.311,"15min":0.311},"max":{"1min":0.66800000000000004,"5min":0.73199999999999998,"15min":0.81399999999999995},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.48999999999999999,"15min":0.47699999999999998},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.27700000000000002},"max":{"1min":0.69999999999999996,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.42399999999999999}]},{"osd":3,"last update":"Mon Oct 16 13:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.443,"15min":0.44},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.59699999999999998,"5min":0.70099999999999996,"15min":0.72599999999999998},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.54300000000000004,"15min":0.54400000000000004},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.82699999999999996,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.35599999999999998}]},{"osd":4,"last update":"Mon Oct 16 13:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.72699999999999998,"15min":0.71799999999999997},"min":{"1min":0.51500000000000001,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.89000000000000001,"5min":1.3600000000000001,"15min":1.3600000000000001},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.72599999999999998,"15min":0.71899999999999997},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.82899999999999996,"5min":1.403,"15min":1.403},"last":0.47999999999999998}]},{"osd":5,"last update":"Mon Oct 16 13:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.68600000000000005,"15min":0.68899999999999995},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.81100000000000005,"5min":1.486,"15min":1.486},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.71199999999999997,"15min":0.69999999999999996},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.90200000000000002,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":0.51600000000000001}]},{"osd":6,"last update":"Mon Oct 16 13:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.67600000000000005,"15min":0.66600000000000004},"min":{"1min":0.44500000000000001,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.80200000000000005,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.72599999999999998,"15min":0.72199999999999998},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.90600000000000003,"5min":1.4379999999999999,"15min":1.4379999999999999},"last":0.504}]},{"osd":7,"last update":"Mon Oct 16 13:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.73899999999999999,"15min":0.72099999999999997},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.94099999999999995,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.72999999999999998,"15min":0.73099999999999998},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.91500000000000004,"5min":1.377,"15min":1.377},"last":0.44600000000000001}]}]},{"osd":3,"up_from":13,"seq":55834575987,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56212,"kb_used_data":2824,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315628,"statfs":{"total":96636764160,"available":96579203072,"internally_reserved":0,"allocated":2891776,"data_stored":1483622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.40300000000000002,"15min":0.40999999999999998},"min":{"1min":0.23499999999999999,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.57199999999999995,"5min":0.64200000000000002,"15min":0.72899999999999998},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.41499999999999998,"15min":0.42799999999999999},"min":{"1min":0.27300000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.60099999999999998,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.376}]},{"osd":1,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.42699999999999999,"15min":0.434},"min":{"1min":0.26700000000000002,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.55900000000000005,"5min":0.68200000000000005,"15min":0.76300000000000001},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.435,"5min":0.45900000000000002,"15min":0.47999999999999998},"min":{"1min":0.27700000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.65500000000000003,"5min":0.70699999999999996,"15min":0.77800000000000002},"last":0.42899999999999999}]},{"osd":2,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.47999999999999998,"15min":0.49299999999999999},"min":{"1min":0.33800000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.621,"5min":1.167,"15min":1.167},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.501,"15min":0.52300000000000002},"min":{"1min":0.312,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.625,"5min":0.73299999999999998,"15min":0.78700000000000003},"last":0.54000000000000004}]},{"osd":4,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.65100000000000002,"15min":0.66500000000000004},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.91200000000000003,"5min":0.998,"15min":1.137},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.68899999999999995,"15min":0.69399999999999995},"min":{"1min":0.441,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.087,"5min":1.2110000000000001,"15min":1.2909999999999999},"last":0.65500000000000003}]},{"osd":5,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.67700000000000005,"15min":0.70499999999999996},"min":{"1min":0.45500000000000002,"5min":0.42899999999999999,"15min":0.374},"max":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.2729999999999999},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.69299999999999995,"15min":0.69999999999999996},"min":{"1min":0.45600000000000002,"5min":0.432,"15min":0.432},"max":{"1min":0.98799999999999999,"5min":1.0549999999999999,"15min":1.173},"last":0.68000000000000005}]},{"osd":6,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.72899999999999998,"15min":0.71899999999999997},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.41799999999999998},"max":{"1min":0.99099999999999999,"5min":5.0999999999999996,"15min":5.0999999999999996},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.70999999999999996,"15min":0.73399999999999999},"min":{"1min":0.49199999999999999,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.0029999999999999,"5min":1.0309999999999999,"15min":1.2290000000000001},"last":0.72899999999999998}]},{"osd":7,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.71899999999999997,"15min":0.73399999999999999},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.048,"5min":1.048,"15min":1.2569999999999999},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.73299999999999998,"15min":0.746},"min":{"1min":0.501,"5min":0.47799999999999998,"15min":0.43099999999999999},"max":{"1min":1.0369999999999999,"5min":1.262,"15min":1.262},"last":0.70799999999999996}]}]},{"osd":4,"up_from":13,"seq":55834575986,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52052,"kb_used_data":2816,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319788,"statfs":{"total":96636764160,"available":96583462912,"internally_reserved":0,"allocated":2883584,"data_stored":1484146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.60699999999999998,"15min":0.625},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.33800000000000002},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.503},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.623,"15min":0.63100000000000001},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.84099999999999997,"5min":0.878,"15min":0.89500000000000002},"last":0.56399999999999995}]},{"osd":1,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.63,"15min":0.65600000000000003},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.628,"15min":0.63700000000000001},"min":{"1min":0.41099999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.622}]},{"osd":2,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.65200000000000002,"15min":0.65300000000000002},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.83999999999999997,"5min":0.89900000000000002,"15min":0.90600000000000003},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.65600000000000003,"15min":0.65800000000000003},"min":{"1min":0.39600000000000002,"5min":0.39400000000000002,"15min":0.38400000000000001},"max":{"1min":0.88200000000000001,"5min":0.89700000000000002,"15min":0.92600000000000005},"last":0.746}]},{"osd":3,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.63200000000000001,"15min":0.64500000000000002},"min":{"1min":0.40400000000000003,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.78100000000000003,"5min":0.86599999999999999,"15min":0.99199999999999999},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.67400000000000004,"15min":0.68200000000000005},"min":{"1min":0.44800000000000001,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.79100000000000004,"5min":0.872,"15min":0.93899999999999995},"last":0.72699999999999998}]},{"osd":5,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55100000000000005,"15min":0.53000000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.57999999999999996,"15min":0.56899999999999995},"min":{"1min":0.36099999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.82199999999999995,"5min":0.82299999999999995,"15min":0.94899999999999995},"last":0.77300000000000002}]},{"osd":6,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.625,"15min":0.59799999999999998},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.33200000000000002},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92700000000000005},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.747,"5min":0.71399999999999997,"15min":0.69399999999999995},"min":{"1min":0.52600000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.129,"5min":1.129,"15min":1.129},"last":0.875}]},{"osd":7,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.66000000000000003,"15min":0.63500000000000001},"min":{"1min":0.44400000000000001,"5min":0.41999999999999998,"15min":0.35799999999999998},"max":{"1min":0.97099999999999997,"5min":0.97499999999999998,"15min":0.98599999999999999},"last":0.91100000000000003},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.70799999999999996,"15min":0.68400000000000005},"min":{"1min":0.44,"5min":0.433,"15min":0.38200000000000001},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":1.026},"last":0.80500000000000005}]}]},{"osd":5,"up_from":13,"seq":55834575987,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52064,"kb_used_data":2832,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319776,"statfs":{"total":96636764160,"available":96583450624,"internally_reserved":0,"allocated":2899968,"data_stored":1485212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:46:24 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.61399999999999999,"15min":0.61499999999999999},"min":{"1min":0.314,"5min":0.314,"15min":0.26800000000000002},"max":{"1min":0.878,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63,"15min":0.64200000000000002},"min":{"1min":0.38700000000000001,"5min":0.35899999999999999,"15min":0.29699999999999999},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.67500000000000004}]},{"osd":1,"last update":"Mon Oct 16 13:46:24 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.66000000000000003,"15min":0.66200000000000003},"min":{"1min":0.44600000000000001,"5min":0.34300000000000003,"15min":0.29499999999999998},"max":{"1min":1,"5min":1,"15min":1},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.63800000000000001,"15min":0.64000000000000001},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.622}]},{"osd":2,"last update":"Mon Oct 16 13:46:24 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.67000000000000004,"15min":0.65700000000000003},"min":{"1min":0.41699999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.91000000000000003,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.629,"5min":0.65700000000000003,"15min":0.64900000000000002},"min":{"1min":0.42599999999999999,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.83899999999999997,"5min":0.871,"15min":0.93500000000000005},"last":0.73499999999999999}]},{"osd":3,"last update":"Mon Oct 16 13:46:24 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.67400000000000004,"15min":0.66400000000000003},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.85799999999999998},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.69899999999999995,"15min":0.69499999999999995},"min":{"1min":0.41399999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.4199999999999999,"5min":1.4199999999999999,"15min":1.4199999999999999},"last":0.82799999999999996}]},{"osd":4,"last update":"Mon Oct 16 13:46:24 2023","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.54400000000000004,"15min":0.53300000000000003},"min":{"1min":0.34100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.82499999999999996,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.89700000000000002},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.59299999999999997,"15min":0.58899999999999997},"min":{"1min":0.34300000000000003,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.85399999999999998,"5min":0.89000000000000001,"15min":0.93400000000000005},"last":0.99399999999999999}]},{"osd":6,"last update":"Mon Oct 16 13:46:24 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.58899999999999997,"15min":0.57999999999999996},"min":{"1min":0.38700000000000001,"5min":0.315,"15min":0.315},"max":{"1min":0.877,"5min":0.88900000000000001,"15min":0.93200000000000005},"last":0.92000000000000004},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.61899999999999999},"min":{"1min":0.372,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89600000000000002},"last":0.95799999999999996}]},{"osd":7,"last update":"Mon Oct 16 13:46:24 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.64800000000000002,"15min":0.63300000000000001},"min":{"1min":0.378,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.93899999999999995},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.69699999999999995,"15min":0.68400000000000005},"min":{"1min":0.46200000000000002,"5min":0.375,"15min":0.375},"max":{"1min":0.94699999999999995,"5min":0.98299999999999998,"15min":1.0229999999999999},"last":0.97299999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T13:46:43.241 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-16T13:46:43.242 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:46:43.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.391+0000 7f0ab63e3700 1 Processor -- start 2023-10-16T13:46:43.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.392+0000 7f0ab63e3700 1 -- start start 2023-10-16T13:46:43.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.392+0000 7f0ab63e3700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f0ab015b860 0x7f0ab006a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:43.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.392+0000 7f0ab63e3700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0ab00b2be0 0x7f0ab006ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:43.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.392+0000 7f0ab63e3700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ab00d11c0 con 0x7f0ab00cd770 2023-10-16T13:46:43.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.392+0000 7f0ab63e3700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ab00d03d0 con 0x7f0ab00b2be0 2023-10-16T13:46:43.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.392+0000 7f0ab63e3700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ab00cd360 con 0x7f0ab015b860 2023-10-16T13:46:43.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.392+0000 7f0aaf7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0ab00b2be0 0x7f0ab006ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:43.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.393+0000 7f0aaf7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0ab00b2be0 0x7f0ab006ad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53596/0 (socket says 172.21.15.16:53596) 2023-10-16T13:46:43.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.393+0000 7f0aaf7fe700 1 -- 172.21.15.16:0/3434927599 learned_addr learned my addr 172.21.15.16:0/3434927599 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:43.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.393+0000 7f0aaffff700 1 --2- 172.21.15.16:0/3434927599 >> v2:172.21.15.49:3300/0 conn(0x7f0ab015b860 0x7f0ab006a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:43.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.393+0000 7f0aaf7fe700 1 -- 172.21.15.16:0/3434927599 >> v2:172.21.15.49:3300/0 conn(0x7f0ab015b860 msgr2=0x7f0ab006a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:43.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.393+0000 7f0aaf7fe700 1 --2- 172.21.15.16:0/3434927599 >> v2:172.21.15.49:3300/0 conn(0x7f0ab015b860 0x7f0ab006a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:43.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.393+0000 7f0aaf7fe700 1 -- 172.21.15.16:0/3434927599 >> v1:172.21.15.49:6789/0 conn(0x7f0ab00cd770 legacy=0x7f0ab00719e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:43.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.393+0000 7f0aaf7fe700 1 -- 172.21.15.16:0/3434927599 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ab00d05b0 con 0x7f0ab00b2be0 2023-10-16T13:46:43.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.393+0000 7f0aaf7fe700 1 --2- 172.21.15.16:0/3434927599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ab00b2be0 0x7f0ab006ad20 secure :-1 s=READY pgs=2526 cs=0 l=1 rev1=1 crypto rx=0x7f0ab0072e20 tx=0x7f0aa4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=434303cb3c514545 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:43.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.393+0000 7f0aaeffd700 1 -- 172.21.15.16:0/3434927599 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa4011830 con 0x7f0ab00b2be0 2023-10-16T13:46:43.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.393+0000 7f0aaeffd700 1 -- 172.21.15.16:0/3434927599 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0aa4011990 con 0x7f0ab00b2be0 2023-10-16T13:46:43.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.393+0000 7f0aaeffd700 1 -- 172.21.15.16:0/3434927599 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa4011d20 con 0x7f0ab00b2be0 2023-10-16T13:46:43.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.394+0000 7f0ab63e3700 1 -- 172.21.15.16:0/3434927599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ab00b2be0 msgr2=0x7f0ab006ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:43.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.394+0000 7f0ab63e3700 1 --2- 172.21.15.16:0/3434927599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ab00b2be0 0x7f0ab006ad20 secure :-1 s=READY pgs=2526 cs=0 l=1 rev1=1 crypto rx=0x7f0ab0072e20 tx=0x7f0aa4009bd0 comp rx=0 tx=0).stop 2023-10-16T13:46:43.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.394+0000 7f0ab63e3700 1 -- 172.21.15.16:0/3434927599 shutdown_connections 2023-10-16T13:46:43.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.394+0000 7f0ab63e3700 1 --2- 172.21.15.16:0/3434927599 >> v2:172.21.15.49:3300/0 conn(0x7f0ab015b860 0x7f0ab006a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:43.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.394+0000 7f0ab63e3700 1 --2- 172.21.15.16:0/3434927599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ab00b2be0 0x7f0ab006ad20 unknown :-1 s=CLOSED pgs=2526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:43.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.394+0000 7f0ab63e3700 1 -- 172.21.15.16:0/3434927599 >> 172.21.15.16:0/3434927599 conn(0x7f0ab00b1fd0 msgr2=0x7f0ab00b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:43.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.394+0000 7f0ab63e3700 1 -- 172.21.15.16:0/3434927599 shutdown_connections 2023-10-16T13:46:43.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.394+0000 7f0ab63e3700 1 -- 172.21.15.16:0/3434927599 wait complete. 2023-10-16T13:46:43.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.395+0000 7f0ab63e3700 1 Processor -- start 2023-10-16T13:46:43.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.395+0000 7f0ab63e3700 1 -- start start 2023-10-16T13:46:43.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.396+0000 7f0ab63e3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ab00b2be0 0x7f0ab0147b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:43.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.396+0000 7f0ab63e3700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0ab00cd770 0x7f0ab01471d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:43.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.396+0000 7f0ab63e3700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0ab015b860 0x7f0ab00b3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:43.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.396+0000 7f0ab63e3700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab014da40 con 0x7f0ab00b2be0 2023-10-16T13:46:43.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.396+0000 7f0ab63e3700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab00c6c60 con 0x7f0ab015b860 2023-10-16T13:46:43.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.396+0000 7f0ab63e3700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab00c3f50 con 0x7f0ab00cd770 2023-10-16T13:46:43.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.396+0000 7f0aaffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ab00b2be0 0x7f0ab0147b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:43.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.396+0000 7f0aaffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ab00b2be0 0x7f0ab0147b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53602/0 (socket says 172.21.15.16:53602) 2023-10-16T13:46:43.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.396+0000 7f0aaffff700 1 -- 172.21.15.16:0/3967150385 learned_addr learned my addr 172.21.15.16:0/3967150385 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:43.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.396+0000 7f0aaf7fe700 1 --2- 172.21.15.16:0/3967150385 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0ab00cd770 0x7f0ab01471d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:43.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.396+0000 7f0ab4980700 1 --2- 172.21.15.16:0/3967150385 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0ab015b860 0x7f0ab00b3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:43.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.396+0000 7f0aaffff700 1 -- 172.21.15.16:0/3967150385 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0ab00cd770 msgr2=0x7f0ab01471d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:43.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.396+0000 7f0aaffff700 1 --2- 172.21.15.16:0/3967150385 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0ab00cd770 0x7f0ab01471d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:43.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.396+0000 7f0aaffff700 1 -- 172.21.15.16:0/3967150385 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0ab015b860 msgr2=0x7f0ab00b3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:43.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.396+0000 7f0aaffff700 1 --2- 172.21.15.16:0/3967150385 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0ab015b860 0x7f0ab00b3fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:43.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.396+0000 7f0aaffff700 1 -- 172.21.15.16:0/3967150385 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0aa4009550 con 0x7f0ab00b2be0 2023-10-16T13:46:43.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.397+0000 7f0aaffff700 1 --2- 172.21.15.16:0/3967150385 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ab00b2be0 0x7f0ab0147b00 secure :-1 s=READY pgs=2527 cs=0 l=1 rev1=1 crypto rx=0x7f0aa000e9b0 tx=0x7f0aa000ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:43.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.397+0000 7f0aacff9700 1 -- 172.21.15.16:0/3967150385 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa0009710 con 0x7f0ab00b2be0 2023-10-16T13:46:43.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.397+0000 7f0aacff9700 1 -- 172.21.15.16:0/3967150385 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0aa000ee50 con 0x7f0ab00b2be0 2023-10-16T13:46:43.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.397+0000 7f0ab63e3700 1 -- 172.21.15.16:0/3967150385 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ab00c4230 con 0x7f0ab00b2be0 2023-10-16T13:46:43.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.397+0000 7f0aacff9700 1 -- 172.21.15.16:0/3967150385 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa000ca50 con 0x7f0ab00b2be0 2023-10-16T13:46:43.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.397+0000 7f0ab63e3700 1 -- 172.21.15.16:0/3967150385 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ab00c71e0 con 0x7f0ab00b2be0 2023-10-16T13:46:43.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.398+0000 7f0ab63e3700 1 -- 172.21.15.16:0/3967150385 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ab00b3e20 con 0x7f0ab00b2be0 2023-10-16T13:46:43.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.398+0000 7f0aacff9700 1 -- 172.21.15.16:0/3967150385 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 138) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0aa000cbb0 con 0x7f0ab00b2be0 2023-10-16T13:46:43.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.399+0000 7f0aacff9700 1 --2- 172.21.15.16:0/3967150385 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f0a9c06b020 0x7f0a9c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:43.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.399+0000 7f0aacff9700 1 -- 172.21.15.16:0/3967150385 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 7021+0+0 (secure 0 0 0) 0x7f0aa0089ce0 con 0x7f0ab00b2be0 2023-10-16T13:46:43.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.399+0000 7f0aaf7fe700 1 --2- 172.21.15.16:0/3967150385 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f0a9c06b020 0x7f0a9c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:43.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.399+0000 7f0aaf7fe700 1 --2- 172.21.15.16:0/3967150385 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f0a9c06b020 0x7f0a9c06d4e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0aa4000c00 tx=0x7f0aa4008030 comp rx=0 tx=0).ready entity=mgr.11190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:43.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.403+0000 7f0aacff9700 1 -- 172.21.15.16:0/3967150385 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0aa0054760 con 0x7f0ab00b2be0 2023-10-16T13:46:43.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.654+0000 7f0ab63e3700 1 -- 172.21.15.16:0/3967150385 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0ab004ebb0 con 0x7f0ab00b2be0 2023-10-16T13:46:43.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.654+0000 7f0aacff9700 1 -- 172.21.15.16:0/3967150385 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) v1 ==== 95+0+6154 (secure 0 0 0) 0x7f0aa0021090 con 0x7f0ab00b2be0 2023-10-16T13:46:43.658 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:46:43.658 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":119,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10516,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/4065699432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":4065699432},{"type":"v1","addr":"172.21.15.49:6834","nonce":4065699432}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":108},{"gid":11415,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/1373048456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1373048456},{"type":"v1","addr":"172.21.15.16:6836","nonce":1373048456}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":108}],"filesystems":[{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:46:24.755313+0000","modified":"2023-10-16T13:46:28.011141+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11430},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11430":{"gid":11430,"name":"b","rank":0,"incarnation":109,"state":"up:active","state_seq":3,"addr":"172.21.15.49:6835/2905760119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":2905760119},{"type":"v1","addr":"172.21.15.49:6835","nonce":2905760119}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13},{"mdsmap":{"epoch":119,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:46:38.053490+0000","modified":"2023-10-16T13:46:41.076680+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11418},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11418":{"gid":11418,"name":"a","rank":0,"incarnation":115,"state":"up:active","state_seq":6,"addr":"172.21.15.16:6837/3860406853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":3860406853},{"type":"v1","addr":"172.21.15.16:6837","nonce":3860406853}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2023-10-16T13:46:43.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.656+0000 7f0ab63e3700 1 -- 172.21.15.16:0/3967150385 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f0a9c06b020 msgr2=0x7f0a9c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:43.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.656+0000 7f0ab63e3700 1 --2- 172.21.15.16:0/3967150385 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f0a9c06b020 0x7f0a9c06d4e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0aa4000c00 tx=0x7f0aa4008030 comp rx=0 tx=0).stop 2023-10-16T13:46:43.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.656+0000 7f0ab63e3700 1 -- 172.21.15.16:0/3967150385 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ab00b2be0 msgr2=0x7f0ab0147b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:43.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.656+0000 7f0ab63e3700 1 --2- 172.21.15.16:0/3967150385 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ab00b2be0 0x7f0ab0147b00 secure :-1 s=READY pgs=2527 cs=0 l=1 rev1=1 crypto rx=0x7f0aa000e9b0 tx=0x7f0aa000ecc0 comp rx=0 tx=0).stop 2023-10-16T13:46:43.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.656+0000 7f0ab63e3700 1 -- 172.21.15.16:0/3967150385 shutdown_connections 2023-10-16T13:46:43.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.656+0000 7f0ab63e3700 1 --2- 172.21.15.16:0/3967150385 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0ab015b860 0x7f0ab00b3fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:43.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.656+0000 7f0ab63e3700 1 --2- 172.21.15.16:0/3967150385 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f0a9c06b020 0x7f0a9c06d4e0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:43.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.656+0000 7f0ab63e3700 1 --2- 172.21.15.16:0/3967150385 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ab00b2be0 0x7f0ab0147b00 unknown :-1 s=CLOSED pgs=2527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:43.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.656+0000 7f0ab63e3700 1 --2- 172.21.15.16:0/3967150385 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0ab00cd770 0x7f0ab01471d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:43.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.656+0000 7f0ab63e3700 1 -- 172.21.15.16:0/3967150385 >> 172.21.15.16:0/3967150385 conn(0x7f0ab00b1fd0 msgr2=0x7f0ab0159bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:43.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.657+0000 7f0ab63e3700 1 -- 172.21.15.16:0/3967150385 shutdown_connections 2023-10-16T13:46:43.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:43.657+0000 7f0ab63e3700 1 -- 172.21.15.16:0/3967150385 wait complete. 2023-10-16T13:46:43.663 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 119 2023-10-16T13:46:44.075 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 119, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T13:46:38.053490+0000', 'modified': '2023-10-16T13:46:41.076680+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 11418}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_11418': {'gid': 11418, 'name': 'a', 'rank': 0, 'incarnation': 115, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.16:6837/3860406853', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6835', 'nonce': 3860406853}, {'type': 'v1', 'addr': '172.21.15.16:6837', 'nonce': 3860406853}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'backup_fs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-16T13:46:44.075 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-16T13:46:44.076 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-10-16T13:46:44.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.211+0000 7f2baab79700 1 Processor -- start 2023-10-16T13:46:44.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.211+0000 7f2baab79700 1 -- start start 2023-10-16T13:46:44.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.211+0000 7f2baab79700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f2ba415b860 0x7f2ba406a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:44.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.211+0000 7f2baab79700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ba40d11c0 con 0x7f2ba40b2be0 2023-10-16T13:46:44.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.211+0000 7f2baab79700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ba4072800 con 0x7f2ba40b4490 2023-10-16T13:46:44.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.211+0000 7f2baab79700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ba40d03d0 con 0x7f2ba415b860 2023-10-16T13:46:44.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.212+0000 7f2ba3fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2ba40b2be0 0x7f2ba40719e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42884/0 (socket says 172.21.15.16:42884) 2023-10-16T13:46:44.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.212+0000 7f2ba3fff700 1 -- 172.21.15.16:0/1891618342 learned_addr learned my addr 172.21.15.16:0/1891618342 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:46:44.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.212+0000 7f2ba8915700 1 --2- 172.21.15.16:0/1891618342 >> v2:172.21.15.49:3300/0 conn(0x7f2ba415b860 0x7f2ba406a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:44.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.212+0000 7f2ba37fe700 1 -- 172.21.15.16:0/1891618342 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 707779160 0 0) 0x7f2ba40d11c0 con 0x7f2ba40b2be0 2023-10-16T13:46:44.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.212+0000 7f2ba37fe700 1 -- 172.21.15.16:0/1891618342 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b940035b0 con 0x7f2ba40b2be0 2023-10-16T13:46:44.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.212+0000 7f2ba37fe700 1 -- 172.21.15.16:0/1891618342 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2874780222 0 0) 0x7f2b940035b0 con 0x7f2ba40b2be0 2023-10-16T13:46:44.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.212+0000 7f2ba37fe700 1 -- 172.21.15.16:0/1891618342 >> v2:172.21.15.49:3300/0 conn(0x7f2ba415b860 msgr2=0x7f2ba406a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:44.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.212+0000 7f2ba37fe700 1 --2- 172.21.15.16:0/1891618342 >> v2:172.21.15.49:3300/0 conn(0x7f2ba415b860 0x7f2ba406a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:44.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.212+0000 7f2ba37fe700 1 -- 172.21.15.16:0/1891618342 >> v1:172.21.15.49:6789/0 conn(0x7f2ba40b4490 legacy=0x7f2ba40720f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:44.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.212+0000 7f2ba37fe700 1 -- 172.21.15.16:0/1891618342 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ba40cd360 con 0x7f2ba40b2be0 2023-10-16T13:46:44.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.212+0000 7f2ba37fe700 1 -- 172.21.15.16:0/1891618342 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f2b90001670 con 0x7f2ba40b2be0 2023-10-16T13:46:44.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.212+0000 7f2ba37fe700 1 -- 172.21.15.16:0/1891618342 >> v1:172.21.15.16:6789/0 conn(0x7f2ba40b2be0 legacy=0x7f2ba40719e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:44.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.212+0000 7f2ba37fe700 1 --2- 172.21.15.16:0/1891618342 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b94003a00 0x7f2b94003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:44.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.212+0000 7f2ba37fe700 1 --2- 172.21.15.16:0/1891618342 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2b94004850 0x7f2b94004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:44.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.212+0000 7f2ba37fe700 1 --2- 172.21.15.16:0/1891618342 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2b94005210 0x7f2b9400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:44.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.212+0000 7f2ba37fe700 1 -- 172.21.15.16:0/1891618342 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ba40d03d0 con 0x7f2b94003a00 2023-10-16T13:46:44.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.212+0000 7f2ba37fe700 1 -- 172.21.15.16:0/1891618342 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b9400c0b0 con 0x7f2b94004850 2023-10-16T13:46:44.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.212+0000 7f2ba37fe700 1 -- 172.21.15.16:0/1891618342 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b9400c270 con 0x7f2b94005210 2023-10-16T13:46:44.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.212+0000 7f2ba8915700 1 --2- 172.21.15.16:0/1891618342 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b94003a00 0x7f2b94003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:44.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.213+0000 7f2ba3fff700 1 --2- 172.21.15.16:0/1891618342 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2b94004850 0x7f2b94004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:44.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.213+0000 7f2ba9116700 1 --2- 172.21.15.16:0/1891618342 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2b94005210 0x7f2b9400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:44.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.213+0000 7f2ba8915700 1 -- 172.21.15.16:0/1891618342 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2b94005210 msgr2=0x7f2b9400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:44.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.213+0000 7f2ba8915700 1 --2- 172.21.15.16:0/1891618342 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2b94005210 0x7f2b9400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:44.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.213+0000 7f2ba8915700 1 -- 172.21.15.16:0/1891618342 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2b94004850 msgr2=0x7f2b94004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:44.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.213+0000 7f2ba8915700 1 --2- 172.21.15.16:0/1891618342 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2b94004850 0x7f2b94004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:44.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.213+0000 7f2ba8915700 1 -- 172.21.15.16:0/1891618342 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b940035b0 con 0x7f2b94003a00 2023-10-16T13:46:44.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.213+0000 7f2ba9116700 1 --2- 172.21.15.16:0/1891618342 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2b94005210 0x7f2b9400ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:46:44.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.213+0000 7f2ba9116700 1 -- 172.21.15.16:0/1891618342 reap_dead start 2023-10-16T13:46:44.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.213+0000 7f2ba8915700 1 --2- 172.21.15.16:0/1891618342 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b94003a00 0x7f2b94003e20 secure :-1 s=READY pgs=2529 cs=0 l=1 rev1=1 crypto rx=0x7f2b98009b60 tx=0x7f2b94005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12274937d9d32676 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:44.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.213+0000 7f2ba37fe700 1 -- 172.21.15.16:0/1891618342 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b9800e6c0 con 0x7f2b94003a00 2023-10-16T13:46:44.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.213+0000 7f2ba37fe700 1 -- 172.21.15.16:0/1891618342 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b9800e820 con 0x7f2b94003a00 2023-10-16T13:46:44.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.213+0000 7f2ba37fe700 1 -- 172.21.15.16:0/1891618342 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b9800eb90 con 0x7f2b94003a00 2023-10-16T13:46:44.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.214+0000 7f2baab79700 1 -- 172.21.15.16:0/1891618342 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b94003a00 msgr2=0x7f2b94003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:44.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.214+0000 7f2baab79700 1 --2- 172.21.15.16:0/1891618342 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b94003a00 0x7f2b94003e20 secure :-1 s=READY pgs=2529 cs=0 l=1 rev1=1 crypto rx=0x7f2b98009b60 tx=0x7f2b94005880 comp rx=0 tx=0).stop 2023-10-16T13:46:44.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.214+0000 7f2baab79700 1 -- 172.21.15.16:0/1891618342 shutdown_connections 2023-10-16T13:46:44.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.214+0000 7f2baab79700 1 --2- 172.21.15.16:0/1891618342 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b94003a00 0x7f2b94003e20 unknown :-1 s=CLOSED pgs=2529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:44.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.214+0000 7f2baab79700 1 -- 172.21.15.16:0/1891618342 >> 172.21.15.16:0/1891618342 conn(0x7f2ba40b1fd0 msgr2=0x7f2ba40b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:44.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.214+0000 7f2baab79700 1 -- 172.21.15.16:0/1891618342 shutdown_connections 2023-10-16T13:46:44.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.214+0000 7f2baab79700 1 -- 172.21.15.16:0/1891618342 wait complete. 2023-10-16T13:46:44.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.215+0000 7f2baab79700 1 Processor -- start 2023-10-16T13:46:44.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.215+0000 7f2baab79700 1 -- start start 2023-10-16T13:46:44.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.216+0000 7f2baab79700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b94003a00 0x7f2ba414d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:44.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.216+0000 7f2baab79700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2ba40ce920 0x7f2ba40aff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:44.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.216+0000 7f2baab79700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2ba40ceda0 0x7f2ba40b0480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:44.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.216+0000 7f2baab79700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ba41f6c90 con 0x7f2b94003a00 2023-10-16T13:46:44.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.216+0000 7f2baab79700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ba40cf370 con 0x7f2ba40ce920 2023-10-16T13:46:44.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.216+0000 7f2baab79700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ba41f6360 con 0x7f2ba40ceda0 2023-10-16T13:46:44.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.216+0000 7f2ba8915700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b94003a00 0x7f2ba414d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:44.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.216+0000 7f2ba8915700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b94003a00 0x7f2ba414d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53620/0 (socket says 172.21.15.16:53620) 2023-10-16T13:46:44.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.216+0000 7f2ba8915700 1 -- 172.21.15.16:0/1680489105 learned_addr learned my addr 172.21.15.16:0/1680489105 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:44.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.216+0000 7f2ba3fff700 1 --2- 172.21.15.16:0/1680489105 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2ba40ce920 0x7f2ba40aff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:44.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.216+0000 7f2ba9116700 1 --2- 172.21.15.16:0/1680489105 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2ba40ceda0 0x7f2ba40b0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:44.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.216+0000 7f2ba8915700 1 -- 172.21.15.16:0/1680489105 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2ba40ceda0 msgr2=0x7f2ba40b0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:44.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.216+0000 7f2ba8915700 1 --2- 172.21.15.16:0/1680489105 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2ba40ceda0 0x7f2ba40b0480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:44.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.216+0000 7f2ba8915700 1 -- 172.21.15.16:0/1680489105 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2ba40ce920 msgr2=0x7f2ba40aff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:44.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.216+0000 7f2ba8915700 1 --2- 172.21.15.16:0/1680489105 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2ba40ce920 0x7f2ba40aff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:44.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.216+0000 7f2ba8915700 1 -- 172.21.15.16:0/1680489105 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ba41f6640 con 0x7f2b94003a00 2023-10-16T13:46:44.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.217+0000 7f2ba8915700 1 --2- 172.21.15.16:0/1680489105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b94003a00 0x7f2ba414d820 secure :-1 s=READY pgs=2530 cs=0 l=1 rev1=1 crypto rx=0x7f2b9800a010 tx=0x7f2b980047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:44.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.217+0000 7f2ba17fa700 1 -- 172.21.15.16:0/1680489105 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b9800e6c0 con 0x7f2b94003a00 2023-10-16T13:46:44.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.217+0000 7f2ba17fa700 1 -- 172.21.15.16:0/1680489105 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b980038c0 con 0x7f2b94003a00 2023-10-16T13:46:44.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.217+0000 7f2ba17fa700 1 -- 172.21.15.16:0/1680489105 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b98003c70 con 0x7f2b94003a00 2023-10-16T13:46:44.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.217+0000 7f2baab79700 1 -- 172.21.15.16:0/1680489105 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ba41f6920 con 0x7f2b94003a00 2023-10-16T13:46:44.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.217+0000 7f2baab79700 1 -- 172.21.15.16:0/1680489105 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ba41f7120 con 0x7f2b94003a00 2023-10-16T13:46:44.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.219+0000 7f2ba17fa700 1 -- 172.21.15.16:0/1680489105 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 138) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2b98013470 con 0x7f2b94003a00 2023-10-16T13:46:44.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.219+0000 7f2ba17fa700 1 --2- 172.21.15.16:0/1680489105 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f2b8806b020 0x7f2b8806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:44.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.219+0000 7f2ba17fa700 1 -- 172.21.15.16:0/1680489105 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 7021+0+0 (secure 0 0 0) 0x7f2b9808b450 con 0x7f2b94003a00 2023-10-16T13:46:44.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.219+0000 7f2ba3fff700 1 --2- 172.21.15.16:0/1680489105 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f2b8806b020 0x7f2b8806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:44.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.220+0000 7f2ba3fff700 1 --2- 172.21.15.16:0/1680489105 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f2b8806b020 0x7f2b8806d4e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f2b900017a0 tx=0x7f2b90001590 comp rx=0 tx=0).ready entity=mgr.11190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:44.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.221+0000 7f2bb6d63b80 1 Processor -- start 2023-10-16T13:46:44.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.221+0000 7f2bb6d63b80 1 -- start start 2023-10-16T13:46:44.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.222+0000 7f2bb6d63b80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55f6526ab220 0x55f6526b2a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:44.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.222+0000 7f2bb6d63b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6526b0630 0x55f6526b2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:44.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.222+0000 7f2bb6d63b80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f652686010 0x55f6526866c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:44.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.222+0000 7f2bb6d63b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f6525fcc20 con 0x55f6526b0630 2023-10-16T13:46:44.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.222+0000 7f2bb6d63b80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f6525e1c10 con 0x55f652686010 2023-10-16T13:46:44.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.222+0000 7f2bb6d63b80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f6525df530 con 0x55f6526ab220 2023-10-16T13:46:44.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.222+0000 7f2ba3fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6526b0630 0x55f6526b2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:44.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.222+0000 7f2ba3fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6526b0630 0x55f6526b2fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53630/0 (socket says 172.21.15.16:53630) 2023-10-16T13:46:44.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.222+0000 7f2ba3fff700 1 -- 172.21.15.16:0/2565352730 learned_addr learned my addr 172.21.15.16:0/2565352730 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:44.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.222+0000 7f2ba8915700 1 --2- 172.21.15.16:0/2565352730 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55f6526ab220 0x55f6526b2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:44.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.222+0000 7f2ba9116700 1 --2- 172.21.15.16:0/2565352730 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f652686010 0x55f6526866c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:44.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.222+0000 7f2ba3fff700 1 -- 172.21.15.16:0/2565352730 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55f6526ab220 msgr2=0x55f6526b2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:44.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.222+0000 7f2ba3fff700 1 --2- 172.21.15.16:0/2565352730 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55f6526ab220 0x55f6526b2a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:44.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.222+0000 7f2ba3fff700 1 -- 172.21.15.16:0/2565352730 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f652686010 msgr2=0x55f6526866c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:44.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.222+0000 7f2ba3fff700 1 --2- 172.21.15.16:0/2565352730 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f652686010 0x55f6526866c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:44.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.222+0000 7f2ba3fff700 1 -- 172.21.15.16:0/2565352730 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f6525983a0 con 0x55f6526b0630 2023-10-16T13:46:44.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.222+0000 7f2ba8915700 1 --2- 172.21.15.16:0/2565352730 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55f6526ab220 0x55f6526b2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:46:44.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.222+0000 7f2ba9116700 1 --2- 172.21.15.16:0/2565352730 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f652686010 0x55f6526866c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:46:44.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.222+0000 7f2ba3fff700 1 --2- 172.21.15.16:0/2565352730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6526b0630 0x55f6526b2fb0 secure :-1 s=READY pgs=2531 cs=0 l=1 rev1=1 crypto rx=0x7f2b900072e0 tx=0x7f2b9000c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25c5aa3f11421d5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:44.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.222+0000 7f2b8e7fc700 1 -- 172.21.15.16:0/2565352730 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b90015800 con 0x55f6526b0630 2023-10-16T13:46:44.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.222+0000 7f2b8e7fc700 1 -- 172.21.15.16:0/2565352730 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b90015960 con 0x55f6526b0630 2023-10-16T13:46:44.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.222+0000 7f2b8e7fc700 1 -- 172.21.15.16:0/2565352730 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b90015cd0 con 0x55f6526b0630 2023-10-16T13:46:44.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.223+0000 7f2bb6d63b80 1 -- 172.21.15.16:0/2565352730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6526b0630 msgr2=0x55f6526b2fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:44.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.223+0000 7f2bb6d63b80 1 --2- 172.21.15.16:0/2565352730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6526b0630 0x55f6526b2fb0 secure :-1 s=READY pgs=2531 cs=0 l=1 rev1=1 crypto rx=0x7f2b900072e0 tx=0x7f2b9000c000 comp rx=0 tx=0).stop 2023-10-16T13:46:44.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.223+0000 7f2bb6d63b80 1 -- 172.21.15.16:0/2565352730 shutdown_connections 2023-10-16T13:46:44.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.223+0000 7f2bb6d63b80 1 --2- 172.21.15.16:0/2565352730 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f652686010 0x55f6526866c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:44.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.223+0000 7f2bb6d63b80 1 --2- 172.21.15.16:0/2565352730 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55f6526ab220 0x55f6526b2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:44.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.223+0000 7f2bb6d63b80 1 --2- 172.21.15.16:0/2565352730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6526b0630 0x55f6526b2fb0 unknown :-1 s=CLOSED pgs=2531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:44.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.223+0000 7f2bb6d63b80 1 -- 172.21.15.16:0/2565352730 >> 172.21.15.16:0/2565352730 conn(0x55f6525b5d80 msgr2=0x55f6526d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:44.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.223+0000 7f2bb6d63b80 1 -- 172.21.15.16:0/2565352730 shutdown_connections 2023-10-16T13:46:44.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.223+0000 7f2bb6d63b80 1 -- 172.21.15.16:0/2565352730 wait complete. 2023-10-16T13:46:44.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.224+0000 7f2bb6d63b80 1 -- start start 2023-10-16T13:46:44.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.224+0000 7f2bb6d63b80 1 Processor -- start 2023-10-16T13:46:44.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.225+0000 7f2bb6d63b80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f652686010 0x55f6526b6230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:44.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.225+0000 7f2bb6d63b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6526ab220 0x55f6526aa540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:44.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.225+0000 7f2bb6d63b80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55f6526b0630 0x55f6526dcd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:44.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.225+0000 7f2bb6d63b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f6525dae20 con 0x55f6526ab220 2023-10-16T13:46:44.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.225+0000 7f2bb6d63b80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f6525776c0 con 0x55f652686010 2023-10-16T13:46:44.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.225+0000 7f2bb6d63b80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f652577360 con 0x55f6526b0630 2023-10-16T13:46:44.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.225+0000 7f2bb6d63b80 10 client.0 _ll_register_callbacks cb 0x55f6523d47b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T13:46:44.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.225+0000 7f2ba3fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6526ab220 0x55f6526aa540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:44.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.225+0000 7f2ba3fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6526ab220 0x55f6526aa540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53636/0 (socket says 172.21.15.16:53636) 2023-10-16T13:46:44.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.225+0000 7f2ba3fff700 1 -- 172.21.15.16:0/1683411445 learned_addr learned my addr 172.21.15.16:0/1683411445 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:44.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.225+0000 7f2ba9116700 1 --2- 172.21.15.16:0/1683411445 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55f6526b0630 0x55f6526dcd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:44.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.225+0000 7f2ba8915700 1 --2- 172.21.15.16:0/1683411445 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f652686010 0x55f6526b6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:44.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.225+0000 7f2ba3fff700 1 -- 172.21.15.16:0/1683411445 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55f6526b0630 msgr2=0x55f6526dcd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:44.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.225+0000 7f2ba3fff700 1 --2- 172.21.15.16:0/1683411445 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55f6526b0630 0x55f6526dcd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:44.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.225+0000 7f2ba3fff700 1 -- 172.21.15.16:0/1683411445 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f652686010 msgr2=0x55f6526b6230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:44.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.225+0000 7f2ba3fff700 1 --2- 172.21.15.16:0/1683411445 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f652686010 0x55f6526b6230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:44.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.225+0000 7f2ba3fff700 1 -- 172.21.15.16:0/1683411445 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ba40cd360 con 0x55f6526ab220 2023-10-16T13:46:44.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.225+0000 7f2ba8915700 1 --2- 172.21.15.16:0/1683411445 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f652686010 0x55f6526b6230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:46:44.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.225+0000 7f2ba3fff700 1 --2- 172.21.15.16:0/1683411445 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6526ab220 0x55f6526aa540 secure :-1 s=READY pgs=2532 cs=0 l=1 rev1=1 crypto rx=0x7f2b90021000 tx=0x7f2b90001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8db2cc4c743c9081 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:44.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.225+0000 7f2b837fe700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2023-10-16T13:46:44.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.225+0000 7f2b837fe700 1 -- 172.21.15.16:0/1683411445 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b90015800 con 0x55f6526ab220 2023-10-16T13:46:44.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.226+0000 7f2b837fe700 1 -- 172.21.15.16:0/1683411445 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(229..229 src has 1..229) v4 ==== 7021+0+0 (secure 0 0 0) 0x7f2b9001a070 con 0x55f6526ab220 2023-10-16T13:46:44.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.226+0000 7f2bb6d63b80 1 -- 172.21.15.16:0/1683411445 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f65267b5b0 con 0x55f6526ab220 2023-10-16T13:46:44.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.226+0000 7f2b837fe700 1 -- 172.21.15.16:0/1683411445 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2b90015e20 con 0x55f6526ab220 2023-10-16T13:46:44.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.226+0000 7f2bb6d63b80 10 client.11712 fetch_fsmap learned FSMap version 119 2023-10-16T13:46:44.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.226+0000 7f2bb6d63b80 1 -- 172.21.15.16:0/1683411445 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=2+}) v3 -- 0x55f652405d90 con 0x55f6526ab220 2023-10-16T13:46:44.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.226+0000 7f2b837fe700 1 -- 172.21.15.16:0/1683411445 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55f65267b5b0 con 0x55f6526ab220 2023-10-16T13:46:44.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.226+0000 7f2b837fe700 1 -- 172.21.15.16:0/1683411445 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 119) v1 ==== 3891+0+0 (secure 0 0 0) 0x7f2b90029030 con 0x55f6526ab220 2023-10-16T13:46:44.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.226+0000 7f2bb6d63b80 10 client.11712 fetch_fsmap finished waiting for FSMap version 119 2023-10-16T13:46:44.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.226+0000 7f2bb6d63b80 10 client.11712 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-16T13:46:44.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.226+0000 7f2bb6d63b80 20 client.11712 populate_metadata read hostname 'smithi016' 2023-10-16T13:46:44.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.226+0000 7f2bb6d63b80 1 --2- 172.21.15.16:0/1683411445 >> [v2:172.21.15.16:6835/3860406853,v1:172.21.15.16:6837/3860406853] conn(0x55f6526879f0 0x55f652687dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:44.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.226+0000 7f2bb6d63b80 4 client.11712 mds_command: new command op to 11418 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T13:46:44.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.226+0000 7f2bb6d63b80 1 -- 172.21.15.16:0/1683411445 --> [v2:172.21.15.16:6835/3860406853,v1:172.21.15.16:6837/3860406853] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f6525d3e00 con 0x55f6526879f0 2023-10-16T13:46:44.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.226+0000 7f2ba8915700 1 --2- 172.21.15.16:0/1683411445 >> [v2:172.21.15.16:6835/3860406853,v1:172.21.15.16:6837/3860406853] conn(0x55f6526879f0 0x55f652687dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:44.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.226+0000 7f2b837fe700 10 client.11712 ms_handle_connect on v2:172.21.15.16:6835/3860406853 2023-10-16T13:46:44.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.226+0000 7f2ba8915700 1 --2- 172.21.15.16:0/1683411445 >> [v2:172.21.15.16:6835/3860406853,v1:172.21.15.16:6837/3860406853] conn(0x55f6526879f0 0x55f652687dd0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab6b036fcfb3c9c0 server_cookie=425e5219f348128b in_seq=0 out_seq=0 2023-10-16T13:46:44.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.227+0000 7f2b837fe700 1 -- 172.21.15.16:0/1683411445 <== mds.0 v2:172.21.15.16:6835/3860406853 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55f6525d3e00 con 0x55f6526879f0 2023-10-16T13:46:44.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.227+0000 7f2b837fe700 10 client.11712 handle_command_reply: tid=0 2023-10-16T13:46:44.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.227+0000 7f2bb6d63b80 1 client.11712 shutdown 2023-10-16T13:46:44.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.227+0000 7f2bb6d63b80 1 -- 172.21.15.16:0/1683411445 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6526ab220 msgr2=0x55f6526aa540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:44.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.227+0000 7f2bb6d63b80 1 --2- 172.21.15.16:0/1683411445 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6526ab220 0x55f6526aa540 secure :-1 s=READY pgs=2532 cs=0 l=1 rev1=1 crypto rx=0x7f2b90021000 tx=0x7f2b90001a00 comp rx=0 tx=0).stop 2023-10-16T13:46:44.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.228+0000 7f2bb6d63b80 1 -- 172.21.15.16:0/1683411445 shutdown_connections 2023-10-16T13:46:44.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.228+0000 7f2bb6d63b80 1 --2- 172.21.15.16:0/1683411445 >> [v2:172.21.15.16:6835/3860406853,v1:172.21.15.16:6837/3860406853] conn(0x55f6526879f0 0x55f652687dd0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:44.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.228+0000 7f2bb6d63b80 1 --2- 172.21.15.16:0/1683411445 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55f6526b0630 0x55f6526dcd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:44.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.228+0000 7f2bb6d63b80 1 --2- 172.21.15.16:0/1683411445 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f652686010 0x55f6526b6230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:44.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.228+0000 7f2b837fe700 0 client.11712 ms_handle_reset on v2:172.21.15.16:6835/3860406853 2023-10-16T13:46:44.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.228+0000 7f2bb6d63b80 1 --2- 172.21.15.16:0/1683411445 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6526ab220 0x55f6526aa540 unknown :-1 s=CLOSED pgs=2532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:44.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.228+0000 7f2bb6d63b80 1 -- 172.21.15.16:0/1683411445 >> 172.21.15.16:0/1683411445 conn(0x55f6525b5d80 msgr2=0x55f6526ce520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:44.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.228+0000 7f2bb6d63b80 1 -- 172.21.15.16:0/1683411445 shutdown_connections 2023-10-16T13:46:44.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.228+0000 7f2bb6d63b80 1 -- 172.21.15.16:0/1683411445 wait complete. 2023-10-16T13:46:44.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.228+0000 7f2bb6d63b80 20 client.11712 trim_cache size 0 max 16384 2023-10-16T13:46:44.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.246+0000 7f2bb6d63b80 1 Processor -- start 2023-10-16T13:46:44.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.246+0000 7f2bb6d63b80 1 -- start start 2023-10-16T13:46:44.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.246+0000 7f2bb6d63b80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f652686010 0x55f652688b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:44.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.246+0000 7f2bb6d63b80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55f6526b0630 0x55f6526b51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:44.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.246+0000 7f2bb6d63b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6526c7420 0x55f6526dc0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:44.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.246+0000 7f2bb6d63b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f65245b8d0 con 0x55f6526c7420 2023-10-16T13:46:44.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.246+0000 7f2bb6d63b80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f6526ef9f0 con 0x55f652686010 2023-10-16T13:46:44.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.246+0000 7f2bb6d63b80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f6526c7bd0 con 0x55f6526b0630 2023-10-16T13:46:44.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.247+0000 7f2ba9116700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6526c7420 0x55f6526dc0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:44.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.247+0000 7f2ba9116700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6526c7420 0x55f6526dc0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53646/0 (socket says 172.21.15.16:53646) 2023-10-16T13:46:44.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.247+0000 7f2ba9116700 1 -- 172.21.15.16:0/4288082897 learned_addr learned my addr 172.21.15.16:0/4288082897 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:44.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.247+0000 7f2ba8915700 1 --2- 172.21.15.16:0/4288082897 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f652686010 0x55f652688b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:44.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.247+0000 7f2ba3fff700 1 --2- 172.21.15.16:0/4288082897 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55f6526b0630 0x55f6526b51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:44.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.247+0000 7f2ba9116700 1 -- 172.21.15.16:0/4288082897 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55f6526b0630 msgr2=0x55f6526b51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:44.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.247+0000 7f2ba9116700 1 --2- 172.21.15.16:0/4288082897 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55f6526b0630 0x55f6526b51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:44.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.247+0000 7f2ba9116700 1 -- 172.21.15.16:0/4288082897 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f652686010 msgr2=0x55f652688b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:44.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.247+0000 7f2ba9116700 1 --2- 172.21.15.16:0/4288082897 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f652686010 0x55f652688b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:44.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.247+0000 7f2ba9116700 1 -- 172.21.15.16:0/4288082897 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f6523eec10 con 0x55f6526c7420 2023-10-16T13:46:44.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.247+0000 7f2ba9116700 1 --2- 172.21.15.16:0/4288082897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6526c7420 0x55f6526dc0b0 secure :-1 s=READY pgs=2533 cs=0 l=1 rev1=1 crypto rx=0x7f2ba40b1f60 tx=0x7f2b9c003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96cc4d162286461 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:44.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.247+0000 7f2b8effd700 1 -- 172.21.15.16:0/4288082897 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b9c0056a0 con 0x55f6526c7420 2023-10-16T13:46:44.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.247+0000 7f2b8effd700 1 -- 172.21.15.16:0/4288082897 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b9c005820 con 0x55f6526c7420 2023-10-16T13:46:44.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.247+0000 7f2b8effd700 1 -- 172.21.15.16:0/4288082897 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b9c017430 con 0x55f6526c7420 2023-10-16T13:46:44.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.248+0000 7f2bb6d63b80 1 -- 172.21.15.16:0/4288082897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6526c7420 msgr2=0x55f6526dc0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:44.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.248+0000 7f2bb6d63b80 1 --2- 172.21.15.16:0/4288082897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6526c7420 0x55f6526dc0b0 secure :-1 s=READY pgs=2533 cs=0 l=1 rev1=1 crypto rx=0x7f2ba40b1f60 tx=0x7f2b9c003ce0 comp rx=0 tx=0).stop 2023-10-16T13:46:44.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.248+0000 7f2bb6d63b80 1 -- 172.21.15.16:0/4288082897 shutdown_connections 2023-10-16T13:46:44.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.248+0000 7f2bb6d63b80 1 --2- 172.21.15.16:0/4288082897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6526c7420 0x55f6526dc0b0 unknown :-1 s=CLOSED pgs=2533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:44.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.248+0000 7f2bb6d63b80 1 --2- 172.21.15.16:0/4288082897 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f652686010 0x55f652688b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:44.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.248+0000 7f2bb6d63b80 1 --2- 172.21.15.16:0/4288082897 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55f6526b0630 0x55f6526b51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:44.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.248+0000 7f2bb6d63b80 1 -- 172.21.15.16:0/4288082897 >> 172.21.15.16:0/4288082897 conn(0x55f6525b5d80 msgr2=0x55f6526a99b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:44.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.248+0000 7f2bb6d63b80 1 -- 172.21.15.16:0/4288082897 shutdown_connections 2023-10-16T13:46:44.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.248+0000 7f2bb6d63b80 1 -- 172.21.15.16:0/4288082897 wait complete. 2023-10-16T13:46:44.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.248+0000 7f2bb6d63b80 1 -- start start 2023-10-16T13:46:44.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.249+0000 7f2bb6d63b80 1 Processor -- start 2023-10-16T13:46:44.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.250+0000 7f2bb6d63b80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f652686010 0x55f6526ab600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:44.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.250+0000 7f2bb6d63b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6526b0630 0x55f6526abb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:44.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.250+0000 7f2bb6d63b80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55f6526c7420 0x55f6526863f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:44.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.250+0000 7f2bb6d63b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f6526b0420 con 0x55f6526b0630 2023-10-16T13:46:44.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.250+0000 7f2bb6d63b80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f65267cf40 con 0x55f652686010 2023-10-16T13:46:44.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.250+0000 7f2bb6d63b80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f65267cc50 con 0x55f6526c7420 2023-10-16T13:46:44.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.250+0000 7f2bb6d63b80 10 client.0 _ll_register_callbacks cb 0x7f2b900071e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T13:46:44.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.250+0000 7f2ba3fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6526b0630 0x55f6526abb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:44.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.250+0000 7f2ba3fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6526b0630 0x55f6526abb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53650/0 (socket says 172.21.15.16:53650) 2023-10-16T13:46:44.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.250+0000 7f2ba3fff700 1 -- 172.21.15.16:0/94774888 learned_addr learned my addr 172.21.15.16:0/94774888 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:44.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.250+0000 7f2ba8915700 1 --2- 172.21.15.16:0/94774888 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f652686010 0x55f6526ab600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:44.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.250+0000 7f2ba9116700 1 --2- 172.21.15.16:0/94774888 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55f6526c7420 0x55f6526863f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:44.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.250+0000 7f2ba3fff700 1 -- 172.21.15.16:0/94774888 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55f6526c7420 msgr2=0x55f6526863f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:44.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.250+0000 7f2ba3fff700 1 --2- 172.21.15.16:0/94774888 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55f6526c7420 0x55f6526863f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:44.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.250+0000 7f2ba3fff700 1 -- 172.21.15.16:0/94774888 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f652686010 msgr2=0x55f6526ab600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:44.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.250+0000 7f2ba3fff700 1 --2- 172.21.15.16:0/94774888 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f652686010 0x55f6526ab600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:44.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.250+0000 7f2ba3fff700 1 -- 172.21.15.16:0/94774888 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b9c00f040 con 0x55f6526b0630 2023-10-16T13:46:44.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.250+0000 7f2ba3fff700 1 --2- 172.21.15.16:0/94774888 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6526b0630 0x55f6526abb40 secure :-1 s=READY pgs=2534 cs=0 l=1 rev1=1 crypto rx=0x7f2b90015fd0 tx=0x7f2b90020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de9271be468cd15c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:44.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.250+0000 7f2b837fe700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2023-10-16T13:46:44.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.251+0000 7f2b837fe700 1 -- 172.21.15.16:0/94774888 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b90015800 con 0x55f6526b0630 2023-10-16T13:46:44.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.251+0000 7f2b837fe700 1 -- 172.21.15.16:0/94774888 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(229..229 src has 1..229) v4 ==== 7021+0+0 (secure 0 0 0) 0x7f2b9001a040 con 0x55f6526b0630 2023-10-16T13:46:44.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.251+0000 7f2bb6d63b80 1 -- 172.21.15.16:0/94774888 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2b90015960 con 0x55f6526b0630 2023-10-16T13:46:44.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.251+0000 7f2b837fe700 1 -- 172.21.15.16:0/94774888 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2b90022780 con 0x55f6526b0630 2023-10-16T13:46:44.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.251+0000 7f2bb6d63b80 10 client.11718 fetch_fsmap learned FSMap version 119 2023-10-16T13:46:44.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.251+0000 7f2bb6d63b80 1 -- 172.21.15.16:0/94774888 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=2+}) v3 -- 0x55f65242c200 con 0x55f6526b0630 2023-10-16T13:46:44.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.251+0000 7f2b837fe700 1 -- 172.21.15.16:0/94774888 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b90015960 con 0x55f6526b0630 2023-10-16T13:46:44.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.251+0000 7f2b837fe700 1 -- 172.21.15.16:0/94774888 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 119) v1 ==== 3891+0+0 (secure 0 0 0) 0x7f2b9002c040 con 0x55f6526b0630 2023-10-16T13:46:44.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.251+0000 7f2bb6d63b80 10 client.11718 fetch_fsmap finished waiting for FSMap version 119 2023-10-16T13:46:44.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.251+0000 7f2bb6d63b80 10 client.11718 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-16T13:46:44.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.251+0000 7f2bb6d63b80 20 client.11718 populate_metadata read hostname 'smithi016' 2023-10-16T13:46:44.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.251+0000 7f2bb6d63b80 1 --2- 172.21.15.16:0/94774888 >> [v2:172.21.15.16:6835/3860406853,v1:172.21.15.16:6837/3860406853] conn(0x55f652686930 0x55f652734910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:44.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.251+0000 7f2bb6d63b80 4 client.11718 mds_command: new command op to 11418 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T13:46:44.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.251+0000 7f2bb6d63b80 1 -- 172.21.15.16:0/94774888 --> [v2:172.21.15.16:6835/3860406853,v1:172.21.15.16:6837/3860406853] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55f6525d3b20 con 0x55f652686930 2023-10-16T13:46:44.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.251+0000 7f2ba8915700 1 --2- 172.21.15.16:0/94774888 >> [v2:172.21.15.16:6835/3860406853,v1:172.21.15.16:6837/3860406853] conn(0x55f652686930 0x55f652734910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:44.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.252+0000 7f2b837fe700 10 client.11718 ms_handle_connect on v2:172.21.15.16:6835/3860406853 2023-10-16T13:46:44.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.252+0000 7f2ba8915700 1 --2- 172.21.15.16:0/94774888 >> [v2:172.21.15.16:6835/3860406853,v1:172.21.15.16:6837/3860406853] conn(0x55f652686930 0x55f652734910 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2efc9d3fd97de7ff server_cookie=d3c955c63f1ac89c in_seq=0 out_seq=0 2023-10-16T13:46:44.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.252+0000 7f2b837fe700 1 -- 172.21.15.16:0/94774888 <== mds.0 v2:172.21.15.16:6835/3860406853 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x55f6525d3b20 con 0x55f652686930 2023-10-16T13:46:44.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.252+0000 7f2b837fe700 10 client.11718 handle_command_reply: tid=0 2023-10-16T13:46:44.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.252+0000 7f2bb6d63b80 1 client.11718 shutdown 2023-10-16T13:46:44.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.252+0000 7f2bb6d63b80 1 -- 172.21.15.16:0/94774888 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6526b0630 msgr2=0x55f6526abb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:44.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.252+0000 7f2bb6d63b80 1 --2- 172.21.15.16:0/94774888 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6526b0630 0x55f6526abb40 secure :-1 s=READY pgs=2534 cs=0 l=1 rev1=1 crypto rx=0x7f2b90015fd0 tx=0x7f2b90020000 comp rx=0 tx=0).stop 2023-10-16T13:46:44.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.252+0000 7f2bb6d63b80 1 -- 172.21.15.16:0/94774888 shutdown_connections 2023-10-16T13:46:44.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.252+0000 7f2bb6d63b80 1 --2- 172.21.15.16:0/94774888 >> [v2:172.21.15.16:6835/3860406853,v1:172.21.15.16:6837/3860406853] conn(0x55f652686930 0x55f652734910 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:44.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.252+0000 7f2bb6d63b80 1 --2- 172.21.15.16:0/94774888 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55f6526c7420 0x55f6526863f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:44.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.252+0000 7f2b837fe700 0 client.11718 ms_handle_reset on v2:172.21.15.16:6835/3860406853 2023-10-16T13:46:44.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.252+0000 7f2bb6d63b80 1 --2- 172.21.15.16:0/94774888 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f652686010 0x55f6526ab600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:44.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.252+0000 7f2bb6d63b80 1 --2- 172.21.15.16:0/94774888 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f6526b0630 0x55f6526abb40 unknown :-1 s=CLOSED pgs=2534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:44.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.252+0000 7f2bb6d63b80 1 -- 172.21.15.16:0/94774888 >> 172.21.15.16:0/94774888 conn(0x55f6525b5d80 msgr2=0x55f6526b51e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:44.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.252+0000 7f2bb6d63b80 1 -- 172.21.15.16:0/94774888 shutdown_connections 2023-10-16T13:46:44.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.252+0000 7f2bb6d63b80 1 -- 172.21.15.16:0/94774888 wait complete. 2023-10-16T13:46:44.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.252+0000 7f2bb6d63b80 20 client.11718 trim_cache size 0 max 16384 2023-10-16T13:46:44.276 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-10-16T13:46:44.277 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "5b28dfa1-b21f-474a-9b2e-9fb77af58254", 2023-10-16T13:46:44.277 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 0, 2023-10-16T13:46:44.277 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 11418, 2023-10-16T13:46:44.277 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2023-10-16T13:46:44.277 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2023-10-16T13:46:44.277 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "backup_fs", 2023-10-16T13:46:44.278 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 5.6045300190000003, 2023-10-16T13:46:44.278 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 119, 2023-10-16T13:46:44.278 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 229, 2023-10-16T13:46:44.278 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 229, 2023-10-16T13:46:44.278 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 24.587651412 2023-10-16T13:46:44.278 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-10-16T13:46:44.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.253+0000 7f2baab79700 1 -- 172.21.15.16:0/1680489105 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f2b8806b020 msgr2=0x7f2b8806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:44.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.253+0000 7f2baab79700 1 --2- 172.21.15.16:0/1680489105 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f2b8806b020 0x7f2b8806d4e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f2b900017a0 tx=0x7f2b90001590 comp rx=0 tx=0).stop 2023-10-16T13:46:44.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.253+0000 7f2baab79700 1 -- 172.21.15.16:0/1680489105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b94003a00 msgr2=0x7f2ba414d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:44.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.253+0000 7f2baab79700 1 --2- 172.21.15.16:0/1680489105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b94003a00 0x7f2ba414d820 secure :-1 s=READY pgs=2530 cs=0 l=1 rev1=1 crypto rx=0x7f2b9800a010 tx=0x7f2b980047c0 comp rx=0 tx=0).stop 2023-10-16T13:46:44.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.253+0000 7f2baab79700 1 -- 172.21.15.16:0/1680489105 shutdown_connections 2023-10-16T13:46:44.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.253+0000 7f2baab79700 1 --2- 172.21.15.16:0/1680489105 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2ba40ceda0 0x7f2ba40b0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:44.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.253+0000 7f2baab79700 1 --2- 172.21.15.16:0/1680489105 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f2b8806b020 0x7f2b8806d4e0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:44.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.253+0000 7f2baab79700 1 --2- 172.21.15.16:0/1680489105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b94003a00 0x7f2ba414d820 unknown :-1 s=CLOSED pgs=2530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:44.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.253+0000 7f2baab79700 1 --2- 172.21.15.16:0/1680489105 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2ba40ce920 0x7f2ba40aff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:44.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.253+0000 7f2baab79700 1 -- 172.21.15.16:0/1680489105 >> 172.21.15.16:0/1680489105 conn(0x7f2ba40b1fd0 msgr2=0x7f2ba40b49e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:44.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.253+0000 7f2baab79700 1 -- 172.21.15.16:0/1680489105 shutdown_connections 2023-10-16T13:46:44.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.253+0000 7f2baab79700 1 -- 172.21.15.16:0/1680489105 wait complete. 2023-10-16T13:46:44.282 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-16T13:46:44.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.405+0000 7f2b3080a700 1 Processor -- start 2023-10-16T13:46:44.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.405+0000 7f2b3080a700 1 -- start start 2023-10-16T13:46:44.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.406+0000 7f2b3080a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2b2c15db40 0x7f2b2c0cb980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:44.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.406+0000 7f2b3080a700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f2b2c06a7e0 0x7f2b2c06ac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:44.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.406+0000 7f2b3080a700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b2c0cedb0 con 0x7f2b2c0ce160 2023-10-16T13:46:44.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.406+0000 7f2b3080a700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b2c06b1a0 con 0x7f2b2c06a7e0 2023-10-16T13:46:44.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.406+0000 7f2b3080a700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b2c15df20 con 0x7f2b2c15db40 2023-10-16T13:46:44.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.406+0000 7f2b2ad9d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2b2c0ce160 0x7f2b2c0ce5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42930/0 (socket says 172.21.15.16:42930) 2023-10-16T13:46:44.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.406+0000 7f2b2ad9d700 1 -- 172.21.15.16:0/2114917011 learned_addr learned my addr 172.21.15.16:0/2114917011 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:46:44.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.407+0000 7f2b29d9b700 1 --2- 172.21.15.16:0/2114917011 >> v2:172.21.15.49:3300/0 conn(0x7f2b2c06a7e0 0x7f2b2c06ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:44.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.407+0000 7f2b2a59c700 1 --2- 172.21.15.16:0/2114917011 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2b2c15db40 0x7f2b2c0cb980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:44.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.407+0000 7f2b2959a700 1 -- 172.21.15.16:0/2114917011 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2749473149 0 0) 0x7f2b2c0cedb0 con 0x7f2b2c0ce160 2023-10-16T13:46:44.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.407+0000 7f2b2959a700 1 -- 172.21.15.16:0/2114917011 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b180035b0 con 0x7f2b2c0ce160 2023-10-16T13:46:44.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.407+0000 7f2b29d9b700 1 -- 172.21.15.16:0/2114917011 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2b2c15db40 msgr2=0x7f2b2c0cb980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:44.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.407+0000 7f2b29d9b700 1 --2- 172.21.15.16:0/2114917011 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2b2c15db40 0x7f2b2c0cb980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:44.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.407+0000 7f2b29d9b700 1 -- 172.21.15.16:0/2114917011 >> v1:172.21.15.16:6789/0 conn(0x7f2b2c0ce160 legacy=0x7f2b2c0ce5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:44.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.407+0000 7f2b29d9b700 1 -- 172.21.15.16:0/2114917011 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b2c0d12c0 con 0x7f2b2c06a7e0 2023-10-16T13:46:44.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.407+0000 7f2b2a59c700 1 --2- 172.21.15.16:0/2114917011 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2b2c15db40 0x7f2b2c0cb980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:46:44.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.407+0000 7f2b29d9b700 1 --2- 172.21.15.16:0/2114917011 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2b2c06a7e0 0x7f2b2c06ac60 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f2b1c009840 tx=0x7f2b1c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a6576ccbd1f44d23 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:44.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.407+0000 7f2b2959a700 1 -- 172.21.15.16:0/2114917011 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b1c011830 con 0x7f2b2c06a7e0 2023-10-16T13:46:44.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.407+0000 7f2b2959a700 1 -- 172.21.15.16:0/2114917011 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b1c011990 con 0x7f2b2c06a7e0 2023-10-16T13:46:44.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.407+0000 7f2b2959a700 1 -- 172.21.15.16:0/2114917011 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b1c011d00 con 0x7f2b2c06a7e0 2023-10-16T13:46:44.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.408+0000 7f2b3080a700 1 -- 172.21.15.16:0/2114917011 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2b2c06a7e0 msgr2=0x7f2b2c06ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:44.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.408+0000 7f2b3080a700 1 --2- 172.21.15.16:0/2114917011 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2b2c06a7e0 0x7f2b2c06ac60 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f2b1c009840 tx=0x7f2b1c009b50 comp rx=0 tx=0).stop 2023-10-16T13:46:44.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.408+0000 7f2b3080a700 1 -- 172.21.15.16:0/2114917011 shutdown_connections 2023-10-16T13:46:44.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.408+0000 7f2b3080a700 1 --2- 172.21.15.16:0/2114917011 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2b2c15db40 0x7f2b2c0cb980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:44.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.408+0000 7f2b3080a700 1 --2- 172.21.15.16:0/2114917011 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2b2c06a7e0 0x7f2b2c06ac60 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:44.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.408+0000 7f2b3080a700 1 -- 172.21.15.16:0/2114917011 >> 172.21.15.16:0/2114917011 conn(0x7f2b2c0b1fd0 msgr2=0x7f2b2c0b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:44.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.408+0000 7f2b3080a700 1 -- 172.21.15.16:0/2114917011 shutdown_connections 2023-10-16T13:46:44.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.408+0000 7f2b3080a700 1 -- 172.21.15.16:0/2114917011 wait complete. 2023-10-16T13:46:44.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.409+0000 7f2b3080a700 1 Processor -- start 2023-10-16T13:46:44.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.409+0000 7f2b3080a700 1 -- start start 2023-10-16T13:46:44.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.410+0000 7f2b3080a700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2b2c06a7e0 0x7f2b2c0b0930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:44.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.410+0000 7f2b3080a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2b2c0ce160 0x7f2b2c14b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:44.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.410+0000 7f2b3080a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b2c15db40 0x7f2b2c14bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:44.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.410+0000 7f2b3080a700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b2c0b19e0 con 0x7f2b2c15db40 2023-10-16T13:46:44.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.410+0000 7f2b3080a700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b2c0c6cf0 con 0x7f2b2c06a7e0 2023-10-16T13:46:44.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.410+0000 7f2b3080a700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b2c14c6d0 con 0x7f2b2c0ce160 2023-10-16T13:46:44.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.410+0000 7f2b2ad9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b2c15db40 0x7f2b2c14bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:44.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.410+0000 7f2b2ad9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b2c15db40 0x7f2b2c14bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53664/0 (socket says 172.21.15.16:53664) 2023-10-16T13:46:44.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.410+0000 7f2b2ad9d700 1 -- 172.21.15.16:0/3386799998 learned_addr learned my addr 172.21.15.16:0/3386799998 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:44.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.410+0000 7f2b29d9b700 1 --2- 172.21.15.16:0/3386799998 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2b2c0ce160 0x7f2b2c14b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:44.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.410+0000 7f2b2a59c700 1 --2- 172.21.15.16:0/3386799998 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2b2c06a7e0 0x7f2b2c0b0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:44.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.411+0000 7f2b2ad9d700 1 -- 172.21.15.16:0/3386799998 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2b2c0ce160 msgr2=0x7f2b2c14b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:44.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.411+0000 7f2b2ad9d700 1 --2- 172.21.15.16:0/3386799998 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2b2c0ce160 0x7f2b2c14b7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:44.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.411+0000 7f2b2ad9d700 1 -- 172.21.15.16:0/3386799998 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2b2c06a7e0 msgr2=0x7f2b2c0b0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:44.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.411+0000 7f2b2ad9d700 1 --2- 172.21.15.16:0/3386799998 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2b2c06a7e0 0x7f2b2c0b0930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:44.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.411+0000 7f2b2ad9d700 1 -- 172.21.15.16:0/3386799998 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b140094a0 con 0x7f2b2c15db40 2023-10-16T13:46:44.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.411+0000 7f2b2ad9d700 1 --2- 172.21.15.16:0/3386799998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b2c15db40 0x7f2b2c14bcf0 secure :-1 s=READY pgs=2536 cs=0 l=1 rev1=1 crypto rx=0x7f2b2400c9f0 tx=0x7f2b2400cd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:44.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.411+0000 7f2b22ffd700 1 -- 172.21.15.16:0/3386799998 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b2400a9e0 con 0x7f2b2c15db40 2023-10-16T13:46:44.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.411+0000 7f2b22ffd700 1 -- 172.21.15.16:0/3386799998 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b2400ce90 con 0x7f2b2c15db40 2023-10-16T13:46:44.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.411+0000 7f2b3080a700 1 -- 172.21.15.16:0/3386799998 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b1c0094a0 con 0x7f2b2c15db40 2023-10-16T13:46:44.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.411+0000 7f2b22ffd700 1 -- 172.21.15.16:0/3386799998 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b2400ad60 con 0x7f2b2c15db40 2023-10-16T13:46:44.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.411+0000 7f2b3080a700 1 -- 172.21.15.16:0/3386799998 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b2c14cc90 con 0x7f2b2c15db40 2023-10-16T13:46:44.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.412+0000 7f2b3080a700 1 -- 172.21.15.16:0/3386799998 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b2c04ebb0 con 0x7f2b2c15db40 2023-10-16T13:46:44.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.412+0000 7f2b22ffd700 1 -- 172.21.15.16:0/3386799998 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 138) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2b2401b3b0 con 0x7f2b2c15db40 2023-10-16T13:46:44.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.413+0000 7f2b22ffd700 1 --2- 172.21.15.16:0/3386799998 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f2b0c06b020 0x7f2b0c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:44.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.413+0000 7f2b22ffd700 1 -- 172.21.15.16:0/3386799998 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 7021+0+0 (secure 0 0 0) 0x7f2b24089890 con 0x7f2b2c15db40 2023-10-16T13:46:44.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.413+0000 7f2b2a59c700 1 --2- 172.21.15.16:0/3386799998 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f2b0c06b020 0x7f2b0c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:44.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.413+0000 7f2b2a59c700 1 --2- 172.21.15.16:0/3386799998 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f2b0c06b020 0x7f2b0c06d4e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2b14009d70 tx=0x7f2b14009210 comp rx=0 tx=0).ready entity=mgr.11190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:44.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.418+0000 7f2b22ffd700 1 -- 172.21.15.16:0/3386799998 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2b24054270 con 0x7f2b2c15db40 2023-10-16T13:46:44.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.649+0000 7f2b3080a700 1 -- 172.21.15.16:0/3386799998 --> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f2b2c0caea0 con 0x7f2b0c06b020 2023-10-16T13:46:44.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.651+0000 7f2b22ffd700 1 -- 172.21.15.16:0/3386799998 <== mgr.11190 v2:172.21.15.16:6832/3171422553 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+82 (secure 0 0 0) 0x7f2b140099c0 con 0x7f2b0c06b020 2023-10-16T13:46:44.654 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2023-10-16T13:46:44.655 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:46:44.655 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cephfs" 2023-10-16T13:46:44.655 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:46:44.655 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:46:44.655 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "backup_fs" 2023-10-16T13:46:44.655 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:46:44.656 INFO:teuthology.orchestra.run.smithi016.stdout:] 2023-10-16T13:46:44.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.653+0000 7f2b3080a700 1 -- 172.21.15.16:0/3386799998 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f2b0c06b020 msgr2=0x7f2b0c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:44.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.653+0000 7f2b3080a700 1 --2- 172.21.15.16:0/3386799998 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f2b0c06b020 0x7f2b0c06d4e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2b14009d70 tx=0x7f2b14009210 comp rx=0 tx=0).stop 2023-10-16T13:46:44.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.653+0000 7f2b3080a700 1 -- 172.21.15.16:0/3386799998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b2c15db40 msgr2=0x7f2b2c14bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:44.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.653+0000 7f2b3080a700 1 --2- 172.21.15.16:0/3386799998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b2c15db40 0x7f2b2c14bcf0 secure :-1 s=READY pgs=2536 cs=0 l=1 rev1=1 crypto rx=0x7f2b2400c9f0 tx=0x7f2b2400cd00 comp rx=0 tx=0).stop 2023-10-16T13:46:44.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.653+0000 7f2b3080a700 1 -- 172.21.15.16:0/3386799998 shutdown_connections 2023-10-16T13:46:44.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.653+0000 7f2b3080a700 1 --2- 172.21.15.16:0/3386799998 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f2b0c06b020 0x7f2b0c06d4e0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:44.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.653+0000 7f2b3080a700 1 --2- 172.21.15.16:0/3386799998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b2c15db40 0x7f2b2c14bcf0 unknown :-1 s=CLOSED pgs=2536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:44.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.653+0000 7f2b3080a700 1 --2- 172.21.15.16:0/3386799998 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2b2c06a7e0 0x7f2b2c0b0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:44.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.653+0000 7f2b3080a700 1 --2- 172.21.15.16:0/3386799998 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2b2c0ce160 0x7f2b2c14b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:44.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.653+0000 7f2b3080a700 1 -- 172.21.15.16:0/3386799998 >> 172.21.15.16:0/3386799998 conn(0x7f2b2c0b1fd0 msgr2=0x7f2b2c0cd650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:44.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.653+0000 7f2b3080a700 1 -- 172.21.15.16:0/3386799998 shutdown_connections 2023-10-16T13:46:44.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.653+0000 7f2b3080a700 1 -- 172.21.15.16:0/3386799998 wait complete. 2023-10-16T13:46:44.669 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-16T13:46:44.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.792+0000 7f181dbdb700 1 Processor -- start 2023-10-16T13:46:44.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.793+0000 7f181dbdb700 1 -- start start 2023-10-16T13:46:44.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.793+0000 7f181dbdb700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f181815ec80 0x7f18180cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:44.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.793+0000 7f181dbdb700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18180b2440 con 0x7f18180d2110 2023-10-16T13:46:44.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.793+0000 7f181dbdb700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18180d00e0 con 0x7f181815f060 2023-10-16T13:46:44.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.793+0000 7f181dbdb700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18180b2c50 con 0x7f181815ec80 2023-10-16T13:46:44.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.794+0000 7f1817fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f18180d2110 0x7f18180b4b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42946/0 (socket says 172.21.15.16:42946) 2023-10-16T13:46:44.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.794+0000 7f1817fff700 1 -- 172.21.15.16:0/3051934669 learned_addr learned my addr 172.21.15.16:0/3051934669 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:46:44.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.794+0000 7f18167fc700 1 -- 172.21.15.16:0/3051934669 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 418537483 0 0) 0x7f18180b2440 con 0x7f18180d2110 2023-10-16T13:46:44.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.794+0000 7f18167fc700 1 -- 172.21.15.16:0/3051934669 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18000035b0 con 0x7f18180d2110 2023-10-16T13:46:44.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.794+0000 7f18177fe700 1 --2- 172.21.15.16:0/3051934669 >> v2:172.21.15.49:3300/0 conn(0x7f181815ec80 0x7f18180cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:44.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.794+0000 7f18167fc700 1 -- 172.21.15.16:0/3051934669 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3230552638 0 0) 0x7f18000035b0 con 0x7f18180d2110 2023-10-16T13:46:44.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.794+0000 7f18167fc700 1 -- 172.21.15.16:0/3051934669 >> v2:172.21.15.49:3300/0 conn(0x7f181815ec80 msgr2=0x7f18180cdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:44.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.794+0000 7f18167fc700 1 --2- 172.21.15.16:0/3051934669 >> v2:172.21.15.49:3300/0 conn(0x7f181815ec80 0x7f18180cdec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:44.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.794+0000 7f18167fc700 1 -- 172.21.15.16:0/3051934669 >> v1:172.21.15.49:6789/0 conn(0x7f181815f060 legacy=0x7f18180b4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:44.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.794+0000 7f18167fc700 1 -- 172.21.15.16:0/3051934669 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18180cef60 con 0x7f18180d2110 2023-10-16T13:46:44.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.794+0000 7f18167fc700 1 -- 172.21.15.16:0/3051934669 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f18100019f0 con 0x7f18180d2110 2023-10-16T13:46:44.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.794+0000 7f18167fc700 1 -- 172.21.15.16:0/3051934669 >> v1:172.21.15.16:6789/0 conn(0x7f18180d2110 legacy=0x7f18180b4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:44.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.794+0000 7f18167fc700 1 --2- 172.21.15.16:0/3051934669 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1800003a00 0x7f1800003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:44.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.794+0000 7f18167fc700 1 --2- 172.21.15.16:0/3051934669 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1800004850 0x7f1800004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:44.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.794+0000 7f18167fc700 1 --2- 172.21.15.16:0/3051934669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1800005210 0x7f180000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:44.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.794+0000 7f18167fc700 1 -- 172.21.15.16:0/3051934669 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18180b2c50 con 0x7f1800005210 2023-10-16T13:46:44.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.794+0000 7f18167fc700 1 -- 172.21.15.16:0/3051934669 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f180000c0b0 con 0x7f1800004850 2023-10-16T13:46:44.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.794+0000 7f18167fc700 1 -- 172.21.15.16:0/3051934669 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f180000c270 con 0x7f1800003a00 2023-10-16T13:46:44.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.794+0000 7f1817fff700 1 --2- 172.21.15.16:0/3051934669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1800005210 0x7f180000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:44.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.794+0000 7f18177fe700 1 --2- 172.21.15.16:0/3051934669 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1800003a00 0x7f1800003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:44.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.794+0000 7f1816ffd700 1 --2- 172.21.15.16:0/3051934669 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1800004850 0x7f1800004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:44.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.795+0000 7f1817fff700 1 -- 172.21.15.16:0/3051934669 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1800003a00 msgr2=0x7f1800003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:44.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.795+0000 7f1817fff700 1 --2- 172.21.15.16:0/3051934669 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1800003a00 0x7f1800003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:44.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.795+0000 7f1817fff700 1 -- 172.21.15.16:0/3051934669 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1800004850 msgr2=0x7f1800004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:44.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.795+0000 7f1817fff700 1 --2- 172.21.15.16:0/3051934669 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1800004850 0x7f1800004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:44.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.795+0000 7f1817fff700 1 -- 172.21.15.16:0/3051934669 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18000035b0 con 0x7f1800005210 2023-10-16T13:46:44.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.795+0000 7f1817fff700 1 -- 172.21.15.16:0/3051934669 reap_dead start 2023-10-16T13:46:44.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.795+0000 7f1817fff700 1 --2- 172.21.15.16:0/3051934669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1800005210 0x7f180000ba10 secure :-1 s=READY pgs=2538 cs=0 l=1 rev1=1 crypto rx=0x7f181000a7f0 tx=0x7f1800005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d524126911e2a01 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:44.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.795+0000 7f18167fc700 1 -- 172.21.15.16:0/3051934669 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1810012600 con 0x7f1800005210 2023-10-16T13:46:44.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.795+0000 7f18167fc700 1 -- 172.21.15.16:0/3051934669 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1810012780 con 0x7f1800005210 2023-10-16T13:46:44.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.795+0000 7f18167fc700 1 -- 172.21.15.16:0/3051934669 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1810012b30 con 0x7f1800005210 2023-10-16T13:46:44.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.795+0000 7f181dbdb700 1 -- 172.21.15.16:0/3051934669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1800005210 msgr2=0x7f180000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:44.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.795+0000 7f181dbdb700 1 --2- 172.21.15.16:0/3051934669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1800005210 0x7f180000ba10 secure :-1 s=READY pgs=2538 cs=0 l=1 rev1=1 crypto rx=0x7f181000a7f0 tx=0x7f1800005880 comp rx=0 tx=0).stop 2023-10-16T13:46:44.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.796+0000 7f181dbdb700 1 -- 172.21.15.16:0/3051934669 shutdown_connections 2023-10-16T13:46:44.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.796+0000 7f181dbdb700 1 --2- 172.21.15.16:0/3051934669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1800005210 0x7f180000ba10 unknown :-1 s=CLOSED pgs=2538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:44.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.796+0000 7f181dbdb700 1 -- 172.21.15.16:0/3051934669 >> 172.21.15.16:0/3051934669 conn(0x7f18180d15f0 msgr2=0x7f18180d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:44.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.796+0000 7f181dbdb700 1 -- 172.21.15.16:0/3051934669 shutdown_connections 2023-10-16T13:46:44.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.796+0000 7f181dbdb700 1 -- 172.21.15.16:0/3051934669 wait complete. 2023-10-16T13:46:44.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.797+0000 7f181dbdb700 1 Processor -- start 2023-10-16T13:46:44.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.797+0000 7f181dbdb700 1 -- start start 2023-10-16T13:46:44.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.798+0000 7f181dbdb700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1800005210 0x7f18180cb460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:44.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.798+0000 7f181dbdb700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1818152690 0x7f1818151f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:44.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.798+0000 7f181dbdb700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f18180c8eb0 0x7f18180c9340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:44.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.798+0000 7f181dbdb700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1818153ad0 con 0x7f1818152690 2023-10-16T13:46:44.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.798+0000 7f181dbdb700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18180c9880 con 0x7f1800005210 2023-10-16T13:46:44.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.798+0000 7f181dbdb700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1818152490 con 0x7f18180c8eb0 2023-10-16T13:46:44.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.798+0000 7f1816ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1818152690 0x7f1818151f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:44.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.798+0000 7f1816ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1818152690 0x7f1818151f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53682/0 (socket says 172.21.15.16:53682) 2023-10-16T13:46:44.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.798+0000 7f1816ffd700 1 -- 172.21.15.16:0/664565820 learned_addr learned my addr 172.21.15.16:0/664565820 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:44.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.798+0000 7f18177fe700 1 --2- 172.21.15.16:0/664565820 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1800005210 0x7f18180cb460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:44.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.798+0000 7f1817fff700 1 --2- 172.21.15.16:0/664565820 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f18180c8eb0 0x7f18180c9340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:44.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.798+0000 7f1816ffd700 1 -- 172.21.15.16:0/664565820 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f18180c8eb0 msgr2=0x7f18180c9340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:44.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.798+0000 7f1816ffd700 1 --2- 172.21.15.16:0/664565820 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f18180c8eb0 0x7f18180c9340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:44.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.798+0000 7f1816ffd700 1 -- 172.21.15.16:0/664565820 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1800005210 msgr2=0x7f18180cb460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:44.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.798+0000 7f1816ffd700 1 --2- 172.21.15.16:0/664565820 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1800005210 0x7f18180cb460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:44.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.798+0000 7f1816ffd700 1 -- 172.21.15.16:0/664565820 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1818153ef0 con 0x7f1818152690 2023-10-16T13:46:44.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.798+0000 7f1816ffd700 1 --2- 172.21.15.16:0/664565820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1818152690 0x7f1818151f50 secure :-1 s=READY pgs=2539 cs=0 l=1 rev1=1 crypto rx=0x7f180800f610 tx=0x7f180800f920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:44.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.798+0000 7f17fffff700 1 -- 172.21.15.16:0/664565820 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18080052e0 con 0x7f1818152690 2023-10-16T13:46:44.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.798+0000 7f17fffff700 1 -- 172.21.15.16:0/664565820 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1808005440 con 0x7f1818152690 2023-10-16T13:46:44.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.798+0000 7f181dbdb700 1 -- 172.21.15.16:0/664565820 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1818152e50 con 0x7f1818152690 2023-10-16T13:46:44.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.798+0000 7f17fffff700 1 -- 172.21.15.16:0/664565820 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18080038a0 con 0x7f1818152690 2023-10-16T13:46:44.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.798+0000 7f181dbdb700 1 -- 172.21.15.16:0/664565820 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1818153410 con 0x7f1818152690 2023-10-16T13:46:44.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.799+0000 7f181dbdb700 1 -- 172.21.15.16:0/664565820 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f181804ebb0 con 0x7f1818152690 2023-10-16T13:46:44.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.800+0000 7f17fffff700 1 -- 172.21.15.16:0/664565820 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 138) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1808003a00 con 0x7f1818152690 2023-10-16T13:46:44.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.800+0000 7f17fffff700 1 --2- 172.21.15.16:0/664565820 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f180406b020 0x7f180406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:44.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.801+0000 7f17fffff700 1 -- 172.21.15.16:0/664565820 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 7021+0+0 (secure 0 0 0) 0x7f1808089800 con 0x7f1818152690 2023-10-16T13:46:44.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.801+0000 7f18177fe700 1 --2- 172.21.15.16:0/664565820 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f180406b020 0x7f180406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:44.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.801+0000 7f18177fe700 1 --2- 172.21.15.16:0/664565820 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f180406b020 0x7f180406d4e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f180c009be0 tx=0x7f180c009210 comp rx=0 tx=0).ready entity=mgr.11190 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:44.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:44.805+0000 7f17fffff700 1 -- 172.21.15.16:0/664565820 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1808054310 con 0x7f1818152690 2023-10-16T13:46:45.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:45.066+0000 7f181dbdb700 1 -- 172.21.15.16:0/664565820 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f1818154140 con 0x7f1818152690 2023-10-16T13:46:45.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:45.676+0000 7f17fffff700 1 -- 172.21.15.16:0/664565820 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v139) v1 ==== 92+0+0 (secure 0 0 0) 0x7f180805ce60 con 0x7f1818152690 2023-10-16T13:46:45.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:45.680+0000 7f181dbdb700 1 -- 172.21.15.16:0/664565820 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f180406b020 msgr2=0x7f180406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:45.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:45.680+0000 7f181dbdb700 1 --2- 172.21.15.16:0/664565820 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f180406b020 0x7f180406d4e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f180c009be0 tx=0x7f180c009210 comp rx=0 tx=0).stop 2023-10-16T13:46:45.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:45.680+0000 7f181dbdb700 1 -- 172.21.15.16:0/664565820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1818152690 msgr2=0x7f1818151f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:45.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:45.680+0000 7f181dbdb700 1 --2- 172.21.15.16:0/664565820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1818152690 0x7f1818151f50 secure :-1 s=READY pgs=2539 cs=0 l=1 rev1=1 crypto rx=0x7f180800f610 tx=0x7f180800f920 comp rx=0 tx=0).stop 2023-10-16T13:46:45.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:45.680+0000 7f181dbdb700 1 -- 172.21.15.16:0/664565820 shutdown_connections 2023-10-16T13:46:45.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:45.680+0000 7f181dbdb700 1 --2- 172.21.15.16:0/664565820 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f18180c8eb0 0x7f18180c9340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:45.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:45.680+0000 7f181dbdb700 1 --2- 172.21.15.16:0/664565820 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1800005210 0x7f18180cb460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:45.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:45.680+0000 7f181dbdb700 1 --2- 172.21.15.16:0/664565820 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f180406b020 0x7f180406d4e0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:45.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:45.680+0000 7f181dbdb700 1 --2- 172.21.15.16:0/664565820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1818152690 0x7f1818151f50 secure :-1 s=CLOSED pgs=2539 cs=0 l=1 rev1=1 crypto rx=0x7f180800f610 tx=0x7f180800f920 comp rx=0 tx=0).stop 2023-10-16T13:46:45.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:45.680+0000 7f181dbdb700 1 -- 172.21.15.16:0/664565820 >> 172.21.15.16:0/664565820 conn(0x7f18180d15f0 msgr2=0x7f18180d19d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:45.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:45.681+0000 7f181dbdb700 1 -- 172.21.15.16:0/664565820 shutdown_connections 2023-10-16T13:46:45.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:45.681+0000 7f181dbdb700 1 -- 172.21.15.16:0/664565820 wait complete. 2023-10-16T13:46:45.701 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-16T13:46:45.831 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:46:45.828+0000 7f4f5ed5b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T13:46:45.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:45.843+0000 7f9a96b5a700 1 Processor -- start 2023-10-16T13:46:45.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:45.843+0000 7f9a96b5a700 1 -- start start 2023-10-16T13:46:45.847 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:45.843+0000 7f45c6578200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T13:46:45.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:45.844+0000 7f9a96b5a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9a9006a7e0 0x7f9a9006ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:45.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:45.844+0000 7f9a96b5a700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a9015ecb0 con 0x7f9a9015b2c0 2023-10-16T13:46:45.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:45.844+0000 7f9a96b5a700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a9015b6a0 con 0x7f9a9006e570 2023-10-16T13:46:45.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:45.844+0000 7f9a96b5a700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a9006b190 con 0x7f9a9006a7e0 2023-10-16T13:46:45.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:45.844+0000 7f9a950f7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9a9006a7e0 0x7f9a9006ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:45.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:45.844+0000 7f9a948f6700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f9a9015b2c0 0x7f9a90161340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42988/0 (socket says 172.21.15.16:42988) 2023-10-16T13:46:45.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:45.844+0000 7f9a948f6700 1 -- 172.21.15.16:0/1350092626 learned_addr learned my addr 172.21.15.16:0/1350092626 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:46:45.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:45.845+0000 7f9a950f7700 1 -- 172.21.15.16:0/1350092626 >> v1:172.21.15.49:6789/0 conn(0x7f9a9006e570 legacy=0x7f9a9015e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:45.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:45.845+0000 7f9a950f7700 1 -- 172.21.15.16:0/1350092626 >> v1:172.21.15.16:6789/0 conn(0x7f9a9015b2c0 legacy=0x7f9a90161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:45.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:45.845+0000 7f9a950f7700 1 -- 172.21.15.16:0/1350092626 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a90161b20 con 0x7f9a9006a7e0 2023-10-16T13:46:45.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:45.845+0000 7f9a950f7700 1 --2- 172.21.15.16:0/1350092626 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a9006a7e0 0x7f9a9006ac50 secure :-1 s=READY pgs=2544 cs=0 l=1 rev1=1 crypto rx=0x7f9a88009d90 tx=0x7f9a8800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad70d3c9df00557d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:45.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:45.845+0000 7f9a8f7fe700 1 -- 172.21.15.16:0/1350092626 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a8800e070 con 0x7f9a9006a7e0 2023-10-16T13:46:45.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:45.845+0000 7f9a8f7fe700 1 -- 172.21.15.16:0/1350092626 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a8800bbd0 con 0x7f9a9006a7e0 2023-10-16T13:46:45.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:45.845+0000 7f9a8f7fe700 1 -- 172.21.15.16:0/1350092626 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a88012430 con 0x7f9a9006a7e0 2023-10-16T13:46:45.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:45.846+0000 7f9a96b5a700 1 -- 172.21.15.16:0/1350092626 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a9006a7e0 msgr2=0x7f9a9006ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:45.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:45.846+0000 7f9a96b5a700 1 --2- 172.21.15.16:0/1350092626 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a9006a7e0 0x7f9a9006ac50 secure :-1 s=READY pgs=2544 cs=0 l=1 rev1=1 crypto rx=0x7f9a88009d90 tx=0x7f9a8800b330 comp rx=0 tx=0).stop 2023-10-16T13:46:45.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:45.846+0000 7f9a96b5a700 1 -- 172.21.15.16:0/1350092626 shutdown_connections 2023-10-16T13:46:45.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:45.846+0000 7f9a96b5a700 1 --2- 172.21.15.16:0/1350092626 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a9006a7e0 0x7f9a9006ac50 unknown :-1 s=CLOSED pgs=2544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:45.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:45.846+0000 7f9a96b5a700 1 -- 172.21.15.16:0/1350092626 >> 172.21.15.16:0/1350092626 conn(0x7f9a900b1f40 msgr2=0x7f9a900b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:45.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:45.846+0000 7f9a96b5a700 1 -- 172.21.15.16:0/1350092626 shutdown_connections 2023-10-16T13:46:45.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:45.846+0000 7f9a96b5a700 1 -- 172.21.15.16:0/1350092626 wait complete. 2023-10-16T13:46:45.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:45.847+0000 7f9a96b5a700 1 Processor -- start 2023-10-16T13:46:45.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:45.847+0000 7f9a96b5a700 1 -- start start 2023-10-16T13:46:45.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:45.848+0000 7f9a96b5a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a9006a7e0 0x7f9a900c75e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:45.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:45.848+0000 7f9a96b5a700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9a9006e570 0x7f9a900c6490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:45.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:45.848+0000 7f9a96b5a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9a9015b2c0 0x7f9a900c69d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:45.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:45.848+0000 7f9a96b5a700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a900c5150 con 0x7f9a9006a7e0 2023-10-16T13:46:45.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:45.848+0000 7f9a96b5a700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a900c53f0 con 0x7f9a9006e570 2023-10-16T13:46:45.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:45.848+0000 7f9a96b5a700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a900c56d0 con 0x7f9a9015b2c0 2023-10-16T13:46:45.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:45.848+0000 7f9a948f6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a9006a7e0 0x7f9a900c75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:45.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:45.848+0000 7f9a948f6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a9006a7e0 0x7f9a900c75e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53718/0 (socket says 172.21.15.16:53718) 2023-10-16T13:46:45.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:45.848+0000 7f9a948f6700 1 -- 172.21.15.16:0/227236568 learned_addr learned my addr 172.21.15.16:0/227236568 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:45.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:45.848+0000 7f9a8ffff700 1 --2- 172.21.15.16:0/227236568 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9a9006e570 0x7f9a900c6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:45.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:45.848+0000 7f9a950f7700 1 --2- 172.21.15.16:0/227236568 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9a9015b2c0 0x7f9a900c69d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:45.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:45.849+0000 7f9a948f6700 1 -- 172.21.15.16:0/227236568 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9a9015b2c0 msgr2=0x7f9a900c69d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:45.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:45.849+0000 7f9a948f6700 1 --2- 172.21.15.16:0/227236568 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9a9015b2c0 0x7f9a900c69d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:45.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:45.849+0000 7f9a948f6700 1 -- 172.21.15.16:0/227236568 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9a9006e570 msgr2=0x7f9a900c6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:45.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:45.849+0000 7f9a948f6700 1 --2- 172.21.15.16:0/227236568 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9a9006e570 0x7f9a900c6490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:45.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:45.849+0000 7f9a948f6700 1 -- 172.21.15.16:0/227236568 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a88009a70 con 0x7f9a9006a7e0 2023-10-16T13:46:45.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:45.849+0000 7f9a948f6700 1 --2- 172.21.15.16:0/227236568 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a9006a7e0 0x7f9a900c75e0 secure :-1 s=READY pgs=2545 cs=0 l=1 rev1=1 crypto rx=0x7f9a8400c520 tx=0x7f9a8400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:45.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:45.849+0000 7f9a8d7fa700 1 -- 172.21.15.16:0/227236568 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a84005d70 con 0x7f9a9006a7e0 2023-10-16T13:46:45.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:45.849+0000 7f9a8d7fa700 1 -- 172.21.15.16:0/227236568 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a840070d0 con 0x7f9a9006a7e0 2023-10-16T13:46:45.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:45.849+0000 7f9a96b5a700 1 -- 172.21.15.16:0/227236568 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a900c59b0 con 0x7f9a9006a7e0 2023-10-16T13:46:45.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:45.849+0000 7f9a8d7fa700 1 -- 172.21.15.16:0/227236568 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a84007440 con 0x7f9a9006a7e0 2023-10-16T13:46:45.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:45.849+0000 7f9a96b5a700 1 -- 172.21.15.16:0/227236568 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a900c5f70 con 0x7f9a9006a7e0 2023-10-16T13:46:45.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:45.850+0000 7f9a96b5a700 1 -- 172.21.15.16:0/227236568 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a9004ebb0 con 0x7f9a9006a7e0 2023-10-16T13:46:45.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:45.850+0000 7f9a8d7fa700 1 -- 172.21.15.16:0/227236568 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 139) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f9a840075a0 con 0x7f9a9006a7e0 2023-10-16T13:46:45.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:45.851+0000 7f9a8d7fa700 1 --2- 172.21.15.16:0/227236568 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f9a7406b070 0x7f9a7406d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:45.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:45.851+0000 7f9a8ffff700 1 -- 172.21.15.16:0/227236568 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f9a7406b070 msgr2=0x7f9a7406d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3171422553 2023-10-16T13:46:45.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:45.851+0000 7f9a8d7fa700 1 -- 172.21.15.16:0/227236568 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 7021+0+0 (secure 0 0 0) 0x7f9a84089060 con 0x7f9a9006a7e0 2023-10-16T13:46:45.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:45.851+0000 7f9a8ffff700 1 --2- 172.21.15.16:0/227236568 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f9a7406b070 0x7f9a7406d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:46:45.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:45.855+0000 7f9a8d7fa700 1 -- 172.21.15.16:0/227236568 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9a84053a80 con 0x7f9a9006a7e0 2023-10-16T13:46:45.900 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:46:45.898+0000 7f4f5ed5b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T13:46:45.929 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:45.926+0000 7f45c6578200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T13:46:45.976 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:46:45.973+0000 7f4f5ed5b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T13:46:46.004 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:46.001+0000 7f45c6578200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T13:46:46.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.052+0000 7f9a8ffff700 1 -- 172.21.15.16:0/227236568 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f9a7406b070 msgr2=0x7f9a7406d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3171422553 2023-10-16T13:46:46.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.052+0000 7f9a8ffff700 1 --2- 172.21.15.16:0/227236568 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f9a7406b070 0x7f9a7406d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:46:46.056 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:46:46.054+0000 7f4f5ed5b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T13:46:46.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.074+0000 7f9a96b5a700 1 -- 172.21.15.16:0/227236568 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f9a900b6370 con 0x7f9a9006a7e0 2023-10-16T13:46:46.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.077+0000 7f9a8d7fa700 1 -- 172.21.15.16:0/227236568 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v77)=0 v77) v1 ==== 138+0+0 (secure 0 0 0) 0x7f9a840570a0 con 0x7f9a9006a7e0 2023-10-16T13:46:46.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.079+0000 7f9a96b5a700 1 -- 172.21.15.16:0/227236568 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f9a7406b070 msgr2=0x7f9a7406d530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:46:46.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.079+0000 7f9a96b5a700 1 --2- 172.21.15.16:0/227236568 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f9a7406b070 0x7f9a7406d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:46.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.079+0000 7f9a96b5a700 1 -- 172.21.15.16:0/227236568 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a9006a7e0 msgr2=0x7f9a900c75e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:46.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.079+0000 7f9a96b5a700 1 --2- 172.21.15.16:0/227236568 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a9006a7e0 0x7f9a900c75e0 secure :-1 s=READY pgs=2545 cs=0 l=1 rev1=1 crypto rx=0x7f9a8400c520 tx=0x7f9a8400c830 comp rx=0 tx=0).stop 2023-10-16T13:46:46.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.079+0000 7f9a96b5a700 1 -- 172.21.15.16:0/227236568 shutdown_connections 2023-10-16T13:46:46.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.079+0000 7f9a96b5a700 1 --2- 172.21.15.16:0/227236568 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9a9015b2c0 0x7f9a900c69d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:46.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.079+0000 7f9a96b5a700 1 --2- 172.21.15.16:0/227236568 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f9a7406b070 0x7f9a7406d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:46.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.079+0000 7f9a96b5a700 1 --2- 172.21.15.16:0/227236568 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a9006a7e0 0x7f9a900c75e0 unknown :-1 s=CLOSED pgs=2545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:46.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.079+0000 7f9a96b5a700 1 --2- 172.21.15.16:0/227236568 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9a9006e570 0x7f9a900c6490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:46.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.079+0000 7f9a96b5a700 1 -- 172.21.15.16:0/227236568 >> 172.21.15.16:0/227236568 conn(0x7f9a900b1f40 msgr2=0x7f9a900cce00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:46.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.079+0000 7f9a96b5a700 1 -- 172.21.15.16:0/227236568 shutdown_connections 2023-10-16T13:46:46.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.079+0000 7f9a96b5a700 1 -- 172.21.15.16:0/227236568 wait complete. 2023-10-16T13:46:46.086 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:46.083+0000 7f45c6578200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T13:46:46.096 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-16T13:46:46.204 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:46:46.201+0000 7f4f5ed5b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T13:46:46.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.222+0000 7eff1cc37700 1 Processor -- start 2023-10-16T13:46:46.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.222+0000 7eff1cc37700 1 -- start start 2023-10-16T13:46:46.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.223+0000 7eff1cc37700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7eff180d2430 0x7eff180d1520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:46.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.223+0000 7eff1cc37700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff180d4540 con 0x7eff1815da10 2023-10-16T13:46:46.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.223+0000 7eff1cc37700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff180ce7e0 con 0x7eff180d4060 2023-10-16T13:46:46.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.223+0000 7eff1cc37700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff180cf9b0 con 0x7eff180d2430 2023-10-16T13:46:46.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.223+0000 7eff1659c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7eff1815da10 0x7eff180cf2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:43046/0 (socket says 172.21.15.16:43046) 2023-10-16T13:46:46.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.223+0000 7eff1659c700 1 -- 172.21.15.16:0/3601094247 learned_addr learned my addr 172.21.15.16:0/3601094247 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:46:46.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.223+0000 7eff16d9d700 1 --2- 172.21.15.16:0/3601094247 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7eff180d2430 0x7eff180d1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:46.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.223+0000 7eff1559a700 1 -- 172.21.15.16:0/3601094247 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2362171332 0 0) 0x7eff180d4540 con 0x7eff1815da10 2023-10-16T13:46:46.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.224+0000 7eff1559a700 1 -- 172.21.15.16:0/3601094247 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff000035b0 con 0x7eff1815da10 2023-10-16T13:46:46.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.224+0000 7eff1559a700 1 -- 172.21.15.16:0/3601094247 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2670453259 0 0) 0x7eff180ce7e0 con 0x7eff180d4060 2023-10-16T13:46:46.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.224+0000 7eff1559a700 1 -- 172.21.15.16:0/3601094247 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff180d4540 con 0x7eff180d4060 2023-10-16T13:46:46.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.224+0000 7eff16d9d700 1 -- 172.21.15.16:0/3601094247 >> v1:172.21.15.49:6789/0 conn(0x7eff180d4060 legacy=0x7eff180d1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:46.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.224+0000 7eff16d9d700 1 -- 172.21.15.16:0/3601094247 >> v1:172.21.15.16:6789/0 conn(0x7eff1815da10 legacy=0x7eff180cf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:46.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.224+0000 7eff16d9d700 1 -- 172.21.15.16:0/3601094247 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff180ceac0 con 0x7eff180d2430 2023-10-16T13:46:46.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.224+0000 7eff16d9d700 1 --2- 172.21.15.16:0/3601094247 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7eff180d2430 0x7eff180d1520 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7eff10009dc0 tx=0x7eff1000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=18815ab4c6cf321c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:46.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.224+0000 7eff1559a700 1 -- 172.21.15.16:0/3601094247 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff1000e070 con 0x7eff180d2430 2023-10-16T13:46:46.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.224+0000 7eff1559a700 1 -- 172.21.15.16:0/3601094247 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff1000bb20 con 0x7eff180d2430 2023-10-16T13:46:46.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.224+0000 7eff1559a700 1 -- 172.21.15.16:0/3601094247 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff10012430 con 0x7eff180d2430 2023-10-16T13:46:46.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.225+0000 7eff1cc37700 1 -- 172.21.15.16:0/3601094247 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7eff180d2430 msgr2=0x7eff180d1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:46.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.225+0000 7eff1cc37700 1 --2- 172.21.15.16:0/3601094247 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7eff180d2430 0x7eff180d1520 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7eff10009dc0 tx=0x7eff1000b330 comp rx=0 tx=0).stop 2023-10-16T13:46:46.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.225+0000 7eff1cc37700 1 -- 172.21.15.16:0/3601094247 shutdown_connections 2023-10-16T13:46:46.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.225+0000 7eff1cc37700 1 --2- 172.21.15.16:0/3601094247 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7eff180d2430 0x7eff180d1520 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:46.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.225+0000 7eff1cc37700 1 -- 172.21.15.16:0/3601094247 >> 172.21.15.16:0/3601094247 conn(0x7eff1809c9f0 msgr2=0x7eff180d3740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:46.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.225+0000 7eff1cc37700 1 -- 172.21.15.16:0/3601094247 shutdown_connections 2023-10-16T13:46:46.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.225+0000 7eff1cc37700 1 -- 172.21.15.16:0/3601094247 wait complete. 2023-10-16T13:46:46.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.226+0000 7eff1cc37700 1 Processor -- start 2023-10-16T13:46:46.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.226+0000 7eff1cc37700 1 -- start start 2023-10-16T13:46:46.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.227+0000 7eff1cc37700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff180d2430 0x7eff181f4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:46.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.227+0000 7eff1cc37700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7eff180d4060 0x7eff181f5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:46.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.227+0000 7eff1cc37700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7eff1815da10 0x7eff1815f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:46.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.227+0000 7eff1cc37700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff180b81f0 con 0x7eff180d2430 2023-10-16T13:46:46.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.227+0000 7eff1cc37700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff180b7b70 con 0x7eff180d4060 2023-10-16T13:46:46.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.227+0000 7eff1cc37700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff180cbec0 con 0x7eff1815da10 2023-10-16T13:46:46.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.227+0000 7eff1659c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff180d2430 0x7eff181f4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:46.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.227+0000 7eff1659c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff180d2430 0x7eff181f4090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53778/0 (socket says 172.21.15.16:53778) 2023-10-16T13:46:46.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.227+0000 7eff1659c700 1 -- 172.21.15.16:0/1057205353 learned_addr learned my addr 172.21.15.16:0/1057205353 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:46.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.227+0000 7eff16d9d700 1 --2- 172.21.15.16:0/1057205353 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7eff1815da10 0x7eff1815f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:46.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.227+0000 7eff15d9b700 1 --2- 172.21.15.16:0/1057205353 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7eff180d4060 0x7eff181f5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:46.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.227+0000 7eff16d9d700 1 -- 172.21.15.16:0/1057205353 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7eff180d4060 msgr2=0x7eff181f5cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:46.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.227+0000 7eff16d9d700 1 --2- 172.21.15.16:0/1057205353 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7eff180d4060 0x7eff181f5cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:46.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.227+0000 7eff16d9d700 1 -- 172.21.15.16:0/1057205353 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff180d2430 msgr2=0x7eff181f4090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:46.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.227+0000 7eff16d9d700 1 --2- 172.21.15.16:0/1057205353 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff180d2430 0x7eff181f4090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:46.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.227+0000 7eff16d9d700 1 -- 172.21.15.16:0/1057205353 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff10009a70 con 0x7eff1815da10 2023-10-16T13:46:46.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.227+0000 7eff1659c700 1 --2- 172.21.15.16:0/1057205353 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff180d2430 0x7eff181f4090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:46:46.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.227+0000 7eff15d9b700 1 --2- 172.21.15.16:0/1057205353 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7eff180d4060 0x7eff181f5cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:46:46.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.228+0000 7eff16d9d700 1 --2- 172.21.15.16:0/1057205353 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7eff1815da10 0x7eff1815f120 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7eff10016000 tx=0x7eff10004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:46.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.228+0000 7eff06ffd700 1 -- 172.21.15.16:0/1057205353 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff1001b810 con 0x7eff1815da10 2023-10-16T13:46:46.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.228+0000 7eff06ffd700 1 -- 172.21.15.16:0/1057205353 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff10020070 con 0x7eff1815da10 2023-10-16T13:46:46.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.228+0000 7eff1cc37700 1 -- 172.21.15.16:0/1057205353 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff180b7e10 con 0x7eff1815da10 2023-10-16T13:46:46.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.228+0000 7eff06ffd700 1 -- 172.21.15.16:0/1057205353 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff1001bb90 con 0x7eff1815da10 2023-10-16T13:46:46.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.228+0000 7eff1cc37700 1 -- 172.21.15.16:0/1057205353 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff180cc1a0 con 0x7eff1815da10 2023-10-16T13:46:46.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.229+0000 7eff1cc37700 1 -- 172.21.15.16:0/1057205353 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff180b8330 con 0x7eff1815da10 2023-10-16T13:46:46.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.229+0000 7eff06ffd700 1 -- 172.21.15.16:0/1057205353 <== mon.2 v2:172.21.15.49:3301/0 4 ==== mgrmap(e 139) v1 ==== 89348+0+0 (secure 0 0 0) 0x7eff1001bcf0 con 0x7eff1815da10 2023-10-16T13:46:46.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.230+0000 7eff06ffd700 1 --2- 172.21.15.16:0/1057205353 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7efef806b070 0x7efef806d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:46.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.230+0000 7eff1659c700 1 -- 172.21.15.16:0/1057205353 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7efef806b070 msgr2=0x7efef806d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3171422553 2023-10-16T13:46:46.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.230+0000 7eff06ffd700 1 -- 172.21.15.16:0/1057205353 <== mon.2 v2:172.21.15.49:3301/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 7021+0+0 (secure 0 0 0) 0x7eff10022070 con 0x7eff1815da10 2023-10-16T13:46:46.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.230+0000 7eff1659c700 1 --2- 172.21.15.16:0/1057205353 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7efef806b070 0x7efef806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:46:46.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.235+0000 7eff06ffd700 1 -- 172.21.15.16:0/1057205353 <== mon.2 v2:172.21.15.49:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7eff180b8330 con 0x7eff1815da10 2023-10-16T13:46:46.242 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:46.238+0000 7f45c6578200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T13:46:46.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.431+0000 7eff1659c700 1 -- 172.21.15.16:0/1057205353 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7efef806b070 msgr2=0x7efef806d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3171422553 2023-10-16T13:46:46.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.431+0000 7eff1659c700 1 --2- 172.21.15.16:0/1057205353 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7efef806b070 0x7efef806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:46:46.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.440+0000 7eff1cc37700 1 -- 172.21.15.16:0/1057205353 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7eff180b8330 con 0x7eff1815da10 2023-10-16T13:46:46.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.444+0000 7eff06ffd700 1 -- 172.21.15.16:0/1057205353 <== mon.2 v2:172.21.15.49:3301/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v78)=0 v78) v1 ==== 133+0+0 (secure 0 0 0) 0x7eff180b8330 con 0x7eff1815da10 2023-10-16T13:46:46.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.446+0000 7eff1cc37700 1 -- 172.21.15.16:0/1057205353 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7efef806b070 msgr2=0x7efef806d530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:46:46.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.446+0000 7eff1cc37700 1 --2- 172.21.15.16:0/1057205353 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7efef806b070 0x7efef806d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:46.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.446+0000 7eff1cc37700 1 -- 172.21.15.16:0/1057205353 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7eff1815da10 msgr2=0x7eff1815f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:46.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.446+0000 7eff1cc37700 1 --2- 172.21.15.16:0/1057205353 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7eff1815da10 0x7eff1815f120 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7eff10016000 tx=0x7eff10004ad0 comp rx=0 tx=0).stop 2023-10-16T13:46:46.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.446+0000 7eff1cc37700 1 -- 172.21.15.16:0/1057205353 shutdown_connections 2023-10-16T13:46:46.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.446+0000 7eff1cc37700 1 --2- 172.21.15.16:0/1057205353 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7eff1815da10 0x7eff1815f120 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:46.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.446+0000 7eff1cc37700 1 --2- 172.21.15.16:0/1057205353 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7efef806b070 0x7efef806d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:46.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.446+0000 7eff1cc37700 1 --2- 172.21.15.16:0/1057205353 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff180d2430 0x7eff181f4090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:46.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.446+0000 7eff1cc37700 1 --2- 172.21.15.16:0/1057205353 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7eff180d4060 0x7eff181f5cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:46.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.446+0000 7eff1cc37700 1 -- 172.21.15.16:0/1057205353 >> 172.21.15.16:0/1057205353 conn(0x7eff1809c9f0 msgr2=0x7eff180cf1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:46.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.446+0000 7eff1cc37700 1 -- 172.21.15.16:0/1057205353 shutdown_connections 2023-10-16T13:46:46.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.446+0000 7eff1cc37700 1 -- 172.21.15.16:0/1057205353 wait complete. 2023-10-16T13:46:46.463 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:46:46.463 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p mandatory_fs_argument_test_dir) 2023-10-16T13:46:46.484 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add mandatory_fs_argument_test_dir --fs backup_fs 1M 2023-10-16T13:46:46.547 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:46:46.544+0000 7f4f5ed5b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T13:46:46.590 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:46.586+0000 7f45c6578200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T13:46:46.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.653+0000 7f2ea9f40700 1 Processor -- start 2023-10-16T13:46:46.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.653+0000 7f2ea9f40700 1 -- start start 2023-10-16T13:46:46.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.654+0000 7f2ea9f40700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2ea40ce260 0x7f2ea40ce6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:46.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.654+0000 7f2ea9f40700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ea40b91b0 con 0x7f2ea415d7a0 2023-10-16T13:46:46.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.654+0000 7f2ea9f40700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ea40b7d00 con 0x7f2ea40d1740 2023-10-16T13:46:46.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.654+0000 7f2ea9f40700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ea40b8980 con 0x7f2ea40ce260 2023-10-16T13:46:46.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.654+0000 7f2ea37fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2ea415d7a0 0x7f2ea40b7240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:43096/0 (socket says 172.21.15.16:43096) 2023-10-16T13:46:46.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.654+0000 7f2ea37fe700 1 -- 172.21.15.16:0/1188851094 learned_addr learned my addr 172.21.15.16:0/1188851094 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:46:46.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.654+0000 7f2ea2ffd700 1 --2- 172.21.15.16:0/1188851094 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2ea40ce260 0x7f2ea40ce6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:46.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.655+0000 7f2ea27fc700 1 -- 172.21.15.16:0/1188851094 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2387699798 0 0) 0x7f2ea40b91b0 con 0x7f2ea415d7a0 2023-10-16T13:46:46.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.655+0000 7f2ea27fc700 1 -- 172.21.15.16:0/1188851094 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e8c0035b0 con 0x7f2ea415d7a0 2023-10-16T13:46:46.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.655+0000 7f2ea27fc700 1 -- 172.21.15.16:0/1188851094 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1469728771 0 0) 0x7f2ea40b7d00 con 0x7f2ea40d1740 2023-10-16T13:46:46.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.655+0000 7f2ea27fc700 1 -- 172.21.15.16:0/1188851094 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ea40b91b0 con 0x7f2ea40d1740 2023-10-16T13:46:46.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.655+0000 7f2ea2ffd700 1 -- 172.21.15.16:0/1188851094 >> v1:172.21.15.49:6789/0 conn(0x7f2ea40d1740 legacy=0x7f2ea40d1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:46.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.655+0000 7f2ea2ffd700 1 -- 172.21.15.16:0/1188851094 >> v1:172.21.15.16:6789/0 conn(0x7f2ea415d7a0 legacy=0x7f2ea40b7240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:46.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.655+0000 7f2ea2ffd700 1 -- 172.21.15.16:0/1188851094 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ea40b7fe0 con 0x7f2ea40ce260 2023-10-16T13:46:46.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.655+0000 7f2ea2ffd700 1 --2- 172.21.15.16:0/1188851094 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2ea40ce260 0x7f2ea40ce6e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f2e98009870 tx=0x7f2e98009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a4d782bc6e34f644 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:46.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.655+0000 7f2ea27fc700 1 -- 172.21.15.16:0/1188851094 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e98011830 con 0x7f2ea40ce260 2023-10-16T13:46:46.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.655+0000 7f2ea27fc700 1 -- 172.21.15.16:0/1188851094 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e98011990 con 0x7f2ea40ce260 2023-10-16T13:46:46.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.655+0000 7f2ea27fc700 1 -- 172.21.15.16:0/1188851094 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e98011d20 con 0x7f2ea40ce260 2023-10-16T13:46:46.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.656+0000 7f2ea9f40700 1 -- 172.21.15.16:0/1188851094 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2ea40ce260 msgr2=0x7f2ea40ce6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:46.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.656+0000 7f2ea9f40700 1 --2- 172.21.15.16:0/1188851094 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2ea40ce260 0x7f2ea40ce6e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f2e98009870 tx=0x7f2e98009b80 comp rx=0 tx=0).stop 2023-10-16T13:46:46.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.656+0000 7f2ea9f40700 1 -- 172.21.15.16:0/1188851094 shutdown_connections 2023-10-16T13:46:46.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.656+0000 7f2ea9f40700 1 --2- 172.21.15.16:0/1188851094 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2ea40ce260 0x7f2ea40ce6e0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:46.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.656+0000 7f2ea9f40700 1 -- 172.21.15.16:0/1188851094 >> 172.21.15.16:0/1188851094 conn(0x7f2ea4072330 msgr2=0x7f2ea4072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:46.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.656+0000 7f2ea9f40700 1 -- 172.21.15.16:0/1188851094 shutdown_connections 2023-10-16T13:46:46.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.656+0000 7f2ea9f40700 1 -- 172.21.15.16:0/1188851094 wait complete. 2023-10-16T13:46:46.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.657+0000 7f2ea9f40700 1 Processor -- start 2023-10-16T13:46:46.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.658+0000 7f2ea9f40700 1 -- start start 2023-10-16T13:46:46.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.659+0000 7f2ea9f40700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ea40ce260 0x7f2ea40cb7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:46.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.659+0000 7f2ea9f40700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2ea40d1740 0x7f2ea415eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:46.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.659+0000 7f2ea9f40700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2ea415d7a0 0x7f2ea4159110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:46.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.659+0000 7f2ea9f40700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ea40cbd00 con 0x7f2ea40ce260 2023-10-16T13:46:46.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.659+0000 7f2ea9f40700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ea40caa90 con 0x7f2ea40d1740 2023-10-16T13:46:46.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.659+0000 7f2ea9f40700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ea40c7cb0 con 0x7f2ea415d7a0 2023-10-16T13:46:46.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.659+0000 7f2ea3fff700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2ea415d7a0 0x7f2ea4159110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:46.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.659+0000 7f2ea2ffd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2ea40d1740 0x7f2ea415eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:46.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.659+0000 7f2ea3fff700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2ea415d7a0 0x7f2ea4159110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:42084/0 (socket says 172.21.15.16:42084) 2023-10-16T13:46:46.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.659+0000 7f2ea3fff700 1 -- 172.21.15.16:0/3558319785 learned_addr learned my addr 172.21.15.16:0/3558319785 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:46.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.660+0000 7f2ea37fe700 1 --2- 172.21.15.16:0/3558319785 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ea40ce260 0x7f2ea40cb7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:46.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.660+0000 7f2ea2ffd700 1 -- 172.21.15.16:0/3558319785 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2ea415d7a0 msgr2=0x7f2ea4159110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:46.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.660+0000 7f2ea2ffd700 1 --2- 172.21.15.16:0/3558319785 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2ea415d7a0 0x7f2ea4159110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:46.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.660+0000 7f2ea2ffd700 1 -- 172.21.15.16:0/3558319785 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ea40ce260 msgr2=0x7f2ea40cb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:46.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.660+0000 7f2ea2ffd700 1 --2- 172.21.15.16:0/3558319785 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ea40ce260 0x7f2ea40cb7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:46.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.660+0000 7f2ea2ffd700 1 -- 172.21.15.16:0/3558319785 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e980094a0 con 0x7f2ea40d1740 2023-10-16T13:46:46.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.660+0000 7f2ea3fff700 1 --2- 172.21.15.16:0/3558319785 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2ea415d7a0 0x7f2ea4159110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:46:46.669 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.660+0000 7f2ea37fe700 1 --2- 172.21.15.16:0/3558319785 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ea40ce260 0x7f2ea40cb7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:46:46.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.660+0000 7f2ea2ffd700 1 --2- 172.21.15.16:0/3558319785 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2ea40d1740 0x7f2ea415eb60 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f2ea406b0c0 tx=0x7f2e98004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:46.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.660+0000 7f2e8bfff700 1 -- 172.21.15.16:0/3558319785 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e9801a970 con 0x7f2ea40d1740 2023-10-16T13:46:46.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.660+0000 7f2e8bfff700 1 -- 172.21.15.16:0/3558319785 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e9801aad0 con 0x7f2ea40d1740 2023-10-16T13:46:46.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.660+0000 7f2ea9f40700 1 -- 172.21.15.16:0/3558319785 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ea40cace0 con 0x7f2ea40d1740 2023-10-16T13:46:46.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.660+0000 7f2e8bfff700 1 -- 172.21.15.16:0/3558319785 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e9801ae80 con 0x7f2ea40d1740 2023-10-16T13:46:46.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.660+0000 7f2ea9f40700 1 -- 172.21.15.16:0/3558319785 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ea40cb2a0 con 0x7f2ea40d1740 2023-10-16T13:46:46.671 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:46:46.667+0000 7f4f5ed5b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T13:46:46.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.662+0000 7f2ea9f40700 1 -- 172.21.15.16:0/3558319785 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ea40b6190 con 0x7f2ea40d1740 2023-10-16T13:46:46.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.662+0000 7f2e8bfff700 1 -- 172.21.15.16:0/3558319785 <== mon.1 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 139) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f2e9802d450 con 0x7f2ea40d1740 2023-10-16T13:46:46.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.662+0000 7f2e8bfff700 1 --2- 172.21.15.16:0/3558319785 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f2e9006b070 0x7f2e9006d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:46.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.663+0000 7f2ea37fe700 1 -- 172.21.15.16:0/3558319785 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f2e9006b070 msgr2=0x7f2e9006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3171422553 2023-10-16T13:46:46.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.663+0000 7f2e8bfff700 1 -- 172.21.15.16:0/3558319785 <== mon.1 v2:172.21.15.49:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 7021+0+0 (secure 0 0 0) 0x7f2e9808b0d0 con 0x7f2ea40d1740 2023-10-16T13:46:46.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.663+0000 7f2ea37fe700 1 --2- 172.21.15.16:0/3558319785 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f2e9006b070 0x7f2e9006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:46:46.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.667+0000 7f2e8bfff700 1 -- 172.21.15.16:0/3558319785 <== mon.1 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2e98055d00 con 0x7f2ea40d1740 2023-10-16T13:46:46.715 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:46.712+0000 7f45c6578200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T13:46:46.747 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:46:46.744+0000 7f4f5ed5b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T13:46:46.793 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:46.790+0000 7f45c6578200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T13:46:46.831 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:46:46.828+0000 7f4f5ed5b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T13:46:46.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.864+0000 7f2ea37fe700 1 -- 172.21.15.16:0/3558319785 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f2e9006b070 msgr2=0x7f2e9006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3171422553 2023-10-16T13:46:46.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.864+0000 7f2ea37fe700 1 --2- 172.21.15.16:0/3558319785 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f2e9006b070 0x7f2e9006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:46:46.878 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:46.874+0000 7f45c6578200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T13:46:46.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:46.899+0000 7f2ea9f40700 1 -- 172.21.15.16:0/3558319785 --> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f2ea40b85a0 con 0x7f2e9006b070 2023-10-16T13:46:47.216 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:46:47.214+0000 7f4f5ed5b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T13:46:47.267 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:47.263+0000 7f45c6578200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T13:46:47.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:47.265+0000 7f2ea37fe700 1 -- 172.21.15.16:0/3558319785 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f2e9006b070 msgr2=0x7f2e9006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3171422553 2023-10-16T13:46:47.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:47.265+0000 7f2ea37fe700 1 --2- 172.21.15.16:0/3558319785 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f2e9006b070 0x7f2e9006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-16T13:46:47.822 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:46:47.819+0000 7f4f5ed5b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T13:46:47.881 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:47.877+0000 7f45c6578200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T13:46:47.894 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:46:47.892+0000 7f4f5ed5b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T13:46:47.954 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:47.951+0000 7f45c6578200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T13:46:48.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:48.066+0000 7f2ea37fe700 1 -- 172.21.15.16:0/3558319785 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f2e9006b070 msgr2=0x7f2e9006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3171422553 2023-10-16T13:46:48.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:48.066+0000 7f2ea37fe700 1 --2- 172.21.15.16:0/3558319785 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f2e9006b070 0x7f2e9006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-16T13:46:48.556 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:46:48.553+0000 7f4f5ed5b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:46:48.624 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:48.621+0000 7f45c6578200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:46:48.637 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:46:48.635+0000 7f4f5ed5b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T13:46:48.706 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:48.703+0000 7f45c6578200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T13:46:48.788 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:46:48.785+0000 7f4f5ed5b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:46:48.859 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:48.856+0000 7f45c6578200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:46:49.024 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:46:49.022+0000 7f4f5ed5b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T13:46:49.100 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:49.097+0000 7f45c6578200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T13:46:49.101 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:46:49.099+0000 7f4f5ed5b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T13:46:49.179 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:49.176+0000 7f45c6578200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T13:46:49.377 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:46:49.375+0000 7f4f5ed5b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T13:46:49.458 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:49.455+0000 7f45c6578200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T13:46:49.610 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:46:49.608+0000 7f4f5ed5b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T13:46:49.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:49.667+0000 7f2ea37fe700 1 -- 172.21.15.16:0/3558319785 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f2e9006b070 msgr2=0x7f2e9006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3171422553 2023-10-16T13:46:49.670 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:49.667+0000 7f2ea37fe700 1 --2- 172.21.15.16:0/3558319785 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f2e9006b070 0x7f2e9006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-10-16T13:46:49.696 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:49.693+0000 7f45c6578200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T13:46:49.767 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:46:49.764+0000 7f4f5ed5b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T13:46:49.855 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:49.851+0000 7f45c6578200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T13:46:49.885 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:46:49.882+0000 7f4f5ed5b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T13:46:49.975 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:49.972+0000 7f45c6578200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T13:46:50.046 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:46:50.043+0000 7f4f5ed5b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T13:46:50.141 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:50.137+0000 7f45c6578200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T13:46:50.316 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:46:50.314+0000 7f4f5ed5b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T13:46:50.416 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:50.413+0000 7f45c6578200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T13:46:50.483 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:46:50.481+0000 7f4f5ed5b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T13:46:50.587 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:50.584+0000 7f45c6578200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T13:46:51.036 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:46:51.034+0000 7f4f5ed5b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T13:46:51.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:51.103+0000 7f2e8bfff700 1 -- 172.21.15.16:0/3558319785 <== mon.1 v2:172.21.15.49:3300/0 7 ==== mgrmap(e 140) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f2e980560d0 con 0x7f2ea40d1740 2023-10-16T13:46:51.153 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:51.150+0000 7f45c6578200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T13:46:51.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:51.169+0000 7f2e8bfff700 1 -- 172.21.15.16:0/3558319785 <== mon.1 v2:172.21.15.49:3300/0 8 ==== mgrmap(e 141) v1 ==== 88957+0+0 (secure 0 0 0) 0x7f2e9805d760 con 0x7f2ea40d1740 2023-10-16T13:46:51.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:51.170+0000 7f2e8bfff700 1 -- 172.21.15.16:0/3558319785 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f2e9006b070 msgr2=0x7f2e9006d530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:46:51.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:51.170+0000 7f2e8bfff700 1 --2- 172.21.15.16:0/3558319785 >> [v2:172.21.15.16:6832/3171422553,v1:172.21.15.16:6833/3171422553] conn(0x7f2e9006b070 0x7f2e9006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:51.393 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:51.389+0000 7f453bd58700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:46:51.393 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:51.389+0000 7f453bd58700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:46:51.393 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:51.389+0000 7f453bd58700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:46:51.394 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:51.389+0000 7f453bd58700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:46:51.394 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:51.389+0000 7f453bd58700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:46:51.394 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:51.390+0000 7f4538d52700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:46:51.394 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:51.390+0000 7f4538d52700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:46:51.395 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:51.390+0000 7f4538d52700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:46:51.395 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:51.390+0000 7f4538d52700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:46:51.395 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:51.390+0000 7f4538d52700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:46:51.402 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:51.399+0000 7f453bd58700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:46:51.402 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:51.399+0000 7f453bd58700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:46:51.403 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:51.399+0000 7f453bd58700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:46:51.403 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:51.399+0000 7f453bd58700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:46:51.403 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:51.399+0000 7f453bd58700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:46:51.404 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:51.399+0000 7f4538d52700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:46:51.404 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:51.399+0000 7f4538d52700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:46:51.404 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:51.399+0000 7f4538d52700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:46:51.404 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:51.399+0000 7f4538d52700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:46:51.405 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:51.399+0000 7f4538d52700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:46:52.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.174+0000 7f2e8bfff700 1 -- 172.21.15.16:0/3558319785 <== mon.1 v2:172.21.15.49:3300/0 9 ==== mgrmap(e 142) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f2e980569a0 con 0x7f2ea40d1740 2023-10-16T13:46:52.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.175+0000 7f2e8bfff700 1 --2- 172.21.15.16:0/3558319785 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f2e9001d250 0x7f2e90077ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:52.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.175+0000 7f2e8bfff700 1 -- 172.21.15.16:0/3558319785 --> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f2e9001d630 con 0x7f2e9001d250 2023-10-16T13:46:52.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.176+0000 7f2ea37fe700 1 --2- 172.21.15.16:0/3558319785 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f2e9001d250 0x7f2e90077ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:52.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.178+0000 7f2ea37fe700 1 --2- 172.21.15.16:0/3558319785 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f2e9001d250 0x7f2e90077ce0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f2e940011e0 tx=0x7f2e94007af0 comp rx=0 tx=0).ready entity=mgr.11745 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:52.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.243+0000 7f2e8bfff700 1 -- 172.21.15.16:0/3558319785 <== mgr.11745 v2:172.21.15.16:6832/1506142504 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+53 (secure 0 0 0) 0x7f2e9001d630 con 0x7f2e9001d250 2023-10-16T13:46:52.246 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule set for path mandatory_fs_argument_test_dir 2023-10-16T13:46:52.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.246+0000 7f2ea9f40700 1 -- 172.21.15.16:0/3558319785 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f2e9001d250 msgr2=0x7f2e90077ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:52.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.246+0000 7f2ea9f40700 1 --2- 172.21.15.16:0/3558319785 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f2e9001d250 0x7f2e90077ce0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f2e940011e0 tx=0x7f2e94007af0 comp rx=0 tx=0).stop 2023-10-16T13:46:52.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.246+0000 7f2ea9f40700 1 -- 172.21.15.16:0/3558319785 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2ea40d1740 msgr2=0x7f2ea415eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:52.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.246+0000 7f2ea9f40700 1 --2- 172.21.15.16:0/3558319785 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2ea40d1740 0x7f2ea415eb60 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f2ea406b0c0 tx=0x7f2e98004750 comp rx=0 tx=0).stop 2023-10-16T13:46:52.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.246+0000 7f2ea3fff700 1 -- 172.21.15.16:0/3558319785 reap_dead start 2023-10-16T13:46:52.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.246+0000 7f2ea9f40700 1 -- 172.21.15.16:0/3558319785 shutdown_connections 2023-10-16T13:46:52.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.246+0000 7f2ea9f40700 1 -- 172.21.15.16:0/3558319785 >> 172.21.15.16:0/3558319785 conn(0x7f2ea4072330 msgr2=0x7f2ea41601e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:52.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.246+0000 7f2ea9f40700 1 -- 172.21.15.16:0/3558319785 shutdown_connections 2023-10-16T13:46:52.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.246+0000 7f2ea9f40700 1 -- 172.21.15.16:0/3558319785 wait complete. 2023-10-16T13:46:52.262 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path mandatory_fs_argument_test_dir 2023-10-16T13:46:52.263 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate mandatory_fs_argument_test_dir --fs backup_fs 1M 2023-10-16T13:46:52.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.394+0000 7f83c26a2700 1 Processor -- start 2023-10-16T13:46:52.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.394+0000 7f83c26a2700 1 -- start start 2023-10-16T13:46:52.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.394+0000 7f83c26a2700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f83bc15d890 0x7f83bc0ce5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:52.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.395+0000 7f83c26a2700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83bc0b6800 con 0x7f83bc0ceb20 2023-10-16T13:46:52.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.395+0000 7f83c26a2700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83bc0d2580 con 0x7f83bc0d06b0 2023-10-16T13:46:52.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.395+0000 7f83c26a2700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83bc0d33e0 con 0x7f83bc15d890 2023-10-16T13:46:52.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.395+0000 7f83bb7fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f83bc0ceb20 0x7f83bc0cffa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:43348/0 (socket says 172.21.15.16:43348) 2023-10-16T13:46:52.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.395+0000 7f83bb7fe700 1 -- 172.21.15.16:0/1233564827 learned_addr learned my addr 172.21.15.16:0/1233564827 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:46:52.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.395+0000 7f83bbfff700 1 --2- 172.21.15.16:0/1233564827 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f83bc15d890 0x7f83bc0ce5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:52.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.395+0000 7f83baffd700 1 -- 172.21.15.16:0/1233564827 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 671885615 0 0) 0x7f83bc0b6800 con 0x7f83bc0ceb20 2023-10-16T13:46:52.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.395+0000 7f83baffd700 1 -- 172.21.15.16:0/1233564827 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83a40035b0 con 0x7f83bc0ceb20 2023-10-16T13:46:52.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.395+0000 7f83baffd700 1 -- 172.21.15.16:0/1233564827 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3338628611 0 0) 0x7f83a40035b0 con 0x7f83bc0ceb20 2023-10-16T13:46:52.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.395+0000 7f83baffd700 1 -- 172.21.15.16:0/1233564827 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f83bc15d890 msgr2=0x7f83bc0ce5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:52.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.395+0000 7f83baffd700 1 --2- 172.21.15.16:0/1233564827 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f83bc15d890 0x7f83bc0ce5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:52.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.395+0000 7f83baffd700 1 -- 172.21.15.16:0/1233564827 >> v1:172.21.15.49:6789/0 conn(0x7f83bc0d06b0 legacy=0x7f83bc0b6b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:52.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.395+0000 7f83baffd700 1 -- 172.21.15.16:0/1233564827 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83bc0d27d0 con 0x7f83bc0ceb20 2023-10-16T13:46:52.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.395+0000 7f83baffd700 1 -- 172.21.15.16:0/1233564827 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f83b0001670 con 0x7f83bc0ceb20 2023-10-16T13:46:52.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.395+0000 7f83baffd700 1 -- 172.21.15.16:0/1233564827 >> v1:172.21.15.16:6789/0 conn(0x7f83bc0ceb20 legacy=0x7f83bc0cffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:52.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.395+0000 7f83baffd700 1 --2- 172.21.15.16:0/1233564827 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f83bc15d890 0x7f83a4003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:52.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.395+0000 7f83baffd700 1 --2- 172.21.15.16:0/1233564827 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83a4003fb0 0x7f83a4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:52.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.395+0000 7f83baffd700 1 --2- 172.21.15.16:0/1233564827 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f83a4004d90 0x7f83a4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:52.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.395+0000 7f83baffd700 1 -- 172.21.15.16:0/1233564827 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83bc0d33e0 con 0x7f83a4003fb0 2023-10-16T13:46:52.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.395+0000 7f83baffd700 1 -- 172.21.15.16:0/1233564827 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83a400bb80 con 0x7f83a4004d90 2023-10-16T13:46:52.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.395+0000 7f83baffd700 1 -- 172.21.15.16:0/1233564827 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83a400bd40 con 0x7f83bc15d890 2023-10-16T13:46:52.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.396+0000 7f83bb7fe700 1 --2- 172.21.15.16:0/1233564827 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83a4003fb0 0x7f83a4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:52.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.396+0000 7f83c0c3f700 1 --2- 172.21.15.16:0/1233564827 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f83a4004d90 0x7f83a4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:52.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.396+0000 7f83bbfff700 1 --2- 172.21.15.16:0/1233564827 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f83bc15d890 0x7f83a4003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:52.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.396+0000 7f83bb7fe700 1 -- 172.21.15.16:0/1233564827 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f83bc15d890 msgr2=0x7f83a4003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:52.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.396+0000 7f83bb7fe700 1 --2- 172.21.15.16:0/1233564827 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f83bc15d890 0x7f83a4003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:52.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.396+0000 7f83bb7fe700 1 -- 172.21.15.16:0/1233564827 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f83a4004d90 msgr2=0x7f83a4005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:52.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.396+0000 7f83bb7fe700 1 --2- 172.21.15.16:0/1233564827 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f83a4004d90 0x7f83a4005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:52.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.396+0000 7f83bb7fe700 1 -- 172.21.15.16:0/1233564827 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83a40035b0 con 0x7f83a4003fb0 2023-10-16T13:46:52.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.396+0000 7f83bb7fe700 1 --2- 172.21.15.16:0/1233564827 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83a4003fb0 0x7f83a4004850 secure :-1 s=READY pgs=2565 cs=0 l=1 rev1=1 crypto rx=0x7f83b000c240 tx=0x7f83a4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2cbab71a79e0c05 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:52.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.396+0000 7f83baffd700 1 -- 172.21.15.16:0/1233564827 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83b0007c10 con 0x7f83a4003fb0 2023-10-16T13:46:52.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.396+0000 7f83baffd700 1 -- 172.21.15.16:0/1233564827 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83b000cde0 con 0x7f83a4003fb0 2023-10-16T13:46:52.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.396+0000 7f83baffd700 1 -- 172.21.15.16:0/1233564827 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83b0005ba0 con 0x7f83a4003fb0 2023-10-16T13:46:52.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.397+0000 7f83c26a2700 1 -- 172.21.15.16:0/1233564827 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83a4003fb0 msgr2=0x7f83a4004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:52.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.397+0000 7f83c26a2700 1 --2- 172.21.15.16:0/1233564827 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83a4003fb0 0x7f83a4004850 secure :-1 s=READY pgs=2565 cs=0 l=1 rev1=1 crypto rx=0x7f83b000c240 tx=0x7f83a4005880 comp rx=0 tx=0).stop 2023-10-16T13:46:52.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.397+0000 7f83c0c3f700 1 -- 172.21.15.16:0/1233564827 reap_dead start 2023-10-16T13:46:52.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.397+0000 7f83c26a2700 1 -- 172.21.15.16:0/1233564827 shutdown_connections 2023-10-16T13:46:52.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.397+0000 7f83c26a2700 1 -- 172.21.15.16:0/1233564827 >> 172.21.15.16:0/1233564827 conn(0x7f83bc072360 msgr2=0x7f83bc072770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:52.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.397+0000 7f83c26a2700 1 -- 172.21.15.16:0/1233564827 shutdown_connections 2023-10-16T13:46:52.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.397+0000 7f83c26a2700 1 -- 172.21.15.16:0/1233564827 wait complete. 2023-10-16T13:46:52.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.398+0000 7f83c26a2700 1 Processor -- start 2023-10-16T13:46:52.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.399+0000 7f83c26a2700 1 -- start start 2023-10-16T13:46:52.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.399+0000 7f83c26a2700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f83a4003fb0 0x7f83bc15a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:52.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.399+0000 7f83c26a2700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f83bc156e70 0x7f83bc0c93b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:52.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.399+0000 7f83c26a2700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83bc158900 0x7f83bc0cc180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:52.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.399+0000 7f83c26a2700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83bc0c98f0 con 0x7f83bc158900 2023-10-16T13:46:52.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.399+0000 7f83c26a2700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83bc159230 con 0x7f83bc156e70 2023-10-16T13:46:52.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.399+0000 7f83c26a2700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83bc159510 con 0x7f83a4003fb0 2023-10-16T13:46:52.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.400+0000 7f83c0c3f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83bc158900 0x7f83bc0cc180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:52.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.400+0000 7f83c0c3f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83bc158900 0x7f83bc0cc180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54088/0 (socket says 172.21.15.16:54088) 2023-10-16T13:46:52.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.400+0000 7f83c0c3f700 1 -- 172.21.15.16:0/2694459162 learned_addr learned my addr 172.21.15.16:0/2694459162 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:52.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.400+0000 7f83bbfff700 1 --2- 172.21.15.16:0/2694459162 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f83a4003fb0 0x7f83bc15a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:52.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.400+0000 7f83bb7fe700 1 --2- 172.21.15.16:0/2694459162 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f83bc156e70 0x7f83bc0c93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:52.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.400+0000 7f83c0c3f700 1 -- 172.21.15.16:0/2694459162 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f83a4003fb0 msgr2=0x7f83bc15a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:52.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.400+0000 7f83c0c3f700 1 --2- 172.21.15.16:0/2694459162 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f83a4003fb0 0x7f83bc15a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:52.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.400+0000 7f83c0c3f700 1 -- 172.21.15.16:0/2694459162 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f83bc156e70 msgr2=0x7f83bc0c93b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:52.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.400+0000 7f83c0c3f700 1 --2- 172.21.15.16:0/2694459162 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f83bc156e70 0x7f83bc0c93b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:52.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.400+0000 7f83c0c3f700 1 -- 172.21.15.16:0/2694459162 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83bc1597f0 con 0x7f83bc158900 2023-10-16T13:46:52.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.400+0000 7f83bb7fe700 1 --2- 172.21.15.16:0/2694459162 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f83bc156e70 0x7f83bc0c93b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:46:52.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.400+0000 7f83c0c3f700 1 --2- 172.21.15.16:0/2694459162 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83bc158900 0x7f83bc0cc180 secure :-1 s=READY pgs=2566 cs=0 l=1 rev1=1 crypto rx=0x7f83bc0d0c20 tx=0x7f83b400d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:52.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.400+0000 7f83b8ff9700 1 -- 172.21.15.16:0/2694459162 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83b4003c10 con 0x7f83bc158900 2023-10-16T13:46:52.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.400+0000 7f83b8ff9700 1 -- 172.21.15.16:0/2694459162 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83b4003d90 con 0x7f83bc158900 2023-10-16T13:46:52.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.400+0000 7f83c26a2700 1 -- 172.21.15.16:0/2694459162 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83bc159ad0 con 0x7f83bc158900 2023-10-16T13:46:52.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.400+0000 7f83b8ff9700 1 -- 172.21.15.16:0/2694459162 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83b400dbd0 con 0x7f83bc158900 2023-10-16T13:46:52.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.400+0000 7f83c26a2700 1 -- 172.21.15.16:0/2694459162 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83bc158db0 con 0x7f83bc158900 2023-10-16T13:46:52.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.401+0000 7f83c26a2700 1 -- 172.21.15.16:0/2694459162 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83bc159ff0 con 0x7f83bc158900 2023-10-16T13:46:52.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.402+0000 7f83b8ff9700 1 -- 172.21.15.16:0/2694459162 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 142) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f83b400dd30 con 0x7f83bc158900 2023-10-16T13:46:52.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.402+0000 7f83b8ff9700 1 --2- 172.21.15.16:0/2694459162 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f83a806ada0 0x7f83a806d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:52.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.402+0000 7f83b8ff9700 1 -- 172.21.15.16:0/2694459162 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(230..230 src has 1..230) v4 ==== 7365+0+0 (secure 0 0 0) 0x7f83b4089b60 con 0x7f83bc158900 2023-10-16T13:46:52.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.402+0000 7f83bbfff700 1 --2- 172.21.15.16:0/2694459162 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f83a806ada0 0x7f83a806d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:52.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.403+0000 7f83bbfff700 1 --2- 172.21.15.16:0/2694459162 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f83a806ada0 0x7f83a806d260 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f83ac009aa0 tx=0x7f83ac008040 comp rx=0 tx=0).ready entity=mgr.11745 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:52.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.407+0000 7f83b8ff9700 1 -- 172.21.15.16:0/2694459162 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f83bc159ff0 con 0x7f83bc158900 2023-10-16T13:46:52.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.637+0000 7f83c26a2700 1 -- 172.21.15.16:0/2694459162 --> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f83bc0d2d90 con 0x7f83a806ada0 2023-10-16T13:46:52.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.690+0000 7f83b8ff9700 1 -- 172.21.15.16:0/2694459162 <== mgr.11745 v2:172.21.15.16:6832/1506142504 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f83ac0137d0 con 0x7f83a806ada0 2023-10-16T13:46:52.693 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule activated for path mandatory_fs_argument_test_dir 2023-10-16T13:46:52.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.692+0000 7f83c26a2700 1 -- 172.21.15.16:0/2694459162 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f83a806ada0 msgr2=0x7f83a806d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:52.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.692+0000 7f83c26a2700 1 --2- 172.21.15.16:0/2694459162 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f83a806ada0 0x7f83a806d260 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f83ac009aa0 tx=0x7f83ac008040 comp rx=0 tx=0).stop 2023-10-16T13:46:52.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.692+0000 7f83c26a2700 1 -- 172.21.15.16:0/2694459162 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83bc158900 msgr2=0x7f83bc0cc180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:52.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.692+0000 7f83c26a2700 1 --2- 172.21.15.16:0/2694459162 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83bc158900 0x7f83bc0cc180 secure :-1 s=READY pgs=2566 cs=0 l=1 rev1=1 crypto rx=0x7f83bc0d0c20 tx=0x7f83b400d430 comp rx=0 tx=0).stop 2023-10-16T13:46:52.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.692+0000 7f83c26a2700 1 -- 172.21.15.16:0/2694459162 shutdown_connections 2023-10-16T13:46:52.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.692+0000 7f83c26a2700 1 --2- 172.21.15.16:0/2694459162 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f83a806ada0 0x7f83a806d260 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:52.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.692+0000 7f83c26a2700 1 --2- 172.21.15.16:0/2694459162 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83bc158900 0x7f83bc0cc180 unknown :-1 s=CLOSED pgs=2566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:52.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.692+0000 7f83c26a2700 1 --2- 172.21.15.16:0/2694459162 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f83a4003fb0 0x7f83bc15a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:52.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.692+0000 7f83c26a2700 1 --2- 172.21.15.16:0/2694459162 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f83bc156e70 0x7f83bc0c93b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:52.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.692+0000 7f83c26a2700 1 -- 172.21.15.16:0/2694459162 >> 172.21.15.16:0/2694459162 conn(0x7f83bc072360 msgr2=0x7f83bc15cab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:52.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.692+0000 7f83c26a2700 1 -- 172.21.15.16:0/2694459162 shutdown_connections 2023-10-16T13:46:52.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.692+0000 7f83c26a2700 1 -- 172.21.15.16:0/2694459162 wait complete. 2023-10-16T13:46:52.707 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path mandatory_fs_argument_test_dir 2023-10-16T13:46:52.707 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention add mandatory_fs_argument_test_dir --fs backup_fs 1M 2023-10-16T13:46:52.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.837+0000 7fd52a3e4700 1 Processor -- start 2023-10-16T13:46:52.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.837+0000 7fd52a3e4700 1 -- start start 2023-10-16T13:46:52.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.838+0000 7fd52a3e4700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd5240724a0 0x7fd5240728d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:52.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.838+0000 7fd52a3e4700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5240afd10 con 0x7fd52415ea90 2023-10-16T13:46:52.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.838+0000 7fd52a3e4700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd524071fa0 con 0x7fd52415e6b0 2023-10-16T13:46:52.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.838+0000 7fd52a3e4700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd524072e10 con 0x7fd5240724a0 2023-10-16T13:46:52.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.838+0000 7fd528981700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd5240724a0 0x7fd5240728d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:52.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.838+0000 7fd5237fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd52415ea90 0x7fd5240cba00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:43368/0 (socket says 172.21.15.16:43368) 2023-10-16T13:46:52.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.838+0000 7fd5237fe700 1 -- 172.21.15.16:0/291775915 learned_addr learned my addr 172.21.15.16:0/291775915 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:46:52.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.838+0000 7fd528981700 1 -- 172.21.15.16:0/291775915 >> v1:172.21.15.49:6789/0 conn(0x7fd52415e6b0 legacy=0x7fd52415abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:52.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.838+0000 7fd528981700 1 -- 172.21.15.16:0/291775915 >> v1:172.21.15.16:6789/0 conn(0x7fd52415ea90 legacy=0x7fd5240cba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:52.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.838+0000 7fd528981700 1 -- 172.21.15.16:0/291775915 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd52415b400 con 0x7fd5240724a0 2023-10-16T13:46:52.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.838+0000 7fd522ffd700 1 -- 172.21.15.16:0/291775915 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1101936776 0 0) 0x7fd50c001110 con 0x7fd52415ea90 2023-10-16T13:46:52.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.838+0000 7fd528981700 1 --2- 172.21.15.16:0/291775915 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5240724a0 0x7fd5240728d0 secure :-1 s=READY pgs=2568 cs=0 l=1 rev1=1 crypto rx=0x7fd51c00bfa0 tx=0x7fd51c00d330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d94da80b8d99979 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:52.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.838+0000 7fd522ffd700 1 -- 172.21.15.16:0/291775915 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd51c010070 con 0x7fd5240724a0 2023-10-16T13:46:52.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.839+0000 7fd522ffd700 1 -- 172.21.15.16:0/291775915 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd51c00b2e0 con 0x7fd5240724a0 2023-10-16T13:46:52.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.839+0000 7fd522ffd700 1 -- 172.21.15.16:0/291775915 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd51c00dcf0 con 0x7fd5240724a0 2023-10-16T13:46:52.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.839+0000 7fd52a3e4700 1 -- 172.21.15.16:0/291775915 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5240724a0 msgr2=0x7fd5240728d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:52.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.839+0000 7fd52a3e4700 1 --2- 172.21.15.16:0/291775915 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5240724a0 0x7fd5240728d0 secure :-1 s=READY pgs=2568 cs=0 l=1 rev1=1 crypto rx=0x7fd51c00bfa0 tx=0x7fd51c00d330 comp rx=0 tx=0).stop 2023-10-16T13:46:52.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.839+0000 7fd52a3e4700 1 -- 172.21.15.16:0/291775915 shutdown_connections 2023-10-16T13:46:52.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.839+0000 7fd52a3e4700 1 --2- 172.21.15.16:0/291775915 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5240724a0 0x7fd5240728d0 unknown :-1 s=CLOSED pgs=2568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:52.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.839+0000 7fd52a3e4700 1 -- 172.21.15.16:0/291775915 >> 172.21.15.16:0/291775915 conn(0x7fd52406bce0 msgr2=0x7fd52406c0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:52.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.839+0000 7fd52a3e4700 1 -- 172.21.15.16:0/291775915 shutdown_connections 2023-10-16T13:46:52.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.839+0000 7fd52a3e4700 1 -- 172.21.15.16:0/291775915 wait complete. 2023-10-16T13:46:52.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.841+0000 7fd52a3e4700 1 Processor -- start 2023-10-16T13:46:52.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.841+0000 7fd52a3e4700 1 -- start start 2023-10-16T13:46:52.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.842+0000 7fd52a3e4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5240724a0 0x7fd52415d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:52.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.842+0000 7fd52a3e4700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd52415e6b0 0x7fd52415bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:52.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.842+0000 7fd52a3e4700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd52415ea90 0x7fd5240b5610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:52.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.842+0000 7fd52a3e4700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd52414c9c0 con 0x7fd5240724a0 2023-10-16T13:46:52.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.842+0000 7fd52a3e4700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd52415d9e0 con 0x7fd52415e6b0 2023-10-16T13:46:52.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.842+0000 7fd52a3e4700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd52415c500 con 0x7fd52415ea90 2023-10-16T13:46:52.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.842+0000 7fd523fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5240724a0 0x7fd52415d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:52.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.842+0000 7fd523fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5240724a0 0x7fd52415d4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54098/0 (socket says 172.21.15.16:54098) 2023-10-16T13:46:52.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.842+0000 7fd523fff700 1 -- 172.21.15.16:0/3187507882 learned_addr learned my addr 172.21.15.16:0/3187507882 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:52.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.842+0000 7fd5237fe700 1 --2- 172.21.15.16:0/3187507882 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd52415e6b0 0x7fd52415bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:52.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.842+0000 7fd528981700 1 --2- 172.21.15.16:0/3187507882 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd52415ea90 0x7fd5240b5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:52.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.842+0000 7fd523fff700 1 -- 172.21.15.16:0/3187507882 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd52415ea90 msgr2=0x7fd5240b5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:52.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.842+0000 7fd523fff700 1 --2- 172.21.15.16:0/3187507882 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd52415ea90 0x7fd5240b5610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:52.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.842+0000 7fd523fff700 1 -- 172.21.15.16:0/3187507882 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd52415e6b0 msgr2=0x7fd52415bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:52.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.842+0000 7fd523fff700 1 --2- 172.21.15.16:0/3187507882 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd52415e6b0 0x7fd52415bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:52.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.842+0000 7fd523fff700 1 -- 172.21.15.16:0/3187507882 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd51c00bc80 con 0x7fd5240724a0 2023-10-16T13:46:52.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.842+0000 7fd5237fe700 1 --2- 172.21.15.16:0/3187507882 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd52415e6b0 0x7fd52415bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:46:52.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.842+0000 7fd523fff700 1 --2- 172.21.15.16:0/3187507882 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5240724a0 0x7fd52415d4a0 secure :-1 s=READY pgs=2569 cs=0 l=1 rev1=1 crypto rx=0x7fd51400c4e0 tx=0x7fd51400c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:52.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.843+0000 7fd520ff9700 1 -- 172.21.15.16:0/3187507882 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5140052d0 con 0x7fd5240724a0 2023-10-16T13:46:52.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.843+0000 7fd520ff9700 1 -- 172.21.15.16:0/3187507882 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd514005430 con 0x7fd5240724a0 2023-10-16T13:46:52.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.843+0000 7fd52a3e4700 1 -- 172.21.15.16:0/3187507882 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd52415c7e0 con 0x7fd5240724a0 2023-10-16T13:46:52.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.843+0000 7fd520ff9700 1 -- 172.21.15.16:0/3187507882 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5140038c0 con 0x7fd5240724a0 2023-10-16T13:46:52.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.843+0000 7fd52a3e4700 1 -- 172.21.15.16:0/3187507882 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5240c7520 con 0x7fd5240724a0 2023-10-16T13:46:52.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.844+0000 7fd52a3e4700 1 -- 172.21.15.16:0/3187507882 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd52404ebb0 con 0x7fd5240724a0 2023-10-16T13:46:52.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.844+0000 7fd520ff9700 1 -- 172.21.15.16:0/3187507882 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 142) v1 ==== 89084+0+0 (secure 0 0 0) 0x7fd514003a20 con 0x7fd5240724a0 2023-10-16T13:46:52.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.845+0000 7fd520ff9700 1 --2- 172.21.15.16:0/3187507882 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7fd50406ada0 0x7fd50406d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:52.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.845+0000 7fd520ff9700 1 -- 172.21.15.16:0/3187507882 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(230..230 src has 1..230) v4 ==== 7365+0+0 (secure 0 0 0) 0x7fd514088e80 con 0x7fd5240724a0 2023-10-16T13:46:52.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.845+0000 7fd5237fe700 1 --2- 172.21.15.16:0/3187507882 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7fd50406ada0 0x7fd50406d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:52.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.845+0000 7fd5237fe700 1 --2- 172.21.15.16:0/3187507882 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7fd50406ada0 0x7fd50406d260 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd5240c3f90 tx=0x7fd50c009040 comp rx=0 tx=0).ready entity=mgr.11745 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:52.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:52.849+0000 7fd520ff9700 1 -- 172.21.15.16:0/3187507882 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd514053b20 con 0x7fd5240724a0 2023-10-16T13:46:53.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.082+0000 7fd52a3e4700 1 -- 172.21.15.16:0/3187507882 --> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "retention_spec_or_period": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fd52415b9c0 con 0x7fd50406ada0 2023-10-16T13:46:53.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.139+0000 7fd520ff9700 1 -- 172.21.15.16:0/3187507882 <== mgr.11745 v2:172.21.15.16:6832/1506142504 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+55 (secure 0 0 0) 0x7fd50c004870 con 0x7fd50406ada0 2023-10-16T13:46:53.142 INFO:teuthology.orchestra.run.smithi016.stdout:Retention added to path mandatory_fs_argument_test_dir 2023-10-16T13:46:53.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.141+0000 7fd52a3e4700 1 -- 172.21.15.16:0/3187507882 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7fd50406ada0 msgr2=0x7fd50406d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:53.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.141+0000 7fd52a3e4700 1 --2- 172.21.15.16:0/3187507882 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7fd50406ada0 0x7fd50406d260 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd5240c3f90 tx=0x7fd50c009040 comp rx=0 tx=0).stop 2023-10-16T13:46:53.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.141+0000 7fd52a3e4700 1 -- 172.21.15.16:0/3187507882 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5240724a0 msgr2=0x7fd52415d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:53.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.141+0000 7fd52a3e4700 1 --2- 172.21.15.16:0/3187507882 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5240724a0 0x7fd52415d4a0 secure :-1 s=READY pgs=2569 cs=0 l=1 rev1=1 crypto rx=0x7fd51400c4e0 tx=0x7fd51400c8a0 comp rx=0 tx=0).stop 2023-10-16T13:46:53.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.141+0000 7fd52a3e4700 1 -- 172.21.15.16:0/3187507882 shutdown_connections 2023-10-16T13:46:53.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.141+0000 7fd52a3e4700 1 --2- 172.21.15.16:0/3187507882 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7fd50406ada0 0x7fd50406d260 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:53.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.141+0000 7fd52a3e4700 1 --2- 172.21.15.16:0/3187507882 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd52415ea90 0x7fd5240b5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:53.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.141+0000 7fd52a3e4700 1 --2- 172.21.15.16:0/3187507882 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5240724a0 0x7fd52415d4a0 unknown :-1 s=CLOSED pgs=2569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:53.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.141+0000 7fd52a3e4700 1 --2- 172.21.15.16:0/3187507882 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd52415e6b0 0x7fd52415bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:53.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.141+0000 7fd52a3e4700 1 -- 172.21.15.16:0/3187507882 >> 172.21.15.16:0/3187507882 conn(0x7fd52406bce0 msgr2=0x7fd5240b0b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:53.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.141+0000 7fd52a3e4700 1 -- 172.21.15.16:0/3187507882 shutdown_connections 2023-10-16T13:46:53.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.141+0000 7fd52a3e4700 1 -- 172.21.15.16:0/3187507882 wait complete. 2023-10-16T13:46:53.156 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention added to path mandatory_fs_argument_test_dir 2023-10-16T13:46:53.156 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list mandatory_fs_argument_test_dir --fs backup_fs --format json 2023-10-16T13:46:53.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.290+0000 7faca771c700 1 Processor -- start 2023-10-16T13:46:53.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.290+0000 7faca771c700 1 -- start start 2023-10-16T13:46:53.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.291+0000 7faca771c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faca0155a60 0x7faca015fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:53.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.291+0000 7faca771c700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faca0160190 con 0x7faca0155e40 2023-10-16T13:46:53.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.291+0000 7faca771c700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faca015c780 con 0x7faca00cba40 2023-10-16T13:46:53.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.291+0000 7faca771c700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faca015f240 con 0x7faca0155a60 2023-10-16T13:46:53.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.291+0000 7faca4cb7700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7faca0155e40 0x7faca015bc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:43398/0 (socket says 172.21.15.16:43398) 2023-10-16T13:46:53.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.291+0000 7faca4cb7700 1 -- 172.21.15.16:0/387798576 learned_addr learned my addr 172.21.15.16:0/387798576 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:46:53.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.291+0000 7faca54b8700 1 --2- 172.21.15.16:0/387798576 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faca0155a60 0x7faca015fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:53.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.291+0000 7fac93fff700 1 -- 172.21.15.16:0/387798576 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3589950985 0 0) 0x7faca0160190 con 0x7faca0155e40 2023-10-16T13:46:53.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.291+0000 7fac93fff700 1 -- 172.21.15.16:0/387798576 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac8c0035b0 con 0x7faca0155e40 2023-10-16T13:46:53.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.292+0000 7fac93fff700 1 -- 172.21.15.16:0/387798576 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2499826241 0 0) 0x7faca015c780 con 0x7faca00cba40 2023-10-16T13:46:53.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.292+0000 7fac93fff700 1 -- 172.21.15.16:0/387798576 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faca0160190 con 0x7faca00cba40 2023-10-16T13:46:53.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.292+0000 7faca54b8700 1 -- 172.21.15.16:0/387798576 >> v1:172.21.15.49:6789/0 conn(0x7faca00cba40 legacy=0x7faca00cefc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:53.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.292+0000 7faca54b8700 1 -- 172.21.15.16:0/387798576 >> v1:172.21.15.16:6789/0 conn(0x7faca0155e40 legacy=0x7faca015bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:53.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.292+0000 7faca54b8700 1 -- 172.21.15.16:0/387798576 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faca015ca60 con 0x7faca0155a60 2023-10-16T13:46:53.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.292+0000 7faca54b8700 1 --2- 172.21.15.16:0/387798576 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faca0155a60 0x7faca015fc50 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fac9c009870 tx=0x7fac9c009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bda8e2163e85adbd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:53.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.292+0000 7fac93fff700 1 -- 172.21.15.16:0/387798576 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac9c011830 con 0x7faca0155a60 2023-10-16T13:46:53.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.292+0000 7fac93fff700 1 -- 172.21.15.16:0/387798576 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac9c011990 con 0x7faca0155a60 2023-10-16T13:46:53.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.292+0000 7fac93fff700 1 -- 172.21.15.16:0/387798576 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac9c011d20 con 0x7faca0155a60 2023-10-16T13:46:53.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.293+0000 7faca771c700 1 -- 172.21.15.16:0/387798576 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faca0155a60 msgr2=0x7faca015fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:53.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.293+0000 7faca771c700 1 --2- 172.21.15.16:0/387798576 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faca0155a60 0x7faca015fc50 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fac9c009870 tx=0x7fac9c009b80 comp rx=0 tx=0).stop 2023-10-16T13:46:53.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.293+0000 7faca771c700 1 -- 172.21.15.16:0/387798576 shutdown_connections 2023-10-16T13:46:53.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.293+0000 7faca771c700 1 --2- 172.21.15.16:0/387798576 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faca0155a60 0x7faca015fc50 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:53.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.293+0000 7faca771c700 1 -- 172.21.15.16:0/387798576 >> 172.21.15.16:0/387798576 conn(0x7faca006a7e0 msgr2=0x7faca006abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:53.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.293+0000 7faca771c700 1 -- 172.21.15.16:0/387798576 shutdown_connections 2023-10-16T13:46:53.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.293+0000 7faca771c700 1 -- 172.21.15.16:0/387798576 wait complete. 2023-10-16T13:46:53.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.294+0000 7faca771c700 1 Processor -- start 2023-10-16T13:46:53.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.294+0000 7faca771c700 1 -- start start 2023-10-16T13:46:53.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.295+0000 7faca771c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faca00cba40 0x7faca01f4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:53.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.295+0000 7faca771c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faca0155a60 0x7faca00c9f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:53.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.295+0000 7faca771c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faca0155e40 0x7faca00b6a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:53.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.295+0000 7faca771c700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faca01f5570 con 0x7faca0155e40 2023-10-16T13:46:53.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.295+0000 7faca771c700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faca00b6f90 con 0x7faca0155a60 2023-10-16T13:46:53.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.295+0000 7faca771c700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faca01f1020 con 0x7faca00cba40 2023-10-16T13:46:53.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.295+0000 7faca5cb9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faca0155e40 0x7faca00b6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:53.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.295+0000 7faca5cb9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faca0155e40 0x7faca00b6a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54132/0 (socket says 172.21.15.16:54132) 2023-10-16T13:46:53.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.295+0000 7faca5cb9700 1 -- 172.21.15.16:0/975053015 learned_addr learned my addr 172.21.15.16:0/975053015 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:53.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.295+0000 7faca54b8700 1 --2- 172.21.15.16:0/975053015 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faca00cba40 0x7faca01f4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:53.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.295+0000 7faca4cb7700 1 --2- 172.21.15.16:0/975053015 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faca0155a60 0x7faca00c9f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:53.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.295+0000 7faca5cb9700 1 -- 172.21.15.16:0/975053015 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faca00cba40 msgr2=0x7faca01f4610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:53.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.295+0000 7faca5cb9700 1 --2- 172.21.15.16:0/975053015 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faca00cba40 0x7faca01f4610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:53.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.295+0000 7faca5cb9700 1 -- 172.21.15.16:0/975053015 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faca0155a60 msgr2=0x7faca00c9f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:53.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.295+0000 7faca5cb9700 1 --2- 172.21.15.16:0/975053015 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faca0155a60 0x7faca00c9f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:53.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.295+0000 7faca5cb9700 1 -- 172.21.15.16:0/975053015 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac9c0094a0 con 0x7faca0155e40 2023-10-16T13:46:53.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.295+0000 7faca5cb9700 1 --2- 172.21.15.16:0/975053015 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faca0155e40 0x7faca00b6a50 secure :-1 s=READY pgs=2573 cs=0 l=1 rev1=1 crypto rx=0x7fac9800cc20 tx=0x7fac9800cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:53.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.295+0000 7fac91ffb700 1 -- 172.21.15.16:0/975053015 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac98003c00 con 0x7faca0155e40 2023-10-16T13:46:53.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.295+0000 7fac91ffb700 1 -- 172.21.15.16:0/975053015 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac98003d60 con 0x7faca0155e40 2023-10-16T13:46:53.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.295+0000 7faca771c700 1 -- 172.21.15.16:0/975053015 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faca01f4cd0 con 0x7faca0155e40 2023-10-16T13:46:53.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.295+0000 7fac91ffb700 1 -- 172.21.15.16:0/975053015 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac9800aae0 con 0x7faca0155e40 2023-10-16T13:46:53.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.296+0000 7faca771c700 1 -- 172.21.15.16:0/975053015 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faca01f5290 con 0x7faca0155e40 2023-10-16T13:46:53.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.297+0000 7faca771c700 1 -- 172.21.15.16:0/975053015 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faca00678b0 con 0x7faca0155e40 2023-10-16T13:46:53.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.297+0000 7fac91ffb700 1 -- 172.21.15.16:0/975053015 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 142) v1 ==== 89084+0+0 (secure 0 0 0) 0x7fac9800ac40 con 0x7faca0155e40 2023-10-16T13:46:53.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.297+0000 7fac91ffb700 1 --2- 172.21.15.16:0/975053015 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7fac8406ada0 0x7fac8406d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:53.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.298+0000 7fac91ffb700 1 -- 172.21.15.16:0/975053015 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(230..230 src has 1..230) v4 ==== 7365+0+0 (secure 0 0 0) 0x7fac98089900 con 0x7faca0155e40 2023-10-16T13:46:53.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.298+0000 7faca54b8700 1 --2- 172.21.15.16:0/975053015 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7fac8406ada0 0x7fac8406d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:53.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.298+0000 7faca54b8700 1 --2- 172.21.15.16:0/975053015 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7fac8406ada0 0x7fac8406d260 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fac9c0097c0 tx=0x7fac9c01c040 comp rx=0 tx=0).ready entity=mgr.11745 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:53.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.302+0000 7fac91ffb700 1 -- 172.21.15.16:0/975053015 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fac98054670 con 0x7faca0155e40 2023-10-16T13:46:53.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.533+0000 7faca771c700 1 -- 172.21.15.16:0/975053015 --> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7faca015f730 con 0x7fac8406ada0 2023-10-16T13:46:53.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.540+0000 7fac91ffb700 1 -- 172.21.15.16:0/975053015 <== mgr.11745 v2:172.21.15.16:6832/1506142504 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+88 (secure 0 0 0) 0x7fac9c011b00 con 0x7fac8406ada0 2023-10-16T13:46:53.542 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:46:53.543 INFO:teuthology.orchestra.run.smithi016.stdout:{"path": "mandatory_fs_argument_test_dir", "schedule": ["1M"], "retention": [{"M": 1}]} 2023-10-16T13:46:53.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.542+0000 7faca771c700 1 -- 172.21.15.16:0/975053015 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7fac8406ada0 msgr2=0x7fac8406d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:53.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.542+0000 7faca771c700 1 --2- 172.21.15.16:0/975053015 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7fac8406ada0 0x7fac8406d260 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fac9c0097c0 tx=0x7fac9c01c040 comp rx=0 tx=0).stop 2023-10-16T13:46:53.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.542+0000 7faca771c700 1 -- 172.21.15.16:0/975053015 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faca0155e40 msgr2=0x7faca00b6a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:53.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.542+0000 7faca771c700 1 --2- 172.21.15.16:0/975053015 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faca0155e40 0x7faca00b6a50 secure :-1 s=READY pgs=2573 cs=0 l=1 rev1=1 crypto rx=0x7fac9800cc20 tx=0x7fac9800cf30 comp rx=0 tx=0).stop 2023-10-16T13:46:53.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.542+0000 7faca771c700 1 -- 172.21.15.16:0/975053015 shutdown_connections 2023-10-16T13:46:53.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.542+0000 7faca771c700 1 --2- 172.21.15.16:0/975053015 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7fac8406ada0 0x7fac8406d260 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:53.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.542+0000 7faca771c700 1 --2- 172.21.15.16:0/975053015 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faca0155e40 0x7faca00b6a50 unknown :-1 s=CLOSED pgs=2573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:53.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.542+0000 7faca771c700 1 --2- 172.21.15.16:0/975053015 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faca00cba40 0x7faca01f4610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:53.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.542+0000 7faca771c700 1 --2- 172.21.15.16:0/975053015 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faca0155a60 0x7faca00c9f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:53.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.542+0000 7faca771c700 1 -- 172.21.15.16:0/975053015 >> 172.21.15.16:0/975053015 conn(0x7faca006a7e0 msgr2=0x7faca00cefc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:53.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.542+0000 7faca771c700 1 -- 172.21.15.16:0/975053015 shutdown_connections 2023-10-16T13:46:53.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.542+0000 7faca771c700 1 -- 172.21.15.16:0/975053015 wait complete. 2023-10-16T13:46:53.557 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "mandatory_fs_argument_test_dir", "schedule": ["1M"], "retention": [{"M": 1}]} 2023-10-16T13:46:53.557 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status mandatory_fs_argument_test_dir --fs backup_fs --format json 2023-10-16T13:46:53.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.685+0000 7f5d90b9c700 1 Processor -- start 2023-10-16T13:46:53.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.685+0000 7f5d90b9c700 1 -- start start 2023-10-16T13:46:53.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.686+0000 7f5d90b9c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5d8c15a3d0 0x7f5d8c06d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:53.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.686+0000 7f5d90b9c700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f5d8c15e0b0 0x7f5d8c0ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:53.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.686+0000 7f5d90b9c700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d8c0cb870 con 0x7f5d8c0cf430 2023-10-16T13:46:53.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.686+0000 7f5d90b9c700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d8c15f0c0 con 0x7f5d8c15a3d0 2023-10-16T13:46:53.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.686+0000 7f5d90b9c700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d8c072990 con 0x7f5d8c15e0b0 2023-10-16T13:46:53.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.686+0000 7f5d8a59c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5d8c15a3d0 0x7f5d8c06d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:53.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.686+0000 7f5d8ad9d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f5d8c0cf430 0x7f5d8c0cdeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:43414/0 (socket says 172.21.15.16:43414) 2023-10-16T13:46:53.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.686+0000 7f5d8a59c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5d8c15a3d0 0x7f5d8c06d910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54136/0 (socket says 172.21.15.16:54136) 2023-10-16T13:46:53.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.686+0000 7f5d8ad9d700 1 -- 172.21.15.16:0/2373962352 learned_addr learned my addr 172.21.15.16:0/2373962352 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:46:53.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.687+0000 7f5d89d9b700 1 --2- 172.21.15.16:0/2373962352 >> v2:172.21.15.49:3300/0 conn(0x7f5d8c15e0b0 0x7f5d8c0ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:53.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.687+0000 7f5d8a59c700 1 -- 172.21.15.16:0/2373962352 >> v2:172.21.15.49:3300/0 conn(0x7f5d8c15e0b0 msgr2=0x7f5d8c0ceef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:53.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.687+0000 7f5d8a59c700 1 --2- 172.21.15.16:0/2373962352 >> v2:172.21.15.49:3300/0 conn(0x7f5d8c15e0b0 0x7f5d8c0ceef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:53.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.687+0000 7f5d8a59c700 1 -- 172.21.15.16:0/2373962352 >> v1:172.21.15.16:6789/0 conn(0x7f5d8c0cf430 legacy=0x7f5d8c0cdeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:53.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.687+0000 7f5d8959a700 1 -- 172.21.15.16:0/2373962352 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2909843649 0 0) 0x7f5d8c0cb870 con 0x7f5d8c0cf430 2023-10-16T13:46:53.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.687+0000 7f5d8a59c700 1 -- 172.21.15.16:0/2373962352 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d8c06e040 con 0x7f5d8c15a3d0 2023-10-16T13:46:53.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.687+0000 7f5d8a59c700 1 --2- 172.21.15.16:0/2373962352 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d8c15a3d0 0x7f5d8c06d910 secure :-1 s=READY pgs=2575 cs=0 l=1 rev1=1 crypto rx=0x7f5d8c15e010 tx=0x7f5d80009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd638ec270d83f4d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:53.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.687+0000 7f5d8959a700 1 -- 172.21.15.16:0/2373962352 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d80011830 con 0x7f5d8c15a3d0 2023-10-16T13:46:53.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.687+0000 7f5d8959a700 1 -- 172.21.15.16:0/2373962352 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d80011990 con 0x7f5d8c15a3d0 2023-10-16T13:46:53.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.687+0000 7f5d8959a700 1 -- 172.21.15.16:0/2373962352 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d80011d20 con 0x7f5d8c15a3d0 2023-10-16T13:46:53.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.687+0000 7f5d90b9c700 1 -- 172.21.15.16:0/2373962352 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d8c15a3d0 msgr2=0x7f5d8c06d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:53.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.687+0000 7f5d90b9c700 1 --2- 172.21.15.16:0/2373962352 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d8c15a3d0 0x7f5d8c06d910 secure :-1 s=READY pgs=2575 cs=0 l=1 rev1=1 crypto rx=0x7f5d8c15e010 tx=0x7f5d80009b50 comp rx=0 tx=0).stop 2023-10-16T13:46:53.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.688+0000 7f5d90b9c700 1 -- 172.21.15.16:0/2373962352 shutdown_connections 2023-10-16T13:46:53.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.688+0000 7f5d90b9c700 1 --2- 172.21.15.16:0/2373962352 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d8c15a3d0 0x7f5d8c06d910 unknown :-1 s=CLOSED pgs=2575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:53.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.688+0000 7f5d90b9c700 1 --2- 172.21.15.16:0/2373962352 >> v2:172.21.15.49:3300/0 conn(0x7f5d8c15e0b0 0x7f5d8c0ceef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:53.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.688+0000 7f5d90b9c700 1 -- 172.21.15.16:0/2373962352 >> 172.21.15.16:0/2373962352 conn(0x7f5d8c06f430 msgr2=0x7f5d8c06f810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:53.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.688+0000 7f5d90b9c700 1 -- 172.21.15.16:0/2373962352 shutdown_connections 2023-10-16T13:46:53.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.688+0000 7f5d90b9c700 1 -- 172.21.15.16:0/2373962352 wait complete. 2023-10-16T13:46:53.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.689+0000 7f5d90b9c700 1 Processor -- start 2023-10-16T13:46:53.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.689+0000 7f5d90b9c700 1 -- start start 2023-10-16T13:46:53.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.690+0000 7f5d90b9c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d8c0cf430 0x7f5d8c159000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:53.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.690+0000 7f5d90b9c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5d8c15a3d0 0x7f5d8c0c4040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:53.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.690+0000 7f5d90b9c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5d8c15e0b0 0x7f5d8c0cde60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:53.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.690+0000 7f5d90b9c700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d8c1f2ca0 con 0x7f5d8c0cf430 2023-10-16T13:46:53.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.690+0000 7f5d90b9c700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d8c0aca70 con 0x7f5d8c15a3d0 2023-10-16T13:46:53.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.690+0000 7f5d90b9c700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d8c0acd50 con 0x7f5d8c15e0b0 2023-10-16T13:46:53.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.690+0000 7f5d8a59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d8c0cf430 0x7f5d8c159000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:53.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.690+0000 7f5d8a59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d8c0cf430 0x7f5d8c159000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54142/0 (socket says 172.21.15.16:54142) 2023-10-16T13:46:53.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.690+0000 7f5d8a59c700 1 -- 172.21.15.16:0/3280837578 learned_addr learned my addr 172.21.15.16:0/3280837578 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:53.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.690+0000 7f5d89d9b700 1 --2- 172.21.15.16:0/3280837578 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5d8c15a3d0 0x7f5d8c0c4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:53.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.690+0000 7f5d8ad9d700 1 --2- 172.21.15.16:0/3280837578 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5d8c15e0b0 0x7f5d8c0cde60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:53.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.690+0000 7f5d8a59c700 1 -- 172.21.15.16:0/3280837578 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5d8c15e0b0 msgr2=0x7f5d8c0cde60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:53.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.690+0000 7f5d8a59c700 1 --2- 172.21.15.16:0/3280837578 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5d8c15e0b0 0x7f5d8c0cde60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:53.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.690+0000 7f5d8a59c700 1 -- 172.21.15.16:0/3280837578 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5d8c15a3d0 msgr2=0x7f5d8c0c4040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:53.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.690+0000 7f5d8a59c700 1 --2- 172.21.15.16:0/3280837578 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5d8c15a3d0 0x7f5d8c0c4040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:53.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.690+0000 7f5d8a59c700 1 -- 172.21.15.16:0/3280837578 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d800094a0 con 0x7f5d8c0cf430 2023-10-16T13:46:53.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.690+0000 7f5d8ad9d700 1 --2- 172.21.15.16:0/3280837578 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5d8c15e0b0 0x7f5d8c0cde60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:46:53.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.690+0000 7f5d8a59c700 1 --2- 172.21.15.16:0/3280837578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d8c0cf430 0x7f5d8c159000 secure :-1 s=READY pgs=2576 cs=0 l=1 rev1=1 crypto rx=0x7f5d800097c0 tx=0x7f5d80004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:53.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.690+0000 7f5d7affd700 1 -- 172.21.15.16:0/3280837578 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d800036a0 con 0x7f5d8c0cf430 2023-10-16T13:46:53.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.690+0000 7f5d7affd700 1 -- 172.21.15.16:0/3280837578 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d8001f070 con 0x7f5d8c0cf430 2023-10-16T13:46:53.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.691+0000 7f5d7affd700 1 -- 172.21.15.16:0/3280837578 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d80003a20 con 0x7f5d8c0cf430 2023-10-16T13:46:53.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.691+0000 7f5d90b9c700 1 -- 172.21.15.16:0/3280837578 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d8c0ad030 con 0x7f5d8c0cf430 2023-10-16T13:46:53.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.691+0000 7f5d90b9c700 1 -- 172.21.15.16:0/3280837578 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d8c0ce3a0 con 0x7f5d8c0cf430 2023-10-16T13:46:53.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.691+0000 7f5d90b9c700 1 -- 172.21.15.16:0/3280837578 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d8c0ad550 con 0x7f5d8c0cf430 2023-10-16T13:46:53.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.693+0000 7f5d7affd700 1 -- 172.21.15.16:0/3280837578 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 142) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f5d80003b80 con 0x7f5d8c0cf430 2023-10-16T13:46:53.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.693+0000 7f5d7affd700 1 --2- 172.21.15.16:0/3280837578 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f5d6c06ada0 0x7f5d6c06d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:53.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.693+0000 7f5d7affd700 1 -- 172.21.15.16:0/3280837578 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(230..230 src has 1..230) v4 ==== 7365+0+0 (secure 0 0 0) 0x7f5d80021070 con 0x7f5d8c0cf430 2023-10-16T13:46:53.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.693+0000 7f5d89d9b700 1 --2- 172.21.15.16:0/3280837578 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f5d6c06ada0 0x7f5d6c06d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:53.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.693+0000 7f5d89d9b700 1 --2- 172.21.15.16:0/3280837578 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f5d6c06ada0 0x7f5d6c06d260 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f5d8c0cdbf0 tx=0x7f5d7c008040 comp rx=0 tx=0).ready entity=mgr.11745 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:53.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.697+0000 7f5d7affd700 1 -- 172.21.15.16:0/3280837578 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5d800563f0 con 0x7f5d8c0cf430 2023-10-16T13:46:53.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.925+0000 7f5d90b9c700 1 -- 172.21.15.16:0/3280837578 --> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5d8c06e320 con 0x7f5d6c06ada0 2023-10-16T13:46:53.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.931+0000 7f5d7affd700 1 -- 172.21.15.16:0/3280837578 <== mgr.11745 v2:172.21.15.16:6832/1506142504 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+353 (secure 0 0 0) 0x7f5d7c0076b0 con 0x7f5d6c06ada0 2023-10-16T13:46:53.933 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:46:53.934 INFO:teuthology.orchestra.run.smithi016.stdout:[{"fs": "backup_fs", "subvol": null, "group": null, "path": "mandatory_fs_argument_test_dir", "rel_path": "mandatory_fs_argument_test_dir", "schedule": "1M", "retention": {"M": 1}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T13:46:52", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true}] 2023-10-16T13:46:53.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.933+0000 7f5d90b9c700 1 -- 172.21.15.16:0/3280837578 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f5d6c06ada0 msgr2=0x7f5d6c06d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:53.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.933+0000 7f5d90b9c700 1 --2- 172.21.15.16:0/3280837578 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f5d6c06ada0 0x7f5d6c06d260 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f5d8c0cdbf0 tx=0x7f5d7c008040 comp rx=0 tx=0).stop 2023-10-16T13:46:53.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.933+0000 7f5d90b9c700 1 -- 172.21.15.16:0/3280837578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d8c0cf430 msgr2=0x7f5d8c159000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:53.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.933+0000 7f5d90b9c700 1 --2- 172.21.15.16:0/3280837578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d8c0cf430 0x7f5d8c159000 secure :-1 s=READY pgs=2576 cs=0 l=1 rev1=1 crypto rx=0x7f5d800097c0 tx=0x7f5d80004750 comp rx=0 tx=0).stop 2023-10-16T13:46:53.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.933+0000 7f5d90b9c700 1 -- 172.21.15.16:0/3280837578 shutdown_connections 2023-10-16T13:46:53.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.933+0000 7f5d90b9c700 1 --2- 172.21.15.16:0/3280837578 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f5d6c06ada0 0x7f5d6c06d260 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:53.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.933+0000 7f5d90b9c700 1 --2- 172.21.15.16:0/3280837578 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5d8c15e0b0 0x7f5d8c0cde60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:53.937 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.933+0000 7f5d90b9c700 1 --2- 172.21.15.16:0/3280837578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d8c0cf430 0x7f5d8c159000 unknown :-1 s=CLOSED pgs=2576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:53.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.933+0000 7f5d90b9c700 1 --2- 172.21.15.16:0/3280837578 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5d8c15a3d0 0x7f5d8c0c4040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:53.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.933+0000 7f5d90b9c700 1 -- 172.21.15.16:0/3280837578 >> 172.21.15.16:0/3280837578 conn(0x7f5d8c06f430 msgr2=0x7f5d8c0cc570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:53.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.933+0000 7f5d90b9c700 1 -- 172.21.15.16:0/3280837578 shutdown_connections 2023-10-16T13:46:53.938 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:53.933+0000 7f5d90b9c700 1 -- 172.21.15.16:0/3280837578 wait complete. 2023-10-16T13:46:53.948 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "backup_fs", "subvol": null, "group": null, "path": "mandatory_fs_argument_test_dir", "rel_path": "mandatory_fs_argument_test_dir", "schedule": "1M", "retention": {"M": 1}, "start": "2023-10-16T00:00:00", "created": "2023-10-16T13:46:52", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true}] 2023-10-16T13:46:53.948 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention remove mandatory_fs_argument_test_dir --fs backup_fs 1M 2023-10-16T13:46:54.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.077+0000 7f560b2a1700 1 Processor -- start 2023-10-16T13:46:54.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.078+0000 7f560b2a1700 1 -- start start 2023-10-16T13:46:54.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.078+0000 7f560b2a1700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f560415d8b0 0x7f56040caa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:54.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.078+0000 7f560b2a1700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5604162440 con 0x7f56040cd330 2023-10-16T13:46:54.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.078+0000 7f560b2a1700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5604161c20 con 0x7f56040ceb20 2023-10-16T13:46:54.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.078+0000 7f560b2a1700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5604161e80 con 0x7f560415d8b0 2023-10-16T13:46:54.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.078+0000 7f560983e700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f56040cd330 0x7f5604072380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:43428/0 (socket says 172.21.15.16:43428) 2023-10-16T13:46:54.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.079+0000 7f560983e700 1 -- 172.21.15.16:0/3628325729 learned_addr learned my addr 172.21.15.16:0/3628325729 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:46:54.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.079+0000 7f560903d700 1 --2- 172.21.15.16:0/3628325729 >> v2:172.21.15.49:3300/0 conn(0x7f560415d8b0 0x7f56040caa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:54.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.079+0000 7f55fbfff700 1 -- 172.21.15.16:0/3628325729 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 605374104 0 0) 0x7f5604162440 con 0x7f56040cd330 2023-10-16T13:46:54.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.079+0000 7f55fbfff700 1 -- 172.21.15.16:0/3628325729 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55f00035b0 con 0x7f56040cd330 2023-10-16T13:46:54.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.079+0000 7f55fbfff700 1 -- 172.21.15.16:0/3628325729 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1153411608 0 0) 0x7f55f00035b0 con 0x7f56040cd330 2023-10-16T13:46:54.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.079+0000 7f55fbfff700 1 -- 172.21.15.16:0/3628325729 >> v2:172.21.15.49:3300/0 conn(0x7f560415d8b0 msgr2=0x7f56040caa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:54.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.079+0000 7f55fbfff700 1 --2- 172.21.15.16:0/3628325729 >> v2:172.21.15.49:3300/0 conn(0x7f560415d8b0 0x7f56040caa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:54.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.079+0000 7f55fbfff700 1 -- 172.21.15.16:0/3628325729 >> v1:172.21.15.49:6789/0 conn(0x7f56040ceb20 legacy=0x7f5604161510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:54.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.079+0000 7f55fbfff700 1 -- 172.21.15.16:0/3628325729 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5604162160 con 0x7f56040cd330 2023-10-16T13:46:54.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.079+0000 7f55fbfff700 1 -- 172.21.15.16:0/3628325729 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f55fc0019f0 con 0x7f56040cd330 2023-10-16T13:46:54.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.079+0000 7f55fbfff700 1 -- 172.21.15.16:0/3628325729 >> v1:172.21.15.16:6789/0 conn(0x7f56040cd330 legacy=0x7f5604072380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:54.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.079+0000 7f55fbfff700 1 --2- 172.21.15.16:0/3628325729 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55f0003a00 0x7f55f0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:54.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.079+0000 7f55fbfff700 1 --2- 172.21.15.16:0/3628325729 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f55f0004850 0x7f55f0004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:54.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.079+0000 7f55fbfff700 1 --2- 172.21.15.16:0/3628325729 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f55f0005210 0x7f55f000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:54.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.079+0000 7f55fbfff700 1 -- 172.21.15.16:0/3628325729 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5604161e80 con 0x7f55f0003a00 2023-10-16T13:46:54.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.079+0000 7f55fbfff700 1 -- 172.21.15.16:0/3628325729 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55f000c1b0 con 0x7f55f0005210 2023-10-16T13:46:54.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.079+0000 7f55fbfff700 1 -- 172.21.15.16:0/3628325729 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55f000c370 con 0x7f55f0004850 2023-10-16T13:46:54.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.079+0000 7f560903d700 1 --2- 172.21.15.16:0/3628325729 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55f0003a00 0x7f55f0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:54.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.079+0000 7f560983e700 1 --2- 172.21.15.16:0/3628325729 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f55f0005210 0x7f55f000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:54.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.079+0000 7f560883c700 1 --2- 172.21.15.16:0/3628325729 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f55f0004850 0x7f55f0004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:54.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.080+0000 7f560903d700 1 -- 172.21.15.16:0/3628325729 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f55f0004850 msgr2=0x7f55f0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:54.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.080+0000 7f560903d700 1 --2- 172.21.15.16:0/3628325729 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f55f0004850 0x7f55f0004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:54.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.080+0000 7f560903d700 1 -- 172.21.15.16:0/3628325729 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f55f0005210 msgr2=0x7f55f000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:54.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.080+0000 7f560903d700 1 --2- 172.21.15.16:0/3628325729 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f55f0005210 0x7f55f000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:54.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.080+0000 7f560983e700 1 -- 172.21.15.16:0/3628325729 reap_dead start 2023-10-16T13:46:54.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.080+0000 7f560903d700 1 -- 172.21.15.16:0/3628325729 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55f00035b0 con 0x7f55f0003a00 2023-10-16T13:46:54.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.080+0000 7f560883c700 1 --2- 172.21.15.16:0/3628325729 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f55f0004850 0x7f55f0004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:46:54.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.080+0000 7f560903d700 1 --2- 172.21.15.16:0/3628325729 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55f0003a00 0x7f55f0003e20 secure :-1 s=READY pgs=2578 cs=0 l=1 rev1=1 crypto rx=0x7f55f4009b60 tx=0x7f55f0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35b72a0441732d33 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:54.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.080+0000 7f55fbfff700 1 -- 172.21.15.16:0/3628325729 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55f400e6c0 con 0x7f55f0003a00 2023-10-16T13:46:54.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.080+0000 7f55fbfff700 1 -- 172.21.15.16:0/3628325729 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55f400e820 con 0x7f55f0003a00 2023-10-16T13:46:54.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.080+0000 7f55fbfff700 1 -- 172.21.15.16:0/3628325729 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55f400eb90 con 0x7f55f0003a00 2023-10-16T13:46:54.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.080+0000 7f560b2a1700 1 -- 172.21.15.16:0/3628325729 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55f0003a00 msgr2=0x7f55f0003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:54.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.080+0000 7f560b2a1700 1 --2- 172.21.15.16:0/3628325729 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55f0003a00 0x7f55f0003e20 secure :-1 s=READY pgs=2578 cs=0 l=1 rev1=1 crypto rx=0x7f55f4009b60 tx=0x7f55f0005880 comp rx=0 tx=0).stop 2023-10-16T13:46:54.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.080+0000 7f560b2a1700 1 -- 172.21.15.16:0/3628325729 shutdown_connections 2023-10-16T13:46:54.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.080+0000 7f560b2a1700 1 --2- 172.21.15.16:0/3628325729 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55f0003a00 0x7f55f0003e20 unknown :-1 s=CLOSED pgs=2578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:54.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.080+0000 7f560b2a1700 1 -- 172.21.15.16:0/3628325729 >> 172.21.15.16:0/3628325729 conn(0x7f560406bce0 msgr2=0x7f560406c0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:54.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.081+0000 7f560b2a1700 1 -- 172.21.15.16:0/3628325729 shutdown_connections 2023-10-16T13:46:54.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.081+0000 7f560b2a1700 1 -- 172.21.15.16:0/3628325729 wait complete. 2023-10-16T13:46:54.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.082+0000 7f560b2a1700 1 Processor -- start 2023-10-16T13:46:54.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.082+0000 7f560b2a1700 1 -- start start 2023-10-16T13:46:54.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.082+0000 7f560b2a1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55f0003a00 0x7f56041f7470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:54.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.082+0000 7f560b2a1700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f56040c7480 0x7f56041587a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:54.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.082+0000 7f560b2a1700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f560415a5c0 0x7f560415aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:54.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.082+0000 7f560b2a1700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56041f7a80 con 0x7f55f0003a00 2023-10-16T13:46:54.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.082+0000 7f560b2a1700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f560415afb0 con 0x7f56040c7480 2023-10-16T13:46:54.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.082+0000 7f560b2a1700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56041f4470 con 0x7f560415a5c0 2023-10-16T13:46:54.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.083+0000 7f560903d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55f0003a00 0x7f56041f7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:54.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.083+0000 7f560903d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55f0003a00 0x7f56041f7470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54162/0 (socket says 172.21.15.16:54162) 2023-10-16T13:46:54.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.083+0000 7f560903d700 1 -- 172.21.15.16:0/4104054510 learned_addr learned my addr 172.21.15.16:0/4104054510 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:54.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.083+0000 7f560983e700 1 --2- 172.21.15.16:0/4104054510 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f560415a5c0 0x7f560415aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:54.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.083+0000 7f560883c700 1 --2- 172.21.15.16:0/4104054510 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f56040c7480 0x7f56041587a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:54.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.083+0000 7f560903d700 1 -- 172.21.15.16:0/4104054510 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f560415a5c0 msgr2=0x7f560415aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:54.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.083+0000 7f560903d700 1 --2- 172.21.15.16:0/4104054510 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f560415a5c0 0x7f560415aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:54.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.083+0000 7f560903d700 1 -- 172.21.15.16:0/4104054510 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f56040c7480 msgr2=0x7f56041587a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:54.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.083+0000 7f560903d700 1 --2- 172.21.15.16:0/4104054510 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f56040c7480 0x7f56041587a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:54.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.083+0000 7f560903d700 1 -- 172.21.15.16:0/4104054510 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56041f3470 con 0x7f55f0003a00 2023-10-16T13:46:54.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.083+0000 7f560883c700 1 --2- 172.21.15.16:0/4104054510 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f56040c7480 0x7f56041587a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:46:54.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.083+0000 7f560903d700 1 --2- 172.21.15.16:0/4104054510 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55f0003a00 0x7f56041f7470 secure :-1 s=READY pgs=2579 cs=0 l=1 rev1=1 crypto rx=0x7f55f400e4b0 tx=0x7f55f40047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:54.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.083+0000 7f55f9ffb700 1 -- 172.21.15.16:0/4104054510 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55f400e6c0 con 0x7f55f0003a00 2023-10-16T13:46:54.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.083+0000 7f55f9ffb700 1 -- 172.21.15.16:0/4104054510 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55f4003ad0 con 0x7f55f0003a00 2023-10-16T13:46:54.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.083+0000 7f560b2a1700 1 -- 172.21.15.16:0/4104054510 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5604158ce0 con 0x7f55f0003a00 2023-10-16T13:46:54.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.083+0000 7f55f9ffb700 1 -- 172.21.15.16:0/4104054510 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55f4003e80 con 0x7f55f0003a00 2023-10-16T13:46:54.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.083+0000 7f560b2a1700 1 -- 172.21.15.16:0/4104054510 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56041f36b0 con 0x7f55f0003a00 2023-10-16T13:46:54.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.084+0000 7f560b2a1700 1 -- 172.21.15.16:0/4104054510 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56041f42c0 con 0x7f55f0003a00 2023-10-16T13:46:54.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.085+0000 7f55f9ffb700 1 -- 172.21.15.16:0/4104054510 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 142) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f55f4013470 con 0x7f55f0003a00 2023-10-16T13:46:54.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.085+0000 7f55f9ffb700 1 --2- 172.21.15.16:0/4104054510 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f55e806ada0 0x7f55e806d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:54.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.085+0000 7f55f9ffb700 1 -- 172.21.15.16:0/4104054510 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(230..230 src has 1..230) v4 ==== 7365+0+0 (secure 0 0 0) 0x7f55f408af20 con 0x7f55f0003a00 2023-10-16T13:46:54.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.085+0000 7f560883c700 1 --2- 172.21.15.16:0/4104054510 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f55e806ada0 0x7f55e806d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:54.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.086+0000 7f560883c700 1 --2- 172.21.15.16:0/4104054510 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f55e806ada0 0x7f55e806d260 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f5600000c00 tx=0x7f5600009040 comp rx=0 tx=0).ready entity=mgr.11745 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:54.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.090+0000 7f55f9ffb700 1 -- 172.21.15.16:0/4104054510 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f55f4055db0 con 0x7f55f0003a00 2023-10-16T13:46:54.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.185+0000 7f55f9ffb700 1 -- 172.21.15.16:0/4104054510 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 143) v1 ==== 89451+0+0 (secure 0 0 0) 0x7f55f40096f0 con 0x7f55f0003a00 2023-10-16T13:46:54.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.327+0000 7f560b2a1700 1 -- 172.21.15.16:0/4104054510 --> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention remove", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "retention_spec_or_period": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f5604071e60 con 0x7f55e806ada0 2023-10-16T13:46:54.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.387+0000 7f55f9ffb700 1 -- 172.21.15.16:0/4104054510 <== mgr.11745 v2:172.21.15.16:6832/1506142504 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f560000d050 con 0x7f55e806ada0 2023-10-16T13:46:54.390 INFO:teuthology.orchestra.run.smithi016.stdout:Retention removed from path mandatory_fs_argument_test_dir 2023-10-16T13:46:54.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.389+0000 7f560b2a1700 1 -- 172.21.15.16:0/4104054510 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f55e806ada0 msgr2=0x7f55e806d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:54.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.389+0000 7f560b2a1700 1 --2- 172.21.15.16:0/4104054510 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f55e806ada0 0x7f55e806d260 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f5600000c00 tx=0x7f5600009040 comp rx=0 tx=0).stop 2023-10-16T13:46:54.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.389+0000 7f560b2a1700 1 -- 172.21.15.16:0/4104054510 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55f0003a00 msgr2=0x7f56041f7470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:54.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.389+0000 7f560b2a1700 1 --2- 172.21.15.16:0/4104054510 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55f0003a00 0x7f56041f7470 secure :-1 s=READY pgs=2579 cs=0 l=1 rev1=1 crypto rx=0x7f55f400e4b0 tx=0x7f55f40047c0 comp rx=0 tx=0).stop 2023-10-16T13:46:54.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.389+0000 7f560b2a1700 1 -- 172.21.15.16:0/4104054510 shutdown_connections 2023-10-16T13:46:54.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.389+0000 7f560b2a1700 1 --2- 172.21.15.16:0/4104054510 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f55e806ada0 0x7f55e806d260 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:54.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.389+0000 7f560b2a1700 1 --2- 172.21.15.16:0/4104054510 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f560415a5c0 0x7f560415aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:54.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.389+0000 7f560b2a1700 1 --2- 172.21.15.16:0/4104054510 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55f0003a00 0x7f56041f7470 unknown :-1 s=CLOSED pgs=2579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:54.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.389+0000 7f560b2a1700 1 --2- 172.21.15.16:0/4104054510 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f56040c7480 0x7f56041587a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:54.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.389+0000 7f560b2a1700 1 -- 172.21.15.16:0/4104054510 >> 172.21.15.16:0/4104054510 conn(0x7f560406bce0 msgr2=0x7f560415bad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:54.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.389+0000 7f560b2a1700 1 -- 172.21.15.16:0/4104054510 shutdown_connections 2023-10-16T13:46:54.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.389+0000 7f560b2a1700 1 -- 172.21.15.16:0/4104054510 wait complete. 2023-10-16T13:46:54.404 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention removed from path mandatory_fs_argument_test_dir 2023-10-16T13:46:54.405 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate mandatory_fs_argument_test_dir --fs backup_fs 1M 2023-10-16T13:46:54.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.532+0000 7f2ffadd4700 1 Processor -- start 2023-10-16T13:46:54.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.532+0000 7f2ffadd4700 1 -- start start 2023-10-16T13:46:54.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.533+0000 7f2ffadd4700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f2ff406d980 0x7f2ff415efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:54.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.533+0000 7f2ffadd4700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ff40cdeb0 con 0x7f2ff406a7e0 2023-10-16T13:46:54.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.533+0000 7f2ffadd4700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ff41613e0 con 0x7f2ff4160e60 2023-10-16T13:46:54.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.533+0000 7f2ffadd4700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ff41582d0 con 0x7f2ff406d980 2023-10-16T13:46:54.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.533+0000 7f2ff9371700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2ff406a7e0 0x7f2ff4161870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:43448/0 (socket says 172.21.15.16:43448) 2023-10-16T13:46:54.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.533+0000 7f2ff9371700 1 -- 172.21.15.16:0/2685842316 learned_addr learned my addr 172.21.15.16:0/2685842316 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:46:54.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.533+0000 7f2ff3fff700 1 --2- 172.21.15.16:0/2685842316 >> v2:172.21.15.49:3300/0 conn(0x7f2ff406d980 0x7f2ff415efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:54.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.533+0000 7f2ff37fe700 1 -- 172.21.15.16:0/2685842316 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2029397200 0 0) 0x7f2ff40cdeb0 con 0x7f2ff406a7e0 2023-10-16T13:46:54.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.533+0000 7f2ff37fe700 1 -- 172.21.15.16:0/2685842316 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fe40035b0 con 0x7f2ff406a7e0 2023-10-16T13:46:54.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.533+0000 7f2ff37fe700 1 -- 172.21.15.16:0/2685842316 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4269088581 0 0) 0x7f2ff41613e0 con 0x7f2ff4160e60 2023-10-16T13:46:54.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.533+0000 7f2ff37fe700 1 -- 172.21.15.16:0/2685842316 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ff40cdeb0 con 0x7f2ff4160e60 2023-10-16T13:46:54.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.533+0000 7f2ff37fe700 1 -- 172.21.15.16:0/2685842316 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3547299242 0 0) 0x7f2fe40035b0 con 0x7f2ff406a7e0 2023-10-16T13:46:54.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.533+0000 7f2ff37fe700 1 -- 172.21.15.16:0/2685842316 >> v2:172.21.15.49:3300/0 conn(0x7f2ff406d980 msgr2=0x7f2ff415efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:54.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.533+0000 7f2ff37fe700 1 --2- 172.21.15.16:0/2685842316 >> v2:172.21.15.49:3300/0 conn(0x7f2ff406d980 0x7f2ff415efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:54.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.533+0000 7f2ff37fe700 1 -- 172.21.15.16:0/2685842316 >> v1:172.21.15.49:6789/0 conn(0x7f2ff4160e60 legacy=0x7f2ff415e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:54.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.533+0000 7f2ff37fe700 1 -- 172.21.15.16:0/2685842316 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ff40ce100 con 0x7f2ff406a7e0 2023-10-16T13:46:54.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.533+0000 7f2ff37fe700 1 -- 172.21.15.16:0/2685842316 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f2fec0019f0 con 0x7f2ff406a7e0 2023-10-16T13:46:54.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.533+0000 7f2ff37fe700 1 -- 172.21.15.16:0/2685842316 >> v1:172.21.15.16:6789/0 conn(0x7f2ff406a7e0 legacy=0x7f2ff4161870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:54.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.533+0000 7f2ff37fe700 1 --2- 172.21.15.16:0/2685842316 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2fe4003be0 0x7f2fe4003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:54.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.533+0000 7f2ff37fe700 1 --2- 172.21.15.16:0/2685842316 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2fe4004850 0x7f2fe4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:54.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.533+0000 7f2ff37fe700 1 --2- 172.21.15.16:0/2685842316 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2fe4005260 0x7f2fe400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:54.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.533+0000 7f2ff37fe700 1 -- 172.21.15.16:0/2685842316 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff41582d0 con 0x7f2fe4004850 2023-10-16T13:46:54.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.533+0000 7f2ff37fe700 1 -- 172.21.15.16:0/2685842316 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe400c190 con 0x7f2fe4005260 2023-10-16T13:46:54.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.533+0000 7f2ff37fe700 1 -- 172.21.15.16:0/2685842316 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe400c350 con 0x7f2fe4003be0 2023-10-16T13:46:54.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.534+0000 7f2ff3fff700 1 --2- 172.21.15.16:0/2685842316 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2fe4004850 0x7f2fe4004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:54.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.534+0000 7f2ff9371700 1 --2- 172.21.15.16:0/2685842316 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2fe4005260 0x7f2fe400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:54.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.534+0000 7f2ff8b70700 1 --2- 172.21.15.16:0/2685842316 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2fe4003be0 0x7f2fe4003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:54.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.534+0000 7f2ff3fff700 1 -- 172.21.15.16:0/2685842316 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2fe4003be0 msgr2=0x7f2fe4003fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:54.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.534+0000 7f2ff3fff700 1 --2- 172.21.15.16:0/2685842316 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2fe4003be0 0x7f2fe4003fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:54.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.534+0000 7f2ff3fff700 1 -- 172.21.15.16:0/2685842316 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2fe4005260 msgr2=0x7f2fe400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:54.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.534+0000 7f2ff3fff700 1 --2- 172.21.15.16:0/2685842316 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2fe4005260 0x7f2fe400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:54.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.534+0000 7f2ff3fff700 1 -- 172.21.15.16:0/2685842316 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fe40035b0 con 0x7f2fe4004850 2023-10-16T13:46:54.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.534+0000 7f2ff9371700 1 -- 172.21.15.16:0/2685842316 reap_dead start 2023-10-16T13:46:54.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.534+0000 7f2ff8b70700 1 --2- 172.21.15.16:0/2685842316 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2fe4003be0 0x7f2fe4003fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:46:54.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.534+0000 7f2ff3fff700 1 --2- 172.21.15.16:0/2685842316 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2fe4004850 0x7f2fe4004cd0 secure :-1 s=READY pgs=2581 cs=0 l=1 rev1=1 crypto rx=0x7f2ff4070280 tx=0x7f2fe4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f1808874954ccf server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:54.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.534+0000 7f2ff37fe700 1 -- 172.21.15.16:0/2685842316 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe000e6c0 con 0x7f2fe4004850 2023-10-16T13:46:54.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.534+0000 7f2ff37fe700 1 -- 172.21.15.16:0/2685842316 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fe000e820 con 0x7f2fe4004850 2023-10-16T13:46:54.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.534+0000 7f2ff37fe700 1 -- 172.21.15.16:0/2685842316 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe000eb90 con 0x7f2fe4004850 2023-10-16T13:46:54.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.535+0000 7f2ffadd4700 1 -- 172.21.15.16:0/2685842316 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2fe4004850 msgr2=0x7f2fe4004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:54.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.535+0000 7f2ffadd4700 1 --2- 172.21.15.16:0/2685842316 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2fe4004850 0x7f2fe4004cd0 secure :-1 s=READY pgs=2581 cs=0 l=1 rev1=1 crypto rx=0x7f2ff4070280 tx=0x7f2fe4005880 comp rx=0 tx=0).stop 2023-10-16T13:46:54.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.535+0000 7f2ffadd4700 1 -- 172.21.15.16:0/2685842316 shutdown_connections 2023-10-16T13:46:54.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.535+0000 7f2ffadd4700 1 --2- 172.21.15.16:0/2685842316 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2fe4004850 0x7f2fe4004cd0 unknown :-1 s=CLOSED pgs=2581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:54.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.535+0000 7f2ffadd4700 1 -- 172.21.15.16:0/2685842316 >> 172.21.15.16:0/2685842316 conn(0x7f2ff406f430 msgr2=0x7f2ff406f810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:54.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.535+0000 7f2ffadd4700 1 -- 172.21.15.16:0/2685842316 shutdown_connections 2023-10-16T13:46:54.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.535+0000 7f2ffadd4700 1 -- 172.21.15.16:0/2685842316 wait complete. 2023-10-16T13:46:54.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.536+0000 7f2ffadd4700 1 Processor -- start 2023-10-16T13:46:54.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.536+0000 7f2ffadd4700 1 -- start start 2023-10-16T13:46:54.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.537+0000 7f2ffadd4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2fe4004850 0x7f2ff41f3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:54.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.537+0000 7f2ffadd4700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2ff40c4090 0x7f2ff415f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:54.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.537+0000 7f2ffadd4700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2ff41555f0 0x7f2ff4155a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:54.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.537+0000 7f2ffadd4700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff40b4080 con 0x7f2fe4004850 2023-10-16T13:46:54.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.537+0000 7f2ffadd4700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff4155fa0 con 0x7f2ff40c4090 2023-10-16T13:46:54.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.537+0000 7f2ffadd4700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff40c48a0 con 0x7f2ff41555f0 2023-10-16T13:46:54.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.537+0000 7f2ff8b70700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2fe4004850 0x7f2ff41f3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:54.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.537+0000 7f2ff8b70700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2fe4004850 0x7f2ff41f3e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54182/0 (socket says 172.21.15.16:54182) 2023-10-16T13:46:54.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.537+0000 7f2ff8b70700 1 -- 172.21.15.16:0/2651231139 learned_addr learned my addr 172.21.15.16:0/2651231139 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:54.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.537+0000 7f2ff3fff700 1 --2- 172.21.15.16:0/2651231139 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2ff40c4090 0x7f2ff415f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:54.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.537+0000 7f2ff9371700 1 --2- 172.21.15.16:0/2651231139 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2ff41555f0 0x7f2ff4155a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:54.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.537+0000 7f2ff8b70700 1 -- 172.21.15.16:0/2651231139 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2ff41555f0 msgr2=0x7f2ff4155a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:54.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.537+0000 7f2ff8b70700 1 --2- 172.21.15.16:0/2651231139 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2ff41555f0 0x7f2ff4155a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:54.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.537+0000 7f2ff8b70700 1 -- 172.21.15.16:0/2651231139 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2ff40c4090 msgr2=0x7f2ff415f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:54.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.537+0000 7f2ff8b70700 1 --2- 172.21.15.16:0/2651231139 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2ff40c4090 0x7f2ff415f4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:54.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.537+0000 7f2ff8b70700 1 -- 172.21.15.16:0/2651231139 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ff40c4b80 con 0x7f2fe4004850 2023-10-16T13:46:54.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.538+0000 7f2ff8b70700 1 --2- 172.21.15.16:0/2651231139 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2fe4004850 0x7f2ff41f3e40 secure :-1 s=READY pgs=2582 cs=0 l=1 rev1=1 crypto rx=0x7f2fe800e570 tx=0x7f2fe800e880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:54.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.538+0000 7f2ff17fa700 1 -- 172.21.15.16:0/2651231139 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe80052e0 con 0x7f2fe4004850 2023-10-16T13:46:54.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.538+0000 7f2ff17fa700 1 -- 172.21.15.16:0/2651231139 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fe8005440 con 0x7f2fe4004850 2023-10-16T13:46:54.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.538+0000 7f2ffadd4700 1 -- 172.21.15.16:0/2651231139 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ff40c4e60 con 0x7f2fe4004850 2023-10-16T13:46:54.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.538+0000 7f2ff17fa700 1 -- 172.21.15.16:0/2651231139 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe80038c0 con 0x7f2fe4004850 2023-10-16T13:46:54.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.538+0000 7f2ffadd4700 1 -- 172.21.15.16:0/2651231139 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ff40c5140 con 0x7f2fe4004850 2023-10-16T13:46:54.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.539+0000 7f2ffadd4700 1 -- 172.21.15.16:0/2651231139 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ff40b4250 con 0x7f2fe4004850 2023-10-16T13:46:54.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.539+0000 7f2ff17fa700 1 -- 172.21.15.16:0/2651231139 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 143) v1 ==== 89451+0+0 (secure 0 0 0) 0x7f2fe8003a20 con 0x7f2fe4004850 2023-10-16T13:46:54.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.539+0000 7f2ff17fa700 1 --2- 172.21.15.16:0/2651231139 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f2fd806b190 0x7f2fd806d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:54.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.540+0000 7f2ff17fa700 1 -- 172.21.15.16:0/2651231139 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(230..230 src has 1..230) v4 ==== 7365+0+0 (secure 0 0 0) 0x7f2fe8089750 con 0x7f2fe4004850 2023-10-16T13:46:54.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.540+0000 7f2ff3fff700 1 --2- 172.21.15.16:0/2651231139 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f2fd806b190 0x7f2fd806d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:54.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.540+0000 7f2ff3fff700 1 --2- 172.21.15.16:0/2651231139 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f2fd806b190 0x7f2fd806d650 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f2fe0000c00 tx=0x7f2fe001e040 comp rx=0 tx=0).ready entity=mgr.11745 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:54.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.544+0000 7f2ff17fa700 1 -- 172.21.15.16:0/2651231139 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2ff40b4250 con 0x7f2fe4004850 2023-10-16T13:46:54.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.780+0000 7f2ffadd4700 1 -- 172.21.15.16:0/2651231139 --> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f2ff40ce6c0 con 0x7f2fd806b190 2023-10-16T13:46:54.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.835+0000 7f2ff17fa700 1 -- 172.21.15.16:0/2651231139 <== mgr.11745 v2:172.21.15.16:6832/1506142504 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+61 (secure 0 0 0) 0x7f2fe000ea90 con 0x7f2fd806b190 2023-10-16T13:46:54.837 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule deactivated for path mandatory_fs_argument_test_dir 2023-10-16T13:46:54.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.837+0000 7f2ffadd4700 1 -- 172.21.15.16:0/2651231139 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f2fd806b190 msgr2=0x7f2fd806d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:54.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.837+0000 7f2ffadd4700 1 --2- 172.21.15.16:0/2651231139 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f2fd806b190 0x7f2fd806d650 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f2fe0000c00 tx=0x7f2fe001e040 comp rx=0 tx=0).stop 2023-10-16T13:46:54.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.837+0000 7f2ffadd4700 1 -- 172.21.15.16:0/2651231139 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2fe4004850 msgr2=0x7f2ff41f3e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:54.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.837+0000 7f2ffadd4700 1 --2- 172.21.15.16:0/2651231139 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2fe4004850 0x7f2ff41f3e40 secure :-1 s=READY pgs=2582 cs=0 l=1 rev1=1 crypto rx=0x7f2fe800e570 tx=0x7f2fe800e880 comp rx=0 tx=0).stop 2023-10-16T13:46:54.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.837+0000 7f2ffadd4700 1 -- 172.21.15.16:0/2651231139 shutdown_connections 2023-10-16T13:46:54.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.837+0000 7f2ffadd4700 1 --2- 172.21.15.16:0/2651231139 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f2fd806b190 0x7f2fd806d650 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:54.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.837+0000 7f2ffadd4700 1 --2- 172.21.15.16:0/2651231139 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2ff41555f0 0x7f2ff4155a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:54.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.837+0000 7f2ffadd4700 1 --2- 172.21.15.16:0/2651231139 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2fe4004850 0x7f2ff41f3e40 unknown :-1 s=CLOSED pgs=2582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:54.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.837+0000 7f2ffadd4700 1 --2- 172.21.15.16:0/2651231139 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2ff40c4090 0x7f2ff415f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:54.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.837+0000 7f2ffadd4700 1 -- 172.21.15.16:0/2651231139 >> 172.21.15.16:0/2651231139 conn(0x7f2ff406f430 msgr2=0x7f2ff41562d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:54.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.837+0000 7f2ffadd4700 1 -- 172.21.15.16:0/2651231139 shutdown_connections 2023-10-16T13:46:54.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.837+0000 7f2ffadd4700 1 -- 172.21.15.16:0/2651231139 wait complete. 2023-10-16T13:46:54.851 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path mandatory_fs_argument_test_dir 2023-10-16T13:46:54.852 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove mandatory_fs_argument_test_dir --fs backup_fs 1M 2023-10-16T13:46:54.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.980+0000 7f6ff16a7700 1 Processor -- start 2023-10-16T13:46:54.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.980+0000 7f6ff16a7700 1 -- start start 2023-10-16T13:46:54.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.981+0000 7f6ff16a7700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6fec0ce260 0x7f6fec0ce6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:54.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.981+0000 7f6ff16a7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6fec0d1740 0x7f6fec0d1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:54.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.981+0000 7f6ff16a7700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fec0b91b0 con 0x7f6fec15d7a0 2023-10-16T13:46:54.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.981+0000 7f6ff16a7700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fec0b8980 con 0x7f6fec0d1740 2023-10-16T13:46:54.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.981+0000 7f6ff16a7700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fec0cec20 con 0x7f6fec0ce260 2023-10-16T13:46:54.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.981+0000 7f6feb7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6fec0d1740 0x7f6fec0d1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:54.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.981+0000 7f6feaffd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f6fec15d7a0 0x7f6fec0b7240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:43464/0 (socket says 172.21.15.16:43464) 2023-10-16T13:46:54.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.981+0000 7f6feaffd700 1 -- 172.21.15.16:0/2746340712 learned_addr learned my addr 172.21.15.16:0/2746340712 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:46:54.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.981+0000 7f6fea7fc700 1 --2- 172.21.15.16:0/2746340712 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6fec0ce260 0x7f6fec0ce6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:54.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.981+0000 7f6feb7fe700 1 -- 172.21.15.16:0/2746340712 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6fec0ce260 msgr2=0x7f6fec0ce6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:54.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.981+0000 7f6feb7fe700 1 --2- 172.21.15.16:0/2746340712 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6fec0ce260 0x7f6fec0ce6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:54.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.981+0000 7f6feb7fe700 1 -- 172.21.15.16:0/2746340712 >> v1:172.21.15.16:6789/0 conn(0x7f6fec15d7a0 legacy=0x7f6fec0b7240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:54.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.981+0000 7f6feb7fe700 1 -- 172.21.15.16:0/2746340712 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fec0b7d00 con 0x7f6fec0d1740 2023-10-16T13:46:54.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.981+0000 7f6fe9ffb700 1 -- 172.21.15.16:0/2746340712 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3650903901 0 0) 0x7f6fec0b91b0 con 0x7f6fec15d7a0 2023-10-16T13:46:54.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.981+0000 7f6fea7fc700 1 --2- 172.21.15.16:0/2746340712 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6fec0ce260 0x7f6fec0ce6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:46:54.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.982+0000 7f6feb7fe700 1 --2- 172.21.15.16:0/2746340712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fec0d1740 0x7f6fec0d1bd0 secure :-1 s=READY pgs=2584 cs=0 l=1 rev1=1 crypto rx=0x7f6fe4009d90 tx=0x7f6fe400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=236cd6886f3e42f2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:54.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.982+0000 7f6fe9ffb700 1 -- 172.21.15.16:0/2746340712 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe400e070 con 0x7f6fec0d1740 2023-10-16T13:46:54.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.982+0000 7f6fe9ffb700 1 -- 172.21.15.16:0/2746340712 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fe400bbd0 con 0x7f6fec0d1740 2023-10-16T13:46:54.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.982+0000 7f6fe9ffb700 1 -- 172.21.15.16:0/2746340712 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe4012430 con 0x7f6fec0d1740 2023-10-16T13:46:54.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.982+0000 7f6ff16a7700 1 -- 172.21.15.16:0/2746340712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fec0d1740 msgr2=0x7f6fec0d1bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:54.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.982+0000 7f6ff16a7700 1 --2- 172.21.15.16:0/2746340712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fec0d1740 0x7f6fec0d1bd0 secure :-1 s=READY pgs=2584 cs=0 l=1 rev1=1 crypto rx=0x7f6fe4009d90 tx=0x7f6fe400b330 comp rx=0 tx=0).stop 2023-10-16T13:46:54.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.982+0000 7f6ff16a7700 1 -- 172.21.15.16:0/2746340712 shutdown_connections 2023-10-16T13:46:54.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.982+0000 7f6ff16a7700 1 --2- 172.21.15.16:0/2746340712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fec0d1740 0x7f6fec0d1bd0 unknown :-1 s=CLOSED pgs=2584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:54.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.982+0000 7f6ff16a7700 1 --2- 172.21.15.16:0/2746340712 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6fec0ce260 0x7f6fec0ce6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:54.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.982+0000 7f6ff16a7700 1 -- 172.21.15.16:0/2746340712 >> 172.21.15.16:0/2746340712 conn(0x7f6fec072330 msgr2=0x7f6fec072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:54.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.982+0000 7f6ff16a7700 1 -- 172.21.15.16:0/2746340712 shutdown_connections 2023-10-16T13:46:54.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.982+0000 7f6ff16a7700 1 -- 172.21.15.16:0/2746340712 wait complete. 2023-10-16T13:46:54.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.984+0000 7f6ff16a7700 1 Processor -- start 2023-10-16T13:46:54.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.984+0000 7f6ff16a7700 1 -- start start 2023-10-16T13:46:54.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.984+0000 7f6ff16a7700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6fec0ce260 0x7f6fec0cc800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:54.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.984+0000 7f6ff16a7700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6fec0d1740 0x7f6fec1518e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:54.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.984+0000 7f6ff16a7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fec15d7a0 0x7f6fec0c8390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:54.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.984+0000 7f6ff16a7700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fec151e20 con 0x7f6fec15d7a0 2023-10-16T13:46:54.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.984+0000 7f6ff16a7700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fec0c72f0 con 0x7f6fec0ce260 2023-10-16T13:46:54.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.984+0000 7f6ff16a7700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fec0c75d0 con 0x7f6fec0d1740 2023-10-16T13:46:54.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.984+0000 7f6feb7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fec15d7a0 0x7f6fec0c8390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:54.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.985+0000 7f6feb7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fec15d7a0 0x7f6fec0c8390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54200/0 (socket says 172.21.15.16:54200) 2023-10-16T13:46:54.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.985+0000 7f6feb7fe700 1 -- 172.21.15.16:0/2655231278 learned_addr learned my addr 172.21.15.16:0/2655231278 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:54.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.985+0000 7f6fea7fc700 1 --2- 172.21.15.16:0/2655231278 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6fec0d1740 0x7f6fec1518e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:54.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.985+0000 7f6feaffd700 1 --2- 172.21.15.16:0/2655231278 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6fec0ce260 0x7f6fec0cc800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:54.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.985+0000 7f6feb7fe700 1 -- 172.21.15.16:0/2655231278 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6fec0d1740 msgr2=0x7f6fec1518e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:54.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.985+0000 7f6feb7fe700 1 --2- 172.21.15.16:0/2655231278 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6fec0d1740 0x7f6fec1518e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:54.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.985+0000 7f6feb7fe700 1 -- 172.21.15.16:0/2655231278 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6fec0ce260 msgr2=0x7f6fec0cc800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:54.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.985+0000 7f6feb7fe700 1 --2- 172.21.15.16:0/2655231278 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6fec0ce260 0x7f6fec0cc800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:54.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.985+0000 7f6feb7fe700 1 -- 172.21.15.16:0/2655231278 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fe4009a70 con 0x7f6fec15d7a0 2023-10-16T13:46:54.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.985+0000 7f6feaffd700 1 --2- 172.21.15.16:0/2655231278 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6fec0ce260 0x7f6fec0cc800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:46:54.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.985+0000 7f6feb7fe700 1 --2- 172.21.15.16:0/2655231278 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fec15d7a0 0x7f6fec0c8390 secure :-1 s=READY pgs=2585 cs=0 l=1 rev1=1 crypto rx=0x7f6fec0d4390 tx=0x7f6fe4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:54.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.985+0000 7f6fd37fe700 1 -- 172.21.15.16:0/2655231278 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe4012430 con 0x7f6fec15d7a0 2023-10-16T13:46:54.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.985+0000 7f6fd37fe700 1 -- 172.21.15.16:0/2655231278 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fe4020070 con 0x7f6fec15d7a0 2023-10-16T13:46:54.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.985+0000 7f6fd37fe700 1 -- 172.21.15.16:0/2655231278 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe4004260 con 0x7f6fec15d7a0 2023-10-16T13:46:54.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.985+0000 7f6ff16a7700 1 -- 172.21.15.16:0/2655231278 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fec0c78b0 con 0x7f6fec15d7a0 2023-10-16T13:46:54.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.985+0000 7f6ff16a7700 1 -- 172.21.15.16:0/2655231278 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fec0c7e70 con 0x7f6fec15d7a0 2023-10-16T13:46:54.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.986+0000 7f6ff16a7700 1 -- 172.21.15.16:0/2655231278 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fec04ec20 con 0x7f6fec15d7a0 2023-10-16T13:46:54.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.987+0000 7f6fd37fe700 1 -- 172.21.15.16:0/2655231278 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 143) v1 ==== 89451+0+0 (secure 0 0 0) 0x7f6fe40043c0 con 0x7f6fec15d7a0 2023-10-16T13:46:54.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.987+0000 7f6fd37fe700 1 --2- 172.21.15.16:0/2655231278 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f6fd806b190 0x7f6fd806d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:54.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.987+0000 7f6fd37fe700 1 -- 172.21.15.16:0/2655231278 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(230..230 src has 1..230) v4 ==== 7365+0+0 (secure 0 0 0) 0x7f6fe4022070 con 0x7f6fec15d7a0 2023-10-16T13:46:54.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.987+0000 7f6feaffd700 1 --2- 172.21.15.16:0/2655231278 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f6fd806b190 0x7f6fd806d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:54.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.988+0000 7f6feaffd700 1 --2- 172.21.15.16:0/2655231278 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f6fd806b190 0x7f6fd806d650 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6fe0000f60 tx=0x7f6fe0007ca0 comp rx=0 tx=0).ready entity=mgr.11745 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:54.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:54.992+0000 7f6fd37fe700 1 -- 172.21.15.16:0/2655231278 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6fe40566e0 con 0x7f6fec15d7a0 2023-10-16T13:46:55.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:55.221+0000 7f6ff16a7700 1 -- 172.21.15.16:0/2655231278 --> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f6fec0c7740 con 0x7f6fd806b190 2023-10-16T13:46:55.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:55.285+0000 7f6fd37fe700 1 -- 172.21.15.16:0/2655231278 <== mgr.11745 v2:172.21.15.16:6832/1506142504 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+57 (secure 0 0 0) 0x7f6fec0c7740 con 0x7f6fd806b190 2023-10-16T13:46:55.288 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule removed for path mandatory_fs_argument_test_dir 2023-10-16T13:46:55.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:55.288+0000 7f6ff16a7700 1 -- 172.21.15.16:0/2655231278 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f6fd806b190 msgr2=0x7f6fd806d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:55.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:55.288+0000 7f6ff16a7700 1 --2- 172.21.15.16:0/2655231278 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f6fd806b190 0x7f6fd806d650 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6fe0000f60 tx=0x7f6fe0007ca0 comp rx=0 tx=0).stop 2023-10-16T13:46:55.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:55.288+0000 7f6ff16a7700 1 -- 172.21.15.16:0/2655231278 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fec15d7a0 msgr2=0x7f6fec0c8390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:55.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:55.288+0000 7f6ff16a7700 1 --2- 172.21.15.16:0/2655231278 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fec15d7a0 0x7f6fec0c8390 secure :-1 s=READY pgs=2585 cs=0 l=1 rev1=1 crypto rx=0x7f6fec0d4390 tx=0x7f6fe4004ad0 comp rx=0 tx=0).stop 2023-10-16T13:46:55.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:55.288+0000 7f6ff16a7700 1 -- 172.21.15.16:0/2655231278 shutdown_connections 2023-10-16T13:46:55.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:55.288+0000 7f6ff16a7700 1 --2- 172.21.15.16:0/2655231278 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f6fd806b190 0x7f6fd806d650 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:55.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:55.288+0000 7f6ff16a7700 1 --2- 172.21.15.16:0/2655231278 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fec15d7a0 0x7f6fec0c8390 unknown :-1 s=CLOSED pgs=2585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:55.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:55.288+0000 7f6ff16a7700 1 --2- 172.21.15.16:0/2655231278 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6fec0ce260 0x7f6fec0cc800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:55.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:55.288+0000 7f6ff16a7700 1 --2- 172.21.15.16:0/2655231278 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6fec0d1740 0x7f6fec1518e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:55.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:55.288+0000 7f6ff16a7700 1 -- 172.21.15.16:0/2655231278 >> 172.21.15.16:0/2655231278 conn(0x7f6fec072330 msgr2=0x7f6fec15ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:55.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:55.288+0000 7f6ff16a7700 1 -- 172.21.15.16:0/2655231278 shutdown_connections 2023-10-16T13:46:55.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:55.288+0000 7f6ff16a7700 1 -- 172.21.15.16:0/2655231278 wait complete. 2023-10-16T13:46:55.303 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path mandatory_fs_argument_test_dir 2023-10-16T13:46:55.303 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:46:55.303 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir mandatory_fs_argument_test_dir) 2023-10-16T13:46:55.323 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-16T13:46:55.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:55.487+0000 7f885cb99700 1 Processor -- start 2023-10-16T13:46:55.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:55.487+0000 7f885cb99700 1 -- start start 2023-10-16T13:46:55.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:55.488+0000 7f885cb99700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f885806aef0 0x7f88580ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:55.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:55.488+0000 7f885cb99700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f885806da30 0x7f88580adf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:55.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:55.488+0000 7f885cb99700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88581613c0 con 0x7f8858160fe0 2023-10-16T13:46:55.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:55.488+0000 7f885cb99700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88580cf680 con 0x7f885806aef0 2023-10-16T13:46:55.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:55.488+0000 7f885cb99700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88580adb00 con 0x7f885806da30 2023-10-16T13:46:55.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:55.488+0000 7f8855d9b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f885806aef0 0x7f88580ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:55.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:55.488+0000 7f8855d9b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f885806aef0 0x7f88580ac8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54204/0 (socket says 172.21.15.16:54204) 2023-10-16T13:46:55.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:55.488+0000 7f8855d9b700 1 -- 172.21.15.16:0/3718416365 learned_addr learned my addr 172.21.15.16:0/3718416365 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:55.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:55.488+0000 7f8856d9d700 1 --2- 172.21.15.16:0/3718416365 >> v2:172.21.15.49:3300/0 conn(0x7f885806da30 0x7f88580adf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:55.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:55.489+0000 7f8855d9b700 1 -- 172.21.15.16:0/3718416365 >> v2:172.21.15.49:3300/0 conn(0x7f885806da30 msgr2=0x7f88580adf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:55.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:55.489+0000 7f8855d9b700 1 --2- 172.21.15.16:0/3718416365 >> v2:172.21.15.49:3300/0 conn(0x7f885806da30 0x7f88580adf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:55.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:55.489+0000 7f8855d9b700 1 -- 172.21.15.16:0/3718416365 >> v1:172.21.15.49:6789/0 conn(0x7f8858160fe0 legacy=0x7f885806a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:55.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:55.489+0000 7f8855d9b700 1 -- 172.21.15.16:0/3718416365 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88580acef0 con 0x7f885806aef0 2023-10-16T13:46:55.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:55.489+0000 7f8855d9b700 1 --2- 172.21.15.16:0/3718416365 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f885806aef0 0x7f88580ac8d0 secure :-1 s=READY pgs=2586 cs=0 l=1 rev1=1 crypto rx=0x7f88580b2e40 tx=0x7f8848009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc2576dc8ea9ff6b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:55.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:55.489+0000 7f885559a700 1 -- 172.21.15.16:0/3718416365 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8848004230 con 0x7f885806aef0 2023-10-16T13:46:55.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:55.489+0000 7f885559a700 1 -- 172.21.15.16:0/3718416365 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8848004390 con 0x7f885806aef0 2023-10-16T13:46:55.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:55.489+0000 7f885559a700 1 -- 172.21.15.16:0/3718416365 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88480036a0 con 0x7f885806aef0 2023-10-16T13:46:55.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:55.489+0000 7f885cb99700 1 -- 172.21.15.16:0/3718416365 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f885806aef0 msgr2=0x7f88580ac8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:55.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:55.489+0000 7f885cb99700 1 --2- 172.21.15.16:0/3718416365 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f885806aef0 0x7f88580ac8d0 secure :-1 s=READY pgs=2586 cs=0 l=1 rev1=1 crypto rx=0x7f88580b2e40 tx=0x7f8848009bd0 comp rx=0 tx=0).stop 2023-10-16T13:46:55.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:55.490+0000 7f885cb99700 1 -- 172.21.15.16:0/3718416365 shutdown_connections 2023-10-16T13:46:55.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:55.490+0000 7f885cb99700 1 --2- 172.21.15.16:0/3718416365 >> v2:172.21.15.49:3300/0 conn(0x7f885806da30 0x7f88580adf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:55.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:55.490+0000 7f885cb99700 1 --2- 172.21.15.16:0/3718416365 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f885806aef0 0x7f88580ac8d0 unknown :-1 s=CLOSED pgs=2586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:55.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:55.490+0000 7f885cb99700 1 -- 172.21.15.16:0/3718416365 >> 172.21.15.16:0/3718416365 conn(0x7f88580b1ed0 msgr2=0x7f88580b22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:55.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:55.490+0000 7f885cb99700 1 -- 172.21.15.16:0/3718416365 shutdown_connections 2023-10-16T13:46:55.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:55.490+0000 7f885cb99700 1 -- 172.21.15.16:0/3718416365 wait complete. 2023-10-16T13:46:55.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:55.491+0000 7f885cb99700 1 Processor -- start 2023-10-16T13:46:55.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:55.491+0000 7f885cb99700 1 -- start start 2023-10-16T13:46:55.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:55.492+0000 7f885cb99700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f885806aef0 0x7f88580b2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:55.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:55.492+0000 7f885cb99700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f885806da30 0x7f88580c6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:55.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:55.492+0000 7f885cb99700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8858160fe0 0x7f8858071b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:55.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:55.492+0000 7f885cb99700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88580c5f80 con 0x7f885806aef0 2023-10-16T13:46:55.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:55.492+0000 7f885cb99700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f885815ae20 con 0x7f8858160fe0 2023-10-16T13:46:55.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:55.492+0000 7f885cb99700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88580c5370 con 0x7f885806da30 2023-10-16T13:46:55.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:55.492+0000 7f885659c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f885806aef0 0x7f88580b2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:55.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:55.492+0000 7f885659c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f885806aef0 0x7f88580b2fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54210/0 (socket says 172.21.15.16:54210) 2023-10-16T13:46:55.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:55.492+0000 7f885659c700 1 -- 172.21.15.16:0/1452299744 learned_addr learned my addr 172.21.15.16:0/1452299744 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:55.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:55.492+0000 7f8856d9d700 1 --2- 172.21.15.16:0/1452299744 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8858160fe0 0x7f8858071b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:55.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:55.492+0000 7f8855d9b700 1 --2- 172.21.15.16:0/1452299744 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f885806da30 0x7f88580c6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:55.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:55.492+0000 7f885659c700 1 -- 172.21.15.16:0/1452299744 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f885806da30 msgr2=0x7f88580c6410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:55.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:55.492+0000 7f885659c700 1 --2- 172.21.15.16:0/1452299744 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f885806da30 0x7f88580c6410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:55.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:55.492+0000 7f885659c700 1 -- 172.21.15.16:0/1452299744 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8858160fe0 msgr2=0x7f8858071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:55.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:55.492+0000 7f885659c700 1 --2- 172.21.15.16:0/1452299744 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8858160fe0 0x7f8858071b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:55.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:55.492+0000 7f885659c700 1 -- 172.21.15.16:0/1452299744 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8848009550 con 0x7f885806aef0 2023-10-16T13:46:55.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:55.492+0000 7f8855d9b700 1 --2- 172.21.15.16:0/1452299744 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f885806da30 0x7f88580c6410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:46:55.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:55.492+0000 7f885659c700 1 --2- 172.21.15.16:0/1452299744 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f885806aef0 0x7f88580b2fc0 secure :-1 s=READY pgs=2587 cs=0 l=1 rev1=1 crypto rx=0x7f884c00b450 tx=0x7f884c00b760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:55.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:55.492+0000 7f8846ffd700 1 -- 172.21.15.16:0/1452299744 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f884c004020 con 0x7f885806aef0 2023-10-16T13:46:55.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:55.492+0000 7f8846ffd700 1 -- 172.21.15.16:0/1452299744 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f884c004180 con 0x7f885806aef0 2023-10-16T13:46:55.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:55.493+0000 7f885cb99700 1 -- 172.21.15.16:0/1452299744 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88580c5650 con 0x7f885806aef0 2023-10-16T13:46:55.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:55.493+0000 7f8846ffd700 1 -- 172.21.15.16:0/1452299744 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f884c004530 con 0x7f885806aef0 2023-10-16T13:46:55.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:55.493+0000 7f885cb99700 1 -- 172.21.15.16:0/1452299744 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88580c5c10 con 0x7f885806aef0 2023-10-16T13:46:55.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:55.494+0000 7f885cb99700 1 -- 172.21.15.16:0/1452299744 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f885804ebb0 con 0x7f885806aef0 2023-10-16T13:46:55.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:55.494+0000 7f8846ffd700 1 -- 172.21.15.16:0/1452299744 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 143) v1 ==== 89451+0+0 (secure 0 0 0) 0x7f884c0036d0 con 0x7f885806aef0 2023-10-16T13:46:55.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:55.494+0000 7f8846ffd700 1 --2- 172.21.15.16:0/1452299744 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f883806b190 0x7f883806d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:55.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:55.494+0000 7f8846ffd700 1 -- 172.21.15.16:0/1452299744 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(230..230 src has 1..230) v4 ==== 7365+0+0 (secure 0 0 0) 0x7f884c0898b0 con 0x7f885806aef0 2023-10-16T13:46:55.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:55.494+0000 7f8855d9b700 1 --2- 172.21.15.16:0/1452299744 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f883806b190 0x7f883806d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:55.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:55.495+0000 7f8855d9b700 1 --2- 172.21.15.16:0/1452299744 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f883806b190 0x7f883806d650 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f8848009130 tx=0x7f8848008030 comp rx=0 tx=0).ready entity=mgr.11745 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:55.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:55.499+0000 7f8846ffd700 1 -- 172.21.15.16:0/1452299744 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f884c053f80 con 0x7f885806aef0 2023-10-16T13:46:55.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:55.760+0000 7f885cb99700 1 -- 172.21.15.16:0/1452299744 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f88580cfa40 con 0x7f885806aef0 2023-10-16T13:46:56.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.190+0000 7f8846ffd700 1 -- 172.21.15.16:0/1452299744 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v144) v1 ==== 93+0+0 (secure 0 0 0) 0x7f884c015020 con 0x7f885806aef0 2023-10-16T13:46:56.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.196+0000 7f885cb99700 1 -- 172.21.15.16:0/1452299744 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f883806b190 msgr2=0x7f883806d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:56.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.196+0000 7f885cb99700 1 --2- 172.21.15.16:0/1452299744 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f883806b190 0x7f883806d650 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f8848009130 tx=0x7f8848008030 comp rx=0 tx=0).stop 2023-10-16T13:46:56.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.196+0000 7f885cb99700 1 -- 172.21.15.16:0/1452299744 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f885806aef0 msgr2=0x7f88580b2fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:56.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.197+0000 7f8846ffd700 1 -- 172.21.15.16:0/1452299744 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mgrmap(e 144) v1 ==== 89490+0+0 (secure 0 0 0) 0x7f884c00fdc0 con 0x7f885806aef0 2023-10-16T13:46:56.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.197+0000 7f885cb99700 1 --2- 172.21.15.16:0/1452299744 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f885806aef0 0x7f88580b2fc0 secure :-1 s=READY pgs=2587 cs=0 l=1 rev1=1 crypto rx=0x7f884c00b450 tx=0x7f884c00b760 comp rx=0 tx=0).stop 2023-10-16T13:46:56.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.197+0000 7f885cb99700 1 -- 172.21.15.16:0/1452299744 shutdown_connections 2023-10-16T13:46:56.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.197+0000 7f885cb99700 1 --2- 172.21.15.16:0/1452299744 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f883806b190 0x7f883806d650 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:56.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.197+0000 7f885cb99700 1 --2- 172.21.15.16:0/1452299744 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8858160fe0 0x7f8858071b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:56.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.197+0000 7f885cb99700 1 --2- 172.21.15.16:0/1452299744 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f885806aef0 0x7f88580b2fc0 unknown :-1 s=CLOSED pgs=2587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:56.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.197+0000 7f885cb99700 1 --2- 172.21.15.16:0/1452299744 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f885806da30 0x7f88580c6410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:56.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.197+0000 7f885cb99700 1 -- 172.21.15.16:0/1452299744 >> 172.21.15.16:0/1452299744 conn(0x7f88580b1ed0 msgr2=0x7f885815fbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:56.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.197+0000 7f885cb99700 1 -- 172.21.15.16:0/1452299744 shutdown_connections 2023-10-16T13:46:56.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.197+0000 7f885cb99700 1 -- 172.21.15.16:0/1452299744 wait complete. 2023-10-16T13:46:56.214 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T13:46:56.214 DEBUG:teuthology.orchestra.run.smithi016:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T13:46:56.256 DEBUG:teuthology.orchestra.run.smithi049:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T13:46:56.302 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T13:46:56.302 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T13:46:56.303 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:46:56.320 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T13:46:56.321 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:46:56.346 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:46:56.344+0000 7f0d0a270200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T13:46:56.350 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:56.347+0000 7fb67c01c200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T13:46:56.367 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:46:56.416 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:46:56.413+0000 7f0d0a270200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T13:46:56.419 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:56.417+0000 7fb67c01c200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T13:46:56.490 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:46:56.488+0000 7f0d0a270200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T13:46:56.497 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:56.494+0000 7fb67c01c200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T13:46:56.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.532+0000 7f01b084f700 1 Processor -- start 2023-10-16T13:46:56.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.532+0000 7f01b084f700 1 -- start start 2023-10-16T13:46:56.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.533+0000 7f01b084f700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f01ac15b630 0x7f01ac0d0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:56.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.533+0000 7f01b084f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f01ac15ba10 0x7f01ac0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:56.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.533+0000 7f01b084f700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01ac0b0970 con 0x7f01ac0cdde0 2023-10-16T13:46:56.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.533+0000 7f01b084f700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01ac0cd690 con 0x7f01ac15b630 2023-10-16T13:46:56.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.533+0000 7f01b084f700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01ac0cd970 con 0x7f01ac15ba10 2023-10-16T13:46:56.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.534+0000 7f01a9d9b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f01ac15ba10 0x7f01ac0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:56.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.534+0000 7f01aa59c700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f01ac15b630 0x7f01ac0d0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:56.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.534+0000 7f01a9d9b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f01ac15ba10 0x7f01ac0cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:42548/0 (socket says 172.21.15.16:42548) 2023-10-16T13:46:56.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.534+0000 7f01a9d9b700 1 -- 172.21.15.16:0/2641402447 learned_addr learned my addr 172.21.15.16:0/2641402447 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:56.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.534+0000 7f01aa59c700 1 -- 172.21.15.16:0/2641402447 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f01ac15ba10 msgr2=0x7f01ac0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:56.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.534+0000 7f01aa59c700 1 --2- 172.21.15.16:0/2641402447 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f01ac15ba10 0x7f01ac0cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:56.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.534+0000 7f01aa59c700 1 -- 172.21.15.16:0/2641402447 >> v1:172.21.15.49:6789/0 conn(0x7f01ac0cdde0 legacy=0x7f01ac0cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:56.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.534+0000 7f01aa59c700 1 -- 172.21.15.16:0/2641402447 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01ac0b0c30 con 0x7f01ac15b630 2023-10-16T13:46:56.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.534+0000 7f01a959a700 1 -- 172.21.15.16:0/2641402447 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1689817729 0 0) 0x7f01ac0b0970 con 0x7f01ac0cdde0 2023-10-16T13:46:56.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.534+0000 7f01a9d9b700 1 --2- 172.21.15.16:0/2641402447 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f01ac15ba10 0x7f01ac0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:46:56.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.534+0000 7f01aa59c700 1 --2- 172.21.15.16:0/2641402447 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f01ac15b630 0x7f01ac0d0380 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f019c009840 tx=0x7f019c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f58c54b449eca7af server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:56.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.534+0000 7f01a959a700 1 -- 172.21.15.16:0/2641402447 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f019c011830 con 0x7f01ac15b630 2023-10-16T13:46:56.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.535+0000 7f01a959a700 1 -- 172.21.15.16:0/2641402447 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f019c011990 con 0x7f01ac15b630 2023-10-16T13:46:56.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.535+0000 7f01a959a700 1 -- 172.21.15.16:0/2641402447 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f019c011d20 con 0x7f01ac15b630 2023-10-16T13:46:56.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.535+0000 7f01b084f700 1 -- 172.21.15.16:0/2641402447 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f01ac15b630 msgr2=0x7f01ac0d0380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:56.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.535+0000 7f01b084f700 1 --2- 172.21.15.16:0/2641402447 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f01ac15b630 0x7f01ac0d0380 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f019c009840 tx=0x7f019c009b50 comp rx=0 tx=0).stop 2023-10-16T13:46:56.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.535+0000 7f01b084f700 1 -- 172.21.15.16:0/2641402447 shutdown_connections 2023-10-16T13:46:56.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.535+0000 7f01b084f700 1 --2- 172.21.15.16:0/2641402447 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f01ac15b630 0x7f01ac0d0380 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:56.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.535+0000 7f01b084f700 1 --2- 172.21.15.16:0/2641402447 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f01ac15ba10 0x7f01ac0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:56.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.535+0000 7f01b084f700 1 -- 172.21.15.16:0/2641402447 >> 172.21.15.16:0/2641402447 conn(0x7f01ac0d14b0 msgr2=0x7f01ac0d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:56.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.535+0000 7f01b084f700 1 -- 172.21.15.16:0/2641402447 shutdown_connections 2023-10-16T13:46:56.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.535+0000 7f01b084f700 1 -- 172.21.15.16:0/2641402447 wait complete. 2023-10-16T13:46:56.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.537+0000 7f01b084f700 1 Processor -- start 2023-10-16T13:46:56.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.537+0000 7f01b084f700 1 -- start start 2023-10-16T13:46:56.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.538+0000 7f01b084f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01ac0cdde0 0x7f01ac14f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:56.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.538+0000 7f01b084f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f01ac15b630 0x7f01ac0b4c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:56.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.538+0000 7f01b084f700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f01ac15ba10 0x7f01ac0b6850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:56.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.538+0000 7f01b084f700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01ac0c9290 con 0x7f01ac0cdde0 2023-10-16T13:46:56.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.538+0000 7f01b084f700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01ac0c9570 con 0x7f01ac15ba10 2023-10-16T13:46:56.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.538+0000 7f01b084f700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01ac0c9850 con 0x7f01ac15b630 2023-10-16T13:46:56.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.538+0000 7f01aa59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01ac0cdde0 0x7f01ac14f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:56.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.538+0000 7f01aa59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01ac0cdde0 0x7f01ac14f330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54294/0 (socket says 172.21.15.16:54294) 2023-10-16T13:46:56.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.538+0000 7f01aa59c700 1 -- 172.21.15.16:0/4272983280 learned_addr learned my addr 172.21.15.16:0/4272983280 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:56.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.538+0000 7f01aad9d700 1 --2- 172.21.15.16:0/4272983280 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f01ac15ba10 0x7f01ac0b6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:56.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.538+0000 7f01a9d9b700 1 --2- 172.21.15.16:0/4272983280 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f01ac15b630 0x7f01ac0b4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:56.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.538+0000 7f01aa59c700 1 -- 172.21.15.16:0/4272983280 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f01ac15b630 msgr2=0x7f01ac0b4c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:56.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.538+0000 7f01aa59c700 1 --2- 172.21.15.16:0/4272983280 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f01ac15b630 0x7f01ac0b4c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:56.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.538+0000 7f01aa59c700 1 -- 172.21.15.16:0/4272983280 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f01ac15ba10 msgr2=0x7f01ac0b6850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:56.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.538+0000 7f01aa59c700 1 --2- 172.21.15.16:0/4272983280 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f01ac15ba10 0x7f01ac0b6850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:56.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.538+0000 7f01aa59c700 1 -- 172.21.15.16:0/4272983280 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01a0009550 con 0x7f01ac0cdde0 2023-10-16T13:46:56.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.539+0000 7f01aa59c700 1 --2- 172.21.15.16:0/4272983280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01ac0cdde0 0x7f01ac14f330 secure :-1 s=READY pgs=2589 cs=0 l=1 rev1=1 crypto rx=0x7f019c009fd0 tx=0x7f019c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:56.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.539+0000 7f019affd700 1 -- 172.21.15.16:0/4272983280 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f019c0036a0 con 0x7f01ac0cdde0 2023-10-16T13:46:56.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.539+0000 7f019affd700 1 -- 172.21.15.16:0/4272983280 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f019c01f070 con 0x7f01ac0cdde0 2023-10-16T13:46:56.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.539+0000 7f019affd700 1 -- 172.21.15.16:0/4272983280 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f019c003a60 con 0x7f01ac0cdde0 2023-10-16T13:46:56.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.539+0000 7f01b084f700 1 -- 172.21.15.16:0/4272983280 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f019c0094a0 con 0x7f01ac0cdde0 2023-10-16T13:46:56.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.539+0000 7f01b084f700 1 -- 172.21.15.16:0/4272983280 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01ac0c9e10 con 0x7f01ac0cdde0 2023-10-16T13:46:56.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.540+0000 7f019affd700 1 -- 172.21.15.16:0/4272983280 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 144) v1 ==== 89490+0+0 (secure 0 0 0) 0x7f019c003bc0 con 0x7f01ac0cdde0 2023-10-16T13:46:56.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.540+0000 7f01b084f700 1 -- 172.21.15.16:0/4272983280 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01ac04ebb0 con 0x7f01ac0cdde0 2023-10-16T13:46:56.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.541+0000 7f019affd700 1 --2- 172.21.15.16:0/4272983280 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f018c06b1d0 0x7f018c06d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:56.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.541+0000 7f01a9d9b700 1 -- 172.21.15.16:0/4272983280 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f018c06b1d0 msgr2=0x7f018c06d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1506142504 2023-10-16T13:46:56.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.541+0000 7f019affd700 1 -- 172.21.15.16:0/4272983280 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(230..230 src has 1..230) v4 ==== 7365+0+0 (secure 0 0 0) 0x7f019c021070 con 0x7f01ac0cdde0 2023-10-16T13:46:56.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.541+0000 7f01a9d9b700 1 --2- 172.21.15.16:0/4272983280 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f018c06b1d0 0x7f018c06d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:46:56.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.546+0000 7f019affd700 1 -- 172.21.15.16:0/4272983280 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f019c055fa0 con 0x7f01ac0cdde0 2023-10-16T13:46:56.572 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:46:56.569+0000 7f0d0a270200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T13:46:56.583 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:56.580+0000 7fb67c01c200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T13:46:56.720 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:46:56.718+0000 7f0d0a270200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T13:46:56.733 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:56.730+0000 7fb67c01c200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T13:46:56.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.742+0000 7f01a9d9b700 1 -- 172.21.15.16:0/4272983280 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f018c06b1d0 msgr2=0x7f018c06d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1506142504 2023-10-16T13:46:56.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.742+0000 7f01a9d9b700 1 --2- 172.21.15.16:0/4272983280 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f018c06b1d0 0x7f018c06d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:46:56.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.753+0000 7f01b084f700 1 -- 172.21.15.16:0/4272983280 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f01ac06b5d0 con 0x7f01ac0cdde0 2023-10-16T13:46:56.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.755+0000 7f019affd700 1 -- 172.21.15.16:0/4272983280 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v230) v1 ==== 74+0+21038 (secure 0 0 0) 0x7f019c0595c0 con 0x7f01ac0cdde0 2023-10-16T13:46:56.758 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:46:56.758 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":230,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:46:51.165060+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2023-10-16T13:46:22.489653+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"224","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2023-10-16T13:46:23.729509+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"225","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T13:46:35.800444+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"228","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"backup_fs_data","create_time":"2023-10-16T13:46:37.035323+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"229","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/2321929682":"2023-10-17T13:46:51.165018+0000","172.21.15.16:0/3429506235":"2023-10-17T13:46:51.165018+0000","172.21.15.16:0/3843126404":"2023-10-17T13:46:51.165018+0000","172.21.15.16:0/1439318665":"2023-10-17T13:46:51.165018+0000","172.21.15.16:0/4204021371":"2023-10-17T13:46:51.165018+0000","172.21.15.16:6833/3171422553":"2023-10-17T13:46:51.165018+0000","172.21.15.16:6832/3171422553":"2023-10-17T13:46:51.165018+0000","172.21.15.16:0/1632287797":"2023-10-17T13:46:51.165018+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:46:56.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.757+0000 7f01b084f700 1 -- 172.21.15.16:0/4272983280 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f018c06b1d0 msgr2=0x7f018c06d690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:46:56.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.757+0000 7f01b084f700 1 --2- 172.21.15.16:0/4272983280 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f018c06b1d0 0x7f018c06d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:56.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.757+0000 7f01b084f700 1 -- 172.21.15.16:0/4272983280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01ac0cdde0 msgr2=0x7f01ac14f330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:56.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.757+0000 7f01b084f700 1 --2- 172.21.15.16:0/4272983280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01ac0cdde0 0x7f01ac14f330 secure :-1 s=READY pgs=2589 cs=0 l=1 rev1=1 crypto rx=0x7f019c009fd0 tx=0x7f019c004750 comp rx=0 tx=0).stop 2023-10-16T13:46:56.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.758+0000 7f01b084f700 1 -- 172.21.15.16:0/4272983280 shutdown_connections 2023-10-16T13:46:56.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.758+0000 7f01b084f700 1 --2- 172.21.15.16:0/4272983280 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f018c06b1d0 0x7f018c06d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:56.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.758+0000 7f01b084f700 1 --2- 172.21.15.16:0/4272983280 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f01ac15ba10 0x7f01ac0b6850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:56.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.758+0000 7f01b084f700 1 --2- 172.21.15.16:0/4272983280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01ac0cdde0 0x7f01ac14f330 unknown :-1 s=CLOSED pgs=2589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:56.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.758+0000 7f01b084f700 1 --2- 172.21.15.16:0/4272983280 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f01ac15b630 0x7f01ac0b4c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:56.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.758+0000 7f01b084f700 1 -- 172.21.15.16:0/4272983280 >> 172.21.15.16:0/4272983280 conn(0x7f01ac0d14b0 msgr2=0x7f01ac15a960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:56.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.758+0000 7f01b084f700 1 -- 172.21.15.16:0/4272983280 shutdown_connections 2023-10-16T13:46:56.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.758+0000 7f01b084f700 1 -- 172.21.15.16:0/4272983280 wait complete. 2023-10-16T13:46:56.773 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 27, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T13:46:22.489653+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '224', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 28, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T13:46:23.729509+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '225', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 29, 'pool_name': 'backup_fs_metadata', 'create_time': '2023-10-16T13:46:35.800444+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '228', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 30, 'pool_name': 'backup_fs_data', 'create_time': '2023-10-16T13:46:37.035323+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '229', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T13:46:56.774 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T13:46:56.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.940+0000 7fec5b53e700 1 Processor -- start 2023-10-16T13:46:56.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.940+0000 7fec5b53e700 1 -- start start 2023-10-16T13:46:56.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.941+0000 7fec5b53e700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fec540cbe20 0x7fec540cda60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:56.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.941+0000 7fec5b53e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fec540c6770 0x7fec540cdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:56.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.941+0000 7fec5b53e700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec5415aa20 con 0x7fec5415a640 2023-10-16T13:46:56.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.941+0000 7fec5b53e700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec5406eb40 con 0x7fec540cbe20 2023-10-16T13:46:56.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.941+0000 7fec5b53e700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec5406ee20 con 0x7fec540c6770 2023-10-16T13:46:56.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.941+0000 7fec592da700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fec5415a640 0x7fec540cb710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:43622/0 (socket says 172.21.15.16:43622) 2023-10-16T13:46:56.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.941+0000 7fec592da700 1 -- 172.21.15.16:0/481950280 learned_addr learned my addr 172.21.15.16:0/481950280 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:46:56.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.941+0000 7fec58ad9700 1 --2- 172.21.15.16:0/481950280 >> v2:172.21.15.49:3300/0 conn(0x7fec540cbe20 0x7fec540cda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:56.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.941+0000 7fec59adb700 1 --2- 172.21.15.16:0/481950280 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fec540c6770 0x7fec540cdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:56.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.941+0000 7fec4bfff700 1 -- 172.21.15.16:0/481950280 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1240324900 0 0) 0x7fec44001110 con 0x7fec5415a640 2023-10-16T13:46:56.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.941+0000 7fec4bfff700 1 -- 172.21.15.16:0/481950280 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec400035b0 con 0x7fec5415a640 2023-10-16T13:46:56.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.942+0000 7fec58ad9700 1 -- 172.21.15.16:0/481950280 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fec540c6770 msgr2=0x7fec540cdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:56.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.942+0000 7fec58ad9700 1 --2- 172.21.15.16:0/481950280 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fec540c6770 0x7fec540cdfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:56.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.942+0000 7fec58ad9700 1 -- 172.21.15.16:0/481950280 >> v1:172.21.15.16:6789/0 conn(0x7fec5415a640 legacy=0x7fec540cb710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:56.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.942+0000 7fec58ad9700 1 -- 172.21.15.16:0/481950280 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec5406f100 con 0x7fec540cbe20 2023-10-16T13:46:56.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.942+0000 7fec59adb700 1 --2- 172.21.15.16:0/481950280 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fec540c6770 0x7fec540cdfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:46:56.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.942+0000 7fec58ad9700 1 --2- 172.21.15.16:0/481950280 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fec540cbe20 0x7fec540cda60 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fec50009840 tx=0x7fec50009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=75db1c4cb0d35665 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:56.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.942+0000 7fec4bfff700 1 -- 172.21.15.16:0/481950280 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec50011830 con 0x7fec540cbe20 2023-10-16T13:46:56.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.942+0000 7fec4bfff700 1 -- 172.21.15.16:0/481950280 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec50011990 con 0x7fec540cbe20 2023-10-16T13:46:56.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.942+0000 7fec4bfff700 1 -- 172.21.15.16:0/481950280 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec50011d20 con 0x7fec540cbe20 2023-10-16T13:46:56.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.943+0000 7fec5b53e700 1 -- 172.21.15.16:0/481950280 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fec540cbe20 msgr2=0x7fec540cda60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:56.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.943+0000 7fec5b53e700 1 --2- 172.21.15.16:0/481950280 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fec540cbe20 0x7fec540cda60 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fec50009840 tx=0x7fec50009b50 comp rx=0 tx=0).stop 2023-10-16T13:46:56.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.943+0000 7fec5b53e700 1 -- 172.21.15.16:0/481950280 shutdown_connections 2023-10-16T13:46:56.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.943+0000 7fec5b53e700 1 --2- 172.21.15.16:0/481950280 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fec540c6770 0x7fec540cdfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:56.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.943+0000 7fec5b53e700 1 --2- 172.21.15.16:0/481950280 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fec540cbe20 0x7fec540cda60 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:56.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.943+0000 7fec5b53e700 1 -- 172.21.15.16:0/481950280 >> 172.21.15.16:0/481950280 conn(0x7fec540cc5e0 msgr2=0x7fec540cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:56.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.943+0000 7fec5b53e700 1 -- 172.21.15.16:0/481950280 shutdown_connections 2023-10-16T13:46:56.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.943+0000 7fec5b53e700 1 -- 172.21.15.16:0/481950280 wait complete. 2023-10-16T13:46:56.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.944+0000 7fec5b53e700 1 Processor -- start 2023-10-16T13:46:56.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.944+0000 7fec5b53e700 1 -- start start 2023-10-16T13:46:56.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.945+0000 7fec5b53e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec540c6770 0x7fec54150190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:56.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.945+0000 7fec5b53e700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fec540cbe20 0x7fec541e4150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:56.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.945+0000 7fec5b53e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fec5415a640 0x7fec541e4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:56.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.945+0000 7fec5b53e700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec540ae910 con 0x7fec540c6770 2023-10-16T13:46:56.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.945+0000 7fec5b53e700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec541e4cc0 con 0x7fec540cbe20 2023-10-16T13:46:56.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.945+0000 7fec5b53e700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec541e4fa0 con 0x7fec5415a640 2023-10-16T13:46:56.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.945+0000 7fec592da700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec540c6770 0x7fec54150190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:56.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.945+0000 7fec592da700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec540c6770 0x7fec54150190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54354/0 (socket says 172.21.15.16:54354) 2023-10-16T13:46:56.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.945+0000 7fec592da700 1 -- 172.21.15.16:0/97352429 learned_addr learned my addr 172.21.15.16:0/97352429 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:56.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.945+0000 7fec59adb700 1 --2- 172.21.15.16:0/97352429 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fec5415a640 0x7fec541e4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:56.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.945+0000 7fec58ad9700 1 --2- 172.21.15.16:0/97352429 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fec540cbe20 0x7fec541e4150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:56.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.945+0000 7fec592da700 1 -- 172.21.15.16:0/97352429 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fec5415a640 msgr2=0x7fec541e4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:56.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.945+0000 7fec592da700 1 --2- 172.21.15.16:0/97352429 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fec5415a640 0x7fec541e4690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:56.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.945+0000 7fec592da700 1 -- 172.21.15.16:0/97352429 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fec540cbe20 msgr2=0x7fec541e4150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:56.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.945+0000 7fec592da700 1 --2- 172.21.15.16:0/97352429 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fec540cbe20 0x7fec541e4150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:56.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.945+0000 7fec592da700 1 -- 172.21.15.16:0/97352429 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec4c009a70 con 0x7fec540c6770 2023-10-16T13:46:56.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.945+0000 7fec592da700 1 --2- 172.21.15.16:0/97352429 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec540c6770 0x7fec54150190 secure :-1 s=READY pgs=2591 cs=0 l=1 rev1=1 crypto rx=0x7fec44007fd0 tx=0x7fec4400c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:56.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.946+0000 7fec49ffb700 1 -- 172.21.15.16:0/97352429 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec44005d70 con 0x7fec540c6770 2023-10-16T13:46:56.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.946+0000 7fec49ffb700 1 -- 172.21.15.16:0/97352429 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec440070d0 con 0x7fec540c6770 2023-10-16T13:46:56.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.946+0000 7fec5b53e700 1 -- 172.21.15.16:0/97352429 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec500094a0 con 0x7fec540c6770 2023-10-16T13:46:56.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.946+0000 7fec49ffb700 1 -- 172.21.15.16:0/97352429 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec44007460 con 0x7fec540c6770 2023-10-16T13:46:56.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.946+0000 7fec5b53e700 1 -- 172.21.15.16:0/97352429 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec541e5560 con 0x7fec540c6770 2023-10-16T13:46:56.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.947+0000 7fec5b53e700 1 -- 172.21.15.16:0/97352429 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec5404ebb0 con 0x7fec540c6770 2023-10-16T13:46:56.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.947+0000 7fec49ffb700 1 -- 172.21.15.16:0/97352429 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 144) v1 ==== 89490+0+0 (secure 0 0 0) 0x7fec440075c0 con 0x7fec540c6770 2023-10-16T13:46:56.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.947+0000 7fec49ffb700 1 --2- 172.21.15.16:0/97352429 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7fec3806b1d0 0x7fec3806d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:56.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.947+0000 7fec58ad9700 1 -- 172.21.15.16:0/97352429 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7fec3806b1d0 msgr2=0x7fec3806d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1506142504 2023-10-16T13:46:56.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.948+0000 7fec49ffb700 1 -- 172.21.15.16:0/97352429 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(230..230 src has 1..230) v4 ==== 7365+0+0 (secure 0 0 0) 0x7fec44089260 con 0x7fec540c6770 2023-10-16T13:46:56.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.948+0000 7fec58ad9700 1 --2- 172.21.15.16:0/97352429 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7fec3806b1d0 0x7fec3806d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:46:56.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:56.952+0000 7fec49ffb700 1 -- 172.21.15.16:0/97352429 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fec440539d0 con 0x7fec540c6770 2023-10-16T13:46:57.060 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:46:57.058+0000 7f0d0a270200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T13:46:57.085 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:57.083+0000 7fb67c01c200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T13:46:57.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.149+0000 7fec58ad9700 1 -- 172.21.15.16:0/97352429 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7fec3806b1d0 msgr2=0x7fec3806d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1506142504 2023-10-16T13:46:57.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.149+0000 7fec58ad9700 1 --2- 172.21.15.16:0/97352429 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7fec3806b1d0 0x7fec3806d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:46:57.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.156+0000 7fec5b53e700 1 -- 172.21.15.16:0/97352429 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fec5415aa20 con 0x7fec540c6770 2023-10-16T13:46:57.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.156+0000 7fec49ffb700 1 -- 172.21.15.16:0/97352429 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v230) v1 ==== 93+0+10 (secure 0 0 0) 0x7fec5415aa20 con 0x7fec540c6770 2023-10-16T13:46:57.159 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T13:46:57.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.160+0000 7fec5b53e700 1 -- 172.21.15.16:0/97352429 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7fec3806b1d0 msgr2=0x7fec3806d690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:46:57.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.160+0000 7fec5b53e700 1 --2- 172.21.15.16:0/97352429 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7fec3806b1d0 0x7fec3806d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:57.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.160+0000 7fec5b53e700 1 -- 172.21.15.16:0/97352429 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec540c6770 msgr2=0x7fec54150190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:57.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.160+0000 7fec5b53e700 1 --2- 172.21.15.16:0/97352429 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec540c6770 0x7fec54150190 secure :-1 s=READY pgs=2591 cs=0 l=1 rev1=1 crypto rx=0x7fec44007fd0 tx=0x7fec4400c7b0 comp rx=0 tx=0).stop 2023-10-16T13:46:57.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.160+0000 7fec5b53e700 1 -- 172.21.15.16:0/97352429 shutdown_connections 2023-10-16T13:46:57.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.160+0000 7fec5b53e700 1 --2- 172.21.15.16:0/97352429 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7fec3806b1d0 0x7fec3806d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:57.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.160+0000 7fec5b53e700 1 --2- 172.21.15.16:0/97352429 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fec5415a640 0x7fec541e4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:57.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.160+0000 7fec5b53e700 1 --2- 172.21.15.16:0/97352429 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec540c6770 0x7fec54150190 unknown :-1 s=CLOSED pgs=2591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:57.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.160+0000 7fec5b53e700 1 --2- 172.21.15.16:0/97352429 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fec540cbe20 0x7fec541e4150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:57.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.160+0000 7fec5b53e700 1 -- 172.21.15.16:0/97352429 >> 172.21.15.16:0/97352429 conn(0x7fec540cc5e0 msgr2=0x7fec540cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:57.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.160+0000 7fec5b53e700 1 -- 172.21.15.16:0/97352429 shutdown_connections 2023-10-16T13:46:57.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.160+0000 7fec5b53e700 1 -- 172.21.15.16:0/97352429 wait complete. 2023-10-16T13:46:57.180 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T13:46:57.181 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:46:57.178+0000 7f0d0a270200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T13:46:57.218 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:57.216+0000 7fb67c01c200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T13:46:57.257 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:46:57.254+0000 7f0d0a270200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T13:46:57.297 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:57.295+0000 7fb67c01c200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T13:46:57.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.308+0000 7fe71bd34700 1 Processor -- start 2023-10-16T13:46:57.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.309+0000 7fe71bd34700 1 -- start start 2023-10-16T13:46:57.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.309+0000 7fe71bd34700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fe71415ec80 0x7fe7140cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:57.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.309+0000 7fe71bd34700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7140d2110 0x7fe7140cd1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:57.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.310+0000 7fe71bd34700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7140b2440 con 0x7fe71415f060 2023-10-16T13:46:57.313 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.310+0000 7fe71bd34700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7140b2c50 con 0x7fe71415ec80 2023-10-16T13:46:57.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.310+0000 7fe71bd34700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7140d00e0 con 0x7fe7140d2110 2023-10-16T13:46:57.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.310+0000 7fe71a2d1700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7140d2110 0x7fe7140cd1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:57.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.310+0000 7fe719ad0700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fe71415ec80 0x7fe7140cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:57.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.310+0000 7fe71a2d1700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7140d2110 0x7fe7140cd1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:42626/0 (socket says 172.21.15.16:42626) 2023-10-16T13:46:57.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.310+0000 7fe71a2d1700 1 -- 172.21.15.16:0/3605269999 learned_addr learned my addr 172.21.15.16:0/3605269999 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:57.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.310+0000 7fe719ad0700 1 --2- 172.21.15.16:0/3605269999 >> v2:172.21.15.49:3300/0 conn(0x7fe71415ec80 0x7fe7140cdec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:53426/0 (socket says 172.21.15.16:53426) 2023-10-16T13:46:57.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.310+0000 7fe71a2d1700 1 -- 172.21.15.16:0/3605269999 >> v2:172.21.15.49:3300/0 conn(0x7fe71415ec80 msgr2=0x7fe7140cdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:57.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.310+0000 7fe71a2d1700 1 --2- 172.21.15.16:0/3605269999 >> v2:172.21.15.49:3300/0 conn(0x7fe71415ec80 0x7fe7140cdec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:57.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.310+0000 7fe71a2d1700 1 -- 172.21.15.16:0/3605269999 >> v1:172.21.15.49:6789/0 conn(0x7fe71415f060 legacy=0x7fe7140b4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:57.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.310+0000 7fe71a2d1700 1 -- 172.21.15.16:0/3605269999 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7140cef60 con 0x7fe7140d2110 2023-10-16T13:46:57.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.310+0000 7fe71a2d1700 1 --2- 172.21.15.16:0/3605269999 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7140d2110 0x7fe7140cd1a0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fe70c009e50 tx=0x7fe70c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3497c2e0dc9b452 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:57.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.310+0000 7fe718ace700 1 -- 172.21.15.16:0/3605269999 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe70c00e070 con 0x7fe7140d2110 2023-10-16T13:46:57.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.311+0000 7fe718ace700 1 -- 172.21.15.16:0/3605269999 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe70c00b910 con 0x7fe7140d2110 2023-10-16T13:46:57.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.311+0000 7fe718ace700 1 -- 172.21.15.16:0/3605269999 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe70c00bcc0 con 0x7fe7140d2110 2023-10-16T13:46:57.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.311+0000 7fe71bd34700 1 -- 172.21.15.16:0/3605269999 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7140d2110 msgr2=0x7fe7140cd1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:57.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.311+0000 7fe71bd34700 1 --2- 172.21.15.16:0/3605269999 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7140d2110 0x7fe7140cd1a0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fe70c009e50 tx=0x7fe70c00b330 comp rx=0 tx=0).stop 2023-10-16T13:46:57.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.311+0000 7fe71bd34700 1 -- 172.21.15.16:0/3605269999 shutdown_connections 2023-10-16T13:46:57.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.311+0000 7fe71bd34700 1 --2- 172.21.15.16:0/3605269999 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7140d2110 0x7fe7140cd1a0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:57.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.311+0000 7fe71bd34700 1 --2- 172.21.15.16:0/3605269999 >> v2:172.21.15.49:3300/0 conn(0x7fe71415ec80 0x7fe7140cdec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:57.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.311+0000 7fe71bd34700 1 -- 172.21.15.16:0/3605269999 >> 172.21.15.16:0/3605269999 conn(0x7fe7140d15f0 msgr2=0x7fe7140d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:57.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.311+0000 7fe71bd34700 1 -- 172.21.15.16:0/3605269999 shutdown_connections 2023-10-16T13:46:57.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.311+0000 7fe71bd34700 1 -- 172.21.15.16:0/3605269999 wait complete. 2023-10-16T13:46:57.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.312+0000 7fe71bd34700 1 Processor -- start 2023-10-16T13:46:57.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.313+0000 7fe71bd34700 1 -- start start 2023-10-16T13:46:57.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.313+0000 7fe71bd34700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7140d2110 0x7fe714152770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:57.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.313+0000 7fe71bd34700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe71415ec80 0x7fe7140b4470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:57.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.313+0000 7fe71bd34700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe71415f060 0x7fe7140b49b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:57.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.313+0000 7fe71bd34700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7140c86c0 con 0x7fe71415f060 2023-10-16T13:46:57.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.313+0000 7fe71bd34700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7140c8910 con 0x7fe71415ec80 2023-10-16T13:46:57.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.313+0000 7fe71bd34700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7140c8bf0 con 0x7fe7140d2110 2023-10-16T13:46:57.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.313+0000 7fe71a2d1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe71415f060 0x7fe7140b49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:57.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.313+0000 7fe719ad0700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7140d2110 0x7fe714152770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:57.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.313+0000 7fe71a2d1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe71415f060 0x7fe7140b49b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54374/0 (socket says 172.21.15.16:54374) 2023-10-16T13:46:57.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.313+0000 7fe71a2d1700 1 -- 172.21.15.16:0/1419164984 learned_addr learned my addr 172.21.15.16:0/1419164984 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:57.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.313+0000 7fe7192cf700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe71415ec80 0x7fe7140b4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:57.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.314+0000 7fe719ad0700 1 -- 172.21.15.16:0/1419164984 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe71415ec80 msgr2=0x7fe7140b4470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:57.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.314+0000 7fe719ad0700 1 --2- 172.21.15.16:0/1419164984 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe71415ec80 0x7fe7140b4470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:57.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.314+0000 7fe719ad0700 1 -- 172.21.15.16:0/1419164984 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe71415f060 msgr2=0x7fe7140b49b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:57.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.314+0000 7fe719ad0700 1 --2- 172.21.15.16:0/1419164984 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe71415f060 0x7fe7140b49b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:57.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.314+0000 7fe719ad0700 1 -- 172.21.15.16:0/1419164984 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe70c009b30 con 0x7fe7140d2110 2023-10-16T13:46:57.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.314+0000 7fe71a2d1700 1 --2- 172.21.15.16:0/1419164984 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe71415f060 0x7fe7140b49b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:46:57.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.314+0000 7fe719ad0700 1 --2- 172.21.15.16:0/1419164984 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7140d2110 0x7fe714152770 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe71000e900 tx=0x7fe71000ec10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:57.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.314+0000 7fe70a7fc700 1 -- 172.21.15.16:0/1419164984 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe710013680 con 0x7fe7140d2110 2023-10-16T13:46:57.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.314+0000 7fe70a7fc700 1 -- 172.21.15.16:0/1419164984 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7100137e0 con 0x7fe7140d2110 2023-10-16T13:46:57.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.314+0000 7fe71bd34700 1 -- 172.21.15.16:0/1419164984 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7140c8ed0 con 0x7fe7140d2110 2023-10-16T13:46:57.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.314+0000 7fe70a7fc700 1 -- 172.21.15.16:0/1419164984 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe710013b50 con 0x7fe7140d2110 2023-10-16T13:46:57.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.314+0000 7fe71bd34700 1 -- 172.21.15.16:0/1419164984 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe714152cb0 con 0x7fe7140d2110 2023-10-16T13:46:57.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.315+0000 7fe71bd34700 1 -- 172.21.15.16:0/1419164984 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe714152e70 con 0x7fe7140d2110 2023-10-16T13:46:57.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.316+0000 7fe70a7fc700 1 -- 172.21.15.16:0/1419164984 <== mon.2 v2:172.21.15.49:3301/0 4 ==== mgrmap(e 144) v1 ==== 89490+0+0 (secure 0 0 0) 0x7fe710013cb0 con 0x7fe7140d2110 2023-10-16T13:46:57.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.316+0000 7fe70a7fc700 1 --2- 172.21.15.16:0/1419164984 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7fe6f806b1d0 0x7fe6f806d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:57.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.316+0000 7fe7192cf700 1 -- 172.21.15.16:0/1419164984 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7fe6f806b1d0 msgr2=0x7fe6f806d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1506142504 2023-10-16T13:46:57.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.316+0000 7fe70a7fc700 1 -- 172.21.15.16:0/1419164984 <== mon.2 v2:172.21.15.49:3301/0 5 ==== osd_map(230..230 src has 1..230) v4 ==== 7365+0+0 (secure 0 0 0) 0x7fe71008a290 con 0x7fe7140d2110 2023-10-16T13:46:57.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.316+0000 7fe7192cf700 1 --2- 172.21.15.16:0/1419164984 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7fe6f806b1d0 0x7fe6f806d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:46:57.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.321+0000 7fe70a7fc700 1 -- 172.21.15.16:0/1419164984 <== mon.2 v2:172.21.15.49:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe714152e70 con 0x7fe7140d2110 2023-10-16T13:46:57.341 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:46:57.339+0000 7f0d0a270200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T13:46:57.385 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:57.383+0000 7fb67c01c200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T13:46:57.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.517+0000 7fe7192cf700 1 -- 172.21.15.16:0/1419164984 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7fe6f806b1d0 msgr2=0x7fe6f806d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1506142504 2023-10-16T13:46:57.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.517+0000 7fe7192cf700 1 --2- 172.21.15.16:0/1419164984 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7fe6f806b1d0 0x7fe6f806d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:46:57.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.527+0000 7fe71bd34700 1 -- 172.21.15.16:0/1419164984 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe7140c93f0 con 0x7fe7140d2110 2023-10-16T13:46:57.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.528+0000 7fe70a7fc700 1 -- 172.21.15.16:0/1419164984 <== mon.2 v2:172.21.15.49:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v230) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe7140c93f0 con 0x7fe7140d2110 2023-10-16T13:46:57.530 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-10-16T13:46:57.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.530+0000 7fe71bd34700 1 -- 172.21.15.16:0/1419164984 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7fe6f806b1d0 msgr2=0x7fe6f806d690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:46:57.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.530+0000 7fe71bd34700 1 --2- 172.21.15.16:0/1419164984 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7fe6f806b1d0 0x7fe6f806d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:57.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.530+0000 7fe71bd34700 1 -- 172.21.15.16:0/1419164984 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7140d2110 msgr2=0x7fe714152770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:57.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.530+0000 7fe71bd34700 1 --2- 172.21.15.16:0/1419164984 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7140d2110 0x7fe714152770 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe71000e900 tx=0x7fe71000ec10 comp rx=0 tx=0).stop 2023-10-16T13:46:57.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.530+0000 7fe71bd34700 1 -- 172.21.15.16:0/1419164984 shutdown_connections 2023-10-16T13:46:57.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.530+0000 7fe71bd34700 1 --2- 172.21.15.16:0/1419164984 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7fe6f806b1d0 0x7fe6f806d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:57.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.530+0000 7fe71bd34700 1 --2- 172.21.15.16:0/1419164984 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe71415f060 0x7fe7140b49b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:57.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.530+0000 7fe71bd34700 1 --2- 172.21.15.16:0/1419164984 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7140d2110 0x7fe714152770 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:57.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.530+0000 7fe71bd34700 1 --2- 172.21.15.16:0/1419164984 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe71415ec80 0x7fe7140b4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:57.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.530+0000 7fe71bd34700 1 -- 172.21.15.16:0/1419164984 >> 172.21.15.16:0/1419164984 conn(0x7fe7140d15f0 msgr2=0x7fe7140ce400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:57.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.530+0000 7fe71bd34700 1 -- 172.21.15.16:0/1419164984 shutdown_connections 2023-10-16T13:46:57.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.530+0000 7fe71bd34700 1 -- 172.21.15.16:0/1419164984 wait complete. 2023-10-16T13:46:57.545 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T13:46:57.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.713+0000 7f2586061700 1 Processor -- start 2023-10-16T13:46:57.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.713+0000 7f2586061700 1 -- start start 2023-10-16T13:46:57.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.714+0000 7f2586061700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f25800d4060 0x7f25800d1520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:57.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.714+0000 7f2586061700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25800d4540 con 0x7f25800d1d20 2023-10-16T13:46:57.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.714+0000 7f2586061700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25800ce7e0 con 0x7f258015da10 2023-10-16T13:46:57.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.714+0000 7f2586061700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25800cf9b0 con 0x7f25800d4060 2023-10-16T13:46:57.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.714+0000 7f257effd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f25800d4060 0x7f25800d1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:57.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.714+0000 7f257ffff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f25800d1d20 0x7f258006b860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:43690/0 (socket says 172.21.15.16:43690) 2023-10-16T13:46:57.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.714+0000 7f257effd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f25800d4060 0x7f25800d1520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54414/0 (socket says 172.21.15.16:54414) 2023-10-16T13:46:57.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.714+0000 7f257ffff700 1 -- 172.21.15.16:0/3759464240 learned_addr learned my addr 172.21.15.16:0/3759464240 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:46:57.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.714+0000 7f257effd700 1 -- 172.21.15.16:0/3759464240 >> v1:172.21.15.49:6789/0 conn(0x7f258015da10 legacy=0x7f25800cf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:57.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.714+0000 7f257effd700 1 -- 172.21.15.16:0/3759464240 >> v1:172.21.15.16:6789/0 conn(0x7f25800d1d20 legacy=0x7f258006b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:57.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.714+0000 7f257effd700 1 -- 172.21.15.16:0/3759464240 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25800ceac0 con 0x7f25800d4060 2023-10-16T13:46:57.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.714+0000 7f257effd700 1 --2- 172.21.15.16:0/3759464240 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25800d4060 0x7f25800d1520 secure :-1 s=READY pgs=2593 cs=0 l=1 rev1=1 crypto rx=0x7f2568009840 tx=0x7f2568009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3e8f05f1459d460 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:57.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.715+0000 7f257e7fc700 1 -- 172.21.15.16:0/3759464240 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2568011830 con 0x7f25800d4060 2023-10-16T13:46:57.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.715+0000 7f257e7fc700 1 -- 172.21.15.16:0/3759464240 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2568011990 con 0x7f25800d4060 2023-10-16T13:46:57.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.715+0000 7f257e7fc700 1 -- 172.21.15.16:0/3759464240 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2568011d20 con 0x7f25800d4060 2023-10-16T13:46:57.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.715+0000 7f2586061700 1 -- 172.21.15.16:0/3759464240 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25800d4060 msgr2=0x7f25800d1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:57.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.715+0000 7f2586061700 1 --2- 172.21.15.16:0/3759464240 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25800d4060 0x7f25800d1520 secure :-1 s=READY pgs=2593 cs=0 l=1 rev1=1 crypto rx=0x7f2568009840 tx=0x7f2568009b50 comp rx=0 tx=0).stop 2023-10-16T13:46:57.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.715+0000 7f2586061700 1 -- 172.21.15.16:0/3759464240 shutdown_connections 2023-10-16T13:46:57.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.715+0000 7f2586061700 1 --2- 172.21.15.16:0/3759464240 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25800d4060 0x7f25800d1520 unknown :-1 s=CLOSED pgs=2593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:57.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.715+0000 7f2586061700 1 -- 172.21.15.16:0/3759464240 >> 172.21.15.16:0/3759464240 conn(0x7f258009c9f0 msgr2=0x7f25800d3740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:57.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.715+0000 7f2586061700 1 -- 172.21.15.16:0/3759464240 shutdown_connections 2023-10-16T13:46:57.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.715+0000 7f2586061700 1 -- 172.21.15.16:0/3759464240 wait complete. 2023-10-16T13:46:57.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.717+0000 7f2586061700 1 Processor -- start 2023-10-16T13:46:57.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.717+0000 7f2586061700 1 -- start start 2023-10-16T13:46:57.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.717+0000 7f2586061700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f25800d1d20 0x7f25800cd330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:57.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.717+0000 7f2586061700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25800d4060 0x7f2580157fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:57.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.717+0000 7f2586061700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f258015da10 0x7f25801f2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:57.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.717+0000 7f2586061700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25801f3ff0 con 0x7f25800d4060 2023-10-16T13:46:57.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.718+0000 7f2586061700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25801f3520 con 0x7f25800d1d20 2023-10-16T13:46:57.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.718+0000 7f2586061700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25800b6590 con 0x7f258015da10 2023-10-16T13:46:57.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.718+0000 7f257effd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25800d4060 0x7f2580157fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:57.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.718+0000 7f257effd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25800d4060 0x7f2580157fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54420/0 (socket says 172.21.15.16:54420) 2023-10-16T13:46:57.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.718+0000 7f257effd700 1 -- 172.21.15.16:0/2745862485 learned_addr learned my addr 172.21.15.16:0/2745862485 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:57.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.718+0000 7f257f7fe700 1 --2- 172.21.15.16:0/2745862485 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f25800d1d20 0x7f25800cd330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:57.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.718+0000 7f257ffff700 1 --2- 172.21.15.16:0/2745862485 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f258015da10 0x7f25801f2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:57.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.718+0000 7f257effd700 1 -- 172.21.15.16:0/2745862485 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f258015da10 msgr2=0x7f25801f2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:57.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.718+0000 7f257effd700 1 --2- 172.21.15.16:0/2745862485 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f258015da10 0x7f25801f2fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:57.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.718+0000 7f257effd700 1 -- 172.21.15.16:0/2745862485 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f25800d1d20 msgr2=0x7f25800cd330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:57.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.718+0000 7f257effd700 1 --2- 172.21.15.16:0/2745862485 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f25800d1d20 0x7f25800cd330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:57.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.718+0000 7f257effd700 1 -- 172.21.15.16:0/2745862485 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25680094a0 con 0x7f25800d4060 2023-10-16T13:46:57.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.718+0000 7f257effd700 1 --2- 172.21.15.16:0/2745862485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25800d4060 0x7f2580157fb0 secure :-1 s=READY pgs=2594 cs=0 l=1 rev1=1 crypto rx=0x7f25680090b0 tx=0x7f2568004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:57.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.718+0000 7f2577fff700 1 -- 172.21.15.16:0/2745862485 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25680036a0 con 0x7f25800d4060 2023-10-16T13:46:57.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.718+0000 7f2577fff700 1 -- 172.21.15.16:0/2745862485 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f256801f070 con 0x7f25800d4060 2023-10-16T13:46:57.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.718+0000 7f2577fff700 1 -- 172.21.15.16:0/2745862485 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2568003a20 con 0x7f25800d4060 2023-10-16T13:46:57.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.718+0000 7f2586061700 1 -- 172.21.15.16:0/2745862485 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25800cb0b0 con 0x7f25800d4060 2023-10-16T13:46:57.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.718+0000 7f2586061700 1 -- 172.21.15.16:0/2745862485 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25801f3a30 con 0x7f25800d4060 2023-10-16T13:46:57.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.719+0000 7f2586061700 1 -- 172.21.15.16:0/2745862485 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25800b6360 con 0x7f25800d4060 2023-10-16T13:46:57.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.720+0000 7f2577fff700 1 -- 172.21.15.16:0/2745862485 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 144) v1 ==== 89490+0+0 (secure 0 0 0) 0x7f2568003b80 con 0x7f25800d4060 2023-10-16T13:46:57.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.720+0000 7f2577fff700 1 --2- 172.21.15.16:0/2745862485 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f256006b1d0 0x7f256006d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:57.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.720+0000 7f257f7fe700 1 -- 172.21.15.16:0/2745862485 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f256006b1d0 msgr2=0x7f256006d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1506142504 2023-10-16T13:46:57.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.720+0000 7f2577fff700 1 -- 172.21.15.16:0/2745862485 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(230..230 src has 1..230) v4 ==== 7365+0+0 (secure 0 0 0) 0x7f2568021070 con 0x7f25800d4060 2023-10-16T13:46:57.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.720+0000 7f257f7fe700 1 --2- 172.21.15.16:0/2745862485 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f256006b1d0 0x7f256006d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:46:57.732 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:46:57.728+0000 7f0d0a270200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T13:46:57.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.725+0000 7f2577fff700 1 -- 172.21.15.16:0/2745862485 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f25800b6360 con 0x7f25800d4060 2023-10-16T13:46:57.787 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:57.785+0000 7fb67c01c200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T13:46:57.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.921+0000 7f257f7fe700 1 -- 172.21.15.16:0/2745862485 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f256006b1d0 msgr2=0x7f256006d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1506142504 2023-10-16T13:46:57.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.921+0000 7f257f7fe700 1 --2- 172.21.15.16:0/2745862485 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f256006b1d0 0x7f256006d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:46:57.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.928+0000 7f2586061700 1 -- 172.21.15.16:0/2745862485 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f25800b6360 con 0x7f25800d4060 2023-10-16T13:46:57.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.928+0000 7f2577fff700 1 -- 172.21.15.16:0/2745862485 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v230) v1 ==== 100+0+11 (secure 0 0 0) 0x7f25800b6360 con 0x7f25800d4060 2023-10-16T13:46:57.931 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-10-16T13:46:57.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.930+0000 7f2586061700 1 -- 172.21.15.16:0/2745862485 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f256006b1d0 msgr2=0x7f256006d690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:46:57.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.930+0000 7f2586061700 1 --2- 172.21.15.16:0/2745862485 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f256006b1d0 0x7f256006d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:57.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.930+0000 7f2586061700 1 -- 172.21.15.16:0/2745862485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25800d4060 msgr2=0x7f2580157fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:57.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.930+0000 7f2586061700 1 --2- 172.21.15.16:0/2745862485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25800d4060 0x7f2580157fb0 secure :-1 s=READY pgs=2594 cs=0 l=1 rev1=1 crypto rx=0x7f25680090b0 tx=0x7f2568004750 comp rx=0 tx=0).stop 2023-10-16T13:46:57.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.930+0000 7f2586061700 1 -- 172.21.15.16:0/2745862485 shutdown_connections 2023-10-16T13:46:57.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.930+0000 7f2586061700 1 --2- 172.21.15.16:0/2745862485 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f256006b1d0 0x7f256006d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:57.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.930+0000 7f2586061700 1 --2- 172.21.15.16:0/2745862485 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f258015da10 0x7f25801f2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:57.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.930+0000 7f2586061700 1 --2- 172.21.15.16:0/2745862485 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f25800d1d20 0x7f25800cd330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:57.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.930+0000 7f2586061700 1 --2- 172.21.15.16:0/2745862485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25800d4060 0x7f2580157fb0 unknown :-1 s=CLOSED pgs=2594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:57.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.930+0000 7f2586061700 1 -- 172.21.15.16:0/2745862485 >> 172.21.15.16:0/2745862485 conn(0x7f258009c9f0 msgr2=0x7f2580160100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:57.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.930+0000 7f2586061700 1 -- 172.21.15.16:0/2745862485 shutdown_connections 2023-10-16T13:46:57.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:57.931+0000 7f2586061700 1 -- 172.21.15.16:0/2745862485 wait complete. 2023-10-16T13:46:57.945 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2023-10-16T13:46:58.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.072+0000 7f770cc45700 1 Processor -- start 2023-10-16T13:46:58.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.072+0000 7f770cc45700 1 -- start start 2023-10-16T13:46:58.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.073+0000 7f770cc45700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f770815da10 0x7f77080d1520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:58.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.073+0000 7f770cc45700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f77080d4060 0x7f77080cf2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:58.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.073+0000 7f770cc45700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77080d4540 con 0x7f77080d1d20 2023-10-16T13:46:58.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.073+0000 7f770cc45700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77080ce7e0 con 0x7f770815da10 2023-10-16T13:46:58.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.073+0000 7f770cc45700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77080ceac0 con 0x7f77080d4060 2023-10-16T13:46:58.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.073+0000 7f770659c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f770815da10 0x7f77080d1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:58.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.073+0000 7f770659c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f770815da10 0x7f77080d1520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54428/0 (socket says 172.21.15.16:54428) 2023-10-16T13:46:58.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.073+0000 7f770659c700 1 -- 172.21.15.16:0/3065462396 learned_addr learned my addr 172.21.15.16:0/3065462396 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:58.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.073+0000 7f7705d9b700 1 --2- 172.21.15.16:0/3065462396 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f77080d4060 0x7f77080cf2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:58.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.074+0000 7f770659c700 1 -- 172.21.15.16:0/3065462396 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f77080d4060 msgr2=0x7f77080cf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:58.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.074+0000 7f770659c700 1 --2- 172.21.15.16:0/3065462396 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f77080d4060 0x7f77080cf2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:58.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.074+0000 7f770659c700 1 -- 172.21.15.16:0/3065462396 >> v1:172.21.15.16:6789/0 conn(0x7f77080d1d20 legacy=0x7f770806b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:58.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.074+0000 7f770659c700 1 -- 172.21.15.16:0/3065462396 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77080ceda0 con 0x7f770815da10 2023-10-16T13:46:58.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.074+0000 7f770659c700 1 --2- 172.21.15.16:0/3065462396 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f770815da10 0x7f77080d1520 secure :-1 s=READY pgs=2596 cs=0 l=1 rev1=1 crypto rx=0x7f76f8006fd0 tx=0x7f76f8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84e136842801a200 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:58.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.074+0000 7f770559a700 1 -- 172.21.15.16:0/3065462396 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76f8005e30 con 0x7f770815da10 2023-10-16T13:46:58.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.074+0000 7f770559a700 1 -- 172.21.15.16:0/3065462396 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76f8004030 con 0x7f770815da10 2023-10-16T13:46:58.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.074+0000 7f770559a700 1 -- 172.21.15.16:0/3065462396 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76f80043e0 con 0x7f770815da10 2023-10-16T13:46:58.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.074+0000 7f770cc45700 1 -- 172.21.15.16:0/3065462396 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f770815da10 msgr2=0x7f77080d1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:58.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.074+0000 7f770cc45700 1 --2- 172.21.15.16:0/3065462396 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f770815da10 0x7f77080d1520 secure :-1 s=READY pgs=2596 cs=0 l=1 rev1=1 crypto rx=0x7f76f8006fd0 tx=0x7f76f8009bd0 comp rx=0 tx=0).stop 2023-10-16T13:46:58.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.075+0000 7f770cc45700 1 -- 172.21.15.16:0/3065462396 shutdown_connections 2023-10-16T13:46:58.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.075+0000 7f770cc45700 1 --2- 172.21.15.16:0/3065462396 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f770815da10 0x7f77080d1520 unknown :-1 s=CLOSED pgs=2596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:58.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.075+0000 7f770cc45700 1 --2- 172.21.15.16:0/3065462396 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f77080d4060 0x7f77080cf2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:58.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.075+0000 7f770cc45700 1 -- 172.21.15.16:0/3065462396 >> 172.21.15.16:0/3065462396 conn(0x7f770809c9f0 msgr2=0x7f77080d3740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:58.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.075+0000 7f770cc45700 1 -- 172.21.15.16:0/3065462396 shutdown_connections 2023-10-16T13:46:58.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.075+0000 7f770cc45700 1 -- 172.21.15.16:0/3065462396 wait complete. 2023-10-16T13:46:58.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.076+0000 7f770cc45700 1 Processor -- start 2023-10-16T13:46:58.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.076+0000 7f770cc45700 1 -- start start 2023-10-16T13:46:58.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.077+0000 7f770cc45700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f77080d1d20 0x7f77080a0d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:58.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.077+0000 7f770cc45700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77080d4060 0x7f770815e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:58.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.077+0000 7f770cc45700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f770815da10 0x7f77080ccd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:58.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.077+0000 7f770cc45700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77080a12b0 con 0x7f77080d4060 2023-10-16T13:46:58.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.077+0000 7f770cc45700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f770815f920 con 0x7f77080d1d20 2023-10-16T13:46:58.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.077+0000 7f770cc45700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f770815f360 con 0x7f770815da10 2023-10-16T13:46:58.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.077+0000 7f7705d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77080d4060 0x7f770815e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:58.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.077+0000 7f7705d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77080d4060 0x7f770815e980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54436/0 (socket says 172.21.15.16:54436) 2023-10-16T13:46:58.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.077+0000 7f7705d9b700 1 -- 172.21.15.16:0/401676575 learned_addr learned my addr 172.21.15.16:0/401676575 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:58.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.077+0000 7f770659c700 1 --2- 172.21.15.16:0/401676575 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f77080d1d20 0x7f77080a0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:58.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.077+0000 7f7706d9d700 1 --2- 172.21.15.16:0/401676575 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f770815da10 0x7f77080ccd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:58.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.077+0000 7f7705d9b700 1 -- 172.21.15.16:0/401676575 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f770815da10 msgr2=0x7f77080ccd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:58.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.077+0000 7f7705d9b700 1 --2- 172.21.15.16:0/401676575 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f770815da10 0x7f77080ccd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:58.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.077+0000 7f7705d9b700 1 -- 172.21.15.16:0/401676575 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f77080d1d20 msgr2=0x7f77080a0d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:58.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.077+0000 7f7705d9b700 1 --2- 172.21.15.16:0/401676575 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f77080d1d20 0x7f77080a0d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:58.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.077+0000 7f7705d9b700 1 -- 172.21.15.16:0/401676575 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76f8009500 con 0x7f77080d4060 2023-10-16T13:46:58.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.077+0000 7f7705d9b700 1 --2- 172.21.15.16:0/401676575 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77080d4060 0x7f770815e980 secure :-1 s=READY pgs=2597 cs=0 l=1 rev1=1 crypto rx=0x7f76fc00e980 tx=0x7f76fc00ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:58.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.078+0000 7f76f6ffd700 1 -- 172.21.15.16:0/401676575 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76fc009710 con 0x7f77080d4060 2023-10-16T13:46:58.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.078+0000 7f76f6ffd700 1 -- 172.21.15.16:0/401676575 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76fc00c690 con 0x7f77080d4060 2023-10-16T13:46:58.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.078+0000 7f76f6ffd700 1 -- 172.21.15.16:0/401676575 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76fc00ca40 con 0x7f77080d4060 2023-10-16T13:46:58.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.078+0000 7f770cc45700 1 -- 172.21.15.16:0/401676575 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f770815f640 con 0x7f77080d4060 2023-10-16T13:46:58.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.078+0000 7f770cc45700 1 -- 172.21.15.16:0/401676575 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77080cac90 con 0x7f77080d4060 2023-10-16T13:46:58.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.079+0000 7f770cc45700 1 -- 172.21.15.16:0/401676575 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77080ca7f0 con 0x7f77080d4060 2023-10-16T13:46:58.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.079+0000 7f76f6ffd700 1 -- 172.21.15.16:0/401676575 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 144) v1 ==== 89490+0+0 (secure 0 0 0) 0x7f76fc00cba0 con 0x7f77080d4060 2023-10-16T13:46:58.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.080+0000 7f76f6ffd700 1 --2- 172.21.15.16:0/401676575 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f76e806b1d0 0x7f76e806d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:58.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.080+0000 7f770659c700 1 -- 172.21.15.16:0/401676575 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f76e806b1d0 msgr2=0x7f76e806d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1506142504 2023-10-16T13:46:58.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.080+0000 7f76f6ffd700 1 -- 172.21.15.16:0/401676575 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(230..230 src has 1..230) v4 ==== 7365+0+0 (secure 0 0 0) 0x7f76fc08a0a0 con 0x7f77080d4060 2023-10-16T13:46:58.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.080+0000 7f770659c700 1 --2- 172.21.15.16:0/401676575 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f76e806b1d0 0x7f76e806d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:46:58.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.084+0000 7f76f6ffd700 1 -- 172.21.15.16:0/401676575 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f76fc054780 con 0x7f77080d4060 2023-10-16T13:46:58.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.281+0000 7f770659c700 1 -- 172.21.15.16:0/401676575 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f76e806b1d0 msgr2=0x7f76e806d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1506142504 2023-10-16T13:46:58.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.281+0000 7f770659c700 1 --2- 172.21.15.16:0/401676575 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f76e806b1d0 0x7f76e806d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:46:58.337 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:46:58.334+0000 7f0d0a270200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T13:46:58.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.344+0000 7f770cc45700 1 -- 172.21.15.16:0/401676575 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f77080b2d00 con 0x7f77080d4060 2023-10-16T13:46:58.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.345+0000 7f76f6ffd700 1 -- 172.21.15.16:0/401676575 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v230) v1 ==== 107+0+11 (secure 0 0 0) 0x7f76fc057da0 con 0x7f77080d4060 2023-10-16T13:46:58.347 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-10-16T13:46:58.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.347+0000 7f770cc45700 1 -- 172.21.15.16:0/401676575 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f76e806b1d0 msgr2=0x7f76e806d690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:46:58.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.347+0000 7f770cc45700 1 --2- 172.21.15.16:0/401676575 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f76e806b1d0 0x7f76e806d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:58.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.347+0000 7f770cc45700 1 -- 172.21.15.16:0/401676575 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77080d4060 msgr2=0x7f770815e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:58.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.347+0000 7f770cc45700 1 --2- 172.21.15.16:0/401676575 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77080d4060 0x7f770815e980 secure :-1 s=READY pgs=2597 cs=0 l=1 rev1=1 crypto rx=0x7f76fc00e980 tx=0x7f76fc00ec90 comp rx=0 tx=0).stop 2023-10-16T13:46:58.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.347+0000 7f770cc45700 1 -- 172.21.15.16:0/401676575 shutdown_connections 2023-10-16T13:46:58.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.347+0000 7f770cc45700 1 --2- 172.21.15.16:0/401676575 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f76e806b1d0 0x7f76e806d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:58.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.347+0000 7f770cc45700 1 --2- 172.21.15.16:0/401676575 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f770815da10 0x7f77080ccd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:58.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.347+0000 7f770cc45700 1 --2- 172.21.15.16:0/401676575 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f77080d1d20 0x7f77080a0d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:58.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.347+0000 7f770cc45700 1 --2- 172.21.15.16:0/401676575 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77080d4060 0x7f770815e980 unknown :-1 s=CLOSED pgs=2597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:58.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.347+0000 7f770cc45700 1 -- 172.21.15.16:0/401676575 >> 172.21.15.16:0/401676575 conn(0x7f770809c9f0 msgr2=0x7f77080b3180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:58.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.347+0000 7f770cc45700 1 -- 172.21.15.16:0/401676575 shutdown_connections 2023-10-16T13:46:58.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.347+0000 7f770cc45700 1 -- 172.21.15.16:0/401676575 wait complete. 2023-10-16T13:46:58.363 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2023-10-16T13:46:58.409 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:46:58.407+0000 7f0d0a270200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T13:46:58.414 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:58.412+0000 7fb67c01c200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T13:46:58.490 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:58.488+0000 7fb67c01c200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T13:46:58.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.492+0000 7f17cfcf1700 1 Processor -- start 2023-10-16T13:46:58.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.493+0000 7f17cfcf1700 1 -- start start 2023-10-16T13:46:58.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.494+0000 7f17cfcf1700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f17c80c6770 0x7f17c80ce170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:58.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.494+0000 7f17cfcf1700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17c815aa20 con 0x7f17c80cbe20 2023-10-16T13:46:58.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.494+0000 7f17cfcf1700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17c806eb40 con 0x7f17c815a640 2023-10-16T13:46:58.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.494+0000 7f17cfcf1700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17c806ee20 con 0x7f17c80c6770 2023-10-16T13:46:58.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.494+0000 7f17cd28c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f17c80cbe20 0x7f17c80cda60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:43720/0 (socket says 172.21.15.16:43720) 2023-10-16T13:46:58.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.494+0000 7f17cd28c700 1 -- 172.21.15.16:0/2052278742 learned_addr learned my addr 172.21.15.16:0/2052278742 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:46:58.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.494+0000 7f17ce28e700 1 --2- 172.21.15.16:0/2052278742 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f17c80c6770 0x7f17c80ce170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:58.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.494+0000 7f17cca8b700 1 -- 172.21.15.16:0/2052278742 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 632498788 0 0) 0x7f17bc001110 con 0x7f17c80cbe20 2023-10-16T13:46:58.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.494+0000 7f17cca8b700 1 -- 172.21.15.16:0/2052278742 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17b40035b0 con 0x7f17c80cbe20 2023-10-16T13:46:58.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.494+0000 7f17cca8b700 1 -- 172.21.15.16:0/2052278742 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4198483050 0 0) 0x7f17c806eb40 con 0x7f17c815a640 2023-10-16T13:46:58.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.494+0000 7f17cca8b700 1 -- 172.21.15.16:0/2052278742 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17bc001110 con 0x7f17c815a640 2023-10-16T13:46:58.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.494+0000 7f17ce28e700 1 -- 172.21.15.16:0/2052278742 >> v1:172.21.15.49:6789/0 conn(0x7f17c815a640 legacy=0x7f17c80cb710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:58.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.494+0000 7f17ce28e700 1 -- 172.21.15.16:0/2052278742 >> v1:172.21.15.16:6789/0 conn(0x7f17c80cbe20 legacy=0x7f17c80cda60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:58.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.494+0000 7f17ce28e700 1 -- 172.21.15.16:0/2052278742 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17c806f100 con 0x7f17c80c6770 2023-10-16T13:46:58.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.494+0000 7f17ce28e700 1 --2- 172.21.15.16:0/2052278742 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f17c80c6770 0x7f17c80ce170 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f17c0009dc0 tx=0x7f17c000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7965e6e5bc9e3541 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:58.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.495+0000 7f17cca8b700 1 -- 172.21.15.16:0/2052278742 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c000e070 con 0x7f17c80c6770 2023-10-16T13:46:58.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.495+0000 7f17cca8b700 1 -- 172.21.15.16:0/2052278742 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17c000bb20 con 0x7f17c80c6770 2023-10-16T13:46:58.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.495+0000 7f17cca8b700 1 -- 172.21.15.16:0/2052278742 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c000be90 con 0x7f17c80c6770 2023-10-16T13:46:58.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.495+0000 7f17cfcf1700 1 -- 172.21.15.16:0/2052278742 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f17c80c6770 msgr2=0x7f17c80ce170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:58.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.495+0000 7f17cfcf1700 1 --2- 172.21.15.16:0/2052278742 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f17c80c6770 0x7f17c80ce170 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f17c0009dc0 tx=0x7f17c000b330 comp rx=0 tx=0).stop 2023-10-16T13:46:58.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.495+0000 7f17cfcf1700 1 -- 172.21.15.16:0/2052278742 shutdown_connections 2023-10-16T13:46:58.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.495+0000 7f17cfcf1700 1 --2- 172.21.15.16:0/2052278742 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f17c80c6770 0x7f17c80ce170 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:58.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.495+0000 7f17cfcf1700 1 -- 172.21.15.16:0/2052278742 >> 172.21.15.16:0/2052278742 conn(0x7f17c80cc5e0 msgr2=0x7f17c80cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:58.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.495+0000 7f17cfcf1700 1 -- 172.21.15.16:0/2052278742 shutdown_connections 2023-10-16T13:46:58.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.495+0000 7f17cfcf1700 1 -- 172.21.15.16:0/2052278742 wait complete. 2023-10-16T13:46:58.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.496+0000 7f17cfcf1700 1 Processor -- start 2023-10-16T13:46:58.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.497+0000 7f17cfcf1700 1 -- start start 2023-10-16T13:46:58.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.497+0000 7f17cfcf1700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f17c80c6770 0x7f17c81ef780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:58.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.497+0000 7f17cfcf1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17c80cbe20 0x7f17c81eeff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:58.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.497+0000 7f17cfcf1700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f17c815a640 0x7f17c80cda60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:58.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.497+0000 7f17cfcf1700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c80c8c50 con 0x7f17c80cbe20 2023-10-16T13:46:58.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.497+0000 7f17cfcf1700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c8154b20 con 0x7f17c80c6770 2023-10-16T13:46:58.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.497+0000 7f17cfcf1700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17c81ef530 con 0x7f17c815a640 2023-10-16T13:46:58.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.497+0000 7f17cd28c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17c80cbe20 0x7f17c81eeff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:58.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.497+0000 7f17cd28c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17c80cbe20 0x7f17c81eeff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54452/0 (socket says 172.21.15.16:54452) 2023-10-16T13:46:58.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.497+0000 7f17cd28c700 1 -- 172.21.15.16:0/1179603193 learned_addr learned my addr 172.21.15.16:0/1179603193 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:58.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.497+0000 7f17cda8d700 1 --2- 172.21.15.16:0/1179603193 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f17c80c6770 0x7f17c81ef780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:58.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.497+0000 7f17ce28e700 1 --2- 172.21.15.16:0/1179603193 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f17c815a640 0x7f17c80cda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:58.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.498+0000 7f17cd28c700 1 -- 172.21.15.16:0/1179603193 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f17c815a640 msgr2=0x7f17c80cda60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:58.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.498+0000 7f17cd28c700 1 --2- 172.21.15.16:0/1179603193 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f17c815a640 0x7f17c80cda60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:58.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.498+0000 7f17cd28c700 1 -- 172.21.15.16:0/1179603193 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f17c80c6770 msgr2=0x7f17c81ef780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:58.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.498+0000 7f17cd28c700 1 --2- 172.21.15.16:0/1179603193 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f17c80c6770 0x7f17c81ef780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:58.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.498+0000 7f17cd28c700 1 -- 172.21.15.16:0/1179603193 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17c0009a70 con 0x7f17c80cbe20 2023-10-16T13:46:58.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.498+0000 7f17ce28e700 1 --2- 172.21.15.16:0/1179603193 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f17c815a640 0x7f17c80cda60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:46:58.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.498+0000 7f17cd28c700 1 --2- 172.21.15.16:0/1179603193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17c80cbe20 0x7f17c81eeff0 secure :-1 s=READY pgs=2599 cs=0 l=1 rev1=1 crypto rx=0x7f17bc001eb0 tx=0x7f17bc00c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:58.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.498+0000 7f17ba7fc700 1 -- 172.21.15.16:0/1179603193 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17bc005d70 con 0x7f17c80cbe20 2023-10-16T13:46:58.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.498+0000 7f17ba7fc700 1 -- 172.21.15.16:0/1179603193 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17bc0070d0 con 0x7f17c80cbe20 2023-10-16T13:46:58.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.498+0000 7f17cfcf1700 1 -- 172.21.15.16:0/1179603193 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17c80c80d0 con 0x7f17c80cbe20 2023-10-16T13:46:58.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.498+0000 7f17ba7fc700 1 -- 172.21.15.16:0/1179603193 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17bc007480 con 0x7f17c80cbe20 2023-10-16T13:46:58.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.498+0000 7f17cfcf1700 1 -- 172.21.15.16:0/1179603193 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17c80c8690 con 0x7f17c80cbe20 2023-10-16T13:46:58.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.499+0000 7f17cfcf1700 1 -- 172.21.15.16:0/1179603193 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17c804ebb0 con 0x7f17c80cbe20 2023-10-16T13:46:58.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.500+0000 7f17ba7fc700 1 -- 172.21.15.16:0/1179603193 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 144) v1 ==== 89490+0+0 (secure 0 0 0) 0x7f17bc0075e0 con 0x7f17c80cbe20 2023-10-16T13:46:58.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.500+0000 7f17ba7fc700 1 --2- 172.21.15.16:0/1179603193 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f17ac06b1d0 0x7f17ac06d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:58.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.500+0000 7f17cda8d700 1 -- 172.21.15.16:0/1179603193 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f17ac06b1d0 msgr2=0x7f17ac06d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1506142504 2023-10-16T13:46:58.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.500+0000 7f17ba7fc700 1 -- 172.21.15.16:0/1179603193 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(230..230 src has 1..230) v4 ==== 7365+0+0 (secure 0 0 0) 0x7f17bc089470 con 0x7f17c80cbe20 2023-10-16T13:46:58.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.500+0000 7f17cda8d700 1 --2- 172.21.15.16:0/1179603193 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f17ac06b1d0 0x7f17ac06d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:46:58.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.505+0000 7f17ba7fc700 1 -- 172.21.15.16:0/1179603193 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f17bc053be0 con 0x7f17c80cbe20 2023-10-16T13:46:58.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.701+0000 7f17cda8d700 1 -- 172.21.15.16:0/1179603193 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f17ac06b1d0 msgr2=0x7f17ac06d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1506142504 2023-10-16T13:46:58.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.701+0000 7f17cda8d700 1 --2- 172.21.15.16:0/1179603193 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f17ac06b1d0 0x7f17ac06d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:46:58.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.706+0000 7f17cfcf1700 1 -- 172.21.15.16:0/1179603193 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) v1 -- 0x7f17c815aa20 con 0x7f17c80cbe20 2023-10-16T13:46:58.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.707+0000 7f17ba7fc700 1 -- 172.21.15.16:0/1179603193 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v230) v1 ==== 103+0+11 (secure 0 0 0) 0x7f17c815aa20 con 0x7f17c80cbe20 2023-10-16T13:46:58.710 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-10-16T13:46:58.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.709+0000 7f17cfcf1700 1 -- 172.21.15.16:0/1179603193 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f17ac06b1d0 msgr2=0x7f17ac06d690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:46:58.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.709+0000 7f17cfcf1700 1 --2- 172.21.15.16:0/1179603193 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f17ac06b1d0 0x7f17ac06d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:58.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.709+0000 7f17cfcf1700 1 -- 172.21.15.16:0/1179603193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17c80cbe20 msgr2=0x7f17c81eeff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:58.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.709+0000 7f17cfcf1700 1 --2- 172.21.15.16:0/1179603193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17c80cbe20 0x7f17c81eeff0 secure :-1 s=READY pgs=2599 cs=0 l=1 rev1=1 crypto rx=0x7f17bc001eb0 tx=0x7f17bc00c9e0 comp rx=0 tx=0).stop 2023-10-16T13:46:58.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.709+0000 7f17cfcf1700 1 -- 172.21.15.16:0/1179603193 shutdown_connections 2023-10-16T13:46:58.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.709+0000 7f17cfcf1700 1 --2- 172.21.15.16:0/1179603193 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f17ac06b1d0 0x7f17ac06d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:58.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.709+0000 7f17cfcf1700 1 --2- 172.21.15.16:0/1179603193 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f17c815a640 0x7f17c80cda60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:58.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.709+0000 7f17cfcf1700 1 --2- 172.21.15.16:0/1179603193 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f17c80c6770 0x7f17c81ef780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:58.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.709+0000 7f17cfcf1700 1 --2- 172.21.15.16:0/1179603193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17c80cbe20 0x7f17c81eeff0 unknown :-1 s=CLOSED pgs=2599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:58.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.709+0000 7f17cfcf1700 1 -- 172.21.15.16:0/1179603193 >> 172.21.15.16:0/1179603193 conn(0x7f17c80cc5e0 msgr2=0x7f17c80af680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:58.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.709+0000 7f17cfcf1700 1 -- 172.21.15.16:0/1179603193 shutdown_connections 2023-10-16T13:46:58.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.709+0000 7f17cfcf1700 1 -- 172.21.15.16:0/1179603193 wait complete. 2023-10-16T13:46:58.724 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T13:46:58.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.848+0000 7fefa0d73700 1 Processor -- start 2023-10-16T13:46:58.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.849+0000 7fefa0d73700 1 -- start start 2023-10-16T13:46:58.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.849+0000 7fefa0d73700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fef9c15d900 0x7fef9c0d2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:58.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.849+0000 7fefa0d73700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fef9c0ce9f0 0x7fef9c0cdc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:58.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.849+0000 7fefa0d73700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fef9c0d2860 0x7fef9c0d2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:58.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.849+0000 7fefa0d73700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef9c0d3230 con 0x7fef9c0d2860 2023-10-16T13:46:58.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.849+0000 7fefa0d73700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef9c0ce1a0 con 0x7fef9c0ce9f0 2023-10-16T13:46:58.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.849+0000 7fefa0d73700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef9c0cee70 con 0x7fef9c15d900 2023-10-16T13:46:58.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.850+0000 7fef9ad9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fef9c0d2860 0x7fef9c0d2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:58.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.850+0000 7fef9ad9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fef9c0d2860 0x7fef9c0d2cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54464/0 (socket says 172.21.15.16:54464) 2023-10-16T13:46:58.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.850+0000 7fef9a59c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fef9c15d900 0x7fef9c0d2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:58.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.850+0000 7fef99d9b700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fef9c0ce9f0 0x7fef9c0cdc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:58.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.850+0000 7fef9ad9d700 1 -- 172.21.15.16:0/3788231672 learned_addr learned my addr 172.21.15.16:0/3788231672 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:58.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.850+0000 7fef9a59c700 1 -- 172.21.15.16:0/3788231672 >> v2:172.21.15.49:3300/0 conn(0x7fef9c0ce9f0 msgr2=0x7fef9c0cdc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:58.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.850+0000 7fef9a59c700 1 --2- 172.21.15.16:0/3788231672 >> v2:172.21.15.49:3300/0 conn(0x7fef9c0ce9f0 0x7fef9c0cdc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:58.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.850+0000 7fef9a59c700 1 -- 172.21.15.16:0/3788231672 >> v2:172.21.15.16:3300/0 conn(0x7fef9c0d2860 msgr2=0x7fef9c0d2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:58.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.850+0000 7fef9a59c700 1 --2- 172.21.15.16:0/3788231672 >> v2:172.21.15.16:3300/0 conn(0x7fef9c0d2860 0x7fef9c0d2cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:58.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.850+0000 7fef9a59c700 1 -- 172.21.15.16:0/3788231672 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef9c0ce400 con 0x7fef9c15d900 2023-10-16T13:46:58.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.850+0000 7fef9ad9d700 1 --2- 172.21.15.16:0/3788231672 >> v2:172.21.15.16:3300/0 conn(0x7fef9c0d2860 0x7fef9c0d2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:46:58.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.850+0000 7fef9a59c700 1 --2- 172.21.15.16:0/3788231672 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fef9c15d900 0x7fef9c0d2040 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fef9c0b6220 tx=0x7fef8c009b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d5296e1bd16a9a7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:58.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.850+0000 7fef9959a700 1 -- 172.21.15.16:0/3788231672 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef8c011830 con 0x7fef9c15d900 2023-10-16T13:46:58.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.851+0000 7fef9959a700 1 -- 172.21.15.16:0/3788231672 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef8c011990 con 0x7fef9c15d900 2023-10-16T13:46:58.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.851+0000 7fef9959a700 1 -- 172.21.15.16:0/3788231672 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef8c011d20 con 0x7fef9c15d900 2023-10-16T13:46:58.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.851+0000 7fefa0d73700 1 -- 172.21.15.16:0/3788231672 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fef9c15d900 msgr2=0x7fef9c0d2040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:58.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.851+0000 7fefa0d73700 1 --2- 172.21.15.16:0/3788231672 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fef9c15d900 0x7fef9c0d2040 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fef9c0b6220 tx=0x7fef8c009b20 comp rx=0 tx=0).stop 2023-10-16T13:46:58.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.851+0000 7fefa0d73700 1 -- 172.21.15.16:0/3788231672 shutdown_connections 2023-10-16T13:46:58.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.851+0000 7fefa0d73700 1 --2- 172.21.15.16:0/3788231672 >> v2:172.21.15.16:3300/0 conn(0x7fef9c0d2860 0x7fef9c0d2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:58.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.851+0000 7fefa0d73700 1 --2- 172.21.15.16:0/3788231672 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fef9c15d900 0x7fef9c0d2040 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:58.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.851+0000 7fefa0d73700 1 --2- 172.21.15.16:0/3788231672 >> v2:172.21.15.49:3300/0 conn(0x7fef9c0ce9f0 0x7fef9c0cdc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:58.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.851+0000 7fefa0d73700 1 -- 172.21.15.16:0/3788231672 >> 172.21.15.16:0/3788231672 conn(0x7fef9c072330 msgr2=0x7fef9c072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:58.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.851+0000 7fefa0d73700 1 -- 172.21.15.16:0/3788231672 shutdown_connections 2023-10-16T13:46:58.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.851+0000 7fefa0d73700 1 -- 172.21.15.16:0/3788231672 wait complete. 2023-10-16T13:46:58.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.853+0000 7fefa0d73700 1 Processor -- start 2023-10-16T13:46:58.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.853+0000 7fefa0d73700 1 -- start start 2023-10-16T13:46:58.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.853+0000 7fefa0d73700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fef9c0ce9f0 0x7fef9c14f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:58.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.853+0000 7fefa0d73700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fef9c0d2860 0x7fef9c0c8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:58.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.853+0000 7fefa0d73700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef9c15d900 0x7fef9c152f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:58.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.853+0000 7fefa0d73700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef9c0c7590 con 0x7fef9c15d900 2023-10-16T13:46:58.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.853+0000 7fefa0d73700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef9c0c7870 con 0x7fef9c0ce9f0 2023-10-16T13:46:58.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.853+0000 7fefa0d73700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef9c0c7b50 con 0x7fef9c0d2860 2023-10-16T13:46:58.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.854+0000 7fef9ad9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef9c15d900 0x7fef9c152f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:58.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.854+0000 7fef9a59c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fef9c0ce9f0 0x7fef9c14f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:58.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.854+0000 7fef99d9b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fef9c0d2860 0x7fef9c0c8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:58.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.854+0000 7fef9a59c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fef9c0ce9f0 0x7fef9c14f210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:53528/0 (socket says 172.21.15.16:53528) 2023-10-16T13:46:58.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.854+0000 7fef99d9b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fef9c0d2860 0x7fef9c0c8dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:42726/0 (socket says 172.21.15.16:42726) 2023-10-16T13:46:58.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.854+0000 7fef9a59c700 1 -- 172.21.15.16:0/3874433026 learned_addr learned my addr 172.21.15.16:0/3874433026 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:58.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.854+0000 7fef99d9b700 1 -- 172.21.15.16:0/3874433026 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fef9c0ce9f0 msgr2=0x7fef9c14f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:58.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.854+0000 7fef99d9b700 1 --2- 172.21.15.16:0/3874433026 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fef9c0ce9f0 0x7fef9c14f210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:58.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.854+0000 7fef99d9b700 1 -- 172.21.15.16:0/3874433026 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef9c15d900 msgr2=0x7fef9c152f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:58.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.854+0000 7fef99d9b700 1 --2- 172.21.15.16:0/3874433026 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef9c15d900 0x7fef9c152f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:58.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.854+0000 7fef99d9b700 1 -- 172.21.15.16:0/3874433026 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef94009b30 con 0x7fef9c0d2860 2023-10-16T13:46:58.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.854+0000 7fef99d9b700 1 --2- 172.21.15.16:0/3874433026 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fef9c0d2860 0x7fef9c0c8dc0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fef9000e870 tx=0x7fef9000eb80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:58.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.854+0000 7fef8affd700 1 -- 172.21.15.16:0/3874433026 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef90013680 con 0x7fef9c0d2860 2023-10-16T13:46:58.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.854+0000 7fef8affd700 1 -- 172.21.15.16:0/3874433026 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef900137e0 con 0x7fef9c0d2860 2023-10-16T13:46:58.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.854+0000 7fefa0d73700 1 -- 172.21.15.16:0/3874433026 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef8c0094a0 con 0x7fef9c0d2860 2023-10-16T13:46:58.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.854+0000 7fef8affd700 1 -- 172.21.15.16:0/3874433026 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef90013b50 con 0x7fef9c0d2860 2023-10-16T13:46:58.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.854+0000 7fefa0d73700 1 -- 172.21.15.16:0/3874433026 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef9c0c8110 con 0x7fef9c0d2860 2023-10-16T13:46:58.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.856+0000 7fefa0d73700 1 -- 172.21.15.16:0/3874433026 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef9c158a60 con 0x7fef9c0d2860 2023-10-16T13:46:58.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.856+0000 7fef8affd700 1 -- 172.21.15.16:0/3874433026 <== mon.2 v2:172.21.15.49:3301/0 4 ==== mgrmap(e 144) v1 ==== 89490+0+0 (secure 0 0 0) 0x7fef90013cb0 con 0x7fef9c0d2860 2023-10-16T13:46:58.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.856+0000 7fef8affd700 1 --2- 172.21.15.16:0/3874433026 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7fef7c06b1d0 0x7fef7c06d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:58.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.856+0000 7fef9a59c700 1 -- 172.21.15.16:0/3874433026 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7fef7c06b1d0 msgr2=0x7fef7c06d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1506142504 2023-10-16T13:46:58.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.856+0000 7fef8affd700 1 -- 172.21.15.16:0/3874433026 <== mon.2 v2:172.21.15.49:3301/0 5 ==== osd_map(230..230 src has 1..230) v4 ==== 7365+0+0 (secure 0 0 0) 0x7fef90089f30 con 0x7fef9c0d2860 2023-10-16T13:46:58.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.856+0000 7fef9a59c700 1 --2- 172.21.15.16:0/3874433026 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7fef7c06b1d0 0x7fef7c06d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:46:58.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:58.861+0000 7fef8affd700 1 -- 172.21.15.16:0/3874433026 <== mon.2 v2:172.21.15.49:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fef90054610 con 0x7fef9c0d2860 2023-10-16T13:46:59.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.057+0000 7fef9a59c700 1 -- 172.21.15.16:0/3874433026 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7fef7c06b1d0 msgr2=0x7fef7c06d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1506142504 2023-10-16T13:46:59.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.057+0000 7fef9a59c700 1 --2- 172.21.15.16:0/3874433026 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7fef7c06b1d0 0x7fef7c06d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:46:59.073 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:46:59.070+0000 7f0d0a270200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:46:59.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.082+0000 7fefa0d73700 1 -- 172.21.15.16:0/3874433026 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fef9c04ebb0 con 0x7fef9c0d2860 2023-10-16T13:46:59.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.082+0000 7fef8affd700 1 -- 172.21.15.16:0/3874433026 <== mon.2 v2:172.21.15.49:3301/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v230) v1 ==== 80+0+470 (secure 0 0 0) 0x7fef900211f0 con 0x7fef9c0d2860 2023-10-16T13:46:59.085 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2321929682 2023-10-17T13:46:51.165018+0000 2023-10-16T13:46:59.085 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3429506235 2023-10-17T13:46:51.165018+0000 2023-10-16T13:46:59.085 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3843126404 2023-10-17T13:46:51.165018+0000 2023-10-16T13:46:59.085 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/1439318665 2023-10-17T13:46:51.165018+0000 2023-10-16T13:46:59.086 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/4204021371 2023-10-17T13:46:51.165018+0000 2023-10-16T13:46:59.086 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6833/3171422553 2023-10-17T13:46:51.165018+0000 2023-10-16T13:46:59.086 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6832/3171422553 2023-10-17T13:46:51.165018+0000 2023-10-16T13:46:59.086 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/1632287797 2023-10-17T13:46:51.165018+0000 2023-10-16T13:46:59.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.084+0000 7fefa0d73700 1 -- 172.21.15.16:0/3874433026 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7fef7c06b1d0 msgr2=0x7fef7c06d690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:46:59.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.084+0000 7fefa0d73700 1 --2- 172.21.15.16:0/3874433026 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7fef7c06b1d0 0x7fef7c06d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:59.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.084+0000 7fefa0d73700 1 -- 172.21.15.16:0/3874433026 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fef9c0d2860 msgr2=0x7fef9c0c8dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:59.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.084+0000 7fefa0d73700 1 --2- 172.21.15.16:0/3874433026 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fef9c0d2860 0x7fef9c0c8dc0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fef9000e870 tx=0x7fef9000eb80 comp rx=0 tx=0).stop 2023-10-16T13:46:59.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.084+0000 7fefa0d73700 1 -- 172.21.15.16:0/3874433026 shutdown_connections 2023-10-16T13:46:59.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.084+0000 7fefa0d73700 1 --2- 172.21.15.16:0/3874433026 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7fef7c06b1d0 0x7fef7c06d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:59.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.084+0000 7fefa0d73700 1 --2- 172.21.15.16:0/3874433026 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef9c15d900 0x7fef9c152f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:59.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.084+0000 7fefa0d73700 1 --2- 172.21.15.16:0/3874433026 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fef9c0ce9f0 0x7fef9c14f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:59.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.084+0000 7fefa0d73700 1 --2- 172.21.15.16:0/3874433026 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fef9c0d2860 0x7fef9c0c8dc0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:59.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.084+0000 7fefa0d73700 1 -- 172.21.15.16:0/3874433026 >> 172.21.15.16:0/3874433026 conn(0x7fef9c072330 msgr2=0x7fef9c0d0610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:59.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.085+0000 7fefa0d73700 1 -- 172.21.15.16:0/3874433026 shutdown_connections 2023-10-16T13:46:59.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.085+0000 7fefa0d73700 1 -- 172.21.15.16:0/3874433026 wait complete. 2023-10-16T13:46:59.090 INFO:teuthology.orchestra.run.smithi016.stderr:listed 8 entries 2023-10-16T13:46:59.100 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-16T13:46:59.100 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:46:59.101 DEBUG:teuthology.orchestra.run.smithi016:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-16T13:46:59.141 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2023-10-16T13:46:59.142 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:46:59.142 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-16T13:46:59.155 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:46:59.152+0000 7f0d0a270200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T13:46:59.176 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:59.174+0000 7fb67c01c200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:46:59.192 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T13:46:59.193 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:46:59.211 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284204k,nr_inodes=4071051,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=753 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260540k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T13:46:59.211 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T13:46:59.212 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:46:59.249 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16287364k,nr_inodes=4071841,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20251 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3261172k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T13:46:59.249 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T13:46:59.250 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:46:59.259 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:59.257+0000 7fb67c01c200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T13:46:59.269 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284204k,nr_inodes=4071051,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=753 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260540k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T13:46:59.269 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:46:59.307 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:46:59.304+0000 7f0d0a270200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:46:59.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.393+0000 7f1a47c47700 1 Processor -- start 2023-10-16T13:46:59.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.393+0000 7f1a47c47700 1 -- start start 2023-10-16T13:46:59.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.394+0000 7f1a47c47700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f1a4015d8d0 0x7f1a400d1c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:59.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.394+0000 7f1a47c47700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1a400d41c0 0x7f1a400d0040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:59.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.394+0000 7f1a47c47700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1a400d0580 0x7f1a400ce240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:59.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.394+0000 7f1a47c47700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a400d0a10 con 0x7f1a400d0580 2023-10-16T13:46:59.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.394+0000 7f1a47c47700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a4006ba00 con 0x7f1a4015d8d0 2023-10-16T13:46:59.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.394+0000 7f1a47c47700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a400d2150 con 0x7f1a400d41c0 2023-10-16T13:46:59.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.394+0000 7f1a461e4700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1a400d0580 0x7f1a400ce240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:59.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.394+0000 7f1a461e4700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1a400d0580 0x7f1a400ce240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54514/0 (socket says 172.21.15.16:54514) 2023-10-16T13:46:59.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.394+0000 7f1a461e4700 1 -- 172.21.15.16:0/795745204 learned_addr learned my addr 172.21.15.16:0/795745204 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:59.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.394+0000 7f1a459e3700 1 --2- 172.21.15.16:0/795745204 >> v2:172.21.15.49:3300/0 conn(0x7f1a4015d8d0 0x7f1a400d1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:59.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.395+0000 7f1a451e2700 1 --2- 172.21.15.16:0/795745204 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1a400d41c0 0x7f1a400d0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:59.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.395+0000 7f1a461e4700 1 -- 172.21.15.16:0/795745204 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1a400d41c0 msgr2=0x7f1a400d0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:59.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.395+0000 7f1a461e4700 1 --2- 172.21.15.16:0/795745204 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1a400d41c0 0x7f1a400d0040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:59.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.395+0000 7f1a461e4700 1 -- 172.21.15.16:0/795745204 >> v2:172.21.15.49:3300/0 conn(0x7f1a4015d8d0 msgr2=0x7f1a400d1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:46:59.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.395+0000 7f1a461e4700 1 --2- 172.21.15.16:0/795745204 >> v2:172.21.15.49:3300/0 conn(0x7f1a4015d8d0 0x7f1a400d1c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:59.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.395+0000 7f1a461e4700 1 -- 172.21.15.16:0/795745204 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a400d26d0 con 0x7f1a400d0580 2023-10-16T13:46:59.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.395+0000 7f1a459e3700 1 --2- 172.21.15.16:0/795745204 >> v2:172.21.15.49:3300/0 conn(0x7f1a4015d8d0 0x7f1a400d1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:46:59.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.395+0000 7f1a461e4700 1 --2- 172.21.15.16:0/795745204 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a400d0580 0x7f1a400ce240 secure :-1 s=READY pgs=2600 cs=0 l=1 rev1=1 crypto rx=0x7f1a4008b460 tx=0x7f1a3800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38a1b31c9d661920 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:59.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.395+0000 7f1a449e1700 1 -- 172.21.15.16:0/795745204 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a3800e070 con 0x7f1a400d0580 2023-10-16T13:46:59.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.395+0000 7f1a449e1700 1 -- 172.21.15.16:0/795745204 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a3800b910 con 0x7f1a400d0580 2023-10-16T13:46:59.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.395+0000 7f1a449e1700 1 -- 172.21.15.16:0/795745204 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a3800bcc0 con 0x7f1a400d0580 2023-10-16T13:46:59.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.396+0000 7f1a47c47700 1 -- 172.21.15.16:0/795745204 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a400d0580 msgr2=0x7f1a400ce240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:59.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.396+0000 7f1a47c47700 1 --2- 172.21.15.16:0/795745204 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a400d0580 0x7f1a400ce240 secure :-1 s=READY pgs=2600 cs=0 l=1 rev1=1 crypto rx=0x7f1a4008b460 tx=0x7f1a3800b330 comp rx=0 tx=0).stop 2023-10-16T13:46:59.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.396+0000 7f1a47c47700 1 -- 172.21.15.16:0/795745204 shutdown_connections 2023-10-16T13:46:59.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.396+0000 7f1a47c47700 1 --2- 172.21.15.16:0/795745204 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a400d0580 0x7f1a400ce240 unknown :-1 s=CLOSED pgs=2600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:59.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.396+0000 7f1a47c47700 1 --2- 172.21.15.16:0/795745204 >> v2:172.21.15.49:3300/0 conn(0x7f1a4015d8d0 0x7f1a400d1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:59.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.396+0000 7f1a47c47700 1 --2- 172.21.15.16:0/795745204 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1a400d41c0 0x7f1a400d0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:59.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.396+0000 7f1a47c47700 1 -- 172.21.15.16:0/795745204 >> 172.21.15.16:0/795745204 conn(0x7f1a400d3630 msgr2=0x7f1a400d3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:59.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.396+0000 7f1a47c47700 1 -- 172.21.15.16:0/795745204 shutdown_connections 2023-10-16T13:46:59.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.396+0000 7f1a47c47700 1 -- 172.21.15.16:0/795745204 wait complete. 2023-10-16T13:46:59.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.397+0000 7f1a47c47700 1 Processor -- start 2023-10-16T13:46:59.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.397+0000 7f1a47c47700 1 -- start start 2023-10-16T13:46:59.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.398+0000 7f1a47c47700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a400d0580 0x7f1a4014f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:59.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.398+0000 7f1a47c47700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1a400d41c0 0x7f1a401f6da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:59.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.398+0000 7f1a47c47700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1a4015d8d0 0x7f1a401e6960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:59.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.398+0000 7f1a47c47700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a401e6290 con 0x7f1a400d0580 2023-10-16T13:46:59.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.398+0000 7f1a47c47700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a4014fd90 con 0x7f1a4015d8d0 2023-10-16T13:46:59.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.398+0000 7f1a47c47700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a401e5610 con 0x7f1a400d41c0 2023-10-16T13:46:59.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.398+0000 7f1a459e3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a400d0580 0x7f1a4014f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:59.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.398+0000 7f1a459e3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a400d0580 0x7f1a4014f850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54516/0 (socket says 172.21.15.16:54516) 2023-10-16T13:46:59.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.398+0000 7f1a459e3700 1 -- 172.21.15.16:0/2119043408 learned_addr learned my addr 172.21.15.16:0/2119043408 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:59.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.398+0000 7f1a451e2700 1 --2- 172.21.15.16:0/2119043408 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1a400d41c0 0x7f1a401f6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:59.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.398+0000 7f1a461e4700 1 --2- 172.21.15.16:0/2119043408 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1a4015d8d0 0x7f1a401e6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:59.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.398+0000 7f1a459e3700 1 -- 172.21.15.16:0/2119043408 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1a400d41c0 msgr2=0x7f1a401f6da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:59.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.398+0000 7f1a459e3700 1 --2- 172.21.15.16:0/2119043408 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1a400d41c0 0x7f1a401f6da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:59.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.398+0000 7f1a459e3700 1 -- 172.21.15.16:0/2119043408 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1a4015d8d0 msgr2=0x7f1a401e6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:59.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.398+0000 7f1a459e3700 1 --2- 172.21.15.16:0/2119043408 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1a4015d8d0 0x7f1a401e6960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:59.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.398+0000 7f1a459e3700 1 -- 172.21.15.16:0/2119043408 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a3c0094a0 con 0x7f1a400d0580 2023-10-16T13:46:59.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.398+0000 7f1a459e3700 1 --2- 172.21.15.16:0/2119043408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a400d0580 0x7f1a4014f850 secure :-1 s=READY pgs=2601 cs=0 l=1 rev1=1 crypto rx=0x7f1a3c00ebe0 tx=0x7f1a3c00eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:59.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.399+0000 7f1a367fc700 1 -- 172.21.15.16:0/2119043408 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a3c00c850 con 0x7f1a400d0580 2023-10-16T13:46:59.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.399+0000 7f1a367fc700 1 -- 172.21.15.16:0/2119043408 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a3c00c9b0 con 0x7f1a400d0580 2023-10-16T13:46:59.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.399+0000 7f1a47c47700 1 -- 172.21.15.16:0/2119043408 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a38009b30 con 0x7f1a400d0580 2023-10-16T13:46:59.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.399+0000 7f1a367fc700 1 -- 172.21.15.16:0/2119043408 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a3c00cd40 con 0x7f1a400d0580 2023-10-16T13:46:59.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.399+0000 7f1a47c47700 1 -- 172.21.15.16:0/2119043408 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a401e5bd0 con 0x7f1a400d0580 2023-10-16T13:46:59.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.400+0000 7f1a47c47700 1 -- 172.21.15.16:0/2119043408 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a4004ebb0 con 0x7f1a400d0580 2023-10-16T13:46:59.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.400+0000 7f1a367fc700 1 -- 172.21.15.16:0/2119043408 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 144) v1 ==== 89490+0+0 (secure 0 0 0) 0x7f1a3c013470 con 0x7f1a400d0580 2023-10-16T13:46:59.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.401+0000 7f1a367fc700 1 --2- 172.21.15.16:0/2119043408 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f1a2406b1d0 0x7f1a2406d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:59.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.401+0000 7f1a451e2700 1 -- 172.21.15.16:0/2119043408 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f1a2406b1d0 msgr2=0x7f1a2406d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1506142504 2023-10-16T13:46:59.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.401+0000 7f1a367fc700 1 -- 172.21.15.16:0/2119043408 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(230..230 src has 1..230) v4 ==== 7365+0+0 (secure 0 0 0) 0x7f1a3c08a1b0 con 0x7f1a400d0580 2023-10-16T13:46:59.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.401+0000 7f1a451e2700 1 --2- 172.21.15.16:0/2119043408 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f1a2406b1d0 0x7f1a2406d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:46:59.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.405+0000 7f1a367fc700 1 -- 172.21.15.16:0/2119043408 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1a3c054890 con 0x7f1a400d0580 2023-10-16T13:46:59.419 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:59.417+0000 7fb67c01c200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:46:59.547 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:46:59.544+0000 7f0d0a270200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T13:46:59.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.602+0000 7f1a451e2700 1 -- 172.21.15.16:0/2119043408 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f1a2406b1d0 msgr2=0x7f1a2406d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1506142504 2023-10-16T13:46:59.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.602+0000 7f1a451e2700 1 --2- 172.21.15.16:0/2119043408 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f1a2406b1d0 0x7f1a2406d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:46:59.625 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:46:59.623+0000 7f0d0a270200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T13:46:59.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.659+0000 7f1a47c47700 1 -- 172.21.15.16:0/2119043408 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a400cf2a0 con 0x7f1a400d0580 2023-10-16T13:46:59.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.659+0000 7f1a367fc700 1 -- 172.21.15.16:0/2119043408 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) v1 ==== 95+0+6154 (secure 0 0 0) 0x7f1a3c00cea0 con 0x7f1a400d0580 2023-10-16T13:46:59.662 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:46:59.662 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":119,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10516,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/4065699432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":4065699432},{"type":"v1","addr":"172.21.15.49:6834","nonce":4065699432}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":108},{"gid":11415,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/1373048456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1373048456},{"type":"v1","addr":"172.21.15.16:6836","nonce":1373048456}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":108}],"filesystems":[{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:46:24.755313+0000","modified":"2023-10-16T13:46:28.011141+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11430},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11430":{"gid":11430,"name":"b","rank":0,"incarnation":109,"state":"up:active","state_seq":3,"addr":"172.21.15.49:6835/2905760119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":2905760119},{"type":"v1","addr":"172.21.15.49:6835","nonce":2905760119}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13},{"mdsmap":{"epoch":119,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:46:38.053490+0000","modified":"2023-10-16T13:46:41.076680+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11418},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11418":{"gid":11418,"name":"a","rank":0,"incarnation":115,"state":"up:active","state_seq":6,"addr":"172.21.15.16:6837/3860406853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":3860406853},{"type":"v1","addr":"172.21.15.16:6837","nonce":3860406853}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2023-10-16T13:46:59.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.661+0000 7f1a47c47700 1 -- 172.21.15.16:0/2119043408 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f1a2406b1d0 msgr2=0x7f1a2406d690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:46:59.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.662+0000 7f1a47c47700 1 --2- 172.21.15.16:0/2119043408 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f1a2406b1d0 0x7f1a2406d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:59.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.662+0000 7f1a47c47700 1 -- 172.21.15.16:0/2119043408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a400d0580 msgr2=0x7f1a4014f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:59.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.662+0000 7f1a47c47700 1 --2- 172.21.15.16:0/2119043408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a400d0580 0x7f1a4014f850 secure :-1 s=READY pgs=2601 cs=0 l=1 rev1=1 crypto rx=0x7f1a3c00ebe0 tx=0x7f1a3c00eef0 comp rx=0 tx=0).stop 2023-10-16T13:46:59.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.662+0000 7f1a47c47700 1 -- 172.21.15.16:0/2119043408 shutdown_connections 2023-10-16T13:46:59.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.662+0000 7f1a47c47700 1 --2- 172.21.15.16:0/2119043408 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f1a2406b1d0 0x7f1a2406d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:59.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.662+0000 7f1a47c47700 1 --2- 172.21.15.16:0/2119043408 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1a4015d8d0 0x7f1a401e6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:59.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.662+0000 7f1a47c47700 1 --2- 172.21.15.16:0/2119043408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a400d0580 0x7f1a4014f850 unknown :-1 s=CLOSED pgs=2601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:59.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.662+0000 7f1a47c47700 1 --2- 172.21.15.16:0/2119043408 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1a400d41c0 0x7f1a401f6da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:59.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.662+0000 7f1a47c47700 1 -- 172.21.15.16:0/2119043408 >> 172.21.15.16:0/2119043408 conn(0x7f1a400d3630 msgr2=0x7f1a400ca700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:59.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.662+0000 7f1a47c47700 1 -- 172.21.15.16:0/2119043408 shutdown_connections 2023-10-16T13:46:59.667 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.662+0000 7f1a47c47700 1 -- 172.21.15.16:0/2119043408 wait complete. 2023-10-16T13:46:59.667 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 119 2023-10-16T13:46:59.667 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:59.663+0000 7fb67c01c200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T13:46:59.677 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:46:59.748 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:46:59.745+0000 7fb67c01c200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T13:46:59.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.802+0000 7f677b47e700 1 Processor -- start 2023-10-16T13:46:59.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.802+0000 7f677b47e700 1 -- start start 2023-10-16T13:46:59.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.803+0000 7f677b47e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f67740ce380 0x7f67740d0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:59.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.803+0000 7f677b47e700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67740d1210 con 0x7f67740b2b50 2023-10-16T13:46:59.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.803+0000 7f677b47e700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67740cbc30 con 0x7f67740cdfa0 2023-10-16T13:46:59.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.803+0000 7f677b47e700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67740cbf10 con 0x7f67740ce380 2023-10-16T13:46:59.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.803+0000 7f6778a19700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f67740b2b50 0x7f67740d04a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:43800/0 (socket says 172.21.15.16:43800) 2023-10-16T13:46:59.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.803+0000 7f6778a19700 1 -- 172.21.15.16:0/1428963512 learned_addr learned my addr 172.21.15.16:0/1428963512 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:46:59.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.803+0000 7f6779a1b700 1 --2- 172.21.15.16:0/1428963512 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f67740ce380 0x7f67740d0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:59.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.803+0000 7f676bfff700 1 -- 172.21.15.16:0/1428963512 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3505062482 0 0) 0x7f67740d1210 con 0x7f67740b2b50 2023-10-16T13:46:59.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.803+0000 7f676bfff700 1 -- 172.21.15.16:0/1428963512 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67600035b0 con 0x7f67740b2b50 2023-10-16T13:46:59.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.804+0000 7f676bfff700 1 -- 172.21.15.16:0/1428963512 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1506609392 0 0) 0x7f67740cbc30 con 0x7f67740cdfa0 2023-10-16T13:46:59.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.804+0000 7f676bfff700 1 -- 172.21.15.16:0/1428963512 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67740d1210 con 0x7f67740cdfa0 2023-10-16T13:46:59.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.804+0000 7f6779a1b700 1 -- 172.21.15.16:0/1428963512 >> v1:172.21.15.49:6789/0 conn(0x7f67740cdfa0 legacy=0x7f67740b2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:59.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.804+0000 7f6779a1b700 1 -- 172.21.15.16:0/1428963512 >> v1:172.21.15.16:6789/0 conn(0x7f67740b2b50 legacy=0x7f67740d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:59.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.804+0000 7f6779a1b700 1 -- 172.21.15.16:0/1428963512 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67740cc1f0 con 0x7f67740ce380 2023-10-16T13:46:59.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.804+0000 7f6779a1b700 1 --2- 172.21.15.16:0/1428963512 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f67740ce380 0x7f67740d0bb0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f676c009dc0 tx=0x7f676c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=18b55926f10b1545 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:59.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.804+0000 7f676bfff700 1 -- 172.21.15.16:0/1428963512 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f676c00e070 con 0x7f67740ce380 2023-10-16T13:46:59.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.804+0000 7f676bfff700 1 -- 172.21.15.16:0/1428963512 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f676c00bb20 con 0x7f67740ce380 2023-10-16T13:46:59.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.804+0000 7f676bfff700 1 -- 172.21.15.16:0/1428963512 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f676c012430 con 0x7f67740ce380 2023-10-16T13:46:59.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.805+0000 7f677b47e700 1 -- 172.21.15.16:0/1428963512 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f67740ce380 msgr2=0x7f67740d0bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:59.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.805+0000 7f677b47e700 1 --2- 172.21.15.16:0/1428963512 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f67740ce380 0x7f67740d0bb0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f676c009dc0 tx=0x7f676c00b330 comp rx=0 tx=0).stop 2023-10-16T13:46:59.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.805+0000 7f677b47e700 1 -- 172.21.15.16:0/1428963512 shutdown_connections 2023-10-16T13:46:59.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.805+0000 7f677b47e700 1 --2- 172.21.15.16:0/1428963512 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f67740ce380 0x7f67740d0bb0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:59.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.805+0000 7f677b47e700 1 -- 172.21.15.16:0/1428963512 >> 172.21.15.16:0/1428963512 conn(0x7f67740d14b0 msgr2=0x7f67740d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:46:59.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.805+0000 7f677b47e700 1 -- 172.21.15.16:0/1428963512 shutdown_connections 2023-10-16T13:46:59.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.805+0000 7f677b47e700 1 -- 172.21.15.16:0/1428963512 wait complete. 2023-10-16T13:46:59.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.806+0000 7f677b47e700 1 Processor -- start 2023-10-16T13:46:59.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.806+0000 7f677b47e700 1 -- start start 2023-10-16T13:46:59.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.807+0000 7f677b47e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f67740b2b50 0x7f67740ca6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:59.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.807+0000 7f677b47e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f67740cdfa0 0x7f677409d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:59.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.807+0000 7f677b47e700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f67740ce380 0x7f677409b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:59.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.807+0000 7f677b47e700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f677409e890 con 0x7f67740cdfa0 2023-10-16T13:46:59.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.807+0000 7f677b47e700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f677409dc10 con 0x7f67740ce380 2023-10-16T13:46:59.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.807+0000 7f677b47e700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f677409def0 con 0x7f67740b2b50 2023-10-16T13:46:59.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.807+0000 7f6778a19700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f67740cdfa0 0x7f677409d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:59.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.807+0000 7f677921a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f67740b2b50 0x7f67740ca6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:59.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.807+0000 7f6778a19700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f67740cdfa0 0x7f677409d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54534/0 (socket says 172.21.15.16:54534) 2023-10-16T13:46:59.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.807+0000 7f6779a1b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f67740ce380 0x7f677409b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:46:59.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.807+0000 7f677921a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f67740b2b50 0x7f67740ca6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:42790/0 (socket says 172.21.15.16:42790) 2023-10-16T13:46:59.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.807+0000 7f6778a19700 1 -- 172.21.15.16:0/1241347965 learned_addr learned my addr 172.21.15.16:0/1241347965 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:46:59.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.808+0000 7f6778a19700 1 -- 172.21.15.16:0/1241347965 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f67740b2b50 msgr2=0x7f67740ca6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:59.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.808+0000 7f6778a19700 1 --2- 172.21.15.16:0/1241347965 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f67740b2b50 0x7f67740ca6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:59.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.808+0000 7f6778a19700 1 -- 172.21.15.16:0/1241347965 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f67740ce380 msgr2=0x7f677409b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:46:59.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.808+0000 7f6778a19700 1 --2- 172.21.15.16:0/1241347965 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f67740ce380 0x7f677409b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:46:59.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.808+0000 7f6778a19700 1 -- 172.21.15.16:0/1241347965 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f676c009a70 con 0x7f67740cdfa0 2023-10-16T13:46:59.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.808+0000 7f677921a700 1 --2- 172.21.15.16:0/1241347965 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f67740b2b50 0x7f67740ca6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:46:59.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.808+0000 7f6779a1b700 1 --2- 172.21.15.16:0/1241347965 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f67740ce380 0x7f677409b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:46:59.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.808+0000 7f6778a19700 1 --2- 172.21.15.16:0/1241347965 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f67740cdfa0 0x7f677409d3c0 secure :-1 s=READY pgs=2603 cs=0 l=1 rev1=1 crypto rx=0x7f676400c700 tx=0x7f676400ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:46:59.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.808+0000 7f6769ffb700 1 -- 172.21.15.16:0/1241347965 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6764005d70 con 0x7f67740cdfa0 2023-10-16T13:46:59.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.808+0000 7f6769ffb700 1 -- 172.21.15.16:0/1241347965 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67640070d0 con 0x7f67740cdfa0 2023-10-16T13:46:59.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.808+0000 7f677b47e700 1 -- 172.21.15.16:0/1241347965 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f677409e1d0 con 0x7f67740cdfa0 2023-10-16T13:46:59.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.808+0000 7f6769ffb700 1 -- 172.21.15.16:0/1241347965 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6764007440 con 0x7f67740cdfa0 2023-10-16T13:46:59.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.808+0000 7f677b47e700 1 -- 172.21.15.16:0/1241347965 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67740cac00 con 0x7f67740cdfa0 2023-10-16T13:46:59.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.809+0000 7f677b47e700 1 -- 172.21.15.16:0/1241347965 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f677409e9d0 con 0x7f67740cdfa0 2023-10-16T13:46:59.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.810+0000 7f6769ffb700 1 -- 172.21.15.16:0/1241347965 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 144) v1 ==== 89490+0+0 (secure 0 0 0) 0x7f67640075a0 con 0x7f67740cdfa0 2023-10-16T13:46:59.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.810+0000 7f6769ffb700 1 --2- 172.21.15.16:0/1241347965 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f675806b1d0 0x7f675806d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:46:59.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.810+0000 7f677921a700 1 -- 172.21.15.16:0/1241347965 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f675806b1d0 msgr2=0x7f675806d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1506142504 2023-10-16T13:46:59.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.810+0000 7f6769ffb700 1 -- 172.21.15.16:0/1241347965 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(230..230 src has 1..230) v4 ==== 7365+0+0 (secure 0 0 0) 0x7f67640893b0 con 0x7f67740cdfa0 2023-10-16T13:46:59.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.810+0000 7f677921a700 1 --2- 172.21.15.16:0/1241347965 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f675806b1d0 0x7f675806d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:46:59.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:46:59.815+0000 7f6769ffb700 1 -- 172.21.15.16:0/1241347965 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f677409e9d0 con 0x7f67740cdfa0 2023-10-16T13:46:59.902 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:46:59.899+0000 7f0d0a270200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T13:47:00.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.011+0000 7f677921a700 1 -- 172.21.15.16:0/1241347965 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f675806b1d0 msgr2=0x7f675806d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1506142504 2023-10-16T13:47:00.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.011+0000 7f677921a700 1 --2- 172.21.15.16:0/1241347965 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f675806b1d0 0x7f675806d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:47:00.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.024+0000 7f677b47e700 1 -- 172.21.15.16:0/1241347965 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f677409e9d0 con 0x7f67740cdfa0 2023-10-16T13:47:00.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.025+0000 7f6769ffb700 1 -- 172.21.15.16:0/1241347965 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v230) v1 ==== 74+0+21038 (secure 0 0 0) 0x7f677409e9d0 con 0x7f67740cdfa0 2023-10-16T13:47:00.028 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:47:00.028 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":230,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:46:51.165060+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2023-10-16T13:46:22.489653+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"224","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2023-10-16T13:46:23.729509+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"225","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T13:46:35.800444+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"228","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"backup_fs_data","create_time":"2023-10-16T13:46:37.035323+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"229","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/2321929682":"2023-10-17T13:46:51.165018+0000","172.21.15.16:0/3429506235":"2023-10-17T13:46:51.165018+0000","172.21.15.16:0/3843126404":"2023-10-17T13:46:51.165018+0000","172.21.15.16:0/1439318665":"2023-10-17T13:46:51.165018+0000","172.21.15.16:0/4204021371":"2023-10-17T13:46:51.165018+0000","172.21.15.16:6833/3171422553":"2023-10-17T13:46:51.165018+0000","172.21.15.16:6832/3171422553":"2023-10-17T13:46:51.165018+0000","172.21.15.16:0/1632287797":"2023-10-17T13:46:51.165018+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:47:00.030 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:00.027+0000 7fb67c01c200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T13:47:00.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.027+0000 7f677b47e700 1 -- 172.21.15.16:0/1241347965 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f675806b1d0 msgr2=0x7f675806d690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:47:00.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.027+0000 7f677b47e700 1 --2- 172.21.15.16:0/1241347965 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f675806b1d0 0x7f675806d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:00.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.027+0000 7f677b47e700 1 -- 172.21.15.16:0/1241347965 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f67740cdfa0 msgr2=0x7f677409d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:00.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.027+0000 7f677b47e700 1 --2- 172.21.15.16:0/1241347965 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f67740cdfa0 0x7f677409d3c0 secure :-1 s=READY pgs=2603 cs=0 l=1 rev1=1 crypto rx=0x7f676400c700 tx=0x7f676400ca10 comp rx=0 tx=0).stop 2023-10-16T13:47:00.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.028+0000 7f677b47e700 1 -- 172.21.15.16:0/1241347965 shutdown_connections 2023-10-16T13:47:00.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.028+0000 7f677b47e700 1 --2- 172.21.15.16:0/1241347965 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f675806b1d0 0x7f675806d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:00.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.028+0000 7f677b47e700 1 --2- 172.21.15.16:0/1241347965 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f67740ce380 0x7f677409b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:00.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.028+0000 7f677b47e700 1 --2- 172.21.15.16:0/1241347965 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f67740b2b50 0x7f67740ca6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:00.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.028+0000 7f677b47e700 1 --2- 172.21.15.16:0/1241347965 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f67740cdfa0 0x7f677409d3c0 unknown :-1 s=CLOSED pgs=2603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:00.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.028+0000 7f677b47e700 1 -- 172.21.15.16:0/1241347965 >> 172.21.15.16:0/1241347965 conn(0x7f67740d14b0 msgr2=0x7f6774160930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:00.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.028+0000 7f677b47e700 1 -- 172.21.15.16:0/1241347965 shutdown_connections 2023-10-16T13:47:00.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.028+0000 7f677b47e700 1 -- 172.21.15.16:0/1241347965 wait complete. 2023-10-16T13:47:00.043 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 27, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T13:46:22.489653+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '224', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 28, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T13:46:23.729509+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '225', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 29, 'pool_name': 'backup_fs_metadata', 'create_time': '2023-10-16T13:46:35.800444+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '228', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 30, 'pool_name': 'backup_fs_data', 'create_time': '2023-10-16T13:46:37.035323+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '229', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T13:47:00.044 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T13:47:00.134 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:47:00.132+0000 7f0d0a270200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T13:47:00.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.171+0000 7f21b1d2d700 1 Processor -- start 2023-10-16T13:47:00.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.171+0000 7f21b1d2d700 1 -- start start 2023-10-16T13:47:00.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.172+0000 7f21b1d2d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f21ac15b790 0x7f21ac0cda60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:00.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.172+0000 7f21b1d2d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f21ac15bb70 0x7f21ac06d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:00.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.172+0000 7f21b1d2d700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f21ac06dea0 0x7f21ac15f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:00.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.172+0000 7f21b1d2d700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21ac06e310 con 0x7f21ac15bb70 2023-10-16T13:47:00.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.172+0000 7f21b1d2d700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21ac0ce0e0 con 0x7f21ac06dea0 2023-10-16T13:47:00.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.172+0000 7f21b1d2d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21ac06a7e0 con 0x7f21ac15b790 2023-10-16T13:47:00.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.172+0000 7f21aaffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f21ac15bb70 0x7f21ac06d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:00.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.172+0000 7f21aaffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f21ac15bb70 0x7f21ac06d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54544/0 (socket says 172.21.15.16:54544) 2023-10-16T13:47:00.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.172+0000 7f21aaffd700 1 -- 172.21.15.16:0/162718545 learned_addr learned my addr 172.21.15.16:0/162718545 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:00.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.172+0000 7f21abfff700 1 --2- 172.21.15.16:0/162718545 >> v2:172.21.15.49:3300/0 conn(0x7f21ac06dea0 0x7f21ac15f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:00.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.172+0000 7f21ab7fe700 1 --2- 172.21.15.16:0/162718545 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f21ac15b790 0x7f21ac0cda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:00.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.172+0000 7f21aaffd700 1 -- 172.21.15.16:0/162718545 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f21ac15b790 msgr2=0x7f21ac0cda60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:00.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.172+0000 7f21aaffd700 1 --2- 172.21.15.16:0/162718545 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f21ac15b790 0x7f21ac0cda60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:00.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.172+0000 7f21aaffd700 1 -- 172.21.15.16:0/162718545 >> v2:172.21.15.49:3300/0 conn(0x7f21ac06dea0 msgr2=0x7f21ac15f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:00.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.172+0000 7f21aaffd700 1 --2- 172.21.15.16:0/162718545 >> v2:172.21.15.49:3300/0 conn(0x7f21ac06dea0 0x7f21ac15f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:00.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.172+0000 7f21aaffd700 1 -- 172.21.15.16:0/162718545 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21ac15c410 con 0x7f21ac15bb70 2023-10-16T13:47:00.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.173+0000 7f21aaffd700 1 --2- 172.21.15.16:0/162718545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21ac15bb70 0x7f21ac06d960 secure :-1 s=READY pgs=2604 cs=0 l=1 rev1=1 crypto rx=0x7f21ac070fb0 tx=0x7f219c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44319f3b9ad4c63f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:00.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.173+0000 7f21aa7fc700 1 -- 172.21.15.16:0/162718545 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f219c011830 con 0x7f21ac15bb70 2023-10-16T13:47:00.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.173+0000 7f21aa7fc700 1 -- 172.21.15.16:0/162718545 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f219c011990 con 0x7f21ac15bb70 2023-10-16T13:47:00.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.173+0000 7f21aa7fc700 1 -- 172.21.15.16:0/162718545 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f219c011d20 con 0x7f21ac15bb70 2023-10-16T13:47:00.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.173+0000 7f21b1d2d700 1 -- 172.21.15.16:0/162718545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21ac15bb70 msgr2=0x7f21ac06d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:00.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.173+0000 7f21b1d2d700 1 --2- 172.21.15.16:0/162718545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21ac15bb70 0x7f21ac06d960 secure :-1 s=READY pgs=2604 cs=0 l=1 rev1=1 crypto rx=0x7f21ac070fb0 tx=0x7f219c009bd0 comp rx=0 tx=0).stop 2023-10-16T13:47:00.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.173+0000 7f21b1d2d700 1 -- 172.21.15.16:0/162718545 shutdown_connections 2023-10-16T13:47:00.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.173+0000 7f21b1d2d700 1 --2- 172.21.15.16:0/162718545 >> v2:172.21.15.49:3300/0 conn(0x7f21ac06dea0 0x7f21ac15f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:00.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.173+0000 7f21b1d2d700 1 --2- 172.21.15.16:0/162718545 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f21ac15b790 0x7f21ac0cda60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:00.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.173+0000 7f21b1d2d700 1 --2- 172.21.15.16:0/162718545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21ac15bb70 0x7f21ac06d960 unknown :-1 s=CLOSED pgs=2604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:00.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.173+0000 7f21b1d2d700 1 -- 172.21.15.16:0/162718545 >> 172.21.15.16:0/162718545 conn(0x7f21ac0cc5e0 msgr2=0x7f21ac0cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:00.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.174+0000 7f21b1d2d700 1 -- 172.21.15.16:0/162718545 shutdown_connections 2023-10-16T13:47:00.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.174+0000 7f21b1d2d700 1 -- 172.21.15.16:0/162718545 wait complete. 2023-10-16T13:47:00.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.175+0000 7f21b1d2d700 1 Processor -- start 2023-10-16T13:47:00.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.175+0000 7f21b1d2d700 1 -- start start 2023-10-16T13:47:00.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.176+0000 7f21b1d2d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21ac06dea0 0x7f21ac152340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:00.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.176+0000 7f21b1d2d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f21ac15b790 0x7f21ac0c9540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:00.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.176+0000 7f21b1d2d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f21ac15bb70 0x7f21ac06b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:00.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.176+0000 7f21b1d2d700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21ac0c7440 con 0x7f21ac06dea0 2023-10-16T13:47:00.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.176+0000 7f21b1d2d700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21ac152880 con 0x7f21ac15bb70 2023-10-16T13:47:00.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.176+0000 7f21b1d2d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21ac0c5ed0 con 0x7f21ac15b790 2023-10-16T13:47:00.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.176+0000 7f21ab7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21ac06dea0 0x7f21ac152340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:00.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.176+0000 7f21ab7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21ac06dea0 0x7f21ac152340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54548/0 (socket says 172.21.15.16:54548) 2023-10-16T13:47:00.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.176+0000 7f21ab7fe700 1 -- 172.21.15.16:0/3500061764 learned_addr learned my addr 172.21.15.16:0/3500061764 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:00.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.176+0000 7f21aaffd700 1 --2- 172.21.15.16:0/3500061764 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f21ac15b790 0x7f21ac0c9540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:00.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.176+0000 7f21abfff700 1 --2- 172.21.15.16:0/3500061764 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f21ac15bb70 0x7f21ac06b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:00.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.176+0000 7f21ab7fe700 1 -- 172.21.15.16:0/3500061764 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f21ac15b790 msgr2=0x7f21ac0c9540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:00.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.176+0000 7f21ab7fe700 1 --2- 172.21.15.16:0/3500061764 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f21ac15b790 0x7f21ac0c9540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:00.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.176+0000 7f21ab7fe700 1 -- 172.21.15.16:0/3500061764 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f21ac15bb70 msgr2=0x7f21ac06b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:00.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.176+0000 7f21ab7fe700 1 --2- 172.21.15.16:0/3500061764 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f21ac15bb70 0x7f21ac06b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:00.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.176+0000 7f21ab7fe700 1 -- 172.21.15.16:0/3500061764 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f219c009550 con 0x7f21ac06dea0 2023-10-16T13:47:00.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.176+0000 7f21ab7fe700 1 --2- 172.21.15.16:0/3500061764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21ac06dea0 0x7f21ac152340 secure :-1 s=READY pgs=2605 cs=0 l=1 rev1=1 crypto rx=0x7f219400e870 tx=0x7f219400eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:00.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.176+0000 7f21a3fff700 1 -- 172.21.15.16:0/3500061764 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2194009590 con 0x7f21ac06dea0 2023-10-16T13:47:00.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.176+0000 7f21a3fff700 1 -- 172.21.15.16:0/3500061764 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2194013680 con 0x7f21ac06dea0 2023-10-16T13:47:00.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.176+0000 7f21b1d2d700 1 -- 172.21.15.16:0/3500061764 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21ac0c61b0 con 0x7f21ac06dea0 2023-10-16T13:47:00.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.176+0000 7f21a3fff700 1 -- 172.21.15.16:0/3500061764 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21940139f0 con 0x7f21ac06dea0 2023-10-16T13:47:00.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.177+0000 7f21b1d2d700 1 -- 172.21.15.16:0/3500061764 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21ac0c70d0 con 0x7f21ac06dea0 2023-10-16T13:47:00.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.178+0000 7f21b1d2d700 1 -- 172.21.15.16:0/3500061764 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21ac04ebb0 con 0x7f21ac06dea0 2023-10-16T13:47:00.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.178+0000 7f21a3fff700 1 -- 172.21.15.16:0/3500061764 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 144) v1 ==== 89490+0+0 (secure 0 0 0) 0x7f2194013b50 con 0x7f21ac06dea0 2023-10-16T13:47:00.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.178+0000 7f21a3fff700 1 --2- 172.21.15.16:0/3500061764 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f218c06b1d0 0x7f218c06d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:00.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.178+0000 7f21aaffd700 1 -- 172.21.15.16:0/3500061764 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f218c06b1d0 msgr2=0x7f218c06d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1506142504 2023-10-16T13:47:00.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.178+0000 7f21a3fff700 1 -- 172.21.15.16:0/3500061764 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(230..230 src has 1..230) v4 ==== 7365+0+0 (secure 0 0 0) 0x7f2194089e70 con 0x7f21ac06dea0 2023-10-16T13:47:00.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.178+0000 7f21aaffd700 1 --2- 172.21.15.16:0/3500061764 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f218c06b1d0 0x7f218c06d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:47:00.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.187+0000 7f21a3fff700 1 -- 172.21.15.16:0/3500061764 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2194054550 con 0x7f21ac06dea0 2023-10-16T13:47:00.283 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:00.281+0000 7fb67c01c200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T13:47:00.291 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:47:00.289+0000 7f0d0a270200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T13:47:00.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.379+0000 7f21aaffd700 1 -- 172.21.15.16:0/3500061764 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f218c06b1d0 msgr2=0x7f218c06d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1506142504 2023-10-16T13:47:00.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.379+0000 7f21aaffd700 1 --2- 172.21.15.16:0/3500061764 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f218c06b1d0 0x7f218c06d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:47:00.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.388+0000 7f21b1d2d700 1 -- 172.21.15.16:0/3500061764 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f21ac0cedb0 con 0x7f21ac06dea0 2023-10-16T13:47:00.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.389+0000 7f21a3fff700 1 -- 172.21.15.16:0/3500061764 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v230) v1 ==== 93+0+10 (secure 0 0 0) 0x7f21940210a0 con 0x7f21ac06dea0 2023-10-16T13:47:00.391 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T13:47:00.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.391+0000 7f21b1d2d700 1 -- 172.21.15.16:0/3500061764 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f218c06b1d0 msgr2=0x7f218c06d690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:47:00.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.391+0000 7f21b1d2d700 1 --2- 172.21.15.16:0/3500061764 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f218c06b1d0 0x7f218c06d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:00.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.391+0000 7f21b1d2d700 1 -- 172.21.15.16:0/3500061764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21ac06dea0 msgr2=0x7f21ac152340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:00.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.391+0000 7f21b1d2d700 1 --2- 172.21.15.16:0/3500061764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21ac06dea0 0x7f21ac152340 secure :-1 s=READY pgs=2605 cs=0 l=1 rev1=1 crypto rx=0x7f219400e870 tx=0x7f219400eb80 comp rx=0 tx=0).stop 2023-10-16T13:47:00.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.391+0000 7f21b1d2d700 1 -- 172.21.15.16:0/3500061764 shutdown_connections 2023-10-16T13:47:00.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.391+0000 7f21b1d2d700 1 --2- 172.21.15.16:0/3500061764 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f218c06b1d0 0x7f218c06d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:00.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.391+0000 7f21b1d2d700 1 --2- 172.21.15.16:0/3500061764 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f21ac15bb70 0x7f21ac06b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:00.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.391+0000 7f21b1d2d700 1 --2- 172.21.15.16:0/3500061764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21ac06dea0 0x7f21ac152340 unknown :-1 s=CLOSED pgs=2605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:00.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.391+0000 7f21b1d2d700 1 --2- 172.21.15.16:0/3500061764 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f21ac15b790 0x7f21ac0c9540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:00.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.391+0000 7f21b1d2d700 1 -- 172.21.15.16:0/3500061764 >> 172.21.15.16:0/3500061764 conn(0x7f21ac0cc5e0 msgr2=0x7f21ac15a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:00.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.391+0000 7f21b1d2d700 1 -- 172.21.15.16:0/3500061764 shutdown_connections 2023-10-16T13:47:00.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.391+0000 7f21b1d2d700 1 -- 172.21.15.16:0/3500061764 wait complete. 2023-10-16T13:47:00.406 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T13:47:00.411 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:47:00.409+0000 7f0d0a270200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T13:47:00.449 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:00.447+0000 7fb67c01c200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T13:47:00.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.534+0000 7f2246dfd700 1 Processor -- start 2023-10-16T13:47:00.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.534+0000 7f2246dfd700 1 -- start start 2023-10-16T13:47:00.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.535+0000 7f2246dfd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f224015b2c0 0x7f2240161340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:00.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.535+0000 7f2246dfd700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f224006a7e0 0x7f224006ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:00.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.535+0000 7f2246dfd700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f224015ecb0 con 0x7f224006e570 2023-10-16T13:47:00.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.535+0000 7f2246dfd700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f224006b190 con 0x7f224015b2c0 2023-10-16T13:47:00.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.535+0000 7f2246dfd700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2240161880 con 0x7f224006a7e0 2023-10-16T13:47:00.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.535+0000 7f2244b99700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f224015b2c0 0x7f2240161340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:00.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.535+0000 7f2244b99700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f224015b2c0 0x7f2240161340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54558/0 (socket says 172.21.15.16:54558) 2023-10-16T13:47:00.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.535+0000 7f2244b99700 1 -- 172.21.15.16:0/2317808317 learned_addr learned my addr 172.21.15.16:0/2317808317 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:00.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.535+0000 7f224539a700 1 --2- 172.21.15.16:0/2317808317 >> v2:172.21.15.49:3300/0 conn(0x7f224006a7e0 0x7f224006ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:00.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.535+0000 7f2244b99700 1 -- 172.21.15.16:0/2317808317 >> v2:172.21.15.49:3300/0 conn(0x7f224006a7e0 msgr2=0x7f224006ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:00.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.535+0000 7f2244b99700 1 --2- 172.21.15.16:0/2317808317 >> v2:172.21.15.49:3300/0 conn(0x7f224006a7e0 0x7f224006ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:00.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.535+0000 7f2244b99700 1 -- 172.21.15.16:0/2317808317 >> v1:172.21.15.49:6789/0 conn(0x7f224006e570 legacy=0x7f224015e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:00.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.535+0000 7f2244b99700 1 -- 172.21.15.16:0/2317808317 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f224015b6a0 con 0x7f224015b2c0 2023-10-16T13:47:00.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.536+0000 7f2244b99700 1 --2- 172.21.15.16:0/2317808317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f224015b2c0 0x7f2240161340 secure :-1 s=READY pgs=2606 cs=0 l=1 rev1=1 crypto rx=0x7f224006b880 tx=0x7f2234009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c2abf5c24491bca server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:00.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.536+0000 7f223f7fe700 1 -- 172.21.15.16:0/2317808317 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2234011830 con 0x7f224015b2c0 2023-10-16T13:47:00.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.536+0000 7f223f7fe700 1 -- 172.21.15.16:0/2317808317 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2234011990 con 0x7f224015b2c0 2023-10-16T13:47:00.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.536+0000 7f223f7fe700 1 -- 172.21.15.16:0/2317808317 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2234011d20 con 0x7f224015b2c0 2023-10-16T13:47:00.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.536+0000 7f2246dfd700 1 -- 172.21.15.16:0/2317808317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f224015b2c0 msgr2=0x7f2240161340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:00.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.536+0000 7f2246dfd700 1 --2- 172.21.15.16:0/2317808317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f224015b2c0 0x7f2240161340 secure :-1 s=READY pgs=2606 cs=0 l=1 rev1=1 crypto rx=0x7f224006b880 tx=0x7f2234009bd0 comp rx=0 tx=0).stop 2023-10-16T13:47:00.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.536+0000 7f2246dfd700 1 -- 172.21.15.16:0/2317808317 shutdown_connections 2023-10-16T13:47:00.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.536+0000 7f2246dfd700 1 --2- 172.21.15.16:0/2317808317 >> v2:172.21.15.49:3300/0 conn(0x7f224006a7e0 0x7f224006ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:00.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.536+0000 7f2246dfd700 1 --2- 172.21.15.16:0/2317808317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f224015b2c0 0x7f2240161340 unknown :-1 s=CLOSED pgs=2606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:00.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.536+0000 7f2246dfd700 1 -- 172.21.15.16:0/2317808317 >> 172.21.15.16:0/2317808317 conn(0x7f22400b1f40 msgr2=0x7f22400b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:00.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.536+0000 7f2246dfd700 1 -- 172.21.15.16:0/2317808317 shutdown_connections 2023-10-16T13:47:00.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.536+0000 7f2246dfd700 1 -- 172.21.15.16:0/2317808317 wait complete. 2023-10-16T13:47:00.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.538+0000 7f2246dfd700 1 Processor -- start 2023-10-16T13:47:00.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.538+0000 7f2246dfd700 1 -- start start 2023-10-16T13:47:00.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.538+0000 7f2246dfd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f224006a7e0 0x7f224014cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:00.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.539+0000 7f2246dfd700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f224006e570 0x7f22400cd250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:00.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.539+0000 7f2246dfd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f224015b2c0 0x7f224015cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:00.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.539+0000 7f2246dfd700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f224014d0c0 con 0x7f224015b2c0 2023-10-16T13:47:00.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.539+0000 7f2246dfd700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22400ca2b0 con 0x7f224006a7e0 2023-10-16T13:47:00.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.539+0000 7f2246dfd700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22400c96a0 con 0x7f224006e570 2023-10-16T13:47:00.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.539+0000 7f224539a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f224015b2c0 0x7f224015cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:00.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.539+0000 7f2244b99700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f224006a7e0 0x7f224014cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:00.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.539+0000 7f2244b99700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f224006a7e0 0x7f224014cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:53624/0 (socket says 172.21.15.16:53624) 2023-10-16T13:47:00.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.539+0000 7f2244b99700 1 -- 172.21.15.16:0/1234026151 learned_addr learned my addr 172.21.15.16:0/1234026151 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:00.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.539+0000 7f224539a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f224015b2c0 0x7f224015cf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54566/0 (socket says 172.21.15.16:54566) 2023-10-16T13:47:00.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.539+0000 7f223ffff700 1 --2- 172.21.15.16:0/1234026151 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f224006e570 0x7f22400cd250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:00.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.539+0000 7f224539a700 1 -- 172.21.15.16:0/1234026151 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f224006e570 msgr2=0x7f22400cd250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:00.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.539+0000 7f224539a700 1 --2- 172.21.15.16:0/1234026151 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f224006e570 0x7f22400cd250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:00.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.539+0000 7f224539a700 1 -- 172.21.15.16:0/1234026151 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f224006a7e0 msgr2=0x7f224014cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:00.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.539+0000 7f224539a700 1 --2- 172.21.15.16:0/1234026151 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f224006a7e0 0x7f224014cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:00.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.539+0000 7f224539a700 1 -- 172.21.15.16:0/1234026151 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2234009550 con 0x7f224015b2c0 2023-10-16T13:47:00.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.539+0000 7f2244b99700 1 --2- 172.21.15.16:0/1234026151 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f224006a7e0 0x7f224014cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:47:00.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.539+0000 7f223ffff700 1 --2- 172.21.15.16:0/1234026151 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f224006e570 0x7f22400cd250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:47:00.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.539+0000 7f224539a700 1 --2- 172.21.15.16:0/1234026151 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f224015b2c0 0x7f224015cf10 secure :-1 s=READY pgs=2607 cs=0 l=1 rev1=1 crypto rx=0x7f223800eba0 tx=0x7f223800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:00.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.539+0000 7f223d7fa700 1 -- 172.21.15.16:0/1234026151 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2238009a80 con 0x7f224015b2c0 2023-10-16T13:47:00.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.539+0000 7f223d7fa700 1 -- 172.21.15.16:0/1234026151 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f223800f040 con 0x7f224015b2c0 2023-10-16T13:47:00.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.539+0000 7f2246dfd700 1 -- 172.21.15.16:0/1234026151 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22400c9980 con 0x7f224015b2c0 2023-10-16T13:47:00.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.540+0000 7f223d7fa700 1 -- 172.21.15.16:0/1234026151 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f223800cc40 con 0x7f224015b2c0 2023-10-16T13:47:00.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.540+0000 7f2246dfd700 1 -- 172.21.15.16:0/1234026151 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22400c9f40 con 0x7f224015b2c0 2023-10-16T13:47:00.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.541+0000 7f2246dfd700 1 -- 172.21.15.16:0/1234026151 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22400ca420 con 0x7f224015b2c0 2023-10-16T13:47:00.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.541+0000 7f223d7fa700 1 -- 172.21.15.16:0/1234026151 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 144) v1 ==== 89490+0+0 (secure 0 0 0) 0x7f2238014480 con 0x7f224015b2c0 2023-10-16T13:47:00.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.541+0000 7f223d7fa700 1 --2- 172.21.15.16:0/1234026151 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f222406b1d0 0x7f222406d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:00.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.542+0000 7f2244b99700 1 -- 172.21.15.16:0/1234026151 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f222406b1d0 msgr2=0x7f222406d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1506142504 2023-10-16T13:47:00.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.542+0000 7f223d7fa700 1 -- 172.21.15.16:0/1234026151 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(230..230 src has 1..230) v4 ==== 7365+0+0 (secure 0 0 0) 0x7f223808abc0 con 0x7f224015b2c0 2023-10-16T13:47:00.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.542+0000 7f2244b99700 1 --2- 172.21.15.16:0/1234026151 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f222406b1d0 0x7f222406d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:47:00.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.546+0000 7f223d7fa700 1 -- 172.21.15.16:0/1234026151 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2238055110 con 0x7f224015b2c0 2023-10-16T13:47:00.574 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:47:00.571+0000 7f0d0a270200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T13:47:00.574 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:00.571+0000 7fb67c01c200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T13:47:00.741 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:00.739+0000 7fb67c01c200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T13:47:00.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.743+0000 7f2244b99700 1 -- 172.21.15.16:0/1234026151 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f222406b1d0 msgr2=0x7f222406d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1506142504 2023-10-16T13:47:00.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.743+0000 7f2244b99700 1 --2- 172.21.15.16:0/1234026151 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f222406b1d0 0x7f222406d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:47:00.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.752+0000 7f2246dfd700 1 -- 172.21.15.16:0/1234026151 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f224004ebb0 con 0x7f224015b2c0 2023-10-16T13:47:00.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.752+0000 7f223d7fa700 1 -- 172.21.15.16:0/1234026151 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v230) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2238058730 con 0x7f224015b2c0 2023-10-16T13:47:00.755 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-10-16T13:47:00.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.755+0000 7f2246dfd700 1 -- 172.21.15.16:0/1234026151 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f222406b1d0 msgr2=0x7f222406d690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:47:00.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.755+0000 7f2246dfd700 1 --2- 172.21.15.16:0/1234026151 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f222406b1d0 0x7f222406d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:00.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.755+0000 7f2246dfd700 1 -- 172.21.15.16:0/1234026151 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f224015b2c0 msgr2=0x7f224015cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:00.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.755+0000 7f2246dfd700 1 --2- 172.21.15.16:0/1234026151 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f224015b2c0 0x7f224015cf10 secure :-1 s=READY pgs=2607 cs=0 l=1 rev1=1 crypto rx=0x7f223800eba0 tx=0x7f223800eeb0 comp rx=0 tx=0).stop 2023-10-16T13:47:00.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.755+0000 7f2246dfd700 1 -- 172.21.15.16:0/1234026151 shutdown_connections 2023-10-16T13:47:00.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.755+0000 7f2246dfd700 1 --2- 172.21.15.16:0/1234026151 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f222406b1d0 0x7f222406d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:00.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.755+0000 7f2246dfd700 1 --2- 172.21.15.16:0/1234026151 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f224015b2c0 0x7f224015cf10 unknown :-1 s=CLOSED pgs=2607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:00.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.755+0000 7f2246dfd700 1 --2- 172.21.15.16:0/1234026151 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f224006a7e0 0x7f224014cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:00.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.755+0000 7f2246dfd700 1 --2- 172.21.15.16:0/1234026151 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f224006e570 0x7f22400cd250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:00.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.755+0000 7f2246dfd700 1 -- 172.21.15.16:0/1234026151 >> 172.21.15.16:0/1234026151 conn(0x7f22400b1f40 msgr2=0x7f224015e4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:00.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.755+0000 7f2246dfd700 1 -- 172.21.15.16:0/1234026151 shutdown_connections 2023-10-16T13:47:00.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.755+0000 7f2246dfd700 1 -- 172.21.15.16:0/1234026151 wait complete. 2023-10-16T13:47:00.770 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T13:47:00.846 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:47:00.843+0000 7f0d0a270200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T13:47:00.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.942+0000 7fa3b480b700 1 Processor -- start 2023-10-16T13:47:00.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.942+0000 7fa3b480b700 1 -- start start 2023-10-16T13:47:00.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.943+0000 7fa3b480b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa3b00ce5d0 0x7fa3b00cc190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:00.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.943+0000 7fa3b480b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa3b015f060 0x7fa3b00cc6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:00.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.943+0000 7fa3b480b700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3b00b2440 con 0x7fa3b015ec80 2023-10-16T13:47:00.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.943+0000 7fa3b480b700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3b00b2c50 con 0x7fa3b015f060 2023-10-16T13:47:00.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.943+0000 7fa3b480b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b00d00e0 con 0x7fa3b00ce5d0 2023-10-16T13:47:00.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.943+0000 7fa3aed9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa3b015f060 0x7fa3b00cc6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:00.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.943+0000 7fa3ae59c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa3b015ec80 0x7fa3b00cdec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:43846/0 (socket says 172.21.15.16:43846) 2023-10-16T13:47:00.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.943+0000 7fa3aed9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa3b015f060 0x7fa3b00cc6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54576/0 (socket says 172.21.15.16:54576) 2023-10-16T13:47:00.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.943+0000 7fa3ae59c700 1 -- 172.21.15.16:0/1467644158 learned_addr learned my addr 172.21.15.16:0/1467644158 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:47:00.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.943+0000 7fa3add9b700 1 --2- 172.21.15.16:0/1467644158 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa3b00ce5d0 0x7fa3b00cc190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:00.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.943+0000 7fa3aed9d700 1 -- 172.21.15.16:0/1467644158 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa3b00ce5d0 msgr2=0x7fa3b00cc190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:00.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.943+0000 7fa3aed9d700 1 --2- 172.21.15.16:0/1467644158 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa3b00ce5d0 0x7fa3b00cc190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:00.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.943+0000 7fa3aed9d700 1 -- 172.21.15.16:0/1467644158 >> v1:172.21.15.16:6789/0 conn(0x7fa3b015ec80 legacy=0x7fa3b00cdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:00.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.943+0000 7fa3aed9d700 1 -- 172.21.15.16:0/1467644158 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3b00cef60 con 0x7fa3b015f060 2023-10-16T13:47:00.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.943+0000 7fa3ad59a700 1 -- 172.21.15.16:0/1467644158 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1564098850 0 0) 0x7fa3b00b2440 con 0x7fa3b015ec80 2023-10-16T13:47:00.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.943+0000 7fa3aed9d700 1 --2- 172.21.15.16:0/1467644158 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3b015f060 0x7fa3b00cc6d0 secure :-1 s=READY pgs=2609 cs=0 l=1 rev1=1 crypto rx=0x7fa3a8009d90 tx=0x7fa3a800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b0b8838980b9d68 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:00.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.943+0000 7fa3ad59a700 1 -- 172.21.15.16:0/1467644158 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a800e070 con 0x7fa3b015f060 2023-10-16T13:47:00.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.943+0000 7fa3ad59a700 1 -- 172.21.15.16:0/1467644158 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3a800bb20 con 0x7fa3b015f060 2023-10-16T13:47:00.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.944+0000 7fa3ad59a700 1 -- 172.21.15.16:0/1467644158 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a8012430 con 0x7fa3b015f060 2023-10-16T13:47:00.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.944+0000 7fa3b480b700 1 -- 172.21.15.16:0/1467644158 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3b015f060 msgr2=0x7fa3b00cc6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:00.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.944+0000 7fa3b480b700 1 --2- 172.21.15.16:0/1467644158 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3b015f060 0x7fa3b00cc6d0 secure :-1 s=READY pgs=2609 cs=0 l=1 rev1=1 crypto rx=0x7fa3a8009d90 tx=0x7fa3a800b330 comp rx=0 tx=0).stop 2023-10-16T13:47:00.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.944+0000 7fa3b480b700 1 -- 172.21.15.16:0/1467644158 shutdown_connections 2023-10-16T13:47:00.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.944+0000 7fa3b480b700 1 --2- 172.21.15.16:0/1467644158 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3b015f060 0x7fa3b00cc6d0 unknown :-1 s=CLOSED pgs=2609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:00.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.944+0000 7fa3b480b700 1 --2- 172.21.15.16:0/1467644158 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa3b00ce5d0 0x7fa3b00cc190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:00.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.944+0000 7fa3b480b700 1 -- 172.21.15.16:0/1467644158 >> 172.21.15.16:0/1467644158 conn(0x7fa3b00d15f0 msgr2=0x7fa3b00d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:00.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.944+0000 7fa3b480b700 1 -- 172.21.15.16:0/1467644158 shutdown_connections 2023-10-16T13:47:00.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.944+0000 7fa3b480b700 1 -- 172.21.15.16:0/1467644158 wait complete. 2023-10-16T13:47:00.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.946+0000 7fa3b480b700 1 Processor -- start 2023-10-16T13:47:00.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.946+0000 7fa3b480b700 1 -- start start 2023-10-16T13:47:00.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.946+0000 7fa3b480b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa3b00ce5d0 0x7fa3b00c7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:00.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.946+0000 7fa3b480b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3b015ec80 0x7fa3b015cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:00.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.946+0000 7fa3b480b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa3b015f060 0x7fa3b009d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:00.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.946+0000 7fa3b480b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b00c5b10 con 0x7fa3b015ec80 2023-10-16T13:47:00.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.946+0000 7fa3b480b700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b00c5dd0 con 0x7fa3b00ce5d0 2023-10-16T13:47:00.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.946+0000 7fa3b480b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b00c60b0 con 0x7fa3b015f060 2023-10-16T13:47:00.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.946+0000 7fa3add9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3b015ec80 0x7fa3b015cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:00.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.947+0000 7fa3add9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3b015ec80 0x7fa3b015cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54580/0 (socket says 172.21.15.16:54580) 2023-10-16T13:47:00.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.947+0000 7fa3add9b700 1 -- 172.21.15.16:0/2400119923 learned_addr learned my addr 172.21.15.16:0/2400119923 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:00.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.947+0000 7fa3aed9d700 1 --2- 172.21.15.16:0/2400119923 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa3b015f060 0x7fa3b009d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:00.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.947+0000 7fa3ae59c700 1 --2- 172.21.15.16:0/2400119923 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa3b00ce5d0 0x7fa3b00c7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:00.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.947+0000 7fa3add9b700 1 -- 172.21.15.16:0/2400119923 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa3b015f060 msgr2=0x7fa3b009d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:00.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.947+0000 7fa3add9b700 1 --2- 172.21.15.16:0/2400119923 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa3b015f060 0x7fa3b009d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:00.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.947+0000 7fa3add9b700 1 -- 172.21.15.16:0/2400119923 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa3b00ce5d0 msgr2=0x7fa3b00c7bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:00.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.947+0000 7fa3add9b700 1 --2- 172.21.15.16:0/2400119923 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa3b00ce5d0 0x7fa3b00c7bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:00.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.947+0000 7fa3add9b700 1 -- 172.21.15.16:0/2400119923 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3a8009a70 con 0x7fa3b015ec80 2023-10-16T13:47:00.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.947+0000 7fa3aed9d700 1 --2- 172.21.15.16:0/2400119923 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa3b015f060 0x7fa3b009d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:47:00.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.947+0000 7fa3add9b700 1 --2- 172.21.15.16:0/2400119923 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3b015ec80 0x7fa3b015cd40 secure :-1 s=READY pgs=2610 cs=0 l=1 rev1=1 crypto rx=0x7fa39800e9b0 tx=0x7fa39800ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:00.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.947+0000 7fa3a6ffd700 1 -- 172.21.15.16:0/2400119923 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa398009710 con 0x7fa3b015ec80 2023-10-16T13:47:00.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.947+0000 7fa3a6ffd700 1 -- 172.21.15.16:0/2400119923 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa39800ee50 con 0x7fa3b015ec80 2023-10-16T13:47:00.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.947+0000 7fa3b480b700 1 -- 172.21.15.16:0/2400119923 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3b00c6390 con 0x7fa3b015ec80 2023-10-16T13:47:00.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.947+0000 7fa3a6ffd700 1 -- 172.21.15.16:0/2400119923 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa39800ca50 con 0x7fa3b015ec80 2023-10-16T13:47:00.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.947+0000 7fa3b480b700 1 -- 172.21.15.16:0/2400119923 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3b00c8130 con 0x7fa3b015ec80 2023-10-16T13:47:00.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.949+0000 7fa3b480b700 1 -- 172.21.15.16:0/2400119923 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3b00c68b0 con 0x7fa3b015ec80 2023-10-16T13:47:00.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.949+0000 7fa3a6ffd700 1 -- 172.21.15.16:0/2400119923 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 144) v1 ==== 89490+0+0 (secure 0 0 0) 0x7fa39800cbb0 con 0x7fa3b015ec80 2023-10-16T13:47:00.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.949+0000 7fa3a6ffd700 1 --2- 172.21.15.16:0/2400119923 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7fa39006b1d0 0x7fa39006d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:00.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.949+0000 7fa3ae59c700 1 -- 172.21.15.16:0/2400119923 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7fa39006b1d0 msgr2=0x7fa39006d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1506142504 2023-10-16T13:47:00.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.949+0000 7fa3a6ffd700 1 -- 172.21.15.16:0/2400119923 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(230..230 src has 1..230) v4 ==== 7365+0+0 (secure 0 0 0) 0x7fa39808a090 con 0x7fa3b015ec80 2023-10-16T13:47:00.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.949+0000 7fa3ae59c700 1 --2- 172.21.15.16:0/2400119923 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7fa39006b1d0 0x7fa39006d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:47:00.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:00.954+0000 7fa3a6ffd700 1 -- 172.21.15.16:0/2400119923 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa3b00c68b0 con 0x7fa3b015ec80 2023-10-16T13:47:01.015 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:47:01.013+0000 7f0d0a270200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T13:47:01.027 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:01.025+0000 7fb67c01c200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T13:47:01.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.150+0000 7fa3ae59c700 1 -- 172.21.15.16:0/2400119923 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7fa39006b1d0 msgr2=0x7fa39006d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1506142504 2023-10-16T13:47:01.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.150+0000 7fa3ae59c700 1 --2- 172.21.15.16:0/2400119923 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7fa39006b1d0 0x7fa39006d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:47:01.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.156+0000 7fa3b480b700 1 -- 172.21.15.16:0/2400119923 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa3b00c68b0 con 0x7fa3b015ec80 2023-10-16T13:47:01.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.156+0000 7fa3a6ffd700 1 -- 172.21.15.16:0/2400119923 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v230) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa3b00c68b0 con 0x7fa3b015ec80 2023-10-16T13:47:01.159 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-10-16T13:47:01.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.158+0000 7fa3b480b700 1 -- 172.21.15.16:0/2400119923 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7fa39006b1d0 msgr2=0x7fa39006d690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:47:01.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.158+0000 7fa3b480b700 1 --2- 172.21.15.16:0/2400119923 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7fa39006b1d0 0x7fa39006d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:01.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.158+0000 7fa3b480b700 1 -- 172.21.15.16:0/2400119923 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3b015ec80 msgr2=0x7fa3b015cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:01.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.158+0000 7fa3b480b700 1 --2- 172.21.15.16:0/2400119923 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3b015ec80 0x7fa3b015cd40 secure :-1 s=READY pgs=2610 cs=0 l=1 rev1=1 crypto rx=0x7fa39800e9b0 tx=0x7fa39800ecc0 comp rx=0 tx=0).stop 2023-10-16T13:47:01.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.159+0000 7fa3b480b700 1 -- 172.21.15.16:0/2400119923 shutdown_connections 2023-10-16T13:47:01.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.159+0000 7fa3b480b700 1 --2- 172.21.15.16:0/2400119923 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7fa39006b1d0 0x7fa39006d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:01.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.159+0000 7fa3b480b700 1 --2- 172.21.15.16:0/2400119923 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa3b015f060 0x7fa3b009d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:01.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.159+0000 7fa3b480b700 1 --2- 172.21.15.16:0/2400119923 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa3b00ce5d0 0x7fa3b00c7bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:01.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.159+0000 7fa3b480b700 1 --2- 172.21.15.16:0/2400119923 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa3b015ec80 0x7fa3b015cd40 unknown :-1 s=CLOSED pgs=2610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:01.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.159+0000 7fa3b480b700 1 -- 172.21.15.16:0/2400119923 >> 172.21.15.16:0/2400119923 conn(0x7fa3b00d15f0 msgr2=0x7fa3b015de00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:01.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.159+0000 7fa3b480b700 1 -- 172.21.15.16:0/2400119923 shutdown_connections 2023-10-16T13:47:01.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.159+0000 7fa3b480b700 1 -- 172.21.15.16:0/2400119923 wait complete. 2023-10-16T13:47:01.173 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2023-10-16T13:47:01.204 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:01.202+0000 7fb67c01c200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T13:47:01.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.301+0000 7f9962ab7700 1 Processor -- start 2023-10-16T13:47:01.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.301+0000 7f9962ab7700 1 -- start start 2023-10-16T13:47:01.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.302+0000 7f9962ab7700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f995c15ec80 0x7f995c0cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:01.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.302+0000 7f9962ab7700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f995c0d2110 0x7f995c0cd1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:01.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.302+0000 7f9962ab7700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f995c0b2440 con 0x7f995c15f060 2023-10-16T13:47:01.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.302+0000 7f9962ab7700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f995c0b2c50 con 0x7f995c0d2110 2023-10-16T13:47:01.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.302+0000 7f9962ab7700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f995c0d00e0 con 0x7f995c15ec80 2023-10-16T13:47:01.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.303+0000 7f9960853700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f995c15ec80 0x7f995c0cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:01.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.303+0000 7f9961054700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f995c0d2110 0x7f995c0cd1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:01.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.303+0000 7f995bfff700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7f995c15f060 0x7f995c0b4440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:51044/0 (socket says 172.21.15.16:51044) 2023-10-16T13:47:01.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.303+0000 7f995bfff700 1 -- 172.21.15.16:0/3808846824 learned_addr learned my addr 172.21.15.16:0/3808846824 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:47:01.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.303+0000 7f9960853700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f995c15ec80 0x7f995c0cdec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:42846/0 (socket says 172.21.15.16:42846) 2023-10-16T13:47:01.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.303+0000 7f995b7fe700 1 -- 172.21.15.16:0/3808846824 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1825678800 0 0) 0x7f995c0b2440 con 0x7f995c15f060 2023-10-16T13:47:01.308 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.303+0000 7f995b7fe700 1 -- 172.21.15.16:0/3808846824 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f994c0035b0 con 0x7f995c15f060 2023-10-16T13:47:01.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.303+0000 7f9960853700 1 -- 172.21.15.16:0/3808846824 >> v2:172.21.15.49:3300/0 conn(0x7f995c0d2110 msgr2=0x7f995c0cd1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:01.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.303+0000 7f9960853700 1 --2- 172.21.15.16:0/3808846824 >> v2:172.21.15.49:3300/0 conn(0x7f995c0d2110 0x7f995c0cd1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:01.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.303+0000 7f9960853700 1 -- 172.21.15.16:0/3808846824 >> v1:172.21.15.49:6789/0 conn(0x7f995c15f060 legacy=0x7f995c0b4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:01.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.303+0000 7f9960853700 1 -- 172.21.15.16:0/3808846824 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f995c0cef60 con 0x7f995c15ec80 2023-10-16T13:47:01.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.303+0000 7f9961054700 1 --2- 172.21.15.16:0/3808846824 >> v2:172.21.15.49:3300/0 conn(0x7f995c0d2110 0x7f995c0cd1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:47:01.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.303+0000 7f9960853700 1 --2- 172.21.15.16:0/3808846824 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f995c15ec80 0x7f995c0cdec0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f9950009840 tx=0x7f9950009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4aab6bfc4dd07e08 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:01.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.303+0000 7f995b7fe700 1 -- 172.21.15.16:0/3808846824 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9950011830 con 0x7f995c15ec80 2023-10-16T13:47:01.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.303+0000 7f995b7fe700 1 -- 172.21.15.16:0/3808846824 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9950011990 con 0x7f995c15ec80 2023-10-16T13:47:01.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.303+0000 7f995b7fe700 1 -- 172.21.15.16:0/3808846824 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9950011d20 con 0x7f995c15ec80 2023-10-16T13:47:01.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.304+0000 7f9962ab7700 1 -- 172.21.15.16:0/3808846824 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f995c15ec80 msgr2=0x7f995c0cdec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:01.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.304+0000 7f9962ab7700 1 --2- 172.21.15.16:0/3808846824 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f995c15ec80 0x7f995c0cdec0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f9950009840 tx=0x7f9950009b50 comp rx=0 tx=0).stop 2023-10-16T13:47:01.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.304+0000 7f9962ab7700 1 -- 172.21.15.16:0/3808846824 shutdown_connections 2023-10-16T13:47:01.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.304+0000 7f9962ab7700 1 --2- 172.21.15.16:0/3808846824 >> v2:172.21.15.49:3300/0 conn(0x7f995c0d2110 0x7f995c0cd1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:01.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.304+0000 7f9962ab7700 1 --2- 172.21.15.16:0/3808846824 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f995c15ec80 0x7f995c0cdec0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:01.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.304+0000 7f9962ab7700 1 -- 172.21.15.16:0/3808846824 >> 172.21.15.16:0/3808846824 conn(0x7f995c0d15f0 msgr2=0x7f995c0d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:01.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.304+0000 7f9962ab7700 1 -- 172.21.15.16:0/3808846824 shutdown_connections 2023-10-16T13:47:01.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.304+0000 7f9962ab7700 1 -- 172.21.15.16:0/3808846824 wait complete. 2023-10-16T13:47:01.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.306+0000 7f9962ab7700 1 Processor -- start 2023-10-16T13:47:01.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.306+0000 7f9962ab7700 1 -- start start 2023-10-16T13:47:01.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.307+0000 7f9962ab7700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f995c0d2110 0x7f995c0b61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:01.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.307+0000 7f9962ab7700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f995c15ec80 0x7f995c159e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:01.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.307+0000 7f9962ab7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f995c15f060 0x7f995c0c9d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:01.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.307+0000 7f9962ab7700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f995c0c92b0 con 0x7f995c15f060 2023-10-16T13:47:01.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.307+0000 7f9962ab7700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f995c1503b0 con 0x7f995c15ec80 2023-10-16T13:47:01.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.307+0000 7f9962ab7700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f995c0c86c0 con 0x7f995c0d2110 2023-10-16T13:47:01.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.307+0000 7f9961054700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f995c15f060 0x7f995c0c9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:01.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.307+0000 7f995bfff700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f995c15ec80 0x7f995c159e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:01.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.307+0000 7f9961054700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f995c15f060 0x7f995c0c9d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54598/0 (socket says 172.21.15.16:54598) 2023-10-16T13:47:01.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.307+0000 7f9960853700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f995c0d2110 0x7f995c0b61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:01.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.307+0000 7f9961054700 1 -- 172.21.15.16:0/753362524 learned_addr learned my addr 172.21.15.16:0/753362524 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:01.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.307+0000 7f9961054700 1 -- 172.21.15.16:0/753362524 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f995c0d2110 msgr2=0x7f995c0b61d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:01.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.307+0000 7f9961054700 1 --2- 172.21.15.16:0/753362524 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f995c0d2110 0x7f995c0b61d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:01.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.307+0000 7f9961054700 1 -- 172.21.15.16:0/753362524 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f995c15ec80 msgr2=0x7f995c159e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:01.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.307+0000 7f9961054700 1 --2- 172.21.15.16:0/753362524 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f995c15ec80 0x7f995c159e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:01.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.307+0000 7f9961054700 1 -- 172.21.15.16:0/753362524 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9954009a70 con 0x7f995c15f060 2023-10-16T13:47:01.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.307+0000 7f9960853700 1 --2- 172.21.15.16:0/753362524 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f995c0d2110 0x7f995c0b61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:47:01.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.307+0000 7f9961054700 1 --2- 172.21.15.16:0/753362524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f995c15f060 0x7f995c0c9d30 secure :-1 s=READY pgs=2611 cs=0 l=1 rev1=1 crypto rx=0x7f9954009f00 tx=0x7f995400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:01.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.308+0000 7f99597fa700 1 -- 172.21.15.16:0/753362524 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9954009d70 con 0x7f995c15f060 2023-10-16T13:47:01.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.308+0000 7f99597fa700 1 -- 172.21.15.16:0/753362524 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f995400f040 con 0x7f995c15f060 2023-10-16T13:47:01.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.308+0000 7f9962ab7700 1 -- 172.21.15.16:0/753362524 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99500094a0 con 0x7f995c15f060 2023-10-16T13:47:01.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.308+0000 7f99597fa700 1 -- 172.21.15.16:0/753362524 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f995400cd10 con 0x7f995c15f060 2023-10-16T13:47:01.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.308+0000 7f9962ab7700 1 -- 172.21.15.16:0/753362524 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f995c0c8bf0 con 0x7f995c15f060 2023-10-16T13:47:01.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.309+0000 7f9962ab7700 1 -- 172.21.15.16:0/753362524 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f995c0c93f0 con 0x7f995c15f060 2023-10-16T13:47:01.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.310+0000 7f99597fa700 1 -- 172.21.15.16:0/753362524 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 144) v1 ==== 89490+0+0 (secure 0 0 0) 0x7f9954003a20 con 0x7f995c15f060 2023-10-16T13:47:01.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.311+0000 7f99597fa700 1 --2- 172.21.15.16:0/753362524 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f994006b200 0x7f994006d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:01.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.311+0000 7f9960853700 1 -- 172.21.15.16:0/753362524 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f994006b200 msgr2=0x7f994006d6c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1506142504 2023-10-16T13:47:01.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.311+0000 7f99597fa700 1 -- 172.21.15.16:0/753362524 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(230..230 src has 1..230) v4 ==== 7365+0+0 (secure 0 0 0) 0x7f995408b150 con 0x7f995c15f060 2023-10-16T13:47:01.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.311+0000 7f9960853700 1 --2- 172.21.15.16:0/753362524 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f994006b200 0x7f994006d6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:47:01.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.314+0000 7f99597fa700 1 -- 172.21.15.16:0/753362524 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9954055690 con 0x7f995c15f060 2023-10-16T13:47:01.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.512+0000 7f9960853700 1 -- 172.21.15.16:0/753362524 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f994006b200 msgr2=0x7f994006d6c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1506142504 2023-10-16T13:47:01.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.512+0000 7f9960853700 1 --2- 172.21.15.16:0/753362524 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f994006b200 0x7f994006d6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:47:01.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.518+0000 7f9962ab7700 1 -- 172.21.15.16:0/753362524 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f995c0c93f0 con 0x7f995c15f060 2023-10-16T13:47:01.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.518+0000 7f99597fa700 1 -- 172.21.15.16:0/753362524 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v230) v1 ==== 107+0+11 (secure 0 0 0) 0x7f995c0c93f0 con 0x7f995c15f060 2023-10-16T13:47:01.521 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-10-16T13:47:01.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.520+0000 7f9962ab7700 1 -- 172.21.15.16:0/753362524 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f994006b200 msgr2=0x7f994006d6c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:47:01.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.520+0000 7f9962ab7700 1 --2- 172.21.15.16:0/753362524 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f994006b200 0x7f994006d6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:01.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.520+0000 7f9962ab7700 1 -- 172.21.15.16:0/753362524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f995c15f060 msgr2=0x7f995c0c9d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:01.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.520+0000 7f9962ab7700 1 --2- 172.21.15.16:0/753362524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f995c15f060 0x7f995c0c9d30 secure :-1 s=READY pgs=2611 cs=0 l=1 rev1=1 crypto rx=0x7f9954009f00 tx=0x7f995400c3a0 comp rx=0 tx=0).stop 2023-10-16T13:47:01.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.520+0000 7f9962ab7700 1 -- 172.21.15.16:0/753362524 shutdown_connections 2023-10-16T13:47:01.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.520+0000 7f9962ab7700 1 --2- 172.21.15.16:0/753362524 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f994006b200 0x7f994006d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:01.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.520+0000 7f9962ab7700 1 --2- 172.21.15.16:0/753362524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f995c15f060 0x7f995c0c9d30 unknown :-1 s=CLOSED pgs=2611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:01.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.520+0000 7f9962ab7700 1 --2- 172.21.15.16:0/753362524 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f995c0d2110 0x7f995c0b61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:01.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.520+0000 7f9962ab7700 1 --2- 172.21.15.16:0/753362524 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f995c15ec80 0x7f995c159e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:01.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.520+0000 7f9962ab7700 1 -- 172.21.15.16:0/753362524 >> 172.21.15.16:0/753362524 conn(0x7f995c0d15f0 msgr2=0x7f995c15c8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:01.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.520+0000 7f9962ab7700 1 -- 172.21.15.16:0/753362524 shutdown_connections 2023-10-16T13:47:01.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.520+0000 7f9962ab7700 1 -- 172.21.15.16:0/753362524 wait complete. 2023-10-16T13:47:01.535 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2023-10-16T13:47:01.569 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:47:01.567+0000 7f0d0a270200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T13:47:01.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.704+0000 7f1e46cfe700 1 Processor -- start 2023-10-16T13:47:01.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.704+0000 7f1e46cfe700 1 -- start start 2023-10-16T13:47:01.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.705+0000 7f1e46cfe700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1e4015b070 0x7f1e4015b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:01.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.705+0000 7f1e46cfe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1e4006e170 0x7f1e400cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:01.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.705+0000 7f1e46cfe700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e400cef70 con 0x7f1e400cb880 2023-10-16T13:47:01.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.705+0000 7f1e46cfe700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e400ce360 con 0x7f1e4006e170 2023-10-16T13:47:01.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.705+0000 7f1e46cfe700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e400ce640 con 0x7f1e4015b070 2023-10-16T13:47:01.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.705+0000 7f1e4529b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1e4006e170 0x7f1e400cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:01.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.705+0000 7f1e3ffff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f1e400cb880 0x7f1e4006da60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:43880/0 (socket says 172.21.15.16:43880) 2023-10-16T13:47:01.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.705+0000 7f1e3ffff700 1 -- 172.21.15.16:0/3851690679 learned_addr learned my addr 172.21.15.16:0/3851690679 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:47:01.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.705+0000 7f1e44a9a700 1 --2- 172.21.15.16:0/3851690679 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1e4015b070 0x7f1e4015b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:01.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.705+0000 7f1e4529b700 1 -- 172.21.15.16:0/3851690679 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1e4015b070 msgr2=0x7f1e4015b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:01.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.705+0000 7f1e4529b700 1 --2- 172.21.15.16:0/3851690679 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1e4015b070 0x7f1e4015b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:01.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.705+0000 7f1e4529b700 1 -- 172.21.15.16:0/3851690679 >> v1:172.21.15.16:6789/0 conn(0x7f1e400cb880 legacy=0x7f1e4006da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:01.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.705+0000 7f1e4529b700 1 -- 172.21.15.16:0/3851690679 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e400ce920 con 0x7f1e4006e170 2023-10-16T13:47:01.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.705+0000 7f1e3f7fe700 1 -- 172.21.15.16:0/3851690679 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 674428612 0 0) 0x7f1e400cef70 con 0x7f1e400cb880 2023-10-16T13:47:01.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.706+0000 7f1e4529b700 1 --2- 172.21.15.16:0/3851690679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e4006e170 0x7f1e400cbd00 secure :-1 s=READY pgs=2613 cs=0 l=1 rev1=1 crypto rx=0x7f1e38009d90 tx=0x7f1e3800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78324b13faf6b351 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:01.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.706+0000 7f1e3f7fe700 1 -- 172.21.15.16:0/3851690679 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e3800e070 con 0x7f1e4006e170 2023-10-16T13:47:01.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.706+0000 7f1e3f7fe700 1 -- 172.21.15.16:0/3851690679 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e3800bb20 con 0x7f1e4006e170 2023-10-16T13:47:01.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.706+0000 7f1e3f7fe700 1 -- 172.21.15.16:0/3851690679 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e38012430 con 0x7f1e4006e170 2023-10-16T13:47:01.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.706+0000 7f1e46cfe700 1 -- 172.21.15.16:0/3851690679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e4006e170 msgr2=0x7f1e400cbd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:01.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.706+0000 7f1e46cfe700 1 --2- 172.21.15.16:0/3851690679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e4006e170 0x7f1e400cbd00 secure :-1 s=READY pgs=2613 cs=0 l=1 rev1=1 crypto rx=0x7f1e38009d90 tx=0x7f1e3800b330 comp rx=0 tx=0).stop 2023-10-16T13:47:01.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.706+0000 7f1e46cfe700 1 -- 172.21.15.16:0/3851690679 shutdown_connections 2023-10-16T13:47:01.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.706+0000 7f1e46cfe700 1 --2- 172.21.15.16:0/3851690679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e4006e170 0x7f1e400cbd00 unknown :-1 s=CLOSED pgs=2613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:01.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.706+0000 7f1e46cfe700 1 --2- 172.21.15.16:0/3851690679 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1e4015b070 0x7f1e4015b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:01.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.706+0000 7f1e46cfe700 1 -- 172.21.15.16:0/3851690679 >> 172.21.15.16:0/3851690679 conn(0x7f1e400b1f40 msgr2=0x7f1e400b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:01.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.707+0000 7f1e46cfe700 1 -- 172.21.15.16:0/3851690679 shutdown_connections 2023-10-16T13:47:01.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.707+0000 7f1e46cfe700 1 -- 172.21.15.16:0/3851690679 wait complete. 2023-10-16T13:47:01.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.708+0000 7f1e46cfe700 1 Processor -- start 2023-10-16T13:47:01.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.708+0000 7f1e46cfe700 1 -- start start 2023-10-16T13:47:01.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.709+0000 7f1e46cfe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1e4006e170 0x7f1e4014a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:01.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.709+0000 7f1e46cfe700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1e400cb880 0x7f1e4015d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:01.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.709+0000 7f1e46cfe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e4015b070 0x7f1e400b57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:01.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.709+0000 7f1e46cfe700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e401573e0 con 0x7f1e4015b070 2023-10-16T13:47:01.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.709+0000 7f1e46cfe700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e4014a6d0 con 0x7f1e4006e170 2023-10-16T13:47:01.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.709+0000 7f1e46cfe700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e40157800 con 0x7f1e400cb880 2023-10-16T13:47:01.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.709+0000 7f1e4529b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e4015b070 0x7f1e400b57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:01.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.709+0000 7f1e4529b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e4015b070 0x7f1e400b57c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54614/0 (socket says 172.21.15.16:54614) 2023-10-16T13:47:01.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.709+0000 7f1e4529b700 1 -- 172.21.15.16:0/455618829 learned_addr learned my addr 172.21.15.16:0/455618829 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:01.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.709+0000 7f1e3ffff700 1 --2- 172.21.15.16:0/455618829 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1e400cb880 0x7f1e4015d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:01.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.709+0000 7f1e44a9a700 1 --2- 172.21.15.16:0/455618829 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1e4006e170 0x7f1e4014a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:01.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.709+0000 7f1e4529b700 1 -- 172.21.15.16:0/455618829 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1e400cb880 msgr2=0x7f1e4015d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:01.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.709+0000 7f1e4529b700 1 --2- 172.21.15.16:0/455618829 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1e400cb880 0x7f1e4015d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:01.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.709+0000 7f1e4529b700 1 -- 172.21.15.16:0/455618829 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1e4006e170 msgr2=0x7f1e4014a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:01.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.709+0000 7f1e4529b700 1 --2- 172.21.15.16:0/455618829 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1e4006e170 0x7f1e4014a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:01.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.709+0000 7f1e4529b700 1 -- 172.21.15.16:0/455618829 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e38009a70 con 0x7f1e4015b070 2023-10-16T13:47:01.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.709+0000 7f1e44a9a700 1 --2- 172.21.15.16:0/455618829 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1e4006e170 0x7f1e4014a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:47:01.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.709+0000 7f1e4529b700 1 --2- 172.21.15.16:0/455618829 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e4015b070 0x7f1e400b57c0 secure :-1 s=READY pgs=2614 cs=0 l=1 rev1=1 crypto rx=0x7f1e40070620 tx=0x7f1e38004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:01.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.709+0000 7f1e3d7fa700 1 -- 172.21.15.16:0/455618829 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e38012430 con 0x7f1e4015b070 2023-10-16T13:47:01.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.710+0000 7f1e46cfe700 1 -- 172.21.15.16:0/455618829 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e400c7930 con 0x7f1e4015b070 2023-10-16T13:47:01.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.710+0000 7f1e3d7fa700 1 -- 172.21.15.16:0/455618829 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e38020070 con 0x7f1e4015b070 2023-10-16T13:47:01.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.710+0000 7f1e3d7fa700 1 -- 172.21.15.16:0/455618829 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e38004280 con 0x7f1e4015b070 2023-10-16T13:47:01.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.710+0000 7f1e46cfe700 1 -- 172.21.15.16:0/455618829 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e40157000 con 0x7f1e4015b070 2023-10-16T13:47:01.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.710+0000 7f1e46cfe700 1 -- 172.21.15.16:0/455618829 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e40156b90 con 0x7f1e4015b070 2023-10-16T13:47:01.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.711+0000 7f1e3d7fa700 1 -- 172.21.15.16:0/455618829 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 144) v1 ==== 89490+0+0 (secure 0 0 0) 0x7f1e380043e0 con 0x7f1e4015b070 2023-10-16T13:47:01.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.711+0000 7f1e3d7fa700 1 --2- 172.21.15.16:0/455618829 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f1e2406b1d0 0x7f1e2406d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:01.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.712+0000 7f1e44a9a700 1 -- 172.21.15.16:0/455618829 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f1e2406b1d0 msgr2=0x7f1e2406d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1506142504 2023-10-16T13:47:01.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.712+0000 7f1e3d7fa700 1 -- 172.21.15.16:0/455618829 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(230..230 src has 1..230) v4 ==== 7365+0+0 (secure 0 0 0) 0x7f1e38022070 con 0x7f1e4015b070 2023-10-16T13:47:01.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.712+0000 7f1e44a9a700 1 --2- 172.21.15.16:0/455618829 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f1e2406b1d0 0x7f1e2406d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:47:01.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.720+0000 7f1e3d7fa700 1 -- 172.21.15.16:0/455618829 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1e38092080 con 0x7f1e4015b070 2023-10-16T13:47:01.790 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:01.788+0000 7fb67c01c200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T13:47:01.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.809+0000 7f1e3d7fa700 1 -- 172.21.15.16:0/455618829 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 145) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f1e380568c0 con 0x7f1e4015b070 2023-10-16T13:47:01.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.809+0000 7f1e3d7fa700 1 -- 172.21.15.16:0/455618829 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f1e2406b1d0 msgr2=0x7f1e2406d690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:47:01.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.809+0000 7f1e3d7fa700 1 --2- 172.21.15.16:0/455618829 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f1e2406b1d0 0x7f1e2406d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:01.881 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:01.878+0000 7fb5f4802700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:47:01.882 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:01.878+0000 7fb5f4802700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:47:01.882 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:01.878+0000 7fb5f4802700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:47:01.882 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:01.878+0000 7fb5f7808700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:47:01.882 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:01.878+0000 7fb5f4802700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:47:01.882 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:01.878+0000 7fb5f4802700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:47:01.883 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:01.878+0000 7fb5f7808700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:47:01.883 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:01.878+0000 7fb5f7808700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:47:01.883 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:01.878+0000 7fb5f7808700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:47:01.883 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:01.878+0000 7fb5f7808700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:47:01.890 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:01.887+0000 7fb5f4802700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:47:01.890 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:01.887+0000 7fb5f4802700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:47:01.890 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:01.887+0000 7fb5f4802700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:47:01.890 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:01.887+0000 7fb5f4802700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:47:01.890 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:01.887+0000 7fb5f4802700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:47:01.891 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:01.888+0000 7fb5f7808700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:47:01.891 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:01.888+0000 7fb5f7808700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:47:01.891 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:01.888+0000 7fb5f7808700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:47:01.891 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:01.888+0000 7fb5f7808700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:47:01.892 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:01.888+0000 7fb5f7808700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:47:01.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.946+0000 7f1e46cfe700 1 -- 172.21.15.16:0/455618829 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) v1 -- 0x7f1e40157520 con 0x7f1e4015b070 2023-10-16T13:47:01.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.947+0000 7f1e3d7fa700 1 -- 172.21.15.16:0/455618829 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v231) v1 ==== 103+0+11 (secure 0 0 0) 0x7f1e3802c020 con 0x7f1e4015b070 2023-10-16T13:47:01.949 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-10-16T13:47:01.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.949+0000 7f1e46cfe700 1 -- 172.21.15.16:0/455618829 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e4015b070 msgr2=0x7f1e400b57c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:01.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.949+0000 7f1e46cfe700 1 --2- 172.21.15.16:0/455618829 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e4015b070 0x7f1e400b57c0 secure :-1 s=READY pgs=2614 cs=0 l=1 rev1=1 crypto rx=0x7f1e40070620 tx=0x7f1e38004ad0 comp rx=0 tx=0).stop 2023-10-16T13:47:01.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.949+0000 7f1e46cfe700 1 -- 172.21.15.16:0/455618829 shutdown_connections 2023-10-16T13:47:01.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.949+0000 7f1e46cfe700 1 --2- 172.21.15.16:0/455618829 >> [v2:172.21.15.16:6832/1506142504,v1:172.21.15.16:6833/1506142504] conn(0x7f1e2406b1d0 0x7f1e2406d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:01.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.949+0000 7f1e46cfe700 1 --2- 172.21.15.16:0/455618829 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e4015b070 0x7f1e400b57c0 unknown :-1 s=CLOSED pgs=2614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:01.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.949+0000 7f1e46cfe700 1 --2- 172.21.15.16:0/455618829 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1e4006e170 0x7f1e4014a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:01.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.949+0000 7f1e46cfe700 1 --2- 172.21.15.16:0/455618829 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1e400cb880 0x7f1e4015d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:01.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.949+0000 7f1e46cfe700 1 -- 172.21.15.16:0/455618829 >> 172.21.15.16:0/455618829 conn(0x7f1e400b1f40 msgr2=0x7f1e400d0c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:01.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.949+0000 7f1e46cfe700 1 -- 172.21.15.16:0/455618829 shutdown_connections 2023-10-16T13:47:01.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:01.949+0000 7f1e46cfe700 1 -- 172.21.15.16:0/455618829 wait complete. 2023-10-16T13:47:01.964 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:47:02.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.087+0000 7ffb0995e700 1 Processor -- start 2023-10-16T13:47:02.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.087+0000 7ffb0995e700 1 -- start start 2023-10-16T13:47:02.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.088+0000 7ffb0995e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ffb040d41c0 0x7ffb040d0040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:02.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.088+0000 7ffb0995e700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7ffb040d0580 0x7ffb0406c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:02.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.088+0000 7ffb0995e700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb0406ba00 con 0x7ffb0415d8d0 2023-10-16T13:47:02.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.088+0000 7ffb0995e700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb040d0a10 con 0x7ffb040d41c0 2023-10-16T13:47:02.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.088+0000 7ffb0995e700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb040d26d0 con 0x7ffb040d0580 2023-10-16T13:47:02.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.088+0000 7ffb027fc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ffb040d41c0 0x7ffb040d0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:02.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.088+0000 7ffb02ffd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ffb0415d8d0 0x7ffb040d1c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:43992/0 (socket says 172.21.15.16:43992) 2023-10-16T13:47:02.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.088+0000 7ffb02ffd700 1 -- 172.21.15.16:0/1860889105 learned_addr learned my addr 172.21.15.16:0/1860889105 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:47:02.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.088+0000 7ffb037fe700 1 --2- 172.21.15.16:0/1860889105 >> v2:172.21.15.49:3300/0 conn(0x7ffb040d0580 0x7ffb0406c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:02.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.088+0000 7ffb027fc700 1 -- 172.21.15.16:0/1860889105 >> v2:172.21.15.49:3300/0 conn(0x7ffb040d0580 msgr2=0x7ffb0406c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:02.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.088+0000 7ffb027fc700 1 --2- 172.21.15.16:0/1860889105 >> v2:172.21.15.49:3300/0 conn(0x7ffb040d0580 0x7ffb0406c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:02.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.088+0000 7ffb027fc700 1 -- 172.21.15.16:0/1860889105 >> v1:172.21.15.16:6789/0 conn(0x7ffb0415d8d0 legacy=0x7ffb040d1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:02.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.088+0000 7ffb027fc700 1 -- 172.21.15.16:0/1860889105 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb040d23f0 con 0x7ffb040d41c0 2023-10-16T13:47:02.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.088+0000 7ffb01ffb700 1 -- 172.21.15.16:0/1860889105 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 407929279 0 0) 0x7ffb0406ba00 con 0x7ffb0415d8d0 2023-10-16T13:47:02.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.088+0000 7ffb037fe700 1 --2- 172.21.15.16:0/1860889105 >> v2:172.21.15.49:3300/0 conn(0x7ffb040d0580 0x7ffb0406c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:47:02.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.088+0000 7ffb027fc700 1 --2- 172.21.15.16:0/1860889105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb040d41c0 0x7ffb040d0040 secure :-1 s=READY pgs=2630 cs=0 l=1 rev1=1 crypto rx=0x7ffb0406b2d0 tx=0x7ffaf4009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d4a6c872d0d25c7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:02.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.089+0000 7ffb01ffb700 1 -- 172.21.15.16:0/1860889105 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf4011830 con 0x7ffb040d41c0 2023-10-16T13:47:02.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.089+0000 7ffb01ffb700 1 -- 172.21.15.16:0/1860889105 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaf4011990 con 0x7ffb040d41c0 2023-10-16T13:47:02.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.089+0000 7ffb01ffb700 1 -- 172.21.15.16:0/1860889105 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf4011d20 con 0x7ffb040d41c0 2023-10-16T13:47:02.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.089+0000 7ffb0995e700 1 -- 172.21.15.16:0/1860889105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb040d41c0 msgr2=0x7ffb040d0040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:02.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.089+0000 7ffb0995e700 1 --2- 172.21.15.16:0/1860889105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb040d41c0 0x7ffb040d0040 secure :-1 s=READY pgs=2630 cs=0 l=1 rev1=1 crypto rx=0x7ffb0406b2d0 tx=0x7ffaf4009b20 comp rx=0 tx=0).stop 2023-10-16T13:47:02.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.089+0000 7ffb0995e700 1 -- 172.21.15.16:0/1860889105 shutdown_connections 2023-10-16T13:47:02.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.089+0000 7ffb0995e700 1 --2- 172.21.15.16:0/1860889105 >> v2:172.21.15.49:3300/0 conn(0x7ffb040d0580 0x7ffb0406c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:02.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.089+0000 7ffb0995e700 1 --2- 172.21.15.16:0/1860889105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb040d41c0 0x7ffb040d0040 unknown :-1 s=CLOSED pgs=2630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:02.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.089+0000 7ffb0995e700 1 -- 172.21.15.16:0/1860889105 >> 172.21.15.16:0/1860889105 conn(0x7ffb040d3630 msgr2=0x7ffb040d3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:02.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.089+0000 7ffb0995e700 1 -- 172.21.15.16:0/1860889105 shutdown_connections 2023-10-16T13:47:02.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.089+0000 7ffb0995e700 1 -- 172.21.15.16:0/1860889105 wait complete. 2023-10-16T13:47:02.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.090+0000 7ffb0995e700 1 Processor -- start 2023-10-16T13:47:02.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.091+0000 7ffb0995e700 1 -- start start 2023-10-16T13:47:02.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.091+0000 7ffb0995e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ffb040d0580 0x7ffb0414e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:02.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.091+0000 7ffb0995e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb040d41c0 0x7ffb040ccd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:02.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.091+0000 7ffb0995e700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ffb0415d8d0 0x7ffb040cd250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:02.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.091+0000 7ffb0995e700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb040c7f90 con 0x7ffb040d41c0 2023-10-16T13:47:02.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.091+0000 7ffb0995e700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb040c8270 con 0x7ffb0415d8d0 2023-10-16T13:47:02.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.091+0000 7ffb0995e700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb040c8550 con 0x7ffb040d0580 2023-10-16T13:47:02.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.091+0000 7ffb027fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb040d41c0 0x7ffb040ccd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:02.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.091+0000 7ffb027fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb040d41c0 0x7ffb040ccd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54726/0 (socket says 172.21.15.16:54726) 2023-10-16T13:47:02.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.091+0000 7ffb027fc700 1 -- 172.21.15.16:0/3297932782 learned_addr learned my addr 172.21.15.16:0/3297932782 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:02.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.092+0000 7ffb02ffd700 1 --2- 172.21.15.16:0/3297932782 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ffb040d0580 0x7ffb0414e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:02.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.092+0000 7ffb037fe700 1 --2- 172.21.15.16:0/3297932782 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ffb0415d8d0 0x7ffb040cd250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:02.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.092+0000 7ffb027fc700 1 -- 172.21.15.16:0/3297932782 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ffb040d0580 msgr2=0x7ffb0414e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:02.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.092+0000 7ffb027fc700 1 --2- 172.21.15.16:0/3297932782 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ffb040d0580 0x7ffb0414e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:02.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.092+0000 7ffb027fc700 1 -- 172.21.15.16:0/3297932782 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ffb0415d8d0 msgr2=0x7ffb040cd250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:02.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.092+0000 7ffb027fc700 1 --2- 172.21.15.16:0/3297932782 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ffb0415d8d0 0x7ffb040cd250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:02.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.092+0000 7ffb027fc700 1 -- 172.21.15.16:0/3297932782 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaf40094a0 con 0x7ffb040d41c0 2023-10-16T13:47:02.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.092+0000 7ffb037fe700 1 --2- 172.21.15.16:0/3297932782 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ffb0415d8d0 0x7ffb040cd250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:47:02.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.092+0000 7ffb02ffd700 1 --2- 172.21.15.16:0/3297932782 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ffb040d0580 0x7ffb0414e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:47:02.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.092+0000 7ffb027fc700 1 --2- 172.21.15.16:0/3297932782 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb040d41c0 0x7ffb040ccd10 secure :-1 s=READY pgs=2631 cs=0 l=1 rev1=1 crypto rx=0x7ffaf40090b0 tx=0x7ffaf4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:02.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.092+0000 7ffafbfff700 1 -- 172.21.15.16:0/3297932782 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf40038a0 con 0x7ffb040d41c0 2023-10-16T13:47:02.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.092+0000 7ffafbfff700 1 -- 172.21.15.16:0/3297932782 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaf401f070 con 0x7ffb040d41c0 2023-10-16T13:47:02.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.092+0000 7ffb0995e700 1 -- 172.21.15.16:0/3297932782 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb040c8830 con 0x7ffb040d41c0 2023-10-16T13:47:02.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.092+0000 7ffafbfff700 1 -- 172.21.15.16:0/3297932782 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf4003c20 con 0x7ffb040d41c0 2023-10-16T13:47:02.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.092+0000 7ffb0995e700 1 -- 172.21.15.16:0/3297932782 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb0414ed70 con 0x7ffb040d41c0 2023-10-16T13:47:02.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.093+0000 7ffb0995e700 1 -- 172.21.15.16:0/3297932782 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb0404ebb0 con 0x7ffb040d41c0 2023-10-16T13:47:02.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.093+0000 7ffafbfff700 1 -- 172.21.15.16:0/3297932782 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 145) v1 ==== 88940+0+0 (secure 0 0 0) 0x7ffaf401a430 con 0x7ffb040d41c0 2023-10-16T13:47:02.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.094+0000 7ffafbfff700 1 -- 172.21.15.16:0/3297932782 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 7838+0+0 (secure 0 0 0) 0x7ffaf4021070 con 0x7ffb040d41c0 2023-10-16T13:47:02.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.098+0000 7ffafbfff700 1 -- 172.21.15.16:0/3297932782 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ffaf4055fd0 con 0x7ffb040d41c0 2023-10-16T13:47:02.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.351+0000 7ffb0995e700 1 -- 172.21.15.16:0/3297932782 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffb040b1e70 con 0x7ffb040d41c0 2023-10-16T13:47:02.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.351+0000 7ffafbfff700 1 -- 172.21.15.16:0/3297932782 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) v1 ==== 95+0+6154 (secure 0 0 0) 0x7ffaf402b070 con 0x7ffb040d41c0 2023-10-16T13:47:02.354 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:47:02.354 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":119,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10516,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/4065699432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":4065699432},{"type":"v1","addr":"172.21.15.49:6834","nonce":4065699432}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":108},{"gid":11415,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/1373048456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1373048456},{"type":"v1","addr":"172.21.15.16:6836","nonce":1373048456}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":108}],"filesystems":[{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:46:24.755313+0000","modified":"2023-10-16T13:46:28.011141+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11430},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11430":{"gid":11430,"name":"b","rank":0,"incarnation":109,"state":"up:active","state_seq":3,"addr":"172.21.15.49:6835/2905760119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":2905760119},{"type":"v1","addr":"172.21.15.49:6835","nonce":2905760119}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13},{"mdsmap":{"epoch":119,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:46:38.053490+0000","modified":"2023-10-16T13:46:41.076680+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11418},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11418":{"gid":11418,"name":"a","rank":0,"incarnation":115,"state":"up:active","state_seq":6,"addr":"172.21.15.16:6837/3860406853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":3860406853},{"type":"v1","addr":"172.21.15.16:6837","nonce":3860406853}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2023-10-16T13:47:02.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.353+0000 7ffb0995e700 1 -- 172.21.15.16:0/3297932782 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb040d41c0 msgr2=0x7ffb040ccd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:02.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.353+0000 7ffb0995e700 1 --2- 172.21.15.16:0/3297932782 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb040d41c0 0x7ffb040ccd10 secure :-1 s=READY pgs=2631 cs=0 l=1 rev1=1 crypto rx=0x7ffaf40090b0 tx=0x7ffaf4004750 comp rx=0 tx=0).stop 2023-10-16T13:47:02.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.353+0000 7ffb0995e700 1 -- 172.21.15.16:0/3297932782 shutdown_connections 2023-10-16T13:47:02.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.353+0000 7ffb0995e700 1 --2- 172.21.15.16:0/3297932782 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ffb0415d8d0 0x7ffb040cd250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:02.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.353+0000 7ffb0995e700 1 --2- 172.21.15.16:0/3297932782 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ffb040d0580 0x7ffb0414e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:02.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.353+0000 7ffb0995e700 1 --2- 172.21.15.16:0/3297932782 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb040d41c0 0x7ffb040ccd10 unknown :-1 s=CLOSED pgs=2631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:02.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.353+0000 7ffb0995e700 1 -- 172.21.15.16:0/3297932782 >> 172.21.15.16:0/3297932782 conn(0x7ffb040d3630 msgr2=0x7ffb0415c9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:02.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.353+0000 7ffb0995e700 1 -- 172.21.15.16:0/3297932782 shutdown_connections 2023-10-16T13:47:02.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.353+0000 7ffb0995e700 1 -- 172.21.15.16:0/3297932782 wait complete. 2023-10-16T13:47:02.358 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 119 2023-10-16T13:47:02.368 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:47:02.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.491+0000 7f0ab3ed0700 1 Processor -- start 2023-10-16T13:47:02.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.491+0000 7f0ab3ed0700 1 -- start start 2023-10-16T13:47:02.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.492+0000 7f0ab3ed0700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0aac15d8d0 0x7f0aac0d1c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:02.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.492+0000 7f0ab3ed0700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0aac0d41c0 0x7f0aac0d0040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:02.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.492+0000 7f0ab3ed0700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f0aac0d0580 0x7f0aac0ce240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:02.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.492+0000 7f0ab3ed0700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0aac0d0a10 con 0x7f0aac15d8d0 2023-10-16T13:47:02.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.492+0000 7f0ab3ed0700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0aac06ba00 con 0x7f0aac0d0580 2023-10-16T13:47:02.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.492+0000 7f0ab3ed0700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0aac0d2150 con 0x7f0aac0d41c0 2023-10-16T13:47:02.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.492+0000 7f0ab1c6c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0aac15d8d0 0x7f0aac0d1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:02.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.492+0000 7f0ab1c6c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0aac15d8d0 0x7f0aac0d1c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54730/0 (socket says 172.21.15.16:54730) 2023-10-16T13:47:02.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.492+0000 7f0ab1c6c700 1 -- 172.21.15.16:0/1729975781 learned_addr learned my addr 172.21.15.16:0/1729975781 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:02.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.492+0000 7f0ab146b700 1 --2- 172.21.15.16:0/1729975781 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0aac0d41c0 0x7f0aac0d0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:02.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.492+0000 7f0ab246d700 1 --2- 172.21.15.16:0/1729975781 >> v2:172.21.15.49:3300/0 conn(0x7f0aac0d0580 0x7f0aac0ce240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:02.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.492+0000 7f0ab1c6c700 1 -- 172.21.15.16:0/1729975781 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0aac0d41c0 msgr2=0x7f0aac0d0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:02.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.492+0000 7f0ab1c6c700 1 --2- 172.21.15.16:0/1729975781 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0aac0d41c0 0x7f0aac0d0040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:02.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.492+0000 7f0ab1c6c700 1 -- 172.21.15.16:0/1729975781 >> v2:172.21.15.49:3300/0 conn(0x7f0aac0d0580 msgr2=0x7f0aac0ce240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:02.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.492+0000 7f0ab1c6c700 1 --2- 172.21.15.16:0/1729975781 >> v2:172.21.15.49:3300/0 conn(0x7f0aac0d0580 0x7f0aac0ce240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:02.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.492+0000 7f0ab1c6c700 1 -- 172.21.15.16:0/1729975781 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0aac0d26d0 con 0x7f0aac15d8d0 2023-10-16T13:47:02.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.492+0000 7f0ab146b700 1 --2- 172.21.15.16:0/1729975781 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0aac0d41c0 0x7f0aac0d0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:47:02.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.492+0000 7f0ab246d700 1 --2- 172.21.15.16:0/1729975781 >> v2:172.21.15.49:3300/0 conn(0x7f0aac0d0580 0x7f0aac0ce240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:47:02.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.492+0000 7f0ab1c6c700 1 --2- 172.21.15.16:0/1729975781 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aac15d8d0 0x7f0aac0d1c10 secure :-1 s=READY pgs=2632 cs=0 l=1 rev1=1 crypto rx=0x7f0aac0c70f0 tx=0x7f0aa8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fb2810941a2f47a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:02.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.492+0000 7f0ab0c6a700 1 -- 172.21.15.16:0/1729975781 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa800b870 con 0x7f0aac15d8d0 2023-10-16T13:47:02.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.492+0000 7f0ab0c6a700 1 -- 172.21.15.16:0/1729975781 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0aa800b9d0 con 0x7f0aac15d8d0 2023-10-16T13:47:02.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.492+0000 7f0ab0c6a700 1 -- 172.21.15.16:0/1729975781 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa800bd60 con 0x7f0aac15d8d0 2023-10-16T13:47:02.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.493+0000 7f0ab3ed0700 1 -- 172.21.15.16:0/1729975781 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aac15d8d0 msgr2=0x7f0aac0d1c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:02.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.493+0000 7f0ab3ed0700 1 --2- 172.21.15.16:0/1729975781 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aac15d8d0 0x7f0aac0d1c10 secure :-1 s=READY pgs=2632 cs=0 l=1 rev1=1 crypto rx=0x7f0aac0c70f0 tx=0x7f0aa8009bd0 comp rx=0 tx=0).stop 2023-10-16T13:47:02.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.493+0000 7f0ab3ed0700 1 -- 172.21.15.16:0/1729975781 shutdown_connections 2023-10-16T13:47:02.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.493+0000 7f0ab3ed0700 1 --2- 172.21.15.16:0/1729975781 >> v2:172.21.15.49:3300/0 conn(0x7f0aac0d0580 0x7f0aac0ce240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:02.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.493+0000 7f0ab3ed0700 1 --2- 172.21.15.16:0/1729975781 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aac15d8d0 0x7f0aac0d1c10 unknown :-1 s=CLOSED pgs=2632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:02.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.493+0000 7f0ab3ed0700 1 --2- 172.21.15.16:0/1729975781 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0aac0d41c0 0x7f0aac0d0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:02.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.493+0000 7f0ab3ed0700 1 -- 172.21.15.16:0/1729975781 >> 172.21.15.16:0/1729975781 conn(0x7f0aac0d3630 msgr2=0x7f0aac0d3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:02.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.493+0000 7f0ab3ed0700 1 -- 172.21.15.16:0/1729975781 shutdown_connections 2023-10-16T13:47:02.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.493+0000 7f0ab3ed0700 1 -- 172.21.15.16:0/1729975781 wait complete. 2023-10-16T13:47:02.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.495+0000 7f0ab3ed0700 1 Processor -- start 2023-10-16T13:47:02.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.495+0000 7f0ab3ed0700 1 -- start start 2023-10-16T13:47:02.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.496+0000 7f0ab3ed0700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0aac0d0580 0x7f0aac14ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:02.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.496+0000 7f0ab3ed0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aac0d41c0 0x7f0aac09f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:02.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.496+0000 7f0ab3ed0700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0aac15d8d0 0x7f0aac09d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:02.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.496+0000 7f0ab3ed0700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aac14cf80 con 0x7f0aac0d41c0 2023-10-16T13:47:02.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.496+0000 7f0ab3ed0700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aac09fd00 con 0x7f0aac0d0580 2023-10-16T13:47:02.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.496+0000 7f0ab3ed0700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0aac09ffe0 con 0x7f0aac15d8d0 2023-10-16T13:47:02.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.496+0000 7f0ab146b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aac0d41c0 0x7f0aac09f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:02.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.496+0000 7f0ab146b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aac0d41c0 0x7f0aac09f220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54738/0 (socket says 172.21.15.16:54738) 2023-10-16T13:47:02.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.496+0000 7f0ab146b700 1 -- 172.21.15.16:0/172766559 learned_addr learned my addr 172.21.15.16:0/172766559 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:02.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.496+0000 7f0ab246d700 1 --2- 172.21.15.16:0/172766559 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0aac15d8d0 0x7f0aac09d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:02.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.496+0000 7f0ab1c6c700 1 --2- 172.21.15.16:0/172766559 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0aac0d0580 0x7f0aac14ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:02.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.496+0000 7f0ab146b700 1 -- 172.21.15.16:0/172766559 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0aac15d8d0 msgr2=0x7f0aac09d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:02.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.496+0000 7f0ab146b700 1 --2- 172.21.15.16:0/172766559 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0aac15d8d0 0x7f0aac09d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:02.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.496+0000 7f0ab146b700 1 -- 172.21.15.16:0/172766559 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0aac0d0580 msgr2=0x7f0aac14ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:02.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.496+0000 7f0ab146b700 1 --2- 172.21.15.16:0/172766559 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0aac0d0580 0x7f0aac14ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:02.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.496+0000 7f0ab146b700 1 -- 172.21.15.16:0/172766559 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0aa8009550 con 0x7f0aac0d41c0 2023-10-16T13:47:02.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.496+0000 7f0ab246d700 1 --2- 172.21.15.16:0/172766559 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0aac15d8d0 0x7f0aac09d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:47:02.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.496+0000 7f0ab1c6c700 1 --2- 172.21.15.16:0/172766559 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0aac0d0580 0x7f0aac14ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:47:02.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.496+0000 7f0ab146b700 1 --2- 172.21.15.16:0/172766559 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aac0d41c0 0x7f0aac09f220 secure :-1 s=READY pgs=2633 cs=0 l=1 rev1=1 crypto rx=0x7f0a9c00c750 tx=0x7f0a9c00cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:02.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.496+0000 7f0aa27fc700 1 -- 172.21.15.16:0/172766559 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a9c011800 con 0x7f0aac0d41c0 2023-10-16T13:47:02.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.496+0000 7f0aa27fc700 1 -- 172.21.15.16:0/172766559 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a9c011960 con 0x7f0aac0d41c0 2023-10-16T13:47:02.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.497+0000 7f0ab3ed0700 1 -- 172.21.15.16:0/172766559 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0aac0a02c0 con 0x7f0aac0d41c0 2023-10-16T13:47:02.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.497+0000 7f0aa27fc700 1 -- 172.21.15.16:0/172766559 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a9c011cd0 con 0x7f0aac0d41c0 2023-10-16T13:47:02.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.497+0000 7f0ab3ed0700 1 -- 172.21.15.16:0/172766559 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0aac0a0880 con 0x7f0aac0d41c0 2023-10-16T13:47:02.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.497+0000 7f0ab3ed0700 1 -- 172.21.15.16:0/172766559 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0aac04ebb0 con 0x7f0aac0d41c0 2023-10-16T13:47:02.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.498+0000 7f0aa27fc700 1 -- 172.21.15.16:0/172766559 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 145) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f0a9c021350 con 0x7f0aac0d41c0 2023-10-16T13:47:02.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.499+0000 7f0aa27fc700 1 -- 172.21.15.16:0/172766559 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 7838+0+0 (secure 0 0 0) 0x7f0a9c089840 con 0x7f0aac0d41c0 2023-10-16T13:47:02.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.503+0000 7f0aa27fc700 1 -- 172.21.15.16:0/172766559 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0a9c0544f0 con 0x7f0aac0d41c0 2023-10-16T13:47:02.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.710+0000 7f0ab3ed0700 1 -- 172.21.15.16:0/172766559 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0aac0686c0 con 0x7f0aac0d41c0 2023-10-16T13:47:02.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.712+0000 7f0aa27fc700 1 -- 172.21.15.16:0/172766559 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v231) v1 ==== 74+0+21725 (secure 0 0 0) 0x7f0a9c01e020 con 0x7f0aac0d41c0 2023-10-16T13:47:02.714 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:47:02.715 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":231,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:47:01.801913+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2023-10-16T13:46:22.489653+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"224","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2023-10-16T13:46:23.729509+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"225","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T13:46:35.800444+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"228","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"backup_fs_data","create_time":"2023-10-16T13:46:37.035323+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"229","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/1827947757":"2023-10-17T13:47:01.801871+0000","172.21.15.16:0/1694897016":"2023-10-17T13:47:01.801871+0000","172.21.15.16:0/3760456275":"2023-10-17T13:47:01.801871+0000","172.21.15.16:0/3651530055":"2023-10-17T13:47:01.801871+0000","172.21.15.16:0/850397235":"2023-10-17T13:47:01.801871+0000","172.21.15.16:6833/1506142504":"2023-10-17T13:47:01.801871+0000","172.21.15.16:0/4239949371":"2023-10-17T13:47:01.801871+0000","172.21.15.16:0/2321929682":"2023-10-17T13:46:51.165018+0000","172.21.15.16:0/3155275276":"2023-10-17T13:47:01.801871+0000","172.21.15.16:0/3429506235":"2023-10-17T13:46:51.165018+0000","172.21.15.16:0/4090083700":"2023-10-17T13:47:01.801871+0000","172.21.15.16:0/3843126404":"2023-10-17T13:46:51.165018+0000","172.21.15.16:6832/1506142504":"2023-10-17T13:47:01.801871+0000","172.21.15.16:0/1439318665":"2023-10-17T13:46:51.165018+0000","172.21.15.16:0/4204021371":"2023-10-17T13:46:51.165018+0000","172.21.15.16:6833/3171422553":"2023-10-17T13:46:51.165018+0000","172.21.15.16:6832/3171422553":"2023-10-17T13:46:51.165018+0000","172.21.15.16:0/1691810194":"2023-10-17T13:47:01.801871+0000","172.21.15.16:0/1632287797":"2023-10-17T13:46:51.165018+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:47:02.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.714+0000 7f0ab3ed0700 1 -- 172.21.15.16:0/172766559 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aac0d41c0 msgr2=0x7f0aac09f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:02.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.714+0000 7f0ab3ed0700 1 --2- 172.21.15.16:0/172766559 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aac0d41c0 0x7f0aac09f220 secure :-1 s=READY pgs=2633 cs=0 l=1 rev1=1 crypto rx=0x7f0a9c00c750 tx=0x7f0a9c00cb10 comp rx=0 tx=0).stop 2023-10-16T13:47:02.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.714+0000 7f0ab3ed0700 1 -- 172.21.15.16:0/172766559 shutdown_connections 2023-10-16T13:47:02.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.714+0000 7f0ab3ed0700 1 --2- 172.21.15.16:0/172766559 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0aac15d8d0 0x7f0aac09d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:02.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.714+0000 7f0ab3ed0700 1 --2- 172.21.15.16:0/172766559 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0aac0d0580 0x7f0aac14ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:02.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.714+0000 7f0ab3ed0700 1 --2- 172.21.15.16:0/172766559 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0aac0d41c0 0x7f0aac09f220 unknown :-1 s=CLOSED pgs=2633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:02.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.714+0000 7f0ab3ed0700 1 -- 172.21.15.16:0/172766559 >> 172.21.15.16:0/172766559 conn(0x7f0aac0d3630 msgr2=0x7f0aac0c9ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:02.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.714+0000 7f0ab3ed0700 1 -- 172.21.15.16:0/172766559 shutdown_connections 2023-10-16T13:47:02.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.714+0000 7f0ab3ed0700 1 -- 172.21.15.16:0/172766559 wait complete. 2023-10-16T13:47:02.729 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-16T13:47:02.729 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:47:02.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.855+0000 7f4440cee700 1 Processor -- start 2023-10-16T13:47:02.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.855+0000 7f4440cee700 1 -- start start 2023-10-16T13:47:02.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.856+0000 7f4440cee700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f443c15b860 0x7f443c06a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:02.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.856+0000 7f4440cee700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f443c0b4490 0x7f443c06ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:02.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.856+0000 7f4440cee700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f443c0d11c0 con 0x7f443c0b2be0 2023-10-16T13:47:02.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.856+0000 7f4440cee700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f443c0d03d0 con 0x7f443c15b860 2023-10-16T13:47:02.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.856+0000 7f4440cee700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f443c0cd360 con 0x7f443c0b4490 2023-10-16T13:47:02.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.857+0000 7f4439d9b700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f443c0b2be0 0x7f443c0719e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44040/0 (socket says 172.21.15.16:44040) 2023-10-16T13:47:02.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.857+0000 7f4439d9b700 1 -- 172.21.15.16:0/4099833352 learned_addr learned my addr 172.21.15.16:0/4099833352 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:47:02.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.857+0000 7f443a59c700 1 --2- 172.21.15.16:0/4099833352 >> v2:172.21.15.49:3300/0 conn(0x7f443c15b860 0x7f443c06a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:02.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.857+0000 7f443ad9d700 1 --2- 172.21.15.16:0/4099833352 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f443c0b4490 0x7f443c06ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:02.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.857+0000 7f443959a700 1 -- 172.21.15.16:0/4099833352 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1132323686 0 0) 0x7f443c0d11c0 con 0x7f443c0b2be0 2023-10-16T13:47:02.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.857+0000 7f443959a700 1 -- 172.21.15.16:0/4099833352 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44240035b0 con 0x7f443c0b2be0 2023-10-16T13:47:02.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.857+0000 7f443959a700 1 -- 172.21.15.16:0/4099833352 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2874056347 0 0) 0x7f44240035b0 con 0x7f443c0b2be0 2023-10-16T13:47:02.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.857+0000 7f443959a700 1 -- 172.21.15.16:0/4099833352 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f443c0b4490 msgr2=0x7f443c06ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:02.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.857+0000 7f443959a700 1 --2- 172.21.15.16:0/4099833352 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f443c0b4490 0x7f443c06ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:02.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.857+0000 7f443959a700 1 -- 172.21.15.16:0/4099833352 >> v2:172.21.15.49:3300/0 conn(0x7f443c15b860 msgr2=0x7f443c06a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:02.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.857+0000 7f443959a700 1 --2- 172.21.15.16:0/4099833352 >> v2:172.21.15.49:3300/0 conn(0x7f443c15b860 0x7f443c06a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:02.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.857+0000 7f443959a700 1 -- 172.21.15.16:0/4099833352 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f443c0d05b0 con 0x7f443c0b2be0 2023-10-16T13:47:02.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.857+0000 7f443959a700 1 -- 172.21.15.16:0/4099833352 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f4430001670 con 0x7f443c0b2be0 2023-10-16T13:47:02.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.857+0000 7f443959a700 1 -- 172.21.15.16:0/4099833352 >> v1:172.21.15.16:6789/0 conn(0x7f443c0b2be0 legacy=0x7f443c0719e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:02.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.857+0000 7f443959a700 1 --2- 172.21.15.16:0/4099833352 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f443c0b4490 0x7f4424003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:02.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.857+0000 7f443959a700 1 --2- 172.21.15.16:0/4099833352 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4424003f80 0x7f4424004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:02.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.857+0000 7f443959a700 1 --2- 172.21.15.16:0/4099833352 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4424004d90 0x7f4424005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:02.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.857+0000 7f443959a700 1 -- 172.21.15.16:0/4099833352 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f443c0d03d0 con 0x7f4424003f80 2023-10-16T13:47:02.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.857+0000 7f443959a700 1 -- 172.21.15.16:0/4099833352 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f443c0cd360 con 0x7f4424004d90 2023-10-16T13:47:02.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.857+0000 7f443959a700 1 -- 172.21.15.16:0/4099833352 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f442400ba10 con 0x7f443c0b4490 2023-10-16T13:47:02.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.857+0000 7f443a59c700 1 --2- 172.21.15.16:0/4099833352 >> v2:172.21.15.49:3300/0 conn(0x7f443c15b860 0x7f443c06a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:47:02.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.857+0000 7f4439d9b700 1 --2- 172.21.15.16:0/4099833352 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4424003f80 0x7f4424004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:02.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.858+0000 7f443ad9d700 1 --2- 172.21.15.16:0/4099833352 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4424004d90 0x7f4424005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:02.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.858+0000 7f443a59c700 1 --2- 172.21.15.16:0/4099833352 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f443c0b4490 0x7f4424003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:02.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.858+0000 7f4439d9b700 1 -- 172.21.15.16:0/4099833352 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f443c0b4490 msgr2=0x7f4424003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:02.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.858+0000 7f4439d9b700 1 --2- 172.21.15.16:0/4099833352 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f443c0b4490 0x7f4424003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:02.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.858+0000 7f4439d9b700 1 -- 172.21.15.16:0/4099833352 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4424004d90 msgr2=0x7f4424005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:02.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.858+0000 7f4439d9b700 1 --2- 172.21.15.16:0/4099833352 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4424004d90 0x7f4424005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:02.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.858+0000 7f4439d9b700 1 -- 172.21.15.16:0/4099833352 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44240035b0 con 0x7f4424003f80 2023-10-16T13:47:02.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.858+0000 7f443a59c700 1 --2- 172.21.15.16:0/4099833352 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f443c0b4490 0x7f4424003a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:47:02.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.858+0000 7f443ad9d700 1 --2- 172.21.15.16:0/4099833352 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4424004d90 0x7f4424005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:47:02.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.858+0000 7f4439d9b700 1 --2- 172.21.15.16:0/4099833352 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4424003f80 0x7f4424004850 secure :-1 s=READY pgs=2635 cs=0 l=1 rev1=1 crypto rx=0x7f443000c2f0 tx=0x7f4424005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d29f47686398863e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:02.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.858+0000 7f443959a700 1 -- 172.21.15.16:0/4099833352 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f443000cdd0 con 0x7f4424003f80 2023-10-16T13:47:02.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.858+0000 7f443959a700 1 -- 172.21.15.16:0/4099833352 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4430005960 con 0x7f4424003f80 2023-10-16T13:47:02.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.858+0000 7f443959a700 1 -- 172.21.15.16:0/4099833352 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4430005d10 con 0x7f4424003f80 2023-10-16T13:47:02.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.858+0000 7f4440cee700 1 -- 172.21.15.16:0/4099833352 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4424003f80 msgr2=0x7f4424004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:02.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.858+0000 7f4440cee700 1 --2- 172.21.15.16:0/4099833352 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4424003f80 0x7f4424004850 secure :-1 s=READY pgs=2635 cs=0 l=1 rev1=1 crypto rx=0x7f443000c2f0 tx=0x7f4424005880 comp rx=0 tx=0).stop 2023-10-16T13:47:02.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.858+0000 7f443ad9d700 1 -- 172.21.15.16:0/4099833352 reap_dead start 2023-10-16T13:47:02.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.859+0000 7f4440cee700 1 -- 172.21.15.16:0/4099833352 shutdown_connections 2023-10-16T13:47:02.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.859+0000 7f4440cee700 1 -- 172.21.15.16:0/4099833352 >> 172.21.15.16:0/4099833352 conn(0x7f443c0b1fd0 msgr2=0x7f443c0b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:02.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.859+0000 7f4440cee700 1 -- 172.21.15.16:0/4099833352 shutdown_connections 2023-10-16T13:47:02.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.859+0000 7f4440cee700 1 -- 172.21.15.16:0/4099833352 wait complete. 2023-10-16T13:47:02.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.860+0000 7f4440cee700 1 Processor -- start 2023-10-16T13:47:02.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.860+0000 7f4440cee700 1 -- start start 2023-10-16T13:47:02.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.861+0000 7f4440cee700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4424003f80 0x7f443c0c9e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:02.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.861+0000 7f4440cee700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f443c14d7e0 0x7f443c0b0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:02.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.861+0000 7f4440cee700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f443c14dbe0 0x7f443c0b14e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:02.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.861+0000 7f4440cee700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f443c14d350 con 0x7f443c14dbe0 2023-10-16T13:47:02.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.861+0000 7f4440cee700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f443c159ef0 con 0x7f4424003f80 2023-10-16T13:47:02.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.861+0000 7f4440cee700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f443c0ca340 con 0x7f443c14d7e0 2023-10-16T13:47:02.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.861+0000 7f443ad9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f443c14dbe0 0x7f443c0b14e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:02.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.861+0000 7f443ad9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f443c14dbe0 0x7f443c0b14e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54780/0 (socket says 172.21.15.16:54780) 2023-10-16T13:47:02.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.861+0000 7f443ad9d700 1 -- 172.21.15.16:0/1180736424 learned_addr learned my addr 172.21.15.16:0/1180736424 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:02.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.861+0000 7f4439d9b700 1 --2- 172.21.15.16:0/1180736424 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f443c14d7e0 0x7f443c0b0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:02.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.861+0000 7f443a59c700 1 --2- 172.21.15.16:0/1180736424 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4424003f80 0x7f443c0c9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:02.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.861+0000 7f443ad9d700 1 -- 172.21.15.16:0/1180736424 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f443c14d7e0 msgr2=0x7f443c0b0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:02.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.861+0000 7f443ad9d700 1 --2- 172.21.15.16:0/1180736424 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f443c14d7e0 0x7f443c0b0fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:02.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.861+0000 7f443ad9d700 1 -- 172.21.15.16:0/1180736424 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4424003f80 msgr2=0x7f443c0c9e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:02.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.861+0000 7f443ad9d700 1 --2- 172.21.15.16:0/1180736424 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4424003f80 0x7f443c0c9e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:02.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.861+0000 7f443ad9d700 1 -- 172.21.15.16:0/1180736424 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f443c14c740 con 0x7f443c14dbe0 2023-10-16T13:47:02.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.861+0000 7f443ad9d700 1 --2- 172.21.15.16:0/1180736424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f443c14dbe0 0x7f443c0b14e0 secure :-1 s=READY pgs=2636 cs=0 l=1 rev1=1 crypto rx=0x7f443c0cce90 tx=0x7f44340111d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:02.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.861+0000 7f442affd700 1 -- 172.21.15.16:0/1180736424 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f443400e030 con 0x7f443c14dbe0 2023-10-16T13:47:02.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.861+0000 7f442affd700 1 -- 172.21.15.16:0/1180736424 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4434011a60 con 0x7f443c14dbe0 2023-10-16T13:47:02.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.862+0000 7f4440cee700 1 -- 172.21.15.16:0/1180736424 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f443c14ca20 con 0x7f443c14dbe0 2023-10-16T13:47:02.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.862+0000 7f442affd700 1 -- 172.21.15.16:0/1180736424 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4434011e10 con 0x7f443c14dbe0 2023-10-16T13:47:02.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.862+0000 7f4440cee700 1 -- 172.21.15.16:0/1180736424 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f443c14cfe0 con 0x7f443c14dbe0 2023-10-16T13:47:02.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.862+0000 7f4440cee700 1 -- 172.21.15.16:0/1180736424 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f443c04ebb0 con 0x7f443c14dbe0 2023-10-16T13:47:02.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.863+0000 7f442affd700 1 -- 172.21.15.16:0/1180736424 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 146) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f4434016470 con 0x7f443c14dbe0 2023-10-16T13:47:02.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.863+0000 7f442affd700 1 --2- 172.21.15.16:0/1180736424 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f441c06ad50 0x7f441c06d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:02.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.863+0000 7f442affd700 1 -- 172.21.15.16:0/1180736424 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 7838+0+0 (secure 0 0 0) 0x7f443400f070 con 0x7f443c14dbe0 2023-10-16T13:47:02.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.864+0000 7f443a59c700 1 --2- 172.21.15.16:0/1180736424 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f441c06ad50 0x7f441c06d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:02.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.864+0000 7f443a59c700 1 --2- 172.21.15.16:0/1180736424 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f441c06ad50 0x7f441c06d210 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f442c009d50 tx=0x7f442c008040 comp rx=0 tx=0).ready entity=mgr.11886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:02.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:02.868+0000 7f442affd700 1 -- 172.21.15.16:0/1180736424 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4434054b30 con 0x7f443c14dbe0 2023-10-16T13:47:03.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.121+0000 7f4440cee700 1 -- 172.21.15.16:0/1180736424 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f443c14d490 con 0x7f443c14dbe0 2023-10-16T13:47:03.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.122+0000 7f442affd700 1 -- 172.21.15.16:0/1180736424 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) v1 ==== 95+0+6154 (secure 0 0 0) 0x7f443401a020 con 0x7f443c14dbe0 2023-10-16T13:47:03.124 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:47:03.124 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":119,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10516,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/4065699432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":4065699432},{"type":"v1","addr":"172.21.15.49:6834","nonce":4065699432}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":108},{"gid":11415,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/1373048456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1373048456},{"type":"v1","addr":"172.21.15.16:6836","nonce":1373048456}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":108}],"filesystems":[{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:46:24.755313+0000","modified":"2023-10-16T13:46:28.011141+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11430},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11430":{"gid":11430,"name":"b","rank":0,"incarnation":109,"state":"up:active","state_seq":3,"addr":"172.21.15.49:6835/2905760119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":2905760119},{"type":"v1","addr":"172.21.15.49:6835","nonce":2905760119}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13},{"mdsmap":{"epoch":119,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:46:38.053490+0000","modified":"2023-10-16T13:46:41.076680+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11418},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11418":{"gid":11418,"name":"a","rank":0,"incarnation":115,"state":"up:active","state_seq":6,"addr":"172.21.15.16:6837/3860406853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":3860406853},{"type":"v1","addr":"172.21.15.16:6837","nonce":3860406853}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2023-10-16T13:47:03.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.124+0000 7f4440cee700 1 -- 172.21.15.16:0/1180736424 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f441c06ad50 msgr2=0x7f441c06d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:03.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.124+0000 7f4440cee700 1 --2- 172.21.15.16:0/1180736424 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f441c06ad50 0x7f441c06d210 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f442c009d50 tx=0x7f442c008040 comp rx=0 tx=0).stop 2023-10-16T13:47:03.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.124+0000 7f4440cee700 1 -- 172.21.15.16:0/1180736424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f443c14dbe0 msgr2=0x7f443c0b14e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:03.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.124+0000 7f4440cee700 1 --2- 172.21.15.16:0/1180736424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f443c14dbe0 0x7f443c0b14e0 secure :-1 s=READY pgs=2636 cs=0 l=1 rev1=1 crypto rx=0x7f443c0cce90 tx=0x7f44340111d0 comp rx=0 tx=0).stop 2023-10-16T13:47:03.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.124+0000 7f4440cee700 1 -- 172.21.15.16:0/1180736424 shutdown_connections 2023-10-16T13:47:03.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.124+0000 7f4440cee700 1 --2- 172.21.15.16:0/1180736424 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f441c06ad50 0x7f441c06d210 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:03.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.124+0000 7f4440cee700 1 --2- 172.21.15.16:0/1180736424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f443c14dbe0 0x7f443c0b14e0 unknown :-1 s=CLOSED pgs=2636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:03.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.124+0000 7f4440cee700 1 --2- 172.21.15.16:0/1180736424 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4424003f80 0x7f443c0c9e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:03.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.124+0000 7f4440cee700 1 --2- 172.21.15.16:0/1180736424 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f443c14d7e0 0x7f443c0b0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:03.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.124+0000 7f4440cee700 1 -- 172.21.15.16:0/1180736424 >> 172.21.15.16:0/1180736424 conn(0x7f443c0b1fd0 msgr2=0x7f443c0718e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:03.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.124+0000 7f4440cee700 1 -- 172.21.15.16:0/1180736424 shutdown_connections 2023-10-16T13:47:03.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.124+0000 7f4440cee700 1 -- 172.21.15.16:0/1180736424 wait complete. 2023-10-16T13:47:03.129 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 119 2023-10-16T13:47:03.138 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:47:03.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.261+0000 7f33e9be2700 1 Processor -- start 2023-10-16T13:47:03.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.261+0000 7f33e9be2700 1 -- start start 2023-10-16T13:47:03.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.261+0000 7f33e9be2700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f33e406b7f0 0x7f33e406bc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:03.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.261+0000 7f33e9be2700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33e40c3490 con 0x7f33e415b9f0 2023-10-16T13:47:03.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.261+0000 7f33e9be2700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33e40cdad0 con 0x7f33e406d940 2023-10-16T13:47:03.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.262+0000 7f33e9be2700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33e406c1b0 con 0x7f33e406b7f0 2023-10-16T13:47:03.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.262+0000 7f33e37fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f33e415b9f0 0x7f33e40b29a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44058/0 (socket says 172.21.15.16:44058) 2023-10-16T13:47:03.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.262+0000 7f33e37fe700 1 -- 172.21.15.16:0/2762397061 learned_addr learned my addr 172.21.15.16:0/2762397061 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:47:03.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.262+0000 7f33e2ffd700 1 --2- 172.21.15.16:0/2762397061 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f33e406b7f0 0x7f33e406bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:03.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.262+0000 7f33e27fc700 1 -- 172.21.15.16:0/2762397061 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3455375055 0 0) 0x7f33e40c3490 con 0x7f33e415b9f0 2023-10-16T13:47:03.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.262+0000 7f33e27fc700 1 -- 172.21.15.16:0/2762397061 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33d00035b0 con 0x7f33e415b9f0 2023-10-16T13:47:03.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.262+0000 7f33e27fc700 1 -- 172.21.15.16:0/2762397061 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1110418200 0 0) 0x7f33e40cdad0 con 0x7f33e406d940 2023-10-16T13:47:03.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.262+0000 7f33e27fc700 1 -- 172.21.15.16:0/2762397061 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33e40c3490 con 0x7f33e406d940 2023-10-16T13:47:03.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.262+0000 7f33e2ffd700 1 -- 172.21.15.16:0/2762397061 >> v1:172.21.15.49:6789/0 conn(0x7f33e406d940 legacy=0x7f33e40ceb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:03.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.262+0000 7f33e2ffd700 1 -- 172.21.15.16:0/2762397061 >> v1:172.21.15.16:6789/0 conn(0x7f33e415b9f0 legacy=0x7f33e40b29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:03.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.262+0000 7f33e27fc700 1 -- 172.21.15.16:0/2762397061 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 219040325 0 0) 0x7f33d00035b0 con 0x7f33e415b9f0 2023-10-16T13:47:03.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.262+0000 7f33e2ffd700 1 -- 172.21.15.16:0/2762397061 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33e40cddb0 con 0x7f33e406b7f0 2023-10-16T13:47:03.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.262+0000 7f33e2ffd700 1 --2- 172.21.15.16:0/2762397061 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f33e406b7f0 0x7f33e406bc70 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f33cc009870 tx=0x7f33cc009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c644e8b41302bbe5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:03.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.263+0000 7f33e27fc700 1 -- 172.21.15.16:0/2762397061 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33d40019f0 con 0x7f33e406b7f0 2023-10-16T13:47:03.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.263+0000 7f33e27fc700 1 -- 172.21.15.16:0/2762397061 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33cc004030 con 0x7f33e406b7f0 2023-10-16T13:47:03.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.263+0000 7f33e27fc700 1 -- 172.21.15.16:0/2762397061 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33cc0043c0 con 0x7f33e406b7f0 2023-10-16T13:47:03.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.263+0000 7f33e9be2700 1 -- 172.21.15.16:0/2762397061 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f33e406b7f0 msgr2=0x7f33e406bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:03.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.263+0000 7f33e9be2700 1 --2- 172.21.15.16:0/2762397061 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f33e406b7f0 0x7f33e406bc70 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f33cc009870 tx=0x7f33cc009b80 comp rx=0 tx=0).stop 2023-10-16T13:47:03.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.263+0000 7f33e9be2700 1 -- 172.21.15.16:0/2762397061 shutdown_connections 2023-10-16T13:47:03.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.263+0000 7f33e9be2700 1 --2- 172.21.15.16:0/2762397061 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f33e406b7f0 0x7f33e406bc70 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:03.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.263+0000 7f33e9be2700 1 -- 172.21.15.16:0/2762397061 >> 172.21.15.16:0/2762397061 conn(0x7f33e40cc5b0 msgr2=0x7f33e40cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:03.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.263+0000 7f33e9be2700 1 -- 172.21.15.16:0/2762397061 shutdown_connections 2023-10-16T13:47:03.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.263+0000 7f33e9be2700 1 -- 172.21.15.16:0/2762397061 wait complete. 2023-10-16T13:47:03.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.264+0000 7f33e9be2700 1 Processor -- start 2023-10-16T13:47:03.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.265+0000 7f33e9be2700 1 -- start start 2023-10-16T13:47:03.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.265+0000 7f33e9be2700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f33e406b7f0 0x7f33e41f2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:03.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.265+0000 7f33e9be2700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33e406d940 0x7f33e41f3230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:03.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.265+0000 7f33e9be2700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f33e40ca8a0 0x7f33e4157790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:03.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.265+0000 7f33e9be2700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33e414c6b0 con 0x7f33e406d940 2023-10-16T13:47:03.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.265+0000 7f33e9be2700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33e40cada0 con 0x7f33e406b7f0 2023-10-16T13:47:03.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.265+0000 7f33e9be2700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33e40cb080 con 0x7f33e40ca8a0 2023-10-16T13:47:03.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.265+0000 7f33e2ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33e406d940 0x7f33e41f3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:03.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.265+0000 7f33e2ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33e406d940 0x7f33e41f3230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54792/0 (socket says 172.21.15.16:54792) 2023-10-16T13:47:03.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.265+0000 7f33e2ffd700 1 -- 172.21.15.16:0/204214625 learned_addr learned my addr 172.21.15.16:0/204214625 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:03.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.266+0000 7f33e37fe700 1 --2- 172.21.15.16:0/204214625 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f33e406b7f0 0x7f33e41f2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:03.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.266+0000 7f33e3fff700 1 --2- 172.21.15.16:0/204214625 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f33e40ca8a0 0x7f33e4157790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:03.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.266+0000 7f33e2ffd700 1 -- 172.21.15.16:0/204214625 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f33e40ca8a0 msgr2=0x7f33e4157790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:03.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.266+0000 7f33e2ffd700 1 --2- 172.21.15.16:0/204214625 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f33e40ca8a0 0x7f33e4157790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:03.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.266+0000 7f33e2ffd700 1 -- 172.21.15.16:0/204214625 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f33e406b7f0 msgr2=0x7f33e41f2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:03.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.266+0000 7f33e2ffd700 1 --2- 172.21.15.16:0/204214625 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f33e406b7f0 0x7f33e41f2cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:03.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.266+0000 7f33e2ffd700 1 -- 172.21.15.16:0/204214625 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33cc0094a0 con 0x7f33e406d940 2023-10-16T13:47:03.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.266+0000 7f33e2ffd700 1 --2- 172.21.15.16:0/204214625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33e406d940 0x7f33e41f3230 secure :-1 s=READY pgs=2638 cs=0 l=1 rev1=1 crypto rx=0x7f33cc0097c0 tx=0x7f33cc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:03.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.266+0000 7f33dbfff700 1 -- 172.21.15.16:0/204214625 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33cc003db0 con 0x7f33e406d940 2023-10-16T13:47:03.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.266+0000 7f33dbfff700 1 -- 172.21.15.16:0/204214625 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33cc01f070 con 0x7f33e406d940 2023-10-16T13:47:03.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.266+0000 7f33e9be2700 1 -- 172.21.15.16:0/204214625 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33e40cb360 con 0x7f33e406d940 2023-10-16T13:47:03.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.266+0000 7f33dbfff700 1 -- 172.21.15.16:0/204214625 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33cc011870 con 0x7f33e406d940 2023-10-16T13:47:03.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.266+0000 7f33e9be2700 1 -- 172.21.15.16:0/204214625 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33e40cb920 con 0x7f33e406d940 2023-10-16T13:47:03.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.267+0000 7f33e9be2700 1 -- 172.21.15.16:0/204214625 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33e404ebb0 con 0x7f33e406d940 2023-10-16T13:47:03.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.268+0000 7f33dbfff700 1 -- 172.21.15.16:0/204214625 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 146) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f33cc021030 con 0x7f33e406d940 2023-10-16T13:47:03.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.268+0000 7f33dbfff700 1 --2- 172.21.15.16:0/204214625 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f33c406ad50 0x7f33c406d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:03.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.268+0000 7f33dbfff700 1 -- 172.21.15.16:0/204214625 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 7838+0+0 (secure 0 0 0) 0x7f33cc08adf0 con 0x7f33e406d940 2023-10-16T13:47:03.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.268+0000 7f33e37fe700 1 --2- 172.21.15.16:0/204214625 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f33c406ad50 0x7f33c406d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:03.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.269+0000 7f33e37fe700 1 --2- 172.21.15.16:0/204214625 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f33c406ad50 0x7f33c406d210 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f33d40012e0 tx=0x7f33d4001f80 comp rx=0 tx=0).ready entity=mgr.11886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:03.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.273+0000 7f33dbfff700 1 -- 172.21.15.16:0/204214625 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f33cc055ab0 con 0x7f33e406d940 2023-10-16T13:47:03.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.524+0000 7f33e9be2700 1 -- 172.21.15.16:0/204214625 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f33e40b5ab0 con 0x7f33e406d940 2023-10-16T13:47:03.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.524+0000 7f33dbfff700 1 -- 172.21.15.16:0/204214625 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) v1 ==== 95+0+6154 (secure 0 0 0) 0x7f33cc0590d0 con 0x7f33e406d940 2023-10-16T13:47:03.527 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:47:03.527 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":119,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10516,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/4065699432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":4065699432},{"type":"v1","addr":"172.21.15.49:6834","nonce":4065699432}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":108},{"gid":11415,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/1373048456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1373048456},{"type":"v1","addr":"172.21.15.16:6836","nonce":1373048456}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":108}],"filesystems":[{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:46:24.755313+0000","modified":"2023-10-16T13:46:28.011141+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11430},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11430":{"gid":11430,"name":"b","rank":0,"incarnation":109,"state":"up:active","state_seq":3,"addr":"172.21.15.49:6835/2905760119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":2905760119},{"type":"v1","addr":"172.21.15.49:6835","nonce":2905760119}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13},{"mdsmap":{"epoch":119,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:46:38.053490+0000","modified":"2023-10-16T13:46:41.076680+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11418},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11418":{"gid":11418,"name":"a","rank":0,"incarnation":115,"state":"up:active","state_seq":6,"addr":"172.21.15.16:6837/3860406853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":3860406853},{"type":"v1","addr":"172.21.15.16:6837","nonce":3860406853}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2023-10-16T13:47:03.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.526+0000 7f33e9be2700 1 -- 172.21.15.16:0/204214625 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f33c406ad50 msgr2=0x7f33c406d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:03.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.526+0000 7f33e9be2700 1 --2- 172.21.15.16:0/204214625 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f33c406ad50 0x7f33c406d210 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f33d40012e0 tx=0x7f33d4001f80 comp rx=0 tx=0).stop 2023-10-16T13:47:03.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.526+0000 7f33e9be2700 1 -- 172.21.15.16:0/204214625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33e406d940 msgr2=0x7f33e41f3230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:03.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.526+0000 7f33e9be2700 1 --2- 172.21.15.16:0/204214625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33e406d940 0x7f33e41f3230 secure :-1 s=READY pgs=2638 cs=0 l=1 rev1=1 crypto rx=0x7f33cc0097c0 tx=0x7f33cc004750 comp rx=0 tx=0).stop 2023-10-16T13:47:03.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.527+0000 7f33e9be2700 1 -- 172.21.15.16:0/204214625 shutdown_connections 2023-10-16T13:47:03.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.527+0000 7f33e9be2700 1 --2- 172.21.15.16:0/204214625 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f33c406ad50 0x7f33c406d210 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:03.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.527+0000 7f33e9be2700 1 --2- 172.21.15.16:0/204214625 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f33e40ca8a0 0x7f33e4157790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:03.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.527+0000 7f33e9be2700 1 --2- 172.21.15.16:0/204214625 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f33e406b7f0 0x7f33e41f2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:03.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.527+0000 7f33e9be2700 1 --2- 172.21.15.16:0/204214625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33e406d940 0x7f33e41f3230 unknown :-1 s=CLOSED pgs=2638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:03.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.527+0000 7f33e9be2700 1 -- 172.21.15.16:0/204214625 >> 172.21.15.16:0/204214625 conn(0x7f33e40cc5b0 msgr2=0x7f33e40714f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:03.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.527+0000 7f33e9be2700 1 -- 172.21.15.16:0/204214625 shutdown_connections 2023-10-16T13:47:03.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.527+0000 7f33e9be2700 1 -- 172.21.15.16:0/204214625 wait complete. 2023-10-16T13:47:03.532 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 119 2023-10-16T13:47:03.541 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:47:03.573 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:47:03.587 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:47:03.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.665+0000 7fc7aa1c0700 1 Processor -- start 2023-10-16T13:47:03.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.666+0000 7fc7aa1c0700 1 -- start start 2023-10-16T13:47:03.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.666+0000 7fc7aa1c0700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fc7a40cc8a0 0x7fc7a415e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:03.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.666+0000 7fc7aa1c0700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7a4161c30 con 0x7fc7a406a7e0 2023-10-16T13:47:03.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.666+0000 7fc7aa1c0700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7a4161f10 con 0x7fc7a415b280 2023-10-16T13:47:03.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.666+0000 7fc7aa1c0700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7a41621f0 con 0x7fc7a40cc8a0 2023-10-16T13:47:03.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.666+0000 7fc7a3fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc7a406a7e0 0x7fc7a40d02b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44076/0 (socket says 172.21.15.16:44076) 2023-10-16T13:47:03.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.666+0000 7fc7a3fff700 1 -- 172.21.15.16:0/4022723733 learned_addr learned my addr 172.21.15.16:0/4022723733 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:47:03.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.667+0000 7fc7a2ffd700 1 --2- 172.21.15.16:0/4022723733 >> v2:172.21.15.49:3300/0 conn(0x7fc7a40cc8a0 0x7fc7a415e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:03.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.667+0000 7fc7a27fc700 1 -- 172.21.15.16:0/4022723733 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2338652863 0 0) 0x7fc7a4161c30 con 0x7fc7a406a7e0 2023-10-16T13:47:03.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.667+0000 7fc7a27fc700 1 -- 172.21.15.16:0/4022723733 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7900035b0 con 0x7fc7a406a7e0 2023-10-16T13:47:03.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.667+0000 7fc7a27fc700 1 -- 172.21.15.16:0/4022723733 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1368405495 0 0) 0x7fc7900035b0 con 0x7fc7a406a7e0 2023-10-16T13:47:03.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.667+0000 7fc7a27fc700 1 -- 172.21.15.16:0/4022723733 >> v2:172.21.15.49:3300/0 conn(0x7fc7a40cc8a0 msgr2=0x7fc7a415e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:03.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.667+0000 7fc7a27fc700 1 --2- 172.21.15.16:0/4022723733 >> v2:172.21.15.49:3300/0 conn(0x7fc7a40cc8a0 0x7fc7a415e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:03.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.667+0000 7fc7a27fc700 1 -- 172.21.15.16:0/4022723733 >> v1:172.21.15.49:6789/0 conn(0x7fc7a415b280 legacy=0x7fc7a41613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:03.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.667+0000 7fc7a27fc700 1 -- 172.21.15.16:0/4022723733 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7a415df00 con 0x7fc7a406a7e0 2023-10-16T13:47:03.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.667+0000 7fc7a27fc700 1 -- 172.21.15.16:0/4022723733 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7fc79c0019f0 con 0x7fc7a406a7e0 2023-10-16T13:47:03.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.667+0000 7fc7a27fc700 1 -- 172.21.15.16:0/4022723733 >> v1:172.21.15.16:6789/0 conn(0x7fc7a406a7e0 legacy=0x7fc7a40d02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:03.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.667+0000 7fc7a27fc700 1 --2- 172.21.15.16:0/4022723733 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc790003a00 0x7fc790003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:03.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.667+0000 7fc7a27fc700 1 --2- 172.21.15.16:0/4022723733 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc790004400 0x7fc7900048d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:03.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.667+0000 7fc7a27fc700 1 --2- 172.21.15.16:0/4022723733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7900050d0 0x7fc79000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:03.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.667+0000 7fc7a27fc700 1 -- 172.21.15.16:0/4022723733 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a41621f0 con 0x7fc7900050d0 2023-10-16T13:47:03.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.667+0000 7fc7a27fc700 1 -- 172.21.15.16:0/4022723733 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc79000c030 con 0x7fc790003a00 2023-10-16T13:47:03.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.667+0000 7fc7a27fc700 1 -- 172.21.15.16:0/4022723733 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc79000c1f0 con 0x7fc790004400 2023-10-16T13:47:03.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.667+0000 7fc7a2ffd700 1 --2- 172.21.15.16:0/4022723733 >> v2:172.21.15.49:3300/0 conn(0x7fc7a40cc8a0 0x7fc7a415e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:47:03.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.667+0000 7fc7a3fff700 1 --2- 172.21.15.16:0/4022723733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7900050d0 0x7fc79000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:03.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.667+0000 7fc7a37fe700 1 --2- 172.21.15.16:0/4022723733 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc790003a00 0x7fc790003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:03.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.668+0000 7fc7a2ffd700 1 --2- 172.21.15.16:0/4022723733 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc790004400 0x7fc7900048d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:03.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.668+0000 7fc7a3fff700 1 -- 172.21.15.16:0/4022723733 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc790004400 msgr2=0x7fc7900048d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:03.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.668+0000 7fc7a3fff700 1 --2- 172.21.15.16:0/4022723733 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc790004400 0x7fc7900048d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:03.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.668+0000 7fc7a3fff700 1 -- 172.21.15.16:0/4022723733 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc790003a00 msgr2=0x7fc790003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:03.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.668+0000 7fc7a3fff700 1 --2- 172.21.15.16:0/4022723733 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc790003a00 0x7fc790003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:03.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.668+0000 7fc7a3fff700 1 -- 172.21.15.16:0/4022723733 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7900035b0 con 0x7fc7900050d0 2023-10-16T13:47:03.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.668+0000 7fc7a3fff700 1 -- 172.21.15.16:0/4022723733 reap_dead start 2023-10-16T13:47:03.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.668+0000 7fc7a3fff700 1 --2- 172.21.15.16:0/4022723733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7900050d0 0x7fc79000ba10 secure :-1 s=READY pgs=2640 cs=0 l=1 rev1=1 crypto rx=0x7fc79c00a830 tx=0x7fc790005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=404b26038a2e100f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:03.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.668+0000 7fc7a27fc700 1 -- 172.21.15.16:0/4022723733 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc79c012600 con 0x7fc7900050d0 2023-10-16T13:47:03.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.668+0000 7fc7a27fc700 1 -- 172.21.15.16:0/4022723733 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc79c012780 con 0x7fc7900050d0 2023-10-16T13:47:03.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.668+0000 7fc7a27fc700 1 -- 172.21.15.16:0/4022723733 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc79c012b30 con 0x7fc7900050d0 2023-10-16T13:47:03.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.668+0000 7fc7aa1c0700 1 -- 172.21.15.16:0/4022723733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7900050d0 msgr2=0x7fc79000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:03.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.668+0000 7fc7aa1c0700 1 --2- 172.21.15.16:0/4022723733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7900050d0 0x7fc79000ba10 secure :-1 s=READY pgs=2640 cs=0 l=1 rev1=1 crypto rx=0x7fc79c00a830 tx=0x7fc790005880 comp rx=0 tx=0).stop 2023-10-16T13:47:03.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.669+0000 7fc7aa1c0700 1 -- 172.21.15.16:0/4022723733 shutdown_connections 2023-10-16T13:47:03.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.669+0000 7fc7aa1c0700 1 --2- 172.21.15.16:0/4022723733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7900050d0 0x7fc79000ba10 unknown :-1 s=CLOSED pgs=2640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:03.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.669+0000 7fc7aa1c0700 1 -- 172.21.15.16:0/4022723733 >> 172.21.15.16:0/4022723733 conn(0x7fc7a40b1fd0 msgr2=0x7fc7a40b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:03.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.669+0000 7fc7aa1c0700 1 -- 172.21.15.16:0/4022723733 shutdown_connections 2023-10-16T13:47:03.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.669+0000 7fc7aa1c0700 1 -- 172.21.15.16:0/4022723733 wait complete. 2023-10-16T13:47:03.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.670+0000 7fc7aa1c0700 1 Processor -- start 2023-10-16T13:47:03.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.670+0000 7fc7aa1c0700 1 -- start start 2023-10-16T13:47:03.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.671+0000 7fc7aa1c0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7900050d0 0x7fc7a40c3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:03.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.671+0000 7fc7aa1c0700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc7a4159970 0x7fc7a40d0270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:03.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.671+0000 7fc7aa1c0700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc7a40d07b0 0x7fc7a40c97e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:03.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.671+0000 7fc7aa1c0700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a40d0c30 con 0x7fc7900050d0 2023-10-16T13:47:03.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.671+0000 7fc7aa1c0700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a40af310 con 0x7fc7a4159970 2023-10-16T13:47:03.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.671+0000 7fc7aa1c0700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a40c75b0 con 0x7fc7a40d07b0 2023-10-16T13:47:03.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.671+0000 7fc7a37fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7900050d0 0x7fc7a40c3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:03.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.671+0000 7fc7a37fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7900050d0 0x7fc7a40c3f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54810/0 (socket says 172.21.15.16:54810) 2023-10-16T13:47:03.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.671+0000 7fc7a37fe700 1 -- 172.21.15.16:0/661012556 learned_addr learned my addr 172.21.15.16:0/661012556 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:03.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.672+0000 7fc7a2ffd700 1 --2- 172.21.15.16:0/661012556 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc7a4159970 0x7fc7a40d0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:03.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.672+0000 7fc7a3fff700 1 --2- 172.21.15.16:0/661012556 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc7a40d07b0 0x7fc7a40c97e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:03.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.672+0000 7fc7a37fe700 1 -- 172.21.15.16:0/661012556 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc7a40d07b0 msgr2=0x7fc7a40c97e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:03.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.672+0000 7fc7a37fe700 1 --2- 172.21.15.16:0/661012556 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc7a40d07b0 0x7fc7a40c97e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:03.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.672+0000 7fc7a37fe700 1 -- 172.21.15.16:0/661012556 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc7a4159970 msgr2=0x7fc7a40d0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:03.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.672+0000 7fc7a37fe700 1 --2- 172.21.15.16:0/661012556 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc7a4159970 0x7fc7a40d0270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:03.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.672+0000 7fc7a37fe700 1 -- 172.21.15.16:0/661012556 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7a40c7890 con 0x7fc7900050d0 2023-10-16T13:47:03.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.672+0000 7fc7a37fe700 1 --2- 172.21.15.16:0/661012556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7900050d0 0x7fc7a40c3f70 secure :-1 s=READY pgs=2641 cs=0 l=1 rev1=1 crypto rx=0x7fc79400f780 tx=0x7fc79400fb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:03.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.672+0000 7fc79bfff700 1 -- 172.21.15.16:0/661012556 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7940052e0 con 0x7fc7900050d0 2023-10-16T13:47:03.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.672+0000 7fc79bfff700 1 -- 172.21.15.16:0/661012556 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc794005460 con 0x7fc7900050d0 2023-10-16T13:47:03.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.672+0000 7fc7aa1c0700 1 -- 172.21.15.16:0/661012556 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7a40c7b70 con 0x7fc7900050d0 2023-10-16T13:47:03.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.672+0000 7fc79bfff700 1 -- 172.21.15.16:0/661012556 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7940038c0 con 0x7fc7900050d0 2023-10-16T13:47:03.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.672+0000 7fc7aa1c0700 1 -- 172.21.15.16:0/661012556 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7a40c44b0 con 0x7fc7900050d0 2023-10-16T13:47:03.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.673+0000 7fc7aa1c0700 1 -- 172.21.15.16:0/661012556 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7a40c8090 con 0x7fc7900050d0 2023-10-16T13:47:03.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.674+0000 7fc79bfff700 1 -- 172.21.15.16:0/661012556 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 146) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fc794003a20 con 0x7fc7900050d0 2023-10-16T13:47:03.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.674+0000 7fc79bfff700 1 --2- 172.21.15.16:0/661012556 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fc78406ad50 0x7fc78406d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:03.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.674+0000 7fc79bfff700 1 -- 172.21.15.16:0/661012556 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 7838+0+0 (secure 0 0 0) 0x7fc794089910 con 0x7fc7900050d0 2023-10-16T13:47:03.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.674+0000 7fc7a2ffd700 1 --2- 172.21.15.16:0/661012556 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fc78406ad50 0x7fc78406d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:03.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.674+0000 7fc7a2ffd700 1 --2- 172.21.15.16:0/661012556 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fc78406ad50 0x7fc78406d210 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc7a41f5fd0 tx=0x7fc78c008040 comp rx=0 tx=0).ready entity=mgr.11886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:03.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.678+0000 7fc79bfff700 1 -- 172.21.15.16:0/661012556 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc7a40c8090 con 0x7fc7900050d0 2023-10-16T13:47:03.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.889+0000 7fc7aa1c0700 1 -- 172.21.15.16:0/661012556 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc7a40c8090 con 0x7fc7900050d0 2023-10-16T13:47:03.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.890+0000 7fc79bfff700 1 -- 172.21.15.16:0/661012556 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v231) v1 ==== 74+0+21725 (secure 0 0 0) 0x7fc7a40c8090 con 0x7fc7900050d0 2023-10-16T13:47:03.893 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:47:03.893 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":231,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:47:01.801913+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2023-10-16T13:46:22.489653+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"224","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2023-10-16T13:46:23.729509+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"225","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T13:46:35.800444+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"228","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"backup_fs_data","create_time":"2023-10-16T13:46:37.035323+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"229","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/1827947757":"2023-10-17T13:47:01.801871+0000","172.21.15.16:0/1694897016":"2023-10-17T13:47:01.801871+0000","172.21.15.16:0/3760456275":"2023-10-17T13:47:01.801871+0000","172.21.15.16:0/3651530055":"2023-10-17T13:47:01.801871+0000","172.21.15.16:0/850397235":"2023-10-17T13:47:01.801871+0000","172.21.15.16:6833/1506142504":"2023-10-17T13:47:01.801871+0000","172.21.15.16:0/4239949371":"2023-10-17T13:47:01.801871+0000","172.21.15.16:0/2321929682":"2023-10-17T13:46:51.165018+0000","172.21.15.16:0/3155275276":"2023-10-17T13:47:01.801871+0000","172.21.15.16:0/3429506235":"2023-10-17T13:46:51.165018+0000","172.21.15.16:0/4090083700":"2023-10-17T13:47:01.801871+0000","172.21.15.16:0/3843126404":"2023-10-17T13:46:51.165018+0000","172.21.15.16:6832/1506142504":"2023-10-17T13:47:01.801871+0000","172.21.15.16:0/1439318665":"2023-10-17T13:46:51.165018+0000","172.21.15.16:0/4204021371":"2023-10-17T13:46:51.165018+0000","172.21.15.16:6833/3171422553":"2023-10-17T13:46:51.165018+0000","172.21.15.16:6832/3171422553":"2023-10-17T13:46:51.165018+0000","172.21.15.16:0/1691810194":"2023-10-17T13:47:01.801871+0000","172.21.15.16:0/1632287797":"2023-10-17T13:46:51.165018+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:47:03.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.892+0000 7fc7aa1c0700 1 -- 172.21.15.16:0/661012556 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fc78406ad50 msgr2=0x7fc78406d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:03.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.892+0000 7fc7aa1c0700 1 --2- 172.21.15.16:0/661012556 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fc78406ad50 0x7fc78406d210 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc7a41f5fd0 tx=0x7fc78c008040 comp rx=0 tx=0).stop 2023-10-16T13:47:03.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.892+0000 7fc7aa1c0700 1 -- 172.21.15.16:0/661012556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7900050d0 msgr2=0x7fc7a40c3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:03.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.892+0000 7fc7aa1c0700 1 --2- 172.21.15.16:0/661012556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7900050d0 0x7fc7a40c3f70 secure :-1 s=READY pgs=2641 cs=0 l=1 rev1=1 crypto rx=0x7fc79400f780 tx=0x7fc79400fb40 comp rx=0 tx=0).stop 2023-10-16T13:47:03.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.892+0000 7fc7aa1c0700 1 -- 172.21.15.16:0/661012556 shutdown_connections 2023-10-16T13:47:03.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.892+0000 7fc7aa1c0700 1 --2- 172.21.15.16:0/661012556 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fc78406ad50 0x7fc78406d210 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:03.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.892+0000 7fc7aa1c0700 1 --2- 172.21.15.16:0/661012556 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc7a40d07b0 0x7fc7a40c97e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:03.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.892+0000 7fc7aa1c0700 1 --2- 172.21.15.16:0/661012556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7900050d0 0x7fc7a40c3f70 unknown :-1 s=CLOSED pgs=2641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:03.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.892+0000 7fc7aa1c0700 1 --2- 172.21.15.16:0/661012556 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc7a4159970 0x7fc7a40d0270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:03.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.892+0000 7fc7aa1c0700 1 -- 172.21.15.16:0/661012556 >> 172.21.15.16:0/661012556 conn(0x7fc7a40b1fd0 msgr2=0x7fc7a4161150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:03.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.893+0000 7fc7aa1c0700 1 -- 172.21.15.16:0/661012556 shutdown_connections 2023-10-16T13:47:03.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:03.893+0000 7fc7aa1c0700 1 -- 172.21.15.16:0/661012556 wait complete. 2023-10-16T13:47:03.907 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-16T13:47:04.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.070+0000 7fc5c993a700 1 Processor -- start 2023-10-16T13:47:04.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.071+0000 7fc5c993a700 1 -- start start 2023-10-16T13:47:04.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.071+0000 7fc5c993a700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fc5c415bbf0 0x7fc5c40b2f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:04.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.071+0000 7fc5c993a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc5c40b34c0 0x7fc5c40d00d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:04.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.071+0000 7fc5c993a700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5c40b38c0 con 0x7fc5c415b810 2023-10-16T13:47:04.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.071+0000 7fc5c993a700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5c40b7300 con 0x7fc5c40b34c0 2023-10-16T13:47:04.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.071+0000 7fc5c993a700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5c4160380 con 0x7fc5c415bbf0 2023-10-16T13:47:04.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.071+0000 7fc5c37fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc5c40b34c0 0x7fc5c40d00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:04.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.072+0000 7fc5c37fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc5c40b34c0 0x7fc5c40d00d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54840/0 (socket says 172.21.15.16:54840) 2023-10-16T13:47:04.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.072+0000 7fc5c37fe700 1 -- 172.21.15.16:0/496370122 learned_addr learned my addr 172.21.15.16:0/496370122 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:04.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.072+0000 7fc5c27fc700 1 --2- 172.21.15.16:0/496370122 >> v2:172.21.15.49:3300/0 conn(0x7fc5c415bbf0 0x7fc5c40b2f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:04.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.072+0000 7fc5c37fe700 1 -- 172.21.15.16:0/496370122 >> v2:172.21.15.49:3300/0 conn(0x7fc5c415bbf0 msgr2=0x7fc5c40b2f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:04.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.072+0000 7fc5c37fe700 1 --2- 172.21.15.16:0/496370122 >> v2:172.21.15.49:3300/0 conn(0x7fc5c415bbf0 0x7fc5c40b2f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:04.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.072+0000 7fc5c37fe700 1 -- 172.21.15.16:0/496370122 >> v1:172.21.15.49:6789/0 conn(0x7fc5c415b810 legacy=0x7fc5c415f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:04.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.072+0000 7fc5c37fe700 1 -- 172.21.15.16:0/496370122 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5c41600a0 con 0x7fc5c40b34c0 2023-10-16T13:47:04.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.072+0000 7fc5c37fe700 1 --2- 172.21.15.16:0/496370122 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5c40b34c0 0x7fc5c40d00d0 secure :-1 s=READY pgs=2644 cs=0 l=1 rev1=1 crypto rx=0x7fc5c406e5f0 tx=0x7fc5bc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81d3ceb36a01a974 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:04.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.072+0000 7fc5c1ffb700 1 -- 172.21.15.16:0/496370122 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5bc00e070 con 0x7fc5c40b34c0 2023-10-16T13:47:04.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.072+0000 7fc5c1ffb700 1 -- 172.21.15.16:0/496370122 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5bc003a20 con 0x7fc5c40b34c0 2023-10-16T13:47:04.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.072+0000 7fc5c1ffb700 1 -- 172.21.15.16:0/496370122 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5bc003dd0 con 0x7fc5c40b34c0 2023-10-16T13:47:04.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.073+0000 7fc5c993a700 1 -- 172.21.15.16:0/496370122 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5c40b34c0 msgr2=0x7fc5c40d00d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:04.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.073+0000 7fc5c993a700 1 --2- 172.21.15.16:0/496370122 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5c40b34c0 0x7fc5c40d00d0 secure :-1 s=READY pgs=2644 cs=0 l=1 rev1=1 crypto rx=0x7fc5c406e5f0 tx=0x7fc5bc00b330 comp rx=0 tx=0).stop 2023-10-16T13:47:04.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.073+0000 7fc5c993a700 1 -- 172.21.15.16:0/496370122 shutdown_connections 2023-10-16T13:47:04.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.073+0000 7fc5c993a700 1 --2- 172.21.15.16:0/496370122 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5c40b34c0 0x7fc5c40d00d0 unknown :-1 s=CLOSED pgs=2644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:04.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.073+0000 7fc5c993a700 1 --2- 172.21.15.16:0/496370122 >> v2:172.21.15.49:3300/0 conn(0x7fc5c415bbf0 0x7fc5c40b2f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:04.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.073+0000 7fc5c993a700 1 -- 172.21.15.16:0/496370122 >> 172.21.15.16:0/496370122 conn(0x7fc5c40cdf80 msgr2=0x7fc5c40ce390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:04.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.073+0000 7fc5c993a700 1 -- 172.21.15.16:0/496370122 shutdown_connections 2023-10-16T13:47:04.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.073+0000 7fc5c993a700 1 -- 172.21.15.16:0/496370122 wait complete. 2023-10-16T13:47:04.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.074+0000 7fc5c993a700 1 Processor -- start 2023-10-16T13:47:04.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.074+0000 7fc5c993a700 1 -- start start 2023-10-16T13:47:04.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.075+0000 7fc5c993a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc5c40b34c0 0x7fc5c41f1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:04.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.075+0000 7fc5c993a700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc5c415b810 0x7fc5c40c8540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:04.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.075+0000 7fc5c993a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5c415bbf0 0x7fc5c40b56d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:04.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.075+0000 7fc5c993a700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c40c9530 con 0x7fc5c415bbf0 2023-10-16T13:47:04.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.075+0000 7fc5c993a700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c4150130 con 0x7fc5c415b810 2023-10-16T13:47:04.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.075+0000 7fc5c993a700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c40c8a80 con 0x7fc5c40b34c0 2023-10-16T13:47:04.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.075+0000 7fc5c37fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5c415bbf0 0x7fc5c40b56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:04.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.075+0000 7fc5c37fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5c415bbf0 0x7fc5c40b56d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54844/0 (socket says 172.21.15.16:54844) 2023-10-16T13:47:04.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.075+0000 7fc5c37fe700 1 -- 172.21.15.16:0/2519299136 learned_addr learned my addr 172.21.15.16:0/2519299136 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:04.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.075+0000 7fc5c2ffd700 1 --2- 172.21.15.16:0/2519299136 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc5c40b34c0 0x7fc5c41f1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:04.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.075+0000 7fc5c27fc700 1 --2- 172.21.15.16:0/2519299136 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc5c415b810 0x7fc5c40c8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:04.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.075+0000 7fc5c37fe700 1 -- 172.21.15.16:0/2519299136 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc5c40b34c0 msgr2=0x7fc5c41f1ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:04.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.075+0000 7fc5c37fe700 1 --2- 172.21.15.16:0/2519299136 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc5c40b34c0 0x7fc5c41f1ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:04.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.075+0000 7fc5c37fe700 1 -- 172.21.15.16:0/2519299136 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc5c415b810 msgr2=0x7fc5c40c8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:04.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.075+0000 7fc5c37fe700 1 --2- 172.21.15.16:0/2519299136 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc5c415b810 0x7fc5c40c8540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:04.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.075+0000 7fc5c37fe700 1 -- 172.21.15.16:0/2519299136 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5bc009b30 con 0x7fc5c415bbf0 2023-10-16T13:47:04.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.076+0000 7fc5c37fe700 1 --2- 172.21.15.16:0/2519299136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5c415bbf0 0x7fc5c40b56d0 secure :-1 s=READY pgs=2645 cs=0 l=1 rev1=1 crypto rx=0x7fc5bc015040 tx=0x7fc5bc004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:04.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.076+0000 7fc5bbfff700 1 -- 172.21.15.16:0/2519299136 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5bc004440 con 0x7fc5c415bbf0 2023-10-16T13:47:04.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.076+0000 7fc5bbfff700 1 -- 172.21.15.16:0/2519299136 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5bc0045c0 con 0x7fc5c415bbf0 2023-10-16T13:47:04.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.076+0000 7fc5c993a700 1 -- 172.21.15.16:0/2519299136 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5c40c8c90 con 0x7fc5c415bbf0 2023-10-16T13:47:04.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.076+0000 7fc5bbfff700 1 -- 172.21.15.16:0/2519299136 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5bc004970 con 0x7fc5c415bbf0 2023-10-16T13:47:04.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.076+0000 7fc5c993a700 1 -- 172.21.15.16:0/2519299136 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5c40c9250 con 0x7fc5c415bbf0 2023-10-16T13:47:04.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.077+0000 7fc5c993a700 1 -- 172.21.15.16:0/2519299136 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5c404ebb0 con 0x7fc5c415bbf0 2023-10-16T13:47:04.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.077+0000 7fc5bbfff700 1 -- 172.21.15.16:0/2519299136 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 146) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fc5bc012430 con 0x7fc5c415bbf0 2023-10-16T13:47:04.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.078+0000 7fc5bbfff700 1 --2- 172.21.15.16:0/2519299136 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fc5a406ad50 0x7fc5a406d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:04.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.078+0000 7fc5bbfff700 1 -- 172.21.15.16:0/2519299136 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 7838+0+0 (secure 0 0 0) 0x7fc5bc08af20 con 0x7fc5c415bbf0 2023-10-16T13:47:04.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.078+0000 7fc5c2ffd700 1 --2- 172.21.15.16:0/2519299136 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fc5a406ad50 0x7fc5a406d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:04.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.078+0000 7fc5c2ffd700 1 --2- 172.21.15.16:0/2519299136 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fc5a406ad50 0x7fc5a406d210 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc5b4004f30 tx=0x7fc5b4009040 comp rx=0 tx=0).ready entity=mgr.11886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:04.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.082+0000 7fc5bbfff700 1 -- 172.21.15.16:0/2519299136 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc5bc055ac0 con 0x7fc5c415bbf0 2023-10-16T13:47:04.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.336+0000 7fc5c993a700 1 -- 172.21.15.16:0/2519299136 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fc5c40cb700 con 0x7fc5c415bbf0 2023-10-16T13:47:04.339 INFO:tasks.ceph.mon.a.smithi016.stderr:2023-10-16T13:47:04.337+0000 7f9fc8535700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-16T13:47:04.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.338+0000 7fc5bbfff700 1 -- 172.21.15.16:0/2519299136 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v120) v1 ==== 161+0+0 (secure 0 0 0) 0x7fc5bc017090 con 0x7fc5c415bbf0 2023-10-16T13:47:04.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.342+0000 7fc5c993a700 1 -- 172.21.15.16:0/2519299136 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fc5a406ad50 msgr2=0x7fc5a406d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:04.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.342+0000 7fc5c993a700 1 --2- 172.21.15.16:0/2519299136 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fc5a406ad50 0x7fc5a406d210 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc5b4004f30 tx=0x7fc5b4009040 comp rx=0 tx=0).stop 2023-10-16T13:47:04.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.343+0000 7fc5c993a700 1 -- 172.21.15.16:0/2519299136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5c415bbf0 msgr2=0x7fc5c40b56d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:04.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.343+0000 7fc5c993a700 1 --2- 172.21.15.16:0/2519299136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5c415bbf0 0x7fc5c40b56d0 secure :-1 s=READY pgs=2645 cs=0 l=1 rev1=1 crypto rx=0x7fc5bc015040 tx=0x7fc5bc004ad0 comp rx=0 tx=0).stop 2023-10-16T13:47:04.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.343+0000 7fc5c993a700 1 -- 172.21.15.16:0/2519299136 shutdown_connections 2023-10-16T13:47:04.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.343+0000 7fc5c993a700 1 --2- 172.21.15.16:0/2519299136 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fc5a406ad50 0x7fc5a406d210 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:04.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.343+0000 7fc5c993a700 1 --2- 172.21.15.16:0/2519299136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5c415bbf0 0x7fc5c40b56d0 unknown :-1 s=CLOSED pgs=2645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:04.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.343+0000 7fc5c993a700 1 --2- 172.21.15.16:0/2519299136 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc5c40b34c0 0x7fc5c41f1ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:04.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.343+0000 7fc5c993a700 1 --2- 172.21.15.16:0/2519299136 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc5c415b810 0x7fc5c40c8540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:04.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.343+0000 7fc5c993a700 1 -- 172.21.15.16:0/2519299136 >> 172.21.15.16:0/2519299136 conn(0x7fc5c40cdf80 msgr2=0x7fc5c415a700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:04.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.343+0000 7fc5c993a700 1 -- 172.21.15.16:0/2519299136 shutdown_connections 2023-10-16T13:47:04.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.343+0000 7fc5c993a700 1 -- 172.21.15.16:0/2519299136 wait complete. 2023-10-16T13:47:04.348 INFO:tasks.ceph.mds.b.smithi049.stderr: -5445> 2023-10-16T13:46:31.550+0000 7fa1c43aa700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T13:47:04.348 INFO:teuthology.orchestra.run.smithi016.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-16T13:47:04.358 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-16T13:47:04.380 INFO:tasks.ceph.mds.b.smithi049.stdout:starting mds.b at 2023-10-16T13:47:04.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.483+0000 7f5020455700 1 Processor -- start 2023-10-16T13:47:04.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.483+0000 7f5020455700 1 -- start start 2023-10-16T13:47:04.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.484+0000 7f5020455700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f50180cb880 0x7f50180cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:04.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.484+0000 7f5020455700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f501806da60 0x7f501806def0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:04.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.484+0000 7f5020455700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50180cef70 con 0x7f501815b070 2023-10-16T13:47:04.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.484+0000 7f5020455700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f501806e430 con 0x7f501806da60 2023-10-16T13:47:04.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.484+0000 7f5020455700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50180ce360 con 0x7f50180cb880 2023-10-16T13:47:04.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.484+0000 7f501e9f2700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f501806da60 0x7f501806def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:04.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.484+0000 7f501e9f2700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f501806da60 0x7f501806def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54852/0 (socket says 172.21.15.16:54852) 2023-10-16T13:47:04.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.484+0000 7f501e9f2700 1 -- 172.21.15.16:0/823095837 learned_addr learned my addr 172.21.15.16:0/823095837 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:04.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.484+0000 7f501e1f1700 1 --1- 172.21.15.16:0/823095837 >> v1:172.21.15.16:6789/0 conn(0x7f501815b070 0x7f501815b450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44122/0 (socket says 172.21.15.16:44122) 2023-10-16T13:47:04.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.484+0000 7f501d9f0700 1 --2- 172.21.15.16:0/823095837 >> v2:172.21.15.49:3300/0 conn(0x7f50180cb880 0x7f50180cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:04.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.484+0000 7f501e9f2700 1 -- 172.21.15.16:0/823095837 >> v2:172.21.15.49:3300/0 conn(0x7f50180cb880 msgr2=0x7f50180cbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:04.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.484+0000 7f501e9f2700 1 --2- 172.21.15.16:0/823095837 >> v2:172.21.15.49:3300/0 conn(0x7f50180cb880 0x7f50180cbd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:04.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.484+0000 7f501e9f2700 1 -- 172.21.15.16:0/823095837 >> v1:172.21.15.16:6789/0 conn(0x7f501815b070 legacy=0x7f501815b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:04.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.484+0000 7f501e9f2700 1 -- 172.21.15.16:0/823095837 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50180ce640 con 0x7f501806da60 2023-10-16T13:47:04.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.485+0000 7f501e9f2700 1 --2- 172.21.15.16:0/823095837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f501806da60 0x7f501806def0 secure :-1 s=READY pgs=2648 cs=0 l=1 rev1=1 crypto rx=0x7f5018070500 tx=0x7f501000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c24c51df37f81f40 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:04.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.485+0000 7f501d1ef700 1 -- 172.21.15.16:0/823095837 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f501000e070 con 0x7f501806da60 2023-10-16T13:47:04.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.485+0000 7f501d1ef700 1 -- 172.21.15.16:0/823095837 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f501000b910 con 0x7f501806da60 2023-10-16T13:47:04.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.485+0000 7f501d1ef700 1 -- 172.21.15.16:0/823095837 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f501000bcc0 con 0x7f501806da60 2023-10-16T13:47:04.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.485+0000 7f5020455700 1 -- 172.21.15.16:0/823095837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f501806da60 msgr2=0x7f501806def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:04.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.485+0000 7f5020455700 1 --2- 172.21.15.16:0/823095837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f501806da60 0x7f501806def0 secure :-1 s=READY pgs=2648 cs=0 l=1 rev1=1 crypto rx=0x7f5018070500 tx=0x7f501000b330 comp rx=0 tx=0).stop 2023-10-16T13:47:04.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.485+0000 7f5020455700 1 -- 172.21.15.16:0/823095837 shutdown_connections 2023-10-16T13:47:04.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.485+0000 7f5020455700 1 --2- 172.21.15.16:0/823095837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f501806da60 0x7f501806def0 unknown :-1 s=CLOSED pgs=2648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:04.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.485+0000 7f5020455700 1 --2- 172.21.15.16:0/823095837 >> v2:172.21.15.49:3300/0 conn(0x7f50180cb880 0x7f50180cbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:04.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.485+0000 7f5020455700 1 -- 172.21.15.16:0/823095837 >> 172.21.15.16:0/823095837 conn(0x7f50180b1f40 msgr2=0x7f50180b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:04.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.485+0000 7f5020455700 1 -- 172.21.15.16:0/823095837 shutdown_connections 2023-10-16T13:47:04.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.486+0000 7f5020455700 1 -- 172.21.15.16:0/823095837 wait complete. 2023-10-16T13:47:04.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.487+0000 7f5020455700 1 Processor -- start 2023-10-16T13:47:04.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.487+0000 7f5020455700 1 -- start start 2023-10-16T13:47:04.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.487+0000 7f5020455700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f501806da60 0x7f5018153d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:04.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.487+0000 7f5020455700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50180cb880 0x7f501815c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:04.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.487+0000 7f5020455700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f501815b070 0x7f5018155190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:04.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.487+0000 7f5020455700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5018154570 con 0x7f50180cb880 2023-10-16T13:47:04.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.487+0000 7f5020455700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50181538c0 con 0x7f501806da60 2023-10-16T13:47:04.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.487+0000 7f5020455700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50180ca570 con 0x7f501815b070 2023-10-16T13:47:04.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.487+0000 7f501d9f0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50180cb880 0x7f501815c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:04.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.487+0000 7f501d9f0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50180cb880 0x7f501815c710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54856/0 (socket says 172.21.15.16:54856) 2023-10-16T13:47:04.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.488+0000 7f501d9f0700 1 -- 172.21.15.16:0/3870016814 learned_addr learned my addr 172.21.15.16:0/3870016814 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:04.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.488+0000 7f501e9f2700 1 --2- 172.21.15.16:0/3870016814 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f501815b070 0x7f5018155190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:04.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.488+0000 7f501e1f1700 1 --2- 172.21.15.16:0/3870016814 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f501806da60 0x7f5018153d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:04.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.488+0000 7f501d9f0700 1 -- 172.21.15.16:0/3870016814 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f501815b070 msgr2=0x7f5018155190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:04.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.488+0000 7f501d9f0700 1 --2- 172.21.15.16:0/3870016814 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f501815b070 0x7f5018155190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:04.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.488+0000 7f501d9f0700 1 -- 172.21.15.16:0/3870016814 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f501806da60 msgr2=0x7f5018153d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:04.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.488+0000 7f501d9f0700 1 --2- 172.21.15.16:0/3870016814 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f501806da60 0x7f5018153d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:04.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.488+0000 7f501d9f0700 1 -- 172.21.15.16:0/3870016814 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5010009b30 con 0x7f50180cb880 2023-10-16T13:47:04.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.488+0000 7f501d9f0700 1 --2- 172.21.15.16:0/3870016814 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50180cb880 0x7f501815c710 secure :-1 s=READY pgs=2649 cs=0 l=1 rev1=1 crypto rx=0x7f501400e9b0 tx=0x7f501400ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:04.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.488+0000 7f500effd700 1 -- 172.21.15.16:0/3870016814 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5014009710 con 0x7f50180cb880 2023-10-16T13:47:04.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.488+0000 7f500effd700 1 -- 172.21.15.16:0/3870016814 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f501400c850 con 0x7f50180cb880 2023-10-16T13:47:04.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.488+0000 7f5020455700 1 -- 172.21.15.16:0/3870016814 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50180c9fb0 con 0x7f50180cb880 2023-10-16T13:47:04.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.488+0000 7f500effd700 1 -- 172.21.15.16:0/3870016814 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f501400cbc0 con 0x7f50180cb880 2023-10-16T13:47:04.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.488+0000 7f5020455700 1 -- 172.21.15.16:0/3870016814 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50180c9c60 con 0x7f50180cb880 2023-10-16T13:47:04.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.489+0000 7f5020455700 1 -- 172.21.15.16:0/3870016814 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f501804ebb0 con 0x7f50180cb880 2023-10-16T13:47:04.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.490+0000 7f500effd700 1 -- 172.21.15.16:0/3870016814 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 146) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f501400cd20 con 0x7f50180cb880 2023-10-16T13:47:04.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.490+0000 7f500effd700 1 --2- 172.21.15.16:0/3870016814 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f4ffc06ad50 0x7f4ffc06d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:04.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.490+0000 7f500effd700 1 -- 172.21.15.16:0/3870016814 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 7924+0+0 (secure 0 0 0) 0x7f5014089c10 con 0x7f50180cb880 2023-10-16T13:47:04.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.490+0000 7f501e1f1700 1 --2- 172.21.15.16:0/3870016814 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f4ffc06ad50 0x7f4ffc06d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:04.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.491+0000 7f501e1f1700 1 --2- 172.21.15.16:0/3870016814 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f4ffc06ad50 0x7f4ffc06d210 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5008000f60 tx=0x7f5008009040 comp rx=0 tx=0).ready entity=mgr.11886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:04.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.495+0000 7f500effd700 1 -- 172.21.15.16:0/3870016814 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5014054750 con 0x7f50180cb880 2023-10-16T13:47:04.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.750+0000 7f5020455700 1 -- 172.21.15.16:0/3870016814 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f50180b2c10 con 0x7f50180cb880 2023-10-16T13:47:04.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:04.816+0000 7f500effd700 1 -- 172.21.15.16:0/3870016814 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 147) v1 ==== 89377+0+0 (secure 0 0 0) 0x7f5014054b20 con 0x7f50180cb880 2023-10-16T13:47:05.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:05.340+0000 7f500effd700 1 -- 172.21.15.16:0/3870016814 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v121) v1 ==== 104+0+0 (secure 0 0 0) 0x7f5014087820 con 0x7f50180cb880 2023-10-16T13:47:05.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:05.343+0000 7f5020455700 1 -- 172.21.15.16:0/3870016814 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f4ffc06ad50 msgr2=0x7f4ffc06d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:05.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:05.343+0000 7f5020455700 1 --2- 172.21.15.16:0/3870016814 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f4ffc06ad50 0x7f4ffc06d210 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5008000f60 tx=0x7f5008009040 comp rx=0 tx=0).stop 2023-10-16T13:47:05.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:05.343+0000 7f5020455700 1 -- 172.21.15.16:0/3870016814 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50180cb880 msgr2=0x7f501815c710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:05.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:05.343+0000 7f5020455700 1 --2- 172.21.15.16:0/3870016814 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50180cb880 0x7f501815c710 secure :-1 s=READY pgs=2649 cs=0 l=1 rev1=1 crypto rx=0x7f501400e9b0 tx=0x7f501400ed70 comp rx=0 tx=0).stop 2023-10-16T13:47:05.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:05.343+0000 7f5020455700 1 -- 172.21.15.16:0/3870016814 shutdown_connections 2023-10-16T13:47:05.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:05.343+0000 7f5020455700 1 --2- 172.21.15.16:0/3870016814 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f4ffc06ad50 0x7f4ffc06d210 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:05.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:05.343+0000 7f5020455700 1 --2- 172.21.15.16:0/3870016814 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f501815b070 0x7f5018155190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:05.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:05.343+0000 7f5020455700 1 --2- 172.21.15.16:0/3870016814 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f501806da60 0x7f5018153d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:05.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:05.343+0000 7f5020455700 1 --2- 172.21.15.16:0/3870016814 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50180cb880 0x7f501815c710 unknown :-1 s=CLOSED pgs=2649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:05.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:05.343+0000 7f5020455700 1 -- 172.21.15.16:0/3870016814 >> 172.21.15.16:0/3870016814 conn(0x7f50180b1f40 msgr2=0x7f50180d0380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:05.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:05.343+0000 7f5020455700 1 -- 172.21.15.16:0/3870016814 shutdown_connections 2023-10-16T13:47:05.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:05.343+0000 7f5020455700 1 -- 172.21.15.16:0/3870016814 wait complete. 2023-10-16T13:47:05.361 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-16T13:47:05.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:05.494+0000 7f2f81dba700 1 Processor -- start 2023-10-16T13:47:05.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:05.494+0000 7f2f81dba700 1 -- start start 2023-10-16T13:47:05.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:05.495+0000 7f2f81dba700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f2f7c0cf2c0 0x7f2f7c0aef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:05.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:05.495+0000 7f2f81dba700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2f7c0cbde0 0x7f2f7c0c5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:05.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:05.495+0000 7f2f81dba700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f7c0c49b0 con 0x7f2f7c0cd600 2023-10-16T13:47:05.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:05.495+0000 7f2f81dba700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f7c0c4c20 con 0x7f2f7c0cbde0 2023-10-16T13:47:05.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:05.495+0000 7f2f81dba700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f7c0c4f00 con 0x7f2f7c0cf2c0 2023-10-16T13:47:05.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:05.495+0000 7f2f7bfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2f7c0cbde0 0x7f2f7c0c5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:05.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:05.495+0000 7f2f7bfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2f7c0cbde0 0x7f2f7c0c5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54866/0 (socket says 172.21.15.16:54866) 2023-10-16T13:47:05.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:05.495+0000 7f2f7bfff700 1 -- 172.21.15.16:0/1128169825 learned_addr learned my addr 172.21.15.16:0/1128169825 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:05.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:05.495+0000 7f2f7affd700 1 --1- 172.21.15.16:0/1128169825 >> v1:172.21.15.16:6789/0 conn(0x7f2f7c0cd600 0x7f2f7c0cb6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44138/0 (socket says 172.21.15.16:44138) 2023-10-16T13:47:05.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:05.495+0000 7f2f7b7fe700 1 --2- 172.21.15.16:0/1128169825 >> v2:172.21.15.49:3300/0 conn(0x7f2f7c0cf2c0 0x7f2f7c0aef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:05.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:05.495+0000 7f2f7bfff700 1 -- 172.21.15.16:0/1128169825 >> v2:172.21.15.49:3300/0 conn(0x7f2f7c0cf2c0 msgr2=0x7f2f7c0aef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:05.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:05.495+0000 7f2f7bfff700 1 --2- 172.21.15.16:0/1128169825 >> v2:172.21.15.49:3300/0 conn(0x7f2f7c0cf2c0 0x7f2f7c0aef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:05.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:05.495+0000 7f2f7bfff700 1 -- 172.21.15.16:0/1128169825 >> v1:172.21.15.16:6789/0 conn(0x7f2f7c0cd600 legacy=0x7f2f7c0cb6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:05.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:05.495+0000 7f2f7bfff700 1 -- 172.21.15.16:0/1128169825 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f7c0c51e0 con 0x7f2f7c0cbde0 2023-10-16T13:47:05.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:05.496+0000 7f2f7bfff700 1 --2- 172.21.15.16:0/1128169825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f7c0cbde0 0x7f2f7c0c5700 secure :-1 s=READY pgs=2651 cs=0 l=1 rev1=1 crypto rx=0x7f2f7c0cd3c0 tx=0x7f2f7400d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fbdd2fdcfd8603a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:05.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:05.496+0000 7f2f7a7fc700 1 -- 172.21.15.16:0/1128169825 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f74010070 con 0x7f2f7c0cbde0 2023-10-16T13:47:05.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:05.496+0000 7f2f7a7fc700 1 -- 172.21.15.16:0/1128169825 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f7400dce0 con 0x7f2f7c0cbde0 2023-10-16T13:47:05.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:05.496+0000 7f2f7a7fc700 1 -- 172.21.15.16:0/1128169825 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f7400b310 con 0x7f2f7c0cbde0 2023-10-16T13:47:05.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:05.496+0000 7f2f81dba700 1 -- 172.21.15.16:0/1128169825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f7c0cbde0 msgr2=0x7f2f7c0c5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:05.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:05.496+0000 7f2f81dba700 1 --2- 172.21.15.16:0/1128169825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f7c0cbde0 0x7f2f7c0c5700 secure :-1 s=READY pgs=2651 cs=0 l=1 rev1=1 crypto rx=0x7f2f7c0cd3c0 tx=0x7f2f7400d430 comp rx=0 tx=0).stop 2023-10-16T13:47:05.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:05.496+0000 7f2f81dba700 1 -- 172.21.15.16:0/1128169825 shutdown_connections 2023-10-16T13:47:05.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:05.496+0000 7f2f81dba700 1 --2- 172.21.15.16:0/1128169825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f7c0cbde0 0x7f2f7c0c5700 unknown :-1 s=CLOSED pgs=2651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:05.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:05.496+0000 7f2f81dba700 1 --2- 172.21.15.16:0/1128169825 >> v2:172.21.15.49:3300/0 conn(0x7f2f7c0cf2c0 0x7f2f7c0aef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:05.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:05.496+0000 7f2f81dba700 1 -- 172.21.15.16:0/1128169825 >> 172.21.15.16:0/1128169825 conn(0x7f2f7c0cc4b0 msgr2=0x7f2f7c0cc8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:05.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:05.496+0000 7f2f81dba700 1 -- 172.21.15.16:0/1128169825 shutdown_connections 2023-10-16T13:47:05.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:05.496+0000 7f2f81dba700 1 -- 172.21.15.16:0/1128169825 wait complete. 2023-10-16T13:47:05.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:05.498+0000 7f2f81dba700 1 Processor -- start 2023-10-16T13:47:05.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:05.498+0000 7f2f81dba700 1 -- start start 2023-10-16T13:47:05.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:05.498+0000 7f2f81dba700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2f7c0cbde0 0x7f2f7c150110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:05.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:05.499+0000 7f2f81dba700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f7c0cd600 0x7f2f7c06d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:05.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:05.499+0000 7f2f81dba700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2f7c0cf2c0 0x7f2f7c15dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:05.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:05.499+0000 7f2f81dba700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f7c06e5d0 con 0x7f2f7c0cd600 2023-10-16T13:47:05.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:05.499+0000 7f2f81dba700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f7c06ea60 con 0x7f2f7c0cbde0 2023-10-16T13:47:05.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:05.499+0000 7f2f81dba700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f7c15e530 con 0x7f2f7c0cf2c0 2023-10-16T13:47:05.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:05.499+0000 7f2f7affd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f7c0cd600 0x7f2f7c06d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:05.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:05.499+0000 7f2f7affd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f7c0cd600 0x7f2f7c06d2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54870/0 (socket says 172.21.15.16:54870) 2023-10-16T13:47:05.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:05.499+0000 7f2f7affd700 1 -- 172.21.15.16:0/152570803 learned_addr learned my addr 172.21.15.16:0/152570803 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:05.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:05.499+0000 7f2f7b7fe700 1 --2- 172.21.15.16:0/152570803 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2f7c0cbde0 0x7f2f7c150110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:05.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:05.499+0000 7f2f7bfff700 1 --2- 172.21.15.16:0/152570803 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2f7c0cf2c0 0x7f2f7c15dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:05.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:05.499+0000 7f2f7affd700 1 -- 172.21.15.16:0/152570803 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2f7c0cf2c0 msgr2=0x7f2f7c15dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:05.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:05.499+0000 7f2f7affd700 1 --2- 172.21.15.16:0/152570803 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2f7c0cf2c0 0x7f2f7c15dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:05.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:05.499+0000 7f2f7affd700 1 -- 172.21.15.16:0/152570803 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2f7c0cbde0 msgr2=0x7f2f7c150110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:05.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:05.499+0000 7f2f7affd700 1 --2- 172.21.15.16:0/152570803 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2f7c0cbde0 0x7f2f7c150110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:05.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:05.499+0000 7f2f7affd700 1 -- 172.21.15.16:0/152570803 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f7400bd40 con 0x7f2f7c0cd600 2023-10-16T13:47:05.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:05.499+0000 7f2f7b7fe700 1 --2- 172.21.15.16:0/152570803 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2f7c0cbde0 0x7f2f7c150110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:47:05.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:05.499+0000 7f2f7affd700 1 --2- 172.21.15.16:0/152570803 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f7c0cd600 0x7f2f7c06d2e0 secure :-1 s=READY pgs=2652 cs=0 l=1 rev1=1 crypto rx=0x7f2f6400c520 tx=0x7f2f6400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:05.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:05.499+0000 7f2f73fff700 1 -- 172.21.15.16:0/152570803 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f64005cb0 con 0x7f2f7c0cd600 2023-10-16T13:47:05.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:05.499+0000 7f2f73fff700 1 -- 172.21.15.16:0/152570803 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f64005e10 con 0x7f2f7c0cd600 2023-10-16T13:47:05.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:05.499+0000 7f2f73fff700 1 -- 172.21.15.16:0/152570803 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f640072d0 con 0x7f2f7c0cd600 2023-10-16T13:47:05.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:05.499+0000 7f2f81dba700 1 -- 172.21.15.16:0/152570803 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f7c06d9c0 con 0x7f2f7c0cd600 2023-10-16T13:47:05.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:05.499+0000 7f2f81dba700 1 -- 172.21.15.16:0/152570803 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f7c06df80 con 0x7f2f7c0cd600 2023-10-16T13:47:05.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:05.500+0000 7f2f81dba700 1 -- 172.21.15.16:0/152570803 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f7c06e780 con 0x7f2f7c0cd600 2023-10-16T13:47:05.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:05.501+0000 7f2f73fff700 1 -- 172.21.15.16:0/152570803 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 147) v1 ==== 89377+0+0 (secure 0 0 0) 0x7f2f64007430 con 0x7f2f7c0cd600 2023-10-16T13:47:05.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:05.501+0000 7f2f73fff700 1 --2- 172.21.15.16:0/152570803 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f2f5c06b0b0 0x7f2f5c06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:05.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:05.501+0000 7f2f73fff700 1 -- 172.21.15.16:0/152570803 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 7924+0+0 (secure 0 0 0) 0x7f2f64089520 con 0x7f2f7c0cd600 2023-10-16T13:47:05.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:05.501+0000 7f2f7b7fe700 1 --2- 172.21.15.16:0/152570803 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f2f5c06b0b0 0x7f2f5c06d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:05.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:05.502+0000 7f2f7b7fe700 1 --2- 172.21.15.16:0/152570803 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f2f5c06b0b0 0x7f2f5c06d570 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c009ac0 tx=0x7f2f6c008040 comp rx=0 tx=0).ready entity=mgr.11886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:05.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:05.506+0000 7f2f73fff700 1 -- 172.21.15.16:0/152570803 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2f7c06e780 con 0x7f2f7c0cd600 2023-10-16T13:47:05.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:05.711+0000 7f2f81dba700 1 -- 172.21.15.16:0/152570803 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f2f7c06e780 con 0x7f2f7c0cd600 2023-10-16T13:47:06.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:06.344+0000 7f2f73fff700 1 -- 172.21.15.16:0/152570803 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v233) v1 ==== 181+0+0 (secure 0 0 0) 0x7f2f7c06e780 con 0x7f2f7c0cd600 2023-10-16T13:47:06.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:06.350+0000 7f2f81dba700 1 -- 172.21.15.16:0/152570803 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f2f5c06b0b0 msgr2=0x7f2f5c06d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:06.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:06.350+0000 7f2f81dba700 1 --2- 172.21.15.16:0/152570803 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f2f5c06b0b0 0x7f2f5c06d570 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c009ac0 tx=0x7f2f6c008040 comp rx=0 tx=0).stop 2023-10-16T13:47:06.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:06.350+0000 7f2f81dba700 1 -- 172.21.15.16:0/152570803 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f7c0cd600 msgr2=0x7f2f7c06d2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:06.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:06.350+0000 7f2f81dba700 1 --2- 172.21.15.16:0/152570803 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f7c0cd600 0x7f2f7c06d2e0 secure :-1 s=READY pgs=2652 cs=0 l=1 rev1=1 crypto rx=0x7f2f6400c520 tx=0x7f2f6400c8e0 comp rx=0 tx=0).stop 2023-10-16T13:47:06.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:06.350+0000 7f2f81dba700 1 -- 172.21.15.16:0/152570803 shutdown_connections 2023-10-16T13:47:06.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:06.350+0000 7f2f81dba700 1 --2- 172.21.15.16:0/152570803 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f2f5c06b0b0 0x7f2f5c06d570 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:06.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:06.350+0000 7f2f81dba700 1 --2- 172.21.15.16:0/152570803 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2f7c0cf2c0 0x7f2f7c15dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:06.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:06.350+0000 7f2f81dba700 1 --2- 172.21.15.16:0/152570803 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2f7c0cbde0 0x7f2f7c150110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:06.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:06.350+0000 7f2f81dba700 1 --2- 172.21.15.16:0/152570803 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f7c0cd600 0x7f2f7c06d2e0 secure :-1 s=CLOSED pgs=2652 cs=0 l=1 rev1=1 crypto rx=0x7f2f6400c520 tx=0x7f2f6400c8e0 comp rx=0 tx=0).stop 2023-10-16T13:47:06.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:06.351+0000 7f2f81dba700 1 -- 172.21.15.16:0/152570803 >> 172.21.15.16:0/152570803 conn(0x7f2f7c0cc4b0 msgr2=0x7f2f7c0b0f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:06.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:06.355+0000 7f2f81dba700 1 -- 172.21.15.16:0/152570803 shutdown_connections 2023-10-16T13:47:06.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:06.355+0000 7f2f81dba700 1 -- 172.21.15.16:0/152570803 wait complete. 2023-10-16T13:47:06.361 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' removed 2023-10-16T13:47:06.376 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-16T13:47:06.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:06.508+0000 7fbaf5ebf700 1 Processor -- start 2023-10-16T13:47:06.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:06.509+0000 7fbaf5ebf700 1 -- start start 2023-10-16T13:47:06.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:06.509+0000 7fbaf5ebf700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fbaf015d920 0x7fbaf00cdef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:06.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:06.509+0000 7fbaf5ebf700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbaf00726b0 0x7fbaf006a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:06.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:06.509+0000 7fbaf5ebf700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbaf00ceb90 con 0x7fbaf006ad20 2023-10-16T13:47:06.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:06.509+0000 7fbaf5ebf700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbaf015dd00 con 0x7fbaf00726b0 2023-10-16T13:47:06.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:06.509+0000 7fbaf5ebf700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbaf006b150 con 0x7fbaf015d920 2023-10-16T13:47:06.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:06.509+0000 7fbaeeffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbaf00726b0 0x7fbaf006a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:06.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:06.510+0000 7fbaeeffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbaf00726b0 0x7fbaf006a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54876/0 (socket says 172.21.15.16:54876) 2023-10-16T13:47:06.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:06.510+0000 7fbaeeffd700 1 -- 172.21.15.16:0/2342027129 learned_addr learned my addr 172.21.15.16:0/2342027129 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:06.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:06.510+0000 7fbaef7fe700 1 --2- 172.21.15.16:0/2342027129 >> v2:172.21.15.49:3300/0 conn(0x7fbaf015d920 0x7fbaf00cdef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:06.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:06.510+0000 7fbaeeffd700 1 -- 172.21.15.16:0/2342027129 >> v2:172.21.15.49:3300/0 conn(0x7fbaf015d920 msgr2=0x7fbaf00cdef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:06.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:06.510+0000 7fbaeeffd700 1 --2- 172.21.15.16:0/2342027129 >> v2:172.21.15.49:3300/0 conn(0x7fbaf015d920 0x7fbaf00cdef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:06.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:06.510+0000 7fbaeeffd700 1 -- 172.21.15.16:0/2342027129 >> v1:172.21.15.16:6789/0 conn(0x7fbaf006ad20 legacy=0x7fbaf00ccd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:06.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:06.510+0000 7fbaeeffd700 1 -- 172.21.15.16:0/2342027129 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaf00ce5d0 con 0x7fbaf00726b0 2023-10-16T13:47:06.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:06.510+0000 7fbaeeffd700 1 --2- 172.21.15.16:0/2342027129 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbaf00726b0 0x7fbaf006a7e0 secure :-1 s=READY pgs=2654 cs=0 l=1 rev1=1 crypto rx=0x7fbaf00b2f40 tx=0x7fbae0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37508004aec84c7c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:06.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:06.510+0000 7fbaee7fc700 1 -- 172.21.15.16:0/2342027129 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbae0011830 con 0x7fbaf00726b0 2023-10-16T13:47:06.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:06.510+0000 7fbaee7fc700 1 -- 172.21.15.16:0/2342027129 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbae0011990 con 0x7fbaf00726b0 2023-10-16T13:47:06.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:06.510+0000 7fbaee7fc700 1 -- 172.21.15.16:0/2342027129 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbae0011d20 con 0x7fbaf00726b0 2023-10-16T13:47:06.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:06.511+0000 7fbaf5ebf700 1 -- 172.21.15.16:0/2342027129 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbaf00726b0 msgr2=0x7fbaf006a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:06.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:06.511+0000 7fbaf5ebf700 1 --2- 172.21.15.16:0/2342027129 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbaf00726b0 0x7fbaf006a7e0 secure :-1 s=READY pgs=2654 cs=0 l=1 rev1=1 crypto rx=0x7fbaf00b2f40 tx=0x7fbae0009bd0 comp rx=0 tx=0).stop 2023-10-16T13:47:06.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:06.511+0000 7fbaf5ebf700 1 -- 172.21.15.16:0/2342027129 shutdown_connections 2023-10-16T13:47:06.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:06.511+0000 7fbaf5ebf700 1 --2- 172.21.15.16:0/2342027129 >> v2:172.21.15.49:3300/0 conn(0x7fbaf015d920 0x7fbaf00cdef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:06.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:06.511+0000 7fbaf5ebf700 1 --2- 172.21.15.16:0/2342027129 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbaf00726b0 0x7fbaf006a7e0 unknown :-1 s=CLOSED pgs=2654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:06.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:06.511+0000 7fbaf5ebf700 1 -- 172.21.15.16:0/2342027129 >> 172.21.15.16:0/2342027129 conn(0x7fbaf00b1ed0 msgr2=0x7fbaf00b22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:06.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:06.511+0000 7fbaf5ebf700 1 -- 172.21.15.16:0/2342027129 shutdown_connections 2023-10-16T13:47:06.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:06.511+0000 7fbaf5ebf700 1 -- 172.21.15.16:0/2342027129 wait complete. 2023-10-16T13:47:06.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:06.512+0000 7fbaf5ebf700 1 Processor -- start 2023-10-16T13:47:06.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:06.512+0000 7fbaf5ebf700 1 -- start start 2023-10-16T13:47:06.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:06.513+0000 7fbaf5ebf700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbaf006ad20 0x7fbaf01533a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:06.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:06.513+0000 7fbaf5ebf700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbaf00726b0 0x7fbaf015de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:06.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:06.513+0000 7fbaf5ebf700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbaf015d920 0x7fbaf00cb210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:06.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:06.513+0000 7fbaf5ebf700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf015f9b0 con 0x7fbaf015d920 2023-10-16T13:47:06.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:06.513+0000 7fbaf5ebf700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf015e910 con 0x7fbaf006ad20 2023-10-16T13:47:06.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:06.513+0000 7fbaf5ebf700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf015ebf0 con 0x7fbaf00726b0 2023-10-16T13:47:06.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:06.513+0000 7fbaeffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbaf015d920 0x7fbaf00cb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:06.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:06.513+0000 7fbaeffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbaf015d920 0x7fbaf00cb210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54886/0 (socket says 172.21.15.16:54886) 2023-10-16T13:47:06.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:06.513+0000 7fbaeffff700 1 -- 172.21.15.16:0/2380091337 learned_addr learned my addr 172.21.15.16:0/2380091337 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:06.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:06.513+0000 7fbaef7fe700 1 --2- 172.21.15.16:0/2380091337 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbaf006ad20 0x7fbaf01533a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:06.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:06.513+0000 7fbaeeffd700 1 --2- 172.21.15.16:0/2380091337 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbaf00726b0 0x7fbaf015de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:06.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:06.513+0000 7fbaeffff700 1 -- 172.21.15.16:0/2380091337 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbaf00726b0 msgr2=0x7fbaf015de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:06.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:06.513+0000 7fbaeffff700 1 --2- 172.21.15.16:0/2380091337 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbaf00726b0 0x7fbaf015de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:06.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:06.513+0000 7fbaeffff700 1 -- 172.21.15.16:0/2380091337 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbaf006ad20 msgr2=0x7fbaf01533a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:06.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:06.513+0000 7fbaeffff700 1 --2- 172.21.15.16:0/2380091337 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbaf006ad20 0x7fbaf01533a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:06.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:06.513+0000 7fbaeffff700 1 -- 172.21.15.16:0/2380091337 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbae0009550 con 0x7fbaf015d920 2023-10-16T13:47:06.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:06.513+0000 7fbaeffff700 1 --2- 172.21.15.16:0/2380091337 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbaf015d920 0x7fbaf00cb210 secure :-1 s=READY pgs=2655 cs=0 l=1 rev1=1 crypto rx=0x7fbae800cc50 tx=0x7fbae800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:06.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:06.514+0000 7fbae7fff700 1 -- 172.21.15.16:0/2380091337 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbae8003c00 con 0x7fbaf015d920 2023-10-16T13:47:06.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:06.514+0000 7fbae7fff700 1 -- 172.21.15.16:0/2380091337 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbae8003d60 con 0x7fbaf015d920 2023-10-16T13:47:06.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:06.514+0000 7fbaf5ebf700 1 -- 172.21.15.16:0/2380091337 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbaf015eed0 con 0x7fbaf015d920 2023-10-16T13:47:06.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:06.514+0000 7fbae7fff700 1 -- 172.21.15.16:0/2380091337 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbae800aab0 con 0x7fbaf015d920 2023-10-16T13:47:06.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:06.514+0000 7fbaf5ebf700 1 -- 172.21.15.16:0/2380091337 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbaf015f490 con 0x7fbaf015d920 2023-10-16T13:47:06.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:06.515+0000 7fbaf5ebf700 1 -- 172.21.15.16:0/2380091337 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbaf015f6b0 con 0x7fbaf015d920 2023-10-16T13:47:06.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:06.515+0000 7fbae7fff700 1 -- 172.21.15.16:0/2380091337 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 148) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fbae800ac10 con 0x7fbaf015d920 2023-10-16T13:47:06.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:06.516+0000 7fbae7fff700 1 --2- 172.21.15.16:0/2380091337 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fbad006b140 0x7fbad006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:06.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:06.516+0000 7fbae7fff700 1 -- 172.21.15.16:0/2380091337 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(233..233 src has 1..233) v4 ==== 7486+0+0 (secure 0 0 0) 0x7fbae808a0a0 con 0x7fbaf015d920 2023-10-16T13:47:06.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:06.516+0000 7fbaef7fe700 1 --2- 172.21.15.16:0/2380091337 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fbad006b140 0x7fbad006d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:06.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:06.516+0000 7fbaef7fe700 1 --2- 172.21.15.16:0/2380091337 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fbad006b140 0x7fbad006d600 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fbaf01f6060 tx=0x7fbad8009210 comp rx=0 tx=0).ready entity=mgr.11886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:06.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:06.520+0000 7fbae7fff700 1 -- 172.21.15.16:0/2380091337 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbae8054700 con 0x7fbaf015d920 2023-10-16T13:47:06.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:06.724+0000 7fbaf5ebf700 1 -- 172.21.15.16:0/2380091337 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fbaf004ec20 con 0x7fbaf015d920 2023-10-16T13:47:07.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.346+0000 7fbae7fff700 1 -- 172.21.15.16:0/2380091337 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v234) v1 ==== 169+0+0 (secure 0 0 0) 0x7fbae8057d20 con 0x7fbaf015d920 2023-10-16T13:47:07.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.351+0000 7fbaf5ebf700 1 -- 172.21.15.16:0/2380091337 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fbad006b140 msgr2=0x7fbad006d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:07.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.351+0000 7fbaf5ebf700 1 --2- 172.21.15.16:0/2380091337 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fbad006b140 0x7fbad006d600 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fbaf01f6060 tx=0x7fbad8009210 comp rx=0 tx=0).stop 2023-10-16T13:47:07.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.351+0000 7fbaf5ebf700 1 -- 172.21.15.16:0/2380091337 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbaf015d920 msgr2=0x7fbaf00cb210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:07.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.351+0000 7fbaf5ebf700 1 --2- 172.21.15.16:0/2380091337 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbaf015d920 0x7fbaf00cb210 secure :-1 s=READY pgs=2655 cs=0 l=1 rev1=1 crypto rx=0x7fbae800cc50 tx=0x7fbae800cf60 comp rx=0 tx=0).stop 2023-10-16T13:47:07.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.351+0000 7fbaf5ebf700 1 -- 172.21.15.16:0/2380091337 shutdown_connections 2023-10-16T13:47:07.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.351+0000 7fbaf5ebf700 1 --2- 172.21.15.16:0/2380091337 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fbad006b140 0x7fbad006d600 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:07.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.351+0000 7fbaf5ebf700 1 --2- 172.21.15.16:0/2380091337 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbaf015d920 0x7fbaf00cb210 unknown :-1 s=CLOSED pgs=2655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:07.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.351+0000 7fbaf5ebf700 1 --2- 172.21.15.16:0/2380091337 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbaf006ad20 0x7fbaf01533a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:07.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.351+0000 7fbaf5ebf700 1 --2- 172.21.15.16:0/2380091337 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbaf00726b0 0x7fbaf015de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:07.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.351+0000 7fbaf5ebf700 1 -- 172.21.15.16:0/2380091337 >> 172.21.15.16:0/2380091337 conn(0x7fbaf00b1ed0 msgr2=0x7fbaf006ddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:07.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.351+0000 7fbaf5ebf700 1 -- 172.21.15.16:0/2380091337 shutdown_connections 2023-10-16T13:47:07.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.351+0000 7fbaf5ebf700 1 -- 172.21.15.16:0/2380091337 wait complete. 2023-10-16T13:47:07.356 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' removed 2023-10-16T13:47:07.372 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:47:07.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.501+0000 7ffaa537b700 1 Processor -- start 2023-10-16T13:47:07.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.502+0000 7ffaa537b700 1 -- start start 2023-10-16T13:47:07.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.502+0000 7ffaa537b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ffaa00d0750 0x7ffaa0168450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:07.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.502+0000 7ffaa537b700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaa006ba00 con 0x7ffaa00d41c0 2023-10-16T13:47:07.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.502+0000 7ffaa537b700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaa00d26d0 con 0x7ffaa015d8d0 2023-10-16T13:47:07.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.502+0000 7ffaa537b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa00d23f0 con 0x7ffaa00d0750 2023-10-16T13:47:07.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.502+0000 7ffa9e7fc700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ffaa00d41c0 0x7ffaa00d0040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44166/0 (socket says 172.21.15.16:44166) 2023-10-16T13:47:07.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.502+0000 7ffa9e7fc700 1 -- 172.21.15.16:0/1105885396 learned_addr learned my addr 172.21.15.16:0/1105885396 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:47:07.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.503+0000 7ffa9f7fe700 1 --2- 172.21.15.16:0/1105885396 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ffaa00d0750 0x7ffaa0168450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:07.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.503+0000 7ffa9dffb700 1 -- 172.21.15.16:0/1105885396 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1494777860 0 0) 0x7ffaa006ba00 con 0x7ffaa00d41c0 2023-10-16T13:47:07.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.503+0000 7ffa9dffb700 1 -- 172.21.15.16:0/1105885396 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa880035b0 con 0x7ffaa00d41c0 2023-10-16T13:47:07.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.503+0000 7ffa9dffb700 1 -- 172.21.15.16:0/1105885396 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1147422989 0 0) 0x7ffa880035b0 con 0x7ffaa00d41c0 2023-10-16T13:47:07.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.503+0000 7ffa9dffb700 1 -- 172.21.15.16:0/1105885396 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ffaa00d0750 msgr2=0x7ffaa0168450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:07.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.503+0000 7ffa9dffb700 1 --2- 172.21.15.16:0/1105885396 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ffaa00d0750 0x7ffaa0168450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:07.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.503+0000 7ffa9dffb700 1 -- 172.21.15.16:0/1105885396 >> v1:172.21.15.49:6789/0 conn(0x7ffaa015d8d0 legacy=0x7ffaa00d1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:07.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.503+0000 7ffa9dffb700 1 -- 172.21.15.16:0/1105885396 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaa006bca0 con 0x7ffaa00d41c0 2023-10-16T13:47:07.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.503+0000 7ffa9dffb700 1 -- 172.21.15.16:0/1105885396 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7ffa94001670 con 0x7ffaa00d41c0 2023-10-16T13:47:07.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.503+0000 7ffa9dffb700 1 -- 172.21.15.16:0/1105885396 >> v1:172.21.15.16:6789/0 conn(0x7ffaa00d41c0 legacy=0x7ffaa00d0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:07.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.503+0000 7ffa9dffb700 1 --2- 172.21.15.16:0/1105885396 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa88003a30 0x7ffa88003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:07.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.503+0000 7ffa9dffb700 1 --2- 172.21.15.16:0/1105885396 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ffaa00d0750 0x7ffa88004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:07.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.503+0000 7ffa9dffb700 1 --2- 172.21.15.16:0/1105885396 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ffa88004d90 0x7ffa88005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:07.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.503+0000 7ffa9dffb700 1 -- 172.21.15.16:0/1105885396 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa00d23f0 con 0x7ffa88003a30 2023-10-16T13:47:07.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.503+0000 7ffa9dffb700 1 -- 172.21.15.16:0/1105885396 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa8800baf0 con 0x7ffa88004d90 2023-10-16T13:47:07.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.503+0000 7ffa9dffb700 1 -- 172.21.15.16:0/1105885396 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa8800bcb0 con 0x7ffaa00d0750 2023-10-16T13:47:07.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.503+0000 7ffa9effd700 1 --2- 172.21.15.16:0/1105885396 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa88003a30 0x7ffa88003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:07.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.503+0000 7ffa9f7fe700 1 --2- 172.21.15.16:0/1105885396 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ffa88004d90 0x7ffa88005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:07.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.503+0000 7ffa9e7fc700 1 --2- 172.21.15.16:0/1105885396 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ffaa00d0750 0x7ffa88004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:07.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.504+0000 7ffa9effd700 1 -- 172.21.15.16:0/1105885396 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ffaa00d0750 msgr2=0x7ffa88004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:07.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.504+0000 7ffa9effd700 1 --2- 172.21.15.16:0/1105885396 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ffaa00d0750 0x7ffa88004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:07.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.504+0000 7ffa9effd700 1 -- 172.21.15.16:0/1105885396 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ffa88004d90 msgr2=0x7ffa88005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:07.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.504+0000 7ffa9effd700 1 --2- 172.21.15.16:0/1105885396 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ffa88004d90 0x7ffa88005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:07.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.504+0000 7ffa9effd700 1 -- 172.21.15.16:0/1105885396 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa880035b0 con 0x7ffa88003a30 2023-10-16T13:47:07.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.504+0000 7ffa9e7fc700 1 --2- 172.21.15.16:0/1105885396 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ffaa00d0750 0x7ffa88004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:47:07.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.504+0000 7ffa9effd700 1 --2- 172.21.15.16:0/1105885396 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa88003a30 0x7ffa88003e50 secure :-1 s=READY pgs=2657 cs=0 l=1 rev1=1 crypto rx=0x7ffa90001800 tx=0x7ffa88005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bf31510d80e2503 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:07.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.504+0000 7ffa9dffb700 1 -- 172.21.15.16:0/1105885396 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa900012f0 con 0x7ffa88003a30 2023-10-16T13:47:07.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.504+0000 7ffa9dffb700 1 -- 172.21.15.16:0/1105885396 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa900052f0 con 0x7ffa88003a30 2023-10-16T13:47:07.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.504+0000 7ffa9dffb700 1 -- 172.21.15.16:0/1105885396 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa90006860 con 0x7ffa88003a30 2023-10-16T13:47:07.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.504+0000 7ffaa537b700 1 -- 172.21.15.16:0/1105885396 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa88003a30 msgr2=0x7ffa88003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:07.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.504+0000 7ffaa537b700 1 --2- 172.21.15.16:0/1105885396 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa88003a30 0x7ffa88003e50 secure :-1 s=READY pgs=2657 cs=0 l=1 rev1=1 crypto rx=0x7ffa90001800 tx=0x7ffa88005880 comp rx=0 tx=0).stop 2023-10-16T13:47:07.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.504+0000 7ffa9f7fe700 1 -- 172.21.15.16:0/1105885396 reap_dead start 2023-10-16T13:47:07.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.504+0000 7ffaa537b700 1 -- 172.21.15.16:0/1105885396 shutdown_connections 2023-10-16T13:47:07.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.504+0000 7ffaa537b700 1 -- 172.21.15.16:0/1105885396 >> 172.21.15.16:0/1105885396 conn(0x7ffaa00d3630 msgr2=0x7ffaa00d3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:07.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.504+0000 7ffaa537b700 1 -- 172.21.15.16:0/1105885396 shutdown_connections 2023-10-16T13:47:07.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.504+0000 7ffaa537b700 1 -- 172.21.15.16:0/1105885396 wait complete. 2023-10-16T13:47:07.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.506+0000 7ffaa537b700 1 Processor -- start 2023-10-16T13:47:07.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.506+0000 7ffaa537b700 1 -- start start 2023-10-16T13:47:07.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.506+0000 7ffaa537b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa88003a30 0x7ffaa00c9c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:07.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.507+0000 7ffaa537b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ffaa0155b20 0x7ffaa0149900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:07.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.507+0000 7ffaa537b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ffaa00b1bb0 0x7ffaa00b2040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:07.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.507+0000 7ffaa537b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa00b2580 con 0x7ffa88003a30 2023-10-16T13:47:07.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.507+0000 7ffaa537b700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa00c8190 con 0x7ffaa00b1bb0 2023-10-16T13:47:07.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.507+0000 7ffaa537b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa00c83e0 con 0x7ffaa0155b20 2023-10-16T13:47:07.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.507+0000 7ffa9effd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa88003a30 0x7ffaa00c9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:07.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.507+0000 7ffa9effd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa88003a30 0x7ffaa00c9c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54902/0 (socket says 172.21.15.16:54902) 2023-10-16T13:47:07.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.507+0000 7ffa9effd700 1 -- 172.21.15.16:0/380824668 learned_addr learned my addr 172.21.15.16:0/380824668 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:07.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.507+0000 7ffa9effd700 1 -- 172.21.15.16:0/380824668 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ffaa0155b20 msgr2=0x7ffaa0149900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:07.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.507+0000 7ffa9effd700 1 --2- 172.21.15.16:0/380824668 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ffaa0155b20 0x7ffaa0149900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:07.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.507+0000 7ffa9effd700 1 -- 172.21.15.16:0/380824668 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ffaa00b1bb0 msgr2=0x7ffaa00b2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:07.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.507+0000 7ffa9effd700 1 --2- 172.21.15.16:0/380824668 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ffaa00b1bb0 0x7ffaa00b2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:07.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.507+0000 7ffa9effd700 1 -- 172.21.15.16:0/380824668 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaa00c86c0 con 0x7ffa88003a30 2023-10-16T13:47:07.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.507+0000 7ffa9effd700 1 --2- 172.21.15.16:0/380824668 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa88003a30 0x7ffaa00c9c80 secure :-1 s=READY pgs=2658 cs=0 l=1 rev1=1 crypto rx=0x7ffa8800bac0 tx=0x7ffa90001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:07.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.507+0000 7ffa877fe700 1 -- 172.21.15.16:0/380824668 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa90006cf0 con 0x7ffa88003a30 2023-10-16T13:47:07.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.507+0000 7ffa877fe700 1 -- 172.21.15.16:0/380824668 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa90006e70 con 0x7ffa88003a30 2023-10-16T13:47:07.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.507+0000 7ffaa537b700 1 -- 172.21.15.16:0/380824668 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaa00c89a0 con 0x7ffa88003a30 2023-10-16T13:47:07.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.507+0000 7ffa877fe700 1 -- 172.21.15.16:0/380824668 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa90007240 con 0x7ffa88003a30 2023-10-16T13:47:07.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.507+0000 7ffaa537b700 1 -- 172.21.15.16:0/380824668 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaa00c8f60 con 0x7ffa88003a30 2023-10-16T13:47:07.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.508+0000 7ffaa537b700 1 -- 172.21.15.16:0/380824668 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffaa00b2580 con 0x7ffa88003a30 2023-10-16T13:47:07.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.509+0000 7ffa877fe700 1 -- 172.21.15.16:0/380824668 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 148) v1 ==== 89433+0+0 (secure 0 0 0) 0x7ffa900073a0 con 0x7ffa88003a30 2023-10-16T13:47:07.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.509+0000 7ffa877fe700 1 --2- 172.21.15.16:0/380824668 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7ffa8c06b140 0x7ffa8c06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:07.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.509+0000 7ffa877fe700 1 -- 172.21.15.16:0/380824668 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 7072+0+0 (secure 0 0 0) 0x7ffa9008b1c0 con 0x7ffa88003a30 2023-10-16T13:47:07.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.509+0000 7ffa9e7fc700 1 --2- 172.21.15.16:0/380824668 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7ffa8c06b140 0x7ffa8c06d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:07.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.510+0000 7ffa9e7fc700 1 --2- 172.21.15.16:0/380824668 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7ffa8c06b140 0x7ffa8c06d600 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ffa940017a0 tx=0x7ffa94009040 comp rx=0 tx=0).ready entity=mgr.11886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:07.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.514+0000 7ffa877fe700 1 -- 172.21.15.16:0/380824668 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ffa90055b60 con 0x7ffa88003a30 2023-10-16T13:47:07.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.730+0000 7ffaa537b700 1 -- 172.21.15.16:0/380824668 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffaa00c90f0 con 0x7ffa88003a30 2023-10-16T13:47:07.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.731+0000 7ffa877fe700 1 -- 172.21.15.16:0/380824668 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v234) v1 ==== 74+0+18143 (secure 0 0 0) 0x7ffa90059180 con 0x7ffa88003a30 2023-10-16T13:47:07.734 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:47:07.735 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":234,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:47:07.345418+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T13:46:35.800444+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"228","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"backup_fs_data","create_time":"2023-10-16T13:46:37.035323+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"229","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.49:6835/2905760119":"2023-10-17T13:47:04.337914+0000","172.21.15.16:0/1827947757":"2023-10-17T13:47:01.801871+0000","172.21.15.16:0/1694897016":"2023-10-17T13:47:01.801871+0000","172.21.15.16:0/3760456275":"2023-10-17T13:47:01.801871+0000","172.21.15.16:0/3651530055":"2023-10-17T13:47:01.801871+0000","172.21.15.16:0/850397235":"2023-10-17T13:47:01.801871+0000","172.21.15.16:6833/1506142504":"2023-10-17T13:47:01.801871+0000","172.21.15.16:0/4239949371":"2023-10-17T13:47:01.801871+0000","172.21.15.16:0/2321929682":"2023-10-17T13:46:51.165018+0000","172.21.15.16:0/3155275276":"2023-10-17T13:47:01.801871+0000","172.21.15.16:0/3429506235":"2023-10-17T13:46:51.165018+0000","172.21.15.49:6833/2905760119":"2023-10-17T13:47:04.337914+0000","172.21.15.16:0/4090083700":"2023-10-17T13:47:01.801871+0000","172.21.15.16:0/3843126404":"2023-10-17T13:46:51.165018+0000","172.21.15.16:6832/1506142504":"2023-10-17T13:47:01.801871+0000","172.21.15.16:0/1439318665":"2023-10-17T13:46:51.165018+0000","172.21.15.16:0/4204021371":"2023-10-17T13:46:51.165018+0000","172.21.15.16:6833/3171422553":"2023-10-17T13:46:51.165018+0000","172.21.15.16:6832/3171422553":"2023-10-17T13:46:51.165018+0000","172.21.15.16:0/1691810194":"2023-10-17T13:47:01.801871+0000","172.21.15.16:0/1632287797":"2023-10-17T13:46:51.165018+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:47:07.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.734+0000 7ffaa537b700 1 -- 172.21.15.16:0/380824668 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7ffa8c06b140 msgr2=0x7ffa8c06d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:07.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.734+0000 7ffaa537b700 1 --2- 172.21.15.16:0/380824668 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7ffa8c06b140 0x7ffa8c06d600 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ffa940017a0 tx=0x7ffa94009040 comp rx=0 tx=0).stop 2023-10-16T13:47:07.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.734+0000 7ffaa537b700 1 -- 172.21.15.16:0/380824668 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa88003a30 msgr2=0x7ffaa00c9c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:07.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.734+0000 7ffaa537b700 1 --2- 172.21.15.16:0/380824668 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa88003a30 0x7ffaa00c9c80 secure :-1 s=READY pgs=2658 cs=0 l=1 rev1=1 crypto rx=0x7ffa8800bac0 tx=0x7ffa90001120 comp rx=0 tx=0).stop 2023-10-16T13:47:07.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.734+0000 7ffaa537b700 1 -- 172.21.15.16:0/380824668 shutdown_connections 2023-10-16T13:47:07.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.734+0000 7ffaa537b700 1 --2- 172.21.15.16:0/380824668 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7ffa8c06b140 0x7ffa8c06d600 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:07.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.734+0000 7ffaa537b700 1 --2- 172.21.15.16:0/380824668 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ffaa00b1bb0 0x7ffaa00b2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:07.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.734+0000 7ffaa537b700 1 --2- 172.21.15.16:0/380824668 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa88003a30 0x7ffaa00c9c80 unknown :-1 s=CLOSED pgs=2658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:07.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.734+0000 7ffaa537b700 1 --2- 172.21.15.16:0/380824668 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ffaa0155b20 0x7ffaa0149900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:07.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.734+0000 7ffaa537b700 1 -- 172.21.15.16:0/380824668 >> 172.21.15.16:0/380824668 conn(0x7ffaa00d3630 msgr2=0x7ffaa00ceb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:07.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.734+0000 7ffaa537b700 1 -- 172.21.15.16:0/380824668 shutdown_connections 2023-10-16T13:47:07.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.734+0000 7ffaa537b700 1 -- 172.21.15.16:0/380824668 wait complete. 2023-10-16T13:47:07.748 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 29, 'pool_name': 'backup_fs_metadata', 'create_time': '2023-10-16T13:46:35.800444+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '228', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 30, 'pool_name': 'backup_fs_data', 'create_time': '2023-10-16T13:46:37.035323+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '229', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T13:47:07.749 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T13:47:07.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.875+0000 7f023bbb6700 1 Processor -- start 2023-10-16T13:47:07.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.875+0000 7f023bbb6700 1 -- start start 2023-10-16T13:47:07.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.876+0000 7f023bbb6700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f023415b2c0 0x7f0234161340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:07.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.876+0000 7f023bbb6700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f023406e570 0x7f023415e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:07.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.876+0000 7f023bbb6700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0234161880 con 0x7f023406a7e0 2023-10-16T13:47:07.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.876+0000 7f023bbb6700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f023415b6a0 con 0x7f023415b2c0 2023-10-16T13:47:07.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.876+0000 7f023bbb6700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0234161b20 con 0x7f023406e570 2023-10-16T13:47:07.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.876+0000 7f0239151700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f023406e570 0x7f023415e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:07.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.876+0000 7f0239952700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f023415b2c0 0x7f0234161340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:07.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.876+0000 7f023a153700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7f023406a7e0 0x7f0234071a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:51368/0 (socket says 172.21.15.16:51368) 2023-10-16T13:47:07.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.876+0000 7f023a153700 1 -- 172.21.15.16:0/2269152182 learned_addr learned my addr 172.21.15.16:0/2269152182 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:47:07.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.877+0000 7f0239151700 1 -- 172.21.15.16:0/2269152182 >> v2:172.21.15.49:3300/0 conn(0x7f023415b2c0 msgr2=0x7f0234161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:07.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.877+0000 7f0239151700 1 --2- 172.21.15.16:0/2269152182 >> v2:172.21.15.49:3300/0 conn(0x7f023415b2c0 0x7f0234161340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:07.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.877+0000 7f0239151700 1 -- 172.21.15.16:0/2269152182 >> v1:172.21.15.49:6789/0 conn(0x7f023406a7e0 legacy=0x7f0234071a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:07.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.877+0000 7f0239151700 1 -- 172.21.15.16:0/2269152182 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0234161e00 con 0x7f023406e570 2023-10-16T13:47:07.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.877+0000 7f0239952700 1 --2- 172.21.15.16:0/2269152182 >> v2:172.21.15.49:3300/0 conn(0x7f023415b2c0 0x7f0234161340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:47:07.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.877+0000 7f0239151700 1 --2- 172.21.15.16:0/2269152182 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f023406e570 0x7f023415e5a0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f0230009840 tx=0x7f0230009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9c87e17847aa4270 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:07.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.877+0000 7f0238950700 1 -- 172.21.15.16:0/2269152182 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0230011830 con 0x7f023406e570 2023-10-16T13:47:07.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.877+0000 7f0238950700 1 -- 172.21.15.16:0/2269152182 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0230011990 con 0x7f023406e570 2023-10-16T13:47:07.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.877+0000 7f0238950700 1 -- 172.21.15.16:0/2269152182 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0230011d20 con 0x7f023406e570 2023-10-16T13:47:07.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.878+0000 7f023bbb6700 1 -- 172.21.15.16:0/2269152182 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f023406e570 msgr2=0x7f023415e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:07.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.878+0000 7f023bbb6700 1 --2- 172.21.15.16:0/2269152182 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f023406e570 0x7f023415e5a0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f0230009840 tx=0x7f0230009b50 comp rx=0 tx=0).stop 2023-10-16T13:47:07.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.878+0000 7f023bbb6700 1 -- 172.21.15.16:0/2269152182 shutdown_connections 2023-10-16T13:47:07.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.878+0000 7f023bbb6700 1 --2- 172.21.15.16:0/2269152182 >> v2:172.21.15.49:3300/0 conn(0x7f023415b2c0 0x7f0234161340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:07.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.878+0000 7f023bbb6700 1 --2- 172.21.15.16:0/2269152182 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f023406e570 0x7f023415e5a0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:07.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.878+0000 7f023bbb6700 1 -- 172.21.15.16:0/2269152182 >> 172.21.15.16:0/2269152182 conn(0x7f02340b1f40 msgr2=0x7f02340b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:07.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.878+0000 7f023bbb6700 1 -- 172.21.15.16:0/2269152182 shutdown_connections 2023-10-16T13:47:07.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.878+0000 7f023bbb6700 1 -- 172.21.15.16:0/2269152182 wait complete. 2023-10-16T13:47:07.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.879+0000 7f023bbb6700 1 Processor -- start 2023-10-16T13:47:07.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.879+0000 7f023bbb6700 1 -- start start 2023-10-16T13:47:07.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.880+0000 7f023bbb6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f023406a7e0 0x7f02340c7730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:07.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.880+0000 7f023bbb6700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f023406e570 0x7f02341f6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:07.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.880+0000 7f023bbb6700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f023415b2c0 0x7f0234149470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:07.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.880+0000 7f023bbb6700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0234158670 con 0x7f023406a7e0 2023-10-16T13:47:07.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.880+0000 7f023bbb6700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02341f6d80 con 0x7f023406e570 2023-10-16T13:47:07.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.880+0000 7f023bbb6700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02341f7060 con 0x7f023415b2c0 2023-10-16T13:47:07.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.880+0000 7f0239952700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f023406a7e0 0x7f02340c7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:07.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.880+0000 7f0239952700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f023406a7e0 0x7f02340c7730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54914/0 (socket says 172.21.15.16:54914) 2023-10-16T13:47:07.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.880+0000 7f0239952700 1 -- 172.21.15.16:0/548881459 learned_addr learned my addr 172.21.15.16:0/548881459 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:07.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.880+0000 7f023a153700 1 --2- 172.21.15.16:0/548881459 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f023415b2c0 0x7f0234149470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:07.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.880+0000 7f0239151700 1 --2- 172.21.15.16:0/548881459 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f023406e570 0x7f02341f6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:07.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.880+0000 7f0239952700 1 -- 172.21.15.16:0/548881459 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f023415b2c0 msgr2=0x7f0234149470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:07.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.880+0000 7f0239952700 1 --2- 172.21.15.16:0/548881459 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f023415b2c0 0x7f0234149470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:07.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.880+0000 7f0239952700 1 -- 172.21.15.16:0/548881459 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f023406e570 msgr2=0x7f02341f6330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:07.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.880+0000 7f0239952700 1 --2- 172.21.15.16:0/548881459 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f023406e570 0x7f02341f6330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:07.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.880+0000 7f0239952700 1 -- 172.21.15.16:0/548881459 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02240094a0 con 0x7f023406a7e0 2023-10-16T13:47:07.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.880+0000 7f0239952700 1 --2- 172.21.15.16:0/548881459 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f023406a7e0 0x7f02340c7730 secure :-1 s=READY pgs=2659 cs=0 l=1 rev1=1 crypto rx=0x7f022400ebe0 tx=0x7f022400eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:07.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.881+0000 7f022a7fc700 1 -- 172.21.15.16:0/548881459 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f022400c850 con 0x7f023406a7e0 2023-10-16T13:47:07.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.881+0000 7f022a7fc700 1 -- 172.21.15.16:0/548881459 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f022400c9b0 con 0x7f023406a7e0 2023-10-16T13:47:07.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.881+0000 7f023bbb6700 1 -- 172.21.15.16:0/548881459 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02300094a0 con 0x7f023406a7e0 2023-10-16T13:47:07.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.881+0000 7f022a7fc700 1 -- 172.21.15.16:0/548881459 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f022400cd40 con 0x7f023406a7e0 2023-10-16T13:47:07.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.881+0000 7f023bbb6700 1 -- 172.21.15.16:0/548881459 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f023414d0d0 con 0x7f023406a7e0 2023-10-16T13:47:07.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.882+0000 7f023bbb6700 1 -- 172.21.15.16:0/548881459 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0234158950 con 0x7f023406a7e0 2023-10-16T13:47:07.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.882+0000 7f022a7fc700 1 -- 172.21.15.16:0/548881459 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 148) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f0224013470 con 0x7f023406a7e0 2023-10-16T13:47:07.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.882+0000 7f022a7fc700 1 --2- 172.21.15.16:0/548881459 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f021806b140 0x7f021806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:07.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.882+0000 7f022a7fc700 1 -- 172.21.15.16:0/548881459 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 7072+0+0 (secure 0 0 0) 0x7f0224089f40 con 0x7f023406a7e0 2023-10-16T13:47:07.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.882+0000 7f0239151700 1 --2- 172.21.15.16:0/548881459 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f021806b140 0x7f021806d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:07.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.883+0000 7f0239151700 1 --2- 172.21.15.16:0/548881459 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f021806b140 0x7f021806d600 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0230000c00 tx=0x7f023001c040 comp rx=0 tx=0).ready entity=mgr.11886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:07.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:07.887+0000 7f022a7fc700 1 -- 172.21.15.16:0/548881459 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f02240547d0 con 0x7f023406a7e0 2023-10-16T13:47:08.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.087+0000 7f023bbb6700 1 -- 172.21.15.16:0/548881459 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f02341f72a0 con 0x7f023406a7e0 2023-10-16T13:47:08.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.088+0000 7f022a7fc700 1 -- 172.21.15.16:0/548881459 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v234) v1 ==== 93+0+10 (secure 0 0 0) 0x7f02240170a0 con 0x7f023406a7e0 2023-10-16T13:47:08.090 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T13:47:08.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.090+0000 7f023bbb6700 1 -- 172.21.15.16:0/548881459 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f021806b140 msgr2=0x7f021806d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:08.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.090+0000 7f023bbb6700 1 --2- 172.21.15.16:0/548881459 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f021806b140 0x7f021806d600 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0230000c00 tx=0x7f023001c040 comp rx=0 tx=0).stop 2023-10-16T13:47:08.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.090+0000 7f023bbb6700 1 -- 172.21.15.16:0/548881459 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f023406a7e0 msgr2=0x7f02340c7730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:08.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.090+0000 7f023bbb6700 1 --2- 172.21.15.16:0/548881459 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f023406a7e0 0x7f02340c7730 secure :-1 s=READY pgs=2659 cs=0 l=1 rev1=1 crypto rx=0x7f022400ebe0 tx=0x7f022400eef0 comp rx=0 tx=0).stop 2023-10-16T13:47:08.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.090+0000 7f023bbb6700 1 -- 172.21.15.16:0/548881459 shutdown_connections 2023-10-16T13:47:08.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.090+0000 7f023bbb6700 1 --2- 172.21.15.16:0/548881459 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f021806b140 0x7f021806d600 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:08.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.090+0000 7f023bbb6700 1 --2- 172.21.15.16:0/548881459 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f023415b2c0 0x7f0234149470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:08.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.090+0000 7f023bbb6700 1 --2- 172.21.15.16:0/548881459 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f023406a7e0 0x7f02340c7730 unknown :-1 s=CLOSED pgs=2659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:08.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.090+0000 7f023bbb6700 1 --2- 172.21.15.16:0/548881459 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f023406e570 0x7f02341f6330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:08.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.090+0000 7f023bbb6700 1 -- 172.21.15.16:0/548881459 >> 172.21.15.16:0/548881459 conn(0x7f02340b1f40 msgr2=0x7f023415b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:08.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.090+0000 7f023bbb6700 1 -- 172.21.15.16:0/548881459 shutdown_connections 2023-10-16T13:47:08.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.090+0000 7f023bbb6700 1 -- 172.21.15.16:0/548881459 wait complete. 2023-10-16T13:47:08.104 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2023-10-16T13:47:08.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.229+0000 7f1776ca5700 1 Processor -- start 2023-10-16T13:47:08.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.230+0000 7f1776ca5700 1 -- start start 2023-10-16T13:47:08.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.230+0000 7f1776ca5700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f177015f170 0x7f17700c5bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:08.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.230+0000 7f1776ca5700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17700cee50 con 0x7f17700cb6f0 2023-10-16T13:47:08.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.230+0000 7f1776ca5700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17700c5020 con 0x7f17700cde90 2023-10-16T13:47:08.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.230+0000 7f1776ca5700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17700d2340 con 0x7f177015f170 2023-10-16T13:47:08.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.230+0000 7f1775242700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f177015f170 0x7f17700c5bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:08.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.231+0000 7f1775242700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f177015f170 0x7f17700c5bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54926/0 (socket says 172.21.15.16:54926) 2023-10-16T13:47:08.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.231+0000 7f1775242700 1 -- 172.21.15.16:0/2419151308 learned_addr learned my addr 172.21.15.16:0/2419151308 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:08.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.231+0000 7f1775242700 1 -- 172.21.15.16:0/2419151308 >> v1:172.21.15.49:6789/0 conn(0x7f17700cde90 legacy=0x7f177015ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:08.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.231+0000 7f1775242700 1 -- 172.21.15.16:0/2419151308 >> v1:172.21.15.16:6789/0 conn(0x7f17700cb6f0 legacy=0x7f17700c54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:08.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.231+0000 7f1775242700 1 -- 172.21.15.16:0/2419151308 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17700ccdd0 con 0x7f177015f170 2023-10-16T13:47:08.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.231+0000 7f1775242700 1 --2- 172.21.15.16:0/2419151308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f177015f170 0x7f17700c5bc0 secure :-1 s=READY pgs=2661 cs=0 l=1 rev1=1 crypto rx=0x7f1768009e50 tx=0x7f176800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7a931ef0b797bd9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:08.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.231+0000 7f176f7fe700 1 -- 172.21.15.16:0/2419151308 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f176800e070 con 0x7f177015f170 2023-10-16T13:47:08.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.231+0000 7f176f7fe700 1 -- 172.21.15.16:0/2419151308 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1768003a20 con 0x7f177015f170 2023-10-16T13:47:08.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.231+0000 7f176f7fe700 1 -- 172.21.15.16:0/2419151308 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1768003dd0 con 0x7f177015f170 2023-10-16T13:47:08.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.232+0000 7f1776ca5700 1 -- 172.21.15.16:0/2419151308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f177015f170 msgr2=0x7f17700c5bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:08.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.232+0000 7f1776ca5700 1 --2- 172.21.15.16:0/2419151308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f177015f170 0x7f17700c5bc0 secure :-1 s=READY pgs=2661 cs=0 l=1 rev1=1 crypto rx=0x7f1768009e50 tx=0x7f176800b330 comp rx=0 tx=0).stop 2023-10-16T13:47:08.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.232+0000 7f1776ca5700 1 -- 172.21.15.16:0/2419151308 shutdown_connections 2023-10-16T13:47:08.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.232+0000 7f1776ca5700 1 --2- 172.21.15.16:0/2419151308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f177015f170 0x7f17700c5bc0 unknown :-1 s=CLOSED pgs=2661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:08.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.232+0000 7f1776ca5700 1 -- 172.21.15.16:0/2419151308 >> 172.21.15.16:0/2419151308 conn(0x7f17700d14d0 msgr2=0x7f17700d18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:08.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.232+0000 7f1776ca5700 1 -- 172.21.15.16:0/2419151308 shutdown_connections 2023-10-16T13:47:08.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.232+0000 7f1776ca5700 1 -- 172.21.15.16:0/2419151308 wait complete. 2023-10-16T13:47:08.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.233+0000 7f1776ca5700 1 Processor -- start 2023-10-16T13:47:08.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.233+0000 7f1776ca5700 1 -- start start 2023-10-16T13:47:08.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.234+0000 7f1776ca5700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f17700cb6f0 0x7f177006d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:08.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.234+0000 7f1776ca5700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f17700cde90 0x7f1770152880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:08.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.234+0000 7f1776ca5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f177015f170 0x7f177006bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:08.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.234+0000 7f1776ca5700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f177006c3f0 con 0x7f177015f170 2023-10-16T13:47:08.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.234+0000 7f1776ca5700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f177006c6d0 con 0x7f17700cb6f0 2023-10-16T13:47:08.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.234+0000 7f1776ca5700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f177006c9b0 con 0x7f17700cde90 2023-10-16T13:47:08.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.234+0000 7f1775242700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f177015f170 0x7f177006bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:08.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.234+0000 7f1775242700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f177015f170 0x7f177006bac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54932/0 (socket says 172.21.15.16:54932) 2023-10-16T13:47:08.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.234+0000 7f1775242700 1 -- 172.21.15.16:0/543442429 learned_addr learned my addr 172.21.15.16:0/543442429 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:08.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.234+0000 7f1774a41700 1 --2- 172.21.15.16:0/543442429 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f17700cb6f0 0x7f177006d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:08.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.234+0000 7f176ffff700 1 --2- 172.21.15.16:0/543442429 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f17700cde90 0x7f1770152880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:08.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.234+0000 7f1775242700 1 -- 172.21.15.16:0/543442429 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f17700cde90 msgr2=0x7f1770152880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:08.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.234+0000 7f1775242700 1 --2- 172.21.15.16:0/543442429 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f17700cde90 0x7f1770152880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:08.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.234+0000 7f1775242700 1 -- 172.21.15.16:0/543442429 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f17700cb6f0 msgr2=0x7f177006d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:08.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.234+0000 7f1775242700 1 --2- 172.21.15.16:0/543442429 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f17700cb6f0 0x7f177006d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:08.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.234+0000 7f1775242700 1 -- 172.21.15.16:0/543442429 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1768009b30 con 0x7f177015f170 2023-10-16T13:47:08.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.234+0000 7f1775242700 1 --2- 172.21.15.16:0/543442429 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f177015f170 0x7f177006bac0 secure :-1 s=READY pgs=2662 cs=0 l=1 rev1=1 crypto rx=0x7f1768015040 tx=0x7f1768004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:08.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.235+0000 7f176d7fa700 1 -- 172.21.15.16:0/543442429 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1768004440 con 0x7f177015f170 2023-10-16T13:47:08.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.235+0000 7f176d7fa700 1 -- 172.21.15.16:0/543442429 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17680045a0 con 0x7f177015f170 2023-10-16T13:47:08.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.235+0000 7f1776ca5700 1 -- 172.21.15.16:0/543442429 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f177006cc90 con 0x7f177015f170 2023-10-16T13:47:08.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.235+0000 7f176d7fa700 1 -- 172.21.15.16:0/543442429 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1768004950 con 0x7f177015f170 2023-10-16T13:47:08.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.235+0000 7f1776ca5700 1 -- 172.21.15.16:0/543442429 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1770152dc0 con 0x7f177015f170 2023-10-16T13:47:08.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.236+0000 7f1776ca5700 1 -- 172.21.15.16:0/543442429 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f177004ebb0 con 0x7f177015f170 2023-10-16T13:47:08.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.236+0000 7f176d7fa700 1 -- 172.21.15.16:0/543442429 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 148) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f1768012430 con 0x7f177015f170 2023-10-16T13:47:08.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.237+0000 7f176d7fa700 1 --2- 172.21.15.16:0/543442429 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f175406b140 0x7f175406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:08.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.237+0000 7f176d7fa700 1 -- 172.21.15.16:0/543442429 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 7072+0+0 (secure 0 0 0) 0x7f176808b4e0 con 0x7f177015f170 2023-10-16T13:47:08.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.237+0000 7f1774a41700 1 --2- 172.21.15.16:0/543442429 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f175406b140 0x7f175406d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:08.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.237+0000 7f1774a41700 1 --2- 172.21.15.16:0/543442429 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f175406b140 0x7f175406d600 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1764007fd0 tx=0x7f1764007ec0 comp rx=0 tx=0).ready entity=mgr.11886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:08.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.241+0000 7f176d7fa700 1 -- 172.21.15.16:0/543442429 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f17680596f0 con 0x7f177015f170 2023-10-16T13:47:08.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.448+0000 7f1776ca5700 1 -- 172.21.15.16:0/543442429 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f17700cd360 con 0x7f177015f170 2023-10-16T13:47:08.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.449+0000 7f176d7fa700 1 -- 172.21.15.16:0/543442429 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v234) v1 ==== 107+0+11 (secure 0 0 0) 0x7f1768059280 con 0x7f177015f170 2023-10-16T13:47:08.452 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-10-16T13:47:08.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.451+0000 7f1776ca5700 1 -- 172.21.15.16:0/543442429 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f175406b140 msgr2=0x7f175406d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:08.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.451+0000 7f1776ca5700 1 --2- 172.21.15.16:0/543442429 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f175406b140 0x7f175406d600 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1764007fd0 tx=0x7f1764007ec0 comp rx=0 tx=0).stop 2023-10-16T13:47:08.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.451+0000 7f1776ca5700 1 -- 172.21.15.16:0/543442429 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f177015f170 msgr2=0x7f177006bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:08.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.451+0000 7f1776ca5700 1 --2- 172.21.15.16:0/543442429 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f177015f170 0x7f177006bac0 secure :-1 s=READY pgs=2662 cs=0 l=1 rev1=1 crypto rx=0x7f1768015040 tx=0x7f1768004ad0 comp rx=0 tx=0).stop 2023-10-16T13:47:08.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.451+0000 7f1776ca5700 1 -- 172.21.15.16:0/543442429 shutdown_connections 2023-10-16T13:47:08.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.451+0000 7f1776ca5700 1 --2- 172.21.15.16:0/543442429 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f175406b140 0x7f175406d600 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:08.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.451+0000 7f1776ca5700 1 --2- 172.21.15.16:0/543442429 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f177015f170 0x7f177006bac0 unknown :-1 s=CLOSED pgs=2662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:08.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.451+0000 7f1776ca5700 1 --2- 172.21.15.16:0/543442429 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f17700cb6f0 0x7f177006d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:08.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.451+0000 7f1776ca5700 1 --2- 172.21.15.16:0/543442429 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f17700cde90 0x7f1770152880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:08.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.451+0000 7f1776ca5700 1 -- 172.21.15.16:0/543442429 >> 172.21.15.16:0/543442429 conn(0x7f17700d14d0 msgr2=0x7f17700d18b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:08.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.451+0000 7f1776ca5700 1 -- 172.21.15.16:0/543442429 shutdown_connections 2023-10-16T13:47:08.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.451+0000 7f1776ca5700 1 -- 172.21.15.16:0/543442429 wait complete. 2023-10-16T13:47:08.465 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2023-10-16T13:47:08.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.591+0000 7f38e87c4700 1 Processor -- start 2023-10-16T13:47:08.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.591+0000 7f38e87c4700 1 -- start start 2023-10-16T13:47:08.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.592+0000 7f38e87c4700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f38e015ec80 0x7f38e00cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:08.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.592+0000 7f38e87c4700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f38e00d2110 0x7f38e00cd1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:08.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.592+0000 7f38e87c4700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38e00b2440 con 0x7f38e015f060 2023-10-16T13:47:08.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.592+0000 7f38e87c4700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38e00b2c50 con 0x7f38e00d2110 2023-10-16T13:47:08.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.592+0000 7f38e87c4700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38e00d00e0 con 0x7f38e015ec80 2023-10-16T13:47:08.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.592+0000 7f38e6560700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f38e015ec80 0x7f38e00cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:08.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.592+0000 7f38e6d61700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f38e00d2110 0x7f38e00cd1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:08.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.592+0000 7f38e5d5f700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7f38e015f060 0x7f38e00b4440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:51392/0 (socket says 172.21.15.16:51392) 2023-10-16T13:47:08.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.592+0000 7f38e5d5f700 1 -- 172.21.15.16:0/3809302839 learned_addr learned my addr 172.21.15.16:0/3809302839 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:47:08.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.592+0000 7f38e555e700 1 -- 172.21.15.16:0/3809302839 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 223149355 0 0) 0x7f38e00b2440 con 0x7f38e015f060 2023-10-16T13:47:08.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.592+0000 7f38e555e700 1 -- 172.21.15.16:0/3809302839 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38cc0035b0 con 0x7f38e015f060 2023-10-16T13:47:08.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.592+0000 7f38e6560700 1 -- 172.21.15.16:0/3809302839 >> v2:172.21.15.49:3300/0 conn(0x7f38e00d2110 msgr2=0x7f38e00cd1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:08.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.592+0000 7f38e6560700 1 --2- 172.21.15.16:0/3809302839 >> v2:172.21.15.49:3300/0 conn(0x7f38e00d2110 0x7f38e00cd1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:08.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.592+0000 7f38e6560700 1 -- 172.21.15.16:0/3809302839 >> v1:172.21.15.49:6789/0 conn(0x7f38e015f060 legacy=0x7f38e00b4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:08.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.592+0000 7f38e6560700 1 -- 172.21.15.16:0/3809302839 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38e00cef60 con 0x7f38e015ec80 2023-10-16T13:47:08.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.592+0000 7f38e6d61700 1 --2- 172.21.15.16:0/3809302839 >> v2:172.21.15.49:3300/0 conn(0x7f38e00d2110 0x7f38e00cd1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:47:08.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.593+0000 7f38e6560700 1 --2- 172.21.15.16:0/3809302839 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f38e015ec80 0x7f38e00cdec0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f38dc009840 tx=0x7f38dc009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eec77c021a22a5a4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:08.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.593+0000 7f38e555e700 1 -- 172.21.15.16:0/3809302839 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38dc011830 con 0x7f38e015ec80 2023-10-16T13:47:08.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.593+0000 7f38e555e700 1 -- 172.21.15.16:0/3809302839 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38dc011990 con 0x7f38e015ec80 2023-10-16T13:47:08.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.593+0000 7f38e555e700 1 -- 172.21.15.16:0/3809302839 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38dc011d20 con 0x7f38e015ec80 2023-10-16T13:47:08.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.593+0000 7f38e87c4700 1 -- 172.21.15.16:0/3809302839 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f38e015ec80 msgr2=0x7f38e00cdec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:08.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.593+0000 7f38e87c4700 1 --2- 172.21.15.16:0/3809302839 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f38e015ec80 0x7f38e00cdec0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f38dc009840 tx=0x7f38dc009b50 comp rx=0 tx=0).stop 2023-10-16T13:47:08.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.594+0000 7f38e87c4700 1 -- 172.21.15.16:0/3809302839 shutdown_connections 2023-10-16T13:47:08.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.594+0000 7f38e87c4700 1 --2- 172.21.15.16:0/3809302839 >> v2:172.21.15.49:3300/0 conn(0x7f38e00d2110 0x7f38e00cd1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:08.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.594+0000 7f38e87c4700 1 --2- 172.21.15.16:0/3809302839 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f38e015ec80 0x7f38e00cdec0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:08.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.594+0000 7f38e87c4700 1 -- 172.21.15.16:0/3809302839 >> 172.21.15.16:0/3809302839 conn(0x7f38e00d15f0 msgr2=0x7f38e00d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:08.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.594+0000 7f38e87c4700 1 -- 172.21.15.16:0/3809302839 shutdown_connections 2023-10-16T13:47:08.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.594+0000 7f38e87c4700 1 -- 172.21.15.16:0/3809302839 wait complete. 2023-10-16T13:47:08.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.595+0000 7f38e87c4700 1 Processor -- start 2023-10-16T13:47:08.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.595+0000 7f38e87c4700 1 -- start start 2023-10-16T13:47:08.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.596+0000 7f38e87c4700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f38e00d2110 0x7f38e00c4030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:08.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.596+0000 7f38e87c4700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f38e015ec80 0x7f38e00caf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:08.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.596+0000 7f38e87c4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38e015f060 0x7f38e00c5100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:08.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.596+0000 7f38e87c4700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38e015b480 con 0x7f38e015f060 2023-10-16T13:47:08.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.596+0000 7f38e87c4700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38e015b760 con 0x7f38e00d2110 2023-10-16T13:47:08.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.596+0000 7f38e87c4700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38e015ba40 con 0x7f38e015ec80 2023-10-16T13:47:08.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.596+0000 7f38e6d61700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38e015f060 0x7f38e00c5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:08.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.596+0000 7f38e6d61700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38e015f060 0x7f38e00c5100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54946/0 (socket says 172.21.15.16:54946) 2023-10-16T13:47:08.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.596+0000 7f38e6d61700 1 -- 172.21.15.16:0/2729435759 learned_addr learned my addr 172.21.15.16:0/2729435759 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:08.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.596+0000 7f38e6560700 1 --2- 172.21.15.16:0/2729435759 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f38e00d2110 0x7f38e00c4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:08.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.596+0000 7f38e5d5f700 1 --2- 172.21.15.16:0/2729435759 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f38e015ec80 0x7f38e00caf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:08.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.596+0000 7f38e6d61700 1 -- 172.21.15.16:0/2729435759 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f38e015ec80 msgr2=0x7f38e00caf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:08.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.596+0000 7f38e6d61700 1 --2- 172.21.15.16:0/2729435759 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f38e015ec80 0x7f38e00caf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:08.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.596+0000 7f38e6d61700 1 -- 172.21.15.16:0/2729435759 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f38e00d2110 msgr2=0x7f38e00c4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:08.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.596+0000 7f38e6d61700 1 --2- 172.21.15.16:0/2729435759 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f38e00d2110 0x7f38e00c4030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:08.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.596+0000 7f38e6d61700 1 -- 172.21.15.16:0/2729435759 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38d8009a70 con 0x7f38e015f060 2023-10-16T13:47:08.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.596+0000 7f38e6d61700 1 --2- 172.21.15.16:0/2729435759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38e015f060 0x7f38e00c5100 secure :-1 s=READY pgs=2663 cs=0 l=1 rev1=1 crypto rx=0x7f38d800efc0 tx=0x7f38d800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:08.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.596+0000 7f38d6ffd700 1 -- 172.21.15.16:0/2729435759 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38d800ca90 con 0x7f38e015f060 2023-10-16T13:47:08.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.596+0000 7f38d6ffd700 1 -- 172.21.15.16:0/2729435759 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38d800f040 con 0x7f38e015f060 2023-10-16T13:47:08.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.596+0000 7f38e87c4700 1 -- 172.21.15.16:0/2729435759 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38dc0094a0 con 0x7f38e015f060 2023-10-16T13:47:08.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.596+0000 7f38d6ffd700 1 -- 172.21.15.16:0/2729435759 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38d800ce10 con 0x7f38e015f060 2023-10-16T13:47:08.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.597+0000 7f38e87c4700 1 -- 172.21.15.16:0/2729435759 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38e015c000 con 0x7f38e015f060 2023-10-16T13:47:08.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.598+0000 7f38e87c4700 1 -- 172.21.15.16:0/2729435759 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38e004ebb0 con 0x7f38e015f060 2023-10-16T13:47:08.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.598+0000 7f38d6ffd700 1 -- 172.21.15.16:0/2729435759 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 148) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f38d8014480 con 0x7f38e015f060 2023-10-16T13:47:08.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.598+0000 7f38d6ffd700 1 --2- 172.21.15.16:0/2729435759 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f38c406b140 0x7f38c406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:08.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.599+0000 7f38d6ffd700 1 -- 172.21.15.16:0/2729435759 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 7072+0+0 (secure 0 0 0) 0x7f38d808a6b0 con 0x7f38e015f060 2023-10-16T13:47:08.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.599+0000 7f38e6560700 1 --2- 172.21.15.16:0/2729435759 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f38c406b140 0x7f38c406d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:08.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.599+0000 7f38e6560700 1 --2- 172.21.15.16:0/2729435759 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f38c406b140 0x7f38c406d600 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f38dc000c00 tx=0x7f38dc008030 comp rx=0 tx=0).ready entity=mgr.11886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:08.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.603+0000 7f38d6ffd700 1 -- 172.21.15.16:0/2729435759 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f38d8054da0 con 0x7f38e015f060 2023-10-16T13:47:08.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.808+0000 7f38e87c4700 1 -- 172.21.15.16:0/2729435759 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) v1 -- 0x7f38e00b6490 con 0x7f38e015f060 2023-10-16T13:47:08.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.808+0000 7f38d6ffd700 1 -- 172.21.15.16:0/2729435759 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v234) v1 ==== 103+0+11 (secure 0 0 0) 0x7f38d8009d70 con 0x7f38e015f060 2023-10-16T13:47:08.811 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-10-16T13:47:08.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.810+0000 7f38e87c4700 1 -- 172.21.15.16:0/2729435759 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f38c406b140 msgr2=0x7f38c406d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:08.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.810+0000 7f38e87c4700 1 --2- 172.21.15.16:0/2729435759 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f38c406b140 0x7f38c406d600 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f38dc000c00 tx=0x7f38dc008030 comp rx=0 tx=0).stop 2023-10-16T13:47:08.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.810+0000 7f38e87c4700 1 -- 172.21.15.16:0/2729435759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38e015f060 msgr2=0x7f38e00c5100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:08.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.810+0000 7f38e87c4700 1 --2- 172.21.15.16:0/2729435759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38e015f060 0x7f38e00c5100 secure :-1 s=READY pgs=2663 cs=0 l=1 rev1=1 crypto rx=0x7f38d800efc0 tx=0x7f38d800c3a0 comp rx=0 tx=0).stop 2023-10-16T13:47:08.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.811+0000 7f38e87c4700 1 -- 172.21.15.16:0/2729435759 shutdown_connections 2023-10-16T13:47:08.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.811+0000 7f38e87c4700 1 --2- 172.21.15.16:0/2729435759 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f38c406b140 0x7f38c406d600 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:08.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.811+0000 7f38e87c4700 1 --2- 172.21.15.16:0/2729435759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38e015f060 0x7f38e00c5100 unknown :-1 s=CLOSED pgs=2663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:08.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.811+0000 7f38e87c4700 1 --2- 172.21.15.16:0/2729435759 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f38e00d2110 0x7f38e00c4030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:08.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.811+0000 7f38e87c4700 1 --2- 172.21.15.16:0/2729435759 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f38e015ec80 0x7f38e00caf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:08.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.811+0000 7f38e87c4700 1 -- 172.21.15.16:0/2729435759 >> 172.21.15.16:0/2729435759 conn(0x7f38e00d15f0 msgr2=0x7f38e00b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:08.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.811+0000 7f38e87c4700 1 -- 172.21.15.16:0/2729435759 shutdown_connections 2023-10-16T13:47:08.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.811+0000 7f38e87c4700 1 -- 172.21.15.16:0/2729435759 wait complete. 2023-10-16T13:47:08.825 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:47:08.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.948+0000 7f708fcb5700 1 Processor -- start 2023-10-16T13:47:08.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.948+0000 7f708fcb5700 1 -- start start 2023-10-16T13:47:08.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.949+0000 7f708fcb5700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f708815db40 0x7f70880cb980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:08.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.949+0000 7f708fcb5700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70880cedb0 con 0x7f708806a7e0 2023-10-16T13:47:08.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.949+0000 7f708fcb5700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70880d12c0 con 0x7f70880ce870 2023-10-16T13:47:08.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.949+0000 7f708fcb5700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f708815df20 con 0x7f708815db40 2023-10-16T13:47:08.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.949+0000 7f708da51700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f708815db40 0x7f70880cb980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:08.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.950+0000 7f708da51700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f708815db40 0x7f70880cb980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54950/0 (socket says 172.21.15.16:54950) 2023-10-16T13:47:08.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.950+0000 7f708da51700 1 -- 172.21.15.16:0/4236748690 learned_addr learned my addr 172.21.15.16:0/4236748690 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:08.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.950+0000 7f708d250700 1 --1- 172.21.15.16:0/4236748690 >> v1:172.21.15.16:6789/0 conn(0x7f708806a7e0 0x7f70880ce160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44224/0 (socket says 172.21.15.16:44224) 2023-10-16T13:47:08.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.950+0000 7f708da51700 1 -- 172.21.15.16:0/4236748690 >> v1:172.21.15.49:6789/0 conn(0x7f70880ce870 legacy=0x7f70880cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:08.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.950+0000 7f708da51700 1 -- 172.21.15.16:0/4236748690 >> v1:172.21.15.16:6789/0 conn(0x7f708806a7e0 legacy=0x7f70880ce160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:08.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.950+0000 7f708da51700 1 -- 172.21.15.16:0/4236748690 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70880cabc0 con 0x7f708815db40 2023-10-16T13:47:08.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.950+0000 7f708da51700 1 --2- 172.21.15.16:0/4236748690 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f708815db40 0x7f70880cb980 secure :-1 s=READY pgs=2665 cs=0 l=1 rev1=1 crypto rx=0x7f70780098f0 tx=0x7f7078009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc754071f9e1c574 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:08.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.950+0000 7f708ca4f700 1 -- 172.21.15.16:0/4236748690 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f707800b8a0 con 0x7f708815db40 2023-10-16T13:47:08.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.950+0000 7f708ca4f700 1 -- 172.21.15.16:0/4236748690 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f707800ba00 con 0x7f708815db40 2023-10-16T13:47:08.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.950+0000 7f708ca4f700 1 -- 172.21.15.16:0/4236748690 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f707800bd90 con 0x7f708815db40 2023-10-16T13:47:08.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.951+0000 7f708fcb5700 1 -- 172.21.15.16:0/4236748690 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f708815db40 msgr2=0x7f70880cb980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:08.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.951+0000 7f708fcb5700 1 --2- 172.21.15.16:0/4236748690 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f708815db40 0x7f70880cb980 secure :-1 s=READY pgs=2665 cs=0 l=1 rev1=1 crypto rx=0x7f70780098f0 tx=0x7f7078009c00 comp rx=0 tx=0).stop 2023-10-16T13:47:08.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.951+0000 7f708fcb5700 1 -- 172.21.15.16:0/4236748690 shutdown_connections 2023-10-16T13:47:08.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.951+0000 7f708fcb5700 1 --2- 172.21.15.16:0/4236748690 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f708815db40 0x7f70880cb980 unknown :-1 s=CLOSED pgs=2665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:08.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.951+0000 7f708fcb5700 1 -- 172.21.15.16:0/4236748690 >> 172.21.15.16:0/4236748690 conn(0x7f70880b1fd0 msgr2=0x7f70880b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:08.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.951+0000 7f708fcb5700 1 -- 172.21.15.16:0/4236748690 shutdown_connections 2023-10-16T13:47:08.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.951+0000 7f708fcb5700 1 -- 172.21.15.16:0/4236748690 wait complete. 2023-10-16T13:47:08.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.952+0000 7f708fcb5700 1 Processor -- start 2023-10-16T13:47:08.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.952+0000 7f708fcb5700 1 -- start start 2023-10-16T13:47:08.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.953+0000 7f708fcb5700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f708806a7e0 0x7f7088157790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:08.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.953+0000 7f708fcb5700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f70880ce870 0x7f70881f2e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:08.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.953+0000 7f708fcb5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f708815db40 0x7f70880c8c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:08.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.953+0000 7f708fcb5700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70880c9c90 con 0x7f708815db40 2023-10-16T13:47:08.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.953+0000 7f708fcb5700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70880c96d0 con 0x7f70880ce870 2023-10-16T13:47:08.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.953+0000 7f708fcb5700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70880c99b0 con 0x7f708806a7e0 2023-10-16T13:47:08.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.953+0000 7f708e252700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f708815db40 0x7f70880c8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:08.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.953+0000 7f708e252700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f708815db40 0x7f70880c8c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54960/0 (socket says 172.21.15.16:54960) 2023-10-16T13:47:08.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.953+0000 7f708e252700 1 -- 172.21.15.16:0/1694288449 learned_addr learned my addr 172.21.15.16:0/1694288449 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:08.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.953+0000 7f708d250700 1 --2- 172.21.15.16:0/1694288449 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f70880ce870 0x7f70881f2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:08.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.953+0000 7f708da51700 1 --2- 172.21.15.16:0/1694288449 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f708806a7e0 0x7f7088157790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:08.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.953+0000 7f708e252700 1 -- 172.21.15.16:0/1694288449 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f708806a7e0 msgr2=0x7f7088157790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:08.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.953+0000 7f708e252700 1 --2- 172.21.15.16:0/1694288449 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f708806a7e0 0x7f7088157790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:08.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.953+0000 7f708e252700 1 -- 172.21.15.16:0/1694288449 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f70880ce870 msgr2=0x7f70881f2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:08.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.953+0000 7f708e252700 1 --2- 172.21.15.16:0/1694288449 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f70880ce870 0x7f70881f2e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:08.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.953+0000 7f708e252700 1 -- 172.21.15.16:0/1694288449 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7078009550 con 0x7f708815db40 2023-10-16T13:47:08.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.953+0000 7f708e252700 1 --2- 172.21.15.16:0/1694288449 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f708815db40 0x7f70880c8c80 secure :-1 s=READY pgs=2666 cs=0 l=1 rev1=1 crypto rx=0x7f708000ca70 tx=0x7f708000cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:08.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.954+0000 7f707e7fc700 1 -- 172.21.15.16:0/1694288449 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7080003c00 con 0x7f708815db40 2023-10-16T13:47:08.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.954+0000 7f707e7fc700 1 -- 172.21.15.16:0/1694288449 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7080003d60 con 0x7f708815db40 2023-10-16T13:47:08.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.954+0000 7f708fcb5700 1 -- 172.21.15.16:0/1694288449 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7088156f90 con 0x7f708815db40 2023-10-16T13:47:08.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.954+0000 7f707e7fc700 1 -- 172.21.15.16:0/1694288449 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f708000aa20 con 0x7f708815db40 2023-10-16T13:47:08.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.954+0000 7f708fcb5700 1 -- 172.21.15.16:0/1694288449 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7088157cd0 con 0x7f708815db40 2023-10-16T13:47:08.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.955+0000 7f708fcb5700 1 -- 172.21.15.16:0/1694288449 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70881574b0 con 0x7f708815db40 2023-10-16T13:47:08.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.955+0000 7f707e7fc700 1 -- 172.21.15.16:0/1694288449 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 148) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f708000ab80 con 0x7f708815db40 2023-10-16T13:47:08.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.956+0000 7f707e7fc700 1 --2- 172.21.15.16:0/1694288449 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f706c06b140 0x7f706c06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:08.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.956+0000 7f707e7fc700 1 -- 172.21.15.16:0/1694288449 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 7072+0+0 (secure 0 0 0) 0x7f7080089e40 con 0x7f708815db40 2023-10-16T13:47:08.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.956+0000 7f708da51700 1 --2- 172.21.15.16:0/1694288449 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f706c06b140 0x7f706c06d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:08.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.956+0000 7f708da51700 1 --2- 172.21.15.16:0/1694288449 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f706c06b140 0x7f706c06d600 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f7078009c40 tx=0x7f7078008030 comp rx=0 tx=0).ready entity=mgr.11886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:08.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:08.960+0000 7f707e7fc700 1 -- 172.21.15.16:0/1694288449 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f70881574b0 con 0x7f708815db40 2023-10-16T13:47:09.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.210+0000 7f708fcb5700 1 -- 172.21.15.16:0/1694288449 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f70881574b0 con 0x7f708815db40 2023-10-16T13:47:09.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.211+0000 7f707e7fc700 1 -- 172.21.15.16:0/1694288449 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) v1 ==== 95+0+4931 (secure 0 0 0) 0x7f70881574b0 con 0x7f708815db40 2023-10-16T13:47:09.214 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:47:09.214 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":121,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9509,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/1167594604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":1167594604},{"type":"v1","addr":"172.21.15.49:6835","nonce":1167594604}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":121},{"gid":10516,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/4065699432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":4065699432},{"type":"v1","addr":"172.21.15.49:6834","nonce":4065699432}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":108},{"gid":11415,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/1373048456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1373048456},{"type":"v1","addr":"172.21.15.16:6836","nonce":1373048456}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":108}],"filesystems":[{"mdsmap":{"epoch":119,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:46:38.053490+0000","modified":"2023-10-16T13:46:41.076680+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11418},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11418":{"gid":11418,"name":"a","rank":0,"incarnation":115,"state":"up:active","state_seq":6,"addr":"172.21.15.16:6837/3860406853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":3860406853},{"type":"v1","addr":"172.21.15.16:6837","nonce":3860406853}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2023-10-16T13:47:09.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.213+0000 7f708fcb5700 1 -- 172.21.15.16:0/1694288449 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f706c06b140 msgr2=0x7f706c06d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:09.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.213+0000 7f708fcb5700 1 --2- 172.21.15.16:0/1694288449 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f706c06b140 0x7f706c06d600 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f7078009c40 tx=0x7f7078008030 comp rx=0 tx=0).stop 2023-10-16T13:47:09.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.213+0000 7f708fcb5700 1 -- 172.21.15.16:0/1694288449 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f708815db40 msgr2=0x7f70880c8c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:09.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.213+0000 7f708fcb5700 1 --2- 172.21.15.16:0/1694288449 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f708815db40 0x7f70880c8c80 secure :-1 s=READY pgs=2666 cs=0 l=1 rev1=1 crypto rx=0x7f708000ca70 tx=0x7f708000cd80 comp rx=0 tx=0).stop 2023-10-16T13:47:09.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.213+0000 7f708fcb5700 1 -- 172.21.15.16:0/1694288449 shutdown_connections 2023-10-16T13:47:09.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.213+0000 7f708fcb5700 1 --2- 172.21.15.16:0/1694288449 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f706c06b140 0x7f706c06d600 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:09.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.213+0000 7f708fcb5700 1 --2- 172.21.15.16:0/1694288449 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f708815db40 0x7f70880c8c80 unknown :-1 s=CLOSED pgs=2666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:09.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.213+0000 7f708fcb5700 1 --2- 172.21.15.16:0/1694288449 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f708806a7e0 0x7f7088157790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:09.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.213+0000 7f708fcb5700 1 --2- 172.21.15.16:0/1694288449 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f70880ce870 0x7f70881f2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:09.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.213+0000 7f708fcb5700 1 -- 172.21.15.16:0/1694288449 >> 172.21.15.16:0/1694288449 conn(0x7f70880b1fd0 msgr2=0x7f708815a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:09.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.213+0000 7f708fcb5700 1 -- 172.21.15.16:0/1694288449 shutdown_connections 2023-10-16T13:47:09.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.213+0000 7f708fcb5700 1 -- 172.21.15.16:0/1694288449 wait complete. 2023-10-16T13:47:09.219 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 121 2023-10-16T13:47:09.227 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:47:09.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.350+0000 7fc3cd939700 1 Processor -- start 2023-10-16T13:47:09.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.350+0000 7fc3cd939700 1 -- start start 2023-10-16T13:47:09.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.352+0000 7fc3cd939700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc3c80ce010 0x7fc3c80cef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:09.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.352+0000 7fc3cd939700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3c8072a50 con 0x7fc3c815a5c0 2023-10-16T13:47:09.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.352+0000 7fc3cd939700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3c80cba30 con 0x7fc3c80d0ae0 2023-10-16T13:47:09.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.352+0000 7fc3cd939700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3c80cf440 con 0x7fc3c80ce010 2023-10-16T13:47:09.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.352+0000 7fc3c6ffd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc3c815a5c0 0x7fc3c80d03d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44238/0 (socket says 172.21.15.16:44238) 2023-10-16T13:47:09.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.352+0000 7fc3c6ffd700 1 -- 172.21.15.16:0/674236520 learned_addr learned my addr 172.21.15.16:0/674236520 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:47:09.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.352+0000 7fc3c77fe700 1 --2- 172.21.15.16:0/674236520 >> v2:172.21.15.16:3300/0 conn(0x7fc3c80ce010 0x7fc3c80cef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:09.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.352+0000 7fc3c77fe700 1 -- 172.21.15.16:0/674236520 >> v1:172.21.15.49:6789/0 conn(0x7fc3c80d0ae0 legacy=0x7fc3c80cc1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:09.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.352+0000 7fc3c77fe700 1 -- 172.21.15.16:0/674236520 >> v1:172.21.15.16:6789/0 conn(0x7fc3c815a5c0 legacy=0x7fc3c80d03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:09.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.352+0000 7fc3c77fe700 1 -- 172.21.15.16:0/674236520 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3c80cf630 con 0x7fc3c80ce010 2023-10-16T13:47:09.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.352+0000 7fc3c77fe700 1 --2- 172.21.15.16:0/674236520 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3c80ce010 0x7fc3c80cef00 secure :-1 s=READY pgs=2668 cs=0 l=1 rev1=1 crypto rx=0x7fc3c0009d90 tx=0x7fc3c000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30b93dfa6043cf30 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:09.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.352+0000 7fc3c5ffb700 1 -- 172.21.15.16:0/674236520 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3c000bdd0 con 0x7fc3c80ce010 2023-10-16T13:47:09.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.352+0000 7fc3c5ffb700 1 -- 172.21.15.16:0/674236520 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3c0011460 con 0x7fc3c80ce010 2023-10-16T13:47:09.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.352+0000 7fc3c5ffb700 1 -- 172.21.15.16:0/674236520 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3c0011810 con 0x7fc3c80ce010 2023-10-16T13:47:09.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.353+0000 7fc3cd939700 1 -- 172.21.15.16:0/674236520 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3c80ce010 msgr2=0x7fc3c80cef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:09.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.353+0000 7fc3cd939700 1 --2- 172.21.15.16:0/674236520 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3c80ce010 0x7fc3c80cef00 secure :-1 s=READY pgs=2668 cs=0 l=1 rev1=1 crypto rx=0x7fc3c0009d90 tx=0x7fc3c000b330 comp rx=0 tx=0).stop 2023-10-16T13:47:09.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.353+0000 7fc3cd939700 1 -- 172.21.15.16:0/674236520 shutdown_connections 2023-10-16T13:47:09.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.353+0000 7fc3cd939700 1 --2- 172.21.15.16:0/674236520 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3c80ce010 0x7fc3c80cef00 unknown :-1 s=CLOSED pgs=2668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:09.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.353+0000 7fc3cd939700 1 -- 172.21.15.16:0/674236520 >> 172.21.15.16:0/674236520 conn(0x7fc3c80b1fd0 msgr2=0x7fc3c80b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:09.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.353+0000 7fc3cd939700 1 -- 172.21.15.16:0/674236520 shutdown_connections 2023-10-16T13:47:09.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.353+0000 7fc3cd939700 1 -- 172.21.15.16:0/674236520 wait complete. 2023-10-16T13:47:09.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.355+0000 7fc3cd939700 1 Processor -- start 2023-10-16T13:47:09.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.355+0000 7fc3cd939700 1 -- start start 2023-10-16T13:47:09.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.356+0000 7fc3cd939700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3c80ce010 0x7fc3c80ccef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:09.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.356+0000 7fc3cd939700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc3c80d0ae0 0x7fc3c81505e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:09.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.356+0000 7fc3cd939700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc3c815a5c0 0x7fc3c8150b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:09.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.356+0000 7fc3cd939700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3c81f33b0 con 0x7fc3c80ce010 2023-10-16T13:47:09.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.356+0000 7fc3cd939700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3c81f2830 con 0x7fc3c80d0ae0 2023-10-16T13:47:09.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.356+0000 7fc3cd939700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3c81f2b10 con 0x7fc3c815a5c0 2023-10-16T13:47:09.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.356+0000 7fc3c6ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3c80ce010 0x7fc3c80ccef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:09.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.356+0000 7fc3c6ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3c80ce010 0x7fc3c80ccef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54970/0 (socket says 172.21.15.16:54970) 2023-10-16T13:47:09.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.356+0000 7fc3c6ffd700 1 -- 172.21.15.16:0/2570152877 learned_addr learned my addr 172.21.15.16:0/2570152877 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:09.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.356+0000 7fc3c67fc700 1 --2- 172.21.15.16:0/2570152877 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc3c80d0ae0 0x7fc3c81505e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:09.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.356+0000 7fc3c77fe700 1 --2- 172.21.15.16:0/2570152877 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc3c815a5c0 0x7fc3c8150b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:09.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.356+0000 7fc3c6ffd700 1 -- 172.21.15.16:0/2570152877 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc3c815a5c0 msgr2=0x7fc3c8150b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:09.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.356+0000 7fc3c6ffd700 1 --2- 172.21.15.16:0/2570152877 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc3c815a5c0 0x7fc3c8150b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:09.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.356+0000 7fc3c6ffd700 1 -- 172.21.15.16:0/2570152877 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc3c80d0ae0 msgr2=0x7fc3c81505e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:09.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.356+0000 7fc3c6ffd700 1 --2- 172.21.15.16:0/2570152877 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc3c80d0ae0 0x7fc3c81505e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:09.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.356+0000 7fc3c6ffd700 1 -- 172.21.15.16:0/2570152877 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3c0009a70 con 0x7fc3c80ce010 2023-10-16T13:47:09.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.356+0000 7fc3c6ffd700 1 --2- 172.21.15.16:0/2570152877 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3c80ce010 0x7fc3c80ccef0 secure :-1 s=READY pgs=2669 cs=0 l=1 rev1=1 crypto rx=0x7fc3bc00c480 tx=0x7fc3bc00c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:09.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.356+0000 7fc3affff700 1 -- 172.21.15.16:0/2570152877 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3bc005d70 con 0x7fc3c80ce010 2023-10-16T13:47:09.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.356+0000 7fc3affff700 1 -- 172.21.15.16:0/2570152877 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3bc0070d0 con 0x7fc3c80ce010 2023-10-16T13:47:09.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.356+0000 7fc3affff700 1 -- 172.21.15.16:0/2570152877 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3bc007440 con 0x7fc3c80ce010 2023-10-16T13:47:09.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.357+0000 7fc3cd939700 1 -- 172.21.15.16:0/2570152877 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3c81f2df0 con 0x7fc3c80ce010 2023-10-16T13:47:09.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.357+0000 7fc3cd939700 1 -- 172.21.15.16:0/2570152877 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3c81f2450 con 0x7fc3c80ce010 2023-10-16T13:47:09.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.358+0000 7fc3cd939700 1 -- 172.21.15.16:0/2570152877 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3c81f25e0 con 0x7fc3c80ce010 2023-10-16T13:47:09.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.358+0000 7fc3affff700 1 -- 172.21.15.16:0/2570152877 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 148) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fc3bc0075a0 con 0x7fc3c80ce010 2023-10-16T13:47:09.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.359+0000 7fc3affff700 1 --2- 172.21.15.16:0/2570152877 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fc3b406b170 0x7fc3b406d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:09.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.359+0000 7fc3c67fc700 1 --2- 172.21.15.16:0/2570152877 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fc3b406b170 0x7fc3b406d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:09.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.359+0000 7fc3affff700 1 -- 172.21.15.16:0/2570152877 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 7072+0+0 (secure 0 0 0) 0x7fc3bc089310 con 0x7fc3c80ce010 2023-10-16T13:47:09.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.359+0000 7fc3c67fc700 1 --2- 172.21.15.16:0/2570152877 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fc3b406b170 0x7fc3b406d630 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fc3c815b8d0 tx=0x7fc3b8009040 comp rx=0 tx=0).ready entity=mgr.11886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:09.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.365+0000 7fc3affff700 1 -- 172.21.15.16:0/2570152877 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc3bc053c90 con 0x7fc3c80ce010 2023-10-16T13:47:09.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.568+0000 7fc3cd939700 1 -- 172.21.15.16:0/2570152877 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc3c804ebb0 con 0x7fc3c80ce010 2023-10-16T13:47:09.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.570+0000 7fc3affff700 1 -- 172.21.15.16:0/2570152877 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v234) v1 ==== 74+0+18143 (secure 0 0 0) 0x7fc3bc0572b0 con 0x7fc3c80ce010 2023-10-16T13:47:09.573 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:47:09.573 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":234,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:47:07.345418+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T13:46:35.800444+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"228","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"backup_fs_data","create_time":"2023-10-16T13:46:37.035323+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"229","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.49:6835/2905760119":"2023-10-17T13:47:04.337914+0000","172.21.15.16:0/1827947757":"2023-10-17T13:47:01.801871+0000","172.21.15.16:0/1694897016":"2023-10-17T13:47:01.801871+0000","172.21.15.16:0/3760456275":"2023-10-17T13:47:01.801871+0000","172.21.15.16:0/3651530055":"2023-10-17T13:47:01.801871+0000","172.21.15.16:0/850397235":"2023-10-17T13:47:01.801871+0000","172.21.15.16:6833/1506142504":"2023-10-17T13:47:01.801871+0000","172.21.15.16:0/4239949371":"2023-10-17T13:47:01.801871+0000","172.21.15.16:0/2321929682":"2023-10-17T13:46:51.165018+0000","172.21.15.16:0/3155275276":"2023-10-17T13:47:01.801871+0000","172.21.15.16:0/3429506235":"2023-10-17T13:46:51.165018+0000","172.21.15.49:6833/2905760119":"2023-10-17T13:47:04.337914+0000","172.21.15.16:0/4090083700":"2023-10-17T13:47:01.801871+0000","172.21.15.16:0/3843126404":"2023-10-17T13:46:51.165018+0000","172.21.15.16:6832/1506142504":"2023-10-17T13:47:01.801871+0000","172.21.15.16:0/1439318665":"2023-10-17T13:46:51.165018+0000","172.21.15.16:0/4204021371":"2023-10-17T13:46:51.165018+0000","172.21.15.16:6833/3171422553":"2023-10-17T13:46:51.165018+0000","172.21.15.16:6832/3171422553":"2023-10-17T13:46:51.165018+0000","172.21.15.16:0/1691810194":"2023-10-17T13:47:01.801871+0000","172.21.15.16:0/1632287797":"2023-10-17T13:46:51.165018+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:47:09.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.572+0000 7fc3cd939700 1 -- 172.21.15.16:0/2570152877 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fc3b406b170 msgr2=0x7fc3b406d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:09.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.572+0000 7fc3cd939700 1 --2- 172.21.15.16:0/2570152877 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fc3b406b170 0x7fc3b406d630 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fc3c815b8d0 tx=0x7fc3b8009040 comp rx=0 tx=0).stop 2023-10-16T13:47:09.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.572+0000 7fc3cd939700 1 -- 172.21.15.16:0/2570152877 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3c80ce010 msgr2=0x7fc3c80ccef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:09.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.572+0000 7fc3cd939700 1 --2- 172.21.15.16:0/2570152877 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3c80ce010 0x7fc3c80ccef0 secure :-1 s=READY pgs=2669 cs=0 l=1 rev1=1 crypto rx=0x7fc3bc00c480 tx=0x7fc3bc00c840 comp rx=0 tx=0).stop 2023-10-16T13:47:09.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.572+0000 7fc3cd939700 1 -- 172.21.15.16:0/2570152877 shutdown_connections 2023-10-16T13:47:09.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.572+0000 7fc3cd939700 1 --2- 172.21.15.16:0/2570152877 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fc3b406b170 0x7fc3b406d630 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:09.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.572+0000 7fc3cd939700 1 --2- 172.21.15.16:0/2570152877 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc3c815a5c0 0x7fc3c8150b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:09.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.572+0000 7fc3cd939700 1 --2- 172.21.15.16:0/2570152877 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3c80ce010 0x7fc3c80ccef0 unknown :-1 s=CLOSED pgs=2669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:09.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.572+0000 7fc3cd939700 1 --2- 172.21.15.16:0/2570152877 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc3c80d0ae0 0x7fc3c81505e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:09.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.572+0000 7fc3cd939700 1 -- 172.21.15.16:0/2570152877 >> 172.21.15.16:0/2570152877 conn(0x7fc3c80b1fd0 msgr2=0x7fc3c80d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:09.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.572+0000 7fc3cd939700 1 -- 172.21.15.16:0/2570152877 shutdown_connections 2023-10-16T13:47:09.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.572+0000 7fc3cd939700 1 -- 172.21.15.16:0/2570152877 wait complete. 2023-10-16T13:47:09.587 INFO:tasks.cephfs.filesystem:Destroying file system backup_fs and related pools 2023-10-16T13:47:09.587 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:47:09.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.712+0000 7f2ba250b700 1 Processor -- start 2023-10-16T13:47:09.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.712+0000 7f2ba250b700 1 -- start start 2023-10-16T13:47:09.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.713+0000 7f2ba250b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2b9c06a7e0 0x7f2b9c06ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:09.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.713+0000 7f2ba250b700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b9c06b150 con 0x7f2b9c0cc8a0 2023-10-16T13:47:09.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.713+0000 7f2ba250b700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b9c161c30 con 0x7f2b9c15b280 2023-10-16T13:47:09.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.713+0000 7f2ba250b700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b9c161f10 con 0x7f2b9c06a7e0 2023-10-16T13:47:09.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.713+0000 7f2ba0aa8700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2b9c06a7e0 0x7f2b9c06ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:09.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.713+0000 7f2ba0aa8700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2b9c06a7e0 0x7f2b9c06ac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54982/0 (socket says 172.21.15.16:54982) 2023-10-16T13:47:09.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.713+0000 7f2ba0aa8700 1 -- 172.21.15.16:0/3071733929 learned_addr learned my addr 172.21.15.16:0/3071733929 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:09.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.713+0000 7f2b9b7fe700 1 --1- 172.21.15.16:0/3071733929 >> v1:172.21.15.16:6789/0 conn(0x7f2b9c0cc8a0 0x7f2b9c15e700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44254/0 (socket says 172.21.15.16:44254) 2023-10-16T13:47:09.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.713+0000 7f2ba0aa8700 1 -- 172.21.15.16:0/3071733929 >> v1:172.21.15.49:6789/0 conn(0x7f2b9c15b280 legacy=0x7f2b9c1613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:09.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.713+0000 7f2ba0aa8700 1 -- 172.21.15.16:0/3071733929 >> v1:172.21.15.16:6789/0 conn(0x7f2b9c0cc8a0 legacy=0x7f2b9c15e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:09.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.713+0000 7f2ba0aa8700 1 -- 172.21.15.16:0/3071733929 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b9c1621f0 con 0x7f2b9c06a7e0 2023-10-16T13:47:09.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.713+0000 7f2ba0aa8700 1 --2- 172.21.15.16:0/3071733929 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b9c06a7e0 0x7f2b9c06ac10 secure :-1 s=READY pgs=2671 cs=0 l=1 rev1=1 crypto rx=0x7f2b94009e50 tx=0x7f2b9400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bac3d76933da0dc3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:09.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.713+0000 7f2b9affd700 1 -- 172.21.15.16:0/3071733929 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b9400e070 con 0x7f2b9c06a7e0 2023-10-16T13:47:09.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.713+0000 7f2b9affd700 1 -- 172.21.15.16:0/3071733929 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b9400b9c0 con 0x7f2b9c06a7e0 2023-10-16T13:47:09.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.713+0000 7f2b9affd700 1 -- 172.21.15.16:0/3071733929 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b9400bd70 con 0x7f2b9c06a7e0 2023-10-16T13:47:09.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.714+0000 7f2ba250b700 1 -- 172.21.15.16:0/3071733929 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b9c06a7e0 msgr2=0x7f2b9c06ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:09.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.714+0000 7f2ba250b700 1 --2- 172.21.15.16:0/3071733929 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b9c06a7e0 0x7f2b9c06ac10 secure :-1 s=READY pgs=2671 cs=0 l=1 rev1=1 crypto rx=0x7f2b94009e50 tx=0x7f2b9400b330 comp rx=0 tx=0).stop 2023-10-16T13:47:09.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.714+0000 7f2ba250b700 1 -- 172.21.15.16:0/3071733929 shutdown_connections 2023-10-16T13:47:09.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.714+0000 7f2ba250b700 1 --2- 172.21.15.16:0/3071733929 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b9c06a7e0 0x7f2b9c06ac10 unknown :-1 s=CLOSED pgs=2671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:09.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.714+0000 7f2ba250b700 1 -- 172.21.15.16:0/3071733929 >> 172.21.15.16:0/3071733929 conn(0x7f2b9c0b1fd0 msgr2=0x7f2b9c0b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:09.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.714+0000 7f2ba250b700 1 -- 172.21.15.16:0/3071733929 shutdown_connections 2023-10-16T13:47:09.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.714+0000 7f2ba250b700 1 -- 172.21.15.16:0/3071733929 wait complete. 2023-10-16T13:47:09.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.715+0000 7f2ba250b700 1 Processor -- start 2023-10-16T13:47:09.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.715+0000 7f2ba250b700 1 -- start start 2023-10-16T13:47:09.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.716+0000 7f2ba250b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b9c06a7e0 0x7f2b9c1588d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:09.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.716+0000 7f2ba250b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2b9c0cc8a0 0x7f2b9c0c83b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:09.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.716+0000 7f2ba250b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2b9c15b280 0x7f2b9c15b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:09.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.716+0000 7f2ba250b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b9c0c6020 con 0x7f2b9c06a7e0 2023-10-16T13:47:09.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.716+0000 7f2ba250b700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b9c0c56f0 con 0x7f2b9c0cc8a0 2023-10-16T13:47:09.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.716+0000 7f2ba250b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b9c0c59d0 con 0x7f2b9c15b280 2023-10-16T13:47:09.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.716+0000 7f2b9bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b9c06a7e0 0x7f2b9c1588d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:09.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.716+0000 7f2b9bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b9c06a7e0 0x7f2b9c1588d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54984/0 (socket says 172.21.15.16:54984) 2023-10-16T13:47:09.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.716+0000 7f2b9bfff700 1 -- 172.21.15.16:0/1686591752 learned_addr learned my addr 172.21.15.16:0/1686591752 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:09.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.716+0000 7f2b9b7fe700 1 --2- 172.21.15.16:0/1686591752 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2b9c0cc8a0 0x7f2b9c0c83b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:09.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.717+0000 7f2b9bfff700 1 -- 172.21.15.16:0/1686591752 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2b9c15b280 msgr2=0x7f2b9c15b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:09.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.717+0000 7f2b9bfff700 1 --2- 172.21.15.16:0/1686591752 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2b9c15b280 0x7f2b9c15b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:09.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.717+0000 7f2b9bfff700 1 -- 172.21.15.16:0/1686591752 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2b9c0cc8a0 msgr2=0x7f2b9c0c83b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:09.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.717+0000 7f2b9bfff700 1 --2- 172.21.15.16:0/1686591752 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2b9c0cc8a0 0x7f2b9c0c83b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:09.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.717+0000 7f2b9bfff700 1 -- 172.21.15.16:0/1686591752 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b94009b30 con 0x7f2b9c06a7e0 2023-10-16T13:47:09.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.717+0000 7f2b9bfff700 1 --2- 172.21.15.16:0/1686591752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b9c06a7e0 0x7f2b9c1588d0 secure :-1 s=READY pgs=2672 cs=0 l=1 rev1=1 crypto rx=0x7f2b8c00c450 tx=0x7f2b8c00c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:09.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.717+0000 7f2b98ff9700 1 -- 172.21.15.16:0/1686591752 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b8c0052d0 con 0x7f2b9c06a7e0 2023-10-16T13:47:09.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.717+0000 7f2b98ff9700 1 -- 172.21.15.16:0/1686591752 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b8c005430 con 0x7f2b9c06a7e0 2023-10-16T13:47:09.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.717+0000 7f2ba250b700 1 -- 172.21.15.16:0/1686591752 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b9c0c5cb0 con 0x7f2b9c06a7e0 2023-10-16T13:47:09.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.717+0000 7f2b98ff9700 1 -- 172.21.15.16:0/1686591752 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b8c0038c0 con 0x7f2b9c06a7e0 2023-10-16T13:47:09.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.717+0000 7f2ba250b700 1 -- 172.21.15.16:0/1686591752 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b9c158e10 con 0x7f2b9c06a7e0 2023-10-16T13:47:09.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.718+0000 7f2ba250b700 1 -- 172.21.15.16:0/1686591752 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b9c04ebb0 con 0x7f2b9c06a7e0 2023-10-16T13:47:09.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.718+0000 7f2b98ff9700 1 -- 172.21.15.16:0/1686591752 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 148) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f2b8c003a20 con 0x7f2b9c06a7e0 2023-10-16T13:47:09.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.719+0000 7f2b98ff9700 1 --2- 172.21.15.16:0/1686591752 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f2b8806b140 0x7f2b8806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:09.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.719+0000 7f2b98ff9700 1 -- 172.21.15.16:0/1686591752 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 7072+0+0 (secure 0 0 0) 0x7f2b8c089290 con 0x7f2b9c06a7e0 2023-10-16T13:47:09.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.719+0000 7f2b9b7fe700 1 --2- 172.21.15.16:0/1686591752 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f2b8806b140 0x7f2b8806d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:09.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.719+0000 7f2b9b7fe700 1 --2- 172.21.15.16:0/1686591752 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f2b8806b140 0x7f2b8806d600 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2b9c0afa20 tx=0x7f2b90009040 comp rx=0 tx=0).ready entity=mgr.11886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:09.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.723+0000 7f2b98ff9700 1 -- 172.21.15.16:0/1686591752 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2b8c053b60 con 0x7f2b9c06a7e0 2023-10-16T13:47:09.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.971+0000 7f2ba250b700 1 -- 172.21.15.16:0/1686591752 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2b9c0cc050 con 0x7f2b9c06a7e0 2023-10-16T13:47:09.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.972+0000 7f2b98ff9700 1 -- 172.21.15.16:0/1686591752 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) v1 ==== 95+0+4931 (secure 0 0 0) 0x7f2b8c057180 con 0x7f2b9c06a7e0 2023-10-16T13:47:09.975 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:47:09.975 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":121,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9509,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/1167594604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":1167594604},{"type":"v1","addr":"172.21.15.49:6835","nonce":1167594604}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":121},{"gid":10516,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/4065699432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":4065699432},{"type":"v1","addr":"172.21.15.49:6834","nonce":4065699432}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":108},{"gid":11415,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/1373048456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1373048456},{"type":"v1","addr":"172.21.15.16:6836","nonce":1373048456}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":108}],"filesystems":[{"mdsmap":{"epoch":119,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:46:38.053490+0000","modified":"2023-10-16T13:46:41.076680+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11418},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11418":{"gid":11418,"name":"a","rank":0,"incarnation":115,"state":"up:active","state_seq":6,"addr":"172.21.15.16:6837/3860406853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":3860406853},{"type":"v1","addr":"172.21.15.16:6837","nonce":3860406853}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2023-10-16T13:47:09.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.974+0000 7f2ba250b700 1 -- 172.21.15.16:0/1686591752 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f2b8806b140 msgr2=0x7f2b8806d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:09.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.974+0000 7f2ba250b700 1 --2- 172.21.15.16:0/1686591752 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f2b8806b140 0x7f2b8806d600 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2b9c0afa20 tx=0x7f2b90009040 comp rx=0 tx=0).stop 2023-10-16T13:47:09.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.974+0000 7f2ba250b700 1 -- 172.21.15.16:0/1686591752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b9c06a7e0 msgr2=0x7f2b9c1588d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:09.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.974+0000 7f2ba250b700 1 --2- 172.21.15.16:0/1686591752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b9c06a7e0 0x7f2b9c1588d0 secure :-1 s=READY pgs=2672 cs=0 l=1 rev1=1 crypto rx=0x7f2b8c00c450 tx=0x7f2b8c00c810 comp rx=0 tx=0).stop 2023-10-16T13:47:09.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.974+0000 7f2ba250b700 1 -- 172.21.15.16:0/1686591752 shutdown_connections 2023-10-16T13:47:09.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.974+0000 7f2ba250b700 1 --2- 172.21.15.16:0/1686591752 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f2b8806b140 0x7f2b8806d600 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:09.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.974+0000 7f2ba250b700 1 --2- 172.21.15.16:0/1686591752 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2b9c15b280 0x7f2b9c15b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:09.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.974+0000 7f2ba250b700 1 --2- 172.21.15.16:0/1686591752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b9c06a7e0 0x7f2b9c1588d0 unknown :-1 s=CLOSED pgs=2672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:09.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.974+0000 7f2ba250b700 1 --2- 172.21.15.16:0/1686591752 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2b9c0cc8a0 0x7f2b9c0c83b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:09.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.974+0000 7f2ba250b700 1 -- 172.21.15.16:0/1686591752 >> 172.21.15.16:0/1686591752 conn(0x7f2b9c0b1fd0 msgr2=0x7f2b9c0b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:09.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.974+0000 7f2ba250b700 1 -- 172.21.15.16:0/1686591752 shutdown_connections 2023-10-16T13:47:09.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:09.974+0000 7f2ba250b700 1 -- 172.21.15.16:0/1686591752 wait complete. 2023-10-16T13:47:09.980 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 121 2023-10-16T13:47:09.989 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:47:10.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.112+0000 7fc40f063700 1 Processor -- start 2023-10-16T13:47:10.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.112+0000 7fc40f063700 1 -- start start 2023-10-16T13:47:10.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.113+0000 7fc40f063700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc4080cf4a0 0x7fc4080726e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:10.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.113+0000 7fc40f063700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc4080cf880 0x7fc4080b52f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:10.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.113+0000 7fc40f063700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc40815bc80 con 0x7fc4080b5830 2023-10-16T13:47:10.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.113+0000 7fc40f063700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc408072c20 con 0x7fc4080cf880 2023-10-16T13:47:10.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.113+0000 7fc40f063700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc40815b070 con 0x7fc4080cf4a0 2023-10-16T13:47:10.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.113+0000 7fc407fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc4080cf880 0x7fc4080b52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:10.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.113+0000 7fc407fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc4080cf880 0x7fc4080b52f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54992/0 (socket says 172.21.15.16:54992) 2023-10-16T13:47:10.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.113+0000 7fc407fff700 1 -- 172.21.15.16:0/3905269836 learned_addr learned my addr 172.21.15.16:0/3905269836 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:10.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.113+0000 7fc40cdff700 1 --2- 172.21.15.16:0/3905269836 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc4080cf4a0 0x7fc4080726e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:10.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.114+0000 7fc407fff700 1 -- 172.21.15.16:0/3905269836 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc4080cf4a0 msgr2=0x7fc4080726e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:10.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.114+0000 7fc407fff700 1 --2- 172.21.15.16:0/3905269836 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc4080cf4a0 0x7fc4080726e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:10.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.114+0000 7fc407fff700 1 -- 172.21.15.16:0/3905269836 >> v1:172.21.15.16:6789/0 conn(0x7fc4080b5830 legacy=0x7fc40815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:10.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.114+0000 7fc4077fe700 1 -- 172.21.15.16:0/3905269836 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 58795964 0 0) 0x7fc40815bc80 con 0x7fc4080b5830 2023-10-16T13:47:10.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.114+0000 7fc407fff700 1 -- 172.21.15.16:0/3905269836 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc40815b350 con 0x7fc4080cf880 2023-10-16T13:47:10.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.114+0000 7fc407fff700 1 --2- 172.21.15.16:0/3905269836 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4080cf880 0x7fc4080b52f0 secure :-1 s=READY pgs=2674 cs=0 l=1 rev1=1 crypto rx=0x7fc3f40098f0 tx=0x7fc3f4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a875e02503c873af server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:10.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.114+0000 7fc4077fe700 1 -- 172.21.15.16:0/3905269836 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f400b8a0 con 0x7fc4080cf880 2023-10-16T13:47:10.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.114+0000 7fc4077fe700 1 -- 172.21.15.16:0/3905269836 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3f400ba00 con 0x7fc4080cf880 2023-10-16T13:47:10.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.114+0000 7fc4077fe700 1 -- 172.21.15.16:0/3905269836 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f400bd90 con 0x7fc4080cf880 2023-10-16T13:47:10.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.115+0000 7fc40f063700 1 -- 172.21.15.16:0/3905269836 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4080cf880 msgr2=0x7fc4080b52f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:10.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.115+0000 7fc40f063700 1 --2- 172.21.15.16:0/3905269836 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4080cf880 0x7fc4080b52f0 secure :-1 s=READY pgs=2674 cs=0 l=1 rev1=1 crypto rx=0x7fc3f40098f0 tx=0x7fc3f4009c00 comp rx=0 tx=0).stop 2023-10-16T13:47:10.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.115+0000 7fc40f063700 1 -- 172.21.15.16:0/3905269836 shutdown_connections 2023-10-16T13:47:10.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.115+0000 7fc40f063700 1 --2- 172.21.15.16:0/3905269836 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc4080cf4a0 0x7fc4080726e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:10.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.115+0000 7fc40f063700 1 --2- 172.21.15.16:0/3905269836 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4080cf880 0x7fc4080b52f0 unknown :-1 s=CLOSED pgs=2674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:10.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.115+0000 7fc40f063700 1 -- 172.21.15.16:0/3905269836 >> 172.21.15.16:0/3905269836 conn(0x7fc4080b1fd0 msgr2=0x7fc4080b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:10.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.115+0000 7fc40f063700 1 -- 172.21.15.16:0/3905269836 shutdown_connections 2023-10-16T13:47:10.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.115+0000 7fc40f063700 1 -- 172.21.15.16:0/3905269836 wait complete. 2023-10-16T13:47:10.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.116+0000 7fc40f063700 1 Processor -- start 2023-10-16T13:47:10.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.116+0000 7fc40f063700 1 -- start start 2023-10-16T13:47:10.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.117+0000 7fc40f063700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4080b5830 0x7fc4080c99e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:10.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.117+0000 7fc40f063700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc4080cf4a0 0x7fc4080c7510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:10.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.117+0000 7fc40f063700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc4080cf880 0x7fc4080cd270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:10.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.117+0000 7fc40f063700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc40815cab0 con 0x7fc4080b5830 2023-10-16T13:47:10.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.117+0000 7fc40f063700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4080c7ba0 con 0x7fc4080cf880 2023-10-16T13:47:10.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.117+0000 7fc40f063700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4080c7e80 con 0x7fc4080cf4a0 2023-10-16T13:47:10.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.118+0000 7fc40cdff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4080b5830 0x7fc4080c99e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:10.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.118+0000 7fc40cdff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4080b5830 0x7fc4080c99e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54998/0 (socket says 172.21.15.16:54998) 2023-10-16T13:47:10.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.118+0000 7fc40cdff700 1 -- 172.21.15.16:0/1174937239 learned_addr learned my addr 172.21.15.16:0/1174937239 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:10.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.118+0000 7fc40cdff700 1 -- 172.21.15.16:0/1174937239 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc4080cf4a0 msgr2=0x7fc4080c7510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:10.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.118+0000 7fc40cdff700 1 --2- 172.21.15.16:0/1174937239 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc4080cf4a0 0x7fc4080c7510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:10.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.118+0000 7fc40cdff700 1 -- 172.21.15.16:0/1174937239 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc4080cf880 msgr2=0x7fc4080cd270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:10.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.118+0000 7fc40cdff700 1 --2- 172.21.15.16:0/1174937239 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc4080cf880 0x7fc4080cd270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:10.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.118+0000 7fc40cdff700 1 -- 172.21.15.16:0/1174937239 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3f4009550 con 0x7fc4080b5830 2023-10-16T13:47:10.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.118+0000 7fc40cdff700 1 --2- 172.21.15.16:0/1174937239 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4080b5830 0x7fc4080c99e0 secure :-1 s=READY pgs=2675 cs=0 l=1 rev1=1 crypto rx=0x7fc3fc00e810 tx=0x7fc3fc00ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:10.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.118+0000 7fc4057fa700 1 -- 172.21.15.16:0/1174937239 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3fc013610 con 0x7fc4080b5830 2023-10-16T13:47:10.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.118+0000 7fc4057fa700 1 -- 172.21.15.16:0/1174937239 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3fc013770 con 0x7fc4080b5830 2023-10-16T13:47:10.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.118+0000 7fc40f063700 1 -- 172.21.15.16:0/1174937239 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4080c8160 con 0x7fc4080b5830 2023-10-16T13:47:10.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.118+0000 7fc4057fa700 1 -- 172.21.15.16:0/1174937239 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3fc013b20 con 0x7fc4080b5830 2023-10-16T13:47:10.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.118+0000 7fc40f063700 1 -- 172.21.15.16:0/1174937239 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4080cd7b0 con 0x7fc4080b5830 2023-10-16T13:47:10.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.119+0000 7fc40f063700 1 -- 172.21.15.16:0/1174937239 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4080c8680 con 0x7fc4080b5830 2023-10-16T13:47:10.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.120+0000 7fc4057fa700 1 -- 172.21.15.16:0/1174937239 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 148) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fc3fc013c80 con 0x7fc4080b5830 2023-10-16T13:47:10.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.120+0000 7fc4057fa700 1 --2- 172.21.15.16:0/1174937239 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fc3ec06b140 0x7fc3ec06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:10.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.120+0000 7fc4057fa700 1 -- 172.21.15.16:0/1174937239 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 7072+0+0 (secure 0 0 0) 0x7fc3fc089d10 con 0x7fc4080b5830 2023-10-16T13:47:10.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.120+0000 7fc407fff700 1 --2- 172.21.15.16:0/1174937239 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fc3ec06b140 0x7fc3ec06d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:10.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.121+0000 7fc407fff700 1 --2- 172.21.15.16:0/1174937239 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fc3ec06b140 0x7fc3ec06d600 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc4080c8d90 tx=0x7fc3f4010000 comp rx=0 tx=0).ready entity=mgr.11886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:10.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.125+0000 7fc4057fa700 1 -- 172.21.15.16:0/1174937239 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc4080c8680 con 0x7fc4080b5830 2023-10-16T13:47:10.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.376+0000 7fc40f063700 1 -- 172.21.15.16:0/1174937239 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc4080c8680 con 0x7fc4080b5830 2023-10-16T13:47:10.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.377+0000 7fc4057fa700 1 -- 172.21.15.16:0/1174937239 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) v1 ==== 95+0+4931 (secure 0 0 0) 0x7fc4080c8680 con 0x7fc4080b5830 2023-10-16T13:47:10.380 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:47:10.380 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":121,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9509,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/1167594604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":1167594604},{"type":"v1","addr":"172.21.15.49:6835","nonce":1167594604}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":121},{"gid":10516,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/4065699432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":4065699432},{"type":"v1","addr":"172.21.15.49:6834","nonce":4065699432}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":108},{"gid":11415,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/1373048456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1373048456},{"type":"v1","addr":"172.21.15.16:6836","nonce":1373048456}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":108}],"filesystems":[{"mdsmap":{"epoch":119,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:46:38.053490+0000","modified":"2023-10-16T13:46:41.076680+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11418},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11418":{"gid":11418,"name":"a","rank":0,"incarnation":115,"state":"up:active","state_seq":6,"addr":"172.21.15.16:6837/3860406853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":3860406853},{"type":"v1","addr":"172.21.15.16:6837","nonce":3860406853}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2023-10-16T13:47:10.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.379+0000 7fc40f063700 1 -- 172.21.15.16:0/1174937239 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fc3ec06b140 msgr2=0x7fc3ec06d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:10.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.379+0000 7fc40f063700 1 --2- 172.21.15.16:0/1174937239 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fc3ec06b140 0x7fc3ec06d600 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc4080c8d90 tx=0x7fc3f4010000 comp rx=0 tx=0).stop 2023-10-16T13:47:10.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.379+0000 7fc40f063700 1 -- 172.21.15.16:0/1174937239 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4080b5830 msgr2=0x7fc4080c99e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:10.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.379+0000 7fc40f063700 1 --2- 172.21.15.16:0/1174937239 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4080b5830 0x7fc4080c99e0 secure :-1 s=READY pgs=2675 cs=0 l=1 rev1=1 crypto rx=0x7fc3fc00e810 tx=0x7fc3fc00ebd0 comp rx=0 tx=0).stop 2023-10-16T13:47:10.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.379+0000 7fc40f063700 1 -- 172.21.15.16:0/1174937239 shutdown_connections 2023-10-16T13:47:10.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.379+0000 7fc40f063700 1 --2- 172.21.15.16:0/1174937239 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fc3ec06b140 0x7fc3ec06d600 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:10.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.379+0000 7fc40f063700 1 --2- 172.21.15.16:0/1174937239 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc4080cf880 0x7fc4080cd270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:10.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.379+0000 7fc40f063700 1 --2- 172.21.15.16:0/1174937239 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4080b5830 0x7fc4080c99e0 unknown :-1 s=CLOSED pgs=2675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:10.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.379+0000 7fc40f063700 1 --2- 172.21.15.16:0/1174937239 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc4080cf4a0 0x7fc4080c7510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:10.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.379+0000 7fc40f063700 1 -- 172.21.15.16:0/1174937239 >> 172.21.15.16:0/1174937239 conn(0x7fc4080b1fd0 msgr2=0x7fc40815cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:10.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.379+0000 7fc40f063700 1 -- 172.21.15.16:0/1174937239 shutdown_connections 2023-10-16T13:47:10.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.379+0000 7fc40f063700 1 -- 172.21.15.16:0/1174937239 wait complete. 2023-10-16T13:47:10.384 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 121 2023-10-16T13:47:10.394 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:47:10.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.516+0000 7f221a773700 1 Processor -- start 2023-10-16T13:47:10.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.516+0000 7f221a773700 1 -- start start 2023-10-16T13:47:10.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.517+0000 7f221a773700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f221415b280 0x7f22141613e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:10.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.517+0000 7f221a773700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f22140cc8a0 0x7f221415e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:10.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.517+0000 7f221a773700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2214161c30 con 0x7f221406a7e0 2023-10-16T13:47:10.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.517+0000 7f221a773700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2214161f10 con 0x7f22140cc8a0 2023-10-16T13:47:10.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.517+0000 7f221a773700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22141621f0 con 0x7f221415b280 2023-10-16T13:47:10.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.517+0000 7f22137fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f22140cc8a0 0x7f221415e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:10.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.517+0000 7f22137fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f22140cc8a0 0x7f221415e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55008/0 (socket says 172.21.15.16:55008) 2023-10-16T13:47:10.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.517+0000 7f22137fe700 1 -- 172.21.15.16:0/2024759967 learned_addr learned my addr 172.21.15.16:0/2024759967 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:10.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.518+0000 7f2213fff700 1 --2- 172.21.15.16:0/2024759967 >> v2:172.21.15.49:3300/0 conn(0x7f221415b280 0x7f22141613e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:10.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.518+0000 7f22137fe700 1 -- 172.21.15.16:0/2024759967 >> v2:172.21.15.49:3300/0 conn(0x7f221415b280 msgr2=0x7f22141613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:10.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.518+0000 7f22137fe700 1 --2- 172.21.15.16:0/2024759967 >> v2:172.21.15.49:3300/0 conn(0x7f221415b280 0x7f22141613e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:10.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.518+0000 7f22137fe700 1 -- 172.21.15.16:0/2024759967 >> v1:172.21.15.49:6789/0 conn(0x7f221406a7e0 legacy=0x7f22140d02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:10.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.518+0000 7f22137fe700 1 -- 172.21.15.16:0/2024759967 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f221415df00 con 0x7f22140cc8a0 2023-10-16T13:47:10.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.518+0000 7f22137fe700 1 --2- 172.21.15.16:0/2024759967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22140cc8a0 0x7f221415e700 secure :-1 s=READY pgs=2676 cs=0 l=1 rev1=1 crypto rx=0x7f221406b8a0 tx=0x7f2204009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73323403fd23989 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:10.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.518+0000 7f2212ffd700 1 -- 172.21.15.16:0/2024759967 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2204004230 con 0x7f22140cc8a0 2023-10-16T13:47:10.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.518+0000 7f2212ffd700 1 -- 172.21.15.16:0/2024759967 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2204004390 con 0x7f22140cc8a0 2023-10-16T13:47:10.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.518+0000 7f2212ffd700 1 -- 172.21.15.16:0/2024759967 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22040036a0 con 0x7f22140cc8a0 2023-10-16T13:47:10.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.519+0000 7f221a773700 1 -- 172.21.15.16:0/2024759967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22140cc8a0 msgr2=0x7f221415e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:10.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.519+0000 7f221a773700 1 --2- 172.21.15.16:0/2024759967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22140cc8a0 0x7f221415e700 secure :-1 s=READY pgs=2676 cs=0 l=1 rev1=1 crypto rx=0x7f221406b8a0 tx=0x7f2204009bd0 comp rx=0 tx=0).stop 2023-10-16T13:47:10.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.519+0000 7f221a773700 1 -- 172.21.15.16:0/2024759967 shutdown_connections 2023-10-16T13:47:10.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.519+0000 7f221a773700 1 --2- 172.21.15.16:0/2024759967 >> v2:172.21.15.49:3300/0 conn(0x7f221415b280 0x7f22141613e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:10.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.519+0000 7f221a773700 1 --2- 172.21.15.16:0/2024759967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22140cc8a0 0x7f221415e700 unknown :-1 s=CLOSED pgs=2676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:10.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.519+0000 7f221a773700 1 -- 172.21.15.16:0/2024759967 >> 172.21.15.16:0/2024759967 conn(0x7f22140b1fd0 msgr2=0x7f22140b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:10.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.519+0000 7f221a773700 1 -- 172.21.15.16:0/2024759967 shutdown_connections 2023-10-16T13:47:10.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.519+0000 7f221a773700 1 -- 172.21.15.16:0/2024759967 wait complete. 2023-10-16T13:47:10.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.520+0000 7f221a773700 1 Processor -- start 2023-10-16T13:47:10.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.520+0000 7f221a773700 1 -- start start 2023-10-16T13:47:10.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.521+0000 7f221a773700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f221406a7e0 0x7f22140c39f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:10.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.521+0000 7f221a773700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f22140cc8a0 0x7f2214159b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:10.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.521+0000 7f221a773700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f221415b280 0x7f22140c3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:10.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.521+0000 7f221a773700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22140c44b0 con 0x7f221406a7e0 2023-10-16T13:47:10.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.521+0000 7f221a773700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22141f64e0 con 0x7f221415b280 2023-10-16T13:47:10.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.521+0000 7f221a773700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22141f67c0 con 0x7f22140cc8a0 2023-10-16T13:47:10.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.521+0000 7f2213fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f221406a7e0 0x7f22140c39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:10.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.521+0000 7f2213fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f221406a7e0 0x7f22140c39f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55012/0 (socket says 172.21.15.16:55012) 2023-10-16T13:47:10.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.521+0000 7f2213fff700 1 -- 172.21.15.16:0/1814068550 learned_addr learned my addr 172.21.15.16:0/1814068550 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:10.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.521+0000 7f2218d10700 1 --2- 172.21.15.16:0/1814068550 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f221415b280 0x7f22140c3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:10.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.521+0000 7f22137fe700 1 --2- 172.21.15.16:0/1814068550 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f22140cc8a0 0x7f2214159b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:10.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.521+0000 7f2213fff700 1 -- 172.21.15.16:0/1814068550 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f22140cc8a0 msgr2=0x7f2214159b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:10.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.521+0000 7f2213fff700 1 --2- 172.21.15.16:0/1814068550 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f22140cc8a0 0x7f2214159b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:10.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.521+0000 7f2213fff700 1 -- 172.21.15.16:0/1814068550 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f221415b280 msgr2=0x7f22140c3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:10.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.521+0000 7f2213fff700 1 --2- 172.21.15.16:0/1814068550 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f221415b280 0x7f22140c3f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:10.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.521+0000 7f2213fff700 1 -- 172.21.15.16:0/1814068550 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2204009550 con 0x7f221406a7e0 2023-10-16T13:47:10.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.521+0000 7f22137fe700 1 --2- 172.21.15.16:0/1814068550 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f22140cc8a0 0x7f2214159b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:47:10.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.521+0000 7f2213fff700 1 --2- 172.21.15.16:0/1814068550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f221406a7e0 0x7f22140c39f0 secure :-1 s=READY pgs=2677 cs=0 l=1 rev1=1 crypto rx=0x7f21fc00c7f0 tx=0x7f21fc00cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:10.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.521+0000 7f2210ff9700 1 -- 172.21.15.16:0/1814068550 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21fc011800 con 0x7f221406a7e0 2023-10-16T13:47:10.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.522+0000 7f2210ff9700 1 -- 172.21.15.16:0/1814068550 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21fc011960 con 0x7f221406a7e0 2023-10-16T13:47:10.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.522+0000 7f221a773700 1 -- 172.21.15.16:0/1814068550 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22141f6aa0 con 0x7f221406a7e0 2023-10-16T13:47:10.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.522+0000 7f2210ff9700 1 -- 172.21.15.16:0/1814068550 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21fc011cf0 con 0x7f221406a7e0 2023-10-16T13:47:10.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.522+0000 7f221a773700 1 -- 172.21.15.16:0/1814068550 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22141f7060 con 0x7f221406a7e0 2023-10-16T13:47:10.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.523+0000 7f221a773700 1 -- 172.21.15.16:0/1814068550 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22140c83e0 con 0x7f221406a7e0 2023-10-16T13:47:10.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.523+0000 7f2210ff9700 1 -- 172.21.15.16:0/1814068550 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 148) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f21fc018360 con 0x7f221406a7e0 2023-10-16T13:47:10.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.523+0000 7f2210ff9700 1 --2- 172.21.15.16:0/1814068550 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f21f406b140 0x7f21f406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:10.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.524+0000 7f2210ff9700 1 -- 172.21.15.16:0/1814068550 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 7072+0+0 (secure 0 0 0) 0x7f21fc08a080 con 0x7f221406a7e0 2023-10-16T13:47:10.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.524+0000 7f22137fe700 1 --2- 172.21.15.16:0/1814068550 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f21f406b140 0x7f21f406d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:10.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.524+0000 7f22137fe700 1 --2- 172.21.15.16:0/1814068550 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f21f406b140 0x7f21f406d600 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f221406b8a0 tx=0x7f2204008030 comp rx=0 tx=0).ready entity=mgr.11886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:10.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.528+0000 7f2210ff9700 1 -- 172.21.15.16:0/1814068550 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f21fc054780 con 0x7f221406a7e0 2023-10-16T13:47:10.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.735+0000 7f221a773700 1 -- 172.21.15.16:0/1814068550 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f221404ebb0 con 0x7f221406a7e0 2023-10-16T13:47:10.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.736+0000 7f2210ff9700 1 -- 172.21.15.16:0/1814068550 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v234) v1 ==== 74+0+18143 (secure 0 0 0) 0x7f21fc057da0 con 0x7f221406a7e0 2023-10-16T13:47:10.739 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:47:10.739 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":234,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:47:07.345418+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T13:46:35.800444+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"228","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"backup_fs_data","create_time":"2023-10-16T13:46:37.035323+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"229","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.49:6835/2905760119":"2023-10-17T13:47:04.337914+0000","172.21.15.16:0/1827947757":"2023-10-17T13:47:01.801871+0000","172.21.15.16:0/1694897016":"2023-10-17T13:47:01.801871+0000","172.21.15.16:0/3760456275":"2023-10-17T13:47:01.801871+0000","172.21.15.16:0/3651530055":"2023-10-17T13:47:01.801871+0000","172.21.15.16:0/850397235":"2023-10-17T13:47:01.801871+0000","172.21.15.16:6833/1506142504":"2023-10-17T13:47:01.801871+0000","172.21.15.16:0/4239949371":"2023-10-17T13:47:01.801871+0000","172.21.15.16:0/2321929682":"2023-10-17T13:46:51.165018+0000","172.21.15.16:0/3155275276":"2023-10-17T13:47:01.801871+0000","172.21.15.16:0/3429506235":"2023-10-17T13:46:51.165018+0000","172.21.15.49:6833/2905760119":"2023-10-17T13:47:04.337914+0000","172.21.15.16:0/4090083700":"2023-10-17T13:47:01.801871+0000","172.21.15.16:0/3843126404":"2023-10-17T13:46:51.165018+0000","172.21.15.16:6832/1506142504":"2023-10-17T13:47:01.801871+0000","172.21.15.16:0/1439318665":"2023-10-17T13:46:51.165018+0000","172.21.15.16:0/4204021371":"2023-10-17T13:46:51.165018+0000","172.21.15.16:6833/3171422553":"2023-10-17T13:46:51.165018+0000","172.21.15.16:6832/3171422553":"2023-10-17T13:46:51.165018+0000","172.21.15.16:0/1691810194":"2023-10-17T13:47:01.801871+0000","172.21.15.16:0/1632287797":"2023-10-17T13:46:51.165018+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:47:10.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.738+0000 7f221a773700 1 -- 172.21.15.16:0/1814068550 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f21f406b140 msgr2=0x7f21f406d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:10.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.738+0000 7f221a773700 1 --2- 172.21.15.16:0/1814068550 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f21f406b140 0x7f21f406d600 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f221406b8a0 tx=0x7f2204008030 comp rx=0 tx=0).stop 2023-10-16T13:47:10.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.738+0000 7f221a773700 1 -- 172.21.15.16:0/1814068550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f221406a7e0 msgr2=0x7f22140c39f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:10.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.738+0000 7f221a773700 1 --2- 172.21.15.16:0/1814068550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f221406a7e0 0x7f22140c39f0 secure :-1 s=READY pgs=2677 cs=0 l=1 rev1=1 crypto rx=0x7f21fc00c7f0 tx=0x7f21fc00cbb0 comp rx=0 tx=0).stop 2023-10-16T13:47:10.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.738+0000 7f221a773700 1 -- 172.21.15.16:0/1814068550 shutdown_connections 2023-10-16T13:47:10.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.738+0000 7f221a773700 1 --2- 172.21.15.16:0/1814068550 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f21f406b140 0x7f21f406d600 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:10.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.738+0000 7f221a773700 1 --2- 172.21.15.16:0/1814068550 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f221415b280 0x7f22140c3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:10.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.738+0000 7f221a773700 1 --2- 172.21.15.16:0/1814068550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f221406a7e0 0x7f22140c39f0 unknown :-1 s=CLOSED pgs=2677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:10.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.738+0000 7f221a773700 1 --2- 172.21.15.16:0/1814068550 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f22140cc8a0 0x7f2214159b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:10.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.738+0000 7f221a773700 1 -- 172.21.15.16:0/1814068550 >> 172.21.15.16:0/1814068550 conn(0x7f22140b1fd0 msgr2=0x7f2214071b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:10.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.738+0000 7f221a773700 1 -- 172.21.15.16:0/1814068550 shutdown_connections 2023-10-16T13:47:10.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.738+0000 7f221a773700 1 -- 172.21.15.16:0/1814068550 wait complete. 2023-10-16T13:47:10.754 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail backup_fs 2023-10-16T13:47:10.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.877+0000 7fa25a589700 1 Processor -- start 2023-10-16T13:47:10.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.877+0000 7fa25a589700 1 -- start start 2023-10-16T13:47:10.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.878+0000 7fa25a589700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fa25415b280 0x7fa2541613e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:10.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.878+0000 7fa25a589700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa2540cc8a0 0x7fa25415e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:10.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.878+0000 7fa25a589700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa254161c30 con 0x7fa25406a7e0 2023-10-16T13:47:10.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.878+0000 7fa25a589700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa254161f10 con 0x7fa25415b280 2023-10-16T13:47:10.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.878+0000 7fa25a589700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2541621f0 con 0x7fa2540cc8a0 2023-10-16T13:47:10.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.878+0000 7fa258b26700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa25406a7e0 0x7fa2540d02b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44298/0 (socket says 172.21.15.16:44298) 2023-10-16T13:47:10.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.878+0000 7fa258b26700 1 -- 172.21.15.16:0/3097144845 learned_addr learned my addr 172.21.15.16:0/3097144845 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:47:10.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.879+0000 7fa253fff700 1 --2- 172.21.15.16:0/3097144845 >> v2:172.21.15.49:3300/0 conn(0x7fa25415b280 0x7fa2541613e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:10.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.879+0000 7fa2537fe700 1 --2- 172.21.15.16:0/3097144845 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa2540cc8a0 0x7fa25415e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:10.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.879+0000 7fa252ffd700 1 -- 172.21.15.16:0/3097144845 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 351830752 0 0) 0x7fa254161c30 con 0x7fa25406a7e0 2023-10-16T13:47:10.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.879+0000 7fa252ffd700 1 -- 172.21.15.16:0/3097144845 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa23c0035b0 con 0x7fa25406a7e0 2023-10-16T13:47:10.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.879+0000 7fa252ffd700 1 -- 172.21.15.16:0/3097144845 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2710555086 0 0) 0x7fa23c0035b0 con 0x7fa25406a7e0 2023-10-16T13:47:10.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.879+0000 7fa252ffd700 1 -- 172.21.15.16:0/3097144845 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa2540cc8a0 msgr2=0x7fa25415e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:10.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.879+0000 7fa252ffd700 1 --2- 172.21.15.16:0/3097144845 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa2540cc8a0 0x7fa25415e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:10.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.879+0000 7fa252ffd700 1 -- 172.21.15.16:0/3097144845 >> v2:172.21.15.49:3300/0 conn(0x7fa25415b280 msgr2=0x7fa2541613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:10.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.879+0000 7fa252ffd700 1 --2- 172.21.15.16:0/3097144845 >> v2:172.21.15.49:3300/0 conn(0x7fa25415b280 0x7fa2541613e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:10.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.879+0000 7fa252ffd700 1 -- 172.21.15.16:0/3097144845 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa25415df00 con 0x7fa25406a7e0 2023-10-16T13:47:10.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.879+0000 7fa252ffd700 1 -- 172.21.15.16:0/3097144845 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7fa24c0019f0 con 0x7fa25406a7e0 2023-10-16T13:47:10.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.879+0000 7fa252ffd700 1 -- 172.21.15.16:0/3097144845 >> v1:172.21.15.16:6789/0 conn(0x7fa25406a7e0 legacy=0x7fa2540d02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:10.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.879+0000 7fa252ffd700 1 --2- 172.21.15.16:0/3097144845 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa23c003a00 0x7fa23c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:10.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.879+0000 7fa252ffd700 1 --2- 172.21.15.16:0/3097144845 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa2540cc8a0 0x7fa23c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:10.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.879+0000 7fa252ffd700 1 --2- 172.21.15.16:0/3097144845 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa23c004d90 0x7fa23c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:10.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.879+0000 7fa252ffd700 1 -- 172.21.15.16:0/3097144845 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa254161f10 con 0x7fa23c003a00 2023-10-16T13:47:10.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.879+0000 7fa252ffd700 1 -- 172.21.15.16:0/3097144845 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2541621f0 con 0x7fa23c004d90 2023-10-16T13:47:10.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.879+0000 7fa252ffd700 1 -- 172.21.15.16:0/3097144845 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa23c00ba10 con 0x7fa2540cc8a0 2023-10-16T13:47:10.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.879+0000 7fa253fff700 1 --2- 172.21.15.16:0/3097144845 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa23c003a00 0x7fa23c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:10.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.879+0000 7fa2537fe700 1 --2- 172.21.15.16:0/3097144845 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa2540cc8a0 0x7fa23c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:10.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.879+0000 7fa258b26700 1 --2- 172.21.15.16:0/3097144845 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa23c004d90 0x7fa23c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:10.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.879+0000 7fa253fff700 1 -- 172.21.15.16:0/3097144845 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa2540cc8a0 msgr2=0x7fa23c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:10.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.879+0000 7fa253fff700 1 --2- 172.21.15.16:0/3097144845 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa2540cc8a0 0x7fa23c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:10.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.880+0000 7fa253fff700 1 -- 172.21.15.16:0/3097144845 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa23c004d90 msgr2=0x7fa23c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:10.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.880+0000 7fa253fff700 1 --2- 172.21.15.16:0/3097144845 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa23c004d90 0x7fa23c005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:10.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.880+0000 7fa253fff700 1 -- 172.21.15.16:0/3097144845 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa23c0035b0 con 0x7fa23c003a00 2023-10-16T13:47:10.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.880+0000 7fa253fff700 1 --2- 172.21.15.16:0/3097144845 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa23c003a00 0x7fa23c003e20 secure :-1 s=READY pgs=2679 cs=0 l=1 rev1=1 crypto rx=0x7fa244009b90 tx=0x7fa23c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d37b408a33aaf0f4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:10.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.880+0000 7fa252ffd700 1 -- 172.21.15.16:0/3097144845 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa24400e6c0 con 0x7fa23c003a00 2023-10-16T13:47:10.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.880+0000 7fa252ffd700 1 -- 172.21.15.16:0/3097144845 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa24400e820 con 0x7fa23c003a00 2023-10-16T13:47:10.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.880+0000 7fa252ffd700 1 -- 172.21.15.16:0/3097144845 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa24400eb90 con 0x7fa23c003a00 2023-10-16T13:47:10.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.880+0000 7fa25a589700 1 -- 172.21.15.16:0/3097144845 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa23c003a00 msgr2=0x7fa23c003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:10.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.880+0000 7fa25a589700 1 --2- 172.21.15.16:0/3097144845 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa23c003a00 0x7fa23c003e20 secure :-1 s=READY pgs=2679 cs=0 l=1 rev1=1 crypto rx=0x7fa244009b90 tx=0x7fa23c005880 comp rx=0 tx=0).stop 2023-10-16T13:47:10.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.880+0000 7fa258b26700 1 -- 172.21.15.16:0/3097144845 reap_dead start 2023-10-16T13:47:10.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.880+0000 7fa25a589700 1 -- 172.21.15.16:0/3097144845 shutdown_connections 2023-10-16T13:47:10.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.880+0000 7fa25a589700 1 -- 172.21.15.16:0/3097144845 >> 172.21.15.16:0/3097144845 conn(0x7fa2540b1fd0 msgr2=0x7fa2540b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:10.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.880+0000 7fa25a589700 1 -- 172.21.15.16:0/3097144845 shutdown_connections 2023-10-16T13:47:10.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.881+0000 7fa25a589700 1 -- 172.21.15.16:0/3097144845 wait complete. 2023-10-16T13:47:10.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.882+0000 7fa25a589700 1 Processor -- start 2023-10-16T13:47:10.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.882+0000 7fa25a589700 1 -- start start 2023-10-16T13:47:10.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.883+0000 7fa25a589700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa23c003a00 0x7fa254148410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:10.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.883+0000 7fa25a589700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa254159f20 0x7fa25415ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:10.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.883+0000 7fa25a589700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa2540c9d90 0x7fa2540cb0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:10.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.883+0000 7fa25a589700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2540c8a50 con 0x7fa254159f20 2023-10-16T13:47:10.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.883+0000 7fa25a589700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2540c8cf0 con 0x7fa23c003a00 2023-10-16T13:47:10.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.883+0000 7fa25a589700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2540c8fd0 con 0x7fa2540c9d90 2023-10-16T13:47:10.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.883+0000 7fa2537fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa254159f20 0x7fa25415ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:10.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.883+0000 7fa2537fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa254159f20 0x7fa25415ffb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55034/0 (socket says 172.21.15.16:55034) 2023-10-16T13:47:10.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.883+0000 7fa2537fe700 1 -- 172.21.15.16:0/2103891024 learned_addr learned my addr 172.21.15.16:0/2103891024 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:10.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.883+0000 7fa253fff700 1 --2- 172.21.15.16:0/2103891024 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa23c003a00 0x7fa254148410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:10.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.883+0000 7fa258b26700 1 --2- 172.21.15.16:0/2103891024 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa2540c9d90 0x7fa2540cb0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:10.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.883+0000 7fa2537fe700 1 -- 172.21.15.16:0/2103891024 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa2540c9d90 msgr2=0x7fa2540cb0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:10.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.883+0000 7fa2537fe700 1 --2- 172.21.15.16:0/2103891024 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa2540c9d90 0x7fa2540cb0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:10.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.883+0000 7fa2537fe700 1 -- 172.21.15.16:0/2103891024 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa23c003a00 msgr2=0x7fa254148410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:10.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.883+0000 7fa2537fe700 1 --2- 172.21.15.16:0/2103891024 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa23c003a00 0x7fa254148410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:10.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.883+0000 7fa2537fe700 1 -- 172.21.15.16:0/2103891024 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2540c92b0 con 0x7fa254159f20 2023-10-16T13:47:10.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.883+0000 7fa253fff700 1 --2- 172.21.15.16:0/2103891024 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa23c003a00 0x7fa254148410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:47:10.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.883+0000 7fa2537fe700 1 --2- 172.21.15.16:0/2103891024 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa254159f20 0x7fa25415ffb0 secure :-1 s=READY pgs=2680 cs=0 l=1 rev1=1 crypto rx=0x7fa248011aa0 tx=0x7fa248011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:10.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.883+0000 7fa250ff9700 1 -- 172.21.15.16:0/2103891024 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa24800f980 con 0x7fa254159f20 2023-10-16T13:47:10.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.883+0000 7fa250ff9700 1 -- 172.21.15.16:0/2103891024 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa24800fae0 con 0x7fa254159f20 2023-10-16T13:47:10.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.883+0000 7fa250ff9700 1 -- 172.21.15.16:0/2103891024 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa24800fe90 con 0x7fa254159f20 2023-10-16T13:47:10.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.883+0000 7fa25a589700 1 -- 172.21.15.16:0/2103891024 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2540c9590 con 0x7fa254159f20 2023-10-16T13:47:10.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.884+0000 7fa25a589700 1 -- 172.21.15.16:0/2103891024 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa254148950 con 0x7fa254159f20 2023-10-16T13:47:10.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.885+0000 7fa25a589700 1 -- 172.21.15.16:0/2103891024 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa25404ebb0 con 0x7fa254159f20 2023-10-16T13:47:10.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.885+0000 7fa250ff9700 1 -- 172.21.15.16:0/2103891024 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 148) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fa248014440 con 0x7fa254159f20 2023-10-16T13:47:10.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.885+0000 7fa250ff9700 1 --2- 172.21.15.16:0/2103891024 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fa24006b140 0x7fa24006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:10.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.885+0000 7fa250ff9700 1 -- 172.21.15.16:0/2103891024 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 7072+0+0 (secure 0 0 0) 0x7fa24808a750 con 0x7fa254159f20 2023-10-16T13:47:10.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.885+0000 7fa253fff700 1 --2- 172.21.15.16:0/2103891024 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fa24006b140 0x7fa24006d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:10.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.886+0000 7fa253fff700 1 --2- 172.21.15.16:0/2103891024 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fa24006b140 0x7fa24006d600 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa2440090b0 tx=0x7fa244008030 comp rx=0 tx=0).ready entity=mgr.11886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:10.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:10.890+0000 7fa250ff9700 1 -- 172.21.15.16:0/2103891024 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa248055010 con 0x7fa254159f20 2023-10-16T13:47:11.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.139+0000 7fa25a589700 1 -- 172.21.15.16:0/2103891024 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "backup_fs"} v 0) v1 -- 0x7fa2540b2600 con 0x7fa254159f20 2023-10-16T13:47:11.142 INFO:tasks.ceph.mon.a.smithi016.stderr:2023-10-16T13:47:11.140+0000 7f9fc8535700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-16T13:47:11.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.141+0000 7fa250ff9700 1 -- 172.21.15.16:0/2103891024 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "backup_fs"}]=0 backup_fs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v122) v1 ==== 167+0+0 (secure 0 0 0) 0x7fa248058630 con 0x7fa254159f20 2023-10-16T13:47:11.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.144+0000 7fa25a589700 1 -- 172.21.15.16:0/2103891024 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fa24006b140 msgr2=0x7fa24006d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:11.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.144+0000 7fa25a589700 1 --2- 172.21.15.16:0/2103891024 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fa24006b140 0x7fa24006d600 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa2440090b0 tx=0x7fa244008030 comp rx=0 tx=0).stop 2023-10-16T13:47:11.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.144+0000 7fa25a589700 1 -- 172.21.15.16:0/2103891024 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa254159f20 msgr2=0x7fa25415ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:11.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.144+0000 7fa25a589700 1 --2- 172.21.15.16:0/2103891024 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa254159f20 0x7fa25415ffb0 secure :-1 s=READY pgs=2680 cs=0 l=1 rev1=1 crypto rx=0x7fa248011aa0 tx=0x7fa248011e60 comp rx=0 tx=0).stop 2023-10-16T13:47:11.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.144+0000 7fa25a589700 1 -- 172.21.15.16:0/2103891024 shutdown_connections 2023-10-16T13:47:11.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.144+0000 7fa25a589700 1 --2- 172.21.15.16:0/2103891024 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fa24006b140 0x7fa24006d600 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:11.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.144+0000 7fa25a589700 1 --2- 172.21.15.16:0/2103891024 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa2540c9d90 0x7fa2540cb0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:11.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.144+0000 7fa25a589700 1 --2- 172.21.15.16:0/2103891024 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa23c003a00 0x7fa254148410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:11.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.144+0000 7fa25a589700 1 --2- 172.21.15.16:0/2103891024 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa254159f20 0x7fa25415ffb0 unknown :-1 s=CLOSED pgs=2680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:11.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.144+0000 7fa25a589700 1 -- 172.21.15.16:0/2103891024 >> 172.21.15.16:0/2103891024 conn(0x7fa2540b1fd0 msgr2=0x7fa254072680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:11.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.144+0000 7fa25a589700 1 -- 172.21.15.16:0/2103891024 shutdown_connections 2023-10-16T13:47:11.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.144+0000 7fa25a589700 1 -- 172.21.15.16:0/2103891024 wait complete. 2023-10-16T13:47:11.149 INFO:teuthology.orchestra.run.smithi016.stderr:backup_fs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-16T13:47:11.150 INFO:tasks.ceph.mds.a.smithi016.stderr: -1805> 2023-10-16T13:46:40.547+0000 7f9a762b4700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T13:47:11.160 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm backup_fs --yes-i-really-mean-it 2023-10-16T13:47:11.175 INFO:tasks.ceph.mds.a.smithi016.stdout:starting mds.a at 2023-10-16T13:47:11.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.285+0000 7f8a12648700 1 Processor -- start 2023-10-16T13:47:11.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.285+0000 7f8a12648700 1 -- start start 2023-10-16T13:47:11.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.286+0000 7f8a12648700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8a0c06e570 0x7f8a0c15e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:11.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.286+0000 7f8a12648700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a0c15b6a0 con 0x7f8a0c15b2c0 2023-10-16T13:47:11.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.286+0000 7f8a12648700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a0c161b20 con 0x7f8a0c06a7e0 2023-10-16T13:47:11.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.286+0000 7f8a12648700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a0c161e00 con 0x7f8a0c06e570 2023-10-16T13:47:11.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.286+0000 7f8a0bfff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f8a0c15b2c0 0x7f8a0c161340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44326/0 (socket says 172.21.15.16:44326) 2023-10-16T13:47:11.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.286+0000 7f8a0bfff700 1 -- 172.21.15.16:0/914012160 learned_addr learned my addr 172.21.15.16:0/914012160 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:47:11.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.287+0000 7f8a0b7fe700 1 --2- 172.21.15.16:0/914012160 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8a0c06e570 0x7f8a0c15e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:11.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.287+0000 7f8a0affd700 1 -- 172.21.15.16:0/914012160 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 333782247 0 0) 0x7f8a0c15b6a0 con 0x7f8a0c15b2c0 2023-10-16T13:47:11.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.287+0000 7f8a0affd700 1 -- 172.21.15.16:0/914012160 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89f80035b0 con 0x7f8a0c15b2c0 2023-10-16T13:47:11.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.287+0000 7f8a0affd700 1 -- 172.21.15.16:0/914012160 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2916369181 0 0) 0x7f8a0c161b20 con 0x7f8a0c06a7e0 2023-10-16T13:47:11.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.287+0000 7f8a0affd700 1 -- 172.21.15.16:0/914012160 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a0c15b6a0 con 0x7f8a0c06a7e0 2023-10-16T13:47:11.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.287+0000 7f8a0affd700 1 -- 172.21.15.16:0/914012160 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2058706374 0 0) 0x7f89f80035b0 con 0x7f8a0c15b2c0 2023-10-16T13:47:11.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.287+0000 7f8a0b7fe700 1 -- 172.21.15.16:0/914012160 >> v1:172.21.15.49:6789/0 conn(0x7f8a0c06a7e0 legacy=0x7f8a0c071a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:11.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.287+0000 7f8a0b7fe700 1 -- 172.21.15.16:0/914012160 >> v1:172.21.15.16:6789/0 conn(0x7f8a0c15b2c0 legacy=0x7f8a0c161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:11.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.287+0000 7f8a0b7fe700 1 -- 172.21.15.16:0/914012160 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a0c1620e0 con 0x7f8a0c06e570 2023-10-16T13:47:11.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.287+0000 7f8a0b7fe700 1 --2- 172.21.15.16:0/914012160 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8a0c06e570 0x7f8a0c15e5a0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f89f4009870 tx=0x7f89f4009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b9cf5260cd31606 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:11.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.287+0000 7f8a0affd700 1 -- 172.21.15.16:0/914012160 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89fc001670 con 0x7f8a0c06e570 2023-10-16T13:47:11.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.287+0000 7f8a0affd700 1 -- 172.21.15.16:0/914012160 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89f4004030 con 0x7f8a0c06e570 2023-10-16T13:47:11.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.287+0000 7f8a0affd700 1 -- 172.21.15.16:0/914012160 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89f40043a0 con 0x7f8a0c06e570 2023-10-16T13:47:11.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.288+0000 7f8a12648700 1 -- 172.21.15.16:0/914012160 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8a0c06e570 msgr2=0x7f8a0c15e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:11.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.288+0000 7f8a12648700 1 --2- 172.21.15.16:0/914012160 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8a0c06e570 0x7f8a0c15e5a0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f89f4009870 tx=0x7f89f4009b80 comp rx=0 tx=0).stop 2023-10-16T13:47:11.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.288+0000 7f8a12648700 1 -- 172.21.15.16:0/914012160 shutdown_connections 2023-10-16T13:47:11.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.288+0000 7f8a12648700 1 --2- 172.21.15.16:0/914012160 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8a0c06e570 0x7f8a0c15e5a0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:11.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.288+0000 7f8a12648700 1 -- 172.21.15.16:0/914012160 >> 172.21.15.16:0/914012160 conn(0x7f8a0c0b1f40 msgr2=0x7f8a0c0b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:11.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.288+0000 7f8a12648700 1 -- 172.21.15.16:0/914012160 shutdown_connections 2023-10-16T13:47:11.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.288+0000 7f8a12648700 1 -- 172.21.15.16:0/914012160 wait complete. 2023-10-16T13:47:11.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.289+0000 7f8a12648700 1 Processor -- start 2023-10-16T13:47:11.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.290+0000 7f8a12648700 1 -- start start 2023-10-16T13:47:11.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.290+0000 7f8a12648700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8a0c06a7e0 0x7f8a0c1f2a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:11.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.290+0000 7f8a12648700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a0c06e570 0x7f8a0c0c3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:11.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.290+0000 7f8a12648700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8a0c15b2c0 0x7f8a0c0b4500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:11.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.290+0000 7f8a12648700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a0c0c6570 con 0x7f8a0c06e570 2023-10-16T13:47:11.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.290+0000 7f8a12648700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a0c0c6850 con 0x7f8a0c06a7e0 2023-10-16T13:47:11.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.290+0000 7f8a12648700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a0c0c6b30 con 0x7f8a0c15b2c0 2023-10-16T13:47:11.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.290+0000 7f8a0b7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a0c06e570 0x7f8a0c0c3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:11.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.290+0000 7f8a0b7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a0c06e570 0x7f8a0c0c3ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55060/0 (socket says 172.21.15.16:55060) 2023-10-16T13:47:11.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.290+0000 7f8a0b7fe700 1 -- 172.21.15.16:0/671944269 learned_addr learned my addr 172.21.15.16:0/671944269 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:11.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.291+0000 7f8a0bfff700 1 --2- 172.21.15.16:0/671944269 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8a0c06a7e0 0x7f8a0c1f2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:11.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.291+0000 7f8a0b7fe700 1 -- 172.21.15.16:0/671944269 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8a0c15b2c0 msgr2=0x7f8a0c0b4500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:11.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.291+0000 7f8a10be5700 1 --2- 172.21.15.16:0/671944269 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8a0c15b2c0 0x7f8a0c0b4500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:11.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.291+0000 7f8a0b7fe700 1 --2- 172.21.15.16:0/671944269 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8a0c15b2c0 0x7f8a0c0b4500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:11.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.291+0000 7f8a0b7fe700 1 -- 172.21.15.16:0/671944269 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8a0c06a7e0 msgr2=0x7f8a0c1f2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:11.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.291+0000 7f8a0b7fe700 1 --2- 172.21.15.16:0/671944269 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8a0c06a7e0 0x7f8a0c1f2a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:11.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.291+0000 7f8a0b7fe700 1 -- 172.21.15.16:0/671944269 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89f40094a0 con 0x7f8a0c06e570 2023-10-16T13:47:11.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.291+0000 7f8a0bfff700 1 --2- 172.21.15.16:0/671944269 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8a0c06a7e0 0x7f8a0c1f2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:47:11.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.291+0000 7f8a10be5700 1 --2- 172.21.15.16:0/671944269 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8a0c15b2c0 0x7f8a0c0b4500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:47:11.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.291+0000 7f8a0b7fe700 1 --2- 172.21.15.16:0/671944269 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a0c06e570 0x7f8a0c0c3ba0 secure :-1 s=READY pgs=2683 cs=0 l=1 rev1=1 crypto rx=0x7f89f40090b0 tx=0x7f89f4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:11.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.291+0000 7f8a08ff9700 1 -- 172.21.15.16:0/671944269 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89f4004590 con 0x7f8a0c06e570 2023-10-16T13:47:11.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.291+0000 7f8a08ff9700 1 -- 172.21.15.16:0/671944269 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89f401f070 con 0x7f8a0c06e570 2023-10-16T13:47:11.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.291+0000 7f8a12648700 1 -- 172.21.15.16:0/671944269 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a0c0c6e10 con 0x7f8a0c06e570 2023-10-16T13:47:11.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.291+0000 7f8a08ff9700 1 -- 172.21.15.16:0/671944269 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89f4011870 con 0x7f8a0c06e570 2023-10-16T13:47:11.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.291+0000 7f8a12648700 1 -- 172.21.15.16:0/671944269 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a0c0c6240 con 0x7f8a0c06e570 2023-10-16T13:47:11.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.292+0000 7f8a12648700 1 -- 172.21.15.16:0/671944269 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a0c15ea60 con 0x7f8a0c06e570 2023-10-16T13:47:11.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.292+0000 7f8a08ff9700 1 -- 172.21.15.16:0/671944269 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 148) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f89f40119d0 con 0x7f8a0c06e570 2023-10-16T13:47:11.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.293+0000 7f8a08ff9700 1 --2- 172.21.15.16:0/671944269 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f89ec06b140 0x7f89ec06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:11.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.293+0000 7f8a08ff9700 1 -- 172.21.15.16:0/671944269 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 7158+0+0 (secure 0 0 0) 0x7f89f4021070 con 0x7f8a0c06e570 2023-10-16T13:47:11.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.293+0000 7f8a0bfff700 1 --2- 172.21.15.16:0/671944269 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f89ec06b140 0x7f89ec06d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:11.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.293+0000 7f8a0bfff700 1 --2- 172.21.15.16:0/671944269 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f89ec06b140 0x7f89ec06d600 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f89fc0011e0 tx=0x7f89fc009040 comp rx=0 tx=0).ready entity=mgr.11886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:11.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.298+0000 7f8a08ff9700 1 -- 172.21.15.16:0/671944269 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f89f40560f0 con 0x7f8a0c06e570 2023-10-16T13:47:11.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:11.547+0000 7f8a12648700 1 -- 172.21.15.16:0/671944269 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "backup_fs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f8a0c04ebb0 con 0x7f8a0c06e570 2023-10-16T13:47:12.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:12.143+0000 7f8a08ff9700 1 -- 172.21.15.16:0/671944269 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "backup_fs", "yes_i_really_mean_it": true}]=0 v123) v1 ==== 107+0+0 (secure 0 0 0) 0x7f89f4059710 con 0x7f8a0c06e570 2023-10-16T13:47:12.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:12.146+0000 7f8a12648700 1 -- 172.21.15.16:0/671944269 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f89ec06b140 msgr2=0x7f89ec06d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:12.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:12.146+0000 7f8a12648700 1 --2- 172.21.15.16:0/671944269 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f89ec06b140 0x7f89ec06d600 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f89fc0011e0 tx=0x7f89fc009040 comp rx=0 tx=0).stop 2023-10-16T13:47:12.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:12.146+0000 7f8a12648700 1 -- 172.21.15.16:0/671944269 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a0c06e570 msgr2=0x7f8a0c0c3ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:12.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:12.146+0000 7f8a12648700 1 --2- 172.21.15.16:0/671944269 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a0c06e570 0x7f8a0c0c3ba0 secure :-1 s=READY pgs=2683 cs=0 l=1 rev1=1 crypto rx=0x7f89f40090b0 tx=0x7f89f4004750 comp rx=0 tx=0).stop 2023-10-16T13:47:12.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:12.146+0000 7f8a12648700 1 -- 172.21.15.16:0/671944269 shutdown_connections 2023-10-16T13:47:12.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:12.146+0000 7f8a12648700 1 --2- 172.21.15.16:0/671944269 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f89ec06b140 0x7f89ec06d600 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:12.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:12.146+0000 7f8a12648700 1 --2- 172.21.15.16:0/671944269 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8a0c15b2c0 0x7f8a0c0b4500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:12.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:12.146+0000 7f8a12648700 1 --2- 172.21.15.16:0/671944269 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8a0c06a7e0 0x7f8a0c1f2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:12.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:12.146+0000 7f8a12648700 1 --2- 172.21.15.16:0/671944269 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a0c06e570 0x7f8a0c0c3ba0 unknown :-1 s=CLOSED pgs=2683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:12.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:12.146+0000 7f8a12648700 1 -- 172.21.15.16:0/671944269 >> 172.21.15.16:0/671944269 conn(0x7f8a0c0b1f40 msgr2=0x7f8a0c0cc030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:12.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:12.146+0000 7f8a12648700 1 -- 172.21.15.16:0/671944269 shutdown_connections 2023-10-16T13:47:12.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:12.146+0000 7f8a12648700 1 -- 172.21.15.16:0/671944269 wait complete. 2023-10-16T13:47:12.164 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm backup_fs_metadata backup_fs_metadata --yes-i-really-really-mean-it 2023-10-16T13:47:12.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:12.340+0000 7faf5e8ac700 1 Processor -- start 2023-10-16T13:47:12.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:12.341+0000 7faf5e8ac700 1 -- start start 2023-10-16T13:47:12.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:12.341+0000 7faf5e8ac700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faf58157b30 0x7faf580cefc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:12.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:12.341+0000 7faf5e8ac700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf58142fc0 con 0x7faf58161d20 2023-10-16T13:47:12.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:12.341+0000 7faf5e8ac700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf5806d900 con 0x7faf58157f10 2023-10-16T13:47:12.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:12.341+0000 7faf5e8ac700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf58143210 con 0x7faf58157b30 2023-10-16T13:47:12.346 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:12.341+0000 7faf5ce49700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7faf58161d20 0x7faf58167700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44346/0 (socket says 172.21.15.16:44346) 2023-10-16T13:47:12.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:12.341+0000 7faf5ce49700 1 -- 172.21.15.16:0/1333070602 learned_addr learned my addr 172.21.15.16:0/1333070602 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:47:12.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:12.342+0000 7faf57fff700 1 --2- 172.21.15.16:0/1333070602 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faf58157b30 0x7faf580cefc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:12.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:12.342+0000 7faf56ffd700 1 -- 172.21.15.16:0/1333070602 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3502441100 0 0) 0x7faf58142fc0 con 0x7faf58161d20 2023-10-16T13:47:12.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:12.342+0000 7faf56ffd700 1 -- 172.21.15.16:0/1333070602 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf400035b0 con 0x7faf58161d20 2023-10-16T13:47:12.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:12.342+0000 7faf56ffd700 1 -- 172.21.15.16:0/1333070602 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 229490986 0 0) 0x7faf5806d900 con 0x7faf58157f10 2023-10-16T13:47:12.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:12.342+0000 7faf56ffd700 1 -- 172.21.15.16:0/1333070602 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf58142fc0 con 0x7faf58157f10 2023-10-16T13:47:12.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:12.342+0000 7faf57fff700 1 -- 172.21.15.16:0/1333070602 >> v1:172.21.15.49:6789/0 conn(0x7faf58157f10 legacy=0x7faf58161610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:12.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:12.342+0000 7faf57fff700 1 -- 172.21.15.16:0/1333070602 >> v1:172.21.15.16:6789/0 conn(0x7faf58161d20 legacy=0x7faf58167700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:12.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:12.342+0000 7faf56ffd700 1 -- 172.21.15.16:0/1333070602 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 514550648 0 0) 0x7faf400035b0 con 0x7faf58161d20 2023-10-16T13:47:12.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:12.342+0000 7faf57fff700 1 -- 172.21.15.16:0/1333070602 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf581434f0 con 0x7faf58157b30 2023-10-16T13:47:12.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:12.342+0000 7faf57fff700 1 --2- 172.21.15.16:0/1333070602 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faf58157b30 0x7faf580cefc0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7faf48009870 tx=0x7faf48009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2face288bfe60928 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:12.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:12.342+0000 7faf56ffd700 1 -- 172.21.15.16:0/1333070602 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf500019f0 con 0x7faf58157b30 2023-10-16T13:47:12.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:12.342+0000 7faf56ffd700 1 -- 172.21.15.16:0/1333070602 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf48004030 con 0x7faf58157b30 2023-10-16T13:47:12.349 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:12.343+0000 7faf56ffd700 1 -- 172.21.15.16:0/1333070602 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf480043c0 con 0x7faf58157b30 2023-10-16T13:47:12.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:12.343+0000 7faf5e8ac700 1 -- 172.21.15.16:0/1333070602 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faf58157b30 msgr2=0x7faf580cefc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:12.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:12.343+0000 7faf5e8ac700 1 --2- 172.21.15.16:0/1333070602 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faf58157b30 0x7faf580cefc0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7faf48009870 tx=0x7faf48009b80 comp rx=0 tx=0).stop 2023-10-16T13:47:12.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:12.343+0000 7faf5e8ac700 1 -- 172.21.15.16:0/1333070602 shutdown_connections 2023-10-16T13:47:12.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:12.343+0000 7faf5e8ac700 1 --2- 172.21.15.16:0/1333070602 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faf58157b30 0x7faf580cefc0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:12.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:12.343+0000 7faf5e8ac700 1 -- 172.21.15.16:0/1333070602 >> 172.21.15.16:0/1333070602 conn(0x7faf5806a7e0 msgr2=0x7faf5806abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:12.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:12.343+0000 7faf5e8ac700 1 -- 172.21.15.16:0/1333070602 shutdown_connections 2023-10-16T13:47:12.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:12.343+0000 7faf5e8ac700 1 -- 172.21.15.16:0/1333070602 wait complete. 2023-10-16T13:47:12.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:12.344+0000 7faf5e8ac700 1 Processor -- start 2023-10-16T13:47:12.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:12.345+0000 7faf5e8ac700 1 -- start start 2023-10-16T13:47:12.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:12.345+0000 7faf5e8ac700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faf58157b30 0x7faf580c6f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:12.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:12.345+0000 7faf5e8ac700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faf58157f10 0x7faf580b49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:12.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:12.345+0000 7faf5e8ac700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf58161d20 0x7faf581f6c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:12.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:12.345+0000 7faf5e8ac700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf58158510 con 0x7faf58161d20 2023-10-16T13:47:12.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:12.345+0000 7faf5e8ac700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf580b4f30 con 0x7faf58157b30 2023-10-16T13:47:12.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:12.345+0000 7faf5e8ac700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf581f71d0 con 0x7faf58157f10 2023-10-16T13:47:12.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:12.345+0000 7faf5ce49700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf58161d20 0x7faf581f6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:12.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:12.346+0000 7faf5ce49700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf58161d20 0x7faf581f6c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55078/0 (socket says 172.21.15.16:55078) 2023-10-16T13:47:12.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:12.346+0000 7faf5ce49700 1 -- 172.21.15.16:0/654157099 learned_addr learned my addr 172.21.15.16:0/654157099 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:12.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:12.346+0000 7faf57fff700 1 --2- 172.21.15.16:0/654157099 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faf58157b30 0x7faf580c6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:12.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:12.346+0000 7faf577fe700 1 --2- 172.21.15.16:0/654157099 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faf58157f10 0x7faf580b49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:12.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:12.346+0000 7faf5ce49700 1 -- 172.21.15.16:0/654157099 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faf58157f10 msgr2=0x7faf580b49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:12.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:12.346+0000 7faf5ce49700 1 --2- 172.21.15.16:0/654157099 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faf58157f10 0x7faf580b49f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:12.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:12.346+0000 7faf5ce49700 1 -- 172.21.15.16:0/654157099 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faf58157b30 msgr2=0x7faf580c6f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:12.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:12.346+0000 7faf5ce49700 1 --2- 172.21.15.16:0/654157099 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faf58157b30 0x7faf580c6f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:12.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:12.346+0000 7faf5ce49700 1 -- 172.21.15.16:0/654157099 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf480094a0 con 0x7faf58161d20 2023-10-16T13:47:12.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:12.346+0000 7faf5ce49700 1 --2- 172.21.15.16:0/654157099 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf58161d20 0x7faf581f6c90 secure :-1 s=READY pgs=2685 cs=0 l=1 rev1=1 crypto rx=0x7faf5000cc60 tx=0x7faf5000cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:12.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:12.346+0000 7faf54ff9700 1 -- 172.21.15.16:0/654157099 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf5000aa20 con 0x7faf58161d20 2023-10-16T13:47:12.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:12.346+0000 7faf54ff9700 1 -- 172.21.15.16:0/654157099 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf5000aba0 con 0x7faf58161d20 2023-10-16T13:47:12.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:12.346+0000 7faf5e8ac700 1 -- 172.21.15.16:0/654157099 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf580b3fa0 con 0x7faf58161d20 2023-10-16T13:47:12.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:12.346+0000 7faf54ff9700 1 -- 172.21.15.16:0/654157099 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf500070d0 con 0x7faf58161d20 2023-10-16T13:47:12.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:12.346+0000 7faf5e8ac700 1 -- 172.21.15.16:0/654157099 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf580b4560 con 0x7faf58161d20 2023-10-16T13:47:12.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:12.347+0000 7faf5e8ac700 1 -- 172.21.15.16:0/654157099 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf580b46f0 con 0x7faf58161d20 2023-10-16T13:47:12.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:12.348+0000 7faf54ff9700 1 -- 172.21.15.16:0/654157099 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 148) v1 ==== 89433+0+0 (secure 0 0 0) 0x7faf50007230 con 0x7faf58161d20 2023-10-16T13:47:12.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:12.348+0000 7faf54ff9700 1 --2- 172.21.15.16:0/654157099 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7faf4406b170 0x7faf4406d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:12.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:12.348+0000 7faf54ff9700 1 -- 172.21.15.16:0/654157099 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 7158+0+0 (secure 0 0 0) 0x7faf50089cc0 con 0x7faf58161d20 2023-10-16T13:47:12.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:12.348+0000 7faf57fff700 1 --2- 172.21.15.16:0/654157099 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7faf4406b170 0x7faf4406d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:12.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:12.349+0000 7faf57fff700 1 --2- 172.21.15.16:0/654157099 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7faf4406b170 0x7faf4406d630 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7faf480097c0 tx=0x7faf48008030 comp rx=0 tx=0).ready entity=mgr.11886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:12.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:12.353+0000 7faf54ff9700 1 -- 172.21.15.16:0/654157099 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7faf500544f0 con 0x7faf58161d20 2023-10-16T13:47:12.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:12.561+0000 7faf5e8ac700 1 -- 172.21.15.16:0/654157099 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "backup_fs_metadata", "pool2": "backup_fs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7faf5804ec20 con 0x7faf58161d20 2023-10-16T13:47:13.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.373+0000 7faf54ff9700 1 -- 172.21.15.16:0/654157099 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "backup_fs_metadata", "pool2": "backup_fs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'backup_fs_metadata' removed v236) v1 ==== 190+0+0 (secure 0 0 0) 0x7faf50018020 con 0x7faf58161d20 2023-10-16T13:47:13.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.377+0000 7faf5e8ac700 1 -- 172.21.15.16:0/654157099 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7faf4406b170 msgr2=0x7faf4406d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:13.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.377+0000 7faf5e8ac700 1 --2- 172.21.15.16:0/654157099 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7faf4406b170 0x7faf4406d630 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7faf480097c0 tx=0x7faf48008030 comp rx=0 tx=0).stop 2023-10-16T13:47:13.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.377+0000 7faf5e8ac700 1 -- 172.21.15.16:0/654157099 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf58161d20 msgr2=0x7faf581f6c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:13.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.377+0000 7faf5e8ac700 1 --2- 172.21.15.16:0/654157099 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf58161d20 0x7faf581f6c90 secure :-1 s=READY pgs=2685 cs=0 l=1 rev1=1 crypto rx=0x7faf5000cc60 tx=0x7faf5000cf70 comp rx=0 tx=0).stop 2023-10-16T13:47:13.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.377+0000 7faf5e8ac700 1 -- 172.21.15.16:0/654157099 shutdown_connections 2023-10-16T13:47:13.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.377+0000 7faf5e8ac700 1 --2- 172.21.15.16:0/654157099 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7faf4406b170 0x7faf4406d630 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:13.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.377+0000 7faf5e8ac700 1 --2- 172.21.15.16:0/654157099 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf58161d20 0x7faf581f6c90 unknown :-1 s=CLOSED pgs=2685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:13.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.377+0000 7faf5e8ac700 1 --2- 172.21.15.16:0/654157099 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faf58157b30 0x7faf580c6f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:13.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.377+0000 7faf5e8ac700 1 --2- 172.21.15.16:0/654157099 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faf58157f10 0x7faf580b49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:13.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.377+0000 7faf5e8ac700 1 -- 172.21.15.16:0/654157099 >> 172.21.15.16:0/654157099 conn(0x7faf5806a7e0 msgr2=0x7faf580b0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:13.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.378+0000 7faf5e8ac700 1 -- 172.21.15.16:0/654157099 shutdown_connections 2023-10-16T13:47:13.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.378+0000 7faf5e8ac700 1 -- 172.21.15.16:0/654157099 wait complete. 2023-10-16T13:47:13.383 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'backup_fs_metadata' removed 2023-10-16T13:47:13.397 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm backup_fs_data backup_fs_data --yes-i-really-really-mean-it 2023-10-16T13:47:13.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.530+0000 7fb7abdfa700 1 Processor -- start 2023-10-16T13:47:13.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.531+0000 7fb7abdfa700 1 -- start start 2023-10-16T13:47:13.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.531+0000 7fb7abdfa700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fb7a4155e40 0x7fb7a415bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:13.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.531+0000 7fb7abdfa700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb7a415c190 0x7fb7a40cba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:13.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.531+0000 7fb7abdfa700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7a415c780 con 0x7fb7a4155a60 2023-10-16T13:47:13.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.531+0000 7fb7abdfa700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7a415f240 con 0x7fb7a415c190 2023-10-16T13:47:13.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.531+0000 7fb7abdfa700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7a415ca60 con 0x7fb7a4155e40 2023-10-16T13:47:13.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.532+0000 7fb7aa397700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb7a415c190 0x7fb7a40cba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:13.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.532+0000 7fb7a9b96700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb7a4155a60 0x7fb7a415fc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44356/0 (socket says 172.21.15.16:44356) 2023-10-16T13:47:13.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.532+0000 7fb7a9b96700 1 -- 172.21.15.16:0/1604026322 learned_addr learned my addr 172.21.15.16:0/1604026322 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:47:13.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.532+0000 7fb7a9395700 1 --2- 172.21.15.16:0/1604026322 >> v2:172.21.15.49:3300/0 conn(0x7fb7a4155e40 0x7fb7a415bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:13.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.532+0000 7fb7aa397700 1 -- 172.21.15.16:0/1604026322 >> v2:172.21.15.49:3300/0 conn(0x7fb7a4155e40 msgr2=0x7fb7a415bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:13.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.532+0000 7fb7aa397700 1 --2- 172.21.15.16:0/1604026322 >> v2:172.21.15.49:3300/0 conn(0x7fb7a4155e40 0x7fb7a415bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:13.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.532+0000 7fb7aa397700 1 -- 172.21.15.16:0/1604026322 >> v1:172.21.15.16:6789/0 conn(0x7fb7a4155a60 legacy=0x7fb7a415fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:13.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.532+0000 7fb7aa397700 1 -- 172.21.15.16:0/1604026322 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7a415f450 con 0x7fb7a415c190 2023-10-16T13:47:13.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.532+0000 7fb7a9395700 1 --2- 172.21.15.16:0/1604026322 >> v2:172.21.15.49:3300/0 conn(0x7fb7a4155e40 0x7fb7a415bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:47:13.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.532+0000 7fb7a8b94700 1 -- 172.21.15.16:0/1604026322 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4048690213 0 0) 0x7fb7a415c780 con 0x7fb7a4155a60 2023-10-16T13:47:13.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.532+0000 7fb7aa397700 1 --2- 172.21.15.16:0/1604026322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7a415c190 0x7fb7a40cba40 secure :-1 s=READY pgs=2687 cs=0 l=1 rev1=1 crypto rx=0x7fb7a40d0860 tx=0x7fb79c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb95de5a6a1e69dc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:13.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.532+0000 7fb7a8b94700 1 -- 172.21.15.16:0/1604026322 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb79c00e070 con 0x7fb7a415c190 2023-10-16T13:47:13.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.532+0000 7fb7a8b94700 1 -- 172.21.15.16:0/1604026322 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb79c00bbd0 con 0x7fb7a415c190 2023-10-16T13:47:13.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.532+0000 7fb7a8b94700 1 -- 172.21.15.16:0/1604026322 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb79c012430 con 0x7fb7a415c190 2023-10-16T13:47:13.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.533+0000 7fb7abdfa700 1 -- 172.21.15.16:0/1604026322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7a415c190 msgr2=0x7fb7a40cba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:13.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.533+0000 7fb7abdfa700 1 --2- 172.21.15.16:0/1604026322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7a415c190 0x7fb7a40cba40 secure :-1 s=READY pgs=2687 cs=0 l=1 rev1=1 crypto rx=0x7fb7a40d0860 tx=0x7fb79c00b330 comp rx=0 tx=0).stop 2023-10-16T13:47:13.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.533+0000 7fb7abdfa700 1 -- 172.21.15.16:0/1604026322 shutdown_connections 2023-10-16T13:47:13.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.533+0000 7fb7abdfa700 1 --2- 172.21.15.16:0/1604026322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7a415c190 0x7fb7a40cba40 unknown :-1 s=CLOSED pgs=2687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:13.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.533+0000 7fb7abdfa700 1 --2- 172.21.15.16:0/1604026322 >> v2:172.21.15.49:3300/0 conn(0x7fb7a4155e40 0x7fb7a415bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:13.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.533+0000 7fb7abdfa700 1 -- 172.21.15.16:0/1604026322 >> 172.21.15.16:0/1604026322 conn(0x7fb7a406a7e0 msgr2=0x7fb7a406abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:13.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.533+0000 7fb7abdfa700 1 -- 172.21.15.16:0/1604026322 shutdown_connections 2023-10-16T13:47:13.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.533+0000 7fb7abdfa700 1 -- 172.21.15.16:0/1604026322 wait complete. 2023-10-16T13:47:13.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.534+0000 7fb7abdfa700 1 Processor -- start 2023-10-16T13:47:13.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.534+0000 7fb7abdfa700 1 -- start start 2023-10-16T13:47:13.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.535+0000 7fb7abdfa700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb7a4155a60 0x7fb7a40c84c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:13.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.535+0000 7fb7abdfa700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb7a4155e40 0x7fb7a41f0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:13.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.535+0000 7fb7abdfa700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7a415c190 0x7fb7a41f0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:13.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.535+0000 7fb7abdfa700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a4146e30 con 0x7fb7a415c190 2023-10-16T13:47:13.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.535+0000 7fb7abdfa700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a40c9b70 con 0x7fb7a4155e40 2023-10-16T13:47:13.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.535+0000 7fb7abdfa700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a40c8f60 con 0x7fb7a4155a60 2023-10-16T13:47:13.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.535+0000 7fb7aa397700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7a415c190 0x7fb7a41f0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:13.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.535+0000 7fb7aa397700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7a415c190 0x7fb7a41f0ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55092/0 (socket says 172.21.15.16:55092) 2023-10-16T13:47:13.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.535+0000 7fb7aa397700 1 -- 172.21.15.16:0/2105973264 learned_addr learned my addr 172.21.15.16:0/2105973264 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:13.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.535+0000 7fb7a9b96700 1 --2- 172.21.15.16:0/2105973264 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb7a4155a60 0x7fb7a40c84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:13.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.535+0000 7fb7a9395700 1 --2- 172.21.15.16:0/2105973264 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb7a4155e40 0x7fb7a41f0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:13.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.535+0000 7fb7aa397700 1 -- 172.21.15.16:0/2105973264 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb7a4155a60 msgr2=0x7fb7a40c84c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:13.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.535+0000 7fb7aa397700 1 --2- 172.21.15.16:0/2105973264 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb7a4155a60 0x7fb7a40c84c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:13.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.535+0000 7fb7aa397700 1 -- 172.21.15.16:0/2105973264 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb7a4155e40 msgr2=0x7fb7a41f0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:13.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.535+0000 7fb7aa397700 1 --2- 172.21.15.16:0/2105973264 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb7a4155e40 0x7fb7a41f0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:13.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.535+0000 7fb7aa397700 1 -- 172.21.15.16:0/2105973264 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb79c009a70 con 0x7fb7a415c190 2023-10-16T13:47:13.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.535+0000 7fb7a9395700 1 --2- 172.21.15.16:0/2105973264 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb7a4155e40 0x7fb7a41f0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:47:13.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.535+0000 7fb7a9b96700 1 --2- 172.21.15.16:0/2105973264 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb7a4155a60 0x7fb7a40c84c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:47:13.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.535+0000 7fb7aa397700 1 --2- 172.21.15.16:0/2105973264 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7a415c190 0x7fb7a41f0ba0 secure :-1 s=READY pgs=2688 cs=0 l=1 rev1=1 crypto rx=0x7fb7a40cec60 tx=0x7fb79c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:13.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.536+0000 7fb79a7fc700 1 -- 172.21.15.16:0/2105973264 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb79c012430 con 0x7fb7a415c190 2023-10-16T13:47:13.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.536+0000 7fb79a7fc700 1 -- 172.21.15.16:0/2105973264 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb79c020070 con 0x7fb7a415c190 2023-10-16T13:47:13.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.536+0000 7fb7abdfa700 1 -- 172.21.15.16:0/2105973264 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7a40c9240 con 0x7fb7a415c190 2023-10-16T13:47:13.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.536+0000 7fb79a7fc700 1 -- 172.21.15.16:0/2105973264 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb79c004280 con 0x7fb7a415c190 2023-10-16T13:47:13.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.536+0000 7fb7abdfa700 1 -- 172.21.15.16:0/2105973264 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7a40c9800 con 0x7fb7a415c190 2023-10-16T13:47:13.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.537+0000 7fb7abdfa700 1 -- 172.21.15.16:0/2105973264 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7a40c99c0 con 0x7fb7a415c190 2023-10-16T13:47:13.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.537+0000 7fb79a7fc700 1 -- 172.21.15.16:0/2105973264 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 148) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fb79c0043e0 con 0x7fb7a415c190 2023-10-16T13:47:13.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.537+0000 7fb79a7fc700 1 --2- 172.21.15.16:0/2105973264 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fb78806b140 0x7fb78806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:13.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.538+0000 7fb79a7fc700 1 -- 172.21.15.16:0/2105973264 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 6714+0+0 (secure 0 0 0) 0x7fb79c022070 con 0x7fb7a415c190 2023-10-16T13:47:13.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.538+0000 7fb7a9b96700 1 --2- 172.21.15.16:0/2105973264 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fb78806b140 0x7fb78806d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:13.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.538+0000 7fb7a9b96700 1 --2- 172.21.15.16:0/2105973264 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fb78806b140 0x7fb78806d600 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb794000f60 tx=0x7fb794007ca0 comp rx=0 tx=0).ready entity=mgr.11886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:13.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.542+0000 7fb79a7fc700 1 -- 172.21.15.16:0/2105973264 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb79c056570 con 0x7fb7a415c190 2023-10-16T13:47:13.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:13.748+0000 7fb7abdfa700 1 -- 172.21.15.16:0/2105973264 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "backup_fs_data", "pool2": "backup_fs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb7a40c9ce0 con 0x7fb7a415c190 2023-10-16T13:47:14.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.378+0000 7fb79a7fc700 1 -- 172.21.15.16:0/2105973264 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "backup_fs_data", "pool2": "backup_fs_data", "yes_i_really_really_mean_it": true}]=0 pool 'backup_fs_data' removed v237) v1 ==== 178+0+0 (secure 0 0 0) 0x7fb79c02c0d0 con 0x7fb7a415c190 2023-10-16T13:47:14.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.383+0000 7fb7abdfa700 1 -- 172.21.15.16:0/2105973264 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fb78806b140 msgr2=0x7fb78806d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:14.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.383+0000 7fb7abdfa700 1 --2- 172.21.15.16:0/2105973264 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fb78806b140 0x7fb78806d600 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb794000f60 tx=0x7fb794007ca0 comp rx=0 tx=0).stop 2023-10-16T13:47:14.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.383+0000 7fb7abdfa700 1 -- 172.21.15.16:0/2105973264 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7a415c190 msgr2=0x7fb7a41f0ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:14.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.383+0000 7fb7abdfa700 1 --2- 172.21.15.16:0/2105973264 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7a415c190 0x7fb7a41f0ba0 secure :-1 s=READY pgs=2688 cs=0 l=1 rev1=1 crypto rx=0x7fb7a40cec60 tx=0x7fb79c004ad0 comp rx=0 tx=0).stop 2023-10-16T13:47:14.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.383+0000 7fb7abdfa700 1 -- 172.21.15.16:0/2105973264 shutdown_connections 2023-10-16T13:47:14.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.383+0000 7fb7abdfa700 1 --2- 172.21.15.16:0/2105973264 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fb78806b140 0x7fb78806d600 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:14.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.383+0000 7fb7abdfa700 1 --2- 172.21.15.16:0/2105973264 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7a415c190 0x7fb7a41f0ba0 unknown :-1 s=CLOSED pgs=2688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:14.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.383+0000 7fb7abdfa700 1 --2- 172.21.15.16:0/2105973264 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb7a4155a60 0x7fb7a40c84c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:14.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.383+0000 7fb7abdfa700 1 --2- 172.21.15.16:0/2105973264 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb7a4155e40 0x7fb7a41f0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:14.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.383+0000 7fb7abdfa700 1 -- 172.21.15.16:0/2105973264 >> 172.21.15.16:0/2105973264 conn(0x7fb7a406a7e0 msgr2=0x7fb7a40cd950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:14.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.383+0000 7fb7abdfa700 1 -- 172.21.15.16:0/2105973264 shutdown_connections 2023-10-16T13:47:14.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.384+0000 7fb7abdfa700 1 -- 172.21.15.16:0/2105973264 wait complete. 2023-10-16T13:47:14.390 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'backup_fs_data' removed 2023-10-16T13:47:14.403 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-16T13:47:14.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.535+0000 7f1d85a86700 1 Processor -- start 2023-10-16T13:47:14.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.536+0000 7f1d85a86700 1 -- start start 2023-10-16T13:47:14.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.536+0000 7f1d85a86700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1d8006ca10 0x7f1d800ae8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:14.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.536+0000 7f1d85a86700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d800b20f0 con 0x7f1d800ce6a0 2023-10-16T13:47:14.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.536+0000 7f1d85a86700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d800d03d0 con 0x7f1d800d2110 2023-10-16T13:47:14.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.536+0000 7f1d85a86700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d800b2340 con 0x7f1d8006ca10 2023-10-16T13:47:14.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.537+0000 7f1d7f7fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f1d800ce6a0 0x7f1d8006c300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44370/0 (socket says 172.21.15.16:44370) 2023-10-16T13:47:14.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.537+0000 7f1d7f7fe700 1 -- 172.21.15.16:0/3489979753 learned_addr learned my addr 172.21.15.16:0/3489979753 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:47:14.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.537+0000 7f1d7ffff700 1 --2- 172.21.15.16:0/3489979753 >> v2:172.21.15.16:3300/0 conn(0x7f1d8006ca10 0x7f1d800ae8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:14.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.537+0000 7f1d7e7fc700 1 -- 172.21.15.16:0/3489979753 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3477372994 0 0) 0x7f1d800b20f0 con 0x7f1d800ce6a0 2023-10-16T13:47:14.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.537+0000 7f1d7e7fc700 1 -- 172.21.15.16:0/3489979753 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d680035b0 con 0x7f1d800ce6a0 2023-10-16T13:47:14.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.537+0000 7f1d7ffff700 1 -- 172.21.15.16:0/3489979753 >> v1:172.21.15.49:6789/0 conn(0x7f1d800d2110 legacy=0x7f1d800ae1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:14.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.537+0000 7f1d7ffff700 1 -- 172.21.15.16:0/3489979753 >> v1:172.21.15.16:6789/0 conn(0x7f1d800ce6a0 legacy=0x7f1d8006c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:14.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.537+0000 7f1d7ffff700 1 -- 172.21.15.16:0/3489979753 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d8006bb00 con 0x7f1d8006ca10 2023-10-16T13:47:14.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.537+0000 7f1d7ffff700 1 --2- 172.21.15.16:0/3489979753 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d8006ca10 0x7f1d800ae8f0 secure :-1 s=READY pgs=2690 cs=0 l=1 rev1=1 crypto rx=0x7f1d78009d90 tx=0x7f1d7800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11887d1f697b9d3e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:14.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.537+0000 7f1d7e7fc700 1 -- 172.21.15.16:0/3489979753 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d7800e070 con 0x7f1d8006ca10 2023-10-16T13:47:14.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.537+0000 7f1d7e7fc700 1 -- 172.21.15.16:0/3489979753 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d7800bb20 con 0x7f1d8006ca10 2023-10-16T13:47:14.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.537+0000 7f1d7e7fc700 1 -- 172.21.15.16:0/3489979753 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d78012430 con 0x7f1d8006ca10 2023-10-16T13:47:14.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.538+0000 7f1d85a86700 1 -- 172.21.15.16:0/3489979753 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d8006ca10 msgr2=0x7f1d800ae8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:14.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.538+0000 7f1d85a86700 1 --2- 172.21.15.16:0/3489979753 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d8006ca10 0x7f1d800ae8f0 secure :-1 s=READY pgs=2690 cs=0 l=1 rev1=1 crypto rx=0x7f1d78009d90 tx=0x7f1d7800b330 comp rx=0 tx=0).stop 2023-10-16T13:47:14.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.538+0000 7f1d85a86700 1 -- 172.21.15.16:0/3489979753 shutdown_connections 2023-10-16T13:47:14.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.538+0000 7f1d85a86700 1 --2- 172.21.15.16:0/3489979753 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d8006ca10 0x7f1d800ae8f0 unknown :-1 s=CLOSED pgs=2690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:14.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.538+0000 7f1d85a86700 1 -- 172.21.15.16:0/3489979753 >> 172.21.15.16:0/3489979753 conn(0x7f1d800d15f0 msgr2=0x7f1d800d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:14.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.538+0000 7f1d85a86700 1 -- 172.21.15.16:0/3489979753 shutdown_connections 2023-10-16T13:47:14.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.538+0000 7f1d85a86700 1 -- 172.21.15.16:0/3489979753 wait complete. 2023-10-16T13:47:14.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.539+0000 7f1d85a86700 1 Processor -- start 2023-10-16T13:47:14.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.539+0000 7f1d85a86700 1 -- start start 2023-10-16T13:47:14.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.540+0000 7f1d85a86700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1d8006ca10 0x7f1d8015aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:14.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.540+0000 7f1d85a86700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1d800ce6a0 0x7f1d800c7220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:14.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.540+0000 7f1d85a86700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d800d2110 0x7f1d800c7760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:14.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.540+0000 7f1d85a86700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d800b6aa0 con 0x7f1d800d2110 2023-10-16T13:47:14.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.540+0000 7f1d85a86700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d800b5b80 con 0x7f1d8006ca10 2023-10-16T13:47:14.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.540+0000 7f1d85a86700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d800b5e20 con 0x7f1d800ce6a0 2023-10-16T13:47:14.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.540+0000 7f1d7ffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d800d2110 0x7f1d800c7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:14.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.540+0000 7f1d7ffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d800d2110 0x7f1d800c7760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55106/0 (socket says 172.21.15.16:55106) 2023-10-16T13:47:14.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.540+0000 7f1d7ffff700 1 -- 172.21.15.16:0/828129076 learned_addr learned my addr 172.21.15.16:0/828129076 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:14.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.540+0000 7f1d7f7fe700 1 --2- 172.21.15.16:0/828129076 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1d8006ca10 0x7f1d8015aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:14.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.540+0000 7f1d7ffff700 1 -- 172.21.15.16:0/828129076 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1d800ce6a0 msgr2=0x7f1d800c7220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:14.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.540+0000 7f1d7ffff700 1 --2- 172.21.15.16:0/828129076 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1d800ce6a0 0x7f1d800c7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:14.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.540+0000 7f1d7ffff700 1 -- 172.21.15.16:0/828129076 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1d8006ca10 msgr2=0x7f1d8015aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:14.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.540+0000 7f1d7ffff700 1 --2- 172.21.15.16:0/828129076 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1d8006ca10 0x7f1d8015aeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:14.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.540+0000 7f1d7ffff700 1 -- 172.21.15.16:0/828129076 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d78009a70 con 0x7f1d800d2110 2023-10-16T13:47:14.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.540+0000 7f1d7ffff700 1 --2- 172.21.15.16:0/828129076 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d800d2110 0x7f1d800c7760 secure :-1 s=READY pgs=2691 cs=0 l=1 rev1=1 crypto rx=0x7f1d80067580 tx=0x7f1d78004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:14.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.541+0000 7f1d67fff700 1 -- 172.21.15.16:0/828129076 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d78012430 con 0x7f1d800d2110 2023-10-16T13:47:14.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.541+0000 7f1d67fff700 1 -- 172.21.15.16:0/828129076 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d78020070 con 0x7f1d800d2110 2023-10-16T13:47:14.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.541+0000 7f1d85a86700 1 -- 172.21.15.16:0/828129076 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d800b6100 con 0x7f1d800d2110 2023-10-16T13:47:14.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.541+0000 7f1d67fff700 1 -- 172.21.15.16:0/828129076 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d78004280 con 0x7f1d800d2110 2023-10-16T13:47:14.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.541+0000 7f1d85a86700 1 -- 172.21.15.16:0/828129076 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d800b66c0 con 0x7f1d800d2110 2023-10-16T13:47:14.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.542+0000 7f1d85a86700 1 -- 172.21.15.16:0/828129076 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d8004ebb0 con 0x7f1d800d2110 2023-10-16T13:47:14.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.542+0000 7f1d67fff700 1 -- 172.21.15.16:0/828129076 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 148) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f1d780043e0 con 0x7f1d800d2110 2023-10-16T13:47:14.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.542+0000 7f1d67fff700 1 --2- 172.21.15.16:0/828129076 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f1d6c06b140 0x7f1d6c06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:14.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.543+0000 7f1d67fff700 1 -- 172.21.15.16:0/828129076 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 6294+0+0 (secure 0 0 0) 0x7f1d78022070 con 0x7f1d800d2110 2023-10-16T13:47:14.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.543+0000 7f1d7f7fe700 1 --2- 172.21.15.16:0/828129076 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f1d6c06b140 0x7f1d6c06d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:14.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.543+0000 7f1d7f7fe700 1 --2- 172.21.15.16:0/828129076 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f1d6c06b140 0x7f1d6c06d600 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1d740011e0 tx=0x7f1d74009040 comp rx=0 tx=0).ready entity=mgr.11886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:14.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.547+0000 7f1d67fff700 1 -- 172.21.15.16:0/828129076 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1d78056930 con 0x7f1d800d2110 2023-10-16T13:47:14.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.751+0000 7f1d85a86700 1 -- 172.21.15.16:0/828129076 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f1d8006cf30 con 0x7f1d800d2110 2023-10-16T13:47:14.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.754+0000 7f1d67fff700 1 -- 172.21.15.16:0/828129076 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v79) v1 ==== 115+0+0 (secure 0 0 0) 0x7f1d78059f50 con 0x7f1d800d2110 2023-10-16T13:47:14.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.757+0000 7f1d85a86700 1 -- 172.21.15.16:0/828129076 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f1d6c06b140 msgr2=0x7f1d6c06d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:14.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.757+0000 7f1d85a86700 1 --2- 172.21.15.16:0/828129076 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f1d6c06b140 0x7f1d6c06d600 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1d740011e0 tx=0x7f1d74009040 comp rx=0 tx=0).stop 2023-10-16T13:47:14.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.757+0000 7f1d85a86700 1 -- 172.21.15.16:0/828129076 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d800d2110 msgr2=0x7f1d800c7760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:14.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.757+0000 7f1d85a86700 1 --2- 172.21.15.16:0/828129076 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d800d2110 0x7f1d800c7760 secure :-1 s=READY pgs=2691 cs=0 l=1 rev1=1 crypto rx=0x7f1d80067580 tx=0x7f1d78004ad0 comp rx=0 tx=0).stop 2023-10-16T13:47:14.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.757+0000 7f1d85a86700 1 -- 172.21.15.16:0/828129076 shutdown_connections 2023-10-16T13:47:14.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.757+0000 7f1d85a86700 1 --2- 172.21.15.16:0/828129076 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f1d6c06b140 0x7f1d6c06d600 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:14.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.757+0000 7f1d85a86700 1 --2- 172.21.15.16:0/828129076 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d800d2110 0x7f1d800c7760 unknown :-1 s=CLOSED pgs=2691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:14.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.757+0000 7f1d85a86700 1 --2- 172.21.15.16:0/828129076 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1d8006ca10 0x7f1d8015aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:14.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.757+0000 7f1d85a86700 1 --2- 172.21.15.16:0/828129076 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1d800ce6a0 0x7f1d800c7220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:14.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.757+0000 7f1d85a86700 1 -- 172.21.15.16:0/828129076 >> 172.21.15.16:0/828129076 conn(0x7f1d800d15f0 msgr2=0x7f1d800cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:14.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.757+0000 7f1d85a86700 1 -- 172.21.15.16:0/828129076 shutdown_connections 2023-10-16T13:47:14.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.757+0000 7f1d85a86700 1 -- 172.21.15.16:0/828129076 wait complete. 2023-10-16T13:47:14.773 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-16T13:47:14.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.897+0000 7ff4e3e5b700 1 Processor -- start 2023-10-16T13:47:14.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.898+0000 7ff4e3e5b700 1 -- start start 2023-10-16T13:47:14.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.898+0000 7ff4e3e5b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff4dc15da10 0x7ff4dc0d1520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:14.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.898+0000 7ff4e3e5b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff4dc0d4060 0x7ff4dc0cf2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:14.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.898+0000 7ff4e3e5b700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7ff4dc0d1d20 0x7ff4dc0d2130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:14.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.898+0000 7ff4e3e5b700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4dc0d2670 con 0x7ff4dc0d4060 2023-10-16T13:47:14.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.898+0000 7ff4e3e5b700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4dc0d4540 con 0x7ff4dc0d1d20 2023-10-16T13:47:14.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.898+0000 7ff4e3e5b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4dc0ce7e0 con 0x7ff4dc15da10 2023-10-16T13:47:14.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.899+0000 7ff4e13f6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff4dc0d4060 0x7ff4dc0cf2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:14.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.899+0000 7ff4e13f6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff4dc0d4060 0x7ff4dc0cf2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55110/0 (socket says 172.21.15.16:55110) 2023-10-16T13:47:14.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.899+0000 7ff4e13f6700 1 -- 172.21.15.16:0/1913574579 learned_addr learned my addr 172.21.15.16:0/1913574579 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:14.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.899+0000 7ff4e23f8700 1 --2- 172.21.15.16:0/1913574579 >> v2:172.21.15.49:3300/0 conn(0x7ff4dc0d1d20 0x7ff4dc0d2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:14.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.899+0000 7ff4e1bf7700 1 --2- 172.21.15.16:0/1913574579 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff4dc15da10 0x7ff4dc0d1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:14.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.899+0000 7ff4e13f6700 1 -- 172.21.15.16:0/1913574579 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff4dc15da10 msgr2=0x7ff4dc0d1520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:14.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.899+0000 7ff4e13f6700 1 --2- 172.21.15.16:0/1913574579 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff4dc15da10 0x7ff4dc0d1520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:14.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.899+0000 7ff4e13f6700 1 -- 172.21.15.16:0/1913574579 >> v2:172.21.15.49:3300/0 conn(0x7ff4dc0d1d20 msgr2=0x7ff4dc0d2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:14.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.899+0000 7ff4e13f6700 1 --2- 172.21.15.16:0/1913574579 >> v2:172.21.15.49:3300/0 conn(0x7ff4dc0d1d20 0x7ff4dc0d2130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:14.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.899+0000 7ff4e13f6700 1 -- 172.21.15.16:0/1913574579 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4dc0ceac0 con 0x7ff4dc0d4060 2023-10-16T13:47:14.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.899+0000 7ff4e23f8700 1 --2- 172.21.15.16:0/1913574579 >> v2:172.21.15.49:3300/0 conn(0x7ff4dc0d1d20 0x7ff4dc0d2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:47:14.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.899+0000 7ff4e13f6700 1 --2- 172.21.15.16:0/1913574579 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4dc0d4060 0x7ff4dc0cf2a0 secure :-1 s=READY pgs=2692 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc0d3550 tx=0x7ff4d8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6196d802664eefa server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:14.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.899+0000 7ff4e0bf5700 1 -- 172.21.15.16:0/1913574579 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d800b870 con 0x7ff4dc0d4060 2023-10-16T13:47:14.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.899+0000 7ff4e0bf5700 1 -- 172.21.15.16:0/1913574579 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4d800b9d0 con 0x7ff4dc0d4060 2023-10-16T13:47:14.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.899+0000 7ff4e0bf5700 1 -- 172.21.15.16:0/1913574579 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d800bd60 con 0x7ff4dc0d4060 2023-10-16T13:47:14.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.900+0000 7ff4e3e5b700 1 -- 172.21.15.16:0/1913574579 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4dc0d4060 msgr2=0x7ff4dc0cf2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:14.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.900+0000 7ff4e3e5b700 1 --2- 172.21.15.16:0/1913574579 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4dc0d4060 0x7ff4dc0cf2a0 secure :-1 s=READY pgs=2692 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc0d3550 tx=0x7ff4d8009bd0 comp rx=0 tx=0).stop 2023-10-16T13:47:14.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.900+0000 7ff4e3e5b700 1 -- 172.21.15.16:0/1913574579 shutdown_connections 2023-10-16T13:47:14.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.900+0000 7ff4e3e5b700 1 --2- 172.21.15.16:0/1913574579 >> v2:172.21.15.49:3300/0 conn(0x7ff4dc0d1d20 0x7ff4dc0d2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:14.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.900+0000 7ff4e3e5b700 1 --2- 172.21.15.16:0/1913574579 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff4dc15da10 0x7ff4dc0d1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:14.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.900+0000 7ff4e3e5b700 1 --2- 172.21.15.16:0/1913574579 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4dc0d4060 0x7ff4dc0cf2a0 unknown :-1 s=CLOSED pgs=2692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:14.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.900+0000 7ff4e3e5b700 1 -- 172.21.15.16:0/1913574579 >> 172.21.15.16:0/1913574579 conn(0x7ff4dc09c9f0 msgr2=0x7ff4dc0d3740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:14.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.900+0000 7ff4e3e5b700 1 -- 172.21.15.16:0/1913574579 shutdown_connections 2023-10-16T13:47:14.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.900+0000 7ff4e3e5b700 1 -- 172.21.15.16:0/1913574579 wait complete. 2023-10-16T13:47:14.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.901+0000 7ff4e3e5b700 1 Processor -- start 2023-10-16T13:47:14.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.902+0000 7ff4e3e5b700 1 -- start start 2023-10-16T13:47:14.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.902+0000 7ff4e3e5b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff4dc0d1d20 0x7ff4dc1f6890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:14.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.903+0000 7ff4e3e5b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4dc0d4060 0x7ff4dc1f6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:14.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.903+0000 7ff4e3e5b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff4dc15da10 0x7ff4dc0cb940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:14.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.903+0000 7ff4e3e5b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4dc0cc670 con 0x7ff4dc0d4060 2023-10-16T13:47:14.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.903+0000 7ff4e3e5b700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4dc0cc950 con 0x7ff4dc0d1d20 2023-10-16T13:47:14.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.903+0000 7ff4e3e5b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4dc0ccc30 con 0x7ff4dc15da10 2023-10-16T13:47:14.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.903+0000 7ff4e13f6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4dc0d4060 0x7ff4dc1f6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:14.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.903+0000 7ff4e13f6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4dc0d4060 0x7ff4dc1f6dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55118/0 (socket says 172.21.15.16:55118) 2023-10-16T13:47:14.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.903+0000 7ff4e13f6700 1 -- 172.21.15.16:0/2524464010 learned_addr learned my addr 172.21.15.16:0/2524464010 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:14.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.903+0000 7ff4e1bf7700 1 --2- 172.21.15.16:0/2524464010 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff4dc0d1d20 0x7ff4dc1f6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:14.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.903+0000 7ff4e23f8700 1 --2- 172.21.15.16:0/2524464010 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff4dc15da10 0x7ff4dc0cb940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:14.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.903+0000 7ff4e13f6700 1 -- 172.21.15.16:0/2524464010 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff4dc15da10 msgr2=0x7ff4dc0cb940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:14.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.903+0000 7ff4e13f6700 1 --2- 172.21.15.16:0/2524464010 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff4dc15da10 0x7ff4dc0cb940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:14.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.903+0000 7ff4e13f6700 1 -- 172.21.15.16:0/2524464010 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff4dc0d1d20 msgr2=0x7ff4dc1f6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:14.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.903+0000 7ff4e13f6700 1 --2- 172.21.15.16:0/2524464010 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff4dc0d1d20 0x7ff4dc1f6890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:14.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.903+0000 7ff4e13f6700 1 -- 172.21.15.16:0/2524464010 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4d8009550 con 0x7ff4dc0d4060 2023-10-16T13:47:14.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.903+0000 7ff4e23f8700 1 --2- 172.21.15.16:0/2524464010 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff4dc15da10 0x7ff4dc0cb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:47:14.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.903+0000 7ff4e13f6700 1 --2- 172.21.15.16:0/2524464010 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4dc0d4060 0x7ff4dc1f6dd0 secure :-1 s=READY pgs=2693 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc08b640 tx=0x7ff4d8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:14.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.903+0000 7ff4d27fc700 1 -- 172.21.15.16:0/2524464010 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d8004310 con 0x7ff4dc0d4060 2023-10-16T13:47:14.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.903+0000 7ff4d27fc700 1 -- 172.21.15.16:0/2524464010 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4d801f070 con 0x7ff4dc0d4060 2023-10-16T13:47:14.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.903+0000 7ff4e3e5b700 1 -- 172.21.15.16:0/2524464010 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4dc0ccf10 con 0x7ff4dc0d4060 2023-10-16T13:47:14.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.903+0000 7ff4d27fc700 1 -- 172.21.15.16:0/2524464010 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d80036a0 con 0x7ff4dc0d4060 2023-10-16T13:47:14.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.903+0000 7ff4e3e5b700 1 -- 172.21.15.16:0/2524464010 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4dc0cbe80 con 0x7ff4dc0d4060 2023-10-16T13:47:14.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.904+0000 7ff4e3e5b700 1 -- 172.21.15.16:0/2524464010 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4dc15af00 con 0x7ff4dc0d4060 2023-10-16T13:47:14.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.905+0000 7ff4d27fc700 1 -- 172.21.15.16:0/2524464010 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 148) v1 ==== 89433+0+0 (secure 0 0 0) 0x7ff4d8021030 con 0x7ff4dc0d4060 2023-10-16T13:47:14.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.905+0000 7ff4d27fc700 1 --2- 172.21.15.16:0/2524464010 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7ff4c006b140 0x7ff4c006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:14.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.905+0000 7ff4d27fc700 1 -- 172.21.15.16:0/2524464010 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 6294+0+0 (secure 0 0 0) 0x7ff4d808b0b0 con 0x7ff4dc0d4060 2023-10-16T13:47:14.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.905+0000 7ff4e1bf7700 1 --2- 172.21.15.16:0/2524464010 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7ff4c006b140 0x7ff4c006d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:14.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.906+0000 7ff4e1bf7700 1 --2- 172.21.15.16:0/2524464010 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7ff4c006b140 0x7ff4c006d600 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc0a1540 tx=0x7ff4cc008040 comp rx=0 tx=0).ready entity=mgr.11886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:14.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:14.910+0000 7ff4d27fc700 1 -- 172.21.15.16:0/2524464010 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff4d8055d60 con 0x7ff4dc0d4060 2023-10-16T13:47:15.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.117+0000 7ff4e3e5b700 1 -- 172.21.15.16:0/2524464010 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7ff4dc0cd0a0 con 0x7ff4dc0d4060 2023-10-16T13:47:15.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.120+0000 7ff4d27fc700 1 -- 172.21.15.16:0/2524464010 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v80) v1 ==== 104+0+0 (secure 0 0 0) 0x7ff4d8059380 con 0x7ff4dc0d4060 2023-10-16T13:47:15.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.122+0000 7ff4e3e5b700 1 -- 172.21.15.16:0/2524464010 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7ff4c006b140 msgr2=0x7ff4c006d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:15.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.122+0000 7ff4e3e5b700 1 --2- 172.21.15.16:0/2524464010 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7ff4c006b140 0x7ff4c006d600 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc0a1540 tx=0x7ff4cc008040 comp rx=0 tx=0).stop 2023-10-16T13:47:15.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.122+0000 7ff4e3e5b700 1 -- 172.21.15.16:0/2524464010 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4dc0d4060 msgr2=0x7ff4dc1f6dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:15.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.122+0000 7ff4e3e5b700 1 --2- 172.21.15.16:0/2524464010 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4dc0d4060 0x7ff4dc1f6dd0 secure :-1 s=READY pgs=2693 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc08b640 tx=0x7ff4d8004750 comp rx=0 tx=0).stop 2023-10-16T13:47:15.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.122+0000 7ff4e3e5b700 1 -- 172.21.15.16:0/2524464010 shutdown_connections 2023-10-16T13:47:15.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.122+0000 7ff4e3e5b700 1 --2- 172.21.15.16:0/2524464010 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7ff4c006b140 0x7ff4c006d600 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:15.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.123+0000 7ff4e3e5b700 1 --2- 172.21.15.16:0/2524464010 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff4dc15da10 0x7ff4dc0cb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:15.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.123+0000 7ff4e3e5b700 1 --2- 172.21.15.16:0/2524464010 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff4dc0d1d20 0x7ff4dc1f6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:15.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.123+0000 7ff4e3e5b700 1 --2- 172.21.15.16:0/2524464010 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4dc0d4060 0x7ff4dc1f6dd0 unknown :-1 s=CLOSED pgs=2693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:15.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.123+0000 7ff4e3e5b700 1 -- 172.21.15.16:0/2524464010 >> 172.21.15.16:0/2524464010 conn(0x7ff4dc09c9f0 msgr2=0x7ff4dc06c340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:15.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.123+0000 7ff4e3e5b700 1 -- 172.21.15.16:0/2524464010 shutdown_connections 2023-10-16T13:47:15.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.123+0000 7ff4e3e5b700 1 -- 172.21.15.16:0/2524464010 wait complete. 2023-10-16T13:47:15.137 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-16T13:47:15.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.262+0000 7feca02e3700 1 Processor -- start 2023-10-16T13:47:15.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.262+0000 7feca02e3700 1 -- start start 2023-10-16T13:47:15.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.263+0000 7feca02e3700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fec98160fe0 0x7fec980ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:15.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.263+0000 7feca02e3700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fec9806da30 0x7fec9806a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:15.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.263+0000 7feca02e3700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fec9806ad20 0x7fec98071a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:15.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.263+0000 7feca02e3700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec980cf680 con 0x7fec9806ad20 2023-10-16T13:47:15.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.263+0000 7feca02e3700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec9806b1b0 con 0x7fec98160fe0 2023-10-16T13:47:15.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.263+0000 7feca02e3700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec981613c0 con 0x7fec9806da30 2023-10-16T13:47:15.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.263+0000 7fec9e880700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fec9806ad20 0x7fec98071a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:15.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.263+0000 7fec9e880700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fec9806ad20 0x7fec98071a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55128/0 (socket says 172.21.15.16:55128) 2023-10-16T13:47:15.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.263+0000 7fec9e880700 1 -- 172.21.15.16:0/1845943754 learned_addr learned my addr 172.21.15.16:0/1845943754 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:15.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.263+0000 7fec9e07f700 1 --2- 172.21.15.16:0/1845943754 >> v2:172.21.15.49:3300/0 conn(0x7fec98160fe0 0x7fec980ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:15.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.263+0000 7fec9d87e700 1 --2- 172.21.15.16:0/1845943754 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fec9806da30 0x7fec9806a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:15.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.263+0000 7fec9e880700 1 -- 172.21.15.16:0/1845943754 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fec9806da30 msgr2=0x7fec9806a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:15.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.263+0000 7fec9e880700 1 --2- 172.21.15.16:0/1845943754 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fec9806da30 0x7fec9806a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:15.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.263+0000 7fec9e880700 1 -- 172.21.15.16:0/1845943754 >> v2:172.21.15.49:3300/0 conn(0x7fec98160fe0 msgr2=0x7fec980ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:15.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.263+0000 7fec9e880700 1 --2- 172.21.15.16:0/1845943754 >> v2:172.21.15.49:3300/0 conn(0x7fec98160fe0 0x7fec980ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:15.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.263+0000 7fec9e880700 1 -- 172.21.15.16:0/1845943754 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec980adb00 con 0x7fec9806ad20 2023-10-16T13:47:15.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.264+0000 7fec9e880700 1 --2- 172.21.15.16:0/1845943754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec9806ad20 0x7fec98071a00 secure :-1 s=READY pgs=2694 cs=0 l=1 rev1=1 crypto rx=0x7fec980b2d40 tx=0x7fec9000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eccb2c216a53ab36 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:15.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.264+0000 7fec9d07d700 1 -- 172.21.15.16:0/1845943754 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec9000e070 con 0x7fec9806ad20 2023-10-16T13:47:15.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.264+0000 7fec9d07d700 1 -- 172.21.15.16:0/1845943754 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec9000b910 con 0x7fec9806ad20 2023-10-16T13:47:15.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.264+0000 7fec9d07d700 1 -- 172.21.15.16:0/1845943754 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec9000bcc0 con 0x7fec9806ad20 2023-10-16T13:47:15.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.264+0000 7feca02e3700 1 -- 172.21.15.16:0/1845943754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec9806ad20 msgr2=0x7fec98071a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:15.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.264+0000 7feca02e3700 1 --2- 172.21.15.16:0/1845943754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec9806ad20 0x7fec98071a00 secure :-1 s=READY pgs=2694 cs=0 l=1 rev1=1 crypto rx=0x7fec980b2d40 tx=0x7fec9000b330 comp rx=0 tx=0).stop 2023-10-16T13:47:15.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.264+0000 7feca02e3700 1 -- 172.21.15.16:0/1845943754 shutdown_connections 2023-10-16T13:47:15.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.264+0000 7feca02e3700 1 --2- 172.21.15.16:0/1845943754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec9806ad20 0x7fec98071a00 unknown :-1 s=CLOSED pgs=2694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:15.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.264+0000 7feca02e3700 1 --2- 172.21.15.16:0/1845943754 >> v2:172.21.15.49:3300/0 conn(0x7fec98160fe0 0x7fec980ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:15.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.264+0000 7feca02e3700 1 --2- 172.21.15.16:0/1845943754 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fec9806da30 0x7fec9806a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:15.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.264+0000 7feca02e3700 1 -- 172.21.15.16:0/1845943754 >> 172.21.15.16:0/1845943754 conn(0x7fec980b1ed0 msgr2=0x7fec980b22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:15.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.264+0000 7feca02e3700 1 -- 172.21.15.16:0/1845943754 shutdown_connections 2023-10-16T13:47:15.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.264+0000 7feca02e3700 1 -- 172.21.15.16:0/1845943754 wait complete. 2023-10-16T13:47:15.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.266+0000 7feca02e3700 1 Processor -- start 2023-10-16T13:47:15.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.266+0000 7feca02e3700 1 -- start start 2023-10-16T13:47:15.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.266+0000 7feca02e3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec9806ad20 0x7fec980b3230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:15.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.266+0000 7feca02e3700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fec9806da30 0x7fec980c4c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:15.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.266+0000 7feca02e3700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fec98160fe0 0x7fec980c5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:15.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.266+0000 7feca02e3700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec980c6390 con 0x7fec9806ad20 2023-10-16T13:47:15.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.266+0000 7feca02e3700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec9814bb40 con 0x7fec98160fe0 2023-10-16T13:47:15.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.266+0000 7feca02e3700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec980c5780 con 0x7fec9806da30 2023-10-16T13:47:15.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.267+0000 7fec9e07f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec9806ad20 0x7fec980b3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:15.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.267+0000 7fec9e07f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec9806ad20 0x7fec980b3230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55130/0 (socket says 172.21.15.16:55130) 2023-10-16T13:47:15.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.267+0000 7fec9e07f700 1 -- 172.21.15.16:0/3663267592 learned_addr learned my addr 172.21.15.16:0/3663267592 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:15.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.267+0000 7fec9e880700 1 --2- 172.21.15.16:0/3663267592 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fec98160fe0 0x7fec980c5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:15.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.267+0000 7fec9d87e700 1 --2- 172.21.15.16:0/3663267592 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fec9806da30 0x7fec980c4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:15.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.267+0000 7fec9e07f700 1 -- 172.21.15.16:0/3663267592 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fec9806da30 msgr2=0x7fec980c4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:15.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.267+0000 7fec9e07f700 1 --2- 172.21.15.16:0/3663267592 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fec9806da30 0x7fec980c4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:15.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.267+0000 7fec9e07f700 1 -- 172.21.15.16:0/3663267592 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fec98160fe0 msgr2=0x7fec980c5150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:15.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.267+0000 7fec9e07f700 1 --2- 172.21.15.16:0/3663267592 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fec98160fe0 0x7fec980c5150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:15.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.267+0000 7fec9e07f700 1 -- 172.21.15.16:0/3663267592 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec90009b30 con 0x7fec9806ad20 2023-10-16T13:47:15.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.267+0000 7fec9d87e700 1 --2- 172.21.15.16:0/3663267592 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fec9806da30 0x7fec980c4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:47:15.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.267+0000 7fec9e07f700 1 --2- 172.21.15.16:0/3663267592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec9806ad20 0x7fec980b3230 secure :-1 s=READY pgs=2695 cs=0 l=1 rev1=1 crypto rx=0x7fec8800e930 tx=0x7fec8800ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:15.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.267+0000 7fec8effd700 1 -- 172.21.15.16:0/3663267592 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec88009650 con 0x7fec9806ad20 2023-10-16T13:47:15.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.267+0000 7fec8effd700 1 -- 172.21.15.16:0/3663267592 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec8800ee80 con 0x7fec9806ad20 2023-10-16T13:47:15.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.267+0000 7feca02e3700 1 -- 172.21.15.16:0/3663267592 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec980c5a60 con 0x7fec9806ad20 2023-10-16T13:47:15.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.267+0000 7fec8effd700 1 -- 172.21.15.16:0/3663267592 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec8800ca50 con 0x7fec9806ad20 2023-10-16T13:47:15.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.267+0000 7feca02e3700 1 -- 172.21.15.16:0/3663267592 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec980c6020 con 0x7fec9806ad20 2023-10-16T13:47:15.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.268+0000 7feca02e3700 1 -- 172.21.15.16:0/3663267592 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec9804ebb0 con 0x7fec9806ad20 2023-10-16T13:47:15.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.269+0000 7fec8effd700 1 -- 172.21.15.16:0/3663267592 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 148) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fec8800cbb0 con 0x7fec9806ad20 2023-10-16T13:47:15.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.269+0000 7fec8effd700 1 --2- 172.21.15.16:0/3663267592 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fec7c06b140 0x7fec7c06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:15.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.269+0000 7fec8effd700 1 -- 172.21.15.16:0/3663267592 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 6294+0+0 (secure 0 0 0) 0x7fec88089f90 con 0x7fec9806ad20 2023-10-16T13:47:15.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.269+0000 7fec9d87e700 1 --2- 172.21.15.16:0/3663267592 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fec7c06b140 0x7fec7c06d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:15.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.270+0000 7fec9d87e700 1 --2- 172.21.15.16:0/3663267592 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fec7c06b140 0x7fec7c06d600 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fec9815d3a0 tx=0x7fec94008040 comp rx=0 tx=0).ready entity=mgr.11886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:15.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.274+0000 7fec8effd700 1 -- 172.21.15.16:0/3663267592 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fec880bcfe0 con 0x7fec9806ad20 2023-10-16T13:47:15.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.480+0000 7feca02e3700 1 -- 172.21.15.16:0/3663267592 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7fec980c61e0 con 0x7fec9806ad20 2023-10-16T13:47:15.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.483+0000 7fec8effd700 1 -- 172.21.15.16:0/3663267592 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v81) v1 ==== 120+0+0 (secure 0 0 0) 0x7fec880170b0 con 0x7fec9806ad20 2023-10-16T13:47:15.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.485+0000 7feca02e3700 1 -- 172.21.15.16:0/3663267592 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fec7c06b140 msgr2=0x7fec7c06d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:15.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.485+0000 7feca02e3700 1 --2- 172.21.15.16:0/3663267592 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fec7c06b140 0x7fec7c06d600 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fec9815d3a0 tx=0x7fec94008040 comp rx=0 tx=0).stop 2023-10-16T13:47:15.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.485+0000 7feca02e3700 1 -- 172.21.15.16:0/3663267592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec9806ad20 msgr2=0x7fec980b3230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:15.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.485+0000 7feca02e3700 1 --2- 172.21.15.16:0/3663267592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec9806ad20 0x7fec980b3230 secure :-1 s=READY pgs=2695 cs=0 l=1 rev1=1 crypto rx=0x7fec8800e930 tx=0x7fec8800ec40 comp rx=0 tx=0).stop 2023-10-16T13:47:15.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.485+0000 7feca02e3700 1 -- 172.21.15.16:0/3663267592 shutdown_connections 2023-10-16T13:47:15.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.485+0000 7feca02e3700 1 --2- 172.21.15.16:0/3663267592 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fec7c06b140 0x7fec7c06d600 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:15.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.485+0000 7feca02e3700 1 --2- 172.21.15.16:0/3663267592 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fec98160fe0 0x7fec980c5150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:15.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.485+0000 7feca02e3700 1 --2- 172.21.15.16:0/3663267592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec9806ad20 0x7fec980b3230 unknown :-1 s=CLOSED pgs=2695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:15.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.485+0000 7feca02e3700 1 --2- 172.21.15.16:0/3663267592 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fec9806da30 0x7fec980c4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:15.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.485+0000 7feca02e3700 1 -- 172.21.15.16:0/3663267592 >> 172.21.15.16:0/3663267592 conn(0x7fec980b1ed0 msgr2=0x7fec980adf90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:15.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.486+0000 7feca02e3700 1 -- 172.21.15.16:0/3663267592 shutdown_connections 2023-10-16T13:47:15.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.486+0000 7feca02e3700 1 -- 172.21.15.16:0/3663267592 wait complete. 2023-10-16T13:47:15.501 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs' 2023-10-16T13:47:15.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.628+0000 7f46ae5ba700 1 Processor -- start 2023-10-16T13:47:15.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.628+0000 7f46ae5ba700 1 -- start start 2023-10-16T13:47:15.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.629+0000 7f46ae5ba700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f46a815d8d0 0x7f46a80d1c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:15.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.629+0000 7f46ae5ba700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f46a80d0750 0x7f46a806c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:15.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.629+0000 7f46ae5ba700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46a806ba00 con 0x7f46a80d41c0 2023-10-16T13:47:15.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.629+0000 7f46ae5ba700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46a80d2150 con 0x7f46a80d0750 2023-10-16T13:47:15.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.629+0000 7f46ae5ba700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46a80d26d0 con 0x7f46a815d8d0 2023-10-16T13:47:15.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.629+0000 7f46acb57700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f46a80d0750 0x7f46a806c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:15.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.629+0000 7f46acb57700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f46a80d0750 0x7f46a806c4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55142/0 (socket says 172.21.15.16:55142) 2023-10-16T13:47:15.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.629+0000 7f46acb57700 1 -- 172.21.15.16:0/308675810 learned_addr learned my addr 172.21.15.16:0/308675810 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:15.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.629+0000 7f46a7fff700 1 --2- 172.21.15.16:0/308675810 >> v2:172.21.15.49:3300/0 conn(0x7f46a815d8d0 0x7f46a80d1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:15.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.629+0000 7f46acb57700 1 -- 172.21.15.16:0/308675810 >> v2:172.21.15.49:3300/0 conn(0x7f46a815d8d0 msgr2=0x7f46a80d1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:15.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.629+0000 7f46acb57700 1 --2- 172.21.15.16:0/308675810 >> v2:172.21.15.49:3300/0 conn(0x7f46a815d8d0 0x7f46a80d1c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:15.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.629+0000 7f46acb57700 1 -- 172.21.15.16:0/308675810 >> v1:172.21.15.49:6789/0 conn(0x7f46a80d41c0 legacy=0x7f46a80d0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:15.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.629+0000 7f46acb57700 1 -- 172.21.15.16:0/308675810 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46a80d23f0 con 0x7f46a80d0750 2023-10-16T13:47:15.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.629+0000 7f46a7fff700 1 --2- 172.21.15.16:0/308675810 >> v2:172.21.15.49:3300/0 conn(0x7f46a815d8d0 0x7f46a80d1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:47:15.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.629+0000 7f46acb57700 1 --2- 172.21.15.16:0/308675810 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46a80d0750 0x7f46a806c4a0 secure :-1 s=READY pgs=2696 cs=0 l=1 rev1=1 crypto rx=0x7f46a808b460 tx=0x7f46a000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76c983d1f4b652cb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:15.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.630+0000 7f46a6ffd700 1 -- 172.21.15.16:0/308675810 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46a000e070 con 0x7f46a80d0750 2023-10-16T13:47:15.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.630+0000 7f46a6ffd700 1 -- 172.21.15.16:0/308675810 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46a000b9c0 con 0x7f46a80d0750 2023-10-16T13:47:15.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.630+0000 7f46a6ffd700 1 -- 172.21.15.16:0/308675810 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46a000bd70 con 0x7f46a80d0750 2023-10-16T13:47:15.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.630+0000 7f46ae5ba700 1 -- 172.21.15.16:0/308675810 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46a80d0750 msgr2=0x7f46a806c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:15.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.630+0000 7f46ae5ba700 1 --2- 172.21.15.16:0/308675810 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46a80d0750 0x7f46a806c4a0 secure :-1 s=READY pgs=2696 cs=0 l=1 rev1=1 crypto rx=0x7f46a808b460 tx=0x7f46a000b330 comp rx=0 tx=0).stop 2023-10-16T13:47:15.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.630+0000 7f46ae5ba700 1 -- 172.21.15.16:0/308675810 shutdown_connections 2023-10-16T13:47:15.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.630+0000 7f46ae5ba700 1 --2- 172.21.15.16:0/308675810 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46a80d0750 0x7f46a806c4a0 unknown :-1 s=CLOSED pgs=2696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:15.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.630+0000 7f46ae5ba700 1 --2- 172.21.15.16:0/308675810 >> v2:172.21.15.49:3300/0 conn(0x7f46a815d8d0 0x7f46a80d1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:15.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.630+0000 7f46ae5ba700 1 -- 172.21.15.16:0/308675810 >> 172.21.15.16:0/308675810 conn(0x7f46a80d3630 msgr2=0x7f46a80d3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:15.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.630+0000 7f46ae5ba700 1 -- 172.21.15.16:0/308675810 shutdown_connections 2023-10-16T13:47:15.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.630+0000 7f46ae5ba700 1 -- 172.21.15.16:0/308675810 wait complete. 2023-10-16T13:47:15.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.631+0000 7f46ae5ba700 1 Processor -- start 2023-10-16T13:47:15.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.632+0000 7f46ae5ba700 1 -- start start 2023-10-16T13:47:15.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.632+0000 7f46ae5ba700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f46a80d0750 0x7f46a815aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:15.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.632+0000 7f46ae5ba700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46a80d41c0 0x7f46a809db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:15.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.632+0000 7f46ae5ba700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f46a815d8d0 0x7f46a809f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:15.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.632+0000 7f46ae5ba700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46a81602c0 con 0x7f46a80d41c0 2023-10-16T13:47:15.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.632+0000 7f46ae5ba700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46a815b010 con 0x7f46a80d0750 2023-10-16T13:47:15.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.632+0000 7f46ae5ba700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46a8160a60 con 0x7f46a815d8d0 2023-10-16T13:47:15.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.632+0000 7f46a77fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46a80d41c0 0x7f46a809db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:15.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.632+0000 7f46a77fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46a80d41c0 0x7f46a809db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55146/0 (socket says 172.21.15.16:55146) 2023-10-16T13:47:15.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.632+0000 7f46a77fe700 1 -- 172.21.15.16:0/623604240 learned_addr learned my addr 172.21.15.16:0/623604240 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:15.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.632+0000 7f46a7fff700 1 --2- 172.21.15.16:0/623604240 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f46a80d0750 0x7f46a815aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:15.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.633+0000 7f46acb57700 1 --2- 172.21.15.16:0/623604240 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f46a815d8d0 0x7f46a809f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:15.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.633+0000 7f46a77fe700 1 -- 172.21.15.16:0/623604240 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f46a815d8d0 msgr2=0x7f46a809f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:15.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.633+0000 7f46a77fe700 1 --2- 172.21.15.16:0/623604240 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f46a815d8d0 0x7f46a809f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:15.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.633+0000 7f46a77fe700 1 -- 172.21.15.16:0/623604240 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f46a80d0750 msgr2=0x7f46a815aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:15.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.633+0000 7f46a77fe700 1 --2- 172.21.15.16:0/623604240 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f46a80d0750 0x7f46a815aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:15.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.633+0000 7f46a77fe700 1 -- 172.21.15.16:0/623604240 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46a0009b30 con 0x7f46a80d41c0 2023-10-16T13:47:15.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.633+0000 7f46acb57700 1 --2- 172.21.15.16:0/623604240 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f46a815d8d0 0x7f46a809f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:47:15.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.633+0000 7f46a77fe700 1 --2- 172.21.15.16:0/623604240 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46a80d41c0 0x7f46a809db00 secure :-1 s=READY pgs=2697 cs=0 l=1 rev1=1 crypto rx=0x7f469000c510 tx=0x7f469000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:15.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.633+0000 7f46a4ff9700 1 -- 172.21.15.16:0/623604240 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46900052d0 con 0x7f46a80d41c0 2023-10-16T13:47:15.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.633+0000 7f46a4ff9700 1 -- 172.21.15.16:0/623604240 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4690005450 con 0x7f46a80d41c0 2023-10-16T13:47:15.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.633+0000 7f46ae5ba700 1 -- 172.21.15.16:0/623604240 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46a81604a0 con 0x7f46a80d41c0 2023-10-16T13:47:15.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.633+0000 7f46a4ff9700 1 -- 172.21.15.16:0/623604240 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46900038c0 con 0x7f46a80d41c0 2023-10-16T13:47:15.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.633+0000 7f46ae5ba700 1 -- 172.21.15.16:0/623604240 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46a81f4200 con 0x7f46a80d41c0 2023-10-16T13:47:15.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.634+0000 7f46ae5ba700 1 -- 172.21.15.16:0/623604240 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46a80d2c00 con 0x7f46a80d41c0 2023-10-16T13:47:15.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.634+0000 7f46a4ff9700 1 -- 172.21.15.16:0/623604240 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 148) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f4690003a20 con 0x7f46a80d41c0 2023-10-16T13:47:15.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.635+0000 7f46a4ff9700 1 --2- 172.21.15.16:0/623604240 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f468806b140 0x7f468806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:15.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.635+0000 7f46a4ff9700 1 -- 172.21.15.16:0/623604240 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 6294+0+0 (secure 0 0 0) 0x7f4690089040 con 0x7f46a80d41c0 2023-10-16T13:47:15.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.635+0000 7f46a7fff700 1 --2- 172.21.15.16:0/623604240 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f468806b140 0x7f468806d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:15.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.635+0000 7f46a7fff700 1 --2- 172.21.15.16:0/623604240 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f468806b140 0x7f468806d600 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f469800c900 tx=0x7f4698008040 comp rx=0 tx=0).ready entity=mgr.11886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:15.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.639+0000 7f46a4ff9700 1 -- 172.21.15.16:0/623604240 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4690053b80 con 0x7f46a80d41c0 2023-10-16T13:47:15.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:15.927+0000 7f46ae5ba700 1 -- 172.21.15.16:0/623604240 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs"]} v 0) v1 -- 0x7f46a806bc70 con 0x7f46a80d41c0 2023-10-16T13:47:16.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.381+0000 7f46a4ff9700 1 -- 172.21.15.16:0/623604240 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs"]}]=0 v3176) v1 ==== 186+0+0 (secure 0 0 0) 0x7f4690017100 con 0x7f46a80d41c0 2023-10-16T13:47:16.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.383+0000 7f46ae5ba700 1 -- 172.21.15.16:0/623604240 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f468806b140 msgr2=0x7f468806d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:16.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.383+0000 7f46ae5ba700 1 --2- 172.21.15.16:0/623604240 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f468806b140 0x7f468806d600 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f469800c900 tx=0x7f4698008040 comp rx=0 tx=0).stop 2023-10-16T13:47:16.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.383+0000 7f46ae5ba700 1 -- 172.21.15.16:0/623604240 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46a80d41c0 msgr2=0x7f46a809db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:16.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.383+0000 7f46ae5ba700 1 --2- 172.21.15.16:0/623604240 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46a80d41c0 0x7f46a809db00 secure :-1 s=READY pgs=2697 cs=0 l=1 rev1=1 crypto rx=0x7f469000c510 tx=0x7f469000c8d0 comp rx=0 tx=0).stop 2023-10-16T13:47:16.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.384+0000 7f46ae5ba700 1 -- 172.21.15.16:0/623604240 shutdown_connections 2023-10-16T13:47:16.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.384+0000 7f46ae5ba700 1 --2- 172.21.15.16:0/623604240 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f468806b140 0x7f468806d600 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:16.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.384+0000 7f46ae5ba700 1 --2- 172.21.15.16:0/623604240 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f46a815d8d0 0x7f46a809f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:16.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.384+0000 7f46ae5ba700 1 --2- 172.21.15.16:0/623604240 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f46a80d0750 0x7f46a815aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:16.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.384+0000 7f46ae5ba700 1 --2- 172.21.15.16:0/623604240 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46a80d41c0 0x7f46a809db00 unknown :-1 s=CLOSED pgs=2697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:16.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.384+0000 7f46ae5ba700 1 -- 172.21.15.16:0/623604240 >> 172.21.15.16:0/623604240 conn(0x7f46a80d3630 msgr2=0x7f46a80d0020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:16.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.384+0000 7f46ae5ba700 1 -- 172.21.15.16:0/623604240 shutdown_connections 2023-10-16T13:47:16.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.384+0000 7f46ae5ba700 1 -- 172.21.15.16:0/623604240 wait complete. 2023-10-16T13:47:16.401 INFO:tasks.cephfs_test_runner:Test command succes with --fs argument for non-default fs ... ok 2023-10-16T13:47:16.402 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_without_fs_argument (tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument) Test command fails without --fs argument in presence of multiple fs 2023-10-16T13:47:16.402 INFO:tasks.cephfs_test_runner:test_snap_schedule_without_fs_argument (tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument) 2023-10-16T13:47:16.403 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:47:16.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.533+0000 7f1a29835700 1 Processor -- start 2023-10-16T13:47:16.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.533+0000 7f1a29835700 1 -- start start 2023-10-16T13:47:16.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.534+0000 7f1a29835700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1a2415d8d0 0x7f1a240d1c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:16.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.534+0000 7f1a29835700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f1a240d0750 0x7f1a2406c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:16.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.534+0000 7f1a29835700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a2406ba00 con 0x7f1a240d41c0 2023-10-16T13:47:16.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.534+0000 7f1a29835700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a240d2150 con 0x7f1a2415d8d0 2023-10-16T13:47:16.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.534+0000 7f1a29835700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a240d26d0 con 0x7f1a240d0750 2023-10-16T13:47:16.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.534+0000 7f1a22ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1a2415d8d0 0x7f1a240d1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:16.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.534+0000 7f1a227fc700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f1a240d41c0 0x7f1a240d0040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44426/0 (socket says 172.21.15.16:44426) 2023-10-16T13:47:16.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.534+0000 7f1a227fc700 1 -- 172.21.15.16:0/3359978646 learned_addr learned my addr 172.21.15.16:0/3359978646 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:47:16.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.534+0000 7f1a22ffd700 1 --2- 172.21.15.16:0/3359978646 >> v2:172.21.15.16:3300/0 conn(0x7f1a2415d8d0 0x7f1a240d1c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55152/0 (socket says 172.21.15.16:55152) 2023-10-16T13:47:16.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.534+0000 7f1a237fe700 1 --2- 172.21.15.16:0/3359978646 >> v2:172.21.15.49:3300/0 conn(0x7f1a240d0750 0x7f1a2406c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:16.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.534+0000 7f1a22ffd700 1 -- 172.21.15.16:0/3359978646 >> v2:172.21.15.49:3300/0 conn(0x7f1a240d0750 msgr2=0x7f1a2406c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:16.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.534+0000 7f1a22ffd700 1 --2- 172.21.15.16:0/3359978646 >> v2:172.21.15.49:3300/0 conn(0x7f1a240d0750 0x7f1a2406c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:16.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.534+0000 7f1a22ffd700 1 -- 172.21.15.16:0/3359978646 >> v1:172.21.15.16:6789/0 conn(0x7f1a240d41c0 legacy=0x7f1a240d0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:16.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.534+0000 7f1a22ffd700 1 -- 172.21.15.16:0/3359978646 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a240d23f0 con 0x7f1a2415d8d0 2023-10-16T13:47:16.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.534+0000 7f1a21ffb700 1 -- 172.21.15.16:0/3359978646 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 911557334 0 0) 0x7f1a2406ba00 con 0x7f1a240d41c0 2023-10-16T13:47:16.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.534+0000 7f1a22ffd700 1 --2- 172.21.15.16:0/3359978646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a2415d8d0 0x7f1a240d1c10 secure :-1 s=READY pgs=2699 cs=0 l=1 rev1=1 crypto rx=0x7f1a240d4640 tx=0x7f1a14009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45e484aa23f2a56e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:16.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.534+0000 7f1a21ffb700 1 -- 172.21.15.16:0/3359978646 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a14011830 con 0x7f1a2415d8d0 2023-10-16T13:47:16.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.534+0000 7f1a21ffb700 1 -- 172.21.15.16:0/3359978646 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a14011990 con 0x7f1a2415d8d0 2023-10-16T13:47:16.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.534+0000 7f1a21ffb700 1 -- 172.21.15.16:0/3359978646 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a14011d20 con 0x7f1a2415d8d0 2023-10-16T13:47:16.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.535+0000 7f1a29835700 1 -- 172.21.15.16:0/3359978646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a2415d8d0 msgr2=0x7f1a240d1c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:16.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.535+0000 7f1a29835700 1 --2- 172.21.15.16:0/3359978646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a2415d8d0 0x7f1a240d1c10 secure :-1 s=READY pgs=2699 cs=0 l=1 rev1=1 crypto rx=0x7f1a240d4640 tx=0x7f1a14009b20 comp rx=0 tx=0).stop 2023-10-16T13:47:16.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.535+0000 7f1a29835700 1 -- 172.21.15.16:0/3359978646 shutdown_connections 2023-10-16T13:47:16.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.535+0000 7f1a29835700 1 --2- 172.21.15.16:0/3359978646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a2415d8d0 0x7f1a240d1c10 unknown :-1 s=CLOSED pgs=2699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:16.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.535+0000 7f1a29835700 1 --2- 172.21.15.16:0/3359978646 >> v2:172.21.15.49:3300/0 conn(0x7f1a240d0750 0x7f1a2406c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:16.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.535+0000 7f1a29835700 1 -- 172.21.15.16:0/3359978646 >> 172.21.15.16:0/3359978646 conn(0x7f1a240d3630 msgr2=0x7f1a240d3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:16.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.535+0000 7f1a29835700 1 -- 172.21.15.16:0/3359978646 shutdown_connections 2023-10-16T13:47:16.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.535+0000 7f1a29835700 1 -- 172.21.15.16:0/3359978646 wait complete. 2023-10-16T13:47:16.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.536+0000 7f1a29835700 1 Processor -- start 2023-10-16T13:47:16.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.536+0000 7f1a29835700 1 -- start start 2023-10-16T13:47:16.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.537+0000 7f1a29835700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1a240d0750 0x7f1a241e6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:16.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.537+0000 7f1a29835700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1a240d41c0 0x7f1a2415a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:16.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.537+0000 7f1a29835700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a2415d8d0 0x7f1a240cc4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:16.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.537+0000 7f1a29835700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a2415aa00 con 0x7f1a2415d8d0 2023-10-16T13:47:16.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.537+0000 7f1a29835700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a240ccfc0 con 0x7f1a240d0750 2023-10-16T13:47:16.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.537+0000 7f1a29835700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a240cca10 con 0x7f1a240d41c0 2023-10-16T13:47:16.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.537+0000 7f1a237fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a2415d8d0 0x7f1a240cc4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:16.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.537+0000 7f1a237fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a2415d8d0 0x7f1a240cc4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55158/0 (socket says 172.21.15.16:55158) 2023-10-16T13:47:16.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.537+0000 7f1a237fe700 1 -- 172.21.15.16:0/2913792152 learned_addr learned my addr 172.21.15.16:0/2913792152 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:16.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.537+0000 7f1a237fe700 1 -- 172.21.15.16:0/2913792152 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1a240d41c0 msgr2=0x7f1a2415a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:16.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.537+0000 7f1a237fe700 1 --2- 172.21.15.16:0/2913792152 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1a240d41c0 0x7f1a2415a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:16.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.537+0000 7f1a237fe700 1 -- 172.21.15.16:0/2913792152 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1a240d0750 msgr2=0x7f1a241e6bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:16.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.537+0000 7f1a237fe700 1 --2- 172.21.15.16:0/2913792152 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1a240d0750 0x7f1a241e6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:16.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.538+0000 7f1a237fe700 1 -- 172.21.15.16:0/2913792152 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a140094a0 con 0x7f1a2415d8d0 2023-10-16T13:47:16.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.538+0000 7f1a237fe700 1 --2- 172.21.15.16:0/2913792152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a2415d8d0 0x7f1a240cc4d0 secure :-1 s=READY pgs=2700 cs=0 l=1 rev1=1 crypto rx=0x7f1a1c00efc0 tx=0x7f1a1c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:16.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.538+0000 7f1a0bfff700 1 -- 172.21.15.16:0/2913792152 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a1c00cd40 con 0x7f1a2415d8d0 2023-10-16T13:47:16.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.538+0000 7f1a0bfff700 1 -- 172.21.15.16:0/2913792152 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a1c00f040 con 0x7f1a2415d8d0 2023-10-16T13:47:16.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.538+0000 7f1a29835700 1 -- 172.21.15.16:0/2913792152 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a240ccce0 con 0x7f1a2415d8d0 2023-10-16T13:47:16.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.538+0000 7f1a0bfff700 1 -- 172.21.15.16:0/2913792152 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a1c0146c0 con 0x7f1a2415d8d0 2023-10-16T13:47:16.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.538+0000 7f1a29835700 1 -- 172.21.15.16:0/2913792152 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a241e63d0 con 0x7f1a2415d8d0 2023-10-16T13:47:16.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.539+0000 7f1a29835700 1 -- 172.21.15.16:0/2913792152 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a2404ebb0 con 0x7f1a2415d8d0 2023-10-16T13:47:16.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.539+0000 7f1a0bfff700 1 -- 172.21.15.16:0/2913792152 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 148) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f1a1c014820 con 0x7f1a2415d8d0 2023-10-16T13:47:16.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.540+0000 7f1a0bfff700 1 --2- 172.21.15.16:0/2913792152 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f1a1006b140 0x7f1a1006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:16.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.540+0000 7f1a0bfff700 1 -- 172.21.15.16:0/2913792152 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 6294+0+0 (secure 0 0 0) 0x7f1a1c08aa30 con 0x7f1a2415d8d0 2023-10-16T13:47:16.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.540+0000 7f1a22ffd700 1 --2- 172.21.15.16:0/2913792152 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f1a1006b140 0x7f1a1006d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:16.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.540+0000 7f1a22ffd700 1 --2- 172.21.15.16:0/2913792152 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f1a1006b140 0x7f1a1006d600 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1a140090b0 tx=0x7f1a14010000 comp rx=0 tx=0).ready entity=mgr.11886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:16.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.544+0000 7f1a0bfff700 1 -- 172.21.15.16:0/2913792152 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1a1c0554b0 con 0x7f1a2415d8d0 2023-10-16T13:47:16.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.750+0000 7f1a29835700 1 -- 172.21.15.16:0/2913792152 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1a240b1620 con 0x7f1a2415d8d0 2023-10-16T13:47:16.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.751+0000 7f1a0bfff700 1 -- 172.21.15.16:0/2913792152 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v237) v1 ==== 74+0+14617 (secure 0 0 0) 0x7f1a1c018090 con 0x7f1a2415d8d0 2023-10-16T13:47:16.754 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:47:16.754 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":237,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:47:14.374769+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6835/3860406853":"2023-10-17T13:47:11.140567+0000","172.21.15.49:6835/2905760119":"2023-10-17T13:47:04.337914+0000","172.21.15.16:0/1827947757":"2023-10-17T13:47:01.801871+0000","172.21.15.16:0/1694897016":"2023-10-17T13:47:01.801871+0000","172.21.15.16:0/3760456275":"2023-10-17T13:47:01.801871+0000","172.21.15.16:6837/3860406853":"2023-10-17T13:47:11.140567+0000","172.21.15.16:0/3651530055":"2023-10-17T13:47:01.801871+0000","172.21.15.16:0/850397235":"2023-10-17T13:47:01.801871+0000","172.21.15.16:6833/1506142504":"2023-10-17T13:47:01.801871+0000","172.21.15.16:0/4239949371":"2023-10-17T13:47:01.801871+0000","172.21.15.16:0/2321929682":"2023-10-17T13:46:51.165018+0000","172.21.15.16:0/3155275276":"2023-10-17T13:47:01.801871+0000","172.21.15.16:0/3429506235":"2023-10-17T13:46:51.165018+0000","172.21.15.49:6833/2905760119":"2023-10-17T13:47:04.337914+0000","172.21.15.16:0/4090083700":"2023-10-17T13:47:01.801871+0000","172.21.15.16:0/3843126404":"2023-10-17T13:46:51.165018+0000","172.21.15.16:6832/1506142504":"2023-10-17T13:47:01.801871+0000","172.21.15.16:0/1439318665":"2023-10-17T13:46:51.165018+0000","172.21.15.16:0/4204021371":"2023-10-17T13:46:51.165018+0000","172.21.15.16:6833/3171422553":"2023-10-17T13:46:51.165018+0000","172.21.15.16:6832/3171422553":"2023-10-17T13:46:51.165018+0000","172.21.15.16:0/1691810194":"2023-10-17T13:47:01.801871+0000","172.21.15.16:0/1632287797":"2023-10-17T13:46:51.165018+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:47:16.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.753+0000 7f1a29835700 1 -- 172.21.15.16:0/2913792152 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f1a1006b140 msgr2=0x7f1a1006d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:16.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.753+0000 7f1a29835700 1 --2- 172.21.15.16:0/2913792152 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f1a1006b140 0x7f1a1006d600 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1a140090b0 tx=0x7f1a14010000 comp rx=0 tx=0).stop 2023-10-16T13:47:16.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.753+0000 7f1a29835700 1 -- 172.21.15.16:0/2913792152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a2415d8d0 msgr2=0x7f1a240cc4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:16.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.753+0000 7f1a29835700 1 --2- 172.21.15.16:0/2913792152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a2415d8d0 0x7f1a240cc4d0 secure :-1 s=READY pgs=2700 cs=0 l=1 rev1=1 crypto rx=0x7f1a1c00efc0 tx=0x7f1a1c00c3a0 comp rx=0 tx=0).stop 2023-10-16T13:47:16.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.753+0000 7f1a29835700 1 -- 172.21.15.16:0/2913792152 shutdown_connections 2023-10-16T13:47:16.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.753+0000 7f1a29835700 1 --2- 172.21.15.16:0/2913792152 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f1a1006b140 0x7f1a1006d600 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:16.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.753+0000 7f1a29835700 1 --2- 172.21.15.16:0/2913792152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a2415d8d0 0x7f1a240cc4d0 unknown :-1 s=CLOSED pgs=2700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:16.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.753+0000 7f1a29835700 1 --2- 172.21.15.16:0/2913792152 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1a240d0750 0x7f1a241e6bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:16.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.753+0000 7f1a29835700 1 --2- 172.21.15.16:0/2913792152 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1a240d41c0 0x7f1a2415a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:16.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.753+0000 7f1a29835700 1 -- 172.21.15.16:0/2913792152 >> 172.21.15.16:0/2913792152 conn(0x7f1a240d3630 msgr2=0x7f1a2415c950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:16.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.753+0000 7f1a29835700 1 -- 172.21.15.16:0/2913792152 shutdown_connections 2023-10-16T13:47:16.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.753+0000 7f1a29835700 1 -- 172.21.15.16:0/2913792152 wait complete. 2023-10-16T13:47:16.768 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T13:47:16.769 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T13:47:16.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.895+0000 7f66af66c700 1 Processor -- start 2023-10-16T13:47:16.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.895+0000 7f66af66c700 1 -- start start 2023-10-16T13:47:16.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.896+0000 7f66af66c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f66a815b2c0 0x7f66a8161340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:16.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.896+0000 7f66af66c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f66a806e570 0x7f66a815e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:16.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.896+0000 7f66af66c700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f66a806a7e0 0x7f66a806ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:16.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.896+0000 7f66af66c700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66a806b1b0 con 0x7f66a815b2c0 2023-10-16T13:47:16.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.896+0000 7f66af66c700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66a8161880 con 0x7f66a806a7e0 2023-10-16T13:47:16.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.896+0000 7f66af66c700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66a815b6a0 con 0x7f66a806e570 2023-10-16T13:47:16.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.896+0000 7f66ad408700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f66a815b2c0 0x7f66a8161340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:16.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.896+0000 7f66ad408700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f66a815b2c0 0x7f66a8161340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55164/0 (socket says 172.21.15.16:55164) 2023-10-16T13:47:16.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.896+0000 7f66ad408700 1 -- 172.21.15.16:0/3632763604 learned_addr learned my addr 172.21.15.16:0/3632763604 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:16.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.896+0000 7f66adc09700 1 --2- 172.21.15.16:0/3632763604 >> v2:172.21.15.49:3300/0 conn(0x7f66a806a7e0 0x7f66a806ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:16.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.896+0000 7f66acc07700 1 --2- 172.21.15.16:0/3632763604 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f66a806e570 0x7f66a815e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:16.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.896+0000 7f66ad408700 1 -- 172.21.15.16:0/3632763604 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f66a806e570 msgr2=0x7f66a815e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:16.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.896+0000 7f66ad408700 1 --2- 172.21.15.16:0/3632763604 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f66a806e570 0x7f66a815e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:16.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.896+0000 7f66ad408700 1 -- 172.21.15.16:0/3632763604 >> v2:172.21.15.49:3300/0 conn(0x7f66a806a7e0 msgr2=0x7f66a806ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:16.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.896+0000 7f66ad408700 1 --2- 172.21.15.16:0/3632763604 >> v2:172.21.15.49:3300/0 conn(0x7f66a806a7e0 0x7f66a806ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:16.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.896+0000 7f66ad408700 1 -- 172.21.15.16:0/3632763604 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66a8161b20 con 0x7f66a815b2c0 2023-10-16T13:47:16.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.896+0000 7f66ad408700 1 --2- 172.21.15.16:0/3632763604 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66a815b2c0 0x7f66a8161340 secure :-1 s=READY pgs=2701 cs=0 l=1 rev1=1 crypto rx=0x7f66a80ca970 tx=0x7f66a4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f60a039e3ed56c9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:16.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.897+0000 7f669ffff700 1 -- 172.21.15.16:0/3632763604 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66a4011830 con 0x7f66a815b2c0 2023-10-16T13:47:16.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.897+0000 7f669ffff700 1 -- 172.21.15.16:0/3632763604 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66a4011990 con 0x7f66a815b2c0 2023-10-16T13:47:16.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.897+0000 7f669ffff700 1 -- 172.21.15.16:0/3632763604 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66a4011d20 con 0x7f66a815b2c0 2023-10-16T13:47:16.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.897+0000 7f66af66c700 1 -- 172.21.15.16:0/3632763604 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66a815b2c0 msgr2=0x7f66a8161340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:16.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.897+0000 7f66af66c700 1 --2- 172.21.15.16:0/3632763604 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66a815b2c0 0x7f66a8161340 secure :-1 s=READY pgs=2701 cs=0 l=1 rev1=1 crypto rx=0x7f66a80ca970 tx=0x7f66a4009bd0 comp rx=0 tx=0).stop 2023-10-16T13:47:16.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.897+0000 7f66af66c700 1 -- 172.21.15.16:0/3632763604 shutdown_connections 2023-10-16T13:47:16.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.897+0000 7f66af66c700 1 --2- 172.21.15.16:0/3632763604 >> v2:172.21.15.49:3300/0 conn(0x7f66a806a7e0 0x7f66a806ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:16.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.897+0000 7f66af66c700 1 --2- 172.21.15.16:0/3632763604 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66a815b2c0 0x7f66a8161340 unknown :-1 s=CLOSED pgs=2701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:16.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.897+0000 7f66af66c700 1 --2- 172.21.15.16:0/3632763604 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f66a806e570 0x7f66a815e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:16.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.897+0000 7f66af66c700 1 -- 172.21.15.16:0/3632763604 >> 172.21.15.16:0/3632763604 conn(0x7f66a80b1f40 msgr2=0x7f66a80b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:16.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.897+0000 7f66af66c700 1 -- 172.21.15.16:0/3632763604 shutdown_connections 2023-10-16T13:47:16.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.897+0000 7f66af66c700 1 -- 172.21.15.16:0/3632763604 wait complete. 2023-10-16T13:47:16.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.898+0000 7f66af66c700 1 Processor -- start 2023-10-16T13:47:16.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.898+0000 7f66af66c700 1 -- start start 2023-10-16T13:47:16.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.899+0000 7f66af66c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f66a806a7e0 0x7f66a80b5b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:16.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.899+0000 7f66af66c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f66a806e570 0x7f66a80c6d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:16.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.899+0000 7f66af66c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66a815b2c0 0x7f66a80b3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:16.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.899+0000 7f66af66c700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66a80c7280 con 0x7f66a815b2c0 2023-10-16T13:47:16.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.899+0000 7f66af66c700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66a80b6150 con 0x7f66a806a7e0 2023-10-16T13:47:16.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.899+0000 7f66af66c700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66a80c8510 con 0x7f66a806e570 2023-10-16T13:47:16.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.899+0000 7f66adc09700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66a815b2c0 0x7f66a80b3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:16.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.899+0000 7f66adc09700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66a815b2c0 0x7f66a80b3660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55174/0 (socket says 172.21.15.16:55174) 2023-10-16T13:47:16.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.899+0000 7f66adc09700 1 -- 172.21.15.16:0/267856205 learned_addr learned my addr 172.21.15.16:0/267856205 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:16.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.900+0000 7f66acc07700 1 --2- 172.21.15.16:0/267856205 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f66a806e570 0x7f66a80c6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:16.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.900+0000 7f66ad408700 1 --2- 172.21.15.16:0/267856205 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f66a806a7e0 0x7f66a80b5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:16.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.900+0000 7f66adc09700 1 -- 172.21.15.16:0/267856205 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f66a806e570 msgr2=0x7f66a80c6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:16.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.900+0000 7f66adc09700 1 --2- 172.21.15.16:0/267856205 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f66a806e570 0x7f66a80c6d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:16.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.900+0000 7f66adc09700 1 -- 172.21.15.16:0/267856205 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f66a806a7e0 msgr2=0x7f66a80b5b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:16.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.900+0000 7f66adc09700 1 --2- 172.21.15.16:0/267856205 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f66a806a7e0 0x7f66a80b5b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:16.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.900+0000 7f66adc09700 1 -- 172.21.15.16:0/267856205 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66a4009550 con 0x7f66a815b2c0 2023-10-16T13:47:16.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.900+0000 7f66ad408700 1 --2- 172.21.15.16:0/267856205 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f66a806a7e0 0x7f66a80b5b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:47:16.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.900+0000 7f66adc09700 1 --2- 172.21.15.16:0/267856205 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66a815b2c0 0x7f66a80b3660 secure :-1 s=READY pgs=2702 cs=0 l=1 rev1=1 crypto rx=0x7f66a000efc0 tx=0x7f66a000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:16.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.900+0000 7f669dffb700 1 -- 172.21.15.16:0/267856205 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66a0009ba0 con 0x7f66a815b2c0 2023-10-16T13:47:16.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.900+0000 7f669dffb700 1 -- 172.21.15.16:0/267856205 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66a000f040 con 0x7f66a815b2c0 2023-10-16T13:47:16.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.900+0000 7f66af66c700 1 -- 172.21.15.16:0/267856205 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66a8152e60 con 0x7f66a815b2c0 2023-10-16T13:47:16.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.900+0000 7f669dffb700 1 -- 172.21.15.16:0/267856205 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66a0014480 con 0x7f66a815b2c0 2023-10-16T13:47:16.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.900+0000 7f66af66c700 1 -- 172.21.15.16:0/267856205 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66a8153420 con 0x7f66a815b2c0 2023-10-16T13:47:16.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.901+0000 7f66af66c700 1 -- 172.21.15.16:0/267856205 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66a80c8730 con 0x7f66a815b2c0 2023-10-16T13:47:16.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.902+0000 7f669dffb700 1 -- 172.21.15.16:0/267856205 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 148) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f66a00145e0 con 0x7f66a815b2c0 2023-10-16T13:47:16.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.902+0000 7f669dffb700 1 --2- 172.21.15.16:0/267856205 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f668c06b140 0x7f668c06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:16.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.902+0000 7f669dffb700 1 -- 172.21.15.16:0/267856205 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 6294+0+0 (secure 0 0 0) 0x7f66a008a5e0 con 0x7f66a815b2c0 2023-10-16T13:47:16.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.902+0000 7f66ad408700 1 --2- 172.21.15.16:0/267856205 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f668c06b140 0x7f668c06d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:16.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.903+0000 7f66ad408700 1 --2- 172.21.15.16:0/267856205 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f668c06b140 0x7f668c06d600 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f66a4009fd0 tx=0x7f66a4008030 comp rx=0 tx=0).ready entity=mgr.11886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:16.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:16.907+0000 7f669dffb700 1 -- 172.21.15.16:0/267856205 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f66a80c8730 con 0x7f66a815b2c0 2023-10-16T13:47:17.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.109+0000 7f66af66c700 1 -- 172.21.15.16:0/267856205 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f66a80c86d0 con 0x7f66a815b2c0 2023-10-16T13:47:17.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.110+0000 7f669dffb700 1 -- 172.21.15.16:0/267856205 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v237) v1 ==== 93+0+10 (secure 0 0 0) 0x7f66a0054fd0 con 0x7f66a815b2c0 2023-10-16T13:47:17.113 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T13:47:17.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.112+0000 7f66af66c700 1 -- 172.21.15.16:0/267856205 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f668c06b140 msgr2=0x7f668c06d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:17.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.112+0000 7f66af66c700 1 --2- 172.21.15.16:0/267856205 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f668c06b140 0x7f668c06d600 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f66a4009fd0 tx=0x7f66a4008030 comp rx=0 tx=0).stop 2023-10-16T13:47:17.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.112+0000 7f66af66c700 1 -- 172.21.15.16:0/267856205 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66a815b2c0 msgr2=0x7f66a80b3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:17.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.112+0000 7f66af66c700 1 --2- 172.21.15.16:0/267856205 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66a815b2c0 0x7f66a80b3660 secure :-1 s=READY pgs=2702 cs=0 l=1 rev1=1 crypto rx=0x7f66a000efc0 tx=0x7f66a000c3a0 comp rx=0 tx=0).stop 2023-10-16T13:47:17.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.112+0000 7f66af66c700 1 -- 172.21.15.16:0/267856205 shutdown_connections 2023-10-16T13:47:17.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.112+0000 7f66af66c700 1 --2- 172.21.15.16:0/267856205 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f668c06b140 0x7f668c06d600 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:17.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.112+0000 7f66af66c700 1 --2- 172.21.15.16:0/267856205 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66a815b2c0 0x7f66a80b3660 unknown :-1 s=CLOSED pgs=2702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:17.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.112+0000 7f66af66c700 1 --2- 172.21.15.16:0/267856205 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f66a806a7e0 0x7f66a80b5b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:17.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.112+0000 7f66af66c700 1 --2- 172.21.15.16:0/267856205 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f66a806e570 0x7f66a80c6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:17.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.112+0000 7f66af66c700 1 -- 172.21.15.16:0/267856205 >> 172.21.15.16:0/267856205 conn(0x7f66a80b1f40 msgr2=0x7f66a8161120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:17.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.112+0000 7f66af66c700 1 -- 172.21.15.16:0/267856205 shutdown_connections 2023-10-16T13:47:17.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.112+0000 7f66af66c700 1 -- 172.21.15.16:0/267856205 wait complete. 2023-10-16T13:47:17.127 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument' 2023-10-16T13:47:17.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.250+0000 7fea1b475700 1 Processor -- start 2023-10-16T13:47:17.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.250+0000 7fea1b475700 1 -- start start 2023-10-16T13:47:17.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.251+0000 7fea1b475700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fea140cf4a0 0x7fea140726e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:17.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.251+0000 7fea1b475700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fea140b5a00 0x7fea1415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:17.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.251+0000 7fea1b475700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea1415bc80 con 0x7fea140cf880 2023-10-16T13:47:17.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.251+0000 7fea1b475700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea140b5e60 con 0x7fea140b5a00 2023-10-16T13:47:17.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.251+0000 7fea1b475700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea14072c20 con 0x7fea140cf4a0 2023-10-16T13:47:17.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.251+0000 7fea18a10700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fea140cf880 0x7fea140b52f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44454/0 (socket says 172.21.15.16:44454) 2023-10-16T13:47:17.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.251+0000 7fea18a10700 1 -- 172.21.15.16:0/1355338378 learned_addr learned my addr 172.21.15.16:0/1355338378 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:47:17.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.251+0000 7fea19a12700 1 --2- 172.21.15.16:0/1355338378 >> v2:172.21.15.49:3300/0 conn(0x7fea140b5a00 0x7fea1415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:17.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.251+0000 7fea19211700 1 --2- 172.21.15.16:0/1355338378 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fea140cf4a0 0x7fea140726e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:17.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.251+0000 7fea0bfff700 1 -- 172.21.15.16:0/1355338378 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 911964729 0 0) 0x7fea1415bc80 con 0x7fea140cf880 2023-10-16T13:47:17.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.251+0000 7fea0bfff700 1 -- 172.21.15.16:0/1355338378 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea000035b0 con 0x7fea140cf880 2023-10-16T13:47:17.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.252+0000 7fea19211700 1 -- 172.21.15.16:0/1355338378 >> v2:172.21.15.49:3300/0 conn(0x7fea140b5a00 msgr2=0x7fea1415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:17.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.252+0000 7fea19211700 1 --2- 172.21.15.16:0/1355338378 >> v2:172.21.15.49:3300/0 conn(0x7fea140b5a00 0x7fea1415a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:17.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.252+0000 7fea19211700 1 -- 172.21.15.16:0/1355338378 >> v1:172.21.15.16:6789/0 conn(0x7fea140cf880 legacy=0x7fea140b52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:17.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.252+0000 7fea19211700 1 -- 172.21.15.16:0/1355338378 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea1415b070 con 0x7fea140cf4a0 2023-10-16T13:47:17.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.252+0000 7fea19a12700 1 --2- 172.21.15.16:0/1355338378 >> v2:172.21.15.49:3300/0 conn(0x7fea140b5a00 0x7fea1415a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:47:17.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.252+0000 7fea19211700 1 --2- 172.21.15.16:0/1355338378 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fea140cf4a0 0x7fea140726e0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fea04009840 tx=0x7fea04009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3a447ff06545cc16 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:17.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.252+0000 7fea0bfff700 1 -- 172.21.15.16:0/1355338378 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea04011830 con 0x7fea140cf4a0 2023-10-16T13:47:17.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.252+0000 7fea0bfff700 1 -- 172.21.15.16:0/1355338378 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea04011990 con 0x7fea140cf4a0 2023-10-16T13:47:17.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.252+0000 7fea0bfff700 1 -- 172.21.15.16:0/1355338378 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea04011d20 con 0x7fea140cf4a0 2023-10-16T13:47:17.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.253+0000 7fea1b475700 1 -- 172.21.15.16:0/1355338378 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fea140cf4a0 msgr2=0x7fea140726e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:17.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.253+0000 7fea1b475700 1 --2- 172.21.15.16:0/1355338378 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fea140cf4a0 0x7fea140726e0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fea04009840 tx=0x7fea04009b50 comp rx=0 tx=0).stop 2023-10-16T13:47:17.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.253+0000 7fea1b475700 1 -- 172.21.15.16:0/1355338378 shutdown_connections 2023-10-16T13:47:17.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.253+0000 7fea1b475700 1 --2- 172.21.15.16:0/1355338378 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fea140cf4a0 0x7fea140726e0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:17.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.253+0000 7fea1b475700 1 --2- 172.21.15.16:0/1355338378 >> v2:172.21.15.49:3300/0 conn(0x7fea140b5a00 0x7fea1415a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:17.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.253+0000 7fea1b475700 1 -- 172.21.15.16:0/1355338378 >> 172.21.15.16:0/1355338378 conn(0x7fea140b1fd0 msgr2=0x7fea140b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:17.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.253+0000 7fea1b475700 1 -- 172.21.15.16:0/1355338378 shutdown_connections 2023-10-16T13:47:17.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.253+0000 7fea1b475700 1 -- 172.21.15.16:0/1355338378 wait complete. 2023-10-16T13:47:17.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.254+0000 7fea1b475700 1 Processor -- start 2023-10-16T13:47:17.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.254+0000 7fea1b475700 1 -- start start 2023-10-16T13:47:17.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.255+0000 7fea1b475700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fea140b5a00 0x7fea140c68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:17.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.255+0000 7fea1b475700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fea140cf4a0 0x7fea1414b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:17.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.255+0000 7fea1b475700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea140cf880 0x7fea140ce2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:17.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.255+0000 7fea1b475700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea140b4f00 con 0x7fea140cf880 2023-10-16T13:47:17.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.255+0000 7fea1b475700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea140b42f0 con 0x7fea140cf4a0 2023-10-16T13:47:17.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.255+0000 7fea1b475700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea140b45d0 con 0x7fea140b5a00 2023-10-16T13:47:17.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.255+0000 7fea19a12700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea140cf880 0x7fea140ce2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:17.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.255+0000 7fea19a12700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea140cf880 0x7fea140ce2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55188/0 (socket says 172.21.15.16:55188) 2023-10-16T13:47:17.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.255+0000 7fea19a12700 1 -- 172.21.15.16:0/3130824789 learned_addr learned my addr 172.21.15.16:0/3130824789 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:17.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.255+0000 7fea19211700 1 --2- 172.21.15.16:0/3130824789 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fea140b5a00 0x7fea140c68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:17.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.255+0000 7fea18a10700 1 --2- 172.21.15.16:0/3130824789 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fea140cf4a0 0x7fea1414b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:17.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.255+0000 7fea19a12700 1 -- 172.21.15.16:0/3130824789 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fea140b5a00 msgr2=0x7fea140c68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:17.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.255+0000 7fea19a12700 1 --2- 172.21.15.16:0/3130824789 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fea140b5a00 0x7fea140c68d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:17.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.255+0000 7fea19a12700 1 -- 172.21.15.16:0/3130824789 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fea140cf4a0 msgr2=0x7fea1414b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:17.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.255+0000 7fea19a12700 1 --2- 172.21.15.16:0/3130824789 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fea140cf4a0 0x7fea1414b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:17.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.255+0000 7fea19a12700 1 -- 172.21.15.16:0/3130824789 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea0c009a70 con 0x7fea140cf880 2023-10-16T13:47:17.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.255+0000 7fea19211700 1 --2- 172.21.15.16:0/3130824789 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fea140b5a00 0x7fea140c68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:47:17.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.255+0000 7fea18a10700 1 --2- 172.21.15.16:0/3130824789 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fea140cf4a0 0x7fea1414b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:47:17.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.255+0000 7fea19a12700 1 --2- 172.21.15.16:0/3130824789 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea140cf880 0x7fea140ce2c0 secure :-1 s=READY pgs=2704 cs=0 l=1 rev1=1 crypto rx=0x7fea0c00efc0 tx=0x7fea0c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:17.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.255+0000 7fea09ffb700 1 -- 172.21.15.16:0/3130824789 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea0c009d70 con 0x7fea140cf880 2023-10-16T13:47:17.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.255+0000 7fea09ffb700 1 -- 172.21.15.16:0/3130824789 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea0c00f040 con 0x7fea140cf880 2023-10-16T13:47:17.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.255+0000 7fea1b475700 1 -- 172.21.15.16:0/3130824789 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea040094a0 con 0x7fea140cf880 2023-10-16T13:47:17.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.255+0000 7fea09ffb700 1 -- 172.21.15.16:0/3130824789 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea0c00ce00 con 0x7fea140cf880 2023-10-16T13:47:17.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.256+0000 7fea1b475700 1 -- 172.21.15.16:0/3130824789 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea140b4b90 con 0x7fea140cf880 2023-10-16T13:47:17.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.256+0000 7fea1b475700 1 -- 172.21.15.16:0/3130824789 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea1404ebb0 con 0x7fea140cf880 2023-10-16T13:47:17.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.257+0000 7fea09ffb700 1 -- 172.21.15.16:0/3130824789 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 148) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fea0c014480 con 0x7fea140cf880 2023-10-16T13:47:17.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.257+0000 7fea09ffb700 1 --2- 172.21.15.16:0/3130824789 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fe9f806b140 0x7fe9f806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:17.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.257+0000 7fea09ffb700 1 -- 172.21.15.16:0/3130824789 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 6294+0+0 (secure 0 0 0) 0x7fea0c08a550 con 0x7fea140cf880 2023-10-16T13:47:17.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.257+0000 7fea19211700 1 --2- 172.21.15.16:0/3130824789 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fe9f806b140 0x7fe9f806d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:17.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.258+0000 7fea19211700 1 --2- 172.21.15.16:0/3130824789 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fe9f806b140 0x7fe9f806d600 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fea040090b0 tx=0x7fea04008030 comp rx=0 tx=0).ready entity=mgr.11886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:17.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.262+0000 7fea09ffb700 1 -- 172.21.15.16:0/3130824789 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fea0c054ec0 con 0x7fea140cf880 2023-10-16T13:47:17.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:17.553+0000 7fea1b475700 1 -- 172.21.15.16:0/3130824789 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument"]} v 0) v1 -- 0x7fea140aff70 con 0x7fea140cf880 2023-10-16T13:47:18.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.386+0000 7fea09ffb700 1 -- 172.21.15.16:0/3130824789 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument"]}]=0 v3178) v1 ==== 190+0+0 (secure 0 0 0) 0x7fea0c0584e0 con 0x7fea140cf880 2023-10-16T13:47:18.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.391+0000 7fea1b475700 1 -- 172.21.15.16:0/3130824789 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fe9f806b140 msgr2=0x7fe9f806d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:18.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.391+0000 7fea1b475700 1 --2- 172.21.15.16:0/3130824789 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fe9f806b140 0x7fe9f806d600 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fea040090b0 tx=0x7fea04008030 comp rx=0 tx=0).stop 2023-10-16T13:47:18.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.391+0000 7fea1b475700 1 -- 172.21.15.16:0/3130824789 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea140cf880 msgr2=0x7fea140ce2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:18.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.391+0000 7fea1b475700 1 --2- 172.21.15.16:0/3130824789 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea140cf880 0x7fea140ce2c0 secure :-1 s=READY pgs=2704 cs=0 l=1 rev1=1 crypto rx=0x7fea0c00efc0 tx=0x7fea0c00c3a0 comp rx=0 tx=0).stop 2023-10-16T13:47:18.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.391+0000 7fea1b475700 1 -- 172.21.15.16:0/3130824789 shutdown_connections 2023-10-16T13:47:18.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.391+0000 7fea1b475700 1 --2- 172.21.15.16:0/3130824789 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fe9f806b140 0x7fe9f806d600 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:18.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.391+0000 7fea1b475700 1 --2- 172.21.15.16:0/3130824789 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea140cf880 0x7fea140ce2c0 unknown :-1 s=CLOSED pgs=2704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:18.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.391+0000 7fea1b475700 1 --2- 172.21.15.16:0/3130824789 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fea140b5a00 0x7fea140c68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:18.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.391+0000 7fea1b475700 1 --2- 172.21.15.16:0/3130824789 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fea140cf4a0 0x7fea1414b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:18.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.391+0000 7fea1b475700 1 -- 172.21.15.16:0/3130824789 >> 172.21.15.16:0/3130824789 conn(0x7fea140b1fd0 msgr2=0x7fea140b1800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:18.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.391+0000 7fea1b475700 1 -- 172.21.15.16:0/3130824789 shutdown_connections 2023-10-16T13:47:18.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.391+0000 7fea1b475700 1 -- 172.21.15.16:0/3130824789 wait complete. 2023-10-16T13:47:18.408 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-16T13:47:18.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.542+0000 7fd2536fa700 1 Processor -- start 2023-10-16T13:47:18.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.542+0000 7fd2536fa700 1 -- start start 2023-10-16T13:47:18.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.543+0000 7fd2536fa700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd24c15b2c0 0x7fd24c161340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:18.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.543+0000 7fd2536fa700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fd24c06e570 0x7fd24c15e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:18.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.543+0000 7fd2536fa700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd24c161880 con 0x7fd24c06a7e0 2023-10-16T13:47:18.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.543+0000 7fd2536fa700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd24c15b6a0 con 0x7fd24c06e570 2023-10-16T13:47:18.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.543+0000 7fd2536fa700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd24c161b20 con 0x7fd24c15b2c0 2023-10-16T13:47:18.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.543+0000 7fd251c97700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd24c06a7e0 0x7fd24c071a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44468/0 (socket says 172.21.15.16:44468) 2023-10-16T13:47:18.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.543+0000 7fd251c97700 1 -- 172.21.15.16:0/1293795730 learned_addr learned my addr 172.21.15.16:0/1293795730 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:47:18.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.544+0000 7fd250c95700 1 --2- 172.21.15.16:0/1293795730 >> v2:172.21.15.49:3300/0 conn(0x7fd24c06e570 0x7fd24c15e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:18.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.544+0000 7fd243fff700 1 -- 172.21.15.16:0/1293795730 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4021966561 0 0) 0x7fd24c161880 con 0x7fd24c06a7e0 2023-10-16T13:47:18.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.544+0000 7fd251496700 1 --2- 172.21.15.16:0/1293795730 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd24c15b2c0 0x7fd24c161340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:18.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.544+0000 7fd243fff700 1 -- 172.21.15.16:0/1293795730 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2380035b0 con 0x7fd24c06a7e0 2023-10-16T13:47:18.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.544+0000 7fd243fff700 1 -- 172.21.15.16:0/1293795730 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4036442471 0 0) 0x7fd2380035b0 con 0x7fd24c06a7e0 2023-10-16T13:47:18.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.544+0000 7fd243fff700 1 -- 172.21.15.16:0/1293795730 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd24c15b2c0 msgr2=0x7fd24c161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:18.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.544+0000 7fd243fff700 1 --2- 172.21.15.16:0/1293795730 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd24c15b2c0 0x7fd24c161340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:18.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.544+0000 7fd243fff700 1 -- 172.21.15.16:0/1293795730 >> v2:172.21.15.49:3300/0 conn(0x7fd24c06e570 msgr2=0x7fd24c15e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:18.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.544+0000 7fd243fff700 1 --2- 172.21.15.16:0/1293795730 >> v2:172.21.15.49:3300/0 conn(0x7fd24c06e570 0x7fd24c15e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:18.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.544+0000 7fd243fff700 1 -- 172.21.15.16:0/1293795730 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd24c161e00 con 0x7fd24c06a7e0 2023-10-16T13:47:18.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.544+0000 7fd243fff700 1 -- 172.21.15.16:0/1293795730 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7fd2440019f0 con 0x7fd24c06a7e0 2023-10-16T13:47:18.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.544+0000 7fd243fff700 1 -- 172.21.15.16:0/1293795730 >> v1:172.21.15.16:6789/0 conn(0x7fd24c06a7e0 legacy=0x7fd24c071a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:18.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.544+0000 7fd243fff700 1 --2- 172.21.15.16:0/1293795730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd238003a00 0x7fd238003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:18.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.544+0000 7fd243fff700 1 --2- 172.21.15.16:0/1293795730 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd24c15b2c0 0x7fd238004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:18.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.544+0000 7fd243fff700 1 --2- 172.21.15.16:0/1293795730 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd238004d90 0x7fd238005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:18.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.544+0000 7fd243fff700 1 -- 172.21.15.16:0/1293795730 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd24c15b6a0 con 0x7fd238003a00 2023-10-16T13:47:18.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.544+0000 7fd243fff700 1 -- 172.21.15.16:0/1293795730 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd24c161b20 con 0x7fd238004d90 2023-10-16T13:47:18.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.544+0000 7fd243fff700 1 -- 172.21.15.16:0/1293795730 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd23800ba10 con 0x7fd24c15b2c0 2023-10-16T13:47:18.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.544+0000 7fd251496700 1 --2- 172.21.15.16:0/1293795730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd238003a00 0x7fd238003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:18.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.544+0000 7fd250c95700 1 --2- 172.21.15.16:0/1293795730 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd24c15b2c0 0x7fd238004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:18.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.544+0000 7fd251c97700 1 --2- 172.21.15.16:0/1293795730 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd238004d90 0x7fd238005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:18.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.544+0000 7fd251496700 1 -- 172.21.15.16:0/1293795730 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd24c15b2c0 msgr2=0x7fd238004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:18.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.544+0000 7fd251496700 1 --2- 172.21.15.16:0/1293795730 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd24c15b2c0 0x7fd238004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:18.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.544+0000 7fd251496700 1 -- 172.21.15.16:0/1293795730 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd238004d90 msgr2=0x7fd238005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:18.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.544+0000 7fd251496700 1 --2- 172.21.15.16:0/1293795730 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd238004d90 0x7fd238005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:18.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.544+0000 7fd251496700 1 -- 172.21.15.16:0/1293795730 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2380035b0 con 0x7fd238003a00 2023-10-16T13:47:18.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.545+0000 7fd251496700 1 --2- 172.21.15.16:0/1293795730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd238003a00 0x7fd238003e20 secure :-1 s=READY pgs=2706 cs=0 l=1 rev1=1 crypto rx=0x7fd23c009b60 tx=0x7fd238005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fe031afb03d384a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:18.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.545+0000 7fd243fff700 1 -- 172.21.15.16:0/1293795730 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd23c00e6c0 con 0x7fd238003a00 2023-10-16T13:47:18.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.545+0000 7fd243fff700 1 -- 172.21.15.16:0/1293795730 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd23c00e820 con 0x7fd238003a00 2023-10-16T13:47:18.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.545+0000 7fd243fff700 1 -- 172.21.15.16:0/1293795730 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd23c00eb90 con 0x7fd238003a00 2023-10-16T13:47:18.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.545+0000 7fd2536fa700 1 -- 172.21.15.16:0/1293795730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd238003a00 msgr2=0x7fd238003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:18.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.545+0000 7fd2536fa700 1 --2- 172.21.15.16:0/1293795730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd238003a00 0x7fd238003e20 secure :-1 s=READY pgs=2706 cs=0 l=1 rev1=1 crypto rx=0x7fd23c009b60 tx=0x7fd238005880 comp rx=0 tx=0).stop 2023-10-16T13:47:18.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.545+0000 7fd251c97700 1 -- 172.21.15.16:0/1293795730 reap_dead start 2023-10-16T13:47:18.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.545+0000 7fd2536fa700 1 -- 172.21.15.16:0/1293795730 shutdown_connections 2023-10-16T13:47:18.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.545+0000 7fd2536fa700 1 -- 172.21.15.16:0/1293795730 >> 172.21.15.16:0/1293795730 conn(0x7fd24c0b1f40 msgr2=0x7fd24c0b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:18.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.545+0000 7fd2536fa700 1 -- 172.21.15.16:0/1293795730 shutdown_connections 2023-10-16T13:47:18.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.545+0000 7fd2536fa700 1 -- 172.21.15.16:0/1293795730 wait complete. 2023-10-16T13:47:18.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.547+0000 7fd2536fa700 1 Processor -- start 2023-10-16T13:47:18.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.547+0000 7fd2536fa700 1 -- start start 2023-10-16T13:47:18.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.547+0000 7fd2536fa700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd238003a00 0x7fd24c14f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:18.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.547+0000 7fd2536fa700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd24c0ca0d0 0x7fd24c15eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:18.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.547+0000 7fd2536fa700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd24c1f43f0 0x7fd24c1601b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:18.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.547+0000 7fd2536fa700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd24c14fec0 con 0x7fd238003a00 2023-10-16T13:47:18.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.547+0000 7fd2536fa700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd24c0c74c0 con 0x7fd24c1f43f0 2023-10-16T13:47:18.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.547+0000 7fd2536fa700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd24c0c77a0 con 0x7fd24c0ca0d0 2023-10-16T13:47:18.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.548+0000 7fd251496700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd238003a00 0x7fd24c14f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:18.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.548+0000 7fd251496700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd238003a00 0x7fd24c14f980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55204/0 (socket says 172.21.15.16:55204) 2023-10-16T13:47:18.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.548+0000 7fd251496700 1 -- 172.21.15.16:0/3432406927 learned_addr learned my addr 172.21.15.16:0/3432406927 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:18.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.548+0000 7fd251c97700 1 --2- 172.21.15.16:0/3432406927 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd24c1f43f0 0x7fd24c1601b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:18.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.548+0000 7fd250c95700 1 --2- 172.21.15.16:0/3432406927 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd24c0ca0d0 0x7fd24c15eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:18.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.548+0000 7fd251496700 1 -- 172.21.15.16:0/3432406927 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd24c0ca0d0 msgr2=0x7fd24c15eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:18.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.548+0000 7fd251496700 1 --2- 172.21.15.16:0/3432406927 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd24c0ca0d0 0x7fd24c15eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:18.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.548+0000 7fd251496700 1 -- 172.21.15.16:0/3432406927 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd24c1f43f0 msgr2=0x7fd24c1601b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:18.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.548+0000 7fd251496700 1 --2- 172.21.15.16:0/3432406927 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd24c1f43f0 0x7fd24c1601b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:18.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.548+0000 7fd251496700 1 -- 172.21.15.16:0/3432406927 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd24c0c7a80 con 0x7fd238003a00 2023-10-16T13:47:18.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.548+0000 7fd251496700 1 --2- 172.21.15.16:0/3432406927 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd238003a00 0x7fd24c14f980 secure :-1 s=READY pgs=2707 cs=0 l=1 rev1=1 crypto rx=0x7fd238004730 tx=0x7fd23c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:18.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.548+0000 7fd241ffb700 1 -- 172.21.15.16:0/3432406927 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd23c021070 con 0x7fd238003a00 2023-10-16T13:47:18.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.548+0000 7fd241ffb700 1 -- 172.21.15.16:0/3432406927 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd23c004520 con 0x7fd238003a00 2023-10-16T13:47:18.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.548+0000 7fd241ffb700 1 -- 172.21.15.16:0/3432406927 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd23c00e6c0 con 0x7fd238003a00 2023-10-16T13:47:18.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.548+0000 7fd2536fa700 1 -- 172.21.15.16:0/3432406927 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd24c0c7d60 con 0x7fd238003a00 2023-10-16T13:47:18.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.548+0000 7fd2536fa700 1 -- 172.21.15.16:0/3432406927 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd24c15f3f0 con 0x7fd238003a00 2023-10-16T13:47:18.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.549+0000 7fd2536fa700 1 -- 172.21.15.16:0/3432406927 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd24c04ebb0 con 0x7fd238003a00 2023-10-16T13:47:18.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.550+0000 7fd241ffb700 1 -- 172.21.15.16:0/3432406927 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 148) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fd23c025070 con 0x7fd238003a00 2023-10-16T13:47:18.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.550+0000 7fd241ffb700 1 --2- 172.21.15.16:0/3432406927 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fd23006b140 0x7fd23006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:18.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.550+0000 7fd241ffb700 1 -- 172.21.15.16:0/3432406927 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 6294+0+0 (secure 0 0 0) 0x7fd23c08ac80 con 0x7fd238003a00 2023-10-16T13:47:18.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.550+0000 7fd250c95700 1 --2- 172.21.15.16:0/3432406927 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fd23006b140 0x7fd23006d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:18.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.551+0000 7fd250c95700 1 --2- 172.21.15.16:0/3432406927 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fd23006b140 0x7fd23006d600 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd24c072250 tx=0x7fd248008040 comp rx=0 tx=0).ready entity=mgr.11886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:18.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.555+0000 7fd241ffb700 1 -- 172.21.15.16:0/3432406927 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd23c055810 con 0x7fd238003a00 2023-10-16T13:47:18.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.760+0000 7fd2536fa700 1 -- 172.21.15.16:0/3432406927 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fd24c15f580 con 0x7fd238003a00 2023-10-16T13:47:18.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.763+0000 7fd241ffb700 1 -- 172.21.15.16:0/3432406927 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v82)=0 v82) v1 ==== 122+0+0 (secure 0 0 0) 0x7fd23c02c0c0 con 0x7fd238003a00 2023-10-16T13:47:18.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.765+0000 7fd2536fa700 1 -- 172.21.15.16:0/3432406927 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fd23006b140 msgr2=0x7fd23006d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:18.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.765+0000 7fd2536fa700 1 --2- 172.21.15.16:0/3432406927 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fd23006b140 0x7fd23006d600 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd24c072250 tx=0x7fd248008040 comp rx=0 tx=0).stop 2023-10-16T13:47:18.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.765+0000 7fd2536fa700 1 -- 172.21.15.16:0/3432406927 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd238003a00 msgr2=0x7fd24c14f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:18.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.765+0000 7fd2536fa700 1 --2- 172.21.15.16:0/3432406927 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd238003a00 0x7fd24c14f980 secure :-1 s=READY pgs=2707 cs=0 l=1 rev1=1 crypto rx=0x7fd238004730 tx=0x7fd23c0047c0 comp rx=0 tx=0).stop 2023-10-16T13:47:18.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.765+0000 7fd2536fa700 1 -- 172.21.15.16:0/3432406927 shutdown_connections 2023-10-16T13:47:18.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.765+0000 7fd2536fa700 1 --2- 172.21.15.16:0/3432406927 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fd23006b140 0x7fd23006d600 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:18.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.765+0000 7fd2536fa700 1 --2- 172.21.15.16:0/3432406927 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd24c1f43f0 0x7fd24c1601b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:18.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.765+0000 7fd2536fa700 1 --2- 172.21.15.16:0/3432406927 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd238003a00 0x7fd24c14f980 unknown :-1 s=CLOSED pgs=2707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:18.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.765+0000 7fd2536fa700 1 --2- 172.21.15.16:0/3432406927 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd24c0ca0d0 0x7fd24c15eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:18.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.766+0000 7fd2536fa700 1 -- 172.21.15.16:0/3432406927 >> 172.21.15.16:0/3432406927 conn(0x7fd24c0b1f40 msgr2=0x7fd24c15be80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:18.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.766+0000 7fd2536fa700 1 -- 172.21.15.16:0/3432406927 shutdown_connections 2023-10-16T13:47:18.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:18.766+0000 7fd2536fa700 1 -- 172.21.15.16:0/3432406927 wait complete. 2023-10-16T13:47:18.780 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T13:47:18.781 DEBUG:teuthology.orchestra.run.smithi016:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T13:47:18.826 DEBUG:teuthology.orchestra.run.smithi049:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T13:47:18.876 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T13:47:18.876 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T13:47:18.877 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:47:18.895 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16287364k,nr_inodes=4071841,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20251 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3261172k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T13:47:18.895 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T13:47:18.895 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:47:18.915 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284204k,nr_inodes=4071051,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=753 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260540k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T13:47:18.916 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T13:47:18.916 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:47:18.952 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16287364k,nr_inodes=4071841,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20251 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3261172k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T13:47:18.953 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T13:47:18.953 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:47:18.974 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284204k,nr_inodes=4071051,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=753 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260540k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T13:47:18.975 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-16T13:47:18.975 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:47:19.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.115+0000 7f658df27700 1 Processor -- start 2023-10-16T13:47:19.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.115+0000 7f658df27700 1 -- start start 2023-10-16T13:47:19.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.116+0000 7f658df27700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f65880cf4a0 0x7f65880726e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:19.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.116+0000 7f658df27700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f65880b5a00 0x7f658815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:19.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.116+0000 7f658df27700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f658815bc80 con 0x7f65880cf880 2023-10-16T13:47:19.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.116+0000 7f658df27700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65880b5e70 con 0x7f65880cf4a0 2023-10-16T13:47:19.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.116+0000 7f658df27700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6588072c20 con 0x7f65880b5a00 2023-10-16T13:47:19.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.116+0000 7f65877fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f65880cf4a0 0x7f65880726e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:19.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.116+0000 7f65877fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f65880cf4a0 0x7f65880726e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55212/0 (socket says 172.21.15.16:55212) 2023-10-16T13:47:19.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.116+0000 7f65877fe700 1 -- 172.21.15.16:0/4061454308 learned_addr learned my addr 172.21.15.16:0/4061454308 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:19.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.116+0000 7f6587fff700 1 --2- 172.21.15.16:0/4061454308 >> v2:172.21.15.49:3300/0 conn(0x7f65880b5a00 0x7f658815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:19.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.116+0000 7f65877fe700 1 -- 172.21.15.16:0/4061454308 >> v2:172.21.15.49:3300/0 conn(0x7f65880b5a00 msgr2=0x7f658815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:19.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.116+0000 7f65877fe700 1 --2- 172.21.15.16:0/4061454308 >> v2:172.21.15.49:3300/0 conn(0x7f65880b5a00 0x7f658815a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:19.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.116+0000 7f65877fe700 1 -- 172.21.15.16:0/4061454308 >> v1:172.21.15.49:6789/0 conn(0x7f65880cf880 legacy=0x7f65880b52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:19.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.116+0000 7f65877fe700 1 -- 172.21.15.16:0/4061454308 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f658815b070 con 0x7f65880cf4a0 2023-10-16T13:47:19.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.116+0000 7f65877fe700 1 --2- 172.21.15.16:0/4061454308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65880cf4a0 0x7f65880726e0 secure :-1 s=READY pgs=2708 cs=0 l=1 rev1=1 crypto rx=0x7f65880cdeb0 tx=0x7f6570009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b9146026ed4c4c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:19.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.117+0000 7f65867fc700 1 -- 172.21.15.16:0/4061454308 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6570004230 con 0x7f65880cf4a0 2023-10-16T13:47:19.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.117+0000 7f65867fc700 1 -- 172.21.15.16:0/4061454308 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6570004390 con 0x7f65880cf4a0 2023-10-16T13:47:19.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.117+0000 7f65867fc700 1 -- 172.21.15.16:0/4061454308 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65700036a0 con 0x7f65880cf4a0 2023-10-16T13:47:19.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.117+0000 7f658df27700 1 -- 172.21.15.16:0/4061454308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65880cf4a0 msgr2=0x7f65880726e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:19.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.117+0000 7f658df27700 1 --2- 172.21.15.16:0/4061454308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65880cf4a0 0x7f65880726e0 secure :-1 s=READY pgs=2708 cs=0 l=1 rev1=1 crypto rx=0x7f65880cdeb0 tx=0x7f6570009bd0 comp rx=0 tx=0).stop 2023-10-16T13:47:19.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.117+0000 7f658df27700 1 -- 172.21.15.16:0/4061454308 shutdown_connections 2023-10-16T13:47:19.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.117+0000 7f658df27700 1 --2- 172.21.15.16:0/4061454308 >> v2:172.21.15.49:3300/0 conn(0x7f65880b5a00 0x7f658815a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:19.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.117+0000 7f658df27700 1 --2- 172.21.15.16:0/4061454308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65880cf4a0 0x7f65880726e0 unknown :-1 s=CLOSED pgs=2708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:19.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.117+0000 7f658df27700 1 -- 172.21.15.16:0/4061454308 >> 172.21.15.16:0/4061454308 conn(0x7f65880b1fd0 msgr2=0x7f65880b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:19.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.117+0000 7f658df27700 1 -- 172.21.15.16:0/4061454308 shutdown_connections 2023-10-16T13:47:19.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.117+0000 7f658df27700 1 -- 172.21.15.16:0/4061454308 wait complete. 2023-10-16T13:47:19.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.118+0000 7f658df27700 1 Processor -- start 2023-10-16T13:47:19.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.119+0000 7f658df27700 1 -- start start 2023-10-16T13:47:19.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.119+0000 7f658df27700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65880b5a00 0x7f65880b4260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:19.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.119+0000 7f658df27700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f65880cf4a0 0x7f65880aec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:19.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.119+0000 7f658df27700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f65880cf880 0x7f65880c7840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:19.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.119+0000 7f658df27700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65881f42f0 con 0x7f65880b5a00 2023-10-16T13:47:19.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.119+0000 7f658df27700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65880adea0 con 0x7f65880cf4a0 2023-10-16T13:47:19.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.119+0000 7f658df27700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65880ae180 con 0x7f65880cf880 2023-10-16T13:47:19.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.119+0000 7f65877fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65880b5a00 0x7f65880b4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:19.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.119+0000 7f65877fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65880b5a00 0x7f65880b4260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55218/0 (socket says 172.21.15.16:55218) 2023-10-16T13:47:19.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.119+0000 7f65877fe700 1 -- 172.21.15.16:0/1904948065 learned_addr learned my addr 172.21.15.16:0/1904948065 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:19.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.120+0000 7f65877fe700 1 -- 172.21.15.16:0/1904948065 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f65880cf880 msgr2=0x7f65880c7840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:19.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.120+0000 7f65877fe700 1 --2- 172.21.15.16:0/1904948065 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f65880cf880 0x7f65880c7840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:19.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.120+0000 7f6586ffd700 1 --2- 172.21.15.16:0/1904948065 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f65880cf4a0 0x7f65880aec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:19.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.120+0000 7f65877fe700 1 -- 172.21.15.16:0/1904948065 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f65880cf4a0 msgr2=0x7f65880aec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:19.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.120+0000 7f65877fe700 1 --2- 172.21.15.16:0/1904948065 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f65880cf4a0 0x7f65880aec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:19.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.120+0000 7f65877fe700 1 -- 172.21.15.16:0/1904948065 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6570009550 con 0x7f65880b5a00 2023-10-16T13:47:19.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.120+0000 7f6586ffd700 1 --2- 172.21.15.16:0/1904948065 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f65880cf4a0 0x7f65880aec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:47:19.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.120+0000 7f65877fe700 1 --2- 172.21.15.16:0/1904948065 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65880b5a00 0x7f65880b4260 secure :-1 s=READY pgs=2709 cs=0 l=1 rev1=1 crypto rx=0x7f6570009130 tx=0x7f6570004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:19.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.120+0000 7f657ffff700 1 -- 172.21.15.16:0/1904948065 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6570003d40 con 0x7f65880b5a00 2023-10-16T13:47:19.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.120+0000 7f657ffff700 1 -- 172.21.15.16:0/1904948065 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f657001f070 con 0x7f65880b5a00 2023-10-16T13:47:19.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.120+0000 7f658df27700 1 -- 172.21.15.16:0/1904948065 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65880ae460 con 0x7f65880b5a00 2023-10-16T13:47:19.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.120+0000 7f657ffff700 1 -- 172.21.15.16:0/1904948065 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6570011870 con 0x7f65880b5a00 2023-10-16T13:47:19.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.120+0000 7f658df27700 1 -- 172.21.15.16:0/1904948065 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65880b47a0 con 0x7f65880b5a00 2023-10-16T13:47:19.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.121+0000 7f658df27700 1 -- 172.21.15.16:0/1904948065 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65880ae980 con 0x7f65880b5a00 2023-10-16T13:47:19.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.121+0000 7f657ffff700 1 -- 172.21.15.16:0/1904948065 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 148) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f6570021030 con 0x7f65880b5a00 2023-10-16T13:47:19.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.122+0000 7f657ffff700 1 --2- 172.21.15.16:0/1904948065 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f656806b140 0x7f656806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:19.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.122+0000 7f657ffff700 1 -- 172.21.15.16:0/1904948065 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 6294+0+0 (secure 0 0 0) 0x7f657008b240 con 0x7f65880b5a00 2023-10-16T13:47:19.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.122+0000 7f6586ffd700 1 --2- 172.21.15.16:0/1904948065 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f656806b140 0x7f656806d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:19.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.122+0000 7f6586ffd700 1 --2- 172.21.15.16:0/1904948065 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f656806b140 0x7f656806d600 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6578006fd0 tx=0x7f6578008040 comp rx=0 tx=0).ready entity=mgr.11886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:19.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.126+0000 7f657ffff700 1 -- 172.21.15.16:0/1904948065 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f65880ae980 con 0x7f65880b5a00 2023-10-16T13:47:19.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.374+0000 7f658df27700 1 -- 172.21.15.16:0/1904948065 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f65880ae980 con 0x7f65880b5a00 2023-10-16T13:47:19.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.375+0000 7f657ffff700 1 -- 172.21.15.16:0/1904948065 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+3707 (secure 0 0 0) 0x7f6570029e90 con 0x7f65880b5a00 2023-10-16T13:47:19.378 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:47:19.378 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":123,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9509,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/1167594604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":1167594604},{"type":"v1","addr":"172.21.15.49:6835","nonce":1167594604}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":121},{"gid":10516,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/4065699432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":4065699432},{"type":"v1","addr":"172.21.15.49:6834","nonce":4065699432}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":108},{"gid":11415,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/1373048456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1373048456},{"type":"v1","addr":"172.21.15.16:6836","nonce":1373048456}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":108},{"gid":12159,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2872983530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":2872983530},{"type":"v1","addr":"172.21.15.16:6837","nonce":2872983530}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":123}],"filesystems":[]} 2023-10-16T13:47:19.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.377+0000 7f658df27700 1 -- 172.21.15.16:0/1904948065 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f656806b140 msgr2=0x7f656806d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:19.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.377+0000 7f658df27700 1 --2- 172.21.15.16:0/1904948065 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f656806b140 0x7f656806d600 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6578006fd0 tx=0x7f6578008040 comp rx=0 tx=0).stop 2023-10-16T13:47:19.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.377+0000 7f658df27700 1 -- 172.21.15.16:0/1904948065 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65880b5a00 msgr2=0x7f65880b4260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:19.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.377+0000 7f658df27700 1 --2- 172.21.15.16:0/1904948065 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65880b5a00 0x7f65880b4260 secure :-1 s=READY pgs=2709 cs=0 l=1 rev1=1 crypto rx=0x7f6570009130 tx=0x7f6570004750 comp rx=0 tx=0).stop 2023-10-16T13:47:19.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.377+0000 7f658df27700 1 -- 172.21.15.16:0/1904948065 shutdown_connections 2023-10-16T13:47:19.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.377+0000 7f658df27700 1 --2- 172.21.15.16:0/1904948065 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f656806b140 0x7f656806d600 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:19.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.377+0000 7f658df27700 1 --2- 172.21.15.16:0/1904948065 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f65880cf880 0x7f65880c7840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:19.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.377+0000 7f658df27700 1 --2- 172.21.15.16:0/1904948065 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65880b5a00 0x7f65880b4260 unknown :-1 s=CLOSED pgs=2709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:19.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.377+0000 7f658df27700 1 --2- 172.21.15.16:0/1904948065 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f65880cf4a0 0x7f65880aec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:19.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.377+0000 7f658df27700 1 -- 172.21.15.16:0/1904948065 >> 172.21.15.16:0/1904948065 conn(0x7f65880b1fd0 msgr2=0x7f658815ccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:19.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.377+0000 7f658df27700 1 -- 172.21.15.16:0/1904948065 shutdown_connections 2023-10-16T13:47:19.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.377+0000 7f658df27700 1 -- 172.21.15.16:0/1904948065 wait complete. 2023-10-16T13:47:19.382 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 123 2023-10-16T13:47:19.392 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-16T13:47:19.392 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-16T13:47:19.393 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-16T13:47:19.393 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T13:47:19.393 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-16T13:47:19.394 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-16T13:47:19.394 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-16T13:47:19.394 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T13:47:19.395 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-16T13:47:19.395 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-16T13:47:19.395 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-16T13:47:19.395 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T13:47:19.396 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-16T13:47:19.396 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-16T13:47:19.396 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-16T13:47:19.396 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T13:47:19.403 INFO:tasks.ceph.mds.a:Stopped 2023-10-16T13:47:19.403 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-16T13:47:19.404 INFO:tasks.ceph.mds.c:Stopped 2023-10-16T13:47:19.404 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-16T13:47:19.406 INFO:tasks.ceph.mds.b:Stopped 2023-10-16T13:47:19.407 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-16T13:47:19.407 INFO:tasks.ceph.mds.d:Stopped 2023-10-16T13:47:19.408 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-16T13:47:19.446 INFO:tasks.ceph.mds.a:Started 2023-10-16T13:47:19.447 DEBUG:teuthology.parallel:result is None 2023-10-16T13:47:19.448 INFO:tasks.ceph.mds.c:Started 2023-10-16T13:47:19.448 DEBUG:teuthology.parallel:result is None 2023-10-16T13:47:19.449 INFO:tasks.ceph.mds.b:Started 2023-10-16T13:47:19.449 DEBUG:teuthology.parallel:result is None 2023-10-16T13:47:19.450 INFO:tasks.ceph.mds.d:Started 2023-10-16T13:47:19.450 DEBUG:teuthology.parallel:result is None 2023-10-16T13:47:19.451 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-16T13:47:19.529 INFO:tasks.ceph.mds.a.smithi016.stdout:starting mds.a at 2023-10-16T13:47:19.529 INFO:tasks.ceph.mds.c.smithi016.stdout:starting mds.c at 2023-10-16T13:47:19.540 INFO:tasks.ceph.mds.b.smithi049.stdout:starting mds.b at 2023-10-16T13:47:19.540 INFO:tasks.ceph.mds.d.smithi049.stdout:starting mds.d at 2023-10-16T13:47:19.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.582+0000 7f4d3ae3b700 1 Processor -- start 2023-10-16T13:47:19.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.582+0000 7f4d3ae3b700 1 -- start start 2023-10-16T13:47:19.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.583+0000 7f4d3ae3b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4d340cdfa0 0x7f4d340b2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:19.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.583+0000 7f4d3ae3b700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f4d340d0bb0 0x7f4d340b2980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:19.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.583+0000 7f4d3ae3b700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d340d0fc0 con 0x7f4d340ce380 2023-10-16T13:47:19.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.583+0000 7f4d3ae3b700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d340d1210 con 0x7f4d340cdfa0 2023-10-16T13:47:19.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.583+0000 7f4d3ae3b700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d340cbc30 con 0x7f4d340d0bb0 2023-10-16T13:47:19.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.583+0000 7f4d38bd7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4d340cdfa0 0x7f4d340b2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:19.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.583+0000 7f4d38bd7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4d340cdfa0 0x7f4d340b2440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55250/0 (socket says 172.21.15.16:55250) 2023-10-16T13:47:19.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.583+0000 7f4d38bd7700 1 -- 172.21.15.16:0/95117844 learned_addr learned my addr 172.21.15.16:0/95117844 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:19.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.583+0000 7f4d33fff700 1 --1- 172.21.15.16:0/95117844 >> v1:172.21.15.16:6789/0 conn(0x7f4d340ce380 0x7f4d340d04a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44526/0 (socket says 172.21.15.16:44526) 2023-10-16T13:47:19.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.583+0000 7f4d393d8700 1 --2- 172.21.15.16:0/95117844 >> v2:172.21.15.49:3300/0 conn(0x7f4d340d0bb0 0x7f4d340b2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:19.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.583+0000 7f4d38bd7700 1 -- 172.21.15.16:0/95117844 >> v2:172.21.15.49:3300/0 conn(0x7f4d340d0bb0 msgr2=0x7f4d340b2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:19.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.583+0000 7f4d38bd7700 1 --2- 172.21.15.16:0/95117844 >> v2:172.21.15.49:3300/0 conn(0x7f4d340d0bb0 0x7f4d340b2980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:19.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.583+0000 7f4d38bd7700 1 -- 172.21.15.16:0/95117844 >> v1:172.21.15.16:6789/0 conn(0x7f4d340ce380 legacy=0x7f4d340d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:19.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.583+0000 7f4d38bd7700 1 -- 172.21.15.16:0/95117844 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d340cbf10 con 0x7f4d340cdfa0 2023-10-16T13:47:19.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.584+0000 7f4d38bd7700 1 --2- 172.21.15.16:0/95117844 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d340cdfa0 0x7f4d340b2440 secure :-1 s=READY pgs=2717 cs=0 l=1 rev1=1 crypto rx=0x7f4d340cf870 tx=0x7f4d28009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2287469b34666ee server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:19.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.584+0000 7f4d337fe700 1 -- 172.21.15.16:0/95117844 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d28011830 con 0x7f4d340cdfa0 2023-10-16T13:47:19.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.584+0000 7f4d337fe700 1 -- 172.21.15.16:0/95117844 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d28011990 con 0x7f4d340cdfa0 2023-10-16T13:47:19.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.584+0000 7f4d337fe700 1 -- 172.21.15.16:0/95117844 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d28011d20 con 0x7f4d340cdfa0 2023-10-16T13:47:19.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.584+0000 7f4d3ae3b700 1 -- 172.21.15.16:0/95117844 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d340cdfa0 msgr2=0x7f4d340b2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:19.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.584+0000 7f4d3ae3b700 1 --2- 172.21.15.16:0/95117844 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d340cdfa0 0x7f4d340b2440 secure :-1 s=READY pgs=2717 cs=0 l=1 rev1=1 crypto rx=0x7f4d340cf870 tx=0x7f4d28009bd0 comp rx=0 tx=0).stop 2023-10-16T13:47:19.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.584+0000 7f4d3ae3b700 1 -- 172.21.15.16:0/95117844 shutdown_connections 2023-10-16T13:47:19.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.584+0000 7f4d3ae3b700 1 --2- 172.21.15.16:0/95117844 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d340cdfa0 0x7f4d340b2440 unknown :-1 s=CLOSED pgs=2717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:19.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.584+0000 7f4d3ae3b700 1 --2- 172.21.15.16:0/95117844 >> v2:172.21.15.49:3300/0 conn(0x7f4d340d0bb0 0x7f4d340b2980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:19.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.584+0000 7f4d3ae3b700 1 -- 172.21.15.16:0/95117844 >> 172.21.15.16:0/95117844 conn(0x7f4d340d14b0 msgr2=0x7f4d340d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:19.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.584+0000 7f4d3ae3b700 1 -- 172.21.15.16:0/95117844 shutdown_connections 2023-10-16T13:47:19.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.584+0000 7f4d3ae3b700 1 -- 172.21.15.16:0/95117844 wait complete. 2023-10-16T13:47:19.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.586+0000 7f4d3ae3b700 1 Processor -- start 2023-10-16T13:47:19.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.586+0000 7f4d3ae3b700 1 -- start start 2023-10-16T13:47:19.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.587+0000 7f4d3ae3b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d340cdfa0 0x7f4d3414f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:19.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.587+0000 7f4d3ae3b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4d340ce380 0x7f4d3409b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:19.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.587+0000 7f4d3ae3b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4d340d0bb0 0x7f4d3409b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:19.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.587+0000 7f4d3ae3b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d3414f540 con 0x7f4d340cdfa0 2023-10-16T13:47:19.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.587+0000 7f4d3ae3b700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d340b3140 con 0x7f4d340ce380 2023-10-16T13:47:19.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.587+0000 7f4d3ae3b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d340b3420 con 0x7f4d340d0bb0 2023-10-16T13:47:19.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.587+0000 7f4d38bd7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d340cdfa0 0x7f4d3414f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:19.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.587+0000 7f4d38bd7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d340cdfa0 0x7f4d3414f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55256/0 (socket says 172.21.15.16:55256) 2023-10-16T13:47:19.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.587+0000 7f4d38bd7700 1 -- 172.21.15.16:0/2239240935 learned_addr learned my addr 172.21.15.16:0/2239240935 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:19.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.588+0000 7f4d33fff700 1 --2- 172.21.15.16:0/2239240935 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4d340ce380 0x7f4d3409b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:19.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.588+0000 7f4d393d8700 1 --2- 172.21.15.16:0/2239240935 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4d340d0bb0 0x7f4d3409b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:19.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.588+0000 7f4d38bd7700 1 -- 172.21.15.16:0/2239240935 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4d340d0bb0 msgr2=0x7f4d3409b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:19.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.588+0000 7f4d38bd7700 1 --2- 172.21.15.16:0/2239240935 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4d340d0bb0 0x7f4d3409b770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:19.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.588+0000 7f4d38bd7700 1 -- 172.21.15.16:0/2239240935 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4d340ce380 msgr2=0x7f4d3409b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:19.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.588+0000 7f4d38bd7700 1 --2- 172.21.15.16:0/2239240935 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4d340ce380 0x7f4d3409b230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:19.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.588+0000 7f4d38bd7700 1 -- 172.21.15.16:0/2239240935 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d28009550 con 0x7f4d340cdfa0 2023-10-16T13:47:19.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.588+0000 7f4d33fff700 1 --2- 172.21.15.16:0/2239240935 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4d340ce380 0x7f4d3409b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:47:19.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.588+0000 7f4d38bd7700 1 --2- 172.21.15.16:0/2239240935 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d340cdfa0 0x7f4d3414f000 secure :-1 s=READY pgs=2718 cs=0 l=1 rev1=1 crypto rx=0x7f4d28000c00 tx=0x7f4d28004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:19.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.588+0000 7f4d317fa700 1 -- 172.21.15.16:0/2239240935 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d28004570 con 0x7f4d340cdfa0 2023-10-16T13:47:19.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.588+0000 7f4d317fa700 1 -- 172.21.15.16:0/2239240935 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d2801f070 con 0x7f4d340cdfa0 2023-10-16T13:47:19.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.588+0000 7f4d3ae3b700 1 -- 172.21.15.16:0/2239240935 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d340b3700 con 0x7f4d340cdfa0 2023-10-16T13:47:19.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.588+0000 7f4d317fa700 1 -- 172.21.15.16:0/2239240935 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d280038e0 con 0x7f4d340cdfa0 2023-10-16T13:47:19.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.588+0000 7f4d3ae3b700 1 -- 172.21.15.16:0/2239240935 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d3409d2b0 con 0x7f4d340cdfa0 2023-10-16T13:47:19.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.589+0000 7f4d3ae3b700 1 -- 172.21.15.16:0/2239240935 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d34066970 con 0x7f4d340cdfa0 2023-10-16T13:47:19.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.589+0000 7f4d317fa700 1 -- 172.21.15.16:0/2239240935 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 148) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f4d28021030 con 0x7f4d340cdfa0 2023-10-16T13:47:19.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.590+0000 7f4d317fa700 1 --2- 172.21.15.16:0/2239240935 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f4d1806b140 0x7f4d1806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:19.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.590+0000 7f4d317fa700 1 -- 172.21.15.16:0/2239240935 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 6294+0+0 (secure 0 0 0) 0x7f4d2808b110 con 0x7f4d340cdfa0 2023-10-16T13:47:19.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.590+0000 7f4d33fff700 1 --2- 172.21.15.16:0/2239240935 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f4d1806b140 0x7f4d1806d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:19.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.590+0000 7f4d33fff700 1 --2- 172.21.15.16:0/2239240935 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f4d1806b140 0x7f4d1806d600 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4d340d1f40 tx=0x7f4d20009040 comp rx=0 tx=0).ready entity=mgr.11886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:19.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.594+0000 7f4d317fa700 1 -- 172.21.15.16:0/2239240935 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4d28055ca0 con 0x7f4d340cdfa0 2023-10-16T13:47:19.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:19.816+0000 7f4d3ae3b700 1 -- 172.21.15.16:0/2239240935 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f4d3404ebb0 con 0x7f4d340cdfa0 2023-10-16T13:47:20.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.398+0000 7f4d317fa700 1 -- 172.21.15.16:0/2239240935 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v239) v1 ==== 97+0+0 (secure 0 0 0) 0x7f4d280592c0 con 0x7f4d340cdfa0 2023-10-16T13:47:20.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.402+0000 7f4d3ae3b700 1 -- 172.21.15.16:0/2239240935 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f4d1806b140 msgr2=0x7f4d1806d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:20.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.402+0000 7f4d3ae3b700 1 --2- 172.21.15.16:0/2239240935 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f4d1806b140 0x7f4d1806d600 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4d340d1f40 tx=0x7f4d20009040 comp rx=0 tx=0).stop 2023-10-16T13:47:20.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.402+0000 7f4d3ae3b700 1 -- 172.21.15.16:0/2239240935 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d340cdfa0 msgr2=0x7f4d3414f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:20.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.402+0000 7f4d3ae3b700 1 --2- 172.21.15.16:0/2239240935 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d340cdfa0 0x7f4d3414f000 secure :-1 s=READY pgs=2718 cs=0 l=1 rev1=1 crypto rx=0x7f4d28000c00 tx=0x7f4d28004750 comp rx=0 tx=0).stop 2023-10-16T13:47:20.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.404+0000 7f4d3ae3b700 1 -- 172.21.15.16:0/2239240935 shutdown_connections 2023-10-16T13:47:20.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.404+0000 7f4d3ae3b700 1 --2- 172.21.15.16:0/2239240935 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f4d1806b140 0x7f4d1806d600 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:20.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.404+0000 7f4d3ae3b700 1 --2- 172.21.15.16:0/2239240935 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4d340d0bb0 0x7f4d3409b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:20.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.404+0000 7f4d3ae3b700 1 --2- 172.21.15.16:0/2239240935 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d340cdfa0 0x7f4d3414f000 unknown :-1 s=CLOSED pgs=2718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:20.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.404+0000 7f4d3ae3b700 1 --2- 172.21.15.16:0/2239240935 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4d340ce380 0x7f4d3409b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:20.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.404+0000 7f4d3ae3b700 1 -- 172.21.15.16:0/2239240935 >> 172.21.15.16:0/2239240935 conn(0x7f4d340d14b0 msgr2=0x7f4d340b6750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:20.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.404+0000 7f4d3ae3b700 1 -- 172.21.15.16:0/2239240935 shutdown_connections 2023-10-16T13:47:20.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.404+0000 7f4d3ae3b700 1 -- 172.21.15.16:0/2239240935 wait complete. 2023-10-16T13:47:20.408 INFO:teuthology.orchestra.run.smithi016.stderr: removed all blocklist entries 2023-10-16T13:47:20.421 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-16T13:47:20.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.546+0000 7fec23766700 1 Processor -- start 2023-10-16T13:47:20.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.546+0000 7fec23766700 1 -- start start 2023-10-16T13:47:20.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.547+0000 7fec23766700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fec1c15b630 0x7fec1c0d0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:20.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.547+0000 7fec23766700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fec1c15ba10 0x7fec1c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:20.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.547+0000 7fec23766700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fec1c0cdde0 0x7fec1c0ce210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:20.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.547+0000 7fec23766700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec1c0ce750 con 0x7fec1c15b630 2023-10-16T13:47:20.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.547+0000 7fec23766700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec1c0b0970 con 0x7fec1c15ba10 2023-10-16T13:47:20.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.547+0000 7fec23766700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec1c0cd690 con 0x7fec1c0cdde0 2023-10-16T13:47:20.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.547+0000 7fec21502700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fec1c15b630 0x7fec1c0d0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:20.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.547+0000 7fec21502700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fec1c15b630 0x7fec1c0d0380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55266/0 (socket says 172.21.15.16:55266) 2023-10-16T13:47:20.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.547+0000 7fec21502700 1 -- 172.21.15.16:0/1847272150 learned_addr learned my addr 172.21.15.16:0/1847272150 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:20.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.547+0000 7fec20d01700 1 --2- 172.21.15.16:0/1847272150 >> v2:172.21.15.49:3300/0 conn(0x7fec1c15ba10 0x7fec1c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:20.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.547+0000 7fec21d03700 1 --2- 172.21.15.16:0/1847272150 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fec1c0cdde0 0x7fec1c0ce210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:20.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.547+0000 7fec21502700 1 -- 172.21.15.16:0/1847272150 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fec1c0cdde0 msgr2=0x7fec1c0ce210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:20.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.547+0000 7fec21502700 1 --2- 172.21.15.16:0/1847272150 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fec1c0cdde0 0x7fec1c0ce210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:20.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.547+0000 7fec21502700 1 -- 172.21.15.16:0/1847272150 >> v2:172.21.15.49:3300/0 conn(0x7fec1c15ba10 msgr2=0x7fec1c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:20.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.547+0000 7fec21502700 1 --2- 172.21.15.16:0/1847272150 >> v2:172.21.15.49:3300/0 conn(0x7fec1c15ba10 0x7fec1c0cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:20.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.547+0000 7fec21502700 1 -- 172.21.15.16:0/1847272150 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec1c0cd970 con 0x7fec1c15b630 2023-10-16T13:47:20.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.547+0000 7fec21502700 1 --2- 172.21.15.16:0/1847272150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec1c15b630 0x7fec1c0d0380 secure :-1 s=READY pgs=2719 cs=0 l=1 rev1=1 crypto rx=0x7fec1c0b2f50 tx=0x7fec18009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=652973f074514508 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:20.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.547+0000 7fec13fff700 1 -- 172.21.15.16:0/1847272150 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec1800b870 con 0x7fec1c15b630 2023-10-16T13:47:20.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.547+0000 7fec13fff700 1 -- 172.21.15.16:0/1847272150 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec1800b9d0 con 0x7fec1c15b630 2023-10-16T13:47:20.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.547+0000 7fec13fff700 1 -- 172.21.15.16:0/1847272150 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec1800bd60 con 0x7fec1c15b630 2023-10-16T13:47:20.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.548+0000 7fec23766700 1 -- 172.21.15.16:0/1847272150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec1c15b630 msgr2=0x7fec1c0d0380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:20.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.548+0000 7fec23766700 1 --2- 172.21.15.16:0/1847272150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec1c15b630 0x7fec1c0d0380 secure :-1 s=READY pgs=2719 cs=0 l=1 rev1=1 crypto rx=0x7fec1c0b2f50 tx=0x7fec18009bd0 comp rx=0 tx=0).stop 2023-10-16T13:47:20.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.548+0000 7fec23766700 1 -- 172.21.15.16:0/1847272150 shutdown_connections 2023-10-16T13:47:20.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.548+0000 7fec23766700 1 --2- 172.21.15.16:0/1847272150 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fec1c0cdde0 0x7fec1c0ce210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:20.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.548+0000 7fec23766700 1 --2- 172.21.15.16:0/1847272150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec1c15b630 0x7fec1c0d0380 unknown :-1 s=CLOSED pgs=2719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:20.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.548+0000 7fec23766700 1 --2- 172.21.15.16:0/1847272150 >> v2:172.21.15.49:3300/0 conn(0x7fec1c15ba10 0x7fec1c0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:20.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.548+0000 7fec23766700 1 -- 172.21.15.16:0/1847272150 >> 172.21.15.16:0/1847272150 conn(0x7fec1c0d14b0 msgr2=0x7fec1c0d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:20.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.548+0000 7fec23766700 1 -- 172.21.15.16:0/1847272150 shutdown_connections 2023-10-16T13:47:20.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.548+0000 7fec23766700 1 -- 172.21.15.16:0/1847272150 wait complete. 2023-10-16T13:47:20.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.550+0000 7fec23766700 1 Processor -- start 2023-10-16T13:47:20.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.550+0000 7fec23766700 1 -- start start 2023-10-16T13:47:20.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.551+0000 7fec23766700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fec1c0cdde0 0x7fec1c0af340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:20.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.551+0000 7fec23766700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fec1c15b630 0x7fec1c0b5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:20.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.551+0000 7fec23766700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec1c15ba10 0x7fec1c0b5a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:20.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.551+0000 7fec23766700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec1c15d5c0 con 0x7fec1c15ba10 2023-10-16T13:47:20.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.551+0000 7fec23766700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec1c15e420 con 0x7fec1c15b630 2023-10-16T13:47:20.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.551+0000 7fec23766700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec1c15d810 con 0x7fec1c0cdde0 2023-10-16T13:47:20.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.551+0000 7fec21d03700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec1c15ba10 0x7fec1c0b5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:20.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.551+0000 7fec21d03700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec1c15ba10 0x7fec1c0b5a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55276/0 (socket says 172.21.15.16:55276) 2023-10-16T13:47:20.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.551+0000 7fec21d03700 1 -- 172.21.15.16:0/1655716776 learned_addr learned my addr 172.21.15.16:0/1655716776 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:20.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.551+0000 7fec20d01700 1 --2- 172.21.15.16:0/1655716776 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fec1c15b630 0x7fec1c0b5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:20.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.551+0000 7fec21502700 1 --2- 172.21.15.16:0/1655716776 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fec1c0cdde0 0x7fec1c0af340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:20.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.551+0000 7fec21d03700 1 -- 172.21.15.16:0/1655716776 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fec1c0cdde0 msgr2=0x7fec1c0af340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:20.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.551+0000 7fec21d03700 1 --2- 172.21.15.16:0/1655716776 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fec1c0cdde0 0x7fec1c0af340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:20.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.551+0000 7fec21d03700 1 -- 172.21.15.16:0/1655716776 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fec1c15b630 msgr2=0x7fec1c0b5500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:20.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.551+0000 7fec21d03700 1 --2- 172.21.15.16:0/1655716776 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fec1c15b630 0x7fec1c0b5500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:20.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.551+0000 7fec21d03700 1 -- 172.21.15.16:0/1655716776 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec18009550 con 0x7fec1c15ba10 2023-10-16T13:47:20.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.551+0000 7fec21502700 1 --2- 172.21.15.16:0/1655716776 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fec1c0cdde0 0x7fec1c0af340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:47:20.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.551+0000 7fec20d01700 1 --2- 172.21.15.16:0/1655716776 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fec1c15b630 0x7fec1c0b5500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:47:20.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.551+0000 7fec21d03700 1 --2- 172.21.15.16:0/1655716776 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec1c15ba10 0x7fec1c0b5a40 secure :-1 s=READY pgs=2720 cs=0 l=1 rev1=1 crypto rx=0x7fec1400ccf0 tx=0x7fec1400a570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:20.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.551+0000 7fec11ffb700 1 -- 172.21.15.16:0/1655716776 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec14003a20 con 0x7fec1c15ba10 2023-10-16T13:47:20.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.551+0000 7fec11ffb700 1 -- 172.21.15.16:0/1655716776 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec14003b80 con 0x7fec1c15ba10 2023-10-16T13:47:20.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.551+0000 7fec23766700 1 -- 172.21.15.16:0/1655716776 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec1c15daf0 con 0x7fec1c15ba10 2023-10-16T13:47:20.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.551+0000 7fec11ffb700 1 -- 172.21.15.16:0/1655716776 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec14004020 con 0x7fec1c15ba10 2023-10-16T13:47:20.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.552+0000 7fec23766700 1 -- 172.21.15.16:0/1655716776 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec1c15e0b0 con 0x7fec1c15ba10 2023-10-16T13:47:20.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.553+0000 7fec23766700 1 -- 172.21.15.16:0/1655716776 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec1c04ebb0 con 0x7fec1c15ba10 2023-10-16T13:47:20.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.553+0000 7fec11ffb700 1 -- 172.21.15.16:0/1655716776 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 148) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fec14004180 con 0x7fec1c15ba10 2023-10-16T13:47:20.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.553+0000 7fec11ffb700 1 --2- 172.21.15.16:0/1655716776 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fec0006b140 0x7fec0006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:20.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.553+0000 7fec11ffb700 1 -- 172.21.15.16:0/1655716776 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(241..241 src has 1..241) v4 ==== 5305+0+0 (secure 0 0 0) 0x7fec14012070 con 0x7fec1c15ba10 2023-10-16T13:47:20.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.553+0000 7fec21502700 1 --2- 172.21.15.16:0/1655716776 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fec0006b140 0x7fec0006d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:20.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.554+0000 7fec21502700 1 --2- 172.21.15.16:0/1655716776 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fec0006b140 0x7fec0006d600 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fec18000c00 tx=0x7fec18008030 comp rx=0 tx=0).ready entity=mgr.11886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:20.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.558+0000 7fec11ffb700 1 -- 172.21.15.16:0/1655716776 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fec14054f00 con 0x7fec1c15ba10 2023-10-16T13:47:20.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.843+0000 7fec23766700 1 -- 172.21.15.16:0/1655716776 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fec1c06b5d0 con 0x7fec1c15ba10 2023-10-16T13:47:20.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.844+0000 7fec11ffb700 1 -- 172.21.15.16:0/1655716776 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v56) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fec14058520 con 0x7fec1c15ba10 2023-10-16T13:47:20.847 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:47:20.847 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-10-16T13:47:20.847 INFO:teuthology.orchestra.run.smithi016.stdout: "auth_dump": [ 2023-10-16T13:47:20.847 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:47:20.848 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.a", 2023-10-16T13:47:20.848 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOKC1l/rBtOhAA/ob5WUvpz8Glp4E+xPZOYA==", 2023-10-16T13:47:20.848 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:47:20.848 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T13:47:20.848 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-10-16T13:47:20.849 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-10-16T13:47:20.849 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:47:20.849 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:47:20.849 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:47:20.849 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:47:20.850 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.b", 2023-10-16T13:47:20.850 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1lKAVsCxAAvhXuLY6i1+Iz1yq5BVPMNQ==", 2023-10-16T13:47:20.850 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:47:20.850 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T13:47:20.850 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-10-16T13:47:20.851 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-10-16T13:47:20.851 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:47:20.851 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:47:20.851 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:47:20.851 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:47:20.852 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.c", 2023-10-16T13:47:20.852 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1lvBowBhAA3waJNYhdgs8PGCYqYNYUog==", 2023-10-16T13:47:20.852 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:47:20.852 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T13:47:20.852 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-10-16T13:47:20.853 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-10-16T13:47:20.853 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:47:20.853 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:47:20.853 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:47:20.853 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:47:20.853 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.d", 2023-10-16T13:47:20.854 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1lS/fnEhAAfNYgQ5BW86JMjYm/bBfnaQ==", 2023-10-16T13:47:20.854 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:47:20.854 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T13:47:20.854 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-10-16T13:47:20.854 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-10-16T13:47:20.855 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:47:20.855 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:47:20.855 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:47:20.855 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:47:20.855 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.0", 2023-10-16T13:47:20.855 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAQKC1lbxllOhAAZeLdp2GMl6HqQPaQjKlNqw==", 2023-10-16T13:47:20.856 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:47:20.856 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T13:47:20.856 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T13:47:20.856 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:47:20.856 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:47:20.857 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:47:20.857 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:47:20.857 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.1", 2023-10-16T13:47:20.857 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQATKC1lnU6KOhAAHxvlbmzBptf1keGgL+Rbkg==", 2023-10-16T13:47:20.857 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:47:20.858 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T13:47:20.858 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T13:47:20.858 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:47:20.858 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:47:20.859 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:47:20.859 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:47:20.859 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.2", 2023-10-16T13:47:20.859 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAXKC1l9jFyARAAr/NAwtWhBMut9sAAoDEGWg==", 2023-10-16T13:47:20.859 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:47:20.859 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T13:47:20.860 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T13:47:20.860 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:47:20.860 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:47:20.860 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:47:20.860 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:47:20.861 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.3", 2023-10-16T13:47:20.861 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAaKC1lbw/lARAAGJ/9DNb5/7kLFSq7vrLtXg==", 2023-10-16T13:47:20.861 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:47:20.861 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T13:47:20.861 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T13:47:20.861 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:47:20.862 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:47:20.862 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:47:20.862 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:47:20.862 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.4", 2023-10-16T13:47:20.862 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAeKC1lZb7jERAAYpvb1xcSli9CTq7L8ChLOA==", 2023-10-16T13:47:20.863 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:47:20.863 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T13:47:20.863 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T13:47:20.863 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:47:20.863 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:47:20.863 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:47:20.864 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:47:20.864 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.5", 2023-10-16T13:47:20.864 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAhKC1l/4HuEBAAqfnRqjknij9KES/fqiyIBQ==", 2023-10-16T13:47:20.864 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:47:20.864 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T13:47:20.864 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T13:47:20.865 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:47:20.865 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:47:20.865 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:47:20.865 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:47:20.865 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.6", 2023-10-16T13:47:20.865 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAkKC1lzoUkEBAA+Q82OiBMNag8+R316WQtUw==", 2023-10-16T13:47:20.866 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:47:20.866 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T13:47:20.866 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T13:47:20.866 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:47:20.866 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:47:20.867 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:47:20.867 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:47:20.867 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.7", 2023-10-16T13:47:20.867 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAnKC1ljUu4DxAAc+G4anJANC2GhLJ8wb6gYg==", 2023-10-16T13:47:20.867 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:47:20.867 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T13:47:20.868 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T13:47:20.868 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:47:20.868 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:47:20.868 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:47:20.868 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:47:20.868 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.0", 2023-10-16T13:47:20.869 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1l2l6PFxAA9QyXtBAHdGAXpwH0JfknuQ==", 2023-10-16T13:47:20.869 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:47:20.869 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T13:47:20.869 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow r", 2023-10-16T13:47:20.869 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-16T13:47:20.870 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:47:20.870 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:47:20.870 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:47:20.870 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.1", 2023-10-16T13:47:20.870 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1lAcHFIxAAcNt2bmv2R6j5L3DYkdw7eQ==", 2023-10-16T13:47:20.870 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:47:20.871 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T13:47:20.871 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-10-16T13:47:20.871 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-10-16T13:47:20.871 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-10-16T13:47:20.871 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:47:20.872 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:47:20.872 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:47:20.872 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.2", 2023-10-16T13:47:20.872 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1lA6wnGxAAdtaqYhtIJu8qENMrD8xvVA==", 2023-10-16T13:47:20.872 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:47:20.872 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T13:47:20.873 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-10-16T13:47:20.873 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-10-16T13:47:20.873 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-10-16T13:47:20.873 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:47:20.873 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:47:20.874 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:47:20.874 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.3", 2023-10-16T13:47:20.874 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1l/ddAJxAA3fNKKn9XkXas1BlDN+ggjw==", 2023-10-16T13:47:20.874 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:47:20.874 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T13:47:20.874 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-10-16T13:47:20.875 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-10-16T13:47:20.875 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-10-16T13:47:20.875 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:47:20.875 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:47:20.875 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:47:20.876 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.admin", 2023-10-16T13:47:20.877 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOKC1ld5jUHBAAc6esnCce3dSEZn3wVjhIlQ==", 2023-10-16T13:47:20.877 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:47:20.877 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2023-10-16T13:47:20.877 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-10-16T13:47:20.878 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-10-16T13:47:20.878 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:47:20.878 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:47:20.878 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:47:20.878 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:47:20.879 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mgr.x", 2023-10-16T13:47:20.879 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOKC1l6b3vNhAAmj5xbfid2sZM2KfHmAf1Wg==", 2023-10-16T13:47:20.879 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:47:20.879 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2023-10-16T13:47:20.879 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile mgr", 2023-10-16T13:47:20.879 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:47:20.880 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:47:20.880 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:47:20.880 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:47:20.880 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mgr.y", 2023-10-16T13:47:20.880 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOKC1lLz2nMhAA8NqFYZ5nDKkzPBMrkwsVuQ==", 2023-10-16T13:47:20.880 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:47:20.881 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2023-10-16T13:47:20.881 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile mgr", 2023-10-16T13:47:20.881 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:47:20.881 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:47:20.881 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:47:20.881 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-10-16T13:47:20.882 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-10-16T13:47:20.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.846+0000 7fec23766700 1 -- 172.21.15.16:0/1655716776 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fec0006b140 msgr2=0x7fec0006d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:20.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.846+0000 7fec23766700 1 --2- 172.21.15.16:0/1655716776 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fec0006b140 0x7fec0006d600 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fec18000c00 tx=0x7fec18008030 comp rx=0 tx=0).stop 2023-10-16T13:47:20.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.846+0000 7fec23766700 1 -- 172.21.15.16:0/1655716776 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec1c15ba10 msgr2=0x7fec1c0b5a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:20.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.846+0000 7fec23766700 1 --2- 172.21.15.16:0/1655716776 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec1c15ba10 0x7fec1c0b5a40 secure :-1 s=READY pgs=2720 cs=0 l=1 rev1=1 crypto rx=0x7fec1400ccf0 tx=0x7fec1400a570 comp rx=0 tx=0).stop 2023-10-16T13:47:20.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.846+0000 7fec23766700 1 -- 172.21.15.16:0/1655716776 shutdown_connections 2023-10-16T13:47:20.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.846+0000 7fec23766700 1 --2- 172.21.15.16:0/1655716776 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fec0006b140 0x7fec0006d600 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:20.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.846+0000 7fec23766700 1 --2- 172.21.15.16:0/1655716776 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec1c15ba10 0x7fec1c0b5a40 unknown :-1 s=CLOSED pgs=2720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:20.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.846+0000 7fec23766700 1 --2- 172.21.15.16:0/1655716776 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fec1c0cdde0 0x7fec1c0af340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:20.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.846+0000 7fec23766700 1 --2- 172.21.15.16:0/1655716776 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fec1c15b630 0x7fec1c0b5500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:20.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.846+0000 7fec23766700 1 -- 172.21.15.16:0/1655716776 >> 172.21.15.16:0/1655716776 conn(0x7fec1c0d14b0 msgr2=0x7fec1c0b2f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:20.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.846+0000 7fec23766700 1 -- 172.21.15.16:0/1655716776 shutdown_connections 2023-10-16T13:47:20.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:20.846+0000 7fec23766700 1 -- 172.21.15.16:0/1655716776 wait complete. 2023-10-16T13:47:20.886 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:47:21.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.009+0000 7ff456f7d700 1 Processor -- start 2023-10-16T13:47:21.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.009+0000 7ff456f7d700 1 -- start start 2023-10-16T13:47:21.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.010+0000 7ff456f7d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff4500cf4a0 0x7ff4500726e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:21.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.010+0000 7ff456f7d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff4500b5a00 0x7ff45015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:21.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.010+0000 7ff456f7d700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff45015bc80 con 0x7ff4500cf880 2023-10-16T13:47:21.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.010+0000 7ff456f7d700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4500b5e40 con 0x7ff4500cf4a0 2023-10-16T13:47:21.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.010+0000 7ff456f7d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff450072c20 con 0x7ff4500b5a00 2023-10-16T13:47:21.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.010+0000 7ff454d19700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff4500cf4a0 0x7ff4500726e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:21.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.010+0000 7ff454d19700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff4500cf4a0 0x7ff4500726e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55280/0 (socket says 172.21.15.16:55280) 2023-10-16T13:47:21.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.011+0000 7ff454d19700 1 -- 172.21.15.16:0/1412694208 learned_addr learned my addr 172.21.15.16:0/1412694208 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:21.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.011+0000 7ff45551a700 1 --2- 172.21.15.16:0/1412694208 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff4500b5a00 0x7ff45015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:21.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.011+0000 7ff454d19700 1 -- 172.21.15.16:0/1412694208 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff4500b5a00 msgr2=0x7ff45015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:21.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.011+0000 7ff454d19700 1 --2- 172.21.15.16:0/1412694208 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff4500b5a00 0x7ff45015a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:21.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.011+0000 7ff454d19700 1 -- 172.21.15.16:0/1412694208 >> v1:172.21.15.16:6789/0 conn(0x7ff4500cf880 legacy=0x7ff4500b52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:21.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.011+0000 7ff454d19700 1 -- 172.21.15.16:0/1412694208 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff45015b070 con 0x7ff4500cf4a0 2023-10-16T13:47:21.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.011+0000 7ff44f7fe700 1 -- 172.21.15.16:0/1412694208 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2899323765 0 0) 0x7ff45015bc80 con 0x7ff4500cf880 2023-10-16T13:47:21.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.011+0000 7ff454d19700 1 --2- 172.21.15.16:0/1412694208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4500cf4a0 0x7ff4500726e0 secure :-1 s=READY pgs=2722 cs=0 l=1 rev1=1 crypto rx=0x7ff4440098f0 tx=0x7ff444009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8fa8e732f5512d2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:21.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.011+0000 7ff44f7fe700 1 -- 172.21.15.16:0/1412694208 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff44400b8a0 con 0x7ff4500cf4a0 2023-10-16T13:47:21.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.011+0000 7ff44f7fe700 1 -- 172.21.15.16:0/1412694208 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff44400ba00 con 0x7ff4500cf4a0 2023-10-16T13:47:21.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.011+0000 7ff44f7fe700 1 -- 172.21.15.16:0/1412694208 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff44400bd90 con 0x7ff4500cf4a0 2023-10-16T13:47:21.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.012+0000 7ff456f7d700 1 -- 172.21.15.16:0/1412694208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4500cf4a0 msgr2=0x7ff4500726e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:21.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.012+0000 7ff456f7d700 1 --2- 172.21.15.16:0/1412694208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4500cf4a0 0x7ff4500726e0 secure :-1 s=READY pgs=2722 cs=0 l=1 rev1=1 crypto rx=0x7ff4440098f0 tx=0x7ff444009c00 comp rx=0 tx=0).stop 2023-10-16T13:47:21.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.012+0000 7ff456f7d700 1 -- 172.21.15.16:0/1412694208 shutdown_connections 2023-10-16T13:47:21.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.012+0000 7ff456f7d700 1 --2- 172.21.15.16:0/1412694208 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff4500b5a00 0x7ff45015a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:21.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.012+0000 7ff456f7d700 1 --2- 172.21.15.16:0/1412694208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4500cf4a0 0x7ff4500726e0 unknown :-1 s=CLOSED pgs=2722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:21.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.012+0000 7ff456f7d700 1 -- 172.21.15.16:0/1412694208 >> 172.21.15.16:0/1412694208 conn(0x7ff4500b1fd0 msgr2=0x7ff4500b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:21.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.012+0000 7ff456f7d700 1 -- 172.21.15.16:0/1412694208 shutdown_connections 2023-10-16T13:47:21.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.012+0000 7ff456f7d700 1 -- 172.21.15.16:0/1412694208 wait complete. 2023-10-16T13:47:21.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.013+0000 7ff456f7d700 1 Processor -- start 2023-10-16T13:47:21.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.013+0000 7ff456f7d700 1 -- start start 2023-10-16T13:47:21.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.015+0000 7ff456f7d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff4500b5a00 0x7ff450159900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:21.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.015+0000 7ff456f7d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4500cf4a0 0x7ff4501f13a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:21.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.015+0000 7ff456f7d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff4500cf880 0x7ff4501f3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:21.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.015+0000 7ff456f7d700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4501f4eb0 con 0x7ff4500cf4a0 2023-10-16T13:47:21.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.015+0000 7ff456f7d700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4501f0950 con 0x7ff4500cf880 2023-10-16T13:47:21.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.015+0000 7ff456f7d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4501f18e0 con 0x7ff4500b5a00 2023-10-16T13:47:21.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.015+0000 7ff44ffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4500cf4a0 0x7ff4501f13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:21.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.015+0000 7ff44ffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4500cf4a0 0x7ff4501f13a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55288/0 (socket says 172.21.15.16:55288) 2023-10-16T13:47:21.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.015+0000 7ff44ffff700 1 -- 172.21.15.16:0/3530232824 learned_addr learned my addr 172.21.15.16:0/3530232824 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:21.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.015+0000 7ff454d19700 1 --2- 172.21.15.16:0/3530232824 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff4500b5a00 0x7ff450159900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:21.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.015+0000 7ff45551a700 1 --2- 172.21.15.16:0/3530232824 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff4500cf880 0x7ff4501f3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:21.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.015+0000 7ff44ffff700 1 -- 172.21.15.16:0/3530232824 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff4500b5a00 msgr2=0x7ff450159900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:21.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.015+0000 7ff44ffff700 1 --2- 172.21.15.16:0/3530232824 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff4500b5a00 0x7ff450159900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:21.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.015+0000 7ff44ffff700 1 -- 172.21.15.16:0/3530232824 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff4500cf880 msgr2=0x7ff4501f3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:21.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.015+0000 7ff44ffff700 1 --2- 172.21.15.16:0/3530232824 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff4500cf880 0x7ff4501f3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:21.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.015+0000 7ff44ffff700 1 -- 172.21.15.16:0/3530232824 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff444009550 con 0x7ff4500cf4a0 2023-10-16T13:47:21.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.015+0000 7ff44ffff700 1 --2- 172.21.15.16:0/3530232824 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4500cf4a0 0x7ff4501f13a0 secure :-1 s=READY pgs=2723 cs=0 l=1 rev1=1 crypto rx=0x7ff43c00c580 tx=0x7ff43c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:21.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.015+0000 7ff44d7fa700 1 -- 172.21.15.16:0/3530232824 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff43c0054e0 con 0x7ff4500cf4a0 2023-10-16T13:47:21.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.015+0000 7ff44d7fa700 1 -- 172.21.15.16:0/3530232824 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff43c003680 con 0x7ff4500cf4a0 2023-10-16T13:47:21.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.015+0000 7ff456f7d700 1 -- 172.21.15.16:0/3530232824 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4501f48f0 con 0x7ff4500cf4a0 2023-10-16T13:47:21.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.015+0000 7ff44d7fa700 1 -- 172.21.15.16:0/3530232824 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff43c003a30 con 0x7ff4500cf4a0 2023-10-16T13:47:21.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.015+0000 7ff456f7d700 1 -- 172.21.15.16:0/3530232824 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4501f0ba0 con 0x7ff4500cf4a0 2023-10-16T13:47:21.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.016+0000 7ff456f7d700 1 -- 172.21.15.16:0/3530232824 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff45004ebb0 con 0x7ff4500cf4a0 2023-10-16T13:47:21.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.017+0000 7ff44d7fa700 1 -- 172.21.15.16:0/3530232824 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 148) v1 ==== 89433+0+0 (secure 0 0 0) 0x7ff43c003b90 con 0x7ff4500cf4a0 2023-10-16T13:47:21.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.017+0000 7ff44d7fa700 1 --2- 172.21.15.16:0/3530232824 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7ff43406b140 0x7ff43406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:21.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.018+0000 7ff44d7fa700 1 -- 172.21.15.16:0/3530232824 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(241..241 src has 1..241) v4 ==== 5305+0+0 (secure 0 0 0) 0x7ff43c088d00 con 0x7ff4500cf4a0 2023-10-16T13:47:21.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.018+0000 7ff454d19700 1 --2- 172.21.15.16:0/3530232824 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7ff43406b140 0x7ff43406d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:21.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.018+0000 7ff454d19700 1 --2- 172.21.15.16:0/3530232824 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7ff43406b140 0x7ff43406d600 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff4500aa620 tx=0x7ff444008030 comp rx=0 tx=0).ready entity=mgr.11886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:21.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.022+0000 7ff44d7fa700 1 -- 172.21.15.16:0/3530232824 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff43c053cb0 con 0x7ff4500cf4a0 2023-10-16T13:47:21.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.226+0000 7ff456f7d700 1 -- 172.21.15.16:0/3530232824 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff450159f70 con 0x7ff4500cf4a0 2023-10-16T13:47:21.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.227+0000 7ff44d7fa700 1 -- 172.21.15.16:0/3530232824 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v241) v1 ==== 74+0+13169 (secure 0 0 0) 0x7ff43c017070 con 0x7ff4500cf4a0 2023-10-16T13:47:21.230 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:47:21.230 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":241,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:47:20.406013+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:47:21.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.229+0000 7ff456f7d700 1 -- 172.21.15.16:0/3530232824 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7ff43406b140 msgr2=0x7ff43406d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:21.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.229+0000 7ff456f7d700 1 --2- 172.21.15.16:0/3530232824 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7ff43406b140 0x7ff43406d600 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff4500aa620 tx=0x7ff444008030 comp rx=0 tx=0).stop 2023-10-16T13:47:21.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.229+0000 7ff456f7d700 1 -- 172.21.15.16:0/3530232824 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4500cf4a0 msgr2=0x7ff4501f13a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:21.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.229+0000 7ff456f7d700 1 --2- 172.21.15.16:0/3530232824 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4500cf4a0 0x7ff4501f13a0 secure :-1 s=READY pgs=2723 cs=0 l=1 rev1=1 crypto rx=0x7ff43c00c580 tx=0x7ff43c00c940 comp rx=0 tx=0).stop 2023-10-16T13:47:21.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.229+0000 7ff456f7d700 1 -- 172.21.15.16:0/3530232824 shutdown_connections 2023-10-16T13:47:21.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.229+0000 7ff456f7d700 1 --2- 172.21.15.16:0/3530232824 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7ff43406b140 0x7ff43406d600 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:21.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.229+0000 7ff456f7d700 1 --2- 172.21.15.16:0/3530232824 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff4500cf880 0x7ff4501f3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:21.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.229+0000 7ff456f7d700 1 --2- 172.21.15.16:0/3530232824 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff4500b5a00 0x7ff450159900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:21.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.229+0000 7ff456f7d700 1 --2- 172.21.15.16:0/3530232824 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4500cf4a0 0x7ff4501f13a0 unknown :-1 s=CLOSED pgs=2723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:21.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.229+0000 7ff456f7d700 1 -- 172.21.15.16:0/3530232824 >> 172.21.15.16:0/3530232824 conn(0x7ff4500b1fd0 msgr2=0x7ff4500b1380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:21.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.229+0000 7ff456f7d700 1 -- 172.21.15.16:0/3530232824 shutdown_connections 2023-10-16T13:47:21.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.229+0000 7ff456f7d700 1 -- 172.21.15.16:0/3530232824 wait complete. 2023-10-16T13:47:21.244 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T13:47:21.245 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T13:47:21.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.371+0000 7fb3e9624700 1 Processor -- start 2023-10-16T13:47:21.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.371+0000 7fb3e9624700 1 -- start start 2023-10-16T13:47:21.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.371+0000 7fb3e9624700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb3e415d8c0 0x7fb3e40b2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:21.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.371+0000 7fb3e9624700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb3e4162390 0x7fb3e40b34e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:21.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.371+0000 7fb3e9624700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fb3e406be40 0x7fb3e406c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:21.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.371+0000 7fb3e9624700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3e415f180 con 0x7fb3e4162390 2023-10-16T13:47:21.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.371+0000 7fb3e9624700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3e4161f00 con 0x7fb3e406be40 2023-10-16T13:47:21.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.371+0000 7fb3e9624700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e415eea0 con 0x7fb3e415d8c0 2023-10-16T13:47:21.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.372+0000 7fb3e27fc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb3e4162390 0x7fb3e40b34e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:21.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.372+0000 7fb3e27fc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb3e4162390 0x7fb3e40b34e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55296/0 (socket says 172.21.15.16:55296) 2023-10-16T13:47:21.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.372+0000 7fb3e27fc700 1 -- 172.21.15.16:0/2766421192 learned_addr learned my addr 172.21.15.16:0/2766421192 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:21.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.372+0000 7fb3e2ffd700 1 --2- 172.21.15.16:0/2766421192 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb3e415d8c0 0x7fb3e40b2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:21.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.372+0000 7fb3e37fe700 1 --2- 172.21.15.16:0/2766421192 >> v2:172.21.15.49:3300/0 conn(0x7fb3e406be40 0x7fb3e406c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:21.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.372+0000 7fb3e27fc700 1 -- 172.21.15.16:0/2766421192 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb3e415d8c0 msgr2=0x7fb3e40b2fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:21.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.372+0000 7fb3e27fc700 1 --2- 172.21.15.16:0/2766421192 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb3e415d8c0 0x7fb3e40b2fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:21.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.372+0000 7fb3e27fc700 1 -- 172.21.15.16:0/2766421192 >> v2:172.21.15.49:3300/0 conn(0x7fb3e406be40 msgr2=0x7fb3e406c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:21.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.372+0000 7fb3e27fc700 1 --2- 172.21.15.16:0/2766421192 >> v2:172.21.15.49:3300/0 conn(0x7fb3e406be40 0x7fb3e406c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:21.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.372+0000 7fb3e27fc700 1 -- 172.21.15.16:0/2766421192 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3e41618b0 con 0x7fb3e4162390 2023-10-16T13:47:21.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.372+0000 7fb3e27fc700 1 --2- 172.21.15.16:0/2766421192 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3e4162390 0x7fb3e40b34e0 secure :-1 s=READY pgs=2724 cs=0 l=1 rev1=1 crypto rx=0x7fb3e40c4f60 tx=0x7fb3d8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9b88e376022f25c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:21.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.372+0000 7fb3e1ffb700 1 -- 172.21.15.16:0/2766421192 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3d800b870 con 0x7fb3e4162390 2023-10-16T13:47:21.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.372+0000 7fb3e1ffb700 1 -- 172.21.15.16:0/2766421192 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3d800b9d0 con 0x7fb3e4162390 2023-10-16T13:47:21.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.372+0000 7fb3e1ffb700 1 -- 172.21.15.16:0/2766421192 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3d800bd60 con 0x7fb3e4162390 2023-10-16T13:47:21.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.373+0000 7fb3e9624700 1 -- 172.21.15.16:0/2766421192 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3e4162390 msgr2=0x7fb3e40b34e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:21.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.373+0000 7fb3e9624700 1 --2- 172.21.15.16:0/2766421192 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3e4162390 0x7fb3e40b34e0 secure :-1 s=READY pgs=2724 cs=0 l=1 rev1=1 crypto rx=0x7fb3e40c4f60 tx=0x7fb3d8009bd0 comp rx=0 tx=0).stop 2023-10-16T13:47:21.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.373+0000 7fb3e9624700 1 -- 172.21.15.16:0/2766421192 shutdown_connections 2023-10-16T13:47:21.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.373+0000 7fb3e9624700 1 --2- 172.21.15.16:0/2766421192 >> v2:172.21.15.49:3300/0 conn(0x7fb3e406be40 0x7fb3e406c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:21.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.373+0000 7fb3e9624700 1 --2- 172.21.15.16:0/2766421192 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb3e415d8c0 0x7fb3e40b2fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:21.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.373+0000 7fb3e9624700 1 --2- 172.21.15.16:0/2766421192 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3e4162390 0x7fb3e40b34e0 unknown :-1 s=CLOSED pgs=2724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:21.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.373+0000 7fb3e9624700 1 -- 172.21.15.16:0/2766421192 >> 172.21.15.16:0/2766421192 conn(0x7fb3e40cdfa0 msgr2=0x7fb3e40ce3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:21.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.373+0000 7fb3e9624700 1 -- 172.21.15.16:0/2766421192 shutdown_connections 2023-10-16T13:47:21.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.373+0000 7fb3e9624700 1 -- 172.21.15.16:0/2766421192 wait complete. 2023-10-16T13:47:21.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.374+0000 7fb3e9624700 1 Processor -- start 2023-10-16T13:47:21.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.374+0000 7fb3e9624700 1 -- start start 2023-10-16T13:47:21.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.375+0000 7fb3e9624700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb3e406be40 0x7fb3e40c8560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:21.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.375+0000 7fb3e9624700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3e415d8c0 0x7fb3e4149eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:21.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.375+0000 7fb3e9624700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb3e4162390 0x7fb3e4146aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:21.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.375+0000 7fb3e9624700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e414a3f0 con 0x7fb3e415d8c0 2023-10-16T13:47:21.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.375+0000 7fb3e9624700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e40c8c00 con 0x7fb3e406be40 2023-10-16T13:47:21.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.375+0000 7fb3e9624700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e40c8ee0 con 0x7fb3e4162390 2023-10-16T13:47:21.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.375+0000 7fb3e27fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3e415d8c0 0x7fb3e4149eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:21.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.375+0000 7fb3e27fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3e415d8c0 0x7fb3e4149eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55302/0 (socket says 172.21.15.16:55302) 2023-10-16T13:47:21.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.375+0000 7fb3e27fc700 1 -- 172.21.15.16:0/2123456764 learned_addr learned my addr 172.21.15.16:0/2123456764 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:21.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.375+0000 7fb3e37fe700 1 --2- 172.21.15.16:0/2123456764 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb3e4162390 0x7fb3e4146aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:21.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.375+0000 7fb3e2ffd700 1 --2- 172.21.15.16:0/2123456764 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb3e406be40 0x7fb3e40c8560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:21.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.375+0000 7fb3e27fc700 1 -- 172.21.15.16:0/2123456764 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb3e4162390 msgr2=0x7fb3e4146aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:21.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.375+0000 7fb3e27fc700 1 --2- 172.21.15.16:0/2123456764 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb3e4162390 0x7fb3e4146aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:21.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.375+0000 7fb3e27fc700 1 -- 172.21.15.16:0/2123456764 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb3e406be40 msgr2=0x7fb3e40c8560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:21.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.375+0000 7fb3e27fc700 1 --2- 172.21.15.16:0/2123456764 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb3e406be40 0x7fb3e40c8560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:21.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.375+0000 7fb3e27fc700 1 -- 172.21.15.16:0/2123456764 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3d8009550 con 0x7fb3e415d8c0 2023-10-16T13:47:21.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.376+0000 7fb3e27fc700 1 --2- 172.21.15.16:0/2123456764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3e415d8c0 0x7fb3e4149eb0 secure :-1 s=READY pgs=2725 cs=0 l=1 rev1=1 crypto rx=0x7fb3d8000c00 tx=0x7fb3d8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:21.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.376+0000 7fb3cb7fe700 1 -- 172.21.15.16:0/2123456764 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3d8004540 con 0x7fb3e415d8c0 2023-10-16T13:47:21.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.376+0000 7fb3cb7fe700 1 -- 172.21.15.16:0/2123456764 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3d80036a0 con 0x7fb3e415d8c0 2023-10-16T13:47:21.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.376+0000 7fb3e9624700 1 -- 172.21.15.16:0/2123456764 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3e40c91c0 con 0x7fb3e415d8c0 2023-10-16T13:47:21.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.376+0000 7fb3cb7fe700 1 -- 172.21.15.16:0/2123456764 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3d8003a50 con 0x7fb3e415d8c0 2023-10-16T13:47:21.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.376+0000 7fb3e9624700 1 -- 172.21.15.16:0/2123456764 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3e40c9c00 con 0x7fb3e415d8c0 2023-10-16T13:47:21.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.377+0000 7fb3e9624700 1 -- 172.21.15.16:0/2123456764 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3e404ebb0 con 0x7fb3e415d8c0 2023-10-16T13:47:21.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.377+0000 7fb3cb7fe700 1 -- 172.21.15.16:0/2123456764 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 148) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fb3d8021070 con 0x7fb3e415d8c0 2023-10-16T13:47:21.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.378+0000 7fb3cb7fe700 1 --2- 172.21.15.16:0/2123456764 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fb3d006b170 0x7fb3d006d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:21.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.378+0000 7fb3cb7fe700 1 -- 172.21.15.16:0/2123456764 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(241..241 src has 1..241) v4 ==== 5305+0+0 (secure 0 0 0) 0x7fb3d808ae70 con 0x7fb3e415d8c0 2023-10-16T13:47:21.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.378+0000 7fb3e2ffd700 1 --2- 172.21.15.16:0/2123456764 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fb3d006b170 0x7fb3d006d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:21.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.378+0000 7fb3e2ffd700 1 --2- 172.21.15.16:0/2123456764 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fb3d006b170 0x7fb3d006d630 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb3d4009950 tx=0x7fb3d4008040 comp rx=0 tx=0).ready entity=mgr.11886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:21.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.382+0000 7fb3cb7fe700 1 -- 172.21.15.16:0/2123456764 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb3d8055de0 con 0x7fb3e415d8c0 2023-10-16T13:47:21.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.587+0000 7fb3e9624700 1 -- 172.21.15.16:0/2123456764 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb3e40b74a0 con 0x7fb3e415d8c0 2023-10-16T13:47:21.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.587+0000 7fb3cb7fe700 1 -- 172.21.15.16:0/2123456764 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v241) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb3d8015020 con 0x7fb3e415d8c0 2023-10-16T13:47:21.590 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T13:47:21.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.590+0000 7fb3e9624700 1 -- 172.21.15.16:0/2123456764 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fb3d006b170 msgr2=0x7fb3d006d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:21.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.590+0000 7fb3e9624700 1 --2- 172.21.15.16:0/2123456764 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fb3d006b170 0x7fb3d006d630 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb3d4009950 tx=0x7fb3d4008040 comp rx=0 tx=0).stop 2023-10-16T13:47:21.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.590+0000 7fb3e9624700 1 -- 172.21.15.16:0/2123456764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3e415d8c0 msgr2=0x7fb3e4149eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:21.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.590+0000 7fb3e9624700 1 --2- 172.21.15.16:0/2123456764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3e415d8c0 0x7fb3e4149eb0 secure :-1 s=READY pgs=2725 cs=0 l=1 rev1=1 crypto rx=0x7fb3d8000c00 tx=0x7fb3d8004750 comp rx=0 tx=0).stop 2023-10-16T13:47:21.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.590+0000 7fb3e9624700 1 -- 172.21.15.16:0/2123456764 shutdown_connections 2023-10-16T13:47:21.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.590+0000 7fb3e9624700 1 --2- 172.21.15.16:0/2123456764 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fb3d006b170 0x7fb3d006d630 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:21.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.590+0000 7fb3e9624700 1 --2- 172.21.15.16:0/2123456764 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb3e4162390 0x7fb3e4146aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:21.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.590+0000 7fb3e9624700 1 --2- 172.21.15.16:0/2123456764 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb3e406be40 0x7fb3e40c8560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:21.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.590+0000 7fb3e9624700 1 --2- 172.21.15.16:0/2123456764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3e415d8c0 0x7fb3e4149eb0 unknown :-1 s=CLOSED pgs=2725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:21.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.590+0000 7fb3e9624700 1 -- 172.21.15.16:0/2123456764 >> 172.21.15.16:0/2123456764 conn(0x7fb3e40cdfa0 msgr2=0x7fb3e406cd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:21.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.590+0000 7fb3e9624700 1 -- 172.21.15.16:0/2123456764 shutdown_connections 2023-10-16T13:47:21.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.590+0000 7fb3e9624700 1 -- 172.21.15.16:0/2123456764 wait complete. 2023-10-16T13:47:21.605 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-16T13:47:21.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.728+0000 7fedfe191700 1 Processor -- start 2023-10-16T13:47:21.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.728+0000 7fedfe191700 1 -- start start 2023-10-16T13:47:21.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.729+0000 7fedfe191700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fedf80d41c0 0x7fedf80d0040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:21.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.729+0000 7fedfe191700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fedf80d0580 0x7fedf806c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:21.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.729+0000 7fedfe191700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedf806ba00 con 0x7fedf815d8d0 2023-10-16T13:47:21.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.729+0000 7fedfe191700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedf80d0a10 con 0x7fedf80d0580 2023-10-16T13:47:21.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.729+0000 7fedfe191700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedf80d26d0 con 0x7fedf80d41c0 2023-10-16T13:47:21.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.729+0000 7fedf6ffd700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fedf80d41c0 0x7fedf80d0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:21.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.729+0000 7fedf7fff700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fedf80d0580 0x7fedf806c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:21.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.729+0000 7fedf6ffd700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fedf80d41c0 0x7fedf80d0040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:43570/0 (socket says 172.21.15.16:43570) 2023-10-16T13:47:21.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.729+0000 7fedf77fe700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7fedf815d8d0 0x7fedf80d1c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:51766/0 (socket says 172.21.15.16:51766) 2023-10-16T13:47:21.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.729+0000 7fedf6ffd700 1 -- 172.21.15.16:0/2302698102 learned_addr learned my addr 172.21.15.16:0/2302698102 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:21.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.729+0000 7fedf7fff700 1 -- 172.21.15.16:0/2302698102 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fedf80d41c0 msgr2=0x7fedf80d0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:21.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.729+0000 7fedf7fff700 1 --2- 172.21.15.16:0/2302698102 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fedf80d41c0 0x7fedf80d0040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:21.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.729+0000 7fedf7fff700 1 -- 172.21.15.16:0/2302698102 >> v1:172.21.15.49:6789/0 conn(0x7fedf815d8d0 legacy=0x7fedf80d1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:21.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.729+0000 7fedf7fff700 1 -- 172.21.15.16:0/2302698102 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedf80d23f0 con 0x7fedf80d0580 2023-10-16T13:47:21.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.729+0000 7fedf6ffd700 1 --2- 172.21.15.16:0/2302698102 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fedf80d41c0 0x7fedf80d0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:47:21.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.730+0000 7fedf7fff700 1 --2- 172.21.15.16:0/2302698102 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fedf80d0580 0x7fedf806c4a0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fedf0009dc0 tx=0x7fedf000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=12bc1f35ae59bad5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:21.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.730+0000 7fedf67fc700 1 -- 172.21.15.16:0/2302698102 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf000e070 con 0x7fedf80d0580 2023-10-16T13:47:21.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.730+0000 7fedf67fc700 1 -- 172.21.15.16:0/2302698102 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedf000bae0 con 0x7fedf80d0580 2023-10-16T13:47:21.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.730+0000 7fedf67fc700 1 -- 172.21.15.16:0/2302698102 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf000be90 con 0x7fedf80d0580 2023-10-16T13:47:21.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.730+0000 7fedfe191700 1 -- 172.21.15.16:0/2302698102 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fedf80d0580 msgr2=0x7fedf806c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:21.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.730+0000 7fedfe191700 1 --2- 172.21.15.16:0/2302698102 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fedf80d0580 0x7fedf806c4a0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fedf0009dc0 tx=0x7fedf000b330 comp rx=0 tx=0).stop 2023-10-16T13:47:21.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.730+0000 7fedfe191700 1 -- 172.21.15.16:0/2302698102 shutdown_connections 2023-10-16T13:47:21.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.730+0000 7fedfe191700 1 --2- 172.21.15.16:0/2302698102 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fedf80d0580 0x7fedf806c4a0 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:21.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.730+0000 7fedfe191700 1 --2- 172.21.15.16:0/2302698102 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fedf80d41c0 0x7fedf80d0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:21.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.730+0000 7fedfe191700 1 -- 172.21.15.16:0/2302698102 >> 172.21.15.16:0/2302698102 conn(0x7fedf80d3630 msgr2=0x7fedf80d3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:21.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.730+0000 7fedfe191700 1 -- 172.21.15.16:0/2302698102 shutdown_connections 2023-10-16T13:47:21.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.731+0000 7fedfe191700 1 -- 172.21.15.16:0/2302698102 wait complete. 2023-10-16T13:47:21.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.732+0000 7fedfe191700 1 Processor -- start 2023-10-16T13:47:21.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.732+0000 7fedfe191700 1 -- start start 2023-10-16T13:47:21.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.733+0000 7fedfe191700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedf80d0580 0x7fedf81f2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:21.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.733+0000 7fedfe191700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fedf80d41c0 0x7fedf815aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:21.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.733+0000 7fedfe191700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fedf815d8d0 0x7fedf815afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:21.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.733+0000 7fedfe191700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedf80c95b0 con 0x7fedf80d0580 2023-10-16T13:47:21.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.733+0000 7fedfe191700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedf81f3210 con 0x7fedf80d41c0 2023-10-16T13:47:21.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.733+0000 7fedfe191700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedf809d4b0 con 0x7fedf815d8d0 2023-10-16T13:47:21.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.733+0000 7fedf77fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedf80d0580 0x7fedf81f2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:21.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.733+0000 7fedf77fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedf80d0580 0x7fedf81f2cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55318/0 (socket says 172.21.15.16:55318) 2023-10-16T13:47:21.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.733+0000 7fedf77fe700 1 -- 172.21.15.16:0/711356079 learned_addr learned my addr 172.21.15.16:0/711356079 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:21.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.734+0000 7fedf6ffd700 1 --2- 172.21.15.16:0/711356079 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fedf80d41c0 0x7fedf815aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:21.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.734+0000 7fedf7fff700 1 --2- 172.21.15.16:0/711356079 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fedf815d8d0 0x7fedf815afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:21.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.734+0000 7fedf77fe700 1 -- 172.21.15.16:0/711356079 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fedf815d8d0 msgr2=0x7fedf815afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:21.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.734+0000 7fedf77fe700 1 --2- 172.21.15.16:0/711356079 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fedf815d8d0 0x7fedf815afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:21.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.734+0000 7fedf77fe700 1 -- 172.21.15.16:0/711356079 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fedf80d41c0 msgr2=0x7fedf815aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:21.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.734+0000 7fedf77fe700 1 --2- 172.21.15.16:0/711356079 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fedf80d41c0 0x7fedf815aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:21.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.734+0000 7fedf77fe700 1 -- 172.21.15.16:0/711356079 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fede8009550 con 0x7fedf80d0580 2023-10-16T13:47:21.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.734+0000 7fedf7fff700 1 --2- 172.21.15.16:0/711356079 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fedf815d8d0 0x7fedf815afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:47:21.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.734+0000 7fedf6ffd700 1 --2- 172.21.15.16:0/711356079 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fedf80d41c0 0x7fedf815aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:47:21.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.734+0000 7fedf77fe700 1 --2- 172.21.15.16:0/711356079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedf80d0580 0x7fedf81f2cd0 secure :-1 s=READY pgs=2726 cs=0 l=1 rev1=1 crypto rx=0x7fede000c530 tx=0x7fede000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:21.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.734+0000 7fedeffff700 1 -- 172.21.15.16:0/711356079 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fede0005cb0 con 0x7fedf80d0580 2023-10-16T13:47:21.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.734+0000 7fedeffff700 1 -- 172.21.15.16:0/711356079 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fede0005e10 con 0x7fedf80d0580 2023-10-16T13:47:21.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.734+0000 7fedfe191700 1 -- 172.21.15.16:0/711356079 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedf0009a70 con 0x7fedf80d0580 2023-10-16T13:47:21.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.734+0000 7fedeffff700 1 -- 172.21.15.16:0/711356079 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fede00072d0 con 0x7fedf80d0580 2023-10-16T13:47:21.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.734+0000 7fedfe191700 1 -- 172.21.15.16:0/711356079 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedf80c92d0 con 0x7fedf80d0580 2023-10-16T13:47:21.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.735+0000 7fedfe191700 1 -- 172.21.15.16:0/711356079 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedf809d310 con 0x7fedf80d0580 2023-10-16T13:47:21.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.736+0000 7fedeffff700 1 -- 172.21.15.16:0/711356079 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 148) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fede0007430 con 0x7fedf80d0580 2023-10-16T13:47:21.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.736+0000 7fedeffff700 1 --2- 172.21.15.16:0/711356079 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fedd806b140 0x7fedd806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:21.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.736+0000 7fedeffff700 1 -- 172.21.15.16:0/711356079 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(241..241 src has 1..241) v4 ==== 5305+0+0 (secure 0 0 0) 0x7fede0088ec0 con 0x7fedf80d0580 2023-10-16T13:47:21.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.736+0000 7fedf6ffd700 1 --2- 172.21.15.16:0/711356079 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fedd806b140 0x7fedd806d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:21.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.737+0000 7fedf6ffd700 1 --2- 172.21.15.16:0/711356079 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fedd806b140 0x7fedd806d600 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fede8009db0 tx=0x7fede8008040 comp rx=0 tx=0).ready entity=mgr.11886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:21.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.740+0000 7fedeffff700 1 -- 172.21.15.16:0/711356079 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fede0053ee0 con 0x7fedf80d0580 2023-10-16T13:47:21.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.946+0000 7fedfe191700 1 -- 172.21.15.16:0/711356079 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fedf809d310 con 0x7fedf80d0580 2023-10-16T13:47:21.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.946+0000 7fedeffff700 1 -- 172.21.15.16:0/711356079 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v241) v1 ==== 84+0+65 (secure 0 0 0) 0x7fedf809d310 con 0x7fedf80d0580 2023-10-16T13:47:21.949 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:47:21.949 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2023-10-16T13:47:21.949 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:47:21.949 INFO:teuthology.orchestra.run.smithi016.stdout: "poolnum": 1, 2023-10-16T13:47:21.950 INFO:teuthology.orchestra.run.smithi016.stdout: "poolname": ".mgr" 2023-10-16T13:47:21.950 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:47:21.950 INFO:teuthology.orchestra.run.smithi016.stdout:] 2023-10-16T13:47:21.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.948+0000 7fedfe191700 1 -- 172.21.15.16:0/711356079 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fedd806b140 msgr2=0x7fedd806d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:21.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.948+0000 7fedfe191700 1 --2- 172.21.15.16:0/711356079 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fedd806b140 0x7fedd806d600 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fede8009db0 tx=0x7fede8008040 comp rx=0 tx=0).stop 2023-10-16T13:47:21.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.948+0000 7fedfe191700 1 -- 172.21.15.16:0/711356079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedf80d0580 msgr2=0x7fedf81f2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:21.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.948+0000 7fedfe191700 1 --2- 172.21.15.16:0/711356079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedf80d0580 0x7fedf81f2cd0 secure :-1 s=READY pgs=2726 cs=0 l=1 rev1=1 crypto rx=0x7fede000c530 tx=0x7fede000c8f0 comp rx=0 tx=0).stop 2023-10-16T13:47:21.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.948+0000 7fedfe191700 1 -- 172.21.15.16:0/711356079 shutdown_connections 2023-10-16T13:47:21.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.948+0000 7fedfe191700 1 --2- 172.21.15.16:0/711356079 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fedd806b140 0x7fedd806d600 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:21.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.948+0000 7fedfe191700 1 --2- 172.21.15.16:0/711356079 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fedf815d8d0 0x7fedf815afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:21.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.948+0000 7fedfe191700 1 --2- 172.21.15.16:0/711356079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedf80d0580 0x7fedf81f2cd0 unknown :-1 s=CLOSED pgs=2726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:21.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.948+0000 7fedfe191700 1 --2- 172.21.15.16:0/711356079 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fedf80d41c0 0x7fedf815aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:21.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.948+0000 7fedfe191700 1 -- 172.21.15.16:0/711356079 >> 172.21.15.16:0/711356079 conn(0x7fedf80d3630 msgr2=0x7fedf815cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:21.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.949+0000 7fedfe191700 1 -- 172.21.15.16:0/711356079 shutdown_connections 2023-10-16T13:47:21.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:21.949+0000 7fedfe191700 1 -- 172.21.15.16:0/711356079 wait complete. 2023-10-16T13:47:21.963 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-16T13:47:21.964 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-16T13:47:22.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.090+0000 7f8995fbc700 1 Processor -- start 2023-10-16T13:47:22.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.090+0000 7f8995fbc700 1 -- start start 2023-10-16T13:47:22.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.091+0000 7f8995fbc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8990159c10 0x7f8990071140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:22.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.091+0000 7f8995fbc700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8990071680 0x7f89900ae8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:22.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.091+0000 7f8995fbc700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f899015e5d0 con 0x7f89900aedf0 2023-10-16T13:47:22.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.091+0000 7f8995fbc700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8990071b00 con 0x7f8990159c10 2023-10-16T13:47:22.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.091+0000 7f8995fbc700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f899015da00 con 0x7f8990071680 2023-10-16T13:47:22.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.091+0000 7f898f7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8990159c10 0x7f8990071140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:22.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.091+0000 7f898f7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8990159c10 0x7f8990071140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55324/0 (socket says 172.21.15.16:55324) 2023-10-16T13:47:22.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.091+0000 7f898f7fe700 1 -- 172.21.15.16:0/2772228757 learned_addr learned my addr 172.21.15.16:0/2772228757 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:22.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.092+0000 7f898effd700 1 --2- 172.21.15.16:0/2772228757 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8990071680 0x7f89900ae8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:22.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.092+0000 7f898f7fe700 1 -- 172.21.15.16:0/2772228757 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8990071680 msgr2=0x7f89900ae8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:22.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.092+0000 7f898f7fe700 1 --2- 172.21.15.16:0/2772228757 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8990071680 0x7f89900ae8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:22.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.092+0000 7f898f7fe700 1 -- 172.21.15.16:0/2772228757 >> v1:172.21.15.16:6789/0 conn(0x7f89900aedf0 legacy=0x7f899006d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:22.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.092+0000 7f898f7fe700 1 -- 172.21.15.16:0/2772228757 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f899015dca0 con 0x7f8990159c10 2023-10-16T13:47:22.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.092+0000 7f898f7fe700 1 --2- 172.21.15.16:0/2772228757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8990159c10 0x7f8990071140 secure :-1 s=READY pgs=2728 cs=0 l=1 rev1=1 crypto rx=0x7f897800bb90 tx=0x7f897800bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9915b3ae5b64b2e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:22.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.092+0000 7f898e7fc700 1 -- 172.21.15.16:0/2772228757 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f897800dd50 con 0x7f8990159c10 2023-10-16T13:47:22.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.092+0000 7f898e7fc700 1 -- 172.21.15.16:0/2772228757 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8978010470 con 0x7f8990159c10 2023-10-16T13:47:22.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.092+0000 7f898e7fc700 1 -- 172.21.15.16:0/2772228757 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8978010800 con 0x7f8990159c10 2023-10-16T13:47:22.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.093+0000 7f8995fbc700 1 -- 172.21.15.16:0/2772228757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8990159c10 msgr2=0x7f8990071140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:22.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.093+0000 7f8995fbc700 1 --2- 172.21.15.16:0/2772228757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8990159c10 0x7f8990071140 secure :-1 s=READY pgs=2728 cs=0 l=1 rev1=1 crypto rx=0x7f897800bb90 tx=0x7f897800bea0 comp rx=0 tx=0).stop 2023-10-16T13:47:22.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.093+0000 7f8995fbc700 1 -- 172.21.15.16:0/2772228757 shutdown_connections 2023-10-16T13:47:22.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.093+0000 7f8995fbc700 1 --2- 172.21.15.16:0/2772228757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8990159c10 0x7f8990071140 unknown :-1 s=CLOSED pgs=2728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:22.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.093+0000 7f8995fbc700 1 --2- 172.21.15.16:0/2772228757 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8990071680 0x7f89900ae8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:22.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.093+0000 7f8995fbc700 1 -- 172.21.15.16:0/2772228757 >> 172.21.15.16:0/2772228757 conn(0x7f89900cfa90 msgr2=0x7f89900cfea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:22.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.093+0000 7f8995fbc700 1 -- 172.21.15.16:0/2772228757 shutdown_connections 2023-10-16T13:47:22.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.093+0000 7f8995fbc700 1 -- 172.21.15.16:0/2772228757 wait complete. 2023-10-16T13:47:22.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.094+0000 7f8995fbc700 1 Processor -- start 2023-10-16T13:47:22.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.094+0000 7f8995fbc700 1 -- start start 2023-10-16T13:47:22.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.095+0000 7f8995fbc700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8990071680 0x7f89901442d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:22.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.095+0000 7f8995fbc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89900aedf0 0x7f89900b2790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:22.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.095+0000 7f8995fbc700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8990159c10 0x7f89900ae4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:22.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.095+0000 7f8995fbc700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f899014d960 con 0x7f89900aedf0 2023-10-16T13:47:22.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.095+0000 7f8995fbc700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89900b1330 con 0x7f8990159c10 2023-10-16T13:47:22.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.095+0000 7f8995fbc700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89900acca0 con 0x7f8990071680 2023-10-16T13:47:22.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.095+0000 7f898effd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89900aedf0 0x7f89900b2790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:22.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.095+0000 7f898effd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89900aedf0 0x7f89900b2790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55330/0 (socket says 172.21.15.16:55330) 2023-10-16T13:47:22.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.095+0000 7f898effd700 1 -- 172.21.15.16:0/1331677871 learned_addr learned my addr 172.21.15.16:0/1331677871 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:22.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.095+0000 7f898ffff700 1 --2- 172.21.15.16:0/1331677871 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8990159c10 0x7f89900ae4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:22.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.095+0000 7f898f7fe700 1 --2- 172.21.15.16:0/1331677871 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8990071680 0x7f89901442d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:22.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.095+0000 7f898effd700 1 -- 172.21.15.16:0/1331677871 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8990071680 msgr2=0x7f89901442d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:22.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.095+0000 7f898effd700 1 --2- 172.21.15.16:0/1331677871 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8990071680 0x7f89901442d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:22.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.095+0000 7f898effd700 1 -- 172.21.15.16:0/1331677871 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8990159c10 msgr2=0x7f89900ae4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:22.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.095+0000 7f898effd700 1 --2- 172.21.15.16:0/1331677871 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8990159c10 0x7f89900ae4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:22.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.095+0000 7f898effd700 1 -- 172.21.15.16:0/1331677871 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f897800b7f0 con 0x7f89900aedf0 2023-10-16T13:47:22.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.095+0000 7f898f7fe700 1 --2- 172.21.15.16:0/1331677871 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8990071680 0x7f89901442d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:47:22.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.095+0000 7f898effd700 1 --2- 172.21.15.16:0/1331677871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89900aedf0 0x7f89900b2790 secure :-1 s=READY pgs=2729 cs=0 l=1 rev1=1 crypto rx=0x7f898000e810 tx=0x7f898000eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:22.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.096+0000 7f8987fff700 1 -- 172.21.15.16:0/1331677871 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8980013610 con 0x7f89900aedf0 2023-10-16T13:47:22.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.096+0000 7f8987fff700 1 -- 172.21.15.16:0/1331677871 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8980013770 con 0x7f89900aedf0 2023-10-16T13:47:22.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.096+0000 7f8995fbc700 1 -- 172.21.15.16:0/1331677871 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89900b0ce0 con 0x7f89900aedf0 2023-10-16T13:47:22.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.096+0000 7f8987fff700 1 -- 172.21.15.16:0/1331677871 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8980013b20 con 0x7f89900aedf0 2023-10-16T13:47:22.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.096+0000 7f8995fbc700 1 -- 172.21.15.16:0/1331677871 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89900b1070 con 0x7f89900aedf0 2023-10-16T13:47:22.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.097+0000 7f8995fbc700 1 -- 172.21.15.16:0/1331677871 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f899004ec20 con 0x7f89900aedf0 2023-10-16T13:47:22.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.097+0000 7f8987fff700 1 -- 172.21.15.16:0/1331677871 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 148) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f8980013c80 con 0x7f89900aedf0 2023-10-16T13:47:22.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.098+0000 7f8987fff700 1 --2- 172.21.15.16:0/1331677871 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f897006b140 0x7f897006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:22.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.098+0000 7f8987fff700 1 -- 172.21.15.16:0/1331677871 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(241..241 src has 1..241) v4 ==== 5305+0+0 (secure 0 0 0) 0x7f8980089590 con 0x7f89900aedf0 2023-10-16T13:47:22.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.098+0000 7f898f7fe700 1 --2- 172.21.15.16:0/1331677871 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f897006b140 0x7f897006d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:22.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.098+0000 7f898f7fe700 1 --2- 172.21.15.16:0/1331677871 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f897006b140 0x7f897006d600 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f897800bee0 tx=0x7f8978013000 comp rx=0 tx=0).ready entity=mgr.11886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:22.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.102+0000 7f8987fff700 1 -- 172.21.15.16:0/1331677871 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8980054370 con 0x7f89900aedf0 2023-10-16T13:47:22.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.303+0000 7f8995fbc700 1 -- 172.21.15.16:0/1331677871 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f89900b5c80 con 0x7f89900aedf0 2023-10-16T13:47:22.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.403+0000 7f8987fff700 1 -- 172.21.15.16:0/1331677871 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v242) v1 ==== 138+0+0 (secure 0 0 0) 0x7f8980057990 con 0x7f89900aedf0 2023-10-16T13:47:22.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.406+0000 7f8995fbc700 1 -- 172.21.15.16:0/1331677871 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f897006b140 msgr2=0x7f897006d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:22.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.406+0000 7f8995fbc700 1 --2- 172.21.15.16:0/1331677871 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f897006b140 0x7f897006d600 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f897800bee0 tx=0x7f8978013000 comp rx=0 tx=0).stop 2023-10-16T13:47:22.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.406+0000 7f8995fbc700 1 -- 172.21.15.16:0/1331677871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89900aedf0 msgr2=0x7f89900b2790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:22.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.406+0000 7f8995fbc700 1 --2- 172.21.15.16:0/1331677871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89900aedf0 0x7f89900b2790 secure :-1 s=READY pgs=2729 cs=0 l=1 rev1=1 crypto rx=0x7f898000e810 tx=0x7f898000eb20 comp rx=0 tx=0).stop 2023-10-16T13:47:22.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.406+0000 7f8995fbc700 1 -- 172.21.15.16:0/1331677871 shutdown_connections 2023-10-16T13:47:22.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.406+0000 7f8995fbc700 1 --2- 172.21.15.16:0/1331677871 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f897006b140 0x7f897006d600 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:22.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.406+0000 7f8995fbc700 1 --2- 172.21.15.16:0/1331677871 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8990159c10 0x7f89900ae4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:22.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.406+0000 7f8995fbc700 1 --2- 172.21.15.16:0/1331677871 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8990071680 0x7f89901442d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:22.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.406+0000 7f8995fbc700 1 --2- 172.21.15.16:0/1331677871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89900aedf0 0x7f89900b2790 unknown :-1 s=CLOSED pgs=2729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:22.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.406+0000 7f8995fbc700 1 -- 172.21.15.16:0/1331677871 >> 172.21.15.16:0/1331677871 conn(0x7f89900cfa90 msgr2=0x7f8990153910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:22.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.406+0000 7f8995fbc700 1 -- 172.21.15.16:0/1331677871 shutdown_connections 2023-10-16T13:47:22.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.406+0000 7f8995fbc700 1 -- 172.21.15.16:0/1331677871 wait complete. 2023-10-16T13:47:22.413 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' created 2023-10-16T13:47:22.424 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-16T13:47:22.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.555+0000 7fc3e9bd9700 1 Processor -- start 2023-10-16T13:47:22.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.555+0000 7fc3e9bd9700 1 -- start start 2023-10-16T13:47:22.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.555+0000 7fc3e9bd9700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc3e40cb880 0x7fc3e40cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:22.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.556+0000 7fc3e9bd9700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3e40cef70 con 0x7fc3e415b070 2023-10-16T13:47:22.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.556+0000 7fc3e9bd9700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3e40ce360 con 0x7fc3e406da60 2023-10-16T13:47:22.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.556+0000 7fc3e9bd9700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3e40ce640 con 0x7fc3e40cb880 2023-10-16T13:47:22.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.556+0000 7fc3e2ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc3e40cb880 0x7fc3e40cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:22.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.556+0000 7fc3e2ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc3e40cb880 0x7fc3e40cbd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55340/0 (socket says 172.21.15.16:55340) 2023-10-16T13:47:22.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.556+0000 7fc3e2ffd700 1 -- 172.21.15.16:0/1640415099 learned_addr learned my addr 172.21.15.16:0/1640415099 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:22.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.556+0000 7fc3e2ffd700 1 -- 172.21.15.16:0/1640415099 >> v1:172.21.15.49:6789/0 conn(0x7fc3e406da60 legacy=0x7fc3e40cf400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:22.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.556+0000 7fc3e2ffd700 1 -- 172.21.15.16:0/1640415099 >> v1:172.21.15.16:6789/0 conn(0x7fc3e415b070 legacy=0x7fc3e415b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:22.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.556+0000 7fc3e2ffd700 1 -- 172.21.15.16:0/1640415099 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3e40ce920 con 0x7fc3e40cb880 2023-10-16T13:47:22.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.556+0000 7fc3e2ffd700 1 --2- 172.21.15.16:0/1640415099 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3e40cb880 0x7fc3e40cbd00 secure :-1 s=READY pgs=2731 cs=0 l=1 rev1=1 crypto rx=0x7fc3d80098f0 tx=0x7fc3d8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32a72832ade42d5a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:22.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.556+0000 7fc3e27fc700 1 -- 172.21.15.16:0/1640415099 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d800b8a0 con 0x7fc3e40cb880 2023-10-16T13:47:22.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.556+0000 7fc3e27fc700 1 -- 172.21.15.16:0/1640415099 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3d800ba00 con 0x7fc3e40cb880 2023-10-16T13:47:22.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.556+0000 7fc3e27fc700 1 -- 172.21.15.16:0/1640415099 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d800bd90 con 0x7fc3e40cb880 2023-10-16T13:47:22.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.557+0000 7fc3e9bd9700 1 -- 172.21.15.16:0/1640415099 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3e40cb880 msgr2=0x7fc3e40cbd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:22.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.557+0000 7fc3e9bd9700 1 --2- 172.21.15.16:0/1640415099 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3e40cb880 0x7fc3e40cbd00 secure :-1 s=READY pgs=2731 cs=0 l=1 rev1=1 crypto rx=0x7fc3d80098f0 tx=0x7fc3d8009c00 comp rx=0 tx=0).stop 2023-10-16T13:47:22.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.557+0000 7fc3e9bd9700 1 -- 172.21.15.16:0/1640415099 shutdown_connections 2023-10-16T13:47:22.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.557+0000 7fc3e9bd9700 1 --2- 172.21.15.16:0/1640415099 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3e40cb880 0x7fc3e40cbd00 unknown :-1 s=CLOSED pgs=2731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:22.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.557+0000 7fc3e9bd9700 1 -- 172.21.15.16:0/1640415099 >> 172.21.15.16:0/1640415099 conn(0x7fc3e40b1f40 msgr2=0x7fc3e40b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:22.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.557+0000 7fc3e9bd9700 1 -- 172.21.15.16:0/1640415099 shutdown_connections 2023-10-16T13:47:22.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.557+0000 7fc3e9bd9700 1 -- 172.21.15.16:0/1640415099 wait complete. 2023-10-16T13:47:22.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.558+0000 7fc3e9bd9700 1 Processor -- start 2023-10-16T13:47:22.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.558+0000 7fc3e9bd9700 1 -- start start 2023-10-16T13:47:22.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.559+0000 7fc3e9bd9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3e406da60 0x7fc3e41546a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:22.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.559+0000 7fc3e9bd9700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc3e40cb880 0x7fc3e4157360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:22.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.559+0000 7fc3e9bd9700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc3e415b070 0x7fc3e41578a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:22.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.559+0000 7fc3e9bd9700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e4154be0 con 0x7fc3e406da60 2023-10-16T13:47:22.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.559+0000 7fc3e9bd9700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e40c74c0 con 0x7fc3e40cb880 2023-10-16T13:47:22.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.559+0000 7fc3e9bd9700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e40b0140 con 0x7fc3e415b070 2023-10-16T13:47:22.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.559+0000 7fc3e37fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3e406da60 0x7fc3e41546a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:22.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.559+0000 7fc3e37fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3e406da60 0x7fc3e41546a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55344/0 (socket says 172.21.15.16:55344) 2023-10-16T13:47:22.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.559+0000 7fc3e37fe700 1 -- 172.21.15.16:0/1662301413 learned_addr learned my addr 172.21.15.16:0/1662301413 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:22.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.559+0000 7fc3e2ffd700 1 --2- 172.21.15.16:0/1662301413 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc3e40cb880 0x7fc3e4157360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:22.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.559+0000 7fc3e3fff700 1 --2- 172.21.15.16:0/1662301413 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc3e415b070 0x7fc3e41578a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:22.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.559+0000 7fc3e37fe700 1 -- 172.21.15.16:0/1662301413 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc3e415b070 msgr2=0x7fc3e41578a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:22.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.559+0000 7fc3e37fe700 1 --2- 172.21.15.16:0/1662301413 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc3e415b070 0x7fc3e41578a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:22.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.559+0000 7fc3e37fe700 1 -- 172.21.15.16:0/1662301413 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc3e40cb880 msgr2=0x7fc3e4157360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:22.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.559+0000 7fc3e37fe700 1 --2- 172.21.15.16:0/1662301413 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc3e40cb880 0x7fc3e4157360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:22.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.560+0000 7fc3e37fe700 1 -- 172.21.15.16:0/1662301413 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3d8009550 con 0x7fc3e406da60 2023-10-16T13:47:22.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.560+0000 7fc3e37fe700 1 --2- 172.21.15.16:0/1662301413 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3e406da60 0x7fc3e41546a0 secure :-1 s=READY pgs=2732 cs=0 l=1 rev1=1 crypto rx=0x7fc3d400c450 tx=0x7fc3d400c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:22.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.560+0000 7fc3cbfff700 1 -- 172.21.15.16:0/1662301413 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d40052d0 con 0x7fc3e406da60 2023-10-16T13:47:22.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.560+0000 7fc3cbfff700 1 -- 172.21.15.16:0/1662301413 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3d4005430 con 0x7fc3e406da60 2023-10-16T13:47:22.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.560+0000 7fc3cbfff700 1 -- 172.21.15.16:0/1662301413 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d40038c0 con 0x7fc3e406da60 2023-10-16T13:47:22.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.560+0000 7fc3e9bd9700 1 -- 172.21.15.16:0/1662301413 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3e40b0420 con 0x7fc3e406da60 2023-10-16T13:47:22.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.560+0000 7fc3e9bd9700 1 -- 172.21.15.16:0/1662301413 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3e40b09e0 con 0x7fc3e406da60 2023-10-16T13:47:22.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.561+0000 7fc3e9bd9700 1 -- 172.21.15.16:0/1662301413 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3e40cdf80 con 0x7fc3e406da60 2023-10-16T13:47:22.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.562+0000 7fc3cbfff700 1 -- 172.21.15.16:0/1662301413 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 148) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fc3d4003a20 con 0x7fc3e406da60 2023-10-16T13:47:22.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.562+0000 7fc3cbfff700 1 --2- 172.21.15.16:0/1662301413 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fc3d006b140 0x7fc3d006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:22.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.562+0000 7fc3cbfff700 1 -- 172.21.15.16:0/1662301413 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(242..242 src has 1..242) v4 ==== 5675+0+0 (secure 0 0 0) 0x7fc3d4088fd0 con 0x7fc3e406da60 2023-10-16T13:47:22.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.562+0000 7fc3e2ffd700 1 --2- 172.21.15.16:0/1662301413 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fc3d006b140 0x7fc3d006d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:22.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.563+0000 7fc3e2ffd700 1 --2- 172.21.15.16:0/1662301413 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fc3d006b140 0x7fc3d006d600 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc3d8009c40 tx=0x7fc3d8008030 comp rx=0 tx=0).ready entity=mgr.11886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:22.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.567+0000 7fc3cbfff700 1 -- 172.21.15.16:0/1662301413 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc3d4053f80 con 0x7fc3e406da60 2023-10-16T13:47:22.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:22.771+0000 7fc3e9bd9700 1 -- 172.21.15.16:0/1662301413 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fc3e40cebd0 con 0x7fc3e406da60 2023-10-16T13:47:23.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.410+0000 7fc3cbfff700 1 -- 172.21.15.16:0/1662301413 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v243) v1 ==== 170+0+0 (secure 0 0 0) 0x7fc3d40170d0 con 0x7fc3e406da60 2023-10-16T13:47:23.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.420+0000 7fc3e9bd9700 1 -- 172.21.15.16:0/1662301413 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fc3d006b140 msgr2=0x7fc3d006d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:23.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.420+0000 7fc3e9bd9700 1 --2- 172.21.15.16:0/1662301413 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fc3d006b140 0x7fc3d006d600 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc3d8009c40 tx=0x7fc3d8008030 comp rx=0 tx=0).stop 2023-10-16T13:47:23.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.420+0000 7fc3e9bd9700 1 -- 172.21.15.16:0/1662301413 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3e406da60 msgr2=0x7fc3e41546a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:23.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.420+0000 7fc3e9bd9700 1 --2- 172.21.15.16:0/1662301413 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3e406da60 0x7fc3e41546a0 secure :-1 s=READY pgs=2732 cs=0 l=1 rev1=1 crypto rx=0x7fc3d400c450 tx=0x7fc3d400c760 comp rx=0 tx=0).stop 2023-10-16T13:47:23.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.420+0000 7fc3e9bd9700 1 -- 172.21.15.16:0/1662301413 shutdown_connections 2023-10-16T13:47:23.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.420+0000 7fc3e9bd9700 1 --2- 172.21.15.16:0/1662301413 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fc3d006b140 0x7fc3d006d600 secure :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc3d8009c40 tx=0x7fc3d8008030 comp rx=0 tx=0).stop 2023-10-16T13:47:23.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.420+0000 7fc3e9bd9700 1 --2- 172.21.15.16:0/1662301413 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc3e415b070 0x7fc3e41578a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:23.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.420+0000 7fc3e9bd9700 1 --2- 172.21.15.16:0/1662301413 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3e406da60 0x7fc3e41546a0 secure :-1 s=CLOSED pgs=2732 cs=0 l=1 rev1=1 crypto rx=0x7fc3d400c450 tx=0x7fc3d400c760 comp rx=0 tx=0).stop 2023-10-16T13:47:23.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.420+0000 7fc3e9bd9700 1 --2- 172.21.15.16:0/1662301413 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc3e40cb880 0x7fc3e4157360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:23.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.420+0000 7fc3e9bd9700 1 -- 172.21.15.16:0/1662301413 >> 172.21.15.16:0/1662301413 conn(0x7fc3e40b1f40 msgr2=0x7fc3e415d460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:23.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.426+0000 7fc3e9bd9700 1 -- 172.21.15.16:0/1662301413 shutdown_connections 2023-10-16T13:47:23.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.430+0000 7fc3e9bd9700 1 -- 172.21.15.16:0/1662301413 wait complete. 2023-10-16T13:47:23.437 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' created 2023-10-16T13:47:23.451 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-16T13:47:23.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.582+0000 7fe153231700 1 Processor -- start 2023-10-16T13:47:23.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.582+0000 7fe153231700 1 -- start start 2023-10-16T13:47:23.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.583+0000 7fe153231700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fe14c0ce6a0 0x7fe14c06c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:23.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.583+0000 7fe153231700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe14c0d2110 0x7fe14c06c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:23.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.583+0000 7fe153231700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe14c0b2620 0x7fe14c0b0f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:23.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.583+0000 7fe153231700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe14c0b20f0 con 0x7fe14c0d2110 2023-10-16T13:47:23.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.583+0000 7fe153231700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe14c0d03d0 con 0x7fe14c0ce6a0 2023-10-16T13:47:23.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.583+0000 7fe153231700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe14c0b2340 con 0x7fe14c0b2620 2023-10-16T13:47:23.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.583+0000 7fe14bfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe14c0d2110 0x7fe14c06c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:23.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.583+0000 7fe14bfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe14c0d2110 0x7fe14c06c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55354/0 (socket says 172.21.15.16:55354) 2023-10-16T13:47:23.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.583+0000 7fe14bfff700 1 -- 172.21.15.16:0/459202456 learned_addr learned my addr 172.21.15.16:0/459202456 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:23.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.583+0000 7fe150fcd700 1 --2- 172.21.15.16:0/459202456 >> v2:172.21.15.49:3300/0 conn(0x7fe14c0ce6a0 0x7fe14c06c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:23.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.583+0000 7fe1517ce700 1 --2- 172.21.15.16:0/459202456 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe14c0b2620 0x7fe14c0b0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:23.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.583+0000 7fe14bfff700 1 -- 172.21.15.16:0/459202456 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe14c0b2620 msgr2=0x7fe14c0b0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:23.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.583+0000 7fe14bfff700 1 --2- 172.21.15.16:0/459202456 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe14c0b2620 0x7fe14c0b0f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:23.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.583+0000 7fe14bfff700 1 -- 172.21.15.16:0/459202456 >> v2:172.21.15.49:3300/0 conn(0x7fe14c0ce6a0 msgr2=0x7fe14c06c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:23.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.583+0000 7fe14bfff700 1 --2- 172.21.15.16:0/459202456 >> v2:172.21.15.49:3300/0 conn(0x7fe14c0ce6a0 0x7fe14c06c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:23.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.583+0000 7fe14bfff700 1 -- 172.21.15.16:0/459202456 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe14c06bb00 con 0x7fe14c0d2110 2023-10-16T13:47:23.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.583+0000 7fe14bfff700 1 --2- 172.21.15.16:0/459202456 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe14c0d2110 0x7fe14c06c840 secure :-1 s=READY pgs=2733 cs=0 l=1 rev1=1 crypto rx=0x7fe14c067b50 tx=0x7fe138009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8142790a61296b05 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:23.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.584+0000 7fe14b7fe700 1 -- 172.21.15.16:0/459202456 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe13800b870 con 0x7fe14c0d2110 2023-10-16T13:47:23.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.584+0000 7fe14b7fe700 1 -- 172.21.15.16:0/459202456 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe13800b9d0 con 0x7fe14c0d2110 2023-10-16T13:47:23.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.584+0000 7fe14b7fe700 1 -- 172.21.15.16:0/459202456 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe13800bd60 con 0x7fe14c0d2110 2023-10-16T13:47:23.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.584+0000 7fe153231700 1 -- 172.21.15.16:0/459202456 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe14c0d2110 msgr2=0x7fe14c06c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:23.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.584+0000 7fe153231700 1 --2- 172.21.15.16:0/459202456 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe14c0d2110 0x7fe14c06c840 secure :-1 s=READY pgs=2733 cs=0 l=1 rev1=1 crypto rx=0x7fe14c067b50 tx=0x7fe138009bd0 comp rx=0 tx=0).stop 2023-10-16T13:47:23.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.584+0000 7fe153231700 1 -- 172.21.15.16:0/459202456 shutdown_connections 2023-10-16T13:47:23.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.584+0000 7fe153231700 1 --2- 172.21.15.16:0/459202456 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe14c0b2620 0x7fe14c0b0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:23.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.584+0000 7fe153231700 1 --2- 172.21.15.16:0/459202456 >> v2:172.21.15.49:3300/0 conn(0x7fe14c0ce6a0 0x7fe14c06c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:23.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.584+0000 7fe153231700 1 --2- 172.21.15.16:0/459202456 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe14c0d2110 0x7fe14c06c840 unknown :-1 s=CLOSED pgs=2733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:23.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.584+0000 7fe153231700 1 -- 172.21.15.16:0/459202456 >> 172.21.15.16:0/459202456 conn(0x7fe14c0d15f0 msgr2=0x7fe14c0d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:23.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.584+0000 7fe153231700 1 -- 172.21.15.16:0/459202456 shutdown_connections 2023-10-16T13:47:23.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.584+0000 7fe153231700 1 -- 172.21.15.16:0/459202456 wait complete. 2023-10-16T13:47:23.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.585+0000 7fe153231700 1 Processor -- start 2023-10-16T13:47:23.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.585+0000 7fe153231700 1 -- start start 2023-10-16T13:47:23.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.586+0000 7fe153231700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe14c0b2620 0x7fe14c09d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:23.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.586+0000 7fe153231700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe14c0ce6a0 0x7fe14c1faf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:23.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.586+0000 7fe153231700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe14c0d2110 0x7fe14c0b5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:23.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.586+0000 7fe153231700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe14c09d860 con 0x7fe14c0d2110 2023-10-16T13:47:23.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.586+0000 7fe153231700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe14c09ce90 con 0x7fe14c0ce6a0 2023-10-16T13:47:23.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.586+0000 7fe153231700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe14c1fb440 con 0x7fe14c0b2620 2023-10-16T13:47:23.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.587+0000 7fe1517ce700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe14c0d2110 0x7fe14c0b5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:23.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.587+0000 7fe150fcd700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe14c0b2620 0x7fe14c09d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:23.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.587+0000 7fe1517ce700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe14c0d2110 0x7fe14c0b5770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55360/0 (socket says 172.21.15.16:55360) 2023-10-16T13:47:23.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.587+0000 7fe1517ce700 1 -- 172.21.15.16:0/2544765206 learned_addr learned my addr 172.21.15.16:0/2544765206 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:23.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.587+0000 7fe14bfff700 1 --2- 172.21.15.16:0/2544765206 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe14c0ce6a0 0x7fe14c1faf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:23.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.587+0000 7fe1517ce700 1 -- 172.21.15.16:0/2544765206 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe14c0b2620 msgr2=0x7fe14c09d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:23.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.587+0000 7fe1517ce700 1 --2- 172.21.15.16:0/2544765206 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe14c0b2620 0x7fe14c09d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:23.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.587+0000 7fe1517ce700 1 -- 172.21.15.16:0/2544765206 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe14c0ce6a0 msgr2=0x7fe14c1faf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:23.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.587+0000 7fe1517ce700 1 --2- 172.21.15.16:0/2544765206 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe14c0ce6a0 0x7fe14c1faf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:23.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.587+0000 7fe1517ce700 1 -- 172.21.15.16:0/2544765206 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe138009550 con 0x7fe14c0d2110 2023-10-16T13:47:23.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.587+0000 7fe1517ce700 1 --2- 172.21.15.16:0/2544765206 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe14c0d2110 0x7fe14c0b5770 secure :-1 s=READY pgs=2734 cs=0 l=1 rev1=1 crypto rx=0x7fe144007e90 tx=0x7fe14400e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:23.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.587+0000 7fe1497fa700 1 -- 172.21.15.16:0/2544765206 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe14400ed30 con 0x7fe14c0d2110 2023-10-16T13:47:23.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.587+0000 7fe1497fa700 1 -- 172.21.15.16:0/2544765206 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe14400eeb0 con 0x7fe14c0d2110 2023-10-16T13:47:23.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.587+0000 7fe153231700 1 -- 172.21.15.16:0/2544765206 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe14c09f010 con 0x7fe14c0d2110 2023-10-16T13:47:23.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.587+0000 7fe1497fa700 1 -- 172.21.15.16:0/2544765206 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe144014640 con 0x7fe14c0d2110 2023-10-16T13:47:23.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.587+0000 7fe153231700 1 -- 172.21.15.16:0/2544765206 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe14c09c840 con 0x7fe14c0d2110 2023-10-16T13:47:23.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.588+0000 7fe153231700 1 -- 172.21.15.16:0/2544765206 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe14c09d040 con 0x7fe14c0d2110 2023-10-16T13:47:23.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.589+0000 7fe1497fa700 1 -- 172.21.15.16:0/2544765206 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 148) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fe1440147a0 con 0x7fe14c0d2110 2023-10-16T13:47:23.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.589+0000 7fe1497fa700 1 --2- 172.21.15.16:0/2544765206 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fe13006b140 0x7fe13006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:23.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.590+0000 7fe1497fa700 1 -- 172.21.15.16:0/2544765206 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fe14408acb0 con 0x7fe14c0d2110 2023-10-16T13:47:23.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.590+0000 7fe150fcd700 1 --2- 172.21.15.16:0/2544765206 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fe13006b140 0x7fe13006d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:23.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.590+0000 7fe150fcd700 1 --2- 172.21.15.16:0/2544765206 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fe13006b140 0x7fe13006d600 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe140006b20 tx=0x7fe140008040 comp rx=0 tx=0).ready entity=mgr.11886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:23.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.594+0000 7fe1497fa700 1 -- 172.21.15.16:0/2544765206 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe144055ae0 con 0x7fe14c0d2110 2023-10-16T13:47:23.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:23.841+0000 7fe153231700 1 -- 172.21.15.16:0/2544765206 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fe14c09d040 con 0x7fe14c0d2110 2023-10-16T13:47:24.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.424+0000 7fe1497fa700 1 -- 172.21.15.16:0/2544765206 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 31 and data pool 32 v126) v1 ==== 174+0+0 (secure 0 0 0) 0x7fe14c09d040 con 0x7fe14c0d2110 2023-10-16T13:47:24.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.449+0000 7fe153231700 1 -- 172.21.15.16:0/2544765206 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fe13006b140 msgr2=0x7fe13006d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:24.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.449+0000 7fe153231700 1 --2- 172.21.15.16:0/2544765206 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fe13006b140 0x7fe13006d600 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe140006b20 tx=0x7fe140008040 comp rx=0 tx=0).stop 2023-10-16T13:47:24.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.449+0000 7fe153231700 1 -- 172.21.15.16:0/2544765206 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe14c0d2110 msgr2=0x7fe14c0b5770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:24.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.449+0000 7fe153231700 1 --2- 172.21.15.16:0/2544765206 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe14c0d2110 0x7fe14c0b5770 secure :-1 s=READY pgs=2734 cs=0 l=1 rev1=1 crypto rx=0x7fe144007e90 tx=0x7fe14400e3c0 comp rx=0 tx=0).stop 2023-10-16T13:47:24.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.450+0000 7fe153231700 1 -- 172.21.15.16:0/2544765206 shutdown_connections 2023-10-16T13:47:24.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.450+0000 7fe153231700 1 --2- 172.21.15.16:0/2544765206 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fe13006b140 0x7fe13006d600 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:24.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.450+0000 7fe153231700 1 --2- 172.21.15.16:0/2544765206 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe14c0d2110 0x7fe14c0b5770 unknown :-1 s=CLOSED pgs=2734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:24.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.450+0000 7fe153231700 1 --2- 172.21.15.16:0/2544765206 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe14c0b2620 0x7fe14c09d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:24.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.450+0000 7fe153231700 1 --2- 172.21.15.16:0/2544765206 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe14c0ce6a0 0x7fe14c1faf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:24.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.450+0000 7fe153231700 1 -- 172.21.15.16:0/2544765206 >> 172.21.15.16:0/2544765206 conn(0x7fe14c0d15f0 msgr2=0x7fe14c160c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:24.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.450+0000 7fe153231700 1 -- 172.21.15.16:0/2544765206 shutdown_connections 2023-10-16T13:47:24.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.451+0000 7fe153231700 1 -- 172.21.15.16:0/2544765206 wait complete. 2023-10-16T13:47:24.455 INFO:teuthology.orchestra.run.smithi016.stderr:new fs with metadata pool 31 and data pool 32 2023-10-16T13:47:24.469 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:47:24.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.595+0000 7fea9ffdf700 1 Processor -- start 2023-10-16T13:47:24.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.595+0000 7fea9ffdf700 1 -- start start 2023-10-16T13:47:24.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.596+0000 7fea9ffdf700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fea9815b630 0x7fea980cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:24.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.596+0000 7fea9ffdf700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fea9815ba10 0x7fea980cdde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:24.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.596+0000 7fea9ffdf700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fea980ce320 0x7fea980cfca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:24.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.596+0000 7fea9ffdf700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea980cdb20 con 0x7fea980ce320 2023-10-16T13:47:24.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.596+0000 7fea9ffdf700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea980ce770 con 0x7fea9815ba10 2023-10-16T13:47:24.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.596+0000 7fea9ffdf700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea980cc2a0 con 0x7fea9815b630 2023-10-16T13:47:24.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.596+0000 7fea9e57c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fea980ce320 0x7fea980cfca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:24.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.596+0000 7fea9e57c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fea980ce320 0x7fea980cfca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55386/0 (socket says 172.21.15.16:55386) 2023-10-16T13:47:24.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.596+0000 7fea9e57c700 1 -- 172.21.15.16:0/2404962761 learned_addr learned my addr 172.21.15.16:0/2404962761 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:24.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.596+0000 7fea9d57a700 1 --2- 172.21.15.16:0/2404962761 >> v2:172.21.15.49:3300/0 conn(0x7fea9815ba10 0x7fea980cdde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:24.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.596+0000 7fea9dd7b700 1 --2- 172.21.15.16:0/2404962761 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fea9815b630 0x7fea980cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:24.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.597+0000 7fea9e57c700 1 -- 172.21.15.16:0/2404962761 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fea9815b630 msgr2=0x7fea980cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:24.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.597+0000 7fea9e57c700 1 --2- 172.21.15.16:0/2404962761 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fea9815b630 0x7fea980cd5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:24.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.597+0000 7fea9e57c700 1 -- 172.21.15.16:0/2404962761 >> v2:172.21.15.49:3300/0 conn(0x7fea9815ba10 msgr2=0x7fea980cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:24.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.597+0000 7fea9e57c700 1 --2- 172.21.15.16:0/2404962761 >> v2:172.21.15.49:3300/0 conn(0x7fea9815ba10 0x7fea980cdde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:24.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.597+0000 7fea9e57c700 1 -- 172.21.15.16:0/2404962761 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea980cc540 con 0x7fea980ce320 2023-10-16T13:47:24.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.597+0000 7fea9d57a700 1 --2- 172.21.15.16:0/2404962761 >> v2:172.21.15.49:3300/0 conn(0x7fea9815ba10 0x7fea980cdde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:47:24.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.597+0000 7fea9e57c700 1 --2- 172.21.15.16:0/2404962761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea980ce320 0x7fea980cfca0 secure :-1 s=READY pgs=2735 cs=0 l=1 rev1=1 crypto rx=0x7fea980727f0 tx=0x7fea8800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6529ecdbb25479f6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:24.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.597+0000 7fea9cd79700 1 -- 172.21.15.16:0/2404962761 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea8800e070 con 0x7fea980ce320 2023-10-16T13:47:24.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.597+0000 7fea9cd79700 1 -- 172.21.15.16:0/2404962761 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea8800b9c0 con 0x7fea980ce320 2023-10-16T13:47:24.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.597+0000 7fea9cd79700 1 -- 172.21.15.16:0/2404962761 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea8800bd70 con 0x7fea980ce320 2023-10-16T13:47:24.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.598+0000 7fea9ffdf700 1 -- 172.21.15.16:0/2404962761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea980ce320 msgr2=0x7fea980cfca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:24.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.598+0000 7fea9ffdf700 1 --2- 172.21.15.16:0/2404962761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea980ce320 0x7fea980cfca0 secure :-1 s=READY pgs=2735 cs=0 l=1 rev1=1 crypto rx=0x7fea980727f0 tx=0x7fea8800b330 comp rx=0 tx=0).stop 2023-10-16T13:47:24.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.598+0000 7fea9ffdf700 1 -- 172.21.15.16:0/2404962761 shutdown_connections 2023-10-16T13:47:24.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.598+0000 7fea9ffdf700 1 --2- 172.21.15.16:0/2404962761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea980ce320 0x7fea980cfca0 unknown :-1 s=CLOSED pgs=2735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:24.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.598+0000 7fea9ffdf700 1 --2- 172.21.15.16:0/2404962761 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fea9815b630 0x7fea980cd5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:24.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.598+0000 7fea9ffdf700 1 --2- 172.21.15.16:0/2404962761 >> v2:172.21.15.49:3300/0 conn(0x7fea9815ba10 0x7fea980cdde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:24.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.598+0000 7fea9ffdf700 1 -- 172.21.15.16:0/2404962761 >> 172.21.15.16:0/2404962761 conn(0x7fea980d14b0 msgr2=0x7fea980d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:24.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.598+0000 7fea9ffdf700 1 -- 172.21.15.16:0/2404962761 shutdown_connections 2023-10-16T13:47:24.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.598+0000 7fea9ffdf700 1 -- 172.21.15.16:0/2404962761 wait complete. 2023-10-16T13:47:24.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.599+0000 7fea9ffdf700 1 Processor -- start 2023-10-16T13:47:24.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.599+0000 7fea9ffdf700 1 -- start start 2023-10-16T13:47:24.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.600+0000 7fea9ffdf700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fea980ce320 0x7fea98154a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:24.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.600+0000 7fea9ffdf700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fea9815b630 0x7fea980c8580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:24.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.600+0000 7fea9ffdf700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea9815ba10 0x7fea981f3630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:24.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.600+0000 7fea9ffdf700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea98154fc0 con 0x7fea9815ba10 2023-10-16T13:47:24.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.600+0000 7fea9ffdf700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea9806d840 con 0x7fea9815b630 2023-10-16T13:47:24.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.600+0000 7fea9ffdf700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea981f4ee0 con 0x7fea980ce320 2023-10-16T13:47:24.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.600+0000 7fea9e57c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea9815ba10 0x7fea981f3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:24.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.600+0000 7fea9e57c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea9815ba10 0x7fea981f3630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55392/0 (socket says 172.21.15.16:55392) 2023-10-16T13:47:24.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.600+0000 7fea9e57c700 1 -- 172.21.15.16:0/2918064694 learned_addr learned my addr 172.21.15.16:0/2918064694 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:24.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.600+0000 7fea9dd7b700 1 --2- 172.21.15.16:0/2918064694 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fea980ce320 0x7fea98154a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:24.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.600+0000 7fea9d57a700 1 --2- 172.21.15.16:0/2918064694 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fea9815b630 0x7fea980c8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:24.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.600+0000 7fea9e57c700 1 -- 172.21.15.16:0/2918064694 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fea980ce320 msgr2=0x7fea98154a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:24.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.600+0000 7fea9e57c700 1 --2- 172.21.15.16:0/2918064694 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fea980ce320 0x7fea98154a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:24.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.600+0000 7fea9e57c700 1 -- 172.21.15.16:0/2918064694 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fea9815b630 msgr2=0x7fea980c8580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:24.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.600+0000 7fea9e57c700 1 --2- 172.21.15.16:0/2918064694 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fea9815b630 0x7fea980c8580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:24.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.600+0000 7fea9e57c700 1 -- 172.21.15.16:0/2918064694 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea88009b30 con 0x7fea9815ba10 2023-10-16T13:47:24.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.600+0000 7fea9e57c700 1 --2- 172.21.15.16:0/2918064694 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea9815ba10 0x7fea981f3630 secure :-1 s=READY pgs=2736 cs=0 l=1 rev1=1 crypto rx=0x7fea98072680 tx=0x7fea88004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:24.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.600+0000 7fea967fc700 1 -- 172.21.15.16:0/2918064694 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea880099b0 con 0x7fea9815ba10 2023-10-16T13:47:24.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.601+0000 7fea967fc700 1 -- 172.21.15.16:0/2918064694 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea88004250 con 0x7fea9815ba10 2023-10-16T13:47:24.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.601+0000 7fea967fc700 1 -- 172.21.15.16:0/2918064694 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea88004600 con 0x7fea9815ba10 2023-10-16T13:47:24.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.601+0000 7fea9ffdf700 1 -- 172.21.15.16:0/2918064694 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea980c8ac0 con 0x7fea9815ba10 2023-10-16T13:47:24.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.601+0000 7fea9ffdf700 1 -- 172.21.15.16:0/2918064694 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea981f42d0 con 0x7fea9815ba10 2023-10-16T13:47:24.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.601+0000 7fea9ffdf700 1 -- 172.21.15.16:0/2918064694 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea9804ebb0 con 0x7fea9815ba10 2023-10-16T13:47:24.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.602+0000 7fea967fc700 1 -- 172.21.15.16:0/2918064694 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 148) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fea88004760 con 0x7fea9815ba10 2023-10-16T13:47:24.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.602+0000 7fea967fc700 1 --2- 172.21.15.16:0/2918064694 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fea8406b140 0x7fea8406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:24.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.603+0000 7fea967fc700 1 -- 172.21.15.16:0/2918064694 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(244..244 src has 1..244) v4 ==== 6157+0+0 (secure 0 0 0) 0x7fea8808af80 con 0x7fea9815ba10 2023-10-16T13:47:24.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.603+0000 7fea9dd7b700 1 --2- 172.21.15.16:0/2918064694 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fea8406b140 0x7fea8406d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:24.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.603+0000 7fea9dd7b700 1 --2- 172.21.15.16:0/2918064694 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fea8406b140 0x7fea8406d600 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fea90006fd0 tx=0x7fea90008040 comp rx=0 tx=0).ready entity=mgr.11886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:24.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.607+0000 7fea967fc700 1 -- 172.21.15.16:0/2918064694 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fea880bdfe0 con 0x7fea9815ba10 2023-10-16T13:47:24.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.813+0000 7fea9ffdf700 1 -- 172.21.15.16:0/2918064694 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fea9806b5d0 con 0x7fea9815ba10 2023-10-16T13:47:24.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.815+0000 7fea967fc700 1 -- 172.21.15.16:0/2918064694 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v244) v1 ==== 74+0+16894 (secure 0 0 0) 0x7fea8801f020 con 0x7fea9815ba10 2023-10-16T13:47:24.817 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:47:24.818 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":244,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:47:23.843013+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2023-10-16T13:47:22.338349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"244","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2023-10-16T13:47:22.806327+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"244","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":243,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":243,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":243,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":243,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":243,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":243,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":243,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":243,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:47:24.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.817+0000 7fea9ffdf700 1 -- 172.21.15.16:0/2918064694 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fea8406b140 msgr2=0x7fea8406d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:24.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.817+0000 7fea9ffdf700 1 --2- 172.21.15.16:0/2918064694 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fea8406b140 0x7fea8406d600 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fea90006fd0 tx=0x7fea90008040 comp rx=0 tx=0).stop 2023-10-16T13:47:24.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.817+0000 7fea9ffdf700 1 -- 172.21.15.16:0/2918064694 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea9815ba10 msgr2=0x7fea981f3630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:24.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.817+0000 7fea9ffdf700 1 --2- 172.21.15.16:0/2918064694 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea9815ba10 0x7fea981f3630 secure :-1 s=READY pgs=2736 cs=0 l=1 rev1=1 crypto rx=0x7fea98072680 tx=0x7fea88004ad0 comp rx=0 tx=0).stop 2023-10-16T13:47:24.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.817+0000 7fea9ffdf700 1 -- 172.21.15.16:0/2918064694 shutdown_connections 2023-10-16T13:47:24.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.817+0000 7fea9ffdf700 1 --2- 172.21.15.16:0/2918064694 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fea8406b140 0x7fea8406d600 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:24.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.817+0000 7fea9ffdf700 1 --2- 172.21.15.16:0/2918064694 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea9815ba10 0x7fea981f3630 unknown :-1 s=CLOSED pgs=2736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:24.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.817+0000 7fea9ffdf700 1 --2- 172.21.15.16:0/2918064694 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fea980ce320 0x7fea98154a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:24.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.817+0000 7fea9ffdf700 1 --2- 172.21.15.16:0/2918064694 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fea9815b630 0x7fea980c8580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:24.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.817+0000 7fea9ffdf700 1 -- 172.21.15.16:0/2918064694 >> 172.21.15.16:0/2918064694 conn(0x7fea980d14b0 msgr2=0x7fea980b5260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:24.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.817+0000 7fea9ffdf700 1 -- 172.21.15.16:0/2918064694 shutdown_connections 2023-10-16T13:47:24.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.817+0000 7fea9ffdf700 1 -- 172.21.15.16:0/2918064694 wait complete. 2023-10-16T13:47:24.832 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:47:24.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.954+0000 7f6328250700 1 Processor -- start 2023-10-16T13:47:24.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.954+0000 7f6328250700 1 -- start start 2023-10-16T13:47:24.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.955+0000 7f6328250700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f632015b280 0x7f63201613e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:24.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.955+0000 7f6328250700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f632006a7e0 0x7f632006ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:24.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.955+0000 7f6328250700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f632006b150 con 0x7f63200cc8a0 2023-10-16T13:47:24.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.955+0000 7f6328250700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6320161c30 con 0x7f632015b280 2023-10-16T13:47:24.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.955+0000 7f6328250700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6320161f10 con 0x7f632006a7e0 2023-10-16T13:47:24.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.955+0000 7f6325fec700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f632015b280 0x7f63201613e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:24.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.955+0000 7f6325fec700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f632015b280 0x7f63201613e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55396/0 (socket says 172.21.15.16:55396) 2023-10-16T13:47:24.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.955+0000 7f6325fec700 1 -- 172.21.15.16:0/1045225454 learned_addr learned my addr 172.21.15.16:0/1045225454 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:24.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.956+0000 7f63267ed700 1 --2- 172.21.15.16:0/1045225454 >> v2:172.21.15.49:3300/0 conn(0x7f632006a7e0 0x7f632006ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:24.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.956+0000 7f6325fec700 1 -- 172.21.15.16:0/1045225454 >> v2:172.21.15.49:3300/0 conn(0x7f632006a7e0 msgr2=0x7f632006ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:24.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.956+0000 7f6325fec700 1 --2- 172.21.15.16:0/1045225454 >> v2:172.21.15.49:3300/0 conn(0x7f632006a7e0 0x7f632006ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:24.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.956+0000 7f6325fec700 1 -- 172.21.15.16:0/1045225454 >> v1:172.21.15.49:6789/0 conn(0x7f63200cc8a0 legacy=0x7f632015e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:24.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.956+0000 7f6325fec700 1 -- 172.21.15.16:0/1045225454 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63201621f0 con 0x7f632015b280 2023-10-16T13:47:24.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.956+0000 7f6325fec700 1 --2- 172.21.15.16:0/1045225454 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f632015b280 0x7f63201613e0 secure :-1 s=READY pgs=2737 cs=0 l=1 rev1=1 crypto rx=0x7f63200cbed0 tx=0x7f6310009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adaddbf42bd22fdf server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:24.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.956+0000 7f6324fea700 1 -- 172.21.15.16:0/1045225454 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6310011830 con 0x7f632015b280 2023-10-16T13:47:24.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.956+0000 7f6324fea700 1 -- 172.21.15.16:0/1045225454 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6310011990 con 0x7f632015b280 2023-10-16T13:47:24.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.956+0000 7f6324fea700 1 -- 172.21.15.16:0/1045225454 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6310011d20 con 0x7f632015b280 2023-10-16T13:47:24.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.957+0000 7f6328250700 1 -- 172.21.15.16:0/1045225454 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f632015b280 msgr2=0x7f63201613e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:24.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.957+0000 7f6328250700 1 --2- 172.21.15.16:0/1045225454 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f632015b280 0x7f63201613e0 secure :-1 s=READY pgs=2737 cs=0 l=1 rev1=1 crypto rx=0x7f63200cbed0 tx=0x7f6310009bd0 comp rx=0 tx=0).stop 2023-10-16T13:47:24.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.957+0000 7f6328250700 1 -- 172.21.15.16:0/1045225454 shutdown_connections 2023-10-16T13:47:24.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.957+0000 7f6328250700 1 --2- 172.21.15.16:0/1045225454 >> v2:172.21.15.49:3300/0 conn(0x7f632006a7e0 0x7f632006ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:24.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.957+0000 7f6328250700 1 --2- 172.21.15.16:0/1045225454 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f632015b280 0x7f63201613e0 unknown :-1 s=CLOSED pgs=2737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:24.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.957+0000 7f6328250700 1 -- 172.21.15.16:0/1045225454 >> 172.21.15.16:0/1045225454 conn(0x7f63200b1fd0 msgr2=0x7f63200b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:24.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.957+0000 7f6328250700 1 -- 172.21.15.16:0/1045225454 shutdown_connections 2023-10-16T13:47:24.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.957+0000 7f6328250700 1 -- 172.21.15.16:0/1045225454 wait complete. 2023-10-16T13:47:24.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.958+0000 7f6328250700 1 Processor -- start 2023-10-16T13:47:24.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.958+0000 7f6328250700 1 -- start start 2023-10-16T13:47:24.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.959+0000 7f6328250700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f632006a7e0 0x7f6320151c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:24.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.959+0000 7f6328250700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f63200cc8a0 0x7f632015fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:24.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.959+0000 7f6328250700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f632015b280 0x7f63200c3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:24.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.959+0000 7f6328250700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6320152150 con 0x7f63200cc8a0 2023-10-16T13:47:24.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.959+0000 7f6328250700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6320160050 con 0x7f632006a7e0 2023-10-16T13:47:24.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.959+0000 7f6328250700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f632015ea70 con 0x7f632015b280 2023-10-16T13:47:24.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.959+0000 7f63257eb700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f63200cc8a0 0x7f632015fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:24.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.959+0000 7f63257eb700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f63200cc8a0 0x7f632015fb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55402/0 (socket says 172.21.15.16:55402) 2023-10-16T13:47:24.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.959+0000 7f63257eb700 1 -- 172.21.15.16:0/2963670586 learned_addr learned my addr 172.21.15.16:0/2963670586 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:24.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.959+0000 7f6325fec700 1 --2- 172.21.15.16:0/2963670586 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f632006a7e0 0x7f6320151c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:24.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.959+0000 7f63267ed700 1 --2- 172.21.15.16:0/2963670586 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f632015b280 0x7f63200c3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:24.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.959+0000 7f63257eb700 1 -- 172.21.15.16:0/2963670586 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f632015b280 msgr2=0x7f63200c3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:24.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.959+0000 7f63257eb700 1 --2- 172.21.15.16:0/2963670586 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f632015b280 0x7f63200c3f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:24.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.959+0000 7f63257eb700 1 -- 172.21.15.16:0/2963670586 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f632006a7e0 msgr2=0x7f6320151c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:24.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.959+0000 7f63257eb700 1 --2- 172.21.15.16:0/2963670586 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f632006a7e0 0x7f6320151c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:24.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.959+0000 7f63257eb700 1 -- 172.21.15.16:0/2963670586 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6310009550 con 0x7f63200cc8a0 2023-10-16T13:47:24.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.959+0000 7f63267ed700 1 --2- 172.21.15.16:0/2963670586 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f632015b280 0x7f63200c3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:47:24.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.959+0000 7f63257eb700 1 --2- 172.21.15.16:0/2963670586 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f63200cc8a0 0x7f632015fb10 secure :-1 s=READY pgs=2738 cs=0 l=1 rev1=1 crypto rx=0x7f631c00c4b0 tx=0x7f631c00c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:24.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.959+0000 7f63167fc700 1 -- 172.21.15.16:0/2963670586 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f631c0052d0 con 0x7f63200cc8a0 2023-10-16T13:47:24.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.959+0000 7f63167fc700 1 -- 172.21.15.16:0/2963670586 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f631c005430 con 0x7f63200cc8a0 2023-10-16T13:47:24.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.959+0000 7f6328250700 1 -- 172.21.15.16:0/2963670586 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f632015ed50 con 0x7f63200cc8a0 2023-10-16T13:47:24.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.959+0000 7f63167fc700 1 -- 172.21.15.16:0/2963670586 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f631c0038c0 con 0x7f63200cc8a0 2023-10-16T13:47:24.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.960+0000 7f6328250700 1 -- 172.21.15.16:0/2963670586 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f632015f310 con 0x7f63200cc8a0 2023-10-16T13:47:24.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.960+0000 7f6328250700 1 -- 172.21.15.16:0/2963670586 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f632004ebb0 con 0x7f63200cc8a0 2023-10-16T13:47:24.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.961+0000 7f63167fc700 1 -- 172.21.15.16:0/2963670586 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 148) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f631c003a20 con 0x7f63200cc8a0 2023-10-16T13:47:24.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.961+0000 7f63167fc700 1 --2- 172.21.15.16:0/2963670586 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f630406b140 0x7f630406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:24.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.961+0000 7f63167fc700 1 -- 172.21.15.16:0/2963670586 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(244..244 src has 1..244) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f631c088f00 con 0x7f63200cc8a0 2023-10-16T13:47:24.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.961+0000 7f6325fec700 1 --2- 172.21.15.16:0/2963670586 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f630406b140 0x7f630406d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:24.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.962+0000 7f6325fec700 1 --2- 172.21.15.16:0/2963670586 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f630406b140 0x7f630406d600 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6310009c10 tx=0x7f6310008030 comp rx=0 tx=0).ready entity=mgr.11886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:24.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:24.966+0000 7f63167fc700 1 -- 172.21.15.16:0/2963670586 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f631c053b60 con 0x7f63200cc8a0 2023-10-16T13:47:25.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:25.170+0000 7f6328250700 1 -- 172.21.15.16:0/2963670586 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f632015da20 con 0x7f63200cc8a0 2023-10-16T13:47:25.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:25.171+0000 7f63167fc700 1 -- 172.21.15.16:0/2963670586 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v244) v1 ==== 74+0+16894 (secure 0 0 0) 0x7f631c017090 con 0x7f63200cc8a0 2023-10-16T13:47:25.174 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:47:25.174 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":244,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:47:23.843013+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2023-10-16T13:47:22.338349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"244","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2023-10-16T13:47:22.806327+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"244","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":243,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":243,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":243,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":243,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":243,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":243,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":243,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":243,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:47:25.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:25.173+0000 7f6328250700 1 -- 172.21.15.16:0/2963670586 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f630406b140 msgr2=0x7f630406d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:25.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:25.173+0000 7f6328250700 1 --2- 172.21.15.16:0/2963670586 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f630406b140 0x7f630406d600 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6310009c10 tx=0x7f6310008030 comp rx=0 tx=0).stop 2023-10-16T13:47:25.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:25.173+0000 7f6328250700 1 -- 172.21.15.16:0/2963670586 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f63200cc8a0 msgr2=0x7f632015fb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:25.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:25.173+0000 7f6328250700 1 --2- 172.21.15.16:0/2963670586 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f63200cc8a0 0x7f632015fb10 secure :-1 s=READY pgs=2738 cs=0 l=1 rev1=1 crypto rx=0x7f631c00c4b0 tx=0x7f631c00c7c0 comp rx=0 tx=0).stop 2023-10-16T13:47:25.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:25.173+0000 7f6328250700 1 -- 172.21.15.16:0/2963670586 shutdown_connections 2023-10-16T13:47:25.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:25.173+0000 7f6328250700 1 --2- 172.21.15.16:0/2963670586 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f630406b140 0x7f630406d600 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:25.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:25.173+0000 7f6328250700 1 --2- 172.21.15.16:0/2963670586 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f632015b280 0x7f63200c3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:25.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:25.173+0000 7f6328250700 1 --2- 172.21.15.16:0/2963670586 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f632006a7e0 0x7f6320151c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:25.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:25.173+0000 7f6328250700 1 --2- 172.21.15.16:0/2963670586 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f63200cc8a0 0x7f632015fb10 unknown :-1 s=CLOSED pgs=2738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:25.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:25.173+0000 7f6328250700 1 -- 172.21.15.16:0/2963670586 >> 172.21.15.16:0/2963670586 conn(0x7f63200b1fd0 msgr2=0x7f63200b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:25.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:25.173+0000 7f6328250700 1 -- 172.21.15.16:0/2963670586 shutdown_connections 2023-10-16T13:47:25.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:25.173+0000 7f6328250700 1 -- 172.21.15.16:0/2963670586 wait complete. 2023-10-16T13:47:25.188 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-16T13:47:25.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:25.314+0000 7fb5ef2c1700 1 Processor -- start 2023-10-16T13:47:25.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:25.314+0000 7fb5ef2c1700 1 -- start start 2023-10-16T13:47:25.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:25.315+0000 7fb5ef2c1700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb5e80cb880 0x7fb5e80cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:25.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:25.315+0000 7fb5ef2c1700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fb5e806da60 0x7fb5e806def0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:25.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:25.315+0000 7fb5ef2c1700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5e80cef70 con 0x7fb5e815b070 2023-10-16T13:47:25.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:25.315+0000 7fb5ef2c1700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5e806e430 con 0x7fb5e80cb880 2023-10-16T13:47:25.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:25.315+0000 7fb5ef2c1700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5e80ce360 con 0x7fb5e806da60 2023-10-16T13:47:25.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:25.315+0000 7fb5ec85c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb5e80cb880 0x7fb5e80cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:25.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:25.315+0000 7fb5ec85c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb5e80cb880 0x7fb5e80cbd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55410/0 (socket says 172.21.15.16:55410) 2023-10-16T13:47:25.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:25.315+0000 7fb5ec85c700 1 -- 172.21.15.16:0/1404805862 learned_addr learned my addr 172.21.15.16:0/1404805862 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:25.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:25.315+0000 7fb5ed85e700 1 --2- 172.21.15.16:0/1404805862 >> v2:172.21.15.49:3300/0 conn(0x7fb5e806da60 0x7fb5e806def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:25.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:25.315+0000 7fb5ec85c700 1 -- 172.21.15.16:0/1404805862 >> v2:172.21.15.49:3300/0 conn(0x7fb5e806da60 msgr2=0x7fb5e806def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:25.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:25.315+0000 7fb5ec85c700 1 --2- 172.21.15.16:0/1404805862 >> v2:172.21.15.49:3300/0 conn(0x7fb5e806da60 0x7fb5e806def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:25.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:25.315+0000 7fb5ec85c700 1 -- 172.21.15.16:0/1404805862 >> v1:172.21.15.16:6789/0 conn(0x7fb5e815b070 legacy=0x7fb5e815b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:25.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:25.315+0000 7fb5dffff700 1 -- 172.21.15.16:0/1404805862 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2624371544 0 0) 0x7fb5e80cef70 con 0x7fb5e815b070 2023-10-16T13:47:25.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:25.315+0000 7fb5ec85c700 1 -- 172.21.15.16:0/1404805862 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5e80ce640 con 0x7fb5e80cb880 2023-10-16T13:47:25.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:25.316+0000 7fb5ec85c700 1 --2- 172.21.15.16:0/1404805862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5e80cb880 0x7fb5e80cbd00 secure :-1 s=READY pgs=2740 cs=0 l=1 rev1=1 crypto rx=0x7fb5e806edc0 tx=0x7fb5e4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6885d55132d9cfdf server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:25.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:25.316+0000 7fb5dffff700 1 -- 172.21.15.16:0/1404805862 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5e4011830 con 0x7fb5e80cb880 2023-10-16T13:47:25.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:25.316+0000 7fb5dffff700 1 -- 172.21.15.16:0/1404805862 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5e4011990 con 0x7fb5e80cb880 2023-10-16T13:47:25.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:25.316+0000 7fb5dffff700 1 -- 172.21.15.16:0/1404805862 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5e4011d20 con 0x7fb5e80cb880 2023-10-16T13:47:25.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:25.316+0000 7fb5ef2c1700 1 -- 172.21.15.16:0/1404805862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5e80cb880 msgr2=0x7fb5e80cbd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:25.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:25.316+0000 7fb5ef2c1700 1 --2- 172.21.15.16:0/1404805862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5e80cb880 0x7fb5e80cbd00 secure :-1 s=READY pgs=2740 cs=0 l=1 rev1=1 crypto rx=0x7fb5e806edc0 tx=0x7fb5e4009bd0 comp rx=0 tx=0).stop 2023-10-16T13:47:25.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:25.316+0000 7fb5ef2c1700 1 -- 172.21.15.16:0/1404805862 shutdown_connections 2023-10-16T13:47:25.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:25.316+0000 7fb5ef2c1700 1 --2- 172.21.15.16:0/1404805862 >> v2:172.21.15.49:3300/0 conn(0x7fb5e806da60 0x7fb5e806def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:25.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:25.316+0000 7fb5ef2c1700 1 --2- 172.21.15.16:0/1404805862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5e80cb880 0x7fb5e80cbd00 unknown :-1 s=CLOSED pgs=2740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:25.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:25.316+0000 7fb5ef2c1700 1 -- 172.21.15.16:0/1404805862 >> 172.21.15.16:0/1404805862 conn(0x7fb5e80b1f40 msgr2=0x7fb5e80b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:25.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:25.316+0000 7fb5ef2c1700 1 -- 172.21.15.16:0/1404805862 shutdown_connections 2023-10-16T13:47:25.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:25.316+0000 7fb5ef2c1700 1 -- 172.21.15.16:0/1404805862 wait complete. 2023-10-16T13:47:25.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:25.318+0000 7fb5ef2c1700 1 Processor -- start 2023-10-16T13:47:25.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:25.318+0000 7fb5ef2c1700 1 -- start start 2023-10-16T13:47:25.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:25.318+0000 7fb5ef2c1700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb5e806da60 0x7fb5e815d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:25.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:25.318+0000 7fb5ef2c1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5e80cb880 0x7fb5e80c90d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:25.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:25.318+0000 7fb5ef2c1700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb5e815b070 0x7fb5e80ad920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:25.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:25.318+0000 7fb5ef2c1700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e80c8cb0 con 0x7fb5e80cb880 2023-10-16T13:47:25.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:25.318+0000 7fb5ef2c1700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e815dbe0 con 0x7fb5e815b070 2023-10-16T13:47:25.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:25.318+0000 7fb5ef2c1700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e8154430 con 0x7fb5e806da60 2023-10-16T13:47:25.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:25.318+0000 7fb5ec85c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5e80cb880 0x7fb5e80c90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:25.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:25.319+0000 7fb5ec85c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5e80cb880 0x7fb5e80c90d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55416/0 (socket says 172.21.15.16:55416) 2023-10-16T13:47:25.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:25.319+0000 7fb5ec85c700 1 -- 172.21.15.16:0/3578401067 learned_addr learned my addr 172.21.15.16:0/3578401067 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:25.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:25.319+0000 7fb5ed05d700 1 --2- 172.21.15.16:0/3578401067 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb5e806da60 0x7fb5e815d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:25.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:25.319+0000 7fb5ed85e700 1 --2- 172.21.15.16:0/3578401067 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb5e815b070 0x7fb5e80ad920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:25.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:25.319+0000 7fb5ec85c700 1 -- 172.21.15.16:0/3578401067 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb5e806da60 msgr2=0x7fb5e815d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:25.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:25.319+0000 7fb5ec85c700 1 --2- 172.21.15.16:0/3578401067 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb5e806da60 0x7fb5e815d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:25.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:25.319+0000 7fb5ec85c700 1 -- 172.21.15.16:0/3578401067 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb5e815b070 msgr2=0x7fb5e80ad920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:25.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:25.319+0000 7fb5ec85c700 1 --2- 172.21.15.16:0/3578401067 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb5e815b070 0x7fb5e80ad920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:25.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:25.319+0000 7fb5ec85c700 1 -- 172.21.15.16:0/3578401067 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5e4009550 con 0x7fb5e80cb880 2023-10-16T13:47:25.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:25.319+0000 7fb5ed85e700 1 --2- 172.21.15.16:0/3578401067 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb5e815b070 0x7fb5e80ad920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:47:25.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:25.319+0000 7fb5ec85c700 1 --2- 172.21.15.16:0/3578401067 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5e80cb880 0x7fb5e80c90d0 secure :-1 s=READY pgs=2741 cs=0 l=1 rev1=1 crypto rx=0x7fb5e4009c10 tx=0x7fb5e4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:25.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:25.319+0000 7fb5ddffb700 1 -- 172.21.15.16:0/3578401067 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5e40036a0 con 0x7fb5e80cb880 2023-10-16T13:47:25.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:25.319+0000 7fb5ddffb700 1 -- 172.21.15.16:0/3578401067 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5e4003800 con 0x7fb5e80cb880 2023-10-16T13:47:25.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:25.319+0000 7fb5ddffb700 1 -- 172.21.15.16:0/3578401067 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5e4003b70 con 0x7fb5e80cb880 2023-10-16T13:47:25.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:25.319+0000 7fb5ef2c1700 1 -- 172.21.15.16:0/3578401067 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5e80c8070 con 0x7fb5e80cb880 2023-10-16T13:47:25.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:25.319+0000 7fb5ef2c1700 1 -- 172.21.15.16:0/3578401067 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5e80c85f0 con 0x7fb5e80cb880 2023-10-16T13:47:25.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:25.320+0000 7fb5ef2c1700 1 -- 172.21.15.16:0/3578401067 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5e80c8df0 con 0x7fb5e80cb880 2023-10-16T13:47:25.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:25.321+0000 7fb5ddffb700 1 -- 172.21.15.16:0/3578401067 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 148) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fb5e4003cd0 con 0x7fb5e80cb880 2023-10-16T13:47:25.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:25.321+0000 7fb5ddffb700 1 --2- 172.21.15.16:0/3578401067 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fb5cc06b140 0x7fb5cc06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:25.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:25.321+0000 7fb5ed05d700 1 --2- 172.21.15.16:0/3578401067 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fb5cc06b140 0x7fb5cc06d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:25.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:25.321+0000 7fb5ddffb700 1 -- 172.21.15.16:0/3578401067 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(244..244 src has 1..244) v4 ==== 6157+0+0 (secure 0 0 0) 0x7fb5e4021030 con 0x7fb5e80cb880 2023-10-16T13:47:25.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:25.322+0000 7fb5ed05d700 1 --2- 172.21.15.16:0/3578401067 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fb5cc06b140 0x7fb5cc06d600 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb5d8000f60 tx=0x7fb5d8009040 comp rx=0 tx=0).ready entity=mgr.11886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:25.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:25.326+0000 7fb5ddffb700 1 -- 172.21.15.16:0/3578401067 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb5e80c8df0 con 0x7fb5e80cb880 2023-10-16T13:47:25.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:25.583+0000 7fb5ef2c1700 1 -- 172.21.15.16:0/3578401067 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fb5e80c8df0 con 0x7fb5e80cb880 2023-10-16T13:47:26.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.433+0000 7fb5ddffb700 1 -- 172.21.15.16:0/3578401067 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v129) v1 ==== 118+0+0 (secure 0 0 0) 0x7fb5e80c8df0 con 0x7fb5e80cb880 2023-10-16T13:47:26.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.436+0000 7fb5ef2c1700 1 -- 172.21.15.16:0/3578401067 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fb5cc06b140 msgr2=0x7fb5cc06d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:26.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.436+0000 7fb5ef2c1700 1 --2- 172.21.15.16:0/3578401067 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fb5cc06b140 0x7fb5cc06d600 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb5d8000f60 tx=0x7fb5d8009040 comp rx=0 tx=0).stop 2023-10-16T13:47:26.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.436+0000 7fb5ef2c1700 1 -- 172.21.15.16:0/3578401067 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5e80cb880 msgr2=0x7fb5e80c90d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:26.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.436+0000 7fb5ef2c1700 1 --2- 172.21.15.16:0/3578401067 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5e80cb880 0x7fb5e80c90d0 secure :-1 s=READY pgs=2741 cs=0 l=1 rev1=1 crypto rx=0x7fb5e4009c10 tx=0x7fb5e4004750 comp rx=0 tx=0).stop 2023-10-16T13:47:26.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.436+0000 7fb5ef2c1700 1 -- 172.21.15.16:0/3578401067 shutdown_connections 2023-10-16T13:47:26.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.436+0000 7fb5ef2c1700 1 --2- 172.21.15.16:0/3578401067 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fb5cc06b140 0x7fb5cc06d600 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:26.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.436+0000 7fb5ef2c1700 1 --2- 172.21.15.16:0/3578401067 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb5e815b070 0x7fb5e80ad920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:26.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.436+0000 7fb5ef2c1700 1 --2- 172.21.15.16:0/3578401067 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb5e806da60 0x7fb5e815d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:26.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.436+0000 7fb5ef2c1700 1 --2- 172.21.15.16:0/3578401067 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5e80cb880 0x7fb5e80c90d0 unknown :-1 s=CLOSED pgs=2741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:26.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.436+0000 7fb5ef2c1700 1 -- 172.21.15.16:0/3578401067 >> 172.21.15.16:0/3578401067 conn(0x7fb5e80b1f40 msgr2=0x7fb5e80cf400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:26.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.436+0000 7fb5ef2c1700 1 -- 172.21.15.16:0/3578401067 shutdown_connections 2023-10-16T13:47:26.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.436+0000 7fb5ef2c1700 1 -- 172.21.15.16:0/3578401067 wait complete. 2023-10-16T13:47:26.454 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-16T13:47:26.454 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-16T13:47:26.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.587+0000 7fbc0c6b7700 1 Processor -- start 2023-10-16T13:47:26.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.587+0000 7fbc0c6b7700 1 -- start start 2023-10-16T13:47:26.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.588+0000 7fbc0c6b7700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fbc040ce6a0 0x7fbc0406c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:26.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.588+0000 7fbc0c6b7700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbc040b2620 0x7fbc0406c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:26.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.588+0000 7fbc0c6b7700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc040b20f0 con 0x7fbc040d2110 2023-10-16T13:47:26.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.588+0000 7fbc0c6b7700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc040af1f0 con 0x7fbc040ce6a0 2023-10-16T13:47:26.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.588+0000 7fbc0c6b7700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc040d03d0 con 0x7fbc040b2620 2023-10-16T13:47:26.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.588+0000 7fbc09c52700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fbc040d2110 0x7fbc040b0f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44696/0 (socket says 172.21.15.16:44696) 2023-10-16T13:47:26.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.588+0000 7fbc09c52700 1 -- 172.21.15.16:0/1640901365 learned_addr learned my addr 172.21.15.16:0/1640901365 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:47:26.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.588+0000 7fbc0a453700 1 --2- 172.21.15.16:0/1640901365 >> v2:172.21.15.49:3300/0 conn(0x7fbc040ce6a0 0x7fbc0406c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:26.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.588+0000 7fbc0ac54700 1 --2- 172.21.15.16:0/1640901365 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbc040b2620 0x7fbc0406c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:26.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.588+0000 7fbc09451700 1 -- 172.21.15.16:0/1640901365 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2952970547 0 0) 0x7fbc040b20f0 con 0x7fbc040d2110 2023-10-16T13:47:26.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.588+0000 7fbc09451700 1 -- 172.21.15.16:0/1640901365 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbf00035b0 con 0x7fbc040d2110 2023-10-16T13:47:26.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.589+0000 7fbc09451700 1 -- 172.21.15.16:0/1640901365 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2567088230 0 0) 0x7fbbf00035b0 con 0x7fbc040d2110 2023-10-16T13:47:26.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.589+0000 7fbc09451700 1 -- 172.21.15.16:0/1640901365 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbc040b2620 msgr2=0x7fbc0406c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:26.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.589+0000 7fbc09451700 1 --2- 172.21.15.16:0/1640901365 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbc040b2620 0x7fbc0406c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:26.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.589+0000 7fbc09451700 1 -- 172.21.15.16:0/1640901365 >> v2:172.21.15.49:3300/0 conn(0x7fbc040ce6a0 msgr2=0x7fbc0406c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:26.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.589+0000 7fbc09451700 1 --2- 172.21.15.16:0/1640901365 >> v2:172.21.15.49:3300/0 conn(0x7fbc040ce6a0 0x7fbc0406c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:26.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.589+0000 7fbc09451700 1 -- 172.21.15.16:0/1640901365 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc040b2340 con 0x7fbc040d2110 2023-10-16T13:47:26.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.589+0000 7fbc09451700 1 -- 172.21.15.16:0/1640901365 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7fbbf4001670 con 0x7fbc040d2110 2023-10-16T13:47:26.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.589+0000 7fbc09451700 1 -- 172.21.15.16:0/1640901365 >> v1:172.21.15.16:6789/0 conn(0x7fbc040d2110 legacy=0x7fbc040b0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:26.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.589+0000 7fbc09451700 1 --2- 172.21.15.16:0/1640901365 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbbf0003a00 0x7fbbf0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:26.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.589+0000 7fbc09451700 1 --2- 172.21.15.16:0/1640901365 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbf0004850 0x7fbbf0004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:26.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.589+0000 7fbc09451700 1 --2- 172.21.15.16:0/1640901365 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbc040b2620 0x7fbbf0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:26.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.589+0000 7fbc09451700 1 -- 172.21.15.16:0/1640901365 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc040af1f0 con 0x7fbbf0004850 2023-10-16T13:47:26.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.589+0000 7fbc09451700 1 -- 172.21.15.16:0/1640901365 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc040d03d0 con 0x7fbbf0003a00 2023-10-16T13:47:26.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.589+0000 7fbc09451700 1 -- 172.21.15.16:0/1640901365 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbf000ba10 con 0x7fbc040b2620 2023-10-16T13:47:26.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.589+0000 7fbc09c52700 1 --2- 172.21.15.16:0/1640901365 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbf0004850 0x7fbbf0004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:26.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.589+0000 7fbc0a453700 1 --2- 172.21.15.16:0/1640901365 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbbf0003a00 0x7fbbf0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:26.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.589+0000 7fbc0ac54700 1 --2- 172.21.15.16:0/1640901365 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbc040b2620 0x7fbbf0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:26.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.589+0000 7fbc09c52700 1 -- 172.21.15.16:0/1640901365 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbc040b2620 msgr2=0x7fbbf0005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:26.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.589+0000 7fbc09c52700 1 --2- 172.21.15.16:0/1640901365 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbc040b2620 0x7fbbf0005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:26.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.589+0000 7fbc09c52700 1 -- 172.21.15.16:0/1640901365 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbbf0003a00 msgr2=0x7fbbf0003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:26.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.589+0000 7fbc09c52700 1 --2- 172.21.15.16:0/1640901365 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbbf0003a00 0x7fbbf0003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:26.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.589+0000 7fbc09c52700 1 -- 172.21.15.16:0/1640901365 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbf00035b0 con 0x7fbbf0004850 2023-10-16T13:47:26.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.589+0000 7fbc09c52700 1 --2- 172.21.15.16:0/1640901365 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbf0004850 0x7fbbf0004cd0 secure :-1 s=READY pgs=2743 cs=0 l=1 rev1=1 crypto rx=0x7fbc04072b10 tx=0x7fbbf0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=671a039d1d601462 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:26.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.589+0000 7fbc09451700 1 -- 172.21.15.16:0/1640901365 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbf400cdd0 con 0x7fbbf0004850 2023-10-16T13:47:26.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.589+0000 7fbc09451700 1 -- 172.21.15.16:0/1640901365 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbf4005960 con 0x7fbbf0004850 2023-10-16T13:47:26.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.589+0000 7fbc09451700 1 -- 172.21.15.16:0/1640901365 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbf4005d10 con 0x7fbbf0004850 2023-10-16T13:47:26.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.590+0000 7fbc0c6b7700 1 -- 172.21.15.16:0/1640901365 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbf0004850 msgr2=0x7fbbf0004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:26.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.590+0000 7fbc0c6b7700 1 --2- 172.21.15.16:0/1640901365 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbf0004850 0x7fbbf0004cd0 secure :-1 s=READY pgs=2743 cs=0 l=1 rev1=1 crypto rx=0x7fbc04072b10 tx=0x7fbbf0005880 comp rx=0 tx=0).stop 2023-10-16T13:47:26.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.590+0000 7fbc0ac54700 1 -- 172.21.15.16:0/1640901365 reap_dead start 2023-10-16T13:47:26.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.590+0000 7fbc0c6b7700 1 -- 172.21.15.16:0/1640901365 shutdown_connections 2023-10-16T13:47:26.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.590+0000 7fbc0c6b7700 1 -- 172.21.15.16:0/1640901365 >> 172.21.15.16:0/1640901365 conn(0x7fbc040d15f0 msgr2=0x7fbc040d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:26.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.590+0000 7fbc0c6b7700 1 -- 172.21.15.16:0/1640901365 shutdown_connections 2023-10-16T13:47:26.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.590+0000 7fbc0c6b7700 1 -- 172.21.15.16:0/1640901365 wait complete. 2023-10-16T13:47:26.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.591+0000 7fbc0c6b7700 1 Processor -- start 2023-10-16T13:47:26.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.591+0000 7fbc0c6b7700 1 -- start start 2023-10-16T13:47:26.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.592+0000 7fbc0c6b7700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbbf0004850 0x7fbc040c90d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:26.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.592+0000 7fbc0c6b7700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbc040c6250 0x7fbc0414d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:26.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.592+0000 7fbc0c6b7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc040c6f50 0x7fbc040c73d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:26.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.592+0000 7fbc0c6b7700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc040c7910 con 0x7fbc040c6f50 2023-10-16T13:47:26.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.592+0000 7fbc0c6b7700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc04158540 con 0x7fbbf0004850 2023-10-16T13:47:26.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.592+0000 7fbc0c6b7700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc040ca230 con 0x7fbc040c6250 2023-10-16T13:47:26.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.592+0000 7fbc0ac54700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc040c6f50 0x7fbc040c73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:26.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.592+0000 7fbc0ac54700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc040c6f50 0x7fbc040c73d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55438/0 (socket says 172.21.15.16:55438) 2023-10-16T13:47:26.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.592+0000 7fbc0ac54700 1 -- 172.21.15.16:0/696383542 learned_addr learned my addr 172.21.15.16:0/696383542 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:26.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.592+0000 7fbc0a453700 1 --2- 172.21.15.16:0/696383542 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbbf0004850 0x7fbc040c90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:26.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.592+0000 7fbc09c52700 1 --2- 172.21.15.16:0/696383542 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbc040c6250 0x7fbc0414d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:26.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.593+0000 7fbc0ac54700 1 -- 172.21.15.16:0/696383542 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbc040c6250 msgr2=0x7fbc0414d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:26.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.593+0000 7fbc0ac54700 1 --2- 172.21.15.16:0/696383542 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbc040c6250 0x7fbc0414d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:26.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.593+0000 7fbc0ac54700 1 -- 172.21.15.16:0/696383542 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbbf0004850 msgr2=0x7fbc040c90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:26.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.593+0000 7fbc0ac54700 1 --2- 172.21.15.16:0/696383542 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbbf0004850 0x7fbc040c90d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:26.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.593+0000 7fbc0ac54700 1 -- 172.21.15.16:0/696383542 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc040c96b0 con 0x7fbc040c6f50 2023-10-16T13:47:26.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.593+0000 7fbc0ac54700 1 --2- 172.21.15.16:0/696383542 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc040c6f50 0x7fbc040c73d0 secure :-1 s=READY pgs=2744 cs=0 l=1 rev1=1 crypto rx=0x7fbc0406ea20 tx=0x7fbbfc011ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:26.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.593+0000 7fbbfaffd700 1 -- 172.21.15.16:0/696383542 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbfc00e040 con 0x7fbc040c6f50 2023-10-16T13:47:26.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.593+0000 7fbbfaffd700 1 -- 172.21.15.16:0/696383542 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbfc00f040 con 0x7fbc040c6f50 2023-10-16T13:47:26.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.593+0000 7fbc0c6b7700 1 -- 172.21.15.16:0/696383542 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc040c9990 con 0x7fbc040c6f50 2023-10-16T13:47:26.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.593+0000 7fbbfaffd700 1 -- 172.21.15.16:0/696383542 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbfc00fc70 con 0x7fbc040c6f50 2023-10-16T13:47:26.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.593+0000 7fbc0c6b7700 1 -- 172.21.15.16:0/696383542 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc040c9f50 con 0x7fbc040c6f50 2023-10-16T13:47:26.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.594+0000 7fbc0c6b7700 1 -- 172.21.15.16:0/696383542 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc04158680 con 0x7fbc040c6f50 2023-10-16T13:47:26.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.594+0000 7fbbfaffd700 1 -- 172.21.15.16:0/696383542 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 148) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fbbfc022460 con 0x7fbc040c6f50 2023-10-16T13:47:26.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.595+0000 7fbbfaffd700 1 --2- 172.21.15.16:0/696383542 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fbbe806b170 0x7fbbe806d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:26.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.595+0000 7fbc0a453700 1 --2- 172.21.15.16:0/696383542 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fbbe806b170 0x7fbbe806d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:26.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.595+0000 7fbbfaffd700 1 -- 172.21.15.16:0/696383542 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(245..245 src has 1..245) v4 ==== 6157+0+0 (secure 0 0 0) 0x7fbbfc08a0e0 con 0x7fbc040c6f50 2023-10-16T13:47:26.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.595+0000 7fbc0a453700 1 --2- 172.21.15.16:0/696383542 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fbbe806b170 0x7fbbe806d630 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fbc00009c60 tx=0x7fbc00009210 comp rx=0 tx=0).ready entity=mgr.11886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:26.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.600+0000 7fbbfaffd700 1 -- 172.21.15.16:0/696383542 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbbfc054ad0 con 0x7fbc040c6f50 2023-10-16T13:47:26.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:26.850+0000 7fbc0c6b7700 1 -- 172.21.15.16:0/696383542 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fbc0404ebb0 con 0x7fbc040c6f50 2023-10-16T13:47:27.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.434+0000 7fbbfaffd700 1 -- 172.21.15.16:0/696383542 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v130) v1 ==== 122+0+0 (secure 0 0 0) 0x7fbbfc0580f0 con 0x7fbc040c6f50 2023-10-16T13:47:27.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.437+0000 7fbc0c6b7700 1 -- 172.21.15.16:0/696383542 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fbbe806b170 msgr2=0x7fbbe806d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:27.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.437+0000 7fbc0c6b7700 1 --2- 172.21.15.16:0/696383542 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fbbe806b170 0x7fbbe806d630 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fbc00009c60 tx=0x7fbc00009210 comp rx=0 tx=0).stop 2023-10-16T13:47:27.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.437+0000 7fbc0c6b7700 1 -- 172.21.15.16:0/696383542 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc040c6f50 msgr2=0x7fbc040c73d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:27.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.437+0000 7fbc0c6b7700 1 --2- 172.21.15.16:0/696383542 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc040c6f50 0x7fbc040c73d0 secure :-1 s=READY pgs=2744 cs=0 l=1 rev1=1 crypto rx=0x7fbc0406ea20 tx=0x7fbbfc011ed0 comp rx=0 tx=0).stop 2023-10-16T13:47:27.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.438+0000 7fbc0c6b7700 1 -- 172.21.15.16:0/696383542 shutdown_connections 2023-10-16T13:47:27.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.438+0000 7fbc0c6b7700 1 --2- 172.21.15.16:0/696383542 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fbbe806b170 0x7fbbe806d630 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:27.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.438+0000 7fbc0c6b7700 1 --2- 172.21.15.16:0/696383542 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc040c6f50 0x7fbc040c73d0 unknown :-1 s=CLOSED pgs=2744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:27.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.438+0000 7fbc0c6b7700 1 --2- 172.21.15.16:0/696383542 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbbf0004850 0x7fbc040c90d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:27.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.438+0000 7fbc0c6b7700 1 --2- 172.21.15.16:0/696383542 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbc040c6250 0x7fbc0414d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:27.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.438+0000 7fbc0c6b7700 1 -- 172.21.15.16:0/696383542 >> 172.21.15.16:0/696383542 conn(0x7fbc040d15f0 msgr2=0x7fbc040afcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:27.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.438+0000 7fbc0c6b7700 1 -- 172.21.15.16:0/696383542 shutdown_connections 2023-10-16T13:47:27.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.438+0000 7fbc0c6b7700 1 -- 172.21.15.16:0/696383542 wait complete. 2023-10-16T13:47:27.455 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-16T13:47:27.456 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:47:27.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.585+0000 7ff27518d700 1 Processor -- start 2023-10-16T13:47:27.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.585+0000 7ff27518d700 1 -- start start 2023-10-16T13:47:27.589 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.585+0000 7ff27518d700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7ff27015b280 0x7ff2701613e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:27.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.585+0000 7ff27518d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff27006a7e0 0x7ff27006ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:27.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.585+0000 7ff27518d700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff27006b150 con 0x7ff2700cc8a0 2023-10-16T13:47:27.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.585+0000 7ff27518d700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff270161c30 con 0x7ff27006a7e0 2023-10-16T13:47:27.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.585+0000 7ff27518d700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff270161f10 con 0x7ff27015b280 2023-10-16T13:47:27.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.586+0000 7ff26f59e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff27006a7e0 0x7ff27006ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:27.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.586+0000 7ff26f59e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff27006a7e0 0x7ff27006ac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55446/0 (socket says 172.21.15.16:55446) 2023-10-16T13:47:27.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.586+0000 7ff26f59e700 1 -- 172.21.15.16:0/3467812907 learned_addr learned my addr 172.21.15.16:0/3467812907 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:27.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.586+0000 7ff26ed9d700 1 --2- 172.21.15.16:0/3467812907 >> v2:172.21.15.49:3300/0 conn(0x7ff27015b280 0x7ff2701613e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:27.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.586+0000 7ff26f59e700 1 -- 172.21.15.16:0/3467812907 >> v2:172.21.15.49:3300/0 conn(0x7ff27015b280 msgr2=0x7ff2701613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:27.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.586+0000 7ff26f59e700 1 --2- 172.21.15.16:0/3467812907 >> v2:172.21.15.49:3300/0 conn(0x7ff27015b280 0x7ff2701613e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:27.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.586+0000 7ff26f59e700 1 -- 172.21.15.16:0/3467812907 >> v1:172.21.15.49:6789/0 conn(0x7ff2700cc8a0 legacy=0x7ff27015e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:27.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.586+0000 7ff26f59e700 1 -- 172.21.15.16:0/3467812907 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2701621f0 con 0x7ff27006a7e0 2023-10-16T13:47:27.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.586+0000 7ff26f59e700 1 --2- 172.21.15.16:0/3467812907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff27006a7e0 0x7ff27006ac10 secure :-1 s=READY pgs=2745 cs=0 l=1 rev1=1 crypto rx=0x7ff27006b8a0 tx=0x7ff26800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25a31242537bb6fb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:27.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.586+0000 7ff26dd9b700 1 -- 172.21.15.16:0/3467812907 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff26800e070 con 0x7ff27006a7e0 2023-10-16T13:47:27.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.586+0000 7ff26dd9b700 1 -- 172.21.15.16:0/3467812907 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff26800b9c0 con 0x7ff27006a7e0 2023-10-16T13:47:27.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.586+0000 7ff26dd9b700 1 -- 172.21.15.16:0/3467812907 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff26800bd70 con 0x7ff27006a7e0 2023-10-16T13:47:27.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.587+0000 7ff27518d700 1 -- 172.21.15.16:0/3467812907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff27006a7e0 msgr2=0x7ff27006ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:27.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.587+0000 7ff27518d700 1 --2- 172.21.15.16:0/3467812907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff27006a7e0 0x7ff27006ac10 secure :-1 s=READY pgs=2745 cs=0 l=1 rev1=1 crypto rx=0x7ff27006b8a0 tx=0x7ff26800b330 comp rx=0 tx=0).stop 2023-10-16T13:47:27.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.587+0000 7ff27518d700 1 -- 172.21.15.16:0/3467812907 shutdown_connections 2023-10-16T13:47:27.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.587+0000 7ff27518d700 1 --2- 172.21.15.16:0/3467812907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff27006a7e0 0x7ff27006ac10 unknown :-1 s=CLOSED pgs=2745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:27.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.587+0000 7ff27518d700 1 --2- 172.21.15.16:0/3467812907 >> v2:172.21.15.49:3300/0 conn(0x7ff27015b280 0x7ff2701613e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:27.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.587+0000 7ff27518d700 1 -- 172.21.15.16:0/3467812907 >> 172.21.15.16:0/3467812907 conn(0x7ff2700b1fd0 msgr2=0x7ff2700b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:27.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.587+0000 7ff27518d700 1 -- 172.21.15.16:0/3467812907 shutdown_connections 2023-10-16T13:47:27.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.587+0000 7ff27518d700 1 -- 172.21.15.16:0/3467812907 wait complete. 2023-10-16T13:47:27.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.588+0000 7ff27518d700 1 Processor -- start 2023-10-16T13:47:27.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.588+0000 7ff27518d700 1 -- start start 2023-10-16T13:47:27.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.589+0000 7ff27518d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff27006a7e0 0x7ff2700ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:27.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.589+0000 7ff27518d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff2700cc8a0 0x7ff270158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:27.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.589+0000 7ff27518d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff27015b280 0x7ff270159e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:27.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.589+0000 7ff27518d700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff270159460 con 0x7ff27006a7e0 2023-10-16T13:47:27.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.589+0000 7ff27518d700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff27015a390 con 0x7ff27015b280 2023-10-16T13:47:27.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.589+0000 7ff27518d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff270159180 con 0x7ff2700cc8a0 2023-10-16T13:47:27.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.589+0000 7ff26ed9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff27006a7e0 0x7ff2700ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:27.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.589+0000 7ff26ed9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff27006a7e0 0x7ff2700ac8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55448/0 (socket says 172.21.15.16:55448) 2023-10-16T13:47:27.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.589+0000 7ff26ed9d700 1 -- 172.21.15.16:0/274985882 learned_addr learned my addr 172.21.15.16:0/274985882 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:27.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.589+0000 7ff26f59e700 1 --2- 172.21.15.16:0/274985882 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff27015b280 0x7ff270159e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:27.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.589+0000 7ff26e59c700 1 --2- 172.21.15.16:0/274985882 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff2700cc8a0 0x7ff270158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:27.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.589+0000 7ff26ed9d700 1 -- 172.21.15.16:0/274985882 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff2700cc8a0 msgr2=0x7ff270158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:27.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.589+0000 7ff26ed9d700 1 --2- 172.21.15.16:0/274985882 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff2700cc8a0 0x7ff270158ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:27.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.589+0000 7ff26ed9d700 1 -- 172.21.15.16:0/274985882 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff27015b280 msgr2=0x7ff270159e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:27.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.589+0000 7ff26ed9d700 1 --2- 172.21.15.16:0/274985882 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff27015b280 0x7ff270159e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:27.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.589+0000 7ff26ed9d700 1 -- 172.21.15.16:0/274985882 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff268009b30 con 0x7ff27006a7e0 2023-10-16T13:47:27.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.590+0000 7ff26ed9d700 1 --2- 172.21.15.16:0/274985882 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff27006a7e0 0x7ff2700ac8d0 secure :-1 s=READY pgs=2746 cs=0 l=1 rev1=1 crypto rx=0x7ff26000e810 tx=0x7ff26000ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:27.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.590+0000 7ff2577fe700 1 -- 172.21.15.16:0/274985882 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff260013610 con 0x7ff27006a7e0 2023-10-16T13:47:27.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.590+0000 7ff2577fe700 1 -- 172.21.15.16:0/274985882 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff260013770 con 0x7ff27006a7e0 2023-10-16T13:47:27.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.590+0000 7ff27518d700 1 -- 172.21.15.16:0/274985882 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2700c40a0 con 0x7ff27006a7e0 2023-10-16T13:47:27.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.590+0000 7ff2577fe700 1 -- 172.21.15.16:0/274985882 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff260013b20 con 0x7ff27006a7e0 2023-10-16T13:47:27.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.590+0000 7ff27518d700 1 -- 172.21.15.16:0/274985882 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2700c4660 con 0x7ff27006a7e0 2023-10-16T13:47:27.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.591+0000 7ff27518d700 1 -- 172.21.15.16:0/274985882 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff27004ebb0 con 0x7ff27006a7e0 2023-10-16T13:47:27.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.591+0000 7ff2577fe700 1 -- 172.21.15.16:0/274985882 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 148) v1 ==== 89433+0+0 (secure 0 0 0) 0x7ff260013c80 con 0x7ff27006a7e0 2023-10-16T13:47:27.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.592+0000 7ff2577fe700 1 --2- 172.21.15.16:0/274985882 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7ff25c06b140 0x7ff25c06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:27.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.592+0000 7ff2577fe700 1 -- 172.21.15.16:0/274985882 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(245..245 src has 1..245) v4 ==== 6157+0+0 (secure 0 0 0) 0x7ff2600899d0 con 0x7ff27006a7e0 2023-10-16T13:47:27.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.592+0000 7ff26e59c700 1 --2- 172.21.15.16:0/274985882 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7ff25c06b140 0x7ff25c06d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:27.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.592+0000 7ff26e59c700 1 --2- 172.21.15.16:0/274985882 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7ff25c06b140 0x7ff25c06d600 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff270071da0 tx=0x7ff264009040 comp rx=0 tx=0).ready entity=mgr.11886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:27.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.596+0000 7ff2577fe700 1 -- 172.21.15.16:0/274985882 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff260054460 con 0x7ff27006a7e0 2023-10-16T13:47:27.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.844+0000 7ff27518d700 1 -- 172.21.15.16:0/274985882 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff2700cad20 con 0x7ff27006a7e0 2023-10-16T13:47:27.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.845+0000 7ff2577fe700 1 -- 172.21.15.16:0/274985882 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 130 v130) v1 ==== 95+0+4922 (secure 0 0 0) 0x7ff260057a80 con 0x7ff27006a7e0 2023-10-16T13:47:27.848 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:47:27.848 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":130,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9596,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/3939647668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":3939647668},{"type":"v1","addr":"172.21.15.49:6834","nonce":3939647668}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":125},{"gid":11185,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/2890113513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2890113513},{"type":"v1","addr":"172.21.15.16:6836","nonce":2890113513}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":125},{"gid":11197,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/680748068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":680748068},{"type":"v1","addr":"172.21.15.49:6835","nonce":680748068}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":125}],"filesystems":[{"mdsmap":{"epoch":130,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:47:23.843333+0000","modified":"2023-10-16T13:47:26.851597+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":12252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12252":{"gid":12252,"name":"a","rank":0,"incarnation":126,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/303278795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":303278795},{"type":"v1","addr":"172.21.15.16:6837","nonce":303278795}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15}]} 2023-10-16T13:47:27.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.847+0000 7ff27518d700 1 -- 172.21.15.16:0/274985882 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7ff25c06b140 msgr2=0x7ff25c06d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:27.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.847+0000 7ff27518d700 1 --2- 172.21.15.16:0/274985882 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7ff25c06b140 0x7ff25c06d600 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff270071da0 tx=0x7ff264009040 comp rx=0 tx=0).stop 2023-10-16T13:47:27.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.847+0000 7ff27518d700 1 -- 172.21.15.16:0/274985882 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff27006a7e0 msgr2=0x7ff2700ac8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:27.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.847+0000 7ff27518d700 1 --2- 172.21.15.16:0/274985882 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff27006a7e0 0x7ff2700ac8d0 secure :-1 s=READY pgs=2746 cs=0 l=1 rev1=1 crypto rx=0x7ff26000e810 tx=0x7ff26000ebd0 comp rx=0 tx=0).stop 2023-10-16T13:47:27.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.847+0000 7ff27518d700 1 -- 172.21.15.16:0/274985882 shutdown_connections 2023-10-16T13:47:27.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.847+0000 7ff27518d700 1 --2- 172.21.15.16:0/274985882 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7ff25c06b140 0x7ff25c06d600 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:27.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.847+0000 7ff27518d700 1 --2- 172.21.15.16:0/274985882 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff27015b280 0x7ff270159e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:27.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.847+0000 7ff27518d700 1 --2- 172.21.15.16:0/274985882 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff27006a7e0 0x7ff2700ac8d0 unknown :-1 s=CLOSED pgs=2746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:27.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.847+0000 7ff27518d700 1 --2- 172.21.15.16:0/274985882 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff2700cc8a0 0x7ff270158ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:27.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.847+0000 7ff27518d700 1 -- 172.21.15.16:0/274985882 >> 172.21.15.16:0/274985882 conn(0x7ff2700b1fd0 msgr2=0x7ff2700d0270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:27.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.847+0000 7ff27518d700 1 -- 172.21.15.16:0/274985882 shutdown_connections 2023-10-16T13:47:27.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.847+0000 7ff27518d700 1 -- 172.21.15.16:0/274985882 wait complete. 2023-10-16T13:47:27.853 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 130 2023-10-16T13:47:27.862 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:47:27.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.985+0000 7ff17652e700 1 Processor -- start 2023-10-16T13:47:27.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.985+0000 7ff17652e700 1 -- start start 2023-10-16T13:47:27.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.987+0000 7ff17652e700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7ff1700cf4a0 0x7ff1700726e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:27.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.987+0000 7ff17652e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff1700cf880 0x7ff1700b52f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:27.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.987+0000 7ff17652e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff1700b5830 0x7ff17015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:27.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.987+0000 7ff17652e700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff17015bc80 con 0x7ff1700b5830 2023-10-16T13:47:27.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.987+0000 7ff17652e700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff170072c20 con 0x7ff1700cf4a0 2023-10-16T13:47:27.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.987+0000 7ff17652e700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff17015b070 con 0x7ff1700cf880 2023-10-16T13:47:27.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.987+0000 7ff174acb700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff1700b5830 0x7ff17015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:27.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.987+0000 7ff174acb700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff1700b5830 0x7ff17015a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55458/0 (socket says 172.21.15.16:55458) 2023-10-16T13:47:27.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.987+0000 7ff174acb700 1 -- 172.21.15.16:0/2369665205 learned_addr learned my addr 172.21.15.16:0/2369665205 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:27.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.987+0000 7ff16ffff700 1 --2- 172.21.15.16:0/2369665205 >> v2:172.21.15.49:3300/0 conn(0x7ff1700cf4a0 0x7ff1700726e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:27.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.987+0000 7ff16f7fe700 1 --2- 172.21.15.16:0/2369665205 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff1700cf880 0x7ff1700b52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:27.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.987+0000 7ff174acb700 1 -- 172.21.15.16:0/2369665205 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff1700cf880 msgr2=0x7ff1700b52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:27.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.987+0000 7ff174acb700 1 --2- 172.21.15.16:0/2369665205 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff1700cf880 0x7ff1700b52f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:27.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.987+0000 7ff174acb700 1 -- 172.21.15.16:0/2369665205 >> v2:172.21.15.49:3300/0 conn(0x7ff1700cf4a0 msgr2=0x7ff1700726e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:27.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.987+0000 7ff174acb700 1 --2- 172.21.15.16:0/2369665205 >> v2:172.21.15.49:3300/0 conn(0x7ff1700cf4a0 0x7ff1700726e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:27.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.987+0000 7ff174acb700 1 -- 172.21.15.16:0/2369665205 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff17015b350 con 0x7ff1700b5830 2023-10-16T13:47:27.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.987+0000 7ff174acb700 1 --2- 172.21.15.16:0/2369665205 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1700b5830 0x7ff17015a090 secure :-1 s=READY pgs=2747 cs=0 l=1 rev1=1 crypto rx=0x7ff1700b2d20 tx=0x7ff168009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=972116a71e2a67d6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:27.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.987+0000 7ff16effd700 1 -- 172.21.15.16:0/2369665205 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff168011830 con 0x7ff1700b5830 2023-10-16T13:47:27.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.987+0000 7ff16effd700 1 -- 172.21.15.16:0/2369665205 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff168011990 con 0x7ff1700b5830 2023-10-16T13:47:27.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.987+0000 7ff16effd700 1 -- 172.21.15.16:0/2369665205 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff168011d20 con 0x7ff1700b5830 2023-10-16T13:47:27.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.988+0000 7ff17652e700 1 -- 172.21.15.16:0/2369665205 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1700b5830 msgr2=0x7ff17015a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:27.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.988+0000 7ff17652e700 1 --2- 172.21.15.16:0/2369665205 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1700b5830 0x7ff17015a090 secure :-1 s=READY pgs=2747 cs=0 l=1 rev1=1 crypto rx=0x7ff1700b2d20 tx=0x7ff168009bd0 comp rx=0 tx=0).stop 2023-10-16T13:47:27.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.988+0000 7ff17652e700 1 -- 172.21.15.16:0/2369665205 shutdown_connections 2023-10-16T13:47:27.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.988+0000 7ff17652e700 1 --2- 172.21.15.16:0/2369665205 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1700b5830 0x7ff17015a090 unknown :-1 s=CLOSED pgs=2747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:27.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.988+0000 7ff17652e700 1 --2- 172.21.15.16:0/2369665205 >> v2:172.21.15.49:3300/0 conn(0x7ff1700cf4a0 0x7ff1700726e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:27.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.988+0000 7ff17652e700 1 --2- 172.21.15.16:0/2369665205 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff1700cf880 0x7ff1700b52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:27.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.988+0000 7ff17652e700 1 -- 172.21.15.16:0/2369665205 >> 172.21.15.16:0/2369665205 conn(0x7ff1700b1fd0 msgr2=0x7ff1700b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:27.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.988+0000 7ff17652e700 1 -- 172.21.15.16:0/2369665205 shutdown_connections 2023-10-16T13:47:27.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.988+0000 7ff17652e700 1 -- 172.21.15.16:0/2369665205 wait complete. 2023-10-16T13:47:27.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.990+0000 7ff17652e700 1 Processor -- start 2023-10-16T13:47:28.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.990+0000 7ff17652e700 1 -- start start 2023-10-16T13:47:28.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.991+0000 7ff17652e700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff1700b5830 0x7ff17014e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:28.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.991+0000 7ff17652e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff1700cf4a0 0x7ff1700b0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:28.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.991+0000 7ff17652e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1700cf880 0x7ff1700b1000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:28.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.991+0000 7ff17652e700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1700ca4e0 con 0x7ff1700cf880 2023-10-16T13:47:28.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.991+0000 7ff17652e700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff17014e9b0 con 0x7ff1700b5830 2023-10-16T13:47:28.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.991+0000 7ff17652e700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff17015a150 con 0x7ff1700cf4a0 2023-10-16T13:47:28.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.991+0000 7ff174acb700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1700cf880 0x7ff1700b1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:28.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.991+0000 7ff174acb700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1700cf880 0x7ff1700b1000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55464/0 (socket says 172.21.15.16:55464) 2023-10-16T13:47:28.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.991+0000 7ff174acb700 1 -- 172.21.15.16:0/969735967 learned_addr learned my addr 172.21.15.16:0/969735967 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:28.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.991+0000 7ff16ffff700 1 --2- 172.21.15.16:0/969735967 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff1700b5830 0x7ff17014e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:28.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.991+0000 7ff16f7fe700 1 --2- 172.21.15.16:0/969735967 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff1700cf4a0 0x7ff1700b0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:28.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.991+0000 7ff174acb700 1 -- 172.21.15.16:0/969735967 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff1700cf4a0 msgr2=0x7ff1700b0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:28.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.991+0000 7ff174acb700 1 --2- 172.21.15.16:0/969735967 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff1700cf4a0 0x7ff1700b0ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:28.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.991+0000 7ff174acb700 1 -- 172.21.15.16:0/969735967 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff1700b5830 msgr2=0x7ff17014e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:28.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.991+0000 7ff174acb700 1 --2- 172.21.15.16:0/969735967 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff1700b5830 0x7ff17014e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:28.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.991+0000 7ff174acb700 1 -- 172.21.15.16:0/969735967 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff168009550 con 0x7ff1700cf880 2023-10-16T13:47:28.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.991+0000 7ff16ffff700 1 --2- 172.21.15.16:0/969735967 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff1700b5830 0x7ff17014e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:47:28.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.991+0000 7ff174acb700 1 --2- 172.21.15.16:0/969735967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1700cf880 0x7ff1700b1000 secure :-1 s=READY pgs=2748 cs=0 l=1 rev1=1 crypto rx=0x7ff168009c10 tx=0x7ff168004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:28.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.992+0000 7ff16cff9700 1 -- 172.21.15.16:0/969735967 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff168004300 con 0x7ff1700cf880 2023-10-16T13:47:28.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.992+0000 7ff16cff9700 1 -- 172.21.15.16:0/969735967 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff16801f070 con 0x7ff1700cf880 2023-10-16T13:47:28.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.992+0000 7ff17652e700 1 -- 172.21.15.16:0/969735967 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff17015a430 con 0x7ff1700cf880 2023-10-16T13:47:28.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.992+0000 7ff16cff9700 1 -- 172.21.15.16:0/969735967 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1680036a0 con 0x7ff1700cf880 2023-10-16T13:47:28.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.992+0000 7ff17652e700 1 -- 172.21.15.16:0/969735967 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1700c6880 con 0x7ff1700cf880 2023-10-16T13:47:28.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.992+0000 7ff17652e700 1 -- 172.21.15.16:0/969735967 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1700c6400 con 0x7ff1700cf880 2023-10-16T13:47:28.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.993+0000 7ff16cff9700 1 -- 172.21.15.16:0/969735967 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 148) v1 ==== 89433+0+0 (secure 0 0 0) 0x7ff168021030 con 0x7ff1700cf880 2023-10-16T13:47:28.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.994+0000 7ff16cff9700 1 --2- 172.21.15.16:0/969735967 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7ff15006b140 0x7ff15006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:28.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.994+0000 7ff16cff9700 1 -- 172.21.15.16:0/969735967 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(245..245 src has 1..245) v4 ==== 6157+0+0 (secure 0 0 0) 0x7ff16808ad80 con 0x7ff1700cf880 2023-10-16T13:47:28.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.994+0000 7ff16ffff700 1 --2- 172.21.15.16:0/969735967 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7ff15006b140 0x7ff15006d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:28.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.994+0000 7ff16ffff700 1 --2- 172.21.15.16:0/969735967 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7ff15006b140 0x7ff15006d600 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff1700afa80 tx=0x7ff16000b040 comp rx=0 tx=0).ready entity=mgr.11886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:28.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:27.998+0000 7ff16cff9700 1 -- 172.21.15.16:0/969735967 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff1700c6400 con 0x7ff1700cf880 2023-10-16T13:47:28.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.203+0000 7ff17652e700 1 -- 172.21.15.16:0/969735967 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff1700c6400 con 0x7ff1700cf880 2023-10-16T13:47:28.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.204+0000 7ff16cff9700 1 -- 172.21.15.16:0/969735967 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v245) v1 ==== 74+0+16881 (secure 0 0 0) 0x7ff1700c6400 con 0x7ff1700cf880 2023-10-16T13:47:28.207 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:47:28.207 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":245,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:47:25.413253+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2023-10-16T13:47:22.338349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"244","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2023-10-16T13:47:22.806327+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"245","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":243,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":243,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":243,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":243,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":243,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":243,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":243,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":243,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:47:28.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.207+0000 7ff17652e700 1 -- 172.21.15.16:0/969735967 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7ff15006b140 msgr2=0x7ff15006d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:28.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.207+0000 7ff17652e700 1 --2- 172.21.15.16:0/969735967 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7ff15006b140 0x7ff15006d600 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff1700afa80 tx=0x7ff16000b040 comp rx=0 tx=0).stop 2023-10-16T13:47:28.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.207+0000 7ff17652e700 1 -- 172.21.15.16:0/969735967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1700cf880 msgr2=0x7ff1700b1000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:28.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.207+0000 7ff17652e700 1 --2- 172.21.15.16:0/969735967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1700cf880 0x7ff1700b1000 secure :-1 s=READY pgs=2748 cs=0 l=1 rev1=1 crypto rx=0x7ff168009c10 tx=0x7ff168004750 comp rx=0 tx=0).stop 2023-10-16T13:47:28.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.207+0000 7ff17652e700 1 -- 172.21.15.16:0/969735967 shutdown_connections 2023-10-16T13:47:28.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.207+0000 7ff17652e700 1 --2- 172.21.15.16:0/969735967 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7ff15006b140 0x7ff15006d600 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:28.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.207+0000 7ff17652e700 1 --2- 172.21.15.16:0/969735967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1700cf880 0x7ff1700b1000 unknown :-1 s=CLOSED pgs=2748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:28.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.207+0000 7ff17652e700 1 --2- 172.21.15.16:0/969735967 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff1700b5830 0x7ff17014e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:28.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.207+0000 7ff17652e700 1 --2- 172.21.15.16:0/969735967 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff1700cf4a0 0x7ff1700b0ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:28.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.207+0000 7ff17652e700 1 -- 172.21.15.16:0/969735967 >> 172.21.15.16:0/969735967 conn(0x7ff1700b1fd0 msgr2=0x7ff1700b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:28.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.207+0000 7ff17652e700 1 -- 172.21.15.16:0/969735967 shutdown_connections 2023-10-16T13:47:28.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.207+0000 7ff17652e700 1 -- 172.21.15.16:0/969735967 wait complete. 2023-10-16T13:47:28.222 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-16T13:47:28.222 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T13:47:28.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.345+0000 7fb8eab15700 1 Processor -- start 2023-10-16T13:47:28.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.345+0000 7fb8eab15700 1 -- start start 2023-10-16T13:47:28.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.345+0000 7fb8eab15700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb8e406b7f0 0x7fb8e406bc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:28.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.345+0000 7fb8eab15700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fb8e406d940 0x7fb8e406ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:28.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.345+0000 7fb8eab15700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8e40c3490 con 0x7fb8e415b9f0 2023-10-16T13:47:28.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.345+0000 7fb8eab15700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8e406c1b0 con 0x7fb8e406d940 2023-10-16T13:47:28.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.345+0000 7fb8eab15700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e406e300 con 0x7fb8e406b7f0 2023-10-16T13:47:28.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.346+0000 7fb8e88b1700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb8e415b9f0 0x7fb8e40b29a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44742/0 (socket says 172.21.15.16:44742) 2023-10-16T13:47:28.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.346+0000 7fb8e88b1700 1 -- 172.21.15.16:0/1123266736 learned_addr learned my addr 172.21.15.16:0/1123266736 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:47:28.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.346+0000 7fb8e3fff700 1 --2- 172.21.15.16:0/1123266736 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb8e406b7f0 0x7fb8e406bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:28.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.346+0000 7fb8e90b2700 1 --2- 172.21.15.16:0/1123266736 >> v2:172.21.15.49:3300/0 conn(0x7fb8e406d940 0x7fb8e406ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:28.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.346+0000 7fb8e37fe700 1 -- 172.21.15.16:0/1123266736 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3755912877 0 0) 0x7fb8e40c3490 con 0x7fb8e415b9f0 2023-10-16T13:47:28.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.346+0000 7fb8e37fe700 1 -- 172.21.15.16:0/1123266736 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8d40035b0 con 0x7fb8e415b9f0 2023-10-16T13:47:28.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.346+0000 7fb8e37fe700 1 -- 172.21.15.16:0/1123266736 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3130430818 0 0) 0x7fb8d40035b0 con 0x7fb8e415b9f0 2023-10-16T13:47:28.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.346+0000 7fb8e37fe700 1 -- 172.21.15.16:0/1123266736 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb8e406b7f0 msgr2=0x7fb8e406bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:28.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.346+0000 7fb8e37fe700 1 --2- 172.21.15.16:0/1123266736 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb8e406b7f0 0x7fb8e406bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:28.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.346+0000 7fb8e37fe700 1 -- 172.21.15.16:0/1123266736 >> v2:172.21.15.49:3300/0 conn(0x7fb8e406d940 msgr2=0x7fb8e406ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:28.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.346+0000 7fb8e37fe700 1 --2- 172.21.15.16:0/1123266736 >> v2:172.21.15.49:3300/0 conn(0x7fb8e406d940 0x7fb8e406ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:28.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.346+0000 7fb8e37fe700 1 -- 172.21.15.16:0/1123266736 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8e40cdad0 con 0x7fb8e415b9f0 2023-10-16T13:47:28.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.346+0000 7fb8e37fe700 1 -- 172.21.15.16:0/1123266736 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7fb8d8001670 con 0x7fb8e415b9f0 2023-10-16T13:47:28.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.346+0000 7fb8e37fe700 1 -- 172.21.15.16:0/1123266736 >> v1:172.21.15.16:6789/0 conn(0x7fb8e415b9f0 legacy=0x7fb8e40b29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:28.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.346+0000 7fb8e37fe700 1 --2- 172.21.15.16:0/1123266736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8d4003a00 0x7fb8d4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:28.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.346+0000 7fb8e37fe700 1 --2- 172.21.15.16:0/1123266736 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb8d4004850 0x7fb8d4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:28.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.346+0000 7fb8e37fe700 1 --2- 172.21.15.16:0/1123266736 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb8e406b7f0 0x7fb8d4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:28.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.346+0000 7fb8e37fe700 1 -- 172.21.15.16:0/1123266736 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e406c1b0 con 0x7fb8d4003a00 2023-10-16T13:47:28.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.346+0000 7fb8e37fe700 1 -- 172.21.15.16:0/1123266736 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e406e300 con 0x7fb8d4004850 2023-10-16T13:47:28.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.346+0000 7fb8e37fe700 1 -- 172.21.15.16:0/1123266736 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8d400ba10 con 0x7fb8e406b7f0 2023-10-16T13:47:28.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.346+0000 7fb8e88b1700 1 --2- 172.21.15.16:0/1123266736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8d4003a00 0x7fb8d4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:28.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.346+0000 7fb8e3fff700 1 --2- 172.21.15.16:0/1123266736 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb8d4004850 0x7fb8d4004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:28.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.346+0000 7fb8e90b2700 1 --2- 172.21.15.16:0/1123266736 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb8e406b7f0 0x7fb8d4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:28.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.347+0000 7fb8e88b1700 1 -- 172.21.15.16:0/1123266736 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb8e406b7f0 msgr2=0x7fb8d4005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:28.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.347+0000 7fb8e88b1700 1 --2- 172.21.15.16:0/1123266736 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb8e406b7f0 0x7fb8d4005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:28.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.347+0000 7fb8e88b1700 1 -- 172.21.15.16:0/1123266736 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb8d4004850 msgr2=0x7fb8d4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:28.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.347+0000 7fb8e88b1700 1 --2- 172.21.15.16:0/1123266736 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb8d4004850 0x7fb8d4004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:28.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.347+0000 7fb8e88b1700 1 -- 172.21.15.16:0/1123266736 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8d40035b0 con 0x7fb8d4003a00 2023-10-16T13:47:28.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.347+0000 7fb8e90b2700 1 --2- 172.21.15.16:0/1123266736 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb8e406b7f0 0x7fb8d4005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:47:28.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.347+0000 7fb8e88b1700 1 --2- 172.21.15.16:0/1123266736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8d4003a00 0x7fb8d4003e20 secure :-1 s=READY pgs=2750 cs=0 l=1 rev1=1 crypto rx=0x7fb8d800c320 tx=0x7fb8d4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b77d82904433dbe server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:28.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.347+0000 7fb8e37fe700 1 -- 172.21.15.16:0/1123266736 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8d800ce60 con 0x7fb8d4003a00 2023-10-16T13:47:28.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.347+0000 7fb8e37fe700 1 -- 172.21.15.16:0/1123266736 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8d8005960 con 0x7fb8d4003a00 2023-10-16T13:47:28.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.347+0000 7fb8e37fe700 1 -- 172.21.15.16:0/1123266736 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8d8005cf0 con 0x7fb8d4003a00 2023-10-16T13:47:28.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.347+0000 7fb8eab15700 1 -- 172.21.15.16:0/1123266736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8d4003a00 msgr2=0x7fb8d4003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:28.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.347+0000 7fb8eab15700 1 --2- 172.21.15.16:0/1123266736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8d4003a00 0x7fb8d4003e20 secure :-1 s=READY pgs=2750 cs=0 l=1 rev1=1 crypto rx=0x7fb8d800c320 tx=0x7fb8d4005880 comp rx=0 tx=0).stop 2023-10-16T13:47:28.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.347+0000 7fb8e90b2700 1 -- 172.21.15.16:0/1123266736 reap_dead start 2023-10-16T13:47:28.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.348+0000 7fb8eab15700 1 -- 172.21.15.16:0/1123266736 shutdown_connections 2023-10-16T13:47:28.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.348+0000 7fb8eab15700 1 -- 172.21.15.16:0/1123266736 >> 172.21.15.16:0/1123266736 conn(0x7fb8e40cc5b0 msgr2=0x7fb8e40cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:28.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.348+0000 7fb8eab15700 1 -- 172.21.15.16:0/1123266736 shutdown_connections 2023-10-16T13:47:28.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.348+0000 7fb8eab15700 1 -- 172.21.15.16:0/1123266736 wait complete. 2023-10-16T13:47:28.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.349+0000 7fb8eab15700 1 Processor -- start 2023-10-16T13:47:28.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.349+0000 7fb8eab15700 1 -- start start 2023-10-16T13:47:28.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.350+0000 7fb8eab15700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb8d4003a00 0x7fb8e40b3a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:28.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.350+0000 7fb8eab15700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb8e40c7270 0x7fb8e40b3f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:28.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.350+0000 7fb8eab15700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8e40cb0d0 0x7fb8e40cb560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:28.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.350+0000 7fb8eab15700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e40c3620 con 0x7fb8e40cb0d0 2023-10-16T13:47:28.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.350+0000 7fb8eab15700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e40c4480 con 0x7fb8e40c7270 2023-10-16T13:47:28.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.350+0000 7fb8eab15700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e414f950 con 0x7fb8d4003a00 2023-10-16T13:47:28.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.350+0000 7fb8e90b2700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8e40cb0d0 0x7fb8e40cb560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:28.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.350+0000 7fb8e90b2700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8e40cb0d0 0x7fb8e40cb560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55484/0 (socket says 172.21.15.16:55484) 2023-10-16T13:47:28.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.350+0000 7fb8e90b2700 1 -- 172.21.15.16:0/3076509554 learned_addr learned my addr 172.21.15.16:0/3076509554 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:28.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.350+0000 7fb8e88b1700 1 --2- 172.21.15.16:0/3076509554 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb8d4003a00 0x7fb8e40b3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:28.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.350+0000 7fb8e3fff700 1 --2- 172.21.15.16:0/3076509554 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb8e40c7270 0x7fb8e40b3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:28.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.350+0000 7fb8e90b2700 1 -- 172.21.15.16:0/3076509554 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb8d4003a00 msgr2=0x7fb8e40b3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:28.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.350+0000 7fb8e90b2700 1 --2- 172.21.15.16:0/3076509554 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb8d4003a00 0x7fb8e40b3a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:28.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.350+0000 7fb8e90b2700 1 -- 172.21.15.16:0/3076509554 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb8e40c7270 msgr2=0x7fb8e40b3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:28.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.350+0000 7fb8e90b2700 1 --2- 172.21.15.16:0/3076509554 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb8e40c7270 0x7fb8e40b3f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:28.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.350+0000 7fb8e90b2700 1 -- 172.21.15.16:0/3076509554 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8e40c3870 con 0x7fb8e40cb0d0 2023-10-16T13:47:28.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.350+0000 7fb8e90b2700 1 --2- 172.21.15.16:0/3076509554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8e40cb0d0 0x7fb8e40cb560 secure :-1 s=READY pgs=2751 cs=0 l=1 rev1=1 crypto rx=0x7fb8e40c9d70 tx=0x7fb8dc0104a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:28.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.350+0000 7fb8e17fa700 1 -- 172.21.15.16:0/3076509554 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8dc01b070 con 0x7fb8e40cb0d0 2023-10-16T13:47:28.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.351+0000 7fb8e17fa700 1 -- 172.21.15.16:0/3076509554 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8dc00f040 con 0x7fb8e40cb0d0 2023-10-16T13:47:28.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.351+0000 7fb8eab15700 1 -- 172.21.15.16:0/3076509554 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8e40c3b50 con 0x7fb8e40cb0d0 2023-10-16T13:47:28.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.351+0000 7fb8e17fa700 1 -- 172.21.15.16:0/3076509554 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8dc00e050 con 0x7fb8e40cb0d0 2023-10-16T13:47:28.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.351+0000 7fb8eab15700 1 -- 172.21.15.16:0/3076509554 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8e40c4110 con 0x7fb8e40cb0d0 2023-10-16T13:47:28.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.351+0000 7fb8eab15700 1 -- 172.21.15.16:0/3076509554 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8e40c42d0 con 0x7fb8e40cb0d0 2023-10-16T13:47:28.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.352+0000 7fb8e17fa700 1 -- 172.21.15.16:0/3076509554 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 148) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fb8dc015440 con 0x7fb8e40cb0d0 2023-10-16T13:47:28.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.352+0000 7fb8e17fa700 1 --2- 172.21.15.16:0/3076509554 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fb8c806b140 0x7fb8c806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:28.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.353+0000 7fb8e88b1700 1 --2- 172.21.15.16:0/3076509554 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fb8c806b140 0x7fb8c806d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:28.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.353+0000 7fb8e17fa700 1 -- 172.21.15.16:0/3076509554 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(245..245 src has 1..245) v4 ==== 6157+0+0 (secure 0 0 0) 0x7fb8dc089c30 con 0x7fb8e40cb0d0 2023-10-16T13:47:28.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.353+0000 7fb8e88b1700 1 --2- 172.21.15.16:0/3076509554 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fb8c806b140 0x7fb8c806d600 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb8d8008fd0 tx=0x7fb8d800a000 comp rx=0 tx=0).ready entity=mgr.11886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:28.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.357+0000 7fb8e17fa700 1 -- 172.21.15.16:0/3076509554 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb8dc054740 con 0x7fb8e40cb0d0 2023-10-16T13:47:28.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.559+0000 7fb8eab15700 1 -- 172.21.15.16:0/3076509554 --> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb8e40b44a0 con 0x7fb8c806b140 2023-10-16T13:47:28.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.568+0000 7fb8e17fa700 1 -- 172.21.15.16:0/3076509554 <== mgr.11886 v2:172.21.15.16:6832/1736104576 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+285751 (secure 0 0 0) 0x7fb8e40b44a0 con 0x7fb8c806b140 2023-10-16T13:47:28.574 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:47:28.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.572+0000 7fb8eab15700 1 -- 172.21.15.16:0/3076509554 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fb8c806b140 msgr2=0x7fb8c806d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:28.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.572+0000 7fb8eab15700 1 --2- 172.21.15.16:0/3076509554 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fb8c806b140 0x7fb8c806d600 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb8d8008fd0 tx=0x7fb8d800a000 comp rx=0 tx=0).stop 2023-10-16T13:47:28.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.572+0000 7fb8eab15700 1 -- 172.21.15.16:0/3076509554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8e40cb0d0 msgr2=0x7fb8e40cb560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:28.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.572+0000 7fb8eab15700 1 --2- 172.21.15.16:0/3076509554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8e40cb0d0 0x7fb8e40cb560 secure :-1 s=READY pgs=2751 cs=0 l=1 rev1=1 crypto rx=0x7fb8e40c9d70 tx=0x7fb8dc0104a0 comp rx=0 tx=0).stop 2023-10-16T13:47:28.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.572+0000 7fb8eab15700 1 -- 172.21.15.16:0/3076509554 shutdown_connections 2023-10-16T13:47:28.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.572+0000 7fb8eab15700 1 --2- 172.21.15.16:0/3076509554 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fb8c806b140 0x7fb8c806d600 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:28.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.572+0000 7fb8eab15700 1 --2- 172.21.15.16:0/3076509554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8e40cb0d0 0x7fb8e40cb560 unknown :-1 s=CLOSED pgs=2751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:28.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.572+0000 7fb8eab15700 1 --2- 172.21.15.16:0/3076509554 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb8d4003a00 0x7fb8e40b3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:28.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.572+0000 7fb8eab15700 1 --2- 172.21.15.16:0/3076509554 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb8e40c7270 0x7fb8e40b3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:28.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.572+0000 7fb8eab15700 1 -- 172.21.15.16:0/3076509554 >> 172.21.15.16:0/3076509554 conn(0x7fb8e40cc5b0 msgr2=0x7fb8e40711b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:28.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.572+0000 7fb8eab15700 1 -- 172.21.15.16:0/3076509554 shutdown_connections 2023-10-16T13:47:28.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.572+0000 7fb8eab15700 1 -- 172.21.15.16:0/3076509554 wait complete. 2023-10-16T13:47:28.578 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-10-16T13:47:28.587 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":30,"stamp":"2023-10-16T13:47:27.821538+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":656,"num_write_kb":2422,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":401,"ondisk_log_size":401,"up":150,"acting":150,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":105,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":431520,"kb_used_data":25208,"kb_used_omap":12,"kb_used_meta":406195,"kb_avail":754543200,"statfs":{"total":773094113280,"available":772652236800,"internally_reserved":0,"allocated":25812992,"data_stored":13882230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12740,"internal_metadata":415944252},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":37,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.416850"},"pg_stats":[{"pgid":"32.1f","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.418005+0000","last_change":"2023-10-16T13:47:24.429332+0000","last_active":"2023-10-16T13:47:25.418005+0000","last_peered":"2023-10-16T13:47:25.418005+0000","last_clean":"2023-10-16T13:47:25.418005+0000","last_became_active":"2023-10-16T13:47:24.428151+0000","last_became_peered":"2023-10-16T13:47:24.428151+0000","last_unstale":"2023-10-16T13:47:25.418005+0000","last_undegraded":"2023-10-16T13:47:25.418005+0000","last_fullsized":"2023-10-16T13:47:25.418005+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:51:49.567279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.1e","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.416620+0000","last_change":"2023-10-16T13:47:24.439610+0000","last_active":"2023-10-16T13:47:25.416620+0000","last_peered":"2023-10-16T13:47:25.416620+0000","last_clean":"2023-10-16T13:47:25.416620+0000","last_became_active":"2023-10-16T13:47:24.439212+0000","last_became_peered":"2023-10-16T13:47:24.439212+0000","last_unstale":"2023-10-16T13:47:25.416620+0000","last_undegraded":"2023-10-16T13:47:25.416620+0000","last_fullsized":"2023-10-16T13:47:25.416620+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:47:24.106272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1d","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100591+0000","last_change":"2023-10-16T13:47:24.432216+0000","last_active":"2023-10-16T13:47:26.100591+0000","last_peered":"2023-10-16T13:47:26.100591+0000","last_clean":"2023-10-16T13:47:26.100591+0000","last_became_active":"2023-10-16T13:47:24.431937+0000","last_became_peered":"2023-10-16T13:47:24.431937+0000","last_unstale":"2023-10-16T13:47:26.100591+0000","last_undegraded":"2023-10-16T13:47:26.100591+0000","last_fullsized":"2023-10-16T13:47:26.100591+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:23:10.580869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1c","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100430+0000","last_change":"2023-10-16T13:47:24.440252+0000","last_active":"2023-10-16T13:47:26.100430+0000","last_peered":"2023-10-16T13:47:26.100430+0000","last_clean":"2023-10-16T13:47:26.100430+0000","last_became_active":"2023-10-16T13:47:24.439982+0000","last_became_peered":"2023-10-16T13:47:24.439982+0000","last_unstale":"2023-10-16T13:47:26.100430+0000","last_undegraded":"2023-10-16T13:47:26.100430+0000","last_fullsized":"2023-10-16T13:47:26.100430+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:54:02.930019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1b","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.101595+0000","last_change":"2023-10-16T13:47:24.440049+0000","last_active":"2023-10-16T13:47:26.101595+0000","last_peered":"2023-10-16T13:47:26.101595+0000","last_clean":"2023-10-16T13:47:26.101595+0000","last_became_active":"2023-10-16T13:47:24.439750+0000","last_became_peered":"2023-10-16T13:47:24.439750+0000","last_unstale":"2023-10-16T13:47:26.101595+0000","last_undegraded":"2023-10-16T13:47:26.101595+0000","last_fullsized":"2023-10-16T13:47:26.101595+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:33:40.094175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1a","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.417711+0000","last_change":"2023-10-16T13:47:24.433661+0000","last_active":"2023-10-16T13:47:25.417711+0000","last_peered":"2023-10-16T13:47:25.417711+0000","last_clean":"2023-10-16T13:47:25.417711+0000","last_became_active":"2023-10-16T13:47:24.433417+0000","last_became_peered":"2023-10-16T13:47:24.433417+0000","last_unstale":"2023-10-16T13:47:25.417711+0000","last_undegraded":"2023-10-16T13:47:25.417711+0000","last_fullsized":"2023-10-16T13:47:25.417711+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:16:02.391378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T13:47:23.404948+0000","last_change":"2023-10-16T13:47:23.404948+0000","last_active":"2023-10-16T13:47:23.404948+0000","last_peered":"2023-10-16T13:47:23.404948+0000","last_clean":"2023-10-16T13:47:23.404948+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:23.404948+0000","last_undegraded":"2023-10-16T13:47:23.404948+0000","last_fullsized":"2023-10-16T13:47:23.404948+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.101429+0000","last_change":"2023-10-16T13:47:24.426442+0000","last_active":"2023-10-16T13:47:26.101429+0000","last_peered":"2023-10-16T13:47:26.101429+0000","last_clean":"2023-10-16T13:47:26.101429+0000","last_became_active":"2023-10-16T13:47:24.425929+0000","last_became_peered":"2023-10-16T13:47:24.425929+0000","last_unstale":"2023-10-16T13:47:26.101429+0000","last_undegraded":"2023-10-16T13:47:26.101429+0000","last_fullsized":"2023-10-16T13:47:26.101429+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:05:13.565304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.418008+0000","last_change":"2023-10-16T13:47:24.426459+0000","last_active":"2023-10-16T13:47:25.418008+0000","last_peered":"2023-10-16T13:47:25.418008+0000","last_clean":"2023-10-16T13:47:25.418008+0000","last_became_active":"2023-10-16T13:47:24.426169+0000","last_became_peered":"2023-10-16T13:47:24.426169+0000","last_unstale":"2023-10-16T13:47:25.418008+0000","last_undegraded":"2023-10-16T13:47:25.418008+0000","last_fullsized":"2023-10-16T13:47:25.418008+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:57:42.170954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100664+0000","last_change":"2023-10-16T13:47:24.427333+0000","last_active":"2023-10-16T13:47:26.100664+0000","last_peered":"2023-10-16T13:47:26.100664+0000","last_clean":"2023-10-16T13:47:26.100664+0000","last_became_active":"2023-10-16T13:47:24.427034+0000","last_became_peered":"2023-10-16T13:47:24.427034+0000","last_unstale":"2023-10-16T13:47:26.100664+0000","last_undegraded":"2023-10-16T13:47:26.100664+0000","last_fullsized":"2023-10-16T13:47:26.100664+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:27:20.181386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.15","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100502+0000","last_change":"2023-10-16T13:47:24.425903+0000","last_active":"2023-10-16T13:47:26.100502+0000","last_peered":"2023-10-16T13:47:26.100502+0000","last_clean":"2023-10-16T13:47:26.100502+0000","last_became_active":"2023-10-16T13:47:24.425593+0000","last_became_peered":"2023-10-16T13:47:24.425593+0000","last_unstale":"2023-10-16T13:47:26.100502+0000","last_undegraded":"2023-10-16T13:47:26.100502+0000","last_fullsized":"2023-10-16T13:47:26.100502+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:46:35.236231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.14","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.417564+0000","last_change":"2023-10-16T13:47:24.438927+0000","last_active":"2023-10-16T13:47:25.417564+0000","last_peered":"2023-10-16T13:47:25.417564+0000","last_clean":"2023-10-16T13:47:25.417564+0000","last_became_active":"2023-10-16T13:47:24.438389+0000","last_became_peered":"2023-10-16T13:47:24.438389+0000","last_unstale":"2023-10-16T13:47:25.417564+0000","last_undegraded":"2023-10-16T13:47:25.417564+0000","last_fullsized":"2023-10-16T13:47:25.417564+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:44:00.360485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T13:47:23.404948+0000","last_change":"2023-10-16T13:47:23.404948+0000","last_active":"2023-10-16T13:47:23.404948+0000","last_peered":"2023-10-16T13:47:23.404948+0000","last_clean":"2023-10-16T13:47:23.404948+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:23.404948+0000","last_undegraded":"2023-10-16T13:47:23.404948+0000","last_fullsized":"2023-10-16T13:47:23.404948+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.099722+0000","last_change":"2023-10-16T13:47:24.437693+0000","last_active":"2023-10-16T13:47:26.099722+0000","last_peered":"2023-10-16T13:47:26.099722+0000","last_clean":"2023-10-16T13:47:26.099722+0000","last_became_active":"2023-10-16T13:47:24.437434+0000","last_became_peered":"2023-10-16T13:47:24.437434+0000","last_unstale":"2023-10-16T13:47:26.099722+0000","last_undegraded":"2023-10-16T13:47:26.099722+0000","last_fullsized":"2023-10-16T13:47:26.099722+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:47:56.221847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T13:47:23.404948+0000","last_change":"2023-10-16T13:47:23.404948+0000","last_active":"2023-10-16T13:47:23.404948+0000","last_peered":"2023-10-16T13:47:23.404948+0000","last_clean":"2023-10-16T13:47:23.404948+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:23.404948+0000","last_undegraded":"2023-10-16T13:47:23.404948+0000","last_fullsized":"2023-10-16T13:47:23.404948+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.10","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.099717+0000","last_change":"2023-10-16T13:47:24.441060+0000","last_active":"2023-10-16T13:47:26.099717+0000","last_peered":"2023-10-16T13:47:26.099717+0000","last_clean":"2023-10-16T13:47:26.099717+0000","last_became_active":"2023-10-16T13:47:24.440745+0000","last_became_peered":"2023-10-16T13:47:24.440745+0000","last_unstale":"2023-10-16T13:47:26.099717+0000","last_undegraded":"2023-10-16T13:47:26.099717+0000","last_fullsized":"2023-10-16T13:47:26.099717+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:48:26.805075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T13:47:23.404948+0000","last_change":"2023-10-16T13:47:23.404948+0000","last_active":"2023-10-16T13:47:23.404948+0000","last_peered":"2023-10-16T13:47:23.404948+0000","last_clean":"2023-10-16T13:47:23.404948+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:23.404948+0000","last_undegraded":"2023-10-16T13:47:23.404948+0000","last_fullsized":"2023-10-16T13:47:23.404948+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.e","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100703+0000","last_change":"2023-10-16T13:47:24.433506+0000","last_active":"2023-10-16T13:47:26.100703+0000","last_peered":"2023-10-16T13:47:26.100703+0000","last_clean":"2023-10-16T13:47:26.100703+0000","last_became_active":"2023-10-16T13:47:24.433218+0000","last_became_peered":"2023-10-16T13:47:24.433218+0000","last_unstale":"2023-10-16T13:47:26.100703+0000","last_undegraded":"2023-10-16T13:47:26.100703+0000","last_fullsized":"2023-10-16T13:47:26.100703+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:04:28.651883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T13:47:23.404948+0000","last_change":"2023-10-16T13:47:23.404948+0000","last_active":"2023-10-16T13:47:23.404948+0000","last_peered":"2023-10-16T13:47:23.404948+0000","last_clean":"2023-10-16T13:47:23.404948+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:23.404948+0000","last_undegraded":"2023-10-16T13:47:23.404948+0000","last_fullsized":"2023-10-16T13:47:23.404948+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.416732+0000","last_change":"2023-10-16T13:47:24.435000+0000","last_active":"2023-10-16T13:47:25.416732+0000","last_peered":"2023-10-16T13:47:25.416732+0000","last_clean":"2023-10-16T13:47:25.416732+0000","last_became_active":"2023-10-16T13:47:24.434568+0000","last_became_peered":"2023-10-16T13:47:24.434568+0000","last_unstale":"2023-10-16T13:47:25.416732+0000","last_undegraded":"2023-10-16T13:47:25.416732+0000","last_fullsized":"2023-10-16T13:47:25.416732+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:31:46.967495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T13:47:23.404948+0000","last_change":"2023-10-16T13:47:23.404948+0000","last_active":"2023-10-16T13:47:23.404948+0000","last_peered":"2023-10-16T13:47:23.404948+0000","last_clean":"2023-10-16T13:47:23.404948+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:23.404948+0000","last_undegraded":"2023-10-16T13:47:23.404948+0000","last_fullsized":"2023-10-16T13:47:23.404948+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T13:47:23.404948+0000","last_change":"2023-10-16T13:47:23.404948+0000","last_active":"2023-10-16T13:47:23.404948+0000","last_peered":"2023-10-16T13:47:23.404948+0000","last_clean":"2023-10-16T13:47:23.404948+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:23.404948+0000","last_undegraded":"2023-10-16T13:47:23.404948+0000","last_fullsized":"2023-10-16T13:47:23.404948+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T13:47:23.404948+0000","last_change":"2023-10-16T13:47:23.404948+0000","last_active":"2023-10-16T13:47:23.404948+0000","last_peered":"2023-10-16T13:47:23.404948+0000","last_clean":"2023-10-16T13:47:23.404948+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:23.404948+0000","last_undegraded":"2023-10-16T13:47:23.404948+0000","last_fullsized":"2023-10-16T13:47:23.404948+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T13:47:23.404948+0000","last_change":"2023-10-16T13:47:23.404948+0000","last_active":"2023-10-16T13:47:23.404948+0000","last_peered":"2023-10-16T13:47:23.404948+0000","last_clean":"2023-10-16T13:47:23.404948+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:23.404948+0000","last_undegraded":"2023-10-16T13:47:23.404948+0000","last_fullsized":"2023-10-16T13:47:23.404948+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.7","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.416641+0000","last_change":"2023-10-16T13:47:24.433031+0000","last_active":"2023-10-16T13:47:25.416641+0000","last_peered":"2023-10-16T13:47:25.416641+0000","last_clean":"2023-10-16T13:47:25.416641+0000","last_became_active":"2023-10-16T13:47:24.432700+0000","last_became_peered":"2023-10-16T13:47:24.432700+0000","last_unstale":"2023-10-16T13:47:25.416641+0000","last_undegraded":"2023-10-16T13:47:25.416641+0000","last_fullsized":"2023-10-16T13:47:25.416641+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:17:10.080681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T13:47:23.404948+0000","last_change":"2023-10-16T13:47:23.404948+0000","last_active":"2023-10-16T13:47:23.404948+0000","last_peered":"2023-10-16T13:47:23.404948+0000","last_clean":"2023-10-16T13:47:23.404948+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:23.404948+0000","last_undegraded":"2023-10-16T13:47:23.404948+0000","last_fullsized":"2023-10-16T13:47:23.404948+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.101557+0000","last_change":"2023-10-16T13:47:24.428801+0000","last_active":"2023-10-16T13:47:26.101557+0000","last_peered":"2023-10-16T13:47:26.101557+0000","last_clean":"2023-10-16T13:47:26.101557+0000","last_became_active":"2023-10-16T13:47:24.428511+0000","last_became_peered":"2023-10-16T13:47:24.428511+0000","last_unstale":"2023-10-16T13:47:26.101557+0000","last_undegraded":"2023-10-16T13:47:26.101557+0000","last_fullsized":"2023-10-16T13:47:26.101557+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:27:55.268379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.417855+0000","last_change":"2023-10-16T13:47:24.429772+0000","last_active":"2023-10-16T13:47:25.417855+0000","last_peered":"2023-10-16T13:47:25.417855+0000","last_clean":"2023-10-16T13:47:25.417855+0000","last_became_active":"2023-10-16T13:47:24.429496+0000","last_became_peered":"2023-10-16T13:47:24.429496+0000","last_unstale":"2023-10-16T13:47:25.417855+0000","last_undegraded":"2023-10-16T13:47:25.417855+0000","last_fullsized":"2023-10-16T13:47:25.417855+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:09:48.751438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.099824+0000","last_change":"2023-10-16T13:47:24.437263+0000","last_active":"2023-10-16T13:47:26.099824+0000","last_peered":"2023-10-16T13:47:26.099824+0000","last_clean":"2023-10-16T13:47:26.099824+0000","last_became_active":"2023-10-16T13:47:24.437023+0000","last_became_peered":"2023-10-16T13:47:24.437023+0000","last_unstale":"2023-10-16T13:47:26.099824+0000","last_undegraded":"2023-10-16T13:47:26.099824+0000","last_fullsized":"2023-10-16T13:47:26.099824+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:59:15.061907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.099775+0000","last_change":"2023-10-16T13:47:24.419847+0000","last_active":"2023-10-16T13:47:26.099775+0000","last_peered":"2023-10-16T13:47:26.099775+0000","last_clean":"2023-10-16T13:47:26.099775+0000","last_became_active":"2023-10-16T13:47:24.419520+0000","last_became_peered":"2023-10-16T13:47:24.419520+0000","last_unstale":"2023-10-16T13:47:26.099775+0000","last_undegraded":"2023-10-16T13:47:26.099775+0000","last_fullsized":"2023-10-16T13:47:26.099775+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:11:44.932011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.416267+0000","last_change":"2023-10-16T13:47:24.435926+0000","last_active":"2023-10-16T13:47:25.416267+0000","last_peered":"2023-10-16T13:47:25.416267+0000","last_clean":"2023-10-16T13:47:25.416267+0000","last_became_active":"2023-10-16T13:47:24.434753+0000","last_became_peered":"2023-10-16T13:47:24.434753+0000","last_unstale":"2023-10-16T13:47:25.416267+0000","last_undegraded":"2023-10-16T13:47:25.416267+0000","last_fullsized":"2023-10-16T13:47:25.416267+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:13:51.332934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.417066+0000","last_change":"2023-10-16T13:47:24.423908+0000","last_active":"2023-10-16T13:47:25.417066+0000","last_peered":"2023-10-16T13:47:25.417066+0000","last_clean":"2023-10-16T13:47:25.417066+0000","last_became_active":"2023-10-16T13:47:24.423530+0000","last_became_peered":"2023-10-16T13:47:24.423530+0000","last_unstale":"2023-10-16T13:47:25.417066+0000","last_undegraded":"2023-10-16T13:47:25.417066+0000","last_fullsized":"2023-10-16T13:47:25.417066+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:50:50.396650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.20","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.416162+0000","last_change":"2023-10-16T13:47:24.436480+0000","last_active":"2023-10-16T13:47:25.416162+0000","last_peered":"2023-10-16T13:47:25.416162+0000","last_clean":"2023-10-16T13:47:25.416162+0000","last_became_active":"2023-10-16T13:47:24.435309+0000","last_became_peered":"2023-10-16T13:47:24.435309+0000","last_unstale":"2023-10-16T13:47:25.416162+0000","last_undegraded":"2023-10-16T13:47:25.416162+0000","last_fullsized":"2023-10-16T13:47:25.416162+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:28:08.947506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1f","version":"244'4","reported_seq":16,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.452436+0000","last_change":"2023-10-16T13:47:23.425349+0000","last_active":"2023-10-16T13:47:24.452436+0000","last_peered":"2023-10-16T13:47:24.452436+0000","last_clean":"2023-10-16T13:47:24.452436+0000","last_became_active":"2023-10-16T13:47:23.425029+0000","last_became_peered":"2023-10-16T13:47:23.425029+0000","last_unstale":"2023-10-16T13:47:24.452436+0000","last_undegraded":"2023-10-16T13:47:24.452436+0000","last_fullsized":"2023-10-16T13:47:24.452436+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:57:35.834032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T13:47:23.404948+0000","last_change":"2023-10-16T13:47:23.404948+0000","last_active":"2023-10-16T13:47:23.404948+0000","last_peered":"2023-10-16T13:47:23.404948+0000","last_clean":"2023-10-16T13:47:23.404948+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:23.404948+0000","last_undegraded":"2023-10-16T13:47:23.404948+0000","last_fullsized":"2023-10-16T13:47:23.404948+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.1d","version":"0'0","reported_seq":1,"reported_epoch":242,"state":"creating+peering","last_fresh":"2023-10-16T13:47:22.414037+0000","last_change":"2023-10-16T13:47:22.414037+0000","last_active":"2023-10-16T13:47:22.400586+0000","last_peered":"2023-10-16T13:47:22.400586+0000","last_clean":"2023-10-16T13:47:22.400586+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:22.414037+0000","last_undegraded":"2023-10-16T13:47:22.414037+0000","last_fullsized":"2023-10-16T13:47:22.414037+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.23","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.417464+0000","last_change":"2023-10-16T13:47:24.432436+0000","last_active":"2023-10-16T13:47:25.417464+0000","last_peered":"2023-10-16T13:47:25.417464+0000","last_clean":"2023-10-16T13:47:25.417464+0000","last_became_active":"2023-10-16T13:47:24.428465+0000","last_became_peered":"2023-10-16T13:47:24.428465+0000","last_unstale":"2023-10-16T13:47:25.417464+0000","last_undegraded":"2023-10-16T13:47:25.417464+0000","last_fullsized":"2023-10-16T13:47:25.417464+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:16:50.560075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1c","version":"244'2","reported_seq":14,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.447290+0000","last_change":"2023-10-16T13:47:23.427686+0000","last_active":"2023-10-16T13:47:24.447290+0000","last_peered":"2023-10-16T13:47:24.447290+0000","last_clean":"2023-10-16T13:47:24.447290+0000","last_became_active":"2023-10-16T13:47:23.427013+0000","last_became_peered":"2023-10-16T13:47:23.427013+0000","last_unstale":"2023-10-16T13:47:24.447290+0000","last_undegraded":"2023-10-16T13:47:24.447290+0000","last_fullsized":"2023-10-16T13:47:24.447290+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:37:17.532164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.24","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100065+0000","last_change":"2023-10-16T13:47:24.438644+0000","last_active":"2023-10-16T13:47:26.100065+0000","last_peered":"2023-10-16T13:47:26.100065+0000","last_clean":"2023-10-16T13:47:26.100065+0000","last_became_active":"2023-10-16T13:47:24.438344+0000","last_became_peered":"2023-10-16T13:47:24.438344+0000","last_unstale":"2023-10-16T13:47:26.100065+0000","last_undegraded":"2023-10-16T13:47:26.100065+0000","last_fullsized":"2023-10-16T13:47:26.100065+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:21:19.269920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.1b","version":"0'0","reported_seq":1,"reported_epoch":242,"state":"creating+peering","last_fresh":"2023-10-16T13:47:22.413990+0000","last_change":"2023-10-16T13:47:22.413990+0000","last_active":"2023-10-16T13:47:22.400586+0000","last_peered":"2023-10-16T13:47:22.400586+0000","last_clean":"2023-10-16T13:47:22.400586+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:22.413990+0000","last_undegraded":"2023-10-16T13:47:22.413990+0000","last_fullsized":"2023-10-16T13:47:22.413990+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T13:47:23.404948+0000","last_change":"2023-10-16T13:47:23.404948+0000","last_active":"2023-10-16T13:47:23.404948+0000","last_peered":"2023-10-16T13:47:23.404948+0000","last_clean":"2023-10-16T13:47:23.404948+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:23.404948+0000","last_undegraded":"2023-10-16T13:47:23.404948+0000","last_fullsized":"2023-10-16T13:47:23.404948+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.1a","version":"244'1","reported_seq":13,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.441041+0000","last_change":"2023-10-16T13:47:23.425695+0000","last_active":"2023-10-16T13:47:24.441041+0000","last_peered":"2023-10-16T13:47:24.441041+0000","last_clean":"2023-10-16T13:47:24.441041+0000","last_became_active":"2023-10-16T13:47:23.425407+0000","last_became_peered":"2023-10-16T13:47:23.425407+0000","last_unstale":"2023-10-16T13:47:24.441041+0000","last_undegraded":"2023-10-16T13:47:24.441041+0000","last_fullsized":"2023-10-16T13:47:24.441041+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:19:34.313200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.26","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100538+0000","last_change":"2023-10-16T13:47:24.439616+0000","last_active":"2023-10-16T13:47:26.100538+0000","last_peered":"2023-10-16T13:47:26.100538+0000","last_clean":"2023-10-16T13:47:26.100538+0000","last_became_active":"2023-10-16T13:47:24.435678+0000","last_became_peered":"2023-10-16T13:47:24.435678+0000","last_unstale":"2023-10-16T13:47:26.100538+0000","last_undegraded":"2023-10-16T13:47:26.100538+0000","last_fullsized":"2023-10-16T13:47:26.100538+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:39:16.296666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.418484+0000","last_change":"2023-10-16T13:47:23.431681+0000","last_active":"2023-10-16T13:47:24.418484+0000","last_peered":"2023-10-16T13:47:24.418484+0000","last_clean":"2023-10-16T13:47:24.418484+0000","last_became_active":"2023-10-16T13:47:23.431317+0000","last_became_peered":"2023-10-16T13:47:23.431317+0000","last_unstale":"2023-10-16T13:47:24.418484+0000","last_undegraded":"2023-10-16T13:47:24.418484+0000","last_fullsized":"2023-10-16T13:47:24.418484+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:20:27.852907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.27","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.101829+0000","last_change":"2023-10-16T13:47:24.423771+0000","last_active":"2023-10-16T13:47:26.101829+0000","last_peered":"2023-10-16T13:47:26.101829+0000","last_clean":"2023-10-16T13:47:26.101829+0000","last_became_active":"2023-10-16T13:47:24.423459+0000","last_became_peered":"2023-10-16T13:47:24.423459+0000","last_unstale":"2023-10-16T13:47:26.101829+0000","last_undegraded":"2023-10-16T13:47:26.101829+0000","last_fullsized":"2023-10-16T13:47:26.101829+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:27:22.393558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.421767+0000","last_change":"2023-10-16T13:47:23.433310+0000","last_active":"2023-10-16T13:47:24.421767+0000","last_peered":"2023-10-16T13:47:24.421767+0000","last_clean":"2023-10-16T13:47:24.421767+0000","last_became_active":"2023-10-16T13:47:23.433018+0000","last_became_peered":"2023-10-16T13:47:23.433018+0000","last_unstale":"2023-10-16T13:47:24.421767+0000","last_undegraded":"2023-10-16T13:47:24.421767+0000","last_fullsized":"2023-10-16T13:47:24.421767+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:49:45.992161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T13:47:23.404948+0000","last_change":"2023-10-16T13:47:23.404948+0000","last_active":"2023-10-16T13:47:23.404948+0000","last_peered":"2023-10-16T13:47:23.404948+0000","last_clean":"2023-10-16T13:47:23.404948+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:23.404948+0000","last_undegraded":"2023-10-16T13:47:23.404948+0000","last_fullsized":"2023-10-16T13:47:23.404948+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.415527+0000","last_change":"2023-10-16T13:47:23.428721+0000","last_active":"2023-10-16T13:47:24.415527+0000","last_peered":"2023-10-16T13:47:24.415527+0000","last_clean":"2023-10-16T13:47:24.415527+0000","last_became_active":"2023-10-16T13:47:23.428349+0000","last_became_peered":"2023-10-16T13:47:23.428349+0000","last_unstale":"2023-10-16T13:47:24.415527+0000","last_undegraded":"2023-10-16T13:47:24.415527+0000","last_fullsized":"2023-10-16T13:47:24.415527+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:59:11.116638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T13:47:23.404948+0000","last_change":"2023-10-16T13:47:23.404948+0000","last_active":"2023-10-16T13:47:23.404948+0000","last_peered":"2023-10-16T13:47:23.404948+0000","last_clean":"2023-10-16T13:47:23.404948+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:23.404948+0000","last_undegraded":"2023-10-16T13:47:23.404948+0000","last_fullsized":"2023-10-16T13:47:23.404948+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.421423+0000","last_change":"2023-10-16T13:47:23.433545+0000","last_active":"2023-10-16T13:47:24.421423+0000","last_peered":"2023-10-16T13:47:24.421423+0000","last_clean":"2023-10-16T13:47:24.421423+0000","last_became_active":"2023-10-16T13:47:23.433275+0000","last_became_peered":"2023-10-16T13:47:23.433275+0000","last_unstale":"2023-10-16T13:47:24.421423+0000","last_undegraded":"2023-10-16T13:47:24.421423+0000","last_fullsized":"2023-10-16T13:47:24.421423+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:21:17.525717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.2a","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100402+0000","last_change":"2023-10-16T13:47:24.424605+0000","last_active":"2023-10-16T13:47:26.100402+0000","last_peered":"2023-10-16T13:47:26.100402+0000","last_clean":"2023-10-16T13:47:26.100402+0000","last_became_active":"2023-10-16T13:47:24.424313+0000","last_became_peered":"2023-10-16T13:47:24.424313+0000","last_unstale":"2023-10-16T13:47:26.100402+0000","last_undegraded":"2023-10-16T13:47:26.100402+0000","last_fullsized":"2023-10-16T13:47:26.100402+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:40:33.368782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.15","version":"244'3","reported_seq":15,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.446109+0000","last_change":"2023-10-16T13:47:23.431025+0000","last_active":"2023-10-16T13:47:24.446109+0000","last_peered":"2023-10-16T13:47:24.446109+0000","last_clean":"2023-10-16T13:47:24.446109+0000","last_became_active":"2023-10-16T13:47:23.430710+0000","last_became_peered":"2023-10-16T13:47:23.430710+0000","last_unstale":"2023-10-16T13:47:24.446109+0000","last_undegraded":"2023-10-16T13:47:24.446109+0000","last_fullsized":"2023-10-16T13:47:24.446109+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:52:27.207432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T13:47:23.404948+0000","last_change":"2023-10-16T13:47:23.404948+0000","last_active":"2023-10-16T13:47:23.404948+0000","last_peered":"2023-10-16T13:47:23.404948+0000","last_clean":"2023-10-16T13:47:23.404948+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:23.404948+0000","last_undegraded":"2023-10-16T13:47:23.404948+0000","last_fullsized":"2023-10-16T13:47:23.404948+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.14","version":"244'6","reported_seq":18,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.453158+0000","last_change":"2023-10-16T13:47:23.430528+0000","last_active":"2023-10-16T13:47:24.453158+0000","last_peered":"2023-10-16T13:47:24.453158+0000","last_clean":"2023-10-16T13:47:24.453158+0000","last_became_active":"2023-10-16T13:47:23.429721+0000","last_became_peered":"2023-10-16T13:47:23.429721+0000","last_unstale":"2023-10-16T13:47:24.453158+0000","last_undegraded":"2023-10-16T13:47:24.453158+0000","last_fullsized":"2023-10-16T13:47:24.453158+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:07:56.379842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.2c","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.416579+0000","last_change":"2023-10-16T13:47:24.436132+0000","last_active":"2023-10-16T13:47:25.416579+0000","last_peered":"2023-10-16T13:47:25.416579+0000","last_clean":"2023-10-16T13:47:25.416579+0000","last_became_active":"2023-10-16T13:47:24.435272+0000","last_became_peered":"2023-10-16T13:47:24.435272+0000","last_unstale":"2023-10-16T13:47:25.416579+0000","last_undegraded":"2023-10-16T13:47:25.416579+0000","last_fullsized":"2023-10-16T13:47:25.416579+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:07:59.434149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.13","version":"0'0","reported_seq":1,"reported_epoch":242,"state":"creating+peering","last_fresh":"2023-10-16T13:47:22.412143+0000","last_change":"2023-10-16T13:47:22.412143+0000","last_active":"2023-10-16T13:47:22.400586+0000","last_peered":"2023-10-16T13:47:22.400586+0000","last_clean":"2023-10-16T13:47:22.400586+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:22.412143+0000","last_undegraded":"2023-10-16T13:47:22.412143+0000","last_fullsized":"2023-10-16T13:47:22.412143+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T13:47:23.404948+0000","last_change":"2023-10-16T13:47:23.404948+0000","last_active":"2023-10-16T13:47:23.404948+0000","last_peered":"2023-10-16T13:47:23.404948+0000","last_clean":"2023-10-16T13:47:23.404948+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:23.404948+0000","last_undegraded":"2023-10-16T13:47:23.404948+0000","last_fullsized":"2023-10-16T13:47:23.404948+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.418559+0000","last_change":"2023-10-16T13:47:23.433222+0000","last_active":"2023-10-16T13:47:24.418559+0000","last_peered":"2023-10-16T13:47:24.418559+0000","last_clean":"2023-10-16T13:47:24.418559+0000","last_became_active":"2023-10-16T13:47:23.432936+0000","last_became_peered":"2023-10-16T13:47:23.432936+0000","last_unstale":"2023-10-16T13:47:24.418559+0000","last_undegraded":"2023-10-16T13:47:24.418559+0000","last_fullsized":"2023-10-16T13:47:24.418559+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:47:59.336633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T13:47:23.404948+0000","last_change":"2023-10-16T13:47:23.404948+0000","last_active":"2023-10-16T13:47:23.404948+0000","last_peered":"2023-10-16T13:47:23.404948+0000","last_clean":"2023-10-16T13:47:23.404948+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:23.404948+0000","last_undegraded":"2023-10-16T13:47:23.404948+0000","last_fullsized":"2023-10-16T13:47:23.404948+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.417965+0000","last_change":"2023-10-16T13:47:23.425872+0000","last_active":"2023-10-16T13:47:24.417965+0000","last_peered":"2023-10-16T13:47:24.417965+0000","last_clean":"2023-10-16T13:47:24.417965+0000","last_became_active":"2023-10-16T13:47:23.425569+0000","last_became_peered":"2023-10-16T13:47:23.425569+0000","last_unstale":"2023-10-16T13:47:24.417965+0000","last_undegraded":"2023-10-16T13:47:24.417965+0000","last_fullsized":"2023-10-16T13:47:24.417965+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:09:18.086723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2f","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.101809+0000","last_change":"2023-10-16T13:47:24.437030+0000","last_active":"2023-10-16T13:47:26.101809+0000","last_peered":"2023-10-16T13:47:26.101809+0000","last_clean":"2023-10-16T13:47:26.101809+0000","last_became_active":"2023-10-16T13:47:24.436734+0000","last_became_peered":"2023-10-16T13:47:24.436734+0000","last_unstale":"2023-10-16T13:47:26.101809+0000","last_undegraded":"2023-10-16T13:47:26.101809+0000","last_fullsized":"2023-10-16T13:47:26.101809+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:28:53.363293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.10","version":"244'8","reported_seq":20,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.453887+0000","last_change":"2023-10-16T13:47:23.426967+0000","last_active":"2023-10-16T13:47:24.453887+0000","last_peered":"2023-10-16T13:47:24.453887+0000","last_clean":"2023-10-16T13:47:24.453887+0000","last_became_active":"2023-10-16T13:47:23.426614+0000","last_became_peered":"2023-10-16T13:47:23.426614+0000","last_unstale":"2023-10-16T13:47:24.453887+0000","last_undegraded":"2023-10-16T13:47:24.453887+0000","last_fullsized":"2023-10-16T13:47:24.453887+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:39:12.211841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.30","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.101224+0000","last_change":"2023-10-16T13:47:24.437081+0000","last_active":"2023-10-16T13:47:26.101224+0000","last_peered":"2023-10-16T13:47:26.101224+0000","last_clean":"2023-10-16T13:47:26.101224+0000","last_became_active":"2023-10-16T13:47:24.436501+0000","last_became_peered":"2023-10-16T13:47:24.436501+0000","last_unstale":"2023-10-16T13:47:26.101224+0000","last_undegraded":"2023-10-16T13:47:26.101224+0000","last_fullsized":"2023-10-16T13:47:26.101224+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:23:30.831964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.f","version":"244'1","reported_seq":13,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.445592+0000","last_change":"2023-10-16T13:47:23.432161+0000","last_active":"2023-10-16T13:47:24.445592+0000","last_peered":"2023-10-16T13:47:24.445592+0000","last_clean":"2023-10-16T13:47:24.445592+0000","last_became_active":"2023-10-16T13:47:23.431822+0000","last_became_peered":"2023-10-16T13:47:23.431822+0000","last_unstale":"2023-10-16T13:47:24.445592+0000","last_undegraded":"2023-10-16T13:47:24.445592+0000","last_fullsized":"2023-10-16T13:47:24.445592+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:15:50.867051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.31","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100923+0000","last_change":"2023-10-16T13:47:24.431186+0000","last_active":"2023-10-16T13:47:26.100923+0000","last_peered":"2023-10-16T13:47:26.100923+0000","last_clean":"2023-10-16T13:47:26.100923+0000","last_became_active":"2023-10-16T13:47:24.430892+0000","last_became_peered":"2023-10-16T13:47:24.430892+0000","last_unstale":"2023-10-16T13:47:26.100923+0000","last_undegraded":"2023-10-16T13:47:26.100923+0000","last_fullsized":"2023-10-16T13:47:26.100923+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:12:21.693658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.e","version":"244'2","reported_seq":14,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.448939+0000","last_change":"2023-10-16T13:47:23.424549+0000","last_active":"2023-10-16T13:47:24.448939+0000","last_peered":"2023-10-16T13:47:24.448939+0000","last_clean":"2023-10-16T13:47:24.448939+0000","last_became_active":"2023-10-16T13:47:23.424111+0000","last_became_peered":"2023-10-16T13:47:23.424111+0000","last_unstale":"2023-10-16T13:47:24.448939+0000","last_undegraded":"2023-10-16T13:47:24.448939+0000","last_fullsized":"2023-10-16T13:47:24.448939+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:36:35.137200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.32","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.416229+0000","last_change":"2023-10-16T13:47:24.432940+0000","last_active":"2023-10-16T13:47:25.416229+0000","last_peered":"2023-10-16T13:47:25.416229+0000","last_clean":"2023-10-16T13:47:25.416229+0000","last_became_active":"2023-10-16T13:47:24.432610+0000","last_became_peered":"2023-10-16T13:47:24.432610+0000","last_unstale":"2023-10-16T13:47:25.416229+0000","last_undegraded":"2023-10-16T13:47:25.416229+0000","last_fullsized":"2023-10-16T13:47:25.416229+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:59:08.104923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.d","version":"0'0","reported_seq":1,"reported_epoch":242,"state":"creating+peering","last_fresh":"2023-10-16T13:47:22.413572+0000","last_change":"2023-10-16T13:47:22.413572+0000","last_active":"2023-10-16T13:47:22.400586+0000","last_peered":"2023-10-16T13:47:22.400586+0000","last_clean":"2023-10-16T13:47:22.400586+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:22.413572+0000","last_undegraded":"2023-10-16T13:47:22.413572+0000","last_fullsized":"2023-10-16T13:47:22.413572+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.33","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100188+0000","last_change":"2023-10-16T13:47:24.427239+0000","last_active":"2023-10-16T13:47:26.100188+0000","last_peered":"2023-10-16T13:47:26.100188+0000","last_clean":"2023-10-16T13:47:26.100188+0000","last_became_active":"2023-10-16T13:47:24.426919+0000","last_became_peered":"2023-10-16T13:47:24.426919+0000","last_unstale":"2023-10-16T13:47:26.100188+0000","last_undegraded":"2023-10-16T13:47:26.100188+0000","last_fullsized":"2023-10-16T13:47:26.100188+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:56:16.580840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":1,"reported_epoch":242,"state":"creating+peering","last_fresh":"2023-10-16T13:47:22.412157+0000","last_change":"2023-10-16T13:47:22.412157+0000","last_active":"2023-10-16T13:47:22.400586+0000","last_peered":"2023-10-16T13:47:22.400586+0000","last_clean":"2023-10-16T13:47:22.400586+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:22.412157+0000","last_undegraded":"2023-10-16T13:47:22.412157+0000","last_fullsized":"2023-10-16T13:47:22.412157+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.34","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100514+0000","last_change":"2023-10-16T13:47:24.444364+0000","last_active":"2023-10-16T13:47:26.100514+0000","last_peered":"2023-10-16T13:47:26.100514+0000","last_clean":"2023-10-16T13:47:26.100514+0000","last_became_active":"2023-10-16T13:47:24.444045+0000","last_became_peered":"2023-10-16T13:47:24.444045+0000","last_unstale":"2023-10-16T13:47:26.100514+0000","last_undegraded":"2023-10-16T13:47:26.100514+0000","last_fullsized":"2023-10-16T13:47:26.100514+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:13:11.795921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.b","version":"244'2","reported_seq":14,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.449802+0000","last_change":"2023-10-16T13:47:23.434957+0000","last_active":"2023-10-16T13:47:24.449802+0000","last_peered":"2023-10-16T13:47:24.449802+0000","last_clean":"2023-10-16T13:47:24.449802+0000","last_became_active":"2023-10-16T13:47:23.434637+0000","last_became_peered":"2023-10-16T13:47:23.434637+0000","last_unstale":"2023-10-16T13:47:24.449802+0000","last_undegraded":"2023-10-16T13:47:24.449802+0000","last_fullsized":"2023-10-16T13:47:24.449802+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:23:47.081423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.35","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.416431+0000","last_change":"2023-10-16T13:47:24.431892+0000","last_active":"2023-10-16T13:47:25.416431+0000","last_peered":"2023-10-16T13:47:25.416431+0000","last_clean":"2023-10-16T13:47:25.416431+0000","last_became_active":"2023-10-16T13:47:24.431624+0000","last_became_peered":"2023-10-16T13:47:24.431624+0000","last_unstale":"2023-10-16T13:47:25.416431+0000","last_undegraded":"2023-10-16T13:47:25.416431+0000","last_fullsized":"2023-10-16T13:47:25.416431+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:49:13.422163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":1,"reported_epoch":242,"state":"creating+peering","last_fresh":"2023-10-16T13:47:22.412844+0000","last_change":"2023-10-16T13:47:22.412844+0000","last_active":"2023-10-16T13:47:22.400586+0000","last_peered":"2023-10-16T13:47:22.400586+0000","last_clean":"2023-10-16T13:47:22.400586+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:22.412844+0000","last_undegraded":"2023-10-16T13:47:22.412844+0000","last_fullsized":"2023-10-16T13:47:22.412844+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T13:47:23.404948+0000","last_change":"2023-10-16T13:47:23.404948+0000","last_active":"2023-10-16T13:47:23.404948+0000","last_peered":"2023-10-16T13:47:23.404948+0000","last_clean":"2023-10-16T13:47:23.404948+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:23.404948+0000","last_undegraded":"2023-10-16T13:47:23.404948+0000","last_fullsized":"2023-10-16T13:47:23.404948+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":1,"reported_epoch":242,"state":"creating+peering","last_fresh":"2023-10-16T13:47:22.412735+0000","last_change":"2023-10-16T13:47:22.412735+0000","last_active":"2023-10-16T13:47:22.400586+0000","last_peered":"2023-10-16T13:47:22.400586+0000","last_clean":"2023-10-16T13:47:22.400586+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:22.412735+0000","last_undegraded":"2023-10-16T13:47:22.412735+0000","last_fullsized":"2023-10-16T13:47:22.412735+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T13:47:23.404948+0000","last_change":"2023-10-16T13:47:23.404948+0000","last_active":"2023-10-16T13:47:23.404948+0000","last_peered":"2023-10-16T13:47:23.404948+0000","last_clean":"2023-10-16T13:47:23.404948+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:23.404948+0000","last_undegraded":"2023-10-16T13:47:23.404948+0000","last_fullsized":"2023-10-16T13:47:23.404948+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.415241+0000","last_change":"2023-10-16T13:47:23.433223+0000","last_active":"2023-10-16T13:47:24.415241+0000","last_peered":"2023-10-16T13:47:24.415241+0000","last_clean":"2023-10-16T13:47:24.415241+0000","last_became_active":"2023-10-16T13:47:23.432935+0000","last_became_peered":"2023-10-16T13:47:23.432935+0000","last_unstale":"2023-10-16T13:47:24.415241+0000","last_undegraded":"2023-10-16T13:47:24.415241+0000","last_fullsized":"2023-10-16T13:47:24.415241+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:40:59.034875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T13:47:23.404948+0000","last_change":"2023-10-16T13:47:23.404948+0000","last_active":"2023-10-16T13:47:23.404948+0000","last_peered":"2023-10-16T13:47:23.404948+0000","last_clean":"2023-10-16T13:47:23.404948+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:23.404948+0000","last_undegraded":"2023-10-16T13:47:23.404948+0000","last_fullsized":"2023-10-16T13:47:23.404948+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.7","version":"244'2","reported_seq":14,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.451763+0000","last_change":"2023-10-16T13:47:23.430865+0000","last_active":"2023-10-16T13:47:24.451763+0000","last_peered":"2023-10-16T13:47:24.451763+0000","last_clean":"2023-10-16T13:47:24.451763+0000","last_became_active":"2023-10-16T13:47:23.430493+0000","last_became_peered":"2023-10-16T13:47:23.430493+0000","last_unstale":"2023-10-16T13:47:24.451763+0000","last_undegraded":"2023-10-16T13:47:24.451763+0000","last_fullsized":"2023-10-16T13:47:24.451763+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:59:14.057546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T13:47:23.404948+0000","last_change":"2023-10-16T13:47:23.404948+0000","last_active":"2023-10-16T13:47:23.404948+0000","last_peered":"2023-10-16T13:47:23.404948+0000","last_clean":"2023-10-16T13:47:23.404948+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:23.404948+0000","last_undegraded":"2023-10-16T13:47:23.404948+0000","last_fullsized":"2023-10-16T13:47:23.404948+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.417912+0000","last_change":"2023-10-16T13:47:23.430851+0000","last_active":"2023-10-16T13:47:24.417912+0000","last_peered":"2023-10-16T13:47:24.417912+0000","last_clean":"2023-10-16T13:47:24.417912+0000","last_became_active":"2023-10-16T13:47:23.430526+0000","last_became_peered":"2023-10-16T13:47:23.430526+0000","last_unstale":"2023-10-16T13:47:24.417912+0000","last_undegraded":"2023-10-16T13:47:24.417912+0000","last_fullsized":"2023-10-16T13:47:24.417912+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:40:16.878541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.3a","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100155+0000","last_change":"2023-10-16T13:47:24.440347+0000","last_active":"2023-10-16T13:47:26.100155+0000","last_peered":"2023-10-16T13:47:26.100155+0000","last_clean":"2023-10-16T13:47:26.100155+0000","last_became_active":"2023-10-16T13:47:24.440041+0000","last_became_peered":"2023-10-16T13:47:24.440041+0000","last_unstale":"2023-10-16T13:47:26.100155+0000","last_undegraded":"2023-10-16T13:47:26.100155+0000","last_fullsized":"2023-10-16T13:47:26.100155+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:19:38.293171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.415325+0000","last_change":"2023-10-16T13:47:23.427399+0000","last_active":"2023-10-16T13:47:24.415325+0000","last_peered":"2023-10-16T13:47:24.415325+0000","last_clean":"2023-10-16T13:47:24.415325+0000","last_became_active":"2023-10-16T13:47:23.427092+0000","last_became_peered":"2023-10-16T13:47:23.427092+0000","last_unstale":"2023-10-16T13:47:24.415325+0000","last_undegraded":"2023-10-16T13:47:24.415325+0000","last_fullsized":"2023-10-16T13:47:24.415325+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:10:56.739817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3b","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.101556+0000","last_change":"2023-10-16T13:47:24.436718+0000","last_active":"2023-10-16T13:47:26.101556+0000","last_peered":"2023-10-16T13:47:26.101556+0000","last_clean":"2023-10-16T13:47:26.101556+0000","last_became_active":"2023-10-16T13:47:24.436371+0000","last_became_peered":"2023-10-16T13:47:24.436371+0000","last_unstale":"2023-10-16T13:47:26.101556+0000","last_undegraded":"2023-10-16T13:47:26.101556+0000","last_fullsized":"2023-10-16T13:47:26.101556+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:58:12.134727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.416043+0000","last_change":"2023-10-16T13:47:23.433833+0000","last_active":"2023-10-16T13:47:24.416043+0000","last_peered":"2023-10-16T13:47:24.416043+0000","last_clean":"2023-10-16T13:47:24.416043+0000","last_became_active":"2023-10-16T13:47:23.433381+0000","last_became_peered":"2023-10-16T13:47:23.433381+0000","last_unstale":"2023-10-16T13:47:24.416043+0000","last_undegraded":"2023-10-16T13:47:24.416043+0000","last_fullsized":"2023-10-16T13:47:24.416043+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:09:34.543970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3c","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.099990+0000","last_change":"2023-10-16T13:47:24.424889+0000","last_active":"2023-10-16T13:47:26.099990+0000","last_peered":"2023-10-16T13:47:26.099990+0000","last_clean":"2023-10-16T13:47:26.099990+0000","last_became_active":"2023-10-16T13:47:24.424573+0000","last_became_peered":"2023-10-16T13:47:24.424573+0000","last_unstale":"2023-10-16T13:47:26.099990+0000","last_undegraded":"2023-10-16T13:47:26.099990+0000","last_fullsized":"2023-10-16T13:47:26.099990+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:23:40.809515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":1,"reported_epoch":242,"state":"creating+peering","last_fresh":"2023-10-16T13:47:22.413180+0000","last_change":"2023-10-16T13:47:22.413180+0000","last_active":"2023-10-16T13:47:22.400586+0000","last_peered":"2023-10-16T13:47:22.400586+0000","last_clean":"2023-10-16T13:47:22.400586+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:22.413180+0000","last_undegraded":"2023-10-16T13:47:22.413180+0000","last_fullsized":"2023-10-16T13:47:22.413180+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.3d","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.417707+0000","last_change":"2023-10-16T13:47:24.430727+0000","last_active":"2023-10-16T13:47:25.417707+0000","last_peered":"2023-10-16T13:47:25.417707+0000","last_clean":"2023-10-16T13:47:25.417707+0000","last_became_active":"2023-10-16T13:47:24.430424+0000","last_became_peered":"2023-10-16T13:47:24.430424+0000","last_unstale":"2023-10-16T13:47:25.417707+0000","last_undegraded":"2023-10-16T13:47:25.417707+0000","last_fullsized":"2023-10-16T13:47:25.417707+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:05:49.279256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.418627+0000","last_change":"2023-10-16T13:47:23.430779+0000","last_active":"2023-10-16T13:47:24.418627+0000","last_peered":"2023-10-16T13:47:24.418627+0000","last_clean":"2023-10-16T13:47:24.418627+0000","last_became_active":"2023-10-16T13:47:23.429996+0000","last_became_peered":"2023-10-16T13:47:23.429996+0000","last_unstale":"2023-10-16T13:47:24.418627+0000","last_undegraded":"2023-10-16T13:47:24.418627+0000","last_fullsized":"2023-10-16T13:47:24.418627+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:17:22.079776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.3e","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.416486+0000","last_change":"2023-10-16T13:47:24.436378+0000","last_active":"2023-10-16T13:47:25.416486+0000","last_peered":"2023-10-16T13:47:25.416486+0000","last_clean":"2023-10-16T13:47:25.416486+0000","last_became_active":"2023-10-16T13:47:24.435495+0000","last_became_peered":"2023-10-16T13:47:24.435495+0000","last_unstale":"2023-10-16T13:47:25.416486+0000","last_undegraded":"2023-10-16T13:47:25.416486+0000","last_fullsized":"2023-10-16T13:47:25.416486+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:18:23.073022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.420122+0000","last_change":"2023-10-16T13:47:23.433945+0000","last_active":"2023-10-16T13:47:24.420122+0000","last_peered":"2023-10-16T13:47:24.420122+0000","last_clean":"2023-10-16T13:47:24.420122+0000","last_became_active":"2023-10-16T13:47:23.433576+0000","last_became_peered":"2023-10-16T13:47:23.433576+0000","last_unstale":"2023-10-16T13:47:24.420122+0000","last_undegraded":"2023-10-16T13:47:24.420122+0000","last_fullsized":"2023-10-16T13:47:24.420122+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:09:34.543970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T13:47:23.404948+0000","last_change":"2023-10-16T13:47:23.404948+0000","last_active":"2023-10-16T13:47:23.404948+0000","last_peered":"2023-10-16T13:47:23.404948+0000","last_clean":"2023-10-16T13:47:23.404948+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:23.404948+0000","last_undegraded":"2023-10-16T13:47:23.404948+0000","last_fullsized":"2023-10-16T13:47:23.404948+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":1,"reported_epoch":242,"state":"creating+peering","last_fresh":"2023-10-16T13:47:22.412533+0000","last_change":"2023-10-16T13:47:22.412533+0000","last_active":"2023-10-16T13:47:22.400586+0000","last_peered":"2023-10-16T13:47:22.400586+0000","last_clean":"2023-10-16T13:47:22.400586+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:22.412533+0000","last_undegraded":"2023-10-16T13:47:22.412533+0000","last_fullsized":"2023-10-16T13:47:22.412533+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.21","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.417066+0000","last_change":"2023-10-16T13:47:24.431657+0000","last_active":"2023-10-16T13:47:25.417066+0000","last_peered":"2023-10-16T13:47:25.417066+0000","last_clean":"2023-10-16T13:47:25.417066+0000","last_became_active":"2023-10-16T13:47:24.431385+0000","last_became_peered":"2023-10-16T13:47:24.431385+0000","last_unstale":"2023-10-16T13:47:25.417066+0000","last_undegraded":"2023-10-16T13:47:25.417066+0000","last_fullsized":"2023-10-16T13:47:25.417066+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:59:09.754185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1e","version":"0'0","reported_seq":1,"reported_epoch":242,"state":"creating+peering","last_fresh":"2023-10-16T13:47:22.406892+0000","last_change":"2023-10-16T13:47:22.406892+0000","last_active":"2023-10-16T13:47:22.400586+0000","last_peered":"2023-10-16T13:47:22.400586+0000","last_clean":"2023-10-16T13:47:22.400586+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:22.406892+0000","last_undegraded":"2023-10-16T13:47:22.406892+0000","last_fullsized":"2023-10-16T13:47:22.406892+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"231'370","reported_seq":1295,"reported_epoch":241,"state":"active+clean","last_fresh":"2023-10-16T13:47:20.908271+0000","last_change":"2023-10-16T12:10:34.746820+0000","last_active":"2023-10-16T13:47:20.908271+0000","last_peered":"2023-10-16T13:47:20.908271+0000","last_clean":"2023-10-16T13:47:20.908271+0000","last_became_active":"2023-10-16T12:10:34.745574+0000","last_became_peered":"2023-10-16T12:10:34.745574+0000","last_unstale":"2023-10-16T13:47:20.908271+0000","last_undegraded":"2023-10-16T13:47:20.908271+0000","last_fullsized":"2023-10-16T13:47:20.908271+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_clean_scrub_stamp":"2023-10-16T12:10:33.577111+0000","objects_scrubbed":0,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:14.461986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":32,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":84,"acting":84,"num_store_stats":5},{"poolid":31,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":37,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":4626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31,"ondisk_log_size":31,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":370,"ondisk_log_size":370,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575995,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56936,"kb_used_data":3548,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314904,"statfs":{"total":96636764160,"available":96578461696,"internally_reserved":0,"allocated":3633152,"data_stored":2159844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54655435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:46:50 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.60299999999999998,"15min":0.63200000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.33000000000000002},"max":{"1min":0.78600000000000003,"5min":0.87,"15min":0.93799999999999994},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.60699999999999998,"15min":0.626},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.72099999999999997,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.63900000000000001}]},{"osd":1,"last update":"Mon Oct 16 13:46:50 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.61299999999999999,"15min":0.63900000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.79000000000000004,"5min":0.84599999999999997,"15min":0.95899999999999996},"last":0.497},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.622,"15min":0.63100000000000001},"min":{"1min":0.38200000000000001,"5min":0.375,"15min":0.375},"max":{"1min":0.76200000000000001,"5min":0.79100000000000004,"15min":0.86399999999999999},"last":0.54900000000000004}]},{"osd":2,"last update":"Mon Oct 16 13:46:50 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.67400000000000004,"15min":0.68600000000000005},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.80300000000000005,"5min":0.83899999999999997,"15min":0.88200000000000001},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.68899999999999995,"15min":0.69799999999999995},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.39400000000000002},"max":{"1min":0.83699999999999997,"5min":0.92800000000000005,"15min":0.93799999999999994},"last":0.66100000000000003}]},{"osd":3,"last update":"Mon Oct 16 13:46:50 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.63800000000000001,"15min":0.65900000000000003},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.85299999999999998,"5min":0.89200000000000002,"15min":0.91300000000000003},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.65000000000000002,"15min":0.66000000000000003},"min":{"1min":0.435,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.88,"5min":1.24,"15min":1.24},"last":0.69599999999999995}]},{"osd":4,"last update":"Mon Oct 16 13:46:50 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.52000000000000002,"15min":0.51900000000000002},"min":{"1min":0.311,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.71099999999999997,"5min":0.745,"15min":0.85299999999999998},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.621,"15min":0.61799999999999999},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.33100000000000002},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.70899999999999996}]},{"osd":5,"last update":"Mon Oct 16 13:46:50 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.54900000000000004,"15min":0.54700000000000004},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.85699999999999998,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.58399999999999996,"15min":0.58699999999999997},"min":{"1min":0.41099999999999998,"5min":0.38400000000000001,"15min":0.35099999999999998},"max":{"1min":0.81100000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.73999999999999999}]},{"osd":6,"last update":"Mon Oct 16 13:46:50 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.60499999999999998,"15min":0.61299999999999999},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.85499999999999998,"5min":0.874,"15min":0.95899999999999996},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.626,"15min":0.61899999999999999},"min":{"1min":0.39400000000000002,"5min":0.378,"15min":0.35199999999999998},"max":{"1min":0.92100000000000004,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.76200000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575994,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60572,"kb_used_data":3024,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311268,"statfs":{"total":96636764160,"available":96574738432,"internally_reserved":0,"allocated":3096576,"data_stored":1596547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58915271},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:46:36 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.65300000000000002,"15min":0.66800000000000004},"min":{"1min":0.41499999999999998,"5min":0.375,"15min":0.28699999999999998},"max":{"1min":0.82599999999999996,"5min":0.89400000000000002,"15min":0.92300000000000004},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.64500000000000002,"15min":0.66400000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.80400000000000005,"5min":0.91400000000000003,"15min":0.93799999999999994},"last":0.68200000000000005}]},{"osd":1,"last update":"Mon Oct 16 13:46:36 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.58599999999999997,"15min":0.624},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.33100000000000002},"max":{"1min":0.70299999999999996,"5min":0.86899999999999999,"15min":0.90900000000000003},"last":0.504},{"interface":"front","average":{"1min":0.63,"5min":0.61399999999999999,"15min":0.62},"min":{"1min":0.41699999999999998,"5min":0.39800000000000002,"15min":0.38300000000000001},"max":{"1min":0.85799999999999998,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.42199999999999999}]},{"osd":2,"last update":"Mon Oct 16 13:46:36 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.622,"15min":0.63300000000000001},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.85799999999999998,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.59999999999999998,"15min":0.624},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.749,"5min":0.89000000000000001,"15min":0.91000000000000003},"last":0.44500000000000001}]},{"osd":3,"last update":"Mon Oct 16 13:46:36 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63900000000000001,"15min":0.66300000000000003},"min":{"1min":0.439,"5min":0.39900000000000002,"15min":0.38200000000000001},"max":{"1min":0.874,"5min":0.92400000000000004,"15min":0.95599999999999996},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.69099999999999995,"15min":0.71599999999999997},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.42299999999999999},"max":{"1min":0.85099999999999998,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.48099999999999998}]},{"osd":4,"last update":"Mon Oct 16 13:46:36 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.51800000000000002,"15min":0.52700000000000002},"min":{"1min":0.39000000000000001,"5min":0.35999999999999999,"15min":0.315},"max":{"1min":0.78000000000000003,"5min":0.85599999999999998,"15min":0.95799999999999996},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61699999999999999,"15min":0.62},"min":{"1min":0.375,"5min":0.35499999999999998,"15min":0.32900000000000001},"max":{"1min":0.83699999999999997,"5min":0.98799999999999999,"15min":1.0309999999999999},"last":0.64600000000000002}]},{"osd":5,"last update":"Mon Oct 16 13:46:36 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.59799999999999998,"15min":0.59399999999999997},"min":{"1min":0.39800000000000002,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.81100000000000005,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.66500000000000004,"15min":0.66000000000000003},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.308},"max":{"1min":0.85899999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.53500000000000003}]},{"osd":7,"last update":"Mon Oct 16 13:46:36 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.60899999999999999,"15min":0.60499999999999998},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.80000000000000004,"5min":1.081,"15min":1.081},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65200000000000002},"min":{"1min":0.42799999999999999,"5min":0.41899999999999998,"15min":0.375},"max":{"1min":0.91500000000000004,"5min":1.099,"15min":1.099},"last":0.629}]}]},{"osd":1,"up_from":13,"seq":55834575994,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52252,"kb_used_data":3016,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319588,"statfs":{"total":96636764160,"available":96583258112,"internally_reserved":0,"allocated":3088384,"data_stored":1596435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":50395593},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:46:55 2023","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.39500000000000002,"15min":0.40699999999999997},"min":{"1min":0.27600000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.51300000000000001,"5min":0.65400000000000003,"15min":0.65800000000000003},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.47599999999999998,"15min":0.49199999999999999},"min":{"1min":0.32000000000000001,"5min":0.253,"15min":0.253},"max":{"1min":0.56000000000000005,"5min":0.70999999999999996,"15min":0.77200000000000002},"last":0.67000000000000004}]},{"osd":2,"last update":"Mon Oct 16 13:46:55 2023","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.436,"15min":0.45100000000000001},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.58599999999999997,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.48099999999999998,"15min":0.48999999999999999},"min":{"1min":0.34399999999999997,"5min":0.315,"15min":0.30199999999999999},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.71399999999999997},"last":0.65200000000000002}]},{"osd":3,"last update":"Mon Oct 16 13:46:55 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.42899999999999999,"15min":0.42799999999999999},"min":{"1min":0.307,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.50800000000000001,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.46100000000000002,"15min":0.47499999999999998},"min":{"1min":0.29799999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.52800000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.69399999999999995}]},{"osd":4,"last update":"Mon Oct 16 13:46:55 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.64400000000000002,"15min":0.66200000000000003},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.42399999999999999},"max":{"1min":0.78600000000000003,"5min":0.95899999999999996,"15min":1.0069999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.64200000000000002,"15min":0.66600000000000004},"min":{"1min":0.42899999999999999,"5min":0.37,"15min":0.37},"max":{"1min":0.69999999999999996,"5min":0.91300000000000003,"15min":0.997},"last":0.747}]},{"osd":5,"last update":"Mon Oct 16 13:46:55 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.68200000000000005,"15min":0.70699999999999996},"min":{"1min":0.45800000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.83999999999999997,"5min":0.97499999999999998,"15min":1.0760000000000001},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.70499999999999996,"15min":0.72599999999999998},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.81399999999999995,"5min":1.0469999999999999,"15min":1.0660000000000001},"last":0.73199999999999998}]},{"osd":6,"last update":"Mon Oct 16 13:46:55 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.67500000000000004,"15min":0.68000000000000005},"min":{"1min":0.46200000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.75900000000000001,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.68700000000000006,"15min":0.70499999999999996},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.80400000000000005,"5min":1.03,"15min":1.083},"last":0.72299999999999998}]},{"osd":7,"last update":"Mon Oct 16 13:46:55 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.70899999999999996,"15min":0.73399999999999999},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.77300000000000002,"5min":1.0629999999999999,"15min":1.0660000000000001},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.72899999999999998,"15min":0.746},"min":{"1min":0.496,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.82099999999999995,"5min":1.042,"15min":1.0580000000000001},"last":0.87}]}]},{"osd":0,"up_from":13,"seq":55834575995,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48676,"kb_used_data":3608,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323164,"statfs":{"total":96636764160,"available":96586919936,"internally_reserved":0,"allocated":3694592,"data_stored":2188415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 13:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.40400000000000003,"15min":0.42099999999999999},"min":{"1min":0.29099999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.60699999999999998,"5min":0.65800000000000003,"15min":0.71399999999999997},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.42199999999999999,"15min":0.439},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.82199999999999995},"last":0.41999999999999998}]},{"osd":2,"last update":"Mon Oct 16 13:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.443,"15min":0.45400000000000001},"min":{"1min":0.28599999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.61399999999999999,"5min":0.65900000000000003,"15min":0.75700000000000001},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.46400000000000002,"15min":0.48499999999999999},"min":{"1min":0.29799999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.83299999999999996},"last":0.60499999999999998}]},{"osd":3,"last update":"Mon Oct 16 13:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.42699999999999999,"15min":0.45300000000000001},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.56499999999999995,"5min":0.63400000000000001,"15min":0.78600000000000003},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.50900000000000001,"15min":0.52000000000000002},"min":{"1min":0.317,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.66400000000000003,"5min":0.82999999999999996,"15min":0.86399999999999999},"last":0.57199999999999995}]},{"osd":4,"last update":"Mon Oct 16 13:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.66400000000000003,"15min":0.68400000000000005},"min":{"1min":0.42399999999999999,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.66500000000000004,"15min":0.68899999999999995},"min":{"1min":0.499,"5min":0.375,"15min":0.375},"max":{"1min":1.21,"5min":1.21,"15min":1.21},"last":0.78200000000000003}]},{"osd":5,"last update":"Mon Oct 16 13:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.66400000000000003,"15min":0.68999999999999995},"min":{"1min":0.41999999999999998,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.242,"5min":1.242,"15min":1.242},"last":0.751},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.70599999999999996,"15min":0.72299999999999998},"min":{"1min":0.497,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.998,"5min":1.081,"15min":1.081},"last":0.63200000000000001}]},{"osd":6,"last update":"Mon Oct 16 13:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.71199999999999997,"15min":0.72899999999999998},"min":{"1min":0.51900000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.96599999999999997,"5min":1.034,"15min":1.103},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.70399999999999996,"15min":0.72299999999999998},"min":{"1min":0.51000000000000001,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":0.79600000000000004}]},{"osd":7,"last update":"Mon Oct 16 13:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.71199999999999997,"15min":0.74199999999999999},"min":{"1min":0.48399999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.1919999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.72499999999999998,"15min":0.73499999999999999},"min":{"1min":0.51300000000000001,"5min":0.433,"15min":0.433},"max":{"1min":1.0469999999999999,"5min":1.087,"15min":1.087},"last":0.64700000000000002}]}]},{"osd":2,"up_from":13,"seq":55834575995,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52252,"kb_used_data":3024,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319588,"statfs":{"total":96636764160,"available":96583258112,"internally_reserved":0,"allocated":3096576,"data_stored":1597087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.41499999999999998,"15min":0.40899999999999997},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.623,"5min":0.71699999999999997,"15min":0.77500000000000002},"last":0.438},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.47699999999999998,"15min":0.47999999999999998},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.52800000000000002}]},{"osd":1,"last update":"Mon Oct 16 13:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.502,"15min":0.5},"min":{"1min":0.317,"5min":0.311,"15min":0.311},"max":{"1min":0.66800000000000004,"5min":0.73199999999999998,"15min":0.81399999999999995},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.48999999999999999,"15min":0.47699999999999998},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.27700000000000002},"max":{"1min":0.69999999999999996,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.58599999999999997}]},{"osd":3,"last update":"Mon Oct 16 13:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.443,"15min":0.44},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.59699999999999998,"5min":0.70099999999999996,"15min":0.72599999999999998},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.54300000000000004,"15min":0.54400000000000004},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.82699999999999996,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.74099999999999999}]},{"osd":4,"last update":"Mon Oct 16 13:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.72699999999999998,"15min":0.71799999999999997},"min":{"1min":0.51500000000000001,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.89000000000000001,"5min":1.3600000000000001,"15min":1.3600000000000001},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.72599999999999998,"15min":0.71899999999999997},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.82899999999999996,"5min":1.403,"15min":1.403},"last":0.91400000000000003}]},{"osd":5,"last update":"Mon Oct 16 13:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.68600000000000005,"15min":0.68899999999999995},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.81100000000000005,"5min":1.486,"15min":1.486},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.71199999999999997,"15min":0.69999999999999996},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.90200000000000002,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":0.879}]},{"osd":6,"last update":"Mon Oct 16 13:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.67600000000000005,"15min":0.66600000000000004},"min":{"1min":0.44500000000000001,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.80200000000000005,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.72599999999999998,"15min":0.72199999999999998},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.90600000000000003,"5min":1.4379999999999999,"15min":1.4379999999999999},"last":0.94299999999999995}]},{"osd":7,"last update":"Mon Oct 16 13:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.73899999999999999,"15min":0.72099999999999997},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.94099999999999995,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":0.97099999999999997},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.72999999999999998,"15min":0.73099999999999998},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.91500000000000004,"5min":1.377,"15min":1.377},"last":1.0429999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575994,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56428,"kb_used_data":3040,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315412,"statfs":{"total":96636764160,"available":96578981888,"internally_reserved":0,"allocated":3112960,"data_stored":1598685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.40300000000000002,"15min":0.40999999999999998},"min":{"1min":0.23499999999999999,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.57199999999999995,"5min":0.64200000000000002,"15min":0.72899999999999998},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.41499999999999998,"15min":0.42799999999999999},"min":{"1min":0.27300000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.60099999999999998,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.312}]},{"osd":1,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.42699999999999999,"15min":0.434},"min":{"1min":0.26700000000000002,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.55900000000000005,"5min":0.68200000000000005,"15min":0.76300000000000001},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.435,"5min":0.45900000000000002,"15min":0.47999999999999998},"min":{"1min":0.27700000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.65500000000000003,"5min":0.70699999999999996,"15min":0.77800000000000002},"last":0.441}]},{"osd":2,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.47999999999999998,"15min":0.49299999999999999},"min":{"1min":0.33800000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.621,"5min":1.167,"15min":1.167},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.501,"15min":0.52300000000000002},"min":{"1min":0.312,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.625,"5min":0.73299999999999998,"15min":0.78700000000000003},"last":0.49199999999999999}]},{"osd":4,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.65100000000000002,"15min":0.66500000000000004},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.91200000000000003,"5min":0.998,"15min":1.137},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.68899999999999995,"15min":0.69399999999999995},"min":{"1min":0.441,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.087,"5min":1.2110000000000001,"15min":1.2909999999999999},"last":0.72799999999999998}]},{"osd":5,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.67700000000000005,"15min":0.70499999999999996},"min":{"1min":0.45500000000000002,"5min":0.42899999999999999,"15min":0.374},"max":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.2729999999999999},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.69299999999999995,"15min":0.69999999999999996},"min":{"1min":0.45600000000000002,"5min":0.432,"15min":0.432},"max":{"1min":0.98799999999999999,"5min":1.0549999999999999,"15min":1.173},"last":0.68600000000000005}]},{"osd":6,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.72899999999999998,"15min":0.71899999999999997},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.41799999999999998},"max":{"1min":0.99099999999999999,"5min":5.0999999999999996,"15min":5.0999999999999996},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.70999999999999996,"15min":0.73399999999999999},"min":{"1min":0.49199999999999999,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.0029999999999999,"5min":1.0309999999999999,"15min":1.2290000000000001},"last":0.80000000000000004}]},{"osd":7,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.71899999999999997,"15min":0.73399999999999999},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.048,"5min":1.048,"15min":1.2569999999999999},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.73299999999999998,"15min":0.746},"min":{"1min":0.501,"5min":0.47799999999999998,"15min":0.43099999999999999},"max":{"1min":1.0369999999999999,"5min":1.262,"15min":1.262},"last":0.66800000000000004}]}]},{"osd":4,"up_from":13,"seq":55834575992,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52208,"kb_used_data":2980,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319632,"statfs":{"total":96636764160,"available":96583303168,"internally_reserved":0,"allocated":3051520,"data_stored":1575741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.60699999999999998,"15min":0.625},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.33800000000000002},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.623,"15min":0.63100000000000001},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.84099999999999997,"5min":0.878,"15min":0.89500000000000002},"last":0.41699999999999998}]},{"osd":1,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.63,"15min":0.65600000000000003},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.628,"15min":0.63700000000000001},"min":{"1min":0.41099999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.50700000000000001}]},{"osd":2,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.65200000000000002,"15min":0.65300000000000002},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.83999999999999997,"5min":0.89900000000000002,"15min":0.90600000000000003},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.65600000000000003,"15min":0.65800000000000003},"min":{"1min":0.39600000000000002,"5min":0.39400000000000002,"15min":0.38400000000000001},"max":{"1min":0.88200000000000001,"5min":0.89700000000000002,"15min":0.92600000000000005},"last":0.438}]},{"osd":3,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.63200000000000001,"15min":0.64500000000000002},"min":{"1min":0.40400000000000003,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.78100000000000003,"5min":0.86599999999999999,"15min":0.99199999999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.67400000000000004,"15min":0.68200000000000005},"min":{"1min":0.44800000000000001,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.79100000000000004,"5min":0.872,"15min":0.93899999999999995},"last":0.46300000000000002}]},{"osd":5,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55100000000000005,"15min":0.53000000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.57999999999999996,"15min":0.56899999999999995},"min":{"1min":0.36099999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.82199999999999995,"5min":0.82299999999999995,"15min":0.94899999999999995},"last":0.59799999999999998}]},{"osd":6,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.625,"15min":0.59799999999999998},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.33200000000000002},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92700000000000005},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.747,"5min":0.71399999999999997,"15min":0.69399999999999995},"min":{"1min":0.52600000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.129,"5min":1.129,"15min":1.129},"last":0.61599999999999999}]},{"osd":7,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.66000000000000003,"15min":0.63500000000000001},"min":{"1min":0.44400000000000001,"5min":0.41999999999999998,"15min":0.35799999999999998},"max":{"1min":0.97099999999999997,"5min":0.97499999999999998,"15min":0.98599999999999999},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.70799999999999996,"15min":0.68400000000000005},"min":{"1min":0.44,"5min":0.433,"15min":0.38200000000000001},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":1.026},"last":0.71599999999999997}]}]},{"osd":5,"up_from":13,"seq":55834575993,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52196,"kb_used_data":2968,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319644,"statfs":{"total":96636764160,"available":96583315456,"internally_reserved":0,"allocated":3039232,"data_stored":1569476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:46:24 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.61399999999999999,"15min":0.61499999999999999},"min":{"1min":0.314,"5min":0.314,"15min":0.26800000000000002},"max":{"1min":0.878,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63,"15min":0.64200000000000002},"min":{"1min":0.38700000000000001,"5min":0.35899999999999999,"15min":0.29699999999999999},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.42099999999999999}]},{"osd":1,"last update":"Mon Oct 16 13:46:24 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.66000000000000003,"15min":0.66200000000000003},"min":{"1min":0.44600000000000001,"5min":0.34300000000000003,"15min":0.29499999999999998},"max":{"1min":1,"5min":1,"15min":1},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.63800000000000001,"15min":0.64000000000000001},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.58899999999999997}]},{"osd":2,"last update":"Mon Oct 16 13:46:24 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.67000000000000004,"15min":0.65700000000000003},"min":{"1min":0.41699999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.91000000000000003,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.629,"5min":0.65700000000000003,"15min":0.64900000000000002},"min":{"1min":0.42599999999999999,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.83899999999999997,"5min":0.871,"15min":0.93500000000000005},"last":0.51000000000000001}]},{"osd":3,"last update":"Mon Oct 16 13:46:24 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.67400000000000004,"15min":0.66400000000000003},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.69899999999999995,"15min":0.69499999999999995},"min":{"1min":0.41399999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.4199999999999999,"5min":1.4199999999999999,"15min":1.4199999999999999},"last":0.54500000000000004}]},{"osd":4,"last update":"Mon Oct 16 13:46:24 2023","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.54400000000000004,"15min":0.53300000000000003},"min":{"1min":0.34100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.82499999999999996,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.59299999999999997,"15min":0.58899999999999997},"min":{"1min":0.34300000000000003,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.85399999999999998,"5min":0.89000000000000001,"15min":0.93400000000000005},"last":0.753}]},{"osd":6,"last update":"Mon Oct 16 13:46:24 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.58899999999999997,"15min":0.57999999999999996},"min":{"1min":0.38700000000000001,"5min":0.315,"15min":0.315},"max":{"1min":0.877,"5min":0.88900000000000001,"15min":0.93200000000000005},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.61899999999999999},"min":{"1min":0.372,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89600000000000002},"last":0.68200000000000005}]},{"osd":7,"last update":"Mon Oct 16 13:46:24 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.64800000000000002,"15min":0.63300000000000001},"min":{"1min":0.378,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.69699999999999995,"15min":0.68400000000000005},"min":{"1min":0.46200000000000002,"5min":0.375,"15min":0.375},"max":{"1min":0.94699999999999995,"5min":0.98299999999999998,"15min":1.0229999999999999},"last":0.72399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T13:47:28.593 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T13:47:28.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.749+0000 7f509b545700 1 Processor -- start 2023-10-16T13:47:28.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.750+0000 7f509b545700 1 -- start start 2023-10-16T13:47:28.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.750+0000 7f509b545700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f50940cdfa0 0x7f50940b2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:28.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.750+0000 7f509b545700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f50940d0bb0 0x7f50940b2980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:28.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.750+0000 7f509b545700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50940d0fc0 con 0x7f50940ce380 2023-10-16T13:47:28.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.750+0000 7f509b545700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50940d1210 con 0x7f50940cdfa0 2023-10-16T13:47:28.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.750+0000 7f509b545700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50940cbc30 con 0x7f50940d0bb0 2023-10-16T13:47:28.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.751+0000 7f50992e1700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f50940cdfa0 0x7f50940b2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:28.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.751+0000 7f50992e1700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f50940cdfa0 0x7f50940b2440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55488/0 (socket says 172.21.15.16:55488) 2023-10-16T13:47:28.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.751+0000 7f50992e1700 1 -- 172.21.15.16:0/4202890193 learned_addr learned my addr 172.21.15.16:0/4202890193 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:28.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.751+0000 7f5098ae0700 1 --1- 172.21.15.16:0/4202890193 >> v1:172.21.15.16:6789/0 conn(0x7f50940ce380 0x7f50940d04a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44762/0 (socket says 172.21.15.16:44762) 2023-10-16T13:47:28.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.751+0000 7f5099ae2700 1 --2- 172.21.15.16:0/4202890193 >> v2:172.21.15.49:3300/0 conn(0x7f50940d0bb0 0x7f50940b2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:28.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.751+0000 7f50992e1700 1 -- 172.21.15.16:0/4202890193 >> v2:172.21.15.49:3300/0 conn(0x7f50940d0bb0 msgr2=0x7f50940b2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:28.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.751+0000 7f50992e1700 1 --2- 172.21.15.16:0/4202890193 >> v2:172.21.15.49:3300/0 conn(0x7f50940d0bb0 0x7f50940b2980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:28.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.751+0000 7f50992e1700 1 -- 172.21.15.16:0/4202890193 >> v1:172.21.15.16:6789/0 conn(0x7f50940ce380 legacy=0x7f50940d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:28.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.751+0000 7f50992e1700 1 -- 172.21.15.16:0/4202890193 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50940cbf10 con 0x7f50940cdfa0 2023-10-16T13:47:28.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.751+0000 7f50992e1700 1 --2- 172.21.15.16:0/4202890193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50940cdfa0 0x7f50940b2440 secure :-1 s=READY pgs=2753 cs=0 l=1 rev1=1 crypto rx=0x7f50940cf870 tx=0x7f5084009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5e1f2b65286a613 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:28.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.751+0000 7f508bfff700 1 -- 172.21.15.16:0/4202890193 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f508400b870 con 0x7f50940cdfa0 2023-10-16T13:47:28.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.751+0000 7f508bfff700 1 -- 172.21.15.16:0/4202890193 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f508400b9d0 con 0x7f50940cdfa0 2023-10-16T13:47:28.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.751+0000 7f508bfff700 1 -- 172.21.15.16:0/4202890193 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f508400bd60 con 0x7f50940cdfa0 2023-10-16T13:47:28.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.752+0000 7f509b545700 1 -- 172.21.15.16:0/4202890193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50940cdfa0 msgr2=0x7f50940b2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:28.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.752+0000 7f509b545700 1 --2- 172.21.15.16:0/4202890193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50940cdfa0 0x7f50940b2440 secure :-1 s=READY pgs=2753 cs=0 l=1 rev1=1 crypto rx=0x7f50940cf870 tx=0x7f5084009bd0 comp rx=0 tx=0).stop 2023-10-16T13:47:28.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.752+0000 7f509b545700 1 -- 172.21.15.16:0/4202890193 shutdown_connections 2023-10-16T13:47:28.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.752+0000 7f509b545700 1 --2- 172.21.15.16:0/4202890193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50940cdfa0 0x7f50940b2440 unknown :-1 s=CLOSED pgs=2753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:28.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.752+0000 7f509b545700 1 --2- 172.21.15.16:0/4202890193 >> v2:172.21.15.49:3300/0 conn(0x7f50940d0bb0 0x7f50940b2980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:28.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.752+0000 7f509b545700 1 -- 172.21.15.16:0/4202890193 >> 172.21.15.16:0/4202890193 conn(0x7f50940d14b0 msgr2=0x7f50940d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:28.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.752+0000 7f509b545700 1 -- 172.21.15.16:0/4202890193 shutdown_connections 2023-10-16T13:47:28.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.752+0000 7f509b545700 1 -- 172.21.15.16:0/4202890193 wait complete. 2023-10-16T13:47:28.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.753+0000 7f509b545700 1 Processor -- start 2023-10-16T13:47:28.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.753+0000 7f509b545700 1 -- start start 2023-10-16T13:47:28.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.754+0000 7f509b545700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f50940cdfa0 0x7f50940c7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:28.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.754+0000 7f509b545700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50940ce380 0x7f50940c8580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:28.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.754+0000 7f509b545700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f50940d0bb0 0x7f50940c9c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:28.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.754+0000 7f509b545700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5094157400 con 0x7f50940ce380 2023-10-16T13:47:28.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.754+0000 7f509b545700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50940ca1d0 con 0x7f50940d0bb0 2023-10-16T13:47:28.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.754+0000 7f509b545700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50940c8bf0 con 0x7f50940cdfa0 2023-10-16T13:47:28.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.754+0000 7f5098ae0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50940ce380 0x7f50940c8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:28.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.754+0000 7f5098ae0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50940ce380 0x7f50940c8580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55496/0 (socket says 172.21.15.16:55496) 2023-10-16T13:47:28.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.754+0000 7f5098ae0700 1 -- 172.21.15.16:0/879800638 learned_addr learned my addr 172.21.15.16:0/879800638 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:28.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.754+0000 7f5099ae2700 1 --2- 172.21.15.16:0/879800638 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f50940d0bb0 0x7f50940c9c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:28.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.754+0000 7f50992e1700 1 --2- 172.21.15.16:0/879800638 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f50940cdfa0 0x7f50940c7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:28.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.754+0000 7f5098ae0700 1 -- 172.21.15.16:0/879800638 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f50940cdfa0 msgr2=0x7f50940c7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:28.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.754+0000 7f5098ae0700 1 --2- 172.21.15.16:0/879800638 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f50940cdfa0 0x7f50940c7ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:28.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.754+0000 7f5098ae0700 1 -- 172.21.15.16:0/879800638 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f50940d0bb0 msgr2=0x7f50940c9c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:28.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.754+0000 7f5098ae0700 1 --2- 172.21.15.16:0/879800638 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f50940d0bb0 0x7f50940c9c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:28.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.754+0000 7f5098ae0700 1 -- 172.21.15.16:0/879800638 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5084009550 con 0x7f50940ce380 2023-10-16T13:47:28.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.755+0000 7f5098ae0700 1 --2- 172.21.15.16:0/879800638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50940ce380 0x7f50940c8580 secure :-1 s=READY pgs=2754 cs=0 l=1 rev1=1 crypto rx=0x7f509000c520 tx=0x7f509000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:28.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.755+0000 7f5089ffb700 1 -- 172.21.15.16:0/879800638 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5090005cb0 con 0x7f50940ce380 2023-10-16T13:47:28.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.755+0000 7f5089ffb700 1 -- 172.21.15.16:0/879800638 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5090005e10 con 0x7f50940ce380 2023-10-16T13:47:28.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.755+0000 7f509b545700 1 -- 172.21.15.16:0/879800638 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50940c8ed0 con 0x7f50940ce380 2023-10-16T13:47:28.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.755+0000 7f5089ffb700 1 -- 172.21.15.16:0/879800638 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50900072d0 con 0x7f50940ce380 2023-10-16T13:47:28.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.755+0000 7f509b545700 1 -- 172.21.15.16:0/879800638 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50940c9490 con 0x7f50940ce380 2023-10-16T13:47:28.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.756+0000 7f509b545700 1 -- 172.21.15.16:0/879800638 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f509404ebb0 con 0x7f50940ce380 2023-10-16T13:47:28.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.756+0000 7f5089ffb700 1 -- 172.21.15.16:0/879800638 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 148) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f5090007430 con 0x7f50940ce380 2023-10-16T13:47:28.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.757+0000 7f5089ffb700 1 --2- 172.21.15.16:0/879800638 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f507806b140 0x7f507806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:28.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.757+0000 7f5089ffb700 1 -- 172.21.15.16:0/879800638 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(245..245 src has 1..245) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f5090088da0 con 0x7f50940ce380 2023-10-16T13:47:28.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.757+0000 7f50992e1700 1 --2- 172.21.15.16:0/879800638 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f507806b140 0x7f507806d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:28.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.757+0000 7f50992e1700 1 --2- 172.21.15.16:0/879800638 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f507806b140 0x7f507806d600 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5084000c00 tx=0x7f5084008030 comp rx=0 tx=0).ready entity=mgr.11886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:28.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.762+0000 7f5089ffb700 1 -- 172.21.15.16:0/879800638 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5090053a70 con 0x7f50940ce380 2023-10-16T13:47:28.964 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.962+0000 7f509b545700 1 -- 172.21.15.16:0/879800638 --> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f50940cc4d0 con 0x7f507806b140 2023-10-16T13:47:28.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.970+0000 7f5089ffb700 1 -- 172.21.15.16:0/879800638 <== mgr.11886 v2:172.21.15.16:6832/1736104576 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+285751 (secure 0 0 0) 0x7f508400bbc0 con 0x7f507806b140 2023-10-16T13:47:28.976 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:47:28.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.973+0000 7f509b545700 1 -- 172.21.15.16:0/879800638 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f507806b140 msgr2=0x7f507806d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:28.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.973+0000 7f509b545700 1 --2- 172.21.15.16:0/879800638 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f507806b140 0x7f507806d600 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5084000c00 tx=0x7f5084008030 comp rx=0 tx=0).stop 2023-10-16T13:47:28.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.973+0000 7f509b545700 1 -- 172.21.15.16:0/879800638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50940ce380 msgr2=0x7f50940c8580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:28.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.973+0000 7f509b545700 1 --2- 172.21.15.16:0/879800638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50940ce380 0x7f50940c8580 secure :-1 s=READY pgs=2754 cs=0 l=1 rev1=1 crypto rx=0x7f509000c520 tx=0x7f509000c830 comp rx=0 tx=0).stop 2023-10-16T13:47:28.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.973+0000 7f509b545700 1 -- 172.21.15.16:0/879800638 shutdown_connections 2023-10-16T13:47:28.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.973+0000 7f509b545700 1 --2- 172.21.15.16:0/879800638 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f507806b140 0x7f507806d600 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:28.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.973+0000 7f509b545700 1 --2- 172.21.15.16:0/879800638 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f50940d0bb0 0x7f50940c9c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:28.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.973+0000 7f509b545700 1 --2- 172.21.15.16:0/879800638 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f50940cdfa0 0x7f50940c7ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:28.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.973+0000 7f509b545700 1 --2- 172.21.15.16:0/879800638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50940ce380 0x7f50940c8580 unknown :-1 s=CLOSED pgs=2754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:28.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.973+0000 7f509b545700 1 -- 172.21.15.16:0/879800638 >> 172.21.15.16:0/879800638 conn(0x7f50940d14b0 msgr2=0x7f509406eba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:28.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.973+0000 7f509b545700 1 -- 172.21.15.16:0/879800638 shutdown_connections 2023-10-16T13:47:28.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:28.973+0000 7f509b545700 1 -- 172.21.15.16:0/879800638 wait complete. 2023-10-16T13:47:28.981 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-10-16T13:47:28.988 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":30,"stamp":"2023-10-16T13:47:27.821538+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":656,"num_write_kb":2422,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":401,"ondisk_log_size":401,"up":150,"acting":150,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":105,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":431520,"kb_used_data":25208,"kb_used_omap":12,"kb_used_meta":406195,"kb_avail":754543200,"statfs":{"total":773094113280,"available":772652236800,"internally_reserved":0,"allocated":25812992,"data_stored":13882230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12740,"internal_metadata":415944252},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":37,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.416850"},"pg_stats":[{"pgid":"32.1f","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.418005+0000","last_change":"2023-10-16T13:47:24.429332+0000","last_active":"2023-10-16T13:47:25.418005+0000","last_peered":"2023-10-16T13:47:25.418005+0000","last_clean":"2023-10-16T13:47:25.418005+0000","last_became_active":"2023-10-16T13:47:24.428151+0000","last_became_peered":"2023-10-16T13:47:24.428151+0000","last_unstale":"2023-10-16T13:47:25.418005+0000","last_undegraded":"2023-10-16T13:47:25.418005+0000","last_fullsized":"2023-10-16T13:47:25.418005+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:51:49.567279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.1e","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.416620+0000","last_change":"2023-10-16T13:47:24.439610+0000","last_active":"2023-10-16T13:47:25.416620+0000","last_peered":"2023-10-16T13:47:25.416620+0000","last_clean":"2023-10-16T13:47:25.416620+0000","last_became_active":"2023-10-16T13:47:24.439212+0000","last_became_peered":"2023-10-16T13:47:24.439212+0000","last_unstale":"2023-10-16T13:47:25.416620+0000","last_undegraded":"2023-10-16T13:47:25.416620+0000","last_fullsized":"2023-10-16T13:47:25.416620+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:47:24.106272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1d","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100591+0000","last_change":"2023-10-16T13:47:24.432216+0000","last_active":"2023-10-16T13:47:26.100591+0000","last_peered":"2023-10-16T13:47:26.100591+0000","last_clean":"2023-10-16T13:47:26.100591+0000","last_became_active":"2023-10-16T13:47:24.431937+0000","last_became_peered":"2023-10-16T13:47:24.431937+0000","last_unstale":"2023-10-16T13:47:26.100591+0000","last_undegraded":"2023-10-16T13:47:26.100591+0000","last_fullsized":"2023-10-16T13:47:26.100591+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:23:10.580869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1c","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100430+0000","last_change":"2023-10-16T13:47:24.440252+0000","last_active":"2023-10-16T13:47:26.100430+0000","last_peered":"2023-10-16T13:47:26.100430+0000","last_clean":"2023-10-16T13:47:26.100430+0000","last_became_active":"2023-10-16T13:47:24.439982+0000","last_became_peered":"2023-10-16T13:47:24.439982+0000","last_unstale":"2023-10-16T13:47:26.100430+0000","last_undegraded":"2023-10-16T13:47:26.100430+0000","last_fullsized":"2023-10-16T13:47:26.100430+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:54:02.930019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1b","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.101595+0000","last_change":"2023-10-16T13:47:24.440049+0000","last_active":"2023-10-16T13:47:26.101595+0000","last_peered":"2023-10-16T13:47:26.101595+0000","last_clean":"2023-10-16T13:47:26.101595+0000","last_became_active":"2023-10-16T13:47:24.439750+0000","last_became_peered":"2023-10-16T13:47:24.439750+0000","last_unstale":"2023-10-16T13:47:26.101595+0000","last_undegraded":"2023-10-16T13:47:26.101595+0000","last_fullsized":"2023-10-16T13:47:26.101595+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:33:40.094175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1a","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.417711+0000","last_change":"2023-10-16T13:47:24.433661+0000","last_active":"2023-10-16T13:47:25.417711+0000","last_peered":"2023-10-16T13:47:25.417711+0000","last_clean":"2023-10-16T13:47:25.417711+0000","last_became_active":"2023-10-16T13:47:24.433417+0000","last_became_peered":"2023-10-16T13:47:24.433417+0000","last_unstale":"2023-10-16T13:47:25.417711+0000","last_undegraded":"2023-10-16T13:47:25.417711+0000","last_fullsized":"2023-10-16T13:47:25.417711+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:16:02.391378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T13:47:23.404948+0000","last_change":"2023-10-16T13:47:23.404948+0000","last_active":"2023-10-16T13:47:23.404948+0000","last_peered":"2023-10-16T13:47:23.404948+0000","last_clean":"2023-10-16T13:47:23.404948+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:23.404948+0000","last_undegraded":"2023-10-16T13:47:23.404948+0000","last_fullsized":"2023-10-16T13:47:23.404948+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.101429+0000","last_change":"2023-10-16T13:47:24.426442+0000","last_active":"2023-10-16T13:47:26.101429+0000","last_peered":"2023-10-16T13:47:26.101429+0000","last_clean":"2023-10-16T13:47:26.101429+0000","last_became_active":"2023-10-16T13:47:24.425929+0000","last_became_peered":"2023-10-16T13:47:24.425929+0000","last_unstale":"2023-10-16T13:47:26.101429+0000","last_undegraded":"2023-10-16T13:47:26.101429+0000","last_fullsized":"2023-10-16T13:47:26.101429+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:05:13.565304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.418008+0000","last_change":"2023-10-16T13:47:24.426459+0000","last_active":"2023-10-16T13:47:25.418008+0000","last_peered":"2023-10-16T13:47:25.418008+0000","last_clean":"2023-10-16T13:47:25.418008+0000","last_became_active":"2023-10-16T13:47:24.426169+0000","last_became_peered":"2023-10-16T13:47:24.426169+0000","last_unstale":"2023-10-16T13:47:25.418008+0000","last_undegraded":"2023-10-16T13:47:25.418008+0000","last_fullsized":"2023-10-16T13:47:25.418008+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:57:42.170954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100664+0000","last_change":"2023-10-16T13:47:24.427333+0000","last_active":"2023-10-16T13:47:26.100664+0000","last_peered":"2023-10-16T13:47:26.100664+0000","last_clean":"2023-10-16T13:47:26.100664+0000","last_became_active":"2023-10-16T13:47:24.427034+0000","last_became_peered":"2023-10-16T13:47:24.427034+0000","last_unstale":"2023-10-16T13:47:26.100664+0000","last_undegraded":"2023-10-16T13:47:26.100664+0000","last_fullsized":"2023-10-16T13:47:26.100664+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:27:20.181386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.15","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100502+0000","last_change":"2023-10-16T13:47:24.425903+0000","last_active":"2023-10-16T13:47:26.100502+0000","last_peered":"2023-10-16T13:47:26.100502+0000","last_clean":"2023-10-16T13:47:26.100502+0000","last_became_active":"2023-10-16T13:47:24.425593+0000","last_became_peered":"2023-10-16T13:47:24.425593+0000","last_unstale":"2023-10-16T13:47:26.100502+0000","last_undegraded":"2023-10-16T13:47:26.100502+0000","last_fullsized":"2023-10-16T13:47:26.100502+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:46:35.236231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.14","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.417564+0000","last_change":"2023-10-16T13:47:24.438927+0000","last_active":"2023-10-16T13:47:25.417564+0000","last_peered":"2023-10-16T13:47:25.417564+0000","last_clean":"2023-10-16T13:47:25.417564+0000","last_became_active":"2023-10-16T13:47:24.438389+0000","last_became_peered":"2023-10-16T13:47:24.438389+0000","last_unstale":"2023-10-16T13:47:25.417564+0000","last_undegraded":"2023-10-16T13:47:25.417564+0000","last_fullsized":"2023-10-16T13:47:25.417564+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:44:00.360485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T13:47:23.404948+0000","last_change":"2023-10-16T13:47:23.404948+0000","last_active":"2023-10-16T13:47:23.404948+0000","last_peered":"2023-10-16T13:47:23.404948+0000","last_clean":"2023-10-16T13:47:23.404948+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:23.404948+0000","last_undegraded":"2023-10-16T13:47:23.404948+0000","last_fullsized":"2023-10-16T13:47:23.404948+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.099722+0000","last_change":"2023-10-16T13:47:24.437693+0000","last_active":"2023-10-16T13:47:26.099722+0000","last_peered":"2023-10-16T13:47:26.099722+0000","last_clean":"2023-10-16T13:47:26.099722+0000","last_became_active":"2023-10-16T13:47:24.437434+0000","last_became_peered":"2023-10-16T13:47:24.437434+0000","last_unstale":"2023-10-16T13:47:26.099722+0000","last_undegraded":"2023-10-16T13:47:26.099722+0000","last_fullsized":"2023-10-16T13:47:26.099722+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:47:56.221847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T13:47:23.404948+0000","last_change":"2023-10-16T13:47:23.404948+0000","last_active":"2023-10-16T13:47:23.404948+0000","last_peered":"2023-10-16T13:47:23.404948+0000","last_clean":"2023-10-16T13:47:23.404948+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:23.404948+0000","last_undegraded":"2023-10-16T13:47:23.404948+0000","last_fullsized":"2023-10-16T13:47:23.404948+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.10","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.099717+0000","last_change":"2023-10-16T13:47:24.441060+0000","last_active":"2023-10-16T13:47:26.099717+0000","last_peered":"2023-10-16T13:47:26.099717+0000","last_clean":"2023-10-16T13:47:26.099717+0000","last_became_active":"2023-10-16T13:47:24.440745+0000","last_became_peered":"2023-10-16T13:47:24.440745+0000","last_unstale":"2023-10-16T13:47:26.099717+0000","last_undegraded":"2023-10-16T13:47:26.099717+0000","last_fullsized":"2023-10-16T13:47:26.099717+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:48:26.805075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T13:47:23.404948+0000","last_change":"2023-10-16T13:47:23.404948+0000","last_active":"2023-10-16T13:47:23.404948+0000","last_peered":"2023-10-16T13:47:23.404948+0000","last_clean":"2023-10-16T13:47:23.404948+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:23.404948+0000","last_undegraded":"2023-10-16T13:47:23.404948+0000","last_fullsized":"2023-10-16T13:47:23.404948+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.e","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100703+0000","last_change":"2023-10-16T13:47:24.433506+0000","last_active":"2023-10-16T13:47:26.100703+0000","last_peered":"2023-10-16T13:47:26.100703+0000","last_clean":"2023-10-16T13:47:26.100703+0000","last_became_active":"2023-10-16T13:47:24.433218+0000","last_became_peered":"2023-10-16T13:47:24.433218+0000","last_unstale":"2023-10-16T13:47:26.100703+0000","last_undegraded":"2023-10-16T13:47:26.100703+0000","last_fullsized":"2023-10-16T13:47:26.100703+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:04:28.651883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T13:47:23.404948+0000","last_change":"2023-10-16T13:47:23.404948+0000","last_active":"2023-10-16T13:47:23.404948+0000","last_peered":"2023-10-16T13:47:23.404948+0000","last_clean":"2023-10-16T13:47:23.404948+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:23.404948+0000","last_undegraded":"2023-10-16T13:47:23.404948+0000","last_fullsized":"2023-10-16T13:47:23.404948+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.416732+0000","last_change":"2023-10-16T13:47:24.435000+0000","last_active":"2023-10-16T13:47:25.416732+0000","last_peered":"2023-10-16T13:47:25.416732+0000","last_clean":"2023-10-16T13:47:25.416732+0000","last_became_active":"2023-10-16T13:47:24.434568+0000","last_became_peered":"2023-10-16T13:47:24.434568+0000","last_unstale":"2023-10-16T13:47:25.416732+0000","last_undegraded":"2023-10-16T13:47:25.416732+0000","last_fullsized":"2023-10-16T13:47:25.416732+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:31:46.967495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T13:47:23.404948+0000","last_change":"2023-10-16T13:47:23.404948+0000","last_active":"2023-10-16T13:47:23.404948+0000","last_peered":"2023-10-16T13:47:23.404948+0000","last_clean":"2023-10-16T13:47:23.404948+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:23.404948+0000","last_undegraded":"2023-10-16T13:47:23.404948+0000","last_fullsized":"2023-10-16T13:47:23.404948+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T13:47:23.404948+0000","last_change":"2023-10-16T13:47:23.404948+0000","last_active":"2023-10-16T13:47:23.404948+0000","last_peered":"2023-10-16T13:47:23.404948+0000","last_clean":"2023-10-16T13:47:23.404948+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:23.404948+0000","last_undegraded":"2023-10-16T13:47:23.404948+0000","last_fullsized":"2023-10-16T13:47:23.404948+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T13:47:23.404948+0000","last_change":"2023-10-16T13:47:23.404948+0000","last_active":"2023-10-16T13:47:23.404948+0000","last_peered":"2023-10-16T13:47:23.404948+0000","last_clean":"2023-10-16T13:47:23.404948+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:23.404948+0000","last_undegraded":"2023-10-16T13:47:23.404948+0000","last_fullsized":"2023-10-16T13:47:23.404948+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T13:47:23.404948+0000","last_change":"2023-10-16T13:47:23.404948+0000","last_active":"2023-10-16T13:47:23.404948+0000","last_peered":"2023-10-16T13:47:23.404948+0000","last_clean":"2023-10-16T13:47:23.404948+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:23.404948+0000","last_undegraded":"2023-10-16T13:47:23.404948+0000","last_fullsized":"2023-10-16T13:47:23.404948+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.7","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.416641+0000","last_change":"2023-10-16T13:47:24.433031+0000","last_active":"2023-10-16T13:47:25.416641+0000","last_peered":"2023-10-16T13:47:25.416641+0000","last_clean":"2023-10-16T13:47:25.416641+0000","last_became_active":"2023-10-16T13:47:24.432700+0000","last_became_peered":"2023-10-16T13:47:24.432700+0000","last_unstale":"2023-10-16T13:47:25.416641+0000","last_undegraded":"2023-10-16T13:47:25.416641+0000","last_fullsized":"2023-10-16T13:47:25.416641+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:17:10.080681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T13:47:23.404948+0000","last_change":"2023-10-16T13:47:23.404948+0000","last_active":"2023-10-16T13:47:23.404948+0000","last_peered":"2023-10-16T13:47:23.404948+0000","last_clean":"2023-10-16T13:47:23.404948+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:23.404948+0000","last_undegraded":"2023-10-16T13:47:23.404948+0000","last_fullsized":"2023-10-16T13:47:23.404948+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.101557+0000","last_change":"2023-10-16T13:47:24.428801+0000","last_active":"2023-10-16T13:47:26.101557+0000","last_peered":"2023-10-16T13:47:26.101557+0000","last_clean":"2023-10-16T13:47:26.101557+0000","last_became_active":"2023-10-16T13:47:24.428511+0000","last_became_peered":"2023-10-16T13:47:24.428511+0000","last_unstale":"2023-10-16T13:47:26.101557+0000","last_undegraded":"2023-10-16T13:47:26.101557+0000","last_fullsized":"2023-10-16T13:47:26.101557+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:27:55.268379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.417855+0000","last_change":"2023-10-16T13:47:24.429772+0000","last_active":"2023-10-16T13:47:25.417855+0000","last_peered":"2023-10-16T13:47:25.417855+0000","last_clean":"2023-10-16T13:47:25.417855+0000","last_became_active":"2023-10-16T13:47:24.429496+0000","last_became_peered":"2023-10-16T13:47:24.429496+0000","last_unstale":"2023-10-16T13:47:25.417855+0000","last_undegraded":"2023-10-16T13:47:25.417855+0000","last_fullsized":"2023-10-16T13:47:25.417855+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:09:48.751438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.099824+0000","last_change":"2023-10-16T13:47:24.437263+0000","last_active":"2023-10-16T13:47:26.099824+0000","last_peered":"2023-10-16T13:47:26.099824+0000","last_clean":"2023-10-16T13:47:26.099824+0000","last_became_active":"2023-10-16T13:47:24.437023+0000","last_became_peered":"2023-10-16T13:47:24.437023+0000","last_unstale":"2023-10-16T13:47:26.099824+0000","last_undegraded":"2023-10-16T13:47:26.099824+0000","last_fullsized":"2023-10-16T13:47:26.099824+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:59:15.061907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.099775+0000","last_change":"2023-10-16T13:47:24.419847+0000","last_active":"2023-10-16T13:47:26.099775+0000","last_peered":"2023-10-16T13:47:26.099775+0000","last_clean":"2023-10-16T13:47:26.099775+0000","last_became_active":"2023-10-16T13:47:24.419520+0000","last_became_peered":"2023-10-16T13:47:24.419520+0000","last_unstale":"2023-10-16T13:47:26.099775+0000","last_undegraded":"2023-10-16T13:47:26.099775+0000","last_fullsized":"2023-10-16T13:47:26.099775+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:11:44.932011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.416267+0000","last_change":"2023-10-16T13:47:24.435926+0000","last_active":"2023-10-16T13:47:25.416267+0000","last_peered":"2023-10-16T13:47:25.416267+0000","last_clean":"2023-10-16T13:47:25.416267+0000","last_became_active":"2023-10-16T13:47:24.434753+0000","last_became_peered":"2023-10-16T13:47:24.434753+0000","last_unstale":"2023-10-16T13:47:25.416267+0000","last_undegraded":"2023-10-16T13:47:25.416267+0000","last_fullsized":"2023-10-16T13:47:25.416267+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:13:51.332934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.417066+0000","last_change":"2023-10-16T13:47:24.423908+0000","last_active":"2023-10-16T13:47:25.417066+0000","last_peered":"2023-10-16T13:47:25.417066+0000","last_clean":"2023-10-16T13:47:25.417066+0000","last_became_active":"2023-10-16T13:47:24.423530+0000","last_became_peered":"2023-10-16T13:47:24.423530+0000","last_unstale":"2023-10-16T13:47:25.417066+0000","last_undegraded":"2023-10-16T13:47:25.417066+0000","last_fullsized":"2023-10-16T13:47:25.417066+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:50:50.396650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.20","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.416162+0000","last_change":"2023-10-16T13:47:24.436480+0000","last_active":"2023-10-16T13:47:25.416162+0000","last_peered":"2023-10-16T13:47:25.416162+0000","last_clean":"2023-10-16T13:47:25.416162+0000","last_became_active":"2023-10-16T13:47:24.435309+0000","last_became_peered":"2023-10-16T13:47:24.435309+0000","last_unstale":"2023-10-16T13:47:25.416162+0000","last_undegraded":"2023-10-16T13:47:25.416162+0000","last_fullsized":"2023-10-16T13:47:25.416162+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:28:08.947506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1f","version":"244'4","reported_seq":16,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.452436+0000","last_change":"2023-10-16T13:47:23.425349+0000","last_active":"2023-10-16T13:47:24.452436+0000","last_peered":"2023-10-16T13:47:24.452436+0000","last_clean":"2023-10-16T13:47:24.452436+0000","last_became_active":"2023-10-16T13:47:23.425029+0000","last_became_peered":"2023-10-16T13:47:23.425029+0000","last_unstale":"2023-10-16T13:47:24.452436+0000","last_undegraded":"2023-10-16T13:47:24.452436+0000","last_fullsized":"2023-10-16T13:47:24.452436+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:57:35.834032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T13:47:23.404948+0000","last_change":"2023-10-16T13:47:23.404948+0000","last_active":"2023-10-16T13:47:23.404948+0000","last_peered":"2023-10-16T13:47:23.404948+0000","last_clean":"2023-10-16T13:47:23.404948+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:23.404948+0000","last_undegraded":"2023-10-16T13:47:23.404948+0000","last_fullsized":"2023-10-16T13:47:23.404948+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.1d","version":"0'0","reported_seq":1,"reported_epoch":242,"state":"creating+peering","last_fresh":"2023-10-16T13:47:22.414037+0000","last_change":"2023-10-16T13:47:22.414037+0000","last_active":"2023-10-16T13:47:22.400586+0000","last_peered":"2023-10-16T13:47:22.400586+0000","last_clean":"2023-10-16T13:47:22.400586+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:22.414037+0000","last_undegraded":"2023-10-16T13:47:22.414037+0000","last_fullsized":"2023-10-16T13:47:22.414037+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.23","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.417464+0000","last_change":"2023-10-16T13:47:24.432436+0000","last_active":"2023-10-16T13:47:25.417464+0000","last_peered":"2023-10-16T13:47:25.417464+0000","last_clean":"2023-10-16T13:47:25.417464+0000","last_became_active":"2023-10-16T13:47:24.428465+0000","last_became_peered":"2023-10-16T13:47:24.428465+0000","last_unstale":"2023-10-16T13:47:25.417464+0000","last_undegraded":"2023-10-16T13:47:25.417464+0000","last_fullsized":"2023-10-16T13:47:25.417464+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:16:50.560075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1c","version":"244'2","reported_seq":14,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.447290+0000","last_change":"2023-10-16T13:47:23.427686+0000","last_active":"2023-10-16T13:47:24.447290+0000","last_peered":"2023-10-16T13:47:24.447290+0000","last_clean":"2023-10-16T13:47:24.447290+0000","last_became_active":"2023-10-16T13:47:23.427013+0000","last_became_peered":"2023-10-16T13:47:23.427013+0000","last_unstale":"2023-10-16T13:47:24.447290+0000","last_undegraded":"2023-10-16T13:47:24.447290+0000","last_fullsized":"2023-10-16T13:47:24.447290+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:37:17.532164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.24","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100065+0000","last_change":"2023-10-16T13:47:24.438644+0000","last_active":"2023-10-16T13:47:26.100065+0000","last_peered":"2023-10-16T13:47:26.100065+0000","last_clean":"2023-10-16T13:47:26.100065+0000","last_became_active":"2023-10-16T13:47:24.438344+0000","last_became_peered":"2023-10-16T13:47:24.438344+0000","last_unstale":"2023-10-16T13:47:26.100065+0000","last_undegraded":"2023-10-16T13:47:26.100065+0000","last_fullsized":"2023-10-16T13:47:26.100065+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:21:19.269920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.1b","version":"0'0","reported_seq":1,"reported_epoch":242,"state":"creating+peering","last_fresh":"2023-10-16T13:47:22.413990+0000","last_change":"2023-10-16T13:47:22.413990+0000","last_active":"2023-10-16T13:47:22.400586+0000","last_peered":"2023-10-16T13:47:22.400586+0000","last_clean":"2023-10-16T13:47:22.400586+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:22.413990+0000","last_undegraded":"2023-10-16T13:47:22.413990+0000","last_fullsized":"2023-10-16T13:47:22.413990+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T13:47:23.404948+0000","last_change":"2023-10-16T13:47:23.404948+0000","last_active":"2023-10-16T13:47:23.404948+0000","last_peered":"2023-10-16T13:47:23.404948+0000","last_clean":"2023-10-16T13:47:23.404948+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:23.404948+0000","last_undegraded":"2023-10-16T13:47:23.404948+0000","last_fullsized":"2023-10-16T13:47:23.404948+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.1a","version":"244'1","reported_seq":13,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.441041+0000","last_change":"2023-10-16T13:47:23.425695+0000","last_active":"2023-10-16T13:47:24.441041+0000","last_peered":"2023-10-16T13:47:24.441041+0000","last_clean":"2023-10-16T13:47:24.441041+0000","last_became_active":"2023-10-16T13:47:23.425407+0000","last_became_peered":"2023-10-16T13:47:23.425407+0000","last_unstale":"2023-10-16T13:47:24.441041+0000","last_undegraded":"2023-10-16T13:47:24.441041+0000","last_fullsized":"2023-10-16T13:47:24.441041+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:19:34.313200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.26","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100538+0000","last_change":"2023-10-16T13:47:24.439616+0000","last_active":"2023-10-16T13:47:26.100538+0000","last_peered":"2023-10-16T13:47:26.100538+0000","last_clean":"2023-10-16T13:47:26.100538+0000","last_became_active":"2023-10-16T13:47:24.435678+0000","last_became_peered":"2023-10-16T13:47:24.435678+0000","last_unstale":"2023-10-16T13:47:26.100538+0000","last_undegraded":"2023-10-16T13:47:26.100538+0000","last_fullsized":"2023-10-16T13:47:26.100538+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:39:16.296666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.418484+0000","last_change":"2023-10-16T13:47:23.431681+0000","last_active":"2023-10-16T13:47:24.418484+0000","last_peered":"2023-10-16T13:47:24.418484+0000","last_clean":"2023-10-16T13:47:24.418484+0000","last_became_active":"2023-10-16T13:47:23.431317+0000","last_became_peered":"2023-10-16T13:47:23.431317+0000","last_unstale":"2023-10-16T13:47:24.418484+0000","last_undegraded":"2023-10-16T13:47:24.418484+0000","last_fullsized":"2023-10-16T13:47:24.418484+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:20:27.852907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.27","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.101829+0000","last_change":"2023-10-16T13:47:24.423771+0000","last_active":"2023-10-16T13:47:26.101829+0000","last_peered":"2023-10-16T13:47:26.101829+0000","last_clean":"2023-10-16T13:47:26.101829+0000","last_became_active":"2023-10-16T13:47:24.423459+0000","last_became_peered":"2023-10-16T13:47:24.423459+0000","last_unstale":"2023-10-16T13:47:26.101829+0000","last_undegraded":"2023-10-16T13:47:26.101829+0000","last_fullsized":"2023-10-16T13:47:26.101829+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:27:22.393558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.421767+0000","last_change":"2023-10-16T13:47:23.433310+0000","last_active":"2023-10-16T13:47:24.421767+0000","last_peered":"2023-10-16T13:47:24.421767+0000","last_clean":"2023-10-16T13:47:24.421767+0000","last_became_active":"2023-10-16T13:47:23.433018+0000","last_became_peered":"2023-10-16T13:47:23.433018+0000","last_unstale":"2023-10-16T13:47:24.421767+0000","last_undegraded":"2023-10-16T13:47:24.421767+0000","last_fullsized":"2023-10-16T13:47:24.421767+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:49:45.992161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T13:47:23.404948+0000","last_change":"2023-10-16T13:47:23.404948+0000","last_active":"2023-10-16T13:47:23.404948+0000","last_peered":"2023-10-16T13:47:23.404948+0000","last_clean":"2023-10-16T13:47:23.404948+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:23.404948+0000","last_undegraded":"2023-10-16T13:47:23.404948+0000","last_fullsized":"2023-10-16T13:47:23.404948+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.415527+0000","last_change":"2023-10-16T13:47:23.428721+0000","last_active":"2023-10-16T13:47:24.415527+0000","last_peered":"2023-10-16T13:47:24.415527+0000","last_clean":"2023-10-16T13:47:24.415527+0000","last_became_active":"2023-10-16T13:47:23.428349+0000","last_became_peered":"2023-10-16T13:47:23.428349+0000","last_unstale":"2023-10-16T13:47:24.415527+0000","last_undegraded":"2023-10-16T13:47:24.415527+0000","last_fullsized":"2023-10-16T13:47:24.415527+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:59:11.116638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T13:47:23.404948+0000","last_change":"2023-10-16T13:47:23.404948+0000","last_active":"2023-10-16T13:47:23.404948+0000","last_peered":"2023-10-16T13:47:23.404948+0000","last_clean":"2023-10-16T13:47:23.404948+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:23.404948+0000","last_undegraded":"2023-10-16T13:47:23.404948+0000","last_fullsized":"2023-10-16T13:47:23.404948+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.421423+0000","last_change":"2023-10-16T13:47:23.433545+0000","last_active":"2023-10-16T13:47:24.421423+0000","last_peered":"2023-10-16T13:47:24.421423+0000","last_clean":"2023-10-16T13:47:24.421423+0000","last_became_active":"2023-10-16T13:47:23.433275+0000","last_became_peered":"2023-10-16T13:47:23.433275+0000","last_unstale":"2023-10-16T13:47:24.421423+0000","last_undegraded":"2023-10-16T13:47:24.421423+0000","last_fullsized":"2023-10-16T13:47:24.421423+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:21:17.525717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.2a","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100402+0000","last_change":"2023-10-16T13:47:24.424605+0000","last_active":"2023-10-16T13:47:26.100402+0000","last_peered":"2023-10-16T13:47:26.100402+0000","last_clean":"2023-10-16T13:47:26.100402+0000","last_became_active":"2023-10-16T13:47:24.424313+0000","last_became_peered":"2023-10-16T13:47:24.424313+0000","last_unstale":"2023-10-16T13:47:26.100402+0000","last_undegraded":"2023-10-16T13:47:26.100402+0000","last_fullsized":"2023-10-16T13:47:26.100402+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:40:33.368782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.15","version":"244'3","reported_seq":15,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.446109+0000","last_change":"2023-10-16T13:47:23.431025+0000","last_active":"2023-10-16T13:47:24.446109+0000","last_peered":"2023-10-16T13:47:24.446109+0000","last_clean":"2023-10-16T13:47:24.446109+0000","last_became_active":"2023-10-16T13:47:23.430710+0000","last_became_peered":"2023-10-16T13:47:23.430710+0000","last_unstale":"2023-10-16T13:47:24.446109+0000","last_undegraded":"2023-10-16T13:47:24.446109+0000","last_fullsized":"2023-10-16T13:47:24.446109+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:52:27.207432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T13:47:23.404948+0000","last_change":"2023-10-16T13:47:23.404948+0000","last_active":"2023-10-16T13:47:23.404948+0000","last_peered":"2023-10-16T13:47:23.404948+0000","last_clean":"2023-10-16T13:47:23.404948+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:23.404948+0000","last_undegraded":"2023-10-16T13:47:23.404948+0000","last_fullsized":"2023-10-16T13:47:23.404948+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.14","version":"244'6","reported_seq":18,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.453158+0000","last_change":"2023-10-16T13:47:23.430528+0000","last_active":"2023-10-16T13:47:24.453158+0000","last_peered":"2023-10-16T13:47:24.453158+0000","last_clean":"2023-10-16T13:47:24.453158+0000","last_became_active":"2023-10-16T13:47:23.429721+0000","last_became_peered":"2023-10-16T13:47:23.429721+0000","last_unstale":"2023-10-16T13:47:24.453158+0000","last_undegraded":"2023-10-16T13:47:24.453158+0000","last_fullsized":"2023-10-16T13:47:24.453158+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:07:56.379842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.2c","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.416579+0000","last_change":"2023-10-16T13:47:24.436132+0000","last_active":"2023-10-16T13:47:25.416579+0000","last_peered":"2023-10-16T13:47:25.416579+0000","last_clean":"2023-10-16T13:47:25.416579+0000","last_became_active":"2023-10-16T13:47:24.435272+0000","last_became_peered":"2023-10-16T13:47:24.435272+0000","last_unstale":"2023-10-16T13:47:25.416579+0000","last_undegraded":"2023-10-16T13:47:25.416579+0000","last_fullsized":"2023-10-16T13:47:25.416579+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:07:59.434149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.13","version":"0'0","reported_seq":1,"reported_epoch":242,"state":"creating+peering","last_fresh":"2023-10-16T13:47:22.412143+0000","last_change":"2023-10-16T13:47:22.412143+0000","last_active":"2023-10-16T13:47:22.400586+0000","last_peered":"2023-10-16T13:47:22.400586+0000","last_clean":"2023-10-16T13:47:22.400586+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:22.412143+0000","last_undegraded":"2023-10-16T13:47:22.412143+0000","last_fullsized":"2023-10-16T13:47:22.412143+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T13:47:23.404948+0000","last_change":"2023-10-16T13:47:23.404948+0000","last_active":"2023-10-16T13:47:23.404948+0000","last_peered":"2023-10-16T13:47:23.404948+0000","last_clean":"2023-10-16T13:47:23.404948+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:23.404948+0000","last_undegraded":"2023-10-16T13:47:23.404948+0000","last_fullsized":"2023-10-16T13:47:23.404948+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.418559+0000","last_change":"2023-10-16T13:47:23.433222+0000","last_active":"2023-10-16T13:47:24.418559+0000","last_peered":"2023-10-16T13:47:24.418559+0000","last_clean":"2023-10-16T13:47:24.418559+0000","last_became_active":"2023-10-16T13:47:23.432936+0000","last_became_peered":"2023-10-16T13:47:23.432936+0000","last_unstale":"2023-10-16T13:47:24.418559+0000","last_undegraded":"2023-10-16T13:47:24.418559+0000","last_fullsized":"2023-10-16T13:47:24.418559+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:47:59.336633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T13:47:23.404948+0000","last_change":"2023-10-16T13:47:23.404948+0000","last_active":"2023-10-16T13:47:23.404948+0000","last_peered":"2023-10-16T13:47:23.404948+0000","last_clean":"2023-10-16T13:47:23.404948+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:23.404948+0000","last_undegraded":"2023-10-16T13:47:23.404948+0000","last_fullsized":"2023-10-16T13:47:23.404948+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.417965+0000","last_change":"2023-10-16T13:47:23.425872+0000","last_active":"2023-10-16T13:47:24.417965+0000","last_peered":"2023-10-16T13:47:24.417965+0000","last_clean":"2023-10-16T13:47:24.417965+0000","last_became_active":"2023-10-16T13:47:23.425569+0000","last_became_peered":"2023-10-16T13:47:23.425569+0000","last_unstale":"2023-10-16T13:47:24.417965+0000","last_undegraded":"2023-10-16T13:47:24.417965+0000","last_fullsized":"2023-10-16T13:47:24.417965+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:09:18.086723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2f","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.101809+0000","last_change":"2023-10-16T13:47:24.437030+0000","last_active":"2023-10-16T13:47:26.101809+0000","last_peered":"2023-10-16T13:47:26.101809+0000","last_clean":"2023-10-16T13:47:26.101809+0000","last_became_active":"2023-10-16T13:47:24.436734+0000","last_became_peered":"2023-10-16T13:47:24.436734+0000","last_unstale":"2023-10-16T13:47:26.101809+0000","last_undegraded":"2023-10-16T13:47:26.101809+0000","last_fullsized":"2023-10-16T13:47:26.101809+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:28:53.363293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.10","version":"244'8","reported_seq":20,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.453887+0000","last_change":"2023-10-16T13:47:23.426967+0000","last_active":"2023-10-16T13:47:24.453887+0000","last_peered":"2023-10-16T13:47:24.453887+0000","last_clean":"2023-10-16T13:47:24.453887+0000","last_became_active":"2023-10-16T13:47:23.426614+0000","last_became_peered":"2023-10-16T13:47:23.426614+0000","last_unstale":"2023-10-16T13:47:24.453887+0000","last_undegraded":"2023-10-16T13:47:24.453887+0000","last_fullsized":"2023-10-16T13:47:24.453887+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:39:12.211841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.30","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.101224+0000","last_change":"2023-10-16T13:47:24.437081+0000","last_active":"2023-10-16T13:47:26.101224+0000","last_peered":"2023-10-16T13:47:26.101224+0000","last_clean":"2023-10-16T13:47:26.101224+0000","last_became_active":"2023-10-16T13:47:24.436501+0000","last_became_peered":"2023-10-16T13:47:24.436501+0000","last_unstale":"2023-10-16T13:47:26.101224+0000","last_undegraded":"2023-10-16T13:47:26.101224+0000","last_fullsized":"2023-10-16T13:47:26.101224+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:23:30.831964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.f","version":"244'1","reported_seq":13,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.445592+0000","last_change":"2023-10-16T13:47:23.432161+0000","last_active":"2023-10-16T13:47:24.445592+0000","last_peered":"2023-10-16T13:47:24.445592+0000","last_clean":"2023-10-16T13:47:24.445592+0000","last_became_active":"2023-10-16T13:47:23.431822+0000","last_became_peered":"2023-10-16T13:47:23.431822+0000","last_unstale":"2023-10-16T13:47:24.445592+0000","last_undegraded":"2023-10-16T13:47:24.445592+0000","last_fullsized":"2023-10-16T13:47:24.445592+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:15:50.867051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.31","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100923+0000","last_change":"2023-10-16T13:47:24.431186+0000","last_active":"2023-10-16T13:47:26.100923+0000","last_peered":"2023-10-16T13:47:26.100923+0000","last_clean":"2023-10-16T13:47:26.100923+0000","last_became_active":"2023-10-16T13:47:24.430892+0000","last_became_peered":"2023-10-16T13:47:24.430892+0000","last_unstale":"2023-10-16T13:47:26.100923+0000","last_undegraded":"2023-10-16T13:47:26.100923+0000","last_fullsized":"2023-10-16T13:47:26.100923+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:12:21.693658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.e","version":"244'2","reported_seq":14,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.448939+0000","last_change":"2023-10-16T13:47:23.424549+0000","last_active":"2023-10-16T13:47:24.448939+0000","last_peered":"2023-10-16T13:47:24.448939+0000","last_clean":"2023-10-16T13:47:24.448939+0000","last_became_active":"2023-10-16T13:47:23.424111+0000","last_became_peered":"2023-10-16T13:47:23.424111+0000","last_unstale":"2023-10-16T13:47:24.448939+0000","last_undegraded":"2023-10-16T13:47:24.448939+0000","last_fullsized":"2023-10-16T13:47:24.448939+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:36:35.137200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.32","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.416229+0000","last_change":"2023-10-16T13:47:24.432940+0000","last_active":"2023-10-16T13:47:25.416229+0000","last_peered":"2023-10-16T13:47:25.416229+0000","last_clean":"2023-10-16T13:47:25.416229+0000","last_became_active":"2023-10-16T13:47:24.432610+0000","last_became_peered":"2023-10-16T13:47:24.432610+0000","last_unstale":"2023-10-16T13:47:25.416229+0000","last_undegraded":"2023-10-16T13:47:25.416229+0000","last_fullsized":"2023-10-16T13:47:25.416229+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:59:08.104923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.d","version":"0'0","reported_seq":1,"reported_epoch":242,"state":"creating+peering","last_fresh":"2023-10-16T13:47:22.413572+0000","last_change":"2023-10-16T13:47:22.413572+0000","last_active":"2023-10-16T13:47:22.400586+0000","last_peered":"2023-10-16T13:47:22.400586+0000","last_clean":"2023-10-16T13:47:22.400586+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:22.413572+0000","last_undegraded":"2023-10-16T13:47:22.413572+0000","last_fullsized":"2023-10-16T13:47:22.413572+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.33","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100188+0000","last_change":"2023-10-16T13:47:24.427239+0000","last_active":"2023-10-16T13:47:26.100188+0000","last_peered":"2023-10-16T13:47:26.100188+0000","last_clean":"2023-10-16T13:47:26.100188+0000","last_became_active":"2023-10-16T13:47:24.426919+0000","last_became_peered":"2023-10-16T13:47:24.426919+0000","last_unstale":"2023-10-16T13:47:26.100188+0000","last_undegraded":"2023-10-16T13:47:26.100188+0000","last_fullsized":"2023-10-16T13:47:26.100188+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:56:16.580840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":1,"reported_epoch":242,"state":"creating+peering","last_fresh":"2023-10-16T13:47:22.412157+0000","last_change":"2023-10-16T13:47:22.412157+0000","last_active":"2023-10-16T13:47:22.400586+0000","last_peered":"2023-10-16T13:47:22.400586+0000","last_clean":"2023-10-16T13:47:22.400586+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:22.412157+0000","last_undegraded":"2023-10-16T13:47:22.412157+0000","last_fullsized":"2023-10-16T13:47:22.412157+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.34","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100514+0000","last_change":"2023-10-16T13:47:24.444364+0000","last_active":"2023-10-16T13:47:26.100514+0000","last_peered":"2023-10-16T13:47:26.100514+0000","last_clean":"2023-10-16T13:47:26.100514+0000","last_became_active":"2023-10-16T13:47:24.444045+0000","last_became_peered":"2023-10-16T13:47:24.444045+0000","last_unstale":"2023-10-16T13:47:26.100514+0000","last_undegraded":"2023-10-16T13:47:26.100514+0000","last_fullsized":"2023-10-16T13:47:26.100514+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:13:11.795921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.b","version":"244'2","reported_seq":14,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.449802+0000","last_change":"2023-10-16T13:47:23.434957+0000","last_active":"2023-10-16T13:47:24.449802+0000","last_peered":"2023-10-16T13:47:24.449802+0000","last_clean":"2023-10-16T13:47:24.449802+0000","last_became_active":"2023-10-16T13:47:23.434637+0000","last_became_peered":"2023-10-16T13:47:23.434637+0000","last_unstale":"2023-10-16T13:47:24.449802+0000","last_undegraded":"2023-10-16T13:47:24.449802+0000","last_fullsized":"2023-10-16T13:47:24.449802+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:23:47.081423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.35","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.416431+0000","last_change":"2023-10-16T13:47:24.431892+0000","last_active":"2023-10-16T13:47:25.416431+0000","last_peered":"2023-10-16T13:47:25.416431+0000","last_clean":"2023-10-16T13:47:25.416431+0000","last_became_active":"2023-10-16T13:47:24.431624+0000","last_became_peered":"2023-10-16T13:47:24.431624+0000","last_unstale":"2023-10-16T13:47:25.416431+0000","last_undegraded":"2023-10-16T13:47:25.416431+0000","last_fullsized":"2023-10-16T13:47:25.416431+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:49:13.422163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":1,"reported_epoch":242,"state":"creating+peering","last_fresh":"2023-10-16T13:47:22.412844+0000","last_change":"2023-10-16T13:47:22.412844+0000","last_active":"2023-10-16T13:47:22.400586+0000","last_peered":"2023-10-16T13:47:22.400586+0000","last_clean":"2023-10-16T13:47:22.400586+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:22.412844+0000","last_undegraded":"2023-10-16T13:47:22.412844+0000","last_fullsized":"2023-10-16T13:47:22.412844+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T13:47:23.404948+0000","last_change":"2023-10-16T13:47:23.404948+0000","last_active":"2023-10-16T13:47:23.404948+0000","last_peered":"2023-10-16T13:47:23.404948+0000","last_clean":"2023-10-16T13:47:23.404948+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:23.404948+0000","last_undegraded":"2023-10-16T13:47:23.404948+0000","last_fullsized":"2023-10-16T13:47:23.404948+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":1,"reported_epoch":242,"state":"creating+peering","last_fresh":"2023-10-16T13:47:22.412735+0000","last_change":"2023-10-16T13:47:22.412735+0000","last_active":"2023-10-16T13:47:22.400586+0000","last_peered":"2023-10-16T13:47:22.400586+0000","last_clean":"2023-10-16T13:47:22.400586+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:22.412735+0000","last_undegraded":"2023-10-16T13:47:22.412735+0000","last_fullsized":"2023-10-16T13:47:22.412735+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T13:47:23.404948+0000","last_change":"2023-10-16T13:47:23.404948+0000","last_active":"2023-10-16T13:47:23.404948+0000","last_peered":"2023-10-16T13:47:23.404948+0000","last_clean":"2023-10-16T13:47:23.404948+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:23.404948+0000","last_undegraded":"2023-10-16T13:47:23.404948+0000","last_fullsized":"2023-10-16T13:47:23.404948+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.415241+0000","last_change":"2023-10-16T13:47:23.433223+0000","last_active":"2023-10-16T13:47:24.415241+0000","last_peered":"2023-10-16T13:47:24.415241+0000","last_clean":"2023-10-16T13:47:24.415241+0000","last_became_active":"2023-10-16T13:47:23.432935+0000","last_became_peered":"2023-10-16T13:47:23.432935+0000","last_unstale":"2023-10-16T13:47:24.415241+0000","last_undegraded":"2023-10-16T13:47:24.415241+0000","last_fullsized":"2023-10-16T13:47:24.415241+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:40:59.034875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T13:47:23.404948+0000","last_change":"2023-10-16T13:47:23.404948+0000","last_active":"2023-10-16T13:47:23.404948+0000","last_peered":"2023-10-16T13:47:23.404948+0000","last_clean":"2023-10-16T13:47:23.404948+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:23.404948+0000","last_undegraded":"2023-10-16T13:47:23.404948+0000","last_fullsized":"2023-10-16T13:47:23.404948+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.7","version":"244'2","reported_seq":14,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.451763+0000","last_change":"2023-10-16T13:47:23.430865+0000","last_active":"2023-10-16T13:47:24.451763+0000","last_peered":"2023-10-16T13:47:24.451763+0000","last_clean":"2023-10-16T13:47:24.451763+0000","last_became_active":"2023-10-16T13:47:23.430493+0000","last_became_peered":"2023-10-16T13:47:23.430493+0000","last_unstale":"2023-10-16T13:47:24.451763+0000","last_undegraded":"2023-10-16T13:47:24.451763+0000","last_fullsized":"2023-10-16T13:47:24.451763+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:59:14.057546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T13:47:23.404948+0000","last_change":"2023-10-16T13:47:23.404948+0000","last_active":"2023-10-16T13:47:23.404948+0000","last_peered":"2023-10-16T13:47:23.404948+0000","last_clean":"2023-10-16T13:47:23.404948+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:23.404948+0000","last_undegraded":"2023-10-16T13:47:23.404948+0000","last_fullsized":"2023-10-16T13:47:23.404948+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.417912+0000","last_change":"2023-10-16T13:47:23.430851+0000","last_active":"2023-10-16T13:47:24.417912+0000","last_peered":"2023-10-16T13:47:24.417912+0000","last_clean":"2023-10-16T13:47:24.417912+0000","last_became_active":"2023-10-16T13:47:23.430526+0000","last_became_peered":"2023-10-16T13:47:23.430526+0000","last_unstale":"2023-10-16T13:47:24.417912+0000","last_undegraded":"2023-10-16T13:47:24.417912+0000","last_fullsized":"2023-10-16T13:47:24.417912+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:40:16.878541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.3a","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100155+0000","last_change":"2023-10-16T13:47:24.440347+0000","last_active":"2023-10-16T13:47:26.100155+0000","last_peered":"2023-10-16T13:47:26.100155+0000","last_clean":"2023-10-16T13:47:26.100155+0000","last_became_active":"2023-10-16T13:47:24.440041+0000","last_became_peered":"2023-10-16T13:47:24.440041+0000","last_unstale":"2023-10-16T13:47:26.100155+0000","last_undegraded":"2023-10-16T13:47:26.100155+0000","last_fullsized":"2023-10-16T13:47:26.100155+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:19:38.293171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.415325+0000","last_change":"2023-10-16T13:47:23.427399+0000","last_active":"2023-10-16T13:47:24.415325+0000","last_peered":"2023-10-16T13:47:24.415325+0000","last_clean":"2023-10-16T13:47:24.415325+0000","last_became_active":"2023-10-16T13:47:23.427092+0000","last_became_peered":"2023-10-16T13:47:23.427092+0000","last_unstale":"2023-10-16T13:47:24.415325+0000","last_undegraded":"2023-10-16T13:47:24.415325+0000","last_fullsized":"2023-10-16T13:47:24.415325+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:10:56.739817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3b","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.101556+0000","last_change":"2023-10-16T13:47:24.436718+0000","last_active":"2023-10-16T13:47:26.101556+0000","last_peered":"2023-10-16T13:47:26.101556+0000","last_clean":"2023-10-16T13:47:26.101556+0000","last_became_active":"2023-10-16T13:47:24.436371+0000","last_became_peered":"2023-10-16T13:47:24.436371+0000","last_unstale":"2023-10-16T13:47:26.101556+0000","last_undegraded":"2023-10-16T13:47:26.101556+0000","last_fullsized":"2023-10-16T13:47:26.101556+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:58:12.134727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.416043+0000","last_change":"2023-10-16T13:47:23.433833+0000","last_active":"2023-10-16T13:47:24.416043+0000","last_peered":"2023-10-16T13:47:24.416043+0000","last_clean":"2023-10-16T13:47:24.416043+0000","last_became_active":"2023-10-16T13:47:23.433381+0000","last_became_peered":"2023-10-16T13:47:23.433381+0000","last_unstale":"2023-10-16T13:47:24.416043+0000","last_undegraded":"2023-10-16T13:47:24.416043+0000","last_fullsized":"2023-10-16T13:47:24.416043+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:09:34.543970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3c","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.099990+0000","last_change":"2023-10-16T13:47:24.424889+0000","last_active":"2023-10-16T13:47:26.099990+0000","last_peered":"2023-10-16T13:47:26.099990+0000","last_clean":"2023-10-16T13:47:26.099990+0000","last_became_active":"2023-10-16T13:47:24.424573+0000","last_became_peered":"2023-10-16T13:47:24.424573+0000","last_unstale":"2023-10-16T13:47:26.099990+0000","last_undegraded":"2023-10-16T13:47:26.099990+0000","last_fullsized":"2023-10-16T13:47:26.099990+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:23:40.809515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":1,"reported_epoch":242,"state":"creating+peering","last_fresh":"2023-10-16T13:47:22.413180+0000","last_change":"2023-10-16T13:47:22.413180+0000","last_active":"2023-10-16T13:47:22.400586+0000","last_peered":"2023-10-16T13:47:22.400586+0000","last_clean":"2023-10-16T13:47:22.400586+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:22.413180+0000","last_undegraded":"2023-10-16T13:47:22.413180+0000","last_fullsized":"2023-10-16T13:47:22.413180+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.3d","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.417707+0000","last_change":"2023-10-16T13:47:24.430727+0000","last_active":"2023-10-16T13:47:25.417707+0000","last_peered":"2023-10-16T13:47:25.417707+0000","last_clean":"2023-10-16T13:47:25.417707+0000","last_became_active":"2023-10-16T13:47:24.430424+0000","last_became_peered":"2023-10-16T13:47:24.430424+0000","last_unstale":"2023-10-16T13:47:25.417707+0000","last_undegraded":"2023-10-16T13:47:25.417707+0000","last_fullsized":"2023-10-16T13:47:25.417707+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:05:49.279256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.418627+0000","last_change":"2023-10-16T13:47:23.430779+0000","last_active":"2023-10-16T13:47:24.418627+0000","last_peered":"2023-10-16T13:47:24.418627+0000","last_clean":"2023-10-16T13:47:24.418627+0000","last_became_active":"2023-10-16T13:47:23.429996+0000","last_became_peered":"2023-10-16T13:47:23.429996+0000","last_unstale":"2023-10-16T13:47:24.418627+0000","last_undegraded":"2023-10-16T13:47:24.418627+0000","last_fullsized":"2023-10-16T13:47:24.418627+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:17:22.079776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.3e","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.416486+0000","last_change":"2023-10-16T13:47:24.436378+0000","last_active":"2023-10-16T13:47:25.416486+0000","last_peered":"2023-10-16T13:47:25.416486+0000","last_clean":"2023-10-16T13:47:25.416486+0000","last_became_active":"2023-10-16T13:47:24.435495+0000","last_became_peered":"2023-10-16T13:47:24.435495+0000","last_unstale":"2023-10-16T13:47:25.416486+0000","last_undegraded":"2023-10-16T13:47:25.416486+0000","last_fullsized":"2023-10-16T13:47:25.416486+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:18:23.073022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.420122+0000","last_change":"2023-10-16T13:47:23.433945+0000","last_active":"2023-10-16T13:47:24.420122+0000","last_peered":"2023-10-16T13:47:24.420122+0000","last_clean":"2023-10-16T13:47:24.420122+0000","last_became_active":"2023-10-16T13:47:23.433576+0000","last_became_peered":"2023-10-16T13:47:23.433576+0000","last_unstale":"2023-10-16T13:47:24.420122+0000","last_undegraded":"2023-10-16T13:47:24.420122+0000","last_fullsized":"2023-10-16T13:47:24.420122+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:09:34.543970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T13:47:23.404948+0000","last_change":"2023-10-16T13:47:23.404948+0000","last_active":"2023-10-16T13:47:23.404948+0000","last_peered":"2023-10-16T13:47:23.404948+0000","last_clean":"2023-10-16T13:47:23.404948+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:23.404948+0000","last_undegraded":"2023-10-16T13:47:23.404948+0000","last_fullsized":"2023-10-16T13:47:23.404948+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":1,"reported_epoch":242,"state":"creating+peering","last_fresh":"2023-10-16T13:47:22.412533+0000","last_change":"2023-10-16T13:47:22.412533+0000","last_active":"2023-10-16T13:47:22.400586+0000","last_peered":"2023-10-16T13:47:22.400586+0000","last_clean":"2023-10-16T13:47:22.400586+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:22.412533+0000","last_undegraded":"2023-10-16T13:47:22.412533+0000","last_fullsized":"2023-10-16T13:47:22.412533+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.21","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.417066+0000","last_change":"2023-10-16T13:47:24.431657+0000","last_active":"2023-10-16T13:47:25.417066+0000","last_peered":"2023-10-16T13:47:25.417066+0000","last_clean":"2023-10-16T13:47:25.417066+0000","last_became_active":"2023-10-16T13:47:24.431385+0000","last_became_peered":"2023-10-16T13:47:24.431385+0000","last_unstale":"2023-10-16T13:47:25.417066+0000","last_undegraded":"2023-10-16T13:47:25.417066+0000","last_fullsized":"2023-10-16T13:47:25.417066+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:59:09.754185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1e","version":"0'0","reported_seq":1,"reported_epoch":242,"state":"creating+peering","last_fresh":"2023-10-16T13:47:22.406892+0000","last_change":"2023-10-16T13:47:22.406892+0000","last_active":"2023-10-16T13:47:22.400586+0000","last_peered":"2023-10-16T13:47:22.400586+0000","last_clean":"2023-10-16T13:47:22.400586+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:22.406892+0000","last_undegraded":"2023-10-16T13:47:22.406892+0000","last_fullsized":"2023-10-16T13:47:22.406892+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"231'370","reported_seq":1295,"reported_epoch":241,"state":"active+clean","last_fresh":"2023-10-16T13:47:20.908271+0000","last_change":"2023-10-16T12:10:34.746820+0000","last_active":"2023-10-16T13:47:20.908271+0000","last_peered":"2023-10-16T13:47:20.908271+0000","last_clean":"2023-10-16T13:47:20.908271+0000","last_became_active":"2023-10-16T12:10:34.745574+0000","last_became_peered":"2023-10-16T12:10:34.745574+0000","last_unstale":"2023-10-16T13:47:20.908271+0000","last_undegraded":"2023-10-16T13:47:20.908271+0000","last_fullsized":"2023-10-16T13:47:20.908271+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_clean_scrub_stamp":"2023-10-16T12:10:33.577111+0000","objects_scrubbed":0,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:14.461986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":32,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":84,"acting":84,"num_store_stats":5},{"poolid":31,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":37,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":4626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31,"ondisk_log_size":31,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":370,"ondisk_log_size":370,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575995,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56936,"kb_used_data":3548,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314904,"statfs":{"total":96636764160,"available":96578461696,"internally_reserved":0,"allocated":3633152,"data_stored":2159844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54655435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:46:50 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.60299999999999998,"15min":0.63200000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.33000000000000002},"max":{"1min":0.78600000000000003,"5min":0.87,"15min":0.93799999999999994},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.60699999999999998,"15min":0.626},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.72099999999999997,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.63900000000000001}]},{"osd":1,"last update":"Mon Oct 16 13:46:50 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.61299999999999999,"15min":0.63900000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.79000000000000004,"5min":0.84599999999999997,"15min":0.95899999999999996},"last":0.497},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.622,"15min":0.63100000000000001},"min":{"1min":0.38200000000000001,"5min":0.375,"15min":0.375},"max":{"1min":0.76200000000000001,"5min":0.79100000000000004,"15min":0.86399999999999999},"last":0.54900000000000004}]},{"osd":2,"last update":"Mon Oct 16 13:46:50 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.67400000000000004,"15min":0.68600000000000005},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.80300000000000005,"5min":0.83899999999999997,"15min":0.88200000000000001},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.68899999999999995,"15min":0.69799999999999995},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.39400000000000002},"max":{"1min":0.83699999999999997,"5min":0.92800000000000005,"15min":0.93799999999999994},"last":0.66100000000000003}]},{"osd":3,"last update":"Mon Oct 16 13:46:50 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.63800000000000001,"15min":0.65900000000000003},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.85299999999999998,"5min":0.89200000000000002,"15min":0.91300000000000003},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.65000000000000002,"15min":0.66000000000000003},"min":{"1min":0.435,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.88,"5min":1.24,"15min":1.24},"last":0.69599999999999995}]},{"osd":4,"last update":"Mon Oct 16 13:46:50 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.52000000000000002,"15min":0.51900000000000002},"min":{"1min":0.311,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.71099999999999997,"5min":0.745,"15min":0.85299999999999998},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.621,"15min":0.61799999999999999},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.33100000000000002},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.70899999999999996}]},{"osd":5,"last update":"Mon Oct 16 13:46:50 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.54900000000000004,"15min":0.54700000000000004},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.85699999999999998,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.58399999999999996,"15min":0.58699999999999997},"min":{"1min":0.41099999999999998,"5min":0.38400000000000001,"15min":0.35099999999999998},"max":{"1min":0.81100000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.73999999999999999}]},{"osd":6,"last update":"Mon Oct 16 13:46:50 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.60499999999999998,"15min":0.61299999999999999},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.85499999999999998,"5min":0.874,"15min":0.95899999999999996},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.626,"15min":0.61899999999999999},"min":{"1min":0.39400000000000002,"5min":0.378,"15min":0.35199999999999998},"max":{"1min":0.92100000000000004,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.76200000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575994,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60572,"kb_used_data":3024,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311268,"statfs":{"total":96636764160,"available":96574738432,"internally_reserved":0,"allocated":3096576,"data_stored":1596547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58915271},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:46:36 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.65300000000000002,"15min":0.66800000000000004},"min":{"1min":0.41499999999999998,"5min":0.375,"15min":0.28699999999999998},"max":{"1min":0.82599999999999996,"5min":0.89400000000000002,"15min":0.92300000000000004},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.64500000000000002,"15min":0.66400000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.80400000000000005,"5min":0.91400000000000003,"15min":0.93799999999999994},"last":0.68200000000000005}]},{"osd":1,"last update":"Mon Oct 16 13:46:36 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.58599999999999997,"15min":0.624},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.33100000000000002},"max":{"1min":0.70299999999999996,"5min":0.86899999999999999,"15min":0.90900000000000003},"last":0.504},{"interface":"front","average":{"1min":0.63,"5min":0.61399999999999999,"15min":0.62},"min":{"1min":0.41699999999999998,"5min":0.39800000000000002,"15min":0.38300000000000001},"max":{"1min":0.85799999999999998,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.42199999999999999}]},{"osd":2,"last update":"Mon Oct 16 13:46:36 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.622,"15min":0.63300000000000001},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.85799999999999998,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.59999999999999998,"15min":0.624},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.749,"5min":0.89000000000000001,"15min":0.91000000000000003},"last":0.44500000000000001}]},{"osd":3,"last update":"Mon Oct 16 13:46:36 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63900000000000001,"15min":0.66300000000000003},"min":{"1min":0.439,"5min":0.39900000000000002,"15min":0.38200000000000001},"max":{"1min":0.874,"5min":0.92400000000000004,"15min":0.95599999999999996},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.69099999999999995,"15min":0.71599999999999997},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.42299999999999999},"max":{"1min":0.85099999999999998,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.48099999999999998}]},{"osd":4,"last update":"Mon Oct 16 13:46:36 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.51800000000000002,"15min":0.52700000000000002},"min":{"1min":0.39000000000000001,"5min":0.35999999999999999,"15min":0.315},"max":{"1min":0.78000000000000003,"5min":0.85599999999999998,"15min":0.95799999999999996},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61699999999999999,"15min":0.62},"min":{"1min":0.375,"5min":0.35499999999999998,"15min":0.32900000000000001},"max":{"1min":0.83699999999999997,"5min":0.98799999999999999,"15min":1.0309999999999999},"last":0.64600000000000002}]},{"osd":5,"last update":"Mon Oct 16 13:46:36 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.59799999999999998,"15min":0.59399999999999997},"min":{"1min":0.39800000000000002,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.81100000000000005,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.66500000000000004,"15min":0.66000000000000003},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.308},"max":{"1min":0.85899999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.53500000000000003}]},{"osd":7,"last update":"Mon Oct 16 13:46:36 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.60899999999999999,"15min":0.60499999999999998},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.80000000000000004,"5min":1.081,"15min":1.081},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65200000000000002},"min":{"1min":0.42799999999999999,"5min":0.41899999999999998,"15min":0.375},"max":{"1min":0.91500000000000004,"5min":1.099,"15min":1.099},"last":0.629}]}]},{"osd":1,"up_from":13,"seq":55834575994,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52252,"kb_used_data":3016,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319588,"statfs":{"total":96636764160,"available":96583258112,"internally_reserved":0,"allocated":3088384,"data_stored":1596435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":50395593},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:46:55 2023","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.39500000000000002,"15min":0.40699999999999997},"min":{"1min":0.27600000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.51300000000000001,"5min":0.65400000000000003,"15min":0.65800000000000003},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.47599999999999998,"15min":0.49199999999999999},"min":{"1min":0.32000000000000001,"5min":0.253,"15min":0.253},"max":{"1min":0.56000000000000005,"5min":0.70999999999999996,"15min":0.77200000000000002},"last":0.67000000000000004}]},{"osd":2,"last update":"Mon Oct 16 13:46:55 2023","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.436,"15min":0.45100000000000001},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.58599999999999997,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.48099999999999998,"15min":0.48999999999999999},"min":{"1min":0.34399999999999997,"5min":0.315,"15min":0.30199999999999999},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.71399999999999997},"last":0.65200000000000002}]},{"osd":3,"last update":"Mon Oct 16 13:46:55 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.42899999999999999,"15min":0.42799999999999999},"min":{"1min":0.307,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.50800000000000001,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.46100000000000002,"15min":0.47499999999999998},"min":{"1min":0.29799999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.52800000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.69399999999999995}]},{"osd":4,"last update":"Mon Oct 16 13:46:55 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.64400000000000002,"15min":0.66200000000000003},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.42399999999999999},"max":{"1min":0.78600000000000003,"5min":0.95899999999999996,"15min":1.0069999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.64200000000000002,"15min":0.66600000000000004},"min":{"1min":0.42899999999999999,"5min":0.37,"15min":0.37},"max":{"1min":0.69999999999999996,"5min":0.91300000000000003,"15min":0.997},"last":0.747}]},{"osd":5,"last update":"Mon Oct 16 13:46:55 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.68200000000000005,"15min":0.70699999999999996},"min":{"1min":0.45800000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.83999999999999997,"5min":0.97499999999999998,"15min":1.0760000000000001},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.70499999999999996,"15min":0.72599999999999998},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.81399999999999995,"5min":1.0469999999999999,"15min":1.0660000000000001},"last":0.73199999999999998}]},{"osd":6,"last update":"Mon Oct 16 13:46:55 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.67500000000000004,"15min":0.68000000000000005},"min":{"1min":0.46200000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.75900000000000001,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.68700000000000006,"15min":0.70499999999999996},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.80400000000000005,"5min":1.03,"15min":1.083},"last":0.72299999999999998}]},{"osd":7,"last update":"Mon Oct 16 13:46:55 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.70899999999999996,"15min":0.73399999999999999},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.77300000000000002,"5min":1.0629999999999999,"15min":1.0660000000000001},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.72899999999999998,"15min":0.746},"min":{"1min":0.496,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.82099999999999995,"5min":1.042,"15min":1.0580000000000001},"last":0.87}]}]},{"osd":0,"up_from":13,"seq":55834575995,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48676,"kb_used_data":3608,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323164,"statfs":{"total":96636764160,"available":96586919936,"internally_reserved":0,"allocated":3694592,"data_stored":2188415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 13:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.40400000000000003,"15min":0.42099999999999999},"min":{"1min":0.29099999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.60699999999999998,"5min":0.65800000000000003,"15min":0.71399999999999997},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.42199999999999999,"15min":0.439},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.82199999999999995},"last":0.41999999999999998}]},{"osd":2,"last update":"Mon Oct 16 13:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.443,"15min":0.45400000000000001},"min":{"1min":0.28599999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.61399999999999999,"5min":0.65900000000000003,"15min":0.75700000000000001},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.46400000000000002,"15min":0.48499999999999999},"min":{"1min":0.29799999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.83299999999999996},"last":0.60499999999999998}]},{"osd":3,"last update":"Mon Oct 16 13:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.42699999999999999,"15min":0.45300000000000001},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.56499999999999995,"5min":0.63400000000000001,"15min":0.78600000000000003},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.50900000000000001,"15min":0.52000000000000002},"min":{"1min":0.317,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.66400000000000003,"5min":0.82999999999999996,"15min":0.86399999999999999},"last":0.57199999999999995}]},{"osd":4,"last update":"Mon Oct 16 13:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.66400000000000003,"15min":0.68400000000000005},"min":{"1min":0.42399999999999999,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.66500000000000004,"15min":0.68899999999999995},"min":{"1min":0.499,"5min":0.375,"15min":0.375},"max":{"1min":1.21,"5min":1.21,"15min":1.21},"last":0.78200000000000003}]},{"osd":5,"last update":"Mon Oct 16 13:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.66400000000000003,"15min":0.68999999999999995},"min":{"1min":0.41999999999999998,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.242,"5min":1.242,"15min":1.242},"last":0.751},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.70599999999999996,"15min":0.72299999999999998},"min":{"1min":0.497,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.998,"5min":1.081,"15min":1.081},"last":0.63200000000000001}]},{"osd":6,"last update":"Mon Oct 16 13:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.71199999999999997,"15min":0.72899999999999998},"min":{"1min":0.51900000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.96599999999999997,"5min":1.034,"15min":1.103},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.70399999999999996,"15min":0.72299999999999998},"min":{"1min":0.51000000000000001,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":0.79600000000000004}]},{"osd":7,"last update":"Mon Oct 16 13:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.71199999999999997,"15min":0.74199999999999999},"min":{"1min":0.48399999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.1919999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.72499999999999998,"15min":0.73499999999999999},"min":{"1min":0.51300000000000001,"5min":0.433,"15min":0.433},"max":{"1min":1.0469999999999999,"5min":1.087,"15min":1.087},"last":0.64700000000000002}]}]},{"osd":2,"up_from":13,"seq":55834575995,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52252,"kb_used_data":3024,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319588,"statfs":{"total":96636764160,"available":96583258112,"internally_reserved":0,"allocated":3096576,"data_stored":1597087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.41499999999999998,"15min":0.40899999999999997},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.623,"5min":0.71699999999999997,"15min":0.77500000000000002},"last":0.438},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.47699999999999998,"15min":0.47999999999999998},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.52800000000000002}]},{"osd":1,"last update":"Mon Oct 16 13:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.502,"15min":0.5},"min":{"1min":0.317,"5min":0.311,"15min":0.311},"max":{"1min":0.66800000000000004,"5min":0.73199999999999998,"15min":0.81399999999999995},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.48999999999999999,"15min":0.47699999999999998},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.27700000000000002},"max":{"1min":0.69999999999999996,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.58599999999999997}]},{"osd":3,"last update":"Mon Oct 16 13:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.443,"15min":0.44},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.59699999999999998,"5min":0.70099999999999996,"15min":0.72599999999999998},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.54300000000000004,"15min":0.54400000000000004},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.82699999999999996,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.74099999999999999}]},{"osd":4,"last update":"Mon Oct 16 13:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.72699999999999998,"15min":0.71799999999999997},"min":{"1min":0.51500000000000001,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.89000000000000001,"5min":1.3600000000000001,"15min":1.3600000000000001},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.72599999999999998,"15min":0.71899999999999997},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.82899999999999996,"5min":1.403,"15min":1.403},"last":0.91400000000000003}]},{"osd":5,"last update":"Mon Oct 16 13:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.68600000000000005,"15min":0.68899999999999995},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.81100000000000005,"5min":1.486,"15min":1.486},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.71199999999999997,"15min":0.69999999999999996},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.90200000000000002,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":0.879}]},{"osd":6,"last update":"Mon Oct 16 13:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.67600000000000005,"15min":0.66600000000000004},"min":{"1min":0.44500000000000001,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.80200000000000005,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.72599999999999998,"15min":0.72199999999999998},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.90600000000000003,"5min":1.4379999999999999,"15min":1.4379999999999999},"last":0.94299999999999995}]},{"osd":7,"last update":"Mon Oct 16 13:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.73899999999999999,"15min":0.72099999999999997},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.94099999999999995,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":0.97099999999999997},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.72999999999999998,"15min":0.73099999999999998},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.91500000000000004,"5min":1.377,"15min":1.377},"last":1.0429999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575994,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56428,"kb_used_data":3040,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315412,"statfs":{"total":96636764160,"available":96578981888,"internally_reserved":0,"allocated":3112960,"data_stored":1598685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.40300000000000002,"15min":0.40999999999999998},"min":{"1min":0.23499999999999999,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.57199999999999995,"5min":0.64200000000000002,"15min":0.72899999999999998},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.41499999999999998,"15min":0.42799999999999999},"min":{"1min":0.27300000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.60099999999999998,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.312}]},{"osd":1,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.42699999999999999,"15min":0.434},"min":{"1min":0.26700000000000002,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.55900000000000005,"5min":0.68200000000000005,"15min":0.76300000000000001},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.435,"5min":0.45900000000000002,"15min":0.47999999999999998},"min":{"1min":0.27700000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.65500000000000003,"5min":0.70699999999999996,"15min":0.77800000000000002},"last":0.441}]},{"osd":2,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.47999999999999998,"15min":0.49299999999999999},"min":{"1min":0.33800000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.621,"5min":1.167,"15min":1.167},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.501,"15min":0.52300000000000002},"min":{"1min":0.312,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.625,"5min":0.73299999999999998,"15min":0.78700000000000003},"last":0.49199999999999999}]},{"osd":4,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.65100000000000002,"15min":0.66500000000000004},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.91200000000000003,"5min":0.998,"15min":1.137},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.68899999999999995,"15min":0.69399999999999995},"min":{"1min":0.441,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.087,"5min":1.2110000000000001,"15min":1.2909999999999999},"last":0.72799999999999998}]},{"osd":5,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.67700000000000005,"15min":0.70499999999999996},"min":{"1min":0.45500000000000002,"5min":0.42899999999999999,"15min":0.374},"max":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.2729999999999999},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.69299999999999995,"15min":0.69999999999999996},"min":{"1min":0.45600000000000002,"5min":0.432,"15min":0.432},"max":{"1min":0.98799999999999999,"5min":1.0549999999999999,"15min":1.173},"last":0.68600000000000005}]},{"osd":6,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.72899999999999998,"15min":0.71899999999999997},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.41799999999999998},"max":{"1min":0.99099999999999999,"5min":5.0999999999999996,"15min":5.0999999999999996},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.70999999999999996,"15min":0.73399999999999999},"min":{"1min":0.49199999999999999,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.0029999999999999,"5min":1.0309999999999999,"15min":1.2290000000000001},"last":0.80000000000000004}]},{"osd":7,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.71899999999999997,"15min":0.73399999999999999},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.048,"5min":1.048,"15min":1.2569999999999999},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.73299999999999998,"15min":0.746},"min":{"1min":0.501,"5min":0.47799999999999998,"15min":0.43099999999999999},"max":{"1min":1.0369999999999999,"5min":1.262,"15min":1.262},"last":0.66800000000000004}]}]},{"osd":4,"up_from":13,"seq":55834575992,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52208,"kb_used_data":2980,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319632,"statfs":{"total":96636764160,"available":96583303168,"internally_reserved":0,"allocated":3051520,"data_stored":1575741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.60699999999999998,"15min":0.625},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.33800000000000002},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.623,"15min":0.63100000000000001},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.84099999999999997,"5min":0.878,"15min":0.89500000000000002},"last":0.41699999999999998}]},{"osd":1,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.63,"15min":0.65600000000000003},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.628,"15min":0.63700000000000001},"min":{"1min":0.41099999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.50700000000000001}]},{"osd":2,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.65200000000000002,"15min":0.65300000000000002},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.83999999999999997,"5min":0.89900000000000002,"15min":0.90600000000000003},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.65600000000000003,"15min":0.65800000000000003},"min":{"1min":0.39600000000000002,"5min":0.39400000000000002,"15min":0.38400000000000001},"max":{"1min":0.88200000000000001,"5min":0.89700000000000002,"15min":0.92600000000000005},"last":0.438}]},{"osd":3,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.63200000000000001,"15min":0.64500000000000002},"min":{"1min":0.40400000000000003,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.78100000000000003,"5min":0.86599999999999999,"15min":0.99199999999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.67400000000000004,"15min":0.68200000000000005},"min":{"1min":0.44800000000000001,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.79100000000000004,"5min":0.872,"15min":0.93899999999999995},"last":0.46300000000000002}]},{"osd":5,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55100000000000005,"15min":0.53000000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.57999999999999996,"15min":0.56899999999999995},"min":{"1min":0.36099999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.82199999999999995,"5min":0.82299999999999995,"15min":0.94899999999999995},"last":0.59799999999999998}]},{"osd":6,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.625,"15min":0.59799999999999998},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.33200000000000002},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92700000000000005},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.747,"5min":0.71399999999999997,"15min":0.69399999999999995},"min":{"1min":0.52600000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.129,"5min":1.129,"15min":1.129},"last":0.61599999999999999}]},{"osd":7,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.66000000000000003,"15min":0.63500000000000001},"min":{"1min":0.44400000000000001,"5min":0.41999999999999998,"15min":0.35799999999999998},"max":{"1min":0.97099999999999997,"5min":0.97499999999999998,"15min":0.98599999999999999},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.70799999999999996,"15min":0.68400000000000005},"min":{"1min":0.44,"5min":0.433,"15min":0.38200000000000001},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":1.026},"last":0.71599999999999997}]}]},{"osd":5,"up_from":13,"seq":55834575993,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52196,"kb_used_data":2968,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319644,"statfs":{"total":96636764160,"available":96583315456,"internally_reserved":0,"allocated":3039232,"data_stored":1569476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:46:24 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.61399999999999999,"15min":0.61499999999999999},"min":{"1min":0.314,"5min":0.314,"15min":0.26800000000000002},"max":{"1min":0.878,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63,"15min":0.64200000000000002},"min":{"1min":0.38700000000000001,"5min":0.35899999999999999,"15min":0.29699999999999999},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.42099999999999999}]},{"osd":1,"last update":"Mon Oct 16 13:46:24 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.66000000000000003,"15min":0.66200000000000003},"min":{"1min":0.44600000000000001,"5min":0.34300000000000003,"15min":0.29499999999999998},"max":{"1min":1,"5min":1,"15min":1},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.63800000000000001,"15min":0.64000000000000001},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.58899999999999997}]},{"osd":2,"last update":"Mon Oct 16 13:46:24 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.67000000000000004,"15min":0.65700000000000003},"min":{"1min":0.41699999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.91000000000000003,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.629,"5min":0.65700000000000003,"15min":0.64900000000000002},"min":{"1min":0.42599999999999999,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.83899999999999997,"5min":0.871,"15min":0.93500000000000005},"last":0.51000000000000001}]},{"osd":3,"last update":"Mon Oct 16 13:46:24 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.67400000000000004,"15min":0.66400000000000003},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.69899999999999995,"15min":0.69499999999999995},"min":{"1min":0.41399999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.4199999999999999,"5min":1.4199999999999999,"15min":1.4199999999999999},"last":0.54500000000000004}]},{"osd":4,"last update":"Mon Oct 16 13:46:24 2023","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.54400000000000004,"15min":0.53300000000000003},"min":{"1min":0.34100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.82499999999999996,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.59299999999999997,"15min":0.58899999999999997},"min":{"1min":0.34300000000000003,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.85399999999999998,"5min":0.89000000000000001,"15min":0.93400000000000005},"last":0.753}]},{"osd":6,"last update":"Mon Oct 16 13:46:24 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.58899999999999997,"15min":0.57999999999999996},"min":{"1min":0.38700000000000001,"5min":0.315,"15min":0.315},"max":{"1min":0.877,"5min":0.88900000000000001,"15min":0.93200000000000005},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.61899999999999999},"min":{"1min":0.372,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89600000000000002},"last":0.68200000000000005}]},{"osd":7,"last update":"Mon Oct 16 13:46:24 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.64800000000000002,"15min":0.63300000000000001},"min":{"1min":0.378,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.69699999999999995,"15min":0.68400000000000005},"min":{"1min":0.46200000000000002,"5min":0.375,"15min":0.375},"max":{"1min":0.94699999999999995,"5min":0.98299999999999998,"15min":1.0229999999999999},"last":0.72399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T13:47:28.993 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T13:47:29.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.151+0000 7fb231d04700 1 Processor -- start 2023-10-16T13:47:29.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.151+0000 7fb231d04700 1 -- start start 2023-10-16T13:47:29.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.152+0000 7fb231d04700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb22c0cf880 0x7fb22c0b52f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:29.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.152+0000 7fb231d04700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fb22c0b5830 0x7fb22c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:29.156 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.152+0000 7fb231d04700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb22c15bc80 con 0x7fb22c0cf4a0 2023-10-16T13:47:29.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.152+0000 7fb231d04700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb22c15b070 con 0x7fb22c0b5830 2023-10-16T13:47:29.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.152+0000 7fb231d04700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb22c15b350 con 0x7fb22c0cf880 2023-10-16T13:47:29.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.152+0000 7fb22b7fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb22c0cf4a0 0x7fb22c0726e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44776/0 (socket says 172.21.15.16:44776) 2023-10-16T13:47:29.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.152+0000 7fb22b7fe700 1 -- 172.21.15.16:0/1459195816 learned_addr learned my addr 172.21.15.16:0/1459195816 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:47:29.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.152+0000 7fb22bfff700 1 --2- 172.21.15.16:0/1459195816 >> v2:172.21.15.49:3300/0 conn(0x7fb22c0b5830 0x7fb22c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:29.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.152+0000 7fb22affd700 1 --2- 172.21.15.16:0/1459195816 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb22c0cf880 0x7fb22c0b52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:29.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.152+0000 7fb22a7fc700 1 -- 172.21.15.16:0/1459195816 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2537984149 0 0) 0x7fb22c15bc80 con 0x7fb22c0cf4a0 2023-10-16T13:47:29.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.152+0000 7fb22a7fc700 1 -- 172.21.15.16:0/1459195816 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2140035b0 con 0x7fb22c0cf4a0 2023-10-16T13:47:29.158 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.152+0000 7fb22a7fc700 1 -- 172.21.15.16:0/1459195816 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2623385586 0 0) 0x7fb2140035b0 con 0x7fb22c0cf4a0 2023-10-16T13:47:29.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.152+0000 7fb22affd700 1 -- 172.21.15.16:0/1459195816 >> v2:172.21.15.49:3300/0 conn(0x7fb22c0b5830 msgr2=0x7fb22c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:29.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.152+0000 7fb22affd700 1 --2- 172.21.15.16:0/1459195816 >> v2:172.21.15.49:3300/0 conn(0x7fb22c0b5830 0x7fb22c15a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:29.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.152+0000 7fb22affd700 1 -- 172.21.15.16:0/1459195816 >> v1:172.21.15.16:6789/0 conn(0x7fb22c0cf4a0 legacy=0x7fb22c0726e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:29.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.152+0000 7fb22affd700 1 -- 172.21.15.16:0/1459195816 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb22c15b630 con 0x7fb22c0cf880 2023-10-16T13:47:29.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.152+0000 7fb22bfff700 1 --2- 172.21.15.16:0/1459195816 >> v2:172.21.15.49:3300/0 conn(0x7fb22c0b5830 0x7fb22c15a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:47:29.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.152+0000 7fb22affd700 1 --2- 172.21.15.16:0/1459195816 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb22c0cf880 0x7fb22c0b52f0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fb220009840 tx=0x7fb220009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d79a3027b939d135 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:29.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.153+0000 7fb22a7fc700 1 -- 172.21.15.16:0/1459195816 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb21c001670 con 0x7fb22c0cf880 2023-10-16T13:47:29.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.153+0000 7fb22a7fc700 1 -- 172.21.15.16:0/1459195816 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb220004030 con 0x7fb22c0cf880 2023-10-16T13:47:29.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.153+0000 7fb22a7fc700 1 -- 172.21.15.16:0/1459195816 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2200043a0 con 0x7fb22c0cf880 2023-10-16T13:47:29.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.153+0000 7fb231d04700 1 -- 172.21.15.16:0/1459195816 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb22c0cf880 msgr2=0x7fb22c0b52f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:29.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.153+0000 7fb231d04700 1 --2- 172.21.15.16:0/1459195816 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb22c0cf880 0x7fb22c0b52f0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fb220009840 tx=0x7fb220009b50 comp rx=0 tx=0).stop 2023-10-16T13:47:29.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.153+0000 7fb231d04700 1 -- 172.21.15.16:0/1459195816 shutdown_connections 2023-10-16T13:47:29.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.153+0000 7fb231d04700 1 --2- 172.21.15.16:0/1459195816 >> v2:172.21.15.49:3300/0 conn(0x7fb22c0b5830 0x7fb22c15a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:29.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.153+0000 7fb231d04700 1 --2- 172.21.15.16:0/1459195816 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb22c0cf880 0x7fb22c0b52f0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:29.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.153+0000 7fb231d04700 1 -- 172.21.15.16:0/1459195816 >> 172.21.15.16:0/1459195816 conn(0x7fb22c0b1fd0 msgr2=0x7fb22c0b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:29.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.153+0000 7fb231d04700 1 -- 172.21.15.16:0/1459195816 shutdown_connections 2023-10-16T13:47:29.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.153+0000 7fb231d04700 1 -- 172.21.15.16:0/1459195816 wait complete. 2023-10-16T13:47:29.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.154+0000 7fb231d04700 1 Processor -- start 2023-10-16T13:47:29.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.155+0000 7fb231d04700 1 -- start start 2023-10-16T13:47:29.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.155+0000 7fb231d04700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb22c0b5830 0x7fb22c0ad8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:29.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.155+0000 7fb231d04700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb22c0cf4a0 0x7fb22c157690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:29.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.155+0000 7fb231d04700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb22c0cf880 0x7fb22c0aebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:29.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.155+0000 7fb231d04700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb22c06a7e0 con 0x7fb22c0cf4a0 2023-10-16T13:47:29.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.155+0000 7fb231d04700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb22c0ade10 con 0x7fb22c0cf880 2023-10-16T13:47:29.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.155+0000 7fb231d04700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb22c06aa30 con 0x7fb22c0b5830 2023-10-16T13:47:29.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.155+0000 7fb22affd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb22c0cf4a0 0x7fb22c157690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:29.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.156+0000 7fb22affd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb22c0cf4a0 0x7fb22c157690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55510/0 (socket says 172.21.15.16:55510) 2023-10-16T13:47:29.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.156+0000 7fb22affd700 1 -- 172.21.15.16:0/2444418790 learned_addr learned my addr 172.21.15.16:0/2444418790 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:29.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.156+0000 7fb22bfff700 1 --2- 172.21.15.16:0/2444418790 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb22c0cf880 0x7fb22c0aebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:29.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.156+0000 7fb22b7fe700 1 --2- 172.21.15.16:0/2444418790 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb22c0b5830 0x7fb22c0ad8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:29.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.156+0000 7fb22affd700 1 -- 172.21.15.16:0/2444418790 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb22c0b5830 msgr2=0x7fb22c0ad8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:29.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.156+0000 7fb22affd700 1 --2- 172.21.15.16:0/2444418790 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb22c0b5830 0x7fb22c0ad8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:29.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.156+0000 7fb22affd700 1 -- 172.21.15.16:0/2444418790 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb22c0cf880 msgr2=0x7fb22c0aebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:29.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.156+0000 7fb22affd700 1 --2- 172.21.15.16:0/2444418790 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb22c0cf880 0x7fb22c0aebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:29.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.156+0000 7fb22affd700 1 -- 172.21.15.16:0/2444418790 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb224009a70 con 0x7fb22c0cf4a0 2023-10-16T13:47:29.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.156+0000 7fb22bfff700 1 --2- 172.21.15.16:0/2444418790 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb22c0cf880 0x7fb22c0aebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:47:29.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.156+0000 7fb22affd700 1 --2- 172.21.15.16:0/2444418790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb22c0cf4a0 0x7fb22c157690 secure :-1 s=READY pgs=2756 cs=0 l=1 rev1=1 crypto rx=0x7fb220000c00 tx=0x7fb220004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:29.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.156+0000 7fb213fff700 1 -- 172.21.15.16:0/2444418790 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb220003df0 con 0x7fb22c0cf4a0 2023-10-16T13:47:29.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.156+0000 7fb213fff700 1 -- 172.21.15.16:0/2444418790 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb22001f070 con 0x7fb22c0cf4a0 2023-10-16T13:47:29.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.156+0000 7fb231d04700 1 -- 172.21.15.16:0/2444418790 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2200094a0 con 0x7fb22c0cf4a0 2023-10-16T13:47:29.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.156+0000 7fb213fff700 1 -- 172.21.15.16:0/2444418790 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb220011870 con 0x7fb22c0cf4a0 2023-10-16T13:47:29.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.156+0000 7fb231d04700 1 -- 172.21.15.16:0/2444418790 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb22c06aff0 con 0x7fb22c0cf4a0 2023-10-16T13:47:29.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.157+0000 7fb231d04700 1 -- 172.21.15.16:0/2444418790 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb22c04ebb0 con 0x7fb22c0cf4a0 2023-10-16T13:47:29.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.158+0000 7fb213fff700 1 -- 172.21.15.16:0/2444418790 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 148) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fb220021030 con 0x7fb22c0cf4a0 2023-10-16T13:47:29.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.158+0000 7fb213fff700 1 --2- 172.21.15.16:0/2444418790 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fb21806b140 0x7fb21806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:29.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.158+0000 7fb22b7fe700 1 --2- 172.21.15.16:0/2444418790 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fb21806b140 0x7fb21806d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:29.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.158+0000 7fb213fff700 1 -- 172.21.15.16:0/2444418790 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(245..245 src has 1..245) v4 ==== 6157+0+0 (secure 0 0 0) 0x7fb22008af30 con 0x7fb22c0cf4a0 2023-10-16T13:47:29.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.159+0000 7fb22b7fe700 1 --2- 172.21.15.16:0/2444418790 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fb21806b140 0x7fb21806d600 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb21c001eb0 tx=0x7fb21c001c60 comp rx=0 tx=0).ready entity=mgr.11886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:29.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.163+0000 7fb213fff700 1 -- 172.21.15.16:0/2444418790 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb220059700 con 0x7fb22c0cf4a0 2023-10-16T13:47:29.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.362+0000 7fb231d04700 1 -- 172.21.15.16:0/2444418790 --> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb22c15b910 con 0x7fb21806b140 2023-10-16T13:47:29.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.370+0000 7fb213fff700 1 -- 172.21.15.16:0/2444418790 <== mgr.11886 v2:172.21.15.16:6832/1736104576 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+285751 (secure 0 0 0) 0x7fb21c001670 con 0x7fb21806b140 2023-10-16T13:47:29.376 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:47:29.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.373+0000 7fb231d04700 1 -- 172.21.15.16:0/2444418790 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fb21806b140 msgr2=0x7fb21806d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:29.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.373+0000 7fb231d04700 1 --2- 172.21.15.16:0/2444418790 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fb21806b140 0x7fb21806d600 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb21c001eb0 tx=0x7fb21c001c60 comp rx=0 tx=0).stop 2023-10-16T13:47:29.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.373+0000 7fb231d04700 1 -- 172.21.15.16:0/2444418790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb22c0cf4a0 msgr2=0x7fb22c157690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:29.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.373+0000 7fb231d04700 1 --2- 172.21.15.16:0/2444418790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb22c0cf4a0 0x7fb22c157690 secure :-1 s=READY pgs=2756 cs=0 l=1 rev1=1 crypto rx=0x7fb220000c00 tx=0x7fb220004750 comp rx=0 tx=0).stop 2023-10-16T13:47:29.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.373+0000 7fb231d04700 1 -- 172.21.15.16:0/2444418790 shutdown_connections 2023-10-16T13:47:29.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.373+0000 7fb231d04700 1 --2- 172.21.15.16:0/2444418790 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fb21806b140 0x7fb21806d600 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:29.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.373+0000 7fb231d04700 1 --2- 172.21.15.16:0/2444418790 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb22c0cf880 0x7fb22c0aebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:29.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.373+0000 7fb231d04700 1 --2- 172.21.15.16:0/2444418790 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb22c0b5830 0x7fb22c0ad8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:29.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.373+0000 7fb231d04700 1 --2- 172.21.15.16:0/2444418790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb22c0cf4a0 0x7fb22c157690 unknown :-1 s=CLOSED pgs=2756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:29.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.373+0000 7fb231d04700 1 -- 172.21.15.16:0/2444418790 >> 172.21.15.16:0/2444418790 conn(0x7fb22c0b1fd0 msgr2=0x7fb22c15a3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:29.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.373+0000 7fb231d04700 1 -- 172.21.15.16:0/2444418790 shutdown_connections 2023-10-16T13:47:29.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.373+0000 7fb231d04700 1 -- 172.21.15.16:0/2444418790 wait complete. 2023-10-16T13:47:29.381 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-10-16T13:47:29.388 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":30,"stamp":"2023-10-16T13:47:27.821538+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":656,"num_write_kb":2422,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":401,"ondisk_log_size":401,"up":150,"acting":150,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":105,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":431520,"kb_used_data":25208,"kb_used_omap":12,"kb_used_meta":406195,"kb_avail":754543200,"statfs":{"total":773094113280,"available":772652236800,"internally_reserved":0,"allocated":25812992,"data_stored":13882230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12740,"internal_metadata":415944252},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":37,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.416850"},"pg_stats":[{"pgid":"32.1f","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.418005+0000","last_change":"2023-10-16T13:47:24.429332+0000","last_active":"2023-10-16T13:47:25.418005+0000","last_peered":"2023-10-16T13:47:25.418005+0000","last_clean":"2023-10-16T13:47:25.418005+0000","last_became_active":"2023-10-16T13:47:24.428151+0000","last_became_peered":"2023-10-16T13:47:24.428151+0000","last_unstale":"2023-10-16T13:47:25.418005+0000","last_undegraded":"2023-10-16T13:47:25.418005+0000","last_fullsized":"2023-10-16T13:47:25.418005+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:51:49.567279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.1e","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.416620+0000","last_change":"2023-10-16T13:47:24.439610+0000","last_active":"2023-10-16T13:47:25.416620+0000","last_peered":"2023-10-16T13:47:25.416620+0000","last_clean":"2023-10-16T13:47:25.416620+0000","last_became_active":"2023-10-16T13:47:24.439212+0000","last_became_peered":"2023-10-16T13:47:24.439212+0000","last_unstale":"2023-10-16T13:47:25.416620+0000","last_undegraded":"2023-10-16T13:47:25.416620+0000","last_fullsized":"2023-10-16T13:47:25.416620+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:47:24.106272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1d","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100591+0000","last_change":"2023-10-16T13:47:24.432216+0000","last_active":"2023-10-16T13:47:26.100591+0000","last_peered":"2023-10-16T13:47:26.100591+0000","last_clean":"2023-10-16T13:47:26.100591+0000","last_became_active":"2023-10-16T13:47:24.431937+0000","last_became_peered":"2023-10-16T13:47:24.431937+0000","last_unstale":"2023-10-16T13:47:26.100591+0000","last_undegraded":"2023-10-16T13:47:26.100591+0000","last_fullsized":"2023-10-16T13:47:26.100591+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:23:10.580869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1c","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100430+0000","last_change":"2023-10-16T13:47:24.440252+0000","last_active":"2023-10-16T13:47:26.100430+0000","last_peered":"2023-10-16T13:47:26.100430+0000","last_clean":"2023-10-16T13:47:26.100430+0000","last_became_active":"2023-10-16T13:47:24.439982+0000","last_became_peered":"2023-10-16T13:47:24.439982+0000","last_unstale":"2023-10-16T13:47:26.100430+0000","last_undegraded":"2023-10-16T13:47:26.100430+0000","last_fullsized":"2023-10-16T13:47:26.100430+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:54:02.930019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1b","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.101595+0000","last_change":"2023-10-16T13:47:24.440049+0000","last_active":"2023-10-16T13:47:26.101595+0000","last_peered":"2023-10-16T13:47:26.101595+0000","last_clean":"2023-10-16T13:47:26.101595+0000","last_became_active":"2023-10-16T13:47:24.439750+0000","last_became_peered":"2023-10-16T13:47:24.439750+0000","last_unstale":"2023-10-16T13:47:26.101595+0000","last_undegraded":"2023-10-16T13:47:26.101595+0000","last_fullsized":"2023-10-16T13:47:26.101595+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:33:40.094175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1a","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.417711+0000","last_change":"2023-10-16T13:47:24.433661+0000","last_active":"2023-10-16T13:47:25.417711+0000","last_peered":"2023-10-16T13:47:25.417711+0000","last_clean":"2023-10-16T13:47:25.417711+0000","last_became_active":"2023-10-16T13:47:24.433417+0000","last_became_peered":"2023-10-16T13:47:24.433417+0000","last_unstale":"2023-10-16T13:47:25.417711+0000","last_undegraded":"2023-10-16T13:47:25.417711+0000","last_fullsized":"2023-10-16T13:47:25.417711+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:16:02.391378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T13:47:23.404948+0000","last_change":"2023-10-16T13:47:23.404948+0000","last_active":"2023-10-16T13:47:23.404948+0000","last_peered":"2023-10-16T13:47:23.404948+0000","last_clean":"2023-10-16T13:47:23.404948+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:23.404948+0000","last_undegraded":"2023-10-16T13:47:23.404948+0000","last_fullsized":"2023-10-16T13:47:23.404948+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.101429+0000","last_change":"2023-10-16T13:47:24.426442+0000","last_active":"2023-10-16T13:47:26.101429+0000","last_peered":"2023-10-16T13:47:26.101429+0000","last_clean":"2023-10-16T13:47:26.101429+0000","last_became_active":"2023-10-16T13:47:24.425929+0000","last_became_peered":"2023-10-16T13:47:24.425929+0000","last_unstale":"2023-10-16T13:47:26.101429+0000","last_undegraded":"2023-10-16T13:47:26.101429+0000","last_fullsized":"2023-10-16T13:47:26.101429+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:05:13.565304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.418008+0000","last_change":"2023-10-16T13:47:24.426459+0000","last_active":"2023-10-16T13:47:25.418008+0000","last_peered":"2023-10-16T13:47:25.418008+0000","last_clean":"2023-10-16T13:47:25.418008+0000","last_became_active":"2023-10-16T13:47:24.426169+0000","last_became_peered":"2023-10-16T13:47:24.426169+0000","last_unstale":"2023-10-16T13:47:25.418008+0000","last_undegraded":"2023-10-16T13:47:25.418008+0000","last_fullsized":"2023-10-16T13:47:25.418008+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:57:42.170954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100664+0000","last_change":"2023-10-16T13:47:24.427333+0000","last_active":"2023-10-16T13:47:26.100664+0000","last_peered":"2023-10-16T13:47:26.100664+0000","last_clean":"2023-10-16T13:47:26.100664+0000","last_became_active":"2023-10-16T13:47:24.427034+0000","last_became_peered":"2023-10-16T13:47:24.427034+0000","last_unstale":"2023-10-16T13:47:26.100664+0000","last_undegraded":"2023-10-16T13:47:26.100664+0000","last_fullsized":"2023-10-16T13:47:26.100664+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:27:20.181386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.15","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100502+0000","last_change":"2023-10-16T13:47:24.425903+0000","last_active":"2023-10-16T13:47:26.100502+0000","last_peered":"2023-10-16T13:47:26.100502+0000","last_clean":"2023-10-16T13:47:26.100502+0000","last_became_active":"2023-10-16T13:47:24.425593+0000","last_became_peered":"2023-10-16T13:47:24.425593+0000","last_unstale":"2023-10-16T13:47:26.100502+0000","last_undegraded":"2023-10-16T13:47:26.100502+0000","last_fullsized":"2023-10-16T13:47:26.100502+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:46:35.236231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.14","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.417564+0000","last_change":"2023-10-16T13:47:24.438927+0000","last_active":"2023-10-16T13:47:25.417564+0000","last_peered":"2023-10-16T13:47:25.417564+0000","last_clean":"2023-10-16T13:47:25.417564+0000","last_became_active":"2023-10-16T13:47:24.438389+0000","last_became_peered":"2023-10-16T13:47:24.438389+0000","last_unstale":"2023-10-16T13:47:25.417564+0000","last_undegraded":"2023-10-16T13:47:25.417564+0000","last_fullsized":"2023-10-16T13:47:25.417564+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:44:00.360485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T13:47:23.404948+0000","last_change":"2023-10-16T13:47:23.404948+0000","last_active":"2023-10-16T13:47:23.404948+0000","last_peered":"2023-10-16T13:47:23.404948+0000","last_clean":"2023-10-16T13:47:23.404948+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:23.404948+0000","last_undegraded":"2023-10-16T13:47:23.404948+0000","last_fullsized":"2023-10-16T13:47:23.404948+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.099722+0000","last_change":"2023-10-16T13:47:24.437693+0000","last_active":"2023-10-16T13:47:26.099722+0000","last_peered":"2023-10-16T13:47:26.099722+0000","last_clean":"2023-10-16T13:47:26.099722+0000","last_became_active":"2023-10-16T13:47:24.437434+0000","last_became_peered":"2023-10-16T13:47:24.437434+0000","last_unstale":"2023-10-16T13:47:26.099722+0000","last_undegraded":"2023-10-16T13:47:26.099722+0000","last_fullsized":"2023-10-16T13:47:26.099722+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:47:56.221847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T13:47:23.404948+0000","last_change":"2023-10-16T13:47:23.404948+0000","last_active":"2023-10-16T13:47:23.404948+0000","last_peered":"2023-10-16T13:47:23.404948+0000","last_clean":"2023-10-16T13:47:23.404948+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:23.404948+0000","last_undegraded":"2023-10-16T13:47:23.404948+0000","last_fullsized":"2023-10-16T13:47:23.404948+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.10","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.099717+0000","last_change":"2023-10-16T13:47:24.441060+0000","last_active":"2023-10-16T13:47:26.099717+0000","last_peered":"2023-10-16T13:47:26.099717+0000","last_clean":"2023-10-16T13:47:26.099717+0000","last_became_active":"2023-10-16T13:47:24.440745+0000","last_became_peered":"2023-10-16T13:47:24.440745+0000","last_unstale":"2023-10-16T13:47:26.099717+0000","last_undegraded":"2023-10-16T13:47:26.099717+0000","last_fullsized":"2023-10-16T13:47:26.099717+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:48:26.805075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T13:47:23.404948+0000","last_change":"2023-10-16T13:47:23.404948+0000","last_active":"2023-10-16T13:47:23.404948+0000","last_peered":"2023-10-16T13:47:23.404948+0000","last_clean":"2023-10-16T13:47:23.404948+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:23.404948+0000","last_undegraded":"2023-10-16T13:47:23.404948+0000","last_fullsized":"2023-10-16T13:47:23.404948+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.e","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100703+0000","last_change":"2023-10-16T13:47:24.433506+0000","last_active":"2023-10-16T13:47:26.100703+0000","last_peered":"2023-10-16T13:47:26.100703+0000","last_clean":"2023-10-16T13:47:26.100703+0000","last_became_active":"2023-10-16T13:47:24.433218+0000","last_became_peered":"2023-10-16T13:47:24.433218+0000","last_unstale":"2023-10-16T13:47:26.100703+0000","last_undegraded":"2023-10-16T13:47:26.100703+0000","last_fullsized":"2023-10-16T13:47:26.100703+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:04:28.651883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T13:47:23.404948+0000","last_change":"2023-10-16T13:47:23.404948+0000","last_active":"2023-10-16T13:47:23.404948+0000","last_peered":"2023-10-16T13:47:23.404948+0000","last_clean":"2023-10-16T13:47:23.404948+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:23.404948+0000","last_undegraded":"2023-10-16T13:47:23.404948+0000","last_fullsized":"2023-10-16T13:47:23.404948+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.416732+0000","last_change":"2023-10-16T13:47:24.435000+0000","last_active":"2023-10-16T13:47:25.416732+0000","last_peered":"2023-10-16T13:47:25.416732+0000","last_clean":"2023-10-16T13:47:25.416732+0000","last_became_active":"2023-10-16T13:47:24.434568+0000","last_became_peered":"2023-10-16T13:47:24.434568+0000","last_unstale":"2023-10-16T13:47:25.416732+0000","last_undegraded":"2023-10-16T13:47:25.416732+0000","last_fullsized":"2023-10-16T13:47:25.416732+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:31:46.967495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T13:47:23.404948+0000","last_change":"2023-10-16T13:47:23.404948+0000","last_active":"2023-10-16T13:47:23.404948+0000","last_peered":"2023-10-16T13:47:23.404948+0000","last_clean":"2023-10-16T13:47:23.404948+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:23.404948+0000","last_undegraded":"2023-10-16T13:47:23.404948+0000","last_fullsized":"2023-10-16T13:47:23.404948+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T13:47:23.404948+0000","last_change":"2023-10-16T13:47:23.404948+0000","last_active":"2023-10-16T13:47:23.404948+0000","last_peered":"2023-10-16T13:47:23.404948+0000","last_clean":"2023-10-16T13:47:23.404948+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:23.404948+0000","last_undegraded":"2023-10-16T13:47:23.404948+0000","last_fullsized":"2023-10-16T13:47:23.404948+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T13:47:23.404948+0000","last_change":"2023-10-16T13:47:23.404948+0000","last_active":"2023-10-16T13:47:23.404948+0000","last_peered":"2023-10-16T13:47:23.404948+0000","last_clean":"2023-10-16T13:47:23.404948+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:23.404948+0000","last_undegraded":"2023-10-16T13:47:23.404948+0000","last_fullsized":"2023-10-16T13:47:23.404948+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T13:47:23.404948+0000","last_change":"2023-10-16T13:47:23.404948+0000","last_active":"2023-10-16T13:47:23.404948+0000","last_peered":"2023-10-16T13:47:23.404948+0000","last_clean":"2023-10-16T13:47:23.404948+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:23.404948+0000","last_undegraded":"2023-10-16T13:47:23.404948+0000","last_fullsized":"2023-10-16T13:47:23.404948+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.7","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.416641+0000","last_change":"2023-10-16T13:47:24.433031+0000","last_active":"2023-10-16T13:47:25.416641+0000","last_peered":"2023-10-16T13:47:25.416641+0000","last_clean":"2023-10-16T13:47:25.416641+0000","last_became_active":"2023-10-16T13:47:24.432700+0000","last_became_peered":"2023-10-16T13:47:24.432700+0000","last_unstale":"2023-10-16T13:47:25.416641+0000","last_undegraded":"2023-10-16T13:47:25.416641+0000","last_fullsized":"2023-10-16T13:47:25.416641+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:17:10.080681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T13:47:23.404948+0000","last_change":"2023-10-16T13:47:23.404948+0000","last_active":"2023-10-16T13:47:23.404948+0000","last_peered":"2023-10-16T13:47:23.404948+0000","last_clean":"2023-10-16T13:47:23.404948+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:23.404948+0000","last_undegraded":"2023-10-16T13:47:23.404948+0000","last_fullsized":"2023-10-16T13:47:23.404948+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.101557+0000","last_change":"2023-10-16T13:47:24.428801+0000","last_active":"2023-10-16T13:47:26.101557+0000","last_peered":"2023-10-16T13:47:26.101557+0000","last_clean":"2023-10-16T13:47:26.101557+0000","last_became_active":"2023-10-16T13:47:24.428511+0000","last_became_peered":"2023-10-16T13:47:24.428511+0000","last_unstale":"2023-10-16T13:47:26.101557+0000","last_undegraded":"2023-10-16T13:47:26.101557+0000","last_fullsized":"2023-10-16T13:47:26.101557+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:27:55.268379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.417855+0000","last_change":"2023-10-16T13:47:24.429772+0000","last_active":"2023-10-16T13:47:25.417855+0000","last_peered":"2023-10-16T13:47:25.417855+0000","last_clean":"2023-10-16T13:47:25.417855+0000","last_became_active":"2023-10-16T13:47:24.429496+0000","last_became_peered":"2023-10-16T13:47:24.429496+0000","last_unstale":"2023-10-16T13:47:25.417855+0000","last_undegraded":"2023-10-16T13:47:25.417855+0000","last_fullsized":"2023-10-16T13:47:25.417855+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:09:48.751438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.099824+0000","last_change":"2023-10-16T13:47:24.437263+0000","last_active":"2023-10-16T13:47:26.099824+0000","last_peered":"2023-10-16T13:47:26.099824+0000","last_clean":"2023-10-16T13:47:26.099824+0000","last_became_active":"2023-10-16T13:47:24.437023+0000","last_became_peered":"2023-10-16T13:47:24.437023+0000","last_unstale":"2023-10-16T13:47:26.099824+0000","last_undegraded":"2023-10-16T13:47:26.099824+0000","last_fullsized":"2023-10-16T13:47:26.099824+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:59:15.061907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.099775+0000","last_change":"2023-10-16T13:47:24.419847+0000","last_active":"2023-10-16T13:47:26.099775+0000","last_peered":"2023-10-16T13:47:26.099775+0000","last_clean":"2023-10-16T13:47:26.099775+0000","last_became_active":"2023-10-16T13:47:24.419520+0000","last_became_peered":"2023-10-16T13:47:24.419520+0000","last_unstale":"2023-10-16T13:47:26.099775+0000","last_undegraded":"2023-10-16T13:47:26.099775+0000","last_fullsized":"2023-10-16T13:47:26.099775+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:11:44.932011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.416267+0000","last_change":"2023-10-16T13:47:24.435926+0000","last_active":"2023-10-16T13:47:25.416267+0000","last_peered":"2023-10-16T13:47:25.416267+0000","last_clean":"2023-10-16T13:47:25.416267+0000","last_became_active":"2023-10-16T13:47:24.434753+0000","last_became_peered":"2023-10-16T13:47:24.434753+0000","last_unstale":"2023-10-16T13:47:25.416267+0000","last_undegraded":"2023-10-16T13:47:25.416267+0000","last_fullsized":"2023-10-16T13:47:25.416267+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:13:51.332934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.417066+0000","last_change":"2023-10-16T13:47:24.423908+0000","last_active":"2023-10-16T13:47:25.417066+0000","last_peered":"2023-10-16T13:47:25.417066+0000","last_clean":"2023-10-16T13:47:25.417066+0000","last_became_active":"2023-10-16T13:47:24.423530+0000","last_became_peered":"2023-10-16T13:47:24.423530+0000","last_unstale":"2023-10-16T13:47:25.417066+0000","last_undegraded":"2023-10-16T13:47:25.417066+0000","last_fullsized":"2023-10-16T13:47:25.417066+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:50:50.396650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.20","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.416162+0000","last_change":"2023-10-16T13:47:24.436480+0000","last_active":"2023-10-16T13:47:25.416162+0000","last_peered":"2023-10-16T13:47:25.416162+0000","last_clean":"2023-10-16T13:47:25.416162+0000","last_became_active":"2023-10-16T13:47:24.435309+0000","last_became_peered":"2023-10-16T13:47:24.435309+0000","last_unstale":"2023-10-16T13:47:25.416162+0000","last_undegraded":"2023-10-16T13:47:25.416162+0000","last_fullsized":"2023-10-16T13:47:25.416162+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:28:08.947506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1f","version":"244'4","reported_seq":16,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.452436+0000","last_change":"2023-10-16T13:47:23.425349+0000","last_active":"2023-10-16T13:47:24.452436+0000","last_peered":"2023-10-16T13:47:24.452436+0000","last_clean":"2023-10-16T13:47:24.452436+0000","last_became_active":"2023-10-16T13:47:23.425029+0000","last_became_peered":"2023-10-16T13:47:23.425029+0000","last_unstale":"2023-10-16T13:47:24.452436+0000","last_undegraded":"2023-10-16T13:47:24.452436+0000","last_fullsized":"2023-10-16T13:47:24.452436+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:57:35.834032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T13:47:23.404948+0000","last_change":"2023-10-16T13:47:23.404948+0000","last_active":"2023-10-16T13:47:23.404948+0000","last_peered":"2023-10-16T13:47:23.404948+0000","last_clean":"2023-10-16T13:47:23.404948+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:23.404948+0000","last_undegraded":"2023-10-16T13:47:23.404948+0000","last_fullsized":"2023-10-16T13:47:23.404948+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.1d","version":"0'0","reported_seq":1,"reported_epoch":242,"state":"creating+peering","last_fresh":"2023-10-16T13:47:22.414037+0000","last_change":"2023-10-16T13:47:22.414037+0000","last_active":"2023-10-16T13:47:22.400586+0000","last_peered":"2023-10-16T13:47:22.400586+0000","last_clean":"2023-10-16T13:47:22.400586+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:22.414037+0000","last_undegraded":"2023-10-16T13:47:22.414037+0000","last_fullsized":"2023-10-16T13:47:22.414037+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.23","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.417464+0000","last_change":"2023-10-16T13:47:24.432436+0000","last_active":"2023-10-16T13:47:25.417464+0000","last_peered":"2023-10-16T13:47:25.417464+0000","last_clean":"2023-10-16T13:47:25.417464+0000","last_became_active":"2023-10-16T13:47:24.428465+0000","last_became_peered":"2023-10-16T13:47:24.428465+0000","last_unstale":"2023-10-16T13:47:25.417464+0000","last_undegraded":"2023-10-16T13:47:25.417464+0000","last_fullsized":"2023-10-16T13:47:25.417464+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:16:50.560075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1c","version":"244'2","reported_seq":14,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.447290+0000","last_change":"2023-10-16T13:47:23.427686+0000","last_active":"2023-10-16T13:47:24.447290+0000","last_peered":"2023-10-16T13:47:24.447290+0000","last_clean":"2023-10-16T13:47:24.447290+0000","last_became_active":"2023-10-16T13:47:23.427013+0000","last_became_peered":"2023-10-16T13:47:23.427013+0000","last_unstale":"2023-10-16T13:47:24.447290+0000","last_undegraded":"2023-10-16T13:47:24.447290+0000","last_fullsized":"2023-10-16T13:47:24.447290+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:37:17.532164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.24","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100065+0000","last_change":"2023-10-16T13:47:24.438644+0000","last_active":"2023-10-16T13:47:26.100065+0000","last_peered":"2023-10-16T13:47:26.100065+0000","last_clean":"2023-10-16T13:47:26.100065+0000","last_became_active":"2023-10-16T13:47:24.438344+0000","last_became_peered":"2023-10-16T13:47:24.438344+0000","last_unstale":"2023-10-16T13:47:26.100065+0000","last_undegraded":"2023-10-16T13:47:26.100065+0000","last_fullsized":"2023-10-16T13:47:26.100065+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:21:19.269920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.1b","version":"0'0","reported_seq":1,"reported_epoch":242,"state":"creating+peering","last_fresh":"2023-10-16T13:47:22.413990+0000","last_change":"2023-10-16T13:47:22.413990+0000","last_active":"2023-10-16T13:47:22.400586+0000","last_peered":"2023-10-16T13:47:22.400586+0000","last_clean":"2023-10-16T13:47:22.400586+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:22.413990+0000","last_undegraded":"2023-10-16T13:47:22.413990+0000","last_fullsized":"2023-10-16T13:47:22.413990+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T13:47:23.404948+0000","last_change":"2023-10-16T13:47:23.404948+0000","last_active":"2023-10-16T13:47:23.404948+0000","last_peered":"2023-10-16T13:47:23.404948+0000","last_clean":"2023-10-16T13:47:23.404948+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:23.404948+0000","last_undegraded":"2023-10-16T13:47:23.404948+0000","last_fullsized":"2023-10-16T13:47:23.404948+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.1a","version":"244'1","reported_seq":13,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.441041+0000","last_change":"2023-10-16T13:47:23.425695+0000","last_active":"2023-10-16T13:47:24.441041+0000","last_peered":"2023-10-16T13:47:24.441041+0000","last_clean":"2023-10-16T13:47:24.441041+0000","last_became_active":"2023-10-16T13:47:23.425407+0000","last_became_peered":"2023-10-16T13:47:23.425407+0000","last_unstale":"2023-10-16T13:47:24.441041+0000","last_undegraded":"2023-10-16T13:47:24.441041+0000","last_fullsized":"2023-10-16T13:47:24.441041+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:19:34.313200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.26","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100538+0000","last_change":"2023-10-16T13:47:24.439616+0000","last_active":"2023-10-16T13:47:26.100538+0000","last_peered":"2023-10-16T13:47:26.100538+0000","last_clean":"2023-10-16T13:47:26.100538+0000","last_became_active":"2023-10-16T13:47:24.435678+0000","last_became_peered":"2023-10-16T13:47:24.435678+0000","last_unstale":"2023-10-16T13:47:26.100538+0000","last_undegraded":"2023-10-16T13:47:26.100538+0000","last_fullsized":"2023-10-16T13:47:26.100538+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:39:16.296666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.418484+0000","last_change":"2023-10-16T13:47:23.431681+0000","last_active":"2023-10-16T13:47:24.418484+0000","last_peered":"2023-10-16T13:47:24.418484+0000","last_clean":"2023-10-16T13:47:24.418484+0000","last_became_active":"2023-10-16T13:47:23.431317+0000","last_became_peered":"2023-10-16T13:47:23.431317+0000","last_unstale":"2023-10-16T13:47:24.418484+0000","last_undegraded":"2023-10-16T13:47:24.418484+0000","last_fullsized":"2023-10-16T13:47:24.418484+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:20:27.852907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.27","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.101829+0000","last_change":"2023-10-16T13:47:24.423771+0000","last_active":"2023-10-16T13:47:26.101829+0000","last_peered":"2023-10-16T13:47:26.101829+0000","last_clean":"2023-10-16T13:47:26.101829+0000","last_became_active":"2023-10-16T13:47:24.423459+0000","last_became_peered":"2023-10-16T13:47:24.423459+0000","last_unstale":"2023-10-16T13:47:26.101829+0000","last_undegraded":"2023-10-16T13:47:26.101829+0000","last_fullsized":"2023-10-16T13:47:26.101829+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:27:22.393558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.421767+0000","last_change":"2023-10-16T13:47:23.433310+0000","last_active":"2023-10-16T13:47:24.421767+0000","last_peered":"2023-10-16T13:47:24.421767+0000","last_clean":"2023-10-16T13:47:24.421767+0000","last_became_active":"2023-10-16T13:47:23.433018+0000","last_became_peered":"2023-10-16T13:47:23.433018+0000","last_unstale":"2023-10-16T13:47:24.421767+0000","last_undegraded":"2023-10-16T13:47:24.421767+0000","last_fullsized":"2023-10-16T13:47:24.421767+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:49:45.992161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T13:47:23.404948+0000","last_change":"2023-10-16T13:47:23.404948+0000","last_active":"2023-10-16T13:47:23.404948+0000","last_peered":"2023-10-16T13:47:23.404948+0000","last_clean":"2023-10-16T13:47:23.404948+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:23.404948+0000","last_undegraded":"2023-10-16T13:47:23.404948+0000","last_fullsized":"2023-10-16T13:47:23.404948+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.415527+0000","last_change":"2023-10-16T13:47:23.428721+0000","last_active":"2023-10-16T13:47:24.415527+0000","last_peered":"2023-10-16T13:47:24.415527+0000","last_clean":"2023-10-16T13:47:24.415527+0000","last_became_active":"2023-10-16T13:47:23.428349+0000","last_became_peered":"2023-10-16T13:47:23.428349+0000","last_unstale":"2023-10-16T13:47:24.415527+0000","last_undegraded":"2023-10-16T13:47:24.415527+0000","last_fullsized":"2023-10-16T13:47:24.415527+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:59:11.116638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T13:47:23.404948+0000","last_change":"2023-10-16T13:47:23.404948+0000","last_active":"2023-10-16T13:47:23.404948+0000","last_peered":"2023-10-16T13:47:23.404948+0000","last_clean":"2023-10-16T13:47:23.404948+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:23.404948+0000","last_undegraded":"2023-10-16T13:47:23.404948+0000","last_fullsized":"2023-10-16T13:47:23.404948+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.421423+0000","last_change":"2023-10-16T13:47:23.433545+0000","last_active":"2023-10-16T13:47:24.421423+0000","last_peered":"2023-10-16T13:47:24.421423+0000","last_clean":"2023-10-16T13:47:24.421423+0000","last_became_active":"2023-10-16T13:47:23.433275+0000","last_became_peered":"2023-10-16T13:47:23.433275+0000","last_unstale":"2023-10-16T13:47:24.421423+0000","last_undegraded":"2023-10-16T13:47:24.421423+0000","last_fullsized":"2023-10-16T13:47:24.421423+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:21:17.525717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.2a","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100402+0000","last_change":"2023-10-16T13:47:24.424605+0000","last_active":"2023-10-16T13:47:26.100402+0000","last_peered":"2023-10-16T13:47:26.100402+0000","last_clean":"2023-10-16T13:47:26.100402+0000","last_became_active":"2023-10-16T13:47:24.424313+0000","last_became_peered":"2023-10-16T13:47:24.424313+0000","last_unstale":"2023-10-16T13:47:26.100402+0000","last_undegraded":"2023-10-16T13:47:26.100402+0000","last_fullsized":"2023-10-16T13:47:26.100402+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:40:33.368782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.15","version":"244'3","reported_seq":15,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.446109+0000","last_change":"2023-10-16T13:47:23.431025+0000","last_active":"2023-10-16T13:47:24.446109+0000","last_peered":"2023-10-16T13:47:24.446109+0000","last_clean":"2023-10-16T13:47:24.446109+0000","last_became_active":"2023-10-16T13:47:23.430710+0000","last_became_peered":"2023-10-16T13:47:23.430710+0000","last_unstale":"2023-10-16T13:47:24.446109+0000","last_undegraded":"2023-10-16T13:47:24.446109+0000","last_fullsized":"2023-10-16T13:47:24.446109+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:52:27.207432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T13:47:23.404948+0000","last_change":"2023-10-16T13:47:23.404948+0000","last_active":"2023-10-16T13:47:23.404948+0000","last_peered":"2023-10-16T13:47:23.404948+0000","last_clean":"2023-10-16T13:47:23.404948+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:23.404948+0000","last_undegraded":"2023-10-16T13:47:23.404948+0000","last_fullsized":"2023-10-16T13:47:23.404948+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.14","version":"244'6","reported_seq":18,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.453158+0000","last_change":"2023-10-16T13:47:23.430528+0000","last_active":"2023-10-16T13:47:24.453158+0000","last_peered":"2023-10-16T13:47:24.453158+0000","last_clean":"2023-10-16T13:47:24.453158+0000","last_became_active":"2023-10-16T13:47:23.429721+0000","last_became_peered":"2023-10-16T13:47:23.429721+0000","last_unstale":"2023-10-16T13:47:24.453158+0000","last_undegraded":"2023-10-16T13:47:24.453158+0000","last_fullsized":"2023-10-16T13:47:24.453158+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:07:56.379842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.2c","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.416579+0000","last_change":"2023-10-16T13:47:24.436132+0000","last_active":"2023-10-16T13:47:25.416579+0000","last_peered":"2023-10-16T13:47:25.416579+0000","last_clean":"2023-10-16T13:47:25.416579+0000","last_became_active":"2023-10-16T13:47:24.435272+0000","last_became_peered":"2023-10-16T13:47:24.435272+0000","last_unstale":"2023-10-16T13:47:25.416579+0000","last_undegraded":"2023-10-16T13:47:25.416579+0000","last_fullsized":"2023-10-16T13:47:25.416579+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:07:59.434149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.13","version":"0'0","reported_seq":1,"reported_epoch":242,"state":"creating+peering","last_fresh":"2023-10-16T13:47:22.412143+0000","last_change":"2023-10-16T13:47:22.412143+0000","last_active":"2023-10-16T13:47:22.400586+0000","last_peered":"2023-10-16T13:47:22.400586+0000","last_clean":"2023-10-16T13:47:22.400586+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:22.412143+0000","last_undegraded":"2023-10-16T13:47:22.412143+0000","last_fullsized":"2023-10-16T13:47:22.412143+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T13:47:23.404948+0000","last_change":"2023-10-16T13:47:23.404948+0000","last_active":"2023-10-16T13:47:23.404948+0000","last_peered":"2023-10-16T13:47:23.404948+0000","last_clean":"2023-10-16T13:47:23.404948+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:23.404948+0000","last_undegraded":"2023-10-16T13:47:23.404948+0000","last_fullsized":"2023-10-16T13:47:23.404948+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.418559+0000","last_change":"2023-10-16T13:47:23.433222+0000","last_active":"2023-10-16T13:47:24.418559+0000","last_peered":"2023-10-16T13:47:24.418559+0000","last_clean":"2023-10-16T13:47:24.418559+0000","last_became_active":"2023-10-16T13:47:23.432936+0000","last_became_peered":"2023-10-16T13:47:23.432936+0000","last_unstale":"2023-10-16T13:47:24.418559+0000","last_undegraded":"2023-10-16T13:47:24.418559+0000","last_fullsized":"2023-10-16T13:47:24.418559+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:47:59.336633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T13:47:23.404948+0000","last_change":"2023-10-16T13:47:23.404948+0000","last_active":"2023-10-16T13:47:23.404948+0000","last_peered":"2023-10-16T13:47:23.404948+0000","last_clean":"2023-10-16T13:47:23.404948+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:23.404948+0000","last_undegraded":"2023-10-16T13:47:23.404948+0000","last_fullsized":"2023-10-16T13:47:23.404948+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.417965+0000","last_change":"2023-10-16T13:47:23.425872+0000","last_active":"2023-10-16T13:47:24.417965+0000","last_peered":"2023-10-16T13:47:24.417965+0000","last_clean":"2023-10-16T13:47:24.417965+0000","last_became_active":"2023-10-16T13:47:23.425569+0000","last_became_peered":"2023-10-16T13:47:23.425569+0000","last_unstale":"2023-10-16T13:47:24.417965+0000","last_undegraded":"2023-10-16T13:47:24.417965+0000","last_fullsized":"2023-10-16T13:47:24.417965+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:09:18.086723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2f","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.101809+0000","last_change":"2023-10-16T13:47:24.437030+0000","last_active":"2023-10-16T13:47:26.101809+0000","last_peered":"2023-10-16T13:47:26.101809+0000","last_clean":"2023-10-16T13:47:26.101809+0000","last_became_active":"2023-10-16T13:47:24.436734+0000","last_became_peered":"2023-10-16T13:47:24.436734+0000","last_unstale":"2023-10-16T13:47:26.101809+0000","last_undegraded":"2023-10-16T13:47:26.101809+0000","last_fullsized":"2023-10-16T13:47:26.101809+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:28:53.363293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.10","version":"244'8","reported_seq":20,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.453887+0000","last_change":"2023-10-16T13:47:23.426967+0000","last_active":"2023-10-16T13:47:24.453887+0000","last_peered":"2023-10-16T13:47:24.453887+0000","last_clean":"2023-10-16T13:47:24.453887+0000","last_became_active":"2023-10-16T13:47:23.426614+0000","last_became_peered":"2023-10-16T13:47:23.426614+0000","last_unstale":"2023-10-16T13:47:24.453887+0000","last_undegraded":"2023-10-16T13:47:24.453887+0000","last_fullsized":"2023-10-16T13:47:24.453887+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:39:12.211841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.30","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.101224+0000","last_change":"2023-10-16T13:47:24.437081+0000","last_active":"2023-10-16T13:47:26.101224+0000","last_peered":"2023-10-16T13:47:26.101224+0000","last_clean":"2023-10-16T13:47:26.101224+0000","last_became_active":"2023-10-16T13:47:24.436501+0000","last_became_peered":"2023-10-16T13:47:24.436501+0000","last_unstale":"2023-10-16T13:47:26.101224+0000","last_undegraded":"2023-10-16T13:47:26.101224+0000","last_fullsized":"2023-10-16T13:47:26.101224+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:23:30.831964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.f","version":"244'1","reported_seq":13,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.445592+0000","last_change":"2023-10-16T13:47:23.432161+0000","last_active":"2023-10-16T13:47:24.445592+0000","last_peered":"2023-10-16T13:47:24.445592+0000","last_clean":"2023-10-16T13:47:24.445592+0000","last_became_active":"2023-10-16T13:47:23.431822+0000","last_became_peered":"2023-10-16T13:47:23.431822+0000","last_unstale":"2023-10-16T13:47:24.445592+0000","last_undegraded":"2023-10-16T13:47:24.445592+0000","last_fullsized":"2023-10-16T13:47:24.445592+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:15:50.867051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.31","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100923+0000","last_change":"2023-10-16T13:47:24.431186+0000","last_active":"2023-10-16T13:47:26.100923+0000","last_peered":"2023-10-16T13:47:26.100923+0000","last_clean":"2023-10-16T13:47:26.100923+0000","last_became_active":"2023-10-16T13:47:24.430892+0000","last_became_peered":"2023-10-16T13:47:24.430892+0000","last_unstale":"2023-10-16T13:47:26.100923+0000","last_undegraded":"2023-10-16T13:47:26.100923+0000","last_fullsized":"2023-10-16T13:47:26.100923+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:12:21.693658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.e","version":"244'2","reported_seq":14,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.448939+0000","last_change":"2023-10-16T13:47:23.424549+0000","last_active":"2023-10-16T13:47:24.448939+0000","last_peered":"2023-10-16T13:47:24.448939+0000","last_clean":"2023-10-16T13:47:24.448939+0000","last_became_active":"2023-10-16T13:47:23.424111+0000","last_became_peered":"2023-10-16T13:47:23.424111+0000","last_unstale":"2023-10-16T13:47:24.448939+0000","last_undegraded":"2023-10-16T13:47:24.448939+0000","last_fullsized":"2023-10-16T13:47:24.448939+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:36:35.137200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.32","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.416229+0000","last_change":"2023-10-16T13:47:24.432940+0000","last_active":"2023-10-16T13:47:25.416229+0000","last_peered":"2023-10-16T13:47:25.416229+0000","last_clean":"2023-10-16T13:47:25.416229+0000","last_became_active":"2023-10-16T13:47:24.432610+0000","last_became_peered":"2023-10-16T13:47:24.432610+0000","last_unstale":"2023-10-16T13:47:25.416229+0000","last_undegraded":"2023-10-16T13:47:25.416229+0000","last_fullsized":"2023-10-16T13:47:25.416229+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:59:08.104923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.d","version":"0'0","reported_seq":1,"reported_epoch":242,"state":"creating+peering","last_fresh":"2023-10-16T13:47:22.413572+0000","last_change":"2023-10-16T13:47:22.413572+0000","last_active":"2023-10-16T13:47:22.400586+0000","last_peered":"2023-10-16T13:47:22.400586+0000","last_clean":"2023-10-16T13:47:22.400586+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:22.413572+0000","last_undegraded":"2023-10-16T13:47:22.413572+0000","last_fullsized":"2023-10-16T13:47:22.413572+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.33","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100188+0000","last_change":"2023-10-16T13:47:24.427239+0000","last_active":"2023-10-16T13:47:26.100188+0000","last_peered":"2023-10-16T13:47:26.100188+0000","last_clean":"2023-10-16T13:47:26.100188+0000","last_became_active":"2023-10-16T13:47:24.426919+0000","last_became_peered":"2023-10-16T13:47:24.426919+0000","last_unstale":"2023-10-16T13:47:26.100188+0000","last_undegraded":"2023-10-16T13:47:26.100188+0000","last_fullsized":"2023-10-16T13:47:26.100188+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:56:16.580840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":1,"reported_epoch":242,"state":"creating+peering","last_fresh":"2023-10-16T13:47:22.412157+0000","last_change":"2023-10-16T13:47:22.412157+0000","last_active":"2023-10-16T13:47:22.400586+0000","last_peered":"2023-10-16T13:47:22.400586+0000","last_clean":"2023-10-16T13:47:22.400586+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:22.412157+0000","last_undegraded":"2023-10-16T13:47:22.412157+0000","last_fullsized":"2023-10-16T13:47:22.412157+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.34","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100514+0000","last_change":"2023-10-16T13:47:24.444364+0000","last_active":"2023-10-16T13:47:26.100514+0000","last_peered":"2023-10-16T13:47:26.100514+0000","last_clean":"2023-10-16T13:47:26.100514+0000","last_became_active":"2023-10-16T13:47:24.444045+0000","last_became_peered":"2023-10-16T13:47:24.444045+0000","last_unstale":"2023-10-16T13:47:26.100514+0000","last_undegraded":"2023-10-16T13:47:26.100514+0000","last_fullsized":"2023-10-16T13:47:26.100514+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:13:11.795921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.b","version":"244'2","reported_seq":14,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.449802+0000","last_change":"2023-10-16T13:47:23.434957+0000","last_active":"2023-10-16T13:47:24.449802+0000","last_peered":"2023-10-16T13:47:24.449802+0000","last_clean":"2023-10-16T13:47:24.449802+0000","last_became_active":"2023-10-16T13:47:23.434637+0000","last_became_peered":"2023-10-16T13:47:23.434637+0000","last_unstale":"2023-10-16T13:47:24.449802+0000","last_undegraded":"2023-10-16T13:47:24.449802+0000","last_fullsized":"2023-10-16T13:47:24.449802+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:23:47.081423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.35","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.416431+0000","last_change":"2023-10-16T13:47:24.431892+0000","last_active":"2023-10-16T13:47:25.416431+0000","last_peered":"2023-10-16T13:47:25.416431+0000","last_clean":"2023-10-16T13:47:25.416431+0000","last_became_active":"2023-10-16T13:47:24.431624+0000","last_became_peered":"2023-10-16T13:47:24.431624+0000","last_unstale":"2023-10-16T13:47:25.416431+0000","last_undegraded":"2023-10-16T13:47:25.416431+0000","last_fullsized":"2023-10-16T13:47:25.416431+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:49:13.422163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":1,"reported_epoch":242,"state":"creating+peering","last_fresh":"2023-10-16T13:47:22.412844+0000","last_change":"2023-10-16T13:47:22.412844+0000","last_active":"2023-10-16T13:47:22.400586+0000","last_peered":"2023-10-16T13:47:22.400586+0000","last_clean":"2023-10-16T13:47:22.400586+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:22.412844+0000","last_undegraded":"2023-10-16T13:47:22.412844+0000","last_fullsized":"2023-10-16T13:47:22.412844+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T13:47:23.404948+0000","last_change":"2023-10-16T13:47:23.404948+0000","last_active":"2023-10-16T13:47:23.404948+0000","last_peered":"2023-10-16T13:47:23.404948+0000","last_clean":"2023-10-16T13:47:23.404948+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:23.404948+0000","last_undegraded":"2023-10-16T13:47:23.404948+0000","last_fullsized":"2023-10-16T13:47:23.404948+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":1,"reported_epoch":242,"state":"creating+peering","last_fresh":"2023-10-16T13:47:22.412735+0000","last_change":"2023-10-16T13:47:22.412735+0000","last_active":"2023-10-16T13:47:22.400586+0000","last_peered":"2023-10-16T13:47:22.400586+0000","last_clean":"2023-10-16T13:47:22.400586+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:22.412735+0000","last_undegraded":"2023-10-16T13:47:22.412735+0000","last_fullsized":"2023-10-16T13:47:22.412735+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T13:47:23.404948+0000","last_change":"2023-10-16T13:47:23.404948+0000","last_active":"2023-10-16T13:47:23.404948+0000","last_peered":"2023-10-16T13:47:23.404948+0000","last_clean":"2023-10-16T13:47:23.404948+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:23.404948+0000","last_undegraded":"2023-10-16T13:47:23.404948+0000","last_fullsized":"2023-10-16T13:47:23.404948+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.415241+0000","last_change":"2023-10-16T13:47:23.433223+0000","last_active":"2023-10-16T13:47:24.415241+0000","last_peered":"2023-10-16T13:47:24.415241+0000","last_clean":"2023-10-16T13:47:24.415241+0000","last_became_active":"2023-10-16T13:47:23.432935+0000","last_became_peered":"2023-10-16T13:47:23.432935+0000","last_unstale":"2023-10-16T13:47:24.415241+0000","last_undegraded":"2023-10-16T13:47:24.415241+0000","last_fullsized":"2023-10-16T13:47:24.415241+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:40:59.034875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T13:47:23.404948+0000","last_change":"2023-10-16T13:47:23.404948+0000","last_active":"2023-10-16T13:47:23.404948+0000","last_peered":"2023-10-16T13:47:23.404948+0000","last_clean":"2023-10-16T13:47:23.404948+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:23.404948+0000","last_undegraded":"2023-10-16T13:47:23.404948+0000","last_fullsized":"2023-10-16T13:47:23.404948+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.7","version":"244'2","reported_seq":14,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.451763+0000","last_change":"2023-10-16T13:47:23.430865+0000","last_active":"2023-10-16T13:47:24.451763+0000","last_peered":"2023-10-16T13:47:24.451763+0000","last_clean":"2023-10-16T13:47:24.451763+0000","last_became_active":"2023-10-16T13:47:23.430493+0000","last_became_peered":"2023-10-16T13:47:23.430493+0000","last_unstale":"2023-10-16T13:47:24.451763+0000","last_undegraded":"2023-10-16T13:47:24.451763+0000","last_fullsized":"2023-10-16T13:47:24.451763+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:59:14.057546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T13:47:23.404948+0000","last_change":"2023-10-16T13:47:23.404948+0000","last_active":"2023-10-16T13:47:23.404948+0000","last_peered":"2023-10-16T13:47:23.404948+0000","last_clean":"2023-10-16T13:47:23.404948+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:23.404948+0000","last_undegraded":"2023-10-16T13:47:23.404948+0000","last_fullsized":"2023-10-16T13:47:23.404948+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.417912+0000","last_change":"2023-10-16T13:47:23.430851+0000","last_active":"2023-10-16T13:47:24.417912+0000","last_peered":"2023-10-16T13:47:24.417912+0000","last_clean":"2023-10-16T13:47:24.417912+0000","last_became_active":"2023-10-16T13:47:23.430526+0000","last_became_peered":"2023-10-16T13:47:23.430526+0000","last_unstale":"2023-10-16T13:47:24.417912+0000","last_undegraded":"2023-10-16T13:47:24.417912+0000","last_fullsized":"2023-10-16T13:47:24.417912+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:40:16.878541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.3a","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100155+0000","last_change":"2023-10-16T13:47:24.440347+0000","last_active":"2023-10-16T13:47:26.100155+0000","last_peered":"2023-10-16T13:47:26.100155+0000","last_clean":"2023-10-16T13:47:26.100155+0000","last_became_active":"2023-10-16T13:47:24.440041+0000","last_became_peered":"2023-10-16T13:47:24.440041+0000","last_unstale":"2023-10-16T13:47:26.100155+0000","last_undegraded":"2023-10-16T13:47:26.100155+0000","last_fullsized":"2023-10-16T13:47:26.100155+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:19:38.293171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.415325+0000","last_change":"2023-10-16T13:47:23.427399+0000","last_active":"2023-10-16T13:47:24.415325+0000","last_peered":"2023-10-16T13:47:24.415325+0000","last_clean":"2023-10-16T13:47:24.415325+0000","last_became_active":"2023-10-16T13:47:23.427092+0000","last_became_peered":"2023-10-16T13:47:23.427092+0000","last_unstale":"2023-10-16T13:47:24.415325+0000","last_undegraded":"2023-10-16T13:47:24.415325+0000","last_fullsized":"2023-10-16T13:47:24.415325+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:10:56.739817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3b","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.101556+0000","last_change":"2023-10-16T13:47:24.436718+0000","last_active":"2023-10-16T13:47:26.101556+0000","last_peered":"2023-10-16T13:47:26.101556+0000","last_clean":"2023-10-16T13:47:26.101556+0000","last_became_active":"2023-10-16T13:47:24.436371+0000","last_became_peered":"2023-10-16T13:47:24.436371+0000","last_unstale":"2023-10-16T13:47:26.101556+0000","last_undegraded":"2023-10-16T13:47:26.101556+0000","last_fullsized":"2023-10-16T13:47:26.101556+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:58:12.134727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.416043+0000","last_change":"2023-10-16T13:47:23.433833+0000","last_active":"2023-10-16T13:47:24.416043+0000","last_peered":"2023-10-16T13:47:24.416043+0000","last_clean":"2023-10-16T13:47:24.416043+0000","last_became_active":"2023-10-16T13:47:23.433381+0000","last_became_peered":"2023-10-16T13:47:23.433381+0000","last_unstale":"2023-10-16T13:47:24.416043+0000","last_undegraded":"2023-10-16T13:47:24.416043+0000","last_fullsized":"2023-10-16T13:47:24.416043+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:09:34.543970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3c","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.099990+0000","last_change":"2023-10-16T13:47:24.424889+0000","last_active":"2023-10-16T13:47:26.099990+0000","last_peered":"2023-10-16T13:47:26.099990+0000","last_clean":"2023-10-16T13:47:26.099990+0000","last_became_active":"2023-10-16T13:47:24.424573+0000","last_became_peered":"2023-10-16T13:47:24.424573+0000","last_unstale":"2023-10-16T13:47:26.099990+0000","last_undegraded":"2023-10-16T13:47:26.099990+0000","last_fullsized":"2023-10-16T13:47:26.099990+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:23:40.809515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":1,"reported_epoch":242,"state":"creating+peering","last_fresh":"2023-10-16T13:47:22.413180+0000","last_change":"2023-10-16T13:47:22.413180+0000","last_active":"2023-10-16T13:47:22.400586+0000","last_peered":"2023-10-16T13:47:22.400586+0000","last_clean":"2023-10-16T13:47:22.400586+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:22.413180+0000","last_undegraded":"2023-10-16T13:47:22.413180+0000","last_fullsized":"2023-10-16T13:47:22.413180+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.3d","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.417707+0000","last_change":"2023-10-16T13:47:24.430727+0000","last_active":"2023-10-16T13:47:25.417707+0000","last_peered":"2023-10-16T13:47:25.417707+0000","last_clean":"2023-10-16T13:47:25.417707+0000","last_became_active":"2023-10-16T13:47:24.430424+0000","last_became_peered":"2023-10-16T13:47:24.430424+0000","last_unstale":"2023-10-16T13:47:25.417707+0000","last_undegraded":"2023-10-16T13:47:25.417707+0000","last_fullsized":"2023-10-16T13:47:25.417707+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:05:49.279256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.418627+0000","last_change":"2023-10-16T13:47:23.430779+0000","last_active":"2023-10-16T13:47:24.418627+0000","last_peered":"2023-10-16T13:47:24.418627+0000","last_clean":"2023-10-16T13:47:24.418627+0000","last_became_active":"2023-10-16T13:47:23.429996+0000","last_became_peered":"2023-10-16T13:47:23.429996+0000","last_unstale":"2023-10-16T13:47:24.418627+0000","last_undegraded":"2023-10-16T13:47:24.418627+0000","last_fullsized":"2023-10-16T13:47:24.418627+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:17:22.079776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.3e","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.416486+0000","last_change":"2023-10-16T13:47:24.436378+0000","last_active":"2023-10-16T13:47:25.416486+0000","last_peered":"2023-10-16T13:47:25.416486+0000","last_clean":"2023-10-16T13:47:25.416486+0000","last_became_active":"2023-10-16T13:47:24.435495+0000","last_became_peered":"2023-10-16T13:47:24.435495+0000","last_unstale":"2023-10-16T13:47:25.416486+0000","last_undegraded":"2023-10-16T13:47:25.416486+0000","last_fullsized":"2023-10-16T13:47:25.416486+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:18:23.073022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.420122+0000","last_change":"2023-10-16T13:47:23.433945+0000","last_active":"2023-10-16T13:47:24.420122+0000","last_peered":"2023-10-16T13:47:24.420122+0000","last_clean":"2023-10-16T13:47:24.420122+0000","last_became_active":"2023-10-16T13:47:23.433576+0000","last_became_peered":"2023-10-16T13:47:23.433576+0000","last_unstale":"2023-10-16T13:47:24.420122+0000","last_undegraded":"2023-10-16T13:47:24.420122+0000","last_fullsized":"2023-10-16T13:47:24.420122+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:09:34.543970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-16T13:47:23.404948+0000","last_change":"2023-10-16T13:47:23.404948+0000","last_active":"2023-10-16T13:47:23.404948+0000","last_peered":"2023-10-16T13:47:23.404948+0000","last_clean":"2023-10-16T13:47:23.404948+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:23.404948+0000","last_undegraded":"2023-10-16T13:47:23.404948+0000","last_fullsized":"2023-10-16T13:47:23.404948+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":1,"reported_epoch":242,"state":"creating+peering","last_fresh":"2023-10-16T13:47:22.412533+0000","last_change":"2023-10-16T13:47:22.412533+0000","last_active":"2023-10-16T13:47:22.400586+0000","last_peered":"2023-10-16T13:47:22.400586+0000","last_clean":"2023-10-16T13:47:22.400586+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:22.412533+0000","last_undegraded":"2023-10-16T13:47:22.412533+0000","last_fullsized":"2023-10-16T13:47:22.412533+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.21","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.417066+0000","last_change":"2023-10-16T13:47:24.431657+0000","last_active":"2023-10-16T13:47:25.417066+0000","last_peered":"2023-10-16T13:47:25.417066+0000","last_clean":"2023-10-16T13:47:25.417066+0000","last_became_active":"2023-10-16T13:47:24.431385+0000","last_became_peered":"2023-10-16T13:47:24.431385+0000","last_unstale":"2023-10-16T13:47:25.417066+0000","last_undegraded":"2023-10-16T13:47:25.417066+0000","last_fullsized":"2023-10-16T13:47:25.417066+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:59:09.754185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1e","version":"0'0","reported_seq":1,"reported_epoch":242,"state":"creating+peering","last_fresh":"2023-10-16T13:47:22.406892+0000","last_change":"2023-10-16T13:47:22.406892+0000","last_active":"2023-10-16T13:47:22.400586+0000","last_peered":"2023-10-16T13:47:22.400586+0000","last_clean":"2023-10-16T13:47:22.400586+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-16T13:47:22.406892+0000","last_undegraded":"2023-10-16T13:47:22.406892+0000","last_fullsized":"2023-10-16T13:47:22.406892+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"231'370","reported_seq":1295,"reported_epoch":241,"state":"active+clean","last_fresh":"2023-10-16T13:47:20.908271+0000","last_change":"2023-10-16T12:10:34.746820+0000","last_active":"2023-10-16T13:47:20.908271+0000","last_peered":"2023-10-16T13:47:20.908271+0000","last_clean":"2023-10-16T13:47:20.908271+0000","last_became_active":"2023-10-16T12:10:34.745574+0000","last_became_peered":"2023-10-16T12:10:34.745574+0000","last_unstale":"2023-10-16T13:47:20.908271+0000","last_undegraded":"2023-10-16T13:47:20.908271+0000","last_fullsized":"2023-10-16T13:47:20.908271+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_clean_scrub_stamp":"2023-10-16T12:10:33.577111+0000","objects_scrubbed":0,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:14.461986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":32,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":84,"acting":84,"num_store_stats":5},{"poolid":31,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":37,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":4626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31,"ondisk_log_size":31,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":370,"ondisk_log_size":370,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575995,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56936,"kb_used_data":3548,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314904,"statfs":{"total":96636764160,"available":96578461696,"internally_reserved":0,"allocated":3633152,"data_stored":2159844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54655435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:46:50 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.60299999999999998,"15min":0.63200000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.33000000000000002},"max":{"1min":0.78600000000000003,"5min":0.87,"15min":0.93799999999999994},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.60699999999999998,"15min":0.626},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.72099999999999997,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.63900000000000001}]},{"osd":1,"last update":"Mon Oct 16 13:46:50 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.61299999999999999,"15min":0.63900000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.79000000000000004,"5min":0.84599999999999997,"15min":0.95899999999999996},"last":0.497},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.622,"15min":0.63100000000000001},"min":{"1min":0.38200000000000001,"5min":0.375,"15min":0.375},"max":{"1min":0.76200000000000001,"5min":0.79100000000000004,"15min":0.86399999999999999},"last":0.54900000000000004}]},{"osd":2,"last update":"Mon Oct 16 13:46:50 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.67400000000000004,"15min":0.68600000000000005},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.80300000000000005,"5min":0.83899999999999997,"15min":0.88200000000000001},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.68899999999999995,"15min":0.69799999999999995},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.39400000000000002},"max":{"1min":0.83699999999999997,"5min":0.92800000000000005,"15min":0.93799999999999994},"last":0.66100000000000003}]},{"osd":3,"last update":"Mon Oct 16 13:46:50 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.63800000000000001,"15min":0.65900000000000003},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.85299999999999998,"5min":0.89200000000000002,"15min":0.91300000000000003},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.65000000000000002,"15min":0.66000000000000003},"min":{"1min":0.435,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.88,"5min":1.24,"15min":1.24},"last":0.69599999999999995}]},{"osd":4,"last update":"Mon Oct 16 13:46:50 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.52000000000000002,"15min":0.51900000000000002},"min":{"1min":0.311,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.71099999999999997,"5min":0.745,"15min":0.85299999999999998},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.621,"15min":0.61799999999999999},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.33100000000000002},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.70899999999999996}]},{"osd":5,"last update":"Mon Oct 16 13:46:50 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.54900000000000004,"15min":0.54700000000000004},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.85699999999999998,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.58399999999999996,"15min":0.58699999999999997},"min":{"1min":0.41099999999999998,"5min":0.38400000000000001,"15min":0.35099999999999998},"max":{"1min":0.81100000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.73999999999999999}]},{"osd":6,"last update":"Mon Oct 16 13:46:50 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.60499999999999998,"15min":0.61299999999999999},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.85499999999999998,"5min":0.874,"15min":0.95899999999999996},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.626,"15min":0.61899999999999999},"min":{"1min":0.39400000000000002,"5min":0.378,"15min":0.35199999999999998},"max":{"1min":0.92100000000000004,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.76200000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575994,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60572,"kb_used_data":3024,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311268,"statfs":{"total":96636764160,"available":96574738432,"internally_reserved":0,"allocated":3096576,"data_stored":1596547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58915271},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:46:36 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.65300000000000002,"15min":0.66800000000000004},"min":{"1min":0.41499999999999998,"5min":0.375,"15min":0.28699999999999998},"max":{"1min":0.82599999999999996,"5min":0.89400000000000002,"15min":0.92300000000000004},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.64500000000000002,"15min":0.66400000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.80400000000000005,"5min":0.91400000000000003,"15min":0.93799999999999994},"last":0.68200000000000005}]},{"osd":1,"last update":"Mon Oct 16 13:46:36 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.58599999999999997,"15min":0.624},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.33100000000000002},"max":{"1min":0.70299999999999996,"5min":0.86899999999999999,"15min":0.90900000000000003},"last":0.504},{"interface":"front","average":{"1min":0.63,"5min":0.61399999999999999,"15min":0.62},"min":{"1min":0.41699999999999998,"5min":0.39800000000000002,"15min":0.38300000000000001},"max":{"1min":0.85799999999999998,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.42199999999999999}]},{"osd":2,"last update":"Mon Oct 16 13:46:36 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.622,"15min":0.63300000000000001},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.85799999999999998,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.59999999999999998,"15min":0.624},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.749,"5min":0.89000000000000001,"15min":0.91000000000000003},"last":0.44500000000000001}]},{"osd":3,"last update":"Mon Oct 16 13:46:36 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63900000000000001,"15min":0.66300000000000003},"min":{"1min":0.439,"5min":0.39900000000000002,"15min":0.38200000000000001},"max":{"1min":0.874,"5min":0.92400000000000004,"15min":0.95599999999999996},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.69099999999999995,"15min":0.71599999999999997},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.42299999999999999},"max":{"1min":0.85099999999999998,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.48099999999999998}]},{"osd":4,"last update":"Mon Oct 16 13:46:36 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.51800000000000002,"15min":0.52700000000000002},"min":{"1min":0.39000000000000001,"5min":0.35999999999999999,"15min":0.315},"max":{"1min":0.78000000000000003,"5min":0.85599999999999998,"15min":0.95799999999999996},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61699999999999999,"15min":0.62},"min":{"1min":0.375,"5min":0.35499999999999998,"15min":0.32900000000000001},"max":{"1min":0.83699999999999997,"5min":0.98799999999999999,"15min":1.0309999999999999},"last":0.64600000000000002}]},{"osd":5,"last update":"Mon Oct 16 13:46:36 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.59799999999999998,"15min":0.59399999999999997},"min":{"1min":0.39800000000000002,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.81100000000000005,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.66500000000000004,"15min":0.66000000000000003},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.308},"max":{"1min":0.85899999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.53500000000000003}]},{"osd":7,"last update":"Mon Oct 16 13:46:36 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.60899999999999999,"15min":0.60499999999999998},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.80000000000000004,"5min":1.081,"15min":1.081},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65200000000000002},"min":{"1min":0.42799999999999999,"5min":0.41899999999999998,"15min":0.375},"max":{"1min":0.91500000000000004,"5min":1.099,"15min":1.099},"last":0.629}]}]},{"osd":1,"up_from":13,"seq":55834575994,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52252,"kb_used_data":3016,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319588,"statfs":{"total":96636764160,"available":96583258112,"internally_reserved":0,"allocated":3088384,"data_stored":1596435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":50395593},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:46:55 2023","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.39500000000000002,"15min":0.40699999999999997},"min":{"1min":0.27600000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.51300000000000001,"5min":0.65400000000000003,"15min":0.65800000000000003},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.47599999999999998,"15min":0.49199999999999999},"min":{"1min":0.32000000000000001,"5min":0.253,"15min":0.253},"max":{"1min":0.56000000000000005,"5min":0.70999999999999996,"15min":0.77200000000000002},"last":0.67000000000000004}]},{"osd":2,"last update":"Mon Oct 16 13:46:55 2023","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.436,"15min":0.45100000000000001},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.58599999999999997,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.48099999999999998,"15min":0.48999999999999999},"min":{"1min":0.34399999999999997,"5min":0.315,"15min":0.30199999999999999},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.71399999999999997},"last":0.65200000000000002}]},{"osd":3,"last update":"Mon Oct 16 13:46:55 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.42899999999999999,"15min":0.42799999999999999},"min":{"1min":0.307,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.50800000000000001,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.46100000000000002,"15min":0.47499999999999998},"min":{"1min":0.29799999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.52800000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.69399999999999995}]},{"osd":4,"last update":"Mon Oct 16 13:46:55 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.64400000000000002,"15min":0.66200000000000003},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.42399999999999999},"max":{"1min":0.78600000000000003,"5min":0.95899999999999996,"15min":1.0069999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.64200000000000002,"15min":0.66600000000000004},"min":{"1min":0.42899999999999999,"5min":0.37,"15min":0.37},"max":{"1min":0.69999999999999996,"5min":0.91300000000000003,"15min":0.997},"last":0.747}]},{"osd":5,"last update":"Mon Oct 16 13:46:55 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.68200000000000005,"15min":0.70699999999999996},"min":{"1min":0.45800000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.83999999999999997,"5min":0.97499999999999998,"15min":1.0760000000000001},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.70499999999999996,"15min":0.72599999999999998},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.81399999999999995,"5min":1.0469999999999999,"15min":1.0660000000000001},"last":0.73199999999999998}]},{"osd":6,"last update":"Mon Oct 16 13:46:55 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.67500000000000004,"15min":0.68000000000000005},"min":{"1min":0.46200000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.75900000000000001,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.68700000000000006,"15min":0.70499999999999996},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.80400000000000005,"5min":1.03,"15min":1.083},"last":0.72299999999999998}]},{"osd":7,"last update":"Mon Oct 16 13:46:55 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.70899999999999996,"15min":0.73399999999999999},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.77300000000000002,"5min":1.0629999999999999,"15min":1.0660000000000001},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.72899999999999998,"15min":0.746},"min":{"1min":0.496,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.82099999999999995,"5min":1.042,"15min":1.0580000000000001},"last":0.87}]}]},{"osd":0,"up_from":13,"seq":55834575995,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48676,"kb_used_data":3608,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323164,"statfs":{"total":96636764160,"available":96586919936,"internally_reserved":0,"allocated":3694592,"data_stored":2188415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 13:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.40400000000000003,"15min":0.42099999999999999},"min":{"1min":0.29099999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.60699999999999998,"5min":0.65800000000000003,"15min":0.71399999999999997},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.42199999999999999,"15min":0.439},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.82199999999999995},"last":0.41999999999999998}]},{"osd":2,"last update":"Mon Oct 16 13:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.443,"15min":0.45400000000000001},"min":{"1min":0.28599999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.61399999999999999,"5min":0.65900000000000003,"15min":0.75700000000000001},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.46400000000000002,"15min":0.48499999999999999},"min":{"1min":0.29799999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.83299999999999996},"last":0.60499999999999998}]},{"osd":3,"last update":"Mon Oct 16 13:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.42699999999999999,"15min":0.45300000000000001},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.56499999999999995,"5min":0.63400000000000001,"15min":0.78600000000000003},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.50900000000000001,"15min":0.52000000000000002},"min":{"1min":0.317,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.66400000000000003,"5min":0.82999999999999996,"15min":0.86399999999999999},"last":0.57199999999999995}]},{"osd":4,"last update":"Mon Oct 16 13:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.66400000000000003,"15min":0.68400000000000005},"min":{"1min":0.42399999999999999,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.66500000000000004,"15min":0.68899999999999995},"min":{"1min":0.499,"5min":0.375,"15min":0.375},"max":{"1min":1.21,"5min":1.21,"15min":1.21},"last":0.78200000000000003}]},{"osd":5,"last update":"Mon Oct 16 13:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.66400000000000003,"15min":0.68999999999999995},"min":{"1min":0.41999999999999998,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.242,"5min":1.242,"15min":1.242},"last":0.751},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.70599999999999996,"15min":0.72299999999999998},"min":{"1min":0.497,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.998,"5min":1.081,"15min":1.081},"last":0.63200000000000001}]},{"osd":6,"last update":"Mon Oct 16 13:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.71199999999999997,"15min":0.72899999999999998},"min":{"1min":0.51900000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.96599999999999997,"5min":1.034,"15min":1.103},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.70399999999999996,"15min":0.72299999999999998},"min":{"1min":0.51000000000000001,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":0.79600000000000004}]},{"osd":7,"last update":"Mon Oct 16 13:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.71199999999999997,"15min":0.74199999999999999},"min":{"1min":0.48399999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.1919999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.72499999999999998,"15min":0.73499999999999999},"min":{"1min":0.51300000000000001,"5min":0.433,"15min":0.433},"max":{"1min":1.0469999999999999,"5min":1.087,"15min":1.087},"last":0.64700000000000002}]}]},{"osd":2,"up_from":13,"seq":55834575995,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52252,"kb_used_data":3024,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319588,"statfs":{"total":96636764160,"available":96583258112,"internally_reserved":0,"allocated":3096576,"data_stored":1597087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.41499999999999998,"15min":0.40899999999999997},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.623,"5min":0.71699999999999997,"15min":0.77500000000000002},"last":0.438},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.47699999999999998,"15min":0.47999999999999998},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.52800000000000002}]},{"osd":1,"last update":"Mon Oct 16 13:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.502,"15min":0.5},"min":{"1min":0.317,"5min":0.311,"15min":0.311},"max":{"1min":0.66800000000000004,"5min":0.73199999999999998,"15min":0.81399999999999995},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.48999999999999999,"15min":0.47699999999999998},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.27700000000000002},"max":{"1min":0.69999999999999996,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.58599999999999997}]},{"osd":3,"last update":"Mon Oct 16 13:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.443,"15min":0.44},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.59699999999999998,"5min":0.70099999999999996,"15min":0.72599999999999998},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.54300000000000004,"15min":0.54400000000000004},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.82699999999999996,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.74099999999999999}]},{"osd":4,"last update":"Mon Oct 16 13:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.72699999999999998,"15min":0.71799999999999997},"min":{"1min":0.51500000000000001,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.89000000000000001,"5min":1.3600000000000001,"15min":1.3600000000000001},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.72599999999999998,"15min":0.71899999999999997},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.82899999999999996,"5min":1.403,"15min":1.403},"last":0.91400000000000003}]},{"osd":5,"last update":"Mon Oct 16 13:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.68600000000000005,"15min":0.68899999999999995},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.81100000000000005,"5min":1.486,"15min":1.486},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.71199999999999997,"15min":0.69999999999999996},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.90200000000000002,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":0.879}]},{"osd":6,"last update":"Mon Oct 16 13:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.67600000000000005,"15min":0.66600000000000004},"min":{"1min":0.44500000000000001,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.80200000000000005,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.72599999999999998,"15min":0.72199999999999998},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.90600000000000003,"5min":1.4379999999999999,"15min":1.4379999999999999},"last":0.94299999999999995}]},{"osd":7,"last update":"Mon Oct 16 13:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.73899999999999999,"15min":0.72099999999999997},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.94099999999999995,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":0.97099999999999997},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.72999999999999998,"15min":0.73099999999999998},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.91500000000000004,"5min":1.377,"15min":1.377},"last":1.0429999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575994,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56428,"kb_used_data":3040,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315412,"statfs":{"total":96636764160,"available":96578981888,"internally_reserved":0,"allocated":3112960,"data_stored":1598685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.40300000000000002,"15min":0.40999999999999998},"min":{"1min":0.23499999999999999,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.57199999999999995,"5min":0.64200000000000002,"15min":0.72899999999999998},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.41499999999999998,"15min":0.42799999999999999},"min":{"1min":0.27300000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.60099999999999998,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.312}]},{"osd":1,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.42699999999999999,"15min":0.434},"min":{"1min":0.26700000000000002,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.55900000000000005,"5min":0.68200000000000005,"15min":0.76300000000000001},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.435,"5min":0.45900000000000002,"15min":0.47999999999999998},"min":{"1min":0.27700000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.65500000000000003,"5min":0.70699999999999996,"15min":0.77800000000000002},"last":0.441}]},{"osd":2,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.47999999999999998,"15min":0.49299999999999999},"min":{"1min":0.33800000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.621,"5min":1.167,"15min":1.167},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.501,"15min":0.52300000000000002},"min":{"1min":0.312,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.625,"5min":0.73299999999999998,"15min":0.78700000000000003},"last":0.49199999999999999}]},{"osd":4,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.65100000000000002,"15min":0.66500000000000004},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.91200000000000003,"5min":0.998,"15min":1.137},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.68899999999999995,"15min":0.69399999999999995},"min":{"1min":0.441,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.087,"5min":1.2110000000000001,"15min":1.2909999999999999},"last":0.72799999999999998}]},{"osd":5,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.67700000000000005,"15min":0.70499999999999996},"min":{"1min":0.45500000000000002,"5min":0.42899999999999999,"15min":0.374},"max":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.2729999999999999},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.69299999999999995,"15min":0.69999999999999996},"min":{"1min":0.45600000000000002,"5min":0.432,"15min":0.432},"max":{"1min":0.98799999999999999,"5min":1.0549999999999999,"15min":1.173},"last":0.68600000000000005}]},{"osd":6,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.72899999999999998,"15min":0.71899999999999997},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.41799999999999998},"max":{"1min":0.99099999999999999,"5min":5.0999999999999996,"15min":5.0999999999999996},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.70999999999999996,"15min":0.73399999999999999},"min":{"1min":0.49199999999999999,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.0029999999999999,"5min":1.0309999999999999,"15min":1.2290000000000001},"last":0.80000000000000004}]},{"osd":7,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.71899999999999997,"15min":0.73399999999999999},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.048,"5min":1.048,"15min":1.2569999999999999},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.73299999999999998,"15min":0.746},"min":{"1min":0.501,"5min":0.47799999999999998,"15min":0.43099999999999999},"max":{"1min":1.0369999999999999,"5min":1.262,"15min":1.262},"last":0.66800000000000004}]}]},{"osd":4,"up_from":13,"seq":55834575992,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52208,"kb_used_data":2980,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319632,"statfs":{"total":96636764160,"available":96583303168,"internally_reserved":0,"allocated":3051520,"data_stored":1575741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.60699999999999998,"15min":0.625},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.33800000000000002},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.623,"15min":0.63100000000000001},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.84099999999999997,"5min":0.878,"15min":0.89500000000000002},"last":0.41699999999999998}]},{"osd":1,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.63,"15min":0.65600000000000003},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.628,"15min":0.63700000000000001},"min":{"1min":0.41099999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.50700000000000001}]},{"osd":2,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.65200000000000002,"15min":0.65300000000000002},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.83999999999999997,"5min":0.89900000000000002,"15min":0.90600000000000003},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.65600000000000003,"15min":0.65800000000000003},"min":{"1min":0.39600000000000002,"5min":0.39400000000000002,"15min":0.38400000000000001},"max":{"1min":0.88200000000000001,"5min":0.89700000000000002,"15min":0.92600000000000005},"last":0.438}]},{"osd":3,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.63200000000000001,"15min":0.64500000000000002},"min":{"1min":0.40400000000000003,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.78100000000000003,"5min":0.86599999999999999,"15min":0.99199999999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.67400000000000004,"15min":0.68200000000000005},"min":{"1min":0.44800000000000001,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.79100000000000004,"5min":0.872,"15min":0.93899999999999995},"last":0.46300000000000002}]},{"osd":5,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55100000000000005,"15min":0.53000000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.57999999999999996,"15min":0.56899999999999995},"min":{"1min":0.36099999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.82199999999999995,"5min":0.82299999999999995,"15min":0.94899999999999995},"last":0.59799999999999998}]},{"osd":6,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.625,"15min":0.59799999999999998},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.33200000000000002},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92700000000000005},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.747,"5min":0.71399999999999997,"15min":0.69399999999999995},"min":{"1min":0.52600000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.129,"5min":1.129,"15min":1.129},"last":0.61599999999999999}]},{"osd":7,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.66000000000000003,"15min":0.63500000000000001},"min":{"1min":0.44400000000000001,"5min":0.41999999999999998,"15min":0.35799999999999998},"max":{"1min":0.97099999999999997,"5min":0.97499999999999998,"15min":0.98599999999999999},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.70799999999999996,"15min":0.68400000000000005},"min":{"1min":0.44,"5min":0.433,"15min":0.38200000000000001},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":1.026},"last":0.71599999999999997}]}]},{"osd":5,"up_from":13,"seq":55834575993,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52196,"kb_used_data":2968,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319644,"statfs":{"total":96636764160,"available":96583315456,"internally_reserved":0,"allocated":3039232,"data_stored":1569476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:46:24 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.61399999999999999,"15min":0.61499999999999999},"min":{"1min":0.314,"5min":0.314,"15min":0.26800000000000002},"max":{"1min":0.878,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63,"15min":0.64200000000000002},"min":{"1min":0.38700000000000001,"5min":0.35899999999999999,"15min":0.29699999999999999},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.42099999999999999}]},{"osd":1,"last update":"Mon Oct 16 13:46:24 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.66000000000000003,"15min":0.66200000000000003},"min":{"1min":0.44600000000000001,"5min":0.34300000000000003,"15min":0.29499999999999998},"max":{"1min":1,"5min":1,"15min":1},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.63800000000000001,"15min":0.64000000000000001},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.58899999999999997}]},{"osd":2,"last update":"Mon Oct 16 13:46:24 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.67000000000000004,"15min":0.65700000000000003},"min":{"1min":0.41699999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.91000000000000003,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.629,"5min":0.65700000000000003,"15min":0.64900000000000002},"min":{"1min":0.42599999999999999,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.83899999999999997,"5min":0.871,"15min":0.93500000000000005},"last":0.51000000000000001}]},{"osd":3,"last update":"Mon Oct 16 13:46:24 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.67400000000000004,"15min":0.66400000000000003},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.69899999999999995,"15min":0.69499999999999995},"min":{"1min":0.41399999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.4199999999999999,"5min":1.4199999999999999,"15min":1.4199999999999999},"last":0.54500000000000004}]},{"osd":4,"last update":"Mon Oct 16 13:46:24 2023","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.54400000000000004,"15min":0.53300000000000003},"min":{"1min":0.34100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.82499999999999996,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.59299999999999997,"15min":0.58899999999999997},"min":{"1min":0.34300000000000003,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.85399999999999998,"5min":0.89000000000000001,"15min":0.93400000000000005},"last":0.753}]},{"osd":6,"last update":"Mon Oct 16 13:46:24 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.58899999999999997,"15min":0.57999999999999996},"min":{"1min":0.38700000000000001,"5min":0.315,"15min":0.315},"max":{"1min":0.877,"5min":0.88900000000000001,"15min":0.93200000000000005},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.61899999999999999},"min":{"1min":0.372,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89600000000000002},"last":0.68200000000000005}]},{"osd":7,"last update":"Mon Oct 16 13:46:24 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.64800000000000002,"15min":0.63300000000000001},"min":{"1min":0.378,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.69699999999999995,"15min":0.68400000000000005},"min":{"1min":0.46200000000000002,"5min":0.375,"15min":0.375},"max":{"1min":0.94699999999999995,"5min":0.98299999999999998,"15min":1.0229999999999999},"last":0.72399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T13:47:29.393 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.19 is not active+clean 2023-10-16T13:47:29.394 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.19', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T13:47:23.404948+0000', 'last_change': '2023-10-16T13:47:23.404948+0000', 'last_active': '2023-10-16T13:47:23.404948+0000', 'last_peered': '2023-10-16T13:47:23.404948+0000', 'last_clean': '2023-10-16T13:47:23.404948+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T13:47:23.404948+0000', 'last_undegraded': '2023-10-16T13:47:23.404948+0000', 'last_fullsized': '2023-10-16T13:47:23.404948+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T13:47:23.404948+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T13:47:23.404948+0000', 'last_clean_scrub_stamp': '2023-10-16T13:47:23.404948+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T13:47:29.394 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.13 is not active+clean 2023-10-16T13:47:29.395 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.13', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T13:47:23.404948+0000', 'last_change': '2023-10-16T13:47:23.404948+0000', 'last_active': '2023-10-16T13:47:23.404948+0000', 'last_peered': '2023-10-16T13:47:23.404948+0000', 'last_clean': '2023-10-16T13:47:23.404948+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T13:47:23.404948+0000', 'last_undegraded': '2023-10-16T13:47:23.404948+0000', 'last_fullsized': '2023-10-16T13:47:23.404948+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T13:47:23.404948+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T13:47:23.404948+0000', 'last_clean_scrub_stamp': '2023-10-16T13:47:23.404948+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T13:47:29.395 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.11 is not active+clean 2023-10-16T13:47:29.395 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.11', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T13:47:23.404948+0000', 'last_change': '2023-10-16T13:47:23.404948+0000', 'last_active': '2023-10-16T13:47:23.404948+0000', 'last_peered': '2023-10-16T13:47:23.404948+0000', 'last_clean': '2023-10-16T13:47:23.404948+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T13:47:23.404948+0000', 'last_undegraded': '2023-10-16T13:47:23.404948+0000', 'last_fullsized': '2023-10-16T13:47:23.404948+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T13:47:23.404948+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T13:47:23.404948+0000', 'last_clean_scrub_stamp': '2023-10-16T13:47:23.404948+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T13:47:29.396 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.f is not active+clean 2023-10-16T13:47:29.396 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T13:47:23.404948+0000', 'last_change': '2023-10-16T13:47:23.404948+0000', 'last_active': '2023-10-16T13:47:23.404948+0000', 'last_peered': '2023-10-16T13:47:23.404948+0000', 'last_clean': '2023-10-16T13:47:23.404948+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T13:47:23.404948+0000', 'last_undegraded': '2023-10-16T13:47:23.404948+0000', 'last_fullsized': '2023-10-16T13:47:23.404948+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T13:47:23.404948+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T13:47:23.404948+0000', 'last_clean_scrub_stamp': '2023-10-16T13:47:23.404948+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T13:47:29.396 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.d is not active+clean 2023-10-16T13:47:29.397 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T13:47:23.404948+0000', 'last_change': '2023-10-16T13:47:23.404948+0000', 'last_active': '2023-10-16T13:47:23.404948+0000', 'last_peered': '2023-10-16T13:47:23.404948+0000', 'last_clean': '2023-10-16T13:47:23.404948+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T13:47:23.404948+0000', 'last_undegraded': '2023-10-16T13:47:23.404948+0000', 'last_fullsized': '2023-10-16T13:47:23.404948+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T13:47:23.404948+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T13:47:23.404948+0000', 'last_clean_scrub_stamp': '2023-10-16T13:47:23.404948+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T13:47:29.397 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.b is not active+clean 2023-10-16T13:47:29.398 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T13:47:23.404948+0000', 'last_change': '2023-10-16T13:47:23.404948+0000', 'last_active': '2023-10-16T13:47:23.404948+0000', 'last_peered': '2023-10-16T13:47:23.404948+0000', 'last_clean': '2023-10-16T13:47:23.404948+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T13:47:23.404948+0000', 'last_undegraded': '2023-10-16T13:47:23.404948+0000', 'last_fullsized': '2023-10-16T13:47:23.404948+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T13:47:23.404948+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T13:47:23.404948+0000', 'last_clean_scrub_stamp': '2023-10-16T13:47:23.404948+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T13:47:29.398 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.a is not active+clean 2023-10-16T13:47:29.399 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T13:47:23.404948+0000', 'last_change': '2023-10-16T13:47:23.404948+0000', 'last_active': '2023-10-16T13:47:23.404948+0000', 'last_peered': '2023-10-16T13:47:23.404948+0000', 'last_clean': '2023-10-16T13:47:23.404948+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T13:47:23.404948+0000', 'last_undegraded': '2023-10-16T13:47:23.404948+0000', 'last_fullsized': '2023-10-16T13:47:23.404948+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T13:47:23.404948+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T13:47:23.404948+0000', 'last_clean_scrub_stamp': '2023-10-16T13:47:23.404948+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T13:47:29.399 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.9 is not active+clean 2023-10-16T13:47:29.399 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.9', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T13:47:23.404948+0000', 'last_change': '2023-10-16T13:47:23.404948+0000', 'last_active': '2023-10-16T13:47:23.404948+0000', 'last_peered': '2023-10-16T13:47:23.404948+0000', 'last_clean': '2023-10-16T13:47:23.404948+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T13:47:23.404948+0000', 'last_undegraded': '2023-10-16T13:47:23.404948+0000', 'last_fullsized': '2023-10-16T13:47:23.404948+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T13:47:23.404948+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T13:47:23.404948+0000', 'last_clean_scrub_stamp': '2023-10-16T13:47:23.404948+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T13:47:29.400 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.8 is not active+clean 2023-10-16T13:47:29.400 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.8', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T13:47:23.404948+0000', 'last_change': '2023-10-16T13:47:23.404948+0000', 'last_active': '2023-10-16T13:47:23.404948+0000', 'last_peered': '2023-10-16T13:47:23.404948+0000', 'last_clean': '2023-10-16T13:47:23.404948+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T13:47:23.404948+0000', 'last_undegraded': '2023-10-16T13:47:23.404948+0000', 'last_fullsized': '2023-10-16T13:47:23.404948+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T13:47:23.404948+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T13:47:23.404948+0000', 'last_clean_scrub_stamp': '2023-10-16T13:47:23.404948+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T13:47:29.400 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.6 is not active+clean 2023-10-16T13:47:29.401 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.6', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T13:47:23.404948+0000', 'last_change': '2023-10-16T13:47:23.404948+0000', 'last_active': '2023-10-16T13:47:23.404948+0000', 'last_peered': '2023-10-16T13:47:23.404948+0000', 'last_clean': '2023-10-16T13:47:23.404948+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T13:47:23.404948+0000', 'last_undegraded': '2023-10-16T13:47:23.404948+0000', 'last_fullsized': '2023-10-16T13:47:23.404948+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T13:47:23.404948+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T13:47:23.404948+0000', 'last_clean_scrub_stamp': '2023-10-16T13:47:23.404948+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T13:47:29.401 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.22 is not active+clean 2023-10-16T13:47:29.401 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.22', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T13:47:23.404948+0000', 'last_change': '2023-10-16T13:47:23.404948+0000', 'last_active': '2023-10-16T13:47:23.404948+0000', 'last_peered': '2023-10-16T13:47:23.404948+0000', 'last_clean': '2023-10-16T13:47:23.404948+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T13:47:23.404948+0000', 'last_undegraded': '2023-10-16T13:47:23.404948+0000', 'last_fullsized': '2023-10-16T13:47:23.404948+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T13:47:23.404948+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T13:47:23.404948+0000', 'last_clean_scrub_stamp': '2023-10-16T13:47:23.404948+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T13:47:29.402 INFO:tasks.cephfs.filesystem.ceph_manager:PG 31.1d is not active+clean 2023-10-16T13:47:29.402 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '31.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 242, 'state': 'creating+peering', 'last_fresh': '2023-10-16T13:47:22.414037+0000', 'last_change': '2023-10-16T13:47:22.414037+0000', 'last_active': '2023-10-16T13:47:22.400586+0000', 'last_peered': '2023-10-16T13:47:22.400586+0000', 'last_clean': '2023-10-16T13:47:22.400586+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T13:47:22.414037+0000', 'last_undegraded': '2023-10-16T13:47:22.414037+0000', 'last_fullsized': '2023-10-16T13:47:22.414037+0000', 'mapping_epoch': 242, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 242, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T13:47:22.400586+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T13:47:22.400586+0000', 'last_clean_scrub_stamp': '2023-10-16T13:47:22.400586+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-10-16T13:47:29.403 INFO:tasks.cephfs.filesystem.ceph_manager:PG 31.1b is not active+clean 2023-10-16T13:47:29.403 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '31.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 242, 'state': 'creating+peering', 'last_fresh': '2023-10-16T13:47:22.413990+0000', 'last_change': '2023-10-16T13:47:22.413990+0000', 'last_active': '2023-10-16T13:47:22.400586+0000', 'last_peered': '2023-10-16T13:47:22.400586+0000', 'last_clean': '2023-10-16T13:47:22.400586+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T13:47:22.413990+0000', 'last_undegraded': '2023-10-16T13:47:22.413990+0000', 'last_fullsized': '2023-10-16T13:47:22.413990+0000', 'mapping_epoch': 242, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 242, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T13:47:22.400586+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T13:47:22.400586+0000', 'last_clean_scrub_stamp': '2023-10-16T13:47:22.400586+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-16T13:47:29.404 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.25 is not active+clean 2023-10-16T13:47:29.404 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.25', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T13:47:23.404948+0000', 'last_change': '2023-10-16T13:47:23.404948+0000', 'last_active': '2023-10-16T13:47:23.404948+0000', 'last_peered': '2023-10-16T13:47:23.404948+0000', 'last_clean': '2023-10-16T13:47:23.404948+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T13:47:23.404948+0000', 'last_undegraded': '2023-10-16T13:47:23.404948+0000', 'last_fullsized': '2023-10-16T13:47:23.404948+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T13:47:23.404948+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T13:47:23.404948+0000', 'last_clean_scrub_stamp': '2023-10-16T13:47:23.404948+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T13:47:29.404 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.28 is not active+clean 2023-10-16T13:47:29.405 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.28', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T13:47:23.404948+0000', 'last_change': '2023-10-16T13:47:23.404948+0000', 'last_active': '2023-10-16T13:47:23.404948+0000', 'last_peered': '2023-10-16T13:47:23.404948+0000', 'last_clean': '2023-10-16T13:47:23.404948+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T13:47:23.404948+0000', 'last_undegraded': '2023-10-16T13:47:23.404948+0000', 'last_fullsized': '2023-10-16T13:47:23.404948+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T13:47:23.404948+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T13:47:23.404948+0000', 'last_clean_scrub_stamp': '2023-10-16T13:47:23.404948+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T13:47:29.405 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.29 is not active+clean 2023-10-16T13:47:29.405 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.29', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T13:47:23.404948+0000', 'last_change': '2023-10-16T13:47:23.404948+0000', 'last_active': '2023-10-16T13:47:23.404948+0000', 'last_peered': '2023-10-16T13:47:23.404948+0000', 'last_clean': '2023-10-16T13:47:23.404948+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T13:47:23.404948+0000', 'last_undegraded': '2023-10-16T13:47:23.404948+0000', 'last_fullsized': '2023-10-16T13:47:23.404948+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T13:47:23.404948+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T13:47:23.404948+0000', 'last_clean_scrub_stamp': '2023-10-16T13:47:23.404948+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T13:47:29.406 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.2b is not active+clean 2023-10-16T13:47:29.406 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.2b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T13:47:23.404948+0000', 'last_change': '2023-10-16T13:47:23.404948+0000', 'last_active': '2023-10-16T13:47:23.404948+0000', 'last_peered': '2023-10-16T13:47:23.404948+0000', 'last_clean': '2023-10-16T13:47:23.404948+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T13:47:23.404948+0000', 'last_undegraded': '2023-10-16T13:47:23.404948+0000', 'last_fullsized': '2023-10-16T13:47:23.404948+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T13:47:23.404948+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T13:47:23.404948+0000', 'last_clean_scrub_stamp': '2023-10-16T13:47:23.404948+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T13:47:29.407 INFO:tasks.cephfs.filesystem.ceph_manager:PG 31.13 is not active+clean 2023-10-16T13:47:29.407 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '31.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 242, 'state': 'creating+peering', 'last_fresh': '2023-10-16T13:47:22.412143+0000', 'last_change': '2023-10-16T13:47:22.412143+0000', 'last_active': '2023-10-16T13:47:22.400586+0000', 'last_peered': '2023-10-16T13:47:22.400586+0000', 'last_clean': '2023-10-16T13:47:22.400586+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T13:47:22.412143+0000', 'last_undegraded': '2023-10-16T13:47:22.412143+0000', 'last_fullsized': '2023-10-16T13:47:22.412143+0000', 'mapping_epoch': 242, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 242, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T13:47:22.400586+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T13:47:22.400586+0000', 'last_clean_scrub_stamp': '2023-10-16T13:47:22.400586+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-16T13:47:29.408 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.2d is not active+clean 2023-10-16T13:47:29.408 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.2d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T13:47:23.404948+0000', 'last_change': '2023-10-16T13:47:23.404948+0000', 'last_active': '2023-10-16T13:47:23.404948+0000', 'last_peered': '2023-10-16T13:47:23.404948+0000', 'last_clean': '2023-10-16T13:47:23.404948+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T13:47:23.404948+0000', 'last_undegraded': '2023-10-16T13:47:23.404948+0000', 'last_fullsized': '2023-10-16T13:47:23.404948+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T13:47:23.404948+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T13:47:23.404948+0000', 'last_clean_scrub_stamp': '2023-10-16T13:47:23.404948+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T13:47:29.408 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.2e is not active+clean 2023-10-16T13:47:29.409 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.2e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T13:47:23.404948+0000', 'last_change': '2023-10-16T13:47:23.404948+0000', 'last_active': '2023-10-16T13:47:23.404948+0000', 'last_peered': '2023-10-16T13:47:23.404948+0000', 'last_clean': '2023-10-16T13:47:23.404948+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T13:47:23.404948+0000', 'last_undegraded': '2023-10-16T13:47:23.404948+0000', 'last_fullsized': '2023-10-16T13:47:23.404948+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T13:47:23.404948+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T13:47:23.404948+0000', 'last_clean_scrub_stamp': '2023-10-16T13:47:23.404948+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T13:47:29.409 INFO:tasks.cephfs.filesystem.ceph_manager:PG 31.d is not active+clean 2023-10-16T13:47:29.410 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '31.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 242, 'state': 'creating+peering', 'last_fresh': '2023-10-16T13:47:22.413572+0000', 'last_change': '2023-10-16T13:47:22.413572+0000', 'last_active': '2023-10-16T13:47:22.400586+0000', 'last_peered': '2023-10-16T13:47:22.400586+0000', 'last_clean': '2023-10-16T13:47:22.400586+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T13:47:22.413572+0000', 'last_undegraded': '2023-10-16T13:47:22.413572+0000', 'last_fullsized': '2023-10-16T13:47:22.413572+0000', 'mapping_epoch': 242, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 242, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T13:47:22.400586+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T13:47:22.400586+0000', 'last_clean_scrub_stamp': '2023-10-16T13:47:22.400586+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-10-16T13:47:29.410 INFO:tasks.cephfs.filesystem.ceph_manager:PG 31.c is not active+clean 2023-10-16T13:47:29.410 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '31.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 242, 'state': 'creating+peering', 'last_fresh': '2023-10-16T13:47:22.412157+0000', 'last_change': '2023-10-16T13:47:22.412157+0000', 'last_active': '2023-10-16T13:47:22.400586+0000', 'last_peered': '2023-10-16T13:47:22.400586+0000', 'last_clean': '2023-10-16T13:47:22.400586+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T13:47:22.412157+0000', 'last_undegraded': '2023-10-16T13:47:22.412157+0000', 'last_fullsized': '2023-10-16T13:47:22.412157+0000', 'mapping_epoch': 242, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 242, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T13:47:22.400586+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T13:47:22.400586+0000', 'last_clean_scrub_stamp': '2023-10-16T13:47:22.400586+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-16T13:47:29.411 INFO:tasks.cephfs.filesystem.ceph_manager:PG 31.a is not active+clean 2023-10-16T13:47:29.411 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '31.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 242, 'state': 'creating+peering', 'last_fresh': '2023-10-16T13:47:22.412844+0000', 'last_change': '2023-10-16T13:47:22.412844+0000', 'last_active': '2023-10-16T13:47:22.400586+0000', 'last_peered': '2023-10-16T13:47:22.400586+0000', 'last_clean': '2023-10-16T13:47:22.400586+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T13:47:22.412844+0000', 'last_undegraded': '2023-10-16T13:47:22.412844+0000', 'last_fullsized': '2023-10-16T13:47:22.412844+0000', 'mapping_epoch': 242, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 242, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T13:47:22.400586+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T13:47:22.400586+0000', 'last_clean_scrub_stamp': '2023-10-16T13:47:22.400586+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-10-16T13:47:29.412 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.36 is not active+clean 2023-10-16T13:47:29.412 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.36', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T13:47:23.404948+0000', 'last_change': '2023-10-16T13:47:23.404948+0000', 'last_active': '2023-10-16T13:47:23.404948+0000', 'last_peered': '2023-10-16T13:47:23.404948+0000', 'last_clean': '2023-10-16T13:47:23.404948+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T13:47:23.404948+0000', 'last_undegraded': '2023-10-16T13:47:23.404948+0000', 'last_fullsized': '2023-10-16T13:47:23.404948+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T13:47:23.404948+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T13:47:23.404948+0000', 'last_clean_scrub_stamp': '2023-10-16T13:47:23.404948+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T13:47:29.413 INFO:tasks.cephfs.filesystem.ceph_manager:PG 31.9 is not active+clean 2023-10-16T13:47:29.413 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '31.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 242, 'state': 'creating+peering', 'last_fresh': '2023-10-16T13:47:22.412735+0000', 'last_change': '2023-10-16T13:47:22.412735+0000', 'last_active': '2023-10-16T13:47:22.400586+0000', 'last_peered': '2023-10-16T13:47:22.400586+0000', 'last_clean': '2023-10-16T13:47:22.400586+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T13:47:22.412735+0000', 'last_undegraded': '2023-10-16T13:47:22.412735+0000', 'last_fullsized': '2023-10-16T13:47:22.412735+0000', 'mapping_epoch': 242, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 242, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T13:47:22.400586+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T13:47:22.400586+0000', 'last_clean_scrub_stamp': '2023-10-16T13:47:22.400586+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-10-16T13:47:29.413 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.37 is not active+clean 2023-10-16T13:47:29.414 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.37', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T13:47:23.404948+0000', 'last_change': '2023-10-16T13:47:23.404948+0000', 'last_active': '2023-10-16T13:47:23.404948+0000', 'last_peered': '2023-10-16T13:47:23.404948+0000', 'last_clean': '2023-10-16T13:47:23.404948+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T13:47:23.404948+0000', 'last_undegraded': '2023-10-16T13:47:23.404948+0000', 'last_fullsized': '2023-10-16T13:47:23.404948+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T13:47:23.404948+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T13:47:23.404948+0000', 'last_clean_scrub_stamp': '2023-10-16T13:47:23.404948+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T13:47:29.414 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.38 is not active+clean 2023-10-16T13:47:29.415 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.38', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T13:47:23.404948+0000', 'last_change': '2023-10-16T13:47:23.404948+0000', 'last_active': '2023-10-16T13:47:23.404948+0000', 'last_peered': '2023-10-16T13:47:23.404948+0000', 'last_clean': '2023-10-16T13:47:23.404948+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T13:47:23.404948+0000', 'last_undegraded': '2023-10-16T13:47:23.404948+0000', 'last_fullsized': '2023-10-16T13:47:23.404948+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T13:47:23.404948+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T13:47:23.404948+0000', 'last_clean_scrub_stamp': '2023-10-16T13:47:23.404948+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T13:47:29.415 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.39 is not active+clean 2023-10-16T13:47:29.416 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.39', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T13:47:23.404948+0000', 'last_change': '2023-10-16T13:47:23.404948+0000', 'last_active': '2023-10-16T13:47:23.404948+0000', 'last_peered': '2023-10-16T13:47:23.404948+0000', 'last_clean': '2023-10-16T13:47:23.404948+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T13:47:23.404948+0000', 'last_undegraded': '2023-10-16T13:47:23.404948+0000', 'last_fullsized': '2023-10-16T13:47:23.404948+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T13:47:23.404948+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T13:47:23.404948+0000', 'last_clean_scrub_stamp': '2023-10-16T13:47:23.404948+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T13:47:29.416 INFO:tasks.cephfs.filesystem.ceph_manager:PG 31.3 is not active+clean 2023-10-16T13:47:29.417 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '31.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 242, 'state': 'creating+peering', 'last_fresh': '2023-10-16T13:47:22.413180+0000', 'last_change': '2023-10-16T13:47:22.413180+0000', 'last_active': '2023-10-16T13:47:22.400586+0000', 'last_peered': '2023-10-16T13:47:22.400586+0000', 'last_clean': '2023-10-16T13:47:22.400586+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T13:47:22.413180+0000', 'last_undegraded': '2023-10-16T13:47:22.413180+0000', 'last_fullsized': '2023-10-16T13:47:22.413180+0000', 'mapping_epoch': 242, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 242, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T13:47:22.400586+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T13:47:22.400586+0000', 'last_clean_scrub_stamp': '2023-10-16T13:47:22.400586+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-10-16T13:47:29.417 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.3f is not active+clean 2023-10-16T13:47:29.417 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.3f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-16T13:47:23.404948+0000', 'last_change': '2023-10-16T13:47:23.404948+0000', 'last_active': '2023-10-16T13:47:23.404948+0000', 'last_peered': '2023-10-16T13:47:23.404948+0000', 'last_clean': '2023-10-16T13:47:23.404948+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T13:47:23.404948+0000', 'last_undegraded': '2023-10-16T13:47:23.404948+0000', 'last_fullsized': '2023-10-16T13:47:23.404948+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T13:47:23.404948+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T13:47:23.404948+0000', 'last_clean_scrub_stamp': '2023-10-16T13:47:23.404948+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-16T13:47:29.418 INFO:tasks.cephfs.filesystem.ceph_manager:PG 31.0 is not active+clean 2023-10-16T13:47:29.418 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '31.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 242, 'state': 'creating+peering', 'last_fresh': '2023-10-16T13:47:22.412533+0000', 'last_change': '2023-10-16T13:47:22.412533+0000', 'last_active': '2023-10-16T13:47:22.400586+0000', 'last_peered': '2023-10-16T13:47:22.400586+0000', 'last_clean': '2023-10-16T13:47:22.400586+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T13:47:22.412533+0000', 'last_undegraded': '2023-10-16T13:47:22.412533+0000', 'last_fullsized': '2023-10-16T13:47:22.412533+0000', 'mapping_epoch': 242, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 242, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T13:47:22.400586+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T13:47:22.400586+0000', 'last_clean_scrub_stamp': '2023-10-16T13:47:22.400586+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-10-16T13:47:29.419 INFO:tasks.cephfs.filesystem.ceph_manager:PG 31.1e is not active+clean 2023-10-16T13:47:29.419 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '31.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 242, 'state': 'creating+peering', 'last_fresh': '2023-10-16T13:47:22.406892+0000', 'last_change': '2023-10-16T13:47:22.406892+0000', 'last_active': '2023-10-16T13:47:22.400586+0000', 'last_peered': '2023-10-16T13:47:22.400586+0000', 'last_clean': '2023-10-16T13:47:22.400586+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-16T13:47:22.406892+0000', 'last_undegraded': '2023-10-16T13:47:22.406892+0000', 'last_fullsized': '2023-10-16T13:47:22.406892+0000', 'mapping_epoch': 242, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 242, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-16T13:47:22.400586+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-16T13:47:22.400586+0000', 'last_clean_scrub_stamp': '2023-10-16T13:47:22.400586+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-10-16T13:47:29.420 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2023-10-16T13:47:29.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.550+0000 7f599d28d700 1 Processor -- start 2023-10-16T13:47:29.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.551+0000 7f599d28d700 1 -- start start 2023-10-16T13:47:29.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.551+0000 7f599d28d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f599815db50 0x7f599806b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:29.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.551+0000 7f599d28d700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f599815df30 con 0x7f59980cf950 2023-10-16T13:47:29.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.551+0000 7f599d28d700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59980cbcf0 con 0x7f5998161650 2023-10-16T13:47:29.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.551+0000 7f599d28d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59980cbb60 con 0x7f599815db50 2023-10-16T13:47:29.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.552+0000 7f59977fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f59980cf950 0x7f59980cc180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44794/0 (socket says 172.21.15.16:44794) 2023-10-16T13:47:29.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.552+0000 7f59977fe700 1 -- 172.21.15.16:0/1040317687 learned_addr learned my addr 172.21.15.16:0/1040317687 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:47:29.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.552+0000 7f5996ffd700 1 --2- 172.21.15.16:0/1040317687 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f599815db50 0x7f599806b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:29.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.552+0000 7f5995ffb700 1 -- 172.21.15.16:0/1040317687 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3148172106 0 0) 0x7f5990001490 con 0x7f59980cf950 2023-10-16T13:47:29.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.552+0000 7f5995ffb700 1 -- 172.21.15.16:0/1040317687 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59840035b0 con 0x7f59980cf950 2023-10-16T13:47:29.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.552+0000 7f5995ffb700 1 -- 172.21.15.16:0/1040317687 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 112127732 0 0) 0x7f59980cbcf0 con 0x7f5998161650 2023-10-16T13:47:29.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.552+0000 7f5995ffb700 1 -- 172.21.15.16:0/1040317687 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5990001490 con 0x7f5998161650 2023-10-16T13:47:29.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.552+0000 7f5995ffb700 1 -- 172.21.15.16:0/1040317687 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3020061347 0 0) 0x7f59840035b0 con 0x7f59980cf950 2023-10-16T13:47:29.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.552+0000 7f5995ffb700 1 -- 172.21.15.16:0/1040317687 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f599815db50 msgr2=0x7f599806b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:29.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.552+0000 7f5995ffb700 1 --2- 172.21.15.16:0/1040317687 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f599815db50 0x7f599806b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:29.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.552+0000 7f5995ffb700 1 -- 172.21.15.16:0/1040317687 >> v1:172.21.15.49:6789/0 conn(0x7f5998161650 legacy=0x7f59980cf240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:29.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.552+0000 7f5995ffb700 1 -- 172.21.15.16:0/1040317687 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59980cb810 con 0x7f59980cf950 2023-10-16T13:47:29.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.552+0000 7f5995ffb700 1 -- 172.21.15.16:0/1040317687 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f5990001b80 con 0x7f59980cf950 2023-10-16T13:47:29.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.552+0000 7f5996ffd700 1 --2- 172.21.15.16:0/1040317687 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f599815db50 0x7f599806b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:47:29.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.552+0000 7f5995ffb700 1 -- 172.21.15.16:0/1040317687 >> v1:172.21.15.16:6789/0 conn(0x7f59980cf950 legacy=0x7f59980cc180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:29.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.552+0000 7f5995ffb700 1 --2- 172.21.15.16:0/1040317687 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5984003bc0 0x7f5984003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:29.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.552+0000 7f5995ffb700 1 --2- 172.21.15.16:0/1040317687 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f599815db50 0x7f5984004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:29.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.552+0000 7f5995ffb700 1 --2- 172.21.15.16:0/1040317687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5984004d90 0x7f5984005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:29.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.552+0000 7f5995ffb700 1 -- 172.21.15.16:0/1040317687 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59980cbb60 con 0x7f5984004d90 2023-10-16T13:47:29.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.552+0000 7f5995ffb700 1 -- 172.21.15.16:0/1040317687 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f598400bac0 con 0x7f5984003bc0 2023-10-16T13:47:29.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.552+0000 7f5995ffb700 1 -- 172.21.15.16:0/1040317687 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f598400bc00 con 0x7f599815db50 2023-10-16T13:47:29.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.552+0000 7f59977fe700 1 --2- 172.21.15.16:0/1040317687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5984004d90 0x7f5984005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:29.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.552+0000 7f5996ffd700 1 --2- 172.21.15.16:0/1040317687 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5984003bc0 0x7f5984003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:29.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.552+0000 7f59967fc700 1 --2- 172.21.15.16:0/1040317687 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f599815db50 0x7f5984004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:29.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.553+0000 7f59977fe700 1 -- 172.21.15.16:0/1040317687 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f599815db50 msgr2=0x7f5984004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:29.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.553+0000 7f59977fe700 1 --2- 172.21.15.16:0/1040317687 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f599815db50 0x7f5984004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:29.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.553+0000 7f59977fe700 1 -- 172.21.15.16:0/1040317687 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5984003bc0 msgr2=0x7f5984003fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:29.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.553+0000 7f59977fe700 1 --2- 172.21.15.16:0/1040317687 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5984003bc0 0x7f5984003fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:29.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.553+0000 7f59977fe700 1 -- 172.21.15.16:0/1040317687 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59880094a0 con 0x7f5984004d90 2023-10-16T13:47:29.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.553+0000 7f59977fe700 1 --2- 172.21.15.16:0/1040317687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5984004d90 0x7f5984005210 secure :-1 s=READY pgs=2758 cs=0 l=1 rev1=1 crypto rx=0x7f599000a7e0 tx=0x7f5984005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7d2b6c29671b92e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:29.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.553+0000 7f5995ffb700 1 -- 172.21.15.16:0/1040317687 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5990012600 con 0x7f5984004d90 2023-10-16T13:47:29.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.553+0000 7f5995ffb700 1 -- 172.21.15.16:0/1040317687 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5990012780 con 0x7f5984004d90 2023-10-16T13:47:29.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.553+0000 7f5995ffb700 1 -- 172.21.15.16:0/1040317687 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5990012b30 con 0x7f5984004d90 2023-10-16T13:47:29.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.553+0000 7f599d28d700 1 -- 172.21.15.16:0/1040317687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5984004d90 msgr2=0x7f5984005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:29.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.553+0000 7f599d28d700 1 --2- 172.21.15.16:0/1040317687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5984004d90 0x7f5984005210 secure :-1 s=READY pgs=2758 cs=0 l=1 rev1=1 crypto rx=0x7f599000a7e0 tx=0x7f5984005880 comp rx=0 tx=0).stop 2023-10-16T13:47:29.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.553+0000 7f59977fe700 1 -- 172.21.15.16:0/1040317687 reap_dead start 2023-10-16T13:47:29.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.554+0000 7f599d28d700 1 -- 172.21.15.16:0/1040317687 shutdown_connections 2023-10-16T13:47:29.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.554+0000 7f599d28d700 1 -- 172.21.15.16:0/1040317687 >> 172.21.15.16:0/1040317687 conn(0x7f59980cdf70 msgr2=0x7f59980ce380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:29.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.554+0000 7f599d28d700 1 -- 172.21.15.16:0/1040317687 shutdown_connections 2023-10-16T13:47:29.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.554+0000 7f599d28d700 1 -- 172.21.15.16:0/1040317687 wait complete. 2023-10-16T13:47:29.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.555+0000 7f599d28d700 1 Processor -- start 2023-10-16T13:47:29.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.555+0000 7f599d28d700 1 -- start start 2023-10-16T13:47:29.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.556+0000 7f599d28d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5984004d90 0x7f5998155df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:29.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.556+0000 7f599d28d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f59980b4da0 0x7f59980c85a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:29.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.556+0000 7f599d28d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59980b5f80 0x7f59980b6400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:29.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.556+0000 7f599d28d700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59980b6940 con 0x7f59980b5f80 2023-10-16T13:47:29.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.556+0000 7f599d28d700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5998156330 con 0x7f59980b4da0 2023-10-16T13:47:29.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.556+0000 7f599d28d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59980b5b40 con 0x7f5984004d90 2023-10-16T13:47:29.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.556+0000 7f59977fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59980b5f80 0x7f59980b6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:29.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.556+0000 7f59977fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59980b5f80 0x7f59980b6400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55532/0 (socket says 172.21.15.16:55532) 2023-10-16T13:47:29.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.556+0000 7f59977fe700 1 -- 172.21.15.16:0/1153617047 learned_addr learned my addr 172.21.15.16:0/1153617047 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:29.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.556+0000 7f5996ffd700 1 --2- 172.21.15.16:0/1153617047 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5984004d90 0x7f5998155df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:29.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.556+0000 7f59967fc700 1 --2- 172.21.15.16:0/1153617047 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f59980b4da0 0x7f59980c85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:29.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.556+0000 7f59977fe700 1 -- 172.21.15.16:0/1153617047 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5984004d90 msgr2=0x7f5998155df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:29.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.556+0000 7f59977fe700 1 --2- 172.21.15.16:0/1153617047 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5984004d90 0x7f5998155df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:29.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.556+0000 7f59977fe700 1 -- 172.21.15.16:0/1153617047 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f59980b4da0 msgr2=0x7f59980c85a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:29.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.556+0000 7f59977fe700 1 --2- 172.21.15.16:0/1153617047 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f59980b4da0 0x7f59980c85a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:29.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.556+0000 7f59977fe700 1 -- 172.21.15.16:0/1153617047 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59980b52a0 con 0x7f59980b5f80 2023-10-16T13:47:29.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.556+0000 7f5996ffd700 1 --2- 172.21.15.16:0/1153617047 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5984004d90 0x7f5998155df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:47:29.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.556+0000 7f59977fe700 1 --2- 172.21.15.16:0/1153617047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59980b5f80 0x7f59980b6400 secure :-1 s=READY pgs=2759 cs=0 l=1 rev1=1 crypto rx=0x7f5984005830 tx=0x7f5990001e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:29.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.556+0000 7f598f7fe700 1 -- 172.21.15.16:0/1153617047 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59900011c0 con 0x7f59980b5f80 2023-10-16T13:47:29.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.556+0000 7f598f7fe700 1 -- 172.21.15.16:0/1153617047 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f599001e430 con 0x7f59980b5f80 2023-10-16T13:47:29.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.557+0000 7f598f7fe700 1 -- 172.21.15.16:0/1153617047 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f599001e7a0 con 0x7f59980b5f80 2023-10-16T13:47:29.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.557+0000 7f599d28d700 1 -- 172.21.15.16:0/1153617047 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59980b5580 con 0x7f59980b5f80 2023-10-16T13:47:29.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.557+0000 7f599d28d700 1 -- 172.21.15.16:0/1153617047 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f599815e630 con 0x7f59980b5f80 2023-10-16T13:47:29.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.558+0000 7f599d28d700 1 -- 172.21.15.16:0/1153617047 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59980d0f50 con 0x7f59980b5f80 2023-10-16T13:47:29.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.558+0000 7f598f7fe700 1 -- 172.21.15.16:0/1153617047 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 148) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f599001e900 con 0x7f59980b5f80 2023-10-16T13:47:29.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.558+0000 7f598f7fe700 1 --2- 172.21.15.16:0/1153617047 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f597806b140 0x7f597806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:29.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.558+0000 7f598f7fe700 1 -- 172.21.15.16:0/1153617047 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(245..245 src has 1..245) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f599008b8a0 con 0x7f59980b5f80 2023-10-16T13:47:29.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.558+0000 7f5996ffd700 1 --2- 172.21.15.16:0/1153617047 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f597806b140 0x7f597806d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:29.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.559+0000 7f5996ffd700 1 --2- 172.21.15.16:0/1153617047 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f597806b140 0x7f597806d600 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5988009df0 tx=0x7f5988008040 comp rx=0 tx=0).ready entity=mgr.11886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:29.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.563+0000 7f598f7fe700 1 -- 172.21.15.16:0/1153617047 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5990056590 con 0x7f59980b5f80 2023-10-16T13:47:29.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.852+0000 7f599d28d700 1 -- 172.21.15.16:0/1153617047 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f599815df30 con 0x7f59980b5f80 2023-10-16T13:47:29.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.853+0000 7f598f7fe700 1 -- 172.21.15.16:0/1153617047 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+3376 (secure 0 0 0) 0x7f5990017d40 con 0x7f59980b5f80 2023-10-16T13:47:29.855 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:47:29.856 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":5828,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":245,"num_osds":8,"num_up_osds":8,"osd_up_since":1697458231,"num_in_osds":8,"osd_in_since":1697458223,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":65},{"state_name":"unknown","count":22},{"state_name":"creating+peering","count":10}],"num_pgs":97,"num_pools":3,"num_objects":19,"data_bytes":592674,"bytes_used":441876480,"bytes_avail":772652236800,"bytes_total":773094113280,"unknown_pgs_ratio":0.22680412232875824,"inactive_pgs_ratio":0.10309278219938278,"write_bytes_sec":1134,"read_op_per_sec":0,"write_op_per_sec":6},"fsmap":{"epoch":130,"id":15,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":15,"rank":0,"name":"a","status":"up:active","gid":12252}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":27,"modified":"2023-10-16T13:47:11.818544+0000","services":{"mds":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"mgr":{"daemons":{"summary":"","x":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"y":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"mon":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"4":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"5":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"6":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"7":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2023-10-16T13:47:29.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.855+0000 7f599d28d700 1 -- 172.21.15.16:0/1153617047 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f597806b140 msgr2=0x7f597806d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:29.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.855+0000 7f599d28d700 1 --2- 172.21.15.16:0/1153617047 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f597806b140 0x7f597806d600 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5988009df0 tx=0x7f5988008040 comp rx=0 tx=0).stop 2023-10-16T13:47:29.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.855+0000 7f599d28d700 1 -- 172.21.15.16:0/1153617047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59980b5f80 msgr2=0x7f59980b6400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:29.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.855+0000 7f599d28d700 1 --2- 172.21.15.16:0/1153617047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59980b5f80 0x7f59980b6400 secure :-1 s=READY pgs=2759 cs=0 l=1 rev1=1 crypto rx=0x7f5984005830 tx=0x7f5990001e90 comp rx=0 tx=0).stop 2023-10-16T13:47:29.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.855+0000 7f599d28d700 1 -- 172.21.15.16:0/1153617047 shutdown_connections 2023-10-16T13:47:29.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.855+0000 7f599d28d700 1 --2- 172.21.15.16:0/1153617047 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f597806b140 0x7f597806d600 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:29.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.855+0000 7f599d28d700 1 --2- 172.21.15.16:0/1153617047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59980b5f80 0x7f59980b6400 unknown :-1 s=CLOSED pgs=2759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:29.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.855+0000 7f599d28d700 1 --2- 172.21.15.16:0/1153617047 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5984004d90 0x7f5998155df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:29.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.855+0000 7f599d28d700 1 --2- 172.21.15.16:0/1153617047 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f59980b4da0 0x7f59980c85a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:29.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.855+0000 7f599d28d700 1 -- 172.21.15.16:0/1153617047 >> 172.21.15.16:0/1153617047 conn(0x7f59980cdf70 msgr2=0x7f59980b0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:29.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.855+0000 7f599d28d700 1 -- 172.21.15.16:0/1153617047 shutdown_connections 2023-10-16T13:47:29.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.855+0000 7f599d28d700 1 -- 172.21.15.16:0/1153617047 wait complete. 2023-10-16T13:47:29.870 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2023-10-16T13:47:29.870 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T13:47:29.996 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.994+0000 7f7474f83700 1 Processor -- start 2023-10-16T13:47:29.997 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.994+0000 7f7474f83700 1 -- start start 2023-10-16T13:47:29.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.995+0000 7f7474f83700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f747015d910 0x7f74700cb1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:29.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.995+0000 7f7474f83700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f747006a910 con 0x7f7470159cc0 2023-10-16T13:47:29.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.995+0000 7f7474f83700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f747006abf0 con 0x7f74700cf4a0 2023-10-16T13:47:29.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.995+0000 7f7474f83700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74700cbf50 con 0x7f747015d910 2023-10-16T13:47:30.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.995+0000 7f746dd9b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f747015d910 0x7f74700cb1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:30.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.995+0000 7f746dd9b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f747015d910 0x7f74700cb1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55538/0 (socket says 172.21.15.16:55538) 2023-10-16T13:47:30.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.995+0000 7f746dd9b700 1 -- 172.21.15.16:0/1741870715 learned_addr learned my addr 172.21.15.16:0/1741870715 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:30.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.995+0000 7f746e59c700 1 --1- 172.21.15.16:0/1741870715 >> v1:172.21.15.16:6789/0 conn(0x7f7470159cc0 0x7f747006b9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44810/0 (socket says 172.21.15.16:44810) 2023-10-16T13:47:30.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.995+0000 7f746dd9b700 1 -- 172.21.15.16:0/1741870715 >> v1:172.21.15.49:6789/0 conn(0x7f74700cf4a0 legacy=0x7f747006e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:30.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.995+0000 7f746dd9b700 1 -- 172.21.15.16:0/1741870715 >> v1:172.21.15.16:6789/0 conn(0x7f7470159cc0 legacy=0x7f747006b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:30.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.995+0000 7f746dd9b700 1 -- 172.21.15.16:0/1741870715 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f747006aed0 con 0x7f747015d910 2023-10-16T13:47:30.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.995+0000 7f746dd9b700 1 --2- 172.21.15.16:0/1741870715 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f747015d910 0x7f74700cb1f0 secure :-1 s=READY pgs=2761 cs=0 l=1 rev1=1 crypto rx=0x7f746400bb90 tx=0x7f746400bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d8bcc1ff3dfbdfc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:30.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.996+0000 7f746d59a700 1 -- 172.21.15.16:0/1741870715 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f746400dd10 con 0x7f747015d910 2023-10-16T13:47:30.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.996+0000 7f746d59a700 1 -- 172.21.15.16:0/1741870715 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f746400de70 con 0x7f747015d910 2023-10-16T13:47:30.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.996+0000 7f746d59a700 1 -- 172.21.15.16:0/1741870715 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7464004230 con 0x7f747015d910 2023-10-16T13:47:30.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.996+0000 7f7474f83700 1 -- 172.21.15.16:0/1741870715 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f747015d910 msgr2=0x7f74700cb1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:30.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.996+0000 7f7474f83700 1 --2- 172.21.15.16:0/1741870715 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f747015d910 0x7f74700cb1f0 secure :-1 s=READY pgs=2761 cs=0 l=1 rev1=1 crypto rx=0x7f746400bb90 tx=0x7f746400bea0 comp rx=0 tx=0).stop 2023-10-16T13:47:30.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.996+0000 7f7474f83700 1 -- 172.21.15.16:0/1741870715 shutdown_connections 2023-10-16T13:47:30.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.996+0000 7f7474f83700 1 --2- 172.21.15.16:0/1741870715 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f747015d910 0x7f74700cb1f0 unknown :-1 s=CLOSED pgs=2761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:30.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.996+0000 7f7474f83700 1 -- 172.21.15.16:0/1741870715 >> 172.21.15.16:0/1741870715 conn(0x7f74700cc5e0 msgr2=0x7f74700cc9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:30.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.996+0000 7f7474f83700 1 -- 172.21.15.16:0/1741870715 shutdown_connections 2023-10-16T13:47:30.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.996+0000 7f7474f83700 1 -- 172.21.15.16:0/1741870715 wait complete. 2023-10-16T13:47:30.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.998+0000 7f7474f83700 1 Processor -- start 2023-10-16T13:47:30.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.998+0000 7f7474f83700 1 -- start start 2023-10-16T13:47:30.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.998+0000 7f7474f83700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74700cf4a0 0x7f747014d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:30.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.999+0000 7f7474f83700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7470159cc0 0x7f74700ae8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:30.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.999+0000 7f7474f83700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f747015d910 0x7f74700ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:30.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.999+0000 7f7474f83700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f747014dbe0 con 0x7f74700cf4a0 2023-10-16T13:47:30.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.999+0000 7f7474f83700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f747014c970 con 0x7f747015d910 2023-10-16T13:47:30.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.999+0000 7f7474f83700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74700af2f0 con 0x7f7470159cc0 2023-10-16T13:47:30.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.999+0000 7f746e59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74700cf4a0 0x7f747014d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:30.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.999+0000 7f746e59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74700cf4a0 0x7f747014d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55542/0 (socket says 172.21.15.16:55542) 2023-10-16T13:47:30.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.999+0000 7f746e59c700 1 -- 172.21.15.16:0/683881752 learned_addr learned my addr 172.21.15.16:0/683881752 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:30.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.999+0000 7f746dd9b700 1 --2- 172.21.15.16:0/683881752 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7470159cc0 0x7f74700ae8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:30.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.999+0000 7f746ed9d700 1 --2- 172.21.15.16:0/683881752 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f747015d910 0x7f74700ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:30.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.999+0000 7f746e59c700 1 -- 172.21.15.16:0/683881752 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7470159cc0 msgr2=0x7f74700ae8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:30.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.999+0000 7f746e59c700 1 --2- 172.21.15.16:0/683881752 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7470159cc0 0x7f74700ae8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:30.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.999+0000 7f746e59c700 1 -- 172.21.15.16:0/683881752 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f747015d910 msgr2=0x7f74700ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:30.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.999+0000 7f746e59c700 1 --2- 172.21.15.16:0/683881752 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f747015d910 0x7f74700ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:30.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.999+0000 7f746e59c700 1 -- 172.21.15.16:0/683881752 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f746400b7f0 con 0x7f74700cf4a0 2023-10-16T13:47:30.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.999+0000 7f746ed9d700 1 --2- 172.21.15.16:0/683881752 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f747015d910 0x7f74700ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:47:30.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.999+0000 7f746e59c700 1 --2- 172.21.15.16:0/683881752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74700cf4a0 0x7f747014d6a0 secure :-1 s=READY pgs=2762 cs=0 l=1 rev1=1 crypto rx=0x7f746000c490 tx=0x7f746000c7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:30.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.999+0000 7f745effd700 1 -- 172.21.15.16:0/683881752 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7460005cb0 con 0x7f74700cf4a0 2023-10-16T13:47:30.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.999+0000 7f745effd700 1 -- 172.21.15.16:0/683881752 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7460005e10 con 0x7f74700cf4a0 2023-10-16T13:47:30.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.999+0000 7f7474f83700 1 -- 172.21.15.16:0/683881752 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f747014cbc0 con 0x7f74700cf4a0 2023-10-16T13:47:30.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.999+0000 7f745effd700 1 -- 172.21.15.16:0/683881752 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74600072d0 con 0x7f74700cf4a0 2023-10-16T13:47:30.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:29.999+0000 7f7474f83700 1 -- 172.21.15.16:0/683881752 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f747014d180 con 0x7f74700cf4a0 2023-10-16T13:47:30.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:30.000+0000 7f7474f83700 1 -- 172.21.15.16:0/683881752 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f747014d3a0 con 0x7f74700cf4a0 2023-10-16T13:47:30.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:30.001+0000 7f745effd700 1 -- 172.21.15.16:0/683881752 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 148) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f7460007430 con 0x7f74700cf4a0 2023-10-16T13:47:30.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:30.001+0000 7f745effd700 1 --2- 172.21.15.16:0/683881752 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f745006b140 0x7f745006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:30.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:30.001+0000 7f745effd700 1 -- 172.21.15.16:0/683881752 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(245..245 src has 1..245) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f7460088e60 con 0x7f74700cf4a0 2023-10-16T13:47:30.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:30.001+0000 7f746dd9b700 1 --2- 172.21.15.16:0/683881752 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f745006b140 0x7f745006d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:30.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:30.002+0000 7f746dd9b700 1 --2- 172.21.15.16:0/683881752 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f745006b140 0x7f745006d600 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f746400b340 tx=0x7f746401c040 comp rx=0 tx=0).ready entity=mgr.11886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:30.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:30.006+0000 7f745effd700 1 -- 172.21.15.16:0/683881752 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7460053a90 con 0x7f74700cf4a0 2023-10-16T13:47:30.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:30.204+0000 7f7474f83700 1 -- 172.21.15.16:0/683881752 --> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f747006b490 con 0x7f745006b140 2023-10-16T13:47:30.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:30.212+0000 7f745effd700 1 -- 172.21.15.16:0/683881752 <== mgr.11886 v2:172.21.15.16:6832/1736104576 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289879 (secure 0 0 0) 0x7f7464004030 con 0x7f745006b140 2023-10-16T13:47:30.218 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:47:30.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:30.215+0000 7f7474f83700 1 -- 172.21.15.16:0/683881752 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f745006b140 msgr2=0x7f745006d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:30.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:30.215+0000 7f7474f83700 1 --2- 172.21.15.16:0/683881752 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f745006b140 0x7f745006d600 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f746400b340 tx=0x7f746401c040 comp rx=0 tx=0).stop 2023-10-16T13:47:30.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:30.215+0000 7f7474f83700 1 -- 172.21.15.16:0/683881752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74700cf4a0 msgr2=0x7f747014d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:30.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:30.215+0000 7f7474f83700 1 --2- 172.21.15.16:0/683881752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74700cf4a0 0x7f747014d6a0 secure :-1 s=READY pgs=2762 cs=0 l=1 rev1=1 crypto rx=0x7f746000c490 tx=0x7f746000c7a0 comp rx=0 tx=0).stop 2023-10-16T13:47:30.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:30.216+0000 7f7474f83700 1 -- 172.21.15.16:0/683881752 shutdown_connections 2023-10-16T13:47:30.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:30.216+0000 7f7474f83700 1 --2- 172.21.15.16:0/683881752 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f745006b140 0x7f745006d600 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:30.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:30.216+0000 7f7474f83700 1 --2- 172.21.15.16:0/683881752 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f747015d910 0x7f74700ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:30.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:30.216+0000 7f7474f83700 1 --2- 172.21.15.16:0/683881752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74700cf4a0 0x7f747014d6a0 unknown :-1 s=CLOSED pgs=2762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:30.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:30.216+0000 7f7474f83700 1 --2- 172.21.15.16:0/683881752 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7470159cc0 0x7f74700ae8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:30.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:30.216+0000 7f7474f83700 1 -- 172.21.15.16:0/683881752 >> 172.21.15.16:0/683881752 conn(0x7f74700cc5e0 msgr2=0x7f747014f330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:30.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:30.216+0000 7f7474f83700 1 -- 172.21.15.16:0/683881752 shutdown_connections 2023-10-16T13:47:30.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:30.216+0000 7f7474f83700 1 -- 172.21.15.16:0/683881752 wait complete. 2023-10-16T13:47:30.223 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-10-16T13:47:30.230 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":31,"stamp":"2023-10-16T13:47:29.821921+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":663,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":408,"ondisk_log_size":408,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":152,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":431664,"kb_used_data":25352,"kb_used_omap":12,"kb_used_meta":406195,"kb_avail":754543056,"statfs":{"total":773094113280,"available":772652089344,"internally_reserved":0,"allocated":25960448,"data_stored":13957036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12740,"internal_metadata":415944252},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.409617"},"pg_stats":[{"pgid":"32.1f","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.418005+0000","last_change":"2023-10-16T13:47:24.429332+0000","last_active":"2023-10-16T13:47:25.418005+0000","last_peered":"2023-10-16T13:47:25.418005+0000","last_clean":"2023-10-16T13:47:25.418005+0000","last_became_active":"2023-10-16T13:47:24.428151+0000","last_became_peered":"2023-10-16T13:47:24.428151+0000","last_unstale":"2023-10-16T13:47:25.418005+0000","last_undegraded":"2023-10-16T13:47:25.418005+0000","last_fullsized":"2023-10-16T13:47:25.418005+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:51:49.567279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.1e","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.416620+0000","last_change":"2023-10-16T13:47:24.439610+0000","last_active":"2023-10-16T13:47:25.416620+0000","last_peered":"2023-10-16T13:47:25.416620+0000","last_clean":"2023-10-16T13:47:25.416620+0000","last_became_active":"2023-10-16T13:47:24.439212+0000","last_became_peered":"2023-10-16T13:47:24.439212+0000","last_unstale":"2023-10-16T13:47:25.416620+0000","last_undegraded":"2023-10-16T13:47:25.416620+0000","last_fullsized":"2023-10-16T13:47:25.416620+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:47:24.106272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1d","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100591+0000","last_change":"2023-10-16T13:47:24.432216+0000","last_active":"2023-10-16T13:47:26.100591+0000","last_peered":"2023-10-16T13:47:26.100591+0000","last_clean":"2023-10-16T13:47:26.100591+0000","last_became_active":"2023-10-16T13:47:24.431937+0000","last_became_peered":"2023-10-16T13:47:24.431937+0000","last_unstale":"2023-10-16T13:47:26.100591+0000","last_undegraded":"2023-10-16T13:47:26.100591+0000","last_fullsized":"2023-10-16T13:47:26.100591+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:23:10.580869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1c","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100430+0000","last_change":"2023-10-16T13:47:24.440252+0000","last_active":"2023-10-16T13:47:26.100430+0000","last_peered":"2023-10-16T13:47:26.100430+0000","last_clean":"2023-10-16T13:47:26.100430+0000","last_became_active":"2023-10-16T13:47:24.439982+0000","last_became_peered":"2023-10-16T13:47:24.439982+0000","last_unstale":"2023-10-16T13:47:26.100430+0000","last_undegraded":"2023-10-16T13:47:26.100430+0000","last_fullsized":"2023-10-16T13:47:26.100430+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:54:02.930019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1b","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.101595+0000","last_change":"2023-10-16T13:47:24.440049+0000","last_active":"2023-10-16T13:47:26.101595+0000","last_peered":"2023-10-16T13:47:26.101595+0000","last_clean":"2023-10-16T13:47:26.101595+0000","last_became_active":"2023-10-16T13:47:24.439750+0000","last_became_peered":"2023-10-16T13:47:24.439750+0000","last_unstale":"2023-10-16T13:47:26.101595+0000","last_undegraded":"2023-10-16T13:47:26.101595+0000","last_fullsized":"2023-10-16T13:47:26.101595+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:33:40.094175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1a","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.417711+0000","last_change":"2023-10-16T13:47:24.433661+0000","last_active":"2023-10-16T13:47:25.417711+0000","last_peered":"2023-10-16T13:47:25.417711+0000","last_clean":"2023-10-16T13:47:25.417711+0000","last_became_active":"2023-10-16T13:47:24.433417+0000","last_became_peered":"2023-10-16T13:47:24.433417+0000","last_unstale":"2023-10-16T13:47:25.417711+0000","last_undegraded":"2023-10-16T13:47:25.417711+0000","last_fullsized":"2023-10-16T13:47:25.417711+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:16:02.391378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.101505+0000","last_change":"2023-10-16T13:47:24.437673+0000","last_active":"2023-10-16T13:47:26.101505+0000","last_peered":"2023-10-16T13:47:26.101505+0000","last_clean":"2023-10-16T13:47:26.101505+0000","last_became_active":"2023-10-16T13:47:24.437479+0000","last_became_peered":"2023-10-16T13:47:24.437479+0000","last_unstale":"2023-10-16T13:47:26.101505+0000","last_undegraded":"2023-10-16T13:47:26.101505+0000","last_fullsized":"2023-10-16T13:47:26.101505+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:23:58.749586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.101429+0000","last_change":"2023-10-16T13:47:24.426442+0000","last_active":"2023-10-16T13:47:26.101429+0000","last_peered":"2023-10-16T13:47:26.101429+0000","last_clean":"2023-10-16T13:47:26.101429+0000","last_became_active":"2023-10-16T13:47:24.425929+0000","last_became_peered":"2023-10-16T13:47:24.425929+0000","last_unstale":"2023-10-16T13:47:26.101429+0000","last_undegraded":"2023-10-16T13:47:26.101429+0000","last_fullsized":"2023-10-16T13:47:26.101429+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:05:13.565304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.418008+0000","last_change":"2023-10-16T13:47:24.426459+0000","last_active":"2023-10-16T13:47:25.418008+0000","last_peered":"2023-10-16T13:47:25.418008+0000","last_clean":"2023-10-16T13:47:25.418008+0000","last_became_active":"2023-10-16T13:47:24.426169+0000","last_became_peered":"2023-10-16T13:47:24.426169+0000","last_unstale":"2023-10-16T13:47:25.418008+0000","last_undegraded":"2023-10-16T13:47:25.418008+0000","last_fullsized":"2023-10-16T13:47:25.418008+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:57:42.170954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100664+0000","last_change":"2023-10-16T13:47:24.427333+0000","last_active":"2023-10-16T13:47:26.100664+0000","last_peered":"2023-10-16T13:47:26.100664+0000","last_clean":"2023-10-16T13:47:26.100664+0000","last_became_active":"2023-10-16T13:47:24.427034+0000","last_became_peered":"2023-10-16T13:47:24.427034+0000","last_unstale":"2023-10-16T13:47:26.100664+0000","last_undegraded":"2023-10-16T13:47:26.100664+0000","last_fullsized":"2023-10-16T13:47:26.100664+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:27:20.181386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.15","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100502+0000","last_change":"2023-10-16T13:47:24.425903+0000","last_active":"2023-10-16T13:47:26.100502+0000","last_peered":"2023-10-16T13:47:26.100502+0000","last_clean":"2023-10-16T13:47:26.100502+0000","last_became_active":"2023-10-16T13:47:24.425593+0000","last_became_peered":"2023-10-16T13:47:24.425593+0000","last_unstale":"2023-10-16T13:47:26.100502+0000","last_undegraded":"2023-10-16T13:47:26.100502+0000","last_fullsized":"2023-10-16T13:47:26.100502+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:46:35.236231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.14","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.417564+0000","last_change":"2023-10-16T13:47:24.438927+0000","last_active":"2023-10-16T13:47:25.417564+0000","last_peered":"2023-10-16T13:47:25.417564+0000","last_clean":"2023-10-16T13:47:25.417564+0000","last_became_active":"2023-10-16T13:47:24.438389+0000","last_became_peered":"2023-10-16T13:47:24.438389+0000","last_unstale":"2023-10-16T13:47:25.417564+0000","last_undegraded":"2023-10-16T13:47:25.417564+0000","last_fullsized":"2023-10-16T13:47:25.417564+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:44:00.360485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.13","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100623+0000","last_change":"2023-10-16T13:47:24.445148+0000","last_active":"2023-10-16T13:47:26.100623+0000","last_peered":"2023-10-16T13:47:26.100623+0000","last_clean":"2023-10-16T13:47:26.100623+0000","last_became_active":"2023-10-16T13:47:24.444896+0000","last_became_peered":"2023-10-16T13:47:24.444896+0000","last_unstale":"2023-10-16T13:47:26.100623+0000","last_undegraded":"2023-10-16T13:47:26.100623+0000","last_fullsized":"2023-10-16T13:47:26.100623+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:35:13.886134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.099722+0000","last_change":"2023-10-16T13:47:24.437693+0000","last_active":"2023-10-16T13:47:26.099722+0000","last_peered":"2023-10-16T13:47:26.099722+0000","last_clean":"2023-10-16T13:47:26.099722+0000","last_became_active":"2023-10-16T13:47:24.437434+0000","last_became_peered":"2023-10-16T13:47:24.437434+0000","last_unstale":"2023-10-16T13:47:26.099722+0000","last_undegraded":"2023-10-16T13:47:26.099722+0000","last_fullsized":"2023-10-16T13:47:26.099722+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:47:56.221847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.101989+0000","last_change":"2023-10-16T13:47:24.435339+0000","last_active":"2023-10-16T13:47:26.101989+0000","last_peered":"2023-10-16T13:47:26.101989+0000","last_clean":"2023-10-16T13:47:26.101989+0000","last_became_active":"2023-10-16T13:47:24.434237+0000","last_became_peered":"2023-10-16T13:47:24.434237+0000","last_unstale":"2023-10-16T13:47:26.101989+0000","last_undegraded":"2023-10-16T13:47:26.101989+0000","last_fullsized":"2023-10-16T13:47:26.101989+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:21:21.083320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.10","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.099717+0000","last_change":"2023-10-16T13:47:24.441060+0000","last_active":"2023-10-16T13:47:26.099717+0000","last_peered":"2023-10-16T13:47:26.099717+0000","last_clean":"2023-10-16T13:47:26.099717+0000","last_became_active":"2023-10-16T13:47:24.440745+0000","last_became_peered":"2023-10-16T13:47:24.440745+0000","last_unstale":"2023-10-16T13:47:26.099717+0000","last_undegraded":"2023-10-16T13:47:26.099717+0000","last_fullsized":"2023-10-16T13:47:26.099717+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:48:26.805075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.f","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.101109+0000","last_change":"2023-10-16T13:47:24.438572+0000","last_active":"2023-10-16T13:47:26.101109+0000","last_peered":"2023-10-16T13:47:26.101109+0000","last_clean":"2023-10-16T13:47:26.101109+0000","last_became_active":"2023-10-16T13:47:24.438283+0000","last_became_peered":"2023-10-16T13:47:24.438283+0000","last_unstale":"2023-10-16T13:47:26.101109+0000","last_undegraded":"2023-10-16T13:47:26.101109+0000","last_fullsized":"2023-10-16T13:47:26.101109+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:48:21.499862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.e","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100703+0000","last_change":"2023-10-16T13:47:24.433506+0000","last_active":"2023-10-16T13:47:26.100703+0000","last_peered":"2023-10-16T13:47:26.100703+0000","last_clean":"2023-10-16T13:47:26.100703+0000","last_became_active":"2023-10-16T13:47:24.433218+0000","last_became_peered":"2023-10-16T13:47:24.433218+0000","last_unstale":"2023-10-16T13:47:26.100703+0000","last_undegraded":"2023-10-16T13:47:26.100703+0000","last_fullsized":"2023-10-16T13:47:26.100703+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:04:28.651883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.d","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.102503+0000","last_change":"2023-10-16T13:47:24.443171+0000","last_active":"2023-10-16T13:47:26.102503+0000","last_peered":"2023-10-16T13:47:26.102503+0000","last_clean":"2023-10-16T13:47:26.102503+0000","last_became_active":"2023-10-16T13:47:24.442811+0000","last_became_peered":"2023-10-16T13:47:24.442811+0000","last_unstale":"2023-10-16T13:47:26.102503+0000","last_undegraded":"2023-10-16T13:47:26.102503+0000","last_fullsized":"2023-10-16T13:47:26.102503+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:56:16.580840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.416732+0000","last_change":"2023-10-16T13:47:24.435000+0000","last_active":"2023-10-16T13:47:25.416732+0000","last_peered":"2023-10-16T13:47:25.416732+0000","last_clean":"2023-10-16T13:47:25.416732+0000","last_became_active":"2023-10-16T13:47:24.434568+0000","last_became_peered":"2023-10-16T13:47:24.434568+0000","last_unstale":"2023-10-16T13:47:25.416732+0000","last_undegraded":"2023-10-16T13:47:25.416732+0000","last_fullsized":"2023-10-16T13:47:25.416732+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:31:46.967495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.b","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100991+0000","last_change":"2023-10-16T13:47:24.432842+0000","last_active":"2023-10-16T13:47:26.100991+0000","last_peered":"2023-10-16T13:47:26.100991+0000","last_clean":"2023-10-16T13:47:26.100991+0000","last_became_active":"2023-10-16T13:47:24.432528+0000","last_became_peered":"2023-10-16T13:47:24.432528+0000","last_unstale":"2023-10-16T13:47:26.100991+0000","last_undegraded":"2023-10-16T13:47:26.100991+0000","last_fullsized":"2023-10-16T13:47:26.100991+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:10:57.744179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.101341+0000","last_change":"2023-10-16T13:47:24.439953+0000","last_active":"2023-10-16T13:47:26.101341+0000","last_peered":"2023-10-16T13:47:26.101341+0000","last_clean":"2023-10-16T13:47:26.101341+0000","last_became_active":"2023-10-16T13:47:24.439662+0000","last_became_peered":"2023-10-16T13:47:24.439662+0000","last_unstale":"2023-10-16T13:47:26.101341+0000","last_undegraded":"2023-10-16T13:47:26.101341+0000","last_fullsized":"2023-10-16T13:47:26.101341+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:16:33.663778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.102248+0000","last_change":"2023-10-16T13:47:24.442582+0000","last_active":"2023-10-16T13:47:26.102248+0000","last_peered":"2023-10-16T13:47:26.102248+0000","last_clean":"2023-10-16T13:47:26.102248+0000","last_became_active":"2023-10-16T13:47:24.442246+0000","last_became_peered":"2023-10-16T13:47:24.442246+0000","last_unstale":"2023-10-16T13:47:26.102248+0000","last_undegraded":"2023-10-16T13:47:26.102248+0000","last_fullsized":"2023-10-16T13:47:26.102248+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:48:26.805075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100231+0000","last_change":"2023-10-16T13:47:24.437336+0000","last_active":"2023-10-16T13:47:26.100231+0000","last_peered":"2023-10-16T13:47:26.100231+0000","last_clean":"2023-10-16T13:47:26.100231+0000","last_became_active":"2023-10-16T13:47:24.437042+0000","last_became_peered":"2023-10-16T13:47:24.437042+0000","last_unstale":"2023-10-16T13:47:26.100231+0000","last_undegraded":"2023-10-16T13:47:26.100231+0000","last_fullsized":"2023-10-16T13:47:26.100231+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:07:16.886792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.7","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.416641+0000","last_change":"2023-10-16T13:47:24.433031+0000","last_active":"2023-10-16T13:47:25.416641+0000","last_peered":"2023-10-16T13:47:25.416641+0000","last_clean":"2023-10-16T13:47:25.416641+0000","last_became_active":"2023-10-16T13:47:24.432700+0000","last_became_peered":"2023-10-16T13:47:24.432700+0000","last_unstale":"2023-10-16T13:47:25.416641+0000","last_undegraded":"2023-10-16T13:47:25.416641+0000","last_fullsized":"2023-10-16T13:47:25.416641+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:17:10.080681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.6","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.101710+0000","last_change":"2023-10-16T13:47:24.434972+0000","last_active":"2023-10-16T13:47:26.101710+0000","last_peered":"2023-10-16T13:47:26.101710+0000","last_clean":"2023-10-16T13:47:26.101710+0000","last_became_active":"2023-10-16T13:47:24.434639+0000","last_became_peered":"2023-10-16T13:47:24.434639+0000","last_unstale":"2023-10-16T13:47:26.101710+0000","last_undegraded":"2023-10-16T13:47:26.101710+0000","last_fullsized":"2023-10-16T13:47:26.101710+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:06:13.278420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.101557+0000","last_change":"2023-10-16T13:47:24.428801+0000","last_active":"2023-10-16T13:47:26.101557+0000","last_peered":"2023-10-16T13:47:26.101557+0000","last_clean":"2023-10-16T13:47:26.101557+0000","last_became_active":"2023-10-16T13:47:24.428511+0000","last_became_peered":"2023-10-16T13:47:24.428511+0000","last_unstale":"2023-10-16T13:47:26.101557+0000","last_undegraded":"2023-10-16T13:47:26.101557+0000","last_fullsized":"2023-10-16T13:47:26.101557+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:27:55.268379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.417855+0000","last_change":"2023-10-16T13:47:24.429772+0000","last_active":"2023-10-16T13:47:25.417855+0000","last_peered":"2023-10-16T13:47:25.417855+0000","last_clean":"2023-10-16T13:47:25.417855+0000","last_became_active":"2023-10-16T13:47:24.429496+0000","last_became_peered":"2023-10-16T13:47:24.429496+0000","last_unstale":"2023-10-16T13:47:25.417855+0000","last_undegraded":"2023-10-16T13:47:25.417855+0000","last_fullsized":"2023-10-16T13:47:25.417855+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:09:48.751438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.099824+0000","last_change":"2023-10-16T13:47:24.437263+0000","last_active":"2023-10-16T13:47:26.099824+0000","last_peered":"2023-10-16T13:47:26.099824+0000","last_clean":"2023-10-16T13:47:26.099824+0000","last_became_active":"2023-10-16T13:47:24.437023+0000","last_became_peered":"2023-10-16T13:47:24.437023+0000","last_unstale":"2023-10-16T13:47:26.099824+0000","last_undegraded":"2023-10-16T13:47:26.099824+0000","last_fullsized":"2023-10-16T13:47:26.099824+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:59:15.061907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.099775+0000","last_change":"2023-10-16T13:47:24.419847+0000","last_active":"2023-10-16T13:47:26.099775+0000","last_peered":"2023-10-16T13:47:26.099775+0000","last_clean":"2023-10-16T13:47:26.099775+0000","last_became_active":"2023-10-16T13:47:24.419520+0000","last_became_peered":"2023-10-16T13:47:24.419520+0000","last_unstale":"2023-10-16T13:47:26.099775+0000","last_undegraded":"2023-10-16T13:47:26.099775+0000","last_fullsized":"2023-10-16T13:47:26.099775+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:11:44.932011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.416267+0000","last_change":"2023-10-16T13:47:24.435926+0000","last_active":"2023-10-16T13:47:25.416267+0000","last_peered":"2023-10-16T13:47:25.416267+0000","last_clean":"2023-10-16T13:47:25.416267+0000","last_became_active":"2023-10-16T13:47:24.434753+0000","last_became_peered":"2023-10-16T13:47:24.434753+0000","last_unstale":"2023-10-16T13:47:25.416267+0000","last_undegraded":"2023-10-16T13:47:25.416267+0000","last_fullsized":"2023-10-16T13:47:25.416267+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:13:51.332934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.417066+0000","last_change":"2023-10-16T13:47:24.423908+0000","last_active":"2023-10-16T13:47:25.417066+0000","last_peered":"2023-10-16T13:47:25.417066+0000","last_clean":"2023-10-16T13:47:25.417066+0000","last_became_active":"2023-10-16T13:47:24.423530+0000","last_became_peered":"2023-10-16T13:47:24.423530+0000","last_unstale":"2023-10-16T13:47:25.417066+0000","last_undegraded":"2023-10-16T13:47:25.417066+0000","last_fullsized":"2023-10-16T13:47:25.417066+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:50:50.396650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.20","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.416162+0000","last_change":"2023-10-16T13:47:24.436480+0000","last_active":"2023-10-16T13:47:25.416162+0000","last_peered":"2023-10-16T13:47:25.416162+0000","last_clean":"2023-10-16T13:47:25.416162+0000","last_became_active":"2023-10-16T13:47:24.435309+0000","last_became_peered":"2023-10-16T13:47:24.435309+0000","last_unstale":"2023-10-16T13:47:25.416162+0000","last_undegraded":"2023-10-16T13:47:25.416162+0000","last_fullsized":"2023-10-16T13:47:25.416162+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:28:08.947506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1f","version":"244'4","reported_seq":16,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.452436+0000","last_change":"2023-10-16T13:47:23.425349+0000","last_active":"2023-10-16T13:47:24.452436+0000","last_peered":"2023-10-16T13:47:24.452436+0000","last_clean":"2023-10-16T13:47:24.452436+0000","last_became_active":"2023-10-16T13:47:23.425029+0000","last_became_peered":"2023-10-16T13:47:23.425029+0000","last_unstale":"2023-10-16T13:47:24.452436+0000","last_undegraded":"2023-10-16T13:47:24.452436+0000","last_fullsized":"2023-10-16T13:47:24.452436+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:57:35.834032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.22","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.102212+0000","last_change":"2023-10-16T13:47:24.431062+0000","last_active":"2023-10-16T13:47:26.102212+0000","last_peered":"2023-10-16T13:47:26.102212+0000","last_clean":"2023-10-16T13:47:26.102212+0000","last_became_active":"2023-10-16T13:47:24.430771+0000","last_became_peered":"2023-10-16T13:47:24.430771+0000","last_unstale":"2023-10-16T13:47:26.102212+0000","last_undegraded":"2023-10-16T13:47:26.102212+0000","last_fullsized":"2023-10-16T13:47:26.102212+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:23:48.085785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.1d","version":"244'2","reported_seq":14,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.448937+0000","last_change":"2023-10-16T13:47:23.428572+0000","last_active":"2023-10-16T13:47:24.448937+0000","last_peered":"2023-10-16T13:47:24.448937+0000","last_clean":"2023-10-16T13:47:24.448937+0000","last_became_active":"2023-10-16T13:47:23.428285+0000","last_became_peered":"2023-10-16T13:47:23.428285+0000","last_unstale":"2023-10-16T13:47:24.448937+0000","last_undegraded":"2023-10-16T13:47:24.448937+0000","last_fullsized":"2023-10-16T13:47:24.448937+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:26:03.650516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.23","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.417464+0000","last_change":"2023-10-16T13:47:24.432436+0000","last_active":"2023-10-16T13:47:25.417464+0000","last_peered":"2023-10-16T13:47:25.417464+0000","last_clean":"2023-10-16T13:47:25.417464+0000","last_became_active":"2023-10-16T13:47:24.428465+0000","last_became_peered":"2023-10-16T13:47:24.428465+0000","last_unstale":"2023-10-16T13:47:25.417464+0000","last_undegraded":"2023-10-16T13:47:25.417464+0000","last_fullsized":"2023-10-16T13:47:25.417464+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:16:50.560075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1c","version":"244'2","reported_seq":14,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.447290+0000","last_change":"2023-10-16T13:47:23.427686+0000","last_active":"2023-10-16T13:47:24.447290+0000","last_peered":"2023-10-16T13:47:24.447290+0000","last_clean":"2023-10-16T13:47:24.447290+0000","last_became_active":"2023-10-16T13:47:23.427013+0000","last_became_peered":"2023-10-16T13:47:23.427013+0000","last_unstale":"2023-10-16T13:47:24.447290+0000","last_undegraded":"2023-10-16T13:47:24.447290+0000","last_fullsized":"2023-10-16T13:47:24.447290+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:37:17.532164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.24","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100065+0000","last_change":"2023-10-16T13:47:24.438644+0000","last_active":"2023-10-16T13:47:26.100065+0000","last_peered":"2023-10-16T13:47:26.100065+0000","last_clean":"2023-10-16T13:47:26.100065+0000","last_became_active":"2023-10-16T13:47:24.438344+0000","last_became_peered":"2023-10-16T13:47:24.438344+0000","last_unstale":"2023-10-16T13:47:26.100065+0000","last_undegraded":"2023-10-16T13:47:26.100065+0000","last_fullsized":"2023-10-16T13:47:26.100065+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:21:19.269920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.1b","version":"244'1","reported_seq":13,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.448817+0000","last_change":"2023-10-16T13:47:23.427234+0000","last_active":"2023-10-16T13:47:24.448817+0000","last_peered":"2023-10-16T13:47:24.448817+0000","last_clean":"2023-10-16T13:47:24.448817+0000","last_became_active":"2023-10-16T13:47:23.426948+0000","last_became_peered":"2023-10-16T13:47:23.426948+0000","last_unstale":"2023-10-16T13:47:24.448817+0000","last_undegraded":"2023-10-16T13:47:24.448817+0000","last_fullsized":"2023-10-16T13:47:24.448817+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:11:43.927649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.25","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100878+0000","last_change":"2023-10-16T13:47:24.432803+0000","last_active":"2023-10-16T13:47:26.100878+0000","last_peered":"2023-10-16T13:47:26.100878+0000","last_clean":"2023-10-16T13:47:26.100878+0000","last_became_active":"2023-10-16T13:47:24.432470+0000","last_became_peered":"2023-10-16T13:47:24.432470+0000","last_unstale":"2023-10-16T13:47:26.100878+0000","last_undegraded":"2023-10-16T13:47:26.100878+0000","last_fullsized":"2023-10-16T13:47:26.100878+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:41:00.039237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.1a","version":"244'1","reported_seq":13,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.441041+0000","last_change":"2023-10-16T13:47:23.425695+0000","last_active":"2023-10-16T13:47:24.441041+0000","last_peered":"2023-10-16T13:47:24.441041+0000","last_clean":"2023-10-16T13:47:24.441041+0000","last_became_active":"2023-10-16T13:47:23.425407+0000","last_became_peered":"2023-10-16T13:47:23.425407+0000","last_unstale":"2023-10-16T13:47:24.441041+0000","last_undegraded":"2023-10-16T13:47:24.441041+0000","last_fullsized":"2023-10-16T13:47:24.441041+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:19:34.313200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.26","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100538+0000","last_change":"2023-10-16T13:47:24.439616+0000","last_active":"2023-10-16T13:47:26.100538+0000","last_peered":"2023-10-16T13:47:26.100538+0000","last_clean":"2023-10-16T13:47:26.100538+0000","last_became_active":"2023-10-16T13:47:24.435678+0000","last_became_peered":"2023-10-16T13:47:24.435678+0000","last_unstale":"2023-10-16T13:47:26.100538+0000","last_undegraded":"2023-10-16T13:47:26.100538+0000","last_fullsized":"2023-10-16T13:47:26.100538+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:39:16.296666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.418484+0000","last_change":"2023-10-16T13:47:23.431681+0000","last_active":"2023-10-16T13:47:24.418484+0000","last_peered":"2023-10-16T13:47:24.418484+0000","last_clean":"2023-10-16T13:47:24.418484+0000","last_became_active":"2023-10-16T13:47:23.431317+0000","last_became_peered":"2023-10-16T13:47:23.431317+0000","last_unstale":"2023-10-16T13:47:24.418484+0000","last_undegraded":"2023-10-16T13:47:24.418484+0000","last_fullsized":"2023-10-16T13:47:24.418484+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:20:27.852907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.27","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.101829+0000","last_change":"2023-10-16T13:47:24.423771+0000","last_active":"2023-10-16T13:47:26.101829+0000","last_peered":"2023-10-16T13:47:26.101829+0000","last_clean":"2023-10-16T13:47:26.101829+0000","last_became_active":"2023-10-16T13:47:24.423459+0000","last_became_peered":"2023-10-16T13:47:24.423459+0000","last_unstale":"2023-10-16T13:47:26.101829+0000","last_undegraded":"2023-10-16T13:47:26.101829+0000","last_fullsized":"2023-10-16T13:47:26.101829+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:27:22.393558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.421767+0000","last_change":"2023-10-16T13:47:23.433310+0000","last_active":"2023-10-16T13:47:24.421767+0000","last_peered":"2023-10-16T13:47:24.421767+0000","last_clean":"2023-10-16T13:47:24.421767+0000","last_became_active":"2023-10-16T13:47:23.433018+0000","last_became_peered":"2023-10-16T13:47:23.433018+0000","last_unstale":"2023-10-16T13:47:24.421767+0000","last_undegraded":"2023-10-16T13:47:24.421767+0000","last_fullsized":"2023-10-16T13:47:24.421767+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:49:45.992161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.28","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100635+0000","last_change":"2023-10-16T13:47:24.433316+0000","last_active":"2023-10-16T13:47:26.100635+0000","last_peered":"2023-10-16T13:47:26.100635+0000","last_clean":"2023-10-16T13:47:26.100635+0000","last_became_active":"2023-10-16T13:47:24.429160+0000","last_became_peered":"2023-10-16T13:47:24.429160+0000","last_unstale":"2023-10-16T13:47:26.100635+0000","last_undegraded":"2023-10-16T13:47:26.100635+0000","last_fullsized":"2023-10-16T13:47:26.100635+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:12:35.760185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.415527+0000","last_change":"2023-10-16T13:47:23.428721+0000","last_active":"2023-10-16T13:47:24.415527+0000","last_peered":"2023-10-16T13:47:24.415527+0000","last_clean":"2023-10-16T13:47:24.415527+0000","last_became_active":"2023-10-16T13:47:23.428349+0000","last_became_peered":"2023-10-16T13:47:23.428349+0000","last_unstale":"2023-10-16T13:47:24.415527+0000","last_undegraded":"2023-10-16T13:47:24.415527+0000","last_fullsized":"2023-10-16T13:47:24.415527+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:59:11.116638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.29","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100592+0000","last_change":"2023-10-16T13:47:24.440635+0000","last_active":"2023-10-16T13:47:26.100592+0000","last_peered":"2023-10-16T13:47:26.100592+0000","last_clean":"2023-10-16T13:47:26.100592+0000","last_became_active":"2023-10-16T13:47:24.440223+0000","last_became_peered":"2023-10-16T13:47:24.440223+0000","last_unstale":"2023-10-16T13:47:26.100592+0000","last_undegraded":"2023-10-16T13:47:26.100592+0000","last_fullsized":"2023-10-16T13:47:26.100592+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:22:53.895897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.421423+0000","last_change":"2023-10-16T13:47:23.433545+0000","last_active":"2023-10-16T13:47:24.421423+0000","last_peered":"2023-10-16T13:47:24.421423+0000","last_clean":"2023-10-16T13:47:24.421423+0000","last_became_active":"2023-10-16T13:47:23.433275+0000","last_became_peered":"2023-10-16T13:47:23.433275+0000","last_unstale":"2023-10-16T13:47:24.421423+0000","last_undegraded":"2023-10-16T13:47:24.421423+0000","last_fullsized":"2023-10-16T13:47:24.421423+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:21:17.525717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.2a","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100402+0000","last_change":"2023-10-16T13:47:24.424605+0000","last_active":"2023-10-16T13:47:26.100402+0000","last_peered":"2023-10-16T13:47:26.100402+0000","last_clean":"2023-10-16T13:47:26.100402+0000","last_became_active":"2023-10-16T13:47:24.424313+0000","last_became_peered":"2023-10-16T13:47:24.424313+0000","last_unstale":"2023-10-16T13:47:26.100402+0000","last_undegraded":"2023-10-16T13:47:26.100402+0000","last_fullsized":"2023-10-16T13:47:26.100402+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:40:33.368782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.15","version":"244'3","reported_seq":15,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.446109+0000","last_change":"2023-10-16T13:47:23.431025+0000","last_active":"2023-10-16T13:47:24.446109+0000","last_peered":"2023-10-16T13:47:24.446109+0000","last_clean":"2023-10-16T13:47:24.446109+0000","last_became_active":"2023-10-16T13:47:23.430710+0000","last_became_peered":"2023-10-16T13:47:23.430710+0000","last_unstale":"2023-10-16T13:47:24.446109+0000","last_undegraded":"2023-10-16T13:47:24.446109+0000","last_fullsized":"2023-10-16T13:47:24.446109+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:52:27.207432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2b","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100831+0000","last_change":"2023-10-16T13:47:24.427810+0000","last_active":"2023-10-16T13:47:26.100831+0000","last_peered":"2023-10-16T13:47:26.100831+0000","last_clean":"2023-10-16T13:47:26.100831+0000","last_became_active":"2023-10-16T13:47:24.427276+0000","last_became_peered":"2023-10-16T13:47:24.427276+0000","last_unstale":"2023-10-16T13:47:26.100831+0000","last_undegraded":"2023-10-16T13:47:26.100831+0000","last_fullsized":"2023-10-16T13:47:26.100831+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:16:12.551854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.14","version":"244'6","reported_seq":18,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.453158+0000","last_change":"2023-10-16T13:47:23.430528+0000","last_active":"2023-10-16T13:47:24.453158+0000","last_peered":"2023-10-16T13:47:24.453158+0000","last_clean":"2023-10-16T13:47:24.453158+0000","last_became_active":"2023-10-16T13:47:23.429721+0000","last_became_peered":"2023-10-16T13:47:23.429721+0000","last_unstale":"2023-10-16T13:47:24.453158+0000","last_undegraded":"2023-10-16T13:47:24.453158+0000","last_fullsized":"2023-10-16T13:47:24.453158+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:07:56.379842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.2c","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.416579+0000","last_change":"2023-10-16T13:47:24.436132+0000","last_active":"2023-10-16T13:47:25.416579+0000","last_peered":"2023-10-16T13:47:25.416579+0000","last_clean":"2023-10-16T13:47:25.416579+0000","last_became_active":"2023-10-16T13:47:24.435272+0000","last_became_peered":"2023-10-16T13:47:24.435272+0000","last_unstale":"2023-10-16T13:47:25.416579+0000","last_undegraded":"2023-10-16T13:47:25.416579+0000","last_fullsized":"2023-10-16T13:47:25.416579+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:07:59.434149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.13","version":"244'2","reported_seq":14,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.450104+0000","last_change":"2023-10-16T13:47:23.426620+0000","last_active":"2023-10-16T13:47:24.450104+0000","last_peered":"2023-10-16T13:47:24.450104+0000","last_clean":"2023-10-16T13:47:24.450104+0000","last_became_active":"2023-10-16T13:47:23.426328+0000","last_became_peered":"2023-10-16T13:47:23.426328+0000","last_unstale":"2023-10-16T13:47:24.450104+0000","last_undegraded":"2023-10-16T13:47:24.450104+0000","last_fullsized":"2023-10-16T13:47:24.450104+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:23:39.805153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.2d","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100631+0000","last_change":"2023-10-16T13:47:24.431734+0000","last_active":"2023-10-16T13:47:26.100631+0000","last_peered":"2023-10-16T13:47:26.100631+0000","last_clean":"2023-10-16T13:47:26.100631+0000","last_became_active":"2023-10-16T13:47:24.431414+0000","last_became_peered":"2023-10-16T13:47:24.431414+0000","last_unstale":"2023-10-16T13:47:26.100631+0000","last_undegraded":"2023-10-16T13:47:26.100631+0000","last_fullsized":"2023-10-16T13:47:26.100631+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:01:31.790432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.418559+0000","last_change":"2023-10-16T13:47:23.433222+0000","last_active":"2023-10-16T13:47:24.418559+0000","last_peered":"2023-10-16T13:47:24.418559+0000","last_clean":"2023-10-16T13:47:24.418559+0000","last_became_active":"2023-10-16T13:47:23.432936+0000","last_became_peered":"2023-10-16T13:47:23.432936+0000","last_unstale":"2023-10-16T13:47:24.418559+0000","last_undegraded":"2023-10-16T13:47:24.418559+0000","last_fullsized":"2023-10-16T13:47:24.418559+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:47:59.336633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2e","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.102342+0000","last_change":"2023-10-16T13:47:24.438281+0000","last_active":"2023-10-16T13:47:26.102342+0000","last_peered":"2023-10-16T13:47:26.102342+0000","last_clean":"2023-10-16T13:47:26.102342+0000","last_became_active":"2023-10-16T13:47:24.437976+0000","last_became_peered":"2023-10-16T13:47:24.437976+0000","last_unstale":"2023-10-16T13:47:26.102342+0000","last_undegraded":"2023-10-16T13:47:26.102342+0000","last_fullsized":"2023-10-16T13:47:26.102342+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:19:38.293171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.417965+0000","last_change":"2023-10-16T13:47:23.425872+0000","last_active":"2023-10-16T13:47:24.417965+0000","last_peered":"2023-10-16T13:47:24.417965+0000","last_clean":"2023-10-16T13:47:24.417965+0000","last_became_active":"2023-10-16T13:47:23.425569+0000","last_became_peered":"2023-10-16T13:47:23.425569+0000","last_unstale":"2023-10-16T13:47:24.417965+0000","last_undegraded":"2023-10-16T13:47:24.417965+0000","last_fullsized":"2023-10-16T13:47:24.417965+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:09:18.086723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2f","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.101809+0000","last_change":"2023-10-16T13:47:24.437030+0000","last_active":"2023-10-16T13:47:26.101809+0000","last_peered":"2023-10-16T13:47:26.101809+0000","last_clean":"2023-10-16T13:47:26.101809+0000","last_became_active":"2023-10-16T13:47:24.436734+0000","last_became_peered":"2023-10-16T13:47:24.436734+0000","last_unstale":"2023-10-16T13:47:26.101809+0000","last_undegraded":"2023-10-16T13:47:26.101809+0000","last_fullsized":"2023-10-16T13:47:26.101809+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:28:53.363293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.10","version":"244'8","reported_seq":20,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.453887+0000","last_change":"2023-10-16T13:47:23.426967+0000","last_active":"2023-10-16T13:47:24.453887+0000","last_peered":"2023-10-16T13:47:24.453887+0000","last_clean":"2023-10-16T13:47:24.453887+0000","last_became_active":"2023-10-16T13:47:23.426614+0000","last_became_peered":"2023-10-16T13:47:23.426614+0000","last_unstale":"2023-10-16T13:47:24.453887+0000","last_undegraded":"2023-10-16T13:47:24.453887+0000","last_fullsized":"2023-10-16T13:47:24.453887+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:39:12.211841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.30","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.101224+0000","last_change":"2023-10-16T13:47:24.437081+0000","last_active":"2023-10-16T13:47:26.101224+0000","last_peered":"2023-10-16T13:47:26.101224+0000","last_clean":"2023-10-16T13:47:26.101224+0000","last_became_active":"2023-10-16T13:47:24.436501+0000","last_became_peered":"2023-10-16T13:47:24.436501+0000","last_unstale":"2023-10-16T13:47:26.101224+0000","last_undegraded":"2023-10-16T13:47:26.101224+0000","last_fullsized":"2023-10-16T13:47:26.101224+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:23:30.831964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.f","version":"244'1","reported_seq":13,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.445592+0000","last_change":"2023-10-16T13:47:23.432161+0000","last_active":"2023-10-16T13:47:24.445592+0000","last_peered":"2023-10-16T13:47:24.445592+0000","last_clean":"2023-10-16T13:47:24.445592+0000","last_became_active":"2023-10-16T13:47:23.431822+0000","last_became_peered":"2023-10-16T13:47:23.431822+0000","last_unstale":"2023-10-16T13:47:24.445592+0000","last_undegraded":"2023-10-16T13:47:24.445592+0000","last_fullsized":"2023-10-16T13:47:24.445592+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:15:50.867051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.31","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100923+0000","last_change":"2023-10-16T13:47:24.431186+0000","last_active":"2023-10-16T13:47:26.100923+0000","last_peered":"2023-10-16T13:47:26.100923+0000","last_clean":"2023-10-16T13:47:26.100923+0000","last_became_active":"2023-10-16T13:47:24.430892+0000","last_became_peered":"2023-10-16T13:47:24.430892+0000","last_unstale":"2023-10-16T13:47:26.100923+0000","last_undegraded":"2023-10-16T13:47:26.100923+0000","last_fullsized":"2023-10-16T13:47:26.100923+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:12:21.693658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.e","version":"244'2","reported_seq":14,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.448939+0000","last_change":"2023-10-16T13:47:23.424549+0000","last_active":"2023-10-16T13:47:24.448939+0000","last_peered":"2023-10-16T13:47:24.448939+0000","last_clean":"2023-10-16T13:47:24.448939+0000","last_became_active":"2023-10-16T13:47:23.424111+0000","last_became_peered":"2023-10-16T13:47:23.424111+0000","last_unstale":"2023-10-16T13:47:24.448939+0000","last_undegraded":"2023-10-16T13:47:24.448939+0000","last_fullsized":"2023-10-16T13:47:24.448939+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:36:35.137200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.32","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.416229+0000","last_change":"2023-10-16T13:47:24.432940+0000","last_active":"2023-10-16T13:47:25.416229+0000","last_peered":"2023-10-16T13:47:25.416229+0000","last_clean":"2023-10-16T13:47:25.416229+0000","last_became_active":"2023-10-16T13:47:24.432610+0000","last_became_peered":"2023-10-16T13:47:24.432610+0000","last_unstale":"2023-10-16T13:47:25.416229+0000","last_undegraded":"2023-10-16T13:47:25.416229+0000","last_fullsized":"2023-10-16T13:47:25.416229+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:59:08.104923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.d","version":"244'1","reported_seq":13,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.448969+0000","last_change":"2023-10-16T13:47:23.428449+0000","last_active":"2023-10-16T13:47:24.448969+0000","last_peered":"2023-10-16T13:47:24.448969+0000","last_clean":"2023-10-16T13:47:24.448969+0000","last_became_active":"2023-10-16T13:47:23.428130+0000","last_became_peered":"2023-10-16T13:47:23.428130+0000","last_unstale":"2023-10-16T13:47:24.448969+0000","last_undegraded":"2023-10-16T13:47:24.448969+0000","last_fullsized":"2023-10-16T13:47:24.448969+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:24:27.601821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.33","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100188+0000","last_change":"2023-10-16T13:47:24.427239+0000","last_active":"2023-10-16T13:47:26.100188+0000","last_peered":"2023-10-16T13:47:26.100188+0000","last_clean":"2023-10-16T13:47:26.100188+0000","last_became_active":"2023-10-16T13:47:24.426919+0000","last_became_peered":"2023-10-16T13:47:24.426919+0000","last_unstale":"2023-10-16T13:47:26.100188+0000","last_undegraded":"2023-10-16T13:47:26.100188+0000","last_fullsized":"2023-10-16T13:47:26.100188+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:56:16.580840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.425656+0000","last_change":"2023-10-16T13:47:23.427051+0000","last_active":"2023-10-16T13:47:24.425656+0000","last_peered":"2023-10-16T13:47:24.425656+0000","last_clean":"2023-10-16T13:47:24.425656+0000","last_became_active":"2023-10-16T13:47:23.426741+0000","last_became_peered":"2023-10-16T13:47:23.426741+0000","last_unstale":"2023-10-16T13:47:24.425656+0000","last_undegraded":"2023-10-16T13:47:24.425656+0000","last_fullsized":"2023-10-16T13:47:24.425656+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:37:17.532164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.34","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100514+0000","last_change":"2023-10-16T13:47:24.444364+0000","last_active":"2023-10-16T13:47:26.100514+0000","last_peered":"2023-10-16T13:47:26.100514+0000","last_clean":"2023-10-16T13:47:26.100514+0000","last_became_active":"2023-10-16T13:47:24.444045+0000","last_became_peered":"2023-10-16T13:47:24.444045+0000","last_unstale":"2023-10-16T13:47:26.100514+0000","last_undegraded":"2023-10-16T13:47:26.100514+0000","last_fullsized":"2023-10-16T13:47:26.100514+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:13:11.795921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.b","version":"244'2","reported_seq":14,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.449802+0000","last_change":"2023-10-16T13:47:23.434957+0000","last_active":"2023-10-16T13:47:24.449802+0000","last_peered":"2023-10-16T13:47:24.449802+0000","last_clean":"2023-10-16T13:47:24.449802+0000","last_became_active":"2023-10-16T13:47:23.434637+0000","last_became_peered":"2023-10-16T13:47:23.434637+0000","last_unstale":"2023-10-16T13:47:24.449802+0000","last_undegraded":"2023-10-16T13:47:24.449802+0000","last_fullsized":"2023-10-16T13:47:24.449802+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:23:47.081423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.35","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.416431+0000","last_change":"2023-10-16T13:47:24.431892+0000","last_active":"2023-10-16T13:47:25.416431+0000","last_peered":"2023-10-16T13:47:25.416431+0000","last_clean":"2023-10-16T13:47:25.416431+0000","last_became_active":"2023-10-16T13:47:24.431624+0000","last_became_peered":"2023-10-16T13:47:24.431624+0000","last_unstale":"2023-10-16T13:47:25.416431+0000","last_undegraded":"2023-10-16T13:47:25.416431+0000","last_fullsized":"2023-10-16T13:47:25.416431+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:49:13.422163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.441201+0000","last_change":"2023-10-16T13:47:23.435342+0000","last_active":"2023-10-16T13:47:24.441201+0000","last_peered":"2023-10-16T13:47:24.441201+0000","last_clean":"2023-10-16T13:47:24.441201+0000","last_became_active":"2023-10-16T13:47:23.434999+0000","last_became_peered":"2023-10-16T13:47:23.434999+0000","last_unstale":"2023-10-16T13:47:24.441201+0000","last_undegraded":"2023-10-16T13:47:24.441201+0000","last_fullsized":"2023-10-16T13:47:24.441201+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:20:03.252858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.36","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100687+0000","last_change":"2023-10-16T13:47:24.435751+0000","last_active":"2023-10-16T13:47:26.100687+0000","last_peered":"2023-10-16T13:47:26.100687+0000","last_clean":"2023-10-16T13:47:26.100687+0000","last_became_active":"2023-10-16T13:47:24.434491+0000","last_became_peered":"2023-10-16T13:47:24.434491+0000","last_unstale":"2023-10-16T13:47:26.100687+0000","last_undegraded":"2023-10-16T13:47:26.100687+0000","last_fullsized":"2023-10-16T13:47:26.100687+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:02:56.979689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.440901+0000","last_change":"2023-10-16T13:47:23.427022+0000","last_active":"2023-10-16T13:47:24.440901+0000","last_peered":"2023-10-16T13:47:24.440901+0000","last_clean":"2023-10-16T13:47:24.440901+0000","last_became_active":"2023-10-16T13:47:23.426682+0000","last_became_peered":"2023-10-16T13:47:23.426682+0000","last_unstale":"2023-10-16T13:47:24.440901+0000","last_undegraded":"2023-10-16T13:47:24.440901+0000","last_fullsized":"2023-10-16T13:47:24.440901+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:05:08.936096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.37","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100950+0000","last_change":"2023-10-16T13:47:24.436357+0000","last_active":"2023-10-16T13:47:26.100950+0000","last_peered":"2023-10-16T13:47:26.100950+0000","last_clean":"2023-10-16T13:47:26.100950+0000","last_became_active":"2023-10-16T13:47:24.436050+0000","last_became_peered":"2023-10-16T13:47:24.436050+0000","last_unstale":"2023-10-16T13:47:26.100950+0000","last_undegraded":"2023-10-16T13:47:26.100950+0000","last_fullsized":"2023-10-16T13:47:26.100950+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:15:49.933818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.415241+0000","last_change":"2023-10-16T13:47:23.433223+0000","last_active":"2023-10-16T13:47:24.415241+0000","last_peered":"2023-10-16T13:47:24.415241+0000","last_clean":"2023-10-16T13:47:24.415241+0000","last_became_active":"2023-10-16T13:47:23.432935+0000","last_became_peered":"2023-10-16T13:47:23.432935+0000","last_unstale":"2023-10-16T13:47:24.415241+0000","last_undegraded":"2023-10-16T13:47:24.415241+0000","last_fullsized":"2023-10-16T13:47:24.415241+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:40:59.034875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.38","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100456+0000","last_change":"2023-10-16T13:47:24.437795+0000","last_active":"2023-10-16T13:47:26.100456+0000","last_peered":"2023-10-16T13:47:26.100456+0000","last_clean":"2023-10-16T13:47:26.100456+0000","last_became_active":"2023-10-16T13:47:24.437493+0000","last_became_peered":"2023-10-16T13:47:24.437493+0000","last_unstale":"2023-10-16T13:47:26.100456+0000","last_undegraded":"2023-10-16T13:47:26.100456+0000","last_fullsized":"2023-10-16T13:47:26.100456+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:32:36.771595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.7","version":"244'2","reported_seq":14,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.451763+0000","last_change":"2023-10-16T13:47:23.430865+0000","last_active":"2023-10-16T13:47:24.451763+0000","last_peered":"2023-10-16T13:47:24.451763+0000","last_clean":"2023-10-16T13:47:24.451763+0000","last_became_active":"2023-10-16T13:47:23.430493+0000","last_became_peered":"2023-10-16T13:47:23.430493+0000","last_unstale":"2023-10-16T13:47:24.451763+0000","last_undegraded":"2023-10-16T13:47:24.451763+0000","last_fullsized":"2023-10-16T13:47:24.451763+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:59:14.057546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.39","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.101988+0000","last_change":"2023-10-16T13:47:24.443917+0000","last_active":"2023-10-16T13:47:26.101988+0000","last_peered":"2023-10-16T13:47:26.101988+0000","last_clean":"2023-10-16T13:47:26.101988+0000","last_became_active":"2023-10-16T13:47:24.443547+0000","last_became_peered":"2023-10-16T13:47:24.443547+0000","last_unstale":"2023-10-16T13:47:26.101988+0000","last_undegraded":"2023-10-16T13:47:26.101988+0000","last_fullsized":"2023-10-16T13:47:26.101988+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:21:18.530078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.417912+0000","last_change":"2023-10-16T13:47:23.430851+0000","last_active":"2023-10-16T13:47:24.417912+0000","last_peered":"2023-10-16T13:47:24.417912+0000","last_clean":"2023-10-16T13:47:24.417912+0000","last_became_active":"2023-10-16T13:47:23.430526+0000","last_became_peered":"2023-10-16T13:47:23.430526+0000","last_unstale":"2023-10-16T13:47:24.417912+0000","last_undegraded":"2023-10-16T13:47:24.417912+0000","last_fullsized":"2023-10-16T13:47:24.417912+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:40:16.878541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.3a","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100155+0000","last_change":"2023-10-16T13:47:24.440347+0000","last_active":"2023-10-16T13:47:26.100155+0000","last_peered":"2023-10-16T13:47:26.100155+0000","last_clean":"2023-10-16T13:47:26.100155+0000","last_became_active":"2023-10-16T13:47:24.440041+0000","last_became_peered":"2023-10-16T13:47:24.440041+0000","last_unstale":"2023-10-16T13:47:26.100155+0000","last_undegraded":"2023-10-16T13:47:26.100155+0000","last_fullsized":"2023-10-16T13:47:26.100155+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:19:38.293171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.415325+0000","last_change":"2023-10-16T13:47:23.427399+0000","last_active":"2023-10-16T13:47:24.415325+0000","last_peered":"2023-10-16T13:47:24.415325+0000","last_clean":"2023-10-16T13:47:24.415325+0000","last_became_active":"2023-10-16T13:47:23.427092+0000","last_became_peered":"2023-10-16T13:47:23.427092+0000","last_unstale":"2023-10-16T13:47:24.415325+0000","last_undegraded":"2023-10-16T13:47:24.415325+0000","last_fullsized":"2023-10-16T13:47:24.415325+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:10:56.739817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3b","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.101556+0000","last_change":"2023-10-16T13:47:24.436718+0000","last_active":"2023-10-16T13:47:26.101556+0000","last_peered":"2023-10-16T13:47:26.101556+0000","last_clean":"2023-10-16T13:47:26.101556+0000","last_became_active":"2023-10-16T13:47:24.436371+0000","last_became_peered":"2023-10-16T13:47:24.436371+0000","last_unstale":"2023-10-16T13:47:26.101556+0000","last_undegraded":"2023-10-16T13:47:26.101556+0000","last_fullsized":"2023-10-16T13:47:26.101556+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:58:12.134727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.416043+0000","last_change":"2023-10-16T13:47:23.433833+0000","last_active":"2023-10-16T13:47:24.416043+0000","last_peered":"2023-10-16T13:47:24.416043+0000","last_clean":"2023-10-16T13:47:24.416043+0000","last_became_active":"2023-10-16T13:47:23.433381+0000","last_became_peered":"2023-10-16T13:47:23.433381+0000","last_unstale":"2023-10-16T13:47:24.416043+0000","last_undegraded":"2023-10-16T13:47:24.416043+0000","last_fullsized":"2023-10-16T13:47:24.416043+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:09:34.543970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3c","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.099990+0000","last_change":"2023-10-16T13:47:24.424889+0000","last_active":"2023-10-16T13:47:26.099990+0000","last_peered":"2023-10-16T13:47:26.099990+0000","last_clean":"2023-10-16T13:47:26.099990+0000","last_became_active":"2023-10-16T13:47:24.424573+0000","last_became_peered":"2023-10-16T13:47:24.424573+0000","last_unstale":"2023-10-16T13:47:26.099990+0000","last_undegraded":"2023-10-16T13:47:26.099990+0000","last_fullsized":"2023-10-16T13:47:26.099990+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:23:40.809515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.441247+0000","last_change":"2023-10-16T13:47:23.435920+0000","last_active":"2023-10-16T13:47:24.441247+0000","last_peered":"2023-10-16T13:47:24.441247+0000","last_clean":"2023-10-16T13:47:24.441247+0000","last_became_active":"2023-10-16T13:47:23.435675+0000","last_became_peered":"2023-10-16T13:47:23.435675+0000","last_unstale":"2023-10-16T13:47:24.441247+0000","last_undegraded":"2023-10-16T13:47:24.441247+0000","last_fullsized":"2023-10-16T13:47:24.441247+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:01:54.403340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.3d","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.417707+0000","last_change":"2023-10-16T13:47:24.430727+0000","last_active":"2023-10-16T13:47:25.417707+0000","last_peered":"2023-10-16T13:47:25.417707+0000","last_clean":"2023-10-16T13:47:25.417707+0000","last_became_active":"2023-10-16T13:47:24.430424+0000","last_became_peered":"2023-10-16T13:47:24.430424+0000","last_unstale":"2023-10-16T13:47:25.417707+0000","last_undegraded":"2023-10-16T13:47:25.417707+0000","last_fullsized":"2023-10-16T13:47:25.417707+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:05:49.279256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.418627+0000","last_change":"2023-10-16T13:47:23.430779+0000","last_active":"2023-10-16T13:47:24.418627+0000","last_peered":"2023-10-16T13:47:24.418627+0000","last_clean":"2023-10-16T13:47:24.418627+0000","last_became_active":"2023-10-16T13:47:23.429996+0000","last_became_peered":"2023-10-16T13:47:23.429996+0000","last_unstale":"2023-10-16T13:47:24.418627+0000","last_undegraded":"2023-10-16T13:47:24.418627+0000","last_fullsized":"2023-10-16T13:47:24.418627+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:17:22.079776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.3e","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.416486+0000","last_change":"2023-10-16T13:47:24.436378+0000","last_active":"2023-10-16T13:47:25.416486+0000","last_peered":"2023-10-16T13:47:25.416486+0000","last_clean":"2023-10-16T13:47:25.416486+0000","last_became_active":"2023-10-16T13:47:24.435495+0000","last_became_peered":"2023-10-16T13:47:24.435495+0000","last_unstale":"2023-10-16T13:47:25.416486+0000","last_undegraded":"2023-10-16T13:47:25.416486+0000","last_fullsized":"2023-10-16T13:47:25.416486+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:18:23.073022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.420122+0000","last_change":"2023-10-16T13:47:23.433945+0000","last_active":"2023-10-16T13:47:24.420122+0000","last_peered":"2023-10-16T13:47:24.420122+0000","last_clean":"2023-10-16T13:47:24.420122+0000","last_became_active":"2023-10-16T13:47:23.433576+0000","last_became_peered":"2023-10-16T13:47:23.433576+0000","last_unstale":"2023-10-16T13:47:24.420122+0000","last_undegraded":"2023-10-16T13:47:24.420122+0000","last_fullsized":"2023-10-16T13:47:24.420122+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:09:34.543970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.3f","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100803+0000","last_change":"2023-10-16T13:47:24.441874+0000","last_active":"2023-10-16T13:47:26.100803+0000","last_peered":"2023-10-16T13:47:26.100803+0000","last_clean":"2023-10-16T13:47:26.100803+0000","last_became_active":"2023-10-16T13:47:24.441466+0000","last_became_peered":"2023-10-16T13:47:24.441466+0000","last_unstale":"2023-10-16T13:47:26.100803+0000","last_undegraded":"2023-10-16T13:47:26.100803+0000","last_fullsized":"2023-10-16T13:47:26.100803+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:22:04.707731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.437625+0000","last_change":"2023-10-16T13:47:23.423270+0000","last_active":"2023-10-16T13:47:24.437625+0000","last_peered":"2023-10-16T13:47:24.437625+0000","last_clean":"2023-10-16T13:47:24.437625+0000","last_became_active":"2023-10-16T13:47:23.422889+0000","last_became_peered":"2023-10-16T13:47:23.422889+0000","last_unstale":"2023-10-16T13:47:24.437625+0000","last_undegraded":"2023-10-16T13:47:24.437625+0000","last_fullsized":"2023-10-16T13:47:24.437625+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:30:37.923537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.21","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.417066+0000","last_change":"2023-10-16T13:47:24.431657+0000","last_active":"2023-10-16T13:47:25.417066+0000","last_peered":"2023-10-16T13:47:25.417066+0000","last_clean":"2023-10-16T13:47:25.417066+0000","last_became_active":"2023-10-16T13:47:24.431385+0000","last_became_peered":"2023-10-16T13:47:24.431385+0000","last_unstale":"2023-10-16T13:47:25.417066+0000","last_undegraded":"2023-10-16T13:47:25.417066+0000","last_fullsized":"2023-10-16T13:47:25.417066+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:59:09.754185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1e","version":"244'1","reported_seq":13,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.448637+0000","last_change":"2023-10-16T13:47:23.429601+0000","last_active":"2023-10-16T13:47:24.448637+0000","last_peered":"2023-10-16T13:47:24.448637+0000","last_clean":"2023-10-16T13:47:24.448637+0000","last_became_active":"2023-10-16T13:47:23.429298+0000","last_became_peered":"2023-10-16T13:47:23.429298+0000","last_unstale":"2023-10-16T13:47:24.448637+0000","last_undegraded":"2023-10-16T13:47:24.448637+0000","last_fullsized":"2023-10-16T13:47:24.448637+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:20:00.827725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"231'370","reported_seq":1298,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.425828+0000","last_change":"2023-10-16T12:10:34.746820+0000","last_active":"2023-10-16T13:47:24.425828+0000","last_peered":"2023-10-16T13:47:24.425828+0000","last_clean":"2023-10-16T13:47:24.425828+0000","last_became_active":"2023-10-16T12:10:34.745574+0000","last_became_peered":"2023-10-16T12:10:34.745574+0000","last_unstale":"2023-10-16T13:47:24.425828+0000","last_undegraded":"2023-10-16T13:47:24.425828+0000","last_fullsized":"2023-10-16T13:47:24.425828+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_clean_scrub_stamp":"2023-10-16T12:10:33.577111+0000","objects_scrubbed":0,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:14.461986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":32,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":31,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":370,"ondisk_log_size":370,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575996,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56992,"kb_used_data":3604,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314848,"statfs":{"total":96636764160,"available":96578404352,"internally_reserved":0,"allocated":3690496,"data_stored":2186927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54655435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:46:50 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.60299999999999998,"15min":0.63200000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.33000000000000002},"max":{"1min":0.78600000000000003,"5min":0.87,"15min":0.93799999999999994},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.60699999999999998,"15min":0.626},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.72099999999999997,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.57499999999999996}]},{"osd":1,"last update":"Mon Oct 16 13:46:50 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.61299999999999999,"15min":0.63900000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.79000000000000004,"5min":0.84599999999999997,"15min":0.95899999999999996},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.622,"15min":0.63100000000000001},"min":{"1min":0.38200000000000001,"5min":0.375,"15min":0.375},"max":{"1min":0.76200000000000001,"5min":0.79100000000000004,"15min":0.86399999999999999},"last":0.65000000000000002}]},{"osd":2,"last update":"Mon Oct 16 13:46:50 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.67400000000000004,"15min":0.68600000000000005},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.80300000000000005,"5min":0.83899999999999997,"15min":0.88200000000000001},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.68899999999999995,"15min":0.69799999999999995},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.39400000000000002},"max":{"1min":0.83699999999999997,"5min":0.92800000000000005,"15min":0.93799999999999994},"last":0.73099999999999998}]},{"osd":3,"last update":"Mon Oct 16 13:46:50 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.63800000000000001,"15min":0.65900000000000003},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.85299999999999998,"5min":0.89200000000000002,"15min":0.91300000000000003},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.65000000000000002,"15min":0.66000000000000003},"min":{"1min":0.435,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.88,"5min":1.24,"15min":1.24},"last":0.71499999999999997}]},{"osd":4,"last update":"Mon Oct 16 13:46:50 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.52000000000000002,"15min":0.51900000000000002},"min":{"1min":0.311,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.71099999999999997,"5min":0.745,"15min":0.85299999999999998},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.621,"15min":0.61799999999999999},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.33100000000000002},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.52800000000000002}]},{"osd":5,"last update":"Mon Oct 16 13:46:50 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.54900000000000004,"15min":0.54700000000000004},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.85699999999999998,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.58399999999999996,"15min":0.58699999999999997},"min":{"1min":0.41099999999999998,"5min":0.38400000000000001,"15min":0.35099999999999998},"max":{"1min":0.81100000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.48199999999999998}]},{"osd":6,"last update":"Mon Oct 16 13:46:50 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.60499999999999998,"15min":0.61299999999999999},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.85499999999999998,"5min":0.874,"15min":0.95899999999999996},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.626,"15min":0.61899999999999999},"min":{"1min":0.39400000000000002,"5min":0.378,"15min":0.35199999999999998},"max":{"1min":0.92100000000000004,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.46500000000000002}]}]},{"osd":6,"up_from":13,"seq":55834575994,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60572,"kb_used_data":3024,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311268,"statfs":{"total":96636764160,"available":96574738432,"internally_reserved":0,"allocated":3096576,"data_stored":1596547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58915271},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:46:36 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.65300000000000002,"15min":0.66800000000000004},"min":{"1min":0.41499999999999998,"5min":0.375,"15min":0.28699999999999998},"max":{"1min":0.82599999999999996,"5min":0.89400000000000002,"15min":0.92300000000000004},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.64500000000000002,"15min":0.66400000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.80400000000000005,"5min":0.91400000000000003,"15min":0.93799999999999994},"last":0.68200000000000005}]},{"osd":1,"last update":"Mon Oct 16 13:46:36 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.58599999999999997,"15min":0.624},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.33100000000000002},"max":{"1min":0.70299999999999996,"5min":0.86899999999999999,"15min":0.90900000000000003},"last":0.504},{"interface":"front","average":{"1min":0.63,"5min":0.61399999999999999,"15min":0.62},"min":{"1min":0.41699999999999998,"5min":0.39800000000000002,"15min":0.38300000000000001},"max":{"1min":0.85799999999999998,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.42199999999999999}]},{"osd":2,"last update":"Mon Oct 16 13:46:36 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.622,"15min":0.63300000000000001},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.85799999999999998,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.59999999999999998,"15min":0.624},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.749,"5min":0.89000000000000001,"15min":0.91000000000000003},"last":0.44500000000000001}]},{"osd":3,"last update":"Mon Oct 16 13:46:36 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63900000000000001,"15min":0.66300000000000003},"min":{"1min":0.439,"5min":0.39900000000000002,"15min":0.38200000000000001},"max":{"1min":0.874,"5min":0.92400000000000004,"15min":0.95599999999999996},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.69099999999999995,"15min":0.71599999999999997},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.42299999999999999},"max":{"1min":0.85099999999999998,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.48099999999999998}]},{"osd":4,"last update":"Mon Oct 16 13:46:36 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.51800000000000002,"15min":0.52700000000000002},"min":{"1min":0.39000000000000001,"5min":0.35999999999999999,"15min":0.315},"max":{"1min":0.78000000000000003,"5min":0.85599999999999998,"15min":0.95799999999999996},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61699999999999999,"15min":0.62},"min":{"1min":0.375,"5min":0.35499999999999998,"15min":0.32900000000000001},"max":{"1min":0.83699999999999997,"5min":0.98799999999999999,"15min":1.0309999999999999},"last":0.64600000000000002}]},{"osd":5,"last update":"Mon Oct 16 13:46:36 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.59799999999999998,"15min":0.59399999999999997},"min":{"1min":0.39800000000000002,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.81100000000000005,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.66500000000000004,"15min":0.66000000000000003},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.308},"max":{"1min":0.85899999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.53500000000000003}]},{"osd":7,"last update":"Mon Oct 16 13:46:36 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.60899999999999999,"15min":0.60499999999999998},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.80000000000000004,"5min":1.081,"15min":1.081},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65200000000000002},"min":{"1min":0.42799999999999999,"5min":0.41899999999999998,"15min":0.375},"max":{"1min":0.91500000000000004,"5min":1.099,"15min":1.099},"last":0.629}]}]},{"osd":1,"up_from":13,"seq":55834575994,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52252,"kb_used_data":3016,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319588,"statfs":{"total":96636764160,"available":96583258112,"internally_reserved":0,"allocated":3088384,"data_stored":1596435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":50395593},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:46:55 2023","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.39500000000000002,"15min":0.40699999999999997},"min":{"1min":0.27600000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.51300000000000001,"5min":0.65400000000000003,"15min":0.65800000000000003},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.47599999999999998,"15min":0.49199999999999999},"min":{"1min":0.32000000000000001,"5min":0.253,"15min":0.253},"max":{"1min":0.56000000000000005,"5min":0.70999999999999996,"15min":0.77200000000000002},"last":0.67000000000000004}]},{"osd":2,"last update":"Mon Oct 16 13:46:55 2023","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.436,"15min":0.45100000000000001},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.58599999999999997,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.48099999999999998,"15min":0.48999999999999999},"min":{"1min":0.34399999999999997,"5min":0.315,"15min":0.30199999999999999},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.71399999999999997},"last":0.65200000000000002}]},{"osd":3,"last update":"Mon Oct 16 13:46:55 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.42899999999999999,"15min":0.42799999999999999},"min":{"1min":0.307,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.50800000000000001,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.46100000000000002,"15min":0.47499999999999998},"min":{"1min":0.29799999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.52800000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.69399999999999995}]},{"osd":4,"last update":"Mon Oct 16 13:46:55 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.64400000000000002,"15min":0.66200000000000003},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.42399999999999999},"max":{"1min":0.78600000000000003,"5min":0.95899999999999996,"15min":1.0069999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.64200000000000002,"15min":0.66600000000000004},"min":{"1min":0.42899999999999999,"5min":0.37,"15min":0.37},"max":{"1min":0.69999999999999996,"5min":0.91300000000000003,"15min":0.997},"last":0.747}]},{"osd":5,"last update":"Mon Oct 16 13:46:55 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.68200000000000005,"15min":0.70699999999999996},"min":{"1min":0.45800000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.83999999999999997,"5min":0.97499999999999998,"15min":1.0760000000000001},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.70499999999999996,"15min":0.72599999999999998},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.81399999999999995,"5min":1.0469999999999999,"15min":1.0660000000000001},"last":0.73199999999999998}]},{"osd":6,"last update":"Mon Oct 16 13:46:55 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.67500000000000004,"15min":0.68000000000000005},"min":{"1min":0.46200000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.75900000000000001,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.68700000000000006,"15min":0.70499999999999996},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.80400000000000005,"5min":1.03,"15min":1.083},"last":0.72299999999999998}]},{"osd":7,"last update":"Mon Oct 16 13:46:55 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.70899999999999996,"15min":0.73399999999999999},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.77300000000000002,"5min":1.0629999999999999,"15min":1.0660000000000001},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.72899999999999998,"15min":0.746},"min":{"1min":0.496,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.82099999999999995,"5min":1.042,"15min":1.0580000000000001},"last":0.87}]}]},{"osd":0,"up_from":13,"seq":55834575995,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48676,"kb_used_data":3608,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323164,"statfs":{"total":96636764160,"available":96586919936,"internally_reserved":0,"allocated":3694592,"data_stored":2188415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 13:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.40400000000000003,"15min":0.42099999999999999},"min":{"1min":0.29099999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.60699999999999998,"5min":0.65800000000000003,"15min":0.71399999999999997},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.42199999999999999,"15min":0.439},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.82199999999999995},"last":0.41999999999999998}]},{"osd":2,"last update":"Mon Oct 16 13:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.443,"15min":0.45400000000000001},"min":{"1min":0.28599999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.61399999999999999,"5min":0.65900000000000003,"15min":0.75700000000000001},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.46400000000000002,"15min":0.48499999999999999},"min":{"1min":0.29799999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.83299999999999996},"last":0.60499999999999998}]},{"osd":3,"last update":"Mon Oct 16 13:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.42699999999999999,"15min":0.45300000000000001},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.56499999999999995,"5min":0.63400000000000001,"15min":0.78600000000000003},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.50900000000000001,"15min":0.52000000000000002},"min":{"1min":0.317,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.66400000000000003,"5min":0.82999999999999996,"15min":0.86399999999999999},"last":0.57199999999999995}]},{"osd":4,"last update":"Mon Oct 16 13:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.66400000000000003,"15min":0.68400000000000005},"min":{"1min":0.42399999999999999,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.66500000000000004,"15min":0.68899999999999995},"min":{"1min":0.499,"5min":0.375,"15min":0.375},"max":{"1min":1.21,"5min":1.21,"15min":1.21},"last":0.78200000000000003}]},{"osd":5,"last update":"Mon Oct 16 13:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.66400000000000003,"15min":0.68999999999999995},"min":{"1min":0.41999999999999998,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.242,"5min":1.242,"15min":1.242},"last":0.751},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.70599999999999996,"15min":0.72299999999999998},"min":{"1min":0.497,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.998,"5min":1.081,"15min":1.081},"last":0.63200000000000001}]},{"osd":6,"last update":"Mon Oct 16 13:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.71199999999999997,"15min":0.72899999999999998},"min":{"1min":0.51900000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.96599999999999997,"5min":1.034,"15min":1.103},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.70399999999999996,"15min":0.72299999999999998},"min":{"1min":0.51000000000000001,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":0.79600000000000004}]},{"osd":7,"last update":"Mon Oct 16 13:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.71199999999999997,"15min":0.74199999999999999},"min":{"1min":0.48399999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.1919999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.72499999999999998,"15min":0.73499999999999999},"min":{"1min":0.51300000000000001,"5min":0.433,"15min":0.433},"max":{"1min":1.0469999999999999,"5min":1.087,"15min":1.087},"last":0.64700000000000002}]}]},{"osd":2,"up_from":13,"seq":55834575995,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52252,"kb_used_data":3024,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319588,"statfs":{"total":96636764160,"available":96583258112,"internally_reserved":0,"allocated":3096576,"data_stored":1597087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.41499999999999998,"15min":0.40899999999999997},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.623,"5min":0.71699999999999997,"15min":0.77500000000000002},"last":0.438},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.47699999999999998,"15min":0.47999999999999998},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.52800000000000002}]},{"osd":1,"last update":"Mon Oct 16 13:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.502,"15min":0.5},"min":{"1min":0.317,"5min":0.311,"15min":0.311},"max":{"1min":0.66800000000000004,"5min":0.73199999999999998,"15min":0.81399999999999995},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.48999999999999999,"15min":0.47699999999999998},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.27700000000000002},"max":{"1min":0.69999999999999996,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.58599999999999997}]},{"osd":3,"last update":"Mon Oct 16 13:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.443,"15min":0.44},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.59699999999999998,"5min":0.70099999999999996,"15min":0.72599999999999998},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.54300000000000004,"15min":0.54400000000000004},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.82699999999999996,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.74099999999999999}]},{"osd":4,"last update":"Mon Oct 16 13:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.72699999999999998,"15min":0.71799999999999997},"min":{"1min":0.51500000000000001,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.89000000000000001,"5min":1.3600000000000001,"15min":1.3600000000000001},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.72599999999999998,"15min":0.71899999999999997},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.82899999999999996,"5min":1.403,"15min":1.403},"last":0.91400000000000003}]},{"osd":5,"last update":"Mon Oct 16 13:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.68600000000000005,"15min":0.68899999999999995},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.81100000000000005,"5min":1.486,"15min":1.486},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.71199999999999997,"15min":0.69999999999999996},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.90200000000000002,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":0.879}]},{"osd":6,"last update":"Mon Oct 16 13:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.67600000000000005,"15min":0.66600000000000004},"min":{"1min":0.44500000000000001,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.80200000000000005,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.72599999999999998,"15min":0.72199999999999998},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.90600000000000003,"5min":1.4379999999999999,"15min":1.4379999999999999},"last":0.94299999999999995}]},{"osd":7,"last update":"Mon Oct 16 13:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.73899999999999999,"15min":0.72099999999999997},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.94099999999999995,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":0.97099999999999997},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.72999999999999998,"15min":0.73099999999999998},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.91500000000000004,"5min":1.377,"15min":1.377},"last":1.0429999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575994,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56428,"kb_used_data":3040,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315412,"statfs":{"total":96636764160,"available":96578981888,"internally_reserved":0,"allocated":3112960,"data_stored":1598685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.40300000000000002,"15min":0.40999999999999998},"min":{"1min":0.23499999999999999,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.57199999999999995,"5min":0.64200000000000002,"15min":0.72899999999999998},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.41499999999999998,"15min":0.42799999999999999},"min":{"1min":0.27300000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.60099999999999998,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.312}]},{"osd":1,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.42699999999999999,"15min":0.434},"min":{"1min":0.26700000000000002,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.55900000000000005,"5min":0.68200000000000005,"15min":0.76300000000000001},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.435,"5min":0.45900000000000002,"15min":0.47999999999999998},"min":{"1min":0.27700000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.65500000000000003,"5min":0.70699999999999996,"15min":0.77800000000000002},"last":0.441}]},{"osd":2,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.47999999999999998,"15min":0.49299999999999999},"min":{"1min":0.33800000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.621,"5min":1.167,"15min":1.167},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.501,"15min":0.52300000000000002},"min":{"1min":0.312,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.625,"5min":0.73299999999999998,"15min":0.78700000000000003},"last":0.49199999999999999}]},{"osd":4,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.65100000000000002,"15min":0.66500000000000004},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.91200000000000003,"5min":0.998,"15min":1.137},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.68899999999999995,"15min":0.69399999999999995},"min":{"1min":0.441,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.087,"5min":1.2110000000000001,"15min":1.2909999999999999},"last":0.72799999999999998}]},{"osd":5,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.67700000000000005,"15min":0.70499999999999996},"min":{"1min":0.45500000000000002,"5min":0.42899999999999999,"15min":0.374},"max":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.2729999999999999},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.69299999999999995,"15min":0.69999999999999996},"min":{"1min":0.45600000000000002,"5min":0.432,"15min":0.432},"max":{"1min":0.98799999999999999,"5min":1.0549999999999999,"15min":1.173},"last":0.68600000000000005}]},{"osd":6,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.72899999999999998,"15min":0.71899999999999997},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.41799999999999998},"max":{"1min":0.99099999999999999,"5min":5.0999999999999996,"15min":5.0999999999999996},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.70999999999999996,"15min":0.73399999999999999},"min":{"1min":0.49199999999999999,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.0029999999999999,"5min":1.0309999999999999,"15min":1.2290000000000001},"last":0.80000000000000004}]},{"osd":7,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.71899999999999997,"15min":0.73399999999999999},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.048,"5min":1.048,"15min":1.2569999999999999},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.73299999999999998,"15min":0.746},"min":{"1min":0.501,"5min":0.47799999999999998,"15min":0.43099999999999999},"max":{"1min":1.0369999999999999,"5min":1.262,"15min":1.262},"last":0.66800000000000004}]}]},{"osd":4,"up_from":13,"seq":55834575993,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52248,"kb_used_data":3020,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319592,"statfs":{"total":96636764160,"available":96583262208,"internally_reserved":0,"allocated":3092480,"data_stored":1596505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.60699999999999998,"15min":0.625},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.33800000000000002},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.623,"15min":0.63100000000000001},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.84099999999999997,"5min":0.878,"15min":0.89500000000000002},"last":0.55800000000000005}]},{"osd":1,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.63,"15min":0.65600000000000003},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.628,"15min":0.63700000000000001},"min":{"1min":0.41099999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.65200000000000002}]},{"osd":2,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.65200000000000002,"15min":0.65300000000000002},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.83999999999999997,"5min":0.89900000000000002,"15min":0.90600000000000003},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.65600000000000003,"15min":0.65800000000000003},"min":{"1min":0.39600000000000002,"5min":0.39400000000000002,"15min":0.38400000000000001},"max":{"1min":0.88200000000000001,"5min":0.89700000000000002,"15min":0.92600000000000005},"last":0.622}]},{"osd":3,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.63200000000000001,"15min":0.64500000000000002},"min":{"1min":0.40400000000000003,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.78100000000000003,"5min":0.86599999999999999,"15min":0.99199999999999999},"last":0.502},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.67400000000000004,"15min":0.68200000000000005},"min":{"1min":0.44800000000000001,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.79100000000000004,"5min":0.872,"15min":0.93899999999999995},"last":0.48199999999999998}]},{"osd":5,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55100000000000005,"15min":0.53000000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.57999999999999996,"15min":0.56899999999999995},"min":{"1min":0.36099999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.82199999999999995,"5min":0.82299999999999995,"15min":0.94899999999999995},"last":0.60499999999999998}]},{"osd":6,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.625,"15min":0.59799999999999998},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.33200000000000002},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92700000000000005},"last":0.378},{"interface":"front","average":{"1min":0.747,"5min":0.71399999999999997,"15min":0.69399999999999995},"min":{"1min":0.52600000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.129,"5min":1.129,"15min":1.129},"last":0.58999999999999997}]},{"osd":7,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.66000000000000003,"15min":0.63500000000000001},"min":{"1min":0.44400000000000001,"5min":0.41999999999999998,"15min":0.35799999999999998},"max":{"1min":0.97099999999999997,"5min":0.97499999999999998,"15min":0.98599999999999999},"last":0.432},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.70799999999999996,"15min":0.68400000000000005},"min":{"1min":0.44,"5min":0.433,"15min":0.38200000000000001},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":1.026},"last":0.66700000000000004}]}]},{"osd":5,"up_from":13,"seq":55834575994,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52244,"kb_used_data":3016,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319596,"statfs":{"total":96636764160,"available":96583266304,"internally_reserved":0,"allocated":3088384,"data_stored":1596435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:46:24 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.61399999999999999,"15min":0.61499999999999999},"min":{"1min":0.314,"5min":0.314,"15min":0.26800000000000002},"max":{"1min":0.878,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.746},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63,"15min":0.64200000000000002},"min":{"1min":0.38700000000000001,"5min":0.35899999999999999,"15min":0.29699999999999999},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.63500000000000001}]},{"osd":1,"last update":"Mon Oct 16 13:46:24 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.66000000000000003,"15min":0.66200000000000003},"min":{"1min":0.44600000000000001,"5min":0.34300000000000003,"15min":0.29499999999999998},"max":{"1min":1,"5min":1,"15min":1},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.63800000000000001,"15min":0.64000000000000001},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.72999999999999998}]},{"osd":2,"last update":"Mon Oct 16 13:46:24 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.67000000000000004,"15min":0.65700000000000003},"min":{"1min":0.41699999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.91000000000000003,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.621},{"interface":"front","average":{"1min":0.629,"5min":0.65700000000000003,"15min":0.64900000000000002},"min":{"1min":0.42599999999999999,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.83899999999999997,"5min":0.871,"15min":0.93500000000000005},"last":0.80200000000000005}]},{"osd":3,"last update":"Mon Oct 16 13:46:24 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.67400000000000004,"15min":0.66400000000000003},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.69899999999999995,"15min":0.69499999999999995},"min":{"1min":0.41399999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.4199999999999999,"5min":1.4199999999999999,"15min":1.4199999999999999},"last":0.77300000000000002}]},{"osd":4,"last update":"Mon Oct 16 13:46:24 2023","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.54400000000000004,"15min":0.53300000000000003},"min":{"1min":0.34100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.82499999999999996,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.59299999999999997,"15min":0.58899999999999997},"min":{"1min":0.34300000000000003,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.85399999999999998,"5min":0.89000000000000001,"15min":0.93400000000000005},"last":0.47999999999999998}]},{"osd":6,"last update":"Mon Oct 16 13:46:24 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.58899999999999997,"15min":0.57999999999999996},"min":{"1min":0.38700000000000001,"5min":0.315,"15min":0.315},"max":{"1min":0.877,"5min":0.88900000000000001,"15min":0.93200000000000005},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.61899999999999999},"min":{"1min":0.372,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89600000000000002},"last":0.504}]},{"osd":7,"last update":"Mon Oct 16 13:46:24 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.64800000000000002,"15min":0.63300000000000001},"min":{"1min":0.378,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.69699999999999995,"15min":0.68400000000000005},"min":{"1min":0.46200000000000002,"5min":0.375,"15min":0.375},"max":{"1min":0.94699999999999995,"5min":0.98299999999999998,"15min":1.0229999999999999},"last":0.53400000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T13:47:30.406 INFO:tasks.ceph.mds.a.smithi016.stderr:2023-10-16T13:47:30.403+0000 7f99d9b55700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T13:47:33.266 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T13:47:33.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.417+0000 7f629534c700 1 Processor -- start 2023-10-16T13:47:33.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.417+0000 7f629534c700 1 -- start start 2023-10-16T13:47:33.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.418+0000 7f629534c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f629015d8d0 0x7f62900d1c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:33.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.418+0000 7f629534c700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f629006ba00 con 0x7f62900d0750 2023-10-16T13:47:33.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.418+0000 7f629534c700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62900d2150 con 0x7f62900d41c0 2023-10-16T13:47:33.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.418+0000 7f629534c700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62900d26d0 con 0x7f629015d8d0 2023-10-16T13:47:33.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.418+0000 7f628effd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f629015d8d0 0x7f62900d1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:33.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.418+0000 7f628effd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f629015d8d0 0x7f62900d1c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55550/0 (socket says 172.21.15.16:55550) 2023-10-16T13:47:33.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.418+0000 7f628effd700 1 -- 172.21.15.16:0/725015030 learned_addr learned my addr 172.21.15.16:0/725015030 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:33.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.418+0000 7f628effd700 1 -- 172.21.15.16:0/725015030 >> v1:172.21.15.49:6789/0 conn(0x7f62900d41c0 legacy=0x7f62900d0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:33.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.418+0000 7f628effd700 1 -- 172.21.15.16:0/725015030 >> v1:172.21.15.16:6789/0 conn(0x7f62900d0750 legacy=0x7f629006c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:33.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.418+0000 7f628effd700 1 -- 172.21.15.16:0/725015030 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62900d23f0 con 0x7f629015d8d0 2023-10-16T13:47:33.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.418+0000 7f628effd700 1 --2- 172.21.15.16:0/725015030 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f629015d8d0 0x7f62900d1c10 secure :-1 s=READY pgs=2764 cs=0 l=1 rev1=1 crypto rx=0x7f62800098f0 tx=0x7f6280009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=426220c16721857a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:33.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.418+0000 7f628dffb700 1 -- 172.21.15.16:0/725015030 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f628000b8a0 con 0x7f629015d8d0 2023-10-16T13:47:33.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.419+0000 7f628dffb700 1 -- 172.21.15.16:0/725015030 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f628000ba00 con 0x7f629015d8d0 2023-10-16T13:47:33.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.419+0000 7f628dffb700 1 -- 172.21.15.16:0/725015030 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f628000bd90 con 0x7f629015d8d0 2023-10-16T13:47:33.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.419+0000 7f629534c700 1 -- 172.21.15.16:0/725015030 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f629015d8d0 msgr2=0x7f62900d1c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:33.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.419+0000 7f629534c700 1 --2- 172.21.15.16:0/725015030 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f629015d8d0 0x7f62900d1c10 secure :-1 s=READY pgs=2764 cs=0 l=1 rev1=1 crypto rx=0x7f62800098f0 tx=0x7f6280009c00 comp rx=0 tx=0).stop 2023-10-16T13:47:33.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.419+0000 7f629534c700 1 -- 172.21.15.16:0/725015030 shutdown_connections 2023-10-16T13:47:33.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.419+0000 7f629534c700 1 --2- 172.21.15.16:0/725015030 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f629015d8d0 0x7f62900d1c10 unknown :-1 s=CLOSED pgs=2764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:33.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.419+0000 7f629534c700 1 -- 172.21.15.16:0/725015030 >> 172.21.15.16:0/725015030 conn(0x7f62900d3630 msgr2=0x7f62900d3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:33.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.419+0000 7f629534c700 1 -- 172.21.15.16:0/725015030 shutdown_connections 2023-10-16T13:47:33.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.419+0000 7f629534c700 1 -- 172.21.15.16:0/725015030 wait complete. 2023-10-16T13:47:33.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.420+0000 7f629534c700 1 Processor -- start 2023-10-16T13:47:33.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.421+0000 7f629534c700 1 -- start start 2023-10-16T13:47:33.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.421+0000 7f629534c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f62900d0750 0x7f62901596e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:33.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.421+0000 7f629534c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62900d41c0 0x7f62900c9930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:33.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.421+0000 7f629534c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f629015d8d0 0x7f62900b1910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:33.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.421+0000 7f629534c700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62900c85c0 con 0x7f62900d41c0 2023-10-16T13:47:33.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.421+0000 7f629534c700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62901f2b80 con 0x7f62900d0750 2023-10-16T13:47:33.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.421+0000 7f629534c700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62900c7520 con 0x7f629015d8d0 2023-10-16T13:47:33.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.421+0000 7f628e7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62900d41c0 0x7f62900c9930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:33.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.421+0000 7f628e7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62900d41c0 0x7f62900c9930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55558/0 (socket says 172.21.15.16:55558) 2023-10-16T13:47:33.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.421+0000 7f628e7fc700 1 -- 172.21.15.16:0/1682206533 learned_addr learned my addr 172.21.15.16:0/1682206533 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:33.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.422+0000 7f628effd700 1 --2- 172.21.15.16:0/1682206533 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f62900d0750 0x7f62901596e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:33.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.422+0000 7f628f7fe700 1 --2- 172.21.15.16:0/1682206533 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f629015d8d0 0x7f62900b1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:33.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.422+0000 7f628e7fc700 1 -- 172.21.15.16:0/1682206533 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f629015d8d0 msgr2=0x7f62900b1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:33.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.422+0000 7f628e7fc700 1 --2- 172.21.15.16:0/1682206533 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f629015d8d0 0x7f62900b1910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:33.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.422+0000 7f628e7fc700 1 -- 172.21.15.16:0/1682206533 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f62900d0750 msgr2=0x7f62901596e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:33.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.422+0000 7f628e7fc700 1 --2- 172.21.15.16:0/1682206533 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f62900d0750 0x7f62901596e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:33.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.422+0000 7f628e7fc700 1 -- 172.21.15.16:0/1682206533 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6280009550 con 0x7f62900d41c0 2023-10-16T13:47:33.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.422+0000 7f628f7fe700 1 --2- 172.21.15.16:0/1682206533 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f629015d8d0 0x7f62900b1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:47:33.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.422+0000 7f628e7fc700 1 --2- 172.21.15.16:0/1682206533 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62900d41c0 0x7f62900c9930 secure :-1 s=READY pgs=2765 cs=0 l=1 rev1=1 crypto rx=0x7f628400c4e0 tx=0x7f628400c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:33.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.422+0000 7f62777fe700 1 -- 172.21.15.16:0/1682206533 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62840052d0 con 0x7f62900d41c0 2023-10-16T13:47:33.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.422+0000 7f62777fe700 1 -- 172.21.15.16:0/1682206533 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6284005430 con 0x7f62900d41c0 2023-10-16T13:47:33.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.422+0000 7f629534c700 1 -- 172.21.15.16:0/1682206533 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62900c7800 con 0x7f62900d41c0 2023-10-16T13:47:33.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.422+0000 7f62777fe700 1 -- 172.21.15.16:0/1682206533 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62840038c0 con 0x7f62900d41c0 2023-10-16T13:47:33.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.422+0000 7f629534c700 1 -- 172.21.15.16:0/1682206533 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62900c7dc0 con 0x7f62900d41c0 2023-10-16T13:47:33.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.423+0000 7f629534c700 1 -- 172.21.15.16:0/1682206533 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f629004ebb0 con 0x7f62900d41c0 2023-10-16T13:47:33.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.424+0000 7f62777fe700 1 -- 172.21.15.16:0/1682206533 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 148) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f6284003a20 con 0x7f62900d41c0 2023-10-16T13:47:33.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.424+0000 7f62777fe700 1 --2- 172.21.15.16:0/1682206533 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f627c06b140 0x7f627c06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:33.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.424+0000 7f62777fe700 1 -- 172.21.15.16:0/1682206533 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(245..245 src has 1..245) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f6284088fd0 con 0x7f62900d41c0 2023-10-16T13:47:33.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.424+0000 7f628effd700 1 --2- 172.21.15.16:0/1682206533 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f627c06b140 0x7f627c06d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:33.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.424+0000 7f628effd700 1 --2- 172.21.15.16:0/1682206533 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f627c06b140 0x7f627c06d600 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6280014040 tx=0x7f6280008030 comp rx=0 tx=0).ready entity=mgr.11886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:33.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.429+0000 7f62777fe700 1 -- 172.21.15.16:0/1682206533 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6284053c30 con 0x7f62900d41c0 2023-10-16T13:47:33.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.628+0000 7f629534c700 1 -- 172.21.15.16:0/1682206533 --> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f629006bf80 con 0x7f627c06b140 2023-10-16T13:47:33.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.636+0000 7f62777fe700 1 -- 172.21.15.16:0/1682206533 <== mgr.11886 v2:172.21.15.16:6832/1736104576 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289879 (secure 0 0 0) 0x7f628000bbf0 con 0x7f627c06b140 2023-10-16T13:47:33.642 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:47:33.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.639+0000 7f629534c700 1 -- 172.21.15.16:0/1682206533 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f627c06b140 msgr2=0x7f627c06d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:33.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.639+0000 7f629534c700 1 --2- 172.21.15.16:0/1682206533 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f627c06b140 0x7f627c06d600 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6280014040 tx=0x7f6280008030 comp rx=0 tx=0).stop 2023-10-16T13:47:33.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.639+0000 7f629534c700 1 -- 172.21.15.16:0/1682206533 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62900d41c0 msgr2=0x7f62900c9930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:33.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.639+0000 7f629534c700 1 --2- 172.21.15.16:0/1682206533 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62900d41c0 0x7f62900c9930 secure :-1 s=READY pgs=2765 cs=0 l=1 rev1=1 crypto rx=0x7f628400c4e0 tx=0x7f628400c8a0 comp rx=0 tx=0).stop 2023-10-16T13:47:33.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.639+0000 7f629534c700 1 -- 172.21.15.16:0/1682206533 shutdown_connections 2023-10-16T13:47:33.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.639+0000 7f629534c700 1 --2- 172.21.15.16:0/1682206533 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f627c06b140 0x7f627c06d600 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:33.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.639+0000 7f629534c700 1 --2- 172.21.15.16:0/1682206533 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f629015d8d0 0x7f62900b1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:33.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.639+0000 7f629534c700 1 --2- 172.21.15.16:0/1682206533 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f62900d0750 0x7f62901596e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:33.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.639+0000 7f629534c700 1 --2- 172.21.15.16:0/1682206533 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62900d41c0 0x7f62900c9930 unknown :-1 s=CLOSED pgs=2765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:33.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.639+0000 7f629534c700 1 -- 172.21.15.16:0/1682206533 >> 172.21.15.16:0/1682206533 conn(0x7f62900d3630 msgr2=0x7f629015e1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:33.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.639+0000 7f629534c700 1 -- 172.21.15.16:0/1682206533 shutdown_connections 2023-10-16T13:47:33.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.639+0000 7f629534c700 1 -- 172.21.15.16:0/1682206533 wait complete. 2023-10-16T13:47:33.646 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-10-16T13:47:33.654 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":32,"stamp":"2023-10-16T13:47:31.822503+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":663,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":408,"ondisk_log_size":408,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":152,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":431664,"kb_used_data":25352,"kb_used_omap":12,"kb_used_meta":406195,"kb_avail":754543056,"statfs":{"total":773094113280,"available":772652089344,"internally_reserved":0,"allocated":25960448,"data_stored":13957036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12740,"internal_metadata":415944252},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001900"},"pg_stats":[{"pgid":"32.1f","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.418005+0000","last_change":"2023-10-16T13:47:24.429332+0000","last_active":"2023-10-16T13:47:25.418005+0000","last_peered":"2023-10-16T13:47:25.418005+0000","last_clean":"2023-10-16T13:47:25.418005+0000","last_became_active":"2023-10-16T13:47:24.428151+0000","last_became_peered":"2023-10-16T13:47:24.428151+0000","last_unstale":"2023-10-16T13:47:25.418005+0000","last_undegraded":"2023-10-16T13:47:25.418005+0000","last_fullsized":"2023-10-16T13:47:25.418005+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:51:49.567279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.1e","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.416620+0000","last_change":"2023-10-16T13:47:24.439610+0000","last_active":"2023-10-16T13:47:25.416620+0000","last_peered":"2023-10-16T13:47:25.416620+0000","last_clean":"2023-10-16T13:47:25.416620+0000","last_became_active":"2023-10-16T13:47:24.439212+0000","last_became_peered":"2023-10-16T13:47:24.439212+0000","last_unstale":"2023-10-16T13:47:25.416620+0000","last_undegraded":"2023-10-16T13:47:25.416620+0000","last_fullsized":"2023-10-16T13:47:25.416620+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:47:24.106272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1d","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100591+0000","last_change":"2023-10-16T13:47:24.432216+0000","last_active":"2023-10-16T13:47:26.100591+0000","last_peered":"2023-10-16T13:47:26.100591+0000","last_clean":"2023-10-16T13:47:26.100591+0000","last_became_active":"2023-10-16T13:47:24.431937+0000","last_became_peered":"2023-10-16T13:47:24.431937+0000","last_unstale":"2023-10-16T13:47:26.100591+0000","last_undegraded":"2023-10-16T13:47:26.100591+0000","last_fullsized":"2023-10-16T13:47:26.100591+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:23:10.580869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1c","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100430+0000","last_change":"2023-10-16T13:47:24.440252+0000","last_active":"2023-10-16T13:47:26.100430+0000","last_peered":"2023-10-16T13:47:26.100430+0000","last_clean":"2023-10-16T13:47:26.100430+0000","last_became_active":"2023-10-16T13:47:24.439982+0000","last_became_peered":"2023-10-16T13:47:24.439982+0000","last_unstale":"2023-10-16T13:47:26.100430+0000","last_undegraded":"2023-10-16T13:47:26.100430+0000","last_fullsized":"2023-10-16T13:47:26.100430+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:54:02.930019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1b","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.101595+0000","last_change":"2023-10-16T13:47:24.440049+0000","last_active":"2023-10-16T13:47:26.101595+0000","last_peered":"2023-10-16T13:47:26.101595+0000","last_clean":"2023-10-16T13:47:26.101595+0000","last_became_active":"2023-10-16T13:47:24.439750+0000","last_became_peered":"2023-10-16T13:47:24.439750+0000","last_unstale":"2023-10-16T13:47:26.101595+0000","last_undegraded":"2023-10-16T13:47:26.101595+0000","last_fullsized":"2023-10-16T13:47:26.101595+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:33:40.094175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1a","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.417711+0000","last_change":"2023-10-16T13:47:24.433661+0000","last_active":"2023-10-16T13:47:25.417711+0000","last_peered":"2023-10-16T13:47:25.417711+0000","last_clean":"2023-10-16T13:47:25.417711+0000","last_became_active":"2023-10-16T13:47:24.433417+0000","last_became_peered":"2023-10-16T13:47:24.433417+0000","last_unstale":"2023-10-16T13:47:25.417711+0000","last_undegraded":"2023-10-16T13:47:25.417711+0000","last_fullsized":"2023-10-16T13:47:25.417711+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:16:02.391378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.101505+0000","last_change":"2023-10-16T13:47:24.437673+0000","last_active":"2023-10-16T13:47:26.101505+0000","last_peered":"2023-10-16T13:47:26.101505+0000","last_clean":"2023-10-16T13:47:26.101505+0000","last_became_active":"2023-10-16T13:47:24.437479+0000","last_became_peered":"2023-10-16T13:47:24.437479+0000","last_unstale":"2023-10-16T13:47:26.101505+0000","last_undegraded":"2023-10-16T13:47:26.101505+0000","last_fullsized":"2023-10-16T13:47:26.101505+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:23:58.749586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.101429+0000","last_change":"2023-10-16T13:47:24.426442+0000","last_active":"2023-10-16T13:47:26.101429+0000","last_peered":"2023-10-16T13:47:26.101429+0000","last_clean":"2023-10-16T13:47:26.101429+0000","last_became_active":"2023-10-16T13:47:24.425929+0000","last_became_peered":"2023-10-16T13:47:24.425929+0000","last_unstale":"2023-10-16T13:47:26.101429+0000","last_undegraded":"2023-10-16T13:47:26.101429+0000","last_fullsized":"2023-10-16T13:47:26.101429+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:05:13.565304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.418008+0000","last_change":"2023-10-16T13:47:24.426459+0000","last_active":"2023-10-16T13:47:25.418008+0000","last_peered":"2023-10-16T13:47:25.418008+0000","last_clean":"2023-10-16T13:47:25.418008+0000","last_became_active":"2023-10-16T13:47:24.426169+0000","last_became_peered":"2023-10-16T13:47:24.426169+0000","last_unstale":"2023-10-16T13:47:25.418008+0000","last_undegraded":"2023-10-16T13:47:25.418008+0000","last_fullsized":"2023-10-16T13:47:25.418008+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:57:42.170954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100664+0000","last_change":"2023-10-16T13:47:24.427333+0000","last_active":"2023-10-16T13:47:26.100664+0000","last_peered":"2023-10-16T13:47:26.100664+0000","last_clean":"2023-10-16T13:47:26.100664+0000","last_became_active":"2023-10-16T13:47:24.427034+0000","last_became_peered":"2023-10-16T13:47:24.427034+0000","last_unstale":"2023-10-16T13:47:26.100664+0000","last_undegraded":"2023-10-16T13:47:26.100664+0000","last_fullsized":"2023-10-16T13:47:26.100664+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:27:20.181386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.15","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100502+0000","last_change":"2023-10-16T13:47:24.425903+0000","last_active":"2023-10-16T13:47:26.100502+0000","last_peered":"2023-10-16T13:47:26.100502+0000","last_clean":"2023-10-16T13:47:26.100502+0000","last_became_active":"2023-10-16T13:47:24.425593+0000","last_became_peered":"2023-10-16T13:47:24.425593+0000","last_unstale":"2023-10-16T13:47:26.100502+0000","last_undegraded":"2023-10-16T13:47:26.100502+0000","last_fullsized":"2023-10-16T13:47:26.100502+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:46:35.236231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.14","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.417564+0000","last_change":"2023-10-16T13:47:24.438927+0000","last_active":"2023-10-16T13:47:25.417564+0000","last_peered":"2023-10-16T13:47:25.417564+0000","last_clean":"2023-10-16T13:47:25.417564+0000","last_became_active":"2023-10-16T13:47:24.438389+0000","last_became_peered":"2023-10-16T13:47:24.438389+0000","last_unstale":"2023-10-16T13:47:25.417564+0000","last_undegraded":"2023-10-16T13:47:25.417564+0000","last_fullsized":"2023-10-16T13:47:25.417564+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:44:00.360485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.13","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100623+0000","last_change":"2023-10-16T13:47:24.445148+0000","last_active":"2023-10-16T13:47:26.100623+0000","last_peered":"2023-10-16T13:47:26.100623+0000","last_clean":"2023-10-16T13:47:26.100623+0000","last_became_active":"2023-10-16T13:47:24.444896+0000","last_became_peered":"2023-10-16T13:47:24.444896+0000","last_unstale":"2023-10-16T13:47:26.100623+0000","last_undegraded":"2023-10-16T13:47:26.100623+0000","last_fullsized":"2023-10-16T13:47:26.100623+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:35:13.886134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.099722+0000","last_change":"2023-10-16T13:47:24.437693+0000","last_active":"2023-10-16T13:47:26.099722+0000","last_peered":"2023-10-16T13:47:26.099722+0000","last_clean":"2023-10-16T13:47:26.099722+0000","last_became_active":"2023-10-16T13:47:24.437434+0000","last_became_peered":"2023-10-16T13:47:24.437434+0000","last_unstale":"2023-10-16T13:47:26.099722+0000","last_undegraded":"2023-10-16T13:47:26.099722+0000","last_fullsized":"2023-10-16T13:47:26.099722+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:47:56.221847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.101989+0000","last_change":"2023-10-16T13:47:24.435339+0000","last_active":"2023-10-16T13:47:26.101989+0000","last_peered":"2023-10-16T13:47:26.101989+0000","last_clean":"2023-10-16T13:47:26.101989+0000","last_became_active":"2023-10-16T13:47:24.434237+0000","last_became_peered":"2023-10-16T13:47:24.434237+0000","last_unstale":"2023-10-16T13:47:26.101989+0000","last_undegraded":"2023-10-16T13:47:26.101989+0000","last_fullsized":"2023-10-16T13:47:26.101989+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:21:21.083320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.10","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.099717+0000","last_change":"2023-10-16T13:47:24.441060+0000","last_active":"2023-10-16T13:47:26.099717+0000","last_peered":"2023-10-16T13:47:26.099717+0000","last_clean":"2023-10-16T13:47:26.099717+0000","last_became_active":"2023-10-16T13:47:24.440745+0000","last_became_peered":"2023-10-16T13:47:24.440745+0000","last_unstale":"2023-10-16T13:47:26.099717+0000","last_undegraded":"2023-10-16T13:47:26.099717+0000","last_fullsized":"2023-10-16T13:47:26.099717+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:48:26.805075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.f","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.101109+0000","last_change":"2023-10-16T13:47:24.438572+0000","last_active":"2023-10-16T13:47:26.101109+0000","last_peered":"2023-10-16T13:47:26.101109+0000","last_clean":"2023-10-16T13:47:26.101109+0000","last_became_active":"2023-10-16T13:47:24.438283+0000","last_became_peered":"2023-10-16T13:47:24.438283+0000","last_unstale":"2023-10-16T13:47:26.101109+0000","last_undegraded":"2023-10-16T13:47:26.101109+0000","last_fullsized":"2023-10-16T13:47:26.101109+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:48:21.499862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.e","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100703+0000","last_change":"2023-10-16T13:47:24.433506+0000","last_active":"2023-10-16T13:47:26.100703+0000","last_peered":"2023-10-16T13:47:26.100703+0000","last_clean":"2023-10-16T13:47:26.100703+0000","last_became_active":"2023-10-16T13:47:24.433218+0000","last_became_peered":"2023-10-16T13:47:24.433218+0000","last_unstale":"2023-10-16T13:47:26.100703+0000","last_undegraded":"2023-10-16T13:47:26.100703+0000","last_fullsized":"2023-10-16T13:47:26.100703+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:04:28.651883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.d","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.102503+0000","last_change":"2023-10-16T13:47:24.443171+0000","last_active":"2023-10-16T13:47:26.102503+0000","last_peered":"2023-10-16T13:47:26.102503+0000","last_clean":"2023-10-16T13:47:26.102503+0000","last_became_active":"2023-10-16T13:47:24.442811+0000","last_became_peered":"2023-10-16T13:47:24.442811+0000","last_unstale":"2023-10-16T13:47:26.102503+0000","last_undegraded":"2023-10-16T13:47:26.102503+0000","last_fullsized":"2023-10-16T13:47:26.102503+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:56:16.580840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.416732+0000","last_change":"2023-10-16T13:47:24.435000+0000","last_active":"2023-10-16T13:47:25.416732+0000","last_peered":"2023-10-16T13:47:25.416732+0000","last_clean":"2023-10-16T13:47:25.416732+0000","last_became_active":"2023-10-16T13:47:24.434568+0000","last_became_peered":"2023-10-16T13:47:24.434568+0000","last_unstale":"2023-10-16T13:47:25.416732+0000","last_undegraded":"2023-10-16T13:47:25.416732+0000","last_fullsized":"2023-10-16T13:47:25.416732+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:31:46.967495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.b","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100991+0000","last_change":"2023-10-16T13:47:24.432842+0000","last_active":"2023-10-16T13:47:26.100991+0000","last_peered":"2023-10-16T13:47:26.100991+0000","last_clean":"2023-10-16T13:47:26.100991+0000","last_became_active":"2023-10-16T13:47:24.432528+0000","last_became_peered":"2023-10-16T13:47:24.432528+0000","last_unstale":"2023-10-16T13:47:26.100991+0000","last_undegraded":"2023-10-16T13:47:26.100991+0000","last_fullsized":"2023-10-16T13:47:26.100991+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:10:57.744179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.101341+0000","last_change":"2023-10-16T13:47:24.439953+0000","last_active":"2023-10-16T13:47:26.101341+0000","last_peered":"2023-10-16T13:47:26.101341+0000","last_clean":"2023-10-16T13:47:26.101341+0000","last_became_active":"2023-10-16T13:47:24.439662+0000","last_became_peered":"2023-10-16T13:47:24.439662+0000","last_unstale":"2023-10-16T13:47:26.101341+0000","last_undegraded":"2023-10-16T13:47:26.101341+0000","last_fullsized":"2023-10-16T13:47:26.101341+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:16:33.663778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.102248+0000","last_change":"2023-10-16T13:47:24.442582+0000","last_active":"2023-10-16T13:47:26.102248+0000","last_peered":"2023-10-16T13:47:26.102248+0000","last_clean":"2023-10-16T13:47:26.102248+0000","last_became_active":"2023-10-16T13:47:24.442246+0000","last_became_peered":"2023-10-16T13:47:24.442246+0000","last_unstale":"2023-10-16T13:47:26.102248+0000","last_undegraded":"2023-10-16T13:47:26.102248+0000","last_fullsized":"2023-10-16T13:47:26.102248+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:48:26.805075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100231+0000","last_change":"2023-10-16T13:47:24.437336+0000","last_active":"2023-10-16T13:47:26.100231+0000","last_peered":"2023-10-16T13:47:26.100231+0000","last_clean":"2023-10-16T13:47:26.100231+0000","last_became_active":"2023-10-16T13:47:24.437042+0000","last_became_peered":"2023-10-16T13:47:24.437042+0000","last_unstale":"2023-10-16T13:47:26.100231+0000","last_undegraded":"2023-10-16T13:47:26.100231+0000","last_fullsized":"2023-10-16T13:47:26.100231+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:07:16.886792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.7","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.416641+0000","last_change":"2023-10-16T13:47:24.433031+0000","last_active":"2023-10-16T13:47:25.416641+0000","last_peered":"2023-10-16T13:47:25.416641+0000","last_clean":"2023-10-16T13:47:25.416641+0000","last_became_active":"2023-10-16T13:47:24.432700+0000","last_became_peered":"2023-10-16T13:47:24.432700+0000","last_unstale":"2023-10-16T13:47:25.416641+0000","last_undegraded":"2023-10-16T13:47:25.416641+0000","last_fullsized":"2023-10-16T13:47:25.416641+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:17:10.080681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.6","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.101710+0000","last_change":"2023-10-16T13:47:24.434972+0000","last_active":"2023-10-16T13:47:26.101710+0000","last_peered":"2023-10-16T13:47:26.101710+0000","last_clean":"2023-10-16T13:47:26.101710+0000","last_became_active":"2023-10-16T13:47:24.434639+0000","last_became_peered":"2023-10-16T13:47:24.434639+0000","last_unstale":"2023-10-16T13:47:26.101710+0000","last_undegraded":"2023-10-16T13:47:26.101710+0000","last_fullsized":"2023-10-16T13:47:26.101710+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:06:13.278420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.101557+0000","last_change":"2023-10-16T13:47:24.428801+0000","last_active":"2023-10-16T13:47:26.101557+0000","last_peered":"2023-10-16T13:47:26.101557+0000","last_clean":"2023-10-16T13:47:26.101557+0000","last_became_active":"2023-10-16T13:47:24.428511+0000","last_became_peered":"2023-10-16T13:47:24.428511+0000","last_unstale":"2023-10-16T13:47:26.101557+0000","last_undegraded":"2023-10-16T13:47:26.101557+0000","last_fullsized":"2023-10-16T13:47:26.101557+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:27:55.268379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.417855+0000","last_change":"2023-10-16T13:47:24.429772+0000","last_active":"2023-10-16T13:47:25.417855+0000","last_peered":"2023-10-16T13:47:25.417855+0000","last_clean":"2023-10-16T13:47:25.417855+0000","last_became_active":"2023-10-16T13:47:24.429496+0000","last_became_peered":"2023-10-16T13:47:24.429496+0000","last_unstale":"2023-10-16T13:47:25.417855+0000","last_undegraded":"2023-10-16T13:47:25.417855+0000","last_fullsized":"2023-10-16T13:47:25.417855+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:09:48.751438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.099824+0000","last_change":"2023-10-16T13:47:24.437263+0000","last_active":"2023-10-16T13:47:26.099824+0000","last_peered":"2023-10-16T13:47:26.099824+0000","last_clean":"2023-10-16T13:47:26.099824+0000","last_became_active":"2023-10-16T13:47:24.437023+0000","last_became_peered":"2023-10-16T13:47:24.437023+0000","last_unstale":"2023-10-16T13:47:26.099824+0000","last_undegraded":"2023-10-16T13:47:26.099824+0000","last_fullsized":"2023-10-16T13:47:26.099824+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:59:15.061907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.099775+0000","last_change":"2023-10-16T13:47:24.419847+0000","last_active":"2023-10-16T13:47:26.099775+0000","last_peered":"2023-10-16T13:47:26.099775+0000","last_clean":"2023-10-16T13:47:26.099775+0000","last_became_active":"2023-10-16T13:47:24.419520+0000","last_became_peered":"2023-10-16T13:47:24.419520+0000","last_unstale":"2023-10-16T13:47:26.099775+0000","last_undegraded":"2023-10-16T13:47:26.099775+0000","last_fullsized":"2023-10-16T13:47:26.099775+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:11:44.932011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.416267+0000","last_change":"2023-10-16T13:47:24.435926+0000","last_active":"2023-10-16T13:47:25.416267+0000","last_peered":"2023-10-16T13:47:25.416267+0000","last_clean":"2023-10-16T13:47:25.416267+0000","last_became_active":"2023-10-16T13:47:24.434753+0000","last_became_peered":"2023-10-16T13:47:24.434753+0000","last_unstale":"2023-10-16T13:47:25.416267+0000","last_undegraded":"2023-10-16T13:47:25.416267+0000","last_fullsized":"2023-10-16T13:47:25.416267+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:13:51.332934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.417066+0000","last_change":"2023-10-16T13:47:24.423908+0000","last_active":"2023-10-16T13:47:25.417066+0000","last_peered":"2023-10-16T13:47:25.417066+0000","last_clean":"2023-10-16T13:47:25.417066+0000","last_became_active":"2023-10-16T13:47:24.423530+0000","last_became_peered":"2023-10-16T13:47:24.423530+0000","last_unstale":"2023-10-16T13:47:25.417066+0000","last_undegraded":"2023-10-16T13:47:25.417066+0000","last_fullsized":"2023-10-16T13:47:25.417066+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:50:50.396650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.20","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.416162+0000","last_change":"2023-10-16T13:47:24.436480+0000","last_active":"2023-10-16T13:47:25.416162+0000","last_peered":"2023-10-16T13:47:25.416162+0000","last_clean":"2023-10-16T13:47:25.416162+0000","last_became_active":"2023-10-16T13:47:24.435309+0000","last_became_peered":"2023-10-16T13:47:24.435309+0000","last_unstale":"2023-10-16T13:47:25.416162+0000","last_undegraded":"2023-10-16T13:47:25.416162+0000","last_fullsized":"2023-10-16T13:47:25.416162+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:28:08.947506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1f","version":"244'4","reported_seq":16,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.452436+0000","last_change":"2023-10-16T13:47:23.425349+0000","last_active":"2023-10-16T13:47:24.452436+0000","last_peered":"2023-10-16T13:47:24.452436+0000","last_clean":"2023-10-16T13:47:24.452436+0000","last_became_active":"2023-10-16T13:47:23.425029+0000","last_became_peered":"2023-10-16T13:47:23.425029+0000","last_unstale":"2023-10-16T13:47:24.452436+0000","last_undegraded":"2023-10-16T13:47:24.452436+0000","last_fullsized":"2023-10-16T13:47:24.452436+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:57:35.834032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.22","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.102212+0000","last_change":"2023-10-16T13:47:24.431062+0000","last_active":"2023-10-16T13:47:26.102212+0000","last_peered":"2023-10-16T13:47:26.102212+0000","last_clean":"2023-10-16T13:47:26.102212+0000","last_became_active":"2023-10-16T13:47:24.430771+0000","last_became_peered":"2023-10-16T13:47:24.430771+0000","last_unstale":"2023-10-16T13:47:26.102212+0000","last_undegraded":"2023-10-16T13:47:26.102212+0000","last_fullsized":"2023-10-16T13:47:26.102212+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:23:48.085785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.1d","version":"244'2","reported_seq":14,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.448937+0000","last_change":"2023-10-16T13:47:23.428572+0000","last_active":"2023-10-16T13:47:24.448937+0000","last_peered":"2023-10-16T13:47:24.448937+0000","last_clean":"2023-10-16T13:47:24.448937+0000","last_became_active":"2023-10-16T13:47:23.428285+0000","last_became_peered":"2023-10-16T13:47:23.428285+0000","last_unstale":"2023-10-16T13:47:24.448937+0000","last_undegraded":"2023-10-16T13:47:24.448937+0000","last_fullsized":"2023-10-16T13:47:24.448937+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:26:03.650516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.23","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.417464+0000","last_change":"2023-10-16T13:47:24.432436+0000","last_active":"2023-10-16T13:47:25.417464+0000","last_peered":"2023-10-16T13:47:25.417464+0000","last_clean":"2023-10-16T13:47:25.417464+0000","last_became_active":"2023-10-16T13:47:24.428465+0000","last_became_peered":"2023-10-16T13:47:24.428465+0000","last_unstale":"2023-10-16T13:47:25.417464+0000","last_undegraded":"2023-10-16T13:47:25.417464+0000","last_fullsized":"2023-10-16T13:47:25.417464+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:16:50.560075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1c","version":"244'2","reported_seq":14,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.447290+0000","last_change":"2023-10-16T13:47:23.427686+0000","last_active":"2023-10-16T13:47:24.447290+0000","last_peered":"2023-10-16T13:47:24.447290+0000","last_clean":"2023-10-16T13:47:24.447290+0000","last_became_active":"2023-10-16T13:47:23.427013+0000","last_became_peered":"2023-10-16T13:47:23.427013+0000","last_unstale":"2023-10-16T13:47:24.447290+0000","last_undegraded":"2023-10-16T13:47:24.447290+0000","last_fullsized":"2023-10-16T13:47:24.447290+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:37:17.532164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.24","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100065+0000","last_change":"2023-10-16T13:47:24.438644+0000","last_active":"2023-10-16T13:47:26.100065+0000","last_peered":"2023-10-16T13:47:26.100065+0000","last_clean":"2023-10-16T13:47:26.100065+0000","last_became_active":"2023-10-16T13:47:24.438344+0000","last_became_peered":"2023-10-16T13:47:24.438344+0000","last_unstale":"2023-10-16T13:47:26.100065+0000","last_undegraded":"2023-10-16T13:47:26.100065+0000","last_fullsized":"2023-10-16T13:47:26.100065+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:21:19.269920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.1b","version":"244'1","reported_seq":13,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.448817+0000","last_change":"2023-10-16T13:47:23.427234+0000","last_active":"2023-10-16T13:47:24.448817+0000","last_peered":"2023-10-16T13:47:24.448817+0000","last_clean":"2023-10-16T13:47:24.448817+0000","last_became_active":"2023-10-16T13:47:23.426948+0000","last_became_peered":"2023-10-16T13:47:23.426948+0000","last_unstale":"2023-10-16T13:47:24.448817+0000","last_undegraded":"2023-10-16T13:47:24.448817+0000","last_fullsized":"2023-10-16T13:47:24.448817+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:11:43.927649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.25","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100878+0000","last_change":"2023-10-16T13:47:24.432803+0000","last_active":"2023-10-16T13:47:26.100878+0000","last_peered":"2023-10-16T13:47:26.100878+0000","last_clean":"2023-10-16T13:47:26.100878+0000","last_became_active":"2023-10-16T13:47:24.432470+0000","last_became_peered":"2023-10-16T13:47:24.432470+0000","last_unstale":"2023-10-16T13:47:26.100878+0000","last_undegraded":"2023-10-16T13:47:26.100878+0000","last_fullsized":"2023-10-16T13:47:26.100878+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:41:00.039237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.1a","version":"244'1","reported_seq":13,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.441041+0000","last_change":"2023-10-16T13:47:23.425695+0000","last_active":"2023-10-16T13:47:24.441041+0000","last_peered":"2023-10-16T13:47:24.441041+0000","last_clean":"2023-10-16T13:47:24.441041+0000","last_became_active":"2023-10-16T13:47:23.425407+0000","last_became_peered":"2023-10-16T13:47:23.425407+0000","last_unstale":"2023-10-16T13:47:24.441041+0000","last_undegraded":"2023-10-16T13:47:24.441041+0000","last_fullsized":"2023-10-16T13:47:24.441041+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:19:34.313200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.26","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100538+0000","last_change":"2023-10-16T13:47:24.439616+0000","last_active":"2023-10-16T13:47:26.100538+0000","last_peered":"2023-10-16T13:47:26.100538+0000","last_clean":"2023-10-16T13:47:26.100538+0000","last_became_active":"2023-10-16T13:47:24.435678+0000","last_became_peered":"2023-10-16T13:47:24.435678+0000","last_unstale":"2023-10-16T13:47:26.100538+0000","last_undegraded":"2023-10-16T13:47:26.100538+0000","last_fullsized":"2023-10-16T13:47:26.100538+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:39:16.296666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.418484+0000","last_change":"2023-10-16T13:47:23.431681+0000","last_active":"2023-10-16T13:47:24.418484+0000","last_peered":"2023-10-16T13:47:24.418484+0000","last_clean":"2023-10-16T13:47:24.418484+0000","last_became_active":"2023-10-16T13:47:23.431317+0000","last_became_peered":"2023-10-16T13:47:23.431317+0000","last_unstale":"2023-10-16T13:47:24.418484+0000","last_undegraded":"2023-10-16T13:47:24.418484+0000","last_fullsized":"2023-10-16T13:47:24.418484+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:20:27.852907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.27","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.101829+0000","last_change":"2023-10-16T13:47:24.423771+0000","last_active":"2023-10-16T13:47:26.101829+0000","last_peered":"2023-10-16T13:47:26.101829+0000","last_clean":"2023-10-16T13:47:26.101829+0000","last_became_active":"2023-10-16T13:47:24.423459+0000","last_became_peered":"2023-10-16T13:47:24.423459+0000","last_unstale":"2023-10-16T13:47:26.101829+0000","last_undegraded":"2023-10-16T13:47:26.101829+0000","last_fullsized":"2023-10-16T13:47:26.101829+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:27:22.393558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.421767+0000","last_change":"2023-10-16T13:47:23.433310+0000","last_active":"2023-10-16T13:47:24.421767+0000","last_peered":"2023-10-16T13:47:24.421767+0000","last_clean":"2023-10-16T13:47:24.421767+0000","last_became_active":"2023-10-16T13:47:23.433018+0000","last_became_peered":"2023-10-16T13:47:23.433018+0000","last_unstale":"2023-10-16T13:47:24.421767+0000","last_undegraded":"2023-10-16T13:47:24.421767+0000","last_fullsized":"2023-10-16T13:47:24.421767+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:49:45.992161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.28","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100635+0000","last_change":"2023-10-16T13:47:24.433316+0000","last_active":"2023-10-16T13:47:26.100635+0000","last_peered":"2023-10-16T13:47:26.100635+0000","last_clean":"2023-10-16T13:47:26.100635+0000","last_became_active":"2023-10-16T13:47:24.429160+0000","last_became_peered":"2023-10-16T13:47:24.429160+0000","last_unstale":"2023-10-16T13:47:26.100635+0000","last_undegraded":"2023-10-16T13:47:26.100635+0000","last_fullsized":"2023-10-16T13:47:26.100635+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:12:35.760185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.415527+0000","last_change":"2023-10-16T13:47:23.428721+0000","last_active":"2023-10-16T13:47:24.415527+0000","last_peered":"2023-10-16T13:47:24.415527+0000","last_clean":"2023-10-16T13:47:24.415527+0000","last_became_active":"2023-10-16T13:47:23.428349+0000","last_became_peered":"2023-10-16T13:47:23.428349+0000","last_unstale":"2023-10-16T13:47:24.415527+0000","last_undegraded":"2023-10-16T13:47:24.415527+0000","last_fullsized":"2023-10-16T13:47:24.415527+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:59:11.116638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.29","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100592+0000","last_change":"2023-10-16T13:47:24.440635+0000","last_active":"2023-10-16T13:47:26.100592+0000","last_peered":"2023-10-16T13:47:26.100592+0000","last_clean":"2023-10-16T13:47:26.100592+0000","last_became_active":"2023-10-16T13:47:24.440223+0000","last_became_peered":"2023-10-16T13:47:24.440223+0000","last_unstale":"2023-10-16T13:47:26.100592+0000","last_undegraded":"2023-10-16T13:47:26.100592+0000","last_fullsized":"2023-10-16T13:47:26.100592+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:22:53.895897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.421423+0000","last_change":"2023-10-16T13:47:23.433545+0000","last_active":"2023-10-16T13:47:24.421423+0000","last_peered":"2023-10-16T13:47:24.421423+0000","last_clean":"2023-10-16T13:47:24.421423+0000","last_became_active":"2023-10-16T13:47:23.433275+0000","last_became_peered":"2023-10-16T13:47:23.433275+0000","last_unstale":"2023-10-16T13:47:24.421423+0000","last_undegraded":"2023-10-16T13:47:24.421423+0000","last_fullsized":"2023-10-16T13:47:24.421423+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:21:17.525717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.2a","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100402+0000","last_change":"2023-10-16T13:47:24.424605+0000","last_active":"2023-10-16T13:47:26.100402+0000","last_peered":"2023-10-16T13:47:26.100402+0000","last_clean":"2023-10-16T13:47:26.100402+0000","last_became_active":"2023-10-16T13:47:24.424313+0000","last_became_peered":"2023-10-16T13:47:24.424313+0000","last_unstale":"2023-10-16T13:47:26.100402+0000","last_undegraded":"2023-10-16T13:47:26.100402+0000","last_fullsized":"2023-10-16T13:47:26.100402+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:40:33.368782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.15","version":"244'3","reported_seq":15,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.446109+0000","last_change":"2023-10-16T13:47:23.431025+0000","last_active":"2023-10-16T13:47:24.446109+0000","last_peered":"2023-10-16T13:47:24.446109+0000","last_clean":"2023-10-16T13:47:24.446109+0000","last_became_active":"2023-10-16T13:47:23.430710+0000","last_became_peered":"2023-10-16T13:47:23.430710+0000","last_unstale":"2023-10-16T13:47:24.446109+0000","last_undegraded":"2023-10-16T13:47:24.446109+0000","last_fullsized":"2023-10-16T13:47:24.446109+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:52:27.207432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2b","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100831+0000","last_change":"2023-10-16T13:47:24.427810+0000","last_active":"2023-10-16T13:47:26.100831+0000","last_peered":"2023-10-16T13:47:26.100831+0000","last_clean":"2023-10-16T13:47:26.100831+0000","last_became_active":"2023-10-16T13:47:24.427276+0000","last_became_peered":"2023-10-16T13:47:24.427276+0000","last_unstale":"2023-10-16T13:47:26.100831+0000","last_undegraded":"2023-10-16T13:47:26.100831+0000","last_fullsized":"2023-10-16T13:47:26.100831+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:16:12.551854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.14","version":"244'6","reported_seq":18,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.453158+0000","last_change":"2023-10-16T13:47:23.430528+0000","last_active":"2023-10-16T13:47:24.453158+0000","last_peered":"2023-10-16T13:47:24.453158+0000","last_clean":"2023-10-16T13:47:24.453158+0000","last_became_active":"2023-10-16T13:47:23.429721+0000","last_became_peered":"2023-10-16T13:47:23.429721+0000","last_unstale":"2023-10-16T13:47:24.453158+0000","last_undegraded":"2023-10-16T13:47:24.453158+0000","last_fullsized":"2023-10-16T13:47:24.453158+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:07:56.379842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.2c","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.416579+0000","last_change":"2023-10-16T13:47:24.436132+0000","last_active":"2023-10-16T13:47:25.416579+0000","last_peered":"2023-10-16T13:47:25.416579+0000","last_clean":"2023-10-16T13:47:25.416579+0000","last_became_active":"2023-10-16T13:47:24.435272+0000","last_became_peered":"2023-10-16T13:47:24.435272+0000","last_unstale":"2023-10-16T13:47:25.416579+0000","last_undegraded":"2023-10-16T13:47:25.416579+0000","last_fullsized":"2023-10-16T13:47:25.416579+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:07:59.434149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.13","version":"244'2","reported_seq":14,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.450104+0000","last_change":"2023-10-16T13:47:23.426620+0000","last_active":"2023-10-16T13:47:24.450104+0000","last_peered":"2023-10-16T13:47:24.450104+0000","last_clean":"2023-10-16T13:47:24.450104+0000","last_became_active":"2023-10-16T13:47:23.426328+0000","last_became_peered":"2023-10-16T13:47:23.426328+0000","last_unstale":"2023-10-16T13:47:24.450104+0000","last_undegraded":"2023-10-16T13:47:24.450104+0000","last_fullsized":"2023-10-16T13:47:24.450104+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:23:39.805153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.2d","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100631+0000","last_change":"2023-10-16T13:47:24.431734+0000","last_active":"2023-10-16T13:47:26.100631+0000","last_peered":"2023-10-16T13:47:26.100631+0000","last_clean":"2023-10-16T13:47:26.100631+0000","last_became_active":"2023-10-16T13:47:24.431414+0000","last_became_peered":"2023-10-16T13:47:24.431414+0000","last_unstale":"2023-10-16T13:47:26.100631+0000","last_undegraded":"2023-10-16T13:47:26.100631+0000","last_fullsized":"2023-10-16T13:47:26.100631+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:01:31.790432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.418559+0000","last_change":"2023-10-16T13:47:23.433222+0000","last_active":"2023-10-16T13:47:24.418559+0000","last_peered":"2023-10-16T13:47:24.418559+0000","last_clean":"2023-10-16T13:47:24.418559+0000","last_became_active":"2023-10-16T13:47:23.432936+0000","last_became_peered":"2023-10-16T13:47:23.432936+0000","last_unstale":"2023-10-16T13:47:24.418559+0000","last_undegraded":"2023-10-16T13:47:24.418559+0000","last_fullsized":"2023-10-16T13:47:24.418559+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:47:59.336633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2e","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.102342+0000","last_change":"2023-10-16T13:47:24.438281+0000","last_active":"2023-10-16T13:47:26.102342+0000","last_peered":"2023-10-16T13:47:26.102342+0000","last_clean":"2023-10-16T13:47:26.102342+0000","last_became_active":"2023-10-16T13:47:24.437976+0000","last_became_peered":"2023-10-16T13:47:24.437976+0000","last_unstale":"2023-10-16T13:47:26.102342+0000","last_undegraded":"2023-10-16T13:47:26.102342+0000","last_fullsized":"2023-10-16T13:47:26.102342+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:19:38.293171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.417965+0000","last_change":"2023-10-16T13:47:23.425872+0000","last_active":"2023-10-16T13:47:24.417965+0000","last_peered":"2023-10-16T13:47:24.417965+0000","last_clean":"2023-10-16T13:47:24.417965+0000","last_became_active":"2023-10-16T13:47:23.425569+0000","last_became_peered":"2023-10-16T13:47:23.425569+0000","last_unstale":"2023-10-16T13:47:24.417965+0000","last_undegraded":"2023-10-16T13:47:24.417965+0000","last_fullsized":"2023-10-16T13:47:24.417965+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:09:18.086723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2f","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.101809+0000","last_change":"2023-10-16T13:47:24.437030+0000","last_active":"2023-10-16T13:47:26.101809+0000","last_peered":"2023-10-16T13:47:26.101809+0000","last_clean":"2023-10-16T13:47:26.101809+0000","last_became_active":"2023-10-16T13:47:24.436734+0000","last_became_peered":"2023-10-16T13:47:24.436734+0000","last_unstale":"2023-10-16T13:47:26.101809+0000","last_undegraded":"2023-10-16T13:47:26.101809+0000","last_fullsized":"2023-10-16T13:47:26.101809+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:28:53.363293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.10","version":"244'8","reported_seq":20,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.453887+0000","last_change":"2023-10-16T13:47:23.426967+0000","last_active":"2023-10-16T13:47:24.453887+0000","last_peered":"2023-10-16T13:47:24.453887+0000","last_clean":"2023-10-16T13:47:24.453887+0000","last_became_active":"2023-10-16T13:47:23.426614+0000","last_became_peered":"2023-10-16T13:47:23.426614+0000","last_unstale":"2023-10-16T13:47:24.453887+0000","last_undegraded":"2023-10-16T13:47:24.453887+0000","last_fullsized":"2023-10-16T13:47:24.453887+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:39:12.211841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.30","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.101224+0000","last_change":"2023-10-16T13:47:24.437081+0000","last_active":"2023-10-16T13:47:26.101224+0000","last_peered":"2023-10-16T13:47:26.101224+0000","last_clean":"2023-10-16T13:47:26.101224+0000","last_became_active":"2023-10-16T13:47:24.436501+0000","last_became_peered":"2023-10-16T13:47:24.436501+0000","last_unstale":"2023-10-16T13:47:26.101224+0000","last_undegraded":"2023-10-16T13:47:26.101224+0000","last_fullsized":"2023-10-16T13:47:26.101224+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:23:30.831964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.f","version":"244'1","reported_seq":13,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.445592+0000","last_change":"2023-10-16T13:47:23.432161+0000","last_active":"2023-10-16T13:47:24.445592+0000","last_peered":"2023-10-16T13:47:24.445592+0000","last_clean":"2023-10-16T13:47:24.445592+0000","last_became_active":"2023-10-16T13:47:23.431822+0000","last_became_peered":"2023-10-16T13:47:23.431822+0000","last_unstale":"2023-10-16T13:47:24.445592+0000","last_undegraded":"2023-10-16T13:47:24.445592+0000","last_fullsized":"2023-10-16T13:47:24.445592+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:15:50.867051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.31","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100923+0000","last_change":"2023-10-16T13:47:24.431186+0000","last_active":"2023-10-16T13:47:26.100923+0000","last_peered":"2023-10-16T13:47:26.100923+0000","last_clean":"2023-10-16T13:47:26.100923+0000","last_became_active":"2023-10-16T13:47:24.430892+0000","last_became_peered":"2023-10-16T13:47:24.430892+0000","last_unstale":"2023-10-16T13:47:26.100923+0000","last_undegraded":"2023-10-16T13:47:26.100923+0000","last_fullsized":"2023-10-16T13:47:26.100923+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:12:21.693658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.e","version":"244'2","reported_seq":14,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.448939+0000","last_change":"2023-10-16T13:47:23.424549+0000","last_active":"2023-10-16T13:47:24.448939+0000","last_peered":"2023-10-16T13:47:24.448939+0000","last_clean":"2023-10-16T13:47:24.448939+0000","last_became_active":"2023-10-16T13:47:23.424111+0000","last_became_peered":"2023-10-16T13:47:23.424111+0000","last_unstale":"2023-10-16T13:47:24.448939+0000","last_undegraded":"2023-10-16T13:47:24.448939+0000","last_fullsized":"2023-10-16T13:47:24.448939+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:36:35.137200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.32","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.416229+0000","last_change":"2023-10-16T13:47:24.432940+0000","last_active":"2023-10-16T13:47:25.416229+0000","last_peered":"2023-10-16T13:47:25.416229+0000","last_clean":"2023-10-16T13:47:25.416229+0000","last_became_active":"2023-10-16T13:47:24.432610+0000","last_became_peered":"2023-10-16T13:47:24.432610+0000","last_unstale":"2023-10-16T13:47:25.416229+0000","last_undegraded":"2023-10-16T13:47:25.416229+0000","last_fullsized":"2023-10-16T13:47:25.416229+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:59:08.104923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.d","version":"244'1","reported_seq":13,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.448969+0000","last_change":"2023-10-16T13:47:23.428449+0000","last_active":"2023-10-16T13:47:24.448969+0000","last_peered":"2023-10-16T13:47:24.448969+0000","last_clean":"2023-10-16T13:47:24.448969+0000","last_became_active":"2023-10-16T13:47:23.428130+0000","last_became_peered":"2023-10-16T13:47:23.428130+0000","last_unstale":"2023-10-16T13:47:24.448969+0000","last_undegraded":"2023-10-16T13:47:24.448969+0000","last_fullsized":"2023-10-16T13:47:24.448969+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:24:27.601821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.33","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100188+0000","last_change":"2023-10-16T13:47:24.427239+0000","last_active":"2023-10-16T13:47:26.100188+0000","last_peered":"2023-10-16T13:47:26.100188+0000","last_clean":"2023-10-16T13:47:26.100188+0000","last_became_active":"2023-10-16T13:47:24.426919+0000","last_became_peered":"2023-10-16T13:47:24.426919+0000","last_unstale":"2023-10-16T13:47:26.100188+0000","last_undegraded":"2023-10-16T13:47:26.100188+0000","last_fullsized":"2023-10-16T13:47:26.100188+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:56:16.580840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.425656+0000","last_change":"2023-10-16T13:47:23.427051+0000","last_active":"2023-10-16T13:47:24.425656+0000","last_peered":"2023-10-16T13:47:24.425656+0000","last_clean":"2023-10-16T13:47:24.425656+0000","last_became_active":"2023-10-16T13:47:23.426741+0000","last_became_peered":"2023-10-16T13:47:23.426741+0000","last_unstale":"2023-10-16T13:47:24.425656+0000","last_undegraded":"2023-10-16T13:47:24.425656+0000","last_fullsized":"2023-10-16T13:47:24.425656+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:37:17.532164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.34","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100514+0000","last_change":"2023-10-16T13:47:24.444364+0000","last_active":"2023-10-16T13:47:26.100514+0000","last_peered":"2023-10-16T13:47:26.100514+0000","last_clean":"2023-10-16T13:47:26.100514+0000","last_became_active":"2023-10-16T13:47:24.444045+0000","last_became_peered":"2023-10-16T13:47:24.444045+0000","last_unstale":"2023-10-16T13:47:26.100514+0000","last_undegraded":"2023-10-16T13:47:26.100514+0000","last_fullsized":"2023-10-16T13:47:26.100514+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:13:11.795921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.b","version":"244'2","reported_seq":14,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.449802+0000","last_change":"2023-10-16T13:47:23.434957+0000","last_active":"2023-10-16T13:47:24.449802+0000","last_peered":"2023-10-16T13:47:24.449802+0000","last_clean":"2023-10-16T13:47:24.449802+0000","last_became_active":"2023-10-16T13:47:23.434637+0000","last_became_peered":"2023-10-16T13:47:23.434637+0000","last_unstale":"2023-10-16T13:47:24.449802+0000","last_undegraded":"2023-10-16T13:47:24.449802+0000","last_fullsized":"2023-10-16T13:47:24.449802+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:23:47.081423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.35","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.416431+0000","last_change":"2023-10-16T13:47:24.431892+0000","last_active":"2023-10-16T13:47:25.416431+0000","last_peered":"2023-10-16T13:47:25.416431+0000","last_clean":"2023-10-16T13:47:25.416431+0000","last_became_active":"2023-10-16T13:47:24.431624+0000","last_became_peered":"2023-10-16T13:47:24.431624+0000","last_unstale":"2023-10-16T13:47:25.416431+0000","last_undegraded":"2023-10-16T13:47:25.416431+0000","last_fullsized":"2023-10-16T13:47:25.416431+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:49:13.422163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.441201+0000","last_change":"2023-10-16T13:47:23.435342+0000","last_active":"2023-10-16T13:47:24.441201+0000","last_peered":"2023-10-16T13:47:24.441201+0000","last_clean":"2023-10-16T13:47:24.441201+0000","last_became_active":"2023-10-16T13:47:23.434999+0000","last_became_peered":"2023-10-16T13:47:23.434999+0000","last_unstale":"2023-10-16T13:47:24.441201+0000","last_undegraded":"2023-10-16T13:47:24.441201+0000","last_fullsized":"2023-10-16T13:47:24.441201+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:20:03.252858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.36","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100687+0000","last_change":"2023-10-16T13:47:24.435751+0000","last_active":"2023-10-16T13:47:26.100687+0000","last_peered":"2023-10-16T13:47:26.100687+0000","last_clean":"2023-10-16T13:47:26.100687+0000","last_became_active":"2023-10-16T13:47:24.434491+0000","last_became_peered":"2023-10-16T13:47:24.434491+0000","last_unstale":"2023-10-16T13:47:26.100687+0000","last_undegraded":"2023-10-16T13:47:26.100687+0000","last_fullsized":"2023-10-16T13:47:26.100687+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:02:56.979689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.440901+0000","last_change":"2023-10-16T13:47:23.427022+0000","last_active":"2023-10-16T13:47:24.440901+0000","last_peered":"2023-10-16T13:47:24.440901+0000","last_clean":"2023-10-16T13:47:24.440901+0000","last_became_active":"2023-10-16T13:47:23.426682+0000","last_became_peered":"2023-10-16T13:47:23.426682+0000","last_unstale":"2023-10-16T13:47:24.440901+0000","last_undegraded":"2023-10-16T13:47:24.440901+0000","last_fullsized":"2023-10-16T13:47:24.440901+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:05:08.936096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.37","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100950+0000","last_change":"2023-10-16T13:47:24.436357+0000","last_active":"2023-10-16T13:47:26.100950+0000","last_peered":"2023-10-16T13:47:26.100950+0000","last_clean":"2023-10-16T13:47:26.100950+0000","last_became_active":"2023-10-16T13:47:24.436050+0000","last_became_peered":"2023-10-16T13:47:24.436050+0000","last_unstale":"2023-10-16T13:47:26.100950+0000","last_undegraded":"2023-10-16T13:47:26.100950+0000","last_fullsized":"2023-10-16T13:47:26.100950+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:15:49.933818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.415241+0000","last_change":"2023-10-16T13:47:23.433223+0000","last_active":"2023-10-16T13:47:24.415241+0000","last_peered":"2023-10-16T13:47:24.415241+0000","last_clean":"2023-10-16T13:47:24.415241+0000","last_became_active":"2023-10-16T13:47:23.432935+0000","last_became_peered":"2023-10-16T13:47:23.432935+0000","last_unstale":"2023-10-16T13:47:24.415241+0000","last_undegraded":"2023-10-16T13:47:24.415241+0000","last_fullsized":"2023-10-16T13:47:24.415241+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:40:59.034875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.38","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100456+0000","last_change":"2023-10-16T13:47:24.437795+0000","last_active":"2023-10-16T13:47:26.100456+0000","last_peered":"2023-10-16T13:47:26.100456+0000","last_clean":"2023-10-16T13:47:26.100456+0000","last_became_active":"2023-10-16T13:47:24.437493+0000","last_became_peered":"2023-10-16T13:47:24.437493+0000","last_unstale":"2023-10-16T13:47:26.100456+0000","last_undegraded":"2023-10-16T13:47:26.100456+0000","last_fullsized":"2023-10-16T13:47:26.100456+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:32:36.771595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.7","version":"244'2","reported_seq":14,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.451763+0000","last_change":"2023-10-16T13:47:23.430865+0000","last_active":"2023-10-16T13:47:24.451763+0000","last_peered":"2023-10-16T13:47:24.451763+0000","last_clean":"2023-10-16T13:47:24.451763+0000","last_became_active":"2023-10-16T13:47:23.430493+0000","last_became_peered":"2023-10-16T13:47:23.430493+0000","last_unstale":"2023-10-16T13:47:24.451763+0000","last_undegraded":"2023-10-16T13:47:24.451763+0000","last_fullsized":"2023-10-16T13:47:24.451763+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:59:14.057546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.39","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.101988+0000","last_change":"2023-10-16T13:47:24.443917+0000","last_active":"2023-10-16T13:47:26.101988+0000","last_peered":"2023-10-16T13:47:26.101988+0000","last_clean":"2023-10-16T13:47:26.101988+0000","last_became_active":"2023-10-16T13:47:24.443547+0000","last_became_peered":"2023-10-16T13:47:24.443547+0000","last_unstale":"2023-10-16T13:47:26.101988+0000","last_undegraded":"2023-10-16T13:47:26.101988+0000","last_fullsized":"2023-10-16T13:47:26.101988+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:21:18.530078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.417912+0000","last_change":"2023-10-16T13:47:23.430851+0000","last_active":"2023-10-16T13:47:24.417912+0000","last_peered":"2023-10-16T13:47:24.417912+0000","last_clean":"2023-10-16T13:47:24.417912+0000","last_became_active":"2023-10-16T13:47:23.430526+0000","last_became_peered":"2023-10-16T13:47:23.430526+0000","last_unstale":"2023-10-16T13:47:24.417912+0000","last_undegraded":"2023-10-16T13:47:24.417912+0000","last_fullsized":"2023-10-16T13:47:24.417912+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:40:16.878541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.3a","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100155+0000","last_change":"2023-10-16T13:47:24.440347+0000","last_active":"2023-10-16T13:47:26.100155+0000","last_peered":"2023-10-16T13:47:26.100155+0000","last_clean":"2023-10-16T13:47:26.100155+0000","last_became_active":"2023-10-16T13:47:24.440041+0000","last_became_peered":"2023-10-16T13:47:24.440041+0000","last_unstale":"2023-10-16T13:47:26.100155+0000","last_undegraded":"2023-10-16T13:47:26.100155+0000","last_fullsized":"2023-10-16T13:47:26.100155+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:19:38.293171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.415325+0000","last_change":"2023-10-16T13:47:23.427399+0000","last_active":"2023-10-16T13:47:24.415325+0000","last_peered":"2023-10-16T13:47:24.415325+0000","last_clean":"2023-10-16T13:47:24.415325+0000","last_became_active":"2023-10-16T13:47:23.427092+0000","last_became_peered":"2023-10-16T13:47:23.427092+0000","last_unstale":"2023-10-16T13:47:24.415325+0000","last_undegraded":"2023-10-16T13:47:24.415325+0000","last_fullsized":"2023-10-16T13:47:24.415325+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:10:56.739817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3b","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.101556+0000","last_change":"2023-10-16T13:47:24.436718+0000","last_active":"2023-10-16T13:47:26.101556+0000","last_peered":"2023-10-16T13:47:26.101556+0000","last_clean":"2023-10-16T13:47:26.101556+0000","last_became_active":"2023-10-16T13:47:24.436371+0000","last_became_peered":"2023-10-16T13:47:24.436371+0000","last_unstale":"2023-10-16T13:47:26.101556+0000","last_undegraded":"2023-10-16T13:47:26.101556+0000","last_fullsized":"2023-10-16T13:47:26.101556+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:58:12.134727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.416043+0000","last_change":"2023-10-16T13:47:23.433833+0000","last_active":"2023-10-16T13:47:24.416043+0000","last_peered":"2023-10-16T13:47:24.416043+0000","last_clean":"2023-10-16T13:47:24.416043+0000","last_became_active":"2023-10-16T13:47:23.433381+0000","last_became_peered":"2023-10-16T13:47:23.433381+0000","last_unstale":"2023-10-16T13:47:24.416043+0000","last_undegraded":"2023-10-16T13:47:24.416043+0000","last_fullsized":"2023-10-16T13:47:24.416043+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:09:34.543970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3c","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.099990+0000","last_change":"2023-10-16T13:47:24.424889+0000","last_active":"2023-10-16T13:47:26.099990+0000","last_peered":"2023-10-16T13:47:26.099990+0000","last_clean":"2023-10-16T13:47:26.099990+0000","last_became_active":"2023-10-16T13:47:24.424573+0000","last_became_peered":"2023-10-16T13:47:24.424573+0000","last_unstale":"2023-10-16T13:47:26.099990+0000","last_undegraded":"2023-10-16T13:47:26.099990+0000","last_fullsized":"2023-10-16T13:47:26.099990+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:23:40.809515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.441247+0000","last_change":"2023-10-16T13:47:23.435920+0000","last_active":"2023-10-16T13:47:24.441247+0000","last_peered":"2023-10-16T13:47:24.441247+0000","last_clean":"2023-10-16T13:47:24.441247+0000","last_became_active":"2023-10-16T13:47:23.435675+0000","last_became_peered":"2023-10-16T13:47:23.435675+0000","last_unstale":"2023-10-16T13:47:24.441247+0000","last_undegraded":"2023-10-16T13:47:24.441247+0000","last_fullsized":"2023-10-16T13:47:24.441247+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:01:54.403340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.3d","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.417707+0000","last_change":"2023-10-16T13:47:24.430727+0000","last_active":"2023-10-16T13:47:25.417707+0000","last_peered":"2023-10-16T13:47:25.417707+0000","last_clean":"2023-10-16T13:47:25.417707+0000","last_became_active":"2023-10-16T13:47:24.430424+0000","last_became_peered":"2023-10-16T13:47:24.430424+0000","last_unstale":"2023-10-16T13:47:25.417707+0000","last_undegraded":"2023-10-16T13:47:25.417707+0000","last_fullsized":"2023-10-16T13:47:25.417707+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:05:49.279256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.418627+0000","last_change":"2023-10-16T13:47:23.430779+0000","last_active":"2023-10-16T13:47:24.418627+0000","last_peered":"2023-10-16T13:47:24.418627+0000","last_clean":"2023-10-16T13:47:24.418627+0000","last_became_active":"2023-10-16T13:47:23.429996+0000","last_became_peered":"2023-10-16T13:47:23.429996+0000","last_unstale":"2023-10-16T13:47:24.418627+0000","last_undegraded":"2023-10-16T13:47:24.418627+0000","last_fullsized":"2023-10-16T13:47:24.418627+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:17:22.079776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.3e","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.416486+0000","last_change":"2023-10-16T13:47:24.436378+0000","last_active":"2023-10-16T13:47:25.416486+0000","last_peered":"2023-10-16T13:47:25.416486+0000","last_clean":"2023-10-16T13:47:25.416486+0000","last_became_active":"2023-10-16T13:47:24.435495+0000","last_became_peered":"2023-10-16T13:47:24.435495+0000","last_unstale":"2023-10-16T13:47:25.416486+0000","last_undegraded":"2023-10-16T13:47:25.416486+0000","last_fullsized":"2023-10-16T13:47:25.416486+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:18:23.073022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.420122+0000","last_change":"2023-10-16T13:47:23.433945+0000","last_active":"2023-10-16T13:47:24.420122+0000","last_peered":"2023-10-16T13:47:24.420122+0000","last_clean":"2023-10-16T13:47:24.420122+0000","last_became_active":"2023-10-16T13:47:23.433576+0000","last_became_peered":"2023-10-16T13:47:23.433576+0000","last_unstale":"2023-10-16T13:47:24.420122+0000","last_undegraded":"2023-10-16T13:47:24.420122+0000","last_fullsized":"2023-10-16T13:47:24.420122+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:09:34.543970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.3f","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:26.100803+0000","last_change":"2023-10-16T13:47:24.441874+0000","last_active":"2023-10-16T13:47:26.100803+0000","last_peered":"2023-10-16T13:47:26.100803+0000","last_clean":"2023-10-16T13:47:26.100803+0000","last_became_active":"2023-10-16T13:47:24.441466+0000","last_became_peered":"2023-10-16T13:47:24.441466+0000","last_unstale":"2023-10-16T13:47:26.100803+0000","last_undegraded":"2023-10-16T13:47:26.100803+0000","last_fullsized":"2023-10-16T13:47:26.100803+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:22:04.707731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":12,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.437625+0000","last_change":"2023-10-16T13:47:23.423270+0000","last_active":"2023-10-16T13:47:24.437625+0000","last_peered":"2023-10-16T13:47:24.437625+0000","last_clean":"2023-10-16T13:47:24.437625+0000","last_became_active":"2023-10-16T13:47:23.422889+0000","last_became_peered":"2023-10-16T13:47:23.422889+0000","last_unstale":"2023-10-16T13:47:24.437625+0000","last_undegraded":"2023-10-16T13:47:24.437625+0000","last_fullsized":"2023-10-16T13:47:24.437625+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:30:37.923537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.21","version":"0'0","reported_seq":11,"reported_epoch":245,"state":"active+clean","last_fresh":"2023-10-16T13:47:25.417066+0000","last_change":"2023-10-16T13:47:24.431657+0000","last_active":"2023-10-16T13:47:25.417066+0000","last_peered":"2023-10-16T13:47:25.417066+0000","last_clean":"2023-10-16T13:47:25.417066+0000","last_became_active":"2023-10-16T13:47:24.431385+0000","last_became_peered":"2023-10-16T13:47:24.431385+0000","last_unstale":"2023-10-16T13:47:25.417066+0000","last_undegraded":"2023-10-16T13:47:25.417066+0000","last_fullsized":"2023-10-16T13:47:25.417066+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:59:09.754185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1e","version":"244'1","reported_seq":13,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.448637+0000","last_change":"2023-10-16T13:47:23.429601+0000","last_active":"2023-10-16T13:47:24.448637+0000","last_peered":"2023-10-16T13:47:24.448637+0000","last_clean":"2023-10-16T13:47:24.448637+0000","last_became_active":"2023-10-16T13:47:23.429298+0000","last_became_peered":"2023-10-16T13:47:23.429298+0000","last_unstale":"2023-10-16T13:47:24.448637+0000","last_undegraded":"2023-10-16T13:47:24.448637+0000","last_fullsized":"2023-10-16T13:47:24.448637+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:20:00.827725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"231'370","reported_seq":1298,"reported_epoch":244,"state":"active+clean","last_fresh":"2023-10-16T13:47:24.425828+0000","last_change":"2023-10-16T12:10:34.746820+0000","last_active":"2023-10-16T13:47:24.425828+0000","last_peered":"2023-10-16T13:47:24.425828+0000","last_clean":"2023-10-16T13:47:24.425828+0000","last_became_active":"2023-10-16T12:10:34.745574+0000","last_became_peered":"2023-10-16T12:10:34.745574+0000","last_unstale":"2023-10-16T13:47:24.425828+0000","last_undegraded":"2023-10-16T13:47:24.425828+0000","last_fullsized":"2023-10-16T13:47:24.425828+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_clean_scrub_stamp":"2023-10-16T12:10:33.577111+0000","objects_scrubbed":0,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:14.461986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":32,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":31,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":370,"ondisk_log_size":370,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575996,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56992,"kb_used_data":3604,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314848,"statfs":{"total":96636764160,"available":96578404352,"internally_reserved":0,"allocated":3690496,"data_stored":2186927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54655435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:46:50 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.60299999999999998,"15min":0.63200000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.33000000000000002},"max":{"1min":0.78600000000000003,"5min":0.87,"15min":0.93799999999999994},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.60699999999999998,"15min":0.626},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.72099999999999997,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.57499999999999996}]},{"osd":1,"last update":"Mon Oct 16 13:46:50 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.61299999999999999,"15min":0.63900000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.79000000000000004,"5min":0.84599999999999997,"15min":0.95899999999999996},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.622,"15min":0.63100000000000001},"min":{"1min":0.38200000000000001,"5min":0.375,"15min":0.375},"max":{"1min":0.76200000000000001,"5min":0.79100000000000004,"15min":0.86399999999999999},"last":0.65000000000000002}]},{"osd":2,"last update":"Mon Oct 16 13:46:50 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.67400000000000004,"15min":0.68600000000000005},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.80300000000000005,"5min":0.83899999999999997,"15min":0.88200000000000001},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.68899999999999995,"15min":0.69799999999999995},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.39400000000000002},"max":{"1min":0.83699999999999997,"5min":0.92800000000000005,"15min":0.93799999999999994},"last":0.73099999999999998}]},{"osd":3,"last update":"Mon Oct 16 13:46:50 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.63800000000000001,"15min":0.65900000000000003},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.85299999999999998,"5min":0.89200000000000002,"15min":0.91300000000000003},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.65000000000000002,"15min":0.66000000000000003},"min":{"1min":0.435,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.88,"5min":1.24,"15min":1.24},"last":0.71499999999999997}]},{"osd":4,"last update":"Mon Oct 16 13:46:50 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.52000000000000002,"15min":0.51900000000000002},"min":{"1min":0.311,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.71099999999999997,"5min":0.745,"15min":0.85299999999999998},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.621,"15min":0.61799999999999999},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.33100000000000002},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.52800000000000002}]},{"osd":5,"last update":"Mon Oct 16 13:46:50 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.54900000000000004,"15min":0.54700000000000004},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.85699999999999998,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.58399999999999996,"15min":0.58699999999999997},"min":{"1min":0.41099999999999998,"5min":0.38400000000000001,"15min":0.35099999999999998},"max":{"1min":0.81100000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.48199999999999998}]},{"osd":6,"last update":"Mon Oct 16 13:46:50 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.60499999999999998,"15min":0.61299999999999999},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.85499999999999998,"5min":0.874,"15min":0.95899999999999996},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.626,"15min":0.61899999999999999},"min":{"1min":0.39400000000000002,"5min":0.378,"15min":0.35199999999999998},"max":{"1min":0.92100000000000004,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.46500000000000002}]}]},{"osd":6,"up_from":13,"seq":55834575994,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60572,"kb_used_data":3024,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311268,"statfs":{"total":96636764160,"available":96574738432,"internally_reserved":0,"allocated":3096576,"data_stored":1596547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58915271},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:46:36 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.65300000000000002,"15min":0.66800000000000004},"min":{"1min":0.41499999999999998,"5min":0.375,"15min":0.28699999999999998},"max":{"1min":0.82599999999999996,"5min":0.89400000000000002,"15min":0.92300000000000004},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.64500000000000002,"15min":0.66400000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.80400000000000005,"5min":0.91400000000000003,"15min":0.93799999999999994},"last":0.68200000000000005}]},{"osd":1,"last update":"Mon Oct 16 13:46:36 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.58599999999999997,"15min":0.624},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.33100000000000002},"max":{"1min":0.70299999999999996,"5min":0.86899999999999999,"15min":0.90900000000000003},"last":0.504},{"interface":"front","average":{"1min":0.63,"5min":0.61399999999999999,"15min":0.62},"min":{"1min":0.41699999999999998,"5min":0.39800000000000002,"15min":0.38300000000000001},"max":{"1min":0.85799999999999998,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.42199999999999999}]},{"osd":2,"last update":"Mon Oct 16 13:46:36 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.622,"15min":0.63300000000000001},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.85799999999999998,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.59999999999999998,"15min":0.624},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.749,"5min":0.89000000000000001,"15min":0.91000000000000003},"last":0.44500000000000001}]},{"osd":3,"last update":"Mon Oct 16 13:46:36 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63900000000000001,"15min":0.66300000000000003},"min":{"1min":0.439,"5min":0.39900000000000002,"15min":0.38200000000000001},"max":{"1min":0.874,"5min":0.92400000000000004,"15min":0.95599999999999996},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.69099999999999995,"15min":0.71599999999999997},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.42299999999999999},"max":{"1min":0.85099999999999998,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.48099999999999998}]},{"osd":4,"last update":"Mon Oct 16 13:46:36 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.51800000000000002,"15min":0.52700000000000002},"min":{"1min":0.39000000000000001,"5min":0.35999999999999999,"15min":0.315},"max":{"1min":0.78000000000000003,"5min":0.85599999999999998,"15min":0.95799999999999996},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61699999999999999,"15min":0.62},"min":{"1min":0.375,"5min":0.35499999999999998,"15min":0.32900000000000001},"max":{"1min":0.83699999999999997,"5min":0.98799999999999999,"15min":1.0309999999999999},"last":0.64600000000000002}]},{"osd":5,"last update":"Mon Oct 16 13:46:36 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.59799999999999998,"15min":0.59399999999999997},"min":{"1min":0.39800000000000002,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.81100000000000005,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.66500000000000004,"15min":0.66000000000000003},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.308},"max":{"1min":0.85899999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.53500000000000003}]},{"osd":7,"last update":"Mon Oct 16 13:46:36 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.60899999999999999,"15min":0.60499999999999998},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.80000000000000004,"5min":1.081,"15min":1.081},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65200000000000002},"min":{"1min":0.42799999999999999,"5min":0.41899999999999998,"15min":0.375},"max":{"1min":0.91500000000000004,"5min":1.099,"15min":1.099},"last":0.629}]}]},{"osd":1,"up_from":13,"seq":55834575994,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52252,"kb_used_data":3016,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319588,"statfs":{"total":96636764160,"available":96583258112,"internally_reserved":0,"allocated":3088384,"data_stored":1596435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":50395593},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:46:55 2023","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.39500000000000002,"15min":0.40699999999999997},"min":{"1min":0.27600000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.51300000000000001,"5min":0.65400000000000003,"15min":0.65800000000000003},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.47599999999999998,"15min":0.49199999999999999},"min":{"1min":0.32000000000000001,"5min":0.253,"15min":0.253},"max":{"1min":0.56000000000000005,"5min":0.70999999999999996,"15min":0.77200000000000002},"last":0.67000000000000004}]},{"osd":2,"last update":"Mon Oct 16 13:46:55 2023","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.436,"15min":0.45100000000000001},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.58599999999999997,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.48099999999999998,"15min":0.48999999999999999},"min":{"1min":0.34399999999999997,"5min":0.315,"15min":0.30199999999999999},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.71399999999999997},"last":0.65200000000000002}]},{"osd":3,"last update":"Mon Oct 16 13:46:55 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.42899999999999999,"15min":0.42799999999999999},"min":{"1min":0.307,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.50800000000000001,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.46100000000000002,"15min":0.47499999999999998},"min":{"1min":0.29799999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.52800000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.69399999999999995}]},{"osd":4,"last update":"Mon Oct 16 13:46:55 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.64400000000000002,"15min":0.66200000000000003},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.42399999999999999},"max":{"1min":0.78600000000000003,"5min":0.95899999999999996,"15min":1.0069999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.64200000000000002,"15min":0.66600000000000004},"min":{"1min":0.42899999999999999,"5min":0.37,"15min":0.37},"max":{"1min":0.69999999999999996,"5min":0.91300000000000003,"15min":0.997},"last":0.747}]},{"osd":5,"last update":"Mon Oct 16 13:46:55 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.68200000000000005,"15min":0.70699999999999996},"min":{"1min":0.45800000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.83999999999999997,"5min":0.97499999999999998,"15min":1.0760000000000001},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.70499999999999996,"15min":0.72599999999999998},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.81399999999999995,"5min":1.0469999999999999,"15min":1.0660000000000001},"last":0.73199999999999998}]},{"osd":6,"last update":"Mon Oct 16 13:46:55 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.67500000000000004,"15min":0.68000000000000005},"min":{"1min":0.46200000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.75900000000000001,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.68700000000000006,"15min":0.70499999999999996},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.80400000000000005,"5min":1.03,"15min":1.083},"last":0.72299999999999998}]},{"osd":7,"last update":"Mon Oct 16 13:46:55 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.70899999999999996,"15min":0.73399999999999999},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.77300000000000002,"5min":1.0629999999999999,"15min":1.0660000000000001},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.72899999999999998,"15min":0.746},"min":{"1min":0.496,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.82099999999999995,"5min":1.042,"15min":1.0580000000000001},"last":0.87}]}]},{"osd":0,"up_from":13,"seq":55834575995,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48676,"kb_used_data":3608,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323164,"statfs":{"total":96636764160,"available":96586919936,"internally_reserved":0,"allocated":3694592,"data_stored":2188415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 13:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.40400000000000003,"15min":0.42099999999999999},"min":{"1min":0.29099999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.60699999999999998,"5min":0.65800000000000003,"15min":0.71399999999999997},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.42199999999999999,"15min":0.439},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.82199999999999995},"last":0.41999999999999998}]},{"osd":2,"last update":"Mon Oct 16 13:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.443,"15min":0.45400000000000001},"min":{"1min":0.28599999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.61399999999999999,"5min":0.65900000000000003,"15min":0.75700000000000001},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.46400000000000002,"15min":0.48499999999999999},"min":{"1min":0.29799999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.83299999999999996},"last":0.60499999999999998}]},{"osd":3,"last update":"Mon Oct 16 13:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.42699999999999999,"15min":0.45300000000000001},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.56499999999999995,"5min":0.63400000000000001,"15min":0.78600000000000003},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.50900000000000001,"15min":0.52000000000000002},"min":{"1min":0.317,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.66400000000000003,"5min":0.82999999999999996,"15min":0.86399999999999999},"last":0.57199999999999995}]},{"osd":4,"last update":"Mon Oct 16 13:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.66400000000000003,"15min":0.68400000000000005},"min":{"1min":0.42399999999999999,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.66500000000000004,"15min":0.68899999999999995},"min":{"1min":0.499,"5min":0.375,"15min":0.375},"max":{"1min":1.21,"5min":1.21,"15min":1.21},"last":0.78200000000000003}]},{"osd":5,"last update":"Mon Oct 16 13:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.66400000000000003,"15min":0.68999999999999995},"min":{"1min":0.41999999999999998,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.242,"5min":1.242,"15min":1.242},"last":0.751},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.70599999999999996,"15min":0.72299999999999998},"min":{"1min":0.497,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.998,"5min":1.081,"15min":1.081},"last":0.63200000000000001}]},{"osd":6,"last update":"Mon Oct 16 13:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.71199999999999997,"15min":0.72899999999999998},"min":{"1min":0.51900000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.96599999999999997,"5min":1.034,"15min":1.103},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.70399999999999996,"15min":0.72299999999999998},"min":{"1min":0.51000000000000001,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":0.79600000000000004}]},{"osd":7,"last update":"Mon Oct 16 13:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.71199999999999997,"15min":0.74199999999999999},"min":{"1min":0.48399999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.1919999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.72499999999999998,"15min":0.73499999999999999},"min":{"1min":0.51300000000000001,"5min":0.433,"15min":0.433},"max":{"1min":1.0469999999999999,"5min":1.087,"15min":1.087},"last":0.64700000000000002}]}]},{"osd":2,"up_from":13,"seq":55834575995,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52252,"kb_used_data":3024,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319588,"statfs":{"total":96636764160,"available":96583258112,"internally_reserved":0,"allocated":3096576,"data_stored":1597087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.41499999999999998,"15min":0.40899999999999997},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.623,"5min":0.71699999999999997,"15min":0.77500000000000002},"last":0.438},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.47699999999999998,"15min":0.47999999999999998},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.52800000000000002}]},{"osd":1,"last update":"Mon Oct 16 13:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.502,"15min":0.5},"min":{"1min":0.317,"5min":0.311,"15min":0.311},"max":{"1min":0.66800000000000004,"5min":0.73199999999999998,"15min":0.81399999999999995},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.48999999999999999,"15min":0.47699999999999998},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.27700000000000002},"max":{"1min":0.69999999999999996,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.58599999999999997}]},{"osd":3,"last update":"Mon Oct 16 13:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.443,"15min":0.44},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.59699999999999998,"5min":0.70099999999999996,"15min":0.72599999999999998},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.54300000000000004,"15min":0.54400000000000004},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.82699999999999996,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.74099999999999999}]},{"osd":4,"last update":"Mon Oct 16 13:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.72699999999999998,"15min":0.71799999999999997},"min":{"1min":0.51500000000000001,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.89000000000000001,"5min":1.3600000000000001,"15min":1.3600000000000001},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.72599999999999998,"15min":0.71899999999999997},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.82899999999999996,"5min":1.403,"15min":1.403},"last":0.91400000000000003}]},{"osd":5,"last update":"Mon Oct 16 13:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.68600000000000005,"15min":0.68899999999999995},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.81100000000000005,"5min":1.486,"15min":1.486},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.71199999999999997,"15min":0.69999999999999996},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.90200000000000002,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":0.879}]},{"osd":6,"last update":"Mon Oct 16 13:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.67600000000000005,"15min":0.66600000000000004},"min":{"1min":0.44500000000000001,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.80200000000000005,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.72599999999999998,"15min":0.72199999999999998},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.90600000000000003,"5min":1.4379999999999999,"15min":1.4379999999999999},"last":0.94299999999999995}]},{"osd":7,"last update":"Mon Oct 16 13:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.73899999999999999,"15min":0.72099999999999997},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.94099999999999995,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":0.97099999999999997},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.72999999999999998,"15min":0.73099999999999998},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.91500000000000004,"5min":1.377,"15min":1.377},"last":1.0429999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575994,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56428,"kb_used_data":3040,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315412,"statfs":{"total":96636764160,"available":96578981888,"internally_reserved":0,"allocated":3112960,"data_stored":1598685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.40300000000000002,"15min":0.40999999999999998},"min":{"1min":0.23499999999999999,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.57199999999999995,"5min":0.64200000000000002,"15min":0.72899999999999998},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.41499999999999998,"15min":0.42799999999999999},"min":{"1min":0.27300000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.60099999999999998,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.312}]},{"osd":1,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.42699999999999999,"15min":0.434},"min":{"1min":0.26700000000000002,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.55900000000000005,"5min":0.68200000000000005,"15min":0.76300000000000001},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.435,"5min":0.45900000000000002,"15min":0.47999999999999998},"min":{"1min":0.27700000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.65500000000000003,"5min":0.70699999999999996,"15min":0.77800000000000002},"last":0.441}]},{"osd":2,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.47999999999999998,"15min":0.49299999999999999},"min":{"1min":0.33800000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.621,"5min":1.167,"15min":1.167},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.501,"15min":0.52300000000000002},"min":{"1min":0.312,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.625,"5min":0.73299999999999998,"15min":0.78700000000000003},"last":0.49199999999999999}]},{"osd":4,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.65100000000000002,"15min":0.66500000000000004},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.91200000000000003,"5min":0.998,"15min":1.137},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.68899999999999995,"15min":0.69399999999999995},"min":{"1min":0.441,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.087,"5min":1.2110000000000001,"15min":1.2909999999999999},"last":0.72799999999999998}]},{"osd":5,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.67700000000000005,"15min":0.70499999999999996},"min":{"1min":0.45500000000000002,"5min":0.42899999999999999,"15min":0.374},"max":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.2729999999999999},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.69299999999999995,"15min":0.69999999999999996},"min":{"1min":0.45600000000000002,"5min":0.432,"15min":0.432},"max":{"1min":0.98799999999999999,"5min":1.0549999999999999,"15min":1.173},"last":0.68600000000000005}]},{"osd":6,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.72899999999999998,"15min":0.71899999999999997},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.41799999999999998},"max":{"1min":0.99099999999999999,"5min":5.0999999999999996,"15min":5.0999999999999996},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.70999999999999996,"15min":0.73399999999999999},"min":{"1min":0.49199999999999999,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.0029999999999999,"5min":1.0309999999999999,"15min":1.2290000000000001},"last":0.80000000000000004}]},{"osd":7,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.71899999999999997,"15min":0.73399999999999999},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.048,"5min":1.048,"15min":1.2569999999999999},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.73299999999999998,"15min":0.746},"min":{"1min":0.501,"5min":0.47799999999999998,"15min":0.43099999999999999},"max":{"1min":1.0369999999999999,"5min":1.262,"15min":1.262},"last":0.66800000000000004}]}]},{"osd":4,"up_from":13,"seq":55834575993,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52248,"kb_used_data":3020,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319592,"statfs":{"total":96636764160,"available":96583262208,"internally_reserved":0,"allocated":3092480,"data_stored":1596505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.60699999999999998,"15min":0.625},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.33800000000000002},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.623,"15min":0.63100000000000001},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.84099999999999997,"5min":0.878,"15min":0.89500000000000002},"last":0.55800000000000005}]},{"osd":1,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.63,"15min":0.65600000000000003},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.628,"15min":0.63700000000000001},"min":{"1min":0.41099999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.65200000000000002}]},{"osd":2,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.65200000000000002,"15min":0.65300000000000002},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.83999999999999997,"5min":0.89900000000000002,"15min":0.90600000000000003},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.65600000000000003,"15min":0.65800000000000003},"min":{"1min":0.39600000000000002,"5min":0.39400000000000002,"15min":0.38400000000000001},"max":{"1min":0.88200000000000001,"5min":0.89700000000000002,"15min":0.92600000000000005},"last":0.622}]},{"osd":3,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.63200000000000001,"15min":0.64500000000000002},"min":{"1min":0.40400000000000003,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.78100000000000003,"5min":0.86599999999999999,"15min":0.99199999999999999},"last":0.502},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.67400000000000004,"15min":0.68200000000000005},"min":{"1min":0.44800000000000001,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.79100000000000004,"5min":0.872,"15min":0.93899999999999995},"last":0.48199999999999998}]},{"osd":5,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55100000000000005,"15min":0.53000000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.57999999999999996,"15min":0.56899999999999995},"min":{"1min":0.36099999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.82199999999999995,"5min":0.82299999999999995,"15min":0.94899999999999995},"last":0.60499999999999998}]},{"osd":6,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.625,"15min":0.59799999999999998},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.33200000000000002},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92700000000000005},"last":0.378},{"interface":"front","average":{"1min":0.747,"5min":0.71399999999999997,"15min":0.69399999999999995},"min":{"1min":0.52600000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.129,"5min":1.129,"15min":1.129},"last":0.58999999999999997}]},{"osd":7,"last update":"Mon Oct 16 13:46:40 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.66000000000000003,"15min":0.63500000000000001},"min":{"1min":0.44400000000000001,"5min":0.41999999999999998,"15min":0.35799999999999998},"max":{"1min":0.97099999999999997,"5min":0.97499999999999998,"15min":0.98599999999999999},"last":0.432},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.70799999999999996,"15min":0.68400000000000005},"min":{"1min":0.44,"5min":0.433,"15min":0.38200000000000001},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":1.026},"last":0.66700000000000004}]}]},{"osd":5,"up_from":13,"seq":55834575994,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52244,"kb_used_data":3016,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319596,"statfs":{"total":96636764160,"available":96583266304,"internally_reserved":0,"allocated":3088384,"data_stored":1596435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:46:24 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.61399999999999999,"15min":0.61499999999999999},"min":{"1min":0.314,"5min":0.314,"15min":0.26800000000000002},"max":{"1min":0.878,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.746},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63,"15min":0.64200000000000002},"min":{"1min":0.38700000000000001,"5min":0.35899999999999999,"15min":0.29699999999999999},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.63500000000000001}]},{"osd":1,"last update":"Mon Oct 16 13:46:24 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.66000000000000003,"15min":0.66200000000000003},"min":{"1min":0.44600000000000001,"5min":0.34300000000000003,"15min":0.29499999999999998},"max":{"1min":1,"5min":1,"15min":1},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.63800000000000001,"15min":0.64000000000000001},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.72999999999999998}]},{"osd":2,"last update":"Mon Oct 16 13:46:24 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.67000000000000004,"15min":0.65700000000000003},"min":{"1min":0.41699999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.91000000000000003,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.621},{"interface":"front","average":{"1min":0.629,"5min":0.65700000000000003,"15min":0.64900000000000002},"min":{"1min":0.42599999999999999,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.83899999999999997,"5min":0.871,"15min":0.93500000000000005},"last":0.80200000000000005}]},{"osd":3,"last update":"Mon Oct 16 13:46:24 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.67400000000000004,"15min":0.66400000000000003},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.69899999999999995,"15min":0.69499999999999995},"min":{"1min":0.41399999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.4199999999999999,"5min":1.4199999999999999,"15min":1.4199999999999999},"last":0.77300000000000002}]},{"osd":4,"last update":"Mon Oct 16 13:46:24 2023","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.54400000000000004,"15min":0.53300000000000003},"min":{"1min":0.34100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.82499999999999996,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.59299999999999997,"15min":0.58899999999999997},"min":{"1min":0.34300000000000003,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.85399999999999998,"5min":0.89000000000000001,"15min":0.93400000000000005},"last":0.47999999999999998}]},{"osd":6,"last update":"Mon Oct 16 13:46:24 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.58899999999999997,"15min":0.57999999999999996},"min":{"1min":0.38700000000000001,"5min":0.315,"15min":0.315},"max":{"1min":0.877,"5min":0.88900000000000001,"15min":0.93200000000000005},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.61899999999999999},"min":{"1min":0.372,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89600000000000002},"last":0.504}]},{"osd":7,"last update":"Mon Oct 16 13:46:24 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.64800000000000002,"15min":0.63300000000000001},"min":{"1min":0.378,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.69699999999999995,"15min":0.68400000000000005},"min":{"1min":0.46200000000000002,"5min":0.375,"15min":0.375},"max":{"1min":0.94699999999999995,"5min":0.98299999999999998,"15min":1.0229999999999999},"last":0.53400000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T13:47:33.659 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-16T13:47:33.660 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-16T13:47:33.661 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:47:33.697 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:47:33.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.826+0000 7f13f3cc0700 1 Processor -- start 2023-10-16T13:47:33.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.826+0000 7f13f3cc0700 1 -- start start 2023-10-16T13:47:33.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.827+0000 7f13f3cc0700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f13ec0cb880 0x7f13ec0cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:33.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.827+0000 7f13f3cc0700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13ec0cef70 con 0x7f13ec15b070 2023-10-16T13:47:33.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.827+0000 7f13f3cc0700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13ec0ce360 con 0x7f13ec06da60 2023-10-16T13:47:33.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.827+0000 7f13f3cc0700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13ec0ce640 con 0x7f13ec0cb880 2023-10-16T13:47:33.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.827+0000 7f13f1a5c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f13ec15b070 0x7f13ec15b450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44838/0 (socket says 172.21.15.16:44838) 2023-10-16T13:47:33.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.827+0000 7f13f1a5c700 1 -- 172.21.15.16:0/482298099 learned_addr learned my addr 172.21.15.16:0/482298099 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:47:33.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.827+0000 7f13f125b700 1 --2- 172.21.15.16:0/482298099 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f13ec0cb880 0x7f13ec0cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:33.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.827+0000 7f13f0a5a700 1 -- 172.21.15.16:0/482298099 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2645076554 0 0) 0x7f13ec0cef70 con 0x7f13ec15b070 2023-10-16T13:47:33.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.827+0000 7f13f0a5a700 1 -- 172.21.15.16:0/482298099 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13d40057c0 con 0x7f13ec15b070 2023-10-16T13:47:33.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.827+0000 7f13f0a5a700 1 -- 172.21.15.16:0/482298099 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4217751876 0 0) 0x7f13ec0ce360 con 0x7f13ec06da60 2023-10-16T13:47:33.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.827+0000 7f13f0a5a700 1 -- 172.21.15.16:0/482298099 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13ec0cef70 con 0x7f13ec06da60 2023-10-16T13:47:33.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.827+0000 7f13f125b700 1 -- 172.21.15.16:0/482298099 >> v1:172.21.15.49:6789/0 conn(0x7f13ec06da60 legacy=0x7f13ec0cf400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:33.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.827+0000 7f13f125b700 1 -- 172.21.15.16:0/482298099 >> v1:172.21.15.16:6789/0 conn(0x7f13ec15b070 legacy=0x7f13ec15b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:33.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.827+0000 7f13f0a5a700 1 -- 172.21.15.16:0/482298099 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3639541890 0 0) 0x7f13d40057c0 con 0x7f13ec15b070 2023-10-16T13:47:33.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.827+0000 7f13f125b700 1 -- 172.21.15.16:0/482298099 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13ec0ce920 con 0x7f13ec0cb880 2023-10-16T13:47:33.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.827+0000 7f13f125b700 1 --2- 172.21.15.16:0/482298099 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f13ec0cb880 0x7f13ec0cbd00 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f13e0009870 tx=0x7f13e0009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba1ea4497aefe5d9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:33.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.828+0000 7f13f0a5a700 1 -- 172.21.15.16:0/482298099 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13e40019f0 con 0x7f13ec0cb880 2023-10-16T13:47:33.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.828+0000 7f13f0a5a700 1 -- 172.21.15.16:0/482298099 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13e0004030 con 0x7f13ec0cb880 2023-10-16T13:47:33.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.828+0000 7f13f0a5a700 1 -- 172.21.15.16:0/482298099 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13e00043a0 con 0x7f13ec0cb880 2023-10-16T13:47:33.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.828+0000 7f13f3cc0700 1 -- 172.21.15.16:0/482298099 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f13ec0cb880 msgr2=0x7f13ec0cbd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:33.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.828+0000 7f13f3cc0700 1 --2- 172.21.15.16:0/482298099 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f13ec0cb880 0x7f13ec0cbd00 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f13e0009870 tx=0x7f13e0009b80 comp rx=0 tx=0).stop 2023-10-16T13:47:33.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.828+0000 7f13f3cc0700 1 -- 172.21.15.16:0/482298099 shutdown_connections 2023-10-16T13:47:33.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.828+0000 7f13f3cc0700 1 --2- 172.21.15.16:0/482298099 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f13ec0cb880 0x7f13ec0cbd00 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:33.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.828+0000 7f13f3cc0700 1 -- 172.21.15.16:0/482298099 >> 172.21.15.16:0/482298099 conn(0x7f13ec0b1f40 msgr2=0x7f13ec0b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:33.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.828+0000 7f13f3cc0700 1 -- 172.21.15.16:0/482298099 shutdown_connections 2023-10-16T13:47:33.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.828+0000 7f13f3cc0700 1 -- 172.21.15.16:0/482298099 wait complete. 2023-10-16T13:47:33.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.829+0000 7f13f3cc0700 1 Processor -- start 2023-10-16T13:47:33.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.829+0000 7f13f3cc0700 1 -- start start 2023-10-16T13:47:33.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.830+0000 7f13f3cc0700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f13ec06da60 0x7f13ec1515e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:33.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.830+0000 7f13f3cc0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13ec0cb880 0x7f13ec15d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:33.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.830+0000 7f13f3cc0700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f13ec15b070 0x7f13ec0d05c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:33.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.830+0000 7f13f3cc0700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13ec1f2a40 con 0x7f13ec0cb880 2023-10-16T13:47:33.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.830+0000 7f13f3cc0700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13ec151b20 con 0x7f13ec06da60 2023-10-16T13:47:33.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.830+0000 7f13f3cc0700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13ec1f1ec0 con 0x7f13ec15b070 2023-10-16T13:47:33.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.830+0000 7f13f125b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13ec0cb880 0x7f13ec15d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:33.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.830+0000 7f13f125b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13ec0cb880 0x7f13ec15d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55572/0 (socket says 172.21.15.16:55572) 2023-10-16T13:47:33.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.830+0000 7f13f125b700 1 -- 172.21.15.16:0/3317925994 learned_addr learned my addr 172.21.15.16:0/3317925994 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:33.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.830+0000 7f13f1a5c700 1 --2- 172.21.15.16:0/3317925994 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f13ec06da60 0x7f13ec1515e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:33.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.831+0000 7f13f225d700 1 --2- 172.21.15.16:0/3317925994 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f13ec15b070 0x7f13ec0d05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:33.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.831+0000 7f13f125b700 1 -- 172.21.15.16:0/3317925994 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f13ec15b070 msgr2=0x7f13ec0d05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:33.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.831+0000 7f13f125b700 1 --2- 172.21.15.16:0/3317925994 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f13ec15b070 0x7f13ec0d05c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:33.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.831+0000 7f13f125b700 1 -- 172.21.15.16:0/3317925994 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f13ec06da60 msgr2=0x7f13ec1515e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:33.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.831+0000 7f13f125b700 1 --2- 172.21.15.16:0/3317925994 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f13ec06da60 0x7f13ec1515e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:33.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.831+0000 7f13f125b700 1 -- 172.21.15.16:0/3317925994 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13e00094a0 con 0x7f13ec0cb880 2023-10-16T13:47:33.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.831+0000 7f13f225d700 1 --2- 172.21.15.16:0/3317925994 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f13ec15b070 0x7f13ec0d05c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:47:33.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.831+0000 7f13f125b700 1 --2- 172.21.15.16:0/3317925994 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13ec0cb880 0x7f13ec15d3c0 secure :-1 s=READY pgs=2767 cs=0 l=1 rev1=1 crypto rx=0x7f13e00097c0 tx=0x7f13e0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:33.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.831+0000 7f13ea7fc700 1 -- 172.21.15.16:0/3317925994 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13e00045b0 con 0x7f13ec0cb880 2023-10-16T13:47:33.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.831+0000 7f13ea7fc700 1 -- 172.21.15.16:0/3317925994 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13e001f070 con 0x7f13ec0cb880 2023-10-16T13:47:33.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.831+0000 7f13f3cc0700 1 -- 172.21.15.16:0/3317925994 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13ec1f21a0 con 0x7f13ec0cb880 2023-10-16T13:47:33.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.831+0000 7f13ea7fc700 1 -- 172.21.15.16:0/3317925994 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13e0011870 con 0x7f13ec0cb880 2023-10-16T13:47:33.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.831+0000 7f13f3cc0700 1 -- 172.21.15.16:0/3317925994 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13ec1f2760 con 0x7f13ec0cb880 2023-10-16T13:47:33.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.832+0000 7f13f3cc0700 1 -- 172.21.15.16:0/3317925994 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13ec15b4e0 con 0x7f13ec0cb880 2023-10-16T13:47:33.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.832+0000 7f13ea7fc700 1 -- 172.21.15.16:0/3317925994 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 148) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f13e00119d0 con 0x7f13ec0cb880 2023-10-16T13:47:33.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.833+0000 7f13ea7fc700 1 --2- 172.21.15.16:0/3317925994 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f13d806b140 0x7f13d806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:33.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.833+0000 7f13ea7fc700 1 -- 172.21.15.16:0/3317925994 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(245..245 src has 1..245) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f13e0021070 con 0x7f13ec0cb880 2023-10-16T13:47:33.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.833+0000 7f13f1a5c700 1 --2- 172.21.15.16:0/3317925994 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f13d806b140 0x7f13d806d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:33.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.833+0000 7f13f1a5c700 1 --2- 172.21.15.16:0/3317925994 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f13d806b140 0x7f13d806d600 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f13e40070a0 tx=0x7f13e40017d0 comp rx=0 tx=0).ready entity=mgr.11886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:33.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:33.837+0000 7f13ea7fc700 1 -- 172.21.15.16:0/3317925994 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f13ec15b4e0 con 0x7f13ec0cb880 2023-10-16T13:47:34.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.123+0000 7f13f3cc0700 1 -- 172.21.15.16:0/3317925994 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f13ec15b4e0 con 0x7f13ec0cb880 2023-10-16T13:47:34.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.125+0000 7f13ea7fc700 1 -- 172.21.15.16:0/3317925994 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v57) v1 ==== 190+0+147 (secure 0 0 0) 0x7f13ec15b4e0 con 0x7f13ec0cb880 2023-10-16T13:47:34.127 INFO:teuthology.orchestra.run.smithi016.stdout:[client.0] 2023-10-16T13:47:34.127 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAPKC1l2l6PFxAA9QyXtBAHdGAXpwH0JfknuQ== 2023-10-16T13:47:34.128 INFO:teuthology.orchestra.run.smithi016.stdout: caps mds = "allow" 2023-10-16T13:47:34.128 INFO:teuthology.orchestra.run.smithi016.stdout: caps mon = "allow r" 2023-10-16T13:47:34.128 INFO:teuthology.orchestra.run.smithi016.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-16T13:47:34.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.128+0000 7f13f3cc0700 1 -- 172.21.15.16:0/3317925994 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f13d806b140 msgr2=0x7f13d806d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:34.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.128+0000 7f13f3cc0700 1 --2- 172.21.15.16:0/3317925994 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f13d806b140 0x7f13d806d600 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f13e40070a0 tx=0x7f13e40017d0 comp rx=0 tx=0).stop 2023-10-16T13:47:34.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.128+0000 7f13f3cc0700 1 -- 172.21.15.16:0/3317925994 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13ec0cb880 msgr2=0x7f13ec15d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:34.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.128+0000 7f13f3cc0700 1 --2- 172.21.15.16:0/3317925994 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13ec0cb880 0x7f13ec15d3c0 secure :-1 s=READY pgs=2767 cs=0 l=1 rev1=1 crypto rx=0x7f13e00097c0 tx=0x7f13e0004750 comp rx=0 tx=0).stop 2023-10-16T13:47:34.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.128+0000 7f13f3cc0700 1 -- 172.21.15.16:0/3317925994 shutdown_connections 2023-10-16T13:47:34.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.128+0000 7f13f3cc0700 1 --2- 172.21.15.16:0/3317925994 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f13d806b140 0x7f13d806d600 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:34.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.128+0000 7f13f3cc0700 1 --2- 172.21.15.16:0/3317925994 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f13ec15b070 0x7f13ec0d05c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:34.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.128+0000 7f13f3cc0700 1 --2- 172.21.15.16:0/3317925994 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f13ec06da60 0x7f13ec1515e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:34.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.128+0000 7f13f3cc0700 1 --2- 172.21.15.16:0/3317925994 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13ec0cb880 0x7f13ec15d3c0 unknown :-1 s=CLOSED pgs=2767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:34.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.128+0000 7f13f3cc0700 1 -- 172.21.15.16:0/3317925994 >> 172.21.15.16:0/3317925994 conn(0x7f13ec0b1f40 msgr2=0x7f13ec0ccf10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:34.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.128+0000 7f13f3cc0700 1 -- 172.21.15.16:0/3317925994 shutdown_connections 2023-10-16T13:47:34.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.128+0000 7f13f3cc0700 1 -- 172.21.15.16:0/3317925994 wait complete. 2023-10-16T13:47:34.133 INFO:teuthology.orchestra.run.smithi016.stderr:updated caps for client.0 2023-10-16T13:47:34.143 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:47:34.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.265+0000 7f4b678ca700 1 Processor -- start 2023-10-16T13:47:34.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.265+0000 7f4b678ca700 1 -- start start 2023-10-16T13:47:34.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.266+0000 7f4b678ca700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4b6006a7e0 0x7f4b6006ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:34.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.266+0000 7f4b678ca700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b60161c30 con 0x7f4b6015b280 2023-10-16T13:47:34.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.266+0000 7f4b678ca700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b60161f10 con 0x7f4b600cc8a0 2023-10-16T13:47:34.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.266+0000 7f4b678ca700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b6006b190 con 0x7f4b6006a7e0 2023-10-16T13:47:34.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.266+0000 7f4b65666700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f4b6015b280 0x7f4b601613e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44852/0 (socket says 172.21.15.16:44852) 2023-10-16T13:47:34.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.266+0000 7f4b65666700 1 -- 172.21.15.16:0/1137820790 learned_addr learned my addr 172.21.15.16:0/1137820790 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:47:34.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.266+0000 7f4b65e67700 1 --2- 172.21.15.16:0/1137820790 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4b6006a7e0 0x7f4b6006ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:34.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.266+0000 7f4b57fff700 1 -- 172.21.15.16:0/1137820790 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2873467076 0 0) 0x7f4b60161c30 con 0x7f4b6015b280 2023-10-16T13:47:34.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.266+0000 7f4b57fff700 1 -- 172.21.15.16:0/1137820790 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b4c0035b0 con 0x7f4b6015b280 2023-10-16T13:47:34.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.267+0000 7f4b57fff700 1 -- 172.21.15.16:0/1137820790 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 543781464 0 0) 0x7f4b60161f10 con 0x7f4b600cc8a0 2023-10-16T13:47:34.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.267+0000 7f4b57fff700 1 -- 172.21.15.16:0/1137820790 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b60161c30 con 0x7f4b600cc8a0 2023-10-16T13:47:34.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.267+0000 7f4b57fff700 1 -- 172.21.15.16:0/1137820790 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3552624844 0 0) 0x7f4b4c0035b0 con 0x7f4b6015b280 2023-10-16T13:47:34.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.267+0000 7f4b57fff700 1 -- 172.21.15.16:0/1137820790 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4b6006a7e0 msgr2=0x7f4b6006ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:34.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.267+0000 7f4b57fff700 1 --2- 172.21.15.16:0/1137820790 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4b6006a7e0 0x7f4b6006ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:34.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.267+0000 7f4b57fff700 1 -- 172.21.15.16:0/1137820790 >> v1:172.21.15.49:6789/0 conn(0x7f4b600cc8a0 legacy=0x7f4b6015e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:34.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.267+0000 7f4b57fff700 1 -- 172.21.15.16:0/1137820790 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b601621f0 con 0x7f4b6015b280 2023-10-16T13:47:34.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.267+0000 7f4b57fff700 1 -- 172.21.15.16:0/1137820790 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f4b50001670 con 0x7f4b6015b280 2023-10-16T13:47:34.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.267+0000 7f4b57fff700 1 -- 172.21.15.16:0/1137820790 >> v1:172.21.15.16:6789/0 conn(0x7f4b6015b280 legacy=0x7f4b601613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:34.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.267+0000 7f4b57fff700 1 --2- 172.21.15.16:0/1137820790 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4b4c003bc0 0x7f4b4c003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:34.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.267+0000 7f4b57fff700 1 --2- 172.21.15.16:0/1137820790 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4b6006a7e0 0x7f4b4c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:34.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.267+0000 7f4b57fff700 1 --2- 172.21.15.16:0/1137820790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b4c004d90 0x7f4b4c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:34.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.267+0000 7f4b57fff700 1 -- 172.21.15.16:0/1137820790 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b6006b190 con 0x7f4b4c004d90 2023-10-16T13:47:34.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.267+0000 7f4b57fff700 1 -- 172.21.15.16:0/1137820790 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b4c00bac0 con 0x7f4b4c003bc0 2023-10-16T13:47:34.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.267+0000 7f4b57fff700 1 -- 172.21.15.16:0/1137820790 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b4c00bc80 con 0x7f4b6006a7e0 2023-10-16T13:47:34.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.267+0000 7f4b65e67700 1 --2- 172.21.15.16:0/1137820790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b4c004d90 0x7f4b4c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:34.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.267+0000 7f4b65666700 1 --2- 172.21.15.16:0/1137820790 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4b4c003bc0 0x7f4b4c003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:34.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.267+0000 7f4b64e65700 1 --2- 172.21.15.16:0/1137820790 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4b6006a7e0 0x7f4b4c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:34.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.267+0000 7f4b65e67700 1 -- 172.21.15.16:0/1137820790 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4b6006a7e0 msgr2=0x7f4b4c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:34.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.267+0000 7f4b65e67700 1 --2- 172.21.15.16:0/1137820790 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4b6006a7e0 0x7f4b4c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:34.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.267+0000 7f4b65e67700 1 -- 172.21.15.16:0/1137820790 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4b4c003bc0 msgr2=0x7f4b4c003fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:34.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.267+0000 7f4b65e67700 1 --2- 172.21.15.16:0/1137820790 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4b4c003bc0 0x7f4b4c003fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:34.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.267+0000 7f4b65e67700 1 -- 172.21.15.16:0/1137820790 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b4c0035b0 con 0x7f4b4c004d90 2023-10-16T13:47:34.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.267+0000 7f4b65e67700 1 --2- 172.21.15.16:0/1137820790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b4c004d90 0x7f4b4c005210 secure :-1 s=READY pgs=2769 cs=0 l=1 rev1=1 crypto rx=0x7f4b58009fc0 tx=0x7f4b4c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d44cd0522cd197f5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:34.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.268+0000 7f4b57fff700 1 -- 172.21.15.16:0/1137820790 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b5800ec30 con 0x7f4b4c004d90 2023-10-16T13:47:34.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.268+0000 7f4b57fff700 1 -- 172.21.15.16:0/1137820790 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b5800f040 con 0x7f4b4c004d90 2023-10-16T13:47:34.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.268+0000 7f4b57fff700 1 -- 172.21.15.16:0/1137820790 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b58014480 con 0x7f4b4c004d90 2023-10-16T13:47:34.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.268+0000 7f4b678ca700 1 -- 172.21.15.16:0/1137820790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b4c004d90 msgr2=0x7f4b4c005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:34.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.268+0000 7f4b678ca700 1 --2- 172.21.15.16:0/1137820790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b4c004d90 0x7f4b4c005210 secure :-1 s=READY pgs=2769 cs=0 l=1 rev1=1 crypto rx=0x7f4b58009fc0 tx=0x7f4b4c005880 comp rx=0 tx=0).stop 2023-10-16T13:47:34.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.268+0000 7f4b65e67700 1 -- 172.21.15.16:0/1137820790 reap_dead start 2023-10-16T13:47:34.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.268+0000 7f4b678ca700 1 -- 172.21.15.16:0/1137820790 shutdown_connections 2023-10-16T13:47:34.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.268+0000 7f4b678ca700 1 -- 172.21.15.16:0/1137820790 >> 172.21.15.16:0/1137820790 conn(0x7f4b600b1fd0 msgr2=0x7f4b600b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:34.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.268+0000 7f4b678ca700 1 -- 172.21.15.16:0/1137820790 shutdown_connections 2023-10-16T13:47:34.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.268+0000 7f4b678ca700 1 -- 172.21.15.16:0/1137820790 wait complete. 2023-10-16T13:47:34.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.269+0000 7f4b678ca700 1 Processor -- start 2023-10-16T13:47:34.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.269+0000 7f4b678ca700 1 -- start start 2023-10-16T13:47:34.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.270+0000 7f4b678ca700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4b4c004d90 0x7f4b600c7f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:34.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.270+0000 7f4b678ca700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4b600c9810 0x7f4b60159c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:34.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.270+0000 7f4b678ca700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b6015c950 0x7f4b6015cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:34.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.270+0000 7f4b678ca700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b6015d310 con 0x7f4b6015c950 2023-10-16T13:47:34.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.270+0000 7f4b678ca700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b6015a160 con 0x7f4b4c004d90 2023-10-16T13:47:34.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.270+0000 7f4b678ca700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b600c5390 con 0x7f4b600c9810 2023-10-16T13:47:34.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.270+0000 7f4b65e67700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b6015c950 0x7f4b6015cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:34.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.270+0000 7f4b65e67700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b6015c950 0x7f4b6015cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55594/0 (socket says 172.21.15.16:55594) 2023-10-16T13:47:34.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.270+0000 7f4b65e67700 1 -- 172.21.15.16:0/3220561084 learned_addr learned my addr 172.21.15.16:0/3220561084 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:34.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.270+0000 7f4b65666700 1 --2- 172.21.15.16:0/3220561084 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4b4c004d90 0x7f4b600c7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:34.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.271+0000 7f4b64e65700 1 --2- 172.21.15.16:0/3220561084 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4b600c9810 0x7f4b60159c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:34.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.271+0000 7f4b65e67700 1 -- 172.21.15.16:0/3220561084 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4b600c9810 msgr2=0x7f4b60159c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:34.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.271+0000 7f4b65e67700 1 --2- 172.21.15.16:0/3220561084 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4b600c9810 0x7f4b60159c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:34.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.271+0000 7f4b65e67700 1 -- 172.21.15.16:0/3220561084 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4b4c004d90 msgr2=0x7f4b600c7f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:34.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.271+0000 7f4b65e67700 1 --2- 172.21.15.16:0/3220561084 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4b4c004d90 0x7f4b600c7f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:34.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.271+0000 7f4b65e67700 1 -- 172.21.15.16:0/3220561084 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b600c9d80 con 0x7f4b6015c950 2023-10-16T13:47:34.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.271+0000 7f4b65666700 1 --2- 172.21.15.16:0/3220561084 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4b4c004d90 0x7f4b600c7f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:47:34.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.271+0000 7f4b65e67700 1 --2- 172.21.15.16:0/3220561084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b6015c950 0x7f4b6015cdd0 secure :-1 s=READY pgs=2770 cs=0 l=1 rev1=1 crypto rx=0x7f4b5800e700 tx=0x7f4b58009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:34.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.271+0000 7f4b55ffb700 1 -- 172.21.15.16:0/3220561084 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b58007000 con 0x7f4b6015c950 2023-10-16T13:47:34.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.271+0000 7f4b55ffb700 1 -- 172.21.15.16:0/3220561084 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b58015d80 con 0x7f4b6015c950 2023-10-16T13:47:34.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.271+0000 7f4b678ca700 1 -- 172.21.15.16:0/3220561084 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b600ca060 con 0x7f4b6015c950 2023-10-16T13:47:34.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.271+0000 7f4b55ffb700 1 -- 172.21.15.16:0/3220561084 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b580276b0 con 0x7f4b6015c950 2023-10-16T13:47:34.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.271+0000 7f4b678ca700 1 -- 172.21.15.16:0/3220561084 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b600ca340 con 0x7f4b6015c950 2023-10-16T13:47:34.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.272+0000 7f4b678ca700 1 -- 172.21.15.16:0/3220561084 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b6004ebb0 con 0x7f4b6015c950 2023-10-16T13:47:34.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.273+0000 7f4b55ffb700 1 -- 172.21.15.16:0/3220561084 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 148) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f4b58024070 con 0x7f4b6015c950 2023-10-16T13:47:34.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.273+0000 7f4b55ffb700 1 --2- 172.21.15.16:0/3220561084 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f4b4406b140 0x7f4b4406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:34.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.273+0000 7f4b55ffb700 1 -- 172.21.15.16:0/3220561084 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(245..245 src has 1..245) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f4b5808b860 con 0x7f4b6015c950 2023-10-16T13:47:34.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.273+0000 7f4b65666700 1 --2- 172.21.15.16:0/3220561084 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f4b4406b140 0x7f4b4406d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:34.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.274+0000 7f4b65666700 1 --2- 172.21.15.16:0/3220561084 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f4b4406b140 0x7f4b4406d600 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4b50001eb0 tx=0x7f4b50009040 comp rx=0 tx=0).ready entity=mgr.11886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:34.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.277+0000 7f4b55ffb700 1 -- 172.21.15.16:0/3220561084 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4b580565a0 con 0x7f4b6015c950 2023-10-16T13:47:34.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.527+0000 7f4b678ca700 1 -- 172.21.15.16:0/3220561084 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4b600c54d0 con 0x7f4b6015c950 2023-10-16T13:47:34.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.528+0000 7f4b55ffb700 1 -- 172.21.15.16:0/3220561084 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 130 v130) v1 ==== 95+0+4922 (secure 0 0 0) 0x7f4b5802c090 con 0x7f4b6015c950 2023-10-16T13:47:34.531 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:47:34.531 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":130,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9596,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/3939647668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":3939647668},{"type":"v1","addr":"172.21.15.49:6834","nonce":3939647668}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":125},{"gid":11185,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/2890113513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2890113513},{"type":"v1","addr":"172.21.15.16:6836","nonce":2890113513}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":125},{"gid":11197,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/680748068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":680748068},{"type":"v1","addr":"172.21.15.49:6835","nonce":680748068}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":125}],"filesystems":[{"mdsmap":{"epoch":130,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:47:23.843333+0000","modified":"2023-10-16T13:47:26.851597+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":12252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12252":{"gid":12252,"name":"a","rank":0,"incarnation":126,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/303278795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":303278795},{"type":"v1","addr":"172.21.15.16:6837","nonce":303278795}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15}]} 2023-10-16T13:47:34.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.530+0000 7f4b678ca700 1 -- 172.21.15.16:0/3220561084 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f4b4406b140 msgr2=0x7f4b4406d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:34.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.530+0000 7f4b678ca700 1 --2- 172.21.15.16:0/3220561084 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f4b4406b140 0x7f4b4406d600 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4b50001eb0 tx=0x7f4b50009040 comp rx=0 tx=0).stop 2023-10-16T13:47:34.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.530+0000 7f4b678ca700 1 -- 172.21.15.16:0/3220561084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b6015c950 msgr2=0x7f4b6015cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:34.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.530+0000 7f4b678ca700 1 --2- 172.21.15.16:0/3220561084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b6015c950 0x7f4b6015cdd0 secure :-1 s=READY pgs=2770 cs=0 l=1 rev1=1 crypto rx=0x7f4b5800e700 tx=0x7f4b58009750 comp rx=0 tx=0).stop 2023-10-16T13:47:34.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.530+0000 7f4b678ca700 1 -- 172.21.15.16:0/3220561084 shutdown_connections 2023-10-16T13:47:34.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.530+0000 7f4b678ca700 1 --2- 172.21.15.16:0/3220561084 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f4b4406b140 0x7f4b4406d600 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:34.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.530+0000 7f4b678ca700 1 --2- 172.21.15.16:0/3220561084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b6015c950 0x7f4b6015cdd0 unknown :-1 s=CLOSED pgs=2770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:34.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.530+0000 7f4b678ca700 1 --2- 172.21.15.16:0/3220561084 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4b4c004d90 0x7f4b600c7f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:34.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.530+0000 7f4b678ca700 1 --2- 172.21.15.16:0/3220561084 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4b600c9810 0x7f4b60159c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:34.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.530+0000 7f4b678ca700 1 -- 172.21.15.16:0/3220561084 >> 172.21.15.16:0/3220561084 conn(0x7f4b600b1fd0 msgr2=0x7f4b600cb2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:34.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.530+0000 7f4b678ca700 1 -- 172.21.15.16:0/3220561084 shutdown_connections 2023-10-16T13:47:34.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:34.530+0000 7f4b678ca700 1 -- 172.21.15.16:0/3220561084 wait complete. 2023-10-16T13:47:34.536 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 130 2023-10-16T13:47:34.947 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 130, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T13:47:23.843333+0000', 'modified': '2023-10-16T13:47:26.851597+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 12252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_12252': {'gid': 12252, 'name': 'a', 'rank': 0, 'incarnation': 126, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.16:6837/303278795', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6835', 'nonce': 303278795}, {'type': 'v1', 'addr': '172.21.15.16:6837', 'nonce': 303278795}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [32], 'metadata_pool': 31, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-16T13:47:34.947 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-16T13:47:34.948 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-10-16T13:47:35.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.078+0000 7f29bf7a7700 1 Processor -- start 2023-10-16T13:47:35.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.078+0000 7f29bf7a7700 1 -- start start 2023-10-16T13:47:35.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.079+0000 7f29bf7a7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f29b80ce4f0 0x7f29b80d0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:35.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.079+0000 7f29bf7a7700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f29b815ba10 0x7f29b80cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:35.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.079+0000 7f29bf7a7700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29b80b0970 con 0x7f29b815b630 2023-10-16T13:47:35.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.079+0000 7f29bf7a7700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29b80cd690 con 0x7f29b80ce4f0 2023-10-16T13:47:35.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.079+0000 7f29bf7a7700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29b80cd970 con 0x7f29b815ba10 2023-10-16T13:47:35.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.079+0000 7f29bcd42700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f29b80ce4f0 0x7f29b80d0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:35.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.079+0000 7f29bcd42700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f29b80ce4f0 0x7f29b80d0380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55600/0 (socket says 172.21.15.16:55600) 2023-10-16T13:47:35.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.079+0000 7f29bcd42700 1 -- 172.21.15.16:0/2172271230 learned_addr learned my addr 172.21.15.16:0/2172271230 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:35.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.079+0000 7f29bdd44700 1 --2- 172.21.15.16:0/2172271230 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f29b815ba10 0x7f29b80cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:35.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.079+0000 7f29bcd42700 1 -- 172.21.15.16:0/2172271230 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f29b815ba10 msgr2=0x7f29b80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:35.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.079+0000 7f29bcd42700 1 --2- 172.21.15.16:0/2172271230 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f29b815ba10 0x7f29b80cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:35.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.079+0000 7f29bcd42700 1 -- 172.21.15.16:0/2172271230 >> v1:172.21.15.49:6789/0 conn(0x7f29b815b630 legacy=0x7f29b80cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:35.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.079+0000 7f29bcd42700 1 -- 172.21.15.16:0/2172271230 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29b80b0c30 con 0x7f29b80ce4f0 2023-10-16T13:47:35.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.079+0000 7f29bcd42700 1 --2- 172.21.15.16:0/2172271230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29b80ce4f0 0x7f29b80d0380 secure :-1 s=READY pgs=2771 cs=0 l=1 rev1=1 crypto rx=0x7f29a80098f0 tx=0x7f29a8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=503fc9ebd6c57b22 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:35.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.080+0000 7f29affff700 1 -- 172.21.15.16:0/2172271230 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29a800b8a0 con 0x7f29b80ce4f0 2023-10-16T13:47:35.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.080+0000 7f29affff700 1 -- 172.21.15.16:0/2172271230 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29a800ba00 con 0x7f29b80ce4f0 2023-10-16T13:47:35.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.080+0000 7f29affff700 1 -- 172.21.15.16:0/2172271230 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29a800bd90 con 0x7f29b80ce4f0 2023-10-16T13:47:35.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.080+0000 7f29bf7a7700 1 -- 172.21.15.16:0/2172271230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29b80ce4f0 msgr2=0x7f29b80d0380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:35.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.080+0000 7f29bf7a7700 1 --2- 172.21.15.16:0/2172271230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29b80ce4f0 0x7f29b80d0380 secure :-1 s=READY pgs=2771 cs=0 l=1 rev1=1 crypto rx=0x7f29a80098f0 tx=0x7f29a8009c00 comp rx=0 tx=0).stop 2023-10-16T13:47:35.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.080+0000 7f29bf7a7700 1 -- 172.21.15.16:0/2172271230 shutdown_connections 2023-10-16T13:47:35.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.080+0000 7f29bf7a7700 1 --2- 172.21.15.16:0/2172271230 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f29b815ba10 0x7f29b80cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:35.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.080+0000 7f29bf7a7700 1 --2- 172.21.15.16:0/2172271230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29b80ce4f0 0x7f29b80d0380 unknown :-1 s=CLOSED pgs=2771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:35.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.080+0000 7f29bf7a7700 1 -- 172.21.15.16:0/2172271230 >> 172.21.15.16:0/2172271230 conn(0x7f29b80d14b0 msgr2=0x7f29b80d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:35.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.080+0000 7f29bf7a7700 1 -- 172.21.15.16:0/2172271230 shutdown_connections 2023-10-16T13:47:35.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.080+0000 7f29bf7a7700 1 -- 172.21.15.16:0/2172271230 wait complete. 2023-10-16T13:47:35.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.081+0000 7f29bf7a7700 1 Processor -- start 2023-10-16T13:47:35.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.081+0000 7f29bf7a7700 1 -- start start 2023-10-16T13:47:35.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.082+0000 7f29bf7a7700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f29b80ce4f0 0x7f29b80c85f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:35.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.082+0000 7f29bf7a7700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f29b815b630 0x7f29b814e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:35.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.082+0000 7f29bf7a7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29b815ba10 0x7f29b8151cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:35.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.082+0000 7f29bf7a7700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29b806c300 con 0x7f29b815ba10 2023-10-16T13:47:35.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.082+0000 7f29bf7a7700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29b806d2a0 con 0x7f29b80ce4f0 2023-10-16T13:47:35.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.082+0000 7f29bf7a7700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29b80c8b30 con 0x7f29b815b630 2023-10-16T13:47:35.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.082+0000 7f29bdd44700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29b815ba10 0x7f29b8151cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:35.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.082+0000 7f29bdd44700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29b815ba10 0x7f29b8151cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55608/0 (socket says 172.21.15.16:55608) 2023-10-16T13:47:35.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.082+0000 7f29bdd44700 1 -- 172.21.15.16:0/2325935048 learned_addr learned my addr 172.21.15.16:0/2325935048 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:35.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.082+0000 7f29bd543700 1 --2- 172.21.15.16:0/2325935048 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f29b80ce4f0 0x7f29b80c85f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:35.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.082+0000 7f29bcd42700 1 --2- 172.21.15.16:0/2325935048 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f29b815b630 0x7f29b814e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:35.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.083+0000 7f29bdd44700 1 -- 172.21.15.16:0/2325935048 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f29b815b630 msgr2=0x7f29b814e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:35.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.083+0000 7f29bdd44700 1 --2- 172.21.15.16:0/2325935048 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f29b815b630 0x7f29b814e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:35.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.083+0000 7f29bdd44700 1 -- 172.21.15.16:0/2325935048 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f29b80ce4f0 msgr2=0x7f29b80c85f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:35.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.083+0000 7f29bdd44700 1 --2- 172.21.15.16:0/2325935048 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f29b80ce4f0 0x7f29b80c85f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:35.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.083+0000 7f29bdd44700 1 -- 172.21.15.16:0/2325935048 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29a8009550 con 0x7f29b815ba10 2023-10-16T13:47:35.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.083+0000 7f29bcd42700 1 --2- 172.21.15.16:0/2325935048 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f29b815b630 0x7f29b814e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:47:35.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.083+0000 7f29bdd44700 1 --2- 172.21.15.16:0/2325935048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29b815ba10 0x7f29b8151cc0 secure :-1 s=READY pgs=2772 cs=0 l=1 rev1=1 crypto rx=0x7f29b0007e60 tx=0x7f29b000e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:35.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.083+0000 7f29adffb700 1 -- 172.21.15.16:0/2325935048 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29b000ed80 con 0x7f29b815ba10 2023-10-16T13:47:35.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.083+0000 7f29adffb700 1 -- 172.21.15.16:0/2325935048 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29b0014420 con 0x7f29b815ba10 2023-10-16T13:47:35.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.083+0000 7f29adffb700 1 -- 172.21.15.16:0/2325935048 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29b00147d0 con 0x7f29b815ba10 2023-10-16T13:47:35.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.083+0000 7f29bf7a7700 1 -- 172.21.15.16:0/2325935048 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29b806c7c0 con 0x7f29b815ba10 2023-10-16T13:47:35.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.083+0000 7f29bf7a7700 1 -- 172.21.15.16:0/2325935048 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29b806cd80 con 0x7f29b815ba10 2023-10-16T13:47:35.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.084+0000 7f29adffb700 1 -- 172.21.15.16:0/2325935048 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 148) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f29b0014930 con 0x7f29b815ba10 2023-10-16T13:47:35.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.085+0000 7f29adffb700 1 --2- 172.21.15.16:0/2325935048 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f299c06b140 0x7f299c06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:35.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.085+0000 7f29adffb700 1 -- 172.21.15.16:0/2325935048 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(245..245 src has 1..245) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f29b008a1b0 con 0x7f29b815ba10 2023-10-16T13:47:35.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.085+0000 7f29bd543700 1 --2- 172.21.15.16:0/2325935048 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f299c06b140 0x7f299c06d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:35.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.085+0000 7f29bd543700 1 --2- 172.21.15.16:0/2325935048 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f299c06b140 0x7f299c06d600 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f29b4007fd0 tx=0x7f29b4009040 comp rx=0 tx=0).ready entity=mgr.11886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:35.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.087+0000 7f29cb991b80 1 Processor -- start 2023-10-16T13:47:35.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.087+0000 7f29cb991b80 1 -- start start 2023-10-16T13:47:35.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.088+0000 7f29cb991b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd441b5220 0x55bd441bca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:35.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.088+0000 7f29cb991b80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55bd441ba630 0x55bd441bcfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:35.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.088+0000 7f29cb991b80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55bd44190010 0x55bd441906c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:35.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.088+0000 7f29cb991b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd44106c20 con 0x55bd441b5220 2023-10-16T13:47:35.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.088+0000 7f29cb991b80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd440ebc10 con 0x55bd44190010 2023-10-16T13:47:35.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.088+0000 7f29cb991b80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bd440e9530 con 0x55bd441ba630 2023-10-16T13:47:35.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.088+0000 7f29bd543700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd441b5220 0x55bd441bca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:35.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.088+0000 7f29bd543700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd441b5220 0x55bd441bca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55612/0 (socket says 172.21.15.16:55612) 2023-10-16T13:47:35.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.088+0000 7f29bd543700 1 -- 172.21.15.16:0/2593565014 learned_addr learned my addr 172.21.15.16:0/2593565014 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:35.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.088+0000 7f29bdd44700 1 --2- 172.21.15.16:0/2593565014 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55bd44190010 0x55bd441906c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:35.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.088+0000 7f29bcd42700 1 --2- 172.21.15.16:0/2593565014 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55bd441ba630 0x55bd441bcfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:35.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.088+0000 7f29bd543700 1 -- 172.21.15.16:0/2593565014 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55bd441ba630 msgr2=0x55bd441bcfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:35.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.088+0000 7f29bd543700 1 --2- 172.21.15.16:0/2593565014 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55bd441ba630 0x55bd441bcfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:35.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.088+0000 7f29bd543700 1 -- 172.21.15.16:0/2593565014 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55bd44190010 msgr2=0x55bd441906c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:35.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.088+0000 7f29bd543700 1 --2- 172.21.15.16:0/2593565014 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55bd44190010 0x55bd441906c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:35.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.088+0000 7f29bd543700 1 -- 172.21.15.16:0/2593565014 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bd440a23a0 con 0x55bd441b5220 2023-10-16T13:47:35.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.088+0000 7f29bd543700 1 --2- 172.21.15.16:0/2593565014 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd441b5220 0x55bd441bca70 secure :-1 s=READY pgs=2773 cs=0 l=1 rev1=1 crypto rx=0x7f29b400f7c0 tx=0x7f29b4012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfea1dc1e6ea45d6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:35.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.088+0000 7f299affd700 1 -- 172.21.15.16:0/2593565014 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29b4014450 con 0x55bd441b5220 2023-10-16T13:47:35.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.088+0000 7f299affd700 1 -- 172.21.15.16:0/2593565014 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29b40145b0 con 0x55bd441b5220 2023-10-16T13:47:35.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.088+0000 7f299affd700 1 -- 172.21.15.16:0/2593565014 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29b4014980 con 0x55bd441b5220 2023-10-16T13:47:35.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.089+0000 7f29cb991b80 1 -- 172.21.15.16:0/2593565014 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd441b5220 msgr2=0x55bd441bca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:35.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.089+0000 7f29cb991b80 1 --2- 172.21.15.16:0/2593565014 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd441b5220 0x55bd441bca70 secure :-1 s=READY pgs=2773 cs=0 l=1 rev1=1 crypto rx=0x7f29b400f7c0 tx=0x7f29b4012000 comp rx=0 tx=0).stop 2023-10-16T13:47:35.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.089+0000 7f29cb991b80 1 -- 172.21.15.16:0/2593565014 shutdown_connections 2023-10-16T13:47:35.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.089+0000 7f29cb991b80 1 --2- 172.21.15.16:0/2593565014 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55bd44190010 0x55bd441906c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:35.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.089+0000 7f29cb991b80 1 --2- 172.21.15.16:0/2593565014 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd441b5220 0x55bd441bca70 unknown :-1 s=CLOSED pgs=2773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:35.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.089+0000 7f29cb991b80 1 --2- 172.21.15.16:0/2593565014 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55bd441ba630 0x55bd441bcfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:35.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.089+0000 7f29cb991b80 1 -- 172.21.15.16:0/2593565014 >> 172.21.15.16:0/2593565014 conn(0x55bd440bfd80 msgr2=0x55bd441daff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:35.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.089+0000 7f29cb991b80 1 -- 172.21.15.16:0/2593565014 shutdown_connections 2023-10-16T13:47:35.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.089+0000 7f29cb991b80 1 -- 172.21.15.16:0/2593565014 wait complete. 2023-10-16T13:47:35.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.090+0000 7f29cb991b80 1 -- start start 2023-10-16T13:47:35.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.090+0000 7f29cb991b80 1 Processor -- start 2023-10-16T13:47:35.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.091+0000 7f29cb991b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd44190010 0x55bd441d7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:35.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.091+0000 7f29cb991b80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55bd441b5220 0x55bd441919f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:35.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.091+0000 7f29cb991b80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55bd441ba630 0x55bd44191f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:35.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.091+0000 7f29cb991b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd440e4e20 con 0x55bd44190010 2023-10-16T13:47:35.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.091+0000 7f29cb991b80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd440816c0 con 0x55bd441ba630 2023-10-16T13:47:35.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.091+0000 7f29cb991b80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bd44081360 con 0x55bd441b5220 2023-10-16T13:47:35.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.091+0000 7f29cb991b80 10 client.0 _ll_register_callbacks cb 0x55bd43ede7b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T13:47:35.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.091+0000 7f29bd543700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd44190010 0x55bd441d7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:35.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.091+0000 7f29bd543700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd44190010 0x55bd441d7aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55618/0 (socket says 172.21.15.16:55618) 2023-10-16T13:47:35.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.091+0000 7f29bd543700 1 -- 172.21.15.16:0/3204259309 learned_addr learned my addr 172.21.15.16:0/3204259309 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:35.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.091+0000 7f29bcd42700 1 --2- 172.21.15.16:0/3204259309 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55bd441b5220 0x55bd441919f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:35.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.091+0000 7f29bdd44700 1 --2- 172.21.15.16:0/3204259309 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55bd441ba630 0x55bd44191f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:35.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.091+0000 7f29bd543700 1 -- 172.21.15.16:0/3204259309 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55bd441b5220 msgr2=0x55bd441919f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:35.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.091+0000 7f29bd543700 1 --2- 172.21.15.16:0/3204259309 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55bd441b5220 0x55bd441919f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:35.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.091+0000 7f29bd543700 1 -- 172.21.15.16:0/3204259309 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55bd441ba630 msgr2=0x55bd44191f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:35.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.091+0000 7f29bd543700 1 --2- 172.21.15.16:0/3204259309 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55bd441ba630 0x55bd44191f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:35.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.091+0000 7f29bd543700 1 -- 172.21.15.16:0/3204259309 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29b80b0970 con 0x55bd44190010 2023-10-16T13:47:35.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.091+0000 7f29bd543700 1 --2- 172.21.15.16:0/3204259309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd44190010 0x55bd441d7aa0 secure :-1 s=READY pgs=2774 cs=0 l=1 rev1=1 crypto rx=0x7f29b4023040 tx=0x7f29b4001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f7ddf8663c5c884 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:35.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.091+0000 7f2993fff700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2023-10-16T13:47:35.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.091+0000 7f2993fff700 1 -- 172.21.15.16:0/3204259309 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29b4014450 con 0x55bd44190010 2023-10-16T13:47:35.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.092+0000 7f2993fff700 1 -- 172.21.15.16:0/3204259309 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(245..245 src has 1..245) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f29b401c040 con 0x55bd44190010 2023-10-16T13:47:35.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.092+0000 7f29cb991b80 1 -- 172.21.15.16:0/3204259309 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55bd441855b0 con 0x55bd44190010 2023-10-16T13:47:35.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.092+0000 7f2993fff700 1 -- 172.21.15.16:0/3204259309 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=130) v2 ==== 24+0+0 (secure 0 0 0) 0x7f29b4001d80 con 0x55bd44190010 2023-10-16T13:47:35.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.092+0000 7f29cb991b80 10 client.12429 fetch_fsmap learned FSMap version 130 2023-10-16T13:47:35.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.092+0000 7f29cb991b80 1 -- 172.21.15.16:0/3204259309 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=130,monmap=2+}) v3 -- 0x55bd43f0fd90 con 0x55bd44190010 2023-10-16T13:47:35.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.092+0000 7f2993fff700 1 -- 172.21.15.16:0/3204259309 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55bd441855b0 con 0x55bd44190010 2023-10-16T13:47:35.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.092+0000 7f2993fff700 1 -- 172.21.15.16:0/3204259309 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 130) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f29b4006a60 con 0x55bd44190010 2023-10-16T13:47:35.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.092+0000 7f29cb991b80 10 client.12429 fetch_fsmap finished waiting for FSMap version 130 2023-10-16T13:47:35.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.092+0000 7f29cb991b80 10 client.12429 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-16T13:47:35.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.092+0000 7f29cb991b80 20 client.12429 populate_metadata read hostname 'smithi016' 2023-10-16T13:47:35.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.092+0000 7f29cb991b80 1 --2- 172.21.15.16:0/3204259309 >> [v2:172.21.15.16:6835/303278795,v1:172.21.15.16:6837/303278795] conn(0x55bd441b8e80 0x55bd441b9260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:35.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.092+0000 7f29cb991b80 4 client.12429 mds_command: new command op to 12252 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T13:47:35.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.092+0000 7f29cb991b80 1 -- 172.21.15.16:0/3204259309 --> [v2:172.21.15.16:6835/303278795,v1:172.21.15.16:6837/303278795] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bd440dde00 con 0x55bd441b8e80 2023-10-16T13:47:35.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.092+0000 7f29bcd42700 1 --2- 172.21.15.16:0/3204259309 >> [v2:172.21.15.16:6835/303278795,v1:172.21.15.16:6837/303278795] conn(0x55bd441b8e80 0x55bd441b9260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:35.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.092+0000 7f2993fff700 10 client.12429 ms_handle_connect on v2:172.21.15.16:6835/303278795 2023-10-16T13:47:35.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.092+0000 7f29bcd42700 1 --2- 172.21.15.16:0/3204259309 >> [v2:172.21.15.16:6835/303278795,v1:172.21.15.16:6837/303278795] conn(0x55bd441b8e80 0x55bd441b9260 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6bd196bb6ab93b8a server_cookie=6ceaa8470ff2b17d in_seq=0 out_seq=0 2023-10-16T13:47:35.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.093+0000 7f2993fff700 1 -- 172.21.15.16:0/3204259309 <== mds.0 v2:172.21.15.16:6835/303278795 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55bd440dde00 con 0x55bd441b8e80 2023-10-16T13:47:35.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.093+0000 7f2993fff700 10 client.12429 handle_command_reply: tid=0 2023-10-16T13:47:35.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.093+0000 7f29cb991b80 1 client.12429 shutdown 2023-10-16T13:47:35.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.093+0000 7f29cb991b80 1 -- 172.21.15.16:0/3204259309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd44190010 msgr2=0x55bd441d7aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:35.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.093+0000 7f29cb991b80 1 --2- 172.21.15.16:0/3204259309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd44190010 0x55bd441d7aa0 secure :-1 s=READY pgs=2774 cs=0 l=1 rev1=1 crypto rx=0x7f29b4023040 tx=0x7f29b4001120 comp rx=0 tx=0).stop 2023-10-16T13:47:35.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.093+0000 7f29cb991b80 1 -- 172.21.15.16:0/3204259309 shutdown_connections 2023-10-16T13:47:35.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.093+0000 7f29cb991b80 1 --2- 172.21.15.16:0/3204259309 >> [v2:172.21.15.16:6835/303278795,v1:172.21.15.16:6837/303278795] conn(0x55bd441b8e80 0x55bd441b9260 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:35.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.093+0000 7f29cb991b80 1 --2- 172.21.15.16:0/3204259309 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55bd441ba630 0x55bd44191f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:35.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.093+0000 7f2993fff700 0 client.12429 ms_handle_reset on v2:172.21.15.16:6835/303278795 2023-10-16T13:47:35.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.093+0000 7f29cb991b80 1 --2- 172.21.15.16:0/3204259309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd44190010 0x55bd441d7aa0 unknown :-1 s=CLOSED pgs=2774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:35.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.093+0000 7f29cb991b80 1 --2- 172.21.15.16:0/3204259309 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55bd441b5220 0x55bd441919f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:35.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.093+0000 7f29cb991b80 1 -- 172.21.15.16:0/3204259309 >> 172.21.15.16:0/3204259309 conn(0x55bd440bfd80 msgr2=0x55bd441d85b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:35.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.093+0000 7f29cb991b80 1 -- 172.21.15.16:0/3204259309 shutdown_connections 2023-10-16T13:47:35.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.093+0000 7f29cb991b80 1 -- 172.21.15.16:0/3204259309 wait complete. 2023-10-16T13:47:35.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.094+0000 7f29cb991b80 20 client.12429 trim_cache size 0 max 16384 2023-10-16T13:47:35.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.112+0000 7f29cb991b80 1 Processor -- start 2023-10-16T13:47:35.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.112+0000 7f29cb991b80 1 -- start start 2023-10-16T13:47:35.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.112+0000 7f29cb991b80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55bd44190010 0x55bd441b7230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:35.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.112+0000 7f29cb991b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd441b5220 0x55bd441d1420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:35.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.112+0000 7f29cb991b80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55bd441ba630 0x55bd441d1960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:35.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.112+0000 7f29cb991b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd43f658d0 con 0x55bd441b5220 2023-10-16T13:47:35.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.112+0000 7f29cb991b80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd4418dbe0 con 0x55bd44190010 2023-10-16T13:47:35.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.112+0000 7f29cb991b80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bd4418ef60 con 0x55bd441ba630 2023-10-16T13:47:35.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.112+0000 7f29bcd42700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd441b5220 0x55bd441d1420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:35.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.112+0000 7f29bcd42700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd441b5220 0x55bd441d1420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55628/0 (socket says 172.21.15.16:55628) 2023-10-16T13:47:35.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.112+0000 7f29bcd42700 1 -- 172.21.15.16:0/3330657475 learned_addr learned my addr 172.21.15.16:0/3330657475 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:35.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.113+0000 7f29bd543700 1 --2- 172.21.15.16:0/3330657475 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55bd44190010 0x55bd441b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:35.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.113+0000 7f29bcd42700 1 -- 172.21.15.16:0/3330657475 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55bd441ba630 msgr2=0x55bd441d1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:35.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.113+0000 7f29bdd44700 1 --2- 172.21.15.16:0/3330657475 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55bd441ba630 0x55bd441d1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:35.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.113+0000 7f29bcd42700 1 --2- 172.21.15.16:0/3330657475 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55bd441ba630 0x55bd441d1960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:35.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.113+0000 7f29bcd42700 1 -- 172.21.15.16:0/3330657475 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55bd44190010 msgr2=0x55bd441b7230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:35.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.113+0000 7f29bcd42700 1 --2- 172.21.15.16:0/3330657475 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55bd44190010 0x55bd441b7230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:35.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.113+0000 7f29bcd42700 1 -- 172.21.15.16:0/3330657475 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bd43ef8c10 con 0x55bd441b5220 2023-10-16T13:47:35.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.113+0000 7f29bd543700 1 --2- 172.21.15.16:0/3330657475 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55bd44190010 0x55bd441b7230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:47:35.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.113+0000 7f29bdd44700 1 --2- 172.21.15.16:0/3330657475 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55bd441ba630 0x55bd441d1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:47:35.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.113+0000 7f29bcd42700 1 --2- 172.21.15.16:0/3330657475 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd441b5220 0x55bd441d1420 secure :-1 s=READY pgs=2775 cs=0 l=1 rev1=1 crypto rx=0x7f29a8000c00 tx=0x7f29a8010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e2b2f8151cd6db8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:35.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.113+0000 7f299b7fe700 1 -- 172.21.15.16:0/3330657475 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29a8021030 con 0x55bd441b5220 2023-10-16T13:47:35.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.113+0000 7f299b7fe700 1 -- 172.21.15.16:0/3330657475 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29a8022040 con 0x55bd441b5220 2023-10-16T13:47:35.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.113+0000 7f299b7fe700 1 -- 172.21.15.16:0/3330657475 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29a8028980 con 0x55bd441b5220 2023-10-16T13:47:35.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.114+0000 7f29cb991b80 1 -- 172.21.15.16:0/3330657475 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd441b5220 msgr2=0x55bd441d1420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:35.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.114+0000 7f29cb991b80 1 --2- 172.21.15.16:0/3330657475 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd441b5220 0x55bd441d1420 secure :-1 s=READY pgs=2775 cs=0 l=1 rev1=1 crypto rx=0x7f29a8000c00 tx=0x7f29a8010000 comp rx=0 tx=0).stop 2023-10-16T13:47:35.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.114+0000 7f29cb991b80 1 -- 172.21.15.16:0/3330657475 shutdown_connections 2023-10-16T13:47:35.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.114+0000 7f29cb991b80 1 --2- 172.21.15.16:0/3330657475 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55bd441ba630 0x55bd441d1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:35.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.114+0000 7f29cb991b80 1 --2- 172.21.15.16:0/3330657475 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55bd44190010 0x55bd441b7230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:35.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.114+0000 7f29cb991b80 1 --2- 172.21.15.16:0/3330657475 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd441b5220 0x55bd441d1420 unknown :-1 s=CLOSED pgs=2775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:35.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.114+0000 7f29cb991b80 1 -- 172.21.15.16:0/3330657475 >> 172.21.15.16:0/3330657475 conn(0x55bd440bfd80 msgr2=0x55bd4418ddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:35.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.114+0000 7f29cb991b80 1 -- 172.21.15.16:0/3330657475 shutdown_connections 2023-10-16T13:47:35.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.114+0000 7f29cb991b80 1 -- 172.21.15.16:0/3330657475 wait complete. 2023-10-16T13:47:35.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.114+0000 7f29cb991b80 1 -- start start 2023-10-16T13:47:35.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.114+0000 7f29cb991b80 1 Processor -- start 2023-10-16T13:47:35.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.115+0000 7f29cb991b80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55bd44190010 0x55bd4423e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:35.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.115+0000 7f29cb991b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd441b5220 0x55bd441bf6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:35.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.115+0000 7f29cb991b80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55bd441ba630 0x55bd441bfbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:35.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.115+0000 7f29cb991b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd441d4c40 con 0x55bd441b5220 2023-10-16T13:47:35.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.115+0000 7f29cb991b80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd441b7750 con 0x55bd44190010 2023-10-16T13:47:35.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.115+0000 7f29cb991b80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bd441b79a0 con 0x55bd441ba630 2023-10-16T13:47:35.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.115+0000 7f29cb991b80 10 client.0 _ll_register_callbacks cb 0x7f29b400f690 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T13:47:35.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.115+0000 7f29bcd42700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd441b5220 0x55bd441bf6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:35.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.115+0000 7f29bcd42700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd441b5220 0x55bd441bf6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55634/0 (socket says 172.21.15.16:55634) 2023-10-16T13:47:35.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.115+0000 7f29bcd42700 1 -- 172.21.15.16:0/444555503 learned_addr learned my addr 172.21.15.16:0/444555503 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:35.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.116+0000 7f29bd543700 1 --2- 172.21.15.16:0/444555503 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55bd44190010 0x55bd4423e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:35.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.116+0000 7f29bdd44700 1 --2- 172.21.15.16:0/444555503 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55bd441ba630 0x55bd441bfbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:35.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.116+0000 7f29bcd42700 1 -- 172.21.15.16:0/444555503 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55bd441ba630 msgr2=0x55bd441bfbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:35.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.116+0000 7f29bcd42700 1 --2- 172.21.15.16:0/444555503 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55bd441ba630 0x55bd441bfbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:35.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.116+0000 7f29bcd42700 1 -- 172.21.15.16:0/444555503 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55bd44190010 msgr2=0x55bd4423e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:35.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.116+0000 7f29bcd42700 1 --2- 172.21.15.16:0/444555503 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55bd44190010 0x55bd4423e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:35.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.116+0000 7f29bcd42700 1 -- 172.21.15.16:0/444555503 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29b80b0c30 con 0x55bd441b5220 2023-10-16T13:47:35.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.116+0000 7f29bcd42700 1 --2- 172.21.15.16:0/444555503 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd441b5220 0x55bd441bf6a0 secure :-1 s=READY pgs=2776 cs=0 l=1 rev1=1 crypto rx=0x7f29a8000c00 tx=0x7f29a80047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3f44ca567da52a7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:35.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.116+0000 7f2993fff700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2023-10-16T13:47:35.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.116+0000 7f2993fff700 1 -- 172.21.15.16:0/444555503 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29a8021040 con 0x55bd441b5220 2023-10-16T13:47:35.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.116+0000 7f2993fff700 1 -- 172.21.15.16:0/444555503 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(245..245 src has 1..245) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f29a8028b80 con 0x55bd441b5220 2023-10-16T13:47:35.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.116+0000 7f29cb991b80 1 -- 172.21.15.16:0/444555503 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f29b40145b0 con 0x55bd441b5220 2023-10-16T13:47:35.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.116+0000 7f2993fff700 1 -- 172.21.15.16:0/444555503 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=130) v2 ==== 24+0+0 (secure 0 0 0) 0x7f29a8028d60 con 0x55bd441b5220 2023-10-16T13:47:35.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.116+0000 7f29cb991b80 10 client.12435 fetch_fsmap learned FSMap version 130 2023-10-16T13:47:35.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.116+0000 7f29cb991b80 1 -- 172.21.15.16:0/444555503 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=130,monmap=2+}) v3 -- 0x55bd43f36200 con 0x55bd441b5220 2023-10-16T13:47:35.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.116+0000 7f2993fff700 1 -- 172.21.15.16:0/444555503 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29b40145b0 con 0x55bd441b5220 2023-10-16T13:47:35.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.116+0000 7f2993fff700 1 -- 172.21.15.16:0/444555503 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 130) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f29a8004070 con 0x55bd441b5220 2023-10-16T13:47:35.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.117+0000 7f29cb991b80 10 client.12435 fetch_fsmap finished waiting for FSMap version 130 2023-10-16T13:47:35.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.117+0000 7f29cb991b80 10 client.12435 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-16T13:47:35.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.117+0000 7f29cb991b80 20 client.12435 populate_metadata read hostname 'smithi016' 2023-10-16T13:47:35.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.117+0000 7f29cb991b80 1 --2- 172.21.15.16:0/444555503 >> [v2:172.21.15.16:6835/303278795,v1:172.21.15.16:6837/303278795] conn(0x55bd44186f40 0x55bd44187320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:35.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.117+0000 7f29cb991b80 4 client.12435 mds_command: new command op to 12252 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T13:47:35.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.117+0000 7f29cb991b80 1 -- 172.21.15.16:0/444555503 --> [v2:172.21.15.16:6835/303278795,v1:172.21.15.16:6837/303278795] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55bd440ddb20 con 0x55bd44186f40 2023-10-16T13:47:35.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.117+0000 7f29bd543700 1 --2- 172.21.15.16:0/444555503 >> [v2:172.21.15.16:6835/303278795,v1:172.21.15.16:6837/303278795] conn(0x55bd44186f40 0x55bd44187320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:35.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.117+0000 7f2993fff700 10 client.12435 ms_handle_connect on v2:172.21.15.16:6835/303278795 2023-10-16T13:47:35.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.117+0000 7f29bd543700 1 --2- 172.21.15.16:0/444555503 >> [v2:172.21.15.16:6835/303278795,v1:172.21.15.16:6837/303278795] conn(0x55bd44186f40 0x55bd44187320 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2a8629a772196fd6 server_cookie=2ad0d114bb5a9ca4 in_seq=0 out_seq=0 2023-10-16T13:47:35.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.117+0000 7f2993fff700 1 -- 172.21.15.16:0/444555503 <== mds.0 v2:172.21.15.16:6835/303278795 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+323 (crc 0 0 0) 0x55bd440ddb20 con 0x55bd44186f40 2023-10-16T13:47:35.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.117+0000 7f2993fff700 10 client.12435 handle_command_reply: tid=0 2023-10-16T13:47:35.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.117+0000 7f29cb991b80 1 client.12435 shutdown 2023-10-16T13:47:35.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.117+0000 7f29cb991b80 1 -- 172.21.15.16:0/444555503 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd441b5220 msgr2=0x55bd441bf6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:35.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.117+0000 7f29cb991b80 1 --2- 172.21.15.16:0/444555503 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd441b5220 0x55bd441bf6a0 secure :-1 s=READY pgs=2776 cs=0 l=1 rev1=1 crypto rx=0x7f29a8000c00 tx=0x7f29a80047c0 comp rx=0 tx=0).stop 2023-10-16T13:47:35.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.117+0000 7f29cb991b80 1 -- 172.21.15.16:0/444555503 shutdown_connections 2023-10-16T13:47:35.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.117+0000 7f29cb991b80 1 --2- 172.21.15.16:0/444555503 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55bd441ba630 0x55bd441bfbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:35.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.117+0000 7f29cb991b80 1 --2- 172.21.15.16:0/444555503 >> [v2:172.21.15.16:6835/303278795,v1:172.21.15.16:6837/303278795] conn(0x55bd44186f40 0x55bd44187320 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:35.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.117+0000 7f29cb991b80 1 --2- 172.21.15.16:0/444555503 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55bd44190010 0x55bd4423e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:35.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.117+0000 7f29cb991b80 1 --2- 172.21.15.16:0/444555503 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd441b5220 0x55bd441bf6a0 unknown :-1 s=CLOSED pgs=2776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:35.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.117+0000 7f29cb991b80 1 -- 172.21.15.16:0/444555503 >> 172.21.15.16:0/444555503 conn(0x55bd440bfd80 msgr2=0x55bd441bca60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:35.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.118+0000 7f2993fff700 0 client.12435 ms_handle_reset on v2:172.21.15.16:6835/303278795 2023-10-16T13:47:35.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.118+0000 7f29cb991b80 1 -- 172.21.15.16:0/444555503 shutdown_connections 2023-10-16T13:47:35.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.118+0000 7f29cb991b80 1 -- 172.21.15.16:0/444555503 wait complete. 2023-10-16T13:47:35.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.118+0000 7f29cb991b80 20 client.12435 trim_cache size 0 max 16384 2023-10-16T13:47:35.140 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-10-16T13:47:35.140 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "5b28dfa1-b21f-474a-9b2e-9fb77af58254", 2023-10-16T13:47:35.140 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 0, 2023-10-16T13:47:35.141 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 12252, 2023-10-16T13:47:35.141 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2023-10-16T13:47:35.141 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2023-10-16T13:47:35.141 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2023-10-16T13:47:35.141 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 10.692952881, 2023-10-16T13:47:35.142 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 130, 2023-10-16T13:47:35.142 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 245, 2023-10-16T13:47:35.142 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 245, 2023-10-16T13:47:35.142 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 15.588964061 2023-10-16T13:47:35.142 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-10-16T13:47:35.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.118+0000 7f29bf7a7700 1 -- 172.21.15.16:0/2325935048 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f299c06b140 msgr2=0x7f299c06d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:35.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.118+0000 7f29bf7a7700 1 --2- 172.21.15.16:0/2325935048 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f299c06b140 0x7f299c06d600 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f29b4007fd0 tx=0x7f29b4009040 comp rx=0 tx=0).stop 2023-10-16T13:47:35.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.118+0000 7f29bf7a7700 1 -- 172.21.15.16:0/2325935048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29b815ba10 msgr2=0x7f29b8151cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:35.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.118+0000 7f29bf7a7700 1 --2- 172.21.15.16:0/2325935048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29b815ba10 0x7f29b8151cc0 secure :-1 s=READY pgs=2772 cs=0 l=1 rev1=1 crypto rx=0x7f29b0007e60 tx=0x7f29b000e3c0 comp rx=0 tx=0).stop 2023-10-16T13:47:35.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.118+0000 7f29bf7a7700 1 -- 172.21.15.16:0/2325935048 shutdown_connections 2023-10-16T13:47:35.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.118+0000 7f29bf7a7700 1 --2- 172.21.15.16:0/2325935048 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f299c06b140 0x7f299c06d600 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:35.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.118+0000 7f29bf7a7700 1 --2- 172.21.15.16:0/2325935048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29b815ba10 0x7f29b8151cc0 unknown :-1 s=CLOSED pgs=2772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:35.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.118+0000 7f29bf7a7700 1 --2- 172.21.15.16:0/2325935048 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f29b80ce4f0 0x7f29b80c85f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:35.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.118+0000 7f29bf7a7700 1 --2- 172.21.15.16:0/2325935048 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f29b815b630 0x7f29b814e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:35.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.118+0000 7f29bf7a7700 1 -- 172.21.15.16:0/2325935048 >> 172.21.15.16:0/2325935048 conn(0x7f29b80d14b0 msgr2=0x7f29b806fa30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:35.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.118+0000 7f29bf7a7700 1 -- 172.21.15.16:0/2325935048 shutdown_connections 2023-10-16T13:47:35.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.118+0000 7f29bf7a7700 1 -- 172.21.15.16:0/2325935048 wait complete. 2023-10-16T13:47:35.146 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T13:47:35.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.269+0000 7f99811a9700 1 Processor -- start 2023-10-16T13:47:35.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.269+0000 7f99811a9700 1 -- start start 2023-10-16T13:47:35.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.270+0000 7f99811a9700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f997c0cf4a0 0x7f997c0726e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:35.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.270+0000 7f99811a9700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f997c0cf880 0x7f997c0b52f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:35.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.270+0000 7f99811a9700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f997c0b5830 0x7f997c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:35.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.270+0000 7f99811a9700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f997c15bc80 con 0x7f997c0cf4a0 2023-10-16T13:47:35.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.270+0000 7f99811a9700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f997c072c20 con 0x7f997c0b5830 2023-10-16T13:47:35.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.270+0000 7f99811a9700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f997c15b070 con 0x7f997c0cf880 2023-10-16T13:47:35.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.270+0000 7f997ad9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f997c0cf4a0 0x7f997c0726e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:35.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.270+0000 7f997ad9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f997c0cf4a0 0x7f997c0726e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55640/0 (socket says 172.21.15.16:55640) 2023-10-16T13:47:35.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.270+0000 7f997ad9d700 1 -- 172.21.15.16:0/2801814710 learned_addr learned my addr 172.21.15.16:0/2801814710 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:35.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.270+0000 7f997b59e700 1 --2- 172.21.15.16:0/2801814710 >> v2:172.21.15.49:3300/0 conn(0x7f997c0b5830 0x7f997c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:35.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.270+0000 7f997a59c700 1 --2- 172.21.15.16:0/2801814710 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f997c0cf880 0x7f997c0b52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:35.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.270+0000 7f997ad9d700 1 -- 172.21.15.16:0/2801814710 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f997c0cf880 msgr2=0x7f997c0b52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:35.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.270+0000 7f997ad9d700 1 --2- 172.21.15.16:0/2801814710 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f997c0cf880 0x7f997c0b52f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:35.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.270+0000 7f997ad9d700 1 -- 172.21.15.16:0/2801814710 >> v2:172.21.15.49:3300/0 conn(0x7f997c0b5830 msgr2=0x7f997c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:35.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.270+0000 7f997ad9d700 1 --2- 172.21.15.16:0/2801814710 >> v2:172.21.15.49:3300/0 conn(0x7f997c0b5830 0x7f997c15a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:35.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.270+0000 7f997ad9d700 1 -- 172.21.15.16:0/2801814710 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f997c15b350 con 0x7f997c0cf4a0 2023-10-16T13:47:35.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.270+0000 7f997ad9d700 1 --2- 172.21.15.16:0/2801814710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f997c0cf4a0 0x7f997c0726e0 secure :-1 s=READY pgs=2777 cs=0 l=1 rev1=1 crypto rx=0x7f997c071da0 tx=0x7f9970009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d281822c9f0fcec7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:35.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.271+0000 7f9979d9b700 1 -- 172.21.15.16:0/2801814710 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9970011830 con 0x7f997c0cf4a0 2023-10-16T13:47:35.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.271+0000 7f9979d9b700 1 -- 172.21.15.16:0/2801814710 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9970011990 con 0x7f997c0cf4a0 2023-10-16T13:47:35.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.271+0000 7f9979d9b700 1 -- 172.21.15.16:0/2801814710 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9970011d20 con 0x7f997c0cf4a0 2023-10-16T13:47:35.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.271+0000 7f99811a9700 1 -- 172.21.15.16:0/2801814710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f997c0cf4a0 msgr2=0x7f997c0726e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:35.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.271+0000 7f99811a9700 1 --2- 172.21.15.16:0/2801814710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f997c0cf4a0 0x7f997c0726e0 secure :-1 s=READY pgs=2777 cs=0 l=1 rev1=1 crypto rx=0x7f997c071da0 tx=0x7f9970009bd0 comp rx=0 tx=0).stop 2023-10-16T13:47:35.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.271+0000 7f99811a9700 1 -- 172.21.15.16:0/2801814710 shutdown_connections 2023-10-16T13:47:35.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.271+0000 7f99811a9700 1 --2- 172.21.15.16:0/2801814710 >> v2:172.21.15.49:3300/0 conn(0x7f997c0b5830 0x7f997c15a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:35.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.271+0000 7f99811a9700 1 --2- 172.21.15.16:0/2801814710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f997c0cf4a0 0x7f997c0726e0 unknown :-1 s=CLOSED pgs=2777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:35.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.271+0000 7f99811a9700 1 --2- 172.21.15.16:0/2801814710 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f997c0cf880 0x7f997c0b52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:35.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.271+0000 7f99811a9700 1 -- 172.21.15.16:0/2801814710 >> 172.21.15.16:0/2801814710 conn(0x7f997c0b1fd0 msgr2=0x7f997c0b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:35.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.271+0000 7f99811a9700 1 -- 172.21.15.16:0/2801814710 shutdown_connections 2023-10-16T13:47:35.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.271+0000 7f99811a9700 1 -- 172.21.15.16:0/2801814710 wait complete. 2023-10-16T13:47:35.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.273+0000 7f99811a9700 1 Processor -- start 2023-10-16T13:47:35.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.273+0000 7f99811a9700 1 -- start start 2023-10-16T13:47:35.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.274+0000 7f99811a9700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f997c0b5830 0x7f997c14dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:35.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.274+0000 7f99811a9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f997c0cf4a0 0x7f997c157150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:35.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.274+0000 7f99811a9700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f997c0cf880 0x7f997c0afe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:35.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.274+0000 7f99811a9700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f997c157b30 con 0x7f997c0cf4a0 2023-10-16T13:47:35.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.274+0000 7f99811a9700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f997c157e10 con 0x7f997c0cf880 2023-10-16T13:47:35.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.274+0000 7f99811a9700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f997c1580f0 con 0x7f997c0b5830 2023-10-16T13:47:35.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.274+0000 7f997a59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f997c0cf4a0 0x7f997c157150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:35.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.274+0000 7f997a59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f997c0cf4a0 0x7f997c157150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55648/0 (socket says 172.21.15.16:55648) 2023-10-16T13:47:35.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.274+0000 7f997a59c700 1 -- 172.21.15.16:0/791723631 learned_addr learned my addr 172.21.15.16:0/791723631 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:35.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.274+0000 7f997b59e700 1 --2- 172.21.15.16:0/791723631 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f997c0cf880 0x7f997c0afe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:35.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.274+0000 7f997ad9d700 1 --2- 172.21.15.16:0/791723631 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f997c0b5830 0x7f997c14dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:35.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.274+0000 7f997a59c700 1 -- 172.21.15.16:0/791723631 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f997c0b5830 msgr2=0x7f997c14dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:35.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.274+0000 7f997a59c700 1 --2- 172.21.15.16:0/791723631 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f997c0b5830 0x7f997c14dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:35.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.274+0000 7f997a59c700 1 -- 172.21.15.16:0/791723631 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f997c0cf880 msgr2=0x7f997c0afe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:35.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.274+0000 7f997a59c700 1 --2- 172.21.15.16:0/791723631 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f997c0cf880 0x7f997c0afe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:35.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.274+0000 7f997a59c700 1 -- 172.21.15.16:0/791723631 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9970009550 con 0x7f997c0cf4a0 2023-10-16T13:47:35.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.274+0000 7f997a59c700 1 --2- 172.21.15.16:0/791723631 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f997c0cf4a0 0x7f997c157150 secure :-1 s=READY pgs=2778 cs=0 l=1 rev1=1 crypto rx=0x7f996c00e870 tx=0x7f996c00eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:35.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.274+0000 7f99637fe700 1 -- 172.21.15.16:0/791723631 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f996c009590 con 0x7f997c0cf4a0 2023-10-16T13:47:35.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.274+0000 7f99637fe700 1 -- 172.21.15.16:0/791723631 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f996c013680 con 0x7f997c0cf4a0 2023-10-16T13:47:35.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.274+0000 7f99811a9700 1 -- 172.21.15.16:0/791723631 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f997c1583d0 con 0x7f997c0cf4a0 2023-10-16T13:47:35.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.274+0000 7f99637fe700 1 -- 172.21.15.16:0/791723631 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f996c0139f0 con 0x7f997c0cf4a0 2023-10-16T13:47:35.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.275+0000 7f99811a9700 1 -- 172.21.15.16:0/791723631 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f997c159200 con 0x7f997c0cf4a0 2023-10-16T13:47:35.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.275+0000 7f99811a9700 1 -- 172.21.15.16:0/791723631 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f997c1588f0 con 0x7f997c0cf4a0 2023-10-16T13:47:35.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.276+0000 7f99637fe700 1 -- 172.21.15.16:0/791723631 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 148) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f996c013b50 con 0x7f997c0cf4a0 2023-10-16T13:47:35.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.276+0000 7f99637fe700 1 --2- 172.21.15.16:0/791723631 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f996806b170 0x7f996806d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:35.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.277+0000 7f99637fe700 1 -- 172.21.15.16:0/791723631 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(245..245 src has 1..245) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f996c089960 con 0x7f997c0cf4a0 2023-10-16T13:47:35.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.277+0000 7f997ad9d700 1 --2- 172.21.15.16:0/791723631 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f996806b170 0x7f996806d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:35.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.277+0000 7f997ad9d700 1 --2- 172.21.15.16:0/791723631 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f996806b170 0x7f996806d630 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9970009c10 tx=0x7f9970008030 comp rx=0 tx=0).ready entity=mgr.11886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:35.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.281+0000 7f99637fe700 1 -- 172.21.15.16:0/791723631 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f997c1588f0 con 0x7f997c0cf4a0 2023-10-16T13:47:35.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.532+0000 7f99811a9700 1 -- 172.21.15.16:0/791723631 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f997c1588f0 con 0x7f997c0cf4a0 2023-10-16T13:47:35.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.532+0000 7f99637fe700 1 -- 172.21.15.16:0/791723631 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v130) v1 ==== 53+0+73 (secure 0 0 0) 0x7f997c1588f0 con 0x7f997c0cf4a0 2023-10-16T13:47:35.535 INFO:teuthology.orchestra.run.smithi016.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-16T13:47:35.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.534+0000 7f99811a9700 1 -- 172.21.15.16:0/791723631 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f996806b170 msgr2=0x7f996806d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:35.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.534+0000 7f99811a9700 1 --2- 172.21.15.16:0/791723631 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f996806b170 0x7f996806d630 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9970009c10 tx=0x7f9970008030 comp rx=0 tx=0).stop 2023-10-16T13:47:35.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.534+0000 7f99811a9700 1 -- 172.21.15.16:0/791723631 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f997c0cf4a0 msgr2=0x7f997c157150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:35.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.534+0000 7f99811a9700 1 --2- 172.21.15.16:0/791723631 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f997c0cf4a0 0x7f997c157150 secure :-1 s=READY pgs=2778 cs=0 l=1 rev1=1 crypto rx=0x7f996c00e870 tx=0x7f996c00eb80 comp rx=0 tx=0).stop 2023-10-16T13:47:35.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.535+0000 7f99811a9700 1 -- 172.21.15.16:0/791723631 shutdown_connections 2023-10-16T13:47:35.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.535+0000 7f99811a9700 1 --2- 172.21.15.16:0/791723631 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f996806b170 0x7f996806d630 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:35.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.535+0000 7f99811a9700 1 --2- 172.21.15.16:0/791723631 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f997c0cf880 0x7f997c0afe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:35.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.535+0000 7f99811a9700 1 --2- 172.21.15.16:0/791723631 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f997c0b5830 0x7f997c14dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:35.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.535+0000 7f99811a9700 1 --2- 172.21.15.16:0/791723631 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f997c0cf4a0 0x7f997c157150 unknown :-1 s=CLOSED pgs=2778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:35.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.535+0000 7f99811a9700 1 -- 172.21.15.16:0/791723631 >> 172.21.15.16:0/791723631 conn(0x7f997c0b1fd0 msgr2=0x7f997c0cdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:35.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.535+0000 7f99811a9700 1 -- 172.21.15.16:0/791723631 shutdown_connections 2023-10-16T13:47:35.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:35.535+0000 7f99811a9700 1 -- 172.21.15.16:0/791723631 wait complete. 2023-10-16T13:47:35.549 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-16T13:47:35.550 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T13:47:35.550 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi016.front.sepia.ceph.com 2023-10-16T13:47:35.550 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-16T13:47:35.551 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-16T13:47:35.551 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-16T13:47:35.551 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-16T13:47:35.552 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T13:47:35.552 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T13:47:35.552 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:47:35.553 DEBUG:teuthology.orchestra.run.smithi016:> ip addr 2023-10-16T13:47:35.571 INFO:teuthology.orchestra.run.smithi016.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T13:47:35.571 INFO:teuthology.orchestra.run.smithi016.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T13:47:35.571 INFO:teuthology.orchestra.run.smithi016.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T13:47:35.571 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T13:47:35.571 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 ::1/128 scope host 2023-10-16T13:47:35.571 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T13:47:35.572 INFO:teuthology.orchestra.run.smithi016.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T13:47:35.572 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e0 brd ff:ff:ff:ff:ff:ff 2023-10-16T13:47:35.572 INFO:teuthology.orchestra.run.smithi016.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T13:47:35.572 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e1 brd ff:ff:ff:ff:ff:ff 2023-10-16T13:47:35.572 INFO:teuthology.orchestra.run.smithi016.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T13:47:35.573 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:64 brd ff:ff:ff:ff:ff:ff 2023-10-16T13:47:35.573 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.21.15.16/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-16T13:47:35.573 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft 36481sec preferred_lft 36481sec 2023-10-16T13:47:35.573 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::ec4:7aff:febd:1564/64 scope link 2023-10-16T13:47:35.573 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T13:47:35.574 INFO:teuthology.orchestra.run.smithi016.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T13:47:35.574 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:65 brd ff:ff:ff:ff:ff:ff 2023-10-16T13:47:35.574 INFO:teuthology.orchestra.run.smithi016.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T13:47:35.574 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 36:cf:ea:25:db:f7 brd ff:ff:ff:ff:ff:ff 2023-10-16T13:47:35.574 INFO:teuthology.orchestra.run.smithi016.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-16T13:47:35.575 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T13:47:35.575 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::cfc:c5ff:fe52:a786/64 scope link 2023-10-16T13:47:35.575 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T13:47:35.575 INFO:teuthology.orchestra.run.smithi016.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T13:47:35.575 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 6e:62:9d:a4:43:23 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T13:47:35.576 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::6c62:9dff:fea4:4323/64 scope link 2023-10-16T13:47:35.576 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T13:47:35.576 INFO:teuthology.orchestra.run.smithi016.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T13:47:35.576 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 36:cf:ea:25:db:f7 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-16T13:47:35.576 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::34cf:eaff:fe25:dbf7/64 scope link 2023-10-16T13:47:35.576 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T13:47:35.577 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:47:35.577 DEBUG:teuthology.orchestra.run.smithi016:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T13:47:35.637 INFO:teuthology.orchestra.run.smithi016.stdout:1 2023-10-16T13:47:35.639 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:47:35.639 DEBUG:teuthology.orchestra.run.smithi016:> route 2023-10-16T13:47:35.698 INFO:teuthology.orchestra.run.smithi016.stdout:Kernel IP routing table 2023-10-16T13:47:35.698 INFO:teuthology.orchestra.run.smithi016.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T13:47:35.699 INFO:teuthology.orchestra.run.smithi016.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-16T13:47:35.699 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-16T13:47:35.699 INFO:teuthology.orchestra.run.smithi016.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T13:47:35.699 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:47:35.700 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec bash -c ' 2023-10-16T13:47:35.700 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2023-10-16T13:47:35.700 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-16T13:47:35.701 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-16T13:47:35.701 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-16T13:47:35.701 DEBUG:teuthology.orchestra.run.smithi016:> ') 2023-10-16T13:47:35.805 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:47:35.805 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2023-10-16T13:47:35.822 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-16T13:47:35.823 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-16T13:47:35.823 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T13:47:35.823 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:47:35.824 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set brx.0 up 2023-10-16T13:47:35.890 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T13:47:35.890 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-16T13:47:35.947 INFO:teuthology.orchestra.run.smithi016.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-16T13:47:35.947 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T13:47:35.948 DEBUG:teuthology.orchestra.run.smithi016:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-16T13:47:36.004 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-16T13:47:36.004 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:47:36.004 DEBUG:teuthology.orchestra.run.smithi016:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T13:47:36.091 INFO:teuthology.orchestra.run.smithi016.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T13:47:36.091 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T13:47:36.091 INFO:teuthology.orchestra.run.smithi016.stdout:invalid new device string format 2023-10-16T13:47:36.100 INFO:teuthology.orchestra.run.smithi016.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T13:47:36.100 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T13:47:36.100 INFO:teuthology.orchestra.run.smithi016.stdout:invalid new device string format 2023-10-16T13:47:36.100 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: resolved to: "172.21.15.16:3300,172.21.15.49:3300,172.21.15.49:3301" 2023-10-16T13:47:36.101 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T13:47:36.173 INFO:teuthology.orchestra.run.smithi016.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T13:47:36.173 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T13:47:36.173 INFO:teuthology.orchestra.run.smithi016.stdout:invalid new device string format 2023-10-16T13:47:36.173 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: resolved to: "172.21.15.16:3300,172.21.15.49:3300,172.21.15.49:3301" 2023-10-16T13:47:36.173 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T13:47:36.174 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: resolved to: "172.21.15.16:6789,172.21.15.49:6789,172.21.15.49:6790" 2023-10-16T13:47:36.174 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.16:6789,172.21.15.49:6789,172.21.15.49:6790:/ 2023-10-16T13:47:36.174 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=5b28dfa1-b21f-474a-9b2e-9fb77af58254" will pass to kernel 2023-10-16T13:47:36.175 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-16T13:47:36.175 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:47:36.176 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-16T13:47:36.204 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:47:36.204 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T13:47:36.294 INFO:teuthology.orchestra.run.smithi016.stdout:client9704 2023-10-16T13:47:36.294 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T13:47:36.294 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:47:36.312 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2023-10-16T13:47:36.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.479+0000 7f377beab700 1 Processor -- start 2023-10-16T13:47:36.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.479+0000 7f377beab700 1 -- start start 2023-10-16T13:47:36.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.480+0000 7f377beab700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3774157b30 0x7f37740cefc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:36.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.480+0000 7f377beab700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f3774157f10 0x7f37740cf500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:36.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.480+0000 7f377beab700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3774142fc0 con 0x7f3774161610 2023-10-16T13:47:36.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.480+0000 7f377beab700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f377406d900 con 0x7f3774157f10 2023-10-16T13:47:36.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.480+0000 7f377beab700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3774143210 con 0x7f3774157b30 2023-10-16T13:47:36.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.480+0000 7f377a448700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f3774161610 0x7f3774161a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44936/0 (socket says 172.21.15.16:44936) 2023-10-16T13:47:36.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.480+0000 7f377a448700 1 -- 172.21.15.16:0/1300431018 learned_addr learned my addr 172.21.15.16:0/1300431018 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:47:36.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.480+0000 7f3779446700 1 --2- 172.21.15.16:0/1300431018 >> v2:172.21.15.49:3300/0 conn(0x7f3774157f10 0x7f37740cf500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:36.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.480+0000 7f3779c47700 1 --2- 172.21.15.16:0/1300431018 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3774157b30 0x7f37740cefc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:36.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.480+0000 7f3778c45700 1 -- 172.21.15.16:0/1300431018 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2418783291 0 0) 0x7f3774142fc0 con 0x7f3774161610 2023-10-16T13:47:36.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.480+0000 7f3778c45700 1 -- 172.21.15.16:0/1300431018 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37600035b0 con 0x7f3774161610 2023-10-16T13:47:36.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.480+0000 7f3778c45700 1 -- 172.21.15.16:0/1300431018 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2538084968 0 0) 0x7f37600035b0 con 0x7f3774161610 2023-10-16T13:47:36.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.480+0000 7f3778c45700 1 -- 172.21.15.16:0/1300431018 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3774157b30 msgr2=0x7f37740cefc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:36.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.480+0000 7f3778c45700 1 --2- 172.21.15.16:0/1300431018 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3774157b30 0x7f37740cefc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:36.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.480+0000 7f3778c45700 1 -- 172.21.15.16:0/1300431018 >> v2:172.21.15.49:3300/0 conn(0x7f3774157f10 msgr2=0x7f37740cf500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:36.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.480+0000 7f3778c45700 1 --2- 172.21.15.16:0/1300431018 >> v2:172.21.15.49:3300/0 conn(0x7f3774157f10 0x7f37740cf500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:36.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.480+0000 7f3778c45700 1 -- 172.21.15.16:0/1300431018 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37741434f0 con 0x7f3774161610 2023-10-16T13:47:36.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.480+0000 7f3778c45700 1 -- 172.21.15.16:0/1300431018 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f376c0019f0 con 0x7f3774161610 2023-10-16T13:47:36.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.480+0000 7f3778c45700 1 -- 172.21.15.16:0/1300431018 >> v1:172.21.15.16:6789/0 conn(0x7f3774161610 legacy=0x7f3774161a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:36.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.480+0000 7f3778c45700 1 --2- 172.21.15.16:0/1300431018 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3774157b30 0x7f3760003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:36.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.480+0000 7f3778c45700 1 --2- 172.21.15.16:0/1300431018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3760003f80 0x7f3760004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:36.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.480+0000 7f3778c45700 1 --2- 172.21.15.16:0/1300431018 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3760004d90 0x7f3760005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:36.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.480+0000 7f3778c45700 1 -- 172.21.15.16:0/1300431018 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f377406d900 con 0x7f3760003f80 2023-10-16T13:47:36.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.480+0000 7f3778c45700 1 -- 172.21.15.16:0/1300431018 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3774143210 con 0x7f3760004d90 2023-10-16T13:47:36.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.480+0000 7f3778c45700 1 -- 172.21.15.16:0/1300431018 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f376000ba10 con 0x7f3774157b30 2023-10-16T13:47:36.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.480+0000 7f3779446700 1 --2- 172.21.15.16:0/1300431018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3760003f80 0x7f3760004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:36.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.481+0000 7f3779c47700 1 --2- 172.21.15.16:0/1300431018 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3774157b30 0x7f3760003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:36.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.481+0000 7f377a448700 1 --2- 172.21.15.16:0/1300431018 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3760004d90 0x7f3760005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:36.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.481+0000 7f3779446700 1 -- 172.21.15.16:0/1300431018 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3774157b30 msgr2=0x7f3760003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:36.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.481+0000 7f3779446700 1 --2- 172.21.15.16:0/1300431018 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3774157b30 0x7f3760003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:36.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.481+0000 7f3779446700 1 -- 172.21.15.16:0/1300431018 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3760004d90 msgr2=0x7f3760005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:36.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.481+0000 7f3779446700 1 --2- 172.21.15.16:0/1300431018 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3760004d90 0x7f3760005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:36.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.481+0000 7f3779446700 1 -- 172.21.15.16:0/1300431018 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37600035b0 con 0x7f3760003f80 2023-10-16T13:47:36.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.481+0000 7f377a448700 1 --2- 172.21.15.16:0/1300431018 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3760004d90 0x7f3760005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:47:36.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.481+0000 7f3779446700 1 --2- 172.21.15.16:0/1300431018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3760003f80 0x7f3760004850 secure :-1 s=READY pgs=2780 cs=0 l=1 rev1=1 crypto rx=0x7f3764009b60 tx=0x7f3760005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec89af1e546cff15 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:36.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.481+0000 7f3778c45700 1 -- 172.21.15.16:0/1300431018 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f376400e660 con 0x7f3760003f80 2023-10-16T13:47:36.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.481+0000 7f3778c45700 1 -- 172.21.15.16:0/1300431018 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f376400e7c0 con 0x7f3760003f80 2023-10-16T13:47:36.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.481+0000 7f3778c45700 1 -- 172.21.15.16:0/1300431018 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f376400eb30 con 0x7f3760003f80 2023-10-16T13:47:36.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.482+0000 7f377beab700 1 -- 172.21.15.16:0/1300431018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3760003f80 msgr2=0x7f3760004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:36.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.482+0000 7f377beab700 1 --2- 172.21.15.16:0/1300431018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3760003f80 0x7f3760004850 secure :-1 s=READY pgs=2780 cs=0 l=1 rev1=1 crypto rx=0x7f3764009b60 tx=0x7f3760005880 comp rx=0 tx=0).stop 2023-10-16T13:47:36.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.482+0000 7f377a448700 1 -- 172.21.15.16:0/1300431018 reap_dead start 2023-10-16T13:47:36.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.482+0000 7f377beab700 1 -- 172.21.15.16:0/1300431018 shutdown_connections 2023-10-16T13:47:36.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.482+0000 7f377beab700 1 -- 172.21.15.16:0/1300431018 >> 172.21.15.16:0/1300431018 conn(0x7f377406a7e0 msgr2=0x7f377406abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:36.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.482+0000 7f377beab700 1 -- 172.21.15.16:0/1300431018 shutdown_connections 2023-10-16T13:47:36.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.482+0000 7f377beab700 1 -- 172.21.15.16:0/1300431018 wait complete. 2023-10-16T13:47:36.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.483+0000 7f377beab700 1 Processor -- start 2023-10-16T13:47:36.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.483+0000 7f377beab700 1 -- start start 2023-10-16T13:47:36.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.484+0000 7f377beab700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3760003f80 0x7f3774152e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:36.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.484+0000 7f377beab700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f37740aff10 0x7f37741f6980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:36.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.484+0000 7f377beab700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f37741f6ec0 0x7f37741f3390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:36.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.484+0000 7f377beab700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37741f7370 con 0x7f3760003f80 2023-10-16T13:47:36.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.484+0000 7f377beab700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37741534b0 con 0x7f37741f6ec0 2023-10-16T13:47:36.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.484+0000 7f377beab700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3774153790 con 0x7f37740aff10 2023-10-16T13:47:36.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.484+0000 7f3779c47700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3760003f80 0x7f3774152e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:36.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.484+0000 7f3779c47700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3760003f80 0x7f3774152e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55670/0 (socket says 172.21.15.16:55670) 2023-10-16T13:47:36.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.484+0000 7f3779c47700 1 -- 172.21.15.16:0/3848831512 learned_addr learned my addr 172.21.15.16:0/3848831512 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:36.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.484+0000 7f377a448700 1 --2- 172.21.15.16:0/3848831512 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f37741f6ec0 0x7f37741f3390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:36.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.484+0000 7f3779446700 1 --2- 172.21.15.16:0/3848831512 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f37740aff10 0x7f37741f6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:36.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.484+0000 7f3779c47700 1 -- 172.21.15.16:0/3848831512 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f37740aff10 msgr2=0x7f37741f6980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:36.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.484+0000 7f3779c47700 1 --2- 172.21.15.16:0/3848831512 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f37740aff10 0x7f37741f6980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:36.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.484+0000 7f3779c47700 1 -- 172.21.15.16:0/3848831512 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f37741f6ec0 msgr2=0x7f37741f3390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:36.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.484+0000 7f3779c47700 1 --2- 172.21.15.16:0/3848831512 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f37741f6ec0 0x7f37741f3390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:36.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.484+0000 7f3779c47700 1 -- 172.21.15.16:0/3848831512 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3774153a70 con 0x7f3760003f80 2023-10-16T13:47:36.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.484+0000 7f3779c47700 1 --2- 172.21.15.16:0/3848831512 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3760003f80 0x7f3774152e60 secure :-1 s=READY pgs=2781 cs=0 l=1 rev1=1 crypto rx=0x7f3770011aa0 tx=0x7f3770011db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:36.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.484+0000 7f376a7fc700 1 -- 172.21.15.16:0/3848831512 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3770011ee0 con 0x7f3760003f80 2023-10-16T13:47:36.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.484+0000 7f376a7fc700 1 -- 172.21.15.16:0/3848831512 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f377000f940 con 0x7f3760003f80 2023-10-16T13:47:36.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.484+0000 7f377beab700 1 -- 172.21.15.16:0/3848831512 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3774153d50 con 0x7f3760003f80 2023-10-16T13:47:36.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.484+0000 7f376a7fc700 1 -- 172.21.15.16:0/3848831512 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f377000fcf0 con 0x7f3760003f80 2023-10-16T13:47:36.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.484+0000 7f377beab700 1 -- 172.21.15.16:0/3848831512 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37740b0390 con 0x7f3760003f80 2023-10-16T13:47:36.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.485+0000 7f377beab700 1 -- 172.21.15.16:0/3848831512 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37740cefa0 con 0x7f3760003f80 2023-10-16T13:47:36.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.486+0000 7f376a7fc700 1 -- 172.21.15.16:0/3848831512 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 148) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f3770014440 con 0x7f3760003f80 2023-10-16T13:47:36.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.486+0000 7f376a7fc700 1 --2- 172.21.15.16:0/3848831512 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f375806b140 0x7f375806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:36.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.486+0000 7f376a7fc700 1 -- 172.21.15.16:0/3848831512 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(245..245 src has 1..245) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f377008a210 con 0x7f3760003f80 2023-10-16T13:47:36.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.486+0000 7f3779446700 1 --2- 172.21.15.16:0/3848831512 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f375806b140 0x7f375806d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:36.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.487+0000 7f3779446700 1 --2- 172.21.15.16:0/3848831512 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f375806b140 0x7f375806d600 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f37640090b0 tx=0x7f376401f040 comp rx=0 tx=0).ready entity=mgr.11886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:36.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.491+0000 7f376a7fc700 1 -- 172.21.15.16:0/3848831512 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f37740cefa0 con 0x7f3760003f80 2023-10-16T13:47:36.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.742+0000 7f377beab700 1 -- 172.21.15.16:0/3848831512 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f37740cefa0 con 0x7f3760003f80 2023-10-16T13:47:36.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.883+0000 7f376a7fc700 1 -- 172.21.15.16:0/3848831512 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v131) v1 ==== 136+0+0 (secure 0 0 0) 0x7f37740cefa0 con 0x7f3760003f80 2023-10-16T13:47:36.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.886+0000 7f377beab700 1 -- 172.21.15.16:0/3848831512 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f375806b140 msgr2=0x7f375806d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:36.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.886+0000 7f377beab700 1 --2- 172.21.15.16:0/3848831512 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f375806b140 0x7f375806d600 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f37640090b0 tx=0x7f376401f040 comp rx=0 tx=0).stop 2023-10-16T13:47:36.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.886+0000 7f377beab700 1 -- 172.21.15.16:0/3848831512 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3760003f80 msgr2=0x7f3774152e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:36.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.886+0000 7f377beab700 1 --2- 172.21.15.16:0/3848831512 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3760003f80 0x7f3774152e60 secure :-1 s=READY pgs=2781 cs=0 l=1 rev1=1 crypto rx=0x7f3770011aa0 tx=0x7f3770011db0 comp rx=0 tx=0).stop 2023-10-16T13:47:36.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.886+0000 7f377beab700 1 -- 172.21.15.16:0/3848831512 shutdown_connections 2023-10-16T13:47:36.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.886+0000 7f377beab700 1 --2- 172.21.15.16:0/3848831512 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f375806b140 0x7f375806d600 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:36.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.886+0000 7f377beab700 1 --2- 172.21.15.16:0/3848831512 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f37741f6ec0 0x7f37741f3390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:36.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.886+0000 7f377beab700 1 --2- 172.21.15.16:0/3848831512 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3760003f80 0x7f3774152e60 unknown :-1 s=CLOSED pgs=2781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:36.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.886+0000 7f377beab700 1 --2- 172.21.15.16:0/3848831512 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f37740aff10 0x7f37741f6980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:36.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.886+0000 7f377beab700 1 -- 172.21.15.16:0/3848831512 >> 172.21.15.16:0/3848831512 conn(0x7f377406a7e0 msgr2=0x7f377406ea90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:36.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.886+0000 7f377beab700 1 -- 172.21.15.16:0/3848831512 shutdown_connections 2023-10-16T13:47:36.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:36.886+0000 7f377beab700 1 -- 172.21.15.16:0/3848831512 wait complete. 2023-10-16T13:47:36.901 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:47:37.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.025+0000 7efef3af9700 1 Processor -- start 2023-10-16T13:47:37.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.025+0000 7efef3af9700 1 -- start start 2023-10-16T13:47:37.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.026+0000 7efef3af9700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efeec15d750 0x7efeec0cf920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:37.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.026+0000 7efef3af9700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7efeec15db30 0x7efeec06c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:37.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.026+0000 7efef3af9700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeec0d0070 con 0x7efeec161490 2023-10-16T13:47:37.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.026+0000 7efef3af9700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efeec0cfe60 con 0x7efeec15db30 2023-10-16T13:47:37.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.026+0000 7efef3af9700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efeec0d0350 con 0x7efeec15d750 2023-10-16T13:47:37.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.026+0000 7efef2096700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7efeec161490 0x7efeec0b2060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44954/0 (socket says 172.21.15.16:44954) 2023-10-16T13:47:37.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.026+0000 7efef2096700 1 -- 172.21.15.16:0/242560906 learned_addr learned my addr 172.21.15.16:0/242560906 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:47:37.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.026+0000 7efef1094700 1 --2- 172.21.15.16:0/242560906 >> v2:172.21.15.49:3300/0 conn(0x7efeec15db30 0x7efeec06c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:37.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.026+0000 7efef1895700 1 --2- 172.21.15.16:0/242560906 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efeec15d750 0x7efeec0cf920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:37.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.026+0000 7efef0893700 1 -- 172.21.15.16:0/242560906 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 87309544 0 0) 0x7efeec0d0070 con 0x7efeec161490 2023-10-16T13:47:37.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.026+0000 7efef0893700 1 -- 172.21.15.16:0/242560906 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efed80035b0 con 0x7efeec161490 2023-10-16T13:47:37.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.026+0000 7efef1895700 1 -- 172.21.15.16:0/242560906 >> v2:172.21.15.49:3300/0 conn(0x7efeec15db30 msgr2=0x7efeec06c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:37.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.026+0000 7efef1895700 1 --2- 172.21.15.16:0/242560906 >> v2:172.21.15.49:3300/0 conn(0x7efeec15db30 0x7efeec06c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:37.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.026+0000 7efef1895700 1 -- 172.21.15.16:0/242560906 >> v1:172.21.15.16:6789/0 conn(0x7efeec161490 legacy=0x7efeec0b2060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:37.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.026+0000 7efef1895700 1 -- 172.21.15.16:0/242560906 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeec0d0630 con 0x7efeec15d750 2023-10-16T13:47:37.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.026+0000 7efef1094700 1 --2- 172.21.15.16:0/242560906 >> v2:172.21.15.49:3300/0 conn(0x7efeec15db30 0x7efeec06c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:47:37.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.027+0000 7efef1895700 1 --2- 172.21.15.16:0/242560906 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efeec15d750 0x7efeec0cf920 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7efee8009820 tx=0x7efee8009b30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8653a06a1f44f1f8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:37.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.027+0000 7efef0893700 1 -- 172.21.15.16:0/242560906 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee8005e30 con 0x7efeec15d750 2023-10-16T13:47:37.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.027+0000 7efef0893700 1 -- 172.21.15.16:0/242560906 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efee8006030 con 0x7efeec15d750 2023-10-16T13:47:37.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.027+0000 7efef0893700 1 -- 172.21.15.16:0/242560906 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee80063a0 con 0x7efeec15d750 2023-10-16T13:47:37.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.027+0000 7efef3af9700 1 -- 172.21.15.16:0/242560906 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efeec15d750 msgr2=0x7efeec0cf920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:37.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.027+0000 7efef3af9700 1 --2- 172.21.15.16:0/242560906 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efeec15d750 0x7efeec0cf920 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7efee8009820 tx=0x7efee8009b30 comp rx=0 tx=0).stop 2023-10-16T13:47:37.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.027+0000 7efef3af9700 1 -- 172.21.15.16:0/242560906 shutdown_connections 2023-10-16T13:47:37.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.027+0000 7efef3af9700 1 --2- 172.21.15.16:0/242560906 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efeec15d750 0x7efeec0cf920 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:37.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.027+0000 7efef3af9700 1 --2- 172.21.15.16:0/242560906 >> v2:172.21.15.49:3300/0 conn(0x7efeec15db30 0x7efeec06c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:37.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.027+0000 7efef3af9700 1 -- 172.21.15.16:0/242560906 >> 172.21.15.16:0/242560906 conn(0x7efeec0d14b0 msgr2=0x7efeec0d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:37.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.027+0000 7efef3af9700 1 -- 172.21.15.16:0/242560906 shutdown_connections 2023-10-16T13:47:37.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.027+0000 7efef3af9700 1 -- 172.21.15.16:0/242560906 wait complete. 2023-10-16T13:47:37.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.029+0000 7efef3af9700 1 Processor -- start 2023-10-16T13:47:37.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.029+0000 7efef3af9700 1 -- start start 2023-10-16T13:47:37.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.029+0000 7efef3af9700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efeec15d750 0x7efeec154b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:37.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.029+0000 7efef3af9700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efeec15db30 0x7efeec09e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:37.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.029+0000 7efef3af9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efeec161490 0x7efeec0b6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:37.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.029+0000 7efef3af9700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeec158c90 con 0x7efeec161490 2023-10-16T13:47:37.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.029+0000 7efef3af9700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeec1f3120 con 0x7efeec15db30 2023-10-16T13:47:37.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.029+0000 7efef3af9700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efeec1f3400 con 0x7efeec15d750 2023-10-16T13:47:37.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.029+0000 7efef2096700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efeec161490 0x7efeec0b6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:37.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.029+0000 7efef2096700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efeec161490 0x7efeec0b6190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55686/0 (socket says 172.21.15.16:55686) 2023-10-16T13:47:37.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.030+0000 7efef2096700 1 -- 172.21.15.16:0/40455427 learned_addr learned my addr 172.21.15.16:0/40455427 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:37.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.030+0000 7efef1094700 1 --2- 172.21.15.16:0/40455427 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efeec15db30 0x7efeec09e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:37.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.030+0000 7efef1895700 1 --2- 172.21.15.16:0/40455427 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efeec15d750 0x7efeec154b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:37.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.030+0000 7efef2096700 1 -- 172.21.15.16:0/40455427 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efeec15d750 msgr2=0x7efeec154b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:37.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.030+0000 7efef2096700 1 --2- 172.21.15.16:0/40455427 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efeec15d750 0x7efeec154b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:37.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.030+0000 7efef2096700 1 -- 172.21.15.16:0/40455427 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efeec15db30 msgr2=0x7efeec09e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:37.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.030+0000 7efef2096700 1 --2- 172.21.15.16:0/40455427 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efeec15db30 0x7efeec09e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:37.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.030+0000 7efef2096700 1 -- 172.21.15.16:0/40455427 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efedc0094a0 con 0x7efeec161490 2023-10-16T13:47:37.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.030+0000 7efef2096700 1 --2- 172.21.15.16:0/40455427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efeec161490 0x7efeec0b6190 secure :-1 s=READY pgs=2783 cs=0 l=1 rev1=1 crypto rx=0x7efee400caa0 tx=0x7efee400ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:37.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.030+0000 7efee27fc700 1 -- 172.21.15.16:0/40455427 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee400aa80 con 0x7efeec161490 2023-10-16T13:47:37.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.030+0000 7efee27fc700 1 -- 172.21.15.16:0/40455427 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efee400abe0 con 0x7efeec161490 2023-10-16T13:47:37.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.030+0000 7efee27fc700 1 -- 172.21.15.16:0/40455427 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee4012400 con 0x7efeec161490 2023-10-16T13:47:37.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.030+0000 7efef3af9700 1 -- 172.21.15.16:0/40455427 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efee8009430 con 0x7efeec161490 2023-10-16T13:47:37.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.030+0000 7efef3af9700 1 -- 172.21.15.16:0/40455427 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efeec1f39c0 con 0x7efeec161490 2023-10-16T13:47:37.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.031+0000 7efef3af9700 1 -- 172.21.15.16:0/40455427 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efeec04ebb0 con 0x7efeec161490 2023-10-16T13:47:37.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.032+0000 7efee27fc700 1 -- 172.21.15.16:0/40455427 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 148) v1 ==== 89433+0+0 (secure 0 0 0) 0x7efee401b3b0 con 0x7efeec161490 2023-10-16T13:47:37.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.032+0000 7efee27fc700 1 --2- 172.21.15.16:0/40455427 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7efed006b140 0x7efed006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:37.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.032+0000 7efee27fc700 1 -- 172.21.15.16:0/40455427 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(245..245 src has 1..245) v4 ==== 6157+0+0 (secure 0 0 0) 0x7efee4089920 con 0x7efeec161490 2023-10-16T13:47:37.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.032+0000 7efef1895700 1 --2- 172.21.15.16:0/40455427 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7efed006b140 0x7efed006d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:37.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.033+0000 7efef1895700 1 --2- 172.21.15.16:0/40455427 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7efed006b140 0x7efed006d600 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7efee8013040 tx=0x7efee8008070 comp rx=0 tx=0).ready entity=mgr.11886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:37.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.037+0000 7efee27fc700 1 -- 172.21.15.16:0/40455427 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7efee40544b0 con 0x7efeec161490 2023-10-16T13:47:37.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.242+0000 7efef3af9700 1 -- 172.21.15.16:0/40455427 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efeec06b5d0 con 0x7efeec161490 2023-10-16T13:47:37.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.243+0000 7efee27fc700 1 -- 172.21.15.16:0/40455427 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v245) v1 ==== 74+0+16881 (secure 0 0 0) 0x7efee4057ad0 con 0x7efeec161490 2023-10-16T13:47:37.246 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:47:37.246 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":245,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:47:25.413253+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2023-10-16T13:47:22.338349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"244","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2023-10-16T13:47:22.806327+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"245","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":243,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":243,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":243,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":243,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":243,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":243,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":243,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":243,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:47:37.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.245+0000 7efef3af9700 1 -- 172.21.15.16:0/40455427 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7efed006b140 msgr2=0x7efed006d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:37.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.245+0000 7efef3af9700 1 --2- 172.21.15.16:0/40455427 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7efed006b140 0x7efed006d600 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7efee8013040 tx=0x7efee8008070 comp rx=0 tx=0).stop 2023-10-16T13:47:37.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.245+0000 7efef3af9700 1 -- 172.21.15.16:0/40455427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efeec161490 msgr2=0x7efeec0b6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:37.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.245+0000 7efef3af9700 1 --2- 172.21.15.16:0/40455427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efeec161490 0x7efeec0b6190 secure :-1 s=READY pgs=2783 cs=0 l=1 rev1=1 crypto rx=0x7efee400caa0 tx=0x7efee400ce60 comp rx=0 tx=0).stop 2023-10-16T13:47:37.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.245+0000 7efef3af9700 1 -- 172.21.15.16:0/40455427 shutdown_connections 2023-10-16T13:47:37.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.245+0000 7efef3af9700 1 --2- 172.21.15.16:0/40455427 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7efed006b140 0x7efed006d600 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:37.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.245+0000 7efef3af9700 1 --2- 172.21.15.16:0/40455427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efeec161490 0x7efeec0b6190 unknown :-1 s=CLOSED pgs=2783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:37.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.245+0000 7efef3af9700 1 --2- 172.21.15.16:0/40455427 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efeec15d750 0x7efeec154b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:37.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.245+0000 7efef3af9700 1 --2- 172.21.15.16:0/40455427 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efeec15db30 0x7efeec09e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:37.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.245+0000 7efef3af9700 1 -- 172.21.15.16:0/40455427 >> 172.21.15.16:0/40455427 conn(0x7efeec0d14b0 msgr2=0x7efeec06bcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:37.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.245+0000 7efef3af9700 1 -- 172.21.15.16:0/40455427 shutdown_connections 2023-10-16T13:47:37.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.245+0000 7efef3af9700 1 -- 172.21.15.16:0/40455427 wait complete. 2023-10-16T13:47:37.260 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 31, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T13:47:22.338349+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '244', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 32, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T13:47:22.806327+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '245', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T13:47:37.261 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T13:47:37.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.387+0000 7f08880d6700 1 Processor -- start 2023-10-16T13:47:37.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.388+0000 7f08880d6700 1 -- start start 2023-10-16T13:47:37.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.388+0000 7f08880d6700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f088006a7e0 0x7f088006ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:37.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.388+0000 7f08880d6700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f088015ecb0 con 0x7f088015b2c0 2023-10-16T13:47:37.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.388+0000 7f08880d6700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f088015b6a0 con 0x7f088006e570 2023-10-16T13:47:37.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.388+0000 7f08880d6700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f088006b1b0 con 0x7f088006a7e0 2023-10-16T13:47:37.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.389+0000 7f0885e72700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f088015b2c0 0x7f0880161340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44964/0 (socket says 172.21.15.16:44964) 2023-10-16T13:47:37.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.389+0000 7f0885e72700 1 -- 172.21.15.16:0/4107563706 learned_addr learned my addr 172.21.15.16:0/4107563706 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:47:37.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.389+0000 7f0886673700 1 --2- 172.21.15.16:0/4107563706 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f088006a7e0 0x7f088006ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:37.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.389+0000 7f0884e70700 1 -- 172.21.15.16:0/4107563706 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2770785134 0 0) 0x7f088015ecb0 con 0x7f088015b2c0 2023-10-16T13:47:37.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.389+0000 7f0884e70700 1 -- 172.21.15.16:0/4107563706 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f086c0035b0 con 0x7f088015b2c0 2023-10-16T13:47:37.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.389+0000 7f0884e70700 1 -- 172.21.15.16:0/4107563706 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1330309164 0 0) 0x7f088015b6a0 con 0x7f088006e570 2023-10-16T13:47:37.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.389+0000 7f0884e70700 1 -- 172.21.15.16:0/4107563706 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f088015ecb0 con 0x7f088006e570 2023-10-16T13:47:37.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.389+0000 7f0884e70700 1 -- 172.21.15.16:0/4107563706 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1616457198 0 0) 0x7f086c0035b0 con 0x7f088015b2c0 2023-10-16T13:47:37.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.389+0000 7f0886673700 1 -- 172.21.15.16:0/4107563706 >> v1:172.21.15.49:6789/0 conn(0x7f088006e570 legacy=0x7f088015e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:37.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.389+0000 7f0886673700 1 -- 172.21.15.16:0/4107563706 >> v1:172.21.15.16:6789/0 conn(0x7f088015b2c0 legacy=0x7f0880161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:37.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.389+0000 7f0886673700 1 -- 172.21.15.16:0/4107563706 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0880161b20 con 0x7f088006a7e0 2023-10-16T13:47:37.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.389+0000 7f0886673700 1 --2- 172.21.15.16:0/4107563706 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f088006a7e0 0x7f088006ac70 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f0878009dc0 tx=0x7f087800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=56a25bfd20026d98 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:37.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.390+0000 7f0884e70700 1 -- 172.21.15.16:0/4107563706 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f087800e070 con 0x7f088006a7e0 2023-10-16T13:47:37.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.390+0000 7f0884e70700 1 -- 172.21.15.16:0/4107563706 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0878003a20 con 0x7f088006a7e0 2023-10-16T13:47:37.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.390+0000 7f0884e70700 1 -- 172.21.15.16:0/4107563706 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0874001670 con 0x7f088006a7e0 2023-10-16T13:47:37.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.390+0000 7f08880d6700 1 -- 172.21.15.16:0/4107563706 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f088006a7e0 msgr2=0x7f088006ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:37.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.390+0000 7f08880d6700 1 --2- 172.21.15.16:0/4107563706 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f088006a7e0 0x7f088006ac70 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f0878009dc0 tx=0x7f087800b330 comp rx=0 tx=0).stop 2023-10-16T13:47:37.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.390+0000 7f08880d6700 1 -- 172.21.15.16:0/4107563706 shutdown_connections 2023-10-16T13:47:37.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.390+0000 7f08880d6700 1 --2- 172.21.15.16:0/4107563706 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f088006a7e0 0x7f088006ac70 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:37.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.390+0000 7f08880d6700 1 -- 172.21.15.16:0/4107563706 >> 172.21.15.16:0/4107563706 conn(0x7f08800b1f40 msgr2=0x7f08800b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:37.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.390+0000 7f08880d6700 1 -- 172.21.15.16:0/4107563706 shutdown_connections 2023-10-16T13:47:37.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.390+0000 7f08880d6700 1 -- 172.21.15.16:0/4107563706 wait complete. 2023-10-16T13:47:37.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.391+0000 7f08880d6700 1 Processor -- start 2023-10-16T13:47:37.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.392+0000 7f08880d6700 1 -- start start 2023-10-16T13:47:37.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.392+0000 7f08880d6700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f088006a7e0 0x7f08800c5f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:37.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.392+0000 7f08880d6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f088006e570 0x7f08800c9670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:37.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.392+0000 7f08880d6700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f088015b2c0 0x7f088015d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:37.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.392+0000 7f08880d6700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08800b3aa0 con 0x7f088006e570 2023-10-16T13:47:37.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.392+0000 7f08880d6700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08800ca0e0 con 0x7f088006a7e0 2023-10-16T13:47:37.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.392+0000 7f08880d6700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08800ca3c0 con 0x7f088015b2c0 2023-10-16T13:47:37.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.392+0000 7f0885671700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f088006e570 0x7f08800c9670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:37.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.392+0000 7f0885671700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f088006e570 0x7f08800c9670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55698/0 (socket says 172.21.15.16:55698) 2023-10-16T13:47:37.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.392+0000 7f0885671700 1 -- 172.21.15.16:0/899945090 learned_addr learned my addr 172.21.15.16:0/899945090 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:37.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.393+0000 7f0885e72700 1 --2- 172.21.15.16:0/899945090 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f088006a7e0 0x7f08800c5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:37.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.393+0000 7f0886673700 1 --2- 172.21.15.16:0/899945090 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f088015b2c0 0x7f088015d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:37.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.393+0000 7f0885671700 1 -- 172.21.15.16:0/899945090 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f088015b2c0 msgr2=0x7f088015d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:37.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.393+0000 7f0885671700 1 --2- 172.21.15.16:0/899945090 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f088015b2c0 0x7f088015d220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:37.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.393+0000 7f0885671700 1 -- 172.21.15.16:0/899945090 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f088006a7e0 msgr2=0x7f08800c5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:37.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.393+0000 7f0885671700 1 --2- 172.21.15.16:0/899945090 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f088006a7e0 0x7f08800c5f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:37.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.393+0000 7f0885671700 1 -- 172.21.15.16:0/899945090 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0878009a70 con 0x7f088006e570 2023-10-16T13:47:37.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.393+0000 7f0885671700 1 --2- 172.21.15.16:0/899945090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f088006e570 0x7f08800c9670 secure :-1 s=READY pgs=2785 cs=0 l=1 rev1=1 crypto rx=0x7f087c00c520 tx=0x7f087c00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:37.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.393+0000 7f08727fc700 1 -- 172.21.15.16:0/899945090 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f087c0052d0 con 0x7f088006e570 2023-10-16T13:47:37.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.393+0000 7f08727fc700 1 -- 172.21.15.16:0/899945090 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f087c005430 con 0x7f088006e570 2023-10-16T13:47:37.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.393+0000 7f08880d6700 1 -- 172.21.15.16:0/899945090 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08800b3730 con 0x7f088006e570 2023-10-16T13:47:37.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.393+0000 7f08727fc700 1 -- 172.21.15.16:0/899945090 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f087c0038c0 con 0x7f088006e570 2023-10-16T13:47:37.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.393+0000 7f08880d6700 1 -- 172.21.15.16:0/899945090 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08800ca6a0 con 0x7f088006e570 2023-10-16T13:47:37.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.394+0000 7f08880d6700 1 -- 172.21.15.16:0/899945090 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08800b3c10 con 0x7f088006e570 2023-10-16T13:47:37.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.395+0000 7f08727fc700 1 -- 172.21.15.16:0/899945090 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 148) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f087c003a20 con 0x7f088006e570 2023-10-16T13:47:37.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.395+0000 7f08727fc700 1 --2- 172.21.15.16:0/899945090 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f086406b140 0x7f086406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:37.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.395+0000 7f0885e72700 1 --2- 172.21.15.16:0/899945090 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f086406b140 0x7f086406d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:37.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.395+0000 7f08727fc700 1 -- 172.21.15.16:0/899945090 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(245..245 src has 1..245) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f087c089080 con 0x7f088006e570 2023-10-16T13:47:37.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.395+0000 7f0885e72700 1 --2- 172.21.15.16:0/899945090 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f086406b140 0x7f086406d600 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0874000f60 tx=0x7f0874001c00 comp rx=0 tx=0).ready entity=mgr.11886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:37.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.400+0000 7f08727fc700 1 -- 172.21.15.16:0/899945090 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f087c053ce0 con 0x7f088006e570 2023-10-16T13:47:37.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.604+0000 7f08880d6700 1 -- 172.21.15.16:0/899945090 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f088004ebb0 con 0x7f088006e570 2023-10-16T13:47:37.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.605+0000 7f08727fc700 1 -- 172.21.15.16:0/899945090 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v245) v1 ==== 93+0+10 (secure 0 0 0) 0x7f087c057300 con 0x7f088006e570 2023-10-16T13:47:37.607 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T13:47:37.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.607+0000 7f08880d6700 1 -- 172.21.15.16:0/899945090 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f086406b140 msgr2=0x7f086406d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:37.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.607+0000 7f08880d6700 1 --2- 172.21.15.16:0/899945090 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f086406b140 0x7f086406d600 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0874000f60 tx=0x7f0874001c00 comp rx=0 tx=0).stop 2023-10-16T13:47:37.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.607+0000 7f08880d6700 1 -- 172.21.15.16:0/899945090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f088006e570 msgr2=0x7f08800c9670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:37.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.607+0000 7f08880d6700 1 --2- 172.21.15.16:0/899945090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f088006e570 0x7f08800c9670 secure :-1 s=READY pgs=2785 cs=0 l=1 rev1=1 crypto rx=0x7f087c00c520 tx=0x7f087c00c8e0 comp rx=0 tx=0).stop 2023-10-16T13:47:37.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.607+0000 7f08880d6700 1 -- 172.21.15.16:0/899945090 shutdown_connections 2023-10-16T13:47:37.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.607+0000 7f08880d6700 1 --2- 172.21.15.16:0/899945090 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f086406b140 0x7f086406d600 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:37.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.607+0000 7f08880d6700 1 --2- 172.21.15.16:0/899945090 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f088015b2c0 0x7f088015d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:37.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.607+0000 7f08880d6700 1 --2- 172.21.15.16:0/899945090 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f088006a7e0 0x7f08800c5f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:37.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.607+0000 7f08880d6700 1 --2- 172.21.15.16:0/899945090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f088006e570 0x7f08800c9670 unknown :-1 s=CLOSED pgs=2785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:37.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.607+0000 7f08880d6700 1 -- 172.21.15.16:0/899945090 >> 172.21.15.16:0/899945090 conn(0x7f08800b1f40 msgr2=0x7f08800d0030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:37.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.607+0000 7f08880d6700 1 -- 172.21.15.16:0/899945090 shutdown_connections 2023-10-16T13:47:37.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.607+0000 7f08880d6700 1 -- 172.21.15.16:0/899945090 wait complete. 2023-10-16T13:47:37.622 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T13:47:37.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.747+0000 7fdd795bb700 1 Processor -- start 2023-10-16T13:47:37.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.747+0000 7fdd795bb700 1 -- start start 2023-10-16T13:47:37.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.748+0000 7fdd795bb700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdd7406a7e0 0x7fdd7406ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:37.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.748+0000 7fdd795bb700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd7415ecb0 con 0x7fdd7415b2c0 2023-10-16T13:47:37.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.748+0000 7fdd795bb700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd7415b6a0 con 0x7fdd7406e570 2023-10-16T13:47:37.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.748+0000 7fdd795bb700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd7406b1b0 con 0x7fdd7406a7e0 2023-10-16T13:47:37.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.748+0000 7fdd72ffd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fdd7415b2c0 0x7fdd74161340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44978/0 (socket says 172.21.15.16:44978) 2023-10-16T13:47:37.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.748+0000 7fdd72ffd700 1 -- 172.21.15.16:0/94521652 learned_addr learned my addr 172.21.15.16:0/94521652 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:47:37.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.749+0000 7fdd737fe700 1 --2- 172.21.15.16:0/94521652 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdd7406a7e0 0x7fdd7406ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:37.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.749+0000 7fdd71ffb700 1 -- 172.21.15.16:0/94521652 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2894437850 0 0) 0x7fdd7415ecb0 con 0x7fdd7415b2c0 2023-10-16T13:47:37.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.749+0000 7fdd71ffb700 1 -- 172.21.15.16:0/94521652 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd600035b0 con 0x7fdd7415b2c0 2023-10-16T13:47:37.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.749+0000 7fdd71ffb700 1 -- 172.21.15.16:0/94521652 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2193720594 0 0) 0x7fdd600035b0 con 0x7fdd7415b2c0 2023-10-16T13:47:37.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.749+0000 7fdd71ffb700 1 -- 172.21.15.16:0/94521652 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdd7406a7e0 msgr2=0x7fdd7406ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:37.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.749+0000 7fdd71ffb700 1 --2- 172.21.15.16:0/94521652 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdd7406a7e0 0x7fdd7406ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:37.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.749+0000 7fdd71ffb700 1 -- 172.21.15.16:0/94521652 >> v1:172.21.15.49:6789/0 conn(0x7fdd7406e570 legacy=0x7fdd7415e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:37.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.749+0000 7fdd71ffb700 1 -- 172.21.15.16:0/94521652 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd74161b20 con 0x7fdd7415b2c0 2023-10-16T13:47:37.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.749+0000 7fdd71ffb700 1 -- 172.21.15.16:0/94521652 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7fdd64001670 con 0x7fdd7415b2c0 2023-10-16T13:47:37.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.749+0000 7fdd71ffb700 1 -- 172.21.15.16:0/94521652 >> v1:172.21.15.16:6789/0 conn(0x7fdd7415b2c0 legacy=0x7fdd74161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:37.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.749+0000 7fdd71ffb700 1 --2- 172.21.15.16:0/94521652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd60003a30 0x7fdd60003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:37.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.749+0000 7fdd71ffb700 1 --2- 172.21.15.16:0/94521652 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdd7406a7e0 0x7fdd60004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:37.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.749+0000 7fdd71ffb700 1 --2- 172.21.15.16:0/94521652 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdd60004d90 0x7fdd60005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:37.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.749+0000 7fdd71ffb700 1 -- 172.21.15.16:0/94521652 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd7406b1b0 con 0x7fdd60003a30 2023-10-16T13:47:37.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.749+0000 7fdd71ffb700 1 -- 172.21.15.16:0/94521652 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd6000bac0 con 0x7fdd60004d90 2023-10-16T13:47:37.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.749+0000 7fdd71ffb700 1 -- 172.21.15.16:0/94521652 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd6000bc80 con 0x7fdd7406a7e0 2023-10-16T13:47:37.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.749+0000 7fdd72ffd700 1 --2- 172.21.15.16:0/94521652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd60003a30 0x7fdd60003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:37.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.749+0000 7fdd737fe700 1 --2- 172.21.15.16:0/94521652 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdd60004d90 0x7fdd60005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:37.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.749+0000 7fdd727fc700 1 --2- 172.21.15.16:0/94521652 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdd7406a7e0 0x7fdd60004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:37.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.749+0000 7fdd72ffd700 1 -- 172.21.15.16:0/94521652 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdd7406a7e0 msgr2=0x7fdd60004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:37.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.749+0000 7fdd72ffd700 1 --2- 172.21.15.16:0/94521652 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdd7406a7e0 0x7fdd60004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:37.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.749+0000 7fdd72ffd700 1 -- 172.21.15.16:0/94521652 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdd60004d90 msgr2=0x7fdd60005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:37.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.749+0000 7fdd72ffd700 1 --2- 172.21.15.16:0/94521652 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdd60004d90 0x7fdd60005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:37.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.749+0000 7fdd72ffd700 1 -- 172.21.15.16:0/94521652 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd600035b0 con 0x7fdd60003a30 2023-10-16T13:47:37.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.750+0000 7fdd737fe700 1 --2- 172.21.15.16:0/94521652 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdd60004d90 0x7fdd60005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:47:37.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.750+0000 7fdd72ffd700 1 --2- 172.21.15.16:0/94521652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd60003a30 0x7fdd60003e10 secure :-1 s=READY pgs=2787 cs=0 l=1 rev1=1 crypto rx=0x7fdd6400c2f0 tx=0x7fdd60005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbc00184064ad086 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:37.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.750+0000 7fdd71ffb700 1 -- 172.21.15.16:0/94521652 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd6400ce30 con 0x7fdd60003a30 2023-10-16T13:47:37.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.750+0000 7fdd71ffb700 1 -- 172.21.15.16:0/94521652 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd64005960 con 0x7fdd60003a30 2023-10-16T13:47:37.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.750+0000 7fdd71ffb700 1 -- 172.21.15.16:0/94521652 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd64005cf0 con 0x7fdd60003a30 2023-10-16T13:47:37.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.750+0000 7fdd795bb700 1 -- 172.21.15.16:0/94521652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd60003a30 msgr2=0x7fdd60003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:37.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.750+0000 7fdd795bb700 1 --2- 172.21.15.16:0/94521652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd60003a30 0x7fdd60003e10 secure :-1 s=READY pgs=2787 cs=0 l=1 rev1=1 crypto rx=0x7fdd6400c2f0 tx=0x7fdd60005880 comp rx=0 tx=0).stop 2023-10-16T13:47:37.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.750+0000 7fdd737fe700 1 -- 172.21.15.16:0/94521652 reap_dead start 2023-10-16T13:47:37.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.750+0000 7fdd795bb700 1 -- 172.21.15.16:0/94521652 shutdown_connections 2023-10-16T13:47:37.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.750+0000 7fdd795bb700 1 -- 172.21.15.16:0/94521652 >> 172.21.15.16:0/94521652 conn(0x7fdd740b1f40 msgr2=0x7fdd740b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:37.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.750+0000 7fdd795bb700 1 -- 172.21.15.16:0/94521652 shutdown_connections 2023-10-16T13:47:37.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.750+0000 7fdd795bb700 1 -- 172.21.15.16:0/94521652 wait complete. 2023-10-16T13:47:37.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.752+0000 7fdd795bb700 1 Processor -- start 2023-10-16T13:47:37.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.752+0000 7fdd795bb700 1 -- start start 2023-10-16T13:47:37.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.752+0000 7fdd795bb700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdd60003a30 0x7fdd741594d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:37.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.753+0000 7fdd795bb700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdd740c7590 0x7fdd74160470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:37.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.753+0000 7fdd795bb700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd740c7a10 0x7fdd741609b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:37.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.753+0000 7fdd795bb700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd740c9010 con 0x7fdd740c7a10 2023-10-16T13:47:37.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.753+0000 7fdd795bb700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd740c8b80 con 0x7fdd740c7590 2023-10-16T13:47:37.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.753+0000 7fdd795bb700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd740c7f70 con 0x7fdd60003a30 2023-10-16T13:47:37.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.753+0000 7fdd737fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd740c7a10 0x7fdd741609b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:37.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.753+0000 7fdd737fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd740c7a10 0x7fdd741609b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55718/0 (socket says 172.21.15.16:55718) 2023-10-16T13:47:37.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.753+0000 7fdd737fe700 1 -- 172.21.15.16:0/3298191708 learned_addr learned my addr 172.21.15.16:0/3298191708 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:37.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.753+0000 7fdd727fc700 1 --2- 172.21.15.16:0/3298191708 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdd740c7590 0x7fdd74160470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:37.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.753+0000 7fdd72ffd700 1 --2- 172.21.15.16:0/3298191708 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdd60003a30 0x7fdd741594d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:37.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.753+0000 7fdd737fe700 1 -- 172.21.15.16:0/3298191708 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdd60003a30 msgr2=0x7fdd741594d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:37.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.753+0000 7fdd737fe700 1 --2- 172.21.15.16:0/3298191708 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdd60003a30 0x7fdd741594d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:37.768 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.753+0000 7fdd737fe700 1 -- 172.21.15.16:0/3298191708 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdd740c7590 msgr2=0x7fdd74160470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:37.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.753+0000 7fdd737fe700 1 --2- 172.21.15.16:0/3298191708 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdd740c7590 0x7fdd74160470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:37.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.753+0000 7fdd737fe700 1 -- 172.21.15.16:0/3298191708 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd740c8250 con 0x7fdd740c7a10 2023-10-16T13:47:37.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.753+0000 7fdd737fe700 1 --2- 172.21.15.16:0/3298191708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd740c7a10 0x7fdd741609b0 secure :-1 s=READY pgs=2788 cs=0 l=1 rev1=1 crypto rx=0x7fdd7406b430 tx=0x7fdd6c0104a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:37.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.753+0000 7fdd6b7fe700 1 -- 172.21.15.16:0/3298191708 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd6c01b070 con 0x7fdd740c7a10 2023-10-16T13:47:37.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.753+0000 7fdd6b7fe700 1 -- 172.21.15.16:0/3298191708 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd6c00f040 con 0x7fdd740c7a10 2023-10-16T13:47:37.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.753+0000 7fdd795bb700 1 -- 172.21.15.16:0/3298191708 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd740c8530 con 0x7fdd740c7a10 2023-10-16T13:47:37.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.753+0000 7fdd6b7fe700 1 -- 172.21.15.16:0/3298191708 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd6c00e050 con 0x7fdd740c7a10 2023-10-16T13:47:37.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.753+0000 7fdd795bb700 1 -- 172.21.15.16:0/3298191708 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd74159a10 con 0x7fdd740c7a10 2023-10-16T13:47:37.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.754+0000 7fdd795bb700 1 -- 172.21.15.16:0/3298191708 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd740c8d10 con 0x7fdd740c7a10 2023-10-16T13:47:37.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.755+0000 7fdd6b7fe700 1 -- 172.21.15.16:0/3298191708 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 148) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fdd6c015440 con 0x7fdd740c7a10 2023-10-16T13:47:37.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.755+0000 7fdd6b7fe700 1 --2- 172.21.15.16:0/3298191708 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fdd5406b140 0x7fdd5406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:37.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.755+0000 7fdd6b7fe700 1 -- 172.21.15.16:0/3298191708 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(245..245 src has 1..245) v4 ==== 6157+0+0 (secure 0 0 0) 0x7fdd6c089c40 con 0x7fdd740c7a10 2023-10-16T13:47:37.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.755+0000 7fdd72ffd700 1 --2- 172.21.15.16:0/3298191708 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fdd5406b140 0x7fdd5406d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:37.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.756+0000 7fdd72ffd700 1 --2- 172.21.15.16:0/3298191708 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fdd5406b140 0x7fdd5406d600 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fdd64008fd0 tx=0x7fdd64007000 comp rx=0 tx=0).ready entity=mgr.11886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:37.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.760+0000 7fdd6b7fe700 1 -- 172.21.15.16:0/3298191708 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdd6c054750 con 0x7fdd740c7a10 2023-10-16T13:47:37.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.962+0000 7fdd795bb700 1 -- 172.21.15.16:0/3298191708 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fdd74159ba0 con 0x7fdd740c7a10 2023-10-16T13:47:37.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.963+0000 7fdd6b7fe700 1 -- 172.21.15.16:0/3298191708 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v245) v1 ==== 104+0+11 (secure 0 0 0) 0x7fdd74159ba0 con 0x7fdd740c7a10 2023-10-16T13:47:37.966 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-10-16T13:47:37.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.965+0000 7fdd795bb700 1 -- 172.21.15.16:0/3298191708 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fdd5406b140 msgr2=0x7fdd5406d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:37.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.965+0000 7fdd795bb700 1 --2- 172.21.15.16:0/3298191708 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fdd5406b140 0x7fdd5406d600 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fdd64008fd0 tx=0x7fdd64007000 comp rx=0 tx=0).stop 2023-10-16T13:47:37.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.965+0000 7fdd795bb700 1 -- 172.21.15.16:0/3298191708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd740c7a10 msgr2=0x7fdd741609b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:37.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.965+0000 7fdd795bb700 1 --2- 172.21.15.16:0/3298191708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd740c7a10 0x7fdd741609b0 secure :-1 s=READY pgs=2788 cs=0 l=1 rev1=1 crypto rx=0x7fdd7406b430 tx=0x7fdd6c0104a0 comp rx=0 tx=0).stop 2023-10-16T13:47:37.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.965+0000 7fdd795bb700 1 -- 172.21.15.16:0/3298191708 shutdown_connections 2023-10-16T13:47:37.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.965+0000 7fdd795bb700 1 --2- 172.21.15.16:0/3298191708 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fdd5406b140 0x7fdd5406d600 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:37.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.965+0000 7fdd795bb700 1 --2- 172.21.15.16:0/3298191708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd740c7a10 0x7fdd741609b0 unknown :-1 s=CLOSED pgs=2788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:37.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.965+0000 7fdd795bb700 1 --2- 172.21.15.16:0/3298191708 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdd60003a30 0x7fdd741594d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:37.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.965+0000 7fdd795bb700 1 --2- 172.21.15.16:0/3298191708 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdd740c7590 0x7fdd74160470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:37.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.965+0000 7fdd795bb700 1 -- 172.21.15.16:0/3298191708 >> 172.21.15.16:0/3298191708 conn(0x7fdd740b1f40 msgr2=0x7fdd740b4f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:37.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.965+0000 7fdd795bb700 1 -- 172.21.15.16:0/3298191708 shutdown_connections 2023-10-16T13:47:37.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:37.965+0000 7fdd795bb700 1 -- 172.21.15.16:0/3298191708 wait complete. 2023-10-16T13:47:37.980 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T13:47:38.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.106+0000 7ff742b4d700 1 Processor -- start 2023-10-16T13:47:38.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.106+0000 7ff742b4d700 1 -- start start 2023-10-16T13:47:38.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.107+0000 7ff742b4d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff73c15b070 0x7ff73c15b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:38.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.107+0000 7ff742b4d700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7ff73c06e170 0x7ff73c0cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:38.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.107+0000 7ff742b4d700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff73c0cef70 con 0x7ff73c0cb880 2023-10-16T13:47:38.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.107+0000 7ff742b4d700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff73c0ce360 con 0x7ff73c15b070 2023-10-16T13:47:38.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.107+0000 7ff742b4d700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff73c0ce640 con 0x7ff73c06e170 2023-10-16T13:47:38.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.107+0000 7ff7408e9700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff73c15b070 0x7ff73c15b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:38.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.107+0000 7ff7408e9700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff73c15b070 0x7ff73c15b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55722/0 (socket says 172.21.15.16:55722) 2023-10-16T13:47:38.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.107+0000 7ff7408e9700 1 -- 172.21.15.16:0/1794313529 learned_addr learned my addr 172.21.15.16:0/1794313529 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:38.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.107+0000 7ff7410ea700 1 --2- 172.21.15.16:0/1794313529 >> v2:172.21.15.49:3300/0 conn(0x7ff73c06e170 0x7ff73c0cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:38.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.107+0000 7ff7408e9700 1 -- 172.21.15.16:0/1794313529 >> v2:172.21.15.49:3300/0 conn(0x7ff73c06e170 msgr2=0x7ff73c0cbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:38.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.107+0000 7ff7408e9700 1 --2- 172.21.15.16:0/1794313529 >> v2:172.21.15.49:3300/0 conn(0x7ff73c06e170 0x7ff73c0cbd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:38.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.107+0000 7ff7408e9700 1 -- 172.21.15.16:0/1794313529 >> v1:172.21.15.16:6789/0 conn(0x7ff73c0cb880 legacy=0x7ff73c06da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:38.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.107+0000 7ff7408e9700 1 -- 172.21.15.16:0/1794313529 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff73c0ce920 con 0x7ff73c15b070 2023-10-16T13:47:38.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.107+0000 7ff7408e9700 1 --2- 172.21.15.16:0/1794313529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff73c15b070 0x7ff73c15b450 secure :-1 s=READY pgs=2790 cs=0 l=1 rev1=1 crypto rx=0x7ff73c15aad0 tx=0x7ff730009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6701ea87419291ed server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:38.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.108+0000 7ff73b7fe700 1 -- 172.21.15.16:0/1794313529 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff730011830 con 0x7ff73c15b070 2023-10-16T13:47:38.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.108+0000 7ff73b7fe700 1 -- 172.21.15.16:0/1794313529 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff730011990 con 0x7ff73c15b070 2023-10-16T13:47:38.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.108+0000 7ff73b7fe700 1 -- 172.21.15.16:0/1794313529 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff730011d20 con 0x7ff73c15b070 2023-10-16T13:47:38.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.108+0000 7ff742b4d700 1 -- 172.21.15.16:0/1794313529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff73c15b070 msgr2=0x7ff73c15b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:38.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.108+0000 7ff742b4d700 1 --2- 172.21.15.16:0/1794313529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff73c15b070 0x7ff73c15b450 secure :-1 s=READY pgs=2790 cs=0 l=1 rev1=1 crypto rx=0x7ff73c15aad0 tx=0x7ff730009bd0 comp rx=0 tx=0).stop 2023-10-16T13:47:38.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.108+0000 7ff742b4d700 1 -- 172.21.15.16:0/1794313529 shutdown_connections 2023-10-16T13:47:38.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.108+0000 7ff742b4d700 1 --2- 172.21.15.16:0/1794313529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff73c15b070 0x7ff73c15b450 unknown :-1 s=CLOSED pgs=2790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:38.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.108+0000 7ff742b4d700 1 --2- 172.21.15.16:0/1794313529 >> v2:172.21.15.49:3300/0 conn(0x7ff73c06e170 0x7ff73c0cbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:38.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.108+0000 7ff742b4d700 1 -- 172.21.15.16:0/1794313529 >> 172.21.15.16:0/1794313529 conn(0x7ff73c0b1f40 msgr2=0x7ff73c0b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:38.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.108+0000 7ff742b4d700 1 -- 172.21.15.16:0/1794313529 shutdown_connections 2023-10-16T13:47:38.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.108+0000 7ff742b4d700 1 -- 172.21.15.16:0/1794313529 wait complete. 2023-10-16T13:47:38.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.109+0000 7ff742b4d700 1 Processor -- start 2023-10-16T13:47:38.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.109+0000 7ff742b4d700 1 -- start start 2023-10-16T13:47:38.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.110+0000 7ff742b4d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff73c06e170 0x7ff73c0ca3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:38.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.110+0000 7ff742b4d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff73c0cb880 0x7ff73c0b00d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:38.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.110+0000 7ff742b4d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff73c15b070 0x7ff73c0c59c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:38.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.110+0000 7ff742b4d700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff73c0c43c0 con 0x7ff73c15b070 2023-10-16T13:47:38.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.110+0000 7ff742b4d700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff73c0c46a0 con 0x7ff73c0cb880 2023-10-16T13:47:38.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.110+0000 7ff742b4d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff73c0c4980 con 0x7ff73c06e170 2023-10-16T13:47:38.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.110+0000 7ff7410ea700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff73c15b070 0x7ff73c0c59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:38.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.111+0000 7ff7410ea700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff73c15b070 0x7ff73c0c59c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55732/0 (socket says 172.21.15.16:55732) 2023-10-16T13:47:38.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.111+0000 7ff7410ea700 1 -- 172.21.15.16:0/4231908793 learned_addr learned my addr 172.21.15.16:0/4231908793 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:38.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.111+0000 7ff73bfff700 1 --2- 172.21.15.16:0/4231908793 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff73c0cb880 0x7ff73c0b00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:38.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.111+0000 7ff7408e9700 1 --2- 172.21.15.16:0/4231908793 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff73c06e170 0x7ff73c0ca3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:38.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.111+0000 7ff7410ea700 1 -- 172.21.15.16:0/4231908793 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff73c06e170 msgr2=0x7ff73c0ca3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:38.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.111+0000 7ff7410ea700 1 --2- 172.21.15.16:0/4231908793 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff73c06e170 0x7ff73c0ca3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:38.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.111+0000 7ff7410ea700 1 -- 172.21.15.16:0/4231908793 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff73c0cb880 msgr2=0x7ff73c0b00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:38.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.111+0000 7ff7410ea700 1 --2- 172.21.15.16:0/4231908793 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff73c0cb880 0x7ff73c0b00d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:38.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.111+0000 7ff7410ea700 1 -- 172.21.15.16:0/4231908793 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff730009550 con 0x7ff73c15b070 2023-10-16T13:47:38.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.111+0000 7ff7408e9700 1 --2- 172.21.15.16:0/4231908793 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff73c06e170 0x7ff73c0ca3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:47:38.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.111+0000 7ff7410ea700 1 --2- 172.21.15.16:0/4231908793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff73c15b070 0x7ff73c0c59c0 secure :-1 s=READY pgs=2791 cs=0 l=1 rev1=1 crypto rx=0x7ff73400ef90 tx=0x7ff73400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:38.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.111+0000 7ff7397fa700 1 -- 172.21.15.16:0/4231908793 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff734009ca0 con 0x7ff73c15b070 2023-10-16T13:47:38.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.111+0000 7ff7397fa700 1 -- 172.21.15.16:0/4231908793 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff73400f040 con 0x7ff73c15b070 2023-10-16T13:47:38.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.111+0000 7ff7397fa700 1 -- 172.21.15.16:0/4231908793 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff734014480 con 0x7ff73c15b070 2023-10-16T13:47:38.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.111+0000 7ff742b4d700 1 -- 172.21.15.16:0/4231908793 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff73c0c4c60 con 0x7ff73c15b070 2023-10-16T13:47:38.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.111+0000 7ff742b4d700 1 -- 172.21.15.16:0/4231908793 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff73c0ca900 con 0x7ff73c15b070 2023-10-16T13:47:38.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.112+0000 7ff742b4d700 1 -- 172.21.15.16:0/4231908793 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff73c0b4080 con 0x7ff73c15b070 2023-10-16T13:47:38.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.113+0000 7ff7397fa700 1 -- 172.21.15.16:0/4231908793 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 148) v1 ==== 89433+0+0 (secure 0 0 0) 0x7ff7340145e0 con 0x7ff73c15b070 2023-10-16T13:47:38.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.113+0000 7ff7397fa700 1 --2- 172.21.15.16:0/4231908793 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7ff72006b140 0x7ff72006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:38.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.113+0000 7ff7397fa700 1 -- 172.21.15.16:0/4231908793 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(245..245 src has 1..245) v4 ==== 6157+0+0 (secure 0 0 0) 0x7ff73408a700 con 0x7ff73c15b070 2023-10-16T13:47:38.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.113+0000 7ff7408e9700 1 --2- 172.21.15.16:0/4231908793 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7ff72006b140 0x7ff72006d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:38.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.114+0000 7ff7408e9700 1 --2- 172.21.15.16:0/4231908793 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7ff72006b140 0x7ff72006d600 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff730014040 tx=0x7ff730008030 comp rx=0 tx=0).ready entity=mgr.11886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:38.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.118+0000 7ff7397fa700 1 -- 172.21.15.16:0/4231908793 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff73c0b4080 con 0x7ff73c15b070 2023-10-16T13:47:38.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.321+0000 7ff742b4d700 1 -- 172.21.15.16:0/4231908793 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff73c0b4080 con 0x7ff73c15b070 2023-10-16T13:47:38.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.322+0000 7ff7397fa700 1 -- 172.21.15.16:0/4231908793 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v245) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff73c0b4080 con 0x7ff73c15b070 2023-10-16T13:47:38.324 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-10-16T13:47:38.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.324+0000 7ff742b4d700 1 -- 172.21.15.16:0/4231908793 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7ff72006b140 msgr2=0x7ff72006d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:38.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.324+0000 7ff742b4d700 1 --2- 172.21.15.16:0/4231908793 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7ff72006b140 0x7ff72006d600 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff730014040 tx=0x7ff730008030 comp rx=0 tx=0).stop 2023-10-16T13:47:38.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.324+0000 7ff742b4d700 1 -- 172.21.15.16:0/4231908793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff73c15b070 msgr2=0x7ff73c0c59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:38.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.324+0000 7ff742b4d700 1 --2- 172.21.15.16:0/4231908793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff73c15b070 0x7ff73c0c59c0 secure :-1 s=READY pgs=2791 cs=0 l=1 rev1=1 crypto rx=0x7ff73400ef90 tx=0x7ff73400c3a0 comp rx=0 tx=0).stop 2023-10-16T13:47:38.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.324+0000 7ff742b4d700 1 -- 172.21.15.16:0/4231908793 shutdown_connections 2023-10-16T13:47:38.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.324+0000 7ff742b4d700 1 --2- 172.21.15.16:0/4231908793 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7ff72006b140 0x7ff72006d600 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:38.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.324+0000 7ff742b4d700 1 --2- 172.21.15.16:0/4231908793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff73c15b070 0x7ff73c0c59c0 unknown :-1 s=CLOSED pgs=2791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:38.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.324+0000 7ff742b4d700 1 --2- 172.21.15.16:0/4231908793 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff73c06e170 0x7ff73c0ca3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:38.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.324+0000 7ff742b4d700 1 --2- 172.21.15.16:0/4231908793 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff73c0cb880 0x7ff73c0b00d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:38.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.324+0000 7ff742b4d700 1 -- 172.21.15.16:0/4231908793 >> 172.21.15.16:0/4231908793 conn(0x7ff73c0b1f40 msgr2=0x7ff73c0b1760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:38.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.324+0000 7ff742b4d700 1 -- 172.21.15.16:0/4231908793 shutdown_connections 2023-10-16T13:47:38.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.324+0000 7ff742b4d700 1 -- 172.21.15.16:0/4231908793 wait complete. 2023-10-16T13:47:38.339 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-16T13:47:38.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.461+0000 7f0f3b8e3700 1 Processor -- start 2023-10-16T13:47:38.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.462+0000 7f0f3b8e3700 1 -- start start 2023-10-16T13:47:38.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.462+0000 7f0f3b8e3700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f340d0ae0 0x7f0f340cef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:38.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.462+0000 7f0f3b8e3700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f0f340ce010 0x7f0f340cc1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:38.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.462+0000 7f0f3b8e3700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f34072a50 con 0x7f0f3415a5c0 2023-10-16T13:47:38.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.462+0000 7f0f3b8e3700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f340cf440 con 0x7f0f340ce010 2023-10-16T13:47:38.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.462+0000 7f0f3b8e3700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f340cba30 con 0x7f0f340d0ae0 2023-10-16T13:47:38.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.463+0000 7f0f38e7e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f340d0ae0 0x7f0f340cef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:38.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.463+0000 7f0f39e80700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f0f340ce010 0x7f0f340cc1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:38.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.463+0000 7f0f38e7e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f340d0ae0 0x7f0f340cef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:43996/0 (socket says 172.21.15.16:43996) 2023-10-16T13:47:38.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.463+0000 7f0f39e80700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f0f340ce010 0x7f0f340cc1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:54802/0 (socket says 172.21.15.16:54802) 2023-10-16T13:47:38.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.463+0000 7f0f38e7e700 1 -- 172.21.15.16:0/2024489744 learned_addr learned my addr 172.21.15.16:0/2024489744 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:38.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.463+0000 7f0f3967f700 1 --1- 172.21.15.16:0/2024489744 >> v1:172.21.15.49:6789/0 conn(0x7f0f3415a5c0 0x7f0f340d03d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:52192/0 (socket says 172.21.15.16:52192) 2023-10-16T13:47:38.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.463+0000 7f0f27fff700 1 -- 172.21.15.16:0/2024489744 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3594456975 0 0) 0x7f0f34072a50 con 0x7f0f3415a5c0 2023-10-16T13:47:38.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.463+0000 7f0f27fff700 1 -- 172.21.15.16:0/2024489744 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f200035b0 con 0x7f0f3415a5c0 2023-10-16T13:47:38.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.463+0000 7f0f39e80700 1 -- 172.21.15.16:0/2024489744 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f340d0ae0 msgr2=0x7f0f340cef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:38.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.463+0000 7f0f39e80700 1 --2- 172.21.15.16:0/2024489744 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f340d0ae0 0x7f0f340cef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:38.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.463+0000 7f0f39e80700 1 -- 172.21.15.16:0/2024489744 >> v1:172.21.15.49:6789/0 conn(0x7f0f3415a5c0 legacy=0x7f0f340d03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:38.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.463+0000 7f0f39e80700 1 -- 172.21.15.16:0/2024489744 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f340cf630 con 0x7f0f340ce010 2023-10-16T13:47:38.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.463+0000 7f0f38e7e700 1 --2- 172.21.15.16:0/2024489744 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f340d0ae0 0x7f0f340cef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:47:38.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.464+0000 7f0f39e80700 1 --2- 172.21.15.16:0/2024489744 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f340ce010 0x7f0f340cc1f0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f0f2c009d90 tx=0x7f0f2c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6cad228d82bc1d1f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:38.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.464+0000 7f0f27fff700 1 -- 172.21.15.16:0/2024489744 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f2c00e070 con 0x7f0f340ce010 2023-10-16T13:47:38.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.464+0000 7f0f27fff700 1 -- 172.21.15.16:0/2024489744 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f2c00bb20 con 0x7f0f340ce010 2023-10-16T13:47:38.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.464+0000 7f0f27fff700 1 -- 172.21.15.16:0/2024489744 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f2c012430 con 0x7f0f340ce010 2023-10-16T13:47:38.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.464+0000 7f0f3b8e3700 1 -- 172.21.15.16:0/2024489744 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f340ce010 msgr2=0x7f0f340cc1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:38.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.464+0000 7f0f3b8e3700 1 --2- 172.21.15.16:0/2024489744 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f340ce010 0x7f0f340cc1f0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f0f2c009d90 tx=0x7f0f2c00b330 comp rx=0 tx=0).stop 2023-10-16T13:47:38.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.464+0000 7f0f3b8e3700 1 -- 172.21.15.16:0/2024489744 shutdown_connections 2023-10-16T13:47:38.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.464+0000 7f0f3b8e3700 1 --2- 172.21.15.16:0/2024489744 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f340ce010 0x7f0f340cc1f0 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:38.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.464+0000 7f0f3b8e3700 1 --2- 172.21.15.16:0/2024489744 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f340d0ae0 0x7f0f340cef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:38.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.464+0000 7f0f3b8e3700 1 -- 172.21.15.16:0/2024489744 >> 172.21.15.16:0/2024489744 conn(0x7f0f340b1fd0 msgr2=0x7f0f340b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:38.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.465+0000 7f0f3b8e3700 1 -- 172.21.15.16:0/2024489744 shutdown_connections 2023-10-16T13:47:38.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.465+0000 7f0f3b8e3700 1 -- 172.21.15.16:0/2024489744 wait complete. 2023-10-16T13:47:38.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.466+0000 7f0f3b8e3700 1 Processor -- start 2023-10-16T13:47:38.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.466+0000 7f0f3b8e3700 1 -- start start 2023-10-16T13:47:38.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.467+0000 7f0f3b8e3700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f340ce010 0x7f0f340ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:38.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.467+0000 7f0f3b8e3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f340d0ae0 0x7f0f3406a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:38.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.467+0000 7f0f3b8e3700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f3415a5c0 0x7f0f3406ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:38.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.467+0000 7f0f3b8e3700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f34158c90 con 0x7f0f340d0ae0 2023-10-16T13:47:38.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.467+0000 7f0f3b8e3700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f340ad120 con 0x7f0f340ce010 2023-10-16T13:47:38.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.467+0000 7f0f3b8e3700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f340af1d0 con 0x7f0f3415a5c0 2023-10-16T13:47:38.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.467+0000 7f0f38e7e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f340d0ae0 0x7f0f3406a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:38.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.467+0000 7f0f38e7e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f340d0ae0 0x7f0f3406a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55744/0 (socket says 172.21.15.16:55744) 2023-10-16T13:47:38.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.467+0000 7f0f38e7e700 1 -- 172.21.15.16:0/2064978408 learned_addr learned my addr 172.21.15.16:0/2064978408 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:38.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.467+0000 7f0f3967f700 1 --2- 172.21.15.16:0/2064978408 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f340ce010 0x7f0f340ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:38.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.467+0000 7f0f39e80700 1 --2- 172.21.15.16:0/2064978408 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f3415a5c0 0x7f0f3406ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:38.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.468+0000 7f0f38e7e700 1 -- 172.21.15.16:0/2064978408 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f3415a5c0 msgr2=0x7f0f3406ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:38.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.468+0000 7f0f38e7e700 1 --2- 172.21.15.16:0/2064978408 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f3415a5c0 0x7f0f3406ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:38.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.468+0000 7f0f38e7e700 1 -- 172.21.15.16:0/2064978408 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f340ce010 msgr2=0x7f0f340ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:38.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.468+0000 7f0f38e7e700 1 --2- 172.21.15.16:0/2064978408 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f340ce010 0x7f0f340ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:38.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.468+0000 7f0f38e7e700 1 -- 172.21.15.16:0/2064978408 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f30009550 con 0x7f0f340d0ae0 2023-10-16T13:47:38.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.468+0000 7f0f3967f700 1 --2- 172.21.15.16:0/2064978408 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f340ce010 0x7f0f340ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:47:38.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.468+0000 7f0f39e80700 1 --2- 172.21.15.16:0/2064978408 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f3415a5c0 0x7f0f3406ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:47:38.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.468+0000 7f0f38e7e700 1 --2- 172.21.15.16:0/2064978408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f340d0ae0 0x7f0f3406a7e0 secure :-1 s=READY pgs=2792 cs=0 l=1 rev1=1 crypto rx=0x7f0f3000e9e0 tx=0x7f0f3000eda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:38.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.468+0000 7f0f25ffb700 1 -- 172.21.15.16:0/2064978408 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f3000c7f0 con 0x7f0f340d0ae0 2023-10-16T13:47:38.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.468+0000 7f0f25ffb700 1 -- 172.21.15.16:0/2064978408 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f3000c950 con 0x7f0f340d0ae0 2023-10-16T13:47:38.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.468+0000 7f0f3b8e3700 1 -- 172.21.15.16:0/2064978408 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f2c009a70 con 0x7f0f340d0ae0 2023-10-16T13:47:38.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.468+0000 7f0f25ffb700 1 -- 172.21.15.16:0/2064978408 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f3000ccc0 con 0x7f0f340d0ae0 2023-10-16T13:47:38.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.468+0000 7f0f3b8e3700 1 -- 172.21.15.16:0/2064978408 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f340af790 con 0x7f0f340d0ae0 2023-10-16T13:47:38.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.469+0000 7f0f3b8e3700 1 -- 172.21.15.16:0/2064978408 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f3404ebb0 con 0x7f0f340d0ae0 2023-10-16T13:47:38.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.470+0000 7f0f25ffb700 1 -- 172.21.15.16:0/2064978408 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 148) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f0f30013470 con 0x7f0f340d0ae0 2023-10-16T13:47:38.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.470+0000 7f0f25ffb700 1 --2- 172.21.15.16:0/2064978408 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f0f1806b140 0x7f0f1806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:38.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.470+0000 7f0f25ffb700 1 -- 172.21.15.16:0/2064978408 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(245..245 src has 1..245) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f0f3008a000 con 0x7f0f340d0ae0 2023-10-16T13:47:38.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.470+0000 7f0f3967f700 1 --2- 172.21.15.16:0/2064978408 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f0f1806b140 0x7f0f1806d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:38.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.471+0000 7f0f3967f700 1 --2- 172.21.15.16:0/2064978408 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f0f1806b140 0x7f0f1806d600 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0f280011e0 tx=0x7f0f28007ca0 comp rx=0 tx=0).ready entity=mgr.11886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:38.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.475+0000 7f0f25ffb700 1 -- 172.21.15.16:0/2064978408 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0f30054ba0 con 0x7f0f340d0ae0 2023-10-16T13:47:38.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.679+0000 7f0f3b8e3700 1 -- 172.21.15.16:0/2064978408 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f0f340ace10 con 0x7f0f340d0ae0 2023-10-16T13:47:38.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.680+0000 7f0f25ffb700 1 -- 172.21.15.16:0/2064978408 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v245) v1 ==== 84+0+209 (secure 0 0 0) 0x7f0f300581c0 con 0x7f0f340d0ae0 2023-10-16T13:47:38.683 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:47:38.683 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2023-10-16T13:47:38.683 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:47:38.683 INFO:teuthology.orchestra.run.smithi016.stdout: "poolnum": 1, 2023-10-16T13:47:38.683 INFO:teuthology.orchestra.run.smithi016.stdout: "poolname": ".mgr" 2023-10-16T13:47:38.684 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:47:38.684 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:47:38.684 INFO:teuthology.orchestra.run.smithi016.stdout: "poolnum": 31, 2023-10-16T13:47:38.684 INFO:teuthology.orchestra.run.smithi016.stdout: "poolname": "cephfs_metadata" 2023-10-16T13:47:38.685 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:47:38.685 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:47:38.685 INFO:teuthology.orchestra.run.smithi016.stdout: "poolnum": 32, 2023-10-16T13:47:38.685 INFO:teuthology.orchestra.run.smithi016.stdout: "poolname": "cephfs_data" 2023-10-16T13:47:38.685 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:47:38.686 INFO:teuthology.orchestra.run.smithi016.stdout:] 2023-10-16T13:47:38.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.682+0000 7f0f3b8e3700 1 -- 172.21.15.16:0/2064978408 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f0f1806b140 msgr2=0x7f0f1806d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:38.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.682+0000 7f0f3b8e3700 1 --2- 172.21.15.16:0/2064978408 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f0f1806b140 0x7f0f1806d600 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0f280011e0 tx=0x7f0f28007ca0 comp rx=0 tx=0).stop 2023-10-16T13:47:38.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.682+0000 7f0f3b8e3700 1 -- 172.21.15.16:0/2064978408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f340d0ae0 msgr2=0x7f0f3406a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:38.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.682+0000 7f0f3b8e3700 1 --2- 172.21.15.16:0/2064978408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f340d0ae0 0x7f0f3406a7e0 secure :-1 s=READY pgs=2792 cs=0 l=1 rev1=1 crypto rx=0x7f0f3000e9e0 tx=0x7f0f3000eda0 comp rx=0 tx=0).stop 2023-10-16T13:47:38.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.682+0000 7f0f3b8e3700 1 -- 172.21.15.16:0/2064978408 shutdown_connections 2023-10-16T13:47:38.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.682+0000 7f0f3b8e3700 1 --2- 172.21.15.16:0/2064978408 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f0f1806b140 0x7f0f1806d600 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:38.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.682+0000 7f0f3b8e3700 1 --2- 172.21.15.16:0/2064978408 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f3415a5c0 0x7f0f3406ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:38.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.682+0000 7f0f3b8e3700 1 --2- 172.21.15.16:0/2064978408 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f340ce010 0x7f0f340ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:38.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.682+0000 7f0f3b8e3700 1 --2- 172.21.15.16:0/2064978408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f340d0ae0 0x7f0f3406a7e0 unknown :-1 s=CLOSED pgs=2792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:38.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.682+0000 7f0f3b8e3700 1 -- 172.21.15.16:0/2064978408 >> 172.21.15.16:0/2064978408 conn(0x7f0f340b1fd0 msgr2=0x7f0f3415e1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:38.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.682+0000 7f0f3b8e3700 1 -- 172.21.15.16:0/2064978408 shutdown_connections 2023-10-16T13:47:38.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.682+0000 7f0f3b8e3700 1 -- 172.21.15.16:0/2064978408 wait complete. 2023-10-16T13:47:38.696 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'backup_fs' 2023-10-16T13:47:38.697 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create backup_fs_metadata --pg_num_min 64 2023-10-16T13:47:38.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.823+0000 7fb9131f1700 1 Processor -- start 2023-10-16T13:47:38.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.823+0000 7fb9131f1700 1 -- start start 2023-10-16T13:47:38.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.824+0000 7fb9131f1700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fb90c160e60 0x7fb90c15e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:38.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.824+0000 7fb9131f1700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb90c06d980 0x7fb90c15efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:38.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.824+0000 7fb9131f1700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb90c06a7e0 0x7fb90c06ac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:38.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.824+0000 7fb9131f1700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb90c0cdeb0 con 0x7fb90c06d980 2023-10-16T13:47:38.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.824+0000 7fb9131f1700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb90c1613e0 con 0x7fb90c160e60 2023-10-16T13:47:38.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.824+0000 7fb9131f1700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb90c06b160 con 0x7fb90c06a7e0 2023-10-16T13:47:38.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.824+0000 7fb90bfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb90c06d980 0x7fb90c15efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:38.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.824+0000 7fb90bfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb90c06d980 0x7fb90c15efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55752/0 (socket says 172.21.15.16:55752) 2023-10-16T13:47:38.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.824+0000 7fb90bfff700 1 -- 172.21.15.16:0/1598527887 learned_addr learned my addr 172.21.15.16:0/1598527887 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:38.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.824+0000 7fb910f8d700 1 --2- 172.21.15.16:0/1598527887 >> v2:172.21.15.49:3300/0 conn(0x7fb90c160e60 0x7fb90c15e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:38.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.824+0000 7fb91178e700 1 --2- 172.21.15.16:0/1598527887 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb90c06a7e0 0x7fb90c06ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:38.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.824+0000 7fb90bfff700 1 -- 172.21.15.16:0/1598527887 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb90c06a7e0 msgr2=0x7fb90c06ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:38.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.824+0000 7fb90bfff700 1 --2- 172.21.15.16:0/1598527887 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb90c06a7e0 0x7fb90c06ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:38.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.824+0000 7fb90bfff700 1 -- 172.21.15.16:0/1598527887 >> v2:172.21.15.49:3300/0 conn(0x7fb90c160e60 msgr2=0x7fb90c15e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:38.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.824+0000 7fb90bfff700 1 --2- 172.21.15.16:0/1598527887 >> v2:172.21.15.49:3300/0 conn(0x7fb90c160e60 0x7fb90c15e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:38.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.824+0000 7fb90bfff700 1 -- 172.21.15.16:0/1598527887 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb90c1582d0 con 0x7fb90c06d980 2023-10-16T13:47:38.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.825+0000 7fb90bfff700 1 --2- 172.21.15.16:0/1598527887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb90c06d980 0x7fb90c15efb0 secure :-1 s=READY pgs=2793 cs=0 l=1 rev1=1 crypto rx=0x7fb90c073050 tx=0x7fb8f8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f22b8360c4d26795 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:38.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.825+0000 7fb90b7fe700 1 -- 172.21.15.16:0/1598527887 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8f800b870 con 0x7fb90c06d980 2023-10-16T13:47:38.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.825+0000 7fb90b7fe700 1 -- 172.21.15.16:0/1598527887 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8f800b9d0 con 0x7fb90c06d980 2023-10-16T13:47:38.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.825+0000 7fb90b7fe700 1 -- 172.21.15.16:0/1598527887 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8f800bd60 con 0x7fb90c06d980 2023-10-16T13:47:38.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.825+0000 7fb9131f1700 1 -- 172.21.15.16:0/1598527887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb90c06d980 msgr2=0x7fb90c15efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:38.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.825+0000 7fb9131f1700 1 --2- 172.21.15.16:0/1598527887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb90c06d980 0x7fb90c15efb0 secure :-1 s=READY pgs=2793 cs=0 l=1 rev1=1 crypto rx=0x7fb90c073050 tx=0x7fb8f8009bd0 comp rx=0 tx=0).stop 2023-10-16T13:47:38.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.825+0000 7fb9131f1700 1 -- 172.21.15.16:0/1598527887 shutdown_connections 2023-10-16T13:47:38.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.825+0000 7fb9131f1700 1 --2- 172.21.15.16:0/1598527887 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb90c06a7e0 0x7fb90c06ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:38.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.825+0000 7fb9131f1700 1 --2- 172.21.15.16:0/1598527887 >> v2:172.21.15.49:3300/0 conn(0x7fb90c160e60 0x7fb90c15e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:38.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.825+0000 7fb9131f1700 1 --2- 172.21.15.16:0/1598527887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb90c06d980 0x7fb90c15efb0 unknown :-1 s=CLOSED pgs=2793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:38.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.825+0000 7fb9131f1700 1 -- 172.21.15.16:0/1598527887 >> 172.21.15.16:0/1598527887 conn(0x7fb90c06f430 msgr2=0x7fb90c06f810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:38.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.825+0000 7fb9131f1700 1 -- 172.21.15.16:0/1598527887 shutdown_connections 2023-10-16T13:47:38.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.825+0000 7fb9131f1700 1 -- 172.21.15.16:0/1598527887 wait complete. 2023-10-16T13:47:38.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.827+0000 7fb9131f1700 1 Processor -- start 2023-10-16T13:47:38.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.827+0000 7fb9131f1700 1 -- start start 2023-10-16T13:47:38.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.828+0000 7fb9131f1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb90c06a7e0 0x7fb90c0b3df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:38.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.828+0000 7fb9131f1700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb90c06d980 0x7fb90c14e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:38.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.828+0000 7fb9131f1700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb90c160e60 0x7fb90c0b5180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:38.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.828+0000 7fb9131f1700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb90c0b39d0 con 0x7fb90c06a7e0 2023-10-16T13:47:38.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.828+0000 7fb9131f1700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb90c0b5c40 con 0x7fb90c160e60 2023-10-16T13:47:38.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.828+0000 7fb9131f1700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb90c0b4330 con 0x7fb90c06d980 2023-10-16T13:47:38.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.828+0000 7fb910f8d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb90c06a7e0 0x7fb90c0b3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:38.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.828+0000 7fb910f8d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb90c06a7e0 0x7fb90c0b3df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55756/0 (socket says 172.21.15.16:55756) 2023-10-16T13:47:38.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.828+0000 7fb910f8d700 1 -- 172.21.15.16:0/1729032315 learned_addr learned my addr 172.21.15.16:0/1729032315 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:38.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.828+0000 7fb910f8d700 1 -- 172.21.15.16:0/1729032315 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb90c06d980 msgr2=0x7fb90c14e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:38.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.828+0000 7fb910f8d700 1 --2- 172.21.15.16:0/1729032315 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb90c06d980 0x7fb90c14e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:38.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.828+0000 7fb910f8d700 1 -- 172.21.15.16:0/1729032315 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb90c160e60 msgr2=0x7fb90c0b5180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:38.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.828+0000 7fb91178e700 1 --2- 172.21.15.16:0/1729032315 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb90c160e60 0x7fb90c0b5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:38.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.828+0000 7fb910f8d700 1 --2- 172.21.15.16:0/1729032315 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb90c160e60 0x7fb90c0b5180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:38.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.828+0000 7fb910f8d700 1 -- 172.21.15.16:0/1729032315 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8f8009550 con 0x7fb90c06a7e0 2023-10-16T13:47:38.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.828+0000 7fb910f8d700 1 --2- 172.21.15.16:0/1729032315 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb90c06a7e0 0x7fb90c0b3df0 secure :-1 s=READY pgs=2794 cs=0 l=1 rev1=1 crypto rx=0x7fb90000d800 tx=0x7fb90000dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:38.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.828+0000 7fb9097fa700 1 -- 172.21.15.16:0/1729032315 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9000125f0 con 0x7fb90c06a7e0 2023-10-16T13:47:38.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.828+0000 7fb9097fa700 1 -- 172.21.15.16:0/1729032315 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb900012750 con 0x7fb90c06a7e0 2023-10-16T13:47:38.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.828+0000 7fb9097fa700 1 -- 172.21.15.16:0/1729032315 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb900012b00 con 0x7fb90c06a7e0 2023-10-16T13:47:38.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.828+0000 7fb9131f1700 1 -- 172.21.15.16:0/1729032315 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb90c0b3030 con 0x7fb90c06a7e0 2023-10-16T13:47:38.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.828+0000 7fb9131f1700 1 -- 172.21.15.16:0/1729032315 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb90c0b35f0 con 0x7fb90c06a7e0 2023-10-16T13:47:38.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.829+0000 7fb9131f1700 1 -- 172.21.15.16:0/1729032315 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb90c04ec20 con 0x7fb90c06a7e0 2023-10-16T13:47:38.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.830+0000 7fb9097fa700 1 -- 172.21.15.16:0/1729032315 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 148) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fb900012c60 con 0x7fb90c06a7e0 2023-10-16T13:47:38.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.830+0000 7fb9097fa700 1 --2- 172.21.15.16:0/1729032315 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fb8f006b140 0x7fb8f006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:38.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.830+0000 7fb9097fa700 1 -- 172.21.15.16:0/1729032315 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(245..245 src has 1..245) v4 ==== 6157+0+0 (secure 0 0 0) 0x7fb900089f50 con 0x7fb90c06a7e0 2023-10-16T13:47:38.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.830+0000 7fb90bfff700 1 --2- 172.21.15.16:0/1729032315 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fb8f006b140 0x7fb8f006d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:38.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.831+0000 7fb90bfff700 1 --2- 172.21.15.16:0/1729032315 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fb8f006b140 0x7fb8f006d600 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb90c073050 tx=0x7fb8f8010000 comp rx=0 tx=0).ready entity=mgr.11886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:38.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:38.835+0000 7fb9097fa700 1 -- 172.21.15.16:0/1729032315 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb900054ae0 con 0x7fb90c06a7e0 2023-10-16T13:47:39.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:39.040+0000 7fb9131f1700 1 -- 172.21.15.16:0/1729032315 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "backup_fs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fb90c0b6260 con 0x7fb90c06a7e0 2023-10-16T13:47:39.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:39.895+0000 7fb9097fa700 1 -- 172.21.15.16:0/1729032315 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "backup_fs_metadata", "pg_num_min": 64}]=0 pool 'backup_fs_metadata' created v246) v1 ==== 144+0+0 (secure 0 0 0) 0x7fb900058100 con 0x7fb90c06a7e0 2023-10-16T13:47:39.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:39.899+0000 7fb9131f1700 1 -- 172.21.15.16:0/1729032315 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fb8f006b140 msgr2=0x7fb8f006d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:39.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:39.899+0000 7fb9131f1700 1 --2- 172.21.15.16:0/1729032315 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fb8f006b140 0x7fb8f006d600 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb90c073050 tx=0x7fb8f8010000 comp rx=0 tx=0).stop 2023-10-16T13:47:39.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:39.899+0000 7fb9131f1700 1 -- 172.21.15.16:0/1729032315 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb90c06a7e0 msgr2=0x7fb90c0b3df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:39.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:39.899+0000 7fb9131f1700 1 --2- 172.21.15.16:0/1729032315 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb90c06a7e0 0x7fb90c0b3df0 secure :-1 s=READY pgs=2794 cs=0 l=1 rev1=1 crypto rx=0x7fb90000d800 tx=0x7fb90000dbc0 comp rx=0 tx=0).stop 2023-10-16T13:47:39.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:39.900+0000 7fb9131f1700 1 -- 172.21.15.16:0/1729032315 shutdown_connections 2023-10-16T13:47:39.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:39.900+0000 7fb9131f1700 1 --2- 172.21.15.16:0/1729032315 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fb8f006b140 0x7fb8f006d600 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:39.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:39.900+0000 7fb9131f1700 1 --2- 172.21.15.16:0/1729032315 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb90c160e60 0x7fb90c0b5180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:39.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:39.900+0000 7fb9131f1700 1 --2- 172.21.15.16:0/1729032315 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb90c06a7e0 0x7fb90c0b3df0 unknown :-1 s=CLOSED pgs=2794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:39.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:39.900+0000 7fb9131f1700 1 --2- 172.21.15.16:0/1729032315 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb90c06d980 0x7fb90c14e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:39.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:39.900+0000 7fb9131f1700 1 -- 172.21.15.16:0/1729032315 >> 172.21.15.16:0/1729032315 conn(0x7fb90c06f430 msgr2=0x7fb90c1508a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:39.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:39.901+0000 7fb9131f1700 1 -- 172.21.15.16:0/1729032315 shutdown_connections 2023-10-16T13:47:39.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:39.906+0000 7fb9131f1700 1 -- 172.21.15.16:0/1729032315 wait complete. 2023-10-16T13:47:39.912 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'backup_fs_metadata' created 2023-10-16T13:47:39.926 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create backup_fs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-16T13:47:40.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:40.062+0000 7fdc57b5a700 1 Processor -- start 2023-10-16T13:47:40.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:40.062+0000 7fdc57b5a700 1 -- start start 2023-10-16T13:47:40.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:40.063+0000 7fdc57b5a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdc5015ec80 0x7fdc500cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:40.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:40.063+0000 7fdc57b5a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdc500d2110 0x7fdc500cd1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:40.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:40.063+0000 7fdc57b5a700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc500b2440 con 0x7fdc5015f060 2023-10-16T13:47:40.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:40.063+0000 7fdc57b5a700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc500b2c50 con 0x7fdc500d2110 2023-10-16T13:47:40.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:40.063+0000 7fdc57b5a700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc500d00e0 con 0x7fdc5015ec80 2023-10-16T13:47:40.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:40.063+0000 7fdc560f7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdc500d2110 0x7fdc500cd1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:40.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:40.063+0000 7fdc560f7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdc500d2110 0x7fdc500cd1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55768/0 (socket says 172.21.15.16:55768) 2023-10-16T13:47:40.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:40.063+0000 7fdc550f5700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fdc5015f060 0x7fdc500b4440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:45038/0 (socket says 172.21.15.16:45038) 2023-10-16T13:47:40.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:40.063+0000 7fdc560f7700 1 -- 172.21.15.16:0/3683065454 learned_addr learned my addr 172.21.15.16:0/3683065454 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:40.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:40.063+0000 7fdc558f6700 1 --2- 172.21.15.16:0/3683065454 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdc5015ec80 0x7fdc500cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:40.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:40.063+0000 7fdc560f7700 1 -- 172.21.15.16:0/3683065454 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdc5015ec80 msgr2=0x7fdc500cdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:40.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:40.063+0000 7fdc560f7700 1 --2- 172.21.15.16:0/3683065454 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdc5015ec80 0x7fdc500cdec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:40.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:40.063+0000 7fdc560f7700 1 -- 172.21.15.16:0/3683065454 >> v1:172.21.15.16:6789/0 conn(0x7fdc5015f060 legacy=0x7fdc500b4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:40.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:40.063+0000 7fdc560f7700 1 -- 172.21.15.16:0/3683065454 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc500cef60 con 0x7fdc500d2110 2023-10-16T13:47:40.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:40.063+0000 7fdc560f7700 1 --2- 172.21.15.16:0/3683065454 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc500d2110 0x7fdc500cd1a0 secure :-1 s=READY pgs=2796 cs=0 l=1 rev1=1 crypto rx=0x7fdc48009e50 tx=0x7fdc4800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a98e360248c15db0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:40.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:40.064+0000 7fdc548f4700 1 -- 172.21.15.16:0/3683065454 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc4800e070 con 0x7fdc500d2110 2023-10-16T13:47:40.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:40.064+0000 7fdc548f4700 1 -- 172.21.15.16:0/3683065454 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc4800b9c0 con 0x7fdc500d2110 2023-10-16T13:47:40.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:40.064+0000 7fdc548f4700 1 -- 172.21.15.16:0/3683065454 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc4800bd70 con 0x7fdc500d2110 2023-10-16T13:47:40.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:40.064+0000 7fdc57b5a700 1 -- 172.21.15.16:0/3683065454 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc500d2110 msgr2=0x7fdc500cd1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:40.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:40.064+0000 7fdc57b5a700 1 --2- 172.21.15.16:0/3683065454 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc500d2110 0x7fdc500cd1a0 secure :-1 s=READY pgs=2796 cs=0 l=1 rev1=1 crypto rx=0x7fdc48009e50 tx=0x7fdc4800b330 comp rx=0 tx=0).stop 2023-10-16T13:47:40.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:40.064+0000 7fdc57b5a700 1 -- 172.21.15.16:0/3683065454 shutdown_connections 2023-10-16T13:47:40.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:40.064+0000 7fdc57b5a700 1 --2- 172.21.15.16:0/3683065454 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc500d2110 0x7fdc500cd1a0 unknown :-1 s=CLOSED pgs=2796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:40.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:40.064+0000 7fdc57b5a700 1 --2- 172.21.15.16:0/3683065454 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdc5015ec80 0x7fdc500cdec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:40.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:40.064+0000 7fdc57b5a700 1 -- 172.21.15.16:0/3683065454 >> 172.21.15.16:0/3683065454 conn(0x7fdc500d15f0 msgr2=0x7fdc500d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:40.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:40.064+0000 7fdc57b5a700 1 -- 172.21.15.16:0/3683065454 shutdown_connections 2023-10-16T13:47:40.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:40.064+0000 7fdc57b5a700 1 -- 172.21.15.16:0/3683065454 wait complete. 2023-10-16T13:47:40.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:40.066+0000 7fdc57b5a700 1 Processor -- start 2023-10-16T13:47:40.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:40.066+0000 7fdc57b5a700 1 -- start start 2023-10-16T13:47:40.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:40.066+0000 7fdc57b5a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc500d2110 0x7fdc5015a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:40.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:40.066+0000 7fdc57b5a700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdc5015ec80 0x7fdc5015d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:40.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:40.066+0000 7fdc57b5a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdc5015f060 0x7fdc5015d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:40.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:40.066+0000 7fdc57b5a700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc5006c300 con 0x7fdc500d2110 2023-10-16T13:47:40.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:40.066+0000 7fdc57b5a700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc5006c5a0 con 0x7fdc5015ec80 2023-10-16T13:47:40.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:40.066+0000 7fdc57b5a700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc5015aa80 con 0x7fdc5015f060 2023-10-16T13:47:40.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:40.066+0000 7fdc558f6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc500d2110 0x7fdc5015a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:40.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:40.066+0000 7fdc558f6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc500d2110 0x7fdc5015a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55770/0 (socket says 172.21.15.16:55770) 2023-10-16T13:47:40.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:40.067+0000 7fdc558f6700 1 -- 172.21.15.16:0/27155023 learned_addr learned my addr 172.21.15.16:0/27155023 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:40.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:40.067+0000 7fdc558f6700 1 -- 172.21.15.16:0/27155023 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdc5015f060 msgr2=0x7fdc5015d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:40.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:40.067+0000 7fdc558f6700 1 --2- 172.21.15.16:0/27155023 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdc5015f060 0x7fdc5015d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:40.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:40.067+0000 7fdc558f6700 1 -- 172.21.15.16:0/27155023 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdc5015ec80 msgr2=0x7fdc5015d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:40.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:40.067+0000 7fdc550f5700 1 --2- 172.21.15.16:0/27155023 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdc5015ec80 0x7fdc5015d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:40.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:40.067+0000 7fdc558f6700 1 --2- 172.21.15.16:0/27155023 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdc5015ec80 0x7fdc5015d360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:40.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:40.067+0000 7fdc558f6700 1 -- 172.21.15.16:0/27155023 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc48009b30 con 0x7fdc500d2110 2023-10-16T13:47:40.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:40.067+0000 7fdc558f6700 1 --2- 172.21.15.16:0/27155023 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc500d2110 0x7fdc5015a540 secure :-1 s=READY pgs=2797 cs=0 l=1 rev1=1 crypto rx=0x7fdc4c00e810 tx=0x7fdc4c00ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:40.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:40.067+0000 7fdc467fc700 1 -- 172.21.15.16:0/27155023 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc4c013610 con 0x7fdc500d2110 2023-10-16T13:47:40.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:40.067+0000 7fdc467fc700 1 -- 172.21.15.16:0/27155023 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc4c013770 con 0x7fdc500d2110 2023-10-16T13:47:40.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:40.067+0000 7fdc57b5a700 1 -- 172.21.15.16:0/27155023 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc500b62e0 con 0x7fdc500d2110 2023-10-16T13:47:40.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:40.067+0000 7fdc467fc700 1 -- 172.21.15.16:0/27155023 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc4c013b20 con 0x7fdc500d2110 2023-10-16T13:47:40.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:40.067+0000 7fdc57b5a700 1 -- 172.21.15.16:0/27155023 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc500b68a0 con 0x7fdc500d2110 2023-10-16T13:47:40.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:40.068+0000 7fdc57b5a700 1 -- 172.21.15.16:0/27155023 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc5006c880 con 0x7fdc500d2110 2023-10-16T13:47:40.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:40.069+0000 7fdc467fc700 1 -- 172.21.15.16:0/27155023 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 148) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fdc4c013c80 con 0x7fdc500d2110 2023-10-16T13:47:40.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:40.069+0000 7fdc467fc700 1 --2- 172.21.15.16:0/27155023 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fdc3406b140 0x7fdc3406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:40.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:40.069+0000 7fdc467fc700 1 -- 172.21.15.16:0/27155023 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(246..246 src has 1..246) v4 ==== 6530+0+0 (secure 0 0 0) 0x7fdc4c089f20 con 0x7fdc500d2110 2023-10-16T13:47:40.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:40.069+0000 7fdc550f5700 1 --2- 172.21.15.16:0/27155023 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fdc3406b140 0x7fdc3406d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:40.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:40.070+0000 7fdc550f5700 1 --2- 172.21.15.16:0/27155023 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fdc3406b140 0x7fdc3406d600 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fdc500b2410 tx=0x7fdc40009040 comp rx=0 tx=0).ready entity=mgr.11886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:40.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:40.073+0000 7fdc467fc700 1 -- 172.21.15.16:0/27155023 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdc4c0548b0 con 0x7fdc500d2110 2023-10-16T13:47:40.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:40.279+0000 7fdc57b5a700 1 -- 172.21.15.16:0/27155023 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "backup_fs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fdc500b6ac0 con 0x7fdc500d2110 2023-10-16T13:47:40.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:40.898+0000 7fdc467fc700 1 -- 172.21.15.16:0/27155023 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "backup_fs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'backup_fs_data' created v247) v1 ==== 176+0+0 (secure 0 0 0) 0x7fdc4c0210d0 con 0x7fdc500d2110 2023-10-16T13:47:40.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:40.902+0000 7fdc57b5a700 1 -- 172.21.15.16:0/27155023 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fdc3406b140 msgr2=0x7fdc3406d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:40.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:40.902+0000 7fdc57b5a700 1 --2- 172.21.15.16:0/27155023 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fdc3406b140 0x7fdc3406d600 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fdc500b2410 tx=0x7fdc40009040 comp rx=0 tx=0).stop 2023-10-16T13:47:40.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:40.902+0000 7fdc57b5a700 1 -- 172.21.15.16:0/27155023 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc500d2110 msgr2=0x7fdc5015a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:40.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:40.902+0000 7fdc57b5a700 1 --2- 172.21.15.16:0/27155023 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc500d2110 0x7fdc5015a540 secure :-1 s=READY pgs=2797 cs=0 l=1 rev1=1 crypto rx=0x7fdc4c00e810 tx=0x7fdc4c00ebd0 comp rx=0 tx=0).stop 2023-10-16T13:47:40.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:40.902+0000 7fdc57b5a700 1 -- 172.21.15.16:0/27155023 shutdown_connections 2023-10-16T13:47:40.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:40.902+0000 7fdc57b5a700 1 --2- 172.21.15.16:0/27155023 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fdc3406b140 0x7fdc3406d600 secure :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fdc500b2410 tx=0x7fdc40009040 comp rx=0 tx=0).stop 2023-10-16T13:47:40.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:40.903+0000 7fdc57b5a700 1 --2- 172.21.15.16:0/27155023 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdc5015f060 0x7fdc5015d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:40.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:40.903+0000 7fdc57b5a700 1 --2- 172.21.15.16:0/27155023 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc500d2110 0x7fdc5015a540 secure :-1 s=CLOSED pgs=2797 cs=0 l=1 rev1=1 crypto rx=0x7fdc4c00e810 tx=0x7fdc4c00ebd0 comp rx=0 tx=0).stop 2023-10-16T13:47:40.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:40.903+0000 7fdc57b5a700 1 --2- 172.21.15.16:0/27155023 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdc5015ec80 0x7fdc5015d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:40.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:40.903+0000 7fdc57b5a700 1 -- 172.21.15.16:0/27155023 >> 172.21.15.16:0/27155023 conn(0x7fdc500d15f0 msgr2=0x7fdc500b4170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:40.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:40.903+0000 7fdc57b5a700 1 -- 172.21.15.16:0/27155023 shutdown_connections 2023-10-16T13:47:40.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:40.904+0000 7fdc57b5a700 1 -- 172.21.15.16:0/27155023 wait complete. 2023-10-16T13:47:40.925 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'backup_fs_data' created 2023-10-16T13:47:40.936 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new backup_fs backup_fs_metadata backup_fs_data 2023-10-16T13:47:41.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.067+0000 7f7ea9423700 1 Processor -- start 2023-10-16T13:47:41.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.067+0000 7f7ea9423700 1 -- start start 2023-10-16T13:47:41.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.068+0000 7f7ea9423700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7ea4159de0 0x7f7ea415a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:41.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.068+0000 7f7ea9423700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f7ea40cf4c0 0x7f7ea406aac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:41.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.068+0000 7f7ea9423700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ea415dda0 con 0x7f7ea406b000 2023-10-16T13:47:41.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.068+0000 7f7ea9423700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ea415ea20 con 0x7f7ea40cf4c0 2023-10-16T13:47:41.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.068+0000 7f7ea9423700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ea415e080 con 0x7f7ea4159de0 2023-10-16T13:47:41.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.068+0000 7f7ea37fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f7ea406b000 0x7f7ea406b4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:45056/0 (socket says 172.21.15.16:45056) 2023-10-16T13:47:41.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.068+0000 7f7ea37fe700 1 -- 172.21.15.16:0/1593284481 learned_addr learned my addr 172.21.15.16:0/1593284481 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:47:41.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.068+0000 7f7ea27fc700 1 --2- 172.21.15.16:0/1593284481 >> v2:172.21.15.49:3300/0 conn(0x7f7ea40cf4c0 0x7f7ea406aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:41.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.068+0000 7f7ea2ffd700 1 --2- 172.21.15.16:0/1593284481 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7ea4159de0 0x7f7ea415a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:41.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.069+0000 7f7ea1ffb700 1 -- 172.21.15.16:0/1593284481 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3407806924 0 0) 0x7f7ea415dda0 con 0x7f7ea406b000 2023-10-16T13:47:41.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.069+0000 7f7ea1ffb700 1 -- 172.21.15.16:0/1593284481 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e8c0035b0 con 0x7f7ea406b000 2023-10-16T13:47:41.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.069+0000 7f7ea1ffb700 1 -- 172.21.15.16:0/1593284481 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 608995235 0 0) 0x7f7e8c0035b0 con 0x7f7ea406b000 2023-10-16T13:47:41.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.069+0000 7f7ea1ffb700 1 -- 172.21.15.16:0/1593284481 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7ea4159de0 msgr2=0x7f7ea415a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:41.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.069+0000 7f7ea1ffb700 1 --2- 172.21.15.16:0/1593284481 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7ea4159de0 0x7f7ea415a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:41.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.069+0000 7f7ea1ffb700 1 -- 172.21.15.16:0/1593284481 >> v2:172.21.15.49:3300/0 conn(0x7f7ea40cf4c0 msgr2=0x7f7ea406aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:41.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.069+0000 7f7ea1ffb700 1 --2- 172.21.15.16:0/1593284481 >> v2:172.21.15.49:3300/0 conn(0x7f7ea40cf4c0 0x7f7ea406aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:41.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.069+0000 7f7ea1ffb700 1 -- 172.21.15.16:0/1593284481 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ea415e360 con 0x7f7ea406b000 2023-10-16T13:47:41.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.069+0000 7f7ea1ffb700 1 -- 172.21.15.16:0/1593284481 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f7e9c0019f0 con 0x7f7ea406b000 2023-10-16T13:47:41.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.069+0000 7f7ea1ffb700 1 -- 172.21.15.16:0/1593284481 >> v1:172.21.15.16:6789/0 conn(0x7f7ea406b000 legacy=0x7f7ea406b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:41.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.069+0000 7f7ea1ffb700 1 --2- 172.21.15.16:0/1593284481 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e8c003a00 0x7f7e8c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:41.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.069+0000 7f7ea1ffb700 1 --2- 172.21.15.16:0/1593284481 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7e8c004850 0x7f7e8c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:41.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.069+0000 7f7ea1ffb700 1 --2- 172.21.15.16:0/1593284481 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7ea4159de0 0x7f7e8c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:41.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.069+0000 7f7ea1ffb700 1 -- 172.21.15.16:0/1593284481 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ea415ea20 con 0x7f7e8c003a00 2023-10-16T13:47:41.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.069+0000 7f7ea1ffb700 1 -- 172.21.15.16:0/1593284481 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ea415e080 con 0x7f7e8c004850 2023-10-16T13:47:41.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.069+0000 7f7ea1ffb700 1 -- 172.21.15.16:0/1593284481 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e8c00ba10 con 0x7f7ea4159de0 2023-10-16T13:47:41.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.069+0000 7f7ea2ffd700 1 --2- 172.21.15.16:0/1593284481 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e8c003a00 0x7f7e8c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:41.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.069+0000 7f7ea27fc700 1 --2- 172.21.15.16:0/1593284481 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7e8c004850 0x7f7e8c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:41.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.069+0000 7f7ea37fe700 1 --2- 172.21.15.16:0/1593284481 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7ea4159de0 0x7f7e8c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:41.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.069+0000 7f7ea2ffd700 1 -- 172.21.15.16:0/1593284481 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7ea4159de0 msgr2=0x7f7e8c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:41.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.069+0000 7f7ea2ffd700 1 --2- 172.21.15.16:0/1593284481 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7ea4159de0 0x7f7e8c005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:41.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.069+0000 7f7ea2ffd700 1 -- 172.21.15.16:0/1593284481 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7e8c004850 msgr2=0x7f7e8c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:41.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.069+0000 7f7ea2ffd700 1 --2- 172.21.15.16:0/1593284481 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7e8c004850 0x7f7e8c004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:41.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.069+0000 7f7ea2ffd700 1 -- 172.21.15.16:0/1593284481 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e8c0035b0 con 0x7f7e8c003a00 2023-10-16T13:47:41.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.069+0000 7f7ea2ffd700 1 --2- 172.21.15.16:0/1593284481 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e8c003a00 0x7f7e8c003e20 secure :-1 s=READY pgs=2799 cs=0 l=1 rev1=1 crypto rx=0x7f7e98009b90 tx=0x7f7e8c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=793e687b7b832acf server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:41.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.070+0000 7f7ea1ffb700 1 -- 172.21.15.16:0/1593284481 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e9800e6c0 con 0x7f7e8c003a00 2023-10-16T13:47:41.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.070+0000 7f7ea1ffb700 1 -- 172.21.15.16:0/1593284481 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e9800e820 con 0x7f7e8c003a00 2023-10-16T13:47:41.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.070+0000 7f7ea1ffb700 1 -- 172.21.15.16:0/1593284481 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e9800eb90 con 0x7f7e8c003a00 2023-10-16T13:47:41.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.070+0000 7f7ea9423700 1 -- 172.21.15.16:0/1593284481 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e8c003a00 msgr2=0x7f7e8c003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:41.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.070+0000 7f7ea9423700 1 --2- 172.21.15.16:0/1593284481 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e8c003a00 0x7f7e8c003e20 secure :-1 s=READY pgs=2799 cs=0 l=1 rev1=1 crypto rx=0x7f7e98009b90 tx=0x7f7e8c005880 comp rx=0 tx=0).stop 2023-10-16T13:47:41.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.070+0000 7f7ea37fe700 1 -- 172.21.15.16:0/1593284481 reap_dead start 2023-10-16T13:47:41.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.070+0000 7f7ea9423700 1 -- 172.21.15.16:0/1593284481 shutdown_connections 2023-10-16T13:47:41.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.070+0000 7f7ea9423700 1 -- 172.21.15.16:0/1593284481 >> 172.21.15.16:0/1593284481 conn(0x7f7ea40cc5e0 msgr2=0x7f7ea40cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:41.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.070+0000 7f7ea9423700 1 -- 172.21.15.16:0/1593284481 shutdown_connections 2023-10-16T13:47:41.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.070+0000 7f7ea9423700 1 -- 172.21.15.16:0/1593284481 wait complete. 2023-10-16T13:47:41.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.072+0000 7f7ea9423700 1 Processor -- start 2023-10-16T13:47:41.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.072+0000 7f7ea9423700 1 -- start start 2023-10-16T13:47:41.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.072+0000 7f7ea9423700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7e8c003a00 0x7f7ea4148cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:41.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.072+0000 7f7ea9423700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7ea40b3790 0x7f7ea40b3c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:41.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.072+0000 7f7ea9423700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ea414eef0 0x7f7ea414f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:41.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.072+0000 7f7ea9423700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ea414a460 con 0x7f7ea414eef0 2023-10-16T13:47:41.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.072+0000 7f7ea9423700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ea414f8b0 con 0x7f7e8c003a00 2023-10-16T13:47:41.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.072+0000 7f7ea9423700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ea4149590 con 0x7f7ea40b3790 2023-10-16T13:47:41.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.072+0000 7f7ea37fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ea414eef0 0x7f7ea414f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:41.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.073+0000 7f7ea37fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ea414eef0 0x7f7ea414f370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55794/0 (socket says 172.21.15.16:55794) 2023-10-16T13:47:41.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.073+0000 7f7ea37fe700 1 -- 172.21.15.16:0/2320827170 learned_addr learned my addr 172.21.15.16:0/2320827170 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:41.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.073+0000 7f7ea2ffd700 1 --2- 172.21.15.16:0/2320827170 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7e8c003a00 0x7f7ea4148cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:41.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.073+0000 7f7ea27fc700 1 --2- 172.21.15.16:0/2320827170 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7ea40b3790 0x7f7ea40b3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:41.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.073+0000 7f7ea37fe700 1 -- 172.21.15.16:0/2320827170 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7ea40b3790 msgr2=0x7f7ea40b3c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:41.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.073+0000 7f7ea37fe700 1 --2- 172.21.15.16:0/2320827170 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7ea40b3790 0x7f7ea40b3c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:41.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.073+0000 7f7ea37fe700 1 -- 172.21.15.16:0/2320827170 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7e8c003a00 msgr2=0x7f7ea4148cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:41.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.073+0000 7f7ea37fe700 1 --2- 172.21.15.16:0/2320827170 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7e8c003a00 0x7f7ea4148cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:41.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.073+0000 7f7ea37fe700 1 -- 172.21.15.16:0/2320827170 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ea41497e0 con 0x7f7ea414eef0 2023-10-16T13:47:41.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.073+0000 7f7ea37fe700 1 --2- 172.21.15.16:0/2320827170 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ea414eef0 0x7f7ea414f370 secure :-1 s=READY pgs=2800 cs=0 l=1 rev1=1 crypto rx=0x7f7e9c001330 tx=0x7f7e9c00a920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:41.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.073+0000 7f7e8b7fe700 1 -- 172.21.15.16:0/2320827170 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e9c015610 con 0x7f7ea414eef0 2023-10-16T13:47:41.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.073+0000 7f7e8b7fe700 1 -- 172.21.15.16:0/2320827170 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e9c015790 con 0x7f7ea414eef0 2023-10-16T13:47:41.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.073+0000 7f7e8b7fe700 1 -- 172.21.15.16:0/2320827170 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e9c015b40 con 0x7f7ea414eef0 2023-10-16T13:47:41.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.073+0000 7f7ea9423700 1 -- 172.21.15.16:0/2320827170 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ea4149ac0 con 0x7f7ea414eef0 2023-10-16T13:47:41.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.073+0000 7f7ea9423700 1 -- 172.21.15.16:0/2320827170 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ea414a080 con 0x7f7ea414eef0 2023-10-16T13:47:41.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.074+0000 7f7ea9423700 1 -- 172.21.15.16:0/2320827170 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ea414a5a0 con 0x7f7ea414eef0 2023-10-16T13:47:41.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.075+0000 7f7e8b7fe700 1 -- 172.21.15.16:0/2320827170 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 148) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f7e9c015ca0 con 0x7f7ea414eef0 2023-10-16T13:47:41.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.075+0000 7f7e8b7fe700 1 --2- 172.21.15.16:0/2320827170 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f7e9006b140 0x7f7e9006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:41.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.075+0000 7f7e8b7fe700 1 -- 172.21.15.16:0/2320827170 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(247..247 src has 1..247) v4 ==== 6915+0+0 (secure 0 0 0) 0x7f7e9c08cc40 con 0x7f7ea414eef0 2023-10-16T13:47:41.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.075+0000 7f7ea2ffd700 1 --2- 172.21.15.16:0/2320827170 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f7e9006b140 0x7f7e9006d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:41.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.076+0000 7f7ea2ffd700 1 --2- 172.21.15.16:0/2320827170 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f7e9006b140 0x7f7e9006d600 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7e980090b0 tx=0x7f7e98008030 comp rx=0 tx=0).ready entity=mgr.11886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:41.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.080+0000 7f7e8b7fe700 1 -- 172.21.15.16:0/2320827170 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7ea414a5a0 con 0x7f7ea414eef0 2023-10-16T13:47:41.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.330+0000 7f7ea9423700 1 -- 172.21.15.16:0/2320827170 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "backup_fs", "metadata": "backup_fs_metadata", "data": "backup_fs_data"} v 0) v1 -- 0x7f7ea414a5a0 con 0x7f7ea414eef0 2023-10-16T13:47:41.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.920+0000 7f7e8b7fe700 1 -- 172.21.15.16:0/2320827170 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "backup_fs", "metadata": "backup_fs_metadata", "data": "backup_fs_data"}]=0 new fs with metadata pool 33 and data pool 34 v132) v1 ==== 183+0+0 (secure 0 0 0) 0x7f7ea414a5a0 con 0x7f7ea414eef0 2023-10-16T13:47:41.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.925+0000 7f7ea9423700 1 -- 172.21.15.16:0/2320827170 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f7e9006b140 msgr2=0x7f7e9006d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:41.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.925+0000 7f7ea9423700 1 --2- 172.21.15.16:0/2320827170 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f7e9006b140 0x7f7e9006d600 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7e980090b0 tx=0x7f7e98008030 comp rx=0 tx=0).stop 2023-10-16T13:47:41.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.926+0000 7f7ea9423700 1 -- 172.21.15.16:0/2320827170 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ea414eef0 msgr2=0x7f7ea414f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:41.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.926+0000 7f7ea9423700 1 --2- 172.21.15.16:0/2320827170 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ea414eef0 0x7f7ea414f370 secure :-1 s=READY pgs=2800 cs=0 l=1 rev1=1 crypto rx=0x7f7e9c001330 tx=0x7f7e9c00a920 comp rx=0 tx=0).stop 2023-10-16T13:47:41.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.929+0000 7f7ea9423700 1 -- 172.21.15.16:0/2320827170 shutdown_connections 2023-10-16T13:47:41.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.929+0000 7f7ea9423700 1 --2- 172.21.15.16:0/2320827170 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f7e9006b140 0x7f7e9006d600 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:41.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.929+0000 7f7ea9423700 1 --2- 172.21.15.16:0/2320827170 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ea414eef0 0x7f7ea414f370 unknown :-1 s=CLOSED pgs=2800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:41.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.929+0000 7f7ea9423700 1 --2- 172.21.15.16:0/2320827170 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7e8c003a00 0x7f7ea4148cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:41.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.929+0000 7f7ea9423700 1 --2- 172.21.15.16:0/2320827170 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7ea40b3790 0x7f7ea40b3c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:41.932 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.929+0000 7f7ea9423700 1 -- 172.21.15.16:0/2320827170 >> 172.21.15.16:0/2320827170 conn(0x7f7ea40cc5e0 msgr2=0x7f7ea40caaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:41.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.931+0000 7f7ea9423700 1 -- 172.21.15.16:0/2320827170 shutdown_connections 2023-10-16T13:47:41.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:41.932+0000 7f7ea9423700 1 -- 172.21.15.16:0/2320827170 wait complete. 2023-10-16T13:47:41.943 INFO:teuthology.orchestra.run.smithi016.stderr:new fs with metadata pool 33 and data pool 34 2023-10-16T13:47:41.969 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:47:42.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.097+0000 7fccc657f700 1 Processor -- start 2023-10-16T13:47:42.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.097+0000 7fccc657f700 1 -- start start 2023-10-16T13:47:42.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.098+0000 7fccc657f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fccc00ce4f0 0x7fccc00cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:42.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.098+0000 7fccc657f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fccc015ba10 0x7fccc00cfca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:42.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.098+0000 7fccc657f700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccc00cc2a0 con 0x7fccc015b630 2023-10-16T13:47:42.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.098+0000 7fccc657f700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccc00cdb20 con 0x7fccc00ce4f0 2023-10-16T13:47:42.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.098+0000 7fccc657f700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccc00cc540 con 0x7fccc015ba10 2023-10-16T13:47:42.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.098+0000 7fccbf7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fccc00ce4f0 0x7fccc00cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:42.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.098+0000 7fccbf7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fccc00ce4f0 0x7fccc00cd5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55798/0 (socket says 172.21.15.16:55798) 2023-10-16T13:47:42.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.098+0000 7fccbf7fe700 1 -- 172.21.15.16:0/1000090956 learned_addr learned my addr 172.21.15.16:0/1000090956 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:42.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.098+0000 7fccc4b1c700 1 --2- 172.21.15.16:0/1000090956 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fccc015ba10 0x7fccc00cfca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:42.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.098+0000 7fccbf7fe700 1 -- 172.21.15.16:0/1000090956 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fccc015ba10 msgr2=0x7fccc00cfca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:42.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.098+0000 7fccbf7fe700 1 --2- 172.21.15.16:0/1000090956 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fccc015ba10 0x7fccc00cfca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:42.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.098+0000 7fccbf7fe700 1 -- 172.21.15.16:0/1000090956 >> v1:172.21.15.49:6789/0 conn(0x7fccc015b630 legacy=0x7fccc00cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:42.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.098+0000 7fccbf7fe700 1 -- 172.21.15.16:0/1000090956 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccc00cc820 con 0x7fccc00ce4f0 2023-10-16T13:47:42.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.098+0000 7fccc4b1c700 1 --2- 172.21.15.16:0/1000090956 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fccc015ba10 0x7fccc00cfca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:47:42.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.099+0000 7fccbf7fe700 1 --2- 172.21.15.16:0/1000090956 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccc00ce4f0 0x7fccc00cd5e0 secure :-1 s=READY pgs=2801 cs=0 l=1 rev1=1 crypto rx=0x7fccb00098f0 tx=0x7fccb0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=592eca56b912643a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:42.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.099+0000 7fccbeffd700 1 -- 172.21.15.16:0/1000090956 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb0004230 con 0x7fccc00ce4f0 2023-10-16T13:47:42.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.099+0000 7fccbeffd700 1 -- 172.21.15.16:0/1000090956 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccb0004390 con 0x7fccc00ce4f0 2023-10-16T13:47:42.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.099+0000 7fccbeffd700 1 -- 172.21.15.16:0/1000090956 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb00036a0 con 0x7fccc00ce4f0 2023-10-16T13:47:42.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.099+0000 7fccc657f700 1 -- 172.21.15.16:0/1000090956 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccc00ce4f0 msgr2=0x7fccc00cd5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:42.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.099+0000 7fccc657f700 1 --2- 172.21.15.16:0/1000090956 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccc00ce4f0 0x7fccc00cd5e0 secure :-1 s=READY pgs=2801 cs=0 l=1 rev1=1 crypto rx=0x7fccb00098f0 tx=0x7fccb0009c00 comp rx=0 tx=0).stop 2023-10-16T13:47:42.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.099+0000 7fccc657f700 1 -- 172.21.15.16:0/1000090956 shutdown_connections 2023-10-16T13:47:42.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.099+0000 7fccc657f700 1 --2- 172.21.15.16:0/1000090956 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fccc015ba10 0x7fccc00cfca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:42.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.099+0000 7fccc657f700 1 --2- 172.21.15.16:0/1000090956 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccc00ce4f0 0x7fccc00cd5e0 unknown :-1 s=CLOSED pgs=2801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:42.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.099+0000 7fccc657f700 1 -- 172.21.15.16:0/1000090956 >> 172.21.15.16:0/1000090956 conn(0x7fccc00d14b0 msgr2=0x7fccc00d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:42.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.099+0000 7fccc657f700 1 -- 172.21.15.16:0/1000090956 shutdown_connections 2023-10-16T13:47:42.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.099+0000 7fccc657f700 1 -- 172.21.15.16:0/1000090956 wait complete. 2023-10-16T13:47:42.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.101+0000 7fccc657f700 1 Processor -- start 2023-10-16T13:47:42.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.101+0000 7fccc657f700 1 -- start start 2023-10-16T13:47:42.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.102+0000 7fccc657f700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fccc00ce4f0 0x7fccc0154ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:42.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.102+0000 7fccc657f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fccc015b630 0x7fccc006c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:42.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.102+0000 7fccc657f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccc015ba10 0x7fccc00b4c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:42.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.102+0000 7fccc657f700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccc006c7f0 con 0x7fccc015ba10 2023-10-16T13:47:42.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.102+0000 7fccc657f700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccc00c6a50 con 0x7fccc00ce4f0 2023-10-16T13:47:42.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.102+0000 7fccc657f700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccc00c6d30 con 0x7fccc015b630 2023-10-16T13:47:42.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.102+0000 7fccc4b1c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccc015ba10 0x7fccc00b4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:42.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.102+0000 7fccc4b1c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccc015ba10 0x7fccc00b4c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55806/0 (socket says 172.21.15.16:55806) 2023-10-16T13:47:42.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.102+0000 7fccc4b1c700 1 -- 172.21.15.16:0/1596614471 learned_addr learned my addr 172.21.15.16:0/1596614471 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:42.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.102+0000 7fccbffff700 1 --2- 172.21.15.16:0/1596614471 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fccc00ce4f0 0x7fccc0154ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:42.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.102+0000 7fccbf7fe700 1 --2- 172.21.15.16:0/1596614471 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fccc015b630 0x7fccc006c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:42.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.102+0000 7fccc4b1c700 1 -- 172.21.15.16:0/1596614471 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fccc015b630 msgr2=0x7fccc006c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:42.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.102+0000 7fccc4b1c700 1 --2- 172.21.15.16:0/1596614471 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fccc015b630 0x7fccc006c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:42.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.102+0000 7fccc4b1c700 1 -- 172.21.15.16:0/1596614471 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fccc00ce4f0 msgr2=0x7fccc0154ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:42.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.102+0000 7fccc4b1c700 1 --2- 172.21.15.16:0/1596614471 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fccc00ce4f0 0x7fccc0154ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:42.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.102+0000 7fccc4b1c700 1 -- 172.21.15.16:0/1596614471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccb0009550 con 0x7fccc015ba10 2023-10-16T13:47:42.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.103+0000 7fccc4b1c700 1 --2- 172.21.15.16:0/1596614471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccc015ba10 0x7fccc00b4c60 secure :-1 s=READY pgs=2802 cs=0 l=1 rev1=1 crypto rx=0x7fccb800cef0 tx=0x7fccb800cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:42.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.103+0000 7fccbcff9700 1 -- 172.21.15.16:0/1596614471 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb800ad00 con 0x7fccc015ba10 2023-10-16T13:47:42.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.103+0000 7fccbcff9700 1 -- 172.21.15.16:0/1596614471 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccb800ae60 con 0x7fccc015ba10 2023-10-16T13:47:42.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.103+0000 7fccbcff9700 1 -- 172.21.15.16:0/1596614471 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb8015680 con 0x7fccc015ba10 2023-10-16T13:47:42.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.103+0000 7fccc657f700 1 -- 172.21.15.16:0/1596614471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccc00c7010 con 0x7fccc015ba10 2023-10-16T13:47:42.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.103+0000 7fccc657f700 1 -- 172.21.15.16:0/1596614471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccc00c8e20 con 0x7fccc015ba10 2023-10-16T13:47:42.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.104+0000 7fccc657f700 1 -- 172.21.15.16:0/1596614471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccc00c7530 con 0x7fccc015ba10 2023-10-16T13:47:42.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.104+0000 7fccbcff9700 1 -- 172.21.15.16:0/1596614471 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 148) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fccb800d430 con 0x7fccc015ba10 2023-10-16T13:47:42.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.105+0000 7fccbcff9700 1 --2- 172.21.15.16:0/1596614471 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fcca006b140 0x7fcca006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:42.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.105+0000 7fccbcff9700 1 -- 172.21.15.16:0/1596614471 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 7021+0+0 (secure 0 0 0) 0x7fccb808a360 con 0x7fccc015ba10 2023-10-16T13:47:42.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.105+0000 7fccbffff700 1 --2- 172.21.15.16:0/1596614471 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fcca006b140 0x7fcca006d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:42.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.105+0000 7fccbffff700 1 --2- 172.21.15.16:0/1596614471 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fcca006b140 0x7fcca006d600 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fcca80050c0 tx=0x7fcca8007f10 comp rx=0 tx=0).ready entity=mgr.11886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:42.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.109+0000 7fccbcff9700 1 -- 172.21.15.16:0/1596614471 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fccb8054c20 con 0x7fccc015ba10 2023-10-16T13:47:42.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.313+0000 7fccc657f700 1 -- 172.21.15.16:0/1596614471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fccc004ebb0 con 0x7fccc015ba10 2023-10-16T13:47:42.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.315+0000 7fccbcff9700 1 -- 172.21.15.16:0/1596614471 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v248) v1 ==== 74+0+20554 (secure 0 0 0) 0x7fccb8058240 con 0x7fccc015ba10 2023-10-16T13:47:42.317 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:47:42.318 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":248,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:47:41.331754+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2023-10-16T13:47:22.338349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"244","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2023-10-16T13:47:22.806327+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"245","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T13:47:39.075528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"248","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"backup_fs_data","create_time":"2023-10-16T13:47:40.314029+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"248","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:47:42.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.317+0000 7fccc657f700 1 -- 172.21.15.16:0/1596614471 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fcca006b140 msgr2=0x7fcca006d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:42.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.317+0000 7fccc657f700 1 --2- 172.21.15.16:0/1596614471 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fcca006b140 0x7fcca006d600 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fcca80050c0 tx=0x7fcca8007f10 comp rx=0 tx=0).stop 2023-10-16T13:47:42.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.317+0000 7fccc657f700 1 -- 172.21.15.16:0/1596614471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccc015ba10 msgr2=0x7fccc00b4c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:42.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.317+0000 7fccc657f700 1 --2- 172.21.15.16:0/1596614471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccc015ba10 0x7fccc00b4c60 secure :-1 s=READY pgs=2802 cs=0 l=1 rev1=1 crypto rx=0x7fccb800cef0 tx=0x7fccb800cf20 comp rx=0 tx=0).stop 2023-10-16T13:47:42.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.317+0000 7fccc657f700 1 -- 172.21.15.16:0/1596614471 shutdown_connections 2023-10-16T13:47:42.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.317+0000 7fccc657f700 1 --2- 172.21.15.16:0/1596614471 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fcca006b140 0x7fcca006d600 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:42.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.317+0000 7fccc657f700 1 --2- 172.21.15.16:0/1596614471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccc015ba10 0x7fccc00b4c60 unknown :-1 s=CLOSED pgs=2802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:42.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.317+0000 7fccc657f700 1 --2- 172.21.15.16:0/1596614471 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fccc00ce4f0 0x7fccc0154ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:42.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.317+0000 7fccc657f700 1 --2- 172.21.15.16:0/1596614471 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fccc015b630 0x7fccc006c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:42.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.317+0000 7fccc657f700 1 -- 172.21.15.16:0/1596614471 >> 172.21.15.16:0/1596614471 conn(0x7fccc00d14b0 msgr2=0x7fccc00b4300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:42.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.317+0000 7fccc657f700 1 -- 172.21.15.16:0/1596614471 shutdown_connections 2023-10-16T13:47:42.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.317+0000 7fccc657f700 1 -- 172.21.15.16:0/1596614471 wait complete. 2023-10-16T13:47:42.332 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:47:42.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.455+0000 7f0c09f57700 1 Processor -- start 2023-10-16T13:47:42.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.455+0000 7f0c09f57700 1 -- start start 2023-10-16T13:47:42.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.456+0000 7f0c09f57700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0c0415db30 0x7f0c040cf920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:42.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.456+0000 7f0c09f57700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c040d0070 con 0x7f0c04161490 2023-10-16T13:47:42.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.456+0000 7f0c09f57700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c040d0350 con 0x7f0c0415d750 2023-10-16T13:47:42.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.456+0000 7f0c09f57700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c040cfe60 con 0x7f0c0415db30 2023-10-16T13:47:42.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.456+0000 7f0c03fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0c04161490 0x7f0c040b2060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:45088/0 (socket says 172.21.15.16:45088) 2023-10-16T13:47:42.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.456+0000 7f0c03fff700 1 -- 172.21.15.16:0/2794719899 learned_addr learned my addr 172.21.15.16:0/2794719899 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:47:42.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.456+0000 7f0c02ffd700 1 --2- 172.21.15.16:0/2794719899 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0c0415db30 0x7f0c040cf920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:42.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.456+0000 7f0c027fc700 1 -- 172.21.15.16:0/2794719899 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1765501151 0 0) 0x7f0c040d0070 con 0x7f0c04161490 2023-10-16T13:47:42.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.456+0000 7f0c027fc700 1 -- 172.21.15.16:0/2794719899 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bf00035b0 con 0x7f0c04161490 2023-10-16T13:47:42.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.457+0000 7f0c027fc700 1 -- 172.21.15.16:0/2794719899 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2861731539 0 0) 0x7f0c040d0350 con 0x7f0c0415d750 2023-10-16T13:47:42.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.457+0000 7f0c027fc700 1 -- 172.21.15.16:0/2794719899 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c040d0070 con 0x7f0c0415d750 2023-10-16T13:47:42.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.457+0000 7f0c02ffd700 1 -- 172.21.15.16:0/2794719899 >> v1:172.21.15.49:6789/0 conn(0x7f0c0415d750 legacy=0x7f0c0406c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:42.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.457+0000 7f0c02ffd700 1 -- 172.21.15.16:0/2794719899 >> v1:172.21.15.16:6789/0 conn(0x7f0c04161490 legacy=0x7f0c040b2060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:42.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.457+0000 7f0c027fc700 1 -- 172.21.15.16:0/2794719899 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3180322794 0 0) 0x7f0bf00035b0 con 0x7f0c04161490 2023-10-16T13:47:42.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.457+0000 7f0c02ffd700 1 -- 172.21.15.16:0/2794719899 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c040d0630 con 0x7f0c0415db30 2023-10-16T13:47:42.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.457+0000 7f0c02ffd700 1 --2- 172.21.15.16:0/2794719899 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0c0415db30 0x7f0c040cf920 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f0bec009870 tx=0x7f0bec009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=26deb97f354e46f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:42.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.457+0000 7f0c027fc700 1 -- 172.21.15.16:0/2794719899 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bfc0019f0 con 0x7f0c0415db30 2023-10-16T13:47:42.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.457+0000 7f0c027fc700 1 -- 172.21.15.16:0/2794719899 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bec004030 con 0x7f0c0415db30 2023-10-16T13:47:42.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.457+0000 7f0c027fc700 1 -- 172.21.15.16:0/2794719899 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bec0043c0 con 0x7f0c0415db30 2023-10-16T13:47:42.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.458+0000 7f0c09f57700 1 -- 172.21.15.16:0/2794719899 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0c0415db30 msgr2=0x7f0c040cf920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:42.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.458+0000 7f0c09f57700 1 --2- 172.21.15.16:0/2794719899 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0c0415db30 0x7f0c040cf920 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f0bec009870 tx=0x7f0bec009b80 comp rx=0 tx=0).stop 2023-10-16T13:47:42.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.458+0000 7f0c09f57700 1 -- 172.21.15.16:0/2794719899 shutdown_connections 2023-10-16T13:47:42.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.458+0000 7f0c09f57700 1 --2- 172.21.15.16:0/2794719899 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0c0415db30 0x7f0c040cf920 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:42.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.458+0000 7f0c09f57700 1 -- 172.21.15.16:0/2794719899 >> 172.21.15.16:0/2794719899 conn(0x7f0c040d14b0 msgr2=0x7f0c040d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:42.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.458+0000 7f0c09f57700 1 -- 172.21.15.16:0/2794719899 shutdown_connections 2023-10-16T13:47:42.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.458+0000 7f0c09f57700 1 -- 172.21.15.16:0/2794719899 wait complete. 2023-10-16T13:47:42.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.459+0000 7f0c09f57700 1 Processor -- start 2023-10-16T13:47:42.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.459+0000 7f0c09f57700 1 -- start start 2023-10-16T13:47:42.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.460+0000 7f0c09f57700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c0415d750 0x7f0c040c4f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:42.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.460+0000 7f0c09f57700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0c0415db30 0x7f0c0409cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:42.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.460+0000 7f0c09f57700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0c04161490 0x7f0c041f39b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:42.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.460+0000 7f0c09f57700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c0409d1c0 con 0x7f0c0415d750 2023-10-16T13:47:42.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.460+0000 7f0c09f57700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c0414e1c0 con 0x7f0c04161490 2023-10-16T13:47:42.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.460+0000 7f0c09f57700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c041f3ef0 con 0x7f0c0415db30 2023-10-16T13:47:42.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.460+0000 7f0c037fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c0415d750 0x7f0c040c4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:42.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.460+0000 7f0c037fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c0415d750 0x7f0c040c4f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55816/0 (socket says 172.21.15.16:55816) 2023-10-16T13:47:42.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.460+0000 7f0c037fe700 1 -- 172.21.15.16:0/2941565517 learned_addr learned my addr 172.21.15.16:0/2941565517 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:42.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.460+0000 7f0c02ffd700 1 --2- 172.21.15.16:0/2941565517 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0c0415db30 0x7f0c0409cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:42.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.460+0000 7f0c03fff700 1 --2- 172.21.15.16:0/2941565517 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0c04161490 0x7f0c041f39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:42.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.460+0000 7f0c037fe700 1 -- 172.21.15.16:0/2941565517 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0c0415db30 msgr2=0x7f0c0409cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:42.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.460+0000 7f0c037fe700 1 --2- 172.21.15.16:0/2941565517 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0c0415db30 0x7f0c0409cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:42.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.460+0000 7f0c037fe700 1 -- 172.21.15.16:0/2941565517 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0c04161490 msgr2=0x7f0c041f39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:42.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.460+0000 7f0c037fe700 1 --2- 172.21.15.16:0/2941565517 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0c04161490 0x7f0c041f39b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:42.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.460+0000 7f0c037fe700 1 -- 172.21.15.16:0/2941565517 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bec0094a0 con 0x7f0c0415d750 2023-10-16T13:47:42.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.460+0000 7f0c03fff700 1 --2- 172.21.15.16:0/2941565517 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0c04161490 0x7f0c041f39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:47:42.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.460+0000 7f0c037fe700 1 --2- 172.21.15.16:0/2941565517 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c0415d750 0x7f0c040c4f10 secure :-1 s=READY pgs=2804 cs=0 l=1 rev1=1 crypto rx=0x7f0bf400c700 tx=0x7f0bf400ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:42.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.460+0000 7f0bfbfff700 1 -- 172.21.15.16:0/2941565517 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf40052d0 con 0x7f0c0415d750 2023-10-16T13:47:42.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.460+0000 7f0bfbfff700 1 -- 172.21.15.16:0/2941565517 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bf4005430 con 0x7f0c0415d750 2023-10-16T13:47:42.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.461+0000 7f0c09f57700 1 -- 172.21.15.16:0/2941565517 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c040c6070 con 0x7f0c0415d750 2023-10-16T13:47:42.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.461+0000 7f0bfbfff700 1 -- 172.21.15.16:0/2941565517 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf40038f0 con 0x7f0c0415d750 2023-10-16T13:47:42.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.461+0000 7f0c09f57700 1 -- 172.21.15.16:0/2941565517 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c040c57d0 con 0x7f0c0415d750 2023-10-16T13:47:42.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.462+0000 7f0c09f57700 1 -- 172.21.15.16:0/2941565517 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c0404ebb0 con 0x7f0c0415d750 2023-10-16T13:47:42.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.462+0000 7f0bfbfff700 1 -- 172.21.15.16:0/2941565517 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 148) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f0bf4003a50 con 0x7f0c0415d750 2023-10-16T13:47:42.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.462+0000 7f0bfbfff700 1 --2- 172.21.15.16:0/2941565517 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f0be406b140 0x7f0be406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:42.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.463+0000 7f0bfbfff700 1 -- 172.21.15.16:0/2941565517 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 7021+0+0 (secure 0 0 0) 0x7f0bf4089360 con 0x7f0c0415d750 2023-10-16T13:47:42.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.463+0000 7f0c02ffd700 1 --2- 172.21.15.16:0/2941565517 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f0be406b140 0x7f0be406d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:42.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.463+0000 7f0c02ffd700 1 --2- 172.21.15.16:0/2941565517 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f0be406b140 0x7f0be406d600 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0c0415df60 tx=0x7f0bec008030 comp rx=0 tx=0).ready entity=mgr.11886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:42.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.467+0000 7f0bfbfff700 1 -- 172.21.15.16:0/2941565517 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0bf4053c60 con 0x7f0c0415d750 2023-10-16T13:47:42.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.673+0000 7f0c09f57700 1 -- 172.21.15.16:0/2941565517 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0c0406bd00 con 0x7f0c0415d750 2023-10-16T13:47:42.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.675+0000 7f0bfbfff700 1 -- 172.21.15.16:0/2941565517 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v248) v1 ==== 74+0+20554 (secure 0 0 0) 0x7f0bf4057280 con 0x7f0c0415d750 2023-10-16T13:47:42.677 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:47:42.678 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":248,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:47:41.331754+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2023-10-16T13:47:22.338349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"244","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2023-10-16T13:47:22.806327+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"245","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T13:47:39.075528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"248","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"backup_fs_data","create_time":"2023-10-16T13:47:40.314029+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"248","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:47:42.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.677+0000 7f0c09f57700 1 -- 172.21.15.16:0/2941565517 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f0be406b140 msgr2=0x7f0be406d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:42.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.677+0000 7f0c09f57700 1 --2- 172.21.15.16:0/2941565517 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f0be406b140 0x7f0be406d600 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0c0415df60 tx=0x7f0bec008030 comp rx=0 tx=0).stop 2023-10-16T13:47:42.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.677+0000 7f0c09f57700 1 -- 172.21.15.16:0/2941565517 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c0415d750 msgr2=0x7f0c040c4f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:42.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.677+0000 7f0c09f57700 1 --2- 172.21.15.16:0/2941565517 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c0415d750 0x7f0c040c4f10 secure :-1 s=READY pgs=2804 cs=0 l=1 rev1=1 crypto rx=0x7f0bf400c700 tx=0x7f0bf400ca10 comp rx=0 tx=0).stop 2023-10-16T13:47:42.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.677+0000 7f0c09f57700 1 -- 172.21.15.16:0/2941565517 shutdown_connections 2023-10-16T13:47:42.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.677+0000 7f0c09f57700 1 --2- 172.21.15.16:0/2941565517 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f0be406b140 0x7f0be406d600 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:42.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.677+0000 7f0c09f57700 1 --2- 172.21.15.16:0/2941565517 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0c04161490 0x7f0c041f39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:42.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.677+0000 7f0c09f57700 1 --2- 172.21.15.16:0/2941565517 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c0415d750 0x7f0c040c4f10 unknown :-1 s=CLOSED pgs=2804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:42.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.677+0000 7f0c09f57700 1 --2- 172.21.15.16:0/2941565517 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0c0415db30 0x7f0c0409cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:42.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.677+0000 7f0c09f57700 1 -- 172.21.15.16:0/2941565517 >> 172.21.15.16:0/2941565517 conn(0x7f0c040d14b0 msgr2=0x7f0c0415df90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:42.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.677+0000 7f0c09f57700 1 -- 172.21.15.16:0/2941565517 shutdown_connections 2023-10-16T13:47:42.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.677+0000 7f0c09f57700 1 -- 172.21.15.16:0/2941565517 wait complete. 2023-10-16T13:47:42.692 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set backup_fs standby_count_wanted 0 2023-10-16T13:47:42.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.817+0000 7f0a138e2700 1 Processor -- start 2023-10-16T13:47:42.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.818+0000 7f0a138e2700 1 -- start start 2023-10-16T13:47:42.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.819+0000 7f0a138e2700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0a0c159de0 0x7f0a0c15a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:42.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.819+0000 7f0a138e2700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f0a0c0cb1c0 0x7f0a0c0cb670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:42.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.819+0000 7f0a138e2700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a0c15dda0 con 0x7f0a0c0cf4c0 2023-10-16T13:47:42.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.819+0000 7f0a138e2700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a0c15ea20 con 0x7f0a0c0cb1c0 2023-10-16T13:47:42.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.819+0000 7f0a138e2700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a0c0cbbb0 con 0x7f0a0c159de0 2023-10-16T13:47:42.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.819+0000 7f0a1167e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0a0c159de0 0x7f0a0c15a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:42.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.819+0000 7f0a11e7f700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f0a0c0cb1c0 0x7f0a0c0cb670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:42.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.819+0000 7f0a1167e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0a0c159de0 0x7f0a0c15a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:44082/0 (socket says 172.21.15.16:44082) 2023-10-16T13:47:42.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.819+0000 7f0a1167e700 1 -- 172.21.15.16:0/233783954 learned_addr learned my addr 172.21.15.16:0/233783954 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:42.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.819+0000 7f0a10e7d700 1 --1- 172.21.15.16:0/233783954 >> v1:172.21.15.49:6789/0 conn(0x7f0a0c0cf4c0 0x7f0a0c06aac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:52280/0 (socket says 172.21.15.16:52280) 2023-10-16T13:47:42.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.819+0000 7f0a1167e700 1 -- 172.21.15.16:0/233783954 >> v2:172.21.15.49:3300/0 conn(0x7f0a0c0cb1c0 msgr2=0x7f0a0c0cb670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:42.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.819+0000 7f0a1167e700 1 --2- 172.21.15.16:0/233783954 >> v2:172.21.15.49:3300/0 conn(0x7f0a0c0cb1c0 0x7f0a0c0cb670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:42.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.819+0000 7f0a1167e700 1 -- 172.21.15.16:0/233783954 >> v1:172.21.15.49:6789/0 conn(0x7f0a0c0cf4c0 legacy=0x7f0a0c06aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:42.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.819+0000 7f0a1167e700 1 -- 172.21.15.16:0/233783954 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a0c15e080 con 0x7f0a0c159de0 2023-10-16T13:47:42.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.819+0000 7f0a11e7f700 1 --2- 172.21.15.16:0/233783954 >> v2:172.21.15.49:3300/0 conn(0x7f0a0c0cb1c0 0x7f0a0c0cb670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:47:42.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.819+0000 7f0a1167e700 1 --2- 172.21.15.16:0/233783954 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0a0c159de0 0x7f0a0c15a1c0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f0a080098f0 tx=0x7f0a08009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=96e20d93ca4d0f18 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:42.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.820+0000 7f0a03fff700 1 -- 172.21.15.16:0/233783954 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a08011830 con 0x7f0a0c159de0 2023-10-16T13:47:42.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.820+0000 7f0a03fff700 1 -- 172.21.15.16:0/233783954 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a08011990 con 0x7f0a0c159de0 2023-10-16T13:47:42.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.820+0000 7f0a03fff700 1 -- 172.21.15.16:0/233783954 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a08011d20 con 0x7f0a0c159de0 2023-10-16T13:47:42.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.820+0000 7f0a138e2700 1 -- 172.21.15.16:0/233783954 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0a0c159de0 msgr2=0x7f0a0c15a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:42.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.820+0000 7f0a138e2700 1 --2- 172.21.15.16:0/233783954 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0a0c159de0 0x7f0a0c15a1c0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f0a080098f0 tx=0x7f0a08009c00 comp rx=0 tx=0).stop 2023-10-16T13:47:42.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.820+0000 7f0a138e2700 1 -- 172.21.15.16:0/233783954 shutdown_connections 2023-10-16T13:47:42.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.820+0000 7f0a138e2700 1 --2- 172.21.15.16:0/233783954 >> v2:172.21.15.49:3300/0 conn(0x7f0a0c0cb1c0 0x7f0a0c0cb670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:42.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.820+0000 7f0a138e2700 1 --2- 172.21.15.16:0/233783954 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0a0c159de0 0x7f0a0c15a1c0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:42.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.820+0000 7f0a138e2700 1 -- 172.21.15.16:0/233783954 >> 172.21.15.16:0/233783954 conn(0x7f0a0c0cc5e0 msgr2=0x7f0a0c0cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:42.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.820+0000 7f0a138e2700 1 -- 172.21.15.16:0/233783954 shutdown_connections 2023-10-16T13:47:42.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.820+0000 7f0a138e2700 1 -- 172.21.15.16:0/233783954 wait complete. 2023-10-16T13:47:42.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.822+0000 7f0a138e2700 1 Processor -- start 2023-10-16T13:47:42.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.822+0000 7f0a138e2700 1 -- start start 2023-10-16T13:47:42.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.823+0000 7f0a138e2700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0a0c0cb1c0 0x7f0a0c0c5d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:42.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.823+0000 7f0a138e2700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a0c0cf4c0 0x7f0a0c0ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:42.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.823+0000 7f0a138e2700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0a0c159de0 0x7f0a0c0c6c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:42.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.823+0000 7f0a138e2700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a0c0ade50 con 0x7f0a0c0cf4c0 2023-10-16T13:47:42.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.823+0000 7f0a138e2700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a0c0ad240 con 0x7f0a0c159de0 2023-10-16T13:47:42.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.823+0000 7f0a138e2700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a0c0ad520 con 0x7f0a0c0cb1c0 2023-10-16T13:47:42.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.823+0000 7f0a10e7d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a0c0cf4c0 0x7f0a0c0ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:42.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.823+0000 7f0a10e7d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a0c0cf4c0 0x7f0a0c0ac8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55830/0 (socket says 172.21.15.16:55830) 2023-10-16T13:47:42.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.823+0000 7f0a10e7d700 1 -- 172.21.15.16:0/805280992 learned_addr learned my addr 172.21.15.16:0/805280992 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:42.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.823+0000 7f0a1167e700 1 --2- 172.21.15.16:0/805280992 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0a0c0cb1c0 0x7f0a0c0c5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:42.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.823+0000 7f0a11e7f700 1 --2- 172.21.15.16:0/805280992 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0a0c159de0 0x7f0a0c0c6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:42.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.823+0000 7f0a10e7d700 1 -- 172.21.15.16:0/805280992 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0a0c0cb1c0 msgr2=0x7f0a0c0c5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:42.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.823+0000 7f0a10e7d700 1 --2- 172.21.15.16:0/805280992 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0a0c0cb1c0 0x7f0a0c0c5d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:42.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.823+0000 7f0a10e7d700 1 -- 172.21.15.16:0/805280992 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0a0c159de0 msgr2=0x7f0a0c0c6c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:42.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.823+0000 7f0a10e7d700 1 --2- 172.21.15.16:0/805280992 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0a0c159de0 0x7f0a0c0c6c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:42.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.824+0000 7f0a10e7d700 1 -- 172.21.15.16:0/805280992 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a04009a70 con 0x7f0a0c0cf4c0 2023-10-16T13:47:42.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.824+0000 7f0a10e7d700 1 --2- 172.21.15.16:0/805280992 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a0c0cf4c0 0x7f0a0c0ac8d0 secure :-1 s=READY pgs=2805 cs=0 l=1 rev1=1 crypto rx=0x7f09fc00c520 tx=0x7f09fc00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:42.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.824+0000 7f0a01ffb700 1 -- 172.21.15.16:0/805280992 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09fc005cb0 con 0x7f0a0c0cf4c0 2023-10-16T13:47:42.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.824+0000 7f0a01ffb700 1 -- 172.21.15.16:0/805280992 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09fc005e10 con 0x7f0a0c0cf4c0 2023-10-16T13:47:42.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.824+0000 7f0a01ffb700 1 -- 172.21.15.16:0/805280992 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09fc0072d0 con 0x7f0a0c0cf4c0 2023-10-16T13:47:42.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.824+0000 7f0a138e2700 1 -- 172.21.15.16:0/805280992 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a08009550 con 0x7f0a0c0cf4c0 2023-10-16T13:47:42.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.824+0000 7f0a138e2700 1 -- 172.21.15.16:0/805280992 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a0c0adae0 con 0x7f0a0c0cf4c0 2023-10-16T13:47:42.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.825+0000 7f0a138e2700 1 -- 172.21.15.16:0/805280992 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a0c0ae000 con 0x7f0a0c0cf4c0 2023-10-16T13:47:42.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.826+0000 7f0a01ffb700 1 -- 172.21.15.16:0/805280992 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 148) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f09fc007430 con 0x7f0a0c0cf4c0 2023-10-16T13:47:42.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.826+0000 7f0a01ffb700 1 --2- 172.21.15.16:0/805280992 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f09f006b140 0x7f09f006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:42.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.826+0000 7f0a01ffb700 1 -- 172.21.15.16:0/805280992 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 7021+0+0 (secure 0 0 0) 0x7f09fc0890e0 con 0x7f0a0c0cf4c0 2023-10-16T13:47:42.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.826+0000 7f0a1167e700 1 --2- 172.21.15.16:0/805280992 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f09f006b140 0x7f09f006d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:42.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.826+0000 7f0a1167e700 1 --2- 172.21.15.16:0/805280992 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f09f006b140 0x7f09f006d600 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0a08009c40 tx=0x7f0a0801d040 comp rx=0 tx=0).ready entity=mgr.11886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:42.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:42.830+0000 7f0a01ffb700 1 -- 172.21.15.16:0/805280992 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0a0c0ae000 con 0x7f0a0c0cf4c0 2023-10-16T13:47:43.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:43.084+0000 7f0a138e2700 1 -- 172.21.15.16:0/805280992 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "backup_fs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f0a0c04ebb0 con 0x7f0a0c0cf4c0 2023-10-16T13:47:43.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:43.951+0000 7f0a01ffb700 1 -- 172.21.15.16:0/805280992 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "backup_fs", "var": "standby_count_wanted", "val": "0"}]=0 v134) v1 ==== 121+0+0 (secure 0 0 0) 0x7f09fc0170c0 con 0x7f0a0c0cf4c0 2023-10-16T13:47:43.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:43.954+0000 7f0a138e2700 1 -- 172.21.15.16:0/805280992 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f09f006b140 msgr2=0x7f09f006d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:43.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:43.954+0000 7f0a138e2700 1 --2- 172.21.15.16:0/805280992 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f09f006b140 0x7f09f006d600 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0a08009c40 tx=0x7f0a0801d040 comp rx=0 tx=0).stop 2023-10-16T13:47:43.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:43.954+0000 7f0a138e2700 1 -- 172.21.15.16:0/805280992 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a0c0cf4c0 msgr2=0x7f0a0c0ac8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:43.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:43.954+0000 7f0a138e2700 1 --2- 172.21.15.16:0/805280992 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a0c0cf4c0 0x7f0a0c0ac8d0 secure :-1 s=READY pgs=2805 cs=0 l=1 rev1=1 crypto rx=0x7f09fc00c520 tx=0x7f09fc00c830 comp rx=0 tx=0).stop 2023-10-16T13:47:43.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:43.954+0000 7f0a138e2700 1 -- 172.21.15.16:0/805280992 shutdown_connections 2023-10-16T13:47:43.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:43.954+0000 7f0a138e2700 1 --2- 172.21.15.16:0/805280992 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f09f006b140 0x7f09f006d600 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:43.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:43.954+0000 7f0a138e2700 1 --2- 172.21.15.16:0/805280992 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0a0c159de0 0x7f0a0c0c6c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:43.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:43.954+0000 7f0a138e2700 1 --2- 172.21.15.16:0/805280992 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0a0c0cb1c0 0x7f0a0c0c5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:43.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:43.954+0000 7f0a138e2700 1 --2- 172.21.15.16:0/805280992 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a0c0cf4c0 0x7f0a0c0ac8d0 unknown :-1 s=CLOSED pgs=2805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:43.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:43.954+0000 7f0a138e2700 1 -- 172.21.15.16:0/805280992 >> 172.21.15.16:0/805280992 conn(0x7f0a0c0cc5e0 msgr2=0x7f0a0c15a1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:43.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:43.955+0000 7f0a138e2700 1 -- 172.21.15.16:0/805280992 shutdown_connections 2023-10-16T13:47:43.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:43.955+0000 7f0a138e2700 1 -- 172.21.15.16:0/805280992 wait complete. 2023-10-16T13:47:43.972 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-16T13:47:43.973 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set backup_fs allow_standby_replay false 2023-10-16T13:47:44.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.105+0000 7f4db95a0700 1 Processor -- start 2023-10-16T13:47:44.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.105+0000 7f4db95a0700 1 -- start start 2023-10-16T13:47:44.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.106+0000 7f4db95a0700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4db415b490 0x7f4db406b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:44.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.106+0000 7f4db95a0700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4db40cd7a0 0x7f4db40cddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:44.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.106+0000 7f4db95a0700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f4db40ce310 0x7f4db40717e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:44.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.106+0000 7f4db95a0700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4db4161fa0 con 0x7f4db40cd7a0 2023-10-16T13:47:44.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.106+0000 7f4db95a0700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4db40ce7a0 con 0x7f4db40ce310 2023-10-16T13:47:44.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.106+0000 7f4db95a0700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4db406b6e0 con 0x7f4db415b490 2023-10-16T13:47:44.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.106+0000 7f4db37fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4db40cd7a0 0x7f4db40cddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:44.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.106+0000 7f4db37fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4db40cd7a0 0x7f4db40cddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55836/0 (socket says 172.21.15.16:55836) 2023-10-16T13:47:44.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.106+0000 7f4db37fe700 1 -- 172.21.15.16:0/2475109369 learned_addr learned my addr 172.21.15.16:0/2475109369 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:44.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.106+0000 7f4db8d9f700 1 --2- 172.21.15.16:0/2475109369 >> v2:172.21.15.49:3300/0 conn(0x7f4db40ce310 0x7f4db40717e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:44.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.107+0000 7f4db3fff700 1 --2- 172.21.15.16:0/2475109369 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4db415b490 0x7f4db406b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:44.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.107+0000 7f4db37fe700 1 -- 172.21.15.16:0/2475109369 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4db415b490 msgr2=0x7f4db406b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:44.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.107+0000 7f4db37fe700 1 --2- 172.21.15.16:0/2475109369 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4db415b490 0x7f4db406b1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:44.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.107+0000 7f4db37fe700 1 -- 172.21.15.16:0/2475109369 >> v2:172.21.15.49:3300/0 conn(0x7f4db40ce310 msgr2=0x7f4db40717e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:44.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.107+0000 7f4db37fe700 1 --2- 172.21.15.16:0/2475109369 >> v2:172.21.15.49:3300/0 conn(0x7f4db40ce310 0x7f4db40717e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:44.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.107+0000 7f4db37fe700 1 -- 172.21.15.16:0/2475109369 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4db4161320 con 0x7f4db40cd7a0 2023-10-16T13:47:44.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.107+0000 7f4db8d9f700 1 --2- 172.21.15.16:0/2475109369 >> v2:172.21.15.49:3300/0 conn(0x7f4db40ce310 0x7f4db40717e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:47:44.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.107+0000 7f4db3fff700 1 --2- 172.21.15.16:0/2475109369 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4db415b490 0x7f4db406b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:47:44.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.107+0000 7f4db37fe700 1 --2- 172.21.15.16:0/2475109369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4db40cd7a0 0x7f4db40cddd0 secure :-1 s=READY pgs=2806 cs=0 l=1 rev1=1 crypto rx=0x7f4db40b2d70 tx=0x7f4d9c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c0818d43f0c4955 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:44.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.107+0000 7f4db2ffd700 1 -- 172.21.15.16:0/2475109369 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d9c00b870 con 0x7f4db40cd7a0 2023-10-16T13:47:44.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.107+0000 7f4db2ffd700 1 -- 172.21.15.16:0/2475109369 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d9c00b9d0 con 0x7f4db40cd7a0 2023-10-16T13:47:44.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.107+0000 7f4db2ffd700 1 -- 172.21.15.16:0/2475109369 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d9c00bd60 con 0x7f4db40cd7a0 2023-10-16T13:47:44.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.108+0000 7f4db95a0700 1 -- 172.21.15.16:0/2475109369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4db40cd7a0 msgr2=0x7f4db40cddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:44.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.108+0000 7f4db95a0700 1 --2- 172.21.15.16:0/2475109369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4db40cd7a0 0x7f4db40cddd0 secure :-1 s=READY pgs=2806 cs=0 l=1 rev1=1 crypto rx=0x7f4db40b2d70 tx=0x7f4d9c009bd0 comp rx=0 tx=0).stop 2023-10-16T13:47:44.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.108+0000 7f4db95a0700 1 -- 172.21.15.16:0/2475109369 shutdown_connections 2023-10-16T13:47:44.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.108+0000 7f4db95a0700 1 --2- 172.21.15.16:0/2475109369 >> v2:172.21.15.49:3300/0 conn(0x7f4db40ce310 0x7f4db40717e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:44.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.108+0000 7f4db95a0700 1 --2- 172.21.15.16:0/2475109369 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4db415b490 0x7f4db406b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:44.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.108+0000 7f4db95a0700 1 --2- 172.21.15.16:0/2475109369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4db40cd7a0 0x7f4db40cddd0 unknown :-1 s=CLOSED pgs=2806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:44.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.108+0000 7f4db95a0700 1 -- 172.21.15.16:0/2475109369 >> 172.21.15.16:0/2475109369 conn(0x7f4db40ccc80 msgr2=0x7f4db40cd090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:44.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.108+0000 7f4db95a0700 1 -- 172.21.15.16:0/2475109369 shutdown_connections 2023-10-16T13:47:44.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.108+0000 7f4db95a0700 1 -- 172.21.15.16:0/2475109369 wait complete. 2023-10-16T13:47:44.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.109+0000 7f4db95a0700 1 Processor -- start 2023-10-16T13:47:44.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.109+0000 7f4db95a0700 1 -- start start 2023-10-16T13:47:44.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.110+0000 7f4db95a0700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4db40cd7a0 0x7f4db40b4140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:44.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.110+0000 7f4db95a0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4db40ce310 0x7f4db40c9860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:44.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.110+0000 7f4db95a0700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4db415b490 0x7f4db41f6540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:44.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.110+0000 7f4db95a0700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db40c7370 con 0x7f4db40ce310 2023-10-16T13:47:44.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.110+0000 7f4db95a0700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db40c7650 con 0x7f4db40cd7a0 2023-10-16T13:47:44.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.110+0000 7f4db95a0700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4db40c7930 con 0x7f4db415b490 2023-10-16T13:47:44.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.110+0000 7f4db37fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4db40ce310 0x7f4db40c9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:44.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.110+0000 7f4db37fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4db40ce310 0x7f4db40c9860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55844/0 (socket says 172.21.15.16:55844) 2023-10-16T13:47:44.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.110+0000 7f4db37fe700 1 -- 172.21.15.16:0/3693628379 learned_addr learned my addr 172.21.15.16:0/3693628379 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:44.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.110+0000 7f4db3fff700 1 --2- 172.21.15.16:0/3693628379 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4db40cd7a0 0x7f4db40b4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:44.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.110+0000 7f4db8d9f700 1 --2- 172.21.15.16:0/3693628379 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4db415b490 0x7f4db41f6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:44.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.110+0000 7f4db37fe700 1 -- 172.21.15.16:0/3693628379 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4db415b490 msgr2=0x7f4db41f6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:44.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.110+0000 7f4db37fe700 1 --2- 172.21.15.16:0/3693628379 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4db415b490 0x7f4db41f6540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:44.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.110+0000 7f4db37fe700 1 -- 172.21.15.16:0/3693628379 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4db40cd7a0 msgr2=0x7f4db40b4140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:44.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.110+0000 7f4db37fe700 1 --2- 172.21.15.16:0/3693628379 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4db40cd7a0 0x7f4db40b4140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:44.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.110+0000 7f4db37fe700 1 -- 172.21.15.16:0/3693628379 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d9c009550 con 0x7f4db40ce310 2023-10-16T13:47:44.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.110+0000 7f4db37fe700 1 --2- 172.21.15.16:0/3693628379 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4db40ce310 0x7f4db40c9860 secure :-1 s=READY pgs=2807 cs=0 l=1 rev1=1 crypto rx=0x7f4d9c000c00 tx=0x7f4d9c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:44.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.110+0000 7f4db0ff9700 1 -- 172.21.15.16:0/3693628379 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d9c004250 con 0x7f4db40ce310 2023-10-16T13:47:44.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.111+0000 7f4db0ff9700 1 -- 172.21.15.16:0/3693628379 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d9c01e070 con 0x7f4db40ce310 2023-10-16T13:47:44.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.111+0000 7f4db95a0700 1 -- 172.21.15.16:0/3693628379 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4db40c7c10 con 0x7f4db40ce310 2023-10-16T13:47:44.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.111+0000 7f4db0ff9700 1 -- 172.21.15.16:0/3693628379 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d9c0045b0 con 0x7f4db40ce310 2023-10-16T13:47:44.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.111+0000 7f4db95a0700 1 -- 172.21.15.16:0/3693628379 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4db40b4680 con 0x7f4db40ce310 2023-10-16T13:47:44.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.111+0000 7f4db95a0700 1 -- 172.21.15.16:0/3693628379 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4db40c8130 con 0x7f4db40ce310 2023-10-16T13:47:44.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.112+0000 7f4db0ff9700 1 -- 172.21.15.16:0/3693628379 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 148) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f4d9c020030 con 0x7f4db40ce310 2023-10-16T13:47:44.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.112+0000 7f4db0ff9700 1 --2- 172.21.15.16:0/3693628379 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f4d9406b140 0x7f4d9406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:44.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.112+0000 7f4db0ff9700 1 -- 172.21.15.16:0/3693628379 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 7021+0+0 (secure 0 0 0) 0x7f4d9c08ad40 con 0x7f4db40ce310 2023-10-16T13:47:44.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.113+0000 7f4db3fff700 1 --2- 172.21.15.16:0/3693628379 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f4d9406b140 0x7f4d9406d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:44.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.113+0000 7f4db3fff700 1 --2- 172.21.15.16:0/3693628379 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f4d9406b140 0x7f4d9406d600 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f4da400c940 tx=0x7f4da4008040 comp rx=0 tx=0).ready entity=mgr.11886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:44.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.117+0000 7f4db0ff9700 1 -- 172.21.15.16:0/3693628379 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4db40c8130 con 0x7f4db40ce310 2023-10-16T13:47:44.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.367+0000 7f4db95a0700 1 -- 172.21.15.16:0/3693628379 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "backup_fs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f4db40c8130 con 0x7f4db40ce310 2023-10-16T13:47:44.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.952+0000 7f4db0ff9700 1 -- 172.21.15.16:0/3693628379 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "backup_fs", "var": "allow_standby_replay", "val": "false"}]=0 v135) v1 ==== 125+0+0 (secure 0 0 0) 0x7f4db40c8130 con 0x7f4db40ce310 2023-10-16T13:47:44.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.955+0000 7f4db95a0700 1 -- 172.21.15.16:0/3693628379 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f4d9406b140 msgr2=0x7f4d9406d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:44.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.955+0000 7f4db95a0700 1 --2- 172.21.15.16:0/3693628379 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f4d9406b140 0x7f4d9406d600 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f4da400c940 tx=0x7f4da4008040 comp rx=0 tx=0).stop 2023-10-16T13:47:44.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.955+0000 7f4db95a0700 1 -- 172.21.15.16:0/3693628379 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4db40ce310 msgr2=0x7f4db40c9860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:44.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.955+0000 7f4db95a0700 1 --2- 172.21.15.16:0/3693628379 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4db40ce310 0x7f4db40c9860 secure :-1 s=READY pgs=2807 cs=0 l=1 rev1=1 crypto rx=0x7f4d9c000c00 tx=0x7f4d9c004750 comp rx=0 tx=0).stop 2023-10-16T13:47:44.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.955+0000 7f4db95a0700 1 -- 172.21.15.16:0/3693628379 shutdown_connections 2023-10-16T13:47:44.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.955+0000 7f4db95a0700 1 --2- 172.21.15.16:0/3693628379 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f4d9406b140 0x7f4d9406d600 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:44.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.955+0000 7f4db95a0700 1 --2- 172.21.15.16:0/3693628379 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4db415b490 0x7f4db41f6540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:44.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.955+0000 7f4db95a0700 1 --2- 172.21.15.16:0/3693628379 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4db40cd7a0 0x7f4db40b4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:44.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.955+0000 7f4db95a0700 1 --2- 172.21.15.16:0/3693628379 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4db40ce310 0x7f4db40c9860 unknown :-1 s=CLOSED pgs=2807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:44.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.955+0000 7f4db95a0700 1 -- 172.21.15.16:0/3693628379 >> 172.21.15.16:0/3693628379 conn(0x7f4db40ccc80 msgr2=0x7f4db415a780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:44.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.955+0000 7f4db95a0700 1 -- 172.21.15.16:0/3693628379 shutdown_connections 2023-10-16T13:47:44.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:44.955+0000 7f4db95a0700 1 -- 172.21.15.16:0/3693628379 wait complete. 2023-10-16T13:47:44.973 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'backup_fs' 2023-10-16T13:47:44.974 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:47:45.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.102+0000 7fc3ec74a700 1 Processor -- start 2023-10-16T13:47:45.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.102+0000 7fc3ec74a700 1 -- start start 2023-10-16T13:47:45.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.103+0000 7fc3ec74a700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fc3e40ce870 0x7fc3e40cb980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:45.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.103+0000 7fc3ec74a700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3e40cedb0 con 0x7fc3e406aef0 2023-10-16T13:47:45.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.103+0000 7fc3ec74a700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3e40d12c0 con 0x7fc3e415db40 2023-10-16T13:47:45.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.103+0000 7fc3ec74a700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3e415df20 con 0x7fc3e40ce870 2023-10-16T13:47:45.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.103+0000 7fc3e9ce5700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc3e406aef0 0x7fc3e40ce160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:45126/0 (socket says 172.21.15.16:45126) 2023-10-16T13:47:45.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.103+0000 7fc3e9ce5700 1 -- 172.21.15.16:0/1995071238 learned_addr learned my addr 172.21.15.16:0/1995071238 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:47:45.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.104+0000 7fc3eace7700 1 --2- 172.21.15.16:0/1995071238 >> v2:172.21.15.49:3300/0 conn(0x7fc3e40ce870 0x7fc3e40cb980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:45.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.104+0000 7fc3e94e4700 1 -- 172.21.15.16:0/1995071238 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1599710102 0 0) 0x7fc3e40cedb0 con 0x7fc3e406aef0 2023-10-16T13:47:45.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.104+0000 7fc3e94e4700 1 -- 172.21.15.16:0/1995071238 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3d00035b0 con 0x7fc3e406aef0 2023-10-16T13:47:45.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.104+0000 7fc3e94e4700 1 -- 172.21.15.16:0/1995071238 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1043600281 0 0) 0x7fc3d00035b0 con 0x7fc3e406aef0 2023-10-16T13:47:45.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.104+0000 7fc3e94e4700 1 -- 172.21.15.16:0/1995071238 >> v2:172.21.15.49:3300/0 conn(0x7fc3e40ce870 msgr2=0x7fc3e40cb980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:45.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.104+0000 7fc3e94e4700 1 --2- 172.21.15.16:0/1995071238 >> v2:172.21.15.49:3300/0 conn(0x7fc3e40ce870 0x7fc3e40cb980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:45.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.104+0000 7fc3e94e4700 1 -- 172.21.15.16:0/1995071238 >> v1:172.21.15.49:6789/0 conn(0x7fc3e415db40 legacy=0x7fc3e406a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:45.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.104+0000 7fc3e94e4700 1 -- 172.21.15.16:0/1995071238 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3e40cabc0 con 0x7fc3e406aef0 2023-10-16T13:47:45.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.104+0000 7fc3e94e4700 1 -- 172.21.15.16:0/1995071238 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7fc3e0001670 con 0x7fc3e406aef0 2023-10-16T13:47:45.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.104+0000 7fc3e94e4700 1 -- 172.21.15.16:0/1995071238 >> v1:172.21.15.16:6789/0 conn(0x7fc3e406aef0 legacy=0x7fc3e40ce160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:45.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.104+0000 7fc3e94e4700 1 --2- 172.21.15.16:0/1995071238 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3d0003a00 0x7fc3d0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:45.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.104+0000 7fc3e94e4700 1 --2- 172.21.15.16:0/1995071238 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc3d0004850 0x7fc3d0004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:45.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.104+0000 7fc3e94e4700 1 --2- 172.21.15.16:0/1995071238 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc3d0005330 0x7fc3d000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:45.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.104+0000 7fc3e94e4700 1 -- 172.21.15.16:0/1995071238 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e415df20 con 0x7fc3d0003a00 2023-10-16T13:47:45.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.104+0000 7fc3e94e4700 1 -- 172.21.15.16:0/1995071238 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d000c200 con 0x7fc3d0005330 2023-10-16T13:47:45.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.104+0000 7fc3e94e4700 1 -- 172.21.15.16:0/1995071238 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d000c3c0 con 0x7fc3d0004850 2023-10-16T13:47:45.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.104+0000 7fc3ea4e6700 1 --2- 172.21.15.16:0/1995071238 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3d0003a00 0x7fc3d0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:45.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.104+0000 7fc3e9ce5700 1 --2- 172.21.15.16:0/1995071238 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc3d0004850 0x7fc3d0004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:45.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.104+0000 7fc3eace7700 1 --2- 172.21.15.16:0/1995071238 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc3d0005330 0x7fc3d000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:45.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.104+0000 7fc3ea4e6700 1 -- 172.21.15.16:0/1995071238 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc3d0004850 msgr2=0x7fc3d0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:45.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.104+0000 7fc3ea4e6700 1 --2- 172.21.15.16:0/1995071238 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc3d0004850 0x7fc3d0004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:45.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.104+0000 7fc3ea4e6700 1 -- 172.21.15.16:0/1995071238 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc3d0005330 msgr2=0x7fc3d000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:45.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.104+0000 7fc3ea4e6700 1 --2- 172.21.15.16:0/1995071238 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc3d0005330 0x7fc3d000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:45.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.104+0000 7fc3eace7700 1 -- 172.21.15.16:0/1995071238 reap_dead start 2023-10-16T13:47:45.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.104+0000 7fc3ea4e6700 1 -- 172.21.15.16:0/1995071238 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3d00035b0 con 0x7fc3d0003a00 2023-10-16T13:47:45.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.104+0000 7fc3e9ce5700 1 --2- 172.21.15.16:0/1995071238 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc3d0004850 0x7fc3d0004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:47:45.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.105+0000 7fc3ea4e6700 1 --2- 172.21.15.16:0/1995071238 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3d0003a00 0x7fc3d0003e20 secure :-1 s=READY pgs=2809 cs=0 l=1 rev1=1 crypto rx=0x7fc3d4007fd0 tx=0x7fc3d0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c82b1535e85be01 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:45.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.105+0000 7fc3e94e4700 1 -- 172.21.15.16:0/1995071238 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d40052d0 con 0x7fc3d0003a00 2023-10-16T13:47:45.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.105+0000 7fc3e94e4700 1 -- 172.21.15.16:0/1995071238 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3d4005450 con 0x7fc3d0003a00 2023-10-16T13:47:45.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.105+0000 7fc3e94e4700 1 -- 172.21.15.16:0/1995071238 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d40038c0 con 0x7fc3d0003a00 2023-10-16T13:47:45.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.105+0000 7fc3ec74a700 1 -- 172.21.15.16:0/1995071238 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3d0003a00 msgr2=0x7fc3d0003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:45.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.105+0000 7fc3ec74a700 1 --2- 172.21.15.16:0/1995071238 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3d0003a00 0x7fc3d0003e20 secure :-1 s=READY pgs=2809 cs=0 l=1 rev1=1 crypto rx=0x7fc3d4007fd0 tx=0x7fc3d0005880 comp rx=0 tx=0).stop 2023-10-16T13:47:45.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.105+0000 7fc3ec74a700 1 -- 172.21.15.16:0/1995071238 shutdown_connections 2023-10-16T13:47:45.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.105+0000 7fc3ec74a700 1 --2- 172.21.15.16:0/1995071238 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3d0003a00 0x7fc3d0003e20 unknown :-1 s=CLOSED pgs=2809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:45.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.105+0000 7fc3ec74a700 1 -- 172.21.15.16:0/1995071238 >> 172.21.15.16:0/1995071238 conn(0x7fc3e40b1fd0 msgr2=0x7fc3e40b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:45.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.105+0000 7fc3ec74a700 1 -- 172.21.15.16:0/1995071238 shutdown_connections 2023-10-16T13:47:45.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.106+0000 7fc3ec74a700 1 -- 172.21.15.16:0/1995071238 wait complete. 2023-10-16T13:47:45.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.107+0000 7fc3ec74a700 1 Processor -- start 2023-10-16T13:47:45.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.107+0000 7fc3ec74a700 1 -- start start 2023-10-16T13:47:45.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.107+0000 7fc3ec74a700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc3d0003a00 0x7fc3e40c7b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:45.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.107+0000 7fc3ec74a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc3e41506f0 0x7fc3e41f6dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:45.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.107+0000 7fc3ec74a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3e415b390 0x7fc3e415b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:45.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.107+0000 7fc3ec74a700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e415bd50 con 0x7fc3e415b390 2023-10-16T13:47:45.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.107+0000 7fc3ec74a700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e4151fa0 con 0x7fc3d0003a00 2023-10-16T13:47:45.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.107+0000 7fc3ec74a700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e41f7300 con 0x7fc3e41506f0 2023-10-16T13:47:45.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.108+0000 7fc3eace7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3e415b390 0x7fc3e415b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:45.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.108+0000 7fc3eace7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3e415b390 0x7fc3e415b810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55866/0 (socket says 172.21.15.16:55866) 2023-10-16T13:47:45.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.108+0000 7fc3eace7700 1 -- 172.21.15.16:0/3941552684 learned_addr learned my addr 172.21.15.16:0/3941552684 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:45.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.108+0000 7fc3ea4e6700 1 --2- 172.21.15.16:0/3941552684 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc3d0003a00 0x7fc3e40c7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:45.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.108+0000 7fc3e9ce5700 1 --2- 172.21.15.16:0/3941552684 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc3e41506f0 0x7fc3e41f6dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:45.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.108+0000 7fc3eace7700 1 -- 172.21.15.16:0/3941552684 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc3e41506f0 msgr2=0x7fc3e41f6dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:45.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.108+0000 7fc3eace7700 1 --2- 172.21.15.16:0/3941552684 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc3e41506f0 0x7fc3e41f6dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:45.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.108+0000 7fc3eace7700 1 -- 172.21.15.16:0/3941552684 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc3d0003a00 msgr2=0x7fc3e40c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:45.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.108+0000 7fc3eace7700 1 --2- 172.21.15.16:0/3941552684 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc3d0003a00 0x7fc3e40c7b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:45.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.108+0000 7fc3eace7700 1 -- 172.21.15.16:0/3941552684 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3e4150bf0 con 0x7fc3e415b390 2023-10-16T13:47:45.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.108+0000 7fc3ea4e6700 1 --2- 172.21.15.16:0/3941552684 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc3d0003a00 0x7fc3e40c7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:47:45.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.108+0000 7fc3eace7700 1 --2- 172.21.15.16:0/3941552684 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3e415b390 0x7fc3e415b810 secure :-1 s=READY pgs=2810 cs=0 l=1 rev1=1 crypto rx=0x7fc3e40cece0 tx=0x7fc3dc011ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:45.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.108+0000 7fc3daffd700 1 -- 172.21.15.16:0/3941552684 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3dc00e040 con 0x7fc3e415b390 2023-10-16T13:47:45.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.108+0000 7fc3daffd700 1 -- 172.21.15.16:0/3941552684 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3dc00f040 con 0x7fc3e415b390 2023-10-16T13:47:45.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.108+0000 7fc3daffd700 1 -- 172.21.15.16:0/3941552684 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3dc00fd30 con 0x7fc3e415b390 2023-10-16T13:47:45.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.108+0000 7fc3ec74a700 1 -- 172.21.15.16:0/3941552684 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3e4150ed0 con 0x7fc3e415b390 2023-10-16T13:47:45.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.108+0000 7fc3ec74a700 1 -- 172.21.15.16:0/3941552684 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3e4151490 con 0x7fc3e415b390 2023-10-16T13:47:45.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.109+0000 7fc3ec74a700 1 -- 172.21.15.16:0/3941552684 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3e404ebb0 con 0x7fc3e415b390 2023-10-16T13:47:45.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.110+0000 7fc3daffd700 1 -- 172.21.15.16:0/3941552684 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 148) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fc3dc015430 con 0x7fc3e415b390 2023-10-16T13:47:45.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.110+0000 7fc3daffd700 1 --2- 172.21.15.16:0/3941552684 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fc3c806b140 0x7fc3c806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:45.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.110+0000 7fc3daffd700 1 -- 172.21.15.16:0/3941552684 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 7021+0+0 (secure 0 0 0) 0x7fc3dc08a590 con 0x7fc3e415b390 2023-10-16T13:47:45.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.110+0000 7fc3ea4e6700 1 --2- 172.21.15.16:0/3941552684 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fc3c806b140 0x7fc3c806d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:45.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.111+0000 7fc3ea4e6700 1 --2- 172.21.15.16:0/3941552684 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fc3c806b140 0x7fc3c806d600 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc3d4001eb0 tx=0x7fc3d4009030 comp rx=0 tx=0).ready entity=mgr.11886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:45.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.115+0000 7fc3daffd700 1 -- 172.21.15.16:0/3941552684 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc3dc054cb0 con 0x7fc3e415b390 2023-10-16T13:47:45.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.366+0000 7fc3ec74a700 1 -- 172.21.15.16:0/3941552684 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc3e4072bd0 con 0x7fc3e415b390 2023-10-16T13:47:45.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.366+0000 7fc3daffd700 1 -- 172.21.15.16:0/3941552684 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) v1 ==== 95+0+6147 (secure 0 0 0) 0x7fc3dc0582d0 con 0x7fc3e415b390 2023-10-16T13:47:45.370 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:47:45.371 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":135,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9596,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/3939647668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":3939647668},{"type":"v1","addr":"172.21.15.49:6834","nonce":3939647668}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":125},{"gid":11185,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/2890113513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2890113513},{"type":"v1","addr":"172.21.15.16:6836","nonce":2890113513}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":125}],"filesystems":[{"mdsmap":{"epoch":130,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:47:23.843333+0000","modified":"2023-10-16T13:47:26.851597+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":12252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12252":{"gid":12252,"name":"a","rank":0,"incarnation":126,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/303278795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":303278795},{"type":"v1","addr":"172.21.15.16:6837","nonce":303278795}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:47:41.332096+0000","modified":"2023-10-16T13:47:44.368830+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11197},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11197":{"gid":11197,"name":"d","rank":0,"incarnation":132,"state":"up:active","state_seq":7,"addr":"172.21.15.49:6835/680748068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":680748068},{"type":"v1","addr":"172.21.15.49:6835","nonce":680748068}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2023-10-16T13:47:45.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.368+0000 7fc3ec74a700 1 -- 172.21.15.16:0/3941552684 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fc3c806b140 msgr2=0x7fc3c806d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:45.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.368+0000 7fc3ec74a700 1 --2- 172.21.15.16:0/3941552684 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fc3c806b140 0x7fc3c806d600 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc3d4001eb0 tx=0x7fc3d4009030 comp rx=0 tx=0).stop 2023-10-16T13:47:45.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.368+0000 7fc3ec74a700 1 -- 172.21.15.16:0/3941552684 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3e415b390 msgr2=0x7fc3e415b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:45.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.368+0000 7fc3ec74a700 1 --2- 172.21.15.16:0/3941552684 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3e415b390 0x7fc3e415b810 secure :-1 s=READY pgs=2810 cs=0 l=1 rev1=1 crypto rx=0x7fc3e40cece0 tx=0x7fc3dc011ed0 comp rx=0 tx=0).stop 2023-10-16T13:47:45.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.369+0000 7fc3ec74a700 1 -- 172.21.15.16:0/3941552684 shutdown_connections 2023-10-16T13:47:45.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.369+0000 7fc3ec74a700 1 --2- 172.21.15.16:0/3941552684 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fc3c806b140 0x7fc3c806d600 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:45.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.369+0000 7fc3ec74a700 1 --2- 172.21.15.16:0/3941552684 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3e415b390 0x7fc3e415b810 unknown :-1 s=CLOSED pgs=2810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:45.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.369+0000 7fc3ec74a700 1 --2- 172.21.15.16:0/3941552684 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc3d0003a00 0x7fc3e40c7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:45.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.369+0000 7fc3ec74a700 1 --2- 172.21.15.16:0/3941552684 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc3e41506f0 0x7fc3e41f6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:45.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.369+0000 7fc3ec74a700 1 -- 172.21.15.16:0/3941552684 >> 172.21.15.16:0/3941552684 conn(0x7fc3e40b1fd0 msgr2=0x7fc3e4072330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:45.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.369+0000 7fc3ec74a700 1 -- 172.21.15.16:0/3941552684 shutdown_connections 2023-10-16T13:47:45.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.369+0000 7fc3ec74a700 1 -- 172.21.15.16:0/3941552684 wait complete. 2023-10-16T13:47:45.375 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 135 2023-10-16T13:47:45.384 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:47:45.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.506+0000 7f396521a700 1 Processor -- start 2023-10-16T13:47:45.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.507+0000 7f396521a700 1 -- start start 2023-10-16T13:47:45.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.507+0000 7f396521a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f39600ce870 0x7f39600cb980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:45.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.507+0000 7f396521a700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39600cedb0 con 0x7f396015db40 2023-10-16T13:47:45.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.507+0000 7f396521a700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39600d12c0 con 0x7f396006aef0 2023-10-16T13:47:45.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.507+0000 7f396521a700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f396015df20 con 0x7f39600ce870 2023-10-16T13:47:45.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.507+0000 7f395ed9d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f396015db40 0x7f396006a7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:45144/0 (socket says 172.21.15.16:45144) 2023-10-16T13:47:45.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.507+0000 7f395ed9d700 1 -- 172.21.15.16:0/1675637042 learned_addr learned my addr 172.21.15.16:0/1675637042 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:47:45.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.508+0000 7f395f59e700 1 --2- 172.21.15.16:0/1675637042 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f39600ce870 0x7f39600cb980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:45.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.508+0000 7f395dd9b700 1 -- 172.21.15.16:0/1675637042 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3717706338 0 0) 0x7f39600cedb0 con 0x7f396015db40 2023-10-16T13:47:45.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.508+0000 7f395dd9b700 1 -- 172.21.15.16:0/1675637042 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f394c0035b0 con 0x7f396015db40 2023-10-16T13:47:45.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.508+0000 7f395dd9b700 1 -- 172.21.15.16:0/1675637042 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3245368386 0 0) 0x7f39600d12c0 con 0x7f396006aef0 2023-10-16T13:47:45.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.508+0000 7f395dd9b700 1 -- 172.21.15.16:0/1675637042 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39600cedb0 con 0x7f396006aef0 2023-10-16T13:47:45.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.508+0000 7f395f59e700 1 -- 172.21.15.16:0/1675637042 >> v1:172.21.15.49:6789/0 conn(0x7f396006aef0 legacy=0x7f39600ce160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:45.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.508+0000 7f395f59e700 1 -- 172.21.15.16:0/1675637042 >> v1:172.21.15.16:6789/0 conn(0x7f396015db40 legacy=0x7f396006a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:45.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.508+0000 7f395f59e700 1 -- 172.21.15.16:0/1675637042 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39600cabc0 con 0x7f39600ce870 2023-10-16T13:47:45.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.508+0000 7f395f59e700 1 --2- 172.21.15.16:0/1675637042 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f39600ce870 0x7f39600cb980 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f3950009870 tx=0x7f3950009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a1803def4ece9f0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:45.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.508+0000 7f395dd9b700 1 -- 172.21.15.16:0/1675637042 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3950011830 con 0x7f39600ce870 2023-10-16T13:47:45.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.508+0000 7f395dd9b700 1 -- 172.21.15.16:0/1675637042 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3950011990 con 0x7f39600ce870 2023-10-16T13:47:45.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.508+0000 7f395dd9b700 1 -- 172.21.15.16:0/1675637042 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3950011d20 con 0x7f39600ce870 2023-10-16T13:47:45.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.509+0000 7f396521a700 1 -- 172.21.15.16:0/1675637042 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f39600ce870 msgr2=0x7f39600cb980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:45.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.509+0000 7f396521a700 1 --2- 172.21.15.16:0/1675637042 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f39600ce870 0x7f39600cb980 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f3950009870 tx=0x7f3950009b80 comp rx=0 tx=0).stop 2023-10-16T13:47:45.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.509+0000 7f396521a700 1 -- 172.21.15.16:0/1675637042 shutdown_connections 2023-10-16T13:47:45.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.509+0000 7f396521a700 1 --2- 172.21.15.16:0/1675637042 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f39600ce870 0x7f39600cb980 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:45.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.509+0000 7f396521a700 1 -- 172.21.15.16:0/1675637042 >> 172.21.15.16:0/1675637042 conn(0x7f39600b1fd0 msgr2=0x7f39600b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:45.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.509+0000 7f396521a700 1 -- 172.21.15.16:0/1675637042 shutdown_connections 2023-10-16T13:47:45.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.509+0000 7f396521a700 1 -- 172.21.15.16:0/1675637042 wait complete. 2023-10-16T13:47:45.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.510+0000 7f396521a700 1 Processor -- start 2023-10-16T13:47:45.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.510+0000 7f396521a700 1 -- start start 2023-10-16T13:47:45.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.511+0000 7f396521a700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f396006aef0 0x7f3960151f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:45.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.511+0000 7f396521a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f39600ce870 0x7f39600c5b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:45.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.511+0000 7f396521a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f396015db40 0x7f39600c6080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:45.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.511+0000 7f396521a700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39600ca3f0 con 0x7f396015db40 2023-10-16T13:47:45.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.511+0000 7f396521a700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39600c4a40 con 0x7f396006aef0 2023-10-16T13:47:45.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.511+0000 7f396521a700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39601524a0 con 0x7f39600ce870 2023-10-16T13:47:45.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.511+0000 7f395f59e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f396015db40 0x7f39600c6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:45.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.511+0000 7f395f59e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f396015db40 0x7f39600c6080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55880/0 (socket says 172.21.15.16:55880) 2023-10-16T13:47:45.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.511+0000 7f395f59e700 1 -- 172.21.15.16:0/3678865346 learned_addr learned my addr 172.21.15.16:0/3678865346 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:45.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.511+0000 7f395e59c700 1 --2- 172.21.15.16:0/3678865346 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f39600ce870 0x7f39600c5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:45.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.512+0000 7f395ed9d700 1 --2- 172.21.15.16:0/3678865346 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f396006aef0 0x7f3960151f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:45.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.512+0000 7f395f59e700 1 -- 172.21.15.16:0/3678865346 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f39600ce870 msgr2=0x7f39600c5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:45.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.512+0000 7f395f59e700 1 --2- 172.21.15.16:0/3678865346 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f39600ce870 0x7f39600c5b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:45.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.512+0000 7f395f59e700 1 -- 172.21.15.16:0/3678865346 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f396006aef0 msgr2=0x7f3960151f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:45.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.512+0000 7f395f59e700 1 --2- 172.21.15.16:0/3678865346 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f396006aef0 0x7f3960151f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:45.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.512+0000 7f395f59e700 1 -- 172.21.15.16:0/3678865346 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39500094a0 con 0x7f396015db40 2023-10-16T13:47:45.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.512+0000 7f395ed9d700 1 --2- 172.21.15.16:0/3678865346 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f396006aef0 0x7f3960151f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:47:45.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.512+0000 7f395f59e700 1 --2- 172.21.15.16:0/3678865346 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f396015db40 0x7f39600c6080 secure :-1 s=READY pgs=2812 cs=0 l=1 rev1=1 crypto rx=0x7f39500090b0 tx=0x7f3950004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:45.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.512+0000 7f39577fe700 1 -- 172.21.15.16:0/3678865346 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f395001a6e0 con 0x7f396015db40 2023-10-16T13:47:45.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.512+0000 7f39577fe700 1 -- 172.21.15.16:0/3678865346 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f395001a840 con 0x7f396015db40 2023-10-16T13:47:45.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.512+0000 7f396521a700 1 -- 172.21.15.16:0/3678865346 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39600c4110 con 0x7f396015db40 2023-10-16T13:47:45.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.512+0000 7f39577fe700 1 -- 172.21.15.16:0/3678865346 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f395001abf0 con 0x7f396015db40 2023-10-16T13:47:45.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.512+0000 7f396521a700 1 -- 172.21.15.16:0/3678865346 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39600c46d0 con 0x7f396015db40 2023-10-16T13:47:45.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.513+0000 7f396521a700 1 -- 172.21.15.16:0/3678865346 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39600c4bd0 con 0x7f396015db40 2023-10-16T13:47:45.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.514+0000 7f39577fe700 1 -- 172.21.15.16:0/3678865346 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 148) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f3950022070 con 0x7f396015db40 2023-10-16T13:47:45.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.514+0000 7f39577fe700 1 --2- 172.21.15.16:0/3678865346 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f394006b140 0x7f394006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:45.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.514+0000 7f39577fe700 1 -- 172.21.15.16:0/3678865346 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 7021+0+0 (secure 0 0 0) 0x7f395008ad80 con 0x7f396015db40 2023-10-16T13:47:45.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.514+0000 7f395ed9d700 1 --2- 172.21.15.16:0/3678865346 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f394006b140 0x7f394006d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:45.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.514+0000 7f395ed9d700 1 --2- 172.21.15.16:0/3678865346 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f394006b140 0x7f394006d600 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f39580012e0 tx=0x7f3958009040 comp rx=0 tx=0).ready entity=mgr.11886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:45.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.518+0000 7f39577fe700 1 -- 172.21.15.16:0/3678865346 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f39500555c0 con 0x7f396015db40 2023-10-16T13:47:45.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.722+0000 7f396521a700 1 -- 172.21.15.16:0/3678865346 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f39600c5920 con 0x7f396015db40 2023-10-16T13:47:45.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.724+0000 7f39577fe700 1 -- 172.21.15.16:0/3678865346 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v249) v1 ==== 74+0+20541 (secure 0 0 0) 0x7f3950058be0 con 0x7f396015db40 2023-10-16T13:47:45.728 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:47:45.728 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":249,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:47:42.921047+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2023-10-16T13:47:22.338349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"244","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2023-10-16T13:47:22.806327+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"245","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T13:47:39.075528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"248","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"backup_fs_data","create_time":"2023-10-16T13:47:40.314029+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"249","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:47:45.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.726+0000 7f396521a700 1 -- 172.21.15.16:0/3678865346 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f394006b140 msgr2=0x7f394006d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:45.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.726+0000 7f396521a700 1 --2- 172.21.15.16:0/3678865346 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f394006b140 0x7f394006d600 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f39580012e0 tx=0x7f3958009040 comp rx=0 tx=0).stop 2023-10-16T13:47:45.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.726+0000 7f396521a700 1 -- 172.21.15.16:0/3678865346 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f396015db40 msgr2=0x7f39600c6080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:45.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.726+0000 7f396521a700 1 --2- 172.21.15.16:0/3678865346 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f396015db40 0x7f39600c6080 secure :-1 s=READY pgs=2812 cs=0 l=1 rev1=1 crypto rx=0x7f39500090b0 tx=0x7f3950004750 comp rx=0 tx=0).stop 2023-10-16T13:47:45.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.726+0000 7f396521a700 1 -- 172.21.15.16:0/3678865346 shutdown_connections 2023-10-16T13:47:45.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.726+0000 7f396521a700 1 --2- 172.21.15.16:0/3678865346 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f394006b140 0x7f394006d600 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:45.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.726+0000 7f396521a700 1 --2- 172.21.15.16:0/3678865346 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f396015db40 0x7f39600c6080 unknown :-1 s=CLOSED pgs=2812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:45.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.726+0000 7f396521a700 1 --2- 172.21.15.16:0/3678865346 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f396006aef0 0x7f3960151f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:45.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.726+0000 7f396521a700 1 --2- 172.21.15.16:0/3678865346 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f39600ce870 0x7f39600c5b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:45.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.726+0000 7f396521a700 1 -- 172.21.15.16:0/3678865346 >> 172.21.15.16:0/3678865346 conn(0x7f39600b1fd0 msgr2=0x7f39600ce130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:45.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.726+0000 7f396521a700 1 -- 172.21.15.16:0/3678865346 shutdown_connections 2023-10-16T13:47:45.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.726+0000 7f396521a700 1 -- 172.21.15.16:0/3678865346 wait complete. 2023-10-16T13:47:45.742 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-16T13:47:45.743 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T13:47:45.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.865+0000 7f411a6d4700 1 Processor -- start 2023-10-16T13:47:45.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.865+0000 7f411a6d4700 1 -- start start 2023-10-16T13:47:45.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.865+0000 7f411a6d4700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f411415b280 0x7f41141613e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:45.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.865+0000 7f411a6d4700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f41140cc8a0 0x7f411415e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:45.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.865+0000 7f411a6d4700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f411406a7e0 0x7f411406ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:45.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.865+0000 7f411a6d4700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f411406b190 con 0x7f411415b280 2023-10-16T13:47:45.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.865+0000 7f411a6d4700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4114161c30 con 0x7f41140cc8a0 2023-10-16T13:47:45.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.865+0000 7f411a6d4700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4114161f10 con 0x7f411406a7e0 2023-10-16T13:47:45.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.866+0000 7f4113fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f411415b280 0x7f41141613e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:45.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.866+0000 7f4113fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f411415b280 0x7f41141613e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55884/0 (socket says 172.21.15.16:55884) 2023-10-16T13:47:45.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.866+0000 7f4113fff700 1 -- 172.21.15.16:0/1813141420 learned_addr learned my addr 172.21.15.16:0/1813141420 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:45.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.866+0000 7f4113fff700 1 -- 172.21.15.16:0/1813141420 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f411406a7e0 msgr2=0x7f411406ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:45.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.866+0000 7f4118c71700 1 --2- 172.21.15.16:0/1813141420 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f411406a7e0 0x7f411406ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:45.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.866+0000 7f41137fe700 1 --2- 172.21.15.16:0/1813141420 >> v2:172.21.15.49:3300/0 conn(0x7f41140cc8a0 0x7f411415e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:45.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.866+0000 7f4113fff700 1 --2- 172.21.15.16:0/1813141420 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f411406a7e0 0x7f411406ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:45.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.866+0000 7f4113fff700 1 -- 172.21.15.16:0/1813141420 >> v2:172.21.15.49:3300/0 conn(0x7f41140cc8a0 msgr2=0x7f411415e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:45.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.866+0000 7f4113fff700 1 --2- 172.21.15.16:0/1813141420 >> v2:172.21.15.49:3300/0 conn(0x7f41140cc8a0 0x7f411415e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:45.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.866+0000 7f4113fff700 1 -- 172.21.15.16:0/1813141420 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41141621f0 con 0x7f411415b280 2023-10-16T13:47:45.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.866+0000 7f41137fe700 1 --2- 172.21.15.16:0/1813141420 >> v2:172.21.15.49:3300/0 conn(0x7f41140cc8a0 0x7f411415e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:47:45.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.866+0000 7f4118c71700 1 --2- 172.21.15.16:0/1813141420 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f411406a7e0 0x7f411406ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:47:45.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.866+0000 7f4113fff700 1 --2- 172.21.15.16:0/1813141420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f411415b280 0x7f41141613e0 secure :-1 s=READY pgs=2813 cs=0 l=1 rev1=1 crypto rx=0x7f4114071970 tx=0x7f4104009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bac0add194a73fcf server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:45.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.866+0000 7f4112ffd700 1 -- 172.21.15.16:0/1813141420 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f410400b870 con 0x7f411415b280 2023-10-16T13:47:45.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.866+0000 7f4112ffd700 1 -- 172.21.15.16:0/1813141420 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f410400b9d0 con 0x7f411415b280 2023-10-16T13:47:45.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.866+0000 7f4112ffd700 1 -- 172.21.15.16:0/1813141420 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f410400bd60 con 0x7f411415b280 2023-10-16T13:47:45.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.867+0000 7f411a6d4700 1 -- 172.21.15.16:0/1813141420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f411415b280 msgr2=0x7f41141613e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:45.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.867+0000 7f411a6d4700 1 --2- 172.21.15.16:0/1813141420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f411415b280 0x7f41141613e0 secure :-1 s=READY pgs=2813 cs=0 l=1 rev1=1 crypto rx=0x7f4114071970 tx=0x7f4104009bd0 comp rx=0 tx=0).stop 2023-10-16T13:47:45.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.867+0000 7f411a6d4700 1 -- 172.21.15.16:0/1813141420 shutdown_connections 2023-10-16T13:47:45.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.867+0000 7f411a6d4700 1 --2- 172.21.15.16:0/1813141420 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f411406a7e0 0x7f411406ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:45.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.867+0000 7f411a6d4700 1 --2- 172.21.15.16:0/1813141420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f411415b280 0x7f41141613e0 unknown :-1 s=CLOSED pgs=2813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:45.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.867+0000 7f411a6d4700 1 --2- 172.21.15.16:0/1813141420 >> v2:172.21.15.49:3300/0 conn(0x7f41140cc8a0 0x7f411415e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:45.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.867+0000 7f411a6d4700 1 -- 172.21.15.16:0/1813141420 >> 172.21.15.16:0/1813141420 conn(0x7f41140b1fd0 msgr2=0x7f41140b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:45.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.867+0000 7f411a6d4700 1 -- 172.21.15.16:0/1813141420 shutdown_connections 2023-10-16T13:47:45.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.867+0000 7f411a6d4700 1 -- 172.21.15.16:0/1813141420 wait complete. 2023-10-16T13:47:45.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.868+0000 7f411a6d4700 1 Processor -- start 2023-10-16T13:47:45.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.868+0000 7f411a6d4700 1 -- start start 2023-10-16T13:47:45.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.869+0000 7f411a6d4700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f411406a7e0 0x7f4114156a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:45.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.869+0000 7f411a6d4700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f41140cc8a0 0x7f41141fa450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:45.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.869+0000 7f411a6d4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f411415b280 0x7f41141fa990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:45.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.869+0000 7f411a6d4700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41140c3a70 con 0x7f411415b280 2023-10-16T13:47:45.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.869+0000 7f411a6d4700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41140ca170 con 0x7f41140cc8a0 2023-10-16T13:47:45.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.869+0000 7f411a6d4700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41140ca450 con 0x7f411406a7e0 2023-10-16T13:47:45.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.869+0000 7f4118c71700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f411415b280 0x7f41141fa990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:45.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.869+0000 7f4118c71700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f411415b280 0x7f41141fa990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55894/0 (socket says 172.21.15.16:55894) 2023-10-16T13:47:45.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.869+0000 7f4118c71700 1 -- 172.21.15.16:0/3387535414 learned_addr learned my addr 172.21.15.16:0/3387535414 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:45.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.869+0000 7f4113fff700 1 --2- 172.21.15.16:0/3387535414 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f411406a7e0 0x7f4114156a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:45.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.869+0000 7f41137fe700 1 --2- 172.21.15.16:0/3387535414 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f41140cc8a0 0x7f41141fa450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:45.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.870+0000 7f4118c71700 1 -- 172.21.15.16:0/3387535414 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f411406a7e0 msgr2=0x7f4114156a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:45.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.870+0000 7f4118c71700 1 --2- 172.21.15.16:0/3387535414 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f411406a7e0 0x7f4114156a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:45.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.870+0000 7f4118c71700 1 -- 172.21.15.16:0/3387535414 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f41140cc8a0 msgr2=0x7f41141fa450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:45.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.870+0000 7f4118c71700 1 --2- 172.21.15.16:0/3387535414 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f41140cc8a0 0x7f41141fa450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:45.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.870+0000 7f4118c71700 1 -- 172.21.15.16:0/3387535414 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4104009550 con 0x7f411415b280 2023-10-16T13:47:45.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.870+0000 7f41137fe700 1 --2- 172.21.15.16:0/3387535414 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f41140cc8a0 0x7f41141fa450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:47:45.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.870+0000 7f4113fff700 1 --2- 172.21.15.16:0/3387535414 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f411406a7e0 0x7f4114156a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:47:45.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.870+0000 7f4118c71700 1 --2- 172.21.15.16:0/3387535414 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f411415b280 0x7f41141fa990 secure :-1 s=READY pgs=2814 cs=0 l=1 rev1=1 crypto rx=0x7f410c00cef0 tx=0x7f410c00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:45.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.870+0000 7f4110ff9700 1 -- 172.21.15.16:0/3387535414 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f410c00add0 con 0x7f411415b280 2023-10-16T13:47:45.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.870+0000 7f4110ff9700 1 -- 172.21.15.16:0/3387535414 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f410c015440 con 0x7f411415b280 2023-10-16T13:47:45.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.870+0000 7f4110ff9700 1 -- 172.21.15.16:0/3387535414 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f410c0157f0 con 0x7f411415b280 2023-10-16T13:47:45.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.870+0000 7f411a6d4700 1 -- 172.21.15.16:0/3387535414 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f411415e7a0 con 0x7f411415b280 2023-10-16T13:47:45.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.870+0000 7f411a6d4700 1 -- 172.21.15.16:0/3387535414 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41140c33a0 con 0x7f411415b280 2023-10-16T13:47:45.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.871+0000 7f411a6d4700 1 -- 172.21.15.16:0/3387535414 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f411415e600 con 0x7f411415b280 2023-10-16T13:47:45.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.871+0000 7f4110ff9700 1 -- 172.21.15.16:0/3387535414 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 148) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f410c00d430 con 0x7f411415b280 2023-10-16T13:47:45.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.872+0000 7f4110ff9700 1 --2- 172.21.15.16:0/3387535414 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f410006b140 0x7f410006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:45.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.872+0000 7f4110ff9700 1 -- 172.21.15.16:0/3387535414 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 7021+0+0 (secure 0 0 0) 0x7f410c08a4c0 con 0x7f411415b280 2023-10-16T13:47:45.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.872+0000 7f4113fff700 1 --2- 172.21.15.16:0/3387535414 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f410006b140 0x7f410006d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:45.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.872+0000 7f4113fff700 1 --2- 172.21.15.16:0/3387535414 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f410006b140 0x7f410006d600 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4104014040 tx=0x7f4104008030 comp rx=0 tx=0).ready entity=mgr.11886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:45.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:45.877+0000 7f4110ff9700 1 -- 172.21.15.16:0/3387535414 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f411415e600 con 0x7f411415b280 2023-10-16T13:47:46.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.076+0000 7f411a6d4700 1 -- 172.21.15.16:0/3387535414 --> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f411415e1e0 con 0x7f410006b140 2023-10-16T13:47:46.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.090+0000 7f4110ff9700 1 -- 172.21.15.16:0/3387535414 <== mgr.11886 v2:172.21.15.16:6832/1736104576 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+529161 (secure 0 0 0) 0x7f410400bda0 con 0x7f410006b140 2023-10-16T13:47:46.099 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:47:46.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.094+0000 7f411a6d4700 1 -- 172.21.15.16:0/3387535414 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f410006b140 msgr2=0x7f410006d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:46.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.094+0000 7f411a6d4700 1 --2- 172.21.15.16:0/3387535414 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f410006b140 0x7f410006d600 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4104014040 tx=0x7f4104008030 comp rx=0 tx=0).stop 2023-10-16T13:47:46.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.094+0000 7f411a6d4700 1 -- 172.21.15.16:0/3387535414 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f411415b280 msgr2=0x7f41141fa990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:46.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.094+0000 7f411a6d4700 1 --2- 172.21.15.16:0/3387535414 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f411415b280 0x7f41141fa990 secure :-1 s=READY pgs=2814 cs=0 l=1 rev1=1 crypto rx=0x7f410c00cef0 tx=0x7f410c00a340 comp rx=0 tx=0).stop 2023-10-16T13:47:46.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.095+0000 7f411a6d4700 1 -- 172.21.15.16:0/3387535414 shutdown_connections 2023-10-16T13:47:46.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.095+0000 7f411a6d4700 1 --2- 172.21.15.16:0/3387535414 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f410006b140 0x7f410006d600 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:46.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.095+0000 7f411a6d4700 1 --2- 172.21.15.16:0/3387535414 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f411415b280 0x7f41141fa990 unknown :-1 s=CLOSED pgs=2814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:46.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.095+0000 7f411a6d4700 1 --2- 172.21.15.16:0/3387535414 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f411406a7e0 0x7f4114156a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:46.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.095+0000 7f411a6d4700 1 --2- 172.21.15.16:0/3387535414 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f41140cc8a0 0x7f41141fa450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:46.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.095+0000 7f411a6d4700 1 -- 172.21.15.16:0/3387535414 >> 172.21.15.16:0/3387535414 conn(0x7f41140b1fd0 msgr2=0x7f41140caa70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:46.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.095+0000 7f411a6d4700 1 -- 172.21.15.16:0/3387535414 shutdown_connections 2023-10-16T13:47:46.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.095+0000 7f411a6d4700 1 -- 172.21.15.16:0/3387535414 wait complete. 2023-10-16T13:47:46.106 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-10-16T13:47:46.111 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":43,"stamp":"2023-10-16T13:47:45.825850+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596435,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":711,"num_write_kb":2447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":450,"ondisk_log_size":450,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":302,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":450068,"kb_used_data":25688,"kb_used_omap":12,"kb_used_meta":424243,"kb_avail":754524652,"statfs":{"total":773094113280,"available":772633243648,"internally_reserved":0,"allocated":26304512,"data_stored":14138623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12740,"internal_metadata":434425404},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3397,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":46,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.928850"},"pg_stats":[{"pgid":"34.1d","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.953163+0000","last_change":"2023-10-16T13:47:41.943421+0000","last_active":"2023-10-16T13:47:41.953163+0000","last_peered":"2023-10-16T13:47:41.953163+0000","last_clean":"2023-10-16T13:47:41.953163+0000","last_became_active":"2023-10-16T13:47:41.942988+0000","last_became_peered":"2023-10-16T13:47:41.942988+0000","last_unstale":"2023-10-16T13:47:41.953163+0000","last_undegraded":"2023-10-16T13:47:41.953163+0000","last_fullsized":"2023-10-16T13:47:41.953163+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:45:39.186801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1e","version":"248'1","reported_seq":12,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.948433+0000","last_change":"2023-10-16T13:47:40.912655+0000","last_active":"2023-10-16T13:47:41.948433+0000","last_peered":"2023-10-16T13:47:41.948433+0000","last_clean":"2023-10-16T13:47:41.948433+0000","last_became_active":"2023-10-16T13:47:40.912369+0000","last_became_peered":"2023-10-16T13:47:40.912369+0000","last_unstale":"2023-10-16T13:47:41.948433+0000","last_undegraded":"2023-10-16T13:47:41.948433+0000","last_fullsized":"2023-10-16T13:47:41.948433+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_clean_scrub_stamp":"2023-10-16T13:47:39.890588+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:50:08.706472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1f","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.908471+0000","last_change":"2023-10-16T13:47:24.429332+0000","last_active":"2023-10-16T13:47:40.908471+0000","last_peered":"2023-10-16T13:47:40.908471+0000","last_clean":"2023-10-16T13:47:40.908471+0000","last_became_active":"2023-10-16T13:47:24.428151+0000","last_became_peered":"2023-10-16T13:47:24.428151+0000","last_unstale":"2023-10-16T13:47:40.908471+0000","last_undegraded":"2023-10-16T13:47:40.908471+0000","last_fullsized":"2023-10-16T13:47:40.908471+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:51:49.567279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1c","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.952421+0000","last_change":"2023-10-16T13:47:41.940099+0000","last_active":"2023-10-16T13:47:41.952421+0000","last_peered":"2023-10-16T13:47:41.952421+0000","last_clean":"2023-10-16T13:47:41.952421+0000","last_became_active":"2023-10-16T13:47:41.939640+0000","last_became_peered":"2023-10-16T13:47:41.939640+0000","last_unstale":"2023-10-16T13:47:41.952421+0000","last_undegraded":"2023-10-16T13:47:41.952421+0000","last_fullsized":"2023-10-16T13:47:41.952421+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:22:07.291807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.1f","version":"248'4","reported_seq":15,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.953154+0000","last_change":"2023-10-16T13:47:40.924092+0000","last_active":"2023-10-16T13:47:41.953154+0000","last_peered":"2023-10-16T13:47:41.953154+0000","last_clean":"2023-10-16T13:47:41.953154+0000","last_became_active":"2023-10-16T13:47:40.923664+0000","last_became_peered":"2023-10-16T13:47:40.923664+0000","last_unstale":"2023-10-16T13:47:41.953154+0000","last_undegraded":"2023-10-16T13:47:41.953154+0000","last_fullsized":"2023-10-16T13:47:41.953154+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_clean_scrub_stamp":"2023-10-16T13:47:39.890588+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:16:06.419458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1e","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.919140+0000","last_change":"2023-10-16T13:47:24.439610+0000","last_active":"2023-10-16T13:47:40.919140+0000","last_peered":"2023-10-16T13:47:40.919140+0000","last_clean":"2023-10-16T13:47:40.919140+0000","last_became_active":"2023-10-16T13:47:24.439212+0000","last_became_peered":"2023-10-16T13:47:24.439212+0000","last_unstale":"2023-10-16T13:47:40.919140+0000","last_undegraded":"2023-10-16T13:47:40.919140+0000","last_fullsized":"2023-10-16T13:47:40.919140+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:47:24.106272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.1f","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.961266+0000","last_change":"2023-10-16T13:47:41.954643+0000","last_active":"2023-10-16T13:47:41.961266+0000","last_peered":"2023-10-16T13:47:41.961266+0000","last_clean":"2023-10-16T13:47:41.961266+0000","last_became_active":"2023-10-16T13:47:41.954321+0000","last_became_peered":"2023-10-16T13:47:41.954321+0000","last_unstale":"2023-10-16T13:47:41.961266+0000","last_undegraded":"2023-10-16T13:47:41.961266+0000","last_fullsized":"2023-10-16T13:47:41.961266+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:23:28.072263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1c","version":"248'2","reported_seq":13,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.945744+0000","last_change":"2023-10-16T13:47:40.920627+0000","last_active":"2023-10-16T13:47:41.945744+0000","last_peered":"2023-10-16T13:47:41.945744+0000","last_clean":"2023-10-16T13:47:41.945744+0000","last_became_active":"2023-10-16T13:47:40.920289+0000","last_became_peered":"2023-10-16T13:47:40.920289+0000","last_unstale":"2023-10-16T13:47:41.945744+0000","last_undegraded":"2023-10-16T13:47:41.945744+0000","last_fullsized":"2023-10-16T13:47:41.945744+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_clean_scrub_stamp":"2023-10-16T13:47:39.890588+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:37:19.404184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1d","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.917110+0000","last_change":"2023-10-16T13:47:24.432216+0000","last_active":"2023-10-16T13:47:40.917110+0000","last_peered":"2023-10-16T13:47:40.917110+0000","last_clean":"2023-10-16T13:47:40.917110+0000","last_became_active":"2023-10-16T13:47:24.431937+0000","last_became_peered":"2023-10-16T13:47:24.431937+0000","last_unstale":"2023-10-16T13:47:40.917110+0000","last_undegraded":"2023-10-16T13:47:40.917110+0000","last_fullsized":"2023-10-16T13:47:40.917110+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:23:10.580869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.1e","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.957795+0000","last_change":"2023-10-16T13:47:41.955036+0000","last_active":"2023-10-16T13:47:41.957795+0000","last_peered":"2023-10-16T13:47:41.957795+0000","last_clean":"2023-10-16T13:47:41.957795+0000","last_became_active":"2023-10-16T13:47:41.954734+0000","last_became_peered":"2023-10-16T13:47:41.954734+0000","last_unstale":"2023-10-16T13:47:41.957795+0000","last_undegraded":"2023-10-16T13:47:41.957795+0000","last_fullsized":"2023-10-16T13:47:41.957795+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:53:38.009404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1d","version":"248'2","reported_seq":13,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.960404+0000","last_change":"2023-10-16T13:47:40.924994+0000","last_active":"2023-10-16T13:47:41.960404+0000","last_peered":"2023-10-16T13:47:41.960404+0000","last_clean":"2023-10-16T13:47:41.960404+0000","last_became_active":"2023-10-16T13:47:40.924683+0000","last_became_peered":"2023-10-16T13:47:40.924683+0000","last_unstale":"2023-10-16T13:47:41.960404+0000","last_undegraded":"2023-10-16T13:47:41.960404+0000","last_fullsized":"2023-10-16T13:47:41.960404+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_clean_scrub_stamp":"2023-10-16T13:47:39.890588+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:52:44.697434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.1c","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.917406+0000","last_change":"2023-10-16T13:47:24.440252+0000","last_active":"2023-10-16T13:47:40.917406+0000","last_peered":"2023-10-16T13:47:40.917406+0000","last_clean":"2023-10-16T13:47:40.917406+0000","last_became_active":"2023-10-16T13:47:24.439982+0000","last_became_peered":"2023-10-16T13:47:24.439982+0000","last_unstale":"2023-10-16T13:47:40.917406+0000","last_undegraded":"2023-10-16T13:47:40.917406+0000","last_fullsized":"2023-10-16T13:47:40.917406+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:54:02.930019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.19","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.952047+0000","last_change":"2023-10-16T13:47:41.948411+0000","last_active":"2023-10-16T13:47:41.952047+0000","last_peered":"2023-10-16T13:47:41.952047+0000","last_clean":"2023-10-16T13:47:41.952047+0000","last_became_active":"2023-10-16T13:47:41.948123+0000","last_became_peered":"2023-10-16T13:47:41.948123+0000","last_unstale":"2023-10-16T13:47:41.952047+0000","last_undegraded":"2023-10-16T13:47:41.952047+0000","last_fullsized":"2023-10-16T13:47:41.952047+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:33:03.023121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1a","version":"248'1","reported_seq":12,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.953618+0000","last_change":"2023-10-16T13:47:40.906916+0000","last_active":"2023-10-16T13:47:41.953618+0000","last_peered":"2023-10-16T13:47:41.953618+0000","last_clean":"2023-10-16T13:47:41.953618+0000","last_became_active":"2023-10-16T13:47:40.906594+0000","last_became_peered":"2023-10-16T13:47:40.906594+0000","last_unstale":"2023-10-16T13:47:41.953618+0000","last_undegraded":"2023-10-16T13:47:41.953618+0000","last_fullsized":"2023-10-16T13:47:41.953618+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_clean_scrub_stamp":"2023-10-16T13:47:39.890588+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:11:06.086887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.1b","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.908372+0000","last_change":"2023-10-16T13:47:24.440049+0000","last_active":"2023-10-16T13:47:40.908372+0000","last_peered":"2023-10-16T13:47:40.908372+0000","last_clean":"2023-10-16T13:47:40.908372+0000","last_became_active":"2023-10-16T13:47:24.439750+0000","last_became_peered":"2023-10-16T13:47:24.439750+0000","last_unstale":"2023-10-16T13:47:40.908372+0000","last_undegraded":"2023-10-16T13:47:40.908372+0000","last_fullsized":"2023-10-16T13:47:40.908372+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:33:40.094175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.18","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.948706+0000","last_change":"2023-10-16T13:47:41.945478+0000","last_active":"2023-10-16T13:47:41.948706+0000","last_peered":"2023-10-16T13:47:41.948706+0000","last_clean":"2023-10-16T13:47:41.948706+0000","last_became_active":"2023-10-16T13:47:41.945033+0000","last_became_peered":"2023-10-16T13:47:41.945033+0000","last_unstale":"2023-10-16T13:47:41.948706+0000","last_undegraded":"2023-10-16T13:47:41.948706+0000","last_fullsized":"2023-10-16T13:47:41.948706+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:42:05.562484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1b","version":"248'1","reported_seq":12,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.945214+0000","last_change":"2023-10-16T13:47:40.923823+0000","last_active":"2023-10-16T13:47:41.945214+0000","last_peered":"2023-10-16T13:47:41.945214+0000","last_clean":"2023-10-16T13:47:41.945214+0000","last_became_active":"2023-10-16T13:47:40.923524+0000","last_became_peered":"2023-10-16T13:47:40.923524+0000","last_unstale":"2023-10-16T13:47:41.945214+0000","last_undegraded":"2023-10-16T13:47:41.945214+0000","last_fullsized":"2023-10-16T13:47:41.945214+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_clean_scrub_stamp":"2023-10-16T13:47:39.890588+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:27:11.351621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1a","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.909354+0000","last_change":"2023-10-16T13:47:24.433661+0000","last_active":"2023-10-16T13:47:40.909354+0000","last_peered":"2023-10-16T13:47:40.909354+0000","last_clean":"2023-10-16T13:47:40.909354+0000","last_became_active":"2023-10-16T13:47:24.433417+0000","last_became_peered":"2023-10-16T13:47:24.433417+0000","last_unstale":"2023-10-16T13:47:40.909354+0000","last_undegraded":"2023-10-16T13:47:40.909354+0000","last_fullsized":"2023-10-16T13:47:40.909354+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:16:02.391378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1b","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.955228+0000","last_change":"2023-10-16T13:47:41.950957+0000","last_active":"2023-10-16T13:47:41.955228+0000","last_peered":"2023-10-16T13:47:41.955228+0000","last_clean":"2023-10-16T13:47:41.955228+0000","last_became_active":"2023-10-16T13:47:41.945092+0000","last_became_peered":"2023-10-16T13:47:41.945092+0000","last_unstale":"2023-10-16T13:47:41.955228+0000","last_undegraded":"2023-10-16T13:47:41.955228+0000","last_fullsized":"2023-10-16T13:47:41.955228+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:14:41.165689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.18","version":"0'0","reported_seq":11,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.932088+0000","last_change":"2023-10-16T13:47:40.924893+0000","last_active":"2023-10-16T13:47:41.932088+0000","last_peered":"2023-10-16T13:47:41.932088+0000","last_clean":"2023-10-16T13:47:41.932088+0000","last_became_active":"2023-10-16T13:47:40.924606+0000","last_became_peered":"2023-10-16T13:47:40.924606+0000","last_unstale":"2023-10-16T13:47:41.932088+0000","last_undegraded":"2023-10-16T13:47:41.932088+0000","last_fullsized":"2023-10-16T13:47:41.932088+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_clean_scrub_stamp":"2023-10-16T13:47:39.890588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:16:08.357053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.909760+0000","last_change":"2023-10-16T13:47:24.437673+0000","last_active":"2023-10-16T13:47:40.909760+0000","last_peered":"2023-10-16T13:47:40.909760+0000","last_clean":"2023-10-16T13:47:40.909760+0000","last_became_active":"2023-10-16T13:47:24.437479+0000","last_became_peered":"2023-10-16T13:47:24.437479+0000","last_unstale":"2023-10-16T13:47:40.909760+0000","last_undegraded":"2023-10-16T13:47:40.909760+0000","last_fullsized":"2023-10-16T13:47:40.909760+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:23:58.749586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.1a","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.954976+0000","last_change":"2023-10-16T13:47:41.946361+0000","last_active":"2023-10-16T13:47:41.954976+0000","last_peered":"2023-10-16T13:47:41.954976+0000","last_clean":"2023-10-16T13:47:41.954976+0000","last_became_active":"2023-10-16T13:47:41.946074+0000","last_became_peered":"2023-10-16T13:47:41.946074+0000","last_unstale":"2023-10-16T13:47:41.954976+0000","last_undegraded":"2023-10-16T13:47:41.954976+0000","last_fullsized":"2023-10-16T13:47:41.954976+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:59:32.553302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.19","version":"0'0","reported_seq":11,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.932544+0000","last_change":"2023-10-16T13:47:40.921334+0000","last_active":"2023-10-16T13:47:41.932544+0000","last_peered":"2023-10-16T13:47:41.932544+0000","last_clean":"2023-10-16T13:47:41.932544+0000","last_became_active":"2023-10-16T13:47:40.920988+0000","last_became_peered":"2023-10-16T13:47:40.920988+0000","last_unstale":"2023-10-16T13:47:41.932544+0000","last_undegraded":"2023-10-16T13:47:41.932544+0000","last_fullsized":"2023-10-16T13:47:41.932544+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_clean_scrub_stamp":"2023-10-16T13:47:39.890588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:09:43.358758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.907346+0000","last_change":"2023-10-16T13:47:24.426442+0000","last_active":"2023-10-16T13:47:40.907346+0000","last_peered":"2023-10-16T13:47:40.907346+0000","last_clean":"2023-10-16T13:47:40.907346+0000","last_became_active":"2023-10-16T13:47:24.425929+0000","last_became_peered":"2023-10-16T13:47:24.425929+0000","last_unstale":"2023-10-16T13:47:40.907346+0000","last_undegraded":"2023-10-16T13:47:40.907346+0000","last_fullsized":"2023-10-16T13:47:40.907346+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:05:13.565304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.15","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.953140+0000","last_change":"2023-10-16T13:47:41.943255+0000","last_active":"2023-10-16T13:47:41.953140+0000","last_peered":"2023-10-16T13:47:41.953140+0000","last_clean":"2023-10-16T13:47:41.953140+0000","last_became_active":"2023-10-16T13:47:41.942596+0000","last_became_peered":"2023-10-16T13:47:41.942596+0000","last_unstale":"2023-10-16T13:47:41.953140+0000","last_undegraded":"2023-10-16T13:47:41.953140+0000","last_fullsized":"2023-10-16T13:47:41.953140+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:45:27.820639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.16","version":"0'0","reported_seq":11,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.934608+0000","last_change":"2023-10-16T13:47:40.922500+0000","last_active":"2023-10-16T13:47:41.934608+0000","last_peered":"2023-10-16T13:47:41.934608+0000","last_clean":"2023-10-16T13:47:41.934608+0000","last_became_active":"2023-10-16T13:47:40.921368+0000","last_became_peered":"2023-10-16T13:47:40.921368+0000","last_unstale":"2023-10-16T13:47:41.934608+0000","last_undegraded":"2023-10-16T13:47:41.934608+0000","last_fullsized":"2023-10-16T13:47:41.934608+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_clean_scrub_stamp":"2023-10-16T13:47:39.890588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:14:48.165605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.908613+0000","last_change":"2023-10-16T13:47:24.426459+0000","last_active":"2023-10-16T13:47:40.908613+0000","last_peered":"2023-10-16T13:47:40.908613+0000","last_clean":"2023-10-16T13:47:40.908613+0000","last_became_active":"2023-10-16T13:47:24.426169+0000","last_became_peered":"2023-10-16T13:47:24.426169+0000","last_unstale":"2023-10-16T13:47:40.908613+0000","last_undegraded":"2023-10-16T13:47:40.908613+0000","last_fullsized":"2023-10-16T13:47:40.908613+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:57:42.170954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.14","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.955137+0000","last_change":"2023-10-16T13:47:41.952226+0000","last_active":"2023-10-16T13:47:41.955137+0000","last_peered":"2023-10-16T13:47:41.955137+0000","last_clean":"2023-10-16T13:47:41.955137+0000","last_became_active":"2023-10-16T13:47:41.951929+0000","last_became_peered":"2023-10-16T13:47:41.951929+0000","last_unstale":"2023-10-16T13:47:41.955137+0000","last_undegraded":"2023-10-16T13:47:41.955137+0000","last_fullsized":"2023-10-16T13:47:41.955137+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:11:32.758621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.17","version":"0'0","reported_seq":11,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.925971+0000","last_change":"2023-10-16T13:47:40.921347+0000","last_active":"2023-10-16T13:47:41.925971+0000","last_peered":"2023-10-16T13:47:41.925971+0000","last_clean":"2023-10-16T13:47:41.925971+0000","last_became_active":"2023-10-16T13:47:40.921032+0000","last_became_peered":"2023-10-16T13:47:40.921032+0000","last_unstale":"2023-10-16T13:47:41.925971+0000","last_undegraded":"2023-10-16T13:47:41.925971+0000","last_fullsized":"2023-10-16T13:47:41.925971+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_clean_scrub_stamp":"2023-10-16T13:47:39.890588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:39:28.570014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.917620+0000","last_change":"2023-10-16T13:47:24.427333+0000","last_active":"2023-10-16T13:47:40.917620+0000","last_peered":"2023-10-16T13:47:40.917620+0000","last_clean":"2023-10-16T13:47:40.917620+0000","last_became_active":"2023-10-16T13:47:24.427034+0000","last_became_peered":"2023-10-16T13:47:24.427034+0000","last_unstale":"2023-10-16T13:47:40.917620+0000","last_undegraded":"2023-10-16T13:47:40.917620+0000","last_fullsized":"2023-10-16T13:47:40.917620+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:27:20.181386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.17","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.952402+0000","last_change":"2023-10-16T13:47:41.945082+0000","last_active":"2023-10-16T13:47:41.952402+0000","last_peered":"2023-10-16T13:47:41.952402+0000","last_clean":"2023-10-16T13:47:41.952402+0000","last_became_active":"2023-10-16T13:47:41.944627+0000","last_became_peered":"2023-10-16T13:47:41.944627+0000","last_unstale":"2023-10-16T13:47:41.952402+0000","last_undegraded":"2023-10-16T13:47:41.952402+0000","last_fullsized":"2023-10-16T13:47:41.952402+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:16:55.340334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.14","version":"248'6","reported_seq":17,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.958613+0000","last_change":"2023-10-16T13:47:40.920578+0000","last_active":"2023-10-16T13:47:41.958613+0000","last_peered":"2023-10-16T13:47:41.958613+0000","last_clean":"2023-10-16T13:47:41.958613+0000","last_became_active":"2023-10-16T13:47:40.920319+0000","last_became_peered":"2023-10-16T13:47:40.920319+0000","last_unstale":"2023-10-16T13:47:41.958613+0000","last_undegraded":"2023-10-16T13:47:41.958613+0000","last_fullsized":"2023-10-16T13:47:41.958613+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_clean_scrub_stamp":"2023-10-16T13:47:39.890588+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:22:21.193371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.15","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.917198+0000","last_change":"2023-10-16T13:47:24.425903+0000","last_active":"2023-10-16T13:47:40.917198+0000","last_peered":"2023-10-16T13:47:40.917198+0000","last_clean":"2023-10-16T13:47:40.917198+0000","last_became_active":"2023-10-16T13:47:24.425593+0000","last_became_peered":"2023-10-16T13:47:24.425593+0000","last_unstale":"2023-10-16T13:47:40.917198+0000","last_undegraded":"2023-10-16T13:47:40.917198+0000","last_fullsized":"2023-10-16T13:47:40.917198+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:46:35.236231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.16","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.956175+0000","last_change":"2023-10-16T13:47:41.955747+0000","last_active":"2023-10-16T13:47:41.956175+0000","last_peered":"2023-10-16T13:47:41.956175+0000","last_clean":"2023-10-16T13:47:41.956175+0000","last_became_active":"2023-10-16T13:47:41.955443+0000","last_became_peered":"2023-10-16T13:47:41.955443+0000","last_unstale":"2023-10-16T13:47:41.956175+0000","last_undegraded":"2023-10-16T13:47:41.956175+0000","last_fullsized":"2023-10-16T13:47:41.956175+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:11:07.092642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.15","version":"248'3","reported_seq":14,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.955416+0000","last_change":"2023-10-16T13:47:40.920585+0000","last_active":"2023-10-16T13:47:41.955416+0000","last_peered":"2023-10-16T13:47:41.955416+0000","last_clean":"2023-10-16T13:47:41.955416+0000","last_became_active":"2023-10-16T13:47:40.920279+0000","last_became_peered":"2023-10-16T13:47:40.920279+0000","last_unstale":"2023-10-16T13:47:41.955416+0000","last_undegraded":"2023-10-16T13:47:41.955416+0000","last_fullsized":"2023-10-16T13:47:41.955416+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_clean_scrub_stamp":"2023-10-16T13:47:39.890588+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:20:13.942596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.14","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.908088+0000","last_change":"2023-10-16T13:47:24.438927+0000","last_active":"2023-10-16T13:47:40.908088+0000","last_peered":"2023-10-16T13:47:40.908088+0000","last_clean":"2023-10-16T13:47:40.908088+0000","last_became_active":"2023-10-16T13:47:24.438389+0000","last_became_peered":"2023-10-16T13:47:24.438389+0000","last_unstale":"2023-10-16T13:47:40.908088+0000","last_undegraded":"2023-10-16T13:47:40.908088+0000","last_fullsized":"2023-10-16T13:47:40.908088+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:44:00.360485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.11","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.952808+0000","last_change":"2023-10-16T13:47:41.949123+0000","last_active":"2023-10-16T13:47:41.952808+0000","last_peered":"2023-10-16T13:47:41.952808+0000","last_clean":"2023-10-16T13:47:41.952808+0000","last_became_active":"2023-10-16T13:47:41.940490+0000","last_became_peered":"2023-10-16T13:47:41.940490+0000","last_unstale":"2023-10-16T13:47:41.952808+0000","last_undegraded":"2023-10-16T13:47:41.952808+0000","last_fullsized":"2023-10-16T13:47:41.952808+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:48:48.259775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.12","version":"0'0","reported_seq":11,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.927024+0000","last_change":"2023-10-16T13:47:40.916172+0000","last_active":"2023-10-16T13:47:41.927024+0000","last_peered":"2023-10-16T13:47:41.927024+0000","last_clean":"2023-10-16T13:47:41.927024+0000","last_became_active":"2023-10-16T13:47:40.915949+0000","last_became_peered":"2023-10-16T13:47:40.915949+0000","last_unstale":"2023-10-16T13:47:41.927024+0000","last_undegraded":"2023-10-16T13:47:41.927024+0000","last_fullsized":"2023-10-16T13:47:41.927024+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_clean_scrub_stamp":"2023-10-16T13:47:39.890588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:10:20.628054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.13","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.911535+0000","last_change":"2023-10-16T13:47:24.445148+0000","last_active":"2023-10-16T13:47:40.911535+0000","last_peered":"2023-10-16T13:47:40.911535+0000","last_clean":"2023-10-16T13:47:40.911535+0000","last_became_active":"2023-10-16T13:47:24.444896+0000","last_became_peered":"2023-10-16T13:47:24.444896+0000","last_unstale":"2023-10-16T13:47:40.911535+0000","last_undegraded":"2023-10-16T13:47:40.911535+0000","last_fullsized":"2023-10-16T13:47:40.911535+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:35:13.886134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.10","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.942446+0000","last_change":"2023-10-16T13:47:41.940848+0000","last_active":"2023-10-16T13:47:41.942446+0000","last_peered":"2023-10-16T13:47:41.942446+0000","last_clean":"2023-10-16T13:47:41.942446+0000","last_became_active":"2023-10-16T13:47:41.940524+0000","last_became_peered":"2023-10-16T13:47:41.940524+0000","last_unstale":"2023-10-16T13:47:41.942446+0000","last_undegraded":"2023-10-16T13:47:41.942446+0000","last_fullsized":"2023-10-16T13:47:41.942446+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:16:57.274456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.13","version":"248'2","reported_seq":13,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.956305+0000","last_change":"2023-10-16T13:47:40.924961+0000","last_active":"2023-10-16T13:47:41.956305+0000","last_peered":"2023-10-16T13:47:41.956305+0000","last_clean":"2023-10-16T13:47:41.956305+0000","last_became_active":"2023-10-16T13:47:40.924672+0000","last_became_peered":"2023-10-16T13:47:40.924672+0000","last_unstale":"2023-10-16T13:47:41.956305+0000","last_undegraded":"2023-10-16T13:47:41.956305+0000","last_fullsized":"2023-10-16T13:47:41.956305+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_clean_scrub_stamp":"2023-10-16T13:47:39.890588+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:50:03.482163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.903870+0000","last_change":"2023-10-16T13:47:24.437693+0000","last_active":"2023-10-16T13:47:40.903870+0000","last_peered":"2023-10-16T13:47:40.903870+0000","last_clean":"2023-10-16T13:47:40.903870+0000","last_became_active":"2023-10-16T13:47:24.437434+0000","last_became_peered":"2023-10-16T13:47:24.437434+0000","last_unstale":"2023-10-16T13:47:40.903870+0000","last_undegraded":"2023-10-16T13:47:40.903870+0000","last_fullsized":"2023-10-16T13:47:40.903870+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:47:56.221847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.13","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.945182+0000","last_change":"2023-10-16T13:47:41.939870+0000","last_active":"2023-10-16T13:47:41.945182+0000","last_peered":"2023-10-16T13:47:41.945182+0000","last_clean":"2023-10-16T13:47:41.945182+0000","last_became_active":"2023-10-16T13:47:41.939443+0000","last_became_peered":"2023-10-16T13:47:41.939443+0000","last_unstale":"2023-10-16T13:47:41.945182+0000","last_undegraded":"2023-10-16T13:47:41.945182+0000","last_fullsized":"2023-10-16T13:47:41.945182+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:04:17.705847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.10","version":"248'8","reported_seq":19,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.961146+0000","last_change":"2023-10-16T13:47:40.920255+0000","last_active":"2023-10-16T13:47:41.961146+0000","last_peered":"2023-10-16T13:47:41.961146+0000","last_clean":"2023-10-16T13:47:41.961146+0000","last_became_active":"2023-10-16T13:47:40.918169+0000","last_became_peered":"2023-10-16T13:47:40.918169+0000","last_unstale":"2023-10-16T13:47:41.961146+0000","last_undegraded":"2023-10-16T13:47:41.961146+0000","last_fullsized":"2023-10-16T13:47:41.961146+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_clean_scrub_stamp":"2023-10-16T13:47:39.890588+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:29:57.627195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.916781+0000","last_change":"2023-10-16T13:47:24.435339+0000","last_active":"2023-10-16T13:47:40.916781+0000","last_peered":"2023-10-16T13:47:40.916781+0000","last_clean":"2023-10-16T13:47:40.916781+0000","last_became_active":"2023-10-16T13:47:24.434237+0000","last_became_peered":"2023-10-16T13:47:24.434237+0000","last_unstale":"2023-10-16T13:47:40.916781+0000","last_undegraded":"2023-10-16T13:47:40.916781+0000","last_fullsized":"2023-10-16T13:47:40.916781+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:21:21.083320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.12","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.946754+0000","last_change":"2023-10-16T13:47:41.943481+0000","last_active":"2023-10-16T13:47:41.946754+0000","last_peered":"2023-10-16T13:47:41.946754+0000","last_clean":"2023-10-16T13:47:41.946754+0000","last_became_active":"2023-10-16T13:47:41.943190+0000","last_became_peered":"2023-10-16T13:47:41.943190+0000","last_unstale":"2023-10-16T13:47:41.946754+0000","last_undegraded":"2023-10-16T13:47:41.946754+0000","last_fullsized":"2023-10-16T13:47:41.946754+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:16:24.457537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.11","version":"0'0","reported_seq":11,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.937163+0000","last_change":"2023-10-16T13:47:40.914735+0000","last_active":"2023-10-16T13:47:41.937163+0000","last_peered":"2023-10-16T13:47:41.937163+0000","last_clean":"2023-10-16T13:47:41.937163+0000","last_became_active":"2023-10-16T13:47:40.914441+0000","last_became_peered":"2023-10-16T13:47:40.914441+0000","last_unstale":"2023-10-16T13:47:41.937163+0000","last_undegraded":"2023-10-16T13:47:41.937163+0000","last_fullsized":"2023-10-16T13:47:41.937163+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_clean_scrub_stamp":"2023-10-16T13:47:39.890588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:35:10.098500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.10","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.912641+0000","last_change":"2023-10-16T13:47:24.441060+0000","last_active":"2023-10-16T13:47:40.912641+0000","last_peered":"2023-10-16T13:47:40.912641+0000","last_clean":"2023-10-16T13:47:40.912641+0000","last_became_active":"2023-10-16T13:47:24.440745+0000","last_became_peered":"2023-10-16T13:47:24.440745+0000","last_unstale":"2023-10-16T13:47:40.912641+0000","last_undegraded":"2023-10-16T13:47:40.912641+0000","last_fullsized":"2023-10-16T13:47:40.912641+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:48:26.805075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.d","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.952889+0000","last_change":"2023-10-16T13:47:41.949823+0000","last_active":"2023-10-16T13:47:41.952889+0000","last_peered":"2023-10-16T13:47:41.952889+0000","last_clean":"2023-10-16T13:47:41.952889+0000","last_became_active":"2023-10-16T13:47:41.949551+0000","last_became_peered":"2023-10-16T13:47:41.949551+0000","last_unstale":"2023-10-16T13:47:41.952889+0000","last_undegraded":"2023-10-16T13:47:41.952889+0000","last_fullsized":"2023-10-16T13:47:41.952889+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:42:46.808296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.e","version":"248'2","reported_seq":13,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.946445+0000","last_change":"2023-10-16T13:47:40.924491+0000","last_active":"2023-10-16T13:47:41.946445+0000","last_peered":"2023-10-16T13:47:41.946445+0000","last_clean":"2023-10-16T13:47:41.946445+0000","last_became_active":"2023-10-16T13:47:40.924197+0000","last_became_peered":"2023-10-16T13:47:40.924197+0000","last_unstale":"2023-10-16T13:47:41.946445+0000","last_undegraded":"2023-10-16T13:47:41.946445+0000","last_fullsized":"2023-10-16T13:47:41.946445+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_clean_scrub_stamp":"2023-10-16T13:47:39.890588+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:17:47.585967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.f","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.920388+0000","last_change":"2023-10-16T13:47:24.438572+0000","last_active":"2023-10-16T13:47:40.920388+0000","last_peered":"2023-10-16T13:47:40.920388+0000","last_clean":"2023-10-16T13:47:40.920388+0000","last_became_active":"2023-10-16T13:47:24.438283+0000","last_became_peered":"2023-10-16T13:47:24.438283+0000","last_unstale":"2023-10-16T13:47:40.920388+0000","last_undegraded":"2023-10-16T13:47:40.920388+0000","last_fullsized":"2023-10-16T13:47:40.920388+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:48:21.499862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.c","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.947191+0000","last_change":"2023-10-16T13:47:41.941965+0000","last_active":"2023-10-16T13:47:41.947191+0000","last_peered":"2023-10-16T13:47:41.947191+0000","last_clean":"2023-10-16T13:47:41.947191+0000","last_became_active":"2023-10-16T13:47:41.941661+0000","last_became_peered":"2023-10-16T13:47:41.941661+0000","last_unstale":"2023-10-16T13:47:41.947191+0000","last_undegraded":"2023-10-16T13:47:41.947191+0000","last_fullsized":"2023-10-16T13:47:41.947191+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:18:34.935509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.f","version":"248'1","reported_seq":12,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.955586+0000","last_change":"2023-10-16T13:47:40.925860+0000","last_active":"2023-10-16T13:47:41.955586+0000","last_peered":"2023-10-16T13:47:41.955586+0000","last_clean":"2023-10-16T13:47:41.955586+0000","last_became_active":"2023-10-16T13:47:40.925444+0000","last_became_peered":"2023-10-16T13:47:40.925444+0000","last_unstale":"2023-10-16T13:47:41.955586+0000","last_undegraded":"2023-10-16T13:47:41.955586+0000","last_fullsized":"2023-10-16T13:47:41.955586+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_clean_scrub_stamp":"2023-10-16T13:47:39.890588+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:44:11.539188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.e","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.917726+0000","last_change":"2023-10-16T13:47:24.433506+0000","last_active":"2023-10-16T13:47:40.917726+0000","last_peered":"2023-10-16T13:47:40.917726+0000","last_clean":"2023-10-16T13:47:40.917726+0000","last_became_active":"2023-10-16T13:47:24.433218+0000","last_became_peered":"2023-10-16T13:47:24.433218+0000","last_unstale":"2023-10-16T13:47:40.917726+0000","last_undegraded":"2023-10-16T13:47:40.917726+0000","last_fullsized":"2023-10-16T13:47:40.917726+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:04:28.651883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.f","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.956437+0000","last_change":"2023-10-16T13:47:41.955137+0000","last_active":"2023-10-16T13:47:41.956437+0000","last_peered":"2023-10-16T13:47:41.956437+0000","last_clean":"2023-10-16T13:47:41.956437+0000","last_became_active":"2023-10-16T13:47:41.954831+0000","last_became_peered":"2023-10-16T13:47:41.954831+0000","last_unstale":"2023-10-16T13:47:41.956437+0000","last_undegraded":"2023-10-16T13:47:41.956437+0000","last_fullsized":"2023-10-16T13:47:41.956437+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:20:04.850917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.c","version":"0'0","reported_seq":11,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.930092+0000","last_change":"2023-10-16T13:47:40.925515+0000","last_active":"2023-10-16T13:47:41.930092+0000","last_peered":"2023-10-16T13:47:41.930092+0000","last_clean":"2023-10-16T13:47:41.930092+0000","last_became_active":"2023-10-16T13:47:40.925203+0000","last_became_peered":"2023-10-16T13:47:40.925203+0000","last_unstale":"2023-10-16T13:47:41.930092+0000","last_undegraded":"2023-10-16T13:47:41.930092+0000","last_fullsized":"2023-10-16T13:47:41.930092+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_clean_scrub_stamp":"2023-10-16T13:47:39.890588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:03:13.465329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.d","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.916690+0000","last_change":"2023-10-16T13:47:24.443171+0000","last_active":"2023-10-16T13:47:40.916690+0000","last_peered":"2023-10-16T13:47:40.916690+0000","last_clean":"2023-10-16T13:47:40.916690+0000","last_became_active":"2023-10-16T13:47:24.442811+0000","last_became_peered":"2023-10-16T13:47:24.442811+0000","last_unstale":"2023-10-16T13:47:40.916690+0000","last_undegraded":"2023-10-16T13:47:40.916690+0000","last_fullsized":"2023-10-16T13:47:40.916690+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:56:16.580840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.e","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.953264+0000","last_change":"2023-10-16T13:47:41.940562+0000","last_active":"2023-10-16T13:47:41.953264+0000","last_peered":"2023-10-16T13:47:41.953264+0000","last_clean":"2023-10-16T13:47:41.953264+0000","last_became_active":"2023-10-16T13:47:41.940278+0000","last_became_peered":"2023-10-16T13:47:41.940278+0000","last_unstale":"2023-10-16T13:47:41.953264+0000","last_undegraded":"2023-10-16T13:47:41.953264+0000","last_fullsized":"2023-10-16T13:47:41.953264+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:45:09.969851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.d","version":"248'1","reported_seq":12,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.961641+0000","last_change":"2023-10-16T13:47:40.920222+0000","last_active":"2023-10-16T13:47:41.961641+0000","last_peered":"2023-10-16T13:47:41.961641+0000","last_clean":"2023-10-16T13:47:41.961641+0000","last_became_active":"2023-10-16T13:47:40.918810+0000","last_became_peered":"2023-10-16T13:47:40.918810+0000","last_unstale":"2023-10-16T13:47:41.961641+0000","last_undegraded":"2023-10-16T13:47:41.961641+0000","last_fullsized":"2023-10-16T13:47:41.961641+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_clean_scrub_stamp":"2023-10-16T13:47:39.890588+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:48:43.290715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.919136+0000","last_change":"2023-10-16T13:47:24.435000+0000","last_active":"2023-10-16T13:47:40.919136+0000","last_peered":"2023-10-16T13:47:40.919136+0000","last_clean":"2023-10-16T13:47:40.919136+0000","last_became_active":"2023-10-16T13:47:24.434568+0000","last_became_peered":"2023-10-16T13:47:24.434568+0000","last_unstale":"2023-10-16T13:47:40.919136+0000","last_undegraded":"2023-10-16T13:47:40.919136+0000","last_fullsized":"2023-10-16T13:47:40.919136+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:31:46.967495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.9","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.956331+0000","last_change":"2023-10-16T13:47:41.954083+0000","last_active":"2023-10-16T13:47:41.956331+0000","last_peered":"2023-10-16T13:47:41.956331+0000","last_clean":"2023-10-16T13:47:41.956331+0000","last_became_active":"2023-10-16T13:47:41.953721+0000","last_became_peered":"2023-10-16T13:47:41.953721+0000","last_unstale":"2023-10-16T13:47:41.956331+0000","last_undegraded":"2023-10-16T13:47:41.956331+0000","last_fullsized":"2023-10-16T13:47:41.956331+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:36:49.327966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.a","version":"0'0","reported_seq":11,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.932545+0000","last_change":"2023-10-16T13:47:40.927530+0000","last_active":"2023-10-16T13:47:41.932545+0000","last_peered":"2023-10-16T13:47:41.932545+0000","last_clean":"2023-10-16T13:47:41.932545+0000","last_became_active":"2023-10-16T13:47:40.927244+0000","last_became_peered":"2023-10-16T13:47:40.927244+0000","last_unstale":"2023-10-16T13:47:41.932545+0000","last_undegraded":"2023-10-16T13:47:41.932545+0000","last_fullsized":"2023-10-16T13:47:41.932545+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_clean_scrub_stamp":"2023-10-16T13:47:39.890588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:53:13.673249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.b","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.910729+0000","last_change":"2023-10-16T13:47:24.432842+0000","last_active":"2023-10-16T13:47:40.910729+0000","last_peered":"2023-10-16T13:47:40.910729+0000","last_clean":"2023-10-16T13:47:40.910729+0000","last_became_active":"2023-10-16T13:47:24.432528+0000","last_became_peered":"2023-10-16T13:47:24.432528+0000","last_unstale":"2023-10-16T13:47:40.910729+0000","last_undegraded":"2023-10-16T13:47:40.910729+0000","last_fullsized":"2023-10-16T13:47:40.910729+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:10:57.744179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.8","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.961869+0000","last_change":"2023-10-16T13:47:41.954409+0000","last_active":"2023-10-16T13:47:41.961869+0000","last_peered":"2023-10-16T13:47:41.961869+0000","last_clean":"2023-10-16T13:47:41.961869+0000","last_became_active":"2023-10-16T13:47:41.954075+0000","last_became_peered":"2023-10-16T13:47:41.954075+0000","last_unstale":"2023-10-16T13:47:41.961869+0000","last_undegraded":"2023-10-16T13:47:41.961869+0000","last_fullsized":"2023-10-16T13:47:41.961869+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:52:17.219170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.b","version":"248'2","reported_seq":13,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.959020+0000","last_change":"2023-10-16T13:47:40.918271+0000","last_active":"2023-10-16T13:47:41.959020+0000","last_peered":"2023-10-16T13:47:41.959020+0000","last_clean":"2023-10-16T13:47:41.959020+0000","last_became_active":"2023-10-16T13:47:40.917978+0000","last_became_peered":"2023-10-16T13:47:40.917978+0000","last_unstale":"2023-10-16T13:47:41.959020+0000","last_undegraded":"2023-10-16T13:47:41.959020+0000","last_fullsized":"2023-10-16T13:47:41.959020+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_clean_scrub_stamp":"2023-10-16T13:47:39.890588+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:31:49.189007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.911633+0000","last_change":"2023-10-16T13:47:24.439953+0000","last_active":"2023-10-16T13:47:40.911633+0000","last_peered":"2023-10-16T13:47:40.911633+0000","last_clean":"2023-10-16T13:47:40.911633+0000","last_became_active":"2023-10-16T13:47:24.439662+0000","last_became_peered":"2023-10-16T13:47:24.439662+0000","last_unstale":"2023-10-16T13:47:40.911633+0000","last_undegraded":"2023-10-16T13:47:40.911633+0000","last_fullsized":"2023-10-16T13:47:40.911633+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:16:33.663778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.b","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.934998+0000","last_change":"2023-10-16T13:47:41.934609+0000","last_active":"2023-10-16T13:47:41.934998+0000","last_peered":"2023-10-16T13:47:41.934998+0000","last_clean":"2023-10-16T13:47:41.934998+0000","last_became_active":"2023-10-16T13:47:41.934325+0000","last_became_peered":"2023-10-16T13:47:41.934325+0000","last_unstale":"2023-10-16T13:47:41.934998+0000","last_undegraded":"2023-10-16T13:47:41.934998+0000","last_fullsized":"2023-10-16T13:47:41.934998+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:13:29.287316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.8","version":"0'0","reported_seq":11,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.938607+0000","last_change":"2023-10-16T13:47:40.910759+0000","last_active":"2023-10-16T13:47:41.938607+0000","last_peered":"2023-10-16T13:47:41.938607+0000","last_clean":"2023-10-16T13:47:41.938607+0000","last_became_active":"2023-10-16T13:47:40.910438+0000","last_became_peered":"2023-10-16T13:47:40.910438+0000","last_unstale":"2023-10-16T13:47:41.938607+0000","last_undegraded":"2023-10-16T13:47:41.938607+0000","last_fullsized":"2023-10-16T13:47:41.938607+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_clean_scrub_stamp":"2023-10-16T13:47:39.890588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:03:49.084910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.916407+0000","last_change":"2023-10-16T13:47:24.442582+0000","last_active":"2023-10-16T13:47:40.916407+0000","last_peered":"2023-10-16T13:47:40.916407+0000","last_clean":"2023-10-16T13:47:40.916407+0000","last_became_active":"2023-10-16T13:47:24.442246+0000","last_became_peered":"2023-10-16T13:47:24.442246+0000","last_unstale":"2023-10-16T13:47:40.916407+0000","last_undegraded":"2023-10-16T13:47:40.916407+0000","last_fullsized":"2023-10-16T13:47:40.916407+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:48:26.805075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.a","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.955126+0000","last_change":"2023-10-16T13:47:41.951343+0000","last_active":"2023-10-16T13:47:41.955126+0000","last_peered":"2023-10-16T13:47:41.955126+0000","last_clean":"2023-10-16T13:47:41.955126+0000","last_became_active":"2023-10-16T13:47:41.951030+0000","last_became_peered":"2023-10-16T13:47:41.951030+0000","last_unstale":"2023-10-16T13:47:41.955126+0000","last_undegraded":"2023-10-16T13:47:41.955126+0000","last_fullsized":"2023-10-16T13:47:41.955126+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:01:00.671432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.9","version":"0'0","reported_seq":11,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.925335+0000","last_change":"2023-10-16T13:47:40.926244+0000","last_active":"2023-10-16T13:47:41.925335+0000","last_peered":"2023-10-16T13:47:41.925335+0000","last_clean":"2023-10-16T13:47:41.925335+0000","last_became_active":"2023-10-16T13:47:40.924533+0000","last_became_peered":"2023-10-16T13:47:40.924533+0000","last_unstale":"2023-10-16T13:47:41.925335+0000","last_undegraded":"2023-10-16T13:47:41.925335+0000","last_fullsized":"2023-10-16T13:47:41.925335+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_clean_scrub_stamp":"2023-10-16T13:47:39.890588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:21:13.301295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.911998+0000","last_change":"2023-10-16T13:47:24.437336+0000","last_active":"2023-10-16T13:47:40.911998+0000","last_peered":"2023-10-16T13:47:40.911998+0000","last_clean":"2023-10-16T13:47:40.911998+0000","last_became_active":"2023-10-16T13:47:24.437042+0000","last_became_peered":"2023-10-16T13:47:24.437042+0000","last_unstale":"2023-10-16T13:47:40.911998+0000","last_undegraded":"2023-10-16T13:47:40.911998+0000","last_fullsized":"2023-10-16T13:47:40.911998+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:07:16.886792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.5","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.956114+0000","last_change":"2023-10-16T13:47:41.953682+0000","last_active":"2023-10-16T13:47:41.956114+0000","last_peered":"2023-10-16T13:47:41.956114+0000","last_clean":"2023-10-16T13:47:41.956114+0000","last_became_active":"2023-10-16T13:47:41.953349+0000","last_became_peered":"2023-10-16T13:47:41.953349+0000","last_unstale":"2023-10-16T13:47:41.956114+0000","last_undegraded":"2023-10-16T13:47:41.956114+0000","last_fullsized":"2023-10-16T13:47:41.956114+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:04:46.143278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.6","version":"0'0","reported_seq":11,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.933870+0000","last_change":"2023-10-16T13:47:40.925445+0000","last_active":"2023-10-16T13:47:41.933870+0000","last_peered":"2023-10-16T13:47:41.933870+0000","last_clean":"2023-10-16T13:47:41.933870+0000","last_became_active":"2023-10-16T13:47:40.925135+0000","last_became_peered":"2023-10-16T13:47:40.925135+0000","last_unstale":"2023-10-16T13:47:41.933870+0000","last_undegraded":"2023-10-16T13:47:41.933870+0000","last_fullsized":"2023-10-16T13:47:41.933870+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_clean_scrub_stamp":"2023-10-16T13:47:39.890588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:26:23.723585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.7","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.918630+0000","last_change":"2023-10-16T13:47:24.433031+0000","last_active":"2023-10-16T13:47:40.918630+0000","last_peered":"2023-10-16T13:47:40.918630+0000","last_clean":"2023-10-16T13:47:40.918630+0000","last_became_active":"2023-10-16T13:47:24.432700+0000","last_became_peered":"2023-10-16T13:47:24.432700+0000","last_unstale":"2023-10-16T13:47:40.918630+0000","last_undegraded":"2023-10-16T13:47:40.918630+0000","last_fullsized":"2023-10-16T13:47:40.918630+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:17:10.080681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.4","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.953868+0000","last_change":"2023-10-16T13:47:41.948145+0000","last_active":"2023-10-16T13:47:41.953868+0000","last_peered":"2023-10-16T13:47:41.953868+0000","last_clean":"2023-10-16T13:47:41.953868+0000","last_became_active":"2023-10-16T13:47:41.947487+0000","last_became_peered":"2023-10-16T13:47:41.947487+0000","last_unstale":"2023-10-16T13:47:41.953868+0000","last_undegraded":"2023-10-16T13:47:41.953868+0000","last_fullsized":"2023-10-16T13:47:41.953868+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:52:17.219170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.7","version":"248'2","reported_seq":13,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.953896+0000","last_change":"2023-10-16T13:47:40.922011+0000","last_active":"2023-10-16T13:47:41.953896+0000","last_peered":"2023-10-16T13:47:41.953896+0000","last_clean":"2023-10-16T13:47:41.953896+0000","last_became_active":"2023-10-16T13:47:40.921705+0000","last_became_peered":"2023-10-16T13:47:40.921705+0000","last_unstale":"2023-10-16T13:47:41.953896+0000","last_undegraded":"2023-10-16T13:47:41.953896+0000","last_fullsized":"2023-10-16T13:47:41.953896+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_clean_scrub_stamp":"2023-10-16T13:47:39.890588+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:07:57.036479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.6","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.909762+0000","last_change":"2023-10-16T13:47:24.434972+0000","last_active":"2023-10-16T13:47:40.909762+0000","last_peered":"2023-10-16T13:47:40.909762+0000","last_clean":"2023-10-16T13:47:40.909762+0000","last_became_active":"2023-10-16T13:47:24.434639+0000","last_became_peered":"2023-10-16T13:47:24.434639+0000","last_unstale":"2023-10-16T13:47:40.909762+0000","last_undegraded":"2023-10-16T13:47:40.909762+0000","last_fullsized":"2023-10-16T13:47:40.909762+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:06:13.278420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.7","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.952706+0000","last_change":"2023-10-16T13:47:41.950972+0000","last_active":"2023-10-16T13:47:41.952706+0000","last_peered":"2023-10-16T13:47:41.952706+0000","last_clean":"2023-10-16T13:47:41.952706+0000","last_became_active":"2023-10-16T13:47:41.950422+0000","last_became_peered":"2023-10-16T13:47:41.950422+0000","last_unstale":"2023-10-16T13:47:41.952706+0000","last_undegraded":"2023-10-16T13:47:41.952706+0000","last_fullsized":"2023-10-16T13:47:41.952706+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:20:24.501093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.4","version":"0'0","reported_seq":11,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.945774+0000","last_change":"2023-10-16T13:47:40.919382+0000","last_active":"2023-10-16T13:47:41.945774+0000","last_peered":"2023-10-16T13:47:41.945774+0000","last_clean":"2023-10-16T13:47:41.945774+0000","last_became_active":"2023-10-16T13:47:40.918911+0000","last_became_peered":"2023-10-16T13:47:40.918911+0000","last_unstale":"2023-10-16T13:47:41.945774+0000","last_undegraded":"2023-10-16T13:47:41.945774+0000","last_fullsized":"2023-10-16T13:47:41.945774+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_clean_scrub_stamp":"2023-10-16T13:47:39.890588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:24:04.571425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.907830+0000","last_change":"2023-10-16T13:47:24.428801+0000","last_active":"2023-10-16T13:47:40.907830+0000","last_peered":"2023-10-16T13:47:40.907830+0000","last_clean":"2023-10-16T13:47:40.907830+0000","last_became_active":"2023-10-16T13:47:24.428511+0000","last_became_peered":"2023-10-16T13:47:24.428511+0000","last_unstale":"2023-10-16T13:47:40.907830+0000","last_undegraded":"2023-10-16T13:47:40.907830+0000","last_fullsized":"2023-10-16T13:47:40.907830+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:27:55.268379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.6","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.954858+0000","last_change":"2023-10-16T13:47:41.946681+0000","last_active":"2023-10-16T13:47:41.954858+0000","last_peered":"2023-10-16T13:47:41.954858+0000","last_clean":"2023-10-16T13:47:41.954858+0000","last_became_active":"2023-10-16T13:47:41.946384+0000","last_became_peered":"2023-10-16T13:47:41.946384+0000","last_unstale":"2023-10-16T13:47:41.954858+0000","last_undegraded":"2023-10-16T13:47:41.954858+0000","last_fullsized":"2023-10-16T13:47:41.954858+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:23:58.300909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.5","version":"0'0","reported_seq":11,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.946799+0000","last_change":"2023-10-16T13:47:40.919745+0000","last_active":"2023-10-16T13:47:41.946799+0000","last_peered":"2023-10-16T13:47:41.946799+0000","last_clean":"2023-10-16T13:47:41.946799+0000","last_became_active":"2023-10-16T13:47:40.919442+0000","last_became_peered":"2023-10-16T13:47:40.919442+0000","last_unstale":"2023-10-16T13:47:41.946799+0000","last_undegraded":"2023-10-16T13:47:41.946799+0000","last_fullsized":"2023-10-16T13:47:41.946799+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_clean_scrub_stamp":"2023-10-16T13:47:39.890588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:19:54.778811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.907995+0000","last_change":"2023-10-16T13:47:24.429772+0000","last_active":"2023-10-16T13:47:40.907995+0000","last_peered":"2023-10-16T13:47:40.907995+0000","last_clean":"2023-10-16T13:47:40.907995+0000","last_became_active":"2023-10-16T13:47:24.429496+0000","last_became_peered":"2023-10-16T13:47:24.429496+0000","last_unstale":"2023-10-16T13:47:40.907995+0000","last_undegraded":"2023-10-16T13:47:40.907995+0000","last_fullsized":"2023-10-16T13:47:40.907995+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:09:48.751438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.952024+0000","last_change":"2023-10-16T13:47:41.946231+0000","last_active":"2023-10-16T13:47:41.952024+0000","last_peered":"2023-10-16T13:47:41.952024+0000","last_clean":"2023-10-16T13:47:41.952024+0000","last_became_active":"2023-10-16T13:47:41.945943+0000","last_became_peered":"2023-10-16T13:47:41.945943+0000","last_unstale":"2023-10-16T13:47:41.952024+0000","last_undegraded":"2023-10-16T13:47:41.952024+0000","last_fullsized":"2023-10-16T13:47:41.952024+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:27:37.672780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.2","version":"0'0","reported_seq":11,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.931458+0000","last_change":"2023-10-16T13:47:40.923159+0000","last_active":"2023-10-16T13:47:41.931458+0000","last_peered":"2023-10-16T13:47:41.931458+0000","last_clean":"2023-10-16T13:47:41.931458+0000","last_became_active":"2023-10-16T13:47:40.922887+0000","last_became_peered":"2023-10-16T13:47:40.922887+0000","last_unstale":"2023-10-16T13:47:41.931458+0000","last_undegraded":"2023-10-16T13:47:41.931458+0000","last_fullsized":"2023-10-16T13:47:41.931458+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_clean_scrub_stamp":"2023-10-16T13:47:39.890588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:16:50.149418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.902331+0000","last_change":"2023-10-16T13:47:24.437263+0000","last_active":"2023-10-16T13:47:40.902331+0000","last_peered":"2023-10-16T13:47:40.902331+0000","last_clean":"2023-10-16T13:47:40.902331+0000","last_became_active":"2023-10-16T13:47:24.437023+0000","last_became_peered":"2023-10-16T13:47:24.437023+0000","last_unstale":"2023-10-16T13:47:40.902331+0000","last_undegraded":"2023-10-16T13:47:40.902331+0000","last_fullsized":"2023-10-16T13:47:40.902331+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:59:15.061907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.0","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.955100+0000","last_change":"2023-10-16T13:47:41.948549+0000","last_active":"2023-10-16T13:47:41.955100+0000","last_peered":"2023-10-16T13:47:41.955100+0000","last_clean":"2023-10-16T13:47:41.955100+0000","last_became_active":"2023-10-16T13:47:41.948269+0000","last_became_peered":"2023-10-16T13:47:41.948269+0000","last_unstale":"2023-10-16T13:47:41.955100+0000","last_undegraded":"2023-10-16T13:47:41.955100+0000","last_fullsized":"2023-10-16T13:47:41.955100+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:50:47.237986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.3","version":"0'0","reported_seq":11,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.931248+0000","last_change":"2023-10-16T13:47:40.923995+0000","last_active":"2023-10-16T13:47:41.931248+0000","last_peered":"2023-10-16T13:47:41.931248+0000","last_clean":"2023-10-16T13:47:41.931248+0000","last_became_active":"2023-10-16T13:47:40.923691+0000","last_became_peered":"2023-10-16T13:47:40.923691+0000","last_unstale":"2023-10-16T13:47:41.931248+0000","last_undegraded":"2023-10-16T13:47:41.931248+0000","last_fullsized":"2023-10-16T13:47:41.931248+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_clean_scrub_stamp":"2023-10-16T13:47:39.890588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:48:37.985502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.903986+0000","last_change":"2023-10-16T13:47:24.419847+0000","last_active":"2023-10-16T13:47:40.903986+0000","last_peered":"2023-10-16T13:47:40.903986+0000","last_clean":"2023-10-16T13:47:40.903986+0000","last_became_active":"2023-10-16T13:47:24.419520+0000","last_became_peered":"2023-10-16T13:47:24.419520+0000","last_unstale":"2023-10-16T13:47:40.903986+0000","last_undegraded":"2023-10-16T13:47:40.903986+0000","last_fullsized":"2023-10-16T13:47:40.903986+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:11:44.932011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.3","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.939712+0000","last_change":"2023-10-16T13:47:41.938833+0000","last_active":"2023-10-16T13:47:41.939712+0000","last_peered":"2023-10-16T13:47:41.939712+0000","last_clean":"2023-10-16T13:47:41.939712+0000","last_became_active":"2023-10-16T13:47:41.938389+0000","last_became_peered":"2023-10-16T13:47:41.938389+0000","last_unstale":"2023-10-16T13:47:41.939712+0000","last_undegraded":"2023-10-16T13:47:41.939712+0000","last_fullsized":"2023-10-16T13:47:41.939712+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:16:01.800647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.0","version":"0'0","reported_seq":11,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.934728+0000","last_change":"2023-10-16T13:47:40.917887+0000","last_active":"2023-10-16T13:47:41.934728+0000","last_peered":"2023-10-16T13:47:41.934728+0000","last_clean":"2023-10-16T13:47:41.934728+0000","last_became_active":"2023-10-16T13:47:40.917596+0000","last_became_peered":"2023-10-16T13:47:40.917596+0000","last_unstale":"2023-10-16T13:47:41.934728+0000","last_undegraded":"2023-10-16T13:47:41.934728+0000","last_fullsized":"2023-10-16T13:47:41.934728+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_clean_scrub_stamp":"2023-10-16T13:47:39.890588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:11:54.742851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.918413+0000","last_change":"2023-10-16T13:47:24.435926+0000","last_active":"2023-10-16T13:47:40.918413+0000","last_peered":"2023-10-16T13:47:40.918413+0000","last_clean":"2023-10-16T13:47:40.918413+0000","last_became_active":"2023-10-16T13:47:24.434753+0000","last_became_peered":"2023-10-16T13:47:24.434753+0000","last_unstale":"2023-10-16T13:47:40.918413+0000","last_undegraded":"2023-10-16T13:47:40.918413+0000","last_fullsized":"2023-10-16T13:47:40.918413+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:13:51.332934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.2","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.956911+0000","last_change":"2023-10-16T13:47:41.952646+0000","last_active":"2023-10-16T13:47:41.956911+0000","last_peered":"2023-10-16T13:47:41.956911+0000","last_clean":"2023-10-16T13:47:41.956911+0000","last_became_active":"2023-10-16T13:47:41.952280+0000","last_became_peered":"2023-10-16T13:47:41.952280+0000","last_unstale":"2023-10-16T13:47:41.956911+0000","last_undegraded":"2023-10-16T13:47:41.956911+0000","last_fullsized":"2023-10-16T13:47:41.956911+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:28:20.482489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1","version":"0'0","reported_seq":11,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.945117+0000","last_change":"2023-10-16T13:47:40.919383+0000","last_active":"2023-10-16T13:47:41.945117+0000","last_peered":"2023-10-16T13:47:41.945117+0000","last_clean":"2023-10-16T13:47:41.945117+0000","last_became_active":"2023-10-16T13:47:40.919053+0000","last_became_peered":"2023-10-16T13:47:40.919053+0000","last_unstale":"2023-10-16T13:47:41.945117+0000","last_undegraded":"2023-10-16T13:47:41.945117+0000","last_fullsized":"2023-10-16T13:47:41.945117+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_clean_scrub_stamp":"2023-10-16T13:47:39.890588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:21:37.568960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.909357+0000","last_change":"2023-10-16T13:47:24.423908+0000","last_active":"2023-10-16T13:47:40.909357+0000","last_peered":"2023-10-16T13:47:40.909357+0000","last_clean":"2023-10-16T13:47:40.909357+0000","last_became_active":"2023-10-16T13:47:24.423530+0000","last_became_peered":"2023-10-16T13:47:24.423530+0000","last_unstale":"2023-10-16T13:47:40.909357+0000","last_undegraded":"2023-10-16T13:47:40.909357+0000","last_fullsized":"2023-10-16T13:47:40.909357+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:50:50.396650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.22","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.952670+0000","last_change":"2023-10-16T13:47:41.950769+0000","last_active":"2023-10-16T13:47:41.952670+0000","last_peered":"2023-10-16T13:47:41.952670+0000","last_clean":"2023-10-16T13:47:41.952670+0000","last_became_active":"2023-10-16T13:47:41.950103+0000","last_became_peered":"2023-10-16T13:47:41.950103+0000","last_unstale":"2023-10-16T13:47:41.952670+0000","last_undegraded":"2023-10-16T13:47:41.952670+0000","last_fullsized":"2023-10-16T13:47:41.952670+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:27:37.672780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.20","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.915340+0000","last_change":"2023-10-16T13:47:24.436480+0000","last_active":"2023-10-16T13:47:40.915340+0000","last_peered":"2023-10-16T13:47:40.915340+0000","last_clean":"2023-10-16T13:47:40.915340+0000","last_became_active":"2023-10-16T13:47:24.435309+0000","last_became_peered":"2023-10-16T13:47:24.435309+0000","last_unstale":"2023-10-16T13:47:40.915340+0000","last_undegraded":"2023-10-16T13:47:40.915340+0000","last_fullsized":"2023-10-16T13:47:40.915340+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:28:08.947506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1f","version":"244'4","reported_seq":19,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.908378+0000","last_change":"2023-10-16T13:47:23.425349+0000","last_active":"2023-10-16T13:47:40.908378+0000","last_peered":"2023-10-16T13:47:40.908378+0000","last_clean":"2023-10-16T13:47:40.908378+0000","last_became_active":"2023-10-16T13:47:23.425029+0000","last_became_peered":"2023-10-16T13:47:23.425029+0000","last_unstale":"2023-10-16T13:47:40.908378+0000","last_undegraded":"2023-10-16T13:47:40.908378+0000","last_fullsized":"2023-10-16T13:47:40.908378+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:57:35.834032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.20","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.956232+0000","last_change":"2023-10-16T13:47:41.948172+0000","last_active":"2023-10-16T13:47:41.956232+0000","last_peered":"2023-10-16T13:47:41.956232+0000","last_clean":"2023-10-16T13:47:41.956232+0000","last_became_active":"2023-10-16T13:47:41.947882+0000","last_became_peered":"2023-10-16T13:47:41.947882+0000","last_unstale":"2023-10-16T13:47:41.956232+0000","last_undegraded":"2023-10-16T13:47:41.956232+0000","last_fullsized":"2023-10-16T13:47:41.956232+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:54:20.421413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.22","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.916376+0000","last_change":"2023-10-16T13:47:24.431062+0000","last_active":"2023-10-16T13:47:40.916376+0000","last_peered":"2023-10-16T13:47:40.916376+0000","last_clean":"2023-10-16T13:47:40.916376+0000","last_became_active":"2023-10-16T13:47:24.430771+0000","last_became_peered":"2023-10-16T13:47:24.430771+0000","last_unstale":"2023-10-16T13:47:40.916376+0000","last_undegraded":"2023-10-16T13:47:40.916376+0000","last_fullsized":"2023-10-16T13:47:40.916376+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:23:48.085785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.1d","version":"244'2","reported_seq":17,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.919891+0000","last_change":"2023-10-16T13:47:23.428572+0000","last_active":"2023-10-16T13:47:40.919891+0000","last_peered":"2023-10-16T13:47:40.919891+0000","last_clean":"2023-10-16T13:47:40.919891+0000","last_became_active":"2023-10-16T13:47:23.428285+0000","last_became_peered":"2023-10-16T13:47:23.428285+0000","last_unstale":"2023-10-16T13:47:40.919891+0000","last_undegraded":"2023-10-16T13:47:40.919891+0000","last_fullsized":"2023-10-16T13:47:40.919891+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:26:03.650516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.21","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.960427+0000","last_change":"2023-10-16T13:47:41.953912+0000","last_active":"2023-10-16T13:47:41.960427+0000","last_peered":"2023-10-16T13:47:41.960427+0000","last_clean":"2023-10-16T13:47:41.960427+0000","last_became_active":"2023-10-16T13:47:41.953631+0000","last_became_peered":"2023-10-16T13:47:41.953631+0000","last_unstale":"2023-10-16T13:47:41.960427+0000","last_undegraded":"2023-10-16T13:47:41.960427+0000","last_fullsized":"2023-10-16T13:47:41.960427+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:43:37.671188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.23","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.907866+0000","last_change":"2023-10-16T13:47:24.432436+0000","last_active":"2023-10-16T13:47:40.907866+0000","last_peered":"2023-10-16T13:47:40.907866+0000","last_clean":"2023-10-16T13:47:40.907866+0000","last_became_active":"2023-10-16T13:47:24.428465+0000","last_became_peered":"2023-10-16T13:47:24.428465+0000","last_unstale":"2023-10-16T13:47:40.907866+0000","last_undegraded":"2023-10-16T13:47:40.907866+0000","last_fullsized":"2023-10-16T13:47:40.907866+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:16:50.560075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1c","version":"244'2","reported_seq":17,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.903280+0000","last_change":"2023-10-16T13:47:23.427686+0000","last_active":"2023-10-16T13:47:40.903280+0000","last_peered":"2023-10-16T13:47:40.903280+0000","last_clean":"2023-10-16T13:47:40.903280+0000","last_became_active":"2023-10-16T13:47:23.427013+0000","last_became_peered":"2023-10-16T13:47:23.427013+0000","last_unstale":"2023-10-16T13:47:40.903280+0000","last_undegraded":"2023-10-16T13:47:40.903280+0000","last_fullsized":"2023-10-16T13:47:40.903280+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:37:17.532164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.26","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.933821+0000","last_change":"2023-10-16T13:47:41.933043+0000","last_active":"2023-10-16T13:47:41.933821+0000","last_peered":"2023-10-16T13:47:41.933821+0000","last_clean":"2023-10-16T13:47:41.933821+0000","last_became_active":"2023-10-16T13:47:41.932784+0000","last_became_peered":"2023-10-16T13:47:41.932784+0000","last_unstale":"2023-10-16T13:47:41.933821+0000","last_undegraded":"2023-10-16T13:47:41.933821+0000","last_fullsized":"2023-10-16T13:47:41.933821+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:04:46.143278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.24","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.903182+0000","last_change":"2023-10-16T13:47:24.438644+0000","last_active":"2023-10-16T13:47:40.903182+0000","last_peered":"2023-10-16T13:47:40.903182+0000","last_clean":"2023-10-16T13:47:40.903182+0000","last_became_active":"2023-10-16T13:47:24.438344+0000","last_became_peered":"2023-10-16T13:47:24.438344+0000","last_unstale":"2023-10-16T13:47:40.903182+0000","last_undegraded":"2023-10-16T13:47:40.903182+0000","last_fullsized":"2023-10-16T13:47:40.903182+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:21:19.269920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.1b","version":"244'1","reported_seq":16,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.918079+0000","last_change":"2023-10-16T13:47:23.427234+0000","last_active":"2023-10-16T13:47:40.918079+0000","last_peered":"2023-10-16T13:47:40.918079+0000","last_clean":"2023-10-16T13:47:40.918079+0000","last_became_active":"2023-10-16T13:47:23.426948+0000","last_became_peered":"2023-10-16T13:47:23.426948+0000","last_unstale":"2023-10-16T13:47:40.918079+0000","last_undegraded":"2023-10-16T13:47:40.918079+0000","last_fullsized":"2023-10-16T13:47:40.918079+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:11:43.927649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.27","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.954896+0000","last_change":"2023-10-16T13:47:41.948872+0000","last_active":"2023-10-16T13:47:41.954896+0000","last_peered":"2023-10-16T13:47:41.954896+0000","last_clean":"2023-10-16T13:47:41.954896+0000","last_became_active":"2023-10-16T13:47:41.948548+0000","last_became_peered":"2023-10-16T13:47:41.948548+0000","last_unstale":"2023-10-16T13:47:41.954896+0000","last_undegraded":"2023-10-16T13:47:41.954896+0000","last_fullsized":"2023-10-16T13:47:41.954896+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:21:36.761315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.25","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.908796+0000","last_change":"2023-10-16T13:47:24.432803+0000","last_active":"2023-10-16T13:47:40.908796+0000","last_peered":"2023-10-16T13:47:40.908796+0000","last_clean":"2023-10-16T13:47:40.908796+0000","last_became_active":"2023-10-16T13:47:24.432470+0000","last_became_peered":"2023-10-16T13:47:24.432470+0000","last_unstale":"2023-10-16T13:47:40.908796+0000","last_undegraded":"2023-10-16T13:47:40.908796+0000","last_fullsized":"2023-10-16T13:47:40.908796+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:41:00.039237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.1a","version":"244'1","reported_seq":16,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.904098+0000","last_change":"2023-10-16T13:47:23.425695+0000","last_active":"2023-10-16T13:47:40.904098+0000","last_peered":"2023-10-16T13:47:40.904098+0000","last_clean":"2023-10-16T13:47:40.904098+0000","last_became_active":"2023-10-16T13:47:23.425407+0000","last_became_peered":"2023-10-16T13:47:23.425407+0000","last_unstale":"2023-10-16T13:47:40.904098+0000","last_undegraded":"2023-10-16T13:47:40.904098+0000","last_fullsized":"2023-10-16T13:47:40.904098+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:19:34.313200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.24","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.952382+0000","last_change":"2023-10-16T13:47:41.946625+0000","last_active":"2023-10-16T13:47:41.952382+0000","last_peered":"2023-10-16T13:47:41.952382+0000","last_clean":"2023-10-16T13:47:41.952382+0000","last_became_active":"2023-10-16T13:47:41.942547+0000","last_became_peered":"2023-10-16T13:47:41.942547+0000","last_unstale":"2023-10-16T13:47:41.952382+0000","last_undegraded":"2023-10-16T13:47:41.952382+0000","last_fullsized":"2023-10-16T13:47:41.952382+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:22:48.838787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.26","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.917522+0000","last_change":"2023-10-16T13:47:24.439616+0000","last_active":"2023-10-16T13:47:40.917522+0000","last_peered":"2023-10-16T13:47:40.917522+0000","last_clean":"2023-10-16T13:47:40.917522+0000","last_became_active":"2023-10-16T13:47:24.435678+0000","last_became_peered":"2023-10-16T13:47:24.435678+0000","last_unstale":"2023-10-16T13:47:40.917522+0000","last_undegraded":"2023-10-16T13:47:40.917522+0000","last_fullsized":"2023-10-16T13:47:40.917522+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:39:16.296666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.912378+0000","last_change":"2023-10-16T13:47:23.431681+0000","last_active":"2023-10-16T13:47:40.912378+0000","last_peered":"2023-10-16T13:47:40.912378+0000","last_clean":"2023-10-16T13:47:40.912378+0000","last_became_active":"2023-10-16T13:47:23.431317+0000","last_became_peered":"2023-10-16T13:47:23.431317+0000","last_unstale":"2023-10-16T13:47:40.912378+0000","last_undegraded":"2023-10-16T13:47:40.912378+0000","last_fullsized":"2023-10-16T13:47:40.912378+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:20:27.852907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.25","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.956367+0000","last_change":"2023-10-16T13:47:41.954814+0000","last_active":"2023-10-16T13:47:41.956367+0000","last_peered":"2023-10-16T13:47:41.956367+0000","last_clean":"2023-10-16T13:47:41.956367+0000","last_became_active":"2023-10-16T13:47:41.954529+0000","last_became_peered":"2023-10-16T13:47:41.954529+0000","last_unstale":"2023-10-16T13:47:41.956367+0000","last_undegraded":"2023-10-16T13:47:41.956367+0000","last_fullsized":"2023-10-16T13:47:41.956367+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:09:36.582479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.27","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.909914+0000","last_change":"2023-10-16T13:47:24.423771+0000","last_active":"2023-10-16T13:47:40.909914+0000","last_peered":"2023-10-16T13:47:40.909914+0000","last_clean":"2023-10-16T13:47:40.909914+0000","last_became_active":"2023-10-16T13:47:24.423459+0000","last_became_peered":"2023-10-16T13:47:24.423459+0000","last_unstale":"2023-10-16T13:47:40.909914+0000","last_undegraded":"2023-10-16T13:47:40.909914+0000","last_fullsized":"2023-10-16T13:47:40.909914+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:27:22.393558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.909463+0000","last_change":"2023-10-16T13:47:23.433310+0000","last_active":"2023-10-16T13:47:40.909463+0000","last_peered":"2023-10-16T13:47:40.909463+0000","last_clean":"2023-10-16T13:47:40.909463+0000","last_became_active":"2023-10-16T13:47:23.433018+0000","last_became_peered":"2023-10-16T13:47:23.433018+0000","last_unstale":"2023-10-16T13:47:40.909463+0000","last_undegraded":"2023-10-16T13:47:40.909463+0000","last_fullsized":"2023-10-16T13:47:40.909463+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:49:45.992161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.2a","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.940455+0000","last_change":"2023-10-16T13:47:41.939916+0000","last_active":"2023-10-16T13:47:41.940455+0000","last_peered":"2023-10-16T13:47:41.940455+0000","last_clean":"2023-10-16T13:47:41.940455+0000","last_became_active":"2023-10-16T13:47:41.939601+0000","last_became_peered":"2023-10-16T13:47:41.939601+0000","last_unstale":"2023-10-16T13:47:41.940455+0000","last_undegraded":"2023-10-16T13:47:41.940455+0000","last_fullsized":"2023-10-16T13:47:41.940455+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:28:15.447879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.28","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.906743+0000","last_change":"2023-10-16T13:47:24.433316+0000","last_active":"2023-10-16T13:47:40.906743+0000","last_peered":"2023-10-16T13:47:40.906743+0000","last_clean":"2023-10-16T13:47:40.906743+0000","last_became_active":"2023-10-16T13:47:24.429160+0000","last_became_peered":"2023-10-16T13:47:24.429160+0000","last_unstale":"2023-10-16T13:47:40.906743+0000","last_undegraded":"2023-10-16T13:47:40.906743+0000","last_fullsized":"2023-10-16T13:47:40.906743+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:12:35.760185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.904819+0000","last_change":"2023-10-16T13:47:23.428721+0000","last_active":"2023-10-16T13:47:40.904819+0000","last_peered":"2023-10-16T13:47:40.904819+0000","last_clean":"2023-10-16T13:47:40.904819+0000","last_became_active":"2023-10-16T13:47:23.428349+0000","last_became_peered":"2023-10-16T13:47:23.428349+0000","last_unstale":"2023-10-16T13:47:40.904819+0000","last_undegraded":"2023-10-16T13:47:40.904819+0000","last_fullsized":"2023-10-16T13:47:40.904819+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:59:11.116638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.2b","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.959655+0000","last_change":"2023-10-16T13:47:41.958608+0000","last_active":"2023-10-16T13:47:41.959655+0000","last_peered":"2023-10-16T13:47:41.959655+0000","last_clean":"2023-10-16T13:47:41.959655+0000","last_became_active":"2023-10-16T13:47:41.958334+0000","last_became_peered":"2023-10-16T13:47:41.958334+0000","last_unstale":"2023-10-16T13:47:41.959655+0000","last_undegraded":"2023-10-16T13:47:41.959655+0000","last_fullsized":"2023-10-16T13:47:41.959655+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:57:54.329788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.29","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.910632+0000","last_change":"2023-10-16T13:47:24.440635+0000","last_active":"2023-10-16T13:47:40.910632+0000","last_peered":"2023-10-16T13:47:40.910632+0000","last_clean":"2023-10-16T13:47:40.910632+0000","last_became_active":"2023-10-16T13:47:24.440223+0000","last_became_peered":"2023-10-16T13:47:24.440223+0000","last_unstale":"2023-10-16T13:47:40.910632+0000","last_undegraded":"2023-10-16T13:47:40.910632+0000","last_fullsized":"2023-10-16T13:47:40.910632+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:22:53.895897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.917821+0000","last_change":"2023-10-16T13:47:23.433545+0000","last_active":"2023-10-16T13:47:40.917821+0000","last_peered":"2023-10-16T13:47:40.917821+0000","last_clean":"2023-10-16T13:47:40.917821+0000","last_became_active":"2023-10-16T13:47:23.433275+0000","last_became_peered":"2023-10-16T13:47:23.433275+0000","last_unstale":"2023-10-16T13:47:40.917821+0000","last_undegraded":"2023-10-16T13:47:40.917821+0000","last_fullsized":"2023-10-16T13:47:40.917821+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:21:17.525717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.28","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.952753+0000","last_change":"2023-10-16T13:47:41.948368+0000","last_active":"2023-10-16T13:47:41.952753+0000","last_peered":"2023-10-16T13:47:41.952753+0000","last_clean":"2023-10-16T13:47:41.952753+0000","last_became_active":"2023-10-16T13:47:41.948097+0000","last_became_peered":"2023-10-16T13:47:41.948097+0000","last_unstale":"2023-10-16T13:47:41.952753+0000","last_undegraded":"2023-10-16T13:47:41.952753+0000","last_fullsized":"2023-10-16T13:47:41.952753+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:32:31.777807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.2a","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.912923+0000","last_change":"2023-10-16T13:47:24.424605+0000","last_active":"2023-10-16T13:47:40.912923+0000","last_peered":"2023-10-16T13:47:40.912923+0000","last_clean":"2023-10-16T13:47:40.912923+0000","last_became_active":"2023-10-16T13:47:24.424313+0000","last_became_peered":"2023-10-16T13:47:24.424313+0000","last_unstale":"2023-10-16T13:47:40.912923+0000","last_undegraded":"2023-10-16T13:47:40.912923+0000","last_fullsized":"2023-10-16T13:47:40.912923+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:40:33.368782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.15","version":"244'3","reported_seq":18,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.907932+0000","last_change":"2023-10-16T13:47:23.431025+0000","last_active":"2023-10-16T13:47:40.907932+0000","last_peered":"2023-10-16T13:47:40.907932+0000","last_clean":"2023-10-16T13:47:40.907932+0000","last_became_active":"2023-10-16T13:47:23.430710+0000","last_became_peered":"2023-10-16T13:47:23.430710+0000","last_unstale":"2023-10-16T13:47:40.907932+0000","last_undegraded":"2023-10-16T13:47:40.907932+0000","last_fullsized":"2023-10-16T13:47:40.907932+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:52:27.207432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.29","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.940690+0000","last_change":"2023-10-16T13:47:41.937466+0000","last_active":"2023-10-16T13:47:41.940690+0000","last_peered":"2023-10-16T13:47:41.940690+0000","last_clean":"2023-10-16T13:47:41.940690+0000","last_became_active":"2023-10-16T13:47:41.937116+0000","last_became_peered":"2023-10-16T13:47:41.937116+0000","last_unstale":"2023-10-16T13:47:41.940690+0000","last_undegraded":"2023-10-16T13:47:41.940690+0000","last_fullsized":"2023-10-16T13:47:41.940690+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:44:12.544942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.2b","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.908612+0000","last_change":"2023-10-16T13:47:24.427810+0000","last_active":"2023-10-16T13:47:40.908612+0000","last_peered":"2023-10-16T13:47:40.908612+0000","last_clean":"2023-10-16T13:47:40.908612+0000","last_became_active":"2023-10-16T13:47:24.427276+0000","last_became_peered":"2023-10-16T13:47:24.427276+0000","last_unstale":"2023-10-16T13:47:40.908612+0000","last_undegraded":"2023-10-16T13:47:40.908612+0000","last_fullsized":"2023-10-16T13:47:40.908612+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:16:12.551854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.14","version":"245'9","reported_seq":24,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.908096+0000","last_change":"2023-10-16T13:47:23.430528+0000","last_active":"2023-10-16T13:47:40.908096+0000","last_peered":"2023-10-16T13:47:40.908096+0000","last_clean":"2023-10-16T13:47:40.908096+0000","last_became_active":"2023-10-16T13:47:23.429721+0000","last_became_peered":"2023-10-16T13:47:23.429721+0000","last_unstale":"2023-10-16T13:47:40.908096+0000","last_undegraded":"2023-10-16T13:47:40.908096+0000","last_fullsized":"2023-10-16T13:47:40.908096+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:07:56.379842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2859,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.2e","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.957806+0000","last_change":"2023-10-16T13:47:41.954884+0000","last_active":"2023-10-16T13:47:41.957806+0000","last_peered":"2023-10-16T13:47:41.957806+0000","last_clean":"2023-10-16T13:47:41.957806+0000","last_became_active":"2023-10-16T13:47:41.954332+0000","last_became_peered":"2023-10-16T13:47:41.954332+0000","last_unstale":"2023-10-16T13:47:41.957806+0000","last_undegraded":"2023-10-16T13:47:41.957806+0000","last_fullsized":"2023-10-16T13:47:41.957806+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:06:03.891766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.2c","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.919047+0000","last_change":"2023-10-16T13:47:24.436132+0000","last_active":"2023-10-16T13:47:40.919047+0000","last_peered":"2023-10-16T13:47:40.919047+0000","last_clean":"2023-10-16T13:47:40.919047+0000","last_became_active":"2023-10-16T13:47:24.435272+0000","last_became_peered":"2023-10-16T13:47:24.435272+0000","last_unstale":"2023-10-16T13:47:40.919047+0000","last_undegraded":"2023-10-16T13:47:40.919047+0000","last_fullsized":"2023-10-16T13:47:40.919047+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:07:59.434149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.13","version":"244'2","reported_seq":17,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.918177+0000","last_change":"2023-10-16T13:47:23.426620+0000","last_active":"2023-10-16T13:47:40.918177+0000","last_peered":"2023-10-16T13:47:40.918177+0000","last_clean":"2023-10-16T13:47:40.918177+0000","last_became_active":"2023-10-16T13:47:23.426328+0000","last_became_peered":"2023-10-16T13:47:23.426328+0000","last_unstale":"2023-10-16T13:47:40.918177+0000","last_undegraded":"2023-10-16T13:47:40.918177+0000","last_fullsized":"2023-10-16T13:47:40.918177+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:23:39.805153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.2f","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.954728+0000","last_change":"2023-10-16T13:47:41.949649+0000","last_active":"2023-10-16T13:47:41.954728+0000","last_peered":"2023-10-16T13:47:41.954728+0000","last_clean":"2023-10-16T13:47:41.954728+0000","last_became_active":"2023-10-16T13:47:41.949363+0000","last_became_peered":"2023-10-16T13:47:41.949363+0000","last_unstale":"2023-10-16T13:47:41.954728+0000","last_undegraded":"2023-10-16T13:47:41.954728+0000","last_fullsized":"2023-10-16T13:47:41.954728+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:57:15.234272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2d","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.908823+0000","last_change":"2023-10-16T13:47:24.431734+0000","last_active":"2023-10-16T13:47:40.908823+0000","last_peered":"2023-10-16T13:47:40.908823+0000","last_clean":"2023-10-16T13:47:40.908823+0000","last_became_active":"2023-10-16T13:47:24.431414+0000","last_became_peered":"2023-10-16T13:47:24.431414+0000","last_unstale":"2023-10-16T13:47:40.908823+0000","last_undegraded":"2023-10-16T13:47:40.908823+0000","last_fullsized":"2023-10-16T13:47:40.908823+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:01:31.790432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.906560+0000","last_change":"2023-10-16T13:47:23.433222+0000","last_active":"2023-10-16T13:47:40.906560+0000","last_peered":"2023-10-16T13:47:40.906560+0000","last_clean":"2023-10-16T13:47:40.906560+0000","last_became_active":"2023-10-16T13:47:23.432936+0000","last_became_peered":"2023-10-16T13:47:23.432936+0000","last_unstale":"2023-10-16T13:47:40.906560+0000","last_undegraded":"2023-10-16T13:47:40.906560+0000","last_fullsized":"2023-10-16T13:47:40.906560+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:47:59.336633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.2c","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.953890+0000","last_change":"2023-10-16T13:47:41.948335+0000","last_active":"2023-10-16T13:47:41.953890+0000","last_peered":"2023-10-16T13:47:41.953890+0000","last_clean":"2023-10-16T13:47:41.953890+0000","last_became_active":"2023-10-16T13:47:41.947856+0000","last_became_peered":"2023-10-16T13:47:41.947856+0000","last_unstale":"2023-10-16T13:47:41.953890+0000","last_undegraded":"2023-10-16T13:47:41.953890+0000","last_fullsized":"2023-10-16T13:47:41.953890+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:21:14.307049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.2e","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.917851+0000","last_change":"2023-10-16T13:47:24.438281+0000","last_active":"2023-10-16T13:47:40.917851+0000","last_peered":"2023-10-16T13:47:40.917851+0000","last_clean":"2023-10-16T13:47:40.917851+0000","last_became_active":"2023-10-16T13:47:24.437976+0000","last_became_peered":"2023-10-16T13:47:24.437976+0000","last_unstale":"2023-10-16T13:47:40.917851+0000","last_undegraded":"2023-10-16T13:47:40.917851+0000","last_fullsized":"2023-10-16T13:47:40.917851+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:19:38.293171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.905879+0000","last_change":"2023-10-16T13:47:23.425872+0000","last_active":"2023-10-16T13:47:40.905879+0000","last_peered":"2023-10-16T13:47:40.905879+0000","last_clean":"2023-10-16T13:47:40.905879+0000","last_became_active":"2023-10-16T13:47:23.425569+0000","last_became_peered":"2023-10-16T13:47:23.425569+0000","last_unstale":"2023-10-16T13:47:40.905879+0000","last_undegraded":"2023-10-16T13:47:40.905879+0000","last_fullsized":"2023-10-16T13:47:40.905879+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:09:18.086723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.2d","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.954148+0000","last_change":"2023-10-16T13:47:41.947743+0000","last_active":"2023-10-16T13:47:41.954148+0000","last_peered":"2023-10-16T13:47:41.954148+0000","last_clean":"2023-10-16T13:47:41.954148+0000","last_became_active":"2023-10-16T13:47:41.947502+0000","last_became_peered":"2023-10-16T13:47:41.947502+0000","last_unstale":"2023-10-16T13:47:41.954148+0000","last_undegraded":"2023-10-16T13:47:41.954148+0000","last_fullsized":"2023-10-16T13:47:41.954148+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:23:25.484030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2f","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.909795+0000","last_change":"2023-10-16T13:47:24.437030+0000","last_active":"2023-10-16T13:47:40.909795+0000","last_peered":"2023-10-16T13:47:40.909795+0000","last_clean":"2023-10-16T13:47:40.909795+0000","last_became_active":"2023-10-16T13:47:24.436734+0000","last_became_peered":"2023-10-16T13:47:24.436734+0000","last_unstale":"2023-10-16T13:47:40.909795+0000","last_undegraded":"2023-10-16T13:47:40.909795+0000","last_fullsized":"2023-10-16T13:47:40.909795+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:28:53.363293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.10","version":"244'8","reported_seq":23,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.915239+0000","last_change":"2023-10-16T13:47:23.426967+0000","last_active":"2023-10-16T13:47:40.915239+0000","last_peered":"2023-10-16T13:47:40.915239+0000","last_clean":"2023-10-16T13:47:40.915239+0000","last_became_active":"2023-10-16T13:47:23.426614+0000","last_became_peered":"2023-10-16T13:47:23.426614+0000","last_unstale":"2023-10-16T13:47:40.915239+0000","last_undegraded":"2023-10-16T13:47:40.915239+0000","last_fullsized":"2023-10-16T13:47:40.915239+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:39:12.211841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.32","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.960379+0000","last_change":"2023-10-16T13:47:41.953476+0000","last_active":"2023-10-16T13:47:41.960379+0000","last_peered":"2023-10-16T13:47:41.960379+0000","last_clean":"2023-10-16T13:47:41.960379+0000","last_became_active":"2023-10-16T13:47:41.953194+0000","last_became_peered":"2023-10-16T13:47:41.953194+0000","last_unstale":"2023-10-16T13:47:41.960379+0000","last_undegraded":"2023-10-16T13:47:41.960379+0000","last_fullsized":"2023-10-16T13:47:41.960379+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:20:24.501093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.30","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.907242+0000","last_change":"2023-10-16T13:47:24.437081+0000","last_active":"2023-10-16T13:47:40.907242+0000","last_peered":"2023-10-16T13:47:40.907242+0000","last_clean":"2023-10-16T13:47:40.907242+0000","last_became_active":"2023-10-16T13:47:24.436501+0000","last_became_peered":"2023-10-16T13:47:24.436501+0000","last_unstale":"2023-10-16T13:47:40.907242+0000","last_undegraded":"2023-10-16T13:47:40.907242+0000","last_fullsized":"2023-10-16T13:47:40.907242+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:23:30.831964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.f","version":"244'1","reported_seq":16,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.908273+0000","last_change":"2023-10-16T13:47:23.432161+0000","last_active":"2023-10-16T13:47:40.908273+0000","last_peered":"2023-10-16T13:47:40.908273+0000","last_clean":"2023-10-16T13:47:40.908273+0000","last_became_active":"2023-10-16T13:47:23.431822+0000","last_became_peered":"2023-10-16T13:47:23.431822+0000","last_unstale":"2023-10-16T13:47:40.908273+0000","last_undegraded":"2023-10-16T13:47:40.908273+0000","last_fullsized":"2023-10-16T13:47:40.908273+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:15:50.867051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.33","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.954838+0000","last_change":"2023-10-16T13:47:41.946832+0000","last_active":"2023-10-16T13:47:41.954838+0000","last_peered":"2023-10-16T13:47:41.954838+0000","last_clean":"2023-10-16T13:47:41.954838+0000","last_became_active":"2023-10-16T13:47:41.944213+0000","last_became_peered":"2023-10-16T13:47:41.944213+0000","last_unstale":"2023-10-16T13:47:41.954838+0000","last_undegraded":"2023-10-16T13:47:41.954838+0000","last_fullsized":"2023-10-16T13:47:41.954838+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:59:46.409384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.31","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.905467+0000","last_change":"2023-10-16T13:47:24.431186+0000","last_active":"2023-10-16T13:47:40.905467+0000","last_peered":"2023-10-16T13:47:40.905467+0000","last_clean":"2023-10-16T13:47:40.905467+0000","last_became_active":"2023-10-16T13:47:24.430892+0000","last_became_peered":"2023-10-16T13:47:24.430892+0000","last_unstale":"2023-10-16T13:47:40.905467+0000","last_undegraded":"2023-10-16T13:47:40.905467+0000","last_fullsized":"2023-10-16T13:47:40.905467+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:12:21.693658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.e","version":"244'2","reported_seq":17,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.918964+0000","last_change":"2023-10-16T13:47:23.424549+0000","last_active":"2023-10-16T13:47:40.918964+0000","last_peered":"2023-10-16T13:47:40.918964+0000","last_clean":"2023-10-16T13:47:40.918964+0000","last_became_active":"2023-10-16T13:47:23.424111+0000","last_became_peered":"2023-10-16T13:47:23.424111+0000","last_unstale":"2023-10-16T13:47:40.918964+0000","last_undegraded":"2023-10-16T13:47:40.918964+0000","last_fullsized":"2023-10-16T13:47:40.918964+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:36:35.137200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.30","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.939840+0000","last_change":"2023-10-16T13:47:41.938201+0000","last_active":"2023-10-16T13:47:41.939840+0000","last_peered":"2023-10-16T13:47:41.939840+0000","last_clean":"2023-10-16T13:47:41.939840+0000","last_became_active":"2023-10-16T13:47:41.937814+0000","last_became_peered":"2023-10-16T13:47:41.937814+0000","last_unstale":"2023-10-16T13:47:41.939840+0000","last_undegraded":"2023-10-16T13:47:41.939840+0000","last_fullsized":"2023-10-16T13:47:41.939840+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:48:11.978315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.32","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.918418+0000","last_change":"2023-10-16T13:47:24.432940+0000","last_active":"2023-10-16T13:47:40.918418+0000","last_peered":"2023-10-16T13:47:40.918418+0000","last_clean":"2023-10-16T13:47:40.918418+0000","last_became_active":"2023-10-16T13:47:24.432610+0000","last_became_peered":"2023-10-16T13:47:24.432610+0000","last_unstale":"2023-10-16T13:47:40.918418+0000","last_undegraded":"2023-10-16T13:47:40.918418+0000","last_fullsized":"2023-10-16T13:47:40.918418+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:59:08.104923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.d","version":"244'1","reported_seq":16,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.920060+0000","last_change":"2023-10-16T13:47:23.428449+0000","last_active":"2023-10-16T13:47:40.920060+0000","last_peered":"2023-10-16T13:47:40.920060+0000","last_clean":"2023-10-16T13:47:40.920060+0000","last_became_active":"2023-10-16T13:47:23.428130+0000","last_became_peered":"2023-10-16T13:47:23.428130+0000","last_unstale":"2023-10-16T13:47:40.920060+0000","last_undegraded":"2023-10-16T13:47:40.920060+0000","last_fullsized":"2023-10-16T13:47:40.920060+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:24:27.601821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.31","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.961246+0000","last_change":"2023-10-16T13:47:41.954571+0000","last_active":"2023-10-16T13:47:41.961246+0000","last_peered":"2023-10-16T13:47:41.961246+0000","last_clean":"2023-10-16T13:47:41.961246+0000","last_became_active":"2023-10-16T13:47:41.954283+0000","last_became_peered":"2023-10-16T13:47:41.954283+0000","last_unstale":"2023-10-16T13:47:41.961246+0000","last_undegraded":"2023-10-16T13:47:41.961246+0000","last_fullsized":"2023-10-16T13:47:41.961246+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:46:52.727625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.33","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.915842+0000","last_change":"2023-10-16T13:47:24.427239+0000","last_active":"2023-10-16T13:47:40.915842+0000","last_peered":"2023-10-16T13:47:40.915842+0000","last_clean":"2023-10-16T13:47:40.915842+0000","last_became_active":"2023-10-16T13:47:24.426919+0000","last_became_peered":"2023-10-16T13:47:24.426919+0000","last_unstale":"2023-10-16T13:47:40.915842+0000","last_undegraded":"2023-10-16T13:47:40.915842+0000","last_fullsized":"2023-10-16T13:47:40.915842+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:56:16.580840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.916522+0000","last_change":"2023-10-16T13:47:23.427051+0000","last_active":"2023-10-16T13:47:40.916522+0000","last_peered":"2023-10-16T13:47:40.916522+0000","last_clean":"2023-10-16T13:47:40.916522+0000","last_became_active":"2023-10-16T13:47:23.426741+0000","last_became_peered":"2023-10-16T13:47:23.426741+0000","last_unstale":"2023-10-16T13:47:40.916522+0000","last_undegraded":"2023-10-16T13:47:40.916522+0000","last_fullsized":"2023-10-16T13:47:40.916522+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:37:17.532164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.36","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.954049+0000","last_change":"2023-10-16T13:47:41.946845+0000","last_active":"2023-10-16T13:47:41.954049+0000","last_peered":"2023-10-16T13:47:41.954049+0000","last_clean":"2023-10-16T13:47:41.954049+0000","last_became_active":"2023-10-16T13:47:41.946499+0000","last_became_peered":"2023-10-16T13:47:41.946499+0000","last_unstale":"2023-10-16T13:47:41.954049+0000","last_undegraded":"2023-10-16T13:47:41.954049+0000","last_fullsized":"2023-10-16T13:47:41.954049+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:45:21.170147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.34","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.917342+0000","last_change":"2023-10-16T13:47:24.444364+0000","last_active":"2023-10-16T13:47:40.917342+0000","last_peered":"2023-10-16T13:47:40.917342+0000","last_clean":"2023-10-16T13:47:40.917342+0000","last_became_active":"2023-10-16T13:47:24.444045+0000","last_became_peered":"2023-10-16T13:47:24.444045+0000","last_unstale":"2023-10-16T13:47:40.917342+0000","last_undegraded":"2023-10-16T13:47:40.917342+0000","last_fullsized":"2023-10-16T13:47:40.917342+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:13:11.795921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.b","version":"244'2","reported_seq":17,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.915739+0000","last_change":"2023-10-16T13:47:23.434957+0000","last_active":"2023-10-16T13:47:40.915739+0000","last_peered":"2023-10-16T13:47:40.915739+0000","last_clean":"2023-10-16T13:47:40.915739+0000","last_became_active":"2023-10-16T13:47:23.434637+0000","last_became_peered":"2023-10-16T13:47:23.434637+0000","last_unstale":"2023-10-16T13:47:40.915739+0000","last_undegraded":"2023-10-16T13:47:40.915739+0000","last_fullsized":"2023-10-16T13:47:40.915739+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:23:47.081423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.37","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.950618+0000","last_change":"2023-10-16T13:47:41.946099+0000","last_active":"2023-10-16T13:47:41.950618+0000","last_peered":"2023-10-16T13:47:41.950618+0000","last_clean":"2023-10-16T13:47:41.950618+0000","last_became_active":"2023-10-16T13:47:41.945829+0000","last_became_peered":"2023-10-16T13:47:41.945829+0000","last_unstale":"2023-10-16T13:47:41.950618+0000","last_undegraded":"2023-10-16T13:47:41.950618+0000","last_fullsized":"2023-10-16T13:47:41.950618+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:33:03.023121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.35","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.918818+0000","last_change":"2023-10-16T13:47:24.431892+0000","last_active":"2023-10-16T13:47:40.918818+0000","last_peered":"2023-10-16T13:47:40.918818+0000","last_clean":"2023-10-16T13:47:40.918818+0000","last_became_active":"2023-10-16T13:47:24.431624+0000","last_became_peered":"2023-10-16T13:47:24.431624+0000","last_unstale":"2023-10-16T13:47:40.918818+0000","last_undegraded":"2023-10-16T13:47:40.918818+0000","last_fullsized":"2023-10-16T13:47:40.918818+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:49:13.422163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.911855+0000","last_change":"2023-10-16T13:47:23.435342+0000","last_active":"2023-10-16T13:47:40.911855+0000","last_peered":"2023-10-16T13:47:40.911855+0000","last_clean":"2023-10-16T13:47:40.911855+0000","last_became_active":"2023-10-16T13:47:23.434999+0000","last_became_peered":"2023-10-16T13:47:23.434999+0000","last_unstale":"2023-10-16T13:47:40.911855+0000","last_undegraded":"2023-10-16T13:47:40.911855+0000","last_fullsized":"2023-10-16T13:47:40.911855+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:20:03.252858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.34","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.942822+0000","last_change":"2023-10-16T13:47:41.941663+0000","last_active":"2023-10-16T13:47:41.942822+0000","last_peered":"2023-10-16T13:47:41.942822+0000","last_clean":"2023-10-16T13:47:41.942822+0000","last_became_active":"2023-10-16T13:47:41.941198+0000","last_became_peered":"2023-10-16T13:47:41.941198+0000","last_unstale":"2023-10-16T13:47:41.942822+0000","last_undegraded":"2023-10-16T13:47:41.942822+0000","last_fullsized":"2023-10-16T13:47:41.942822+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:28:20.482489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.36","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.921197+0000","last_change":"2023-10-16T13:47:24.435751+0000","last_active":"2023-10-16T13:47:40.921197+0000","last_peered":"2023-10-16T13:47:40.921197+0000","last_clean":"2023-10-16T13:47:40.921197+0000","last_became_active":"2023-10-16T13:47:24.434491+0000","last_became_peered":"2023-10-16T13:47:24.434491+0000","last_unstale":"2023-10-16T13:47:40.921197+0000","last_undegraded":"2023-10-16T13:47:40.921197+0000","last_fullsized":"2023-10-16T13:47:40.921197+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:02:56.979689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.910768+0000","last_change":"2023-10-16T13:47:23.427022+0000","last_active":"2023-10-16T13:47:40.910768+0000","last_peered":"2023-10-16T13:47:40.910768+0000","last_clean":"2023-10-16T13:47:40.910768+0000","last_became_active":"2023-10-16T13:47:23.426682+0000","last_became_peered":"2023-10-16T13:47:23.426682+0000","last_unstale":"2023-10-16T13:47:40.910768+0000","last_undegraded":"2023-10-16T13:47:40.910768+0000","last_fullsized":"2023-10-16T13:47:40.910768+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:05:08.936096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.35","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.952350+0000","last_change":"2023-10-16T13:47:41.945372+0000","last_active":"2023-10-16T13:47:41.952350+0000","last_peered":"2023-10-16T13:47:41.952350+0000","last_clean":"2023-10-16T13:47:41.952350+0000","last_became_active":"2023-10-16T13:47:41.945069+0000","last_became_peered":"2023-10-16T13:47:41.945069+0000","last_unstale":"2023-10-16T13:47:41.952350+0000","last_undegraded":"2023-10-16T13:47:41.952350+0000","last_fullsized":"2023-10-16T13:47:41.952350+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:40:35.374297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.37","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.920489+0000","last_change":"2023-10-16T13:47:24.436357+0000","last_active":"2023-10-16T13:47:40.920489+0000","last_peered":"2023-10-16T13:47:40.920489+0000","last_clean":"2023-10-16T13:47:40.920489+0000","last_became_active":"2023-10-16T13:47:24.436050+0000","last_became_peered":"2023-10-16T13:47:24.436050+0000","last_unstale":"2023-10-16T13:47:40.920489+0000","last_undegraded":"2023-10-16T13:47:40.920489+0000","last_fullsized":"2023-10-16T13:47:40.920489+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:15:49.933818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.901663+0000","last_change":"2023-10-16T13:47:23.433223+0000","last_active":"2023-10-16T13:47:40.901663+0000","last_peered":"2023-10-16T13:47:40.901663+0000","last_clean":"2023-10-16T13:47:40.901663+0000","last_became_active":"2023-10-16T13:47:23.432935+0000","last_became_peered":"2023-10-16T13:47:23.432935+0000","last_unstale":"2023-10-16T13:47:40.901663+0000","last_undegraded":"2023-10-16T13:47:40.901663+0000","last_fullsized":"2023-10-16T13:47:40.901663+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:40:59.034875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.3a","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.952689+0000","last_change":"2023-10-16T13:47:41.950978+0000","last_active":"2023-10-16T13:47:41.952689+0000","last_peered":"2023-10-16T13:47:41.952689+0000","last_clean":"2023-10-16T13:47:41.952689+0000","last_became_active":"2023-10-16T13:47:41.950480+0000","last_became_peered":"2023-10-16T13:47:41.950480+0000","last_unstale":"2023-10-16T13:47:41.952689+0000","last_undegraded":"2023-10-16T13:47:41.952689+0000","last_fullsized":"2023-10-16T13:47:41.952689+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:23:28.072263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.38","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.906642+0000","last_change":"2023-10-16T13:47:24.437795+0000","last_active":"2023-10-16T13:47:40.906642+0000","last_peered":"2023-10-16T13:47:40.906642+0000","last_clean":"2023-10-16T13:47:40.906642+0000","last_became_active":"2023-10-16T13:47:24.437493+0000","last_became_peered":"2023-10-16T13:47:24.437493+0000","last_unstale":"2023-10-16T13:47:40.906642+0000","last_undegraded":"2023-10-16T13:47:40.906642+0000","last_fullsized":"2023-10-16T13:47:40.906642+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:32:36.771595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.7","version":"244'2","reported_seq":17,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.916978+0000","last_change":"2023-10-16T13:47:23.430865+0000","last_active":"2023-10-16T13:47:40.916978+0000","last_peered":"2023-10-16T13:47:40.916978+0000","last_clean":"2023-10-16T13:47:40.916978+0000","last_became_active":"2023-10-16T13:47:23.430493+0000","last_became_peered":"2023-10-16T13:47:23.430493+0000","last_unstale":"2023-10-16T13:47:40.916978+0000","last_undegraded":"2023-10-16T13:47:40.916978+0000","last_fullsized":"2023-10-16T13:47:40.916978+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:59:14.057546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.3b","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.959084+0000","last_change":"2023-10-16T13:47:41.953921+0000","last_active":"2023-10-16T13:47:41.959084+0000","last_peered":"2023-10-16T13:47:41.959084+0000","last_clean":"2023-10-16T13:47:41.959084+0000","last_became_active":"2023-10-16T13:47:41.953626+0000","last_became_peered":"2023-10-16T13:47:41.953626+0000","last_unstale":"2023-10-16T13:47:41.959084+0000","last_undegraded":"2023-10-16T13:47:41.959084+0000","last_fullsized":"2023-10-16T13:47:41.959084+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:33:32.802746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.39","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.916543+0000","last_change":"2023-10-16T13:47:24.443917+0000","last_active":"2023-10-16T13:47:40.916543+0000","last_peered":"2023-10-16T13:47:40.916543+0000","last_clean":"2023-10-16T13:47:40.916543+0000","last_became_active":"2023-10-16T13:47:24.443547+0000","last_became_peered":"2023-10-16T13:47:24.443547+0000","last_unstale":"2023-10-16T13:47:40.916543+0000","last_undegraded":"2023-10-16T13:47:40.916543+0000","last_fullsized":"2023-10-16T13:47:40.916543+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:21:18.530078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.6","version":"245'1","reported_seq":16,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.908521+0000","last_change":"2023-10-16T13:47:23.430851+0000","last_active":"2023-10-16T13:47:40.908521+0000","last_peered":"2023-10-16T13:47:40.908521+0000","last_clean":"2023-10-16T13:47:40.908521+0000","last_became_active":"2023-10-16T13:47:23.430526+0000","last_became_peered":"2023-10-16T13:47:23.430526+0000","last_unstale":"2023-10-16T13:47:40.908521+0000","last_undegraded":"2023-10-16T13:47:40.908521+0000","last_fullsized":"2023-10-16T13:47:40.908521+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:40:16.878541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.38","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.960492+0000","last_change":"2023-10-16T13:47:41.956508+0000","last_active":"2023-10-16T13:47:41.960492+0000","last_peered":"2023-10-16T13:47:41.960492+0000","last_clean":"2023-10-16T13:47:41.960492+0000","last_became_active":"2023-10-16T13:47:41.956242+0000","last_became_peered":"2023-10-16T13:47:41.956242+0000","last_unstale":"2023-10-16T13:47:41.960492+0000","last_undegraded":"2023-10-16T13:47:41.960492+0000","last_fullsized":"2023-10-16T13:47:41.960492+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:10:25.581028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.3a","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.912818+0000","last_change":"2023-10-16T13:47:24.440347+0000","last_active":"2023-10-16T13:47:40.912818+0000","last_peered":"2023-10-16T13:47:40.912818+0000","last_clean":"2023-10-16T13:47:40.912818+0000","last_became_active":"2023-10-16T13:47:24.440041+0000","last_became_peered":"2023-10-16T13:47:24.440041+0000","last_unstale":"2023-10-16T13:47:40.912818+0000","last_undegraded":"2023-10-16T13:47:40.912818+0000","last_fullsized":"2023-10-16T13:47:40.912818+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:19:38.293171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.902848+0000","last_change":"2023-10-16T13:47:23.427399+0000","last_active":"2023-10-16T13:47:40.902848+0000","last_peered":"2023-10-16T13:47:40.902848+0000","last_clean":"2023-10-16T13:47:40.902848+0000","last_became_active":"2023-10-16T13:47:23.427092+0000","last_became_peered":"2023-10-16T13:47:23.427092+0000","last_unstale":"2023-10-16T13:47:40.902848+0000","last_undegraded":"2023-10-16T13:47:40.902848+0000","last_fullsized":"2023-10-16T13:47:40.902848+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:10:56.739817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.39","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.954251+0000","last_change":"2023-10-16T13:47:41.945354+0000","last_active":"2023-10-16T13:47:41.954251+0000","last_peered":"2023-10-16T13:47:41.954251+0000","last_clean":"2023-10-16T13:47:41.954251+0000","last_became_active":"2023-10-16T13:47:41.944660+0000","last_became_peered":"2023-10-16T13:47:41.944660+0000","last_unstale":"2023-10-16T13:47:41.954251+0000","last_undegraded":"2023-10-16T13:47:41.954251+0000","last_fullsized":"2023-10-16T13:47:41.954251+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:12:02.423405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.3b","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.908235+0000","last_change":"2023-10-16T13:47:24.436718+0000","last_active":"2023-10-16T13:47:40.908235+0000","last_peered":"2023-10-16T13:47:40.908235+0000","last_clean":"2023-10-16T13:47:40.908235+0000","last_became_active":"2023-10-16T13:47:24.436371+0000","last_became_peered":"2023-10-16T13:47:24.436371+0000","last_unstale":"2023-10-16T13:47:40.908235+0000","last_undegraded":"2023-10-16T13:47:40.908235+0000","last_fullsized":"2023-10-16T13:47:40.908235+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:58:12.134727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.903396+0000","last_change":"2023-10-16T13:47:23.433833+0000","last_active":"2023-10-16T13:47:40.903396+0000","last_peered":"2023-10-16T13:47:40.903396+0000","last_clean":"2023-10-16T13:47:40.903396+0000","last_became_active":"2023-10-16T13:47:23.433381+0000","last_became_peered":"2023-10-16T13:47:23.433381+0000","last_unstale":"2023-10-16T13:47:40.903396+0000","last_undegraded":"2023-10-16T13:47:40.903396+0000","last_fullsized":"2023-10-16T13:47:40.903396+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:09:34.543970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.3e","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.952818+0000","last_change":"2023-10-16T13:47:41.946300+0000","last_active":"2023-10-16T13:47:41.952818+0000","last_peered":"2023-10-16T13:47:41.952818+0000","last_clean":"2023-10-16T13:47:41.952818+0000","last_became_active":"2023-10-16T13:47:41.946000+0000","last_became_peered":"2023-10-16T13:47:41.946000+0000","last_unstale":"2023-10-16T13:47:41.952818+0000","last_undegraded":"2023-10-16T13:47:41.952818+0000","last_fullsized":"2023-10-16T13:47:41.952818+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:39:33.788061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3c","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.903058+0000","last_change":"2023-10-16T13:47:24.424889+0000","last_active":"2023-10-16T13:47:40.903058+0000","last_peered":"2023-10-16T13:47:40.903058+0000","last_clean":"2023-10-16T13:47:40.903058+0000","last_became_active":"2023-10-16T13:47:24.424573+0000","last_became_peered":"2023-10-16T13:47:24.424573+0000","last_unstale":"2023-10-16T13:47:40.903058+0000","last_undegraded":"2023-10-16T13:47:40.903058+0000","last_fullsized":"2023-10-16T13:47:40.903058+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:23:40.809515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.911573+0000","last_change":"2023-10-16T13:47:23.435920+0000","last_active":"2023-10-16T13:47:40.911573+0000","last_peered":"2023-10-16T13:47:40.911573+0000","last_clean":"2023-10-16T13:47:40.911573+0000","last_became_active":"2023-10-16T13:47:23.435675+0000","last_became_peered":"2023-10-16T13:47:23.435675+0000","last_unstale":"2023-10-16T13:47:40.911573+0000","last_undegraded":"2023-10-16T13:47:40.911573+0000","last_fullsized":"2023-10-16T13:47:40.911573+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:01:54.403340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.3f","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.952423+0000","last_change":"2023-10-16T13:47:41.945288+0000","last_active":"2023-10-16T13:47:41.952423+0000","last_peered":"2023-10-16T13:47:41.952423+0000","last_clean":"2023-10-16T13:47:41.952423+0000","last_became_active":"2023-10-16T13:47:41.942669+0000","last_became_peered":"2023-10-16T13:47:41.942669+0000","last_unstale":"2023-10-16T13:47:41.952423+0000","last_undegraded":"2023-10-16T13:47:41.952423+0000","last_fullsized":"2023-10-16T13:47:41.952423+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:55:04.579725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.3d","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.907802+0000","last_change":"2023-10-16T13:47:24.430727+0000","last_active":"2023-10-16T13:47:40.907802+0000","last_peered":"2023-10-16T13:47:40.907802+0000","last_clean":"2023-10-16T13:47:40.907802+0000","last_became_active":"2023-10-16T13:47:24.430424+0000","last_became_peered":"2023-10-16T13:47:24.430424+0000","last_unstale":"2023-10-16T13:47:40.907802+0000","last_undegraded":"2023-10-16T13:47:40.907802+0000","last_fullsized":"2023-10-16T13:47:40.907802+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:05:49.279256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.906650+0000","last_change":"2023-10-16T13:47:23.430779+0000","last_active":"2023-10-16T13:47:40.906650+0000","last_peered":"2023-10-16T13:47:40.906650+0000","last_clean":"2023-10-16T13:47:40.906650+0000","last_became_active":"2023-10-16T13:47:23.429996+0000","last_became_peered":"2023-10-16T13:47:23.429996+0000","last_unstale":"2023-10-16T13:47:40.906650+0000","last_undegraded":"2023-10-16T13:47:40.906650+0000","last_fullsized":"2023-10-16T13:47:40.906650+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:17:22.079776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.3c","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.955157+0000","last_change":"2023-10-16T13:47:41.952726+0000","last_active":"2023-10-16T13:47:41.955157+0000","last_peered":"2023-10-16T13:47:41.955157+0000","last_clean":"2023-10-16T13:47:41.955157+0000","last_became_active":"2023-10-16T13:47:41.952452+0000","last_became_peered":"2023-10-16T13:47:41.952452+0000","last_unstale":"2023-10-16T13:47:41.955157+0000","last_undegraded":"2023-10-16T13:47:41.955157+0000","last_fullsized":"2023-10-16T13:47:41.955157+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:13:48.625511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.3e","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.919255+0000","last_change":"2023-10-16T13:47:24.436378+0000","last_active":"2023-10-16T13:47:40.919255+0000","last_peered":"2023-10-16T13:47:40.919255+0000","last_clean":"2023-10-16T13:47:40.919255+0000","last_became_active":"2023-10-16T13:47:24.435495+0000","last_became_peered":"2023-10-16T13:47:24.435495+0000","last_unstale":"2023-10-16T13:47:40.919255+0000","last_undegraded":"2023-10-16T13:47:40.919255+0000","last_fullsized":"2023-10-16T13:47:40.919255+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:18:23.073022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.912463+0000","last_change":"2023-10-16T13:47:23.433945+0000","last_active":"2023-10-16T13:47:40.912463+0000","last_peered":"2023-10-16T13:47:40.912463+0000","last_clean":"2023-10-16T13:47:40.912463+0000","last_became_active":"2023-10-16T13:47:23.433576+0000","last_became_peered":"2023-10-16T13:47:23.433576+0000","last_unstale":"2023-10-16T13:47:40.912463+0000","last_undegraded":"2023-10-16T13:47:40.912463+0000","last_fullsized":"2023-10-16T13:47:40.912463+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:09:34.543970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.3d","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.955058+0000","last_change":"2023-10-16T13:47:41.947937+0000","last_active":"2023-10-16T13:47:41.955058+0000","last_peered":"2023-10-16T13:47:41.955058+0000","last_clean":"2023-10-16T13:47:41.955058+0000","last_became_active":"2023-10-16T13:47:41.947634+0000","last_became_peered":"2023-10-16T13:47:41.947634+0000","last_unstale":"2023-10-16T13:47:41.955058+0000","last_undegraded":"2023-10-16T13:47:41.955058+0000","last_fullsized":"2023-10-16T13:47:41.955058+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:21:36.021473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.3f","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.920581+0000","last_change":"2023-10-16T13:47:24.441874+0000","last_active":"2023-10-16T13:47:40.920581+0000","last_peered":"2023-10-16T13:47:40.920581+0000","last_clean":"2023-10-16T13:47:40.920581+0000","last_became_active":"2023-10-16T13:47:24.441466+0000","last_became_peered":"2023-10-16T13:47:24.441466+0000","last_unstale":"2023-10-16T13:47:40.920581+0000","last_undegraded":"2023-10-16T13:47:40.920581+0000","last_fullsized":"2023-10-16T13:47:40.920581+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:22:04.707731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.912926+0000","last_change":"2023-10-16T13:47:23.423270+0000","last_active":"2023-10-16T13:47:40.912926+0000","last_peered":"2023-10-16T13:47:40.912926+0000","last_clean":"2023-10-16T13:47:40.912926+0000","last_became_active":"2023-10-16T13:47:23.422889+0000","last_became_peered":"2023-10-16T13:47:23.422889+0000","last_unstale":"2023-10-16T13:47:40.912926+0000","last_undegraded":"2023-10-16T13:47:40.912926+0000","last_fullsized":"2023-10-16T13:47:40.912926+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:30:37.923537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.23","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.955194+0000","last_change":"2023-10-16T13:47:41.948103+0000","last_active":"2023-10-16T13:47:41.955194+0000","last_peered":"2023-10-16T13:47:41.955194+0000","last_clean":"2023-10-16T13:47:41.955194+0000","last_became_active":"2023-10-16T13:47:41.947098+0000","last_became_peered":"2023-10-16T13:47:41.947098+0000","last_unstale":"2023-10-16T13:47:41.955194+0000","last_undegraded":"2023-10-16T13:47:41.955194+0000","last_fullsized":"2023-10-16T13:47:41.955194+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:05:43.949107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.21","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.906700+0000","last_change":"2023-10-16T13:47:24.431657+0000","last_active":"2023-10-16T13:47:40.906700+0000","last_peered":"2023-10-16T13:47:40.906700+0000","last_clean":"2023-10-16T13:47:40.906700+0000","last_became_active":"2023-10-16T13:47:24.431385+0000","last_became_peered":"2023-10-16T13:47:24.431385+0000","last_unstale":"2023-10-16T13:47:40.906700+0000","last_undegraded":"2023-10-16T13:47:40.906700+0000","last_fullsized":"2023-10-16T13:47:40.906700+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:59:09.754185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1e","version":"244'1","reported_seq":16,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.909995+0000","last_change":"2023-10-16T13:47:23.429601+0000","last_active":"2023-10-16T13:47:40.909995+0000","last_peered":"2023-10-16T13:47:40.909995+0000","last_clean":"2023-10-16T13:47:40.909995+0000","last_became_active":"2023-10-16T13:47:23.429298+0000","last_became_peered":"2023-10-16T13:47:23.429298+0000","last_unstale":"2023-10-16T13:47:40.909995+0000","last_undegraded":"2023-10-16T13:47:40.909995+0000","last_fullsized":"2023-10-16T13:47:40.909995+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:20:00.827725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"231'370","reported_seq":1301,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.916219+0000","last_change":"2023-10-16T12:10:34.746820+0000","last_active":"2023-10-16T13:47:40.916219+0000","last_peered":"2023-10-16T13:47:40.916219+0000","last_clean":"2023-10-16T13:47:40.916219+0000","last_became_active":"2023-10-16T12:10:34.745574+0000","last_became_peered":"2023-10-16T12:10:34.745574+0000","last_unstale":"2023-10-16T13:47:40.916219+0000","last_undegraded":"2023-10-16T13:47:40.916219+0000","last_fullsized":"2023-10-16T13:47:40.916219+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_clean_scrub_stamp":"2023-10-16T12:10:33.577111+0000","objects_scrubbed":0,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:14.461986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":34,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":33,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":32,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":31,"num_pg":32,"stat_sum":{"num_bytes":3657,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":370,"ondisk_log_size":370,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575999,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57040,"kb_used_data":3648,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314800,"statfs":{"total":96636764160,"available":96578355200,"internally_reserved":0,"allocated":3735552,"data_stored":2210331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54655435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:46:50 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.60299999999999998,"15min":0.63200000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.33000000000000002},"max":{"1min":0.78600000000000003,"5min":0.87,"15min":0.93799999999999994},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.60699999999999998,"15min":0.626},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.72099999999999997,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.45700000000000002}]},{"osd":1,"last update":"Mon Oct 16 13:46:50 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.61299999999999999,"15min":0.63900000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.79000000000000004,"5min":0.84599999999999997,"15min":0.95899999999999996},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.622,"15min":0.63100000000000001},"min":{"1min":0.38200000000000001,"5min":0.375,"15min":0.375},"max":{"1min":0.76200000000000001,"5min":0.79100000000000004,"15min":0.86399999999999999},"last":0.39800000000000002}]},{"osd":2,"last update":"Mon Oct 16 13:46:50 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.67400000000000004,"15min":0.68600000000000005},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.80300000000000005,"5min":0.83899999999999997,"15min":0.88200000000000001},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.68899999999999995,"15min":0.69799999999999995},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.39400000000000002},"max":{"1min":0.83699999999999997,"5min":0.92800000000000005,"15min":0.93799999999999994},"last":0.433}]},{"osd":3,"last update":"Mon Oct 16 13:46:50 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.63800000000000001,"15min":0.65900000000000003},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.85299999999999998,"5min":0.89200000000000002,"15min":0.91300000000000003},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.65000000000000002,"15min":0.66000000000000003},"min":{"1min":0.435,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.88,"5min":1.24,"15min":1.24},"last":0.51000000000000001}]},{"osd":4,"last update":"Mon Oct 16 13:46:50 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.52000000000000002,"15min":0.51900000000000002},"min":{"1min":0.311,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.71099999999999997,"5min":0.745,"15min":0.85299999999999998},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.621,"15min":0.61799999999999999},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.33100000000000002},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.64200000000000002}]},{"osd":5,"last update":"Mon Oct 16 13:46:50 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.54900000000000004,"15min":0.54700000000000004},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.85699999999999998,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.58399999999999996,"15min":0.58699999999999997},"min":{"1min":0.41099999999999998,"5min":0.38400000000000001,"15min":0.35099999999999998},"max":{"1min":0.81100000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.63100000000000001}]},{"osd":6,"last update":"Mon Oct 16 13:46:50 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.60499999999999998,"15min":0.61299999999999999},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.85499999999999998,"5min":0.874,"15min":0.95899999999999996},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.626,"15min":0.61899999999999999},"min":{"1min":0.39400000000000002,"5min":0.378,"15min":0.35199999999999998},"max":{"1min":0.92100000000000004,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.67000000000000004}]}]},{"osd":6,"up_from":13,"seq":55834575997,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60600,"kb_used_data":3048,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311240,"statfs":{"total":96636764160,"available":96574709760,"internally_reserved":0,"allocated":3121152,"data_stored":1611254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58915271},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:47:39 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.64100000000000001,"15min":0.66300000000000003},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.29599999999999999},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.63700000000000001,"15min":0.65800000000000003},"min":{"1min":0.36499999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.81100000000000005,"5min":0.91400000000000003,"15min":0.93799999999999994},"last":0.373}]},{"osd":1,"last update":"Mon Oct 16 13:47:39 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.57099999999999995,"15min":0.61799999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.76600000000000001,"5min":0.86899999999999999,"15min":0.90900000000000003},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.61599999999999999,"15min":0.61699999999999999},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.51500000000000001}]},{"osd":2,"last update":"Mon Oct 16 13:47:39 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.60799999999999998,"15min":0.627},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.79600000000000004,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.59699999999999998,"15min":0.61799999999999999},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.2769999999999999,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.56000000000000005}]},{"osd":3,"last update":"Mon Oct 16 13:47:39 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.625,"15min":0.65600000000000003},"min":{"1min":0.40100000000000002,"5min":0.39900000000000002,"15min":0.38200000000000001},"max":{"1min":0.75600000000000001,"5min":0.92400000000000004,"15min":0.95599999999999996},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.68400000000000005,"15min":0.71199999999999997},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.42299999999999999},"max":{"1min":0.85199999999999998,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.63200000000000001}]},{"osd":4,"last update":"Mon Oct 16 13:47:39 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.52500000000000002,"15min":0.53000000000000003},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.748,"5min":0.85599999999999998,"15min":0.95799999999999996},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.61199999999999999,"15min":0.621},"min":{"1min":0.38400000000000001,"5min":0.35499999999999998,"15min":0.32900000000000001},"max":{"1min":0.79100000000000004,"5min":0.98799999999999999,"15min":1.0309999999999999},"last":0.61699999999999999}]},{"osd":5,"last update":"Mon Oct 16 13:47:39 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.58799999999999997,"15min":0.59499999999999997},"min":{"1min":0.34399999999999997,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.80000000000000004,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.64000000000000001,"15min":0.65900000000000003},"min":{"1min":0.36099999999999999,"5min":0.33700000000000002,"15min":0.308},"max":{"1min":0.86399999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.68200000000000005}]},{"osd":7,"last update":"Mon Oct 16 13:47:39 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.58899999999999997,"15min":0.60299999999999998},"min":{"1min":0.38700000000000001,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.79300000000000004,"5min":1.081,"15min":1.081},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.63,"5min":0.64100000000000001,"15min":0.65400000000000003},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.375},"max":{"1min":0.83499999999999996,"5min":1.099,"15min":1.099},"last":0.64300000000000002}]}]},{"osd":1,"up_from":13,"seq":55834575997,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52288,"kb_used_data":3060,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319552,"statfs":{"total":96636764160,"available":96583221248,"internally_reserved":0,"allocated":3133440,"data_stored":1619359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":50395593},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:46:55 2023","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.39500000000000002,"15min":0.40699999999999997},"min":{"1min":0.27600000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.51300000000000001,"5min":0.65400000000000003,"15min":0.65800000000000003},"last":0.23000000000000001},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.47599999999999998,"15min":0.49199999999999999},"min":{"1min":0.32000000000000001,"5min":0.253,"15min":0.253},"max":{"1min":0.56000000000000005,"5min":0.70999999999999996,"15min":0.77200000000000002},"last":0.29099999999999998}]},{"osd":2,"last update":"Mon Oct 16 13:46:55 2023","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.436,"15min":0.45100000000000001},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.58599999999999997,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.48099999999999998,"15min":0.48999999999999999},"min":{"1min":0.34399999999999997,"5min":0.315,"15min":0.30199999999999999},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.71399999999999997},"last":0.308}]},{"osd":3,"last update":"Mon Oct 16 13:46:55 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.42899999999999999,"15min":0.42799999999999999},"min":{"1min":0.307,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.50800000000000001,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.254},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.46100000000000002,"15min":0.47499999999999998},"min":{"1min":0.29799999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.52800000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.34899999999999998}]},{"osd":4,"last update":"Mon Oct 16 13:46:55 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.64400000000000002,"15min":0.66200000000000003},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.42399999999999999},"max":{"1min":0.78600000000000003,"5min":0.95899999999999996,"15min":1.0069999999999999},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.64200000000000002,"15min":0.66600000000000004},"min":{"1min":0.42899999999999999,"5min":0.37,"15min":0.37},"max":{"1min":0.69999999999999996,"5min":0.91300000000000003,"15min":0.997},"last":0.69399999999999995}]},{"osd":5,"last update":"Mon Oct 16 13:46:55 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.68200000000000005,"15min":0.70699999999999996},"min":{"1min":0.45800000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.83999999999999997,"5min":0.97499999999999998,"15min":1.0760000000000001},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.70499999999999996,"15min":0.72599999999999998},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.81399999999999995,"5min":1.0469999999999999,"15min":1.0660000000000001},"last":0.58499999999999996}]},{"osd":6,"last update":"Mon Oct 16 13:46:55 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.67500000000000004,"15min":0.68000000000000005},"min":{"1min":0.46200000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.75900000000000001,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.623},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.68700000000000006,"15min":0.70499999999999996},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.80400000000000005,"5min":1.03,"15min":1.083},"last":0.54400000000000004}]},{"osd":7,"last update":"Mon Oct 16 13:46:55 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.70899999999999996,"15min":0.73399999999999999},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.77300000000000002,"5min":1.0629999999999999,"15min":1.0660000000000001},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.72899999999999998,"15min":0.746},"min":{"1min":0.496,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.82099999999999995,"5min":1.042,"15min":1.0580000000000001},"last":0.67000000000000004}]}]},{"osd":0,"up_from":13,"seq":55834575998,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48724,"kb_used_data":3648,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323116,"statfs":{"total":96636764160,"available":96586870784,"internally_reserved":0,"allocated":3735552,"data_stored":2212532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 13:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.40400000000000003,"15min":0.42099999999999999},"min":{"1min":0.29099999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.60699999999999998,"5min":0.65800000000000003,"15min":0.71399999999999997},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.42199999999999999,"15min":0.439},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.82199999999999995},"last":0.32800000000000001}]},{"osd":2,"last update":"Mon Oct 16 13:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.443,"15min":0.45400000000000001},"min":{"1min":0.28599999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.61399999999999999,"5min":0.65900000000000003,"15min":0.75700000000000001},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.46400000000000002,"15min":0.48499999999999999},"min":{"1min":0.29799999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.83299999999999996},"last":0.39300000000000002}]},{"osd":3,"last update":"Mon Oct 16 13:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.42699999999999999,"15min":0.45300000000000001},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.56499999999999995,"5min":0.63400000000000001,"15min":0.78600000000000003},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.50900000000000001,"15min":0.52000000000000002},"min":{"1min":0.317,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.66400000000000003,"5min":0.82999999999999996,"15min":0.86399999999999999},"last":0.309}]},{"osd":4,"last update":"Mon Oct 16 13:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.66400000000000003,"15min":0.68400000000000005},"min":{"1min":0.42399999999999999,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.66500000000000004,"15min":0.68899999999999995},"min":{"1min":0.499,"5min":0.375,"15min":0.375},"max":{"1min":1.21,"5min":1.21,"15min":1.21},"last":0.74399999999999999}]},{"osd":5,"last update":"Mon Oct 16 13:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.66400000000000003,"15min":0.68999999999999995},"min":{"1min":0.41999999999999998,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.242,"5min":1.242,"15min":1.242},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.70599999999999996,"15min":0.72299999999999998},"min":{"1min":0.497,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.998,"5min":1.081,"15min":1.081},"last":0.76000000000000001}]},{"osd":6,"last update":"Mon Oct 16 13:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.71199999999999997,"15min":0.72899999999999998},"min":{"1min":0.51900000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.96599999999999997,"5min":1.034,"15min":1.103},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.70399999999999996,"15min":0.72299999999999998},"min":{"1min":0.51000000000000001,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":0.77400000000000002}]},{"osd":7,"last update":"Mon Oct 16 13:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.71199999999999997,"15min":0.74199999999999999},"min":{"1min":0.48399999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.1919999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.72499999999999998,"15min":0.73499999999999999},"min":{"1min":0.51300000000000001,"5min":0.433,"15min":0.433},"max":{"1min":1.0469999999999999,"5min":1.087,"15min":1.087},"last":0.66200000000000003}]}]},{"osd":2,"up_from":13,"seq":55834575998,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52312,"kb_used_data":3080,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319528,"statfs":{"total":96636764160,"available":96583196672,"internally_reserved":0,"allocated":3153920,"data_stored":1622103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.39800000000000002,"15min":0.40699999999999997},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.54100000000000004,"5min":0.71699999999999997,"15min":0.77500000000000002},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.45800000000000002,"15min":0.47599999999999998},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.69799999999999995,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.38200000000000001}]},{"osd":1,"last update":"Mon Oct 16 13:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.47899999999999998,"15min":0.495},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.61899999999999999,"5min":0.73199999999999998,"15min":0.81399999999999995},"last":0.433},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.47099999999999997,"15min":0.47299999999999998},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.59799999999999998,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.40600000000000003}]},{"osd":3,"last update":"Mon Oct 16 13:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.432,"15min":0.438},"min":{"1min":0.26300000000000001,"5min":0.255,"15min":0.255},"max":{"1min":0.63,"5min":0.70099999999999996,"15min":0.72599999999999998},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.52400000000000002,"15min":0.54100000000000004},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.74099999999999999,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.27000000000000002}]},{"osd":4,"last update":"Mon Oct 16 13:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.69299999999999995,"15min":0.71199999999999997},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.94399999999999995,"5min":1.3600000000000001,"15min":1.3600000000000001},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.70399999999999996,"15min":0.71599999999999997},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.196,"5min":1.403,"15min":1.403},"last":0.49399999999999999}]},{"osd":5,"last update":"Mon Oct 16 13:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.65900000000000003,"15min":0.68400000000000005},"min":{"1min":0.46700000000000003,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.82999999999999996,"5min":1.486,"15min":1.486},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.68200000000000005,"15min":0.69599999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.879,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":0.50900000000000001}]},{"osd":6,"last update":"Mon Oct 16 13:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.65800000000000003,"15min":0.66500000000000004},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.087,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.69799999999999995,"15min":0.71799999999999997},"min":{"1min":0.46000000000000002,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.94299999999999995,"5min":1.4379999999999999,"15min":1.4379999999999999},"last":0.52600000000000002}]},{"osd":7,"last update":"Mon Oct 16 13:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.70499999999999996,"15min":0.71499999999999997},"min":{"1min":0.42599999999999999,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.97099999999999997,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.70299999999999996,"15min":0.72899999999999998},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.0429999999999999,"5min":1.377,"15min":1.377},"last":0.56499999999999995}]}]},{"osd":3,"up_from":13,"seq":55834575997,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56472,"kb_used_data":3080,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315368,"statfs":{"total":96636764160,"available":96578936832,"internally_reserved":0,"allocated":3153920,"data_stored":1622822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:47:42 2023","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.39300000000000002,"15min":0.40699999999999997},"min":{"1min":0.27400000000000002,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.60099999999999998,"5min":0.64200000000000002,"15min":0.72899999999999998},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.40000000000000002,"15min":0.42099999999999999},"min":{"1min":0.27000000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.47899999999999998,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.34499999999999997}]},{"osd":1,"last update":"Mon Oct 16 13:47:42 2023","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.40799999999999997,"15min":0.42699999999999999},"min":{"1min":0.27100000000000002,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.46200000000000002,"5min":0.68200000000000005,"15min":0.76300000000000001},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.44600000000000001,"15min":0.47499999999999998},"min":{"1min":0.309,"5min":0.253,"15min":0.253},"max":{"1min":0.65400000000000003,"5min":0.70699999999999996,"15min":0.77800000000000002},"last":0.36899999999999999}]},{"osd":2,"last update":"Mon Oct 16 13:47:42 2023","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.46600000000000003,"15min":0.48499999999999999},"min":{"1min":0.317,"5min":0.316,"15min":0.316},"max":{"1min":0.64700000000000002,"5min":1.167,"15min":1.167},"last":0.317},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.48399999999999999,"15min":0.51400000000000001},"min":{"1min":0.32400000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.58099999999999996,"5min":0.73299999999999998,"15min":0.78700000000000003},"last":0.39700000000000002}]},{"osd":4,"last update":"Mon Oct 16 13:47:42 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.626,"15min":0.65800000000000003},"min":{"1min":0.46000000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.69399999999999995,"5min":0.998,"15min":1.137},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.66600000000000004,"15min":0.68899999999999995},"min":{"1min":0.44600000000000001,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.86299999999999999,"5min":1.2110000000000001,"15min":1.2909999999999999},"last":0.45300000000000001}]},{"osd":5,"last update":"Mon Oct 16 13:47:42 2023","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.65700000000000003,"15min":0.69999999999999996},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.374},"max":{"1min":0.85099999999999998,"5min":1.0680000000000001,"15min":1.2729999999999999},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.66900000000000004,"15min":0.69399999999999995},"min":{"1min":0.48799999999999999,"5min":0.45600000000000002,"15min":0.432},"max":{"1min":0.77400000000000002,"5min":1.0549999999999999,"15min":1.173},"last":0.58299999999999996}]},{"osd":6,"last update":"Mon Oct 16 13:47:42 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.70999999999999996,"15min":0.71399999999999997},"min":{"1min":0.434,"5min":0.42999999999999999,"15min":0.41799999999999998},"max":{"1min":0.80900000000000005,"5min":5.0999999999999996,"15min":5.0999999999999996},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.69299999999999995,"15min":0.72699999999999998},"min":{"1min":0.47399999999999998,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.89100000000000001,"5min":1.0309999999999999,"15min":1.2290000000000001},"last":0.48499999999999999}]},{"osd":7,"last update":"Mon Oct 16 13:47:42 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.69599999999999995,"15min":0.72899999999999998},"min":{"1min":0.497,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.82799999999999996,"5min":1.048,"15min":1.2569999999999999},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.62,"5min":0.70299999999999996,"15min":0.73599999999999999},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.43099999999999999},"max":{"1min":0.90400000000000003,"5min":1.262,"15min":1.262},"last":0.47299999999999998}]}]},{"osd":4,"up_from":13,"seq":55834575996,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52300,"kb_used_data":3068,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319540,"statfs":{"total":96636764160,"available":96583208960,"internally_reserved":0,"allocated":3141632,"data_stored":1620917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:47:41 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.57999999999999996,"15min":0.623},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.81599999999999995,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.60099999999999998,"15min":0.63},"min":{"1min":0.39300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.77300000000000002,"5min":0.878,"15min":0.89500000000000002},"last":0.61099999999999999}]},{"osd":1,"last update":"Mon Oct 16 13:47:41 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.60799999999999998,"15min":0.65200000000000002},"min":{"1min":0.33700000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.78900000000000003,"5min":0.877,"15min":0.877},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.60899999999999999,"15min":0.63700000000000001},"min":{"1min":0.42199999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.80800000000000005,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.64200000000000002}]},{"osd":2,"last update":"Mon Oct 16 13:47:41 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.623,"15min":0.64900000000000002},"min":{"1min":0.39000000000000001,"5min":0.378,"15min":0.378},"max":{"1min":0.69399999999999995,"5min":0.89900000000000002,"15min":0.90600000000000003},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.64200000000000002,"15min":0.65900000000000003},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.77500000000000002,"5min":0.89700000000000002,"15min":0.92600000000000005},"last":0.71999999999999997}]},{"osd":3,"last update":"Mon Oct 16 13:47:41 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.61699999999999999,"15min":0.64300000000000002},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.76000000000000001,"5min":0.86599999999999999,"15min":0.99199999999999999},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.65300000000000002,"15min":0.68000000000000005},"min":{"1min":0.45600000000000002,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.93899999999999995},"last":0.54800000000000004}]},{"osd":5,"last update":"Mon Oct 16 13:47:41 2023","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.54400000000000004,"15min":0.53300000000000003},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.73799999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.58099999999999996,"15min":0.56799999999999995},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.753,"5min":0.82199999999999995,"15min":0.94899999999999995},"last":0.57499999999999996}]},{"osd":6,"last update":"Mon Oct 16 13:47:41 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.627,"15min":0.60099999999999998},"min":{"1min":0.378,"5min":0.36399999999999999,"15min":0.33200000000000002},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.627},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.70999999999999996,"15min":0.69499999999999995},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.39400000000000002},"max":{"1min":0.84299999999999997,"5min":1.129,"15min":1.129},"last":0.746}]},{"osd":7,"last update":"Mon Oct 16 13:47:41 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.65800000000000003,"15min":0.63500000000000001},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.38900000000000001},"max":{"1min":0.754,"5min":0.97499999999999998,"15min":0.98599999999999999},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.71299999999999997,"15min":0.68700000000000006},"min":{"1min":0.48699999999999999,"5min":0.44,"15min":0.38200000000000001},"max":{"1min":0.84699999999999998,"5min":0.97599999999999998,"15min":1.026},"last":0.66500000000000004}]}]},{"osd":5,"up_from":13,"seq":55834575997,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70332,"kb_used_data":3056,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94301508,"statfs":{"total":96636764160,"available":96564744192,"internally_reserved":0,"allocated":3129344,"data_stored":1619305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:47:28 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.59599999999999997,"15min":0.61399999999999999},"min":{"1min":0.379,"5min":0.314,"15min":0.26800000000000002},"max":{"1min":0.79200000000000004,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.61799999999999999,"15min":0.63700000000000001},"min":{"1min":0.38600000000000001,"5min":0.35899999999999999,"15min":0.29699999999999999},"max":{"1min":0.78600000000000003,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.57099999999999995}]},{"osd":1,"last update":"Mon Oct 16 13:47:28 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.64700000000000002,"15min":0.65900000000000003},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.29499999999999998},"max":{"1min":0.82799999999999996,"5min":1,"15min":1},"last":0.504},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.61299999999999999,"15min":0.63600000000000001},"min":{"1min":0.378,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.82099999999999995,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.59599999999999997}]},{"osd":2,"last update":"Mon Oct 16 13:47:28 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.65400000000000003,"15min":0.65500000000000003},"min":{"1min":0.48699999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.80500000000000005,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.63400000000000001,"15min":0.64700000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.80200000000000005,"5min":0.871,"15min":0.93500000000000005},"last":0.55400000000000005}]},{"osd":3,"last update":"Mon Oct 16 13:47:28 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.66200000000000003,"15min":0.66500000000000004},"min":{"1min":0.42199999999999999,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.85799999999999998,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.629},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.68999999999999995,"15min":0.69399999999999995},"min":{"1min":0.45500000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.82799999999999996,"5min":1.4199999999999999,"15min":1.4199999999999999},"last":0.46899999999999997}]},{"osd":4,"last update":"Mon Oct 16 13:47:28 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.54800000000000004,"15min":0.53800000000000003},"min":{"1min":0.39500000000000002,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.89700000000000002,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.60799999999999998,"15min":0.59499999999999997},"min":{"1min":0.40999999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.67200000000000004}]},{"osd":6,"last update":"Mon Oct 16 13:47:28 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.60099999999999998,"15min":0.58799999999999997},"min":{"1min":0.443,"5min":0.315,"15min":0.315},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.93200000000000005},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.629,"15min":0.625},"min":{"1min":0.501,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.49099999999999999}]},{"osd":7,"last update":"Mon Oct 16 13:47:28 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.65500000000000003,"15min":0.63900000000000001},"min":{"1min":0.47599999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.93899999999999995,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.70699999999999996,"15min":0.68899999999999995},"min":{"1min":0.53400000000000003,"5min":0.375,"15min":0.375},"max":{"1min":0.97299999999999998,"5min":0.98299999999999998,"15min":1.0229999999999999},"last":0.64600000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T13:47:46.121 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T13:47:46.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.273+0000 7f5b29039700 1 Processor -- start 2023-10-16T13:47:46.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.273+0000 7f5b29039700 1 -- start start 2023-10-16T13:47:46.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.274+0000 7f5b29039700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f5b2406aef0 0x7f5b240b3480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:46.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.274+0000 7f5b29039700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b240d11c0 con 0x7f5b2415b860 2023-10-16T13:47:46.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.274+0000 7f5b29039700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b240cd360 con 0x7f5b240b2be0 2023-10-16T13:47:46.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.274+0000 7f5b29039700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b240d03d0 con 0x7f5b2406aef0 2023-10-16T13:47:46.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.274+0000 7f5b22d9d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f5b2415b860 0x7f5b2406a7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:45172/0 (socket says 172.21.15.16:45172) 2023-10-16T13:47:46.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.274+0000 7f5b22d9d700 1 -- 172.21.15.16:0/1161295295 learned_addr learned my addr 172.21.15.16:0/1161295295 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:47:46.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.274+0000 7f5b2259c700 1 --2- 172.21.15.16:0/1161295295 >> v2:172.21.15.49:3300/0 conn(0x7f5b2406aef0 0x7f5b240b3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:46.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.274+0000 7f5b21d9b700 1 -- 172.21.15.16:0/1161295295 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 241293741 0 0) 0x7f5b240d11c0 con 0x7f5b2415b860 2023-10-16T13:47:46.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.274+0000 7f5b21d9b700 1 -- 172.21.15.16:0/1161295295 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b0c0035b0 con 0x7f5b2415b860 2023-10-16T13:47:46.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.274+0000 7f5b21d9b700 1 -- 172.21.15.16:0/1161295295 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 424289047 0 0) 0x7f5b0c0035b0 con 0x7f5b2415b860 2023-10-16T13:47:46.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.274+0000 7f5b21d9b700 1 -- 172.21.15.16:0/1161295295 >> v2:172.21.15.49:3300/0 conn(0x7f5b2406aef0 msgr2=0x7f5b240b3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:46.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.274+0000 7f5b21d9b700 1 --2- 172.21.15.16:0/1161295295 >> v2:172.21.15.49:3300/0 conn(0x7f5b2406aef0 0x7f5b240b3480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:46.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.274+0000 7f5b21d9b700 1 -- 172.21.15.16:0/1161295295 >> v1:172.21.15.49:6789/0 conn(0x7f5b240b2be0 legacy=0x7f5b240b39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:46.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.274+0000 7f5b21d9b700 1 -- 172.21.15.16:0/1161295295 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b240d05b0 con 0x7f5b2415b860 2023-10-16T13:47:46.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.274+0000 7f5b21d9b700 1 -- 172.21.15.16:0/1161295295 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f5b14001670 con 0x7f5b2415b860 2023-10-16T13:47:46.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.274+0000 7f5b21d9b700 1 -- 172.21.15.16:0/1161295295 >> v1:172.21.15.16:6789/0 conn(0x7f5b2415b860 legacy=0x7f5b2406a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:46.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.274+0000 7f5b21d9b700 1 --2- 172.21.15.16:0/1161295295 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5b0c003a00 0x7f5b0c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:46.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.274+0000 7f5b21d9b700 1 --2- 172.21.15.16:0/1161295295 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b0c004850 0x7f5b0c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:46.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.274+0000 7f5b21d9b700 1 --2- 172.21.15.16:0/1161295295 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5b0c005210 0x7f5b0c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:46.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.274+0000 7f5b21d9b700 1 -- 172.21.15.16:0/1161295295 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b240d03d0 con 0x7f5b0c004850 2023-10-16T13:47:46.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.274+0000 7f5b21d9b700 1 -- 172.21.15.16:0/1161295295 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b0c00c0b0 con 0x7f5b0c003a00 2023-10-16T13:47:46.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.274+0000 7f5b21d9b700 1 -- 172.21.15.16:0/1161295295 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b0c00c270 con 0x7f5b0c005210 2023-10-16T13:47:46.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.275+0000 7f5b2259c700 1 --2- 172.21.15.16:0/1161295295 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b0c004850 0x7f5b0c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:46.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.275+0000 7f5b22d9d700 1 --2- 172.21.15.16:0/1161295295 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5b0c003a00 0x7f5b0c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:46.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.275+0000 7f5b2359e700 1 --2- 172.21.15.16:0/1161295295 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5b0c005210 0x7f5b0c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:46.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.275+0000 7f5b2259c700 1 -- 172.21.15.16:0/1161295295 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5b0c005210 msgr2=0x7f5b0c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:46.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.275+0000 7f5b2259c700 1 --2- 172.21.15.16:0/1161295295 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5b0c005210 0x7f5b0c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:46.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.275+0000 7f5b2259c700 1 -- 172.21.15.16:0/1161295295 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5b0c003a00 msgr2=0x7f5b0c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:46.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.275+0000 7f5b2259c700 1 --2- 172.21.15.16:0/1161295295 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5b0c003a00 0x7f5b0c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:46.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.275+0000 7f5b2259c700 1 -- 172.21.15.16:0/1161295295 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b0c0035b0 con 0x7f5b0c004850 2023-10-16T13:47:46.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.275+0000 7f5b2359e700 1 -- 172.21.15.16:0/1161295295 reap_dead start 2023-10-16T13:47:46.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.275+0000 7f5b2259c700 1 --2- 172.21.15.16:0/1161295295 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b0c004850 0x7f5b0c004cd0 secure :-1 s=READY pgs=2816 cs=0 l=1 rev1=1 crypto rx=0x7f5b18009b60 tx=0x7f5b0c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f3c20c08c450e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:46.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.275+0000 7f5b21d9b700 1 -- 172.21.15.16:0/1161295295 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b1800e6c0 con 0x7f5b0c004850 2023-10-16T13:47:46.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.275+0000 7f5b21d9b700 1 -- 172.21.15.16:0/1161295295 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b1800e820 con 0x7f5b0c004850 2023-10-16T13:47:46.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.275+0000 7f5b21d9b700 1 -- 172.21.15.16:0/1161295295 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b1800eb90 con 0x7f5b0c004850 2023-10-16T13:47:46.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.276+0000 7f5b29039700 1 -- 172.21.15.16:0/1161295295 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b0c004850 msgr2=0x7f5b0c004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:46.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.276+0000 7f5b29039700 1 --2- 172.21.15.16:0/1161295295 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b0c004850 0x7f5b0c004cd0 secure :-1 s=READY pgs=2816 cs=0 l=1 rev1=1 crypto rx=0x7f5b18009b60 tx=0x7f5b0c005880 comp rx=0 tx=0).stop 2023-10-16T13:47:46.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.276+0000 7f5b29039700 1 -- 172.21.15.16:0/1161295295 shutdown_connections 2023-10-16T13:47:46.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.276+0000 7f5b29039700 1 --2- 172.21.15.16:0/1161295295 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b0c004850 0x7f5b0c004cd0 unknown :-1 s=CLOSED pgs=2816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:46.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.276+0000 7f5b29039700 1 -- 172.21.15.16:0/1161295295 >> 172.21.15.16:0/1161295295 conn(0x7f5b240b1fd0 msgr2=0x7f5b240b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:46.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.276+0000 7f5b29039700 1 -- 172.21.15.16:0/1161295295 shutdown_connections 2023-10-16T13:47:46.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.276+0000 7f5b29039700 1 -- 172.21.15.16:0/1161295295 wait complete. 2023-10-16T13:47:46.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.277+0000 7f5b29039700 1 Processor -- start 2023-10-16T13:47:46.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.277+0000 7f5b29039700 1 -- start start 2023-10-16T13:47:46.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.278+0000 7f5b29039700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5b0c004850 0x7f5b2414d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:46.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.278+0000 7f5b29039700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5b240ceda0 0x7f5b2415c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:46.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.278+0000 7f5b29039700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b2415c960 0x7f5b241f5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:46.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.278+0000 7f5b29039700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b240accb0 con 0x7f5b2415c960 2023-10-16T13:47:46.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.278+0000 7f5b29039700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b240acf90 con 0x7f5b240ceda0 2023-10-16T13:47:46.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.278+0000 7f5b29039700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b240ad270 con 0x7f5b0c004850 2023-10-16T13:47:46.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.278+0000 7f5b2359e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b2415c960 0x7f5b241f5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:46.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.278+0000 7f5b2359e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b2415c960 0x7f5b241f5ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55912/0 (socket says 172.21.15.16:55912) 2023-10-16T13:47:46.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.278+0000 7f5b2359e700 1 -- 172.21.15.16:0/1544431532 learned_addr learned my addr 172.21.15.16:0/1544431532 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:46.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.278+0000 7f5b22d9d700 1 --2- 172.21.15.16:0/1544431532 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5b0c004850 0x7f5b2414d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:46.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.278+0000 7f5b2259c700 1 --2- 172.21.15.16:0/1544431532 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5b240ceda0 0x7f5b2415c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:46.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.278+0000 7f5b2359e700 1 -- 172.21.15.16:0/1544431532 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5b0c004850 msgr2=0x7f5b2414d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:46.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.278+0000 7f5b2359e700 1 --2- 172.21.15.16:0/1544431532 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5b0c004850 0x7f5b2414d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:46.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.278+0000 7f5b2359e700 1 -- 172.21.15.16:0/1544431532 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5b240ceda0 msgr2=0x7f5b2415c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:46.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.278+0000 7f5b2359e700 1 --2- 172.21.15.16:0/1544431532 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5b240ceda0 0x7f5b2415c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:46.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.278+0000 7f5b2359e700 1 -- 172.21.15.16:0/1544431532 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b240ad550 con 0x7f5b2415c960 2023-10-16T13:47:46.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.279+0000 7f5b2359e700 1 --2- 172.21.15.16:0/1544431532 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b2415c960 0x7f5b241f5ff0 secure :-1 s=READY pgs=2817 cs=0 l=1 rev1=1 crypto rx=0x7f5b240cce90 tx=0x7f5b1c00ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:46.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.279+0000 7f5b0b7fe700 1 -- 172.21.15.16:0/1544431532 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b1c00ee80 con 0x7f5b2415c960 2023-10-16T13:47:46.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.279+0000 7f5b0b7fe700 1 -- 172.21.15.16:0/1544431532 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b1c003c10 con 0x7f5b2415c960 2023-10-16T13:47:46.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.279+0000 7f5b29039700 1 -- 172.21.15.16:0/1544431532 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b240ad830 con 0x7f5b2415c960 2023-10-16T13:47:46.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.279+0000 7f5b0b7fe700 1 -- 172.21.15.16:0/1544431532 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b1c0050d0 con 0x7f5b2415c960 2023-10-16T13:47:46.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.279+0000 7f5b29039700 1 -- 172.21.15.16:0/1544431532 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b240cf220 con 0x7f5b2415c960 2023-10-16T13:47:46.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.280+0000 7f5b29039700 1 -- 172.21.15.16:0/1544431532 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b240ada20 con 0x7f5b2415c960 2023-10-16T13:47:46.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.280+0000 7f5b0b7fe700 1 -- 172.21.15.16:0/1544431532 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 148) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f5b1c005230 con 0x7f5b2415c960 2023-10-16T13:47:46.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.281+0000 7f5b0b7fe700 1 --2- 172.21.15.16:0/1544431532 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f5b1006b140 0x7f5b1006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:46.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.281+0000 7f5b0b7fe700 1 -- 172.21.15.16:0/1544431532 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 7021+0+0 (secure 0 0 0) 0x7f5b1c089cb0 con 0x7f5b2415c960 2023-10-16T13:47:46.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.281+0000 7f5b22d9d700 1 --2- 172.21.15.16:0/1544431532 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f5b1006b140 0x7f5b1006d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:46.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.281+0000 7f5b22d9d700 1 --2- 172.21.15.16:0/1544431532 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f5b1006b140 0x7f5b1006d600 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5b14001050 tx=0x7f5b14000f80 comp rx=0 tx=0).ready entity=mgr.11886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:46.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.285+0000 7f5b0b7fe700 1 -- 172.21.15.16:0/1544431532 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5b1c054570 con 0x7f5b2415c960 2023-10-16T13:47:46.409 INFO:tasks.ceph.mds.d.smithi049.stderr:2023-10-16T13:47:46.406+0000 7f9f1a76b700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T13:47:46.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.484+0000 7f5b29039700 1 -- 172.21.15.16:0/1544431532 --> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5b240d0b70 con 0x7f5b1006b140 2023-10-16T13:47:46.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.498+0000 7f5b0b7fe700 1 -- 172.21.15.16:0/1544431532 <== mgr.11886 v2:172.21.15.16:6832/1736104576 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+529161 (secure 0 0 0) 0x7f5b140070c0 con 0x7f5b1006b140 2023-10-16T13:47:46.507 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:47:46.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.502+0000 7f5b29039700 1 -- 172.21.15.16:0/1544431532 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f5b1006b140 msgr2=0x7f5b1006d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:46.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.502+0000 7f5b29039700 1 --2- 172.21.15.16:0/1544431532 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f5b1006b140 0x7f5b1006d600 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5b14001050 tx=0x7f5b14000f80 comp rx=0 tx=0).stop 2023-10-16T13:47:46.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.502+0000 7f5b29039700 1 -- 172.21.15.16:0/1544431532 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b2415c960 msgr2=0x7f5b241f5ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:46.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.502+0000 7f5b29039700 1 --2- 172.21.15.16:0/1544431532 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b2415c960 0x7f5b241f5ff0 secure :-1 s=READY pgs=2817 cs=0 l=1 rev1=1 crypto rx=0x7f5b240cce90 tx=0x7f5b1c00ed50 comp rx=0 tx=0).stop 2023-10-16T13:47:46.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.502+0000 7f5b29039700 1 -- 172.21.15.16:0/1544431532 shutdown_connections 2023-10-16T13:47:46.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.502+0000 7f5b29039700 1 --2- 172.21.15.16:0/1544431532 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f5b1006b140 0x7f5b1006d600 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:46.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.502+0000 7f5b29039700 1 --2- 172.21.15.16:0/1544431532 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b2415c960 0x7f5b241f5ff0 unknown :-1 s=CLOSED pgs=2817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:46.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.502+0000 7f5b29039700 1 --2- 172.21.15.16:0/1544431532 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5b0c004850 0x7f5b2414d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:46.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.502+0000 7f5b29039700 1 --2- 172.21.15.16:0/1544431532 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5b240ceda0 0x7f5b2415c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:46.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.502+0000 7f5b29039700 1 -- 172.21.15.16:0/1544431532 >> 172.21.15.16:0/1544431532 conn(0x7f5b240b1fd0 msgr2=0x7f5b240b4640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:46.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.502+0000 7f5b29039700 1 -- 172.21.15.16:0/1544431532 shutdown_connections 2023-10-16T13:47:46.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.502+0000 7f5b29039700 1 -- 172.21.15.16:0/1544431532 wait complete. 2023-10-16T13:47:46.515 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-10-16T13:47:46.518 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":43,"stamp":"2023-10-16T13:47:45.825850+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596435,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":711,"num_write_kb":2447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":450,"ondisk_log_size":450,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":302,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":450068,"kb_used_data":25688,"kb_used_omap":12,"kb_used_meta":424243,"kb_avail":754524652,"statfs":{"total":773094113280,"available":772633243648,"internally_reserved":0,"allocated":26304512,"data_stored":14138623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12740,"internal_metadata":434425404},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3397,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":46,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.928850"},"pg_stats":[{"pgid":"34.1d","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.953163+0000","last_change":"2023-10-16T13:47:41.943421+0000","last_active":"2023-10-16T13:47:41.953163+0000","last_peered":"2023-10-16T13:47:41.953163+0000","last_clean":"2023-10-16T13:47:41.953163+0000","last_became_active":"2023-10-16T13:47:41.942988+0000","last_became_peered":"2023-10-16T13:47:41.942988+0000","last_unstale":"2023-10-16T13:47:41.953163+0000","last_undegraded":"2023-10-16T13:47:41.953163+0000","last_fullsized":"2023-10-16T13:47:41.953163+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:45:39.186801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1e","version":"248'1","reported_seq":12,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.948433+0000","last_change":"2023-10-16T13:47:40.912655+0000","last_active":"2023-10-16T13:47:41.948433+0000","last_peered":"2023-10-16T13:47:41.948433+0000","last_clean":"2023-10-16T13:47:41.948433+0000","last_became_active":"2023-10-16T13:47:40.912369+0000","last_became_peered":"2023-10-16T13:47:40.912369+0000","last_unstale":"2023-10-16T13:47:41.948433+0000","last_undegraded":"2023-10-16T13:47:41.948433+0000","last_fullsized":"2023-10-16T13:47:41.948433+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_clean_scrub_stamp":"2023-10-16T13:47:39.890588+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:50:08.706472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1f","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.908471+0000","last_change":"2023-10-16T13:47:24.429332+0000","last_active":"2023-10-16T13:47:40.908471+0000","last_peered":"2023-10-16T13:47:40.908471+0000","last_clean":"2023-10-16T13:47:40.908471+0000","last_became_active":"2023-10-16T13:47:24.428151+0000","last_became_peered":"2023-10-16T13:47:24.428151+0000","last_unstale":"2023-10-16T13:47:40.908471+0000","last_undegraded":"2023-10-16T13:47:40.908471+0000","last_fullsized":"2023-10-16T13:47:40.908471+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:51:49.567279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1c","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.952421+0000","last_change":"2023-10-16T13:47:41.940099+0000","last_active":"2023-10-16T13:47:41.952421+0000","last_peered":"2023-10-16T13:47:41.952421+0000","last_clean":"2023-10-16T13:47:41.952421+0000","last_became_active":"2023-10-16T13:47:41.939640+0000","last_became_peered":"2023-10-16T13:47:41.939640+0000","last_unstale":"2023-10-16T13:47:41.952421+0000","last_undegraded":"2023-10-16T13:47:41.952421+0000","last_fullsized":"2023-10-16T13:47:41.952421+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:22:07.291807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.1f","version":"248'4","reported_seq":15,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.953154+0000","last_change":"2023-10-16T13:47:40.924092+0000","last_active":"2023-10-16T13:47:41.953154+0000","last_peered":"2023-10-16T13:47:41.953154+0000","last_clean":"2023-10-16T13:47:41.953154+0000","last_became_active":"2023-10-16T13:47:40.923664+0000","last_became_peered":"2023-10-16T13:47:40.923664+0000","last_unstale":"2023-10-16T13:47:41.953154+0000","last_undegraded":"2023-10-16T13:47:41.953154+0000","last_fullsized":"2023-10-16T13:47:41.953154+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_clean_scrub_stamp":"2023-10-16T13:47:39.890588+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:16:06.419458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1e","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.919140+0000","last_change":"2023-10-16T13:47:24.439610+0000","last_active":"2023-10-16T13:47:40.919140+0000","last_peered":"2023-10-16T13:47:40.919140+0000","last_clean":"2023-10-16T13:47:40.919140+0000","last_became_active":"2023-10-16T13:47:24.439212+0000","last_became_peered":"2023-10-16T13:47:24.439212+0000","last_unstale":"2023-10-16T13:47:40.919140+0000","last_undegraded":"2023-10-16T13:47:40.919140+0000","last_fullsized":"2023-10-16T13:47:40.919140+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:47:24.106272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.1f","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.961266+0000","last_change":"2023-10-16T13:47:41.954643+0000","last_active":"2023-10-16T13:47:41.961266+0000","last_peered":"2023-10-16T13:47:41.961266+0000","last_clean":"2023-10-16T13:47:41.961266+0000","last_became_active":"2023-10-16T13:47:41.954321+0000","last_became_peered":"2023-10-16T13:47:41.954321+0000","last_unstale":"2023-10-16T13:47:41.961266+0000","last_undegraded":"2023-10-16T13:47:41.961266+0000","last_fullsized":"2023-10-16T13:47:41.961266+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:23:28.072263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1c","version":"248'2","reported_seq":13,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.945744+0000","last_change":"2023-10-16T13:47:40.920627+0000","last_active":"2023-10-16T13:47:41.945744+0000","last_peered":"2023-10-16T13:47:41.945744+0000","last_clean":"2023-10-16T13:47:41.945744+0000","last_became_active":"2023-10-16T13:47:40.920289+0000","last_became_peered":"2023-10-16T13:47:40.920289+0000","last_unstale":"2023-10-16T13:47:41.945744+0000","last_undegraded":"2023-10-16T13:47:41.945744+0000","last_fullsized":"2023-10-16T13:47:41.945744+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_clean_scrub_stamp":"2023-10-16T13:47:39.890588+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:37:19.404184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1d","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.917110+0000","last_change":"2023-10-16T13:47:24.432216+0000","last_active":"2023-10-16T13:47:40.917110+0000","last_peered":"2023-10-16T13:47:40.917110+0000","last_clean":"2023-10-16T13:47:40.917110+0000","last_became_active":"2023-10-16T13:47:24.431937+0000","last_became_peered":"2023-10-16T13:47:24.431937+0000","last_unstale":"2023-10-16T13:47:40.917110+0000","last_undegraded":"2023-10-16T13:47:40.917110+0000","last_fullsized":"2023-10-16T13:47:40.917110+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:23:10.580869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.1e","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.957795+0000","last_change":"2023-10-16T13:47:41.955036+0000","last_active":"2023-10-16T13:47:41.957795+0000","last_peered":"2023-10-16T13:47:41.957795+0000","last_clean":"2023-10-16T13:47:41.957795+0000","last_became_active":"2023-10-16T13:47:41.954734+0000","last_became_peered":"2023-10-16T13:47:41.954734+0000","last_unstale":"2023-10-16T13:47:41.957795+0000","last_undegraded":"2023-10-16T13:47:41.957795+0000","last_fullsized":"2023-10-16T13:47:41.957795+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:53:38.009404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1d","version":"248'2","reported_seq":13,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.960404+0000","last_change":"2023-10-16T13:47:40.924994+0000","last_active":"2023-10-16T13:47:41.960404+0000","last_peered":"2023-10-16T13:47:41.960404+0000","last_clean":"2023-10-16T13:47:41.960404+0000","last_became_active":"2023-10-16T13:47:40.924683+0000","last_became_peered":"2023-10-16T13:47:40.924683+0000","last_unstale":"2023-10-16T13:47:41.960404+0000","last_undegraded":"2023-10-16T13:47:41.960404+0000","last_fullsized":"2023-10-16T13:47:41.960404+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_clean_scrub_stamp":"2023-10-16T13:47:39.890588+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:52:44.697434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.1c","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.917406+0000","last_change":"2023-10-16T13:47:24.440252+0000","last_active":"2023-10-16T13:47:40.917406+0000","last_peered":"2023-10-16T13:47:40.917406+0000","last_clean":"2023-10-16T13:47:40.917406+0000","last_became_active":"2023-10-16T13:47:24.439982+0000","last_became_peered":"2023-10-16T13:47:24.439982+0000","last_unstale":"2023-10-16T13:47:40.917406+0000","last_undegraded":"2023-10-16T13:47:40.917406+0000","last_fullsized":"2023-10-16T13:47:40.917406+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:54:02.930019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.19","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.952047+0000","last_change":"2023-10-16T13:47:41.948411+0000","last_active":"2023-10-16T13:47:41.952047+0000","last_peered":"2023-10-16T13:47:41.952047+0000","last_clean":"2023-10-16T13:47:41.952047+0000","last_became_active":"2023-10-16T13:47:41.948123+0000","last_became_peered":"2023-10-16T13:47:41.948123+0000","last_unstale":"2023-10-16T13:47:41.952047+0000","last_undegraded":"2023-10-16T13:47:41.952047+0000","last_fullsized":"2023-10-16T13:47:41.952047+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:33:03.023121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1a","version":"248'1","reported_seq":12,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.953618+0000","last_change":"2023-10-16T13:47:40.906916+0000","last_active":"2023-10-16T13:47:41.953618+0000","last_peered":"2023-10-16T13:47:41.953618+0000","last_clean":"2023-10-16T13:47:41.953618+0000","last_became_active":"2023-10-16T13:47:40.906594+0000","last_became_peered":"2023-10-16T13:47:40.906594+0000","last_unstale":"2023-10-16T13:47:41.953618+0000","last_undegraded":"2023-10-16T13:47:41.953618+0000","last_fullsized":"2023-10-16T13:47:41.953618+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_clean_scrub_stamp":"2023-10-16T13:47:39.890588+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:11:06.086887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.1b","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.908372+0000","last_change":"2023-10-16T13:47:24.440049+0000","last_active":"2023-10-16T13:47:40.908372+0000","last_peered":"2023-10-16T13:47:40.908372+0000","last_clean":"2023-10-16T13:47:40.908372+0000","last_became_active":"2023-10-16T13:47:24.439750+0000","last_became_peered":"2023-10-16T13:47:24.439750+0000","last_unstale":"2023-10-16T13:47:40.908372+0000","last_undegraded":"2023-10-16T13:47:40.908372+0000","last_fullsized":"2023-10-16T13:47:40.908372+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:33:40.094175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.18","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.948706+0000","last_change":"2023-10-16T13:47:41.945478+0000","last_active":"2023-10-16T13:47:41.948706+0000","last_peered":"2023-10-16T13:47:41.948706+0000","last_clean":"2023-10-16T13:47:41.948706+0000","last_became_active":"2023-10-16T13:47:41.945033+0000","last_became_peered":"2023-10-16T13:47:41.945033+0000","last_unstale":"2023-10-16T13:47:41.948706+0000","last_undegraded":"2023-10-16T13:47:41.948706+0000","last_fullsized":"2023-10-16T13:47:41.948706+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:42:05.562484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1b","version":"248'1","reported_seq":12,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.945214+0000","last_change":"2023-10-16T13:47:40.923823+0000","last_active":"2023-10-16T13:47:41.945214+0000","last_peered":"2023-10-16T13:47:41.945214+0000","last_clean":"2023-10-16T13:47:41.945214+0000","last_became_active":"2023-10-16T13:47:40.923524+0000","last_became_peered":"2023-10-16T13:47:40.923524+0000","last_unstale":"2023-10-16T13:47:41.945214+0000","last_undegraded":"2023-10-16T13:47:41.945214+0000","last_fullsized":"2023-10-16T13:47:41.945214+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_clean_scrub_stamp":"2023-10-16T13:47:39.890588+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:27:11.351621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1a","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.909354+0000","last_change":"2023-10-16T13:47:24.433661+0000","last_active":"2023-10-16T13:47:40.909354+0000","last_peered":"2023-10-16T13:47:40.909354+0000","last_clean":"2023-10-16T13:47:40.909354+0000","last_became_active":"2023-10-16T13:47:24.433417+0000","last_became_peered":"2023-10-16T13:47:24.433417+0000","last_unstale":"2023-10-16T13:47:40.909354+0000","last_undegraded":"2023-10-16T13:47:40.909354+0000","last_fullsized":"2023-10-16T13:47:40.909354+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:16:02.391378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1b","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.955228+0000","last_change":"2023-10-16T13:47:41.950957+0000","last_active":"2023-10-16T13:47:41.955228+0000","last_peered":"2023-10-16T13:47:41.955228+0000","last_clean":"2023-10-16T13:47:41.955228+0000","last_became_active":"2023-10-16T13:47:41.945092+0000","last_became_peered":"2023-10-16T13:47:41.945092+0000","last_unstale":"2023-10-16T13:47:41.955228+0000","last_undegraded":"2023-10-16T13:47:41.955228+0000","last_fullsized":"2023-10-16T13:47:41.955228+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:14:41.165689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.18","version":"0'0","reported_seq":11,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.932088+0000","last_change":"2023-10-16T13:47:40.924893+0000","last_active":"2023-10-16T13:47:41.932088+0000","last_peered":"2023-10-16T13:47:41.932088+0000","last_clean":"2023-10-16T13:47:41.932088+0000","last_became_active":"2023-10-16T13:47:40.924606+0000","last_became_peered":"2023-10-16T13:47:40.924606+0000","last_unstale":"2023-10-16T13:47:41.932088+0000","last_undegraded":"2023-10-16T13:47:41.932088+0000","last_fullsized":"2023-10-16T13:47:41.932088+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_clean_scrub_stamp":"2023-10-16T13:47:39.890588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:16:08.357053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.909760+0000","last_change":"2023-10-16T13:47:24.437673+0000","last_active":"2023-10-16T13:47:40.909760+0000","last_peered":"2023-10-16T13:47:40.909760+0000","last_clean":"2023-10-16T13:47:40.909760+0000","last_became_active":"2023-10-16T13:47:24.437479+0000","last_became_peered":"2023-10-16T13:47:24.437479+0000","last_unstale":"2023-10-16T13:47:40.909760+0000","last_undegraded":"2023-10-16T13:47:40.909760+0000","last_fullsized":"2023-10-16T13:47:40.909760+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:23:58.749586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.1a","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.954976+0000","last_change":"2023-10-16T13:47:41.946361+0000","last_active":"2023-10-16T13:47:41.954976+0000","last_peered":"2023-10-16T13:47:41.954976+0000","last_clean":"2023-10-16T13:47:41.954976+0000","last_became_active":"2023-10-16T13:47:41.946074+0000","last_became_peered":"2023-10-16T13:47:41.946074+0000","last_unstale":"2023-10-16T13:47:41.954976+0000","last_undegraded":"2023-10-16T13:47:41.954976+0000","last_fullsized":"2023-10-16T13:47:41.954976+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:59:32.553302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.19","version":"0'0","reported_seq":11,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.932544+0000","last_change":"2023-10-16T13:47:40.921334+0000","last_active":"2023-10-16T13:47:41.932544+0000","last_peered":"2023-10-16T13:47:41.932544+0000","last_clean":"2023-10-16T13:47:41.932544+0000","last_became_active":"2023-10-16T13:47:40.920988+0000","last_became_peered":"2023-10-16T13:47:40.920988+0000","last_unstale":"2023-10-16T13:47:41.932544+0000","last_undegraded":"2023-10-16T13:47:41.932544+0000","last_fullsized":"2023-10-16T13:47:41.932544+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_clean_scrub_stamp":"2023-10-16T13:47:39.890588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:09:43.358758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.907346+0000","last_change":"2023-10-16T13:47:24.426442+0000","last_active":"2023-10-16T13:47:40.907346+0000","last_peered":"2023-10-16T13:47:40.907346+0000","last_clean":"2023-10-16T13:47:40.907346+0000","last_became_active":"2023-10-16T13:47:24.425929+0000","last_became_peered":"2023-10-16T13:47:24.425929+0000","last_unstale":"2023-10-16T13:47:40.907346+0000","last_undegraded":"2023-10-16T13:47:40.907346+0000","last_fullsized":"2023-10-16T13:47:40.907346+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:05:13.565304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.15","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.953140+0000","last_change":"2023-10-16T13:47:41.943255+0000","last_active":"2023-10-16T13:47:41.953140+0000","last_peered":"2023-10-16T13:47:41.953140+0000","last_clean":"2023-10-16T13:47:41.953140+0000","last_became_active":"2023-10-16T13:47:41.942596+0000","last_became_peered":"2023-10-16T13:47:41.942596+0000","last_unstale":"2023-10-16T13:47:41.953140+0000","last_undegraded":"2023-10-16T13:47:41.953140+0000","last_fullsized":"2023-10-16T13:47:41.953140+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:45:27.820639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.16","version":"0'0","reported_seq":11,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.934608+0000","last_change":"2023-10-16T13:47:40.922500+0000","last_active":"2023-10-16T13:47:41.934608+0000","last_peered":"2023-10-16T13:47:41.934608+0000","last_clean":"2023-10-16T13:47:41.934608+0000","last_became_active":"2023-10-16T13:47:40.921368+0000","last_became_peered":"2023-10-16T13:47:40.921368+0000","last_unstale":"2023-10-16T13:47:41.934608+0000","last_undegraded":"2023-10-16T13:47:41.934608+0000","last_fullsized":"2023-10-16T13:47:41.934608+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_clean_scrub_stamp":"2023-10-16T13:47:39.890588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:14:48.165605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.908613+0000","last_change":"2023-10-16T13:47:24.426459+0000","last_active":"2023-10-16T13:47:40.908613+0000","last_peered":"2023-10-16T13:47:40.908613+0000","last_clean":"2023-10-16T13:47:40.908613+0000","last_became_active":"2023-10-16T13:47:24.426169+0000","last_became_peered":"2023-10-16T13:47:24.426169+0000","last_unstale":"2023-10-16T13:47:40.908613+0000","last_undegraded":"2023-10-16T13:47:40.908613+0000","last_fullsized":"2023-10-16T13:47:40.908613+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:57:42.170954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.14","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.955137+0000","last_change":"2023-10-16T13:47:41.952226+0000","last_active":"2023-10-16T13:47:41.955137+0000","last_peered":"2023-10-16T13:47:41.955137+0000","last_clean":"2023-10-16T13:47:41.955137+0000","last_became_active":"2023-10-16T13:47:41.951929+0000","last_became_peered":"2023-10-16T13:47:41.951929+0000","last_unstale":"2023-10-16T13:47:41.955137+0000","last_undegraded":"2023-10-16T13:47:41.955137+0000","last_fullsized":"2023-10-16T13:47:41.955137+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:11:32.758621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.17","version":"0'0","reported_seq":11,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.925971+0000","last_change":"2023-10-16T13:47:40.921347+0000","last_active":"2023-10-16T13:47:41.925971+0000","last_peered":"2023-10-16T13:47:41.925971+0000","last_clean":"2023-10-16T13:47:41.925971+0000","last_became_active":"2023-10-16T13:47:40.921032+0000","last_became_peered":"2023-10-16T13:47:40.921032+0000","last_unstale":"2023-10-16T13:47:41.925971+0000","last_undegraded":"2023-10-16T13:47:41.925971+0000","last_fullsized":"2023-10-16T13:47:41.925971+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_clean_scrub_stamp":"2023-10-16T13:47:39.890588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:39:28.570014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.917620+0000","last_change":"2023-10-16T13:47:24.427333+0000","last_active":"2023-10-16T13:47:40.917620+0000","last_peered":"2023-10-16T13:47:40.917620+0000","last_clean":"2023-10-16T13:47:40.917620+0000","last_became_active":"2023-10-16T13:47:24.427034+0000","last_became_peered":"2023-10-16T13:47:24.427034+0000","last_unstale":"2023-10-16T13:47:40.917620+0000","last_undegraded":"2023-10-16T13:47:40.917620+0000","last_fullsized":"2023-10-16T13:47:40.917620+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:27:20.181386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.17","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.952402+0000","last_change":"2023-10-16T13:47:41.945082+0000","last_active":"2023-10-16T13:47:41.952402+0000","last_peered":"2023-10-16T13:47:41.952402+0000","last_clean":"2023-10-16T13:47:41.952402+0000","last_became_active":"2023-10-16T13:47:41.944627+0000","last_became_peered":"2023-10-16T13:47:41.944627+0000","last_unstale":"2023-10-16T13:47:41.952402+0000","last_undegraded":"2023-10-16T13:47:41.952402+0000","last_fullsized":"2023-10-16T13:47:41.952402+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:16:55.340334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.14","version":"248'6","reported_seq":17,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.958613+0000","last_change":"2023-10-16T13:47:40.920578+0000","last_active":"2023-10-16T13:47:41.958613+0000","last_peered":"2023-10-16T13:47:41.958613+0000","last_clean":"2023-10-16T13:47:41.958613+0000","last_became_active":"2023-10-16T13:47:40.920319+0000","last_became_peered":"2023-10-16T13:47:40.920319+0000","last_unstale":"2023-10-16T13:47:41.958613+0000","last_undegraded":"2023-10-16T13:47:41.958613+0000","last_fullsized":"2023-10-16T13:47:41.958613+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_clean_scrub_stamp":"2023-10-16T13:47:39.890588+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:22:21.193371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.15","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.917198+0000","last_change":"2023-10-16T13:47:24.425903+0000","last_active":"2023-10-16T13:47:40.917198+0000","last_peered":"2023-10-16T13:47:40.917198+0000","last_clean":"2023-10-16T13:47:40.917198+0000","last_became_active":"2023-10-16T13:47:24.425593+0000","last_became_peered":"2023-10-16T13:47:24.425593+0000","last_unstale":"2023-10-16T13:47:40.917198+0000","last_undegraded":"2023-10-16T13:47:40.917198+0000","last_fullsized":"2023-10-16T13:47:40.917198+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:46:35.236231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.16","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.956175+0000","last_change":"2023-10-16T13:47:41.955747+0000","last_active":"2023-10-16T13:47:41.956175+0000","last_peered":"2023-10-16T13:47:41.956175+0000","last_clean":"2023-10-16T13:47:41.956175+0000","last_became_active":"2023-10-16T13:47:41.955443+0000","last_became_peered":"2023-10-16T13:47:41.955443+0000","last_unstale":"2023-10-16T13:47:41.956175+0000","last_undegraded":"2023-10-16T13:47:41.956175+0000","last_fullsized":"2023-10-16T13:47:41.956175+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:11:07.092642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.15","version":"248'3","reported_seq":14,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.955416+0000","last_change":"2023-10-16T13:47:40.920585+0000","last_active":"2023-10-16T13:47:41.955416+0000","last_peered":"2023-10-16T13:47:41.955416+0000","last_clean":"2023-10-16T13:47:41.955416+0000","last_became_active":"2023-10-16T13:47:40.920279+0000","last_became_peered":"2023-10-16T13:47:40.920279+0000","last_unstale":"2023-10-16T13:47:41.955416+0000","last_undegraded":"2023-10-16T13:47:41.955416+0000","last_fullsized":"2023-10-16T13:47:41.955416+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_clean_scrub_stamp":"2023-10-16T13:47:39.890588+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:20:13.942596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.14","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.908088+0000","last_change":"2023-10-16T13:47:24.438927+0000","last_active":"2023-10-16T13:47:40.908088+0000","last_peered":"2023-10-16T13:47:40.908088+0000","last_clean":"2023-10-16T13:47:40.908088+0000","last_became_active":"2023-10-16T13:47:24.438389+0000","last_became_peered":"2023-10-16T13:47:24.438389+0000","last_unstale":"2023-10-16T13:47:40.908088+0000","last_undegraded":"2023-10-16T13:47:40.908088+0000","last_fullsized":"2023-10-16T13:47:40.908088+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:44:00.360485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.11","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.952808+0000","last_change":"2023-10-16T13:47:41.949123+0000","last_active":"2023-10-16T13:47:41.952808+0000","last_peered":"2023-10-16T13:47:41.952808+0000","last_clean":"2023-10-16T13:47:41.952808+0000","last_became_active":"2023-10-16T13:47:41.940490+0000","last_became_peered":"2023-10-16T13:47:41.940490+0000","last_unstale":"2023-10-16T13:47:41.952808+0000","last_undegraded":"2023-10-16T13:47:41.952808+0000","last_fullsized":"2023-10-16T13:47:41.952808+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:48:48.259775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.12","version":"0'0","reported_seq":11,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.927024+0000","last_change":"2023-10-16T13:47:40.916172+0000","last_active":"2023-10-16T13:47:41.927024+0000","last_peered":"2023-10-16T13:47:41.927024+0000","last_clean":"2023-10-16T13:47:41.927024+0000","last_became_active":"2023-10-16T13:47:40.915949+0000","last_became_peered":"2023-10-16T13:47:40.915949+0000","last_unstale":"2023-10-16T13:47:41.927024+0000","last_undegraded":"2023-10-16T13:47:41.927024+0000","last_fullsized":"2023-10-16T13:47:41.927024+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_clean_scrub_stamp":"2023-10-16T13:47:39.890588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:10:20.628054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.13","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.911535+0000","last_change":"2023-10-16T13:47:24.445148+0000","last_active":"2023-10-16T13:47:40.911535+0000","last_peered":"2023-10-16T13:47:40.911535+0000","last_clean":"2023-10-16T13:47:40.911535+0000","last_became_active":"2023-10-16T13:47:24.444896+0000","last_became_peered":"2023-10-16T13:47:24.444896+0000","last_unstale":"2023-10-16T13:47:40.911535+0000","last_undegraded":"2023-10-16T13:47:40.911535+0000","last_fullsized":"2023-10-16T13:47:40.911535+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:35:13.886134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.10","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.942446+0000","last_change":"2023-10-16T13:47:41.940848+0000","last_active":"2023-10-16T13:47:41.942446+0000","last_peered":"2023-10-16T13:47:41.942446+0000","last_clean":"2023-10-16T13:47:41.942446+0000","last_became_active":"2023-10-16T13:47:41.940524+0000","last_became_peered":"2023-10-16T13:47:41.940524+0000","last_unstale":"2023-10-16T13:47:41.942446+0000","last_undegraded":"2023-10-16T13:47:41.942446+0000","last_fullsized":"2023-10-16T13:47:41.942446+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:16:57.274456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.13","version":"248'2","reported_seq":13,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.956305+0000","last_change":"2023-10-16T13:47:40.924961+0000","last_active":"2023-10-16T13:47:41.956305+0000","last_peered":"2023-10-16T13:47:41.956305+0000","last_clean":"2023-10-16T13:47:41.956305+0000","last_became_active":"2023-10-16T13:47:40.924672+0000","last_became_peered":"2023-10-16T13:47:40.924672+0000","last_unstale":"2023-10-16T13:47:41.956305+0000","last_undegraded":"2023-10-16T13:47:41.956305+0000","last_fullsized":"2023-10-16T13:47:41.956305+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_clean_scrub_stamp":"2023-10-16T13:47:39.890588+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:50:03.482163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.903870+0000","last_change":"2023-10-16T13:47:24.437693+0000","last_active":"2023-10-16T13:47:40.903870+0000","last_peered":"2023-10-16T13:47:40.903870+0000","last_clean":"2023-10-16T13:47:40.903870+0000","last_became_active":"2023-10-16T13:47:24.437434+0000","last_became_peered":"2023-10-16T13:47:24.437434+0000","last_unstale":"2023-10-16T13:47:40.903870+0000","last_undegraded":"2023-10-16T13:47:40.903870+0000","last_fullsized":"2023-10-16T13:47:40.903870+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:47:56.221847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.13","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.945182+0000","last_change":"2023-10-16T13:47:41.939870+0000","last_active":"2023-10-16T13:47:41.945182+0000","last_peered":"2023-10-16T13:47:41.945182+0000","last_clean":"2023-10-16T13:47:41.945182+0000","last_became_active":"2023-10-16T13:47:41.939443+0000","last_became_peered":"2023-10-16T13:47:41.939443+0000","last_unstale":"2023-10-16T13:47:41.945182+0000","last_undegraded":"2023-10-16T13:47:41.945182+0000","last_fullsized":"2023-10-16T13:47:41.945182+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:04:17.705847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.10","version":"248'8","reported_seq":19,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.961146+0000","last_change":"2023-10-16T13:47:40.920255+0000","last_active":"2023-10-16T13:47:41.961146+0000","last_peered":"2023-10-16T13:47:41.961146+0000","last_clean":"2023-10-16T13:47:41.961146+0000","last_became_active":"2023-10-16T13:47:40.918169+0000","last_became_peered":"2023-10-16T13:47:40.918169+0000","last_unstale":"2023-10-16T13:47:41.961146+0000","last_undegraded":"2023-10-16T13:47:41.961146+0000","last_fullsized":"2023-10-16T13:47:41.961146+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_clean_scrub_stamp":"2023-10-16T13:47:39.890588+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:29:57.627195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.916781+0000","last_change":"2023-10-16T13:47:24.435339+0000","last_active":"2023-10-16T13:47:40.916781+0000","last_peered":"2023-10-16T13:47:40.916781+0000","last_clean":"2023-10-16T13:47:40.916781+0000","last_became_active":"2023-10-16T13:47:24.434237+0000","last_became_peered":"2023-10-16T13:47:24.434237+0000","last_unstale":"2023-10-16T13:47:40.916781+0000","last_undegraded":"2023-10-16T13:47:40.916781+0000","last_fullsized":"2023-10-16T13:47:40.916781+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:21:21.083320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.12","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.946754+0000","last_change":"2023-10-16T13:47:41.943481+0000","last_active":"2023-10-16T13:47:41.946754+0000","last_peered":"2023-10-16T13:47:41.946754+0000","last_clean":"2023-10-16T13:47:41.946754+0000","last_became_active":"2023-10-16T13:47:41.943190+0000","last_became_peered":"2023-10-16T13:47:41.943190+0000","last_unstale":"2023-10-16T13:47:41.946754+0000","last_undegraded":"2023-10-16T13:47:41.946754+0000","last_fullsized":"2023-10-16T13:47:41.946754+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:16:24.457537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.11","version":"0'0","reported_seq":11,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.937163+0000","last_change":"2023-10-16T13:47:40.914735+0000","last_active":"2023-10-16T13:47:41.937163+0000","last_peered":"2023-10-16T13:47:41.937163+0000","last_clean":"2023-10-16T13:47:41.937163+0000","last_became_active":"2023-10-16T13:47:40.914441+0000","last_became_peered":"2023-10-16T13:47:40.914441+0000","last_unstale":"2023-10-16T13:47:41.937163+0000","last_undegraded":"2023-10-16T13:47:41.937163+0000","last_fullsized":"2023-10-16T13:47:41.937163+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_clean_scrub_stamp":"2023-10-16T13:47:39.890588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:35:10.098500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.10","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.912641+0000","last_change":"2023-10-16T13:47:24.441060+0000","last_active":"2023-10-16T13:47:40.912641+0000","last_peered":"2023-10-16T13:47:40.912641+0000","last_clean":"2023-10-16T13:47:40.912641+0000","last_became_active":"2023-10-16T13:47:24.440745+0000","last_became_peered":"2023-10-16T13:47:24.440745+0000","last_unstale":"2023-10-16T13:47:40.912641+0000","last_undegraded":"2023-10-16T13:47:40.912641+0000","last_fullsized":"2023-10-16T13:47:40.912641+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:48:26.805075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.d","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.952889+0000","last_change":"2023-10-16T13:47:41.949823+0000","last_active":"2023-10-16T13:47:41.952889+0000","last_peered":"2023-10-16T13:47:41.952889+0000","last_clean":"2023-10-16T13:47:41.952889+0000","last_became_active":"2023-10-16T13:47:41.949551+0000","last_became_peered":"2023-10-16T13:47:41.949551+0000","last_unstale":"2023-10-16T13:47:41.952889+0000","last_undegraded":"2023-10-16T13:47:41.952889+0000","last_fullsized":"2023-10-16T13:47:41.952889+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:42:46.808296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.e","version":"248'2","reported_seq":13,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.946445+0000","last_change":"2023-10-16T13:47:40.924491+0000","last_active":"2023-10-16T13:47:41.946445+0000","last_peered":"2023-10-16T13:47:41.946445+0000","last_clean":"2023-10-16T13:47:41.946445+0000","last_became_active":"2023-10-16T13:47:40.924197+0000","last_became_peered":"2023-10-16T13:47:40.924197+0000","last_unstale":"2023-10-16T13:47:41.946445+0000","last_undegraded":"2023-10-16T13:47:41.946445+0000","last_fullsized":"2023-10-16T13:47:41.946445+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_clean_scrub_stamp":"2023-10-16T13:47:39.890588+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:17:47.585967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.f","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.920388+0000","last_change":"2023-10-16T13:47:24.438572+0000","last_active":"2023-10-16T13:47:40.920388+0000","last_peered":"2023-10-16T13:47:40.920388+0000","last_clean":"2023-10-16T13:47:40.920388+0000","last_became_active":"2023-10-16T13:47:24.438283+0000","last_became_peered":"2023-10-16T13:47:24.438283+0000","last_unstale":"2023-10-16T13:47:40.920388+0000","last_undegraded":"2023-10-16T13:47:40.920388+0000","last_fullsized":"2023-10-16T13:47:40.920388+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:48:21.499862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.c","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.947191+0000","last_change":"2023-10-16T13:47:41.941965+0000","last_active":"2023-10-16T13:47:41.947191+0000","last_peered":"2023-10-16T13:47:41.947191+0000","last_clean":"2023-10-16T13:47:41.947191+0000","last_became_active":"2023-10-16T13:47:41.941661+0000","last_became_peered":"2023-10-16T13:47:41.941661+0000","last_unstale":"2023-10-16T13:47:41.947191+0000","last_undegraded":"2023-10-16T13:47:41.947191+0000","last_fullsized":"2023-10-16T13:47:41.947191+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:18:34.935509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.f","version":"248'1","reported_seq":12,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.955586+0000","last_change":"2023-10-16T13:47:40.925860+0000","last_active":"2023-10-16T13:47:41.955586+0000","last_peered":"2023-10-16T13:47:41.955586+0000","last_clean":"2023-10-16T13:47:41.955586+0000","last_became_active":"2023-10-16T13:47:40.925444+0000","last_became_peered":"2023-10-16T13:47:40.925444+0000","last_unstale":"2023-10-16T13:47:41.955586+0000","last_undegraded":"2023-10-16T13:47:41.955586+0000","last_fullsized":"2023-10-16T13:47:41.955586+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_clean_scrub_stamp":"2023-10-16T13:47:39.890588+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:44:11.539188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.e","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.917726+0000","last_change":"2023-10-16T13:47:24.433506+0000","last_active":"2023-10-16T13:47:40.917726+0000","last_peered":"2023-10-16T13:47:40.917726+0000","last_clean":"2023-10-16T13:47:40.917726+0000","last_became_active":"2023-10-16T13:47:24.433218+0000","last_became_peered":"2023-10-16T13:47:24.433218+0000","last_unstale":"2023-10-16T13:47:40.917726+0000","last_undegraded":"2023-10-16T13:47:40.917726+0000","last_fullsized":"2023-10-16T13:47:40.917726+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:04:28.651883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.f","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.956437+0000","last_change":"2023-10-16T13:47:41.955137+0000","last_active":"2023-10-16T13:47:41.956437+0000","last_peered":"2023-10-16T13:47:41.956437+0000","last_clean":"2023-10-16T13:47:41.956437+0000","last_became_active":"2023-10-16T13:47:41.954831+0000","last_became_peered":"2023-10-16T13:47:41.954831+0000","last_unstale":"2023-10-16T13:47:41.956437+0000","last_undegraded":"2023-10-16T13:47:41.956437+0000","last_fullsized":"2023-10-16T13:47:41.956437+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:20:04.850917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.c","version":"0'0","reported_seq":11,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.930092+0000","last_change":"2023-10-16T13:47:40.925515+0000","last_active":"2023-10-16T13:47:41.930092+0000","last_peered":"2023-10-16T13:47:41.930092+0000","last_clean":"2023-10-16T13:47:41.930092+0000","last_became_active":"2023-10-16T13:47:40.925203+0000","last_became_peered":"2023-10-16T13:47:40.925203+0000","last_unstale":"2023-10-16T13:47:41.930092+0000","last_undegraded":"2023-10-16T13:47:41.930092+0000","last_fullsized":"2023-10-16T13:47:41.930092+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_clean_scrub_stamp":"2023-10-16T13:47:39.890588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:03:13.465329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.d","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.916690+0000","last_change":"2023-10-16T13:47:24.443171+0000","last_active":"2023-10-16T13:47:40.916690+0000","last_peered":"2023-10-16T13:47:40.916690+0000","last_clean":"2023-10-16T13:47:40.916690+0000","last_became_active":"2023-10-16T13:47:24.442811+0000","last_became_peered":"2023-10-16T13:47:24.442811+0000","last_unstale":"2023-10-16T13:47:40.916690+0000","last_undegraded":"2023-10-16T13:47:40.916690+0000","last_fullsized":"2023-10-16T13:47:40.916690+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:56:16.580840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.e","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.953264+0000","last_change":"2023-10-16T13:47:41.940562+0000","last_active":"2023-10-16T13:47:41.953264+0000","last_peered":"2023-10-16T13:47:41.953264+0000","last_clean":"2023-10-16T13:47:41.953264+0000","last_became_active":"2023-10-16T13:47:41.940278+0000","last_became_peered":"2023-10-16T13:47:41.940278+0000","last_unstale":"2023-10-16T13:47:41.953264+0000","last_undegraded":"2023-10-16T13:47:41.953264+0000","last_fullsized":"2023-10-16T13:47:41.953264+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:45:09.969851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.d","version":"248'1","reported_seq":12,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.961641+0000","last_change":"2023-10-16T13:47:40.920222+0000","last_active":"2023-10-16T13:47:41.961641+0000","last_peered":"2023-10-16T13:47:41.961641+0000","last_clean":"2023-10-16T13:47:41.961641+0000","last_became_active":"2023-10-16T13:47:40.918810+0000","last_became_peered":"2023-10-16T13:47:40.918810+0000","last_unstale":"2023-10-16T13:47:41.961641+0000","last_undegraded":"2023-10-16T13:47:41.961641+0000","last_fullsized":"2023-10-16T13:47:41.961641+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_clean_scrub_stamp":"2023-10-16T13:47:39.890588+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:48:43.290715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.919136+0000","last_change":"2023-10-16T13:47:24.435000+0000","last_active":"2023-10-16T13:47:40.919136+0000","last_peered":"2023-10-16T13:47:40.919136+0000","last_clean":"2023-10-16T13:47:40.919136+0000","last_became_active":"2023-10-16T13:47:24.434568+0000","last_became_peered":"2023-10-16T13:47:24.434568+0000","last_unstale":"2023-10-16T13:47:40.919136+0000","last_undegraded":"2023-10-16T13:47:40.919136+0000","last_fullsized":"2023-10-16T13:47:40.919136+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:31:46.967495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.9","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.956331+0000","last_change":"2023-10-16T13:47:41.954083+0000","last_active":"2023-10-16T13:47:41.956331+0000","last_peered":"2023-10-16T13:47:41.956331+0000","last_clean":"2023-10-16T13:47:41.956331+0000","last_became_active":"2023-10-16T13:47:41.953721+0000","last_became_peered":"2023-10-16T13:47:41.953721+0000","last_unstale":"2023-10-16T13:47:41.956331+0000","last_undegraded":"2023-10-16T13:47:41.956331+0000","last_fullsized":"2023-10-16T13:47:41.956331+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:36:49.327966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.a","version":"0'0","reported_seq":11,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.932545+0000","last_change":"2023-10-16T13:47:40.927530+0000","last_active":"2023-10-16T13:47:41.932545+0000","last_peered":"2023-10-16T13:47:41.932545+0000","last_clean":"2023-10-16T13:47:41.932545+0000","last_became_active":"2023-10-16T13:47:40.927244+0000","last_became_peered":"2023-10-16T13:47:40.927244+0000","last_unstale":"2023-10-16T13:47:41.932545+0000","last_undegraded":"2023-10-16T13:47:41.932545+0000","last_fullsized":"2023-10-16T13:47:41.932545+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_clean_scrub_stamp":"2023-10-16T13:47:39.890588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:53:13.673249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.b","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.910729+0000","last_change":"2023-10-16T13:47:24.432842+0000","last_active":"2023-10-16T13:47:40.910729+0000","last_peered":"2023-10-16T13:47:40.910729+0000","last_clean":"2023-10-16T13:47:40.910729+0000","last_became_active":"2023-10-16T13:47:24.432528+0000","last_became_peered":"2023-10-16T13:47:24.432528+0000","last_unstale":"2023-10-16T13:47:40.910729+0000","last_undegraded":"2023-10-16T13:47:40.910729+0000","last_fullsized":"2023-10-16T13:47:40.910729+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:10:57.744179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.8","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.961869+0000","last_change":"2023-10-16T13:47:41.954409+0000","last_active":"2023-10-16T13:47:41.961869+0000","last_peered":"2023-10-16T13:47:41.961869+0000","last_clean":"2023-10-16T13:47:41.961869+0000","last_became_active":"2023-10-16T13:47:41.954075+0000","last_became_peered":"2023-10-16T13:47:41.954075+0000","last_unstale":"2023-10-16T13:47:41.961869+0000","last_undegraded":"2023-10-16T13:47:41.961869+0000","last_fullsized":"2023-10-16T13:47:41.961869+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:52:17.219170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.b","version":"248'2","reported_seq":13,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.959020+0000","last_change":"2023-10-16T13:47:40.918271+0000","last_active":"2023-10-16T13:47:41.959020+0000","last_peered":"2023-10-16T13:47:41.959020+0000","last_clean":"2023-10-16T13:47:41.959020+0000","last_became_active":"2023-10-16T13:47:40.917978+0000","last_became_peered":"2023-10-16T13:47:40.917978+0000","last_unstale":"2023-10-16T13:47:41.959020+0000","last_undegraded":"2023-10-16T13:47:41.959020+0000","last_fullsized":"2023-10-16T13:47:41.959020+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_clean_scrub_stamp":"2023-10-16T13:47:39.890588+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:31:49.189007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.911633+0000","last_change":"2023-10-16T13:47:24.439953+0000","last_active":"2023-10-16T13:47:40.911633+0000","last_peered":"2023-10-16T13:47:40.911633+0000","last_clean":"2023-10-16T13:47:40.911633+0000","last_became_active":"2023-10-16T13:47:24.439662+0000","last_became_peered":"2023-10-16T13:47:24.439662+0000","last_unstale":"2023-10-16T13:47:40.911633+0000","last_undegraded":"2023-10-16T13:47:40.911633+0000","last_fullsized":"2023-10-16T13:47:40.911633+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:16:33.663778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.b","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.934998+0000","last_change":"2023-10-16T13:47:41.934609+0000","last_active":"2023-10-16T13:47:41.934998+0000","last_peered":"2023-10-16T13:47:41.934998+0000","last_clean":"2023-10-16T13:47:41.934998+0000","last_became_active":"2023-10-16T13:47:41.934325+0000","last_became_peered":"2023-10-16T13:47:41.934325+0000","last_unstale":"2023-10-16T13:47:41.934998+0000","last_undegraded":"2023-10-16T13:47:41.934998+0000","last_fullsized":"2023-10-16T13:47:41.934998+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:13:29.287316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.8","version":"0'0","reported_seq":11,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.938607+0000","last_change":"2023-10-16T13:47:40.910759+0000","last_active":"2023-10-16T13:47:41.938607+0000","last_peered":"2023-10-16T13:47:41.938607+0000","last_clean":"2023-10-16T13:47:41.938607+0000","last_became_active":"2023-10-16T13:47:40.910438+0000","last_became_peered":"2023-10-16T13:47:40.910438+0000","last_unstale":"2023-10-16T13:47:41.938607+0000","last_undegraded":"2023-10-16T13:47:41.938607+0000","last_fullsized":"2023-10-16T13:47:41.938607+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_clean_scrub_stamp":"2023-10-16T13:47:39.890588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:03:49.084910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.916407+0000","last_change":"2023-10-16T13:47:24.442582+0000","last_active":"2023-10-16T13:47:40.916407+0000","last_peered":"2023-10-16T13:47:40.916407+0000","last_clean":"2023-10-16T13:47:40.916407+0000","last_became_active":"2023-10-16T13:47:24.442246+0000","last_became_peered":"2023-10-16T13:47:24.442246+0000","last_unstale":"2023-10-16T13:47:40.916407+0000","last_undegraded":"2023-10-16T13:47:40.916407+0000","last_fullsized":"2023-10-16T13:47:40.916407+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:48:26.805075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.a","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.955126+0000","last_change":"2023-10-16T13:47:41.951343+0000","last_active":"2023-10-16T13:47:41.955126+0000","last_peered":"2023-10-16T13:47:41.955126+0000","last_clean":"2023-10-16T13:47:41.955126+0000","last_became_active":"2023-10-16T13:47:41.951030+0000","last_became_peered":"2023-10-16T13:47:41.951030+0000","last_unstale":"2023-10-16T13:47:41.955126+0000","last_undegraded":"2023-10-16T13:47:41.955126+0000","last_fullsized":"2023-10-16T13:47:41.955126+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:01:00.671432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.9","version":"0'0","reported_seq":11,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.925335+0000","last_change":"2023-10-16T13:47:40.926244+0000","last_active":"2023-10-16T13:47:41.925335+0000","last_peered":"2023-10-16T13:47:41.925335+0000","last_clean":"2023-10-16T13:47:41.925335+0000","last_became_active":"2023-10-16T13:47:40.924533+0000","last_became_peered":"2023-10-16T13:47:40.924533+0000","last_unstale":"2023-10-16T13:47:41.925335+0000","last_undegraded":"2023-10-16T13:47:41.925335+0000","last_fullsized":"2023-10-16T13:47:41.925335+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_clean_scrub_stamp":"2023-10-16T13:47:39.890588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:21:13.301295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.911998+0000","last_change":"2023-10-16T13:47:24.437336+0000","last_active":"2023-10-16T13:47:40.911998+0000","last_peered":"2023-10-16T13:47:40.911998+0000","last_clean":"2023-10-16T13:47:40.911998+0000","last_became_active":"2023-10-16T13:47:24.437042+0000","last_became_peered":"2023-10-16T13:47:24.437042+0000","last_unstale":"2023-10-16T13:47:40.911998+0000","last_undegraded":"2023-10-16T13:47:40.911998+0000","last_fullsized":"2023-10-16T13:47:40.911998+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:07:16.886792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.5","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.956114+0000","last_change":"2023-10-16T13:47:41.953682+0000","last_active":"2023-10-16T13:47:41.956114+0000","last_peered":"2023-10-16T13:47:41.956114+0000","last_clean":"2023-10-16T13:47:41.956114+0000","last_became_active":"2023-10-16T13:47:41.953349+0000","last_became_peered":"2023-10-16T13:47:41.953349+0000","last_unstale":"2023-10-16T13:47:41.956114+0000","last_undegraded":"2023-10-16T13:47:41.956114+0000","last_fullsized":"2023-10-16T13:47:41.956114+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:04:46.143278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.6","version":"0'0","reported_seq":11,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.933870+0000","last_change":"2023-10-16T13:47:40.925445+0000","last_active":"2023-10-16T13:47:41.933870+0000","last_peered":"2023-10-16T13:47:41.933870+0000","last_clean":"2023-10-16T13:47:41.933870+0000","last_became_active":"2023-10-16T13:47:40.925135+0000","last_became_peered":"2023-10-16T13:47:40.925135+0000","last_unstale":"2023-10-16T13:47:41.933870+0000","last_undegraded":"2023-10-16T13:47:41.933870+0000","last_fullsized":"2023-10-16T13:47:41.933870+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_clean_scrub_stamp":"2023-10-16T13:47:39.890588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:26:23.723585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.7","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.918630+0000","last_change":"2023-10-16T13:47:24.433031+0000","last_active":"2023-10-16T13:47:40.918630+0000","last_peered":"2023-10-16T13:47:40.918630+0000","last_clean":"2023-10-16T13:47:40.918630+0000","last_became_active":"2023-10-16T13:47:24.432700+0000","last_became_peered":"2023-10-16T13:47:24.432700+0000","last_unstale":"2023-10-16T13:47:40.918630+0000","last_undegraded":"2023-10-16T13:47:40.918630+0000","last_fullsized":"2023-10-16T13:47:40.918630+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:17:10.080681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.4","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.953868+0000","last_change":"2023-10-16T13:47:41.948145+0000","last_active":"2023-10-16T13:47:41.953868+0000","last_peered":"2023-10-16T13:47:41.953868+0000","last_clean":"2023-10-16T13:47:41.953868+0000","last_became_active":"2023-10-16T13:47:41.947487+0000","last_became_peered":"2023-10-16T13:47:41.947487+0000","last_unstale":"2023-10-16T13:47:41.953868+0000","last_undegraded":"2023-10-16T13:47:41.953868+0000","last_fullsized":"2023-10-16T13:47:41.953868+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:52:17.219170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.7","version":"248'2","reported_seq":13,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.953896+0000","last_change":"2023-10-16T13:47:40.922011+0000","last_active":"2023-10-16T13:47:41.953896+0000","last_peered":"2023-10-16T13:47:41.953896+0000","last_clean":"2023-10-16T13:47:41.953896+0000","last_became_active":"2023-10-16T13:47:40.921705+0000","last_became_peered":"2023-10-16T13:47:40.921705+0000","last_unstale":"2023-10-16T13:47:41.953896+0000","last_undegraded":"2023-10-16T13:47:41.953896+0000","last_fullsized":"2023-10-16T13:47:41.953896+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_clean_scrub_stamp":"2023-10-16T13:47:39.890588+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:07:57.036479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.6","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.909762+0000","last_change":"2023-10-16T13:47:24.434972+0000","last_active":"2023-10-16T13:47:40.909762+0000","last_peered":"2023-10-16T13:47:40.909762+0000","last_clean":"2023-10-16T13:47:40.909762+0000","last_became_active":"2023-10-16T13:47:24.434639+0000","last_became_peered":"2023-10-16T13:47:24.434639+0000","last_unstale":"2023-10-16T13:47:40.909762+0000","last_undegraded":"2023-10-16T13:47:40.909762+0000","last_fullsized":"2023-10-16T13:47:40.909762+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:06:13.278420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.7","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.952706+0000","last_change":"2023-10-16T13:47:41.950972+0000","last_active":"2023-10-16T13:47:41.952706+0000","last_peered":"2023-10-16T13:47:41.952706+0000","last_clean":"2023-10-16T13:47:41.952706+0000","last_became_active":"2023-10-16T13:47:41.950422+0000","last_became_peered":"2023-10-16T13:47:41.950422+0000","last_unstale":"2023-10-16T13:47:41.952706+0000","last_undegraded":"2023-10-16T13:47:41.952706+0000","last_fullsized":"2023-10-16T13:47:41.952706+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:20:24.501093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.4","version":"0'0","reported_seq":11,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.945774+0000","last_change":"2023-10-16T13:47:40.919382+0000","last_active":"2023-10-16T13:47:41.945774+0000","last_peered":"2023-10-16T13:47:41.945774+0000","last_clean":"2023-10-16T13:47:41.945774+0000","last_became_active":"2023-10-16T13:47:40.918911+0000","last_became_peered":"2023-10-16T13:47:40.918911+0000","last_unstale":"2023-10-16T13:47:41.945774+0000","last_undegraded":"2023-10-16T13:47:41.945774+0000","last_fullsized":"2023-10-16T13:47:41.945774+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_clean_scrub_stamp":"2023-10-16T13:47:39.890588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:24:04.571425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.907830+0000","last_change":"2023-10-16T13:47:24.428801+0000","last_active":"2023-10-16T13:47:40.907830+0000","last_peered":"2023-10-16T13:47:40.907830+0000","last_clean":"2023-10-16T13:47:40.907830+0000","last_became_active":"2023-10-16T13:47:24.428511+0000","last_became_peered":"2023-10-16T13:47:24.428511+0000","last_unstale":"2023-10-16T13:47:40.907830+0000","last_undegraded":"2023-10-16T13:47:40.907830+0000","last_fullsized":"2023-10-16T13:47:40.907830+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:27:55.268379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.6","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.954858+0000","last_change":"2023-10-16T13:47:41.946681+0000","last_active":"2023-10-16T13:47:41.954858+0000","last_peered":"2023-10-16T13:47:41.954858+0000","last_clean":"2023-10-16T13:47:41.954858+0000","last_became_active":"2023-10-16T13:47:41.946384+0000","last_became_peered":"2023-10-16T13:47:41.946384+0000","last_unstale":"2023-10-16T13:47:41.954858+0000","last_undegraded":"2023-10-16T13:47:41.954858+0000","last_fullsized":"2023-10-16T13:47:41.954858+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:23:58.300909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.5","version":"0'0","reported_seq":11,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.946799+0000","last_change":"2023-10-16T13:47:40.919745+0000","last_active":"2023-10-16T13:47:41.946799+0000","last_peered":"2023-10-16T13:47:41.946799+0000","last_clean":"2023-10-16T13:47:41.946799+0000","last_became_active":"2023-10-16T13:47:40.919442+0000","last_became_peered":"2023-10-16T13:47:40.919442+0000","last_unstale":"2023-10-16T13:47:41.946799+0000","last_undegraded":"2023-10-16T13:47:41.946799+0000","last_fullsized":"2023-10-16T13:47:41.946799+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_clean_scrub_stamp":"2023-10-16T13:47:39.890588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:19:54.778811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.907995+0000","last_change":"2023-10-16T13:47:24.429772+0000","last_active":"2023-10-16T13:47:40.907995+0000","last_peered":"2023-10-16T13:47:40.907995+0000","last_clean":"2023-10-16T13:47:40.907995+0000","last_became_active":"2023-10-16T13:47:24.429496+0000","last_became_peered":"2023-10-16T13:47:24.429496+0000","last_unstale":"2023-10-16T13:47:40.907995+0000","last_undegraded":"2023-10-16T13:47:40.907995+0000","last_fullsized":"2023-10-16T13:47:40.907995+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:09:48.751438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.952024+0000","last_change":"2023-10-16T13:47:41.946231+0000","last_active":"2023-10-16T13:47:41.952024+0000","last_peered":"2023-10-16T13:47:41.952024+0000","last_clean":"2023-10-16T13:47:41.952024+0000","last_became_active":"2023-10-16T13:47:41.945943+0000","last_became_peered":"2023-10-16T13:47:41.945943+0000","last_unstale":"2023-10-16T13:47:41.952024+0000","last_undegraded":"2023-10-16T13:47:41.952024+0000","last_fullsized":"2023-10-16T13:47:41.952024+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:27:37.672780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.2","version":"0'0","reported_seq":11,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.931458+0000","last_change":"2023-10-16T13:47:40.923159+0000","last_active":"2023-10-16T13:47:41.931458+0000","last_peered":"2023-10-16T13:47:41.931458+0000","last_clean":"2023-10-16T13:47:41.931458+0000","last_became_active":"2023-10-16T13:47:40.922887+0000","last_became_peered":"2023-10-16T13:47:40.922887+0000","last_unstale":"2023-10-16T13:47:41.931458+0000","last_undegraded":"2023-10-16T13:47:41.931458+0000","last_fullsized":"2023-10-16T13:47:41.931458+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_clean_scrub_stamp":"2023-10-16T13:47:39.890588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:16:50.149418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.902331+0000","last_change":"2023-10-16T13:47:24.437263+0000","last_active":"2023-10-16T13:47:40.902331+0000","last_peered":"2023-10-16T13:47:40.902331+0000","last_clean":"2023-10-16T13:47:40.902331+0000","last_became_active":"2023-10-16T13:47:24.437023+0000","last_became_peered":"2023-10-16T13:47:24.437023+0000","last_unstale":"2023-10-16T13:47:40.902331+0000","last_undegraded":"2023-10-16T13:47:40.902331+0000","last_fullsized":"2023-10-16T13:47:40.902331+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:59:15.061907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.0","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.955100+0000","last_change":"2023-10-16T13:47:41.948549+0000","last_active":"2023-10-16T13:47:41.955100+0000","last_peered":"2023-10-16T13:47:41.955100+0000","last_clean":"2023-10-16T13:47:41.955100+0000","last_became_active":"2023-10-16T13:47:41.948269+0000","last_became_peered":"2023-10-16T13:47:41.948269+0000","last_unstale":"2023-10-16T13:47:41.955100+0000","last_undegraded":"2023-10-16T13:47:41.955100+0000","last_fullsized":"2023-10-16T13:47:41.955100+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:50:47.237986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.3","version":"0'0","reported_seq":11,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.931248+0000","last_change":"2023-10-16T13:47:40.923995+0000","last_active":"2023-10-16T13:47:41.931248+0000","last_peered":"2023-10-16T13:47:41.931248+0000","last_clean":"2023-10-16T13:47:41.931248+0000","last_became_active":"2023-10-16T13:47:40.923691+0000","last_became_peered":"2023-10-16T13:47:40.923691+0000","last_unstale":"2023-10-16T13:47:41.931248+0000","last_undegraded":"2023-10-16T13:47:41.931248+0000","last_fullsized":"2023-10-16T13:47:41.931248+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_clean_scrub_stamp":"2023-10-16T13:47:39.890588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:48:37.985502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.903986+0000","last_change":"2023-10-16T13:47:24.419847+0000","last_active":"2023-10-16T13:47:40.903986+0000","last_peered":"2023-10-16T13:47:40.903986+0000","last_clean":"2023-10-16T13:47:40.903986+0000","last_became_active":"2023-10-16T13:47:24.419520+0000","last_became_peered":"2023-10-16T13:47:24.419520+0000","last_unstale":"2023-10-16T13:47:40.903986+0000","last_undegraded":"2023-10-16T13:47:40.903986+0000","last_fullsized":"2023-10-16T13:47:40.903986+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:11:44.932011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.3","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.939712+0000","last_change":"2023-10-16T13:47:41.938833+0000","last_active":"2023-10-16T13:47:41.939712+0000","last_peered":"2023-10-16T13:47:41.939712+0000","last_clean":"2023-10-16T13:47:41.939712+0000","last_became_active":"2023-10-16T13:47:41.938389+0000","last_became_peered":"2023-10-16T13:47:41.938389+0000","last_unstale":"2023-10-16T13:47:41.939712+0000","last_undegraded":"2023-10-16T13:47:41.939712+0000","last_fullsized":"2023-10-16T13:47:41.939712+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:16:01.800647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.0","version":"0'0","reported_seq":11,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.934728+0000","last_change":"2023-10-16T13:47:40.917887+0000","last_active":"2023-10-16T13:47:41.934728+0000","last_peered":"2023-10-16T13:47:41.934728+0000","last_clean":"2023-10-16T13:47:41.934728+0000","last_became_active":"2023-10-16T13:47:40.917596+0000","last_became_peered":"2023-10-16T13:47:40.917596+0000","last_unstale":"2023-10-16T13:47:41.934728+0000","last_undegraded":"2023-10-16T13:47:41.934728+0000","last_fullsized":"2023-10-16T13:47:41.934728+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_clean_scrub_stamp":"2023-10-16T13:47:39.890588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:11:54.742851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.918413+0000","last_change":"2023-10-16T13:47:24.435926+0000","last_active":"2023-10-16T13:47:40.918413+0000","last_peered":"2023-10-16T13:47:40.918413+0000","last_clean":"2023-10-16T13:47:40.918413+0000","last_became_active":"2023-10-16T13:47:24.434753+0000","last_became_peered":"2023-10-16T13:47:24.434753+0000","last_unstale":"2023-10-16T13:47:40.918413+0000","last_undegraded":"2023-10-16T13:47:40.918413+0000","last_fullsized":"2023-10-16T13:47:40.918413+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:13:51.332934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.2","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.956911+0000","last_change":"2023-10-16T13:47:41.952646+0000","last_active":"2023-10-16T13:47:41.956911+0000","last_peered":"2023-10-16T13:47:41.956911+0000","last_clean":"2023-10-16T13:47:41.956911+0000","last_became_active":"2023-10-16T13:47:41.952280+0000","last_became_peered":"2023-10-16T13:47:41.952280+0000","last_unstale":"2023-10-16T13:47:41.956911+0000","last_undegraded":"2023-10-16T13:47:41.956911+0000","last_fullsized":"2023-10-16T13:47:41.956911+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:28:20.482489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1","version":"0'0","reported_seq":11,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.945117+0000","last_change":"2023-10-16T13:47:40.919383+0000","last_active":"2023-10-16T13:47:41.945117+0000","last_peered":"2023-10-16T13:47:41.945117+0000","last_clean":"2023-10-16T13:47:41.945117+0000","last_became_active":"2023-10-16T13:47:40.919053+0000","last_became_peered":"2023-10-16T13:47:40.919053+0000","last_unstale":"2023-10-16T13:47:41.945117+0000","last_undegraded":"2023-10-16T13:47:41.945117+0000","last_fullsized":"2023-10-16T13:47:41.945117+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":246,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:39.890588+0000","last_clean_scrub_stamp":"2023-10-16T13:47:39.890588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:21:37.568960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.909357+0000","last_change":"2023-10-16T13:47:24.423908+0000","last_active":"2023-10-16T13:47:40.909357+0000","last_peered":"2023-10-16T13:47:40.909357+0000","last_clean":"2023-10-16T13:47:40.909357+0000","last_became_active":"2023-10-16T13:47:24.423530+0000","last_became_peered":"2023-10-16T13:47:24.423530+0000","last_unstale":"2023-10-16T13:47:40.909357+0000","last_undegraded":"2023-10-16T13:47:40.909357+0000","last_fullsized":"2023-10-16T13:47:40.909357+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:50:50.396650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.22","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.952670+0000","last_change":"2023-10-16T13:47:41.950769+0000","last_active":"2023-10-16T13:47:41.952670+0000","last_peered":"2023-10-16T13:47:41.952670+0000","last_clean":"2023-10-16T13:47:41.952670+0000","last_became_active":"2023-10-16T13:47:41.950103+0000","last_became_peered":"2023-10-16T13:47:41.950103+0000","last_unstale":"2023-10-16T13:47:41.952670+0000","last_undegraded":"2023-10-16T13:47:41.952670+0000","last_fullsized":"2023-10-16T13:47:41.952670+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:27:37.672780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.20","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.915340+0000","last_change":"2023-10-16T13:47:24.436480+0000","last_active":"2023-10-16T13:47:40.915340+0000","last_peered":"2023-10-16T13:47:40.915340+0000","last_clean":"2023-10-16T13:47:40.915340+0000","last_became_active":"2023-10-16T13:47:24.435309+0000","last_became_peered":"2023-10-16T13:47:24.435309+0000","last_unstale":"2023-10-16T13:47:40.915340+0000","last_undegraded":"2023-10-16T13:47:40.915340+0000","last_fullsized":"2023-10-16T13:47:40.915340+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:28:08.947506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1f","version":"244'4","reported_seq":19,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.908378+0000","last_change":"2023-10-16T13:47:23.425349+0000","last_active":"2023-10-16T13:47:40.908378+0000","last_peered":"2023-10-16T13:47:40.908378+0000","last_clean":"2023-10-16T13:47:40.908378+0000","last_became_active":"2023-10-16T13:47:23.425029+0000","last_became_peered":"2023-10-16T13:47:23.425029+0000","last_unstale":"2023-10-16T13:47:40.908378+0000","last_undegraded":"2023-10-16T13:47:40.908378+0000","last_fullsized":"2023-10-16T13:47:40.908378+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:57:35.834032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.20","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.956232+0000","last_change":"2023-10-16T13:47:41.948172+0000","last_active":"2023-10-16T13:47:41.956232+0000","last_peered":"2023-10-16T13:47:41.956232+0000","last_clean":"2023-10-16T13:47:41.956232+0000","last_became_active":"2023-10-16T13:47:41.947882+0000","last_became_peered":"2023-10-16T13:47:41.947882+0000","last_unstale":"2023-10-16T13:47:41.956232+0000","last_undegraded":"2023-10-16T13:47:41.956232+0000","last_fullsized":"2023-10-16T13:47:41.956232+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:54:20.421413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.22","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.916376+0000","last_change":"2023-10-16T13:47:24.431062+0000","last_active":"2023-10-16T13:47:40.916376+0000","last_peered":"2023-10-16T13:47:40.916376+0000","last_clean":"2023-10-16T13:47:40.916376+0000","last_became_active":"2023-10-16T13:47:24.430771+0000","last_became_peered":"2023-10-16T13:47:24.430771+0000","last_unstale":"2023-10-16T13:47:40.916376+0000","last_undegraded":"2023-10-16T13:47:40.916376+0000","last_fullsized":"2023-10-16T13:47:40.916376+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:23:48.085785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.1d","version":"244'2","reported_seq":17,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.919891+0000","last_change":"2023-10-16T13:47:23.428572+0000","last_active":"2023-10-16T13:47:40.919891+0000","last_peered":"2023-10-16T13:47:40.919891+0000","last_clean":"2023-10-16T13:47:40.919891+0000","last_became_active":"2023-10-16T13:47:23.428285+0000","last_became_peered":"2023-10-16T13:47:23.428285+0000","last_unstale":"2023-10-16T13:47:40.919891+0000","last_undegraded":"2023-10-16T13:47:40.919891+0000","last_fullsized":"2023-10-16T13:47:40.919891+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:26:03.650516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.21","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.960427+0000","last_change":"2023-10-16T13:47:41.953912+0000","last_active":"2023-10-16T13:47:41.960427+0000","last_peered":"2023-10-16T13:47:41.960427+0000","last_clean":"2023-10-16T13:47:41.960427+0000","last_became_active":"2023-10-16T13:47:41.953631+0000","last_became_peered":"2023-10-16T13:47:41.953631+0000","last_unstale":"2023-10-16T13:47:41.960427+0000","last_undegraded":"2023-10-16T13:47:41.960427+0000","last_fullsized":"2023-10-16T13:47:41.960427+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:43:37.671188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.23","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.907866+0000","last_change":"2023-10-16T13:47:24.432436+0000","last_active":"2023-10-16T13:47:40.907866+0000","last_peered":"2023-10-16T13:47:40.907866+0000","last_clean":"2023-10-16T13:47:40.907866+0000","last_became_active":"2023-10-16T13:47:24.428465+0000","last_became_peered":"2023-10-16T13:47:24.428465+0000","last_unstale":"2023-10-16T13:47:40.907866+0000","last_undegraded":"2023-10-16T13:47:40.907866+0000","last_fullsized":"2023-10-16T13:47:40.907866+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:16:50.560075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1c","version":"244'2","reported_seq":17,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.903280+0000","last_change":"2023-10-16T13:47:23.427686+0000","last_active":"2023-10-16T13:47:40.903280+0000","last_peered":"2023-10-16T13:47:40.903280+0000","last_clean":"2023-10-16T13:47:40.903280+0000","last_became_active":"2023-10-16T13:47:23.427013+0000","last_became_peered":"2023-10-16T13:47:23.427013+0000","last_unstale":"2023-10-16T13:47:40.903280+0000","last_undegraded":"2023-10-16T13:47:40.903280+0000","last_fullsized":"2023-10-16T13:47:40.903280+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:37:17.532164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.26","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.933821+0000","last_change":"2023-10-16T13:47:41.933043+0000","last_active":"2023-10-16T13:47:41.933821+0000","last_peered":"2023-10-16T13:47:41.933821+0000","last_clean":"2023-10-16T13:47:41.933821+0000","last_became_active":"2023-10-16T13:47:41.932784+0000","last_became_peered":"2023-10-16T13:47:41.932784+0000","last_unstale":"2023-10-16T13:47:41.933821+0000","last_undegraded":"2023-10-16T13:47:41.933821+0000","last_fullsized":"2023-10-16T13:47:41.933821+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:04:46.143278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.24","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.903182+0000","last_change":"2023-10-16T13:47:24.438644+0000","last_active":"2023-10-16T13:47:40.903182+0000","last_peered":"2023-10-16T13:47:40.903182+0000","last_clean":"2023-10-16T13:47:40.903182+0000","last_became_active":"2023-10-16T13:47:24.438344+0000","last_became_peered":"2023-10-16T13:47:24.438344+0000","last_unstale":"2023-10-16T13:47:40.903182+0000","last_undegraded":"2023-10-16T13:47:40.903182+0000","last_fullsized":"2023-10-16T13:47:40.903182+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:21:19.269920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.1b","version":"244'1","reported_seq":16,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.918079+0000","last_change":"2023-10-16T13:47:23.427234+0000","last_active":"2023-10-16T13:47:40.918079+0000","last_peered":"2023-10-16T13:47:40.918079+0000","last_clean":"2023-10-16T13:47:40.918079+0000","last_became_active":"2023-10-16T13:47:23.426948+0000","last_became_peered":"2023-10-16T13:47:23.426948+0000","last_unstale":"2023-10-16T13:47:40.918079+0000","last_undegraded":"2023-10-16T13:47:40.918079+0000","last_fullsized":"2023-10-16T13:47:40.918079+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:11:43.927649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.27","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.954896+0000","last_change":"2023-10-16T13:47:41.948872+0000","last_active":"2023-10-16T13:47:41.954896+0000","last_peered":"2023-10-16T13:47:41.954896+0000","last_clean":"2023-10-16T13:47:41.954896+0000","last_became_active":"2023-10-16T13:47:41.948548+0000","last_became_peered":"2023-10-16T13:47:41.948548+0000","last_unstale":"2023-10-16T13:47:41.954896+0000","last_undegraded":"2023-10-16T13:47:41.954896+0000","last_fullsized":"2023-10-16T13:47:41.954896+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:21:36.761315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.25","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.908796+0000","last_change":"2023-10-16T13:47:24.432803+0000","last_active":"2023-10-16T13:47:40.908796+0000","last_peered":"2023-10-16T13:47:40.908796+0000","last_clean":"2023-10-16T13:47:40.908796+0000","last_became_active":"2023-10-16T13:47:24.432470+0000","last_became_peered":"2023-10-16T13:47:24.432470+0000","last_unstale":"2023-10-16T13:47:40.908796+0000","last_undegraded":"2023-10-16T13:47:40.908796+0000","last_fullsized":"2023-10-16T13:47:40.908796+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:41:00.039237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.1a","version":"244'1","reported_seq":16,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.904098+0000","last_change":"2023-10-16T13:47:23.425695+0000","last_active":"2023-10-16T13:47:40.904098+0000","last_peered":"2023-10-16T13:47:40.904098+0000","last_clean":"2023-10-16T13:47:40.904098+0000","last_became_active":"2023-10-16T13:47:23.425407+0000","last_became_peered":"2023-10-16T13:47:23.425407+0000","last_unstale":"2023-10-16T13:47:40.904098+0000","last_undegraded":"2023-10-16T13:47:40.904098+0000","last_fullsized":"2023-10-16T13:47:40.904098+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:19:34.313200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.24","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.952382+0000","last_change":"2023-10-16T13:47:41.946625+0000","last_active":"2023-10-16T13:47:41.952382+0000","last_peered":"2023-10-16T13:47:41.952382+0000","last_clean":"2023-10-16T13:47:41.952382+0000","last_became_active":"2023-10-16T13:47:41.942547+0000","last_became_peered":"2023-10-16T13:47:41.942547+0000","last_unstale":"2023-10-16T13:47:41.952382+0000","last_undegraded":"2023-10-16T13:47:41.952382+0000","last_fullsized":"2023-10-16T13:47:41.952382+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:22:48.838787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.26","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.917522+0000","last_change":"2023-10-16T13:47:24.439616+0000","last_active":"2023-10-16T13:47:40.917522+0000","last_peered":"2023-10-16T13:47:40.917522+0000","last_clean":"2023-10-16T13:47:40.917522+0000","last_became_active":"2023-10-16T13:47:24.435678+0000","last_became_peered":"2023-10-16T13:47:24.435678+0000","last_unstale":"2023-10-16T13:47:40.917522+0000","last_undegraded":"2023-10-16T13:47:40.917522+0000","last_fullsized":"2023-10-16T13:47:40.917522+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:39:16.296666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.912378+0000","last_change":"2023-10-16T13:47:23.431681+0000","last_active":"2023-10-16T13:47:40.912378+0000","last_peered":"2023-10-16T13:47:40.912378+0000","last_clean":"2023-10-16T13:47:40.912378+0000","last_became_active":"2023-10-16T13:47:23.431317+0000","last_became_peered":"2023-10-16T13:47:23.431317+0000","last_unstale":"2023-10-16T13:47:40.912378+0000","last_undegraded":"2023-10-16T13:47:40.912378+0000","last_fullsized":"2023-10-16T13:47:40.912378+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:20:27.852907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.25","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.956367+0000","last_change":"2023-10-16T13:47:41.954814+0000","last_active":"2023-10-16T13:47:41.956367+0000","last_peered":"2023-10-16T13:47:41.956367+0000","last_clean":"2023-10-16T13:47:41.956367+0000","last_became_active":"2023-10-16T13:47:41.954529+0000","last_became_peered":"2023-10-16T13:47:41.954529+0000","last_unstale":"2023-10-16T13:47:41.956367+0000","last_undegraded":"2023-10-16T13:47:41.956367+0000","last_fullsized":"2023-10-16T13:47:41.956367+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:09:36.582479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.27","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.909914+0000","last_change":"2023-10-16T13:47:24.423771+0000","last_active":"2023-10-16T13:47:40.909914+0000","last_peered":"2023-10-16T13:47:40.909914+0000","last_clean":"2023-10-16T13:47:40.909914+0000","last_became_active":"2023-10-16T13:47:24.423459+0000","last_became_peered":"2023-10-16T13:47:24.423459+0000","last_unstale":"2023-10-16T13:47:40.909914+0000","last_undegraded":"2023-10-16T13:47:40.909914+0000","last_fullsized":"2023-10-16T13:47:40.909914+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:27:22.393558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.909463+0000","last_change":"2023-10-16T13:47:23.433310+0000","last_active":"2023-10-16T13:47:40.909463+0000","last_peered":"2023-10-16T13:47:40.909463+0000","last_clean":"2023-10-16T13:47:40.909463+0000","last_became_active":"2023-10-16T13:47:23.433018+0000","last_became_peered":"2023-10-16T13:47:23.433018+0000","last_unstale":"2023-10-16T13:47:40.909463+0000","last_undegraded":"2023-10-16T13:47:40.909463+0000","last_fullsized":"2023-10-16T13:47:40.909463+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:49:45.992161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.2a","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.940455+0000","last_change":"2023-10-16T13:47:41.939916+0000","last_active":"2023-10-16T13:47:41.940455+0000","last_peered":"2023-10-16T13:47:41.940455+0000","last_clean":"2023-10-16T13:47:41.940455+0000","last_became_active":"2023-10-16T13:47:41.939601+0000","last_became_peered":"2023-10-16T13:47:41.939601+0000","last_unstale":"2023-10-16T13:47:41.940455+0000","last_undegraded":"2023-10-16T13:47:41.940455+0000","last_fullsized":"2023-10-16T13:47:41.940455+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:28:15.447879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.28","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.906743+0000","last_change":"2023-10-16T13:47:24.433316+0000","last_active":"2023-10-16T13:47:40.906743+0000","last_peered":"2023-10-16T13:47:40.906743+0000","last_clean":"2023-10-16T13:47:40.906743+0000","last_became_active":"2023-10-16T13:47:24.429160+0000","last_became_peered":"2023-10-16T13:47:24.429160+0000","last_unstale":"2023-10-16T13:47:40.906743+0000","last_undegraded":"2023-10-16T13:47:40.906743+0000","last_fullsized":"2023-10-16T13:47:40.906743+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:12:35.760185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.904819+0000","last_change":"2023-10-16T13:47:23.428721+0000","last_active":"2023-10-16T13:47:40.904819+0000","last_peered":"2023-10-16T13:47:40.904819+0000","last_clean":"2023-10-16T13:47:40.904819+0000","last_became_active":"2023-10-16T13:47:23.428349+0000","last_became_peered":"2023-10-16T13:47:23.428349+0000","last_unstale":"2023-10-16T13:47:40.904819+0000","last_undegraded":"2023-10-16T13:47:40.904819+0000","last_fullsized":"2023-10-16T13:47:40.904819+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:59:11.116638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.2b","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.959655+0000","last_change":"2023-10-16T13:47:41.958608+0000","last_active":"2023-10-16T13:47:41.959655+0000","last_peered":"2023-10-16T13:47:41.959655+0000","last_clean":"2023-10-16T13:47:41.959655+0000","last_became_active":"2023-10-16T13:47:41.958334+0000","last_became_peered":"2023-10-16T13:47:41.958334+0000","last_unstale":"2023-10-16T13:47:41.959655+0000","last_undegraded":"2023-10-16T13:47:41.959655+0000","last_fullsized":"2023-10-16T13:47:41.959655+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:57:54.329788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.29","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.910632+0000","last_change":"2023-10-16T13:47:24.440635+0000","last_active":"2023-10-16T13:47:40.910632+0000","last_peered":"2023-10-16T13:47:40.910632+0000","last_clean":"2023-10-16T13:47:40.910632+0000","last_became_active":"2023-10-16T13:47:24.440223+0000","last_became_peered":"2023-10-16T13:47:24.440223+0000","last_unstale":"2023-10-16T13:47:40.910632+0000","last_undegraded":"2023-10-16T13:47:40.910632+0000","last_fullsized":"2023-10-16T13:47:40.910632+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:22:53.895897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.917821+0000","last_change":"2023-10-16T13:47:23.433545+0000","last_active":"2023-10-16T13:47:40.917821+0000","last_peered":"2023-10-16T13:47:40.917821+0000","last_clean":"2023-10-16T13:47:40.917821+0000","last_became_active":"2023-10-16T13:47:23.433275+0000","last_became_peered":"2023-10-16T13:47:23.433275+0000","last_unstale":"2023-10-16T13:47:40.917821+0000","last_undegraded":"2023-10-16T13:47:40.917821+0000","last_fullsized":"2023-10-16T13:47:40.917821+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:21:17.525717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.28","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.952753+0000","last_change":"2023-10-16T13:47:41.948368+0000","last_active":"2023-10-16T13:47:41.952753+0000","last_peered":"2023-10-16T13:47:41.952753+0000","last_clean":"2023-10-16T13:47:41.952753+0000","last_became_active":"2023-10-16T13:47:41.948097+0000","last_became_peered":"2023-10-16T13:47:41.948097+0000","last_unstale":"2023-10-16T13:47:41.952753+0000","last_undegraded":"2023-10-16T13:47:41.952753+0000","last_fullsized":"2023-10-16T13:47:41.952753+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:32:31.777807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.2a","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.912923+0000","last_change":"2023-10-16T13:47:24.424605+0000","last_active":"2023-10-16T13:47:40.912923+0000","last_peered":"2023-10-16T13:47:40.912923+0000","last_clean":"2023-10-16T13:47:40.912923+0000","last_became_active":"2023-10-16T13:47:24.424313+0000","last_became_peered":"2023-10-16T13:47:24.424313+0000","last_unstale":"2023-10-16T13:47:40.912923+0000","last_undegraded":"2023-10-16T13:47:40.912923+0000","last_fullsized":"2023-10-16T13:47:40.912923+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:40:33.368782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.15","version":"244'3","reported_seq":18,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.907932+0000","last_change":"2023-10-16T13:47:23.431025+0000","last_active":"2023-10-16T13:47:40.907932+0000","last_peered":"2023-10-16T13:47:40.907932+0000","last_clean":"2023-10-16T13:47:40.907932+0000","last_became_active":"2023-10-16T13:47:23.430710+0000","last_became_peered":"2023-10-16T13:47:23.430710+0000","last_unstale":"2023-10-16T13:47:40.907932+0000","last_undegraded":"2023-10-16T13:47:40.907932+0000","last_fullsized":"2023-10-16T13:47:40.907932+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:52:27.207432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.29","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.940690+0000","last_change":"2023-10-16T13:47:41.937466+0000","last_active":"2023-10-16T13:47:41.940690+0000","last_peered":"2023-10-16T13:47:41.940690+0000","last_clean":"2023-10-16T13:47:41.940690+0000","last_became_active":"2023-10-16T13:47:41.937116+0000","last_became_peered":"2023-10-16T13:47:41.937116+0000","last_unstale":"2023-10-16T13:47:41.940690+0000","last_undegraded":"2023-10-16T13:47:41.940690+0000","last_fullsized":"2023-10-16T13:47:41.940690+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:44:12.544942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.2b","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.908612+0000","last_change":"2023-10-16T13:47:24.427810+0000","last_active":"2023-10-16T13:47:40.908612+0000","last_peered":"2023-10-16T13:47:40.908612+0000","last_clean":"2023-10-16T13:47:40.908612+0000","last_became_active":"2023-10-16T13:47:24.427276+0000","last_became_peered":"2023-10-16T13:47:24.427276+0000","last_unstale":"2023-10-16T13:47:40.908612+0000","last_undegraded":"2023-10-16T13:47:40.908612+0000","last_fullsized":"2023-10-16T13:47:40.908612+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:16:12.551854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.14","version":"245'9","reported_seq":24,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.908096+0000","last_change":"2023-10-16T13:47:23.430528+0000","last_active":"2023-10-16T13:47:40.908096+0000","last_peered":"2023-10-16T13:47:40.908096+0000","last_clean":"2023-10-16T13:47:40.908096+0000","last_became_active":"2023-10-16T13:47:23.429721+0000","last_became_peered":"2023-10-16T13:47:23.429721+0000","last_unstale":"2023-10-16T13:47:40.908096+0000","last_undegraded":"2023-10-16T13:47:40.908096+0000","last_fullsized":"2023-10-16T13:47:40.908096+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:07:56.379842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2859,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.2e","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.957806+0000","last_change":"2023-10-16T13:47:41.954884+0000","last_active":"2023-10-16T13:47:41.957806+0000","last_peered":"2023-10-16T13:47:41.957806+0000","last_clean":"2023-10-16T13:47:41.957806+0000","last_became_active":"2023-10-16T13:47:41.954332+0000","last_became_peered":"2023-10-16T13:47:41.954332+0000","last_unstale":"2023-10-16T13:47:41.957806+0000","last_undegraded":"2023-10-16T13:47:41.957806+0000","last_fullsized":"2023-10-16T13:47:41.957806+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:06:03.891766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.2c","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.919047+0000","last_change":"2023-10-16T13:47:24.436132+0000","last_active":"2023-10-16T13:47:40.919047+0000","last_peered":"2023-10-16T13:47:40.919047+0000","last_clean":"2023-10-16T13:47:40.919047+0000","last_became_active":"2023-10-16T13:47:24.435272+0000","last_became_peered":"2023-10-16T13:47:24.435272+0000","last_unstale":"2023-10-16T13:47:40.919047+0000","last_undegraded":"2023-10-16T13:47:40.919047+0000","last_fullsized":"2023-10-16T13:47:40.919047+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:07:59.434149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.13","version":"244'2","reported_seq":17,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.918177+0000","last_change":"2023-10-16T13:47:23.426620+0000","last_active":"2023-10-16T13:47:40.918177+0000","last_peered":"2023-10-16T13:47:40.918177+0000","last_clean":"2023-10-16T13:47:40.918177+0000","last_became_active":"2023-10-16T13:47:23.426328+0000","last_became_peered":"2023-10-16T13:47:23.426328+0000","last_unstale":"2023-10-16T13:47:40.918177+0000","last_undegraded":"2023-10-16T13:47:40.918177+0000","last_fullsized":"2023-10-16T13:47:40.918177+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:23:39.805153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.2f","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.954728+0000","last_change":"2023-10-16T13:47:41.949649+0000","last_active":"2023-10-16T13:47:41.954728+0000","last_peered":"2023-10-16T13:47:41.954728+0000","last_clean":"2023-10-16T13:47:41.954728+0000","last_became_active":"2023-10-16T13:47:41.949363+0000","last_became_peered":"2023-10-16T13:47:41.949363+0000","last_unstale":"2023-10-16T13:47:41.954728+0000","last_undegraded":"2023-10-16T13:47:41.954728+0000","last_fullsized":"2023-10-16T13:47:41.954728+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:57:15.234272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2d","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.908823+0000","last_change":"2023-10-16T13:47:24.431734+0000","last_active":"2023-10-16T13:47:40.908823+0000","last_peered":"2023-10-16T13:47:40.908823+0000","last_clean":"2023-10-16T13:47:40.908823+0000","last_became_active":"2023-10-16T13:47:24.431414+0000","last_became_peered":"2023-10-16T13:47:24.431414+0000","last_unstale":"2023-10-16T13:47:40.908823+0000","last_undegraded":"2023-10-16T13:47:40.908823+0000","last_fullsized":"2023-10-16T13:47:40.908823+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:01:31.790432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.906560+0000","last_change":"2023-10-16T13:47:23.433222+0000","last_active":"2023-10-16T13:47:40.906560+0000","last_peered":"2023-10-16T13:47:40.906560+0000","last_clean":"2023-10-16T13:47:40.906560+0000","last_became_active":"2023-10-16T13:47:23.432936+0000","last_became_peered":"2023-10-16T13:47:23.432936+0000","last_unstale":"2023-10-16T13:47:40.906560+0000","last_undegraded":"2023-10-16T13:47:40.906560+0000","last_fullsized":"2023-10-16T13:47:40.906560+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:47:59.336633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.2c","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.953890+0000","last_change":"2023-10-16T13:47:41.948335+0000","last_active":"2023-10-16T13:47:41.953890+0000","last_peered":"2023-10-16T13:47:41.953890+0000","last_clean":"2023-10-16T13:47:41.953890+0000","last_became_active":"2023-10-16T13:47:41.947856+0000","last_became_peered":"2023-10-16T13:47:41.947856+0000","last_unstale":"2023-10-16T13:47:41.953890+0000","last_undegraded":"2023-10-16T13:47:41.953890+0000","last_fullsized":"2023-10-16T13:47:41.953890+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:21:14.307049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.2e","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.917851+0000","last_change":"2023-10-16T13:47:24.438281+0000","last_active":"2023-10-16T13:47:40.917851+0000","last_peered":"2023-10-16T13:47:40.917851+0000","last_clean":"2023-10-16T13:47:40.917851+0000","last_became_active":"2023-10-16T13:47:24.437976+0000","last_became_peered":"2023-10-16T13:47:24.437976+0000","last_unstale":"2023-10-16T13:47:40.917851+0000","last_undegraded":"2023-10-16T13:47:40.917851+0000","last_fullsized":"2023-10-16T13:47:40.917851+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:19:38.293171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.905879+0000","last_change":"2023-10-16T13:47:23.425872+0000","last_active":"2023-10-16T13:47:40.905879+0000","last_peered":"2023-10-16T13:47:40.905879+0000","last_clean":"2023-10-16T13:47:40.905879+0000","last_became_active":"2023-10-16T13:47:23.425569+0000","last_became_peered":"2023-10-16T13:47:23.425569+0000","last_unstale":"2023-10-16T13:47:40.905879+0000","last_undegraded":"2023-10-16T13:47:40.905879+0000","last_fullsized":"2023-10-16T13:47:40.905879+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:09:18.086723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.2d","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.954148+0000","last_change":"2023-10-16T13:47:41.947743+0000","last_active":"2023-10-16T13:47:41.954148+0000","last_peered":"2023-10-16T13:47:41.954148+0000","last_clean":"2023-10-16T13:47:41.954148+0000","last_became_active":"2023-10-16T13:47:41.947502+0000","last_became_peered":"2023-10-16T13:47:41.947502+0000","last_unstale":"2023-10-16T13:47:41.954148+0000","last_undegraded":"2023-10-16T13:47:41.954148+0000","last_fullsized":"2023-10-16T13:47:41.954148+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:23:25.484030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2f","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.909795+0000","last_change":"2023-10-16T13:47:24.437030+0000","last_active":"2023-10-16T13:47:40.909795+0000","last_peered":"2023-10-16T13:47:40.909795+0000","last_clean":"2023-10-16T13:47:40.909795+0000","last_became_active":"2023-10-16T13:47:24.436734+0000","last_became_peered":"2023-10-16T13:47:24.436734+0000","last_unstale":"2023-10-16T13:47:40.909795+0000","last_undegraded":"2023-10-16T13:47:40.909795+0000","last_fullsized":"2023-10-16T13:47:40.909795+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:28:53.363293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.10","version":"244'8","reported_seq":23,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.915239+0000","last_change":"2023-10-16T13:47:23.426967+0000","last_active":"2023-10-16T13:47:40.915239+0000","last_peered":"2023-10-16T13:47:40.915239+0000","last_clean":"2023-10-16T13:47:40.915239+0000","last_became_active":"2023-10-16T13:47:23.426614+0000","last_became_peered":"2023-10-16T13:47:23.426614+0000","last_unstale":"2023-10-16T13:47:40.915239+0000","last_undegraded":"2023-10-16T13:47:40.915239+0000","last_fullsized":"2023-10-16T13:47:40.915239+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:39:12.211841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.32","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.960379+0000","last_change":"2023-10-16T13:47:41.953476+0000","last_active":"2023-10-16T13:47:41.960379+0000","last_peered":"2023-10-16T13:47:41.960379+0000","last_clean":"2023-10-16T13:47:41.960379+0000","last_became_active":"2023-10-16T13:47:41.953194+0000","last_became_peered":"2023-10-16T13:47:41.953194+0000","last_unstale":"2023-10-16T13:47:41.960379+0000","last_undegraded":"2023-10-16T13:47:41.960379+0000","last_fullsized":"2023-10-16T13:47:41.960379+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:20:24.501093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.30","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.907242+0000","last_change":"2023-10-16T13:47:24.437081+0000","last_active":"2023-10-16T13:47:40.907242+0000","last_peered":"2023-10-16T13:47:40.907242+0000","last_clean":"2023-10-16T13:47:40.907242+0000","last_became_active":"2023-10-16T13:47:24.436501+0000","last_became_peered":"2023-10-16T13:47:24.436501+0000","last_unstale":"2023-10-16T13:47:40.907242+0000","last_undegraded":"2023-10-16T13:47:40.907242+0000","last_fullsized":"2023-10-16T13:47:40.907242+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:23:30.831964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.f","version":"244'1","reported_seq":16,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.908273+0000","last_change":"2023-10-16T13:47:23.432161+0000","last_active":"2023-10-16T13:47:40.908273+0000","last_peered":"2023-10-16T13:47:40.908273+0000","last_clean":"2023-10-16T13:47:40.908273+0000","last_became_active":"2023-10-16T13:47:23.431822+0000","last_became_peered":"2023-10-16T13:47:23.431822+0000","last_unstale":"2023-10-16T13:47:40.908273+0000","last_undegraded":"2023-10-16T13:47:40.908273+0000","last_fullsized":"2023-10-16T13:47:40.908273+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:15:50.867051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.33","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.954838+0000","last_change":"2023-10-16T13:47:41.946832+0000","last_active":"2023-10-16T13:47:41.954838+0000","last_peered":"2023-10-16T13:47:41.954838+0000","last_clean":"2023-10-16T13:47:41.954838+0000","last_became_active":"2023-10-16T13:47:41.944213+0000","last_became_peered":"2023-10-16T13:47:41.944213+0000","last_unstale":"2023-10-16T13:47:41.954838+0000","last_undegraded":"2023-10-16T13:47:41.954838+0000","last_fullsized":"2023-10-16T13:47:41.954838+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:59:46.409384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.31","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.905467+0000","last_change":"2023-10-16T13:47:24.431186+0000","last_active":"2023-10-16T13:47:40.905467+0000","last_peered":"2023-10-16T13:47:40.905467+0000","last_clean":"2023-10-16T13:47:40.905467+0000","last_became_active":"2023-10-16T13:47:24.430892+0000","last_became_peered":"2023-10-16T13:47:24.430892+0000","last_unstale":"2023-10-16T13:47:40.905467+0000","last_undegraded":"2023-10-16T13:47:40.905467+0000","last_fullsized":"2023-10-16T13:47:40.905467+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:12:21.693658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.e","version":"244'2","reported_seq":17,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.918964+0000","last_change":"2023-10-16T13:47:23.424549+0000","last_active":"2023-10-16T13:47:40.918964+0000","last_peered":"2023-10-16T13:47:40.918964+0000","last_clean":"2023-10-16T13:47:40.918964+0000","last_became_active":"2023-10-16T13:47:23.424111+0000","last_became_peered":"2023-10-16T13:47:23.424111+0000","last_unstale":"2023-10-16T13:47:40.918964+0000","last_undegraded":"2023-10-16T13:47:40.918964+0000","last_fullsized":"2023-10-16T13:47:40.918964+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:36:35.137200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.30","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.939840+0000","last_change":"2023-10-16T13:47:41.938201+0000","last_active":"2023-10-16T13:47:41.939840+0000","last_peered":"2023-10-16T13:47:41.939840+0000","last_clean":"2023-10-16T13:47:41.939840+0000","last_became_active":"2023-10-16T13:47:41.937814+0000","last_became_peered":"2023-10-16T13:47:41.937814+0000","last_unstale":"2023-10-16T13:47:41.939840+0000","last_undegraded":"2023-10-16T13:47:41.939840+0000","last_fullsized":"2023-10-16T13:47:41.939840+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:48:11.978315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.32","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.918418+0000","last_change":"2023-10-16T13:47:24.432940+0000","last_active":"2023-10-16T13:47:40.918418+0000","last_peered":"2023-10-16T13:47:40.918418+0000","last_clean":"2023-10-16T13:47:40.918418+0000","last_became_active":"2023-10-16T13:47:24.432610+0000","last_became_peered":"2023-10-16T13:47:24.432610+0000","last_unstale":"2023-10-16T13:47:40.918418+0000","last_undegraded":"2023-10-16T13:47:40.918418+0000","last_fullsized":"2023-10-16T13:47:40.918418+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:59:08.104923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.d","version":"244'1","reported_seq":16,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.920060+0000","last_change":"2023-10-16T13:47:23.428449+0000","last_active":"2023-10-16T13:47:40.920060+0000","last_peered":"2023-10-16T13:47:40.920060+0000","last_clean":"2023-10-16T13:47:40.920060+0000","last_became_active":"2023-10-16T13:47:23.428130+0000","last_became_peered":"2023-10-16T13:47:23.428130+0000","last_unstale":"2023-10-16T13:47:40.920060+0000","last_undegraded":"2023-10-16T13:47:40.920060+0000","last_fullsized":"2023-10-16T13:47:40.920060+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:24:27.601821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.31","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.961246+0000","last_change":"2023-10-16T13:47:41.954571+0000","last_active":"2023-10-16T13:47:41.961246+0000","last_peered":"2023-10-16T13:47:41.961246+0000","last_clean":"2023-10-16T13:47:41.961246+0000","last_became_active":"2023-10-16T13:47:41.954283+0000","last_became_peered":"2023-10-16T13:47:41.954283+0000","last_unstale":"2023-10-16T13:47:41.961246+0000","last_undegraded":"2023-10-16T13:47:41.961246+0000","last_fullsized":"2023-10-16T13:47:41.961246+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:46:52.727625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.33","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.915842+0000","last_change":"2023-10-16T13:47:24.427239+0000","last_active":"2023-10-16T13:47:40.915842+0000","last_peered":"2023-10-16T13:47:40.915842+0000","last_clean":"2023-10-16T13:47:40.915842+0000","last_became_active":"2023-10-16T13:47:24.426919+0000","last_became_peered":"2023-10-16T13:47:24.426919+0000","last_unstale":"2023-10-16T13:47:40.915842+0000","last_undegraded":"2023-10-16T13:47:40.915842+0000","last_fullsized":"2023-10-16T13:47:40.915842+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:56:16.580840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.916522+0000","last_change":"2023-10-16T13:47:23.427051+0000","last_active":"2023-10-16T13:47:40.916522+0000","last_peered":"2023-10-16T13:47:40.916522+0000","last_clean":"2023-10-16T13:47:40.916522+0000","last_became_active":"2023-10-16T13:47:23.426741+0000","last_became_peered":"2023-10-16T13:47:23.426741+0000","last_unstale":"2023-10-16T13:47:40.916522+0000","last_undegraded":"2023-10-16T13:47:40.916522+0000","last_fullsized":"2023-10-16T13:47:40.916522+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:37:17.532164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.36","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.954049+0000","last_change":"2023-10-16T13:47:41.946845+0000","last_active":"2023-10-16T13:47:41.954049+0000","last_peered":"2023-10-16T13:47:41.954049+0000","last_clean":"2023-10-16T13:47:41.954049+0000","last_became_active":"2023-10-16T13:47:41.946499+0000","last_became_peered":"2023-10-16T13:47:41.946499+0000","last_unstale":"2023-10-16T13:47:41.954049+0000","last_undegraded":"2023-10-16T13:47:41.954049+0000","last_fullsized":"2023-10-16T13:47:41.954049+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:45:21.170147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.34","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.917342+0000","last_change":"2023-10-16T13:47:24.444364+0000","last_active":"2023-10-16T13:47:40.917342+0000","last_peered":"2023-10-16T13:47:40.917342+0000","last_clean":"2023-10-16T13:47:40.917342+0000","last_became_active":"2023-10-16T13:47:24.444045+0000","last_became_peered":"2023-10-16T13:47:24.444045+0000","last_unstale":"2023-10-16T13:47:40.917342+0000","last_undegraded":"2023-10-16T13:47:40.917342+0000","last_fullsized":"2023-10-16T13:47:40.917342+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:13:11.795921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.b","version":"244'2","reported_seq":17,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.915739+0000","last_change":"2023-10-16T13:47:23.434957+0000","last_active":"2023-10-16T13:47:40.915739+0000","last_peered":"2023-10-16T13:47:40.915739+0000","last_clean":"2023-10-16T13:47:40.915739+0000","last_became_active":"2023-10-16T13:47:23.434637+0000","last_became_peered":"2023-10-16T13:47:23.434637+0000","last_unstale":"2023-10-16T13:47:40.915739+0000","last_undegraded":"2023-10-16T13:47:40.915739+0000","last_fullsized":"2023-10-16T13:47:40.915739+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:23:47.081423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.37","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.950618+0000","last_change":"2023-10-16T13:47:41.946099+0000","last_active":"2023-10-16T13:47:41.950618+0000","last_peered":"2023-10-16T13:47:41.950618+0000","last_clean":"2023-10-16T13:47:41.950618+0000","last_became_active":"2023-10-16T13:47:41.945829+0000","last_became_peered":"2023-10-16T13:47:41.945829+0000","last_unstale":"2023-10-16T13:47:41.950618+0000","last_undegraded":"2023-10-16T13:47:41.950618+0000","last_fullsized":"2023-10-16T13:47:41.950618+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:33:03.023121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.35","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.918818+0000","last_change":"2023-10-16T13:47:24.431892+0000","last_active":"2023-10-16T13:47:40.918818+0000","last_peered":"2023-10-16T13:47:40.918818+0000","last_clean":"2023-10-16T13:47:40.918818+0000","last_became_active":"2023-10-16T13:47:24.431624+0000","last_became_peered":"2023-10-16T13:47:24.431624+0000","last_unstale":"2023-10-16T13:47:40.918818+0000","last_undegraded":"2023-10-16T13:47:40.918818+0000","last_fullsized":"2023-10-16T13:47:40.918818+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:49:13.422163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.911855+0000","last_change":"2023-10-16T13:47:23.435342+0000","last_active":"2023-10-16T13:47:40.911855+0000","last_peered":"2023-10-16T13:47:40.911855+0000","last_clean":"2023-10-16T13:47:40.911855+0000","last_became_active":"2023-10-16T13:47:23.434999+0000","last_became_peered":"2023-10-16T13:47:23.434999+0000","last_unstale":"2023-10-16T13:47:40.911855+0000","last_undegraded":"2023-10-16T13:47:40.911855+0000","last_fullsized":"2023-10-16T13:47:40.911855+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:20:03.252858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.34","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.942822+0000","last_change":"2023-10-16T13:47:41.941663+0000","last_active":"2023-10-16T13:47:41.942822+0000","last_peered":"2023-10-16T13:47:41.942822+0000","last_clean":"2023-10-16T13:47:41.942822+0000","last_became_active":"2023-10-16T13:47:41.941198+0000","last_became_peered":"2023-10-16T13:47:41.941198+0000","last_unstale":"2023-10-16T13:47:41.942822+0000","last_undegraded":"2023-10-16T13:47:41.942822+0000","last_fullsized":"2023-10-16T13:47:41.942822+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:28:20.482489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.36","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.921197+0000","last_change":"2023-10-16T13:47:24.435751+0000","last_active":"2023-10-16T13:47:40.921197+0000","last_peered":"2023-10-16T13:47:40.921197+0000","last_clean":"2023-10-16T13:47:40.921197+0000","last_became_active":"2023-10-16T13:47:24.434491+0000","last_became_peered":"2023-10-16T13:47:24.434491+0000","last_unstale":"2023-10-16T13:47:40.921197+0000","last_undegraded":"2023-10-16T13:47:40.921197+0000","last_fullsized":"2023-10-16T13:47:40.921197+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:02:56.979689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.910768+0000","last_change":"2023-10-16T13:47:23.427022+0000","last_active":"2023-10-16T13:47:40.910768+0000","last_peered":"2023-10-16T13:47:40.910768+0000","last_clean":"2023-10-16T13:47:40.910768+0000","last_became_active":"2023-10-16T13:47:23.426682+0000","last_became_peered":"2023-10-16T13:47:23.426682+0000","last_unstale":"2023-10-16T13:47:40.910768+0000","last_undegraded":"2023-10-16T13:47:40.910768+0000","last_fullsized":"2023-10-16T13:47:40.910768+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:05:08.936096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.35","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.952350+0000","last_change":"2023-10-16T13:47:41.945372+0000","last_active":"2023-10-16T13:47:41.952350+0000","last_peered":"2023-10-16T13:47:41.952350+0000","last_clean":"2023-10-16T13:47:41.952350+0000","last_became_active":"2023-10-16T13:47:41.945069+0000","last_became_peered":"2023-10-16T13:47:41.945069+0000","last_unstale":"2023-10-16T13:47:41.952350+0000","last_undegraded":"2023-10-16T13:47:41.952350+0000","last_fullsized":"2023-10-16T13:47:41.952350+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:40:35.374297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.37","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.920489+0000","last_change":"2023-10-16T13:47:24.436357+0000","last_active":"2023-10-16T13:47:40.920489+0000","last_peered":"2023-10-16T13:47:40.920489+0000","last_clean":"2023-10-16T13:47:40.920489+0000","last_became_active":"2023-10-16T13:47:24.436050+0000","last_became_peered":"2023-10-16T13:47:24.436050+0000","last_unstale":"2023-10-16T13:47:40.920489+0000","last_undegraded":"2023-10-16T13:47:40.920489+0000","last_fullsized":"2023-10-16T13:47:40.920489+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:15:49.933818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.901663+0000","last_change":"2023-10-16T13:47:23.433223+0000","last_active":"2023-10-16T13:47:40.901663+0000","last_peered":"2023-10-16T13:47:40.901663+0000","last_clean":"2023-10-16T13:47:40.901663+0000","last_became_active":"2023-10-16T13:47:23.432935+0000","last_became_peered":"2023-10-16T13:47:23.432935+0000","last_unstale":"2023-10-16T13:47:40.901663+0000","last_undegraded":"2023-10-16T13:47:40.901663+0000","last_fullsized":"2023-10-16T13:47:40.901663+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:40:59.034875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.3a","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.952689+0000","last_change":"2023-10-16T13:47:41.950978+0000","last_active":"2023-10-16T13:47:41.952689+0000","last_peered":"2023-10-16T13:47:41.952689+0000","last_clean":"2023-10-16T13:47:41.952689+0000","last_became_active":"2023-10-16T13:47:41.950480+0000","last_became_peered":"2023-10-16T13:47:41.950480+0000","last_unstale":"2023-10-16T13:47:41.952689+0000","last_undegraded":"2023-10-16T13:47:41.952689+0000","last_fullsized":"2023-10-16T13:47:41.952689+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:23:28.072263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.38","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.906642+0000","last_change":"2023-10-16T13:47:24.437795+0000","last_active":"2023-10-16T13:47:40.906642+0000","last_peered":"2023-10-16T13:47:40.906642+0000","last_clean":"2023-10-16T13:47:40.906642+0000","last_became_active":"2023-10-16T13:47:24.437493+0000","last_became_peered":"2023-10-16T13:47:24.437493+0000","last_unstale":"2023-10-16T13:47:40.906642+0000","last_undegraded":"2023-10-16T13:47:40.906642+0000","last_fullsized":"2023-10-16T13:47:40.906642+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:32:36.771595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.7","version":"244'2","reported_seq":17,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.916978+0000","last_change":"2023-10-16T13:47:23.430865+0000","last_active":"2023-10-16T13:47:40.916978+0000","last_peered":"2023-10-16T13:47:40.916978+0000","last_clean":"2023-10-16T13:47:40.916978+0000","last_became_active":"2023-10-16T13:47:23.430493+0000","last_became_peered":"2023-10-16T13:47:23.430493+0000","last_unstale":"2023-10-16T13:47:40.916978+0000","last_undegraded":"2023-10-16T13:47:40.916978+0000","last_fullsized":"2023-10-16T13:47:40.916978+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:59:14.057546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.3b","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.959084+0000","last_change":"2023-10-16T13:47:41.953921+0000","last_active":"2023-10-16T13:47:41.959084+0000","last_peered":"2023-10-16T13:47:41.959084+0000","last_clean":"2023-10-16T13:47:41.959084+0000","last_became_active":"2023-10-16T13:47:41.953626+0000","last_became_peered":"2023-10-16T13:47:41.953626+0000","last_unstale":"2023-10-16T13:47:41.959084+0000","last_undegraded":"2023-10-16T13:47:41.959084+0000","last_fullsized":"2023-10-16T13:47:41.959084+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:33:32.802746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.39","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.916543+0000","last_change":"2023-10-16T13:47:24.443917+0000","last_active":"2023-10-16T13:47:40.916543+0000","last_peered":"2023-10-16T13:47:40.916543+0000","last_clean":"2023-10-16T13:47:40.916543+0000","last_became_active":"2023-10-16T13:47:24.443547+0000","last_became_peered":"2023-10-16T13:47:24.443547+0000","last_unstale":"2023-10-16T13:47:40.916543+0000","last_undegraded":"2023-10-16T13:47:40.916543+0000","last_fullsized":"2023-10-16T13:47:40.916543+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:21:18.530078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.6","version":"245'1","reported_seq":16,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.908521+0000","last_change":"2023-10-16T13:47:23.430851+0000","last_active":"2023-10-16T13:47:40.908521+0000","last_peered":"2023-10-16T13:47:40.908521+0000","last_clean":"2023-10-16T13:47:40.908521+0000","last_became_active":"2023-10-16T13:47:23.430526+0000","last_became_peered":"2023-10-16T13:47:23.430526+0000","last_unstale":"2023-10-16T13:47:40.908521+0000","last_undegraded":"2023-10-16T13:47:40.908521+0000","last_fullsized":"2023-10-16T13:47:40.908521+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:40:16.878541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.38","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.960492+0000","last_change":"2023-10-16T13:47:41.956508+0000","last_active":"2023-10-16T13:47:41.960492+0000","last_peered":"2023-10-16T13:47:41.960492+0000","last_clean":"2023-10-16T13:47:41.960492+0000","last_became_active":"2023-10-16T13:47:41.956242+0000","last_became_peered":"2023-10-16T13:47:41.956242+0000","last_unstale":"2023-10-16T13:47:41.960492+0000","last_undegraded":"2023-10-16T13:47:41.960492+0000","last_fullsized":"2023-10-16T13:47:41.960492+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:10:25.581028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.3a","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.912818+0000","last_change":"2023-10-16T13:47:24.440347+0000","last_active":"2023-10-16T13:47:40.912818+0000","last_peered":"2023-10-16T13:47:40.912818+0000","last_clean":"2023-10-16T13:47:40.912818+0000","last_became_active":"2023-10-16T13:47:24.440041+0000","last_became_peered":"2023-10-16T13:47:24.440041+0000","last_unstale":"2023-10-16T13:47:40.912818+0000","last_undegraded":"2023-10-16T13:47:40.912818+0000","last_fullsized":"2023-10-16T13:47:40.912818+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:19:38.293171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.902848+0000","last_change":"2023-10-16T13:47:23.427399+0000","last_active":"2023-10-16T13:47:40.902848+0000","last_peered":"2023-10-16T13:47:40.902848+0000","last_clean":"2023-10-16T13:47:40.902848+0000","last_became_active":"2023-10-16T13:47:23.427092+0000","last_became_peered":"2023-10-16T13:47:23.427092+0000","last_unstale":"2023-10-16T13:47:40.902848+0000","last_undegraded":"2023-10-16T13:47:40.902848+0000","last_fullsized":"2023-10-16T13:47:40.902848+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:10:56.739817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.39","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.954251+0000","last_change":"2023-10-16T13:47:41.945354+0000","last_active":"2023-10-16T13:47:41.954251+0000","last_peered":"2023-10-16T13:47:41.954251+0000","last_clean":"2023-10-16T13:47:41.954251+0000","last_became_active":"2023-10-16T13:47:41.944660+0000","last_became_peered":"2023-10-16T13:47:41.944660+0000","last_unstale":"2023-10-16T13:47:41.954251+0000","last_undegraded":"2023-10-16T13:47:41.954251+0000","last_fullsized":"2023-10-16T13:47:41.954251+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:12:02.423405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.3b","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.908235+0000","last_change":"2023-10-16T13:47:24.436718+0000","last_active":"2023-10-16T13:47:40.908235+0000","last_peered":"2023-10-16T13:47:40.908235+0000","last_clean":"2023-10-16T13:47:40.908235+0000","last_became_active":"2023-10-16T13:47:24.436371+0000","last_became_peered":"2023-10-16T13:47:24.436371+0000","last_unstale":"2023-10-16T13:47:40.908235+0000","last_undegraded":"2023-10-16T13:47:40.908235+0000","last_fullsized":"2023-10-16T13:47:40.908235+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:58:12.134727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.903396+0000","last_change":"2023-10-16T13:47:23.433833+0000","last_active":"2023-10-16T13:47:40.903396+0000","last_peered":"2023-10-16T13:47:40.903396+0000","last_clean":"2023-10-16T13:47:40.903396+0000","last_became_active":"2023-10-16T13:47:23.433381+0000","last_became_peered":"2023-10-16T13:47:23.433381+0000","last_unstale":"2023-10-16T13:47:40.903396+0000","last_undegraded":"2023-10-16T13:47:40.903396+0000","last_fullsized":"2023-10-16T13:47:40.903396+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:09:34.543970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.3e","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.952818+0000","last_change":"2023-10-16T13:47:41.946300+0000","last_active":"2023-10-16T13:47:41.952818+0000","last_peered":"2023-10-16T13:47:41.952818+0000","last_clean":"2023-10-16T13:47:41.952818+0000","last_became_active":"2023-10-16T13:47:41.946000+0000","last_became_peered":"2023-10-16T13:47:41.946000+0000","last_unstale":"2023-10-16T13:47:41.952818+0000","last_undegraded":"2023-10-16T13:47:41.952818+0000","last_fullsized":"2023-10-16T13:47:41.952818+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:39:33.788061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3c","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.903058+0000","last_change":"2023-10-16T13:47:24.424889+0000","last_active":"2023-10-16T13:47:40.903058+0000","last_peered":"2023-10-16T13:47:40.903058+0000","last_clean":"2023-10-16T13:47:40.903058+0000","last_became_active":"2023-10-16T13:47:24.424573+0000","last_became_peered":"2023-10-16T13:47:24.424573+0000","last_unstale":"2023-10-16T13:47:40.903058+0000","last_undegraded":"2023-10-16T13:47:40.903058+0000","last_fullsized":"2023-10-16T13:47:40.903058+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:23:40.809515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.911573+0000","last_change":"2023-10-16T13:47:23.435920+0000","last_active":"2023-10-16T13:47:40.911573+0000","last_peered":"2023-10-16T13:47:40.911573+0000","last_clean":"2023-10-16T13:47:40.911573+0000","last_became_active":"2023-10-16T13:47:23.435675+0000","last_became_peered":"2023-10-16T13:47:23.435675+0000","last_unstale":"2023-10-16T13:47:40.911573+0000","last_undegraded":"2023-10-16T13:47:40.911573+0000","last_fullsized":"2023-10-16T13:47:40.911573+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:01:54.403340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.3f","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.952423+0000","last_change":"2023-10-16T13:47:41.945288+0000","last_active":"2023-10-16T13:47:41.952423+0000","last_peered":"2023-10-16T13:47:41.952423+0000","last_clean":"2023-10-16T13:47:41.952423+0000","last_became_active":"2023-10-16T13:47:41.942669+0000","last_became_peered":"2023-10-16T13:47:41.942669+0000","last_unstale":"2023-10-16T13:47:41.952423+0000","last_undegraded":"2023-10-16T13:47:41.952423+0000","last_fullsized":"2023-10-16T13:47:41.952423+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:55:04.579725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.3d","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.907802+0000","last_change":"2023-10-16T13:47:24.430727+0000","last_active":"2023-10-16T13:47:40.907802+0000","last_peered":"2023-10-16T13:47:40.907802+0000","last_clean":"2023-10-16T13:47:40.907802+0000","last_became_active":"2023-10-16T13:47:24.430424+0000","last_became_peered":"2023-10-16T13:47:24.430424+0000","last_unstale":"2023-10-16T13:47:40.907802+0000","last_undegraded":"2023-10-16T13:47:40.907802+0000","last_fullsized":"2023-10-16T13:47:40.907802+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:05:49.279256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.906650+0000","last_change":"2023-10-16T13:47:23.430779+0000","last_active":"2023-10-16T13:47:40.906650+0000","last_peered":"2023-10-16T13:47:40.906650+0000","last_clean":"2023-10-16T13:47:40.906650+0000","last_became_active":"2023-10-16T13:47:23.429996+0000","last_became_peered":"2023-10-16T13:47:23.429996+0000","last_unstale":"2023-10-16T13:47:40.906650+0000","last_undegraded":"2023-10-16T13:47:40.906650+0000","last_fullsized":"2023-10-16T13:47:40.906650+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:17:22.079776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.3c","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.955157+0000","last_change":"2023-10-16T13:47:41.952726+0000","last_active":"2023-10-16T13:47:41.955157+0000","last_peered":"2023-10-16T13:47:41.955157+0000","last_clean":"2023-10-16T13:47:41.955157+0000","last_became_active":"2023-10-16T13:47:41.952452+0000","last_became_peered":"2023-10-16T13:47:41.952452+0000","last_unstale":"2023-10-16T13:47:41.955157+0000","last_undegraded":"2023-10-16T13:47:41.955157+0000","last_fullsized":"2023-10-16T13:47:41.955157+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:13:48.625511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.3e","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.919255+0000","last_change":"2023-10-16T13:47:24.436378+0000","last_active":"2023-10-16T13:47:40.919255+0000","last_peered":"2023-10-16T13:47:40.919255+0000","last_clean":"2023-10-16T13:47:40.919255+0000","last_became_active":"2023-10-16T13:47:24.435495+0000","last_became_peered":"2023-10-16T13:47:24.435495+0000","last_unstale":"2023-10-16T13:47:40.919255+0000","last_undegraded":"2023-10-16T13:47:40.919255+0000","last_fullsized":"2023-10-16T13:47:40.919255+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:18:23.073022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.912463+0000","last_change":"2023-10-16T13:47:23.433945+0000","last_active":"2023-10-16T13:47:40.912463+0000","last_peered":"2023-10-16T13:47:40.912463+0000","last_clean":"2023-10-16T13:47:40.912463+0000","last_became_active":"2023-10-16T13:47:23.433576+0000","last_became_peered":"2023-10-16T13:47:23.433576+0000","last_unstale":"2023-10-16T13:47:40.912463+0000","last_undegraded":"2023-10-16T13:47:40.912463+0000","last_fullsized":"2023-10-16T13:47:40.912463+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:09:34.543970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.3d","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.955058+0000","last_change":"2023-10-16T13:47:41.947937+0000","last_active":"2023-10-16T13:47:41.955058+0000","last_peered":"2023-10-16T13:47:41.955058+0000","last_clean":"2023-10-16T13:47:41.955058+0000","last_became_active":"2023-10-16T13:47:41.947634+0000","last_became_peered":"2023-10-16T13:47:41.947634+0000","last_unstale":"2023-10-16T13:47:41.955058+0000","last_undegraded":"2023-10-16T13:47:41.955058+0000","last_fullsized":"2023-10-16T13:47:41.955058+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:21:36.021473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.3f","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.920581+0000","last_change":"2023-10-16T13:47:24.441874+0000","last_active":"2023-10-16T13:47:40.920581+0000","last_peered":"2023-10-16T13:47:40.920581+0000","last_clean":"2023-10-16T13:47:40.920581+0000","last_became_active":"2023-10-16T13:47:24.441466+0000","last_became_peered":"2023-10-16T13:47:24.441466+0000","last_unstale":"2023-10-16T13:47:40.920581+0000","last_undegraded":"2023-10-16T13:47:40.920581+0000","last_fullsized":"2023-10-16T13:47:40.920581+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:22:04.707731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.912926+0000","last_change":"2023-10-16T13:47:23.423270+0000","last_active":"2023-10-16T13:47:40.912926+0000","last_peered":"2023-10-16T13:47:40.912926+0000","last_clean":"2023-10-16T13:47:40.912926+0000","last_became_active":"2023-10-16T13:47:23.422889+0000","last_became_peered":"2023-10-16T13:47:23.422889+0000","last_unstale":"2023-10-16T13:47:40.912926+0000","last_undegraded":"2023-10-16T13:47:40.912926+0000","last_fullsized":"2023-10-16T13:47:40.912926+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:30:37.923537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.23","version":"0'0","reported_seq":9,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-16T13:47:41.955194+0000","last_change":"2023-10-16T13:47:41.948103+0000","last_active":"2023-10-16T13:47:41.955194+0000","last_peered":"2023-10-16T13:47:41.955194+0000","last_clean":"2023-10-16T13:47:41.955194+0000","last_became_active":"2023-10-16T13:47:41.947098+0000","last_became_peered":"2023-10-16T13:47:41.947098+0000","last_unstale":"2023-10-16T13:47:41.955194+0000","last_undegraded":"2023-10-16T13:47:41.955194+0000","last_fullsized":"2023-10-16T13:47:41.955194+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:40.896342+0000","last_clean_scrub_stamp":"2023-10-16T13:47:40.896342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:05:43.949107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.21","version":"0'0","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.906700+0000","last_change":"2023-10-16T13:47:24.431657+0000","last_active":"2023-10-16T13:47:40.906700+0000","last_peered":"2023-10-16T13:47:40.906700+0000","last_clean":"2023-10-16T13:47:40.906700+0000","last_became_active":"2023-10-16T13:47:24.431385+0000","last_became_peered":"2023-10-16T13:47:24.431385+0000","last_unstale":"2023-10-16T13:47:40.906700+0000","last_undegraded":"2023-10-16T13:47:40.906700+0000","last_fullsized":"2023-10-16T13:47:40.906700+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:23.404948+0000","last_clean_scrub_stamp":"2023-10-16T13:47:23.404948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:59:09.754185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1e","version":"244'1","reported_seq":16,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.909995+0000","last_change":"2023-10-16T13:47:23.429601+0000","last_active":"2023-10-16T13:47:40.909995+0000","last_peered":"2023-10-16T13:47:40.909995+0000","last_clean":"2023-10-16T13:47:40.909995+0000","last_became_active":"2023-10-16T13:47:23.429298+0000","last_became_peered":"2023-10-16T13:47:23.429298+0000","last_unstale":"2023-10-16T13:47:40.909995+0000","last_undegraded":"2023-10-16T13:47:40.909995+0000","last_fullsized":"2023-10-16T13:47:40.909995+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:47:22.400586+0000","last_clean_scrub_stamp":"2023-10-16T13:47:22.400586+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:20:00.827725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"231'370","reported_seq":1301,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-16T13:47:40.916219+0000","last_change":"2023-10-16T12:10:34.746820+0000","last_active":"2023-10-16T13:47:40.916219+0000","last_peered":"2023-10-16T13:47:40.916219+0000","last_clean":"2023-10-16T13:47:40.916219+0000","last_became_active":"2023-10-16T12:10:34.745574+0000","last_became_peered":"2023-10-16T12:10:34.745574+0000","last_unstale":"2023-10-16T13:47:40.916219+0000","last_undegraded":"2023-10-16T13:47:40.916219+0000","last_fullsized":"2023-10-16T13:47:40.916219+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_clean_scrub_stamp":"2023-10-16T12:10:33.577111+0000","objects_scrubbed":0,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:14.461986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":34,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":33,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":32,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":31,"num_pg":32,"stat_sum":{"num_bytes":3657,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":370,"ondisk_log_size":370,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575999,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57040,"kb_used_data":3648,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314800,"statfs":{"total":96636764160,"available":96578355200,"internally_reserved":0,"allocated":3735552,"data_stored":2210331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54655435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:46:50 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.60299999999999998,"15min":0.63200000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.33000000000000002},"max":{"1min":0.78600000000000003,"5min":0.87,"15min":0.93799999999999994},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.60699999999999998,"15min":0.626},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.72099999999999997,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.45700000000000002}]},{"osd":1,"last update":"Mon Oct 16 13:46:50 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.61299999999999999,"15min":0.63900000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.79000000000000004,"5min":0.84599999999999997,"15min":0.95899999999999996},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.622,"15min":0.63100000000000001},"min":{"1min":0.38200000000000001,"5min":0.375,"15min":0.375},"max":{"1min":0.76200000000000001,"5min":0.79100000000000004,"15min":0.86399999999999999},"last":0.39800000000000002}]},{"osd":2,"last update":"Mon Oct 16 13:46:50 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.67400000000000004,"15min":0.68600000000000005},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.80300000000000005,"5min":0.83899999999999997,"15min":0.88200000000000001},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.68899999999999995,"15min":0.69799999999999995},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.39400000000000002},"max":{"1min":0.83699999999999997,"5min":0.92800000000000005,"15min":0.93799999999999994},"last":0.433}]},{"osd":3,"last update":"Mon Oct 16 13:46:50 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.63800000000000001,"15min":0.65900000000000003},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.85299999999999998,"5min":0.89200000000000002,"15min":0.91300000000000003},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.65000000000000002,"15min":0.66000000000000003},"min":{"1min":0.435,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.88,"5min":1.24,"15min":1.24},"last":0.51000000000000001}]},{"osd":4,"last update":"Mon Oct 16 13:46:50 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.52000000000000002,"15min":0.51900000000000002},"min":{"1min":0.311,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.71099999999999997,"5min":0.745,"15min":0.85299999999999998},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.621,"15min":0.61799999999999999},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.33100000000000002},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.64200000000000002}]},{"osd":5,"last update":"Mon Oct 16 13:46:50 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.54900000000000004,"15min":0.54700000000000004},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.85699999999999998,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.58399999999999996,"15min":0.58699999999999997},"min":{"1min":0.41099999999999998,"5min":0.38400000000000001,"15min":0.35099999999999998},"max":{"1min":0.81100000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.63100000000000001}]},{"osd":6,"last update":"Mon Oct 16 13:46:50 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.60499999999999998,"15min":0.61299999999999999},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.85499999999999998,"5min":0.874,"15min":0.95899999999999996},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.626,"15min":0.61899999999999999},"min":{"1min":0.39400000000000002,"5min":0.378,"15min":0.35199999999999998},"max":{"1min":0.92100000000000004,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.67000000000000004}]}]},{"osd":6,"up_from":13,"seq":55834575997,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60600,"kb_used_data":3048,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311240,"statfs":{"total":96636764160,"available":96574709760,"internally_reserved":0,"allocated":3121152,"data_stored":1611254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58915271},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:47:39 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.64100000000000001,"15min":0.66300000000000003},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.29599999999999999},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.63700000000000001,"15min":0.65800000000000003},"min":{"1min":0.36499999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.81100000000000005,"5min":0.91400000000000003,"15min":0.93799999999999994},"last":0.373}]},{"osd":1,"last update":"Mon Oct 16 13:47:39 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.57099999999999995,"15min":0.61799999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.76600000000000001,"5min":0.86899999999999999,"15min":0.90900000000000003},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.61599999999999999,"15min":0.61699999999999999},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.51500000000000001}]},{"osd":2,"last update":"Mon Oct 16 13:47:39 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.60799999999999998,"15min":0.627},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.79600000000000004,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.59699999999999998,"15min":0.61799999999999999},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.2769999999999999,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.56000000000000005}]},{"osd":3,"last update":"Mon Oct 16 13:47:39 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.625,"15min":0.65600000000000003},"min":{"1min":0.40100000000000002,"5min":0.39900000000000002,"15min":0.38200000000000001},"max":{"1min":0.75600000000000001,"5min":0.92400000000000004,"15min":0.95599999999999996},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.68400000000000005,"15min":0.71199999999999997},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.42299999999999999},"max":{"1min":0.85199999999999998,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.63200000000000001}]},{"osd":4,"last update":"Mon Oct 16 13:47:39 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.52500000000000002,"15min":0.53000000000000003},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.748,"5min":0.85599999999999998,"15min":0.95799999999999996},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.61199999999999999,"15min":0.621},"min":{"1min":0.38400000000000001,"5min":0.35499999999999998,"15min":0.32900000000000001},"max":{"1min":0.79100000000000004,"5min":0.98799999999999999,"15min":1.0309999999999999},"last":0.61699999999999999}]},{"osd":5,"last update":"Mon Oct 16 13:47:39 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.58799999999999997,"15min":0.59499999999999997},"min":{"1min":0.34399999999999997,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.80000000000000004,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.64000000000000001,"15min":0.65900000000000003},"min":{"1min":0.36099999999999999,"5min":0.33700000000000002,"15min":0.308},"max":{"1min":0.86399999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.68200000000000005}]},{"osd":7,"last update":"Mon Oct 16 13:47:39 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.58899999999999997,"15min":0.60299999999999998},"min":{"1min":0.38700000000000001,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.79300000000000004,"5min":1.081,"15min":1.081},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.63,"5min":0.64100000000000001,"15min":0.65400000000000003},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.375},"max":{"1min":0.83499999999999996,"5min":1.099,"15min":1.099},"last":0.64300000000000002}]}]},{"osd":1,"up_from":13,"seq":55834575997,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52288,"kb_used_data":3060,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319552,"statfs":{"total":96636764160,"available":96583221248,"internally_reserved":0,"allocated":3133440,"data_stored":1619359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":50395593},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:46:55 2023","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.39500000000000002,"15min":0.40699999999999997},"min":{"1min":0.27600000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.51300000000000001,"5min":0.65400000000000003,"15min":0.65800000000000003},"last":0.23000000000000001},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.47599999999999998,"15min":0.49199999999999999},"min":{"1min":0.32000000000000001,"5min":0.253,"15min":0.253},"max":{"1min":0.56000000000000005,"5min":0.70999999999999996,"15min":0.77200000000000002},"last":0.29099999999999998}]},{"osd":2,"last update":"Mon Oct 16 13:46:55 2023","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.436,"15min":0.45100000000000001},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.58599999999999997,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.48099999999999998,"15min":0.48999999999999999},"min":{"1min":0.34399999999999997,"5min":0.315,"15min":0.30199999999999999},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.71399999999999997},"last":0.308}]},{"osd":3,"last update":"Mon Oct 16 13:46:55 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.42899999999999999,"15min":0.42799999999999999},"min":{"1min":0.307,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.50800000000000001,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.254},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.46100000000000002,"15min":0.47499999999999998},"min":{"1min":0.29799999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.52800000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.34899999999999998}]},{"osd":4,"last update":"Mon Oct 16 13:46:55 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.64400000000000002,"15min":0.66200000000000003},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.42399999999999999},"max":{"1min":0.78600000000000003,"5min":0.95899999999999996,"15min":1.0069999999999999},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.64200000000000002,"15min":0.66600000000000004},"min":{"1min":0.42899999999999999,"5min":0.37,"15min":0.37},"max":{"1min":0.69999999999999996,"5min":0.91300000000000003,"15min":0.997},"last":0.69399999999999995}]},{"osd":5,"last update":"Mon Oct 16 13:46:55 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.68200000000000005,"15min":0.70699999999999996},"min":{"1min":0.45800000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.83999999999999997,"5min":0.97499999999999998,"15min":1.0760000000000001},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.70499999999999996,"15min":0.72599999999999998},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.81399999999999995,"5min":1.0469999999999999,"15min":1.0660000000000001},"last":0.58499999999999996}]},{"osd":6,"last update":"Mon Oct 16 13:46:55 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.67500000000000004,"15min":0.68000000000000005},"min":{"1min":0.46200000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.75900000000000001,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.623},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.68700000000000006,"15min":0.70499999999999996},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.80400000000000005,"5min":1.03,"15min":1.083},"last":0.54400000000000004}]},{"osd":7,"last update":"Mon Oct 16 13:46:55 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.70899999999999996,"15min":0.73399999999999999},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.77300000000000002,"5min":1.0629999999999999,"15min":1.0660000000000001},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.72899999999999998,"15min":0.746},"min":{"1min":0.496,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.82099999999999995,"5min":1.042,"15min":1.0580000000000001},"last":0.67000000000000004}]}]},{"osd":0,"up_from":13,"seq":55834575998,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48724,"kb_used_data":3648,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323116,"statfs":{"total":96636764160,"available":96586870784,"internally_reserved":0,"allocated":3735552,"data_stored":2212532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 13:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.40400000000000003,"15min":0.42099999999999999},"min":{"1min":0.29099999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.60699999999999998,"5min":0.65800000000000003,"15min":0.71399999999999997},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.42199999999999999,"15min":0.439},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.82199999999999995},"last":0.32800000000000001}]},{"osd":2,"last update":"Mon Oct 16 13:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.443,"15min":0.45400000000000001},"min":{"1min":0.28599999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.61399999999999999,"5min":0.65900000000000003,"15min":0.75700000000000001},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.46400000000000002,"15min":0.48499999999999999},"min":{"1min":0.29799999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.83299999999999996},"last":0.39300000000000002}]},{"osd":3,"last update":"Mon Oct 16 13:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.42699999999999999,"15min":0.45300000000000001},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.56499999999999995,"5min":0.63400000000000001,"15min":0.78600000000000003},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.50900000000000001,"15min":0.52000000000000002},"min":{"1min":0.317,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.66400000000000003,"5min":0.82999999999999996,"15min":0.86399999999999999},"last":0.309}]},{"osd":4,"last update":"Mon Oct 16 13:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.66400000000000003,"15min":0.68400000000000005},"min":{"1min":0.42399999999999999,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.66500000000000004,"15min":0.68899999999999995},"min":{"1min":0.499,"5min":0.375,"15min":0.375},"max":{"1min":1.21,"5min":1.21,"15min":1.21},"last":0.74399999999999999}]},{"osd":5,"last update":"Mon Oct 16 13:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.66400000000000003,"15min":0.68999999999999995},"min":{"1min":0.41999999999999998,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.242,"5min":1.242,"15min":1.242},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.70599999999999996,"15min":0.72299999999999998},"min":{"1min":0.497,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.998,"5min":1.081,"15min":1.081},"last":0.76000000000000001}]},{"osd":6,"last update":"Mon Oct 16 13:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.71199999999999997,"15min":0.72899999999999998},"min":{"1min":0.51900000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.96599999999999997,"5min":1.034,"15min":1.103},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.70399999999999996,"15min":0.72299999999999998},"min":{"1min":0.51000000000000001,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":0.77400000000000002}]},{"osd":7,"last update":"Mon Oct 16 13:46:43 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.71199999999999997,"15min":0.74199999999999999},"min":{"1min":0.48399999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.1919999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.72499999999999998,"15min":0.73499999999999999},"min":{"1min":0.51300000000000001,"5min":0.433,"15min":0.433},"max":{"1min":1.0469999999999999,"5min":1.087,"15min":1.087},"last":0.66200000000000003}]}]},{"osd":2,"up_from":13,"seq":55834575998,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52312,"kb_used_data":3080,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319528,"statfs":{"total":96636764160,"available":96583196672,"internally_reserved":0,"allocated":3153920,"data_stored":1622103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.39800000000000002,"15min":0.40699999999999997},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.54100000000000004,"5min":0.71699999999999997,"15min":0.77500000000000002},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.45800000000000002,"15min":0.47599999999999998},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.69799999999999995,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.38200000000000001}]},{"osd":1,"last update":"Mon Oct 16 13:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.47899999999999998,"15min":0.495},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.61899999999999999,"5min":0.73199999999999998,"15min":0.81399999999999995},"last":0.433},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.47099999999999997,"15min":0.47299999999999998},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.59799999999999998,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.40600000000000003}]},{"osd":3,"last update":"Mon Oct 16 13:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.432,"15min":0.438},"min":{"1min":0.26300000000000001,"5min":0.255,"15min":0.255},"max":{"1min":0.63,"5min":0.70099999999999996,"15min":0.72599999999999998},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.52400000000000002,"15min":0.54100000000000004},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.74099999999999999,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.27000000000000002}]},{"osd":4,"last update":"Mon Oct 16 13:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.69299999999999995,"15min":0.71199999999999997},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.94399999999999995,"5min":1.3600000000000001,"15min":1.3600000000000001},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.70399999999999996,"15min":0.71599999999999997},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.196,"5min":1.403,"15min":1.403},"last":0.49399999999999999}]},{"osd":5,"last update":"Mon Oct 16 13:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.65900000000000003,"15min":0.68400000000000005},"min":{"1min":0.46700000000000003,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.82999999999999996,"5min":1.486,"15min":1.486},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.68200000000000005,"15min":0.69599999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.879,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":0.50900000000000001}]},{"osd":6,"last update":"Mon Oct 16 13:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.65800000000000003,"15min":0.66500000000000004},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.087,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.69799999999999995,"15min":0.71799999999999997},"min":{"1min":0.46000000000000002,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.94299999999999995,"5min":1.4379999999999999,"15min":1.4379999999999999},"last":0.52600000000000002}]},{"osd":7,"last update":"Mon Oct 16 13:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.70499999999999996,"15min":0.71499999999999997},"min":{"1min":0.42599999999999999,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.97099999999999997,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.70299999999999996,"15min":0.72899999999999998},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.0429999999999999,"5min":1.377,"15min":1.377},"last":0.56499999999999995}]}]},{"osd":3,"up_from":13,"seq":55834575997,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56472,"kb_used_data":3080,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315368,"statfs":{"total":96636764160,"available":96578936832,"internally_reserved":0,"allocated":3153920,"data_stored":1622822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:47:42 2023","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.39300000000000002,"15min":0.40699999999999997},"min":{"1min":0.27400000000000002,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.60099999999999998,"5min":0.64200000000000002,"15min":0.72899999999999998},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.40000000000000002,"15min":0.42099999999999999},"min":{"1min":0.27000000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.47899999999999998,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.34499999999999997}]},{"osd":1,"last update":"Mon Oct 16 13:47:42 2023","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.40799999999999997,"15min":0.42699999999999999},"min":{"1min":0.27100000000000002,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.46200000000000002,"5min":0.68200000000000005,"15min":0.76300000000000001},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.44600000000000001,"15min":0.47499999999999998},"min":{"1min":0.309,"5min":0.253,"15min":0.253},"max":{"1min":0.65400000000000003,"5min":0.70699999999999996,"15min":0.77800000000000002},"last":0.36899999999999999}]},{"osd":2,"last update":"Mon Oct 16 13:47:42 2023","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.46600000000000003,"15min":0.48499999999999999},"min":{"1min":0.317,"5min":0.316,"15min":0.316},"max":{"1min":0.64700000000000002,"5min":1.167,"15min":1.167},"last":0.317},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.48399999999999999,"15min":0.51400000000000001},"min":{"1min":0.32400000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.58099999999999996,"5min":0.73299999999999998,"15min":0.78700000000000003},"last":0.39700000000000002}]},{"osd":4,"last update":"Mon Oct 16 13:47:42 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.626,"15min":0.65800000000000003},"min":{"1min":0.46000000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.69399999999999995,"5min":0.998,"15min":1.137},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.66600000000000004,"15min":0.68899999999999995},"min":{"1min":0.44600000000000001,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.86299999999999999,"5min":1.2110000000000001,"15min":1.2909999999999999},"last":0.45300000000000001}]},{"osd":5,"last update":"Mon Oct 16 13:47:42 2023","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.65700000000000003,"15min":0.69999999999999996},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.374},"max":{"1min":0.85099999999999998,"5min":1.0680000000000001,"15min":1.2729999999999999},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.66900000000000004,"15min":0.69399999999999995},"min":{"1min":0.48799999999999999,"5min":0.45600000000000002,"15min":0.432},"max":{"1min":0.77400000000000002,"5min":1.0549999999999999,"15min":1.173},"last":0.58299999999999996}]},{"osd":6,"last update":"Mon Oct 16 13:47:42 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.70999999999999996,"15min":0.71399999999999997},"min":{"1min":0.434,"5min":0.42999999999999999,"15min":0.41799999999999998},"max":{"1min":0.80900000000000005,"5min":5.0999999999999996,"15min":5.0999999999999996},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.69299999999999995,"15min":0.72699999999999998},"min":{"1min":0.47399999999999998,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.89100000000000001,"5min":1.0309999999999999,"15min":1.2290000000000001},"last":0.48499999999999999}]},{"osd":7,"last update":"Mon Oct 16 13:47:42 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.69599999999999995,"15min":0.72899999999999998},"min":{"1min":0.497,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.82799999999999996,"5min":1.048,"15min":1.2569999999999999},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.62,"5min":0.70299999999999996,"15min":0.73599999999999999},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.43099999999999999},"max":{"1min":0.90400000000000003,"5min":1.262,"15min":1.262},"last":0.47299999999999998}]}]},{"osd":4,"up_from":13,"seq":55834575996,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52300,"kb_used_data":3068,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319540,"statfs":{"total":96636764160,"available":96583208960,"internally_reserved":0,"allocated":3141632,"data_stored":1620917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:47:41 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.57999999999999996,"15min":0.623},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.81599999999999995,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.60099999999999998,"15min":0.63},"min":{"1min":0.39300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.77300000000000002,"5min":0.878,"15min":0.89500000000000002},"last":0.61099999999999999}]},{"osd":1,"last update":"Mon Oct 16 13:47:41 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.60799999999999998,"15min":0.65200000000000002},"min":{"1min":0.33700000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.78900000000000003,"5min":0.877,"15min":0.877},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.60899999999999999,"15min":0.63700000000000001},"min":{"1min":0.42199999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.80800000000000005,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.64200000000000002}]},{"osd":2,"last update":"Mon Oct 16 13:47:41 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.623,"15min":0.64900000000000002},"min":{"1min":0.39000000000000001,"5min":0.378,"15min":0.378},"max":{"1min":0.69399999999999995,"5min":0.89900000000000002,"15min":0.90600000000000003},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.64200000000000002,"15min":0.65900000000000003},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.77500000000000002,"5min":0.89700000000000002,"15min":0.92600000000000005},"last":0.71999999999999997}]},{"osd":3,"last update":"Mon Oct 16 13:47:41 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.61699999999999999,"15min":0.64300000000000002},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.76000000000000001,"5min":0.86599999999999999,"15min":0.99199999999999999},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.65300000000000002,"15min":0.68000000000000005},"min":{"1min":0.45600000000000002,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.93899999999999995},"last":0.54800000000000004}]},{"osd":5,"last update":"Mon Oct 16 13:47:41 2023","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.54400000000000004,"15min":0.53300000000000003},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.73799999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.58099999999999996,"15min":0.56799999999999995},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.753,"5min":0.82199999999999995,"15min":0.94899999999999995},"last":0.57499999999999996}]},{"osd":6,"last update":"Mon Oct 16 13:47:41 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.627,"15min":0.60099999999999998},"min":{"1min":0.378,"5min":0.36399999999999999,"15min":0.33200000000000002},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.627},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.70999999999999996,"15min":0.69499999999999995},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.39400000000000002},"max":{"1min":0.84299999999999997,"5min":1.129,"15min":1.129},"last":0.746}]},{"osd":7,"last update":"Mon Oct 16 13:47:41 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.65800000000000003,"15min":0.63500000000000001},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.38900000000000001},"max":{"1min":0.754,"5min":0.97499999999999998,"15min":0.98599999999999999},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.71299999999999997,"15min":0.68700000000000006},"min":{"1min":0.48699999999999999,"5min":0.44,"15min":0.38200000000000001},"max":{"1min":0.84699999999999998,"5min":0.97599999999999998,"15min":1.026},"last":0.66500000000000004}]}]},{"osd":5,"up_from":13,"seq":55834575997,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70332,"kb_used_data":3056,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94301508,"statfs":{"total":96636764160,"available":96564744192,"internally_reserved":0,"allocated":3129344,"data_stored":1619305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:47:28 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.59599999999999997,"15min":0.61399999999999999},"min":{"1min":0.379,"5min":0.314,"15min":0.26800000000000002},"max":{"1min":0.79200000000000004,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.61799999999999999,"15min":0.63700000000000001},"min":{"1min":0.38600000000000001,"5min":0.35899999999999999,"15min":0.29699999999999999},"max":{"1min":0.78600000000000003,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.57099999999999995}]},{"osd":1,"last update":"Mon Oct 16 13:47:28 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.64700000000000002,"15min":0.65900000000000003},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.29499999999999998},"max":{"1min":0.82799999999999996,"5min":1,"15min":1},"last":0.504},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.61299999999999999,"15min":0.63600000000000001},"min":{"1min":0.378,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.82099999999999995,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.59599999999999997}]},{"osd":2,"last update":"Mon Oct 16 13:47:28 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.65400000000000003,"15min":0.65500000000000003},"min":{"1min":0.48699999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.80500000000000005,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.63400000000000001,"15min":0.64700000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.80200000000000005,"5min":0.871,"15min":0.93500000000000005},"last":0.55400000000000005}]},{"osd":3,"last update":"Mon Oct 16 13:47:28 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.66200000000000003,"15min":0.66500000000000004},"min":{"1min":0.42199999999999999,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.85799999999999998,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.629},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.68999999999999995,"15min":0.69399999999999995},"min":{"1min":0.45500000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.82799999999999996,"5min":1.4199999999999999,"15min":1.4199999999999999},"last":0.46899999999999997}]},{"osd":4,"last update":"Mon Oct 16 13:47:28 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.54800000000000004,"15min":0.53800000000000003},"min":{"1min":0.39500000000000002,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.89700000000000002,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.60799999999999998,"15min":0.59499999999999997},"min":{"1min":0.40999999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.67200000000000004}]},{"osd":6,"last update":"Mon Oct 16 13:47:28 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.60099999999999998,"15min":0.58799999999999997},"min":{"1min":0.443,"5min":0.315,"15min":0.315},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.93200000000000005},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.629,"15min":0.625},"min":{"1min":0.501,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.49099999999999999}]},{"osd":7,"last update":"Mon Oct 16 13:47:28 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.65500000000000003,"15min":0.63900000000000001},"min":{"1min":0.47599999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.93899999999999995,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.70699999999999996,"15min":0.68899999999999995},"min":{"1min":0.53400000000000003,"5min":0.375,"15min":0.375},"max":{"1min":0.97299999999999998,"5min":0.98299999999999998,"15min":1.0229999999999999},"last":0.64600000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T13:47:46.527 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-16T13:47:46.528 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:47:46.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.680+0000 7f83a55e9700 1 Processor -- start 2023-10-16T13:47:46.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.680+0000 7f83a55e9700 1 -- start start 2023-10-16T13:47:46.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.681+0000 7f83a55e9700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f83a00d41c0 0x7f83a00d0040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:46.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.681+0000 7f83a55e9700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f83a00d0580 0x7f83a006c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:46.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.681+0000 7f83a55e9700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83a006ba00 con 0x7f83a015d8d0 2023-10-16T13:47:46.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.681+0000 7f83a55e9700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83a00d0a10 con 0x7f83a00d41c0 2023-10-16T13:47:46.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.681+0000 7f83a55e9700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83a00d26d0 con 0x7f83a00d0580 2023-10-16T13:47:46.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.681+0000 7f839f7fe700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f83a00d0580 0x7f83a006c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:46.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.681+0000 7f839e7fc700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f83a00d41c0 0x7f83a00d0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:46.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.681+0000 7f839f7fe700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f83a00d0580 0x7f83a006c4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:44178/0 (socket says 172.21.15.16:44178) 2023-10-16T13:47:46.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.681+0000 7f839e7fc700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f83a00d41c0 0x7f83a00d0040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:54980/0 (socket says 172.21.15.16:54980) 2023-10-16T13:47:46.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.681+0000 7f839effd700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7f83a015d8d0 0x7f83a00d1c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:52372/0 (socket says 172.21.15.16:52372) 2023-10-16T13:47:46.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.681+0000 7f839f7fe700 1 -- 172.21.15.16:0/1034663946 learned_addr learned my addr 172.21.15.16:0/1034663946 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:46.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.681+0000 7f839e7fc700 1 -- 172.21.15.16:0/1034663946 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f83a00d0580 msgr2=0x7f83a006c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:46.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.681+0000 7f839e7fc700 1 --2- 172.21.15.16:0/1034663946 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f83a00d0580 0x7f83a006c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:46.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.681+0000 7f839e7fc700 1 -- 172.21.15.16:0/1034663946 >> v1:172.21.15.49:6789/0 conn(0x7f83a015d8d0 legacy=0x7f83a00d1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:46.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.681+0000 7f839e7fc700 1 -- 172.21.15.16:0/1034663946 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83a00d23f0 con 0x7f83a00d41c0 2023-10-16T13:47:46.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.681+0000 7f839f7fe700 1 --2- 172.21.15.16:0/1034663946 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f83a00d0580 0x7f83a006c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:47:46.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.682+0000 7f839e7fc700 1 --2- 172.21.15.16:0/1034663946 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f83a00d41c0 0x7f83a00d0040 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f8394009840 tx=0x7f8394009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=97ebce4494b914ad server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:46.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.682+0000 7f839dffb700 1 -- 172.21.15.16:0/1034663946 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8394011830 con 0x7f83a00d41c0 2023-10-16T13:47:46.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.682+0000 7f839dffb700 1 -- 172.21.15.16:0/1034663946 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8394011990 con 0x7f83a00d41c0 2023-10-16T13:47:46.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.682+0000 7f839dffb700 1 -- 172.21.15.16:0/1034663946 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8394011d20 con 0x7f83a00d41c0 2023-10-16T13:47:46.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.682+0000 7f83a55e9700 1 -- 172.21.15.16:0/1034663946 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f83a00d41c0 msgr2=0x7f83a00d0040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:46.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.682+0000 7f83a55e9700 1 --2- 172.21.15.16:0/1034663946 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f83a00d41c0 0x7f83a00d0040 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f8394009840 tx=0x7f8394009b50 comp rx=0 tx=0).stop 2023-10-16T13:47:46.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.682+0000 7f83a55e9700 1 -- 172.21.15.16:0/1034663946 shutdown_connections 2023-10-16T13:47:46.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.682+0000 7f83a55e9700 1 --2- 172.21.15.16:0/1034663946 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f83a00d0580 0x7f83a006c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:46.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.682+0000 7f83a55e9700 1 --2- 172.21.15.16:0/1034663946 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f83a00d41c0 0x7f83a00d0040 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:46.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.682+0000 7f83a55e9700 1 -- 172.21.15.16:0/1034663946 >> 172.21.15.16:0/1034663946 conn(0x7f83a00d3630 msgr2=0x7f83a00d3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:46.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.682+0000 7f83a55e9700 1 -- 172.21.15.16:0/1034663946 shutdown_connections 2023-10-16T13:47:46.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.683+0000 7f83a55e9700 1 -- 172.21.15.16:0/1034663946 wait complete. 2023-10-16T13:47:46.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.684+0000 7f83a55e9700 1 Processor -- start 2023-10-16T13:47:46.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.684+0000 7f83a55e9700 1 -- start start 2023-10-16T13:47:46.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.685+0000 7f83a55e9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83a00d0580 0x7f83a009d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:46.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.685+0000 7f83a55e9700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f83a00d41c0 0x7f83a015bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:46.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.685+0000 7f83a55e9700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f83a015d8d0 0x7f83a01f6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:46.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.685+0000 7f83a55e9700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83a009d850 con 0x7f83a00d0580 2023-10-16T13:47:46.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.685+0000 7f83a55e9700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83a009fa30 con 0x7f83a00d41c0 2023-10-16T13:47:46.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.685+0000 7f83a55e9700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83a009fd10 con 0x7f83a015d8d0 2023-10-16T13:47:46.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.685+0000 7f839effd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83a00d0580 0x7f83a009d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:46.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.685+0000 7f839effd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83a00d0580 0x7f83a009d310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55922/0 (socket says 172.21.15.16:55922) 2023-10-16T13:47:46.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.685+0000 7f839effd700 1 -- 172.21.15.16:0/3655961962 learned_addr learned my addr 172.21.15.16:0/3655961962 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:46.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.685+0000 7f839e7fc700 1 --2- 172.21.15.16:0/3655961962 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f83a00d41c0 0x7f83a015bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:46.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.685+0000 7f839f7fe700 1 --2- 172.21.15.16:0/3655961962 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f83a015d8d0 0x7f83a01f6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:46.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.685+0000 7f839effd700 1 -- 172.21.15.16:0/3655961962 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f83a015d8d0 msgr2=0x7f83a01f6180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:46.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.685+0000 7f839effd700 1 --2- 172.21.15.16:0/3655961962 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f83a015d8d0 0x7f83a01f6180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:46.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.685+0000 7f839effd700 1 -- 172.21.15.16:0/3655961962 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f83a00d41c0 msgr2=0x7f83a015bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:46.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.685+0000 7f839effd700 1 --2- 172.21.15.16:0/3655961962 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f83a00d41c0 0x7f83a015bbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:46.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.685+0000 7f839effd700 1 -- 172.21.15.16:0/3655961962 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8398009b30 con 0x7f83a00d0580 2023-10-16T13:47:46.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.685+0000 7f839effd700 1 --2- 172.21.15.16:0/3655961962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83a00d0580 0x7f83a009d310 secure :-1 s=READY pgs=2818 cs=0 l=1 rev1=1 crypto rx=0x7f839000c430 tx=0x7f839000c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:46.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.685+0000 7f83877fe700 1 -- 172.21.15.16:0/3655961962 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8390005cb0 con 0x7f83a00d0580 2023-10-16T13:47:46.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.685+0000 7f83877fe700 1 -- 172.21.15.16:0/3655961962 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8390005e10 con 0x7f83a00d0580 2023-10-16T13:47:46.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.685+0000 7f83877fe700 1 -- 172.21.15.16:0/3655961962 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83900072d0 con 0x7f83a00d0580 2023-10-16T13:47:46.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.685+0000 7f83a55e9700 1 -- 172.21.15.16:0/3655961962 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83940094a0 con 0x7f83a00d0580 2023-10-16T13:47:46.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.686+0000 7f83a55e9700 1 -- 172.21.15.16:0/3655961962 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83a00a02d0 con 0x7f83a00d0580 2023-10-16T13:47:46.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.687+0000 7f83a55e9700 1 -- 172.21.15.16:0/3655961962 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83a004ebb0 con 0x7f83a00d0580 2023-10-16T13:47:46.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.687+0000 7f83877fe700 1 -- 172.21.15.16:0/3655961962 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 148) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f8390007430 con 0x7f83a00d0580 2023-10-16T13:47:46.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.687+0000 7f83877fe700 1 --2- 172.21.15.16:0/3655961962 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f838c06b140 0x7f838c06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:46.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.687+0000 7f83877fe700 1 -- 172.21.15.16:0/3655961962 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 7021+0+0 (secure 0 0 0) 0x7f8390089240 con 0x7f83a00d0580 2023-10-16T13:47:46.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.687+0000 7f839e7fc700 1 --2- 172.21.15.16:0/3655961962 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f838c06b140 0x7f838c06d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:46.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.688+0000 7f839e7fc700 1 --2- 172.21.15.16:0/3655961962 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f838c06b140 0x7f838c06d600 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f83a00b8880 tx=0x7f8394008030 comp rx=0 tx=0).ready entity=mgr.11886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:46.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.692+0000 7f83877fe700 1 -- 172.21.15.16:0/3655961962 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8390053b30 con 0x7f83a00d0580 2023-10-16T13:47:46.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.942+0000 7f83a55e9700 1 -- 172.21.15.16:0/3655961962 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f83a00d1cb0 con 0x7f83a00d0580 2023-10-16T13:47:46.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.943+0000 7f83877fe700 1 -- 172.21.15.16:0/3655961962 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) v1 ==== 95+0+6147 (secure 0 0 0) 0x7f8390057150 con 0x7f83a00d0580 2023-10-16T13:47:46.947 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:47:46.947 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":135,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9596,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/3939647668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":3939647668},{"type":"v1","addr":"172.21.15.49:6834","nonce":3939647668}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":125},{"gid":11185,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/2890113513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2890113513},{"type":"v1","addr":"172.21.15.16:6836","nonce":2890113513}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":125}],"filesystems":[{"mdsmap":{"epoch":130,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:47:23.843333+0000","modified":"2023-10-16T13:47:26.851597+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":12252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12252":{"gid":12252,"name":"a","rank":0,"incarnation":126,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/303278795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":303278795},{"type":"v1","addr":"172.21.15.16:6837","nonce":303278795}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:47:41.332096+0000","modified":"2023-10-16T13:47:44.368830+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11197},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11197":{"gid":11197,"name":"d","rank":0,"incarnation":132,"state":"up:active","state_seq":7,"addr":"172.21.15.49:6835/680748068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":680748068},{"type":"v1","addr":"172.21.15.49:6835","nonce":680748068}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2023-10-16T13:47:46.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.945+0000 7f83a55e9700 1 -- 172.21.15.16:0/3655961962 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f838c06b140 msgr2=0x7f838c06d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:46.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.945+0000 7f83a55e9700 1 --2- 172.21.15.16:0/3655961962 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f838c06b140 0x7f838c06d600 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f83a00b8880 tx=0x7f8394008030 comp rx=0 tx=0).stop 2023-10-16T13:47:46.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.945+0000 7f83a55e9700 1 -- 172.21.15.16:0/3655961962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83a00d0580 msgr2=0x7f83a009d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:46.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.945+0000 7f83a55e9700 1 --2- 172.21.15.16:0/3655961962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83a00d0580 0x7f83a009d310 secure :-1 s=READY pgs=2818 cs=0 l=1 rev1=1 crypto rx=0x7f839000c430 tx=0x7f839000c7f0 comp rx=0 tx=0).stop 2023-10-16T13:47:46.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.945+0000 7f83a55e9700 1 -- 172.21.15.16:0/3655961962 shutdown_connections 2023-10-16T13:47:46.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.945+0000 7f83a55e9700 1 --2- 172.21.15.16:0/3655961962 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f838c06b140 0x7f838c06d600 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:46.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.945+0000 7f83a55e9700 1 --2- 172.21.15.16:0/3655961962 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f83a015d8d0 0x7f83a01f6180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:46.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.945+0000 7f83a55e9700 1 --2- 172.21.15.16:0/3655961962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83a00d0580 0x7f83a009d310 unknown :-1 s=CLOSED pgs=2818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:46.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.945+0000 7f83a55e9700 1 --2- 172.21.15.16:0/3655961962 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f83a00d41c0 0x7f83a015bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:46.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.945+0000 7f83a55e9700 1 -- 172.21.15.16:0/3655961962 >> 172.21.15.16:0/3655961962 conn(0x7f83a00d3630 msgr2=0x7f83a00cd3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:46.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.946+0000 7f83a55e9700 1 -- 172.21.15.16:0/3655961962 shutdown_connections 2023-10-16T13:47:46.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:46.946+0000 7f83a55e9700 1 -- 172.21.15.16:0/3655961962 wait complete. 2023-10-16T13:47:46.952 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 135 2023-10-16T13:47:47.364 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 135, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T13:47:41.332096+0000', 'modified': '2023-10-16T13:47:44.368830+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 11197}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_11197': {'gid': 11197, 'name': 'd', 'rank': 0, 'incarnation': 132, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.49:6835/680748068', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6833', 'nonce': 680748068}, {'type': 'v1', 'addr': '172.21.15.49:6835', 'nonce': 680748068}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'backup_fs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-16T13:47:47.365 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-16T13:47:47.365 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2023-10-16T13:47:47.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.495+0000 7f8b0a4a6700 1 Processor -- start 2023-10-16T13:47:47.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.495+0000 7f8b0a4a6700 1 -- start start 2023-10-16T13:47:47.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.496+0000 7f8b0a4a6700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8b040cf880 0x7f8b040b52f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:47.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.496+0000 7f8b0a4a6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8b040b5830 0x7f8b0415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:47.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.496+0000 7f8b0a4a6700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b0415bc80 con 0x7f8b040cf4a0 2023-10-16T13:47:47.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.496+0000 7f8b0a4a6700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b0415b070 con 0x7f8b040b5830 2023-10-16T13:47:47.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.496+0000 7f8b0a4a6700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b0415b350 con 0x7f8b040cf880 2023-10-16T13:47:47.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.496+0000 7f8b08a43700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8b040b5830 0x7f8b0415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:47.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.496+0000 7f8b08a43700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8b040b5830 0x7f8b0415a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55934/0 (socket says 172.21.15.16:55934) 2023-10-16T13:47:47.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.496+0000 7f8b08a43700 1 -- 172.21.15.16:0/3600684902 learned_addr learned my addr 172.21.15.16:0/3600684902 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:47.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.496+0000 7f8b037fe700 1 --2- 172.21.15.16:0/3600684902 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8b040cf880 0x7f8b040b52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:47.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.496+0000 7f8b08a43700 1 -- 172.21.15.16:0/3600684902 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8b040cf880 msgr2=0x7f8b040b52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:47.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.496+0000 7f8b08a43700 1 --2- 172.21.15.16:0/3600684902 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8b040cf880 0x7f8b040b52f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:47.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.496+0000 7f8b08a43700 1 -- 172.21.15.16:0/3600684902 >> v1:172.21.15.16:6789/0 conn(0x7f8b040cf4a0 legacy=0x7f8b040726e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:47.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.496+0000 7f8b08a43700 1 -- 172.21.15.16:0/3600684902 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b0415b630 con 0x7f8b040b5830 2023-10-16T13:47:47.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.496+0000 7f8b08a43700 1 --2- 172.21.15.16:0/3600684902 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b040b5830 0x7f8b0415a090 secure :-1 s=READY pgs=2820 cs=0 l=1 rev1=1 crypto rx=0x7f8afc009e50 tx=0x7f8afc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7db437d68eacf256 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:47.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.496+0000 7f8b02ffd700 1 -- 172.21.15.16:0/3600684902 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8afc00e070 con 0x7f8b040b5830 2023-10-16T13:47:47.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.497+0000 7f8b02ffd700 1 -- 172.21.15.16:0/3600684902 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8afc00b9c0 con 0x7f8b040b5830 2023-10-16T13:47:47.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.497+0000 7f8b02ffd700 1 -- 172.21.15.16:0/3600684902 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8afc00bd70 con 0x7f8b040b5830 2023-10-16T13:47:47.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.497+0000 7f8b0a4a6700 1 -- 172.21.15.16:0/3600684902 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b040b5830 msgr2=0x7f8b0415a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:47.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.497+0000 7f8b0a4a6700 1 --2- 172.21.15.16:0/3600684902 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b040b5830 0x7f8b0415a090 secure :-1 s=READY pgs=2820 cs=0 l=1 rev1=1 crypto rx=0x7f8afc009e50 tx=0x7f8afc00b330 comp rx=0 tx=0).stop 2023-10-16T13:47:47.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.497+0000 7f8b0a4a6700 1 -- 172.21.15.16:0/3600684902 shutdown_connections 2023-10-16T13:47:47.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.497+0000 7f8b0a4a6700 1 --2- 172.21.15.16:0/3600684902 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b040b5830 0x7f8b0415a090 unknown :-1 s=CLOSED pgs=2820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:47.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.497+0000 7f8b0a4a6700 1 --2- 172.21.15.16:0/3600684902 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8b040cf880 0x7f8b040b52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:47.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.497+0000 7f8b0a4a6700 1 -- 172.21.15.16:0/3600684902 >> 172.21.15.16:0/3600684902 conn(0x7f8b040b1fd0 msgr2=0x7f8b040b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:47.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.497+0000 7f8b0a4a6700 1 -- 172.21.15.16:0/3600684902 shutdown_connections 2023-10-16T13:47:47.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.497+0000 7f8b0a4a6700 1 -- 172.21.15.16:0/3600684902 wait complete. 2023-10-16T13:47:47.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.498+0000 7f8b0a4a6700 1 Processor -- start 2023-10-16T13:47:47.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.498+0000 7f8b0a4a6700 1 -- start start 2023-10-16T13:47:47.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.499+0000 7f8b0a4a6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b040b5830 0x7f8b040c9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:47.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.499+0000 7f8b0a4a6700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8b040cf4a0 0x7f8b04156270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:47.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.499+0000 7f8b0a4a6700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8b040cf880 0x7f8b040b11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:47.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.499+0000 7f8b0a4a6700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b0414a3e0 con 0x7f8b040b5830 2023-10-16T13:47:47.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.499+0000 7f8b0a4a6700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b040ca400 con 0x7f8b040cf880 2023-10-16T13:47:47.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.499+0000 7f8b0a4a6700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b0414a630 con 0x7f8b040cf4a0 2023-10-16T13:47:47.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.499+0000 7f8b03fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b040b5830 0x7f8b040c9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:47.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.499+0000 7f8b03fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b040b5830 0x7f8b040c9ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55936/0 (socket says 172.21.15.16:55936) 2023-10-16T13:47:47.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.499+0000 7f8b03fff700 1 -- 172.21.15.16:0/497541202 learned_addr learned my addr 172.21.15.16:0/497541202 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:47.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.500+0000 7f8b037fe700 1 --2- 172.21.15.16:0/497541202 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8b040cf4a0 0x7f8b04156270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:47.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.500+0000 7f8b03fff700 1 -- 172.21.15.16:0/497541202 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8b040cf4a0 msgr2=0x7f8b04156270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:47.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.500+0000 7f8b03fff700 1 --2- 172.21.15.16:0/497541202 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8b040cf4a0 0x7f8b04156270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:47.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.500+0000 7f8b03fff700 1 -- 172.21.15.16:0/497541202 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8b040cf880 msgr2=0x7f8b040b11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:47.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.500+0000 7f8b08a43700 1 --2- 172.21.15.16:0/497541202 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8b040cf880 0x7f8b040b11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:47.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.500+0000 7f8b03fff700 1 --2- 172.21.15.16:0/497541202 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8b040cf880 0x7f8b040b11a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:47.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.500+0000 7f8b03fff700 1 -- 172.21.15.16:0/497541202 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8afc009b30 con 0x7f8b040b5830 2023-10-16T13:47:47.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.500+0000 7f8b03fff700 1 --2- 172.21.15.16:0/497541202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b040b5830 0x7f8b040c9ec0 secure :-1 s=READY pgs=2821 cs=0 l=1 rev1=1 crypto rx=0x7f8aec00c510 tx=0x7f8aec00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:47.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.500+0000 7f8b00ff9700 1 -- 172.21.15.16:0/497541202 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aec0052d0 con 0x7f8b040b5830 2023-10-16T13:47:47.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.500+0000 7f8b00ff9700 1 -- 172.21.15.16:0/497541202 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8aec005430 con 0x7f8b040b5830 2023-10-16T13:47:47.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.500+0000 7f8b00ff9700 1 -- 172.21.15.16:0/497541202 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aec0038c0 con 0x7f8b040b5830 2023-10-16T13:47:47.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.500+0000 7f8b0a4a6700 1 -- 172.21.15.16:0/497541202 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b0414a910 con 0x7f8b040b5830 2023-10-16T13:47:47.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.500+0000 7f8b0a4a6700 1 -- 172.21.15.16:0/497541202 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b0414aed0 con 0x7f8b040b5830 2023-10-16T13:47:47.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.502+0000 7f8b00ff9700 1 -- 172.21.15.16:0/497541202 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 148) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f8aec003a20 con 0x7f8b040b5830 2023-10-16T13:47:47.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.502+0000 7f8b00ff9700 1 --2- 172.21.15.16:0/497541202 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f8ae406b140 0x7f8ae406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:47.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.502+0000 7f8b00ff9700 1 -- 172.21.15.16:0/497541202 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 7021+0+0 (secure 0 0 0) 0x7f8aec089300 con 0x7f8b040b5830 2023-10-16T13:47:47.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.502+0000 7f8b037fe700 1 --2- 172.21.15.16:0/497541202 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f8ae406b140 0x7f8ae406d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:47.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.502+0000 7f8b037fe700 1 --2- 172.21.15.16:0/497541202 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f8ae406b140 0x7f8ae406d600 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f8b0414d600 tx=0x7f8af4008040 comp rx=0 tx=0).ready entity=mgr.11886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:47.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.504+0000 7f8b16690b80 1 Processor -- start 2023-10-16T13:47:47.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.504+0000 7f8b16690b80 1 -- start start 2023-10-16T13:47:47.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.505+0000 7f8b16690b80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x560fb772fb40 0x560fb772ff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:47.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.505+0000 7f8b16690b80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560fb7756f30 0x560fb772f280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:47.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.505+0000 7f8b16690b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560fb772af00 0x560fb7756300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:47.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.505+0000 7f8b16690b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x560fb76a3c20 con 0x560fb772af00 2023-10-16T13:47:47.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.505+0000 7f8b16690b80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x560fb7688c10 con 0x560fb7756f30 2023-10-16T13:47:47.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.505+0000 7f8b16690b80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x560fb7686530 con 0x560fb772fb40 2023-10-16T13:47:47.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.505+0000 7f8b08a43700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560fb772af00 0x560fb7756300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:47.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.505+0000 7f8b08a43700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560fb772af00 0x560fb7756300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55948/0 (socket says 172.21.15.16:55948) 2023-10-16T13:47:47.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.505+0000 7f8b08a43700 1 -- 172.21.15.16:0/2604509910 learned_addr learned my addr 172.21.15.16:0/2604509910 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:47.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.505+0000 7f8b037fe700 1 --2- 172.21.15.16:0/2604509910 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560fb7756f30 0x560fb772f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:47.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.505+0000 7f8b03fff700 1 --2- 172.21.15.16:0/2604509910 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x560fb772fb40 0x560fb772ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:47.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.505+0000 7f8b08a43700 1 -- 172.21.15.16:0/2604509910 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x560fb772fb40 msgr2=0x560fb772ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:47.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.505+0000 7f8b08a43700 1 --2- 172.21.15.16:0/2604509910 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x560fb772fb40 0x560fb772ff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:47.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.505+0000 7f8b08a43700 1 -- 172.21.15.16:0/2604509910 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560fb7756f30 msgr2=0x560fb772f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:47.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.505+0000 7f8b08a43700 1 --2- 172.21.15.16:0/2604509910 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560fb7756f30 0x560fb772f280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:47.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.505+0000 7f8b08a43700 1 -- 172.21.15.16:0/2604509910 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560fb763f3a0 con 0x560fb772af00 2023-10-16T13:47:47.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.505+0000 7f8b03fff700 1 --2- 172.21.15.16:0/2604509910 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x560fb772fb40 0x560fb772ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:47:47.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.505+0000 7f8b08a43700 1 --2- 172.21.15.16:0/2604509910 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560fb772af00 0x560fb7756300 secure :-1 s=READY pgs=2822 cs=0 l=1 rev1=1 crypto rx=0x7f8b040cdfc0 tx=0x7f8afc010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0fa006cf67e572c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:47.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.505+0000 7f8af9ffb700 1 -- 172.21.15.16:0/2604509910 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8afc003a20 con 0x560fb772af00 2023-10-16T13:47:47.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.505+0000 7f8af9ffb700 1 -- 172.21.15.16:0/2604509910 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8afc003b80 con 0x560fb772af00 2023-10-16T13:47:47.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.505+0000 7f8af9ffb700 1 -- 172.21.15.16:0/2604509910 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8afc004040 con 0x560fb772af00 2023-10-16T13:47:47.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.506+0000 7f8b16690b80 1 -- 172.21.15.16:0/2604509910 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560fb772af00 msgr2=0x560fb7756300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:47.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.506+0000 7f8b16690b80 1 --2- 172.21.15.16:0/2604509910 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560fb772af00 0x560fb7756300 secure :-1 s=READY pgs=2822 cs=0 l=1 rev1=1 crypto rx=0x7f8b040cdfc0 tx=0x7f8afc010000 comp rx=0 tx=0).stop 2023-10-16T13:47:47.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.506+0000 7f8b16690b80 1 -- 172.21.15.16:0/2604509910 shutdown_connections 2023-10-16T13:47:47.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.506+0000 7f8b16690b80 1 --2- 172.21.15.16:0/2604509910 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560fb772af00 0x560fb7756300 unknown :-1 s=CLOSED pgs=2822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:47.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.506+0000 7f8b16690b80 1 --2- 172.21.15.16:0/2604509910 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x560fb772fb40 0x560fb772ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:47.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.506+0000 7f8b16690b80 1 --2- 172.21.15.16:0/2604509910 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560fb7756f30 0x560fb772f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:47.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.506+0000 7f8b16690b80 1 -- 172.21.15.16:0/2604509910 >> 172.21.15.16:0/2604509910 conn(0x560fb765cd80 msgr2=0x560fb7777ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:47.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.506+0000 7f8b16690b80 1 -- 172.21.15.16:0/2604509910 shutdown_connections 2023-10-16T13:47:47.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.506+0000 7f8b16690b80 1 -- 172.21.15.16:0/2604509910 wait complete. 2023-10-16T13:47:47.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.507+0000 7f8b16690b80 1 -- start start 2023-10-16T13:47:47.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.507+0000 7f8b16690b80 1 Processor -- start 2023-10-16T13:47:47.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.508+0000 7f8b16690b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560fb772af00 0x560fb774fc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:47.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.508+0000 7f8b16690b80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x560fb772fb40 0x560fb77593a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:47.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.508+0000 7f8b16690b80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560fb7756f30 0x560fb7776520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:47.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.508+0000 7f8b16690b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x560fb7681e20 con 0x560fb772af00 2023-10-16T13:47:47.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.508+0000 7f8b16690b80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x560fb761e6c0 con 0x560fb7756f30 2023-10-16T13:47:47.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.508+0000 7f8b16690b80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x560fb761e360 con 0x560fb772fb40 2023-10-16T13:47:47.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.508+0000 7f8b03fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560fb772af00 0x560fb774fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:47.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.508+0000 7f8b16690b80 10 client.0 _ll_register_callbacks cb 0x560fb747b7b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T13:47:47.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.508+0000 7f8b03fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560fb772af00 0x560fb774fc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55950/0 (socket says 172.21.15.16:55950) 2023-10-16T13:47:47.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.508+0000 7f8b03fff700 1 -- 172.21.15.16:0/3656902403 learned_addr learned my addr 172.21.15.16:0/3656902403 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:47.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.508+0000 7f8b08a43700 1 --2- 172.21.15.16:0/3656902403 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560fb7756f30 0x560fb7776520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:47.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.508+0000 7f8b037fe700 1 --2- 172.21.15.16:0/3656902403 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x560fb772fb40 0x560fb77593a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:47.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.508+0000 7f8b03fff700 1 -- 172.21.15.16:0/3656902403 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x560fb772fb40 msgr2=0x560fb77593a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:47.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.508+0000 7f8b03fff700 1 --2- 172.21.15.16:0/3656902403 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x560fb772fb40 0x560fb77593a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:47.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.508+0000 7f8b03fff700 1 -- 172.21.15.16:0/3656902403 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560fb7756f30 msgr2=0x560fb7776520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:47.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.508+0000 7f8b03fff700 1 --2- 172.21.15.16:0/3656902403 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560fb7756f30 0x560fb7776520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:47.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.508+0000 7f8b03fff700 1 -- 172.21.15.16:0/3656902403 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b0415b630 con 0x560fb772af00 2023-10-16T13:47:47.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.508+0000 7f8b037fe700 1 --2- 172.21.15.16:0/3656902403 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x560fb772fb40 0x560fb77593a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:47:47.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.508+0000 7f8b03fff700 1 --2- 172.21.15.16:0/3656902403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560fb772af00 0x560fb774fc40 secure :-1 s=READY pgs=2823 cs=0 l=1 rev1=1 crypto rx=0x7f8aec053dc0 tx=0x7f8aec057510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c89302d7351a1c3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:47.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.508+0000 7f8ae2ffd700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2023-10-16T13:47:47.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.508+0000 7f8ae2ffd700 1 -- 172.21.15.16:0/3656902403 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aec05ba10 con 0x560fb772af00 2023-10-16T13:47:47.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.508+0000 7f8ae2ffd700 1 -- 172.21.15.16:0/3656902403 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(249..249 src has 1..249) v4 ==== 7021+0+0 (secure 0 0 0) 0x7f8aec05fbc0 con 0x560fb772af00 2023-10-16T13:47:47.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.508+0000 7f8b16690b80 1 -- 172.21.15.16:0/3656902403 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560fb77225b0 con 0x560fb772af00 2023-10-16T13:47:47.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.509+0000 7f8ae2ffd700 1 -- 172.21.15.16:0/3656902403 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=135) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8aec056db0 con 0x560fb772af00 2023-10-16T13:47:47.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.509+0000 7f8b16690b80 10 client.12561 fetch_fsmap learned FSMap version 135 2023-10-16T13:47:47.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.509+0000 7f8b16690b80 1 -- 172.21.15.16:0/3656902403 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=135,monmap=2+}) v3 -- 0x560fb74acd90 con 0x560fb772af00 2023-10-16T13:47:47.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.509+0000 7f8ae2ffd700 1 -- 172.21.15.16:0/3656902403 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x560fb77225b0 con 0x560fb772af00 2023-10-16T13:47:47.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.509+0000 7f8ae2ffd700 1 -- 172.21.15.16:0/3656902403 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 135) v1 ==== 3891+0+0 (secure 0 0 0) 0x7f8aec05f730 con 0x560fb772af00 2023-10-16T13:47:47.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.509+0000 7f8b16690b80 10 client.12561 fetch_fsmap finished waiting for FSMap version 135 2023-10-16T13:47:47.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.509+0000 7f8b16690b80 10 client.12561 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-16T13:47:47.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.509+0000 7f8b16690b80 20 client.12561 populate_metadata read hostname 'smithi016' 2023-10-16T13:47:47.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.509+0000 7f8b16690b80 1 --2- 172.21.15.16:0/3656902403 >> [v2:172.21.15.49:6833/680748068,v1:172.21.15.49:6835/680748068] conn(0x560fb7729e70 0x560fb772a250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:47.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.509+0000 7f8b16690b80 4 client.12561 mds_command: new command op to 11197 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T13:47:47.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.509+0000 7f8b16690b80 1 -- 172.21.15.16:0/3656902403 --> [v2:172.21.15.49:6833/680748068,v1:172.21.15.49:6835/680748068] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560fb767ae00 con 0x560fb7729e70 2023-10-16T13:47:47.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.509+0000 7f8b037fe700 1 --2- 172.21.15.16:0/3656902403 >> [v2:172.21.15.49:6833/680748068,v1:172.21.15.49:6835/680748068] conn(0x560fb7729e70 0x560fb772a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:47.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.510+0000 7f8ae2ffd700 10 client.12561 ms_handle_connect on v2:172.21.15.49:6833/680748068 2023-10-16T13:47:47.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.510+0000 7f8b037fe700 1 --2- 172.21.15.16:0/3656902403 >> [v2:172.21.15.49:6833/680748068,v1:172.21.15.49:6835/680748068] conn(0x560fb7729e70 0x560fb772a250 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=25e4d3d18c8a193a server_cookie=ca0e68cbc7cdcd12 in_seq=0 out_seq=0 2023-10-16T13:47:47.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.510+0000 7f8ae2ffd700 1 -- 172.21.15.16:0/3656902403 <== mds.0 v2:172.21.15.49:6833/680748068 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x560fb767ae00 con 0x560fb7729e70 2023-10-16T13:47:47.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.510+0000 7f8ae2ffd700 10 client.12561 handle_command_reply: tid=0 2023-10-16T13:47:47.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.510+0000 7f8b16690b80 1 client.12561 shutdown 2023-10-16T13:47:47.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.510+0000 7f8b16690b80 1 -- 172.21.15.16:0/3656902403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560fb772af00 msgr2=0x560fb774fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:47.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.510+0000 7f8b16690b80 1 --2- 172.21.15.16:0/3656902403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560fb772af00 0x560fb774fc40 secure :-1 s=READY pgs=2823 cs=0 l=1 rev1=1 crypto rx=0x7f8aec053dc0 tx=0x7f8aec057510 comp rx=0 tx=0).stop 2023-10-16T13:47:47.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.511+0000 7f8b16690b80 1 -- 172.21.15.16:0/3656902403 shutdown_connections 2023-10-16T13:47:47.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.511+0000 7f8b16690b80 1 --2- 172.21.15.16:0/3656902403 >> [v2:172.21.15.49:6833/680748068,v1:172.21.15.49:6835/680748068] conn(0x560fb7729e70 0x560fb772a250 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:47.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.511+0000 7f8b16690b80 1 --2- 172.21.15.16:0/3656902403 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560fb7756f30 0x560fb7776520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:47.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.511+0000 7f8b16690b80 1 --2- 172.21.15.16:0/3656902403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560fb772af00 0x560fb774fc40 unknown :-1 s=CLOSED pgs=2823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:47.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.511+0000 7f8ae2ffd700 0 client.12561 ms_handle_reset on v2:172.21.15.49:6833/680748068 2023-10-16T13:47:47.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.511+0000 7f8b16690b80 1 --2- 172.21.15.16:0/3656902403 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x560fb772fb40 0x560fb77593a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:47.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.511+0000 7f8b16690b80 1 -- 172.21.15.16:0/3656902403 >> 172.21.15.16:0/3656902403 conn(0x560fb765cd80 msgr2=0x560fb772f1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:47.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.511+0000 7f8b16690b80 1 -- 172.21.15.16:0/3656902403 shutdown_connections 2023-10-16T13:47:47.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.511+0000 7f8b16690b80 1 -- 172.21.15.16:0/3656902403 wait complete. 2023-10-16T13:47:47.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.511+0000 7f8b16690b80 20 client.12561 trim_cache size 0 max 16384 2023-10-16T13:47:47.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.529+0000 7f8b16690b80 1 Processor -- start 2023-10-16T13:47:47.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.529+0000 7f8b16690b80 1 -- start start 2023-10-16T13:47:47.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.529+0000 7f8b16690b80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560fb772af00 0x560fb772e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:47.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.529+0000 7f8b16690b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560fb772fb40 0x560fb77562e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:47.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.529+0000 7f8b16690b80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x560fb7756f30 0x560fb7750540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:47.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.529+0000 7f8b16690b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x560fb75028d0 con 0x560fb772fb40 2023-10-16T13:47:47.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.529+0000 7f8b16690b80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x560fb77053c0 con 0x560fb772af00 2023-10-16T13:47:47.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.529+0000 7f8b16690b80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x560fb7759ce0 con 0x560fb7756f30 2023-10-16T13:47:47.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.530+0000 7f8b037fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560fb772fb40 0x560fb77562e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:47.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.530+0000 7f8b037fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560fb772fb40 0x560fb77562e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55960/0 (socket says 172.21.15.16:55960) 2023-10-16T13:47:47.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.530+0000 7f8b037fe700 1 -- 172.21.15.16:0/6147093 learned_addr learned my addr 172.21.15.16:0/6147093 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:47.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.530+0000 7f8b03fff700 1 --2- 172.21.15.16:0/6147093 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560fb772af00 0x560fb772e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:47.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.530+0000 7f8b08a43700 1 --2- 172.21.15.16:0/6147093 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x560fb7756f30 0x560fb7750540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:47.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.530+0000 7f8b037fe700 1 -- 172.21.15.16:0/6147093 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x560fb7756f30 msgr2=0x560fb7750540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:47.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.530+0000 7f8b037fe700 1 --2- 172.21.15.16:0/6147093 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x560fb7756f30 0x560fb7750540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:47.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.530+0000 7f8b037fe700 1 -- 172.21.15.16:0/6147093 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560fb772af00 msgr2=0x560fb772e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:47.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.530+0000 7f8b037fe700 1 --2- 172.21.15.16:0/6147093 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560fb772af00 0x560fb772e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:47.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.530+0000 7f8b037fe700 1 -- 172.21.15.16:0/6147093 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560fb7759fb0 con 0x560fb772fb40 2023-10-16T13:47:47.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.530+0000 7f8b037fe700 1 --2- 172.21.15.16:0/6147093 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560fb772fb40 0x560fb77562e0 secure :-1 s=READY pgs=2824 cs=0 l=1 rev1=1 crypto rx=0x7f8b040b6520 tx=0x7f8af4009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc61d34383656a6f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:47.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.530+0000 7f8afa7fc700 1 -- 172.21.15.16:0/6147093 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8af4015df0 con 0x560fb772fb40 2023-10-16T13:47:47.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.530+0000 7f8afa7fc700 1 -- 172.21.15.16:0/6147093 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8af4027150 con 0x560fb772fb40 2023-10-16T13:47:47.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.530+0000 7f8afa7fc700 1 -- 172.21.15.16:0/6147093 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8af40274e0 con 0x560fb772fb40 2023-10-16T13:47:47.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.531+0000 7f8b16690b80 1 -- 172.21.15.16:0/6147093 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560fb772fb40 msgr2=0x560fb77562e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:47.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.531+0000 7f8b16690b80 1 --2- 172.21.15.16:0/6147093 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560fb772fb40 0x560fb77562e0 secure :-1 s=READY pgs=2824 cs=0 l=1 rev1=1 crypto rx=0x7f8b040b6520 tx=0x7f8af4009f90 comp rx=0 tx=0).stop 2023-10-16T13:47:47.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.531+0000 7f8b16690b80 1 -- 172.21.15.16:0/6147093 shutdown_connections 2023-10-16T13:47:47.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.531+0000 7f8b16690b80 1 --2- 172.21.15.16:0/6147093 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x560fb7756f30 0x560fb7750540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:47.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.531+0000 7f8b16690b80 1 --2- 172.21.15.16:0/6147093 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560fb772af00 0x560fb772e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:47.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.531+0000 7f8b16690b80 1 --2- 172.21.15.16:0/6147093 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560fb772fb40 0x560fb77562e0 unknown :-1 s=CLOSED pgs=2824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:47.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.531+0000 7f8b16690b80 1 -- 172.21.15.16:0/6147093 >> 172.21.15.16:0/6147093 conn(0x560fb765cd80 msgr2=0x560fb775a1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:47.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.531+0000 7f8b16690b80 1 -- 172.21.15.16:0/6147093 shutdown_connections 2023-10-16T13:47:47.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.531+0000 7f8b16690b80 1 -- 172.21.15.16:0/6147093 wait complete. 2023-10-16T13:47:47.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.531+0000 7f8b16690b80 1 -- start start 2023-10-16T13:47:47.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.532+0000 7f8b16690b80 1 Processor -- start 2023-10-16T13:47:47.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.532+0000 7f8b16690b80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x560fb772af00 0x560fb77e3100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:47.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.532+0000 7f8b16690b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560fb772fb40 0x560fb772d590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:47.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.532+0000 7f8b16690b80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560fb7756f30 0x560fb7791d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:47.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.532+0000 7f8b16690b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x560fb772d170 con 0x560fb772fb40 2023-10-16T13:47:47.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.532+0000 7f8b16690b80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x560fb7778c30 con 0x560fb7756f30 2023-10-16T13:47:47.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.532+0000 7f8b16690b80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x560fb77e3640 con 0x560fb772af00 2023-10-16T13:47:47.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.532+0000 7f8b16690b80 10 client.0 _ll_register_callbacks cb 0x7f8aec057220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T13:47:47.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.532+0000 7f8b037fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560fb772fb40 0x560fb772d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:47.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.533+0000 7f8b037fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560fb772fb40 0x560fb772d590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55966/0 (socket says 172.21.15.16:55966) 2023-10-16T13:47:47.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.533+0000 7f8b037fe700 1 -- 172.21.15.16:0/1450876868 learned_addr learned my addr 172.21.15.16:0/1450876868 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:47.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.533+0000 7f8b08a43700 1 --2- 172.21.15.16:0/1450876868 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560fb7756f30 0x560fb7791d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:47.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.533+0000 7f8b03fff700 1 --2- 172.21.15.16:0/1450876868 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x560fb772af00 0x560fb77e3100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:47.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.533+0000 7f8b037fe700 1 -- 172.21.15.16:0/1450876868 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x560fb772af00 msgr2=0x560fb77e3100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:47.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.533+0000 7f8b037fe700 1 --2- 172.21.15.16:0/1450876868 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x560fb772af00 0x560fb77e3100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:47.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.533+0000 7f8b037fe700 1 -- 172.21.15.16:0/1450876868 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560fb7756f30 msgr2=0x560fb7791d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:47.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.533+0000 7f8b037fe700 1 --2- 172.21.15.16:0/1450876868 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560fb7756f30 0x560fb7791d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:47.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.533+0000 7f8b037fe700 1 -- 172.21.15.16:0/1450876868 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8af401d040 con 0x560fb772fb40 2023-10-16T13:47:47.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.533+0000 7f8b037fe700 1 --2- 172.21.15.16:0/1450876868 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560fb772fb40 0x560fb772d590 secure :-1 s=READY pgs=2825 cs=0 l=1 rev1=1 crypto rx=0x7f8af4025040 tx=0x7f8af4011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9446f9f72eaa404 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:47.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.533+0000 7f8ae2ffd700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2023-10-16T13:47:47.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.533+0000 7f8ae2ffd700 1 -- 172.21.15.16:0/1450876868 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8af4019070 con 0x560fb772fb40 2023-10-16T13:47:47.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.533+0000 7f8ae2ffd700 1 -- 172.21.15.16:0/1450876868 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(249..249 src has 1..249) v4 ==== 7021+0+0 (secure 0 0 0) 0x7f8af40272c0 con 0x560fb772fb40 2023-10-16T13:47:47.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.533+0000 7f8b16690b80 1 -- 172.21.15.16:0/1450876868 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8afc003b80 con 0x560fb772fb40 2023-10-16T13:47:47.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.533+0000 7f8ae2ffd700 1 -- 172.21.15.16:0/1450876868 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=135) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8af4027480 con 0x560fb772fb40 2023-10-16T13:47:47.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.533+0000 7f8b16690b80 10 client.12567 fetch_fsmap learned FSMap version 135 2023-10-16T13:47:47.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.533+0000 7f8b16690b80 1 -- 172.21.15.16:0/1450876868 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=135,monmap=2+}) v3 -- 0x560fb7495c10 con 0x560fb772fb40 2023-10-16T13:47:47.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.533+0000 7f8ae2ffd700 1 -- 172.21.15.16:0/1450876868 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8afc003b80 con 0x560fb772fb40 2023-10-16T13:47:47.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.534+0000 7f8ae2ffd700 1 -- 172.21.15.16:0/1450876868 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 135) v1 ==== 3891+0+0 (secure 0 0 0) 0x7f8af403a420 con 0x560fb772fb40 2023-10-16T13:47:47.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.534+0000 7f8b16690b80 10 client.12567 fetch_fsmap finished waiting for FSMap version 135 2023-10-16T13:47:47.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.534+0000 7f8b16690b80 10 client.12567 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-16T13:47:47.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.534+0000 7f8b16690b80 20 client.12567 populate_metadata read hostname 'smithi016' 2023-10-16T13:47:47.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.534+0000 7f8b16690b80 1 --2- 172.21.15.16:0/1450876868 >> [v2:172.21.15.49:6833/680748068,v1:172.21.15.49:6835/680748068] conn(0x560fb77564f0 0x560fb77568d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:47.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.534+0000 7f8b16690b80 4 client.12567 mds_command: new command op to 11197 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T13:47:47.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.534+0000 7f8b16690b80 1 -- 172.21.15.16:0/1450876868 --> [v2:172.21.15.49:6833/680748068,v1:172.21.15.49:6835/680748068] -- command(tid 0: {"prefix": "status"}) v1 -- 0x560fb767ab20 con 0x560fb77564f0 2023-10-16T13:47:47.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.534+0000 7f8b03fff700 1 --2- 172.21.15.16:0/1450876868 >> [v2:172.21.15.49:6833/680748068,v1:172.21.15.49:6835/680748068] conn(0x560fb77564f0 0x560fb77568d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:47.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.534+0000 7f8ae2ffd700 10 client.12567 ms_handle_connect on v2:172.21.15.49:6833/680748068 2023-10-16T13:47:47.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.534+0000 7f8b03fff700 1 --2- 172.21.15.16:0/1450876868 >> [v2:172.21.15.49:6833/680748068,v1:172.21.15.49:6835/680748068] conn(0x560fb77564f0 0x560fb77568d0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bda3c4c99f0d101b server_cookie=5cce842081530ae in_seq=0 out_seq=0 2023-10-16T13:47:47.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.534+0000 7f8ae2ffd700 1 -- 172.21.15.16:0/1450876868 <== mds.0 v2:172.21.15.49:6833/680748068 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+338 (crc 0 0 0) 0x560fb767ab20 con 0x560fb77564f0 2023-10-16T13:47:47.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.534+0000 7f8ae2ffd700 10 client.12567 handle_command_reply: tid=0 2023-10-16T13:47:47.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.534+0000 7f8b16690b80 1 client.12567 shutdown 2023-10-16T13:47:47.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.535+0000 7f8b16690b80 1 -- 172.21.15.16:0/1450876868 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560fb772fb40 msgr2=0x560fb772d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:47.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.535+0000 7f8b16690b80 1 --2- 172.21.15.16:0/1450876868 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560fb772fb40 0x560fb772d590 secure :-1 s=READY pgs=2825 cs=0 l=1 rev1=1 crypto rx=0x7f8af4025040 tx=0x7f8af4011000 comp rx=0 tx=0).stop 2023-10-16T13:47:47.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.535+0000 7f8b16690b80 1 -- 172.21.15.16:0/1450876868 shutdown_connections 2023-10-16T13:47:47.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.535+0000 7f8b16690b80 1 --2- 172.21.15.16:0/1450876868 >> [v2:172.21.15.49:6833/680748068,v1:172.21.15.49:6835/680748068] conn(0x560fb77564f0 0x560fb77568d0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:47.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.535+0000 7f8b16690b80 1 --2- 172.21.15.16:0/1450876868 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560fb7756f30 0x560fb7791d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:47.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.535+0000 7f8ae2ffd700 0 client.12567 ms_handle_reset on v2:172.21.15.49:6833/680748068 2023-10-16T13:47:47.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.535+0000 7f8b16690b80 1 --2- 172.21.15.16:0/1450876868 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x560fb772af00 0x560fb77e3100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:47.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.535+0000 7f8b16690b80 1 --2- 172.21.15.16:0/1450876868 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560fb772fb40 0x560fb772d590 unknown :-1 s=CLOSED pgs=2825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:47.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.535+0000 7f8b16690b80 1 -- 172.21.15.16:0/1450876868 >> 172.21.15.16:0/1450876868 conn(0x560fb765cd80 msgr2=0x560fb775a140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:47.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.535+0000 7f8b16690b80 1 -- 172.21.15.16:0/1450876868 shutdown_connections 2023-10-16T13:47:47.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.535+0000 7f8b16690b80 1 -- 172.21.15.16:0/1450876868 wait complete. 2023-10-16T13:47:47.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.535+0000 7f8b16690b80 20 client.12567 trim_cache size 0 max 16384 2023-10-16T13:47:47.557 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-10-16T13:47:47.558 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "5b28dfa1-b21f-474a-9b2e-9fb77af58254", 2023-10-16T13:47:47.558 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 0, 2023-10-16T13:47:47.558 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 11197, 2023-10-16T13:47:47.558 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2023-10-16T13:47:47.558 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2023-10-16T13:47:47.559 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "backup_fs", 2023-10-16T13:47:47.559 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 5.6145044559999997, 2023-10-16T13:47:47.559 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 135, 2023-10-16T13:47:47.559 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 249, 2023-10-16T13:47:47.559 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 249, 2023-10-16T13:47:47.560 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 27.996149255999999 2023-10-16T13:47:47.560 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-10-16T13:47:47.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.535+0000 7f8b0a4a6700 1 -- 172.21.15.16:0/497541202 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f8ae406b140 msgr2=0x7f8ae406d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:47.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.535+0000 7f8b0a4a6700 1 --2- 172.21.15.16:0/497541202 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f8ae406b140 0x7f8ae406d600 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f8b0414d600 tx=0x7f8af4008040 comp rx=0 tx=0).stop 2023-10-16T13:47:47.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.535+0000 7f8b0a4a6700 1 -- 172.21.15.16:0/497541202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b040b5830 msgr2=0x7f8b040c9ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:47.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.535+0000 7f8b0a4a6700 1 --2- 172.21.15.16:0/497541202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b040b5830 0x7f8b040c9ec0 secure :-1 s=READY pgs=2821 cs=0 l=1 rev1=1 crypto rx=0x7f8aec00c510 tx=0x7f8aec00c8d0 comp rx=0 tx=0).stop 2023-10-16T13:47:47.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.535+0000 7f8b0a4a6700 1 -- 172.21.15.16:0/497541202 shutdown_connections 2023-10-16T13:47:47.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.535+0000 7f8b0a4a6700 1 --2- 172.21.15.16:0/497541202 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f8ae406b140 0x7f8ae406d600 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:47.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.535+0000 7f8b0a4a6700 1 --2- 172.21.15.16:0/497541202 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8b040cf880 0x7f8b040b11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:47.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.535+0000 7f8b0a4a6700 1 --2- 172.21.15.16:0/497541202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b040b5830 0x7f8b040c9ec0 unknown :-1 s=CLOSED pgs=2821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:47.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.535+0000 7f8b0a4a6700 1 --2- 172.21.15.16:0/497541202 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8b040cf4a0 0x7f8b04156270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:47.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.535+0000 7f8b0a4a6700 1 -- 172.21.15.16:0/497541202 >> 172.21.15.16:0/497541202 conn(0x7f8b040b1fd0 msgr2=0x7f8b040afe40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:47.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.535+0000 7f8b0a4a6700 1 -- 172.21.15.16:0/497541202 shutdown_connections 2023-10-16T13:47:47.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.535+0000 7f8b0a4a6700 1 -- 172.21.15.16:0/497541202 wait complete. 2023-10-16T13:47:47.564 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-16T13:47:47.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.686+0000 7f5a684ad700 1 Processor -- start 2023-10-16T13:47:47.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.686+0000 7f5a684ad700 1 -- start start 2023-10-16T13:47:47.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.687+0000 7f5a684ad700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5a600cc8a0 0x7f5a6015e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:47.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.687+0000 7f5a684ad700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a60161c30 con 0x7f5a6015b280 2023-10-16T13:47:47.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.687+0000 7f5a684ad700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a60161f10 con 0x7f5a6006a7e0 2023-10-16T13:47:47.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.687+0000 7f5a684ad700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a601621f0 con 0x7f5a600cc8a0 2023-10-16T13:47:47.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.687+0000 7f5a66249700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f5a6015b280 0x7f5a601613e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:45246/0 (socket says 172.21.15.16:45246) 2023-10-16T13:47:47.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.687+0000 7f5a66249700 1 -- 172.21.15.16:0/3224790403 learned_addr learned my addr 172.21.15.16:0/3224790403 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:47:47.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.687+0000 7f5a65a48700 1 --2- 172.21.15.16:0/3224790403 >> v2:172.21.15.16:3300/0 conn(0x7f5a600cc8a0 0x7f5a6015e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:47.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.687+0000 7f5a65a48700 1 -- 172.21.15.16:0/3224790403 >> v1:172.21.15.49:6789/0 conn(0x7f5a6006a7e0 legacy=0x7f5a600d02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:47.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.687+0000 7f5a65a48700 1 -- 172.21.15.16:0/3224790403 >> v1:172.21.15.16:6789/0 conn(0x7f5a6015b280 legacy=0x7f5a601613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:47.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.687+0000 7f5a65a48700 1 -- 172.21.15.16:0/3224790403 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a6015df00 con 0x7f5a600cc8a0 2023-10-16T13:47:47.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.688+0000 7f5a65a48700 1 --2- 172.21.15.16:0/3224790403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a600cc8a0 0x7f5a6015e700 secure :-1 s=READY pgs=2827 cs=0 l=1 rev1=1 crypto rx=0x7f5a5c009840 tx=0x7f5a5c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=690059d40813273b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:47.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.688+0000 7f5a65247700 1 -- 172.21.15.16:0/3224790403 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a5c011830 con 0x7f5a600cc8a0 2023-10-16T13:47:47.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.688+0000 7f5a65247700 1 -- 172.21.15.16:0/3224790403 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a5c011990 con 0x7f5a600cc8a0 2023-10-16T13:47:47.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.688+0000 7f5a65247700 1 -- 172.21.15.16:0/3224790403 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a5c011d20 con 0x7f5a600cc8a0 2023-10-16T13:47:47.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.688+0000 7f5a684ad700 1 -- 172.21.15.16:0/3224790403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a600cc8a0 msgr2=0x7f5a6015e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:47.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.688+0000 7f5a684ad700 1 --2- 172.21.15.16:0/3224790403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a600cc8a0 0x7f5a6015e700 secure :-1 s=READY pgs=2827 cs=0 l=1 rev1=1 crypto rx=0x7f5a5c009840 tx=0x7f5a5c009b50 comp rx=0 tx=0).stop 2023-10-16T13:47:47.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.688+0000 7f5a684ad700 1 -- 172.21.15.16:0/3224790403 shutdown_connections 2023-10-16T13:47:47.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.688+0000 7f5a684ad700 1 --2- 172.21.15.16:0/3224790403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a600cc8a0 0x7f5a6015e700 unknown :-1 s=CLOSED pgs=2827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:47.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.688+0000 7f5a684ad700 1 -- 172.21.15.16:0/3224790403 >> 172.21.15.16:0/3224790403 conn(0x7f5a600b1fd0 msgr2=0x7f5a600b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:47.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.688+0000 7f5a684ad700 1 -- 172.21.15.16:0/3224790403 shutdown_connections 2023-10-16T13:47:47.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.688+0000 7f5a684ad700 1 -- 172.21.15.16:0/3224790403 wait complete. 2023-10-16T13:47:47.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.690+0000 7f5a684ad700 1 Processor -- start 2023-10-16T13:47:47.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.690+0000 7f5a684ad700 1 -- start start 2023-10-16T13:47:47.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.690+0000 7f5a684ad700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5a6006a7e0 0x7f5a6014bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:47.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.690+0000 7f5a684ad700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5a600cc8a0 0x7f5a601586d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:47.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.690+0000 7f5a684ad700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a6015b280 0x7f5a600ad350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:47.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.690+0000 7f5a684ad700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a60156ed0 con 0x7f5a6015b280 2023-10-16T13:47:47.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.690+0000 7f5a684ad700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a601f6b30 con 0x7f5a600cc8a0 2023-10-16T13:47:47.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.690+0000 7f5a684ad700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a601f6e10 con 0x7f5a6006a7e0 2023-10-16T13:47:47.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.690+0000 7f5a66a4a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a6015b280 0x7f5a600ad350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:47.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.691+0000 7f5a66a4a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a6015b280 0x7f5a600ad350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55982/0 (socket says 172.21.15.16:55982) 2023-10-16T13:47:47.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.691+0000 7f5a66a4a700 1 -- 172.21.15.16:0/4110740739 learned_addr learned my addr 172.21.15.16:0/4110740739 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:47.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.691+0000 7f5a66249700 1 --2- 172.21.15.16:0/4110740739 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5a6006a7e0 0x7f5a6014bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:47.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.691+0000 7f5a66a4a700 1 -- 172.21.15.16:0/4110740739 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5a6006a7e0 msgr2=0x7f5a6014bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:47.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.691+0000 7f5a66a4a700 1 --2- 172.21.15.16:0/4110740739 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5a6006a7e0 0x7f5a6014bfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:47.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.691+0000 7f5a66a4a700 1 -- 172.21.15.16:0/4110740739 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5a600cc8a0 msgr2=0x7f5a601586d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:47.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.691+0000 7f5a66a4a700 1 --2- 172.21.15.16:0/4110740739 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5a600cc8a0 0x7f5a601586d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:47.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.691+0000 7f5a66a4a700 1 -- 172.21.15.16:0/4110740739 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a5c0094a0 con 0x7f5a6015b280 2023-10-16T13:47:47.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.691+0000 7f5a66a4a700 1 --2- 172.21.15.16:0/4110740739 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a6015b280 0x7f5a600ad350 secure :-1 s=READY pgs=2828 cs=0 l=1 rev1=1 crypto rx=0x7f5a5800cc50 tx=0x7f5a5800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:47.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.691+0000 7f5a56ffd700 1 -- 172.21.15.16:0/4110740739 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a5800ad80 con 0x7f5a6015b280 2023-10-16T13:47:47.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.691+0000 7f5a56ffd700 1 -- 172.21.15.16:0/4110740739 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a5800ce50 con 0x7f5a6015b280 2023-10-16T13:47:47.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.691+0000 7f5a56ffd700 1 -- 172.21.15.16:0/4110740739 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a58003c40 con 0x7f5a6015b280 2023-10-16T13:47:47.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.691+0000 7f5a684ad700 1 -- 172.21.15.16:0/4110740739 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a601f70f0 con 0x7f5a6015b280 2023-10-16T13:47:47.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.691+0000 7f5a684ad700 1 -- 172.21.15.16:0/4110740739 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a6014c4f0 con 0x7f5a6015b280 2023-10-16T13:47:47.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.692+0000 7f5a684ad700 1 -- 172.21.15.16:0/4110740739 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a60157040 con 0x7f5a6015b280 2023-10-16T13:47:47.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.693+0000 7f5a56ffd700 1 -- 172.21.15.16:0/4110740739 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 148) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f5a580050d0 con 0x7f5a6015b280 2023-10-16T13:47:47.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.693+0000 7f5a56ffd700 1 --2- 172.21.15.16:0/4110740739 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f5a4406b140 0x7f5a4406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:47.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.693+0000 7f5a56ffd700 1 -- 172.21.15.16:0/4110740739 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 7021+0+0 (secure 0 0 0) 0x7f5a5808a040 con 0x7f5a6015b280 2023-10-16T13:47:47.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.693+0000 7f5a66249700 1 --2- 172.21.15.16:0/4110740739 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f5a4406b140 0x7f5a4406d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:47.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.693+0000 7f5a66249700 1 --2- 172.21.15.16:0/4110740739 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f5a4406b140 0x7f5a4406d600 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f5a50005910 tx=0x7f5a50009040 comp rx=0 tx=0).ready entity=mgr.11886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:47.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.698+0000 7f5a56ffd700 1 -- 172.21.15.16:0/4110740739 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5a58054870 con 0x7f5a6015b280 2023-10-16T13:47:47.939 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.936+0000 7f5a684ad700 1 -- 172.21.15.16:0/4110740739 --> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f5a6015e1e0 con 0x7f5a4406b140 2023-10-16T13:47:47.941 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.937+0000 7f5a56ffd700 1 -- 172.21.15.16:0/4110740739 <== mgr.11886 v2:172.21.15.16:6832/1736104576 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+82 (secure 0 0 0) 0x7f5a50001340 con 0x7f5a4406b140 2023-10-16T13:47:47.941 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2023-10-16T13:47:47.941 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:47:47.942 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cephfs" 2023-10-16T13:47:47.942 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:47:47.942 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:47:47.942 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "backup_fs" 2023-10-16T13:47:47.942 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:47:47.943 INFO:teuthology.orchestra.run.smithi016.stdout:] 2023-10-16T13:47:47.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.940+0000 7f5a684ad700 1 -- 172.21.15.16:0/4110740739 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f5a4406b140 msgr2=0x7f5a4406d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:47.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.940+0000 7f5a684ad700 1 --2- 172.21.15.16:0/4110740739 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f5a4406b140 0x7f5a4406d600 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f5a50005910 tx=0x7f5a50009040 comp rx=0 tx=0).stop 2023-10-16T13:47:47.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.940+0000 7f5a684ad700 1 -- 172.21.15.16:0/4110740739 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a6015b280 msgr2=0x7f5a600ad350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:47.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.940+0000 7f5a684ad700 1 --2- 172.21.15.16:0/4110740739 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a6015b280 0x7f5a600ad350 secure :-1 s=READY pgs=2828 cs=0 l=1 rev1=1 crypto rx=0x7f5a5800cc50 tx=0x7f5a5800a340 comp rx=0 tx=0).stop 2023-10-16T13:47:47.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.940+0000 7f5a684ad700 1 -- 172.21.15.16:0/4110740739 shutdown_connections 2023-10-16T13:47:47.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.940+0000 7f5a684ad700 1 --2- 172.21.15.16:0/4110740739 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f5a4406b140 0x7f5a4406d600 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:47.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.940+0000 7f5a684ad700 1 --2- 172.21.15.16:0/4110740739 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a6015b280 0x7f5a600ad350 unknown :-1 s=CLOSED pgs=2828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:47.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.940+0000 7f5a684ad700 1 --2- 172.21.15.16:0/4110740739 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5a6006a7e0 0x7f5a6014bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:47.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.940+0000 7f5a684ad700 1 --2- 172.21.15.16:0/4110740739 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5a600cc8a0 0x7f5a601586d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:47.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.940+0000 7f5a684ad700 1 -- 172.21.15.16:0/4110740739 >> 172.21.15.16:0/4110740739 conn(0x7f5a600b1fd0 msgr2=0x7f5a600b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:47.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.940+0000 7f5a684ad700 1 -- 172.21.15.16:0/4110740739 shutdown_connections 2023-10-16T13:47:47.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:47.940+0000 7f5a684ad700 1 -- 172.21.15.16:0/4110740739 wait complete. 2023-10-16T13:47:47.955 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-16T13:47:48.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:48.080+0000 7f34953c4700 1 Processor -- start 2023-10-16T13:47:48.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:48.080+0000 7f34953c4700 1 -- start start 2023-10-16T13:47:48.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:48.081+0000 7f34953c4700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3490158050 0x7f34900cdf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:48.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:48.081+0000 7f34953c4700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f34900cc1d0 0x7f34900cc660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:48.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:48.081+0000 7f34953c4700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f349006eab0 con 0x7f34900ce470 2023-10-16T13:47:48.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:48.081+0000 7f34953c4700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f349006e4f0 con 0x7f3490158050 2023-10-16T13:47:48.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:48.081+0000 7f34953c4700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f349006e7d0 con 0x7f34900cc1d0 2023-10-16T13:47:48.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:48.081+0000 7f348effd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3490158050 0x7f34900cdf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:48.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:48.081+0000 7f348effd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3490158050 0x7f34900cdf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55986/0 (socket says 172.21.15.16:55986) 2023-10-16T13:47:48.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:48.081+0000 7f348effd700 1 -- 172.21.15.16:0/135401552 learned_addr learned my addr 172.21.15.16:0/135401552 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:48.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:48.081+0000 7f348e7fc700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f34900ce470 0x7f349006da30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:45260/0 (socket says 172.21.15.16:45260) 2023-10-16T13:47:48.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:48.082+0000 7f348f7fe700 1 --2- 172.21.15.16:0/135401552 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f34900cc1d0 0x7f34900cc660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:48.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:48.082+0000 7f348effd700 1 -- 172.21.15.16:0/135401552 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f34900cc1d0 msgr2=0x7f34900cc660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:48.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:48.082+0000 7f348effd700 1 --2- 172.21.15.16:0/135401552 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f34900cc1d0 0x7f34900cc660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:48.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:48.082+0000 7f348effd700 1 -- 172.21.15.16:0/135401552 >> v1:172.21.15.16:6789/0 conn(0x7f34900ce470 legacy=0x7f349006da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:48.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:48.082+0000 7f348effd700 1 -- 172.21.15.16:0/135401552 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34900cb9d0 con 0x7f3490158050 2023-10-16T13:47:48.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:48.082+0000 7f348dffb700 1 -- 172.21.15.16:0/135401552 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 866289054 0 0) 0x7f349006eab0 con 0x7f34900ce470 2023-10-16T13:47:48.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:48.082+0000 7f348effd700 1 --2- 172.21.15.16:0/135401552 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3490158050 0x7f34900cdf30 secure :-1 s=READY pgs=2830 cs=0 l=1 rev1=1 crypto rx=0x7f3480006fd0 tx=0x7f3480009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46162b439a8d5a80 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:48.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:48.082+0000 7f348dffb700 1 -- 172.21.15.16:0/135401552 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3480005e30 con 0x7f3490158050 2023-10-16T13:47:48.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:48.082+0000 7f348dffb700 1 -- 172.21.15.16:0/135401552 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3480004030 con 0x7f3490158050 2023-10-16T13:47:48.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:48.082+0000 7f348dffb700 1 -- 172.21.15.16:0/135401552 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34800043e0 con 0x7f3490158050 2023-10-16T13:47:48.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:48.083+0000 7f34953c4700 1 -- 172.21.15.16:0/135401552 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3490158050 msgr2=0x7f34900cdf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:48.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:48.083+0000 7f34953c4700 1 --2- 172.21.15.16:0/135401552 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3490158050 0x7f34900cdf30 secure :-1 s=READY pgs=2830 cs=0 l=1 rev1=1 crypto rx=0x7f3480006fd0 tx=0x7f3480009bd0 comp rx=0 tx=0).stop 2023-10-16T13:47:48.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:48.083+0000 7f34953c4700 1 -- 172.21.15.16:0/135401552 shutdown_connections 2023-10-16T13:47:48.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:48.083+0000 7f34953c4700 1 --2- 172.21.15.16:0/135401552 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f34900cc1d0 0x7f34900cc660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:48.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:48.083+0000 7f34953c4700 1 --2- 172.21.15.16:0/135401552 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3490158050 0x7f34900cdf30 unknown :-1 s=CLOSED pgs=2830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:48.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:48.083+0000 7f34953c4700 1 -- 172.21.15.16:0/135401552 >> 172.21.15.16:0/135401552 conn(0x7f34900b1ed0 msgr2=0x7f34900b22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:48.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:48.083+0000 7f34953c4700 1 -- 172.21.15.16:0/135401552 shutdown_connections 2023-10-16T13:47:48.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:48.083+0000 7f34953c4700 1 -- 172.21.15.16:0/135401552 wait complete. 2023-10-16T13:47:48.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:48.084+0000 7f34953c4700 1 Processor -- start 2023-10-16T13:47:48.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:48.084+0000 7f34953c4700 1 -- start start 2023-10-16T13:47:48.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:48.085+0000 7f34953c4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34900cc1d0 0x7f349014e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:48.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:48.085+0000 7f34953c4700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f34900ce470 0x7f34900b5750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:48.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:48.085+0000 7f34953c4700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3490158050 0x7f34900cf0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:48.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:48.085+0000 7f34953c4700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f349014ec70 con 0x7f34900cc1d0 2023-10-16T13:47:48.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:48.085+0000 7f34953c4700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34901f3ce0 con 0x7f3490158050 2023-10-16T13:47:48.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:48.085+0000 7f34953c4700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f349015be80 con 0x7f34900ce470 2023-10-16T13:47:48.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:48.085+0000 7f348effd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34900cc1d0 0x7f349014e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:48.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:48.085+0000 7f348effd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34900cc1d0 0x7f349014e730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55992/0 (socket says 172.21.15.16:55992) 2023-10-16T13:47:48.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:48.085+0000 7f348effd700 1 -- 172.21.15.16:0/3894369051 learned_addr learned my addr 172.21.15.16:0/3894369051 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:48.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:48.085+0000 7f348e7fc700 1 --2- 172.21.15.16:0/3894369051 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f34900ce470 0x7f34900b5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:48.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:48.085+0000 7f348f7fe700 1 --2- 172.21.15.16:0/3894369051 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3490158050 0x7f34900cf0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:48.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:48.085+0000 7f348effd700 1 -- 172.21.15.16:0/3894369051 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f34900ce470 msgr2=0x7f34900b5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:48.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:48.085+0000 7f348effd700 1 --2- 172.21.15.16:0/3894369051 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f34900ce470 0x7f34900b5750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:48.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:48.085+0000 7f348effd700 1 -- 172.21.15.16:0/3894369051 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3490158050 msgr2=0x7f34900cf0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:48.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:48.085+0000 7f348effd700 1 --2- 172.21.15.16:0/3894369051 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3490158050 0x7f34900cf0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:48.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:48.085+0000 7f348effd700 1 -- 172.21.15.16:0/3894369051 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3480009500 con 0x7f34900cc1d0 2023-10-16T13:47:48.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:48.085+0000 7f348f7fe700 1 --2- 172.21.15.16:0/3894369051 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3490158050 0x7f34900cf0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:47:48.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:48.085+0000 7f348effd700 1 --2- 172.21.15.16:0/3894369051 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34900cc1d0 0x7f349014e730 secure :-1 s=READY pgs=2831 cs=0 l=1 rev1=1 crypto rx=0x7f3480009c10 tx=0x7f3480004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:48.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:48.086+0000 7f34877fe700 1 -- 172.21.15.16:0/3894369051 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3480003d20 con 0x7f34900cc1d0 2023-10-16T13:47:48.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:48.086+0000 7f34877fe700 1 -- 172.21.15.16:0/3894369051 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f348001c420 con 0x7f34900cc1d0 2023-10-16T13:47:48.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:48.086+0000 7f34953c4700 1 -- 172.21.15.16:0/3894369051 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f349015c160 con 0x7f34900cc1d0 2023-10-16T13:47:48.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:48.086+0000 7f34877fe700 1 -- 172.21.15.16:0/3894369051 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3480003ea0 con 0x7f34900cc1d0 2023-10-16T13:47:48.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:48.086+0000 7f34953c4700 1 -- 172.21.15.16:0/3894369051 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f349015c720 con 0x7f34900cc1d0 2023-10-16T13:47:48.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:48.087+0000 7f34953c4700 1 -- 172.21.15.16:0/3894369051 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f349004ebb0 con 0x7f34900cc1d0 2023-10-16T13:47:48.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:48.087+0000 7f34877fe700 1 -- 172.21.15.16:0/3894369051 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 148) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f348001c7d0 con 0x7f34900cc1d0 2023-10-16T13:47:48.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:48.088+0000 7f34877fe700 1 --2- 172.21.15.16:0/3894369051 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f347006b140 0x7f347006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:48.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:48.088+0000 7f34877fe700 1 -- 172.21.15.16:0/3894369051 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 7021+0+0 (secure 0 0 0) 0x7f348008aee0 con 0x7f34900cc1d0 2023-10-16T13:47:48.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:48.088+0000 7f348e7fc700 1 --2- 172.21.15.16:0/3894369051 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f347006b140 0x7f347006d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:48.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:48.088+0000 7f348e7fc700 1 --2- 172.21.15.16:0/3894369051 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f347006b140 0x7f347006d600 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f3490071bb0 tx=0x7f3478009040 comp rx=0 tx=0).ready entity=mgr.11886 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:48.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:48.092+0000 7f34877fe700 1 -- 172.21.15.16:0/3894369051 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f34800557a0 con 0x7f34900cc1d0 2023-10-16T13:47:48.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:48.358+0000 7f34953c4700 1 -- 172.21.15.16:0/3894369051 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f34900cd1c0 con 0x7f34900cc1d0 2023-10-16T13:47:49.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.347+0000 7f34877fe700 1 -- 172.21.15.16:0/3894369051 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v149) v1 ==== 92+0+0 (secure 0 0 0) 0x7f348002a0a0 con 0x7f34900cc1d0 2023-10-16T13:47:49.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.351+0000 7f34953c4700 1 -- 172.21.15.16:0/3894369051 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f347006b140 msgr2=0x7f347006d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:49.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.351+0000 7f34953c4700 1 --2- 172.21.15.16:0/3894369051 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f347006b140 0x7f347006d600 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f3490071bb0 tx=0x7f3478009040 comp rx=0 tx=0).stop 2023-10-16T13:47:49.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.351+0000 7f34953c4700 1 -- 172.21.15.16:0/3894369051 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34900cc1d0 msgr2=0x7f349014e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:49.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.351+0000 7f34953c4700 1 --2- 172.21.15.16:0/3894369051 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34900cc1d0 0x7f349014e730 secure :-1 s=READY pgs=2831 cs=0 l=1 rev1=1 crypto rx=0x7f3480009c10 tx=0x7f3480004750 comp rx=0 tx=0).stop 2023-10-16T13:47:49.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.353+0000 7f34953c4700 1 -- 172.21.15.16:0/3894369051 shutdown_connections 2023-10-16T13:47:49.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.353+0000 7f34953c4700 1 --2- 172.21.15.16:0/3894369051 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f347006b140 0x7f347006d600 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:49.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.353+0000 7f34953c4700 1 --2- 172.21.15.16:0/3894369051 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3490158050 0x7f34900cf0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:49.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.353+0000 7f34953c4700 1 --2- 172.21.15.16:0/3894369051 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34900cc1d0 0x7f349014e730 unknown :-1 s=CLOSED pgs=2831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:49.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.353+0000 7f34953c4700 1 --2- 172.21.15.16:0/3894369051 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f34900ce470 0x7f34900b5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:49.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.353+0000 7f34953c4700 1 -- 172.21.15.16:0/3894369051 >> 172.21.15.16:0/3894369051 conn(0x7f34900b1ed0 msgr2=0x7f3490158430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:49.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.353+0000 7f34953c4700 1 -- 172.21.15.16:0/3894369051 shutdown_connections 2023-10-16T13:47:49.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.353+0000 7f34953c4700 1 -- 172.21.15.16:0/3894369051 wait complete. 2023-10-16T13:47:49.373 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-16T13:47:49.504 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:47:49.502+0000 7facdb07f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T13:47:49.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.507+0000 7fc957884700 1 Processor -- start 2023-10-16T13:47:49.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.507+0000 7fc957884700 1 -- start start 2023-10-16T13:47:49.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.508+0000 7fc957884700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc9500d2430 0x7fc9500d1520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:49.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.508+0000 7fc957884700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9500d4540 con 0x7fc9500d4060 2023-10-16T13:47:49.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.508+0000 7fc957884700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9500ce7e0 con 0x7fc95015da10 2023-10-16T13:47:49.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.508+0000 7fc957884700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9500cf9b0 con 0x7fc9500d2430 2023-10-16T13:47:49.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.508+0000 7fc954e1f700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc9500d4060 0x7fc9500d1d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:45302/0 (socket says 172.21.15.16:45302) 2023-10-16T13:47:49.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.508+0000 7fc954e1f700 1 -- 172.21.15.16:0/3491045520 learned_addr learned my addr 172.21.15.16:0/3491045520 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:47:49.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.508+0000 7fc955e21700 1 --2- 172.21.15.16:0/3491045520 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc9500d2430 0x7fc9500d1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:49.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.508+0000 7fc947fff700 1 -- 172.21.15.16:0/3491045520 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 989215812 0 0) 0x7fc9500d4540 con 0x7fc9500d4060 2023-10-16T13:47:49.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.508+0000 7fc947fff700 1 -- 172.21.15.16:0/3491045520 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc93c0035b0 con 0x7fc9500d4060 2023-10-16T13:47:49.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.508+0000 7fc947fff700 1 -- 172.21.15.16:0/3491045520 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2123065680 0 0) 0x7fc9500ce7e0 con 0x7fc95015da10 2023-10-16T13:47:49.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.508+0000 7fc947fff700 1 -- 172.21.15.16:0/3491045520 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9500d4540 con 0x7fc95015da10 2023-10-16T13:47:49.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.508+0000 7fc955e21700 1 -- 172.21.15.16:0/3491045520 >> v1:172.21.15.49:6789/0 conn(0x7fc95015da10 legacy=0x7fc9500cf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:49.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.508+0000 7fc955e21700 1 -- 172.21.15.16:0/3491045520 >> v1:172.21.15.16:6789/0 conn(0x7fc9500d4060 legacy=0x7fc9500d1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:49.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.508+0000 7fc955e21700 1 -- 172.21.15.16:0/3491045520 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9500ceac0 con 0x7fc9500d2430 2023-10-16T13:47:49.515 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:49.508+0000 7f01e0c51200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T13:47:49.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.508+0000 7fc955e21700 1 --2- 172.21.15.16:0/3491045520 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc9500d2430 0x7fc9500d1520 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fc948009dc0 tx=0x7fc94800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba5ceb116b28b3e9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:49.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.509+0000 7fc947fff700 1 -- 172.21.15.16:0/3491045520 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc94800e070 con 0x7fc9500d2430 2023-10-16T13:47:49.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.509+0000 7fc947fff700 1 -- 172.21.15.16:0/3491045520 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc94800bb20 con 0x7fc9500d2430 2023-10-16T13:47:49.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.509+0000 7fc947fff700 1 -- 172.21.15.16:0/3491045520 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc948012430 con 0x7fc9500d2430 2023-10-16T13:47:49.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.509+0000 7fc957884700 1 -- 172.21.15.16:0/3491045520 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc9500d2430 msgr2=0x7fc9500d1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:49.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.509+0000 7fc957884700 1 --2- 172.21.15.16:0/3491045520 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc9500d2430 0x7fc9500d1520 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fc948009dc0 tx=0x7fc94800b330 comp rx=0 tx=0).stop 2023-10-16T13:47:49.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.509+0000 7fc957884700 1 -- 172.21.15.16:0/3491045520 shutdown_connections 2023-10-16T13:47:49.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.509+0000 7fc957884700 1 --2- 172.21.15.16:0/3491045520 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc9500d2430 0x7fc9500d1520 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:49.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.509+0000 7fc957884700 1 -- 172.21.15.16:0/3491045520 >> 172.21.15.16:0/3491045520 conn(0x7fc95009c9f0 msgr2=0x7fc9500d3740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:49.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.509+0000 7fc957884700 1 -- 172.21.15.16:0/3491045520 shutdown_connections 2023-10-16T13:47:49.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.509+0000 7fc957884700 1 -- 172.21.15.16:0/3491045520 wait complete. 2023-10-16T13:47:49.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.511+0000 7fc957884700 1 Processor -- start 2023-10-16T13:47:49.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.511+0000 7fc957884700 1 -- start start 2023-10-16T13:47:49.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.511+0000 7fc957884700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc9500d2430 0x7fc950155010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:49.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.511+0000 7fc957884700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc9500d4060 0x7fc9500b1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:49.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.511+0000 7fc957884700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc95015da10 0x7fc9500b1640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:49.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.511+0000 7fc957884700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9500ca940 con 0x7fc95015da10 2023-10-16T13:47:49.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.511+0000 7fc957884700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9500cac20 con 0x7fc9500d2430 2023-10-16T13:47:49.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.511+0000 7fc957884700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9500caf00 con 0x7fc9500d4060 2023-10-16T13:47:49.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.511+0000 7fc955e21700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc95015da10 0x7fc9500b1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:49.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.511+0000 7fc955e21700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc95015da10 0x7fc9500b1640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56036/0 (socket says 172.21.15.16:56036) 2023-10-16T13:47:49.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.511+0000 7fc955e21700 1 -- 172.21.15.16:0/3586966315 learned_addr learned my addr 172.21.15.16:0/3586966315 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:49.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.511+0000 7fc955620700 1 --2- 172.21.15.16:0/3586966315 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc9500d2430 0x7fc950155010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:49.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.511+0000 7fc954e1f700 1 --2- 172.21.15.16:0/3586966315 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc9500d4060 0x7fc9500b1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:49.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.512+0000 7fc955e21700 1 -- 172.21.15.16:0/3586966315 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc9500d4060 msgr2=0x7fc9500b1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:49.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.512+0000 7fc955e21700 1 --2- 172.21.15.16:0/3586966315 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc9500d4060 0x7fc9500b1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:49.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.512+0000 7fc955e21700 1 -- 172.21.15.16:0/3586966315 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc9500d2430 msgr2=0x7fc950155010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:49.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.512+0000 7fc955e21700 1 --2- 172.21.15.16:0/3586966315 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc9500d2430 0x7fc950155010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:49.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.512+0000 7fc955e21700 1 -- 172.21.15.16:0/3586966315 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc948009a70 con 0x7fc95015da10 2023-10-16T13:47:49.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.512+0000 7fc955e21700 1 --2- 172.21.15.16:0/3586966315 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc95015da10 0x7fc9500b1640 secure :-1 s=READY pgs=2837 cs=0 l=1 rev1=1 crypto rx=0x7fc948014040 tx=0x7fc948004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:49.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.512+0000 7fc945ffb700 1 -- 172.21.15.16:0/3586966315 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc94801b810 con 0x7fc95015da10 2023-10-16T13:47:49.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.512+0000 7fc945ffb700 1 -- 172.21.15.16:0/3586966315 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc948020070 con 0x7fc95015da10 2023-10-16T13:47:49.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.512+0000 7fc957884700 1 -- 172.21.15.16:0/3586966315 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9500cb1e0 con 0x7fc95015da10 2023-10-16T13:47:49.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.512+0000 7fc945ffb700 1 -- 172.21.15.16:0/3586966315 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc94801bbb0 con 0x7fc95015da10 2023-10-16T13:47:49.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.512+0000 7fc957884700 1 -- 172.21.15.16:0/3586966315 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc950155550 con 0x7fc95015da10 2023-10-16T13:47:49.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.513+0000 7fc957884700 1 -- 172.21.15.16:0/3586966315 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc950067870 con 0x7fc95015da10 2023-10-16T13:47:49.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.514+0000 7fc945ffb700 1 -- 172.21.15.16:0/3586966315 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 149) v1 ==== 89450+0+0 (secure 0 0 0) 0x7fc94801bd10 con 0x7fc95015da10 2023-10-16T13:47:49.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.515+0000 7fc945ffb700 1 --2- 172.21.15.16:0/3586966315 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fc93406b190 0x7fc93406d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:49.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.515+0000 7fc955620700 1 -- 172.21.15.16:0/3586966315 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fc93406b190 msgr2=0x7fc93406d650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1736104576 2023-10-16T13:47:49.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.515+0000 7fc945ffb700 1 -- 172.21.15.16:0/3586966315 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 7021+0+0 (secure 0 0 0) 0x7fc948022070 con 0x7fc95015da10 2023-10-16T13:47:49.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.515+0000 7fc955620700 1 --2- 172.21.15.16:0/3586966315 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fc93406b190 0x7fc93406d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:47:49.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.519+0000 7fc945ffb700 1 -- 172.21.15.16:0/3586966315 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc948056a90 con 0x7fc95015da10 2023-10-16T13:47:49.573 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:47:49.571+0000 7facdb07f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T13:47:49.582 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:49.579+0000 7f01e0c51200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T13:47:49.648 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:47:49.646+0000 7facdb07f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T13:47:49.658 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:49.655+0000 7f01e0c51200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T13:47:49.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.716+0000 7fc955620700 1 -- 172.21.15.16:0/3586966315 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fc93406b190 msgr2=0x7fc93406d650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1736104576 2023-10-16T13:47:49.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.716+0000 7fc955620700 1 --2- 172.21.15.16:0/3586966315 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fc93406b190 0x7fc93406d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:47:49.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.725+0000 7fc957884700 1 -- 172.21.15.16:0/3586966315 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7fc95004ebb0 con 0x7fc95015da10 2023-10-16T13:47:49.729 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:47:49.727+0000 7facdb07f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T13:47:49.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.728+0000 7fc945ffb700 1 -- 172.21.15.16:0/3586966315 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v83)=0 v83) v1 ==== 138+0+0 (secure 0 0 0) 0x7fc94805a0b0 con 0x7fc95015da10 2023-10-16T13:47:49.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.730+0000 7fc957884700 1 -- 172.21.15.16:0/3586966315 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fc93406b190 msgr2=0x7fc93406d650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:47:49.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.730+0000 7fc957884700 1 --2- 172.21.15.16:0/3586966315 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fc93406b190 0x7fc93406d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:49.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.730+0000 7fc957884700 1 -- 172.21.15.16:0/3586966315 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc95015da10 msgr2=0x7fc9500b1640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:49.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.730+0000 7fc957884700 1 --2- 172.21.15.16:0/3586966315 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc95015da10 0x7fc9500b1640 secure :-1 s=READY pgs=2837 cs=0 l=1 rev1=1 crypto rx=0x7fc948014040 tx=0x7fc948004ad0 comp rx=0 tx=0).stop 2023-10-16T13:47:49.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.730+0000 7fc957884700 1 -- 172.21.15.16:0/3586966315 shutdown_connections 2023-10-16T13:47:49.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.730+0000 7fc957884700 1 --2- 172.21.15.16:0/3586966315 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7fc93406b190 0x7fc93406d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:49.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.730+0000 7fc957884700 1 --2- 172.21.15.16:0/3586966315 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc95015da10 0x7fc9500b1640 unknown :-1 s=CLOSED pgs=2837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:49.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.730+0000 7fc957884700 1 --2- 172.21.15.16:0/3586966315 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc9500d2430 0x7fc950155010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:49.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.730+0000 7fc957884700 1 --2- 172.21.15.16:0/3586966315 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc9500d4060 0x7fc9500b1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:49.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.730+0000 7fc957884700 1 -- 172.21.15.16:0/3586966315 >> 172.21.15.16:0/3586966315 conn(0x7fc95009c9f0 msgr2=0x7fc9500cf1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:49.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.731+0000 7fc957884700 1 -- 172.21.15.16:0/3586966315 shutdown_connections 2023-10-16T13:47:49.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.731+0000 7fc957884700 1 -- 172.21.15.16:0/3586966315 wait complete. 2023-10-16T13:47:49.742 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:49.738+0000 7f01e0c51200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T13:47:49.747 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-16T13:47:49.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.873+0000 7f5c1890b700 1 Processor -- start 2023-10-16T13:47:49.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.873+0000 7f5c1890b700 1 -- start start 2023-10-16T13:47:49.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.874+0000 7f5c1890b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5c1415ec80 0x7f5c140cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:49.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.874+0000 7f5c1890b700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f5c1415f060 0x7f5c140ce400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:49.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.874+0000 7f5c1890b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5c140d2110 0x7f5c140b4440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:49.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.874+0000 7f5c1890b700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c140b2c50 con 0x7f5c1415ec80 2023-10-16T13:47:49.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.874+0000 7f5c1890b700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c140b2440 con 0x7f5c1415f060 2023-10-16T13:47:49.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.874+0000 7f5c1890b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c140d00e0 con 0x7f5c140d2110 2023-10-16T13:47:49.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.875+0000 7f5c1259c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5c1415ec80 0x7f5c140cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:49.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.875+0000 7f5c1259c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5c1415ec80 0x7f5c140cdec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56084/0 (socket says 172.21.15.16:56084) 2023-10-16T13:47:49.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.875+0000 7f5c1259c700 1 -- 172.21.15.16:0/3135434978 learned_addr learned my addr 172.21.15.16:0/3135434978 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:49.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.875+0000 7f5c12d9d700 1 --2- 172.21.15.16:0/3135434978 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5c140d2110 0x7f5c140b4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:49.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.875+0000 7f5c11d9b700 1 --2- 172.21.15.16:0/3135434978 >> v2:172.21.15.49:3300/0 conn(0x7f5c1415f060 0x7f5c140ce400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:49.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.875+0000 7f5c1259c700 1 -- 172.21.15.16:0/3135434978 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5c140d2110 msgr2=0x7f5c140b4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:49.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.875+0000 7f5c1259c700 1 --2- 172.21.15.16:0/3135434978 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5c140d2110 0x7f5c140b4440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:49.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.875+0000 7f5c1259c700 1 -- 172.21.15.16:0/3135434978 >> v2:172.21.15.49:3300/0 conn(0x7f5c1415f060 msgr2=0x7f5c140ce400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:49.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.875+0000 7f5c1259c700 1 --2- 172.21.15.16:0/3135434978 >> v2:172.21.15.49:3300/0 conn(0x7f5c1415f060 0x7f5c140ce400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:49.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.875+0000 7f5c1259c700 1 -- 172.21.15.16:0/3135434978 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c140cef60 con 0x7f5c1415ec80 2023-10-16T13:47:49.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.875+0000 7f5c11d9b700 1 --2- 172.21.15.16:0/3135434978 >> v2:172.21.15.49:3300/0 conn(0x7f5c1415f060 0x7f5c140ce400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:47:49.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.875+0000 7f5c1259c700 1 --2- 172.21.15.16:0/3135434978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c1415ec80 0x7f5c140cdec0 secure :-1 s=READY pgs=2838 cs=0 l=1 rev1=1 crypto rx=0x7f5c140674b0 tx=0x7f5c04009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec5ee388e1e16a3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:49.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.875+0000 7f5c1159a700 1 -- 172.21.15.16:0/3135434978 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c0400b870 con 0x7f5c1415ec80 2023-10-16T13:47:49.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.875+0000 7f5c1159a700 1 -- 172.21.15.16:0/3135434978 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c0400b9d0 con 0x7f5c1415ec80 2023-10-16T13:47:49.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.875+0000 7f5c1159a700 1 -- 172.21.15.16:0/3135434978 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c0400bd60 con 0x7f5c1415ec80 2023-10-16T13:47:49.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.876+0000 7f5c1890b700 1 -- 172.21.15.16:0/3135434978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c1415ec80 msgr2=0x7f5c140cdec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:49.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.876+0000 7f5c1890b700 1 --2- 172.21.15.16:0/3135434978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c1415ec80 0x7f5c140cdec0 secure :-1 s=READY pgs=2838 cs=0 l=1 rev1=1 crypto rx=0x7f5c140674b0 tx=0x7f5c04009bd0 comp rx=0 tx=0).stop 2023-10-16T13:47:49.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.876+0000 7f5c1890b700 1 -- 172.21.15.16:0/3135434978 shutdown_connections 2023-10-16T13:47:49.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.876+0000 7f5c1890b700 1 --2- 172.21.15.16:0/3135434978 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5c140d2110 0x7f5c140b4440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:49.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.876+0000 7f5c1890b700 1 --2- 172.21.15.16:0/3135434978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c1415ec80 0x7f5c140cdec0 unknown :-1 s=CLOSED pgs=2838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:49.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.876+0000 7f5c1890b700 1 --2- 172.21.15.16:0/3135434978 >> v2:172.21.15.49:3300/0 conn(0x7f5c1415f060 0x7f5c140ce400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:49.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.876+0000 7f5c1890b700 1 -- 172.21.15.16:0/3135434978 >> 172.21.15.16:0/3135434978 conn(0x7f5c140d15f0 msgr2=0x7f5c140d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:49.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.876+0000 7f5c1890b700 1 -- 172.21.15.16:0/3135434978 shutdown_connections 2023-10-16T13:47:49.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.876+0000 7f5c1890b700 1 -- 172.21.15.16:0/3135434978 wait complete. 2023-10-16T13:47:49.886 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:47:49.876+0000 7facdb07f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T13:47:49.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.877+0000 7f5c1890b700 1 Processor -- start 2023-10-16T13:47:49.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.877+0000 7f5c1890b700 1 -- start start 2023-10-16T13:47:49.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.878+0000 7f5c1890b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5c140d2110 0x7f5c1415a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:49.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.878+0000 7f5c1890b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5c1415ec80 0x7f5c141f0990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:49.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.878+0000 7f5c1890b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c1415f060 0x7f5c140c9020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:49.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.878+0000 7f5c1890b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c141585b0 con 0x7f5c1415f060 2023-10-16T13:47:49.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.878+0000 7f5c1890b700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c1414a4f0 con 0x7f5c1415ec80 2023-10-16T13:47:49.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.878+0000 7f5c1890b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c1415a9c0 con 0x7f5c140d2110 2023-10-16T13:47:49.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.878+0000 7f5c12d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c1415f060 0x7f5c140c9020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:49.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.878+0000 7f5c1259c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5c140d2110 0x7f5c1415a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:49.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.878+0000 7f5c12d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c1415f060 0x7f5c140c9020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56094/0 (socket says 172.21.15.16:56094) 2023-10-16T13:47:49.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.878+0000 7f5c11d9b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5c1415ec80 0x7f5c141f0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:49.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.878+0000 7f5c12d9d700 1 -- 172.21.15.16:0/1564279361 learned_addr learned my addr 172.21.15.16:0/1564279361 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:49.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.879+0000 7f5c12d9d700 1 -- 172.21.15.16:0/1564279361 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5c140d2110 msgr2=0x7f5c1415a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:49.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.879+0000 7f5c12d9d700 1 --2- 172.21.15.16:0/1564279361 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5c140d2110 0x7f5c1415a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:49.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.879+0000 7f5c12d9d700 1 -- 172.21.15.16:0/1564279361 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5c1415ec80 msgr2=0x7f5c141f0990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:49.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.879+0000 7f5c12d9d700 1 --2- 172.21.15.16:0/1564279361 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5c1415ec80 0x7f5c141f0990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:49.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.879+0000 7f5c12d9d700 1 -- 172.21.15.16:0/1564279361 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c04009550 con 0x7f5c1415f060 2023-10-16T13:47:49.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.879+0000 7f5c12d9d700 1 --2- 172.21.15.16:0/1564279361 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c1415f060 0x7f5c140c9020 secure :-1 s=READY pgs=2839 cs=0 l=1 rev1=1 crypto rx=0x7f5c0c007e60 tx=0x7f5c0c00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:49.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.879+0000 7f5c02ffd700 1 -- 172.21.15.16:0/1564279361 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c0c00ed30 con 0x7f5c1415f060 2023-10-16T13:47:49.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.879+0000 7f5c02ffd700 1 -- 172.21.15.16:0/1564279361 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c0c014420 con 0x7f5c1415f060 2023-10-16T13:47:49.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.879+0000 7f5c1890b700 1 -- 172.21.15.16:0/1564279361 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c1415acd0 con 0x7f5c1415f060 2023-10-16T13:47:49.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.879+0000 7f5c02ffd700 1 -- 172.21.15.16:0/1564279361 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c0c0147d0 con 0x7f5c1415f060 2023-10-16T13:47:49.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.879+0000 7f5c1890b700 1 -- 172.21.15.16:0/1564279361 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c1415af70 con 0x7f5c1415f060 2023-10-16T13:47:49.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.880+0000 7f5c1890b700 1 -- 172.21.15.16:0/1564279361 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c1404ebb0 con 0x7f5c1415f060 2023-10-16T13:47:49.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.881+0000 7f5c02ffd700 1 -- 172.21.15.16:0/1564279361 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 149) v1 ==== 89450+0+0 (secure 0 0 0) 0x7f5c0c014930 con 0x7f5c1415f060 2023-10-16T13:47:49.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.881+0000 7f5c02ffd700 1 --2- 172.21.15.16:0/1564279361 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f5bf406b190 0x7f5bf406d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:49.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.881+0000 7f5c1259c700 1 -- 172.21.15.16:0/1564279361 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f5bf406b190 msgr2=0x7f5bf406d650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1736104576 2023-10-16T13:47:49.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.881+0000 7f5c02ffd700 1 -- 172.21.15.16:0/1564279361 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 7021+0+0 (secure 0 0 0) 0x7f5c0c08b370 con 0x7f5c1415f060 2023-10-16T13:47:49.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.881+0000 7f5c1259c700 1 --2- 172.21.15.16:0/1564279361 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f5bf406b190 0x7f5bf406d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:47:49.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:49.885+0000 7f5c02ffd700 1 -- 172.21.15.16:0/1564279361 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5c0c055d50 con 0x7f5c1415f060 2023-10-16T13:47:49.896 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:49.892+0000 7f01e0c51200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T13:47:50.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.082+0000 7f5c1259c700 1 -- 172.21.15.16:0/1564279361 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f5bf406b190 msgr2=0x7f5bf406d650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1736104576 2023-10-16T13:47:50.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.082+0000 7f5c1259c700 1 --2- 172.21.15.16:0/1564279361 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f5bf406b190 0x7f5bf406d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:47:50.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.094+0000 7f5c1890b700 1 -- 172.21.15.16:0/1564279361 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f5c140d1be0 con 0x7f5c1415f060 2023-10-16T13:47:50.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.097+0000 7f5c02ffd700 1 -- 172.21.15.16:0/1564279361 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v84)=0 v84) v1 ==== 133+0+0 (secure 0 0 0) 0x7f5c0c059370 con 0x7f5c1415f060 2023-10-16T13:47:50.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.099+0000 7f5c1890b700 1 -- 172.21.15.16:0/1564279361 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f5bf406b190 msgr2=0x7f5bf406d650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:47:50.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.099+0000 7f5c1890b700 1 --2- 172.21.15.16:0/1564279361 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f5bf406b190 0x7f5bf406d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:50.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.099+0000 7f5c1890b700 1 -- 172.21.15.16:0/1564279361 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c1415f060 msgr2=0x7f5c140c9020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:50.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.099+0000 7f5c1890b700 1 --2- 172.21.15.16:0/1564279361 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c1415f060 0x7f5c140c9020 secure :-1 s=READY pgs=2839 cs=0 l=1 rev1=1 crypto rx=0x7f5c0c007e60 tx=0x7f5c0c00e3c0 comp rx=0 tx=0).stop 2023-10-16T13:47:50.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.099+0000 7f5c1890b700 1 -- 172.21.15.16:0/1564279361 shutdown_connections 2023-10-16T13:47:50.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.099+0000 7f5c1890b700 1 --2- 172.21.15.16:0/1564279361 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7f5bf406b190 0x7f5bf406d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:50.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.099+0000 7f5c1890b700 1 --2- 172.21.15.16:0/1564279361 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c1415f060 0x7f5c140c9020 unknown :-1 s=CLOSED pgs=2839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:50.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.099+0000 7f5c1890b700 1 --2- 172.21.15.16:0/1564279361 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5c140d2110 0x7f5c1415a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:50.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.099+0000 7f5c1890b700 1 --2- 172.21.15.16:0/1564279361 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5c1415ec80 0x7f5c141f0990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:50.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.099+0000 7f5c1890b700 1 -- 172.21.15.16:0/1564279361 >> 172.21.15.16:0/1564279361 conn(0x7f5c140d15f0 msgr2=0x7f5c140b6bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:50.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.099+0000 7f5c1890b700 1 -- 172.21.15.16:0/1564279361 shutdown_connections 2023-10-16T13:47:50.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.099+0000 7f5c1890b700 1 -- 172.21.15.16:0/1564279361 wait complete. 2023-10-16T13:47:50.116 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:47:50.117 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p mandatory_fs_argument_test_dir) 2023-10-16T13:47:50.139 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add mandatory_fs_argument_test_dir 1M 2023-10-16T13:47:50.221 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:47:50.220+0000 7facdb07f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T13:47:50.252 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:50.249+0000 7f01e0c51200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T13:47:50.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.308+0000 7ff4cee21700 1 Processor -- start 2023-10-16T13:47:50.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.308+0000 7ff4cee21700 1 -- start start 2023-10-16T13:47:50.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.308+0000 7ff4cee21700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff4c815ec80 0x7ff4c80cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:50.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.308+0000 7ff4cee21700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4c80b2440 con 0x7ff4c815f060 2023-10-16T13:47:50.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.308+0000 7ff4cee21700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4c80d00e0 con 0x7ff4c80d2110 2023-10-16T13:47:50.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.308+0000 7ff4cee21700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c80b2c50 con 0x7ff4c815ec80 2023-10-16T13:47:50.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.309+0000 7ff4c7fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff4c815f060 0x7ff4c80b4440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:45418/0 (socket says 172.21.15.16:45418) 2023-10-16T13:47:50.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.309+0000 7ff4c7fff700 1 -- 172.21.15.16:0/2261788797 learned_addr learned my addr 172.21.15.16:0/2261788797 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:47:50.315 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.309+0000 7ff4ccbbd700 1 --2- 172.21.15.16:0/2261788797 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff4c815ec80 0x7ff4c80cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:50.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.309+0000 7ff4c77fe700 1 -- 172.21.15.16:0/2261788797 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 199512285 0 0) 0x7ff4c80b2440 con 0x7ff4c815f060 2023-10-16T13:47:50.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.309+0000 7ff4c77fe700 1 -- 172.21.15.16:0/2261788797 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4b80035b0 con 0x7ff4c815f060 2023-10-16T13:47:50.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.309+0000 7ff4c77fe700 1 -- 172.21.15.16:0/2261788797 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3720650877 0 0) 0x7ff4c80d00e0 con 0x7ff4c80d2110 2023-10-16T13:47:50.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.309+0000 7ff4c77fe700 1 -- 172.21.15.16:0/2261788797 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4c80b2440 con 0x7ff4c80d2110 2023-10-16T13:47:50.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.309+0000 7ff4ccbbd700 1 -- 172.21.15.16:0/2261788797 >> v1:172.21.15.49:6789/0 conn(0x7ff4c80d2110 legacy=0x7ff4c80b4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:50.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.309+0000 7ff4ccbbd700 1 -- 172.21.15.16:0/2261788797 >> v1:172.21.15.16:6789/0 conn(0x7ff4c815f060 legacy=0x7ff4c80b4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:50.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.309+0000 7ff4c77fe700 1 -- 172.21.15.16:0/2261788797 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2144366413 0 0) 0x7ff4b80035b0 con 0x7ff4c815f060 2023-10-16T13:47:50.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.309+0000 7ff4ccbbd700 1 -- 172.21.15.16:0/2261788797 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4c80cef60 con 0x7ff4c815ec80 2023-10-16T13:47:50.317 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.309+0000 7ff4ccbbd700 1 --2- 172.21.15.16:0/2261788797 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff4c815ec80 0x7ff4c80cdec0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc009870 tx=0x7ff4bc009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dad8d7e5b92c8b75 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:50.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.309+0000 7ff4c77fe700 1 -- 172.21.15.16:0/2261788797 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b4001670 con 0x7ff4c815ec80 2023-10-16T13:47:50.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.309+0000 7ff4c77fe700 1 -- 172.21.15.16:0/2261788797 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4bc004030 con 0x7ff4c815ec80 2023-10-16T13:47:50.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.309+0000 7ff4c77fe700 1 -- 172.21.15.16:0/2261788797 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4bc0043c0 con 0x7ff4c815ec80 2023-10-16T13:47:50.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.310+0000 7ff4cee21700 1 -- 172.21.15.16:0/2261788797 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff4c815ec80 msgr2=0x7ff4c80cdec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:50.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.310+0000 7ff4cee21700 1 --2- 172.21.15.16:0/2261788797 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff4c815ec80 0x7ff4c80cdec0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc009870 tx=0x7ff4bc009b80 comp rx=0 tx=0).stop 2023-10-16T13:47:50.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.310+0000 7ff4cee21700 1 -- 172.21.15.16:0/2261788797 shutdown_connections 2023-10-16T13:47:50.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.310+0000 7ff4cee21700 1 --2- 172.21.15.16:0/2261788797 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff4c815ec80 0x7ff4c80cdec0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:50.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.310+0000 7ff4cee21700 1 -- 172.21.15.16:0/2261788797 >> 172.21.15.16:0/2261788797 conn(0x7ff4c80d15f0 msgr2=0x7ff4c80d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:50.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.310+0000 7ff4cee21700 1 -- 172.21.15.16:0/2261788797 shutdown_connections 2023-10-16T13:47:50.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.310+0000 7ff4cee21700 1 -- 172.21.15.16:0/2261788797 wait complete. 2023-10-16T13:47:50.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.311+0000 7ff4cee21700 1 Processor -- start 2023-10-16T13:47:50.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.311+0000 7ff4cee21700 1 -- start start 2023-10-16T13:47:50.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.312+0000 7ff4cee21700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff4c80d2110 0x7ff4c81e5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:50.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.312+0000 7ff4cee21700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff4c815ec80 0x7ff4c8151be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:50.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.312+0000 7ff4cee21700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4c815f060 0x7ff4c80b30e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:50.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.312+0000 7ff4cee21700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c8154de0 con 0x7ff4c815f060 2023-10-16T13:47:50.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.312+0000 7ff4cee21700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c80ca0f0 con 0x7ff4c80d2110 2023-10-16T13:47:50.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.312+0000 7ff4cee21700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c80ca3d0 con 0x7ff4c815ec80 2023-10-16T13:47:50.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.312+0000 7ff4cd3be700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4c815f060 0x7ff4c80b30e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:50.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.312+0000 7ff4c7fff700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff4c815ec80 0x7ff4c8151be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:50.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.312+0000 7ff4ccbbd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff4c80d2110 0x7ff4c81e5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:50.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.313+0000 7ff4c7fff700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff4c815ec80 0x7ff4c8151be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:44410/0 (socket says 172.21.15.16:44410) 2023-10-16T13:47:50.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.313+0000 7ff4cd3be700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4c815f060 0x7ff4c80b30e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56154/0 (socket says 172.21.15.16:56154) 2023-10-16T13:47:50.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.313+0000 7ff4ccbbd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff4c80d2110 0x7ff4c81e5bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:55212/0 (socket says 172.21.15.16:55212) 2023-10-16T13:47:50.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.313+0000 7ff4c7fff700 1 -- 172.21.15.16:0/2444369789 learned_addr learned my addr 172.21.15.16:0/2444369789 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:50.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.313+0000 7ff4c7fff700 1 -- 172.21.15.16:0/2444369789 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff4c80d2110 msgr2=0x7ff4c81e5bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:50.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.313+0000 7ff4c7fff700 1 --2- 172.21.15.16:0/2444369789 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff4c80d2110 0x7ff4c81e5bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:50.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.313+0000 7ff4c7fff700 1 -- 172.21.15.16:0/2444369789 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4c815f060 msgr2=0x7ff4c80b30e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:50.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.313+0000 7ff4c7fff700 1 --2- 172.21.15.16:0/2444369789 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4c815f060 0x7ff4c80b30e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:50.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.313+0000 7ff4c7fff700 1 -- 172.21.15.16:0/2444369789 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4bc0094a0 con 0x7ff4c815ec80 2023-10-16T13:47:50.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.313+0000 7ff4c7fff700 1 --2- 172.21.15.16:0/2444369789 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff4c815ec80 0x7ff4c8151be0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7ff4b400c700 tx=0x7ff4b400ca10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:50.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.313+0000 7ff4c57fa700 1 -- 172.21.15.16:0/2444369789 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b4004c80 con 0x7ff4c815ec80 2023-10-16T13:47:50.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.313+0000 7ff4c57fa700 1 -- 172.21.15.16:0/2444369789 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4b4004de0 con 0x7ff4c815ec80 2023-10-16T13:47:50.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.313+0000 7ff4cee21700 1 -- 172.21.15.16:0/2444369789 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4c80ca6b0 con 0x7ff4c815ec80 2023-10-16T13:47:50.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.313+0000 7ff4c57fa700 1 -- 172.21.15.16:0/2444369789 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b4007310 con 0x7ff4c815ec80 2023-10-16T13:47:50.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.313+0000 7ff4cee21700 1 -- 172.21.15.16:0/2444369789 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4c80cac70 con 0x7ff4c815ec80 2023-10-16T13:47:50.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.314+0000 7ff4cee21700 1 -- 172.21.15.16:0/2444369789 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4c804ebb0 con 0x7ff4c815ec80 2023-10-16T13:47:50.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.315+0000 7ff4c57fa700 1 -- 172.21.15.16:0/2444369789 <== mon.2 v2:172.21.15.49:3301/0 4 ==== mgrmap(e 149) v1 ==== 89450+0+0 (secure 0 0 0) 0x7ff4b4007470 con 0x7ff4c815ec80 2023-10-16T13:47:50.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.315+0000 7ff4c57fa700 1 --2- 172.21.15.16:0/2444369789 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7ff4ac06b190 0x7ff4ac06d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:50.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.315+0000 7ff4ccbbd700 1 -- 172.21.15.16:0/2444369789 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7ff4ac06b190 msgr2=0x7ff4ac06d650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1736104576 2023-10-16T13:47:50.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.315+0000 7ff4c57fa700 1 -- 172.21.15.16:0/2444369789 <== mon.2 v2:172.21.15.49:3301/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 7021+0+0 (secure 0 0 0) 0x7ff4b4089af0 con 0x7ff4c815ec80 2023-10-16T13:47:50.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.315+0000 7ff4ccbbd700 1 --2- 172.21.15.16:0/2444369789 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7ff4ac06b190 0x7ff4ac06d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:47:50.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.320+0000 7ff4c57fa700 1 -- 172.21.15.16:0/2444369789 <== mon.2 v2:172.21.15.49:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff4b40543b0 con 0x7ff4c815ec80 2023-10-16T13:47:50.342 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:47:50.341+0000 7facdb07f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T13:47:50.376 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:50.373+0000 7f01e0c51200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T13:47:50.420 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:47:50.418+0000 7facdb07f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T13:47:50.454 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:50.451+0000 7f01e0c51200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T13:47:50.504 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:47:50.502+0000 7facdb07f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T13:47:50.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.516+0000 7ff4ccbbd700 1 -- 172.21.15.16:0/2444369789 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7ff4ac06b190 msgr2=0x7ff4ac06d650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1736104576 2023-10-16T13:47:50.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.516+0000 7ff4ccbbd700 1 --2- 172.21.15.16:0/2444369789 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7ff4ac06b190 0x7ff4ac06d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:47:50.538 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:50.535+0000 7f01e0c51200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T13:47:50.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.553+0000 7ff4cee21700 1 -- 172.21.15.16:0/2444369789 --> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "mandatory_fs_argument_test_dir", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7ff4c80b2970 con 0x7ff4ac06b190 2023-10-16T13:47:50.890 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:47:50.889+0000 7facdb07f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T13:47:50.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.917+0000 7ff4ccbbd700 1 -- 172.21.15.16:0/2444369789 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7ff4ac06b190 msgr2=0x7ff4ac06d650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1736104576 2023-10-16T13:47:50.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:50.917+0000 7ff4ccbbd700 1 --2- 172.21.15.16:0/2444369789 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7ff4ac06b190 0x7ff4ac06d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-16T13:47:50.926 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:50.923+0000 7f01e0c51200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T13:47:51.500 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:47:51.498+0000 7facdb07f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T13:47:51.537 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:51.533+0000 7f01e0c51200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T13:47:51.573 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:47:51.571+0000 7facdb07f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T13:47:51.610 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:51.607+0000 7f01e0c51200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T13:47:51.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:51.718+0000 7ff4ccbbd700 1 -- 172.21.15.16:0/2444369789 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7ff4ac06b190 msgr2=0x7ff4ac06d650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1736104576 2023-10-16T13:47:51.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:51.718+0000 7ff4ccbbd700 1 --2- 172.21.15.16:0/2444369789 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7ff4ac06b190 0x7ff4ac06d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-16T13:47:52.234 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:47:52.232+0000 7facdb07f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:47:52.277 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:52.274+0000 7f01e0c51200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:47:52.315 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:47:52.313+0000 7facdb07f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T13:47:52.359 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:52.356+0000 7f01e0c51200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T13:47:52.467 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:47:52.465+0000 7facdb07f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:47:52.513 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:52.509+0000 7f01e0c51200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:47:52.704 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:47:52.702+0000 7facdb07f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T13:47:52.752 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:52.749+0000 7f01e0c51200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T13:47:52.782 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:47:52.780+0000 7facdb07f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T13:47:52.830 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:52.827+0000 7f01e0c51200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T13:47:53.058 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:47:53.055+0000 7facdb07f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T13:47:53.109 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:53.106+0000 7f01e0c51200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T13:47:53.289 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:47:53.287+0000 7facdb07f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T13:47:53.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:53.319+0000 7ff4ccbbd700 1 -- 172.21.15.16:0/2444369789 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7ff4ac06b190 msgr2=0x7ff4ac06d650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1736104576 2023-10-16T13:47:53.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:53.319+0000 7ff4ccbbd700 1 --2- 172.21.15.16:0/2444369789 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7ff4ac06b190 0x7ff4ac06d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-10-16T13:47:53.345 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:53.341+0000 7f01e0c51200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T13:47:53.446 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:47:53.443+0000 7facdb07f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T13:47:53.503 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:53.500+0000 7f01e0c51200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T13:47:53.565 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:47:53.563+0000 7facdb07f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T13:47:53.624 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:53.620+0000 7f01e0c51200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T13:47:53.729 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:47:53.726+0000 7facdb07f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T13:47:53.789 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:53.786+0000 7f01e0c51200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T13:47:54.001 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:47:53.998+0000 7facdb07f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T13:47:54.065 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:54.062+0000 7f01e0c51200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T13:47:54.169 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:47:54.167+0000 7facdb07f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T13:47:54.237 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:54.233+0000 7f01e0c51200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T13:47:54.725 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:47:54.723+0000 7facdb07f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T13:47:54.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:54.793+0000 7ff4c57fa700 1 -- 172.21.15.16:0/2444369789 <== mon.2 v2:172.21.15.49:3301/0 7 ==== mgrmap(e 150) v1 ==== 89450+0+0 (secure 0 0 0) 0x7ff4b4054780 con 0x7ff4c815ec80 2023-10-16T13:47:54.805 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:54.802+0000 7f01e0c51200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T13:47:54.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:54.824+0000 7ff4c57fa700 1 -- 172.21.15.16:0/2444369789 <== mon.2 v2:172.21.15.49:3301/0 8 ==== mgrmap(e 151) v1 ==== 88957+0+0 (secure 0 0 0) 0x7ff4b4019260 con 0x7ff4c815ec80 2023-10-16T13:47:54.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:54.824+0000 7ff4c57fa700 1 -- 172.21.15.16:0/2444369789 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7ff4ac06b190 msgr2=0x7ff4ac06d650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:47:54.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:54.824+0000 7ff4c57fa700 1 --2- 172.21.15.16:0/2444369789 >> [v2:172.21.15.16:6832/1736104576,v1:172.21.15.16:6833/1736104576] conn(0x7ff4ac06b190 0x7ff4ac06d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:55.061 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:55.057+0000 7f0156431700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:47:55.061 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:55.057+0000 7f0156431700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:47:55.061 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:55.057+0000 7f0156431700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:47:55.062 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:55.057+0000 7f0156431700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:47:55.062 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:55.057+0000 7f0156431700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:47:55.062 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:55.057+0000 7f015342b700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:47:55.062 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:55.057+0000 7f015342b700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:47:55.062 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:55.057+0000 7f015342b700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:47:55.063 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:55.057+0000 7f015342b700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:47:55.063 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:55.057+0000 7f015342b700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:47:55.072 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:55.069+0000 7f0156431700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:47:55.072 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:55.069+0000 7f0156431700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:47:55.072 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:55.069+0000 7f0156431700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:47:55.072 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:55.069+0000 7f0156431700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:47:55.073 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:55.069+0000 7f0156431700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:47:55.073 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:55.069+0000 7f015342b700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:47:55.073 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:55.069+0000 7f015342b700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:47:55.073 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:55.069+0000 7f015342b700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:47:55.073 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:55.069+0000 7f015342b700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:47:55.074 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:55.069+0000 7f015342b700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:47:55.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:55.836+0000 7ff4c57fa700 1 -- 172.21.15.16:0/2444369789 <== mon.2 v2:172.21.15.49:3301/0 9 ==== mgrmap(e 152) v1 ==== 89084+0+0 (secure 0 0 0) 0x7ff4b4055050 con 0x7ff4c815ec80 2023-10-16T13:47:55.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:55.837+0000 7ff4c57fa700 1 --2- 172.21.15.16:0/2444369789 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7ff4ac01cc70 0x7ff4ac01d050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:55.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:55.837+0000 7ff4c57fa700 1 -- 172.21.15.16:0/2444369789 --> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "mandatory_fs_argument_test_dir", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7ff4ac01d590 con 0x7ff4ac01cc70 2023-10-16T13:47:55.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:55.837+0000 7ff4ccbbd700 1 --2- 172.21.15.16:0/2444369789 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7ff4ac01cc70 0x7ff4ac01d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:55.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:55.840+0000 7ff4ccbbd700 1 --2- 172.21.15.16:0/2444369789 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7ff4ac01cc70 0x7ff4ac01d050 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc0090b0 tx=0x7ff4bc008030 comp rx=0 tx=0).ready entity=mgr.12600 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:55.845 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:55.842+0000 7f015d43f700 -1 mgr.server reply reply (22) Invalid argument filesystem argument is required when there is more than one file system 2023-10-16T13:47:55.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:55.842+0000 7ff4c57fa700 1 -- 172.21.15.16:0/2444369789 <== mgr.12600 v2:172.21.15.16:6832/2221624769 1 ==== mgr_command_reply(tid 0: -22 filesystem argument is required when there is more than one file system) v1 ==== 79+0+0 (secure 0 0 0) 0x7ff4ac01d590 con 0x7ff4ac01cc70 2023-10-16T13:47:55.847 INFO:teuthology.orchestra.run.smithi016.stderr:Error EINVAL: filesystem argument is required when there is more than one file system 2023-10-16T13:47:55.851 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-10-16T13:47:55.852 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:47:55.852 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir mandatory_fs_argument_test_dir) 2023-10-16T13:47:55.874 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-16T13:47:56.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.043+0000 7f797e722700 1 Processor -- start 2023-10-16T13:47:56.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.044+0000 7f797e722700 1 -- start start 2023-10-16T13:47:56.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.044+0000 7f797e722700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f79780ce6a0 0x7f797806c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:56.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.044+0000 7f797e722700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f79780d2110 0x7f797806c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:56.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.044+0000 7f797e722700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f79780b2620 0x7f79780b0f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:56.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.044+0000 7f797e722700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79780b20f0 con 0x7f79780ce6a0 2023-10-16T13:47:56.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.044+0000 7f797e722700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79780d03d0 con 0x7f79780b2620 2023-10-16T13:47:56.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.044+0000 7f797e722700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79780b2340 con 0x7f79780d2110 2023-10-16T13:47:56.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.045+0000 7f7977fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f79780ce6a0 0x7f797806c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:56.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.045+0000 7f7977fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f79780ce6a0 0x7f797806c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56378/0 (socket says 172.21.15.16:56378) 2023-10-16T13:47:56.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.045+0000 7f7977fff700 1 -- 172.21.15.16:0/103573293 learned_addr learned my addr 172.21.15.16:0/103573293 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:56.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.045+0000 7f79777fe700 1 --2- 172.21.15.16:0/103573293 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f79780d2110 0x7f797806c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:56.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.045+0000 7f797ccbf700 1 --2- 172.21.15.16:0/103573293 >> v2:172.21.15.49:3300/0 conn(0x7f79780b2620 0x7f79780b0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:56.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.045+0000 7f7977fff700 1 -- 172.21.15.16:0/103573293 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f79780d2110 msgr2=0x7f797806c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:56.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.045+0000 7f7977fff700 1 --2- 172.21.15.16:0/103573293 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f79780d2110 0x7f797806c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:56.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.045+0000 7f7977fff700 1 -- 172.21.15.16:0/103573293 >> v2:172.21.15.49:3300/0 conn(0x7f79780b2620 msgr2=0x7f79780b0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:56.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.045+0000 7f7977fff700 1 --2- 172.21.15.16:0/103573293 >> v2:172.21.15.49:3300/0 conn(0x7f79780b2620 0x7f79780b0f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:56.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.045+0000 7f7977fff700 1 -- 172.21.15.16:0/103573293 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f797806bb00 con 0x7f79780ce6a0 2023-10-16T13:47:56.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.045+0000 7f797ccbf700 1 --2- 172.21.15.16:0/103573293 >> v2:172.21.15.49:3300/0 conn(0x7f79780b2620 0x7f79780b0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:47:56.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.045+0000 7f7977fff700 1 --2- 172.21.15.16:0/103573293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79780ce6a0 0x7f797806c300 secure :-1 s=READY pgs=2854 cs=0 l=1 rev1=1 crypto rx=0x7f797808bcb0 tx=0x7f7968009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a98b9594efe7372c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:56.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.045+0000 7f7976ffd700 1 -- 172.21.15.16:0/103573293 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f796800b870 con 0x7f79780ce6a0 2023-10-16T13:47:56.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.045+0000 7f7976ffd700 1 -- 172.21.15.16:0/103573293 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f796800b9d0 con 0x7f79780ce6a0 2023-10-16T13:47:56.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.045+0000 7f7976ffd700 1 -- 172.21.15.16:0/103573293 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f796800bd60 con 0x7f79780ce6a0 2023-10-16T13:47:56.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.046+0000 7f797e722700 1 -- 172.21.15.16:0/103573293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79780ce6a0 msgr2=0x7f797806c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:56.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.046+0000 7f797e722700 1 --2- 172.21.15.16:0/103573293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79780ce6a0 0x7f797806c300 secure :-1 s=READY pgs=2854 cs=0 l=1 rev1=1 crypto rx=0x7f797808bcb0 tx=0x7f7968009bd0 comp rx=0 tx=0).stop 2023-10-16T13:47:56.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.046+0000 7f797e722700 1 -- 172.21.15.16:0/103573293 shutdown_connections 2023-10-16T13:47:56.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.046+0000 7f797e722700 1 --2- 172.21.15.16:0/103573293 >> v2:172.21.15.49:3300/0 conn(0x7f79780b2620 0x7f79780b0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:56.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.046+0000 7f797e722700 1 --2- 172.21.15.16:0/103573293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79780ce6a0 0x7f797806c300 unknown :-1 s=CLOSED pgs=2854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:56.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.046+0000 7f797e722700 1 --2- 172.21.15.16:0/103573293 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f79780d2110 0x7f797806c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:56.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.046+0000 7f797e722700 1 -- 172.21.15.16:0/103573293 >> 172.21.15.16:0/103573293 conn(0x7f79780d15f0 msgr2=0x7f79780d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:56.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.046+0000 7f797e722700 1 -- 172.21.15.16:0/103573293 shutdown_connections 2023-10-16T13:47:56.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.046+0000 7f797e722700 1 -- 172.21.15.16:0/103573293 wait complete. 2023-10-16T13:47:56.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.047+0000 7f797e722700 1 Processor -- start 2023-10-16T13:47:56.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.047+0000 7f797e722700 1 -- start start 2023-10-16T13:47:56.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.048+0000 7f797e722700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f79780b2620 0x7f797814b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:56.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.048+0000 7f797e722700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f79780ce6a0 0x7f797815e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:56.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.048+0000 7f797e722700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79780d2110 0x7f797814c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:56.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.048+0000 7f797e722700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79780b5d90 con 0x7f79780d2110 2023-10-16T13:47:56.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.048+0000 7f797e722700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f797815d6c0 con 0x7f79780ce6a0 2023-10-16T13:47:56.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.048+0000 7f797e722700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f797815d8d0 con 0x7f79780b2620 2023-10-16T13:47:56.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.048+0000 7f797ccbf700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79780d2110 0x7f797814c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:56.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.048+0000 7f797ccbf700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79780d2110 0x7f797814c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56388/0 (socket says 172.21.15.16:56388) 2023-10-16T13:47:56.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.048+0000 7f797ccbf700 1 -- 172.21.15.16:0/1824482941 learned_addr learned my addr 172.21.15.16:0/1824482941 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:56.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.048+0000 7f7977fff700 1 --2- 172.21.15.16:0/1824482941 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f79780b2620 0x7f797814b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:56.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.048+0000 7f79777fe700 1 --2- 172.21.15.16:0/1824482941 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f79780ce6a0 0x7f797815e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:56.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.049+0000 7f797ccbf700 1 -- 172.21.15.16:0/1824482941 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f79780b2620 msgr2=0x7f797814b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:56.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.049+0000 7f797ccbf700 1 --2- 172.21.15.16:0/1824482941 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f79780b2620 0x7f797814b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:56.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.049+0000 7f797ccbf700 1 -- 172.21.15.16:0/1824482941 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f79780ce6a0 msgr2=0x7f797815e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:56.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.049+0000 7f797ccbf700 1 --2- 172.21.15.16:0/1824482941 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f79780ce6a0 0x7f797815e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:56.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.049+0000 7f797ccbf700 1 -- 172.21.15.16:0/1824482941 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7968009550 con 0x7f79780d2110 2023-10-16T13:47:56.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.049+0000 7f797ccbf700 1 --2- 172.21.15.16:0/1824482941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79780d2110 0x7f797814c250 secure :-1 s=READY pgs=2855 cs=0 l=1 rev1=1 crypto rx=0x7f797000cef0 tx=0x7f797000cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:56.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.049+0000 7f7974ff9700 1 -- 172.21.15.16:0/1824482941 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f797000ac80 con 0x7f79780d2110 2023-10-16T13:47:56.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.049+0000 7f7974ff9700 1 -- 172.21.15.16:0/1824482941 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f797000ade0 con 0x7f79780d2110 2023-10-16T13:47:56.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.049+0000 7f797e722700 1 -- 172.21.15.16:0/1824482941 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f797815dbb0 con 0x7f79780d2110 2023-10-16T13:47:56.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.049+0000 7f7974ff9700 1 -- 172.21.15.16:0/1824482941 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7970015680 con 0x7f79780d2110 2023-10-16T13:47:56.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.049+0000 7f797e722700 1 -- 172.21.15.16:0/1824482941 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f797815e170 con 0x7f79780d2110 2023-10-16T13:47:56.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.050+0000 7f797e722700 1 -- 172.21.15.16:0/1824482941 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f797815e390 con 0x7f79780d2110 2023-10-16T13:47:56.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.051+0000 7f7974ff9700 1 -- 172.21.15.16:0/1824482941 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 152) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f797000d430 con 0x7f79780d2110 2023-10-16T13:47:56.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.051+0000 7f7974ff9700 1 --2- 172.21.15.16:0/1824482941 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7f795806ada0 0x7f795806d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:56.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.051+0000 7f7974ff9700 1 -- 172.21.15.16:0/1824482941 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(250..250 src has 1..250) v4 ==== 7451+0+0 (secure 0 0 0) 0x7f7970089de0 con 0x7f79780d2110 2023-10-16T13:47:56.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.051+0000 7f7977fff700 1 --2- 172.21.15.16:0/1824482941 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7f795806ada0 0x7f795806d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:56.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.052+0000 7f7977fff700 1 --2- 172.21.15.16:0/1824482941 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7f795806ada0 0x7f795806d260 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7968000c00 tx=0x7f7968008030 comp rx=0 tx=0).ready entity=mgr.12600 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:56.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.056+0000 7f7974ff9700 1 -- 172.21.15.16:0/1824482941 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7970054a80 con 0x7f79780d2110 2023-10-16T13:47:56.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.323+0000 7f797e722700 1 -- 172.21.15.16:0/1824482941 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f797804ebb0 con 0x7f79780d2110 2023-10-16T13:47:56.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.835+0000 7f7974ff9700 1 -- 172.21.15.16:0/1824482941 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v153) v1 ==== 93+0+0 (secure 0 0 0) 0x7f79700580a0 con 0x7f79780d2110 2023-10-16T13:47:56.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.839+0000 7f7974ff9700 1 -- 172.21.15.16:0/1824482941 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mgrmap(e 153) v1 ==== 89434+0+0 (secure 0 0 0) 0x7f7970054e50 con 0x7f79780d2110 2023-10-16T13:47:56.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.840+0000 7f797e722700 1 -- 172.21.15.16:0/1824482941 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7f795806ada0 msgr2=0x7f795806d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:56.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.840+0000 7f797e722700 1 --2- 172.21.15.16:0/1824482941 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7f795806ada0 0x7f795806d260 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7968000c00 tx=0x7f7968008030 comp rx=0 tx=0).stop 2023-10-16T13:47:56.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.840+0000 7f797e722700 1 -- 172.21.15.16:0/1824482941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79780d2110 msgr2=0x7f797814c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:56.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.840+0000 7f797e722700 1 --2- 172.21.15.16:0/1824482941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79780d2110 0x7f797814c250 secure :-1 s=READY pgs=2855 cs=0 l=1 rev1=1 crypto rx=0x7f797000cef0 tx=0x7f797000cf20 comp rx=0 tx=0).stop 2023-10-16T13:47:56.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.840+0000 7f797e722700 1 -- 172.21.15.16:0/1824482941 shutdown_connections 2023-10-16T13:47:56.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.840+0000 7f797e722700 1 --2- 172.21.15.16:0/1824482941 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7f795806ada0 0x7f795806d260 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:56.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.840+0000 7f797e722700 1 --2- 172.21.15.16:0/1824482941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79780d2110 0x7f797814c250 unknown :-1 s=CLOSED pgs=2855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:56.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.840+0000 7f797e722700 1 --2- 172.21.15.16:0/1824482941 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f79780b2620 0x7f797814b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:56.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.840+0000 7f797e722700 1 --2- 172.21.15.16:0/1824482941 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f79780ce6a0 0x7f797815e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:56.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.840+0000 7f797e722700 1 -- 172.21.15.16:0/1824482941 >> 172.21.15.16:0/1824482941 conn(0x7f79780d15f0 msgr2=0x7f79780d19d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:56.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.840+0000 7f797e722700 1 -- 172.21.15.16:0/1824482941 shutdown_connections 2023-10-16T13:47:56.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:56.840+0000 7f797e722700 1 -- 172.21.15.16:0/1824482941 wait complete. 2023-10-16T13:47:56.858 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T13:47:56.859 DEBUG:teuthology.orchestra.run.smithi016:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T13:47:56.901 DEBUG:teuthology.orchestra.run.smithi049:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T13:47:56.945 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T13:47:56.945 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T13:47:56.946 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:47:56.965 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T13:47:56.965 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:47:56.993 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:47:56.990+0000 7f250e765200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T13:47:56.998 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:56.995+0000 7eff06a04200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T13:47:57.015 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:47:57.061 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:47:57.059+0000 7f250e765200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T13:47:57.067 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:57.065+0000 7eff06a04200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T13:47:57.136 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:47:57.133+0000 7f250e765200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T13:47:57.144 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:57.142+0000 7eff06a04200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T13:47:57.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.179+0000 7f5bb026c700 1 Processor -- start 2023-10-16T13:47:57.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.179+0000 7f5bb026c700 1 -- start start 2023-10-16T13:47:57.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.180+0000 7f5bb026c700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f5ba80cdfa0 0x7f5ba80b2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:57.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.180+0000 7f5bb026c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5ba80ce380 0x7f5ba80b2980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:57.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.180+0000 7f5bb026c700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ba80d0fe0 con 0x7f5ba80d04a0 2023-10-16T13:47:57.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.180+0000 7f5bb026c700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ba80d1210 con 0x7f5ba80cdfa0 2023-10-16T13:47:57.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.180+0000 7f5bb026c700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ba80cbc30 con 0x7f5ba80ce380 2023-10-16T13:47:57.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.181+0000 7f5bae809700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f5ba80d04a0 0x7f5ba80d08d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:45718/0 (socket says 172.21.15.16:45718) 2023-10-16T13:47:57.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.181+0000 7f5bae809700 1 -- 172.21.15.16:0/2902502904 learned_addr learned my addr 172.21.15.16:0/2902502904 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:47:57.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.181+0000 7f5bad807700 1 --2- 172.21.15.16:0/2902502904 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5ba80ce380 0x7f5ba80b2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:57.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.181+0000 7f5bae008700 1 --2- 172.21.15.16:0/2902502904 >> v2:172.21.15.49:3300/0 conn(0x7f5ba80cdfa0 0x7f5ba80b2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:57.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.181+0000 7f5bad006700 1 -- 172.21.15.16:0/2902502904 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2572360532 0 0) 0x7f5ba80d0fe0 con 0x7f5ba80d04a0 2023-10-16T13:47:57.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.181+0000 7f5bad006700 1 -- 172.21.15.16:0/2902502904 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b940035b0 con 0x7f5ba80d04a0 2023-10-16T13:47:57.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.181+0000 7f5bad807700 1 -- 172.21.15.16:0/2902502904 >> v2:172.21.15.49:3300/0 conn(0x7f5ba80cdfa0 msgr2=0x7f5ba80b2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:57.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.181+0000 7f5bad807700 1 --2- 172.21.15.16:0/2902502904 >> v2:172.21.15.49:3300/0 conn(0x7f5ba80cdfa0 0x7f5ba80b2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:57.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.181+0000 7f5bad807700 1 -- 172.21.15.16:0/2902502904 >> v1:172.21.15.16:6789/0 conn(0x7f5ba80d04a0 legacy=0x7f5ba80d08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:57.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.181+0000 7f5bad807700 1 -- 172.21.15.16:0/2902502904 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ba80cbf10 con 0x7f5ba80ce380 2023-10-16T13:47:57.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.181+0000 7f5bae008700 1 --2- 172.21.15.16:0/2902502904 >> v2:172.21.15.49:3300/0 conn(0x7f5ba80cdfa0 0x7f5ba80b2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:47:57.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.181+0000 7f5bad807700 1 --2- 172.21.15.16:0/2902502904 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5ba80ce380 0x7f5ba80b2980 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f5b98009840 tx=0x7f5b98009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f6218a9a4fb3585 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:57.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.181+0000 7f5bad006700 1 -- 172.21.15.16:0/2902502904 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b98011830 con 0x7f5ba80ce380 2023-10-16T13:47:57.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.182+0000 7f5bad006700 1 -- 172.21.15.16:0/2902502904 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b98011990 con 0x7f5ba80ce380 2023-10-16T13:47:57.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.182+0000 7f5bad006700 1 -- 172.21.15.16:0/2902502904 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b98011d20 con 0x7f5ba80ce380 2023-10-16T13:47:57.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.182+0000 7f5bb026c700 1 -- 172.21.15.16:0/2902502904 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5ba80ce380 msgr2=0x7f5ba80b2980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:57.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.182+0000 7f5bb026c700 1 --2- 172.21.15.16:0/2902502904 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5ba80ce380 0x7f5ba80b2980 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f5b98009840 tx=0x7f5b98009b50 comp rx=0 tx=0).stop 2023-10-16T13:47:57.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.182+0000 7f5bb026c700 1 -- 172.21.15.16:0/2902502904 shutdown_connections 2023-10-16T13:47:57.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.182+0000 7f5bb026c700 1 --2- 172.21.15.16:0/2902502904 >> v2:172.21.15.49:3300/0 conn(0x7f5ba80cdfa0 0x7f5ba80b2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:57.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.182+0000 7f5bb026c700 1 --2- 172.21.15.16:0/2902502904 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5ba80ce380 0x7f5ba80b2980 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:57.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.182+0000 7f5bb026c700 1 -- 172.21.15.16:0/2902502904 >> 172.21.15.16:0/2902502904 conn(0x7f5ba80d14b0 msgr2=0x7f5ba80d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:57.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.182+0000 7f5bb026c700 1 -- 172.21.15.16:0/2902502904 shutdown_connections 2023-10-16T13:47:57.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.182+0000 7f5bb026c700 1 -- 172.21.15.16:0/2902502904 wait complete. 2023-10-16T13:47:57.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.183+0000 7f5bb026c700 1 Processor -- start 2023-10-16T13:47:57.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.183+0000 7f5bb026c700 1 -- start start 2023-10-16T13:47:57.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.184+0000 7f5bb026c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5ba80cdfa0 0x7f5ba80b5e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:57.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.184+0000 7f5bb026c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5ba80ce380 0x7f5ba81f3c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:57.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.184+0000 7f5bb026c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ba80d04a0 0x7f5ba80c5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:57.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.184+0000 7f5bb026c700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ba80b6340 con 0x7f5ba80d04a0 2023-10-16T13:47:57.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.184+0000 7f5bb026c700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ba81f4180 con 0x7f5ba80cdfa0 2023-10-16T13:47:57.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.184+0000 7f5bb026c700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ba81e6e00 con 0x7f5ba80ce380 2023-10-16T13:47:57.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.184+0000 7f5bae809700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ba80d04a0 0x7f5ba80c5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:57.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.184+0000 7f5bae809700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ba80d04a0 0x7f5ba80c5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56450/0 (socket says 172.21.15.16:56450) 2023-10-16T13:47:57.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.184+0000 7f5bae809700 1 -- 172.21.15.16:0/3564183644 learned_addr learned my addr 172.21.15.16:0/3564183644 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:57.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.184+0000 7f5bad807700 1 --2- 172.21.15.16:0/3564183644 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5ba80ce380 0x7f5ba81f3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:57.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.184+0000 7f5bae008700 1 --2- 172.21.15.16:0/3564183644 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5ba80cdfa0 0x7f5ba80b5e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:57.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.185+0000 7f5bae809700 1 -- 172.21.15.16:0/3564183644 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5ba80ce380 msgr2=0x7f5ba81f3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:57.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.185+0000 7f5bae809700 1 --2- 172.21.15.16:0/3564183644 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5ba80ce380 0x7f5ba81f3c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:57.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.185+0000 7f5bae809700 1 -- 172.21.15.16:0/3564183644 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5ba80cdfa0 msgr2=0x7f5ba80b5e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:57.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.185+0000 7f5bae809700 1 --2- 172.21.15.16:0/3564183644 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5ba80cdfa0 0x7f5ba80b5e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:57.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.185+0000 7f5bae809700 1 -- 172.21.15.16:0/3564183644 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ba40094a0 con 0x7f5ba80d04a0 2023-10-16T13:47:57.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.185+0000 7f5bad807700 1 --2- 172.21.15.16:0/3564183644 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5ba80ce380 0x7f5ba81f3c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:47:57.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.185+0000 7f5bae809700 1 --2- 172.21.15.16:0/3564183644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ba80d04a0 0x7f5ba80c5ee0 secure :-1 s=READY pgs=2858 cs=0 l=1 rev1=1 crypto rx=0x7f5ba000caa0 tx=0x7f5ba000cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:57.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.185+0000 7f5b9effd700 1 -- 172.21.15.16:0/3564183644 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ba000a9e0 con 0x7f5ba80d04a0 2023-10-16T13:47:57.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.185+0000 7f5b9effd700 1 -- 172.21.15.16:0/3564183644 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ba000ab40 con 0x7f5ba80d04a0 2023-10-16T13:47:57.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.185+0000 7f5bb026c700 1 -- 172.21.15.16:0/3564183644 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b980094a0 con 0x7f5ba80d04a0 2023-10-16T13:47:57.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.185+0000 7f5b9effd700 1 -- 172.21.15.16:0/3564183644 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ba0012400 con 0x7f5ba80d04a0 2023-10-16T13:47:57.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.185+0000 7f5bb026c700 1 -- 172.21.15.16:0/3564183644 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ba81e73c0 con 0x7f5ba80d04a0 2023-10-16T13:47:57.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.186+0000 7f5bb026c700 1 -- 172.21.15.16:0/3564183644 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ba804ebb0 con 0x7f5ba80d04a0 2023-10-16T13:47:57.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.187+0000 7f5b9effd700 1 -- 172.21.15.16:0/3564183644 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 153) v1 ==== 89434+0+0 (secure 0 0 0) 0x7f5ba001b3b0 con 0x7f5ba80d04a0 2023-10-16T13:47:57.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.187+0000 7f5b9effd700 1 --2- 172.21.15.16:0/3564183644 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7f5b8c06b140 0x7f5b8c06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:57.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.187+0000 7f5bae008700 1 -- 172.21.15.16:0/3564183644 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7f5b8c06b140 msgr2=0x7f5b8c06d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2221624769 2023-10-16T13:47:57.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.187+0000 7f5b9effd700 1 -- 172.21.15.16:0/3564183644 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(250..250 src has 1..250) v4 ==== 7451+0+0 (secure 0 0 0) 0x7f5ba0089c90 con 0x7f5ba80d04a0 2023-10-16T13:47:57.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.187+0000 7f5bae008700 1 --2- 172.21.15.16:0/3564183644 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7f5b8c06b140 0x7f5b8c06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:47:57.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.192+0000 7f5b9effd700 1 -- 172.21.15.16:0/3564183644 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5ba0054460 con 0x7f5ba80d04a0 2023-10-16T13:47:57.217 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:47:57.215+0000 7f250e765200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T13:47:57.230 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:57.227+0000 7eff06a04200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T13:47:57.365 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:47:57.362+0000 7f250e765200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T13:47:57.380 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:57.377+0000 7eff06a04200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T13:47:57.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.388+0000 7f5bae008700 1 -- 172.21.15.16:0/3564183644 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7f5b8c06b140 msgr2=0x7f5b8c06d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2221624769 2023-10-16T13:47:57.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.388+0000 7f5bae008700 1 --2- 172.21.15.16:0/3564183644 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7f5b8c06b140 0x7f5b8c06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:47:57.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.397+0000 7f5bb026c700 1 -- 172.21.15.16:0/3564183644 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5ba806d3b0 con 0x7f5ba80d04a0 2023-10-16T13:47:57.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.399+0000 7f5b9effd700 1 -- 172.21.15.16:0/3564183644 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v250) v1 ==== 74+0+21164 (secure 0 0 0) 0x7f5ba0057a80 con 0x7f5ba80d04a0 2023-10-16T13:47:57.402 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:47:57.402 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":250,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:47:54.816765+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2023-10-16T13:47:22.338349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"244","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2023-10-16T13:47:22.806327+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"245","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T13:47:39.075528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"248","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"backup_fs_data","create_time":"2023-10-16T13:47:40.314029+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"249","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/3947601914":"2023-10-17T13:47:54.816700+0000","172.21.15.16:0/2759470534":"2023-10-17T13:47:54.816700+0000","172.21.15.16:0/2974892940":"2023-10-17T13:47:54.816700+0000","172.21.15.16:6833/1736104576":"2023-10-17T13:47:54.816700+0000","172.21.15.16:6832/1736104576":"2023-10-17T13:47:54.816700+0000","172.21.15.16:0/318862437":"2023-10-17T13:47:54.816700+0000","172.21.15.16:0/413581668":"2023-10-17T13:47:54.816700+0000","172.21.15.16:0/3740515383":"2023-10-17T13:47:54.816700+0000","172.21.15.16:0/3105924907":"2023-10-17T13:47:54.816700+0000","172.21.15.16:0/3617558038":"2023-10-17T13:47:54.816700+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:47:57.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.401+0000 7f5bb026c700 1 -- 172.21.15.16:0/3564183644 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7f5b8c06b140 msgr2=0x7f5b8c06d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:47:57.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.401+0000 7f5bb026c700 1 --2- 172.21.15.16:0/3564183644 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7f5b8c06b140 0x7f5b8c06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:57.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.401+0000 7f5bb026c700 1 -- 172.21.15.16:0/3564183644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ba80d04a0 msgr2=0x7f5ba80c5ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:57.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.401+0000 7f5bb026c700 1 --2- 172.21.15.16:0/3564183644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ba80d04a0 0x7f5ba80c5ee0 secure :-1 s=READY pgs=2858 cs=0 l=1 rev1=1 crypto rx=0x7f5ba000caa0 tx=0x7f5ba000cdb0 comp rx=0 tx=0).stop 2023-10-16T13:47:57.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.401+0000 7f5bb026c700 1 -- 172.21.15.16:0/3564183644 shutdown_connections 2023-10-16T13:47:57.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.401+0000 7f5bb026c700 1 --2- 172.21.15.16:0/3564183644 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7f5b8c06b140 0x7f5b8c06d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:57.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.401+0000 7f5bb026c700 1 --2- 172.21.15.16:0/3564183644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ba80d04a0 0x7f5ba80c5ee0 unknown :-1 s=CLOSED pgs=2858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:57.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.401+0000 7f5bb026c700 1 --2- 172.21.15.16:0/3564183644 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5ba80cdfa0 0x7f5ba80b5e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:57.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.401+0000 7f5bb026c700 1 --2- 172.21.15.16:0/3564183644 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5ba80ce380 0x7f5ba81f3c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:57.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.401+0000 7f5bb026c700 1 -- 172.21.15.16:0/3564183644 >> 172.21.15.16:0/3564183644 conn(0x7f5ba80d14b0 msgr2=0x7f5ba8156730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:57.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.401+0000 7f5bb026c700 1 -- 172.21.15.16:0/3564183644 shutdown_connections 2023-10-16T13:47:57.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.401+0000 7f5bb026c700 1 -- 172.21.15.16:0/3564183644 wait complete. 2023-10-16T13:47:57.421 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 31, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T13:47:22.338349+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '244', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 32, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T13:47:22.806327+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '245', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 33, 'pool_name': 'backup_fs_metadata', 'create_time': '2023-10-16T13:47:39.075528+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '248', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 34, 'pool_name': 'backup_fs_data', 'create_time': '2023-10-16T13:47:40.314029+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '249', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T13:47:57.422 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T13:47:57.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.588+0000 7f6e7cbe1700 1 Processor -- start 2023-10-16T13:47:57.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.588+0000 7f6e7cbe1700 1 -- start start 2023-10-16T13:47:57.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.589+0000 7f6e7cbe1700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f6e78072b30 0x7f6e78071a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:57.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.589+0000 7f6e7cbe1700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6e780caa40 0x7f6e780cae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:57.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.589+0000 7f6e7cbe1700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e780cb390 con 0x7f6e7815bb40 2023-10-16T13:47:57.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.589+0000 7f6e7cbe1700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e7806ea20 con 0x7f6e780caa40 2023-10-16T13:47:57.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.589+0000 7f6e7cbe1700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e7815c040 con 0x7f6e78072b30 2023-10-16T13:47:57.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.590+0000 7f6e76d9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6e780caa40 0x7f6e780cae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:57.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.590+0000 7f6e76d9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6e780caa40 0x7f6e780cae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56480/0 (socket says 172.21.15.16:56480) 2023-10-16T13:47:57.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.590+0000 7f6e76d9d700 1 -- 172.21.15.16:0/296846583 learned_addr learned my addr 172.21.15.16:0/296846583 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:57.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.590+0000 7f6e7659c700 1 --2- 172.21.15.16:0/296846583 >> v2:172.21.15.49:3300/0 conn(0x7f6e78072b30 0x7f6e78071a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:57.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.590+0000 7f6e76d9d700 1 -- 172.21.15.16:0/296846583 >> v2:172.21.15.49:3300/0 conn(0x7f6e78072b30 msgr2=0x7f6e78071a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:57.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.590+0000 7f6e76d9d700 1 --2- 172.21.15.16:0/296846583 >> v2:172.21.15.49:3300/0 conn(0x7f6e78072b30 0x7f6e78071a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:57.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.590+0000 7f6e76d9d700 1 -- 172.21.15.16:0/296846583 >> v1:172.21.15.49:6789/0 conn(0x7f6e7815bb40 legacy=0x7f6e7806e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:57.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.590+0000 7f6e76d9d700 1 -- 172.21.15.16:0/296846583 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e7815c320 con 0x7f6e780caa40 2023-10-16T13:47:57.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.590+0000 7f6e76d9d700 1 --2- 172.21.15.16:0/296846583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e780caa40 0x7f6e780cae50 secure :-1 s=READY pgs=2859 cs=0 l=1 rev1=1 crypto rx=0x7f6e7806b780 tx=0x7f6e7000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27c2a68e997fbcc6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:57.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.590+0000 7f6e7559a700 1 -- 172.21.15.16:0/296846583 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e7000e070 con 0x7f6e780caa40 2023-10-16T13:47:57.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.590+0000 7f6e7559a700 1 -- 172.21.15.16:0/296846583 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e7000b910 con 0x7f6e780caa40 2023-10-16T13:47:57.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.590+0000 7f6e7559a700 1 -- 172.21.15.16:0/296846583 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e7000bcc0 con 0x7f6e780caa40 2023-10-16T13:47:57.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.591+0000 7f6e7cbe1700 1 -- 172.21.15.16:0/296846583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e780caa40 msgr2=0x7f6e780cae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:57.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.591+0000 7f6e7cbe1700 1 --2- 172.21.15.16:0/296846583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e780caa40 0x7f6e780cae50 secure :-1 s=READY pgs=2859 cs=0 l=1 rev1=1 crypto rx=0x7f6e7806b780 tx=0x7f6e7000b330 comp rx=0 tx=0).stop 2023-10-16T13:47:57.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.591+0000 7f6e7cbe1700 1 -- 172.21.15.16:0/296846583 shutdown_connections 2023-10-16T13:47:57.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.591+0000 7f6e7cbe1700 1 --2- 172.21.15.16:0/296846583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e780caa40 0x7f6e780cae50 unknown :-1 s=CLOSED pgs=2859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:57.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.591+0000 7f6e7cbe1700 1 --2- 172.21.15.16:0/296846583 >> v2:172.21.15.49:3300/0 conn(0x7f6e78072b30 0x7f6e78071a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:57.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.591+0000 7f6e7cbe1700 1 -- 172.21.15.16:0/296846583 >> 172.21.15.16:0/296846583 conn(0x7f6e780b1f40 msgr2=0x7f6e780b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:57.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.591+0000 7f6e7cbe1700 1 -- 172.21.15.16:0/296846583 shutdown_connections 2023-10-16T13:47:57.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.591+0000 7f6e7cbe1700 1 -- 172.21.15.16:0/296846583 wait complete. 2023-10-16T13:47:57.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.593+0000 7f6e7cbe1700 1 Processor -- start 2023-10-16T13:47:57.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.593+0000 7f6e7cbe1700 1 -- start start 2023-10-16T13:47:57.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.593+0000 7f6e7cbe1700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6e78072b30 0x7f6e780c7f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:57.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.593+0000 7f6e7cbe1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e780caa40 0x7f6e781f0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:57.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.593+0000 7f6e7cbe1700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6e7815bb40 0x7f6e78157f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:57.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.593+0000 7f6e7cbe1700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e780c94d0 con 0x7f6e780caa40 2023-10-16T13:47:57.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.593+0000 7f6e7cbe1700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e780c8850 con 0x7f6e78072b30 2023-10-16T13:47:57.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.593+0000 7f6e7cbe1700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e780c8b30 con 0x7f6e7815bb40 2023-10-16T13:47:57.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.593+0000 7f6e75d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e780caa40 0x7f6e781f0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:57.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.593+0000 7f6e75d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e780caa40 0x7f6e781f0650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56484/0 (socket says 172.21.15.16:56484) 2023-10-16T13:47:57.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.593+0000 7f6e75d9b700 1 -- 172.21.15.16:0/1639708513 learned_addr learned my addr 172.21.15.16:0/1639708513 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:57.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.594+0000 7f6e7659c700 1 --2- 172.21.15.16:0/1639708513 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6e78072b30 0x7f6e780c7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:57.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.594+0000 7f6e76d9d700 1 --2- 172.21.15.16:0/1639708513 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6e7815bb40 0x7f6e78157f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:57.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.594+0000 7f6e75d9b700 1 -- 172.21.15.16:0/1639708513 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6e7815bb40 msgr2=0x7f6e78157f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:57.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.594+0000 7f6e75d9b700 1 --2- 172.21.15.16:0/1639708513 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6e7815bb40 0x7f6e78157f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:57.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.594+0000 7f6e75d9b700 1 -- 172.21.15.16:0/1639708513 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6e78072b30 msgr2=0x7f6e780c7f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:57.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.594+0000 7f6e75d9b700 1 --2- 172.21.15.16:0/1639708513 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6e78072b30 0x7f6e780c7f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:57.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.594+0000 7f6e75d9b700 1 -- 172.21.15.16:0/1639708513 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e70009b30 con 0x7f6e780caa40 2023-10-16T13:47:57.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.594+0000 7f6e75d9b700 1 --2- 172.21.15.16:0/1639708513 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e780caa40 0x7f6e781f0650 secure :-1 s=READY pgs=2860 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c00c4b0 tx=0x7f6e6c00c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:57.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.594+0000 7f6e66ffd700 1 -- 172.21.15.16:0/1639708513 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e6c0052d0 con 0x7f6e780caa40 2023-10-16T13:47:57.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.594+0000 7f6e66ffd700 1 -- 172.21.15.16:0/1639708513 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e6c005430 con 0x7f6e780caa40 2023-10-16T13:47:57.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.594+0000 7f6e66ffd700 1 -- 172.21.15.16:0/1639708513 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e6c0038c0 con 0x7f6e780caa40 2023-10-16T13:47:57.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.594+0000 7f6e7cbe1700 1 -- 172.21.15.16:0/1639708513 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e780c8e10 con 0x7f6e780caa40 2023-10-16T13:47:57.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.594+0000 7f6e7cbe1700 1 -- 172.21.15.16:0/1639708513 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e780c8460 con 0x7f6e780caa40 2023-10-16T13:47:57.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.595+0000 7f6e7cbe1700 1 -- 172.21.15.16:0/1639708513 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e7804ebb0 con 0x7f6e780caa40 2023-10-16T13:47:57.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.596+0000 7f6e66ffd700 1 -- 172.21.15.16:0/1639708513 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 153) v1 ==== 89434+0+0 (secure 0 0 0) 0x7f6e6c003a20 con 0x7f6e780caa40 2023-10-16T13:47:57.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.596+0000 7f6e66ffd700 1 --2- 172.21.15.16:0/1639708513 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7f6e5806b140 0x7f6e5806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:57.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.596+0000 7f6e7659c700 1 -- 172.21.15.16:0/1639708513 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7f6e5806b140 msgr2=0x7f6e5806d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2221624769 2023-10-16T13:47:57.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.596+0000 7f6e66ffd700 1 -- 172.21.15.16:0/1639708513 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(250..250 src has 1..250) v4 ==== 7451+0+0 (secure 0 0 0) 0x7f6e6c089300 con 0x7f6e780caa40 2023-10-16T13:47:57.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.596+0000 7f6e7659c700 1 --2- 172.21.15.16:0/1639708513 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7f6e5806b140 0x7f6e5806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:47:57.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.601+0000 7f6e66ffd700 1 -- 172.21.15.16:0/1639708513 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6e6c053ae0 con 0x7f6e780caa40 2023-10-16T13:47:57.706 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:47:57.704+0000 7f250e765200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T13:47:57.734 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:57.732+0000 7eff06a04200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T13:47:57.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.797+0000 7f6e7659c700 1 -- 172.21.15.16:0/1639708513 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7f6e5806b140 msgr2=0x7f6e5806d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2221624769 2023-10-16T13:47:57.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.797+0000 7f6e7659c700 1 --2- 172.21.15.16:0/1639708513 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7f6e5806b140 0x7f6e5806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:47:57.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.803+0000 7f6e7cbe1700 1 -- 172.21.15.16:0/1639708513 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6e780b2d40 con 0x7f6e780caa40 2023-10-16T13:47:57.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.804+0000 7f6e66ffd700 1 -- 172.21.15.16:0/1639708513 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v250) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6e6c057100 con 0x7f6e780caa40 2023-10-16T13:47:57.806 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T13:47:57.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.806+0000 7f6e7cbe1700 1 -- 172.21.15.16:0/1639708513 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7f6e5806b140 msgr2=0x7f6e5806d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:47:57.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.806+0000 7f6e7cbe1700 1 --2- 172.21.15.16:0/1639708513 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7f6e5806b140 0x7f6e5806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:57.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.806+0000 7f6e7cbe1700 1 -- 172.21.15.16:0/1639708513 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e780caa40 msgr2=0x7f6e781f0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:57.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.806+0000 7f6e7cbe1700 1 --2- 172.21.15.16:0/1639708513 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e780caa40 0x7f6e781f0650 secure :-1 s=READY pgs=2860 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c00c4b0 tx=0x7f6e6c00c7c0 comp rx=0 tx=0).stop 2023-10-16T13:47:57.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.806+0000 7f6e7cbe1700 1 -- 172.21.15.16:0/1639708513 shutdown_connections 2023-10-16T13:47:57.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.806+0000 7f6e7cbe1700 1 --2- 172.21.15.16:0/1639708513 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7f6e5806b140 0x7f6e5806d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:57.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.806+0000 7f6e7cbe1700 1 --2- 172.21.15.16:0/1639708513 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6e7815bb40 0x7f6e78157f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:57.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.806+0000 7f6e7cbe1700 1 --2- 172.21.15.16:0/1639708513 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6e78072b30 0x7f6e780c7f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:57.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.806+0000 7f6e7cbe1700 1 --2- 172.21.15.16:0/1639708513 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e780caa40 0x7f6e781f0650 unknown :-1 s=CLOSED pgs=2860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:57.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.806+0000 7f6e7cbe1700 1 -- 172.21.15.16:0/1639708513 >> 172.21.15.16:0/1639708513 conn(0x7f6e780b1f40 msgr2=0x7f6e780d0c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:57.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.806+0000 7f6e7cbe1700 1 -- 172.21.15.16:0/1639708513 shutdown_connections 2023-10-16T13:47:57.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.806+0000 7f6e7cbe1700 1 -- 172.21.15.16:0/1639708513 wait complete. 2023-10-16T13:47:57.823 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T13:47:57.831 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:47:57.829+0000 7f250e765200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T13:47:57.859 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:57.857+0000 7eff06a04200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T13:47:57.909 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:47:57.906+0000 7f250e765200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T13:47:57.937 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:57.935+0000 7eff06a04200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T13:47:57.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.953+0000 7f1e6c0a8700 1 Processor -- start 2023-10-16T13:47:57.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.953+0000 7f1e6c0a8700 1 -- start start 2023-10-16T13:47:57.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.954+0000 7f1e6c0a8700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f1e640d4060 0x7f1e640d1520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:57.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.954+0000 7f1e6c0a8700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1e640d1d20 0x7f1e640d21b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:57.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.954+0000 7f1e6c0a8700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e640d4540 con 0x7f1e6415da10 2023-10-16T13:47:57.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.954+0000 7f1e6c0a8700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e640cf9b0 con 0x7f1e640d1d20 2023-10-16T13:47:57.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.954+0000 7f1e6c0a8700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e640d26f0 con 0x7f1e640d4060 2023-10-16T13:47:57.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.954+0000 7f1e69e44700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f1e6415da10 0x7f1e640cf2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:45766/0 (socket says 172.21.15.16:45766) 2023-10-16T13:47:57.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.954+0000 7f1e6a645700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1e640d1d20 0x7f1e640d21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:57.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.954+0000 7f1e69e44700 1 -- 172.21.15.16:0/1704204939 learned_addr learned my addr 172.21.15.16:0/1704204939 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:47:57.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.954+0000 7f1e69643700 1 --2- 172.21.15.16:0/1704204939 >> v2:172.21.15.49:3300/0 conn(0x7f1e640d4060 0x7f1e640d1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:57.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.955+0000 7f1e6a645700 1 -- 172.21.15.16:0/1704204939 >> v2:172.21.15.49:3300/0 conn(0x7f1e640d4060 msgr2=0x7f1e640d1520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:57.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.955+0000 7f1e6a645700 1 --2- 172.21.15.16:0/1704204939 >> v2:172.21.15.49:3300/0 conn(0x7f1e640d4060 0x7f1e640d1520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:57.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.955+0000 7f1e6a645700 1 -- 172.21.15.16:0/1704204939 >> v1:172.21.15.16:6789/0 conn(0x7f1e6415da10 legacy=0x7f1e640cf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:57.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.955+0000 7f1e68e42700 1 -- 172.21.15.16:0/1704204939 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2198787278 0 0) 0x7f1e640d4540 con 0x7f1e6415da10 2023-10-16T13:47:57.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.955+0000 7f1e6a645700 1 -- 172.21.15.16:0/1704204939 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e640ce7e0 con 0x7f1e640d1d20 2023-10-16T13:47:57.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.955+0000 7f1e6a645700 1 --2- 172.21.15.16:0/1704204939 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e640d1d20 0x7f1e640d21b0 secure :-1 s=READY pgs=2862 cs=0 l=1 rev1=1 crypto rx=0x7f1e640cdf70 tx=0x7f1e5c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d9de7199cea6e9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:57.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.955+0000 7f1e68e42700 1 -- 172.21.15.16:0/1704204939 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e5c00e070 con 0x7f1e640d1d20 2023-10-16T13:47:57.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.955+0000 7f1e68e42700 1 -- 172.21.15.16:0/1704204939 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e5c00bb20 con 0x7f1e640d1d20 2023-10-16T13:47:57.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.955+0000 7f1e68e42700 1 -- 172.21.15.16:0/1704204939 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e5c012430 con 0x7f1e640d1d20 2023-10-16T13:47:57.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.956+0000 7f1e6c0a8700 1 -- 172.21.15.16:0/1704204939 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e640d1d20 msgr2=0x7f1e640d21b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:57.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.956+0000 7f1e6c0a8700 1 --2- 172.21.15.16:0/1704204939 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e640d1d20 0x7f1e640d21b0 secure :-1 s=READY pgs=2862 cs=0 l=1 rev1=1 crypto rx=0x7f1e640cdf70 tx=0x7f1e5c00b330 comp rx=0 tx=0).stop 2023-10-16T13:47:57.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.956+0000 7f1e6c0a8700 1 -- 172.21.15.16:0/1704204939 shutdown_connections 2023-10-16T13:47:57.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.956+0000 7f1e6c0a8700 1 --2- 172.21.15.16:0/1704204939 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e640d1d20 0x7f1e640d21b0 unknown :-1 s=CLOSED pgs=2862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:57.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.956+0000 7f1e6c0a8700 1 --2- 172.21.15.16:0/1704204939 >> v2:172.21.15.49:3300/0 conn(0x7f1e640d4060 0x7f1e640d1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:57.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.956+0000 7f1e6c0a8700 1 -- 172.21.15.16:0/1704204939 >> 172.21.15.16:0/1704204939 conn(0x7f1e6409c9f0 msgr2=0x7f1e640d3740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:57.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.956+0000 7f1e6c0a8700 1 -- 172.21.15.16:0/1704204939 shutdown_connections 2023-10-16T13:47:57.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.956+0000 7f1e6c0a8700 1 -- 172.21.15.16:0/1704204939 wait complete. 2023-10-16T13:47:57.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.957+0000 7f1e6c0a8700 1 Processor -- start 2023-10-16T13:47:57.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.957+0000 7f1e6c0a8700 1 -- start start 2023-10-16T13:47:57.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.958+0000 7f1e6c0a8700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1e640d1d20 0x7f1e640b6d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:57.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.958+0000 7f1e6c0a8700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1e640d4060 0x7f1e640c8270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:57.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.958+0000 7f1e6c0a8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e6415da10 0x7f1e641f22f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:57.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.958+0000 7f1e6c0a8700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e64156700 con 0x7f1e6415da10 2023-10-16T13:47:57.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.958+0000 7f1e6c0a8700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e640b31a0 con 0x7f1e640d4060 2023-10-16T13:47:57.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.958+0000 7f1e6c0a8700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e640b72b0 con 0x7f1e640d1d20 2023-10-16T13:47:57.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.958+0000 7f1e6a645700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e6415da10 0x7f1e641f22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:57.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.958+0000 7f1e6a645700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e6415da10 0x7f1e641f22f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56502/0 (socket says 172.21.15.16:56502) 2023-10-16T13:47:57.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.958+0000 7f1e6a645700 1 -- 172.21.15.16:0/1616294938 learned_addr learned my addr 172.21.15.16:0/1616294938 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:57.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.958+0000 7f1e69643700 1 --2- 172.21.15.16:0/1616294938 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1e640d4060 0x7f1e640c8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:57.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.958+0000 7f1e69e44700 1 --2- 172.21.15.16:0/1616294938 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1e640d1d20 0x7f1e640b6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:57.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.959+0000 7f1e6a645700 1 -- 172.21.15.16:0/1616294938 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1e640d1d20 msgr2=0x7f1e640b6d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:57.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.959+0000 7f1e6a645700 1 --2- 172.21.15.16:0/1616294938 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1e640d1d20 0x7f1e640b6d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:57.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.959+0000 7f1e6a645700 1 -- 172.21.15.16:0/1616294938 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1e640d4060 msgr2=0x7f1e640c8270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:57.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.959+0000 7f1e6a645700 1 --2- 172.21.15.16:0/1616294938 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1e640d4060 0x7f1e640c8270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:57.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.959+0000 7f1e6a645700 1 -- 172.21.15.16:0/1616294938 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e5c009a70 con 0x7f1e6415da10 2023-10-16T13:47:57.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.959+0000 7f1e6a645700 1 --2- 172.21.15.16:0/1616294938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e6415da10 0x7f1e641f22f0 secure :-1 s=READY pgs=2863 cs=0 l=1 rev1=1 crypto rx=0x7f1e5c014040 tx=0x7f1e5c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:57.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.959+0000 7f1e5a7fc700 1 -- 172.21.15.16:0/1616294938 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e5c004780 con 0x7f1e6415da10 2023-10-16T13:47:57.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.959+0000 7f1e5a7fc700 1 -- 172.21.15.16:0/1616294938 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e5c020070 con 0x7f1e6415da10 2023-10-16T13:47:57.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.959+0000 7f1e6c0a8700 1 -- 172.21.15.16:0/1616294938 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e640c87b0 con 0x7f1e6415da10 2023-10-16T13:47:57.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.959+0000 7f1e5a7fc700 1 -- 172.21.15.16:0/1616294938 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e5c0276c0 con 0x7f1e6415da10 2023-10-16T13:47:57.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.959+0000 7f1e6c0a8700 1 -- 172.21.15.16:0/1616294938 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e640b7600 con 0x7f1e6415da10 2023-10-16T13:47:57.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.960+0000 7f1e6c0a8700 1 -- 172.21.15.16:0/1616294938 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e6404ebb0 con 0x7f1e6415da10 2023-10-16T13:47:57.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.960+0000 7f1e5a7fc700 1 -- 172.21.15.16:0/1616294938 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 153) v1 ==== 89434+0+0 (secure 0 0 0) 0x7f1e5c012430 con 0x7f1e6415da10 2023-10-16T13:47:57.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.961+0000 7f1e5a7fc700 1 --2- 172.21.15.16:0/1616294938 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7f1e4806b140 0x7f1e4806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:57.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.961+0000 7f1e69e44700 1 -- 172.21.15.16:0/1616294938 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7f1e4806b140 msgr2=0x7f1e4806d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2221624769 2023-10-16T13:47:57.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.961+0000 7f1e5a7fc700 1 -- 172.21.15.16:0/1616294938 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(250..250 src has 1..250) v4 ==== 7451+0+0 (secure 0 0 0) 0x7f1e5c022070 con 0x7f1e6415da10 2023-10-16T13:47:57.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.961+0000 7f1e69e44700 1 --2- 172.21.15.16:0/1616294938 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7f1e4806b140 0x7f1e4806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:47:57.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:57.966+0000 7f1e5a7fc700 1 -- 172.21.15.16:0/1616294938 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1e5c0564e0 con 0x7f1e6415da10 2023-10-16T13:47:57.995 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:47:57.993+0000 7f250e765200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T13:47:58.028 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:58.025+0000 7eff06a04200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T13:47:58.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.162+0000 7f1e69e44700 1 -- 172.21.15.16:0/1616294938 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7f1e4806b140 msgr2=0x7f1e4806d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2221624769 2023-10-16T13:47:58.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.162+0000 7f1e69e44700 1 --2- 172.21.15.16:0/1616294938 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7f1e4806b140 0x7f1e4806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:47:58.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.170+0000 7f1e6c0a8700 1 -- 172.21.15.16:0/1616294938 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1e640d1b00 con 0x7f1e6415da10 2023-10-16T13:47:58.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.171+0000 7f1e5a7fc700 1 -- 172.21.15.16:0/1616294938 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v250) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1e5c059b00 con 0x7f1e6415da10 2023-10-16T13:47:58.173 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-10-16T13:47:58.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.173+0000 7f1e6c0a8700 1 -- 172.21.15.16:0/1616294938 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7f1e4806b140 msgr2=0x7f1e4806d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:47:58.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.173+0000 7f1e6c0a8700 1 --2- 172.21.15.16:0/1616294938 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7f1e4806b140 0x7f1e4806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:58.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.173+0000 7f1e6c0a8700 1 -- 172.21.15.16:0/1616294938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e6415da10 msgr2=0x7f1e641f22f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:58.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.173+0000 7f1e6c0a8700 1 --2- 172.21.15.16:0/1616294938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e6415da10 0x7f1e641f22f0 secure :-1 s=READY pgs=2863 cs=0 l=1 rev1=1 crypto rx=0x7f1e5c014040 tx=0x7f1e5c004ad0 comp rx=0 tx=0).stop 2023-10-16T13:47:58.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.173+0000 7f1e6c0a8700 1 -- 172.21.15.16:0/1616294938 shutdown_connections 2023-10-16T13:47:58.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.173+0000 7f1e6c0a8700 1 --2- 172.21.15.16:0/1616294938 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7f1e4806b140 0x7f1e4806d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:58.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.173+0000 7f1e6c0a8700 1 --2- 172.21.15.16:0/1616294938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e6415da10 0x7f1e641f22f0 unknown :-1 s=CLOSED pgs=2863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:58.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.173+0000 7f1e6c0a8700 1 --2- 172.21.15.16:0/1616294938 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1e640d1d20 0x7f1e640b6d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:58.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.173+0000 7f1e6c0a8700 1 --2- 172.21.15.16:0/1616294938 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1e640d4060 0x7f1e640c8270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:58.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.173+0000 7f1e6c0a8700 1 -- 172.21.15.16:0/1616294938 >> 172.21.15.16:0/1616294938 conn(0x7f1e6409c9f0 msgr2=0x7f1e6415dff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:58.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.173+0000 7f1e6c0a8700 1 -- 172.21.15.16:0/1616294938 shutdown_connections 2023-10-16T13:47:58.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.173+0000 7f1e6c0a8700 1 -- 172.21.15.16:0/1616294938 wait complete. 2023-10-16T13:47:58.188 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T13:47:58.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.355+0000 7fe04de45700 1 Processor -- start 2023-10-16T13:47:58.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.355+0000 7fe04de45700 1 -- start start 2023-10-16T13:47:58.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.356+0000 7fe04de45700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fe04815ec80 0x7fe0480cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:58.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.356+0000 7fe04de45700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe04815f060 0x7fe0480ce400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:58.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.356+0000 7fe04de45700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0480b2440 con 0x7fe0480d2110 2023-10-16T13:47:58.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.356+0000 7fe04de45700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0480b2c50 con 0x7fe04815f060 2023-10-16T13:47:58.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.356+0000 7fe04de45700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0480d00e0 con 0x7fe04815ec80 2023-10-16T13:47:58.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.356+0000 7fe046ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe04815f060 0x7fe0480ce400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:58.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.356+0000 7fe047fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe0480d2110 0x7fe0480b4440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:45816/0 (socket says 172.21.15.16:45816) 2023-10-16T13:47:58.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.356+0000 7fe047fff700 1 -- 172.21.15.16:0/684950248 learned_addr learned my addr 172.21.15.16:0/684950248 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:47:58.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.356+0000 7fe0477fe700 1 --2- 172.21.15.16:0/684950248 >> v2:172.21.15.49:3300/0 conn(0x7fe04815ec80 0x7fe0480cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:58.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.357+0000 7fe046ffd700 1 -- 172.21.15.16:0/684950248 >> v2:172.21.15.49:3300/0 conn(0x7fe04815ec80 msgr2=0x7fe0480cdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:58.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.357+0000 7fe046ffd700 1 --2- 172.21.15.16:0/684950248 >> v2:172.21.15.49:3300/0 conn(0x7fe04815ec80 0x7fe0480cdec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:58.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.357+0000 7fe046ffd700 1 -- 172.21.15.16:0/684950248 >> v1:172.21.15.16:6789/0 conn(0x7fe0480d2110 legacy=0x7fe0480b4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:58.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.357+0000 7fe0467fc700 1 -- 172.21.15.16:0/684950248 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3803563227 0 0) 0x7fe0480b2440 con 0x7fe0480d2110 2023-10-16T13:47:58.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.357+0000 7fe046ffd700 1 -- 172.21.15.16:0/684950248 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0480cef60 con 0x7fe04815f060 2023-10-16T13:47:58.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.357+0000 7fe046ffd700 1 --2- 172.21.15.16:0/684950248 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe04815f060 0x7fe0480ce400 secure :-1 s=READY pgs=2865 cs=0 l=1 rev1=1 crypto rx=0x7fe048067b20 tx=0x7fe03c009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=991a0b21953bc4c5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:58.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.357+0000 7fe0467fc700 1 -- 172.21.15.16:0/684950248 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe03c004220 con 0x7fe04815f060 2023-10-16T13:47:58.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.357+0000 7fe0467fc700 1 -- 172.21.15.16:0/684950248 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe03c004380 con 0x7fe04815f060 2023-10-16T13:47:58.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.357+0000 7fe0467fc700 1 -- 172.21.15.16:0/684950248 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe03c010890 con 0x7fe04815f060 2023-10-16T13:47:58.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.358+0000 7fe04de45700 1 -- 172.21.15.16:0/684950248 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe04815f060 msgr2=0x7fe0480ce400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:58.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.358+0000 7fe04de45700 1 --2- 172.21.15.16:0/684950248 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe04815f060 0x7fe0480ce400 secure :-1 s=READY pgs=2865 cs=0 l=1 rev1=1 crypto rx=0x7fe048067b20 tx=0x7fe03c009b80 comp rx=0 tx=0).stop 2023-10-16T13:47:58.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.358+0000 7fe04de45700 1 -- 172.21.15.16:0/684950248 shutdown_connections 2023-10-16T13:47:58.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.358+0000 7fe04de45700 1 --2- 172.21.15.16:0/684950248 >> v2:172.21.15.49:3300/0 conn(0x7fe04815ec80 0x7fe0480cdec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:58.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.358+0000 7fe04de45700 1 --2- 172.21.15.16:0/684950248 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe04815f060 0x7fe0480ce400 unknown :-1 s=CLOSED pgs=2865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:58.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.358+0000 7fe04de45700 1 -- 172.21.15.16:0/684950248 >> 172.21.15.16:0/684950248 conn(0x7fe0480d15f0 msgr2=0x7fe0480d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:58.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.358+0000 7fe04de45700 1 -- 172.21.15.16:0/684950248 shutdown_connections 2023-10-16T13:47:58.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.358+0000 7fe04de45700 1 -- 172.21.15.16:0/684950248 wait complete. 2023-10-16T13:47:58.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.359+0000 7fe04de45700 1 Processor -- start 2023-10-16T13:47:58.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.359+0000 7fe04de45700 1 -- start start 2023-10-16T13:47:58.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.360+0000 7fe04de45700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe0480d2110 0x7fe0480c86c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:58.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.360+0000 7fe04de45700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe04815ec80 0x7fe0480c7c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:58.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.360+0000 7fe04de45700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe04815f060 0x7fe04806fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:58.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.360+0000 7fe04de45700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe04814f5d0 con 0x7fe04815f060 2023-10-16T13:47:58.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.360+0000 7fe04de45700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0480b4ad0 con 0x7fe0480d2110 2023-10-16T13:47:58.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.360+0000 7fe04de45700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0480b5090 con 0x7fe04815ec80 2023-10-16T13:47:58.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.360+0000 7fe047fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe04815f060 0x7fe04806fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:58.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.360+0000 7fe047fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe04815f060 0x7fe04806fd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56548/0 (socket says 172.21.15.16:56548) 2023-10-16T13:47:58.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.360+0000 7fe047fff700 1 -- 172.21.15.16:0/583159266 learned_addr learned my addr 172.21.15.16:0/583159266 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:58.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.360+0000 7fe046ffd700 1 --2- 172.21.15.16:0/583159266 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe04815ec80 0x7fe0480c7c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:58.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.360+0000 7fe0477fe700 1 --2- 172.21.15.16:0/583159266 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe0480d2110 0x7fe0480c86c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:58.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.360+0000 7fe047fff700 1 -- 172.21.15.16:0/583159266 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe04815ec80 msgr2=0x7fe0480c7c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:58.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.360+0000 7fe047fff700 1 --2- 172.21.15.16:0/583159266 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe04815ec80 0x7fe0480c7c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:58.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.360+0000 7fe047fff700 1 -- 172.21.15.16:0/583159266 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe0480d2110 msgr2=0x7fe0480c86c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:58.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.360+0000 7fe047fff700 1 --2- 172.21.15.16:0/583159266 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe0480d2110 0x7fe0480c86c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:58.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.360+0000 7fe047fff700 1 -- 172.21.15.16:0/583159266 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe03c0094a0 con 0x7fe04815f060 2023-10-16T13:47:58.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.361+0000 7fe047fff700 1 --2- 172.21.15.16:0/583159266 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe04815f060 0x7fe04806fd80 secure :-1 s=READY pgs=2866 cs=0 l=1 rev1=1 crypto rx=0x7fe04000cca0 tx=0x7fe04000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:58.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.361+0000 7fe02ffff700 1 -- 172.21.15.16:0/583159266 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe04000ab70 con 0x7fe04815f060 2023-10-16T13:47:58.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.361+0000 7fe02ffff700 1 -- 172.21.15.16:0/583159266 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe04000acd0 con 0x7fe04815f060 2023-10-16T13:47:58.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.361+0000 7fe04de45700 1 -- 172.21.15.16:0/583159266 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0480705d0 con 0x7fe04815f060 2023-10-16T13:47:58.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.361+0000 7fe02ffff700 1 -- 172.21.15.16:0/583159266 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe040007310 con 0x7fe04815f060 2023-10-16T13:47:58.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.361+0000 7fe04de45700 1 -- 172.21.15.16:0/583159266 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0480b4d20 con 0x7fe04815f060 2023-10-16T13:47:58.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.362+0000 7fe04de45700 1 -- 172.21.15.16:0/583159266 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe04804ebb0 con 0x7fe04815f060 2023-10-16T13:47:58.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.363+0000 7fe02ffff700 1 -- 172.21.15.16:0/583159266 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 153) v1 ==== 89434+0+0 (secure 0 0 0) 0x7fe040007470 con 0x7fe04815f060 2023-10-16T13:47:58.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.363+0000 7fe02ffff700 1 --2- 172.21.15.16:0/583159266 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7fe03406b140 0x7fe03406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:58.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.363+0000 7fe0477fe700 1 -- 172.21.15.16:0/583159266 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7fe03406b140 msgr2=0x7fe03406d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2221624769 2023-10-16T13:47:58.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.363+0000 7fe02ffff700 1 -- 172.21.15.16:0/583159266 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(250..250 src has 1..250) v4 ==== 7451+0+0 (secure 0 0 0) 0x7fe040089ea0 con 0x7fe04815f060 2023-10-16T13:47:58.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.363+0000 7fe0477fe700 1 --2- 172.21.15.16:0/583159266 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7fe03406b140 0x7fe03406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:47:58.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.368+0000 7fe02ffff700 1 -- 172.21.15.16:0/583159266 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe040054520 con 0x7fe04815f060 2023-10-16T13:47:58.382 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:47:58.379+0000 7f250e765200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T13:47:58.428 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:58.426+0000 7eff06a04200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T13:47:58.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.564+0000 7fe0477fe700 1 -- 172.21.15.16:0/583159266 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7fe03406b140 msgr2=0x7fe03406d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2221624769 2023-10-16T13:47:58.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.564+0000 7fe0477fe700 1 --2- 172.21.15.16:0/583159266 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7fe03406b140 0x7fe03406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:47:58.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.570+0000 7fe04de45700 1 -- 172.21.15.16:0/583159266 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe0480b51d0 con 0x7fe04815f060 2023-10-16T13:47:58.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.571+0000 7fe02ffff700 1 -- 172.21.15.16:0/583159266 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v250) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe040018020 con 0x7fe04815f060 2023-10-16T13:47:58.574 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-10-16T13:47:58.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.573+0000 7fe04de45700 1 -- 172.21.15.16:0/583159266 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7fe03406b140 msgr2=0x7fe03406d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:47:58.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.573+0000 7fe04de45700 1 --2- 172.21.15.16:0/583159266 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7fe03406b140 0x7fe03406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:58.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.573+0000 7fe04de45700 1 -- 172.21.15.16:0/583159266 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe04815f060 msgr2=0x7fe04806fd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:58.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.573+0000 7fe04de45700 1 --2- 172.21.15.16:0/583159266 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe04815f060 0x7fe04806fd80 secure :-1 s=READY pgs=2866 cs=0 l=1 rev1=1 crypto rx=0x7fe04000cca0 tx=0x7fe04000a060 comp rx=0 tx=0).stop 2023-10-16T13:47:58.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.573+0000 7fe04de45700 1 -- 172.21.15.16:0/583159266 shutdown_connections 2023-10-16T13:47:58.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.573+0000 7fe04de45700 1 --2- 172.21.15.16:0/583159266 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7fe03406b140 0x7fe03406d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:58.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.573+0000 7fe04de45700 1 --2- 172.21.15.16:0/583159266 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe04815f060 0x7fe04806fd80 unknown :-1 s=CLOSED pgs=2866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:58.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.573+0000 7fe04de45700 1 --2- 172.21.15.16:0/583159266 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe0480d2110 0x7fe0480c86c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:58.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.573+0000 7fe04de45700 1 --2- 172.21.15.16:0/583159266 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe04815ec80 0x7fe0480c7c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:58.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.573+0000 7fe04de45700 1 -- 172.21.15.16:0/583159266 >> 172.21.15.16:0/583159266 conn(0x7fe0480d15f0 msgr2=0x7fe04806b8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:58.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.573+0000 7fe04de45700 1 -- 172.21.15.16:0/583159266 shutdown_connections 2023-10-16T13:47:58.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.573+0000 7fe04de45700 1 -- 172.21.15.16:0/583159266 wait complete. 2023-10-16T13:47:58.588 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2023-10-16T13:47:58.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.715+0000 7fd8b6f5d700 1 Processor -- start 2023-10-16T13:47:58.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.715+0000 7fd8b6f5d700 1 -- start start 2023-10-16T13:47:58.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.715+0000 7fd8b6f5d700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fd8b015ec80 0x7fd8b00cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:58.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.716+0000 7fd8b6f5d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd8b00d2110 0x7fd8b00cd1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:58.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.716+0000 7fd8b6f5d700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8b00b2440 con 0x7fd8b015f060 2023-10-16T13:47:58.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.716+0000 7fd8b6f5d700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8b00b2c50 con 0x7fd8b015ec80 2023-10-16T13:47:58.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.716+0000 7fd8b6f5d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b00d00e0 con 0x7fd8b00d2110 2023-10-16T13:47:58.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.716+0000 7fd8b4cf9700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fd8b015ec80 0x7fd8b00cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:58.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.716+0000 7fd8affff700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7fd8b015f060 0x7fd8b00b4440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:53012/0 (socket says 172.21.15.16:53012) 2023-10-16T13:47:58.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.716+0000 7fd8b54fa700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd8b00d2110 0x7fd8b00cd1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:58.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.716+0000 7fd8affff700 1 -- 172.21.15.16:0/1639271988 learned_addr learned my addr 172.21.15.16:0/1639271988 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:47:58.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.716+0000 7fd8b4cf9700 1 -- 172.21.15.16:0/1639271988 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd8b00d2110 msgr2=0x7fd8b00cd1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:58.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.716+0000 7fd8b4cf9700 1 --2- 172.21.15.16:0/1639271988 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd8b00d2110 0x7fd8b00cd1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:58.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.716+0000 7fd8b4cf9700 1 -- 172.21.15.16:0/1639271988 >> v1:172.21.15.49:6789/0 conn(0x7fd8b015f060 legacy=0x7fd8b00b4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:58.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.716+0000 7fd8b4cf9700 1 -- 172.21.15.16:0/1639271988 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8b00cef60 con 0x7fd8b015ec80 2023-10-16T13:47:58.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.716+0000 7fd8af7fe700 1 -- 172.21.15.16:0/1639271988 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1753554601 0 0) 0x7fd8b00b2440 con 0x7fd8b015f060 2023-10-16T13:47:58.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.716+0000 7fd8b4cf9700 1 --2- 172.21.15.16:0/1639271988 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd8b015ec80 0x7fd8b00cdec0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7fd8a4009840 tx=0x7fd8a4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=21b10d4456f84783 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:58.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.717+0000 7fd8af7fe700 1 -- 172.21.15.16:0/1639271988 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a4011830 con 0x7fd8b015ec80 2023-10-16T13:47:58.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.717+0000 7fd8af7fe700 1 -- 172.21.15.16:0/1639271988 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8a4011990 con 0x7fd8b015ec80 2023-10-16T13:47:58.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.717+0000 7fd8af7fe700 1 -- 172.21.15.16:0/1639271988 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a4011d20 con 0x7fd8b015ec80 2023-10-16T13:47:58.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.717+0000 7fd8b6f5d700 1 -- 172.21.15.16:0/1639271988 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd8b015ec80 msgr2=0x7fd8b00cdec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:58.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.717+0000 7fd8b6f5d700 1 --2- 172.21.15.16:0/1639271988 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd8b015ec80 0x7fd8b00cdec0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7fd8a4009840 tx=0x7fd8a4009b50 comp rx=0 tx=0).stop 2023-10-16T13:47:58.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.717+0000 7fd8b6f5d700 1 -- 172.21.15.16:0/1639271988 shutdown_connections 2023-10-16T13:47:58.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.717+0000 7fd8b6f5d700 1 --2- 172.21.15.16:0/1639271988 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd8b00d2110 0x7fd8b00cd1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:58.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.717+0000 7fd8b6f5d700 1 --2- 172.21.15.16:0/1639271988 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd8b015ec80 0x7fd8b00cdec0 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:58.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.717+0000 7fd8b6f5d700 1 -- 172.21.15.16:0/1639271988 >> 172.21.15.16:0/1639271988 conn(0x7fd8b00d15f0 msgr2=0x7fd8b00d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:58.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.717+0000 7fd8b6f5d700 1 -- 172.21.15.16:0/1639271988 shutdown_connections 2023-10-16T13:47:58.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.717+0000 7fd8b6f5d700 1 -- 172.21.15.16:0/1639271988 wait complete. 2023-10-16T13:47:58.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.718+0000 7fd8b6f5d700 1 Processor -- start 2023-10-16T13:47:58.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.719+0000 7fd8b6f5d700 1 -- start start 2023-10-16T13:47:58.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.719+0000 7fd8b6f5d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd8b00d2110 0x7fd8b0151a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:58.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.719+0000 7fd8b6f5d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd8b015ec80 0x7fd8b00c86c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:58.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.719+0000 7fd8b6f5d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8b015f060 0x7fd8b00ca330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:58.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.719+0000 7fd8b6f5d700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b01f4420 con 0x7fd8b015f060 2023-10-16T13:47:58.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.719+0000 7fd8b6f5d700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b01f4670 con 0x7fd8b015ec80 2023-10-16T13:47:58.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.719+0000 7fd8b6f5d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b01f4950 con 0x7fd8b00d2110 2023-10-16T13:47:58.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.720+0000 7fd8b54fa700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8b015f060 0x7fd8b00ca330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:58.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.720+0000 7fd8affff700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd8b015ec80 0x7fd8b00c86c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:58.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.720+0000 7fd8b4cf9700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd8b00d2110 0x7fd8b0151a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:58.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.720+0000 7fd8affff700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd8b015ec80 0x7fd8b00c86c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:55624/0 (socket says 172.21.15.16:55624) 2023-10-16T13:47:58.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.720+0000 7fd8b54fa700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8b015f060 0x7fd8b00ca330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56564/0 (socket says 172.21.15.16:56564) 2023-10-16T13:47:58.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.720+0000 7fd8b4cf9700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd8b00d2110 0x7fd8b0151a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:44818/0 (socket says 172.21.15.16:44818) 2023-10-16T13:47:58.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.720+0000 7fd8b54fa700 1 -- 172.21.15.16:0/2987615072 learned_addr learned my addr 172.21.15.16:0/2987615072 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:58.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.720+0000 7fd8affff700 1 -- 172.21.15.16:0/2987615072 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd8b00d2110 msgr2=0x7fd8b0151a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:58.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.720+0000 7fd8affff700 1 --2- 172.21.15.16:0/2987615072 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd8b00d2110 0x7fd8b0151a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:58.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.720+0000 7fd8affff700 1 -- 172.21.15.16:0/2987615072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8b015f060 msgr2=0x7fd8b00ca330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:58.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.720+0000 7fd8affff700 1 --2- 172.21.15.16:0/2987615072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8b015f060 0x7fd8b00ca330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:58.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.720+0000 7fd8affff700 1 -- 172.21.15.16:0/2987615072 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8a40094a0 con 0x7fd8b015ec80 2023-10-16T13:47:58.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.720+0000 7fd8b54fa700 1 --2- 172.21.15.16:0/2987615072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8b015f060 0x7fd8b00ca330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:47:58.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.720+0000 7fd8affff700 1 --2- 172.21.15.16:0/2987615072 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd8b015ec80 0x7fd8b00c86c0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fd89c00c3e0 tx=0x7fd89c00c6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:58.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.720+0000 7fd8ad7fa700 1 -- 172.21.15.16:0/2987615072 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd89c004a70 con 0x7fd8b015ec80 2023-10-16T13:47:58.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.720+0000 7fd8ad7fa700 1 -- 172.21.15.16:0/2987615072 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd89c004bd0 con 0x7fd8b015ec80 2023-10-16T13:47:58.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.720+0000 7fd8b6f5d700 1 -- 172.21.15.16:0/2987615072 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8b01f4c30 con 0x7fd8b015ec80 2023-10-16T13:47:58.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.720+0000 7fd8ad7fa700 1 -- 172.21.15.16:0/2987615072 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd89c0070d0 con 0x7fd8b015ec80 2023-10-16T13:47:58.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.720+0000 7fd8b6f5d700 1 -- 172.21.15.16:0/2987615072 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8b00ca870 con 0x7fd8b015ec80 2023-10-16T13:47:58.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.721+0000 7fd8b6f5d700 1 -- 172.21.15.16:0/2987615072 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8b004ebb0 con 0x7fd8b015ec80 2023-10-16T13:47:58.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.722+0000 7fd8ad7fa700 1 -- 172.21.15.16:0/2987615072 <== mon.1 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 153) v1 ==== 89434+0+0 (secure 0 0 0) 0x7fd89c007230 con 0x7fd8b015ec80 2023-10-16T13:47:58.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.722+0000 7fd8ad7fa700 1 --2- 172.21.15.16:0/2987615072 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7fd89406b140 0x7fd89406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:58.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.722+0000 7fd8b4cf9700 1 -- 172.21.15.16:0/2987615072 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7fd89406b140 msgr2=0x7fd89406d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2221624769 2023-10-16T13:47:58.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.722+0000 7fd8ad7fa700 1 -- 172.21.15.16:0/2987615072 <== mon.1 v2:172.21.15.49:3300/0 5 ==== osd_map(250..250 src has 1..250) v4 ==== 7451+0+0 (secure 0 0 0) 0x7fd89c089c10 con 0x7fd8b015ec80 2023-10-16T13:47:58.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.722+0000 7fd8b4cf9700 1 --2- 172.21.15.16:0/2987615072 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7fd89406b140 0x7fd89406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:47:58.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.727+0000 7fd8ad7fa700 1 -- 172.21.15.16:0/2987615072 <== mon.1 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd89c054280 con 0x7fd8b015ec80 2023-10-16T13:47:58.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.923+0000 7fd8b4cf9700 1 -- 172.21.15.16:0/2987615072 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7fd89406b140 msgr2=0x7fd89406d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2221624769 2023-10-16T13:47:58.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.923+0000 7fd8b4cf9700 1 --2- 172.21.15.16:0/2987615072 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7fd89406b140 0x7fd89406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:47:58.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.978+0000 7fd8b6f5d700 1 -- 172.21.15.16:0/2987615072 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd8b00cd9c0 con 0x7fd8b015ec80 2023-10-16T13:47:58.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.978+0000 7fd8ad7fa700 1 -- 172.21.15.16:0/2987615072 <== mon.1 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v250) v1 ==== 107+0+11 (secure 0 0 0) 0x7fd89c0578a0 con 0x7fd8b015ec80 2023-10-16T13:47:58.981 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-10-16T13:47:58.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.980+0000 7fd8b6f5d700 1 -- 172.21.15.16:0/2987615072 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7fd89406b140 msgr2=0x7fd89406d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:47:58.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.980+0000 7fd8b6f5d700 1 --2- 172.21.15.16:0/2987615072 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7fd89406b140 0x7fd89406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:58.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.980+0000 7fd8b6f5d700 1 -- 172.21.15.16:0/2987615072 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd8b015ec80 msgr2=0x7fd8b00c86c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:58.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.980+0000 7fd8b6f5d700 1 --2- 172.21.15.16:0/2987615072 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd8b015ec80 0x7fd8b00c86c0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fd89c00c3e0 tx=0x7fd89c00c6f0 comp rx=0 tx=0).stop 2023-10-16T13:47:58.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.980+0000 7fd8b6f5d700 1 -- 172.21.15.16:0/2987615072 shutdown_connections 2023-10-16T13:47:58.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.981+0000 7fd8b6f5d700 1 --2- 172.21.15.16:0/2987615072 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7fd89406b140 0x7fd89406d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:58.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.981+0000 7fd8b6f5d700 1 --2- 172.21.15.16:0/2987615072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8b015f060 0x7fd8b00ca330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:58.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.981+0000 7fd8b6f5d700 1 --2- 172.21.15.16:0/2987615072 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd8b00d2110 0x7fd8b0151a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:58.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.981+0000 7fd8b6f5d700 1 --2- 172.21.15.16:0/2987615072 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd8b015ec80 0x7fd8b00c86c0 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:58.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.981+0000 7fd8b6f5d700 1 -- 172.21.15.16:0/2987615072 >> 172.21.15.16:0/2987615072 conn(0x7fd8b00d15f0 msgr2=0x7fd8b015c810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:58.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.981+0000 7fd8b6f5d700 1 -- 172.21.15.16:0/2987615072 shutdown_connections 2023-10-16T13:47:58.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:58.981+0000 7fd8b6f5d700 1 -- 172.21.15.16:0/2987615072 wait complete. 2023-10-16T13:47:58.990 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:47:58.987+0000 7f250e765200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T13:47:58.997 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2023-10-16T13:47:59.048 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:59.045+0000 7eff06a04200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T13:47:59.062 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:47:59.059+0000 7f250e765200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T13:47:59.123 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:59.121+0000 7eff06a04200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T13:47:59.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.127+0000 7ff04b045700 1 Processor -- start 2023-10-16T13:47:59.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.127+0000 7ff04b045700 1 -- start start 2023-10-16T13:47:59.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.128+0000 7ff04b045700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7ff04415ec80 0x7ff0440cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:59.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.128+0000 7ff04b045700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff0440d2110 0x7ff0440cd1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:59.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.128+0000 7ff04b045700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0440b2440 con 0x7ff04415f060 2023-10-16T13:47:59.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.128+0000 7ff04b045700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0440b2c50 con 0x7ff04415ec80 2023-10-16T13:47:59.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.128+0000 7ff04b045700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0440d00e0 con 0x7ff0440d2110 2023-10-16T13:47:59.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.128+0000 7ff048de1700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7ff04415ec80 0x7ff0440cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:59.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.128+0000 7ff043fff700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7ff04415f060 0x7ff0440b4440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:53028/0 (socket says 172.21.15.16:53028) 2023-10-16T13:47:59.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.128+0000 7ff048de1700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7ff04415ec80 0x7ff0440cdec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:55632/0 (socket says 172.21.15.16:55632) 2023-10-16T13:47:59.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.128+0000 7ff043fff700 1 -- 172.21.15.16:0/984459522 learned_addr learned my addr 172.21.15.16:0/984459522 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:47:59.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.129+0000 7ff048de1700 1 -- 172.21.15.16:0/984459522 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff0440d2110 msgr2=0x7ff0440cd1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:59.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.129+0000 7ff048de1700 1 --2- 172.21.15.16:0/984459522 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff0440d2110 0x7ff0440cd1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:59.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.129+0000 7ff048de1700 1 -- 172.21.15.16:0/984459522 >> v1:172.21.15.49:6789/0 conn(0x7ff04415f060 legacy=0x7ff0440b4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:59.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.129+0000 7ff048de1700 1 -- 172.21.15.16:0/984459522 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0440cef60 con 0x7ff04415ec80 2023-10-16T13:47:59.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.129+0000 7ff048de1700 1 --2- 172.21.15.16:0/984459522 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff04415ec80 0x7ff0440cdec0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7ff038009840 tx=0x7ff038009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c557607441a09871 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:59.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.129+0000 7ff0437fe700 1 -- 172.21.15.16:0/984459522 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff038011830 con 0x7ff04415ec80 2023-10-16T13:47:59.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.129+0000 7ff0437fe700 1 -- 172.21.15.16:0/984459522 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff038011990 con 0x7ff04415ec80 2023-10-16T13:47:59.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.129+0000 7ff0437fe700 1 -- 172.21.15.16:0/984459522 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff038011d20 con 0x7ff04415ec80 2023-10-16T13:47:59.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.130+0000 7ff04b045700 1 -- 172.21.15.16:0/984459522 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff04415ec80 msgr2=0x7ff0440cdec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:59.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.130+0000 7ff04b045700 1 --2- 172.21.15.16:0/984459522 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff04415ec80 0x7ff0440cdec0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7ff038009840 tx=0x7ff038009b50 comp rx=0 tx=0).stop 2023-10-16T13:47:59.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.130+0000 7ff04b045700 1 -- 172.21.15.16:0/984459522 shutdown_connections 2023-10-16T13:47:59.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.130+0000 7ff04b045700 1 --2- 172.21.15.16:0/984459522 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff0440d2110 0x7ff0440cd1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:59.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.130+0000 7ff04b045700 1 --2- 172.21.15.16:0/984459522 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff04415ec80 0x7ff0440cdec0 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:59.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.130+0000 7ff04b045700 1 -- 172.21.15.16:0/984459522 >> 172.21.15.16:0/984459522 conn(0x7ff0440d15f0 msgr2=0x7ff0440d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:59.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.130+0000 7ff04b045700 1 -- 172.21.15.16:0/984459522 shutdown_connections 2023-10-16T13:47:59.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.130+0000 7ff04b045700 1 -- 172.21.15.16:0/984459522 wait complete. 2023-10-16T13:47:59.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.131+0000 7ff04b045700 1 Processor -- start 2023-10-16T13:47:59.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.131+0000 7ff04b045700 1 -- start start 2023-10-16T13:47:59.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.132+0000 7ff04b045700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff0440d2110 0x7ff0440b6b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:59.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.132+0000 7ff04b045700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff04415ec80 0x7ff04406ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:59.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.132+0000 7ff04b045700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff04415f060 0x7ff04415d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:59.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.132+0000 7ff04b045700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff044158f40 con 0x7ff04415f060 2023-10-16T13:47:59.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.132+0000 7ff04b045700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0440ca720 con 0x7ff04415ec80 2023-10-16T13:47:59.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.132+0000 7ff04b045700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0440caa00 con 0x7ff0440d2110 2023-10-16T13:47:59.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.132+0000 7ff0495e2700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff04415f060 0x7ff04415d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:59.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.132+0000 7ff043fff700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff04415ec80 0x7ff04406ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:59.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.132+0000 7ff048de1700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff0440d2110 0x7ff0440b6b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:59.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.132+0000 7ff0495e2700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff04415f060 0x7ff04415d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56580/0 (socket says 172.21.15.16:56580) 2023-10-16T13:47:59.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.132+0000 7ff0495e2700 1 -- 172.21.15.16:0/4288782247 learned_addr learned my addr 172.21.15.16:0/4288782247 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:59.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.132+0000 7ff048de1700 1 -- 172.21.15.16:0/4288782247 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff04415ec80 msgr2=0x7ff04406ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:59.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.132+0000 7ff048de1700 1 --2- 172.21.15.16:0/4288782247 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff04415ec80 0x7ff04406ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:59.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.132+0000 7ff048de1700 1 -- 172.21.15.16:0/4288782247 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff04415f060 msgr2=0x7ff04415d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:59.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.132+0000 7ff048de1700 1 --2- 172.21.15.16:0/4288782247 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff04415f060 0x7ff04415d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:59.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.132+0000 7ff048de1700 1 -- 172.21.15.16:0/4288782247 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0380094a0 con 0x7ff0440d2110 2023-10-16T13:47:59.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.132+0000 7ff043fff700 1 --2- 172.21.15.16:0/4288782247 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff04415ec80 0x7ff04406ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:47:59.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.132+0000 7ff0495e2700 1 --2- 172.21.15.16:0/4288782247 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff04415f060 0x7ff04415d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:47:59.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.133+0000 7ff048de1700 1 --2- 172.21.15.16:0/4288782247 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff0440d2110 0x7ff0440b6b20 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7ff038016000 tx=0x7ff038004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:59.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.133+0000 7ff0417fa700 1 -- 172.21.15.16:0/4288782247 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0380036a0 con 0x7ff0440d2110 2023-10-16T13:47:59.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.133+0000 7ff0417fa700 1 -- 172.21.15.16:0/4288782247 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff03801f070 con 0x7ff0440d2110 2023-10-16T13:47:59.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.133+0000 7ff04b045700 1 -- 172.21.15.16:0/4288782247 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0440cace0 con 0x7ff0440d2110 2023-10-16T13:47:59.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.133+0000 7ff0417fa700 1 -- 172.21.15.16:0/4288782247 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff038003a60 con 0x7ff0440d2110 2023-10-16T13:47:59.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.133+0000 7ff04b045700 1 -- 172.21.15.16:0/4288782247 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff04406d780 con 0x7ff0440d2110 2023-10-16T13:47:59.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.134+0000 7ff04b045700 1 -- 172.21.15.16:0/4288782247 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff04406d910 con 0x7ff0440d2110 2023-10-16T13:47:59.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.134+0000 7ff0417fa700 1 -- 172.21.15.16:0/4288782247 <== mon.2 v2:172.21.15.49:3301/0 4 ==== mgrmap(e 153) v1 ==== 89434+0+0 (secure 0 0 0) 0x7ff038003bc0 con 0x7ff0440d2110 2023-10-16T13:47:59.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.135+0000 7ff0417fa700 1 --2- 172.21.15.16:0/4288782247 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7ff02806b140 0x7ff02806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:59.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.135+0000 7ff043fff700 1 -- 172.21.15.16:0/4288782247 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7ff02806b140 msgr2=0x7ff02806d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2221624769 2023-10-16T13:47:59.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.135+0000 7ff0417fa700 1 -- 172.21.15.16:0/4288782247 <== mon.2 v2:172.21.15.49:3301/0 5 ==== osd_map(250..250 src has 1..250) v4 ==== 7451+0+0 (secure 0 0 0) 0x7ff038021070 con 0x7ff0440d2110 2023-10-16T13:47:59.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.135+0000 7ff043fff700 1 --2- 172.21.15.16:0/4288782247 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7ff02806b140 0x7ff02806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:47:59.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.140+0000 7ff0417fa700 1 -- 172.21.15.16:0/4288782247 <== mon.2 v2:172.21.15.49:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff038056060 con 0x7ff0440d2110 2023-10-16T13:47:59.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.336+0000 7ff043fff700 1 -- 172.21.15.16:0/4288782247 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7ff02806b140 msgr2=0x7ff02806d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2221624769 2023-10-16T13:47:59.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.336+0000 7ff043fff700 1 --2- 172.21.15.16:0/4288782247 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7ff02806b140 0x7ff02806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:47:59.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.346+0000 7ff04b045700 1 -- 172.21.15.16:0/4288782247 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) v1 -- 0x7ff04404ebb0 con 0x7ff0440d2110 2023-10-16T13:47:59.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.346+0000 7ff0417fa700 1 -- 172.21.15.16:0/4288782247 <== mon.2 v2:172.21.15.49:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v250) v1 ==== 103+0+11 (secure 0 0 0) 0x7ff038059680 con 0x7ff0440d2110 2023-10-16T13:47:59.349 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-10-16T13:47:59.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.348+0000 7ff04b045700 1 -- 172.21.15.16:0/4288782247 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7ff02806b140 msgr2=0x7ff02806d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:47:59.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.348+0000 7ff04b045700 1 --2- 172.21.15.16:0/4288782247 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7ff02806b140 0x7ff02806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:59.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.348+0000 7ff04b045700 1 -- 172.21.15.16:0/4288782247 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff0440d2110 msgr2=0x7ff0440b6b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:59.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.348+0000 7ff04b045700 1 --2- 172.21.15.16:0/4288782247 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff0440d2110 0x7ff0440b6b20 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7ff038016000 tx=0x7ff038004750 comp rx=0 tx=0).stop 2023-10-16T13:47:59.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.348+0000 7ff04b045700 1 -- 172.21.15.16:0/4288782247 shutdown_connections 2023-10-16T13:47:59.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.348+0000 7ff04b045700 1 --2- 172.21.15.16:0/4288782247 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7ff02806b140 0x7ff02806d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:59.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.348+0000 7ff04b045700 1 --2- 172.21.15.16:0/4288782247 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff04415f060 0x7ff04415d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:59.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.348+0000 7ff04b045700 1 --2- 172.21.15.16:0/4288782247 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff0440d2110 0x7ff0440b6b20 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:59.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.348+0000 7ff04b045700 1 --2- 172.21.15.16:0/4288782247 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff04415ec80 0x7ff04406ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:59.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.348+0000 7ff04b045700 1 -- 172.21.15.16:0/4288782247 >> 172.21.15.16:0/4288782247 conn(0x7ff0440d15f0 msgr2=0x7ff0440d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:59.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.349+0000 7ff04b045700 1 -- 172.21.15.16:0/4288782247 shutdown_connections 2023-10-16T13:47:59.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.349+0000 7ff04b045700 1 -- 172.21.15.16:0/4288782247 wait complete. 2023-10-16T13:47:59.364 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T13:47:59.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.488+0000 7ff6fcd82700 1 Processor -- start 2023-10-16T13:47:59.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.488+0000 7ff6fcd82700 1 -- start start 2023-10-16T13:47:59.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.489+0000 7ff6fcd82700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7ff6f8159c90 0x7ff6f815d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:59.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.489+0000 7ff6fcd82700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff6f80cb190 0x7ff6f815dee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:59.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.489+0000 7ff6fcd82700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6f806b1f0 con 0x7ff6f815a070 2023-10-16T13:47:59.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.489+0000 7ff6fcd82700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6f806b4b0 con 0x7ff6f80cb190 2023-10-16T13:47:59.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.489+0000 7ff6fcd82700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6f806b790 con 0x7ff6f8159c90 2023-10-16T13:47:59.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.489+0000 7ff6f6d9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff6f80cb190 0x7ff6f815dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:59.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.489+0000 7ff6f6d9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff6f80cb190 0x7ff6f815dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56590/0 (socket says 172.21.15.16:56590) 2023-10-16T13:47:59.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.489+0000 7ff6f6d9d700 1 -- 172.21.15.16:0/614641412 learned_addr learned my addr 172.21.15.16:0/614641412 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:59.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.489+0000 7ff6f659c700 1 --2- 172.21.15.16:0/614641412 >> v2:172.21.15.49:3300/0 conn(0x7ff6f8159c90 0x7ff6f815d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:59.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.489+0000 7ff6f559a700 1 -- 172.21.15.16:0/614641412 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3749179462 0 0) 0x7ff6f806b1f0 con 0x7ff6f815a070 2023-10-16T13:47:59.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.489+0000 7ff6f6d9d700 1 -- 172.21.15.16:0/614641412 >> v2:172.21.15.49:3300/0 conn(0x7ff6f8159c90 msgr2=0x7ff6f815d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:47:59.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.489+0000 7ff6f6d9d700 1 --2- 172.21.15.16:0/614641412 >> v2:172.21.15.49:3300/0 conn(0x7ff6f8159c90 0x7ff6f815d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:59.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.489+0000 7ff6f6d9d700 1 -- 172.21.15.16:0/614641412 >> v1:172.21.15.49:6789/0 conn(0x7ff6f815a070 legacy=0x7ff6f80ae520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:59.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.489+0000 7ff6f6d9d700 1 -- 172.21.15.16:0/614641412 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f806ba70 con 0x7ff6f80cb190 2023-10-16T13:47:59.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.489+0000 7ff6f659c700 1 --2- 172.21.15.16:0/614641412 >> v2:172.21.15.49:3300/0 conn(0x7ff6f8159c90 0x7ff6f815d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:47:59.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.489+0000 7ff6f6d9d700 1 --2- 172.21.15.16:0/614641412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6f80cb190 0x7ff6f815dee0 secure :-1 s=READY pgs=2867 cs=0 l=1 rev1=1 crypto rx=0x7ff6f80725a0 tx=0x7ff6f000d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dca8afc41ab7cf22 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:59.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.490+0000 7ff6f559a700 1 -- 172.21.15.16:0/614641412 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6f0010070 con 0x7ff6f80cb190 2023-10-16T13:47:59.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.490+0000 7ff6f559a700 1 -- 172.21.15.16:0/614641412 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6f000dad0 con 0x7ff6f80cb190 2023-10-16T13:47:59.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.490+0000 7ff6f559a700 1 -- 172.21.15.16:0/614641412 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6f000de80 con 0x7ff6f80cb190 2023-10-16T13:47:59.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.490+0000 7ff6fcd82700 1 -- 172.21.15.16:0/614641412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6f80cb190 msgr2=0x7ff6f815dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:59.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.490+0000 7ff6fcd82700 1 --2- 172.21.15.16:0/614641412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6f80cb190 0x7ff6f815dee0 secure :-1 s=READY pgs=2867 cs=0 l=1 rev1=1 crypto rx=0x7ff6f80725a0 tx=0x7ff6f000d430 comp rx=0 tx=0).stop 2023-10-16T13:47:59.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.490+0000 7ff6fcd82700 1 -- 172.21.15.16:0/614641412 shutdown_connections 2023-10-16T13:47:59.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.490+0000 7ff6fcd82700 1 --2- 172.21.15.16:0/614641412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6f80cb190 0x7ff6f815dee0 unknown :-1 s=CLOSED pgs=2867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:59.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.490+0000 7ff6fcd82700 1 --2- 172.21.15.16:0/614641412 >> v2:172.21.15.49:3300/0 conn(0x7ff6f8159c90 0x7ff6f815d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:59.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.490+0000 7ff6fcd82700 1 -- 172.21.15.16:0/614641412 >> 172.21.15.16:0/614641412 conn(0x7ff6f80cc5b0 msgr2=0x7ff6f80cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:59.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.490+0000 7ff6fcd82700 1 -- 172.21.15.16:0/614641412 shutdown_connections 2023-10-16T13:47:59.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.490+0000 7ff6fcd82700 1 -- 172.21.15.16:0/614641412 wait complete. 2023-10-16T13:47:59.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.492+0000 7ff6fcd82700 1 Processor -- start 2023-10-16T13:47:59.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.493+0000 7ff6fcd82700 1 -- start start 2023-10-16T13:47:59.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.494+0000 7ff6fcd82700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff6f80cb190 0x7ff6f80c4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:59.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.494+0000 7ff6fcd82700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6f8159c90 0x7ff6f80c4b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:59.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.494+0000 7ff6fcd82700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff6f815a070 0x7ff6f8149a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:59.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.494+0000 7ff6fcd82700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f80c6040 con 0x7ff6f8159c90 2023-10-16T13:47:59.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.494+0000 7ff6fcd82700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f80c6320 con 0x7ff6f80cb190 2023-10-16T13:47:59.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.494+0000 7ff6fcd82700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f80c53c0 con 0x7ff6f815a070 2023-10-16T13:47:59.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.494+0000 7ff6f5d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6f8159c90 0x7ff6f80c4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:59.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.494+0000 7ff6f5d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6f8159c90 0x7ff6f80c4b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56594/0 (socket says 172.21.15.16:56594) 2023-10-16T13:47:59.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.494+0000 7ff6f5d9b700 1 -- 172.21.15.16:0/3420118849 learned_addr learned my addr 172.21.15.16:0/3420118849 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:47:59.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.494+0000 7ff6f659c700 1 --2- 172.21.15.16:0/3420118849 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff6f80cb190 0x7ff6f80c4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:59.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.494+0000 7ff6f6d9d700 1 --2- 172.21.15.16:0/3420118849 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff6f815a070 0x7ff6f8149a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:47:59.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.494+0000 7ff6f5d9b700 1 -- 172.21.15.16:0/3420118849 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff6f815a070 msgr2=0x7ff6f8149a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:59.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.494+0000 7ff6f5d9b700 1 --2- 172.21.15.16:0/3420118849 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff6f815a070 0x7ff6f8149a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:59.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.494+0000 7ff6f5d9b700 1 -- 172.21.15.16:0/3420118849 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff6f80cb190 msgr2=0x7ff6f80c4640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:59.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.494+0000 7ff6f5d9b700 1 --2- 172.21.15.16:0/3420118849 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff6f80cb190 0x7ff6f80c4640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:59.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.494+0000 7ff6f5d9b700 1 -- 172.21.15.16:0/3420118849 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6e80094a0 con 0x7ff6f8159c90 2023-10-16T13:47:59.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.494+0000 7ff6f659c700 1 --2- 172.21.15.16:0/3420118849 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff6f80cb190 0x7ff6f80c4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:47:59.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.495+0000 7ff6f5d9b700 1 --2- 172.21.15.16:0/3420118849 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6f8159c90 0x7ff6f80c4b80 secure :-1 s=READY pgs=2868 cs=0 l=1 rev1=1 crypto rx=0x7ff6ec007fd0 tx=0x7ff6ec00c730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:47:59.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.495+0000 7ff6e6ffd700 1 -- 172.21.15.16:0/3420118849 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6ec0054e0 con 0x7ff6f8159c90 2023-10-16T13:47:59.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.495+0000 7ff6e6ffd700 1 -- 172.21.15.16:0/3420118849 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6ec00ce30 con 0x7ff6f8159c90 2023-10-16T13:47:59.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.495+0000 7ff6fcd82700 1 -- 172.21.15.16:0/3420118849 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6f000bd40 con 0x7ff6f8159c90 2023-10-16T13:47:59.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.495+0000 7ff6e6ffd700 1 -- 172.21.15.16:0/3420118849 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6ec0038c0 con 0x7ff6f8159c90 2023-10-16T13:47:59.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.495+0000 7ff6fcd82700 1 -- 172.21.15.16:0/3420118849 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6f80c5980 con 0x7ff6f8159c90 2023-10-16T13:47:59.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.496+0000 7ff6fcd82700 1 -- 172.21.15.16:0/3420118849 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6f8066f10 con 0x7ff6f8159c90 2023-10-16T13:47:59.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.497+0000 7ff6e6ffd700 1 -- 172.21.15.16:0/3420118849 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 153) v1 ==== 89434+0+0 (secure 0 0 0) 0x7ff6ec003a20 con 0x7ff6f8159c90 2023-10-16T13:47:59.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.497+0000 7ff6e6ffd700 1 --2- 172.21.15.16:0/3420118849 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7ff6d806b140 0x7ff6d806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:47:59.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.497+0000 7ff6f659c700 1 -- 172.21.15.16:0/3420118849 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7ff6d806b140 msgr2=0x7ff6d806d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2221624769 2023-10-16T13:47:59.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.497+0000 7ff6e6ffd700 1 -- 172.21.15.16:0/3420118849 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(250..250 src has 1..250) v4 ==== 7451+0+0 (secure 0 0 0) 0x7ff6ec089400 con 0x7ff6f8159c90 2023-10-16T13:47:59.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.497+0000 7ff6f659c700 1 --2- 172.21.15.16:0/3420118849 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7ff6d806b140 0x7ff6d806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:47:59.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.502+0000 7ff6e6ffd700 1 -- 172.21.15.16:0/3420118849 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff6ec053a90 con 0x7ff6f8159c90 2023-10-16T13:47:59.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.698+0000 7ff6f659c700 1 -- 172.21.15.16:0/3420118849 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7ff6d806b140 msgr2=0x7ff6d806d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2221624769 2023-10-16T13:47:59.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.698+0000 7ff6f659c700 1 --2- 172.21.15.16:0/3420118849 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7ff6d806b140 0x7ff6d806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:47:59.727 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:47:59.724+0000 7f250e765200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:47:59.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.726+0000 7ff6fcd82700 1 -- 172.21.15.16:0/3420118849 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7ff6f80c6180 con 0x7ff6f8159c90 2023-10-16T13:47:59.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.726+0000 7ff6e6ffd700 1 -- 172.21.15.16:0/3420118849 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v250) v1 ==== 81+0+584 (secure 0 0 0) 0x7ff6f80c6180 con 0x7ff6f8159c90 2023-10-16T13:47:59.729 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3947601914 2023-10-17T13:47:54.816700+0000 2023-10-16T13:47:59.729 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2759470534 2023-10-17T13:47:54.816700+0000 2023-10-16T13:47:59.729 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2974892940 2023-10-17T13:47:54.816700+0000 2023-10-16T13:47:59.730 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6833/1736104576 2023-10-17T13:47:54.816700+0000 2023-10-16T13:47:59.730 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6832/1736104576 2023-10-17T13:47:54.816700+0000 2023-10-16T13:47:59.730 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/318862437 2023-10-17T13:47:54.816700+0000 2023-10-16T13:47:59.730 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/413581668 2023-10-17T13:47:54.816700+0000 2023-10-16T13:47:59.730 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3740515383 2023-10-17T13:47:54.816700+0000 2023-10-16T13:47:59.731 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3105924907 2023-10-17T13:47:54.816700+0000 2023-10-16T13:47:59.731 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3617558038 2023-10-17T13:47:54.816700+0000 2023-10-16T13:47:59.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.728+0000 7ff6fcd82700 1 -- 172.21.15.16:0/3420118849 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7ff6d806b140 msgr2=0x7ff6d806d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:47:59.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.728+0000 7ff6fcd82700 1 --2- 172.21.15.16:0/3420118849 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7ff6d806b140 0x7ff6d806d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:59.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.728+0000 7ff6fcd82700 1 -- 172.21.15.16:0/3420118849 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6f8159c90 msgr2=0x7ff6f80c4b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:47:59.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.728+0000 7ff6fcd82700 1 --2- 172.21.15.16:0/3420118849 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6f8159c90 0x7ff6f80c4b80 secure :-1 s=READY pgs=2868 cs=0 l=1 rev1=1 crypto rx=0x7ff6ec007fd0 tx=0x7ff6ec00c730 comp rx=0 tx=0).stop 2023-10-16T13:47:59.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.728+0000 7ff6fcd82700 1 -- 172.21.15.16:0/3420118849 shutdown_connections 2023-10-16T13:47:59.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.728+0000 7ff6fcd82700 1 --2- 172.21.15.16:0/3420118849 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7ff6d806b140 0x7ff6d806d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:59.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.728+0000 7ff6fcd82700 1 --2- 172.21.15.16:0/3420118849 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff6f815a070 0x7ff6f8149a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:59.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.728+0000 7ff6fcd82700 1 --2- 172.21.15.16:0/3420118849 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff6f80cb190 0x7ff6f80c4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:59.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.728+0000 7ff6fcd82700 1 --2- 172.21.15.16:0/3420118849 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6f8159c90 0x7ff6f80c4b80 unknown :-1 s=CLOSED pgs=2868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:47:59.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.728+0000 7ff6fcd82700 1 -- 172.21.15.16:0/3420118849 >> 172.21.15.16:0/3420118849 conn(0x7ff6f80cc5b0 msgr2=0x7ff6f806a7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:47:59.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.728+0000 7ff6fcd82700 1 -- 172.21.15.16:0/3420118849 shutdown_connections 2023-10-16T13:47:59.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:47:59.729+0000 7ff6fcd82700 1 -- 172.21.15.16:0/3420118849 wait complete. 2023-10-16T13:47:59.734 INFO:teuthology.orchestra.run.smithi016.stderr:listed 10 entries 2023-10-16T13:47:59.743 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-16T13:47:59.744 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:47:59.744 DEBUG:teuthology.orchestra.run.smithi016:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-16T13:47:59.792 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2023-10-16T13:47:59.793 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:47:59.793 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-16T13:47:59.808 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:47:59.805+0000 7f250e765200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T13:47:59.810 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T13:47:59.810 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:47:59.813 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:59.811+0000 7eff06a04200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:47:59.828 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284204k,nr_inodes=4071051,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=753 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260540k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T13:47:59.828 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T13:47:59.829 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:47:59.846 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16287364k,nr_inodes=4071841,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20251 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3261172k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T13:47:59.847 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T13:47:59.847 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:47:59.886 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284204k,nr_inodes=4071051,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=753 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260540k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T13:47:59.887 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:47:59.898 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:47:59.896+0000 7eff06a04200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T13:47:59.961 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:47:59.958+0000 7f250e765200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:48:00.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.011+0000 7f6a90d7c700 1 Processor -- start 2023-10-16T13:48:00.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.011+0000 7f6a90d7c700 1 -- start start 2023-10-16T13:48:00.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.012+0000 7f6a90d7c700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f6a8c0d41c0 0x7f6a8c0d0040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:00.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.012+0000 7f6a90d7c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6a8c0d0580 0x7f6a8c06c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:00.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.012+0000 7f6a90d7c700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a8c06ba00 con 0x7f6a8c15d8d0 2023-10-16T13:48:00.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.012+0000 7f6a90d7c700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a8c0d0a10 con 0x7f6a8c0d41c0 2023-10-16T13:48:00.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.012+0000 7f6a90d7c700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a8c0d26d0 con 0x7f6a8c0d0580 2023-10-16T13:48:00.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.012+0000 7f6a89d9b700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f6a8c0d41c0 0x7f6a8c0d0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:00.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.012+0000 7f6a8ad9d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6a8c0d0580 0x7f6a8c06c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:00.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.012+0000 7f6a8a59c700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7f6a8c15d8d0 0x7f6a8c0d1c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:53084/0 (socket says 172.21.15.16:53084) 2023-10-16T13:48:00.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.012+0000 7f6a8a59c700 1 -- 172.21.15.16:0/2960178316 learned_addr learned my addr 172.21.15.16:0/2960178316 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:48:00.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.012+0000 7f6a89d9b700 1 -- 172.21.15.16:0/2960178316 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6a8c0d0580 msgr2=0x7f6a8c06c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:00.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.012+0000 7f6a89d9b700 1 --2- 172.21.15.16:0/2960178316 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6a8c0d0580 0x7f6a8c06c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:00.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.012+0000 7f6a89d9b700 1 -- 172.21.15.16:0/2960178316 >> v1:172.21.15.49:6789/0 conn(0x7f6a8c15d8d0 legacy=0x7f6a8c0d1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:00.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.012+0000 7f6a89d9b700 1 -- 172.21.15.16:0/2960178316 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a8c0d23f0 con 0x7f6a8c0d41c0 2023-10-16T13:48:00.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.012+0000 7f6a8ad9d700 1 --2- 172.21.15.16:0/2960178316 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6a8c0d0580 0x7f6a8c06c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:48:00.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.012+0000 7f6a89d9b700 1 --2- 172.21.15.16:0/2960178316 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6a8c0d41c0 0x7f6a8c0d0040 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f6a80009840 tx=0x7f6a80009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b646869547442fe0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:00.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.013+0000 7f6a8959a700 1 -- 172.21.15.16:0/2960178316 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a80011830 con 0x7f6a8c0d41c0 2023-10-16T13:48:00.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.013+0000 7f6a8959a700 1 -- 172.21.15.16:0/2960178316 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a80011990 con 0x7f6a8c0d41c0 2023-10-16T13:48:00.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.013+0000 7f6a8959a700 1 -- 172.21.15.16:0/2960178316 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a80011d20 con 0x7f6a8c0d41c0 2023-10-16T13:48:00.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.013+0000 7f6a90d7c700 1 -- 172.21.15.16:0/2960178316 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6a8c0d41c0 msgr2=0x7f6a8c0d0040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:00.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.013+0000 7f6a90d7c700 1 --2- 172.21.15.16:0/2960178316 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6a8c0d41c0 0x7f6a8c0d0040 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f6a80009840 tx=0x7f6a80009b50 comp rx=0 tx=0).stop 2023-10-16T13:48:00.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.013+0000 7f6a90d7c700 1 -- 172.21.15.16:0/2960178316 shutdown_connections 2023-10-16T13:48:00.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.013+0000 7f6a90d7c700 1 --2- 172.21.15.16:0/2960178316 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6a8c0d0580 0x7f6a8c06c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:00.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.013+0000 7f6a90d7c700 1 --2- 172.21.15.16:0/2960178316 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6a8c0d41c0 0x7f6a8c0d0040 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:00.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.013+0000 7f6a90d7c700 1 -- 172.21.15.16:0/2960178316 >> 172.21.15.16:0/2960178316 conn(0x7f6a8c0d3630 msgr2=0x7f6a8c0d3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:00.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.013+0000 7f6a90d7c700 1 -- 172.21.15.16:0/2960178316 shutdown_connections 2023-10-16T13:48:00.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.013+0000 7f6a90d7c700 1 -- 172.21.15.16:0/2960178316 wait complete. 2023-10-16T13:48:00.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.015+0000 7f6a90d7c700 1 Processor -- start 2023-10-16T13:48:00.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.015+0000 7f6a90d7c700 1 -- start start 2023-10-16T13:48:00.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.016+0000 7f6a90d7c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6a8c0d0580 0x7f6a8c15ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:00.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.016+0000 7f6a90d7c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6a8c0d41c0 0x7f6a8c1e6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:00.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.016+0000 7f6a90d7c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a8c15d8d0 0x7f6a8c1f2f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:00.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.016+0000 7f6a90d7c700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a8c0b8bf0 con 0x7f6a8c15d8d0 2023-10-16T13:48:00.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.016+0000 7f6a90d7c700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a8c15b3a0 con 0x7f6a8c0d0580 2023-10-16T13:48:00.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.016+0000 7f6a90d7c700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a8c1e5260 con 0x7f6a8c0d41c0 2023-10-16T13:48:00.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.016+0000 7f6a8ad9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a8c15d8d0 0x7f6a8c1f2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:00.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.016+0000 7f6a8a59c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6a8c0d0580 0x7f6a8c15ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:00.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.016+0000 7f6a8ad9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a8c15d8d0 0x7f6a8c1f2f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56638/0 (socket says 172.21.15.16:56638) 2023-10-16T13:48:00.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.016+0000 7f6a8ad9d700 1 -- 172.21.15.16:0/2514892993 learned_addr learned my addr 172.21.15.16:0/2514892993 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:00.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.016+0000 7f6a89d9b700 1 --2- 172.21.15.16:0/2514892993 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6a8c0d41c0 0x7f6a8c1e6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:00.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.016+0000 7f6a8ad9d700 1 -- 172.21.15.16:0/2514892993 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6a8c0d41c0 msgr2=0x7f6a8c1e6300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:00.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.016+0000 7f6a8ad9d700 1 --2- 172.21.15.16:0/2514892993 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6a8c0d41c0 0x7f6a8c1e6300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:00.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.016+0000 7f6a8ad9d700 1 -- 172.21.15.16:0/2514892993 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6a8c0d0580 msgr2=0x7f6a8c15ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:00.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.016+0000 7f6a8ad9d700 1 --2- 172.21.15.16:0/2514892993 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6a8c0d0580 0x7f6a8c15ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:00.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.016+0000 7f6a8ad9d700 1 -- 172.21.15.16:0/2514892993 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a84009a70 con 0x7f6a8c15d8d0 2023-10-16T13:48:00.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.016+0000 7f6a89d9b700 1 --2- 172.21.15.16:0/2514892993 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6a8c0d41c0 0x7f6a8c1e6300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:48:00.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.016+0000 7f6a8ad9d700 1 --2- 172.21.15.16:0/2514892993 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a8c15d8d0 0x7f6a8c1f2f30 secure :-1 s=READY pgs=2869 cs=0 l=1 rev1=1 crypto rx=0x7f6a8400efc0 tx=0x7f6a8400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:00.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.016+0000 7f6a7affd700 1 -- 172.21.15.16:0/2514892993 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a8400ca90 con 0x7f6a8c15d8d0 2023-10-16T13:48:00.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.016+0000 7f6a7affd700 1 -- 172.21.15.16:0/2514892993 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a8400f040 con 0x7f6a8c15d8d0 2023-10-16T13:48:00.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.016+0000 7f6a90d7c700 1 -- 172.21.15.16:0/2514892993 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a800094a0 con 0x7f6a8c15d8d0 2023-10-16T13:48:00.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.016+0000 7f6a7affd700 1 -- 172.21.15.16:0/2514892993 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a8400ce10 con 0x7f6a8c15d8d0 2023-10-16T13:48:00.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.017+0000 7f6a90d7c700 1 -- 172.21.15.16:0/2514892993 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a8c1e5820 con 0x7f6a8c15d8d0 2023-10-16T13:48:00.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.018+0000 7f6a90d7c700 1 -- 172.21.15.16:0/2514892993 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a8c04ebb0 con 0x7f6a8c15d8d0 2023-10-16T13:48:00.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.018+0000 7f6a7affd700 1 -- 172.21.15.16:0/2514892993 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 153) v1 ==== 89434+0+0 (secure 0 0 0) 0x7f6a84014480 con 0x7f6a8c15d8d0 2023-10-16T13:48:00.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.019+0000 7f6a7affd700 1 --2- 172.21.15.16:0/2514892993 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7f6a6c06b140 0x7f6a6c06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:00.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.019+0000 7f6a8a59c700 1 -- 172.21.15.16:0/2514892993 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7f6a6c06b140 msgr2=0x7f6a6c06d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2221624769 2023-10-16T13:48:00.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.019+0000 7f6a7affd700 1 -- 172.21.15.16:0/2514892993 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(250..250 src has 1..250) v4 ==== 7451+0+0 (secure 0 0 0) 0x7f6a8408a7a0 con 0x7f6a8c15d8d0 2023-10-16T13:48:00.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.019+0000 7f6a8a59c700 1 --2- 172.21.15.16:0/2514892993 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7f6a6c06b140 0x7f6a6c06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:48:00.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.024+0000 7f6a7affd700 1 -- 172.21.15.16:0/2514892993 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6a84054da0 con 0x7f6a8c15d8d0 2023-10-16T13:48:00.067 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:00.065+0000 7eff06a04200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:48:00.200 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:48:00.197+0000 7f250e765200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T13:48:00.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.220+0000 7f6a8a59c700 1 -- 172.21.15.16:0/2514892993 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7f6a6c06b140 msgr2=0x7f6a6c06d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2221624769 2023-10-16T13:48:00.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.220+0000 7f6a8a59c700 1 --2- 172.21.15.16:0/2514892993 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7f6a6c06b140 0x7f6a6c06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:48:00.277 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:48:00.274+0000 7f250e765200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T13:48:00.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.298+0000 7f6a90d7c700 1 -- 172.21.15.16:0/2514892993 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6a8c0d17e0 con 0x7f6a8c15d8d0 2023-10-16T13:48:00.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.298+0000 7f6a7affd700 1 -- 172.21.15.16:0/2514892993 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) v1 ==== 95+0+6147 (secure 0 0 0) 0x7f6a84009d70 con 0x7f6a8c15d8d0 2023-10-16T13:48:00.301 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:48:00.301 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":135,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9596,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/3939647668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":3939647668},{"type":"v1","addr":"172.21.15.49:6834","nonce":3939647668}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":125},{"gid":11185,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/2890113513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2890113513},{"type":"v1","addr":"172.21.15.16:6836","nonce":2890113513}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":125}],"filesystems":[{"mdsmap":{"epoch":130,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:47:23.843333+0000","modified":"2023-10-16T13:47:26.851597+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":12252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12252":{"gid":12252,"name":"a","rank":0,"incarnation":126,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/303278795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":303278795},{"type":"v1","addr":"172.21.15.16:6837","nonce":303278795}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:47:41.332096+0000","modified":"2023-10-16T13:47:44.368830+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11197},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11197":{"gid":11197,"name":"d","rank":0,"incarnation":132,"state":"up:active","state_seq":7,"addr":"172.21.15.49:6835/680748068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":680748068},{"type":"v1","addr":"172.21.15.49:6835","nonce":680748068}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2023-10-16T13:48:00.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.302+0000 7f6a90d7c700 1 -- 172.21.15.16:0/2514892993 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7f6a6c06b140 msgr2=0x7f6a6c06d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:48:00.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.302+0000 7f6a90d7c700 1 --2- 172.21.15.16:0/2514892993 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7f6a6c06b140 0x7f6a6c06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:00.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.302+0000 7f6a90d7c700 1 -- 172.21.15.16:0/2514892993 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a8c15d8d0 msgr2=0x7f6a8c1f2f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:00.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.302+0000 7f6a90d7c700 1 --2- 172.21.15.16:0/2514892993 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a8c15d8d0 0x7f6a8c1f2f30 secure :-1 s=READY pgs=2869 cs=0 l=1 rev1=1 crypto rx=0x7f6a8400efc0 tx=0x7f6a8400c3a0 comp rx=0 tx=0).stop 2023-10-16T13:48:00.305 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.302+0000 7f6a90d7c700 1 -- 172.21.15.16:0/2514892993 shutdown_connections 2023-10-16T13:48:00.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.302+0000 7f6a90d7c700 1 --2- 172.21.15.16:0/2514892993 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7f6a6c06b140 0x7f6a6c06d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:00.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.302+0000 7f6a90d7c700 1 --2- 172.21.15.16:0/2514892993 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a8c15d8d0 0x7f6a8c1f2f30 unknown :-1 s=CLOSED pgs=2869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:00.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.302+0000 7f6a90d7c700 1 --2- 172.21.15.16:0/2514892993 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6a8c0d0580 0x7f6a8c15ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:00.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.302+0000 7f6a90d7c700 1 --2- 172.21.15.16:0/2514892993 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6a8c0d41c0 0x7f6a8c1e6300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:00.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.302+0000 7f6a90d7c700 1 -- 172.21.15.16:0/2514892993 >> 172.21.15.16:0/2514892993 conn(0x7f6a8c0d3630 msgr2=0x7f6a8c06c2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:00.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.302+0000 7f6a90d7c700 1 -- 172.21.15.16:0/2514892993 shutdown_connections 2023-10-16T13:48:00.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.302+0000 7f6a90d7c700 1 -- 172.21.15.16:0/2514892993 wait complete. 2023-10-16T13:48:00.307 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 135 2023-10-16T13:48:00.317 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:48:00.323 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:00.320+0000 7eff06a04200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T13:48:00.405 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:00.403+0000 7eff06a04200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T13:48:00.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.448+0000 7fc1ddee5700 1 Processor -- start 2023-10-16T13:48:00.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.449+0000 7fc1ddee5700 1 -- start start 2023-10-16T13:48:00.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.449+0000 7fc1ddee5700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc1d80cc8a0 0x7fc1d815e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:00.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.449+0000 7fc1ddee5700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1d8161c30 con 0x7fc1d806a7e0 2023-10-16T13:48:00.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.449+0000 7fc1ddee5700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1d8161f10 con 0x7fc1d815b280 2023-10-16T13:48:00.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.449+0000 7fc1ddee5700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1d81621f0 con 0x7fc1d80cc8a0 2023-10-16T13:48:00.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.449+0000 7fc1d6ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc1d80cc8a0 0x7fc1d815e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:00.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.449+0000 7fc1d7fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc1d806a7e0 0x7fc1d80d02b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:45922/0 (socket says 172.21.15.16:45922) 2023-10-16T13:48:00.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.449+0000 7fc1d6ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc1d80cc8a0 0x7fc1d815e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56646/0 (socket says 172.21.15.16:56646) 2023-10-16T13:48:00.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.450+0000 7fc1d7fff700 1 -- 172.21.15.16:0/480547374 learned_addr learned my addr 172.21.15.16:0/480547374 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:48:00.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.450+0000 7fc1d6ffd700 1 -- 172.21.15.16:0/480547374 >> v1:172.21.15.49:6789/0 conn(0x7fc1d815b280 legacy=0x7fc1d81613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:00.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.450+0000 7fc1d6ffd700 1 -- 172.21.15.16:0/480547374 >> v1:172.21.15.16:6789/0 conn(0x7fc1d806a7e0 legacy=0x7fc1d80d02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:00.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.450+0000 7fc1d6ffd700 1 -- 172.21.15.16:0/480547374 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1d815df00 con 0x7fc1d80cc8a0 2023-10-16T13:48:00.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.450+0000 7fc1d67fc700 1 -- 172.21.15.16:0/480547374 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1525767835 0 0) 0x7fc1d8161c30 con 0x7fc1d806a7e0 2023-10-16T13:48:00.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.450+0000 7fc1d6ffd700 1 --2- 172.21.15.16:0/480547374 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1d80cc8a0 0x7fc1d815e700 secure :-1 s=READY pgs=2871 cs=0 l=1 rev1=1 crypto rx=0x7fc1cc009840 tx=0x7fc1cc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb6c6156232633f2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:00.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.450+0000 7fc1d67fc700 1 -- 172.21.15.16:0/480547374 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1cc011830 con 0x7fc1d80cc8a0 2023-10-16T13:48:00.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.450+0000 7fc1d67fc700 1 -- 172.21.15.16:0/480547374 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1cc011990 con 0x7fc1d80cc8a0 2023-10-16T13:48:00.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.450+0000 7fc1d67fc700 1 -- 172.21.15.16:0/480547374 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1cc011d20 con 0x7fc1d80cc8a0 2023-10-16T13:48:00.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.451+0000 7fc1ddee5700 1 -- 172.21.15.16:0/480547374 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1d80cc8a0 msgr2=0x7fc1d815e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:00.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.451+0000 7fc1ddee5700 1 --2- 172.21.15.16:0/480547374 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1d80cc8a0 0x7fc1d815e700 secure :-1 s=READY pgs=2871 cs=0 l=1 rev1=1 crypto rx=0x7fc1cc009840 tx=0x7fc1cc009b50 comp rx=0 tx=0).stop 2023-10-16T13:48:00.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.451+0000 7fc1ddee5700 1 -- 172.21.15.16:0/480547374 shutdown_connections 2023-10-16T13:48:00.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.451+0000 7fc1ddee5700 1 --2- 172.21.15.16:0/480547374 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1d80cc8a0 0x7fc1d815e700 unknown :-1 s=CLOSED pgs=2871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:00.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.451+0000 7fc1ddee5700 1 -- 172.21.15.16:0/480547374 >> 172.21.15.16:0/480547374 conn(0x7fc1d80b1fd0 msgr2=0x7fc1d80b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:00.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.451+0000 7fc1ddee5700 1 -- 172.21.15.16:0/480547374 shutdown_connections 2023-10-16T13:48:00.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.451+0000 7fc1ddee5700 1 -- 172.21.15.16:0/480547374 wait complete. 2023-10-16T13:48:00.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.452+0000 7fc1ddee5700 1 Processor -- start 2023-10-16T13:48:00.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.452+0000 7fc1ddee5700 1 -- start start 2023-10-16T13:48:00.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.453+0000 7fc1ddee5700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc1d806a7e0 0x7fc1d81587a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:00.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.453+0000 7fc1ddee5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1d80cc8a0 0x7fc1d8160160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:00.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.453+0000 7fc1ddee5700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc1d815b280 0x7fc1d806ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:00.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.453+0000 7fc1ddee5700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1d80c6db0 con 0x7fc1d80cc8a0 2023-10-16T13:48:00.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.453+0000 7fc1ddee5700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1d80c6ad0 con 0x7fc1d806a7e0 2023-10-16T13:48:00.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.453+0000 7fc1ddee5700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1d81f63d0 con 0x7fc1d815b280 2023-10-16T13:48:00.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.453+0000 7fc1d6ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1d80cc8a0 0x7fc1d8160160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:00.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.453+0000 7fc1d6ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1d80cc8a0 0x7fc1d8160160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56652/0 (socket says 172.21.15.16:56652) 2023-10-16T13:48:00.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.453+0000 7fc1d6ffd700 1 -- 172.21.15.16:0/1156157838 learned_addr learned my addr 172.21.15.16:0/1156157838 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:00.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.453+0000 7fc1d77fe700 1 --2- 172.21.15.16:0/1156157838 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc1d806a7e0 0x7fc1d81587a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:00.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.453+0000 7fc1d7fff700 1 --2- 172.21.15.16:0/1156157838 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc1d815b280 0x7fc1d806ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:00.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.454+0000 7fc1d6ffd700 1 -- 172.21.15.16:0/1156157838 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc1d815b280 msgr2=0x7fc1d806ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:00.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.454+0000 7fc1d6ffd700 1 --2- 172.21.15.16:0/1156157838 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc1d815b280 0x7fc1d806ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:00.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.454+0000 7fc1d6ffd700 1 -- 172.21.15.16:0/1156157838 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc1d806a7e0 msgr2=0x7fc1d81587a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:00.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.454+0000 7fc1d6ffd700 1 --2- 172.21.15.16:0/1156157838 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc1d806a7e0 0x7fc1d81587a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:00.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.454+0000 7fc1d6ffd700 1 -- 172.21.15.16:0/1156157838 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1cc0094a0 con 0x7fc1d80cc8a0 2023-10-16T13:48:00.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.454+0000 7fc1d6ffd700 1 --2- 172.21.15.16:0/1156157838 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1d80cc8a0 0x7fc1d8160160 secure :-1 s=READY pgs=2872 cs=0 l=1 rev1=1 crypto rx=0x7fc1cc0090b0 tx=0x7fc1cc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:00.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.454+0000 7fc1bffff700 1 -- 172.21.15.16:0/1156157838 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1cc01a7e0 con 0x7fc1d80cc8a0 2023-10-16T13:48:00.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.454+0000 7fc1bffff700 1 -- 172.21.15.16:0/1156157838 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1cc01a940 con 0x7fc1d80cc8a0 2023-10-16T13:48:00.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.454+0000 7fc1ddee5700 1 -- 172.21.15.16:0/1156157838 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1d81f66b0 con 0x7fc1d80cc8a0 2023-10-16T13:48:00.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.454+0000 7fc1bffff700 1 -- 172.21.15.16:0/1156157838 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1cc01acb0 con 0x7fc1d80cc8a0 2023-10-16T13:48:00.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.454+0000 7fc1ddee5700 1 -- 172.21.15.16:0/1156157838 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1d81606a0 con 0x7fc1d80cc8a0 2023-10-16T13:48:00.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.455+0000 7fc1ddee5700 1 -- 172.21.15.16:0/1156157838 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1d81f6bd0 con 0x7fc1d80cc8a0 2023-10-16T13:48:00.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.456+0000 7fc1bffff700 1 -- 172.21.15.16:0/1156157838 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 153) v1 ==== 89434+0+0 (secure 0 0 0) 0x7fc1cc022070 con 0x7fc1d80cc8a0 2023-10-16T13:48:00.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.457+0000 7fc1bffff700 1 --2- 172.21.15.16:0/1156157838 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7fc1c406b140 0x7fc1c406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:00.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.457+0000 7fc1d77fe700 1 -- 172.21.15.16:0/1156157838 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7fc1c406b140 msgr2=0x7fc1c406d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2221624769 2023-10-16T13:48:00.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.457+0000 7fc1bffff700 1 -- 172.21.15.16:0/1156157838 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(250..250 src has 1..250) v4 ==== 7451+0+0 (secure 0 0 0) 0x7fc1cc08b180 con 0x7fc1d80cc8a0 2023-10-16T13:48:00.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.457+0000 7fc1d77fe700 1 --2- 172.21.15.16:0/1156157838 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7fc1c406b140 0x7fc1c406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:48:00.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.461+0000 7fc1bffff700 1 -- 172.21.15.16:0/1156157838 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc1cc055a40 con 0x7fc1d80cc8a0 2023-10-16T13:48:00.554 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:48:00.552+0000 7f250e765200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T13:48:00.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.658+0000 7fc1d77fe700 1 -- 172.21.15.16:0/1156157838 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7fc1c406b140 msgr2=0x7fc1c406d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2221624769 2023-10-16T13:48:00.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.658+0000 7fc1d77fe700 1 --2- 172.21.15.16:0/1156157838 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7fc1c406b140 0x7fc1c406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:48:00.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.669+0000 7fc1ddee5700 1 -- 172.21.15.16:0/1156157838 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc1d81f6bd0 con 0x7fc1d80cc8a0 2023-10-16T13:48:00.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.671+0000 7fc1bffff700 1 -- 172.21.15.16:0/1156157838 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v250) v1 ==== 74+0+21164 (secure 0 0 0) 0x7fc1d81f6bd0 con 0x7fc1d80cc8a0 2023-10-16T13:48:00.673 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:48:00.674 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":250,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:47:54.816765+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2023-10-16T13:47:22.338349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"244","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2023-10-16T13:47:22.806327+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"245","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T13:47:39.075528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"248","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"backup_fs_data","create_time":"2023-10-16T13:47:40.314029+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"249","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/3947601914":"2023-10-17T13:47:54.816700+0000","172.21.15.16:0/2759470534":"2023-10-17T13:47:54.816700+0000","172.21.15.16:0/2974892940":"2023-10-17T13:47:54.816700+0000","172.21.15.16:6833/1736104576":"2023-10-17T13:47:54.816700+0000","172.21.15.16:6832/1736104576":"2023-10-17T13:47:54.816700+0000","172.21.15.16:0/318862437":"2023-10-17T13:47:54.816700+0000","172.21.15.16:0/413581668":"2023-10-17T13:47:54.816700+0000","172.21.15.16:0/3740515383":"2023-10-17T13:47:54.816700+0000","172.21.15.16:0/3105924907":"2023-10-17T13:47:54.816700+0000","172.21.15.16:0/3617558038":"2023-10-17T13:47:54.816700+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:48:00.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.673+0000 7fc1ddee5700 1 -- 172.21.15.16:0/1156157838 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7fc1c406b140 msgr2=0x7fc1c406d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:48:00.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.673+0000 7fc1ddee5700 1 --2- 172.21.15.16:0/1156157838 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7fc1c406b140 0x7fc1c406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:00.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.673+0000 7fc1ddee5700 1 -- 172.21.15.16:0/1156157838 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1d80cc8a0 msgr2=0x7fc1d8160160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:00.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.673+0000 7fc1ddee5700 1 --2- 172.21.15.16:0/1156157838 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1d80cc8a0 0x7fc1d8160160 secure :-1 s=READY pgs=2872 cs=0 l=1 rev1=1 crypto rx=0x7fc1cc0090b0 tx=0x7fc1cc004750 comp rx=0 tx=0).stop 2023-10-16T13:48:00.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.673+0000 7fc1ddee5700 1 -- 172.21.15.16:0/1156157838 shutdown_connections 2023-10-16T13:48:00.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.673+0000 7fc1ddee5700 1 --2- 172.21.15.16:0/1156157838 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7fc1c406b140 0x7fc1c406d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:00.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.673+0000 7fc1ddee5700 1 --2- 172.21.15.16:0/1156157838 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc1d815b280 0x7fc1d806ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:00.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.673+0000 7fc1ddee5700 1 --2- 172.21.15.16:0/1156157838 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc1d806a7e0 0x7fc1d81587a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:00.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.673+0000 7fc1ddee5700 1 --2- 172.21.15.16:0/1156157838 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1d80cc8a0 0x7fc1d8160160 unknown :-1 s=CLOSED pgs=2872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:00.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.673+0000 7fc1ddee5700 1 -- 172.21.15.16:0/1156157838 >> 172.21.15.16:0/1156157838 conn(0x7fc1d80b1fd0 msgr2=0x7fc1d80cb3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:00.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.673+0000 7fc1ddee5700 1 -- 172.21.15.16:0/1156157838 shutdown_connections 2023-10-16T13:48:00.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.673+0000 7fc1ddee5700 1 -- 172.21.15.16:0/1156157838 wait complete. 2023-10-16T13:48:00.689 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 31, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T13:47:22.338349+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '244', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 32, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T13:47:22.806327+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '245', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 33, 'pool_name': 'backup_fs_metadata', 'create_time': '2023-10-16T13:47:39.075528+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '248', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 34, 'pool_name': 'backup_fs_data', 'create_time': '2023-10-16T13:47:40.314029+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '249', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T13:48:00.690 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T13:48:00.691 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:00.688+0000 7eff06a04200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T13:48:00.786 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:48:00.783+0000 7f250e765200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T13:48:00.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.817+0000 7f590824b700 1 Processor -- start 2023-10-16T13:48:00.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.818+0000 7f590824b700 1 -- start start 2023-10-16T13:48:00.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.819+0000 7f590824b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5900160ee0 0x7f59000aeb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:00.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.819+0000 7f590824b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f59000ce890 0x7f590006aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:00.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.819+0000 7f590824b700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59000af4a0 con 0x7f59000af040 2023-10-16T13:48:00.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.819+0000 7f590824b700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59000cf770 con 0x7f5900160ee0 2023-10-16T13:48:00.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.819+0000 7f590824b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f590006aa10 con 0x7f59000ce890 2023-10-16T13:48:00.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.819+0000 7f5905fe7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5900160ee0 0x7f59000aeb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:00.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.819+0000 7f5905fe7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5900160ee0 0x7f59000aeb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56660/0 (socket says 172.21.15.16:56660) 2023-10-16T13:48:00.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.819+0000 7f5905fe7700 1 -- 172.21.15.16:0/2604946330 learned_addr learned my addr 172.21.15.16:0/2604946330 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:00.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.819+0000 7f59067e8700 1 --2- 172.21.15.16:0/2604946330 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f59000ce890 0x7f590006aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:00.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.819+0000 7f5905fe7700 1 -- 172.21.15.16:0/2604946330 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f59000ce890 msgr2=0x7f590006aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:00.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.819+0000 7f5905fe7700 1 --2- 172.21.15.16:0/2604946330 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f59000ce890 0x7f590006aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:00.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.819+0000 7f5905fe7700 1 -- 172.21.15.16:0/2604946330 >> v1:172.21.15.49:6789/0 conn(0x7f59000af040 legacy=0x7f59000ce180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:00.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.819+0000 7f5905fe7700 1 -- 172.21.15.16:0/2604946330 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59000ceed0 con 0x7f5900160ee0 2023-10-16T13:48:00.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.819+0000 7f5905fe7700 1 --2- 172.21.15.16:0/2604946330 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5900160ee0 0x7f59000aeb00 secure :-1 s=READY pgs=2873 cs=0 l=1 rev1=1 crypto rx=0x7f58fc0098f0 tx=0x7f58fc009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=191e3be7bb65647b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:00.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.819+0000 7f5904fe5700 1 -- 172.21.15.16:0/2604946330 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58fc011830 con 0x7f5900160ee0 2023-10-16T13:48:00.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.820+0000 7f5904fe5700 1 -- 172.21.15.16:0/2604946330 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58fc011990 con 0x7f5900160ee0 2023-10-16T13:48:00.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.820+0000 7f5904fe5700 1 -- 172.21.15.16:0/2604946330 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58fc011d20 con 0x7f5900160ee0 2023-10-16T13:48:00.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.820+0000 7f590824b700 1 -- 172.21.15.16:0/2604946330 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5900160ee0 msgr2=0x7f59000aeb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:00.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.820+0000 7f590824b700 1 --2- 172.21.15.16:0/2604946330 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5900160ee0 0x7f59000aeb00 secure :-1 s=READY pgs=2873 cs=0 l=1 rev1=1 crypto rx=0x7f58fc0098f0 tx=0x7f58fc009c00 comp rx=0 tx=0).stop 2023-10-16T13:48:00.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.820+0000 7f590824b700 1 -- 172.21.15.16:0/2604946330 shutdown_connections 2023-10-16T13:48:00.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.820+0000 7f590824b700 1 --2- 172.21.15.16:0/2604946330 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f59000ce890 0x7f590006aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:00.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.820+0000 7f590824b700 1 --2- 172.21.15.16:0/2604946330 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5900160ee0 0x7f59000aeb00 unknown :-1 s=CLOSED pgs=2873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:00.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.820+0000 7f590824b700 1 -- 172.21.15.16:0/2604946330 >> 172.21.15.16:0/2604946330 conn(0x7f59000cc5e0 msgr2=0x7f59000cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:00.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.821+0000 7f590824b700 1 -- 172.21.15.16:0/2604946330 shutdown_connections 2023-10-16T13:48:00.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.821+0000 7f590824b700 1 -- 172.21.15.16:0/2604946330 wait complete. 2023-10-16T13:48:00.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.822+0000 7f590824b700 1 Processor -- start 2023-10-16T13:48:00.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.822+0000 7f590824b700 1 -- start start 2023-10-16T13:48:00.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.823+0000 7f590824b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59000af040 0x7f59001562f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:00.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.823+0000 7f590824b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f59000ce890 0x7f5900158f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:00.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.823+0000 7f590824b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5900160ee0 0x7f5900159490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:00.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.823+0000 7f590824b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59000c43d0 con 0x7f59000af040 2023-10-16T13:48:00.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.823+0000 7f590824b700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59000c37c0 con 0x7f5900160ee0 2023-10-16T13:48:00.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.823+0000 7f590824b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59000c3aa0 con 0x7f59000ce890 2023-10-16T13:48:00.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.823+0000 7f5905fe7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59000af040 0x7f59001562f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:00.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.823+0000 7f5905fe7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59000af040 0x7f59001562f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56666/0 (socket says 172.21.15.16:56666) 2023-10-16T13:48:00.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.823+0000 7f5905fe7700 1 -- 172.21.15.16:0/3058802969 learned_addr learned my addr 172.21.15.16:0/3058802969 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:00.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.823+0000 7f59067e8700 1 --2- 172.21.15.16:0/3058802969 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5900160ee0 0x7f5900159490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:00.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.823+0000 7f59057e6700 1 --2- 172.21.15.16:0/3058802969 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f59000ce890 0x7f5900158f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:00.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.823+0000 7f5905fe7700 1 -- 172.21.15.16:0/3058802969 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f59000ce890 msgr2=0x7f5900158f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:00.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.823+0000 7f5905fe7700 1 --2- 172.21.15.16:0/3058802969 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f59000ce890 0x7f5900158f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:00.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.823+0000 7f5905fe7700 1 -- 172.21.15.16:0/3058802969 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5900160ee0 msgr2=0x7f5900159490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:00.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.823+0000 7f5905fe7700 1 --2- 172.21.15.16:0/3058802969 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5900160ee0 0x7f5900159490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:00.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.823+0000 7f5905fe7700 1 -- 172.21.15.16:0/3058802969 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58fc009550 con 0x7f59000af040 2023-10-16T13:48:00.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.823+0000 7f59057e6700 1 --2- 172.21.15.16:0/3058802969 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f59000ce890 0x7f5900158f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:48:00.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.823+0000 7f59067e8700 1 --2- 172.21.15.16:0/3058802969 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5900160ee0 0x7f5900159490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:48:00.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.823+0000 7f5905fe7700 1 --2- 172.21.15.16:0/3058802969 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59000af040 0x7f59001562f0 secure :-1 s=READY pgs=2874 cs=0 l=1 rev1=1 crypto rx=0x7f58fc009c40 tx=0x7f58fc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:00.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.823+0000 7f58f67fc700 1 -- 172.21.15.16:0/3058802969 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58fc0043c0 con 0x7f59000af040 2023-10-16T13:48:00.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.824+0000 7f58f67fc700 1 -- 172.21.15.16:0/3058802969 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58fc01f070 con 0x7f59000af040 2023-10-16T13:48:00.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.824+0000 7f58f67fc700 1 -- 172.21.15.16:0/3058802969 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58fc0036a0 con 0x7f59000af040 2023-10-16T13:48:00.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.824+0000 7f590824b700 1 -- 172.21.15.16:0/3058802969 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59000c3d80 con 0x7f59000af040 2023-10-16T13:48:00.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.824+0000 7f590824b700 1 -- 172.21.15.16:0/3058802969 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59000c5510 con 0x7f59000af040 2023-10-16T13:48:00.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.825+0000 7f590824b700 1 -- 172.21.15.16:0/3058802969 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f590006f460 con 0x7f59000af040 2023-10-16T13:48:00.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.825+0000 7f58f67fc700 1 -- 172.21.15.16:0/3058802969 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 153) v1 ==== 89434+0+0 (secure 0 0 0) 0x7f58fc021030 con 0x7f59000af040 2023-10-16T13:48:00.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.825+0000 7f58f67fc700 1 --2- 172.21.15.16:0/3058802969 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7f58e406b140 0x7f58e406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:00.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.825+0000 7f59057e6700 1 -- 172.21.15.16:0/3058802969 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7f58e406b140 msgr2=0x7f58e406d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2221624769 2023-10-16T13:48:00.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.825+0000 7f58f67fc700 1 -- 172.21.15.16:0/3058802969 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(250..250 src has 1..250) v4 ==== 7451+0+0 (secure 0 0 0) 0x7f58fc08b500 con 0x7f59000af040 2023-10-16T13:48:00.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.825+0000 7f59057e6700 1 --2- 172.21.15.16:0/3058802969 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7f58e406b140 0x7f58e406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:48:00.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:00.830+0000 7f58f67fc700 1 -- 172.21.15.16:0/3058802969 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f58fc055e00 con 0x7f59000af040 2023-10-16T13:48:00.942 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:48:00.940+0000 7f250e765200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T13:48:00.957 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:00.955+0000 7eff06a04200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T13:48:01.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.026+0000 7f59057e6700 1 -- 172.21.15.16:0/3058802969 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7f58e406b140 msgr2=0x7f58e406d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2221624769 2023-10-16T13:48:01.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.026+0000 7f59057e6700 1 --2- 172.21.15.16:0/3058802969 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7f58e406b140 0x7f58e406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:48:01.061 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:48:01.058+0000 7f250e765200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T13:48:01.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.061+0000 7f590824b700 1 -- 172.21.15.16:0/3058802969 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f590004ebb0 con 0x7f59000af040 2023-10-16T13:48:01.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.061+0000 7f58f67fc700 1 -- 172.21.15.16:0/3058802969 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v250) v1 ==== 93+0+10 (secure 0 0 0) 0x7f58fc059420 con 0x7f59000af040 2023-10-16T13:48:01.064 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T13:48:01.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.063+0000 7f590824b700 1 -- 172.21.15.16:0/3058802969 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7f58e406b140 msgr2=0x7f58e406d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:48:01.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.063+0000 7f590824b700 1 --2- 172.21.15.16:0/3058802969 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7f58e406b140 0x7f58e406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:01.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.064+0000 7f590824b700 1 -- 172.21.15.16:0/3058802969 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59000af040 msgr2=0x7f59001562f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:01.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.064+0000 7f590824b700 1 --2- 172.21.15.16:0/3058802969 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59000af040 0x7f59001562f0 secure :-1 s=READY pgs=2874 cs=0 l=1 rev1=1 crypto rx=0x7f58fc009c40 tx=0x7f58fc004750 comp rx=0 tx=0).stop 2023-10-16T13:48:01.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.064+0000 7f590824b700 1 -- 172.21.15.16:0/3058802969 shutdown_connections 2023-10-16T13:48:01.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.064+0000 7f590824b700 1 --2- 172.21.15.16:0/3058802969 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7f58e406b140 0x7f58e406d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:01.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.064+0000 7f590824b700 1 --2- 172.21.15.16:0/3058802969 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5900160ee0 0x7f5900159490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:01.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.064+0000 7f590824b700 1 --2- 172.21.15.16:0/3058802969 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59000af040 0x7f59001562f0 unknown :-1 s=CLOSED pgs=2874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:01.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.064+0000 7f590824b700 1 --2- 172.21.15.16:0/3058802969 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f59000ce890 0x7f5900158f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:01.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.064+0000 7f590824b700 1 -- 172.21.15.16:0/3058802969 >> 172.21.15.16:0/3058802969 conn(0x7f59000cc5e0 msgr2=0x7f59000ae8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:01.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.064+0000 7f590824b700 1 -- 172.21.15.16:0/3058802969 shutdown_connections 2023-10-16T13:48:01.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.064+0000 7f590824b700 1 -- 172.21.15.16:0/3058802969 wait complete. 2023-10-16T13:48:01.078 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T13:48:01.121 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:01.119+0000 7eff06a04200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T13:48:01.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.206+0000 7f0156219700 1 Processor -- start 2023-10-16T13:48:01.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.206+0000 7f0156219700 1 -- start start 2023-10-16T13:48:01.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.207+0000 7f0156219700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f015015b2c0 0x7f0150161340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:01.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.207+0000 7f0156219700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f015006e570 0x7f015015e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:01.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.207+0000 7f0156219700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0150161880 con 0x7f015006a7e0 2023-10-16T13:48:01.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.207+0000 7f0156219700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f015015b6a0 con 0x7f015006e570 2023-10-16T13:48:01.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.207+0000 7f0156219700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0150161b20 con 0x7f015015b2c0 2023-10-16T13:48:01.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.207+0000 7f014f7fe700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f015015b2c0 0x7f0150161340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:01.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.207+0000 7f014effd700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f015006e570 0x7f015015e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:01.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.207+0000 7f014f7fe700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f015015b2c0 0x7f0150161340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:44934/0 (socket says 172.21.15.16:44934) 2023-10-16T13:48:01.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.207+0000 7f014f7fe700 1 -- 172.21.15.16:0/3418532125 learned_addr learned my addr 172.21.15.16:0/3418532125 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:01.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.207+0000 7f014f7fe700 1 -- 172.21.15.16:0/3418532125 >> v2:172.21.15.49:3300/0 conn(0x7f015006e570 msgr2=0x7f015015e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:01.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.207+0000 7f014f7fe700 1 --2- 172.21.15.16:0/3418532125 >> v2:172.21.15.49:3300/0 conn(0x7f015006e570 0x7f015015e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:01.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.207+0000 7f014f7fe700 1 -- 172.21.15.16:0/3418532125 >> v1:172.21.15.49:6789/0 conn(0x7f015006a7e0 legacy=0x7f0150071a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:01.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.207+0000 7f014f7fe700 1 -- 172.21.15.16:0/3418532125 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0150161e00 con 0x7f015015b2c0 2023-10-16T13:48:01.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.207+0000 7f014f7fe700 1 --2- 172.21.15.16:0/3418532125 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f015015b2c0 0x7f0150161340 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f01400098f0 tx=0x7f0140009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6eca18ecf12e3565 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:01.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.208+0000 7f014e7fc700 1 -- 172.21.15.16:0/3418532125 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0140004230 con 0x7f015015b2c0 2023-10-16T13:48:01.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.208+0000 7f014e7fc700 1 -- 172.21.15.16:0/3418532125 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0140004390 con 0x7f015015b2c0 2023-10-16T13:48:01.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.208+0000 7f014e7fc700 1 -- 172.21.15.16:0/3418532125 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01400036a0 con 0x7f015015b2c0 2023-10-16T13:48:01.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.208+0000 7f0156219700 1 -- 172.21.15.16:0/3418532125 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f015015b2c0 msgr2=0x7f0150161340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:01.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.208+0000 7f0156219700 1 --2- 172.21.15.16:0/3418532125 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f015015b2c0 0x7f0150161340 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f01400098f0 tx=0x7f0140009c00 comp rx=0 tx=0).stop 2023-10-16T13:48:01.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.208+0000 7f0156219700 1 -- 172.21.15.16:0/3418532125 shutdown_connections 2023-10-16T13:48:01.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.208+0000 7f0156219700 1 --2- 172.21.15.16:0/3418532125 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f015015b2c0 0x7f0150161340 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:01.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.208+0000 7f0156219700 1 --2- 172.21.15.16:0/3418532125 >> v2:172.21.15.49:3300/0 conn(0x7f015006e570 0x7f015015e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:01.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.208+0000 7f0156219700 1 -- 172.21.15.16:0/3418532125 >> 172.21.15.16:0/3418532125 conn(0x7f01500b1f40 msgr2=0x7f01500b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:01.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.208+0000 7f0156219700 1 -- 172.21.15.16:0/3418532125 shutdown_connections 2023-10-16T13:48:01.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.208+0000 7f0156219700 1 -- 172.21.15.16:0/3418532125 wait complete. 2023-10-16T13:48:01.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.210+0000 7f0156219700 1 Processor -- start 2023-10-16T13:48:01.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.210+0000 7f0156219700 1 -- start start 2023-10-16T13:48:01.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.211+0000 7f0156219700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f015006a7e0 0x7f01500c9ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:01.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.211+0000 7f0156219700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f015006e570 0x7f0150157740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:01.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.211+0000 7f0156219700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f015015b2c0 0x7f015015f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:01.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.211+0000 7f0156219700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01500b3400 con 0x7f015006e570 2023-10-16T13:48:01.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.211+0000 7f0156219700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01500ca770 con 0x7f015006a7e0 2023-10-16T13:48:01.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.211+0000 7f0156219700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01500ca1b0 con 0x7f015015b2c0 2023-10-16T13:48:01.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.211+0000 7f014effd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f015006e570 0x7f0150157740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:01.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.211+0000 7f014effd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f015006e570 0x7f0150157740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56684/0 (socket says 172.21.15.16:56684) 2023-10-16T13:48:01.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.211+0000 7f014f7fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f015006a7e0 0x7f01500c9ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:01.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.211+0000 7f014effd700 1 -- 172.21.15.16:0/3173724513 learned_addr learned my addr 172.21.15.16:0/3173724513 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:01.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.211+0000 7f014ffff700 1 --2- 172.21.15.16:0/3173724513 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f015015b2c0 0x7f015015f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:01.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.211+0000 7f014f7fe700 1 -- 172.21.15.16:0/3173724513 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f015015b2c0 msgr2=0x7f015015f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:01.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.211+0000 7f014f7fe700 1 --2- 172.21.15.16:0/3173724513 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f015015b2c0 0x7f015015f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:01.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.211+0000 7f014f7fe700 1 -- 172.21.15.16:0/3173724513 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f015006e570 msgr2=0x7f0150157740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:01.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.211+0000 7f014f7fe700 1 --2- 172.21.15.16:0/3173724513 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f015006e570 0x7f0150157740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:01.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.211+0000 7f014f7fe700 1 -- 172.21.15.16:0/3173724513 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0140009550 con 0x7f015006a7e0 2023-10-16T13:48:01.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.211+0000 7f014f7fe700 1 --2- 172.21.15.16:0/3173724513 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f015006a7e0 0x7f01500c9ad0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f0140015000 tx=0x7f0140004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:01.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.211+0000 7f0137fff700 1 -- 172.21.15.16:0/3173724513 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0140003b60 con 0x7f015006a7e0 2023-10-16T13:48:01.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.211+0000 7f0137fff700 1 -- 172.21.15.16:0/3173724513 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f014001f070 con 0x7f015006a7e0 2023-10-16T13:48:01.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.211+0000 7f0137fff700 1 -- 172.21.15.16:0/3173724513 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0140011630 con 0x7f015006a7e0 2023-10-16T13:48:01.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.211+0000 7f0156219700 1 -- 172.21.15.16:0/3173724513 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01500ca490 con 0x7f015006a7e0 2023-10-16T13:48:01.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.212+0000 7f0156219700 1 -- 172.21.15.16:0/3173724513 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01500b1840 con 0x7f015006a7e0 2023-10-16T13:48:01.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.212+0000 7f0156219700 1 -- 172.21.15.16:0/3173724513 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f015004ebb0 con 0x7f015006a7e0 2023-10-16T13:48:01.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.213+0000 7f0137fff700 1 -- 172.21.15.16:0/3173724513 <== mon.1 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 153) v1 ==== 89434+0+0 (secure 0 0 0) 0x7f0140021030 con 0x7f015006a7e0 2023-10-16T13:48:01.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.214+0000 7f0137fff700 1 --2- 172.21.15.16:0/3173724513 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7f013c06b140 0x7f013c06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:01.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.214+0000 7f014effd700 1 -- 172.21.15.16:0/3173724513 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7f013c06b140 msgr2=0x7f013c06d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2221624769 2023-10-16T13:48:01.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.214+0000 7f0137fff700 1 -- 172.21.15.16:0/3173724513 <== mon.1 v2:172.21.15.49:3300/0 5 ==== osd_map(250..250 src has 1..250) v4 ==== 7451+0+0 (secure 0 0 0) 0x7f014008b290 con 0x7f015006a7e0 2023-10-16T13:48:01.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.214+0000 7f014effd700 1 --2- 172.21.15.16:0/3173724513 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7f013c06b140 0x7f013c06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:48:01.224 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.218+0000 7f0137fff700 1 -- 172.21.15.16:0/3173724513 <== mon.1 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0140055a70 con 0x7f015006a7e0 2023-10-16T13:48:01.224 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:48:01.221+0000 7f250e765200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T13:48:01.247 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:01.245+0000 7eff06a04200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T13:48:01.415 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:01.413+0000 7eff06a04200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T13:48:01.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.415+0000 7f014effd700 1 -- 172.21.15.16:0/3173724513 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7f013c06b140 msgr2=0x7f013c06d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2221624769 2023-10-16T13:48:01.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.415+0000 7f014effd700 1 --2- 172.21.15.16:0/3173724513 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7f013c06b140 0x7f013c06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:48:01.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.421+0000 7f0156219700 1 -- 172.21.15.16:0/3173724513 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f01500b6370 con 0x7f015006a7e0 2023-10-16T13:48:01.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.421+0000 7f0137fff700 1 -- 172.21.15.16:0/3173724513 <== mon.1 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v250) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0140059090 con 0x7f015006a7e0 2023-10-16T13:48:01.424 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-10-16T13:48:01.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.423+0000 7f0156219700 1 -- 172.21.15.16:0/3173724513 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7f013c06b140 msgr2=0x7f013c06d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:48:01.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.423+0000 7f0156219700 1 --2- 172.21.15.16:0/3173724513 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7f013c06b140 0x7f013c06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:01.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.423+0000 7f0156219700 1 -- 172.21.15.16:0/3173724513 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f015006a7e0 msgr2=0x7f01500c9ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:01.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.423+0000 7f0156219700 1 --2- 172.21.15.16:0/3173724513 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f015006a7e0 0x7f01500c9ad0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f0140015000 tx=0x7f0140004750 comp rx=0 tx=0).stop 2023-10-16T13:48:01.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.423+0000 7f0156219700 1 -- 172.21.15.16:0/3173724513 shutdown_connections 2023-10-16T13:48:01.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.423+0000 7f0156219700 1 --2- 172.21.15.16:0/3173724513 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7f013c06b140 0x7f013c06d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:01.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.423+0000 7f0156219700 1 --2- 172.21.15.16:0/3173724513 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f015015b2c0 0x7f015015f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:01.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.423+0000 7f0156219700 1 --2- 172.21.15.16:0/3173724513 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f015006a7e0 0x7f01500c9ad0 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:01.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.423+0000 7f0156219700 1 --2- 172.21.15.16:0/3173724513 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f015006e570 0x7f0150157740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:01.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.423+0000 7f0156219700 1 -- 172.21.15.16:0/3173724513 >> 172.21.15.16:0/3173724513 conn(0x7f01500b1f40 msgr2=0x7f015015b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:01.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.424+0000 7f0156219700 1 -- 172.21.15.16:0/3173724513 shutdown_connections 2023-10-16T13:48:01.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.424+0000 7f0156219700 1 -- 172.21.15.16:0/3173724513 wait complete. 2023-10-16T13:48:01.439 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T13:48:01.497 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:48:01.494+0000 7f250e765200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T13:48:01.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.611+0000 7fe0a9ef6700 1 Processor -- start 2023-10-16T13:48:01.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.611+0000 7fe0a9ef6700 1 -- start start 2023-10-16T13:48:01.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.612+0000 7fe0a9ef6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe0a40d2110 0x7fe0a40ae1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:01.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.612+0000 7fe0a9ef6700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe0a406ca10 0x7fe0a40ae720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:01.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.612+0000 7fe0a9ef6700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0a40b20f0 con 0x7fe0a40ce6a0 2023-10-16T13:48:01.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.612+0000 7fe0a9ef6700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0a40d03d0 con 0x7fe0a40d2110 2023-10-16T13:48:01.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.612+0000 7fe0a9ef6700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0a40b2340 con 0x7fe0a406ca10 2023-10-16T13:48:01.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.612+0000 7fe0a2ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe0a40d2110 0x7fe0a40ae1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:01.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.612+0000 7fe0a2ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe0a40d2110 0x7fe0a40ae1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56694/0 (socket says 172.21.15.16:56694) 2023-10-16T13:48:01.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.612+0000 7fe0a2ffd700 1 -- 172.21.15.16:0/243521103 learned_addr learned my addr 172.21.15.16:0/243521103 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:01.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.612+0000 7fe0a3fff700 1 --2- 172.21.15.16:0/243521103 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe0a406ca10 0x7fe0a40ae720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:01.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.612+0000 7fe0a2ffd700 1 -- 172.21.15.16:0/243521103 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe0a406ca10 msgr2=0x7fe0a40ae720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:01.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.612+0000 7fe0a2ffd700 1 --2- 172.21.15.16:0/243521103 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe0a406ca10 0x7fe0a40ae720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:01.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.613+0000 7fe0a2ffd700 1 -- 172.21.15.16:0/243521103 >> v1:172.21.15.49:6789/0 conn(0x7fe0a40ce6a0 legacy=0x7fe0a406c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:01.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.613+0000 7fe0a2ffd700 1 -- 172.21.15.16:0/243521103 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0a406bb00 con 0x7fe0a40d2110 2023-10-16T13:48:01.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.613+0000 7fe0a27fc700 1 -- 172.21.15.16:0/243521103 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2336727866 0 0) 0x7fe0a40b20f0 con 0x7fe0a40ce6a0 2023-10-16T13:48:01.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.613+0000 7fe0a2ffd700 1 --2- 172.21.15.16:0/243521103 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0a40d2110 0x7fe0a40ae1e0 secure :-1 s=READY pgs=2875 cs=0 l=1 rev1=1 crypto rx=0x7fe0940098f0 tx=0x7fe094009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7730466527fc0c58 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:01.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.613+0000 7fe0a27fc700 1 -- 172.21.15.16:0/243521103 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe094011830 con 0x7fe0a40d2110 2023-10-16T13:48:01.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.613+0000 7fe0a27fc700 1 -- 172.21.15.16:0/243521103 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe094011990 con 0x7fe0a40d2110 2023-10-16T13:48:01.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.613+0000 7fe0a27fc700 1 -- 172.21.15.16:0/243521103 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe094011d20 con 0x7fe0a40d2110 2023-10-16T13:48:01.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.613+0000 7fe0a9ef6700 1 -- 172.21.15.16:0/243521103 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0a40d2110 msgr2=0x7fe0a40ae1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:01.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.613+0000 7fe0a9ef6700 1 --2- 172.21.15.16:0/243521103 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0a40d2110 0x7fe0a40ae1e0 secure :-1 s=READY pgs=2875 cs=0 l=1 rev1=1 crypto rx=0x7fe0940098f0 tx=0x7fe094009c00 comp rx=0 tx=0).stop 2023-10-16T13:48:01.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.613+0000 7fe0a9ef6700 1 -- 172.21.15.16:0/243521103 shutdown_connections 2023-10-16T13:48:01.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.613+0000 7fe0a9ef6700 1 --2- 172.21.15.16:0/243521103 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe0a406ca10 0x7fe0a40ae720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:01.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.613+0000 7fe0a9ef6700 1 --2- 172.21.15.16:0/243521103 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0a40d2110 0x7fe0a40ae1e0 unknown :-1 s=CLOSED pgs=2875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:01.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.614+0000 7fe0a9ef6700 1 -- 172.21.15.16:0/243521103 >> 172.21.15.16:0/243521103 conn(0x7fe0a40d15f0 msgr2=0x7fe0a40d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:01.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.614+0000 7fe0a9ef6700 1 -- 172.21.15.16:0/243521103 shutdown_connections 2023-10-16T13:48:01.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.614+0000 7fe0a9ef6700 1 -- 172.21.15.16:0/243521103 wait complete. 2023-10-16T13:48:01.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.615+0000 7fe0a9ef6700 1 Processor -- start 2023-10-16T13:48:01.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.615+0000 7fe0a9ef6700 1 -- start start 2023-10-16T13:48:01.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.616+0000 7fe0a9ef6700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe0a406ca10 0x7fe0a4151e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:01.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.616+0000 7fe0a9ef6700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe0a40ce6a0 0x7fe0a40ccda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:01.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.616+0000 7fe0a9ef6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0a40d2110 0x7fe0a40cd2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:01.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.616+0000 7fe0a9ef6700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0a40b5cc0 con 0x7fe0a40d2110 2023-10-16T13:48:01.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.616+0000 7fe0a9ef6700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0a41f3d80 con 0x7fe0a40ce6a0 2023-10-16T13:48:01.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.616+0000 7fe0a9ef6700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0a40b5360 con 0x7fe0a406ca10 2023-10-16T13:48:01.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.616+0000 7fe0a3fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0a40d2110 0x7fe0a40cd2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:01.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.616+0000 7fe0a3fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0a40d2110 0x7fe0a40cd2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56702/0 (socket says 172.21.15.16:56702) 2023-10-16T13:48:01.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.616+0000 7fe0a3fff700 1 -- 172.21.15.16:0/1640907455 learned_addr learned my addr 172.21.15.16:0/1640907455 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:01.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.616+0000 7fe0a2ffd700 1 --2- 172.21.15.16:0/1640907455 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe0a40ce6a0 0x7fe0a40ccda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:01.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.616+0000 7fe0a37fe700 1 --2- 172.21.15.16:0/1640907455 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe0a406ca10 0x7fe0a4151e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:01.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.616+0000 7fe0a3fff700 1 -- 172.21.15.16:0/1640907455 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe0a406ca10 msgr2=0x7fe0a4151e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:01.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.616+0000 7fe0a3fff700 1 --2- 172.21.15.16:0/1640907455 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe0a406ca10 0x7fe0a4151e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:01.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.616+0000 7fe0a3fff700 1 -- 172.21.15.16:0/1640907455 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe0a40ce6a0 msgr2=0x7fe0a40ccda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:01.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.616+0000 7fe0a3fff700 1 --2- 172.21.15.16:0/1640907455 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe0a40ce6a0 0x7fe0a40ccda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:01.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.616+0000 7fe0a3fff700 1 -- 172.21.15.16:0/1640907455 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe094009550 con 0x7fe0a40d2110 2023-10-16T13:48:01.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.616+0000 7fe0a3fff700 1 --2- 172.21.15.16:0/1640907455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0a40d2110 0x7fe0a40cd2e0 secure :-1 s=READY pgs=2876 cs=0 l=1 rev1=1 crypto rx=0x7fe09c00efc0 tx=0x7fe09c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:01.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.616+0000 7fe08bfff700 1 -- 172.21.15.16:0/1640907455 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe09c009ea0 con 0x7fe0a40d2110 2023-10-16T13:48:01.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.616+0000 7fe08bfff700 1 -- 172.21.15.16:0/1640907455 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe09c00f040 con 0x7fe0a40d2110 2023-10-16T13:48:01.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.616+0000 7fe0a9ef6700 1 -- 172.21.15.16:0/1640907455 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0a4152340 con 0x7fe0a40d2110 2023-10-16T13:48:01.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.616+0000 7fe08bfff700 1 -- 172.21.15.16:0/1640907455 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe09c00ced0 con 0x7fe0a40d2110 2023-10-16T13:48:01.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.616+0000 7fe0a9ef6700 1 -- 172.21.15.16:0/1640907455 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0a40b5600 con 0x7fe0a40d2110 2023-10-16T13:48:01.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.617+0000 7fe0a9ef6700 1 -- 172.21.15.16:0/1640907455 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0a40b5e00 con 0x7fe0a40d2110 2023-10-16T13:48:01.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.618+0000 7fe08bfff700 1 -- 172.21.15.16:0/1640907455 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 153) v1 ==== 89434+0+0 (secure 0 0 0) 0x7fe09c014480 con 0x7fe0a40d2110 2023-10-16T13:48:01.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.618+0000 7fe08bfff700 1 --2- 172.21.15.16:0/1640907455 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7fe09006b140 0x7fe09006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:01.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.619+0000 7fe0a37fe700 1 -- 172.21.15.16:0/1640907455 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7fe09006b140 msgr2=0x7fe09006d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2221624769 2023-10-16T13:48:01.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.619+0000 7fe08bfff700 1 -- 172.21.15.16:0/1640907455 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(250..250 src has 1..250) v4 ==== 7451+0+0 (secure 0 0 0) 0x7fe09c08a920 con 0x7fe0a40d2110 2023-10-16T13:48:01.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.619+0000 7fe0a37fe700 1 --2- 172.21.15.16:0/1640907455 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7fe09006b140 0x7fe09006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:48:01.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.623+0000 7fe08bfff700 1 -- 172.21.15.16:0/1640907455 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe0a40b5e00 con 0x7fe0a40d2110 2023-10-16T13:48:01.665 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:48:01.663+0000 7f250e765200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T13:48:01.700 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:01.698+0000 7eff06a04200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T13:48:01.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.820+0000 7fe0a37fe700 1 -- 172.21.15.16:0/1640907455 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7fe09006b140 msgr2=0x7fe09006d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2221624769 2023-10-16T13:48:01.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.820+0000 7fe0a37fe700 1 --2- 172.21.15.16:0/1640907455 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7fe09006b140 0x7fe09006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:48:01.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.826+0000 7fe0a9ef6700 1 -- 172.21.15.16:0/1640907455 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe0a40b5e00 con 0x7fe0a40d2110 2023-10-16T13:48:01.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.827+0000 7fe08bfff700 1 -- 172.21.15.16:0/1640907455 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v250) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe0a40b5e00 con 0x7fe0a40d2110 2023-10-16T13:48:01.829 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-10-16T13:48:01.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.829+0000 7fe0a9ef6700 1 -- 172.21.15.16:0/1640907455 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7fe09006b140 msgr2=0x7fe09006d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:48:01.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.829+0000 7fe0a9ef6700 1 --2- 172.21.15.16:0/1640907455 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7fe09006b140 0x7fe09006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:01.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.829+0000 7fe0a9ef6700 1 -- 172.21.15.16:0/1640907455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0a40d2110 msgr2=0x7fe0a40cd2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:01.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.829+0000 7fe0a9ef6700 1 --2- 172.21.15.16:0/1640907455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0a40d2110 0x7fe0a40cd2e0 secure :-1 s=READY pgs=2876 cs=0 l=1 rev1=1 crypto rx=0x7fe09c00efc0 tx=0x7fe09c00c3a0 comp rx=0 tx=0).stop 2023-10-16T13:48:01.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.829+0000 7fe0a9ef6700 1 -- 172.21.15.16:0/1640907455 shutdown_connections 2023-10-16T13:48:01.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.829+0000 7fe0a9ef6700 1 --2- 172.21.15.16:0/1640907455 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7fe09006b140 0x7fe09006d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:01.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.829+0000 7fe0a9ef6700 1 --2- 172.21.15.16:0/1640907455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0a40d2110 0x7fe0a40cd2e0 unknown :-1 s=CLOSED pgs=2876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:01.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.829+0000 7fe0a9ef6700 1 --2- 172.21.15.16:0/1640907455 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe0a406ca10 0x7fe0a4151e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:01.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.829+0000 7fe0a9ef6700 1 --2- 172.21.15.16:0/1640907455 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe0a40ce6a0 0x7fe0a40ccda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:01.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.829+0000 7fe0a9ef6700 1 -- 172.21.15.16:0/1640907455 >> 172.21.15.16:0/1640907455 conn(0x7fe0a40d15f0 msgr2=0x7fe0a415f5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:01.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.829+0000 7fe0a9ef6700 1 -- 172.21.15.16:0/1640907455 shutdown_connections 2023-10-16T13:48:01.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.829+0000 7fe0a9ef6700 1 -- 172.21.15.16:0/1640907455 wait complete. 2023-10-16T13:48:01.844 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2023-10-16T13:48:01.879 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:01.877+0000 7eff06a04200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T13:48:01.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.973+0000 7f0bc94a1700 1 Processor -- start 2023-10-16T13:48:01.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.973+0000 7f0bc94a1700 1 -- start start 2023-10-16T13:48:01.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.974+0000 7f0bc94a1700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0bc415b2c0 0x7f0bc4161340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:01.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.974+0000 7f0bc94a1700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0bc406e570 0x7f0bc415e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:01.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.974+0000 7f0bc94a1700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bc4161880 con 0x7f0bc406a7e0 2023-10-16T13:48:01.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.974+0000 7f0bc94a1700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bc415b6a0 con 0x7f0bc415b2c0 2023-10-16T13:48:01.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.974+0000 7f0bc94a1700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc4161b20 con 0x7f0bc406e570 2023-10-16T13:48:01.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.974+0000 7f0bc2ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0bc415b2c0 0x7f0bc4161340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:01.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.974+0000 7f0bc2ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0bc415b2c0 0x7f0bc4161340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56708/0 (socket says 172.21.15.16:56708) 2023-10-16T13:48:01.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.974+0000 7f0bc2ffd700 1 -- 172.21.15.16:0/3795012574 learned_addr learned my addr 172.21.15.16:0/3795012574 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:01.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.974+0000 7f0bc27fc700 1 --2- 172.21.15.16:0/3795012574 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0bc406e570 0x7f0bc415e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:01.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.975+0000 7f0bc2ffd700 1 -- 172.21.15.16:0/3795012574 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0bc406e570 msgr2=0x7f0bc415e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:01.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.975+0000 7f0bc2ffd700 1 --2- 172.21.15.16:0/3795012574 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0bc406e570 0x7f0bc415e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:01.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.975+0000 7f0bc2ffd700 1 -- 172.21.15.16:0/3795012574 >> v1:172.21.15.49:6789/0 conn(0x7f0bc406a7e0 legacy=0x7f0bc4071a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:01.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.975+0000 7f0bc2ffd700 1 -- 172.21.15.16:0/3795012574 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bc4161e00 con 0x7f0bc415b2c0 2023-10-16T13:48:01.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.975+0000 7f0bc1ffb700 1 -- 172.21.15.16:0/3795012574 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4056060012 0 0) 0x7f0bc4161880 con 0x7f0bc406a7e0 2023-10-16T13:48:01.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.975+0000 7f0bc2ffd700 1 --2- 172.21.15.16:0/3795012574 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bc415b2c0 0x7f0bc4161340 secure :-1 s=READY pgs=2877 cs=0 l=1 rev1=1 crypto rx=0x7f0bb40098f0 tx=0x7f0bb4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5d5333b28db0968 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:01.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.975+0000 7f0bc1ffb700 1 -- 172.21.15.16:0/3795012574 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb4011830 con 0x7f0bc415b2c0 2023-10-16T13:48:01.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.975+0000 7f0bc1ffb700 1 -- 172.21.15.16:0/3795012574 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bb4011990 con 0x7f0bc415b2c0 2023-10-16T13:48:01.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.975+0000 7f0bc1ffb700 1 -- 172.21.15.16:0/3795012574 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb4011d20 con 0x7f0bc415b2c0 2023-10-16T13:48:01.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.976+0000 7f0bc94a1700 1 -- 172.21.15.16:0/3795012574 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bc415b2c0 msgr2=0x7f0bc4161340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:01.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.976+0000 7f0bc94a1700 1 --2- 172.21.15.16:0/3795012574 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bc415b2c0 0x7f0bc4161340 secure :-1 s=READY pgs=2877 cs=0 l=1 rev1=1 crypto rx=0x7f0bb40098f0 tx=0x7f0bb4009c00 comp rx=0 tx=0).stop 2023-10-16T13:48:01.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.976+0000 7f0bc94a1700 1 -- 172.21.15.16:0/3795012574 shutdown_connections 2023-10-16T13:48:01.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.976+0000 7f0bc94a1700 1 --2- 172.21.15.16:0/3795012574 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bc415b2c0 0x7f0bc4161340 unknown :-1 s=CLOSED pgs=2877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:01.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.976+0000 7f0bc94a1700 1 --2- 172.21.15.16:0/3795012574 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0bc406e570 0x7f0bc415e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:01.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.976+0000 7f0bc94a1700 1 -- 172.21.15.16:0/3795012574 >> 172.21.15.16:0/3795012574 conn(0x7f0bc40b1f40 msgr2=0x7f0bc40b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:01.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.976+0000 7f0bc94a1700 1 -- 172.21.15.16:0/3795012574 shutdown_connections 2023-10-16T13:48:01.983 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.976+0000 7f0bc94a1700 1 -- 172.21.15.16:0/3795012574 wait complete. 2023-10-16T13:48:01.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.977+0000 7f0bc94a1700 1 Processor -- start 2023-10-16T13:48:01.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.977+0000 7f0bc94a1700 1 -- start start 2023-10-16T13:48:01.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.978+0000 7f0bc94a1700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0bc406a7e0 0x7f0bc414f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:01.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.978+0000 7f0bc94a1700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0bc406e570 0x7f0bc40b51a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:01.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.978+0000 7f0bc94a1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bc415b2c0 0x7f0bc4160ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:01.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.978+0000 7f0bc94a1700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc414fec0 con 0x7f0bc415b2c0 2023-10-16T13:48:01.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.978+0000 7f0bc94a1700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc40c80c0 con 0x7f0bc406a7e0 2023-10-16T13:48:01.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.978+0000 7f0bc94a1700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc40b4100 con 0x7f0bc406e570 2023-10-16T13:48:01.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.978+0000 7f0bc37fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bc415b2c0 0x7f0bc4160ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:01.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.978+0000 7f0bc37fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bc415b2c0 0x7f0bc4160ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56718/0 (socket says 172.21.15.16:56718) 2023-10-16T13:48:01.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.978+0000 7f0bc37fe700 1 -- 172.21.15.16:0/3102376022 learned_addr learned my addr 172.21.15.16:0/3102376022 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:01.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.978+0000 7f0bc27fc700 1 --2- 172.21.15.16:0/3102376022 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0bc406e570 0x7f0bc40b51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:01.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.978+0000 7f0bc2ffd700 1 --2- 172.21.15.16:0/3102376022 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0bc406a7e0 0x7f0bc414f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:01.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.979+0000 7f0bc37fe700 1 -- 172.21.15.16:0/3102376022 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0bc406e570 msgr2=0x7f0bc40b51a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:01.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.979+0000 7f0bc37fe700 1 --2- 172.21.15.16:0/3102376022 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0bc406e570 0x7f0bc40b51a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:01.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.979+0000 7f0bc37fe700 1 -- 172.21.15.16:0/3102376022 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0bc406a7e0 msgr2=0x7f0bc414f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:01.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.979+0000 7f0bc37fe700 1 --2- 172.21.15.16:0/3102376022 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0bc406a7e0 0x7f0bc414f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:01.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.979+0000 7f0bc37fe700 1 -- 172.21.15.16:0/3102376022 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bb4009550 con 0x7f0bc415b2c0 2023-10-16T13:48:01.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.979+0000 7f0bc37fe700 1 --2- 172.21.15.16:0/3102376022 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bc415b2c0 0x7f0bc4160ec0 secure :-1 s=READY pgs=2878 cs=0 l=1 rev1=1 crypto rx=0x7f0bbc00c7a0 tx=0x7f0bbc00cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:01.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.979+0000 7f0bab7fe700 1 -- 172.21.15.16:0/3102376022 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bbc003e30 con 0x7f0bc415b2c0 2023-10-16T13:48:01.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.979+0000 7f0bab7fe700 1 -- 172.21.15.16:0/3102376022 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bbc005070 con 0x7f0bc415b2c0 2023-10-16T13:48:01.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.979+0000 7f0bc94a1700 1 -- 172.21.15.16:0/3102376022 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bc40b43e0 con 0x7f0bc415b2c0 2023-10-16T13:48:01.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.979+0000 7f0bab7fe700 1 -- 172.21.15.16:0/3102376022 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bbc005420 con 0x7f0bc415b2c0 2023-10-16T13:48:01.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.979+0000 7f0bc94a1700 1 -- 172.21.15.16:0/3102376022 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bc40b49a0 con 0x7f0bc415b2c0 2023-10-16T13:48:01.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.980+0000 7f0bc94a1700 1 -- 172.21.15.16:0/3102376022 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bc404ebb0 con 0x7f0bc415b2c0 2023-10-16T13:48:01.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.980+0000 7f0bab7fe700 1 -- 172.21.15.16:0/3102376022 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 153) v1 ==== 89434+0+0 (secure 0 0 0) 0x7f0bbc005580 con 0x7f0bc415b2c0 2023-10-16T13:48:01.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.981+0000 7f0bab7fe700 1 --2- 172.21.15.16:0/3102376022 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7f0bb006b140 0x7f0bb006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:01.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.981+0000 7f0bc2ffd700 1 -- 172.21.15.16:0/3102376022 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7f0bb006b140 msgr2=0x7f0bb006d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2221624769 2023-10-16T13:48:01.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.981+0000 7f0bab7fe700 1 -- 172.21.15.16:0/3102376022 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(250..250 src has 1..250) v4 ==== 7451+0+0 (secure 0 0 0) 0x7f0bbc0895f0 con 0x7f0bc415b2c0 2023-10-16T13:48:01.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.981+0000 7f0bc2ffd700 1 --2- 172.21.15.16:0/3102376022 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7f0bb006b140 0x7f0bb006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:48:01.991 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:01.986+0000 7f0bab7fe700 1 -- 172.21.15.16:0/3102376022 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0bbc057790 con 0x7f0bc415b2c0 2023-10-16T13:48:02.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.182+0000 7f0bc2ffd700 1 -- 172.21.15.16:0/3102376022 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7f0bb006b140 msgr2=0x7f0bb006d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2221624769 2023-10-16T13:48:02.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.182+0000 7f0bc2ffd700 1 --2- 172.21.15.16:0/3102376022 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7f0bb006b140 0x7f0bb006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:48:02.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.204+0000 7f0bc94a1700 1 -- 172.21.15.16:0/3102376022 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0bc40b6370 con 0x7f0bc415b2c0 2023-10-16T13:48:02.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.205+0000 7f0bab7fe700 1 -- 172.21.15.16:0/3102376022 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v250) v1 ==== 107+0+11 (secure 0 0 0) 0x7f0bbc0170b0 con 0x7f0bc415b2c0 2023-10-16T13:48:02.207 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-10-16T13:48:02.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.207+0000 7f0bc94a1700 1 -- 172.21.15.16:0/3102376022 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7f0bb006b140 msgr2=0x7f0bb006d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:48:02.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.207+0000 7f0bc94a1700 1 --2- 172.21.15.16:0/3102376022 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7f0bb006b140 0x7f0bb006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:02.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.207+0000 7f0bc94a1700 1 -- 172.21.15.16:0/3102376022 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bc415b2c0 msgr2=0x7f0bc4160ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:02.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.207+0000 7f0bc94a1700 1 --2- 172.21.15.16:0/3102376022 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bc415b2c0 0x7f0bc4160ec0 secure :-1 s=READY pgs=2878 cs=0 l=1 rev1=1 crypto rx=0x7f0bbc00c7a0 tx=0x7f0bbc00cab0 comp rx=0 tx=0).stop 2023-10-16T13:48:02.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.207+0000 7f0bc94a1700 1 -- 172.21.15.16:0/3102376022 shutdown_connections 2023-10-16T13:48:02.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.207+0000 7f0bc94a1700 1 --2- 172.21.15.16:0/3102376022 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7f0bb006b140 0x7f0bb006d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:02.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.207+0000 7f0bc94a1700 1 --2- 172.21.15.16:0/3102376022 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bc415b2c0 0x7f0bc4160ec0 unknown :-1 s=CLOSED pgs=2878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:02.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.207+0000 7f0bc94a1700 1 --2- 172.21.15.16:0/3102376022 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0bc406a7e0 0x7f0bc414f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:02.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.207+0000 7f0bc94a1700 1 --2- 172.21.15.16:0/3102376022 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0bc406e570 0x7f0bc40b51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:02.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.207+0000 7f0bc94a1700 1 -- 172.21.15.16:0/3102376022 >> 172.21.15.16:0/3102376022 conn(0x7f0bc40b1f40 msgr2=0x7f0bc415c060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:02.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.207+0000 7f0bc94a1700 1 -- 172.21.15.16:0/3102376022 shutdown_connections 2023-10-16T13:48:02.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.207+0000 7f0bc94a1700 1 -- 172.21.15.16:0/3102376022 wait complete. 2023-10-16T13:48:02.220 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:48:02.218+0000 7f250e765200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T13:48:02.222 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2023-10-16T13:48:02.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.391+0000 7f2af55e0700 1 Processor -- start 2023-10-16T13:48:02.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.391+0000 7f2af55e0700 1 -- start start 2023-10-16T13:48:02.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.391+0000 7f2af55e0700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f2af015afc0 0x7f2af0071ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:02.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.392+0000 7f2af55e0700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2af0072010 con 0x7f2af015b850 2023-10-16T13:48:02.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.392+0000 7f2af55e0700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2af015e500 con 0x7f2af015b3a0 2023-10-16T13:48:02.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.392+0000 7f2af55e0700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2af015f080 con 0x7f2af015afc0 2023-10-16T13:48:02.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.392+0000 7f2aef7fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2af015b850 0x7f2af00b24c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46002/0 (socket says 172.21.15.16:46002) 2023-10-16T13:48:02.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.392+0000 7f2aef7fe700 1 -- 172.21.15.16:0/3835622228 learned_addr learned my addr 172.21.15.16:0/3835622228 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:48:02.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.392+0000 7f2aeeffd700 1 --2- 172.21.15.16:0/3835622228 >> v2:172.21.15.49:3300/0 conn(0x7f2af015afc0 0x7f2af0071ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:02.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.392+0000 7f2aedffb700 1 -- 172.21.15.16:0/3835622228 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1302413219 0 0) 0x7f2af0072010 con 0x7f2af015b850 2023-10-16T13:48:02.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.392+0000 7f2aedffb700 1 -- 172.21.15.16:0/3835622228 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ad80035b0 con 0x7f2af015b850 2023-10-16T13:48:02.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.392+0000 7f2aeeffd700 1 -- 172.21.15.16:0/3835622228 >> v1:172.21.15.49:6789/0 conn(0x7f2af015b3a0 legacy=0x7f2af00b0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:02.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.392+0000 7f2aeeffd700 1 -- 172.21.15.16:0/3835622228 >> v1:172.21.15.16:6789/0 conn(0x7f2af015b850 legacy=0x7f2af00b24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:02.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.392+0000 7f2aedffb700 1 -- 172.21.15.16:0/3835622228 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1490023060 0 0) 0x7f2ad80035b0 con 0x7f2af015b850 2023-10-16T13:48:02.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.392+0000 7f2aeeffd700 1 -- 172.21.15.16:0/3835622228 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2af015e7e0 con 0x7f2af015afc0 2023-10-16T13:48:02.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.392+0000 7f2aeeffd700 1 --2- 172.21.15.16:0/3835622228 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2af015afc0 0x7f2af0071ad0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f2ae0009840 tx=0x7f2ae0009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=37ef310f9143b27 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:02.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.393+0000 7f2aedffb700 1 -- 172.21.15.16:0/3835622228 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ae0011830 con 0x7f2af015afc0 2023-10-16T13:48:02.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.393+0000 7f2aedffb700 1 -- 172.21.15.16:0/3835622228 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ae0011990 con 0x7f2af015afc0 2023-10-16T13:48:02.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.393+0000 7f2aedffb700 1 -- 172.21.15.16:0/3835622228 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ae0011d20 con 0x7f2af015afc0 2023-10-16T13:48:02.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.393+0000 7f2af55e0700 1 -- 172.21.15.16:0/3835622228 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2af015afc0 msgr2=0x7f2af0071ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:02.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.393+0000 7f2af55e0700 1 --2- 172.21.15.16:0/3835622228 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2af015afc0 0x7f2af0071ad0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f2ae0009840 tx=0x7f2ae0009b50 comp rx=0 tx=0).stop 2023-10-16T13:48:02.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.393+0000 7f2af55e0700 1 -- 172.21.15.16:0/3835622228 shutdown_connections 2023-10-16T13:48:02.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.393+0000 7f2af55e0700 1 --2- 172.21.15.16:0/3835622228 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2af015afc0 0x7f2af0071ad0 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:02.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.393+0000 7f2af55e0700 1 -- 172.21.15.16:0/3835622228 >> 172.21.15.16:0/3835622228 conn(0x7f2af006a7e0 msgr2=0x7f2af006abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:02.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.393+0000 7f2af55e0700 1 -- 172.21.15.16:0/3835622228 shutdown_connections 2023-10-16T13:48:02.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.393+0000 7f2af55e0700 1 -- 172.21.15.16:0/3835622228 wait complete. 2023-10-16T13:48:02.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.395+0000 7f2af55e0700 1 Processor -- start 2023-10-16T13:48:02.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.395+0000 7f2af55e0700 1 -- start start 2023-10-16T13:48:02.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.395+0000 7f2af55e0700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2af015afc0 0x7f2af00ac8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:02.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.395+0000 7f2af55e0700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2af015b3a0 0x7f2af00cd9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:02.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.395+0000 7f2af55e0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2af015b850 0x7f2af01f6550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:02.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.395+0000 7f2af55e0700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2af0152110 con 0x7f2af015b850 2023-10-16T13:48:02.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.395+0000 7f2af55e0700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2af014b830 con 0x7f2af015afc0 2023-10-16T13:48:02.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.395+0000 7f2af55e0700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2af0151b50 con 0x7f2af015b3a0 2023-10-16T13:48:02.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.396+0000 7f2aef7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2af015b850 0x7f2af01f6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:02.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.396+0000 7f2aef7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2af015b850 0x7f2af01f6550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56734/0 (socket says 172.21.15.16:56734) 2023-10-16T13:48:02.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.396+0000 7f2aef7fe700 1 -- 172.21.15.16:0/2972009803 learned_addr learned my addr 172.21.15.16:0/2972009803 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:02.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.396+0000 7f2aee7fc700 1 --2- 172.21.15.16:0/2972009803 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2af015b3a0 0x7f2af00cd9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:02.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.396+0000 7f2aeeffd700 1 --2- 172.21.15.16:0/2972009803 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2af015afc0 0x7f2af00ac8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:02.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.396+0000 7f2aef7fe700 1 -- 172.21.15.16:0/2972009803 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2af015b3a0 msgr2=0x7f2af00cd9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:02.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.396+0000 7f2aef7fe700 1 --2- 172.21.15.16:0/2972009803 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2af015b3a0 0x7f2af00cd9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:02.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.396+0000 7f2aef7fe700 1 -- 172.21.15.16:0/2972009803 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2af015afc0 msgr2=0x7f2af00ac8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:02.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.396+0000 7f2aef7fe700 1 --2- 172.21.15.16:0/2972009803 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2af015afc0 0x7f2af00ac8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:02.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.396+0000 7f2aef7fe700 1 -- 172.21.15.16:0/2972009803 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ae00094a0 con 0x7f2af015b850 2023-10-16T13:48:02.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.396+0000 7f2aef7fe700 1 --2- 172.21.15.16:0/2972009803 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2af015b850 0x7f2af01f6550 secure :-1 s=READY pgs=2880 cs=0 l=1 rev1=1 crypto rx=0x7f2ae800cad0 tx=0x7f2ae800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:02.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.396+0000 7f2ad77fe700 1 -- 172.21.15.16:0/2972009803 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ae800a9e0 con 0x7f2af015b850 2023-10-16T13:48:02.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.396+0000 7f2ad77fe700 1 -- 172.21.15.16:0/2972009803 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ae800ab40 con 0x7f2af015b850 2023-10-16T13:48:02.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.396+0000 7f2ad77fe700 1 -- 172.21.15.16:0/2972009803 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ae8012400 con 0x7f2af015b850 2023-10-16T13:48:02.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.396+0000 7f2af55e0700 1 -- 172.21.15.16:0/2972009803 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2af0151e30 con 0x7f2af015b850 2023-10-16T13:48:02.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.396+0000 7f2af55e0700 1 -- 172.21.15.16:0/2972009803 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2af0159f60 con 0x7f2af015b850 2023-10-16T13:48:02.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.397+0000 7f2af55e0700 1 -- 172.21.15.16:0/2972009803 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2af0159c80 con 0x7f2af015b850 2023-10-16T13:48:02.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.398+0000 7f2ad77fe700 1 -- 172.21.15.16:0/2972009803 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 153) v1 ==== 89434+0+0 (secure 0 0 0) 0x7f2ae8012560 con 0x7f2af015b850 2023-10-16T13:48:02.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.398+0000 7f2ad77fe700 1 --2- 172.21.15.16:0/2972009803 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7f2adc06b140 0x7f2adc06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:02.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.398+0000 7f2aeeffd700 1 -- 172.21.15.16:0/2972009803 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7f2adc06b140 msgr2=0x7f2adc06d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2221624769 2023-10-16T13:48:02.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.398+0000 7f2ad77fe700 1 -- 172.21.15.16:0/2972009803 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(250..250 src has 1..250) v4 ==== 7451+0+0 (secure 0 0 0) 0x7f2ae808a2c0 con 0x7f2af015b850 2023-10-16T13:48:02.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.398+0000 7f2aeeffd700 1 --2- 172.21.15.16:0/2972009803 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7f2adc06b140 0x7f2adc06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:48:02.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.403+0000 7f2ad77fe700 1 -- 172.21.15.16:0/2972009803 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2af0159c80 con 0x7f2af015b850 2023-10-16T13:48:02.474 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:02.472+0000 7eff06a04200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T13:48:02.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.495+0000 7f2ad77fe700 1 -- 172.21.15.16:0/2972009803 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 154) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f2ae8054d10 con 0x7f2af015b850 2023-10-16T13:48:02.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.495+0000 7f2ad77fe700 1 -- 172.21.15.16:0/2972009803 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7f2adc06b140 msgr2=0x7f2adc06d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:48:02.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.495+0000 7f2ad77fe700 1 --2- 172.21.15.16:0/2972009803 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7f2adc06b140 0x7f2adc06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:02.566 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:02.564+0000 7efe821f0700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:48:02.566 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:02.564+0000 7efe821f0700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:48:02.567 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:02.564+0000 7efe821f0700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:48:02.567 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:02.564+0000 7efe821f0700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:48:02.567 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:02.564+0000 7efe821f0700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:48:02.567 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:02.564+0000 7efe7f1ea700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:48:02.567 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:02.564+0000 7efe7f1ea700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:48:02.567 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:02.564+0000 7efe7f1ea700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:48:02.568 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:02.564+0000 7efe7f1ea700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:48:02.568 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:02.564+0000 7efe7f1ea700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:48:02.576 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:02.574+0000 7efe7f1ea700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:48:02.576 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:02.574+0000 7efe7f1ea700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:48:02.576 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:02.574+0000 7efe7f1ea700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:48:02.577 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:02.574+0000 7efe7f1ea700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:48:02.577 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:02.574+0000 7efe7f1ea700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:48:02.577 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:02.574+0000 7efe821f0700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:48:02.578 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:02.574+0000 7efe821f0700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:48:02.578 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:02.574+0000 7efe821f0700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:48:02.578 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:02.574+0000 7efe821f0700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:48:02.578 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:02.574+0000 7efe821f0700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:48:02.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.625+0000 7f2af55e0700 1 -- 172.21.15.16:0/2972009803 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) v1 -- 0x7f2af0159c80 con 0x7f2af015b850 2023-10-16T13:48:02.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.625+0000 7f2ad77fe700 1 -- 172.21.15.16:0/2972009803 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v251) v1 ==== 103+0+11 (secure 0 0 0) 0x7f2af0159c80 con 0x7f2af015b850 2023-10-16T13:48:02.628 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-10-16T13:48:02.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.627+0000 7f2af55e0700 1 -- 172.21.15.16:0/2972009803 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2af015b850 msgr2=0x7f2af01f6550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:02.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.627+0000 7f2af55e0700 1 --2- 172.21.15.16:0/2972009803 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2af015b850 0x7f2af01f6550 secure :-1 s=READY pgs=2880 cs=0 l=1 rev1=1 crypto rx=0x7f2ae800cad0 tx=0x7f2ae800cde0 comp rx=0 tx=0).stop 2023-10-16T13:48:02.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.627+0000 7f2af55e0700 1 -- 172.21.15.16:0/2972009803 shutdown_connections 2023-10-16T13:48:02.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.627+0000 7f2af55e0700 1 --2- 172.21.15.16:0/2972009803 >> [v2:172.21.15.16:6832/2221624769,v1:172.21.15.16:6833/2221624769] conn(0x7f2adc06b140 0x7f2adc06d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:02.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.627+0000 7f2af55e0700 1 --2- 172.21.15.16:0/2972009803 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2af015b850 0x7f2af01f6550 unknown :-1 s=CLOSED pgs=2880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:02.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.627+0000 7f2af55e0700 1 --2- 172.21.15.16:0/2972009803 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2af015afc0 0x7f2af00ac8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:02.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.627+0000 7f2af55e0700 1 --2- 172.21.15.16:0/2972009803 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2af015b3a0 0x7f2af00cd9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:02.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.627+0000 7f2af55e0700 1 -- 172.21.15.16:0/2972009803 >> 172.21.15.16:0/2972009803 conn(0x7f2af006a7e0 msgr2=0x7f2af015a220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:02.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.627+0000 7f2af55e0700 1 -- 172.21.15.16:0/2972009803 shutdown_connections 2023-10-16T13:48:02.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.627+0000 7f2af55e0700 1 -- 172.21.15.16:0/2972009803 wait complete. 2023-10-16T13:48:02.642 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:48:02.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.765+0000 7f7b0add7700 1 Processor -- start 2023-10-16T13:48:02.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.765+0000 7f7b0add7700 1 -- start start 2023-10-16T13:48:02.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.765+0000 7f7b0add7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7b040cf4a0 0x7f7b040726e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:02.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.765+0000 7f7b0add7700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f7b040b5a00 0x7f7b0415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:02.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.765+0000 7f7b0add7700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b0415bc80 con 0x7f7b040cf880 2023-10-16T13:48:02.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.765+0000 7f7b0add7700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b040b5e70 con 0x7f7b040cf4a0 2023-10-16T13:48:02.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.765+0000 7f7b0add7700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b04072c20 con 0x7f7b040b5a00 2023-10-16T13:48:02.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.766+0000 7f7b08b73700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7b040cf4a0 0x7f7b040726e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:02.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.766+0000 7f7b08b73700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7b040cf4a0 0x7f7b040726e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56838/0 (socket says 172.21.15.16:56838) 2023-10-16T13:48:02.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.766+0000 7f7b08b73700 1 -- 172.21.15.16:0/1802772153 learned_addr learned my addr 172.21.15.16:0/1802772153 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:02.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.766+0000 7f7b09374700 1 --2- 172.21.15.16:0/1802772153 >> v2:172.21.15.49:3300/0 conn(0x7f7b040b5a00 0x7f7b0415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:02.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.766+0000 7f7b08b73700 1 -- 172.21.15.16:0/1802772153 >> v2:172.21.15.49:3300/0 conn(0x7f7b040b5a00 msgr2=0x7f7b0415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:02.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.766+0000 7f7b08b73700 1 --2- 172.21.15.16:0/1802772153 >> v2:172.21.15.49:3300/0 conn(0x7f7b040b5a00 0x7f7b0415a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:02.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.766+0000 7f7b08b73700 1 -- 172.21.15.16:0/1802772153 >> v1:172.21.15.16:6789/0 conn(0x7f7b040cf880 legacy=0x7f7b040b52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:02.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.766+0000 7f7b08b73700 1 -- 172.21.15.16:0/1802772153 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b0415b070 con 0x7f7b040cf4a0 2023-10-16T13:48:02.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.766+0000 7f7b08b73700 1 --2- 172.21.15.16:0/1802772153 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b040cf4a0 0x7f7b040726e0 secure :-1 s=READY pgs=2892 cs=0 l=1 rev1=1 crypto rx=0x7f7b040cdeb0 tx=0x7f7af8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d2c9a6521e8ac8c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:02.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.766+0000 7f7b037fe700 1 -- 172.21.15.16:0/1802772153 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af8011830 con 0x7f7b040cf4a0 2023-10-16T13:48:02.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.766+0000 7f7b037fe700 1 -- 172.21.15.16:0/1802772153 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7af8011990 con 0x7f7b040cf4a0 2023-10-16T13:48:02.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.766+0000 7f7b037fe700 1 -- 172.21.15.16:0/1802772153 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af8011d20 con 0x7f7b040cf4a0 2023-10-16T13:48:02.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.767+0000 7f7b0add7700 1 -- 172.21.15.16:0/1802772153 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b040cf4a0 msgr2=0x7f7b040726e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:02.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.767+0000 7f7b0add7700 1 --2- 172.21.15.16:0/1802772153 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b040cf4a0 0x7f7b040726e0 secure :-1 s=READY pgs=2892 cs=0 l=1 rev1=1 crypto rx=0x7f7b040cdeb0 tx=0x7f7af8009bd0 comp rx=0 tx=0).stop 2023-10-16T13:48:02.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.767+0000 7f7b0add7700 1 -- 172.21.15.16:0/1802772153 shutdown_connections 2023-10-16T13:48:02.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.767+0000 7f7b0add7700 1 --2- 172.21.15.16:0/1802772153 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b040cf4a0 0x7f7b040726e0 unknown :-1 s=CLOSED pgs=2892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:02.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.767+0000 7f7b0add7700 1 --2- 172.21.15.16:0/1802772153 >> v2:172.21.15.49:3300/0 conn(0x7f7b040b5a00 0x7f7b0415a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:02.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.767+0000 7f7b0add7700 1 -- 172.21.15.16:0/1802772153 >> 172.21.15.16:0/1802772153 conn(0x7f7b040b1fd0 msgr2=0x7f7b040b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:02.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.767+0000 7f7b0add7700 1 -- 172.21.15.16:0/1802772153 shutdown_connections 2023-10-16T13:48:02.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.767+0000 7f7b0add7700 1 -- 172.21.15.16:0/1802772153 wait complete. 2023-10-16T13:48:02.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.768+0000 7f7b0add7700 1 Processor -- start 2023-10-16T13:48:02.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.768+0000 7f7b0add7700 1 -- start start 2023-10-16T13:48:02.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.769+0000 7f7b0add7700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7b040b5a00 0x7f7b0414d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:02.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.769+0000 7f7b0add7700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7b040cf4a0 0x7f7b040c7510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:02.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.769+0000 7f7b0add7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b040cf880 0x7f7b0406a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:02.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.769+0000 7f7b0add7700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b040cc640 con 0x7f7b040cf880 2023-10-16T13:48:02.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.769+0000 7f7b0add7700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b040cba30 con 0x7f7b040cf4a0 2023-10-16T13:48:02.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.769+0000 7f7b0add7700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b040cbd10 con 0x7f7b040b5a00 2023-10-16T13:48:02.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.769+0000 7f7b09374700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b040cf880 0x7f7b0406a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:02.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.769+0000 7f7b09374700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b040cf880 0x7f7b0406a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56848/0 (socket says 172.21.15.16:56848) 2023-10-16T13:48:02.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.769+0000 7f7b09374700 1 -- 172.21.15.16:0/477947157 learned_addr learned my addr 172.21.15.16:0/477947157 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:02.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.769+0000 7f7b03fff700 1 --2- 172.21.15.16:0/477947157 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7b040cf4a0 0x7f7b040c7510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:02.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.769+0000 7f7b08b73700 1 --2- 172.21.15.16:0/477947157 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7b040b5a00 0x7f7b0414d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:02.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.769+0000 7f7b09374700 1 -- 172.21.15.16:0/477947157 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7b040b5a00 msgr2=0x7f7b0414d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:02.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.769+0000 7f7b09374700 1 --2- 172.21.15.16:0/477947157 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7b040b5a00 0x7f7b0414d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:02.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.769+0000 7f7b09374700 1 -- 172.21.15.16:0/477947157 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7b040cf4a0 msgr2=0x7f7b040c7510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:02.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.770+0000 7f7b09374700 1 --2- 172.21.15.16:0/477947157 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7b040cf4a0 0x7f7b040c7510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:02.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.770+0000 7f7b09374700 1 -- 172.21.15.16:0/477947157 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7af8009550 con 0x7f7b040cf880 2023-10-16T13:48:02.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.770+0000 7f7b09374700 1 --2- 172.21.15.16:0/477947157 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b040cf880 0x7f7b0406a7e0 secure :-1 s=READY pgs=2893 cs=0 l=1 rev1=1 crypto rx=0x7f7afc00efc0 tx=0x7f7afc00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:02.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.770+0000 7f7b017fa700 1 -- 172.21.15.16:0/477947157 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7afc009ea0 con 0x7f7b040cf880 2023-10-16T13:48:02.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.770+0000 7f7b017fa700 1 -- 172.21.15.16:0/477947157 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7afc00f040 con 0x7f7b040cf880 2023-10-16T13:48:02.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.770+0000 7f7b0add7700 1 -- 172.21.15.16:0/477947157 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b040cbff0 con 0x7f7b040cf880 2023-10-16T13:48:02.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.770+0000 7f7b017fa700 1 -- 172.21.15.16:0/477947157 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7afc00ced0 con 0x7f7b040cf880 2023-10-16T13:48:02.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.770+0000 7f7b0add7700 1 -- 172.21.15.16:0/477947157 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b0414dd80 con 0x7f7b040cf880 2023-10-16T13:48:02.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.771+0000 7f7b0add7700 1 -- 172.21.15.16:0/477947157 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b0404ebb0 con 0x7f7b040cf880 2023-10-16T13:48:02.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.771+0000 7f7b017fa700 1 -- 172.21.15.16:0/477947157 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 154) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f7afc014480 con 0x7f7b040cf880 2023-10-16T13:48:02.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.772+0000 7f7b017fa700 1 -- 172.21.15.16:0/477947157 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(251..251 src has 1..251) v4 ==== 7881+0+0 (secure 0 0 0) 0x7f7afc08a2c0 con 0x7f7b040cf880 2023-10-16T13:48:02.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:02.776+0000 7f7b017fa700 1 -- 172.21.15.16:0/477947157 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7afc054dd0 con 0x7f7b040cf880 2023-10-16T13:48:03.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.024+0000 7f7b0add7700 1 -- 172.21.15.16:0/477947157 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7b040cc780 con 0x7f7b040cf880 2023-10-16T13:48:03.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.025+0000 7f7b017fa700 1 -- 172.21.15.16:0/477947157 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) v1 ==== 95+0+6147 (secure 0 0 0) 0x7f7afc018020 con 0x7f7b040cf880 2023-10-16T13:48:03.027 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:48:03.028 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":135,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9596,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/3939647668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":3939647668},{"type":"v1","addr":"172.21.15.49:6834","nonce":3939647668}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":125},{"gid":11185,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/2890113513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2890113513},{"type":"v1","addr":"172.21.15.16:6836","nonce":2890113513}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":125}],"filesystems":[{"mdsmap":{"epoch":130,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:47:23.843333+0000","modified":"2023-10-16T13:47:26.851597+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":12252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12252":{"gid":12252,"name":"a","rank":0,"incarnation":126,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/303278795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":303278795},{"type":"v1","addr":"172.21.15.16:6837","nonce":303278795}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:47:41.332096+0000","modified":"2023-10-16T13:47:44.368830+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11197},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11197":{"gid":11197,"name":"d","rank":0,"incarnation":132,"state":"up:active","state_seq":7,"addr":"172.21.15.49:6835/680748068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":680748068},{"type":"v1","addr":"172.21.15.49:6835","nonce":680748068}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2023-10-16T13:48:03.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.027+0000 7f7b0add7700 1 -- 172.21.15.16:0/477947157 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b040cf880 msgr2=0x7f7b0406a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:03.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.027+0000 7f7b0add7700 1 --2- 172.21.15.16:0/477947157 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b040cf880 0x7f7b0406a7e0 secure :-1 s=READY pgs=2893 cs=0 l=1 rev1=1 crypto rx=0x7f7afc00efc0 tx=0x7f7afc00c3a0 comp rx=0 tx=0).stop 2023-10-16T13:48:03.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.027+0000 7f7b0add7700 1 -- 172.21.15.16:0/477947157 shutdown_connections 2023-10-16T13:48:03.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.027+0000 7f7b0add7700 1 --2- 172.21.15.16:0/477947157 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b040cf880 0x7f7b0406a7e0 unknown :-1 s=CLOSED pgs=2893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:03.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.027+0000 7f7b0add7700 1 --2- 172.21.15.16:0/477947157 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7b040b5a00 0x7f7b0414d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:03.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.027+0000 7f7b0add7700 1 --2- 172.21.15.16:0/477947157 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7b040cf4a0 0x7f7b040c7510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:03.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.027+0000 7f7b0add7700 1 -- 172.21.15.16:0/477947157 >> 172.21.15.16:0/477947157 conn(0x7f7b040b1fd0 msgr2=0x7f7b040b1800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:03.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.027+0000 7f7b0add7700 1 -- 172.21.15.16:0/477947157 shutdown_connections 2023-10-16T13:48:03.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.027+0000 7f7b0add7700 1 -- 172.21.15.16:0/477947157 wait complete. 2023-10-16T13:48:03.032 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 135 2023-10-16T13:48:03.042 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:48:03.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.165+0000 7f80c6bb6700 1 Processor -- start 2023-10-16T13:48:03.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.165+0000 7f80c6bb6700 1 -- start start 2023-10-16T13:48:03.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.166+0000 7f80c6bb6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f80c00cf880 0x7f80c00b52f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:03.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.166+0000 7f80c6bb6700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80c015bc80 con 0x7f80c00cf4a0 2023-10-16T13:48:03.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.166+0000 7f80c6bb6700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80c015b070 con 0x7f80c00b5830 2023-10-16T13:48:03.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.166+0000 7f80c6bb6700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80c015b350 con 0x7f80c00cf880 2023-10-16T13:48:03.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.166+0000 7f80bffff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f80c00cf880 0x7f80c00b52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:03.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.166+0000 7f80bffff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f80c00cf880 0x7f80c00b52f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56852/0 (socket says 172.21.15.16:56852) 2023-10-16T13:48:03.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.166+0000 7f80bffff700 1 -- 172.21.15.16:0/2147723027 learned_addr learned my addr 172.21.15.16:0/2147723027 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:03.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.166+0000 7f80bffff700 1 -- 172.21.15.16:0/2147723027 >> v1:172.21.15.49:6789/0 conn(0x7f80c00b5830 legacy=0x7f80c015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:03.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.166+0000 7f80bffff700 1 -- 172.21.15.16:0/2147723027 >> v1:172.21.15.16:6789/0 conn(0x7f80c00cf4a0 legacy=0x7f80c00726e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:03.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.166+0000 7f80bffff700 1 -- 172.21.15.16:0/2147723027 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80c015b630 con 0x7f80c00cf880 2023-10-16T13:48:03.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.166+0000 7f80bffff700 1 --2- 172.21.15.16:0/2147723027 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80c00cf880 0x7f80c00b52f0 secure :-1 s=READY pgs=2895 cs=0 l=1 rev1=1 crypto rx=0x7f80ac0098f0 tx=0x7f80ac009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0da6bbd2448425a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:03.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.167+0000 7f80bf7fe700 1 -- 172.21.15.16:0/2147723027 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80ac00b8a0 con 0x7f80c00cf880 2023-10-16T13:48:03.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.167+0000 7f80bf7fe700 1 -- 172.21.15.16:0/2147723027 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80ac00ba00 con 0x7f80c00cf880 2023-10-16T13:48:03.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.167+0000 7f80bf7fe700 1 -- 172.21.15.16:0/2147723027 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80ac00bd90 con 0x7f80c00cf880 2023-10-16T13:48:03.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.167+0000 7f80c6bb6700 1 -- 172.21.15.16:0/2147723027 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80c00cf880 msgr2=0x7f80c00b52f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:03.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.167+0000 7f80c6bb6700 1 --2- 172.21.15.16:0/2147723027 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80c00cf880 0x7f80c00b52f0 secure :-1 s=READY pgs=2895 cs=0 l=1 rev1=1 crypto rx=0x7f80ac0098f0 tx=0x7f80ac009c00 comp rx=0 tx=0).stop 2023-10-16T13:48:03.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.167+0000 7f80c6bb6700 1 -- 172.21.15.16:0/2147723027 shutdown_connections 2023-10-16T13:48:03.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.167+0000 7f80c6bb6700 1 --2- 172.21.15.16:0/2147723027 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80c00cf880 0x7f80c00b52f0 unknown :-1 s=CLOSED pgs=2895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:03.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.167+0000 7f80c6bb6700 1 -- 172.21.15.16:0/2147723027 >> 172.21.15.16:0/2147723027 conn(0x7f80c00b1fd0 msgr2=0x7f80c00b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:03.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.167+0000 7f80c6bb6700 1 -- 172.21.15.16:0/2147723027 shutdown_connections 2023-10-16T13:48:03.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.167+0000 7f80c6bb6700 1 -- 172.21.15.16:0/2147723027 wait complete. 2023-10-16T13:48:03.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.168+0000 7f80c6bb6700 1 Processor -- start 2023-10-16T13:48:03.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.168+0000 7f80c6bb6700 1 -- start start 2023-10-16T13:48:03.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.169+0000 7f80c6bb6700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f80c00b5830 0x7f80c014f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:03.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.169+0000 7f80c6bb6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80c00cf4a0 0x7f80c015c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:03.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.169+0000 7f80c6bb6700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f80c00cf880 0x7f80c015c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:03.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.169+0000 7f80c6bb6700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80c01f7180 con 0x7f80c00cf4a0 2023-10-16T13:48:03.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.169+0000 7f80c6bb6700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80c0159650 con 0x7f80c00cf880 2023-10-16T13:48:03.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.169+0000 7f80c6bb6700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80c00cc980 con 0x7f80c00b5830 2023-10-16T13:48:03.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.169+0000 7f80bffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80c00cf4a0 0x7f80c015c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:03.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.169+0000 7f80bffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80c00cf4a0 0x7f80c015c220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56858/0 (socket says 172.21.15.16:56858) 2023-10-16T13:48:03.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.169+0000 7f80bffff700 1 -- 172.21.15.16:0/1101207702 learned_addr learned my addr 172.21.15.16:0/1101207702 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:03.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.170+0000 7f80bffff700 1 -- 172.21.15.16:0/1101207702 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f80c00b5830 msgr2=0x7f80c014f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:03.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.170+0000 7f80bffff700 1 --2- 172.21.15.16:0/1101207702 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f80c00b5830 0x7f80c014f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:03.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.170+0000 7f80bffff700 1 -- 172.21.15.16:0/1101207702 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f80c00cf880 msgr2=0x7f80c015c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:03.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.170+0000 7f80c5153700 1 --2- 172.21.15.16:0/1101207702 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f80c00cf880 0x7f80c015c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:03.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.170+0000 7f80bffff700 1 --2- 172.21.15.16:0/1101207702 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f80c00cf880 0x7f80c015c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:03.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.170+0000 7f80bffff700 1 -- 172.21.15.16:0/1101207702 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80ac009550 con 0x7f80c00cf4a0 2023-10-16T13:48:03.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.170+0000 7f80bffff700 1 --2- 172.21.15.16:0/1101207702 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80c00cf4a0 0x7f80c015c220 secure :-1 s=READY pgs=2896 cs=0 l=1 rev1=1 crypto rx=0x7f80ac009130 tx=0x7f80ac004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:03.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.170+0000 7f80bd7fa700 1 -- 172.21.15.16:0/1101207702 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80ac0036a0 con 0x7f80c00cf4a0 2023-10-16T13:48:03.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.170+0000 7f80bd7fa700 1 -- 172.21.15.16:0/1101207702 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80ac01e070 con 0x7f80c00cf4a0 2023-10-16T13:48:03.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.170+0000 7f80c6bb6700 1 -- 172.21.15.16:0/1101207702 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80c014fdf0 con 0x7f80c00cf4a0 2023-10-16T13:48:03.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.170+0000 7f80bd7fa700 1 -- 172.21.15.16:0/1101207702 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80ac003a60 con 0x7f80c00cf4a0 2023-10-16T13:48:03.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.170+0000 7f80c6bb6700 1 -- 172.21.15.16:0/1101207702 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80c01f6ac0 con 0x7f80c00cf4a0 2023-10-16T13:48:03.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.171+0000 7f80c6bb6700 1 -- 172.21.15.16:0/1101207702 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80c01f72c0 con 0x7f80c00cf4a0 2023-10-16T13:48:03.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.172+0000 7f80bd7fa700 1 -- 172.21.15.16:0/1101207702 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 154) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f80ac003bc0 con 0x7f80c00cf4a0 2023-10-16T13:48:03.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.172+0000 7f80bd7fa700 1 -- 172.21.15.16:0/1101207702 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(251..251 src has 1..251) v4 ==== 7881+0+0 (secure 0 0 0) 0x7f80ac020070 con 0x7f80c00cf4a0 2023-10-16T13:48:03.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.177+0000 7f80bd7fa700 1 -- 172.21.15.16:0/1101207702 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f80c01f72c0 con 0x7f80c00cf4a0 2023-10-16T13:48:03.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.381+0000 7f80c6bb6700 1 -- 172.21.15.16:0/1101207702 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f80c01f72c0 con 0x7f80c00cf4a0 2023-10-16T13:48:03.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.382+0000 7f80bd7fa700 1 -- 172.21.15.16:0/1101207702 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v251) v1 ==== 74+0+21789 (secure 0 0 0) 0x7f80c01f72c0 con 0x7f80c00cf4a0 2023-10-16T13:48:03.385 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:48:03.385 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":251,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:48:02.485657+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2023-10-16T13:47:22.338349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"244","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2023-10-16T13:47:22.806327+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"245","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T13:47:39.075528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"248","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"backup_fs_data","create_time":"2023-10-16T13:47:40.314029+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"249","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6832/2221624769":"2023-10-17T13:48:02.485614+0000","172.21.15.16:0/2242986614":"2023-10-17T13:48:02.485614+0000","172.21.15.16:0/3610227254":"2023-10-17T13:48:02.485614+0000","172.21.15.16:0/1182888331":"2023-10-17T13:48:02.485614+0000","172.21.15.16:0/2095012141":"2023-10-17T13:48:02.485614+0000","172.21.15.16:0/874722077":"2023-10-17T13:48:02.485614+0000","172.21.15.16:0/2066401556":"2023-10-17T13:48:02.485614+0000","172.21.15.16:0/3947601914":"2023-10-17T13:47:54.816700+0000","172.21.15.16:6833/2221624769":"2023-10-17T13:48:02.485614+0000","172.21.15.16:0/2759470534":"2023-10-17T13:47:54.816700+0000","172.21.15.16:0/2634105683":"2023-10-17T13:48:02.485614+0000","172.21.15.16:0/2974892940":"2023-10-17T13:47:54.816700+0000","172.21.15.16:6833/1736104576":"2023-10-17T13:47:54.816700+0000","172.21.15.16:6832/1736104576":"2023-10-17T13:47:54.816700+0000","172.21.15.16:0/1246409088":"2023-10-17T13:48:02.485614+0000","172.21.15.16:0/318862437":"2023-10-17T13:47:54.816700+0000","172.21.15.16:0/413581668":"2023-10-17T13:47:54.816700+0000","172.21.15.16:0/3740515383":"2023-10-17T13:47:54.816700+0000","172.21.15.16:0/3105924907":"2023-10-17T13:47:54.816700+0000","172.21.15.16:0/3617558038":"2023-10-17T13:47:54.816700+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:48:03.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.384+0000 7f80c6bb6700 1 -- 172.21.15.16:0/1101207702 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80c00cf4a0 msgr2=0x7f80c015c220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:03.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.384+0000 7f80c6bb6700 1 --2- 172.21.15.16:0/1101207702 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80c00cf4a0 0x7f80c015c220 secure :-1 s=READY pgs=2896 cs=0 l=1 rev1=1 crypto rx=0x7f80ac009130 tx=0x7f80ac004750 comp rx=0 tx=0).stop 2023-10-16T13:48:03.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.384+0000 7f80c6bb6700 1 -- 172.21.15.16:0/1101207702 shutdown_connections 2023-10-16T13:48:03.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.384+0000 7f80c6bb6700 1 --2- 172.21.15.16:0/1101207702 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f80c00cf880 0x7f80c015c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:03.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.384+0000 7f80c6bb6700 1 --2- 172.21.15.16:0/1101207702 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f80c00b5830 0x7f80c014f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:03.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.384+0000 7f80c6bb6700 1 --2- 172.21.15.16:0/1101207702 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80c00cf4a0 0x7f80c015c220 unknown :-1 s=CLOSED pgs=2896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:03.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.384+0000 7f80c6bb6700 1 -- 172.21.15.16:0/1101207702 >> 172.21.15.16:0/1101207702 conn(0x7f80c00b1fd0 msgr2=0x7f80c00b1360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:03.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.385+0000 7f80c6bb6700 1 -- 172.21.15.16:0/1101207702 shutdown_connections 2023-10-16T13:48:03.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.385+0000 7f80c6bb6700 1 -- 172.21.15.16:0/1101207702 wait complete. 2023-10-16T13:48:03.400 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-16T13:48:03.400 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:48:03.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.530+0000 7fc259aaf700 1 Processor -- start 2023-10-16T13:48:03.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.530+0000 7fc259aaf700 1 -- start start 2023-10-16T13:48:03.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.531+0000 7fc259aaf700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc25415b280 0x7fc2541613e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:03.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.531+0000 7fc259aaf700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc25406a7e0 0x7fc25406ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:03.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.531+0000 7fc259aaf700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc25406b170 con 0x7fc2540cc8a0 2023-10-16T13:48:03.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.531+0000 7fc259aaf700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc254161c30 con 0x7fc25406a7e0 2023-10-16T13:48:03.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.531+0000 7fc259aaf700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc254161f10 con 0x7fc25415b280 2023-10-16T13:48:03.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.531+0000 7fc253fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc25406a7e0 0x7fc25406ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:03.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.531+0000 7fc253fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc25406a7e0 0x7fc25406ac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56888/0 (socket says 172.21.15.16:56888) 2023-10-16T13:48:03.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.531+0000 7fc253fff700 1 -- 172.21.15.16:0/3219157563 learned_addr learned my addr 172.21.15.16:0/3219157563 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:03.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.531+0000 7fc2537fe700 1 --2- 172.21.15.16:0/3219157563 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc25415b280 0x7fc2541613e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:03.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.531+0000 7fc253fff700 1 -- 172.21.15.16:0/3219157563 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc25415b280 msgr2=0x7fc2541613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:03.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.531+0000 7fc253fff700 1 --2- 172.21.15.16:0/3219157563 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc25415b280 0x7fc2541613e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:03.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.531+0000 7fc253fff700 1 -- 172.21.15.16:0/3219157563 >> v1:172.21.15.49:6789/0 conn(0x7fc2540cc8a0 legacy=0x7fc25415e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:03.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.531+0000 7fc253fff700 1 -- 172.21.15.16:0/3219157563 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2541621f0 con 0x7fc25406a7e0 2023-10-16T13:48:03.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.531+0000 7fc253fff700 1 --2- 172.21.15.16:0/3219157563 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc25406a7e0 0x7fc25406ac30 secure :-1 s=READY pgs=2897 cs=0 l=1 rev1=1 crypto rx=0x7fc24c009e50 tx=0x7fc24c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d0b76c9c31b1a6e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:03.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.531+0000 7fc2527fc700 1 -- 172.21.15.16:0/3219157563 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc24c00e070 con 0x7fc25406a7e0 2023-10-16T13:48:03.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.532+0000 7fc2527fc700 1 -- 172.21.15.16:0/3219157563 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc24c00b9c0 con 0x7fc25406a7e0 2023-10-16T13:48:03.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.532+0000 7fc2527fc700 1 -- 172.21.15.16:0/3219157563 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc24c00bd70 con 0x7fc25406a7e0 2023-10-16T13:48:03.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.532+0000 7fc259aaf700 1 -- 172.21.15.16:0/3219157563 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc25406a7e0 msgr2=0x7fc25406ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:03.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.532+0000 7fc259aaf700 1 --2- 172.21.15.16:0/3219157563 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc25406a7e0 0x7fc25406ac30 secure :-1 s=READY pgs=2897 cs=0 l=1 rev1=1 crypto rx=0x7fc24c009e50 tx=0x7fc24c00b330 comp rx=0 tx=0).stop 2023-10-16T13:48:03.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.532+0000 7fc259aaf700 1 -- 172.21.15.16:0/3219157563 shutdown_connections 2023-10-16T13:48:03.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.532+0000 7fc259aaf700 1 --2- 172.21.15.16:0/3219157563 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc25406a7e0 0x7fc25406ac30 unknown :-1 s=CLOSED pgs=2897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:03.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.532+0000 7fc259aaf700 1 --2- 172.21.15.16:0/3219157563 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc25415b280 0x7fc2541613e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:03.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.532+0000 7fc259aaf700 1 -- 172.21.15.16:0/3219157563 >> 172.21.15.16:0/3219157563 conn(0x7fc2540b1fd0 msgr2=0x7fc2540b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:03.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.532+0000 7fc259aaf700 1 -- 172.21.15.16:0/3219157563 shutdown_connections 2023-10-16T13:48:03.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.532+0000 7fc259aaf700 1 -- 172.21.15.16:0/3219157563 wait complete. 2023-10-16T13:48:03.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.534+0000 7fc259aaf700 1 Processor -- start 2023-10-16T13:48:03.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.534+0000 7fc259aaf700 1 -- start start 2023-10-16T13:48:03.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.535+0000 7fc259aaf700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc25406a7e0 0x7fc2541562c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:03.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.535+0000 7fc259aaf700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc2540cc8a0 0x7fc2540c6640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:03.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.535+0000 7fc259aaf700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc25415b280 0x7fc2541f6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:03.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.535+0000 7fc259aaf700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2541f5e60 con 0x7fc25415b280 2023-10-16T13:48:03.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.535+0000 7fc259aaf700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc254156800 con 0x7fc2540cc8a0 2023-10-16T13:48:03.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.535+0000 7fc259aaf700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2540c6e00 con 0x7fc25406a7e0 2023-10-16T13:48:03.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.535+0000 7fc253fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc25415b280 0x7fc2541f6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:03.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.535+0000 7fc253fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc25415b280 0x7fc2541f6070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56894/0 (socket says 172.21.15.16:56894) 2023-10-16T13:48:03.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.535+0000 7fc253fff700 1 -- 172.21.15.16:0/1942679067 learned_addr learned my addr 172.21.15.16:0/1942679067 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:03.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.535+0000 7fc2537fe700 1 --2- 172.21.15.16:0/1942679067 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc25406a7e0 0x7fc2541562c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:03.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.535+0000 7fc252ffd700 1 --2- 172.21.15.16:0/1942679067 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc2540cc8a0 0x7fc2540c6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:03.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.535+0000 7fc253fff700 1 -- 172.21.15.16:0/1942679067 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc25406a7e0 msgr2=0x7fc2541562c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:03.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.535+0000 7fc253fff700 1 --2- 172.21.15.16:0/1942679067 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc25406a7e0 0x7fc2541562c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:03.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.535+0000 7fc253fff700 1 -- 172.21.15.16:0/1942679067 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc2540cc8a0 msgr2=0x7fc2540c6640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:03.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.535+0000 7fc253fff700 1 --2- 172.21.15.16:0/1942679067 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc2540cc8a0 0x7fc2540c6640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:03.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.535+0000 7fc253fff700 1 -- 172.21.15.16:0/1942679067 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc24c009b30 con 0x7fc25415b280 2023-10-16T13:48:03.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.535+0000 7fc253fff700 1 --2- 172.21.15.16:0/1942679067 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc25415b280 0x7fc2541f6070 secure :-1 s=READY pgs=2898 cs=0 l=1 rev1=1 crypto rx=0x7fc2540d1540 tx=0x7fc24c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:03.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.536+0000 7fc24bfff700 1 -- 172.21.15.16:0/1942679067 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc24c004800 con 0x7fc25415b280 2023-10-16T13:48:03.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.536+0000 7fc24bfff700 1 -- 172.21.15.16:0/1942679067 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc24c004960 con 0x7fc25415b280 2023-10-16T13:48:03.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.536+0000 7fc259aaf700 1 -- 172.21.15.16:0/1942679067 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2540c6b80 con 0x7fc25415b280 2023-10-16T13:48:03.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.536+0000 7fc24bfff700 1 -- 172.21.15.16:0/1942679067 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc24c01c680 con 0x7fc25415b280 2023-10-16T13:48:03.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.536+0000 7fc259aaf700 1 -- 172.21.15.16:0/1942679067 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc25415c210 con 0x7fc25415b280 2023-10-16T13:48:03.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.537+0000 7fc259aaf700 1 -- 172.21.15.16:0/1942679067 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc25404ebb0 con 0x7fc25415b280 2023-10-16T13:48:03.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.537+0000 7fc24bfff700 1 -- 172.21.15.16:0/1942679067 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 155) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fc24c01c7e0 con 0x7fc25415b280 2023-10-16T13:48:03.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.537+0000 7fc24bfff700 1 --2- 172.21.15.16:0/1942679067 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fc23406ad50 0x7fc23406d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:03.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.538+0000 7fc24bfff700 1 -- 172.21.15.16:0/1942679067 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(251..251 src has 1..251) v4 ==== 7881+0+0 (secure 0 0 0) 0x7fc24c08b6b0 con 0x7fc25415b280 2023-10-16T13:48:03.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.538+0000 7fc2537fe700 1 --2- 172.21.15.16:0/1942679067 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fc23406ad50 0x7fc23406d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:03.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.538+0000 7fc2537fe700 1 --2- 172.21.15.16:0/1942679067 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fc23406ad50 0x7fc23406d210 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc23c006b20 tx=0x7fc23c008040 comp rx=0 tx=0).ready entity=mgr.12678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:03.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.542+0000 7fc24bfff700 1 -- 172.21.15.16:0/1942679067 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc24c056220 con 0x7fc25415b280 2023-10-16T13:48:03.776 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:48:03.778 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:48:03.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.796+0000 7fc259aaf700 1 -- 172.21.15.16:0/1942679067 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc2540c3830 con 0x7fc25415b280 2023-10-16T13:48:03.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.796+0000 7fc24bfff700 1 -- 172.21.15.16:0/1942679067 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) v1 ==== 95+0+6147 (secure 0 0 0) 0x7fc24c059840 con 0x7fc25415b280 2023-10-16T13:48:03.799 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:48:03.799 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":135,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9596,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/3939647668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":3939647668},{"type":"v1","addr":"172.21.15.49:6834","nonce":3939647668}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":125},{"gid":11185,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/2890113513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2890113513},{"type":"v1","addr":"172.21.15.16:6836","nonce":2890113513}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":125}],"filesystems":[{"mdsmap":{"epoch":130,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:47:23.843333+0000","modified":"2023-10-16T13:47:26.851597+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":12252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12252":{"gid":12252,"name":"a","rank":0,"incarnation":126,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/303278795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":303278795},{"type":"v1","addr":"172.21.15.16:6837","nonce":303278795}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:47:41.332096+0000","modified":"2023-10-16T13:47:44.368830+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11197},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11197":{"gid":11197,"name":"d","rank":0,"incarnation":132,"state":"up:active","state_seq":7,"addr":"172.21.15.49:6835/680748068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":680748068},{"type":"v1","addr":"172.21.15.49:6835","nonce":680748068}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2023-10-16T13:48:03.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.799+0000 7fc259aaf700 1 -- 172.21.15.16:0/1942679067 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fc23406ad50 msgr2=0x7fc23406d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:03.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.799+0000 7fc259aaf700 1 --2- 172.21.15.16:0/1942679067 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fc23406ad50 0x7fc23406d210 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc23c006b20 tx=0x7fc23c008040 comp rx=0 tx=0).stop 2023-10-16T13:48:03.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.799+0000 7fc259aaf700 1 -- 172.21.15.16:0/1942679067 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc25415b280 msgr2=0x7fc2541f6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:03.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.799+0000 7fc259aaf700 1 --2- 172.21.15.16:0/1942679067 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc25415b280 0x7fc2541f6070 secure :-1 s=READY pgs=2898 cs=0 l=1 rev1=1 crypto rx=0x7fc2540d1540 tx=0x7fc24c004ad0 comp rx=0 tx=0).stop 2023-10-16T13:48:03.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.799+0000 7fc259aaf700 1 -- 172.21.15.16:0/1942679067 shutdown_connections 2023-10-16T13:48:03.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.799+0000 7fc259aaf700 1 --2- 172.21.15.16:0/1942679067 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fc23406ad50 0x7fc23406d210 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:03.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.799+0000 7fc259aaf700 1 --2- 172.21.15.16:0/1942679067 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc25415b280 0x7fc2541f6070 unknown :-1 s=CLOSED pgs=2898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:03.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.799+0000 7fc259aaf700 1 --2- 172.21.15.16:0/1942679067 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc25406a7e0 0x7fc2541562c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:03.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.799+0000 7fc259aaf700 1 --2- 172.21.15.16:0/1942679067 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc2540cc8a0 0x7fc2540c6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:03.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.799+0000 7fc259aaf700 1 -- 172.21.15.16:0/1942679067 >> 172.21.15.16:0/1942679067 conn(0x7fc2540b1fd0 msgr2=0x7fc25415a470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:03.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.799+0000 7fc259aaf700 1 -- 172.21.15.16:0/1942679067 shutdown_connections 2023-10-16T13:48:03.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.799+0000 7fc259aaf700 1 -- 172.21.15.16:0/1942679067 wait complete. 2023-10-16T13:48:03.804 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 135 2023-10-16T13:48:03.814 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:48:04.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.998+0000 7fd6a375d700 1 Processor -- start 2023-10-16T13:48:04.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.998+0000 7fd6a375d700 1 -- start start 2023-10-16T13:48:04.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.999+0000 7fd6a375d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd69c15b630 0x7fd69c0cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:04.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.999+0000 7fd6a375d700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fd69c0ce4f0 0x7fd69c0cfca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:04.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.999+0000 7fd6a375d700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd69c0cc2a0 con 0x7fd69c15ba10 2023-10-16T13:48:04.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.999+0000 7fd6a375d700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd69c0cdb20 con 0x7fd69c15b630 2023-10-16T13:48:04.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.999+0000 7fd6a375d700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd69c0cc540 con 0x7fd69c0ce4f0 2023-10-16T13:48:04.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.999+0000 7fd6a14f9700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd69c15b630 0x7fd69c0cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:04.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.999+0000 7fd6a14f9700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd69c15b630 0x7fd69c0cd5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56898/0 (socket says 172.21.15.16:56898) 2023-10-16T13:48:04.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.999+0000 7fd6a14f9700 1 -- 172.21.15.16:0/3945978181 learned_addr learned my addr 172.21.15.16:0/3945978181 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:04.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.999+0000 7fd6a14f9700 1 -- 172.21.15.16:0/3945978181 >> v2:172.21.15.49:3300/0 conn(0x7fd69c0ce4f0 msgr2=0x7fd69c0cfca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:04.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.999+0000 7fd6a1cfa700 1 --2- 172.21.15.16:0/3945978181 >> v2:172.21.15.49:3300/0 conn(0x7fd69c0ce4f0 0x7fd69c0cfca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:04.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.999+0000 7fd6a14f9700 1 --2- 172.21.15.16:0/3945978181 >> v2:172.21.15.49:3300/0 conn(0x7fd69c0ce4f0 0x7fd69c0cfca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:04.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.999+0000 7fd693fff700 1 -- 172.21.15.16:0/3945978181 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1366948691 0 0) 0x7fd69c0cc2a0 con 0x7fd69c15ba10 2023-10-16T13:48:04.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.999+0000 7fd6a14f9700 1 -- 172.21.15.16:0/3945978181 >> v1:172.21.15.16:6789/0 conn(0x7fd69c15ba10 legacy=0x7fd69c0cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:04.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.999+0000 7fd6a14f9700 1 -- 172.21.15.16:0/3945978181 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd69c0cc820 con 0x7fd69c15b630 2023-10-16T13:48:04.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:03.999+0000 7fd6a1cfa700 1 --2- 172.21.15.16:0/3945978181 >> v2:172.21.15.49:3300/0 conn(0x7fd69c0ce4f0 0x7fd69c0cfca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:48:04.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.000+0000 7fd6a14f9700 1 --2- 172.21.15.16:0/3945978181 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd69c15b630 0x7fd69c0cd5e0 secure :-1 s=READY pgs=2900 cs=0 l=1 rev1=1 crypto rx=0x7fd69c0cba90 tx=0x7fd68c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36e96f2b535e20b3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:04.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.000+0000 7fd693fff700 1 -- 172.21.15.16:0/3945978181 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd68c005e30 con 0x7fd69c15b630 2023-10-16T13:48:04.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.000+0000 7fd693fff700 1 -- 172.21.15.16:0/3945978181 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd68c004030 con 0x7fd69c15b630 2023-10-16T13:48:04.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.000+0000 7fd693fff700 1 -- 172.21.15.16:0/3945978181 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd68c0043e0 con 0x7fd69c15b630 2023-10-16T13:48:04.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.000+0000 7fd6a375d700 1 -- 172.21.15.16:0/3945978181 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd69c15b630 msgr2=0x7fd69c0cd5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:04.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.000+0000 7fd6a375d700 1 --2- 172.21.15.16:0/3945978181 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd69c15b630 0x7fd69c0cd5e0 secure :-1 s=READY pgs=2900 cs=0 l=1 rev1=1 crypto rx=0x7fd69c0cba90 tx=0x7fd68c009bd0 comp rx=0 tx=0).stop 2023-10-16T13:48:04.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.000+0000 7fd6a375d700 1 -- 172.21.15.16:0/3945978181 shutdown_connections 2023-10-16T13:48:04.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.000+0000 7fd6a375d700 1 --2- 172.21.15.16:0/3945978181 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd69c15b630 0x7fd69c0cd5e0 unknown :-1 s=CLOSED pgs=2900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:04.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.000+0000 7fd6a375d700 1 --2- 172.21.15.16:0/3945978181 >> v2:172.21.15.49:3300/0 conn(0x7fd69c0ce4f0 0x7fd69c0cfca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:04.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.000+0000 7fd6a375d700 1 -- 172.21.15.16:0/3945978181 >> 172.21.15.16:0/3945978181 conn(0x7fd69c0d14b0 msgr2=0x7fd69c0d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:04.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.000+0000 7fd6a375d700 1 -- 172.21.15.16:0/3945978181 shutdown_connections 2023-10-16T13:48:04.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.000+0000 7fd6a375d700 1 -- 172.21.15.16:0/3945978181 wait complete. 2023-10-16T13:48:04.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.002+0000 7fd6a375d700 1 Processor -- start 2023-10-16T13:48:04.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.002+0000 7fd6a375d700 1 -- start start 2023-10-16T13:48:04.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.002+0000 7fd6a375d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd69c0ce4f0 0x7fd69c15e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:04.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.002+0000 7fd6a375d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd69c15b630 0x7fd69c1569a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:04.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.002+0000 7fd6a375d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd69c15ba10 0x7fd69c0b22e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:04.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.002+0000 7fd6a375d700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd69c0c9200 con 0x7fd69c0ce4f0 2023-10-16T13:48:04.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.002+0000 7fd6a375d700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd69c1f1ca0 con 0x7fd69c15ba10 2023-10-16T13:48:04.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.002+0000 7fd6a375d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd69c1f1120 con 0x7fd69c15b630 2023-10-16T13:48:04.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.002+0000 7fd6a14f9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd69c0ce4f0 0x7fd69c15e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:04.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.002+0000 7fd6a14f9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd69c0ce4f0 0x7fd69c15e3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56902/0 (socket says 172.21.15.16:56902) 2023-10-16T13:48:04.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.002+0000 7fd6a14f9700 1 -- 172.21.15.16:0/3258629479 learned_addr learned my addr 172.21.15.16:0/3258629479 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:04.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.003+0000 7fd6a14f9700 1 -- 172.21.15.16:0/3258629479 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd69c15b630 msgr2=0x7fd69c1569a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:04.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.003+0000 7fd6a14f9700 1 --2- 172.21.15.16:0/3258629479 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd69c15b630 0x7fd69c1569a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:04.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.003+0000 7fd6a14f9700 1 -- 172.21.15.16:0/3258629479 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd69c15ba10 msgr2=0x7fd69c0b22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:04.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.003+0000 7fd6a14f9700 1 --2- 172.21.15.16:0/3258629479 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd69c15ba10 0x7fd69c0b22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:04.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.003+0000 7fd6a14f9700 1 -- 172.21.15.16:0/3258629479 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd68c009500 con 0x7fd69c0ce4f0 2023-10-16T13:48:04.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.003+0000 7fd6a14f9700 1 --2- 172.21.15.16:0/3258629479 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd69c0ce4f0 0x7fd69c15e3a0 secure :-1 s=READY pgs=2901 cs=0 l=1 rev1=1 crypto rx=0x7fd68c009130 tx=0x7fd68c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:04.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.003+0000 7fd691ffb700 1 -- 172.21.15.16:0/3258629479 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd68c01d630 con 0x7fd69c0ce4f0 2023-10-16T13:48:04.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.003+0000 7fd691ffb700 1 -- 172.21.15.16:0/3258629479 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd68c01d7b0 con 0x7fd69c0ce4f0 2023-10-16T13:48:04.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.003+0000 7fd6a375d700 1 -- 172.21.15.16:0/3258629479 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd69c1f1400 con 0x7fd69c0ce4f0 2023-10-16T13:48:04.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.003+0000 7fd691ffb700 1 -- 172.21.15.16:0/3258629479 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd68c01db60 con 0x7fd69c0ce4f0 2023-10-16T13:48:04.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.003+0000 7fd6a375d700 1 -- 172.21.15.16:0/3258629479 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd69c1f19c0 con 0x7fd69c0ce4f0 2023-10-16T13:48:04.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.004+0000 7fd6a375d700 1 -- 172.21.15.16:0/3258629479 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd69c04ebb0 con 0x7fd69c0ce4f0 2023-10-16T13:48:04.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.005+0000 7fd691ffb700 1 -- 172.21.15.16:0/3258629479 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 155) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fd68c01dcc0 con 0x7fd69c0ce4f0 2023-10-16T13:48:04.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.005+0000 7fd691ffb700 1 --2- 172.21.15.16:0/3258629479 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fd68006ad50 0x7fd68006d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:04.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.005+0000 7fd691ffb700 1 -- 172.21.15.16:0/3258629479 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(251..251 src has 1..251) v4 ==== 7881+0+0 (secure 0 0 0) 0x7fd68c08ae10 con 0x7fd69c0ce4f0 2023-10-16T13:48:04.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.005+0000 7fd6a0cf8700 1 --2- 172.21.15.16:0/3258629479 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fd68006ad50 0x7fd68006d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:04.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.005+0000 7fd6a0cf8700 1 --2- 172.21.15.16:0/3258629479 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fd68006ad50 0x7fd68006d210 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fd698007fd0 tx=0x7fd6980079a0 comp rx=0 tx=0).ready entity=mgr.12678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:04.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.010+0000 7fd691ffb700 1 -- 172.21.15.16:0/3258629479 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd68c055980 con 0x7fd69c0ce4f0 2023-10-16T13:48:04.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.260+0000 7fd6a375d700 1 -- 172.21.15.16:0/3258629479 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd69c15e8e0 con 0x7fd69c0ce4f0 2023-10-16T13:48:04.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.260+0000 7fd691ffb700 1 -- 172.21.15.16:0/3258629479 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) v1 ==== 95+0+6147 (secure 0 0 0) 0x7fd68c016090 con 0x7fd69c0ce4f0 2023-10-16T13:48:04.263 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:48:04.263 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":135,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9596,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/3939647668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":3939647668},{"type":"v1","addr":"172.21.15.49:6834","nonce":3939647668}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":125},{"gid":11185,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/2890113513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2890113513},{"type":"v1","addr":"172.21.15.16:6836","nonce":2890113513}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":125}],"filesystems":[{"mdsmap":{"epoch":130,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:47:23.843333+0000","modified":"2023-10-16T13:47:26.851597+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":12252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12252":{"gid":12252,"name":"a","rank":0,"incarnation":126,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6837/303278795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":303278795},{"type":"v1","addr":"172.21.15.16:6837","nonce":303278795}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:47:41.332096+0000","modified":"2023-10-16T13:47:44.368830+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11197},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11197":{"gid":11197,"name":"d","rank":0,"incarnation":132,"state":"up:active","state_seq":7,"addr":"172.21.15.49:6835/680748068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":680748068},{"type":"v1","addr":"172.21.15.49:6835","nonce":680748068}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2023-10-16T13:48:04.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.262+0000 7fd6a375d700 1 -- 172.21.15.16:0/3258629479 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fd68006ad50 msgr2=0x7fd68006d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:04.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.262+0000 7fd6a375d700 1 --2- 172.21.15.16:0/3258629479 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fd68006ad50 0x7fd68006d210 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fd698007fd0 tx=0x7fd6980079a0 comp rx=0 tx=0).stop 2023-10-16T13:48:04.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.262+0000 7fd6a375d700 1 -- 172.21.15.16:0/3258629479 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd69c0ce4f0 msgr2=0x7fd69c15e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:04.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.262+0000 7fd6a375d700 1 --2- 172.21.15.16:0/3258629479 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd69c0ce4f0 0x7fd69c15e3a0 secure :-1 s=READY pgs=2901 cs=0 l=1 rev1=1 crypto rx=0x7fd68c009130 tx=0x7fd68c004750 comp rx=0 tx=0).stop 2023-10-16T13:48:04.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.262+0000 7fd6a375d700 1 -- 172.21.15.16:0/3258629479 shutdown_connections 2023-10-16T13:48:04.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.262+0000 7fd6a375d700 1 --2- 172.21.15.16:0/3258629479 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fd68006ad50 0x7fd68006d210 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:04.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.262+0000 7fd6a375d700 1 --2- 172.21.15.16:0/3258629479 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd69c15ba10 0x7fd69c0b22e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:04.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.262+0000 7fd6a375d700 1 --2- 172.21.15.16:0/3258629479 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd69c0ce4f0 0x7fd69c15e3a0 unknown :-1 s=CLOSED pgs=2901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:04.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.262+0000 7fd6a375d700 1 --2- 172.21.15.16:0/3258629479 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd69c15b630 0x7fd69c1569a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:04.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.262+0000 7fd6a375d700 1 -- 172.21.15.16:0/3258629479 >> 172.21.15.16:0/3258629479 conn(0x7fd69c0d14b0 msgr2=0x7fd69c0b5550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:04.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.262+0000 7fd6a375d700 1 -- 172.21.15.16:0/3258629479 shutdown_connections 2023-10-16T13:48:04.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.262+0000 7fd6a375d700 1 -- 172.21.15.16:0/3258629479 wait complete. 2023-10-16T13:48:04.268 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 135 2023-10-16T13:48:04.277 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:48:04.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.400+0000 7f4cbfaaf700 1 Processor -- start 2023-10-16T13:48:04.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.400+0000 7f4cbfaaf700 1 -- start start 2023-10-16T13:48:04.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.401+0000 7f4cbfaaf700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4cb80cc8a0 0x7f4cb815e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:04.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.401+0000 7f4cbfaaf700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4cb806a7e0 0x7f4cb806ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:04.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.401+0000 7f4cbfaaf700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cb8161c30 con 0x7f4cb815b280 2023-10-16T13:48:04.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.401+0000 7f4cbfaaf700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cb806b190 con 0x7f4cb80cc8a0 2023-10-16T13:48:04.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.401+0000 7f4cbfaaf700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb8161f10 con 0x7f4cb806a7e0 2023-10-16T13:48:04.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.401+0000 7f4cbd04a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4cb80cc8a0 0x7f4cb815e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:04.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.401+0000 7f4cbd84b700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f4cb815b280 0x7f4cb81613e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46184/0 (socket says 172.21.15.16:46184) 2023-10-16T13:48:04.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.401+0000 7f4cbd84b700 1 -- 172.21.15.16:0/2557246990 learned_addr learned my addr 172.21.15.16:0/2557246990 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:48:04.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.401+0000 7f4cbe04c700 1 --2- 172.21.15.16:0/2557246990 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4cb806a7e0 0x7f4cb806ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:04.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.401+0000 7f4cbc849700 1 -- 172.21.15.16:0/2557246990 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2631836102 0 0) 0x7f4cb8161c30 con 0x7f4cb815b280 2023-10-16T13:48:04.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.401+0000 7f4cbc849700 1 -- 172.21.15.16:0/2557246990 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ca40035b0 con 0x7f4cb815b280 2023-10-16T13:48:04.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.402+0000 7f4cbd04a700 1 -- 172.21.15.16:0/2557246990 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4cb806a7e0 msgr2=0x7f4cb806ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:04.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.402+0000 7f4cbd04a700 1 --2- 172.21.15.16:0/2557246990 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4cb806a7e0 0x7f4cb806ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:04.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.402+0000 7f4cbd04a700 1 -- 172.21.15.16:0/2557246990 >> v1:172.21.15.16:6789/0 conn(0x7f4cb815b280 legacy=0x7f4cb81613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:04.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.402+0000 7f4cbd04a700 1 -- 172.21.15.16:0/2557246990 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cb81621f0 con 0x7f4cb80cc8a0 2023-10-16T13:48:04.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.402+0000 7f4cbd04a700 1 --2- 172.21.15.16:0/2557246990 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cb80cc8a0 0x7f4cb815e700 secure :-1 s=READY pgs=2903 cs=0 l=1 rev1=1 crypto rx=0x7f4cb4009840 tx=0x7f4cb4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=670469610934caf4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:04.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.402+0000 7f4cbc849700 1 -- 172.21.15.16:0/2557246990 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb4011830 con 0x7f4cb80cc8a0 2023-10-16T13:48:04.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.402+0000 7f4cbc849700 1 -- 172.21.15.16:0/2557246990 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cb4011990 con 0x7f4cb80cc8a0 2023-10-16T13:48:04.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.402+0000 7f4cbc849700 1 -- 172.21.15.16:0/2557246990 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb4011d20 con 0x7f4cb80cc8a0 2023-10-16T13:48:04.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.402+0000 7f4cbfaaf700 1 -- 172.21.15.16:0/2557246990 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cb80cc8a0 msgr2=0x7f4cb815e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:04.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.402+0000 7f4cbfaaf700 1 --2- 172.21.15.16:0/2557246990 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cb80cc8a0 0x7f4cb815e700 secure :-1 s=READY pgs=2903 cs=0 l=1 rev1=1 crypto rx=0x7f4cb4009840 tx=0x7f4cb4009b50 comp rx=0 tx=0).stop 2023-10-16T13:48:04.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.402+0000 7f4cbfaaf700 1 -- 172.21.15.16:0/2557246990 shutdown_connections 2023-10-16T13:48:04.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.402+0000 7f4cbfaaf700 1 --2- 172.21.15.16:0/2557246990 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4cb806a7e0 0x7f4cb806ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:04.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.402+0000 7f4cbfaaf700 1 --2- 172.21.15.16:0/2557246990 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cb80cc8a0 0x7f4cb815e700 unknown :-1 s=CLOSED pgs=2903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:04.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.402+0000 7f4cbfaaf700 1 -- 172.21.15.16:0/2557246990 >> 172.21.15.16:0/2557246990 conn(0x7f4cb80b1fd0 msgr2=0x7f4cb80b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:04.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.403+0000 7f4cbfaaf700 1 -- 172.21.15.16:0/2557246990 shutdown_connections 2023-10-16T13:48:04.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.403+0000 7f4cbfaaf700 1 -- 172.21.15.16:0/2557246990 wait complete. 2023-10-16T13:48:04.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.404+0000 7f4cbfaaf700 1 Processor -- start 2023-10-16T13:48:04.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.404+0000 7f4cbfaaf700 1 -- start start 2023-10-16T13:48:04.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.404+0000 7f4cbfaaf700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cb806a7e0 0x7f4cb81f3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:04.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.404+0000 7f4cbfaaf700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4cb80cc8a0 0x7f4cb80c8720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:04.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.404+0000 7f4cbfaaf700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4cb815b280 0x7f4cb80c79b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:04.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.404+0000 7f4cbfaaf700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb81f3bc0 con 0x7f4cb806a7e0 2023-10-16T13:48:04.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.404+0000 7f4cbfaaf700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb80c9800 con 0x7f4cb80cc8a0 2023-10-16T13:48:04.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.404+0000 7f4cbfaaf700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb81f62e0 con 0x7f4cb815b280 2023-10-16T13:48:04.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.405+0000 7f4cbd84b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cb806a7e0 0x7f4cb81f3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:04.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.405+0000 7f4cbd84b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cb806a7e0 0x7f4cb81f3680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56916/0 (socket says 172.21.15.16:56916) 2023-10-16T13:48:04.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.405+0000 7f4cbd84b700 1 -- 172.21.15.16:0/1169117910 learned_addr learned my addr 172.21.15.16:0/1169117910 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:04.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.405+0000 7f4cbd04a700 1 --2- 172.21.15.16:0/1169117910 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4cb80cc8a0 0x7f4cb80c8720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:04.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.405+0000 7f4cbd84b700 1 -- 172.21.15.16:0/1169117910 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4cb815b280 msgr2=0x7f4cb80c79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:04.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.405+0000 7f4cbe04c700 1 --2- 172.21.15.16:0/1169117910 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4cb815b280 0x7f4cb80c79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:04.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.405+0000 7f4cbd84b700 1 --2- 172.21.15.16:0/1169117910 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4cb815b280 0x7f4cb80c79b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:04.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.405+0000 7f4cbd84b700 1 -- 172.21.15.16:0/1169117910 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4cb80cc8a0 msgr2=0x7f4cb80c8720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:04.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.405+0000 7f4cbd84b700 1 --2- 172.21.15.16:0/1169117910 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4cb80cc8a0 0x7f4cb80c8720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:04.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.405+0000 7f4cbd84b700 1 -- 172.21.15.16:0/1169117910 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cb40094a0 con 0x7f4cb806a7e0 2023-10-16T13:48:04.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.405+0000 7f4cbd84b700 1 --2- 172.21.15.16:0/1169117910 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cb806a7e0 0x7f4cb81f3680 secure :-1 s=READY pgs=2904 cs=0 l=1 rev1=1 crypto rx=0x7f4ca800c730 tx=0x7f4ca800caf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:04.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.405+0000 7f4cae7fc700 1 -- 172.21.15.16:0/1169117910 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ca8005d70 con 0x7f4cb806a7e0 2023-10-16T13:48:04.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.405+0000 7f4cae7fc700 1 -- 172.21.15.16:0/1169117910 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ca80070d0 con 0x7f4cb806a7e0 2023-10-16T13:48:04.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.405+0000 7f4cbfaaf700 1 -- 172.21.15.16:0/1169117910 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cb80c8c60 con 0x7f4cb806a7e0 2023-10-16T13:48:04.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.405+0000 7f4cae7fc700 1 -- 172.21.15.16:0/1169117910 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ca8007440 con 0x7f4cb806a7e0 2023-10-16T13:48:04.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.405+0000 7f4cbfaaf700 1 -- 172.21.15.16:0/1169117910 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cb80c9a50 con 0x7f4cb806a7e0 2023-10-16T13:48:04.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.406+0000 7f4cbfaaf700 1 -- 172.21.15.16:0/1169117910 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cb804ebb0 con 0x7f4cb806a7e0 2023-10-16T13:48:04.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.407+0000 7f4cae7fc700 1 -- 172.21.15.16:0/1169117910 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 155) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f4ca80075a0 con 0x7f4cb806a7e0 2023-10-16T13:48:04.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.407+0000 7f4cae7fc700 1 --2- 172.21.15.16:0/1169117910 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f4c9c06ad50 0x7f4c9c06d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:04.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.407+0000 7f4cae7fc700 1 -- 172.21.15.16:0/1169117910 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(251..251 src has 1..251) v4 ==== 7881+0+0 (secure 0 0 0) 0x7f4ca8088ef0 con 0x7f4cb806a7e0 2023-10-16T13:48:04.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.407+0000 7f4cbd04a700 1 --2- 172.21.15.16:0/1169117910 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f4c9c06ad50 0x7f4c9c06d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:04.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.407+0000 7f4cbd04a700 1 --2- 172.21.15.16:0/1169117910 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f4c9c06ad50 0x7f4c9c06d210 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f4cb4009840 tx=0x7f4cb4008030 comp rx=0 tx=0).ready entity=mgr.12678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:04.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.412+0000 7f4cae7fc700 1 -- 172.21.15.16:0/1169117910 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4ca8053b10 con 0x7f4cb806a7e0 2023-10-16T13:48:04.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.624+0000 7f4cbfaaf700 1 -- 172.21.15.16:0/1169117910 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4cb806ca50 con 0x7f4cb806a7e0 2023-10-16T13:48:04.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.625+0000 7f4cae7fc700 1 -- 172.21.15.16:0/1169117910 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v251) v1 ==== 74+0+21789 (secure 0 0 0) 0x7f4ca8017020 con 0x7f4cb806a7e0 2023-10-16T13:48:04.628 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:48:04.628 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":251,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:48:02.485657+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2023-10-16T13:47:22.338349+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"244","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2023-10-16T13:47:22.806327+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"245","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T13:47:39.075528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"248","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"backup_fs_data","create_time":"2023-10-16T13:47:40.314029+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"249","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6832/2221624769":"2023-10-17T13:48:02.485614+0000","172.21.15.16:0/2242986614":"2023-10-17T13:48:02.485614+0000","172.21.15.16:0/3610227254":"2023-10-17T13:48:02.485614+0000","172.21.15.16:0/1182888331":"2023-10-17T13:48:02.485614+0000","172.21.15.16:0/2095012141":"2023-10-17T13:48:02.485614+0000","172.21.15.16:0/874722077":"2023-10-17T13:48:02.485614+0000","172.21.15.16:0/2066401556":"2023-10-17T13:48:02.485614+0000","172.21.15.16:0/3947601914":"2023-10-17T13:47:54.816700+0000","172.21.15.16:6833/2221624769":"2023-10-17T13:48:02.485614+0000","172.21.15.16:0/2759470534":"2023-10-17T13:47:54.816700+0000","172.21.15.16:0/2634105683":"2023-10-17T13:48:02.485614+0000","172.21.15.16:0/2974892940":"2023-10-17T13:47:54.816700+0000","172.21.15.16:6833/1736104576":"2023-10-17T13:47:54.816700+0000","172.21.15.16:6832/1736104576":"2023-10-17T13:47:54.816700+0000","172.21.15.16:0/1246409088":"2023-10-17T13:48:02.485614+0000","172.21.15.16:0/318862437":"2023-10-17T13:47:54.816700+0000","172.21.15.16:0/413581668":"2023-10-17T13:47:54.816700+0000","172.21.15.16:0/3740515383":"2023-10-17T13:47:54.816700+0000","172.21.15.16:0/3105924907":"2023-10-17T13:47:54.816700+0000","172.21.15.16:0/3617558038":"2023-10-17T13:47:54.816700+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:48:04.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.627+0000 7f4cbfaaf700 1 -- 172.21.15.16:0/1169117910 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f4c9c06ad50 msgr2=0x7f4c9c06d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:04.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.627+0000 7f4cbfaaf700 1 --2- 172.21.15.16:0/1169117910 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f4c9c06ad50 0x7f4c9c06d210 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f4cb4009840 tx=0x7f4cb4008030 comp rx=0 tx=0).stop 2023-10-16T13:48:04.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.628+0000 7f4cbfaaf700 1 -- 172.21.15.16:0/1169117910 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cb806a7e0 msgr2=0x7f4cb81f3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:04.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.628+0000 7f4cbfaaf700 1 --2- 172.21.15.16:0/1169117910 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cb806a7e0 0x7f4cb81f3680 secure :-1 s=READY pgs=2904 cs=0 l=1 rev1=1 crypto rx=0x7f4ca800c730 tx=0x7f4ca800caf0 comp rx=0 tx=0).stop 2023-10-16T13:48:04.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.628+0000 7f4cbfaaf700 1 -- 172.21.15.16:0/1169117910 shutdown_connections 2023-10-16T13:48:04.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.628+0000 7f4cbfaaf700 1 --2- 172.21.15.16:0/1169117910 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f4c9c06ad50 0x7f4c9c06d210 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:04.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.628+0000 7f4cbfaaf700 1 --2- 172.21.15.16:0/1169117910 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4cb815b280 0x7f4cb80c79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:04.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.628+0000 7f4cbfaaf700 1 --2- 172.21.15.16:0/1169117910 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cb806a7e0 0x7f4cb81f3680 unknown :-1 s=CLOSED pgs=2904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:04.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.628+0000 7f4cbfaaf700 1 --2- 172.21.15.16:0/1169117910 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4cb80cc8a0 0x7f4cb80c8720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:04.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.628+0000 7f4cbfaaf700 1 -- 172.21.15.16:0/1169117910 >> 172.21.15.16:0/1169117910 conn(0x7f4cb80b1fd0 msgr2=0x7f4cb8071970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:04.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.628+0000 7f4cbfaaf700 1 -- 172.21.15.16:0/1169117910 shutdown_connections 2023-10-16T13:48:04.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.628+0000 7f4cbfaaf700 1 -- 172.21.15.16:0/1169117910 wait complete. 2023-10-16T13:48:04.643 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-16T13:48:04.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.766+0000 7f8cc35e1700 1 Processor -- start 2023-10-16T13:48:04.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.767+0000 7f8cc35e1700 1 -- start start 2023-10-16T13:48:04.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.767+0000 7f8cc35e1700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8cbc15b630 0x7f8cbc0d0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:04.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.767+0000 7f8cc35e1700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cbc0b0970 con 0x7f8cbc0ce4f0 2023-10-16T13:48:04.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.767+0000 7f8cc35e1700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cbc0cd690 con 0x7f8cbc15ba10 2023-10-16T13:48:04.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.767+0000 7f8cc35e1700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cbc0cd970 con 0x7f8cbc15b630 2023-10-16T13:48:04.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.767+0000 7f8cc137d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8cbc15b630 0x7f8cbc0d0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:04.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.767+0000 7f8cc1b7e700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f8cbc0ce4f0 0x7f8cbc0b1430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46216/0 (socket says 172.21.15.16:46216) 2023-10-16T13:48:04.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.767+0000 7f8cc1b7e700 1 -- 172.21.15.16:0/3282508631 learned_addr learned my addr 172.21.15.16:0/3282508631 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:48:04.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.768+0000 7f8cc137d700 1 -- 172.21.15.16:0/3282508631 >> v1:172.21.15.49:6789/0 conn(0x7f8cbc15ba10 legacy=0x7f8cbc0cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:04.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.768+0000 7f8cc137d700 1 -- 172.21.15.16:0/3282508631 >> v1:172.21.15.16:6789/0 conn(0x7f8cbc0ce4f0 legacy=0x7f8cbc0b1430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:04.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.768+0000 7f8cc137d700 1 -- 172.21.15.16:0/3282508631 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cbc0b0c30 con 0x7f8cbc15b630 2023-10-16T13:48:04.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.768+0000 7f8cb3fff700 1 -- 172.21.15.16:0/3282508631 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1150311099 0 0) 0x7f8cbc0b0970 con 0x7f8cbc0ce4f0 2023-10-16T13:48:04.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.768+0000 7f8cc137d700 1 --2- 172.21.15.16:0/3282508631 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cbc15b630 0x7f8cbc0d0380 secure :-1 s=READY pgs=2908 cs=0 l=1 rev1=1 crypto rx=0x7f8cac009840 tx=0x7f8cac009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4a7ca1034e9f2d5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:04.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.768+0000 7f8cb3fff700 1 -- 172.21.15.16:0/3282508631 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cac011830 con 0x7f8cbc15b630 2023-10-16T13:48:04.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.768+0000 7f8cb3fff700 1 -- 172.21.15.16:0/3282508631 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cac011990 con 0x7f8cbc15b630 2023-10-16T13:48:04.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.768+0000 7f8cb3fff700 1 -- 172.21.15.16:0/3282508631 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cac011d20 con 0x7f8cbc15b630 2023-10-16T13:48:04.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.769+0000 7f8cc35e1700 1 -- 172.21.15.16:0/3282508631 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cbc15b630 msgr2=0x7f8cbc0d0380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:04.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.769+0000 7f8cc35e1700 1 --2- 172.21.15.16:0/3282508631 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cbc15b630 0x7f8cbc0d0380 secure :-1 s=READY pgs=2908 cs=0 l=1 rev1=1 crypto rx=0x7f8cac009840 tx=0x7f8cac009b50 comp rx=0 tx=0).stop 2023-10-16T13:48:04.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.769+0000 7f8cc35e1700 1 -- 172.21.15.16:0/3282508631 shutdown_connections 2023-10-16T13:48:04.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.769+0000 7f8cc35e1700 1 --2- 172.21.15.16:0/3282508631 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cbc15b630 0x7f8cbc0d0380 unknown :-1 s=CLOSED pgs=2908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:04.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.769+0000 7f8cc35e1700 1 -- 172.21.15.16:0/3282508631 >> 172.21.15.16:0/3282508631 conn(0x7f8cbc0d14b0 msgr2=0x7f8cbc0d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:04.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.769+0000 7f8cc35e1700 1 -- 172.21.15.16:0/3282508631 shutdown_connections 2023-10-16T13:48:04.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.769+0000 7f8cc35e1700 1 -- 172.21.15.16:0/3282508631 wait complete. 2023-10-16T13:48:04.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.770+0000 7f8cc35e1700 1 Processor -- start 2023-10-16T13:48:04.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.770+0000 7f8cc35e1700 1 -- start start 2023-10-16T13:48:04.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.771+0000 7f8cc35e1700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8cbc0ce4f0 0x7f8cbc150780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:04.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.771+0000 7f8cc35e1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cbc15b630 0x7f8cbc0c6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:04.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.771+0000 7f8cc35e1700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8cbc15ba10 0x7f8cbc0b2c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:04.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.771+0000 7f8cc35e1700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cbc0b4d60 con 0x7f8cbc15b630 2023-10-16T13:48:04.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.771+0000 7f8cc35e1700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cbc0c7430 con 0x7f8cbc15ba10 2023-10-16T13:48:04.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.771+0000 7f8cc35e1700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cbc150cc0 con 0x7f8cbc0ce4f0 2023-10-16T13:48:04.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.771+0000 7f8cc0b7c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cbc15b630 0x7f8cbc0c6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:04.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.771+0000 7f8cc0b7c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cbc15b630 0x7f8cbc0c6ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56948/0 (socket says 172.21.15.16:56948) 2023-10-16T13:48:04.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.771+0000 7f8cc0b7c700 1 -- 172.21.15.16:0/908544269 learned_addr learned my addr 172.21.15.16:0/908544269 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:04.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.771+0000 7f8cc1b7e700 1 --2- 172.21.15.16:0/908544269 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8cbc15ba10 0x7f8cbc0b2c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:04.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.771+0000 7f8cc137d700 1 --2- 172.21.15.16:0/908544269 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8cbc0ce4f0 0x7f8cbc150780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:04.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.771+0000 7f8cc0b7c700 1 -- 172.21.15.16:0/908544269 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8cbc0ce4f0 msgr2=0x7f8cbc150780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:04.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.771+0000 7f8cc0b7c700 1 --2- 172.21.15.16:0/908544269 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8cbc0ce4f0 0x7f8cbc150780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:04.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.771+0000 7f8cc0b7c700 1 -- 172.21.15.16:0/908544269 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8cbc15ba10 msgr2=0x7f8cbc0b2c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:04.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.771+0000 7f8cc0b7c700 1 --2- 172.21.15.16:0/908544269 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8cbc15ba10 0x7f8cbc0b2c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:04.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.771+0000 7f8cc0b7c700 1 -- 172.21.15.16:0/908544269 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cac0094a0 con 0x7f8cbc15b630 2023-10-16T13:48:04.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.771+0000 7f8cc1b7e700 1 --2- 172.21.15.16:0/908544269 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8cbc15ba10 0x7f8cbc0b2c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:48:04.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.771+0000 7f8cc0b7c700 1 --2- 172.21.15.16:0/908544269 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cbc15b630 0x7f8cbc0c6ae0 secure :-1 s=READY pgs=2909 cs=0 l=1 rev1=1 crypto rx=0x7f8cb800c450 tx=0x7f8cb800c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:04.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.771+0000 7f8cb1ffb700 1 -- 172.21.15.16:0/908544269 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cb80052d0 con 0x7f8cbc15b630 2023-10-16T13:48:04.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.771+0000 7f8cb1ffb700 1 -- 172.21.15.16:0/908544269 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cb8005430 con 0x7f8cbc15b630 2023-10-16T13:48:04.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.772+0000 7f8cc35e1700 1 -- 172.21.15.16:0/908544269 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cbc0b4430 con 0x7f8cbc15b630 2023-10-16T13:48:04.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.772+0000 7f8cb1ffb700 1 -- 172.21.15.16:0/908544269 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cb80038c0 con 0x7f8cbc15b630 2023-10-16T13:48:04.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.772+0000 7f8cc35e1700 1 -- 172.21.15.16:0/908544269 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cbc0b49f0 con 0x7f8cbc15b630 2023-10-16T13:48:04.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.773+0000 7f8cc35e1700 1 -- 172.21.15.16:0/908544269 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cbc04ebb0 con 0x7f8cbc15b630 2023-10-16T13:48:04.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.773+0000 7f8cb1ffb700 1 -- 172.21.15.16:0/908544269 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 155) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f8cb8003a20 con 0x7f8cbc15b630 2023-10-16T13:48:04.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.773+0000 7f8cb1ffb700 1 --2- 172.21.15.16:0/908544269 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f8ca006ad50 0x7f8ca006d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:04.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.773+0000 7f8cb1ffb700 1 -- 172.21.15.16:0/908544269 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(251..251 src has 1..251) v4 ==== 7881+0+0 (secure 0 0 0) 0x7f8cb8088e80 con 0x7f8cbc15b630 2023-10-16T13:48:04.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.774+0000 7f8cc137d700 1 --2- 172.21.15.16:0/908544269 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f8ca006ad50 0x7f8ca006d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:04.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.774+0000 7f8cc137d700 1 --2- 172.21.15.16:0/908544269 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f8ca006ad50 0x7f8ca006d210 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f8cbc0b7300 tx=0x7f8cac008030 comp rx=0 tx=0).ready entity=mgr.12678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:04.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:04.778+0000 7f8cb1ffb700 1 -- 172.21.15.16:0/908544269 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8cb80539f0 con 0x7f8cbc15b630 2023-10-16T13:48:05.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:05.032+0000 7f8cc35e1700 1 -- 172.21.15.16:0/908544269 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f8cbc0b7330 con 0x7f8cbc15b630 2023-10-16T13:48:05.035 INFO:tasks.ceph.mon.a.smithi016.stderr:2023-10-16T13:48:05.033+0000 7f9fc8535700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-16T13:48:05.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:05.034+0000 7f8cb1ffb700 1 -- 172.21.15.16:0/908544269 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v136) v1 ==== 161+0+0 (secure 0 0 0) 0x7f8cb8017020 con 0x7f8cbc15b630 2023-10-16T13:48:05.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:05.038+0000 7f8cc35e1700 1 -- 172.21.15.16:0/908544269 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f8ca006ad50 msgr2=0x7f8ca006d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:05.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:05.038+0000 7f8cc35e1700 1 --2- 172.21.15.16:0/908544269 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f8ca006ad50 0x7f8ca006d210 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f8cbc0b7300 tx=0x7f8cac008030 comp rx=0 tx=0).stop 2023-10-16T13:48:05.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:05.038+0000 7f8cc35e1700 1 -- 172.21.15.16:0/908544269 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cbc15b630 msgr2=0x7f8cbc0c6ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:05.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:05.038+0000 7f8cc35e1700 1 --2- 172.21.15.16:0/908544269 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cbc15b630 0x7f8cbc0c6ae0 secure :-1 s=READY pgs=2909 cs=0 l=1 rev1=1 crypto rx=0x7f8cb800c450 tx=0x7f8cb800c810 comp rx=0 tx=0).stop 2023-10-16T13:48:05.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:05.039+0000 7f8cc35e1700 1 -- 172.21.15.16:0/908544269 shutdown_connections 2023-10-16T13:48:05.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:05.039+0000 7f8cc35e1700 1 --2- 172.21.15.16:0/908544269 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f8ca006ad50 0x7f8ca006d210 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:05.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:05.039+0000 7f8cc35e1700 1 --2- 172.21.15.16:0/908544269 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8cbc15ba10 0x7f8cbc0b2c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:05.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:05.039+0000 7f8cc35e1700 1 --2- 172.21.15.16:0/908544269 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8cbc0ce4f0 0x7f8cbc150780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:05.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:05.039+0000 7f8cc35e1700 1 --2- 172.21.15.16:0/908544269 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cbc15b630 0x7f8cbc0c6ae0 unknown :-1 s=CLOSED pgs=2909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:05.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:05.039+0000 7f8cc35e1700 1 -- 172.21.15.16:0/908544269 >> 172.21.15.16:0/908544269 conn(0x7f8cbc0d14b0 msgr2=0x7f8cbc0d1890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:05.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:05.039+0000 7f8cc35e1700 1 -- 172.21.15.16:0/908544269 shutdown_connections 2023-10-16T13:48:05.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:05.039+0000 7f8cc35e1700 1 -- 172.21.15.16:0/908544269 wait complete. 2023-10-16T13:48:05.043 INFO:teuthology.orchestra.run.smithi016.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-16T13:48:05.044 INFO:tasks.ceph.mds.a.smithi016.stderr: -5443> 2023-10-16T13:47:30.403+0000 7f99d9b55700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T13:48:05.054 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-16T13:48:05.074 INFO:tasks.ceph.mds.a.smithi016.stdout:starting mds.a at 2023-10-16T13:48:05.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:05.181+0000 7f787e90f700 1 Processor -- start 2023-10-16T13:48:05.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:05.181+0000 7f787e90f700 1 -- start start 2023-10-16T13:48:05.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:05.182+0000 7f787e90f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f787806a7e0 0x7f787806ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:05.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:05.182+0000 7f787e90f700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f787815ecb0 con 0x7f787815b2c0 2023-10-16T13:48:05.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:05.182+0000 7f787e90f700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f787815b6a0 con 0x7f787806e570 2023-10-16T13:48:05.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:05.182+0000 7f787e90f700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f787806b190 con 0x7f787806a7e0 2023-10-16T13:48:05.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:05.182+0000 7f787ceac700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f787806a7e0 0x7f787806ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:05.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:05.182+0000 7f7877fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f787815b2c0 0x7f7878161340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46240/0 (socket says 172.21.15.16:46240) 2023-10-16T13:48:05.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:05.182+0000 7f7877fff700 1 -- 172.21.15.16:0/1438406300 learned_addr learned my addr 172.21.15.16:0/1438406300 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:48:05.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:05.182+0000 7f787ceac700 1 -- 172.21.15.16:0/1438406300 >> v1:172.21.15.49:6789/0 conn(0x7f787806e570 legacy=0x7f787815e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:05.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:05.182+0000 7f787ceac700 1 -- 172.21.15.16:0/1438406300 >> v1:172.21.15.16:6789/0 conn(0x7f787815b2c0 legacy=0x7f7878161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:05.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:05.182+0000 7f787ceac700 1 -- 172.21.15.16:0/1438406300 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7878161b20 con 0x7f787806a7e0 2023-10-16T13:48:05.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:05.182+0000 7f7876ffd700 1 -- 172.21.15.16:0/1438406300 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1386426335 0 0) 0x7f787815ecb0 con 0x7f787815b2c0 2023-10-16T13:48:05.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:05.182+0000 7f787ceac700 1 --2- 172.21.15.16:0/1438406300 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f787806a7e0 0x7f787806ac50 secure :-1 s=READY pgs=2914 cs=0 l=1 rev1=1 crypto rx=0x7f7870009d90 tx=0x7f787000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39d114f2b42925d9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:05.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:05.183+0000 7f7876ffd700 1 -- 172.21.15.16:0/1438406300 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f787000e070 con 0x7f787806a7e0 2023-10-16T13:48:05.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:05.183+0000 7f7876ffd700 1 -- 172.21.15.16:0/1438406300 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f787000bbd0 con 0x7f787806a7e0 2023-10-16T13:48:05.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:05.183+0000 7f7876ffd700 1 -- 172.21.15.16:0/1438406300 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7870012430 con 0x7f787806a7e0 2023-10-16T13:48:05.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:05.183+0000 7f787e90f700 1 -- 172.21.15.16:0/1438406300 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f787806a7e0 msgr2=0x7f787806ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:05.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:05.183+0000 7f787e90f700 1 --2- 172.21.15.16:0/1438406300 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f787806a7e0 0x7f787806ac50 secure :-1 s=READY pgs=2914 cs=0 l=1 rev1=1 crypto rx=0x7f7870009d90 tx=0x7f787000b330 comp rx=0 tx=0).stop 2023-10-16T13:48:05.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:05.183+0000 7f787e90f700 1 -- 172.21.15.16:0/1438406300 shutdown_connections 2023-10-16T13:48:05.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:05.183+0000 7f787e90f700 1 --2- 172.21.15.16:0/1438406300 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f787806a7e0 0x7f787806ac50 unknown :-1 s=CLOSED pgs=2914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:05.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:05.183+0000 7f787e90f700 1 -- 172.21.15.16:0/1438406300 >> 172.21.15.16:0/1438406300 conn(0x7f78780b1f40 msgr2=0x7f78780b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:05.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:05.183+0000 7f787e90f700 1 -- 172.21.15.16:0/1438406300 shutdown_connections 2023-10-16T13:48:05.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:05.183+0000 7f787e90f700 1 -- 172.21.15.16:0/1438406300 wait complete. 2023-10-16T13:48:05.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:05.185+0000 7f787e90f700 1 Processor -- start 2023-10-16T13:48:05.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:05.185+0000 7f787e90f700 1 -- start start 2023-10-16T13:48:05.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:05.186+0000 7f787e90f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f787806a7e0 0x7f78780c7dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:05.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:05.186+0000 7f787e90f700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f787806e570 0x7f7878156eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:05.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:05.186+0000 7f787e90f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f787815b2c0 0x7f7878159100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:05.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:05.186+0000 7f787e90f700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78780ca630 con 0x7f787806a7e0 2023-10-16T13:48:05.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:05.186+0000 7f787e90f700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78780c8310 con 0x7f787806e570 2023-10-16T13:48:05.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:05.186+0000 7f787e90f700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78780c8590 con 0x7f787815b2c0 2023-10-16T13:48:05.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:05.186+0000 7f7877fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f787806a7e0 0x7f78780c7dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:05.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:05.186+0000 7f7877fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f787806a7e0 0x7f78780c7dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56972/0 (socket says 172.21.15.16:56972) 2023-10-16T13:48:05.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:05.186+0000 7f7877fff700 1 -- 172.21.15.16:0/2106335748 learned_addr learned my addr 172.21.15.16:0/2106335748 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:05.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:05.186+0000 7f78777fe700 1 --2- 172.21.15.16:0/2106335748 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f787806e570 0x7f7878156eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:05.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:05.186+0000 7f787ceac700 1 --2- 172.21.15.16:0/2106335748 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f787815b2c0 0x7f7878159100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:05.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:05.186+0000 7f7877fff700 1 -- 172.21.15.16:0/2106335748 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f787815b2c0 msgr2=0x7f7878159100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:05.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:05.186+0000 7f7877fff700 1 --2- 172.21.15.16:0/2106335748 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f787815b2c0 0x7f7878159100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:05.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:05.186+0000 7f7877fff700 1 -- 172.21.15.16:0/2106335748 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f787806e570 msgr2=0x7f7878156eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:05.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:05.186+0000 7f7877fff700 1 --2- 172.21.15.16:0/2106335748 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f787806e570 0x7f7878156eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:05.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:05.186+0000 7f7877fff700 1 -- 172.21.15.16:0/2106335748 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7870009a70 con 0x7f787806a7e0 2023-10-16T13:48:05.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:05.186+0000 7f78777fe700 1 --2- 172.21.15.16:0/2106335748 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f787806e570 0x7f7878156eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:48:05.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:05.187+0000 7f7877fff700 1 --2- 172.21.15.16:0/2106335748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f787806a7e0 0x7f78780c7dd0 secure :-1 s=READY pgs=2915 cs=0 l=1 rev1=1 crypto rx=0x7f786800c520 tx=0x7f786800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:05.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:05.187+0000 7f7874ff9700 1 -- 172.21.15.16:0/2106335748 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7868004c80 con 0x7f787806a7e0 2023-10-16T13:48:05.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:05.187+0000 7f7874ff9700 1 -- 172.21.15.16:0/2106335748 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7868004e00 con 0x7f787806a7e0 2023-10-16T13:48:05.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:05.187+0000 7f7874ff9700 1 -- 172.21.15.16:0/2106335748 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7868007310 con 0x7f787806a7e0 2023-10-16T13:48:05.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:05.187+0000 7f787e90f700 1 -- 172.21.15.16:0/2106335748 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78780c9f70 con 0x7f787806a7e0 2023-10-16T13:48:05.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:05.187+0000 7f787e90f700 1 -- 172.21.15.16:0/2106335748 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78781573f0 con 0x7f787806a7e0 2023-10-16T13:48:05.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:05.188+0000 7f787e90f700 1 -- 172.21.15.16:0/2106335748 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78780735f0 con 0x7f787806a7e0 2023-10-16T13:48:05.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:05.189+0000 7f7874ff9700 1 -- 172.21.15.16:0/2106335748 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 155) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f7868007470 con 0x7f787806a7e0 2023-10-16T13:48:05.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:05.189+0000 7f7874ff9700 1 --2- 172.21.15.16:0/2106335748 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f786406ad50 0x7f786406d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:05.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:05.189+0000 7f7874ff9700 1 -- 172.21.15.16:0/2106335748 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 7967+0+0 (secure 0 0 0) 0x7f7868089810 con 0x7f787806a7e0 2023-10-16T13:48:05.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:05.189+0000 7f78777fe700 1 --2- 172.21.15.16:0/2106335748 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f786406ad50 0x7f786406d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:05.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:05.190+0000 7f78777fe700 1 --2- 172.21.15.16:0/2106335748 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f786406ad50 0x7f786406d210 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f78780caeb0 tx=0x7f786c009040 comp rx=0 tx=0).ready entity=mgr.12678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:05.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:05.194+0000 7f7874ff9700 1 -- 172.21.15.16:0/2106335748 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7868054330 con 0x7f787806a7e0 2023-10-16T13:48:05.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:05.443+0000 7f787e90f700 1 -- 172.21.15.16:0/2106335748 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f787804ebb0 con 0x7f787806a7e0 2023-10-16T13:48:05.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:05.500+0000 7f7874ff9700 1 -- 172.21.15.16:0/2106335748 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 156) v1 ==== 89377+0+0 (secure 0 0 0) 0x7f786805e0d0 con 0x7f787806a7e0 2023-10-16T13:48:06.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:06.036+0000 7f7874ff9700 1 -- 172.21.15.16:0/2106335748 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v137) v1 ==== 104+0+0 (secure 0 0 0) 0x7f7868073940 con 0x7f787806a7e0 2023-10-16T13:48:06.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:06.039+0000 7f787e90f700 1 -- 172.21.15.16:0/2106335748 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f786406ad50 msgr2=0x7f786406d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:06.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:06.039+0000 7f787e90f700 1 --2- 172.21.15.16:0/2106335748 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f786406ad50 0x7f786406d210 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f78780caeb0 tx=0x7f786c009040 comp rx=0 tx=0).stop 2023-10-16T13:48:06.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:06.039+0000 7f787e90f700 1 -- 172.21.15.16:0/2106335748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f787806a7e0 msgr2=0x7f78780c7dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:06.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:06.039+0000 7f787e90f700 1 --2- 172.21.15.16:0/2106335748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f787806a7e0 0x7f78780c7dd0 secure :-1 s=READY pgs=2915 cs=0 l=1 rev1=1 crypto rx=0x7f786800c520 tx=0x7f786800c8e0 comp rx=0 tx=0).stop 2023-10-16T13:48:06.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:06.039+0000 7f787e90f700 1 -- 172.21.15.16:0/2106335748 shutdown_connections 2023-10-16T13:48:06.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:06.039+0000 7f787e90f700 1 --2- 172.21.15.16:0/2106335748 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f786406ad50 0x7f786406d210 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:06.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:06.039+0000 7f787e90f700 1 --2- 172.21.15.16:0/2106335748 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f787815b2c0 0x7f7878159100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:06.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:06.039+0000 7f787e90f700 1 --2- 172.21.15.16:0/2106335748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f787806a7e0 0x7f78780c7dd0 unknown :-1 s=CLOSED pgs=2915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:06.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:06.039+0000 7f787e90f700 1 --2- 172.21.15.16:0/2106335748 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f787806e570 0x7f7878156eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:06.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:06.039+0000 7f787e90f700 1 -- 172.21.15.16:0/2106335748 >> 172.21.15.16:0/2106335748 conn(0x7f78780b1f40 msgr2=0x7f78780cb120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:06.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:06.039+0000 7f787e90f700 1 -- 172.21.15.16:0/2106335748 shutdown_connections 2023-10-16T13:48:06.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:06.039+0000 7f787e90f700 1 -- 172.21.15.16:0/2106335748 wait complete. 2023-10-16T13:48:06.058 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-16T13:48:06.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:06.234+0000 7fa74e00e700 1 Processor -- start 2023-10-16T13:48:06.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:06.234+0000 7fa74e00e700 1 -- start start 2023-10-16T13:48:06.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:06.235+0000 7fa74e00e700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fa748155a60 0x7fa74815fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:06.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:06.235+0000 7fa74e00e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa748155e40 0x7fa74815bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:06.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:06.235+0000 7fa74e00e700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa748160190 con 0x7fa74815c190 2023-10-16T13:48:06.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:06.235+0000 7fa74e00e700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa74815f240 con 0x7fa748155e40 2023-10-16T13:48:06.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:06.235+0000 7fa74e00e700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa74815c780 con 0x7fa748155a60 2023-10-16T13:48:06.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:06.235+0000 7fa746ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa748155e40 0x7fa74815bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:06.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:06.235+0000 7fa746ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa748155e40 0x7fa74815bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56982/0 (socket says 172.21.15.16:56982) 2023-10-16T13:48:06.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:06.235+0000 7fa746ffd700 1 -- 172.21.15.16:0/3513625006 learned_addr learned my addr 172.21.15.16:0/3513625006 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:06.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:06.235+0000 7fa747fff700 1 --1- 172.21.15.16:0/3513625006 >> v1:172.21.15.16:6789/0 conn(0x7fa74815c190 0x7fa7480cba40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46258/0 (socket says 172.21.15.16:46258) 2023-10-16T13:48:06.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:06.235+0000 7fa7477fe700 1 --2- 172.21.15.16:0/3513625006 >> v2:172.21.15.49:3300/0 conn(0x7fa748155a60 0x7fa74815fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:06.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:06.235+0000 7fa746ffd700 1 -- 172.21.15.16:0/3513625006 >> v2:172.21.15.49:3300/0 conn(0x7fa748155a60 msgr2=0x7fa74815fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:06.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:06.236+0000 7fa746ffd700 1 --2- 172.21.15.16:0/3513625006 >> v2:172.21.15.49:3300/0 conn(0x7fa748155a60 0x7fa74815fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:06.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:06.236+0000 7fa746ffd700 1 -- 172.21.15.16:0/3513625006 >> v1:172.21.15.16:6789/0 conn(0x7fa74815c190 legacy=0x7fa7480cba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:06.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:06.236+0000 7fa746ffd700 1 -- 172.21.15.16:0/3513625006 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa74815ca60 con 0x7fa748155e40 2023-10-16T13:48:06.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:06.236+0000 7fa7467fc700 1 -- 172.21.15.16:0/3513625006 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3167419522 0 0) 0x7fa748160190 con 0x7fa74815c190 2023-10-16T13:48:06.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:06.236+0000 7fa746ffd700 1 --2- 172.21.15.16:0/3513625006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa748155e40 0x7fa74815bc50 secure :-1 s=READY pgs=2917 cs=0 l=1 rev1=1 crypto rx=0x7fa7480d24f0 tx=0x7fa738009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=907b18b5413c64b6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:06.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:06.236+0000 7fa7467fc700 1 -- 172.21.15.16:0/3513625006 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa738011830 con 0x7fa748155e40 2023-10-16T13:48:06.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:06.236+0000 7fa7467fc700 1 -- 172.21.15.16:0/3513625006 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa738011990 con 0x7fa748155e40 2023-10-16T13:48:06.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:06.236+0000 7fa7467fc700 1 -- 172.21.15.16:0/3513625006 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa738011d20 con 0x7fa748155e40 2023-10-16T13:48:06.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:06.236+0000 7fa74e00e700 1 -- 172.21.15.16:0/3513625006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa748155e40 msgr2=0x7fa74815bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:06.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:06.236+0000 7fa74e00e700 1 --2- 172.21.15.16:0/3513625006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa748155e40 0x7fa74815bc50 secure :-1 s=READY pgs=2917 cs=0 l=1 rev1=1 crypto rx=0x7fa7480d24f0 tx=0x7fa738009bd0 comp rx=0 tx=0).stop 2023-10-16T13:48:06.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:06.236+0000 7fa74e00e700 1 -- 172.21.15.16:0/3513625006 shutdown_connections 2023-10-16T13:48:06.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:06.236+0000 7fa74e00e700 1 --2- 172.21.15.16:0/3513625006 >> v2:172.21.15.49:3300/0 conn(0x7fa748155a60 0x7fa74815fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:06.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:06.236+0000 7fa74e00e700 1 --2- 172.21.15.16:0/3513625006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa748155e40 0x7fa74815bc50 unknown :-1 s=CLOSED pgs=2917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:06.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:06.236+0000 7fa74e00e700 1 -- 172.21.15.16:0/3513625006 >> 172.21.15.16:0/3513625006 conn(0x7fa74806a7e0 msgr2=0x7fa74806abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:06.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:06.237+0000 7fa74e00e700 1 -- 172.21.15.16:0/3513625006 shutdown_connections 2023-10-16T13:48:06.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:06.237+0000 7fa74e00e700 1 -- 172.21.15.16:0/3513625006 wait complete. 2023-10-16T13:48:06.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:06.238+0000 7fa74e00e700 1 Processor -- start 2023-10-16T13:48:06.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:06.238+0000 7fa74e00e700 1 -- start start 2023-10-16T13:48:06.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:06.239+0000 7fa74e00e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa748155a60 0x7fa74814e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:06.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:06.239+0000 7fa74e00e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa748155e40 0x7fa7480aff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:06.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:06.239+0000 7fa74e00e700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa74815c190 0x7fa748154030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:06.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:06.239+0000 7fa74e00e700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7480ac8d0 con 0x7fa748155a60 2023-10-16T13:48:06.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:06.239+0000 7fa74e00e700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7480b6710 con 0x7fa74815c190 2023-10-16T13:48:06.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:06.239+0000 7fa74e00e700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7480b69f0 con 0x7fa748155e40 2023-10-16T13:48:06.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:06.239+0000 7fa7477fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa748155a60 0x7fa74814e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:06.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:06.239+0000 7fa7477fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa748155a60 0x7fa74814e7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56986/0 (socket says 172.21.15.16:56986) 2023-10-16T13:48:06.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:06.239+0000 7fa7477fe700 1 -- 172.21.15.16:0/852713627 learned_addr learned my addr 172.21.15.16:0/852713627 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:06.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:06.239+0000 7fa7477fe700 1 -- 172.21.15.16:0/852713627 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa748155e40 msgr2=0x7fa7480aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:06.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:06.239+0000 7fa747fff700 1 --2- 172.21.15.16:0/852713627 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa74815c190 0x7fa748154030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:06.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:06.239+0000 7fa746ffd700 1 --2- 172.21.15.16:0/852713627 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa748155e40 0x7fa7480aff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:06.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:06.239+0000 7fa7477fe700 1 --2- 172.21.15.16:0/852713627 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa748155e40 0x7fa7480aff60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:06.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:06.239+0000 7fa7477fe700 1 -- 172.21.15.16:0/852713627 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa74815c190 msgr2=0x7fa748154030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:06.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:06.239+0000 7fa7477fe700 1 --2- 172.21.15.16:0/852713627 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa74815c190 0x7fa748154030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:06.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:06.239+0000 7fa7477fe700 1 -- 172.21.15.16:0/852713627 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa738009550 con 0x7fa748155a60 2023-10-16T13:48:06.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:06.239+0000 7fa747fff700 1 --2- 172.21.15.16:0/852713627 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa74815c190 0x7fa748154030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:48:06.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:06.239+0000 7fa7477fe700 1 --2- 172.21.15.16:0/852713627 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa748155a60 0x7fa74814e7c0 secure :-1 s=READY pgs=2918 cs=0 l=1 rev1=1 crypto rx=0x7fa73000e9b0 tx=0x7fa73000ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:06.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:06.239+0000 7fa73ffff700 1 -- 172.21.15.16:0/852713627 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa730009710 con 0x7fa748155a60 2023-10-16T13:48:06.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:06.239+0000 7fa73ffff700 1 -- 172.21.15.16:0/852713627 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa73000c850 con 0x7fa748155a60 2023-10-16T13:48:06.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:06.239+0000 7fa74e00e700 1 -- 172.21.15.16:0/852713627 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7480b6cd0 con 0x7fa748155a60 2023-10-16T13:48:06.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:06.239+0000 7fa73ffff700 1 -- 172.21.15.16:0/852713627 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa73000cbc0 con 0x7fa748155a60 2023-10-16T13:48:06.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:06.240+0000 7fa74e00e700 1 -- 172.21.15.16:0/852713627 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa74814ed00 con 0x7fa748155a60 2023-10-16T13:48:06.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:06.240+0000 7fa74e00e700 1 -- 172.21.15.16:0/852713627 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7480acee0 con 0x7fa748155a60 2023-10-16T13:48:06.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:06.241+0000 7fa73ffff700 1 -- 172.21.15.16:0/852713627 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 156) v1 ==== 89377+0+0 (secure 0 0 0) 0x7fa73000cd20 con 0x7fa748155a60 2023-10-16T13:48:06.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:06.241+0000 7fa73ffff700 1 --2- 172.21.15.16:0/852713627 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fa72806b0b0 0x7fa72806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:06.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:06.242+0000 7fa73ffff700 1 -- 172.21.15.16:0/852713627 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 7967+0+0 (secure 0 0 0) 0x7fa73008a320 con 0x7fa748155a60 2023-10-16T13:48:06.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:06.242+0000 7fa746ffd700 1 --2- 172.21.15.16:0/852713627 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fa72806b0b0 0x7fa72806d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:06.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:06.242+0000 7fa746ffd700 1 --2- 172.21.15.16:0/852713627 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fa72806b0b0 0x7fa72806d570 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa748154e50 tx=0x7fa738008030 comp rx=0 tx=0).ready entity=mgr.12678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:06.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:06.246+0000 7fa73ffff700 1 -- 172.21.15.16:0/852713627 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa730054880 con 0x7fa748155a60 2023-10-16T13:48:06.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:06.449+0000 7fa74e00e700 1 -- 172.21.15.16:0/852713627 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fa7480678b0 con 0x7fa748155a60 2023-10-16T13:48:07.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:07.040+0000 7fa73ffff700 1 -- 172.21.15.16:0/852713627 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v253) v1 ==== 181+0+0 (secure 0 0 0) 0x7fa7300210d0 con 0x7fa748155a60 2023-10-16T13:48:07.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:07.050+0000 7fa74e00e700 1 -- 172.21.15.16:0/852713627 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fa72806b0b0 msgr2=0x7fa72806d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:07.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:07.050+0000 7fa74e00e700 1 --2- 172.21.15.16:0/852713627 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fa72806b0b0 0x7fa72806d570 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa748154e50 tx=0x7fa738008030 comp rx=0 tx=0).stop 2023-10-16T13:48:07.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:07.050+0000 7fa74e00e700 1 -- 172.21.15.16:0/852713627 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa748155a60 msgr2=0x7fa74814e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:07.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:07.051+0000 7fa73ffff700 1 -- 172.21.15.16:0/852713627 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fa730054c50 con 0x7fa748155a60 2023-10-16T13:48:07.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:07.051+0000 7fa74e00e700 1 --2- 172.21.15.16:0/852713627 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa748155a60 0x7fa74814e7c0 secure :-1 s=READY pgs=2918 cs=0 l=1 rev1=1 crypto rx=0x7fa73000e9b0 tx=0x7fa73000ed70 comp rx=0 tx=0).stop 2023-10-16T13:48:07.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:07.051+0000 7fa74e00e700 1 -- 172.21.15.16:0/852713627 shutdown_connections 2023-10-16T13:48:07.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:07.051+0000 7fa74e00e700 1 --2- 172.21.15.16:0/852713627 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fa72806b0b0 0x7fa72806d570 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:07.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:07.051+0000 7fa74e00e700 1 --2- 172.21.15.16:0/852713627 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa74815c190 0x7fa748154030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:07.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:07.051+0000 7fa74e00e700 1 --2- 172.21.15.16:0/852713627 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa748155a60 0x7fa74814e7c0 unknown :-1 s=CLOSED pgs=2918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:07.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:07.051+0000 7fa74e00e700 1 --2- 172.21.15.16:0/852713627 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa748155e40 0x7fa7480aff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:07.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:07.051+0000 7fa74e00e700 1 -- 172.21.15.16:0/852713627 >> 172.21.15.16:0/852713627 conn(0x7fa74806a7e0 msgr2=0x7fa7480d13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:07.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:07.051+0000 7fa74e00e700 1 -- 172.21.15.16:0/852713627 shutdown_connections 2023-10-16T13:48:07.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:07.051+0000 7fa74e00e700 1 -- 172.21.15.16:0/852713627 wait complete. 2023-10-16T13:48:07.056 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' removed 2023-10-16T13:48:07.069 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-16T13:48:07.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:07.202+0000 7f0f80f82700 1 Processor -- start 2023-10-16T13:48:07.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:07.202+0000 7f0f80f82700 1 -- start start 2023-10-16T13:48:07.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:07.203+0000 7f0f80f82700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f7c157b30 0x7f0f7c0cefc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:07.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:07.203+0000 7f0f80f82700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0f7c161d20 0x7f0f7c0cf500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:07.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:07.203+0000 7f0f80f82700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f7c142fc0 con 0x7f0f7c157f10 2023-10-16T13:48:07.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:07.203+0000 7f0f80f82700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f7c06d900 con 0x7f0f7c161d20 2023-10-16T13:48:07.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:07.203+0000 7f0f80f82700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f7c143210 con 0x7f0f7c157b30 2023-10-16T13:48:07.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:07.203+0000 7f0f7ad9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0f7c161d20 0x7f0f7c0cf500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:07.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:07.203+0000 7f0f7ad9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0f7c161d20 0x7f0f7c0cf500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56998/0 (socket says 172.21.15.16:56998) 2023-10-16T13:48:07.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:07.203+0000 7f0f7ad9d700 1 -- 172.21.15.16:0/3628410778 learned_addr learned my addr 172.21.15.16:0/3628410778 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:07.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:07.203+0000 7f0f7a59c700 1 --2- 172.21.15.16:0/3628410778 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f7c157b30 0x7f0f7c0cefc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:07.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:07.203+0000 7f0f7ad9d700 1 -- 172.21.15.16:0/3628410778 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f7c157b30 msgr2=0x7f0f7c0cefc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:07.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:07.203+0000 7f0f7ad9d700 1 --2- 172.21.15.16:0/3628410778 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f7c157b30 0x7f0f7c0cefc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:07.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:07.203+0000 7f0f7ad9d700 1 -- 172.21.15.16:0/3628410778 >> v1:172.21.15.49:6789/0 conn(0x7f0f7c157f10 legacy=0x7f0f7c161610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:07.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:07.203+0000 7f0f7ad9d700 1 -- 172.21.15.16:0/3628410778 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f7c1434f0 con 0x7f0f7c161d20 2023-10-16T13:48:07.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:07.203+0000 7f0f7ad9d700 1 --2- 172.21.15.16:0/3628410778 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f7c161d20 0x7f0f7c0cf500 secure :-1 s=READY pgs=2919 cs=0 l=1 rev1=1 crypto rx=0x7f0f74009e50 tx=0x7f0f7400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e7cec1d5cf00c03 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:07.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:07.204+0000 7f0f7959a700 1 -- 172.21.15.16:0/3628410778 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f7400e070 con 0x7f0f7c161d20 2023-10-16T13:48:07.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:07.204+0000 7f0f7959a700 1 -- 172.21.15.16:0/3628410778 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f7400b9c0 con 0x7f0f7c161d20 2023-10-16T13:48:07.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:07.204+0000 7f0f7959a700 1 -- 172.21.15.16:0/3628410778 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f7400bd70 con 0x7f0f7c161d20 2023-10-16T13:48:07.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:07.204+0000 7f0f80f82700 1 -- 172.21.15.16:0/3628410778 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f7c161d20 msgr2=0x7f0f7c0cf500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:07.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:07.204+0000 7f0f80f82700 1 --2- 172.21.15.16:0/3628410778 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f7c161d20 0x7f0f7c0cf500 secure :-1 s=READY pgs=2919 cs=0 l=1 rev1=1 crypto rx=0x7f0f74009e50 tx=0x7f0f7400b330 comp rx=0 tx=0).stop 2023-10-16T13:48:07.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:07.204+0000 7f0f80f82700 1 -- 172.21.15.16:0/3628410778 shutdown_connections 2023-10-16T13:48:07.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:07.204+0000 7f0f80f82700 1 --2- 172.21.15.16:0/3628410778 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f7c161d20 0x7f0f7c0cf500 unknown :-1 s=CLOSED pgs=2919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:07.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:07.204+0000 7f0f80f82700 1 --2- 172.21.15.16:0/3628410778 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f7c157b30 0x7f0f7c0cefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:07.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:07.204+0000 7f0f80f82700 1 -- 172.21.15.16:0/3628410778 >> 172.21.15.16:0/3628410778 conn(0x7f0f7c06a7e0 msgr2=0x7f0f7c06abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:07.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:07.204+0000 7f0f80f82700 1 -- 172.21.15.16:0/3628410778 shutdown_connections 2023-10-16T13:48:07.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:07.204+0000 7f0f80f82700 1 -- 172.21.15.16:0/3628410778 wait complete. 2023-10-16T13:48:07.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:07.206+0000 7f0f80f82700 1 Processor -- start 2023-10-16T13:48:07.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:07.206+0000 7f0f80f82700 1 -- start start 2023-10-16T13:48:07.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:07.206+0000 7f0f80f82700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f7c157b30 0x7f0f7c0ca1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:07.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:07.206+0000 7f0f80f82700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f7c157f10 0x7f0f7c0cae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:07.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:07.206+0000 7f0f80f82700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f7c161d20 0x7f0f7c1543b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:07.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:07.206+0000 7f0f80f82700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f7c0c9ea0 con 0x7f0f7c157f10 2023-10-16T13:48:07.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:07.206+0000 7f0f80f82700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f7c0ca8d0 con 0x7f0f7c157b30 2023-10-16T13:48:07.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:07.206+0000 7f0f80f82700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f7c0c9570 con 0x7f0f7c161d20 2023-10-16T13:48:07.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:07.206+0000 7f0f79d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f7c157f10 0x7f0f7c0cae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:07.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:07.207+0000 7f0f79d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f7c157f10 0x7f0f7c0cae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57002/0 (socket says 172.21.15.16:57002) 2023-10-16T13:48:07.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:07.207+0000 7f0f79d9b700 1 -- 172.21.15.16:0/4126988124 learned_addr learned my addr 172.21.15.16:0/4126988124 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:07.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:07.207+0000 7f0f7a59c700 1 --2- 172.21.15.16:0/4126988124 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f7c157b30 0x7f0f7c0ca1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:07.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:07.207+0000 7f0f7ad9d700 1 --2- 172.21.15.16:0/4126988124 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f7c161d20 0x7f0f7c1543b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:07.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:07.207+0000 7f0f79d9b700 1 -- 172.21.15.16:0/4126988124 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f7c161d20 msgr2=0x7f0f7c1543b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:07.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:07.207+0000 7f0f79d9b700 1 --2- 172.21.15.16:0/4126988124 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f7c161d20 0x7f0f7c1543b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:07.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:07.207+0000 7f0f79d9b700 1 -- 172.21.15.16:0/4126988124 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f7c157b30 msgr2=0x7f0f7c0ca1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:07.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:07.207+0000 7f0f79d9b700 1 --2- 172.21.15.16:0/4126988124 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f7c157b30 0x7f0f7c0ca1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:07.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:07.207+0000 7f0f79d9b700 1 -- 172.21.15.16:0/4126988124 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f74009b30 con 0x7f0f7c157f10 2023-10-16T13:48:07.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:07.207+0000 7f0f7a59c700 1 --2- 172.21.15.16:0/4126988124 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f7c157b30 0x7f0f7c0ca1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:48:07.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:07.207+0000 7f0f79d9b700 1 --2- 172.21.15.16:0/4126988124 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f7c157f10 0x7f0f7c0cae20 secure :-1 s=READY pgs=2920 cs=0 l=1 rev1=1 crypto rx=0x7f0f7000c4e0 tx=0x7f0f7000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:07.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:07.207+0000 7f0f6affd700 1 -- 172.21.15.16:0/4126988124 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f700052d0 con 0x7f0f7c157f10 2023-10-16T13:48:07.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:07.207+0000 7f0f6affd700 1 -- 172.21.15.16:0/4126988124 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f70005430 con 0x7f0f7c157f10 2023-10-16T13:48:07.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:07.207+0000 7f0f6affd700 1 -- 172.21.15.16:0/4126988124 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f700038c0 con 0x7f0f7c157f10 2023-10-16T13:48:07.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:07.207+0000 7f0f80f82700 1 -- 172.21.15.16:0/4126988124 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f7c0c9850 con 0x7f0f7c157f10 2023-10-16T13:48:07.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:07.207+0000 7f0f80f82700 1 -- 172.21.15.16:0/4126988124 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f7c1548f0 con 0x7f0f7c157f10 2023-10-16T13:48:07.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:07.208+0000 7f0f80f82700 1 -- 172.21.15.16:0/4126988124 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f7c04ec20 con 0x7f0f7c157f10 2023-10-16T13:48:07.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:07.209+0000 7f0f6affd700 1 -- 172.21.15.16:0/4126988124 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f0f70003a20 con 0x7f0f7c157f10 2023-10-16T13:48:07.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:07.209+0000 7f0f6affd700 1 --2- 172.21.15.16:0/4126988124 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f0f5c06b140 0x7f0f5c06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:07.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:07.209+0000 7f0f6affd700 1 -- 172.21.15.16:0/4126988124 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(253..253 src has 1..253) v4 ==== 7529+0+0 (secure 0 0 0) 0x7f0f70089530 con 0x7f0f7c157f10 2023-10-16T13:48:07.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:07.209+0000 7f0f7a59c700 1 --2- 172.21.15.16:0/4126988124 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f0f5c06b140 0x7f0f5c06d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:07.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:07.210+0000 7f0f7a59c700 1 --2- 172.21.15.16:0/4126988124 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f0f5c06b140 0x7f0f5c06d600 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c009a00 tx=0x7f0f6c008040 comp rx=0 tx=0).ready entity=mgr.12678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:07.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:07.214+0000 7f0f6affd700 1 -- 172.21.15.16:0/4126988124 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0f70053c30 con 0x7f0f7c157f10 2023-10-16T13:48:07.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:07.425+0000 7f0f80f82700 1 -- 172.21.15.16:0/4126988124 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f0f7c0b3b50 con 0x7f0f7c157f10 2023-10-16T13:48:08.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.350+0000 7f0f6affd700 1 -- 172.21.15.16:0/4126988124 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v254) v1 ==== 169+0+0 (secure 0 0 0) 0x7f0f70057250 con 0x7f0f7c157f10 2023-10-16T13:48:08.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.356+0000 7f0f80f82700 1 -- 172.21.15.16:0/4126988124 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f0f5c06b140 msgr2=0x7f0f5c06d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:08.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.356+0000 7f0f80f82700 1 --2- 172.21.15.16:0/4126988124 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f0f5c06b140 0x7f0f5c06d600 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c009a00 tx=0x7f0f6c008040 comp rx=0 tx=0).stop 2023-10-16T13:48:08.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.356+0000 7f0f80f82700 1 -- 172.21.15.16:0/4126988124 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f7c157f10 msgr2=0x7f0f7c0cae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:08.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.356+0000 7f0f80f82700 1 --2- 172.21.15.16:0/4126988124 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f7c157f10 0x7f0f7c0cae20 secure :-1 s=READY pgs=2920 cs=0 l=1 rev1=1 crypto rx=0x7f0f7000c4e0 tx=0x7f0f7000c8a0 comp rx=0 tx=0).stop 2023-10-16T13:48:08.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.357+0000 7f0f80f82700 1 -- 172.21.15.16:0/4126988124 shutdown_connections 2023-10-16T13:48:08.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.357+0000 7f0f80f82700 1 --2- 172.21.15.16:0/4126988124 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f0f5c06b140 0x7f0f5c06d600 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:08.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.357+0000 7f0f80f82700 1 --2- 172.21.15.16:0/4126988124 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f7c161d20 0x7f0f7c1543b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:08.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.357+0000 7f0f80f82700 1 --2- 172.21.15.16:0/4126988124 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f7c157b30 0x7f0f7c0ca1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:08.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.357+0000 7f0f80f82700 1 --2- 172.21.15.16:0/4126988124 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f7c157f10 0x7f0f7c0cae20 secure :-1 s=CLOSED pgs=2920 cs=0 l=1 rev1=1 crypto rx=0x7f0f7000c4e0 tx=0x7f0f7000c8a0 comp rx=0 tx=0).stop 2023-10-16T13:48:08.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.357+0000 7f0f80f82700 1 -- 172.21.15.16:0/4126988124 >> 172.21.15.16:0/4126988124 conn(0x7f0f7c06a7e0 msgr2=0x7f0f7c160b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:08.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.359+0000 7f0f80f82700 1 -- 172.21.15.16:0/4126988124 shutdown_connections 2023-10-16T13:48:08.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.360+0000 7f0f80f82700 1 -- 172.21.15.16:0/4126988124 wait complete. 2023-10-16T13:48:08.365 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' removed 2023-10-16T13:48:08.380 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:48:08.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.510+0000 7f031e9f7700 1 Processor -- start 2023-10-16T13:48:08.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.510+0000 7f031e9f7700 1 -- start start 2023-10-16T13:48:08.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.511+0000 7f031e9f7700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f031815a5c0 0x7f03180cef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:08.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.511+0000 7f031e9f7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f03180ce010 0x7f03180d03d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:08.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.511+0000 7f031e9f7700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0318072a50 con 0x7f03180d0910 2023-10-16T13:48:08.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.511+0000 7f031e9f7700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03180cf440 con 0x7f03180ce010 2023-10-16T13:48:08.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.511+0000 7f031e9f7700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03180d0da0 con 0x7f031815a5c0 2023-10-16T13:48:08.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.511+0000 7f030ffff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f03180ce010 0x7f03180d03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:08.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.511+0000 7f030ffff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f03180ce010 0x7f03180d03d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57010/0 (socket says 172.21.15.16:57010) 2023-10-16T13:48:08.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.511+0000 7f030ffff700 1 -- 172.21.15.16:0/2318996706 learned_addr learned my addr 172.21.15.16:0/2318996706 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:08.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.511+0000 7f0317fff700 1 --2- 172.21.15.16:0/2318996706 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f031815a5c0 0x7f03180cef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:08.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.511+0000 7f030ffff700 1 -- 172.21.15.16:0/2318996706 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f031815a5c0 msgr2=0x7f03180cef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:08.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.511+0000 7f030ffff700 1 --2- 172.21.15.16:0/2318996706 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f031815a5c0 0x7f03180cef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:08.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.511+0000 7f030ffff700 1 -- 172.21.15.16:0/2318996706 >> v1:172.21.15.49:6789/0 conn(0x7f03180d0910 legacy=0x7f031815e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:08.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.511+0000 7f030ffff700 1 -- 172.21.15.16:0/2318996706 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03180cba30 con 0x7f03180ce010 2023-10-16T13:48:08.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.512+0000 7f030ffff700 1 --2- 172.21.15.16:0/2318996706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03180ce010 0x7f03180d03d0 secure :-1 s=READY pgs=2921 cs=0 l=1 rev1=1 crypto rx=0x7f03000098f0 tx=0x7f0300009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c601f965bf34636f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:08.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.512+0000 7f03177fe700 1 -- 172.21.15.16:0/2318996706 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0300004230 con 0x7f03180ce010 2023-10-16T13:48:08.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.512+0000 7f03177fe700 1 -- 172.21.15.16:0/2318996706 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0300004390 con 0x7f03180ce010 2023-10-16T13:48:08.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.512+0000 7f03177fe700 1 -- 172.21.15.16:0/2318996706 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03000036a0 con 0x7f03180ce010 2023-10-16T13:48:08.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.512+0000 7f031e9f7700 1 -- 172.21.15.16:0/2318996706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03180ce010 msgr2=0x7f03180d03d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:08.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.512+0000 7f031e9f7700 1 --2- 172.21.15.16:0/2318996706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03180ce010 0x7f03180d03d0 secure :-1 s=READY pgs=2921 cs=0 l=1 rev1=1 crypto rx=0x7f03000098f0 tx=0x7f0300009c00 comp rx=0 tx=0).stop 2023-10-16T13:48:08.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.512+0000 7f031e9f7700 1 -- 172.21.15.16:0/2318996706 shutdown_connections 2023-10-16T13:48:08.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.512+0000 7f031e9f7700 1 --2- 172.21.15.16:0/2318996706 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f031815a5c0 0x7f03180cef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:08.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.512+0000 7f031e9f7700 1 --2- 172.21.15.16:0/2318996706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03180ce010 0x7f03180d03d0 unknown :-1 s=CLOSED pgs=2921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:08.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.512+0000 7f031e9f7700 1 -- 172.21.15.16:0/2318996706 >> 172.21.15.16:0/2318996706 conn(0x7f03180b1fd0 msgr2=0x7f03180b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:08.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.512+0000 7f031e9f7700 1 -- 172.21.15.16:0/2318996706 shutdown_connections 2023-10-16T13:48:08.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.512+0000 7f031e9f7700 1 -- 172.21.15.16:0/2318996706 wait complete. 2023-10-16T13:48:08.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.514+0000 7f031e9f7700 1 Processor -- start 2023-10-16T13:48:08.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.514+0000 7f031e9f7700 1 -- start start 2023-10-16T13:48:08.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.514+0000 7f031e9f7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03180ce010 0x7f031815c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:08.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.514+0000 7f031e9f7700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f03180d0910 0x7f03180c6910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:08.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.514+0000 7f031e9f7700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f031815a5c0 0x7f0318146cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:08.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.514+0000 7f031e9f7700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03180ca8d0 con 0x7f03180ce010 2023-10-16T13:48:08.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.514+0000 7f031e9f7700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f031815cc70 con 0x7f03180d0910 2023-10-16T13:48:08.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.514+0000 7f031e9f7700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03180c6e50 con 0x7f031815a5c0 2023-10-16T13:48:08.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.514+0000 7f0317fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03180ce010 0x7f031815c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:08.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.515+0000 7f0317fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03180ce010 0x7f031815c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57014/0 (socket says 172.21.15.16:57014) 2023-10-16T13:48:08.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.515+0000 7f0317fff700 1 -- 172.21.15.16:0/1492802009 learned_addr learned my addr 172.21.15.16:0/1492802009 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:08.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.515+0000 7f030ffff700 1 --2- 172.21.15.16:0/1492802009 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f03180d0910 0x7f03180c6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:08.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.515+0000 7f031cf94700 1 --2- 172.21.15.16:0/1492802009 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f031815a5c0 0x7f0318146cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:08.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.515+0000 7f0317fff700 1 -- 172.21.15.16:0/1492802009 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f031815a5c0 msgr2=0x7f0318146cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:08.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.515+0000 7f0317fff700 1 --2- 172.21.15.16:0/1492802009 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f031815a5c0 0x7f0318146cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:08.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.515+0000 7f0317fff700 1 -- 172.21.15.16:0/1492802009 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f03180d0910 msgr2=0x7f03180c6910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:08.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.515+0000 7f0317fff700 1 --2- 172.21.15.16:0/1492802009 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f03180d0910 0x7f03180c6910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:08.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.515+0000 7f0317fff700 1 -- 172.21.15.16:0/1492802009 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0300009550 con 0x7f03180ce010 2023-10-16T13:48:08.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.515+0000 7f030ffff700 1 --2- 172.21.15.16:0/1492802009 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f03180d0910 0x7f03180c6910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:48:08.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.515+0000 7f0317fff700 1 --2- 172.21.15.16:0/1492802009 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03180ce010 0x7f031815c730 secure :-1 s=READY pgs=2922 cs=0 l=1 rev1=1 crypto rx=0x7f030800c7f0 tx=0x7f030800cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:08.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.515+0000 7f03157fa700 1 -- 172.21.15.16:0/1492802009 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0308011800 con 0x7f03180ce010 2023-10-16T13:48:08.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.515+0000 7f03157fa700 1 -- 172.21.15.16:0/1492802009 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0308011960 con 0x7f03180ce010 2023-10-16T13:48:08.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.515+0000 7f031e9f7700 1 -- 172.21.15.16:0/1492802009 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f031815df20 con 0x7f03180ce010 2023-10-16T13:48:08.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.515+0000 7f03157fa700 1 -- 172.21.15.16:0/1492802009 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0308011cf0 con 0x7f03180ce010 2023-10-16T13:48:08.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.515+0000 7f031e9f7700 1 -- 172.21.15.16:0/1492802009 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f031815d160 con 0x7f03180ce010 2023-10-16T13:48:08.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.516+0000 7f031e9f7700 1 -- 172.21.15.16:0/1492802009 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f031804ebb0 con 0x7f03180ce010 2023-10-16T13:48:08.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.517+0000 7f03157fa700 1 -- 172.21.15.16:0/1492802009 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f0308018360 con 0x7f03180ce010 2023-10-16T13:48:08.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.517+0000 7f03157fa700 1 --2- 172.21.15.16:0/1492802009 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f02f806b140 0x7f02f806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:08.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.517+0000 7f03157fa700 1 -- 172.21.15.16:0/1492802009 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(254..254 src has 1..254) v4 ==== 7115+0+0 (secure 0 0 0) 0x7f030808a020 con 0x7f03180ce010 2023-10-16T13:48:08.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.517+0000 7f030ffff700 1 --2- 172.21.15.16:0/1492802009 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f02f806b140 0x7f02f806d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:08.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.518+0000 7f030ffff700 1 --2- 172.21.15.16:0/1492802009 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f02f806b140 0x7f02f806d600 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0300009c40 tx=0x7f0300008030 comp rx=0 tx=0).ready entity=mgr.12678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:08.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.522+0000 7f03157fa700 1 -- 172.21.15.16:0/1492802009 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f03080546f0 con 0x7f03180ce010 2023-10-16T13:48:08.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.724+0000 7f031e9f7700 1 -- 172.21.15.16:0/1492802009 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0318071af0 con 0x7f03180ce010 2023-10-16T13:48:08.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.726+0000 7f03157fa700 1 -- 172.21.15.16:0/1492802009 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v254) v1 ==== 74+0+18205 (secure 0 0 0) 0x7f0308057d10 con 0x7f03180ce010 2023-10-16T13:48:08.729 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:48:08.729 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":254,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:48:08.349411+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T13:47:39.075528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"248","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"backup_fs_data","create_time":"2023-10-16T13:47:40.314029+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"249","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6835/303278795":"2023-10-17T13:48:05.033844+0000","172.21.15.16:6832/2221624769":"2023-10-17T13:48:02.485614+0000","172.21.15.16:0/2242986614":"2023-10-17T13:48:02.485614+0000","172.21.15.16:0/3610227254":"2023-10-17T13:48:02.485614+0000","172.21.15.16:0/1182888331":"2023-10-17T13:48:02.485614+0000","172.21.15.16:0/2095012141":"2023-10-17T13:48:02.485614+0000","172.21.15.16:0/874722077":"2023-10-17T13:48:02.485614+0000","172.21.15.16:0/2066401556":"2023-10-17T13:48:02.485614+0000","172.21.15.16:0/3947601914":"2023-10-17T13:47:54.816700+0000","172.21.15.16:6833/2221624769":"2023-10-17T13:48:02.485614+0000","172.21.15.16:0/2759470534":"2023-10-17T13:47:54.816700+0000","172.21.15.16:0/2634105683":"2023-10-17T13:48:02.485614+0000","172.21.15.16:0/2974892940":"2023-10-17T13:47:54.816700+0000","172.21.15.16:6833/1736104576":"2023-10-17T13:47:54.816700+0000","172.21.15.16:6832/1736104576":"2023-10-17T13:47:54.816700+0000","172.21.15.16:0/1246409088":"2023-10-17T13:48:02.485614+0000","172.21.15.16:0/318862437":"2023-10-17T13:47:54.816700+0000","172.21.15.16:0/413581668":"2023-10-17T13:47:54.816700+0000","172.21.15.16:6837/303278795":"2023-10-17T13:48:05.033844+0000","172.21.15.16:0/3740515383":"2023-10-17T13:47:54.816700+0000","172.21.15.16:0/3105924907":"2023-10-17T13:47:54.816700+0000","172.21.15.16:0/3617558038":"2023-10-17T13:47:54.816700+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:48:08.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.728+0000 7f031e9f7700 1 -- 172.21.15.16:0/1492802009 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f02f806b140 msgr2=0x7f02f806d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:08.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.728+0000 7f031e9f7700 1 --2- 172.21.15.16:0/1492802009 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f02f806b140 0x7f02f806d600 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0300009c40 tx=0x7f0300008030 comp rx=0 tx=0).stop 2023-10-16T13:48:08.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.728+0000 7f031e9f7700 1 -- 172.21.15.16:0/1492802009 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03180ce010 msgr2=0x7f031815c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:08.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.728+0000 7f031e9f7700 1 --2- 172.21.15.16:0/1492802009 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03180ce010 0x7f031815c730 secure :-1 s=READY pgs=2922 cs=0 l=1 rev1=1 crypto rx=0x7f030800c7f0 tx=0x7f030800cbb0 comp rx=0 tx=0).stop 2023-10-16T13:48:08.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.728+0000 7f031e9f7700 1 -- 172.21.15.16:0/1492802009 shutdown_connections 2023-10-16T13:48:08.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.728+0000 7f031e9f7700 1 --2- 172.21.15.16:0/1492802009 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f02f806b140 0x7f02f806d600 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:08.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.728+0000 7f031e9f7700 1 --2- 172.21.15.16:0/1492802009 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f031815a5c0 0x7f0318146cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:08.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.728+0000 7f031e9f7700 1 --2- 172.21.15.16:0/1492802009 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03180ce010 0x7f031815c730 unknown :-1 s=CLOSED pgs=2922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:08.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.728+0000 7f031e9f7700 1 --2- 172.21.15.16:0/1492802009 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f03180d0910 0x7f03180c6910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:08.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.728+0000 7f031e9f7700 1 -- 172.21.15.16:0/1492802009 >> 172.21.15.16:0/1492802009 conn(0x7f03180b1fd0 msgr2=0x7f031806a7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:08.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.728+0000 7f031e9f7700 1 -- 172.21.15.16:0/1492802009 shutdown_connections 2023-10-16T13:48:08.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.728+0000 7f031e9f7700 1 -- 172.21.15.16:0/1492802009 wait complete. 2023-10-16T13:48:08.743 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 33, 'pool_name': 'backup_fs_metadata', 'create_time': '2023-10-16T13:47:39.075528+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '248', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 34, 'pool_name': 'backup_fs_data', 'create_time': '2023-10-16T13:47:40.314029+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '249', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T13:48:08.744 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T13:48:08.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.869+0000 7f3be5487700 1 Processor -- start 2023-10-16T13:48:08.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.869+0000 7f3be5487700 1 -- start start 2023-10-16T13:48:08.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.870+0000 7f3be5487700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3be015d8d0 0x7f3be00b2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:08.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.870+0000 7f3be5487700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3be00ce350 0x7f3be00ce780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:08.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.870+0000 7f3be5487700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3be00cecc0 con 0x7f3be00b8fa0 2023-10-16T13:48:08.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.870+0000 7f3be5487700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3be00d0bb0 con 0x7f3be015d8d0 2023-10-16T13:48:08.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.870+0000 7f3be5487700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3be015dcb0 con 0x7f3be00ce350 2023-10-16T13:48:08.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.870+0000 7f3bdeffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3be015d8d0 0x7f3be00b2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:08.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.870+0000 7f3bdeffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3be015d8d0 0x7f3be00b2830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57022/0 (socket says 172.21.15.16:57022) 2023-10-16T13:48:08.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.870+0000 7f3bdeffd700 1 -- 172.21.15.16:0/110383541 learned_addr learned my addr 172.21.15.16:0/110383541 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:08.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.870+0000 7f3bde7fc700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f3be00b8fa0 0x7f3be00b6720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46298/0 (socket says 172.21.15.16:46298) 2023-10-16T13:48:08.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.870+0000 7f3bdf7fe700 1 --2- 172.21.15.16:0/110383541 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3be00ce350 0x7f3be00ce780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:08.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.870+0000 7f3bdeffd700 1 -- 172.21.15.16:0/110383541 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3be00ce350 msgr2=0x7f3be00ce780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:08.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.870+0000 7f3bdeffd700 1 --2- 172.21.15.16:0/110383541 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3be00ce350 0x7f3be00ce780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:08.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.871+0000 7f3bdeffd700 1 -- 172.21.15.16:0/110383541 >> v1:172.21.15.16:6789/0 conn(0x7f3be00b8fa0 legacy=0x7f3be00b6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:08.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.871+0000 7f3bdeffd700 1 -- 172.21.15.16:0/110383541 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3be00b2f30 con 0x7f3be015d8d0 2023-10-16T13:48:08.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.871+0000 7f3bdeffd700 1 --2- 172.21.15.16:0/110383541 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3be015d8d0 0x7f3be00b2830 secure :-1 s=READY pgs=2924 cs=0 l=1 rev1=1 crypto rx=0x7f3bd40098f0 tx=0x7f3bd4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9870e4e555c602c0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:08.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.871+0000 7f3bddffb700 1 -- 172.21.15.16:0/110383541 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bd400b8a0 con 0x7f3be015d8d0 2023-10-16T13:48:08.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.871+0000 7f3bddffb700 1 -- 172.21.15.16:0/110383541 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bd400ba00 con 0x7f3be015d8d0 2023-10-16T13:48:08.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.871+0000 7f3bddffb700 1 -- 172.21.15.16:0/110383541 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bd400bd90 con 0x7f3be015d8d0 2023-10-16T13:48:08.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.871+0000 7f3be5487700 1 -- 172.21.15.16:0/110383541 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3be015d8d0 msgr2=0x7f3be00b2830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:08.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.871+0000 7f3be5487700 1 --2- 172.21.15.16:0/110383541 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3be015d8d0 0x7f3be00b2830 secure :-1 s=READY pgs=2924 cs=0 l=1 rev1=1 crypto rx=0x7f3bd40098f0 tx=0x7f3bd4009c00 comp rx=0 tx=0).stop 2023-10-16T13:48:08.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.871+0000 7f3be5487700 1 -- 172.21.15.16:0/110383541 shutdown_connections 2023-10-16T13:48:08.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.871+0000 7f3be5487700 1 --2- 172.21.15.16:0/110383541 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3be00ce350 0x7f3be00ce780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:08.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.871+0000 7f3be5487700 1 --2- 172.21.15.16:0/110383541 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3be015d8d0 0x7f3be00b2830 unknown :-1 s=CLOSED pgs=2924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:08.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.871+0000 7f3be5487700 1 -- 172.21.15.16:0/110383541 >> 172.21.15.16:0/110383541 conn(0x7f3be0072330 msgr2=0x7f3be0072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:08.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.872+0000 7f3be5487700 1 -- 172.21.15.16:0/110383541 shutdown_connections 2023-10-16T13:48:08.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.872+0000 7f3be5487700 1 -- 172.21.15.16:0/110383541 wait complete. 2023-10-16T13:48:08.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.873+0000 7f3be5487700 1 Processor -- start 2023-10-16T13:48:08.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.873+0000 7f3be5487700 1 -- start start 2023-10-16T13:48:08.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.874+0000 7f3be5487700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3be00b8fa0 0x7f3be014f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:08.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.874+0000 7f3be5487700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3be00ce350 0x7f3be014eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:08.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.874+0000 7f3be5487700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3be015d8d0 0x7f3be00cbe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:08.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.874+0000 7f3be5487700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3be00cb830 con 0x7f3be00b8fa0 2023-10-16T13:48:08.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.874+0000 7f3be5487700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3be014f050 con 0x7f3be00ce350 2023-10-16T13:48:08.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.874+0000 7f3be5487700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3be00cb550 con 0x7f3be015d8d0 2023-10-16T13:48:08.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.874+0000 7f3bdeffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3be00b8fa0 0x7f3be014f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:08.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.874+0000 7f3bdeffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3be00b8fa0 0x7f3be014f330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57028/0 (socket says 172.21.15.16:57028) 2023-10-16T13:48:08.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.874+0000 7f3bdeffd700 1 -- 172.21.15.16:0/4008644477 learned_addr learned my addr 172.21.15.16:0/4008644477 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:08.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.875+0000 7f3bde7fc700 1 --2- 172.21.15.16:0/4008644477 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3be00ce350 0x7f3be014eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:08.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.875+0000 7f3bdf7fe700 1 --2- 172.21.15.16:0/4008644477 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3be015d8d0 0x7f3be00cbe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:08.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.875+0000 7f3bdeffd700 1 -- 172.21.15.16:0/4008644477 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3be015d8d0 msgr2=0x7f3be00cbe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:08.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.875+0000 7f3bdeffd700 1 --2- 172.21.15.16:0/4008644477 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3be015d8d0 0x7f3be00cbe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:08.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.875+0000 7f3bdeffd700 1 -- 172.21.15.16:0/4008644477 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3be00ce350 msgr2=0x7f3be014eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:08.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.875+0000 7f3bdeffd700 1 --2- 172.21.15.16:0/4008644477 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3be00ce350 0x7f3be014eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:08.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.875+0000 7f3bdeffd700 1 -- 172.21.15.16:0/4008644477 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bd4009550 con 0x7f3be00b8fa0 2023-10-16T13:48:08.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.875+0000 7f3bde7fc700 1 --2- 172.21.15.16:0/4008644477 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3be00ce350 0x7f3be014eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:48:08.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.875+0000 7f3bdeffd700 1 --2- 172.21.15.16:0/4008644477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3be00b8fa0 0x7f3be014f330 secure :-1 s=READY pgs=2925 cs=0 l=1 rev1=1 crypto rx=0x7f3bd4009c40 tx=0x7f3bd4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:08.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.875+0000 7f3bc77fe700 1 -- 172.21.15.16:0/4008644477 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bd40036a0 con 0x7f3be00b8fa0 2023-10-16T13:48:08.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.875+0000 7f3bc77fe700 1 -- 172.21.15.16:0/4008644477 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bd401e070 con 0x7f3be00b8fa0 2023-10-16T13:48:08.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.875+0000 7f3be5487700 1 -- 172.21.15.16:0/4008644477 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3be00c71e0 con 0x7f3be00b8fa0 2023-10-16T13:48:08.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.875+0000 7f3bc77fe700 1 -- 172.21.15.16:0/4008644477 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bd4003a60 con 0x7f3be00b8fa0 2023-10-16T13:48:08.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.875+0000 7f3be5487700 1 -- 172.21.15.16:0/4008644477 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3be00c77a0 con 0x7f3be00b8fa0 2023-10-16T13:48:08.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.876+0000 7f3be5487700 1 -- 172.21.15.16:0/4008644477 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3be00c6fe0 con 0x7f3be00b8fa0 2023-10-16T13:48:08.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.877+0000 7f3bc77fe700 1 -- 172.21.15.16:0/4008644477 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f3bd4003bc0 con 0x7f3be00b8fa0 2023-10-16T13:48:08.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.877+0000 7f3bc77fe700 1 --2- 172.21.15.16:0/4008644477 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f3bcc06b140 0x7f3bcc06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:08.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.877+0000 7f3bc77fe700 1 -- 172.21.15.16:0/4008644477 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(254..254 src has 1..254) v4 ==== 7115+0+0 (secure 0 0 0) 0x7f3bd4020070 con 0x7f3be00b8fa0 2023-10-16T13:48:08.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.877+0000 7f3bde7fc700 1 --2- 172.21.15.16:0/4008644477 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f3bcc06b140 0x7f3bcc06d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:08.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.877+0000 7f3bde7fc700 1 --2- 172.21.15.16:0/4008644477 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f3bcc06b140 0x7f3bcc06d600 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3be0160df0 tx=0x7f3bd0007dd0 comp rx=0 tx=0).ready entity=mgr.12678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:08.888 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:08.881+0000 7f3bc77fe700 1 -- 172.21.15.16:0/4008644477 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3bd4055c70 con 0x7f3be00b8fa0 2023-10-16T13:48:09.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.086+0000 7f3be5487700 1 -- 172.21.15.16:0/4008644477 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3be004ebb0 con 0x7f3be00b8fa0 2023-10-16T13:48:09.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.086+0000 7f3bc77fe700 1 -- 172.21.15.16:0/4008644477 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v254) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3bd4059290 con 0x7f3be00b8fa0 2023-10-16T13:48:09.089 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T13:48:09.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.088+0000 7f3be5487700 1 -- 172.21.15.16:0/4008644477 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f3bcc06b140 msgr2=0x7f3bcc06d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:09.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.088+0000 7f3be5487700 1 --2- 172.21.15.16:0/4008644477 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f3bcc06b140 0x7f3bcc06d600 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3be0160df0 tx=0x7f3bd0007dd0 comp rx=0 tx=0).stop 2023-10-16T13:48:09.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.088+0000 7f3be5487700 1 -- 172.21.15.16:0/4008644477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3be00b8fa0 msgr2=0x7f3be014f330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:09.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.088+0000 7f3be5487700 1 --2- 172.21.15.16:0/4008644477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3be00b8fa0 0x7f3be014f330 secure :-1 s=READY pgs=2925 cs=0 l=1 rev1=1 crypto rx=0x7f3bd4009c40 tx=0x7f3bd4004750 comp rx=0 tx=0).stop 2023-10-16T13:48:09.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.089+0000 7f3be5487700 1 -- 172.21.15.16:0/4008644477 shutdown_connections 2023-10-16T13:48:09.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.089+0000 7f3be5487700 1 --2- 172.21.15.16:0/4008644477 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f3bcc06b140 0x7f3bcc06d600 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:09.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.089+0000 7f3be5487700 1 --2- 172.21.15.16:0/4008644477 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3be015d8d0 0x7f3be00cbe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:09.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.089+0000 7f3be5487700 1 --2- 172.21.15.16:0/4008644477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3be00b8fa0 0x7f3be014f330 unknown :-1 s=CLOSED pgs=2925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:09.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.089+0000 7f3be5487700 1 --2- 172.21.15.16:0/4008644477 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3be00ce350 0x7f3be014eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:09.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.089+0000 7f3be5487700 1 -- 172.21.15.16:0/4008644477 >> 172.21.15.16:0/4008644477 conn(0x7f3be0072330 msgr2=0x7f3be00d2860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:09.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.089+0000 7f3be5487700 1 -- 172.21.15.16:0/4008644477 shutdown_connections 2023-10-16T13:48:09.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.089+0000 7f3be5487700 1 -- 172.21.15.16:0/4008644477 wait complete. 2023-10-16T13:48:09.103 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2023-10-16T13:48:09.230 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.228+0000 7fa0e0e29700 1 Processor -- start 2023-10-16T13:48:09.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.228+0000 7fa0e0e29700 1 -- start start 2023-10-16T13:48:09.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.229+0000 7fa0e0e29700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fa0dc0d2110 0x7fa0dc0ae1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:09.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.229+0000 7fa0e0e29700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa0dc06ca10 0x7fa0dc0ae720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:09.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.229+0000 7fa0e0e29700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0dc0b20f0 con 0x7fa0dc0ce6a0 2023-10-16T13:48:09.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.229+0000 7fa0e0e29700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0dc0d03d0 con 0x7fa0dc06ca10 2023-10-16T13:48:09.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.229+0000 7fa0e0e29700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0dc0b2340 con 0x7fa0dc0d2110 2023-10-16T13:48:09.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.229+0000 7fa0dad9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa0dc06ca10 0x7fa0dc0ae720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:09.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.229+0000 7fa0da59c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa0dc0ce6a0 0x7fa0dc06c300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46310/0 (socket says 172.21.15.16:46310) 2023-10-16T13:48:09.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.229+0000 7fa0da59c700 1 -- 172.21.15.16:0/480597893 learned_addr learned my addr 172.21.15.16:0/480597893 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:48:09.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.230+0000 7fa0d9d9b700 1 --2- 172.21.15.16:0/480597893 >> v2:172.21.15.49:3300/0 conn(0x7fa0dc0d2110 0x7fa0dc0ae1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:09.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.230+0000 7fa0dad9d700 1 -- 172.21.15.16:0/480597893 >> v2:172.21.15.49:3300/0 conn(0x7fa0dc0d2110 msgr2=0x7fa0dc0ae1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:09.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.230+0000 7fa0dad9d700 1 --2- 172.21.15.16:0/480597893 >> v2:172.21.15.49:3300/0 conn(0x7fa0dc0d2110 0x7fa0dc0ae1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:09.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.230+0000 7fa0dad9d700 1 -- 172.21.15.16:0/480597893 >> v1:172.21.15.16:6789/0 conn(0x7fa0dc0ce6a0 legacy=0x7fa0dc06c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:09.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.230+0000 7fa0dad9d700 1 -- 172.21.15.16:0/480597893 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0dc06bb00 con 0x7fa0dc06ca10 2023-10-16T13:48:09.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.230+0000 7fa0d959a700 1 -- 172.21.15.16:0/480597893 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 639281910 0 0) 0x7fa0dc0b20f0 con 0x7fa0dc0ce6a0 2023-10-16T13:48:09.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.230+0000 7fa0dad9d700 1 --2- 172.21.15.16:0/480597893 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0dc06ca10 0x7fa0dc0ae720 secure :-1 s=READY pgs=2927 cs=0 l=1 rev1=1 crypto rx=0x7fa0dc0cf0f0 tx=0x7fa0d400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52e7ba708b64d1a8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:09.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.230+0000 7fa0d959a700 1 -- 172.21.15.16:0/480597893 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0d400e070 con 0x7fa0dc06ca10 2023-10-16T13:48:09.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.230+0000 7fa0d959a700 1 -- 172.21.15.16:0/480597893 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0d400bbd0 con 0x7fa0dc06ca10 2023-10-16T13:48:09.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.230+0000 7fa0d959a700 1 -- 172.21.15.16:0/480597893 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0d4012430 con 0x7fa0dc06ca10 2023-10-16T13:48:09.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.231+0000 7fa0e0e29700 1 -- 172.21.15.16:0/480597893 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0dc06ca10 msgr2=0x7fa0dc0ae720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:09.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.231+0000 7fa0e0e29700 1 --2- 172.21.15.16:0/480597893 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0dc06ca10 0x7fa0dc0ae720 secure :-1 s=READY pgs=2927 cs=0 l=1 rev1=1 crypto rx=0x7fa0dc0cf0f0 tx=0x7fa0d400b330 comp rx=0 tx=0).stop 2023-10-16T13:48:09.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.231+0000 7fa0e0e29700 1 -- 172.21.15.16:0/480597893 shutdown_connections 2023-10-16T13:48:09.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.231+0000 7fa0e0e29700 1 --2- 172.21.15.16:0/480597893 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0dc06ca10 0x7fa0dc0ae720 unknown :-1 s=CLOSED pgs=2927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:09.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.231+0000 7fa0e0e29700 1 --2- 172.21.15.16:0/480597893 >> v2:172.21.15.49:3300/0 conn(0x7fa0dc0d2110 0x7fa0dc0ae1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:09.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.231+0000 7fa0e0e29700 1 -- 172.21.15.16:0/480597893 >> 172.21.15.16:0/480597893 conn(0x7fa0dc0d15f0 msgr2=0x7fa0dc0d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:09.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.231+0000 7fa0e0e29700 1 -- 172.21.15.16:0/480597893 shutdown_connections 2023-10-16T13:48:09.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.231+0000 7fa0e0e29700 1 -- 172.21.15.16:0/480597893 wait complete. 2023-10-16T13:48:09.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.232+0000 7fa0e0e29700 1 Processor -- start 2023-10-16T13:48:09.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.232+0000 7fa0e0e29700 1 -- start start 2023-10-16T13:48:09.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.233+0000 7fa0e0e29700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa0dc06ca10 0x7fa0dc154990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:09.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.233+0000 7fa0e0e29700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa0dc0ce6a0 0x7fa0dc0b64e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:09.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.233+0000 7fa0e0e29700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0dc0d2110 0x7fa0dc0b6a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:09.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.233+0000 7fa0e0e29700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0dc0b4bc0 con 0x7fa0dc0d2110 2023-10-16T13:48:09.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.233+0000 7fa0e0e29700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0dc0b4e20 con 0x7fa0dc0ce6a0 2023-10-16T13:48:09.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.233+0000 7fa0e0e29700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0dc0b5100 con 0x7fa0dc06ca10 2023-10-16T13:48:09.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.233+0000 7fa0dad9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0dc0d2110 0x7fa0dc0b6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:09.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.233+0000 7fa0dad9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0dc0d2110 0x7fa0dc0b6a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57046/0 (socket says 172.21.15.16:57046) 2023-10-16T13:48:09.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.233+0000 7fa0dad9d700 1 -- 172.21.15.16:0/3356765796 learned_addr learned my addr 172.21.15.16:0/3356765796 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:09.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.233+0000 7fa0d9d9b700 1 --2- 172.21.15.16:0/3356765796 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa0dc0ce6a0 0x7fa0dc0b64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:09.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.233+0000 7fa0da59c700 1 --2- 172.21.15.16:0/3356765796 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa0dc06ca10 0x7fa0dc154990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:09.242 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.233+0000 7fa0dad9d700 1 -- 172.21.15.16:0/3356765796 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa0dc06ca10 msgr2=0x7fa0dc154990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:09.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.233+0000 7fa0dad9d700 1 --2- 172.21.15.16:0/3356765796 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa0dc06ca10 0x7fa0dc154990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:09.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.233+0000 7fa0dad9d700 1 -- 172.21.15.16:0/3356765796 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa0dc0ce6a0 msgr2=0x7fa0dc0b64e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:09.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.233+0000 7fa0dad9d700 1 --2- 172.21.15.16:0/3356765796 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa0dc0ce6a0 0x7fa0dc0b64e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:09.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.233+0000 7fa0dad9d700 1 -- 172.21.15.16:0/3356765796 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0d4009a70 con 0x7fa0dc0d2110 2023-10-16T13:48:09.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.233+0000 7fa0da59c700 1 --2- 172.21.15.16:0/3356765796 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa0dc06ca10 0x7fa0dc154990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:48:09.243 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.233+0000 7fa0dad9d700 1 --2- 172.21.15.16:0/3356765796 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0dc0d2110 0x7fa0dc0b6a20 secure :-1 s=READY pgs=2928 cs=0 l=1 rev1=1 crypto rx=0x7fa0d4014040 tx=0x7fa0d4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:09.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.234+0000 7fa0d2ffd700 1 -- 172.21.15.16:0/3356765796 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0d4012430 con 0x7fa0dc0d2110 2023-10-16T13:48:09.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.234+0000 7fa0d2ffd700 1 -- 172.21.15.16:0/3356765796 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0d4020070 con 0x7fa0dc0d2110 2023-10-16T13:48:09.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.234+0000 7fa0e0e29700 1 -- 172.21.15.16:0/3356765796 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0dc0b53e0 con 0x7fa0dc0d2110 2023-10-16T13:48:09.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.234+0000 7fa0d2ffd700 1 -- 172.21.15.16:0/3356765796 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0d4004280 con 0x7fa0dc0d2110 2023-10-16T13:48:09.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.234+0000 7fa0e0e29700 1 -- 172.21.15.16:0/3356765796 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0dc0b59a0 con 0x7fa0dc0d2110 2023-10-16T13:48:09.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.235+0000 7fa0e0e29700 1 -- 172.21.15.16:0/3356765796 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0dc04ebb0 con 0x7fa0dc0d2110 2023-10-16T13:48:09.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.235+0000 7fa0d2ffd700 1 -- 172.21.15.16:0/3356765796 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fa0d40043e0 con 0x7fa0dc0d2110 2023-10-16T13:48:09.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.235+0000 7fa0d2ffd700 1 --2- 172.21.15.16:0/3356765796 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fa0bc06b140 0x7fa0bc06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:09.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.235+0000 7fa0d2ffd700 1 -- 172.21.15.16:0/3356765796 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(254..254 src has 1..254) v4 ==== 7115+0+0 (secure 0 0 0) 0x7fa0d4022070 con 0x7fa0dc0d2110 2023-10-16T13:48:09.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.236+0000 7fa0da59c700 1 --2- 172.21.15.16:0/3356765796 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fa0bc06b140 0x7fa0bc06d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:09.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.236+0000 7fa0da59c700 1 --2- 172.21.15.16:0/3356765796 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fa0bc06b140 0x7fa0bc06d600 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa0c4000f60 tx=0x7fa0c4007ca0 comp rx=0 tx=0).ready entity=mgr.12678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:09.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.240+0000 7fa0d2ffd700 1 -- 172.21.15.16:0/3356765796 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa0d40569e0 con 0x7fa0dc0d2110 2023-10-16T13:48:09.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.446+0000 7fa0e0e29700 1 -- 172.21.15.16:0/3356765796 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa0dc06c0e0 con 0x7fa0dc0d2110 2023-10-16T13:48:09.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.447+0000 7fa0d2ffd700 1 -- 172.21.15.16:0/3356765796 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v254) v1 ==== 107+0+11 (secure 0 0 0) 0x7fa0d405a000 con 0x7fa0dc0d2110 2023-10-16T13:48:09.449 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-10-16T13:48:09.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.449+0000 7fa0e0e29700 1 -- 172.21.15.16:0/3356765796 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fa0bc06b140 msgr2=0x7fa0bc06d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:09.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.449+0000 7fa0e0e29700 1 --2- 172.21.15.16:0/3356765796 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fa0bc06b140 0x7fa0bc06d600 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa0c4000f60 tx=0x7fa0c4007ca0 comp rx=0 tx=0).stop 2023-10-16T13:48:09.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.449+0000 7fa0e0e29700 1 -- 172.21.15.16:0/3356765796 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0dc0d2110 msgr2=0x7fa0dc0b6a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:09.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.449+0000 7fa0e0e29700 1 --2- 172.21.15.16:0/3356765796 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0dc0d2110 0x7fa0dc0b6a20 secure :-1 s=READY pgs=2928 cs=0 l=1 rev1=1 crypto rx=0x7fa0d4014040 tx=0x7fa0d4004ad0 comp rx=0 tx=0).stop 2023-10-16T13:48:09.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.449+0000 7fa0e0e29700 1 -- 172.21.15.16:0/3356765796 shutdown_connections 2023-10-16T13:48:09.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.449+0000 7fa0e0e29700 1 --2- 172.21.15.16:0/3356765796 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fa0bc06b140 0x7fa0bc06d600 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:09.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.449+0000 7fa0e0e29700 1 --2- 172.21.15.16:0/3356765796 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0dc0d2110 0x7fa0dc0b6a20 unknown :-1 s=CLOSED pgs=2928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:09.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.449+0000 7fa0e0e29700 1 --2- 172.21.15.16:0/3356765796 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa0dc06ca10 0x7fa0dc154990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:09.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.449+0000 7fa0e0e29700 1 --2- 172.21.15.16:0/3356765796 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa0dc0ce6a0 0x7fa0dc0b64e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:09.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.449+0000 7fa0e0e29700 1 -- 172.21.15.16:0/3356765796 >> 172.21.15.16:0/3356765796 conn(0x7fa0dc0d15f0 msgr2=0x7fa0dc160990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:09.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.449+0000 7fa0e0e29700 1 -- 172.21.15.16:0/3356765796 shutdown_connections 2023-10-16T13:48:09.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.449+0000 7fa0e0e29700 1 -- 172.21.15.16:0/3356765796 wait complete. 2023-10-16T13:48:09.464 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2023-10-16T13:48:09.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.590+0000 7fd4f6f70700 1 Processor -- start 2023-10-16T13:48:09.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.590+0000 7fd4f6f70700 1 -- start start 2023-10-16T13:48:09.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.590+0000 7fd4f6f70700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fd4f006e570 0x7fd4f015e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:09.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.590+0000 7fd4f6f70700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd4f006a7e0 0x7fd4f006ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:09.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.590+0000 7fd4f6f70700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4f015b6a0 con 0x7fd4f015b2c0 2023-10-16T13:48:09.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.590+0000 7fd4f6f70700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4f006b190 con 0x7fd4f006a7e0 2023-10-16T13:48:09.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.590+0000 7fd4f6f70700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4f0161b20 con 0x7fd4f006e570 2023-10-16T13:48:09.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.591+0000 7fd4f550d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd4f006a7e0 0x7fd4f006ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:09.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.591+0000 7fd4f550d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd4f006a7e0 0x7fd4f006ac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57054/0 (socket says 172.21.15.16:57054) 2023-10-16T13:48:09.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.591+0000 7fd4f550d700 1 -- 172.21.15.16:0/2146325833 learned_addr learned my addr 172.21.15.16:0/2146325833 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:09.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.591+0000 7fd4effff700 1 --2- 172.21.15.16:0/2146325833 >> v2:172.21.15.49:3300/0 conn(0x7fd4f006e570 0x7fd4f015e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:09.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.591+0000 7fd4f550d700 1 -- 172.21.15.16:0/2146325833 >> v2:172.21.15.49:3300/0 conn(0x7fd4f006e570 msgr2=0x7fd4f015e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:09.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.591+0000 7fd4f550d700 1 --2- 172.21.15.16:0/2146325833 >> v2:172.21.15.49:3300/0 conn(0x7fd4f006e570 0x7fd4f015e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:09.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.591+0000 7fd4f550d700 1 -- 172.21.15.16:0/2146325833 >> v1:172.21.15.49:6789/0 conn(0x7fd4f015b2c0 legacy=0x7fd4f0161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:09.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.591+0000 7fd4f550d700 1 -- 172.21.15.16:0/2146325833 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4f0161e00 con 0x7fd4f006a7e0 2023-10-16T13:48:09.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.591+0000 7fd4f550d700 1 --2- 172.21.15.16:0/2146325833 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4f006a7e0 0x7fd4f006ac50 secure :-1 s=READY pgs=2929 cs=0 l=1 rev1=1 crypto rx=0x7fd4f00d03c0 tx=0x7fd4e800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=263152b64796387 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:09.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.591+0000 7fd4ef7fe700 1 -- 172.21.15.16:0/2146325833 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4e800e070 con 0x7fd4f006a7e0 2023-10-16T13:48:09.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.591+0000 7fd4ef7fe700 1 -- 172.21.15.16:0/2146325833 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4e8003a20 con 0x7fd4f006a7e0 2023-10-16T13:48:09.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.591+0000 7fd4ef7fe700 1 -- 172.21.15.16:0/2146325833 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4e8003dd0 con 0x7fd4f006a7e0 2023-10-16T13:48:09.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.592+0000 7fd4f6f70700 1 -- 172.21.15.16:0/2146325833 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4f006a7e0 msgr2=0x7fd4f006ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:09.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.592+0000 7fd4f6f70700 1 --2- 172.21.15.16:0/2146325833 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4f006a7e0 0x7fd4f006ac50 secure :-1 s=READY pgs=2929 cs=0 l=1 rev1=1 crypto rx=0x7fd4f00d03c0 tx=0x7fd4e800b330 comp rx=0 tx=0).stop 2023-10-16T13:48:09.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.592+0000 7fd4f6f70700 1 -- 172.21.15.16:0/2146325833 shutdown_connections 2023-10-16T13:48:09.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.592+0000 7fd4f6f70700 1 --2- 172.21.15.16:0/2146325833 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4f006a7e0 0x7fd4f006ac50 unknown :-1 s=CLOSED pgs=2929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:09.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.592+0000 7fd4f6f70700 1 --2- 172.21.15.16:0/2146325833 >> v2:172.21.15.49:3300/0 conn(0x7fd4f006e570 0x7fd4f015e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:09.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.592+0000 7fd4f6f70700 1 -- 172.21.15.16:0/2146325833 >> 172.21.15.16:0/2146325833 conn(0x7fd4f00b1f40 msgr2=0x7fd4f00b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:09.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.592+0000 7fd4f6f70700 1 -- 172.21.15.16:0/2146325833 shutdown_connections 2023-10-16T13:48:09.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.592+0000 7fd4f6f70700 1 -- 172.21.15.16:0/2146325833 wait complete. 2023-10-16T13:48:09.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.593+0000 7fd4f6f70700 1 Processor -- start 2023-10-16T13:48:09.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.593+0000 7fd4f6f70700 1 -- start start 2023-10-16T13:48:09.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.594+0000 7fd4f6f70700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd4f006a7e0 0x7fd4f014faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:09.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.594+0000 7fd4f6f70700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd4f006e570 0x7fd4f0155200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:09.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.594+0000 7fd4f6f70700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4f015b2c0 0x7fd4f0155740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:09.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.594+0000 7fd4f6f70700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f00c6ad0 con 0x7fd4f015b2c0 2023-10-16T13:48:09.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.594+0000 7fd4f6f70700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f01613f0 con 0x7fd4f006a7e0 2023-10-16T13:48:09.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.594+0000 7fd4f6f70700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f015e5e0 con 0x7fd4f006e570 2023-10-16T13:48:09.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.594+0000 7fd4f550d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4f015b2c0 0x7fd4f0155740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:09.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.594+0000 7fd4f550d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4f015b2c0 0x7fd4f0155740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57060/0 (socket says 172.21.15.16:57060) 2023-10-16T13:48:09.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.594+0000 7fd4f550d700 1 -- 172.21.15.16:0/2674493851 learned_addr learned my addr 172.21.15.16:0/2674493851 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:09.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.594+0000 7fd4f4d0c700 1 --2- 172.21.15.16:0/2674493851 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd4f006a7e0 0x7fd4f014faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:09.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.594+0000 7fd4effff700 1 --2- 172.21.15.16:0/2674493851 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd4f006e570 0x7fd4f0155200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:09.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.594+0000 7fd4f550d700 1 -- 172.21.15.16:0/2674493851 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd4f006e570 msgr2=0x7fd4f0155200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:09.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.594+0000 7fd4f550d700 1 --2- 172.21.15.16:0/2674493851 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd4f006e570 0x7fd4f0155200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:09.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.594+0000 7fd4f550d700 1 -- 172.21.15.16:0/2674493851 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd4f006a7e0 msgr2=0x7fd4f014faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:09.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.594+0000 7fd4f550d700 1 --2- 172.21.15.16:0/2674493851 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd4f006a7e0 0x7fd4f014faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:09.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.594+0000 7fd4f550d700 1 -- 172.21.15.16:0/2674493851 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4e8009b30 con 0x7fd4f015b2c0 2023-10-16T13:48:09.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.594+0000 7fd4f550d700 1 --2- 172.21.15.16:0/2674493851 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4f015b2c0 0x7fd4f0155740 secure :-1 s=READY pgs=2930 cs=0 l=1 rev1=1 crypto rx=0x7fd4e800b370 tx=0x7fd4e8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:09.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.595+0000 7fd4ed7fa700 1 -- 172.21.15.16:0/2674493851 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4e8004440 con 0x7fd4f015b2c0 2023-10-16T13:48:09.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.595+0000 7fd4ed7fa700 1 -- 172.21.15.16:0/2674493851 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4e80045c0 con 0x7fd4f015b2c0 2023-10-16T13:48:09.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.595+0000 7fd4f6f70700 1 -- 172.21.15.16:0/2674493851 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4f015e8c0 con 0x7fd4f015b2c0 2023-10-16T13:48:09.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.595+0000 7fd4ed7fa700 1 -- 172.21.15.16:0/2674493851 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4e8004970 con 0x7fd4f015b2c0 2023-10-16T13:48:09.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.595+0000 7fd4f6f70700 1 -- 172.21.15.16:0/2674493851 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4f015ee80 con 0x7fd4f015b2c0 2023-10-16T13:48:09.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.596+0000 7fd4f6f70700 1 -- 172.21.15.16:0/2674493851 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4f004ebb0 con 0x7fd4f015b2c0 2023-10-16T13:48:09.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.596+0000 7fd4ed7fa700 1 -- 172.21.15.16:0/2674493851 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fd4e80093c0 con 0x7fd4f015b2c0 2023-10-16T13:48:09.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.597+0000 7fd4ed7fa700 1 --2- 172.21.15.16:0/2674493851 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fd4d406b140 0x7fd4d406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:09.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.597+0000 7fd4ed7fa700 1 -- 172.21.15.16:0/2674493851 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(254..254 src has 1..254) v4 ==== 7115+0+0 (secure 0 0 0) 0x7fd4e808b3b0 con 0x7fd4f015b2c0 2023-10-16T13:48:09.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.597+0000 7fd4f4d0c700 1 --2- 172.21.15.16:0/2674493851 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fd4d406b140 0x7fd4d406d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:09.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.597+0000 7fd4f4d0c700 1 --2- 172.21.15.16:0/2674493851 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fd4d406b140 0x7fd4d406d600 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd4f01f6880 tx=0x7fd4e4007ec0 comp rx=0 tx=0).ready entity=mgr.12678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:09.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.601+0000 7fd4ed7fa700 1 -- 172.21.15.16:0/2674493851 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd4e8059590 con 0x7fd4f015b2c0 2023-10-16T13:48:09.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.803+0000 7fd4f6f70700 1 -- 172.21.15.16:0/2674493851 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) v1 -- 0x7fd4f0161790 con 0x7fd4f015b2c0 2023-10-16T13:48:09.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.803+0000 7fd4ed7fa700 1 -- 172.21.15.16:0/2674493851 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v254) v1 ==== 103+0+11 (secure 0 0 0) 0x7fd4e8059120 con 0x7fd4f015b2c0 2023-10-16T13:48:09.806 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-10-16T13:48:09.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.805+0000 7fd4f6f70700 1 -- 172.21.15.16:0/2674493851 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fd4d406b140 msgr2=0x7fd4d406d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:09.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.805+0000 7fd4f6f70700 1 --2- 172.21.15.16:0/2674493851 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fd4d406b140 0x7fd4d406d600 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd4f01f6880 tx=0x7fd4e4007ec0 comp rx=0 tx=0).stop 2023-10-16T13:48:09.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.805+0000 7fd4f6f70700 1 -- 172.21.15.16:0/2674493851 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4f015b2c0 msgr2=0x7fd4f0155740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:09.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.805+0000 7fd4f6f70700 1 --2- 172.21.15.16:0/2674493851 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4f015b2c0 0x7fd4f0155740 secure :-1 s=READY pgs=2930 cs=0 l=1 rev1=1 crypto rx=0x7fd4e800b370 tx=0x7fd4e8004ad0 comp rx=0 tx=0).stop 2023-10-16T13:48:09.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.805+0000 7fd4f6f70700 1 -- 172.21.15.16:0/2674493851 shutdown_connections 2023-10-16T13:48:09.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.805+0000 7fd4f6f70700 1 --2- 172.21.15.16:0/2674493851 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fd4d406b140 0x7fd4d406d600 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:09.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.805+0000 7fd4f6f70700 1 --2- 172.21.15.16:0/2674493851 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4f015b2c0 0x7fd4f0155740 unknown :-1 s=CLOSED pgs=2930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:09.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.805+0000 7fd4f6f70700 1 --2- 172.21.15.16:0/2674493851 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd4f006a7e0 0x7fd4f014faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:09.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.805+0000 7fd4f6f70700 1 --2- 172.21.15.16:0/2674493851 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd4f006e570 0x7fd4f0155200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:09.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.805+0000 7fd4f6f70700 1 -- 172.21.15.16:0/2674493851 >> 172.21.15.16:0/2674493851 conn(0x7fd4f00b1f40 msgr2=0x7fd4f015bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:09.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.805+0000 7fd4f6f70700 1 -- 172.21.15.16:0/2674493851 shutdown_connections 2023-10-16T13:48:09.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.805+0000 7fd4f6f70700 1 -- 172.21.15.16:0/2674493851 wait complete. 2023-10-16T13:48:09.820 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:48:09.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.942+0000 7f95a98e9700 1 Processor -- start 2023-10-16T13:48:09.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.942+0000 7f95a98e9700 1 -- start start 2023-10-16T13:48:09.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.943+0000 7f95a98e9700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f95a415b860 0x7f95a406a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:09.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.943+0000 7f95a98e9700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f95a40b4490 0x7f95a406ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:09.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.943+0000 7f95a98e9700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95a40d11c0 con 0x7f95a40b2be0 2023-10-16T13:48:09.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.943+0000 7f95a98e9700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95a40d03d0 con 0x7f95a415b860 2023-10-16T13:48:09.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.943+0000 7f95a98e9700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95a40cd360 con 0x7f95a40b4490 2023-10-16T13:48:09.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.943+0000 7f95a27fc700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f95a40b2be0 0x7f95a40719e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46338/0 (socket says 172.21.15.16:46338) 2023-10-16T13:48:09.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.943+0000 7f95a27fc700 1 -- 172.21.15.16:0/3916989266 learned_addr learned my addr 172.21.15.16:0/3916989266 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:48:09.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.943+0000 7f95a37fe700 1 --2- 172.21.15.16:0/3916989266 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f95a40b4490 0x7f95a406ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:09.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.943+0000 7f95a2ffd700 1 --2- 172.21.15.16:0/3916989266 >> v2:172.21.15.49:3300/0 conn(0x7f95a415b860 0x7f95a406a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:09.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.944+0000 7f95a1ffb700 1 -- 172.21.15.16:0/3916989266 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1594542593 0 0) 0x7f95a40d11c0 con 0x7f95a40b2be0 2023-10-16T13:48:09.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.944+0000 7f95a1ffb700 1 -- 172.21.15.16:0/3916989266 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f958c0035b0 con 0x7f95a40b2be0 2023-10-16T13:48:09.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.944+0000 7f95a2ffd700 1 -- 172.21.15.16:0/3916989266 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f95a40b4490 msgr2=0x7f95a406ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:09.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.944+0000 7f95a2ffd700 1 --2- 172.21.15.16:0/3916989266 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f95a40b4490 0x7f95a406ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:09.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.944+0000 7f95a2ffd700 1 -- 172.21.15.16:0/3916989266 >> v1:172.21.15.16:6789/0 conn(0x7f95a40b2be0 legacy=0x7f95a40719e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:09.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.944+0000 7f95a2ffd700 1 -- 172.21.15.16:0/3916989266 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95a40d05b0 con 0x7f95a415b860 2023-10-16T13:48:09.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.944+0000 7f95a37fe700 1 --2- 172.21.15.16:0/3916989266 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f95a40b4490 0x7f95a406ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:48:09.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.944+0000 7f95a2ffd700 1 --2- 172.21.15.16:0/3916989266 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f95a415b860 0x7f95a406a7e0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f9598009840 tx=0x7f9598009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=548c609246003f2c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:09.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.944+0000 7f95a1ffb700 1 -- 172.21.15.16:0/3916989266 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9598011830 con 0x7f95a415b860 2023-10-16T13:48:09.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.944+0000 7f95a1ffb700 1 -- 172.21.15.16:0/3916989266 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9598011990 con 0x7f95a415b860 2023-10-16T13:48:09.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.944+0000 7f95a1ffb700 1 -- 172.21.15.16:0/3916989266 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9598011d20 con 0x7f95a415b860 2023-10-16T13:48:09.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.945+0000 7f95a98e9700 1 -- 172.21.15.16:0/3916989266 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f95a415b860 msgr2=0x7f95a406a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:09.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.945+0000 7f95a98e9700 1 --2- 172.21.15.16:0/3916989266 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f95a415b860 0x7f95a406a7e0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f9598009840 tx=0x7f9598009b50 comp rx=0 tx=0).stop 2023-10-16T13:48:09.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.945+0000 7f95a98e9700 1 -- 172.21.15.16:0/3916989266 shutdown_connections 2023-10-16T13:48:09.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.945+0000 7f95a98e9700 1 --2- 172.21.15.16:0/3916989266 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f95a40b4490 0x7f95a406ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:09.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.945+0000 7f95a98e9700 1 --2- 172.21.15.16:0/3916989266 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f95a415b860 0x7f95a406a7e0 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:09.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.945+0000 7f95a98e9700 1 -- 172.21.15.16:0/3916989266 >> 172.21.15.16:0/3916989266 conn(0x7f95a40b1fd0 msgr2=0x7f95a40b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:09.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.945+0000 7f95a98e9700 1 -- 172.21.15.16:0/3916989266 shutdown_connections 2023-10-16T13:48:09.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.945+0000 7f95a98e9700 1 -- 172.21.15.16:0/3916989266 wait complete. 2023-10-16T13:48:09.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.946+0000 7f95a98e9700 1 Processor -- start 2023-10-16T13:48:09.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.946+0000 7f95a98e9700 1 -- start start 2023-10-16T13:48:09.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.947+0000 7f95a98e9700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f95a40b2be0 0x7f95a4158a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:09.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.947+0000 7f95a98e9700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f95a40b4490 0x7f95a41f43a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:09.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.947+0000 7f95a98e9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95a415b860 0x7f95a40c9a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:09.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.947+0000 7f95a98e9700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95a4158f80 con 0x7f95a415b860 2023-10-16T13:48:09.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.947+0000 7f95a98e9700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95a41f48e0 con 0x7f95a40b4490 2023-10-16T13:48:09.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.947+0000 7f95a98e9700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95a40cf0a0 con 0x7f95a40b2be0 2023-10-16T13:48:09.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.947+0000 7f95a37fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95a415b860 0x7f95a40c9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:09.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.947+0000 7f95a37fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95a415b860 0x7f95a40c9a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57074/0 (socket says 172.21.15.16:57074) 2023-10-16T13:48:09.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.947+0000 7f95a37fe700 1 -- 172.21.15.16:0/602866428 learned_addr learned my addr 172.21.15.16:0/602866428 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:09.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.947+0000 7f95a2ffd700 1 --2- 172.21.15.16:0/602866428 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f95a40b2be0 0x7f95a4158a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:09.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.947+0000 7f95a27fc700 1 --2- 172.21.15.16:0/602866428 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f95a40b4490 0x7f95a41f43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:09.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.948+0000 7f95a37fe700 1 -- 172.21.15.16:0/602866428 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f95a40b2be0 msgr2=0x7f95a4158a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:09.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.948+0000 7f95a37fe700 1 --2- 172.21.15.16:0/602866428 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f95a40b2be0 0x7f95a4158a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:09.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.948+0000 7f95a37fe700 1 -- 172.21.15.16:0/602866428 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f95a40b4490 msgr2=0x7f95a41f43a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:09.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.948+0000 7f95a37fe700 1 --2- 172.21.15.16:0/602866428 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f95a40b4490 0x7f95a41f43a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:09.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.948+0000 7f95a37fe700 1 -- 172.21.15.16:0/602866428 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f959c009a70 con 0x7f95a415b860 2023-10-16T13:48:09.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.948+0000 7f95a37fe700 1 --2- 172.21.15.16:0/602866428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95a415b860 0x7f95a40c9a90 secure :-1 s=READY pgs=2932 cs=0 l=1 rev1=1 crypto rx=0x7f959c00efc0 tx=0x7f959c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:09.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.948+0000 7f958bfff700 1 -- 172.21.15.16:0/602866428 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f959c00cc90 con 0x7f95a415b860 2023-10-16T13:48:09.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.948+0000 7f958bfff700 1 -- 172.21.15.16:0/602866428 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f959c00f040 con 0x7f95a415b860 2023-10-16T13:48:09.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.948+0000 7f95a98e9700 1 -- 172.21.15.16:0/602866428 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95980094a0 con 0x7f95a415b860 2023-10-16T13:48:09.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.948+0000 7f958bfff700 1 -- 172.21.15.16:0/602866428 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f959c014480 con 0x7f95a415b860 2023-10-16T13:48:09.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.948+0000 7f95a98e9700 1 -- 172.21.15.16:0/602866428 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95a40cf660 con 0x7f95a415b860 2023-10-16T13:48:09.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.949+0000 7f95a98e9700 1 -- 172.21.15.16:0/602866428 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95a404ebb0 con 0x7f95a415b860 2023-10-16T13:48:09.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.950+0000 7f958bfff700 1 -- 172.21.15.16:0/602866428 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f959c0145e0 con 0x7f95a415b860 2023-10-16T13:48:09.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.950+0000 7f958bfff700 1 --2- 172.21.15.16:0/602866428 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f959006b140 0x7f959006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:09.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.950+0000 7f958bfff700 1 -- 172.21.15.16:0/602866428 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(254..254 src has 1..254) v4 ==== 7115+0+0 (secure 0 0 0) 0x7f959c08a8f0 con 0x7f95a415b860 2023-10-16T13:48:09.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.950+0000 7f95a2ffd700 1 --2- 172.21.15.16:0/602866428 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f959006b140 0x7f959006d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:09.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.950+0000 7f95a2ffd700 1 --2- 172.21.15.16:0/602866428 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f959006b140 0x7f959006d600 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f9598000c00 tx=0x7f9598008030 comp rx=0 tx=0).ready entity=mgr.12678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:09.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:09.954+0000 7f958bfff700 1 -- 172.21.15.16:0/602866428 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f959c055040 con 0x7f95a415b860 2023-10-16T13:48:10.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.204+0000 7f95a98e9700 1 -- 172.21.15.16:0/602866428 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f95a415ecd0 con 0x7f95a415b860 2023-10-16T13:48:10.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.205+0000 7f958bfff700 1 -- 172.21.15.16:0/602866428 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 137 v137) v1 ==== 95+0+4928 (secure 0 0 0) 0x7f959c009cd0 con 0x7f95a415b860 2023-10-16T13:48:10.207 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:48:10.208 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":137,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9596,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/3939647668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":3939647668},{"type":"v1","addr":"172.21.15.49:6834","nonce":3939647668}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":125},{"gid":11185,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/2890113513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2890113513},{"type":"v1","addr":"172.21.15.16:6836","nonce":2890113513}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":125},{"gid":12864,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/1181609271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":1181609271},{"type":"v1","addr":"172.21.15.16:6837","nonce":1181609271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":137}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:47:41.332096+0000","modified":"2023-10-16T13:47:44.368830+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11197},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11197":{"gid":11197,"name":"d","rank":0,"incarnation":132,"state":"up:active","state_seq":7,"addr":"172.21.15.49:6835/680748068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":680748068},{"type":"v1","addr":"172.21.15.49:6835","nonce":680748068}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2023-10-16T13:48:10.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.207+0000 7f95a98e9700 1 -- 172.21.15.16:0/602866428 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f959006b140 msgr2=0x7f959006d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:10.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.207+0000 7f95a98e9700 1 --2- 172.21.15.16:0/602866428 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f959006b140 0x7f959006d600 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f9598000c00 tx=0x7f9598008030 comp rx=0 tx=0).stop 2023-10-16T13:48:10.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.207+0000 7f95a98e9700 1 -- 172.21.15.16:0/602866428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95a415b860 msgr2=0x7f95a40c9a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:10.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.207+0000 7f95a98e9700 1 --2- 172.21.15.16:0/602866428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95a415b860 0x7f95a40c9a90 secure :-1 s=READY pgs=2932 cs=0 l=1 rev1=1 crypto rx=0x7f959c00efc0 tx=0x7f959c00c3a0 comp rx=0 tx=0).stop 2023-10-16T13:48:10.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.207+0000 7f95a98e9700 1 -- 172.21.15.16:0/602866428 shutdown_connections 2023-10-16T13:48:10.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.207+0000 7f95a98e9700 1 --2- 172.21.15.16:0/602866428 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f959006b140 0x7f959006d600 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:10.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.207+0000 7f95a98e9700 1 --2- 172.21.15.16:0/602866428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95a415b860 0x7f95a40c9a90 unknown :-1 s=CLOSED pgs=2932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:10.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.207+0000 7f95a98e9700 1 --2- 172.21.15.16:0/602866428 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f95a40b2be0 0x7f95a4158a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:10.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.207+0000 7f95a98e9700 1 --2- 172.21.15.16:0/602866428 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f95a40b4490 0x7f95a41f43a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:10.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.207+0000 7f95a98e9700 1 -- 172.21.15.16:0/602866428 >> 172.21.15.16:0/602866428 conn(0x7f95a40b1fd0 msgr2=0x7f95a415e460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:10.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.207+0000 7f95a98e9700 1 -- 172.21.15.16:0/602866428 shutdown_connections 2023-10-16T13:48:10.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.207+0000 7f95a98e9700 1 -- 172.21.15.16:0/602866428 wait complete. 2023-10-16T13:48:10.212 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 137 2023-10-16T13:48:10.222 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:48:10.347 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.345+0000 7fe15de36700 1 Processor -- start 2023-10-16T13:48:10.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.345+0000 7fe15de36700 1 -- start start 2023-10-16T13:48:10.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.346+0000 7fe15de36700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe15815b280 0x7fe1581613e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:10.350 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.346+0000 7fe15de36700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fe1580cc8a0 0x7fe15815e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:10.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.346+0000 7fe15de36700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe158161c30 con 0x7fe15806a7e0 2023-10-16T13:48:10.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.346+0000 7fe15de36700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe158161f10 con 0x7fe1580cc8a0 2023-10-16T13:48:10.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.346+0000 7fe15de36700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1581621f0 con 0x7fe15815b280 2023-10-16T13:48:10.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.346+0000 7fe1577fe700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe15815b280 0x7fe1581613e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:10.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.346+0000 7fe156ffd700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fe1580cc8a0 0x7fe15815e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:10.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.346+0000 7fe156ffd700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fe1580cc8a0 0x7fe15815e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:56142/0 (socket says 172.21.15.16:56142) 2023-10-16T13:48:10.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.346+0000 7fe1577fe700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe15815b280 0x7fe1581613e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:45336/0 (socket says 172.21.15.16:45336) 2023-10-16T13:48:10.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.346+0000 7fe156ffd700 1 -- 172.21.15.16:0/159264837 learned_addr learned my addr 172.21.15.16:0/159264837 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:10.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.346+0000 7fe157fff700 1 --1- 172.21.15.16:0/159264837 >> v1:172.21.15.49:6789/0 conn(0x7fe15806a7e0 0x7fe1580d02b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:53538/0 (socket says 172.21.15.16:53538) 2023-10-16T13:48:10.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.346+0000 7fe156ffd700 1 -- 172.21.15.16:0/159264837 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe15815b280 msgr2=0x7fe1581613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:10.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.346+0000 7fe156ffd700 1 --2- 172.21.15.16:0/159264837 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe15815b280 0x7fe1581613e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:10.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.346+0000 7fe156ffd700 1 -- 172.21.15.16:0/159264837 >> v1:172.21.15.49:6789/0 conn(0x7fe15806a7e0 legacy=0x7fe1580d02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:10.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.346+0000 7fe156ffd700 1 -- 172.21.15.16:0/159264837 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe15815df00 con 0x7fe1580cc8a0 2023-10-16T13:48:10.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.347+0000 7fe156ffd700 1 --2- 172.21.15.16:0/159264837 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe1580cc8a0 0x7fe15815e700 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fe1400098f0 tx=0x7fe140009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c522f347deb4a71 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:10.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.347+0000 7fe1567fc700 1 -- 172.21.15.16:0/159264837 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe140011830 con 0x7fe1580cc8a0 2023-10-16T13:48:10.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.347+0000 7fe1567fc700 1 -- 172.21.15.16:0/159264837 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe140011990 con 0x7fe1580cc8a0 2023-10-16T13:48:10.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.347+0000 7fe1567fc700 1 -- 172.21.15.16:0/159264837 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe140011d20 con 0x7fe1580cc8a0 2023-10-16T13:48:10.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.347+0000 7fe15de36700 1 -- 172.21.15.16:0/159264837 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe1580cc8a0 msgr2=0x7fe15815e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:10.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.347+0000 7fe15de36700 1 --2- 172.21.15.16:0/159264837 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe1580cc8a0 0x7fe15815e700 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fe1400098f0 tx=0x7fe140009c00 comp rx=0 tx=0).stop 2023-10-16T13:48:10.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.347+0000 7fe15de36700 1 -- 172.21.15.16:0/159264837 shutdown_connections 2023-10-16T13:48:10.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.347+0000 7fe15de36700 1 --2- 172.21.15.16:0/159264837 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe15815b280 0x7fe1581613e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:10.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.347+0000 7fe15de36700 1 --2- 172.21.15.16:0/159264837 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe1580cc8a0 0x7fe15815e700 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:10.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.347+0000 7fe15de36700 1 -- 172.21.15.16:0/159264837 >> 172.21.15.16:0/159264837 conn(0x7fe1580b1fd0 msgr2=0x7fe1580b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:10.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.347+0000 7fe15de36700 1 -- 172.21.15.16:0/159264837 shutdown_connections 2023-10-16T13:48:10.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.347+0000 7fe15de36700 1 -- 172.21.15.16:0/159264837 wait complete. 2023-10-16T13:48:10.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.349+0000 7fe15de36700 1 Processor -- start 2023-10-16T13:48:10.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.349+0000 7fe15de36700 1 -- start start 2023-10-16T13:48:10.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.350+0000 7fe15de36700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe15806a7e0 0x7fe1580c9de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:10.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.350+0000 7fe15de36700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1580cc8a0 0x7fe15806ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:10.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.350+0000 7fe15de36700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe15815b280 0x7fe15806b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:10.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.350+0000 7fe15de36700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1580c7510 con 0x7fe1580cc8a0 2023-10-16T13:48:10.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.350+0000 7fe15de36700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1580ca320 con 0x7fe15815b280 2023-10-16T13:48:10.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.350+0000 7fe15de36700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1581f7250 con 0x7fe15806a7e0 2023-10-16T13:48:10.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.350+0000 7fe156ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1580cc8a0 0x7fe15806ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:10.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.350+0000 7fe156ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1580cc8a0 0x7fe15806ac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57086/0 (socket says 172.21.15.16:57086) 2023-10-16T13:48:10.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.350+0000 7fe156ffd700 1 -- 172.21.15.16:0/2071277633 learned_addr learned my addr 172.21.15.16:0/2071277633 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:10.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.350+0000 7fe1577fe700 1 --2- 172.21.15.16:0/2071277633 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe15806a7e0 0x7fe1580c9de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:10.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.350+0000 7fe157fff700 1 --2- 172.21.15.16:0/2071277633 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe15815b280 0x7fe15806b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:10.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.350+0000 7fe156ffd700 1 -- 172.21.15.16:0/2071277633 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe15806a7e0 msgr2=0x7fe1580c9de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:10.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.350+0000 7fe156ffd700 1 --2- 172.21.15.16:0/2071277633 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe15806a7e0 0x7fe1580c9de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:10.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.350+0000 7fe156ffd700 1 -- 172.21.15.16:0/2071277633 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe15815b280 msgr2=0x7fe15806b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:10.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.350+0000 7fe156ffd700 1 --2- 172.21.15.16:0/2071277633 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe15815b280 0x7fe15806b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:10.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.350+0000 7fe156ffd700 1 -- 172.21.15.16:0/2071277633 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe140009550 con 0x7fe1580cc8a0 2023-10-16T13:48:10.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.350+0000 7fe157fff700 1 --2- 172.21.15.16:0/2071277633 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe15815b280 0x7fe15806b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:48:10.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.350+0000 7fe156ffd700 1 --2- 172.21.15.16:0/2071277633 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1580cc8a0 0x7fe15806ac30 secure :-1 s=READY pgs=2933 cs=0 l=1 rev1=1 crypto rx=0x7fe140000c00 tx=0x7fe140004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:10.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.350+0000 7fe14ffff700 1 -- 172.21.15.16:0/2071277633 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe140004480 con 0x7fe1580cc8a0 2023-10-16T13:48:10.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.350+0000 7fe14ffff700 1 -- 172.21.15.16:0/2071277633 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe14001f070 con 0x7fe1580cc8a0 2023-10-16T13:48:10.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.350+0000 7fe14ffff700 1 -- 172.21.15.16:0/2071277633 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1400038c0 con 0x7fe1580cc8a0 2023-10-16T13:48:10.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.350+0000 7fe15de36700 1 -- 172.21.15.16:0/2071277633 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1580c7760 con 0x7fe1580cc8a0 2023-10-16T13:48:10.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.350+0000 7fe15de36700 1 -- 172.21.15.16:0/2071277633 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1580c7d20 con 0x7fe1580cc8a0 2023-10-16T13:48:10.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.351+0000 7fe15de36700 1 -- 172.21.15.16:0/2071277633 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe15804ebb0 con 0x7fe1580cc8a0 2023-10-16T13:48:10.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.352+0000 7fe14ffff700 1 -- 172.21.15.16:0/2071277633 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fe140021030 con 0x7fe1580cc8a0 2023-10-16T13:48:10.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.352+0000 7fe14ffff700 1 --2- 172.21.15.16:0/2071277633 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fe13806b140 0x7fe13806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:10.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.352+0000 7fe14ffff700 1 -- 172.21.15.16:0/2071277633 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(254..254 src has 1..254) v4 ==== 7115+0+0 (secure 0 0 0) 0x7fe14008b350 con 0x7fe1580cc8a0 2023-10-16T13:48:10.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.352+0000 7fe1577fe700 1 --2- 172.21.15.16:0/2071277633 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fe13806b140 0x7fe13806d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:10.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.353+0000 7fe1577fe700 1 --2- 172.21.15.16:0/2071277633 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fe13806b140 0x7fe13806d600 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe148009b40 tx=0x7fe148009210 comp rx=0 tx=0).ready entity=mgr.12678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:10.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.357+0000 7fe14ffff700 1 -- 172.21.15.16:0/2071277633 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe140055cd0 con 0x7fe1580cc8a0 2023-10-16T13:48:10.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.563+0000 7fe15de36700 1 -- 172.21.15.16:0/2071277633 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe1580c3ad0 con 0x7fe1580cc8a0 2023-10-16T13:48:10.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.565+0000 7fe14ffff700 1 -- 172.21.15.16:0/2071277633 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v254) v1 ==== 74+0+18205 (secure 0 0 0) 0x7fe1400592f0 con 0x7fe1580cc8a0 2023-10-16T13:48:10.568 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:48:10.568 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":254,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:48:08.349411+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T13:47:39.075528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"248","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"backup_fs_data","create_time":"2023-10-16T13:47:40.314029+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"249","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6835/303278795":"2023-10-17T13:48:05.033844+0000","172.21.15.16:6832/2221624769":"2023-10-17T13:48:02.485614+0000","172.21.15.16:0/2242986614":"2023-10-17T13:48:02.485614+0000","172.21.15.16:0/3610227254":"2023-10-17T13:48:02.485614+0000","172.21.15.16:0/1182888331":"2023-10-17T13:48:02.485614+0000","172.21.15.16:0/2095012141":"2023-10-17T13:48:02.485614+0000","172.21.15.16:0/874722077":"2023-10-17T13:48:02.485614+0000","172.21.15.16:0/2066401556":"2023-10-17T13:48:02.485614+0000","172.21.15.16:0/3947601914":"2023-10-17T13:47:54.816700+0000","172.21.15.16:6833/2221624769":"2023-10-17T13:48:02.485614+0000","172.21.15.16:0/2759470534":"2023-10-17T13:47:54.816700+0000","172.21.15.16:0/2634105683":"2023-10-17T13:48:02.485614+0000","172.21.15.16:0/2974892940":"2023-10-17T13:47:54.816700+0000","172.21.15.16:6833/1736104576":"2023-10-17T13:47:54.816700+0000","172.21.15.16:6832/1736104576":"2023-10-17T13:47:54.816700+0000","172.21.15.16:0/1246409088":"2023-10-17T13:48:02.485614+0000","172.21.15.16:0/318862437":"2023-10-17T13:47:54.816700+0000","172.21.15.16:0/413581668":"2023-10-17T13:47:54.816700+0000","172.21.15.16:6837/303278795":"2023-10-17T13:48:05.033844+0000","172.21.15.16:0/3740515383":"2023-10-17T13:47:54.816700+0000","172.21.15.16:0/3105924907":"2023-10-17T13:47:54.816700+0000","172.21.15.16:0/3617558038":"2023-10-17T13:47:54.816700+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:48:10.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.567+0000 7fe15de36700 1 -- 172.21.15.16:0/2071277633 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fe13806b140 msgr2=0x7fe13806d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:10.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.567+0000 7fe15de36700 1 --2- 172.21.15.16:0/2071277633 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fe13806b140 0x7fe13806d600 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe148009b40 tx=0x7fe148009210 comp rx=0 tx=0).stop 2023-10-16T13:48:10.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.567+0000 7fe15de36700 1 -- 172.21.15.16:0/2071277633 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1580cc8a0 msgr2=0x7fe15806ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:10.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.567+0000 7fe15de36700 1 --2- 172.21.15.16:0/2071277633 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1580cc8a0 0x7fe15806ac30 secure :-1 s=READY pgs=2933 cs=0 l=1 rev1=1 crypto rx=0x7fe140000c00 tx=0x7fe140004750 comp rx=0 tx=0).stop 2023-10-16T13:48:10.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.567+0000 7fe15de36700 1 -- 172.21.15.16:0/2071277633 shutdown_connections 2023-10-16T13:48:10.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.567+0000 7fe15de36700 1 --2- 172.21.15.16:0/2071277633 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fe13806b140 0x7fe13806d600 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:10.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.567+0000 7fe15de36700 1 --2- 172.21.15.16:0/2071277633 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe15815b280 0x7fe15806b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:10.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.567+0000 7fe15de36700 1 --2- 172.21.15.16:0/2071277633 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe15806a7e0 0x7fe1580c9de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:10.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.567+0000 7fe15de36700 1 --2- 172.21.15.16:0/2071277633 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1580cc8a0 0x7fe15806ac30 unknown :-1 s=CLOSED pgs=2933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:10.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.567+0000 7fe15de36700 1 -- 172.21.15.16:0/2071277633 >> 172.21.15.16:0/2071277633 conn(0x7fe1580b1fd0 msgr2=0x7fe1580cb9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:10.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.567+0000 7fe15de36700 1 -- 172.21.15.16:0/2071277633 shutdown_connections 2023-10-16T13:48:10.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.567+0000 7fe15de36700 1 -- 172.21.15.16:0/2071277633 wait complete. 2023-10-16T13:48:10.582 INFO:tasks.cephfs.filesystem:Destroying file system backup_fs and related pools 2023-10-16T13:48:10.582 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:48:10.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.704+0000 7f9bdae19700 1 Processor -- start 2023-10-16T13:48:10.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.704+0000 7f9bdae19700 1 -- start start 2023-10-16T13:48:10.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.705+0000 7f9bdae19700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f9bd415b280 0x7f9bd41613e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:10.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.705+0000 7f9bdae19700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9bd406a7e0 0x7f9bd406ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:10.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.705+0000 7f9bdae19700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bd406b150 con 0x7f9bd40cc8a0 2023-10-16T13:48:10.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.705+0000 7f9bdae19700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bd4161c30 con 0x7f9bd406a7e0 2023-10-16T13:48:10.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.705+0000 7f9bdae19700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bd4161f10 con 0x7f9bd415b280 2023-10-16T13:48:10.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.705+0000 7f9bd93b6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9bd406a7e0 0x7f9bd406ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:10.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.706+0000 7f9bd93b6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9bd406a7e0 0x7f9bd406ac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57094/0 (socket says 172.21.15.16:57094) 2023-10-16T13:48:10.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.706+0000 7f9bd93b6700 1 -- 172.21.15.16:0/440412287 learned_addr learned my addr 172.21.15.16:0/440412287 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:10.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.706+0000 7f9bd8bb5700 1 --2- 172.21.15.16:0/440412287 >> v2:172.21.15.49:3300/0 conn(0x7f9bd415b280 0x7f9bd41613e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:10.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.706+0000 7f9bd93b6700 1 -- 172.21.15.16:0/440412287 >> v2:172.21.15.49:3300/0 conn(0x7f9bd415b280 msgr2=0x7f9bd41613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:10.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.706+0000 7f9bd93b6700 1 --2- 172.21.15.16:0/440412287 >> v2:172.21.15.49:3300/0 conn(0x7f9bd415b280 0x7f9bd41613e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:10.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.706+0000 7f9bd93b6700 1 -- 172.21.15.16:0/440412287 >> v1:172.21.15.49:6789/0 conn(0x7f9bd40cc8a0 legacy=0x7f9bd415e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:10.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.706+0000 7f9bd93b6700 1 -- 172.21.15.16:0/440412287 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bd41621f0 con 0x7f9bd406a7e0 2023-10-16T13:48:10.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.706+0000 7f9bd93b6700 1 --2- 172.21.15.16:0/440412287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bd406a7e0 0x7f9bd406ac10 secure :-1 s=READY pgs=2934 cs=0 l=1 rev1=1 crypto rx=0x7f9bd406b8a0 tx=0x7f9bcc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c405565164bfe79 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:10.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.706+0000 7f9bd37fe700 1 -- 172.21.15.16:0/440412287 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bcc00e070 con 0x7f9bd406a7e0 2023-10-16T13:48:10.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.706+0000 7f9bd37fe700 1 -- 172.21.15.16:0/440412287 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bcc00b910 con 0x7f9bd406a7e0 2023-10-16T13:48:10.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.706+0000 7f9bd37fe700 1 -- 172.21.15.16:0/440412287 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bcc00bcc0 con 0x7f9bd406a7e0 2023-10-16T13:48:10.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.707+0000 7f9bdae19700 1 -- 172.21.15.16:0/440412287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bd406a7e0 msgr2=0x7f9bd406ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:10.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.707+0000 7f9bdae19700 1 --2- 172.21.15.16:0/440412287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bd406a7e0 0x7f9bd406ac10 secure :-1 s=READY pgs=2934 cs=0 l=1 rev1=1 crypto rx=0x7f9bd406b8a0 tx=0x7f9bcc00b330 comp rx=0 tx=0).stop 2023-10-16T13:48:10.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.707+0000 7f9bdae19700 1 -- 172.21.15.16:0/440412287 shutdown_connections 2023-10-16T13:48:10.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.707+0000 7f9bdae19700 1 --2- 172.21.15.16:0/440412287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bd406a7e0 0x7f9bd406ac10 unknown :-1 s=CLOSED pgs=2934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:10.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.707+0000 7f9bdae19700 1 --2- 172.21.15.16:0/440412287 >> v2:172.21.15.49:3300/0 conn(0x7f9bd415b280 0x7f9bd41613e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:10.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.707+0000 7f9bdae19700 1 -- 172.21.15.16:0/440412287 >> 172.21.15.16:0/440412287 conn(0x7f9bd40b1fd0 msgr2=0x7f9bd40b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:10.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.707+0000 7f9bdae19700 1 -- 172.21.15.16:0/440412287 shutdown_connections 2023-10-16T13:48:10.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.707+0000 7f9bdae19700 1 -- 172.21.15.16:0/440412287 wait complete. 2023-10-16T13:48:10.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.708+0000 7f9bdae19700 1 Processor -- start 2023-10-16T13:48:10.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.708+0000 7f9bdae19700 1 -- start start 2023-10-16T13:48:10.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.709+0000 7f9bdae19700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bd406a7e0 0x7f9bd40ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:10.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.709+0000 7f9bdae19700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9bd40cc8a0 0x7f9bd4158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:10.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.709+0000 7f9bdae19700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9bd415b280 0x7f9bd4159e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:10.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.709+0000 7f9bdae19700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd4159460 con 0x7f9bd406a7e0 2023-10-16T13:48:10.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.709+0000 7f9bdae19700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd415a390 con 0x7f9bd415b280 2023-10-16T13:48:10.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.709+0000 7f9bdae19700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd4159180 con 0x7f9bd40cc8a0 2023-10-16T13:48:10.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.709+0000 7f9bd8bb5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bd406a7e0 0x7f9bd40ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:10.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.709+0000 7f9bd8bb5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bd406a7e0 0x7f9bd40ac8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57098/0 (socket says 172.21.15.16:57098) 2023-10-16T13:48:10.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.709+0000 7f9bd8bb5700 1 -- 172.21.15.16:0/3234273157 learned_addr learned my addr 172.21.15.16:0/3234273157 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:10.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.709+0000 7f9bd93b6700 1 --2- 172.21.15.16:0/3234273157 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9bd415b280 0x7f9bd4159e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:10.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.709+0000 7f9bd3fff700 1 --2- 172.21.15.16:0/3234273157 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9bd40cc8a0 0x7f9bd4158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:10.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.709+0000 7f9bd8bb5700 1 -- 172.21.15.16:0/3234273157 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9bd40cc8a0 msgr2=0x7f9bd4158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:10.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.709+0000 7f9bd8bb5700 1 --2- 172.21.15.16:0/3234273157 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9bd40cc8a0 0x7f9bd4158ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:10.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.709+0000 7f9bd8bb5700 1 -- 172.21.15.16:0/3234273157 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9bd415b280 msgr2=0x7f9bd4159e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:10.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.709+0000 7f9bd8bb5700 1 --2- 172.21.15.16:0/3234273157 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9bd415b280 0x7f9bd4159e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:10.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.709+0000 7f9bd8bb5700 1 -- 172.21.15.16:0/3234273157 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bcc009b30 con 0x7f9bd406a7e0 2023-10-16T13:48:10.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.709+0000 7f9bd8bb5700 1 --2- 172.21.15.16:0/3234273157 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bd406a7e0 0x7f9bd40ac8d0 secure :-1 s=READY pgs=2935 cs=0 l=1 rev1=1 crypto rx=0x7f9bc800e9b0 tx=0x7f9bc800ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:10.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.710+0000 7f9bd17fa700 1 -- 172.21.15.16:0/3234273157 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc8009710 con 0x7f9bd406a7e0 2023-10-16T13:48:10.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.710+0000 7f9bd17fa700 1 -- 172.21.15.16:0/3234273157 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bc800ee50 con 0x7f9bd406a7e0 2023-10-16T13:48:10.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.710+0000 7f9bd17fa700 1 -- 172.21.15.16:0/3234273157 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc800ca50 con 0x7f9bd406a7e0 2023-10-16T13:48:10.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.710+0000 7f9bdae19700 1 -- 172.21.15.16:0/3234273157 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bd40c40a0 con 0x7f9bd406a7e0 2023-10-16T13:48:10.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.710+0000 7f9bdae19700 1 -- 172.21.15.16:0/3234273157 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bd40c4660 con 0x7f9bd406a7e0 2023-10-16T13:48:10.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.711+0000 7f9bdae19700 1 -- 172.21.15.16:0/3234273157 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bd404ebb0 con 0x7f9bd406a7e0 2023-10-16T13:48:10.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.711+0000 7f9bd17fa700 1 -- 172.21.15.16:0/3234273157 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f9bc800cbb0 con 0x7f9bd406a7e0 2023-10-16T13:48:10.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.712+0000 7f9bd17fa700 1 --2- 172.21.15.16:0/3234273157 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f9bb806b140 0x7f9bb806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:10.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.712+0000 7f9bd17fa700 1 -- 172.21.15.16:0/3234273157 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(254..254 src has 1..254) v4 ==== 7115+0+0 (secure 0 0 0) 0x7f9bc8089eb0 con 0x7f9bd406a7e0 2023-10-16T13:48:10.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.712+0000 7f9bd3fff700 1 --2- 172.21.15.16:0/3234273157 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f9bb806b140 0x7f9bb806d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:10.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.712+0000 7f9bd3fff700 1 --2- 172.21.15.16:0/3234273157 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f9bb806b140 0x7f9bb806d600 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9bd4071da0 tx=0x7f9bc0009040 comp rx=0 tx=0).ready entity=mgr.12678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:10.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.716+0000 7f9bd17fa700 1 -- 172.21.15.16:0/3234273157 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9bc8054710 con 0x7f9bd406a7e0 2023-10-16T13:48:10.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.963+0000 7f9bdae19700 1 -- 172.21.15.16:0/3234273157 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9bd40cad20 con 0x7f9bd406a7e0 2023-10-16T13:48:10.965 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.963+0000 7f9bd17fa700 1 -- 172.21.15.16:0/3234273157 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 137 v137) v1 ==== 95+0+4928 (secure 0 0 0) 0x7f9bc8021090 con 0x7f9bd406a7e0 2023-10-16T13:48:10.966 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:48:10.966 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":137,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9596,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/3939647668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":3939647668},{"type":"v1","addr":"172.21.15.49:6834","nonce":3939647668}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":125},{"gid":11185,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/2890113513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2890113513},{"type":"v1","addr":"172.21.15.16:6836","nonce":2890113513}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":125},{"gid":12864,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/1181609271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":1181609271},{"type":"v1","addr":"172.21.15.16:6837","nonce":1181609271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":137}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:47:41.332096+0000","modified":"2023-10-16T13:47:44.368830+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11197},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11197":{"gid":11197,"name":"d","rank":0,"incarnation":132,"state":"up:active","state_seq":7,"addr":"172.21.15.49:6835/680748068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":680748068},{"type":"v1","addr":"172.21.15.49:6835","nonce":680748068}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2023-10-16T13:48:10.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.965+0000 7f9bdae19700 1 -- 172.21.15.16:0/3234273157 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f9bb806b140 msgr2=0x7f9bb806d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:10.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.965+0000 7f9bdae19700 1 --2- 172.21.15.16:0/3234273157 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f9bb806b140 0x7f9bb806d600 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9bd4071da0 tx=0x7f9bc0009040 comp rx=0 tx=0).stop 2023-10-16T13:48:10.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.965+0000 7f9bdae19700 1 -- 172.21.15.16:0/3234273157 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bd406a7e0 msgr2=0x7f9bd40ac8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:10.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.965+0000 7f9bdae19700 1 --2- 172.21.15.16:0/3234273157 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bd406a7e0 0x7f9bd40ac8d0 secure :-1 s=READY pgs=2935 cs=0 l=1 rev1=1 crypto rx=0x7f9bc800e9b0 tx=0x7f9bc800ecc0 comp rx=0 tx=0).stop 2023-10-16T13:48:10.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.965+0000 7f9bdae19700 1 -- 172.21.15.16:0/3234273157 shutdown_connections 2023-10-16T13:48:10.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.965+0000 7f9bdae19700 1 --2- 172.21.15.16:0/3234273157 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f9bb806b140 0x7f9bb806d600 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:10.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.965+0000 7f9bdae19700 1 --2- 172.21.15.16:0/3234273157 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9bd415b280 0x7f9bd4159e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:10.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.965+0000 7f9bdae19700 1 --2- 172.21.15.16:0/3234273157 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bd406a7e0 0x7f9bd40ac8d0 unknown :-1 s=CLOSED pgs=2935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:10.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.965+0000 7f9bdae19700 1 --2- 172.21.15.16:0/3234273157 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9bd40cc8a0 0x7f9bd4158ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:10.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.965+0000 7f9bdae19700 1 -- 172.21.15.16:0/3234273157 >> 172.21.15.16:0/3234273157 conn(0x7f9bd40b1fd0 msgr2=0x7f9bd40d0270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:10.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.965+0000 7f9bdae19700 1 -- 172.21.15.16:0/3234273157 shutdown_connections 2023-10-16T13:48:10.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:10.965+0000 7f9bdae19700 1 -- 172.21.15.16:0/3234273157 wait complete. 2023-10-16T13:48:10.970 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 137 2023-10-16T13:48:10.980 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:48:11.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.103+0000 7fe3eba19700 1 Processor -- start 2023-10-16T13:48:11.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.103+0000 7fe3eba19700 1 -- start start 2023-10-16T13:48:11.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.104+0000 7fe3eba19700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fe3e415b630 0x7fe3e40cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:11.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.104+0000 7fe3eba19700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe3e415ba10 0x7fe3e40cdde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:11.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.104+0000 7fe3eba19700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe3e40ce320 0x7fe3e40cfca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:11.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.104+0000 7fe3eba19700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3e40cdb20 con 0x7fe3e40ce320 2023-10-16T13:48:11.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.104+0000 7fe3eba19700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3e40ce770 con 0x7fe3e415b630 2023-10-16T13:48:11.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.104+0000 7fe3eba19700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3e40cc2a0 con 0x7fe3e415ba10 2023-10-16T13:48:11.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.104+0000 7fe3e9fb6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe3e40ce320 0x7fe3e40cfca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:11.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.104+0000 7fe3e9fb6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe3e40ce320 0x7fe3e40cfca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57110/0 (socket says 172.21.15.16:57110) 2023-10-16T13:48:11.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.104+0000 7fe3e9fb6700 1 -- 172.21.15.16:0/2982445984 learned_addr learned my addr 172.21.15.16:0/2982445984 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:11.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.105+0000 7fe3e97b5700 1 --2- 172.21.15.16:0/2982445984 >> v2:172.21.15.49:3300/0 conn(0x7fe3e415b630 0x7fe3e40cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:11.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.105+0000 7fe3e8fb4700 1 --2- 172.21.15.16:0/2982445984 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe3e415ba10 0x7fe3e40cdde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:11.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.105+0000 7fe3e9fb6700 1 -- 172.21.15.16:0/2982445984 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe3e415ba10 msgr2=0x7fe3e40cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:11.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.105+0000 7fe3e9fb6700 1 --2- 172.21.15.16:0/2982445984 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe3e415ba10 0x7fe3e40cdde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:11.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.105+0000 7fe3e9fb6700 1 -- 172.21.15.16:0/2982445984 >> v2:172.21.15.49:3300/0 conn(0x7fe3e415b630 msgr2=0x7fe3e40cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:11.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.105+0000 7fe3e9fb6700 1 --2- 172.21.15.16:0/2982445984 >> v2:172.21.15.49:3300/0 conn(0x7fe3e415b630 0x7fe3e40cd5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:11.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.105+0000 7fe3e9fb6700 1 -- 172.21.15.16:0/2982445984 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3e40cc540 con 0x7fe3e40ce320 2023-10-16T13:48:11.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.105+0000 7fe3e97b5700 1 --2- 172.21.15.16:0/2982445984 >> v2:172.21.15.49:3300/0 conn(0x7fe3e415b630 0x7fe3e40cd5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:48:11.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.105+0000 7fe3e8fb4700 1 --2- 172.21.15.16:0/2982445984 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe3e415ba10 0x7fe3e40cdde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:48:11.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.105+0000 7fe3e9fb6700 1 --2- 172.21.15.16:0/2982445984 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3e40ce320 0x7fe3e40cfca0 secure :-1 s=READY pgs=2936 cs=0 l=1 rev1=1 crypto rx=0x7fe3e40cefb0 tx=0x7fe3dc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c3fdc553837496 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:11.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.105+0000 7fe3dbfff700 1 -- 172.21.15.16:0/2982445984 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3dc00e070 con 0x7fe3e40ce320 2023-10-16T13:48:11.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.105+0000 7fe3dbfff700 1 -- 172.21.15.16:0/2982445984 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3dc00b9c0 con 0x7fe3e40ce320 2023-10-16T13:48:11.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.105+0000 7fe3dbfff700 1 -- 172.21.15.16:0/2982445984 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3dc00bd70 con 0x7fe3e40ce320 2023-10-16T13:48:11.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.106+0000 7fe3eba19700 1 -- 172.21.15.16:0/2982445984 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3e40ce320 msgr2=0x7fe3e40cfca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:11.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.106+0000 7fe3eba19700 1 --2- 172.21.15.16:0/2982445984 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3e40ce320 0x7fe3e40cfca0 secure :-1 s=READY pgs=2936 cs=0 l=1 rev1=1 crypto rx=0x7fe3e40cefb0 tx=0x7fe3dc00b330 comp rx=0 tx=0).stop 2023-10-16T13:48:11.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.106+0000 7fe3eba19700 1 -- 172.21.15.16:0/2982445984 shutdown_connections 2023-10-16T13:48:11.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.106+0000 7fe3eba19700 1 --2- 172.21.15.16:0/2982445984 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3e40ce320 0x7fe3e40cfca0 unknown :-1 s=CLOSED pgs=2936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:11.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.106+0000 7fe3eba19700 1 --2- 172.21.15.16:0/2982445984 >> v2:172.21.15.49:3300/0 conn(0x7fe3e415b630 0x7fe3e40cd5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:11.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.106+0000 7fe3eba19700 1 --2- 172.21.15.16:0/2982445984 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe3e415ba10 0x7fe3e40cdde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:11.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.106+0000 7fe3eba19700 1 -- 172.21.15.16:0/2982445984 >> 172.21.15.16:0/2982445984 conn(0x7fe3e40d14b0 msgr2=0x7fe3e40d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:11.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.106+0000 7fe3eba19700 1 -- 172.21.15.16:0/2982445984 shutdown_connections 2023-10-16T13:48:11.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.106+0000 7fe3eba19700 1 -- 172.21.15.16:0/2982445984 wait complete. 2023-10-16T13:48:11.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.107+0000 7fe3eba19700 1 Processor -- start 2023-10-16T13:48:11.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.107+0000 7fe3eba19700 1 -- start start 2023-10-16T13:48:11.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.108+0000 7fe3eba19700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe3e40ce320 0x7fe3e40caae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:11.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.108+0000 7fe3eba19700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3e415b630 0x7fe3e406f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:11.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.108+0000 7fe3eba19700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe3e415ba10 0x7fe3e414b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:11.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.108+0000 7fe3eba19700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3e40cb020 con 0x7fe3e415b630 2023-10-16T13:48:11.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.108+0000 7fe3eba19700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3e40ca650 con 0x7fe3e40ce320 2023-10-16T13:48:11.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.108+0000 7fe3eba19700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3e40c9600 con 0x7fe3e415ba10 2023-10-16T13:48:11.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.108+0000 7fe3e8fb4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3e415b630 0x7fe3e406f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:11.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.108+0000 7fe3e8fb4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3e415b630 0x7fe3e406f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57114/0 (socket says 172.21.15.16:57114) 2023-10-16T13:48:11.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.108+0000 7fe3e8fb4700 1 -- 172.21.15.16:0/340144820 learned_addr learned my addr 172.21.15.16:0/340144820 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:11.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.108+0000 7fe3e97b5700 1 --2- 172.21.15.16:0/340144820 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe3e40ce320 0x7fe3e40caae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:11.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.108+0000 7fe3e9fb6700 1 --2- 172.21.15.16:0/340144820 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe3e415ba10 0x7fe3e414b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:11.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.108+0000 7fe3e8fb4700 1 -- 172.21.15.16:0/340144820 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe3e415ba10 msgr2=0x7fe3e414b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:11.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.108+0000 7fe3e8fb4700 1 --2- 172.21.15.16:0/340144820 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe3e415ba10 0x7fe3e414b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:11.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.108+0000 7fe3e8fb4700 1 -- 172.21.15.16:0/340144820 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe3e40ce320 msgr2=0x7fe3e40caae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:11.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.108+0000 7fe3e8fb4700 1 --2- 172.21.15.16:0/340144820 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe3e40ce320 0x7fe3e40caae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:11.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.108+0000 7fe3e8fb4700 1 -- 172.21.15.16:0/340144820 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3dc009b30 con 0x7fe3e415b630 2023-10-16T13:48:11.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.108+0000 7fe3e8fb4700 1 --2- 172.21.15.16:0/340144820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3e415b630 0x7fe3e406f7b0 secure :-1 s=READY pgs=2937 cs=0 l=1 rev1=1 crypto rx=0x7fe3e000c750 tx=0x7fe3e000ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:11.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.108+0000 7fe3d9ffb700 1 -- 172.21.15.16:0/340144820 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3e0011800 con 0x7fe3e415b630 2023-10-16T13:48:11.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.108+0000 7fe3d9ffb700 1 -- 172.21.15.16:0/340144820 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3e0011960 con 0x7fe3e415b630 2023-10-16T13:48:11.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.108+0000 7fe3eba19700 1 -- 172.21.15.16:0/340144820 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3e40c9320 con 0x7fe3e415b630 2023-10-16T13:48:11.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.109+0000 7fe3d9ffb700 1 -- 172.21.15.16:0/340144820 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3e0011cd0 con 0x7fe3e415b630 2023-10-16T13:48:11.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.109+0000 7fe3eba19700 1 -- 172.21.15.16:0/340144820 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3e40ca2e0 con 0x7fe3e415b630 2023-10-16T13:48:11.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.110+0000 7fe3eba19700 1 -- 172.21.15.16:0/340144820 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3e40ca7c0 con 0x7fe3e415b630 2023-10-16T13:48:11.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.110+0000 7fe3d9ffb700 1 -- 172.21.15.16:0/340144820 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fe3e0021350 con 0x7fe3e415b630 2023-10-16T13:48:11.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.110+0000 7fe3d9ffb700 1 --2- 172.21.15.16:0/340144820 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fe3c806b140 0x7fe3c806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:11.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.110+0000 7fe3d9ffb700 1 -- 172.21.15.16:0/340144820 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(254..254 src has 1..254) v4 ==== 7115+0+0 (secure 0 0 0) 0x7fe3e0089e00 con 0x7fe3e415b630 2023-10-16T13:48:11.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.110+0000 7fe3e97b5700 1 --2- 172.21.15.16:0/340144820 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fe3c806b140 0x7fe3c806d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:11.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.111+0000 7fe3e97b5700 1 --2- 172.21.15.16:0/340144820 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fe3c806b140 0x7fe3c806d600 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe3d4006fd0 tx=0x7fe3d4008040 comp rx=0 tx=0).ready entity=mgr.12678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:11.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.115+0000 7fe3d9ffb700 1 -- 172.21.15.16:0/340144820 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe3e0054550 con 0x7fe3e415b630 2023-10-16T13:48:11.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.363+0000 7fe3eba19700 1 -- 172.21.15.16:0/340144820 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe3e40c8380 con 0x7fe3e415b630 2023-10-16T13:48:11.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.364+0000 7fe3d9ffb700 1 -- 172.21.15.16:0/340144820 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 137 v137) v1 ==== 95+0+4928 (secure 0 0 0) 0x7fe3e001e090 con 0x7fe3e415b630 2023-10-16T13:48:11.367 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:48:11.367 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":137,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9596,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/3939647668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":3939647668},{"type":"v1","addr":"172.21.15.49:6834","nonce":3939647668}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":125},{"gid":11185,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/2890113513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2890113513},{"type":"v1","addr":"172.21.15.16:6836","nonce":2890113513}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":125},{"gid":12864,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/1181609271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":1181609271},{"type":"v1","addr":"172.21.15.16:6837","nonce":1181609271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":137}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:47:41.332096+0000","modified":"2023-10-16T13:47:44.368830+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11197},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11197":{"gid":11197,"name":"d","rank":0,"incarnation":132,"state":"up:active","state_seq":7,"addr":"172.21.15.49:6835/680748068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":680748068},{"type":"v1","addr":"172.21.15.49:6835","nonce":680748068}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2023-10-16T13:48:11.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.366+0000 7fe3eba19700 1 -- 172.21.15.16:0/340144820 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fe3c806b140 msgr2=0x7fe3c806d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:11.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.366+0000 7fe3eba19700 1 --2- 172.21.15.16:0/340144820 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fe3c806b140 0x7fe3c806d600 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe3d4006fd0 tx=0x7fe3d4008040 comp rx=0 tx=0).stop 2023-10-16T13:48:11.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.366+0000 7fe3eba19700 1 -- 172.21.15.16:0/340144820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3e415b630 msgr2=0x7fe3e406f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:11.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.366+0000 7fe3eba19700 1 --2- 172.21.15.16:0/340144820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3e415b630 0x7fe3e406f7b0 secure :-1 s=READY pgs=2937 cs=0 l=1 rev1=1 crypto rx=0x7fe3e000c750 tx=0x7fe3e000ca60 comp rx=0 tx=0).stop 2023-10-16T13:48:11.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.366+0000 7fe3eba19700 1 -- 172.21.15.16:0/340144820 shutdown_connections 2023-10-16T13:48:11.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.366+0000 7fe3eba19700 1 --2- 172.21.15.16:0/340144820 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fe3c806b140 0x7fe3c806d600 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:11.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.366+0000 7fe3eba19700 1 --2- 172.21.15.16:0/340144820 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe3e415ba10 0x7fe3e414b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:11.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.366+0000 7fe3eba19700 1 --2- 172.21.15.16:0/340144820 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe3e40ce320 0x7fe3e40caae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:11.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.366+0000 7fe3eba19700 1 --2- 172.21.15.16:0/340144820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3e415b630 0x7fe3e406f7b0 unknown :-1 s=CLOSED pgs=2937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:11.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.366+0000 7fe3eba19700 1 -- 172.21.15.16:0/340144820 >> 172.21.15.16:0/340144820 conn(0x7fe3e40d14b0 msgr2=0x7fe3e40b5000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:11.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.367+0000 7fe3eba19700 1 -- 172.21.15.16:0/340144820 shutdown_connections 2023-10-16T13:48:11.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.367+0000 7fe3eba19700 1 -- 172.21.15.16:0/340144820 wait complete. 2023-10-16T13:48:11.372 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 137 2023-10-16T13:48:11.381 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:48:11.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.504+0000 7f9f5b0aa700 1 Processor -- start 2023-10-16T13:48:11.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.504+0000 7f9f5b0aa700 1 -- start start 2023-10-16T13:48:11.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.504+0000 7f9f5b0aa700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9f540d0ae0 0x7f9f540cef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:11.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.504+0000 7f9f5b0aa700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f54072a50 con 0x7f9f540ce010 2023-10-16T13:48:11.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.504+0000 7f9f5b0aa700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f540cba30 con 0x7f9f5415a5c0 2023-10-16T13:48:11.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.504+0000 7f9f5b0aa700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f540cf440 con 0x7f9f540d0ae0 2023-10-16T13:48:11.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.505+0000 7f9f59647700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f9f540ce010 0x7f9f540cc1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46398/0 (socket says 172.21.15.16:46398) 2023-10-16T13:48:11.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.505+0000 7f9f59647700 1 -- 172.21.15.16:0/35670703 learned_addr learned my addr 172.21.15.16:0/35670703 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:48:11.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.505+0000 7f9f53fff700 1 --2- 172.21.15.16:0/35670703 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9f540d0ae0 0x7f9f540cef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:11.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.505+0000 7f9f537fe700 1 -- 172.21.15.16:0/35670703 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1486753060 0 0) 0x7f9f54072a50 con 0x7f9f540ce010 2023-10-16T13:48:11.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.505+0000 7f9f537fe700 1 -- 172.21.15.16:0/35670703 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f440035b0 con 0x7f9f540ce010 2023-10-16T13:48:11.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.505+0000 7f9f537fe700 1 -- 172.21.15.16:0/35670703 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1829409509 0 0) 0x7f9f440035b0 con 0x7f9f540ce010 2023-10-16T13:48:11.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.505+0000 7f9f537fe700 1 -- 172.21.15.16:0/35670703 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9f540d0ae0 msgr2=0x7f9f540cef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:11.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.505+0000 7f9f537fe700 1 --2- 172.21.15.16:0/35670703 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9f540d0ae0 0x7f9f540cef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:11.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.505+0000 7f9f537fe700 1 -- 172.21.15.16:0/35670703 >> v1:172.21.15.49:6789/0 conn(0x7f9f5415a5c0 legacy=0x7f9f540d03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:11.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.505+0000 7f9f537fe700 1 -- 172.21.15.16:0/35670703 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f540cf630 con 0x7f9f540ce010 2023-10-16T13:48:11.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.505+0000 7f9f537fe700 1 -- 172.21.15.16:0/35670703 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f9f4c0019f0 con 0x7f9f540ce010 2023-10-16T13:48:11.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.505+0000 7f9f537fe700 1 -- 172.21.15.16:0/35670703 >> v1:172.21.15.16:6789/0 conn(0x7f9f540ce010 legacy=0x7f9f540cc1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:11.512 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.505+0000 7f9f537fe700 1 --2- 172.21.15.16:0/35670703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f44003a30 0x7f9f44003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:11.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.505+0000 7f9f537fe700 1 --2- 172.21.15.16:0/35670703 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9f540d0ae0 0x7f9f44004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:11.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.505+0000 7f9f537fe700 1 --2- 172.21.15.16:0/35670703 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9f44004d90 0x7f9f44005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:11.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.505+0000 7f9f537fe700 1 -- 172.21.15.16:0/35670703 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f4400bac0 con 0x7f9f44003a30 2023-10-16T13:48:11.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.505+0000 7f9f537fe700 1 -- 172.21.15.16:0/35670703 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f4400bc00 con 0x7f9f44004d90 2023-10-16T13:48:11.513 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.505+0000 7f9f537fe700 1 -- 172.21.15.16:0/35670703 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f4400bdc0 con 0x7f9f540d0ae0 2023-10-16T13:48:11.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.505+0000 7f9f58e46700 1 --2- 172.21.15.16:0/35670703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f44003a30 0x7f9f44003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:11.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.505+0000 7f9f53fff700 1 --2- 172.21.15.16:0/35670703 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9f540d0ae0 0x7f9f44004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:11.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.505+0000 7f9f59647700 1 --2- 172.21.15.16:0/35670703 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9f44004d90 0x7f9f44005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:11.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.506+0000 7f9f58e46700 1 -- 172.21.15.16:0/35670703 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9f540d0ae0 msgr2=0x7f9f44004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:11.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.506+0000 7f9f58e46700 1 --2- 172.21.15.16:0/35670703 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9f540d0ae0 0x7f9f44004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:11.514 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.506+0000 7f9f58e46700 1 -- 172.21.15.16:0/35670703 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9f44004d90 msgr2=0x7f9f44005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:11.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.506+0000 7f9f58e46700 1 --2- 172.21.15.16:0/35670703 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9f44004d90 0x7f9f44005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:11.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.506+0000 7f9f58e46700 1 -- 172.21.15.16:0/35670703 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f440035b0 con 0x7f9f44003a30 2023-10-16T13:48:11.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.506+0000 7f9f59647700 1 --2- 172.21.15.16:0/35670703 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9f44004d90 0x7f9f44005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:48:11.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.506+0000 7f9f58e46700 1 --2- 172.21.15.16:0/35670703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f44003a30 0x7f9f44003e50 secure :-1 s=READY pgs=2939 cs=0 l=1 rev1=1 crypto rx=0x7f9f48007fd0 tx=0x7f9f44005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae0775cb1f71ad61 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:11.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.506+0000 7f9f537fe700 1 -- 172.21.15.16:0/35670703 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f480052d0 con 0x7f9f44003a30 2023-10-16T13:48:11.515 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.506+0000 7f9f537fe700 1 -- 172.21.15.16:0/35670703 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f48005470 con 0x7f9f44003a30 2023-10-16T13:48:11.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.506+0000 7f9f537fe700 1 -- 172.21.15.16:0/35670703 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f480038c0 con 0x7f9f44003a30 2023-10-16T13:48:11.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.506+0000 7f9f5b0aa700 1 -- 172.21.15.16:0/35670703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f44003a30 msgr2=0x7f9f44003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:11.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.506+0000 7f9f5b0aa700 1 --2- 172.21.15.16:0/35670703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f44003a30 0x7f9f44003e50 secure :-1 s=READY pgs=2939 cs=0 l=1 rev1=1 crypto rx=0x7f9f48007fd0 tx=0x7f9f44005880 comp rx=0 tx=0).stop 2023-10-16T13:48:11.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.506+0000 7f9f59647700 1 -- 172.21.15.16:0/35670703 reap_dead start 2023-10-16T13:48:11.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.506+0000 7f9f5b0aa700 1 -- 172.21.15.16:0/35670703 shutdown_connections 2023-10-16T13:48:11.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.506+0000 7f9f5b0aa700 1 -- 172.21.15.16:0/35670703 >> 172.21.15.16:0/35670703 conn(0x7f9f540b1fd0 msgr2=0x7f9f540b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:11.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.507+0000 7f9f5b0aa700 1 -- 172.21.15.16:0/35670703 shutdown_connections 2023-10-16T13:48:11.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.507+0000 7f9f5b0aa700 1 -- 172.21.15.16:0/35670703 wait complete. 2023-10-16T13:48:11.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.508+0000 7f9f5b0aa700 1 Processor -- start 2023-10-16T13:48:11.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.508+0000 7f9f5b0aa700 1 -- start start 2023-10-16T13:48:11.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.509+0000 7f9f5b0aa700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9f44003a30 0x7f9f540c5c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:11.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.509+0000 7f9f5b0aa700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f541572d0 0x7f9f540c5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:11.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.509+0000 7f9f5b0aa700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9f540b0d70 0x7f9f540b11f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:11.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.509+0000 7f9f5b0aa700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f540ad270 con 0x7f9f541572d0 2023-10-16T13:48:11.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.509+0000 7f9f5b0aa700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f540ca400 con 0x7f9f44003a30 2023-10-16T13:48:11.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.509+0000 7f9f5b0aa700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f54157750 con 0x7f9f540b0d70 2023-10-16T13:48:11.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.509+0000 7f9f53fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f541572d0 0x7f9f540c5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:11.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.509+0000 7f9f53fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f541572d0 0x7f9f540c5270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57134/0 (socket says 172.21.15.16:57134) 2023-10-16T13:48:11.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.509+0000 7f9f53fff700 1 -- 172.21.15.16:0/1072896595 learned_addr learned my addr 172.21.15.16:0/1072896595 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:11.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.509+0000 7f9f59647700 1 --2- 172.21.15.16:0/1072896595 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9f540b0d70 0x7f9f540b11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:11.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.509+0000 7f9f58e46700 1 --2- 172.21.15.16:0/1072896595 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9f44003a30 0x7f9f540c5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:11.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.509+0000 7f9f53fff700 1 -- 172.21.15.16:0/1072896595 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9f540b0d70 msgr2=0x7f9f540b11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:11.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.509+0000 7f9f53fff700 1 --2- 172.21.15.16:0/1072896595 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9f540b0d70 0x7f9f540b11f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:11.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.509+0000 7f9f53fff700 1 -- 172.21.15.16:0/1072896595 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9f44003a30 msgr2=0x7f9f540c5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:11.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.509+0000 7f9f53fff700 1 --2- 172.21.15.16:0/1072896595 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9f44003a30 0x7f9f540c5c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:11.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.509+0000 7f9f53fff700 1 -- 172.21.15.16:0/1072896595 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f54157c00 con 0x7f9f541572d0 2023-10-16T13:48:11.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.509+0000 7f9f59647700 1 --2- 172.21.15.16:0/1072896595 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9f540b0d70 0x7f9f540b11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:48:11.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.509+0000 7f9f58e46700 1 --2- 172.21.15.16:0/1072896595 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9f44003a30 0x7f9f540c5c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:48:11.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.509+0000 7f9f53fff700 1 --2- 172.21.15.16:0/1072896595 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f541572d0 0x7f9f540c5270 secure :-1 s=READY pgs=2940 cs=0 l=1 rev1=1 crypto rx=0x7f9f40011ad0 tx=0x7f9f40011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:11.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.509+0000 7f9f517fa700 1 -- 172.21.15.16:0/1072896595 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f4000f9a0 con 0x7f9f541572d0 2023-10-16T13:48:11.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.509+0000 7f9f517fa700 1 -- 172.21.15.16:0/1072896595 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f4000fb00 con 0x7f9f541572d0 2023-10-16T13:48:11.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.509+0000 7f9f5b0aa700 1 -- 172.21.15.16:0/1072896595 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f540c5900 con 0x7f9f541572d0 2023-10-16T13:48:11.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.509+0000 7f9f517fa700 1 -- 172.21.15.16:0/1072896595 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f4000feb0 con 0x7f9f541572d0 2023-10-16T13:48:11.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.509+0000 7f9f5b0aa700 1 -- 172.21.15.16:0/1072896595 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f540acbb0 con 0x7f9f541572d0 2023-10-16T13:48:11.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.510+0000 7f9f5b0aa700 1 -- 172.21.15.16:0/1072896595 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f540ad3b0 con 0x7f9f541572d0 2023-10-16T13:48:11.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.511+0000 7f9f517fa700 1 -- 172.21.15.16:0/1072896595 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f9f40014440 con 0x7f9f541572d0 2023-10-16T13:48:11.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.511+0000 7f9f517fa700 1 --2- 172.21.15.16:0/1072896595 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f9f3806b140 0x7f9f3806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:11.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.511+0000 7f9f517fa700 1 -- 172.21.15.16:0/1072896595 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(254..254 src has 1..254) v4 ==== 7115+0+0 (secure 0 0 0) 0x7f9f4008a5d0 con 0x7f9f541572d0 2023-10-16T13:48:11.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.511+0000 7f9f58e46700 1 --2- 172.21.15.16:0/1072896595 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f9f3806b140 0x7f9f3806d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:11.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.512+0000 7f9f58e46700 1 --2- 172.21.15.16:0/1072896595 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f9f3806b140 0x7f9f3806d600 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9f48007fd0 tx=0x7f9f48009030 comp rx=0 tx=0).ready entity=mgr.12678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:11.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.516+0000 7f9f517fa700 1 -- 172.21.15.16:0/1072896595 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9f540ad3b0 con 0x7f9f541572d0 2023-10-16T13:48:11.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.719+0000 7f9f5b0aa700 1 -- 172.21.15.16:0/1072896595 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9f540ad3b0 con 0x7f9f541572d0 2023-10-16T13:48:11.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.721+0000 7f9f517fa700 1 -- 172.21.15.16:0/1072896595 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v254) v1 ==== 74+0+18205 (secure 0 0 0) 0x7f9f540ad3b0 con 0x7f9f541572d0 2023-10-16T13:48:11.723 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:48:11.724 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":254,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:48:08.349411+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_metadata","create_time":"2023-10-16T13:47:39.075528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"248","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"backup_fs_data","create_time":"2023-10-16T13:47:40.314029+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"249","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6835/303278795":"2023-10-17T13:48:05.033844+0000","172.21.15.16:6832/2221624769":"2023-10-17T13:48:02.485614+0000","172.21.15.16:0/2242986614":"2023-10-17T13:48:02.485614+0000","172.21.15.16:0/3610227254":"2023-10-17T13:48:02.485614+0000","172.21.15.16:0/1182888331":"2023-10-17T13:48:02.485614+0000","172.21.15.16:0/2095012141":"2023-10-17T13:48:02.485614+0000","172.21.15.16:0/874722077":"2023-10-17T13:48:02.485614+0000","172.21.15.16:0/2066401556":"2023-10-17T13:48:02.485614+0000","172.21.15.16:0/3947601914":"2023-10-17T13:47:54.816700+0000","172.21.15.16:6833/2221624769":"2023-10-17T13:48:02.485614+0000","172.21.15.16:0/2759470534":"2023-10-17T13:47:54.816700+0000","172.21.15.16:0/2634105683":"2023-10-17T13:48:02.485614+0000","172.21.15.16:0/2974892940":"2023-10-17T13:47:54.816700+0000","172.21.15.16:6833/1736104576":"2023-10-17T13:47:54.816700+0000","172.21.15.16:6832/1736104576":"2023-10-17T13:47:54.816700+0000","172.21.15.16:0/1246409088":"2023-10-17T13:48:02.485614+0000","172.21.15.16:0/318862437":"2023-10-17T13:47:54.816700+0000","172.21.15.16:0/413581668":"2023-10-17T13:47:54.816700+0000","172.21.15.16:6837/303278795":"2023-10-17T13:48:05.033844+0000","172.21.15.16:0/3740515383":"2023-10-17T13:47:54.816700+0000","172.21.15.16:0/3105924907":"2023-10-17T13:47:54.816700+0000","172.21.15.16:0/3617558038":"2023-10-17T13:47:54.816700+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:48:11.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.723+0000 7f9f5b0aa700 1 -- 172.21.15.16:0/1072896595 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f9f3806b140 msgr2=0x7f9f3806d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:11.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.723+0000 7f9f5b0aa700 1 --2- 172.21.15.16:0/1072896595 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f9f3806b140 0x7f9f3806d600 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9f48007fd0 tx=0x7f9f48009030 comp rx=0 tx=0).stop 2023-10-16T13:48:11.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.723+0000 7f9f5b0aa700 1 -- 172.21.15.16:0/1072896595 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f541572d0 msgr2=0x7f9f540c5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:11.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.723+0000 7f9f5b0aa700 1 --2- 172.21.15.16:0/1072896595 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f541572d0 0x7f9f540c5270 secure :-1 s=READY pgs=2940 cs=0 l=1 rev1=1 crypto rx=0x7f9f40011ad0 tx=0x7f9f40011e90 comp rx=0 tx=0).stop 2023-10-16T13:48:11.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.723+0000 7f9f5b0aa700 1 -- 172.21.15.16:0/1072896595 shutdown_connections 2023-10-16T13:48:11.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.723+0000 7f9f5b0aa700 1 --2- 172.21.15.16:0/1072896595 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f9f3806b140 0x7f9f3806d600 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:11.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.723+0000 7f9f5b0aa700 1 --2- 172.21.15.16:0/1072896595 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9f540b0d70 0x7f9f540b11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:11.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.723+0000 7f9f5b0aa700 1 --2- 172.21.15.16:0/1072896595 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9f44003a30 0x7f9f540c5c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:11.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.723+0000 7f9f5b0aa700 1 --2- 172.21.15.16:0/1072896595 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f541572d0 0x7f9f540c5270 unknown :-1 s=CLOSED pgs=2940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:11.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.723+0000 7f9f5b0aa700 1 -- 172.21.15.16:0/1072896595 >> 172.21.15.16:0/1072896595 conn(0x7f9f540b1fd0 msgr2=0x7f9f540cebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:11.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.723+0000 7f9f5b0aa700 1 -- 172.21.15.16:0/1072896595 shutdown_connections 2023-10-16T13:48:11.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.723+0000 7f9f5b0aa700 1 -- 172.21.15.16:0/1072896595 wait complete. 2023-10-16T13:48:11.738 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail backup_fs 2023-10-16T13:48:11.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.862+0000 7f2d0fb8f700 1 Processor -- start 2023-10-16T13:48:11.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.862+0000 7f2d0fb8f700 1 -- start start 2023-10-16T13:48:11.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.863+0000 7f2d0fb8f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2d080cb280 0x7f2d0806d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:11.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.863+0000 7f2d0fb8f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2d0806b1f0 0x7f2d0806b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:11.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.863+0000 7f2d0fb8f700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d0806c150 con 0x7f2d08160eb0 2023-10-16T13:48:11.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.863+0000 7f2d0fb8f700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d080af370 con 0x7f2d0806b1f0 2023-10-16T13:48:11.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.863+0000 7f2d0fb8f700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d0806bbe0 con 0x7f2d080cb280 2023-10-16T13:48:11.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.863+0000 7f2d0e12c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2d0806b1f0 0x7f2d0806b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:11.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.863+0000 7f2d0d92b700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2d08160eb0 0x7f2d080cab70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46414/0 (socket says 172.21.15.16:46414) 2023-10-16T13:48:11.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.863+0000 7f2d0d92b700 1 -- 172.21.15.16:0/725983270 learned_addr learned my addr 172.21.15.16:0/725983270 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:48:11.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.864+0000 7f2d0d12a700 1 --2- 172.21.15.16:0/725983270 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2d080cb280 0x7f2d0806d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:11.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.864+0000 7f2d0e12c700 1 -- 172.21.15.16:0/725983270 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2d080cb280 msgr2=0x7f2d0806d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:11.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.864+0000 7f2d0e12c700 1 --2- 172.21.15.16:0/725983270 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2d080cb280 0x7f2d0806d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:11.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.864+0000 7f2d0e12c700 1 -- 172.21.15.16:0/725983270 >> v1:172.21.15.16:6789/0 conn(0x7f2d08160eb0 legacy=0x7f2d080cab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:11.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.864+0000 7f2d0e12c700 1 -- 172.21.15.16:0/725983270 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d0806be70 con 0x7f2d0806b1f0 2023-10-16T13:48:11.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.864+0000 7f2d0c929700 1 -- 172.21.15.16:0/725983270 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2059279268 0 0) 0x7f2d0806c150 con 0x7f2d08160eb0 2023-10-16T13:48:11.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.864+0000 7f2d0e12c700 1 --2- 172.21.15.16:0/725983270 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d0806b1f0 0x7f2d0806b6a0 secure :-1 s=READY pgs=2942 cs=0 l=1 rev1=1 crypto rx=0x7f2d00009d90 tx=0x7f2d0000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b97fa0b9ee266f3b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:11.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.864+0000 7f2d0c929700 1 -- 172.21.15.16:0/725983270 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d0000e070 con 0x7f2d0806b1f0 2023-10-16T13:48:11.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.864+0000 7f2d0c929700 1 -- 172.21.15.16:0/725983270 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d0000bbd0 con 0x7f2d0806b1f0 2023-10-16T13:48:11.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.864+0000 7f2d0c929700 1 -- 172.21.15.16:0/725983270 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d00012430 con 0x7f2d0806b1f0 2023-10-16T13:48:11.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.864+0000 7f2d0fb8f700 1 -- 172.21.15.16:0/725983270 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d0806b1f0 msgr2=0x7f2d0806b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:11.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.864+0000 7f2d0fb8f700 1 --2- 172.21.15.16:0/725983270 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d0806b1f0 0x7f2d0806b6a0 secure :-1 s=READY pgs=2942 cs=0 l=1 rev1=1 crypto rx=0x7f2d00009d90 tx=0x7f2d0000b330 comp rx=0 tx=0).stop 2023-10-16T13:48:11.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.865+0000 7f2d0fb8f700 1 -- 172.21.15.16:0/725983270 shutdown_connections 2023-10-16T13:48:11.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.865+0000 7f2d0fb8f700 1 --2- 172.21.15.16:0/725983270 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d0806b1f0 0x7f2d0806b6a0 unknown :-1 s=CLOSED pgs=2942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:11.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.865+0000 7f2d0fb8f700 1 --2- 172.21.15.16:0/725983270 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2d080cb280 0x7f2d0806d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:11.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.865+0000 7f2d0fb8f700 1 -- 172.21.15.16:0/725983270 >> 172.21.15.16:0/725983270 conn(0x7f2d080cc5b0 msgr2=0x7f2d080cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:11.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.865+0000 7f2d0fb8f700 1 -- 172.21.15.16:0/725983270 shutdown_connections 2023-10-16T13:48:11.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.865+0000 7f2d0fb8f700 1 -- 172.21.15.16:0/725983270 wait complete. 2023-10-16T13:48:11.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.866+0000 7f2d0fb8f700 1 Processor -- start 2023-10-16T13:48:11.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.866+0000 7f2d0fb8f700 1 -- start start 2023-10-16T13:48:11.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.867+0000 7f2d0fb8f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d0806b1f0 0x7f2d081f66f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:11.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.867+0000 7f2d0fb8f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2d080cb280 0x7f2d080b2800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:11.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.867+0000 7f2d0fb8f700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2d08160eb0 0x7f2d0815ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:11.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.867+0000 7f2d0fb8f700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d080c7ee0 con 0x7f2d0806b1f0 2023-10-16T13:48:11.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.867+0000 7f2d0fb8f700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d081526d0 con 0x7f2d08160eb0 2023-10-16T13:48:11.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.867+0000 7f2d0fb8f700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d081f6cd0 con 0x7f2d080cb280 2023-10-16T13:48:11.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.867+0000 7f2d0d92b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d0806b1f0 0x7f2d081f66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:11.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.867+0000 7f2d0d92b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d0806b1f0 0x7f2d081f66f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57146/0 (socket says 172.21.15.16:57146) 2023-10-16T13:48:11.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.867+0000 7f2d0d92b700 1 -- 172.21.15.16:0/1795523751 learned_addr learned my addr 172.21.15.16:0/1795523751 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:11.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.867+0000 7f2d0d12a700 1 --2- 172.21.15.16:0/1795523751 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2d080cb280 0x7f2d080b2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:11.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.867+0000 7f2d0e12c700 1 --2- 172.21.15.16:0/1795523751 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2d08160eb0 0x7f2d0815ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:11.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.867+0000 7f2d0d92b700 1 -- 172.21.15.16:0/1795523751 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2d080cb280 msgr2=0x7f2d080b2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:11.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.867+0000 7f2d0d92b700 1 --2- 172.21.15.16:0/1795523751 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2d080cb280 0x7f2d080b2800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:11.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.867+0000 7f2d0d92b700 1 -- 172.21.15.16:0/1795523751 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2d08160eb0 msgr2=0x7f2d0815ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:11.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.867+0000 7f2d0d92b700 1 --2- 172.21.15.16:0/1795523751 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2d08160eb0 0x7f2d0815ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:11.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.867+0000 7f2d0d92b700 1 -- 172.21.15.16:0/1795523751 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d00009a70 con 0x7f2d0806b1f0 2023-10-16T13:48:11.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.867+0000 7f2d0d92b700 1 --2- 172.21.15.16:0/1795523751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d0806b1f0 0x7f2d081f66f0 secure :-1 s=READY pgs=2943 cs=0 l=1 rev1=1 crypto rx=0x7f2cf80011e0 tx=0x7f2cf800ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:11.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.867+0000 7f2cfe7fc700 1 -- 172.21.15.16:0/1795523751 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cf8004c80 con 0x7f2d0806b1f0 2023-10-16T13:48:11.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.867+0000 7f2cfe7fc700 1 -- 172.21.15.16:0/1795523751 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cf8004e00 con 0x7f2d0806b1f0 2023-10-16T13:48:11.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.867+0000 7f2d0fb8f700 1 -- 172.21.15.16:0/1795523751 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d081f6fb0 con 0x7f2d0806b1f0 2023-10-16T13:48:11.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.867+0000 7f2cfe7fc700 1 -- 172.21.15.16:0/1795523751 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cf8007310 con 0x7f2d0806b1f0 2023-10-16T13:48:11.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.868+0000 7f2d0fb8f700 1 -- 172.21.15.16:0/1795523751 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d0815f180 con 0x7f2d0806b1f0 2023-10-16T13:48:11.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.868+0000 7f2d0fb8f700 1 -- 172.21.15.16:0/1795523751 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d08161320 con 0x7f2d0806b1f0 2023-10-16T13:48:11.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.869+0000 7f2cfe7fc700 1 -- 172.21.15.16:0/1795523751 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f2cf8007470 con 0x7f2d0806b1f0 2023-10-16T13:48:11.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.869+0000 7f2cfe7fc700 1 --2- 172.21.15.16:0/1795523751 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f2cec06b140 0x7f2cec06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:11.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.869+0000 7f2cfe7fc700 1 -- 172.21.15.16:0/1795523751 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(254..254 src has 1..254) v4 ==== 7115+0+0 (secure 0 0 0) 0x7f2cf8089ca0 con 0x7f2d0806b1f0 2023-10-16T13:48:11.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.869+0000 7f2d0d12a700 1 --2- 172.21.15.16:0/1795523751 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f2cec06b140 0x7f2cec06d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:11.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.870+0000 7f2d0d12a700 1 --2- 172.21.15.16:0/1795523751 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f2cec06b140 0x7f2cec06d600 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2d08152930 tx=0x7f2d04008040 comp rx=0 tx=0).ready entity=mgr.12678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:11.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:11.874+0000 7f2cfe7fc700 1 -- 172.21.15.16:0/1795523751 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2cf80543d0 con 0x7f2d0806b1f0 2023-10-16T13:48:12.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.124+0000 7f2d0fb8f700 1 -- 172.21.15.16:0/1795523751 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "backup_fs"} v 0) v1 -- 0x7f2d0804ebb0 con 0x7f2d0806b1f0 2023-10-16T13:48:12.127 INFO:tasks.ceph.mon.a.smithi016.stderr:2023-10-16T13:48:12.124+0000 7f9fc8535700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-16T13:48:12.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.126+0000 7f2cfe7fc700 1 -- 172.21.15.16:0/1795523751 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "backup_fs"}]=0 backup_fs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v138) v1 ==== 167+0+0 (secure 0 0 0) 0x7f2cf80579f0 con 0x7f2d0806b1f0 2023-10-16T13:48:12.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.129+0000 7f2d0fb8f700 1 -- 172.21.15.16:0/1795523751 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f2cec06b140 msgr2=0x7f2cec06d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:12.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.129+0000 7f2d0fb8f700 1 --2- 172.21.15.16:0/1795523751 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f2cec06b140 0x7f2cec06d600 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2d08152930 tx=0x7f2d04008040 comp rx=0 tx=0).stop 2023-10-16T13:48:12.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.129+0000 7f2d0fb8f700 1 -- 172.21.15.16:0/1795523751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d0806b1f0 msgr2=0x7f2d081f66f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:12.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.129+0000 7f2d0fb8f700 1 --2- 172.21.15.16:0/1795523751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d0806b1f0 0x7f2d081f66f0 secure :-1 s=READY pgs=2943 cs=0 l=1 rev1=1 crypto rx=0x7f2cf80011e0 tx=0x7f2cf800ca30 comp rx=0 tx=0).stop 2023-10-16T13:48:12.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.129+0000 7f2d0fb8f700 1 -- 172.21.15.16:0/1795523751 shutdown_connections 2023-10-16T13:48:12.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.129+0000 7f2d0fb8f700 1 --2- 172.21.15.16:0/1795523751 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f2cec06b140 0x7f2cec06d600 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:12.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.129+0000 7f2d0fb8f700 1 --2- 172.21.15.16:0/1795523751 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2d08160eb0 0x7f2d0815ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:12.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.129+0000 7f2d0fb8f700 1 --2- 172.21.15.16:0/1795523751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d0806b1f0 0x7f2d081f66f0 unknown :-1 s=CLOSED pgs=2943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:12.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.129+0000 7f2d0fb8f700 1 --2- 172.21.15.16:0/1795523751 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2d080cb280 0x7f2d080b2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:12.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.129+0000 7f2d0fb8f700 1 -- 172.21.15.16:0/1795523751 >> 172.21.15.16:0/1795523751 conn(0x7f2d080cc5b0 msgr2=0x7f2d0815ff50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:12.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.129+0000 7f2d0fb8f700 1 -- 172.21.15.16:0/1795523751 shutdown_connections 2023-10-16T13:48:12.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.129+0000 7f2d0fb8f700 1 -- 172.21.15.16:0/1795523751 wait complete. 2023-10-16T13:48:12.134 INFO:tasks.ceph.mds.d.smithi049.stderr: -1561> 2023-10-16T13:47:46.406+0000 7f9f1a76b700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T13:48:12.134 INFO:teuthology.orchestra.run.smithi016.stderr:backup_fs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-16T13:48:12.144 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm backup_fs --yes-i-really-mean-it 2023-10-16T13:48:12.162 INFO:tasks.ceph.mds.d.smithi049.stdout:starting mds.d at 2023-10-16T13:48:12.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.269+0000 7f2cc82a4700 1 Processor -- start 2023-10-16T13:48:12.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.269+0000 7f2cc82a4700 1 -- start start 2023-10-16T13:48:12.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.270+0000 7f2cc82a4700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2cc00af210 0x7f2cc00ce180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:12.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.270+0000 7f2cc82a4700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2cc00ce6c0 0x7f2cc006aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:12.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.270+0000 7f2cc82a4700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cc006aa10 con 0x7f2cc0160ee0 2023-10-16T13:48:12.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.270+0000 7f2cc82a4700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cc00cf770 con 0x7f2cc00ce6c0 2023-10-16T13:48:12.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.270+0000 7f2cc82a4700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc00ceed0 con 0x7f2cc00af210 2023-10-16T13:48:12.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.270+0000 7f2cc6841700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2cc00ce6c0 0x7f2cc006aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:12.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.270+0000 7f2cc6841700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2cc00ce6c0 0x7f2cc006aea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57154/0 (socket says 172.21.15.16:57154) 2023-10-16T13:48:12.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.270+0000 7f2cc6040700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2cc0160ee0 0x7f2cc00aeb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46428/0 (socket says 172.21.15.16:46428) 2023-10-16T13:48:12.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.270+0000 7f2cc6841700 1 -- 172.21.15.16:0/1995907167 learned_addr learned my addr 172.21.15.16:0/1995907167 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:12.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.270+0000 7f2cc583f700 1 --2- 172.21.15.16:0/1995907167 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2cc00af210 0x7f2cc00ce180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:12.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.270+0000 7f2cc6841700 1 -- 172.21.15.16:0/1995907167 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2cc00af210 msgr2=0x7f2cc00ce180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:12.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.270+0000 7f2cc6841700 1 --2- 172.21.15.16:0/1995907167 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2cc00af210 0x7f2cc00ce180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:12.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.270+0000 7f2cc6841700 1 -- 172.21.15.16:0/1995907167 >> v1:172.21.15.16:6789/0 conn(0x7f2cc0160ee0 legacy=0x7f2cc00aeb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:12.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.270+0000 7f2cc6841700 1 -- 172.21.15.16:0/1995907167 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cc00cf1b0 con 0x7f2cc00ce6c0 2023-10-16T13:48:12.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.270+0000 7f2cc503e700 1 -- 172.21.15.16:0/1995907167 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2592741897 0 0) 0x7f2cc006aa10 con 0x7f2cc0160ee0 2023-10-16T13:48:12.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.270+0000 7f2cc6841700 1 --2- 172.21.15.16:0/1995907167 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2cc00ce6c0 0x7f2cc006aea0 secure :-1 s=READY pgs=2946 cs=0 l=1 rev1=1 crypto rx=0x7f2cb8009e50 tx=0x7f2cb800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33cd98342e515377 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:12.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.271+0000 7f2cc503e700 1 -- 172.21.15.16:0/1995907167 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cb800e070 con 0x7f2cc00ce6c0 2023-10-16T13:48:12.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.271+0000 7f2cc503e700 1 -- 172.21.15.16:0/1995907167 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cb800b9c0 con 0x7f2cc00ce6c0 2023-10-16T13:48:12.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.271+0000 7f2cc503e700 1 -- 172.21.15.16:0/1995907167 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cb800bd70 con 0x7f2cc00ce6c0 2023-10-16T13:48:12.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.271+0000 7f2cc82a4700 1 -- 172.21.15.16:0/1995907167 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2cc00ce6c0 msgr2=0x7f2cc006aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:12.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.271+0000 7f2cc82a4700 1 --2- 172.21.15.16:0/1995907167 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2cc00ce6c0 0x7f2cc006aea0 secure :-1 s=READY pgs=2946 cs=0 l=1 rev1=1 crypto rx=0x7f2cb8009e50 tx=0x7f2cb800b330 comp rx=0 tx=0).stop 2023-10-16T13:48:12.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.271+0000 7f2cc82a4700 1 -- 172.21.15.16:0/1995907167 shutdown_connections 2023-10-16T13:48:12.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.271+0000 7f2cc82a4700 1 --2- 172.21.15.16:0/1995907167 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2cc00ce6c0 0x7f2cc006aea0 unknown :-1 s=CLOSED pgs=2946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:12.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.271+0000 7f2cc82a4700 1 --2- 172.21.15.16:0/1995907167 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2cc00af210 0x7f2cc00ce180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:12.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.271+0000 7f2cc82a4700 1 -- 172.21.15.16:0/1995907167 >> 172.21.15.16:0/1995907167 conn(0x7f2cc00cc5e0 msgr2=0x7f2cc00cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:12.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.271+0000 7f2cc82a4700 1 -- 172.21.15.16:0/1995907167 shutdown_connections 2023-10-16T13:48:12.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.271+0000 7f2cc82a4700 1 -- 172.21.15.16:0/1995907167 wait complete. 2023-10-16T13:48:12.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.272+0000 7f2cc82a4700 1 Processor -- start 2023-10-16T13:48:12.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.272+0000 7f2cc82a4700 1 -- start start 2023-10-16T13:48:12.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.273+0000 7f2cc82a4700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2cc00af210 0x7f2cc00c3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:12.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.273+0000 7f2cc82a4700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2cc00ce6c0 0x7f2cc01f6d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:12.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.273+0000 7f2cc82a4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2cc0160ee0 0x7f2cc0155770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:12.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.273+0000 7f2cc82a4700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc00c4eb0 con 0x7f2cc0160ee0 2023-10-16T13:48:12.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.273+0000 7f2cc82a4700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc00c42a0 con 0x7f2cc00ce6c0 2023-10-16T13:48:12.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.273+0000 7f2cc82a4700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc00c4580 con 0x7f2cc00af210 2023-10-16T13:48:12.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.273+0000 7f2cc6841700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2cc0160ee0 0x7f2cc0155770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:12.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.273+0000 7f2cc6841700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2cc0160ee0 0x7f2cc0155770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57164/0 (socket says 172.21.15.16:57164) 2023-10-16T13:48:12.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.273+0000 7f2cc6841700 1 -- 172.21.15.16:0/937071184 learned_addr learned my addr 172.21.15.16:0/937071184 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:12.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.273+0000 7f2cc583f700 1 --2- 172.21.15.16:0/937071184 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2cc00ce6c0 0x7f2cc01f6d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:12.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.274+0000 7f2cc6040700 1 --2- 172.21.15.16:0/937071184 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2cc00af210 0x7f2cc00c3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:12.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.274+0000 7f2cc6841700 1 -- 172.21.15.16:0/937071184 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2cc00af210 msgr2=0x7f2cc00c3bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:12.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.274+0000 7f2cc6841700 1 --2- 172.21.15.16:0/937071184 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2cc00af210 0x7f2cc00c3bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:12.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.274+0000 7f2cc6841700 1 -- 172.21.15.16:0/937071184 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2cc00ce6c0 msgr2=0x7f2cc01f6d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:12.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.274+0000 7f2cc6841700 1 --2- 172.21.15.16:0/937071184 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2cc00ce6c0 0x7f2cc01f6d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:12.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.274+0000 7f2cc6841700 1 -- 172.21.15.16:0/937071184 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cb8009b30 con 0x7f2cc0160ee0 2023-10-16T13:48:12.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.274+0000 7f2cc583f700 1 --2- 172.21.15.16:0/937071184 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2cc00ce6c0 0x7f2cc01f6d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:48:12.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.274+0000 7f2cc6841700 1 --2- 172.21.15.16:0/937071184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2cc0160ee0 0x7f2cc0155770 secure :-1 s=READY pgs=2947 cs=0 l=1 rev1=1 crypto rx=0x7f2cb800b730 tx=0x7f2cb8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:12.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.274+0000 7f2cb6ffd700 1 -- 172.21.15.16:0/937071184 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cb80099b0 con 0x7f2cc0160ee0 2023-10-16T13:48:12.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.274+0000 7f2cb6ffd700 1 -- 172.21.15.16:0/937071184 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cb80092e0 con 0x7f2cc0160ee0 2023-10-16T13:48:12.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.274+0000 7f2cc82a4700 1 -- 172.21.15.16:0/937071184 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cc00c4860 con 0x7f2cc0160ee0 2023-10-16T13:48:12.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.274+0000 7f2cb6ffd700 1 -- 172.21.15.16:0/937071184 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cb801c660 con 0x7f2cc0160ee0 2023-10-16T13:48:12.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.274+0000 7f2cc82a4700 1 -- 172.21.15.16:0/937071184 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cc0155cb0 con 0x7f2cc0160ee0 2023-10-16T13:48:12.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.275+0000 7f2cc82a4700 1 -- 172.21.15.16:0/937071184 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cc004ebb0 con 0x7f2cc0160ee0 2023-10-16T13:48:12.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.275+0000 7f2cb6ffd700 1 -- 172.21.15.16:0/937071184 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f2cb801c7c0 con 0x7f2cc0160ee0 2023-10-16T13:48:12.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.276+0000 7f2cb6ffd700 1 --2- 172.21.15.16:0/937071184 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f2ca406b140 0x7f2ca406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:12.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.276+0000 7f2cb6ffd700 1 -- 172.21.15.16:0/937071184 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(255..255 src has 1..255) v4 ==== 7201+0+0 (secure 0 0 0) 0x7f2cb808bdc0 con 0x7f2cc0160ee0 2023-10-16T13:48:12.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.276+0000 7f2cc6040700 1 --2- 172.21.15.16:0/937071184 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f2ca406b140 0x7f2ca406d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:12.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.276+0000 7f2cc6040700 1 --2- 172.21.15.16:0/937071184 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f2ca406b140 0x7f2ca406d600 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2cb0000f60 tx=0x7f2cb0007be0 comp rx=0 tx=0).ready entity=mgr.12678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:12.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.281+0000 7f2cb6ffd700 1 -- 172.21.15.16:0/937071184 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2cb80564b0 con 0x7f2cc0160ee0 2023-10-16T13:48:12.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:12.534+0000 7f2cc82a4700 1 -- 172.21.15.16:0/937071184 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "backup_fs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f2cc00cdba0 con 0x7f2cc0160ee0 2023-10-16T13:48:13.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:13.128+0000 7f2cb6ffd700 1 -- 172.21.15.16:0/937071184 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "backup_fs", "yes_i_really_mean_it": true}]=0 v139) v1 ==== 107+0+0 (secure 0 0 0) 0x7f2cb8059ad0 con 0x7f2cc0160ee0 2023-10-16T13:48:13.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:13.131+0000 7f2cc82a4700 1 -- 172.21.15.16:0/937071184 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f2ca406b140 msgr2=0x7f2ca406d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:13.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:13.131+0000 7f2cc82a4700 1 --2- 172.21.15.16:0/937071184 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f2ca406b140 0x7f2ca406d600 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2cb0000f60 tx=0x7f2cb0007be0 comp rx=0 tx=0).stop 2023-10-16T13:48:13.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:13.131+0000 7f2cc82a4700 1 -- 172.21.15.16:0/937071184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2cc0160ee0 msgr2=0x7f2cc0155770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:13.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:13.131+0000 7f2cc82a4700 1 --2- 172.21.15.16:0/937071184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2cc0160ee0 0x7f2cc0155770 secure :-1 s=READY pgs=2947 cs=0 l=1 rev1=1 crypto rx=0x7f2cb800b730 tx=0x7f2cb8004ad0 comp rx=0 tx=0).stop 2023-10-16T13:48:13.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:13.131+0000 7f2cc82a4700 1 -- 172.21.15.16:0/937071184 shutdown_connections 2023-10-16T13:48:13.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:13.131+0000 7f2cc82a4700 1 --2- 172.21.15.16:0/937071184 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f2ca406b140 0x7f2ca406d600 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:13.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:13.131+0000 7f2cc82a4700 1 --2- 172.21.15.16:0/937071184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2cc0160ee0 0x7f2cc0155770 unknown :-1 s=CLOSED pgs=2947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:13.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:13.131+0000 7f2cc82a4700 1 --2- 172.21.15.16:0/937071184 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2cc00af210 0x7f2cc00c3bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:13.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:13.131+0000 7f2cc82a4700 1 --2- 172.21.15.16:0/937071184 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2cc00ce6c0 0x7f2cc01f6d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:13.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:13.131+0000 7f2cc82a4700 1 -- 172.21.15.16:0/937071184 >> 172.21.15.16:0/937071184 conn(0x7f2cc00cc5e0 msgr2=0x7f2cc006b6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:13.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:13.131+0000 7f2cc82a4700 1 -- 172.21.15.16:0/937071184 shutdown_connections 2023-10-16T13:48:13.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:13.131+0000 7f2cc82a4700 1 -- 172.21.15.16:0/937071184 wait complete. 2023-10-16T13:48:13.149 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm backup_fs_metadata backup_fs_metadata --yes-i-really-really-mean-it 2023-10-16T13:48:13.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:13.283+0000 7f3df4b63700 1 Processor -- start 2023-10-16T13:48:13.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:13.283+0000 7f3df4b63700 1 -- start start 2023-10-16T13:48:13.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:13.284+0000 7f3df4b63700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3df0160e60 0x7f3df015e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:13.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:13.284+0000 7f3df4b63700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f3df006a7e0 0x7f3df006ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:13.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:13.284+0000 7f3df4b63700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3df00cdeb0 con 0x7f3df006d980 2023-10-16T13:48:13.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:13.284+0000 7f3df4b63700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3df015f6c0 con 0x7f3df0160e60 2023-10-16T13:48:13.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:13.284+0000 7f3df4b63700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3df006b190 con 0x7f3df006a7e0 2023-10-16T13:48:13.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:13.284+0000 7f3dee59c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3df0160e60 0x7f3df015e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:13.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:13.284+0000 7f3dee59c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3df0160e60 0x7f3df015e870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57168/0 (socket says 172.21.15.16:57168) 2023-10-16T13:48:13.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:13.284+0000 7f3dee59c700 1 -- 172.21.15.16:0/1284276822 learned_addr learned my addr 172.21.15.16:0/1284276822 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:13.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:13.284+0000 7f3deed9d700 1 --2- 172.21.15.16:0/1284276822 >> v2:172.21.15.49:3300/0 conn(0x7f3df006a7e0 0x7f3df006ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:13.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:13.284+0000 7f3dee59c700 1 -- 172.21.15.16:0/1284276822 >> v2:172.21.15.49:3300/0 conn(0x7f3df006a7e0 msgr2=0x7f3df006ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:13.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:13.284+0000 7f3dee59c700 1 --2- 172.21.15.16:0/1284276822 >> v2:172.21.15.49:3300/0 conn(0x7f3df006a7e0 0x7f3df006ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:13.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:13.284+0000 7f3dee59c700 1 -- 172.21.15.16:0/1284276822 >> v1:172.21.15.49:6789/0 conn(0x7f3df006d980 legacy=0x7f3df015efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:13.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:13.284+0000 7f3dee59c700 1 -- 172.21.15.16:0/1284276822 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3df01613e0 con 0x7f3df0160e60 2023-10-16T13:48:13.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:13.284+0000 7f3deed9d700 1 --2- 172.21.15.16:0/1284276822 >> v2:172.21.15.49:3300/0 conn(0x7f3df006a7e0 0x7f3df006ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:48:13.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:13.285+0000 7f3dee59c700 1 --2- 172.21.15.16:0/1284276822 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3df0160e60 0x7f3df015e870 secure :-1 s=READY pgs=2948 cs=0 l=1 rev1=1 crypto rx=0x7f3df00cf5a0 tx=0x7f3de4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa44f7d182c292e7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:13.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:13.285+0000 7f3ded59a700 1 -- 172.21.15.16:0/1284276822 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3de4004230 con 0x7f3df0160e60 2023-10-16T13:48:13.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:13.285+0000 7f3ded59a700 1 -- 172.21.15.16:0/1284276822 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3de4004390 con 0x7f3df0160e60 2023-10-16T13:48:13.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:13.285+0000 7f3ded59a700 1 -- 172.21.15.16:0/1284276822 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3de40036a0 con 0x7f3df0160e60 2023-10-16T13:48:13.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:13.285+0000 7f3df4b63700 1 -- 172.21.15.16:0/1284276822 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3df0160e60 msgr2=0x7f3df015e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:13.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:13.285+0000 7f3df4b63700 1 --2- 172.21.15.16:0/1284276822 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3df0160e60 0x7f3df015e870 secure :-1 s=READY pgs=2948 cs=0 l=1 rev1=1 crypto rx=0x7f3df00cf5a0 tx=0x7f3de4009c00 comp rx=0 tx=0).stop 2023-10-16T13:48:13.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:13.285+0000 7f3df4b63700 1 -- 172.21.15.16:0/1284276822 shutdown_connections 2023-10-16T13:48:13.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:13.285+0000 7f3df4b63700 1 --2- 172.21.15.16:0/1284276822 >> v2:172.21.15.49:3300/0 conn(0x7f3df006a7e0 0x7f3df006ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:13.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:13.285+0000 7f3df4b63700 1 --2- 172.21.15.16:0/1284276822 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3df0160e60 0x7f3df015e870 unknown :-1 s=CLOSED pgs=2948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:13.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:13.285+0000 7f3df4b63700 1 -- 172.21.15.16:0/1284276822 >> 172.21.15.16:0/1284276822 conn(0x7f3df006f430 msgr2=0x7f3df006f810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:13.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:13.285+0000 7f3df4b63700 1 -- 172.21.15.16:0/1284276822 shutdown_connections 2023-10-16T13:48:13.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:13.285+0000 7f3df4b63700 1 -- 172.21.15.16:0/1284276822 wait complete. 2023-10-16T13:48:13.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:13.287+0000 7f3df4b63700 1 Processor -- start 2023-10-16T13:48:13.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:13.287+0000 7f3df4b63700 1 -- start start 2023-10-16T13:48:13.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:13.287+0000 7f3df4b63700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3df006a7e0 0x7f3df015ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:13.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:13.287+0000 7f3df4b63700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3df006d980 0x7f3df00c4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:13.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:13.288+0000 7f3df4b63700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3df0160e60 0x7f3df00ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:13.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:13.288+0000 7f3df4b63700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3df00c6460 con 0x7f3df006d980 2023-10-16T13:48:13.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:13.288+0000 7f3df4b63700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3df015cfd0 con 0x7f3df006a7e0 2023-10-16T13:48:13.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:13.288+0000 7f3df4b63700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3df00c5850 con 0x7f3df0160e60 2023-10-16T13:48:13.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:13.288+0000 7f3dedd9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3df006d980 0x7f3df00c4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:13.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:13.288+0000 7f3dedd9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3df006d980 0x7f3df00c4d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57174/0 (socket says 172.21.15.16:57174) 2023-10-16T13:48:13.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:13.288+0000 7f3dedd9b700 1 -- 172.21.15.16:0/3405234937 learned_addr learned my addr 172.21.15.16:0/3405234937 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:13.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:13.288+0000 7f3dee59c700 1 --2- 172.21.15.16:0/3405234937 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3df006a7e0 0x7f3df015ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:13.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:13.288+0000 7f3deed9d700 1 --2- 172.21.15.16:0/3405234937 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3df0160e60 0x7f3df00ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:13.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:13.288+0000 7f3dedd9b700 1 -- 172.21.15.16:0/3405234937 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3df0160e60 msgr2=0x7f3df00ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:13.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:13.288+0000 7f3dedd9b700 1 --2- 172.21.15.16:0/3405234937 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3df0160e60 0x7f3df00ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:13.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:13.288+0000 7f3dedd9b700 1 -- 172.21.15.16:0/3405234937 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3df006a7e0 msgr2=0x7f3df015ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:13.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:13.288+0000 7f3dedd9b700 1 --2- 172.21.15.16:0/3405234937 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3df006a7e0 0x7f3df015ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:13.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:13.288+0000 7f3dedd9b700 1 -- 172.21.15.16:0/3405234937 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3de4009550 con 0x7f3df006d980 2023-10-16T13:48:13.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:13.288+0000 7f3dedd9b700 1 --2- 172.21.15.16:0/3405234937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3df006d980 0x7f3df00c4d90 secure :-1 s=READY pgs=2949 cs=0 l=1 rev1=1 crypto rx=0x7f3de000c520 tx=0x7f3de000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:13.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:13.288+0000 7f3ddeffd700 1 -- 172.21.15.16:0/3405234937 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3de0016070 con 0x7f3df006d980 2023-10-16T13:48:13.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:13.288+0000 7f3ddeffd700 1 -- 172.21.15.16:0/3405234937 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3de0006860 con 0x7f3df006d980 2023-10-16T13:48:13.299 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:13.288+0000 7f3df4b63700 1 -- 172.21.15.16:0/3405234937 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3df00c5b30 con 0x7f3df006d980 2023-10-16T13:48:13.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:13.288+0000 7f3ddeffd700 1 -- 172.21.15.16:0/3405234937 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3de0006bf0 con 0x7f3df006d980 2023-10-16T13:48:13.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:13.288+0000 7f3df4b63700 1 -- 172.21.15.16:0/3405234937 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3df00c60f0 con 0x7f3df006d980 2023-10-16T13:48:13.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:13.289+0000 7f3df4b63700 1 -- 172.21.15.16:0/3405234937 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3df00c62b0 con 0x7f3df006d980 2023-10-16T13:48:13.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:13.290+0000 7f3ddeffd700 1 -- 172.21.15.16:0/3405234937 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f3de0006d50 con 0x7f3df006d980 2023-10-16T13:48:13.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:13.290+0000 7f3ddeffd700 1 --2- 172.21.15.16:0/3405234937 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f3dd006b140 0x7f3dd006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:13.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:13.290+0000 7f3ddeffd700 1 -- 172.21.15.16:0/3405234937 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(255..255 src has 1..255) v4 ==== 7201+0+0 (secure 0 0 0) 0x7f3de0089650 con 0x7f3df006d980 2023-10-16T13:48:13.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:13.290+0000 7f3dee59c700 1 --2- 172.21.15.16:0/3405234937 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f3dd006b140 0x7f3dd006d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:13.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:13.291+0000 7f3dee59c700 1 --2- 172.21.15.16:0/3405234937 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f3dd006b140 0x7f3dd006d600 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3de4009130 tx=0x7f3de4008030 comp rx=0 tx=0).ready entity=mgr.12678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:13.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:13.295+0000 7f3ddeffd700 1 -- 172.21.15.16:0/3405234937 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3de0053f20 con 0x7f3df006d980 2023-10-16T13:48:13.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:13.497+0000 7f3df4b63700 1 -- 172.21.15.16:0/3405234937 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "backup_fs_metadata", "pool2": "backup_fs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f3df004ec20 con 0x7f3df006d980 2023-10-16T13:48:14.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:14.373+0000 7f3ddeffd700 1 -- 172.21.15.16:0/3405234937 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "backup_fs_metadata", "pool2": "backup_fs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'backup_fs_metadata' removed v256) v1 ==== 190+0+0 (secure 0 0 0) 0x7f3de0057540 con 0x7f3df006d980 2023-10-16T13:48:14.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:14.378+0000 7f3df4b63700 1 -- 172.21.15.16:0/3405234937 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f3dd006b140 msgr2=0x7f3dd006d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:14.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:14.378+0000 7f3df4b63700 1 --2- 172.21.15.16:0/3405234937 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f3dd006b140 0x7f3dd006d600 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3de4009130 tx=0x7f3de4008030 comp rx=0 tx=0).stop 2023-10-16T13:48:14.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:14.378+0000 7f3df4b63700 1 -- 172.21.15.16:0/3405234937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3df006d980 msgr2=0x7f3df00c4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:14.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:14.378+0000 7f3df4b63700 1 --2- 172.21.15.16:0/3405234937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3df006d980 0x7f3df00c4d90 secure :-1 s=READY pgs=2949 cs=0 l=1 rev1=1 crypto rx=0x7f3de000c520 tx=0x7f3de000c8e0 comp rx=0 tx=0).stop 2023-10-16T13:48:14.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:14.379+0000 7f3df4b63700 1 -- 172.21.15.16:0/3405234937 shutdown_connections 2023-10-16T13:48:14.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:14.379+0000 7f3df4b63700 1 --2- 172.21.15.16:0/3405234937 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f3dd006b140 0x7f3dd006d600 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:14.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:14.379+0000 7f3df4b63700 1 --2- 172.21.15.16:0/3405234937 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3df0160e60 0x7f3df00ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:14.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:14.379+0000 7f3df4b63700 1 --2- 172.21.15.16:0/3405234937 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3df006a7e0 0x7f3df015ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:14.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:14.379+0000 7f3df4b63700 1 --2- 172.21.15.16:0/3405234937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3df006d980 0x7f3df00c4d90 unknown :-1 s=CLOSED pgs=2949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:14.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:14.379+0000 7f3df4b63700 1 -- 172.21.15.16:0/3405234937 >> 172.21.15.16:0/3405234937 conn(0x7f3df006f430 msgr2=0x7f3df0157810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:14.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:14.379+0000 7f3df4b63700 1 -- 172.21.15.16:0/3405234937 shutdown_connections 2023-10-16T13:48:14.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:14.380+0000 7f3df4b63700 1 -- 172.21.15.16:0/3405234937 wait complete. 2023-10-16T13:48:14.385 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'backup_fs_metadata' removed 2023-10-16T13:48:14.399 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm backup_fs_data backup_fs_data --yes-i-really-really-mean-it 2023-10-16T13:48:14.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:14.536+0000 7f564b0fa700 1 Processor -- start 2023-10-16T13:48:14.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:14.536+0000 7f564b0fa700 1 -- start start 2023-10-16T13:48:14.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:14.537+0000 7f564b0fa700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f564415d890 0x7f56440ce5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:14.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:14.537+0000 7f564b0fa700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f56440ceb20 0x7f56440cffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:14.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:14.537+0000 7f564b0fa700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56440b6800 con 0x7f56440d04e0 2023-10-16T13:48:14.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:14.537+0000 7f564b0fa700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56440d2580 con 0x7f56440ceb20 2023-10-16T13:48:14.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:14.537+0000 7f564b0fa700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56440d33e0 con 0x7f564415d890 2023-10-16T13:48:14.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:14.537+0000 7f5643fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f56440ceb20 0x7f56440cffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:14.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:14.537+0000 7f5643fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f56440ceb20 0x7f56440cffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57182/0 (socket says 172.21.15.16:57182) 2023-10-16T13:48:14.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:14.537+0000 7f5643fff700 1 -- 172.21.15.16:0/3515126289 learned_addr learned my addr 172.21.15.16:0/3515126289 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:14.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:14.537+0000 7f5648e96700 1 --2- 172.21.15.16:0/3515126289 >> v2:172.21.15.49:3300/0 conn(0x7f564415d890 0x7f56440ce5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:14.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:14.537+0000 7f5643fff700 1 -- 172.21.15.16:0/3515126289 >> v2:172.21.15.49:3300/0 conn(0x7f564415d890 msgr2=0x7f56440ce5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:14.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:14.537+0000 7f5643fff700 1 --2- 172.21.15.16:0/3515126289 >> v2:172.21.15.49:3300/0 conn(0x7f564415d890 0x7f56440ce5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:14.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:14.537+0000 7f5643fff700 1 -- 172.21.15.16:0/3515126289 >> v1:172.21.15.49:6789/0 conn(0x7f56440d04e0 legacy=0x7f56440b5120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:14.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:14.537+0000 7f5643fff700 1 -- 172.21.15.16:0/3515126289 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56440d0910 con 0x7f56440ceb20 2023-10-16T13:48:14.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:14.537+0000 7f5643fff700 1 --2- 172.21.15.16:0/3515126289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56440ceb20 0x7f56440cffa0 secure :-1 s=READY pgs=2950 cs=0 l=1 rev1=1 crypto rx=0x7f56440d0f70 tx=0x7f5630009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=811749b0bb72ee7a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:14.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:14.538+0000 7f56437fe700 1 -- 172.21.15.16:0/3515126289 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5630011830 con 0x7f56440ceb20 2023-10-16T13:48:14.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:14.538+0000 7f56437fe700 1 -- 172.21.15.16:0/3515126289 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5630011990 con 0x7f56440ceb20 2023-10-16T13:48:14.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:14.538+0000 7f56437fe700 1 -- 172.21.15.16:0/3515126289 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5630011d20 con 0x7f56440ceb20 2023-10-16T13:48:14.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:14.538+0000 7f564b0fa700 1 -- 172.21.15.16:0/3515126289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56440ceb20 msgr2=0x7f56440cffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:14.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:14.538+0000 7f564b0fa700 1 --2- 172.21.15.16:0/3515126289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56440ceb20 0x7f56440cffa0 secure :-1 s=READY pgs=2950 cs=0 l=1 rev1=1 crypto rx=0x7f56440d0f70 tx=0x7f5630009bd0 comp rx=0 tx=0).stop 2023-10-16T13:48:14.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:14.538+0000 7f564b0fa700 1 -- 172.21.15.16:0/3515126289 shutdown_connections 2023-10-16T13:48:14.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:14.538+0000 7f564b0fa700 1 --2- 172.21.15.16:0/3515126289 >> v2:172.21.15.49:3300/0 conn(0x7f564415d890 0x7f56440ce5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:14.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:14.538+0000 7f564b0fa700 1 --2- 172.21.15.16:0/3515126289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56440ceb20 0x7f56440cffa0 unknown :-1 s=CLOSED pgs=2950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:14.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:14.538+0000 7f564b0fa700 1 -- 172.21.15.16:0/3515126289 >> 172.21.15.16:0/3515126289 conn(0x7f5644072360 msgr2=0x7f5644072770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:14.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:14.538+0000 7f564b0fa700 1 -- 172.21.15.16:0/3515126289 shutdown_connections 2023-10-16T13:48:14.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:14.538+0000 7f564b0fa700 1 -- 172.21.15.16:0/3515126289 wait complete. 2023-10-16T13:48:14.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:14.539+0000 7f564b0fa700 1 Processor -- start 2023-10-16T13:48:14.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:14.540+0000 7f564b0fa700 1 -- start start 2023-10-16T13:48:14.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:14.540+0000 7f564b0fa700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56440ceb20 0x7f56441519d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:14.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:14.540+0000 7f564b0fa700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f56440d04e0 0x7f56440c78c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:14.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:14.540+0000 7f564b0fa700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f564415d890 0x7f56440c9650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:14.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:14.540+0000 7f564b0fa700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56441515b0 con 0x7f56440ceb20 2023-10-16T13:48:14.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:14.540+0000 7f564b0fa700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56441506e0 con 0x7f564415d890 2023-10-16T13:48:14.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:14.540+0000 7f564b0fa700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5644150930 con 0x7f56440d04e0 2023-10-16T13:48:14.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:14.541+0000 7f5648e96700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56440ceb20 0x7f56441519d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:14.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:14.541+0000 7f5648e96700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56440ceb20 0x7f56441519d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57186/0 (socket says 172.21.15.16:57186) 2023-10-16T13:48:14.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:14.541+0000 7f5648e96700 1 -- 172.21.15.16:0/3040943571 learned_addr learned my addr 172.21.15.16:0/3040943571 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:14.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:14.541+0000 7f5648e96700 1 -- 172.21.15.16:0/3040943571 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f56440d04e0 msgr2=0x7f56440c78c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:14.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:14.541+0000 7f5648e96700 1 --2- 172.21.15.16:0/3040943571 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f56440d04e0 0x7f56440c78c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:14.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:14.541+0000 7f5648e96700 1 -- 172.21.15.16:0/3040943571 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f564415d890 msgr2=0x7f56440c9650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:14.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:14.541+0000 7f5649697700 1 --2- 172.21.15.16:0/3040943571 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f564415d890 0x7f56440c9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:14.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:14.541+0000 7f5648e96700 1 --2- 172.21.15.16:0/3040943571 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f564415d890 0x7f56440c9650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:14.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:14.541+0000 7f5648e96700 1 -- 172.21.15.16:0/3040943571 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5630009550 con 0x7f56440ceb20 2023-10-16T13:48:14.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:14.541+0000 7f5648e96700 1 --2- 172.21.15.16:0/3040943571 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56440ceb20 0x7f56441519d0 secure :-1 s=READY pgs=2951 cs=0 l=1 rev1=1 crypto rx=0x7f563800e8a0 tx=0x7f563800ec60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:14.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:14.541+0000 7f56417fa700 1 -- 172.21.15.16:0/3040943571 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f563800eea0 con 0x7f56440ceb20 2023-10-16T13:48:14.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:14.541+0000 7f56417fa700 1 -- 172.21.15.16:0/3040943571 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f563800c850 con 0x7f56440ceb20 2023-10-16T13:48:14.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:14.541+0000 7f564b0fa700 1 -- 172.21.15.16:0/3040943571 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5644150c10 con 0x7f56440ceb20 2023-10-16T13:48:14.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:14.541+0000 7f56417fa700 1 -- 172.21.15.16:0/3040943571 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f563800cbc0 con 0x7f56440ceb20 2023-10-16T13:48:14.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:14.541+0000 7f564b0fa700 1 -- 172.21.15.16:0/3040943571 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56441511d0 con 0x7f56440ceb20 2023-10-16T13:48:14.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:14.542+0000 7f564b0fa700 1 -- 172.21.15.16:0/3040943571 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56441516f0 con 0x7f56440ceb20 2023-10-16T13:48:14.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:14.543+0000 7f56417fa700 1 -- 172.21.15.16:0/3040943571 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f563800cd20 con 0x7f56440ceb20 2023-10-16T13:48:14.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:14.543+0000 7f56417fa700 1 --2- 172.21.15.16:0/3040943571 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f562806b140 0x7f562806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:14.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:14.543+0000 7f56417fa700 1 -- 172.21.15.16:0/3040943571 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(256..256 src has 1..256) v4 ==== 6757+0+0 (secure 0 0 0) 0x7f5638089d70 con 0x7f56440ceb20 2023-10-16T13:48:14.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:14.543+0000 7f5643fff700 1 --2- 172.21.15.16:0/3040943571 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f562806b140 0x7f562806d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:14.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:14.544+0000 7f5643fff700 1 --2- 172.21.15.16:0/3040943571 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f562806b140 0x7f562806d600 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f56441f6160 tx=0x7f5630010000 comp rx=0 tx=0).ready entity=mgr.12678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:14.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:14.548+0000 7f56417fa700 1 -- 172.21.15.16:0/3040943571 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f56441516f0 con 0x7f56440ceb20 2023-10-16T13:48:14.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:14.749+0000 7f564b0fa700 1 -- 172.21.15.16:0/3040943571 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "backup_fs_data", "pool2": "backup_fs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f56441516f0 con 0x7f56440ceb20 2023-10-16T13:48:15.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.378+0000 7f56417fa700 1 -- 172.21.15.16:0/3040943571 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "backup_fs_data", "pool2": "backup_fs_data", "yes_i_really_really_mean_it": true}]=0 pool 'backup_fs_data' removed v257) v1 ==== 178+0+0 (secure 0 0 0) 0x7f56441516f0 con 0x7f56440ceb20 2023-10-16T13:48:15.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.383+0000 7f564b0fa700 1 -- 172.21.15.16:0/3040943571 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f562806b140 msgr2=0x7f562806d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:15.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.383+0000 7f564b0fa700 1 --2- 172.21.15.16:0/3040943571 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f562806b140 0x7f562806d600 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f56441f6160 tx=0x7f5630010000 comp rx=0 tx=0).stop 2023-10-16T13:48:15.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.383+0000 7f564b0fa700 1 -- 172.21.15.16:0/3040943571 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56440ceb20 msgr2=0x7f56441519d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:15.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.383+0000 7f564b0fa700 1 --2- 172.21.15.16:0/3040943571 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56440ceb20 0x7f56441519d0 secure :-1 s=READY pgs=2951 cs=0 l=1 rev1=1 crypto rx=0x7f563800e8a0 tx=0x7f563800ec60 comp rx=0 tx=0).stop 2023-10-16T13:48:15.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.384+0000 7f564b0fa700 1 -- 172.21.15.16:0/3040943571 shutdown_connections 2023-10-16T13:48:15.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.384+0000 7f564b0fa700 1 --2- 172.21.15.16:0/3040943571 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f562806b140 0x7f562806d600 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:15.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.384+0000 7f564b0fa700 1 --2- 172.21.15.16:0/3040943571 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f564415d890 0x7f56440c9650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:15.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.384+0000 7f564b0fa700 1 --2- 172.21.15.16:0/3040943571 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f56440ceb20 0x7f56441519d0 unknown :-1 s=CLOSED pgs=2951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:15.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.384+0000 7f564b0fa700 1 --2- 172.21.15.16:0/3040943571 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f56440d04e0 0x7f56440c78c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:15.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.384+0000 7f564b0fa700 1 -- 172.21.15.16:0/3040943571 >> 172.21.15.16:0/3040943571 conn(0x7f5644072360 msgr2=0x7f5644072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:15.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.384+0000 7f564b0fa700 1 -- 172.21.15.16:0/3040943571 shutdown_connections 2023-10-16T13:48:15.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.384+0000 7f564b0fa700 1 -- 172.21.15.16:0/3040943571 wait complete. 2023-10-16T13:48:15.390 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'backup_fs_data' removed 2023-10-16T13:48:15.404 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-16T13:48:15.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.538+0000 7f4f84c60700 1 Processor -- start 2023-10-16T13:48:15.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.538+0000 7f4f84c60700 1 -- start start 2023-10-16T13:48:15.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.539+0000 7f4f84c60700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f4f800ce6a0 0x7f4f8006c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:15.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.539+0000 7f4f84c60700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4f800d2110 0x7f4f8006c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:15.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.539+0000 7f4f84c60700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f800b20f0 con 0x7f4f800b2620 2023-10-16T13:48:15.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.539+0000 7f4f84c60700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f800af1f0 con 0x7f4f800ce6a0 2023-10-16T13:48:15.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.539+0000 7f4f84c60700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f800d03d0 con 0x7f4f800d2110 2023-10-16T13:48:15.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.539+0000 7f4f7e59c700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f4f800ce6a0 0x7f4f8006c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:15.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.539+0000 7f4f7dd9b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4f800d2110 0x7f4f8006c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:15.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.539+0000 7f4f7e59c700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f4f800ce6a0 0x7f4f8006c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:56256/0 (socket says 172.21.15.16:56256) 2023-10-16T13:48:15.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.539+0000 7f4f7e59c700 1 -- 172.21.15.16:0/2192412490 learned_addr learned my addr 172.21.15.16:0/2192412490 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:15.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.539+0000 7f4f7dd9b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4f800d2110 0x7f4f8006c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:45454/0 (socket says 172.21.15.16:45454) 2023-10-16T13:48:15.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.540+0000 7f4f7dd9b700 1 -- 172.21.15.16:0/2192412490 >> v2:172.21.15.49:3300/0 conn(0x7f4f800ce6a0 msgr2=0x7f4f8006c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:15.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.540+0000 7f4f7dd9b700 1 --2- 172.21.15.16:0/2192412490 >> v2:172.21.15.49:3300/0 conn(0x7f4f800ce6a0 0x7f4f8006c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:15.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.540+0000 7f4f7dd9b700 1 -- 172.21.15.16:0/2192412490 >> v1:172.21.15.49:6789/0 conn(0x7f4f800b2620 legacy=0x7f4f800b0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:15.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.540+0000 7f4f7dd9b700 1 -- 172.21.15.16:0/2192412490 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f800b2340 con 0x7f4f800d2110 2023-10-16T13:48:15.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.540+0000 7f4f7e59c700 1 --2- 172.21.15.16:0/2192412490 >> v2:172.21.15.49:3300/0 conn(0x7f4f800ce6a0 0x7f4f8006c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:48:15.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.540+0000 7f4f7dd9b700 1 --2- 172.21.15.16:0/2192412490 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4f800d2110 0x7f4f8006c840 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f4f68009840 tx=0x7f4f68009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb1da4a0c1d2c6aa server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:15.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.540+0000 7f4f7d59a700 1 -- 172.21.15.16:0/2192412490 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f68011830 con 0x7f4f800d2110 2023-10-16T13:48:15.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.540+0000 7f4f7d59a700 1 -- 172.21.15.16:0/2192412490 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f68011990 con 0x7f4f800d2110 2023-10-16T13:48:15.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.540+0000 7f4f7d59a700 1 -- 172.21.15.16:0/2192412490 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f68011d20 con 0x7f4f800d2110 2023-10-16T13:48:15.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.541+0000 7f4f84c60700 1 -- 172.21.15.16:0/2192412490 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4f800d2110 msgr2=0x7f4f8006c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:15.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.541+0000 7f4f84c60700 1 --2- 172.21.15.16:0/2192412490 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4f800d2110 0x7f4f8006c840 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f4f68009840 tx=0x7f4f68009b50 comp rx=0 tx=0).stop 2023-10-16T13:48:15.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.541+0000 7f4f84c60700 1 -- 172.21.15.16:0/2192412490 shutdown_connections 2023-10-16T13:48:15.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.541+0000 7f4f84c60700 1 --2- 172.21.15.16:0/2192412490 >> v2:172.21.15.49:3300/0 conn(0x7f4f800ce6a0 0x7f4f8006c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:15.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.541+0000 7f4f84c60700 1 --2- 172.21.15.16:0/2192412490 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4f800d2110 0x7f4f8006c840 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:15.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.541+0000 7f4f84c60700 1 -- 172.21.15.16:0/2192412490 >> 172.21.15.16:0/2192412490 conn(0x7f4f800d15f0 msgr2=0x7f4f800d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:15.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.541+0000 7f4f84c60700 1 -- 172.21.15.16:0/2192412490 shutdown_connections 2023-10-16T13:48:15.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.541+0000 7f4f84c60700 1 -- 172.21.15.16:0/2192412490 wait complete. 2023-10-16T13:48:15.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.542+0000 7f4f84c60700 1 Processor -- start 2023-10-16T13:48:15.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.542+0000 7f4f84c60700 1 -- start start 2023-10-16T13:48:15.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.543+0000 7f4f84c60700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4f800b2620 0x7f4f800cb0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:15.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.543+0000 7f4f84c60700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4f800ce6a0 0x7f4f8015e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:15.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.543+0000 7f4f84c60700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f800d2110 0x7f4f801f6a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:15.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.543+0000 7f4f84c60700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f800cb600 con 0x7f4f800d2110 2023-10-16T13:48:15.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.543+0000 7f4f84c60700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f8015e960 con 0x7f4f800b2620 2023-10-16T13:48:15.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.543+0000 7f4f84c60700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f800ca020 con 0x7f4f800ce6a0 2023-10-16T13:48:15.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.543+0000 7f4f7e59c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4f800b2620 0x7f4f800cb0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:15.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.543+0000 7f4f7ed9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f800d2110 0x7f4f801f6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:15.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.543+0000 7f4f7e59c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4f800b2620 0x7f4f800cb0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:56262/0 (socket says 172.21.15.16:56262) 2023-10-16T13:48:15.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.543+0000 7f4f7dd9b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4f800ce6a0 0x7f4f8015e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:15.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.543+0000 7f4f7e59c700 1 -- 172.21.15.16:0/1073098886 learned_addr learned my addr 172.21.15.16:0/1073098886 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:15.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.543+0000 7f4f7ed9d700 1 -- 172.21.15.16:0/1073098886 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4f800ce6a0 msgr2=0x7f4f8015e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:15.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.543+0000 7f4f7ed9d700 1 --2- 172.21.15.16:0/1073098886 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4f800ce6a0 0x7f4f8015e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:15.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.543+0000 7f4f7ed9d700 1 -- 172.21.15.16:0/1073098886 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4f800b2620 msgr2=0x7f4f800cb0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:15.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.543+0000 7f4f7ed9d700 1 --2- 172.21.15.16:0/1073098886 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4f800b2620 0x7f4f800cb0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:15.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.543+0000 7f4f7ed9d700 1 -- 172.21.15.16:0/1073098886 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f70009550 con 0x7f4f800d2110 2023-10-16T13:48:15.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.543+0000 7f4f7e59c700 1 --2- 172.21.15.16:0/1073098886 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4f800b2620 0x7f4f800cb0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:48:15.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.544+0000 7f4f7ed9d700 1 --2- 172.21.15.16:0/1073098886 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f800d2110 0x7f4f801f6a20 secure :-1 s=READY pgs=2952 cs=0 l=1 rev1=1 crypto rx=0x7f4f7800cb90 tx=0x7f4f7800cea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:15.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.544+0000 7f4f76ffd700 1 -- 172.21.15.16:0/1073098886 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f78003c00 con 0x7f4f800d2110 2023-10-16T13:48:15.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.544+0000 7f4f76ffd700 1 -- 172.21.15.16:0/1073098886 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f78003d60 con 0x7f4f800d2110 2023-10-16T13:48:15.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.544+0000 7f4f84c60700 1 -- 172.21.15.16:0/1073098886 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f680094a0 con 0x7f4f800d2110 2023-10-16T13:48:15.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.544+0000 7f4f76ffd700 1 -- 172.21.15.16:0/1073098886 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f7800aa80 con 0x7f4f800d2110 2023-10-16T13:48:15.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.544+0000 7f4f84c60700 1 -- 172.21.15.16:0/1073098886 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f800ca5e0 con 0x7f4f800d2110 2023-10-16T13:48:15.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.545+0000 7f4f84c60700 1 -- 172.21.15.16:0/1073098886 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f8004ebb0 con 0x7f4f800d2110 2023-10-16T13:48:15.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.545+0000 7f4f76ffd700 1 -- 172.21.15.16:0/1073098886 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f4f7800abe0 con 0x7f4f800d2110 2023-10-16T13:48:15.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.546+0000 7f4f76ffd700 1 --2- 172.21.15.16:0/1073098886 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f4f6006b140 0x7f4f6006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:15.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.546+0000 7f4f76ffd700 1 -- 172.21.15.16:0/1073098886 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 6337+0+0 (secure 0 0 0) 0x7f4f78089ba0 con 0x7f4f800d2110 2023-10-16T13:48:15.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.546+0000 7f4f7e59c700 1 --2- 172.21.15.16:0/1073098886 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f4f6006b140 0x7f4f6006d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:15.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.546+0000 7f4f7e59c700 1 --2- 172.21.15.16:0/1073098886 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f4f6006b140 0x7f4f6006d600 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4f700114d0 tx=0x7f4f700092c0 comp rx=0 tx=0).ready entity=mgr.12678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:15.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.550+0000 7f4f76ffd700 1 -- 172.21.15.16:0/1073098886 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4f78054680 con 0x7f4f800d2110 2023-10-16T13:48:15.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.755+0000 7f4f84c60700 1 -- 172.21.15.16:0/1073098886 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f4f800d0110 con 0x7f4f800d2110 2023-10-16T13:48:15.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.759+0000 7f4f76ffd700 1 -- 172.21.15.16:0/1073098886 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v85) v1 ==== 115+0+0 (secure 0 0 0) 0x7f4f78057ca0 con 0x7f4f800d2110 2023-10-16T13:48:15.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.761+0000 7f4f84c60700 1 -- 172.21.15.16:0/1073098886 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f4f6006b140 msgr2=0x7f4f6006d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:15.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.761+0000 7f4f84c60700 1 --2- 172.21.15.16:0/1073098886 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f4f6006b140 0x7f4f6006d600 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4f700114d0 tx=0x7f4f700092c0 comp rx=0 tx=0).stop 2023-10-16T13:48:15.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.761+0000 7f4f84c60700 1 -- 172.21.15.16:0/1073098886 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f800d2110 msgr2=0x7f4f801f6a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:15.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.761+0000 7f4f84c60700 1 --2- 172.21.15.16:0/1073098886 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f800d2110 0x7f4f801f6a20 secure :-1 s=READY pgs=2952 cs=0 l=1 rev1=1 crypto rx=0x7f4f7800cb90 tx=0x7f4f7800cea0 comp rx=0 tx=0).stop 2023-10-16T13:48:15.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.761+0000 7f4f84c60700 1 -- 172.21.15.16:0/1073098886 shutdown_connections 2023-10-16T13:48:15.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.761+0000 7f4f84c60700 1 --2- 172.21.15.16:0/1073098886 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f4f6006b140 0x7f4f6006d600 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:15.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.761+0000 7f4f84c60700 1 --2- 172.21.15.16:0/1073098886 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f800d2110 0x7f4f801f6a20 unknown :-1 s=CLOSED pgs=2952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:15.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.761+0000 7f4f84c60700 1 --2- 172.21.15.16:0/1073098886 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4f800b2620 0x7f4f800cb0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:15.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.761+0000 7f4f84c60700 1 --2- 172.21.15.16:0/1073098886 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4f800ce6a0 0x7f4f8015e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:15.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.761+0000 7f4f84c60700 1 -- 172.21.15.16:0/1073098886 >> 172.21.15.16:0/1073098886 conn(0x7f4f800d15f0 msgr2=0x7f4f8006c300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:15.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.762+0000 7f4f84c60700 1 -- 172.21.15.16:0/1073098886 shutdown_connections 2023-10-16T13:48:15.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.762+0000 7f4f84c60700 1 -- 172.21.15.16:0/1073098886 wait complete. 2023-10-16T13:48:15.777 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-16T13:48:15.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.902+0000 7f0f1133f700 1 Processor -- start 2023-10-16T13:48:15.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.902+0000 7f0f1133f700 1 -- start start 2023-10-16T13:48:15.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.903+0000 7f0f1133f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f0c15d8d0 0x7f0f0c0b2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:15.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.903+0000 7f0f1133f700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f0c0b2f30 con 0x7f0f0c0b8fa0 2023-10-16T13:48:15.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.903+0000 7f0f1133f700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f0c0d1c40 con 0x7f0f0c0ce350 2023-10-16T13:48:15.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.903+0000 7f0f1133f700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f0c0d0bb0 con 0x7f0f0c15d8d0 2023-10-16T13:48:15.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.903+0000 7f0f0a7fc700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0f0c0b8fa0 0x7f0f0c0b6720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46482/0 (socket says 172.21.15.16:46482) 2023-10-16T13:48:15.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.903+0000 7f0f0a7fc700 1 -- 172.21.15.16:0/2667472644 learned_addr learned my addr 172.21.15.16:0/2667472644 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:48:15.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.903+0000 7f0f0affd700 1 --2- 172.21.15.16:0/2667472644 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f0c15d8d0 0x7f0f0c0b2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:15.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.903+0000 7f0f09ffb700 1 -- 172.21.15.16:0/2667472644 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1916249085 0 0) 0x7f0f0c0b2f30 con 0x7f0f0c0b8fa0 2023-10-16T13:48:15.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.903+0000 7f0f09ffb700 1 -- 172.21.15.16:0/2667472644 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ef40035b0 con 0x7f0f0c0b8fa0 2023-10-16T13:48:15.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.903+0000 7f0f09ffb700 1 -- 172.21.15.16:0/2667472644 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3286567760 0 0) 0x7f0f0c0d1c40 con 0x7f0f0c0ce350 2023-10-16T13:48:15.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.903+0000 7f0f09ffb700 1 -- 172.21.15.16:0/2667472644 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f0c0b2f30 con 0x7f0f0c0ce350 2023-10-16T13:48:15.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.903+0000 7f0f0affd700 1 -- 172.21.15.16:0/2667472644 >> v1:172.21.15.49:6789/0 conn(0x7f0f0c0ce350 legacy=0x7f0f0c0d29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:15.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.903+0000 7f0f09ffb700 1 -- 172.21.15.16:0/2667472644 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3677323178 0 0) 0x7f0ef40035b0 con 0x7f0f0c0b8fa0 2023-10-16T13:48:15.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.904+0000 7f0f0affd700 1 -- 172.21.15.16:0/2667472644 >> v1:172.21.15.16:6789/0 conn(0x7f0f0c0b8fa0 legacy=0x7f0f0c0b6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:15.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.904+0000 7f0f0affd700 1 -- 172.21.15.16:0/2667472644 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f0c0d1ee0 con 0x7f0f0c15d8d0 2023-10-16T13:48:15.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.904+0000 7f0f0affd700 1 --2- 172.21.15.16:0/2667472644 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f0c15d8d0 0x7f0f0c0b2830 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f0f00009870 tx=0x7f0f00009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=62cee91e711b6a44 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:15.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.904+0000 7f0f09ffb700 1 -- 172.21.15.16:0/2667472644 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0efc001670 con 0x7f0f0c15d8d0 2023-10-16T13:48:15.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.904+0000 7f0f09ffb700 1 -- 172.21.15.16:0/2667472644 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f00004030 con 0x7f0f0c15d8d0 2023-10-16T13:48:15.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.904+0000 7f0f09ffb700 1 -- 172.21.15.16:0/2667472644 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f000043c0 con 0x7f0f0c15d8d0 2023-10-16T13:48:15.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.904+0000 7f0f1133f700 1 -- 172.21.15.16:0/2667472644 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f0c15d8d0 msgr2=0x7f0f0c0b2830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:15.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.904+0000 7f0f1133f700 1 --2- 172.21.15.16:0/2667472644 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f0c15d8d0 0x7f0f0c0b2830 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f0f00009870 tx=0x7f0f00009b80 comp rx=0 tx=0).stop 2023-10-16T13:48:15.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.904+0000 7f0f1133f700 1 -- 172.21.15.16:0/2667472644 shutdown_connections 2023-10-16T13:48:15.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.904+0000 7f0f1133f700 1 --2- 172.21.15.16:0/2667472644 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f0c15d8d0 0x7f0f0c0b2830 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:15.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.904+0000 7f0f1133f700 1 -- 172.21.15.16:0/2667472644 >> 172.21.15.16:0/2667472644 conn(0x7f0f0c072330 msgr2=0x7f0f0c072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:15.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.905+0000 7f0f1133f700 1 -- 172.21.15.16:0/2667472644 shutdown_connections 2023-10-16T13:48:15.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.905+0000 7f0f1133f700 1 -- 172.21.15.16:0/2667472644 wait complete. 2023-10-16T13:48:15.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.906+0000 7f0f1133f700 1 Processor -- start 2023-10-16T13:48:15.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.906+0000 7f0f1133f700 1 -- start start 2023-10-16T13:48:15.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.907+0000 7f0f1133f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f0c0b8fa0 0x7f0f0c0b1f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:15.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.907+0000 7f0f1133f700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f0c0ce350 0x7f0f0c0aa9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:15.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.907+0000 7f0f1133f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f0c15d8d0 0x7f0f0c0ac460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:15.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.907+0000 7f0f1133f700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f0c0abfd0 con 0x7f0f0c0b8fa0 2023-10-16T13:48:15.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.907+0000 7f0f1133f700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f0c0b2460 con 0x7f0f0c0ce350 2023-10-16T13:48:15.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.907+0000 7f0f1133f700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f0c0ab3c0 con 0x7f0f0c15d8d0 2023-10-16T13:48:15.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.907+0000 7f0f0affd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f0c0b8fa0 0x7f0f0c0b1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:15.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.907+0000 7f0f0affd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f0c0b8fa0 0x7f0f0c0b1f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57214/0 (socket says 172.21.15.16:57214) 2023-10-16T13:48:15.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.907+0000 7f0f0affd700 1 -- 172.21.15.16:0/2897473427 learned_addr learned my addr 172.21.15.16:0/2897473427 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:15.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.907+0000 7f0f0a7fc700 1 --2- 172.21.15.16:0/2897473427 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f0c0ce350 0x7f0f0c0aa9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:15.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.907+0000 7f0f0affd700 1 -- 172.21.15.16:0/2897473427 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f0c15d8d0 msgr2=0x7f0f0c0ac460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:15.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.907+0000 7f0f0b7fe700 1 --2- 172.21.15.16:0/2897473427 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f0c15d8d0 0x7f0f0c0ac460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:15.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.907+0000 7f0f0affd700 1 --2- 172.21.15.16:0/2897473427 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f0c15d8d0 0x7f0f0c0ac460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:15.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.907+0000 7f0f0affd700 1 -- 172.21.15.16:0/2897473427 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f0c0ce350 msgr2=0x7f0f0c0aa9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:15.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.907+0000 7f0f0affd700 1 --2- 172.21.15.16:0/2897473427 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f0c0ce350 0x7f0f0c0aa9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:15.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.907+0000 7f0f0affd700 1 -- 172.21.15.16:0/2897473427 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f000094a0 con 0x7f0f0c0b8fa0 2023-10-16T13:48:15.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.907+0000 7f0f0b7fe700 1 --2- 172.21.15.16:0/2897473427 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f0c15d8d0 0x7f0f0c0ac460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:48:15.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.907+0000 7f0f0affd700 1 --2- 172.21.15.16:0/2897473427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f0c0b8fa0 0x7f0f0c0b1f20 secure :-1 s=READY pgs=2954 cs=0 l=1 rev1=1 crypto rx=0x7f0f00000c00 tx=0x7f0f00004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:15.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.907+0000 7f0ef37fe700 1 -- 172.21.15.16:0/2897473427 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f00011630 con 0x7f0f0c0b8fa0 2023-10-16T13:48:15.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.907+0000 7f0ef37fe700 1 -- 172.21.15.16:0/2897473427 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f0001f070 con 0x7f0f0c0b8fa0 2023-10-16T13:48:15.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.907+0000 7f0ef37fe700 1 -- 172.21.15.16:0/2897473427 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f000119f0 con 0x7f0f0c0b8fa0 2023-10-16T13:48:15.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.907+0000 7f0f1133f700 1 -- 172.21.15.16:0/2897473427 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f0c0ab6a0 con 0x7f0f0c0b8fa0 2023-10-16T13:48:15.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.908+0000 7f0f1133f700 1 -- 172.21.15.16:0/2897473427 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f0c0abc60 con 0x7f0f0c0b8fa0 2023-10-16T13:48:15.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.908+0000 7f0f1133f700 1 -- 172.21.15.16:0/2897473427 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f0c04ebb0 con 0x7f0f0c0b8fa0 2023-10-16T13:48:15.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.909+0000 7f0ef37fe700 1 -- 172.21.15.16:0/2897473427 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f0f00011b50 con 0x7f0f0c0b8fa0 2023-10-16T13:48:15.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.909+0000 7f0ef37fe700 1 --2- 172.21.15.16:0/2897473427 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f0ef806b140 0x7f0ef806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:15.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.909+0000 7f0ef37fe700 1 -- 172.21.15.16:0/2897473427 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 6337+0+0 (secure 0 0 0) 0x7f0f00021070 con 0x7f0f0c0b8fa0 2023-10-16T13:48:15.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.909+0000 7f0f0a7fc700 1 --2- 172.21.15.16:0/2897473427 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f0ef806b140 0x7f0ef806d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:15.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.910+0000 7f0f0a7fc700 1 --2- 172.21.15.16:0/2897473427 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f0ef806b140 0x7f0ef806d600 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0efc000f60 tx=0x7f0efc0014b0 comp rx=0 tx=0).ready entity=mgr.12678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:15.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:15.914+0000 7f0ef37fe700 1 -- 172.21.15.16:0/2897473427 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0f00056120 con 0x7f0f0c0b8fa0 2023-10-16T13:48:16.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.119+0000 7f0f1133f700 1 -- 172.21.15.16:0/2897473427 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f0f0c0b5a40 con 0x7f0f0c0b8fa0 2023-10-16T13:48:16.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.122+0000 7f0ef37fe700 1 -- 172.21.15.16:0/2897473427 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v86) v1 ==== 104+0+0 (secure 0 0 0) 0x7f0f00059740 con 0x7f0f0c0b8fa0 2023-10-16T13:48:16.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.125+0000 7f0f1133f700 1 -- 172.21.15.16:0/2897473427 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f0ef806b140 msgr2=0x7f0ef806d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:16.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.125+0000 7f0f1133f700 1 --2- 172.21.15.16:0/2897473427 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f0ef806b140 0x7f0ef806d600 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0efc000f60 tx=0x7f0efc0014b0 comp rx=0 tx=0).stop 2023-10-16T13:48:16.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.125+0000 7f0f1133f700 1 -- 172.21.15.16:0/2897473427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f0c0b8fa0 msgr2=0x7f0f0c0b1f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:16.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.125+0000 7f0f1133f700 1 --2- 172.21.15.16:0/2897473427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f0c0b8fa0 0x7f0f0c0b1f20 secure :-1 s=READY pgs=2954 cs=0 l=1 rev1=1 crypto rx=0x7f0f00000c00 tx=0x7f0f00004750 comp rx=0 tx=0).stop 2023-10-16T13:48:16.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.125+0000 7f0f1133f700 1 -- 172.21.15.16:0/2897473427 shutdown_connections 2023-10-16T13:48:16.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.125+0000 7f0f1133f700 1 --2- 172.21.15.16:0/2897473427 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f0ef806b140 0x7f0ef806d600 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:16.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.125+0000 7f0f1133f700 1 --2- 172.21.15.16:0/2897473427 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0f0c15d8d0 0x7f0f0c0ac460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:16.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.125+0000 7f0f1133f700 1 --2- 172.21.15.16:0/2897473427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f0c0b8fa0 0x7f0f0c0b1f20 unknown :-1 s=CLOSED pgs=2954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:16.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.125+0000 7f0f1133f700 1 --2- 172.21.15.16:0/2897473427 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0f0c0ce350 0x7f0f0c0aa9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:16.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.125+0000 7f0f1133f700 1 -- 172.21.15.16:0/2897473427 >> 172.21.15.16:0/2897473427 conn(0x7f0f0c072330 msgr2=0x7f0f0c15cc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:16.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.125+0000 7f0f1133f700 1 -- 172.21.15.16:0/2897473427 shutdown_connections 2023-10-16T13:48:16.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.125+0000 7f0f1133f700 1 -- 172.21.15.16:0/2897473427 wait complete. 2023-10-16T13:48:16.140 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-16T13:48:16.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.270+0000 7f04b332a700 1 Processor -- start 2023-10-16T13:48:16.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.270+0000 7f04b332a700 1 -- start start 2023-10-16T13:48:16.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.271+0000 7f04b332a700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f04ac06ca10 0x7f04ac0ae8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:16.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.271+0000 7f04b332a700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04ac0b20f0 con 0x7f04ac0d2110 2023-10-16T13:48:16.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.271+0000 7f04b332a700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04ac0d03d0 con 0x7f04ac0ce6a0 2023-10-16T13:48:16.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.271+0000 7f04b332a700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04ac0b2340 con 0x7f04ac06ca10 2023-10-16T13:48:16.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.271+0000 7f04b08c5700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f04ac0d2110 0x7f04ac0ae1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46496/0 (socket says 172.21.15.16:46496) 2023-10-16T13:48:16.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.271+0000 7f04b08c5700 1 -- 172.21.15.16:0/2965106854 learned_addr learned my addr 172.21.15.16:0/2965106854 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:48:16.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.271+0000 7f04b18c7700 1 --2- 172.21.15.16:0/2965106854 >> v2:172.21.15.49:3300/0 conn(0x7f04ac06ca10 0x7f04ac0ae8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:16.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.271+0000 7f04a3fff700 1 -- 172.21.15.16:0/2965106854 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1842320424 0 0) 0x7f04ac0b20f0 con 0x7f04ac0d2110 2023-10-16T13:48:16.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.271+0000 7f04a3fff700 1 -- 172.21.15.16:0/2965106854 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04980035b0 con 0x7f04ac0d2110 2023-10-16T13:48:16.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.271+0000 7f04a3fff700 1 -- 172.21.15.16:0/2965106854 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2665682654 0 0) 0x7f04980035b0 con 0x7f04ac0d2110 2023-10-16T13:48:16.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.271+0000 7f04a3fff700 1 -- 172.21.15.16:0/2965106854 >> v2:172.21.15.49:3300/0 conn(0x7f04ac06ca10 msgr2=0x7f04ac0ae8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:16.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.271+0000 7f04a3fff700 1 --2- 172.21.15.16:0/2965106854 >> v2:172.21.15.49:3300/0 conn(0x7f04ac06ca10 0x7f04ac0ae8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:16.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.271+0000 7f04a3fff700 1 -- 172.21.15.16:0/2965106854 >> v1:172.21.15.49:6789/0 conn(0x7f04ac0ce6a0 legacy=0x7f04ac06c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:16.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.271+0000 7f04a3fff700 1 -- 172.21.15.16:0/2965106854 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04ac06bb00 con 0x7f04ac0d2110 2023-10-16T13:48:16.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.271+0000 7f04a3fff700 1 -- 172.21.15.16:0/2965106854 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f049c001670 con 0x7f04ac0d2110 2023-10-16T13:48:16.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.272+0000 7f04a3fff700 1 -- 172.21.15.16:0/2965106854 >> v1:172.21.15.16:6789/0 conn(0x7f04ac0d2110 legacy=0x7f04ac0ae1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:16.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.272+0000 7f04a3fff700 1 --2- 172.21.15.16:0/2965106854 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0498003a00 0x7f0498003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:16.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.272+0000 7f04a3fff700 1 --2- 172.21.15.16:0/2965106854 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0498004850 0x7f0498004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:16.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.272+0000 7f04a3fff700 1 --2- 172.21.15.16:0/2965106854 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0498005210 0x7f049800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:16.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.272+0000 7f04a3fff700 1 -- 172.21.15.16:0/2965106854 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04ac0b2340 con 0x7f0498005210 2023-10-16T13:48:16.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.272+0000 7f04a3fff700 1 -- 172.21.15.16:0/2965106854 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f049800c0b0 con 0x7f0498003a00 2023-10-16T13:48:16.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.272+0000 7f04a3fff700 1 -- 172.21.15.16:0/2965106854 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f049800c270 con 0x7f0498004850 2023-10-16T13:48:16.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.272+0000 7f04b18c7700 1 --2- 172.21.15.16:0/2965106854 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0498005210 0x7f049800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:16.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.272+0000 7f04b10c6700 1 --2- 172.21.15.16:0/2965106854 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0498003a00 0x7f0498003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:16.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.272+0000 7f04b08c5700 1 --2- 172.21.15.16:0/2965106854 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0498004850 0x7f0498004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:16.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.272+0000 7f04b18c7700 1 -- 172.21.15.16:0/2965106854 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0498004850 msgr2=0x7f0498004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:16.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.272+0000 7f04b18c7700 1 --2- 172.21.15.16:0/2965106854 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0498004850 0x7f0498004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:16.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.272+0000 7f04b18c7700 1 -- 172.21.15.16:0/2965106854 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0498003a00 msgr2=0x7f0498003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:16.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.272+0000 7f04b18c7700 1 --2- 172.21.15.16:0/2965106854 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0498003a00 0x7f0498003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:16.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.272+0000 7f04b18c7700 1 -- 172.21.15.16:0/2965106854 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04980035b0 con 0x7f0498005210 2023-10-16T13:48:16.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.272+0000 7f04b18c7700 1 -- 172.21.15.16:0/2965106854 reap_dead start 2023-10-16T13:48:16.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.272+0000 7f04b18c7700 1 --2- 172.21.15.16:0/2965106854 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0498005210 0x7f049800ba10 secure :-1 s=READY pgs=2956 cs=0 l=1 rev1=1 crypto rx=0x7f04a4009fc0 tx=0x7f0498005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1fd5ae078bfd9bf server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:16.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.272+0000 7f04a3fff700 1 -- 172.21.15.16:0/2965106854 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a4003a20 con 0x7f0498005210 2023-10-16T13:48:16.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.272+0000 7f04a3fff700 1 -- 172.21.15.16:0/2965106854 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04a400f040 con 0x7f0498005210 2023-10-16T13:48:16.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.272+0000 7f04a3fff700 1 -- 172.21.15.16:0/2965106854 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a4003de0 con 0x7f0498005210 2023-10-16T13:48:16.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.273+0000 7f04b332a700 1 -- 172.21.15.16:0/2965106854 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0498005210 msgr2=0x7f049800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:16.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.273+0000 7f04b332a700 1 --2- 172.21.15.16:0/2965106854 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0498005210 0x7f049800ba10 secure :-1 s=READY pgs=2956 cs=0 l=1 rev1=1 crypto rx=0x7f04a4009fc0 tx=0x7f0498005880 comp rx=0 tx=0).stop 2023-10-16T13:48:16.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.273+0000 7f04b332a700 1 -- 172.21.15.16:0/2965106854 shutdown_connections 2023-10-16T13:48:16.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.273+0000 7f04b332a700 1 --2- 172.21.15.16:0/2965106854 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0498005210 0x7f049800ba10 unknown :-1 s=CLOSED pgs=2956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:16.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.273+0000 7f04b332a700 1 -- 172.21.15.16:0/2965106854 >> 172.21.15.16:0/2965106854 conn(0x7f04ac0d15f0 msgr2=0x7f04ac0d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:16.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.273+0000 7f04b332a700 1 -- 172.21.15.16:0/2965106854 shutdown_connections 2023-10-16T13:48:16.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.273+0000 7f04b332a700 1 -- 172.21.15.16:0/2965106854 wait complete. 2023-10-16T13:48:16.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.274+0000 7f04b332a700 1 Processor -- start 2023-10-16T13:48:16.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.274+0000 7f04b332a700 1 -- start start 2023-10-16T13:48:16.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.275+0000 7f04b332a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0498005210 0x7f04ac0b6860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:16.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.275+0000 7f04b332a700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f04ac156120 0x7f04ac1565a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:16.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.275+0000 7f04b332a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04ac0c72f0 0x7f04ac0c7770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:16.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.275+0000 7f04b332a700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04ac0c7cb0 con 0x7f04ac0c72f0 2023-10-16T13:48:16.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.275+0000 7f04b332a700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04ac0b71b0 con 0x7f04ac156120 2023-10-16T13:48:16.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.275+0000 7f04b332a700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04ac0b7490 con 0x7f0498005210 2023-10-16T13:48:16.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.275+0000 7f04b18c7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04ac0c72f0 0x7f04ac0c7770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:16.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.275+0000 7f04b18c7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04ac0c72f0 0x7f04ac0c7770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57238/0 (socket says 172.21.15.16:57238) 2023-10-16T13:48:16.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.275+0000 7f04b18c7700 1 -- 172.21.15.16:0/2584106001 learned_addr learned my addr 172.21.15.16:0/2584106001 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:16.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.275+0000 7f04b10c6700 1 --2- 172.21.15.16:0/2584106001 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0498005210 0x7f04ac0b6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:16.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.275+0000 7f04b08c5700 1 --2- 172.21.15.16:0/2584106001 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f04ac156120 0x7f04ac1565a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:16.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.275+0000 7f04b18c7700 1 -- 172.21.15.16:0/2584106001 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0498005210 msgr2=0x7f04ac0b6860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:16.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.275+0000 7f04b18c7700 1 --2- 172.21.15.16:0/2584106001 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0498005210 0x7f04ac0b6860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:16.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.275+0000 7f04b18c7700 1 -- 172.21.15.16:0/2584106001 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f04ac156120 msgr2=0x7f04ac1565a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:16.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.275+0000 7f04b18c7700 1 --2- 172.21.15.16:0/2584106001 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f04ac156120 0x7f04ac1565a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:16.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.275+0000 7f04b18c7700 1 -- 172.21.15.16:0/2584106001 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04ac0ae280 con 0x7f04ac0c72f0 2023-10-16T13:48:16.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.276+0000 7f04b18c7700 1 --2- 172.21.15.16:0/2584106001 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04ac0c72f0 0x7f04ac0c7770 secure :-1 s=READY pgs=2957 cs=0 l=1 rev1=1 crypto rx=0x7f04ac06d730 tx=0x7f04a4009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:16.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.276+0000 7f04a1ffb700 1 -- 172.21.15.16:0/2584106001 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a4003e10 con 0x7f04ac0c72f0 2023-10-16T13:48:16.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.276+0000 7f04a1ffb700 1 -- 172.21.15.16:0/2584106001 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04a4004230 con 0x7f04ac0c72f0 2023-10-16T13:48:16.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.276+0000 7f04a1ffb700 1 -- 172.21.15.16:0/2584106001 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a40045e0 con 0x7f04ac0c72f0 2023-10-16T13:48:16.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.276+0000 7f04b332a700 1 -- 172.21.15.16:0/2584106001 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04ac0ae560 con 0x7f04ac0c72f0 2023-10-16T13:48:16.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.276+0000 7f04b332a700 1 -- 172.21.15.16:0/2584106001 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04ac0b6da0 con 0x7f04ac0c72f0 2023-10-16T13:48:16.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.277+0000 7f04b332a700 1 -- 172.21.15.16:0/2584106001 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04ac0b6f60 con 0x7f04ac0c72f0 2023-10-16T13:48:16.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.277+0000 7f04a1ffb700 1 -- 172.21.15.16:0/2584106001 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f04a4004740 con 0x7f04ac0c72f0 2023-10-16T13:48:16.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.278+0000 7f04a1ffb700 1 --2- 172.21.15.16:0/2584106001 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f049006b140 0x7f049006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:16.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.278+0000 7f04a1ffb700 1 -- 172.21.15.16:0/2584106001 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 6337+0+0 (secure 0 0 0) 0x7f04a408af80 con 0x7f04ac0c72f0 2023-10-16T13:48:16.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.278+0000 7f04b10c6700 1 --2- 172.21.15.16:0/2584106001 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f049006b140 0x7f049006d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:16.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.278+0000 7f04b10c6700 1 --2- 172.21.15.16:0/2584106001 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f049006b140 0x7f049006d600 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f04ac0ccb70 tx=0x7f04a8009040 comp rx=0 tx=0).ready entity=mgr.12678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:16.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.282+0000 7f04a1ffb700 1 -- 172.21.15.16:0/2584106001 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f04a40559d0 con 0x7f04ac0c72f0 2023-10-16T13:48:16.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.488+0000 7f04b332a700 1 -- 172.21.15.16:0/2584106001 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f04ac04ebb0 con 0x7f04ac0c72f0 2023-10-16T13:48:16.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.491+0000 7f04a1ffb700 1 -- 172.21.15.16:0/2584106001 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v87) v1 ==== 120+0+0 (secure 0 0 0) 0x7f04a40180b0 con 0x7f04ac0c72f0 2023-10-16T13:48:16.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.493+0000 7f04b332a700 1 -- 172.21.15.16:0/2584106001 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f049006b140 msgr2=0x7f049006d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:16.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.493+0000 7f04b332a700 1 --2- 172.21.15.16:0/2584106001 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f049006b140 0x7f049006d600 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f04ac0ccb70 tx=0x7f04a8009040 comp rx=0 tx=0).stop 2023-10-16T13:48:16.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.493+0000 7f04b332a700 1 -- 172.21.15.16:0/2584106001 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04ac0c72f0 msgr2=0x7f04ac0c7770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:16.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.493+0000 7f04b332a700 1 --2- 172.21.15.16:0/2584106001 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04ac0c72f0 0x7f04ac0c7770 secure :-1 s=READY pgs=2957 cs=0 l=1 rev1=1 crypto rx=0x7f04ac06d730 tx=0x7f04a4009750 comp rx=0 tx=0).stop 2023-10-16T13:48:16.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.493+0000 7f04b332a700 1 -- 172.21.15.16:0/2584106001 shutdown_connections 2023-10-16T13:48:16.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.493+0000 7f04b332a700 1 --2- 172.21.15.16:0/2584106001 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f049006b140 0x7f049006d600 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:16.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.493+0000 7f04b332a700 1 --2- 172.21.15.16:0/2584106001 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04ac0c72f0 0x7f04ac0c7770 unknown :-1 s=CLOSED pgs=2957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:16.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.493+0000 7f04b332a700 1 --2- 172.21.15.16:0/2584106001 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0498005210 0x7f04ac0b6860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:16.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.493+0000 7f04b332a700 1 --2- 172.21.15.16:0/2584106001 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f04ac156120 0x7f04ac1565a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:16.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.493+0000 7f04b332a700 1 -- 172.21.15.16:0/2584106001 >> 172.21.15.16:0/2584106001 conn(0x7f04ac0d15f0 msgr2=0x7f04ac0d19d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:16.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.493+0000 7f04b332a700 1 -- 172.21.15.16:0/2584106001 shutdown_connections 2023-10-16T13:48:16.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.493+0000 7f04b332a700 1 -- 172.21.15.16:0/2584106001 wait complete. 2023-10-16T13:48:16.508 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument' 2023-10-16T13:48:16.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.632+0000 7fcb83214700 1 Processor -- start 2023-10-16T13:48:16.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.632+0000 7fcb83214700 1 -- start start 2023-10-16T13:48:16.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.633+0000 7fcb83214700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcb7c15a5c0 0x7fcb7c0cef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:16.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.633+0000 7fcb83214700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fcb7c0ce010 0x7fcb7c0d03d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:16.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.633+0000 7fcb83214700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb7c072a50 con 0x7fcb7c0d0910 2023-10-16T13:48:16.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.633+0000 7fcb83214700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb7c0cf440 con 0x7fcb7c15a5c0 2023-10-16T13:48:16.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.633+0000 7fcb83214700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb7c0d0da0 con 0x7fcb7c0ce010 2023-10-16T13:48:16.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.633+0000 7fcb80fb0700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcb7c15a5c0 0x7fcb7c0cef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:16.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.633+0000 7fcb80fb0700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcb7c15a5c0 0x7fcb7c0cef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57242/0 (socket says 172.21.15.16:57242) 2023-10-16T13:48:16.638 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.633+0000 7fcb80fb0700 1 -- 172.21.15.16:0/242514213 learned_addr learned my addr 172.21.15.16:0/242514213 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:16.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.633+0000 7fcb7bfff700 1 --2- 172.21.15.16:0/242514213 >> v2:172.21.15.49:3300/0 conn(0x7fcb7c0ce010 0x7fcb7c0d03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:16.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.633+0000 7fcb80fb0700 1 -- 172.21.15.16:0/242514213 >> v2:172.21.15.49:3300/0 conn(0x7fcb7c0ce010 msgr2=0x7fcb7c0d03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:16.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.633+0000 7fcb80fb0700 1 --2- 172.21.15.16:0/242514213 >> v2:172.21.15.49:3300/0 conn(0x7fcb7c0ce010 0x7fcb7c0d03d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:16.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.633+0000 7fcb80fb0700 1 -- 172.21.15.16:0/242514213 >> v1:172.21.15.49:6789/0 conn(0x7fcb7c0d0910 legacy=0x7fcb7c15e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:16.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.633+0000 7fcb80fb0700 1 -- 172.21.15.16:0/242514213 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb7c0cba30 con 0x7fcb7c15a5c0 2023-10-16T13:48:16.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.633+0000 7fcb7bfff700 1 --2- 172.21.15.16:0/242514213 >> v2:172.21.15.49:3300/0 conn(0x7fcb7c0ce010 0x7fcb7c0d03d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:48:16.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.633+0000 7fcb80fb0700 1 --2- 172.21.15.16:0/242514213 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb7c15a5c0 0x7fcb7c0cef00 secure :-1 s=READY pgs=2958 cs=0 l=1 rev1=1 crypto rx=0x7fcb7c067660 tx=0x7fcb70009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=892e8837f72a6072 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:16.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.633+0000 7fcb7b7fe700 1 -- 172.21.15.16:0/242514213 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb70004230 con 0x7fcb7c15a5c0 2023-10-16T13:48:16.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.633+0000 7fcb7b7fe700 1 -- 172.21.15.16:0/242514213 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb70004390 con 0x7fcb7c15a5c0 2023-10-16T13:48:16.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.633+0000 7fcb7b7fe700 1 -- 172.21.15.16:0/242514213 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb700036a0 con 0x7fcb7c15a5c0 2023-10-16T13:48:16.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.634+0000 7fcb83214700 1 -- 172.21.15.16:0/242514213 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb7c15a5c0 msgr2=0x7fcb7c0cef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:16.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.634+0000 7fcb83214700 1 --2- 172.21.15.16:0/242514213 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb7c15a5c0 0x7fcb7c0cef00 secure :-1 s=READY pgs=2958 cs=0 l=1 rev1=1 crypto rx=0x7fcb7c067660 tx=0x7fcb70009bd0 comp rx=0 tx=0).stop 2023-10-16T13:48:16.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.634+0000 7fcb83214700 1 -- 172.21.15.16:0/242514213 shutdown_connections 2023-10-16T13:48:16.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.634+0000 7fcb83214700 1 --2- 172.21.15.16:0/242514213 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb7c15a5c0 0x7fcb7c0cef00 unknown :-1 s=CLOSED pgs=2958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:16.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.634+0000 7fcb83214700 1 --2- 172.21.15.16:0/242514213 >> v2:172.21.15.49:3300/0 conn(0x7fcb7c0ce010 0x7fcb7c0d03d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:16.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.634+0000 7fcb83214700 1 -- 172.21.15.16:0/242514213 >> 172.21.15.16:0/242514213 conn(0x7fcb7c0b1fd0 msgr2=0x7fcb7c0b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:16.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.634+0000 7fcb83214700 1 -- 172.21.15.16:0/242514213 shutdown_connections 2023-10-16T13:48:16.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.634+0000 7fcb83214700 1 -- 172.21.15.16:0/242514213 wait complete. 2023-10-16T13:48:16.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.635+0000 7fcb83214700 1 Processor -- start 2023-10-16T13:48:16.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.635+0000 7fcb83214700 1 -- start start 2023-10-16T13:48:16.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.636+0000 7fcb83214700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb7c0ce010 0x7fcb7c0b14e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:16.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.636+0000 7fcb83214700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcb7c0d0910 0x7fcb7c0c9740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:16.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.636+0000 7fcb83214700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcb7c15a5c0 0x7fcb7c0c9c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:16.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.636+0000 7fcb83214700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb7c0c92b0 con 0x7fcb7c0ce010 2023-10-16T13:48:16.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.636+0000 7fcb83214700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb7c0c8980 con 0x7fcb7c0d0910 2023-10-16T13:48:16.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.636+0000 7fcb83214700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb7c0c8c60 con 0x7fcb7c15a5c0 2023-10-16T13:48:16.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.636+0000 7fcb80fb0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb7c0ce010 0x7fcb7c0b14e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:16.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.636+0000 7fcb80fb0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb7c0ce010 0x7fcb7c0b14e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57248/0 (socket says 172.21.15.16:57248) 2023-10-16T13:48:16.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.636+0000 7fcb80fb0700 1 -- 172.21.15.16:0/3161665892 learned_addr learned my addr 172.21.15.16:0/3161665892 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:16.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.636+0000 7fcb7bfff700 1 --2- 172.21.15.16:0/3161665892 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcb7c0d0910 0x7fcb7c0c9740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:16.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.636+0000 7fcb817b1700 1 --2- 172.21.15.16:0/3161665892 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcb7c15a5c0 0x7fcb7c0c9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:16.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.637+0000 7fcb80fb0700 1 -- 172.21.15.16:0/3161665892 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcb7c15a5c0 msgr2=0x7fcb7c0c9c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:16.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.637+0000 7fcb80fb0700 1 --2- 172.21.15.16:0/3161665892 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcb7c15a5c0 0x7fcb7c0c9c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:16.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.637+0000 7fcb80fb0700 1 -- 172.21.15.16:0/3161665892 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcb7c0d0910 msgr2=0x7fcb7c0c9740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:16.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.637+0000 7fcb80fb0700 1 --2- 172.21.15.16:0/3161665892 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcb7c0d0910 0x7fcb7c0c9740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:16.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.637+0000 7fcb80fb0700 1 -- 172.21.15.16:0/3161665892 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb70009550 con 0x7fcb7c0ce010 2023-10-16T13:48:16.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.637+0000 7fcb817b1700 1 --2- 172.21.15.16:0/3161665892 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcb7c15a5c0 0x7fcb7c0c9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:48:16.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.637+0000 7fcb80fb0700 1 --2- 172.21.15.16:0/3161665892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb7c0ce010 0x7fcb7c0b14e0 secure :-1 s=READY pgs=2959 cs=0 l=1 rev1=1 crypto rx=0x7fcb70009130 tx=0x7fcb70004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:16.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.637+0000 7fcb797fa700 1 -- 172.21.15.16:0/3161665892 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb70011630 con 0x7fcb7c0ce010 2023-10-16T13:48:16.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.637+0000 7fcb797fa700 1 -- 172.21.15.16:0/3161665892 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb70011790 con 0x7fcb7c0ce010 2023-10-16T13:48:16.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.637+0000 7fcb83214700 1 -- 172.21.15.16:0/3161665892 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb7c0c8f40 con 0x7fcb7c0ce010 2023-10-16T13:48:16.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.637+0000 7fcb797fa700 1 -- 172.21.15.16:0/3161665892 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb70011b40 con 0x7fcb7c0ce010 2023-10-16T13:48:16.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.637+0000 7fcb83214700 1 -- 172.21.15.16:0/3161665892 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb7c159c70 con 0x7fcb7c0ce010 2023-10-16T13:48:16.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.638+0000 7fcb83214700 1 -- 172.21.15.16:0/3161665892 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb7c0c9590 con 0x7fcb7c0ce010 2023-10-16T13:48:16.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.638+0000 7fcb797fa700 1 -- 172.21.15.16:0/3161665892 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fcb70011ca0 con 0x7fcb7c0ce010 2023-10-16T13:48:16.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.639+0000 7fcb797fa700 1 --2- 172.21.15.16:0/3161665892 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fcb6006b140 0x7fcb6006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:16.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.639+0000 7fcb7bfff700 1 --2- 172.21.15.16:0/3161665892 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fcb6006b140 0x7fcb6006d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:16.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.639+0000 7fcb797fa700 1 -- 172.21.15.16:0/3161665892 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 6337+0+0 (secure 0 0 0) 0x7fcb7008b260 con 0x7fcb7c0ce010 2023-10-16T13:48:16.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.639+0000 7fcb7bfff700 1 --2- 172.21.15.16:0/3161665892 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fcb6006b140 0x7fcb6006d600 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fcb68006fd0 tx=0x7fcb68008040 comp rx=0 tx=0).ready entity=mgr.12678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:16.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.644+0000 7fcb797fa700 1 -- 172.21.15.16:0/3161665892 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcb70055f40 con 0x7fcb7c0ce010 2023-10-16T13:48:16.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:16.931+0000 7fcb83214700 1 -- 172.21.15.16:0/3161665892 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument"]} v 0) v1 -- 0x7fcb7c04ebb0 con 0x7fcb7c0ce010 2023-10-16T13:48:17.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.380+0000 7fcb797fa700 1 -- 172.21.15.16:0/3161665892 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument"]}]=0 v3230) v1 ==== 187+0+0 (secure 0 0 0) 0x7fcb70059560 con 0x7fcb7c0ce010 2023-10-16T13:48:17.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.383+0000 7fcb83214700 1 -- 172.21.15.16:0/3161665892 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fcb6006b140 msgr2=0x7fcb6006d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:17.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.383+0000 7fcb83214700 1 --2- 172.21.15.16:0/3161665892 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fcb6006b140 0x7fcb6006d600 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fcb68006fd0 tx=0x7fcb68008040 comp rx=0 tx=0).stop 2023-10-16T13:48:17.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.383+0000 7fcb83214700 1 -- 172.21.15.16:0/3161665892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb7c0ce010 msgr2=0x7fcb7c0b14e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:17.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.383+0000 7fcb83214700 1 --2- 172.21.15.16:0/3161665892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb7c0ce010 0x7fcb7c0b14e0 secure :-1 s=READY pgs=2959 cs=0 l=1 rev1=1 crypto rx=0x7fcb70009130 tx=0x7fcb70004750 comp rx=0 tx=0).stop 2023-10-16T13:48:17.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.383+0000 7fcb83214700 1 -- 172.21.15.16:0/3161665892 shutdown_connections 2023-10-16T13:48:17.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.383+0000 7fcb83214700 1 --2- 172.21.15.16:0/3161665892 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fcb6006b140 0x7fcb6006d600 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:17.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.383+0000 7fcb83214700 1 --2- 172.21.15.16:0/3161665892 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcb7c15a5c0 0x7fcb7c0c9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:17.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.383+0000 7fcb83214700 1 --2- 172.21.15.16:0/3161665892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb7c0ce010 0x7fcb7c0b14e0 unknown :-1 s=CLOSED pgs=2959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:17.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.383+0000 7fcb83214700 1 --2- 172.21.15.16:0/3161665892 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcb7c0d0910 0x7fcb7c0c9740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:17.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.383+0000 7fcb83214700 1 -- 172.21.15.16:0/3161665892 >> 172.21.15.16:0/3161665892 conn(0x7fcb7c0b1fd0 msgr2=0x7fcb7c15e1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:17.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.383+0000 7fcb83214700 1 -- 172.21.15.16:0/3161665892 shutdown_connections 2023-10-16T13:48:17.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.383+0000 7fcb83214700 1 -- 172.21.15.16:0/3161665892 wait complete. 2023-10-16T13:48:17.400 INFO:tasks.cephfs_test_runner:Test command fails without --fs argument in presence of multiple fs ... ok 2023-10-16T13:48:17.401 INFO:tasks.cephfs_test_runner:Starting test: test_snap_dir_name (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir) Test the correctness of snap directory name 2023-10-16T13:48:17.401 INFO:tasks.cephfs_test_runner:test_snap_dir_name (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir) 2023-10-16T13:48:17.402 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:48:17.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.530+0000 7fc4ae3ae700 1 Processor -- start 2023-10-16T13:48:17.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.530+0000 7fc4ae3ae700 1 -- start start 2023-10-16T13:48:17.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.531+0000 7fc4ae3ae700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc4a815b280 0x7fc4a81613e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:17.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.531+0000 7fc4ae3ae700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fc4a80cc8a0 0x7fc4a815e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:17.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.531+0000 7fc4ae3ae700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4a8161c30 con 0x7fc4a806a7e0 2023-10-16T13:48:17.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.531+0000 7fc4ae3ae700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4a8161f10 con 0x7fc4a815b280 2023-10-16T13:48:17.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.531+0000 7fc4ae3ae700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4a81621f0 con 0x7fc4a80cc8a0 2023-10-16T13:48:17.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.531+0000 7fc4a7fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc4a815b280 0x7fc4a81613e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:17.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.531+0000 7fc4a7fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc4a815b280 0x7fc4a81613e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57256/0 (socket says 172.21.15.16:57256) 2023-10-16T13:48:17.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.531+0000 7fc4a7fff700 1 -- 172.21.15.16:0/3472655453 learned_addr learned my addr 172.21.15.16:0/3472655453 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:17.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.531+0000 7fc4a77fe700 1 --2- 172.21.15.16:0/3472655453 >> v2:172.21.15.49:3300/0 conn(0x7fc4a80cc8a0 0x7fc4a815e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:17.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.531+0000 7fc4a7fff700 1 -- 172.21.15.16:0/3472655453 >> v2:172.21.15.49:3300/0 conn(0x7fc4a80cc8a0 msgr2=0x7fc4a815e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:17.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.531+0000 7fc4a7fff700 1 --2- 172.21.15.16:0/3472655453 >> v2:172.21.15.49:3300/0 conn(0x7fc4a80cc8a0 0x7fc4a815e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:17.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.532+0000 7fc4a7fff700 1 -- 172.21.15.16:0/3472655453 >> v1:172.21.15.49:6789/0 conn(0x7fc4a806a7e0 legacy=0x7fc4a80d02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:17.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.532+0000 7fc4a7fff700 1 -- 172.21.15.16:0/3472655453 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4a815df00 con 0x7fc4a815b280 2023-10-16T13:48:17.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.532+0000 7fc4a77fe700 1 --2- 172.21.15.16:0/3472655453 >> v2:172.21.15.49:3300/0 conn(0x7fc4a80cc8a0 0x7fc4a815e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:48:17.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.532+0000 7fc4a7fff700 1 --2- 172.21.15.16:0/3472655453 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4a815b280 0x7fc4a81613e0 secure :-1 s=READY pgs=2960 cs=0 l=1 rev1=1 crypto rx=0x7fc4a8071970 tx=0x7fc49c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c98a084be428a930 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:17.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.532+0000 7fc4a6ffd700 1 -- 172.21.15.16:0/3472655453 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc49c004230 con 0x7fc4a815b280 2023-10-16T13:48:17.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.532+0000 7fc4a6ffd700 1 -- 172.21.15.16:0/3472655453 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc49c004390 con 0x7fc4a815b280 2023-10-16T13:48:17.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.532+0000 7fc4a6ffd700 1 -- 172.21.15.16:0/3472655453 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc49c0036a0 con 0x7fc4a815b280 2023-10-16T13:48:17.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.532+0000 7fc4ae3ae700 1 -- 172.21.15.16:0/3472655453 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4a815b280 msgr2=0x7fc4a81613e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:17.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.532+0000 7fc4ae3ae700 1 --2- 172.21.15.16:0/3472655453 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4a815b280 0x7fc4a81613e0 secure :-1 s=READY pgs=2960 cs=0 l=1 rev1=1 crypto rx=0x7fc4a8071970 tx=0x7fc49c009bd0 comp rx=0 tx=0).stop 2023-10-16T13:48:17.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.532+0000 7fc4ae3ae700 1 -- 172.21.15.16:0/3472655453 shutdown_connections 2023-10-16T13:48:17.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.532+0000 7fc4ae3ae700 1 --2- 172.21.15.16:0/3472655453 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4a815b280 0x7fc4a81613e0 unknown :-1 s=CLOSED pgs=2960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:17.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.532+0000 7fc4ae3ae700 1 --2- 172.21.15.16:0/3472655453 >> v2:172.21.15.49:3300/0 conn(0x7fc4a80cc8a0 0x7fc4a815e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:17.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.532+0000 7fc4ae3ae700 1 -- 172.21.15.16:0/3472655453 >> 172.21.15.16:0/3472655453 conn(0x7fc4a80b1fd0 msgr2=0x7fc4a80b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:17.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.532+0000 7fc4ae3ae700 1 -- 172.21.15.16:0/3472655453 shutdown_connections 2023-10-16T13:48:17.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.532+0000 7fc4ae3ae700 1 -- 172.21.15.16:0/3472655453 wait complete. 2023-10-16T13:48:17.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.534+0000 7fc4ae3ae700 1 Processor -- start 2023-10-16T13:48:17.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.534+0000 7fc4ae3ae700 1 -- start start 2023-10-16T13:48:17.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.535+0000 7fc4ae3ae700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4a806a7e0 0x7fc4a814f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:17.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.535+0000 7fc4ae3ae700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc4a80cc8a0 0x7fc4a806abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:17.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.535+0000 7fc4ae3ae700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc4a815b280 0x7fc4a806b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:17.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.535+0000 7fc4ae3ae700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a814fcc0 con 0x7fc4a806a7e0 2023-10-16T13:48:17.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.535+0000 7fc4ae3ae700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a80c5190 con 0x7fc4a80cc8a0 2023-10-16T13:48:17.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.535+0000 7fc4ae3ae700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a80c6480 con 0x7fc4a815b280 2023-10-16T13:48:17.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.535+0000 7fc4a7fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4a806a7e0 0x7fc4a814f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:17.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.535+0000 7fc4a7fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4a806a7e0 0x7fc4a814f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57262/0 (socket says 172.21.15.16:57262) 2023-10-16T13:48:17.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.535+0000 7fc4a7fff700 1 -- 172.21.15.16:0/316038328 learned_addr learned my addr 172.21.15.16:0/316038328 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:17.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.535+0000 7fc4a77fe700 1 --2- 172.21.15.16:0/316038328 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc4a80cc8a0 0x7fc4a806abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:17.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.535+0000 7fc4ac94b700 1 --2- 172.21.15.16:0/316038328 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc4a815b280 0x7fc4a806b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:17.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.535+0000 7fc4a7fff700 1 -- 172.21.15.16:0/316038328 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc4a815b280 msgr2=0x7fc4a806b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:17.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.535+0000 7fc4a7fff700 1 --2- 172.21.15.16:0/316038328 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc4a815b280 0x7fc4a806b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:17.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.535+0000 7fc4a7fff700 1 -- 172.21.15.16:0/316038328 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc4a80cc8a0 msgr2=0x7fc4a806abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:17.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.535+0000 7fc4a7fff700 1 --2- 172.21.15.16:0/316038328 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc4a80cc8a0 0x7fc4a806abc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:17.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.535+0000 7fc4a7fff700 1 -- 172.21.15.16:0/316038328 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc49c009550 con 0x7fc4a806a7e0 2023-10-16T13:48:17.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.535+0000 7fc4a77fe700 1 --2- 172.21.15.16:0/316038328 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc4a80cc8a0 0x7fc4a806abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:48:17.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.535+0000 7fc4ac94b700 1 --2- 172.21.15.16:0/316038328 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc4a815b280 0x7fc4a806b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:48:17.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.535+0000 7fc4a7fff700 1 --2- 172.21.15.16:0/316038328 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4a806a7e0 0x7fc4a814f780 secure :-1 s=READY pgs=2961 cs=0 l=1 rev1=1 crypto rx=0x7fc4a80d1540 tx=0x7fc49c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:17.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.535+0000 7fc4a4ff9700 1 -- 172.21.15.16:0/316038328 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc49c003d80 con 0x7fc4a806a7e0 2023-10-16T13:48:17.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.535+0000 7fc4a4ff9700 1 -- 172.21.15.16:0/316038328 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc49c01f070 con 0x7fc4a806a7e0 2023-10-16T13:48:17.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.535+0000 7fc4a4ff9700 1 -- 172.21.15.16:0/316038328 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc49c011870 con 0x7fc4a806a7e0 2023-10-16T13:48:17.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.535+0000 7fc4ae3ae700 1 -- 172.21.15.16:0/316038328 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4a80c53e0 con 0x7fc4a806a7e0 2023-10-16T13:48:17.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.535+0000 7fc4ae3ae700 1 -- 172.21.15.16:0/316038328 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4a80c59a0 con 0x7fc4a806a7e0 2023-10-16T13:48:17.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.536+0000 7fc4ae3ae700 1 -- 172.21.15.16:0/316038328 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4a804ebb0 con 0x7fc4a806a7e0 2023-10-16T13:48:17.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.537+0000 7fc4a4ff9700 1 -- 172.21.15.16:0/316038328 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fc49c021030 con 0x7fc4a806a7e0 2023-10-16T13:48:17.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.537+0000 7fc4a4ff9700 1 --2- 172.21.15.16:0/316038328 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fc49406b140 0x7fc49406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:17.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.537+0000 7fc4a4ff9700 1 -- 172.21.15.16:0/316038328 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 6337+0+0 (secure 0 0 0) 0x7fc49c08b2e0 con 0x7fc4a806a7e0 2023-10-16T13:48:17.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.537+0000 7fc4a77fe700 1 --2- 172.21.15.16:0/316038328 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fc49406b140 0x7fc49406d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:17.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.538+0000 7fc4a77fe700 1 --2- 172.21.15.16:0/316038328 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fc49406b140 0x7fc49406d600 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc4a8153660 tx=0x7fc498008040 comp rx=0 tx=0).ready entity=mgr.12678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:17.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.542+0000 7fc4a4ff9700 1 -- 172.21.15.16:0/316038328 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc49c055fb0 con 0x7fc4a806a7e0 2023-10-16T13:48:17.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.750+0000 7fc4ae3ae700 1 -- 172.21.15.16:0/316038328 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc4a80722e0 con 0x7fc4a806a7e0 2023-10-16T13:48:17.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.751+0000 7fc4a4ff9700 1 -- 172.21.15.16:0/316038328 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v257) v1 ==== 74+0+14673 (secure 0 0 0) 0x7fc49c0595d0 con 0x7fc4a806a7e0 2023-10-16T13:48:17.753 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:48:17.753 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":257,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:48:15.374349+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.49:6833/680748068":"2023-10-17T13:48:12.125392+0000","172.21.15.16:6835/303278795":"2023-10-17T13:48:05.033844+0000","172.21.15.16:6832/2221624769":"2023-10-17T13:48:02.485614+0000","172.21.15.16:0/2242986614":"2023-10-17T13:48:02.485614+0000","172.21.15.16:0/3610227254":"2023-10-17T13:48:02.485614+0000","172.21.15.16:0/1182888331":"2023-10-17T13:48:02.485614+0000","172.21.15.16:0/2095012141":"2023-10-17T13:48:02.485614+0000","172.21.15.16:0/874722077":"2023-10-17T13:48:02.485614+0000","172.21.15.16:0/2066401556":"2023-10-17T13:48:02.485614+0000","172.21.15.16:0/3947601914":"2023-10-17T13:47:54.816700+0000","172.21.15.16:6833/2221624769":"2023-10-17T13:48:02.485614+0000","172.21.15.16:0/2759470534":"2023-10-17T13:47:54.816700+0000","172.21.15.16:0/2634105683":"2023-10-17T13:48:02.485614+0000","172.21.15.16:0/2974892940":"2023-10-17T13:47:54.816700+0000","172.21.15.16:6833/1736104576":"2023-10-17T13:47:54.816700+0000","172.21.15.16:6832/1736104576":"2023-10-17T13:47:54.816700+0000","172.21.15.16:0/1246409088":"2023-10-17T13:48:02.485614+0000","172.21.15.16:0/318862437":"2023-10-17T13:47:54.816700+0000","172.21.15.16:0/413581668":"2023-10-17T13:47:54.816700+0000","172.21.15.49:6835/680748068":"2023-10-17T13:48:12.125392+0000","172.21.15.16:6837/303278795":"2023-10-17T13:48:05.033844+0000","172.21.15.16:0/3740515383":"2023-10-17T13:47:54.816700+0000","172.21.15.16:0/3105924907":"2023-10-17T13:47:54.816700+0000","172.21.15.16:0/3617558038":"2023-10-17T13:47:54.816700+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:48:17.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.752+0000 7fc4ae3ae700 1 -- 172.21.15.16:0/316038328 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fc49406b140 msgr2=0x7fc49406d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:17.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.752+0000 7fc4ae3ae700 1 --2- 172.21.15.16:0/316038328 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fc49406b140 0x7fc49406d600 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc4a8153660 tx=0x7fc498008040 comp rx=0 tx=0).stop 2023-10-16T13:48:17.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.753+0000 7fc4ae3ae700 1 -- 172.21.15.16:0/316038328 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4a806a7e0 msgr2=0x7fc4a814f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:17.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.753+0000 7fc4ae3ae700 1 --2- 172.21.15.16:0/316038328 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4a806a7e0 0x7fc4a814f780 secure :-1 s=READY pgs=2961 cs=0 l=1 rev1=1 crypto rx=0x7fc4a80d1540 tx=0x7fc49c004750 comp rx=0 tx=0).stop 2023-10-16T13:48:17.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.753+0000 7fc4ae3ae700 1 -- 172.21.15.16:0/316038328 shutdown_connections 2023-10-16T13:48:17.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.753+0000 7fc4ae3ae700 1 --2- 172.21.15.16:0/316038328 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fc49406b140 0x7fc49406d600 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:17.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.753+0000 7fc4ae3ae700 1 --2- 172.21.15.16:0/316038328 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc4a815b280 0x7fc4a806b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:17.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.753+0000 7fc4ae3ae700 1 --2- 172.21.15.16:0/316038328 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4a806a7e0 0x7fc4a814f780 unknown :-1 s=CLOSED pgs=2961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:17.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.753+0000 7fc4ae3ae700 1 --2- 172.21.15.16:0/316038328 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc4a80cc8a0 0x7fc4a806abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:17.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.753+0000 7fc4ae3ae700 1 -- 172.21.15.16:0/316038328 >> 172.21.15.16:0/316038328 conn(0x7fc4a80b1fd0 msgr2=0x7fc4a80cac20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:17.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.753+0000 7fc4ae3ae700 1 -- 172.21.15.16:0/316038328 shutdown_connections 2023-10-16T13:48:17.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.753+0000 7fc4ae3ae700 1 -- 172.21.15.16:0/316038328 wait complete. 2023-10-16T13:48:17.767 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T13:48:17.768 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T13:48:17.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.894+0000 7f488c688700 1 Processor -- start 2023-10-16T13:48:17.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.894+0000 7f488c688700 1 -- start start 2023-10-16T13:48:17.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.895+0000 7f488c688700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f48840d2110 0x7f48840ae1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:17.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.895+0000 7f488c688700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f488406ca10 0x7f48840ae720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:17.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.895+0000 7f488c688700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48840b20f0 con 0x7f48840ce6a0 2023-10-16T13:48:17.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.895+0000 7f488c688700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48840d03d0 con 0x7f48840d2110 2023-10-16T13:48:17.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.895+0000 7f488c688700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48840b2340 con 0x7f488406ca10 2023-10-16T13:48:17.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.895+0000 7f4889c23700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f48840d2110 0x7f48840ae1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:17.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.895+0000 7f488a424700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f48840ce6a0 0x7f488406c300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46544/0 (socket says 172.21.15.16:46544) 2023-10-16T13:48:17.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.895+0000 7f488a424700 1 -- 172.21.15.16:0/3947541108 learned_addr learned my addr 172.21.15.16:0/3947541108 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:48:17.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.896+0000 7f488ac25700 1 --2- 172.21.15.16:0/3947541108 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f488406ca10 0x7f48840ae720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:17.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.896+0000 7f4889c23700 1 -- 172.21.15.16:0/3947541108 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f488406ca10 msgr2=0x7f48840ae720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:17.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.896+0000 7f4889c23700 1 --2- 172.21.15.16:0/3947541108 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f488406ca10 0x7f48840ae720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:17.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.896+0000 7f4889c23700 1 -- 172.21.15.16:0/3947541108 >> v1:172.21.15.16:6789/0 conn(0x7f48840ce6a0 legacy=0x7f488406c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:17.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.896+0000 7f4889c23700 1 -- 172.21.15.16:0/3947541108 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f488406bb00 con 0x7f48840d2110 2023-10-16T13:48:17.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.896+0000 7f4889422700 1 -- 172.21.15.16:0/3947541108 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 786951488 0 0) 0x7f48840b20f0 con 0x7f48840ce6a0 2023-10-16T13:48:17.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.896+0000 7f4889c23700 1 --2- 172.21.15.16:0/3947541108 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48840d2110 0x7f48840ae1e0 secure :-1 s=READY pgs=2963 cs=0 l=1 rev1=1 crypto rx=0x7f4880009840 tx=0x7f4880009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1021dd997e2fed12 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:17.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.896+0000 7f4889422700 1 -- 172.21.15.16:0/3947541108 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4880011830 con 0x7f48840d2110 2023-10-16T13:48:17.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.896+0000 7f4889422700 1 -- 172.21.15.16:0/3947541108 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4880011990 con 0x7f48840d2110 2023-10-16T13:48:17.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.896+0000 7f4889422700 1 -- 172.21.15.16:0/3947541108 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4880011d20 con 0x7f48840d2110 2023-10-16T13:48:17.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.897+0000 7f488c688700 1 -- 172.21.15.16:0/3947541108 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48840d2110 msgr2=0x7f48840ae1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:17.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.897+0000 7f488c688700 1 --2- 172.21.15.16:0/3947541108 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48840d2110 0x7f48840ae1e0 secure :-1 s=READY pgs=2963 cs=0 l=1 rev1=1 crypto rx=0x7f4880009840 tx=0x7f4880009b50 comp rx=0 tx=0).stop 2023-10-16T13:48:17.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.897+0000 7f488c688700 1 -- 172.21.15.16:0/3947541108 shutdown_connections 2023-10-16T13:48:17.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.897+0000 7f488c688700 1 --2- 172.21.15.16:0/3947541108 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f488406ca10 0x7f48840ae720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:17.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.897+0000 7f488c688700 1 --2- 172.21.15.16:0/3947541108 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48840d2110 0x7f48840ae1e0 unknown :-1 s=CLOSED pgs=2963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:17.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.897+0000 7f488c688700 1 -- 172.21.15.16:0/3947541108 >> 172.21.15.16:0/3947541108 conn(0x7f48840d15f0 msgr2=0x7f48840d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:17.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.897+0000 7f488c688700 1 -- 172.21.15.16:0/3947541108 shutdown_connections 2023-10-16T13:48:17.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.897+0000 7f488c688700 1 -- 172.21.15.16:0/3947541108 wait complete. 2023-10-16T13:48:17.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.898+0000 7f488c688700 1 Processor -- start 2023-10-16T13:48:17.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.898+0000 7f488c688700 1 -- start start 2023-10-16T13:48:17.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.899+0000 7f488c688700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f488406ca10 0x7f488415d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:17.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.899+0000 7f488c688700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f48840ce6a0 0x7f48840ae450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:17.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.899+0000 7f488c688700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48840d2110 0x7f48840c9e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:17.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.899+0000 7f488c688700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48840b4e70 con 0x7f48840d2110 2023-10-16T13:48:17.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.899+0000 7f488c688700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48840b42a0 con 0x7f488406ca10 2023-10-16T13:48:17.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.899+0000 7f488c688700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48840b4540 con 0x7f48840ce6a0 2023-10-16T13:48:17.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.899+0000 7f488ac25700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48840d2110 0x7f48840c9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:17.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.899+0000 7f488ac25700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48840d2110 0x7f48840c9e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57280/0 (socket says 172.21.15.16:57280) 2023-10-16T13:48:17.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.899+0000 7f488ac25700 1 -- 172.21.15.16:0/1072112327 learned_addr learned my addr 172.21.15.16:0/1072112327 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:17.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.899+0000 7f4889c23700 1 --2- 172.21.15.16:0/1072112327 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f48840ce6a0 0x7f48840ae450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:17.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.899+0000 7f488a424700 1 --2- 172.21.15.16:0/1072112327 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f488406ca10 0x7f488415d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:17.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.899+0000 7f488ac25700 1 -- 172.21.15.16:0/1072112327 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f48840ce6a0 msgr2=0x7f48840ae450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:17.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.899+0000 7f488ac25700 1 --2- 172.21.15.16:0/1072112327 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f48840ce6a0 0x7f48840ae450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:17.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.899+0000 7f488ac25700 1 -- 172.21.15.16:0/1072112327 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f488406ca10 msgr2=0x7f488415d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:17.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.899+0000 7f488ac25700 1 --2- 172.21.15.16:0/1072112327 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f488406ca10 0x7f488415d790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:17.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.899+0000 7f488ac25700 1 -- 172.21.15.16:0/1072112327 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48800094a0 con 0x7f48840d2110 2023-10-16T13:48:17.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.899+0000 7f488a424700 1 --2- 172.21.15.16:0/1072112327 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f488406ca10 0x7f488415d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:48:17.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.899+0000 7f488ac25700 1 --2- 172.21.15.16:0/1072112327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48840d2110 0x7f48840c9e10 secure :-1 s=READY pgs=2964 cs=0 l=1 rev1=1 crypto rx=0x7f487c007e40 tx=0x7f487c00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:17.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.899+0000 7f487affd700 1 -- 172.21.15.16:0/1072112327 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f487c00ede0 con 0x7f48840d2110 2023-10-16T13:48:17.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.899+0000 7f487affd700 1 -- 172.21.15.16:0/1072112327 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f487c014420 con 0x7f48840d2110 2023-10-16T13:48:17.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.899+0000 7f488c688700 1 -- 172.21.15.16:0/1072112327 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48840b4820 con 0x7f48840d2110 2023-10-16T13:48:17.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.899+0000 7f487affd700 1 -- 172.21.15.16:0/1072112327 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f487c0147d0 con 0x7f48840d2110 2023-10-16T13:48:17.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.900+0000 7f488c688700 1 -- 172.21.15.16:0/1072112327 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f488415dcd0 con 0x7f48840d2110 2023-10-16T13:48:17.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.900+0000 7f488c688700 1 -- 172.21.15.16:0/1072112327 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f488415de60 con 0x7f48840d2110 2023-10-16T13:48:17.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.901+0000 7f487affd700 1 -- 172.21.15.16:0/1072112327 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f487c014930 con 0x7f48840d2110 2023-10-16T13:48:17.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.901+0000 7f487affd700 1 --2- 172.21.15.16:0/1072112327 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f486806b140 0x7f486806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:17.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.902+0000 7f487affd700 1 -- 172.21.15.16:0/1072112327 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 6337+0+0 (secure 0 0 0) 0x7f487c08a260 con 0x7f48840d2110 2023-10-16T13:48:17.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.902+0000 7f488a424700 1 --2- 172.21.15.16:0/1072112327 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f486806b140 0x7f486806d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:17.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.902+0000 7f488a424700 1 --2- 172.21.15.16:0/1072112327 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f486806b140 0x7f486806d600 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4874000f60 tx=0x7f4874007ca0 comp rx=0 tx=0).ready entity=mgr.12678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:17.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:17.906+0000 7f487affd700 1 -- 172.21.15.16:0/1072112327 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f488415de60 con 0x7f48840d2110 2023-10-16T13:48:18.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.106+0000 7f488c688700 1 -- 172.21.15.16:0/1072112327 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f488415de60 con 0x7f48840d2110 2023-10-16T13:48:18.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.107+0000 7f487affd700 1 -- 172.21.15.16:0/1072112327 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v257) v1 ==== 93+0+10 (secure 0 0 0) 0x7f488415de60 con 0x7f48840d2110 2023-10-16T13:48:18.110 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T13:48:18.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.109+0000 7f488c688700 1 -- 172.21.15.16:0/1072112327 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f486806b140 msgr2=0x7f486806d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:18.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.109+0000 7f488c688700 1 --2- 172.21.15.16:0/1072112327 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f486806b140 0x7f486806d600 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4874000f60 tx=0x7f4874007ca0 comp rx=0 tx=0).stop 2023-10-16T13:48:18.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.109+0000 7f488c688700 1 -- 172.21.15.16:0/1072112327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48840d2110 msgr2=0x7f48840c9e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:18.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.109+0000 7f488c688700 1 --2- 172.21.15.16:0/1072112327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48840d2110 0x7f48840c9e10 secure :-1 s=READY pgs=2964 cs=0 l=1 rev1=1 crypto rx=0x7f487c007e40 tx=0x7f487c00e3c0 comp rx=0 tx=0).stop 2023-10-16T13:48:18.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.109+0000 7f488c688700 1 -- 172.21.15.16:0/1072112327 shutdown_connections 2023-10-16T13:48:18.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.109+0000 7f488c688700 1 --2- 172.21.15.16:0/1072112327 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f486806b140 0x7f486806d600 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:18.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.109+0000 7f488c688700 1 --2- 172.21.15.16:0/1072112327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48840d2110 0x7f48840c9e10 unknown :-1 s=CLOSED pgs=2964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:18.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.109+0000 7f488c688700 1 --2- 172.21.15.16:0/1072112327 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f488406ca10 0x7f488415d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:18.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.109+0000 7f488c688700 1 --2- 172.21.15.16:0/1072112327 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f48840ce6a0 0x7f48840ae450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:18.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.109+0000 7f488c688700 1 -- 172.21.15.16:0/1072112327 >> 172.21.15.16:0/1072112327 conn(0x7f48840d15f0 msgr2=0x7f48840cf880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:18.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.109+0000 7f488c688700 1 -- 172.21.15.16:0/1072112327 shutdown_connections 2023-10-16T13:48:18.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.109+0000 7f488c688700 1 -- 172.21.15.16:0/1072112327 wait complete. 2023-10-16T13:48:18.124 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name' 2023-10-16T13:48:18.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.246+0000 7fd8262b3700 1 Processor -- start 2023-10-16T13:48:18.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.246+0000 7fd8262b3700 1 -- start start 2023-10-16T13:48:18.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.247+0000 7fd8262b3700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd82015b890 0x7fd8200b2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:18.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.247+0000 7fd8262b3700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fd82006da60 0x7fd82006dee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:18.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.247+0000 7fd8262b3700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8200cdc00 con 0x7fd8200ce9c0 2023-10-16T13:48:18.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.247+0000 7fd8262b3700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8200cf560 con 0x7fd82006da60 2023-10-16T13:48:18.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.247+0000 7fd8262b3700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd82006c1d0 con 0x7fd82015b890 2023-10-16T13:48:18.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.247+0000 7fd824850700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd8200ce9c0 0x7fd8200cee50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46562/0 (socket says 172.21.15.16:46562) 2023-10-16T13:48:18.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.247+0000 7fd824850700 1 -- 172.21.15.16:0/2117939434 learned_addr learned my addr 172.21.15.16:0/2117939434 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:48:18.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.247+0000 7fd81f7fe700 1 --2- 172.21.15.16:0/2117939434 >> v2:172.21.15.49:3300/0 conn(0x7fd82006da60 0x7fd82006dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:18.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.247+0000 7fd81ffff700 1 --2- 172.21.15.16:0/2117939434 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd82015b890 0x7fd8200b2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:18.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.248+0000 7fd81effd700 1 -- 172.21.15.16:0/2117939434 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2296348028 0 0) 0x7fd8200cdc00 con 0x7fd8200ce9c0 2023-10-16T13:48:18.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.248+0000 7fd81effd700 1 -- 172.21.15.16:0/2117939434 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8080035b0 con 0x7fd8200ce9c0 2023-10-16T13:48:18.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.248+0000 7fd81ffff700 1 -- 172.21.15.16:0/2117939434 >> v2:172.21.15.49:3300/0 conn(0x7fd82006da60 msgr2=0x7fd82006dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:18.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.248+0000 7fd81ffff700 1 --2- 172.21.15.16:0/2117939434 >> v2:172.21.15.49:3300/0 conn(0x7fd82006da60 0x7fd82006dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:18.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.248+0000 7fd81ffff700 1 -- 172.21.15.16:0/2117939434 >> v1:172.21.15.16:6789/0 conn(0x7fd8200ce9c0 legacy=0x7fd8200cee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:18.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.248+0000 7fd81effd700 1 -- 172.21.15.16:0/2117939434 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3360649156 0 0) 0x7fd8080035b0 con 0x7fd8200ce9c0 2023-10-16T13:48:18.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.248+0000 7fd81ffff700 1 -- 172.21.15.16:0/2117939434 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8200cdee0 con 0x7fd82015b890 2023-10-16T13:48:18.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.248+0000 7fd81f7fe700 1 --2- 172.21.15.16:0/2117939434 >> v2:172.21.15.49:3300/0 conn(0x7fd82006da60 0x7fd82006dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:48:18.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.248+0000 7fd81ffff700 1 --2- 172.21.15.16:0/2117939434 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd82015b890 0x7fd8200b2a10 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fd810009840 tx=0x7fd810009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=831b1b2cc4bff263 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:18.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.248+0000 7fd81effd700 1 -- 172.21.15.16:0/2117939434 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8180019f0 con 0x7fd82015b890 2023-10-16T13:48:18.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.248+0000 7fd81effd700 1 -- 172.21.15.16:0/2117939434 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd810004030 con 0x7fd82015b890 2023-10-16T13:48:18.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.248+0000 7fd81effd700 1 -- 172.21.15.16:0/2117939434 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8100043a0 con 0x7fd82015b890 2023-10-16T13:48:18.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.249+0000 7fd8262b3700 1 -- 172.21.15.16:0/2117939434 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd82015b890 msgr2=0x7fd8200b2a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:18.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.249+0000 7fd8262b3700 1 --2- 172.21.15.16:0/2117939434 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd82015b890 0x7fd8200b2a10 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fd810009840 tx=0x7fd810009b50 comp rx=0 tx=0).stop 2023-10-16T13:48:18.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.249+0000 7fd8262b3700 1 -- 172.21.15.16:0/2117939434 shutdown_connections 2023-10-16T13:48:18.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.249+0000 7fd8262b3700 1 --2- 172.21.15.16:0/2117939434 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd82015b890 0x7fd8200b2a10 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:18.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.249+0000 7fd8262b3700 1 --2- 172.21.15.16:0/2117939434 >> v2:172.21.15.49:3300/0 conn(0x7fd82006da60 0x7fd82006dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:18.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.249+0000 7fd8262b3700 1 -- 172.21.15.16:0/2117939434 >> 172.21.15.16:0/2117939434 conn(0x7fd8200cc5e0 msgr2=0x7fd8200cc9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:18.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.249+0000 7fd8262b3700 1 -- 172.21.15.16:0/2117939434 shutdown_connections 2023-10-16T13:48:18.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.249+0000 7fd8262b3700 1 -- 172.21.15.16:0/2117939434 wait complete. 2023-10-16T13:48:18.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.250+0000 7fd8262b3700 1 Processor -- start 2023-10-16T13:48:18.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.250+0000 7fd8262b3700 1 -- start start 2023-10-16T13:48:18.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.251+0000 7fd8262b3700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd82006da60 0x7fd82014c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:18.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.251+0000 7fd8262b3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8200ce9c0 0x7fd8200c8fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:18.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.251+0000 7fd8262b3700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd82015b890 0x7fd8200b07a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:18.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.251+0000 7fd8262b3700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd82015ce90 con 0x7fd8200ce9c0 2023-10-16T13:48:18.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.251+0000 7fd8262b3700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8200c9520 con 0x7fd82015b890 2023-10-16T13:48:18.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.251+0000 7fd8262b3700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd82015d0e0 con 0x7fd82006da60 2023-10-16T13:48:18.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.251+0000 7fd81f7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8200ce9c0 0x7fd8200c8fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:18.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.251+0000 7fd81f7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8200ce9c0 0x7fd8200c8fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57292/0 (socket says 172.21.15.16:57292) 2023-10-16T13:48:18.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.251+0000 7fd81f7fe700 1 -- 172.21.15.16:0/178362442 learned_addr learned my addr 172.21.15.16:0/178362442 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:18.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.251+0000 7fd81ffff700 1 --2- 172.21.15.16:0/178362442 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd82006da60 0x7fd82014c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:18.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.251+0000 7fd824850700 1 --2- 172.21.15.16:0/178362442 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd82015b890 0x7fd8200b07a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:18.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.251+0000 7fd81f7fe700 1 -- 172.21.15.16:0/178362442 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd82006da60 msgr2=0x7fd82014c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:18.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.251+0000 7fd81f7fe700 1 --2- 172.21.15.16:0/178362442 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd82006da60 0x7fd82014c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:18.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.251+0000 7fd81f7fe700 1 -- 172.21.15.16:0/178362442 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd82015b890 msgr2=0x7fd8200b07a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:18.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.251+0000 7fd81f7fe700 1 --2- 172.21.15.16:0/178362442 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd82015b890 0x7fd8200b07a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:18.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.251+0000 7fd81f7fe700 1 -- 172.21.15.16:0/178362442 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8140094a0 con 0x7fd8200ce9c0 2023-10-16T13:48:18.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.251+0000 7fd824850700 1 --2- 172.21.15.16:0/178362442 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd82015b890 0x7fd8200b07a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:48:18.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.252+0000 7fd81f7fe700 1 --2- 172.21.15.16:0/178362442 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8200ce9c0 0x7fd8200c8fe0 secure :-1 s=READY pgs=2966 cs=0 l=1 rev1=1 crypto rx=0x7fd81400ebe0 tx=0x7fd814009e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:18.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.252+0000 7fd81cff9700 1 -- 172.21.15.16:0/178362442 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd81400c8b0 con 0x7fd8200ce9c0 2023-10-16T13:48:18.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.252+0000 7fd81cff9700 1 -- 172.21.15.16:0/178362442 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd81400ca10 con 0x7fd8200ce9c0 2023-10-16T13:48:18.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.252+0000 7fd8262b3700 1 -- 172.21.15.16:0/178362442 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8100094a0 con 0x7fd8200ce9c0 2023-10-16T13:48:18.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.252+0000 7fd81cff9700 1 -- 172.21.15.16:0/178362442 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd81400cda0 con 0x7fd8200ce9c0 2023-10-16T13:48:18.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.252+0000 7fd8262b3700 1 -- 172.21.15.16:0/178362442 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd82015d6a0 con 0x7fd8200ce9c0 2023-10-16T13:48:18.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.253+0000 7fd8262b3700 1 -- 172.21.15.16:0/178362442 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd82015dfd0 con 0x7fd8200ce9c0 2023-10-16T13:48:18.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.254+0000 7fd81cff9700 1 -- 172.21.15.16:0/178362442 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fd814013470 con 0x7fd8200ce9c0 2023-10-16T13:48:18.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.254+0000 7fd81cff9700 1 --2- 172.21.15.16:0/178362442 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fd80c06b140 0x7fd80c06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:18.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.254+0000 7fd81cff9700 1 -- 172.21.15.16:0/178362442 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 6337+0+0 (secure 0 0 0) 0x7fd814089ea0 con 0x7fd8200ce9c0 2023-10-16T13:48:18.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.254+0000 7fd81ffff700 1 --2- 172.21.15.16:0/178362442 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fd80c06b140 0x7fd80c06d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:18.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.254+0000 7fd81ffff700 1 --2- 172.21.15.16:0/178362442 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fd80c06b140 0x7fd80c06d600 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd8100090b0 tx=0x7fd810008030 comp rx=0 tx=0).ready entity=mgr.12678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:18.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.258+0000 7fd81cff9700 1 -- 172.21.15.16:0/178362442 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd8140548d0 con 0x7fd8200ce9c0 2023-10-16T13:48:18.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:18.547+0000 7fd8262b3700 1 -- 172.21.15.16:0/178362442 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]} v 0) v1 -- 0x7fd82004ec20 con 0x7fd8200ce9c0 2023-10-16T13:48:19.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.386+0000 7fd81cff9700 1 -- 172.21.15.16:0/178362442 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]}]=0 v3232) v1 ==== 158+0+0 (secure 0 0 0) 0x7fd8140170e0 con 0x7fd8200ce9c0 2023-10-16T13:48:19.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.389+0000 7fd8262b3700 1 -- 172.21.15.16:0/178362442 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fd80c06b140 msgr2=0x7fd80c06d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:19.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.389+0000 7fd8262b3700 1 --2- 172.21.15.16:0/178362442 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fd80c06b140 0x7fd80c06d600 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd8100090b0 tx=0x7fd810008030 comp rx=0 tx=0).stop 2023-10-16T13:48:19.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.389+0000 7fd8262b3700 1 -- 172.21.15.16:0/178362442 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8200ce9c0 msgr2=0x7fd8200c8fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:19.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.389+0000 7fd8262b3700 1 --2- 172.21.15.16:0/178362442 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8200ce9c0 0x7fd8200c8fe0 secure :-1 s=READY pgs=2966 cs=0 l=1 rev1=1 crypto rx=0x7fd81400ebe0 tx=0x7fd814009e20 comp rx=0 tx=0).stop 2023-10-16T13:48:19.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.389+0000 7fd8262b3700 1 -- 172.21.15.16:0/178362442 shutdown_connections 2023-10-16T13:48:19.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.389+0000 7fd8262b3700 1 --2- 172.21.15.16:0/178362442 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fd80c06b140 0x7fd80c06d600 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:19.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.389+0000 7fd8262b3700 1 --2- 172.21.15.16:0/178362442 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd82015b890 0x7fd8200b07a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:19.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.389+0000 7fd8262b3700 1 --2- 172.21.15.16:0/178362442 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd82006da60 0x7fd82014c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:19.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.389+0000 7fd8262b3700 1 --2- 172.21.15.16:0/178362442 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8200ce9c0 0x7fd8200c8fe0 unknown :-1 s=CLOSED pgs=2966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:19.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.389+0000 7fd8262b3700 1 -- 172.21.15.16:0/178362442 >> 172.21.15.16:0/178362442 conn(0x7fd8200cc5e0 msgr2=0x7fd820157280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:19.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.389+0000 7fd8262b3700 1 -- 172.21.15.16:0/178362442 shutdown_connections 2023-10-16T13:48:19.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.389+0000 7fd8262b3700 1 -- 172.21.15.16:0/178362442 wait complete. 2023-10-16T13:48:19.407 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-16T13:48:19.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.540+0000 7fcd89ab6700 1 Processor -- start 2023-10-16T13:48:19.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.540+0000 7fcd89ab6700 1 -- start start 2023-10-16T13:48:19.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.540+0000 7fcd89ab6700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fcd840cb150 0x7fcd84071a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:19.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.540+0000 7fcd89ab6700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd8415c040 con 0x7fcd8415bb40 2023-10-16T13:48:19.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.540+0000 7fcd89ab6700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd8415c320 con 0x7fcd84072b30 2023-10-16T13:48:19.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.540+0000 7fcd89ab6700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd8406ea20 con 0x7fcd840cb150 2023-10-16T13:48:19.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.541+0000 7fcd82ffd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fcd8415bb40 0x7fcd840caa40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46572/0 (socket says 172.21.15.16:46572) 2023-10-16T13:48:19.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.541+0000 7fcd82ffd700 1 -- 172.21.15.16:0/4071800151 learned_addr learned my addr 172.21.15.16:0/4071800151 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:48:19.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.541+0000 7fcd83fff700 1 --2- 172.21.15.16:0/4071800151 >> v2:172.21.15.49:3300/0 conn(0x7fcd840cb150 0x7fcd84071a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:19.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.541+0000 7fcd827fc700 1 -- 172.21.15.16:0/4071800151 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2063838449 0 0) 0x7fcd8415c040 con 0x7fcd8415bb40 2023-10-16T13:48:19.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.541+0000 7fcd827fc700 1 -- 172.21.15.16:0/4071800151 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd6c0035b0 con 0x7fcd8415bb40 2023-10-16T13:48:19.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.541+0000 7fcd827fc700 1 -- 172.21.15.16:0/4071800151 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3130839256 0 0) 0x7fcd6c0035b0 con 0x7fcd8415bb40 2023-10-16T13:48:19.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.541+0000 7fcd827fc700 1 -- 172.21.15.16:0/4071800151 >> v2:172.21.15.49:3300/0 conn(0x7fcd840cb150 msgr2=0x7fcd84071a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:19.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.541+0000 7fcd827fc700 1 --2- 172.21.15.16:0/4071800151 >> v2:172.21.15.49:3300/0 conn(0x7fcd840cb150 0x7fcd84071a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:19.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.541+0000 7fcd827fc700 1 -- 172.21.15.16:0/4071800151 >> v1:172.21.15.49:6789/0 conn(0x7fcd84072b30 legacy=0x7fcd8406e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:19.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.541+0000 7fcd827fc700 1 -- 172.21.15.16:0/4071800151 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd8415c600 con 0x7fcd8415bb40 2023-10-16T13:48:19.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.541+0000 7fcd827fc700 1 -- 172.21.15.16:0/4071800151 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7fcd78001670 con 0x7fcd8415bb40 2023-10-16T13:48:19.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.541+0000 7fcd827fc700 1 -- 172.21.15.16:0/4071800151 >> v1:172.21.15.16:6789/0 conn(0x7fcd8415bb40 legacy=0x7fcd840caa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:19.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.541+0000 7fcd827fc700 1 --2- 172.21.15.16:0/4071800151 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd6c003a00 0x7fcd6c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:19.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.541+0000 7fcd827fc700 1 --2- 172.21.15.16:0/4071800151 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcd6c004850 0x7fcd6c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:19.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.541+0000 7fcd827fc700 1 --2- 172.21.15.16:0/4071800151 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcd6c005210 0x7fcd6c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:19.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.541+0000 7fcd827fc700 1 -- 172.21.15.16:0/4071800151 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd8406ea20 con 0x7fcd6c003a00 2023-10-16T13:48:19.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.541+0000 7fcd827fc700 1 -- 172.21.15.16:0/4071800151 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd6c00c0b0 con 0x7fcd6c005210 2023-10-16T13:48:19.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.541+0000 7fcd827fc700 1 -- 172.21.15.16:0/4071800151 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd6c00c270 con 0x7fcd6c004850 2023-10-16T13:48:19.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.541+0000 7fcd837fe700 1 --2- 172.21.15.16:0/4071800151 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd6c003a00 0x7fcd6c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:19.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.541+0000 7fcd83fff700 1 --2- 172.21.15.16:0/4071800151 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcd6c005210 0x7fcd6c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:19.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.541+0000 7fcd82ffd700 1 --2- 172.21.15.16:0/4071800151 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcd6c004850 0x7fcd6c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:19.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.542+0000 7fcd837fe700 1 -- 172.21.15.16:0/4071800151 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcd6c004850 msgr2=0x7fcd6c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:19.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.542+0000 7fcd837fe700 1 --2- 172.21.15.16:0/4071800151 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcd6c004850 0x7fcd6c004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:19.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.542+0000 7fcd837fe700 1 -- 172.21.15.16:0/4071800151 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcd6c005210 msgr2=0x7fcd6c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:19.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.542+0000 7fcd837fe700 1 --2- 172.21.15.16:0/4071800151 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcd6c005210 0x7fcd6c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:19.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.542+0000 7fcd83fff700 1 -- 172.21.15.16:0/4071800151 reap_dead start 2023-10-16T13:48:19.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.542+0000 7fcd837fe700 1 -- 172.21.15.16:0/4071800151 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd6c0035b0 con 0x7fcd6c003a00 2023-10-16T13:48:19.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.542+0000 7fcd82ffd700 1 --2- 172.21.15.16:0/4071800151 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcd6c004850 0x7fcd6c004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:48:19.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.542+0000 7fcd837fe700 1 --2- 172.21.15.16:0/4071800151 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd6c003a00 0x7fcd6c003e20 secure :-1 s=READY pgs=2968 cs=0 l=1 rev1=1 crypto rx=0x7fcd740017c0 tx=0x7fcd6c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5ff4fe501b2b331 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:19.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.542+0000 7fcd827fc700 1 -- 172.21.15.16:0/4071800151 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd740052d0 con 0x7fcd6c003a00 2023-10-16T13:48:19.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.542+0000 7fcd827fc700 1 -- 172.21.15.16:0/4071800151 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd74005450 con 0x7fcd6c003a00 2023-10-16T13:48:19.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.542+0000 7fcd827fc700 1 -- 172.21.15.16:0/4071800151 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd740038c0 con 0x7fcd6c003a00 2023-10-16T13:48:19.553 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.542+0000 7fcd89ab6700 1 -- 172.21.15.16:0/4071800151 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd6c003a00 msgr2=0x7fcd6c003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:19.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.542+0000 7fcd89ab6700 1 --2- 172.21.15.16:0/4071800151 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd6c003a00 0x7fcd6c003e20 secure :-1 s=READY pgs=2968 cs=0 l=1 rev1=1 crypto rx=0x7fcd740017c0 tx=0x7fcd6c005880 comp rx=0 tx=0).stop 2023-10-16T13:48:19.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.543+0000 7fcd89ab6700 1 -- 172.21.15.16:0/4071800151 shutdown_connections 2023-10-16T13:48:19.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.543+0000 7fcd89ab6700 1 --2- 172.21.15.16:0/4071800151 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd6c003a00 0x7fcd6c003e20 unknown :-1 s=CLOSED pgs=2968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:19.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.543+0000 7fcd89ab6700 1 -- 172.21.15.16:0/4071800151 >> 172.21.15.16:0/4071800151 conn(0x7fcd840b1f40 msgr2=0x7fcd840b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:19.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.543+0000 7fcd89ab6700 1 -- 172.21.15.16:0/4071800151 shutdown_connections 2023-10-16T13:48:19.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.543+0000 7fcd89ab6700 1 -- 172.21.15.16:0/4071800151 wait complete. 2023-10-16T13:48:19.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.544+0000 7fcd89ab6700 1 Processor -- start 2023-10-16T13:48:19.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.544+0000 7fcd89ab6700 1 -- start start 2023-10-16T13:48:19.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.545+0000 7fcd89ab6700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcd6c003a00 0x7fcd8415f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:19.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.545+0000 7fcd89ab6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd840ac8d0 0x7fcd84152fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:19.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.545+0000 7fcd89ab6700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcd840aea80 0x7fcd840aef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:19.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.545+0000 7fcd89ab6700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd840ad650 con 0x7fcd840ac8d0 2023-10-16T13:48:19.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.545+0000 7fcd89ab6700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd840ad090 con 0x7fcd6c003a00 2023-10-16T13:48:19.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.545+0000 7fcd89ab6700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd840ad370 con 0x7fcd840aea80 2023-10-16T13:48:19.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.545+0000 7fcd82ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd840ac8d0 0x7fcd84152fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:19.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.545+0000 7fcd82ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd840ac8d0 0x7fcd84152fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57312/0 (socket says 172.21.15.16:57312) 2023-10-16T13:48:19.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.545+0000 7fcd82ffd700 1 -- 172.21.15.16:0/3743759753 learned_addr learned my addr 172.21.15.16:0/3743759753 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:19.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.545+0000 7fcd837fe700 1 --2- 172.21.15.16:0/3743759753 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcd6c003a00 0x7fcd8415f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:19.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.545+0000 7fcd83fff700 1 --2- 172.21.15.16:0/3743759753 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcd840aea80 0x7fcd840aef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:19.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.545+0000 7fcd82ffd700 1 -- 172.21.15.16:0/3743759753 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcd840aea80 msgr2=0x7fcd840aef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:19.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.545+0000 7fcd82ffd700 1 --2- 172.21.15.16:0/3743759753 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcd840aea80 0x7fcd840aef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:19.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.545+0000 7fcd82ffd700 1 -- 172.21.15.16:0/3743759753 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcd6c003a00 msgr2=0x7fcd8415f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:19.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.545+0000 7fcd82ffd700 1 --2- 172.21.15.16:0/3743759753 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcd6c003a00 0x7fcd8415f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:19.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.545+0000 7fcd82ffd700 1 -- 172.21.15.16:0/3743759753 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd840ce140 con 0x7fcd840ac8d0 2023-10-16T13:48:19.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.545+0000 7fcd82ffd700 1 --2- 172.21.15.16:0/3743759753 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd840ac8d0 0x7fcd84152fd0 secure :-1 s=READY pgs=2969 cs=0 l=1 rev1=1 crypto rx=0x7fcd78000c00 tx=0x7fcd7800ea60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:19.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.545+0000 7fcd6bfff700 1 -- 172.21.15.16:0/3743759753 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd78010510 con 0x7fcd840ac8d0 2023-10-16T13:48:19.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.545+0000 7fcd6bfff700 1 -- 172.21.15.16:0/3743759753 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd78010670 con 0x7fcd840ac8d0 2023-10-16T13:48:19.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.545+0000 7fcd89ab6700 1 -- 172.21.15.16:0/3743759753 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd840ce420 con 0x7fcd840ac8d0 2023-10-16T13:48:19.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.545+0000 7fcd6bfff700 1 -- 172.21.15.16:0/3743759753 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd78010a00 con 0x7fcd840ac8d0 2023-10-16T13:48:19.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.546+0000 7fcd89ab6700 1 -- 172.21.15.16:0/3743759753 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd840acd50 con 0x7fcd840ac8d0 2023-10-16T13:48:19.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.547+0000 7fcd89ab6700 1 -- 172.21.15.16:0/3743759753 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd84071a30 con 0x7fcd840ac8d0 2023-10-16T13:48:19.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.547+0000 7fcd6bfff700 1 -- 172.21.15.16:0/3743759753 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fcd78010b60 con 0x7fcd840ac8d0 2023-10-16T13:48:19.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.548+0000 7fcd6bfff700 1 --2- 172.21.15.16:0/3743759753 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fcd7006b170 0x7fcd7006d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:19.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.548+0000 7fcd837fe700 1 --2- 172.21.15.16:0/3743759753 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fcd7006b170 0x7fcd7006d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:19.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.548+0000 7fcd6bfff700 1 -- 172.21.15.16:0/3743759753 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 6337+0+0 (secure 0 0 0) 0x7fcd78089290 con 0x7fcd840ac8d0 2023-10-16T13:48:19.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.548+0000 7fcd837fe700 1 --2- 172.21.15.16:0/3743759753 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fcd7006b170 0x7fcd7006d630 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fcd74000c00 tx=0x7fcd74009030 comp rx=0 tx=0).ready entity=mgr.12678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:19.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.552+0000 7fcd6bfff700 1 -- 172.21.15.16:0/3743759753 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcd78053ee0 con 0x7fcd840ac8d0 2023-10-16T13:48:19.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.756+0000 7fcd89ab6700 1 -- 172.21.15.16:0/3743759753 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fcd8404ebb0 con 0x7fcd840ac8d0 2023-10-16T13:48:19.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.759+0000 7fcd6bfff700 1 -- 172.21.15.16:0/3743759753 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v88)=0 v88) v1 ==== 122+0+0 (secure 0 0 0) 0x7fcd780190c0 con 0x7fcd840ac8d0 2023-10-16T13:48:19.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.761+0000 7fcd89ab6700 1 -- 172.21.15.16:0/3743759753 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fcd7006b170 msgr2=0x7fcd7006d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:19.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.761+0000 7fcd89ab6700 1 --2- 172.21.15.16:0/3743759753 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fcd7006b170 0x7fcd7006d630 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fcd74000c00 tx=0x7fcd74009030 comp rx=0 tx=0).stop 2023-10-16T13:48:19.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.761+0000 7fcd89ab6700 1 -- 172.21.15.16:0/3743759753 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd840ac8d0 msgr2=0x7fcd84152fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:19.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.761+0000 7fcd89ab6700 1 --2- 172.21.15.16:0/3743759753 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd840ac8d0 0x7fcd84152fd0 secure :-1 s=READY pgs=2969 cs=0 l=1 rev1=1 crypto rx=0x7fcd78000c00 tx=0x7fcd7800ea60 comp rx=0 tx=0).stop 2023-10-16T13:48:19.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.761+0000 7fcd89ab6700 1 -- 172.21.15.16:0/3743759753 shutdown_connections 2023-10-16T13:48:19.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.761+0000 7fcd89ab6700 1 --2- 172.21.15.16:0/3743759753 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fcd7006b170 0x7fcd7006d630 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:19.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.761+0000 7fcd89ab6700 1 --2- 172.21.15.16:0/3743759753 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcd840aea80 0x7fcd840aef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:19.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.761+0000 7fcd89ab6700 1 --2- 172.21.15.16:0/3743759753 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcd6c003a00 0x7fcd8415f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:19.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.761+0000 7fcd89ab6700 1 --2- 172.21.15.16:0/3743759753 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd840ac8d0 0x7fcd84152fd0 unknown :-1 s=CLOSED pgs=2969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:19.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.761+0000 7fcd89ab6700 1 -- 172.21.15.16:0/3743759753 >> 172.21.15.16:0/3743759753 conn(0x7fcd840b1f40 msgr2=0x7fcd840cce60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:19.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.761+0000 7fcd89ab6700 1 -- 172.21.15.16:0/3743759753 shutdown_connections 2023-10-16T13:48:19.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:19.761+0000 7fcd89ab6700 1 -- 172.21.15.16:0/3743759753 wait complete. 2023-10-16T13:48:19.776 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T13:48:19.776 DEBUG:teuthology.orchestra.run.smithi016:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T13:48:19.827 DEBUG:teuthology.orchestra.run.smithi049:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T13:48:19.875 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T13:48:19.876 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T13:48:19.876 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:48:19.893 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16287364k,nr_inodes=4071841,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20251 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3261172k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T13:48:19.894 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T13:48:19.894 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:48:19.914 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284204k,nr_inodes=4071051,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=753 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260540k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T13:48:19.914 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T13:48:19.915 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:48:19.951 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16287364k,nr_inodes=4071841,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20251 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3261172k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T13:48:19.952 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T13:48:19.952 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:48:19.973 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284204k,nr_inodes=4071051,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=753 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260540k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T13:48:19.974 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-16T13:48:19.974 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:48:20.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.113+0000 7f4da64b9700 1 Processor -- start 2023-10-16T13:48:20.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.113+0000 7f4da64b9700 1 -- start start 2023-10-16T13:48:20.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.114+0000 7f4da64b9700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4da015b280 0x7f4da01613e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:20.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.114+0000 7f4da64b9700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f4da00cc8a0 0x7f4da015e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:20.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.114+0000 7f4da64b9700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4da0161c30 con 0x7f4da006a7e0 2023-10-16T13:48:20.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.114+0000 7f4da64b9700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4da0161f10 con 0x7f4da00cc8a0 2023-10-16T13:48:20.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.114+0000 7f4da64b9700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4da01621f0 con 0x7f4da015b280 2023-10-16T13:48:20.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.114+0000 7f4d9f7fe700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f4da00cc8a0 0x7f4da015e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:20.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.114+0000 7f4d9ffff700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4da015b280 0x7f4da01613e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:20.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.114+0000 7f4da4a56700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7f4da006a7e0 0x7f4da00d02b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:53778/0 (socket says 172.21.15.16:53778) 2023-10-16T13:48:20.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.114+0000 7f4da4a56700 1 -- 172.21.15.16:0/459170868 learned_addr learned my addr 172.21.15.16:0/459170868 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:48:20.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.114+0000 7f4d9f7fe700 1 -- 172.21.15.16:0/459170868 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4da015b280 msgr2=0x7f4da01613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:20.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.114+0000 7f4d9f7fe700 1 --2- 172.21.15.16:0/459170868 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4da015b280 0x7f4da01613e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:20.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.114+0000 7f4d9f7fe700 1 -- 172.21.15.16:0/459170868 >> v1:172.21.15.49:6789/0 conn(0x7f4da006a7e0 legacy=0x7f4da00d02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:20.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.114+0000 7f4d9f7fe700 1 -- 172.21.15.16:0/459170868 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4da015df00 con 0x7f4da00cc8a0 2023-10-16T13:48:20.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.114+0000 7f4d9ffff700 1 --2- 172.21.15.16:0/459170868 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4da015b280 0x7f4da01613e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:48:20.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.115+0000 7f4d9f7fe700 1 --2- 172.21.15.16:0/459170868 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4da00cc8a0 0x7f4da015e700 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f4d88009840 tx=0x7f4d88009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f26441e4deb705f7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:20.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.115+0000 7f4d9effd700 1 -- 172.21.15.16:0/459170868 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d88011830 con 0x7f4da00cc8a0 2023-10-16T13:48:20.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.115+0000 7f4d9effd700 1 -- 172.21.15.16:0/459170868 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d88011990 con 0x7f4da00cc8a0 2023-10-16T13:48:20.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.115+0000 7f4d9effd700 1 -- 172.21.15.16:0/459170868 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d88011d20 con 0x7f4da00cc8a0 2023-10-16T13:48:20.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.115+0000 7f4da64b9700 1 -- 172.21.15.16:0/459170868 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4da00cc8a0 msgr2=0x7f4da015e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:20.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.115+0000 7f4da64b9700 1 --2- 172.21.15.16:0/459170868 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4da00cc8a0 0x7f4da015e700 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f4d88009840 tx=0x7f4d88009b50 comp rx=0 tx=0).stop 2023-10-16T13:48:20.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.115+0000 7f4da64b9700 1 -- 172.21.15.16:0/459170868 shutdown_connections 2023-10-16T13:48:20.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.115+0000 7f4da64b9700 1 --2- 172.21.15.16:0/459170868 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4da015b280 0x7f4da01613e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:20.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.115+0000 7f4da64b9700 1 --2- 172.21.15.16:0/459170868 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4da00cc8a0 0x7f4da015e700 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:20.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.115+0000 7f4da64b9700 1 -- 172.21.15.16:0/459170868 >> 172.21.15.16:0/459170868 conn(0x7f4da00b1fd0 msgr2=0x7f4da00b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:20.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.115+0000 7f4da64b9700 1 -- 172.21.15.16:0/459170868 shutdown_connections 2023-10-16T13:48:20.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.116+0000 7f4da64b9700 1 -- 172.21.15.16:0/459170868 wait complete. 2023-10-16T13:48:20.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.117+0000 7f4da64b9700 1 Processor -- start 2023-10-16T13:48:20.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.117+0000 7f4da64b9700 1 -- start start 2023-10-16T13:48:20.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.118+0000 7f4da64b9700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4da006a7e0 0x7f4da00c4380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:20.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.118+0000 7f4da64b9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4da00cc8a0 0x7f4da00c6960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:20.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.118+0000 7f4da64b9700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4da015b280 0x7f4da015b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:20.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.118+0000 7f4da64b9700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4da0145120 con 0x7f4da00cc8a0 2023-10-16T13:48:20.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.118+0000 7f4da64b9700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4da00c7db0 con 0x7f4da015b280 2023-10-16T13:48:20.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.118+0000 7f4da64b9700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4da00c7510 con 0x7f4da006a7e0 2023-10-16T13:48:20.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.118+0000 7f4d9f7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4da00cc8a0 0x7f4da00c6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:20.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.118+0000 7f4d9f7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4da00cc8a0 0x7f4da00c6960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57326/0 (socket says 172.21.15.16:57326) 2023-10-16T13:48:20.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.118+0000 7f4d9f7fe700 1 -- 172.21.15.16:0/2365796527 learned_addr learned my addr 172.21.15.16:0/2365796527 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:20.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.118+0000 7f4da4a56700 1 --2- 172.21.15.16:0/2365796527 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4da015b280 0x7f4da015b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:20.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.118+0000 7f4d9ffff700 1 --2- 172.21.15.16:0/2365796527 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4da006a7e0 0x7f4da00c4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:20.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.118+0000 7f4d9f7fe700 1 -- 172.21.15.16:0/2365796527 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4da006a7e0 msgr2=0x7f4da00c4380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:20.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.118+0000 7f4d9f7fe700 1 --2- 172.21.15.16:0/2365796527 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4da006a7e0 0x7f4da00c4380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:20.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.118+0000 7f4d9f7fe700 1 -- 172.21.15.16:0/2365796527 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4da015b280 msgr2=0x7f4da015b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:20.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.118+0000 7f4d9f7fe700 1 --2- 172.21.15.16:0/2365796527 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4da015b280 0x7f4da015b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:20.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.118+0000 7f4d9f7fe700 1 -- 172.21.15.16:0/2365796527 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d900094a0 con 0x7f4da00cc8a0 2023-10-16T13:48:20.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.118+0000 7f4d9f7fe700 1 --2- 172.21.15.16:0/2365796527 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4da00cc8a0 0x7f4da00c6960 secure :-1 s=READY pgs=2970 cs=0 l=1 rev1=1 crypto rx=0x7f4d88009fd0 tx=0x7f4d88004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:20.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.118+0000 7f4d9cff9700 1 -- 172.21.15.16:0/2365796527 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d880036a0 con 0x7f4da00cc8a0 2023-10-16T13:48:20.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.118+0000 7f4d9cff9700 1 -- 172.21.15.16:0/2365796527 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d8801f070 con 0x7f4da00cc8a0 2023-10-16T13:48:20.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.118+0000 7f4d9cff9700 1 -- 172.21.15.16:0/2365796527 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d88003a60 con 0x7f4da00cc8a0 2023-10-16T13:48:20.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.118+0000 7f4da64b9700 1 -- 172.21.15.16:0/2365796527 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d880094a0 con 0x7f4da00cc8a0 2023-10-16T13:48:20.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.119+0000 7f4da64b9700 1 -- 172.21.15.16:0/2365796527 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4da00c7ad0 con 0x7f4da00cc8a0 2023-10-16T13:48:20.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.119+0000 7f4da64b9700 1 -- 172.21.15.16:0/2365796527 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4da004ebb0 con 0x7f4da00cc8a0 2023-10-16T13:48:20.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.120+0000 7f4d9cff9700 1 -- 172.21.15.16:0/2365796527 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f4d88003bc0 con 0x7f4da00cc8a0 2023-10-16T13:48:20.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.120+0000 7f4d9cff9700 1 --2- 172.21.15.16:0/2365796527 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f4d8006b140 0x7f4d8006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:20.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.120+0000 7f4d9cff9700 1 -- 172.21.15.16:0/2365796527 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 6337+0+0 (secure 0 0 0) 0x7f4d88021070 con 0x7f4da00cc8a0 2023-10-16T13:48:20.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.120+0000 7f4d9ffff700 1 --2- 172.21.15.16:0/2365796527 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f4d8006b140 0x7f4d8006d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:20.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.121+0000 7f4d9ffff700 1 --2- 172.21.15.16:0/2365796527 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f4d8006b140 0x7f4d8006d600 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4d90009cd0 tx=0x7f4d90009210 comp rx=0 tx=0).ready entity=mgr.12678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:20.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.125+0000 7f4d9cff9700 1 -- 172.21.15.16:0/2365796527 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4d88055f50 con 0x7f4da00cc8a0 2023-10-16T13:48:20.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.373+0000 7f4da64b9700 1 -- 172.21.15.16:0/2365796527 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4da0071ce0 con 0x7f4da00cc8a0 2023-10-16T13:48:20.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.374+0000 7f4d9cff9700 1 -- 172.21.15.16:0/2365796527 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 139 v139) v1 ==== 95+0+3707 (secure 0 0 0) 0x7f4d8802be90 con 0x7f4da00cc8a0 2023-10-16T13:48:20.377 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:48:20.377 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":139,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9596,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/3939647668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":3939647668},{"type":"v1","addr":"172.21.15.49:6834","nonce":3939647668}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":125},{"gid":11185,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/2890113513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2890113513},{"type":"v1","addr":"172.21.15.16:6836","nonce":2890113513}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":125},{"gid":11752,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/1948262043","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":1948262043},{"type":"v1","addr":"172.21.15.49:6835","nonce":1948262043}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":139},{"gid":12864,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/1181609271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":1181609271},{"type":"v1","addr":"172.21.15.16:6837","nonce":1181609271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":137}],"filesystems":[]} 2023-10-16T13:48:20.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.376+0000 7f4da64b9700 1 -- 172.21.15.16:0/2365796527 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f4d8006b140 msgr2=0x7f4d8006d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:20.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.376+0000 7f4da64b9700 1 --2- 172.21.15.16:0/2365796527 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f4d8006b140 0x7f4d8006d600 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4d90009cd0 tx=0x7f4d90009210 comp rx=0 tx=0).stop 2023-10-16T13:48:20.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.376+0000 7f4da64b9700 1 -- 172.21.15.16:0/2365796527 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4da00cc8a0 msgr2=0x7f4da00c6960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:20.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.376+0000 7f4da64b9700 1 --2- 172.21.15.16:0/2365796527 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4da00cc8a0 0x7f4da00c6960 secure :-1 s=READY pgs=2970 cs=0 l=1 rev1=1 crypto rx=0x7f4d88009fd0 tx=0x7f4d88004750 comp rx=0 tx=0).stop 2023-10-16T13:48:20.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.376+0000 7f4da64b9700 1 -- 172.21.15.16:0/2365796527 shutdown_connections 2023-10-16T13:48:20.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.376+0000 7f4da64b9700 1 --2- 172.21.15.16:0/2365796527 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f4d8006b140 0x7f4d8006d600 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:20.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.376+0000 7f4da64b9700 1 --2- 172.21.15.16:0/2365796527 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4da015b280 0x7f4da015b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:20.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.376+0000 7f4da64b9700 1 --2- 172.21.15.16:0/2365796527 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4da006a7e0 0x7f4da00c4380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:20.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.376+0000 7f4da64b9700 1 --2- 172.21.15.16:0/2365796527 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4da00cc8a0 0x7f4da00c6960 unknown :-1 s=CLOSED pgs=2970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:20.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.376+0000 7f4da64b9700 1 -- 172.21.15.16:0/2365796527 >> 172.21.15.16:0/2365796527 conn(0x7f4da00b1fd0 msgr2=0x7f4da00724e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:20.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.376+0000 7f4da64b9700 1 -- 172.21.15.16:0/2365796527 shutdown_connections 2023-10-16T13:48:20.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.376+0000 7f4da64b9700 1 -- 172.21.15.16:0/2365796527 wait complete. 2023-10-16T13:48:20.381 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 139 2023-10-16T13:48:20.391 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-16T13:48:20.391 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-16T13:48:20.392 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-16T13:48:20.392 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T13:48:20.392 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-16T13:48:20.393 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-16T13:48:20.393 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-16T13:48:20.393 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T13:48:20.393 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-16T13:48:20.394 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-16T13:48:20.394 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-16T13:48:20.394 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T13:48:20.394 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-16T13:48:20.395 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-16T13:48:20.395 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-16T13:48:20.395 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T13:48:20.402 INFO:tasks.ceph.mds.a:Stopped 2023-10-16T13:48:20.403 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-16T13:48:20.404 INFO:tasks.ceph.mds.c:Stopped 2023-10-16T13:48:20.404 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-16T13:48:20.405 INFO:tasks.ceph.mds.b:Stopped 2023-10-16T13:48:20.405 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-16T13:48:20.406 INFO:tasks.ceph.mds.d:Stopped 2023-10-16T13:48:20.407 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-16T13:48:20.445 INFO:tasks.ceph.mds.a:Started 2023-10-16T13:48:20.446 DEBUG:teuthology.parallel:result is None 2023-10-16T13:48:20.447 INFO:tasks.ceph.mds.c:Started 2023-10-16T13:48:20.447 DEBUG:teuthology.parallel:result is None 2023-10-16T13:48:20.448 INFO:tasks.ceph.mds.b:Started 2023-10-16T13:48:20.448 DEBUG:teuthology.parallel:result is None 2023-10-16T13:48:20.449 INFO:tasks.ceph.mds.d:Started 2023-10-16T13:48:20.449 DEBUG:teuthology.parallel:result is None 2023-10-16T13:48:20.450 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-16T13:48:20.528 INFO:tasks.ceph.mds.a.smithi016.stdout:starting mds.a at 2023-10-16T13:48:20.529 INFO:tasks.ceph.mds.c.smithi016.stdout:starting mds.c at 2023-10-16T13:48:20.540 INFO:tasks.ceph.mds.b.smithi049.stdout:starting mds.b at 2023-10-16T13:48:20.541 INFO:tasks.ceph.mds.d.smithi049.stdout:starting mds.d at 2023-10-16T13:48:20.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.580+0000 7f4308044700 1 Processor -- start 2023-10-16T13:48:20.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.580+0000 7f4308044700 1 -- start start 2023-10-16T13:48:20.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.581+0000 7f4308044700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4300160eb0 0x7f430006d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:20.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.581+0000 7f4308044700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f43000cab70 0x7f43000caff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:20.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.581+0000 7f4308044700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f430006c150 con 0x7f430006b1f0 2023-10-16T13:48:20.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.581+0000 7f4308044700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43000af370 con 0x7f43000cab70 2023-10-16T13:48:20.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.581+0000 7f4308044700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43000cb530 con 0x7f4300160eb0 2023-10-16T13:48:20.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.581+0000 7f43055df700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f43000cab70 0x7f43000caff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:20.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.581+0000 7f43055df700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f43000cab70 0x7f43000caff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57360/0 (socket says 172.21.15.16:57360) 2023-10-16T13:48:20.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.581+0000 7f43055df700 1 -- 172.21.15.16:0/2540250623 learned_addr learned my addr 172.21.15.16:0/2540250623 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:20.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.581+0000 7f4305de0700 1 --2- 172.21.15.16:0/2540250623 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4300160eb0 0x7f430006d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:20.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.581+0000 7f4305de0700 1 -- 172.21.15.16:0/2540250623 >> v2:172.21.15.16:3300/0 conn(0x7f43000cab70 msgr2=0x7f43000caff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:20.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.582+0000 7f4305de0700 1 --2- 172.21.15.16:0/2540250623 >> v2:172.21.15.16:3300/0 conn(0x7f43000cab70 0x7f43000caff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:20.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.582+0000 7f4305de0700 1 -- 172.21.15.16:0/2540250623 >> v1:172.21.15.16:6789/0 conn(0x7f430006b1f0 legacy=0x7f430006b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:20.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.582+0000 7f4305de0700 1 -- 172.21.15.16:0/2540250623 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f430006be70 con 0x7f4300160eb0 2023-10-16T13:48:20.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.582+0000 7f4305de0700 1 --2- 172.21.15.16:0/2540250623 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4300160eb0 0x7f430006d8f0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f42f4009840 tx=0x7f42f4009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=89d878861e41c1ac server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:20.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.582+0000 7f4304dde700 1 -- 172.21.15.16:0/2540250623 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42f4011830 con 0x7f4300160eb0 2023-10-16T13:48:20.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.582+0000 7f4304dde700 1 -- 172.21.15.16:0/2540250623 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42f4011990 con 0x7f4300160eb0 2023-10-16T13:48:20.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.582+0000 7f4304dde700 1 -- 172.21.15.16:0/2540250623 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42f4011d20 con 0x7f4300160eb0 2023-10-16T13:48:20.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.583+0000 7f4308044700 1 -- 172.21.15.16:0/2540250623 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4300160eb0 msgr2=0x7f430006d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:20.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.583+0000 7f4308044700 1 --2- 172.21.15.16:0/2540250623 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4300160eb0 0x7f430006d8f0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f42f4009840 tx=0x7f42f4009b50 comp rx=0 tx=0).stop 2023-10-16T13:48:20.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.583+0000 7f4308044700 1 -- 172.21.15.16:0/2540250623 shutdown_connections 2023-10-16T13:48:20.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.583+0000 7f4308044700 1 --2- 172.21.15.16:0/2540250623 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4300160eb0 0x7f430006d8f0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:20.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.583+0000 7f4308044700 1 --2- 172.21.15.16:0/2540250623 >> v2:172.21.15.16:3300/0 conn(0x7f43000cab70 0x7f43000caff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:20.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.583+0000 7f4308044700 1 -- 172.21.15.16:0/2540250623 >> 172.21.15.16:0/2540250623 conn(0x7f43000cc5b0 msgr2=0x7f43000cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:20.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.583+0000 7f4308044700 1 -- 172.21.15.16:0/2540250623 shutdown_connections 2023-10-16T13:48:20.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.583+0000 7f4308044700 1 -- 172.21.15.16:0/2540250623 wait complete. 2023-10-16T13:48:20.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.584+0000 7f4308044700 1 Processor -- start 2023-10-16T13:48:20.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.584+0000 7f4308044700 1 -- start start 2023-10-16T13:48:20.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.585+0000 7f4308044700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f430006b1f0 0x7f43000c5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:20.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.585+0000 7f4308044700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f43000cab70 0x7f43001f66a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:20.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.585+0000 7f4308044700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4300160eb0 0x7f43001f6be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:20.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.585+0000 7f4308044700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43000c4850 con 0x7f4300160eb0 2023-10-16T13:48:20.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.585+0000 7f4308044700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43000c6500 con 0x7f430006b1f0 2023-10-16T13:48:20.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.585+0000 7f4308044700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43000c53f0 con 0x7f43000cab70 2023-10-16T13:48:20.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.585+0000 7f43065e1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4300160eb0 0x7f43001f6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:20.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.585+0000 7f43065e1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4300160eb0 0x7f43001f6be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57368/0 (socket says 172.21.15.16:57368) 2023-10-16T13:48:20.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.585+0000 7f43065e1700 1 -- 172.21.15.16:0/2011844811 learned_addr learned my addr 172.21.15.16:0/2011844811 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:20.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.585+0000 7f4305de0700 1 --2- 172.21.15.16:0/2011844811 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f430006b1f0 0x7f43000c5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:20.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.585+0000 7f43055df700 1 --2- 172.21.15.16:0/2011844811 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f43000cab70 0x7f43001f66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:20.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.585+0000 7f43065e1700 1 -- 172.21.15.16:0/2011844811 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f43000cab70 msgr2=0x7f43001f66a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:20.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.585+0000 7f43065e1700 1 --2- 172.21.15.16:0/2011844811 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f43000cab70 0x7f43001f66a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:20.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.585+0000 7f43065e1700 1 -- 172.21.15.16:0/2011844811 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f430006b1f0 msgr2=0x7f43000c5fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:20.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.585+0000 7f43065e1700 1 --2- 172.21.15.16:0/2011844811 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f430006b1f0 0x7f43000c5fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:20.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.585+0000 7f43065e1700 1 -- 172.21.15.16:0/2011844811 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42f40094a0 con 0x7f4300160eb0 2023-10-16T13:48:20.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.585+0000 7f43065e1700 1 --2- 172.21.15.16:0/2011844811 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4300160eb0 0x7f43001f6be0 secure :-1 s=READY pgs=2980 cs=0 l=1 rev1=1 crypto rx=0x7f42f800cb50 tx=0x7f42f800ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:20.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.585+0000 7f42f27fc700 1 -- 172.21.15.16:0/2011844811 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42f8003c00 con 0x7f4300160eb0 2023-10-16T13:48:20.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.585+0000 7f42f27fc700 1 -- 172.21.15.16:0/2011844811 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42f8003d60 con 0x7f4300160eb0 2023-10-16T13:48:20.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.586+0000 7f42f27fc700 1 -- 172.21.15.16:0/2011844811 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42f800aa20 con 0x7f4300160eb0 2023-10-16T13:48:20.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.586+0000 7f4308044700 1 -- 172.21.15.16:0/2011844811 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43000c3bd0 con 0x7f4300160eb0 2023-10-16T13:48:20.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.586+0000 7f4308044700 1 -- 172.21.15.16:0/2011844811 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43000c4190 con 0x7f4300160eb0 2023-10-16T13:48:20.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.586+0000 7f4308044700 1 -- 172.21.15.16:0/2011844811 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43000c4990 con 0x7f4300160eb0 2023-10-16T13:48:20.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.587+0000 7f42f27fc700 1 -- 172.21.15.16:0/2011844811 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f42f800ab80 con 0x7f4300160eb0 2023-10-16T13:48:20.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.587+0000 7f42f27fc700 1 --2- 172.21.15.16:0/2011844811 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f42e406b140 0x7f42e406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:20.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.587+0000 7f42f27fc700 1 -- 172.21.15.16:0/2011844811 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 6337+0+0 (secure 0 0 0) 0x7f42f8089ad0 con 0x7f4300160eb0 2023-10-16T13:48:20.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.587+0000 7f4305de0700 1 --2- 172.21.15.16:0/2011844811 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f42e406b140 0x7f42e406d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:20.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.588+0000 7f4305de0700 1 --2- 172.21.15.16:0/2011844811 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f42e406b140 0x7f42e406d600 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f42f4009fd0 tx=0x7f42f4008030 comp rx=0 tx=0).ready entity=mgr.12678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:20.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.592+0000 7f42f27fc700 1 -- 172.21.15.16:0/2011844811 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f43000c4990 con 0x7f4300160eb0 2023-10-16T13:48:20.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:20.813+0000 7f4308044700 1 -- 172.21.15.16:0/2011844811 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f43000c4990 con 0x7f4300160eb0 2023-10-16T13:48:21.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.396+0000 7f42f27fc700 1 -- 172.21.15.16:0/2011844811 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v259) v1 ==== 97+0+0 (secure 0 0 0) 0x7f43000c4990 con 0x7f4300160eb0 2023-10-16T13:48:21.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.400+0000 7f4308044700 1 -- 172.21.15.16:0/2011844811 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f42e406b140 msgr2=0x7f42e406d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:21.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.400+0000 7f4308044700 1 --2- 172.21.15.16:0/2011844811 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f42e406b140 0x7f42e406d600 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f42f4009fd0 tx=0x7f42f4008030 comp rx=0 tx=0).stop 2023-10-16T13:48:21.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.400+0000 7f4308044700 1 -- 172.21.15.16:0/2011844811 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4300160eb0 msgr2=0x7f43001f6be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:21.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.400+0000 7f4308044700 1 --2- 172.21.15.16:0/2011844811 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4300160eb0 0x7f43001f6be0 secure :-1 s=READY pgs=2980 cs=0 l=1 rev1=1 crypto rx=0x7f42f800cb50 tx=0x7f42f800ce60 comp rx=0 tx=0).stop 2023-10-16T13:48:21.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.400+0000 7f4308044700 1 -- 172.21.15.16:0/2011844811 shutdown_connections 2023-10-16T13:48:21.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.400+0000 7f4308044700 1 --2- 172.21.15.16:0/2011844811 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f42e406b140 0x7f42e406d600 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:21.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.400+0000 7f4308044700 1 --2- 172.21.15.16:0/2011844811 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4300160eb0 0x7f43001f6be0 unknown :-1 s=CLOSED pgs=2980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:21.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.400+0000 7f4308044700 1 --2- 172.21.15.16:0/2011844811 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f430006b1f0 0x7f43000c5fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:21.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.400+0000 7f4308044700 1 --2- 172.21.15.16:0/2011844811 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f43000cab70 0x7f43001f66a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:21.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.400+0000 7f4308044700 1 -- 172.21.15.16:0/2011844811 >> 172.21.15.16:0/2011844811 conn(0x7f43000cc5b0 msgr2=0x7f43000686c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:21.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.401+0000 7f4308044700 1 -- 172.21.15.16:0/2011844811 shutdown_connections 2023-10-16T13:48:21.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.401+0000 7f4308044700 1 -- 172.21.15.16:0/2011844811 wait complete. 2023-10-16T13:48:21.405 INFO:teuthology.orchestra.run.smithi016.stderr: removed all blocklist entries 2023-10-16T13:48:21.420 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-16T13:48:21.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.549+0000 7f7931241700 1 Processor -- start 2023-10-16T13:48:21.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.549+0000 7f7931241700 1 -- start start 2023-10-16T13:48:21.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.550+0000 7f7931241700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f792c15b280 0x7f792c1613e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:21.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.550+0000 7f7931241700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f792c06a7e0 0x7f792c06ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:21.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.550+0000 7f7931241700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f792c06b170 con 0x7f792c0cc8a0 2023-10-16T13:48:21.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.550+0000 7f7931241700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f792c161c30 con 0x7f792c06a7e0 2023-10-16T13:48:21.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.550+0000 7f7931241700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f792c161f10 con 0x7f792c15b280 2023-10-16T13:48:21.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.550+0000 7f792a59c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f792c0cc8a0 0x7f792c15e700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46648/0 (socket says 172.21.15.16:46648) 2023-10-16T13:48:21.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.550+0000 7f792a59c700 1 -- 172.21.15.16:0/1154248974 learned_addr learned my addr 172.21.15.16:0/1154248974 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:48:21.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.550+0000 7f792b59e700 1 --2- 172.21.15.16:0/1154248974 >> v2:172.21.15.49:3300/0 conn(0x7f792c06a7e0 0x7f792c06ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:21.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.550+0000 7f792ad9d700 1 --2- 172.21.15.16:0/1154248974 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f792c15b280 0x7f792c1613e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:21.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.550+0000 7f7929d9b700 1 -- 172.21.15.16:0/1154248974 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2211290227 0 0) 0x7f7920001110 con 0x7f792c0cc8a0 2023-10-16T13:48:21.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.550+0000 7f7929d9b700 1 -- 172.21.15.16:0/1154248974 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79140035b0 con 0x7f792c0cc8a0 2023-10-16T13:48:21.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.551+0000 7f7929d9b700 1 -- 172.21.15.16:0/1154248974 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3290425056 0 0) 0x7f79140035b0 con 0x7f792c0cc8a0 2023-10-16T13:48:21.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.551+0000 7f7929d9b700 1 -- 172.21.15.16:0/1154248974 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f792c15b280 msgr2=0x7f792c1613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:21.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.551+0000 7f7929d9b700 1 --2- 172.21.15.16:0/1154248974 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f792c15b280 0x7f792c1613e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:21.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.551+0000 7f7929d9b700 1 -- 172.21.15.16:0/1154248974 >> v2:172.21.15.49:3300/0 conn(0x7f792c06a7e0 msgr2=0x7f792c06ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:21.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.551+0000 7f7929d9b700 1 --2- 172.21.15.16:0/1154248974 >> v2:172.21.15.49:3300/0 conn(0x7f792c06a7e0 0x7f792c06ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:21.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.551+0000 7f7929d9b700 1 -- 172.21.15.16:0/1154248974 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f792c1621f0 con 0x7f792c0cc8a0 2023-10-16T13:48:21.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.551+0000 7f7929d9b700 1 -- 172.21.15.16:0/1154248974 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f7920001800 con 0x7f792c0cc8a0 2023-10-16T13:48:21.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.551+0000 7f7929d9b700 1 -- 172.21.15.16:0/1154248974 >> v1:172.21.15.16:6789/0 conn(0x7f792c0cc8a0 legacy=0x7f792c15e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:21.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.551+0000 7f7929d9b700 1 --2- 172.21.15.16:0/1154248974 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7914003a00 0x7f7914003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:21.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.551+0000 7f7929d9b700 1 --2- 172.21.15.16:0/1154248974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7914004850 0x7f7914004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:21.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.551+0000 7f7929d9b700 1 --2- 172.21.15.16:0/1154248974 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f792c15b280 0x7f7914005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:21.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.551+0000 7f7929d9b700 1 -- 172.21.15.16:0/1154248974 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f792c161c30 con 0x7f7914004850 2023-10-16T13:48:21.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.551+0000 7f7929d9b700 1 -- 172.21.15.16:0/1154248974 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f792c161f10 con 0x7f7914003a00 2023-10-16T13:48:21.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.551+0000 7f7929d9b700 1 -- 172.21.15.16:0/1154248974 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f791400ba10 con 0x7f792c15b280 2023-10-16T13:48:21.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.551+0000 7f792a59c700 1 --2- 172.21.15.16:0/1154248974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7914004850 0x7f7914004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:21.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.551+0000 7f792ad9d700 1 --2- 172.21.15.16:0/1154248974 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7914003a00 0x7f7914003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:21.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.551+0000 7f792b59e700 1 --2- 172.21.15.16:0/1154248974 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f792c15b280 0x7f7914005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:21.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.551+0000 7f792a59c700 1 -- 172.21.15.16:0/1154248974 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f792c15b280 msgr2=0x7f7914005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:21.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.551+0000 7f792a59c700 1 --2- 172.21.15.16:0/1154248974 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f792c15b280 0x7f7914005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:21.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.551+0000 7f792a59c700 1 -- 172.21.15.16:0/1154248974 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7914003a00 msgr2=0x7f7914003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:21.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.551+0000 7f792a59c700 1 --2- 172.21.15.16:0/1154248974 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7914003a00 0x7f7914003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:21.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.551+0000 7f792a59c700 1 -- 172.21.15.16:0/1154248974 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79140035b0 con 0x7f7914004850 2023-10-16T13:48:21.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.551+0000 7f792a59c700 1 --2- 172.21.15.16:0/1154248974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7914004850 0x7f7914004cd0 secure :-1 s=READY pgs=2982 cs=0 l=1 rev1=1 crypto rx=0x7f792000c2f0 tx=0x7f7914005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31cbf87e451f4672 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:21.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.552+0000 7f7929d9b700 1 -- 172.21.15.16:0/1154248974 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f792000ce30 con 0x7f7914004850 2023-10-16T13:48:21.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.552+0000 7f7929d9b700 1 -- 172.21.15.16:0/1154248974 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7920005960 con 0x7f7914004850 2023-10-16T13:48:21.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.552+0000 7f7929d9b700 1 -- 172.21.15.16:0/1154248974 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7920005d10 con 0x7f7914004850 2023-10-16T13:48:21.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.552+0000 7f7931241700 1 -- 172.21.15.16:0/1154248974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7914004850 msgr2=0x7f7914004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:21.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.552+0000 7f7931241700 1 --2- 172.21.15.16:0/1154248974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7914004850 0x7f7914004cd0 secure :-1 s=READY pgs=2982 cs=0 l=1 rev1=1 crypto rx=0x7f792000c2f0 tx=0x7f7914005880 comp rx=0 tx=0).stop 2023-10-16T13:48:21.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.552+0000 7f792b59e700 1 -- 172.21.15.16:0/1154248974 reap_dead start 2023-10-16T13:48:21.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.552+0000 7f7931241700 1 -- 172.21.15.16:0/1154248974 shutdown_connections 2023-10-16T13:48:21.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.552+0000 7f7931241700 1 -- 172.21.15.16:0/1154248974 >> 172.21.15.16:0/1154248974 conn(0x7f792c0b1fd0 msgr2=0x7f792c0b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:21.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.552+0000 7f7931241700 1 -- 172.21.15.16:0/1154248974 shutdown_connections 2023-10-16T13:48:21.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.552+0000 7f7931241700 1 -- 172.21.15.16:0/1154248974 wait complete. 2023-10-16T13:48:21.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.554+0000 7f7931241700 1 Processor -- start 2023-10-16T13:48:21.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.554+0000 7f7931241700 1 -- start start 2023-10-16T13:48:21.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.554+0000 7f7931241700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f792c0aeb20 0x7f792c0aef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:21.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.554+0000 7f7931241700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f792c0caf80 0x7f792c0cb400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:21.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.554+0000 7f7931241700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f792c0c4460 0x7f792c0c4910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:21.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.554+0000 7f7931241700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f792c0c59f0 con 0x7f792c0c4460 2023-10-16T13:48:21.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.554+0000 7f7931241700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f792c0cb940 con 0x7f792c0aeb20 2023-10-16T13:48:21.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.554+0000 7f7931241700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f792c1f3ca0 con 0x7f792c0caf80 2023-10-16T13:48:21.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.555+0000 7f792b59e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f792c0c4460 0x7f792c0c4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:21.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.555+0000 7f792b59e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f792c0c4460 0x7f792c0c4910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57390/0 (socket says 172.21.15.16:57390) 2023-10-16T13:48:21.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.555+0000 7f792b59e700 1 -- 172.21.15.16:0/1247468931 learned_addr learned my addr 172.21.15.16:0/1247468931 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:21.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.555+0000 7f792a59c700 1 --2- 172.21.15.16:0/1247468931 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f792c0caf80 0x7f792c0cb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:21.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.555+0000 7f792ad9d700 1 --2- 172.21.15.16:0/1247468931 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f792c0aeb20 0x7f792c0aef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:21.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.555+0000 7f792b59e700 1 -- 172.21.15.16:0/1247468931 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f792c0caf80 msgr2=0x7f792c0cb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:21.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.555+0000 7f792b59e700 1 --2- 172.21.15.16:0/1247468931 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f792c0caf80 0x7f792c0cb400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:21.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.555+0000 7f792b59e700 1 -- 172.21.15.16:0/1247468931 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f792c0aeb20 msgr2=0x7f792c0aef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:21.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.555+0000 7f792b59e700 1 --2- 172.21.15.16:0/1247468931 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f792c0aeb20 0x7f792c0aef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:21.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.555+0000 7f792b59e700 1 -- 172.21.15.16:0/1247468931 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f792c0c5150 con 0x7f792c0c4460 2023-10-16T13:48:21.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.555+0000 7f792ad9d700 1 --2- 172.21.15.16:0/1247468931 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f792c0aeb20 0x7f792c0aef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:48:21.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.555+0000 7f792b59e700 1 --2- 172.21.15.16:0/1247468931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f792c0c4460 0x7f792c0c4910 secure :-1 s=READY pgs=2983 cs=0 l=1 rev1=1 crypto rx=0x7f792c0caa20 tx=0x7f792400f490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:21.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.555+0000 7f79137fe700 1 -- 172.21.15.16:0/1247468931 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f792400e040 con 0x7f792c0c4460 2023-10-16T13:48:21.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.555+0000 7f79137fe700 1 -- 172.21.15.16:0/1247468931 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f792400f040 con 0x7f792c0c4460 2023-10-16T13:48:21.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.555+0000 7f7931241700 1 -- 172.21.15.16:0/1247468931 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f792c0c5430 con 0x7f792c0c4460 2023-10-16T13:48:21.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.555+0000 7f79137fe700 1 -- 172.21.15.16:0/1247468931 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7924015670 con 0x7f792c0c4460 2023-10-16T13:48:21.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.555+0000 7f7931241700 1 -- 172.21.15.16:0/1247468931 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f792c1f3ef0 con 0x7f792c0c4460 2023-10-16T13:48:21.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.556+0000 7f7931241700 1 -- 172.21.15.16:0/1247468931 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f792c04ebb0 con 0x7f792c0c4460 2023-10-16T13:48:21.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.557+0000 7f79137fe700 1 -- 172.21.15.16:0/1247468931 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f79240157d0 con 0x7f792c0c4460 2023-10-16T13:48:21.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.557+0000 7f79137fe700 1 --2- 172.21.15.16:0/1247468931 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f791806b140 0x7f791806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:21.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.557+0000 7f79137fe700 1 -- 172.21.15.16:0/1247468931 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(261..261 src has 1..261) v4 ==== 5305+0+0 (secure 0 0 0) 0x7f792408a280 con 0x7f792c0c4460 2023-10-16T13:48:21.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.557+0000 7f792ad9d700 1 --2- 172.21.15.16:0/1247468931 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f791806b140 0x7f791806d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:21.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.558+0000 7f792ad9d700 1 --2- 172.21.15.16:0/1247468931 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f791806b140 0x7f791806d600 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f791c009b00 tx=0x7f791c008040 comp rx=0 tx=0).ready entity=mgr.12678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:21.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.562+0000 7f79137fe700 1 -- 172.21.15.16:0/1247468931 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7924055190 con 0x7f792c0c4460 2023-10-16T13:48:21.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.850+0000 7f7931241700 1 -- 172.21.15.16:0/1247468931 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f792c0af770 con 0x7f792c0c4460 2023-10-16T13:48:21.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.850+0000 7f79137fe700 1 -- 172.21.15.16:0/1247468931 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v57) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f79240587b0 con 0x7f792c0c4460 2023-10-16T13:48:21.853 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:48:21.854 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-10-16T13:48:21.854 INFO:teuthology.orchestra.run.smithi016.stdout: "auth_dump": [ 2023-10-16T13:48:21.854 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:48:21.854 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.a", 2023-10-16T13:48:21.855 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOKC1l/rBtOhAA/ob5WUvpz8Glp4E+xPZOYA==", 2023-10-16T13:48:21.855 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:48:21.855 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T13:48:21.855 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-10-16T13:48:21.855 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-10-16T13:48:21.855 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:48:21.856 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:48:21.856 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:48:21.856 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:48:21.856 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.b", 2023-10-16T13:48:21.856 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1lKAVsCxAAvhXuLY6i1+Iz1yq5BVPMNQ==", 2023-10-16T13:48:21.857 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:48:21.857 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T13:48:21.857 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-10-16T13:48:21.857 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-10-16T13:48:21.857 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:48:21.858 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:48:21.858 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:48:21.858 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:48:21.858 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.c", 2023-10-16T13:48:21.858 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1lvBowBhAA3waJNYhdgs8PGCYqYNYUog==", 2023-10-16T13:48:21.858 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:48:21.859 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T13:48:21.859 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-10-16T13:48:21.859 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-10-16T13:48:21.859 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:48:21.859 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:48:21.860 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:48:21.860 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:48:21.860 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.d", 2023-10-16T13:48:21.860 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1lS/fnEhAAfNYgQ5BW86JMjYm/bBfnaQ==", 2023-10-16T13:48:21.860 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:48:21.861 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T13:48:21.861 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-10-16T13:48:21.861 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-10-16T13:48:21.861 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:48:21.861 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:48:21.861 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:48:21.862 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:48:21.862 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.0", 2023-10-16T13:48:21.862 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAQKC1lbxllOhAAZeLdp2GMl6HqQPaQjKlNqw==", 2023-10-16T13:48:21.862 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:48:21.862 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T13:48:21.863 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T13:48:21.863 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:48:21.863 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:48:21.863 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:48:21.863 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:48:21.864 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.1", 2023-10-16T13:48:21.864 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQATKC1lnU6KOhAAHxvlbmzBptf1keGgL+Rbkg==", 2023-10-16T13:48:21.864 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:48:21.864 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T13:48:21.864 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T13:48:21.865 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:48:21.865 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:48:21.865 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:48:21.865 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:48:21.865 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.2", 2023-10-16T13:48:21.866 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAXKC1l9jFyARAAr/NAwtWhBMut9sAAoDEGWg==", 2023-10-16T13:48:21.866 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:48:21.866 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T13:48:21.866 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T13:48:21.866 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:48:21.867 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:48:21.867 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:48:21.867 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:48:21.867 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.3", 2023-10-16T13:48:21.867 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAaKC1lbw/lARAAGJ/9DNb5/7kLFSq7vrLtXg==", 2023-10-16T13:48:21.867 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:48:21.868 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T13:48:21.868 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T13:48:21.868 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:48:21.868 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:48:21.868 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:48:21.868 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:48:21.869 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.4", 2023-10-16T13:48:21.869 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAeKC1lZb7jERAAYpvb1xcSli9CTq7L8ChLOA==", 2023-10-16T13:48:21.869 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:48:21.869 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T13:48:21.869 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T13:48:21.870 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:48:21.870 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:48:21.870 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:48:21.870 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:48:21.870 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.5", 2023-10-16T13:48:21.871 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAhKC1l/4HuEBAAqfnRqjknij9KES/fqiyIBQ==", 2023-10-16T13:48:21.871 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:48:21.871 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T13:48:21.871 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T13:48:21.871 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:48:21.871 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:48:21.872 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:48:21.872 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:48:21.872 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.6", 2023-10-16T13:48:21.872 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAkKC1lzoUkEBAA+Q82OiBMNag8+R316WQtUw==", 2023-10-16T13:48:21.872 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:48:21.872 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T13:48:21.873 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T13:48:21.873 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:48:21.873 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:48:21.873 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:48:21.873 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:48:21.874 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.7", 2023-10-16T13:48:21.874 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAnKC1ljUu4DxAAc+G4anJANC2GhLJ8wb6gYg==", 2023-10-16T13:48:21.874 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:48:21.874 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T13:48:21.874 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T13:48:21.875 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:48:21.875 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:48:21.875 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:48:21.875 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:48:21.875 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.0", 2023-10-16T13:48:21.875 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1l2l6PFxAA9QyXtBAHdGAXpwH0JfknuQ==", 2023-10-16T13:48:21.876 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:48:21.876 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T13:48:21.876 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow r", 2023-10-16T13:48:21.876 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-16T13:48:21.876 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:48:21.876 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:48:21.877 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:48:21.877 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.1", 2023-10-16T13:48:21.877 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1lAcHFIxAAcNt2bmv2R6j5L3DYkdw7eQ==", 2023-10-16T13:48:21.877 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:48:21.877 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T13:48:21.878 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-10-16T13:48:21.878 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-10-16T13:48:21.878 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-10-16T13:48:21.878 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:48:21.878 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:48:21.878 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:48:21.879 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.2", 2023-10-16T13:48:21.879 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1lA6wnGxAAdtaqYhtIJu8qENMrD8xvVA==", 2023-10-16T13:48:21.879 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:48:21.879 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T13:48:21.879 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-10-16T13:48:21.880 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-10-16T13:48:21.880 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-10-16T13:48:21.880 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:48:21.880 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:48:21.880 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:48:21.880 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.3", 2023-10-16T13:48:21.881 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1l/ddAJxAA3fNKKn9XkXas1BlDN+ggjw==", 2023-10-16T13:48:21.881 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:48:21.881 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T13:48:21.881 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-10-16T13:48:21.881 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-10-16T13:48:21.881 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-10-16T13:48:21.882 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:48:21.882 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:48:21.882 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:48:21.882 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.admin", 2023-10-16T13:48:21.882 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOKC1ld5jUHBAAc6esnCce3dSEZn3wVjhIlQ==", 2023-10-16T13:48:21.883 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:48:21.883 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2023-10-16T13:48:21.883 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-10-16T13:48:21.883 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-10-16T13:48:21.883 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:48:21.884 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:48:21.884 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:48:21.884 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:48:21.884 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mgr.x", 2023-10-16T13:48:21.884 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOKC1l6b3vNhAAmj5xbfid2sZM2KfHmAf1Wg==", 2023-10-16T13:48:21.884 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:48:21.885 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2023-10-16T13:48:21.885 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile mgr", 2023-10-16T13:48:21.885 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:48:21.885 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:48:21.885 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:48:21.886 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:48:21.886 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mgr.y", 2023-10-16T13:48:21.886 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOKC1lLz2nMhAA8NqFYZ5nDKkzPBMrkwsVuQ==", 2023-10-16T13:48:21.886 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:48:21.886 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2023-10-16T13:48:21.886 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile mgr", 2023-10-16T13:48:21.887 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:48:21.887 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:48:21.887 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:48:21.887 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-10-16T13:48:21.887 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-10-16T13:48:21.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.852+0000 7f7931241700 1 -- 172.21.15.16:0/1247468931 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f791806b140 msgr2=0x7f791806d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:21.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.852+0000 7f7931241700 1 --2- 172.21.15.16:0/1247468931 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f791806b140 0x7f791806d600 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f791c009b00 tx=0x7f791c008040 comp rx=0 tx=0).stop 2023-10-16T13:48:21.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.852+0000 7f7931241700 1 -- 172.21.15.16:0/1247468931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f792c0c4460 msgr2=0x7f792c0c4910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:21.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.852+0000 7f7931241700 1 --2- 172.21.15.16:0/1247468931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f792c0c4460 0x7f792c0c4910 secure :-1 s=READY pgs=2983 cs=0 l=1 rev1=1 crypto rx=0x7f792c0caa20 tx=0x7f792400f490 comp rx=0 tx=0).stop 2023-10-16T13:48:21.889 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.852+0000 7f7931241700 1 -- 172.21.15.16:0/1247468931 shutdown_connections 2023-10-16T13:48:21.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.852+0000 7f7931241700 1 --2- 172.21.15.16:0/1247468931 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f791806b140 0x7f791806d600 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:21.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.852+0000 7f7931241700 1 --2- 172.21.15.16:0/1247468931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f792c0c4460 0x7f792c0c4910 unknown :-1 s=CLOSED pgs=2983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:21.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.852+0000 7f7931241700 1 --2- 172.21.15.16:0/1247468931 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f792c0aeb20 0x7f792c0aef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:21.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.852+0000 7f7931241700 1 --2- 172.21.15.16:0/1247468931 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f792c0caf80 0x7f792c0cb400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:21.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.852+0000 7f7931241700 1 -- 172.21.15.16:0/1247468931 >> 172.21.15.16:0/1247468931 conn(0x7f792c0b1fd0 msgr2=0x7f792c0b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:21.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.853+0000 7f7931241700 1 -- 172.21.15.16:0/1247468931 shutdown_connections 2023-10-16T13:48:21.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:21.853+0000 7f7931241700 1 -- 172.21.15.16:0/1247468931 wait complete. 2023-10-16T13:48:21.891 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:48:22.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.015+0000 7ff64fe28700 1 Processor -- start 2023-10-16T13:48:22.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.015+0000 7ff64fe28700 1 -- start start 2023-10-16T13:48:22.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.016+0000 7ff64fe28700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff64806b7f0 0x7ff64806bc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:22.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.016+0000 7ff64fe28700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff64806d940 0x7ff64806ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:22.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.016+0000 7ff64fe28700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6480c3490 con 0x7ff64815b9f0 2023-10-16T13:48:22.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.016+0000 7ff64fe28700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff64806c1b0 con 0x7ff64806d940 2023-10-16T13:48:22.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.016+0000 7ff64fe28700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff64806e300 con 0x7ff64806b7f0 2023-10-16T13:48:22.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.016+0000 7ff64e3c5700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff64806d940 0x7ff64806ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:22.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.016+0000 7ff64e3c5700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff64806d940 0x7ff64806ddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57398/0 (socket says 172.21.15.16:57398) 2023-10-16T13:48:22.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.016+0000 7ff64e3c5700 1 -- 172.21.15.16:0/3823114620 learned_addr learned my addr 172.21.15.16:0/3823114620 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:22.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.017+0000 7ff64d3c3700 1 --2- 172.21.15.16:0/3823114620 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff64806b7f0 0x7ff64806bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:22.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.017+0000 7ff64e3c5700 1 -- 172.21.15.16:0/3823114620 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff64806b7f0 msgr2=0x7ff64806bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:22.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.017+0000 7ff64e3c5700 1 --2- 172.21.15.16:0/3823114620 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff64806b7f0 0x7ff64806bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:22.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.017+0000 7ff64e3c5700 1 -- 172.21.15.16:0/3823114620 >> v1:172.21.15.49:6789/0 conn(0x7ff64815b9f0 legacy=0x7ff6480b29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:22.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.017+0000 7ff64e3c5700 1 -- 172.21.15.16:0/3823114620 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6480cdad0 con 0x7ff64806d940 2023-10-16T13:48:22.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.017+0000 7ff64e3c5700 1 --2- 172.21.15.16:0/3823114620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff64806d940 0x7ff64806ddc0 secure :-1 s=READY pgs=2984 cs=0 l=1 rev1=1 crypto rx=0x7ff640009e50 tx=0x7ff64000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=448ff6b8fcdf4967 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:22.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.017+0000 7ff64cbc2700 1 -- 172.21.15.16:0/3823114620 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff64000e070 con 0x7ff64806d940 2023-10-16T13:48:22.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.017+0000 7ff64cbc2700 1 -- 172.21.15.16:0/3823114620 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff64000b9c0 con 0x7ff64806d940 2023-10-16T13:48:22.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.017+0000 7ff64cbc2700 1 -- 172.21.15.16:0/3823114620 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff64000bd70 con 0x7ff64806d940 2023-10-16T13:48:22.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.018+0000 7ff64fe28700 1 -- 172.21.15.16:0/3823114620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff64806d940 msgr2=0x7ff64806ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:22.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.018+0000 7ff64fe28700 1 --2- 172.21.15.16:0/3823114620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff64806d940 0x7ff64806ddc0 secure :-1 s=READY pgs=2984 cs=0 l=1 rev1=1 crypto rx=0x7ff640009e50 tx=0x7ff64000b330 comp rx=0 tx=0).stop 2023-10-16T13:48:22.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.018+0000 7ff64fe28700 1 -- 172.21.15.16:0/3823114620 shutdown_connections 2023-10-16T13:48:22.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.018+0000 7ff64fe28700 1 --2- 172.21.15.16:0/3823114620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff64806d940 0x7ff64806ddc0 unknown :-1 s=CLOSED pgs=2984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:22.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.018+0000 7ff64fe28700 1 --2- 172.21.15.16:0/3823114620 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff64806b7f0 0x7ff64806bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:22.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.018+0000 7ff64fe28700 1 -- 172.21.15.16:0/3823114620 >> 172.21.15.16:0/3823114620 conn(0x7ff6480cc5b0 msgr2=0x7ff6480cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:22.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.018+0000 7ff64fe28700 1 -- 172.21.15.16:0/3823114620 shutdown_connections 2023-10-16T13:48:22.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.018+0000 7ff64fe28700 1 -- 172.21.15.16:0/3823114620 wait complete. 2023-10-16T13:48:22.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.019+0000 7ff64fe28700 1 Processor -- start 2023-10-16T13:48:22.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.019+0000 7ff64fe28700 1 -- start start 2023-10-16T13:48:22.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.020+0000 7ff64fe28700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff64806b7f0 0x7ff64814bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:22.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.020+0000 7ff64fe28700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff64806d940 0x7ff6480c7780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:22.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.020+0000 7ff64fe28700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff64815de50 0x7ff64815e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:22.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.020+0000 7ff64fe28700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6480c7360 con 0x7ff64806b7f0 2023-10-16T13:48:22.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.020+0000 7ff64fe28700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff64815e8d0 con 0x7ff64815de50 2023-10-16T13:48:22.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.020+0000 7ff64fe28700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6480c6ca0 con 0x7ff64806d940 2023-10-16T13:48:22.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.020+0000 7ff64dbc4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff64806b7f0 0x7ff64814bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:22.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.020+0000 7ff64dbc4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff64806b7f0 0x7ff64814bf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57400/0 (socket says 172.21.15.16:57400) 2023-10-16T13:48:22.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.020+0000 7ff64dbc4700 1 -- 172.21.15.16:0/2028093803 learned_addr learned my addr 172.21.15.16:0/2028093803 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:22.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.020+0000 7ff64d3c3700 1 --2- 172.21.15.16:0/2028093803 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff64806d940 0x7ff6480c7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:22.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.020+0000 7ff64e3c5700 1 --2- 172.21.15.16:0/2028093803 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff64815de50 0x7ff64815e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:22.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.020+0000 7ff64dbc4700 1 -- 172.21.15.16:0/2028093803 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff64806d940 msgr2=0x7ff6480c7780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:22.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.020+0000 7ff64dbc4700 1 --2- 172.21.15.16:0/2028093803 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff64806d940 0x7ff6480c7780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:22.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.020+0000 7ff64dbc4700 1 -- 172.21.15.16:0/2028093803 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff64815de50 msgr2=0x7ff64815e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:22.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.020+0000 7ff64dbc4700 1 --2- 172.21.15.16:0/2028093803 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff64815de50 0x7ff64815e2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:22.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.020+0000 7ff64dbc4700 1 -- 172.21.15.16:0/2028093803 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff640009b30 con 0x7ff64806b7f0 2023-10-16T13:48:22.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.020+0000 7ff64dbc4700 1 --2- 172.21.15.16:0/2028093803 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff64806b7f0 0x7ff64814bf00 secure :-1 s=READY pgs=2985 cs=0 l=1 rev1=1 crypto rx=0x7ff64400c4e0 tx=0x7ff64400c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:22.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.020+0000 7ff63a7fc700 1 -- 172.21.15.16:0/2028093803 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6440052d0 con 0x7ff64806b7f0 2023-10-16T13:48:22.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.020+0000 7ff63a7fc700 1 -- 172.21.15.16:0/2028093803 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff644005430 con 0x7ff64806b7f0 2023-10-16T13:48:22.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.020+0000 7ff64fe28700 1 -- 172.21.15.16:0/2028093803 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6480c6f80 con 0x7ff64806b7f0 2023-10-16T13:48:22.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.021+0000 7ff63a7fc700 1 -- 172.21.15.16:0/2028093803 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6440038c0 con 0x7ff64806b7f0 2023-10-16T13:48:22.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.021+0000 7ff64fe28700 1 -- 172.21.15.16:0/2028093803 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff64815ebb0 con 0x7ff64806b7f0 2023-10-16T13:48:22.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.022+0000 7ff64fe28700 1 -- 172.21.15.16:0/2028093803 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff64804ebb0 con 0x7ff64806b7f0 2023-10-16T13:48:22.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.022+0000 7ff63a7fc700 1 -- 172.21.15.16:0/2028093803 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7ff644003a20 con 0x7ff64806b7f0 2023-10-16T13:48:22.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.022+0000 7ff63a7fc700 1 --2- 172.21.15.16:0/2028093803 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7ff62c06b140 0x7ff62c06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:22.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.023+0000 7ff64d3c3700 1 --2- 172.21.15.16:0/2028093803 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7ff62c06b140 0x7ff62c06d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:22.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.023+0000 7ff63a7fc700 1 -- 172.21.15.16:0/2028093803 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(261..261 src has 1..261) v4 ==== 5305+0+0 (secure 0 0 0) 0x7ff644088ae0 con 0x7ff64806b7f0 2023-10-16T13:48:22.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.023+0000 7ff64d3c3700 1 --2- 172.21.15.16:0/2028093803 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7ff62c06b140 0x7ff62c06d600 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff64814c8a0 tx=0x7ff63c008040 comp rx=0 tx=0).ready entity=mgr.12678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:22.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.027+0000 7ff63a7fc700 1 -- 172.21.15.16:0/2028093803 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff644053a90 con 0x7ff64806b7f0 2023-10-16T13:48:22.234 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.232+0000 7ff64fe28700 1 -- 172.21.15.16:0/2028093803 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff6480c75d0 con 0x7ff64806b7f0 2023-10-16T13:48:22.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.233+0000 7ff63a7fc700 1 -- 172.21.15.16:0/2028093803 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v261) v1 ==== 74+0+13169 (secure 0 0 0) 0x7ff644017090 con 0x7ff64806b7f0 2023-10-16T13:48:22.235 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:48:22.235 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":261,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:48:21.404146+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:48:22.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.235+0000 7ff64fe28700 1 -- 172.21.15.16:0/2028093803 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7ff62c06b140 msgr2=0x7ff62c06d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:22.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.235+0000 7ff64fe28700 1 --2- 172.21.15.16:0/2028093803 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7ff62c06b140 0x7ff62c06d600 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff64814c8a0 tx=0x7ff63c008040 comp rx=0 tx=0).stop 2023-10-16T13:48:22.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.235+0000 7ff64fe28700 1 -- 172.21.15.16:0/2028093803 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff64806b7f0 msgr2=0x7ff64814bf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:22.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.235+0000 7ff64fe28700 1 --2- 172.21.15.16:0/2028093803 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff64806b7f0 0x7ff64814bf00 secure :-1 s=READY pgs=2985 cs=0 l=1 rev1=1 crypto rx=0x7ff64400c4e0 tx=0x7ff64400c7f0 comp rx=0 tx=0).stop 2023-10-16T13:48:22.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.235+0000 7ff64fe28700 1 -- 172.21.15.16:0/2028093803 shutdown_connections 2023-10-16T13:48:22.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.235+0000 7ff64fe28700 1 --2- 172.21.15.16:0/2028093803 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7ff62c06b140 0x7ff62c06d600 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:22.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.235+0000 7ff64fe28700 1 --2- 172.21.15.16:0/2028093803 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff64815de50 0x7ff64815e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:22.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.235+0000 7ff64fe28700 1 --2- 172.21.15.16:0/2028093803 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff64806b7f0 0x7ff64814bf00 unknown :-1 s=CLOSED pgs=2985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:22.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.235+0000 7ff64fe28700 1 --2- 172.21.15.16:0/2028093803 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff64806d940 0x7ff6480c7780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:22.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.235+0000 7ff64fe28700 1 -- 172.21.15.16:0/2028093803 >> 172.21.15.16:0/2028093803 conn(0x7ff6480cc5b0 msgr2=0x7ff6480b5320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:22.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.235+0000 7ff64fe28700 1 -- 172.21.15.16:0/2028093803 shutdown_connections 2023-10-16T13:48:22.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.235+0000 7ff64fe28700 1 -- 172.21.15.16:0/2028093803 wait complete. 2023-10-16T13:48:22.250 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T13:48:22.250 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T13:48:22.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.377+0000 7f8b85b73700 1 Processor -- start 2023-10-16T13:48:22.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.377+0000 7f8b85b73700 1 -- start start 2023-10-16T13:48:22.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.378+0000 7f8b85b73700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f8b8015a640 0x7f8b800cb710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:22.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.378+0000 7f8b85b73700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b8015aa20 con 0x7f8b8015e140 2023-10-16T13:48:22.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.378+0000 7f8b85b73700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b800cafd0 con 0x7f8b800c6770 2023-10-16T13:48:22.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.378+0000 7f8b85b73700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b8006eb40 con 0x7f8b8015a640 2023-10-16T13:48:22.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.378+0000 7f8b7ffff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f8b8015e140 0x7f8b800ca8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46686/0 (socket says 172.21.15.16:46686) 2023-10-16T13:48:22.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.378+0000 7f8b7ffff700 1 -- 172.21.15.16:0/2184377158 learned_addr learned my addr 172.21.15.16:0/2184377158 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:48:22.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.378+0000 7f8b7f7fe700 1 --2- 172.21.15.16:0/2184377158 >> v2:172.21.15.49:3300/0 conn(0x7f8b8015a640 0x7f8b800cb710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:22.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.378+0000 7f8b7e7fc700 1 -- 172.21.15.16:0/2184377158 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1832092372 0 0) 0x7f8b78001490 con 0x7f8b8015e140 2023-10-16T13:48:22.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.378+0000 7f8b7e7fc700 1 -- 172.21.15.16:0/2184377158 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b6c0035b0 con 0x7f8b8015e140 2023-10-16T13:48:22.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.379+0000 7f8b7f7fe700 1 -- 172.21.15.16:0/2184377158 >> v1:172.21.15.49:6789/0 conn(0x7f8b800c6770 legacy=0x7f8b800ca1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:22.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.379+0000 7f8b7f7fe700 1 -- 172.21.15.16:0/2184377158 >> v1:172.21.15.16:6789/0 conn(0x7f8b8015e140 legacy=0x7f8b800ca8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:22.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.379+0000 7f8b7e7fc700 1 -- 172.21.15.16:0/2184377158 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2323065737 0 0) 0x7f8b6c0035b0 con 0x7f8b8015e140 2023-10-16T13:48:22.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.379+0000 7f8b7f7fe700 1 -- 172.21.15.16:0/2184377158 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b8006ee20 con 0x7f8b8015a640 2023-10-16T13:48:22.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.379+0000 7f8b7f7fe700 1 --2- 172.21.15.16:0/2184377158 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8b8015a640 0x7f8b800cb710 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f8b68009840 tx=0x7f8b68009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=29d71815298169f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:22.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.379+0000 7f8b7e7fc700 1 -- 172.21.15.16:0/2184377158 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b78001b80 con 0x7f8b8015a640 2023-10-16T13:48:22.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.379+0000 7f8b7e7fc700 1 -- 172.21.15.16:0/2184377158 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b68004030 con 0x7f8b8015a640 2023-10-16T13:48:22.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.379+0000 7f8b7e7fc700 1 -- 172.21.15.16:0/2184377158 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b680043c0 con 0x7f8b8015a640 2023-10-16T13:48:22.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.380+0000 7f8b85b73700 1 -- 172.21.15.16:0/2184377158 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8b8015a640 msgr2=0x7f8b800cb710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:22.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.380+0000 7f8b85b73700 1 --2- 172.21.15.16:0/2184377158 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8b8015a640 0x7f8b800cb710 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f8b68009840 tx=0x7f8b68009b50 comp rx=0 tx=0).stop 2023-10-16T13:48:22.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.380+0000 7f8b85b73700 1 -- 172.21.15.16:0/2184377158 shutdown_connections 2023-10-16T13:48:22.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.380+0000 7f8b85b73700 1 --2- 172.21.15.16:0/2184377158 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8b8015a640 0x7f8b800cb710 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:22.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.380+0000 7f8b85b73700 1 -- 172.21.15.16:0/2184377158 >> 172.21.15.16:0/2184377158 conn(0x7f8b800cc5e0 msgr2=0x7f8b800cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:22.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.380+0000 7f8b85b73700 1 -- 172.21.15.16:0/2184377158 shutdown_connections 2023-10-16T13:48:22.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.380+0000 7f8b85b73700 1 -- 172.21.15.16:0/2184377158 wait complete. 2023-10-16T13:48:22.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.381+0000 7f8b85b73700 1 Processor -- start 2023-10-16T13:48:22.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.381+0000 7f8b85b73700 1 -- start start 2023-10-16T13:48:22.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.382+0000 7f8b85b73700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8b800c6770 0x7f8b8014be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:22.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.382+0000 7f8b85b73700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8b8015a640 0x7f8b800c5660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:22.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.382+0000 7f8b85b73700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b8015e140 0x7f8b800c5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:22.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.382+0000 7f8b85b73700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b8015d350 con 0x7f8b8015e140 2023-10-16T13:48:22.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.382+0000 7f8b85b73700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b8015ca40 con 0x7f8b8015a640 2023-10-16T13:48:22.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.382+0000 7f8b85b73700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b801f4150 con 0x7f8b800c6770 2023-10-16T13:48:22.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.382+0000 7f8b7ffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b8015e140 0x7f8b800c5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:22.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.382+0000 7f8b7ffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b8015e140 0x7f8b800c5ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57418/0 (socket says 172.21.15.16:57418) 2023-10-16T13:48:22.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.382+0000 7f8b7ffff700 1 -- 172.21.15.16:0/439235179 learned_addr learned my addr 172.21.15.16:0/439235179 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:22.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.382+0000 7f8b7f7fe700 1 --2- 172.21.15.16:0/439235179 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8b800c6770 0x7f8b8014be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:22.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.382+0000 7f8b7effd700 1 --2- 172.21.15.16:0/439235179 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8b8015a640 0x7f8b800c5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:22.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.382+0000 7f8b7ffff700 1 -- 172.21.15.16:0/439235179 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8b800c6770 msgr2=0x7f8b8014be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:22.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.382+0000 7f8b7ffff700 1 --2- 172.21.15.16:0/439235179 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8b800c6770 0x7f8b8014be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:22.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.382+0000 7f8b7ffff700 1 -- 172.21.15.16:0/439235179 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8b8015a640 msgr2=0x7f8b800c5660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:22.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.382+0000 7f8b7ffff700 1 --2- 172.21.15.16:0/439235179 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8b8015a640 0x7f8b800c5660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:22.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.382+0000 7f8b7ffff700 1 -- 172.21.15.16:0/439235179 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b680094a0 con 0x7f8b8015e140 2023-10-16T13:48:22.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.382+0000 7f8b7effd700 1 --2- 172.21.15.16:0/439235179 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8b8015a640 0x7f8b800c5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:48:22.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.383+0000 7f8b7ffff700 1 --2- 172.21.15.16:0/439235179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b8015e140 0x7f8b800c5ba0 secure :-1 s=READY pgs=2987 cs=0 l=1 rev1=1 crypto rx=0x7f8b78001cf0 tx=0x7f8b7800c550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:22.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.383+0000 7f8b77fff700 1 -- 172.21.15.16:0/439235179 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b780070d0 con 0x7f8b8015e140 2023-10-16T13:48:22.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.383+0000 7f8b77fff700 1 -- 172.21.15.16:0/439235179 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b78007230 con 0x7f8b8015e140 2023-10-16T13:48:22.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.383+0000 7f8b85b73700 1 -- 172.21.15.16:0/439235179 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b8015cc90 con 0x7f8b8015e140 2023-10-16T13:48:22.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.383+0000 7f8b77fff700 1 -- 172.21.15.16:0/439235179 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b780075e0 con 0x7f8b8015e140 2023-10-16T13:48:22.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.383+0000 7f8b85b73700 1 -- 172.21.15.16:0/439235179 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b8014c3d0 con 0x7f8b8015e140 2023-10-16T13:48:22.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.384+0000 7f8b85b73700 1 -- 172.21.15.16:0/439235179 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b8015d490 con 0x7f8b8015e140 2023-10-16T13:48:22.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.384+0000 7f8b77fff700 1 -- 172.21.15.16:0/439235179 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f8b78007740 con 0x7f8b8015e140 2023-10-16T13:48:22.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.385+0000 7f8b77fff700 1 --2- 172.21.15.16:0/439235179 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f8b6006b140 0x7f8b6006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:22.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.385+0000 7f8b77fff700 1 -- 172.21.15.16:0/439235179 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(261..261 src has 1..261) v4 ==== 5305+0+0 (secure 0 0 0) 0x7f8b7808c870 con 0x7f8b8015e140 2023-10-16T13:48:22.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.385+0000 7f8b7f7fe700 1 --2- 172.21.15.16:0/439235179 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f8b6006b140 0x7f8b6006d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:22.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.385+0000 7f8b7f7fe700 1 --2- 172.21.15.16:0/439235179 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f8b6006b140 0x7f8b6006d600 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8b680090b0 tx=0x7f8b68008030 comp rx=0 tx=0).ready entity=mgr.12678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:22.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.389+0000 7f8b77fff700 1 -- 172.21.15.16:0/439235179 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8b8015d490 con 0x7f8b8015e140 2023-10-16T13:48:22.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.594+0000 7f8b85b73700 1 -- 172.21.15.16:0/439235179 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8b8015d490 con 0x7f8b8015e140 2023-10-16T13:48:22.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.595+0000 7f8b77fff700 1 -- 172.21.15.16:0/439235179 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v261) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8b8015d490 con 0x7f8b8015e140 2023-10-16T13:48:22.598 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T13:48:22.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.597+0000 7f8b85b73700 1 -- 172.21.15.16:0/439235179 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f8b6006b140 msgr2=0x7f8b6006d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:22.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.597+0000 7f8b85b73700 1 --2- 172.21.15.16:0/439235179 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f8b6006b140 0x7f8b6006d600 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8b680090b0 tx=0x7f8b68008030 comp rx=0 tx=0).stop 2023-10-16T13:48:22.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.597+0000 7f8b85b73700 1 -- 172.21.15.16:0/439235179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b8015e140 msgr2=0x7f8b800c5ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:22.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.597+0000 7f8b85b73700 1 --2- 172.21.15.16:0/439235179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b8015e140 0x7f8b800c5ba0 secure :-1 s=READY pgs=2987 cs=0 l=1 rev1=1 crypto rx=0x7f8b78001cf0 tx=0x7f8b7800c550 comp rx=0 tx=0).stop 2023-10-16T13:48:22.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.597+0000 7f8b85b73700 1 -- 172.21.15.16:0/439235179 shutdown_connections 2023-10-16T13:48:22.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.597+0000 7f8b85b73700 1 --2- 172.21.15.16:0/439235179 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f8b6006b140 0x7f8b6006d600 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:22.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.597+0000 7f8b85b73700 1 --2- 172.21.15.16:0/439235179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b8015e140 0x7f8b800c5ba0 unknown :-1 s=CLOSED pgs=2987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:22.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.597+0000 7f8b85b73700 1 --2- 172.21.15.16:0/439235179 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f8b800c6770 0x7f8b8014be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:22.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.597+0000 7f8b85b73700 1 --2- 172.21.15.16:0/439235179 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8b8015a640 0x7f8b800c5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:22.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.597+0000 7f8b85b73700 1 -- 172.21.15.16:0/439235179 >> 172.21.15.16:0/439235179 conn(0x7f8b800cc5e0 msgr2=0x7f8b800c2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:22.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.597+0000 7f8b85b73700 1 -- 172.21.15.16:0/439235179 shutdown_connections 2023-10-16T13:48:22.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.597+0000 7f8b85b73700 1 -- 172.21.15.16:0/439235179 wait complete. 2023-10-16T13:48:22.612 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-16T13:48:22.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.735+0000 7f7150d1e700 1 Processor -- start 2023-10-16T13:48:22.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.735+0000 7f7150d1e700 1 -- start start 2023-10-16T13:48:22.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.736+0000 7f7150d1e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f714c0cf4a0 0x7f714c0726e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:22.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.736+0000 7f7150d1e700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f714c15bc80 con 0x7f714c0cf880 2023-10-16T13:48:22.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.736+0000 7f7150d1e700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f714c0b5e70 con 0x7f714c0b5a00 2023-10-16T13:48:22.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.736+0000 7f7150d1e700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f714c072c20 con 0x7f714c0cf4a0 2023-10-16T13:48:22.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.736+0000 7f714a59c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f714c0cf4a0 0x7f714c0726e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:22.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.736+0000 7f714a59c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f714c0cf4a0 0x7f714c0726e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57424/0 (socket says 172.21.15.16:57424) 2023-10-16T13:48:22.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.736+0000 7f714a59c700 1 -- 172.21.15.16:0/556455791 learned_addr learned my addr 172.21.15.16:0/556455791 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:22.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.736+0000 7f714a59c700 1 -- 172.21.15.16:0/556455791 >> v1:172.21.15.49:6789/0 conn(0x7f714c0b5a00 legacy=0x7f714c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:22.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.736+0000 7f714a59c700 1 -- 172.21.15.16:0/556455791 >> v1:172.21.15.16:6789/0 conn(0x7f714c0cf880 legacy=0x7f714c0b52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:22.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.736+0000 7f714a59c700 1 -- 172.21.15.16:0/556455791 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f714c15b070 con 0x7f714c0cf4a0 2023-10-16T13:48:22.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.736+0000 7f714a59c700 1 --2- 172.21.15.16:0/556455791 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f714c0cf4a0 0x7f714c0726e0 secure :-1 s=READY pgs=2989 cs=0 l=1 rev1=1 crypto rx=0x7f713c0098f0 tx=0x7f713c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb37677597f86967 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:22.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.736+0000 7f714959a700 1 -- 172.21.15.16:0/556455791 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f713c00b8a0 con 0x7f714c0cf4a0 2023-10-16T13:48:22.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.737+0000 7f714959a700 1 -- 172.21.15.16:0/556455791 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f713c00ba00 con 0x7f714c0cf4a0 2023-10-16T13:48:22.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.737+0000 7f714959a700 1 -- 172.21.15.16:0/556455791 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f713c00bd90 con 0x7f714c0cf4a0 2023-10-16T13:48:22.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.737+0000 7f7150d1e700 1 -- 172.21.15.16:0/556455791 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f714c0cf4a0 msgr2=0x7f714c0726e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:22.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.737+0000 7f7150d1e700 1 --2- 172.21.15.16:0/556455791 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f714c0cf4a0 0x7f714c0726e0 secure :-1 s=READY pgs=2989 cs=0 l=1 rev1=1 crypto rx=0x7f713c0098f0 tx=0x7f713c009c00 comp rx=0 tx=0).stop 2023-10-16T13:48:22.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.737+0000 7f7150d1e700 1 -- 172.21.15.16:0/556455791 shutdown_connections 2023-10-16T13:48:22.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.737+0000 7f7150d1e700 1 --2- 172.21.15.16:0/556455791 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f714c0cf4a0 0x7f714c0726e0 unknown :-1 s=CLOSED pgs=2989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:22.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.737+0000 7f7150d1e700 1 -- 172.21.15.16:0/556455791 >> 172.21.15.16:0/556455791 conn(0x7f714c0b1fd0 msgr2=0x7f714c0b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:22.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.737+0000 7f7150d1e700 1 -- 172.21.15.16:0/556455791 shutdown_connections 2023-10-16T13:48:22.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.737+0000 7f7150d1e700 1 -- 172.21.15.16:0/556455791 wait complete. 2023-10-16T13:48:22.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.738+0000 7f7150d1e700 1 Processor -- start 2023-10-16T13:48:22.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.738+0000 7f7150d1e700 1 -- start start 2023-10-16T13:48:22.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.739+0000 7f7150d1e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f714c0b5a00 0x7f714c0c7510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:22.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.739+0000 7f7150d1e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f714c0cf4a0 0x7f714c1477c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:22.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.739+0000 7f7150d1e700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f714c0cf880 0x7f714c0c6680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:22.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.739+0000 7f7150d1e700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f714c0c7a50 con 0x7f714c0b5a00 2023-10-16T13:48:22.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.739+0000 7f7150d1e700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f714c0c5670 con 0x7f714c0cf880 2023-10-16T13:48:22.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.739+0000 7f7150d1e700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f714c0c58c0 con 0x7f714c0cf4a0 2023-10-16T13:48:22.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.739+0000 7f714a59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f714c0b5a00 0x7f714c0c7510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:22.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.739+0000 7f714a59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f714c0b5a00 0x7f714c0c7510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57428/0 (socket says 172.21.15.16:57428) 2023-10-16T13:48:22.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.739+0000 7f714a59c700 1 -- 172.21.15.16:0/1052786141 learned_addr learned my addr 172.21.15.16:0/1052786141 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:22.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.740+0000 7f714a59c700 1 -- 172.21.15.16:0/1052786141 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f714c0cf4a0 msgr2=0x7f714c1477c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:22.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.740+0000 7f7149d9b700 1 --2- 172.21.15.16:0/1052786141 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f714c0cf4a0 0x7f714c1477c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:22.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.740+0000 7f714ad9d700 1 --2- 172.21.15.16:0/1052786141 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f714c0cf880 0x7f714c0c6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:22.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.740+0000 7f714a59c700 1 --2- 172.21.15.16:0/1052786141 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f714c0cf4a0 0x7f714c1477c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:22.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.740+0000 7f714a59c700 1 -- 172.21.15.16:0/1052786141 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f714c0cf880 msgr2=0x7f714c0c6680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:22.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.740+0000 7f714a59c700 1 --2- 172.21.15.16:0/1052786141 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f714c0cf880 0x7f714c0c6680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:22.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.740+0000 7f714a59c700 1 -- 172.21.15.16:0/1052786141 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f713c009550 con 0x7f714c0b5a00 2023-10-16T13:48:22.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.740+0000 7f7149d9b700 1 --2- 172.21.15.16:0/1052786141 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f714c0cf4a0 0x7f714c1477c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:48:22.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.740+0000 7f714a59c700 1 --2- 172.21.15.16:0/1052786141 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f714c0b5a00 0x7f714c0c7510 secure :-1 s=READY pgs=2990 cs=0 l=1 rev1=1 crypto rx=0x7f713c000c00 tx=0x7f713c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:22.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.740+0000 7f713affd700 1 -- 172.21.15.16:0/1052786141 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f713c0038a0 con 0x7f714c0b5a00 2023-10-16T13:48:22.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.740+0000 7f713affd700 1 -- 172.21.15.16:0/1052786141 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f713c003a00 con 0x7f714c0b5a00 2023-10-16T13:48:22.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.740+0000 7f7150d1e700 1 -- 172.21.15.16:0/1052786141 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f714c0c5ba0 con 0x7f714c0b5a00 2023-10-16T13:48:22.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.740+0000 7f713affd700 1 -- 172.21.15.16:0/1052786141 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f713c003db0 con 0x7f714c0b5a00 2023-10-16T13:48:22.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.740+0000 7f7150d1e700 1 -- 172.21.15.16:0/1052786141 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f714c0c6160 con 0x7f714c0b5a00 2023-10-16T13:48:22.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.741+0000 7f7150d1e700 1 -- 172.21.15.16:0/1052786141 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f714c04ebb0 con 0x7f714c0b5a00 2023-10-16T13:48:22.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.742+0000 7f713affd700 1 -- 172.21.15.16:0/1052786141 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f713c01b5c0 con 0x7f714c0b5a00 2023-10-16T13:48:22.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.742+0000 7f713affd700 1 --2- 172.21.15.16:0/1052786141 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f712c06b170 0x7f712c06d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:22.752 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.742+0000 7f713affd700 1 -- 172.21.15.16:0/1052786141 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(261..261 src has 1..261) v4 ==== 5305+0+0 (secure 0 0 0) 0x7f713c08ca30 con 0x7f714c0b5a00 2023-10-16T13:48:22.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.742+0000 7f7149d9b700 1 --2- 172.21.15.16:0/1052786141 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f712c06b170 0x7f712c06d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:22.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.743+0000 7f7149d9b700 1 --2- 172.21.15.16:0/1052786141 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f712c06b170 0x7f712c06d630 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f714c0b57f0 tx=0x7f7140009040 comp rx=0 tx=0).ready entity=mgr.12678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:22.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.747+0000 7f713affd700 1 -- 172.21.15.16:0/1052786141 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f713c0579a0 con 0x7f714c0b5a00 2023-10-16T13:48:22.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.954+0000 7f7150d1e700 1 -- 172.21.15.16:0/1052786141 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f714c0afab0 con 0x7f714c0b5a00 2023-10-16T13:48:22.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.954+0000 7f713affd700 1 -- 172.21.15.16:0/1052786141 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v261) v1 ==== 84+0+65 (secure 0 0 0) 0x7f713c029060 con 0x7f714c0b5a00 2023-10-16T13:48:22.957 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:48:22.957 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2023-10-16T13:48:22.958 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:48:22.958 INFO:teuthology.orchestra.run.smithi016.stdout: "poolnum": 1, 2023-10-16T13:48:22.958 INFO:teuthology.orchestra.run.smithi016.stdout: "poolname": ".mgr" 2023-10-16T13:48:22.958 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:48:22.958 INFO:teuthology.orchestra.run.smithi016.stdout:] 2023-10-16T13:48:22.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.956+0000 7f7150d1e700 1 -- 172.21.15.16:0/1052786141 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f712c06b170 msgr2=0x7f712c06d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:22.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.956+0000 7f7150d1e700 1 --2- 172.21.15.16:0/1052786141 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f712c06b170 0x7f712c06d630 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f714c0b57f0 tx=0x7f7140009040 comp rx=0 tx=0).stop 2023-10-16T13:48:22.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.956+0000 7f7150d1e700 1 -- 172.21.15.16:0/1052786141 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f714c0b5a00 msgr2=0x7f714c0c7510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:22.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.956+0000 7f7150d1e700 1 --2- 172.21.15.16:0/1052786141 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f714c0b5a00 0x7f714c0c7510 secure :-1 s=READY pgs=2990 cs=0 l=1 rev1=1 crypto rx=0x7f713c000c00 tx=0x7f713c004750 comp rx=0 tx=0).stop 2023-10-16T13:48:22.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.956+0000 7f7150d1e700 1 -- 172.21.15.16:0/1052786141 shutdown_connections 2023-10-16T13:48:22.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.956+0000 7f7150d1e700 1 --2- 172.21.15.16:0/1052786141 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f712c06b170 0x7f712c06d630 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:22.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.956+0000 7f7150d1e700 1 --2- 172.21.15.16:0/1052786141 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f714c0cf880 0x7f714c0c6680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:22.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.956+0000 7f7150d1e700 1 --2- 172.21.15.16:0/1052786141 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f714c0b5a00 0x7f714c0c7510 unknown :-1 s=CLOSED pgs=2990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:22.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.956+0000 7f7150d1e700 1 --2- 172.21.15.16:0/1052786141 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f714c0cf4a0 0x7f714c1477c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:22.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.956+0000 7f7150d1e700 1 -- 172.21.15.16:0/1052786141 >> 172.21.15.16:0/1052786141 conn(0x7f714c0b1fd0 msgr2=0x7f714c0afe40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:22.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.956+0000 7f7150d1e700 1 -- 172.21.15.16:0/1052786141 shutdown_connections 2023-10-16T13:48:22.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:22.956+0000 7f7150d1e700 1 -- 172.21.15.16:0/1052786141 wait complete. 2023-10-16T13:48:22.971 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-16T13:48:22.972 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-16T13:48:23.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.098+0000 7f65a7cdf700 1 Processor -- start 2023-10-16T13:48:23.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.098+0000 7f65a7cdf700 1 -- start start 2023-10-16T13:48:23.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.099+0000 7f65a7cdf700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f65a015e0b0 0x7f65a006d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:23.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.099+0000 7f65a7cdf700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65a00cb870 con 0x7f65a00cf600 2023-10-16T13:48:23.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.099+0000 7f65a7cdf700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65a006e040 con 0x7f65a015a3d0 2023-10-16T13:48:23.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.099+0000 7f65a7cdf700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65a015f0c0 con 0x7f65a015e0b0 2023-10-16T13:48:23.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.099+0000 7f65a627c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f65a015e0b0 0x7f65a006d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:23.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.099+0000 7f65a627c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f65a015e0b0 0x7f65a006d910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57440/0 (socket says 172.21.15.16:57440) 2023-10-16T13:48:23.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.099+0000 7f65a627c700 1 -- 172.21.15.16:0/2331771768 learned_addr learned my addr 172.21.15.16:0/2331771768 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:23.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.100+0000 7f65a627c700 1 -- 172.21.15.16:0/2331771768 >> v1:172.21.15.49:6789/0 conn(0x7f65a015a3d0 legacy=0x7f65a00ceef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:23.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.100+0000 7f65a627c700 1 -- 172.21.15.16:0/2331771768 >> v1:172.21.15.16:6789/0 conn(0x7f65a00cf600 legacy=0x7f65a00cbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:23.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.100+0000 7f65a627c700 1 -- 172.21.15.16:0/2331771768 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65a006e320 con 0x7f65a015e0b0 2023-10-16T13:48:23.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.100+0000 7f65a627c700 1 --2- 172.21.15.16:0/2331771768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65a015e0b0 0x7f65a006d910 secure :-1 s=READY pgs=2992 cs=0 l=1 rev1=1 crypto rx=0x7f6598009e50 tx=0x7f659800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b120400c341b6a58 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:23.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.100+0000 7f65a4a79700 1 -- 172.21.15.16:0/2331771768 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f659800e070 con 0x7f65a015e0b0 2023-10-16T13:48:23.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.100+0000 7f65a4a79700 1 -- 172.21.15.16:0/2331771768 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6598003a20 con 0x7f65a015e0b0 2023-10-16T13:48:23.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.100+0000 7f65a4a79700 1 -- 172.21.15.16:0/2331771768 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6598003dd0 con 0x7f65a015e0b0 2023-10-16T13:48:23.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.100+0000 7f65a7cdf700 1 -- 172.21.15.16:0/2331771768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65a015e0b0 msgr2=0x7f65a006d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:23.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.100+0000 7f65a7cdf700 1 --2- 172.21.15.16:0/2331771768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65a015e0b0 0x7f65a006d910 secure :-1 s=READY pgs=2992 cs=0 l=1 rev1=1 crypto rx=0x7f6598009e50 tx=0x7f659800b330 comp rx=0 tx=0).stop 2023-10-16T13:48:23.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.100+0000 7f65a7cdf700 1 -- 172.21.15.16:0/2331771768 shutdown_connections 2023-10-16T13:48:23.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.100+0000 7f65a7cdf700 1 --2- 172.21.15.16:0/2331771768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65a015e0b0 0x7f65a006d910 unknown :-1 s=CLOSED pgs=2992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:23.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.100+0000 7f65a7cdf700 1 -- 172.21.15.16:0/2331771768 >> 172.21.15.16:0/2331771768 conn(0x7f65a006f430 msgr2=0x7f65a006f810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:23.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.101+0000 7f65a7cdf700 1 -- 172.21.15.16:0/2331771768 shutdown_connections 2023-10-16T13:48:23.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.101+0000 7f65a7cdf700 1 -- 172.21.15.16:0/2331771768 wait complete. 2023-10-16T13:48:23.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.102+0000 7f65a7cdf700 1 Processor -- start 2023-10-16T13:48:23.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.102+0000 7f65a7cdf700 1 -- start start 2023-10-16T13:48:23.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.102+0000 7f65a7cdf700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f65a00cf600 0x7f65a00ae950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:23.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.102+0000 7f65a7cdf700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65a015a3d0 0x7f65a00b3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:23.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.102+0000 7f65a7cdf700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f65a015e0b0 0x7f65a006a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:23.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.102+0000 7f65a7cdf700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65a00aee90 con 0x7f65a015a3d0 2023-10-16T13:48:23.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.102+0000 7f65a7cdf700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65a00b5640 con 0x7f65a015e0b0 2023-10-16T13:48:23.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.102+0000 7f65a7cdf700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65a00b4a30 con 0x7f65a00cf600 2023-10-16T13:48:23.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.103+0000 7f65a527a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65a015a3d0 0x7f65a00b3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:23.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.103+0000 7f65a527a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65a015a3d0 0x7f65a00b3f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57444/0 (socket says 172.21.15.16:57444) 2023-10-16T13:48:23.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.103+0000 7f65a527a700 1 -- 172.21.15.16:0/3252896256 learned_addr learned my addr 172.21.15.16:0/3252896256 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:23.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.103+0000 7f65a5a7b700 1 --2- 172.21.15.16:0/3252896256 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f65a00cf600 0x7f65a00ae950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:23.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.103+0000 7f65a627c700 1 --2- 172.21.15.16:0/3252896256 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f65a015e0b0 0x7f65a006a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:23.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.103+0000 7f65a527a700 1 -- 172.21.15.16:0/3252896256 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f65a00cf600 msgr2=0x7f65a00ae950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:23.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.103+0000 7f65a527a700 1 --2- 172.21.15.16:0/3252896256 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f65a00cf600 0x7f65a00ae950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:23.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.103+0000 7f65a527a700 1 -- 172.21.15.16:0/3252896256 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f65a015e0b0 msgr2=0x7f65a006a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:23.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.103+0000 7f65a527a700 1 --2- 172.21.15.16:0/3252896256 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f65a015e0b0 0x7f65a006a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:23.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.103+0000 7f65a527a700 1 -- 172.21.15.16:0/3252896256 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6598009b30 con 0x7f65a015a3d0 2023-10-16T13:48:23.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.103+0000 7f65a527a700 1 --2- 172.21.15.16:0/3252896256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65a015a3d0 0x7f65a00b3f70 secure :-1 s=READY pgs=2993 cs=0 l=1 rev1=1 crypto rx=0x7f659c00c490 tx=0x7f659c00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:23.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.103+0000 7f65967fc700 1 -- 172.21.15.16:0/3252896256 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f659c0052d0 con 0x7f65a015a3d0 2023-10-16T13:48:23.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.103+0000 7f65967fc700 1 -- 172.21.15.16:0/3252896256 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f659c005450 con 0x7f65a015a3d0 2023-10-16T13:48:23.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.103+0000 7f65a7cdf700 1 -- 172.21.15.16:0/3252896256 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65a00b4d10 con 0x7f65a015a3d0 2023-10-16T13:48:23.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.103+0000 7f65967fc700 1 -- 172.21.15.16:0/3252896256 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f659c006aa0 con 0x7f65a015a3d0 2023-10-16T13:48:23.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.103+0000 7f65a7cdf700 1 -- 172.21.15.16:0/3252896256 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65a00b52d0 con 0x7f65a015a3d0 2023-10-16T13:48:23.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.104+0000 7f65a7cdf700 1 -- 172.21.15.16:0/3252896256 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65a00b5490 con 0x7f65a015a3d0 2023-10-16T13:48:23.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.105+0000 7f65967fc700 1 -- 172.21.15.16:0/3252896256 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f659c003680 con 0x7f65a015a3d0 2023-10-16T13:48:23.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.105+0000 7f65967fc700 1 --2- 172.21.15.16:0/3252896256 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f658406b140 0x7f658406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:23.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.105+0000 7f65967fc700 1 -- 172.21.15.16:0/3252896256 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(261..261 src has 1..261) v4 ==== 5305+0+0 (secure 0 0 0) 0x7f659c088a50 con 0x7f65a015a3d0 2023-10-16T13:48:23.115 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.105+0000 7f65a5a7b700 1 --2- 172.21.15.16:0/3252896256 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f658406b140 0x7f658406d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:23.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.106+0000 7f65a5a7b700 1 --2- 172.21.15.16:0/3252896256 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f658406b140 0x7f658406d600 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6590004f30 tx=0x7f6590009040 comp rx=0 tx=0).ready entity=mgr.12678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:23.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.110+0000 7f65967fc700 1 -- 172.21.15.16:0/3252896256 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f659c053a00 con 0x7f65a015a3d0 2023-10-16T13:48:23.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.313+0000 7f65a7cdf700 1 -- 172.21.15.16:0/3252896256 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f65a004ec20 con 0x7f65a015a3d0 2023-10-16T13:48:23.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.399+0000 7f65967fc700 1 -- 172.21.15.16:0/3252896256 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v262) v1 ==== 138+0+0 (secure 0 0 0) 0x7f659c057020 con 0x7f65a015a3d0 2023-10-16T13:48:23.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.401+0000 7f65a7cdf700 1 -- 172.21.15.16:0/3252896256 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f658406b140 msgr2=0x7f658406d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:23.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.401+0000 7f65a7cdf700 1 --2- 172.21.15.16:0/3252896256 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f658406b140 0x7f658406d600 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6590004f30 tx=0x7f6590009040 comp rx=0 tx=0).stop 2023-10-16T13:48:23.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.401+0000 7f65a7cdf700 1 -- 172.21.15.16:0/3252896256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65a015a3d0 msgr2=0x7f65a00b3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:23.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.401+0000 7f65a7cdf700 1 --2- 172.21.15.16:0/3252896256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65a015a3d0 0x7f65a00b3f70 secure :-1 s=READY pgs=2993 cs=0 l=1 rev1=1 crypto rx=0x7f659c00c490 tx=0x7f659c00c850 comp rx=0 tx=0).stop 2023-10-16T13:48:23.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.401+0000 7f65a7cdf700 1 -- 172.21.15.16:0/3252896256 shutdown_connections 2023-10-16T13:48:23.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.401+0000 7f65a7cdf700 1 --2- 172.21.15.16:0/3252896256 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f658406b140 0x7f658406d600 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:23.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.401+0000 7f65a7cdf700 1 --2- 172.21.15.16:0/3252896256 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f65a015e0b0 0x7f65a006a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:23.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.401+0000 7f65a7cdf700 1 --2- 172.21.15.16:0/3252896256 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f65a00cf600 0x7f65a00ae950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:23.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.401+0000 7f65a7cdf700 1 --2- 172.21.15.16:0/3252896256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65a015a3d0 0x7f65a00b3f70 unknown :-1 s=CLOSED pgs=2993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:23.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.402+0000 7f65a7cdf700 1 -- 172.21.15.16:0/3252896256 >> 172.21.15.16:0/3252896256 conn(0x7f65a006f430 msgr2=0x7f65a00cde60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:23.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.402+0000 7f65a7cdf700 1 -- 172.21.15.16:0/3252896256 shutdown_connections 2023-10-16T13:48:23.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.402+0000 7f65a7cdf700 1 -- 172.21.15.16:0/3252896256 wait complete. 2023-10-16T13:48:23.407 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' created 2023-10-16T13:48:23.419 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-16T13:48:23.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.550+0000 7fcb36d97700 1 Processor -- start 2023-10-16T13:48:23.552 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.550+0000 7fcb36d97700 1 -- start start 2023-10-16T13:48:23.554 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.550+0000 7fcb36d97700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcb30072b30 0x7fcb30071a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:23.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.550+0000 7fcb36d97700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcb300caa40 0x7fcb300cae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:23.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.550+0000 7fcb36d97700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb300cb3d0 con 0x7fcb3015bb40 2023-10-16T13:48:23.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.550+0000 7fcb36d97700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb3006ea20 con 0x7fcb30072b30 2023-10-16T13:48:23.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.551+0000 7fcb36d97700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb3015c040 con 0x7fcb300caa40 2023-10-16T13:48:23.555 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.551+0000 7fcb34b33700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcb30072b30 0x7fcb30071a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:23.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.551+0000 7fcb34b33700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcb30072b30 0x7fcb30071a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57450/0 (socket says 172.21.15.16:57450) 2023-10-16T13:48:23.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.551+0000 7fcb34b33700 1 -- 172.21.15.16:0/4208837535 learned_addr learned my addr 172.21.15.16:0/4208837535 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:23.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.551+0000 7fcb35334700 1 --2- 172.21.15.16:0/4208837535 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcb300caa40 0x7fcb300cae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:23.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.551+0000 7fcb34b33700 1 -- 172.21.15.16:0/4208837535 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcb300caa40 msgr2=0x7fcb300cae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:23.556 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.551+0000 7fcb34b33700 1 --2- 172.21.15.16:0/4208837535 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcb300caa40 0x7fcb300cae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:23.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.551+0000 7fcb34b33700 1 -- 172.21.15.16:0/4208837535 >> v1:172.21.15.49:6789/0 conn(0x7fcb3015bb40 legacy=0x7fcb3006e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:23.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.551+0000 7fcb34b33700 1 -- 172.21.15.16:0/4208837535 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb3015c320 con 0x7fcb30072b30 2023-10-16T13:48:23.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.551+0000 7fcb34b33700 1 --2- 172.21.15.16:0/4208837535 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb30072b30 0x7fcb30071a70 secure :-1 s=READY pgs=2994 cs=0 l=1 rev1=1 crypto rx=0x7fcb240098f0 tx=0x7fcb24009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c075fcb5826f42df server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:23.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.551+0000 7fcb2f7fe700 1 -- 172.21.15.16:0/4208837535 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb24004230 con 0x7fcb30072b30 2023-10-16T13:48:23.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.551+0000 7fcb2f7fe700 1 -- 172.21.15.16:0/4208837535 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb24004390 con 0x7fcb30072b30 2023-10-16T13:48:23.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.551+0000 7fcb2f7fe700 1 -- 172.21.15.16:0/4208837535 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb240036a0 con 0x7fcb30072b30 2023-10-16T13:48:23.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.552+0000 7fcb36d97700 1 -- 172.21.15.16:0/4208837535 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb30072b30 msgr2=0x7fcb30071a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:23.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.552+0000 7fcb36d97700 1 --2- 172.21.15.16:0/4208837535 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb30072b30 0x7fcb30071a70 secure :-1 s=READY pgs=2994 cs=0 l=1 rev1=1 crypto rx=0x7fcb240098f0 tx=0x7fcb24009c00 comp rx=0 tx=0).stop 2023-10-16T13:48:23.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.552+0000 7fcb36d97700 1 -- 172.21.15.16:0/4208837535 shutdown_connections 2023-10-16T13:48:23.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.552+0000 7fcb36d97700 1 --2- 172.21.15.16:0/4208837535 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcb300caa40 0x7fcb300cae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:23.558 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.552+0000 7fcb36d97700 1 --2- 172.21.15.16:0/4208837535 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb30072b30 0x7fcb30071a70 unknown :-1 s=CLOSED pgs=2994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:23.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.552+0000 7fcb36d97700 1 -- 172.21.15.16:0/4208837535 >> 172.21.15.16:0/4208837535 conn(0x7fcb300b1f40 msgr2=0x7fcb300b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:23.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.552+0000 7fcb36d97700 1 -- 172.21.15.16:0/4208837535 shutdown_connections 2023-10-16T13:48:23.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.552+0000 7fcb36d97700 1 -- 172.21.15.16:0/4208837535 wait complete. 2023-10-16T13:48:23.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.553+0000 7fcb36d97700 1 Processor -- start 2023-10-16T13:48:23.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.553+0000 7fcb36d97700 1 -- start start 2023-10-16T13:48:23.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.554+0000 7fcb36d97700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcb30072b30 0x7fcb301587f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:23.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.554+0000 7fcb36d97700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcb300caa40 0x7fcb30156d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:23.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.554+0000 7fcb36d97700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb3015bb40 0x7fcb301f2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:23.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.554+0000 7fcb36d97700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb301e6740 con 0x7fcb3015bb40 2023-10-16T13:48:23.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.554+0000 7fcb36d97700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb301e5e10 con 0x7fcb300caa40 2023-10-16T13:48:23.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.554+0000 7fcb36d97700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb301e60f0 con 0x7fcb30072b30 2023-10-16T13:48:23.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.554+0000 7fcb35334700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb3015bb40 0x7fcb301f2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:23.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.554+0000 7fcb35334700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb3015bb40 0x7fcb301f2b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57460/0 (socket says 172.21.15.16:57460) 2023-10-16T13:48:23.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.554+0000 7fcb35334700 1 -- 172.21.15.16:0/3681470494 learned_addr learned my addr 172.21.15.16:0/3681470494 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:23.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.554+0000 7fcb34b33700 1 --2- 172.21.15.16:0/3681470494 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcb30072b30 0x7fcb301587f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:23.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.554+0000 7fcb2ffff700 1 --2- 172.21.15.16:0/3681470494 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcb300caa40 0x7fcb30156d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:23.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.554+0000 7fcb35334700 1 -- 172.21.15.16:0/3681470494 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcb30072b30 msgr2=0x7fcb301587f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:23.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.554+0000 7fcb35334700 1 --2- 172.21.15.16:0/3681470494 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcb30072b30 0x7fcb301587f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:23.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.554+0000 7fcb35334700 1 -- 172.21.15.16:0/3681470494 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcb300caa40 msgr2=0x7fcb30156d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:23.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.554+0000 7fcb35334700 1 --2- 172.21.15.16:0/3681470494 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcb300caa40 0x7fcb30156d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:23.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.554+0000 7fcb35334700 1 -- 172.21.15.16:0/3681470494 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb24009550 con 0x7fcb3015bb40 2023-10-16T13:48:23.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.554+0000 7fcb2ffff700 1 --2- 172.21.15.16:0/3681470494 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcb300caa40 0x7fcb30156d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:48:23.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.555+0000 7fcb35334700 1 --2- 172.21.15.16:0/3681470494 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb3015bb40 0x7fcb301f2b70 secure :-1 s=READY pgs=2995 cs=0 l=1 rev1=1 crypto rx=0x7fcb28007e60 tx=0x7fcb2800e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:23.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.555+0000 7fcb2d7fa700 1 -- 172.21.15.16:0/3681470494 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb2800ede0 con 0x7fcb3015bb40 2023-10-16T13:48:23.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.555+0000 7fcb2d7fa700 1 -- 172.21.15.16:0/3681470494 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb28014420 con 0x7fcb3015bb40 2023-10-16T13:48:23.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.555+0000 7fcb36d97700 1 -- 172.21.15.16:0/3681470494 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb301e63d0 con 0x7fcb3015bb40 2023-10-16T13:48:23.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.555+0000 7fcb2d7fa700 1 -- 172.21.15.16:0/3681470494 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb280147d0 con 0x7fcb3015bb40 2023-10-16T13:48:23.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.555+0000 7fcb36d97700 1 -- 172.21.15.16:0/3681470494 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb301572d0 con 0x7fcb3015bb40 2023-10-16T13:48:23.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.556+0000 7fcb36d97700 1 -- 172.21.15.16:0/3681470494 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb301e68b0 con 0x7fcb3015bb40 2023-10-16T13:48:23.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.557+0000 7fcb2d7fa700 1 -- 172.21.15.16:0/3681470494 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fcb28014930 con 0x7fcb3015bb40 2023-10-16T13:48:23.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.557+0000 7fcb2d7fa700 1 --2- 172.21.15.16:0/3681470494 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fcb1406b140 0x7fcb1406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:23.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.557+0000 7fcb2d7fa700 1 -- 172.21.15.16:0/3681470494 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(262..262 src has 1..262) v4 ==== 5675+0+0 (secure 0 0 0) 0x7fcb2808ae00 con 0x7fcb3015bb40 2023-10-16T13:48:23.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.557+0000 7fcb34b33700 1 --2- 172.21.15.16:0/3681470494 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fcb1406b140 0x7fcb1406d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:23.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.558+0000 7fcb34b33700 1 --2- 172.21.15.16:0/3681470494 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fcb1406b140 0x7fcb1406d600 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fcb24009130 tx=0x7fcb24008030 comp rx=0 tx=0).ready entity=mgr.12678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:23.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.562+0000 7fcb2d7fa700 1 -- 172.21.15.16:0/3681470494 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcb28055d20 con 0x7fcb3015bb40 2023-10-16T13:48:23.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:23.765+0000 7fcb36d97700 1 -- 172.21.15.16:0/3681470494 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fcb3004ebb0 con 0x7fcb3015bb40 2023-10-16T13:48:24.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.401+0000 7fcb2d7fa700 1 -- 172.21.15.16:0/3681470494 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v263) v1 ==== 170+0+0 (secure 0 0 0) 0x7fcb28059340 con 0x7fcb3015bb40 2023-10-16T13:48:24.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.405+0000 7fcb36d97700 1 -- 172.21.15.16:0/3681470494 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fcb1406b140 msgr2=0x7fcb1406d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:24.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.405+0000 7fcb36d97700 1 --2- 172.21.15.16:0/3681470494 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fcb1406b140 0x7fcb1406d600 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fcb24009130 tx=0x7fcb24008030 comp rx=0 tx=0).stop 2023-10-16T13:48:24.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.405+0000 7fcb36d97700 1 -- 172.21.15.16:0/3681470494 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb3015bb40 msgr2=0x7fcb301f2b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:24.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.405+0000 7fcb36d97700 1 --2- 172.21.15.16:0/3681470494 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb3015bb40 0x7fcb301f2b70 secure :-1 s=READY pgs=2995 cs=0 l=1 rev1=1 crypto rx=0x7fcb28007e60 tx=0x7fcb2800e3c0 comp rx=0 tx=0).stop 2023-10-16T13:48:24.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.406+0000 7fcb36d97700 1 -- 172.21.15.16:0/3681470494 shutdown_connections 2023-10-16T13:48:24.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.406+0000 7fcb36d97700 1 --2- 172.21.15.16:0/3681470494 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fcb1406b140 0x7fcb1406d600 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:24.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.406+0000 7fcb36d97700 1 --2- 172.21.15.16:0/3681470494 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb3015bb40 0x7fcb301f2b70 unknown :-1 s=CLOSED pgs=2995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:24.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.406+0000 7fcb36d97700 1 --2- 172.21.15.16:0/3681470494 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcb30072b30 0x7fcb301587f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:24.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.406+0000 7fcb36d97700 1 --2- 172.21.15.16:0/3681470494 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcb300caa40 0x7fcb30156d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:24.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.406+0000 7fcb36d97700 1 -- 172.21.15.16:0/3681470494 >> 172.21.15.16:0/3681470494 conn(0x7fcb300b1f40 msgr2=0x7fcb300afeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:24.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.406+0000 7fcb36d97700 1 -- 172.21.15.16:0/3681470494 shutdown_connections 2023-10-16T13:48:24.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.406+0000 7fcb36d97700 1 -- 172.21.15.16:0/3681470494 wait complete. 2023-10-16T13:48:24.416 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' created 2023-10-16T13:48:24.434 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-16T13:48:24.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.566+0000 7f0b066dd700 1 Processor -- start 2023-10-16T13:48:24.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.566+0000 7f0b066dd700 1 -- start start 2023-10-16T13:48:24.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.567+0000 7f0b066dd700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f0b0006a7e0 0x7f0b0006ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:24.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.567+0000 7f0b066dd700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b0015ecb0 con 0x7f0b0015b2c0 2023-10-16T13:48:24.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.567+0000 7f0b066dd700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b0015b6a0 con 0x7f0b0006e570 2023-10-16T13:48:24.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.567+0000 7f0b066dd700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b0006b190 con 0x7f0b0006a7e0 2023-10-16T13:48:24.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.567+0000 7f0afffff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0b0015b2c0 0x7f0b00161340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46738/0 (socket says 172.21.15.16:46738) 2023-10-16T13:48:24.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.567+0000 7f0afffff700 1 -- 172.21.15.16:0/1824017241 learned_addr learned my addr 172.21.15.16:0/1824017241 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:48:24.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.567+0000 7f0b04c7a700 1 --2- 172.21.15.16:0/1824017241 >> v2:172.21.15.49:3300/0 conn(0x7f0b0006a7e0 0x7f0b0006ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:24.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.567+0000 7f0afeffd700 1 -- 172.21.15.16:0/1824017241 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3795599450 0 0) 0x7f0b0015ecb0 con 0x7f0b0015b2c0 2023-10-16T13:48:24.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.567+0000 7f0afeffd700 1 -- 172.21.15.16:0/1824017241 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ae80035b0 con 0x7f0b0015b2c0 2023-10-16T13:48:24.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.567+0000 7f0afeffd700 1 -- 172.21.15.16:0/1824017241 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1836226878 0 0) 0x7f0ae80035b0 con 0x7f0b0015b2c0 2023-10-16T13:48:24.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.567+0000 7f0afeffd700 1 -- 172.21.15.16:0/1824017241 >> v2:172.21.15.49:3300/0 conn(0x7f0b0006a7e0 msgr2=0x7f0b0006ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:24.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.567+0000 7f0afeffd700 1 --2- 172.21.15.16:0/1824017241 >> v2:172.21.15.49:3300/0 conn(0x7f0b0006a7e0 0x7f0b0006ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:24.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.567+0000 7f0afeffd700 1 -- 172.21.15.16:0/1824017241 >> v1:172.21.15.49:6789/0 conn(0x7f0b0006e570 legacy=0x7f0b0015e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:24.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.567+0000 7f0afeffd700 1 -- 172.21.15.16:0/1824017241 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b00161b20 con 0x7f0b0015b2c0 2023-10-16T13:48:24.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.567+0000 7f0afeffd700 1 -- 172.21.15.16:0/1824017241 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f0af0001670 con 0x7f0b0015b2c0 2023-10-16T13:48:24.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.568+0000 7f0afeffd700 1 -- 172.21.15.16:0/1824017241 >> v1:172.21.15.16:6789/0 conn(0x7f0b0015b2c0 legacy=0x7f0b00161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:24.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.568+0000 7f0afeffd700 1 --2- 172.21.15.16:0/1824017241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ae8003a00 0x7f0ae8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:24.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.568+0000 7f0afeffd700 1 --2- 172.21.15.16:0/1824017241 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0ae8004850 0x7f0ae8004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:24.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.568+0000 7f0afeffd700 1 --2- 172.21.15.16:0/1824017241 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0ae8005210 0x7f0ae800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:24.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.568+0000 7f0afeffd700 1 -- 172.21.15.16:0/1824017241 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b0006b190 con 0x7f0ae8003a00 2023-10-16T13:48:24.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.568+0000 7f0afeffd700 1 -- 172.21.15.16:0/1824017241 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ae800c1b0 con 0x7f0ae8005210 2023-10-16T13:48:24.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.568+0000 7f0afeffd700 1 -- 172.21.15.16:0/1824017241 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ae800c370 con 0x7f0ae8004850 2023-10-16T13:48:24.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.568+0000 7f0afffff700 1 --2- 172.21.15.16:0/1824017241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ae8003a00 0x7f0ae8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:24.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.568+0000 7f0b04c7a700 1 --2- 172.21.15.16:0/1824017241 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0ae8005210 0x7f0ae800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:24.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.568+0000 7f0aff7fe700 1 --2- 172.21.15.16:0/1824017241 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0ae8004850 0x7f0ae8004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:24.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.568+0000 7f0afffff700 1 -- 172.21.15.16:0/1824017241 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0ae8004850 msgr2=0x7f0ae8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:24.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.568+0000 7f0afffff700 1 --2- 172.21.15.16:0/1824017241 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0ae8004850 0x7f0ae8004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:24.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.568+0000 7f0afffff700 1 -- 172.21.15.16:0/1824017241 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0ae8005210 msgr2=0x7f0ae800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:24.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.568+0000 7f0afffff700 1 --2- 172.21.15.16:0/1824017241 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0ae8005210 0x7f0ae800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:24.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.568+0000 7f0b04c7a700 1 -- 172.21.15.16:0/1824017241 reap_dead start 2023-10-16T13:48:24.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.568+0000 7f0afffff700 1 -- 172.21.15.16:0/1824017241 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ae80035b0 con 0x7f0ae8003a00 2023-10-16T13:48:24.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.568+0000 7f0aff7fe700 1 --2- 172.21.15.16:0/1824017241 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0ae8004850 0x7f0ae8004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:48:24.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.568+0000 7f0afffff700 1 --2- 172.21.15.16:0/1824017241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ae8003a00 0x7f0ae8003e20 secure :-1 s=READY pgs=2997 cs=0 l=1 rev1=1 crypto rx=0x7f0af000c240 tx=0x7f0ae8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad0ea3fa8427549d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:24.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.568+0000 7f0afeffd700 1 -- 172.21.15.16:0/1824017241 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0af0007c10 con 0x7f0ae8003a00 2023-10-16T13:48:24.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.568+0000 7f0afeffd700 1 -- 172.21.15.16:0/1824017241 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0af000cd60 con 0x7f0ae8003a00 2023-10-16T13:48:24.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.568+0000 7f0afeffd700 1 -- 172.21.15.16:0/1824017241 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0af0005ba0 con 0x7f0ae8003a00 2023-10-16T13:48:24.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.569+0000 7f0b066dd700 1 -- 172.21.15.16:0/1824017241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ae8003a00 msgr2=0x7f0ae8003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:24.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.569+0000 7f0b066dd700 1 --2- 172.21.15.16:0/1824017241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ae8003a00 0x7f0ae8003e20 secure :-1 s=READY pgs=2997 cs=0 l=1 rev1=1 crypto rx=0x7f0af000c240 tx=0x7f0ae8005880 comp rx=0 tx=0).stop 2023-10-16T13:48:24.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.569+0000 7f0b066dd700 1 -- 172.21.15.16:0/1824017241 shutdown_connections 2023-10-16T13:48:24.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.569+0000 7f0b066dd700 1 --2- 172.21.15.16:0/1824017241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ae8003a00 0x7f0ae8003e20 unknown :-1 s=CLOSED pgs=2997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:24.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.569+0000 7f0b066dd700 1 -- 172.21.15.16:0/1824017241 >> 172.21.15.16:0/1824017241 conn(0x7f0b000b1f40 msgr2=0x7f0b000b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:24.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.569+0000 7f0b066dd700 1 -- 172.21.15.16:0/1824017241 shutdown_connections 2023-10-16T13:48:24.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.569+0000 7f0b066dd700 1 -- 172.21.15.16:0/1824017241 wait complete. 2023-10-16T13:48:24.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.570+0000 7f0b066dd700 1 Processor -- start 2023-10-16T13:48:24.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.570+0000 7f0b066dd700 1 -- start start 2023-10-16T13:48:24.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.571+0000 7f0b066dd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0ae8003a00 0x7f0b000af420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:24.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.571+0000 7f0b066dd700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0b0015c7a0 0x7f0b000c74c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:24.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.571+0000 7f0b066dd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b0015cc20 0x7f0b000c7a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:24.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.571+0000 7f0b066dd700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b0015dd40 con 0x7f0b0015cc20 2023-10-16T13:48:24.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.571+0000 7f0b066dd700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b0015d1c0 con 0x7f0ae8003a00 2023-10-16T13:48:24.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.571+0000 7f0b066dd700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b0015d4a0 con 0x7f0b0015c7a0 2023-10-16T13:48:24.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.571+0000 7f0b04c7a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b0015cc20 0x7f0b000c7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:24.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.571+0000 7f0b04c7a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b0015cc20 0x7f0b000c7a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57480/0 (socket says 172.21.15.16:57480) 2023-10-16T13:48:24.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.571+0000 7f0b04c7a700 1 -- 172.21.15.16:0/3883261620 learned_addr learned my addr 172.21.15.16:0/3883261620 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:24.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.571+0000 7f0afffff700 1 --2- 172.21.15.16:0/3883261620 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0ae8003a00 0x7f0b000af420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:24.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.571+0000 7f0aff7fe700 1 --2- 172.21.15.16:0/3883261620 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0b0015c7a0 0x7f0b000c74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:24.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.572+0000 7f0b04c7a700 1 -- 172.21.15.16:0/3883261620 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0b0015c7a0 msgr2=0x7f0b000c74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:24.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.572+0000 7f0b04c7a700 1 --2- 172.21.15.16:0/3883261620 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0b0015c7a0 0x7f0b000c74c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:24.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.572+0000 7f0b04c7a700 1 -- 172.21.15.16:0/3883261620 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0ae8003a00 msgr2=0x7f0b000af420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:24.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.572+0000 7f0b04c7a700 1 --2- 172.21.15.16:0/3883261620 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0ae8003a00 0x7f0b000af420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:24.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.572+0000 7f0b04c7a700 1 -- 172.21.15.16:0/3883261620 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b0015d780 con 0x7f0b0015cc20 2023-10-16T13:48:24.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.572+0000 7f0b04c7a700 1 --2- 172.21.15.16:0/3883261620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b0015cc20 0x7f0b000c7a00 secure :-1 s=READY pgs=2998 cs=0 l=1 rev1=1 crypto rx=0x7f0b0006b430 tx=0x7f0af8011ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:24.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.572+0000 7f0afcff9700 1 -- 172.21.15.16:0/3883261620 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0af800e040 con 0x7f0b0015cc20 2023-10-16T13:48:24.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.572+0000 7f0afcff9700 1 -- 172.21.15.16:0/3883261620 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0af800f040 con 0x7f0b0015cc20 2023-10-16T13:48:24.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.572+0000 7f0b066dd700 1 -- 172.21.15.16:0/3883261620 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b0015da60 con 0x7f0b0015cc20 2023-10-16T13:48:24.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.572+0000 7f0afcff9700 1 -- 172.21.15.16:0/3883261620 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0af800fbe0 con 0x7f0b0015cc20 2023-10-16T13:48:24.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.572+0000 7f0b066dd700 1 -- 172.21.15.16:0/3883261620 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b0015bbf0 con 0x7f0b0015cc20 2023-10-16T13:48:24.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.573+0000 7f0b066dd700 1 -- 172.21.15.16:0/3883261620 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b0004ebb0 con 0x7f0b0015cc20 2023-10-16T13:48:24.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.574+0000 7f0afcff9700 1 -- 172.21.15.16:0/3883261620 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f0af800fd40 con 0x7f0b0015cc20 2023-10-16T13:48:24.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.574+0000 7f0afcff9700 1 --2- 172.21.15.16:0/3883261620 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f0aec06b140 0x7f0aec06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:24.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.574+0000 7f0afcff9700 1 -- 172.21.15.16:0/3883261620 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(263..263 src has 1..263) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f0af808a2d0 con 0x7f0b0015cc20 2023-10-16T13:48:24.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.574+0000 7f0afffff700 1 --2- 172.21.15.16:0/3883261620 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f0aec06b140 0x7f0aec06d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:24.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.574+0000 7f0afffff700 1 --2- 172.21.15.16:0/3883261620 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f0aec06b140 0x7f0aec06d600 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0af0000c00 tx=0x7f0af0007000 comp rx=0 tx=0).ready entity=mgr.12678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:24.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.579+0000 7f0afcff9700 1 -- 172.21.15.16:0/3883261620 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0af8054d30 con 0x7f0b0015cc20 2023-10-16T13:48:24.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:24.831+0000 7f0b066dd700 1 -- 172.21.15.16:0/3883261620 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f0b000ca720 con 0x7f0b0015cc20 2023-10-16T13:48:25.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.433+0000 7f0afcff9700 1 -- 172.21.15.16:0/3883261620 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 35 and data pool 36 v142) v1 ==== 174+0+0 (secure 0 0 0) 0x7f0af8013310 con 0x7f0b0015cc20 2023-10-16T13:48:25.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.446+0000 7f0b066dd700 1 -- 172.21.15.16:0/3883261620 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f0aec06b140 msgr2=0x7f0aec06d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:25.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.446+0000 7f0b066dd700 1 --2- 172.21.15.16:0/3883261620 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f0aec06b140 0x7f0aec06d600 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0af0000c00 tx=0x7f0af0007000 comp rx=0 tx=0).stop 2023-10-16T13:48:25.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.446+0000 7f0b066dd700 1 -- 172.21.15.16:0/3883261620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b0015cc20 msgr2=0x7f0b000c7a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:25.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.446+0000 7f0b066dd700 1 --2- 172.21.15.16:0/3883261620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b0015cc20 0x7f0b000c7a00 secure :-1 s=READY pgs=2998 cs=0 l=1 rev1=1 crypto rx=0x7f0b0006b430 tx=0x7f0af8011ed0 comp rx=0 tx=0).stop 2023-10-16T13:48:25.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.447+0000 7f0b066dd700 1 -- 172.21.15.16:0/3883261620 shutdown_connections 2023-10-16T13:48:25.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.447+0000 7f0b066dd700 1 --2- 172.21.15.16:0/3883261620 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f0aec06b140 0x7f0aec06d600 secure :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0af0000c00 tx=0x7f0af0007000 comp rx=0 tx=0).stop 2023-10-16T13:48:25.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.447+0000 7f0b066dd700 1 --2- 172.21.15.16:0/3883261620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b0015cc20 0x7f0b000c7a00 unknown :-1 s=CLOSED pgs=2998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:25.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.447+0000 7f0b066dd700 1 --2- 172.21.15.16:0/3883261620 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0ae8003a00 0x7f0b000af420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:25.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.447+0000 7f0b066dd700 1 --2- 172.21.15.16:0/3883261620 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0b0015c7a0 0x7f0b000c74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:25.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.447+0000 7f0b066dd700 1 -- 172.21.15.16:0/3883261620 >> 172.21.15.16:0/3883261620 conn(0x7f0b000b1f40 msgr2=0x7f0b000b1470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:25.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.447+0000 7f0b066dd700 1 -- 172.21.15.16:0/3883261620 shutdown_connections 2023-10-16T13:48:25.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.448+0000 7f0b066dd700 1 -- 172.21.15.16:0/3883261620 wait complete. 2023-10-16T13:48:25.454 INFO:teuthology.orchestra.run.smithi016.stderr:new fs with metadata pool 35 and data pool 36 2023-10-16T13:48:25.469 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:48:25.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.598+0000 7f36e1c7b700 1 Processor -- start 2023-10-16T13:48:25.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.598+0000 7f36e1c7b700 1 -- start start 2023-10-16T13:48:25.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.598+0000 7f36e1c7b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f36dc15b280 0x7f36dc1613e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:25.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.598+0000 7f36e1c7b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f36dc0cc8a0 0x7f36dc15e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:25.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.598+0000 7f36e1c7b700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f36dc06a7e0 0x7f36dc06ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:25.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.598+0000 7f36e1c7b700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36dc06b170 con 0x7f36dc15b280 2023-10-16T13:48:25.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.598+0000 7f36e1c7b700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36dc161c30 con 0x7f36dc06a7e0 2023-10-16T13:48:25.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.598+0000 7f36e1c7b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36dc161f10 con 0x7f36dc0cc8a0 2023-10-16T13:48:25.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.599+0000 7f36db7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f36dc15b280 0x7f36dc1613e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:25.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.599+0000 7f36db7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f36dc15b280 0x7f36dc1613e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57484/0 (socket says 172.21.15.16:57484) 2023-10-16T13:48:25.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.599+0000 7f36db7fe700 1 -- 172.21.15.16:0/1201609570 learned_addr learned my addr 172.21.15.16:0/1201609570 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:25.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.599+0000 7f36db7fe700 1 -- 172.21.15.16:0/1201609570 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f36dc0cc8a0 msgr2=0x7f36dc15e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:25.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.599+0000 7f36daffd700 1 --2- 172.21.15.16:0/1201609570 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f36dc0cc8a0 0x7f36dc15e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:25.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.599+0000 7f36dbfff700 1 --2- 172.21.15.16:0/1201609570 >> v2:172.21.15.49:3300/0 conn(0x7f36dc06a7e0 0x7f36dc06ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:25.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.599+0000 7f36db7fe700 1 --2- 172.21.15.16:0/1201609570 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f36dc0cc8a0 0x7f36dc15e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:25.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.599+0000 7f36db7fe700 1 -- 172.21.15.16:0/1201609570 >> v2:172.21.15.49:3300/0 conn(0x7f36dc06a7e0 msgr2=0x7f36dc06ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:25.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.599+0000 7f36db7fe700 1 --2- 172.21.15.16:0/1201609570 >> v2:172.21.15.49:3300/0 conn(0x7f36dc06a7e0 0x7f36dc06ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:25.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.599+0000 7f36db7fe700 1 -- 172.21.15.16:0/1201609570 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36dc1621f0 con 0x7f36dc15b280 2023-10-16T13:48:25.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.599+0000 7f36dbfff700 1 --2- 172.21.15.16:0/1201609570 >> v2:172.21.15.49:3300/0 conn(0x7f36dc06a7e0 0x7f36dc06ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:48:25.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.599+0000 7f36daffd700 1 --2- 172.21.15.16:0/1201609570 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f36dc0cc8a0 0x7f36dc15e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:48:25.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.599+0000 7f36db7fe700 1 --2- 172.21.15.16:0/1201609570 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36dc15b280 0x7f36dc1613e0 secure :-1 s=READY pgs=2999 cs=0 l=1 rev1=1 crypto rx=0x7f36dc071970 tx=0x7f36cc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3c1a7398e8f88b2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:25.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.599+0000 7f36da7fc700 1 -- 172.21.15.16:0/1201609570 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36cc00b870 con 0x7f36dc15b280 2023-10-16T13:48:25.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.599+0000 7f36da7fc700 1 -- 172.21.15.16:0/1201609570 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36cc00b9d0 con 0x7f36dc15b280 2023-10-16T13:48:25.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.599+0000 7f36da7fc700 1 -- 172.21.15.16:0/1201609570 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36cc00bd60 con 0x7f36dc15b280 2023-10-16T13:48:25.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.600+0000 7f36e1c7b700 1 -- 172.21.15.16:0/1201609570 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36dc15b280 msgr2=0x7f36dc1613e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:25.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.600+0000 7f36e1c7b700 1 --2- 172.21.15.16:0/1201609570 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36dc15b280 0x7f36dc1613e0 secure :-1 s=READY pgs=2999 cs=0 l=1 rev1=1 crypto rx=0x7f36dc071970 tx=0x7f36cc009bd0 comp rx=0 tx=0).stop 2023-10-16T13:48:25.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.600+0000 7f36e1c7b700 1 -- 172.21.15.16:0/1201609570 shutdown_connections 2023-10-16T13:48:25.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.600+0000 7f36e1c7b700 1 --2- 172.21.15.16:0/1201609570 >> v2:172.21.15.49:3300/0 conn(0x7f36dc06a7e0 0x7f36dc06ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:25.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.600+0000 7f36e1c7b700 1 --2- 172.21.15.16:0/1201609570 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36dc15b280 0x7f36dc1613e0 unknown :-1 s=CLOSED pgs=2999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:25.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.600+0000 7f36e1c7b700 1 --2- 172.21.15.16:0/1201609570 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f36dc0cc8a0 0x7f36dc15e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:25.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.600+0000 7f36e1c7b700 1 -- 172.21.15.16:0/1201609570 >> 172.21.15.16:0/1201609570 conn(0x7f36dc0b1fd0 msgr2=0x7f36dc0b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:25.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.600+0000 7f36e1c7b700 1 -- 172.21.15.16:0/1201609570 shutdown_connections 2023-10-16T13:48:25.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.600+0000 7f36e1c7b700 1 -- 172.21.15.16:0/1201609570 wait complete. 2023-10-16T13:48:25.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.601+0000 7f36e1c7b700 1 Processor -- start 2023-10-16T13:48:25.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.602+0000 7f36e1c7b700 1 -- start start 2023-10-16T13:48:25.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.603+0000 7f36e1c7b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f36dc06a7e0 0x7f36dc1f2920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:25.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.603+0000 7f36e1c7b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f36dc0cc8a0 0x7f36dc0cb2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:25.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.603+0000 7f36e1c7b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36dc15b280 0x7f36dc0cb830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:25.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.603+0000 7f36e1c7b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36dc14f430 con 0x7f36dc15b280 2023-10-16T13:48:25.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.603+0000 7f36e1c7b700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36dc0c8050 con 0x7f36dc06a7e0 2023-10-16T13:48:25.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.603+0000 7f36e1c7b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36dc0c8330 con 0x7f36dc0cc8a0 2023-10-16T13:48:25.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.603+0000 7f36dbfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36dc15b280 0x7f36dc0cb830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:25.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.603+0000 7f36dbfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36dc15b280 0x7f36dc0cb830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57494/0 (socket says 172.21.15.16:57494) 2023-10-16T13:48:25.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.603+0000 7f36dbfff700 1 -- 172.21.15.16:0/1677774627 learned_addr learned my addr 172.21.15.16:0/1677774627 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:25.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.603+0000 7f36db7fe700 1 --2- 172.21.15.16:0/1677774627 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f36dc06a7e0 0x7f36dc1f2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:25.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.603+0000 7f36daffd700 1 --2- 172.21.15.16:0/1677774627 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f36dc0cc8a0 0x7f36dc0cb2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:25.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.603+0000 7f36dbfff700 1 -- 172.21.15.16:0/1677774627 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f36dc0cc8a0 msgr2=0x7f36dc0cb2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:25.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.603+0000 7f36dbfff700 1 --2- 172.21.15.16:0/1677774627 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f36dc0cc8a0 0x7f36dc0cb2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:25.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.603+0000 7f36dbfff700 1 -- 172.21.15.16:0/1677774627 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f36dc06a7e0 msgr2=0x7f36dc1f2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:25.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.603+0000 7f36dbfff700 1 --2- 172.21.15.16:0/1677774627 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f36dc06a7e0 0x7f36dc1f2920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:25.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.603+0000 7f36dbfff700 1 -- 172.21.15.16:0/1677774627 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36cc009550 con 0x7f36dc15b280 2023-10-16T13:48:25.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.603+0000 7f36daffd700 1 --2- 172.21.15.16:0/1677774627 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f36dc0cc8a0 0x7f36dc0cb2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:48:25.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.603+0000 7f36dbfff700 1 --2- 172.21.15.16:0/1677774627 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36dc15b280 0x7f36dc0cb830 secure :-1 s=READY pgs=3000 cs=0 l=1 rev1=1 crypto rx=0x7f36d400cef0 tx=0x7f36d400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:25.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.603+0000 7f36c3fff700 1 -- 172.21.15.16:0/1677774627 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36d400add0 con 0x7f36dc15b280 2023-10-16T13:48:25.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.603+0000 7f36c3fff700 1 -- 172.21.15.16:0/1677774627 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36d4015440 con 0x7f36dc15b280 2023-10-16T13:48:25.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.604+0000 7f36c3fff700 1 -- 172.21.15.16:0/1677774627 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36d40157f0 con 0x7f36dc15b280 2023-10-16T13:48:25.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.604+0000 7f36e1c7b700 1 -- 172.21.15.16:0/1677774627 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36dc0c8610 con 0x7f36dc15b280 2023-10-16T13:48:25.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.604+0000 7f36e1c7b700 1 -- 172.21.15.16:0/1677774627 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36dc0c8bd0 con 0x7f36dc15b280 2023-10-16T13:48:25.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.605+0000 7f36e1c7b700 1 -- 172.21.15.16:0/1677774627 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36dc04ebb0 con 0x7f36dc15b280 2023-10-16T13:48:25.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.605+0000 7f36c3fff700 1 -- 172.21.15.16:0/1677774627 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f36d400d430 con 0x7f36dc15b280 2023-10-16T13:48:25.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.605+0000 7f36c3fff700 1 --2- 172.21.15.16:0/1677774627 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f36c806b140 0x7f36c806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:25.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.606+0000 7f36c3fff700 1 -- 172.21.15.16:0/1677774627 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(264..264 src has 1..264) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f36d408a0d0 con 0x7f36dc15b280 2023-10-16T13:48:25.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.606+0000 7f36db7fe700 1 --2- 172.21.15.16:0/1677774627 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f36c806b140 0x7f36c806d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:25.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.606+0000 7f36db7fe700 1 --2- 172.21.15.16:0/1677774627 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f36c806b140 0x7f36c806d600 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f36cc009c10 tx=0x7f36cc008030 comp rx=0 tx=0).ready entity=mgr.12678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:25.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.610+0000 7f36c3fff700 1 -- 172.21.15.16:0/1677774627 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f36d4054cf0 con 0x7f36dc15b280 2023-10-16T13:48:25.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.814+0000 7f36e1c7b700 1 -- 172.21.15.16:0/1677774627 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f36dc0d06d0 con 0x7f36dc15b280 2023-10-16T13:48:25.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.815+0000 7f36c3fff700 1 -- 172.21.15.16:0/1677774627 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v264) v1 ==== 74+0+16830 (secure 0 0 0) 0x7f36d4058310 con 0x7f36dc15b280 2023-10-16T13:48:25.818 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:48:25.818 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":264,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:48:24.832284+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":36,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_metadata","create_time":"2023-10-16T13:48:23.348668+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"264","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_data","create_time":"2023-10-16T13:48:23.799968+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"264","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:48:25.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.817+0000 7f36e1c7b700 1 -- 172.21.15.16:0/1677774627 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f36c806b140 msgr2=0x7f36c806d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:25.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.817+0000 7f36e1c7b700 1 --2- 172.21.15.16:0/1677774627 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f36c806b140 0x7f36c806d600 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f36cc009c10 tx=0x7f36cc008030 comp rx=0 tx=0).stop 2023-10-16T13:48:25.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.817+0000 7f36e1c7b700 1 -- 172.21.15.16:0/1677774627 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36dc15b280 msgr2=0x7f36dc0cb830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:25.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.817+0000 7f36e1c7b700 1 --2- 172.21.15.16:0/1677774627 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36dc15b280 0x7f36dc0cb830 secure :-1 s=READY pgs=3000 cs=0 l=1 rev1=1 crypto rx=0x7f36d400cef0 tx=0x7f36d400a340 comp rx=0 tx=0).stop 2023-10-16T13:48:25.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.817+0000 7f36e1c7b700 1 -- 172.21.15.16:0/1677774627 shutdown_connections 2023-10-16T13:48:25.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.817+0000 7f36e1c7b700 1 --2- 172.21.15.16:0/1677774627 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f36c806b140 0x7f36c806d600 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:25.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.817+0000 7f36e1c7b700 1 --2- 172.21.15.16:0/1677774627 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36dc15b280 0x7f36dc0cb830 unknown :-1 s=CLOSED pgs=3000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:25.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.817+0000 7f36e1c7b700 1 --2- 172.21.15.16:0/1677774627 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f36dc06a7e0 0x7f36dc1f2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:25.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.817+0000 7f36e1c7b700 1 --2- 172.21.15.16:0/1677774627 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f36dc0cc8a0 0x7f36dc0cb2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:25.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.817+0000 7f36e1c7b700 1 -- 172.21.15.16:0/1677774627 >> 172.21.15.16:0/1677774627 conn(0x7f36dc0b1fd0 msgr2=0x7f36dc0d0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:25.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.817+0000 7f36e1c7b700 1 -- 172.21.15.16:0/1677774627 shutdown_connections 2023-10-16T13:48:25.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.817+0000 7f36e1c7b700 1 -- 172.21.15.16:0/1677774627 wait complete. 2023-10-16T13:48:25.833 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:48:25.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.955+0000 7f681b58d700 1 Processor -- start 2023-10-16T13:48:25.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.955+0000 7f681b58d700 1 -- start start 2023-10-16T13:48:25.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.956+0000 7f681b58d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f68140b2b50 0x7f68140d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:25.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.956+0000 7f681b58d700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68140d1210 con 0x7f68140ce380 2023-10-16T13:48:25.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.956+0000 7f681b58d700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68140cbc30 con 0x7f68140cdfa0 2023-10-16T13:48:25.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.956+0000 7f681b58d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68140cbf10 con 0x7f68140b2b50 2023-10-16T13:48:25.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.956+0000 7f6819b2a700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f68140ce380 0x7f68140d09e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46776/0 (socket says 172.21.15.16:46776) 2023-10-16T13:48:25.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.956+0000 7f6819b2a700 1 -- 172.21.15.16:0/213728375 learned_addr learned my addr 172.21.15.16:0/213728375 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:48:25.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.956+0000 7f6818b28700 1 --2- 172.21.15.16:0/213728375 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f68140b2b50 0x7f68140d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:25.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.956+0000 7f680bfff700 1 -- 172.21.15.16:0/213728375 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2244549493 0 0) 0x7f68140d1210 con 0x7f68140ce380 2023-10-16T13:48:25.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.956+0000 7f680bfff700 1 -- 172.21.15.16:0/213728375 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68000035b0 con 0x7f68140ce380 2023-10-16T13:48:25.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.956+0000 7f680bfff700 1 -- 172.21.15.16:0/213728375 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3426783155 0 0) 0x7f68140cbc30 con 0x7f68140cdfa0 2023-10-16T13:48:25.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.956+0000 7f680bfff700 1 -- 172.21.15.16:0/213728375 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68140d1210 con 0x7f68140cdfa0 2023-10-16T13:48:25.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.956+0000 7f6818b28700 1 -- 172.21.15.16:0/213728375 >> v1:172.21.15.49:6789/0 conn(0x7f68140cdfa0 legacy=0x7f68140b2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:25.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.956+0000 7f680bfff700 1 -- 172.21.15.16:0/213728375 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3387302782 0 0) 0x7f68000035b0 con 0x7f68140ce380 2023-10-16T13:48:25.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.956+0000 7f6818b28700 1 -- 172.21.15.16:0/213728375 >> v1:172.21.15.16:6789/0 conn(0x7f68140ce380 legacy=0x7f68140d09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:25.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.956+0000 7f6818b28700 1 -- 172.21.15.16:0/213728375 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68140cc1f0 con 0x7f68140b2b50 2023-10-16T13:48:25.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.957+0000 7f6818b28700 1 --2- 172.21.15.16:0/213728375 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f68140b2b50 0x7f68140d04a0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f6804009870 tx=0x7f6804009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76ce0803c5aa1f44 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:25.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.957+0000 7f680bfff700 1 -- 172.21.15.16:0/213728375 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f680c0019f0 con 0x7f68140b2b50 2023-10-16T13:48:25.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.957+0000 7f680bfff700 1 -- 172.21.15.16:0/213728375 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6804004030 con 0x7f68140b2b50 2023-10-16T13:48:25.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.957+0000 7f680bfff700 1 -- 172.21.15.16:0/213728375 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68040043c0 con 0x7f68140b2b50 2023-10-16T13:48:25.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.957+0000 7f681b58d700 1 -- 172.21.15.16:0/213728375 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f68140b2b50 msgr2=0x7f68140d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:25.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.957+0000 7f681b58d700 1 --2- 172.21.15.16:0/213728375 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f68140b2b50 0x7f68140d04a0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f6804009870 tx=0x7f6804009b80 comp rx=0 tx=0).stop 2023-10-16T13:48:25.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.957+0000 7f681b58d700 1 -- 172.21.15.16:0/213728375 shutdown_connections 2023-10-16T13:48:25.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.957+0000 7f681b58d700 1 --2- 172.21.15.16:0/213728375 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f68140b2b50 0x7f68140d04a0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:25.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.957+0000 7f681b58d700 1 -- 172.21.15.16:0/213728375 >> 172.21.15.16:0/213728375 conn(0x7f68140d14b0 msgr2=0x7f68140d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:25.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.958+0000 7f681b58d700 1 -- 172.21.15.16:0/213728375 shutdown_connections 2023-10-16T13:48:25.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.958+0000 7f681b58d700 1 -- 172.21.15.16:0/213728375 wait complete. 2023-10-16T13:48:25.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.959+0000 7f681b58d700 1 Processor -- start 2023-10-16T13:48:25.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.959+0000 7f681b58d700 1 -- start start 2023-10-16T13:48:25.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.960+0000 7f681b58d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68140b2b50 0x7f681415dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:25.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.960+0000 7f681b58d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f68140cdfa0 0x7f68141545c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:25.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.960+0000 7f681b58d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f68140ce380 0x7f6814155840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:25.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.960+0000 7f681b58d700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68140c51f0 con 0x7f68140b2b50 2023-10-16T13:48:25.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.960+0000 7f681b58d700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68140c6010 con 0x7f68140cdfa0 2023-10-16T13:48:25.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.960+0000 7f681b58d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f681415e2b0 con 0x7f68140ce380 2023-10-16T13:48:25.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.960+0000 7f6819329700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68140b2b50 0x7f681415dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:25.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.960+0000 7f6819329700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68140b2b50 0x7f681415dd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57504/0 (socket says 172.21.15.16:57504) 2023-10-16T13:48:25.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.960+0000 7f6819329700 1 -- 172.21.15.16:0/306843719 learned_addr learned my addr 172.21.15.16:0/306843719 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:25.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.960+0000 7f6818b28700 1 --2- 172.21.15.16:0/306843719 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f68140cdfa0 0x7f68141545c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:25.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.960+0000 7f6819b2a700 1 --2- 172.21.15.16:0/306843719 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f68140ce380 0x7f6814155840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:25.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.960+0000 7f6819329700 1 -- 172.21.15.16:0/306843719 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f68140ce380 msgr2=0x7f6814155840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:25.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.960+0000 7f6819329700 1 --2- 172.21.15.16:0/306843719 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f68140ce380 0x7f6814155840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:25.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.960+0000 7f6819329700 1 -- 172.21.15.16:0/306843719 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f68140cdfa0 msgr2=0x7f68141545c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:25.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.960+0000 7f6819329700 1 --2- 172.21.15.16:0/306843719 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f68140cdfa0 0x7f68141545c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:25.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.960+0000 7f6819329700 1 -- 172.21.15.16:0/306843719 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68040094a0 con 0x7f68140b2b50 2023-10-16T13:48:25.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.960+0000 7f6819329700 1 --2- 172.21.15.16:0/306843719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68140b2b50 0x7f681415dd70 secure :-1 s=READY pgs=3002 cs=0 l=1 rev1=1 crypto rx=0x7f681000c520 tx=0x7f681000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:25.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.960+0000 7f6809ffb700 1 -- 172.21.15.16:0/306843719 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68100052d0 con 0x7f68140b2b50 2023-10-16T13:48:25.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.960+0000 7f6809ffb700 1 -- 172.21.15.16:0/306843719 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6810005430 con 0x7f68140b2b50 2023-10-16T13:48:25.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.960+0000 7f681b58d700 1 -- 172.21.15.16:0/306843719 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68140c5400 con 0x7f68140b2b50 2023-10-16T13:48:25.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.960+0000 7f6809ffb700 1 -- 172.21.15.16:0/306843719 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68100038c0 con 0x7f68140b2b50 2023-10-16T13:48:25.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.961+0000 7f681b58d700 1 -- 172.21.15.16:0/306843719 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68140c59c0 con 0x7f68140b2b50 2023-10-16T13:48:25.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.961+0000 7f681b58d700 1 -- 172.21.15.16:0/306843719 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f681404ebb0 con 0x7f68140b2b50 2023-10-16T13:48:25.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.962+0000 7f6809ffb700 1 -- 172.21.15.16:0/306843719 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f6810003a20 con 0x7f68140b2b50 2023-10-16T13:48:25.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.962+0000 7f6809ffb700 1 --2- 172.21.15.16:0/306843719 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f67f806b140 0x7f67f806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:25.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.962+0000 7f6809ffb700 1 -- 172.21.15.16:0/306843719 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(264..264 src has 1..264) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f6810088ff0 con 0x7f68140b2b50 2023-10-16T13:48:25.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.963+0000 7f6818b28700 1 --2- 172.21.15.16:0/306843719 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f67f806b140 0x7f67f806d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:25.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.963+0000 7f6818b28700 1 --2- 172.21.15.16:0/306843719 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f67f806b140 0x7f67f806d600 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f68140b2f70 tx=0x7f6804008030 comp rx=0 tx=0).ready entity=mgr.12678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:25.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:25.967+0000 7f6809ffb700 1 -- 172.21.15.16:0/306843719 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6810053ce0 con 0x7f68140b2b50 2023-10-16T13:48:26.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.171+0000 7f681b58d700 1 -- 172.21.15.16:0/306843719 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f68140cd3e0 con 0x7f68140b2b50 2023-10-16T13:48:26.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.172+0000 7f6809ffb700 1 -- 172.21.15.16:0/306843719 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v264) v1 ==== 74+0+16830 (secure 0 0 0) 0x7f6810057300 con 0x7f68140b2b50 2023-10-16T13:48:26.175 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:48:26.175 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":264,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:48:24.832284+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":36,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_metadata","create_time":"2023-10-16T13:48:23.348668+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"264","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_data","create_time":"2023-10-16T13:48:23.799968+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"264","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:48:26.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.174+0000 7f681b58d700 1 -- 172.21.15.16:0/306843719 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f67f806b140 msgr2=0x7f67f806d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:26.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.174+0000 7f681b58d700 1 --2- 172.21.15.16:0/306843719 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f67f806b140 0x7f67f806d600 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f68140b2f70 tx=0x7f6804008030 comp rx=0 tx=0).stop 2023-10-16T13:48:26.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.174+0000 7f681b58d700 1 -- 172.21.15.16:0/306843719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68140b2b50 msgr2=0x7f681415dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:26.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.174+0000 7f681b58d700 1 --2- 172.21.15.16:0/306843719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68140b2b50 0x7f681415dd70 secure :-1 s=READY pgs=3002 cs=0 l=1 rev1=1 crypto rx=0x7f681000c520 tx=0x7f681000c8e0 comp rx=0 tx=0).stop 2023-10-16T13:48:26.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.174+0000 7f681b58d700 1 -- 172.21.15.16:0/306843719 shutdown_connections 2023-10-16T13:48:26.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.174+0000 7f681b58d700 1 --2- 172.21.15.16:0/306843719 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f67f806b140 0x7f67f806d600 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:26.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.174+0000 7f681b58d700 1 --2- 172.21.15.16:0/306843719 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f68140ce380 0x7f6814155840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:26.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.174+0000 7f681b58d700 1 --2- 172.21.15.16:0/306843719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68140b2b50 0x7f681415dd70 unknown :-1 s=CLOSED pgs=3002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:26.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.174+0000 7f681b58d700 1 --2- 172.21.15.16:0/306843719 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f68140cdfa0 0x7f68141545c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:26.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.174+0000 7f681b58d700 1 -- 172.21.15.16:0/306843719 >> 172.21.15.16:0/306843719 conn(0x7f68140d14b0 msgr2=0x7f681406c290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:26.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.174+0000 7f681b58d700 1 -- 172.21.15.16:0/306843719 shutdown_connections 2023-10-16T13:48:26.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.174+0000 7f681b58d700 1 -- 172.21.15.16:0/306843719 wait complete. 2023-10-16T13:48:26.189 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-16T13:48:26.318 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.316+0000 7f38d934b700 1 Processor -- start 2023-10-16T13:48:26.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.316+0000 7f38d934b700 1 -- start start 2023-10-16T13:48:26.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.317+0000 7f38d934b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f38d415d8c0 0x7f38d40b2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:26.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.317+0000 7f38d934b700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f38d4162390 0x7f38d40b34e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:26.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.317+0000 7f38d934b700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38d4161f00 con 0x7f38d406be40 2023-10-16T13:48:26.322 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.317+0000 7f38d934b700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38d415f180 con 0x7f38d4162390 2023-10-16T13:48:26.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.317+0000 7f38d934b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38d415ecb0 con 0x7f38d415d8c0 2023-10-16T13:48:26.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.317+0000 7f38d37fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f38d406be40 0x7f38d406c2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46790/0 (socket says 172.21.15.16:46790) 2023-10-16T13:48:26.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.317+0000 7f38d37fe700 1 -- 172.21.15.16:0/4201821358 learned_addr learned my addr 172.21.15.16:0/4201821358 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:48:26.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.317+0000 7f38d2ffd700 1 --2- 172.21.15.16:0/4201821358 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f38d415d8c0 0x7f38d40b2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:26.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.317+0000 7f38d27fc700 1 --2- 172.21.15.16:0/4201821358 >> v2:172.21.15.49:3300/0 conn(0x7f38d4162390 0x7f38d40b34e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:26.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.317+0000 7f38d1ffb700 1 -- 172.21.15.16:0/4201821358 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1422217989 0 0) 0x7f38d4161f00 con 0x7f38d406be40 2023-10-16T13:48:26.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.317+0000 7f38d1ffb700 1 -- 172.21.15.16:0/4201821358 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38bc0035b0 con 0x7f38d406be40 2023-10-16T13:48:26.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.317+0000 7f38d1ffb700 1 -- 172.21.15.16:0/4201821358 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3068259809 0 0) 0x7f38bc0035b0 con 0x7f38d406be40 2023-10-16T13:48:26.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.317+0000 7f38d1ffb700 1 -- 172.21.15.16:0/4201821358 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f38d415d8c0 msgr2=0x7f38d40b2fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:26.324 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.317+0000 7f38d1ffb700 1 --2- 172.21.15.16:0/4201821358 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f38d415d8c0 0x7f38d40b2fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:26.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.318+0000 7f38d1ffb700 1 -- 172.21.15.16:0/4201821358 >> v2:172.21.15.49:3300/0 conn(0x7f38d4162390 msgr2=0x7f38d40b34e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:26.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.318+0000 7f38d1ffb700 1 --2- 172.21.15.16:0/4201821358 >> v2:172.21.15.49:3300/0 conn(0x7f38d4162390 0x7f38d40b34e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:26.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.318+0000 7f38d1ffb700 1 -- 172.21.15.16:0/4201821358 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38d415eea0 con 0x7f38d406be40 2023-10-16T13:48:26.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.318+0000 7f38d1ffb700 1 -- 172.21.15.16:0/4201821358 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f38cc0019f0 con 0x7f38d406be40 2023-10-16T13:48:26.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.318+0000 7f38d1ffb700 1 -- 172.21.15.16:0/4201821358 >> v1:172.21.15.16:6789/0 conn(0x7f38d406be40 legacy=0x7f38d406c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:26.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.318+0000 7f38d1ffb700 1 --2- 172.21.15.16:0/4201821358 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f38bc003a30 0x7f38bc003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:26.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.318+0000 7f38d1ffb700 1 --2- 172.21.15.16:0/4201821358 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f38bc004850 0x7f38bc004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:26.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.318+0000 7f38d1ffb700 1 --2- 172.21.15.16:0/4201821358 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38bc005210 0x7f38bc00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:26.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.318+0000 7f38d1ffb700 1 -- 172.21.15.16:0/4201821358 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38d415f180 con 0x7f38bc005210 2023-10-16T13:48:26.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.318+0000 7f38d1ffb700 1 -- 172.21.15.16:0/4201821358 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38bc00bf50 con 0x7f38bc003a30 2023-10-16T13:48:26.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.318+0000 7f38d1ffb700 1 -- 172.21.15.16:0/4201821358 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38bc00c090 con 0x7f38bc004850 2023-10-16T13:48:26.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.318+0000 7f38d2ffd700 1 --2- 172.21.15.16:0/4201821358 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f38d415d8c0 0x7f38d40b2fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:48:26.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.318+0000 7f38d37fe700 1 --2- 172.21.15.16:0/4201821358 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38bc005210 0x7f38bc00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:26.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.318+0000 7f38d27fc700 1 --2- 172.21.15.16:0/4201821358 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f38bc004850 0x7f38bc004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:26.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.318+0000 7f38d2ffd700 1 --2- 172.21.15.16:0/4201821358 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f38bc003a30 0x7f38bc003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:26.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.318+0000 7f38d37fe700 1 -- 172.21.15.16:0/4201821358 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f38bc004850 msgr2=0x7f38bc004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:26.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.318+0000 7f38d37fe700 1 --2- 172.21.15.16:0/4201821358 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f38bc004850 0x7f38bc004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:26.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.318+0000 7f38d37fe700 1 -- 172.21.15.16:0/4201821358 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f38bc003a30 msgr2=0x7f38bc003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:26.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.318+0000 7f38d37fe700 1 --2- 172.21.15.16:0/4201821358 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f38bc003a30 0x7f38bc003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:26.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.318+0000 7f38d37fe700 1 -- 172.21.15.16:0/4201821358 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38bc0035b0 con 0x7f38bc005210 2023-10-16T13:48:26.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.318+0000 7f38d37fe700 1 --2- 172.21.15.16:0/4201821358 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38bc005210 0x7f38bc00ba10 secure :-1 s=READY pgs=3004 cs=0 l=1 rev1=1 crypto rx=0x7f38cc00a860 tx=0x7f38bc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84bbac0d674cb673 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:26.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.318+0000 7f38d1ffb700 1 -- 172.21.15.16:0/4201821358 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38cc00a2f0 con 0x7f38bc005210 2023-10-16T13:48:26.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.318+0000 7f38d1ffb700 1 -- 172.21.15.16:0/4201821358 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38cc012620 con 0x7f38bc005210 2023-10-16T13:48:26.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.318+0000 7f38d1ffb700 1 -- 172.21.15.16:0/4201821358 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38cc0129d0 con 0x7f38bc005210 2023-10-16T13:48:26.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.319+0000 7f38d934b700 1 -- 172.21.15.16:0/4201821358 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38bc005210 msgr2=0x7f38bc00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:26.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.319+0000 7f38d934b700 1 --2- 172.21.15.16:0/4201821358 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38bc005210 0x7f38bc00ba10 secure :-1 s=READY pgs=3004 cs=0 l=1 rev1=1 crypto rx=0x7f38cc00a860 tx=0x7f38bc005880 comp rx=0 tx=0).stop 2023-10-16T13:48:26.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.319+0000 7f38d37fe700 1 -- 172.21.15.16:0/4201821358 reap_dead start 2023-10-16T13:48:26.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.319+0000 7f38d934b700 1 -- 172.21.15.16:0/4201821358 shutdown_connections 2023-10-16T13:48:26.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.319+0000 7f38d934b700 1 -- 172.21.15.16:0/4201821358 >> 172.21.15.16:0/4201821358 conn(0x7f38d40cdfa0 msgr2=0x7f38d40ce3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:26.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.319+0000 7f38d934b700 1 -- 172.21.15.16:0/4201821358 shutdown_connections 2023-10-16T13:48:26.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.319+0000 7f38d934b700 1 -- 172.21.15.16:0/4201821358 wait complete. 2023-10-16T13:48:26.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.321+0000 7f38d934b700 1 Processor -- start 2023-10-16T13:48:26.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.321+0000 7f38d934b700 1 -- start start 2023-10-16T13:48:26.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.322+0000 7f38d934b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38bc005210 0x7f38d40c8560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:26.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.322+0000 7f38d934b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f38d40cadc0 0x7f38d40ccdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:26.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.322+0000 7f38d934b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f38d40cd300 0x7f38d4144330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:26.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.322+0000 7f38d934b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38d40cb240 con 0x7f38bc005210 2023-10-16T13:48:26.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.322+0000 7f38d934b700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38d40b4420 con 0x7f38d40cadc0 2023-10-16T13:48:26.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.322+0000 7f38d934b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38d40b6360 con 0x7f38d40cd300 2023-10-16T13:48:26.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.322+0000 7f38d2ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38bc005210 0x7f38d40c8560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:26.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.322+0000 7f38d2ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38bc005210 0x7f38d40c8560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57524/0 (socket says 172.21.15.16:57524) 2023-10-16T13:48:26.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.322+0000 7f38d2ffd700 1 -- 172.21.15.16:0/1515916254 learned_addr learned my addr 172.21.15.16:0/1515916254 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:26.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.322+0000 7f38d27fc700 1 --2- 172.21.15.16:0/1515916254 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f38d40cadc0 0x7f38d40ccdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:26.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.322+0000 7f38d37fe700 1 --2- 172.21.15.16:0/1515916254 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f38d40cd300 0x7f38d4144330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:26.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.322+0000 7f38d2ffd700 1 -- 172.21.15.16:0/1515916254 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f38d40cd300 msgr2=0x7f38d4144330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:26.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.322+0000 7f38d2ffd700 1 --2- 172.21.15.16:0/1515916254 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f38d40cd300 0x7f38d4144330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:26.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.322+0000 7f38d2ffd700 1 -- 172.21.15.16:0/1515916254 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f38d40cadc0 msgr2=0x7f38d40ccdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:26.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.322+0000 7f38d2ffd700 1 --2- 172.21.15.16:0/1515916254 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f38d40cadc0 0x7f38d40ccdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:26.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.322+0000 7f38d2ffd700 1 -- 172.21.15.16:0/1515916254 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38d40b6620 con 0x7f38bc005210 2023-10-16T13:48:26.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.322+0000 7f38d2ffd700 1 --2- 172.21.15.16:0/1515916254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38bc005210 0x7f38d40c8560 secure :-1 s=READY pgs=3005 cs=0 l=1 rev1=1 crypto rx=0x7f38c8011800 tx=0x7f38c8011bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:26.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.322+0000 7f38bb7fe700 1 -- 172.21.15.16:0/1515916254 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38c8015870 con 0x7f38bc005210 2023-10-16T13:48:26.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.322+0000 7f38bb7fe700 1 -- 172.21.15.16:0/1515916254 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38c80159d0 con 0x7f38bc005210 2023-10-16T13:48:26.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.322+0000 7f38bb7fe700 1 -- 172.21.15.16:0/1515916254 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38c8015d80 con 0x7f38bc005210 2023-10-16T13:48:26.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.322+0000 7f38d934b700 1 -- 172.21.15.16:0/1515916254 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38d40b6900 con 0x7f38bc005210 2023-10-16T13:48:26.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.323+0000 7f38d934b700 1 -- 172.21.15.16:0/1515916254 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38d40b6ec0 con 0x7f38bc005210 2023-10-16T13:48:26.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.324+0000 7f38d934b700 1 -- 172.21.15.16:0/1515916254 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38d40c8d40 con 0x7f38bc005210 2023-10-16T13:48:26.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.324+0000 7f38bb7fe700 1 -- 172.21.15.16:0/1515916254 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f38c8013430 con 0x7f38bc005210 2023-10-16T13:48:26.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.324+0000 7f38bb7fe700 1 --2- 172.21.15.16:0/1515916254 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f38c006b140 0x7f38c006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:26.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.325+0000 7f38bb7fe700 1 -- 172.21.15.16:0/1515916254 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(264..264 src has 1..264) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f38c808a060 con 0x7f38bc005210 2023-10-16T13:48:26.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.325+0000 7f38d27fc700 1 --2- 172.21.15.16:0/1515916254 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f38c006b140 0x7f38c006d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:26.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.325+0000 7f38d27fc700 1 --2- 172.21.15.16:0/1515916254 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f38c006b140 0x7f38c006d600 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f38d4145530 tx=0x7f38c4008040 comp rx=0 tx=0).ready entity=mgr.12678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:26.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.329+0000 7f38bb7fe700 1 -- 172.21.15.16:0/1515916254 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f38c8054d30 con 0x7f38bc005210 2023-10-16T13:48:26.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:26.584+0000 7f38d934b700 1 -- 172.21.15.16:0/1515916254 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f38d404ebb0 con 0x7f38bc005210 2023-10-16T13:48:27.402 INFO:tasks.ceph.mds.d.smithi049.stderr:2023-10-16T13:48:27.401+0000 7f65323c1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T13:48:27.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:27.441+0000 7f38bb7fe700 1 -- 172.21.15.16:0/1515916254 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v145) v1 ==== 118+0+0 (secure 0 0 0) 0x7f38c8058350 con 0x7f38bc005210 2023-10-16T13:48:27.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:27.444+0000 7f38d934b700 1 -- 172.21.15.16:0/1515916254 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f38c006b140 msgr2=0x7f38c006d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:27.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:27.444+0000 7f38d934b700 1 --2- 172.21.15.16:0/1515916254 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f38c006b140 0x7f38c006d600 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f38d4145530 tx=0x7f38c4008040 comp rx=0 tx=0).stop 2023-10-16T13:48:27.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:27.444+0000 7f38d934b700 1 -- 172.21.15.16:0/1515916254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38bc005210 msgr2=0x7f38d40c8560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:27.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:27.444+0000 7f38d934b700 1 --2- 172.21.15.16:0/1515916254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38bc005210 0x7f38d40c8560 secure :-1 s=READY pgs=3005 cs=0 l=1 rev1=1 crypto rx=0x7f38c8011800 tx=0x7f38c8011bc0 comp rx=0 tx=0).stop 2023-10-16T13:48:27.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:27.444+0000 7f38d934b700 1 -- 172.21.15.16:0/1515916254 shutdown_connections 2023-10-16T13:48:27.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:27.444+0000 7f38d934b700 1 --2- 172.21.15.16:0/1515916254 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f38c006b140 0x7f38c006d600 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:27.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:27.444+0000 7f38d934b700 1 --2- 172.21.15.16:0/1515916254 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f38d40cd300 0x7f38d4144330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:27.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:27.444+0000 7f38d934b700 1 --2- 172.21.15.16:0/1515916254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38bc005210 0x7f38d40c8560 unknown :-1 s=CLOSED pgs=3005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:27.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:27.444+0000 7f38d934b700 1 --2- 172.21.15.16:0/1515916254 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f38d40cadc0 0x7f38d40ccdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:27.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:27.444+0000 7f38d934b700 1 -- 172.21.15.16:0/1515916254 >> 172.21.15.16:0/1515916254 conn(0x7f38d40cdfa0 msgr2=0x7f38d40ce3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:27.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:27.444+0000 7f38d934b700 1 -- 172.21.15.16:0/1515916254 shutdown_connections 2023-10-16T13:48:27.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:27.444+0000 7f38d934b700 1 -- 172.21.15.16:0/1515916254 wait complete. 2023-10-16T13:48:27.462 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-16T13:48:27.462 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-16T13:48:27.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:27.595+0000 7f7766064700 1 Processor -- start 2023-10-16T13:48:27.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:27.596+0000 7f7766064700 1 -- start start 2023-10-16T13:48:27.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:27.596+0000 7f7766064700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7760160fe0 0x7f77600ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:27.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:27.596+0000 7f7766064700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f776006da30 0x7f776006a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:27.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:27.596+0000 7f7766064700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77601613c0 con 0x7f776006ad20 2023-10-16T13:48:27.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:27.596+0000 7f7766064700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77600cf680 con 0x7f776006da30 2023-10-16T13:48:27.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:27.596+0000 7f7766064700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f776006b1b0 con 0x7f7760160fe0 2023-10-16T13:48:27.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:27.596+0000 7f775effd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f776006da30 0x7f776006a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:27.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:27.597+0000 7f775effd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f776006da30 0x7f776006a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57534/0 (socket says 172.21.15.16:57534) 2023-10-16T13:48:27.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:27.597+0000 7f775effd700 1 -- 172.21.15.16:0/3418768626 learned_addr learned my addr 172.21.15.16:0/3418768626 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:27.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:27.597+0000 7f775f7fe700 1 --2- 172.21.15.16:0/3418768626 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7760160fe0 0x7f77600ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:27.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:27.597+0000 7f775effd700 1 -- 172.21.15.16:0/3418768626 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7760160fe0 msgr2=0x7f77600ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:27.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:27.597+0000 7f775effd700 1 --2- 172.21.15.16:0/3418768626 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7760160fe0 0x7f77600ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:27.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:27.597+0000 7f775effd700 1 -- 172.21.15.16:0/3418768626 >> v1:172.21.15.49:6789/0 conn(0x7f776006ad20 legacy=0x7f7760071a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:27.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:27.597+0000 7f775effd700 1 -- 172.21.15.16:0/3418768626 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77600adb00 con 0x7f776006da30 2023-10-16T13:48:27.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:27.597+0000 7f775f7fe700 1 --2- 172.21.15.16:0/3418768626 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7760160fe0 0x7f77600ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:48:27.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:27.597+0000 7f775effd700 1 --2- 172.21.15.16:0/3418768626 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f776006da30 0x7f776006a7e0 secure :-1 s=READY pgs=3006 cs=0 l=1 rev1=1 crypto rx=0x7f77540098f0 tx=0x7f7754009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ca8b59c5db45642 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:27.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:27.597+0000 7f775e7fc700 1 -- 172.21.15.16:0/3418768626 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7754004230 con 0x7f776006da30 2023-10-16T13:48:27.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:27.597+0000 7f775e7fc700 1 -- 172.21.15.16:0/3418768626 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7754004390 con 0x7f776006da30 2023-10-16T13:48:27.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:27.597+0000 7f775e7fc700 1 -- 172.21.15.16:0/3418768626 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77540036a0 con 0x7f776006da30 2023-10-16T13:48:27.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:27.598+0000 7f7766064700 1 -- 172.21.15.16:0/3418768626 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f776006da30 msgr2=0x7f776006a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:27.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:27.598+0000 7f7766064700 1 --2- 172.21.15.16:0/3418768626 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f776006da30 0x7f776006a7e0 secure :-1 s=READY pgs=3006 cs=0 l=1 rev1=1 crypto rx=0x7f77540098f0 tx=0x7f7754009c00 comp rx=0 tx=0).stop 2023-10-16T13:48:27.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:27.598+0000 7f7766064700 1 -- 172.21.15.16:0/3418768626 shutdown_connections 2023-10-16T13:48:27.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:27.598+0000 7f7766064700 1 --2- 172.21.15.16:0/3418768626 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7760160fe0 0x7f77600ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:27.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:27.598+0000 7f7766064700 1 --2- 172.21.15.16:0/3418768626 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f776006da30 0x7f776006a7e0 unknown :-1 s=CLOSED pgs=3006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:27.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:27.598+0000 7f7766064700 1 -- 172.21.15.16:0/3418768626 >> 172.21.15.16:0/3418768626 conn(0x7f77600b1ed0 msgr2=0x7f77600b22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:27.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:27.598+0000 7f7766064700 1 -- 172.21.15.16:0/3418768626 shutdown_connections 2023-10-16T13:48:27.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:27.598+0000 7f7766064700 1 -- 172.21.15.16:0/3418768626 wait complete. 2023-10-16T13:48:27.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:27.599+0000 7f7766064700 1 Processor -- start 2023-10-16T13:48:27.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:27.599+0000 7f7766064700 1 -- start start 2023-10-16T13:48:27.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:27.600+0000 7f7766064700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f776006ad20 0x7f77601536a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:27.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:27.600+0000 7f7766064700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f776006da30 0x7f7760158060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:27.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:27.600+0000 7f7766064700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7760160fe0 0x7f7760156d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:27.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:27.600+0000 7f7766064700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7760154710 con 0x7f7760160fe0 2023-10-16T13:48:27.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:27.600+0000 7f7766064700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77601549f0 con 0x7f776006da30 2023-10-16T13:48:27.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:27.600+0000 7f7766064700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f776015a140 con 0x7f776006ad20 2023-10-16T13:48:27.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:27.600+0000 7f775ffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7760160fe0 0x7f7760156d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:27.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:27.600+0000 7f775ffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7760160fe0 0x7f7760156d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57542/0 (socket says 172.21.15.16:57542) 2023-10-16T13:48:27.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:27.600+0000 7f775ffff700 1 -- 172.21.15.16:0/245153462 learned_addr learned my addr 172.21.15.16:0/245153462 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:27.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:27.600+0000 7f775f7fe700 1 --2- 172.21.15.16:0/245153462 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f776006ad20 0x7f77601536a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:27.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:27.600+0000 7f775effd700 1 --2- 172.21.15.16:0/245153462 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f776006da30 0x7f7760158060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:27.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:27.600+0000 7f775ffff700 1 -- 172.21.15.16:0/245153462 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f776006ad20 msgr2=0x7f77601536a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:27.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:27.600+0000 7f775ffff700 1 --2- 172.21.15.16:0/245153462 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f776006ad20 0x7f77601536a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:27.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:27.600+0000 7f775ffff700 1 -- 172.21.15.16:0/245153462 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f776006da30 msgr2=0x7f7760158060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:27.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:27.600+0000 7f775ffff700 1 --2- 172.21.15.16:0/245153462 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f776006da30 0x7f7760158060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:27.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:27.600+0000 7f775ffff700 1 -- 172.21.15.16:0/245153462 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7754009550 con 0x7f7760160fe0 2023-10-16T13:48:27.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:27.600+0000 7f775effd700 1 --2- 172.21.15.16:0/245153462 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f776006da30 0x7f7760158060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:48:27.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:27.601+0000 7f775ffff700 1 --2- 172.21.15.16:0/245153462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7760160fe0 0x7f7760156d90 secure :-1 s=READY pgs=3007 cs=0 l=1 rev1=1 crypto rx=0x7f775800cb60 tx=0x7f775800ce70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:27.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:27.601+0000 7f7747fff700 1 -- 172.21.15.16:0/245153462 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f775800aa60 con 0x7f7760160fe0 2023-10-16T13:48:27.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:27.601+0000 7f7747fff700 1 -- 172.21.15.16:0/245153462 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f775800abc0 con 0x7f7760160fe0 2023-10-16T13:48:27.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:27.601+0000 7f7747fff700 1 -- 172.21.15.16:0/245153462 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7758012420 con 0x7f7760160fe0 2023-10-16T13:48:27.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:27.601+0000 7f7766064700 1 -- 172.21.15.16:0/245153462 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f776015a420 con 0x7f7760160fe0 2023-10-16T13:48:27.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:27.601+0000 7f7766064700 1 -- 172.21.15.16:0/245153462 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77601543a0 con 0x7f7760160fe0 2023-10-16T13:48:27.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:27.602+0000 7f7766064700 1 -- 172.21.15.16:0/245153462 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7760156ab0 con 0x7f7760160fe0 2023-10-16T13:48:27.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:27.602+0000 7f7747fff700 1 -- 172.21.15.16:0/245153462 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f7758012580 con 0x7f7760160fe0 2023-10-16T13:48:27.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:27.603+0000 7f7747fff700 1 --2- 172.21.15.16:0/245153462 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f774c06b140 0x7f774c06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:27.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:27.603+0000 7f7747fff700 1 -- 172.21.15.16:0/245153462 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f7758089dc0 con 0x7f7760160fe0 2023-10-16T13:48:27.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:27.603+0000 7f775f7fe700 1 --2- 172.21.15.16:0/245153462 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f774c06b140 0x7f774c06d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:27.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:27.603+0000 7f775f7fe700 1 --2- 172.21.15.16:0/245153462 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f774c06b140 0x7f774c06d600 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f77600b5df0 tx=0x7f7750008040 comp rx=0 tx=0).ready entity=mgr.12678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:27.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:27.608+0000 7f7747fff700 1 -- 172.21.15.16:0/245153462 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7760156ab0 con 0x7f7760160fe0 2023-10-16T13:48:27.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:27.857+0000 7f7766064700 1 -- 172.21.15.16:0/245153462 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f7760156a80 con 0x7f7760160fe0 2023-10-16T13:48:28.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.442+0000 7f7747fff700 1 -- 172.21.15.16:0/245153462 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v146) v1 ==== 122+0+0 (secure 0 0 0) 0x7f77580549e0 con 0x7f7760160fe0 2023-10-16T13:48:28.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.445+0000 7f7766064700 1 -- 172.21.15.16:0/245153462 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f774c06b140 msgr2=0x7f774c06d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:28.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.445+0000 7f7766064700 1 --2- 172.21.15.16:0/245153462 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f774c06b140 0x7f774c06d600 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f77600b5df0 tx=0x7f7750008040 comp rx=0 tx=0).stop 2023-10-16T13:48:28.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.445+0000 7f7766064700 1 -- 172.21.15.16:0/245153462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7760160fe0 msgr2=0x7f7760156d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:28.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.445+0000 7f7766064700 1 --2- 172.21.15.16:0/245153462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7760160fe0 0x7f7760156d90 secure :-1 s=READY pgs=3007 cs=0 l=1 rev1=1 crypto rx=0x7f775800cb60 tx=0x7f775800ce70 comp rx=0 tx=0).stop 2023-10-16T13:48:28.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.445+0000 7f7766064700 1 -- 172.21.15.16:0/245153462 shutdown_connections 2023-10-16T13:48:28.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.445+0000 7f7766064700 1 --2- 172.21.15.16:0/245153462 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f774c06b140 0x7f774c06d600 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:28.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.445+0000 7f7766064700 1 --2- 172.21.15.16:0/245153462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7760160fe0 0x7f7760156d90 unknown :-1 s=CLOSED pgs=3007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:28.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.445+0000 7f7766064700 1 --2- 172.21.15.16:0/245153462 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f776006ad20 0x7f77601536a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:28.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.445+0000 7f7766064700 1 --2- 172.21.15.16:0/245153462 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f776006da30 0x7f7760158060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:28.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.445+0000 7f7766064700 1 -- 172.21.15.16:0/245153462 >> 172.21.15.16:0/245153462 conn(0x7f77600b1ed0 msgr2=0x7f77600af120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:28.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.446+0000 7f7766064700 1 -- 172.21.15.16:0/245153462 shutdown_connections 2023-10-16T13:48:28.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.446+0000 7f7766064700 1 -- 172.21.15.16:0/245153462 wait complete. 2023-10-16T13:48:28.463 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-16T13:48:28.464 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:48:28.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.595+0000 7fe70f8ba700 1 Processor -- start 2023-10-16T13:48:28.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.595+0000 7fe70f8ba700 1 -- start start 2023-10-16T13:48:28.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.596+0000 7fe70f8ba700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fe70806aef0 0x7fe7080b35b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:28.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.596+0000 7fe70f8ba700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe7080b2be0 0x7fe7080b3af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:28.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.596+0000 7fe70f8ba700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7080d03b0 con 0x7fe70815b760 2023-10-16T13:48:28.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.596+0000 7fe70f8ba700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe70815bb40 con 0x7fe7080b2be0 2023-10-16T13:48:28.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.596+0000 7fe70f8ba700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7080cb760 con 0x7fe70806aef0 2023-10-16T13:48:28.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.596+0000 7fe70de57700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe7080b2be0 0x7fe7080b3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:28.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.596+0000 7fe70de57700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe7080b2be0 0x7fe7080b3af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57550/0 (socket says 172.21.15.16:57550) 2023-10-16T13:48:28.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.596+0000 7fe70de57700 1 -- 172.21.15.16:0/2509558853 learned_addr learned my addr 172.21.15.16:0/2509558853 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:28.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.596+0000 7fe70ce55700 1 --2- 172.21.15.16:0/2509558853 >> v2:172.21.15.49:3300/0 conn(0x7fe70806aef0 0x7fe7080b35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:28.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.596+0000 7fe70de57700 1 -- 172.21.15.16:0/2509558853 >> v2:172.21.15.49:3300/0 conn(0x7fe70806aef0 msgr2=0x7fe7080b35b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:28.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.596+0000 7fe70de57700 1 --2- 172.21.15.16:0/2509558853 >> v2:172.21.15.49:3300/0 conn(0x7fe70806aef0 0x7fe7080b35b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:28.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.596+0000 7fe70de57700 1 -- 172.21.15.16:0/2509558853 >> v1:172.21.15.16:6789/0 conn(0x7fe70815b760 legacy=0x7fe70806a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:28.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.596+0000 7fe70de57700 1 -- 172.21.15.16:0/2509558853 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7080d0600 con 0x7fe7080b2be0 2023-10-16T13:48:28.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.596+0000 7fe70de57700 1 --2- 172.21.15.16:0/2509558853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7080b2be0 0x7fe7080b3af0 secure :-1 s=READY pgs=3009 cs=0 l=1 rev1=1 crypto rx=0x7fe70806b8c0 tx=0x7fe70000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9ca5c7aa1d52cdb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:28.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.597+0000 7fe6fffff700 1 -- 172.21.15.16:0/2509558853 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe70000e070 con 0x7fe7080b2be0 2023-10-16T13:48:28.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.597+0000 7fe6fffff700 1 -- 172.21.15.16:0/2509558853 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe70000b910 con 0x7fe7080b2be0 2023-10-16T13:48:28.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.597+0000 7fe6fffff700 1 -- 172.21.15.16:0/2509558853 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe70000bcc0 con 0x7fe7080b2be0 2023-10-16T13:48:28.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.597+0000 7fe70f8ba700 1 -- 172.21.15.16:0/2509558853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7080b2be0 msgr2=0x7fe7080b3af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:28.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.597+0000 7fe70f8ba700 1 --2- 172.21.15.16:0/2509558853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7080b2be0 0x7fe7080b3af0 secure :-1 s=READY pgs=3009 cs=0 l=1 rev1=1 crypto rx=0x7fe70806b8c0 tx=0x7fe70000b330 comp rx=0 tx=0).stop 2023-10-16T13:48:28.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.597+0000 7fe70f8ba700 1 -- 172.21.15.16:0/2509558853 shutdown_connections 2023-10-16T13:48:28.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.597+0000 7fe70f8ba700 1 --2- 172.21.15.16:0/2509558853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7080b2be0 0x7fe7080b3af0 unknown :-1 s=CLOSED pgs=3009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:28.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.597+0000 7fe70f8ba700 1 --2- 172.21.15.16:0/2509558853 >> v2:172.21.15.49:3300/0 conn(0x7fe70806aef0 0x7fe7080b35b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:28.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.597+0000 7fe70f8ba700 1 -- 172.21.15.16:0/2509558853 >> 172.21.15.16:0/2509558853 conn(0x7fe7080b1fd0 msgr2=0x7fe7080b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:28.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.597+0000 7fe70f8ba700 1 -- 172.21.15.16:0/2509558853 shutdown_connections 2023-10-16T13:48:28.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.597+0000 7fe70f8ba700 1 -- 172.21.15.16:0/2509558853 wait complete. 2023-10-16T13:48:28.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.598+0000 7fe70f8ba700 1 Processor -- start 2023-10-16T13:48:28.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.598+0000 7fe70f8ba700 1 -- start start 2023-10-16T13:48:28.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.599+0000 7fe70f8ba700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe70806aef0 0x7fe7081f0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:28.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.599+0000 7fe70f8ba700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7080b2be0 0x7fe7080c6600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:28.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.599+0000 7fe70f8ba700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe70815b760 0x7fe7080c3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:28.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.599+0000 7fe70f8ba700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe70814b360 con 0x7fe70815b760 2023-10-16T13:48:28.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.599+0000 7fe70f8ba700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7081f0970 con 0x7fe70806aef0 2023-10-16T13:48:28.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.599+0000 7fe70f8ba700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe70814d760 con 0x7fe7080b2be0 2023-10-16T13:48:28.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.599+0000 7fe70de57700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe70815b760 0x7fe7080c3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:28.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.599+0000 7fe70de57700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe70815b760 0x7fe7080c3f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57556/0 (socket says 172.21.15.16:57556) 2023-10-16T13:48:28.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.599+0000 7fe70de57700 1 -- 172.21.15.16:0/2528130248 learned_addr learned my addr 172.21.15.16:0/2528130248 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:28.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.599+0000 7fe70ce55700 1 --2- 172.21.15.16:0/2528130248 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7080b2be0 0x7fe7080c6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:28.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.599+0000 7fe70d656700 1 --2- 172.21.15.16:0/2528130248 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe70806aef0 0x7fe7081f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:28.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.600+0000 7fe70de57700 1 -- 172.21.15.16:0/2528130248 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7080b2be0 msgr2=0x7fe7080c6600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:28.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.600+0000 7fe70de57700 1 --2- 172.21.15.16:0/2528130248 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7080b2be0 0x7fe7080c6600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:28.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.600+0000 7fe70de57700 1 -- 172.21.15.16:0/2528130248 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe70806aef0 msgr2=0x7fe7081f0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:28.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.600+0000 7fe70de57700 1 --2- 172.21.15.16:0/2528130248 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe70806aef0 0x7fe7081f0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:28.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.600+0000 7fe70de57700 1 -- 172.21.15.16:0/2528130248 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe700009b30 con 0x7fe70815b760 2023-10-16T13:48:28.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.600+0000 7fe70de57700 1 --2- 172.21.15.16:0/2528130248 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe70815b760 0x7fe7080c3f70 secure :-1 s=READY pgs=3010 cs=0 l=1 rev1=1 crypto rx=0x7fe7080d0270 tx=0x7fe700004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:28.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.600+0000 7fe6fdffb700 1 -- 172.21.15.16:0/2528130248 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe700004880 con 0x7fe70815b760 2023-10-16T13:48:28.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.600+0000 7fe6fdffb700 1 -- 172.21.15.16:0/2528130248 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7000099b0 con 0x7fe70815b760 2023-10-16T13:48:28.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.600+0000 7fe70f8ba700 1 -- 172.21.15.16:0/2528130248 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe70814da40 con 0x7fe70815b760 2023-10-16T13:48:28.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.600+0000 7fe6fdffb700 1 -- 172.21.15.16:0/2528130248 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe70001c680 con 0x7fe70815b760 2023-10-16T13:48:28.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.600+0000 7fe70f8ba700 1 -- 172.21.15.16:0/2528130248 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe70814e000 con 0x7fe70815b760 2023-10-16T13:48:28.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.601+0000 7fe70f8ba700 1 -- 172.21.15.16:0/2528130248 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe70804ebb0 con 0x7fe70815b760 2023-10-16T13:48:28.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.602+0000 7fe6fdffb700 1 -- 172.21.15.16:0/2528130248 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fe70001c7e0 con 0x7fe70815b760 2023-10-16T13:48:28.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.602+0000 7fe6fdffb700 1 --2- 172.21.15.16:0/2528130248 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fe6ec06b140 0x7fe6ec06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:28.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.602+0000 7fe70d656700 1 --2- 172.21.15.16:0/2528130248 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fe6ec06b140 0x7fe6ec06d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:28.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.602+0000 7fe6fdffb700 1 -- 172.21.15.16:0/2528130248 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 6157+0+0 (secure 0 0 0) 0x7fe70008b480 con 0x7fe70815b760 2023-10-16T13:48:28.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.603+0000 7fe70d656700 1 --2- 172.21.15.16:0/2528130248 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fe6ec06b140 0x7fe6ec06d600 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe6f8007fd0 tx=0x7fe6f8009040 comp rx=0 tx=0).ready entity=mgr.12678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:28.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.607+0000 7fe6fdffb700 1 -- 172.21.15.16:0/2528130248 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe700055f90 con 0x7fe70815b760 2023-10-16T13:48:28.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.856+0000 7fe70f8ba700 1 -- 172.21.15.16:0/2528130248 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe708068b00 con 0x7fe70815b760 2023-10-16T13:48:28.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.856+0000 7fe6fdffb700 1 -- 172.21.15.16:0/2528130248 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 146 v146) v1 ==== 95+0+4926 (secure 0 0 0) 0x7fe7000595b0 con 0x7fe70815b760 2023-10-16T13:48:28.859 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:48:28.859 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":146,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10112,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/712583564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":712583564},{"type":"v1","addr":"172.21.15.16:6837","nonce":712583564}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":141},{"gid":11836,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/3551114031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":3551114031},{"type":"v1","addr":"172.21.15.49:6834","nonce":3551114031}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":141},{"gid":13050,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/2135643757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2135643757},{"type":"v1","addr":"172.21.15.16:6836","nonce":2135643757}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":140}],"filesystems":[{"mdsmap":{"epoch":146,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:48:24.832614+0000","modified":"2023-10-16T13:48:27.858697+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":13068},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13068":{"gid":13068,"name":"d","rank":0,"incarnation":142,"state":"up:active","state_seq":3,"addr":"172.21.15.49:6835/1985453659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":1985453659},{"type":"v1","addr":"172.21.15.49:6835","nonce":1985453659}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[36],"metadata_pool":35,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":17}]} 2023-10-16T13:48:28.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.858+0000 7fe70f8ba700 1 -- 172.21.15.16:0/2528130248 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fe6ec06b140 msgr2=0x7fe6ec06d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:28.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.858+0000 7fe70f8ba700 1 --2- 172.21.15.16:0/2528130248 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fe6ec06b140 0x7fe6ec06d600 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe6f8007fd0 tx=0x7fe6f8009040 comp rx=0 tx=0).stop 2023-10-16T13:48:28.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.858+0000 7fe70f8ba700 1 -- 172.21.15.16:0/2528130248 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe70815b760 msgr2=0x7fe7080c3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:28.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.858+0000 7fe70f8ba700 1 --2- 172.21.15.16:0/2528130248 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe70815b760 0x7fe7080c3f70 secure :-1 s=READY pgs=3010 cs=0 l=1 rev1=1 crypto rx=0x7fe7080d0270 tx=0x7fe700004ad0 comp rx=0 tx=0).stop 2023-10-16T13:48:28.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.858+0000 7fe70f8ba700 1 -- 172.21.15.16:0/2528130248 shutdown_connections 2023-10-16T13:48:28.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.858+0000 7fe70f8ba700 1 --2- 172.21.15.16:0/2528130248 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fe6ec06b140 0x7fe6ec06d600 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:28.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.858+0000 7fe70f8ba700 1 --2- 172.21.15.16:0/2528130248 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe70815b760 0x7fe7080c3f70 unknown :-1 s=CLOSED pgs=3010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:28.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.858+0000 7fe70f8ba700 1 --2- 172.21.15.16:0/2528130248 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe70806aef0 0x7fe7081f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:28.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.858+0000 7fe70f8ba700 1 --2- 172.21.15.16:0/2528130248 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe7080b2be0 0x7fe7080c6600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:28.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.858+0000 7fe70f8ba700 1 -- 172.21.15.16:0/2528130248 >> 172.21.15.16:0/2528130248 conn(0x7fe7080b1fd0 msgr2=0x7fe7080719a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:28.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.858+0000 7fe70f8ba700 1 -- 172.21.15.16:0/2528130248 shutdown_connections 2023-10-16T13:48:28.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.858+0000 7fe70f8ba700 1 -- 172.21.15.16:0/2528130248 wait complete. 2023-10-16T13:48:28.864 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 146 2023-10-16T13:48:28.873 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:48:28.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.996+0000 7ff618a85700 1 Processor -- start 2023-10-16T13:48:28.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.996+0000 7ff618a85700 1 -- start start 2023-10-16T13:48:29.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.997+0000 7ff618a85700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7ff6140ce4f0 0x7ff6140d0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:29.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.997+0000 7ff618a85700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff61415ba10 0x7ff6140cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:29.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.997+0000 7ff618a85700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6140b0970 con 0x7ff61415b630 2023-10-16T13:48:29.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.997+0000 7ff618a85700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6140cd690 con 0x7ff6140ce4f0 2023-10-16T13:48:29.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.997+0000 7ff618a85700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6140cd970 con 0x7ff61415ba10 2023-10-16T13:48:29.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.997+0000 7ff61259c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff61415b630 0x7ff6140cdde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46834/0 (socket says 172.21.15.16:46834) 2023-10-16T13:48:29.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.997+0000 7ff61259c700 1 -- 172.21.15.16:0/1543262313 learned_addr learned my addr 172.21.15.16:0/1543262313 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:48:29.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.997+0000 7ff612d9d700 1 --2- 172.21.15.16:0/1543262313 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff61415ba10 0x7ff6140cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:29.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.997+0000 7ff611d9b700 1 --2- 172.21.15.16:0/1543262313 >> v2:172.21.15.49:3300/0 conn(0x7ff6140ce4f0 0x7ff6140d0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:29.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.997+0000 7ff61159a700 1 -- 172.21.15.16:0/1543262313 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2277391173 0 0) 0x7ff6140b0970 con 0x7ff61415b630 2023-10-16T13:48:29.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.997+0000 7ff61159a700 1 -- 172.21.15.16:0/1543262313 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5fc0035b0 con 0x7ff61415b630 2023-10-16T13:48:29.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.997+0000 7ff61159a700 1 -- 172.21.15.16:0/1543262313 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 748270527 0 0) 0x7ff5fc0035b0 con 0x7ff61415b630 2023-10-16T13:48:29.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.997+0000 7ff612d9d700 1 -- 172.21.15.16:0/1543262313 >> v2:172.21.15.49:3300/0 conn(0x7ff6140ce4f0 msgr2=0x7ff6140d0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:29.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.997+0000 7ff612d9d700 1 --2- 172.21.15.16:0/1543262313 >> v2:172.21.15.49:3300/0 conn(0x7ff6140ce4f0 0x7ff6140d0380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:29.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.997+0000 7ff612d9d700 1 -- 172.21.15.16:0/1543262313 >> v1:172.21.15.16:6789/0 conn(0x7ff61415b630 legacy=0x7ff6140cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:29.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.997+0000 7ff612d9d700 1 -- 172.21.15.16:0/1543262313 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6140b0c30 con 0x7ff61415ba10 2023-10-16T13:48:29.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.997+0000 7ff611d9b700 1 --2- 172.21.15.16:0/1543262313 >> v2:172.21.15.49:3300/0 conn(0x7ff6140ce4f0 0x7ff6140d0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:48:29.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.998+0000 7ff612d9d700 1 --2- 172.21.15.16:0/1543262313 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff61415ba10 0x7ff6140cf290 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7ff60c009d90 tx=0x7ff60c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=54be6ee1f1cb10da server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:29.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.998+0000 7ff61159a700 1 -- 172.21.15.16:0/1543262313 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff60c00e070 con 0x7ff61415ba10 2023-10-16T13:48:29.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.998+0000 7ff61159a700 1 -- 172.21.15.16:0/1543262313 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff60c003a20 con 0x7ff61415ba10 2023-10-16T13:48:29.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.998+0000 7ff61159a700 1 -- 172.21.15.16:0/1543262313 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff604001670 con 0x7ff61415ba10 2023-10-16T13:48:29.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.998+0000 7ff618a85700 1 -- 172.21.15.16:0/1543262313 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff61415ba10 msgr2=0x7ff6140cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:29.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.998+0000 7ff618a85700 1 --2- 172.21.15.16:0/1543262313 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff61415ba10 0x7ff6140cf290 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7ff60c009d90 tx=0x7ff60c00b330 comp rx=0 tx=0).stop 2023-10-16T13:48:29.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.998+0000 7ff618a85700 1 -- 172.21.15.16:0/1543262313 shutdown_connections 2023-10-16T13:48:29.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.998+0000 7ff618a85700 1 --2- 172.21.15.16:0/1543262313 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff61415ba10 0x7ff6140cf290 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:29.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.998+0000 7ff618a85700 1 --2- 172.21.15.16:0/1543262313 >> v2:172.21.15.49:3300/0 conn(0x7ff6140ce4f0 0x7ff6140d0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:29.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.998+0000 7ff618a85700 1 -- 172.21.15.16:0/1543262313 >> 172.21.15.16:0/1543262313 conn(0x7ff6140d14b0 msgr2=0x7ff6140d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:29.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.998+0000 7ff618a85700 1 -- 172.21.15.16:0/1543262313 shutdown_connections 2023-10-16T13:48:29.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:28.998+0000 7ff618a85700 1 -- 172.21.15.16:0/1543262313 wait complete. 2023-10-16T13:48:29.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.000+0000 7ff618a85700 1 Processor -- start 2023-10-16T13:48:29.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.000+0000 7ff618a85700 1 -- start start 2023-10-16T13:48:29.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.001+0000 7ff618a85700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6140ce4f0 0x7ff61414ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:29.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.001+0000 7ff618a85700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff61415b630 0x7ff61406d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:29.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.001+0000 7ff618a85700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff61415ba10 0x7ff6140af1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:29.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.001+0000 7ff618a85700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6140b70e0 con 0x7ff6140ce4f0 2023-10-16T13:48:29.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.001+0000 7ff618a85700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6140b46e0 con 0x7ff61415b630 2023-10-16T13:48:29.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.001+0000 7ff618a85700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6140b6840 con 0x7ff61415ba10 2023-10-16T13:48:29.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.001+0000 7ff61259c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6140ce4f0 0x7ff61414ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:29.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.001+0000 7ff61259c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6140ce4f0 0x7ff61414ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57566/0 (socket says 172.21.15.16:57566) 2023-10-16T13:48:29.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.001+0000 7ff61259c700 1 -- 172.21.15.16:0/3580591203 learned_addr learned my addr 172.21.15.16:0/3580591203 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:29.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.001+0000 7ff61259c700 1 -- 172.21.15.16:0/3580591203 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff61415ba10 msgr2=0x7ff6140af1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:29.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.001+0000 7ff61259c700 1 --2- 172.21.15.16:0/3580591203 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff61415ba10 0x7ff6140af1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:29.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.001+0000 7ff611d9b700 1 --2- 172.21.15.16:0/3580591203 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff61415b630 0x7ff61406d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:29.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.001+0000 7ff61259c700 1 -- 172.21.15.16:0/3580591203 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff61415b630 msgr2=0x7ff61406d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:29.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.001+0000 7ff61259c700 1 --2- 172.21.15.16:0/3580591203 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff61415b630 0x7ff61406d4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:29.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.001+0000 7ff61259c700 1 -- 172.21.15.16:0/3580591203 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6080094a0 con 0x7ff6140ce4f0 2023-10-16T13:48:29.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.001+0000 7ff61259c700 1 --2- 172.21.15.16:0/3580591203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6140ce4f0 0x7ff61414ccb0 secure :-1 s=READY pgs=3012 cs=0 l=1 rev1=1 crypto rx=0x7ff6040011e0 tx=0x7ff60400cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:29.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.001+0000 7ff602ffd700 1 -- 172.21.15.16:0/3580591203 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff604007560 con 0x7ff6140ce4f0 2023-10-16T13:48:29.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.001+0000 7ff602ffd700 1 -- 172.21.15.16:0/3580591203 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6040076c0 con 0x7ff6140ce4f0 2023-10-16T13:48:29.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.001+0000 7ff618a85700 1 -- 172.21.15.16:0/3580591203 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff60c009a70 con 0x7ff6140ce4f0 2023-10-16T13:48:29.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.001+0000 7ff602ffd700 1 -- 172.21.15.16:0/3580591203 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff604004a90 con 0x7ff6140ce4f0 2023-10-16T13:48:29.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.001+0000 7ff618a85700 1 -- 172.21.15.16:0/3580591203 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6140b6e00 con 0x7ff6140ce4f0 2023-10-16T13:48:29.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.002+0000 7ff618a85700 1 -- 172.21.15.16:0/3580591203 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff61415cf50 con 0x7ff6140ce4f0 2023-10-16T13:48:29.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.003+0000 7ff602ffd700 1 -- 172.21.15.16:0/3580591203 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7ff604004bf0 con 0x7ff6140ce4f0 2023-10-16T13:48:29.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.003+0000 7ff602ffd700 1 --2- 172.21.15.16:0/3580591203 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7ff5f406b140 0x7ff5f406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:29.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.003+0000 7ff602ffd700 1 -- 172.21.15.16:0/3580591203 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 6157+0+0 (secure 0 0 0) 0x7ff604088fb0 con 0x7ff6140ce4f0 2023-10-16T13:48:29.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.003+0000 7ff611d9b700 1 --2- 172.21.15.16:0/3580591203 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7ff5f406b140 0x7ff5f406d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:29.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.004+0000 7ff611d9b700 1 --2- 172.21.15.16:0/3580591203 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7ff5f406b140 0x7ff5f406d600 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff61406d360 tx=0x7ff608008040 comp rx=0 tx=0).ready entity=mgr.12678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:29.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.008+0000 7ff602ffd700 1 -- 172.21.15.16:0/3580591203 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff61415cf50 con 0x7ff6140ce4f0 2023-10-16T13:48:29.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.213+0000 7ff618a85700 1 -- 172.21.15.16:0/3580591203 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff61415cf50 con 0x7ff6140ce4f0 2023-10-16T13:48:29.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.215+0000 7ff602ffd700 1 -- 172.21.15.16:0/3580591203 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v265) v1 ==== 74+0+16817 (secure 0 0 0) 0x7ff61415cf50 con 0x7ff6140ce4f0 2023-10-16T13:48:29.217 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:48:29.218 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":265,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:48:26.417878+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":36,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_metadata","create_time":"2023-10-16T13:48:23.348668+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"264","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_data","create_time":"2023-10-16T13:48:23.799968+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"265","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:48:29.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.217+0000 7ff618a85700 1 -- 172.21.15.16:0/3580591203 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7ff5f406b140 msgr2=0x7ff5f406d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:29.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.217+0000 7ff618a85700 1 --2- 172.21.15.16:0/3580591203 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7ff5f406b140 0x7ff5f406d600 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff61406d360 tx=0x7ff608008040 comp rx=0 tx=0).stop 2023-10-16T13:48:29.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.217+0000 7ff618a85700 1 -- 172.21.15.16:0/3580591203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6140ce4f0 msgr2=0x7ff61414ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:29.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.217+0000 7ff618a85700 1 --2- 172.21.15.16:0/3580591203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6140ce4f0 0x7ff61414ccb0 secure :-1 s=READY pgs=3012 cs=0 l=1 rev1=1 crypto rx=0x7ff6040011e0 tx=0x7ff60400cb00 comp rx=0 tx=0).stop 2023-10-16T13:48:29.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.217+0000 7ff618a85700 1 -- 172.21.15.16:0/3580591203 shutdown_connections 2023-10-16T13:48:29.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.217+0000 7ff618a85700 1 --2- 172.21.15.16:0/3580591203 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7ff5f406b140 0x7ff5f406d600 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:29.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.217+0000 7ff618a85700 1 --2- 172.21.15.16:0/3580591203 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff61415ba10 0x7ff6140af1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:29.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.217+0000 7ff618a85700 1 --2- 172.21.15.16:0/3580591203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6140ce4f0 0x7ff61414ccb0 unknown :-1 s=CLOSED pgs=3012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:29.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.217+0000 7ff618a85700 1 --2- 172.21.15.16:0/3580591203 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff61415b630 0x7ff61406d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:29.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.217+0000 7ff618a85700 1 -- 172.21.15.16:0/3580591203 >> 172.21.15.16:0/3580591203 conn(0x7ff6140d14b0 msgr2=0x7ff61406f880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:29.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.217+0000 7ff618a85700 1 -- 172.21.15.16:0/3580591203 shutdown_connections 2023-10-16T13:48:29.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.217+0000 7ff618a85700 1 -- 172.21.15.16:0/3580591203 wait complete. 2023-10-16T13:48:29.232 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-16T13:48:29.232 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T13:48:29.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.355+0000 7f816bddb700 1 Processor -- start 2023-10-16T13:48:29.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.355+0000 7f816bddb700 1 -- start start 2023-10-16T13:48:29.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.356+0000 7f816bddb700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f81640cf4a0 0x7f81640726e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:29.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.356+0000 7f816bddb700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f81640cf880 0x7f81640b52f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:29.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.356+0000 7f816bddb700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f816415bc80 con 0x7f81640b5830 2023-10-16T13:48:29.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.356+0000 7f816bddb700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8164072c20 con 0x7f81640cf880 2023-10-16T13:48:29.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.356+0000 7f816bddb700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f816415b070 con 0x7f81640cf4a0 2023-10-16T13:48:29.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.356+0000 7f8169376700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f81640cf880 0x7f81640b52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:29.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.356+0000 7f8169376700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f81640cf880 0x7f81640b52f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57576/0 (socket says 172.21.15.16:57576) 2023-10-16T13:48:29.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.356+0000 7f8169376700 1 -- 172.21.15.16:0/33616599 learned_addr learned my addr 172.21.15.16:0/33616599 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:29.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.356+0000 7f8169b77700 1 --2- 172.21.15.16:0/33616599 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f81640cf4a0 0x7f81640726e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:29.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.356+0000 7f8169376700 1 -- 172.21.15.16:0/33616599 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f81640cf4a0 msgr2=0x7f81640726e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:29.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.356+0000 7f8169376700 1 --2- 172.21.15.16:0/33616599 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f81640cf4a0 0x7f81640726e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:29.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.356+0000 7f8169376700 1 -- 172.21.15.16:0/33616599 >> v1:172.21.15.49:6789/0 conn(0x7f81640b5830 legacy=0x7f816415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:29.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.356+0000 7f8169376700 1 -- 172.21.15.16:0/33616599 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f816415b350 con 0x7f81640cf880 2023-10-16T13:48:29.362 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.356+0000 7f8169376700 1 --2- 172.21.15.16:0/33616599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81640cf880 0x7f81640b52f0 secure :-1 s=READY pgs=3013 cs=0 l=1 rev1=1 crypto rx=0x7f81540098f0 tx=0x7f8154009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cd99e0132c4dd5e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:29.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.357+0000 7f8168b75700 1 -- 172.21.15.16:0/33616599 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8154004230 con 0x7f81640cf880 2023-10-16T13:48:29.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.357+0000 7f8168b75700 1 -- 172.21.15.16:0/33616599 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8154004390 con 0x7f81640cf880 2023-10-16T13:48:29.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.357+0000 7f8168b75700 1 -- 172.21.15.16:0/33616599 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81540036a0 con 0x7f81640cf880 2023-10-16T13:48:29.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.357+0000 7f816bddb700 1 -- 172.21.15.16:0/33616599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81640cf880 msgr2=0x7f81640b52f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:29.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.357+0000 7f816bddb700 1 --2- 172.21.15.16:0/33616599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81640cf880 0x7f81640b52f0 secure :-1 s=READY pgs=3013 cs=0 l=1 rev1=1 crypto rx=0x7f81540098f0 tx=0x7f8154009c00 comp rx=0 tx=0).stop 2023-10-16T13:48:29.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.357+0000 7f816bddb700 1 -- 172.21.15.16:0/33616599 shutdown_connections 2023-10-16T13:48:29.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.357+0000 7f816bddb700 1 --2- 172.21.15.16:0/33616599 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f81640cf4a0 0x7f81640726e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:29.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.357+0000 7f816bddb700 1 --2- 172.21.15.16:0/33616599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81640cf880 0x7f81640b52f0 unknown :-1 s=CLOSED pgs=3013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:29.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.357+0000 7f816bddb700 1 -- 172.21.15.16:0/33616599 >> 172.21.15.16:0/33616599 conn(0x7f81640b1fd0 msgr2=0x7f81640b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:29.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.357+0000 7f816bddb700 1 -- 172.21.15.16:0/33616599 shutdown_connections 2023-10-16T13:48:29.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.357+0000 7f816bddb700 1 -- 172.21.15.16:0/33616599 wait complete. 2023-10-16T13:48:29.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.358+0000 7f816bddb700 1 Processor -- start 2023-10-16T13:48:29.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.359+0000 7f816bddb700 1 -- start start 2023-10-16T13:48:29.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.359+0000 7f816bddb700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f81640b5830 0x7f8164151750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:29.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.359+0000 7f816bddb700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81640cf4a0 0x7f8164152130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:29.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.359+0000 7f816bddb700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f81640cf880 0x7f81640c3f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:29.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.359+0000 7f816bddb700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8164153c20 con 0x7f81640cf4a0 2023-10-16T13:48:29.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.359+0000 7f816bddb700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8164152b80 con 0x7f81640cf880 2023-10-16T13:48:29.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.359+0000 7f816bddb700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8164152e60 con 0x7f81640b5830 2023-10-16T13:48:29.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.359+0000 7f8169376700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81640cf4a0 0x7f8164152130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:29.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.359+0000 7f8169376700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81640cf4a0 0x7f8164152130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57582/0 (socket says 172.21.15.16:57582) 2023-10-16T13:48:29.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.359+0000 7f8169376700 1 -- 172.21.15.16:0/779302295 learned_addr learned my addr 172.21.15.16:0/779302295 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:29.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.359+0000 7f816a378700 1 --2- 172.21.15.16:0/779302295 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f81640cf880 0x7f81640c3f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:29.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.360+0000 7f8169b77700 1 --2- 172.21.15.16:0/779302295 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f81640b5830 0x7f8164151750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:29.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.360+0000 7f8169376700 1 -- 172.21.15.16:0/779302295 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f81640b5830 msgr2=0x7f8164151750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:29.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.360+0000 7f8169376700 1 --2- 172.21.15.16:0/779302295 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f81640b5830 0x7f8164151750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:29.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.360+0000 7f8169376700 1 -- 172.21.15.16:0/779302295 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f81640cf880 msgr2=0x7f81640c3f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:29.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.360+0000 7f8169376700 1 --2- 172.21.15.16:0/779302295 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f81640cf880 0x7f81640c3f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:29.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.360+0000 7f8169376700 1 -- 172.21.15.16:0/779302295 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8154009550 con 0x7f81640cf4a0 2023-10-16T13:48:29.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.360+0000 7f8169376700 1 --2- 172.21.15.16:0/779302295 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81640cf4a0 0x7f8164152130 secure :-1 s=READY pgs=3014 cs=0 l=1 rev1=1 crypto rx=0x7f8154000c00 tx=0x7f8154004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:29.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.360+0000 7f815a7fc700 1 -- 172.21.15.16:0/779302295 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f815400b860 con 0x7f81640cf4a0 2023-10-16T13:48:29.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.360+0000 7f815a7fc700 1 -- 172.21.15.16:0/779302295 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f815400b9c0 con 0x7f81640cf4a0 2023-10-16T13:48:29.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.360+0000 7f816bddb700 1 -- 172.21.15.16:0/779302295 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8164153140 con 0x7f81640cf4a0 2023-10-16T13:48:29.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.360+0000 7f815a7fc700 1 -- 172.21.15.16:0/779302295 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f815400bd70 con 0x7f81640cf4a0 2023-10-16T13:48:29.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.360+0000 7f816bddb700 1 -- 172.21.15.16:0/779302295 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8164153700 con 0x7f81640cf4a0 2023-10-16T13:48:29.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.361+0000 7f816bddb700 1 -- 172.21.15.16:0/779302295 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8164073520 con 0x7f81640cf4a0 2023-10-16T13:48:29.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.361+0000 7f815a7fc700 1 -- 172.21.15.16:0/779302295 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f815401b440 con 0x7f81640cf4a0 2023-10-16T13:48:29.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.362+0000 7f815a7fc700 1 --2- 172.21.15.16:0/779302295 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f814806b140 0x7f814806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:29.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.362+0000 7f815a7fc700 1 -- 172.21.15.16:0/779302295 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f815408b250 con 0x7f81640cf4a0 2023-10-16T13:48:29.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.362+0000 7f8169b77700 1 --2- 172.21.15.16:0/779302295 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f814806b140 0x7f814806d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:29.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.362+0000 7f8169b77700 1 --2- 172.21.15.16:0/779302295 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f814806b140 0x7f814806d600 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f81640b2db0 tx=0x7f8160008040 comp rx=0 tx=0).ready entity=mgr.12678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:29.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.366+0000 7f815a7fc700 1 -- 172.21.15.16:0/779302295 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8154055ea0 con 0x7f81640cf4a0 2023-10-16T13:48:29.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.567+0000 7f816bddb700 1 -- 172.21.15.16:0/779302295 --> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f816415b630 con 0x7f814806b140 2023-10-16T13:48:29.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.576+0000 7f815a7fc700 1 -- 172.21.15.16:0/779302295 <== mgr.12678 v2:172.21.15.16:6832/1275231846 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+290166 (secure 0 0 0) 0x7f8160009dd0 con 0x7f814806b140 2023-10-16T13:48:29.581 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:48:29.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.579+0000 7f816bddb700 1 -- 172.21.15.16:0/779302295 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f814806b140 msgr2=0x7f814806d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:29.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.579+0000 7f816bddb700 1 --2- 172.21.15.16:0/779302295 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f814806b140 0x7f814806d600 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f81640b2db0 tx=0x7f8160008040 comp rx=0 tx=0).stop 2023-10-16T13:48:29.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.579+0000 7f816bddb700 1 -- 172.21.15.16:0/779302295 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81640cf4a0 msgr2=0x7f8164152130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:29.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.579+0000 7f816bddb700 1 --2- 172.21.15.16:0/779302295 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81640cf4a0 0x7f8164152130 secure :-1 s=READY pgs=3014 cs=0 l=1 rev1=1 crypto rx=0x7f8154000c00 tx=0x7f8154004750 comp rx=0 tx=0).stop 2023-10-16T13:48:29.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.579+0000 7f816bddb700 1 -- 172.21.15.16:0/779302295 shutdown_connections 2023-10-16T13:48:29.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.579+0000 7f816bddb700 1 --2- 172.21.15.16:0/779302295 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f814806b140 0x7f814806d600 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:29.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.579+0000 7f816bddb700 1 --2- 172.21.15.16:0/779302295 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f81640cf880 0x7f81640c3f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:29.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.579+0000 7f816bddb700 1 --2- 172.21.15.16:0/779302295 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f81640b5830 0x7f8164151750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:29.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.579+0000 7f816bddb700 1 --2- 172.21.15.16:0/779302295 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81640cf4a0 0x7f8164152130 unknown :-1 s=CLOSED pgs=3014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:29.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.579+0000 7f816bddb700 1 -- 172.21.15.16:0/779302295 >> 172.21.15.16:0/779302295 conn(0x7f81640b1fd0 msgr2=0x7f81640ceb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:29.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.579+0000 7f816bddb700 1 -- 172.21.15.16:0/779302295 shutdown_connections 2023-10-16T13:48:29.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.579+0000 7f816bddb700 1 -- 172.21.15.16:0/779302295 wait complete. 2023-10-16T13:48:29.586 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-10-16T13:48:29.594 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":30,"stamp":"2023-10-16T13:48:28.509404+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2420,"num_read_kb":5044,"num_write":683,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":419,"ondisk_log_size":419,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":144,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":464252,"kb_used_data":27400,"kb_used_omap":12,"kb_used_meta":436723,"kb_avail":754510468,"statfs":{"total":773094113280,"available":772618719232,"internally_reserved":0,"allocated":28057600,"data_stored":15108108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12740,"internal_metadata":447204924},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.108657"},"pg_stats":[{"pgid":"36.3f","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.394764+0000","last_change":"2023-10-16T13:48:25.440344+0000","last_active":"2023-10-16T13:48:27.394764+0000","last_peered":"2023-10-16T13:48:27.394764+0000","last_clean":"2023-10-16T13:48:27.394764+0000","last_became_active":"2023-10-16T13:48:25.439797+0000","last_became_peered":"2023-10-16T13:48:25.439797+0000","last_unstale":"2023-10-16T13:48:27.394764+0000","last_undegraded":"2023-10-16T13:48:27.394764+0000","last_fullsized":"2023-10-16T13:48:27.394764+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:24:11.575909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.3e","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.394646+0000","last_change":"2023-10-16T13:48:25.447276+0000","last_active":"2023-10-16T13:48:27.394646+0000","last_peered":"2023-10-16T13:48:27.394646+0000","last_clean":"2023-10-16T13:48:27.394646+0000","last_became_active":"2023-10-16T13:48:25.444498+0000","last_became_peered":"2023-10-16T13:48:25.444498+0000","last_unstale":"2023-10-16T13:48:27.394646+0000","last_undegraded":"2023-10-16T13:48:27.394646+0000","last_fullsized":"2023-10-16T13:48:27.394646+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:40:17.291707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.3d","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.394304+0000","last_change":"2023-10-16T13:48:25.445627+0000","last_active":"2023-10-16T13:48:27.394304+0000","last_peered":"2023-10-16T13:48:27.394304+0000","last_clean":"2023-10-16T13:48:27.394304+0000","last_became_active":"2023-10-16T13:48:25.445233+0000","last_became_peered":"2023-10-16T13:48:25.445233+0000","last_unstale":"2023-10-16T13:48:27.394304+0000","last_undegraded":"2023-10-16T13:48:27.394304+0000","last_fullsized":"2023-10-16T13:48:27.394304+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:46:41.609860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.3c","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.394242+0000","last_change":"2023-10-16T13:48:25.433667+0000","last_active":"2023-10-16T13:48:27.394242+0000","last_peered":"2023-10-16T13:48:27.394242+0000","last_clean":"2023-10-16T13:48:27.394242+0000","last_became_active":"2023-10-16T13:48:25.433338+0000","last_became_peered":"2023-10-16T13:48:25.433338+0000","last_unstale":"2023-10-16T13:48:27.394242+0000","last_undegraded":"2023-10-16T13:48:27.394242+0000","last_fullsized":"2023-10-16T13:48:27.394242+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:15:32.675005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.3b","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.393580+0000","last_change":"2023-10-16T13:48:25.429495+0000","last_active":"2023-10-16T13:48:27.393580+0000","last_peered":"2023-10-16T13:48:27.393580+0000","last_clean":"2023-10-16T13:48:27.393580+0000","last_became_active":"2023-10-16T13:48:25.428576+0000","last_became_peered":"2023-10-16T13:48:25.428576+0000","last_unstale":"2023-10-16T13:48:27.393580+0000","last_undegraded":"2023-10-16T13:48:27.393580+0000","last_fullsized":"2023-10-16T13:48:27.393580+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:41:00.902601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.3a","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.393226+0000","last_change":"2023-10-16T13:48:25.444779+0000","last_active":"2023-10-16T13:48:27.393226+0000","last_peered":"2023-10-16T13:48:27.393226+0000","last_clean":"2023-10-16T13:48:27.393226+0000","last_became_active":"2023-10-16T13:48:25.444495+0000","last_became_peered":"2023-10-16T13:48:25.444495+0000","last_unstale":"2023-10-16T13:48:27.393226+0000","last_undegraded":"2023-10-16T13:48:27.393226+0000","last_fullsized":"2023-10-16T13:48:27.393226+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:11:09.084674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.39","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.164073+0000","last_change":"2023-10-16T13:48:25.441400+0000","last_active":"2023-10-16T13:48:27.164073+0000","last_peered":"2023-10-16T13:48:27.164073+0000","last_clean":"2023-10-16T13:48:27.164073+0000","last_became_active":"2023-10-16T13:48:25.441112+0000","last_became_peered":"2023-10-16T13:48:25.441112+0000","last_unstale":"2023-10-16T13:48:27.164073+0000","last_undegraded":"2023-10-16T13:48:27.164073+0000","last_fullsized":"2023-10-16T13:48:27.164073+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:45:53.473497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.38","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.393265+0000","last_change":"2023-10-16T13:48:25.435146+0000","last_active":"2023-10-16T13:48:27.393265+0000","last_peered":"2023-10-16T13:48:27.393265+0000","last_clean":"2023-10-16T13:48:27.393265+0000","last_became_active":"2023-10-16T13:48:25.434832+0000","last_became_peered":"2023-10-16T13:48:25.434832+0000","last_unstale":"2023-10-16T13:48:27.393265+0000","last_undegraded":"2023-10-16T13:48:27.393265+0000","last_fullsized":"2023-10-16T13:48:27.393265+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:48:55.481961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.37","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.164575+0000","last_change":"2023-10-16T13:48:25.443618+0000","last_active":"2023-10-16T13:48:27.164575+0000","last_peered":"2023-10-16T13:48:27.164575+0000","last_clean":"2023-10-16T13:48:27.164575+0000","last_became_active":"2023-10-16T13:48:25.443345+0000","last_became_peered":"2023-10-16T13:48:25.443345+0000","last_unstale":"2023-10-16T13:48:27.164575+0000","last_undegraded":"2023-10-16T13:48:27.164575+0000","last_fullsized":"2023-10-16T13:48:27.164575+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:11:09.084674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.36","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.164523+0000","last_change":"2023-10-16T13:48:25.434663+0000","last_active":"2023-10-16T13:48:27.164523+0000","last_peered":"2023-10-16T13:48:27.164523+0000","last_clean":"2023-10-16T13:48:27.164523+0000","last_became_active":"2023-10-16T13:48:25.434440+0000","last_became_peered":"2023-10-16T13:48:25.434440+0000","last_unstale":"2023-10-16T13:48:27.164523+0000","last_undegraded":"2023-10-16T13:48:27.164523+0000","last_fullsized":"2023-10-16T13:48:27.164523+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:35:54.607900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.35","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:26.422050+0000","last_change":"2023-10-16T13:48:25.430939+0000","last_active":"2023-10-16T13:48:26.422050+0000","last_peered":"2023-10-16T13:48:26.422050+0000","last_clean":"2023-10-16T13:48:26.422050+0000","last_became_active":"2023-10-16T13:48:25.430596+0000","last_became_peered":"2023-10-16T13:48:25.430596+0000","last_unstale":"2023-10-16T13:48:26.422050+0000","last_undegraded":"2023-10-16T13:48:26.422050+0000","last_fullsized":"2023-10-16T13:48:26.422050+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:47:41.595942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.34","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.395034+0000","last_change":"2023-10-16T13:48:25.440365+0000","last_active":"2023-10-16T13:48:27.395034+0000","last_peered":"2023-10-16T13:48:27.395034+0000","last_clean":"2023-10-16T13:48:27.395034+0000","last_became_active":"2023-10-16T13:48:25.439782+0000","last_became_peered":"2023-10-16T13:48:25.439782+0000","last_unstale":"2023-10-16T13:48:27.395034+0000","last_undegraded":"2023-10-16T13:48:27.395034+0000","last_fullsized":"2023-10-16T13:48:27.395034+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:53:00.722815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.33","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.394234+0000","last_change":"2023-10-16T13:48:25.445304+0000","last_active":"2023-10-16T13:48:27.394234+0000","last_peered":"2023-10-16T13:48:27.394234+0000","last_clean":"2023-10-16T13:48:27.394234+0000","last_became_active":"2023-10-16T13:48:25.444997+0000","last_became_peered":"2023-10-16T13:48:25.444997+0000","last_unstale":"2023-10-16T13:48:27.394234+0000","last_undegraded":"2023-10-16T13:48:27.394234+0000","last_fullsized":"2023-10-16T13:48:27.394234+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:46:04.673793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.32","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.164260+0000","last_change":"2023-10-16T13:48:25.433664+0000","last_active":"2023-10-16T13:48:27.164260+0000","last_peered":"2023-10-16T13:48:27.164260+0000","last_clean":"2023-10-16T13:48:27.164260+0000","last_became_active":"2023-10-16T13:48:25.433352+0000","last_became_peered":"2023-10-16T13:48:25.433352+0000","last_unstale":"2023-10-16T13:48:27.164260+0000","last_undegraded":"2023-10-16T13:48:27.164260+0000","last_fullsized":"2023-10-16T13:48:27.164260+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:00:29.913030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.31","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.393389+0000","last_change":"2023-10-16T13:48:25.435661+0000","last_active":"2023-10-16T13:48:27.393389+0000","last_peered":"2023-10-16T13:48:27.393389+0000","last_clean":"2023-10-16T13:48:27.393389+0000","last_became_active":"2023-10-16T13:48:25.435366+0000","last_became_peered":"2023-10-16T13:48:25.435366+0000","last_unstale":"2023-10-16T13:48:27.393389+0000","last_undegraded":"2023-10-16T13:48:27.393389+0000","last_fullsized":"2023-10-16T13:48:27.393389+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:22:50.795453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.30","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.393363+0000","last_change":"2023-10-16T13:48:25.434843+0000","last_active":"2023-10-16T13:48:27.393363+0000","last_peered":"2023-10-16T13:48:27.393363+0000","last_clean":"2023-10-16T13:48:27.393363+0000","last_became_active":"2023-10-16T13:48:25.434530+0000","last_became_peered":"2023-10-16T13:48:25.434530+0000","last_unstale":"2023-10-16T13:48:27.393363+0000","last_undegraded":"2023-10-16T13:48:27.393363+0000","last_fullsized":"2023-10-16T13:48:27.393363+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:17:13.546895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.2f","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.394821+0000","last_change":"2023-10-16T13:48:25.434325+0000","last_active":"2023-10-16T13:48:27.394821+0000","last_peered":"2023-10-16T13:48:27.394821+0000","last_clean":"2023-10-16T13:48:27.394821+0000","last_became_active":"2023-10-16T13:48:25.433548+0000","last_became_peered":"2023-10-16T13:48:25.433548+0000","last_unstale":"2023-10-16T13:48:27.394821+0000","last_undegraded":"2023-10-16T13:48:27.394821+0000","last_fullsized":"2023-10-16T13:48:27.394821+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:21:29.852309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.2e","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.394496+0000","last_change":"2023-10-16T13:48:25.448916+0000","last_active":"2023-10-16T13:48:27.394496+0000","last_peered":"2023-10-16T13:48:27.394496+0000","last_clean":"2023-10-16T13:48:27.394496+0000","last_became_active":"2023-10-16T13:48:25.446072+0000","last_became_peered":"2023-10-16T13:48:25.446072+0000","last_unstale":"2023-10-16T13:48:27.394496+0000","last_undegraded":"2023-10-16T13:48:27.394496+0000","last_fullsized":"2023-10-16T13:48:27.394496+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:21:02.827127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.2d","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.394411+0000","last_change":"2023-10-16T13:48:25.442491+0000","last_active":"2023-10-16T13:48:27.394411+0000","last_peered":"2023-10-16T13:48:27.394411+0000","last_clean":"2023-10-16T13:48:27.394411+0000","last_became_active":"2023-10-16T13:48:25.442191+0000","last_became_peered":"2023-10-16T13:48:25.442191+0000","last_unstale":"2023-10-16T13:48:27.394411+0000","last_undegraded":"2023-10-16T13:48:27.394411+0000","last_fullsized":"2023-10-16T13:48:27.394411+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:07:14.273460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.2c","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.394093+0000","last_change":"2023-10-16T13:48:25.434792+0000","last_active":"2023-10-16T13:48:27.394093+0000","last_peered":"2023-10-16T13:48:27.394093+0000","last_clean":"2023-10-16T13:48:27.394093+0000","last_became_active":"2023-10-16T13:48:25.434472+0000","last_became_peered":"2023-10-16T13:48:25.434472+0000","last_unstale":"2023-10-16T13:48:27.394093+0000","last_undegraded":"2023-10-16T13:48:27.394093+0000","last_fullsized":"2023-10-16T13:48:27.394093+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:11:58.739219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.2b","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:26.421755+0000","last_change":"2023-10-16T13:48:25.441991+0000","last_active":"2023-10-16T13:48:26.421755+0000","last_peered":"2023-10-16T13:48:26.421755+0000","last_clean":"2023-10-16T13:48:26.421755+0000","last_became_active":"2023-10-16T13:48:25.441592+0000","last_became_peered":"2023-10-16T13:48:25.441592+0000","last_unstale":"2023-10-16T13:48:26.421755+0000","last_undegraded":"2023-10-16T13:48:26.421755+0000","last_fullsized":"2023-10-16T13:48:26.421755+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:28:48.750254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.2a","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:26.421344+0000","last_change":"2023-10-16T13:48:25.434737+0000","last_active":"2023-10-16T13:48:26.421344+0000","last_peered":"2023-10-16T13:48:26.421344+0000","last_clean":"2023-10-16T13:48:26.421344+0000","last_became_active":"2023-10-16T13:48:25.434461+0000","last_became_peered":"2023-10-16T13:48:25.434461+0000","last_unstale":"2023-10-16T13:48:26.421344+0000","last_undegraded":"2023-10-16T13:48:26.421344+0000","last_fullsized":"2023-10-16T13:48:26.421344+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:41:00.902601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.29","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.164129+0000","last_change":"2023-10-16T13:48:25.436144+0000","last_active":"2023-10-16T13:48:27.164129+0000","last_peered":"2023-10-16T13:48:27.164129+0000","last_clean":"2023-10-16T13:48:27.164129+0000","last_became_active":"2023-10-16T13:48:25.435379+0000","last_became_peered":"2023-10-16T13:48:25.435379+0000","last_unstale":"2023-10-16T13:48:27.164129+0000","last_undegraded":"2023-10-16T13:48:27.164129+0000","last_fullsized":"2023-10-16T13:48:27.164129+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:11:05.137455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.28","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:26.421024+0000","last_change":"2023-10-16T13:48:25.438832+0000","last_active":"2023-10-16T13:48:26.421024+0000","last_peered":"2023-10-16T13:48:26.421024+0000","last_clean":"2023-10-16T13:48:26.421024+0000","last_became_active":"2023-10-16T13:48:25.438583+0000","last_became_peered":"2023-10-16T13:48:25.438583+0000","last_unstale":"2023-10-16T13:48:26.421024+0000","last_undegraded":"2023-10-16T13:48:26.421024+0000","last_fullsized":"2023-10-16T13:48:26.421024+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:11:09.084674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.27","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.394939+0000","last_change":"2023-10-16T13:48:25.440490+0000","last_active":"2023-10-16T13:48:27.394939+0000","last_peered":"2023-10-16T13:48:27.394939+0000","last_clean":"2023-10-16T13:48:27.394939+0000","last_became_active":"2023-10-16T13:48:25.440003+0000","last_became_peered":"2023-10-16T13:48:25.440003+0000","last_unstale":"2023-10-16T13:48:27.394939+0000","last_undegraded":"2023-10-16T13:48:27.394939+0000","last_fullsized":"2023-10-16T13:48:27.394939+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:05:29.646924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.26","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.393603+0000","last_change":"2023-10-16T13:48:25.433846+0000","last_active":"2023-10-16T13:48:27.393603+0000","last_peered":"2023-10-16T13:48:27.393603+0000","last_clean":"2023-10-16T13:48:27.393603+0000","last_became_active":"2023-10-16T13:48:25.433463+0000","last_became_peered":"2023-10-16T13:48:25.433463+0000","last_unstale":"2023-10-16T13:48:27.393603+0000","last_undegraded":"2023-10-16T13:48:27.393603+0000","last_fullsized":"2023-10-16T13:48:27.393603+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:08:59.756642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.24","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.393539+0000","last_change":"2023-10-16T13:48:25.436440+0000","last_active":"2023-10-16T13:48:27.393539+0000","last_peered":"2023-10-16T13:48:27.393539+0000","last_clean":"2023-10-16T13:48:27.393539+0000","last_became_active":"2023-10-16T13:48:25.436172+0000","last_became_peered":"2023-10-16T13:48:25.436172+0000","last_unstale":"2023-10-16T13:48:27.393539+0000","last_undegraded":"2023-10-16T13:48:27.393539+0000","last_fullsized":"2023-10-16T13:48:27.393539+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:33:34.431164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.23","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:26.421427+0000","last_change":"2023-10-16T13:48:25.437873+0000","last_active":"2023-10-16T13:48:26.421427+0000","last_peered":"2023-10-16T13:48:26.421427+0000","last_clean":"2023-10-16T13:48:26.421427+0000","last_became_active":"2023-10-16T13:48:25.437581+0000","last_became_peered":"2023-10-16T13:48:25.437581+0000","last_unstale":"2023-10-16T13:48:26.421427+0000","last_undegraded":"2023-10-16T13:48:26.421427+0000","last_fullsized":"2023-10-16T13:48:26.421427+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:35:56.554093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.22","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.393592+0000","last_change":"2023-10-16T13:48:25.435538+0000","last_active":"2023-10-16T13:48:27.393592+0000","last_peered":"2023-10-16T13:48:27.393592+0000","last_clean":"2023-10-16T13:48:27.393592+0000","last_became_active":"2023-10-16T13:48:25.435190+0000","last_became_peered":"2023-10-16T13:48:25.435190+0000","last_unstale":"2023-10-16T13:48:27.393592+0000","last_undegraded":"2023-10-16T13:48:27.393592+0000","last_fullsized":"2023-10-16T13:48:27.393592+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:52:50.562319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.21","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:26.421074+0000","last_change":"2023-10-16T13:48:25.445785+0000","last_active":"2023-10-16T13:48:26.421074+0000","last_peered":"2023-10-16T13:48:26.421074+0000","last_clean":"2023-10-16T13:48:26.421074+0000","last_became_active":"2023-10-16T13:48:25.445489+0000","last_became_peered":"2023-10-16T13:48:25.445489+0000","last_unstale":"2023-10-16T13:48:26.421074+0000","last_undegraded":"2023-10-16T13:48:26.421074+0000","last_fullsized":"2023-10-16T13:48:26.421074+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:30:42.136595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.20","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.393352+0000","last_change":"2023-10-16T13:48:25.441470+0000","last_active":"2023-10-16T13:48:27.393352+0000","last_peered":"2023-10-16T13:48:27.393352+0000","last_clean":"2023-10-16T13:48:27.393352+0000","last_became_active":"2023-10-16T13:48:25.440973+0000","last_became_peered":"2023-10-16T13:48:25.440973+0000","last_unstale":"2023-10-16T13:48:27.393352+0000","last_undegraded":"2023-10-16T13:48:27.393352+0000","last_fullsized":"2023-10-16T13:48:27.393352+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:16:45.304293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.18","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:26.421074+0000","last_change":"2023-10-16T13:48:25.439376+0000","last_active":"2023-10-16T13:48:26.421074+0000","last_peered":"2023-10-16T13:48:26.421074+0000","last_clean":"2023-10-16T13:48:26.421074+0000","last_became_active":"2023-10-16T13:48:25.439055+0000","last_became_peered":"2023-10-16T13:48:25.439055+0000","last_unstale":"2023-10-16T13:48:26.421074+0000","last_undegraded":"2023-10-16T13:48:26.421074+0000","last_fullsized":"2023-10-16T13:48:26.421074+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:04:33.594311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.1f","version":"264'4","reported_seq":16,"reported_epoch":264,"state":"active+clean","last_fresh":"2023-10-16T13:48:25.453425+0000","last_change":"2023-10-16T13:48:24.426828+0000","last_active":"2023-10-16T13:48:25.453425+0000","last_peered":"2023-10-16T13:48:25.453425+0000","last_clean":"2023-10-16T13:48:25.453425+0000","last_became_active":"2023-10-16T13:48:24.426394+0000","last_became_peered":"2023-10-16T13:48:24.426394+0000","last_unstale":"2023-10-16T13:48:25.453425+0000","last_undegraded":"2023-10-16T13:48:25.453425+0000","last_fullsized":"2023-10-16T13:48:25.453425+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_clean_scrub_stamp":"2023-10-16T13:48:23.395571+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:51:29.737215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.19","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.164445+0000","last_change":"2023-10-16T13:48:25.436335+0000","last_active":"2023-10-16T13:48:27.164445+0000","last_peered":"2023-10-16T13:48:27.164445+0000","last_clean":"2023-10-16T13:48:27.164445+0000","last_became_active":"2023-10-16T13:48:25.435822+0000","last_became_peered":"2023-10-16T13:48:25.435822+0000","last_unstale":"2023-10-16T13:48:27.164445+0000","last_undegraded":"2023-10-16T13:48:27.164445+0000","last_fullsized":"2023-10-16T13:48:27.164445+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:12:16.262267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.1e","version":"264'1","reported_seq":13,"reported_epoch":264,"state":"active+clean","last_fresh":"2023-10-16T13:48:25.454493+0000","last_change":"2023-10-16T13:48:24.422098+0000","last_active":"2023-10-16T13:48:25.454493+0000","last_peered":"2023-10-16T13:48:25.454493+0000","last_clean":"2023-10-16T13:48:25.454493+0000","last_became_active":"2023-10-16T13:48:24.421192+0000","last_became_peered":"2023-10-16T13:48:24.421192+0000","last_unstale":"2023-10-16T13:48:25.454493+0000","last_undegraded":"2023-10-16T13:48:25.454493+0000","last_fullsized":"2023-10-16T13:48:25.454493+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_clean_scrub_stamp":"2023-10-16T13:48:23.395571+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:08:40.541463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.1a","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.394877+0000","last_change":"2023-10-16T13:48:25.445268+0000","last_active":"2023-10-16T13:48:27.394877+0000","last_peered":"2023-10-16T13:48:27.394877+0000","last_clean":"2023-10-16T13:48:27.394877+0000","last_became_active":"2023-10-16T13:48:25.444954+0000","last_became_peered":"2023-10-16T13:48:25.444954+0000","last_unstale":"2023-10-16T13:48:27.394877+0000","last_undegraded":"2023-10-16T13:48:27.394877+0000","last_fullsized":"2023-10-16T13:48:27.394877+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:28:21.176426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.1d","version":"264'2","reported_seq":14,"reported_epoch":264,"state":"active+clean","last_fresh":"2023-10-16T13:48:25.454252+0000","last_change":"2023-10-16T13:48:24.422654+0000","last_active":"2023-10-16T13:48:25.454252+0000","last_peered":"2023-10-16T13:48:25.454252+0000","last_clean":"2023-10-16T13:48:25.454252+0000","last_became_active":"2023-10-16T13:48:24.422204+0000","last_became_peered":"2023-10-16T13:48:24.422204+0000","last_unstale":"2023-10-16T13:48:25.454252+0000","last_undegraded":"2023-10-16T13:48:25.454252+0000","last_fullsized":"2023-10-16T13:48:25.454252+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_clean_scrub_stamp":"2023-10-16T13:48:23.395571+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:56:05.560556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.1b","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.164373+0000","last_change":"2023-10-16T13:48:25.443547+0000","last_active":"2023-10-16T13:48:27.164373+0000","last_peered":"2023-10-16T13:48:27.164373+0000","last_clean":"2023-10-16T13:48:27.164373+0000","last_became_active":"2023-10-16T13:48:25.443240+0000","last_became_peered":"2023-10-16T13:48:25.443240+0000","last_unstale":"2023-10-16T13:48:27.164373+0000","last_undegraded":"2023-10-16T13:48:27.164373+0000","last_fullsized":"2023-10-16T13:48:27.164373+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:14:32.129157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.1c","version":"264'2","reported_seq":14,"reported_epoch":264,"state":"active+clean","last_fresh":"2023-10-16T13:48:25.454745+0000","last_change":"2023-10-16T13:48:24.420922+0000","last_active":"2023-10-16T13:48:25.454745+0000","last_peered":"2023-10-16T13:48:25.454745+0000","last_clean":"2023-10-16T13:48:25.454745+0000","last_became_active":"2023-10-16T13:48:24.420438+0000","last_became_peered":"2023-10-16T13:48:24.420438+0000","last_unstale":"2023-10-16T13:48:25.454745+0000","last_undegraded":"2023-10-16T13:48:25.454745+0000","last_fullsized":"2023-10-16T13:48:25.454745+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_clean_scrub_stamp":"2023-10-16T13:48:23.395571+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:35:03.573357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.1c","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.394439+0000","last_change":"2023-10-16T13:48:25.448098+0000","last_active":"2023-10-16T13:48:27.394439+0000","last_peered":"2023-10-16T13:48:27.394439+0000","last_clean":"2023-10-16T13:48:27.394439+0000","last_became_active":"2023-10-16T13:48:25.446119+0000","last_became_peered":"2023-10-16T13:48:25.446119+0000","last_unstale":"2023-10-16T13:48:27.394439+0000","last_undegraded":"2023-10-16T13:48:27.394439+0000","last_fullsized":"2023-10-16T13:48:27.394439+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:40:30.963695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.1b","version":"264'1","reported_seq":13,"reported_epoch":264,"state":"active+clean","last_fresh":"2023-10-16T13:48:25.454386+0000","last_change":"2023-10-16T13:48:24.425433+0000","last_active":"2023-10-16T13:48:25.454386+0000","last_peered":"2023-10-16T13:48:25.454386+0000","last_clean":"2023-10-16T13:48:25.454386+0000","last_became_active":"2023-10-16T13:48:24.425121+0000","last_became_peered":"2023-10-16T13:48:24.425121+0000","last_unstale":"2023-10-16T13:48:25.454386+0000","last_undegraded":"2023-10-16T13:48:25.454386+0000","last_fullsized":"2023-10-16T13:48:25.454386+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_clean_scrub_stamp":"2023-10-16T13:48:23.395571+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:57:16.571463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.1d","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:26.422317+0000","last_change":"2023-10-16T13:48:25.445586+0000","last_active":"2023-10-16T13:48:26.422317+0000","last_peered":"2023-10-16T13:48:26.422317+0000","last_clean":"2023-10-16T13:48:26.422317+0000","last_became_active":"2023-10-16T13:48:25.445292+0000","last_became_peered":"2023-10-16T13:48:25.445292+0000","last_unstale":"2023-10-16T13:48:26.422317+0000","last_undegraded":"2023-10-16T13:48:26.422317+0000","last_fullsized":"2023-10-16T13:48:26.422317+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:08:24.915543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.1a","version":"264'1","reported_seq":13,"reported_epoch":264,"state":"active+clean","last_fresh":"2023-10-16T13:48:25.453362+0000","last_change":"2023-10-16T13:48:24.420701+0000","last_active":"2023-10-16T13:48:25.453362+0000","last_peered":"2023-10-16T13:48:25.453362+0000","last_clean":"2023-10-16T13:48:25.453362+0000","last_became_active":"2023-10-16T13:48:24.420401+0000","last_became_peered":"2023-10-16T13:48:24.420401+0000","last_unstale":"2023-10-16T13:48:25.453362+0000","last_undegraded":"2023-10-16T13:48:25.453362+0000","last_fullsized":"2023-10-16T13:48:25.453362+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_clean_scrub_stamp":"2023-10-16T13:48:23.395571+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:11:04.133038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.1e","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:26.422513+0000","last_change":"2023-10-16T13:48:25.439008+0000","last_active":"2023-10-16T13:48:26.422513+0000","last_peered":"2023-10-16T13:48:26.422513+0000","last_clean":"2023-10-16T13:48:26.422513+0000","last_became_active":"2023-10-16T13:48:25.438631+0000","last_became_peered":"2023-10-16T13:48:25.438631+0000","last_unstale":"2023-10-16T13:48:26.422513+0000","last_undegraded":"2023-10-16T13:48:26.422513+0000","last_fullsized":"2023-10-16T13:48:26.422513+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:11:52.087708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.19","version":"0'0","reported_seq":12,"reported_epoch":264,"state":"active+clean","last_fresh":"2023-10-16T13:48:25.422488+0000","last_change":"2023-10-16T13:48:24.418856+0000","last_active":"2023-10-16T13:48:25.422488+0000","last_peered":"2023-10-16T13:48:25.422488+0000","last_clean":"2023-10-16T13:48:25.422488+0000","last_became_active":"2023-10-16T13:48:24.418545+0000","last_became_peered":"2023-10-16T13:48:24.418545+0000","last_unstale":"2023-10-16T13:48:25.422488+0000","last_undegraded":"2023-10-16T13:48:25.422488+0000","last_fullsized":"2023-10-16T13:48:25.422488+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_clean_scrub_stamp":"2023-10-16T13:48:23.395571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:17:50.550699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.1f","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.394967+0000","last_change":"2023-10-16T13:48:25.434506+0000","last_active":"2023-10-16T13:48:27.394967+0000","last_peered":"2023-10-16T13:48:27.394967+0000","last_clean":"2023-10-16T13:48:27.394967+0000","last_became_active":"2023-10-16T13:48:25.434180+0000","last_became_peered":"2023-10-16T13:48:25.434180+0000","last_unstale":"2023-10-16T13:48:27.394967+0000","last_undegraded":"2023-10-16T13:48:27.394967+0000","last_fullsized":"2023-10-16T13:48:27.394967+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:58:59.048055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.18","version":"0'0","reported_seq":12,"reported_epoch":264,"state":"active+clean","last_fresh":"2023-10-16T13:48:25.428855+0000","last_change":"2023-10-16T13:48:24.423577+0000","last_active":"2023-10-16T13:48:25.428855+0000","last_peered":"2023-10-16T13:48:25.428855+0000","last_clean":"2023-10-16T13:48:25.428855+0000","last_became_active":"2023-10-16T13:48:24.422594+0000","last_became_peered":"2023-10-16T13:48:24.422594+0000","last_unstale":"2023-10-16T13:48:25.428855+0000","last_undegraded":"2023-10-16T13:48:25.428855+0000","last_fullsized":"2023-10-16T13:48:25.428855+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_clean_scrub_stamp":"2023-10-16T13:48:23.395571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:18:00.438377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.10","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.393427+0000","last_change":"2023-10-16T13:48:25.445795+0000","last_active":"2023-10-16T13:48:27.393427+0000","last_peered":"2023-10-16T13:48:27.393427+0000","last_clean":"2023-10-16T13:48:27.393427+0000","last_became_active":"2023-10-16T13:48:25.444626+0000","last_became_peered":"2023-10-16T13:48:25.444626+0000","last_unstale":"2023-10-16T13:48:27.393427+0000","last_undegraded":"2023-10-16T13:48:27.393427+0000","last_fullsized":"2023-10-16T13:48:27.393427+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:38:03.913584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.17","version":"0'0","reported_seq":12,"reported_epoch":264,"state":"active+clean","last_fresh":"2023-10-16T13:48:25.433009+0000","last_change":"2023-10-16T13:48:24.424321+0000","last_active":"2023-10-16T13:48:25.433009+0000","last_peered":"2023-10-16T13:48:25.433009+0000","last_clean":"2023-10-16T13:48:25.433009+0000","last_became_active":"2023-10-16T13:48:24.423989+0000","last_became_peered":"2023-10-16T13:48:24.423989+0000","last_unstale":"2023-10-16T13:48:25.433009+0000","last_undegraded":"2023-10-16T13:48:25.433009+0000","last_fullsized":"2023-10-16T13:48:25.433009+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_clean_scrub_stamp":"2023-10-16T13:48:23.395571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:13:35.750808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.11","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.394279+0000","last_change":"2023-10-16T13:48:25.440171+0000","last_active":"2023-10-16T13:48:27.394279+0000","last_peered":"2023-10-16T13:48:27.394279+0000","last_clean":"2023-10-16T13:48:27.394279+0000","last_became_active":"2023-10-16T13:48:25.439568+0000","last_became_peered":"2023-10-16T13:48:25.439568+0000","last_unstale":"2023-10-16T13:48:27.394279+0000","last_undegraded":"2023-10-16T13:48:27.394279+0000","last_fullsized":"2023-10-16T13:48:27.394279+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:14:12.790962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.16","version":"0'0","reported_seq":12,"reported_epoch":264,"state":"active+clean","last_fresh":"2023-10-16T13:48:25.422535+0000","last_change":"2023-10-16T13:48:24.419409+0000","last_active":"2023-10-16T13:48:25.422535+0000","last_peered":"2023-10-16T13:48:25.422535+0000","last_clean":"2023-10-16T13:48:25.422535+0000","last_became_active":"2023-10-16T13:48:24.419140+0000","last_became_peered":"2023-10-16T13:48:24.419140+0000","last_unstale":"2023-10-16T13:48:25.422535+0000","last_undegraded":"2023-10-16T13:48:25.422535+0000","last_fullsized":"2023-10-16T13:48:25.422535+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_clean_scrub_stamp":"2023-10-16T13:48:23.395571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:45:00.351108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.12","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.393906+0000","last_change":"2023-10-16T13:48:25.433864+0000","last_active":"2023-10-16T13:48:27.393906+0000","last_peered":"2023-10-16T13:48:27.393906+0000","last_clean":"2023-10-16T13:48:27.393906+0000","last_became_active":"2023-10-16T13:48:25.433560+0000","last_became_peered":"2023-10-16T13:48:25.433560+0000","last_unstale":"2023-10-16T13:48:27.393906+0000","last_undegraded":"2023-10-16T13:48:27.393906+0000","last_fullsized":"2023-10-16T13:48:27.393906+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:10:27.868159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.15","version":"264'3","reported_seq":15,"reported_epoch":264,"state":"active+clean","last_fresh":"2023-10-16T13:48:25.452740+0000","last_change":"2023-10-16T13:48:24.421215+0000","last_active":"2023-10-16T13:48:25.452740+0000","last_peered":"2023-10-16T13:48:25.452740+0000","last_clean":"2023-10-16T13:48:25.452740+0000","last_became_active":"2023-10-16T13:48:24.420643+0000","last_became_peered":"2023-10-16T13:48:24.420643+0000","last_unstale":"2023-10-16T13:48:25.452740+0000","last_undegraded":"2023-10-16T13:48:25.452740+0000","last_fullsized":"2023-10-16T13:48:25.452740+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_clean_scrub_stamp":"2023-10-16T13:48:23.395571+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:24:58.740209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.13","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.393710+0000","last_change":"2023-10-16T13:48:25.437813+0000","last_active":"2023-10-16T13:48:27.393710+0000","last_peered":"2023-10-16T13:48:27.393710+0000","last_clean":"2023-10-16T13:48:27.393710+0000","last_became_active":"2023-10-16T13:48:25.437510+0000","last_became_peered":"2023-10-16T13:48:25.437510+0000","last_unstale":"2023-10-16T13:48:27.393710+0000","last_undegraded":"2023-10-16T13:48:27.393710+0000","last_fullsized":"2023-10-16T13:48:27.393710+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:35:56.554093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.14","version":"264'6","reported_seq":18,"reported_epoch":264,"state":"active+clean","last_fresh":"2023-10-16T13:48:25.456240+0000","last_change":"2023-10-16T13:48:24.429280+0000","last_active":"2023-10-16T13:48:25.456240+0000","last_peered":"2023-10-16T13:48:25.456240+0000","last_clean":"2023-10-16T13:48:25.456240+0000","last_became_active":"2023-10-16T13:48:24.428975+0000","last_became_peered":"2023-10-16T13:48:24.428975+0000","last_unstale":"2023-10-16T13:48:25.456240+0000","last_undegraded":"2023-10-16T13:48:25.456240+0000","last_fullsized":"2023-10-16T13:48:25.456240+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_clean_scrub_stamp":"2023-10-16T13:48:23.395571+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:40:12.074998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.14","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.395040+0000","last_change":"2023-10-16T13:48:25.446229+0000","last_active":"2023-10-16T13:48:27.395040+0000","last_peered":"2023-10-16T13:48:27.395040+0000","last_clean":"2023-10-16T13:48:27.395040+0000","last_became_active":"2023-10-16T13:48:25.441305+0000","last_became_peered":"2023-10-16T13:48:25.441305+0000","last_unstale":"2023-10-16T13:48:27.395040+0000","last_undegraded":"2023-10-16T13:48:27.395040+0000","last_fullsized":"2023-10-16T13:48:27.395040+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:21:57.810695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.13","version":"264'2","reported_seq":14,"reported_epoch":264,"state":"active+clean","last_fresh":"2023-10-16T13:48:25.454665+0000","last_change":"2023-10-16T13:48:24.426307+0000","last_active":"2023-10-16T13:48:25.454665+0000","last_peered":"2023-10-16T13:48:25.454665+0000","last_clean":"2023-10-16T13:48:25.454665+0000","last_became_active":"2023-10-16T13:48:24.425993+0000","last_became_peered":"2023-10-16T13:48:24.425993+0000","last_unstale":"2023-10-16T13:48:25.454665+0000","last_undegraded":"2023-10-16T13:48:25.454665+0000","last_fullsized":"2023-10-16T13:48:25.454665+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_clean_scrub_stamp":"2023-10-16T13:48:23.395571+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:51:50.387274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.15","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.394429+0000","last_change":"2023-10-16T13:48:25.440326+0000","last_active":"2023-10-16T13:48:27.394429+0000","last_peered":"2023-10-16T13:48:27.394429+0000","last_clean":"2023-10-16T13:48:27.394429+0000","last_became_active":"2023-10-16T13:48:25.440012+0000","last_became_peered":"2023-10-16T13:48:25.440012+0000","last_unstale":"2023-10-16T13:48:27.394429+0000","last_undegraded":"2023-10-16T13:48:27.394429+0000","last_fullsized":"2023-10-16T13:48:27.394429+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:20:52.895043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.12","version":"0'0","reported_seq":12,"reported_epoch":264,"state":"active+clean","last_fresh":"2023-10-16T13:48:25.428462+0000","last_change":"2023-10-16T13:48:24.423607+0000","last_active":"2023-10-16T13:48:25.428462+0000","last_peered":"2023-10-16T13:48:25.428462+0000","last_clean":"2023-10-16T13:48:25.428462+0000","last_became_active":"2023-10-16T13:48:24.423294+0000","last_became_peered":"2023-10-16T13:48:24.423294+0000","last_unstale":"2023-10-16T13:48:25.428462+0000","last_undegraded":"2023-10-16T13:48:25.428462+0000","last_fullsized":"2023-10-16T13:48:25.428462+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_clean_scrub_stamp":"2023-10-16T13:48:23.395571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:20:38.283795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.16","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.394476+0000","last_change":"2023-10-16T13:48:25.433847+0000","last_active":"2023-10-16T13:48:27.394476+0000","last_peered":"2023-10-16T13:48:27.394476+0000","last_clean":"2023-10-16T13:48:27.394476+0000","last_became_active":"2023-10-16T13:48:25.433563+0000","last_became_peered":"2023-10-16T13:48:25.433563+0000","last_unstale":"2023-10-16T13:48:27.394476+0000","last_undegraded":"2023-10-16T13:48:27.394476+0000","last_fullsized":"2023-10-16T13:48:27.394476+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:00:13.116040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.11","version":"0'0","reported_seq":12,"reported_epoch":264,"state":"active+clean","last_fresh":"2023-10-16T13:48:25.429723+0000","last_change":"2023-10-16T13:48:24.422134+0000","last_active":"2023-10-16T13:48:25.429723+0000","last_peered":"2023-10-16T13:48:25.429723+0000","last_clean":"2023-10-16T13:48:25.429723+0000","last_became_active":"2023-10-16T13:48:24.421631+0000","last_became_peered":"2023-10-16T13:48:24.421631+0000","last_unstale":"2023-10-16T13:48:25.429723+0000","last_undegraded":"2023-10-16T13:48:25.429723+0000","last_fullsized":"2023-10-16T13:48:25.429723+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_clean_scrub_stamp":"2023-10-16T13:48:23.395571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:55:02.920642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.17","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.395059+0000","last_change":"2023-10-16T13:48:25.433217+0000","last_active":"2023-10-16T13:48:27.395059+0000","last_peered":"2023-10-16T13:48:27.395059+0000","last_clean":"2023-10-16T13:48:27.395059+0000","last_became_active":"2023-10-16T13:48:25.432642+0000","last_became_peered":"2023-10-16T13:48:25.432642+0000","last_unstale":"2023-10-16T13:48:27.395059+0000","last_undegraded":"2023-10-16T13:48:27.395059+0000","last_fullsized":"2023-10-16T13:48:27.395059+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:10:05.461399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.10","version":"264'8","reported_seq":20,"reported_epoch":264,"state":"active+clean","last_fresh":"2023-10-16T13:48:25.455985+0000","last_change":"2023-10-16T13:48:24.428346+0000","last_active":"2023-10-16T13:48:25.455985+0000","last_peered":"2023-10-16T13:48:25.455985+0000","last_clean":"2023-10-16T13:48:25.455985+0000","last_became_active":"2023-10-16T13:48:24.428022+0000","last_became_peered":"2023-10-16T13:48:24.428022+0000","last_unstale":"2023-10-16T13:48:25.455985+0000","last_undegraded":"2023-10-16T13:48:25.455985+0000","last_fullsized":"2023-10-16T13:48:25.455985+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_clean_scrub_stamp":"2023-10-16T13:48:23.395571+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:09:41.055965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.8","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.393150+0000","last_change":"2023-10-16T13:48:25.426854+0000","last_active":"2023-10-16T13:48:27.393150+0000","last_peered":"2023-10-16T13:48:27.393150+0000","last_clean":"2023-10-16T13:48:27.393150+0000","last_became_active":"2023-10-16T13:48:25.426546+0000","last_became_peered":"2023-10-16T13:48:25.426546+0000","last_unstale":"2023-10-16T13:48:27.393150+0000","last_undegraded":"2023-10-16T13:48:27.393150+0000","last_fullsized":"2023-10-16T13:48:27.393150+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:35:54.607900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.f","version":"264'1","reported_seq":13,"reported_epoch":264,"state":"active+clean","last_fresh":"2023-10-16T13:48:25.455007+0000","last_change":"2023-10-16T13:48:24.418499+0000","last_active":"2023-10-16T13:48:25.455007+0000","last_peered":"2023-10-16T13:48:25.455007+0000","last_clean":"2023-10-16T13:48:25.455007+0000","last_became_active":"2023-10-16T13:48:24.418095+0000","last_became_peered":"2023-10-16T13:48:24.418095+0000","last_unstale":"2023-10-16T13:48:25.455007+0000","last_undegraded":"2023-10-16T13:48:25.455007+0000","last_fullsized":"2023-10-16T13:48:25.455007+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_clean_scrub_stamp":"2023-10-16T13:48:23.395571+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:41:15.591732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.9","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:26.421154+0000","last_change":"2023-10-16T13:48:25.443855+0000","last_active":"2023-10-16T13:48:26.421154+0000","last_peered":"2023-10-16T13:48:26.421154+0000","last_clean":"2023-10-16T13:48:26.421154+0000","last_became_active":"2023-10-16T13:48:25.439465+0000","last_became_peered":"2023-10-16T13:48:25.439465+0000","last_unstale":"2023-10-16T13:48:26.421154+0000","last_undegraded":"2023-10-16T13:48:26.421154+0000","last_fullsized":"2023-10-16T13:48:26.421154+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:33:34.431164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.e","version":"264'2","reported_seq":14,"reported_epoch":264,"state":"active+clean","last_fresh":"2023-10-16T13:48:25.452636+0000","last_change":"2023-10-16T13:48:24.423791+0000","last_active":"2023-10-16T13:48:25.452636+0000","last_peered":"2023-10-16T13:48:25.452636+0000","last_clean":"2023-10-16T13:48:25.452636+0000","last_became_active":"2023-10-16T13:48:24.423494+0000","last_became_peered":"2023-10-16T13:48:24.423494+0000","last_unstale":"2023-10-16T13:48:25.452636+0000","last_undegraded":"2023-10-16T13:48:25.452636+0000","last_fullsized":"2023-10-16T13:48:25.452636+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_clean_scrub_stamp":"2023-10-16T13:48:23.395571+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:14:31.124740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.a","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.393392+0000","last_change":"2023-10-16T13:48:25.427450+0000","last_active":"2023-10-16T13:48:27.393392+0000","last_peered":"2023-10-16T13:48:27.393392+0000","last_clean":"2023-10-16T13:48:27.393392+0000","last_became_active":"2023-10-16T13:48:25.427170+0000","last_became_peered":"2023-10-16T13:48:25.427170+0000","last_unstale":"2023-10-16T13:48:27.393392+0000","last_undegraded":"2023-10-16T13:48:27.393392+0000","last_fullsized":"2023-10-16T13:48:27.393392+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:04:33.594311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.d","version":"264'1","reported_seq":13,"reported_epoch":264,"state":"active+clean","last_fresh":"2023-10-16T13:48:25.454791+0000","last_change":"2023-10-16T13:48:24.422463+0000","last_active":"2023-10-16T13:48:25.454791+0000","last_peered":"2023-10-16T13:48:25.454791+0000","last_clean":"2023-10-16T13:48:25.454791+0000","last_became_active":"2023-10-16T13:48:24.421831+0000","last_became_peered":"2023-10-16T13:48:24.421831+0000","last_unstale":"2023-10-16T13:48:25.454791+0000","last_undegraded":"2023-10-16T13:48:25.454791+0000","last_fullsized":"2023-10-16T13:48:25.454791+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_clean_scrub_stamp":"2023-10-16T13:48:23.395571+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:26:00.824106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.b","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.394177+0000","last_change":"2023-10-16T13:48:25.438352+0000","last_active":"2023-10-16T13:48:27.394177+0000","last_peered":"2023-10-16T13:48:27.394177+0000","last_clean":"2023-10-16T13:48:27.394177+0000","last_became_active":"2023-10-16T13:48:25.438027+0000","last_became_peered":"2023-10-16T13:48:25.438027+0000","last_unstale":"2023-10-16T13:48:27.394177+0000","last_undegraded":"2023-10-16T13:48:27.394177+0000","last_fullsized":"2023-10-16T13:48:27.394177+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:50:53.215873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.c","version":"0'0","reported_seq":12,"reported_epoch":264,"state":"active+clean","last_fresh":"2023-10-16T13:48:25.422405+0000","last_change":"2023-10-16T13:48:24.421504+0000","last_active":"2023-10-16T13:48:25.422405+0000","last_peered":"2023-10-16T13:48:25.422405+0000","last_clean":"2023-10-16T13:48:25.422405+0000","last_became_active":"2023-10-16T13:48:24.421202+0000","last_became_peered":"2023-10-16T13:48:24.421202+0000","last_unstale":"2023-10-16T13:48:25.422405+0000","last_undegraded":"2023-10-16T13:48:25.422405+0000","last_fullsized":"2023-10-16T13:48:25.422405+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_clean_scrub_stamp":"2023-10-16T13:48:23.395571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:08:57.374827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.c","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:26.421401+0000","last_change":"2023-10-16T13:48:25.444963+0000","last_active":"2023-10-16T13:48:26.421401+0000","last_peered":"2023-10-16T13:48:26.421401+0000","last_clean":"2023-10-16T13:48:26.421401+0000","last_became_active":"2023-10-16T13:48:25.444659+0000","last_became_peered":"2023-10-16T13:48:25.444659+0000","last_unstale":"2023-10-16T13:48:26.421401+0000","last_undegraded":"2023-10-16T13:48:26.421401+0000","last_fullsized":"2023-10-16T13:48:26.421401+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:08:59.756642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.b","version":"264'2","reported_seq":14,"reported_epoch":264,"state":"active+clean","last_fresh":"2023-10-16T13:48:25.454708+0000","last_change":"2023-10-16T13:48:24.415442+0000","last_active":"2023-10-16T13:48:25.454708+0000","last_peered":"2023-10-16T13:48:25.454708+0000","last_clean":"2023-10-16T13:48:25.454708+0000","last_became_active":"2023-10-16T13:48:24.415108+0000","last_became_peered":"2023-10-16T13:48:24.415108+0000","last_unstale":"2023-10-16T13:48:25.454708+0000","last_undegraded":"2023-10-16T13:48:25.454708+0000","last_fullsized":"2023-10-16T13:48:25.454708+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_clean_scrub_stamp":"2023-10-16T13:48:23.395571+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:00:09.744808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.d","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:26.422403+0000","last_change":"2023-10-16T13:48:25.435788+0000","last_active":"2023-10-16T13:48:26.422403+0000","last_peered":"2023-10-16T13:48:26.422403+0000","last_clean":"2023-10-16T13:48:26.422403+0000","last_became_active":"2023-10-16T13:48:25.435458+0000","last_became_peered":"2023-10-16T13:48:25.435458+0000","last_unstale":"2023-10-16T13:48:26.422403+0000","last_undegraded":"2023-10-16T13:48:26.422403+0000","last_fullsized":"2023-10-16T13:48:26.422403+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:18:02.241801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.a","version":"0'0","reported_seq":12,"reported_epoch":264,"state":"active+clean","last_fresh":"2023-10-16T13:48:25.428892+0000","last_change":"2023-10-16T13:48:24.429991+0000","last_active":"2023-10-16T13:48:25.428892+0000","last_peered":"2023-10-16T13:48:25.428892+0000","last_clean":"2023-10-16T13:48:25.428892+0000","last_became_active":"2023-10-16T13:48:24.429622+0000","last_became_peered":"2023-10-16T13:48:24.429622+0000","last_unstale":"2023-10-16T13:48:25.428892+0000","last_undegraded":"2023-10-16T13:48:25.428892+0000","last_fullsized":"2023-10-16T13:48:25.428892+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_clean_scrub_stamp":"2023-10-16T13:48:23.395571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:47:29.855885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.e","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.394644+0000","last_change":"2023-10-16T13:48:25.436515+0000","last_active":"2023-10-16T13:48:27.394644+0000","last_peered":"2023-10-16T13:48:27.394644+0000","last_clean":"2023-10-16T13:48:27.394644+0000","last_became_active":"2023-10-16T13:48:25.436207+0000","last_became_peered":"2023-10-16T13:48:25.436207+0000","last_unstale":"2023-10-16T13:48:27.394644+0000","last_undegraded":"2023-10-16T13:48:27.394644+0000","last_fullsized":"2023-10-16T13:48:27.394644+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:40:28.418592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.9","version":"0'0","reported_seq":12,"reported_epoch":264,"state":"active+clean","last_fresh":"2023-10-16T13:48:25.435597+0000","last_change":"2023-10-16T13:48:24.425322+0000","last_active":"2023-10-16T13:48:25.435597+0000","last_peered":"2023-10-16T13:48:25.435597+0000","last_clean":"2023-10-16T13:48:25.435597+0000","last_became_active":"2023-10-16T13:48:24.424311+0000","last_became_peered":"2023-10-16T13:48:24.424311+0000","last_unstale":"2023-10-16T13:48:25.435597+0000","last_undegraded":"2023-10-16T13:48:25.435597+0000","last_fullsized":"2023-10-16T13:48:25.435597+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_clean_scrub_stamp":"2023-10-16T13:48:23.395571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:12:15.257850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.f","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.393378+0000","last_change":"2023-10-16T13:48:25.444978+0000","last_active":"2023-10-16T13:48:27.393378+0000","last_peered":"2023-10-16T13:48:27.393378+0000","last_clean":"2023-10-16T13:48:27.393378+0000","last_became_active":"2023-10-16T13:48:25.444683+0000","last_became_peered":"2023-10-16T13:48:25.444683+0000","last_unstale":"2023-10-16T13:48:27.393378+0000","last_undegraded":"2023-10-16T13:48:27.393378+0000","last_fullsized":"2023-10-16T13:48:27.393378+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:30:42.136595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.8","version":"0'0","reported_seq":12,"reported_epoch":264,"state":"active+clean","last_fresh":"2023-10-16T13:48:25.423666+0000","last_change":"2023-10-16T13:48:24.426952+0000","last_active":"2023-10-16T13:48:25.423666+0000","last_peered":"2023-10-16T13:48:25.423666+0000","last_clean":"2023-10-16T13:48:25.423666+0000","last_became_active":"2023-10-16T13:48:24.426401+0000","last_became_peered":"2023-10-16T13:48:24.426401+0000","last_unstale":"2023-10-16T13:48:25.423666+0000","last_undegraded":"2023-10-16T13:48:25.423666+0000","last_fullsized":"2023-10-16T13:48:25.423666+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_clean_scrub_stamp":"2023-10-16T13:48:23.395571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:58:42.161578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.0","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.393496+0000","last_change":"2023-10-16T13:48:25.438817+0000","last_active":"2023-10-16T13:48:27.393496+0000","last_peered":"2023-10-16T13:48:27.393496+0000","last_clean":"2023-10-16T13:48:27.393496+0000","last_became_active":"2023-10-16T13:48:25.438538+0000","last_became_peered":"2023-10-16T13:48:25.438538+0000","last_unstale":"2023-10-16T13:48:27.393496+0000","last_undegraded":"2023-10-16T13:48:27.393496+0000","last_fullsized":"2023-10-16T13:48:27.393496+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:24:08.987676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.7","version":"264'2","reported_seq":14,"reported_epoch":264,"state":"active+clean","last_fresh":"2023-10-16T13:48:25.454312+0000","last_change":"2023-10-16T13:48:24.418531+0000","last_active":"2023-10-16T13:48:25.454312+0000","last_peered":"2023-10-16T13:48:25.454312+0000","last_clean":"2023-10-16T13:48:25.454312+0000","last_became_active":"2023-10-16T13:48:24.418119+0000","last_became_peered":"2023-10-16T13:48:24.418119+0000","last_unstale":"2023-10-16T13:48:25.454312+0000","last_undegraded":"2023-10-16T13:48:25.454312+0000","last_fullsized":"2023-10-16T13:48:25.454312+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_clean_scrub_stamp":"2023-10-16T13:48:23.395571+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:17:02.382002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.1","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.393530+0000","last_change":"2023-10-16T13:48:25.438726+0000","last_active":"2023-10-16T13:48:27.393530+0000","last_peered":"2023-10-16T13:48:27.393530+0000","last_clean":"2023-10-16T13:48:27.393530+0000","last_became_active":"2023-10-16T13:48:25.435116+0000","last_became_peered":"2023-10-16T13:48:25.435116+0000","last_unstale":"2023-10-16T13:48:27.393530+0000","last_undegraded":"2023-10-16T13:48:27.393530+0000","last_fullsized":"2023-10-16T13:48:27.393530+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:42:01.034277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.6","version":"0'0","reported_seq":12,"reported_epoch":264,"state":"active+clean","last_fresh":"2023-10-16T13:48:25.429482+0000","last_change":"2023-10-16T13:48:24.423801+0000","last_active":"2023-10-16T13:48:25.429482+0000","last_peered":"2023-10-16T13:48:25.429482+0000","last_clean":"2023-10-16T13:48:25.429482+0000","last_became_active":"2023-10-16T13:48:24.423451+0000","last_became_peered":"2023-10-16T13:48:24.423451+0000","last_unstale":"2023-10-16T13:48:25.429482+0000","last_undegraded":"2023-10-16T13:48:25.429482+0000","last_fullsized":"2023-10-16T13:48:25.429482+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_clean_scrub_stamp":"2023-10-16T13:48:23.395571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:41:33.359406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.2","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.393722+0000","last_change":"2023-10-16T13:48:25.445560+0000","last_active":"2023-10-16T13:48:27.393722+0000","last_peered":"2023-10-16T13:48:27.393722+0000","last_clean":"2023-10-16T13:48:27.393722+0000","last_became_active":"2023-10-16T13:48:25.445064+0000","last_became_peered":"2023-10-16T13:48:25.445064+0000","last_unstale":"2023-10-16T13:48:27.393722+0000","last_undegraded":"2023-10-16T13:48:27.393722+0000","last_fullsized":"2023-10-16T13:48:27.393722+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:57:58.737918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.5","version":"0'0","reported_seq":12,"reported_epoch":264,"state":"active+clean","last_fresh":"2023-10-16T13:48:25.426272+0000","last_change":"2023-10-16T13:48:24.429667+0000","last_active":"2023-10-16T13:48:25.426272+0000","last_peered":"2023-10-16T13:48:25.426272+0000","last_clean":"2023-10-16T13:48:25.426272+0000","last_became_active":"2023-10-16T13:48:24.429339+0000","last_became_peered":"2023-10-16T13:48:24.429339+0000","last_unstale":"2023-10-16T13:48:25.426272+0000","last_undegraded":"2023-10-16T13:48:25.426272+0000","last_fullsized":"2023-10-16T13:48:25.426272+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_clean_scrub_stamp":"2023-10-16T13:48:23.395571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:27:40.075970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.3","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.164453+0000","last_change":"2023-10-16T13:48:25.433691+0000","last_active":"2023-10-16T13:48:27.164453+0000","last_peered":"2023-10-16T13:48:27.164453+0000","last_clean":"2023-10-16T13:48:27.164453+0000","last_became_active":"2023-10-16T13:48:25.433385+0000","last_became_peered":"2023-10-16T13:48:25.433385+0000","last_unstale":"2023-10-16T13:48:27.164453+0000","last_undegraded":"2023-10-16T13:48:27.164453+0000","last_fullsized":"2023-10-16T13:48:27.164453+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:06:27.452752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.4","version":"0'0","reported_seq":12,"reported_epoch":264,"state":"active+clean","last_fresh":"2023-10-16T13:48:25.429614+0000","last_change":"2023-10-16T13:48:24.423134+0000","last_active":"2023-10-16T13:48:25.429614+0000","last_peered":"2023-10-16T13:48:25.429614+0000","last_clean":"2023-10-16T13:48:25.429614+0000","last_became_active":"2023-10-16T13:48:24.422846+0000","last_became_peered":"2023-10-16T13:48:24.422846+0000","last_unstale":"2023-10-16T13:48:25.429614+0000","last_undegraded":"2023-10-16T13:48:25.429614+0000","last_fullsized":"2023-10-16T13:48:25.429614+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_clean_scrub_stamp":"2023-10-16T13:48:23.395571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:47:35.226854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.4","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.164713+0000","last_change":"2023-10-16T13:48:25.435062+0000","last_active":"2023-10-16T13:48:27.164713+0000","last_peered":"2023-10-16T13:48:27.164713+0000","last_clean":"2023-10-16T13:48:27.164713+0000","last_became_active":"2023-10-16T13:48:25.433884+0000","last_became_peered":"2023-10-16T13:48:25.433884+0000","last_unstale":"2023-10-16T13:48:27.164713+0000","last_undegraded":"2023-10-16T13:48:27.164713+0000","last_fullsized":"2023-10-16T13:48:27.164713+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:30:42.136595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.3","version":"0'0","reported_seq":12,"reported_epoch":264,"state":"active+clean","last_fresh":"2023-10-16T13:48:25.422718+0000","last_change":"2023-10-16T13:48:24.423099+0000","last_active":"2023-10-16T13:48:25.422718+0000","last_peered":"2023-10-16T13:48:25.422718+0000","last_clean":"2023-10-16T13:48:25.422718+0000","last_became_active":"2023-10-16T13:48:24.422766+0000","last_became_peered":"2023-10-16T13:48:24.422766+0000","last_unstale":"2023-10-16T13:48:25.422718+0000","last_undegraded":"2023-10-16T13:48:25.422718+0000","last_fullsized":"2023-10-16T13:48:25.422718+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_clean_scrub_stamp":"2023-10-16T13:48:23.395571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:06:49.269879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.5","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.394496+0000","last_change":"2023-10-16T13:48:25.434071+0000","last_active":"2023-10-16T13:48:27.394496+0000","last_peered":"2023-10-16T13:48:27.394496+0000","last_clean":"2023-10-16T13:48:27.394496+0000","last_became_active":"2023-10-16T13:48:25.431980+0000","last_became_peered":"2023-10-16T13:48:25.431980+0000","last_unstale":"2023-10-16T13:48:27.394496+0000","last_undegraded":"2023-10-16T13:48:27.394496+0000","last_fullsized":"2023-10-16T13:48:27.394496+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:38:43.055399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.2","version":"0'0","reported_seq":12,"reported_epoch":264,"state":"active+clean","last_fresh":"2023-10-16T13:48:25.430077+0000","last_change":"2023-10-16T13:48:24.423228+0000","last_active":"2023-10-16T13:48:25.430077+0000","last_peered":"2023-10-16T13:48:25.430077+0000","last_clean":"2023-10-16T13:48:25.430077+0000","last_became_active":"2023-10-16T13:48:24.422695+0000","last_became_peered":"2023-10-16T13:48:24.422695+0000","last_unstale":"2023-10-16T13:48:25.430077+0000","last_undegraded":"2023-10-16T13:48:25.430077+0000","last_fullsized":"2023-10-16T13:48:25.430077+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_clean_scrub_stamp":"2023-10-16T13:48:23.395571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:27:43.879103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.6","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.393740+0000","last_change":"2023-10-16T13:48:25.433978+0000","last_active":"2023-10-16T13:48:27.393740+0000","last_peered":"2023-10-16T13:48:27.393740+0000","last_clean":"2023-10-16T13:48:27.393740+0000","last_became_active":"2023-10-16T13:48:25.433676+0000","last_became_peered":"2023-10-16T13:48:25.433676+0000","last_unstale":"2023-10-16T13:48:27.393740+0000","last_undegraded":"2023-10-16T13:48:27.393740+0000","last_fullsized":"2023-10-16T13:48:27.393740+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:08:24.915543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.1","version":"0'0","reported_seq":12,"reported_epoch":264,"state":"active+clean","last_fresh":"2023-10-16T13:48:25.432357+0000","last_change":"2023-10-16T13:48:24.430561+0000","last_active":"2023-10-16T13:48:25.432357+0000","last_peered":"2023-10-16T13:48:25.432357+0000","last_clean":"2023-10-16T13:48:25.432357+0000","last_became_active":"2023-10-16T13:48:24.430259+0000","last_became_peered":"2023-10-16T13:48:24.430259+0000","last_unstale":"2023-10-16T13:48:25.432357+0000","last_undegraded":"2023-10-16T13:48:25.432357+0000","last_fullsized":"2023-10-16T13:48:25.432357+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_clean_scrub_stamp":"2023-10-16T13:48:23.395571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:02:31.781056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.7","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.394456+0000","last_change":"2023-10-16T13:48:25.445845+0000","last_active":"2023-10-16T13:48:27.394456+0000","last_peered":"2023-10-16T13:48:27.394456+0000","last_clean":"2023-10-16T13:48:27.394456+0000","last_became_active":"2023-10-16T13:48:25.445533+0000","last_became_peered":"2023-10-16T13:48:25.445533+0000","last_unstale":"2023-10-16T13:48:27.394456+0000","last_undegraded":"2023-10-16T13:48:27.394456+0000","last_fullsized":"2023-10-16T13:48:27.394456+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:22:49.036856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.0","version":"0'0","reported_seq":12,"reported_epoch":264,"state":"active+clean","last_fresh":"2023-10-16T13:48:25.423576+0000","last_change":"2023-10-16T13:48:24.426714+0000","last_active":"2023-10-16T13:48:25.423576+0000","last_peered":"2023-10-16T13:48:25.423576+0000","last_clean":"2023-10-16T13:48:25.423576+0000","last_became_active":"2023-10-16T13:48:24.425908+0000","last_became_peered":"2023-10-16T13:48:24.425908+0000","last_unstale":"2023-10-16T13:48:25.423576+0000","last_undegraded":"2023-10-16T13:48:25.423576+0000","last_fullsized":"2023-10-16T13:48:25.423576+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_clean_scrub_stamp":"2023-10-16T13:48:23.395571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:10:48.742062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.25","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.393896+0000","last_change":"2023-10-16T13:48:25.424261+0000","last_active":"2023-10-16T13:48:27.393896+0000","last_peered":"2023-10-16T13:48:27.393896+0000","last_clean":"2023-10-16T13:48:27.393896+0000","last_became_active":"2023-10-16T13:48:25.423811+0000","last_became_peered":"2023-10-16T13:48:25.423811+0000","last_unstale":"2023-10-16T13:48:27.393896+0000","last_undegraded":"2023-10-16T13:48:27.393896+0000","last_fullsized":"2023-10-16T13:48:27.393896+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:50:46.522916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"251'381","reported_seq":1355,"reported_epoch":264,"state":"active+clean","last_fresh":"2023-10-16T13:48:25.424566+0000","last_change":"2023-10-16T12:10:34.746820+0000","last_active":"2023-10-16T13:48:25.424566+0000","last_peered":"2023-10-16T13:48:25.424566+0000","last_clean":"2023-10-16T13:48:25.424566+0000","last_became_active":"2023-10-16T12:10:34.745574+0000","last_became_peered":"2023-10-16T12:10:34.745574+0000","last_unstale":"2023-10-16T13:48:25.424566+0000","last_undegraded":"2023-10-16T13:48:25.424566+0000","last_fullsized":"2023-10-16T13:48:25.424566+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_clean_scrub_stamp":"2023-10-16T12:10:33.577111+0000","objects_scrubbed":0,"log_size":381,"log_dups_size":0,"ondisk_log_size":381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:14.461986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2420,"num_read_kb":5044,"num_write":639,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":35,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":36,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2420,"num_read_kb":5044,"num_write":639,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":381,"ondisk_log_size":381,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576007,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57264,"kb_used_data":3872,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314576,"statfs":{"total":96636764160,"available":96578125824,"internally_reserved":0,"allocated":3964928,"data_stored":2332951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54655435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:47:51 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.57599999999999996,"15min":0.621},"min":{"1min":0.38200000000000001,"5min":0.34200000000000003,"15min":0.33000000000000002},"max":{"1min":0.76600000000000001,"5min":0.87,"15min":0.93799999999999994},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.58599999999999997,"15min":0.61799999999999999},"min":{"1min":0.34899999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.79700000000000004,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.51300000000000001}]},{"osd":1,"last update":"Mon Oct 16 13:47:51 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.59499999999999997,"15min":0.63200000000000001},"min":{"1min":0.35999999999999999,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.78300000000000003,"5min":0.80400000000000005,"15min":0.95899999999999996},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.60099999999999998,"15min":0.624},"min":{"1min":0.39800000000000002,"5min":0.375,"15min":0.375},"max":{"1min":0.76400000000000001,"5min":0.78700000000000003,"15min":0.86399999999999999},"last":0.59299999999999997}]},{"osd":2,"last update":"Mon Oct 16 13:47:51 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.65400000000000003,"15min":0.67800000000000005},"min":{"1min":0.44900000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.78300000000000003,"5min":0.83899999999999997,"15min":0.88200000000000001},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.66700000000000004,"15min":0.69099999999999995},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.81899999999999995,"5min":0.90200000000000002,"15min":0.93799999999999994},"last":0.78300000000000003}]},{"osd":3,"last update":"Mon Oct 16 13:47:51 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.623,"15min":0.65100000000000002},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.85199999999999998,"5min":0.85299999999999998,"15min":0.91300000000000003},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.64600000000000002,"15min":0.65900000000000003},"min":{"1min":0.48099999999999998,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.86499999999999999,"5min":1.24,"15min":1.24},"last":0.46200000000000002}]},{"osd":4,"last update":"Mon Oct 16 13:47:51 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.33200000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.435},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.63100000000000001,"15min":0.621},"min":{"1min":0.45600000000000002,"5min":0.34399999999999997,"15min":0.33100000000000002},"max":{"1min":0.93799999999999994,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.81200000000000006}]},{"osd":5,"last update":"Mon Oct 16 13:47:51 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.56799999999999995,"15min":0.55400000000000005},"min":{"1min":0.35899999999999999,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.998,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.60499999999999998,"15min":0.59399999999999997},"min":{"1min":0.42799999999999999,"5min":0.39700000000000002,"15min":0.35099999999999998},"max":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.52700000000000002}]},{"osd":6,"last update":"Mon Oct 16 13:47:51 2023","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.62,"15min":0.61899999999999999},"min":{"1min":0.48499999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.022,"5min":1.022,"15min":1.022},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64300000000000002,"15min":0.623},"min":{"1min":0.38,"5min":0.378,"15min":0.35199999999999998},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.753}]}]},{"osd":6,"up_from":13,"seq":55834576005,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64980,"kb_used_data":3272,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94306860,"statfs":{"total":96636764160,"available":96570224640,"internally_reserved":0,"allocated":3350528,"data_stored":1740319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":63175111},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:47:39 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.64100000000000001,"15min":0.66300000000000003},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.29599999999999999},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.63700000000000001,"15min":0.65800000000000003},"min":{"1min":0.36499999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.81100000000000005,"5min":0.91400000000000003,"15min":0.93799999999999994},"last":0.50800000000000001}]},{"osd":1,"last update":"Mon Oct 16 13:47:39 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.57099999999999995,"15min":0.61799999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.76600000000000001,"5min":0.86899999999999999,"15min":0.90900000000000003},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.61599999999999999,"15min":0.61699999999999999},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.41199999999999998}]},{"osd":2,"last update":"Mon Oct 16 13:47:39 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.60799999999999998,"15min":0.627},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.79600000000000004,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.59699999999999998,"15min":0.61799999999999999},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.2769999999999999,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.39400000000000002}]},{"osd":3,"last update":"Mon Oct 16 13:47:39 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.625,"15min":0.65600000000000003},"min":{"1min":0.40100000000000002,"5min":0.39900000000000002,"15min":0.38200000000000001},"max":{"1min":0.75600000000000001,"5min":0.92400000000000004,"15min":0.95599999999999996},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.68400000000000005,"15min":0.71199999999999997},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.42299999999999999},"max":{"1min":0.85199999999999998,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.52500000000000002}]},{"osd":4,"last update":"Mon Oct 16 13:47:39 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.52500000000000002,"15min":0.53000000000000003},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.748,"5min":0.85599999999999998,"15min":0.95799999999999996},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.61199999999999999,"15min":0.621},"min":{"1min":0.38400000000000001,"5min":0.35499999999999998,"15min":0.32900000000000001},"max":{"1min":0.79100000000000004,"5min":0.98799999999999999,"15min":1.0309999999999999},"last":0.56399999999999995}]},{"osd":5,"last update":"Mon Oct 16 13:47:39 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.58799999999999997,"15min":0.59499999999999997},"min":{"1min":0.34399999999999997,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.80000000000000004,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.64000000000000001,"15min":0.65900000000000003},"min":{"1min":0.36099999999999999,"5min":0.33700000000000002,"15min":0.308},"max":{"1min":0.86399999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.63800000000000001}]},{"osd":7,"last update":"Mon Oct 16 13:47:39 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.58899999999999997,"15min":0.60299999999999998},"min":{"1min":0.38700000000000001,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.79300000000000004,"5min":1.081,"15min":1.081},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.63,"5min":0.64100000000000001,"15min":0.65400000000000003},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.375},"max":{"1min":0.83499999999999996,"5min":1.099,"15min":1.099},"last":0.54500000000000004}]}]},{"osd":1,"up_from":13,"seq":55834576005,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52504,"kb_used_data":3276,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319336,"statfs":{"total":96636764160,"available":96583000064,"internally_reserved":0,"allocated":3354624,"data_stored":1740409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":50395593},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:47:55 2023","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.38100000000000001,"15min":0.40400000000000003},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.59199999999999997,"5min":0.65400000000000003,"15min":0.65800000000000003},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.45600000000000002,"15min":0.48699999999999999},"min":{"1min":0.29099999999999998,"5min":0.253,"15min":0.253},"max":{"1min":0.67000000000000004,"5min":0.70999999999999996,"15min":0.77200000000000002},"last":0.38500000000000001}]},{"osd":2,"last update":"Mon Oct 16 13:47:55 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.41699999999999998,"15min":0.44600000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.57799999999999996,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.44,"5min":0.46200000000000002,"15min":0.48599999999999999},"min":{"1min":0.308,"5min":0.308,"15min":0.30199999999999999},"max":{"1min":0.65200000000000002,"5min":0.67600000000000005,"15min":0.71399999999999997},"last":0.28799999999999998}]},{"osd":3,"last update":"Mon Oct 16 13:47:55 2023","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.41099999999999998,"15min":0.42499999999999999},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.53200000000000003,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.439,"15min":0.47099999999999997},"min":{"1min":0.27100000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.69399999999999995,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.36599999999999999}]},{"osd":4,"last update":"Mon Oct 16 13:47:55 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.63600000000000001,"15min":0.66000000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.626,"15min":0.66200000000000003},"min":{"1min":0.39900000000000002,"5min":0.37,"15min":0.37},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.997},"last":0.69299999999999995}]},{"osd":5,"last update":"Mon Oct 16 13:47:55 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.67000000000000004,"15min":0.70499999999999996},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":1.0760000000000001},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.68500000000000005,"15min":0.72099999999999997},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.754}]},{"osd":6,"last update":"Mon Oct 16 13:47:55 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.65200000000000002,"15min":0.67700000000000005},"min":{"1min":0.41199999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.80300000000000005,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.67100000000000004,"15min":0.70199999999999996},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.88100000000000001,"5min":0.998,"15min":1.083},"last":0.61099999999999999}]},{"osd":7,"last update":"Mon Oct 16 13:47:55 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.69099999999999995,"15min":0.72899999999999998},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.163,"5min":1.163,"15min":1.163},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.71299999999999997,"15min":0.74299999999999999},"min":{"1min":0.47599999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.92500000000000004,"5min":1.042,"15min":1.0580000000000001},"last":0.76900000000000002}]}]},{"osd":0,"up_from":13,"seq":55834576006,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48928,"kb_used_data":3860,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94322912,"statfs":{"total":96636764160,"available":96586661888,"internally_reserved":0,"allocated":3952640,"data_stored":2331303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 13:47:43 2023","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.39300000000000002,"15min":0.41599999999999998},"min":{"1min":0.245,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.51400000000000001,"5min":0.65100000000000002,"15min":0.71399999999999997},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.40200000000000002,"15min":0.43099999999999999},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.41999999999999998,"5min":0.77700000000000002,"15min":0.82199999999999995},"last":0.29599999999999999}]},{"osd":2,"last update":"Mon Oct 16 13:47:43 2023","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.43099999999999999,"15min":0.44800000000000001},"min":{"1min":0.32000000000000001,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.45900000000000002,"5min":0.65900000000000003,"15min":0.75700000000000001},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.44800000000000001,"15min":0.47899999999999998},"min":{"1min":0.30499999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.60499999999999998,"5min":0.68799999999999994,"15min":0.83299999999999996},"last":0.38200000000000001}]},{"osd":3,"last update":"Mon Oct 16 13:47:43 2023","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.41999999999999998,"15min":0.44900000000000001},"min":{"1min":0.28100000000000003,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.78600000000000003},"last":0.318},{"interface":"front","average":{"1min":0.432,"5min":0.49199999999999999,"15min":0.51400000000000001},"min":{"1min":0.309,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.58799999999999997,"5min":0.82999999999999996,"15min":0.86399999999999999},"last":0.371}]},{"osd":4,"last update":"Mon Oct 16 13:47:43 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.65800000000000003,"15min":0.68000000000000005},"min":{"1min":0.442,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.85799999999999998,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.66300000000000003,"15min":0.68700000000000006},"min":{"1min":0.437,"5min":0.375,"15min":0.375},"max":{"1min":0.90500000000000003,"5min":1.21,"15min":1.21},"last":0.69699999999999995}]},{"osd":5,"last update":"Mon Oct 16 13:47:43 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.65100000000000002,"15min":0.68200000000000005},"min":{"1min":0.46999999999999997,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.81399999999999995,"5min":1.242,"15min":1.242},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.71099999999999997,"15min":0.72199999999999998},"min":{"1min":0.59499999999999997,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.95699999999999996,"5min":1.081,"15min":1.081},"last":0.71399999999999997}]},{"osd":6,"last update":"Mon Oct 16 13:47:43 2023","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.70699999999999996,"15min":0.72599999999999998},"min":{"1min":0.52000000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.90300000000000002,"5min":1.034,"15min":1.103},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.68899999999999995,"15min":0.71799999999999997},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.79600000000000004,"5min":1.131,"15min":1.131},"last":0.68500000000000005}]},{"osd":7,"last update":"Mon Oct 16 13:47:43 2023","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.70599999999999996,"15min":0.73799999999999999},"min":{"1min":0.48799999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.88800000000000001,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.70999999999999996,"15min":0.73099999999999998},"min":{"1min":0.53700000000000003,"5min":0.433,"15min":0.433},"max":{"1min":0.80800000000000005,"5min":1.087,"15min":1.087},"last":0.66600000000000004}]}]},{"osd":2,"up_from":13,"seq":55834576006,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56664,"kb_used_data":3272,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315176,"statfs":{"total":96636764160,"available":96578740224,"internally_reserved":0,"allocated":3350528,"data_stored":1740319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.39800000000000002,"15min":0.40699999999999997},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.54100000000000004,"5min":0.71699999999999997,"15min":0.77500000000000002},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.45800000000000002,"15min":0.47599999999999998},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.69799999999999995,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.40600000000000003}]},{"osd":1,"last update":"Mon Oct 16 13:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.47899999999999998,"15min":0.495},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.61899999999999999,"5min":0.73199999999999998,"15min":0.81399999999999995},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.47099999999999997,"15min":0.47299999999999998},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.59799999999999998,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.315}]},{"osd":3,"last update":"Mon Oct 16 13:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.432,"15min":0.438},"min":{"1min":0.26300000000000001,"5min":0.255,"15min":0.255},"max":{"1min":0.63,"5min":0.70099999999999996,"15min":0.72599999999999998},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.52400000000000002,"15min":0.54100000000000004},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.74099999999999999,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.44}]},{"osd":4,"last update":"Mon Oct 16 13:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.69299999999999995,"15min":0.71199999999999997},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.94399999999999995,"5min":1.3600000000000001,"15min":1.3600000000000001},"last":0.627},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.70399999999999996,"15min":0.71599999999999997},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.196,"5min":1.403,"15min":1.403},"last":0.60599999999999998}]},{"osd":5,"last update":"Mon Oct 16 13:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.65900000000000003,"15min":0.68400000000000005},"min":{"1min":0.46700000000000003,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.82999999999999996,"5min":1.486,"15min":1.486},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.68200000000000005,"15min":0.69599999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.879,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":0.65300000000000002}]},{"osd":6,"last update":"Mon Oct 16 13:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.65800000000000003,"15min":0.66500000000000004},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.087,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.69799999999999995,"15min":0.71799999999999997},"min":{"1min":0.46000000000000002,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.94299999999999995,"5min":1.4379999999999999,"15min":1.4379999999999999},"last":0.51400000000000001}]},{"osd":7,"last update":"Mon Oct 16 13:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.70499999999999996,"15min":0.71499999999999997},"min":{"1min":0.42599999999999999,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.97099999999999997,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":0.503},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.70299999999999996,"15min":0.72899999999999998},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.0429999999999999,"5min":1.377,"15min":1.377},"last":0.56999999999999995}]}]},{"osd":3,"up_from":13,"seq":55834576005,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56672,"kb_used_data":3280,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315168,"statfs":{"total":96636764160,"available":96578732032,"internally_reserved":0,"allocated":3358720,"data_stored":1740411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:47:42 2023","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.39300000000000002,"15min":0.40699999999999997},"min":{"1min":0.27400000000000002,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.60099999999999998,"5min":0.64200000000000002,"15min":0.72899999999999998},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.40000000000000002,"15min":0.42099999999999999},"min":{"1min":0.27000000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.47899999999999998,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.34999999999999998}]},{"osd":1,"last update":"Mon Oct 16 13:47:42 2023","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.40799999999999997,"15min":0.42699999999999999},"min":{"1min":0.27100000000000002,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.46200000000000002,"5min":0.68200000000000005,"15min":0.76300000000000001},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.44600000000000001,"15min":0.47499999999999998},"min":{"1min":0.309,"5min":0.253,"15min":0.253},"max":{"1min":0.65400000000000003,"5min":0.70699999999999996,"15min":0.77800000000000002},"last":0.55000000000000004}]},{"osd":2,"last update":"Mon Oct 16 13:47:42 2023","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.46600000000000003,"15min":0.48499999999999999},"min":{"1min":0.317,"5min":0.316,"15min":0.316},"max":{"1min":0.64700000000000002,"5min":1.167,"15min":1.167},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.48399999999999999,"15min":0.51400000000000001},"min":{"1min":0.32400000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.58099999999999996,"5min":0.73299999999999998,"15min":0.78700000000000003},"last":0.50900000000000001}]},{"osd":4,"last update":"Mon Oct 16 13:47:42 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.626,"15min":0.65800000000000003},"min":{"1min":0.46000000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.69399999999999995,"5min":0.998,"15min":1.137},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.66600000000000004,"15min":0.68899999999999995},"min":{"1min":0.44600000000000001,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.86299999999999999,"5min":1.2110000000000001,"15min":1.2909999999999999},"last":0.63600000000000001}]},{"osd":5,"last update":"Mon Oct 16 13:47:42 2023","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.65700000000000003,"15min":0.69999999999999996},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.374},"max":{"1min":0.85099999999999998,"5min":1.0680000000000001,"15min":1.2729999999999999},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.66900000000000004,"15min":0.69399999999999995},"min":{"1min":0.48799999999999999,"5min":0.45600000000000002,"15min":0.432},"max":{"1min":0.77400000000000002,"5min":1.0549999999999999,"15min":1.173},"last":0.56899999999999995}]},{"osd":6,"last update":"Mon Oct 16 13:47:42 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.70999999999999996,"15min":0.71399999999999997},"min":{"1min":0.434,"5min":0.42999999999999999,"15min":0.41799999999999998},"max":{"1min":0.80900000000000005,"5min":5.0999999999999996,"15min":5.0999999999999996},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.69299999999999995,"15min":0.72699999999999998},"min":{"1min":0.47399999999999998,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.89100000000000001,"5min":1.0309999999999999,"15min":1.2290000000000001},"last":0.80600000000000005}]},{"osd":7,"last update":"Mon Oct 16 13:47:42 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.69599999999999995,"15min":0.72899999999999998},"min":{"1min":0.497,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.82799999999999996,"5min":1.048,"15min":1.2569999999999999},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.62,"5min":0.70299999999999996,"15min":0.73599999999999999},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.43099999999999999},"max":{"1min":0.90400000000000003,"5min":1.262,"15min":1.262},"last":0.70499999999999996}]}]},{"osd":4,"up_from":13,"seq":55834576004,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52520,"kb_used_data":3288,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319320,"statfs":{"total":96636764160,"available":96582983680,"internally_reserved":0,"allocated":3366912,"data_stored":1742021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:47:41 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.57999999999999996,"15min":0.623},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.81599999999999995,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.60099999999999998,"15min":0.63},"min":{"1min":0.39300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.77300000000000002,"5min":0.878,"15min":0.89500000000000002},"last":0.55400000000000005}]},{"osd":1,"last update":"Mon Oct 16 13:47:41 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.60799999999999998,"15min":0.65200000000000002},"min":{"1min":0.33700000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.78900000000000003,"5min":0.877,"15min":0.877},"last":0.501},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.60899999999999999,"15min":0.63700000000000001},"min":{"1min":0.42199999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.80800000000000005,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.33300000000000002}]},{"osd":2,"last update":"Mon Oct 16 13:47:41 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.623,"15min":0.64900000000000002},"min":{"1min":0.39000000000000001,"5min":0.378,"15min":0.378},"max":{"1min":0.69399999999999995,"5min":0.89900000000000002,"15min":0.90600000000000003},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.64200000000000002,"15min":0.65900000000000003},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.77500000000000002,"5min":0.89700000000000002,"15min":0.92600000000000005},"last":0.53500000000000003}]},{"osd":3,"last update":"Mon Oct 16 13:47:41 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.61699999999999999,"15min":0.64300000000000002},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.76000000000000001,"5min":0.86599999999999999,"15min":0.99199999999999999},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.65300000000000002,"15min":0.68000000000000005},"min":{"1min":0.45600000000000002,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.93899999999999995},"last":0.59399999999999997}]},{"osd":5,"last update":"Mon Oct 16 13:47:41 2023","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.54400000000000004,"15min":0.53300000000000003},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.73799999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.58099999999999996,"15min":0.56799999999999995},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.753,"5min":0.82199999999999995,"15min":0.94899999999999995},"last":0.66400000000000003}]},{"osd":6,"last update":"Mon Oct 16 13:47:41 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.627,"15min":0.60099999999999998},"min":{"1min":0.378,"5min":0.36399999999999999,"15min":0.33200000000000002},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.70999999999999996,"15min":0.69499999999999995},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.39400000000000002},"max":{"1min":0.84299999999999997,"5min":1.129,"15min":1.129},"last":0.63600000000000001}]},{"osd":7,"last update":"Mon Oct 16 13:47:41 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.65800000000000003,"15min":0.63500000000000001},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.38900000000000001},"max":{"1min":0.754,"5min":0.97499999999999998,"15min":0.98599999999999999},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.71299999999999997,"15min":0.68700000000000006},"min":{"1min":0.48699999999999999,"5min":0.44,"15min":0.38200000000000001},"max":{"1min":0.84699999999999998,"5min":0.97599999999999998,"15min":1.026},"last":0.69299999999999995}]}]},{"osd":5,"up_from":13,"seq":55834576005,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74720,"kb_used_data":3280,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94297120,"statfs":{"total":96636764160,"available":96560250880,"internally_reserved":0,"allocated":3358720,"data_stored":1740375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:47:28 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.59599999999999997,"15min":0.61399999999999999},"min":{"1min":0.379,"5min":0.314,"15min":0.26800000000000002},"max":{"1min":0.79200000000000004,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.61799999999999999,"15min":0.63700000000000001},"min":{"1min":0.38600000000000001,"5min":0.35899999999999999,"15min":0.29699999999999999},"max":{"1min":0.78600000000000003,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.57699999999999996}]},{"osd":1,"last update":"Mon Oct 16 13:47:28 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.64700000000000002,"15min":0.65900000000000003},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.29499999999999998},"max":{"1min":0.82799999999999996,"5min":1,"15min":1},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.61299999999999999,"15min":0.63600000000000001},"min":{"1min":0.378,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.82099999999999995,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.54200000000000004}]},{"osd":2,"last update":"Mon Oct 16 13:47:28 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.65400000000000003,"15min":0.65500000000000003},"min":{"1min":0.48699999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.80500000000000005,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.63400000000000001,"15min":0.64700000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.80200000000000005,"5min":0.871,"15min":0.93500000000000005},"last":0.55900000000000005}]},{"osd":3,"last update":"Mon Oct 16 13:47:28 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.66200000000000003,"15min":0.66500000000000004},"min":{"1min":0.42199999999999999,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.85799999999999998,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.68999999999999995,"15min":0.69399999999999995},"min":{"1min":0.45500000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.82799999999999996,"5min":1.4199999999999999,"15min":1.4199999999999999},"last":0.48799999999999999}]},{"osd":4,"last update":"Mon Oct 16 13:47:28 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.54800000000000004,"15min":0.53800000000000003},"min":{"1min":0.39500000000000002,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.89700000000000002,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.60799999999999998,"15min":0.59499999999999997},"min":{"1min":0.40999999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.68899999999999995}]},{"osd":6,"last update":"Mon Oct 16 13:47:28 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.60099999999999998,"15min":0.58799999999999997},"min":{"1min":0.443,"5min":0.315,"15min":0.315},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.93200000000000005},"last":0.628},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.629,"15min":0.625},"min":{"1min":0.501,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.52800000000000002}]},{"osd":7,"last update":"Mon Oct 16 13:47:28 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.65500000000000003,"15min":0.63900000000000001},"min":{"1min":0.47599999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.93899999999999995,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.70699999999999996,"15min":0.68899999999999995},"min":{"1min":0.53400000000000003,"5min":0.375,"15min":0.375},"max":{"1min":0.97299999999999998,"5min":0.98299999999999998,"15min":1.0229999999999999},"last":0.60199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T13:48:29.600 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T13:48:29.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.757+0000 7f365507a700 1 Processor -- start 2023-10-16T13:48:29.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.757+0000 7f365507a700 1 -- start start 2023-10-16T13:48:29.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.758+0000 7f365507a700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f36500cc8a0 0x7f365015e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:29.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.758+0000 7f365507a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f365006a7e0 0x7f365006ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:29.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.758+0000 7f365507a700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f365006b150 con 0x7f365015b280 2023-10-16T13:48:29.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.758+0000 7f365507a700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3650161c30 con 0x7f365006a7e0 2023-10-16T13:48:29.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.758+0000 7f365507a700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3650161f10 con 0x7f36500cc8a0 2023-10-16T13:48:29.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.758+0000 7f364f59e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f365006a7e0 0x7f365006ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:29.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.758+0000 7f364f59e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f365006a7e0 0x7f365006ac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57592/0 (socket says 172.21.15.16:57592) 2023-10-16T13:48:29.763 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.758+0000 7f364f59e700 1 -- 172.21.15.16:0/639188203 learned_addr learned my addr 172.21.15.16:0/639188203 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:29.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.758+0000 7f364e59c700 1 --2- 172.21.15.16:0/639188203 >> v2:172.21.15.49:3300/0 conn(0x7f36500cc8a0 0x7f365015e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:29.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.758+0000 7f364f59e700 1 -- 172.21.15.16:0/639188203 >> v2:172.21.15.49:3300/0 conn(0x7f36500cc8a0 msgr2=0x7f365015e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:29.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.758+0000 7f364f59e700 1 --2- 172.21.15.16:0/639188203 >> v2:172.21.15.49:3300/0 conn(0x7f36500cc8a0 0x7f365015e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:29.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.758+0000 7f364f59e700 1 -- 172.21.15.16:0/639188203 >> v1:172.21.15.49:6789/0 conn(0x7f365015b280 legacy=0x7f36501613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:29.764 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.758+0000 7f364f59e700 1 -- 172.21.15.16:0/639188203 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36501621f0 con 0x7f365006a7e0 2023-10-16T13:48:29.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.758+0000 7f364e59c700 1 --2- 172.21.15.16:0/639188203 >> v2:172.21.15.49:3300/0 conn(0x7f36500cc8a0 0x7f365015e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:48:29.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.759+0000 7f364f59e700 1 --2- 172.21.15.16:0/639188203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f365006a7e0 0x7f365006ac10 secure :-1 s=READY pgs=3015 cs=0 l=1 rev1=1 crypto rx=0x7f3650071970 tx=0x7f364800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0b7882fffc0b055 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:29.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.759+0000 7f364dd9b700 1 -- 172.21.15.16:0/639188203 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f364800e070 con 0x7f365006a7e0 2023-10-16T13:48:29.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.759+0000 7f364dd9b700 1 -- 172.21.15.16:0/639188203 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f364800b9c0 con 0x7f365006a7e0 2023-10-16T13:48:29.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.759+0000 7f364dd9b700 1 -- 172.21.15.16:0/639188203 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f364800bd70 con 0x7f365006a7e0 2023-10-16T13:48:29.765 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.759+0000 7f365507a700 1 -- 172.21.15.16:0/639188203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f365006a7e0 msgr2=0x7f365006ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:29.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.759+0000 7f365507a700 1 --2- 172.21.15.16:0/639188203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f365006a7e0 0x7f365006ac10 secure :-1 s=READY pgs=3015 cs=0 l=1 rev1=1 crypto rx=0x7f3650071970 tx=0x7f364800b330 comp rx=0 tx=0).stop 2023-10-16T13:48:29.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.759+0000 7f365507a700 1 -- 172.21.15.16:0/639188203 shutdown_connections 2023-10-16T13:48:29.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.759+0000 7f365507a700 1 --2- 172.21.15.16:0/639188203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f365006a7e0 0x7f365006ac10 unknown :-1 s=CLOSED pgs=3015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:29.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.759+0000 7f365507a700 1 --2- 172.21.15.16:0/639188203 >> v2:172.21.15.49:3300/0 conn(0x7f36500cc8a0 0x7f365015e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:29.766 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.759+0000 7f365507a700 1 -- 172.21.15.16:0/639188203 >> 172.21.15.16:0/639188203 conn(0x7f36500b1fd0 msgr2=0x7f36500b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:29.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.759+0000 7f365507a700 1 -- 172.21.15.16:0/639188203 shutdown_connections 2023-10-16T13:48:29.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.759+0000 7f365507a700 1 -- 172.21.15.16:0/639188203 wait complete. 2023-10-16T13:48:29.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.761+0000 7f365507a700 1 Processor -- start 2023-10-16T13:48:29.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.761+0000 7f365507a700 1 -- start start 2023-10-16T13:48:29.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.762+0000 7f365507a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f365006a7e0 0x7f36501f2790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:29.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.762+0000 7f365507a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f36500cc8a0 0x7f36500c97e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:29.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.762+0000 7f365507a700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f365015b280 0x7f36500c9d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:29.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.762+0000 7f365507a700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36500b15f0 con 0x7f365006a7e0 2023-10-16T13:48:29.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.762+0000 7f365507a700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36500ca260 con 0x7f365015b280 2023-10-16T13:48:29.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.762+0000 7f365507a700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36500b09e0 con 0x7f36500cc8a0 2023-10-16T13:48:29.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.762+0000 7f364ed9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f365006a7e0 0x7f36501f2790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:29.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.762+0000 7f364ed9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f365006a7e0 0x7f36501f2790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57594/0 (socket says 172.21.15.16:57594) 2023-10-16T13:48:29.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.762+0000 7f364ed9d700 1 -- 172.21.15.16:0/230318167 learned_addr learned my addr 172.21.15.16:0/230318167 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:29.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.762+0000 7f364e59c700 1 --2- 172.21.15.16:0/230318167 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f36500cc8a0 0x7f36500c97e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:29.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.762+0000 7f364f59e700 1 --2- 172.21.15.16:0/230318167 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f365015b280 0x7f36500c9d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:29.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.762+0000 7f364ed9d700 1 -- 172.21.15.16:0/230318167 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f36500cc8a0 msgr2=0x7f36500c97e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:29.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.762+0000 7f364ed9d700 1 --2- 172.21.15.16:0/230318167 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f36500cc8a0 0x7f36500c97e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:29.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.762+0000 7f364ed9d700 1 -- 172.21.15.16:0/230318167 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f365015b280 msgr2=0x7f36500c9d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:29.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.762+0000 7f364ed9d700 1 --2- 172.21.15.16:0/230318167 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f365015b280 0x7f36500c9d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:29.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.762+0000 7f364ed9d700 1 -- 172.21.15.16:0/230318167 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3648009b30 con 0x7f365006a7e0 2023-10-16T13:48:29.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.762+0000 7f364f59e700 1 --2- 172.21.15.16:0/230318167 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f365015b280 0x7f36500c9d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:48:29.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.762+0000 7f364ed9d700 1 --2- 172.21.15.16:0/230318167 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f365006a7e0 0x7f36501f2790 secure :-1 s=READY pgs=3016 cs=0 l=1 rev1=1 crypto rx=0x7f364400c450 tx=0x7f364400c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:29.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.762+0000 7f36377fe700 1 -- 172.21.15.16:0/230318167 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36440052d0 con 0x7f365006a7e0 2023-10-16T13:48:29.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.762+0000 7f36377fe700 1 -- 172.21.15.16:0/230318167 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3644005430 con 0x7f365006a7e0 2023-10-16T13:48:29.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.762+0000 7f365507a700 1 -- 172.21.15.16:0/230318167 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36500b0cc0 con 0x7f365006a7e0 2023-10-16T13:48:29.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.763+0000 7f36377fe700 1 -- 172.21.15.16:0/230318167 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36440038c0 con 0x7f365006a7e0 2023-10-16T13:48:29.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.763+0000 7f365507a700 1 -- 172.21.15.16:0/230318167 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36500b1280 con 0x7f365006a7e0 2023-10-16T13:48:29.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.763+0000 7f365507a700 1 -- 172.21.15.16:0/230318167 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f365004ebb0 con 0x7f365006a7e0 2023-10-16T13:48:29.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.764+0000 7f36377fe700 1 -- 172.21.15.16:0/230318167 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f3644003a20 con 0x7f365006a7e0 2023-10-16T13:48:29.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.764+0000 7f36377fe700 1 --2- 172.21.15.16:0/230318167 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f363c06b140 0x7f363c06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:29.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.765+0000 7f36377fe700 1 -- 172.21.15.16:0/230318167 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f3644088ea0 con 0x7f365006a7e0 2023-10-16T13:48:29.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.765+0000 7f364e59c700 1 --2- 172.21.15.16:0/230318167 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f363c06b140 0x7f363c06d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:29.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.765+0000 7f364e59c700 1 --2- 172.21.15.16:0/230318167 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f363c06b140 0x7f363c06d600 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f36500d0fd0 tx=0x7f3640008040 comp rx=0 tx=0).ready entity=mgr.12678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:29.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.769+0000 7f36377fe700 1 -- 172.21.15.16:0/230318167 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3644053b00 con 0x7f365006a7e0 2023-10-16T13:48:29.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.968+0000 7f365507a700 1 -- 172.21.15.16:0/230318167 --> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f365015e1e0 con 0x7f363c06b140 2023-10-16T13:48:29.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.976+0000 7f36377fe700 1 -- 172.21.15.16:0/230318167 <== mgr.12678 v2:172.21.15.16:6832/1275231846 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+290166 (secure 0 0 0) 0x7f3640011720 con 0x7f363c06b140 2023-10-16T13:48:29.982 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:48:29.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.979+0000 7f365507a700 1 -- 172.21.15.16:0/230318167 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f363c06b140 msgr2=0x7f363c06d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:29.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.979+0000 7f365507a700 1 --2- 172.21.15.16:0/230318167 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f363c06b140 0x7f363c06d600 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f36500d0fd0 tx=0x7f3640008040 comp rx=0 tx=0).stop 2023-10-16T13:48:29.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.979+0000 7f365507a700 1 -- 172.21.15.16:0/230318167 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f365006a7e0 msgr2=0x7f36501f2790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:29.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.979+0000 7f365507a700 1 --2- 172.21.15.16:0/230318167 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f365006a7e0 0x7f36501f2790 secure :-1 s=READY pgs=3016 cs=0 l=1 rev1=1 crypto rx=0x7f364400c450 tx=0x7f364400c760 comp rx=0 tx=0).stop 2023-10-16T13:48:29.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.979+0000 7f365507a700 1 -- 172.21.15.16:0/230318167 shutdown_connections 2023-10-16T13:48:29.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.979+0000 7f365507a700 1 --2- 172.21.15.16:0/230318167 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f363c06b140 0x7f363c06d600 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:29.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.980+0000 7f365507a700 1 --2- 172.21.15.16:0/230318167 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f365015b280 0x7f36500c9d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:29.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.980+0000 7f365507a700 1 --2- 172.21.15.16:0/230318167 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f365006a7e0 0x7f36501f2790 unknown :-1 s=CLOSED pgs=3016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:29.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.980+0000 7f365507a700 1 --2- 172.21.15.16:0/230318167 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f36500cc8a0 0x7f36500c97e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:29.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.980+0000 7f365507a700 1 -- 172.21.15.16:0/230318167 >> 172.21.15.16:0/230318167 conn(0x7f36500b1fd0 msgr2=0x7f365015a800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:29.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.980+0000 7f365507a700 1 -- 172.21.15.16:0/230318167 shutdown_connections 2023-10-16T13:48:29.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:29.980+0000 7f365507a700 1 -- 172.21.15.16:0/230318167 wait complete. 2023-10-16T13:48:29.987 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-10-16T13:48:29.994 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":30,"stamp":"2023-10-16T13:48:28.509404+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2420,"num_read_kb":5044,"num_write":683,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":419,"ondisk_log_size":419,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":144,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":464252,"kb_used_data":27400,"kb_used_omap":12,"kb_used_meta":436723,"kb_avail":754510468,"statfs":{"total":773094113280,"available":772618719232,"internally_reserved":0,"allocated":28057600,"data_stored":15108108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12740,"internal_metadata":447204924},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.108657"},"pg_stats":[{"pgid":"36.3f","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.394764+0000","last_change":"2023-10-16T13:48:25.440344+0000","last_active":"2023-10-16T13:48:27.394764+0000","last_peered":"2023-10-16T13:48:27.394764+0000","last_clean":"2023-10-16T13:48:27.394764+0000","last_became_active":"2023-10-16T13:48:25.439797+0000","last_became_peered":"2023-10-16T13:48:25.439797+0000","last_unstale":"2023-10-16T13:48:27.394764+0000","last_undegraded":"2023-10-16T13:48:27.394764+0000","last_fullsized":"2023-10-16T13:48:27.394764+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:24:11.575909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.3e","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.394646+0000","last_change":"2023-10-16T13:48:25.447276+0000","last_active":"2023-10-16T13:48:27.394646+0000","last_peered":"2023-10-16T13:48:27.394646+0000","last_clean":"2023-10-16T13:48:27.394646+0000","last_became_active":"2023-10-16T13:48:25.444498+0000","last_became_peered":"2023-10-16T13:48:25.444498+0000","last_unstale":"2023-10-16T13:48:27.394646+0000","last_undegraded":"2023-10-16T13:48:27.394646+0000","last_fullsized":"2023-10-16T13:48:27.394646+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:40:17.291707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.3d","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.394304+0000","last_change":"2023-10-16T13:48:25.445627+0000","last_active":"2023-10-16T13:48:27.394304+0000","last_peered":"2023-10-16T13:48:27.394304+0000","last_clean":"2023-10-16T13:48:27.394304+0000","last_became_active":"2023-10-16T13:48:25.445233+0000","last_became_peered":"2023-10-16T13:48:25.445233+0000","last_unstale":"2023-10-16T13:48:27.394304+0000","last_undegraded":"2023-10-16T13:48:27.394304+0000","last_fullsized":"2023-10-16T13:48:27.394304+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:46:41.609860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.3c","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.394242+0000","last_change":"2023-10-16T13:48:25.433667+0000","last_active":"2023-10-16T13:48:27.394242+0000","last_peered":"2023-10-16T13:48:27.394242+0000","last_clean":"2023-10-16T13:48:27.394242+0000","last_became_active":"2023-10-16T13:48:25.433338+0000","last_became_peered":"2023-10-16T13:48:25.433338+0000","last_unstale":"2023-10-16T13:48:27.394242+0000","last_undegraded":"2023-10-16T13:48:27.394242+0000","last_fullsized":"2023-10-16T13:48:27.394242+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:15:32.675005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.3b","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.393580+0000","last_change":"2023-10-16T13:48:25.429495+0000","last_active":"2023-10-16T13:48:27.393580+0000","last_peered":"2023-10-16T13:48:27.393580+0000","last_clean":"2023-10-16T13:48:27.393580+0000","last_became_active":"2023-10-16T13:48:25.428576+0000","last_became_peered":"2023-10-16T13:48:25.428576+0000","last_unstale":"2023-10-16T13:48:27.393580+0000","last_undegraded":"2023-10-16T13:48:27.393580+0000","last_fullsized":"2023-10-16T13:48:27.393580+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:41:00.902601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.3a","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.393226+0000","last_change":"2023-10-16T13:48:25.444779+0000","last_active":"2023-10-16T13:48:27.393226+0000","last_peered":"2023-10-16T13:48:27.393226+0000","last_clean":"2023-10-16T13:48:27.393226+0000","last_became_active":"2023-10-16T13:48:25.444495+0000","last_became_peered":"2023-10-16T13:48:25.444495+0000","last_unstale":"2023-10-16T13:48:27.393226+0000","last_undegraded":"2023-10-16T13:48:27.393226+0000","last_fullsized":"2023-10-16T13:48:27.393226+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:11:09.084674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.39","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.164073+0000","last_change":"2023-10-16T13:48:25.441400+0000","last_active":"2023-10-16T13:48:27.164073+0000","last_peered":"2023-10-16T13:48:27.164073+0000","last_clean":"2023-10-16T13:48:27.164073+0000","last_became_active":"2023-10-16T13:48:25.441112+0000","last_became_peered":"2023-10-16T13:48:25.441112+0000","last_unstale":"2023-10-16T13:48:27.164073+0000","last_undegraded":"2023-10-16T13:48:27.164073+0000","last_fullsized":"2023-10-16T13:48:27.164073+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:45:53.473497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.38","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.393265+0000","last_change":"2023-10-16T13:48:25.435146+0000","last_active":"2023-10-16T13:48:27.393265+0000","last_peered":"2023-10-16T13:48:27.393265+0000","last_clean":"2023-10-16T13:48:27.393265+0000","last_became_active":"2023-10-16T13:48:25.434832+0000","last_became_peered":"2023-10-16T13:48:25.434832+0000","last_unstale":"2023-10-16T13:48:27.393265+0000","last_undegraded":"2023-10-16T13:48:27.393265+0000","last_fullsized":"2023-10-16T13:48:27.393265+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:48:55.481961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.37","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.164575+0000","last_change":"2023-10-16T13:48:25.443618+0000","last_active":"2023-10-16T13:48:27.164575+0000","last_peered":"2023-10-16T13:48:27.164575+0000","last_clean":"2023-10-16T13:48:27.164575+0000","last_became_active":"2023-10-16T13:48:25.443345+0000","last_became_peered":"2023-10-16T13:48:25.443345+0000","last_unstale":"2023-10-16T13:48:27.164575+0000","last_undegraded":"2023-10-16T13:48:27.164575+0000","last_fullsized":"2023-10-16T13:48:27.164575+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:11:09.084674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.36","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.164523+0000","last_change":"2023-10-16T13:48:25.434663+0000","last_active":"2023-10-16T13:48:27.164523+0000","last_peered":"2023-10-16T13:48:27.164523+0000","last_clean":"2023-10-16T13:48:27.164523+0000","last_became_active":"2023-10-16T13:48:25.434440+0000","last_became_peered":"2023-10-16T13:48:25.434440+0000","last_unstale":"2023-10-16T13:48:27.164523+0000","last_undegraded":"2023-10-16T13:48:27.164523+0000","last_fullsized":"2023-10-16T13:48:27.164523+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:35:54.607900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.35","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:26.422050+0000","last_change":"2023-10-16T13:48:25.430939+0000","last_active":"2023-10-16T13:48:26.422050+0000","last_peered":"2023-10-16T13:48:26.422050+0000","last_clean":"2023-10-16T13:48:26.422050+0000","last_became_active":"2023-10-16T13:48:25.430596+0000","last_became_peered":"2023-10-16T13:48:25.430596+0000","last_unstale":"2023-10-16T13:48:26.422050+0000","last_undegraded":"2023-10-16T13:48:26.422050+0000","last_fullsized":"2023-10-16T13:48:26.422050+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:47:41.595942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.34","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.395034+0000","last_change":"2023-10-16T13:48:25.440365+0000","last_active":"2023-10-16T13:48:27.395034+0000","last_peered":"2023-10-16T13:48:27.395034+0000","last_clean":"2023-10-16T13:48:27.395034+0000","last_became_active":"2023-10-16T13:48:25.439782+0000","last_became_peered":"2023-10-16T13:48:25.439782+0000","last_unstale":"2023-10-16T13:48:27.395034+0000","last_undegraded":"2023-10-16T13:48:27.395034+0000","last_fullsized":"2023-10-16T13:48:27.395034+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:53:00.722815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.33","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.394234+0000","last_change":"2023-10-16T13:48:25.445304+0000","last_active":"2023-10-16T13:48:27.394234+0000","last_peered":"2023-10-16T13:48:27.394234+0000","last_clean":"2023-10-16T13:48:27.394234+0000","last_became_active":"2023-10-16T13:48:25.444997+0000","last_became_peered":"2023-10-16T13:48:25.444997+0000","last_unstale":"2023-10-16T13:48:27.394234+0000","last_undegraded":"2023-10-16T13:48:27.394234+0000","last_fullsized":"2023-10-16T13:48:27.394234+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:46:04.673793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.32","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.164260+0000","last_change":"2023-10-16T13:48:25.433664+0000","last_active":"2023-10-16T13:48:27.164260+0000","last_peered":"2023-10-16T13:48:27.164260+0000","last_clean":"2023-10-16T13:48:27.164260+0000","last_became_active":"2023-10-16T13:48:25.433352+0000","last_became_peered":"2023-10-16T13:48:25.433352+0000","last_unstale":"2023-10-16T13:48:27.164260+0000","last_undegraded":"2023-10-16T13:48:27.164260+0000","last_fullsized":"2023-10-16T13:48:27.164260+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:00:29.913030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.31","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.393389+0000","last_change":"2023-10-16T13:48:25.435661+0000","last_active":"2023-10-16T13:48:27.393389+0000","last_peered":"2023-10-16T13:48:27.393389+0000","last_clean":"2023-10-16T13:48:27.393389+0000","last_became_active":"2023-10-16T13:48:25.435366+0000","last_became_peered":"2023-10-16T13:48:25.435366+0000","last_unstale":"2023-10-16T13:48:27.393389+0000","last_undegraded":"2023-10-16T13:48:27.393389+0000","last_fullsized":"2023-10-16T13:48:27.393389+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:22:50.795453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.30","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.393363+0000","last_change":"2023-10-16T13:48:25.434843+0000","last_active":"2023-10-16T13:48:27.393363+0000","last_peered":"2023-10-16T13:48:27.393363+0000","last_clean":"2023-10-16T13:48:27.393363+0000","last_became_active":"2023-10-16T13:48:25.434530+0000","last_became_peered":"2023-10-16T13:48:25.434530+0000","last_unstale":"2023-10-16T13:48:27.393363+0000","last_undegraded":"2023-10-16T13:48:27.393363+0000","last_fullsized":"2023-10-16T13:48:27.393363+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:17:13.546895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.2f","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.394821+0000","last_change":"2023-10-16T13:48:25.434325+0000","last_active":"2023-10-16T13:48:27.394821+0000","last_peered":"2023-10-16T13:48:27.394821+0000","last_clean":"2023-10-16T13:48:27.394821+0000","last_became_active":"2023-10-16T13:48:25.433548+0000","last_became_peered":"2023-10-16T13:48:25.433548+0000","last_unstale":"2023-10-16T13:48:27.394821+0000","last_undegraded":"2023-10-16T13:48:27.394821+0000","last_fullsized":"2023-10-16T13:48:27.394821+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:21:29.852309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.2e","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.394496+0000","last_change":"2023-10-16T13:48:25.448916+0000","last_active":"2023-10-16T13:48:27.394496+0000","last_peered":"2023-10-16T13:48:27.394496+0000","last_clean":"2023-10-16T13:48:27.394496+0000","last_became_active":"2023-10-16T13:48:25.446072+0000","last_became_peered":"2023-10-16T13:48:25.446072+0000","last_unstale":"2023-10-16T13:48:27.394496+0000","last_undegraded":"2023-10-16T13:48:27.394496+0000","last_fullsized":"2023-10-16T13:48:27.394496+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:21:02.827127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.2d","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.394411+0000","last_change":"2023-10-16T13:48:25.442491+0000","last_active":"2023-10-16T13:48:27.394411+0000","last_peered":"2023-10-16T13:48:27.394411+0000","last_clean":"2023-10-16T13:48:27.394411+0000","last_became_active":"2023-10-16T13:48:25.442191+0000","last_became_peered":"2023-10-16T13:48:25.442191+0000","last_unstale":"2023-10-16T13:48:27.394411+0000","last_undegraded":"2023-10-16T13:48:27.394411+0000","last_fullsized":"2023-10-16T13:48:27.394411+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:07:14.273460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.2c","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.394093+0000","last_change":"2023-10-16T13:48:25.434792+0000","last_active":"2023-10-16T13:48:27.394093+0000","last_peered":"2023-10-16T13:48:27.394093+0000","last_clean":"2023-10-16T13:48:27.394093+0000","last_became_active":"2023-10-16T13:48:25.434472+0000","last_became_peered":"2023-10-16T13:48:25.434472+0000","last_unstale":"2023-10-16T13:48:27.394093+0000","last_undegraded":"2023-10-16T13:48:27.394093+0000","last_fullsized":"2023-10-16T13:48:27.394093+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:11:58.739219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.2b","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:26.421755+0000","last_change":"2023-10-16T13:48:25.441991+0000","last_active":"2023-10-16T13:48:26.421755+0000","last_peered":"2023-10-16T13:48:26.421755+0000","last_clean":"2023-10-16T13:48:26.421755+0000","last_became_active":"2023-10-16T13:48:25.441592+0000","last_became_peered":"2023-10-16T13:48:25.441592+0000","last_unstale":"2023-10-16T13:48:26.421755+0000","last_undegraded":"2023-10-16T13:48:26.421755+0000","last_fullsized":"2023-10-16T13:48:26.421755+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:28:48.750254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.2a","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:26.421344+0000","last_change":"2023-10-16T13:48:25.434737+0000","last_active":"2023-10-16T13:48:26.421344+0000","last_peered":"2023-10-16T13:48:26.421344+0000","last_clean":"2023-10-16T13:48:26.421344+0000","last_became_active":"2023-10-16T13:48:25.434461+0000","last_became_peered":"2023-10-16T13:48:25.434461+0000","last_unstale":"2023-10-16T13:48:26.421344+0000","last_undegraded":"2023-10-16T13:48:26.421344+0000","last_fullsized":"2023-10-16T13:48:26.421344+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:41:00.902601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.29","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.164129+0000","last_change":"2023-10-16T13:48:25.436144+0000","last_active":"2023-10-16T13:48:27.164129+0000","last_peered":"2023-10-16T13:48:27.164129+0000","last_clean":"2023-10-16T13:48:27.164129+0000","last_became_active":"2023-10-16T13:48:25.435379+0000","last_became_peered":"2023-10-16T13:48:25.435379+0000","last_unstale":"2023-10-16T13:48:27.164129+0000","last_undegraded":"2023-10-16T13:48:27.164129+0000","last_fullsized":"2023-10-16T13:48:27.164129+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:11:05.137455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.28","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:26.421024+0000","last_change":"2023-10-16T13:48:25.438832+0000","last_active":"2023-10-16T13:48:26.421024+0000","last_peered":"2023-10-16T13:48:26.421024+0000","last_clean":"2023-10-16T13:48:26.421024+0000","last_became_active":"2023-10-16T13:48:25.438583+0000","last_became_peered":"2023-10-16T13:48:25.438583+0000","last_unstale":"2023-10-16T13:48:26.421024+0000","last_undegraded":"2023-10-16T13:48:26.421024+0000","last_fullsized":"2023-10-16T13:48:26.421024+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:11:09.084674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.27","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.394939+0000","last_change":"2023-10-16T13:48:25.440490+0000","last_active":"2023-10-16T13:48:27.394939+0000","last_peered":"2023-10-16T13:48:27.394939+0000","last_clean":"2023-10-16T13:48:27.394939+0000","last_became_active":"2023-10-16T13:48:25.440003+0000","last_became_peered":"2023-10-16T13:48:25.440003+0000","last_unstale":"2023-10-16T13:48:27.394939+0000","last_undegraded":"2023-10-16T13:48:27.394939+0000","last_fullsized":"2023-10-16T13:48:27.394939+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:05:29.646924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.26","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.393603+0000","last_change":"2023-10-16T13:48:25.433846+0000","last_active":"2023-10-16T13:48:27.393603+0000","last_peered":"2023-10-16T13:48:27.393603+0000","last_clean":"2023-10-16T13:48:27.393603+0000","last_became_active":"2023-10-16T13:48:25.433463+0000","last_became_peered":"2023-10-16T13:48:25.433463+0000","last_unstale":"2023-10-16T13:48:27.393603+0000","last_undegraded":"2023-10-16T13:48:27.393603+0000","last_fullsized":"2023-10-16T13:48:27.393603+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:08:59.756642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.24","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.393539+0000","last_change":"2023-10-16T13:48:25.436440+0000","last_active":"2023-10-16T13:48:27.393539+0000","last_peered":"2023-10-16T13:48:27.393539+0000","last_clean":"2023-10-16T13:48:27.393539+0000","last_became_active":"2023-10-16T13:48:25.436172+0000","last_became_peered":"2023-10-16T13:48:25.436172+0000","last_unstale":"2023-10-16T13:48:27.393539+0000","last_undegraded":"2023-10-16T13:48:27.393539+0000","last_fullsized":"2023-10-16T13:48:27.393539+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:33:34.431164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.23","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:26.421427+0000","last_change":"2023-10-16T13:48:25.437873+0000","last_active":"2023-10-16T13:48:26.421427+0000","last_peered":"2023-10-16T13:48:26.421427+0000","last_clean":"2023-10-16T13:48:26.421427+0000","last_became_active":"2023-10-16T13:48:25.437581+0000","last_became_peered":"2023-10-16T13:48:25.437581+0000","last_unstale":"2023-10-16T13:48:26.421427+0000","last_undegraded":"2023-10-16T13:48:26.421427+0000","last_fullsized":"2023-10-16T13:48:26.421427+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:35:56.554093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.22","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.393592+0000","last_change":"2023-10-16T13:48:25.435538+0000","last_active":"2023-10-16T13:48:27.393592+0000","last_peered":"2023-10-16T13:48:27.393592+0000","last_clean":"2023-10-16T13:48:27.393592+0000","last_became_active":"2023-10-16T13:48:25.435190+0000","last_became_peered":"2023-10-16T13:48:25.435190+0000","last_unstale":"2023-10-16T13:48:27.393592+0000","last_undegraded":"2023-10-16T13:48:27.393592+0000","last_fullsized":"2023-10-16T13:48:27.393592+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:52:50.562319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.21","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:26.421074+0000","last_change":"2023-10-16T13:48:25.445785+0000","last_active":"2023-10-16T13:48:26.421074+0000","last_peered":"2023-10-16T13:48:26.421074+0000","last_clean":"2023-10-16T13:48:26.421074+0000","last_became_active":"2023-10-16T13:48:25.445489+0000","last_became_peered":"2023-10-16T13:48:25.445489+0000","last_unstale":"2023-10-16T13:48:26.421074+0000","last_undegraded":"2023-10-16T13:48:26.421074+0000","last_fullsized":"2023-10-16T13:48:26.421074+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:30:42.136595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.20","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.393352+0000","last_change":"2023-10-16T13:48:25.441470+0000","last_active":"2023-10-16T13:48:27.393352+0000","last_peered":"2023-10-16T13:48:27.393352+0000","last_clean":"2023-10-16T13:48:27.393352+0000","last_became_active":"2023-10-16T13:48:25.440973+0000","last_became_peered":"2023-10-16T13:48:25.440973+0000","last_unstale":"2023-10-16T13:48:27.393352+0000","last_undegraded":"2023-10-16T13:48:27.393352+0000","last_fullsized":"2023-10-16T13:48:27.393352+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:16:45.304293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.18","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:26.421074+0000","last_change":"2023-10-16T13:48:25.439376+0000","last_active":"2023-10-16T13:48:26.421074+0000","last_peered":"2023-10-16T13:48:26.421074+0000","last_clean":"2023-10-16T13:48:26.421074+0000","last_became_active":"2023-10-16T13:48:25.439055+0000","last_became_peered":"2023-10-16T13:48:25.439055+0000","last_unstale":"2023-10-16T13:48:26.421074+0000","last_undegraded":"2023-10-16T13:48:26.421074+0000","last_fullsized":"2023-10-16T13:48:26.421074+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:04:33.594311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.1f","version":"264'4","reported_seq":16,"reported_epoch":264,"state":"active+clean","last_fresh":"2023-10-16T13:48:25.453425+0000","last_change":"2023-10-16T13:48:24.426828+0000","last_active":"2023-10-16T13:48:25.453425+0000","last_peered":"2023-10-16T13:48:25.453425+0000","last_clean":"2023-10-16T13:48:25.453425+0000","last_became_active":"2023-10-16T13:48:24.426394+0000","last_became_peered":"2023-10-16T13:48:24.426394+0000","last_unstale":"2023-10-16T13:48:25.453425+0000","last_undegraded":"2023-10-16T13:48:25.453425+0000","last_fullsized":"2023-10-16T13:48:25.453425+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_clean_scrub_stamp":"2023-10-16T13:48:23.395571+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:51:29.737215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.19","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.164445+0000","last_change":"2023-10-16T13:48:25.436335+0000","last_active":"2023-10-16T13:48:27.164445+0000","last_peered":"2023-10-16T13:48:27.164445+0000","last_clean":"2023-10-16T13:48:27.164445+0000","last_became_active":"2023-10-16T13:48:25.435822+0000","last_became_peered":"2023-10-16T13:48:25.435822+0000","last_unstale":"2023-10-16T13:48:27.164445+0000","last_undegraded":"2023-10-16T13:48:27.164445+0000","last_fullsized":"2023-10-16T13:48:27.164445+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:12:16.262267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.1e","version":"264'1","reported_seq":13,"reported_epoch":264,"state":"active+clean","last_fresh":"2023-10-16T13:48:25.454493+0000","last_change":"2023-10-16T13:48:24.422098+0000","last_active":"2023-10-16T13:48:25.454493+0000","last_peered":"2023-10-16T13:48:25.454493+0000","last_clean":"2023-10-16T13:48:25.454493+0000","last_became_active":"2023-10-16T13:48:24.421192+0000","last_became_peered":"2023-10-16T13:48:24.421192+0000","last_unstale":"2023-10-16T13:48:25.454493+0000","last_undegraded":"2023-10-16T13:48:25.454493+0000","last_fullsized":"2023-10-16T13:48:25.454493+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_clean_scrub_stamp":"2023-10-16T13:48:23.395571+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:08:40.541463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.1a","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.394877+0000","last_change":"2023-10-16T13:48:25.445268+0000","last_active":"2023-10-16T13:48:27.394877+0000","last_peered":"2023-10-16T13:48:27.394877+0000","last_clean":"2023-10-16T13:48:27.394877+0000","last_became_active":"2023-10-16T13:48:25.444954+0000","last_became_peered":"2023-10-16T13:48:25.444954+0000","last_unstale":"2023-10-16T13:48:27.394877+0000","last_undegraded":"2023-10-16T13:48:27.394877+0000","last_fullsized":"2023-10-16T13:48:27.394877+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:28:21.176426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.1d","version":"264'2","reported_seq":14,"reported_epoch":264,"state":"active+clean","last_fresh":"2023-10-16T13:48:25.454252+0000","last_change":"2023-10-16T13:48:24.422654+0000","last_active":"2023-10-16T13:48:25.454252+0000","last_peered":"2023-10-16T13:48:25.454252+0000","last_clean":"2023-10-16T13:48:25.454252+0000","last_became_active":"2023-10-16T13:48:24.422204+0000","last_became_peered":"2023-10-16T13:48:24.422204+0000","last_unstale":"2023-10-16T13:48:25.454252+0000","last_undegraded":"2023-10-16T13:48:25.454252+0000","last_fullsized":"2023-10-16T13:48:25.454252+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_clean_scrub_stamp":"2023-10-16T13:48:23.395571+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:56:05.560556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.1b","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.164373+0000","last_change":"2023-10-16T13:48:25.443547+0000","last_active":"2023-10-16T13:48:27.164373+0000","last_peered":"2023-10-16T13:48:27.164373+0000","last_clean":"2023-10-16T13:48:27.164373+0000","last_became_active":"2023-10-16T13:48:25.443240+0000","last_became_peered":"2023-10-16T13:48:25.443240+0000","last_unstale":"2023-10-16T13:48:27.164373+0000","last_undegraded":"2023-10-16T13:48:27.164373+0000","last_fullsized":"2023-10-16T13:48:27.164373+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:14:32.129157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.1c","version":"264'2","reported_seq":14,"reported_epoch":264,"state":"active+clean","last_fresh":"2023-10-16T13:48:25.454745+0000","last_change":"2023-10-16T13:48:24.420922+0000","last_active":"2023-10-16T13:48:25.454745+0000","last_peered":"2023-10-16T13:48:25.454745+0000","last_clean":"2023-10-16T13:48:25.454745+0000","last_became_active":"2023-10-16T13:48:24.420438+0000","last_became_peered":"2023-10-16T13:48:24.420438+0000","last_unstale":"2023-10-16T13:48:25.454745+0000","last_undegraded":"2023-10-16T13:48:25.454745+0000","last_fullsized":"2023-10-16T13:48:25.454745+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_clean_scrub_stamp":"2023-10-16T13:48:23.395571+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:35:03.573357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.1c","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.394439+0000","last_change":"2023-10-16T13:48:25.448098+0000","last_active":"2023-10-16T13:48:27.394439+0000","last_peered":"2023-10-16T13:48:27.394439+0000","last_clean":"2023-10-16T13:48:27.394439+0000","last_became_active":"2023-10-16T13:48:25.446119+0000","last_became_peered":"2023-10-16T13:48:25.446119+0000","last_unstale":"2023-10-16T13:48:27.394439+0000","last_undegraded":"2023-10-16T13:48:27.394439+0000","last_fullsized":"2023-10-16T13:48:27.394439+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:40:30.963695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.1b","version":"264'1","reported_seq":13,"reported_epoch":264,"state":"active+clean","last_fresh":"2023-10-16T13:48:25.454386+0000","last_change":"2023-10-16T13:48:24.425433+0000","last_active":"2023-10-16T13:48:25.454386+0000","last_peered":"2023-10-16T13:48:25.454386+0000","last_clean":"2023-10-16T13:48:25.454386+0000","last_became_active":"2023-10-16T13:48:24.425121+0000","last_became_peered":"2023-10-16T13:48:24.425121+0000","last_unstale":"2023-10-16T13:48:25.454386+0000","last_undegraded":"2023-10-16T13:48:25.454386+0000","last_fullsized":"2023-10-16T13:48:25.454386+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_clean_scrub_stamp":"2023-10-16T13:48:23.395571+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:57:16.571463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.1d","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:26.422317+0000","last_change":"2023-10-16T13:48:25.445586+0000","last_active":"2023-10-16T13:48:26.422317+0000","last_peered":"2023-10-16T13:48:26.422317+0000","last_clean":"2023-10-16T13:48:26.422317+0000","last_became_active":"2023-10-16T13:48:25.445292+0000","last_became_peered":"2023-10-16T13:48:25.445292+0000","last_unstale":"2023-10-16T13:48:26.422317+0000","last_undegraded":"2023-10-16T13:48:26.422317+0000","last_fullsized":"2023-10-16T13:48:26.422317+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:08:24.915543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.1a","version":"264'1","reported_seq":13,"reported_epoch":264,"state":"active+clean","last_fresh":"2023-10-16T13:48:25.453362+0000","last_change":"2023-10-16T13:48:24.420701+0000","last_active":"2023-10-16T13:48:25.453362+0000","last_peered":"2023-10-16T13:48:25.453362+0000","last_clean":"2023-10-16T13:48:25.453362+0000","last_became_active":"2023-10-16T13:48:24.420401+0000","last_became_peered":"2023-10-16T13:48:24.420401+0000","last_unstale":"2023-10-16T13:48:25.453362+0000","last_undegraded":"2023-10-16T13:48:25.453362+0000","last_fullsized":"2023-10-16T13:48:25.453362+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_clean_scrub_stamp":"2023-10-16T13:48:23.395571+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:11:04.133038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.1e","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:26.422513+0000","last_change":"2023-10-16T13:48:25.439008+0000","last_active":"2023-10-16T13:48:26.422513+0000","last_peered":"2023-10-16T13:48:26.422513+0000","last_clean":"2023-10-16T13:48:26.422513+0000","last_became_active":"2023-10-16T13:48:25.438631+0000","last_became_peered":"2023-10-16T13:48:25.438631+0000","last_unstale":"2023-10-16T13:48:26.422513+0000","last_undegraded":"2023-10-16T13:48:26.422513+0000","last_fullsized":"2023-10-16T13:48:26.422513+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:11:52.087708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.19","version":"0'0","reported_seq":12,"reported_epoch":264,"state":"active+clean","last_fresh":"2023-10-16T13:48:25.422488+0000","last_change":"2023-10-16T13:48:24.418856+0000","last_active":"2023-10-16T13:48:25.422488+0000","last_peered":"2023-10-16T13:48:25.422488+0000","last_clean":"2023-10-16T13:48:25.422488+0000","last_became_active":"2023-10-16T13:48:24.418545+0000","last_became_peered":"2023-10-16T13:48:24.418545+0000","last_unstale":"2023-10-16T13:48:25.422488+0000","last_undegraded":"2023-10-16T13:48:25.422488+0000","last_fullsized":"2023-10-16T13:48:25.422488+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_clean_scrub_stamp":"2023-10-16T13:48:23.395571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:17:50.550699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.1f","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.394967+0000","last_change":"2023-10-16T13:48:25.434506+0000","last_active":"2023-10-16T13:48:27.394967+0000","last_peered":"2023-10-16T13:48:27.394967+0000","last_clean":"2023-10-16T13:48:27.394967+0000","last_became_active":"2023-10-16T13:48:25.434180+0000","last_became_peered":"2023-10-16T13:48:25.434180+0000","last_unstale":"2023-10-16T13:48:27.394967+0000","last_undegraded":"2023-10-16T13:48:27.394967+0000","last_fullsized":"2023-10-16T13:48:27.394967+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:58:59.048055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.18","version":"0'0","reported_seq":12,"reported_epoch":264,"state":"active+clean","last_fresh":"2023-10-16T13:48:25.428855+0000","last_change":"2023-10-16T13:48:24.423577+0000","last_active":"2023-10-16T13:48:25.428855+0000","last_peered":"2023-10-16T13:48:25.428855+0000","last_clean":"2023-10-16T13:48:25.428855+0000","last_became_active":"2023-10-16T13:48:24.422594+0000","last_became_peered":"2023-10-16T13:48:24.422594+0000","last_unstale":"2023-10-16T13:48:25.428855+0000","last_undegraded":"2023-10-16T13:48:25.428855+0000","last_fullsized":"2023-10-16T13:48:25.428855+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_clean_scrub_stamp":"2023-10-16T13:48:23.395571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:18:00.438377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.10","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.393427+0000","last_change":"2023-10-16T13:48:25.445795+0000","last_active":"2023-10-16T13:48:27.393427+0000","last_peered":"2023-10-16T13:48:27.393427+0000","last_clean":"2023-10-16T13:48:27.393427+0000","last_became_active":"2023-10-16T13:48:25.444626+0000","last_became_peered":"2023-10-16T13:48:25.444626+0000","last_unstale":"2023-10-16T13:48:27.393427+0000","last_undegraded":"2023-10-16T13:48:27.393427+0000","last_fullsized":"2023-10-16T13:48:27.393427+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:38:03.913584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.17","version":"0'0","reported_seq":12,"reported_epoch":264,"state":"active+clean","last_fresh":"2023-10-16T13:48:25.433009+0000","last_change":"2023-10-16T13:48:24.424321+0000","last_active":"2023-10-16T13:48:25.433009+0000","last_peered":"2023-10-16T13:48:25.433009+0000","last_clean":"2023-10-16T13:48:25.433009+0000","last_became_active":"2023-10-16T13:48:24.423989+0000","last_became_peered":"2023-10-16T13:48:24.423989+0000","last_unstale":"2023-10-16T13:48:25.433009+0000","last_undegraded":"2023-10-16T13:48:25.433009+0000","last_fullsized":"2023-10-16T13:48:25.433009+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_clean_scrub_stamp":"2023-10-16T13:48:23.395571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:13:35.750808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.11","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.394279+0000","last_change":"2023-10-16T13:48:25.440171+0000","last_active":"2023-10-16T13:48:27.394279+0000","last_peered":"2023-10-16T13:48:27.394279+0000","last_clean":"2023-10-16T13:48:27.394279+0000","last_became_active":"2023-10-16T13:48:25.439568+0000","last_became_peered":"2023-10-16T13:48:25.439568+0000","last_unstale":"2023-10-16T13:48:27.394279+0000","last_undegraded":"2023-10-16T13:48:27.394279+0000","last_fullsized":"2023-10-16T13:48:27.394279+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:14:12.790962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.16","version":"0'0","reported_seq":12,"reported_epoch":264,"state":"active+clean","last_fresh":"2023-10-16T13:48:25.422535+0000","last_change":"2023-10-16T13:48:24.419409+0000","last_active":"2023-10-16T13:48:25.422535+0000","last_peered":"2023-10-16T13:48:25.422535+0000","last_clean":"2023-10-16T13:48:25.422535+0000","last_became_active":"2023-10-16T13:48:24.419140+0000","last_became_peered":"2023-10-16T13:48:24.419140+0000","last_unstale":"2023-10-16T13:48:25.422535+0000","last_undegraded":"2023-10-16T13:48:25.422535+0000","last_fullsized":"2023-10-16T13:48:25.422535+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_clean_scrub_stamp":"2023-10-16T13:48:23.395571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:45:00.351108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.12","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.393906+0000","last_change":"2023-10-16T13:48:25.433864+0000","last_active":"2023-10-16T13:48:27.393906+0000","last_peered":"2023-10-16T13:48:27.393906+0000","last_clean":"2023-10-16T13:48:27.393906+0000","last_became_active":"2023-10-16T13:48:25.433560+0000","last_became_peered":"2023-10-16T13:48:25.433560+0000","last_unstale":"2023-10-16T13:48:27.393906+0000","last_undegraded":"2023-10-16T13:48:27.393906+0000","last_fullsized":"2023-10-16T13:48:27.393906+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:10:27.868159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.15","version":"264'3","reported_seq":15,"reported_epoch":264,"state":"active+clean","last_fresh":"2023-10-16T13:48:25.452740+0000","last_change":"2023-10-16T13:48:24.421215+0000","last_active":"2023-10-16T13:48:25.452740+0000","last_peered":"2023-10-16T13:48:25.452740+0000","last_clean":"2023-10-16T13:48:25.452740+0000","last_became_active":"2023-10-16T13:48:24.420643+0000","last_became_peered":"2023-10-16T13:48:24.420643+0000","last_unstale":"2023-10-16T13:48:25.452740+0000","last_undegraded":"2023-10-16T13:48:25.452740+0000","last_fullsized":"2023-10-16T13:48:25.452740+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_clean_scrub_stamp":"2023-10-16T13:48:23.395571+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:24:58.740209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.13","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.393710+0000","last_change":"2023-10-16T13:48:25.437813+0000","last_active":"2023-10-16T13:48:27.393710+0000","last_peered":"2023-10-16T13:48:27.393710+0000","last_clean":"2023-10-16T13:48:27.393710+0000","last_became_active":"2023-10-16T13:48:25.437510+0000","last_became_peered":"2023-10-16T13:48:25.437510+0000","last_unstale":"2023-10-16T13:48:27.393710+0000","last_undegraded":"2023-10-16T13:48:27.393710+0000","last_fullsized":"2023-10-16T13:48:27.393710+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:35:56.554093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.14","version":"264'6","reported_seq":18,"reported_epoch":264,"state":"active+clean","last_fresh":"2023-10-16T13:48:25.456240+0000","last_change":"2023-10-16T13:48:24.429280+0000","last_active":"2023-10-16T13:48:25.456240+0000","last_peered":"2023-10-16T13:48:25.456240+0000","last_clean":"2023-10-16T13:48:25.456240+0000","last_became_active":"2023-10-16T13:48:24.428975+0000","last_became_peered":"2023-10-16T13:48:24.428975+0000","last_unstale":"2023-10-16T13:48:25.456240+0000","last_undegraded":"2023-10-16T13:48:25.456240+0000","last_fullsized":"2023-10-16T13:48:25.456240+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_clean_scrub_stamp":"2023-10-16T13:48:23.395571+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:40:12.074998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.14","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.395040+0000","last_change":"2023-10-16T13:48:25.446229+0000","last_active":"2023-10-16T13:48:27.395040+0000","last_peered":"2023-10-16T13:48:27.395040+0000","last_clean":"2023-10-16T13:48:27.395040+0000","last_became_active":"2023-10-16T13:48:25.441305+0000","last_became_peered":"2023-10-16T13:48:25.441305+0000","last_unstale":"2023-10-16T13:48:27.395040+0000","last_undegraded":"2023-10-16T13:48:27.395040+0000","last_fullsized":"2023-10-16T13:48:27.395040+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:21:57.810695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.13","version":"264'2","reported_seq":14,"reported_epoch":264,"state":"active+clean","last_fresh":"2023-10-16T13:48:25.454665+0000","last_change":"2023-10-16T13:48:24.426307+0000","last_active":"2023-10-16T13:48:25.454665+0000","last_peered":"2023-10-16T13:48:25.454665+0000","last_clean":"2023-10-16T13:48:25.454665+0000","last_became_active":"2023-10-16T13:48:24.425993+0000","last_became_peered":"2023-10-16T13:48:24.425993+0000","last_unstale":"2023-10-16T13:48:25.454665+0000","last_undegraded":"2023-10-16T13:48:25.454665+0000","last_fullsized":"2023-10-16T13:48:25.454665+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_clean_scrub_stamp":"2023-10-16T13:48:23.395571+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:51:50.387274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.15","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.394429+0000","last_change":"2023-10-16T13:48:25.440326+0000","last_active":"2023-10-16T13:48:27.394429+0000","last_peered":"2023-10-16T13:48:27.394429+0000","last_clean":"2023-10-16T13:48:27.394429+0000","last_became_active":"2023-10-16T13:48:25.440012+0000","last_became_peered":"2023-10-16T13:48:25.440012+0000","last_unstale":"2023-10-16T13:48:27.394429+0000","last_undegraded":"2023-10-16T13:48:27.394429+0000","last_fullsized":"2023-10-16T13:48:27.394429+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:20:52.895043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.12","version":"0'0","reported_seq":12,"reported_epoch":264,"state":"active+clean","last_fresh":"2023-10-16T13:48:25.428462+0000","last_change":"2023-10-16T13:48:24.423607+0000","last_active":"2023-10-16T13:48:25.428462+0000","last_peered":"2023-10-16T13:48:25.428462+0000","last_clean":"2023-10-16T13:48:25.428462+0000","last_became_active":"2023-10-16T13:48:24.423294+0000","last_became_peered":"2023-10-16T13:48:24.423294+0000","last_unstale":"2023-10-16T13:48:25.428462+0000","last_undegraded":"2023-10-16T13:48:25.428462+0000","last_fullsized":"2023-10-16T13:48:25.428462+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_clean_scrub_stamp":"2023-10-16T13:48:23.395571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:20:38.283795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.16","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.394476+0000","last_change":"2023-10-16T13:48:25.433847+0000","last_active":"2023-10-16T13:48:27.394476+0000","last_peered":"2023-10-16T13:48:27.394476+0000","last_clean":"2023-10-16T13:48:27.394476+0000","last_became_active":"2023-10-16T13:48:25.433563+0000","last_became_peered":"2023-10-16T13:48:25.433563+0000","last_unstale":"2023-10-16T13:48:27.394476+0000","last_undegraded":"2023-10-16T13:48:27.394476+0000","last_fullsized":"2023-10-16T13:48:27.394476+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:00:13.116040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.11","version":"0'0","reported_seq":12,"reported_epoch":264,"state":"active+clean","last_fresh":"2023-10-16T13:48:25.429723+0000","last_change":"2023-10-16T13:48:24.422134+0000","last_active":"2023-10-16T13:48:25.429723+0000","last_peered":"2023-10-16T13:48:25.429723+0000","last_clean":"2023-10-16T13:48:25.429723+0000","last_became_active":"2023-10-16T13:48:24.421631+0000","last_became_peered":"2023-10-16T13:48:24.421631+0000","last_unstale":"2023-10-16T13:48:25.429723+0000","last_undegraded":"2023-10-16T13:48:25.429723+0000","last_fullsized":"2023-10-16T13:48:25.429723+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_clean_scrub_stamp":"2023-10-16T13:48:23.395571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:55:02.920642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.17","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.395059+0000","last_change":"2023-10-16T13:48:25.433217+0000","last_active":"2023-10-16T13:48:27.395059+0000","last_peered":"2023-10-16T13:48:27.395059+0000","last_clean":"2023-10-16T13:48:27.395059+0000","last_became_active":"2023-10-16T13:48:25.432642+0000","last_became_peered":"2023-10-16T13:48:25.432642+0000","last_unstale":"2023-10-16T13:48:27.395059+0000","last_undegraded":"2023-10-16T13:48:27.395059+0000","last_fullsized":"2023-10-16T13:48:27.395059+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:10:05.461399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.10","version":"264'8","reported_seq":20,"reported_epoch":264,"state":"active+clean","last_fresh":"2023-10-16T13:48:25.455985+0000","last_change":"2023-10-16T13:48:24.428346+0000","last_active":"2023-10-16T13:48:25.455985+0000","last_peered":"2023-10-16T13:48:25.455985+0000","last_clean":"2023-10-16T13:48:25.455985+0000","last_became_active":"2023-10-16T13:48:24.428022+0000","last_became_peered":"2023-10-16T13:48:24.428022+0000","last_unstale":"2023-10-16T13:48:25.455985+0000","last_undegraded":"2023-10-16T13:48:25.455985+0000","last_fullsized":"2023-10-16T13:48:25.455985+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_clean_scrub_stamp":"2023-10-16T13:48:23.395571+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:09:41.055965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.8","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.393150+0000","last_change":"2023-10-16T13:48:25.426854+0000","last_active":"2023-10-16T13:48:27.393150+0000","last_peered":"2023-10-16T13:48:27.393150+0000","last_clean":"2023-10-16T13:48:27.393150+0000","last_became_active":"2023-10-16T13:48:25.426546+0000","last_became_peered":"2023-10-16T13:48:25.426546+0000","last_unstale":"2023-10-16T13:48:27.393150+0000","last_undegraded":"2023-10-16T13:48:27.393150+0000","last_fullsized":"2023-10-16T13:48:27.393150+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:35:54.607900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.f","version":"264'1","reported_seq":13,"reported_epoch":264,"state":"active+clean","last_fresh":"2023-10-16T13:48:25.455007+0000","last_change":"2023-10-16T13:48:24.418499+0000","last_active":"2023-10-16T13:48:25.455007+0000","last_peered":"2023-10-16T13:48:25.455007+0000","last_clean":"2023-10-16T13:48:25.455007+0000","last_became_active":"2023-10-16T13:48:24.418095+0000","last_became_peered":"2023-10-16T13:48:24.418095+0000","last_unstale":"2023-10-16T13:48:25.455007+0000","last_undegraded":"2023-10-16T13:48:25.455007+0000","last_fullsized":"2023-10-16T13:48:25.455007+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_clean_scrub_stamp":"2023-10-16T13:48:23.395571+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:41:15.591732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.9","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:26.421154+0000","last_change":"2023-10-16T13:48:25.443855+0000","last_active":"2023-10-16T13:48:26.421154+0000","last_peered":"2023-10-16T13:48:26.421154+0000","last_clean":"2023-10-16T13:48:26.421154+0000","last_became_active":"2023-10-16T13:48:25.439465+0000","last_became_peered":"2023-10-16T13:48:25.439465+0000","last_unstale":"2023-10-16T13:48:26.421154+0000","last_undegraded":"2023-10-16T13:48:26.421154+0000","last_fullsized":"2023-10-16T13:48:26.421154+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:33:34.431164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.e","version":"264'2","reported_seq":14,"reported_epoch":264,"state":"active+clean","last_fresh":"2023-10-16T13:48:25.452636+0000","last_change":"2023-10-16T13:48:24.423791+0000","last_active":"2023-10-16T13:48:25.452636+0000","last_peered":"2023-10-16T13:48:25.452636+0000","last_clean":"2023-10-16T13:48:25.452636+0000","last_became_active":"2023-10-16T13:48:24.423494+0000","last_became_peered":"2023-10-16T13:48:24.423494+0000","last_unstale":"2023-10-16T13:48:25.452636+0000","last_undegraded":"2023-10-16T13:48:25.452636+0000","last_fullsized":"2023-10-16T13:48:25.452636+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_clean_scrub_stamp":"2023-10-16T13:48:23.395571+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:14:31.124740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.a","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.393392+0000","last_change":"2023-10-16T13:48:25.427450+0000","last_active":"2023-10-16T13:48:27.393392+0000","last_peered":"2023-10-16T13:48:27.393392+0000","last_clean":"2023-10-16T13:48:27.393392+0000","last_became_active":"2023-10-16T13:48:25.427170+0000","last_became_peered":"2023-10-16T13:48:25.427170+0000","last_unstale":"2023-10-16T13:48:27.393392+0000","last_undegraded":"2023-10-16T13:48:27.393392+0000","last_fullsized":"2023-10-16T13:48:27.393392+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:04:33.594311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.d","version":"264'1","reported_seq":13,"reported_epoch":264,"state":"active+clean","last_fresh":"2023-10-16T13:48:25.454791+0000","last_change":"2023-10-16T13:48:24.422463+0000","last_active":"2023-10-16T13:48:25.454791+0000","last_peered":"2023-10-16T13:48:25.454791+0000","last_clean":"2023-10-16T13:48:25.454791+0000","last_became_active":"2023-10-16T13:48:24.421831+0000","last_became_peered":"2023-10-16T13:48:24.421831+0000","last_unstale":"2023-10-16T13:48:25.454791+0000","last_undegraded":"2023-10-16T13:48:25.454791+0000","last_fullsized":"2023-10-16T13:48:25.454791+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_clean_scrub_stamp":"2023-10-16T13:48:23.395571+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:26:00.824106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.b","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.394177+0000","last_change":"2023-10-16T13:48:25.438352+0000","last_active":"2023-10-16T13:48:27.394177+0000","last_peered":"2023-10-16T13:48:27.394177+0000","last_clean":"2023-10-16T13:48:27.394177+0000","last_became_active":"2023-10-16T13:48:25.438027+0000","last_became_peered":"2023-10-16T13:48:25.438027+0000","last_unstale":"2023-10-16T13:48:27.394177+0000","last_undegraded":"2023-10-16T13:48:27.394177+0000","last_fullsized":"2023-10-16T13:48:27.394177+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:50:53.215873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.c","version":"0'0","reported_seq":12,"reported_epoch":264,"state":"active+clean","last_fresh":"2023-10-16T13:48:25.422405+0000","last_change":"2023-10-16T13:48:24.421504+0000","last_active":"2023-10-16T13:48:25.422405+0000","last_peered":"2023-10-16T13:48:25.422405+0000","last_clean":"2023-10-16T13:48:25.422405+0000","last_became_active":"2023-10-16T13:48:24.421202+0000","last_became_peered":"2023-10-16T13:48:24.421202+0000","last_unstale":"2023-10-16T13:48:25.422405+0000","last_undegraded":"2023-10-16T13:48:25.422405+0000","last_fullsized":"2023-10-16T13:48:25.422405+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_clean_scrub_stamp":"2023-10-16T13:48:23.395571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:08:57.374827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.c","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:26.421401+0000","last_change":"2023-10-16T13:48:25.444963+0000","last_active":"2023-10-16T13:48:26.421401+0000","last_peered":"2023-10-16T13:48:26.421401+0000","last_clean":"2023-10-16T13:48:26.421401+0000","last_became_active":"2023-10-16T13:48:25.444659+0000","last_became_peered":"2023-10-16T13:48:25.444659+0000","last_unstale":"2023-10-16T13:48:26.421401+0000","last_undegraded":"2023-10-16T13:48:26.421401+0000","last_fullsized":"2023-10-16T13:48:26.421401+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:08:59.756642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.b","version":"264'2","reported_seq":14,"reported_epoch":264,"state":"active+clean","last_fresh":"2023-10-16T13:48:25.454708+0000","last_change":"2023-10-16T13:48:24.415442+0000","last_active":"2023-10-16T13:48:25.454708+0000","last_peered":"2023-10-16T13:48:25.454708+0000","last_clean":"2023-10-16T13:48:25.454708+0000","last_became_active":"2023-10-16T13:48:24.415108+0000","last_became_peered":"2023-10-16T13:48:24.415108+0000","last_unstale":"2023-10-16T13:48:25.454708+0000","last_undegraded":"2023-10-16T13:48:25.454708+0000","last_fullsized":"2023-10-16T13:48:25.454708+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_clean_scrub_stamp":"2023-10-16T13:48:23.395571+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:00:09.744808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.d","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:26.422403+0000","last_change":"2023-10-16T13:48:25.435788+0000","last_active":"2023-10-16T13:48:26.422403+0000","last_peered":"2023-10-16T13:48:26.422403+0000","last_clean":"2023-10-16T13:48:26.422403+0000","last_became_active":"2023-10-16T13:48:25.435458+0000","last_became_peered":"2023-10-16T13:48:25.435458+0000","last_unstale":"2023-10-16T13:48:26.422403+0000","last_undegraded":"2023-10-16T13:48:26.422403+0000","last_fullsized":"2023-10-16T13:48:26.422403+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:18:02.241801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.a","version":"0'0","reported_seq":12,"reported_epoch":264,"state":"active+clean","last_fresh":"2023-10-16T13:48:25.428892+0000","last_change":"2023-10-16T13:48:24.429991+0000","last_active":"2023-10-16T13:48:25.428892+0000","last_peered":"2023-10-16T13:48:25.428892+0000","last_clean":"2023-10-16T13:48:25.428892+0000","last_became_active":"2023-10-16T13:48:24.429622+0000","last_became_peered":"2023-10-16T13:48:24.429622+0000","last_unstale":"2023-10-16T13:48:25.428892+0000","last_undegraded":"2023-10-16T13:48:25.428892+0000","last_fullsized":"2023-10-16T13:48:25.428892+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_clean_scrub_stamp":"2023-10-16T13:48:23.395571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:47:29.855885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.e","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.394644+0000","last_change":"2023-10-16T13:48:25.436515+0000","last_active":"2023-10-16T13:48:27.394644+0000","last_peered":"2023-10-16T13:48:27.394644+0000","last_clean":"2023-10-16T13:48:27.394644+0000","last_became_active":"2023-10-16T13:48:25.436207+0000","last_became_peered":"2023-10-16T13:48:25.436207+0000","last_unstale":"2023-10-16T13:48:27.394644+0000","last_undegraded":"2023-10-16T13:48:27.394644+0000","last_fullsized":"2023-10-16T13:48:27.394644+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:40:28.418592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.9","version":"0'0","reported_seq":12,"reported_epoch":264,"state":"active+clean","last_fresh":"2023-10-16T13:48:25.435597+0000","last_change":"2023-10-16T13:48:24.425322+0000","last_active":"2023-10-16T13:48:25.435597+0000","last_peered":"2023-10-16T13:48:25.435597+0000","last_clean":"2023-10-16T13:48:25.435597+0000","last_became_active":"2023-10-16T13:48:24.424311+0000","last_became_peered":"2023-10-16T13:48:24.424311+0000","last_unstale":"2023-10-16T13:48:25.435597+0000","last_undegraded":"2023-10-16T13:48:25.435597+0000","last_fullsized":"2023-10-16T13:48:25.435597+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_clean_scrub_stamp":"2023-10-16T13:48:23.395571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:12:15.257850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.f","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.393378+0000","last_change":"2023-10-16T13:48:25.444978+0000","last_active":"2023-10-16T13:48:27.393378+0000","last_peered":"2023-10-16T13:48:27.393378+0000","last_clean":"2023-10-16T13:48:27.393378+0000","last_became_active":"2023-10-16T13:48:25.444683+0000","last_became_peered":"2023-10-16T13:48:25.444683+0000","last_unstale":"2023-10-16T13:48:27.393378+0000","last_undegraded":"2023-10-16T13:48:27.393378+0000","last_fullsized":"2023-10-16T13:48:27.393378+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:30:42.136595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.8","version":"0'0","reported_seq":12,"reported_epoch":264,"state":"active+clean","last_fresh":"2023-10-16T13:48:25.423666+0000","last_change":"2023-10-16T13:48:24.426952+0000","last_active":"2023-10-16T13:48:25.423666+0000","last_peered":"2023-10-16T13:48:25.423666+0000","last_clean":"2023-10-16T13:48:25.423666+0000","last_became_active":"2023-10-16T13:48:24.426401+0000","last_became_peered":"2023-10-16T13:48:24.426401+0000","last_unstale":"2023-10-16T13:48:25.423666+0000","last_undegraded":"2023-10-16T13:48:25.423666+0000","last_fullsized":"2023-10-16T13:48:25.423666+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_clean_scrub_stamp":"2023-10-16T13:48:23.395571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:58:42.161578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.0","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.393496+0000","last_change":"2023-10-16T13:48:25.438817+0000","last_active":"2023-10-16T13:48:27.393496+0000","last_peered":"2023-10-16T13:48:27.393496+0000","last_clean":"2023-10-16T13:48:27.393496+0000","last_became_active":"2023-10-16T13:48:25.438538+0000","last_became_peered":"2023-10-16T13:48:25.438538+0000","last_unstale":"2023-10-16T13:48:27.393496+0000","last_undegraded":"2023-10-16T13:48:27.393496+0000","last_fullsized":"2023-10-16T13:48:27.393496+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:24:08.987676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.7","version":"264'2","reported_seq":14,"reported_epoch":264,"state":"active+clean","last_fresh":"2023-10-16T13:48:25.454312+0000","last_change":"2023-10-16T13:48:24.418531+0000","last_active":"2023-10-16T13:48:25.454312+0000","last_peered":"2023-10-16T13:48:25.454312+0000","last_clean":"2023-10-16T13:48:25.454312+0000","last_became_active":"2023-10-16T13:48:24.418119+0000","last_became_peered":"2023-10-16T13:48:24.418119+0000","last_unstale":"2023-10-16T13:48:25.454312+0000","last_undegraded":"2023-10-16T13:48:25.454312+0000","last_fullsized":"2023-10-16T13:48:25.454312+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_clean_scrub_stamp":"2023-10-16T13:48:23.395571+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:17:02.382002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.1","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.393530+0000","last_change":"2023-10-16T13:48:25.438726+0000","last_active":"2023-10-16T13:48:27.393530+0000","last_peered":"2023-10-16T13:48:27.393530+0000","last_clean":"2023-10-16T13:48:27.393530+0000","last_became_active":"2023-10-16T13:48:25.435116+0000","last_became_peered":"2023-10-16T13:48:25.435116+0000","last_unstale":"2023-10-16T13:48:27.393530+0000","last_undegraded":"2023-10-16T13:48:27.393530+0000","last_fullsized":"2023-10-16T13:48:27.393530+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:42:01.034277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.6","version":"0'0","reported_seq":12,"reported_epoch":264,"state":"active+clean","last_fresh":"2023-10-16T13:48:25.429482+0000","last_change":"2023-10-16T13:48:24.423801+0000","last_active":"2023-10-16T13:48:25.429482+0000","last_peered":"2023-10-16T13:48:25.429482+0000","last_clean":"2023-10-16T13:48:25.429482+0000","last_became_active":"2023-10-16T13:48:24.423451+0000","last_became_peered":"2023-10-16T13:48:24.423451+0000","last_unstale":"2023-10-16T13:48:25.429482+0000","last_undegraded":"2023-10-16T13:48:25.429482+0000","last_fullsized":"2023-10-16T13:48:25.429482+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_clean_scrub_stamp":"2023-10-16T13:48:23.395571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:41:33.359406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.2","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.393722+0000","last_change":"2023-10-16T13:48:25.445560+0000","last_active":"2023-10-16T13:48:27.393722+0000","last_peered":"2023-10-16T13:48:27.393722+0000","last_clean":"2023-10-16T13:48:27.393722+0000","last_became_active":"2023-10-16T13:48:25.445064+0000","last_became_peered":"2023-10-16T13:48:25.445064+0000","last_unstale":"2023-10-16T13:48:27.393722+0000","last_undegraded":"2023-10-16T13:48:27.393722+0000","last_fullsized":"2023-10-16T13:48:27.393722+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:57:58.737918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.5","version":"0'0","reported_seq":12,"reported_epoch":264,"state":"active+clean","last_fresh":"2023-10-16T13:48:25.426272+0000","last_change":"2023-10-16T13:48:24.429667+0000","last_active":"2023-10-16T13:48:25.426272+0000","last_peered":"2023-10-16T13:48:25.426272+0000","last_clean":"2023-10-16T13:48:25.426272+0000","last_became_active":"2023-10-16T13:48:24.429339+0000","last_became_peered":"2023-10-16T13:48:24.429339+0000","last_unstale":"2023-10-16T13:48:25.426272+0000","last_undegraded":"2023-10-16T13:48:25.426272+0000","last_fullsized":"2023-10-16T13:48:25.426272+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_clean_scrub_stamp":"2023-10-16T13:48:23.395571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:27:40.075970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.3","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.164453+0000","last_change":"2023-10-16T13:48:25.433691+0000","last_active":"2023-10-16T13:48:27.164453+0000","last_peered":"2023-10-16T13:48:27.164453+0000","last_clean":"2023-10-16T13:48:27.164453+0000","last_became_active":"2023-10-16T13:48:25.433385+0000","last_became_peered":"2023-10-16T13:48:25.433385+0000","last_unstale":"2023-10-16T13:48:27.164453+0000","last_undegraded":"2023-10-16T13:48:27.164453+0000","last_fullsized":"2023-10-16T13:48:27.164453+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:06:27.452752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.4","version":"0'0","reported_seq":12,"reported_epoch":264,"state":"active+clean","last_fresh":"2023-10-16T13:48:25.429614+0000","last_change":"2023-10-16T13:48:24.423134+0000","last_active":"2023-10-16T13:48:25.429614+0000","last_peered":"2023-10-16T13:48:25.429614+0000","last_clean":"2023-10-16T13:48:25.429614+0000","last_became_active":"2023-10-16T13:48:24.422846+0000","last_became_peered":"2023-10-16T13:48:24.422846+0000","last_unstale":"2023-10-16T13:48:25.429614+0000","last_undegraded":"2023-10-16T13:48:25.429614+0000","last_fullsized":"2023-10-16T13:48:25.429614+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_clean_scrub_stamp":"2023-10-16T13:48:23.395571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:47:35.226854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.4","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.164713+0000","last_change":"2023-10-16T13:48:25.435062+0000","last_active":"2023-10-16T13:48:27.164713+0000","last_peered":"2023-10-16T13:48:27.164713+0000","last_clean":"2023-10-16T13:48:27.164713+0000","last_became_active":"2023-10-16T13:48:25.433884+0000","last_became_peered":"2023-10-16T13:48:25.433884+0000","last_unstale":"2023-10-16T13:48:27.164713+0000","last_undegraded":"2023-10-16T13:48:27.164713+0000","last_fullsized":"2023-10-16T13:48:27.164713+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:30:42.136595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.3","version":"0'0","reported_seq":12,"reported_epoch":264,"state":"active+clean","last_fresh":"2023-10-16T13:48:25.422718+0000","last_change":"2023-10-16T13:48:24.423099+0000","last_active":"2023-10-16T13:48:25.422718+0000","last_peered":"2023-10-16T13:48:25.422718+0000","last_clean":"2023-10-16T13:48:25.422718+0000","last_became_active":"2023-10-16T13:48:24.422766+0000","last_became_peered":"2023-10-16T13:48:24.422766+0000","last_unstale":"2023-10-16T13:48:25.422718+0000","last_undegraded":"2023-10-16T13:48:25.422718+0000","last_fullsized":"2023-10-16T13:48:25.422718+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_clean_scrub_stamp":"2023-10-16T13:48:23.395571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:06:49.269879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.5","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.394496+0000","last_change":"2023-10-16T13:48:25.434071+0000","last_active":"2023-10-16T13:48:27.394496+0000","last_peered":"2023-10-16T13:48:27.394496+0000","last_clean":"2023-10-16T13:48:27.394496+0000","last_became_active":"2023-10-16T13:48:25.431980+0000","last_became_peered":"2023-10-16T13:48:25.431980+0000","last_unstale":"2023-10-16T13:48:27.394496+0000","last_undegraded":"2023-10-16T13:48:27.394496+0000","last_fullsized":"2023-10-16T13:48:27.394496+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:38:43.055399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.2","version":"0'0","reported_seq":12,"reported_epoch":264,"state":"active+clean","last_fresh":"2023-10-16T13:48:25.430077+0000","last_change":"2023-10-16T13:48:24.423228+0000","last_active":"2023-10-16T13:48:25.430077+0000","last_peered":"2023-10-16T13:48:25.430077+0000","last_clean":"2023-10-16T13:48:25.430077+0000","last_became_active":"2023-10-16T13:48:24.422695+0000","last_became_peered":"2023-10-16T13:48:24.422695+0000","last_unstale":"2023-10-16T13:48:25.430077+0000","last_undegraded":"2023-10-16T13:48:25.430077+0000","last_fullsized":"2023-10-16T13:48:25.430077+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_clean_scrub_stamp":"2023-10-16T13:48:23.395571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:27:43.879103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.6","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.393740+0000","last_change":"2023-10-16T13:48:25.433978+0000","last_active":"2023-10-16T13:48:27.393740+0000","last_peered":"2023-10-16T13:48:27.393740+0000","last_clean":"2023-10-16T13:48:27.393740+0000","last_became_active":"2023-10-16T13:48:25.433676+0000","last_became_peered":"2023-10-16T13:48:25.433676+0000","last_unstale":"2023-10-16T13:48:27.393740+0000","last_undegraded":"2023-10-16T13:48:27.393740+0000","last_fullsized":"2023-10-16T13:48:27.393740+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:08:24.915543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.1","version":"0'0","reported_seq":12,"reported_epoch":264,"state":"active+clean","last_fresh":"2023-10-16T13:48:25.432357+0000","last_change":"2023-10-16T13:48:24.430561+0000","last_active":"2023-10-16T13:48:25.432357+0000","last_peered":"2023-10-16T13:48:25.432357+0000","last_clean":"2023-10-16T13:48:25.432357+0000","last_became_active":"2023-10-16T13:48:24.430259+0000","last_became_peered":"2023-10-16T13:48:24.430259+0000","last_unstale":"2023-10-16T13:48:25.432357+0000","last_undegraded":"2023-10-16T13:48:25.432357+0000","last_fullsized":"2023-10-16T13:48:25.432357+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_clean_scrub_stamp":"2023-10-16T13:48:23.395571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:02:31.781056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.7","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.394456+0000","last_change":"2023-10-16T13:48:25.445845+0000","last_active":"2023-10-16T13:48:27.394456+0000","last_peered":"2023-10-16T13:48:27.394456+0000","last_clean":"2023-10-16T13:48:27.394456+0000","last_became_active":"2023-10-16T13:48:25.445533+0000","last_became_peered":"2023-10-16T13:48:25.445533+0000","last_unstale":"2023-10-16T13:48:27.394456+0000","last_undegraded":"2023-10-16T13:48:27.394456+0000","last_fullsized":"2023-10-16T13:48:27.394456+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:22:49.036856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.0","version":"0'0","reported_seq":12,"reported_epoch":264,"state":"active+clean","last_fresh":"2023-10-16T13:48:25.423576+0000","last_change":"2023-10-16T13:48:24.426714+0000","last_active":"2023-10-16T13:48:25.423576+0000","last_peered":"2023-10-16T13:48:25.423576+0000","last_clean":"2023-10-16T13:48:25.423576+0000","last_became_active":"2023-10-16T13:48:24.425908+0000","last_became_peered":"2023-10-16T13:48:24.425908+0000","last_unstale":"2023-10-16T13:48:25.423576+0000","last_undegraded":"2023-10-16T13:48:25.423576+0000","last_fullsized":"2023-10-16T13:48:25.423576+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:23.395571+0000","last_clean_scrub_stamp":"2023-10-16T13:48:23.395571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:10:48.742062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.25","version":"0'0","reported_seq":11,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-16T13:48:27.393896+0000","last_change":"2023-10-16T13:48:25.424261+0000","last_active":"2023-10-16T13:48:27.393896+0000","last_peered":"2023-10-16T13:48:27.393896+0000","last_clean":"2023-10-16T13:48:27.393896+0000","last_became_active":"2023-10-16T13:48:25.423811+0000","last_became_peered":"2023-10-16T13:48:25.423811+0000","last_unstale":"2023-10-16T13:48:27.393896+0000","last_undegraded":"2023-10-16T13:48:27.393896+0000","last_fullsized":"2023-10-16T13:48:27.393896+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:48:24.399988+0000","last_clean_scrub_stamp":"2023-10-16T13:48:24.399988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:50:46.522916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"251'381","reported_seq":1355,"reported_epoch":264,"state":"active+clean","last_fresh":"2023-10-16T13:48:25.424566+0000","last_change":"2023-10-16T12:10:34.746820+0000","last_active":"2023-10-16T13:48:25.424566+0000","last_peered":"2023-10-16T13:48:25.424566+0000","last_clean":"2023-10-16T13:48:25.424566+0000","last_became_active":"2023-10-16T12:10:34.745574+0000","last_became_peered":"2023-10-16T12:10:34.745574+0000","last_unstale":"2023-10-16T13:48:25.424566+0000","last_undegraded":"2023-10-16T13:48:25.424566+0000","last_fullsized":"2023-10-16T13:48:25.424566+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_clean_scrub_stamp":"2023-10-16T12:10:33.577111+0000","objects_scrubbed":0,"log_size":381,"log_dups_size":0,"ondisk_log_size":381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:14.461986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2420,"num_read_kb":5044,"num_write":639,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":35,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":36,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2420,"num_read_kb":5044,"num_write":639,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":381,"ondisk_log_size":381,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576007,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57264,"kb_used_data":3872,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314576,"statfs":{"total":96636764160,"available":96578125824,"internally_reserved":0,"allocated":3964928,"data_stored":2332951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54655435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:47:51 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.57599999999999996,"15min":0.621},"min":{"1min":0.38200000000000001,"5min":0.34200000000000003,"15min":0.33000000000000002},"max":{"1min":0.76600000000000001,"5min":0.87,"15min":0.93799999999999994},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.58599999999999997,"15min":0.61799999999999999},"min":{"1min":0.34899999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.79700000000000004,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.51300000000000001}]},{"osd":1,"last update":"Mon Oct 16 13:47:51 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.59499999999999997,"15min":0.63200000000000001},"min":{"1min":0.35999999999999999,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.78300000000000003,"5min":0.80400000000000005,"15min":0.95899999999999996},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.60099999999999998,"15min":0.624},"min":{"1min":0.39800000000000002,"5min":0.375,"15min":0.375},"max":{"1min":0.76400000000000001,"5min":0.78700000000000003,"15min":0.86399999999999999},"last":0.59299999999999997}]},{"osd":2,"last update":"Mon Oct 16 13:47:51 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.65400000000000003,"15min":0.67800000000000005},"min":{"1min":0.44900000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.78300000000000003,"5min":0.83899999999999997,"15min":0.88200000000000001},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.66700000000000004,"15min":0.69099999999999995},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.81899999999999995,"5min":0.90200000000000002,"15min":0.93799999999999994},"last":0.78300000000000003}]},{"osd":3,"last update":"Mon Oct 16 13:47:51 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.623,"15min":0.65100000000000002},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.85199999999999998,"5min":0.85299999999999998,"15min":0.91300000000000003},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.64600000000000002,"15min":0.65900000000000003},"min":{"1min":0.48099999999999998,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.86499999999999999,"5min":1.24,"15min":1.24},"last":0.46200000000000002}]},{"osd":4,"last update":"Mon Oct 16 13:47:51 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.33200000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.435},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.63100000000000001,"15min":0.621},"min":{"1min":0.45600000000000002,"5min":0.34399999999999997,"15min":0.33100000000000002},"max":{"1min":0.93799999999999994,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.81200000000000006}]},{"osd":5,"last update":"Mon Oct 16 13:47:51 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.56799999999999995,"15min":0.55400000000000005},"min":{"1min":0.35899999999999999,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.998,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.60499999999999998,"15min":0.59399999999999997},"min":{"1min":0.42799999999999999,"5min":0.39700000000000002,"15min":0.35099999999999998},"max":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.52700000000000002}]},{"osd":6,"last update":"Mon Oct 16 13:47:51 2023","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.62,"15min":0.61899999999999999},"min":{"1min":0.48499999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.022,"5min":1.022,"15min":1.022},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64300000000000002,"15min":0.623},"min":{"1min":0.38,"5min":0.378,"15min":0.35199999999999998},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.753}]}]},{"osd":6,"up_from":13,"seq":55834576005,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64980,"kb_used_data":3272,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94306860,"statfs":{"total":96636764160,"available":96570224640,"internally_reserved":0,"allocated":3350528,"data_stored":1740319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":63175111},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:47:39 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.64100000000000001,"15min":0.66300000000000003},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.29599999999999999},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.63700000000000001,"15min":0.65800000000000003},"min":{"1min":0.36499999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.81100000000000005,"5min":0.91400000000000003,"15min":0.93799999999999994},"last":0.50800000000000001}]},{"osd":1,"last update":"Mon Oct 16 13:47:39 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.57099999999999995,"15min":0.61799999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.76600000000000001,"5min":0.86899999999999999,"15min":0.90900000000000003},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.61599999999999999,"15min":0.61699999999999999},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.41199999999999998}]},{"osd":2,"last update":"Mon Oct 16 13:47:39 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.60799999999999998,"15min":0.627},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.79600000000000004,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.59699999999999998,"15min":0.61799999999999999},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.2769999999999999,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.39400000000000002}]},{"osd":3,"last update":"Mon Oct 16 13:47:39 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.625,"15min":0.65600000000000003},"min":{"1min":0.40100000000000002,"5min":0.39900000000000002,"15min":0.38200000000000001},"max":{"1min":0.75600000000000001,"5min":0.92400000000000004,"15min":0.95599999999999996},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.68400000000000005,"15min":0.71199999999999997},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.42299999999999999},"max":{"1min":0.85199999999999998,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.52500000000000002}]},{"osd":4,"last update":"Mon Oct 16 13:47:39 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.52500000000000002,"15min":0.53000000000000003},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.748,"5min":0.85599999999999998,"15min":0.95799999999999996},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.61199999999999999,"15min":0.621},"min":{"1min":0.38400000000000001,"5min":0.35499999999999998,"15min":0.32900000000000001},"max":{"1min":0.79100000000000004,"5min":0.98799999999999999,"15min":1.0309999999999999},"last":0.56399999999999995}]},{"osd":5,"last update":"Mon Oct 16 13:47:39 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.58799999999999997,"15min":0.59499999999999997},"min":{"1min":0.34399999999999997,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.80000000000000004,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.64000000000000001,"15min":0.65900000000000003},"min":{"1min":0.36099999999999999,"5min":0.33700000000000002,"15min":0.308},"max":{"1min":0.86399999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.63800000000000001}]},{"osd":7,"last update":"Mon Oct 16 13:47:39 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.58899999999999997,"15min":0.60299999999999998},"min":{"1min":0.38700000000000001,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.79300000000000004,"5min":1.081,"15min":1.081},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.63,"5min":0.64100000000000001,"15min":0.65400000000000003},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.375},"max":{"1min":0.83499999999999996,"5min":1.099,"15min":1.099},"last":0.54500000000000004}]}]},{"osd":1,"up_from":13,"seq":55834576005,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52504,"kb_used_data":3276,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319336,"statfs":{"total":96636764160,"available":96583000064,"internally_reserved":0,"allocated":3354624,"data_stored":1740409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":50395593},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:47:55 2023","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.38100000000000001,"15min":0.40400000000000003},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.59199999999999997,"5min":0.65400000000000003,"15min":0.65800000000000003},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.45600000000000002,"15min":0.48699999999999999},"min":{"1min":0.29099999999999998,"5min":0.253,"15min":0.253},"max":{"1min":0.67000000000000004,"5min":0.70999999999999996,"15min":0.77200000000000002},"last":0.38500000000000001}]},{"osd":2,"last update":"Mon Oct 16 13:47:55 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.41699999999999998,"15min":0.44600000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.57799999999999996,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.44,"5min":0.46200000000000002,"15min":0.48599999999999999},"min":{"1min":0.308,"5min":0.308,"15min":0.30199999999999999},"max":{"1min":0.65200000000000002,"5min":0.67600000000000005,"15min":0.71399999999999997},"last":0.28799999999999998}]},{"osd":3,"last update":"Mon Oct 16 13:47:55 2023","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.41099999999999998,"15min":0.42499999999999999},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.53200000000000003,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.439,"15min":0.47099999999999997},"min":{"1min":0.27100000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.69399999999999995,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.36599999999999999}]},{"osd":4,"last update":"Mon Oct 16 13:47:55 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.63600000000000001,"15min":0.66000000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.626,"15min":0.66200000000000003},"min":{"1min":0.39900000000000002,"5min":0.37,"15min":0.37},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.997},"last":0.69299999999999995}]},{"osd":5,"last update":"Mon Oct 16 13:47:55 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.67000000000000004,"15min":0.70499999999999996},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":1.0760000000000001},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.68500000000000005,"15min":0.72099999999999997},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.754}]},{"osd":6,"last update":"Mon Oct 16 13:47:55 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.65200000000000002,"15min":0.67700000000000005},"min":{"1min":0.41199999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.80300000000000005,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.67100000000000004,"15min":0.70199999999999996},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.88100000000000001,"5min":0.998,"15min":1.083},"last":0.61099999999999999}]},{"osd":7,"last update":"Mon Oct 16 13:47:55 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.69099999999999995,"15min":0.72899999999999998},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.163,"5min":1.163,"15min":1.163},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.71299999999999997,"15min":0.74299999999999999},"min":{"1min":0.47599999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.92500000000000004,"5min":1.042,"15min":1.0580000000000001},"last":0.76900000000000002}]}]},{"osd":0,"up_from":13,"seq":55834576006,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48928,"kb_used_data":3860,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94322912,"statfs":{"total":96636764160,"available":96586661888,"internally_reserved":0,"allocated":3952640,"data_stored":2331303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 13:47:43 2023","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.39300000000000002,"15min":0.41599999999999998},"min":{"1min":0.245,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.51400000000000001,"5min":0.65100000000000002,"15min":0.71399999999999997},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.40200000000000002,"15min":0.43099999999999999},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.41999999999999998,"5min":0.77700000000000002,"15min":0.82199999999999995},"last":0.29599999999999999}]},{"osd":2,"last update":"Mon Oct 16 13:47:43 2023","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.43099999999999999,"15min":0.44800000000000001},"min":{"1min":0.32000000000000001,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.45900000000000002,"5min":0.65900000000000003,"15min":0.75700000000000001},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.44800000000000001,"15min":0.47899999999999998},"min":{"1min":0.30499999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.60499999999999998,"5min":0.68799999999999994,"15min":0.83299999999999996},"last":0.38200000000000001}]},{"osd":3,"last update":"Mon Oct 16 13:47:43 2023","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.41999999999999998,"15min":0.44900000000000001},"min":{"1min":0.28100000000000003,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.78600000000000003},"last":0.318},{"interface":"front","average":{"1min":0.432,"5min":0.49199999999999999,"15min":0.51400000000000001},"min":{"1min":0.309,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.58799999999999997,"5min":0.82999999999999996,"15min":0.86399999999999999},"last":0.371}]},{"osd":4,"last update":"Mon Oct 16 13:47:43 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.65800000000000003,"15min":0.68000000000000005},"min":{"1min":0.442,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.85799999999999998,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.66300000000000003,"15min":0.68700000000000006},"min":{"1min":0.437,"5min":0.375,"15min":0.375},"max":{"1min":0.90500000000000003,"5min":1.21,"15min":1.21},"last":0.69699999999999995}]},{"osd":5,"last update":"Mon Oct 16 13:47:43 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.65100000000000002,"15min":0.68200000000000005},"min":{"1min":0.46999999999999997,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.81399999999999995,"5min":1.242,"15min":1.242},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.71099999999999997,"15min":0.72199999999999998},"min":{"1min":0.59499999999999997,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.95699999999999996,"5min":1.081,"15min":1.081},"last":0.71399999999999997}]},{"osd":6,"last update":"Mon Oct 16 13:47:43 2023","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.70699999999999996,"15min":0.72599999999999998},"min":{"1min":0.52000000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.90300000000000002,"5min":1.034,"15min":1.103},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.68899999999999995,"15min":0.71799999999999997},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.79600000000000004,"5min":1.131,"15min":1.131},"last":0.68500000000000005}]},{"osd":7,"last update":"Mon Oct 16 13:47:43 2023","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.70599999999999996,"15min":0.73799999999999999},"min":{"1min":0.48799999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.88800000000000001,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.70999999999999996,"15min":0.73099999999999998},"min":{"1min":0.53700000000000003,"5min":0.433,"15min":0.433},"max":{"1min":0.80800000000000005,"5min":1.087,"15min":1.087},"last":0.66600000000000004}]}]},{"osd":2,"up_from":13,"seq":55834576006,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56664,"kb_used_data":3272,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315176,"statfs":{"total":96636764160,"available":96578740224,"internally_reserved":0,"allocated":3350528,"data_stored":1740319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.39800000000000002,"15min":0.40699999999999997},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.54100000000000004,"5min":0.71699999999999997,"15min":0.77500000000000002},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.45800000000000002,"15min":0.47599999999999998},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.69799999999999995,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.40600000000000003}]},{"osd":1,"last update":"Mon Oct 16 13:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.47899999999999998,"15min":0.495},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.61899999999999999,"5min":0.73199999999999998,"15min":0.81399999999999995},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.47099999999999997,"15min":0.47299999999999998},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.59799999999999998,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.315}]},{"osd":3,"last update":"Mon Oct 16 13:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.432,"15min":0.438},"min":{"1min":0.26300000000000001,"5min":0.255,"15min":0.255},"max":{"1min":0.63,"5min":0.70099999999999996,"15min":0.72599999999999998},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.52400000000000002,"15min":0.54100000000000004},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.74099999999999999,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.44}]},{"osd":4,"last update":"Mon Oct 16 13:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.69299999999999995,"15min":0.71199999999999997},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.94399999999999995,"5min":1.3600000000000001,"15min":1.3600000000000001},"last":0.627},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.70399999999999996,"15min":0.71599999999999997},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.196,"5min":1.403,"15min":1.403},"last":0.60599999999999998}]},{"osd":5,"last update":"Mon Oct 16 13:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.65900000000000003,"15min":0.68400000000000005},"min":{"1min":0.46700000000000003,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.82999999999999996,"5min":1.486,"15min":1.486},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.68200000000000005,"15min":0.69599999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.879,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":0.65300000000000002}]},{"osd":6,"last update":"Mon Oct 16 13:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.65800000000000003,"15min":0.66500000000000004},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.087,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.69799999999999995,"15min":0.71799999999999997},"min":{"1min":0.46000000000000002,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.94299999999999995,"5min":1.4379999999999999,"15min":1.4379999999999999},"last":0.51400000000000001}]},{"osd":7,"last update":"Mon Oct 16 13:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.70499999999999996,"15min":0.71499999999999997},"min":{"1min":0.42599999999999999,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.97099999999999997,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":0.503},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.70299999999999996,"15min":0.72899999999999998},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.0429999999999999,"5min":1.377,"15min":1.377},"last":0.56999999999999995}]}]},{"osd":3,"up_from":13,"seq":55834576005,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56672,"kb_used_data":3280,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315168,"statfs":{"total":96636764160,"available":96578732032,"internally_reserved":0,"allocated":3358720,"data_stored":1740411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:47:42 2023","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.39300000000000002,"15min":0.40699999999999997},"min":{"1min":0.27400000000000002,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.60099999999999998,"5min":0.64200000000000002,"15min":0.72899999999999998},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.40000000000000002,"15min":0.42099999999999999},"min":{"1min":0.27000000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.47899999999999998,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.34999999999999998}]},{"osd":1,"last update":"Mon Oct 16 13:47:42 2023","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.40799999999999997,"15min":0.42699999999999999},"min":{"1min":0.27100000000000002,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.46200000000000002,"5min":0.68200000000000005,"15min":0.76300000000000001},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.44600000000000001,"15min":0.47499999999999998},"min":{"1min":0.309,"5min":0.253,"15min":0.253},"max":{"1min":0.65400000000000003,"5min":0.70699999999999996,"15min":0.77800000000000002},"last":0.55000000000000004}]},{"osd":2,"last update":"Mon Oct 16 13:47:42 2023","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.46600000000000003,"15min":0.48499999999999999},"min":{"1min":0.317,"5min":0.316,"15min":0.316},"max":{"1min":0.64700000000000002,"5min":1.167,"15min":1.167},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.48399999999999999,"15min":0.51400000000000001},"min":{"1min":0.32400000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.58099999999999996,"5min":0.73299999999999998,"15min":0.78700000000000003},"last":0.50900000000000001}]},{"osd":4,"last update":"Mon Oct 16 13:47:42 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.626,"15min":0.65800000000000003},"min":{"1min":0.46000000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.69399999999999995,"5min":0.998,"15min":1.137},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.66600000000000004,"15min":0.68899999999999995},"min":{"1min":0.44600000000000001,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.86299999999999999,"5min":1.2110000000000001,"15min":1.2909999999999999},"last":0.63600000000000001}]},{"osd":5,"last update":"Mon Oct 16 13:47:42 2023","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.65700000000000003,"15min":0.69999999999999996},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.374},"max":{"1min":0.85099999999999998,"5min":1.0680000000000001,"15min":1.2729999999999999},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.66900000000000004,"15min":0.69399999999999995},"min":{"1min":0.48799999999999999,"5min":0.45600000000000002,"15min":0.432},"max":{"1min":0.77400000000000002,"5min":1.0549999999999999,"15min":1.173},"last":0.56899999999999995}]},{"osd":6,"last update":"Mon Oct 16 13:47:42 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.70999999999999996,"15min":0.71399999999999997},"min":{"1min":0.434,"5min":0.42999999999999999,"15min":0.41799999999999998},"max":{"1min":0.80900000000000005,"5min":5.0999999999999996,"15min":5.0999999999999996},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.69299999999999995,"15min":0.72699999999999998},"min":{"1min":0.47399999999999998,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.89100000000000001,"5min":1.0309999999999999,"15min":1.2290000000000001},"last":0.80600000000000005}]},{"osd":7,"last update":"Mon Oct 16 13:47:42 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.69599999999999995,"15min":0.72899999999999998},"min":{"1min":0.497,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.82799999999999996,"5min":1.048,"15min":1.2569999999999999},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.62,"5min":0.70299999999999996,"15min":0.73599999999999999},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.43099999999999999},"max":{"1min":0.90400000000000003,"5min":1.262,"15min":1.262},"last":0.70499999999999996}]}]},{"osd":4,"up_from":13,"seq":55834576004,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52520,"kb_used_data":3288,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319320,"statfs":{"total":96636764160,"available":96582983680,"internally_reserved":0,"allocated":3366912,"data_stored":1742021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:47:41 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.57999999999999996,"15min":0.623},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.81599999999999995,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.60099999999999998,"15min":0.63},"min":{"1min":0.39300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.77300000000000002,"5min":0.878,"15min":0.89500000000000002},"last":0.55400000000000005}]},{"osd":1,"last update":"Mon Oct 16 13:47:41 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.60799999999999998,"15min":0.65200000000000002},"min":{"1min":0.33700000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.78900000000000003,"5min":0.877,"15min":0.877},"last":0.501},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.60899999999999999,"15min":0.63700000000000001},"min":{"1min":0.42199999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.80800000000000005,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.33300000000000002}]},{"osd":2,"last update":"Mon Oct 16 13:47:41 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.623,"15min":0.64900000000000002},"min":{"1min":0.39000000000000001,"5min":0.378,"15min":0.378},"max":{"1min":0.69399999999999995,"5min":0.89900000000000002,"15min":0.90600000000000003},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.64200000000000002,"15min":0.65900000000000003},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.77500000000000002,"5min":0.89700000000000002,"15min":0.92600000000000005},"last":0.53500000000000003}]},{"osd":3,"last update":"Mon Oct 16 13:47:41 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.61699999999999999,"15min":0.64300000000000002},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.76000000000000001,"5min":0.86599999999999999,"15min":0.99199999999999999},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.65300000000000002,"15min":0.68000000000000005},"min":{"1min":0.45600000000000002,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.93899999999999995},"last":0.59399999999999997}]},{"osd":5,"last update":"Mon Oct 16 13:47:41 2023","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.54400000000000004,"15min":0.53300000000000003},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.73799999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.58099999999999996,"15min":0.56799999999999995},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.753,"5min":0.82199999999999995,"15min":0.94899999999999995},"last":0.66400000000000003}]},{"osd":6,"last update":"Mon Oct 16 13:47:41 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.627,"15min":0.60099999999999998},"min":{"1min":0.378,"5min":0.36399999999999999,"15min":0.33200000000000002},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.70999999999999996,"15min":0.69499999999999995},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.39400000000000002},"max":{"1min":0.84299999999999997,"5min":1.129,"15min":1.129},"last":0.63600000000000001}]},{"osd":7,"last update":"Mon Oct 16 13:47:41 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.65800000000000003,"15min":0.63500000000000001},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.38900000000000001},"max":{"1min":0.754,"5min":0.97499999999999998,"15min":0.98599999999999999},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.71299999999999997,"15min":0.68700000000000006},"min":{"1min":0.48699999999999999,"5min":0.44,"15min":0.38200000000000001},"max":{"1min":0.84699999999999998,"5min":0.97599999999999998,"15min":1.026},"last":0.69299999999999995}]}]},{"osd":5,"up_from":13,"seq":55834576005,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74720,"kb_used_data":3280,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94297120,"statfs":{"total":96636764160,"available":96560250880,"internally_reserved":0,"allocated":3358720,"data_stored":1740375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:47:28 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.59599999999999997,"15min":0.61399999999999999},"min":{"1min":0.379,"5min":0.314,"15min":0.26800000000000002},"max":{"1min":0.79200000000000004,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.61799999999999999,"15min":0.63700000000000001},"min":{"1min":0.38600000000000001,"5min":0.35899999999999999,"15min":0.29699999999999999},"max":{"1min":0.78600000000000003,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.57699999999999996}]},{"osd":1,"last update":"Mon Oct 16 13:47:28 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.64700000000000002,"15min":0.65900000000000003},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.29499999999999998},"max":{"1min":0.82799999999999996,"5min":1,"15min":1},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.61299999999999999,"15min":0.63600000000000001},"min":{"1min":0.378,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.82099999999999995,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.54200000000000004}]},{"osd":2,"last update":"Mon Oct 16 13:47:28 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.65400000000000003,"15min":0.65500000000000003},"min":{"1min":0.48699999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.80500000000000005,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.63400000000000001,"15min":0.64700000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.80200000000000005,"5min":0.871,"15min":0.93500000000000005},"last":0.55900000000000005}]},{"osd":3,"last update":"Mon Oct 16 13:47:28 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.66200000000000003,"15min":0.66500000000000004},"min":{"1min":0.42199999999999999,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.85799999999999998,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.68999999999999995,"15min":0.69399999999999995},"min":{"1min":0.45500000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.82799999999999996,"5min":1.4199999999999999,"15min":1.4199999999999999},"last":0.48799999999999999}]},{"osd":4,"last update":"Mon Oct 16 13:47:28 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.54800000000000004,"15min":0.53800000000000003},"min":{"1min":0.39500000000000002,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.89700000000000002,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.60799999999999998,"15min":0.59499999999999997},"min":{"1min":0.40999999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.68899999999999995}]},{"osd":6,"last update":"Mon Oct 16 13:47:28 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.60099999999999998,"15min":0.58799999999999997},"min":{"1min":0.443,"5min":0.315,"15min":0.315},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.93200000000000005},"last":0.628},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.629,"15min":0.625},"min":{"1min":0.501,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.52800000000000002}]},{"osd":7,"last update":"Mon Oct 16 13:47:28 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.65500000000000003,"15min":0.63900000000000001},"min":{"1min":0.47599999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.93899999999999995,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.70699999999999996,"15min":0.68899999999999995},"min":{"1min":0.53400000000000003,"5min":0.375,"15min":0.375},"max":{"1min":0.97299999999999998,"5min":0.98299999999999998,"15min":1.0229999999999999},"last":0.60199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T13:48:30.000 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-16T13:48:30.001 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-16T13:48:30.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.165+0000 7fc95d575700 1 Processor -- start 2023-10-16T13:48:30.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.165+0000 7fc95d575700 1 -- start start 2023-10-16T13:48:30.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.166+0000 7fc95d575700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc9580cea60 0x7fc9580b2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:30.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.166+0000 7fc95d575700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9580b2f30 con 0x7fc9580b8fa0 2023-10-16T13:48:30.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.166+0000 7fc95d575700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9580d1c40 con 0x7fc95815d8d0 2023-10-16T13:48:30.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.166+0000 7fc95d575700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9580d0bb0 con 0x7fc9580cea60 2023-10-16T13:48:30.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.166+0000 7fc9577fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc9580cea60 0x7fc9580b2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:30.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.166+0000 7fc9577fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc9580cea60 0x7fc9580b2830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57606/0 (socket says 172.21.15.16:57606) 2023-10-16T13:48:30.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.166+0000 7fc9577fe700 1 -- 172.21.15.16:0/2768831010 learned_addr learned my addr 172.21.15.16:0/2768831010 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:30.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.166+0000 7fc9567fc700 1 --1- 172.21.15.16:0/2768831010 >> v1:172.21.15.16:6789/0 conn(0x7fc9580b8fa0 0x7fc9580ce350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46878/0 (socket says 172.21.15.16:46878) 2023-10-16T13:48:30.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.166+0000 7fc9577fe700 1 -- 172.21.15.16:0/2768831010 >> v1:172.21.15.49:6789/0 conn(0x7fc95815d8d0 legacy=0x7fc9580b6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:30.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.166+0000 7fc9577fe700 1 -- 172.21.15.16:0/2768831010 >> v1:172.21.15.16:6789/0 conn(0x7fc9580b8fa0 legacy=0x7fc9580ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:30.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.166+0000 7fc9577fe700 1 -- 172.21.15.16:0/2768831010 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9580d1ee0 con 0x7fc9580cea60 2023-10-16T13:48:30.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.166+0000 7fc9577fe700 1 --2- 172.21.15.16:0/2768831010 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9580cea60 0x7fc9580b2830 secure :-1 s=READY pgs=3018 cs=0 l=1 rev1=1 crypto rx=0x7fc950009e50 tx=0x7fc95000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d63f6bf9e2999085 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:30.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.167+0000 7fc955ffb700 1 -- 172.21.15.16:0/2768831010 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc95000e070 con 0x7fc9580cea60 2023-10-16T13:48:30.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.167+0000 7fc955ffb700 1 -- 172.21.15.16:0/2768831010 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc95000b9c0 con 0x7fc9580cea60 2023-10-16T13:48:30.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.167+0000 7fc955ffb700 1 -- 172.21.15.16:0/2768831010 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc95000bd70 con 0x7fc9580cea60 2023-10-16T13:48:30.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.167+0000 7fc95d575700 1 -- 172.21.15.16:0/2768831010 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9580cea60 msgr2=0x7fc9580b2830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:30.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.167+0000 7fc95d575700 1 --2- 172.21.15.16:0/2768831010 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9580cea60 0x7fc9580b2830 secure :-1 s=READY pgs=3018 cs=0 l=1 rev1=1 crypto rx=0x7fc950009e50 tx=0x7fc95000b330 comp rx=0 tx=0).stop 2023-10-16T13:48:30.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.167+0000 7fc95d575700 1 -- 172.21.15.16:0/2768831010 shutdown_connections 2023-10-16T13:48:30.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.167+0000 7fc95d575700 1 --2- 172.21.15.16:0/2768831010 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9580cea60 0x7fc9580b2830 unknown :-1 s=CLOSED pgs=3018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:30.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.167+0000 7fc95d575700 1 -- 172.21.15.16:0/2768831010 >> 172.21.15.16:0/2768831010 conn(0x7fc958072330 msgr2=0x7fc958072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:30.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.167+0000 7fc95d575700 1 -- 172.21.15.16:0/2768831010 shutdown_connections 2023-10-16T13:48:30.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.167+0000 7fc95d575700 1 -- 172.21.15.16:0/2768831010 wait complete. 2023-10-16T13:48:30.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.169+0000 7fc95d575700 1 Processor -- start 2023-10-16T13:48:30.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.169+0000 7fc95d575700 1 -- start start 2023-10-16T13:48:30.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.169+0000 7fc95d575700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc9580b8fa0 0x7fc9580b1430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:30.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.169+0000 7fc95d575700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9580cea60 0x7fc9580a9fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:30.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.169+0000 7fc95d575700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc95815d8d0 0x7fc9580aa510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:30.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.169+0000 7fc95d575700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9581f4330 con 0x7fc9580cea60 2023-10-16T13:48:30.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.169+0000 7fc95d575700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9580b1970 con 0x7fc95815d8d0 2023-10-16T13:48:30.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.169+0000 7fc95d575700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc95815fc30 con 0x7fc9580b8fa0 2023-10-16T13:48:30.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.169+0000 7fc9567fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9580cea60 0x7fc9580a9fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:30.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.170+0000 7fc9567fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9580cea60 0x7fc9580a9fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57608/0 (socket says 172.21.15.16:57608) 2023-10-16T13:48:30.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.170+0000 7fc9567fc700 1 -- 172.21.15.16:0/2416803899 learned_addr learned my addr 172.21.15.16:0/2416803899 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:30.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.170+0000 7fc9567fc700 1 -- 172.21.15.16:0/2416803899 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc9580b8fa0 msgr2=0x7fc9580b1430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:30.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.170+0000 7fc9577fe700 1 --2- 172.21.15.16:0/2416803899 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc95815d8d0 0x7fc9580aa510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:30.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.170+0000 7fc9567fc700 1 --2- 172.21.15.16:0/2416803899 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc9580b8fa0 0x7fc9580b1430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:30.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.170+0000 7fc9567fc700 1 -- 172.21.15.16:0/2416803899 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc95815d8d0 msgr2=0x7fc9580aa510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:30.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.170+0000 7fc9567fc700 1 --2- 172.21.15.16:0/2416803899 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc95815d8d0 0x7fc9580aa510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:30.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.170+0000 7fc9567fc700 1 -- 172.21.15.16:0/2416803899 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc950009b30 con 0x7fc9580cea60 2023-10-16T13:48:30.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.170+0000 7fc9567fc700 1 --2- 172.21.15.16:0/2416803899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9580cea60 0x7fc9580a9fd0 secure :-1 s=READY pgs=3019 cs=0 l=1 rev1=1 crypto rx=0x7fc94800c500 tx=0x7fc94800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:30.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.170+0000 7fc93f7fe700 1 -- 172.21.15.16:0/2416803899 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc948005cb0 con 0x7fc9580cea60 2023-10-16T13:48:30.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.170+0000 7fc93f7fe700 1 -- 172.21.15.16:0/2416803899 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc948005e10 con 0x7fc9580cea60 2023-10-16T13:48:30.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.170+0000 7fc95d575700 1 -- 172.21.15.16:0/2416803899 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc95815ff10 con 0x7fc9580cea60 2023-10-16T13:48:30.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.170+0000 7fc93f7fe700 1 -- 172.21.15.16:0/2416803899 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9480072d0 con 0x7fc9580cea60 2023-10-16T13:48:30.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.170+0000 7fc95d575700 1 -- 172.21.15.16:0/2416803899 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9581604d0 con 0x7fc9580cea60 2023-10-16T13:48:30.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.171+0000 7fc95d575700 1 -- 172.21.15.16:0/2416803899 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9580d4010 con 0x7fc9580cea60 2023-10-16T13:48:30.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.172+0000 7fc93f7fe700 1 -- 172.21.15.16:0/2416803899 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fc948007430 con 0x7fc9580cea60 2023-10-16T13:48:30.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.172+0000 7fc93f7fe700 1 --2- 172.21.15.16:0/2416803899 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fc94406b110 0x7fc94406d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:30.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.172+0000 7fc93f7fe700 1 -- 172.21.15.16:0/2416803899 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 6157+0+0 (secure 0 0 0) 0x7fc948088f20 con 0x7fc9580cea60 2023-10-16T13:48:30.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.172+0000 7fc956ffd700 1 --2- 172.21.15.16:0/2416803899 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fc94406b110 0x7fc94406d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:30.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.173+0000 7fc956ffd700 1 --2- 172.21.15.16:0/2416803899 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fc94406b110 0x7fc94406d5d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc94c000f60 tx=0x7fc94c007a60 comp rx=0 tx=0).ready entity=mgr.12678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:30.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.177+0000 7fc93f7fe700 1 -- 172.21.15.16:0/2416803899 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc948053b70 con 0x7fc9580cea60 2023-10-16T13:48:30.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.467+0000 7fc95d575700 1 -- 172.21.15.16:0/2416803899 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fc95804ebb0 con 0x7fc9580cea60 2023-10-16T13:48:30.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.469+0000 7fc93f7fe700 1 -- 172.21.15.16:0/2416803899 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v58) v1 ==== 190+0+147 (secure 0 0 0) 0x7fc948057190 con 0x7fc9580cea60 2023-10-16T13:48:30.471 INFO:teuthology.orchestra.run.smithi016.stdout:[client.0] 2023-10-16T13:48:30.472 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAPKC1l2l6PFxAA9QyXtBAHdGAXpwH0JfknuQ== 2023-10-16T13:48:30.472 INFO:teuthology.orchestra.run.smithi016.stdout: caps mds = "allow" 2023-10-16T13:48:30.472 INFO:teuthology.orchestra.run.smithi016.stdout: caps mon = "allow r" 2023-10-16T13:48:30.472 INFO:teuthology.orchestra.run.smithi016.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-16T13:48:30.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.471+0000 7fc95d575700 1 -- 172.21.15.16:0/2416803899 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fc94406b110 msgr2=0x7fc94406d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:30.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.471+0000 7fc95d575700 1 --2- 172.21.15.16:0/2416803899 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fc94406b110 0x7fc94406d5d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc94c000f60 tx=0x7fc94c007a60 comp rx=0 tx=0).stop 2023-10-16T13:48:30.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.471+0000 7fc95d575700 1 -- 172.21.15.16:0/2416803899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9580cea60 msgr2=0x7fc9580a9fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:30.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.471+0000 7fc95d575700 1 --2- 172.21.15.16:0/2416803899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9580cea60 0x7fc9580a9fd0 secure :-1 s=READY pgs=3019 cs=0 l=1 rev1=1 crypto rx=0x7fc94800c500 tx=0x7fc94800c8c0 comp rx=0 tx=0).stop 2023-10-16T13:48:30.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.471+0000 7fc95d575700 1 -- 172.21.15.16:0/2416803899 shutdown_connections 2023-10-16T13:48:30.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.471+0000 7fc95d575700 1 --2- 172.21.15.16:0/2416803899 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fc94406b110 0x7fc94406d5d0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:30.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.471+0000 7fc95d575700 1 --2- 172.21.15.16:0/2416803899 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc95815d8d0 0x7fc9580aa510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:30.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.471+0000 7fc95d575700 1 --2- 172.21.15.16:0/2416803899 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc9580b8fa0 0x7fc9580b1430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:30.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.471+0000 7fc95d575700 1 --2- 172.21.15.16:0/2416803899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9580cea60 0x7fc9580a9fd0 unknown :-1 s=CLOSED pgs=3019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:30.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.471+0000 7fc95d575700 1 -- 172.21.15.16:0/2416803899 >> 172.21.15.16:0/2416803899 conn(0x7fc958072330 msgr2=0x7fc9580b36c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:30.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.471+0000 7fc95d575700 1 -- 172.21.15.16:0/2416803899 shutdown_connections 2023-10-16T13:48:30.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.471+0000 7fc95d575700 1 -- 172.21.15.16:0/2416803899 wait complete. 2023-10-16T13:48:30.476 INFO:teuthology.orchestra.run.smithi016.stderr:updated caps for client.0 2023-10-16T13:48:30.486 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:48:30.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.609+0000 7fbdb61da700 1 Processor -- start 2023-10-16T13:48:30.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.609+0000 7fbdb61da700 1 -- start start 2023-10-16T13:48:30.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.610+0000 7fbdb61da700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fbdb015b280 0x7fbdb01613e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:30.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.610+0000 7fbdb61da700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbdb00cc8a0 0x7fbdb015e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:30.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.610+0000 7fbdb61da700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbdb006a7e0 0x7fbdb006ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:30.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.610+0000 7fbdb61da700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdb006b190 con 0x7fbdb00cc8a0 2023-10-16T13:48:30.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.610+0000 7fbdb61da700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdb0161c30 con 0x7fbdb015b280 2023-10-16T13:48:30.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.610+0000 7fbdb61da700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb0161f10 con 0x7fbdb006a7e0 2023-10-16T13:48:30.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.610+0000 7fbdaeffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbdb00cc8a0 0x7fbdb015e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:30.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.610+0000 7fbdaeffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbdb00cc8a0 0x7fbdb015e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57616/0 (socket says 172.21.15.16:57616) 2023-10-16T13:48:30.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.610+0000 7fbdaeffd700 1 -- 172.21.15.16:0/241700496 learned_addr learned my addr 172.21.15.16:0/241700496 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:30.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.610+0000 7fbdaffff700 1 --2- 172.21.15.16:0/241700496 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbdb006a7e0 0x7fbdb006ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:30.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.610+0000 7fbdaf7fe700 1 --2- 172.21.15.16:0/241700496 >> v2:172.21.15.49:3300/0 conn(0x7fbdb015b280 0x7fbdb01613e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:30.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.611+0000 7fbdaeffd700 1 -- 172.21.15.16:0/241700496 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbdb006a7e0 msgr2=0x7fbdb006ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:30.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.611+0000 7fbdaeffd700 1 --2- 172.21.15.16:0/241700496 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbdb006a7e0 0x7fbdb006ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:30.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.611+0000 7fbdaeffd700 1 -- 172.21.15.16:0/241700496 >> v2:172.21.15.49:3300/0 conn(0x7fbdb015b280 msgr2=0x7fbdb01613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:30.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.611+0000 7fbdaeffd700 1 --2- 172.21.15.16:0/241700496 >> v2:172.21.15.49:3300/0 conn(0x7fbdb015b280 0x7fbdb01613e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:30.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.611+0000 7fbdaeffd700 1 -- 172.21.15.16:0/241700496 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdb01621f0 con 0x7fbdb00cc8a0 2023-10-16T13:48:30.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.611+0000 7fbdaf7fe700 1 --2- 172.21.15.16:0/241700496 >> v2:172.21.15.49:3300/0 conn(0x7fbdb015b280 0x7fbdb01613e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:48:30.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.611+0000 7fbdaeffd700 1 --2- 172.21.15.16:0/241700496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbdb00cc8a0 0x7fbdb015e700 secure :-1 s=READY pgs=3020 cs=0 l=1 rev1=1 crypto rx=0x7fbdb006b8a0 tx=0x7fbda0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a078cbac3347689 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:30.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.611+0000 7fbdae7fc700 1 -- 172.21.15.16:0/241700496 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbda000b870 con 0x7fbdb00cc8a0 2023-10-16T13:48:30.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.611+0000 7fbdae7fc700 1 -- 172.21.15.16:0/241700496 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbda000b9d0 con 0x7fbdb00cc8a0 2023-10-16T13:48:30.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.611+0000 7fbdae7fc700 1 -- 172.21.15.16:0/241700496 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbda000bd60 con 0x7fbdb00cc8a0 2023-10-16T13:48:30.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.612+0000 7fbdb61da700 1 -- 172.21.15.16:0/241700496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbdb00cc8a0 msgr2=0x7fbdb015e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:30.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.612+0000 7fbdb61da700 1 --2- 172.21.15.16:0/241700496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbdb00cc8a0 0x7fbdb015e700 secure :-1 s=READY pgs=3020 cs=0 l=1 rev1=1 crypto rx=0x7fbdb006b8a0 tx=0x7fbda0009bd0 comp rx=0 tx=0).stop 2023-10-16T13:48:30.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.612+0000 7fbdb61da700 1 -- 172.21.15.16:0/241700496 shutdown_connections 2023-10-16T13:48:30.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.612+0000 7fbdb61da700 1 --2- 172.21.15.16:0/241700496 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbdb006a7e0 0x7fbdb006ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:30.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.612+0000 7fbdb61da700 1 --2- 172.21.15.16:0/241700496 >> v2:172.21.15.49:3300/0 conn(0x7fbdb015b280 0x7fbdb01613e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:30.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.612+0000 7fbdb61da700 1 --2- 172.21.15.16:0/241700496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbdb00cc8a0 0x7fbdb015e700 unknown :-1 s=CLOSED pgs=3020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:30.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.612+0000 7fbdb61da700 1 -- 172.21.15.16:0/241700496 >> 172.21.15.16:0/241700496 conn(0x7fbdb00b1fd0 msgr2=0x7fbdb00b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:30.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.612+0000 7fbdb61da700 1 -- 172.21.15.16:0/241700496 shutdown_connections 2023-10-16T13:48:30.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.612+0000 7fbdb61da700 1 -- 172.21.15.16:0/241700496 wait complete. 2023-10-16T13:48:30.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.613+0000 7fbdb61da700 1 Processor -- start 2023-10-16T13:48:30.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.613+0000 7fbdb61da700 1 -- start start 2023-10-16T13:48:30.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.614+0000 7fbdb61da700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbdb006a7e0 0x7fbdb00c9eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:30.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.614+0000 7fbdb61da700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbdb00cc8a0 0x7fbdb00c9710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:30.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.614+0000 7fbdb61da700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbdb015b280 0x7fbdb015f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:30.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.614+0000 7fbdb61da700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb00c7510 con 0x7fbdb00cc8a0 2023-10-16T13:48:30.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.614+0000 7fbdb61da700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb0160c70 con 0x7fbdb015b280 2023-10-16T13:48:30.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.614+0000 7fbdb61da700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb00c41f0 con 0x7fbdb006a7e0 2023-10-16T13:48:30.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.614+0000 7fbdaeffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbdb00cc8a0 0x7fbdb00c9710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:30.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.614+0000 7fbdaeffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbdb00cc8a0 0x7fbdb00c9710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57624/0 (socket says 172.21.15.16:57624) 2023-10-16T13:48:30.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.614+0000 7fbdaeffd700 1 -- 172.21.15.16:0/3385088854 learned_addr learned my addr 172.21.15.16:0/3385088854 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:30.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.614+0000 7fbdaffff700 1 --2- 172.21.15.16:0/3385088854 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbdb015b280 0x7fbdb015f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:30.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.614+0000 7fbdaf7fe700 1 --2- 172.21.15.16:0/3385088854 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbdb006a7e0 0x7fbdb00c9eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:30.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.614+0000 7fbdaeffd700 1 -- 172.21.15.16:0/3385088854 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbdb006a7e0 msgr2=0x7fbdb00c9eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:30.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.614+0000 7fbdaeffd700 1 --2- 172.21.15.16:0/3385088854 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbdb006a7e0 0x7fbdb00c9eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:30.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.614+0000 7fbdaeffd700 1 -- 172.21.15.16:0/3385088854 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbdb015b280 msgr2=0x7fbdb015f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:30.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.614+0000 7fbdaeffd700 1 --2- 172.21.15.16:0/3385088854 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbdb015b280 0x7fbdb015f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:30.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.614+0000 7fbdaeffd700 1 -- 172.21.15.16:0/3385088854 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbda0009550 con 0x7fbdb00cc8a0 2023-10-16T13:48:30.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.614+0000 7fbdaf7fe700 1 --2- 172.21.15.16:0/3385088854 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbdb006a7e0 0x7fbdb00c9eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:48:30.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.614+0000 7fbdaeffd700 1 --2- 172.21.15.16:0/3385088854 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbdb00cc8a0 0x7fbdb00c9710 secure :-1 s=READY pgs=3021 cs=0 l=1 rev1=1 crypto rx=0x7fbda0000c00 tx=0x7fbda0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:30.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.614+0000 7fbda7fff700 1 -- 172.21.15.16:0/3385088854 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbda0004540 con 0x7fbdb00cc8a0 2023-10-16T13:48:30.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.615+0000 7fbda7fff700 1 -- 172.21.15.16:0/3385088854 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbda00036a0 con 0x7fbdb00cc8a0 2023-10-16T13:48:30.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.615+0000 7fbdb61da700 1 -- 172.21.15.16:0/3385088854 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdb00c44d0 con 0x7fbdb00cc8a0 2023-10-16T13:48:30.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.615+0000 7fbda7fff700 1 -- 172.21.15.16:0/3385088854 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbda0003a50 con 0x7fbdb00cc8a0 2023-10-16T13:48:30.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.615+0000 7fbdb61da700 1 -- 172.21.15.16:0/3385088854 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdb00c7760 con 0x7fbdb00cc8a0 2023-10-16T13:48:30.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.615+0000 7fbdb61da700 1 -- 172.21.15.16:0/3385088854 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdb00ce740 con 0x7fbdb00cc8a0 2023-10-16T13:48:30.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.616+0000 7fbda7fff700 1 -- 172.21.15.16:0/3385088854 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fbda0021070 con 0x7fbdb00cc8a0 2023-10-16T13:48:30.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.616+0000 7fbda7fff700 1 --2- 172.21.15.16:0/3385088854 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fbd9006b140 0x7fbd9006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:30.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.616+0000 7fbda7fff700 1 -- 172.21.15.16:0/3385088854 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 6157+0+0 (secure 0 0 0) 0x7fbda008b1c0 con 0x7fbdb00cc8a0 2023-10-16T13:48:30.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.616+0000 7fbdaf7fe700 1 --2- 172.21.15.16:0/3385088854 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fbd9006b140 0x7fbd9006d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:30.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.617+0000 7fbdaf7fe700 1 --2- 172.21.15.16:0/3385088854 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fbd9006b140 0x7fbd9006d600 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fbdb00b0da0 tx=0x7fbd98008040 comp rx=0 tx=0).ready entity=mgr.12678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:30.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.621+0000 7fbda7fff700 1 -- 172.21.15.16:0/3385088854 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbda0055de0 con 0x7fbdb00cc8a0 2023-10-16T13:48:30.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.868+0000 7fbdb61da700 1 -- 172.21.15.16:0/3385088854 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbdb004ebb0 con 0x7fbdb00cc8a0 2023-10-16T13:48:30.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.869+0000 7fbda7fff700 1 -- 172.21.15.16:0/3385088854 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 146 v146) v1 ==== 95+0+4926 (secure 0 0 0) 0x7fbda0059400 con 0x7fbdb00cc8a0 2023-10-16T13:48:30.871 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:48:30.872 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":146,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10112,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/712583564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":712583564},{"type":"v1","addr":"172.21.15.16:6837","nonce":712583564}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":141},{"gid":11836,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/3551114031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":3551114031},{"type":"v1","addr":"172.21.15.49:6834","nonce":3551114031}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":141},{"gid":13050,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/2135643757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2135643757},{"type":"v1","addr":"172.21.15.16:6836","nonce":2135643757}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":140}],"filesystems":[{"mdsmap":{"epoch":146,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:48:24.832614+0000","modified":"2023-10-16T13:48:27.858697+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":13068},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13068":{"gid":13068,"name":"d","rank":0,"incarnation":142,"state":"up:active","state_seq":3,"addr":"172.21.15.49:6835/1985453659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":1985453659},{"type":"v1","addr":"172.21.15.49:6835","nonce":1985453659}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[36],"metadata_pool":35,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":17}]} 2023-10-16T13:48:30.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.871+0000 7fbdb61da700 1 -- 172.21.15.16:0/3385088854 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fbd9006b140 msgr2=0x7fbd9006d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:30.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.871+0000 7fbdb61da700 1 --2- 172.21.15.16:0/3385088854 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fbd9006b140 0x7fbd9006d600 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fbdb00b0da0 tx=0x7fbd98008040 comp rx=0 tx=0).stop 2023-10-16T13:48:30.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.871+0000 7fbdb61da700 1 -- 172.21.15.16:0/3385088854 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbdb00cc8a0 msgr2=0x7fbdb00c9710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:30.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.871+0000 7fbdb61da700 1 --2- 172.21.15.16:0/3385088854 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbdb00cc8a0 0x7fbdb00c9710 secure :-1 s=READY pgs=3021 cs=0 l=1 rev1=1 crypto rx=0x7fbda0000c00 tx=0x7fbda0004750 comp rx=0 tx=0).stop 2023-10-16T13:48:30.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.871+0000 7fbdb61da700 1 -- 172.21.15.16:0/3385088854 shutdown_connections 2023-10-16T13:48:30.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.871+0000 7fbdb61da700 1 --2- 172.21.15.16:0/3385088854 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fbd9006b140 0x7fbd9006d600 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:30.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.871+0000 7fbdb61da700 1 --2- 172.21.15.16:0/3385088854 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbdb015b280 0x7fbdb015f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:30.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.871+0000 7fbdb61da700 1 --2- 172.21.15.16:0/3385088854 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbdb006a7e0 0x7fbdb00c9eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:30.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.871+0000 7fbdb61da700 1 --2- 172.21.15.16:0/3385088854 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbdb00cc8a0 0x7fbdb00c9710 unknown :-1 s=CLOSED pgs=3021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:30.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.871+0000 7fbdb61da700 1 -- 172.21.15.16:0/3385088854 >> 172.21.15.16:0/3385088854 conn(0x7fbdb00b1fd0 msgr2=0x7fbdb00cb690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:30.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.871+0000 7fbdb61da700 1 -- 172.21.15.16:0/3385088854 shutdown_connections 2023-10-16T13:48:30.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:30.871+0000 7fbdb61da700 1 -- 172.21.15.16:0/3385088854 wait complete. 2023-10-16T13:48:30.876 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 146 2023-10-16T13:48:31.289 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 146, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T13:48:24.832614+0000', 'modified': '2023-10-16T13:48:27.858697+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 13068}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_13068': {'gid': 13068, 'name': 'd', 'rank': 0, 'incarnation': 142, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.49:6835/1985453659', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6833', 'nonce': 1985453659}, {'type': 'v1', 'addr': '172.21.15.49:6835', 'nonce': 1985453659}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [36], 'metadata_pool': 35, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-16T13:48:31.289 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-16T13:48:31.290 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2023-10-16T13:48:31.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.421+0000 7f646017b700 1 Processor -- start 2023-10-16T13:48:31.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.422+0000 7f646017b700 1 -- start start 2023-10-16T13:48:31.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.422+0000 7f646017b700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f645815d8d0 0x7f64580d1c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:31.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.422+0000 7f646017b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f64580d0750 0x7f645806c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:31.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.422+0000 7f646017b700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f645806ba00 con 0x7f64580d41c0 2023-10-16T13:48:31.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.422+0000 7f646017b700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64580d2150 con 0x7f64580d0750 2023-10-16T13:48:31.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.422+0000 7f646017b700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64580d26d0 con 0x7f645815d8d0 2023-10-16T13:48:31.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.423+0000 7f645e718700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f64580d0750 0x7f645806c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:31.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.423+0000 7f645e718700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f64580d0750 0x7f645806c4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57634/0 (socket says 172.21.15.16:57634) 2023-10-16T13:48:31.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.423+0000 7f645e718700 1 -- 172.21.15.16:0/46297602 learned_addr learned my addr 172.21.15.16:0/46297602 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:31.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.423+0000 7f645df17700 1 --2- 172.21.15.16:0/46297602 >> v2:172.21.15.49:3300/0 conn(0x7f645815d8d0 0x7f64580d1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:31.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.423+0000 7f645e718700 1 -- 172.21.15.16:0/46297602 >> v2:172.21.15.49:3300/0 conn(0x7f645815d8d0 msgr2=0x7f64580d1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:31.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.423+0000 7f645e718700 1 --2- 172.21.15.16:0/46297602 >> v2:172.21.15.49:3300/0 conn(0x7f645815d8d0 0x7f64580d1c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:31.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.423+0000 7f645e718700 1 -- 172.21.15.16:0/46297602 >> v1:172.21.15.16:6789/0 conn(0x7f64580d41c0 legacy=0x7f64580d0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:31.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.423+0000 7f645e718700 1 -- 172.21.15.16:0/46297602 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64580d23f0 con 0x7f64580d0750 2023-10-16T13:48:31.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.423+0000 7f645e718700 1 --2- 172.21.15.16:0/46297602 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64580d0750 0x7f645806c4a0 secure :-1 s=READY pgs=3023 cs=0 l=1 rev1=1 crypto rx=0x7f645808b460 tx=0x7f645000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f947fe15e5e2033 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:31.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.423+0000 7f645cf15700 1 -- 172.21.15.16:0/46297602 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f645000e070 con 0x7f64580d0750 2023-10-16T13:48:31.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.423+0000 7f645cf15700 1 -- 172.21.15.16:0/46297602 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f645000b9c0 con 0x7f64580d0750 2023-10-16T13:48:31.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.423+0000 7f645cf15700 1 -- 172.21.15.16:0/46297602 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f645000bd70 con 0x7f64580d0750 2023-10-16T13:48:31.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.424+0000 7f646017b700 1 -- 172.21.15.16:0/46297602 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64580d0750 msgr2=0x7f645806c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:31.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.424+0000 7f646017b700 1 --2- 172.21.15.16:0/46297602 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64580d0750 0x7f645806c4a0 secure :-1 s=READY pgs=3023 cs=0 l=1 rev1=1 crypto rx=0x7f645808b460 tx=0x7f645000b330 comp rx=0 tx=0).stop 2023-10-16T13:48:31.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.424+0000 7f646017b700 1 -- 172.21.15.16:0/46297602 shutdown_connections 2023-10-16T13:48:31.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.424+0000 7f646017b700 1 --2- 172.21.15.16:0/46297602 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64580d0750 0x7f645806c4a0 unknown :-1 s=CLOSED pgs=3023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:31.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.424+0000 7f646017b700 1 --2- 172.21.15.16:0/46297602 >> v2:172.21.15.49:3300/0 conn(0x7f645815d8d0 0x7f64580d1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:31.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.424+0000 7f646017b700 1 -- 172.21.15.16:0/46297602 >> 172.21.15.16:0/46297602 conn(0x7f64580d3630 msgr2=0x7f64580d3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:31.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.424+0000 7f646017b700 1 -- 172.21.15.16:0/46297602 shutdown_connections 2023-10-16T13:48:31.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.424+0000 7f646017b700 1 -- 172.21.15.16:0/46297602 wait complete. 2023-10-16T13:48:31.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.425+0000 7f646017b700 1 Processor -- start 2023-10-16T13:48:31.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.426+0000 7f646017b700 1 -- start start 2023-10-16T13:48:31.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.426+0000 7f646017b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f64580d0750 0x7f6458157f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:31.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.426+0000 7f646017b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f64580d41c0 0x7f64581e7100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:31.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.426+0000 7f646017b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f645815d8d0 0x7f64581e7640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:31.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.426+0000 7f646017b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64580c8c70 con 0x7f645815d8d0 2023-10-16T13:48:31.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.426+0000 7f646017b700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6458158470 con 0x7f64580d0750 2023-10-16T13:48:31.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.426+0000 7f646017b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f645815ef60 con 0x7f64580d41c0 2023-10-16T13:48:31.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.427+0000 7f645e718700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f645815d8d0 0x7f64581e7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:31.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.427+0000 7f645e718700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f645815d8d0 0x7f64581e7640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57640/0 (socket says 172.21.15.16:57640) 2023-10-16T13:48:31.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.427+0000 7f645e718700 1 -- 172.21.15.16:0/4219497798 learned_addr learned my addr 172.21.15.16:0/4219497798 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:31.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.427+0000 7f645df17700 1 --2- 172.21.15.16:0/4219497798 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f64580d0750 0x7f6458157f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:31.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.427+0000 7f645d716700 1 --2- 172.21.15.16:0/4219497798 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f64580d41c0 0x7f64581e7100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:31.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.427+0000 7f645e718700 1 -- 172.21.15.16:0/4219497798 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f64580d41c0 msgr2=0x7f64581e7100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:31.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.427+0000 7f645e718700 1 --2- 172.21.15.16:0/4219497798 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f64580d41c0 0x7f64581e7100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:31.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.427+0000 7f645e718700 1 -- 172.21.15.16:0/4219497798 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f64580d0750 msgr2=0x7f6458157f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:31.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.427+0000 7f645e718700 1 --2- 172.21.15.16:0/4219497798 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f64580d0750 0x7f6458157f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:31.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.427+0000 7f645e718700 1 -- 172.21.15.16:0/4219497798 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6450009b30 con 0x7f645815d8d0 2023-10-16T13:48:31.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.427+0000 7f645d716700 1 --2- 172.21.15.16:0/4219497798 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f64580d41c0 0x7f64581e7100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:48:31.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.427+0000 7f645df17700 1 --2- 172.21.15.16:0/4219497798 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f64580d0750 0x7f6458157f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:48:31.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.427+0000 7f645e718700 1 --2- 172.21.15.16:0/4219497798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f645815d8d0 0x7f64581e7640 secure :-1 s=READY pgs=3024 cs=0 l=1 rev1=1 crypto rx=0x7f6450012000 tx=0x7f6450004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:31.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.427+0000 7f644e7fc700 1 -- 172.21.15.16:0/4219497798 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64500099b0 con 0x7f645815d8d0 2023-10-16T13:48:31.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.427+0000 7f644e7fc700 1 -- 172.21.15.16:0/4219497798 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64500092e0 con 0x7f645815d8d0 2023-10-16T13:48:31.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.427+0000 7f646017b700 1 -- 172.21.15.16:0/4219497798 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f645815f240 con 0x7f645815d8d0 2023-10-16T13:48:31.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.427+0000 7f644e7fc700 1 -- 172.21.15.16:0/4219497798 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f645001c640 con 0x7f645815d8d0 2023-10-16T13:48:31.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.427+0000 7f646017b700 1 -- 172.21.15.16:0/4219497798 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f645815f800 con 0x7f645815d8d0 2023-10-16T13:48:31.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.429+0000 7f644e7fc700 1 -- 172.21.15.16:0/4219497798 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f645001c7a0 con 0x7f645815d8d0 2023-10-16T13:48:31.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.429+0000 7f644e7fc700 1 --2- 172.21.15.16:0/4219497798 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f643c06b140 0x7f643c06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:31.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.429+0000 7f644e7fc700 1 -- 172.21.15.16:0/4219497798 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f645008b6f0 con 0x7f645815d8d0 2023-10-16T13:48:31.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.429+0000 7f645df17700 1 --2- 172.21.15.16:0/4219497798 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f643c06b140 0x7f643c06d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:31.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.430+0000 7f645df17700 1 --2- 172.21.15.16:0/4219497798 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f643c06b140 0x7f643c06d600 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6454007a50 tx=0x7f6454008040 comp rx=0 tx=0).ready entity=mgr.12678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:31.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.431+0000 7f646c365b80 1 Processor -- start 2023-10-16T13:48:31.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.431+0000 7f646c365b80 1 -- start start 2023-10-16T13:48:31.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.432+0000 7f646c365b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ed699ba220 0x55ed699c1a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:31.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.432+0000 7f646c365b80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55ed699bf630 0x55ed699c1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:31.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.432+0000 7f646c365b80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ed69995010 0x55ed699956c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:31.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.432+0000 7f646c365b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed6990bc20 con 0x55ed699ba220 2023-10-16T13:48:31.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.432+0000 7f646c365b80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed698f0c10 con 0x55ed69995010 2023-10-16T13:48:31.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.432+0000 7f646c365b80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ed698ee530 con 0x55ed699bf630 2023-10-16T13:48:31.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.432+0000 7f645df17700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ed699ba220 0x55ed699c1a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:31.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.432+0000 7f645df17700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ed699ba220 0x55ed699c1a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57644/0 (socket says 172.21.15.16:57644) 2023-10-16T13:48:31.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.432+0000 7f645df17700 1 -- 172.21.15.16:0/199562337 learned_addr learned my addr 172.21.15.16:0/199562337 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:31.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.432+0000 7f645e718700 1 --2- 172.21.15.16:0/199562337 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ed69995010 0x55ed699956c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:31.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.432+0000 7f645d716700 1 --2- 172.21.15.16:0/199562337 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55ed699bf630 0x55ed699c1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:31.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.432+0000 7f645df17700 1 -- 172.21.15.16:0/199562337 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55ed699bf630 msgr2=0x55ed699c1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:31.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.432+0000 7f645df17700 1 --2- 172.21.15.16:0/199562337 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55ed699bf630 0x55ed699c1fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:31.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.432+0000 7f645df17700 1 -- 172.21.15.16:0/199562337 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ed69995010 msgr2=0x55ed699956c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:31.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.432+0000 7f645df17700 1 --2- 172.21.15.16:0/199562337 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ed69995010 0x55ed699956c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:31.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.433+0000 7f645df17700 1 -- 172.21.15.16:0/199562337 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ed698a73a0 con 0x55ed699ba220 2023-10-16T13:48:31.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.433+0000 7f645d716700 1 --2- 172.21.15.16:0/199562337 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55ed699bf630 0x55ed699c1fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:48:31.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.433+0000 7f645df17700 1 --2- 172.21.15.16:0/199562337 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ed699ba220 0x55ed699c1a70 secure :-1 s=READY pgs=3025 cs=0 l=1 rev1=1 crypto rx=0x7f645400f910 tx=0x7f6454012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f9a3a54ae6ece0d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:31.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.433+0000 7f643b7fe700 1 -- 172.21.15.16:0/199562337 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64540146e0 con 0x55ed699ba220 2023-10-16T13:48:31.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.433+0000 7f643b7fe700 1 -- 172.21.15.16:0/199562337 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6454019070 con 0x55ed699ba220 2023-10-16T13:48:31.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.433+0000 7f643b7fe700 1 -- 172.21.15.16:0/199562337 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6454014a40 con 0x55ed699ba220 2023-10-16T13:48:31.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.433+0000 7f646c365b80 1 -- 172.21.15.16:0/199562337 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ed699ba220 msgr2=0x55ed699c1a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:31.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.433+0000 7f646c365b80 1 --2- 172.21.15.16:0/199562337 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ed699ba220 0x55ed699c1a70 secure :-1 s=READY pgs=3025 cs=0 l=1 rev1=1 crypto rx=0x7f645400f910 tx=0x7f6454012000 comp rx=0 tx=0).stop 2023-10-16T13:48:31.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.433+0000 7f646c365b80 1 -- 172.21.15.16:0/199562337 shutdown_connections 2023-10-16T13:48:31.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.434+0000 7f646c365b80 1 --2- 172.21.15.16:0/199562337 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ed69995010 0x55ed699956c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:31.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.434+0000 7f646c365b80 1 --2- 172.21.15.16:0/199562337 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ed699ba220 0x55ed699c1a70 unknown :-1 s=CLOSED pgs=3025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:31.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.434+0000 7f646c365b80 1 --2- 172.21.15.16:0/199562337 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55ed699bf630 0x55ed699c1fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:31.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.434+0000 7f646c365b80 1 -- 172.21.15.16:0/199562337 >> 172.21.15.16:0/199562337 conn(0x55ed698c4d80 msgr2=0x55ed699dfff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:31.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.434+0000 7f646c365b80 1 -- 172.21.15.16:0/199562337 shutdown_connections 2023-10-16T13:48:31.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.434+0000 7f646c365b80 1 -- 172.21.15.16:0/199562337 wait complete. 2023-10-16T13:48:31.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.434+0000 7f646c365b80 1 -- start start 2023-10-16T13:48:31.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.434+0000 7f646c365b80 1 Processor -- start 2023-10-16T13:48:31.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.435+0000 7f646c365b80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ed69995010 0x55ed699b92b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:31.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.435+0000 7f646c365b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ed699ba220 0x55ed699bdac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:31.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.435+0000 7f646c365b80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55ed699bf630 0x55ed69995450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:31.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.435+0000 7f646c365b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed698e9e20 con 0x55ed699ba220 2023-10-16T13:48:31.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.435+0000 7f646c365b80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed698866c0 con 0x55ed69995010 2023-10-16T13:48:31.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.435+0000 7f646c365b80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ed69886360 con 0x55ed699bf630 2023-10-16T13:48:31.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.435+0000 7f646c365b80 10 client.0 _ll_register_callbacks cb 0x55ed696e37b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T13:48:31.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.435+0000 7f645d716700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ed699ba220 0x55ed699bdac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:31.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.435+0000 7f645d716700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ed699ba220 0x55ed699bdac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57652/0 (socket says 172.21.15.16:57652) 2023-10-16T13:48:31.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.435+0000 7f645d716700 1 -- 172.21.15.16:0/3256154290 learned_addr learned my addr 172.21.15.16:0/3256154290 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:31.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.436+0000 7f645df17700 1 --2- 172.21.15.16:0/3256154290 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ed69995010 0x55ed699b92b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:31.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.436+0000 7f645e718700 1 --2- 172.21.15.16:0/3256154290 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55ed699bf630 0x55ed69995450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:31.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.436+0000 7f645d716700 1 -- 172.21.15.16:0/3256154290 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55ed699bf630 msgr2=0x55ed69995450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:31.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.436+0000 7f645d716700 1 --2- 172.21.15.16:0/3256154290 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55ed699bf630 0x55ed69995450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:31.453 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.436+0000 7f645d716700 1 -- 172.21.15.16:0/3256154290 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ed69995010 msgr2=0x55ed699b92b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:31.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.436+0000 7f645d716700 1 --2- 172.21.15.16:0/3256154290 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ed69995010 0x55ed699b92b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:31.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.436+0000 7f645d716700 1 -- 172.21.15.16:0/3256154290 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f645400e040 con 0x55ed699ba220 2023-10-16T13:48:31.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.436+0000 7f645df17700 1 --2- 172.21.15.16:0/3256154290 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ed69995010 0x55ed699b92b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:48:31.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.436+0000 7f645d716700 1 --2- 172.21.15.16:0/3256154290 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ed699ba220 0x55ed699bdac0 secure :-1 s=READY pgs=3026 cs=0 l=1 rev1=1 crypto rx=0x7f64581e8190 tx=0x7f644800f560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8241210e7ac04ad8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:31.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.436+0000 7f6438ff9700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2023-10-16T13:48:31.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.436+0000 7f6438ff9700 1 -- 172.21.15.16:0/3256154290 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f644800fe10 con 0x55ed699ba220 2023-10-16T13:48:31.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.436+0000 7f6438ff9700 1 -- 172.21.15.16:0/3256154290 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(265..265 src has 1..265) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f64480050d0 con 0x55ed699ba220 2023-10-16T13:48:31.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.436+0000 7f646c365b80 1 -- 172.21.15.16:0/3256154290 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ed6998a5b0 con 0x55ed699ba220 2023-10-16T13:48:31.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.436+0000 7f6438ff9700 1 -- 172.21.15.16:0/3256154290 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=146) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6448005290 con 0x55ed699ba220 2023-10-16T13:48:31.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.436+0000 7f646c365b80 10 client.13197 fetch_fsmap learned FSMap version 146 2023-10-16T13:48:31.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.436+0000 7f646c365b80 1 -- 172.21.15.16:0/3256154290 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=146,monmap=2+}) v3 -- 0x55ed69714d90 con 0x55ed699ba220 2023-10-16T13:48:31.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.436+0000 7f6438ff9700 1 -- 172.21.15.16:0/3256154290 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55ed6998a5b0 con 0x55ed699ba220 2023-10-16T13:48:31.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.436+0000 7f6438ff9700 1 -- 172.21.15.16:0/3256154290 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 146) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f6448012070 con 0x55ed699ba220 2023-10-16T13:48:31.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.436+0000 7f646c365b80 10 client.13197 fetch_fsmap finished waiting for FSMap version 146 2023-10-16T13:48:31.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.436+0000 7f646c365b80 10 client.13197 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-16T13:48:31.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.436+0000 7f646c365b80 20 client.13197 populate_metadata read hostname 'smithi016' 2023-10-16T13:48:31.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.436+0000 7f646c365b80 1 --2- 172.21.15.16:0/3256154290 >> [v2:172.21.15.49:6833/1985453659,v1:172.21.15.49:6835/1985453659] conn(0x55ed699babf0 0x55ed699ef470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:31.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.436+0000 7f646c365b80 4 client.13197 mds_command: new command op to 13068 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T13:48:31.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.436+0000 7f646c365b80 1 -- 172.21.15.16:0/3256154290 --> [v2:172.21.15.49:6833/1985453659,v1:172.21.15.49:6835/1985453659] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ed698e2e00 con 0x55ed699babf0 2023-10-16T13:48:31.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.437+0000 7f645df17700 1 --2- 172.21.15.16:0/3256154290 >> [v2:172.21.15.49:6833/1985453659,v1:172.21.15.49:6835/1985453659] conn(0x55ed699babf0 0x55ed699ef470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:31.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.437+0000 7f645df17700 1 --2- 172.21.15.16:0/3256154290 >> [v2:172.21.15.49:6833/1985453659,v1:172.21.15.49:6835/1985453659] conn(0x55ed699babf0 0x55ed699ef470 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a0035efc7a971a2a server_cookie=2afe4c878eac2511 in_seq=0 out_seq=0 2023-10-16T13:48:31.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.437+0000 7f6438ff9700 10 client.13197 ms_handle_connect on v2:172.21.15.49:6833/1985453659 2023-10-16T13:48:31.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.438+0000 7f6438ff9700 1 -- 172.21.15.16:0/3256154290 <== mds.0 v2:172.21.15.49:6833/1985453659 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55ed698e2e00 con 0x55ed699babf0 2023-10-16T13:48:31.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.438+0000 7f6438ff9700 10 client.13197 handle_command_reply: tid=0 2023-10-16T13:48:31.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.438+0000 7f646c365b80 1 client.13197 shutdown 2023-10-16T13:48:31.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.438+0000 7f646c365b80 1 -- 172.21.15.16:0/3256154290 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ed699ba220 msgr2=0x55ed699bdac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:31.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.438+0000 7f646c365b80 1 --2- 172.21.15.16:0/3256154290 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ed699ba220 0x55ed699bdac0 secure :-1 s=READY pgs=3026 cs=0 l=1 rev1=1 crypto rx=0x7f64581e8190 tx=0x7f644800f560 comp rx=0 tx=0).stop 2023-10-16T13:48:31.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.438+0000 7f646c365b80 1 -- 172.21.15.16:0/3256154290 shutdown_connections 2023-10-16T13:48:31.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.438+0000 7f646c365b80 1 --2- 172.21.15.16:0/3256154290 >> [v2:172.21.15.49:6833/1985453659,v1:172.21.15.49:6835/1985453659] conn(0x55ed699babf0 0x55ed699ef470 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:31.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.438+0000 7f646c365b80 1 --2- 172.21.15.16:0/3256154290 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55ed699bf630 0x55ed69995450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:31.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.438+0000 7f646c365b80 1 --2- 172.21.15.16:0/3256154290 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ed69995010 0x55ed699b92b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:31.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.438+0000 7f646c365b80 1 --2- 172.21.15.16:0/3256154290 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ed699ba220 0x55ed699bdac0 unknown :-1 s=CLOSED pgs=3026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:31.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.438+0000 7f6438ff9700 0 client.13197 ms_handle_reset on v2:172.21.15.49:6833/1985453659 2023-10-16T13:48:31.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.438+0000 7f646c365b80 1 -- 172.21.15.16:0/3256154290 >> 172.21.15.16:0/3256154290 conn(0x55ed698c4d80 msgr2=0x55ed699dd450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:31.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.438+0000 7f646c365b80 1 -- 172.21.15.16:0/3256154290 shutdown_connections 2023-10-16T13:48:31.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.438+0000 7f646c365b80 1 -- 172.21.15.16:0/3256154290 wait complete. 2023-10-16T13:48:31.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.438+0000 7f646c365b80 20 client.13197 trim_cache size 0 max 16384 2023-10-16T13:48:31.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.456+0000 7f646c365b80 1 Processor -- start 2023-10-16T13:48:31.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.457+0000 7f646c365b80 1 -- start start 2023-10-16T13:48:31.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.457+0000 7f646c365b80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ed69995010 0x55ed699b92b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:31.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.457+0000 7f646c365b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ed699ba220 0x55ed699c1a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:31.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.457+0000 7f646c365b80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55ed699bf630 0x55ed699c4410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:31.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.457+0000 7f646c365b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed6976a8d0 con 0x55ed699ba220 2023-10-16T13:48:31.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.457+0000 7f646c365b80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed699eb970 con 0x55ed69995010 2023-10-16T13:48:31.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.457+0000 7f646c365b80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ed699dd5f0 con 0x55ed699bf630 2023-10-16T13:48:31.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.457+0000 7f645d716700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ed699ba220 0x55ed699c1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:31.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.457+0000 7f645d716700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ed699ba220 0x55ed699c1a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57660/0 (socket says 172.21.15.16:57660) 2023-10-16T13:48:31.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.457+0000 7f645d716700 1 -- 172.21.15.16:0/1934837712 learned_addr learned my addr 172.21.15.16:0/1934837712 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:31.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.458+0000 7f645df17700 1 --2- 172.21.15.16:0/1934837712 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ed69995010 0x55ed699b92b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:31.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.458+0000 7f645e718700 1 --2- 172.21.15.16:0/1934837712 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55ed699bf630 0x55ed699c4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:31.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.458+0000 7f645d716700 1 -- 172.21.15.16:0/1934837712 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55ed699bf630 msgr2=0x55ed699c4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:31.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.458+0000 7f645d716700 1 --2- 172.21.15.16:0/1934837712 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55ed699bf630 0x55ed699c4410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:31.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.458+0000 7f645d716700 1 -- 172.21.15.16:0/1934837712 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ed69995010 msgr2=0x55ed699b92b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:31.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.458+0000 7f645d716700 1 --2- 172.21.15.16:0/1934837712 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ed69995010 0x55ed699b92b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:31.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.458+0000 7f645d716700 1 -- 172.21.15.16:0/1934837712 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ed696fdc10 con 0x55ed699ba220 2023-10-16T13:48:31.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.458+0000 7f645d716700 1 --2- 172.21.15.16:0/1934837712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ed699ba220 0x55ed699c1a60 secure :-1 s=READY pgs=3027 cs=0 l=1 rev1=1 crypto rx=0x7f6448000fc0 tx=0x7f644800b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=804f46c0a821444d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:31.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.458+0000 7f643bfff700 1 -- 172.21.15.16:0/1934837712 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f644800fe10 con 0x55ed699ba220 2023-10-16T13:48:31.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.458+0000 7f643bfff700 1 -- 172.21.15.16:0/1934837712 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f644800d050 con 0x55ed699ba220 2023-10-16T13:48:31.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.458+0000 7f643bfff700 1 -- 172.21.15.16:0/1934837712 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6448006a80 con 0x55ed699ba220 2023-10-16T13:48:31.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.459+0000 7f646c365b80 1 -- 172.21.15.16:0/1934837712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ed699ba220 msgr2=0x55ed699c1a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:31.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.459+0000 7f646c365b80 1 --2- 172.21.15.16:0/1934837712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ed699ba220 0x55ed699c1a60 secure :-1 s=READY pgs=3027 cs=0 l=1 rev1=1 crypto rx=0x7f6448000fc0 tx=0x7f644800b000 comp rx=0 tx=0).stop 2023-10-16T13:48:31.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.459+0000 7f646c365b80 1 -- 172.21.15.16:0/1934837712 shutdown_connections 2023-10-16T13:48:31.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.459+0000 7f646c365b80 1 --2- 172.21.15.16:0/1934837712 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55ed699bf630 0x55ed699c4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:31.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.459+0000 7f646c365b80 1 --2- 172.21.15.16:0/1934837712 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ed69995010 0x55ed699b92b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:31.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.459+0000 7f646c365b80 1 --2- 172.21.15.16:0/1934837712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ed699ba220 0x55ed699c1a60 unknown :-1 s=CLOSED pgs=3027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:31.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.459+0000 7f646c365b80 1 -- 172.21.15.16:0/1934837712 >> 172.21.15.16:0/1934837712 conn(0x55ed698c4d80 msgr2=0x55ed699c3c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:31.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.459+0000 7f646c365b80 1 -- 172.21.15.16:0/1934837712 shutdown_connections 2023-10-16T13:48:31.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.459+0000 7f646c365b80 1 -- 172.21.15.16:0/1934837712 wait complete. 2023-10-16T13:48:31.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.459+0000 7f646c365b80 1 -- start start 2023-10-16T13:48:31.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.459+0000 7f646c365b80 1 Processor -- start 2023-10-16T13:48:31.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.460+0000 7f646c365b80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ed69995010 0x55ed699c4870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:31.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.460+0000 7f646c365b80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55ed699ba220 0x55ed699dc450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:31.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.460+0000 7f646c365b80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ed699bf630 0x55ed699dc990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:31.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.460+0000 7f646c365b80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed6998c060 con 0x55ed699bf630 2023-10-16T13:48:31.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.460+0000 7f646c365b80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed6998c270 con 0x55ed69995010 2023-10-16T13:48:31.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.460+0000 7f646c365b80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ed69a445c0 con 0x55ed699ba220 2023-10-16T13:48:31.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.460+0000 7f646c365b80 10 client.0 _ll_register_callbacks cb 0x7f6448001ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T13:48:31.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.460+0000 7f645e718700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ed699bf630 0x55ed699dc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:31.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.460+0000 7f645e718700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ed699bf630 0x55ed699dc990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57668/0 (socket says 172.21.15.16:57668) 2023-10-16T13:48:31.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.460+0000 7f645e718700 1 -- 172.21.15.16:0/3133613226 learned_addr learned my addr 172.21.15.16:0/3133613226 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:31.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.461+0000 7f645df17700 1 --2- 172.21.15.16:0/3133613226 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ed69995010 0x55ed699c4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:31.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.461+0000 7f645d716700 1 --2- 172.21.15.16:0/3133613226 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55ed699ba220 0x55ed699dc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:31.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.461+0000 7f645e718700 1 -- 172.21.15.16:0/3133613226 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55ed699ba220 msgr2=0x55ed699dc450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:31.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.461+0000 7f645e718700 1 --2- 172.21.15.16:0/3133613226 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55ed699ba220 0x55ed699dc450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:31.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.461+0000 7f645e718700 1 -- 172.21.15.16:0/3133613226 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ed69995010 msgr2=0x55ed699c4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:31.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.461+0000 7f645e718700 1 --2- 172.21.15.16:0/3133613226 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ed69995010 0x55ed699c4870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:31.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.461+0000 7f645e718700 1 -- 172.21.15.16:0/3133613226 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ed69714d90 con 0x55ed699bf630 2023-10-16T13:48:31.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.461+0000 7f645e718700 1 --2- 172.21.15.16:0/3133613226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ed699bf630 0x55ed699dc990 secure :-1 s=READY pgs=3028 cs=0 l=1 rev1=1 crypto rx=0x7f6450056320 tx=0x7f645000b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea405ff127b7d115 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:31.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.461+0000 7f6438ff9700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2023-10-16T13:48:31.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.461+0000 7f6438ff9700 1 -- 172.21.15.16:0/3133613226 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f645005df70 con 0x55ed699bf630 2023-10-16T13:48:31.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.461+0000 7f646c365b80 1 -- 172.21.15.16:0/3133613226 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6454019070 con 0x55ed699bf630 2023-10-16T13:48:31.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.461+0000 7f6438ff9700 1 -- 172.21.15.16:0/3133613226 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(265..265 src has 1..265) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f6450062120 con 0x55ed699bf630 2023-10-16T13:48:31.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.461+0000 7f6438ff9700 1 -- 172.21.15.16:0/3133613226 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=146) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6450089250 con 0x55ed699bf630 2023-10-16T13:48:31.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.461+0000 7f646c365b80 10 client.13203 fetch_fsmap learned FSMap version 146 2023-10-16T13:48:31.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.461+0000 7f646c365b80 1 -- 172.21.15.16:0/3133613226 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=146,monmap=2+}) v3 -- 0x55ed6973b200 con 0x55ed699bf630 2023-10-16T13:48:31.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.461+0000 7f6438ff9700 1 -- 172.21.15.16:0/3133613226 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6454019070 con 0x55ed699bf630 2023-10-16T13:48:31.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.461+0000 7f6438ff9700 1 -- 172.21.15.16:0/3133613226 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 146) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f6450076020 con 0x55ed699bf630 2023-10-16T13:48:31.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.461+0000 7f646c365b80 10 client.13203 fetch_fsmap finished waiting for FSMap version 146 2023-10-16T13:48:31.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.461+0000 7f646c365b80 10 client.13203 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-16T13:48:31.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.461+0000 7f646c365b80 20 client.13203 populate_metadata read hostname 'smithi016' 2023-10-16T13:48:31.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.461+0000 7f646c365b80 1 --2- 172.21.15.16:0/3133613226 >> [v2:172.21.15.49:6833/1985453659,v1:172.21.15.49:6835/1985453659] conn(0x55ed69a43a70 0x55ed699f5430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:31.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.462+0000 7f646c365b80 4 client.13203 mds_command: new command op to 13068 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T13:48:31.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.462+0000 7f646c365b80 1 -- 172.21.15.16:0/3133613226 --> [v2:172.21.15.49:6833/1985453659,v1:172.21.15.49:6835/1985453659] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55ed698e2b20 con 0x55ed69a43a70 2023-10-16T13:48:31.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.462+0000 7f645df17700 1 --2- 172.21.15.16:0/3133613226 >> [v2:172.21.15.49:6833/1985453659,v1:172.21.15.49:6835/1985453659] conn(0x55ed69a43a70 0x55ed699f5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:31.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.462+0000 7f6438ff9700 10 client.13203 ms_handle_connect on v2:172.21.15.49:6833/1985453659 2023-10-16T13:48:31.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.462+0000 7f645df17700 1 --2- 172.21.15.16:0/3133613226 >> [v2:172.21.15.49:6833/1985453659,v1:172.21.15.49:6835/1985453659] conn(0x55ed69a43a70 0x55ed699f5430 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8897ab22186c357b server_cookie=930a80b660e8faa0 in_seq=0 out_seq=0 2023-10-16T13:48:31.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.462+0000 7f6438ff9700 1 -- 172.21.15.16:0/3133613226 <== mds.0 v2:172.21.15.49:6833/1985453659 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+335 (crc 0 0 0) 0x55ed698e2b20 con 0x55ed69a43a70 2023-10-16T13:48:31.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.462+0000 7f6438ff9700 10 client.13203 handle_command_reply: tid=0 2023-10-16T13:48:31.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.462+0000 7f646c365b80 1 client.13203 shutdown 2023-10-16T13:48:31.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.462+0000 7f646c365b80 1 -- 172.21.15.16:0/3133613226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ed699bf630 msgr2=0x55ed699dc990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:31.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.462+0000 7f646c365b80 1 --2- 172.21.15.16:0/3133613226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ed699bf630 0x55ed699dc990 secure :-1 s=READY pgs=3028 cs=0 l=1 rev1=1 crypto rx=0x7f6450056320 tx=0x7f645000b7e0 comp rx=0 tx=0).stop 2023-10-16T13:48:31.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.463+0000 7f646c365b80 1 -- 172.21.15.16:0/3133613226 shutdown_connections 2023-10-16T13:48:31.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.463+0000 7f646c365b80 1 --2- 172.21.15.16:0/3133613226 >> [v2:172.21.15.49:6833/1985453659,v1:172.21.15.49:6835/1985453659] conn(0x55ed69a43a70 0x55ed699f5430 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:31.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.463+0000 7f646c365b80 1 --2- 172.21.15.16:0/3133613226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ed699bf630 0x55ed699dc990 unknown :-1 s=CLOSED pgs=3028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:31.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.463+0000 7f6438ff9700 0 client.13203 ms_handle_reset on v2:172.21.15.49:6833/1985453659 2023-10-16T13:48:31.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.463+0000 7f646c365b80 1 --2- 172.21.15.16:0/3133613226 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ed69995010 0x55ed699c4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:31.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.463+0000 7f646c365b80 1 --2- 172.21.15.16:0/3133613226 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x55ed699ba220 0x55ed699dc450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:31.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.463+0000 7f646c365b80 1 -- 172.21.15.16:0/3133613226 >> 172.21.15.16:0/3133613226 conn(0x55ed698c4d80 msgr2=0x55ed699bafd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:31.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.463+0000 7f646c365b80 1 -- 172.21.15.16:0/3133613226 shutdown_connections 2023-10-16T13:48:31.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.463+0000 7f646c365b80 1 -- 172.21.15.16:0/3133613226 wait complete. 2023-10-16T13:48:31.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.463+0000 7f646c365b80 20 client.13203 trim_cache size 0 max 16384 2023-10-16T13:48:31.482 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-10-16T13:48:31.482 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "5b28dfa1-b21f-474a-9b2e-9fb77af58254", 2023-10-16T13:48:31.483 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 0, 2023-10-16T13:48:31.483 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 13068, 2023-10-16T13:48:31.483 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2023-10-16T13:48:31.483 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2023-10-16T13:48:31.483 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2023-10-16T13:48:31.483 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 6.0281289229999997, 2023-10-16T13:48:31.484 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 146, 2023-10-16T13:48:31.484 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 265, 2023-10-16T13:48:31.484 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 265, 2023-10-16T13:48:31.484 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 10.922602443000001 2023-10-16T13:48:31.484 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-10-16T13:48:31.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.463+0000 7f646017b700 1 -- 172.21.15.16:0/4219497798 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f643c06b140 msgr2=0x7f643c06d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:31.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.463+0000 7f646017b700 1 --2- 172.21.15.16:0/4219497798 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f643c06b140 0x7f643c06d600 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6454007a50 tx=0x7f6454008040 comp rx=0 tx=0).stop 2023-10-16T13:48:31.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.463+0000 7f646017b700 1 -- 172.21.15.16:0/4219497798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f645815d8d0 msgr2=0x7f64581e7640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:31.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.463+0000 7f646017b700 1 --2- 172.21.15.16:0/4219497798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f645815d8d0 0x7f64581e7640 secure :-1 s=READY pgs=3024 cs=0 l=1 rev1=1 crypto rx=0x7f6450012000 tx=0x7f6450004ad0 comp rx=0 tx=0).stop 2023-10-16T13:48:31.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.463+0000 7f646017b700 1 -- 172.21.15.16:0/4219497798 shutdown_connections 2023-10-16T13:48:31.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.463+0000 7f646017b700 1 --2- 172.21.15.16:0/4219497798 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f643c06b140 0x7f643c06d600 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:31.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.463+0000 7f646017b700 1 --2- 172.21.15.16:0/4219497798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f645815d8d0 0x7f64581e7640 unknown :-1 s=CLOSED pgs=3024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:31.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.463+0000 7f646017b700 1 --2- 172.21.15.16:0/4219497798 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f64580d0750 0x7f6458157f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:31.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.463+0000 7f646017b700 1 --2- 172.21.15.16:0/4219497798 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f64580d41c0 0x7f64581e7100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:31.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.463+0000 7f646017b700 1 -- 172.21.15.16:0/4219497798 >> 172.21.15.16:0/4219497798 conn(0x7f64580d3630 msgr2=0x7f645815c7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:31.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.463+0000 7f646017b700 1 -- 172.21.15.16:0/4219497798 shutdown_connections 2023-10-16T13:48:31.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.463+0000 7f646017b700 1 -- 172.21.15.16:0/4219497798 wait complete. 2023-10-16T13:48:31.488 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T13:48:31.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.611+0000 7f921e967700 1 Processor -- start 2023-10-16T13:48:31.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.611+0000 7f921e967700 1 -- start start 2023-10-16T13:48:31.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.611+0000 7f921e967700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f92180b2b50 0x7f92180d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:31.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.611+0000 7f921e967700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f92180ce380 0x7f92180d09e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:31.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.611+0000 7f921e967700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92180d1210 con 0x7f92180cdfa0 2023-10-16T13:48:31.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.611+0000 7f921e967700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92180cbc30 con 0x7f92180b2b50 2023-10-16T13:48:31.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.611+0000 7f921e967700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92180cbf10 con 0x7f92180ce380 2023-10-16T13:48:31.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.612+0000 7f92177fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f92180b2b50 0x7f92180d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:31.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.612+0000 7f92177fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f92180b2b50 0x7f92180d04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57674/0 (socket says 172.21.15.16:57674) 2023-10-16T13:48:31.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.612+0000 7f92177fe700 1 -- 172.21.15.16:0/121966049 learned_addr learned my addr 172.21.15.16:0/121966049 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:31.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.612+0000 7f921cf04700 1 --2- 172.21.15.16:0/121966049 >> v2:172.21.15.49:3300/0 conn(0x7f92180ce380 0x7f92180d09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:31.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.612+0000 7f92177fe700 1 -- 172.21.15.16:0/121966049 >> v2:172.21.15.49:3300/0 conn(0x7f92180ce380 msgr2=0x7f92180d09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:31.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.612+0000 7f92177fe700 1 --2- 172.21.15.16:0/121966049 >> v2:172.21.15.49:3300/0 conn(0x7f92180ce380 0x7f92180d09e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:31.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.612+0000 7f92177fe700 1 -- 172.21.15.16:0/121966049 >> v1:172.21.15.49:6789/0 conn(0x7f92180cdfa0 legacy=0x7f92180b2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:31.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.612+0000 7f92177fe700 1 -- 172.21.15.16:0/121966049 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92180cc1f0 con 0x7f92180b2b50 2023-10-16T13:48:31.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.612+0000 7f92177fe700 1 --2- 172.21.15.16:0/121966049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92180b2b50 0x7f92180d04a0 secure :-1 s=READY pgs=3029 cs=0 l=1 rev1=1 crypto rx=0x7f920c0098f0 tx=0x7f920c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d8ed7f123cd6d78 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:31.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.612+0000 7f9216ffd700 1 -- 172.21.15.16:0/121966049 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f920c00b860 con 0x7f92180b2b50 2023-10-16T13:48:31.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.612+0000 7f9216ffd700 1 -- 172.21.15.16:0/121966049 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f920c00b9c0 con 0x7f92180b2b50 2023-10-16T13:48:31.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.612+0000 7f9216ffd700 1 -- 172.21.15.16:0/121966049 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f920c00bd50 con 0x7f92180b2b50 2023-10-16T13:48:31.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.613+0000 7f921e967700 1 -- 172.21.15.16:0/121966049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92180b2b50 msgr2=0x7f92180d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:31.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.613+0000 7f921e967700 1 --2- 172.21.15.16:0/121966049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92180b2b50 0x7f92180d04a0 secure :-1 s=READY pgs=3029 cs=0 l=1 rev1=1 crypto rx=0x7f920c0098f0 tx=0x7f920c009c00 comp rx=0 tx=0).stop 2023-10-16T13:48:31.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.613+0000 7f921e967700 1 -- 172.21.15.16:0/121966049 shutdown_connections 2023-10-16T13:48:31.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.613+0000 7f921e967700 1 --2- 172.21.15.16:0/121966049 >> v2:172.21.15.49:3300/0 conn(0x7f92180ce380 0x7f92180d09e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:31.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.613+0000 7f921e967700 1 --2- 172.21.15.16:0/121966049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92180b2b50 0x7f92180d04a0 unknown :-1 s=CLOSED pgs=3029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:31.619 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.613+0000 7f921e967700 1 -- 172.21.15.16:0/121966049 >> 172.21.15.16:0/121966049 conn(0x7f92180d14b0 msgr2=0x7f92180d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:31.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.613+0000 7f921e967700 1 -- 172.21.15.16:0/121966049 shutdown_connections 2023-10-16T13:48:31.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.613+0000 7f921e967700 1 -- 172.21.15.16:0/121966049 wait complete. 2023-10-16T13:48:31.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.614+0000 7f921e967700 1 Processor -- start 2023-10-16T13:48:31.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.614+0000 7f921e967700 1 -- start start 2023-10-16T13:48:31.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.615+0000 7f921e967700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92180b2b50 0x7f92180c64a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:31.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.615+0000 7f921e967700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f92180cdfa0 0x7f921814f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:31.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.615+0000 7f921e967700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f92180ce380 0x7f92180c96e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:31.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.615+0000 7f921e967700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92181f30d0 con 0x7f92180b2b50 2023-10-16T13:48:31.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.615+0000 7f921e967700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92181f33b0 con 0x7f92180cdfa0 2023-10-16T13:48:31.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.615+0000 7f921e967700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92181f3690 con 0x7f92180ce380 2023-10-16T13:48:31.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.615+0000 7f9217fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92180b2b50 0x7f92180c64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:31.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.615+0000 7f9217fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92180b2b50 0x7f92180c64a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57678/0 (socket says 172.21.15.16:57678) 2023-10-16T13:48:31.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.615+0000 7f9217fff700 1 -- 172.21.15.16:0/371569870 learned_addr learned my addr 172.21.15.16:0/371569870 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:31.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.615+0000 7f92177fe700 1 --2- 172.21.15.16:0/371569870 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f92180cdfa0 0x7f921814f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:31.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.615+0000 7f921cf04700 1 --2- 172.21.15.16:0/371569870 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f92180ce380 0x7f92180c96e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:31.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.615+0000 7f9217fff700 1 -- 172.21.15.16:0/371569870 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f92180ce380 msgr2=0x7f92180c96e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:31.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.615+0000 7f9217fff700 1 --2- 172.21.15.16:0/371569870 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f92180ce380 0x7f92180c96e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:31.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.615+0000 7f9217fff700 1 -- 172.21.15.16:0/371569870 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f92180cdfa0 msgr2=0x7f921814f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:31.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.615+0000 7f9217fff700 1 --2- 172.21.15.16:0/371569870 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f92180cdfa0 0x7f921814f930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:31.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.615+0000 7f9217fff700 1 -- 172.21.15.16:0/371569870 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f920c009550 con 0x7f92180b2b50 2023-10-16T13:48:31.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.615+0000 7f9217fff700 1 --2- 172.21.15.16:0/371569870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92180b2b50 0x7f92180c64a0 secure :-1 s=READY pgs=3030 cs=0 l=1 rev1=1 crypto rx=0x7f920800c4e0 tx=0x7f920800c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:31.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.616+0000 7f9214ff9700 1 -- 172.21.15.16:0/371569870 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92080052d0 con 0x7f92180b2b50 2023-10-16T13:48:31.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.616+0000 7f9214ff9700 1 -- 172.21.15.16:0/371569870 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9208005430 con 0x7f92180b2b50 2023-10-16T13:48:31.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.616+0000 7f921e967700 1 -- 172.21.15.16:0/371569870 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92181f3970 con 0x7f92180b2b50 2023-10-16T13:48:31.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.616+0000 7f9214ff9700 1 -- 172.21.15.16:0/371569870 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92080038c0 con 0x7f92180b2b50 2023-10-16T13:48:31.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.616+0000 7f921e967700 1 -- 172.21.15.16:0/371569870 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f921814fe70 con 0x7f92180b2b50 2023-10-16T13:48:31.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.617+0000 7f921e967700 1 -- 172.21.15.16:0/371569870 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f921804ebb0 con 0x7f92180b2b50 2023-10-16T13:48:31.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.617+0000 7f9214ff9700 1 -- 172.21.15.16:0/371569870 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f9208003a20 con 0x7f92180b2b50 2023-10-16T13:48:31.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.618+0000 7f9214ff9700 1 --2- 172.21.15.16:0/371569870 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f920406b140 0x7f920406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:31.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.618+0000 7f9214ff9700 1 -- 172.21.15.16:0/371569870 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f9208088e80 con 0x7f92180b2b50 2023-10-16T13:48:31.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.618+0000 7f92177fe700 1 --2- 172.21.15.16:0/371569870 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f920406b140 0x7f920406d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:31.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.618+0000 7f92177fe700 1 --2- 172.21.15.16:0/371569870 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f920406b140 0x7f920406d600 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f920c009c40 tx=0x7f920c008030 comp rx=0 tx=0).ready entity=mgr.12678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:31.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.622+0000 7f9214ff9700 1 -- 172.21.15.16:0/371569870 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9208053ae0 con 0x7f92180b2b50 2023-10-16T13:48:31.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.874+0000 7f921e967700 1 -- 172.21.15.16:0/371569870 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f92180712b0 con 0x7f92180b2b50 2023-10-16T13:48:31.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.875+0000 7f9214ff9700 1 -- 172.21.15.16:0/371569870 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v146) v1 ==== 53+0+73 (secure 0 0 0) 0x7f9208017070 con 0x7f92180b2b50 2023-10-16T13:48:31.878 INFO:teuthology.orchestra.run.smithi016.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-16T13:48:31.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.877+0000 7f921e967700 1 -- 172.21.15.16:0/371569870 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f920406b140 msgr2=0x7f920406d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:31.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.877+0000 7f921e967700 1 --2- 172.21.15.16:0/371569870 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f920406b140 0x7f920406d600 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f920c009c40 tx=0x7f920c008030 comp rx=0 tx=0).stop 2023-10-16T13:48:31.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.877+0000 7f921e967700 1 -- 172.21.15.16:0/371569870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92180b2b50 msgr2=0x7f92180c64a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:31.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.877+0000 7f921e967700 1 --2- 172.21.15.16:0/371569870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92180b2b50 0x7f92180c64a0 secure :-1 s=READY pgs=3030 cs=0 l=1 rev1=1 crypto rx=0x7f920800c4e0 tx=0x7f920800c7f0 comp rx=0 tx=0).stop 2023-10-16T13:48:31.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.877+0000 7f921e967700 1 -- 172.21.15.16:0/371569870 shutdown_connections 2023-10-16T13:48:31.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.877+0000 7f921e967700 1 --2- 172.21.15.16:0/371569870 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f920406b140 0x7f920406d600 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:31.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.877+0000 7f921e967700 1 --2- 172.21.15.16:0/371569870 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f92180ce380 0x7f92180c96e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:31.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.877+0000 7f921e967700 1 --2- 172.21.15.16:0/371569870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92180b2b50 0x7f92180c64a0 unknown :-1 s=CLOSED pgs=3030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:31.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.877+0000 7f921e967700 1 --2- 172.21.15.16:0/371569870 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f92180cdfa0 0x7f921814f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:31.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.877+0000 7f921e967700 1 -- 172.21.15.16:0/371569870 >> 172.21.15.16:0/371569870 conn(0x7f92180d14b0 msgr2=0x7f921815f990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:31.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.877+0000 7f921e967700 1 -- 172.21.15.16:0/371569870 shutdown_connections 2023-10-16T13:48:31.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:31.877+0000 7f921e967700 1 -- 172.21.15.16:0/371569870 wait complete. 2023-10-16T13:48:31.892 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-16T13:48:31.892 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T13:48:31.893 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi016.front.sepia.ceph.com 2023-10-16T13:48:31.893 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-16T13:48:31.893 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-16T13:48:31.894 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-16T13:48:31.894 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-16T13:48:31.894 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T13:48:31.895 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T13:48:31.895 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:48:31.895 DEBUG:teuthology.orchestra.run.smithi016:> ip addr 2023-10-16T13:48:31.913 INFO:teuthology.orchestra.run.smithi016.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T13:48:31.913 INFO:teuthology.orchestra.run.smithi016.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T13:48:31.913 INFO:teuthology.orchestra.run.smithi016.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T13:48:31.914 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T13:48:31.914 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 ::1/128 scope host 2023-10-16T13:48:31.914 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T13:48:31.914 INFO:teuthology.orchestra.run.smithi016.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T13:48:31.914 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e0 brd ff:ff:ff:ff:ff:ff 2023-10-16T13:48:31.915 INFO:teuthology.orchestra.run.smithi016.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T13:48:31.915 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e1 brd ff:ff:ff:ff:ff:ff 2023-10-16T13:48:31.915 INFO:teuthology.orchestra.run.smithi016.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T13:48:31.915 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:64 brd ff:ff:ff:ff:ff:ff 2023-10-16T13:48:31.915 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.21.15.16/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-16T13:48:31.916 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft 36424sec preferred_lft 36424sec 2023-10-16T13:48:31.916 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::ec4:7aff:febd:1564/64 scope link 2023-10-16T13:48:31.916 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T13:48:31.916 INFO:teuthology.orchestra.run.smithi016.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T13:48:31.916 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:65 brd ff:ff:ff:ff:ff:ff 2023-10-16T13:48:31.916 INFO:teuthology.orchestra.run.smithi016.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T13:48:31.917 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 36:cf:ea:25:db:f7 brd ff:ff:ff:ff:ff:ff 2023-10-16T13:48:31.917 INFO:teuthology.orchestra.run.smithi016.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-16T13:48:31.917 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T13:48:31.917 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::cfc:c5ff:fe52:a786/64 scope link 2023-10-16T13:48:31.917 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T13:48:31.917 INFO:teuthology.orchestra.run.smithi016.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T13:48:31.918 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 6e:62:9d:a4:43:23 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T13:48:31.918 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::6c62:9dff:fea4:4323/64 scope link 2023-10-16T13:48:31.918 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T13:48:31.918 INFO:teuthology.orchestra.run.smithi016.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T13:48:31.918 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 36:cf:ea:25:db:f7 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-16T13:48:31.919 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::34cf:eaff:fe25:dbf7/64 scope link 2023-10-16T13:48:31.919 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T13:48:31.919 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:48:31.919 DEBUG:teuthology.orchestra.run.smithi016:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T13:48:31.978 INFO:teuthology.orchestra.run.smithi016.stdout:1 2023-10-16T13:48:31.980 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:48:31.980 DEBUG:teuthology.orchestra.run.smithi016:> route 2023-10-16T13:48:32.038 INFO:teuthology.orchestra.run.smithi016.stdout:Kernel IP routing table 2023-10-16T13:48:32.039 INFO:teuthology.orchestra.run.smithi016.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T13:48:32.039 INFO:teuthology.orchestra.run.smithi016.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-16T13:48:32.039 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-16T13:48:32.039 INFO:teuthology.orchestra.run.smithi016.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T13:48:32.040 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:48:32.040 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec bash -c ' 2023-10-16T13:48:32.040 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2023-10-16T13:48:32.041 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-16T13:48:32.041 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-16T13:48:32.041 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-16T13:48:32.042 DEBUG:teuthology.orchestra.run.smithi016:> ') 2023-10-16T13:48:32.145 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:48:32.146 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2023-10-16T13:48:32.163 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-16T13:48:32.163 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-16T13:48:32.163 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T13:48:32.164 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:48:32.164 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set brx.0 up 2023-10-16T13:48:32.230 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T13:48:32.231 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-16T13:48:32.288 INFO:teuthology.orchestra.run.smithi016.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-16T13:48:32.288 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T13:48:32.289 DEBUG:teuthology.orchestra.run.smithi016:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-16T13:48:32.344 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-16T13:48:32.345 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:48:32.345 DEBUG:teuthology.orchestra.run.smithi016:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T13:48:32.431 INFO:teuthology.orchestra.run.smithi016.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T13:48:32.431 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T13:48:32.432 INFO:teuthology.orchestra.run.smithi016.stdout:invalid new device string format 2023-10-16T13:48:32.440 INFO:teuthology.orchestra.run.smithi016.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T13:48:32.441 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T13:48:32.441 INFO:teuthology.orchestra.run.smithi016.stdout:invalid new device string format 2023-10-16T13:48:32.441 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: resolved to: "172.21.15.16:3300,172.21.15.49:3300,172.21.15.49:3301" 2023-10-16T13:48:32.441 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T13:48:32.514 INFO:teuthology.orchestra.run.smithi016.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T13:48:32.515 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T13:48:32.515 INFO:teuthology.orchestra.run.smithi016.stdout:invalid new device string format 2023-10-16T13:48:32.515 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: resolved to: "172.21.15.16:3300,172.21.15.49:3300,172.21.15.49:3301" 2023-10-16T13:48:32.515 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T13:48:32.516 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: resolved to: "172.21.15.16:6789,172.21.15.49:6789,172.21.15.49:6790" 2023-10-16T13:48:32.516 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.16:6789,172.21.15.49:6789,172.21.15.49:6790:/ 2023-10-16T13:48:32.516 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=5b28dfa1-b21f-474a-9b2e-9fb77af58254" will pass to kernel 2023-10-16T13:48:32.517 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-16T13:48:32.517 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:48:32.518 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-16T13:48:32.545 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:48:32.546 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T13:48:32.635 INFO:teuthology.orchestra.run.smithi016.stdout:client10202 2023-10-16T13:48:32.635 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T13:48:32.636 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:48:32.653 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-16T13:48:32.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:32.816+0000 7f72e8da1700 1 Processor -- start 2023-10-16T13:48:32.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:32.816+0000 7f72e8da1700 1 -- start start 2023-10-16T13:48:32.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:32.817+0000 7f72e8da1700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f72e40cf4a0 0x7f72e40726e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:32.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:32.817+0000 7f72e8da1700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72e415bc80 con 0x7f72e40b5a00 2023-10-16T13:48:32.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:32.817+0000 7f72e8da1700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72e40b5e70 con 0x7f72e40cf880 2023-10-16T13:48:32.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:32.817+0000 7f72e8da1700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72e4072c20 con 0x7f72e40cf4a0 2023-10-16T13:48:32.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:32.817+0000 7f72e259c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f72e40cf4a0 0x7f72e40726e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:32.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:32.817+0000 7f72e259c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f72e40cf4a0 0x7f72e40726e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57690/0 (socket says 172.21.15.16:57690) 2023-10-16T13:48:32.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:32.817+0000 7f72e259c700 1 -- 172.21.15.16:0/293824706 learned_addr learned my addr 172.21.15.16:0/293824706 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:32.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:32.818+0000 7f72e259c700 1 -- 172.21.15.16:0/293824706 >> v1:172.21.15.49:6789/0 conn(0x7f72e40cf880 legacy=0x7f72e40b52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:32.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:32.818+0000 7f72e259c700 1 -- 172.21.15.16:0/293824706 >> v1:172.21.15.16:6789/0 conn(0x7f72e40b5a00 legacy=0x7f72e415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:32.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:32.818+0000 7f72e259c700 1 -- 172.21.15.16:0/293824706 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72e415b070 con 0x7f72e40cf4a0 2023-10-16T13:48:32.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:32.818+0000 7f72e259c700 1 --2- 172.21.15.16:0/293824706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72e40cf4a0 0x7f72e40726e0 secure :-1 s=READY pgs=3032 cs=0 l=1 rev1=1 crypto rx=0x7f72d80098f0 tx=0x7f72d8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e204d8c1d066534 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:32.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:32.818+0000 7f72e159a700 1 -- 172.21.15.16:0/293824706 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72d800b8a0 con 0x7f72e40cf4a0 2023-10-16T13:48:32.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:32.818+0000 7f72e159a700 1 -- 172.21.15.16:0/293824706 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72d800ba00 con 0x7f72e40cf4a0 2023-10-16T13:48:32.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:32.818+0000 7f72e159a700 1 -- 172.21.15.16:0/293824706 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72d800bd90 con 0x7f72e40cf4a0 2023-10-16T13:48:32.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:32.819+0000 7f72e8da1700 1 -- 172.21.15.16:0/293824706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72e40cf4a0 msgr2=0x7f72e40726e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:32.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:32.819+0000 7f72e8da1700 1 --2- 172.21.15.16:0/293824706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72e40cf4a0 0x7f72e40726e0 secure :-1 s=READY pgs=3032 cs=0 l=1 rev1=1 crypto rx=0x7f72d80098f0 tx=0x7f72d8009c00 comp rx=0 tx=0).stop 2023-10-16T13:48:32.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:32.819+0000 7f72e8da1700 1 -- 172.21.15.16:0/293824706 shutdown_connections 2023-10-16T13:48:32.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:32.819+0000 7f72e8da1700 1 --2- 172.21.15.16:0/293824706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72e40cf4a0 0x7f72e40726e0 unknown :-1 s=CLOSED pgs=3032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:32.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:32.819+0000 7f72e8da1700 1 -- 172.21.15.16:0/293824706 >> 172.21.15.16:0/293824706 conn(0x7f72e40b1fd0 msgr2=0x7f72e40b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:32.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:32.819+0000 7f72e8da1700 1 -- 172.21.15.16:0/293824706 shutdown_connections 2023-10-16T13:48:32.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:32.819+0000 7f72e8da1700 1 -- 172.21.15.16:0/293824706 wait complete. 2023-10-16T13:48:32.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:32.820+0000 7f72e8da1700 1 Processor -- start 2023-10-16T13:48:32.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:32.821+0000 7f72e8da1700 1 -- start start 2023-10-16T13:48:32.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:32.821+0000 7f72e8da1700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f72e40b5a00 0x7f72e414b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:32.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:32.821+0000 7f72e8da1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72e40cf4a0 0x7f72e40c4390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:32.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:32.821+0000 7f72e8da1700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f72e40cf880 0x7f72e406a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:32.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:32.821+0000 7f72e8da1700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72e41f75e0 con 0x7f72e40cf4a0 2023-10-16T13:48:32.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:32.821+0000 7f72e8da1700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72e41f69d0 con 0x7f72e40cf880 2023-10-16T13:48:32.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:32.821+0000 7f72e8da1700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72e41f6cb0 con 0x7f72e40b5a00 2023-10-16T13:48:32.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:32.822+0000 7f72e1d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72e40cf4a0 0x7f72e40c4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:32.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:32.822+0000 7f72e1d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72e40cf4a0 0x7f72e40c4390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57698/0 (socket says 172.21.15.16:57698) 2023-10-16T13:48:32.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:32.822+0000 7f72e1d9b700 1 -- 172.21.15.16:0/289438432 learned_addr learned my addr 172.21.15.16:0/289438432 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:32.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:32.822+0000 7f72e259c700 1 --2- 172.21.15.16:0/289438432 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f72e40b5a00 0x7f72e414b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:32.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:32.822+0000 7f72e2d9d700 1 --2- 172.21.15.16:0/289438432 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f72e40cf880 0x7f72e406a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:32.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:32.822+0000 7f72e1d9b700 1 -- 172.21.15.16:0/289438432 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f72e40b5a00 msgr2=0x7f72e414b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:32.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:32.822+0000 7f72e1d9b700 1 --2- 172.21.15.16:0/289438432 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f72e40b5a00 0x7f72e414b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:32.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:32.822+0000 7f72e1d9b700 1 -- 172.21.15.16:0/289438432 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f72e40cf880 msgr2=0x7f72e406a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:32.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:32.822+0000 7f72e1d9b700 1 --2- 172.21.15.16:0/289438432 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f72e40cf880 0x7f72e406a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:32.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:32.822+0000 7f72e1d9b700 1 -- 172.21.15.16:0/289438432 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72d8009550 con 0x7f72e40cf4a0 2023-10-16T13:48:32.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:32.822+0000 7f72e1d9b700 1 --2- 172.21.15.16:0/289438432 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72e40cf4a0 0x7f72e40c4390 secure :-1 s=READY pgs=3033 cs=0 l=1 rev1=1 crypto rx=0x7f72d400c450 tx=0x7f72d400c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:32.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:32.822+0000 7f72d2ffd700 1 -- 172.21.15.16:0/289438432 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72d40052d0 con 0x7f72e40cf4a0 2023-10-16T13:48:32.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:32.822+0000 7f72d2ffd700 1 -- 172.21.15.16:0/289438432 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72d4005430 con 0x7f72e40cf4a0 2023-10-16T13:48:32.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:32.822+0000 7f72e8da1700 1 -- 172.21.15.16:0/289438432 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72e41f6f90 con 0x7f72e40cf4a0 2023-10-16T13:48:32.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:32.822+0000 7f72d2ffd700 1 -- 172.21.15.16:0/289438432 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72d40038c0 con 0x7f72e40cf4a0 2023-10-16T13:48:32.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:32.822+0000 7f72e8da1700 1 -- 172.21.15.16:0/289438432 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72e4154e70 con 0x7f72e40cf4a0 2023-10-16T13:48:32.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:32.823+0000 7f72e8da1700 1 -- 172.21.15.16:0/289438432 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72e41f7750 con 0x7f72e40cf4a0 2023-10-16T13:48:32.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:32.824+0000 7f72d2ffd700 1 -- 172.21.15.16:0/289438432 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f72d4003a20 con 0x7f72e40cf4a0 2023-10-16T13:48:32.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:32.824+0000 7f72d2ffd700 1 --2- 172.21.15.16:0/289438432 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f72c406b140 0x7f72c406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:32.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:32.824+0000 7f72d2ffd700 1 -- 172.21.15.16:0/289438432 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f72d4088d80 con 0x7f72e40cf4a0 2023-10-16T13:48:32.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:32.824+0000 7f72e259c700 1 --2- 172.21.15.16:0/289438432 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f72c406b140 0x7f72c406d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:32.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:32.825+0000 7f72e259c700 1 --2- 172.21.15.16:0/289438432 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f72c406b140 0x7f72c406d600 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f72d8009c40 tx=0x7f72d8008030 comp rx=0 tx=0).ready entity=mgr.12678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:32.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:32.829+0000 7f72d2ffd700 1 -- 172.21.15.16:0/289438432 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f72d4053a70 con 0x7f72e40cf4a0 2023-10-16T13:48:33.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:33.061+0000 7f72e8da1700 1 -- 172.21.15.16:0/289438432 --> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f72e415b630 con 0x7f72c406b140 2023-10-16T13:48:33.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:33.063+0000 7f72d2ffd700 1 -- 172.21.15.16:0/289438432 <== mgr.12678 v2:172.21.15.16:6832/1275231846 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f72d800bbf0 con 0x7f72c406b140 2023-10-16T13:48:33.065 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2023-10-16T13:48:33.065 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:48:33.066 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cephfs" 2023-10-16T13:48:33.066 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:48:33.066 INFO:teuthology.orchestra.run.smithi016.stdout:] 2023-10-16T13:48:33.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:33.065+0000 7f72e8da1700 1 -- 172.21.15.16:0/289438432 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f72c406b140 msgr2=0x7f72c406d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:33.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:33.065+0000 7f72e8da1700 1 --2- 172.21.15.16:0/289438432 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f72c406b140 0x7f72c406d600 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f72d8009c40 tx=0x7f72d8008030 comp rx=0 tx=0).stop 2023-10-16T13:48:33.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:33.065+0000 7f72e8da1700 1 -- 172.21.15.16:0/289438432 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72e40cf4a0 msgr2=0x7f72e40c4390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:33.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:33.065+0000 7f72e8da1700 1 --2- 172.21.15.16:0/289438432 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72e40cf4a0 0x7f72e40c4390 secure :-1 s=READY pgs=3033 cs=0 l=1 rev1=1 crypto rx=0x7f72d400c450 tx=0x7f72d400c760 comp rx=0 tx=0).stop 2023-10-16T13:48:33.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:33.065+0000 7f72e8da1700 1 -- 172.21.15.16:0/289438432 shutdown_connections 2023-10-16T13:48:33.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:33.065+0000 7f72e8da1700 1 --2- 172.21.15.16:0/289438432 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f72c406b140 0x7f72c406d600 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:33.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:33.065+0000 7f72e8da1700 1 --2- 172.21.15.16:0/289438432 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f72e40cf880 0x7f72e406a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:33.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:33.065+0000 7f72e8da1700 1 --2- 172.21.15.16:0/289438432 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f72e40b5a00 0x7f72e414b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:33.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:33.065+0000 7f72e8da1700 1 --2- 172.21.15.16:0/289438432 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72e40cf4a0 0x7f72e40c4390 unknown :-1 s=CLOSED pgs=3033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:33.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:33.065+0000 7f72e8da1700 1 -- 172.21.15.16:0/289438432 >> 172.21.15.16:0/289438432 conn(0x7f72e40b1fd0 msgr2=0x7f72e40cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:33.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:33.065+0000 7f72e8da1700 1 -- 172.21.15.16:0/289438432 shutdown_connections 2023-10-16T13:48:33.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:33.065+0000 7f72e8da1700 1 -- 172.21.15.16:0/289438432 wait complete. 2023-10-16T13:48:33.079 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-16T13:48:33.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:33.204+0000 7f2bdb715700 1 Processor -- start 2023-10-16T13:48:33.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:33.204+0000 7f2bdb715700 1 -- start start 2023-10-16T13:48:33.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:33.205+0000 7f2bdb715700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2bd415b070 0x7f2bd415b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:33.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:33.205+0000 7f2bdb715700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2bd40cb880 0x7f2bd40cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:33.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:33.205+0000 7f2bdb715700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bd40cef70 con 0x7f2bd406da60 2023-10-16T13:48:33.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:33.205+0000 7f2bdb715700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bd40ce360 con 0x7f2bd40cb880 2023-10-16T13:48:33.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:33.205+0000 7f2bdb715700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd40ce640 con 0x7f2bd415b070 2023-10-16T13:48:33.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:33.205+0000 7f2bd8cb0700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2bd40cb880 0x7f2bd40cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:33.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:33.205+0000 7f2bd8cb0700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2bd40cb880 0x7f2bd40cbd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57704/0 (socket says 172.21.15.16:57704) 2023-10-16T13:48:33.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:33.205+0000 7f2bd8cb0700 1 -- 172.21.15.16:0/1625335640 learned_addr learned my addr 172.21.15.16:0/1625335640 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:33.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:33.205+0000 7f2bd9cb2700 1 --1- 172.21.15.16:0/1625335640 >> v1:172.21.15.16:6789/0 conn(0x7f2bd406da60 0x7f2bd40cf400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46982/0 (socket says 172.21.15.16:46982) 2023-10-16T13:48:33.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:33.206+0000 7f2bd94b1700 1 --2- 172.21.15.16:0/1625335640 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2bd415b070 0x7f2bd415b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:33.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:33.206+0000 7f2bd8cb0700 1 -- 172.21.15.16:0/1625335640 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2bd415b070 msgr2=0x7f2bd415b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:33.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:33.206+0000 7f2bd8cb0700 1 --2- 172.21.15.16:0/1625335640 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2bd415b070 0x7f2bd415b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:33.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:33.206+0000 7f2bd8cb0700 1 -- 172.21.15.16:0/1625335640 >> v1:172.21.15.16:6789/0 conn(0x7f2bd406da60 legacy=0x7f2bd40cf400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:33.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:33.206+0000 7f2bd8cb0700 1 -- 172.21.15.16:0/1625335640 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bd40ce920 con 0x7f2bd40cb880 2023-10-16T13:48:33.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:33.206+0000 7f2bd8cb0700 1 --2- 172.21.15.16:0/1625335640 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2bd40cb880 0x7f2bd40cbd00 secure :-1 s=READY pgs=3035 cs=0 l=1 rev1=1 crypto rx=0x7f2bd00098f0 tx=0x7f2bd0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28ca6d29606077a3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:33.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:33.206+0000 7f2bcbfff700 1 -- 172.21.15.16:0/1625335640 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bd000b8a0 con 0x7f2bd40cb880 2023-10-16T13:48:33.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:33.206+0000 7f2bcbfff700 1 -- 172.21.15.16:0/1625335640 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bd000ba00 con 0x7f2bd40cb880 2023-10-16T13:48:33.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:33.206+0000 7f2bcbfff700 1 -- 172.21.15.16:0/1625335640 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bd000bd90 con 0x7f2bd40cb880 2023-10-16T13:48:33.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:33.207+0000 7f2bdb715700 1 -- 172.21.15.16:0/1625335640 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2bd40cb880 msgr2=0x7f2bd40cbd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:33.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:33.207+0000 7f2bdb715700 1 --2- 172.21.15.16:0/1625335640 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2bd40cb880 0x7f2bd40cbd00 secure :-1 s=READY pgs=3035 cs=0 l=1 rev1=1 crypto rx=0x7f2bd00098f0 tx=0x7f2bd0009c00 comp rx=0 tx=0).stop 2023-10-16T13:48:33.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:33.207+0000 7f2bdb715700 1 -- 172.21.15.16:0/1625335640 shutdown_connections 2023-10-16T13:48:33.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:33.207+0000 7f2bdb715700 1 --2- 172.21.15.16:0/1625335640 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2bd415b070 0x7f2bd415b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:33.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:33.207+0000 7f2bdb715700 1 --2- 172.21.15.16:0/1625335640 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2bd40cb880 0x7f2bd40cbd00 unknown :-1 s=CLOSED pgs=3035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:33.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:33.207+0000 7f2bdb715700 1 -- 172.21.15.16:0/1625335640 >> 172.21.15.16:0/1625335640 conn(0x7f2bd40b1f40 msgr2=0x7f2bd40b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:33.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:33.207+0000 7f2bdb715700 1 -- 172.21.15.16:0/1625335640 shutdown_connections 2023-10-16T13:48:33.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:33.207+0000 7f2bdb715700 1 -- 172.21.15.16:0/1625335640 wait complete. 2023-10-16T13:48:33.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:33.208+0000 7f2bdb715700 1 Processor -- start 2023-10-16T13:48:33.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:33.208+0000 7f2bdb715700 1 -- start start 2023-10-16T13:48:33.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:33.209+0000 7f2bdb715700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2bd406da60 0x7f2bd40c9680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:33.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:33.209+0000 7f2bdb715700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2bd40cb880 0x7f2bd40c6050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:33.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:33.209+0000 7f2bdb715700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2bd415b070 0x7f2bd40c74c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:33.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:33.209+0000 7f2bdb715700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd40c6590 con 0x7f2bd406da60 2023-10-16T13:48:33.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:33.209+0000 7f2bdb715700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd40c7a00 con 0x7f2bd415b070 2023-10-16T13:48:33.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:33.209+0000 7f2bdb715700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd40c7ca0 con 0x7f2bd40cb880 2023-10-16T13:48:33.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:33.209+0000 7f2bd94b1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2bd406da60 0x7f2bd40c9680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:33.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:33.209+0000 7f2bd94b1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2bd406da60 0x7f2bd40c9680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57710/0 (socket says 172.21.15.16:57710) 2023-10-16T13:48:33.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:33.209+0000 7f2bd94b1700 1 -- 172.21.15.16:0/770508609 learned_addr learned my addr 172.21.15.16:0/770508609 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:33.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:33.209+0000 7f2bd8cb0700 1 --2- 172.21.15.16:0/770508609 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2bd40cb880 0x7f2bd40c6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:33.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:33.209+0000 7f2bd9cb2700 1 --2- 172.21.15.16:0/770508609 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2bd415b070 0x7f2bd40c74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:33.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:33.209+0000 7f2bd94b1700 1 -- 172.21.15.16:0/770508609 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2bd40cb880 msgr2=0x7f2bd40c6050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:33.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:33.209+0000 7f2bd94b1700 1 --2- 172.21.15.16:0/770508609 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2bd40cb880 0x7f2bd40c6050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:33.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:33.209+0000 7f2bd94b1700 1 -- 172.21.15.16:0/770508609 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2bd415b070 msgr2=0x7f2bd40c74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:33.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:33.209+0000 7f2bd94b1700 1 --2- 172.21.15.16:0/770508609 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2bd415b070 0x7f2bd40c74c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:33.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:33.209+0000 7f2bd94b1700 1 -- 172.21.15.16:0/770508609 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bd0009550 con 0x7f2bd406da60 2023-10-16T13:48:33.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:33.209+0000 7f2bd9cb2700 1 --2- 172.21.15.16:0/770508609 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2bd415b070 0x7f2bd40c74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:48:33.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:33.209+0000 7f2bd94b1700 1 --2- 172.21.15.16:0/770508609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2bd406da60 0x7f2bd40c9680 secure :-1 s=READY pgs=3036 cs=0 l=1 rev1=1 crypto rx=0x7f2bc400faa0 tx=0x7f2bc400fdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:33.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:33.209+0000 7f2bc9ffb700 1 -- 172.21.15.16:0/770508609 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc4007ae0 con 0x7f2bd406da60 2023-10-16T13:48:33.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:33.209+0000 7f2bc9ffb700 1 -- 172.21.15.16:0/770508609 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bc4007c40 con 0x7f2bd406da60 2023-10-16T13:48:33.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:33.209+0000 7f2bdb715700 1 -- 172.21.15.16:0/770508609 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bd40c7f80 con 0x7f2bd406da60 2023-10-16T13:48:33.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:33.210+0000 7f2bc9ffb700 1 -- 172.21.15.16:0/770508609 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc401a460 con 0x7f2bd406da60 2023-10-16T13:48:33.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:33.210+0000 7f2bdb715700 1 -- 172.21.15.16:0/770508609 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bd40c8540 con 0x7f2bd406da60 2023-10-16T13:48:33.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:33.211+0000 7f2bdb715700 1 -- 172.21.15.16:0/770508609 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bd404ebb0 con 0x7f2bd406da60 2023-10-16T13:48:33.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:33.211+0000 7f2bc9ffb700 1 -- 172.21.15.16:0/770508609 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 157) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f2bc4011450 con 0x7f2bd406da60 2023-10-16T13:48:33.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:33.211+0000 7f2bc9ffb700 1 --2- 172.21.15.16:0/770508609 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f2bb806b140 0x7f2bb806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:33.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:33.212+0000 7f2bc9ffb700 1 -- 172.21.15.16:0/770508609 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f2bc408c3c0 con 0x7f2bd406da60 2023-10-16T13:48:33.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:33.212+0000 7f2bd8cb0700 1 --2- 172.21.15.16:0/770508609 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f2bb806b140 0x7f2bb806d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:33.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:33.212+0000 7f2bd8cb0700 1 --2- 172.21.15.16:0/770508609 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f2bb806b140 0x7f2bb806d600 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f2bd0009c40 tx=0x7f2bd0008030 comp rx=0 tx=0).ready entity=mgr.12678 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:33.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:33.216+0000 7f2bc9ffb700 1 -- 172.21.15.16:0/770508609 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2bc4056df0 con 0x7f2bd406da60 2023-10-16T13:48:33.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:33.481+0000 7f2bdb715700 1 -- 172.21.15.16:0/770508609 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f2bd4158480 con 0x7f2bd406da60 2023-10-16T13:48:33.848 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:48:33.851 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:48:34.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.483+0000 7f2bc9ffb700 1 -- 172.21.15.16:0/770508609 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v158) v1 ==== 92+0+0 (secure 0 0 0) 0x7f2bc405a410 con 0x7f2bd406da60 2023-10-16T13:48:34.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.489+0000 7f2bdb715700 1 -- 172.21.15.16:0/770508609 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f2bb806b140 msgr2=0x7f2bb806d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:34.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.489+0000 7f2bdb715700 1 --2- 172.21.15.16:0/770508609 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f2bb806b140 0x7f2bb806d600 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f2bd0009c40 tx=0x7f2bd0008030 comp rx=0 tx=0).stop 2023-10-16T13:48:34.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.489+0000 7f2bdb715700 1 -- 172.21.15.16:0/770508609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2bd406da60 msgr2=0x7f2bd40c9680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:34.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.489+0000 7f2bc9ffb700 1 -- 172.21.15.16:0/770508609 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mgrmap(e 158) v1 ==== 89450+0+0 (secure 0 0 0) 0x7f2bc40571c0 con 0x7f2bd406da60 2023-10-16T13:48:34.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.489+0000 7f2bdb715700 1 --2- 172.21.15.16:0/770508609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2bd406da60 0x7f2bd40c9680 secure :-1 s=READY pgs=3036 cs=0 l=1 rev1=1 crypto rx=0x7f2bc400faa0 tx=0x7f2bc400fdb0 comp rx=0 tx=0).stop 2023-10-16T13:48:34.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.489+0000 7f2bdb715700 1 -- 172.21.15.16:0/770508609 shutdown_connections 2023-10-16T13:48:34.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.489+0000 7f2bdb715700 1 --2- 172.21.15.16:0/770508609 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f2bb806b140 0x7f2bb806d600 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:34.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.490+0000 7f2bdb715700 1 --2- 172.21.15.16:0/770508609 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2bd415b070 0x7f2bd40c74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:34.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.490+0000 7f2bdb715700 1 --2- 172.21.15.16:0/770508609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2bd406da60 0x7f2bd40c9680 unknown :-1 s=CLOSED pgs=3036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:34.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.490+0000 7f2bdb715700 1 --2- 172.21.15.16:0/770508609 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2bd40cb880 0x7f2bd40c6050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:34.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.490+0000 7f2bdb715700 1 -- 172.21.15.16:0/770508609 >> 172.21.15.16:0/770508609 conn(0x7f2bd40b1f40 msgr2=0x7f2bd40cd150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:34.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.490+0000 7f2bc9ffb700 1 --2- 172.21.15.16:0/770508609 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f2bb80368c0 0x7f2bb806d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:34.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.490+0000 7f2bdb715700 1 -- 172.21.15.16:0/770508609 shutdown_connections 2023-10-16T13:48:34.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.490+0000 7f2bdb715700 1 --2- 172.21.15.16:0/770508609 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f2bb80368c0 0x7f2bb806d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:34.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.490+0000 7f2bdb715700 1 -- 172.21.15.16:0/770508609 wait complete. 2023-10-16T13:48:34.508 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-16T13:48:34.638 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:48:34.636+0000 7f78fc93e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T13:48:34.646 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:34.644+0000 7f648ebc9200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T13:48:34.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.644+0000 7f0982937700 1 Processor -- start 2023-10-16T13:48:34.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.645+0000 7f0982937700 1 -- start start 2023-10-16T13:48:34.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.645+0000 7f0982937700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f097c15b070 0x7f097c15b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:34.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.645+0000 7f0982937700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f097c0cb880 0x7f097c0cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:34.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.645+0000 7f0982937700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f097c0cef70 con 0x7f097c06da60 2023-10-16T13:48:34.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.645+0000 7f0982937700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f097c0ce360 con 0x7f097c15b070 2023-10-16T13:48:34.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.645+0000 7f0982937700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f097c0ce640 con 0x7f097c0cb880 2023-10-16T13:48:34.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.645+0000 7f0980ed4700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f097c06da60 0x7f097c0cf400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:47024/0 (socket says 172.21.15.16:47024) 2023-10-16T13:48:34.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.646+0000 7f0980ed4700 1 -- 172.21.15.16:0/241431288 learned_addr learned my addr 172.21.15.16:0/241431288 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:48:34.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.646+0000 7f097bfff700 1 --2- 172.21.15.16:0/241431288 >> v2:172.21.15.49:3300/0 conn(0x7f097c15b070 0x7f097c15b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:34.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.646+0000 7f097b7fe700 1 --2- 172.21.15.16:0/241431288 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f097c0cb880 0x7f097c0cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:34.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.646+0000 7f097affd700 1 -- 172.21.15.16:0/241431288 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1290413152 0 0) 0x7f097c0cef70 con 0x7f097c06da60 2023-10-16T13:48:34.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.646+0000 7f097affd700 1 -- 172.21.15.16:0/241431288 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09640057c0 con 0x7f097c06da60 2023-10-16T13:48:34.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.646+0000 7f097bfff700 1 -- 172.21.15.16:0/241431288 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f097c0cb880 msgr2=0x7f097c0cbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:34.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.646+0000 7f097bfff700 1 --2- 172.21.15.16:0/241431288 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f097c0cb880 0x7f097c0cbd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:34.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.646+0000 7f097bfff700 1 -- 172.21.15.16:0/241431288 >> v1:172.21.15.16:6789/0 conn(0x7f097c06da60 legacy=0x7f097c0cf400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:34.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.646+0000 7f097bfff700 1 -- 172.21.15.16:0/241431288 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f097c0ce920 con 0x7f097c15b070 2023-10-16T13:48:34.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.646+0000 7f097bfff700 1 --2- 172.21.15.16:0/241431288 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f097c15b070 0x7f097c15b450 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f096c009840 tx=0x7f096c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=feb26ad0ec3faace server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:34.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.646+0000 7f097affd700 1 -- 172.21.15.16:0/241431288 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f096c011830 con 0x7f097c15b070 2023-10-16T13:48:34.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.646+0000 7f097affd700 1 -- 172.21.15.16:0/241431288 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f096c011990 con 0x7f097c15b070 2023-10-16T13:48:34.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.646+0000 7f097affd700 1 -- 172.21.15.16:0/241431288 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f096c011d20 con 0x7f097c15b070 2023-10-16T13:48:34.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.647+0000 7f0982937700 1 -- 172.21.15.16:0/241431288 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f097c15b070 msgr2=0x7f097c15b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:34.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.647+0000 7f0982937700 1 --2- 172.21.15.16:0/241431288 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f097c15b070 0x7f097c15b450 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f096c009840 tx=0x7f096c009b50 comp rx=0 tx=0).stop 2023-10-16T13:48:34.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.647+0000 7f0982937700 1 -- 172.21.15.16:0/241431288 shutdown_connections 2023-10-16T13:48:34.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.647+0000 7f0982937700 1 --2- 172.21.15.16:0/241431288 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f097c15b070 0x7f097c15b450 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:34.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.647+0000 7f0982937700 1 --2- 172.21.15.16:0/241431288 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f097c0cb880 0x7f097c0cbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:34.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.647+0000 7f0982937700 1 -- 172.21.15.16:0/241431288 >> 172.21.15.16:0/241431288 conn(0x7f097c0b1f40 msgr2=0x7f097c0b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:34.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.647+0000 7f0982937700 1 -- 172.21.15.16:0/241431288 shutdown_connections 2023-10-16T13:48:34.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.647+0000 7f0982937700 1 -- 172.21.15.16:0/241431288 wait complete. 2023-10-16T13:48:34.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.648+0000 7f0982937700 1 Processor -- start 2023-10-16T13:48:34.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.649+0000 7f0982937700 1 -- start start 2023-10-16T13:48:34.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.649+0000 7f0982937700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f097c06da60 0x7f097c0b43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:34.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.649+0000 7f0982937700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f097c0cb880 0x7f097c0c6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:34.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.649+0000 7f0982937700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f097c15b070 0x7f097c153960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:34.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.649+0000 7f0982937700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f097c0c65b0 con 0x7f097c15b070 2023-10-16T13:48:34.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.649+0000 7f0982937700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f097c0b3f20 con 0x7f097c06da60 2023-10-16T13:48:34.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.649+0000 7f0982937700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f097c0b3030 con 0x7f097c0cb880 2023-10-16T13:48:34.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.649+0000 7f0980ed4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f097c15b070 0x7f097c153960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:34.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.650+0000 7f0980ed4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f097c15b070 0x7f097c153960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57756/0 (socket says 172.21.15.16:57756) 2023-10-16T13:48:34.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.650+0000 7f0980ed4700 1 -- 172.21.15.16:0/1242807523 learned_addr learned my addr 172.21.15.16:0/1242807523 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:34.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.650+0000 7f097bfff700 1 --2- 172.21.15.16:0/1242807523 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f097c06da60 0x7f097c0b43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:34.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.650+0000 7f097b7fe700 1 --2- 172.21.15.16:0/1242807523 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f097c0cb880 0x7f097c0c6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:34.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.650+0000 7f0980ed4700 1 -- 172.21.15.16:0/1242807523 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f097c0cb880 msgr2=0x7f097c0c6070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:34.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.650+0000 7f0980ed4700 1 --2- 172.21.15.16:0/1242807523 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f097c0cb880 0x7f097c0c6070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:34.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.650+0000 7f0980ed4700 1 -- 172.21.15.16:0/1242807523 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f097c06da60 msgr2=0x7f097c0b43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:34.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.650+0000 7f0980ed4700 1 --2- 172.21.15.16:0/1242807523 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f097c06da60 0x7f097c0b43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:34.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.650+0000 7f0980ed4700 1 -- 172.21.15.16:0/1242807523 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f096c0094a0 con 0x7f097c15b070 2023-10-16T13:48:34.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.650+0000 7f097bfff700 1 --2- 172.21.15.16:0/1242807523 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f097c06da60 0x7f097c0b43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:48:34.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.650+0000 7f0980ed4700 1 --2- 172.21.15.16:0/1242807523 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f097c15b070 0x7f097c153960 secure :-1 s=READY pgs=3039 cs=0 l=1 rev1=1 crypto rx=0x7f097400cb40 tx=0x7f097400ce50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:34.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.650+0000 7f0978ff9700 1 -- 172.21.15.16:0/1242807523 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f097400a9e0 con 0x7f097c15b070 2023-10-16T13:48:34.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.650+0000 7f0978ff9700 1 -- 172.21.15.16:0/1242807523 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f097400ab40 con 0x7f097c15b070 2023-10-16T13:48:34.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.650+0000 7f0978ff9700 1 -- 172.21.15.16:0/1242807523 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0974012400 con 0x7f097c15b070 2023-10-16T13:48:34.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.650+0000 7f0982937700 1 -- 172.21.15.16:0/1242807523 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f097c0b3310 con 0x7f097c15b070 2023-10-16T13:48:34.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.650+0000 7f0982937700 1 -- 172.21.15.16:0/1242807523 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f097c0b38d0 con 0x7f097c15b070 2023-10-16T13:48:34.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.651+0000 7f0982937700 1 -- 172.21.15.16:0/1242807523 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f097c04ebb0 con 0x7f097c15b070 2023-10-16T13:48:34.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.652+0000 7f0978ff9700 1 -- 172.21.15.16:0/1242807523 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 158) v1 ==== 89450+0+0 (secure 0 0 0) 0x7f097401b3b0 con 0x7f097c15b070 2023-10-16T13:48:34.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.652+0000 7f0978ff9700 1 --2- 172.21.15.16:0/1242807523 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f096806b190 0x7f096806d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:34.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.652+0000 7f097bfff700 1 -- 172.21.15.16:0/1242807523 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f096806b190 msgr2=0x7f096806d650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1275231846 2023-10-16T13:48:34.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.652+0000 7f0978ff9700 1 -- 172.21.15.16:0/1242807523 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f09740898a0 con 0x7f097c15b070 2023-10-16T13:48:34.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.652+0000 7f097bfff700 1 --2- 172.21.15.16:0/1242807523 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f096806b190 0x7f096806d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:48:34.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.657+0000 7f0978ff9700 1 -- 172.21.15.16:0/1242807523 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f09740543b0 con 0x7f097c15b070 2023-10-16T13:48:34.707 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:48:34.705+0000 7f78fc93e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T13:48:34.718 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:34.716+0000 7f648ebc9200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T13:48:34.782 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:48:34.780+0000 7f78fc93e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T13:48:34.794 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:34.792+0000 7f648ebc9200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T13:48:34.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.853+0000 7f097bfff700 1 -- 172.21.15.16:0/1242807523 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f096806b190 msgr2=0x7f096806d650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1275231846 2023-10-16T13:48:34.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.853+0000 7f097bfff700 1 --2- 172.21.15.16:0/1242807523 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f096806b190 0x7f096806d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:48:34.864 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:48:34.861+0000 7f78fc93e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T13:48:34.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.866+0000 7f0982937700 1 -- 172.21.15.16:0/1242807523 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f097c0b4060 con 0x7f097c15b070 2023-10-16T13:48:34.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.870+0000 7f0978ff9700 1 -- 172.21.15.16:0/1242807523 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v89)=0 v89) v1 ==== 138+0+0 (secure 0 0 0) 0x7f09740170d0 con 0x7f097c15b070 2023-10-16T13:48:34.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.872+0000 7f0982937700 1 -- 172.21.15.16:0/1242807523 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f096806b190 msgr2=0x7f096806d650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:48:34.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.872+0000 7f0982937700 1 --2- 172.21.15.16:0/1242807523 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f096806b190 0x7f096806d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:34.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.872+0000 7f0982937700 1 -- 172.21.15.16:0/1242807523 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f097c15b070 msgr2=0x7f097c153960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:34.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.872+0000 7f0982937700 1 --2- 172.21.15.16:0/1242807523 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f097c15b070 0x7f097c153960 secure :-1 s=READY pgs=3039 cs=0 l=1 rev1=1 crypto rx=0x7f097400cb40 tx=0x7f097400ce50 comp rx=0 tx=0).stop 2023-10-16T13:48:34.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.872+0000 7f0982937700 1 -- 172.21.15.16:0/1242807523 shutdown_connections 2023-10-16T13:48:34.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.872+0000 7f0982937700 1 --2- 172.21.15.16:0/1242807523 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f096806b190 0x7f096806d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:34.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.872+0000 7f0982937700 1 --2- 172.21.15.16:0/1242807523 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f097c15b070 0x7f097c153960 unknown :-1 s=CLOSED pgs=3039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:34.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.872+0000 7f0982937700 1 --2- 172.21.15.16:0/1242807523 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f097c06da60 0x7f097c0b43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:34.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.872+0000 7f0982937700 1 --2- 172.21.15.16:0/1242807523 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f097c0cb880 0x7f097c0c6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:34.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.872+0000 7f0982937700 1 -- 172.21.15.16:0/1242807523 >> 172.21.15.16:0/1242807523 conn(0x7f097c0b1f40 msgr2=0x7f097c0cc4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:34.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.873+0000 7f0982937700 1 -- 172.21.15.16:0/1242807523 shutdown_connections 2023-10-16T13:48:34.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:34.873+0000 7f0982937700 1 -- 172.21.15.16:0/1242807523 wait complete. 2023-10-16T13:48:34.879 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:34.876+0000 7f648ebc9200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T13:48:34.888 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-16T13:48:35.012 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:48:35.010+0000 7f78fc93e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T13:48:35.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.015+0000 7f9d7242f700 1 Processor -- start 2023-10-16T13:48:35.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.015+0000 7f9d7242f700 1 -- start start 2023-10-16T13:48:35.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.016+0000 7f9d7242f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9d6c15b2c0 0x7f9d6c161340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:35.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.016+0000 7f9d7242f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9d6c06e570 0x7f9d6c15e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:35.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.016+0000 7f9d7242f700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f9d6c06a7e0 0x7f9d6c06ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:35.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.016+0000 7f9d7242f700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d6c06b1b0 con 0x7f9d6c15b2c0 2023-10-16T13:48:35.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.016+0000 7f9d7242f700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d6c161880 con 0x7f9d6c06a7e0 2023-10-16T13:48:35.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.016+0000 7f9d7242f700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d6c15b6a0 con 0x7f9d6c06e570 2023-10-16T13:48:35.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.016+0000 7f9d6bfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9d6c15b2c0 0x7f9d6c161340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:35.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.016+0000 7f9d6bfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9d6c15b2c0 0x7f9d6c161340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57806/0 (socket says 172.21.15.16:57806) 2023-10-16T13:48:35.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.016+0000 7f9d6bfff700 1 -- 172.21.15.16:0/3932428734 learned_addr learned my addr 172.21.15.16:0/3932428734 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:35.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.016+0000 7f9d6b7fe700 1 --2- 172.21.15.16:0/3932428734 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9d6c06e570 0x7f9d6c15e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:35.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.016+0000 7f9d709cc700 1 --2- 172.21.15.16:0/3932428734 >> v2:172.21.15.49:3300/0 conn(0x7f9d6c06a7e0 0x7f9d6c06ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:35.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.016+0000 7f9d6bfff700 1 -- 172.21.15.16:0/3932428734 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9d6c06e570 msgr2=0x7f9d6c15e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:35.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.016+0000 7f9d6bfff700 1 --2- 172.21.15.16:0/3932428734 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9d6c06e570 0x7f9d6c15e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:35.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.016+0000 7f9d6bfff700 1 -- 172.21.15.16:0/3932428734 >> v2:172.21.15.49:3300/0 conn(0x7f9d6c06a7e0 msgr2=0x7f9d6c06ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:35.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.016+0000 7f9d6bfff700 1 --2- 172.21.15.16:0/3932428734 >> v2:172.21.15.49:3300/0 conn(0x7f9d6c06a7e0 0x7f9d6c06ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:35.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.016+0000 7f9d6bfff700 1 -- 172.21.15.16:0/3932428734 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d6c161b20 con 0x7f9d6c15b2c0 2023-10-16T13:48:35.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.016+0000 7f9d6bfff700 1 --2- 172.21.15.16:0/3932428734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d6c15b2c0 0x7f9d6c161340 secure :-1 s=READY pgs=3040 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c0ca970 tx=0x7f9d5c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=391eddd64e71a165 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:35.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.017+0000 7f9d6affd700 1 -- 172.21.15.16:0/3932428734 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d5c011830 con 0x7f9d6c15b2c0 2023-10-16T13:48:35.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.017+0000 7f9d6affd700 1 -- 172.21.15.16:0/3932428734 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d5c011990 con 0x7f9d6c15b2c0 2023-10-16T13:48:35.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.017+0000 7f9d6affd700 1 -- 172.21.15.16:0/3932428734 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d5c011d20 con 0x7f9d6c15b2c0 2023-10-16T13:48:35.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.017+0000 7f9d7242f700 1 -- 172.21.15.16:0/3932428734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d6c15b2c0 msgr2=0x7f9d6c161340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:35.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.017+0000 7f9d7242f700 1 --2- 172.21.15.16:0/3932428734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d6c15b2c0 0x7f9d6c161340 secure :-1 s=READY pgs=3040 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c0ca970 tx=0x7f9d5c009bd0 comp rx=0 tx=0).stop 2023-10-16T13:48:35.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.017+0000 7f9d7242f700 1 -- 172.21.15.16:0/3932428734 shutdown_connections 2023-10-16T13:48:35.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.017+0000 7f9d7242f700 1 --2- 172.21.15.16:0/3932428734 >> v2:172.21.15.49:3300/0 conn(0x7f9d6c06a7e0 0x7f9d6c06ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:35.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.017+0000 7f9d7242f700 1 --2- 172.21.15.16:0/3932428734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d6c15b2c0 0x7f9d6c161340 unknown :-1 s=CLOSED pgs=3040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:35.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.017+0000 7f9d7242f700 1 --2- 172.21.15.16:0/3932428734 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9d6c06e570 0x7f9d6c15e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:35.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.017+0000 7f9d7242f700 1 -- 172.21.15.16:0/3932428734 >> 172.21.15.16:0/3932428734 conn(0x7f9d6c0b1f40 msgr2=0x7f9d6c0b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:35.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.017+0000 7f9d7242f700 1 -- 172.21.15.16:0/3932428734 shutdown_connections 2023-10-16T13:48:35.025 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.017+0000 7f9d7242f700 1 -- 172.21.15.16:0/3932428734 wait complete. 2023-10-16T13:48:35.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.018+0000 7f9d7242f700 1 Processor -- start 2023-10-16T13:48:35.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.019+0000 7f9d7242f700 1 -- start start 2023-10-16T13:48:35.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.019+0000 7f9d7242f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d6c06a7e0 0x7f9d6c0c74c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:35.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.019+0000 7f9d7242f700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9d6c06e570 0x7f9d6c15f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:35.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.019+0000 7f9d7242f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9d6c15b2c0 0x7f9d6c15f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:35.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.019+0000 7f9d7242f700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d6c0b4a50 con 0x7f9d6c06a7e0 2023-10-16T13:48:35.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.019+0000 7f9d7242f700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d6c0c7b30 con 0x7f9d6c06e570 2023-10-16T13:48:35.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.019+0000 7f9d7242f700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d6c0b4400 con 0x7f9d6c15b2c0 2023-10-16T13:48:35.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.019+0000 7f9d6bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d6c06a7e0 0x7f9d6c0c74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:35.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.019+0000 7f9d6bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d6c06a7e0 0x7f9d6c0c74c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57812/0 (socket says 172.21.15.16:57812) 2023-10-16T13:48:35.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.019+0000 7f9d6bfff700 1 -- 172.21.15.16:0/3212233418 learned_addr learned my addr 172.21.15.16:0/3212233418 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:35.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.020+0000 7f9d6b7fe700 1 --2- 172.21.15.16:0/3212233418 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9d6c06e570 0x7f9d6c15f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:35.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.020+0000 7f9d6bfff700 1 -- 172.21.15.16:0/3212233418 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9d6c15b2c0 msgr2=0x7f9d6c15f5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-16T13:48:35.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.020+0000 7f9d6bfff700 1 --2- 172.21.15.16:0/3212233418 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9d6c15b2c0 0x7f9d6c15f5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:35.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.020+0000 7f9d6bfff700 1 -- 172.21.15.16:0/3212233418 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9d6c06e570 msgr2=0x7f9d6c15f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:35.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.020+0000 7f9d6bfff700 1 --2- 172.21.15.16:0/3212233418 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9d6c06e570 0x7f9d6c15f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:35.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.020+0000 7f9d6bfff700 1 -- 172.21.15.16:0/3212233418 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d5c009550 con 0x7f9d6c06a7e0 2023-10-16T13:48:35.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.020+0000 7f9d6bfff700 1 --2- 172.21.15.16:0/3212233418 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d6c06a7e0 0x7f9d6c0c74c0 secure :-1 s=READY pgs=3041 cs=0 l=1 rev1=1 crypto rx=0x7f9d5c000c00 tx=0x7f9d5c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:35.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.020+0000 7f9d68ff9700 1 -- 172.21.15.16:0/3212233418 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d5c004240 con 0x7f9d6c06a7e0 2023-10-16T13:48:35.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.020+0000 7f9d68ff9700 1 -- 172.21.15.16:0/3212233418 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d5c01f070 con 0x7f9d6c06a7e0 2023-10-16T13:48:35.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.020+0000 7f9d68ff9700 1 -- 172.21.15.16:0/3212233418 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d5c0036a0 con 0x7f9d6c06a7e0 2023-10-16T13:48:35.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.020+0000 7f9d7242f700 1 -- 172.21.15.16:0/3212233418 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d6c0b46e0 con 0x7f9d6c06a7e0 2023-10-16T13:48:35.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.020+0000 7f9d7242f700 1 -- 172.21.15.16:0/3212233418 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d6c1f2eb0 con 0x7f9d6c06a7e0 2023-10-16T13:48:35.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.021+0000 7f9d7242f700 1 -- 172.21.15.16:0/3212233418 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d6c04ebb0 con 0x7f9d6c06a7e0 2023-10-16T13:48:35.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.022+0000 7f9d68ff9700 1 -- 172.21.15.16:0/3212233418 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 158) v1 ==== 89450+0+0 (secure 0 0 0) 0x7f9d5c021030 con 0x7f9d6c06a7e0 2023-10-16T13:48:35.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.022+0000 7f9d68ff9700 1 --2- 172.21.15.16:0/3212233418 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f9d5806b190 0x7f9d5806d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:35.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.022+0000 7f9d6b7fe700 1 -- 172.21.15.16:0/3212233418 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f9d5806b190 msgr2=0x7f9d5806d650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1275231846 2023-10-16T13:48:35.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.022+0000 7f9d68ff9700 1 -- 172.21.15.16:0/3212233418 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f9d5c08b0a0 con 0x7f9d6c06a7e0 2023-10-16T13:48:35.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.022+0000 7f9d6b7fe700 1 --2- 172.21.15.16:0/3212233418 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f9d5806b190 0x7f9d5806d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:48:35.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.030+0000 7f9d68ff9700 1 -- 172.21.15.16:0/3212233418 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9d5c055da0 con 0x7f9d6c06a7e0 2023-10-16T13:48:35.036 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:35.034+0000 7f648ebc9200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T13:48:35.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.223+0000 7f9d6b7fe700 1 -- 172.21.15.16:0/3212233418 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f9d5806b190 msgr2=0x7f9d5806d650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1275231846 2023-10-16T13:48:35.225 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.223+0000 7f9d6b7fe700 1 --2- 172.21.15.16:0/3212233418 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f9d5806b190 0x7f9d5806d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:48:35.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.236+0000 7f9d7242f700 1 -- 172.21.15.16:0/3212233418 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f9d6c0b6370 con 0x7f9d6c06a7e0 2023-10-16T13:48:35.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.239+0000 7f9d68ff9700 1 -- 172.21.15.16:0/3212233418 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v90)=0 v90) v1 ==== 133+0+0 (secure 0 0 0) 0x7f9d5c015020 con 0x7f9d6c06a7e0 2023-10-16T13:48:35.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.241+0000 7f9d7242f700 1 -- 172.21.15.16:0/3212233418 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f9d5806b190 msgr2=0x7f9d5806d650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:48:35.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.241+0000 7f9d7242f700 1 --2- 172.21.15.16:0/3212233418 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f9d5806b190 0x7f9d5806d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:35.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.241+0000 7f9d7242f700 1 -- 172.21.15.16:0/3212233418 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d6c06a7e0 msgr2=0x7f9d6c0c74c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:35.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.241+0000 7f9d7242f700 1 --2- 172.21.15.16:0/3212233418 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d6c06a7e0 0x7f9d6c0c74c0 secure :-1 s=READY pgs=3041 cs=0 l=1 rev1=1 crypto rx=0x7f9d5c000c00 tx=0x7f9d5c004750 comp rx=0 tx=0).stop 2023-10-16T13:48:35.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.241+0000 7f9d7242f700 1 -- 172.21.15.16:0/3212233418 shutdown_connections 2023-10-16T13:48:35.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.241+0000 7f9d7242f700 1 --2- 172.21.15.16:0/3212233418 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7f9d5806b190 0x7f9d5806d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:35.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.241+0000 7f9d7242f700 1 --2- 172.21.15.16:0/3212233418 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9d6c15b2c0 0x7f9d6c15f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:35.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.241+0000 7f9d7242f700 1 --2- 172.21.15.16:0/3212233418 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9d6c06a7e0 0x7f9d6c0c74c0 unknown :-1 s=CLOSED pgs=3041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:35.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.241+0000 7f9d7242f700 1 --2- 172.21.15.16:0/3212233418 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9d6c06e570 0x7f9d6c15f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:35.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.241+0000 7f9d7242f700 1 -- 172.21.15.16:0/3212233418 >> 172.21.15.16:0/3212233418 conn(0x7f9d6c0b1f40 msgr2=0x7f9d6c15b990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:35.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.242+0000 7f9d7242f700 1 -- 172.21.15.16:0/3212233418 shutdown_connections 2023-10-16T13:48:35.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.242+0000 7f9d7242f700 1 -- 172.21.15.16:0/3212233418 wait complete. 2023-10-16T13:48:35.257 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:48:35.257 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-10-16T13:48:35.285 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1m 2023-10-16T13:48:35.358 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:48:35.355+0000 7f78fc93e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T13:48:35.385 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:35.383+0000 7f648ebc9200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T13:48:35.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.453+0000 7fb207ad3700 1 Processor -- start 2023-10-16T13:48:35.456 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.453+0000 7fb207ad3700 1 -- start start 2023-10-16T13:48:35.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.454+0000 7fb207ad3700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fb2000d4060 0x7fb2000d1520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:35.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.454+0000 7fb207ad3700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb2000d1d20 0x7fb2000d21b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:35.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.454+0000 7fb207ad3700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2000d4540 con 0x7fb20015da10 2023-10-16T13:48:35.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.454+0000 7fb207ad3700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2000cf9b0 con 0x7fb2000d4060 2023-10-16T13:48:35.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.454+0000 7fb207ad3700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2000d26f0 con 0x7fb2000d1d20 2023-10-16T13:48:35.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.454+0000 7fb20586f700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb20015da10 0x7fb2000cf2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:47130/0 (socket says 172.21.15.16:47130) 2023-10-16T13:48:35.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.454+0000 7fb20586f700 1 -- 172.21.15.16:0/4039984334 learned_addr learned my addr 172.21.15.16:0/4039984334 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:48:35.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.454+0000 7fb20506e700 1 --2- 172.21.15.16:0/4039984334 >> v2:172.21.15.49:3300/0 conn(0x7fb2000d4060 0x7fb2000d1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:35.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.454+0000 7fb206070700 1 --2- 172.21.15.16:0/4039984334 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb2000d1d20 0x7fb2000d21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:35.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.454+0000 7fb20486d700 1 -- 172.21.15.16:0/4039984334 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 493935743 0 0) 0x7fb2000d4540 con 0x7fb20015da10 2023-10-16T13:48:35.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.454+0000 7fb20486d700 1 -- 172.21.15.16:0/4039984334 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1ec0035b0 con 0x7fb20015da10 2023-10-16T13:48:35.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.454+0000 7fb206070700 1 -- 172.21.15.16:0/4039984334 >> v2:172.21.15.49:3300/0 conn(0x7fb2000d4060 msgr2=0x7fb2000d1520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:35.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.454+0000 7fb206070700 1 --2- 172.21.15.16:0/4039984334 >> v2:172.21.15.49:3300/0 conn(0x7fb2000d4060 0x7fb2000d1520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:35.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.454+0000 7fb206070700 1 -- 172.21.15.16:0/4039984334 >> v1:172.21.15.16:6789/0 conn(0x7fb20015da10 legacy=0x7fb2000cf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:35.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.454+0000 7fb206070700 1 -- 172.21.15.16:0/4039984334 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2000ce7e0 con 0x7fb2000d1d20 2023-10-16T13:48:35.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.454+0000 7fb20506e700 1 --2- 172.21.15.16:0/4039984334 >> v2:172.21.15.49:3300/0 conn(0x7fb2000d4060 0x7fb2000d1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:48:35.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.455+0000 7fb206070700 1 --2- 172.21.15.16:0/4039984334 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb2000d1d20 0x7fb2000d21b0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fb1f8009d90 tx=0x7fb1f800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=811195ab4eae145d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:35.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.455+0000 7fb20486d700 1 -- 172.21.15.16:0/4039984334 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f800e070 con 0x7fb2000d1d20 2023-10-16T13:48:35.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.455+0000 7fb20486d700 1 -- 172.21.15.16:0/4039984334 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1f800bb20 con 0x7fb2000d1d20 2023-10-16T13:48:35.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.455+0000 7fb20486d700 1 -- 172.21.15.16:0/4039984334 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f800be90 con 0x7fb2000d1d20 2023-10-16T13:48:35.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.455+0000 7fb207ad3700 1 -- 172.21.15.16:0/4039984334 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb2000d1d20 msgr2=0x7fb2000d21b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:35.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.455+0000 7fb207ad3700 1 --2- 172.21.15.16:0/4039984334 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb2000d1d20 0x7fb2000d21b0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fb1f8009d90 tx=0x7fb1f800b330 comp rx=0 tx=0).stop 2023-10-16T13:48:35.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.455+0000 7fb207ad3700 1 -- 172.21.15.16:0/4039984334 shutdown_connections 2023-10-16T13:48:35.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.455+0000 7fb207ad3700 1 --2- 172.21.15.16:0/4039984334 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb2000d1d20 0x7fb2000d21b0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:35.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.455+0000 7fb207ad3700 1 --2- 172.21.15.16:0/4039984334 >> v2:172.21.15.49:3300/0 conn(0x7fb2000d4060 0x7fb2000d1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:35.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.455+0000 7fb207ad3700 1 -- 172.21.15.16:0/4039984334 >> 172.21.15.16:0/4039984334 conn(0x7fb20009c9f0 msgr2=0x7fb2000d3740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:35.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.455+0000 7fb207ad3700 1 -- 172.21.15.16:0/4039984334 shutdown_connections 2023-10-16T13:48:35.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.456+0000 7fb207ad3700 1 -- 172.21.15.16:0/4039984334 wait complete. 2023-10-16T13:48:35.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.457+0000 7fb207ad3700 1 Processor -- start 2023-10-16T13:48:35.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.457+0000 7fb207ad3700 1 -- start start 2023-10-16T13:48:35.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.457+0000 7fb207ad3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2000d1d20 0x7fb2000b6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:35.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.457+0000 7fb207ad3700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb2000d4060 0x7fb2000b5c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:35.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.457+0000 7fb207ad3700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb20015da10 0x7fb2000b88d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:35.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.457+0000 7fb207ad3700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2001f7170 con 0x7fb2000d1d20 2023-10-16T13:48:35.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.457+0000 7fb207ad3700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2001f76a0 con 0x7fb20015da10 2023-10-16T13:48:35.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.457+0000 7fb207ad3700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2000b32d0 con 0x7fb2000d4060 2023-10-16T13:48:35.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.458+0000 7fb20586f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2000d1d20 0x7fb2000b6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:35.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.458+0000 7fb20586f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2000d1d20 0x7fb2000b6360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57862/0 (socket says 172.21.15.16:57862) 2023-10-16T13:48:35.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.458+0000 7fb20586f700 1 -- 172.21.15.16:0/180929129 learned_addr learned my addr 172.21.15.16:0/180929129 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:35.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.458+0000 7fb20506e700 1 --2- 172.21.15.16:0/180929129 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb2000d4060 0x7fb2000b5c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:35.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.458+0000 7fb206070700 1 --2- 172.21.15.16:0/180929129 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb20015da10 0x7fb2000b88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:35.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.458+0000 7fb20586f700 1 -- 172.21.15.16:0/180929129 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb2000d4060 msgr2=0x7fb2000b5c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:35.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.458+0000 7fb20586f700 1 --2- 172.21.15.16:0/180929129 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb2000d4060 0x7fb2000b5c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:35.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.458+0000 7fb20586f700 1 -- 172.21.15.16:0/180929129 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb20015da10 msgr2=0x7fb2000b88d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:35.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.458+0000 7fb20586f700 1 --2- 172.21.15.16:0/180929129 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb20015da10 0x7fb2000b88d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:35.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.458+0000 7fb20586f700 1 -- 172.21.15.16:0/180929129 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1f00094a0 con 0x7fb2000d1d20 2023-10-16T13:48:35.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.458+0000 7fb20506e700 1 --2- 172.21.15.16:0/180929129 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb2000d4060 0x7fb2000b5c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:48:35.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.458+0000 7fb20586f700 1 --2- 172.21.15.16:0/180929129 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2000d1d20 0x7fb2000b6360 secure :-1 s=READY pgs=3043 cs=0 l=1 rev1=1 crypto rx=0x7fb1fc00c4d0 tx=0x7fb1fc00c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:35.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.458+0000 7fb1f67fc700 1 -- 172.21.15.16:0/180929129 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1fc0074e0 con 0x7fb2000d1d20 2023-10-16T13:48:35.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.458+0000 7fb207ad3700 1 -- 172.21.15.16:0/180929129 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1f8009a70 con 0x7fb2000d1d20 2023-10-16T13:48:35.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.458+0000 7fb1f67fc700 1 -- 172.21.15.16:0/180929129 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1fc007660 con 0x7fb2000d1d20 2023-10-16T13:48:35.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.458+0000 7fb1f67fc700 1 -- 172.21.15.16:0/180929129 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1fc005a70 con 0x7fb2000d1d20 2023-10-16T13:48:35.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.458+0000 7fb207ad3700 1 -- 172.21.15.16:0/180929129 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2000b3570 con 0x7fb2000d1d20 2023-10-16T13:48:35.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.459+0000 7fb207ad3700 1 -- 172.21.15.16:0/180929129 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb20004ebb0 con 0x7fb2000d1d20 2023-10-16T13:48:35.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.460+0000 7fb1f67fc700 1 -- 172.21.15.16:0/180929129 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 158) v1 ==== 89450+0+0 (secure 0 0 0) 0x7fb1fc005bd0 con 0x7fb2000d1d20 2023-10-16T13:48:35.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.460+0000 7fb1f67fc700 1 --2- 172.21.15.16:0/180929129 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fb1e406b190 0x7fb1e406d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:35.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.461+0000 7fb20506e700 1 -- 172.21.15.16:0/180929129 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fb1e406b190 msgr2=0x7fb1e406d650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1275231846 2023-10-16T13:48:35.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.461+0000 7fb1f67fc700 1 -- 172.21.15.16:0/180929129 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 6157+0+0 (secure 0 0 0) 0x7fb1fc088fc0 con 0x7fb2000d1d20 2023-10-16T13:48:35.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.461+0000 7fb20506e700 1 --2- 172.21.15.16:0/180929129 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fb1e406b190 0x7fb1e406d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:48:35.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.465+0000 7fb1f67fc700 1 -- 172.21.15.16:0/180929129 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb1fc0bbfe0 con 0x7fb2000d1d20 2023-10-16T13:48:35.478 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:48:35.476+0000 7f78fc93e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T13:48:35.509 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:35.507+0000 7f648ebc9200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T13:48:35.555 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:48:35.553+0000 7f78fc93e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T13:48:35.587 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:35.585+0000 7f648ebc9200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T13:48:35.639 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:48:35.637+0000 7f78fc93e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T13:48:35.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.662+0000 7fb20506e700 1 -- 172.21.15.16:0/180929129 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fb1e406b190 msgr2=0x7fb1e406d650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1275231846 2023-10-16T13:48:35.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.662+0000 7fb20506e700 1 --2- 172.21.15.16:0/180929129 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fb1e406b190 0x7fb1e406d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:48:35.672 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:35.670+0000 7f648ebc9200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T13:48:35.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:35.700+0000 7fb207ad3700 1 -- 172.21.15.16:0/180929129 --> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fb2000ceda0 con 0x7fb1e406b190 2023-10-16T13:48:36.027 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:48:36.025+0000 7f78fc93e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T13:48:36.062 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:36.060+0000 7f648ebc9200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T13:48:36.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:36.063+0000 7fb20506e700 1 -- 172.21.15.16:0/180929129 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fb1e406b190 msgr2=0x7fb1e406d650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1275231846 2023-10-16T13:48:36.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:36.063+0000 7fb20506e700 1 --2- 172.21.15.16:0/180929129 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fb1e406b190 0x7fb1e406d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-16T13:48:36.636 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:48:36.633+0000 7f78fc93e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T13:48:36.672 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:36.670+0000 7f648ebc9200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T13:48:36.709 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:48:36.706+0000 7f78fc93e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T13:48:36.746 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:36.744+0000 7f648ebc9200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T13:48:36.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:36.864+0000 7fb20506e700 1 -- 172.21.15.16:0/180929129 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fb1e406b190 msgr2=0x7fb1e406d650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1275231846 2023-10-16T13:48:36.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:36.864+0000 7fb20506e700 1 --2- 172.21.15.16:0/180929129 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fb1e406b190 0x7fb1e406d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-16T13:48:37.374 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:48:37.372+0000 7f78fc93e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:48:37.420 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:37.417+0000 7f648ebc9200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:48:37.456 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:48:37.454+0000 7f78fc93e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T13:48:37.503 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:37.500+0000 7f648ebc9200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T13:48:37.607 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:48:37.605+0000 7f78fc93e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:48:37.656 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:37.653+0000 7f648ebc9200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:48:37.845 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:48:37.842+0000 7f78fc93e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T13:48:37.897 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:37.894+0000 7f648ebc9200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T13:48:37.922 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:48:37.920+0000 7f78fc93e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T13:48:37.975 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:37.973+0000 7f648ebc9200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T13:48:38.200 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:48:38.197+0000 7f78fc93e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T13:48:38.254 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:38.252+0000 7f648ebc9200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T13:48:38.431 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:48:38.429+0000 7f78fc93e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T13:48:38.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:38.465+0000 7fb20506e700 1 -- 172.21.15.16:0/180929129 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fb1e406b190 msgr2=0x7fb1e406d650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1275231846 2023-10-16T13:48:38.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:38.465+0000 7fb20506e700 1 --2- 172.21.15.16:0/180929129 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fb1e406b190 0x7fb1e406d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-10-16T13:48:38.490 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:38.488+0000 7f648ebc9200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T13:48:38.588 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:48:38.585+0000 7f78fc93e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T13:48:38.651 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:38.649+0000 7f648ebc9200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T13:48:38.707 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:48:38.705+0000 7f78fc93e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T13:48:38.773 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:38.770+0000 7f648ebc9200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T13:48:38.871 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:48:38.869+0000 7f78fc93e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T13:48:38.940 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:38.937+0000 7f648ebc9200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T13:48:39.144 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:48:39.142+0000 7f78fc93e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T13:48:39.216 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:39.213+0000 7f648ebc9200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T13:48:39.313 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:48:39.311+0000 7f78fc93e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T13:48:39.387 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:39.385+0000 7f648ebc9200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T13:48:39.869 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:48:39.866+0000 7f78fc93e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T13:48:39.940 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:39.938+0000 7fb1f67fc700 1 -- 172.21.15.16:0/180929129 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 159) v1 ==== 89450+0+0 (secure 0 0 0) 0x7fb1fc053f30 con 0x7fb2000d1d20 2023-10-16T13:48:39.961 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:39.959+0000 7f648ebc9200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T13:48:39.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:39.978+0000 7fb1f67fc700 1 -- 172.21.15.16:0/180929129 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mgrmap(e 160) v1 ==== 88957+0+0 (secure 0 0 0) 0x7fb1fc076500 con 0x7fb2000d1d20 2023-10-16T13:48:39.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:39.978+0000 7fb1f67fc700 1 -- 172.21.15.16:0/180929129 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fb1e406b190 msgr2=0x7fb1e406d650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:48:39.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:39.978+0000 7fb1f67fc700 1 --2- 172.21.15.16:0/180929129 >> [v2:172.21.15.16:6832/1275231846,v1:172.21.15.16:6833/1275231846] conn(0x7fb1e406b190 0x7fb1e406d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:40.121 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:40.118+0000 7f6401ba4700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:48:40.121 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:40.118+0000 7f6401ba4700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:48:40.121 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:40.118+0000 7f6401ba4700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:48:40.121 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:40.118+0000 7f6401ba4700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:48:40.122 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:40.118+0000 7f6401ba4700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:48:40.122 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:40.118+0000 7f6404baa700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:48:40.122 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:40.118+0000 7f6404baa700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:48:40.122 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:40.118+0000 7f6404baa700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:48:40.122 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:40.118+0000 7f6404baa700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:48:40.123 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:40.118+0000 7f6404baa700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:48:40.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:40.982+0000 7fb1f67fc700 1 -- 172.21.15.16:0/180929129 <== mon.0 v2:172.21.15.16:3300/0 9 ==== mgrmap(e 161) v1 ==== 89084+0+0 (secure 0 0 0) 0x7fb1fc054800 con 0x7fb2000d1d20 2023-10-16T13:48:40.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:40.983+0000 7fb1f67fc700 1 --2- 172.21.15.16:0/180929129 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7fb1e401cc70 0x7fb1e401d050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:40.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:40.983+0000 7fb1f67fc700 1 -- 172.21.15.16:0/180929129 --> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fb1e401d590 con 0x7fb1e401cc70 2023-10-16T13:48:40.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:40.985+0000 7fb20506e700 1 --2- 172.21.15.16:0/180929129 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7fb1e401cc70 0x7fb1e401d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:40.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:40.987+0000 7fb20506e700 1 --2- 172.21.15.16:0/180929129 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7fb1e401cc70 0x7fb1e401d050 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb2000b8480 tx=0x7fb1f0009210 comp rx=0 tx=0).ready entity=mgr.13239 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:41.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.050+0000 7fb1f67fc700 1 -- 172.21.15.16:0/180929129 <== mgr.13239 v2:172.21.15.16:6832/2131389174 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fb1e401d590 con 0x7fb1e401cc70 2023-10-16T13:48:41.052 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule set for path snap_test_dir1 2023-10-16T13:48:41.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.052+0000 7fb207ad3700 1 -- 172.21.15.16:0/180929129 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7fb1e401cc70 msgr2=0x7fb1e401d050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:41.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.052+0000 7fb207ad3700 1 --2- 172.21.15.16:0/180929129 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7fb1e401cc70 0x7fb1e401d050 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb2000b8480 tx=0x7fb1f0009210 comp rx=0 tx=0).stop 2023-10-16T13:48:41.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.052+0000 7fb207ad3700 1 -- 172.21.15.16:0/180929129 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2000d1d20 msgr2=0x7fb2000b6360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:41.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.052+0000 7fb207ad3700 1 --2- 172.21.15.16:0/180929129 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2000d1d20 0x7fb2000b6360 secure :-1 s=READY pgs=3043 cs=0 l=1 rev1=1 crypto rx=0x7fb1fc00c4d0 tx=0x7fb1fc00c7e0 comp rx=0 tx=0).stop 2023-10-16T13:48:41.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.052+0000 7fb206070700 1 -- 172.21.15.16:0/180929129 reap_dead start 2023-10-16T13:48:41.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.052+0000 7fb207ad3700 1 -- 172.21.15.16:0/180929129 shutdown_connections 2023-10-16T13:48:41.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.052+0000 7fb207ad3700 1 -- 172.21.15.16:0/180929129 >> 172.21.15.16:0/180929129 conn(0x7fb20009c9f0 msgr2=0x7fb2000cf1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:41.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.052+0000 7fb207ad3700 1 -- 172.21.15.16:0/180929129 shutdown_connections 2023-10-16T13:48:41.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.052+0000 7fb207ad3700 1 -- 172.21.15.16:0/180929129 wait complete. 2023-10-16T13:48:41.068 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2023-10-16T13:48:41.068 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention add snap_test_dir1 1m 2023-10-16T13:48:41.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.198+0000 7f403578f700 1 Processor -- start 2023-10-16T13:48:41.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.198+0000 7f403578f700 1 -- start start 2023-10-16T13:48:41.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.199+0000 7f403578f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f403015d7a0 0x7f40300b7240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:41.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.199+0000 7f403578f700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f40300ce970 0x7f40300d1740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:41.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.199+0000 7f403578f700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40300b91b0 con 0x7f40300b7780 2023-10-16T13:48:41.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.199+0000 7f403578f700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40300b8980 con 0x7f403015d7a0 2023-10-16T13:48:41.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.199+0000 7f403578f700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40300b7d00 con 0x7f40300ce970 2023-10-16T13:48:41.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.199+0000 7f402effd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f403015d7a0 0x7f40300b7240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:41.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.199+0000 7f402e7fc700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f40300b7780 0x7f40300ce260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:47320/0 (socket says 172.21.15.16:47320) 2023-10-16T13:48:41.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.199+0000 7f402e7fc700 1 -- 172.21.15.16:0/3547000983 learned_addr learned my addr 172.21.15.16:0/3547000983 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:48:41.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.199+0000 7f402f7fe700 1 --2- 172.21.15.16:0/3547000983 >> v2:172.21.15.49:3300/0 conn(0x7f40300ce970 0x7f40300d1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:41.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.199+0000 7f402effd700 1 -- 172.21.15.16:0/3547000983 >> v2:172.21.15.49:3300/0 conn(0x7f40300ce970 msgr2=0x7f40300d1740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:41.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.199+0000 7f402effd700 1 --2- 172.21.15.16:0/3547000983 >> v2:172.21.15.49:3300/0 conn(0x7f40300ce970 0x7f40300d1740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:41.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.199+0000 7f402effd700 1 -- 172.21.15.16:0/3547000983 >> v1:172.21.15.16:6789/0 conn(0x7f40300b7780 legacy=0x7f40300ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:41.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.199+0000 7f402effd700 1 -- 172.21.15.16:0/3547000983 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40300b7fe0 con 0x7f403015d7a0 2023-10-16T13:48:41.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.199+0000 7f402effd700 1 --2- 172.21.15.16:0/3547000983 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f403015d7a0 0x7f40300b7240 secure :-1 s=READY pgs=3058 cs=0 l=1 rev1=1 crypto rx=0x7f40300c6ec0 tx=0x7f4020009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7d6022106f639e3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:41.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.200+0000 7f402dffb700 1 -- 172.21.15.16:0/3547000983 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4020004230 con 0x7f403015d7a0 2023-10-16T13:48:41.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.200+0000 7f402dffb700 1 -- 172.21.15.16:0/3547000983 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4020004390 con 0x7f403015d7a0 2023-10-16T13:48:41.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.200+0000 7f402dffb700 1 -- 172.21.15.16:0/3547000983 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40200036a0 con 0x7f403015d7a0 2023-10-16T13:48:41.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.200+0000 7f403578f700 1 -- 172.21.15.16:0/3547000983 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f403015d7a0 msgr2=0x7f40300b7240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:41.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.200+0000 7f403578f700 1 --2- 172.21.15.16:0/3547000983 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f403015d7a0 0x7f40300b7240 secure :-1 s=READY pgs=3058 cs=0 l=1 rev1=1 crypto rx=0x7f40300c6ec0 tx=0x7f4020009b20 comp rx=0 tx=0).stop 2023-10-16T13:48:41.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.200+0000 7f403578f700 1 -- 172.21.15.16:0/3547000983 shutdown_connections 2023-10-16T13:48:41.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.200+0000 7f403578f700 1 --2- 172.21.15.16:0/3547000983 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f403015d7a0 0x7f40300b7240 unknown :-1 s=CLOSED pgs=3058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:41.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.200+0000 7f403578f700 1 --2- 172.21.15.16:0/3547000983 >> v2:172.21.15.49:3300/0 conn(0x7f40300ce970 0x7f40300d1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:41.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.200+0000 7f403578f700 1 -- 172.21.15.16:0/3547000983 >> 172.21.15.16:0/3547000983 conn(0x7f4030072330 msgr2=0x7f4030072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:41.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.200+0000 7f403578f700 1 -- 172.21.15.16:0/3547000983 shutdown_connections 2023-10-16T13:48:41.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.200+0000 7f403578f700 1 -- 172.21.15.16:0/3547000983 wait complete. 2023-10-16T13:48:41.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.202+0000 7f403578f700 1 Processor -- start 2023-10-16T13:48:41.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.202+0000 7f403578f700 1 -- start start 2023-10-16T13:48:41.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.203+0000 7f403578f700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f40300b7780 0x7f40300b1790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:41.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.203+0000 7f403578f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f40300ce970 0x7f40300cb6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:41.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.203+0000 7f403578f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f403015d7a0 0x7f40300cbbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:41.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.203+0000 7f403578f700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40300cc220 con 0x7f403015d7a0 2023-10-16T13:48:41.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.203+0000 7f403578f700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40300cc500 con 0x7f40300b7780 2023-10-16T13:48:41.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.203+0000 7f403578f700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40300cc7e0 con 0x7f40300ce970 2023-10-16T13:48:41.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.203+0000 7f402f7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f403015d7a0 0x7f40300cbbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:41.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.203+0000 7f402f7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f403015d7a0 0x7f40300cbbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58056/0 (socket says 172.21.15.16:58056) 2023-10-16T13:48:41.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.203+0000 7f402f7fe700 1 -- 172.21.15.16:0/2784378600 learned_addr learned my addr 172.21.15.16:0/2784378600 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:41.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.203+0000 7f402effd700 1 --2- 172.21.15.16:0/2784378600 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f40300b7780 0x7f40300b1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:41.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.203+0000 7f402e7fc700 1 --2- 172.21.15.16:0/2784378600 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f40300ce970 0x7f40300cb6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:41.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.203+0000 7f402f7fe700 1 -- 172.21.15.16:0/2784378600 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f40300ce970 msgr2=0x7f40300cb6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:41.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.203+0000 7f402f7fe700 1 --2- 172.21.15.16:0/2784378600 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f40300ce970 0x7f40300cb6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:41.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.203+0000 7f402f7fe700 1 -- 172.21.15.16:0/2784378600 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f40300b7780 msgr2=0x7f40300b1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:41.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.203+0000 7f402f7fe700 1 --2- 172.21.15.16:0/2784378600 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f40300b7780 0x7f40300b1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:41.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.203+0000 7f402f7fe700 1 -- 172.21.15.16:0/2784378600 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40200094a0 con 0x7f403015d7a0 2023-10-16T13:48:41.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.203+0000 7f402e7fc700 1 --2- 172.21.15.16:0/2784378600 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f40300ce970 0x7f40300cb6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:48:41.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.204+0000 7f402f7fe700 1 --2- 172.21.15.16:0/2784378600 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f403015d7a0 0x7f40300cbbf0 secure :-1 s=READY pgs=3059 cs=0 l=1 rev1=1 crypto rx=0x7f4028007e30 tx=0x7f402800e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:41.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.204+0000 7f40177fe700 1 -- 172.21.15.16:0/2784378600 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f402800eea0 con 0x7f403015d7a0 2023-10-16T13:48:41.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.204+0000 7f40177fe700 1 -- 172.21.15.16:0/2784378600 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4028014420 con 0x7f403015d7a0 2023-10-16T13:48:41.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.204+0000 7f403578f700 1 -- 172.21.15.16:0/2784378600 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40300ccac0 con 0x7f403015d7a0 2023-10-16T13:48:41.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.204+0000 7f40177fe700 1 -- 172.21.15.16:0/2784378600 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40280147d0 con 0x7f403015d7a0 2023-10-16T13:48:41.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.204+0000 7f403578f700 1 -- 172.21.15.16:0/2784378600 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40300b3700 con 0x7f403015d7a0 2023-10-16T13:48:41.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.205+0000 7f403578f700 1 -- 172.21.15.16:0/2784378600 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40300ccfe0 con 0x7f403015d7a0 2023-10-16T13:48:41.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.205+0000 7f40177fe700 1 -- 172.21.15.16:0/2784378600 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 161) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f4028014930 con 0x7f403015d7a0 2023-10-16T13:48:41.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.206+0000 7f40177fe700 1 --2- 172.21.15.16:0/2784378600 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f401c06ada0 0x7f401c06d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:41.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.206+0000 7f40177fe700 1 -- 172.21.15.16:0/2784378600 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(266..266 src has 1..266) v4 ==== 6587+0+0 (secure 0 0 0) 0x7f4028089d70 con 0x7f403015d7a0 2023-10-16T13:48:41.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.206+0000 7f402effd700 1 --2- 172.21.15.16:0/2784378600 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f401c06ada0 0x7f401c06d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:41.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.206+0000 7f402effd700 1 --2- 172.21.15.16:0/2784378600 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f401c06ada0 0x7f401c06d260 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f40300c6ec0 tx=0x7f4020013040 comp rx=0 tx=0).ready entity=mgr.13239 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:41.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.210+0000 7f40177fe700 1 -- 172.21.15.16:0/2784378600 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4028054df0 con 0x7f403015d7a0 2023-10-16T13:48:41.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.445+0000 7f403578f700 1 -- 172.21.15.16:0/2784378600 --> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "path": "snap_test_dir1", "retention_spec_or_period": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f40300b85a0 con 0x7f401c06ada0 2023-10-16T13:48:41.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.500+0000 7f40177fe700 1 -- 172.21.15.16:0/2784378600 <== mgr.13239 v2:172.21.15.16:6832/2131389174 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+39 (secure 0 0 0) 0x7f4020004220 con 0x7f401c06ada0 2023-10-16T13:48:41.502 INFO:teuthology.orchestra.run.smithi016.stdout:Retention added to path snap_test_dir1 2023-10-16T13:48:41.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.502+0000 7f403578f700 1 -- 172.21.15.16:0/2784378600 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f401c06ada0 msgr2=0x7f401c06d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:41.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.502+0000 7f403578f700 1 --2- 172.21.15.16:0/2784378600 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f401c06ada0 0x7f401c06d260 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f40300c6ec0 tx=0x7f4020013040 comp rx=0 tx=0).stop 2023-10-16T13:48:41.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.502+0000 7f403578f700 1 -- 172.21.15.16:0/2784378600 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f403015d7a0 msgr2=0x7f40300cbbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:41.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.502+0000 7f403578f700 1 --2- 172.21.15.16:0/2784378600 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f403015d7a0 0x7f40300cbbf0 secure :-1 s=READY pgs=3059 cs=0 l=1 rev1=1 crypto rx=0x7f4028007e30 tx=0x7f402800e3c0 comp rx=0 tx=0).stop 2023-10-16T13:48:41.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.502+0000 7f403578f700 1 -- 172.21.15.16:0/2784378600 shutdown_connections 2023-10-16T13:48:41.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.502+0000 7f403578f700 1 --2- 172.21.15.16:0/2784378600 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f401c06ada0 0x7f401c06d260 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:41.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.502+0000 7f403578f700 1 --2- 172.21.15.16:0/2784378600 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f403015d7a0 0x7f40300cbbf0 unknown :-1 s=CLOSED pgs=3059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:41.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.502+0000 7f403578f700 1 --2- 172.21.15.16:0/2784378600 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f40300b7780 0x7f40300b1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:41.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.502+0000 7f403578f700 1 --2- 172.21.15.16:0/2784378600 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f40300ce970 0x7f40300cb6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:41.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.502+0000 7f403578f700 1 -- 172.21.15.16:0/2784378600 >> 172.21.15.16:0/2784378600 conn(0x7f4030072330 msgr2=0x7f40300b21f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:41.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.502+0000 7f403578f700 1 -- 172.21.15.16:0/2784378600 shutdown_connections 2023-10-16T13:48:41.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.502+0000 7f403578f700 1 -- 172.21.15.16:0/2784378600 wait complete. 2023-10-16T13:48:41.517 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention added to path snap_test_dir1 2023-10-16T13:48:41.518 INFO:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2023-10-16-13_49_00 in ~19s... 2023-10-16T13:48:41.518 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m'], expected_retention: [{'m': 1}] 2023-10-16T13:48:41.518 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json snap_test_dir1 2023-10-16T13:48:41.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.643+0000 7f1a690a2700 1 Processor -- start 2023-10-16T13:48:41.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.643+0000 7f1a690a2700 1 -- start start 2023-10-16T13:48:41.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.644+0000 7f1a690a2700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1a640cb150 0x7f1a64071a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:41.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.644+0000 7f1a690a2700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a6415c040 con 0x7f1a64072b30 2023-10-16T13:48:41.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.644+0000 7f1a690a2700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a6415c320 con 0x7f1a6415bb40 2023-10-16T13:48:41.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.644+0000 7f1a690a2700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6406ea20 con 0x7f1a640cb150 2023-10-16T13:48:41.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.644+0000 7f1a62d9d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f1a64072b30 0x7f1a6406e310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:47334/0 (socket says 172.21.15.16:47334) 2023-10-16T13:48:41.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.644+0000 7f1a62d9d700 1 -- 172.21.15.16:0/43807625 learned_addr learned my addr 172.21.15.16:0/43807625 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:48:41.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.644+0000 7f1a6359e700 1 --2- 172.21.15.16:0/43807625 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1a640cb150 0x7f1a64071a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:41.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.645+0000 7f1a61d9b700 1 -- 172.21.15.16:0/43807625 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 653762248 0 0) 0x7f1a6415c040 con 0x7f1a64072b30 2023-10-16T13:48:41.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.645+0000 7f1a61d9b700 1 -- 172.21.15.16:0/43807625 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a500035b0 con 0x7f1a64072b30 2023-10-16T13:48:41.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.645+0000 7f1a61d9b700 1 -- 172.21.15.16:0/43807625 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3925972839 0 0) 0x7f1a6415c320 con 0x7f1a6415bb40 2023-10-16T13:48:41.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.645+0000 7f1a61d9b700 1 -- 172.21.15.16:0/43807625 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a6415c040 con 0x7f1a6415bb40 2023-10-16T13:48:41.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.645+0000 7f1a61d9b700 1 -- 172.21.15.16:0/43807625 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4070779536 0 0) 0x7f1a500035b0 con 0x7f1a64072b30 2023-10-16T13:48:41.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.645+0000 7f1a61d9b700 1 -- 172.21.15.16:0/43807625 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1a640cb150 msgr2=0x7f1a64071a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:41.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.645+0000 7f1a61d9b700 1 --2- 172.21.15.16:0/43807625 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1a640cb150 0x7f1a64071a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:41.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.645+0000 7f1a61d9b700 1 -- 172.21.15.16:0/43807625 >> v1:172.21.15.49:6789/0 conn(0x7f1a6415bb40 legacy=0x7f1a640caa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:41.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.645+0000 7f1a61d9b700 1 -- 172.21.15.16:0/43807625 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a6415c600 con 0x7f1a64072b30 2023-10-16T13:48:41.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.645+0000 7f1a61d9b700 1 -- 172.21.15.16:0/43807625 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f1a54001670 con 0x7f1a64072b30 2023-10-16T13:48:41.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.645+0000 7f1a61d9b700 1 -- 172.21.15.16:0/43807625 >> v1:172.21.15.16:6789/0 conn(0x7f1a64072b30 legacy=0x7f1a6406e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:41.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.645+0000 7f1a61d9b700 1 --2- 172.21.15.16:0/43807625 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1a640cb150 0x7f1a50003bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:41.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.645+0000 7f1a61d9b700 1 --2- 172.21.15.16:0/43807625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a50004120 0x7f1a50004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:41.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.645+0000 7f1a61d9b700 1 --2- 172.21.15.16:0/43807625 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1a50004d90 0x7f1a50005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:41.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.645+0000 7f1a61d9b700 1 -- 172.21.15.16:0/43807625 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6406ea20 con 0x7f1a50004120 2023-10-16T13:48:41.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.645+0000 7f1a61d9b700 1 -- 172.21.15.16:0/43807625 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5000baf0 con 0x7f1a50004d90 2023-10-16T13:48:41.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.645+0000 7f1a61d9b700 1 -- 172.21.15.16:0/43807625 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5000bcb0 con 0x7f1a640cb150 2023-10-16T13:48:41.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.645+0000 7f1a6259c700 1 --2- 172.21.15.16:0/43807625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a50004120 0x7f1a50004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:41.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.645+0000 7f1a62d9d700 1 --2- 172.21.15.16:0/43807625 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1a640cb150 0x7f1a50003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:41.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.645+0000 7f1a6359e700 1 --2- 172.21.15.16:0/43807625 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1a50004d90 0x7f1a50005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:41.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.645+0000 7f1a6259c700 1 -- 172.21.15.16:0/43807625 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1a640cb150 msgr2=0x7f1a50003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:41.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.645+0000 7f1a6259c700 1 --2- 172.21.15.16:0/43807625 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1a640cb150 0x7f1a50003bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:41.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.645+0000 7f1a6259c700 1 -- 172.21.15.16:0/43807625 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1a50004d90 msgr2=0x7f1a50005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:41.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.645+0000 7f1a6259c700 1 --2- 172.21.15.16:0/43807625 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1a50004d90 0x7f1a50005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:41.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.645+0000 7f1a6259c700 1 -- 172.21.15.16:0/43807625 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a500035b0 con 0x7f1a50004120 2023-10-16T13:48:41.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.646+0000 7f1a6259c700 1 --2- 172.21.15.16:0/43807625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a50004120 0x7f1a50004850 secure :-1 s=READY pgs=3061 cs=0 l=1 rev1=1 crypto rx=0x7f1a6415bf70 tx=0x7f1a50005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8dde67f5d1a99c3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:41.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.646+0000 7f1a61d9b700 1 -- 172.21.15.16:0/43807625 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a4c001430 con 0x7f1a50004120 2023-10-16T13:48:41.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.646+0000 7f1a61d9b700 1 -- 172.21.15.16:0/43807625 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a4c0052d0 con 0x7f1a50004120 2023-10-16T13:48:41.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.646+0000 7f1a61d9b700 1 -- 172.21.15.16:0/43807625 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a4c001d40 con 0x7f1a50004120 2023-10-16T13:48:41.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.646+0000 7f1a690a2700 1 -- 172.21.15.16:0/43807625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a50004120 msgr2=0x7f1a50004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:41.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.646+0000 7f1a690a2700 1 --2- 172.21.15.16:0/43807625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a50004120 0x7f1a50004850 secure :-1 s=READY pgs=3061 cs=0 l=1 rev1=1 crypto rx=0x7f1a6415bf70 tx=0x7f1a50005880 comp rx=0 tx=0).stop 2023-10-16T13:48:41.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.646+0000 7f1a6359e700 1 -- 172.21.15.16:0/43807625 reap_dead start 2023-10-16T13:48:41.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.646+0000 7f1a690a2700 1 -- 172.21.15.16:0/43807625 shutdown_connections 2023-10-16T13:48:41.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.646+0000 7f1a690a2700 1 -- 172.21.15.16:0/43807625 >> 172.21.15.16:0/43807625 conn(0x7f1a640b1f40 msgr2=0x7f1a640b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:41.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.646+0000 7f1a690a2700 1 -- 172.21.15.16:0/43807625 shutdown_connections 2023-10-16T13:48:41.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.646+0000 7f1a690a2700 1 -- 172.21.15.16:0/43807625 wait complete. 2023-10-16T13:48:41.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.648+0000 7f1a690a2700 1 Processor -- start 2023-10-16T13:48:41.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.648+0000 7f1a690a2700 1 -- start start 2023-10-16T13:48:41.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.649+0000 7f1a690a2700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1a50004120 0x7f1a64155d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:41.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.649+0000 7f1a690a2700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a6406e280 0x7f1a640b57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:41.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.649+0000 7f1a690a2700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1a640b2ff0 0x7f1a640b34a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:41.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.649+0000 7f1a690a2700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a640c89a0 con 0x7f1a6406e280 2023-10-16T13:48:41.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.649+0000 7f1a690a2700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a640b5cf0 con 0x7f1a640b2ff0 2023-10-16T13:48:41.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.649+0000 7f1a690a2700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a640c8000 con 0x7f1a50004120 2023-10-16T13:48:41.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.649+0000 7f1a6259c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a6406e280 0x7f1a640b57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:41.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.649+0000 7f1a6259c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a6406e280 0x7f1a640b57b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58074/0 (socket says 172.21.15.16:58074) 2023-10-16T13:48:41.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.649+0000 7f1a6259c700 1 -- 172.21.15.16:0/1674725883 learned_addr learned my addr 172.21.15.16:0/1674725883 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:41.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.649+0000 7f1a6359e700 1 --2- 172.21.15.16:0/1674725883 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1a640b2ff0 0x7f1a640b34a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:41.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.649+0000 7f1a62d9d700 1 --2- 172.21.15.16:0/1674725883 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1a50004120 0x7f1a64155d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:41.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.649+0000 7f1a6259c700 1 -- 172.21.15.16:0/1674725883 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1a50004120 msgr2=0x7f1a64155d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:41.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.649+0000 7f1a6259c700 1 --2- 172.21.15.16:0/1674725883 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1a50004120 0x7f1a64155d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:41.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.649+0000 7f1a6259c700 1 -- 172.21.15.16:0/1674725883 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1a640b2ff0 msgr2=0x7f1a640b34a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:41.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.649+0000 7f1a6259c700 1 --2- 172.21.15.16:0/1674725883 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1a640b2ff0 0x7f1a640b34a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:41.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.649+0000 7f1a6259c700 1 -- 172.21.15.16:0/1674725883 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a640c82e0 con 0x7f1a6406e280 2023-10-16T13:48:41.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.649+0000 7f1a6359e700 1 --2- 172.21.15.16:0/1674725883 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1a640b2ff0 0x7f1a640b34a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:48:41.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.649+0000 7f1a62d9d700 1 --2- 172.21.15.16:0/1674725883 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1a50004120 0x7f1a64155d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:48:41.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.649+0000 7f1a6259c700 1 --2- 172.21.15.16:0/1674725883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a6406e280 0x7f1a640b57b0 secure :-1 s=READY pgs=3062 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c006a90 tx=0x7f1a4c0012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:41.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.649+0000 7f1a5b7fe700 1 -- 172.21.15.16:0/1674725883 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a4c001d40 con 0x7f1a6406e280 2023-10-16T13:48:41.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.650+0000 7f1a5b7fe700 1 -- 172.21.15.16:0/1674725883 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a4c0013d0 con 0x7f1a6406e280 2023-10-16T13:48:41.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.650+0000 7f1a690a2700 1 -- 172.21.15.16:0/1674725883 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a640c85c0 con 0x7f1a6406e280 2023-10-16T13:48:41.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.650+0000 7f1a5b7fe700 1 -- 172.21.15.16:0/1674725883 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a4c007280 con 0x7f1a6406e280 2023-10-16T13:48:41.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.650+0000 7f1a690a2700 1 -- 172.21.15.16:0/1674725883 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a640c8dc0 con 0x7f1a6406e280 2023-10-16T13:48:41.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.650+0000 7f1a690a2700 1 -- 172.21.15.16:0/1674725883 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a6404ebb0 con 0x7f1a6406e280 2023-10-16T13:48:41.665 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.651+0000 7f1a5b7fe700 1 -- 172.21.15.16:0/1674725883 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 161) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f1a4c0073e0 con 0x7f1a6406e280 2023-10-16T13:48:41.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.651+0000 7f1a5b7fe700 1 --2- 172.21.15.16:0/1674725883 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f1a4406ada0 0x7f1a4406d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:41.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.651+0000 7f1a5b7fe700 1 -- 172.21.15.16:0/1674725883 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(266..266 src has 1..266) v4 ==== 6587+0+0 (secure 0 0 0) 0x7f1a4c08a6a0 con 0x7f1a6406e280 2023-10-16T13:48:41.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.651+0000 7f1a62d9d700 1 --2- 172.21.15.16:0/1674725883 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f1a4406ada0 0x7f1a4406d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:41.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.652+0000 7f1a62d9d700 1 --2- 172.21.15.16:0/1674725883 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f1a4406ada0 0x7f1a4406d260 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f1a54000fb0 tx=0x7f1a54009040 comp rx=0 tx=0).ready entity=mgr.13239 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:41.666 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.656+0000 7f1a5b7fe700 1 -- 172.21.15.16:0/1674725883 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1a4c055840 con 0x7f1a6406e280 2023-10-16T13:48:41.887 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.885+0000 7f1a690a2700 1 -- 172.21.15.16:0/1674725883 --> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1a640cc9f0 con 0x7f1a4406ada0 2023-10-16T13:48:41.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.895+0000 7f1a5b7fe700 1 -- 172.21.15.16:0/1674725883 <== mgr.13239 v2:172.21.15.16:6832/2131389174 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+72 (secure 0 0 0) 0x7f1a54001650 con 0x7f1a4406ada0 2023-10-16T13:48:41.897 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:48:41.897 INFO:teuthology.orchestra.run.smithi016.stdout:{"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{"m": 1}]} 2023-10-16T13:48:41.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.897+0000 7f1a690a2700 1 -- 172.21.15.16:0/1674725883 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f1a4406ada0 msgr2=0x7f1a4406d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:41.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.897+0000 7f1a690a2700 1 --2- 172.21.15.16:0/1674725883 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f1a4406ada0 0x7f1a4406d260 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f1a54000fb0 tx=0x7f1a54009040 comp rx=0 tx=0).stop 2023-10-16T13:48:41.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.897+0000 7f1a690a2700 1 -- 172.21.15.16:0/1674725883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a6406e280 msgr2=0x7f1a640b57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:41.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.897+0000 7f1a690a2700 1 --2- 172.21.15.16:0/1674725883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a6406e280 0x7f1a640b57b0 secure :-1 s=READY pgs=3062 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c006a90 tx=0x7f1a4c0012d0 comp rx=0 tx=0).stop 2023-10-16T13:48:41.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.897+0000 7f1a690a2700 1 -- 172.21.15.16:0/1674725883 shutdown_connections 2023-10-16T13:48:41.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.897+0000 7f1a690a2700 1 --2- 172.21.15.16:0/1674725883 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f1a4406ada0 0x7f1a4406d260 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:41.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.897+0000 7f1a690a2700 1 --2- 172.21.15.16:0/1674725883 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1a640b2ff0 0x7f1a640b34a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:41.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.897+0000 7f1a690a2700 1 --2- 172.21.15.16:0/1674725883 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1a50004120 0x7f1a64155d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:41.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.897+0000 7f1a690a2700 1 --2- 172.21.15.16:0/1674725883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a6406e280 0x7f1a640b57b0 unknown :-1 s=CLOSED pgs=3062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:41.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.897+0000 7f1a690a2700 1 -- 172.21.15.16:0/1674725883 >> 172.21.15.16:0/1674725883 conn(0x7f1a640b1f40 msgr2=0x7f1a640cb770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:41.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.897+0000 7f1a690a2700 1 -- 172.21.15.16:0/1674725883 shutdown_connections 2023-10-16T13:48:41.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:41.897+0000 7f1a690a2700 1 -- 172.21.15.16:0/1674725883 wait complete. 2023-10-16T13:48:41.912 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{"m": 1}]} 2023-10-16T13:48:41.912 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1m'], 'retention': [{'m': 1}]} 2023-10-16T13:48:41.912 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2023-10-16T13:48:42.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.041+0000 7f9ee55f7700 1 Processor -- start 2023-10-16T13:48:42.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.041+0000 7f9ee55f7700 1 -- start start 2023-10-16T13:48:42.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.042+0000 7f9ee55f7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9ee006a7e0 0x7f9ee006ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:42.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.042+0000 7f9ee55f7700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ee015ecb0 con 0x7f9ee015b2c0 2023-10-16T13:48:42.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.042+0000 7f9ee55f7700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ee015b6a0 con 0x7f9ee006e570 2023-10-16T13:48:42.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.042+0000 7f9ee55f7700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ee006b190 con 0x7f9ee006a7e0 2023-10-16T13:48:42.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.042+0000 7f9edf7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9ee006a7e0 0x7f9ee006ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:42.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.042+0000 7f9edeffd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f9ee015b2c0 0x7f9ee0161340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:47372/0 (socket says 172.21.15.16:47372) 2023-10-16T13:48:42.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.042+0000 7f9edeffd700 1 -- 172.21.15.16:0/744386900 learned_addr learned my addr 172.21.15.16:0/744386900 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:48:42.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.042+0000 7f9edf7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9ee006a7e0 0x7f9ee006ac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58102/0 (socket says 172.21.15.16:58102) 2023-10-16T13:48:42.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.043+0000 7f9edf7fe700 1 -- 172.21.15.16:0/744386900 >> v1:172.21.15.49:6789/0 conn(0x7f9ee006e570 legacy=0x7f9ee015e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:42.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.043+0000 7f9edf7fe700 1 -- 172.21.15.16:0/744386900 >> v1:172.21.15.16:6789/0 conn(0x7f9ee015b2c0 legacy=0x7f9ee0161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:42.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.043+0000 7f9edf7fe700 1 -- 172.21.15.16:0/744386900 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ee0161b20 con 0x7f9ee006a7e0 2023-10-16T13:48:42.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.043+0000 7f9edf7fe700 1 --2- 172.21.15.16:0/744386900 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ee006a7e0 0x7f9ee006ac50 secure :-1 s=READY pgs=3066 cs=0 l=1 rev1=1 crypto rx=0x7f9ed8009d90 tx=0x7f9ed800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e3baabb459eb3f5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:42.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.043+0000 7f9eddffb700 1 -- 172.21.15.16:0/744386900 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed800e070 con 0x7f9ee006a7e0 2023-10-16T13:48:42.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.043+0000 7f9eddffb700 1 -- 172.21.15.16:0/744386900 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ed800bbd0 con 0x7f9ee006a7e0 2023-10-16T13:48:42.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.043+0000 7f9eddffb700 1 -- 172.21.15.16:0/744386900 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed8012430 con 0x7f9ee006a7e0 2023-10-16T13:48:42.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.044+0000 7f9ee55f7700 1 -- 172.21.15.16:0/744386900 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ee006a7e0 msgr2=0x7f9ee006ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:42.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.044+0000 7f9ee55f7700 1 --2- 172.21.15.16:0/744386900 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ee006a7e0 0x7f9ee006ac50 secure :-1 s=READY pgs=3066 cs=0 l=1 rev1=1 crypto rx=0x7f9ed8009d90 tx=0x7f9ed800b330 comp rx=0 tx=0).stop 2023-10-16T13:48:42.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.044+0000 7f9ee55f7700 1 -- 172.21.15.16:0/744386900 shutdown_connections 2023-10-16T13:48:42.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.044+0000 7f9ee55f7700 1 --2- 172.21.15.16:0/744386900 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ee006a7e0 0x7f9ee006ac50 unknown :-1 s=CLOSED pgs=3066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:42.050 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.044+0000 7f9ee55f7700 1 -- 172.21.15.16:0/744386900 >> 172.21.15.16:0/744386900 conn(0x7f9ee00b1f40 msgr2=0x7f9ee00b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:42.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.044+0000 7f9ee55f7700 1 -- 172.21.15.16:0/744386900 shutdown_connections 2023-10-16T13:48:42.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.044+0000 7f9ee55f7700 1 -- 172.21.15.16:0/744386900 wait complete. 2023-10-16T13:48:42.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.045+0000 7f9ee55f7700 1 Processor -- start 2023-10-16T13:48:42.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.045+0000 7f9ee55f7700 1 -- start start 2023-10-16T13:48:42.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.046+0000 7f9ee55f7700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9ee006a7e0 0x7f9ee00b46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:42.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.046+0000 7f9ee55f7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ee006e570 0x7f9ee00c3b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:42.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.046+0000 7f9ee55f7700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9ee015b2c0 0x7f9ee014f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:42.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.046+0000 7f9ee55f7700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee00c6690 con 0x7f9ee006e570 2023-10-16T13:48:42.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.046+0000 7f9ee55f7700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee00c6970 con 0x7f9ee006a7e0 2023-10-16T13:48:42.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.046+0000 7f9ee55f7700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee00c6c50 con 0x7f9ee015b2c0 2023-10-16T13:48:42.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.046+0000 7f9ede7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ee006e570 0x7f9ee00c3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:42.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.046+0000 7f9ede7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ee006e570 0x7f9ee00c3b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58106/0 (socket says 172.21.15.16:58106) 2023-10-16T13:48:42.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.046+0000 7f9ede7fc700 1 -- 172.21.15.16:0/2363527885 learned_addr learned my addr 172.21.15.16:0/2363527885 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:42.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.046+0000 7f9edeffd700 1 --2- 172.21.15.16:0/2363527885 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9ee006a7e0 0x7f9ee00b46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:42.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.046+0000 7f9edf7fe700 1 --2- 172.21.15.16:0/2363527885 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9ee015b2c0 0x7f9ee014f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:42.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.046+0000 7f9ede7fc700 1 -- 172.21.15.16:0/2363527885 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9ee015b2c0 msgr2=0x7f9ee014f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:42.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.046+0000 7f9ede7fc700 1 --2- 172.21.15.16:0/2363527885 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9ee015b2c0 0x7f9ee014f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:42.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.046+0000 7f9ede7fc700 1 -- 172.21.15.16:0/2363527885 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9ee006a7e0 msgr2=0x7f9ee00b46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:42.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.046+0000 7f9ede7fc700 1 --2- 172.21.15.16:0/2363527885 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9ee006a7e0 0x7f9ee00b46e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:42.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.047+0000 7f9ede7fc700 1 -- 172.21.15.16:0/2363527885 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ed8009a70 con 0x7f9ee006e570 2023-10-16T13:48:42.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.047+0000 7f9ede7fc700 1 --2- 172.21.15.16:0/2363527885 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ee006e570 0x7f9ee00c3b90 secure :-1 s=READY pgs=3067 cs=0 l=1 rev1=1 crypto rx=0x7f9ed400c4e0 tx=0x7f9ed400c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:42.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.047+0000 7f9ec77fe700 1 -- 172.21.15.16:0/2363527885 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed40052d0 con 0x7f9ee006e570 2023-10-16T13:48:42.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.047+0000 7f9ec77fe700 1 -- 172.21.15.16:0/2363527885 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ed4005430 con 0x7f9ee006e570 2023-10-16T13:48:42.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.047+0000 7f9ee55f7700 1 -- 172.21.15.16:0/2363527885 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ee00c6f30 con 0x7f9ee006e570 2023-10-16T13:48:42.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.047+0000 7f9ec77fe700 1 -- 172.21.15.16:0/2363527885 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed40038c0 con 0x7f9ee006e570 2023-10-16T13:48:42.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.047+0000 7f9ee55f7700 1 -- 172.21.15.16:0/2363527885 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ee00c62f0 con 0x7f9ee006e570 2023-10-16T13:48:42.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.048+0000 7f9ee55f7700 1 -- 172.21.15.16:0/2363527885 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ee015e060 con 0x7f9ee006e570 2023-10-16T13:48:42.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.048+0000 7f9ec77fe700 1 -- 172.21.15.16:0/2363527885 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 161) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f9ed4003a20 con 0x7f9ee006e570 2023-10-16T13:48:42.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.049+0000 7f9ec77fe700 1 --2- 172.21.15.16:0/2363527885 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f9ecc06ada0 0x7f9ecc06d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:42.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.049+0000 7f9ec77fe700 1 -- 172.21.15.16:0/2363527885 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(266..266 src has 1..266) v4 ==== 6587+0+0 (secure 0 0 0) 0x7f9ed4088900 con 0x7f9ee006e570 2023-10-16T13:48:42.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.049+0000 7f9edeffd700 1 --2- 172.21.15.16:0/2363527885 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f9ecc06ada0 0x7f9ecc06d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:42.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.049+0000 7f9edeffd700 1 --2- 172.21.15.16:0/2363527885 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f9ecc06ada0 0x7f9ecc06d260 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f9ed0005910 tx=0x7f9ed0009040 comp rx=0 tx=0).ready entity=mgr.13239 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:42.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.054+0000 7f9ec77fe700 1 -- 172.21.15.16:0/2363527885 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9ee015e060 con 0x7f9ee006e570 2023-10-16T13:48:42.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.287+0000 7f9ee55f7700 1 -- 172.21.15.16:0/2363527885 --> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f9ee00c6ae0 con 0x7f9ecc06ada0 2023-10-16T13:48:42.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.353+0000 7f9ec77fe700 1 -- 172.21.15.16:0/2363527885 <== mgr.13239 v2:172.21.15.16:6832/2131389174 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f9ee00c6ae0 con 0x7f9ecc06ada0 2023-10-16T13:48:42.356 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule removed for path snap_test_dir1 2023-10-16T13:48:42.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.355+0000 7f9ee55f7700 1 -- 172.21.15.16:0/2363527885 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f9ecc06ada0 msgr2=0x7f9ecc06d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:42.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.355+0000 7f9ee55f7700 1 --2- 172.21.15.16:0/2363527885 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f9ecc06ada0 0x7f9ecc06d260 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f9ed0005910 tx=0x7f9ed0009040 comp rx=0 tx=0).stop 2023-10-16T13:48:42.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.355+0000 7f9ee55f7700 1 -- 172.21.15.16:0/2363527885 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ee006e570 msgr2=0x7f9ee00c3b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:42.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.355+0000 7f9ee55f7700 1 --2- 172.21.15.16:0/2363527885 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ee006e570 0x7f9ee00c3b90 secure :-1 s=READY pgs=3067 cs=0 l=1 rev1=1 crypto rx=0x7f9ed400c4e0 tx=0x7f9ed400c7f0 comp rx=0 tx=0).stop 2023-10-16T13:48:42.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.355+0000 7f9ee55f7700 1 -- 172.21.15.16:0/2363527885 shutdown_connections 2023-10-16T13:48:42.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.355+0000 7f9ee55f7700 1 --2- 172.21.15.16:0/2363527885 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f9ecc06ada0 0x7f9ecc06d260 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:42.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.355+0000 7f9ee55f7700 1 --2- 172.21.15.16:0/2363527885 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9ee015b2c0 0x7f9ee014f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:42.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.355+0000 7f9ee55f7700 1 --2- 172.21.15.16:0/2363527885 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9ee006a7e0 0x7f9ee00b46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:42.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.355+0000 7f9ee55f7700 1 --2- 172.21.15.16:0/2363527885 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ee006e570 0x7f9ee00c3b90 unknown :-1 s=CLOSED pgs=3067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:42.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.355+0000 7f9ee55f7700 1 -- 172.21.15.16:0/2363527885 >> 172.21.15.16:0/2363527885 conn(0x7f9ee00b1f40 msgr2=0x7f9ee015e4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:42.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.355+0000 7f9ee55f7700 1 -- 172.21.15.16:0/2363527885 shutdown_connections 2023-10-16T13:48:42.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.355+0000 7f9ee55f7700 1 -- 172.21.15.16:0/2363527885 wait complete. 2023-10-16T13:48:42.370 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2023-10-16T13:48:42.371 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:48:42.371 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-16T13:48:42.389 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:48:42.389 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2023-10-16T13:48:42.449 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-16T13:48:42.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.615+0000 7fee75461700 1 Processor -- start 2023-10-16T13:48:42.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.615+0000 7fee75461700 1 -- start start 2023-10-16T13:48:42.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.615+0000 7fee75461700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fee7015b9d0 0x7fee700cbcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:42.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.615+0000 7fee75461700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fee700cc230 0x7fee700b2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:42.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.615+0000 7fee75461700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee7015bdb0 con 0x7fee700b34e0 2023-10-16T13:48:42.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.615+0000 7fee75461700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee700b3970 con 0x7fee7015b9d0 2023-10-16T13:48:42.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.615+0000 7fee75461700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee700cc7b0 con 0x7fee700cc230 2023-10-16T13:48:42.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.616+0000 7fee6effd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fee7015b9d0 0x7fee700cbcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:42.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.616+0000 7fee6effd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fee7015b9d0 0x7fee700cbcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58112/0 (socket says 172.21.15.16:58112) 2023-10-16T13:48:42.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.616+0000 7fee6effd700 1 -- 172.21.15.16:0/2664473575 learned_addr learned my addr 172.21.15.16:0/2664473575 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:42.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.616+0000 7fee6e7fc700 1 --2- 172.21.15.16:0/2664473575 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fee700cc230 0x7fee700b2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:42.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.616+0000 7fee6effd700 1 -- 172.21.15.16:0/2664473575 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fee700cc230 msgr2=0x7fee700b2fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:42.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.616+0000 7fee6effd700 1 --2- 172.21.15.16:0/2664473575 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fee700cc230 0x7fee700b2fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:42.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.616+0000 7fee6effd700 1 -- 172.21.15.16:0/2664473575 >> v1:172.21.15.49:6789/0 conn(0x7fee700b34e0 legacy=0x7fee70070530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:42.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.616+0000 7fee6effd700 1 -- 172.21.15.16:0/2664473575 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee7015d110 con 0x7fee7015b9d0 2023-10-16T13:48:42.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.616+0000 7fee6effd700 1 --2- 172.21.15.16:0/2664473575 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee7015b9d0 0x7fee700cbcf0 secure :-1 s=READY pgs=3068 cs=0 l=1 rev1=1 crypto rx=0x7fee600098f0 tx=0x7fee60009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2d4055e500bff81 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:42.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.616+0000 7fee6dffb700 1 -- 172.21.15.16:0/2664473575 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee6000b8a0 con 0x7fee7015b9d0 2023-10-16T13:48:42.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.616+0000 7fee6dffb700 1 -- 172.21.15.16:0/2664473575 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee6000ba00 con 0x7fee7015b9d0 2023-10-16T13:48:42.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.616+0000 7fee6dffb700 1 -- 172.21.15.16:0/2664473575 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee6000bd90 con 0x7fee7015b9d0 2023-10-16T13:48:42.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.617+0000 7fee75461700 1 -- 172.21.15.16:0/2664473575 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee7015b9d0 msgr2=0x7fee700cbcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:42.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.617+0000 7fee75461700 1 --2- 172.21.15.16:0/2664473575 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee7015b9d0 0x7fee700cbcf0 secure :-1 s=READY pgs=3068 cs=0 l=1 rev1=1 crypto rx=0x7fee600098f0 tx=0x7fee60009c00 comp rx=0 tx=0).stop 2023-10-16T13:48:42.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.617+0000 7fee75461700 1 -- 172.21.15.16:0/2664473575 shutdown_connections 2023-10-16T13:48:42.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.617+0000 7fee75461700 1 --2- 172.21.15.16:0/2664473575 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee7015b9d0 0x7fee700cbcf0 unknown :-1 s=CLOSED pgs=3068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:42.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.617+0000 7fee75461700 1 --2- 172.21.15.16:0/2664473575 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fee700cc230 0x7fee700b2fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:42.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.617+0000 7fee75461700 1 -- 172.21.15.16:0/2664473575 >> 172.21.15.16:0/2664473575 conn(0x7fee700cdfa0 msgr2=0x7fee700ce3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:42.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.617+0000 7fee75461700 1 -- 172.21.15.16:0/2664473575 shutdown_connections 2023-10-16T13:48:42.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.617+0000 7fee75461700 1 -- 172.21.15.16:0/2664473575 wait complete. 2023-10-16T13:48:42.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.619+0000 7fee75461700 1 Processor -- start 2023-10-16T13:48:42.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.619+0000 7fee75461700 1 -- start start 2023-10-16T13:48:42.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.620+0000 7fee75461700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fee700b34e0 0x7fee700b17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:42.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.620+0000 7fee75461700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee700cc230 0x7fee70154df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:42.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.620+0000 7fee75461700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fee7015b9d0 0x7fee70155330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:42.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.620+0000 7fee75461700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee701f2250 con 0x7fee700cc230 2023-10-16T13:48:42.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.620+0000 7fee75461700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee701f2530 con 0x7fee700b34e0 2023-10-16T13:48:42.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.620+0000 7fee75461700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee701f2810 con 0x7fee7015b9d0 2023-10-16T13:48:42.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.620+0000 7fee6e7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee700cc230 0x7fee70154df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:42.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.620+0000 7fee6e7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee700cc230 0x7fee70154df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58120/0 (socket says 172.21.15.16:58120) 2023-10-16T13:48:42.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.620+0000 7fee6e7fc700 1 -- 172.21.15.16:0/1779812494 learned_addr learned my addr 172.21.15.16:0/1779812494 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:42.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.620+0000 7fee6effd700 1 --2- 172.21.15.16:0/1779812494 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fee700b34e0 0x7fee700b17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:42.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.620+0000 7fee6f7fe700 1 --2- 172.21.15.16:0/1779812494 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fee7015b9d0 0x7fee70155330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:42.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.620+0000 7fee6e7fc700 1 -- 172.21.15.16:0/1779812494 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fee7015b9d0 msgr2=0x7fee70155330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:42.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.620+0000 7fee6e7fc700 1 --2- 172.21.15.16:0/1779812494 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fee7015b9d0 0x7fee70155330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:42.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.620+0000 7fee6e7fc700 1 -- 172.21.15.16:0/1779812494 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fee700b34e0 msgr2=0x7fee700b17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:42.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.620+0000 7fee6e7fc700 1 --2- 172.21.15.16:0/1779812494 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fee700b34e0 0x7fee700b17a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:42.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.620+0000 7fee6e7fc700 1 -- 172.21.15.16:0/1779812494 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee60009550 con 0x7fee700cc230 2023-10-16T13:48:42.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.620+0000 7fee6e7fc700 1 --2- 172.21.15.16:0/1779812494 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee700cc230 0x7fee70154df0 secure :-1 s=READY pgs=3069 cs=0 l=1 rev1=1 crypto rx=0x7fee6400e810 tx=0x7fee6400ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:42.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.620+0000 7fee577fe700 1 -- 172.21.15.16:0/1779812494 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee64013610 con 0x7fee700cc230 2023-10-16T13:48:42.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.620+0000 7fee577fe700 1 -- 172.21.15.16:0/1779812494 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee64013770 con 0x7fee700cc230 2023-10-16T13:48:42.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.620+0000 7fee75461700 1 -- 172.21.15.16:0/1779812494 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee701f2af0 con 0x7fee700cc230 2023-10-16T13:48:42.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.621+0000 7fee577fe700 1 -- 172.21.15.16:0/1779812494 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee64013b20 con 0x7fee700cc230 2023-10-16T13:48:42.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.621+0000 7fee75461700 1 -- 172.21.15.16:0/1779812494 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee70152730 con 0x7fee700cc230 2023-10-16T13:48:42.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.621+0000 7fee75461700 1 -- 172.21.15.16:0/1779812494 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee701f3010 con 0x7fee700cc230 2023-10-16T13:48:42.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.622+0000 7fee577fe700 1 -- 172.21.15.16:0/1779812494 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 161) v1 ==== 89084+0+0 (secure 0 0 0) 0x7fee64013c80 con 0x7fee700cc230 2023-10-16T13:48:42.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.622+0000 7fee577fe700 1 --2- 172.21.15.16:0/1779812494 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7fee5c06ada0 0x7fee5c06d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:42.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.622+0000 7fee577fe700 1 -- 172.21.15.16:0/1779812494 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(266..266 src has 1..266) v4 ==== 6587+0+0 (secure 0 0 0) 0x7fee64089310 con 0x7fee700cc230 2023-10-16T13:48:42.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.622+0000 7fee6effd700 1 --2- 172.21.15.16:0/1779812494 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7fee5c06ada0 0x7fee5c06d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:42.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.623+0000 7fee6effd700 1 --2- 172.21.15.16:0/1779812494 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7fee5c06ada0 0x7fee5c06d260 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fee60014040 tx=0x7fee60008030 comp rx=0 tx=0).ready entity=mgr.13239 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:42.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.627+0000 7fee577fe700 1 -- 172.21.15.16:0/1779812494 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fee701f3010 con 0x7fee700cc230 2023-10-16T13:48:42.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.889+0000 7fee75461700 1 -- 172.21.15.16:0/1779812494 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7fee701f3010 con 0x7fee700cc230 2023-10-16T13:48:42.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.987+0000 7fee577fe700 1 -- 172.21.15.16:0/1779812494 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v162) v1 ==== 93+0+0 (secure 0 0 0) 0x7fee701f3010 con 0x7fee700cc230 2023-10-16T13:48:42.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.990+0000 7fee75461700 1 -- 172.21.15.16:0/1779812494 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7fee5c06ada0 msgr2=0x7fee5c06d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:42.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.990+0000 7fee75461700 1 --2- 172.21.15.16:0/1779812494 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7fee5c06ada0 0x7fee5c06d260 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fee60014040 tx=0x7fee60008030 comp rx=0 tx=0).stop 2023-10-16T13:48:42.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.990+0000 7fee75461700 1 -- 172.21.15.16:0/1779812494 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee700cc230 msgr2=0x7fee70154df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:42.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.990+0000 7fee75461700 1 --2- 172.21.15.16:0/1779812494 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee700cc230 0x7fee70154df0 secure :-1 s=READY pgs=3069 cs=0 l=1 rev1=1 crypto rx=0x7fee6400e810 tx=0x7fee6400ebd0 comp rx=0 tx=0).stop 2023-10-16T13:48:42.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.991+0000 7fee75461700 1 -- 172.21.15.16:0/1779812494 shutdown_connections 2023-10-16T13:48:42.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.991+0000 7fee75461700 1 --2- 172.21.15.16:0/1779812494 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7fee5c06ada0 0x7fee5c06d260 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fee60014040 tx=0x7fee60008030 comp rx=0 tx=0).stop 2023-10-16T13:48:42.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.991+0000 7fee75461700 1 --2- 172.21.15.16:0/1779812494 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fee7015b9d0 0x7fee70155330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:42.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.991+0000 7fee75461700 1 --2- 172.21.15.16:0/1779812494 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fee700b34e0 0x7fee700b17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:42.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.991+0000 7fee75461700 1 --2- 172.21.15.16:0/1779812494 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee700cc230 0x7fee70154df0 unknown :-1 s=CLOSED pgs=3069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:42.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.991+0000 7fee75461700 1 -- 172.21.15.16:0/1779812494 >> 172.21.15.16:0/1779812494 conn(0x7fee700cdfa0 msgr2=0x7fee70159cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:42.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.992+0000 7fee75461700 1 -- 172.21.15.16:0/1779812494 shutdown_connections 2023-10-16T13:48:42.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:42.992+0000 7fee75461700 1 -- 172.21.15.16:0/1779812494 wait complete. 2023-10-16T13:48:43.008 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T13:48:43.008 DEBUG:teuthology.orchestra.run.smithi016:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T13:48:43.052 DEBUG:teuthology.orchestra.run.smithi049:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T13:48:43.093 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T13:48:43.093 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T13:48:43.094 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:48:43.111 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T13:48:43.112 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:48:43.139 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:43.137+0000 7fd1d1dca200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T13:48:43.145 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:48:43.142+0000 7f2fb0253200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T13:48:43.156 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:48:43.208 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:43.206+0000 7fd1d1dca200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T13:48:43.214 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:48:43.211+0000 7f2fb0253200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T13:48:43.285 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:43.283+0000 7fd1d1dca200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T13:48:43.289 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:48:43.286+0000 7f2fb0253200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T13:48:43.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.320+0000 7f2bd09a8700 1 Processor -- start 2023-10-16T13:48:43.323 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.321+0000 7f2bd09a8700 1 -- start start 2023-10-16T13:48:43.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.321+0000 7f2bd09a8700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2bcc0d2f70 0x7f2bcc0d2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:43.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.322+0000 7f2bd09a8700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bcc0ce400 con 0x7f2bcc15d900 2023-10-16T13:48:43.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.322+0000 7f2bd09a8700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bcc0cee40 con 0x7f2bcc0ce9f0 2023-10-16T13:48:43.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.322+0000 7f2bd09a8700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bcc15dce0 con 0x7f2bcc0d2f70 2023-10-16T13:48:43.325 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.322+0000 7f2bca59c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2bcc15d900 0x7f2bcc0d2860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:47460/0 (socket says 172.21.15.16:47460) 2023-10-16T13:48:43.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.322+0000 7f2bc9d9b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2bcc0d2f70 0x7f2bcc0d2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:43.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.322+0000 7f2bca59c700 1 -- 172.21.15.16:0/37101362 learned_addr learned my addr 172.21.15.16:0/37101362 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:48:43.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.322+0000 7f2bc9d9b700 1 -- 172.21.15.16:0/37101362 >> v1:172.21.15.49:6789/0 conn(0x7f2bcc0ce9f0 legacy=0x7f2bcc0b6790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:43.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.322+0000 7f2bc9d9b700 1 -- 172.21.15.16:0/37101362 >> v1:172.21.15.16:6789/0 conn(0x7f2bcc15d900 legacy=0x7f2bcc0d2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:43.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.322+0000 7f2bc9d9b700 1 -- 172.21.15.16:0/37101362 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bcc0d1280 con 0x7f2bcc0d2f70 2023-10-16T13:48:43.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.322+0000 7f2bc9d9b700 1 --2- 172.21.15.16:0/37101362 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2bcc0d2f70 0x7f2bcc0d2040 secure :-1 s=READY pgs=3074 cs=0 l=1 rev1=1 crypto rx=0x7f2bc0009840 tx=0x7f2bc0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e00cf91ce73d4fb1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:43.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.322+0000 7f2bc959a700 1 -- 172.21.15.16:0/37101362 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc0011830 con 0x7f2bcc0d2f70 2023-10-16T13:48:43.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.322+0000 7f2bc959a700 1 -- 172.21.15.16:0/37101362 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bc0011990 con 0x7f2bcc0d2f70 2023-10-16T13:48:43.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.322+0000 7f2bc959a700 1 -- 172.21.15.16:0/37101362 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc0011d20 con 0x7f2bcc0d2f70 2023-10-16T13:48:43.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.323+0000 7f2bd09a8700 1 -- 172.21.15.16:0/37101362 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2bcc0d2f70 msgr2=0x7f2bcc0d2040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:43.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.323+0000 7f2bd09a8700 1 --2- 172.21.15.16:0/37101362 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2bcc0d2f70 0x7f2bcc0d2040 secure :-1 s=READY pgs=3074 cs=0 l=1 rev1=1 crypto rx=0x7f2bc0009840 tx=0x7f2bc0009b50 comp rx=0 tx=0).stop 2023-10-16T13:48:43.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.323+0000 7f2bd09a8700 1 -- 172.21.15.16:0/37101362 shutdown_connections 2023-10-16T13:48:43.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.323+0000 7f2bd09a8700 1 --2- 172.21.15.16:0/37101362 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2bcc0d2f70 0x7f2bcc0d2040 unknown :-1 s=CLOSED pgs=3074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:43.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.323+0000 7f2bd09a8700 1 -- 172.21.15.16:0/37101362 >> 172.21.15.16:0/37101362 conn(0x7f2bcc072330 msgr2=0x7f2bcc072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:43.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.323+0000 7f2bd09a8700 1 -- 172.21.15.16:0/37101362 shutdown_connections 2023-10-16T13:48:43.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.323+0000 7f2bd09a8700 1 -- 172.21.15.16:0/37101362 wait complete. 2023-10-16T13:48:43.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.324+0000 7f2bd09a8700 1 Processor -- start 2023-10-16T13:48:43.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.324+0000 7f2bd09a8700 1 -- start start 2023-10-16T13:48:43.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.325+0000 7f2bd09a8700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2bcc0ce9f0 0x7f2bcc0b53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:43.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.325+0000 7f2bd09a8700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2bcc0d2f70 0x7f2bcc0c9700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:43.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.325+0000 7f2bd09a8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2bcc15d900 0x7f2bcc151110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:43.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.325+0000 7f2bd09a8700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bcc0c8a50 con 0x7f2bcc15d900 2023-10-16T13:48:43.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.325+0000 7f2bd09a8700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bcc0c7c00 con 0x7f2bcc0ce9f0 2023-10-16T13:48:43.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.325+0000 7f2bd09a8700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bcc0c7e10 con 0x7f2bcc0d2f70 2023-10-16T13:48:43.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.325+0000 7f2bcad9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2bcc15d900 0x7f2bcc151110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:43.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.325+0000 7f2bcad9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2bcc15d900 0x7f2bcc151110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58196/0 (socket says 172.21.15.16:58196) 2023-10-16T13:48:43.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.325+0000 7f2bca59c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2bcc0ce9f0 0x7f2bcc0b53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:43.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.325+0000 7f2bcad9d700 1 -- 172.21.15.16:0/68357293 learned_addr learned my addr 172.21.15.16:0/68357293 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:43.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.325+0000 7f2bc9d9b700 1 --2- 172.21.15.16:0/68357293 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2bcc0d2f70 0x7f2bcc0c9700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:43.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.326+0000 7f2bcad9d700 1 -- 172.21.15.16:0/68357293 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2bcc0d2f70 msgr2=0x7f2bcc0c9700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:43.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.326+0000 7f2bcad9d700 1 --2- 172.21.15.16:0/68357293 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2bcc0d2f70 0x7f2bcc0c9700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:43.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.326+0000 7f2bcad9d700 1 -- 172.21.15.16:0/68357293 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2bcc0ce9f0 msgr2=0x7f2bcc0b53c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:43.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.326+0000 7f2bcad9d700 1 --2- 172.21.15.16:0/68357293 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2bcc0ce9f0 0x7f2bcc0b53c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:43.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.326+0000 7f2bcad9d700 1 -- 172.21.15.16:0/68357293 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bc00094a0 con 0x7f2bcc15d900 2023-10-16T13:48:43.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.326+0000 7f2bcad9d700 1 --2- 172.21.15.16:0/68357293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2bcc15d900 0x7f2bcc151110 secure :-1 s=READY pgs=3075 cs=0 l=1 rev1=1 crypto rx=0x7f2bc400cc50 tx=0x7f2bc400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:43.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.326+0000 7f2bbaffd700 1 -- 172.21.15.16:0/68357293 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc4003c00 con 0x7f2bcc15d900 2023-10-16T13:48:43.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.326+0000 7f2bbaffd700 1 -- 172.21.15.16:0/68357293 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bc4003d60 con 0x7f2bcc15d900 2023-10-16T13:48:43.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.326+0000 7f2bd09a8700 1 -- 172.21.15.16:0/68357293 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bcc0c80b0 con 0x7f2bcc15d900 2023-10-16T13:48:43.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.326+0000 7f2bbaffd700 1 -- 172.21.15.16:0/68357293 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc400aab0 con 0x7f2bcc15d900 2023-10-16T13:48:43.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.326+0000 7f2bd09a8700 1 -- 172.21.15.16:0/68357293 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bcc0c8670 con 0x7f2bcc15d900 2023-10-16T13:48:43.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.327+0000 7f2bd09a8700 1 -- 172.21.15.16:0/68357293 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bcc0c8b90 con 0x7f2bcc15d900 2023-10-16T13:48:43.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.328+0000 7f2bbaffd700 1 -- 172.21.15.16:0/68357293 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 162) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f2bc400ac10 con 0x7f2bcc15d900 2023-10-16T13:48:43.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.328+0000 7f2bbaffd700 1 --2- 172.21.15.16:0/68357293 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f2bac06b020 0x7f2bac06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:43.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.328+0000 7f2bca59c700 1 -- 172.21.15.16:0/68357293 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f2bac06b020 msgr2=0x7f2bac06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2131389174 2023-10-16T13:48:43.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.328+0000 7f2bbaffd700 1 -- 172.21.15.16:0/68357293 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(266..266 src has 1..266) v4 ==== 6587+0+0 (secure 0 0 0) 0x7f2bc408a5a0 con 0x7f2bcc15d900 2023-10-16T13:48:43.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.328+0000 7f2bca59c700 1 --2- 172.21.15.16:0/68357293 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f2bac06b020 0x7f2bac06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:48:43.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.332+0000 7f2bbaffd700 1 -- 172.21.15.16:0/68357293 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2bc40551c0 con 0x7f2bcc15d900 2023-10-16T13:48:43.372 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:48:43.369+0000 7f2fb0253200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T13:48:43.372 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:43.370+0000 7fd1d1dca200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T13:48:43.521 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:48:43.518+0000 7f2fb0253200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T13:48:43.522 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:43.519+0000 7fd1d1dca200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T13:48:43.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.529+0000 7f2bca59c700 1 -- 172.21.15.16:0/68357293 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f2bac06b020 msgr2=0x7f2bac06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2131389174 2023-10-16T13:48:43.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.529+0000 7f2bca59c700 1 --2- 172.21.15.16:0/68357293 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f2bac06b020 0x7f2bac06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:48:43.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.540+0000 7f2bd09a8700 1 -- 172.21.15.16:0/68357293 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2bcc04ebb0 con 0x7f2bcc15d900 2023-10-16T13:48:43.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.541+0000 7f2bbaffd700 1 -- 172.21.15.16:0/68357293 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v266) v1 ==== 74+0+17440 (secure 0 0 0) 0x7f2bc40587e0 con 0x7f2bcc15d900 2023-10-16T13:48:43.544 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:48:43.544 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":266,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:48:39.972754+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":36,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_metadata","create_time":"2023-10-16T13:48:23.348668+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"264","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_data","create_time":"2023-10-16T13:48:23.799968+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"265","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/3313712117":"2023-10-17T13:48:39.972713+0000","172.21.15.16:0/2202829273":"2023-10-17T13:48:39.972713+0000","172.21.15.16:0/2777724108":"2023-10-17T13:48:39.972713+0000","172.21.15.16:0/2088077503":"2023-10-17T13:48:39.972713+0000","172.21.15.16:0/4096007345":"2023-10-17T13:48:39.972713+0000","172.21.15.16:0/3000870029":"2023-10-17T13:48:39.972713+0000","172.21.15.16:0/3923653515":"2023-10-17T13:48:39.972713+0000","172.21.15.16:0/56667078":"2023-10-17T13:48:39.972713+0000","172.21.15.16:6833/1275231846":"2023-10-17T13:48:39.972713+0000","172.21.15.16:6832/1275231846":"2023-10-17T13:48:39.972713+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:48:43.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.543+0000 7f2bd09a8700 1 -- 172.21.15.16:0/68357293 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f2bac06b020 msgr2=0x7f2bac06d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:48:43.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.543+0000 7f2bd09a8700 1 --2- 172.21.15.16:0/68357293 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f2bac06b020 0x7f2bac06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:43.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.543+0000 7f2bd09a8700 1 -- 172.21.15.16:0/68357293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2bcc15d900 msgr2=0x7f2bcc151110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:43.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.543+0000 7f2bd09a8700 1 --2- 172.21.15.16:0/68357293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2bcc15d900 0x7f2bcc151110 secure :-1 s=READY pgs=3075 cs=0 l=1 rev1=1 crypto rx=0x7f2bc400cc50 tx=0x7f2bc400cf60 comp rx=0 tx=0).stop 2023-10-16T13:48:43.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.543+0000 7f2bd09a8700 1 -- 172.21.15.16:0/68357293 shutdown_connections 2023-10-16T13:48:43.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.543+0000 7f2bd09a8700 1 --2- 172.21.15.16:0/68357293 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f2bac06b020 0x7f2bac06d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:43.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.543+0000 7f2bd09a8700 1 --2- 172.21.15.16:0/68357293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2bcc15d900 0x7f2bcc151110 unknown :-1 s=CLOSED pgs=3075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:43.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.543+0000 7f2bd09a8700 1 --2- 172.21.15.16:0/68357293 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2bcc0ce9f0 0x7f2bcc0b53c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:43.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.543+0000 7f2bd09a8700 1 --2- 172.21.15.16:0/68357293 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2bcc0d2f70 0x7f2bcc0c9700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:43.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.543+0000 7f2bd09a8700 1 -- 172.21.15.16:0/68357293 >> 172.21.15.16:0/68357293 conn(0x7f2bcc072330 msgr2=0x7f2bcc158980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:43.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.543+0000 7f2bd09a8700 1 -- 172.21.15.16:0/68357293 shutdown_connections 2023-10-16T13:48:43.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.543+0000 7f2bd09a8700 1 -- 172.21.15.16:0/68357293 wait complete. 2023-10-16T13:48:43.559 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 35, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T13:48:23.348668+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '264', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 36, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T13:48:23.799968+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '265', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T13:48:43.559 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T13:48:43.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.725+0000 7fd5ba7a6700 1 Processor -- start 2023-10-16T13:48:43.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.725+0000 7fd5ba7a6700 1 -- start start 2023-10-16T13:48:43.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.726+0000 7fd5ba7a6700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fd5b415da10 0x7fd5b40d1520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:43.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.726+0000 7fd5ba7a6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd5b40d4060 0x7fd5b40cf2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:43.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.726+0000 7fd5ba7a6700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd5b40d1d20 0x7fd5b40d21b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:43.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.726+0000 7fd5ba7a6700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5b40d26f0 con 0x7fd5b40d4060 2023-10-16T13:48:43.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.726+0000 7fd5ba7a6700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5b40d4540 con 0x7fd5b415da10 2023-10-16T13:48:43.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.726+0000 7fd5ba7a6700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b40ce7e0 con 0x7fd5b40d1d20 2023-10-16T13:48:43.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.726+0000 7fd5b37fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd5b40d4060 0x7fd5b40cf2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:43.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.726+0000 7fd5b37fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd5b40d4060 0x7fd5b40cf2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58238/0 (socket says 172.21.15.16:58238) 2023-10-16T13:48:43.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.726+0000 7fd5b37fe700 1 -- 172.21.15.16:0/767952183 learned_addr learned my addr 172.21.15.16:0/767952183 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:43.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.726+0000 7fd5b3fff700 1 --2- 172.21.15.16:0/767952183 >> v2:172.21.15.49:3300/0 conn(0x7fd5b415da10 0x7fd5b40d1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:43.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.726+0000 7fd5b8d43700 1 --2- 172.21.15.16:0/767952183 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd5b40d1d20 0x7fd5b40d21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:43.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.726+0000 7fd5b37fe700 1 -- 172.21.15.16:0/767952183 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd5b40d1d20 msgr2=0x7fd5b40d21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:43.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.726+0000 7fd5b37fe700 1 --2- 172.21.15.16:0/767952183 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd5b40d1d20 0x7fd5b40d21b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:43.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.726+0000 7fd5b37fe700 1 -- 172.21.15.16:0/767952183 >> v2:172.21.15.49:3300/0 conn(0x7fd5b415da10 msgr2=0x7fd5b40d1520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:43.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.726+0000 7fd5b37fe700 1 --2- 172.21.15.16:0/767952183 >> v2:172.21.15.49:3300/0 conn(0x7fd5b415da10 0x7fd5b40d1520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:43.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.726+0000 7fd5b37fe700 1 -- 172.21.15.16:0/767952183 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5b40ceac0 con 0x7fd5b40d4060 2023-10-16T13:48:43.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.726+0000 7fd5b37fe700 1 --2- 172.21.15.16:0/767952183 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5b40d4060 0x7fd5b40cf2a0 secure :-1 s=READY pgs=3076 cs=0 l=1 rev1=1 crypto rx=0x7fd5b408b540 tx=0x7fd5a8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a28abf7c7c1ac368 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:43.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.727+0000 7fd5b2ffd700 1 -- 172.21.15.16:0/767952183 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a8011830 con 0x7fd5b40d4060 2023-10-16T13:48:43.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.727+0000 7fd5b2ffd700 1 -- 172.21.15.16:0/767952183 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5a8011990 con 0x7fd5b40d4060 2023-10-16T13:48:43.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.727+0000 7fd5b2ffd700 1 -- 172.21.15.16:0/767952183 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a8011d20 con 0x7fd5b40d4060 2023-10-16T13:48:43.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.727+0000 7fd5ba7a6700 1 -- 172.21.15.16:0/767952183 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5b40d4060 msgr2=0x7fd5b40cf2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:43.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.727+0000 7fd5ba7a6700 1 --2- 172.21.15.16:0/767952183 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5b40d4060 0x7fd5b40cf2a0 secure :-1 s=READY pgs=3076 cs=0 l=1 rev1=1 crypto rx=0x7fd5b408b540 tx=0x7fd5a8009bd0 comp rx=0 tx=0).stop 2023-10-16T13:48:43.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.727+0000 7fd5ba7a6700 1 -- 172.21.15.16:0/767952183 shutdown_connections 2023-10-16T13:48:43.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.727+0000 7fd5ba7a6700 1 --2- 172.21.15.16:0/767952183 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd5b40d1d20 0x7fd5b40d21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:43.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.727+0000 7fd5ba7a6700 1 --2- 172.21.15.16:0/767952183 >> v2:172.21.15.49:3300/0 conn(0x7fd5b415da10 0x7fd5b40d1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:43.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.727+0000 7fd5ba7a6700 1 --2- 172.21.15.16:0/767952183 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5b40d4060 0x7fd5b40cf2a0 unknown :-1 s=CLOSED pgs=3076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:43.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.727+0000 7fd5ba7a6700 1 -- 172.21.15.16:0/767952183 >> 172.21.15.16:0/767952183 conn(0x7fd5b409c9f0 msgr2=0x7fd5b40d3740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:43.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.727+0000 7fd5ba7a6700 1 -- 172.21.15.16:0/767952183 shutdown_connections 2023-10-16T13:48:43.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.727+0000 7fd5ba7a6700 1 -- 172.21.15.16:0/767952183 wait complete. 2023-10-16T13:48:43.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.728+0000 7fd5ba7a6700 1 Processor -- start 2023-10-16T13:48:43.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.729+0000 7fd5ba7a6700 1 -- start start 2023-10-16T13:48:43.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.729+0000 7fd5ba7a6700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd5b40d1d20 0x7fd5b415a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:43.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.729+0000 7fd5ba7a6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5b40d4060 0x7fd5b41f6d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:43.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.729+0000 7fd5ba7a6700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd5b415da10 0x7fd5b40cc820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:43.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.729+0000 7fd5ba7a6700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b41f6600 con 0x7fd5b40d4060 2023-10-16T13:48:43.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.729+0000 7fd5ba7a6700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b41f68e0 con 0x7fd5b415da10 2023-10-16T13:48:43.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.729+0000 7fd5ba7a6700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b41f3880 con 0x7fd5b40d1d20 2023-10-16T13:48:43.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.729+0000 7fd5b37fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5b40d4060 0x7fd5b41f6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:43.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.729+0000 7fd5b37fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5b40d4060 0x7fd5b41f6d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58244/0 (socket says 172.21.15.16:58244) 2023-10-16T13:48:43.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.729+0000 7fd5b37fe700 1 -- 172.21.15.16:0/332609733 learned_addr learned my addr 172.21.15.16:0/332609733 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:43.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.730+0000 7fd5b3fff700 1 --2- 172.21.15.16:0/332609733 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd5b40d1d20 0x7fd5b415a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:43.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.730+0000 7fd5b8d43700 1 --2- 172.21.15.16:0/332609733 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd5b415da10 0x7fd5b40cc820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:43.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.730+0000 7fd5b37fe700 1 -- 172.21.15.16:0/332609733 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd5b40d1d20 msgr2=0x7fd5b415a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:43.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.730+0000 7fd5b37fe700 1 --2- 172.21.15.16:0/332609733 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd5b40d1d20 0x7fd5b415a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:43.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.730+0000 7fd5b37fe700 1 -- 172.21.15.16:0/332609733 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd5b415da10 msgr2=0x7fd5b40cc820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:43.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.730+0000 7fd5b37fe700 1 --2- 172.21.15.16:0/332609733 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd5b415da10 0x7fd5b40cc820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:43.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.730+0000 7fd5b37fe700 1 -- 172.21.15.16:0/332609733 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5a8009550 con 0x7fd5b40d4060 2023-10-16T13:48:43.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.730+0000 7fd5b37fe700 1 --2- 172.21.15.16:0/332609733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5b40d4060 0x7fd5b41f6d70 secure :-1 s=READY pgs=3077 cs=0 l=1 rev1=1 crypto rx=0x7fd5a8009c10 tx=0x7fd5a8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:43.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.730+0000 7fd5b0ff9700 1 -- 172.21.15.16:0/332609733 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a8004240 con 0x7fd5b40d4060 2023-10-16T13:48:43.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.730+0000 7fd5b0ff9700 1 -- 172.21.15.16:0/332609733 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5a801f070 con 0x7fd5b40d4060 2023-10-16T13:48:43.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.730+0000 7fd5ba7a6700 1 -- 172.21.15.16:0/332609733 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5b41f3b60 con 0x7fd5b40d4060 2023-10-16T13:48:43.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.730+0000 7fd5b0ff9700 1 -- 172.21.15.16:0/332609733 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a80036a0 con 0x7fd5b40d4060 2023-10-16T13:48:43.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.730+0000 7fd5ba7a6700 1 -- 172.21.15.16:0/332609733 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5b415a8a0 con 0x7fd5b40d4060 2023-10-16T13:48:43.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.731+0000 7fd5ba7a6700 1 -- 172.21.15.16:0/332609733 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5b404ebb0 con 0x7fd5b40d4060 2023-10-16T13:48:43.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.732+0000 7fd5b0ff9700 1 -- 172.21.15.16:0/332609733 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 162) v1 ==== 89332+0+0 (secure 0 0 0) 0x7fd5a8021030 con 0x7fd5b40d4060 2023-10-16T13:48:43.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.732+0000 7fd5b0ff9700 1 --2- 172.21.15.16:0/332609733 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7fd5a006b020 0x7fd5a006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:43.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.732+0000 7fd5b3fff700 1 -- 172.21.15.16:0/332609733 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7fd5a006b020 msgr2=0x7fd5a006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2131389174 2023-10-16T13:48:43.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.732+0000 7fd5b0ff9700 1 -- 172.21.15.16:0/332609733 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(266..266 src has 1..266) v4 ==== 6587+0+0 (secure 0 0 0) 0x7fd5a808ac30 con 0x7fd5b40d4060 2023-10-16T13:48:43.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.732+0000 7fd5b3fff700 1 --2- 172.21.15.16:0/332609733 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7fd5a006b020 0x7fd5a006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:48:43.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.737+0000 7fd5b0ff9700 1 -- 172.21.15.16:0/332609733 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd5a80558f0 con 0x7fd5b40d4060 2023-10-16T13:48:43.863 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:48:43.861+0000 7f2fb0253200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T13:48:43.871 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:43.869+0000 7fd1d1dca200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T13:48:43.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.933+0000 7fd5b3fff700 1 -- 172.21.15.16:0/332609733 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7fd5a006b020 msgr2=0x7fd5a006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2131389174 2023-10-16T13:48:43.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.933+0000 7fd5b3fff700 1 --2- 172.21.15.16:0/332609733 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7fd5a006b020 0x7fd5a006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:48:43.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.942+0000 7fd5ba7a6700 1 -- 172.21.15.16:0/332609733 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd5b40b2d00 con 0x7fd5b40d4060 2023-10-16T13:48:43.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.942+0000 7fd5b0ff9700 1 -- 172.21.15.16:0/332609733 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v266) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd5a80150a0 con 0x7fd5b40d4060 2023-10-16T13:48:43.945 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T13:48:43.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.944+0000 7fd5ba7a6700 1 -- 172.21.15.16:0/332609733 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7fd5a006b020 msgr2=0x7fd5a006d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:48:43.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.944+0000 7fd5ba7a6700 1 --2- 172.21.15.16:0/332609733 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7fd5a006b020 0x7fd5a006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:43.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.945+0000 7fd5ba7a6700 1 -- 172.21.15.16:0/332609733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5b40d4060 msgr2=0x7fd5b41f6d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:43.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.945+0000 7fd5ba7a6700 1 --2- 172.21.15.16:0/332609733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5b40d4060 0x7fd5b41f6d70 secure :-1 s=READY pgs=3077 cs=0 l=1 rev1=1 crypto rx=0x7fd5a8009c10 tx=0x7fd5a8004750 comp rx=0 tx=0).stop 2023-10-16T13:48:43.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.945+0000 7fd5ba7a6700 1 -- 172.21.15.16:0/332609733 shutdown_connections 2023-10-16T13:48:43.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.945+0000 7fd5ba7a6700 1 --2- 172.21.15.16:0/332609733 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7fd5a006b020 0x7fd5a006d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:43.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.945+0000 7fd5ba7a6700 1 --2- 172.21.15.16:0/332609733 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd5b415da10 0x7fd5b40cc820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:43.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.945+0000 7fd5ba7a6700 1 --2- 172.21.15.16:0/332609733 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd5b40d1d20 0x7fd5b415a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:43.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.945+0000 7fd5ba7a6700 1 --2- 172.21.15.16:0/332609733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5b40d4060 0x7fd5b41f6d70 unknown :-1 s=CLOSED pgs=3077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:43.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.945+0000 7fd5ba7a6700 1 -- 172.21.15.16:0/332609733 >> 172.21.15.16:0/332609733 conn(0x7fd5b409c9f0 msgr2=0x7fd5b406b810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:43.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.945+0000 7fd5ba7a6700 1 -- 172.21.15.16:0/332609733 shutdown_connections 2023-10-16T13:48:43.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:43.945+0000 7fd5ba7a6700 1 -- 172.21.15.16:0/332609733 wait complete. 2023-10-16T13:48:43.961 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T13:48:43.989 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:48:43.986+0000 7f2fb0253200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T13:48:43.994 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:43.992+0000 7fd1d1dca200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T13:48:44.065 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:48:44.063+0000 7f2fb0253200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T13:48:44.072 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:44.070+0000 7fd1d1dca200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T13:48:44.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.089+0000 7f8860ef3700 1 Processor -- start 2023-10-16T13:48:44.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.089+0000 7f8860ef3700 1 -- start start 2023-10-16T13:48:44.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.090+0000 7f8860ef3700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f885c15f060 0x7f885c0cc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:44.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.090+0000 7f8860ef3700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f885c0b2440 con 0x7f885c0ce5d0 2023-10-16T13:48:44.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.090+0000 7f8860ef3700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f885c0d00e0 con 0x7f885c15ec80 2023-10-16T13:48:44.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.090+0000 7f8860ef3700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f885c0b2c50 con 0x7f885c15f060 2023-10-16T13:48:44.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.090+0000 7f8859d9b700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f885c0ce5d0 0x7f885c0cc190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:47526/0 (socket says 172.21.15.16:47526) 2023-10-16T13:48:44.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.090+0000 7f8859d9b700 1 -- 172.21.15.16:0/3705157914 learned_addr learned my addr 172.21.15.16:0/3705157914 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:48:44.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.090+0000 7f885ad9d700 1 --2- 172.21.15.16:0/3705157914 >> v2:172.21.15.49:3300/0 conn(0x7f885c15f060 0x7f885c0cc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:44.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.090+0000 7f885959a700 1 -- 172.21.15.16:0/3705157914 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1667608714 0 0) 0x7f885c0b2440 con 0x7f885c0ce5d0 2023-10-16T13:48:44.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.090+0000 7f885959a700 1 -- 172.21.15.16:0/3705157914 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88440035b0 con 0x7f885c0ce5d0 2023-10-16T13:48:44.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.090+0000 7f885959a700 1 -- 172.21.15.16:0/3705157914 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1289078040 0 0) 0x7f885c0d00e0 con 0x7f885c15ec80 2023-10-16T13:48:44.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.090+0000 7f885959a700 1 -- 172.21.15.16:0/3705157914 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f885c0b2440 con 0x7f885c15ec80 2023-10-16T13:48:44.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.090+0000 7f885ad9d700 1 -- 172.21.15.16:0/3705157914 >> v1:172.21.15.49:6789/0 conn(0x7f885c15ec80 legacy=0x7f885c0cdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:44.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.090+0000 7f885959a700 1 -- 172.21.15.16:0/3705157914 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 973376448 0 0) 0x7f88440035b0 con 0x7f885c0ce5d0 2023-10-16T13:48:44.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.090+0000 7f885ad9d700 1 -- 172.21.15.16:0/3705157914 >> v1:172.21.15.16:6789/0 conn(0x7f885c0ce5d0 legacy=0x7f885c0cc190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:44.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.090+0000 7f885ad9d700 1 -- 172.21.15.16:0/3705157914 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f885c0cef60 con 0x7f885c15f060 2023-10-16T13:48:44.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.091+0000 7f885ad9d700 1 --2- 172.21.15.16:0/3705157914 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f885c15f060 0x7f885c0cc8a0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f8854009dc0 tx=0x7f885400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=702e129480d517bd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:44.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.091+0000 7f885959a700 1 -- 172.21.15.16:0/3705157914 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f885400e070 con 0x7f885c15f060 2023-10-16T13:48:44.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.091+0000 7f885959a700 1 -- 172.21.15.16:0/3705157914 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8854003a20 con 0x7f885c15f060 2023-10-16T13:48:44.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.091+0000 7f885959a700 1 -- 172.21.15.16:0/3705157914 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8850001670 con 0x7f885c15f060 2023-10-16T13:48:44.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.091+0000 7f8860ef3700 1 -- 172.21.15.16:0/3705157914 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f885c15f060 msgr2=0x7f885c0cc8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:44.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.091+0000 7f8860ef3700 1 --2- 172.21.15.16:0/3705157914 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f885c15f060 0x7f885c0cc8a0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f8854009dc0 tx=0x7f885400b330 comp rx=0 tx=0).stop 2023-10-16T13:48:44.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.092+0000 7f8860ef3700 1 -- 172.21.15.16:0/3705157914 shutdown_connections 2023-10-16T13:48:44.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.092+0000 7f8860ef3700 1 --2- 172.21.15.16:0/3705157914 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f885c15f060 0x7f885c0cc8a0 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:44.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.092+0000 7f8860ef3700 1 -- 172.21.15.16:0/3705157914 >> 172.21.15.16:0/3705157914 conn(0x7f885c0d15f0 msgr2=0x7f885c0d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:44.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.092+0000 7f8860ef3700 1 -- 172.21.15.16:0/3705157914 shutdown_connections 2023-10-16T13:48:44.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.092+0000 7f8860ef3700 1 -- 172.21.15.16:0/3705157914 wait complete. 2023-10-16T13:48:44.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.093+0000 7f8860ef3700 1 Processor -- start 2023-10-16T13:48:44.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.093+0000 7f8860ef3700 1 -- start start 2023-10-16T13:48:44.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.094+0000 7f8860ef3700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f885c0ce5d0 0x7f885c151c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:44.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.094+0000 7f8860ef3700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f885c15ec80 0x7f885c15ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:44.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.094+0000 7f8860ef3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f885c15f060 0x7f885c0c7740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:44.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.094+0000 7f8860ef3700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f885c0c72b0 con 0x7f885c15f060 2023-10-16T13:48:44.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.094+0000 7f8860ef3700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f885c152160 con 0x7f885c0ce5d0 2023-10-16T13:48:44.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.094+0000 7f8860ef3700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f885c0c66a0 con 0x7f885c15ec80 2023-10-16T13:48:44.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.094+0000 7f885ad9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f885c15f060 0x7f885c0c7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:44.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.094+0000 7f885a59c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f885c0ce5d0 0x7f885c151c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:44.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.094+0000 7f885ad9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f885c15f060 0x7f885c0c7740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58260/0 (socket says 172.21.15.16:58260) 2023-10-16T13:48:44.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.094+0000 7f885ad9d700 1 -- 172.21.15.16:0/2795512736 learned_addr learned my addr 172.21.15.16:0/2795512736 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:44.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.094+0000 7f8859d9b700 1 --2- 172.21.15.16:0/2795512736 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f885c15ec80 0x7f885c15ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:44.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.094+0000 7f885ad9d700 1 -- 172.21.15.16:0/2795512736 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f885c15ec80 msgr2=0x7f885c15ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:44.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.094+0000 7f885ad9d700 1 --2- 172.21.15.16:0/2795512736 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f885c15ec80 0x7f885c15ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:44.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.094+0000 7f885ad9d700 1 -- 172.21.15.16:0/2795512736 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f885c0ce5d0 msgr2=0x7f885c151c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:44.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.094+0000 7f885ad9d700 1 --2- 172.21.15.16:0/2795512736 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f885c0ce5d0 0x7f885c151c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:44.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.094+0000 7f885ad9d700 1 -- 172.21.15.16:0/2795512736 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8854009a70 con 0x7f885c15f060 2023-10-16T13:48:44.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.095+0000 7f885ad9d700 1 --2- 172.21.15.16:0/2795512736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f885c15f060 0x7f885c0c7740 secure :-1 s=READY pgs=3079 cs=0 l=1 rev1=1 crypto rx=0x7f885400b370 tx=0x7f8854004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:44.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.095+0000 7f884affd700 1 -- 172.21.15.16:0/2795512736 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88540044c0 con 0x7f885c15f060 2023-10-16T13:48:44.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.095+0000 7f884affd700 1 -- 172.21.15.16:0/2795512736 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8854004660 con 0x7f885c15f060 2023-10-16T13:48:44.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.095+0000 7f884affd700 1 -- 172.21.15.16:0/2795512736 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8854012450 con 0x7f885c15f060 2023-10-16T13:48:44.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.095+0000 7f8860ef3700 1 -- 172.21.15.16:0/2795512736 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f885c0c6980 con 0x7f885c15f060 2023-10-16T13:48:44.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.095+0000 7f8860ef3700 1 -- 172.21.15.16:0/2795512736 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f885c0c6f40 con 0x7f885c15f060 2023-10-16T13:48:44.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.096+0000 7f8860ef3700 1 -- 172.21.15.16:0/2795512736 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f885c0c7100 con 0x7f885c15f060 2023-10-16T13:48:44.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.096+0000 7f884affd700 1 -- 172.21.15.16:0/2795512736 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 162) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f88540125b0 con 0x7f885c15f060 2023-10-16T13:48:44.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.097+0000 7f884affd700 1 --2- 172.21.15.16:0/2795512736 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f883c06b020 0x7f883c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:44.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.097+0000 7f885a59c700 1 -- 172.21.15.16:0/2795512736 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f883c06b020 msgr2=0x7f883c06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2131389174 2023-10-16T13:48:44.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.097+0000 7f884affd700 1 -- 172.21.15.16:0/2795512736 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(266..266 src has 1..266) v4 ==== 6587+0+0 (secure 0 0 0) 0x7f885408b4b0 con 0x7f885c15f060 2023-10-16T13:48:44.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.097+0000 7f885a59c700 1 --2- 172.21.15.16:0/2795512736 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f883c06b020 0x7f883c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:48:44.106 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.101+0000 7f884affd700 1 -- 172.21.15.16:0/2795512736 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8854055f40 con 0x7f885c15f060 2023-10-16T13:48:44.150 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:48:44.148+0000 7f2fb0253200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T13:48:44.160 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:44.157+0000 7fd1d1dca200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T13:48:44.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.298+0000 7f885a59c700 1 -- 172.21.15.16:0/2795512736 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f883c06b020 msgr2=0x7f883c06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2131389174 2023-10-16T13:48:44.300 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.298+0000 7f885a59c700 1 --2- 172.21.15.16:0/2795512736 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f883c06b020 0x7f883c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:48:44.306 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.304+0000 7f8860ef3700 1 -- 172.21.15.16:0/2795512736 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f885c04ebb0 con 0x7f885c15f060 2023-10-16T13:48:44.307 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.304+0000 7f884affd700 1 -- 172.21.15.16:0/2795512736 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v266) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8854059560 con 0x7f885c15f060 2023-10-16T13:48:44.307 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-10-16T13:48:44.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.306+0000 7f8860ef3700 1 -- 172.21.15.16:0/2795512736 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f883c06b020 msgr2=0x7f883c06d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:48:44.309 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.306+0000 7f8860ef3700 1 --2- 172.21.15.16:0/2795512736 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f883c06b020 0x7f883c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:44.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.307+0000 7f8860ef3700 1 -- 172.21.15.16:0/2795512736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f885c15f060 msgr2=0x7f885c0c7740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:44.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.307+0000 7f8860ef3700 1 --2- 172.21.15.16:0/2795512736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f885c15f060 0x7f885c0c7740 secure :-1 s=READY pgs=3079 cs=0 l=1 rev1=1 crypto rx=0x7f885400b370 tx=0x7f8854004ad0 comp rx=0 tx=0).stop 2023-10-16T13:48:44.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.307+0000 7f8860ef3700 1 -- 172.21.15.16:0/2795512736 shutdown_connections 2023-10-16T13:48:44.310 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.307+0000 7f8860ef3700 1 --2- 172.21.15.16:0/2795512736 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f883c06b020 0x7f883c06d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:44.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.307+0000 7f8860ef3700 1 --2- 172.21.15.16:0/2795512736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f885c15f060 0x7f885c0c7740 unknown :-1 s=CLOSED pgs=3079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:44.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.307+0000 7f8860ef3700 1 --2- 172.21.15.16:0/2795512736 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f885c0ce5d0 0x7f885c151c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:44.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.307+0000 7f8860ef3700 1 --2- 172.21.15.16:0/2795512736 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f885c15ec80 0x7f885c15ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:44.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.307+0000 7f8860ef3700 1 -- 172.21.15.16:0/2795512736 >> 172.21.15.16:0/2795512736 conn(0x7f885c0d15f0 msgr2=0x7f885c1539a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:44.311 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.307+0000 7f8860ef3700 1 -- 172.21.15.16:0/2795512736 shutdown_connections 2023-10-16T13:48:44.312 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.307+0000 7f8860ef3700 1 -- 172.21.15.16:0/2795512736 wait complete. 2023-10-16T13:48:44.322 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T13:48:44.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.490+0000 7fcafc531700 1 Processor -- start 2023-10-16T13:48:44.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.490+0000 7fcafc531700 1 -- start start 2023-10-16T13:48:44.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.491+0000 7fcafc531700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fcaf406e170 0x7fcaf40cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:44.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.491+0000 7fcafc531700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaf40cef70 con 0x7fcaf40cb880 2023-10-16T13:48:44.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.491+0000 7fcafc531700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaf40ce360 con 0x7fcaf415b070 2023-10-16T13:48:44.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.491+0000 7fcafc531700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcaf40ce640 con 0x7fcaf406e170 2023-10-16T13:48:44.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.491+0000 7fcaf9acc700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fcaf40cb880 0x7fcaf406da60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:47574/0 (socket says 172.21.15.16:47574) 2023-10-16T13:48:44.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.491+0000 7fcaf9acc700 1 -- 172.21.15.16:0/4063356975 learned_addr learned my addr 172.21.15.16:0/4063356975 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:48:44.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.491+0000 7fcafaace700 1 --2- 172.21.15.16:0/4063356975 >> v2:172.21.15.49:3300/0 conn(0x7fcaf406e170 0x7fcaf40cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:44.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.491+0000 7fcaf92cb700 1 -- 172.21.15.16:0/4063356975 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3804110539 0 0) 0x7fcaf40cef70 con 0x7fcaf40cb880 2023-10-16T13:48:44.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.491+0000 7fcaf92cb700 1 -- 172.21.15.16:0/4063356975 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcae00035b0 con 0x7fcaf40cb880 2023-10-16T13:48:44.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.491+0000 7fcaf92cb700 1 -- 172.21.15.16:0/4063356975 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2945457587 0 0) 0x7fcaf40ce360 con 0x7fcaf415b070 2023-10-16T13:48:44.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.491+0000 7fcaf92cb700 1 -- 172.21.15.16:0/4063356975 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcaf40cef70 con 0x7fcaf415b070 2023-10-16T13:48:44.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.491+0000 7fcafaace700 1 -- 172.21.15.16:0/4063356975 >> v1:172.21.15.49:6789/0 conn(0x7fcaf415b070 legacy=0x7fcaf415b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:44.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.491+0000 7fcafaace700 1 -- 172.21.15.16:0/4063356975 >> v1:172.21.15.16:6789/0 conn(0x7fcaf40cb880 legacy=0x7fcaf406da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:44.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.491+0000 7fcafaace700 1 -- 172.21.15.16:0/4063356975 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaf40ce920 con 0x7fcaf406e170 2023-10-16T13:48:44.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.492+0000 7fcafaace700 1 --2- 172.21.15.16:0/4063356975 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcaf406e170 0x7fcaf40cbd00 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7fcaec009d90 tx=0x7fcaec00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=91ff9998ff7e1287 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:44.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.492+0000 7fcaf92cb700 1 -- 172.21.15.16:0/4063356975 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaec00e070 con 0x7fcaf406e170 2023-10-16T13:48:44.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.492+0000 7fcaf92cb700 1 -- 172.21.15.16:0/4063356975 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcaec00bb20 con 0x7fcaf406e170 2023-10-16T13:48:44.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.492+0000 7fcaf92cb700 1 -- 172.21.15.16:0/4063356975 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaec00be90 con 0x7fcaf406e170 2023-10-16T13:48:44.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.492+0000 7fcafc531700 1 -- 172.21.15.16:0/4063356975 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcaf406e170 msgr2=0x7fcaf40cbd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:44.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.492+0000 7fcafc531700 1 --2- 172.21.15.16:0/4063356975 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcaf406e170 0x7fcaf40cbd00 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7fcaec009d90 tx=0x7fcaec00b330 comp rx=0 tx=0).stop 2023-10-16T13:48:44.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.493+0000 7fcafc531700 1 -- 172.21.15.16:0/4063356975 shutdown_connections 2023-10-16T13:48:44.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.493+0000 7fcafc531700 1 --2- 172.21.15.16:0/4063356975 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcaf406e170 0x7fcaf40cbd00 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:44.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.493+0000 7fcafc531700 1 -- 172.21.15.16:0/4063356975 >> 172.21.15.16:0/4063356975 conn(0x7fcaf40b1f40 msgr2=0x7fcaf40b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:44.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.493+0000 7fcafc531700 1 -- 172.21.15.16:0/4063356975 shutdown_connections 2023-10-16T13:48:44.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.493+0000 7fcafc531700 1 -- 172.21.15.16:0/4063356975 wait complete. 2023-10-16T13:48:44.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.494+0000 7fcafc531700 1 Processor -- start 2023-10-16T13:48:44.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.494+0000 7fcafc531700 1 -- start start 2023-10-16T13:48:44.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.495+0000 7fcafc531700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcaf406e170 0x7fcaf41f6a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:44.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.495+0000 7fcafc531700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcaf40cb880 0x7fcaf40ccd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:44.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.495+0000 7fcafc531700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcaf415b070 0x7fcaf4147a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:44.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.495+0000 7fcafc531700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf40c92e0 con 0x7fcaf40cb880 2023-10-16T13:48:44.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.495+0000 7fcafc531700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf4147080 con 0x7fcaf406e170 2023-10-16T13:48:44.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.495+0000 7fcafc531700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf40cd290 con 0x7fcaf415b070 2023-10-16T13:48:44.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.495+0000 7fcaf9acc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcaf40cb880 0x7fcaf40ccd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:44.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.495+0000 7fcaf9acc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcaf40cb880 0x7fcaf40ccd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58308/0 (socket says 172.21.15.16:58308) 2023-10-16T13:48:44.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.495+0000 7fcaf9acc700 1 -- 172.21.15.16:0/2670327195 learned_addr learned my addr 172.21.15.16:0/2670327195 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:44.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.495+0000 7fcafa2cd700 1 --2- 172.21.15.16:0/2670327195 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcaf406e170 0x7fcaf41f6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:44.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.495+0000 7fcafaace700 1 --2- 172.21.15.16:0/2670327195 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcaf415b070 0x7fcaf4147a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:44.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.495+0000 7fcaf9acc700 1 -- 172.21.15.16:0/2670327195 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcaf415b070 msgr2=0x7fcaf4147a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:44.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.495+0000 7fcaf9acc700 1 --2- 172.21.15.16:0/2670327195 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcaf415b070 0x7fcaf4147a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:44.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.495+0000 7fcaf9acc700 1 -- 172.21.15.16:0/2670327195 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcaf406e170 msgr2=0x7fcaf41f6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:44.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.495+0000 7fcaf9acc700 1 --2- 172.21.15.16:0/2670327195 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcaf406e170 0x7fcaf41f6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:44.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.495+0000 7fcaf9acc700 1 -- 172.21.15.16:0/2670327195 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaec009a70 con 0x7fcaf40cb880 2023-10-16T13:48:44.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.495+0000 7fcaf9acc700 1 --2- 172.21.15.16:0/2670327195 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcaf40cb880 0x7fcaf40ccd50 secure :-1 s=READY pgs=3081 cs=0 l=1 rev1=1 crypto rx=0x7fcae400c700 tx=0x7fcae400ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:44.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.495+0000 7fcaeaffd700 1 -- 172.21.15.16:0/2670327195 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcae4005d70 con 0x7fcaf40cb880 2023-10-16T13:48:44.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.495+0000 7fcaeaffd700 1 -- 172.21.15.16:0/2670327195 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcae40070d0 con 0x7fcaf40cb880 2023-10-16T13:48:44.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.495+0000 7fcafc531700 1 -- 172.21.15.16:0/2670327195 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcaf4147fd0 con 0x7fcaf40cb880 2023-10-16T13:48:44.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.495+0000 7fcaeaffd700 1 -- 172.21.15.16:0/2670327195 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcae4007460 con 0x7fcaf40cb880 2023-10-16T13:48:44.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.496+0000 7fcafc531700 1 -- 172.21.15.16:0/2670327195 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcaf4147290 con 0x7fcaf40cb880 2023-10-16T13:48:44.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.496+0000 7fcafc531700 1 -- 172.21.15.16:0/2670327195 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcaf404ebb0 con 0x7fcaf40cb880 2023-10-16T13:48:44.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.497+0000 7fcaeaffd700 1 -- 172.21.15.16:0/2670327195 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 162) v1 ==== 89332+0+0 (secure 0 0 0) 0x7fcae40075c0 con 0x7fcaf40cb880 2023-10-16T13:48:44.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.497+0000 7fcaeaffd700 1 --2- 172.21.15.16:0/2670327195 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7fcad806b050 0x7fcad806d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:44.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.497+0000 7fcafa2cd700 1 -- 172.21.15.16:0/2670327195 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7fcad806b050 msgr2=0x7fcad806d510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2131389174 2023-10-16T13:48:44.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.497+0000 7fcaeaffd700 1 -- 172.21.15.16:0/2670327195 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(266..266 src has 1..266) v4 ==== 6587+0+0 (secure 0 0 0) 0x7fcae4088e80 con 0x7fcaf40cb880 2023-10-16T13:48:44.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.497+0000 7fcafa2cd700 1 --2- 172.21.15.16:0/2670327195 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7fcad806b050 0x7fcad806d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:48:44.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.502+0000 7fcaeaffd700 1 -- 172.21.15.16:0/2670327195 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcae4053aa0 con 0x7fcaf40cb880 2023-10-16T13:48:44.540 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:48:44.537+0000 7f2fb0253200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T13:48:44.556 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:44.554+0000 7fd1d1dca200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T13:48:44.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.698+0000 7fcafa2cd700 1 -- 172.21.15.16:0/2670327195 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7fcad806b050 msgr2=0x7fcad806d510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2131389174 2023-10-16T13:48:44.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.698+0000 7fcafa2cd700 1 --2- 172.21.15.16:0/2670327195 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7fcad806b050 0x7fcad806d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:48:44.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.704+0000 7fcafc531700 1 -- 172.21.15.16:0/2670327195 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fcaf4154e40 con 0x7fcaf40cb880 2023-10-16T13:48:44.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.705+0000 7fcaeaffd700 1 -- 172.21.15.16:0/2670327195 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v266) v1 ==== 100+0+11 (secure 0 0 0) 0x7fcae4017020 con 0x7fcaf40cb880 2023-10-16T13:48:44.708 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-10-16T13:48:44.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.707+0000 7fcafc531700 1 -- 172.21.15.16:0/2670327195 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7fcad806b050 msgr2=0x7fcad806d510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:48:44.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.707+0000 7fcafc531700 1 --2- 172.21.15.16:0/2670327195 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7fcad806b050 0x7fcad806d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:44.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.707+0000 7fcafc531700 1 -- 172.21.15.16:0/2670327195 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcaf40cb880 msgr2=0x7fcaf40ccd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:44.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.707+0000 7fcafc531700 1 --2- 172.21.15.16:0/2670327195 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcaf40cb880 0x7fcaf40ccd50 secure :-1 s=READY pgs=3081 cs=0 l=1 rev1=1 crypto rx=0x7fcae400c700 tx=0x7fcae400ca10 comp rx=0 tx=0).stop 2023-10-16T13:48:44.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.707+0000 7fcafc531700 1 -- 172.21.15.16:0/2670327195 shutdown_connections 2023-10-16T13:48:44.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.707+0000 7fcafc531700 1 --2- 172.21.15.16:0/2670327195 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7fcad806b050 0x7fcad806d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:44.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.707+0000 7fcafc531700 1 --2- 172.21.15.16:0/2670327195 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcaf415b070 0x7fcaf4147a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:44.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.707+0000 7fcafc531700 1 --2- 172.21.15.16:0/2670327195 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcaf406e170 0x7fcaf41f6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:44.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.707+0000 7fcafc531700 1 --2- 172.21.15.16:0/2670327195 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcaf40cb880 0x7fcaf40ccd50 unknown :-1 s=CLOSED pgs=3081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:44.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.707+0000 7fcafc531700 1 -- 172.21.15.16:0/2670327195 >> 172.21.15.16:0/2670327195 conn(0x7fcaf40b1f40 msgr2=0x7fcaf406a7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:44.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.707+0000 7fcafc531700 1 -- 172.21.15.16:0/2670327195 shutdown_connections 2023-10-16T13:48:44.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.707+0000 7fcafc531700 1 -- 172.21.15.16:0/2670327195 wait complete. 2023-10-16T13:48:44.722 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T13:48:44.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.846+0000 7f353f7af700 1 Processor -- start 2023-10-16T13:48:44.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.847+0000 7f353f7af700 1 -- start start 2023-10-16T13:48:44.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.847+0000 7f353f7af700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f353815b630 0x7f35380d0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:44.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.847+0000 7f353f7af700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35380b0970 con 0x7f35380ce4f0 2023-10-16T13:48:44.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.847+0000 7f353f7af700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35380cd690 con 0x7f353815ba10 2023-10-16T13:48:44.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.847+0000 7f353f7af700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35380cd970 con 0x7f353815b630 2023-10-16T13:48:44.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.848+0000 7f353dd4c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f35380ce4f0 0x7f35380b1430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:47592/0 (socket says 172.21.15.16:47592) 2023-10-16T13:48:44.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.848+0000 7f353dd4c700 1 -- 172.21.15.16:0/2973024819 learned_addr learned my addr 172.21.15.16:0/2973024819 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:48:44.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.848+0000 7f353d54b700 1 --2- 172.21.15.16:0/2973024819 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f353815b630 0x7f35380d0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:44.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.848+0000 7f352ffff700 1 -- 172.21.15.16:0/2973024819 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 191094913 0 0) 0x7f35380b0970 con 0x7f35380ce4f0 2023-10-16T13:48:44.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.848+0000 7f352ffff700 1 -- 172.21.15.16:0/2973024819 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35240035b0 con 0x7f35380ce4f0 2023-10-16T13:48:44.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.848+0000 7f352ffff700 1 -- 172.21.15.16:0/2973024819 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2042598278 0 0) 0x7f35380cd690 con 0x7f353815ba10 2023-10-16T13:48:44.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.848+0000 7f352ffff700 1 -- 172.21.15.16:0/2973024819 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35380b0970 con 0x7f353815ba10 2023-10-16T13:48:44.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.848+0000 7f353d54b700 1 -- 172.21.15.16:0/2973024819 >> v1:172.21.15.49:6789/0 conn(0x7f353815ba10 legacy=0x7f35380cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:44.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.848+0000 7f353d54b700 1 -- 172.21.15.16:0/2973024819 >> v1:172.21.15.16:6789/0 conn(0x7f35380ce4f0 legacy=0x7f35380b1430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:44.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.848+0000 7f353d54b700 1 -- 172.21.15.16:0/2973024819 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35380b0c30 con 0x7f353815b630 2023-10-16T13:48:44.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.848+0000 7f353d54b700 1 --2- 172.21.15.16:0/2973024819 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f353815b630 0x7f35380d0380 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f3534009870 tx=0x7f3534009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4da24b347d66f0e8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:44.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.848+0000 7f352ffff700 1 -- 172.21.15.16:0/2973024819 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3534011830 con 0x7f353815b630 2023-10-16T13:48:44.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.849+0000 7f352ffff700 1 -- 172.21.15.16:0/2973024819 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3534011990 con 0x7f353815b630 2023-10-16T13:48:44.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.849+0000 7f352ffff700 1 -- 172.21.15.16:0/2973024819 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3534011d20 con 0x7f353815b630 2023-10-16T13:48:44.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.849+0000 7f353f7af700 1 -- 172.21.15.16:0/2973024819 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f353815b630 msgr2=0x7f35380d0380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:44.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.849+0000 7f353f7af700 1 --2- 172.21.15.16:0/2973024819 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f353815b630 0x7f35380d0380 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f3534009870 tx=0x7f3534009b80 comp rx=0 tx=0).stop 2023-10-16T13:48:44.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.849+0000 7f353f7af700 1 -- 172.21.15.16:0/2973024819 shutdown_connections 2023-10-16T13:48:44.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.849+0000 7f353f7af700 1 --2- 172.21.15.16:0/2973024819 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f353815b630 0x7f35380d0380 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:44.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.849+0000 7f353f7af700 1 -- 172.21.15.16:0/2973024819 >> 172.21.15.16:0/2973024819 conn(0x7f35380d14b0 msgr2=0x7f35380d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:44.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.849+0000 7f353f7af700 1 -- 172.21.15.16:0/2973024819 shutdown_connections 2023-10-16T13:48:44.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.849+0000 7f353f7af700 1 -- 172.21.15.16:0/2973024819 wait complete. 2023-10-16T13:48:44.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.851+0000 7f353f7af700 1 Processor -- start 2023-10-16T13:48:44.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.851+0000 7f353f7af700 1 -- start start 2023-10-16T13:48:44.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.852+0000 7f353f7af700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35380ce4f0 0x7f35380c5e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:44.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.852+0000 7f353f7af700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f353815b630 0x7f3538150a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:44.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.852+0000 7f353f7af700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f353815ba10 0x7f35380c79e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:44.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.852+0000 7f353f7af700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3538150f80 con 0x7f35380ce4f0 2023-10-16T13:48:44.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.852+0000 7f353f7af700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35380c7f20 con 0x7f353815b630 2023-10-16T13:48:44.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.852+0000 7f353f7af700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35380c6440 con 0x7f353815ba10 2023-10-16T13:48:44.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.852+0000 7f353d54b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35380ce4f0 0x7f35380c5e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:44.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.852+0000 7f353d54b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35380ce4f0 0x7f35380c5e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58320/0 (socket says 172.21.15.16:58320) 2023-10-16T13:48:44.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.852+0000 7f353d54b700 1 -- 172.21.15.16:0/4268143888 learned_addr learned my addr 172.21.15.16:0/4268143888 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:44.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.852+0000 7f353cd4a700 1 --2- 172.21.15.16:0/4268143888 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f353815b630 0x7f3538150a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:44.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.852+0000 7f353dd4c700 1 --2- 172.21.15.16:0/4268143888 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f353815ba10 0x7f35380c79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:44.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.852+0000 7f353cd4a700 1 -- 172.21.15.16:0/4268143888 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f353815ba10 msgr2=0x7f35380c79e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:44.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.852+0000 7f353cd4a700 1 --2- 172.21.15.16:0/4268143888 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f353815ba10 0x7f35380c79e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:44.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.852+0000 7f353cd4a700 1 -- 172.21.15.16:0/4268143888 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35380ce4f0 msgr2=0x7f35380c5e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:44.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.852+0000 7f353cd4a700 1 --2- 172.21.15.16:0/4268143888 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35380ce4f0 0x7f35380c5e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:44.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.852+0000 7f353cd4a700 1 -- 172.21.15.16:0/4268143888 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35340094a0 con 0x7f353815b630 2023-10-16T13:48:44.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.852+0000 7f353d54b700 1 --2- 172.21.15.16:0/4268143888 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35380ce4f0 0x7f35380c5e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:48:44.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.853+0000 7f353cd4a700 1 --2- 172.21.15.16:0/4268143888 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f353815b630 0x7f3538150a40 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f352800c3c0 tx=0x7f352800c6d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:44.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.853+0000 7f352dffb700 1 -- 172.21.15.16:0/4268143888 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35280052d0 con 0x7f353815b630 2023-10-16T13:48:44.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.853+0000 7f352dffb700 1 -- 172.21.15.16:0/4268143888 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3528005430 con 0x7f353815b630 2023-10-16T13:48:44.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.853+0000 7f353f7af700 1 -- 172.21.15.16:0/4268143888 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35380c6720 con 0x7f353815b630 2023-10-16T13:48:44.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.853+0000 7f352dffb700 1 -- 172.21.15.16:0/4268143888 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35280038c0 con 0x7f353815b630 2023-10-16T13:48:44.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.853+0000 7f353f7af700 1 -- 172.21.15.16:0/4268143888 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35380c6ce0 con 0x7f353815b630 2023-10-16T13:48:44.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.854+0000 7f353f7af700 1 -- 172.21.15.16:0/4268143888 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3538066a90 con 0x7f353815b630 2023-10-16T13:48:44.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.855+0000 7f352dffb700 1 -- 172.21.15.16:0/4268143888 <== mon.1 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 162) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f3528003a20 con 0x7f353815b630 2023-10-16T13:48:44.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.855+0000 7f352dffb700 1 --2- 172.21.15.16:0/4268143888 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f351c06b020 0x7f351c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:44.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.855+0000 7f353d54b700 1 -- 172.21.15.16:0/4268143888 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f351c06b020 msgr2=0x7f351c06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2131389174 2023-10-16T13:48:44.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.855+0000 7f352dffb700 1 -- 172.21.15.16:0/4268143888 <== mon.1 v2:172.21.15.49:3300/0 5 ==== osd_map(266..266 src has 1..266) v4 ==== 6587+0+0 (secure 0 0 0) 0x7f3528088d80 con 0x7f353815b630 2023-10-16T13:48:44.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.855+0000 7f353d54b700 1 --2- 172.21.15.16:0/4268143888 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f351c06b020 0x7f351c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:48:44.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:44.860+0000 7f352dffb700 1 -- 172.21.15.16:0/4268143888 <== mon.1 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3528053a80 con 0x7f353815b630 2023-10-16T13:48:45.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.056+0000 7f353d54b700 1 -- 172.21.15.16:0/4268143888 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f351c06b020 msgr2=0x7f351c06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2131389174 2023-10-16T13:48:45.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.056+0000 7f353d54b700 1 --2- 172.21.15.16:0/4268143888 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f351c06b020 0x7f351c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:48:45.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.125+0000 7f353f7af700 1 -- 172.21.15.16:0/4268143888 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f353804ebb0 con 0x7f353815b630 2023-10-16T13:48:45.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.126+0000 7f352dffb700 1 -- 172.21.15.16:0/4268143888 <== mon.1 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v266) v1 ==== 81+0+584 (secure 0 0 0) 0x7f35280570a0 con 0x7f353815b630 2023-10-16T13:48:45.128 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3313712117 2023-10-17T13:48:39.972713+0000 2023-10-16T13:48:45.128 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2202829273 2023-10-17T13:48:39.972713+0000 2023-10-16T13:48:45.129 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2777724108 2023-10-17T13:48:39.972713+0000 2023-10-16T13:48:45.129 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2088077503 2023-10-17T13:48:39.972713+0000 2023-10-16T13:48:45.129 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/4096007345 2023-10-17T13:48:39.972713+0000 2023-10-16T13:48:45.129 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3000870029 2023-10-17T13:48:39.972713+0000 2023-10-16T13:48:45.129 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3923653515 2023-10-17T13:48:39.972713+0000 2023-10-16T13:48:45.130 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/56667078 2023-10-17T13:48:39.972713+0000 2023-10-16T13:48:45.130 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6833/1275231846 2023-10-17T13:48:39.972713+0000 2023-10-16T13:48:45.130 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6832/1275231846 2023-10-17T13:48:39.972713+0000 2023-10-16T13:48:45.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.128+0000 7f353f7af700 1 -- 172.21.15.16:0/4268143888 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f351c06b020 msgr2=0x7f351c06d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:48:45.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.128+0000 7f353f7af700 1 --2- 172.21.15.16:0/4268143888 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f351c06b020 0x7f351c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:45.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.128+0000 7f353f7af700 1 -- 172.21.15.16:0/4268143888 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f353815b630 msgr2=0x7f3538150a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:45.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.128+0000 7f353f7af700 1 --2- 172.21.15.16:0/4268143888 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f353815b630 0x7f3538150a40 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f352800c3c0 tx=0x7f352800c6d0 comp rx=0 tx=0).stop 2023-10-16T13:48:45.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.128+0000 7f353f7af700 1 -- 172.21.15.16:0/4268143888 shutdown_connections 2023-10-16T13:48:45.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.128+0000 7f353f7af700 1 --2- 172.21.15.16:0/4268143888 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f351c06b020 0x7f351c06d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:45.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.128+0000 7f353f7af700 1 --2- 172.21.15.16:0/4268143888 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f353815ba10 0x7f35380c79e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:45.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.128+0000 7f353f7af700 1 --2- 172.21.15.16:0/4268143888 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35380ce4f0 0x7f35380c5e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:45.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.128+0000 7f353f7af700 1 --2- 172.21.15.16:0/4268143888 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f353815b630 0x7f3538150a40 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:45.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.128+0000 7f353f7af700 1 -- 172.21.15.16:0/4268143888 >> 172.21.15.16:0/4268143888 conn(0x7f35380d14b0 msgr2=0x7f35380d1890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:45.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.128+0000 7f353f7af700 1 -- 172.21.15.16:0/4268143888 shutdown_connections 2023-10-16T13:48:45.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.128+0000 7f353f7af700 1 -- 172.21.15.16:0/4268143888 wait complete. 2023-10-16T13:48:45.133 INFO:teuthology.orchestra.run.smithi016.stderr:listed 10 entries 2023-10-16T13:48:45.144 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-16T13:48:45.144 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:48:45.145 DEBUG:teuthology.orchestra.run.smithi016:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-16T13:48:45.150 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:48:45.147+0000 7f2fb0253200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T13:48:45.174 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:45.172+0000 7fd1d1dca200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T13:48:45.204 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2023-10-16T13:48:45.205 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:48:45.205 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-16T13:48:45.225 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:48:45.222+0000 7f2fb0253200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T13:48:45.248 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:45.246+0000 7fd1d1dca200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T13:48:45.262 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T13:48:45.262 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:48:45.280 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284204k,nr_inodes=4071051,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=753 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260540k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T13:48:45.281 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T13:48:45.281 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:48:45.320 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16287364k,nr_inodes=4071841,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20251 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3261172k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T13:48:45.320 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T13:48:45.320 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:48:45.338 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284204k,nr_inodes=4071051,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=753 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260540k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T13:48:45.338 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:48:45.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.484+0000 7fc03c52c700 1 Processor -- start 2023-10-16T13:48:45.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.484+0000 7fc03c52c700 1 -- start start 2023-10-16T13:48:45.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.485+0000 7fc03c52c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc03415b630 0x7fc0340d0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:45.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.485+0000 7fc03c52c700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fc03415ba10 0x7fc0340cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:45.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.485+0000 7fc03c52c700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0340b0970 con 0x7fc0340cdde0 2023-10-16T13:48:45.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.485+0000 7fc03c52c700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0340cd690 con 0x7fc03415ba10 2023-10-16T13:48:45.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.485+0000 7fc03c52c700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0340cd970 con 0x7fc03415b630 2023-10-16T13:48:45.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.485+0000 7fc03aac9700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc0340cdde0 0x7fc0340cc2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:47608/0 (socket says 172.21.15.16:47608) 2023-10-16T13:48:45.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.485+0000 7fc03aac9700 1 -- 172.21.15.16:0/1050405496 learned_addr learned my addr 172.21.15.16:0/1050405496 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:48:45.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.485+0000 7fc03a2c8700 1 --2- 172.21.15.16:0/1050405496 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc03415b630 0x7fc0340d0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:45.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.485+0000 7fc039ac7700 1 --2- 172.21.15.16:0/1050405496 >> v2:172.21.15.49:3300/0 conn(0x7fc03415ba10 0x7fc0340cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:45.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.486+0000 7fc0392c6700 1 -- 172.21.15.16:0/1050405496 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1900299799 0 0) 0x7fc0340b0970 con 0x7fc0340cdde0 2023-10-16T13:48:45.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.486+0000 7fc03a2c8700 1 -- 172.21.15.16:0/1050405496 >> v2:172.21.15.49:3300/0 conn(0x7fc03415ba10 msgr2=0x7fc0340cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:45.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.486+0000 7fc03a2c8700 1 --2- 172.21.15.16:0/1050405496 >> v2:172.21.15.49:3300/0 conn(0x7fc03415ba10 0x7fc0340cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:45.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.486+0000 7fc03a2c8700 1 -- 172.21.15.16:0/1050405496 >> v1:172.21.15.16:6789/0 conn(0x7fc0340cdde0 legacy=0x7fc0340cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:45.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.486+0000 7fc03a2c8700 1 -- 172.21.15.16:0/1050405496 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0340b0c30 con 0x7fc03415b630 2023-10-16T13:48:45.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.486+0000 7fc039ac7700 1 --2- 172.21.15.16:0/1050405496 >> v2:172.21.15.49:3300/0 conn(0x7fc03415ba10 0x7fc0340cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:48:45.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.486+0000 7fc03a2c8700 1 --2- 172.21.15.16:0/1050405496 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc03415b630 0x7fc0340d0380 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fc030009840 tx=0x7fc030009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe9528f5eb179062 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:45.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.486+0000 7fc0392c6700 1 -- 172.21.15.16:0/1050405496 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc030011830 con 0x7fc03415b630 2023-10-16T13:48:45.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.486+0000 7fc0392c6700 1 -- 172.21.15.16:0/1050405496 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc030011990 con 0x7fc03415b630 2023-10-16T13:48:45.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.486+0000 7fc0392c6700 1 -- 172.21.15.16:0/1050405496 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc030011d20 con 0x7fc03415b630 2023-10-16T13:48:45.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.487+0000 7fc03c52c700 1 -- 172.21.15.16:0/1050405496 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc03415b630 msgr2=0x7fc0340d0380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:45.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.487+0000 7fc03c52c700 1 --2- 172.21.15.16:0/1050405496 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc03415b630 0x7fc0340d0380 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fc030009840 tx=0x7fc030009b50 comp rx=0 tx=0).stop 2023-10-16T13:48:45.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.487+0000 7fc03c52c700 1 -- 172.21.15.16:0/1050405496 shutdown_connections 2023-10-16T13:48:45.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.487+0000 7fc03c52c700 1 --2- 172.21.15.16:0/1050405496 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc03415b630 0x7fc0340d0380 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:45.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.487+0000 7fc03c52c700 1 --2- 172.21.15.16:0/1050405496 >> v2:172.21.15.49:3300/0 conn(0x7fc03415ba10 0x7fc0340cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:45.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.487+0000 7fc03c52c700 1 -- 172.21.15.16:0/1050405496 >> 172.21.15.16:0/1050405496 conn(0x7fc0340d14b0 msgr2=0x7fc0340d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:45.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.487+0000 7fc03c52c700 1 -- 172.21.15.16:0/1050405496 shutdown_connections 2023-10-16T13:48:45.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.487+0000 7fc03c52c700 1 -- 172.21.15.16:0/1050405496 wait complete. 2023-10-16T13:48:45.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.488+0000 7fc03c52c700 1 Processor -- start 2023-10-16T13:48:45.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.488+0000 7fc03c52c700 1 -- start start 2023-10-16T13:48:45.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.489+0000 7fc03c52c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0340cdde0 0x7fc0340b7000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:45.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.489+0000 7fc03c52c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc03415b630 0x7fc03414b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:45.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.489+0000 7fc03c52c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc03415ba10 0x7fc03414f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:45.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.489+0000 7fc03c52c700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc03414c870 con 0x7fc0340cdde0 2023-10-16T13:48:45.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.489+0000 7fc03c52c700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc03414bdc0 con 0x7fc03415ba10 2023-10-16T13:48:45.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.489+0000 7fc03c52c700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0340c9800 con 0x7fc03415b630 2023-10-16T13:48:45.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.489+0000 7fc03a2c8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0340cdde0 0x7fc0340b7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:45.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.489+0000 7fc03a2c8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0340cdde0 0x7fc0340b7000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58336/0 (socket says 172.21.15.16:58336) 2023-10-16T13:48:45.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.489+0000 7fc03a2c8700 1 -- 172.21.15.16:0/431525874 learned_addr learned my addr 172.21.15.16:0/431525874 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:45.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.489+0000 7fc039ac7700 1 --2- 172.21.15.16:0/431525874 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc03415b630 0x7fc03414b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:45.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.489+0000 7fc03aac9700 1 --2- 172.21.15.16:0/431525874 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc03415ba10 0x7fc03414f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:45.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.489+0000 7fc03a2c8700 1 -- 172.21.15.16:0/431525874 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc03415b630 msgr2=0x7fc03414b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:45.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.489+0000 7fc03a2c8700 1 --2- 172.21.15.16:0/431525874 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc03415b630 0x7fc03414b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:45.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.489+0000 7fc03a2c8700 1 -- 172.21.15.16:0/431525874 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc03415ba10 msgr2=0x7fc03414f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:45.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.489+0000 7fc03a2c8700 1 --2- 172.21.15.16:0/431525874 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc03415ba10 0x7fc03414f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:45.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.489+0000 7fc03a2c8700 1 -- 172.21.15.16:0/431525874 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0240094a0 con 0x7fc0340cdde0 2023-10-16T13:48:45.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.489+0000 7fc03a2c8700 1 --2- 172.21.15.16:0/431525874 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0340cdde0 0x7fc0340b7000 secure :-1 s=READY pgs=3084 cs=0 l=1 rev1=1 crypto rx=0x7fc0300090b0 tx=0x7fc030004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:45.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.490+0000 7fc02affd700 1 -- 172.21.15.16:0/431525874 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc030004550 con 0x7fc0340cdde0 2023-10-16T13:48:45.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.490+0000 7fc02affd700 1 -- 172.21.15.16:0/431525874 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc03001f070 con 0x7fc0340cdde0 2023-10-16T13:48:45.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.490+0000 7fc03c52c700 1 -- 172.21.15.16:0/431525874 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0300094a0 con 0x7fc0340cdde0 2023-10-16T13:48:45.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.490+0000 7fc02affd700 1 -- 172.21.15.16:0/431525874 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0300038e0 con 0x7fc0340cdde0 2023-10-16T13:48:45.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.490+0000 7fc03c52c700 1 -- 172.21.15.16:0/431525874 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc03414c2b0 con 0x7fc0340cdde0 2023-10-16T13:48:45.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.491+0000 7fc03c52c700 1 -- 172.21.15.16:0/431525874 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0340c95d0 con 0x7fc0340cdde0 2023-10-16T13:48:45.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.491+0000 7fc02affd700 1 -- 172.21.15.16:0/431525874 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 162) v1 ==== 89332+0+0 (secure 0 0 0) 0x7fc030003a40 con 0x7fc0340cdde0 2023-10-16T13:48:45.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.492+0000 7fc02affd700 1 --2- 172.21.15.16:0/431525874 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7fc01806b020 0x7fc01806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:45.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.492+0000 7fc039ac7700 1 -- 172.21.15.16:0/431525874 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7fc01806b020 msgr2=0x7fc01806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2131389174 2023-10-16T13:48:45.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.492+0000 7fc02affd700 1 -- 172.21.15.16:0/431525874 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(266..266 src has 1..266) v4 ==== 6587+0+0 (secure 0 0 0) 0x7fc030021070 con 0x7fc0340cdde0 2023-10-16T13:48:45.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.492+0000 7fc039ac7700 1 --2- 172.21.15.16:0/431525874 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7fc01806b020 0x7fc01806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:48:45.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.501+0000 7fc02affd700 1 -- 172.21.15.16:0/431525874 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc030055f30 con 0x7fc0340cdde0 2023-10-16T13:48:45.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.693+0000 7fc039ac7700 1 -- 172.21.15.16:0/431525874 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7fc01806b020 msgr2=0x7fc01806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2131389174 2023-10-16T13:48:45.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.693+0000 7fc039ac7700 1 --2- 172.21.15.16:0/431525874 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7fc01806b020 0x7fc01806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:48:45.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.753+0000 7fc03c52c700 1 -- 172.21.15.16:0/431525874 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc03404ebb0 con 0x7fc0340cdde0 2023-10-16T13:48:45.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.753+0000 7fc02affd700 1 -- 172.21.15.16:0/431525874 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 146 v146) v1 ==== 95+0+4926 (secure 0 0 0) 0x7fc030059550 con 0x7fc0340cdde0 2023-10-16T13:48:45.756 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:48:45.756 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":146,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10112,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/712583564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":712583564},{"type":"v1","addr":"172.21.15.16:6837","nonce":712583564}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":141},{"gid":11836,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/3551114031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":3551114031},{"type":"v1","addr":"172.21.15.49:6834","nonce":3551114031}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":141},{"gid":13050,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/2135643757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2135643757},{"type":"v1","addr":"172.21.15.16:6836","nonce":2135643757}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":140}],"filesystems":[{"mdsmap":{"epoch":146,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:48:24.832614+0000","modified":"2023-10-16T13:48:27.858697+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":13068},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13068":{"gid":13068,"name":"d","rank":0,"incarnation":142,"state":"up:active","state_seq":3,"addr":"172.21.15.49:6835/1985453659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":1985453659},{"type":"v1","addr":"172.21.15.49:6835","nonce":1985453659}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[36],"metadata_pool":35,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":17}]} 2023-10-16T13:48:45.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.755+0000 7fc03c52c700 1 -- 172.21.15.16:0/431525874 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7fc01806b020 msgr2=0x7fc01806d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:48:45.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.755+0000 7fc03c52c700 1 --2- 172.21.15.16:0/431525874 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7fc01806b020 0x7fc01806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:45.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.756+0000 7fc03c52c700 1 -- 172.21.15.16:0/431525874 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0340cdde0 msgr2=0x7fc0340b7000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:45.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.756+0000 7fc03c52c700 1 --2- 172.21.15.16:0/431525874 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0340cdde0 0x7fc0340b7000 secure :-1 s=READY pgs=3084 cs=0 l=1 rev1=1 crypto rx=0x7fc0300090b0 tx=0x7fc030004750 comp rx=0 tx=0).stop 2023-10-16T13:48:45.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.756+0000 7fc03c52c700 1 -- 172.21.15.16:0/431525874 shutdown_connections 2023-10-16T13:48:45.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.756+0000 7fc03c52c700 1 --2- 172.21.15.16:0/431525874 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7fc01806b020 0x7fc01806d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:45.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.756+0000 7fc03c52c700 1 --2- 172.21.15.16:0/431525874 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc03415ba10 0x7fc03414f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:45.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.756+0000 7fc03c52c700 1 --2- 172.21.15.16:0/431525874 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0340cdde0 0x7fc0340b7000 unknown :-1 s=CLOSED pgs=3084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:45.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.756+0000 7fc03c52c700 1 --2- 172.21.15.16:0/431525874 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc03415b630 0x7fc03414b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:45.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.756+0000 7fc03c52c700 1 -- 172.21.15.16:0/431525874 >> 172.21.15.16:0/431525874 conn(0x7fc0340d14b0 msgr2=0x7fc034070480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:45.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.756+0000 7fc03c52c700 1 -- 172.21.15.16:0/431525874 shutdown_connections 2023-10-16T13:48:45.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.756+0000 7fc03c52c700 1 -- 172.21.15.16:0/431525874 wait complete. 2023-10-16T13:48:45.761 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 146 2023-10-16T13:48:45.771 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:48:45.899 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:48:45.897+0000 7f2fb0253200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:48:45.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.898+0000 7f31a81fe700 1 Processor -- start 2023-10-16T13:48:45.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.898+0000 7f31a81fe700 1 -- start start 2023-10-16T13:48:45.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.899+0000 7f31a81fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f31a015b9f0 0x7f31a00b29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:45.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.899+0000 7f31a81fe700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f31a00cf280 0x7f31a006ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:45.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.899+0000 7f31a81fe700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31a00c3490 con 0x7f31a006d940 2023-10-16T13:48:45.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.899+0000 7f31a81fe700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31a006e300 con 0x7f31a015b9f0 2023-10-16T13:48:45.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.899+0000 7f31a81fe700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31a00cf730 con 0x7f31a00cf280 2023-10-16T13:48:45.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.899+0000 7f31a5f9a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f31a015b9f0 0x7f31a00b29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:45.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.899+0000 7f31a5f9a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f31a015b9f0 0x7f31a00b29a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58346/0 (socket says 172.21.15.16:58346) 2023-10-16T13:48:45.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.899+0000 7f31a5f9a700 1 -- 172.21.15.16:0/328226507 learned_addr learned my addr 172.21.15.16:0/328226507 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:45.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.899+0000 7f31a679b700 1 --2- 172.21.15.16:0/328226507 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f31a00cf280 0x7f31a006ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:45.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.900+0000 7f31a5f9a700 1 -- 172.21.15.16:0/328226507 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f31a00cf280 msgr2=0x7f31a006ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:45.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.900+0000 7f31a5f9a700 1 --2- 172.21.15.16:0/328226507 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f31a00cf280 0x7f31a006ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:45.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.900+0000 7f31a5f9a700 1 -- 172.21.15.16:0/328226507 >> v1:172.21.15.49:6789/0 conn(0x7f31a006d940 legacy=0x7f31a00ceb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:45.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.900+0000 7f31a4f98700 1 -- 172.21.15.16:0/328226507 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2913396000 0 0) 0x7f31a00c3490 con 0x7f31a006d940 2023-10-16T13:48:45.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.900+0000 7f31a5f9a700 1 -- 172.21.15.16:0/328226507 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31a00cdad0 con 0x7f31a015b9f0 2023-10-16T13:48:45.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.900+0000 7f31a5f9a700 1 --2- 172.21.15.16:0/328226507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31a015b9f0 0x7f31a00b29a0 secure :-1 s=READY pgs=3085 cs=0 l=1 rev1=1 crypto rx=0x7f319c0098f0 tx=0x7f319c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c7b11a3959b18cf server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:45.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.900+0000 7f31a4f98700 1 -- 172.21.15.16:0/328226507 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f319c011830 con 0x7f31a015b9f0 2023-10-16T13:48:45.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.900+0000 7f31a4f98700 1 -- 172.21.15.16:0/328226507 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f319c011990 con 0x7f31a015b9f0 2023-10-16T13:48:45.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.900+0000 7f31a4f98700 1 -- 172.21.15.16:0/328226507 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f319c011d20 con 0x7f31a015b9f0 2023-10-16T13:48:45.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.901+0000 7f31a81fe700 1 -- 172.21.15.16:0/328226507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31a015b9f0 msgr2=0x7f31a00b29a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:45.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.901+0000 7f31a81fe700 1 --2- 172.21.15.16:0/328226507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31a015b9f0 0x7f31a00b29a0 secure :-1 s=READY pgs=3085 cs=0 l=1 rev1=1 crypto rx=0x7f319c0098f0 tx=0x7f319c009c00 comp rx=0 tx=0).stop 2023-10-16T13:48:45.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.901+0000 7f31a81fe700 1 -- 172.21.15.16:0/328226507 shutdown_connections 2023-10-16T13:48:45.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.901+0000 7f31a81fe700 1 --2- 172.21.15.16:0/328226507 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f31a00cf280 0x7f31a006ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:45.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.901+0000 7f31a81fe700 1 --2- 172.21.15.16:0/328226507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31a015b9f0 0x7f31a00b29a0 unknown :-1 s=CLOSED pgs=3085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:45.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.901+0000 7f31a81fe700 1 -- 172.21.15.16:0/328226507 >> 172.21.15.16:0/328226507 conn(0x7f31a00cc5b0 msgr2=0x7f31a00cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:45.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.901+0000 7f31a81fe700 1 -- 172.21.15.16:0/328226507 shutdown_connections 2023-10-16T13:48:45.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.901+0000 7f31a81fe700 1 -- 172.21.15.16:0/328226507 wait complete. 2023-10-16T13:48:45.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.902+0000 7f31a81fe700 1 Processor -- start 2023-10-16T13:48:45.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.902+0000 7f31a81fe700 1 -- start start 2023-10-16T13:48:45.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.903+0000 7f31a81fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31a006d940 0x7f31a01498f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:45.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.903+0000 7f31a81fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f31a00cf280 0x7f31a00c5e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:45.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.903+0000 7f31a81fe700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f31a00c63a0 0x7f31a00cb2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:45.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.903+0000 7f31a81fe700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31a00b14e0 con 0x7f31a006d940 2023-10-16T13:48:45.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.903+0000 7f31a81fe700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31a00ca220 con 0x7f31a00cf280 2023-10-16T13:48:45.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.903+0000 7f31a81fe700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31a00ca500 con 0x7f31a00c63a0 2023-10-16T13:48:45.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.903+0000 7f31a5f9a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31a006d940 0x7f31a01498f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:45.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.903+0000 7f31a5f9a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31a006d940 0x7f31a01498f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58350/0 (socket says 172.21.15.16:58350) 2023-10-16T13:48:45.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.903+0000 7f31a5f9a700 1 -- 172.21.15.16:0/3544347388 learned_addr learned my addr 172.21.15.16:0/3544347388 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:45.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.903+0000 7f31a679b700 1 --2- 172.21.15.16:0/3544347388 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f31a00c63a0 0x7f31a00cb2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:45.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.903+0000 7f31a5799700 1 --2- 172.21.15.16:0/3544347388 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f31a00cf280 0x7f31a00c5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:45.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.903+0000 7f31a5f9a700 1 -- 172.21.15.16:0/3544347388 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f31a00c63a0 msgr2=0x7f31a00cb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:45.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.903+0000 7f31a5f9a700 1 --2- 172.21.15.16:0/3544347388 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f31a00c63a0 0x7f31a00cb2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:45.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.903+0000 7f31a5f9a700 1 -- 172.21.15.16:0/3544347388 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f31a00cf280 msgr2=0x7f31a00c5e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:45.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.903+0000 7f31a5f9a700 1 --2- 172.21.15.16:0/3544347388 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f31a00cf280 0x7f31a00c5e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:45.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.903+0000 7f31a5f9a700 1 -- 172.21.15.16:0/3544347388 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f319c009550 con 0x7f31a006d940 2023-10-16T13:48:45.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.904+0000 7f31a5f9a700 1 --2- 172.21.15.16:0/3544347388 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31a006d940 0x7f31a01498f0 secure :-1 s=READY pgs=3086 cs=0 l=1 rev1=1 crypto rx=0x7f319c009c40 tx=0x7f319c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:45.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.904+0000 7f31927fc700 1 -- 172.21.15.16:0/3544347388 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f319c004550 con 0x7f31a006d940 2023-10-16T13:48:45.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.904+0000 7f31927fc700 1 -- 172.21.15.16:0/3544347388 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f319c01f070 con 0x7f31a006d940 2023-10-16T13:48:45.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.904+0000 7f31a81fe700 1 -- 172.21.15.16:0/3544347388 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31a00ca7e0 con 0x7f31a006d940 2023-10-16T13:48:45.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.904+0000 7f31927fc700 1 -- 172.21.15.16:0/3544347388 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f319c0038e0 con 0x7f31a006d940 2023-10-16T13:48:45.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.904+0000 7f31a81fe700 1 -- 172.21.15.16:0/3544347388 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31a00cada0 con 0x7f31a006d940 2023-10-16T13:48:45.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.905+0000 7f31a81fe700 1 -- 172.21.15.16:0/3544347388 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31a006ee70 con 0x7f31a006d940 2023-10-16T13:48:45.915 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.905+0000 7f31927fc700 1 -- 172.21.15.16:0/3544347388 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 162) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f319c003a40 con 0x7f31a006d940 2023-10-16T13:48:45.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.906+0000 7f31927fc700 1 --2- 172.21.15.16:0/3544347388 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f318406b020 0x7f318406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:45.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.906+0000 7f31a5799700 1 -- 172.21.15.16:0/3544347388 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f318406b020 msgr2=0x7f318406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2131389174 2023-10-16T13:48:45.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.906+0000 7f31927fc700 1 -- 172.21.15.16:0/3544347388 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(266..266 src has 1..266) v4 ==== 6587+0+0 (secure 0 0 0) 0x7f319c021070 con 0x7f31a006d940 2023-10-16T13:48:45.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.906+0000 7f31a5799700 1 --2- 172.21.15.16:0/3544347388 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f318406b020 0x7f318406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:48:45.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:45.911+0000 7f31927fc700 1 -- 172.21.15.16:0/3544347388 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f319c055f60 con 0x7f31a006d940 2023-10-16T13:48:45.943 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:45.940+0000 7fd1d1dca200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:48:45.981 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:48:45.979+0000 7f2fb0253200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T13:48:46.026 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:46.024+0000 7fd1d1dca200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T13:48:46.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.107+0000 7f31a5799700 1 -- 172.21.15.16:0/3544347388 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f318406b020 msgr2=0x7f318406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2131389174 2023-10-16T13:48:46.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.107+0000 7f31a5799700 1 --2- 172.21.15.16:0/3544347388 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f318406b020 0x7f318406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:48:46.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.117+0000 7f31a81fe700 1 -- 172.21.15.16:0/3544347388 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f31a004ebb0 con 0x7f31a006d940 2023-10-16T13:48:46.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.118+0000 7f31927fc700 1 -- 172.21.15.16:0/3544347388 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v266) v1 ==== 74+0+17440 (secure 0 0 0) 0x7f319c059580 con 0x7f31a006d940 2023-10-16T13:48:46.121 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:48:46.122 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":266,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:48:39.972754+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":36,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_metadata","create_time":"2023-10-16T13:48:23.348668+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"264","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_data","create_time":"2023-10-16T13:48:23.799968+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"265","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/3313712117":"2023-10-17T13:48:39.972713+0000","172.21.15.16:0/2202829273":"2023-10-17T13:48:39.972713+0000","172.21.15.16:0/2777724108":"2023-10-17T13:48:39.972713+0000","172.21.15.16:0/2088077503":"2023-10-17T13:48:39.972713+0000","172.21.15.16:0/4096007345":"2023-10-17T13:48:39.972713+0000","172.21.15.16:0/3000870029":"2023-10-17T13:48:39.972713+0000","172.21.15.16:0/3923653515":"2023-10-17T13:48:39.972713+0000","172.21.15.16:0/56667078":"2023-10-17T13:48:39.972713+0000","172.21.15.16:6833/1275231846":"2023-10-17T13:48:39.972713+0000","172.21.15.16:6832/1275231846":"2023-10-17T13:48:39.972713+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:48:46.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.121+0000 7f31a81fe700 1 -- 172.21.15.16:0/3544347388 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f318406b020 msgr2=0x7f318406d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:48:46.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.121+0000 7f31a81fe700 1 --2- 172.21.15.16:0/3544347388 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f318406b020 0x7f318406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:46.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.121+0000 7f31a81fe700 1 -- 172.21.15.16:0/3544347388 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31a006d940 msgr2=0x7f31a01498f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:46.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.121+0000 7f31a81fe700 1 --2- 172.21.15.16:0/3544347388 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31a006d940 0x7f31a01498f0 secure :-1 s=READY pgs=3086 cs=0 l=1 rev1=1 crypto rx=0x7f319c009c40 tx=0x7f319c004750 comp rx=0 tx=0).stop 2023-10-16T13:48:46.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.121+0000 7f31a81fe700 1 -- 172.21.15.16:0/3544347388 shutdown_connections 2023-10-16T13:48:46.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.121+0000 7f31a81fe700 1 --2- 172.21.15.16:0/3544347388 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f318406b020 0x7f318406d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:46.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.121+0000 7f31a81fe700 1 --2- 172.21.15.16:0/3544347388 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f31a00c63a0 0x7f31a00cb2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:46.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.121+0000 7f31a81fe700 1 --2- 172.21.15.16:0/3544347388 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31a006d940 0x7f31a01498f0 unknown :-1 s=CLOSED pgs=3086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:46.125 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.121+0000 7f31a81fe700 1 --2- 172.21.15.16:0/3544347388 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f31a00cf280 0x7f31a00c5e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:46.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.121+0000 7f31a81fe700 1 -- 172.21.15.16:0/3544347388 >> 172.21.15.16:0/3544347388 conn(0x7f31a00cc5b0 msgr2=0x7f31a00b3030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:46.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.121+0000 7f31a81fe700 1 -- 172.21.15.16:0/3544347388 shutdown_connections 2023-10-16T13:48:46.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.121+0000 7f31a81fe700 1 -- 172.21.15.16:0/3544347388 wait complete. 2023-10-16T13:48:46.132 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:48:46.130+0000 7f2fb0253200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:48:46.136 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 35, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T13:48:23.348668+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '264', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 36, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T13:48:23.799968+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '265', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T13:48:46.137 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T13:48:46.186 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:46.184+0000 7fd1d1dca200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:48:46.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.267+0000 7f64dd342700 1 Processor -- start 2023-10-16T13:48:46.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.267+0000 7f64dd342700 1 -- start start 2023-10-16T13:48:46.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.268+0000 7f64dd342700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f64d815ec80 0x7f64d80cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:46.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.268+0000 7f64dd342700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f64d815f060 0x7f64d80ce400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:46.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.268+0000 7f64dd342700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64d80b2440 con 0x7f64d80d2110 2023-10-16T13:48:46.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.268+0000 7f64dd342700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64d80b2c50 con 0x7f64d815ec80 2023-10-16T13:48:46.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.268+0000 7f64dd342700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64d80d00e0 con 0x7f64d815f060 2023-10-16T13:48:46.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.268+0000 7f64d6ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f64d815ec80 0x7f64d80cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:46.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.268+0000 7f64d77fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f64d80d2110 0x7f64d80b4440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:47666/0 (socket says 172.21.15.16:47666) 2023-10-16T13:48:46.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.268+0000 7f64d77fe700 1 -- 172.21.15.16:0/3002208832 learned_addr learned my addr 172.21.15.16:0/3002208832 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:48:46.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.268+0000 7f64d67fc700 1 --2- 172.21.15.16:0/3002208832 >> v2:172.21.15.49:3300/0 conn(0x7f64d815f060 0x7f64d80ce400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:46.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.268+0000 7f64d6ffd700 1 -- 172.21.15.16:0/3002208832 >> v2:172.21.15.49:3300/0 conn(0x7f64d815f060 msgr2=0x7f64d80ce400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:46.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.268+0000 7f64d6ffd700 1 --2- 172.21.15.16:0/3002208832 >> v2:172.21.15.49:3300/0 conn(0x7f64d815f060 0x7f64d80ce400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:46.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.268+0000 7f64d6ffd700 1 -- 172.21.15.16:0/3002208832 >> v1:172.21.15.16:6789/0 conn(0x7f64d80d2110 legacy=0x7f64d80b4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:46.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.268+0000 7f64d5ffb700 1 -- 172.21.15.16:0/3002208832 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2306199284 0 0) 0x7f64d80b2440 con 0x7f64d80d2110 2023-10-16T13:48:46.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.269+0000 7f64d6ffd700 1 -- 172.21.15.16:0/3002208832 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64d80cef60 con 0x7f64d815ec80 2023-10-16T13:48:46.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.269+0000 7f64d6ffd700 1 --2- 172.21.15.16:0/3002208832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64d815ec80 0x7f64d80cdec0 secure :-1 s=READY pgs=3088 cs=0 l=1 rev1=1 crypto rx=0x7f64d80674b0 tx=0x7f64c8009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f0eb863829d9f5d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:46.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.269+0000 7f64d5ffb700 1 -- 172.21.15.16:0/3002208832 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64c8011830 con 0x7f64d815ec80 2023-10-16T13:48:46.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.269+0000 7f64d5ffb700 1 -- 172.21.15.16:0/3002208832 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64c8011990 con 0x7f64d815ec80 2023-10-16T13:48:46.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.269+0000 7f64d5ffb700 1 -- 172.21.15.16:0/3002208832 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64c8011d20 con 0x7f64d815ec80 2023-10-16T13:48:46.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.269+0000 7f64dd342700 1 -- 172.21.15.16:0/3002208832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64d815ec80 msgr2=0x7f64d80cdec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:46.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.269+0000 7f64dd342700 1 --2- 172.21.15.16:0/3002208832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64d815ec80 0x7f64d80cdec0 secure :-1 s=READY pgs=3088 cs=0 l=1 rev1=1 crypto rx=0x7f64d80674b0 tx=0x7f64c8009b20 comp rx=0 tx=0).stop 2023-10-16T13:48:46.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.269+0000 7f64dd342700 1 -- 172.21.15.16:0/3002208832 shutdown_connections 2023-10-16T13:48:46.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.269+0000 7f64dd342700 1 --2- 172.21.15.16:0/3002208832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64d815ec80 0x7f64d80cdec0 unknown :-1 s=CLOSED pgs=3088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:46.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.269+0000 7f64dd342700 1 --2- 172.21.15.16:0/3002208832 >> v2:172.21.15.49:3300/0 conn(0x7f64d815f060 0x7f64d80ce400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:46.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.269+0000 7f64dd342700 1 -- 172.21.15.16:0/3002208832 >> 172.21.15.16:0/3002208832 conn(0x7f64d80d15f0 msgr2=0x7f64d80d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:46.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.270+0000 7f64dd342700 1 -- 172.21.15.16:0/3002208832 shutdown_connections 2023-10-16T13:48:46.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.270+0000 7f64dd342700 1 -- 172.21.15.16:0/3002208832 wait complete. 2023-10-16T13:48:46.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.271+0000 7f64dd342700 1 Processor -- start 2023-10-16T13:48:46.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.271+0000 7f64dd342700 1 -- start start 2023-10-16T13:48:46.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.272+0000 7f64dd342700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f64d80d2110 0x7f64d80c6710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:46.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.272+0000 7f64dd342700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f64d815ec80 0x7f64d8149980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:46.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.272+0000 7f64dd342700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64d815f060 0x7f64d80c7630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:46.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.272+0000 7f64dd342700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64d8156dc0 con 0x7f64d815f060 2023-10-16T13:48:46.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.272+0000 7f64dd342700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64d80c5670 con 0x7f64d80d2110 2023-10-16T13:48:46.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.272+0000 7f64dd342700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64d80c5950 con 0x7f64d815ec80 2023-10-16T13:48:46.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.272+0000 7f64d77fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64d815f060 0x7f64d80c7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:46.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.272+0000 7f64d6ffd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f64d80d2110 0x7f64d80c6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:46.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.272+0000 7f64d77fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64d815f060 0x7f64d80c7630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58398/0 (socket says 172.21.15.16:58398) 2023-10-16T13:48:46.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.272+0000 7f64d77fe700 1 -- 172.21.15.16:0/1048993873 learned_addr learned my addr 172.21.15.16:0/1048993873 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:46.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.272+0000 7f64d67fc700 1 --2- 172.21.15.16:0/1048993873 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f64d815ec80 0x7f64d8149980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:46.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.272+0000 7f64d77fe700 1 -- 172.21.15.16:0/1048993873 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f64d815ec80 msgr2=0x7f64d8149980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:46.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.272+0000 7f64d77fe700 1 --2- 172.21.15.16:0/1048993873 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f64d815ec80 0x7f64d8149980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:46.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.272+0000 7f64d77fe700 1 -- 172.21.15.16:0/1048993873 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f64d80d2110 msgr2=0x7f64d80c6710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:46.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.272+0000 7f64d77fe700 1 --2- 172.21.15.16:0/1048993873 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f64d80d2110 0x7f64d80c6710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:46.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.272+0000 7f64d77fe700 1 -- 172.21.15.16:0/1048993873 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64c80094a0 con 0x7f64d815f060 2023-10-16T13:48:46.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.272+0000 7f64d77fe700 1 --2- 172.21.15.16:0/1048993873 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64d815f060 0x7f64d80c7630 secure :-1 s=READY pgs=3089 cs=0 l=1 rev1=1 crypto rx=0x7f64d0002230 tx=0x7f64d000cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:46.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.273+0000 7f64bf7fe700 1 -- 172.21.15.16:0/1048993873 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64d000aac0 con 0x7f64d815f060 2023-10-16T13:48:46.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.273+0000 7f64bf7fe700 1 -- 172.21.15.16:0/1048993873 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64d000ac40 con 0x7f64d815f060 2023-10-16T13:48:46.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.273+0000 7f64bf7fe700 1 -- 172.21.15.16:0/1048993873 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64d00070d0 con 0x7f64d815f060 2023-10-16T13:48:46.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.273+0000 7f64dd342700 1 -- 172.21.15.16:0/1048993873 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64d80c5c30 con 0x7f64d815f060 2023-10-16T13:48:46.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.273+0000 7f64dd342700 1 -- 172.21.15.16:0/1048993873 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64d80c61f0 con 0x7f64d815f060 2023-10-16T13:48:46.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.274+0000 7f64dd342700 1 -- 172.21.15.16:0/1048993873 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64d80c63e0 con 0x7f64d815f060 2023-10-16T13:48:46.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.274+0000 7f64bf7fe700 1 -- 172.21.15.16:0/1048993873 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 162) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f64d0007250 con 0x7f64d815f060 2023-10-16T13:48:46.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.275+0000 7f64bf7fe700 1 --2- 172.21.15.16:0/1048993873 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f64c406b020 0x7f64c406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:46.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.275+0000 7f64d6ffd700 1 -- 172.21.15.16:0/1048993873 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f64c406b020 msgr2=0x7f64c406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2131389174 2023-10-16T13:48:46.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.275+0000 7f64bf7fe700 1 -- 172.21.15.16:0/1048993873 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(266..266 src has 1..266) v4 ==== 6587+0+0 (secure 0 0 0) 0x7f64d00894c0 con 0x7f64d815f060 2023-10-16T13:48:46.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.275+0000 7f64d6ffd700 1 --2- 172.21.15.16:0/1048993873 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f64c406b020 0x7f64c406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:48:46.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.279+0000 7f64bf7fe700 1 -- 172.21.15.16:0/1048993873 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f64d00540f0 con 0x7f64d815f060 2023-10-16T13:48:46.372 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:48:46.370+0000 7f2fb0253200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T13:48:46.435 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:46.433+0000 7fd1d1dca200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T13:48:46.450 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:48:46.448+0000 7f2fb0253200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T13:48:46.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.476+0000 7f64d6ffd700 1 -- 172.21.15.16:0/1048993873 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f64c406b020 msgr2=0x7f64c406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2131389174 2023-10-16T13:48:46.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.476+0000 7f64d6ffd700 1 --2- 172.21.15.16:0/1048993873 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f64c406b020 0x7f64c406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:48:46.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.482+0000 7f64dd342700 1 -- 172.21.15.16:0/1048993873 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f64d804ebb0 con 0x7f64d815f060 2023-10-16T13:48:46.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.483+0000 7f64bf7fe700 1 -- 172.21.15.16:0/1048993873 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v266) v1 ==== 93+0+10 (secure 0 0 0) 0x7f64d0057710 con 0x7f64d815f060 2023-10-16T13:48:46.486 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T13:48:46.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.485+0000 7f64dd342700 1 -- 172.21.15.16:0/1048993873 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f64c406b020 msgr2=0x7f64c406d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:48:46.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.485+0000 7f64dd342700 1 --2- 172.21.15.16:0/1048993873 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f64c406b020 0x7f64c406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:46.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.485+0000 7f64dd342700 1 -- 172.21.15.16:0/1048993873 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64d815f060 msgr2=0x7f64d80c7630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:46.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.485+0000 7f64dd342700 1 --2- 172.21.15.16:0/1048993873 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64d815f060 0x7f64d80c7630 secure :-1 s=READY pgs=3089 cs=0 l=1 rev1=1 crypto rx=0x7f64d0002230 tx=0x7f64d000cf50 comp rx=0 tx=0).stop 2023-10-16T13:48:46.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.485+0000 7f64dd342700 1 -- 172.21.15.16:0/1048993873 shutdown_connections 2023-10-16T13:48:46.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.485+0000 7f64dd342700 1 --2- 172.21.15.16:0/1048993873 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f64c406b020 0x7f64c406d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:46.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.485+0000 7f64dd342700 1 --2- 172.21.15.16:0/1048993873 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64d815f060 0x7f64d80c7630 unknown :-1 s=CLOSED pgs=3089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:46.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.485+0000 7f64dd342700 1 --2- 172.21.15.16:0/1048993873 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f64d80d2110 0x7f64d80c6710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:46.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.485+0000 7f64dd342700 1 --2- 172.21.15.16:0/1048993873 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f64d815ec80 0x7f64d8149980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:46.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.485+0000 7f64dd342700 1 -- 172.21.15.16:0/1048993873 >> 172.21.15.16:0/1048993873 conn(0x7f64d80d15f0 msgr2=0x7f64d80b4330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:46.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.485+0000 7f64dd342700 1 -- 172.21.15.16:0/1048993873 shutdown_connections 2023-10-16T13:48:46.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.485+0000 7f64dd342700 1 -- 172.21.15.16:0/1048993873 wait complete. 2023-10-16T13:48:46.501 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T13:48:46.519 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:46.516+0000 7fd1d1dca200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T13:48:46.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.634+0000 7f754d7ae700 1 Processor -- start 2023-10-16T13:48:46.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.634+0000 7f754d7ae700 1 -- start start 2023-10-16T13:48:46.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.635+0000 7f754d7ae700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f754806a7e0 0x7f754806ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:46.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.635+0000 7f754d7ae700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f754815ecb0 con 0x7f754815b2c0 2023-10-16T13:48:46.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.635+0000 7f754d7ae700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f754815b6a0 con 0x7f754806e570 2023-10-16T13:48:46.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.635+0000 7f754d7ae700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f754806b1b0 con 0x7f754806a7e0 2023-10-16T13:48:46.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.635+0000 7f7546ffd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f754815b2c0 0x7f7548161340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:47678/0 (socket says 172.21.15.16:47678) 2023-10-16T13:48:46.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.635+0000 7f7546ffd700 1 -- 172.21.15.16:0/1430374434 learned_addr learned my addr 172.21.15.16:0/1430374434 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:48:46.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.635+0000 7f75477fe700 1 --2- 172.21.15.16:0/1430374434 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f754806a7e0 0x7f754806ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:46.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.635+0000 7f7545ffb700 1 -- 172.21.15.16:0/1430374434 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1026268762 0 0) 0x7f754815ecb0 con 0x7f754815b2c0 2023-10-16T13:48:46.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.635+0000 7f7545ffb700 1 -- 172.21.15.16:0/1430374434 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75300035b0 con 0x7f754815b2c0 2023-10-16T13:48:46.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.635+0000 7f7545ffb700 1 -- 172.21.15.16:0/1430374434 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1164736749 0 0) 0x7f754815b6a0 con 0x7f754806e570 2023-10-16T13:48:46.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.635+0000 7f7545ffb700 1 -- 172.21.15.16:0/1430374434 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f754815ecb0 con 0x7f754806e570 2023-10-16T13:48:46.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.636+0000 7f7545ffb700 1 -- 172.21.15.16:0/1430374434 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2892562681 0 0) 0x7f75300035b0 con 0x7f754815b2c0 2023-10-16T13:48:46.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.636+0000 7f7545ffb700 1 -- 172.21.15.16:0/1430374434 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f754806a7e0 msgr2=0x7f754806ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:46.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.636+0000 7f7545ffb700 1 --2- 172.21.15.16:0/1430374434 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f754806a7e0 0x7f754806ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:46.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.636+0000 7f7545ffb700 1 -- 172.21.15.16:0/1430374434 >> v1:172.21.15.49:6789/0 conn(0x7f754806e570 legacy=0x7f754815e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:46.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.636+0000 7f7545ffb700 1 -- 172.21.15.16:0/1430374434 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7548161b20 con 0x7f754815b2c0 2023-10-16T13:48:46.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.636+0000 7f7545ffb700 1 -- 172.21.15.16:0/1430374434 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f7538001670 con 0x7f754815b2c0 2023-10-16T13:48:46.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.636+0000 7f75477fe700 1 --2- 172.21.15.16:0/1430374434 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f754806a7e0 0x7f754806ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:48:46.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.636+0000 7f7545ffb700 1 -- 172.21.15.16:0/1430374434 >> v1:172.21.15.16:6789/0 conn(0x7f754815b2c0 legacy=0x7f7548161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:46.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.636+0000 7f7545ffb700 1 --2- 172.21.15.16:0/1430374434 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7530003bf0 0x7f7530003fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:46.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.636+0000 7f7545ffb700 1 --2- 172.21.15.16:0/1430374434 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7530004850 0x7f7530004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:46.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.636+0000 7f7545ffb700 1 --2- 172.21.15.16:0/1430374434 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f754806a7e0 0x7f7530005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:46.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.636+0000 7f7545ffb700 1 -- 172.21.15.16:0/1430374434 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f753c001230 con 0x7f7530003bf0 2023-10-16T13:48:46.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.636+0000 7f7545ffb700 1 -- 172.21.15.16:0/1430374434 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f754806b1b0 con 0x7f7530004850 2023-10-16T13:48:46.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.636+0000 7f7545ffb700 1 -- 172.21.15.16:0/1430374434 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f753000bb40 con 0x7f754806a7e0 2023-10-16T13:48:46.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.636+0000 7f7546ffd700 1 --2- 172.21.15.16:0/1430374434 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7530003bf0 0x7f7530003fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:46.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.636+0000 7f75467fc700 1 --2- 172.21.15.16:0/1430374434 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7530004850 0x7f7530004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:46.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.636+0000 7f75477fe700 1 --2- 172.21.15.16:0/1430374434 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f754806a7e0 0x7f7530005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:46.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.636+0000 7f7546ffd700 1 -- 172.21.15.16:0/1430374434 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f754806a7e0 msgr2=0x7f7530005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:46.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.636+0000 7f7546ffd700 1 --2- 172.21.15.16:0/1430374434 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f754806a7e0 0x7f7530005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:46.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.636+0000 7f7546ffd700 1 -- 172.21.15.16:0/1430374434 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7530004850 msgr2=0x7f7530004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:46.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.636+0000 7f7546ffd700 1 --2- 172.21.15.16:0/1430374434 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7530004850 0x7f7530004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:46.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.636+0000 7f7546ffd700 1 -- 172.21.15.16:0/1430374434 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7540009a70 con 0x7f7530003bf0 2023-10-16T13:48:46.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.636+0000 7f7546ffd700 1 --2- 172.21.15.16:0/1430374434 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7530003bf0 0x7f7530003fd0 secure :-1 s=READY pgs=3091 cs=0 l=1 rev1=1 crypto rx=0x7f7538001eb0 tx=0x7f7530005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a696875b2ee07dc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:46.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.637+0000 7f7545ffb700 1 -- 172.21.15.16:0/1430374434 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f753800cd80 con 0x7f7530003bf0 2023-10-16T13:48:46.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.637+0000 7f7545ffb700 1 -- 172.21.15.16:0/1430374434 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7538005960 con 0x7f7530003bf0 2023-10-16T13:48:46.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.637+0000 7f7545ffb700 1 -- 172.21.15.16:0/1430374434 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7538005d10 con 0x7f7530003bf0 2023-10-16T13:48:46.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.637+0000 7f754d7ae700 1 -- 172.21.15.16:0/1430374434 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7530003bf0 msgr2=0x7f7530003fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:46.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.637+0000 7f754d7ae700 1 --2- 172.21.15.16:0/1430374434 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7530003bf0 0x7f7530003fd0 secure :-1 s=READY pgs=3091 cs=0 l=1 rev1=1 crypto rx=0x7f7538001eb0 tx=0x7f7530005880 comp rx=0 tx=0).stop 2023-10-16T13:48:46.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.637+0000 7f75477fe700 1 -- 172.21.15.16:0/1430374434 reap_dead start 2023-10-16T13:48:46.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.637+0000 7f754d7ae700 1 -- 172.21.15.16:0/1430374434 shutdown_connections 2023-10-16T13:48:46.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.637+0000 7f754d7ae700 1 -- 172.21.15.16:0/1430374434 >> 172.21.15.16:0/1430374434 conn(0x7f75480b1f40 msgr2=0x7f75480b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:46.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.637+0000 7f754d7ae700 1 -- 172.21.15.16:0/1430374434 shutdown_connections 2023-10-16T13:48:46.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.637+0000 7f754d7ae700 1 -- 172.21.15.16:0/1430374434 wait complete. 2023-10-16T13:48:46.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.639+0000 7f754d7ae700 1 Processor -- start 2023-10-16T13:48:46.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.639+0000 7f754d7ae700 1 -- start start 2023-10-16T13:48:46.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.640+0000 7f754d7ae700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7530003bf0 0x7f75480c7ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:46.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.640+0000 7f754d7ae700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7548155970 0x7f75480b5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:46.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.640+0000 7f754d7ae700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f75480c9940 0x7f75480c9dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:46.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.640+0000 7f754d7ae700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75480ca300 con 0x7f7530003bf0 2023-10-16T13:48:46.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.640+0000 7f754d7ae700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f754815f290 con 0x7f75480c9940 2023-10-16T13:48:46.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.640+0000 7f754d7ae700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75480b58a0 con 0x7f7548155970 2023-10-16T13:48:46.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.640+0000 7f7546ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7530003bf0 0x7f75480c7ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:46.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.640+0000 7f7546ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7530003bf0 0x7f75480c7ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58416/0 (socket says 172.21.15.16:58416) 2023-10-16T13:48:46.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.640+0000 7f7546ffd700 1 -- 172.21.15.16:0/3710912864 learned_addr learned my addr 172.21.15.16:0/3710912864 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:46.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.640+0000 7f75467fc700 1 --2- 172.21.15.16:0/3710912864 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7548155970 0x7f75480b5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:46.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.640+0000 7f75477fe700 1 --2- 172.21.15.16:0/3710912864 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f75480c9940 0x7f75480c9dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:46.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.640+0000 7f7546ffd700 1 -- 172.21.15.16:0/3710912864 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7548155970 msgr2=0x7f75480b5360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:46.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.640+0000 7f7546ffd700 1 --2- 172.21.15.16:0/3710912864 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7548155970 0x7f75480b5360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:46.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.640+0000 7f7546ffd700 1 -- 172.21.15.16:0/3710912864 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f75480c9940 msgr2=0x7f75480c9dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:46.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.640+0000 7f7546ffd700 1 --2- 172.21.15.16:0/3710912864 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f75480c9940 0x7f75480c9dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:46.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.640+0000 7f7546ffd700 1 -- 172.21.15.16:0/3710912864 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75480c85b0 con 0x7f7530003bf0 2023-10-16T13:48:46.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.640+0000 7f7546ffd700 1 --2- 172.21.15.16:0/3710912864 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7530003bf0 0x7f75480c7ed0 secure :-1 s=READY pgs=3092 cs=0 l=1 rev1=1 crypto rx=0x7f753800c7d0 tx=0x7f7538001980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:46.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.640+0000 7f752f7fe700 1 -- 172.21.15.16:0/3710912864 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7538005cd0 con 0x7f7530003bf0 2023-10-16T13:48:46.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.640+0000 7f754d7ae700 1 -- 172.21.15.16:0/3710912864 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75480c8890 con 0x7f7530003bf0 2023-10-16T13:48:46.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.640+0000 7f752f7fe700 1 -- 172.21.15.16:0/3710912864 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7538005e30 con 0x7f7530003bf0 2023-10-16T13:48:46.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.640+0000 7f752f7fe700 1 -- 172.21.15.16:0/3710912864 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7538004a70 con 0x7f7530003bf0 2023-10-16T13:48:46.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.641+0000 7f754d7ae700 1 -- 172.21.15.16:0/3710912864 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75480c8e50 con 0x7f7530003bf0 2023-10-16T13:48:46.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.641+0000 7f754d7ae700 1 -- 172.21.15.16:0/3710912864 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75480c9070 con 0x7f7530003bf0 2023-10-16T13:48:46.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.642+0000 7f752f7fe700 1 -- 172.21.15.16:0/3710912864 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 162) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f7538004bd0 con 0x7f7530003bf0 2023-10-16T13:48:46.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.642+0000 7f752f7fe700 1 --2- 172.21.15.16:0/3710912864 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f753406b020 0x7f753406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:46.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.643+0000 7f75467fc700 1 -- 172.21.15.16:0/3710912864 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f753406b020 msgr2=0x7f753406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2131389174 2023-10-16T13:48:46.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.643+0000 7f752f7fe700 1 -- 172.21.15.16:0/3710912864 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(266..266 src has 1..266) v4 ==== 6587+0+0 (secure 0 0 0) 0x7f753808b060 con 0x7f7530003bf0 2023-10-16T13:48:46.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.643+0000 7f75467fc700 1 --2- 172.21.15.16:0/3710912864 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f753406b020 0x7f753406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:48:46.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.647+0000 7f752f7fe700 1 -- 172.21.15.16:0/3710912864 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7538055dc0 con 0x7f7530003bf0 2023-10-16T13:48:46.728 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:48:46.725+0000 7f2fb0253200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T13:48:46.808 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:46.806+0000 7fd1d1dca200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T13:48:46.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.844+0000 7f75467fc700 1 -- 172.21.15.16:0/3710912864 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f753406b020 msgr2=0x7f753406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2131389174 2023-10-16T13:48:46.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.844+0000 7f75467fc700 1 --2- 172.21.15.16:0/3710912864 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f753406b020 0x7f753406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:48:46.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.865+0000 7f754d7ae700 1 -- 172.21.15.16:0/3710912864 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f754815f5a0 con 0x7f7530003bf0 2023-10-16T13:48:46.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.866+0000 7f752f7fe700 1 -- 172.21.15.16:0/3710912864 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v266) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7538025e80 con 0x7f7530003bf0 2023-10-16T13:48:46.868 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-10-16T13:48:46.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.868+0000 7f754d7ae700 1 -- 172.21.15.16:0/3710912864 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f753406b020 msgr2=0x7f753406d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:48:46.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.868+0000 7f754d7ae700 1 --2- 172.21.15.16:0/3710912864 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f753406b020 0x7f753406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:46.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.868+0000 7f754d7ae700 1 -- 172.21.15.16:0/3710912864 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7530003bf0 msgr2=0x7f75480c7ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:46.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.868+0000 7f754d7ae700 1 --2- 172.21.15.16:0/3710912864 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7530003bf0 0x7f75480c7ed0 secure :-1 s=READY pgs=3092 cs=0 l=1 rev1=1 crypto rx=0x7f753800c7d0 tx=0x7f7538001980 comp rx=0 tx=0).stop 2023-10-16T13:48:46.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.868+0000 7f754d7ae700 1 -- 172.21.15.16:0/3710912864 shutdown_connections 2023-10-16T13:48:46.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.868+0000 7f754d7ae700 1 --2- 172.21.15.16:0/3710912864 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f753406b020 0x7f753406d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:46.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.868+0000 7f754d7ae700 1 --2- 172.21.15.16:0/3710912864 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f75480c9940 0x7f75480c9dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:46.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.868+0000 7f754d7ae700 1 --2- 172.21.15.16:0/3710912864 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7530003bf0 0x7f75480c7ed0 unknown :-1 s=CLOSED pgs=3092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:46.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.868+0000 7f754d7ae700 1 --2- 172.21.15.16:0/3710912864 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7548155970 0x7f75480b5360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:46.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.868+0000 7f754d7ae700 1 -- 172.21.15.16:0/3710912864 >> 172.21.15.16:0/3710912864 conn(0x7f75480b1f40 msgr2=0x7f75480cd4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:46.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.868+0000 7f754d7ae700 1 -- 172.21.15.16:0/3710912864 shutdown_connections 2023-10-16T13:48:46.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:46.868+0000 7f754d7ae700 1 -- 172.21.15.16:0/3710912864 wait complete. 2023-10-16T13:48:46.883 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T13:48:46.961 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:48:46.958+0000 7f2fb0253200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T13:48:47.050 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:47.048+0000 7fd1d1dca200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T13:48:47.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.051+0000 7f886247e700 1 Processor -- start 2023-10-16T13:48:47.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.051+0000 7f886247e700 1 -- start start 2023-10-16T13:48:47.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.052+0000 7f886247e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f885c15b2c0 0x7f885c161340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:47.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.052+0000 7f886247e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f885c06e570 0x7f885c15e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:47.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.052+0000 7f886247e700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f885c161880 con 0x7f885c06a7e0 2023-10-16T13:48:47.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.052+0000 7f886247e700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f885c15b6a0 con 0x7f885c15b2c0 2023-10-16T13:48:47.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.052+0000 7f886247e700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f885c161b20 con 0x7f885c06e570 2023-10-16T13:48:47.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.052+0000 7f885bfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f885c15b2c0 0x7f885c161340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:47.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.052+0000 7f885bfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f885c15b2c0 0x7f885c161340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58426/0 (socket says 172.21.15.16:58426) 2023-10-16T13:48:47.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.052+0000 7f8860a1b700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f885c06a7e0 0x7f885c071a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:47704/0 (socket says 172.21.15.16:47704) 2023-10-16T13:48:47.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.052+0000 7f885bfff700 1 -- 172.21.15.16:0/3066394752 learned_addr learned my addr 172.21.15.16:0/3066394752 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:47.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.052+0000 7f885b7fe700 1 --2- 172.21.15.16:0/3066394752 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f885c06e570 0x7f885c15e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:47.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.052+0000 7f885bfff700 1 -- 172.21.15.16:0/3066394752 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f885c06e570 msgr2=0x7f885c15e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:47.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.052+0000 7f885bfff700 1 --2- 172.21.15.16:0/3066394752 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f885c06e570 0x7f885c15e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:47.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.052+0000 7f885bfff700 1 -- 172.21.15.16:0/3066394752 >> v1:172.21.15.16:6789/0 conn(0x7f885c06a7e0 legacy=0x7f885c071a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:47.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.053+0000 7f885bfff700 1 -- 172.21.15.16:0/3066394752 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f885c161e00 con 0x7f885c15b2c0 2023-10-16T13:48:47.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.052+0000 7f885affd700 1 -- 172.21.15.16:0/3066394752 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 208046443 0 0) 0x7f885c161880 con 0x7f885c06a7e0 2023-10-16T13:48:47.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.053+0000 7f885bfff700 1 --2- 172.21.15.16:0/3066394752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f885c15b2c0 0x7f885c161340 secure :-1 s=READY pgs=3094 cs=0 l=1 rev1=1 crypto rx=0x7f884c0098f0 tx=0x7f884c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78c3362c4d28327b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:47.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.053+0000 7f885affd700 1 -- 172.21.15.16:0/3066394752 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f884c011830 con 0x7f885c15b2c0 2023-10-16T13:48:47.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.053+0000 7f885affd700 1 -- 172.21.15.16:0/3066394752 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f884c011990 con 0x7f885c15b2c0 2023-10-16T13:48:47.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.053+0000 7f885affd700 1 -- 172.21.15.16:0/3066394752 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f884c011d20 con 0x7f885c15b2c0 2023-10-16T13:48:47.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.053+0000 7f886247e700 1 -- 172.21.15.16:0/3066394752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f885c15b2c0 msgr2=0x7f885c161340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:47.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.053+0000 7f886247e700 1 --2- 172.21.15.16:0/3066394752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f885c15b2c0 0x7f885c161340 secure :-1 s=READY pgs=3094 cs=0 l=1 rev1=1 crypto rx=0x7f884c0098f0 tx=0x7f884c009c00 comp rx=0 tx=0).stop 2023-10-16T13:48:47.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.053+0000 7f886247e700 1 -- 172.21.15.16:0/3066394752 shutdown_connections 2023-10-16T13:48:47.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.053+0000 7f886247e700 1 --2- 172.21.15.16:0/3066394752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f885c15b2c0 0x7f885c161340 unknown :-1 s=CLOSED pgs=3094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:47.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.054+0000 7f886247e700 1 --2- 172.21.15.16:0/3066394752 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f885c06e570 0x7f885c15e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:47.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.054+0000 7f886247e700 1 -- 172.21.15.16:0/3066394752 >> 172.21.15.16:0/3066394752 conn(0x7f885c0b1f40 msgr2=0x7f885c0b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:47.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.054+0000 7f886247e700 1 -- 172.21.15.16:0/3066394752 shutdown_connections 2023-10-16T13:48:47.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.054+0000 7f886247e700 1 -- 172.21.15.16:0/3066394752 wait complete. 2023-10-16T13:48:47.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.055+0000 7f886247e700 1 Processor -- start 2023-10-16T13:48:47.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.055+0000 7f886247e700 1 -- start start 2023-10-16T13:48:47.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.056+0000 7f886247e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f885c06a7e0 0x7f885c1f6800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:47.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.056+0000 7f886247e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f885c06e570 0x7f885c1f6d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:47.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.056+0000 7f886247e700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f885c06ae00 0x7f885c0c7990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:47.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.056+0000 7f886247e700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f885c1f76b0 con 0x7f885c06a7e0 2023-10-16T13:48:47.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.056+0000 7f886247e700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f885c1f73d0 con 0x7f885c06ae00 2023-10-16T13:48:47.064 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.056+0000 7f886247e700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f885c0b4b10 con 0x7f885c06e570 2023-10-16T13:48:47.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.056+0000 7f885bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f885c06a7e0 0x7f885c1f6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:47.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.056+0000 7f885bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f885c06a7e0 0x7f885c1f6800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58432/0 (socket says 172.21.15.16:58432) 2023-10-16T13:48:47.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.056+0000 7f885bfff700 1 -- 172.21.15.16:0/2112360480 learned_addr learned my addr 172.21.15.16:0/2112360480 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:47.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.056+0000 7f885b7fe700 1 --2- 172.21.15.16:0/2112360480 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f885c06e570 0x7f885c1f6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:47.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.056+0000 7f8860a1b700 1 --2- 172.21.15.16:0/2112360480 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f885c06ae00 0x7f885c0c7990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:47.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.056+0000 7f885bfff700 1 -- 172.21.15.16:0/2112360480 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f885c06e570 msgr2=0x7f885c1f6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:47.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.056+0000 7f885bfff700 1 --2- 172.21.15.16:0/2112360480 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f885c06e570 0x7f885c1f6d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:47.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.056+0000 7f885bfff700 1 -- 172.21.15.16:0/2112360480 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f885c06ae00 msgr2=0x7f885c0c7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:47.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.056+0000 7f885bfff700 1 --2- 172.21.15.16:0/2112360480 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f885c06ae00 0x7f885c0c7990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:47.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.056+0000 7f885bfff700 1 -- 172.21.15.16:0/2112360480 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f884c009550 con 0x7f885c06a7e0 2023-10-16T13:48:47.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.056+0000 7f885bfff700 1 --2- 172.21.15.16:0/2112360480 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f885c06a7e0 0x7f885c1f6800 secure :-1 s=READY pgs=3095 cs=0 l=1 rev1=1 crypto rx=0x7f884c009c40 tx=0x7f884c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:47.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.057+0000 7f8858ff9700 1 -- 172.21.15.16:0/2112360480 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f884c004340 con 0x7f885c06a7e0 2023-10-16T13:48:47.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.057+0000 7f8858ff9700 1 -- 172.21.15.16:0/2112360480 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f884c01f070 con 0x7f885c06a7e0 2023-10-16T13:48:47.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.057+0000 7f886247e700 1 -- 172.21.15.16:0/2112360480 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f885c0b4df0 con 0x7f885c06a7e0 2023-10-16T13:48:47.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.057+0000 7f8858ff9700 1 -- 172.21.15.16:0/2112360480 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f884c0036a0 con 0x7f885c06a7e0 2023-10-16T13:48:47.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.057+0000 7f886247e700 1 -- 172.21.15.16:0/2112360480 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f885c0c74c0 con 0x7f885c06a7e0 2023-10-16T13:48:47.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.058+0000 7f886247e700 1 -- 172.21.15.16:0/2112360480 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f885c0b4970 con 0x7f885c06a7e0 2023-10-16T13:48:47.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.058+0000 7f8858ff9700 1 -- 172.21.15.16:0/2112360480 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 162) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f884c021030 con 0x7f885c06a7e0 2023-10-16T13:48:47.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.059+0000 7f8858ff9700 1 --2- 172.21.15.16:0/2112360480 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f884806b020 0x7f884806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:47.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.059+0000 7f885b7fe700 1 -- 172.21.15.16:0/2112360480 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f884806b020 msgr2=0x7f884806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2131389174 2023-10-16T13:48:47.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.059+0000 7f8858ff9700 1 -- 172.21.15.16:0/2112360480 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(266..266 src has 1..266) v4 ==== 6587+0+0 (secure 0 0 0) 0x7f884c08ae60 con 0x7f885c06a7e0 2023-10-16T13:48:47.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.059+0000 7f885b7fe700 1 --2- 172.21.15.16:0/2112360480 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f884806b020 0x7f884806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:48:47.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.063+0000 7f8858ff9700 1 -- 172.21.15.16:0/2112360480 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f885c0b4970 con 0x7f885c06a7e0 2023-10-16T13:48:47.118 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:48:47.116+0000 7f2fb0253200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T13:48:47.213 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:47.210+0000 7fd1d1dca200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T13:48:47.238 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:48:47.235+0000 7f2fb0253200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T13:48:47.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.260+0000 7f885b7fe700 1 -- 172.21.15.16:0/2112360480 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f884806b020 msgr2=0x7f884806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2131389174 2023-10-16T13:48:47.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.260+0000 7f885b7fe700 1 --2- 172.21.15.16:0/2112360480 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f884806b020 0x7f884806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:48:47.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.269+0000 7f886247e700 1 -- 172.21.15.16:0/2112360480 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f885c0b4970 con 0x7f885c06a7e0 2023-10-16T13:48:47.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.269+0000 7f8858ff9700 1 -- 172.21.15.16:0/2112360480 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v266) v1 ==== 100+0+11 (secure 0 0 0) 0x7f885c0b4970 con 0x7f885c06a7e0 2023-10-16T13:48:47.272 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-10-16T13:48:47.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.271+0000 7f886247e700 1 -- 172.21.15.16:0/2112360480 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f884806b020 msgr2=0x7f884806d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:48:47.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.271+0000 7f886247e700 1 --2- 172.21.15.16:0/2112360480 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f884806b020 0x7f884806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:47.274 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.271+0000 7f886247e700 1 -- 172.21.15.16:0/2112360480 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f885c06a7e0 msgr2=0x7f885c1f6800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:47.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.271+0000 7f886247e700 1 --2- 172.21.15.16:0/2112360480 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f885c06a7e0 0x7f885c1f6800 secure :-1 s=READY pgs=3095 cs=0 l=1 rev1=1 crypto rx=0x7f884c009c40 tx=0x7f884c004750 comp rx=0 tx=0).stop 2023-10-16T13:48:47.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.271+0000 7f886247e700 1 -- 172.21.15.16:0/2112360480 shutdown_connections 2023-10-16T13:48:47.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.271+0000 7f886247e700 1 --2- 172.21.15.16:0/2112360480 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f884806b020 0x7f884806d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:47.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.271+0000 7f886247e700 1 --2- 172.21.15.16:0/2112360480 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f885c06ae00 0x7f885c0c7990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:47.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.271+0000 7f886247e700 1 --2- 172.21.15.16:0/2112360480 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f885c06a7e0 0x7f885c1f6800 unknown :-1 s=CLOSED pgs=3095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:47.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.271+0000 7f886247e700 1 --2- 172.21.15.16:0/2112360480 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f885c06e570 0x7f885c1f6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:47.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.271+0000 7f886247e700 1 -- 172.21.15.16:0/2112360480 >> 172.21.15.16:0/2112360480 conn(0x7f885c0b1f40 msgr2=0x7f885c0cce00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:47.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.272+0000 7f886247e700 1 -- 172.21.15.16:0/2112360480 shutdown_connections 2023-10-16T13:48:47.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.272+0000 7f886247e700 1 -- 172.21.15.16:0/2112360480 wait complete. 2023-10-16T13:48:47.287 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:48:47.338 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:47.335+0000 7fd1d1dca200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T13:48:47.402 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:48:47.400+0000 7f2fb0253200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T13:48:47.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.411+0000 7f81ded6e700 1 Processor -- start 2023-10-16T13:48:47.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.411+0000 7f81ded6e700 1 -- start start 2023-10-16T13:48:47.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.412+0000 7f81ded6e700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f81d815b860 0x7f81d806a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:47.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.412+0000 7f81ded6e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f81d80b4490 0x7f81d806ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:47.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.412+0000 7f81ded6e700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81d80d11c0 con 0x7f81d80b2be0 2023-10-16T13:48:47.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.412+0000 7f81ded6e700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81d80d03d0 con 0x7f81d80b4490 2023-10-16T13:48:47.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.412+0000 7f81ded6e700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81d80cd360 con 0x7f81d815b860 2023-10-16T13:48:47.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.412+0000 7f81dd30b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f81d80b4490 0x7f81d806ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:47.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.412+0000 7f81dd30b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f81d80b4490 0x7f81d806ad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58446/0 (socket says 172.21.15.16:58446) 2023-10-16T13:48:47.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.412+0000 7f81dd30b700 1 -- 172.21.15.16:0/1632989664 learned_addr learned my addr 172.21.15.16:0/1632989664 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:47.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.412+0000 7f81dcb0a700 1 --2- 172.21.15.16:0/1632989664 >> v2:172.21.15.49:3300/0 conn(0x7f81d815b860 0x7f81d806a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:47.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.413+0000 7f81dd30b700 1 -- 172.21.15.16:0/1632989664 >> v2:172.21.15.49:3300/0 conn(0x7f81d815b860 msgr2=0x7f81d806a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:47.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.413+0000 7f81dd30b700 1 --2- 172.21.15.16:0/1632989664 >> v2:172.21.15.49:3300/0 conn(0x7f81d815b860 0x7f81d806a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:47.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.413+0000 7f81dd30b700 1 -- 172.21.15.16:0/1632989664 >> v1:172.21.15.49:6789/0 conn(0x7f81d80b2be0 legacy=0x7f81d80719e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:47.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.413+0000 7f81dd30b700 1 -- 172.21.15.16:0/1632989664 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81d80d05b0 con 0x7f81d80b4490 2023-10-16T13:48:47.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.413+0000 7f81dd30b700 1 --2- 172.21.15.16:0/1632989664 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81d80b4490 0x7f81d806ad20 secure :-1 s=READY pgs=3096 cs=0 l=1 rev1=1 crypto rx=0x7f81d8072e20 tx=0x7f81d000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2490051ac43eb1f6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:47.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.413+0000 7f81d77fe700 1 -- 172.21.15.16:0/1632989664 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81d000e070 con 0x7f81d80b4490 2023-10-16T13:48:47.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.413+0000 7f81d77fe700 1 -- 172.21.15.16:0/1632989664 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81d000b910 con 0x7f81d80b4490 2023-10-16T13:48:47.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.413+0000 7f81d77fe700 1 -- 172.21.15.16:0/1632989664 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81d000bcc0 con 0x7f81d80b4490 2023-10-16T13:48:47.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.414+0000 7f81ded6e700 1 -- 172.21.15.16:0/1632989664 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81d80b4490 msgr2=0x7f81d806ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:47.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.414+0000 7f81ded6e700 1 --2- 172.21.15.16:0/1632989664 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81d80b4490 0x7f81d806ad20 secure :-1 s=READY pgs=3096 cs=0 l=1 rev1=1 crypto rx=0x7f81d8072e20 tx=0x7f81d000b330 comp rx=0 tx=0).stop 2023-10-16T13:48:47.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.414+0000 7f81ded6e700 1 -- 172.21.15.16:0/1632989664 shutdown_connections 2023-10-16T13:48:47.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.414+0000 7f81ded6e700 1 --2- 172.21.15.16:0/1632989664 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81d80b4490 0x7f81d806ad20 unknown :-1 s=CLOSED pgs=3096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:47.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.414+0000 7f81ded6e700 1 --2- 172.21.15.16:0/1632989664 >> v2:172.21.15.49:3300/0 conn(0x7f81d815b860 0x7f81d806a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:47.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.414+0000 7f81ded6e700 1 -- 172.21.15.16:0/1632989664 >> 172.21.15.16:0/1632989664 conn(0x7f81d80b1fd0 msgr2=0x7f81d80b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:47.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.414+0000 7f81ded6e700 1 -- 172.21.15.16:0/1632989664 shutdown_connections 2023-10-16T13:48:47.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.414+0000 7f81ded6e700 1 -- 172.21.15.16:0/1632989664 wait complete. 2023-10-16T13:48:47.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.416+0000 7f81ded6e700 1 Processor -- start 2023-10-16T13:48:47.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.416+0000 7f81ded6e700 1 -- start start 2023-10-16T13:48:47.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.417+0000 7f81ded6e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81d80b2be0 0x7f81d81559a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:47.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.417+0000 7f81ded6e700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f81d80b4490 0x7f81d80ce010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:47.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.417+0000 7f81ded6e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f81d815b860 0x7f81d80c5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:47.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.417+0000 7f81ded6e700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81d80acce0 con 0x7f81d80b2be0 2023-10-16T13:48:47.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.417+0000 7f81ded6e700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81d80ce9f0 con 0x7f81d80b4490 2023-10-16T13:48:47.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.417+0000 7f81ded6e700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81d80cecd0 con 0x7f81d815b860 2023-10-16T13:48:47.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.417+0000 7f81dcb0a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81d80b2be0 0x7f81d81559a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:47.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.417+0000 7f81dcb0a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81d80b2be0 0x7f81d81559a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58448/0 (socket says 172.21.15.16:58448) 2023-10-16T13:48:47.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.417+0000 7f81dcb0a700 1 -- 172.21.15.16:0/325272561 learned_addr learned my addr 172.21.15.16:0/325272561 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:47.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.417+0000 7f81d7fff700 1 --2- 172.21.15.16:0/325272561 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f81d80b4490 0x7f81d80ce010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:47.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.417+0000 7f81dd30b700 1 --2- 172.21.15.16:0/325272561 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f81d815b860 0x7f81d80c5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:47.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.418+0000 7f81dcb0a700 1 -- 172.21.15.16:0/325272561 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f81d815b860 msgr2=0x7f81d80c5390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:47.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.418+0000 7f81dcb0a700 1 --2- 172.21.15.16:0/325272561 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f81d815b860 0x7f81d80c5390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:47.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.418+0000 7f81dcb0a700 1 -- 172.21.15.16:0/325272561 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f81d80b4490 msgr2=0x7f81d80ce010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:47.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.418+0000 7f81dcb0a700 1 --2- 172.21.15.16:0/325272561 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f81d80b4490 0x7f81d80ce010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:47.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.418+0000 7f81dcb0a700 1 -- 172.21.15.16:0/325272561 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81d0009b30 con 0x7f81d80b2be0 2023-10-16T13:48:47.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.418+0000 7f81dcb0a700 1 --2- 172.21.15.16:0/325272561 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81d80b2be0 0x7f81d81559a0 secure :-1 s=READY pgs=3097 cs=0 l=1 rev1=1 crypto rx=0x7f81cc00e9b0 tx=0x7f81cc00ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:47.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.418+0000 7f81d57fa700 1 -- 172.21.15.16:0/325272561 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81cc009710 con 0x7f81d80b2be0 2023-10-16T13:48:47.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.418+0000 7f81d57fa700 1 -- 172.21.15.16:0/325272561 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81cc00ee50 con 0x7f81d80b2be0 2023-10-16T13:48:47.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.418+0000 7f81ded6e700 1 -- 172.21.15.16:0/325272561 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81d80cefb0 con 0x7f81d80b2be0 2023-10-16T13:48:47.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.418+0000 7f81d57fa700 1 -- 172.21.15.16:0/325272561 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81cc00ca50 con 0x7f81d80b2be0 2023-10-16T13:48:47.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.418+0000 7f81ded6e700 1 -- 172.21.15.16:0/325272561 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81d8155ee0 con 0x7f81d80b2be0 2023-10-16T13:48:47.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.419+0000 7f81ded6e700 1 -- 172.21.15.16:0/325272561 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81d80acfc0 con 0x7f81d80b2be0 2023-10-16T13:48:47.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.420+0000 7f81d57fa700 1 -- 172.21.15.16:0/325272561 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 162) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f81cc00cbb0 con 0x7f81d80b2be0 2023-10-16T13:48:47.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.420+0000 7f81d57fa700 1 --2- 172.21.15.16:0/325272561 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f81bc06b020 0x7f81bc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:47.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.420+0000 7f81d7fff700 1 -- 172.21.15.16:0/325272561 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f81bc06b020 msgr2=0x7f81bc06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2131389174 2023-10-16T13:48:47.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.421+0000 7f81d57fa700 1 -- 172.21.15.16:0/325272561 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(266..266 src has 1..266) v4 ==== 6587+0+0 (secure 0 0 0) 0x7f81cc089a90 con 0x7f81d80b2be0 2023-10-16T13:48:47.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.421+0000 7f81d7fff700 1 --2- 172.21.15.16:0/325272561 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f81bc06b020 0x7f81bc06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:48:47.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.425+0000 7f81d57fa700 1 -- 172.21.15.16:0/325272561 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f81cc0546b0 con 0x7f81d80b2be0 2023-10-16T13:48:47.511 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:47.509+0000 7fd1d1dca200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T13:48:47.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.621+0000 7f81d7fff700 1 -- 172.21.15.16:0/325272561 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f81bc06b020 msgr2=0x7f81bc06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2131389174 2023-10-16T13:48:47.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.621+0000 7f81d7fff700 1 --2- 172.21.15.16:0/325272561 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f81bc06b020 0x7f81bc06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:48:47.676 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:48:47.673+0000 7f2fb0253200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T13:48:47.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.675+0000 7f81ded6e700 1 -- 172.21.15.16:0/325272561 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f81d804ebb0 con 0x7f81d80b2be0 2023-10-16T13:48:47.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.676+0000 7f81d57fa700 1 -- 172.21.15.16:0/325272561 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 146 v146) v1 ==== 95+0+4926 (secure 0 0 0) 0x7f81cc00ce60 con 0x7f81d80b2be0 2023-10-16T13:48:47.679 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:48:47.680 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":146,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10112,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/712583564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":712583564},{"type":"v1","addr":"172.21.15.16:6837","nonce":712583564}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":141},{"gid":11836,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/3551114031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":3551114031},{"type":"v1","addr":"172.21.15.49:6834","nonce":3551114031}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":141},{"gid":13050,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/2135643757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2135643757},{"type":"v1","addr":"172.21.15.16:6836","nonce":2135643757}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":140}],"filesystems":[{"mdsmap":{"epoch":146,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:48:24.832614+0000","modified":"2023-10-16T13:48:27.858697+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":13068},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13068":{"gid":13068,"name":"d","rank":0,"incarnation":142,"state":"up:active","state_seq":3,"addr":"172.21.15.49:6835/1985453659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":1985453659},{"type":"v1","addr":"172.21.15.49:6835","nonce":1985453659}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[36],"metadata_pool":35,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":17}]} 2023-10-16T13:48:47.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.678+0000 7f81ded6e700 1 -- 172.21.15.16:0/325272561 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f81bc06b020 msgr2=0x7f81bc06d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:48:47.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.678+0000 7f81ded6e700 1 --2- 172.21.15.16:0/325272561 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f81bc06b020 0x7f81bc06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:47.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.678+0000 7f81ded6e700 1 -- 172.21.15.16:0/325272561 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81d80b2be0 msgr2=0x7f81d81559a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:47.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.678+0000 7f81ded6e700 1 --2- 172.21.15.16:0/325272561 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81d80b2be0 0x7f81d81559a0 secure :-1 s=READY pgs=3097 cs=0 l=1 rev1=1 crypto rx=0x7f81cc00e9b0 tx=0x7f81cc00ecc0 comp rx=0 tx=0).stop 2023-10-16T13:48:47.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.678+0000 7f81ded6e700 1 -- 172.21.15.16:0/325272561 shutdown_connections 2023-10-16T13:48:47.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.678+0000 7f81ded6e700 1 --2- 172.21.15.16:0/325272561 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f81bc06b020 0x7f81bc06d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:47.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.678+0000 7f81ded6e700 1 --2- 172.21.15.16:0/325272561 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f81d815b860 0x7f81d80c5390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:47.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.678+0000 7f81ded6e700 1 --2- 172.21.15.16:0/325272561 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81d80b2be0 0x7f81d81559a0 unknown :-1 s=CLOSED pgs=3097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:47.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.678+0000 7f81ded6e700 1 --2- 172.21.15.16:0/325272561 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f81d80b4490 0x7f81d80ce010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:47.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.678+0000 7f81ded6e700 1 -- 172.21.15.16:0/325272561 >> 172.21.15.16:0/325272561 conn(0x7f81d80b1fd0 msgr2=0x7f81d80affd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:47.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.678+0000 7f81ded6e700 1 -- 172.21.15.16:0/325272561 shutdown_connections 2023-10-16T13:48:47.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.678+0000 7f81ded6e700 1 -- 172.21.15.16:0/325272561 wait complete. 2023-10-16T13:48:47.684 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 146 2023-10-16T13:48:47.694 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:48:47.796 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:47.793+0000 7fd1d1dca200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T13:48:47.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.817+0000 7f391b427700 1 Processor -- start 2023-10-16T13:48:47.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.817+0000 7f391b427700 1 -- start start 2023-10-16T13:48:47.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.817+0000 7f391b427700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f391406dcd0 0x7f39141578c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:47.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.818+0000 7f391b427700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f39140cb8a0 0x7f3914157e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:47.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.818+0000 7f391b427700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f391406e0b0 0x7f391415aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:47.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.818+0000 7f391b427700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3914158340 con 0x7f391406e0b0 2023-10-16T13:48:47.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.818+0000 7f391b427700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f391415b7d0 con 0x7f391406dcd0 2023-10-16T13:48:47.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.818+0000 7f391b427700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39140cdfc0 con 0x7f39140cb8a0 2023-10-16T13:48:47.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.818+0000 7f39199c4700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f391406e0b0 0x7f391415aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:47.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.818+0000 7f39199c4700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f391406e0b0 0x7f391415aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58462/0 (socket says 172.21.15.16:58462) 2023-10-16T13:48:47.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.818+0000 7f39199c4700 1 -- 172.21.15.16:0/2745631569 learned_addr learned my addr 172.21.15.16:0/2745631569 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:47.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.818+0000 7f39189c2700 1 --2- 172.21.15.16:0/2745631569 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f39140cb8a0 0x7f3914157e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:47.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.818+0000 7f39191c3700 1 --2- 172.21.15.16:0/2745631569 >> v2:172.21.15.49:3300/0 conn(0x7f391406dcd0 0x7f39141578c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:47.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.818+0000 7f39199c4700 1 -- 172.21.15.16:0/2745631569 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f39140cb8a0 msgr2=0x7f3914157e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:47.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.818+0000 7f39199c4700 1 --2- 172.21.15.16:0/2745631569 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f39140cb8a0 0x7f3914157e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:47.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.818+0000 7f39199c4700 1 -- 172.21.15.16:0/2745631569 >> v2:172.21.15.49:3300/0 conn(0x7f391406dcd0 msgr2=0x7f39141578c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:47.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.818+0000 7f39199c4700 1 --2- 172.21.15.16:0/2745631569 >> v2:172.21.15.49:3300/0 conn(0x7f391406dcd0 0x7f39141578c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:47.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.818+0000 7f39199c4700 1 -- 172.21.15.16:0/2745631569 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39140ce2a0 con 0x7f391406e0b0 2023-10-16T13:48:47.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.818+0000 7f39199c4700 1 --2- 172.21.15.16:0/2745631569 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f391406e0b0 0x7f391415aa60 secure :-1 s=READY pgs=3098 cs=0 l=1 rev1=1 crypto rx=0x7f390c00d150 tx=0x7f390c00d460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=623738dca610f23a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:47.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.818+0000 7f390bfff700 1 -- 172.21.15.16:0/2745631569 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f390c010070 con 0x7f391406e0b0 2023-10-16T13:48:47.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.818+0000 7f390bfff700 1 -- 172.21.15.16:0/2745631569 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f390c00dcd0 con 0x7f391406e0b0 2023-10-16T13:48:47.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.819+0000 7f390bfff700 1 -- 172.21.15.16:0/2745631569 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f390c00b310 con 0x7f391406e0b0 2023-10-16T13:48:47.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.819+0000 7f391b427700 1 -- 172.21.15.16:0/2745631569 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f391406e0b0 msgr2=0x7f391415aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:47.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.819+0000 7f391b427700 1 --2- 172.21.15.16:0/2745631569 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f391406e0b0 0x7f391415aa60 secure :-1 s=READY pgs=3098 cs=0 l=1 rev1=1 crypto rx=0x7f390c00d150 tx=0x7f390c00d460 comp rx=0 tx=0).stop 2023-10-16T13:48:47.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.819+0000 7f391b427700 1 -- 172.21.15.16:0/2745631569 shutdown_connections 2023-10-16T13:48:47.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.819+0000 7f391b427700 1 --2- 172.21.15.16:0/2745631569 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f391406e0b0 0x7f391415aa60 unknown :-1 s=CLOSED pgs=3098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:47.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.819+0000 7f391b427700 1 --2- 172.21.15.16:0/2745631569 >> v2:172.21.15.49:3300/0 conn(0x7f391406dcd0 0x7f39141578c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:47.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.819+0000 7f391b427700 1 --2- 172.21.15.16:0/2745631569 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f39140cb8a0 0x7f3914157e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:47.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.819+0000 7f391b427700 1 -- 172.21.15.16:0/2745631569 >> 172.21.15.16:0/2745631569 conn(0x7f39140b1f40 msgr2=0x7f39140b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:47.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.819+0000 7f391b427700 1 -- 172.21.15.16:0/2745631569 shutdown_connections 2023-10-16T13:48:47.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.819+0000 7f391b427700 1 -- 172.21.15.16:0/2745631569 wait complete. 2023-10-16T13:48:47.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.820+0000 7f391b427700 1 Processor -- start 2023-10-16T13:48:47.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.820+0000 7f391b427700 1 -- start start 2023-10-16T13:48:47.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.821+0000 7f391b427700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f391406dcd0 0x7f39141527b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:47.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.821+0000 7f391b427700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f391406e0b0 0x7f39140ad2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:47.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.821+0000 7f391b427700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f39140cb8a0 0x7f39140ad800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:47.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.821+0000 7f391b427700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3914159120 con 0x7f391406dcd0 2023-10-16T13:48:47.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.821+0000 7f391b427700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39141f7450 con 0x7f391406e0b0 2023-10-16T13:48:47.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.821+0000 7f391b427700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3914158530 con 0x7f39140cb8a0 2023-10-16T13:48:47.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.821+0000 7f39191c3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f391406dcd0 0x7f39141527b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:47.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.821+0000 7f39191c3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f391406dcd0 0x7f39141527b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58464/0 (socket says 172.21.15.16:58464) 2023-10-16T13:48:47.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.821+0000 7f39191c3700 1 -- 172.21.15.16:0/2123903787 learned_addr learned my addr 172.21.15.16:0/2123903787 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:47.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.821+0000 7f39199c4700 1 --2- 172.21.15.16:0/2123903787 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f39140cb8a0 0x7f39140ad800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:47.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.821+0000 7f39189c2700 1 --2- 172.21.15.16:0/2123903787 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f391406e0b0 0x7f39140ad2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:47.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.822+0000 7f39191c3700 1 -- 172.21.15.16:0/2123903787 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f39140cb8a0 msgr2=0x7f39140ad800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:47.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.822+0000 7f39191c3700 1 --2- 172.21.15.16:0/2123903787 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f39140cb8a0 0x7f39140ad800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:47.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.822+0000 7f39191c3700 1 -- 172.21.15.16:0/2123903787 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f391406e0b0 msgr2=0x7f39140ad2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:47.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.822+0000 7f39191c3700 1 --2- 172.21.15.16:0/2123903787 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f391406e0b0 0x7f39140ad2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:47.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.822+0000 7f39191c3700 1 -- 172.21.15.16:0/2123903787 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f390c00bd40 con 0x7f391406dcd0 2023-10-16T13:48:47.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.822+0000 7f39191c3700 1 --2- 172.21.15.16:0/2123903787 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f391406dcd0 0x7f39141527b0 secure :-1 s=READY pgs=3099 cs=0 l=1 rev1=1 crypto rx=0x7f390400e9b0 tx=0x7f390400ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:47.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.822+0000 7f3909ffb700 1 -- 172.21.15.16:0/2123903787 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3904009710 con 0x7f391406dcd0 2023-10-16T13:48:47.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.822+0000 7f3909ffb700 1 -- 172.21.15.16:0/2123903787 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f390400ee50 con 0x7f391406dcd0 2023-10-16T13:48:47.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.822+0000 7f391b427700 1 -- 172.21.15.16:0/2123903787 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39141587f0 con 0x7f391406dcd0 2023-10-16T13:48:47.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.822+0000 7f3909ffb700 1 -- 172.21.15.16:0/2123903787 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f390400ca50 con 0x7f391406dcd0 2023-10-16T13:48:47.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.822+0000 7f391b427700 1 -- 172.21.15.16:0/2123903787 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3914158db0 con 0x7f391406dcd0 2023-10-16T13:48:47.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.823+0000 7f391b427700 1 -- 172.21.15.16:0/2123903787 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f391404ec20 con 0x7f391406dcd0 2023-10-16T13:48:47.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.824+0000 7f3909ffb700 1 -- 172.21.15.16:0/2123903787 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 162) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f390400cbb0 con 0x7f391406dcd0 2023-10-16T13:48:47.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.824+0000 7f3909ffb700 1 --2- 172.21.15.16:0/2123903787 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f38f806b050 0x7f38f806d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:47.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.824+0000 7f39189c2700 1 -- 172.21.15.16:0/2123903787 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f38f806b050 msgr2=0x7f38f806d510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2131389174 2023-10-16T13:48:47.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.824+0000 7f3909ffb700 1 -- 172.21.15.16:0/2123903787 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(266..266 src has 1..266) v4 ==== 6587+0+0 (secure 0 0 0) 0x7f3904089a10 con 0x7f391406dcd0 2023-10-16T13:48:47.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.824+0000 7f39189c2700 1 --2- 172.21.15.16:0/2123903787 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f38f806b050 0x7f38f806d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:48:47.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:47.833+0000 7f3909ffb700 1 -- 172.21.15.16:0/2123903787 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3904054640 con 0x7f391406dcd0 2023-10-16T13:48:47.845 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:48:47.842+0000 7f2fb0253200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T13:48:47.977 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:47.974+0000 7fd1d1dca200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T13:48:48.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.025+0000 7f39189c2700 1 -- 172.21.15.16:0/2123903787 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f38f806b050 msgr2=0x7f38f806d510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2131389174 2023-10-16T13:48:48.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.025+0000 7f39189c2700 1 --2- 172.21.15.16:0/2123903787 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f38f806b050 0x7f38f806d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:48:48.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.039+0000 7f391b427700 1 -- 172.21.15.16:0/2123903787 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f39140ca920 con 0x7f391406dcd0 2023-10-16T13:48:48.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.041+0000 7f3909ffb700 1 -- 172.21.15.16:0/2123903787 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v266) v1 ==== 74+0+17440 (secure 0 0 0) 0x7f3904021090 con 0x7f391406dcd0 2023-10-16T13:48:48.045 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:48:48.045 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":266,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:48:39.972754+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":36,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_metadata","create_time":"2023-10-16T13:48:23.348668+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"264","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_data","create_time":"2023-10-16T13:48:23.799968+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"265","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/3313712117":"2023-10-17T13:48:39.972713+0000","172.21.15.16:0/2202829273":"2023-10-17T13:48:39.972713+0000","172.21.15.16:0/2777724108":"2023-10-17T13:48:39.972713+0000","172.21.15.16:0/2088077503":"2023-10-17T13:48:39.972713+0000","172.21.15.16:0/4096007345":"2023-10-17T13:48:39.972713+0000","172.21.15.16:0/3000870029":"2023-10-17T13:48:39.972713+0000","172.21.15.16:0/3923653515":"2023-10-17T13:48:39.972713+0000","172.21.15.16:0/56667078":"2023-10-17T13:48:39.972713+0000","172.21.15.16:6833/1275231846":"2023-10-17T13:48:39.972713+0000","172.21.15.16:6832/1275231846":"2023-10-17T13:48:39.972713+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:48:48.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.043+0000 7f391b427700 1 -- 172.21.15.16:0/2123903787 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f38f806b050 msgr2=0x7f38f806d510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:48:48.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.043+0000 7f391b427700 1 --2- 172.21.15.16:0/2123903787 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f38f806b050 0x7f38f806d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:48.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.043+0000 7f391b427700 1 -- 172.21.15.16:0/2123903787 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f391406dcd0 msgr2=0x7f39141527b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:48.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.043+0000 7f391b427700 1 --2- 172.21.15.16:0/2123903787 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f391406dcd0 0x7f39141527b0 secure :-1 s=READY pgs=3099 cs=0 l=1 rev1=1 crypto rx=0x7f390400e9b0 tx=0x7f390400ecc0 comp rx=0 tx=0).stop 2023-10-16T13:48:48.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.043+0000 7f391b427700 1 -- 172.21.15.16:0/2123903787 shutdown_connections 2023-10-16T13:48:48.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.043+0000 7f391b427700 1 --2- 172.21.15.16:0/2123903787 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f38f806b050 0x7f38f806d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:48.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.043+0000 7f391b427700 1 --2- 172.21.15.16:0/2123903787 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f39140cb8a0 0x7f39140ad800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:48.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.043+0000 7f391b427700 1 --2- 172.21.15.16:0/2123903787 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f391406dcd0 0x7f39141527b0 unknown :-1 s=CLOSED pgs=3099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:48.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.043+0000 7f391b427700 1 --2- 172.21.15.16:0/2123903787 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f391406e0b0 0x7f39140ad2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:48.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.044+0000 7f391b427700 1 -- 172.21.15.16:0/2123903787 >> 172.21.15.16:0/2123903787 conn(0x7f39140b1f40 msgr2=0x7f39140cf120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:48.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.044+0000 7f391b427700 1 -- 172.21.15.16:0/2123903787 shutdown_connections 2023-10-16T13:48:48.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.044+0000 7f391b427700 1 -- 172.21.15.16:0/2123903787 wait complete. 2023-10-16T13:48:48.060 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-16T13:48:48.060 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:48:48.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.183+0000 7f738a102700 1 Processor -- start 2023-10-16T13:48:48.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.183+0000 7f738a102700 1 -- start start 2023-10-16T13:48:48.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.184+0000 7f738a102700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f738415b280 0x7f73841613e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:48.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.184+0000 7f738a102700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f73840cc8a0 0x7f738415e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:48.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.184+0000 7f738a102700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f738406a7e0 0x7f738406ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:48.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.184+0000 7f738a102700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f738406b170 con 0x7f73840cc8a0 2023-10-16T13:48:48.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.184+0000 7f738a102700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7384161c30 con 0x7f738406a7e0 2023-10-16T13:48:48.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.184+0000 7f738a102700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7384161f10 con 0x7f738415b280 2023-10-16T13:48:48.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.184+0000 7f7382ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f73840cc8a0 0x7f738415e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:48.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.185+0000 7f7382ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f73840cc8a0 0x7f738415e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58476/0 (socket says 172.21.15.16:58476) 2023-10-16T13:48:48.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.185+0000 7f7382ffd700 1 -- 172.21.15.16:0/2160583811 learned_addr learned my addr 172.21.15.16:0/2160583811 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:48.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.185+0000 7f7383fff700 1 --2- 172.21.15.16:0/2160583811 >> v2:172.21.15.49:3300/0 conn(0x7f738406a7e0 0x7f738406ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:48.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.185+0000 7f73837fe700 1 --2- 172.21.15.16:0/2160583811 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f738415b280 0x7f73841613e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:48.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.185+0000 7f7382ffd700 1 -- 172.21.15.16:0/2160583811 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f738415b280 msgr2=0x7f73841613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:48.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.185+0000 7f7382ffd700 1 --2- 172.21.15.16:0/2160583811 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f738415b280 0x7f73841613e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:48.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.185+0000 7f7382ffd700 1 -- 172.21.15.16:0/2160583811 >> v2:172.21.15.49:3300/0 conn(0x7f738406a7e0 msgr2=0x7f738406ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:48.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.185+0000 7f7382ffd700 1 --2- 172.21.15.16:0/2160583811 >> v2:172.21.15.49:3300/0 conn(0x7f738406a7e0 0x7f738406ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:48.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.185+0000 7f7382ffd700 1 -- 172.21.15.16:0/2160583811 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73841621f0 con 0x7f73840cc8a0 2023-10-16T13:48:48.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.185+0000 7f7382ffd700 1 --2- 172.21.15.16:0/2160583811 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73840cc8a0 0x7f738415e700 secure :-1 s=READY pgs=3100 cs=0 l=1 rev1=1 crypto rx=0x7f7384071970 tx=0x7f736c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d30b4912c7ab4176 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:48.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.185+0000 7f73827fc700 1 -- 172.21.15.16:0/2160583811 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f736c011830 con 0x7f73840cc8a0 2023-10-16T13:48:48.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.185+0000 7f73827fc700 1 -- 172.21.15.16:0/2160583811 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f736c011990 con 0x7f73840cc8a0 2023-10-16T13:48:48.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.185+0000 7f73827fc700 1 -- 172.21.15.16:0/2160583811 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f736c011d20 con 0x7f73840cc8a0 2023-10-16T13:48:48.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.186+0000 7f738a102700 1 -- 172.21.15.16:0/2160583811 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73840cc8a0 msgr2=0x7f738415e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:48.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.186+0000 7f738a102700 1 --2- 172.21.15.16:0/2160583811 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73840cc8a0 0x7f738415e700 secure :-1 s=READY pgs=3100 cs=0 l=1 rev1=1 crypto rx=0x7f7384071970 tx=0x7f736c009bd0 comp rx=0 tx=0).stop 2023-10-16T13:48:48.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.186+0000 7f738a102700 1 -- 172.21.15.16:0/2160583811 shutdown_connections 2023-10-16T13:48:48.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.186+0000 7f738a102700 1 --2- 172.21.15.16:0/2160583811 >> v2:172.21.15.49:3300/0 conn(0x7f738406a7e0 0x7f738406ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:48.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.186+0000 7f738a102700 1 --2- 172.21.15.16:0/2160583811 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f738415b280 0x7f73841613e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:48.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.186+0000 7f738a102700 1 --2- 172.21.15.16:0/2160583811 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73840cc8a0 0x7f738415e700 unknown :-1 s=CLOSED pgs=3100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:48.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.186+0000 7f738a102700 1 -- 172.21.15.16:0/2160583811 >> 172.21.15.16:0/2160583811 conn(0x7f73840b1fd0 msgr2=0x7f73840b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:48.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.186+0000 7f738a102700 1 -- 172.21.15.16:0/2160583811 shutdown_connections 2023-10-16T13:48:48.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.186+0000 7f738a102700 1 -- 172.21.15.16:0/2160583811 wait complete. 2023-10-16T13:48:48.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.187+0000 7f738a102700 1 Processor -- start 2023-10-16T13:48:48.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.187+0000 7f738a102700 1 -- start start 2023-10-16T13:48:48.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.188+0000 7f738a102700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f738406a7e0 0x7f73840c75a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:48.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.188+0000 7f738a102700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73840cc8a0 0x7f73840c7db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:48.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.188+0000 7f738a102700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f738415b280 0x7f7384159840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:48.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.188+0000 7f738a102700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73841600e0 con 0x7f73840cc8a0 2023-10-16T13:48:48.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.188+0000 7f738a102700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73840c7ae0 con 0x7f738406a7e0 2023-10-16T13:48:48.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.188+0000 7f738a102700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73840c8520 con 0x7f738415b280 2023-10-16T13:48:48.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.188+0000 7f7382ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73840cc8a0 0x7f73840c7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:48.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.188+0000 7f7382ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73840cc8a0 0x7f73840c7db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58482/0 (socket says 172.21.15.16:58482) 2023-10-16T13:48:48.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.188+0000 7f73837fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f738406a7e0 0x7f73840c75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:48.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.188+0000 7f7382ffd700 1 -- 172.21.15.16:0/657018213 learned_addr learned my addr 172.21.15.16:0/657018213 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:48.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.188+0000 7f7383fff700 1 --2- 172.21.15.16:0/657018213 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f738415b280 0x7f7384159840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:48.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.189+0000 7f7382ffd700 1 -- 172.21.15.16:0/657018213 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f738415b280 msgr2=0x7f7384159840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:48.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.189+0000 7f7382ffd700 1 --2- 172.21.15.16:0/657018213 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f738415b280 0x7f7384159840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:48.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.189+0000 7f7382ffd700 1 -- 172.21.15.16:0/657018213 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f738406a7e0 msgr2=0x7f73840c75a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:48.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.189+0000 7f7382ffd700 1 --2- 172.21.15.16:0/657018213 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f738406a7e0 0x7f73840c75a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:48.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.189+0000 7f7382ffd700 1 -- 172.21.15.16:0/657018213 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f736c009550 con 0x7f73840cc8a0 2023-10-16T13:48:48.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.189+0000 7f7382ffd700 1 --2- 172.21.15.16:0/657018213 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73840cc8a0 0x7f73840c7db0 secure :-1 s=READY pgs=3101 cs=0 l=1 rev1=1 crypto rx=0x7f736c009c10 tx=0x7f736c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:48.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.189+0000 7f737bfff700 1 -- 172.21.15.16:0/657018213 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f736c004240 con 0x7f73840cc8a0 2023-10-16T13:48:48.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.189+0000 7f737bfff700 1 -- 172.21.15.16:0/657018213 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f736c01f070 con 0x7f73840cc8a0 2023-10-16T13:48:48.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.189+0000 7f738a102700 1 -- 172.21.15.16:0/657018213 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73840c82f0 con 0x7f73840cc8a0 2023-10-16T13:48:48.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.189+0000 7f737bfff700 1 -- 172.21.15.16:0/657018213 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f736c0045e0 con 0x7f73840cc8a0 2023-10-16T13:48:48.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.189+0000 7f738a102700 1 -- 172.21.15.16:0/657018213 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f738415f740 con 0x7f73840cc8a0 2023-10-16T13:48:48.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.190+0000 7f738a102700 1 -- 172.21.15.16:0/657018213 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7384160220 con 0x7f73840cc8a0 2023-10-16T13:48:48.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.191+0000 7f737bfff700 1 -- 172.21.15.16:0/657018213 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 162) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f736c021030 con 0x7f73840cc8a0 2023-10-16T13:48:48.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.191+0000 7f737bfff700 1 --2- 172.21.15.16:0/657018213 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f736406b020 0x7f736406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:48.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.191+0000 7f73837fe700 1 -- 172.21.15.16:0/657018213 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f736406b020 msgr2=0x7f736406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2131389174 2023-10-16T13:48:48.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.191+0000 7f737bfff700 1 -- 172.21.15.16:0/657018213 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(266..266 src has 1..266) v4 ==== 6587+0+0 (secure 0 0 0) 0x7f736c08ae60 con 0x7f73840cc8a0 2023-10-16T13:48:48.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.191+0000 7f73837fe700 1 --2- 172.21.15.16:0/657018213 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f736406b020 0x7f736406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:48:48.204 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.195+0000 7f737bfff700 1 -- 172.21.15.16:0/657018213 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f736c0b7fe0 con 0x7f73840cc8a0 2023-10-16T13:48:48.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.392+0000 7f73837fe700 1 -- 172.21.15.16:0/657018213 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f736406b020 msgr2=0x7f736406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/2131389174 2023-10-16T13:48:48.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.392+0000 7f73837fe700 1 --2- 172.21.15.16:0/657018213 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f736406b020 0x7f736406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:48:48.407 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:48:48.405+0000 7f2fb0253200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T13:48:48.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.451+0000 7f738a102700 1 -- 172.21.15.16:0/657018213 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7384160220 con 0x7f73840cc8a0 2023-10-16T13:48:48.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.451+0000 7f737bfff700 1 -- 172.21.15.16:0/657018213 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 146 v146) v1 ==== 95+0+4926 (secure 0 0 0) 0x7f7384160220 con 0x7f73840cc8a0 2023-10-16T13:48:48.455 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:48:48.455 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":146,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10112,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/712583564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":712583564},{"type":"v1","addr":"172.21.15.16:6837","nonce":712583564}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":141},{"gid":11836,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/3551114031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":3551114031},{"type":"v1","addr":"172.21.15.49:6834","nonce":3551114031}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":141},{"gid":13050,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/2135643757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2135643757},{"type":"v1","addr":"172.21.15.16:6836","nonce":2135643757}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":140}],"filesystems":[{"mdsmap":{"epoch":146,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:48:24.832614+0000","modified":"2023-10-16T13:48:27.858697+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":13068},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13068":{"gid":13068,"name":"d","rank":0,"incarnation":142,"state":"up:active","state_seq":3,"addr":"172.21.15.49:6835/1985453659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":1985453659},{"type":"v1","addr":"172.21.15.49:6835","nonce":1985453659}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[36],"metadata_pool":35,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":17}]} 2023-10-16T13:48:48.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.453+0000 7f738a102700 1 -- 172.21.15.16:0/657018213 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f736406b020 msgr2=0x7f736406d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:48:48.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.453+0000 7f738a102700 1 --2- 172.21.15.16:0/657018213 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f736406b020 0x7f736406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:48.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.453+0000 7f738a102700 1 -- 172.21.15.16:0/657018213 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73840cc8a0 msgr2=0x7f73840c7db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:48.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.453+0000 7f738a102700 1 --2- 172.21.15.16:0/657018213 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73840cc8a0 0x7f73840c7db0 secure :-1 s=READY pgs=3101 cs=0 l=1 rev1=1 crypto rx=0x7f736c009c10 tx=0x7f736c004750 comp rx=0 tx=0).stop 2023-10-16T13:48:48.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.453+0000 7f738a102700 1 -- 172.21.15.16:0/657018213 shutdown_connections 2023-10-16T13:48:48.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.453+0000 7f738a102700 1 --2- 172.21.15.16:0/657018213 >> [v2:172.21.15.16:6832/2131389174,v1:172.21.15.16:6833/2131389174] conn(0x7f736406b020 0x7f736406d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:48.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.453+0000 7f738a102700 1 --2- 172.21.15.16:0/657018213 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f738415b280 0x7f7384159840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:48.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.453+0000 7f738a102700 1 --2- 172.21.15.16:0/657018213 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f738406a7e0 0x7f73840c75a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:48.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.453+0000 7f738a102700 1 --2- 172.21.15.16:0/657018213 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73840cc8a0 0x7f73840c7db0 unknown :-1 s=CLOSED pgs=3101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:48.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.453+0000 7f738a102700 1 -- 172.21.15.16:0/657018213 >> 172.21.15.16:0/657018213 conn(0x7f73840b1fd0 msgr2=0x7f73840c3730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:48.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.453+0000 7f738a102700 1 -- 172.21.15.16:0/657018213 shutdown_connections 2023-10-16T13:48:48.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.453+0000 7f738a102700 1 -- 172.21.15.16:0/657018213 wait complete. 2023-10-16T13:48:48.460 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 146 2023-10-16T13:48:48.469 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:48:48.558 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:48.555+0000 7fd1d1dca200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T13:48:48.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.598+0000 7fc5f023d700 1 Processor -- start 2023-10-16T13:48:48.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.598+0000 7fc5f023d700 1 -- start start 2023-10-16T13:48:48.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.599+0000 7fc5f023d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc5e80d41c0 0x7fc5e80d0040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:48.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.599+0000 7fc5f023d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc5e80d0580 0x7fc5e806c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:48.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.599+0000 7fc5f023d700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5e806ba00 con 0x7fc5e815d8d0 2023-10-16T13:48:48.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.599+0000 7fc5f023d700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5e80d0a10 con 0x7fc5e80d41c0 2023-10-16T13:48:48.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.599+0000 7fc5f023d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e80d26d0 con 0x7fc5e80d0580 2023-10-16T13:48:48.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.599+0000 7fc5ed7d8700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc5e80d41c0 0x7fc5e80d0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:48.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.599+0000 7fc5edfd9700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc5e815d8d0 0x7fc5e80d1c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:47782/0 (socket says 172.21.15.16:47782) 2023-10-16T13:48:48.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.599+0000 7fc5edfd9700 1 -- 172.21.15.16:0/1501867338 learned_addr learned my addr 172.21.15.16:0/1501867338 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:48:48.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.599+0000 7fc5ed7d8700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc5e80d41c0 0x7fc5e80d0040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58510/0 (socket says 172.21.15.16:58510) 2023-10-16T13:48:48.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.599+0000 7fc5ee7da700 1 --2- 172.21.15.16:0/1501867338 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc5e80d0580 0x7fc5e806c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:48.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.600+0000 7fc5ed7d8700 1 -- 172.21.15.16:0/1501867338 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc5e80d0580 msgr2=0x7fc5e806c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:48.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.600+0000 7fc5ed7d8700 1 --2- 172.21.15.16:0/1501867338 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc5e80d0580 0x7fc5e806c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:48.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.600+0000 7fc5ed7d8700 1 -- 172.21.15.16:0/1501867338 >> v1:172.21.15.16:6789/0 conn(0x7fc5e815d8d0 legacy=0x7fc5e80d1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:48.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.600+0000 7fc5ecfd7700 1 -- 172.21.15.16:0/1501867338 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 493327596 0 0) 0x7fc5e806ba00 con 0x7fc5e815d8d0 2023-10-16T13:48:48.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.600+0000 7fc5ed7d8700 1 -- 172.21.15.16:0/1501867338 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5e80d23f0 con 0x7fc5e80d41c0 2023-10-16T13:48:48.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.600+0000 7fc5ed7d8700 1 --2- 172.21.15.16:0/1501867338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5e80d41c0 0x7fc5e80d0040 secure :-1 s=READY pgs=3106 cs=0 l=1 rev1=1 crypto rx=0x7fc5e4009840 tx=0x7fc5e4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de030bcdbbef47c4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:48.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.600+0000 7fc5ecfd7700 1 -- 172.21.15.16:0/1501867338 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e4011830 con 0x7fc5e80d41c0 2023-10-16T13:48:48.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.600+0000 7fc5ecfd7700 1 -- 172.21.15.16:0/1501867338 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5e4011990 con 0x7fc5e80d41c0 2023-10-16T13:48:48.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.600+0000 7fc5ecfd7700 1 -- 172.21.15.16:0/1501867338 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e4011d20 con 0x7fc5e80d41c0 2023-10-16T13:48:48.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.600+0000 7fc5f023d700 1 -- 172.21.15.16:0/1501867338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5e80d41c0 msgr2=0x7fc5e80d0040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:48.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.601+0000 7fc5f023d700 1 --2- 172.21.15.16:0/1501867338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5e80d41c0 0x7fc5e80d0040 secure :-1 s=READY pgs=3106 cs=0 l=1 rev1=1 crypto rx=0x7fc5e4009840 tx=0x7fc5e4009b50 comp rx=0 tx=0).stop 2023-10-16T13:48:48.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.601+0000 7fc5f023d700 1 -- 172.21.15.16:0/1501867338 shutdown_connections 2023-10-16T13:48:48.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.601+0000 7fc5f023d700 1 --2- 172.21.15.16:0/1501867338 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc5e80d0580 0x7fc5e806c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:48.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.601+0000 7fc5f023d700 1 --2- 172.21.15.16:0/1501867338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5e80d41c0 0x7fc5e80d0040 unknown :-1 s=CLOSED pgs=3106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:48.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.601+0000 7fc5f023d700 1 -- 172.21.15.16:0/1501867338 >> 172.21.15.16:0/1501867338 conn(0x7fc5e80d3630 msgr2=0x7fc5e80d3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:48.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.601+0000 7fc5f023d700 1 -- 172.21.15.16:0/1501867338 shutdown_connections 2023-10-16T13:48:48.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.601+0000 7fc5f023d700 1 -- 172.21.15.16:0/1501867338 wait complete. 2023-10-16T13:48:48.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.602+0000 7fc5f023d700 1 Processor -- start 2023-10-16T13:48:48.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.602+0000 7fc5f023d700 1 -- start start 2023-10-16T13:48:48.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.603+0000 7fc5f023d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc5e80d0580 0x7fc5e815acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:48.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.603+0000 7fc5f023d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc5e80d41c0 0x7fc5e81f6020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:48.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.603+0000 7fc5f023d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5e815d8d0 0x7fc5e80cc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:48.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.603+0000 7fc5f023d700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e815b210 con 0x7fc5e815d8d0 2023-10-16T13:48:48.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.603+0000 7fc5f023d700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e81f6900 con 0x7fc5e80d41c0 2023-10-16T13:48:48.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.603+0000 7fc5f023d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e81f6be0 con 0x7fc5e80d0580 2023-10-16T13:48:48.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.603+0000 7fc5ee7da700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5e815d8d0 0x7fc5e80cc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:48.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.603+0000 7fc5edfd9700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc5e80d0580 0x7fc5e815acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:48.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.603+0000 7fc5ee7da700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5e815d8d0 0x7fc5e80cc0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58518/0 (socket says 172.21.15.16:58518) 2023-10-16T13:48:48.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.603+0000 7fc5ee7da700 1 -- 172.21.15.16:0/3343354813 learned_addr learned my addr 172.21.15.16:0/3343354813 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:48.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.603+0000 7fc5ed7d8700 1 --2- 172.21.15.16:0/3343354813 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc5e80d41c0 0x7fc5e81f6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:48.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.603+0000 7fc5ee7da700 1 -- 172.21.15.16:0/3343354813 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc5e80d0580 msgr2=0x7fc5e815acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:48.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.603+0000 7fc5ee7da700 1 --2- 172.21.15.16:0/3343354813 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc5e80d0580 0x7fc5e815acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:48.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.603+0000 7fc5ee7da700 1 -- 172.21.15.16:0/3343354813 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc5e80d41c0 msgr2=0x7fc5e81f6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:48.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.603+0000 7fc5ee7da700 1 --2- 172.21.15.16:0/3343354813 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc5e80d41c0 0x7fc5e81f6020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:48.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.603+0000 7fc5ee7da700 1 -- 172.21.15.16:0/3343354813 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5e40094a0 con 0x7fc5e815d8d0 2023-10-16T13:48:48.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.603+0000 7fc5edfd9700 1 --2- 172.21.15.16:0/3343354813 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc5e80d0580 0x7fc5e815acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:48:48.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.603+0000 7fc5ed7d8700 1 --2- 172.21.15.16:0/3343354813 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc5e80d41c0 0x7fc5e81f6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:48:48.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.604+0000 7fc5ee7da700 1 --2- 172.21.15.16:0/3343354813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5e815d8d0 0x7fc5e80cc0d0 secure :-1 s=READY pgs=3107 cs=0 l=1 rev1=1 crypto rx=0x7fc5e000efc0 tx=0x7fc5e000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:48.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.604+0000 7fc5de7fc700 1 -- 172.21.15.16:0/3343354813 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e0009ea0 con 0x7fc5e815d8d0 2023-10-16T13:48:48.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.604+0000 7fc5de7fc700 1 -- 172.21.15.16:0/3343354813 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5e000f040 con 0x7fc5e815d8d0 2023-10-16T13:48:48.616 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.604+0000 7fc5de7fc700 1 -- 172.21.15.16:0/3343354813 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e000ced0 con 0x7fc5e815d8d0 2023-10-16T13:48:48.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.604+0000 7fc5f023d700 1 -- 172.21.15.16:0/3343354813 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5e81f6ec0 con 0x7fc5e815d8d0 2023-10-16T13:48:48.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.604+0000 7fc5f023d700 1 -- 172.21.15.16:0/3343354813 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5e80c74a0 con 0x7fc5e815d8d0 2023-10-16T13:48:48.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.605+0000 7fc5f023d700 1 -- 172.21.15.16:0/3343354813 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5e804ebb0 con 0x7fc5e815d8d0 2023-10-16T13:48:48.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.606+0000 7fc5de7fc700 1 -- 172.21.15.16:0/3343354813 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 163) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fc5e0014480 con 0x7fc5e815d8d0 2023-10-16T13:48:48.617 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.606+0000 7fc5de7fc700 1 -- 172.21.15.16:0/3343354813 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(267..267 src has 1..267) v4 ==== 6931+0+0 (secure 0 0 0) 0x7fc5e0089da0 con 0x7fc5e815d8d0 2023-10-16T13:48:48.618 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.611+0000 7fc5de7fc700 1 -- 172.21.15.16:0/3343354813 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc5e0054dd0 con 0x7fc5e815d8d0 2023-10-16T13:48:48.643 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:48.639+0000 7fd14a5f0700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:48:48.643 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:48.639+0000 7fd14a5f0700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:48:48.643 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:48.639+0000 7fd14a5f0700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:48:48.643 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:48.639+0000 7fd14a5f0700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:48:48.643 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:48.639+0000 7fd14a5f0700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:48:48.644 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:48.639+0000 7fd14d5f6700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:48:48.644 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:48.639+0000 7fd14d5f6700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:48:48.644 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:48.639+0000 7fd14d5f6700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:48:48.644 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:48.639+0000 7fd14d5f6700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:48:48.644 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:48:48.639+0000 7fd14d5f6700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:48:48.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.867+0000 7fc5f023d700 1 -- 172.21.15.16:0/3343354813 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc5e81e7de0 con 0x7fc5e815d8d0 2023-10-16T13:48:48.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.867+0000 7fc5de7fc700 1 -- 172.21.15.16:0/3343354813 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 146 v146) v1 ==== 95+0+4926 (secure 0 0 0) 0x7fc5e0018020 con 0x7fc5e815d8d0 2023-10-16T13:48:48.871 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:48:48.871 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":146,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10112,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/712583564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":712583564},{"type":"v1","addr":"172.21.15.16:6837","nonce":712583564}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":141},{"gid":11836,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/3551114031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":3551114031},{"type":"v1","addr":"172.21.15.49:6834","nonce":3551114031}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":141},{"gid":13050,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/2135643757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2135643757},{"type":"v1","addr":"172.21.15.16:6836","nonce":2135643757}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":140}],"filesystems":[{"mdsmap":{"epoch":146,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:48:24.832614+0000","modified":"2023-10-16T13:48:27.858697+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":13068},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13068":{"gid":13068,"name":"d","rank":0,"incarnation":142,"state":"up:active","state_seq":3,"addr":"172.21.15.49:6835/1985453659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":1985453659},{"type":"v1","addr":"172.21.15.49:6835","nonce":1985453659}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[36],"metadata_pool":35,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":17}]} 2023-10-16T13:48:48.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.869+0000 7fc5f023d700 1 -- 172.21.15.16:0/3343354813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5e815d8d0 msgr2=0x7fc5e80cc0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:48.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.869+0000 7fc5f023d700 1 --2- 172.21.15.16:0/3343354813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5e815d8d0 0x7fc5e80cc0d0 secure :-1 s=READY pgs=3107 cs=0 l=1 rev1=1 crypto rx=0x7fc5e000efc0 tx=0x7fc5e000c3a0 comp rx=0 tx=0).stop 2023-10-16T13:48:48.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.870+0000 7fc5f023d700 1 -- 172.21.15.16:0/3343354813 shutdown_connections 2023-10-16T13:48:48.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.870+0000 7fc5f023d700 1 --2- 172.21.15.16:0/3343354813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5e815d8d0 0x7fc5e80cc0d0 unknown :-1 s=CLOSED pgs=3107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:48.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.870+0000 7fc5f023d700 1 --2- 172.21.15.16:0/3343354813 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc5e80d0580 0x7fc5e815acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:48.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.870+0000 7fc5f023d700 1 --2- 172.21.15.16:0/3343354813 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc5e80d41c0 0x7fc5e81f6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:48.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.870+0000 7fc5f023d700 1 -- 172.21.15.16:0/3343354813 >> 172.21.15.16:0/3343354813 conn(0x7fc5e80d3630 msgr2=0x7fc5e80b1f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:48.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.870+0000 7fc5f023d700 1 -- 172.21.15.16:0/3343354813 shutdown_connections 2023-10-16T13:48:48.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:48.870+0000 7fc5f023d700 1 -- 172.21.15.16:0/3343354813 wait complete. 2023-10-16T13:48:48.875 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 146 2023-10-16T13:48:48.885 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:48:49.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.006+0000 7f2399344700 1 Processor -- start 2023-10-16T13:48:49.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.006+0000 7f2399344700 1 -- start start 2023-10-16T13:48:49.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.007+0000 7f2399344700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2394160d90 0x7f239406a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:49.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.007+0000 7f2399344700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f239406da60 0x7f239406ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:49.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.007+0000 7f2399344700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23940ced00 con 0x7f239406e450 2023-10-16T13:48:49.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.007+0000 7f2399344700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2394162180 con 0x7f239406da60 2023-10-16T13:48:49.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.007+0000 7f2399344700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23940caa40 con 0x7f2394160d90 2023-10-16T13:48:49.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.007+0000 7f23937fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f239406da60 0x7f239406ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:49.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.007+0000 7f23937fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f239406da60 0x7f239406ad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58576/0 (socket says 172.21.15.16:58576) 2023-10-16T13:48:49.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.007+0000 7f23937fe700 1 -- 172.21.15.16:0/552510529 learned_addr learned my addr 172.21.15.16:0/552510529 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:49.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.008+0000 7f2392ffd700 1 --2- 172.21.15.16:0/552510529 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2394160d90 0x7f239406a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:49.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.008+0000 7f23937fe700 1 -- 172.21.15.16:0/552510529 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2394160d90 msgr2=0x7f239406a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:49.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.008+0000 7f23937fe700 1 --2- 172.21.15.16:0/552510529 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2394160d90 0x7f239406a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:49.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.008+0000 7f23937fe700 1 -- 172.21.15.16:0/552510529 >> v1:172.21.15.16:6789/0 conn(0x7f239406e450 legacy=0x7f23940cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:49.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.008+0000 7f23937fe700 1 -- 172.21.15.16:0/552510529 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2394161570 con 0x7f239406da60 2023-10-16T13:48:49.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.008+0000 7f23937fe700 1 --2- 172.21.15.16:0/552510529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f239406da60 0x7f239406ad20 secure :-1 s=READY pgs=3115 cs=0 l=1 rev1=1 crypto rx=0x7f238c009e50 tx=0x7f238c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e7d00202733c05b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:49.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.008+0000 7f2391ffb700 1 -- 172.21.15.16:0/552510529 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f238c00e070 con 0x7f239406da60 2023-10-16T13:48:49.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.008+0000 7f2391ffb700 1 -- 172.21.15.16:0/552510529 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f238c00b9c0 con 0x7f239406da60 2023-10-16T13:48:49.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.008+0000 7f2391ffb700 1 -- 172.21.15.16:0/552510529 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f238c00bd70 con 0x7f239406da60 2023-10-16T13:48:49.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.009+0000 7f2399344700 1 -- 172.21.15.16:0/552510529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f239406da60 msgr2=0x7f239406ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:49.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.009+0000 7f2399344700 1 --2- 172.21.15.16:0/552510529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f239406da60 0x7f239406ad20 secure :-1 s=READY pgs=3115 cs=0 l=1 rev1=1 crypto rx=0x7f238c009e50 tx=0x7f238c00b330 comp rx=0 tx=0).stop 2023-10-16T13:48:49.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.009+0000 7f2399344700 1 -- 172.21.15.16:0/552510529 shutdown_connections 2023-10-16T13:48:49.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.009+0000 7f2399344700 1 --2- 172.21.15.16:0/552510529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f239406da60 0x7f239406ad20 unknown :-1 s=CLOSED pgs=3115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:49.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.009+0000 7f2399344700 1 --2- 172.21.15.16:0/552510529 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2394160d90 0x7f239406a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:49.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.009+0000 7f2399344700 1 -- 172.21.15.16:0/552510529 >> 172.21.15.16:0/552510529 conn(0x7f23940b1f40 msgr2=0x7f23940b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:49.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.009+0000 7f2399344700 1 -- 172.21.15.16:0/552510529 shutdown_connections 2023-10-16T13:48:49.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.009+0000 7f2399344700 1 -- 172.21.15.16:0/552510529 wait complete. 2023-10-16T13:48:49.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.010+0000 7f2399344700 1 Processor -- start 2023-10-16T13:48:49.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.010+0000 7f2399344700 1 -- start start 2023-10-16T13:48:49.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.011+0000 7f2399344700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f239406da60 0x7f23940c64d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:49.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.011+0000 7f2399344700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f239406e450 0x7f239415d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:49.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.011+0000 7f2399344700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2394160d90 0x7f23940b3c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:49.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.011+0000 7f2399344700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f239414f430 con 0x7f2394160d90 2023-10-16T13:48:49.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.011+0000 7f2399344700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23940c6040 con 0x7f239406da60 2023-10-16T13:48:49.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.011+0000 7f2399344700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23940c5430 con 0x7f239406e450 2023-10-16T13:48:49.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.011+0000 7f23937fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2394160d90 0x7f23940b3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:49.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.011+0000 7f23937fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2394160d90 0x7f23940b3c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58582/0 (socket says 172.21.15.16:58582) 2023-10-16T13:48:49.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.011+0000 7f23937fe700 1 -- 172.21.15.16:0/1653525056 learned_addr learned my addr 172.21.15.16:0/1653525056 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:49.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.011+0000 7f2392ffd700 1 --2- 172.21.15.16:0/1653525056 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f239406da60 0x7f23940c64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:49.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.011+0000 7f23927fc700 1 --2- 172.21.15.16:0/1653525056 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f239406e450 0x7f239415d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:49.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.011+0000 7f23937fe700 1 -- 172.21.15.16:0/1653525056 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f239406e450 msgr2=0x7f239415d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:49.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.011+0000 7f23937fe700 1 --2- 172.21.15.16:0/1653525056 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f239406e450 0x7f239415d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:49.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.011+0000 7f23937fe700 1 -- 172.21.15.16:0/1653525056 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f239406da60 msgr2=0x7f23940c64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:49.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.011+0000 7f23937fe700 1 --2- 172.21.15.16:0/1653525056 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f239406da60 0x7f23940c64d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:49.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.011+0000 7f23937fe700 1 -- 172.21.15.16:0/1653525056 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f238c009b30 con 0x7f2394160d90 2023-10-16T13:48:49.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.011+0000 7f23937fe700 1 --2- 172.21.15.16:0/1653525056 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2394160d90 0x7f23940b3c50 secure :-1 s=READY pgs=3116 cs=0 l=1 rev1=1 crypto rx=0x7f238c012000 tx=0x7f238c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:49.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.011+0000 7f237b7fe700 1 -- 172.21.15.16:0/1653525056 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f238c0047c0 con 0x7f2394160d90 2023-10-16T13:48:49.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.012+0000 7f237b7fe700 1 -- 172.21.15.16:0/1653525056 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f238c0099b0 con 0x7f2394160d90 2023-10-16T13:48:49.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.012+0000 7f2399344700 1 -- 172.21.15.16:0/1653525056 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23940c5710 con 0x7f2394160d90 2023-10-16T13:48:49.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.012+0000 7f237b7fe700 1 -- 172.21.15.16:0/1653525056 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f238c01c680 con 0x7f2394160d90 2023-10-16T13:48:49.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.012+0000 7f2399344700 1 -- 172.21.15.16:0/1653525056 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23940c5cd0 con 0x7f2394160d90 2023-10-16T13:48:49.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.013+0000 7f2399344700 1 -- 172.21.15.16:0/1653525056 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23940c61b0 con 0x7f2394160d90 2023-10-16T13:48:49.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.013+0000 7f237b7fe700 1 -- 172.21.15.16:0/1653525056 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 163) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f238c01c7e0 con 0x7f2394160d90 2023-10-16T13:48:49.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.013+0000 7f237b7fe700 1 -- 172.21.15.16:0/1653525056 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(267..267 src has 1..267) v4 ==== 6931+0+0 (secure 0 0 0) 0x7f238c08b070 con 0x7f2394160d90 2023-10-16T13:48:49.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.018+0000 7f237b7fe700 1 -- 172.21.15.16:0/1653525056 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f238c0560a0 con 0x7f2394160d90 2023-10-16T13:48:49.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.224+0000 7f2399344700 1 -- 172.21.15.16:0/1653525056 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f239404ec20 con 0x7f2394160d90 2023-10-16T13:48:49.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.225+0000 7f237b7fe700 1 -- 172.21.15.16:0/1653525056 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v267) v1 ==== 74+0+17941 (secure 0 0 0) 0x7f238c0596c0 con 0x7f2394160d90 2023-10-16T13:48:49.229 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:48:49.229 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":267,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:48:48.569703+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":36,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_metadata","create_time":"2023-10-16T13:48:23.348668+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"264","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_data","create_time":"2023-10-16T13:48:23.799968+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"265","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6833/2131389174":"2023-10-17T13:48:48.569666+0000","172.21.15.16:6832/2131389174":"2023-10-17T13:48:48.569666+0000","172.21.15.16:0/762492059":"2023-10-17T13:48:48.569666+0000","172.21.15.16:0/1604016973":"2023-10-17T13:48:48.569666+0000","172.21.15.16:0/2535628588":"2023-10-17T13:48:48.569666+0000","172.21.15.16:0/3313712117":"2023-10-17T13:48:39.972713+0000","172.21.15.16:0/2881413822":"2023-10-17T13:48:48.569666+0000","172.21.15.16:0/2202829273":"2023-10-17T13:48:39.972713+0000","172.21.15.16:0/2777724108":"2023-10-17T13:48:39.972713+0000","172.21.15.16:0/2088077503":"2023-10-17T13:48:39.972713+0000","172.21.15.16:0/4096007345":"2023-10-17T13:48:39.972713+0000","172.21.15.16:0/3000870029":"2023-10-17T13:48:39.972713+0000","172.21.15.16:0/3746447491":"2023-10-17T13:48:48.569666+0000","172.21.15.16:0/3923653515":"2023-10-17T13:48:39.972713+0000","172.21.15.16:0/1621010103":"2023-10-17T13:48:48.569666+0000","172.21.15.16:0/56667078":"2023-10-17T13:48:39.972713+0000","172.21.15.16:6833/1275231846":"2023-10-17T13:48:39.972713+0000","172.21.15.16:6832/1275231846":"2023-10-17T13:48:39.972713+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:48:49.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.227+0000 7f2399344700 1 -- 172.21.15.16:0/1653525056 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2394160d90 msgr2=0x7f23940b3c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:49.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.227+0000 7f2399344700 1 --2- 172.21.15.16:0/1653525056 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2394160d90 0x7f23940b3c50 secure :-1 s=READY pgs=3116 cs=0 l=1 rev1=1 crypto rx=0x7f238c012000 tx=0x7f238c004ad0 comp rx=0 tx=0).stop 2023-10-16T13:48:49.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.227+0000 7f2399344700 1 -- 172.21.15.16:0/1653525056 shutdown_connections 2023-10-16T13:48:49.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.227+0000 7f2399344700 1 --2- 172.21.15.16:0/1653525056 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2394160d90 0x7f23940b3c50 unknown :-1 s=CLOSED pgs=3116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:49.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.227+0000 7f2399344700 1 --2- 172.21.15.16:0/1653525056 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f239406da60 0x7f23940c64d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:49.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.227+0000 7f2399344700 1 --2- 172.21.15.16:0/1653525056 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f239406e450 0x7f239415d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:49.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.227+0000 7f2399344700 1 -- 172.21.15.16:0/1653525056 >> 172.21.15.16:0/1653525056 conn(0x7f23940b1f40 msgr2=0x7f239415bed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:49.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.228+0000 7f2399344700 1 -- 172.21.15.16:0/1653525056 shutdown_connections 2023-10-16T13:48:49.233 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.228+0000 7f2399344700 1 -- 172.21.15.16:0/1653525056 wait complete. 2023-10-16T13:48:49.243 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-16T13:48:49.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.366+0000 7fc138ebf700 1 Processor -- start 2023-10-16T13:48:49.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.366+0000 7fc138ebf700 1 -- start start 2023-10-16T13:48:49.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.367+0000 7fc138ebf700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc13406aef0 0x7fc1340b3480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:49.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.367+0000 7fc138ebf700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1340d11c0 con 0x7fc1340b2be0 2023-10-16T13:48:49.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.367+0000 7fc138ebf700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1340cd360 con 0x7fc13415b860 2023-10-16T13:48:49.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.367+0000 7fc138ebf700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1340d03d0 con 0x7fc13406aef0 2023-10-16T13:48:49.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.367+0000 7fc131d9b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc13406aef0 0x7fc1340b3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:49.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.367+0000 7fc131d9b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc13406aef0 0x7fc1340b3480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58586/0 (socket says 172.21.15.16:58586) 2023-10-16T13:48:49.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.367+0000 7fc132d9d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc1340b2be0 0x7fc1340b39c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:47862/0 (socket says 172.21.15.16:47862) 2023-10-16T13:48:49.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.367+0000 7fc131d9b700 1 -- 172.21.15.16:0/3922361953 learned_addr learned my addr 172.21.15.16:0/3922361953 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:49.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.367+0000 7fc131d9b700 1 -- 172.21.15.16:0/3922361953 >> v1:172.21.15.49:6789/0 conn(0x7fc13415b860 legacy=0x7fc13406a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:49.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.367+0000 7fc131d9b700 1 -- 172.21.15.16:0/3922361953 >> v1:172.21.15.16:6789/0 conn(0x7fc1340b2be0 legacy=0x7fc1340b39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:49.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.367+0000 7fc131d9b700 1 -- 172.21.15.16:0/3922361953 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1340d05b0 con 0x7fc13406aef0 2023-10-16T13:48:49.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.367+0000 7fc13159a700 1 -- 172.21.15.16:0/3922361953 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3515722456 0 0) 0x7fc1340d11c0 con 0x7fc1340b2be0 2023-10-16T13:48:49.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.368+0000 7fc131d9b700 1 --2- 172.21.15.16:0/3922361953 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc13406aef0 0x7fc1340b3480 secure :-1 s=READY pgs=3118 cs=0 l=1 rev1=1 crypto rx=0x7fc1280098f0 tx=0x7fc128009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57bf2073aadc1e23 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:49.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.368+0000 7fc13159a700 1 -- 172.21.15.16:0/3922361953 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc12800b8a0 con 0x7fc13406aef0 2023-10-16T13:48:49.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.368+0000 7fc13159a700 1 -- 172.21.15.16:0/3922361953 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc12800ba00 con 0x7fc13406aef0 2023-10-16T13:48:49.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.368+0000 7fc13159a700 1 -- 172.21.15.16:0/3922361953 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc12800bd90 con 0x7fc13406aef0 2023-10-16T13:48:49.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.368+0000 7fc138ebf700 1 -- 172.21.15.16:0/3922361953 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc13406aef0 msgr2=0x7fc1340b3480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:49.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.368+0000 7fc138ebf700 1 --2- 172.21.15.16:0/3922361953 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc13406aef0 0x7fc1340b3480 secure :-1 s=READY pgs=3118 cs=0 l=1 rev1=1 crypto rx=0x7fc1280098f0 tx=0x7fc128009c00 comp rx=0 tx=0).stop 2023-10-16T13:48:49.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.368+0000 7fc138ebf700 1 -- 172.21.15.16:0/3922361953 shutdown_connections 2023-10-16T13:48:49.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.368+0000 7fc138ebf700 1 --2- 172.21.15.16:0/3922361953 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc13406aef0 0x7fc1340b3480 unknown :-1 s=CLOSED pgs=3118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:49.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.368+0000 7fc138ebf700 1 -- 172.21.15.16:0/3922361953 >> 172.21.15.16:0/3922361953 conn(0x7fc1340b1fd0 msgr2=0x7fc1340b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:49.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.368+0000 7fc138ebf700 1 -- 172.21.15.16:0/3922361953 shutdown_connections 2023-10-16T13:48:49.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.368+0000 7fc138ebf700 1 -- 172.21.15.16:0/3922361953 wait complete. 2023-10-16T13:48:49.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.370+0000 7fc138ebf700 1 Processor -- start 2023-10-16T13:48:49.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.370+0000 7fc138ebf700 1 -- start start 2023-10-16T13:48:49.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.370+0000 7fc138ebf700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc13406aef0 0x7fc1341f3740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:49.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.370+0000 7fc138ebf700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1340b2be0 0x7fc1341e42f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:49.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.370+0000 7fc138ebf700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc13415b860 0x7fc1341e4830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:49.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.370+0000 7fc138ebf700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1340c5720 con 0x7fc1340b2be0 2023-10-16T13:48:49.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.370+0000 7fc138ebf700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1340c5a00 con 0x7fc13415b860 2023-10-16T13:48:49.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.370+0000 7fc138ebf700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1340c5ce0 con 0x7fc13406aef0 2023-10-16T13:48:49.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.371+0000 7fc131d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1340b2be0 0x7fc1341e42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:49.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.371+0000 7fc131d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1340b2be0 0x7fc1341e42f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58592/0 (socket says 172.21.15.16:58592) 2023-10-16T13:48:49.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.371+0000 7fc131d9b700 1 -- 172.21.15.16:0/4140402048 learned_addr learned my addr 172.21.15.16:0/4140402048 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:49.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.371+0000 7fc132d9d700 1 --2- 172.21.15.16:0/4140402048 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc13415b860 0x7fc1341e4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:49.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.371+0000 7fc13259c700 1 --2- 172.21.15.16:0/4140402048 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc13406aef0 0x7fc1341f3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:49.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.371+0000 7fc131d9b700 1 -- 172.21.15.16:0/4140402048 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc13406aef0 msgr2=0x7fc1341f3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:49.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.371+0000 7fc131d9b700 1 --2- 172.21.15.16:0/4140402048 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc13406aef0 0x7fc1341f3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:49.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.371+0000 7fc131d9b700 1 -- 172.21.15.16:0/4140402048 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc13415b860 msgr2=0x7fc1341e4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:49.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.371+0000 7fc131d9b700 1 --2- 172.21.15.16:0/4140402048 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc13415b860 0x7fc1341e4830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:49.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.371+0000 7fc131d9b700 1 -- 172.21.15.16:0/4140402048 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc128009550 con 0x7fc1340b2be0 2023-10-16T13:48:49.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.371+0000 7fc131d9b700 1 --2- 172.21.15.16:0/4140402048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1340b2be0 0x7fc1341e42f0 secure :-1 s=READY pgs=3119 cs=0 l=1 rev1=1 crypto rx=0x7fc128000c00 tx=0x7fc128004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:49.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.371+0000 7fc122ffd700 1 -- 172.21.15.16:0/4140402048 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc128011970 con 0x7fc1340b2be0 2023-10-16T13:48:49.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.371+0000 7fc122ffd700 1 -- 172.21.15.16:0/4140402048 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc12801f070 con 0x7fc1340b2be0 2023-10-16T13:48:49.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.371+0000 7fc122ffd700 1 -- 172.21.15.16:0/4140402048 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc128011d10 con 0x7fc1340b2be0 2023-10-16T13:48:49.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.371+0000 7fc138ebf700 1 -- 172.21.15.16:0/4140402048 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1340c5fc0 con 0x7fc1340b2be0 2023-10-16T13:48:49.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.371+0000 7fc138ebf700 1 -- 172.21.15.16:0/4140402048 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1341f3c80 con 0x7fc1340b2be0 2023-10-16T13:48:49.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.372+0000 7fc138ebf700 1 -- 172.21.15.16:0/4140402048 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc13404ebb0 con 0x7fc1340b2be0 2023-10-16T13:48:49.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.373+0000 7fc122ffd700 1 -- 172.21.15.16:0/4140402048 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 163) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fc128026480 con 0x7fc1340b2be0 2023-10-16T13:48:49.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.373+0000 7fc122ffd700 1 -- 172.21.15.16:0/4140402048 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(267..267 src has 1..267) v4 ==== 6931+0+0 (secure 0 0 0) 0x7fc128021070 con 0x7fc1340b2be0 2023-10-16T13:48:49.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.378+0000 7fc122ffd700 1 -- 172.21.15.16:0/4140402048 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc128055f60 con 0x7fc1340b2be0 2023-10-16T13:48:49.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.578+0000 7fc122ffd700 1 -- 172.21.15.16:0/4140402048 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 164) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fc128055a10 con 0x7fc1340b2be0 2023-10-16T13:48:49.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.579+0000 7fc122ffd700 1 --2- 172.21.15.16:0/4140402048 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fc114070e80 0x7fc114073270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:49.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.581+0000 7fc13259c700 1 --2- 172.21.15.16:0/4140402048 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fc114070e80 0x7fc114073270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:49.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.582+0000 7fc13259c700 1 --2- 172.21.15.16:0/4140402048 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fc114070e80 0x7fc114073270 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fc124007fd0 tx=0x7fc124009040 comp rx=0 tx=0).ready entity=mgr.13344 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:49.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.640+0000 7fc138ebf700 1 -- 172.21.15.16:0/4140402048 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fc13415eb80 con 0x7fc1340b2be0 2023-10-16T13:48:49.643 INFO:tasks.ceph.mon.a.smithi016.stderr:2023-10-16T13:48:49.640+0000 7f9fc8535700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-16T13:48:49.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.641+0000 7fc122ffd700 1 -- 172.21.15.16:0/4140402048 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v147) v1 ==== 161+0+0 (secure 0 0 0) 0x7fc12805c890 con 0x7fc1340b2be0 2023-10-16T13:48:49.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.645+0000 7fc138ebf700 1 -- 172.21.15.16:0/4140402048 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fc114070e80 msgr2=0x7fc114073270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:49.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.645+0000 7fc138ebf700 1 --2- 172.21.15.16:0/4140402048 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fc114070e80 0x7fc114073270 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fc124007fd0 tx=0x7fc124009040 comp rx=0 tx=0).stop 2023-10-16T13:48:49.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.645+0000 7fc138ebf700 1 -- 172.21.15.16:0/4140402048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1340b2be0 msgr2=0x7fc1341e42f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:49.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.645+0000 7fc138ebf700 1 --2- 172.21.15.16:0/4140402048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1340b2be0 0x7fc1341e42f0 secure :-1 s=READY pgs=3119 cs=0 l=1 rev1=1 crypto rx=0x7fc128000c00 tx=0x7fc128004750 comp rx=0 tx=0).stop 2023-10-16T13:48:49.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.645+0000 7fc138ebf700 1 -- 172.21.15.16:0/4140402048 shutdown_connections 2023-10-16T13:48:49.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.645+0000 7fc138ebf700 1 --2- 172.21.15.16:0/4140402048 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc13415b860 0x7fc1341e4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:49.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.645+0000 7fc138ebf700 1 --2- 172.21.15.16:0/4140402048 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc13406aef0 0x7fc1341f3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:49.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.645+0000 7fc138ebf700 1 --2- 172.21.15.16:0/4140402048 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fc114070e80 0x7fc114073270 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:49.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.645+0000 7fc138ebf700 1 --2- 172.21.15.16:0/4140402048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1340b2be0 0x7fc1341e42f0 unknown :-1 s=CLOSED pgs=3119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:49.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.645+0000 7fc138ebf700 1 -- 172.21.15.16:0/4140402048 >> 172.21.15.16:0/4140402048 conn(0x7fc1340b1fd0 msgr2=0x7fc134159990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:49.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.645+0000 7fc138ebf700 1 -- 172.21.15.16:0/4140402048 shutdown_connections 2023-10-16T13:48:49.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.645+0000 7fc138ebf700 1 -- 172.21.15.16:0/4140402048 wait complete. 2023-10-16T13:48:49.651 INFO:tasks.ceph.mds.d.smithi049.stderr: -5238> 2023-10-16T13:48:27.401+0000 7f65323c1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T13:48:49.651 INFO:teuthology.orchestra.run.smithi016.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-16T13:48:49.661 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-16T13:48:49.683 INFO:tasks.ceph.mds.d.smithi049.stdout:starting mds.d at 2023-10-16T13:48:49.787 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.784+0000 7f6ab3237700 1 Processor -- start 2023-10-16T13:48:49.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.784+0000 7f6ab3237700 1 -- start start 2023-10-16T13:48:49.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.785+0000 7f6ab3237700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6aac072b30 0x7f6aac071a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:49.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.785+0000 7f6ab3237700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6aac0caa40 0x7f6aac0cae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:49.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.785+0000 7f6ab3237700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6aac0cb3d0 con 0x7f6aac15bb40 2023-10-16T13:48:49.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.785+0000 7f6ab3237700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6aac06ea20 con 0x7f6aac0caa40 2023-10-16T13:48:49.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.785+0000 7f6ab3237700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6aac15c040 con 0x7f6aac072b30 2023-10-16T13:48:49.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.786+0000 7f6ab17d4700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6aac0caa40 0x7f6aac0cae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:49.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.786+0000 7f6aabfff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f6aac15bb40 0x7f6aac06e310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:47894/0 (socket says 172.21.15.16:47894) 2023-10-16T13:48:49.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.786+0000 7f6aabfff700 1 -- 172.21.15.16:0/2312528167 learned_addr learned my addr 172.21.15.16:0/2312528167 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:48:49.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.786+0000 7f6ab0fd3700 1 --2- 172.21.15.16:0/2312528167 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6aac072b30 0x7f6aac071a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:49.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.786+0000 7f6aab7fe700 1 -- 172.21.15.16:0/2312528167 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2772469761 0 0) 0x7f6a98001110 con 0x7f6aac15bb40 2023-10-16T13:48:49.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.786+0000 7f6aab7fe700 1 -- 172.21.15.16:0/2312528167 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a9c0035b0 con 0x7f6aac15bb40 2023-10-16T13:48:49.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.786+0000 7f6ab17d4700 1 -- 172.21.15.16:0/2312528167 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6aac072b30 msgr2=0x7f6aac071a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:49.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.786+0000 7f6ab17d4700 1 --2- 172.21.15.16:0/2312528167 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6aac072b30 0x7f6aac071a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:49.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.786+0000 7f6ab17d4700 1 -- 172.21.15.16:0/2312528167 >> v1:172.21.15.16:6789/0 conn(0x7f6aac15bb40 legacy=0x7f6aac06e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:49.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.786+0000 7f6ab17d4700 1 -- 172.21.15.16:0/2312528167 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6aac15c320 con 0x7f6aac0caa40 2023-10-16T13:48:49.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.786+0000 7f6ab17d4700 1 --2- 172.21.15.16:0/2312528167 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6aac0caa40 0x7f6aac0cae90 secure :-1 s=READY pgs=3122 cs=0 l=1 rev1=1 crypto rx=0x7f6aa4009d90 tx=0x7f6aa400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=229fbd91c02ee75f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:49.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.786+0000 7f6aab7fe700 1 -- 172.21.15.16:0/2312528167 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aa400e070 con 0x7f6aac0caa40 2023-10-16T13:48:49.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.786+0000 7f6aab7fe700 1 -- 172.21.15.16:0/2312528167 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6aa400bb20 con 0x7f6aac0caa40 2023-10-16T13:48:49.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.786+0000 7f6aab7fe700 1 -- 172.21.15.16:0/2312528167 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aa4012430 con 0x7f6aac0caa40 2023-10-16T13:48:49.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.787+0000 7f6ab3237700 1 -- 172.21.15.16:0/2312528167 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6aac0caa40 msgr2=0x7f6aac0cae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:49.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.787+0000 7f6ab3237700 1 --2- 172.21.15.16:0/2312528167 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6aac0caa40 0x7f6aac0cae90 secure :-1 s=READY pgs=3122 cs=0 l=1 rev1=1 crypto rx=0x7f6aa4009d90 tx=0x7f6aa400b330 comp rx=0 tx=0).stop 2023-10-16T13:48:49.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.787+0000 7f6ab3237700 1 -- 172.21.15.16:0/2312528167 shutdown_connections 2023-10-16T13:48:49.794 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.787+0000 7f6ab3237700 1 --2- 172.21.15.16:0/2312528167 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6aac0caa40 0x7f6aac0cae90 unknown :-1 s=CLOSED pgs=3122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:49.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.787+0000 7f6ab3237700 1 --2- 172.21.15.16:0/2312528167 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6aac072b30 0x7f6aac071a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:49.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.787+0000 7f6ab3237700 1 -- 172.21.15.16:0/2312528167 >> 172.21.15.16:0/2312528167 conn(0x7f6aac0b1f40 msgr2=0x7f6aac0b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:49.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.787+0000 7f6ab3237700 1 -- 172.21.15.16:0/2312528167 shutdown_connections 2023-10-16T13:48:49.795 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.787+0000 7f6ab3237700 1 -- 172.21.15.16:0/2312528167 wait complete. 2023-10-16T13:48:49.797 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.788+0000 7f6ab3237700 1 Processor -- start 2023-10-16T13:48:49.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.789+0000 7f6ab3237700 1 -- start start 2023-10-16T13:48:49.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.790+0000 7f6ab3237700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6aac072b30 0x7f6aac1585b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:49.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.790+0000 7f6ab3237700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6aac0caa40 0x7f6aac154960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:49.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.790+0000 7f6ab3237700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6aac15bb40 0x7f6aac154ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:49.798 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.790+0000 7f6ab3237700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6aac0c88a0 con 0x7f6aac15bb40 2023-10-16T13:48:49.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.790+0000 7f6ab3237700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6aac0c8a90 con 0x7f6aac072b30 2023-10-16T13:48:49.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.790+0000 7f6ab3237700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6aac0c8d70 con 0x7f6aac0caa40 2023-10-16T13:48:49.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.790+0000 7f6ab17d4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6aac15bb40 0x7f6aac154ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:49.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.790+0000 7f6ab17d4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6aac15bb40 0x7f6aac154ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58630/0 (socket says 172.21.15.16:58630) 2023-10-16T13:48:49.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.790+0000 7f6ab17d4700 1 -- 172.21.15.16:0/2102190964 learned_addr learned my addr 172.21.15.16:0/2102190964 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:49.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.790+0000 7f6aabfff700 1 --2- 172.21.15.16:0/2102190964 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6aac0caa40 0x7f6aac154960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:49.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.790+0000 7f6ab0fd3700 1 --2- 172.21.15.16:0/2102190964 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6aac072b30 0x7f6aac1585b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:49.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.790+0000 7f6ab17d4700 1 -- 172.21.15.16:0/2102190964 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6aac0caa40 msgr2=0x7f6aac154960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:49.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.790+0000 7f6ab17d4700 1 --2- 172.21.15.16:0/2102190964 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6aac0caa40 0x7f6aac154960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:49.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.790+0000 7f6ab17d4700 1 -- 172.21.15.16:0/2102190964 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6aac072b30 msgr2=0x7f6aac1585b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:49.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.790+0000 7f6ab17d4700 1 --2- 172.21.15.16:0/2102190964 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6aac072b30 0x7f6aac1585b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:49.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.790+0000 7f6ab17d4700 1 -- 172.21.15.16:0/2102190964 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6aa4009a70 con 0x7f6aac15bb40 2023-10-16T13:48:49.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.790+0000 7f6ab0fd3700 1 --2- 172.21.15.16:0/2102190964 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6aac072b30 0x7f6aac1585b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:48:49.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.790+0000 7f6ab17d4700 1 --2- 172.21.15.16:0/2102190964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6aac15bb40 0x7f6aac154ea0 secure :-1 s=READY pgs=3123 cs=0 l=1 rev1=1 crypto rx=0x7f6aa4014040 tx=0x7f6aa4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:49.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.790+0000 7f6aa97fa700 1 -- 172.21.15.16:0/2102190964 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aa40129f0 con 0x7f6aac15bb40 2023-10-16T13:48:49.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.790+0000 7f6aa97fa700 1 -- 172.21.15.16:0/2102190964 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6aa4020070 con 0x7f6aac15bb40 2023-10-16T13:48:49.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.790+0000 7f6ab3237700 1 -- 172.21.15.16:0/2102190964 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6aac0c9050 con 0x7f6aac15bb40 2023-10-16T13:48:49.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.790+0000 7f6aa97fa700 1 -- 172.21.15.16:0/2102190964 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aa4012db0 con 0x7f6aac15bb40 2023-10-16T13:48:49.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.790+0000 7f6ab3237700 1 -- 172.21.15.16:0/2102190964 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6aac0c9610 con 0x7f6aac15bb40 2023-10-16T13:48:49.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.791+0000 7f6ab3237700 1 -- 172.21.15.16:0/2102190964 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6aac04ebb0 con 0x7f6aac15bb40 2023-10-16T13:48:49.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.792+0000 7f6aa97fa700 1 -- 172.21.15.16:0/2102190964 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 164) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f6aa4004040 con 0x7f6aac15bb40 2023-10-16T13:48:49.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.792+0000 7f6aa97fa700 1 --2- 172.21.15.16:0/2102190964 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f6a9006ad50 0x7f6a9006d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:49.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.792+0000 7f6aa97fa700 1 -- 172.21.15.16:0/2102190964 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(268..268 src has 1..268) v4 ==== 7017+0+0 (secure 0 0 0) 0x7f6aa4022070 con 0x7f6aac15bb40 2023-10-16T13:48:49.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.792+0000 7f6ab0fd3700 1 --2- 172.21.15.16:0/2102190964 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f6a9006ad50 0x7f6a9006d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:49.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.793+0000 7f6ab0fd3700 1 --2- 172.21.15.16:0/2102190964 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f6a9006ad50 0x7f6a9006d210 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f6aa0009b90 tx=0x7f6aa0008040 comp rx=0 tx=0).ready entity=mgr.13344 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:49.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:49.797+0000 7f6aa97fa700 1 -- 172.21.15.16:0/2102190964 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6aa40564f0 con 0x7f6aac15bb40 2023-10-16T13:48:50.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.051+0000 7f6ab3237700 1 -- 172.21.15.16:0/2102190964 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f6aac0b2d40 con 0x7f6aac15bb40 2023-10-16T13:48:50.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.643+0000 7f6aa97fa700 1 -- 172.21.15.16:0/2102190964 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v148) v1 ==== 104+0+0 (secure 0 0 0) 0x7f6aa4059b10 con 0x7f6aac15bb40 2023-10-16T13:48:50.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.646+0000 7f6ab3237700 1 -- 172.21.15.16:0/2102190964 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f6a9006ad50 msgr2=0x7f6a9006d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:50.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.646+0000 7f6ab3237700 1 --2- 172.21.15.16:0/2102190964 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f6a9006ad50 0x7f6a9006d210 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f6aa0009b90 tx=0x7f6aa0008040 comp rx=0 tx=0).stop 2023-10-16T13:48:50.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.646+0000 7f6ab3237700 1 -- 172.21.15.16:0/2102190964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6aac15bb40 msgr2=0x7f6aac154ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:50.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.646+0000 7f6ab3237700 1 --2- 172.21.15.16:0/2102190964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6aac15bb40 0x7f6aac154ea0 secure :-1 s=READY pgs=3123 cs=0 l=1 rev1=1 crypto rx=0x7f6aa4014040 tx=0x7f6aa4004ad0 comp rx=0 tx=0).stop 2023-10-16T13:48:50.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.646+0000 7f6ab3237700 1 -- 172.21.15.16:0/2102190964 shutdown_connections 2023-10-16T13:48:50.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.646+0000 7f6ab3237700 1 --2- 172.21.15.16:0/2102190964 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f6a9006ad50 0x7f6a9006d210 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:50.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.646+0000 7f6ab3237700 1 --2- 172.21.15.16:0/2102190964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6aac15bb40 0x7f6aac154ea0 unknown :-1 s=CLOSED pgs=3123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:50.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.646+0000 7f6ab3237700 1 --2- 172.21.15.16:0/2102190964 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6aac072b30 0x7f6aac1585b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:50.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.646+0000 7f6ab3237700 1 --2- 172.21.15.16:0/2102190964 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6aac0caa40 0x7f6aac154960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:50.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.646+0000 7f6ab3237700 1 -- 172.21.15.16:0/2102190964 >> 172.21.15.16:0/2102190964 conn(0x7f6aac0b1f40 msgr2=0x7f6aac0cbf40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:50.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.646+0000 7f6ab3237700 1 -- 172.21.15.16:0/2102190964 shutdown_connections 2023-10-16T13:48:50.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.646+0000 7f6ab3237700 1 -- 172.21.15.16:0/2102190964 wait complete. 2023-10-16T13:48:50.665 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-16T13:48:50.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.795+0000 7f43bad25700 1 Processor -- start 2023-10-16T13:48:50.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.796+0000 7f43bad25700 1 -- start start 2023-10-16T13:48:50.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.796+0000 7f43bad25700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f43b406d980 0x7f43b415efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:50.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.796+0000 7f43bad25700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43b40cdeb0 con 0x7f43b406a7e0 2023-10-16T13:48:50.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.796+0000 7f43bad25700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43b41613e0 con 0x7f43b4160e60 2023-10-16T13:48:50.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.796+0000 7f43bad25700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43b41582d0 con 0x7f43b406d980 2023-10-16T13:48:50.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.796+0000 7f43b92c2700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f43b406a7e0 0x7f43b4161870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:47930/0 (socket says 172.21.15.16:47930) 2023-10-16T13:48:50.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.797+0000 7f43b92c2700 1 -- 172.21.15.16:0/1971435611 learned_addr learned my addr 172.21.15.16:0/1971435611 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:48:50.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.797+0000 7f43b3fff700 1 --2- 172.21.15.16:0/1971435611 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f43b406d980 0x7f43b415efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:50.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.797+0000 7f43b37fe700 1 -- 172.21.15.16:0/1971435611 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2851579160 0 0) 0x7f43b40cdeb0 con 0x7f43b406a7e0 2023-10-16T13:48:50.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.797+0000 7f43b37fe700 1 -- 172.21.15.16:0/1971435611 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43a40035b0 con 0x7f43b406a7e0 2023-10-16T13:48:50.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.797+0000 7f43b37fe700 1 -- 172.21.15.16:0/1971435611 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 801770803 0 0) 0x7f43b41613e0 con 0x7f43b4160e60 2023-10-16T13:48:50.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.797+0000 7f43b37fe700 1 -- 172.21.15.16:0/1971435611 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43b40cdeb0 con 0x7f43b4160e60 2023-10-16T13:48:50.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.797+0000 7f43b37fe700 1 -- 172.21.15.16:0/1971435611 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2059516052 0 0) 0x7f43a40035b0 con 0x7f43b406a7e0 2023-10-16T13:48:50.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.797+0000 7f43b37fe700 1 -- 172.21.15.16:0/1971435611 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f43b406d980 msgr2=0x7f43b415efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:50.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.797+0000 7f43b37fe700 1 --2- 172.21.15.16:0/1971435611 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f43b406d980 0x7f43b415efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:50.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.797+0000 7f43b37fe700 1 -- 172.21.15.16:0/1971435611 >> v1:172.21.15.49:6789/0 conn(0x7f43b4160e60 legacy=0x7f43b415e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:50.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.797+0000 7f43b37fe700 1 -- 172.21.15.16:0/1971435611 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43b40ce100 con 0x7f43b406a7e0 2023-10-16T13:48:50.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.797+0000 7f43b37fe700 1 -- 172.21.15.16:0/1971435611 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f43ac0019f0 con 0x7f43b406a7e0 2023-10-16T13:48:50.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.797+0000 7f43b37fe700 1 -- 172.21.15.16:0/1971435611 >> v1:172.21.15.16:6789/0 conn(0x7f43b406a7e0 legacy=0x7f43b4161870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:50.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.797+0000 7f43b37fe700 1 --2- 172.21.15.16:0/1971435611 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43a4003bc0 0x7f43a4003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:50.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.797+0000 7f43b37fe700 1 --2- 172.21.15.16:0/1971435611 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f43b406d980 0x7f43a4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:50.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.797+0000 7f43b37fe700 1 --2- 172.21.15.16:0/1971435611 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f43a4004d90 0x7f43a4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:50.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.797+0000 7f43b37fe700 1 -- 172.21.15.16:0/1971435611 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43b41582d0 con 0x7f43a4003bc0 2023-10-16T13:48:50.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.797+0000 7f43b37fe700 1 -- 172.21.15.16:0/1971435611 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43a400bac0 con 0x7f43a4004d90 2023-10-16T13:48:50.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.797+0000 7f43b37fe700 1 -- 172.21.15.16:0/1971435611 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43a400bc80 con 0x7f43b406d980 2023-10-16T13:48:50.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.797+0000 7f43b8ac1700 1 --2- 172.21.15.16:0/1971435611 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43a4003bc0 0x7f43a4003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:50.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.797+0000 7f43b3fff700 1 --2- 172.21.15.16:0/1971435611 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f43b406d980 0x7f43a4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:50.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.797+0000 7f43b92c2700 1 --2- 172.21.15.16:0/1971435611 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f43a4004d90 0x7f43a4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:50.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.798+0000 7f43b8ac1700 1 -- 172.21.15.16:0/1971435611 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f43b406d980 msgr2=0x7f43a4004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:50.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.798+0000 7f43b8ac1700 1 --2- 172.21.15.16:0/1971435611 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f43b406d980 0x7f43a4004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:50.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.798+0000 7f43b8ac1700 1 -- 172.21.15.16:0/1971435611 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f43a4004d90 msgr2=0x7f43a4005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:50.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.798+0000 7f43b8ac1700 1 --2- 172.21.15.16:0/1971435611 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f43a4004d90 0x7f43a4005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:50.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.798+0000 7f43b8ac1700 1 -- 172.21.15.16:0/1971435611 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43a40035b0 con 0x7f43a4003bc0 2023-10-16T13:48:50.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.798+0000 7f43b92c2700 1 --2- 172.21.15.16:0/1971435611 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f43a4004d90 0x7f43a4005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:48:50.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.798+0000 7f43b8ac1700 1 --2- 172.21.15.16:0/1971435611 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43a4003bc0 0x7f43a4003fc0 secure :-1 s=READY pgs=3127 cs=0 l=1 rev1=1 crypto rx=0x7f43a8001940 tx=0x7f43a4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3aa286c09df0763c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:50.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.798+0000 7f43b37fe700 1 -- 172.21.15.16:0/1971435611 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a8001430 con 0x7f43a4003bc0 2023-10-16T13:48:50.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.798+0000 7f43b37fe700 1 -- 172.21.15.16:0/1971435611 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43a8001d70 con 0x7f43a4003bc0 2023-10-16T13:48:50.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.798+0000 7f43b37fe700 1 -- 172.21.15.16:0/1971435611 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a8005510 con 0x7f43a4003bc0 2023-10-16T13:48:50.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.798+0000 7f43bad25700 1 -- 172.21.15.16:0/1971435611 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43a4003bc0 msgr2=0x7f43a4003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:50.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.798+0000 7f43bad25700 1 --2- 172.21.15.16:0/1971435611 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43a4003bc0 0x7f43a4003fc0 secure :-1 s=READY pgs=3127 cs=0 l=1 rev1=1 crypto rx=0x7f43a8001940 tx=0x7f43a4005880 comp rx=0 tx=0).stop 2023-10-16T13:48:50.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.798+0000 7f43b92c2700 1 -- 172.21.15.16:0/1971435611 reap_dead start 2023-10-16T13:48:50.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.798+0000 7f43bad25700 1 -- 172.21.15.16:0/1971435611 shutdown_connections 2023-10-16T13:48:50.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.798+0000 7f43bad25700 1 -- 172.21.15.16:0/1971435611 >> 172.21.15.16:0/1971435611 conn(0x7f43b406f430 msgr2=0x7f43b406f810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:50.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.799+0000 7f43bad25700 1 -- 172.21.15.16:0/1971435611 shutdown_connections 2023-10-16T13:48:50.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.799+0000 7f43bad25700 1 -- 172.21.15.16:0/1971435611 wait complete. 2023-10-16T13:48:50.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.800+0000 7f43bad25700 1 Processor -- start 2023-10-16T13:48:50.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.800+0000 7f43bad25700 1 -- start start 2023-10-16T13:48:50.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.801+0000 7f43bad25700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f43a4003bc0 0x7f43b414c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:50.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.801+0000 7f43bad25700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43b40acd70 0x7f43b40ad1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:50.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.801+0000 7f43bad25700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f43b40b0220 0x7f43b40b06d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:50.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.801+0000 7f43bad25700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43b40b0c10 con 0x7f43b40acd70 2023-10-16T13:48:50.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.801+0000 7f43bad25700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43b414ce10 con 0x7f43b40b0220 2023-10-16T13:48:50.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.801+0000 7f43bad25700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43b40ad810 con 0x7f43a4003bc0 2023-10-16T13:48:50.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.801+0000 7f43b3fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43b40acd70 0x7f43b40ad1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:50.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.801+0000 7f43b3fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43b40acd70 0x7f43b40ad1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58666/0 (socket says 172.21.15.16:58666) 2023-10-16T13:48:50.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.801+0000 7f43b3fff700 1 -- 172.21.15.16:0/3752040348 learned_addr learned my addr 172.21.15.16:0/3752040348 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:50.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.801+0000 7f43b92c2700 1 --2- 172.21.15.16:0/3752040348 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f43b40b0220 0x7f43b40b06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:50.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.801+0000 7f43b8ac1700 1 --2- 172.21.15.16:0/3752040348 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f43a4003bc0 0x7f43b414c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:50.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.801+0000 7f43b3fff700 1 -- 172.21.15.16:0/3752040348 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f43a4003bc0 msgr2=0x7f43b414c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:50.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.801+0000 7f43b3fff700 1 --2- 172.21.15.16:0/3752040348 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f43a4003bc0 0x7f43b414c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:50.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.801+0000 7f43b3fff700 1 -- 172.21.15.16:0/3752040348 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f43b40b0220 msgr2=0x7f43b40b06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:50.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.801+0000 7f43b3fff700 1 --2- 172.21.15.16:0/3752040348 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f43b40b0220 0x7f43b40b06d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:50.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.801+0000 7f43b3fff700 1 -- 172.21.15.16:0/3752040348 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43b40adaf0 con 0x7f43b40acd70 2023-10-16T13:48:50.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.801+0000 7f43b8ac1700 1 --2- 172.21.15.16:0/3752040348 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f43a4003bc0 0x7f43b414c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:48:50.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.801+0000 7f43b3fff700 1 --2- 172.21.15.16:0/3752040348 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43b40acd70 0x7f43b40ad1f0 secure :-1 s=READY pgs=3128 cs=0 l=1 rev1=1 crypto rx=0x7f43a0011aa0 tx=0x7f43a0011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:50.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.801+0000 7f43b17fa700 1 -- 172.21.15.16:0/3752040348 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a000f8c0 con 0x7f43b40acd70 2023-10-16T13:48:50.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.801+0000 7f43b17fa700 1 -- 172.21.15.16:0/3752040348 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43a000fa20 con 0x7f43b40acd70 2023-10-16T13:48:50.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.801+0000 7f43b17fa700 1 -- 172.21.15.16:0/3752040348 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a000fdd0 con 0x7f43b40acd70 2023-10-16T13:48:50.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.801+0000 7f43bad25700 1 -- 172.21.15.16:0/3752040348 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43b40addd0 con 0x7f43b40acd70 2023-10-16T13:48:50.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.802+0000 7f43bad25700 1 -- 172.21.15.16:0/3752040348 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43b40b4310 con 0x7f43b40acd70 2023-10-16T13:48:50.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.802+0000 7f43bad25700 1 -- 172.21.15.16:0/3752040348 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43b404ec20 con 0x7f43b40acd70 2023-10-16T13:48:50.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.803+0000 7f43b17fa700 1 -- 172.21.15.16:0/3752040348 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 164) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f43a0014440 con 0x7f43b40acd70 2023-10-16T13:48:50.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.803+0000 7f43b17fa700 1 --2- 172.21.15.16:0/3752040348 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f439806ad50 0x7f439806d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:50.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.804+0000 7f43b8ac1700 1 --2- 172.21.15.16:0/3752040348 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f439806ad50 0x7f439806d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:50.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.804+0000 7f43b17fa700 1 -- 172.21.15.16:0/3752040348 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(268..268 src has 1..268) v4 ==== 7017+0+0 (secure 0 0 0) 0x7f43a0089df0 con 0x7f43b40acd70 2023-10-16T13:48:50.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.804+0000 7f43b8ac1700 1 --2- 172.21.15.16:0/3752040348 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f439806ad50 0x7f439806d210 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f43a8015040 tx=0x7f43a800a000 comp rx=0 tx=0).ready entity=mgr.13344 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:50.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:50.808+0000 7f43b17fa700 1 -- 172.21.15.16:0/3752040348 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f43a0054d80 con 0x7f43b40acd70 2023-10-16T13:48:51.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.011+0000 7f43bad25700 1 -- 172.21.15.16:0/3752040348 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f43b4157440 con 0x7f43b40acd70 2023-10-16T13:48:51.593 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.590+0000 7f43b17fa700 1 -- 172.21.15.16:0/3752040348 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 165) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f43a008e070 con 0x7f43b40acd70 2023-10-16T13:48:51.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.645+0000 7f43b17fa700 1 -- 172.21.15.16:0/3752040348 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v269) v1 ==== 181+0+0 (secure 0 0 0) 0x7f43a0041340 con 0x7f43b40acd70 2023-10-16T13:48:51.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.650+0000 7f43bad25700 1 -- 172.21.15.16:0/3752040348 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f439806ad50 msgr2=0x7f439806d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:51.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.650+0000 7f43bad25700 1 --2- 172.21.15.16:0/3752040348 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f439806ad50 0x7f439806d210 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f43a8015040 tx=0x7f43a800a000 comp rx=0 tx=0).stop 2023-10-16T13:48:51.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.650+0000 7f43bad25700 1 -- 172.21.15.16:0/3752040348 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43b40acd70 msgr2=0x7f43b40ad1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:51.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.650+0000 7f43bad25700 1 --2- 172.21.15.16:0/3752040348 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43b40acd70 0x7f43b40ad1f0 secure :-1 s=READY pgs=3128 cs=0 l=1 rev1=1 crypto rx=0x7f43a0011aa0 tx=0x7f43a0011e60 comp rx=0 tx=0).stop 2023-10-16T13:48:51.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.651+0000 7f43bad25700 1 -- 172.21.15.16:0/3752040348 shutdown_connections 2023-10-16T13:48:51.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.651+0000 7f43bad25700 1 --2- 172.21.15.16:0/3752040348 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f43b40b0220 0x7f43b40b06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:51.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.651+0000 7f43bad25700 1 --2- 172.21.15.16:0/3752040348 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f43a4003bc0 0x7f43b414c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:51.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.651+0000 7f43bad25700 1 --2- 172.21.15.16:0/3752040348 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f439806ad50 0x7f439806d210 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:51.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.651+0000 7f43bad25700 1 --2- 172.21.15.16:0/3752040348 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43b40acd70 0x7f43b40ad1f0 unknown :-1 s=CLOSED pgs=3128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:51.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.651+0000 7f43bad25700 1 -- 172.21.15.16:0/3752040348 >> 172.21.15.16:0/3752040348 conn(0x7f43b406f430 msgr2=0x7f43b41604c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:51.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.651+0000 7f43bad25700 1 -- 172.21.15.16:0/3752040348 shutdown_connections 2023-10-16T13:48:51.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.651+0000 7f43bad25700 1 -- 172.21.15.16:0/3752040348 wait complete. 2023-10-16T13:48:51.657 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' removed 2023-10-16T13:48:51.671 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-16T13:48:51.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.803+0000 7f57c4fa0700 1 Processor -- start 2023-10-16T13:48:51.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.803+0000 7f57c4fa0700 1 -- start start 2023-10-16T13:48:51.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.804+0000 7f57c4fa0700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f57c00cb040 0x7f57c015fab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:51.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.804+0000 7f57c4fa0700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57c006b220 con 0x7f57c006e590 2023-10-16T13:48:51.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.804+0000 7f57c4fa0700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57c006a980 con 0x7f57c015b500 2023-10-16T13:48:51.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.804+0000 7f57c4fa0700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57c0160090 con 0x7f57c00cb040 2023-10-16T13:48:51.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.804+0000 7f57bed9d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f57c006e590 0x7f57c00ccc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:47950/0 (socket says 172.21.15.16:47950) 2023-10-16T13:48:51.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.804+0000 7f57bed9d700 1 -- 172.21.15.16:0/4266158563 learned_addr learned my addr 172.21.15.16:0/4266158563 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:48:51.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.804+0000 7f57bdd9b700 1 --2- 172.21.15.16:0/4266158563 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f57c00cb040 0x7f57c015fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:51.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.804+0000 7f57bd59a700 1 -- 172.21.15.16:0/4266158563 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3534410489 0 0) 0x7f57c006b220 con 0x7f57c006e590 2023-10-16T13:48:51.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.804+0000 7f57bd59a700 1 -- 172.21.15.16:0/4266158563 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57a80035b0 con 0x7f57c006e590 2023-10-16T13:48:51.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.804+0000 7f57bd59a700 1 -- 172.21.15.16:0/4266158563 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 106932495 0 0) 0x7f57c006a980 con 0x7f57c015b500 2023-10-16T13:48:51.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.804+0000 7f57bd59a700 1 -- 172.21.15.16:0/4266158563 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57c006b220 con 0x7f57c015b500 2023-10-16T13:48:51.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.804+0000 7f57bd59a700 1 -- 172.21.15.16:0/4266158563 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3505016245 0 0) 0x7f57a80035b0 con 0x7f57c006e590 2023-10-16T13:48:51.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.805+0000 7f57bd59a700 1 -- 172.21.15.16:0/4266158563 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f57c00cb040 msgr2=0x7f57c015fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:51.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.805+0000 7f57bd59a700 1 --2- 172.21.15.16:0/4266158563 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f57c00cb040 0x7f57c015fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:51.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.805+0000 7f57bd59a700 1 -- 172.21.15.16:0/4266158563 >> v1:172.21.15.49:6789/0 conn(0x7f57c015b500 legacy=0x7f57c00ca930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:51.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.805+0000 7f57bd59a700 1 -- 172.21.15.16:0/4266158563 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57c006ac60 con 0x7f57c006e590 2023-10-16T13:48:51.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.805+0000 7f57bd59a700 1 -- 172.21.15.16:0/4266158563 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f57b80019f0 con 0x7f57c006e590 2023-10-16T13:48:51.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.805+0000 7f57bd59a700 1 -- 172.21.15.16:0/4266158563 >> v1:172.21.15.16:6789/0 conn(0x7f57c006e590 legacy=0x7f57c00ccc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:51.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.805+0000 7f57bd59a700 1 --2- 172.21.15.16:0/4266158563 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57a8003bc0 0x7f57a8003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:51.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.805+0000 7f57bd59a700 1 --2- 172.21.15.16:0/4266158563 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f57a8004850 0x7f57a8004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:51.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.805+0000 7f57bd59a700 1 --2- 172.21.15.16:0/4266158563 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f57c00cb040 0x7f57a8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:51.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.805+0000 7f57bd59a700 1 -- 172.21.15.16:0/4266158563 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57c0160090 con 0x7f57a8003bc0 2023-10-16T13:48:51.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.805+0000 7f57bd59a700 1 -- 172.21.15.16:0/4266158563 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57a800bac0 con 0x7f57a8004850 2023-10-16T13:48:51.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.805+0000 7f57bd59a700 1 -- 172.21.15.16:0/4266158563 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57a800bc50 con 0x7f57c00cb040 2023-10-16T13:48:51.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.805+0000 7f57be59c700 1 --2- 172.21.15.16:0/4266158563 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57a8003bc0 0x7f57a8003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:51.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.805+0000 7f57bdd9b700 1 --2- 172.21.15.16:0/4266158563 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f57a8004850 0x7f57a8004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:51.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.805+0000 7f57bed9d700 1 --2- 172.21.15.16:0/4266158563 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f57c00cb040 0x7f57a8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:51.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.805+0000 7f57be59c700 1 -- 172.21.15.16:0/4266158563 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f57c00cb040 msgr2=0x7f57a8005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:51.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.805+0000 7f57be59c700 1 --2- 172.21.15.16:0/4266158563 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f57c00cb040 0x7f57a8005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:51.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.805+0000 7f57be59c700 1 -- 172.21.15.16:0/4266158563 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f57a8004850 msgr2=0x7f57a8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:51.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.805+0000 7f57be59c700 1 --2- 172.21.15.16:0/4266158563 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f57a8004850 0x7f57a8004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:51.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.805+0000 7f57be59c700 1 -- 172.21.15.16:0/4266158563 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57a80035b0 con 0x7f57a8003bc0 2023-10-16T13:48:51.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.805+0000 7f57be59c700 1 --2- 172.21.15.16:0/4266158563 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57a8003bc0 0x7f57a8003fc0 secure :-1 s=READY pgs=3130 cs=0 l=1 rev1=1 crypto rx=0x7f57b4001940 tx=0x7f57a8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e17d98317332ca98 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:51.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.805+0000 7f57bd59a700 1 -- 172.21.15.16:0/4266158563 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b4001430 con 0x7f57a8003bc0 2023-10-16T13:48:51.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.805+0000 7f57bd59a700 1 -- 172.21.15.16:0/4266158563 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57b4001d70 con 0x7f57a8003bc0 2023-10-16T13:48:51.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.805+0000 7f57bd59a700 1 -- 172.21.15.16:0/4266158563 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b4005510 con 0x7f57a8003bc0 2023-10-16T13:48:51.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.806+0000 7f57c4fa0700 1 -- 172.21.15.16:0/4266158563 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57a8003bc0 msgr2=0x7f57a8003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:51.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.806+0000 7f57c4fa0700 1 --2- 172.21.15.16:0/4266158563 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57a8003bc0 0x7f57a8003fc0 secure :-1 s=READY pgs=3130 cs=0 l=1 rev1=1 crypto rx=0x7f57b4001940 tx=0x7f57a8005880 comp rx=0 tx=0).stop 2023-10-16T13:48:51.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.806+0000 7f57bed9d700 1 -- 172.21.15.16:0/4266158563 reap_dead start 2023-10-16T13:48:51.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.806+0000 7f57c4fa0700 1 -- 172.21.15.16:0/4266158563 shutdown_connections 2023-10-16T13:48:51.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.806+0000 7f57c4fa0700 1 -- 172.21.15.16:0/4266158563 >> 172.21.15.16:0/4266158563 conn(0x7f57c006f430 msgr2=0x7f57c006f810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:51.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.806+0000 7f57c4fa0700 1 -- 172.21.15.16:0/4266158563 shutdown_connections 2023-10-16T13:48:51.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.806+0000 7f57c4fa0700 1 -- 172.21.15.16:0/4266158563 wait complete. 2023-10-16T13:48:51.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.807+0000 7f57c4fa0700 1 Processor -- start 2023-10-16T13:48:51.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.807+0000 7f57c4fa0700 1 -- start start 2023-10-16T13:48:51.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.808+0000 7f57c4fa0700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f57a8003bc0 0x7f57c015de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:51.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.808+0000 7f57c4fa0700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f57c00ced90 0x7f57c0158e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:51.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.808+0000 7f57c4fa0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57c00ad380 0x7f57c00ad830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:51.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.808+0000 7f57c4fa0700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57c00add70 con 0x7f57c00ad380 2023-10-16T13:48:51.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.808+0000 7f57c4fa0700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57c00c3df0 con 0x7f57c00ced90 2023-10-16T13:48:51.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.808+0000 7f57c4fa0700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57c00cf350 con 0x7f57a8003bc0 2023-10-16T13:48:51.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.808+0000 7f57bed9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57c00ad380 0x7f57c00ad830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:51.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.808+0000 7f57bed9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57c00ad380 0x7f57c00ad830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58688/0 (socket says 172.21.15.16:58688) 2023-10-16T13:48:51.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.808+0000 7f57bed9d700 1 -- 172.21.15.16:0/566718771 learned_addr learned my addr 172.21.15.16:0/566718771 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:51.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.808+0000 7f57bdd9b700 1 --2- 172.21.15.16:0/566718771 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f57c00ced90 0x7f57c0158e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:51.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.809+0000 7f57be59c700 1 --2- 172.21.15.16:0/566718771 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f57a8003bc0 0x7f57c015de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:51.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.809+0000 7f57bed9d700 1 -- 172.21.15.16:0/566718771 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f57a8003bc0 msgr2=0x7f57c015de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:51.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.809+0000 7f57bed9d700 1 --2- 172.21.15.16:0/566718771 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f57a8003bc0 0x7f57c015de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:51.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.809+0000 7f57bed9d700 1 -- 172.21.15.16:0/566718771 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f57c00ced90 msgr2=0x7f57c0158e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:51.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.809+0000 7f57bed9d700 1 --2- 172.21.15.16:0/566718771 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f57c00ced90 0x7f57c0158e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:51.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.809+0000 7f57bed9d700 1 -- 172.21.15.16:0/566718771 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57c00cf630 con 0x7f57c00ad380 2023-10-16T13:48:51.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.809+0000 7f57be59c700 1 --2- 172.21.15.16:0/566718771 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f57a8003bc0 0x7f57c015de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:48:51.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.809+0000 7f57bed9d700 1 --2- 172.21.15.16:0/566718771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57c00ad380 0x7f57c00ad830 secure :-1 s=READY pgs=3131 cs=0 l=1 rev1=1 crypto rx=0x7f57c00ca3f0 tx=0x7f57b800a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:51.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.809+0000 7f57aeffd700 1 -- 172.21.15.16:0/566718771 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b80133f0 con 0x7f57c00ad380 2023-10-16T13:48:51.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.809+0000 7f57aeffd700 1 -- 172.21.15.16:0/566718771 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57b8013590 con 0x7f57c00ad380 2023-10-16T13:48:51.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.809+0000 7f57aeffd700 1 -- 172.21.15.16:0/566718771 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b8013940 con 0x7f57c00ad380 2023-10-16T13:48:51.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.809+0000 7f57c4fa0700 1 -- 172.21.15.16:0/566718771 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57c00cf910 con 0x7f57c00ad380 2023-10-16T13:48:51.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.809+0000 7f57c4fa0700 1 -- 172.21.15.16:0/566718771 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57c015e350 con 0x7f57c00ad380 2023-10-16T13:48:51.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.810+0000 7f57c4fa0700 1 -- 172.21.15.16:0/566718771 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57c00c3f80 con 0x7f57c00ad380 2023-10-16T13:48:51.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.810+0000 7f57aeffd700 1 -- 172.21.15.16:0/566718771 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 165) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f57b8013aa0 con 0x7f57c00ad380 2023-10-16T13:48:51.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.811+0000 7f57aeffd700 1 --2- 172.21.15.16:0/566718771 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f57a006af90 0x7f57a006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:51.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.811+0000 7f57aeffd700 1 -- 172.21.15.16:0/566718771 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 6579+0+0 (secure 0 0 0) 0x7f57b808a690 con 0x7f57c00ad380 2023-10-16T13:48:51.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.811+0000 7f57be59c700 1 --2- 172.21.15.16:0/566718771 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f57a006af90 0x7f57a006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:51.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.811+0000 7f57be59c700 1 --2- 172.21.15.16:0/566718771 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f57a006af90 0x7f57a006d450 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f57b4015040 tx=0x7f57b400a000 comp rx=0 tx=0).ready entity=mgr.13344 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:51.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:51.815+0000 7f57aeffd700 1 -- 172.21.15.16:0/566718771 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f57c00c3f80 con 0x7f57c00ad380 2023-10-16T13:48:52.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:52.017+0000 7f57c4fa0700 1 -- 172.21.15.16:0/566718771 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f57c00c3f80 con 0x7f57c00ad380 2023-10-16T13:48:52.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:52.646+0000 7f57aeffd700 1 -- 172.21.15.16:0/566718771 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v270) v1 ==== 169+0+0 (secure 0 0 0) 0x7f57c00c3f80 con 0x7f57c00ad380 2023-10-16T13:48:52.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:52.650+0000 7f57c4fa0700 1 -- 172.21.15.16:0/566718771 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f57a006af90 msgr2=0x7f57a006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:52.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:52.650+0000 7f57c4fa0700 1 --2- 172.21.15.16:0/566718771 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f57a006af90 0x7f57a006d450 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f57b4015040 tx=0x7f57b400a000 comp rx=0 tx=0).stop 2023-10-16T13:48:52.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:52.650+0000 7f57c4fa0700 1 -- 172.21.15.16:0/566718771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57c00ad380 msgr2=0x7f57c00ad830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:52.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:52.650+0000 7f57c4fa0700 1 --2- 172.21.15.16:0/566718771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57c00ad380 0x7f57c00ad830 secure :-1 s=READY pgs=3131 cs=0 l=1 rev1=1 crypto rx=0x7f57c00ca3f0 tx=0x7f57b800a660 comp rx=0 tx=0).stop 2023-10-16T13:48:52.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:52.650+0000 7f57c4fa0700 1 -- 172.21.15.16:0/566718771 shutdown_connections 2023-10-16T13:48:52.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:52.650+0000 7f57c4fa0700 1 --2- 172.21.15.16:0/566718771 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f57a006af90 0x7f57a006d450 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:52.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:52.650+0000 7f57c4fa0700 1 --2- 172.21.15.16:0/566718771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57c00ad380 0x7f57c00ad830 unknown :-1 s=CLOSED pgs=3131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:52.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:52.650+0000 7f57c4fa0700 1 --2- 172.21.15.16:0/566718771 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f57a8003bc0 0x7f57c015de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:52.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:52.650+0000 7f57c4fa0700 1 --2- 172.21.15.16:0/566718771 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f57c00ced90 0x7f57c0158e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:52.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:52.650+0000 7f57c4fa0700 1 -- 172.21.15.16:0/566718771 >> 172.21.15.16:0/566718771 conn(0x7f57c006f430 msgr2=0x7f57c006f810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:52.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:52.650+0000 7f57c4fa0700 1 -- 172.21.15.16:0/566718771 shutdown_connections 2023-10-16T13:48:52.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:52.650+0000 7f57c4fa0700 1 -- 172.21.15.16:0/566718771 wait complete. 2023-10-16T13:48:52.656 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' removed 2023-10-16T13:48:52.669 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-16T13:48:52.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:52.795+0000 7fbc4f073700 1 Processor -- start 2023-10-16T13:48:52.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:52.795+0000 7fbc4f073700 1 -- start start 2023-10-16T13:48:52.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:52.796+0000 7fbc4f073700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbc48160fe0 0x7fbc480ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:52.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:52.796+0000 7fbc4f073700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc481613c0 con 0x7fbc4806da30 2023-10-16T13:48:52.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:52.796+0000 7fbc4f073700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc480adb00 con 0x7fbc4806aef0 2023-10-16T13:48:52.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:52.796+0000 7fbc4f073700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc480cf680 con 0x7fbc48160fe0 2023-10-16T13:48:52.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:52.796+0000 7fbc4ce0f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbc48160fe0 0x7fbc480ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:52.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:52.796+0000 7fbc47fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fbc4806da30 0x7fbc4806a7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:47968/0 (socket says 172.21.15.16:47968) 2023-10-16T13:48:52.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:52.796+0000 7fbc47fff700 1 -- 172.21.15.16:0/1824853698 learned_addr learned my addr 172.21.15.16:0/1824853698 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:48:52.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:52.797+0000 7fbc477fe700 1 -- 172.21.15.16:0/1824853698 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3660101615 0 0) 0x7fbc34001110 con 0x7fbc4806da30 2023-10-16T13:48:52.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:52.797+0000 7fbc477fe700 1 -- 172.21.15.16:0/1824853698 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc380035b0 con 0x7fbc4806da30 2023-10-16T13:48:52.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:52.797+0000 7fbc4ce0f700 1 -- 172.21.15.16:0/1824853698 >> v1:172.21.15.49:6789/0 conn(0x7fbc4806aef0 legacy=0x7fbc480adf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:52.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:52.797+0000 7fbc4ce0f700 1 -- 172.21.15.16:0/1824853698 >> v1:172.21.15.16:6789/0 conn(0x7fbc4806da30 legacy=0x7fbc4806a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:52.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:52.797+0000 7fbc4ce0f700 1 -- 172.21.15.16:0/1824853698 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc480acef0 con 0x7fbc48160fe0 2023-10-16T13:48:52.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:52.797+0000 7fbc4ce0f700 1 --2- 172.21.15.16:0/1824853698 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc48160fe0 0x7fbc480ac8d0 secure :-1 s=READY pgs=3133 cs=0 l=1 rev1=1 crypto rx=0x7fbc3c009820 tx=0x7fbc3c009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d5fbf9edcbe1d85 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:52.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:52.797+0000 7fbc477fe700 1 -- 172.21.15.16:0/1824853698 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc3c005e30 con 0x7fbc48160fe0 2023-10-16T13:48:52.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:52.797+0000 7fbc477fe700 1 -- 172.21.15.16:0/1824853698 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc3c006030 con 0x7fbc48160fe0 2023-10-16T13:48:52.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:52.797+0000 7fbc477fe700 1 -- 172.21.15.16:0/1824853698 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc3c0063c0 con 0x7fbc48160fe0 2023-10-16T13:48:52.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:52.798+0000 7fbc4f073700 1 -- 172.21.15.16:0/1824853698 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc48160fe0 msgr2=0x7fbc480ac8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:52.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:52.798+0000 7fbc4f073700 1 --2- 172.21.15.16:0/1824853698 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc48160fe0 0x7fbc480ac8d0 secure :-1 s=READY pgs=3133 cs=0 l=1 rev1=1 crypto rx=0x7fbc3c009820 tx=0x7fbc3c009b30 comp rx=0 tx=0).stop 2023-10-16T13:48:52.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:52.798+0000 7fbc4f073700 1 -- 172.21.15.16:0/1824853698 shutdown_connections 2023-10-16T13:48:52.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:52.798+0000 7fbc4f073700 1 --2- 172.21.15.16:0/1824853698 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc48160fe0 0x7fbc480ac8d0 unknown :-1 s=CLOSED pgs=3133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:52.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:52.798+0000 7fbc4f073700 1 -- 172.21.15.16:0/1824853698 >> 172.21.15.16:0/1824853698 conn(0x7fbc480b1ed0 msgr2=0x7fbc480b22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:52.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:52.798+0000 7fbc4f073700 1 -- 172.21.15.16:0/1824853698 shutdown_connections 2023-10-16T13:48:52.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:52.798+0000 7fbc4f073700 1 -- 172.21.15.16:0/1824853698 wait complete. 2023-10-16T13:48:52.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:52.799+0000 7fbc4f073700 1 Processor -- start 2023-10-16T13:48:52.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:52.799+0000 7fbc4f073700 1 -- start start 2023-10-16T13:48:52.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:52.800+0000 7fbc4f073700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc4806aef0 0x7fbc481560e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:52.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:52.800+0000 7fbc4f073700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbc4806da30 0x7fbc48156970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:52.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:52.800+0000 7fbc4f073700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbc48160fe0 0x7fbc48156eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:52.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:52.800+0000 7fbc4f073700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc480b5960 con 0x7fbc4806aef0 2023-10-16T13:48:52.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:52.800+0000 7fbc4f073700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc480b48c0 con 0x7fbc48160fe0 2023-10-16T13:48:52.809 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:52.800+0000 7fbc4f073700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc480b4ba0 con 0x7fbc4806da30 2023-10-16T13:48:52.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:52.800+0000 7fbc4ce0f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc4806aef0 0x7fbc481560e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:52.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:52.800+0000 7fbc4ce0f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc4806aef0 0x7fbc481560e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58698/0 (socket says 172.21.15.16:58698) 2023-10-16T13:48:52.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:52.800+0000 7fbc4ce0f700 1 -- 172.21.15.16:0/363508640 learned_addr learned my addr 172.21.15.16:0/363508640 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:52.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:52.800+0000 7fbc4d610700 1 --2- 172.21.15.16:0/363508640 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbc48160fe0 0x7fbc48156eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:52.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:52.800+0000 7fbc47fff700 1 --2- 172.21.15.16:0/363508640 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbc4806da30 0x7fbc48156970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:52.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:52.800+0000 7fbc4ce0f700 1 -- 172.21.15.16:0/363508640 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbc4806da30 msgr2=0x7fbc48156970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:52.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:52.800+0000 7fbc4ce0f700 1 --2- 172.21.15.16:0/363508640 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbc4806da30 0x7fbc48156970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:52.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:52.800+0000 7fbc4ce0f700 1 -- 172.21.15.16:0/363508640 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbc48160fe0 msgr2=0x7fbc48156eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:52.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:52.800+0000 7fbc4ce0f700 1 --2- 172.21.15.16:0/363508640 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbc48160fe0 0x7fbc48156eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:52.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:52.800+0000 7fbc4ce0f700 1 -- 172.21.15.16:0/363508640 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc3c009430 con 0x7fbc4806aef0 2023-10-16T13:48:52.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:52.800+0000 7fbc4ce0f700 1 --2- 172.21.15.16:0/363508640 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc4806aef0 0x7fbc481560e0 secure :-1 s=READY pgs=3134 cs=0 l=1 rev1=1 crypto rx=0x7fbc3c009fd0 tx=0x7fbc3c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:52.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:52.800+0000 7fbc457fa700 1 -- 172.21.15.16:0/363508640 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc3c004240 con 0x7fbc4806aef0 2023-10-16T13:48:52.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:52.800+0000 7fbc4f073700 1 -- 172.21.15.16:0/363508640 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc480b4e80 con 0x7fbc4806aef0 2023-10-16T13:48:52.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:52.801+0000 7fbc4f073700 1 -- 172.21.15.16:0/363508640 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc480b5440 con 0x7fbc4806aef0 2023-10-16T13:48:52.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:52.801+0000 7fbc457fa700 1 -- 172.21.15.16:0/363508640 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc3c0043a0 con 0x7fbc4806aef0 2023-10-16T13:48:52.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:52.801+0000 7fbc457fa700 1 -- 172.21.15.16:0/363508640 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc3c0036a0 con 0x7fbc4806aef0 2023-10-16T13:48:52.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:52.802+0000 7fbc4f073700 1 -- 172.21.15.16:0/363508640 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc280052f0 con 0x7fbc4806aef0 2023-10-16T13:48:52.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:52.802+0000 7fbc457fa700 1 -- 172.21.15.16:0/363508640 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 165) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fbc3c022070 con 0x7fbc4806aef0 2023-10-16T13:48:52.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:52.802+0000 7fbc457fa700 1 --2- 172.21.15.16:0/363508640 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fbc2c06af90 0x7fbc2c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:52.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:52.802+0000 7fbc457fa700 1 -- 172.21.15.16:0/363508640 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 6165+0+0 (secure 0 0 0) 0x7fbc3c08b040 con 0x7fbc4806aef0 2023-10-16T13:48:52.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:52.802+0000 7fbc47fff700 1 --2- 172.21.15.16:0/363508640 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fbc2c06af90 0x7fbc2c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:52.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:52.803+0000 7fbc47fff700 1 --2- 172.21.15.16:0/363508640 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fbc2c06af90 0x7fbc2c06d450 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fbc481f4730 tx=0x7fbc34009040 comp rx=0 tx=0).ready entity=mgr.13344 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:52.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:52.807+0000 7fbc457fa700 1 -- 172.21.15.16:0/363508640 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbc3c055eb0 con 0x7fbc4806aef0 2023-10-16T13:48:53.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.012+0000 7fbc4f073700 1 -- 172.21.15.16:0/363508640 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7fbc28005640 con 0x7fbc4806aef0 2023-10-16T13:48:53.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.016+0000 7fbc457fa700 1 -- 172.21.15.16:0/363508640 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v91) v1 ==== 115+0+0 (secure 0 0 0) 0x7fbc3c0594d0 con 0x7fbc4806aef0 2023-10-16T13:48:53.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.018+0000 7fbc4f073700 1 -- 172.21.15.16:0/363508640 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fbc2c06af90 msgr2=0x7fbc2c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:53.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.018+0000 7fbc4f073700 1 --2- 172.21.15.16:0/363508640 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fbc2c06af90 0x7fbc2c06d450 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fbc481f4730 tx=0x7fbc34009040 comp rx=0 tx=0).stop 2023-10-16T13:48:53.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.018+0000 7fbc4f073700 1 -- 172.21.15.16:0/363508640 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc4806aef0 msgr2=0x7fbc481560e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:53.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.018+0000 7fbc4f073700 1 --2- 172.21.15.16:0/363508640 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc4806aef0 0x7fbc481560e0 secure :-1 s=READY pgs=3134 cs=0 l=1 rev1=1 crypto rx=0x7fbc3c009fd0 tx=0x7fbc3c004750 comp rx=0 tx=0).stop 2023-10-16T13:48:53.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.018+0000 7fbc4f073700 1 -- 172.21.15.16:0/363508640 shutdown_connections 2023-10-16T13:48:53.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.018+0000 7fbc4f073700 1 --2- 172.21.15.16:0/363508640 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbc48160fe0 0x7fbc48156eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:53.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.018+0000 7fbc4f073700 1 --2- 172.21.15.16:0/363508640 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fbc2c06af90 0x7fbc2c06d450 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:53.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.018+0000 7fbc4f073700 1 --2- 172.21.15.16:0/363508640 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc4806aef0 0x7fbc481560e0 unknown :-1 s=CLOSED pgs=3134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:53.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.018+0000 7fbc4f073700 1 --2- 172.21.15.16:0/363508640 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbc4806da30 0x7fbc48156970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:53.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.018+0000 7fbc4f073700 1 -- 172.21.15.16:0/363508640 >> 172.21.15.16:0/363508640 conn(0x7fbc480b1ed0 msgr2=0x7fbc480b22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:53.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.018+0000 7fbc4f073700 1 -- 172.21.15.16:0/363508640 shutdown_connections 2023-10-16T13:48:53.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.018+0000 7fbc4f073700 1 -- 172.21.15.16:0/363508640 wait complete. 2023-10-16T13:48:53.035 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-16T13:48:53.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.158+0000 7fe8e66a1700 1 Processor -- start 2023-10-16T13:48:53.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.158+0000 7fe8e66a1700 1 -- start start 2023-10-16T13:48:53.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.159+0000 7fe8e66a1700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe8e015b070 0x7fe8e015b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:53.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.159+0000 7fe8e66a1700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fe8e006e170 0x7fe8e00cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:53.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.159+0000 7fe8e66a1700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8e00cef70 con 0x7fe8e00cb880 2023-10-16T13:48:53.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.159+0000 7fe8e66a1700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8e00ce360 con 0x7fe8e006e170 2023-10-16T13:48:53.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.159+0000 7fe8e66a1700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e00ce640 con 0x7fe8e015b070 2023-10-16T13:48:53.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.159+0000 7fe8df7fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe8e00cb880 0x7fe8e006da60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:47982/0 (socket says 172.21.15.16:47982) 2023-10-16T13:48:53.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.159+0000 7fe8df7fe700 1 -- 172.21.15.16:0/3964914155 learned_addr learned my addr 172.21.15.16:0/3964914155 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:48:53.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.159+0000 7fe8dffff700 1 --2- 172.21.15.16:0/3964914155 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe8e015b070 0x7fe8e015b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:53.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.159+0000 7fe8e4c3e700 1 --2- 172.21.15.16:0/3964914155 >> v2:172.21.15.49:3300/0 conn(0x7fe8e006e170 0x7fe8e00cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:53.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.159+0000 7fe8deffd700 1 -- 172.21.15.16:0/3964914155 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1093602484 0 0) 0x7fe8e00cef70 con 0x7fe8e00cb880 2023-10-16T13:48:53.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.159+0000 7fe8deffd700 1 -- 172.21.15.16:0/3964914155 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8cc0035b0 con 0x7fe8e00cb880 2023-10-16T13:48:53.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.159+0000 7fe8dffff700 1 -- 172.21.15.16:0/3964914155 >> v2:172.21.15.49:3300/0 conn(0x7fe8e006e170 msgr2=0x7fe8e00cbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:53.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.159+0000 7fe8dffff700 1 --2- 172.21.15.16:0/3964914155 >> v2:172.21.15.49:3300/0 conn(0x7fe8e006e170 0x7fe8e00cbd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:53.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.159+0000 7fe8dffff700 1 -- 172.21.15.16:0/3964914155 >> v1:172.21.15.16:6789/0 conn(0x7fe8e00cb880 legacy=0x7fe8e006da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:53.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.159+0000 7fe8dffff700 1 -- 172.21.15.16:0/3964914155 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8e00ce920 con 0x7fe8e015b070 2023-10-16T13:48:53.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.160+0000 7fe8e4c3e700 1 --2- 172.21.15.16:0/3964914155 >> v2:172.21.15.49:3300/0 conn(0x7fe8e006e170 0x7fe8e00cbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:48:53.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.160+0000 7fe8dffff700 1 --2- 172.21.15.16:0/3964914155 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe8e015b070 0x7fe8e015b450 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fe8c8009840 tx=0x7fe8c8009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=96da95ad75c972ef server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:53.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.160+0000 7fe8deffd700 1 -- 172.21.15.16:0/3964914155 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8c8011830 con 0x7fe8e015b070 2023-10-16T13:48:53.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.160+0000 7fe8deffd700 1 -- 172.21.15.16:0/3964914155 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8c8011990 con 0x7fe8e015b070 2023-10-16T13:48:53.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.160+0000 7fe8deffd700 1 -- 172.21.15.16:0/3964914155 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8c8011d20 con 0x7fe8e015b070 2023-10-16T13:48:53.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.160+0000 7fe8e66a1700 1 -- 172.21.15.16:0/3964914155 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe8e015b070 msgr2=0x7fe8e015b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:53.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.160+0000 7fe8e66a1700 1 --2- 172.21.15.16:0/3964914155 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe8e015b070 0x7fe8e015b450 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fe8c8009840 tx=0x7fe8c8009b50 comp rx=0 tx=0).stop 2023-10-16T13:48:53.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.160+0000 7fe8e66a1700 1 -- 172.21.15.16:0/3964914155 shutdown_connections 2023-10-16T13:48:53.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.160+0000 7fe8e66a1700 1 --2- 172.21.15.16:0/3964914155 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe8e015b070 0x7fe8e015b450 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:53.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.160+0000 7fe8e66a1700 1 --2- 172.21.15.16:0/3964914155 >> v2:172.21.15.49:3300/0 conn(0x7fe8e006e170 0x7fe8e00cbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:53.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.160+0000 7fe8e66a1700 1 -- 172.21.15.16:0/3964914155 >> 172.21.15.16:0/3964914155 conn(0x7fe8e00b1f40 msgr2=0x7fe8e00b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:53.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.160+0000 7fe8e66a1700 1 -- 172.21.15.16:0/3964914155 shutdown_connections 2023-10-16T13:48:53.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.160+0000 7fe8e66a1700 1 -- 172.21.15.16:0/3964914155 wait complete. 2023-10-16T13:48:53.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.162+0000 7fe8e66a1700 1 Processor -- start 2023-10-16T13:48:53.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.162+0000 7fe8e66a1700 1 -- start start 2023-10-16T13:48:53.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.163+0000 7fe8e66a1700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe8e006e170 0x7fe8e00c9b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:53.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.163+0000 7fe8e66a1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8e00cb880 0x7fe8e01590e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:53.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.163+0000 7fe8e66a1700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe8e015b070 0x7fe8e01471b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:53.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.163+0000 7fe8e66a1700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e0158cc0 con 0x7fe8e00cb880 2023-10-16T13:48:53.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.163+0000 7fe8e66a1700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e01586d0 con 0x7fe8e015b070 2023-10-16T13:48:53.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.163+0000 7fe8e66a1700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e0147c50 con 0x7fe8e006e170 2023-10-16T13:48:53.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.163+0000 7fe8df7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8e00cb880 0x7fe8e01590e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:53.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.163+0000 7fe8df7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8e00cb880 0x7fe8e01590e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58714/0 (socket says 172.21.15.16:58714) 2023-10-16T13:48:53.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.163+0000 7fe8df7fe700 1 -- 172.21.15.16:0/542807350 learned_addr learned my addr 172.21.15.16:0/542807350 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:53.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.163+0000 7fe8e4c3e700 1 --2- 172.21.15.16:0/542807350 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe8e015b070 0x7fe8e01471b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:53.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.163+0000 7fe8dffff700 1 --2- 172.21.15.16:0/542807350 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe8e006e170 0x7fe8e00c9b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:53.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.163+0000 7fe8df7fe700 1 -- 172.21.15.16:0/542807350 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe8e006e170 msgr2=0x7fe8e00c9b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:53.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.163+0000 7fe8df7fe700 1 --2- 172.21.15.16:0/542807350 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe8e006e170 0x7fe8e00c9b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:53.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.163+0000 7fe8df7fe700 1 -- 172.21.15.16:0/542807350 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe8e015b070 msgr2=0x7fe8e01471b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:53.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.163+0000 7fe8df7fe700 1 --2- 172.21.15.16:0/542807350 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe8e015b070 0x7fe8e01471b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:53.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.163+0000 7fe8df7fe700 1 -- 172.21.15.16:0/542807350 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8d8009a70 con 0x7fe8e00cb880 2023-10-16T13:48:53.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.163+0000 7fe8df7fe700 1 --2- 172.21.15.16:0/542807350 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8e00cb880 0x7fe8e01590e0 secure :-1 s=READY pgs=3136 cs=0 l=1 rev1=1 crypto rx=0x7fe8d000c700 tx=0x7fe8d000cac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:53.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.163+0000 7fe8dcff9700 1 -- 172.21.15.16:0/542807350 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8d0005d70 con 0x7fe8e00cb880 2023-10-16T13:48:53.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.163+0000 7fe8dcff9700 1 -- 172.21.15.16:0/542807350 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8d00070d0 con 0x7fe8e00cb880 2023-10-16T13:48:53.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.163+0000 7fe8e66a1700 1 -- 172.21.15.16:0/542807350 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8c80094a0 con 0x7fe8e00cb880 2023-10-16T13:48:53.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.163+0000 7fe8dcff9700 1 -- 172.21.15.16:0/542807350 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8d0007460 con 0x7fe8e00cb880 2023-10-16T13:48:53.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.164+0000 7fe8e66a1700 1 -- 172.21.15.16:0/542807350 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8e01588e0 con 0x7fe8e00cb880 2023-10-16T13:48:53.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.164+0000 7fe8e66a1700 1 -- 172.21.15.16:0/542807350 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8e0158e00 con 0x7fe8e00cb880 2023-10-16T13:48:53.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.165+0000 7fe8dcff9700 1 -- 172.21.15.16:0/542807350 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 165) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fe8d00075c0 con 0x7fe8e00cb880 2023-10-16T13:48:53.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.165+0000 7fe8dcff9700 1 --2- 172.21.15.16:0/542807350 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fe8c006af90 0x7fe8c006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:53.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.166+0000 7fe8dffff700 1 --2- 172.21.15.16:0/542807350 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fe8c006af90 0x7fe8c006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:53.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.166+0000 7fe8dcff9700 1 -- 172.21.15.16:0/542807350 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 6165+0+0 (secure 0 0 0) 0x7fe8d0088f70 con 0x7fe8e00cb880 2023-10-16T13:48:53.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.166+0000 7fe8dffff700 1 --2- 172.21.15.16:0/542807350 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fe8c006af90 0x7fe8c006d450 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe8c8009fd0 tx=0x7fe8c801c040 comp rx=0 tx=0).ready entity=mgr.13344 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:53.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.170+0000 7fe8dcff9700 1 -- 172.21.15.16:0/542807350 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe8e0158e00 con 0x7fe8e00cb880 2023-10-16T13:48:53.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.374+0000 7fe8e66a1700 1 -- 172.21.15.16:0/542807350 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fe8e0158e00 con 0x7fe8e00cb880 2023-10-16T13:48:53.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.377+0000 7fe8dcff9700 1 -- 172.21.15.16:0/542807350 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v92) v1 ==== 104+0+0 (secure 0 0 0) 0x7fe8e0158e00 con 0x7fe8e00cb880 2023-10-16T13:48:53.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.379+0000 7fe8e66a1700 1 -- 172.21.15.16:0/542807350 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fe8c006af90 msgr2=0x7fe8c006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:53.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.379+0000 7fe8e66a1700 1 --2- 172.21.15.16:0/542807350 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fe8c006af90 0x7fe8c006d450 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe8c8009fd0 tx=0x7fe8c801c040 comp rx=0 tx=0).stop 2023-10-16T13:48:53.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.379+0000 7fe8e66a1700 1 -- 172.21.15.16:0/542807350 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8e00cb880 msgr2=0x7fe8e01590e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:53.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.379+0000 7fe8e66a1700 1 --2- 172.21.15.16:0/542807350 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8e00cb880 0x7fe8e01590e0 secure :-1 s=READY pgs=3136 cs=0 l=1 rev1=1 crypto rx=0x7fe8d000c700 tx=0x7fe8d000cac0 comp rx=0 tx=0).stop 2023-10-16T13:48:53.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.379+0000 7fe8e66a1700 1 -- 172.21.15.16:0/542807350 shutdown_connections 2023-10-16T13:48:53.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.379+0000 7fe8e66a1700 1 --2- 172.21.15.16:0/542807350 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe8e015b070 0x7fe8e01471b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:53.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.379+0000 7fe8e66a1700 1 --2- 172.21.15.16:0/542807350 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe8e006e170 0x7fe8e00c9b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:53.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.379+0000 7fe8e66a1700 1 --2- 172.21.15.16:0/542807350 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fe8c006af90 0x7fe8c006d450 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:53.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.379+0000 7fe8e66a1700 1 --2- 172.21.15.16:0/542807350 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe8e00cb880 0x7fe8e01590e0 unknown :-1 s=CLOSED pgs=3136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:53.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.379+0000 7fe8e66a1700 1 -- 172.21.15.16:0/542807350 >> 172.21.15.16:0/542807350 conn(0x7fe8e00b1f40 msgr2=0x7fe8e00d0c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:53.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.379+0000 7fe8e66a1700 1 -- 172.21.15.16:0/542807350 shutdown_connections 2023-10-16T13:48:53.384 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.379+0000 7fe8e66a1700 1 -- 172.21.15.16:0/542807350 wait complete. 2023-10-16T13:48:53.394 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-16T13:48:53.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.517+0000 7fdcf6857700 1 Processor -- start 2023-10-16T13:48:53.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.518+0000 7fdcf6857700 1 -- start start 2023-10-16T13:48:53.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.518+0000 7fdcf6857700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fdcf015da10 0x7fdcf00d1520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:53.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.518+0000 7fdcf6857700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdcf00d4060 0x7fdcf00cf2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:53.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.518+0000 7fdcf6857700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdcf00d1d20 0x7fdcf00d21b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:53.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.518+0000 7fdcf6857700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcf00d26f0 con 0x7fdcf00d4060 2023-10-16T13:48:53.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.518+0000 7fdcf6857700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcf00d4540 con 0x7fdcf015da10 2023-10-16T13:48:53.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.518+0000 7fdcf6857700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcf00ce7e0 con 0x7fdcf00d1d20 2023-10-16T13:48:53.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.518+0000 7fdcef7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdcf00d4060 0x7fdcf00cf2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:53.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.518+0000 7fdcef7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdcf00d4060 0x7fdcf00cf2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58720/0 (socket says 172.21.15.16:58720) 2023-10-16T13:48:53.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.519+0000 7fdcef7fe700 1 -- 172.21.15.16:0/1114668948 learned_addr learned my addr 172.21.15.16:0/1114668948 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:53.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.519+0000 7fdceffff700 1 --2- 172.21.15.16:0/1114668948 >> v2:172.21.15.49:3300/0 conn(0x7fdcf015da10 0x7fdcf00d1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:53.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.519+0000 7fdcf4df4700 1 --2- 172.21.15.16:0/1114668948 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdcf00d1d20 0x7fdcf00d21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:53.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.519+0000 7fdcef7fe700 1 -- 172.21.15.16:0/1114668948 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdcf00d1d20 msgr2=0x7fdcf00d21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:53.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.519+0000 7fdcef7fe700 1 --2- 172.21.15.16:0/1114668948 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdcf00d1d20 0x7fdcf00d21b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:53.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.519+0000 7fdcef7fe700 1 -- 172.21.15.16:0/1114668948 >> v2:172.21.15.49:3300/0 conn(0x7fdcf015da10 msgr2=0x7fdcf00d1520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:53.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.519+0000 7fdcef7fe700 1 --2- 172.21.15.16:0/1114668948 >> v2:172.21.15.49:3300/0 conn(0x7fdcf015da10 0x7fdcf00d1520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:53.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.519+0000 7fdcef7fe700 1 -- 172.21.15.16:0/1114668948 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcf00ceac0 con 0x7fdcf00d4060 2023-10-16T13:48:53.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.519+0000 7fdcef7fe700 1 --2- 172.21.15.16:0/1114668948 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdcf00d4060 0x7fdcf00cf2a0 secure :-1 s=READY pgs=3137 cs=0 l=1 rev1=1 crypto rx=0x7fdcf008b540 tx=0x7fdcd8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e41b29e5564e074f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:53.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.519+0000 7fdceeffd700 1 -- 172.21.15.16:0/1114668948 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcd800b870 con 0x7fdcf00d4060 2023-10-16T13:48:53.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.519+0000 7fdceeffd700 1 -- 172.21.15.16:0/1114668948 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcd800b9d0 con 0x7fdcf00d4060 2023-10-16T13:48:53.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.519+0000 7fdceeffd700 1 -- 172.21.15.16:0/1114668948 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcd800bd60 con 0x7fdcf00d4060 2023-10-16T13:48:53.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.520+0000 7fdcf6857700 1 -- 172.21.15.16:0/1114668948 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdcf00d4060 msgr2=0x7fdcf00cf2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:53.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.520+0000 7fdcf6857700 1 --2- 172.21.15.16:0/1114668948 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdcf00d4060 0x7fdcf00cf2a0 secure :-1 s=READY pgs=3137 cs=0 l=1 rev1=1 crypto rx=0x7fdcf008b540 tx=0x7fdcd8009bd0 comp rx=0 tx=0).stop 2023-10-16T13:48:53.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.520+0000 7fdcf6857700 1 -- 172.21.15.16:0/1114668948 shutdown_connections 2023-10-16T13:48:53.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.520+0000 7fdcf6857700 1 --2- 172.21.15.16:0/1114668948 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdcf00d1d20 0x7fdcf00d21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:53.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.520+0000 7fdcf6857700 1 --2- 172.21.15.16:0/1114668948 >> v2:172.21.15.49:3300/0 conn(0x7fdcf015da10 0x7fdcf00d1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:53.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.520+0000 7fdcf6857700 1 --2- 172.21.15.16:0/1114668948 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdcf00d4060 0x7fdcf00cf2a0 unknown :-1 s=CLOSED pgs=3137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:53.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.520+0000 7fdcf6857700 1 -- 172.21.15.16:0/1114668948 >> 172.21.15.16:0/1114668948 conn(0x7fdcf009c9f0 msgr2=0x7fdcf00d3740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:53.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.520+0000 7fdcf6857700 1 -- 172.21.15.16:0/1114668948 shutdown_connections 2023-10-16T13:48:53.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.520+0000 7fdcf6857700 1 -- 172.21.15.16:0/1114668948 wait complete. 2023-10-16T13:48:53.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.521+0000 7fdcf6857700 1 Processor -- start 2023-10-16T13:48:53.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.521+0000 7fdcf6857700 1 -- start start 2023-10-16T13:48:53.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.522+0000 7fdcf6857700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdcf00d1d20 0x7fdcf015a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:53.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.522+0000 7fdcf6857700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdcf00d4060 0x7fdcf01f6d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:53.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.522+0000 7fdcf6857700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdcf015da10 0x7fdcf00cc820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:53.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.522+0000 7fdcf6857700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcf01f6600 con 0x7fdcf00d4060 2023-10-16T13:48:53.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.522+0000 7fdcf6857700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcf01f68e0 con 0x7fdcf00d1d20 2023-10-16T13:48:53.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.522+0000 7fdcf6857700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcf01f3880 con 0x7fdcf015da10 2023-10-16T13:48:53.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.522+0000 7fdcef7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdcf00d4060 0x7fdcf01f6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:53.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.522+0000 7fdcef7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdcf00d4060 0x7fdcf01f6d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58728/0 (socket says 172.21.15.16:58728) 2023-10-16T13:48:53.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.522+0000 7fdcef7fe700 1 -- 172.21.15.16:0/1656718249 learned_addr learned my addr 172.21.15.16:0/1656718249 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:53.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.522+0000 7fdceffff700 1 --2- 172.21.15.16:0/1656718249 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdcf00d1d20 0x7fdcf015a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:53.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.522+0000 7fdcf4df4700 1 --2- 172.21.15.16:0/1656718249 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdcf015da10 0x7fdcf00cc820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:53.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.522+0000 7fdcef7fe700 1 -- 172.21.15.16:0/1656718249 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdcf015da10 msgr2=0x7fdcf00cc820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:53.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.522+0000 7fdcef7fe700 1 --2- 172.21.15.16:0/1656718249 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdcf015da10 0x7fdcf00cc820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:53.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.522+0000 7fdcef7fe700 1 -- 172.21.15.16:0/1656718249 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdcf00d1d20 msgr2=0x7fdcf015a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:53.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.522+0000 7fdcef7fe700 1 --2- 172.21.15.16:0/1656718249 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdcf00d1d20 0x7fdcf015a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:53.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.522+0000 7fdcef7fe700 1 -- 172.21.15.16:0/1656718249 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcd8009550 con 0x7fdcf00d4060 2023-10-16T13:48:53.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.523+0000 7fdcef7fe700 1 --2- 172.21.15.16:0/1656718249 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdcf00d4060 0x7fdcf01f6d70 secure :-1 s=READY pgs=3138 cs=0 l=1 rev1=1 crypto rx=0x7fdcd8013000 tx=0x7fdcd8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:53.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.523+0000 7fdcecff9700 1 -- 172.21.15.16:0/1656718249 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcd8004480 con 0x7fdcf00d4060 2023-10-16T13:48:53.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.523+0000 7fdcecff9700 1 -- 172.21.15.16:0/1656718249 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcd80045e0 con 0x7fdcf00d4060 2023-10-16T13:48:53.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.523+0000 7fdcf6857700 1 -- 172.21.15.16:0/1656718249 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcf01f3b60 con 0x7fdcf00d4060 2023-10-16T13:48:53.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.523+0000 7fdcecff9700 1 -- 172.21.15.16:0/1656718249 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcd80038e0 con 0x7fdcf00d4060 2023-10-16T13:48:53.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.523+0000 7fdcf6857700 1 -- 172.21.15.16:0/1656718249 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcf015a8a0 con 0x7fdcf00d4060 2023-10-16T13:48:53.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.524+0000 7fdcf6857700 1 -- 172.21.15.16:0/1656718249 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcf004ebb0 con 0x7fdcf00d4060 2023-10-16T13:48:53.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.524+0000 7fdcecff9700 1 -- 172.21.15.16:0/1656718249 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 165) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fdcd8021070 con 0x7fdcf00d4060 2023-10-16T13:48:53.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.525+0000 7fdcecff9700 1 --2- 172.21.15.16:0/1656718249 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fdcd006af90 0x7fdcd006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:53.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.525+0000 7fdcecff9700 1 -- 172.21.15.16:0/1656718249 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 6165+0+0 (secure 0 0 0) 0x7fdcd808add0 con 0x7fdcf00d4060 2023-10-16T13:48:53.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.525+0000 7fdceffff700 1 --2- 172.21.15.16:0/1656718249 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fdcd006af90 0x7fdcd006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:53.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.525+0000 7fdceffff700 1 --2- 172.21.15.16:0/1656718249 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fdcd006af90 0x7fdcd006d450 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fdce0006fd0 tx=0x7fdce0008040 comp rx=0 tx=0).ready entity=mgr.13344 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:53.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.529+0000 7fdcecff9700 1 -- 172.21.15.16:0/1656718249 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdcd8055e70 con 0x7fdcf00d4060 2023-10-16T13:48:53.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.591+0000 7fdcecff9700 1 -- 172.21.15.16:0/1656718249 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 166) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fdcd8056240 con 0x7fdcf00d4060 2023-10-16T13:48:53.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.748+0000 7fdcf6857700 1 -- 172.21.15.16:0/1656718249 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7fdcf00b2d00 con 0x7fdcf00d4060 2023-10-16T13:48:53.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.751+0000 7fdcecff9700 1 -- 172.21.15.16:0/1656718249 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v93) v1 ==== 120+0+0 (secure 0 0 0) 0x7fdcd806e9d0 con 0x7fdcf00d4060 2023-10-16T13:48:53.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.754+0000 7fdcf6857700 1 -- 172.21.15.16:0/1656718249 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fdcd006af90 msgr2=0x7fdcd006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:53.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.754+0000 7fdcf6857700 1 --2- 172.21.15.16:0/1656718249 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fdcd006af90 0x7fdcd006d450 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fdce0006fd0 tx=0x7fdce0008040 comp rx=0 tx=0).stop 2023-10-16T13:48:53.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.754+0000 7fdcf6857700 1 -- 172.21.15.16:0/1656718249 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdcf00d4060 msgr2=0x7fdcf01f6d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:53.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.754+0000 7fdcf6857700 1 --2- 172.21.15.16:0/1656718249 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdcf00d4060 0x7fdcf01f6d70 secure :-1 s=READY pgs=3138 cs=0 l=1 rev1=1 crypto rx=0x7fdcd8013000 tx=0x7fdcd8004750 comp rx=0 tx=0).stop 2023-10-16T13:48:53.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.754+0000 7fdcf6857700 1 -- 172.21.15.16:0/1656718249 shutdown_connections 2023-10-16T13:48:53.758 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.754+0000 7fdcf6857700 1 --2- 172.21.15.16:0/1656718249 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdcf015da10 0x7fdcf00cc820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:53.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.754+0000 7fdcf6857700 1 --2- 172.21.15.16:0/1656718249 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdcf00d1d20 0x7fdcf015a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:53.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.754+0000 7fdcf6857700 1 --2- 172.21.15.16:0/1656718249 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fdcd006af90 0x7fdcd006d450 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:53.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.754+0000 7fdcf6857700 1 --2- 172.21.15.16:0/1656718249 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdcf00d4060 0x7fdcf01f6d70 unknown :-1 s=CLOSED pgs=3138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:53.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.754+0000 7fdcf6857700 1 -- 172.21.15.16:0/1656718249 >> 172.21.15.16:0/1656718249 conn(0x7fdcf009c9f0 msgr2=0x7fdcf006b810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:53.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.754+0000 7fdcf6857700 1 -- 172.21.15.16:0/1656718249 shutdown_connections 2023-10-16T13:48:53.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.754+0000 7fdcf6857700 1 -- 172.21.15.16:0/1656718249 wait complete. 2023-10-16T13:48:53.771 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name' 2023-10-16T13:48:53.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.893+0000 7f4bd1eb9700 1 Processor -- start 2023-10-16T13:48:53.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.893+0000 7f4bd1eb9700 1 -- start start 2023-10-16T13:48:53.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.894+0000 7f4bd1eb9700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4bcc15b280 0x7f4bcc1613e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:53.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.894+0000 7f4bd1eb9700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f4bcc0cc8a0 0x7f4bcc15e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:53.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.894+0000 7f4bd1eb9700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bcc161c30 con 0x7f4bcc06a7e0 2023-10-16T13:48:53.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.894+0000 7f4bd1eb9700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bcc161f10 con 0x7f4bcc0cc8a0 2023-10-16T13:48:53.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.894+0000 7f4bd1eb9700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bcc1621f0 con 0x7f4bcc15b280 2023-10-16T13:48:53.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.894+0000 7f4bcaffd700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f4bcc0cc8a0 0x7f4bcc15e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:53.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.894+0000 7f4bcb7fe700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4bcc15b280 0x7f4bcc1613e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:53.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.894+0000 7f4bcbfff700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7f4bcc06a7e0 0x7f4bcc0d02b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:55194/0 (socket says 172.21.15.16:55194) 2023-10-16T13:48:53.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.894+0000 7f4bcaffd700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f4bcc0cc8a0 0x7f4bcc15e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:57798/0 (socket says 172.21.15.16:57798) 2023-10-16T13:48:53.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.894+0000 7f4bcbfff700 1 -- 172.21.15.16:0/669356766 learned_addr learned my addr 172.21.15.16:0/669356766 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:48:53.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.895+0000 7f4bcaffd700 1 -- 172.21.15.16:0/669356766 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4bcc15b280 msgr2=0x7f4bcc1613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:53.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.895+0000 7f4bcaffd700 1 --2- 172.21.15.16:0/669356766 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4bcc15b280 0x7f4bcc1613e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:53.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.895+0000 7f4bcaffd700 1 -- 172.21.15.16:0/669356766 >> v1:172.21.15.49:6789/0 conn(0x7f4bcc06a7e0 legacy=0x7f4bcc0d02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:53.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.895+0000 7f4bcaffd700 1 -- 172.21.15.16:0/669356766 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bcc15df00 con 0x7f4bcc0cc8a0 2023-10-16T13:48:53.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.895+0000 7f4bcaffd700 1 --2- 172.21.15.16:0/669356766 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4bcc0cc8a0 0x7f4bcc15e700 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f4bc0009840 tx=0x7f4bc0009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb4a97b2cf33400c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:53.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.895+0000 7f4bca7fc700 1 -- 172.21.15.16:0/669356766 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bc0011830 con 0x7f4bcc0cc8a0 2023-10-16T13:48:53.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.895+0000 7f4bca7fc700 1 -- 172.21.15.16:0/669356766 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bc0011990 con 0x7f4bcc0cc8a0 2023-10-16T13:48:53.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.895+0000 7f4bca7fc700 1 -- 172.21.15.16:0/669356766 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bc0011d20 con 0x7f4bcc0cc8a0 2023-10-16T13:48:53.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.896+0000 7f4bd1eb9700 1 -- 172.21.15.16:0/669356766 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4bcc0cc8a0 msgr2=0x7f4bcc15e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:53.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.896+0000 7f4bd1eb9700 1 --2- 172.21.15.16:0/669356766 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4bcc0cc8a0 0x7f4bcc15e700 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f4bc0009840 tx=0x7f4bc0009b50 comp rx=0 tx=0).stop 2023-10-16T13:48:53.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.896+0000 7f4bd1eb9700 1 -- 172.21.15.16:0/669356766 shutdown_connections 2023-10-16T13:48:53.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.896+0000 7f4bd1eb9700 1 --2- 172.21.15.16:0/669356766 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4bcc15b280 0x7f4bcc1613e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:53.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.896+0000 7f4bd1eb9700 1 --2- 172.21.15.16:0/669356766 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4bcc0cc8a0 0x7f4bcc15e700 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:53.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.896+0000 7f4bd1eb9700 1 -- 172.21.15.16:0/669356766 >> 172.21.15.16:0/669356766 conn(0x7f4bcc0b1fd0 msgr2=0x7f4bcc0b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:53.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.896+0000 7f4bd1eb9700 1 -- 172.21.15.16:0/669356766 shutdown_connections 2023-10-16T13:48:53.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.896+0000 7f4bd1eb9700 1 -- 172.21.15.16:0/669356766 wait complete. 2023-10-16T13:48:53.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.897+0000 7f4bd1eb9700 1 Processor -- start 2023-10-16T13:48:53.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.897+0000 7f4bd1eb9700 1 -- start start 2023-10-16T13:48:53.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.898+0000 7f4bd1eb9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bcc06a7e0 0x7f4bcc0c9de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:53.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.898+0000 7f4bd1eb9700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4bcc0cc8a0 0x7f4bcc06ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:53.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.898+0000 7f4bd1eb9700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4bcc15b280 0x7f4bcc06b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:53.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.898+0000 7f4bd1eb9700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bcc0c7510 con 0x7f4bcc06a7e0 2023-10-16T13:48:53.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.898+0000 7f4bd1eb9700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bcc0ca320 con 0x7f4bcc0cc8a0 2023-10-16T13:48:53.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.898+0000 7f4bd1eb9700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bcc1f7250 con 0x7f4bcc15b280 2023-10-16T13:48:53.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.898+0000 7f4bcb7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bcc06a7e0 0x7f4bcc0c9de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:53.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.898+0000 7f4bcb7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bcc06a7e0 0x7f4bcc0c9de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58740/0 (socket says 172.21.15.16:58740) 2023-10-16T13:48:53.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.898+0000 7f4bcb7fe700 1 -- 172.21.15.16:0/439212353 learned_addr learned my addr 172.21.15.16:0/439212353 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:53.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.898+0000 7f4bcaffd700 1 --2- 172.21.15.16:0/439212353 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4bcc0cc8a0 0x7f4bcc06ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:53.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.898+0000 7f4bcbfff700 1 --2- 172.21.15.16:0/439212353 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4bcc15b280 0x7f4bcc06b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:53.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.898+0000 7f4bcb7fe700 1 -- 172.21.15.16:0/439212353 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4bcc15b280 msgr2=0x7f4bcc06b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:53.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.898+0000 7f4bcb7fe700 1 --2- 172.21.15.16:0/439212353 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4bcc15b280 0x7f4bcc06b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:53.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.898+0000 7f4bcb7fe700 1 -- 172.21.15.16:0/439212353 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4bcc0cc8a0 msgr2=0x7f4bcc06ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:53.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.898+0000 7f4bcb7fe700 1 --2- 172.21.15.16:0/439212353 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4bcc0cc8a0 0x7f4bcc06ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:53.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.898+0000 7f4bcb7fe700 1 -- 172.21.15.16:0/439212353 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bc00094a0 con 0x7f4bcc06a7e0 2023-10-16T13:48:53.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.898+0000 7f4bcaffd700 1 --2- 172.21.15.16:0/439212353 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4bcc0cc8a0 0x7f4bcc06ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:48:53.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.899+0000 7f4bcb7fe700 1 --2- 172.21.15.16:0/439212353 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bcc06a7e0 0x7f4bcc0c9de0 secure :-1 s=READY pgs=3139 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc00e870 tx=0x7f4bbc00eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:53.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.899+0000 7f4bb3fff700 1 -- 172.21.15.16:0/439212353 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bbc009590 con 0x7f4bcc06a7e0 2023-10-16T13:48:53.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.899+0000 7f4bb3fff700 1 -- 172.21.15.16:0/439212353 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bbc013680 con 0x7f4bcc06a7e0 2023-10-16T13:48:53.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.899+0000 7f4bd1eb9700 1 -- 172.21.15.16:0/439212353 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bcc0c7760 con 0x7f4bcc06a7e0 2023-10-16T13:48:53.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.899+0000 7f4bb3fff700 1 -- 172.21.15.16:0/439212353 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bbc0139f0 con 0x7f4bcc06a7e0 2023-10-16T13:48:53.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.899+0000 7f4bd1eb9700 1 -- 172.21.15.16:0/439212353 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bcc0c7d20 con 0x7f4bcc06a7e0 2023-10-16T13:48:53.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.900+0000 7f4bd1eb9700 1 -- 172.21.15.16:0/439212353 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bcc0c7f10 con 0x7f4bcc06a7e0 2023-10-16T13:48:53.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.900+0000 7f4bb3fff700 1 -- 172.21.15.16:0/439212353 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 166) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4bbc013b50 con 0x7f4bcc06a7e0 2023-10-16T13:48:53.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.901+0000 7f4bb3fff700 1 --2- 172.21.15.16:0/439212353 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f4bb806b020 0x7f4bb806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:53.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.901+0000 7f4bb3fff700 1 -- 172.21.15.16:0/439212353 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 6165+0+0 (secure 0 0 0) 0x7f4bbc089710 con 0x7f4bcc06a7e0 2023-10-16T13:48:53.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.901+0000 7f4bcaffd700 1 --2- 172.21.15.16:0/439212353 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f4bb806b020 0x7f4bb806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:53.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.901+0000 7f4bcaffd700 1 --2- 172.21.15.16:0/439212353 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f4bb806b020 0x7f4bb806d4e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4bc0000c00 tx=0x7f4bc0008030 comp rx=0 tx=0).ready entity=mgr.13344 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:53.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:53.905+0000 7f4bb3fff700 1 -- 172.21.15.16:0/439212353 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4bbc054570 con 0x7f4bcc06a7e0 2023-10-16T13:48:54.201 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:54.198+0000 7f4bd1eb9700 1 -- 172.21.15.16:0/439212353 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]} v 0) v1 -- 0x7f4bcc04ebb0 con 0x7f4bcc06a7e0 2023-10-16T13:48:54.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:54.649+0000 7f4bb3fff700 1 -- 172.21.15.16:0/439212353 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]}]=0 v3263) v1 ==== 155+0+0 (secure 0 0 0) 0x7f4bbc057b90 con 0x7f4bcc06a7e0 2023-10-16T13:48:54.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:54.651+0000 7f4bd1eb9700 1 -- 172.21.15.16:0/439212353 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f4bb806b020 msgr2=0x7f4bb806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:54.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:54.651+0000 7f4bd1eb9700 1 --2- 172.21.15.16:0/439212353 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f4bb806b020 0x7f4bb806d4e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4bc0000c00 tx=0x7f4bc0008030 comp rx=0 tx=0).stop 2023-10-16T13:48:54.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:54.651+0000 7f4bd1eb9700 1 -- 172.21.15.16:0/439212353 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bcc06a7e0 msgr2=0x7f4bcc0c9de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:54.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:54.651+0000 7f4bd1eb9700 1 --2- 172.21.15.16:0/439212353 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bcc06a7e0 0x7f4bcc0c9de0 secure :-1 s=READY pgs=3139 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc00e870 tx=0x7f4bbc00eb80 comp rx=0 tx=0).stop 2023-10-16T13:48:54.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:54.651+0000 7f4bd1eb9700 1 -- 172.21.15.16:0/439212353 shutdown_connections 2023-10-16T13:48:54.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:54.651+0000 7f4bd1eb9700 1 --2- 172.21.15.16:0/439212353 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4bcc15b280 0x7f4bcc06b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:54.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:54.651+0000 7f4bd1eb9700 1 --2- 172.21.15.16:0/439212353 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f4bb806b020 0x7f4bb806d4e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:54.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:54.651+0000 7f4bd1eb9700 1 --2- 172.21.15.16:0/439212353 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4bcc06a7e0 0x7f4bcc0c9de0 unknown :-1 s=CLOSED pgs=3139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:54.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:54.651+0000 7f4bd1eb9700 1 --2- 172.21.15.16:0/439212353 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4bcc0cc8a0 0x7f4bcc06ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:54.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:54.651+0000 7f4bd1eb9700 1 -- 172.21.15.16:0/439212353 >> 172.21.15.16:0/439212353 conn(0x7f4bcc0b1fd0 msgr2=0x7f4bcc0cb9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:54.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:54.652+0000 7f4bd1eb9700 1 -- 172.21.15.16:0/439212353 shutdown_connections 2023-10-16T13:48:54.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:54.652+0000 7f4bd1eb9700 1 -- 172.21.15.16:0/439212353 wait complete. 2023-10-16T13:48:54.669 INFO:tasks.cephfs_test_runner:Test the correctness of snap directory name ... ok 2023-10-16T13:48:54.670 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_subvol_and_group_arguments (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments) Test subvol and group arguments to snap-schedule cli. 2023-10-16T13:48:54.670 INFO:tasks.cephfs_test_runner:test_snap_schedule_subvol_and_group_arguments (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments) 2023-10-16T13:48:54.671 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:48:54.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:54.796+0000 7fe5e5426700 1 Processor -- start 2023-10-16T13:48:54.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:54.796+0000 7fe5e5426700 1 -- start start 2023-10-16T13:48:54.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:54.797+0000 7fe5e5426700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe5e015b630 0x7fe5e00d0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:54.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:54.797+0000 7fe5e5426700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fe5e00ce4f0 0x7fe5e00cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:54.803 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:54.797+0000 7fe5e5426700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5e00b0970 con 0x7fe5e015ba10 2023-10-16T13:48:54.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:54.797+0000 7fe5e5426700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5e00cd690 con 0x7fe5e00ce4f0 2023-10-16T13:48:54.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:54.797+0000 7fe5e5426700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e00cd970 con 0x7fe5e015b630 2023-10-16T13:48:54.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:54.797+0000 7fe5de7fc700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe5e015ba10 0x7fe5e00cdde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48022/0 (socket says 172.21.15.16:48022) 2023-10-16T13:48:54.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:54.797+0000 7fe5de7fc700 1 -- 172.21.15.16:0/2227912765 learned_addr learned my addr 172.21.15.16:0/2227912765 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:48:54.804 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:54.798+0000 7fe5deffd700 1 --2- 172.21.15.16:0/2227912765 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe5e015b630 0x7fe5e00d0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:54.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:54.798+0000 7fe5df7fe700 1 --2- 172.21.15.16:0/2227912765 >> v2:172.21.15.49:3300/0 conn(0x7fe5e00ce4f0 0x7fe5e00cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:54.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:54.798+0000 7fe5ddffb700 1 -- 172.21.15.16:0/2227912765 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1419572870 0 0) 0x7fe5e00b0970 con 0x7fe5e015ba10 2023-10-16T13:48:54.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:54.798+0000 7fe5ddffb700 1 -- 172.21.15.16:0/2227912765 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5c80035b0 con 0x7fe5e015ba10 2023-10-16T13:48:54.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:54.798+0000 7fe5deffd700 1 -- 172.21.15.16:0/2227912765 >> v2:172.21.15.49:3300/0 conn(0x7fe5e00ce4f0 msgr2=0x7fe5e00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:54.805 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:54.798+0000 7fe5deffd700 1 --2- 172.21.15.16:0/2227912765 >> v2:172.21.15.49:3300/0 conn(0x7fe5e00ce4f0 0x7fe5e00cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:54.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:54.798+0000 7fe5deffd700 1 -- 172.21.15.16:0/2227912765 >> v1:172.21.15.16:6789/0 conn(0x7fe5e015ba10 legacy=0x7fe5e00cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:54.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:54.798+0000 7fe5deffd700 1 -- 172.21.15.16:0/2227912765 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5e00b0c30 con 0x7fe5e015b630 2023-10-16T13:48:54.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:54.798+0000 7fe5deffd700 1 --2- 172.21.15.16:0/2227912765 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe5e015b630 0x7fe5e00d0380 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fe5d0009840 tx=0x7fe5d0009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cfd6decc8822724b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:54.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:54.798+0000 7fe5ddffb700 1 -- 172.21.15.16:0/2227912765 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5d0011830 con 0x7fe5e015b630 2023-10-16T13:48:54.806 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:54.798+0000 7fe5ddffb700 1 -- 172.21.15.16:0/2227912765 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5d0011990 con 0x7fe5e015b630 2023-10-16T13:48:54.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:54.798+0000 7fe5ddffb700 1 -- 172.21.15.16:0/2227912765 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5d0011d20 con 0x7fe5e015b630 2023-10-16T13:48:54.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:54.799+0000 7fe5e5426700 1 -- 172.21.15.16:0/2227912765 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe5e015b630 msgr2=0x7fe5e00d0380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:54.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:54.799+0000 7fe5e5426700 1 --2- 172.21.15.16:0/2227912765 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe5e015b630 0x7fe5e00d0380 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fe5d0009840 tx=0x7fe5d0009b50 comp rx=0 tx=0).stop 2023-10-16T13:48:54.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:54.799+0000 7fe5e5426700 1 -- 172.21.15.16:0/2227912765 shutdown_connections 2023-10-16T13:48:54.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:54.799+0000 7fe5e5426700 1 --2- 172.21.15.16:0/2227912765 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe5e015b630 0x7fe5e00d0380 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:54.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:54.799+0000 7fe5e5426700 1 --2- 172.21.15.16:0/2227912765 >> v2:172.21.15.49:3300/0 conn(0x7fe5e00ce4f0 0x7fe5e00cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:54.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:54.799+0000 7fe5e5426700 1 -- 172.21.15.16:0/2227912765 >> 172.21.15.16:0/2227912765 conn(0x7fe5e00d14b0 msgr2=0x7fe5e00d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:54.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:54.799+0000 7fe5e5426700 1 -- 172.21.15.16:0/2227912765 shutdown_connections 2023-10-16T13:48:54.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:54.799+0000 7fe5e5426700 1 -- 172.21.15.16:0/2227912765 wait complete. 2023-10-16T13:48:54.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:54.800+0000 7fe5e5426700 1 Processor -- start 2023-10-16T13:48:54.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:54.800+0000 7fe5e5426700 1 -- start start 2023-10-16T13:48:54.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:54.801+0000 7fe5e5426700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5e00ce4f0 0x7fe5e0150ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:54.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:54.801+0000 7fe5e5426700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe5e015b630 0x7fe5e00b66d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:54.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:54.801+0000 7fe5e5426700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe5e015ba10 0x7fe5e00b6c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:54.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:54.801+0000 7fe5e5426700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e0151010 con 0x7fe5e00ce4f0 2023-10-16T13:48:54.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:54.801+0000 7fe5e5426700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e00b5780 con 0x7fe5e015ba10 2023-10-16T13:48:54.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:54.801+0000 7fe5e5426700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e00b46e0 con 0x7fe5e015b630 2023-10-16T13:48:54.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:54.801+0000 7fe5deffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5e00ce4f0 0x7fe5e0150ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:54.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:54.801+0000 7fe5deffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5e00ce4f0 0x7fe5e0150ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58752/0 (socket says 172.21.15.16:58752) 2023-10-16T13:48:54.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:54.801+0000 7fe5deffd700 1 -- 172.21.15.16:0/2906482504 learned_addr learned my addr 172.21.15.16:0/2906482504 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:54.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:54.801+0000 7fe5de7fc700 1 --2- 172.21.15.16:0/2906482504 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe5e015b630 0x7fe5e00b66d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:54.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:54.801+0000 7fe5df7fe700 1 --2- 172.21.15.16:0/2906482504 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe5e015ba10 0x7fe5e00b6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:54.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:54.801+0000 7fe5deffd700 1 -- 172.21.15.16:0/2906482504 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe5e015b630 msgr2=0x7fe5e00b66d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:54.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:54.801+0000 7fe5deffd700 1 --2- 172.21.15.16:0/2906482504 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe5e015b630 0x7fe5e00b66d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:54.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:54.801+0000 7fe5deffd700 1 -- 172.21.15.16:0/2906482504 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe5e015ba10 msgr2=0x7fe5e00b6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:54.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:54.801+0000 7fe5deffd700 1 --2- 172.21.15.16:0/2906482504 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe5e015ba10 0x7fe5e00b6c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:54.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:54.801+0000 7fe5deffd700 1 -- 172.21.15.16:0/2906482504 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5d00094a0 con 0x7fe5e00ce4f0 2023-10-16T13:48:54.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:54.801+0000 7fe5df7fe700 1 --2- 172.21.15.16:0/2906482504 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe5e015ba10 0x7fe5e00b6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:48:54.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:54.802+0000 7fe5deffd700 1 --2- 172.21.15.16:0/2906482504 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5e00ce4f0 0x7fe5e0150ad0 secure :-1 s=READY pgs=3141 cs=0 l=1 rev1=1 crypto rx=0x7fe5d00090b0 tx=0x7fe5d0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:54.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:54.802+0000 7fe5c77fe700 1 -- 172.21.15.16:0/2906482504 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5d00036a0 con 0x7fe5e00ce4f0 2023-10-16T13:48:54.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:54.802+0000 7fe5c77fe700 1 -- 172.21.15.16:0/2906482504 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5d001f070 con 0x7fe5e00ce4f0 2023-10-16T13:48:54.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:54.802+0000 7fe5e5426700 1 -- 172.21.15.16:0/2906482504 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5e00b49c0 con 0x7fe5e00ce4f0 2023-10-16T13:48:54.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:54.802+0000 7fe5c77fe700 1 -- 172.21.15.16:0/2906482504 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5d0003a60 con 0x7fe5e00ce4f0 2023-10-16T13:48:54.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:54.802+0000 7fe5e5426700 1 -- 172.21.15.16:0/2906482504 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5e00b4f80 con 0x7fe5e00ce4f0 2023-10-16T13:48:54.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:54.803+0000 7fe5e5426700 1 -- 172.21.15.16:0/2906482504 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5e004ebb0 con 0x7fe5e00ce4f0 2023-10-16T13:48:54.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:54.803+0000 7fe5c77fe700 1 -- 172.21.15.16:0/2906482504 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 166) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe5d0003bc0 con 0x7fe5e00ce4f0 2023-10-16T13:48:54.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:54.804+0000 7fe5c77fe700 1 --2- 172.21.15.16:0/2906482504 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fe5cc06b020 0x7fe5cc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:54.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:54.804+0000 7fe5c77fe700 1 -- 172.21.15.16:0/2906482504 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 6165+0+0 (secure 0 0 0) 0x7fe5d0021070 con 0x7fe5e00ce4f0 2023-10-16T13:48:54.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:54.804+0000 7fe5de7fc700 1 --2- 172.21.15.16:0/2906482504 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fe5cc06b020 0x7fe5cc06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:54.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:54.804+0000 7fe5de7fc700 1 --2- 172.21.15.16:0/2906482504 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fe5cc06b020 0x7fe5cc06d4e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe5e00c4f60 tx=0x7fe5d4007dc0 comp rx=0 tx=0).ready entity=mgr.13344 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:54.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:54.808+0000 7fe5c77fe700 1 -- 172.21.15.16:0/2906482504 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe5d0055f50 con 0x7fe5e00ce4f0 2023-10-16T13:48:55.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.011+0000 7fe5e5426700 1 -- 172.21.15.16:0/2906482504 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe5e006b5d0 con 0x7fe5e00ce4f0 2023-10-16T13:48:55.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.012+0000 7fe5c77fe700 1 -- 172.21.15.16:0/2906482504 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v270) v1 ==== 74+0+14423 (secure 0 0 0) 0x7fe5d0015090 con 0x7fe5e00ce4f0 2023-10-16T13:48:55.016 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:48:55.016 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":270,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:48:52.646791+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":36,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.49:6835/1985453659":"2023-10-17T13:48:49.642192+0000","172.21.15.49:6833/1985453659":"2023-10-17T13:48:49.642192+0000","172.21.15.16:6833/2131389174":"2023-10-17T13:48:48.569666+0000","172.21.15.16:6832/2131389174":"2023-10-17T13:48:48.569666+0000","172.21.15.16:0/762492059":"2023-10-17T13:48:48.569666+0000","172.21.15.16:0/1604016973":"2023-10-17T13:48:48.569666+0000","172.21.15.16:0/2535628588":"2023-10-17T13:48:48.569666+0000","172.21.15.16:0/3313712117":"2023-10-17T13:48:39.972713+0000","172.21.15.16:0/2881413822":"2023-10-17T13:48:48.569666+0000","172.21.15.16:0/2202829273":"2023-10-17T13:48:39.972713+0000","172.21.15.16:0/2777724108":"2023-10-17T13:48:39.972713+0000","172.21.15.16:0/2088077503":"2023-10-17T13:48:39.972713+0000","172.21.15.16:0/4096007345":"2023-10-17T13:48:39.972713+0000","172.21.15.16:0/3000870029":"2023-10-17T13:48:39.972713+0000","172.21.15.16:0/3746447491":"2023-10-17T13:48:48.569666+0000","172.21.15.16:0/3923653515":"2023-10-17T13:48:39.972713+0000","172.21.15.16:0/1621010103":"2023-10-17T13:48:48.569666+0000","172.21.15.16:0/56667078":"2023-10-17T13:48:39.972713+0000","172.21.15.16:6833/1275231846":"2023-10-17T13:48:39.972713+0000","172.21.15.16:6832/1275231846":"2023-10-17T13:48:39.972713+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:48:55.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.015+0000 7fe5e5426700 1 -- 172.21.15.16:0/2906482504 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fe5cc06b020 msgr2=0x7fe5cc06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:55.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.015+0000 7fe5e5426700 1 --2- 172.21.15.16:0/2906482504 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fe5cc06b020 0x7fe5cc06d4e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe5e00c4f60 tx=0x7fe5d4007dc0 comp rx=0 tx=0).stop 2023-10-16T13:48:55.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.015+0000 7fe5e5426700 1 -- 172.21.15.16:0/2906482504 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5e00ce4f0 msgr2=0x7fe5e0150ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:55.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.015+0000 7fe5e5426700 1 --2- 172.21.15.16:0/2906482504 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5e00ce4f0 0x7fe5e0150ad0 secure :-1 s=READY pgs=3141 cs=0 l=1 rev1=1 crypto rx=0x7fe5d00090b0 tx=0x7fe5d0004750 comp rx=0 tx=0).stop 2023-10-16T13:48:55.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.015+0000 7fe5e5426700 1 -- 172.21.15.16:0/2906482504 shutdown_connections 2023-10-16T13:48:55.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.015+0000 7fe5e5426700 1 --2- 172.21.15.16:0/2906482504 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe5e015ba10 0x7fe5e00b6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:55.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.015+0000 7fe5e5426700 1 --2- 172.21.15.16:0/2906482504 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fe5cc06b020 0x7fe5cc06d4e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:55.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.015+0000 7fe5e5426700 1 --2- 172.21.15.16:0/2906482504 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5e00ce4f0 0x7fe5e0150ad0 unknown :-1 s=CLOSED pgs=3141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:55.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.015+0000 7fe5e5426700 1 --2- 172.21.15.16:0/2906482504 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe5e015b630 0x7fe5e00b66d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:55.020 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.015+0000 7fe5e5426700 1 -- 172.21.15.16:0/2906482504 >> 172.21.15.16:0/2906482504 conn(0x7fe5e00d14b0 msgr2=0x7fe5e006f9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:55.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.015+0000 7fe5e5426700 1 -- 172.21.15.16:0/2906482504 shutdown_connections 2023-10-16T13:48:55.021 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.015+0000 7fe5e5426700 1 -- 172.21.15.16:0/2906482504 wait complete. 2023-10-16T13:48:55.031 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T13:48:55.032 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T13:48:55.159 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.156+0000 7f92e7322700 1 Processor -- start 2023-10-16T13:48:55.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.156+0000 7f92e7322700 1 -- start start 2023-10-16T13:48:55.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.157+0000 7f92e7322700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f92e0160fe0 0x7f92e00ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:55.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.157+0000 7f92e7322700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f92e006da30 0x7f92e006a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:55.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.157+0000 7f92e7322700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92e01613c0 con 0x7f92e006ad20 2023-10-16T13:48:55.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.157+0000 7f92e7322700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92e00cf680 con 0x7f92e0160fe0 2023-10-16T13:48:55.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.157+0000 7f92e7322700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92e006b1b0 con 0x7f92e006da30 2023-10-16T13:48:55.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.157+0000 7f92e48bd700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f92e006da30 0x7f92e006a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:55.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.157+0000 7f92e50be700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f92e0160fe0 0x7f92e00ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:55.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.157+0000 7f92e58bf700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7f92e006ad20 0x7f92e0071a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:55220/0 (socket says 172.21.15.16:55220) 2023-10-16T13:48:55.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.157+0000 7f92e48bd700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f92e006da30 0x7f92e006a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:47018/0 (socket says 172.21.15.16:47018) 2023-10-16T13:48:55.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.157+0000 7f92e58bf700 1 -- 172.21.15.16:0/3301538154 learned_addr learned my addr 172.21.15.16:0/3301538154 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:48:55.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.157+0000 7f92e48bd700 1 -- 172.21.15.16:0/3301538154 >> v2:172.21.15.49:3300/0 conn(0x7f92e0160fe0 msgr2=0x7f92e00ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:55.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.157+0000 7f92e48bd700 1 --2- 172.21.15.16:0/3301538154 >> v2:172.21.15.49:3300/0 conn(0x7f92e0160fe0 0x7f92e00ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:55.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.157+0000 7f92e48bd700 1 -- 172.21.15.16:0/3301538154 >> v1:172.21.15.49:6789/0 conn(0x7f92e006ad20 legacy=0x7f92e0071a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:55.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.157+0000 7f92e48bd700 1 -- 172.21.15.16:0/3301538154 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92e00adb00 con 0x7f92e006da30 2023-10-16T13:48:55.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.158+0000 7f92e48bd700 1 --2- 172.21.15.16:0/3301538154 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f92e006da30 0x7f92e006a7e0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f92d0009840 tx=0x7f92d0009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=84219927e317cefe server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:55.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.158+0000 7f92d7fff700 1 -- 172.21.15.16:0/3301538154 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92d0011830 con 0x7f92e006da30 2023-10-16T13:48:55.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.158+0000 7f92d7fff700 1 -- 172.21.15.16:0/3301538154 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92d0011990 con 0x7f92e006da30 2023-10-16T13:48:55.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.158+0000 7f92d7fff700 1 -- 172.21.15.16:0/3301538154 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92d0011d20 con 0x7f92e006da30 2023-10-16T13:48:55.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.158+0000 7f92e7322700 1 -- 172.21.15.16:0/3301538154 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f92e006da30 msgr2=0x7f92e006a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:55.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.158+0000 7f92e7322700 1 --2- 172.21.15.16:0/3301538154 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f92e006da30 0x7f92e006a7e0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f92d0009840 tx=0x7f92d0009b50 comp rx=0 tx=0).stop 2023-10-16T13:48:55.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.158+0000 7f92e7322700 1 -- 172.21.15.16:0/3301538154 shutdown_connections 2023-10-16T13:48:55.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.158+0000 7f92e7322700 1 --2- 172.21.15.16:0/3301538154 >> v2:172.21.15.49:3300/0 conn(0x7f92e0160fe0 0x7f92e00ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:55.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.158+0000 7f92e7322700 1 --2- 172.21.15.16:0/3301538154 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f92e006da30 0x7f92e006a7e0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:55.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.158+0000 7f92e7322700 1 -- 172.21.15.16:0/3301538154 >> 172.21.15.16:0/3301538154 conn(0x7f92e00b1ed0 msgr2=0x7f92e00b22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:55.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.158+0000 7f92e7322700 1 -- 172.21.15.16:0/3301538154 shutdown_connections 2023-10-16T13:48:55.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.158+0000 7f92e7322700 1 -- 172.21.15.16:0/3301538154 wait complete. 2023-10-16T13:48:55.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.160+0000 7f92e7322700 1 Processor -- start 2023-10-16T13:48:55.169 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.160+0000 7f92e7322700 1 -- start start 2023-10-16T13:48:55.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.160+0000 7f92e7322700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f92e006ad20 0x7f92e0153590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:55.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.160+0000 7f92e7322700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f92e006da30 0x7f92e0154940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:55.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.160+0000 7f92e7322700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92e0160fe0 0x7f92e0154e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:55.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.160+0000 7f92e7322700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92e00c3e80 con 0x7f92e0160fe0 2023-10-16T13:48:55.170 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.160+0000 7f92e7322700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92e00c5140 con 0x7f92e006da30 2023-10-16T13:48:55.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.160+0000 7f92e7322700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92e0153ad0 con 0x7f92e006ad20 2023-10-16T13:48:55.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.160+0000 7f92e58bf700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92e0160fe0 0x7f92e0154e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:55.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.161+0000 7f92e58bf700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92e0160fe0 0x7f92e0154e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58770/0 (socket says 172.21.15.16:58770) 2023-10-16T13:48:55.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.161+0000 7f92e58bf700 1 -- 172.21.15.16:0/699111731 learned_addr learned my addr 172.21.15.16:0/699111731 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:55.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.161+0000 7f92e50be700 1 --2- 172.21.15.16:0/699111731 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f92e006ad20 0x7f92e0153590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:55.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.161+0000 7f92e48bd700 1 --2- 172.21.15.16:0/699111731 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f92e006da30 0x7f92e0154940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:55.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.161+0000 7f92e58bf700 1 -- 172.21.15.16:0/699111731 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f92e006ad20 msgr2=0x7f92e0153590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:55.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.161+0000 7f92e58bf700 1 --2- 172.21.15.16:0/699111731 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f92e006ad20 0x7f92e0153590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:55.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.161+0000 7f92e58bf700 1 -- 172.21.15.16:0/699111731 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f92e006da30 msgr2=0x7f92e0154940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:55.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.161+0000 7f92e58bf700 1 --2- 172.21.15.16:0/699111731 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f92e006da30 0x7f92e0154940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:55.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.161+0000 7f92e58bf700 1 -- 172.21.15.16:0/699111731 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92d00094a0 con 0x7f92e0160fe0 2023-10-16T13:48:55.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.161+0000 7f92e58bf700 1 --2- 172.21.15.16:0/699111731 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92e0160fe0 0x7f92e0154e80 secure :-1 s=READY pgs=3142 cs=0 l=1 rev1=1 crypto rx=0x7f92d800ca70 tx=0x7f92d800cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:55.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.161+0000 7f92d5ffb700 1 -- 172.21.15.16:0/699111731 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92d800a9e0 con 0x7f92e0160fe0 2023-10-16T13:48:55.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.161+0000 7f92d5ffb700 1 -- 172.21.15.16:0/699111731 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92d800ab40 con 0x7f92e0160fe0 2023-10-16T13:48:55.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.161+0000 7f92e7322700 1 -- 172.21.15.16:0/699111731 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92e01554b0 con 0x7f92e0160fe0 2023-10-16T13:48:55.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.161+0000 7f92d5ffb700 1 -- 172.21.15.16:0/699111731 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92d8012400 con 0x7f92e0160fe0 2023-10-16T13:48:55.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.161+0000 7f92e7322700 1 -- 172.21.15.16:0/699111731 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92e0155a70 con 0x7f92e0160fe0 2023-10-16T13:48:55.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.162+0000 7f92e7322700 1 -- 172.21.15.16:0/699111731 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92e004ebb0 con 0x7f92e0160fe0 2023-10-16T13:48:55.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.163+0000 7f92d5ffb700 1 -- 172.21.15.16:0/699111731 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 166) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f92d801b3b0 con 0x7f92e0160fe0 2023-10-16T13:48:55.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.163+0000 7f92d5ffb700 1 --2- 172.21.15.16:0/699111731 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f92c406aff0 0x7f92c406d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:55.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.164+0000 7f92d5ffb700 1 -- 172.21.15.16:0/699111731 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 6165+0+0 (secure 0 0 0) 0x7f92d8089200 con 0x7f92e0160fe0 2023-10-16T13:48:55.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.164+0000 7f92e50be700 1 --2- 172.21.15.16:0/699111731 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f92c406aff0 0x7f92c406d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:55.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.164+0000 7f92e50be700 1 --2- 172.21.15.16:0/699111731 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f92c406aff0 0x7f92c406d4b0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f92dc009b20 tx=0x7f92dc0091c0 comp rx=0 tx=0).ready entity=mgr.13344 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:55.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.168+0000 7f92d5ffb700 1 -- 172.21.15.16:0/699111731 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f92d8054090 con 0x7f92e0160fe0 2023-10-16T13:48:55.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.371+0000 7f92e7322700 1 -- 172.21.15.16:0/699111731 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f92e01613c0 con 0x7f92e0160fe0 2023-10-16T13:48:55.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.372+0000 7f92d5ffb700 1 -- 172.21.15.16:0/699111731 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v270) v1 ==== 93+0+10 (secure 0 0 0) 0x7f92e01613c0 con 0x7f92e0160fe0 2023-10-16T13:48:55.376 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T13:48:55.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.374+0000 7f92e7322700 1 -- 172.21.15.16:0/699111731 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f92c406aff0 msgr2=0x7f92c406d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:55.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.374+0000 7f92e7322700 1 --2- 172.21.15.16:0/699111731 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f92c406aff0 0x7f92c406d4b0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f92dc009b20 tx=0x7f92dc0091c0 comp rx=0 tx=0).stop 2023-10-16T13:48:55.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.374+0000 7f92e7322700 1 -- 172.21.15.16:0/699111731 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92e0160fe0 msgr2=0x7f92e0154e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:55.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.374+0000 7f92e7322700 1 --2- 172.21.15.16:0/699111731 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92e0160fe0 0x7f92e0154e80 secure :-1 s=READY pgs=3142 cs=0 l=1 rev1=1 crypto rx=0x7f92d800ca70 tx=0x7f92d800cd80 comp rx=0 tx=0).stop 2023-10-16T13:48:55.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.374+0000 7f92e7322700 1 -- 172.21.15.16:0/699111731 shutdown_connections 2023-10-16T13:48:55.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.374+0000 7f92e7322700 1 --2- 172.21.15.16:0/699111731 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f92c406aff0 0x7f92c406d4b0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:55.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.374+0000 7f92e7322700 1 --2- 172.21.15.16:0/699111731 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92e0160fe0 0x7f92e0154e80 unknown :-1 s=CLOSED pgs=3142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:55.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.374+0000 7f92e7322700 1 --2- 172.21.15.16:0/699111731 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f92e006ad20 0x7f92e0153590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:55.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.374+0000 7f92e7322700 1 --2- 172.21.15.16:0/699111731 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f92e006da30 0x7f92e0154940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:55.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.374+0000 7f92e7322700 1 -- 172.21.15.16:0/699111731 >> 172.21.15.16:0/699111731 conn(0x7f92e00b1ed0 msgr2=0x7f92e00adcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:55.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.374+0000 7f92e7322700 1 -- 172.21.15.16:0/699111731 shutdown_connections 2023-10-16T13:48:55.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.374+0000 7f92e7322700 1 -- 172.21.15.16:0/699111731 wait complete. 2023-10-16T13:48:55.390 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments.test_snap_schedule_subvol_and_group_arguments' 2023-10-16T13:48:55.516 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.513+0000 7f7fa5df6700 1 Processor -- start 2023-10-16T13:48:55.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.513+0000 7f7fa5df6700 1 -- start start 2023-10-16T13:48:55.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.514+0000 7f7fa5df6700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f7fa00cf880 0x7f7fa00b52f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:55.519 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.514+0000 7f7fa5df6700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7fa00b5830 0x7f7fa015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:55.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.514+0000 7f7fa5df6700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fa015bc80 con 0x7f7fa00cf4a0 2023-10-16T13:48:55.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.514+0000 7f7fa5df6700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fa015b070 con 0x7f7fa00cf880 2023-10-16T13:48:55.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.514+0000 7f7fa5df6700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fa015b350 con 0x7f7fa00b5830 2023-10-16T13:48:55.520 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.514+0000 7f7f9f7fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f7fa00cf4a0 0x7f7fa00726e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48048/0 (socket says 172.21.15.16:48048) 2023-10-16T13:48:55.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.514+0000 7f7f9f7fe700 1 -- 172.21.15.16:0/1192035278 learned_addr learned my addr 172.21.15.16:0/1192035278 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:48:55.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.514+0000 7f7f9ffff700 1 --2- 172.21.15.16:0/1192035278 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7fa00b5830 0x7f7fa015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:55.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.514+0000 7f7f9effd700 1 --2- 172.21.15.16:0/1192035278 >> v2:172.21.15.49:3300/0 conn(0x7f7fa00cf880 0x7f7fa00b52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:55.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.514+0000 7f7f9e7fc700 1 -- 172.21.15.16:0/1192035278 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3112863617 0 0) 0x7f7fa015bc80 con 0x7f7fa00cf4a0 2023-10-16T13:48:55.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.514+0000 7f7f9e7fc700 1 -- 172.21.15.16:0/1192035278 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f8c0035b0 con 0x7f7fa00cf4a0 2023-10-16T13:48:55.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.514+0000 7f7f9effd700 1 -- 172.21.15.16:0/1192035278 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7fa00b5830 msgr2=0x7f7fa015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:55.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.514+0000 7f7f9effd700 1 --2- 172.21.15.16:0/1192035278 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7fa00b5830 0x7f7fa015a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:55.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.514+0000 7f7f9effd700 1 -- 172.21.15.16:0/1192035278 >> v1:172.21.15.16:6789/0 conn(0x7f7fa00cf4a0 legacy=0x7f7fa00726e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:55.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.514+0000 7f7f9effd700 1 -- 172.21.15.16:0/1192035278 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fa015b630 con 0x7f7fa00cf880 2023-10-16T13:48:55.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.515+0000 7f7f9effd700 1 --2- 172.21.15.16:0/1192035278 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7fa00cf880 0x7f7fa00b52f0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f7f88009840 tx=0x7f7f88009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a0e1d4f39142db1a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:55.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.515+0000 7f7f9e7fc700 1 -- 172.21.15.16:0/1192035278 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f88011830 con 0x7f7fa00cf880 2023-10-16T13:48:55.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.515+0000 7f7f9e7fc700 1 -- 172.21.15.16:0/1192035278 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f88011990 con 0x7f7fa00cf880 2023-10-16T13:48:55.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.515+0000 7f7f9e7fc700 1 -- 172.21.15.16:0/1192035278 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f88011d20 con 0x7f7fa00cf880 2023-10-16T13:48:55.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.515+0000 7f7fa5df6700 1 -- 172.21.15.16:0/1192035278 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7fa00cf880 msgr2=0x7f7fa00b52f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:55.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.515+0000 7f7fa5df6700 1 --2- 172.21.15.16:0/1192035278 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7fa00cf880 0x7f7fa00b52f0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f7f88009840 tx=0x7f7f88009b50 comp rx=0 tx=0).stop 2023-10-16T13:48:55.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.515+0000 7f7fa5df6700 1 -- 172.21.15.16:0/1192035278 shutdown_connections 2023-10-16T13:48:55.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.515+0000 7f7fa5df6700 1 --2- 172.21.15.16:0/1192035278 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7fa00b5830 0x7f7fa015a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:55.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.515+0000 7f7fa5df6700 1 --2- 172.21.15.16:0/1192035278 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7fa00cf880 0x7f7fa00b52f0 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:55.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.515+0000 7f7fa5df6700 1 -- 172.21.15.16:0/1192035278 >> 172.21.15.16:0/1192035278 conn(0x7f7fa00b1fd0 msgr2=0x7f7fa00b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:55.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.515+0000 7f7fa5df6700 1 -- 172.21.15.16:0/1192035278 shutdown_connections 2023-10-16T13:48:55.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.515+0000 7f7fa5df6700 1 -- 172.21.15.16:0/1192035278 wait complete. 2023-10-16T13:48:55.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.517+0000 7f7fa5df6700 1 Processor -- start 2023-10-16T13:48:55.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.517+0000 7f7fa5df6700 1 -- start start 2023-10-16T13:48:55.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.517+0000 7f7fa5df6700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7fa00b5830 0x7f7fa014d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:55.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.517+0000 7f7fa5df6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7fa00cf4a0 0x7f7fa00cd670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:55.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.517+0000 7f7fa5df6700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7fa00cf880 0x7f7fa01f2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:55.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.517+0000 7f7fa5df6700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fa01f5e50 con 0x7f7fa00cf4a0 2023-10-16T13:48:55.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.517+0000 7f7fa5df6700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fa01f6c70 con 0x7f7fa00b5830 2023-10-16T13:48:55.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.517+0000 7f7fa5df6700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fa00cdbb0 con 0x7f7fa00cf880 2023-10-16T13:48:55.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.518+0000 7f7f9effd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7fa00cf4a0 0x7f7fa00cd670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:55.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.518+0000 7f7f9effd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7fa00cf4a0 0x7f7fa00cd670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58782/0 (socket says 172.21.15.16:58782) 2023-10-16T13:48:55.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.518+0000 7f7f9effd700 1 -- 172.21.15.16:0/4222808790 learned_addr learned my addr 172.21.15.16:0/4222808790 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:55.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.518+0000 7f7f9f7fe700 1 --2- 172.21.15.16:0/4222808790 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7fa00b5830 0x7f7fa014d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:55.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.518+0000 7f7f9ffff700 1 --2- 172.21.15.16:0/4222808790 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7fa00cf880 0x7f7fa01f2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:55.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.518+0000 7f7f9effd700 1 -- 172.21.15.16:0/4222808790 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7fa00cf880 msgr2=0x7f7fa01f2000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:55.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.518+0000 7f7f9effd700 1 --2- 172.21.15.16:0/4222808790 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7fa00cf880 0x7f7fa01f2000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:55.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.518+0000 7f7f9effd700 1 -- 172.21.15.16:0/4222808790 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7fa00b5830 msgr2=0x7f7fa014d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:55.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.518+0000 7f7f9effd700 1 --2- 172.21.15.16:0/4222808790 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7fa00b5830 0x7f7fa014d380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:55.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.518+0000 7f7f9effd700 1 -- 172.21.15.16:0/4222808790 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f880094a0 con 0x7f7fa00cf4a0 2023-10-16T13:48:55.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.518+0000 7f7f9effd700 1 --2- 172.21.15.16:0/4222808790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7fa00cf4a0 0x7f7fa00cd670 secure :-1 s=READY pgs=3144 cs=0 l=1 rev1=1 crypto rx=0x7f7f880090b0 tx=0x7f7f88004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:55.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.518+0000 7f7f97fff700 1 -- 172.21.15.16:0/4222808790 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f880036a0 con 0x7f7fa00cf4a0 2023-10-16T13:48:55.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.518+0000 7f7f97fff700 1 -- 172.21.15.16:0/4222808790 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f8801e070 con 0x7f7fa00cf4a0 2023-10-16T13:48:55.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.518+0000 7f7fa5df6700 1 -- 172.21.15.16:0/4222808790 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fa01f6060 con 0x7f7fa00cf4a0 2023-10-16T13:48:55.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.518+0000 7f7f97fff700 1 -- 172.21.15.16:0/4222808790 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f88003a60 con 0x7f7fa00cf4a0 2023-10-16T13:48:55.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.518+0000 7f7fa5df6700 1 -- 172.21.15.16:0/4222808790 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fa01f6620 con 0x7f7fa00cf4a0 2023-10-16T13:48:55.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.519+0000 7f7fa5df6700 1 -- 172.21.15.16:0/4222808790 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fa004ebb0 con 0x7f7fa00cf4a0 2023-10-16T13:48:55.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.520+0000 7f7f97fff700 1 -- 172.21.15.16:0/4222808790 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 166) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7f88003bc0 con 0x7f7fa00cf4a0 2023-10-16T13:48:55.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.520+0000 7f7f97fff700 1 --2- 172.21.15.16:0/4222808790 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f7f8006b020 0x7f7f8006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:55.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.520+0000 7f7f97fff700 1 -- 172.21.15.16:0/4222808790 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 6165+0+0 (secure 0 0 0) 0x7f7f8808a7c0 con 0x7f7fa00cf4a0 2023-10-16T13:48:55.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.520+0000 7f7f9f7fe700 1 --2- 172.21.15.16:0/4222808790 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f7f8006b020 0x7f7f8006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:55.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.521+0000 7f7f9f7fe700 1 --2- 172.21.15.16:0/4222808790 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f7f8006b020 0x7f7f8006d4e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7f900011e0 tx=0x7f7f90009040 comp rx=0 tx=0).ready entity=mgr.13344 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:55.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.525+0000 7f7f97fff700 1 -- 172.21.15.16:0/4222808790 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7f88055620 con 0x7f7fa00cf4a0 2023-10-16T13:48:55.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:55.813+0000 7f7fa5df6700 1 -- 172.21.15.16:0/4222808790 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments.test_snap_schedule_subvol_and_group_arguments"]} v 0) v1 -- 0x7f7fa00b5490 con 0x7f7fa00cf4a0 2023-10-16T13:48:56.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.654+0000 7f7f97fff700 1 -- 172.21.15.16:0/4222808790 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments.test_snap_schedule_subvol_and_group_arguments"]}]=0 v3265) v1 ==== 201+0+0 (secure 0 0 0) 0x7f7f8805f070 con 0x7f7fa00cf4a0 2023-10-16T13:48:56.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.658+0000 7f7fa5df6700 1 -- 172.21.15.16:0/4222808790 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f7f8006b020 msgr2=0x7f7f8006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:56.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.658+0000 7f7fa5df6700 1 --2- 172.21.15.16:0/4222808790 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f7f8006b020 0x7f7f8006d4e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7f900011e0 tx=0x7f7f90009040 comp rx=0 tx=0).stop 2023-10-16T13:48:56.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.658+0000 7f7fa5df6700 1 -- 172.21.15.16:0/4222808790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7fa00cf4a0 msgr2=0x7f7fa00cd670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:56.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.658+0000 7f7fa5df6700 1 --2- 172.21.15.16:0/4222808790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7fa00cf4a0 0x7f7fa00cd670 secure :-1 s=READY pgs=3144 cs=0 l=1 rev1=1 crypto rx=0x7f7f880090b0 tx=0x7f7f88004750 comp rx=0 tx=0).stop 2023-10-16T13:48:56.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.658+0000 7f7fa5df6700 1 -- 172.21.15.16:0/4222808790 shutdown_connections 2023-10-16T13:48:56.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.658+0000 7f7fa5df6700 1 --2- 172.21.15.16:0/4222808790 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7fa00cf880 0x7f7fa01f2000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:56.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.658+0000 7f7fa5df6700 1 --2- 172.21.15.16:0/4222808790 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7fa00b5830 0x7f7fa014d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:56.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.658+0000 7f7fa5df6700 1 --2- 172.21.15.16:0/4222808790 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f7f8006b020 0x7f7f8006d4e0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:56.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.658+0000 7f7fa5df6700 1 --2- 172.21.15.16:0/4222808790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7fa00cf4a0 0x7f7fa00cd670 unknown :-1 s=CLOSED pgs=3144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:56.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.658+0000 7f7fa5df6700 1 -- 172.21.15.16:0/4222808790 >> 172.21.15.16:0/4222808790 conn(0x7f7fa00b1fd0 msgr2=0x7f7fa00724d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:56.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.658+0000 7f7fa5df6700 1 -- 172.21.15.16:0/4222808790 shutdown_connections 2023-10-16T13:48:56.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.658+0000 7f7fa5df6700 1 -- 172.21.15.16:0/4222808790 wait complete. 2023-10-16T13:48:56.676 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-16T13:48:56.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.807+0000 7f6422b5d700 1 Processor -- start 2023-10-16T13:48:56.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.807+0000 7f6422b5d700 1 -- start start 2023-10-16T13:48:56.814 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.808+0000 7f6422b5d700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f641c06e570 0x7f641c15e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:56.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.808+0000 7f6422b5d700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f641c15b6a0 con 0x7f641c15b2c0 2023-10-16T13:48:56.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.808+0000 7f6422b5d700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f641c161b20 con 0x7f641c06a7e0 2023-10-16T13:48:56.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.808+0000 7f6422b5d700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f641c161e00 con 0x7f641c06e570 2023-10-16T13:48:56.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.808+0000 7f64208f9700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f641c15b2c0 0x7f641c161340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48062/0 (socket says 172.21.15.16:48062) 2023-10-16T13:48:56.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.808+0000 7f64208f9700 1 -- 172.21.15.16:0/3676929907 learned_addr learned my addr 172.21.15.16:0/3676929907 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:48:56.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.808+0000 7f641bfff700 1 --2- 172.21.15.16:0/3676929907 >> v2:172.21.15.49:3300/0 conn(0x7f641c06e570 0x7f641c15e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:56.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.808+0000 7f641b7fe700 1 -- 172.21.15.16:0/3676929907 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3716720145 0 0) 0x7f641c15b6a0 con 0x7f641c15b2c0 2023-10-16T13:48:56.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.808+0000 7f641b7fe700 1 -- 172.21.15.16:0/3676929907 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f640c0035b0 con 0x7f641c15b2c0 2023-10-16T13:48:56.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.808+0000 7f641b7fe700 1 -- 172.21.15.16:0/3676929907 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2854928752 0 0) 0x7f641c161b20 con 0x7f641c06a7e0 2023-10-16T13:48:56.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.808+0000 7f641b7fe700 1 -- 172.21.15.16:0/3676929907 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f641c15b6a0 con 0x7f641c06a7e0 2023-10-16T13:48:56.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.808+0000 7f641b7fe700 1 -- 172.21.15.16:0/3676929907 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 863491011 0 0) 0x7f640c0035b0 con 0x7f641c15b2c0 2023-10-16T13:48:56.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.808+0000 7f641b7fe700 1 -- 172.21.15.16:0/3676929907 >> v2:172.21.15.49:3300/0 conn(0x7f641c06e570 msgr2=0x7f641c15e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:56.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.808+0000 7f641b7fe700 1 --2- 172.21.15.16:0/3676929907 >> v2:172.21.15.49:3300/0 conn(0x7f641c06e570 0x7f641c15e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:56.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.808+0000 7f641b7fe700 1 -- 172.21.15.16:0/3676929907 >> v1:172.21.15.49:6789/0 conn(0x7f641c06a7e0 legacy=0x7f641c071a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:56.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.808+0000 7f641b7fe700 1 -- 172.21.15.16:0/3676929907 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f641c1620e0 con 0x7f641c15b2c0 2023-10-16T13:48:56.817 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.808+0000 7f641b7fe700 1 -- 172.21.15.16:0/3676929907 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f6410001670 con 0x7f641c15b2c0 2023-10-16T13:48:56.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.808+0000 7f641b7fe700 1 -- 172.21.15.16:0/3676929907 >> v1:172.21.15.16:6789/0 conn(0x7f641c15b2c0 legacy=0x7f641c161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:56.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.808+0000 7f641b7fe700 1 --2- 172.21.15.16:0/3676929907 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f640c003be0 0x7f640c003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:56.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.808+0000 7f641b7fe700 1 --2- 172.21.15.16:0/3676929907 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f640c004850 0x7f640c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:56.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.808+0000 7f641b7fe700 1 --2- 172.21.15.16:0/3676929907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f640c005260 0x7f640c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:56.818 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.808+0000 7f641b7fe700 1 -- 172.21.15.16:0/3676929907 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f641c161e00 con 0x7f640c005260 2023-10-16T13:48:56.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.808+0000 7f641b7fe700 1 -- 172.21.15.16:0/3676929907 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f640c00c190 con 0x7f640c003be0 2023-10-16T13:48:56.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.808+0000 7f641b7fe700 1 -- 172.21.15.16:0/3676929907 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f640c00c350 con 0x7f640c004850 2023-10-16T13:48:56.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.809+0000 7f64210fa700 1 --2- 172.21.15.16:0/3676929907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f640c005260 0x7f640c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:56.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.809+0000 7f64208f9700 1 --2- 172.21.15.16:0/3676929907 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f640c003be0 0x7f640c003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:56.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.809+0000 7f641bfff700 1 --2- 172.21.15.16:0/3676929907 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f640c004850 0x7f640c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:56.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.809+0000 7f64210fa700 1 -- 172.21.15.16:0/3676929907 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f640c004850 msgr2=0x7f640c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:56.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.809+0000 7f64210fa700 1 --2- 172.21.15.16:0/3676929907 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f640c004850 0x7f640c004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:56.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.809+0000 7f64210fa700 1 -- 172.21.15.16:0/3676929907 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f640c003be0 msgr2=0x7f640c003fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:56.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.809+0000 7f64210fa700 1 --2- 172.21.15.16:0/3676929907 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f640c003be0 0x7f640c003fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:56.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.809+0000 7f64210fa700 1 -- 172.21.15.16:0/3676929907 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f640c0035b0 con 0x7f640c005260 2023-10-16T13:48:56.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.809+0000 7f64210fa700 1 -- 172.21.15.16:0/3676929907 reap_dead start 2023-10-16T13:48:56.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.809+0000 7f64210fa700 1 --2- 172.21.15.16:0/3676929907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f640c005260 0x7f640c00ba10 secure :-1 s=READY pgs=3146 cs=0 l=1 rev1=1 crypto rx=0x7f641400c2f0 tx=0x7f640c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a934fb2485d700ce server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:56.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.809+0000 7f641b7fe700 1 -- 172.21.15.16:0/3676929907 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6414001f30 con 0x7f640c005260 2023-10-16T13:48:56.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.809+0000 7f641b7fe700 1 -- 172.21.15.16:0/3676929907 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6414003a00 con 0x7f640c005260 2023-10-16T13:48:56.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.809+0000 7f641b7fe700 1 -- 172.21.15.16:0/3676929907 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6414003db0 con 0x7f640c005260 2023-10-16T13:48:56.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.810+0000 7f6422b5d700 1 -- 172.21.15.16:0/3676929907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f640c005260 msgr2=0x7f640c00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:56.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.810+0000 7f6422b5d700 1 --2- 172.21.15.16:0/3676929907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f640c005260 0x7f640c00ba10 secure :-1 s=READY pgs=3146 cs=0 l=1 rev1=1 crypto rx=0x7f641400c2f0 tx=0x7f640c005880 comp rx=0 tx=0).stop 2023-10-16T13:48:56.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.810+0000 7f6422b5d700 1 -- 172.21.15.16:0/3676929907 shutdown_connections 2023-10-16T13:48:56.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.810+0000 7f6422b5d700 1 --2- 172.21.15.16:0/3676929907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f640c005260 0x7f640c00ba10 unknown :-1 s=CLOSED pgs=3146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:56.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.810+0000 7f6422b5d700 1 -- 172.21.15.16:0/3676929907 >> 172.21.15.16:0/3676929907 conn(0x7f641c0b1f40 msgr2=0x7f641c0b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:56.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.810+0000 7f6422b5d700 1 -- 172.21.15.16:0/3676929907 shutdown_connections 2023-10-16T13:48:56.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.810+0000 7f6422b5d700 1 -- 172.21.15.16:0/3676929907 wait complete. 2023-10-16T13:48:56.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.811+0000 7f6422b5d700 1 Processor -- start 2023-10-16T13:48:56.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.811+0000 7f6422b5d700 1 -- start start 2023-10-16T13:48:56.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.812+0000 7f6422b5d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f640c005260 0x7f641c159680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:56.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.812+0000 7f6422b5d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f641c161300 0x7f641c0c7dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:56.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.812+0000 7f6422b5d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f641c0c9cd0 0x7f641c0ca160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:56.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.812+0000 7f6422b5d700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f641c0b3fc0 con 0x7f641c161300 2023-10-16T13:48:56.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.812+0000 7f6422b5d700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f641c0b42a0 con 0x7f641c0c9cd0 2023-10-16T13:48:56.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.812+0000 7f6422b5d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f641c0b4580 con 0x7f640c005260 2023-10-16T13:48:56.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.812+0000 7f641bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f641c161300 0x7f641c0c7dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:56.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.812+0000 7f641bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f641c161300 0x7f641c0c7dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58802/0 (socket says 172.21.15.16:58802) 2023-10-16T13:48:56.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.812+0000 7f641bfff700 1 -- 172.21.15.16:0/540807198 learned_addr learned my addr 172.21.15.16:0/540807198 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:56.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.812+0000 7f64208f9700 1 --2- 172.21.15.16:0/540807198 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f640c005260 0x7f641c159680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:56.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.812+0000 7f64210fa700 1 --2- 172.21.15.16:0/540807198 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f641c0c9cd0 0x7f641c0ca160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:56.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.812+0000 7f641bfff700 1 -- 172.21.15.16:0/540807198 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f640c005260 msgr2=0x7f641c159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:56.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.812+0000 7f641bfff700 1 --2- 172.21.15.16:0/540807198 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f640c005260 0x7f641c159680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:56.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.812+0000 7f641bfff700 1 -- 172.21.15.16:0/540807198 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f641c0c9cd0 msgr2=0x7f641c0ca160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:56.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.812+0000 7f641bfff700 1 --2- 172.21.15.16:0/540807198 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f641c0c9cd0 0x7f641c0ca160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:56.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.812+0000 7f641bfff700 1 -- 172.21.15.16:0/540807198 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f641c0b4860 con 0x7f641c161300 2023-10-16T13:48:56.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.813+0000 7f641bfff700 1 --2- 172.21.15.16:0/540807198 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f641c161300 0x7f641c0c7dd0 secure :-1 s=READY pgs=3147 cs=0 l=1 rev1=1 crypto rx=0x7f64080119a0 tx=0x7f6408011cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:56.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.813+0000 7f64197fa700 1 -- 172.21.15.16:0/540807198 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f640800f780 con 0x7f641c161300 2023-10-16T13:48:56.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.813+0000 7f64197fa700 1 -- 172.21.15.16:0/540807198 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f640800f8e0 con 0x7f641c161300 2023-10-16T13:48:56.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.813+0000 7f6422b5d700 1 -- 172.21.15.16:0/540807198 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f641c0ca6a0 con 0x7f641c161300 2023-10-16T13:48:56.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.813+0000 7f64197fa700 1 -- 172.21.15.16:0/540807198 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f640800fc70 con 0x7f641c161300 2023-10-16T13:48:56.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.813+0000 7f6422b5d700 1 -- 172.21.15.16:0/540807198 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f641c159bc0 con 0x7f641c161300 2023-10-16T13:48:56.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.814+0000 7f6422b5d700 1 -- 172.21.15.16:0/540807198 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f641c0b49f0 con 0x7f641c161300 2023-10-16T13:48:56.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.814+0000 7f64197fa700 1 -- 172.21.15.16:0/540807198 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 166) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6408014440 con 0x7f641c161300 2023-10-16T13:48:56.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.815+0000 7f64197fa700 1 --2- 172.21.15.16:0/540807198 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f640006b020 0x7f640006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:56.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.815+0000 7f64208f9700 1 --2- 172.21.15.16:0/540807198 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f640006b020 0x7f640006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:56.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.815+0000 7f64197fa700 1 -- 172.21.15.16:0/540807198 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 6165+0+0 (secure 0 0 0) 0x7f6408089f40 con 0x7f641c161300 2023-10-16T13:48:56.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.815+0000 7f64208f9700 1 --2- 172.21.15.16:0/540807198 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f640006b020 0x7f640006d4e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6410000c00 tx=0x7f6410000f80 comp rx=0 tx=0).ready entity=mgr.13344 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:56.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:56.820+0000 7f64197fa700 1 -- 172.21.15.16:0/540807198 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6408054d10 con 0x7f641c161300 2023-10-16T13:48:57.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.025+0000 7f6422b5d700 1 -- 172.21.15.16:0/540807198 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f641c0b4100 con 0x7f641c161300 2023-10-16T13:48:57.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.028+0000 7f64197fa700 1 -- 172.21.15.16:0/540807198 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v94)=0 v94) v1 ==== 122+0+0 (secure 0 0 0) 0x7f641c0b4100 con 0x7f641c161300 2023-10-16T13:48:57.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.030+0000 7f6422b5d700 1 -- 172.21.15.16:0/540807198 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f640006b020 msgr2=0x7f640006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:57.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.030+0000 7f6422b5d700 1 --2- 172.21.15.16:0/540807198 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f640006b020 0x7f640006d4e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6410000c00 tx=0x7f6410000f80 comp rx=0 tx=0).stop 2023-10-16T13:48:57.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.030+0000 7f6422b5d700 1 -- 172.21.15.16:0/540807198 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f641c161300 msgr2=0x7f641c0c7dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:57.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.030+0000 7f6422b5d700 1 --2- 172.21.15.16:0/540807198 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f641c161300 0x7f641c0c7dd0 secure :-1 s=READY pgs=3147 cs=0 l=1 rev1=1 crypto rx=0x7f64080119a0 tx=0x7f6408011cb0 comp rx=0 tx=0).stop 2023-10-16T13:48:57.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.030+0000 7f6422b5d700 1 -- 172.21.15.16:0/540807198 shutdown_connections 2023-10-16T13:48:57.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.030+0000 7f6422b5d700 1 --2- 172.21.15.16:0/540807198 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f641c0c9cd0 0x7f641c0ca160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:57.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.030+0000 7f6422b5d700 1 --2- 172.21.15.16:0/540807198 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f640c005260 0x7f641c159680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:57.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.030+0000 7f6422b5d700 1 --2- 172.21.15.16:0/540807198 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f640006b020 0x7f640006d4e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:57.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.030+0000 7f6422b5d700 1 --2- 172.21.15.16:0/540807198 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f641c161300 0x7f641c0c7dd0 unknown :-1 s=CLOSED pgs=3147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:57.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.030+0000 7f6422b5d700 1 -- 172.21.15.16:0/540807198 >> 172.21.15.16:0/540807198 conn(0x7f641c0b1f40 msgr2=0x7f641c0cb4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:57.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.030+0000 7f6422b5d700 1 -- 172.21.15.16:0/540807198 shutdown_connections 2023-10-16T13:48:57.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.030+0000 7f6422b5d700 1 -- 172.21.15.16:0/540807198 wait complete. 2023-10-16T13:48:57.046 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T13:48:57.046 DEBUG:teuthology.orchestra.run.smithi016:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T13:48:57.088 DEBUG:teuthology.orchestra.run.smithi049:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T13:48:57.140 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T13:48:57.140 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T13:48:57.141 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:48:57.158 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16287364k,nr_inodes=4071841,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20251 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3261172k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T13:48:57.159 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T13:48:57.159 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:48:57.179 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284204k,nr_inodes=4071051,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=753 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260540k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T13:48:57.180 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T13:48:57.180 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:48:57.216 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16287364k,nr_inodes=4071841,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20251 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3261172k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T13:48:57.217 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T13:48:57.217 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:48:57.239 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284204k,nr_inodes=4071051,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=753 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260540k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T13:48:57.239 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-16T13:48:57.240 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:48:57.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.378+0000 7f9324e35700 1 Processor -- start 2023-10-16T13:48:57.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.378+0000 7f9324e35700 1 -- start start 2023-10-16T13:48:57.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.378+0000 7f9324e35700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f932015d900 0x7f93200d2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:57.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.378+0000 7f9324e35700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93200cee70 con 0x7f93200d2f70 2023-10-16T13:48:57.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.378+0000 7f9324e35700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93200ce400 con 0x7f93200ce9f0 2023-10-16T13:48:57.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.378+0000 7f9324e35700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f932015dce0 con 0x7f932015d900 2023-10-16T13:48:57.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.378+0000 7f931ed9d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f93200d2f70 0x7f93200b6790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48082/0 (socket says 172.21.15.16:48082) 2023-10-16T13:48:57.388 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.378+0000 7f931ed9d700 1 -- 172.21.15.16:0/798790918 learned_addr learned my addr 172.21.15.16:0/798790918 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:48:57.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.379+0000 7f931d59a700 1 -- 172.21.15.16:0/798790918 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4187902591 0 0) 0x7f93200cee70 con 0x7f93200d2f70 2023-10-16T13:48:57.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.379+0000 7f931d59a700 1 -- 172.21.15.16:0/798790918 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93080035b0 con 0x7f93200d2f70 2023-10-16T13:48:57.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.379+0000 7f931e59c700 1 --2- 172.21.15.16:0/798790918 >> v2:172.21.15.49:3300/0 conn(0x7f932015d900 0x7f93200d2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:57.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.379+0000 7f931d59a700 1 -- 172.21.15.16:0/798790918 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1799141248 0 0) 0x7f93080035b0 con 0x7f93200d2f70 2023-10-16T13:48:57.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.379+0000 7f931d59a700 1 -- 172.21.15.16:0/798790918 >> v2:172.21.15.49:3300/0 conn(0x7f932015d900 msgr2=0x7f93200d2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:57.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.379+0000 7f931d59a700 1 --2- 172.21.15.16:0/798790918 >> v2:172.21.15.49:3300/0 conn(0x7f932015d900 0x7f93200d2040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:57.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.379+0000 7f931d59a700 1 -- 172.21.15.16:0/798790918 >> v1:172.21.15.49:6789/0 conn(0x7f93200ce9f0 legacy=0x7f93200d2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:57.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.379+0000 7f931d59a700 1 -- 172.21.15.16:0/798790918 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93200d1280 con 0x7f93200d2f70 2023-10-16T13:48:57.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.379+0000 7f931d59a700 1 -- 172.21.15.16:0/798790918 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f93180019f0 con 0x7f93200d2f70 2023-10-16T13:48:57.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.379+0000 7f931d59a700 1 -- 172.21.15.16:0/798790918 >> v1:172.21.15.16:6789/0 conn(0x7f93200d2f70 legacy=0x7f93200b6790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:57.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.379+0000 7f931d59a700 1 --2- 172.21.15.16:0/798790918 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9308003a50 0x7f9308003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:57.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.379+0000 7f931d59a700 1 --2- 172.21.15.16:0/798790918 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9308004850 0x7f9308004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:57.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.379+0000 7f931d59a700 1 --2- 172.21.15.16:0/798790918 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9308005260 0x7f930800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:57.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.379+0000 7f931d59a700 1 -- 172.21.15.16:0/798790918 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f932015dce0 con 0x7f9308004850 2023-10-16T13:48:57.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.379+0000 7f931d59a700 1 -- 172.21.15.16:0/798790918 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f930800c1a0 con 0x7f9308003a50 2023-10-16T13:48:57.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.379+0000 7f931d59a700 1 -- 172.21.15.16:0/798790918 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f930800c360 con 0x7f9308005260 2023-10-16T13:48:57.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.379+0000 7f931dd9b700 1 --2- 172.21.15.16:0/798790918 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9308004850 0x7f9308004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:57.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.379+0000 7f931e59c700 1 --2- 172.21.15.16:0/798790918 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9308003a50 0x7f9308003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:57.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.379+0000 7f931ed9d700 1 --2- 172.21.15.16:0/798790918 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9308005260 0x7f930800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:57.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.379+0000 7f931dd9b700 1 -- 172.21.15.16:0/798790918 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9308005260 msgr2=0x7f930800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:57.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.379+0000 7f931dd9b700 1 --2- 172.21.15.16:0/798790918 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9308005260 0x7f930800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:57.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.379+0000 7f931dd9b700 1 -- 172.21.15.16:0/798790918 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9308003a50 msgr2=0x7f9308003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:57.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.379+0000 7f931dd9b700 1 --2- 172.21.15.16:0/798790918 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9308003a50 0x7f9308003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:57.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.380+0000 7f931dd9b700 1 -- 172.21.15.16:0/798790918 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93080035b0 con 0x7f9308004850 2023-10-16T13:48:57.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.380+0000 7f931ed9d700 1 -- 172.21.15.16:0/798790918 reap_dead start 2023-10-16T13:48:57.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.380+0000 7f931dd9b700 1 --2- 172.21.15.16:0/798790918 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9308004850 0x7f9308004cd0 secure :-1 s=READY pgs=3149 cs=0 l=1 rev1=1 crypto rx=0x7f93100017c0 tx=0x7f9308005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e270ec09bd70d1a7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:57.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.380+0000 7f931d59a700 1 -- 172.21.15.16:0/798790918 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93100052d0 con 0x7f9308004850 2023-10-16T13:48:57.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.380+0000 7f931d59a700 1 -- 172.21.15.16:0/798790918 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9310005430 con 0x7f9308004850 2023-10-16T13:48:57.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.380+0000 7f931d59a700 1 -- 172.21.15.16:0/798790918 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93100038c0 con 0x7f9308004850 2023-10-16T13:48:57.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.380+0000 7f9324e35700 1 -- 172.21.15.16:0/798790918 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9308004850 msgr2=0x7f9308004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:57.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.380+0000 7f9324e35700 1 --2- 172.21.15.16:0/798790918 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9308004850 0x7f9308004cd0 secure :-1 s=READY pgs=3149 cs=0 l=1 rev1=1 crypto rx=0x7f93100017c0 tx=0x7f9308005880 comp rx=0 tx=0).stop 2023-10-16T13:48:57.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.380+0000 7f9324e35700 1 -- 172.21.15.16:0/798790918 shutdown_connections 2023-10-16T13:48:57.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.380+0000 7f9324e35700 1 --2- 172.21.15.16:0/798790918 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9308004850 0x7f9308004cd0 unknown :-1 s=CLOSED pgs=3149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:57.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.380+0000 7f9324e35700 1 -- 172.21.15.16:0/798790918 >> 172.21.15.16:0/798790918 conn(0x7f9320072330 msgr2=0x7f9320072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:57.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.380+0000 7f9324e35700 1 -- 172.21.15.16:0/798790918 shutdown_connections 2023-10-16T13:48:57.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.380+0000 7f9324e35700 1 -- 172.21.15.16:0/798790918 wait complete. 2023-10-16T13:48:57.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.382+0000 7f9324e35700 1 Processor -- start 2023-10-16T13:48:57.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.382+0000 7f9324e35700 1 -- start start 2023-10-16T13:48:57.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.384+0000 7f9324e35700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9308004850 0x7f93201f3860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:57.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.384+0000 7f9324e35700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93201f3da0 0x7f932015bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:57.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.384+0000 7f9324e35700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f93201f4220 0x7f932015c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:57.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.384+0000 7f9324e35700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93200c9650 con 0x7f93201f3da0 2023-10-16T13:48:57.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.384+0000 7f9324e35700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f932015e980 con 0x7f9308004850 2023-10-16T13:48:57.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.384+0000 7f9324e35700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f932015e050 con 0x7f93201f4220 2023-10-16T13:48:57.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.384+0000 7f931dd9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93201f3da0 0x7f932015bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:57.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.384+0000 7f931dd9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93201f3da0 0x7f932015bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58818/0 (socket says 172.21.15.16:58818) 2023-10-16T13:48:57.398 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.384+0000 7f931dd9b700 1 -- 172.21.15.16:0/2590404402 learned_addr learned my addr 172.21.15.16:0/2590404402 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:57.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.384+0000 7f931e59c700 1 --2- 172.21.15.16:0/2590404402 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9308004850 0x7f93201f3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:57.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.384+0000 7f931dd9b700 1 -- 172.21.15.16:0/2590404402 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f93201f4220 msgr2=0x7f932015c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:57.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.384+0000 7f931dd9b700 1 --2- 172.21.15.16:0/2590404402 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f93201f4220 0x7f932015c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:57.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.384+0000 7f931dd9b700 1 -- 172.21.15.16:0/2590404402 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9308004850 msgr2=0x7f93201f3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:57.399 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.384+0000 7f931dd9b700 1 --2- 172.21.15.16:0/2590404402 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9308004850 0x7f93201f3860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:57.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.384+0000 7f931dd9b700 1 -- 172.21.15.16:0/2590404402 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f932015e330 con 0x7f93201f3da0 2023-10-16T13:48:57.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.384+0000 7f931dd9b700 1 --2- 172.21.15.16:0/2590404402 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93201f3da0 0x7f932015bdb0 secure :-1 s=READY pgs=3150 cs=0 l=1 rev1=1 crypto rx=0x7f9310015040 tx=0x7f9310001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:57.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.384+0000 7f930effd700 1 -- 172.21.15.16:0/2590404402 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9310001290 con 0x7f93201f3da0 2023-10-16T13:48:57.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.384+0000 7f930effd700 1 -- 172.21.15.16:0/2590404402 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f931001d480 con 0x7f93201f3da0 2023-10-16T13:48:57.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.384+0000 7f9324e35700 1 -- 172.21.15.16:0/2590404402 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f932015e610 con 0x7f93201f3da0 2023-10-16T13:48:57.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.384+0000 7f930effd700 1 -- 172.21.15.16:0/2590404402 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f931001d830 con 0x7f93201f3da0 2023-10-16T13:48:57.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.384+0000 7f9324e35700 1 -- 172.21.15.16:0/2590404402 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93200c8c50 con 0x7f93201f3da0 2023-10-16T13:48:57.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.384+0000 7f9324e35700 1 -- 172.21.15.16:0/2590404402 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f932004ebb0 con 0x7f93201f3da0 2023-10-16T13:48:57.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.385+0000 7f930effd700 1 -- 172.21.15.16:0/2590404402 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 166) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9310019040 con 0x7f93201f3da0 2023-10-16T13:48:57.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.385+0000 7f930effd700 1 --2- 172.21.15.16:0/2590404402 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f930006b020 0x7f930006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:57.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.385+0000 7f930effd700 1 -- 172.21.15.16:0/2590404402 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 6165+0+0 (secure 0 0 0) 0x7f931008a410 con 0x7f93201f3da0 2023-10-16T13:48:57.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.385+0000 7f931e59c700 1 --2- 172.21.15.16:0/2590404402 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f930006b020 0x7f930006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:57.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.386+0000 7f931e59c700 1 --2- 172.21.15.16:0/2590404402 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f930006b020 0x7f930006d4e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f93140098f0 tx=0x7f9314008040 comp rx=0 tx=0).ready entity=mgr.13344 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:57.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.391+0000 7f930effd700 1 -- 172.21.15.16:0/2590404402 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9310055270 con 0x7f93201f3da0 2023-10-16T13:48:57.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.641+0000 7f9324e35700 1 -- 172.21.15.16:0/2590404402 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f93201e7130 con 0x7f93201f3da0 2023-10-16T13:48:57.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.642+0000 7f930effd700 1 -- 172.21.15.16:0/2590404402 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 148 v148) v1 ==== 95+0+3705 (secure 0 0 0) 0x7f931002de90 con 0x7f93201f3da0 2023-10-16T13:48:57.646 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:48:57.646 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":148,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10112,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/712583564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":712583564},{"type":"v1","addr":"172.21.15.16:6837","nonce":712583564}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":141},{"gid":10406,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/3633071569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":3633071569},{"type":"v1","addr":"172.21.15.49:6835","nonce":3633071569}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":148},{"gid":11836,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/3551114031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":3551114031},{"type":"v1","addr":"172.21.15.49:6834","nonce":3551114031}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":141},{"gid":13050,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/2135643757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":2135643757},{"type":"v1","addr":"172.21.15.16:6836","nonce":2135643757}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":140}],"filesystems":[]} 2023-10-16T13:48:57.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.644+0000 7f9324e35700 1 -- 172.21.15.16:0/2590404402 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f930006b020 msgr2=0x7f930006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:57.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.644+0000 7f9324e35700 1 --2- 172.21.15.16:0/2590404402 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f930006b020 0x7f930006d4e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f93140098f0 tx=0x7f9314008040 comp rx=0 tx=0).stop 2023-10-16T13:48:57.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.644+0000 7f9324e35700 1 -- 172.21.15.16:0/2590404402 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93201f3da0 msgr2=0x7f932015bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:57.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.644+0000 7f9324e35700 1 --2- 172.21.15.16:0/2590404402 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93201f3da0 0x7f932015bdb0 secure :-1 s=READY pgs=3150 cs=0 l=1 rev1=1 crypto rx=0x7f9310015040 tx=0x7f9310001120 comp rx=0 tx=0).stop 2023-10-16T13:48:57.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.644+0000 7f9324e35700 1 -- 172.21.15.16:0/2590404402 shutdown_connections 2023-10-16T13:48:57.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.644+0000 7f9324e35700 1 --2- 172.21.15.16:0/2590404402 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f93201f4220 0x7f932015c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:57.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.644+0000 7f9324e35700 1 --2- 172.21.15.16:0/2590404402 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9308004850 0x7f93201f3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:57.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.644+0000 7f9324e35700 1 --2- 172.21.15.16:0/2590404402 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f930006b020 0x7f930006d4e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:57.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.644+0000 7f9324e35700 1 --2- 172.21.15.16:0/2590404402 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93201f3da0 0x7f932015bdb0 unknown :-1 s=CLOSED pgs=3150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:57.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.644+0000 7f9324e35700 1 -- 172.21.15.16:0/2590404402 >> 172.21.15.16:0/2590404402 conn(0x7f9320072330 msgr2=0x7f93201e60b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:57.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.644+0000 7f9324e35700 1 -- 172.21.15.16:0/2590404402 shutdown_connections 2023-10-16T13:48:57.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.644+0000 7f9324e35700 1 -- 172.21.15.16:0/2590404402 wait complete. 2023-10-16T13:48:57.651 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 148 2023-10-16T13:48:57.660 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-16T13:48:57.661 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-16T13:48:57.661 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-16T13:48:57.661 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T13:48:57.661 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-16T13:48:57.662 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-16T13:48:57.662 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-16T13:48:57.662 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T13:48:57.662 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-16T13:48:57.663 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-16T13:48:57.663 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-16T13:48:57.663 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T13:48:57.663 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-16T13:48:57.664 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-16T13:48:57.664 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-16T13:48:57.664 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T13:48:57.669 INFO:tasks.ceph.mds.a:Stopped 2023-10-16T13:48:57.670 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-16T13:48:57.671 INFO:tasks.ceph.mds.c:Stopped 2023-10-16T13:48:57.671 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-16T13:48:57.674 INFO:tasks.ceph.mds.b:Stopped 2023-10-16T13:48:57.674 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-16T13:48:57.675 INFO:tasks.ceph.mds.d:Stopped 2023-10-16T13:48:57.675 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-16T13:48:57.713 INFO:tasks.ceph.mds.a:Started 2023-10-16T13:48:57.713 DEBUG:teuthology.parallel:result is None 2023-10-16T13:48:57.714 INFO:tasks.ceph.mds.c:Started 2023-10-16T13:48:57.714 DEBUG:teuthology.parallel:result is None 2023-10-16T13:48:57.717 INFO:tasks.ceph.mds.b:Started 2023-10-16T13:48:57.717 DEBUG:teuthology.parallel:result is None 2023-10-16T13:48:57.718 INFO:tasks.ceph.mds.d:Started 2023-10-16T13:48:57.718 DEBUG:teuthology.parallel:result is None 2023-10-16T13:48:57.718 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-16T13:48:57.794 INFO:tasks.ceph.mds.a.smithi016.stdout:starting mds.a at 2023-10-16T13:48:57.795 INFO:tasks.ceph.mds.c.smithi016.stdout:starting mds.c at 2023-10-16T13:48:57.809 INFO:tasks.ceph.mds.d.smithi049.stdout:starting mds.d at 2023-10-16T13:48:57.809 INFO:tasks.ceph.mds.b.smithi049.stdout:starting mds.b at 2023-10-16T13:48:57.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.845+0000 7ffbd59f2700 1 Processor -- start 2023-10-16T13:48:57.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.845+0000 7ffbd59f2700 1 -- start start 2023-10-16T13:48:57.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.846+0000 7ffbd59f2700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ffbd00cf4a0 0x7ffbd00726e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:57.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.846+0000 7ffbd59f2700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7ffbd00cf880 0x7ffbd00b52f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:57.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.846+0000 7ffbd59f2700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbd015bc80 con 0x7ffbd00b5830 2023-10-16T13:48:57.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.846+0000 7ffbd59f2700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbd0072c20 con 0x7ffbd00cf4a0 2023-10-16T13:48:57.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.846+0000 7ffbd59f2700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbd015b070 con 0x7ffbd00cf880 2023-10-16T13:48:57.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.846+0000 7ffbceffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ffbd00cf4a0 0x7ffbd00726e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:57.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.846+0000 7ffbcf7fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ffbd00b5830 0x7ffbd015a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48126/0 (socket says 172.21.15.16:48126) 2023-10-16T13:48:57.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.846+0000 7ffbceffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ffbd00cf4a0 0x7ffbd00726e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58848/0 (socket says 172.21.15.16:58848) 2023-10-16T13:48:57.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.846+0000 7ffbcf7fe700 1 -- 172.21.15.16:0/4169343534 learned_addr learned my addr 172.21.15.16:0/4169343534 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:48:57.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.847+0000 7ffbce7fc700 1 --2- 172.21.15.16:0/4169343534 >> v2:172.21.15.49:3300/0 conn(0x7ffbd00cf880 0x7ffbd00b52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:57.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.847+0000 7ffbceffd700 1 -- 172.21.15.16:0/4169343534 >> v2:172.21.15.49:3300/0 conn(0x7ffbd00cf880 msgr2=0x7ffbd00b52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:57.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.847+0000 7ffbceffd700 1 --2- 172.21.15.16:0/4169343534 >> v2:172.21.15.49:3300/0 conn(0x7ffbd00cf880 0x7ffbd00b52f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:57.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.847+0000 7ffbceffd700 1 -- 172.21.15.16:0/4169343534 >> v1:172.21.15.16:6789/0 conn(0x7ffbd00b5830 legacy=0x7ffbd015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:57.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.847+0000 7ffbceffd700 1 -- 172.21.15.16:0/4169343534 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbd015b350 con 0x7ffbd00cf4a0 2023-10-16T13:48:57.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.847+0000 7ffbceffd700 1 --2- 172.21.15.16:0/4169343534 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffbd00cf4a0 0x7ffbd00726e0 secure :-1 s=READY pgs=3160 cs=0 l=1 rev1=1 crypto rx=0x7ffbd0071da0 tx=0x7ffbc4009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3f6ef3ce1d12b04 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:57.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.847+0000 7ffbcdffb700 1 -- 172.21.15.16:0/4169343534 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbc4011830 con 0x7ffbd00cf4a0 2023-10-16T13:48:57.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.847+0000 7ffbcdffb700 1 -- 172.21.15.16:0/4169343534 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbc4011990 con 0x7ffbd00cf4a0 2023-10-16T13:48:57.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.847+0000 7ffbcdffb700 1 -- 172.21.15.16:0/4169343534 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbc4011d20 con 0x7ffbd00cf4a0 2023-10-16T13:48:57.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.848+0000 7ffbd59f2700 1 -- 172.21.15.16:0/4169343534 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffbd00cf4a0 msgr2=0x7ffbd00726e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:57.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.848+0000 7ffbd59f2700 1 --2- 172.21.15.16:0/4169343534 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffbd00cf4a0 0x7ffbd00726e0 secure :-1 s=READY pgs=3160 cs=0 l=1 rev1=1 crypto rx=0x7ffbd0071da0 tx=0x7ffbc4009b20 comp rx=0 tx=0).stop 2023-10-16T13:48:57.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.848+0000 7ffbd59f2700 1 -- 172.21.15.16:0/4169343534 shutdown_connections 2023-10-16T13:48:57.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.848+0000 7ffbd59f2700 1 --2- 172.21.15.16:0/4169343534 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffbd00cf4a0 0x7ffbd00726e0 unknown :-1 s=CLOSED pgs=3160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:57.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.848+0000 7ffbd59f2700 1 --2- 172.21.15.16:0/4169343534 >> v2:172.21.15.49:3300/0 conn(0x7ffbd00cf880 0x7ffbd00b52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:57.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.848+0000 7ffbd59f2700 1 -- 172.21.15.16:0/4169343534 >> 172.21.15.16:0/4169343534 conn(0x7ffbd00b1fd0 msgr2=0x7ffbd00b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:57.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.848+0000 7ffbd59f2700 1 -- 172.21.15.16:0/4169343534 shutdown_connections 2023-10-16T13:48:57.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.848+0000 7ffbd59f2700 1 -- 172.21.15.16:0/4169343534 wait complete. 2023-10-16T13:48:57.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.849+0000 7ffbd59f2700 1 Processor -- start 2023-10-16T13:48:57.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.849+0000 7ffbd59f2700 1 -- start start 2023-10-16T13:48:57.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.850+0000 7ffbd59f2700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ffbd00b5830 0x7ffbd00c5830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:57.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.850+0000 7ffbd59f2700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffbd00cf4a0 0x7ffbd0158630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:57.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.850+0000 7ffbd59f2700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ffbd00cf880 0x7ffbd015c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:57.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.850+0000 7ffbd59f2700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd00c5d70 con 0x7ffbd00cf4a0 2023-10-16T13:48:57.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.850+0000 7ffbd59f2700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd00c5410 con 0x7ffbd00cf880 2023-10-16T13:48:57.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.850+0000 7ffbd59f2700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd00c4650 con 0x7ffbd00b5830 2023-10-16T13:48:57.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.850+0000 7ffbce7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffbd00cf4a0 0x7ffbd0158630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:57.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.850+0000 7ffbce7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffbd00cf4a0 0x7ffbd0158630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58856/0 (socket says 172.21.15.16:58856) 2023-10-16T13:48:57.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.850+0000 7ffbce7fc700 1 -- 172.21.15.16:0/2802478835 learned_addr learned my addr 172.21.15.16:0/2802478835 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:57.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.850+0000 7ffbceffd700 1 --2- 172.21.15.16:0/2802478835 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ffbd00b5830 0x7ffbd00c5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:57.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.850+0000 7ffbcf7fe700 1 --2- 172.21.15.16:0/2802478835 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ffbd00cf880 0x7ffbd015c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:57.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.850+0000 7ffbce7fc700 1 -- 172.21.15.16:0/2802478835 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ffbd00b5830 msgr2=0x7ffbd00c5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:57.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.850+0000 7ffbce7fc700 1 --2- 172.21.15.16:0/2802478835 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ffbd00b5830 0x7ffbd00c5830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:57.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.850+0000 7ffbce7fc700 1 -- 172.21.15.16:0/2802478835 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ffbd00cf880 msgr2=0x7ffbd015c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:57.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.850+0000 7ffbce7fc700 1 --2- 172.21.15.16:0/2802478835 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ffbd00cf880 0x7ffbd015c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:57.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.850+0000 7ffbce7fc700 1 -- 172.21.15.16:0/2802478835 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbc40094a0 con 0x7ffbd00cf4a0 2023-10-16T13:48:57.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.850+0000 7ffbceffd700 1 --2- 172.21.15.16:0/2802478835 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ffbd00b5830 0x7ffbd00c5830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:48:57.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.850+0000 7ffbce7fc700 1 --2- 172.21.15.16:0/2802478835 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffbd00cf4a0 0x7ffbd0158630 secure :-1 s=READY pgs=3161 cs=0 l=1 rev1=1 crypto rx=0x7ffbc000e9b0 tx=0x7ffbc000ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:57.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.851+0000 7ffbb7fff700 1 -- 172.21.15.16:0/2802478835 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbc0009710 con 0x7ffbd00cf4a0 2023-10-16T13:48:57.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.851+0000 7ffbb7fff700 1 -- 172.21.15.16:0/2802478835 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbc000c850 con 0x7ffbd00cf4a0 2023-10-16T13:48:57.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.851+0000 7ffbd59f2700 1 -- 172.21.15.16:0/2802478835 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbd00c4930 con 0x7ffbd00cf4a0 2023-10-16T13:48:57.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.851+0000 7ffbb7fff700 1 -- 172.21.15.16:0/2802478835 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbc000cbc0 con 0x7ffbd00cf4a0 2023-10-16T13:48:57.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.851+0000 7ffbd59f2700 1 -- 172.21.15.16:0/2802478835 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbd00c4ef0 con 0x7ffbd00cf4a0 2023-10-16T13:48:57.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.852+0000 7ffbd59f2700 1 -- 172.21.15.16:0/2802478835 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbd004ebb0 con 0x7ffbd00cf4a0 2023-10-16T13:48:57.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.852+0000 7ffbb7fff700 1 -- 172.21.15.16:0/2802478835 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 166) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ffbc000cd20 con 0x7ffbd00cf4a0 2023-10-16T13:48:57.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.853+0000 7ffbb7fff700 1 --2- 172.21.15.16:0/2802478835 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7ffbbc06b020 0x7ffbbc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:57.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.853+0000 7ffbb7fff700 1 -- 172.21.15.16:0/2802478835 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 6165+0+0 (secure 0 0 0) 0x7ffbc0089a90 con 0x7ffbd00cf4a0 2023-10-16T13:48:57.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.853+0000 7ffbceffd700 1 --2- 172.21.15.16:0/2802478835 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7ffbbc06b020 0x7ffbbc06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:57.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.853+0000 7ffbceffd700 1 --2- 172.21.15.16:0/2802478835 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7ffbbc06b020 0x7ffbbc06d4e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ffbc40090b0 tx=0x7ffbc4008030 comp rx=0 tx=0).ready entity=mgr.13344 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:57.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:57.858+0000 7ffbb7fff700 1 -- 172.21.15.16:0/2802478835 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ffbc0054850 con 0x7ffbd00cf4a0 2023-10-16T13:48:58.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:58.077+0000 7ffbd59f2700 1 -- 172.21.15.16:0/2802478835 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7ffbd006ca50 con 0x7ffbd00cf4a0 2023-10-16T13:48:58.668 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:58.666+0000 7ffbb7fff700 1 -- 172.21.15.16:0/2802478835 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v272) v1 ==== 97+0+0 (secure 0 0 0) 0x7ffbc0021080 con 0x7ffbd00cf4a0 2023-10-16T13:48:58.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:58.669+0000 7ffbd59f2700 1 -- 172.21.15.16:0/2802478835 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7ffbbc06b020 msgr2=0x7ffbbc06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:58.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:58.669+0000 7ffbd59f2700 1 --2- 172.21.15.16:0/2802478835 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7ffbbc06b020 0x7ffbbc06d4e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ffbc40090b0 tx=0x7ffbc4008030 comp rx=0 tx=0).stop 2023-10-16T13:48:58.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:58.669+0000 7ffbd59f2700 1 -- 172.21.15.16:0/2802478835 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffbd00cf4a0 msgr2=0x7ffbd0158630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:58.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:58.669+0000 7ffbd59f2700 1 --2- 172.21.15.16:0/2802478835 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffbd00cf4a0 0x7ffbd0158630 secure :-1 s=READY pgs=3161 cs=0 l=1 rev1=1 crypto rx=0x7ffbc000e9b0 tx=0x7ffbc000ed70 comp rx=0 tx=0).stop 2023-10-16T13:48:58.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:58.670+0000 7ffbd59f2700 1 -- 172.21.15.16:0/2802478835 shutdown_connections 2023-10-16T13:48:58.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:58.670+0000 7ffbd59f2700 1 --2- 172.21.15.16:0/2802478835 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ffbd00cf880 0x7ffbd015c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:58.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:58.670+0000 7ffbd59f2700 1 --2- 172.21.15.16:0/2802478835 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ffbd00b5830 0x7ffbd00c5830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:58.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:58.670+0000 7ffbd59f2700 1 --2- 172.21.15.16:0/2802478835 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7ffbbc06b020 0x7ffbbc06d4e0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:58.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:58.670+0000 7ffbd59f2700 1 --2- 172.21.15.16:0/2802478835 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffbd00cf4a0 0x7ffbd0158630 unknown :-1 s=CLOSED pgs=3161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:58.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:58.670+0000 7ffbd59f2700 1 -- 172.21.15.16:0/2802478835 >> 172.21.15.16:0/2802478835 conn(0x7ffbd00b1fd0 msgr2=0x7ffbd015d3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:58.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:58.670+0000 7ffbd59f2700 1 -- 172.21.15.16:0/2802478835 shutdown_connections 2023-10-16T13:48:58.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:58.670+0000 7ffbd59f2700 1 -- 172.21.15.16:0/2802478835 wait complete. 2023-10-16T13:48:58.675 INFO:teuthology.orchestra.run.smithi016.stderr: removed all blocklist entries 2023-10-16T13:48:58.689 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-16T13:48:58.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:58.818+0000 7f018c60b700 1 Processor -- start 2023-10-16T13:48:58.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:58.818+0000 7f018c60b700 1 -- start start 2023-10-16T13:48:58.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:58.819+0000 7f018c60b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f018415b860 0x7f018406a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:58.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:58.819+0000 7f018c60b700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f01840b2be0 0x7f018406ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:58.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:58.819+0000 7f018c60b700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01840d11c0 con 0x7f01840cd770 2023-10-16T13:48:58.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:58.819+0000 7f018c60b700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01840d03d0 con 0x7f018415b860 2023-10-16T13:48:58.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:58.819+0000 7f018c60b700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01840cd360 con 0x7f01840b2be0 2023-10-16T13:48:58.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:58.819+0000 7f018a3a7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f018415b860 0x7f018406a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:58.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:58.819+0000 7f018aba8700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f01840cd770 0x7f01840719e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48138/0 (socket says 172.21.15.16:48138) 2023-10-16T13:48:58.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:58.819+0000 7f018aba8700 1 -- 172.21.15.16:0/3112359277 learned_addr learned my addr 172.21.15.16:0/3112359277 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:48:58.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:58.819+0000 7f018a3a7700 1 --2- 172.21.15.16:0/3112359277 >> v2:172.21.15.16:3300/0 conn(0x7f018415b860 0x7f018406a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58864/0 (socket says 172.21.15.16:58864) 2023-10-16T13:48:58.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:58.820+0000 7f0189ba6700 1 --2- 172.21.15.16:0/3112359277 >> v2:172.21.15.49:3300/0 conn(0x7f01840b2be0 0x7f018406ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:58.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:58.820+0000 7f018a3a7700 1 -- 172.21.15.16:0/3112359277 >> v2:172.21.15.49:3300/0 conn(0x7f01840b2be0 msgr2=0x7f018406ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:58.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:58.820+0000 7f018a3a7700 1 --2- 172.21.15.16:0/3112359277 >> v2:172.21.15.49:3300/0 conn(0x7f01840b2be0 0x7f018406ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:58.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:58.820+0000 7f018a3a7700 1 -- 172.21.15.16:0/3112359277 >> v1:172.21.15.16:6789/0 conn(0x7f01840cd770 legacy=0x7f01840719e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:58.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:58.820+0000 7f018a3a7700 1 -- 172.21.15.16:0/3112359277 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01840d05b0 con 0x7f018415b860 2023-10-16T13:48:58.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:58.820+0000 7f01893a5700 1 -- 172.21.15.16:0/3112359277 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1106112206 0 0) 0x7f01840d11c0 con 0x7f01840cd770 2023-10-16T13:48:58.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:58.820+0000 7f018a3a7700 1 --2- 172.21.15.16:0/3112359277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f018415b860 0x7f018406a7e0 secure :-1 s=READY pgs=3163 cs=0 l=1 rev1=1 crypto rx=0x7f01840675a0 tx=0x7f0174009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a0b3d23678a8bc8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:58.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:58.820+0000 7f01893a5700 1 -- 172.21.15.16:0/3112359277 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0174011830 con 0x7f018415b860 2023-10-16T13:48:58.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:58.820+0000 7f01893a5700 1 -- 172.21.15.16:0/3112359277 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0174011990 con 0x7f018415b860 2023-10-16T13:48:58.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:58.820+0000 7f01893a5700 1 -- 172.21.15.16:0/3112359277 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0174011d20 con 0x7f018415b860 2023-10-16T13:48:58.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:58.821+0000 7f018c60b700 1 -- 172.21.15.16:0/3112359277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f018415b860 msgr2=0x7f018406a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:58.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:58.821+0000 7f018c60b700 1 --2- 172.21.15.16:0/3112359277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f018415b860 0x7f018406a7e0 secure :-1 s=READY pgs=3163 cs=0 l=1 rev1=1 crypto rx=0x7f01840675a0 tx=0x7f0174009b20 comp rx=0 tx=0).stop 2023-10-16T13:48:58.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:58.821+0000 7f018c60b700 1 -- 172.21.15.16:0/3112359277 shutdown_connections 2023-10-16T13:48:58.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:58.821+0000 7f018c60b700 1 --2- 172.21.15.16:0/3112359277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f018415b860 0x7f018406a7e0 unknown :-1 s=CLOSED pgs=3163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:58.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:58.821+0000 7f018c60b700 1 --2- 172.21.15.16:0/3112359277 >> v2:172.21.15.49:3300/0 conn(0x7f01840b2be0 0x7f018406ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:58.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:58.821+0000 7f018c60b700 1 -- 172.21.15.16:0/3112359277 >> 172.21.15.16:0/3112359277 conn(0x7f01840b1fd0 msgr2=0x7f01840b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:58.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:58.821+0000 7f018c60b700 1 -- 172.21.15.16:0/3112359277 shutdown_connections 2023-10-16T13:48:58.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:58.821+0000 7f018c60b700 1 -- 172.21.15.16:0/3112359277 wait complete. 2023-10-16T13:48:58.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:58.822+0000 7f018c60b700 1 Processor -- start 2023-10-16T13:48:58.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:58.822+0000 7f018c60b700 1 -- start start 2023-10-16T13:48:58.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:58.823+0000 7f018c60b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f01840b2be0 0x7f01840a9560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:58.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:58.823+0000 7f018c60b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f01840cd770 0x7f01840c9620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:58.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:58.823+0000 7f018c60b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f018415b860 0x7f01840b4130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:58.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:58.823+0000 7f018c60b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01840c9b60 con 0x7f018415b860 2023-10-16T13:48:58.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:58.823+0000 7f018c60b700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01840a84c0 con 0x7f01840b2be0 2023-10-16T13:48:58.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:58.823+0000 7f018c60b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01840a87a0 con 0x7f01840cd770 2023-10-16T13:48:58.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:58.823+0000 7f018aba8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f018415b860 0x7f01840b4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:58.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:58.823+0000 7f018aba8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f018415b860 0x7f01840b4130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58870/0 (socket says 172.21.15.16:58870) 2023-10-16T13:48:58.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:58.823+0000 7f018aba8700 1 -- 172.21.15.16:0/4236184505 learned_addr learned my addr 172.21.15.16:0/4236184505 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:58.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:58.823+0000 7f018a3a7700 1 --2- 172.21.15.16:0/4236184505 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f01840b2be0 0x7f01840a9560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:58.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:58.823+0000 7f0189ba6700 1 --2- 172.21.15.16:0/4236184505 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f01840cd770 0x7f01840c9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:58.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:58.823+0000 7f018aba8700 1 -- 172.21.15.16:0/4236184505 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f01840cd770 msgr2=0x7f01840c9620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:58.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:58.823+0000 7f018aba8700 1 --2- 172.21.15.16:0/4236184505 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f01840cd770 0x7f01840c9620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:58.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:58.823+0000 7f018aba8700 1 -- 172.21.15.16:0/4236184505 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f01840b2be0 msgr2=0x7f01840a9560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:58.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:58.823+0000 7f018aba8700 1 --2- 172.21.15.16:0/4236184505 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f01840b2be0 0x7f01840a9560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:58.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:58.823+0000 7f018aba8700 1 -- 172.21.15.16:0/4236184505 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01740094a0 con 0x7f018415b860 2023-10-16T13:48:58.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:58.823+0000 7f018a3a7700 1 --2- 172.21.15.16:0/4236184505 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f01840b2be0 0x7f01840a9560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:48:58.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:58.824+0000 7f018aba8700 1 --2- 172.21.15.16:0/4236184505 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f018415b860 0x7f01840b4130 secure :-1 s=READY pgs=3164 cs=0 l=1 rev1=1 crypto rx=0x7f017c00cca0 tx=0x7f017c00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:58.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:58.824+0000 7f017affd700 1 -- 172.21.15.16:0/4236184505 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f017c00cea0 con 0x7f018415b860 2023-10-16T13:48:58.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:58.824+0000 7f017affd700 1 -- 172.21.15.16:0/4236184505 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f017c00ade0 con 0x7f018415b860 2023-10-16T13:48:58.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:58.824+0000 7f018c60b700 1 -- 172.21.15.16:0/4236184505 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01840a8a80 con 0x7f018415b860 2023-10-16T13:48:58.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:58.824+0000 7f017affd700 1 -- 172.21.15.16:0/4236184505 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f017c007310 con 0x7f018415b860 2023-10-16T13:48:58.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:58.824+0000 7f018c60b700 1 -- 172.21.15.16:0/4236184505 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01840a9040 con 0x7f018415b860 2023-10-16T13:48:58.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:58.825+0000 7f018c60b700 1 -- 172.21.15.16:0/4236184505 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01840a9260 con 0x7f018415b860 2023-10-16T13:48:58.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:58.825+0000 7f017affd700 1 -- 172.21.15.16:0/4236184505 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 166) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f017c007470 con 0x7f018415b860 2023-10-16T13:48:58.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:58.826+0000 7f017affd700 1 --2- 172.21.15.16:0/4236184505 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f016806b020 0x7f016806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:58.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:58.826+0000 7f017affd700 1 -- 172.21.15.16:0/4236184505 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(274..274 src has 1..274) v4 ==== 5305+0+0 (secure 0 0 0) 0x7f017c0892a0 con 0x7f018415b860 2023-10-16T13:48:58.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:58.826+0000 7f018a3a7700 1 --2- 172.21.15.16:0/4236184505 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f016806b020 0x7f016806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:58.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:58.826+0000 7f018a3a7700 1 --2- 172.21.15.16:0/4236184505 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f016806b020 0x7f016806d4e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0174000c00 tx=0x7f0174008030 comp rx=0 tx=0).ready entity=mgr.13344 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:58.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:58.831+0000 7f017affd700 1 -- 172.21.15.16:0/4236184505 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f017c0543d0 con 0x7f018415b860 2023-10-16T13:48:59.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.115+0000 7f018c60b700 1 -- 172.21.15.16:0/4236184505 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f018404ebb0 con 0x7f018415b860 2023-10-16T13:48:59.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.116+0000 7f017affd700 1 -- 172.21.15.16:0/4236184505 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v58) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f017c0579f0 con 0x7f018415b860 2023-10-16T13:48:59.118 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:48:59.119 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-10-16T13:48:59.119 INFO:teuthology.orchestra.run.smithi016.stdout: "auth_dump": [ 2023-10-16T13:48:59.119 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:48:59.119 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.a", 2023-10-16T13:48:59.119 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOKC1l/rBtOhAA/ob5WUvpz8Glp4E+xPZOYA==", 2023-10-16T13:48:59.120 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:48:59.120 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T13:48:59.120 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-10-16T13:48:59.120 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-10-16T13:48:59.120 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:48:59.121 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:48:59.121 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:48:59.121 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:48:59.121 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.b", 2023-10-16T13:48:59.121 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1lKAVsCxAAvhXuLY6i1+Iz1yq5BVPMNQ==", 2023-10-16T13:48:59.121 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:48:59.122 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T13:48:59.122 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-10-16T13:48:59.122 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-10-16T13:48:59.122 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:48:59.122 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:48:59.123 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:48:59.123 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:48:59.123 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.c", 2023-10-16T13:48:59.123 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1lvBowBhAA3waJNYhdgs8PGCYqYNYUog==", 2023-10-16T13:48:59.123 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:48:59.123 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T13:48:59.124 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-10-16T13:48:59.124 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-10-16T13:48:59.124 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:48:59.124 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:48:59.124 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:48:59.124 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:48:59.125 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.d", 2023-10-16T13:48:59.125 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1lS/fnEhAAfNYgQ5BW86JMjYm/bBfnaQ==", 2023-10-16T13:48:59.125 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:48:59.125 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T13:48:59.125 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-10-16T13:48:59.126 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-10-16T13:48:59.126 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:48:59.126 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:48:59.126 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:48:59.126 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:48:59.126 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.0", 2023-10-16T13:48:59.127 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAQKC1lbxllOhAAZeLdp2GMl6HqQPaQjKlNqw==", 2023-10-16T13:48:59.127 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:48:59.127 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T13:48:59.127 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T13:48:59.127 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:48:59.127 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:48:59.128 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:48:59.128 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:48:59.128 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.1", 2023-10-16T13:48:59.128 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQATKC1lnU6KOhAAHxvlbmzBptf1keGgL+Rbkg==", 2023-10-16T13:48:59.128 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:48:59.129 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T13:48:59.129 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T13:48:59.129 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:48:59.129 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:48:59.129 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:48:59.129 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:48:59.130 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.2", 2023-10-16T13:48:59.130 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAXKC1l9jFyARAAr/NAwtWhBMut9sAAoDEGWg==", 2023-10-16T13:48:59.130 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:48:59.130 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T13:48:59.130 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T13:48:59.130 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:48:59.131 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:48:59.131 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:48:59.131 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:48:59.131 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.3", 2023-10-16T13:48:59.131 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAaKC1lbw/lARAAGJ/9DNb5/7kLFSq7vrLtXg==", 2023-10-16T13:48:59.132 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:48:59.132 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T13:48:59.132 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T13:48:59.132 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:48:59.132 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:48:59.133 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:48:59.133 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:48:59.133 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.4", 2023-10-16T13:48:59.133 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAeKC1lZb7jERAAYpvb1xcSli9CTq7L8ChLOA==", 2023-10-16T13:48:59.133 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:48:59.133 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T13:48:59.134 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T13:48:59.134 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:48:59.134 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:48:59.134 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:48:59.134 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:48:59.134 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.5", 2023-10-16T13:48:59.135 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAhKC1l/4HuEBAAqfnRqjknij9KES/fqiyIBQ==", 2023-10-16T13:48:59.135 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:48:59.135 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T13:48:59.135 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T13:48:59.135 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:48:59.135 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:48:59.136 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:48:59.136 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:48:59.136 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.6", 2023-10-16T13:48:59.136 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAkKC1lzoUkEBAA+Q82OiBMNag8+R316WQtUw==", 2023-10-16T13:48:59.136 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:48:59.136 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T13:48:59.137 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T13:48:59.137 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:48:59.137 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:48:59.137 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:48:59.137 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:48:59.138 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.7", 2023-10-16T13:48:59.138 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAnKC1ljUu4DxAAc+G4anJANC2GhLJ8wb6gYg==", 2023-10-16T13:48:59.138 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:48:59.138 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2023-10-16T13:48:59.138 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2023-10-16T13:48:59.138 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:48:59.139 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:48:59.139 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:48:59.139 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:48:59.139 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.0", 2023-10-16T13:48:59.139 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1l2l6PFxAA9QyXtBAHdGAXpwH0JfknuQ==", 2023-10-16T13:48:59.139 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:48:59.140 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T13:48:59.140 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow r", 2023-10-16T13:48:59.140 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-16T13:48:59.140 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:48:59.140 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:48:59.141 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:48:59.141 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.1", 2023-10-16T13:48:59.141 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1lAcHFIxAAcNt2bmv2R6j5L3DYkdw7eQ==", 2023-10-16T13:48:59.141 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:48:59.141 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T13:48:59.141 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-10-16T13:48:59.142 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-10-16T13:48:59.142 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-10-16T13:48:59.142 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:48:59.142 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:48:59.142 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:48:59.143 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.2", 2023-10-16T13:48:59.143 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1lA6wnGxAAdtaqYhtIJu8qENMrD8xvVA==", 2023-10-16T13:48:59.143 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:48:59.143 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T13:48:59.143 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-10-16T13:48:59.143 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-10-16T13:48:59.144 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-10-16T13:48:59.144 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:48:59.144 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:48:59.144 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:48:59.144 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.3", 2023-10-16T13:48:59.145 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAPKC1l/ddAJxAA3fNKKn9XkXas1BlDN+ggjw==", 2023-10-16T13:48:59.145 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:48:59.145 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2023-10-16T13:48:59.145 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2023-10-16T13:48:59.145 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2023-10-16T13:48:59.145 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2023-10-16T13:48:59.146 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:48:59.146 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:48:59.146 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:48:59.146 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.admin", 2023-10-16T13:48:59.146 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOKC1ld5jUHBAAc6esnCce3dSEZn3wVjhIlQ==", 2023-10-16T13:48:59.147 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:48:59.147 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2023-10-16T13:48:59.147 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2023-10-16T13:48:59.147 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2023-10-16T13:48:59.147 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:48:59.147 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:48:59.148 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:48:59.148 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:48:59.148 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mgr.x", 2023-10-16T13:48:59.148 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOKC1l6b3vNhAAmj5xbfid2sZM2KfHmAf1Wg==", 2023-10-16T13:48:59.148 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:48:59.148 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2023-10-16T13:48:59.149 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile mgr", 2023-10-16T13:48:59.149 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:48:59.149 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:48:59.149 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-10-16T13:48:59.149 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:48:59.150 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mgr.y", 2023-10-16T13:48:59.150 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAOKC1lLz2nMhAA8NqFYZ5nDKkzPBMrkwsVuQ==", 2023-10-16T13:48:59.150 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2023-10-16T13:48:59.150 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2023-10-16T13:48:59.150 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile mgr", 2023-10-16T13:48:59.151 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2023-10-16T13:48:59.151 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:48:59.151 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:48:59.151 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-10-16T13:48:59.151 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-10-16T13:48:59.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.118+0000 7f018c60b700 1 -- 172.21.15.16:0/4236184505 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f016806b020 msgr2=0x7f016806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:59.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.118+0000 7f018c60b700 1 --2- 172.21.15.16:0/4236184505 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f016806b020 0x7f016806d4e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0174000c00 tx=0x7f0174008030 comp rx=0 tx=0).stop 2023-10-16T13:48:59.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.118+0000 7f018c60b700 1 -- 172.21.15.16:0/4236184505 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f018415b860 msgr2=0x7f01840b4130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:59.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.118+0000 7f018c60b700 1 --2- 172.21.15.16:0/4236184505 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f018415b860 0x7f01840b4130 secure :-1 s=READY pgs=3164 cs=0 l=1 rev1=1 crypto rx=0x7f017c00cca0 tx=0x7f017c00a340 comp rx=0 tx=0).stop 2023-10-16T13:48:59.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.118+0000 7f018c60b700 1 -- 172.21.15.16:0/4236184505 shutdown_connections 2023-10-16T13:48:59.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.118+0000 7f018c60b700 1 --2- 172.21.15.16:0/4236184505 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f016806b020 0x7f016806d4e0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:59.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.118+0000 7f018c60b700 1 --2- 172.21.15.16:0/4236184505 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f018415b860 0x7f01840b4130 unknown :-1 s=CLOSED pgs=3164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:59.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.118+0000 7f018c60b700 1 --2- 172.21.15.16:0/4236184505 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f01840b2be0 0x7f01840a9560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:59.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.118+0000 7f018c60b700 1 --2- 172.21.15.16:0/4236184505 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f01840cd770 0x7f01840c9620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:59.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.118+0000 7f018c60b700 1 -- 172.21.15.16:0/4236184505 >> 172.21.15.16:0/4236184505 conn(0x7f01840b1fd0 msgr2=0x7f018415a900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:59.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.118+0000 7f018c60b700 1 -- 172.21.15.16:0/4236184505 shutdown_connections 2023-10-16T13:48:59.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.118+0000 7f018c60b700 1 -- 172.21.15.16:0/4236184505 wait complete. 2023-10-16T13:48:59.155 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:48:59.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.279+0000 7fb41f5b3700 1 Processor -- start 2023-10-16T13:48:59.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.279+0000 7fb41f5b3700 1 -- start start 2023-10-16T13:48:59.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.280+0000 7fb41f5b3700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fb4180cc8a0 0x7fb41815e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:59.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.280+0000 7fb41f5b3700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb41806a7e0 0x7fb41806ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:59.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.280+0000 7fb41f5b3700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb418161c30 con 0x7fb41815b280 2023-10-16T13:48:59.284 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.280+0000 7fb41f5b3700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb41806b190 con 0x7fb4180cc8a0 2023-10-16T13:48:59.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.280+0000 7fb41f5b3700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb418161f10 con 0x7fb41806a7e0 2023-10-16T13:48:59.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.281+0000 7fb41cb4e700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fb4180cc8a0 0x7fb41815e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:59.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.281+0000 7fb41db50700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb41806a7e0 0x7fb41806ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:59.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.281+0000 7fb41d34f700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7fb41815b280 0x7fb4181613e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:55330/0 (socket says 172.21.15.16:55330) 2023-10-16T13:48:59.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.281+0000 7fb41d34f700 1 -- 172.21.15.16:0/2823210810 learned_addr learned my addr 172.21.15.16:0/2823210810 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:48:59.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.281+0000 7fb41db50700 1 -- 172.21.15.16:0/2823210810 >> v2:172.21.15.49:3300/0 conn(0x7fb4180cc8a0 msgr2=0x7fb41815e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:59.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.281+0000 7fb41db50700 1 --2- 172.21.15.16:0/2823210810 >> v2:172.21.15.49:3300/0 conn(0x7fb4180cc8a0 0x7fb41815e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:59.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.281+0000 7fb41db50700 1 -- 172.21.15.16:0/2823210810 >> v1:172.21.15.49:6789/0 conn(0x7fb41815b280 legacy=0x7fb4181613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:59.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.281+0000 7fb41db50700 1 -- 172.21.15.16:0/2823210810 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4181621f0 con 0x7fb41806a7e0 2023-10-16T13:48:59.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.281+0000 7fb41cb4e700 1 --2- 172.21.15.16:0/2823210810 >> v2:172.21.15.49:3300/0 conn(0x7fb4180cc8a0 0x7fb41815e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:48:59.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.281+0000 7fb41db50700 1 --2- 172.21.15.16:0/2823210810 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb41806a7e0 0x7fb41806ac50 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fb410009d90 tx=0x7fb41000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=165203d36baaae2f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:59.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.281+0000 7fb40ffff700 1 -- 172.21.15.16:0/2823210810 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb41000e070 con 0x7fb41806a7e0 2023-10-16T13:48:59.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.281+0000 7fb40ffff700 1 -- 172.21.15.16:0/2823210810 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb41000bb20 con 0x7fb41806a7e0 2023-10-16T13:48:59.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.281+0000 7fb40ffff700 1 -- 172.21.15.16:0/2823210810 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb410012430 con 0x7fb41806a7e0 2023-10-16T13:48:59.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.282+0000 7fb41f5b3700 1 -- 172.21.15.16:0/2823210810 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb41806a7e0 msgr2=0x7fb41806ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:59.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.282+0000 7fb41f5b3700 1 --2- 172.21.15.16:0/2823210810 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb41806a7e0 0x7fb41806ac50 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fb410009d90 tx=0x7fb41000b330 comp rx=0 tx=0).stop 2023-10-16T13:48:59.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.282+0000 7fb41f5b3700 1 -- 172.21.15.16:0/2823210810 shutdown_connections 2023-10-16T13:48:59.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.282+0000 7fb41f5b3700 1 --2- 172.21.15.16:0/2823210810 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb41806a7e0 0x7fb41806ac50 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:59.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.282+0000 7fb41f5b3700 1 --2- 172.21.15.16:0/2823210810 >> v2:172.21.15.49:3300/0 conn(0x7fb4180cc8a0 0x7fb41815e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:59.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.282+0000 7fb41f5b3700 1 -- 172.21.15.16:0/2823210810 >> 172.21.15.16:0/2823210810 conn(0x7fb4180b1fd0 msgr2=0x7fb4180b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:59.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.282+0000 7fb41f5b3700 1 -- 172.21.15.16:0/2823210810 shutdown_connections 2023-10-16T13:48:59.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.282+0000 7fb41f5b3700 1 -- 172.21.15.16:0/2823210810 wait complete. 2023-10-16T13:48:59.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.283+0000 7fb41f5b3700 1 Processor -- start 2023-10-16T13:48:59.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.283+0000 7fb41f5b3700 1 -- start start 2023-10-16T13:48:59.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.284+0000 7fb41f5b3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb41806a7e0 0x7fb4180af460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:59.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.284+0000 7fb41f5b3700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb4180cc8a0 0x7fb4180ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:59.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.284+0000 7fb41f5b3700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb41815b280 0x7fb4180ace10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:59.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.284+0000 7fb41f5b3700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb41815c350 con 0x7fb41806a7e0 2023-10-16T13:48:59.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.284+0000 7fb41f5b3700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb41815c630 con 0x7fb41815b280 2023-10-16T13:48:59.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.284+0000 7fb41f5b3700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb41815c910 con 0x7fb4180cc8a0 2023-10-16T13:48:59.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.284+0000 7fb41d34f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb41806a7e0 0x7fb4180af460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:59.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.284+0000 7fb41d34f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb41806a7e0 0x7fb4180af460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58880/0 (socket says 172.21.15.16:58880) 2023-10-16T13:48:59.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.284+0000 7fb41d34f700 1 -- 172.21.15.16:0/1438187319 learned_addr learned my addr 172.21.15.16:0/1438187319 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:59.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.284+0000 7fb41db50700 1 --2- 172.21.15.16:0/1438187319 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb41815b280 0x7fb4180ace10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:59.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.284+0000 7fb41cb4e700 1 --2- 172.21.15.16:0/1438187319 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb4180cc8a0 0x7fb4180ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:59.294 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.284+0000 7fb41d34f700 1 -- 172.21.15.16:0/1438187319 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb4180cc8a0 msgr2=0x7fb4180ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:59.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.284+0000 7fb41d34f700 1 --2- 172.21.15.16:0/1438187319 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb4180cc8a0 0x7fb4180ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:59.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.284+0000 7fb41d34f700 1 -- 172.21.15.16:0/1438187319 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb41815b280 msgr2=0x7fb4180ace10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:59.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.284+0000 7fb41d34f700 1 --2- 172.21.15.16:0/1438187319 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb41815b280 0x7fb4180ace10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:59.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.284+0000 7fb41d34f700 1 -- 172.21.15.16:0/1438187319 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4140094a0 con 0x7fb41806a7e0 2023-10-16T13:48:59.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.285+0000 7fb41d34f700 1 --2- 172.21.15.16:0/1438187319 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb41806a7e0 0x7fb4180af460 secure :-1 s=READY pgs=3165 cs=0 l=1 rev1=1 crypto rx=0x7fb40800c520 tx=0x7fb40800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:59.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.285+0000 7fb40dffb700 1 -- 172.21.15.16:0/1438187319 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb408005d70 con 0x7fb41806a7e0 2023-10-16T13:48:59.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.285+0000 7fb40dffb700 1 -- 172.21.15.16:0/1438187319 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4080070d0 con 0x7fb41806a7e0 2023-10-16T13:48:59.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.285+0000 7fb40dffb700 1 -- 172.21.15.16:0/1438187319 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb408007440 con 0x7fb41806a7e0 2023-10-16T13:48:59.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.285+0000 7fb41f5b3700 1 -- 172.21.15.16:0/1438187319 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb410009a70 con 0x7fb41806a7e0 2023-10-16T13:48:59.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.285+0000 7fb41f5b3700 1 -- 172.21.15.16:0/1438187319 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb41815ced0 con 0x7fb41806a7e0 2023-10-16T13:48:59.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.286+0000 7fb41f5b3700 1 -- 172.21.15.16:0/1438187319 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb41804ebb0 con 0x7fb41806a7e0 2023-10-16T13:48:59.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.286+0000 7fb40dffb700 1 -- 172.21.15.16:0/1438187319 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 166) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb4080075a0 con 0x7fb41806a7e0 2023-10-16T13:48:59.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.287+0000 7fb40dffb700 1 --2- 172.21.15.16:0/1438187319 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fb3fc06b020 0x7fb3fc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:59.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.287+0000 7fb40dffb700 1 -- 172.21.15.16:0/1438187319 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(274..274 src has 1..274) v4 ==== 5305+0+0 (secure 0 0 0) 0x7fb4080888e0 con 0x7fb41806a7e0 2023-10-16T13:48:59.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.287+0000 7fb41cb4e700 1 --2- 172.21.15.16:0/1438187319 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fb3fc06b020 0x7fb3fc06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:59.297 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.287+0000 7fb41cb4e700 1 --2- 172.21.15.16:0/1438187319 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fb3fc06b020 0x7fb3fc06d4e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb4180c3480 tx=0x7fb414009210 comp rx=0 tx=0).ready entity=mgr.13344 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:59.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.291+0000 7fb40dffb700 1 -- 172.21.15.16:0/1438187319 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb408053a20 con 0x7fb41806a7e0 2023-10-16T13:48:59.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.496+0000 7fb41f5b3700 1 -- 172.21.15.16:0/1438187319 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb4180d12d0 con 0x7fb41806a7e0 2023-10-16T13:48:59.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.496+0000 7fb40dffb700 1 -- 172.21.15.16:0/1438187319 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v274) v1 ==== 74+0+13169 (secure 0 0 0) 0x7fb408017090 con 0x7fb41806a7e0 2023-10-16T13:48:59.499 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:48:59.499 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":274,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:48:58.674171+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":36,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":263,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:48:59.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.499+0000 7fb41f5b3700 1 -- 172.21.15.16:0/1438187319 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fb3fc06b020 msgr2=0x7fb3fc06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:59.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.499+0000 7fb41f5b3700 1 --2- 172.21.15.16:0/1438187319 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fb3fc06b020 0x7fb3fc06d4e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb4180c3480 tx=0x7fb414009210 comp rx=0 tx=0).stop 2023-10-16T13:48:59.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.499+0000 7fb41f5b3700 1 -- 172.21.15.16:0/1438187319 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb41806a7e0 msgr2=0x7fb4180af460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:59.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.499+0000 7fb41f5b3700 1 --2- 172.21.15.16:0/1438187319 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb41806a7e0 0x7fb4180af460 secure :-1 s=READY pgs=3165 cs=0 l=1 rev1=1 crypto rx=0x7fb40800c520 tx=0x7fb40800c830 comp rx=0 tx=0).stop 2023-10-16T13:48:59.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.499+0000 7fb41f5b3700 1 -- 172.21.15.16:0/1438187319 shutdown_connections 2023-10-16T13:48:59.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.499+0000 7fb41f5b3700 1 --2- 172.21.15.16:0/1438187319 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb41815b280 0x7fb4180ace10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:59.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.499+0000 7fb41f5b3700 1 --2- 172.21.15.16:0/1438187319 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fb3fc06b020 0x7fb3fc06d4e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:59.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.499+0000 7fb41f5b3700 1 --2- 172.21.15.16:0/1438187319 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb41806a7e0 0x7fb4180af460 unknown :-1 s=CLOSED pgs=3165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:59.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.499+0000 7fb41f5b3700 1 --2- 172.21.15.16:0/1438187319 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb4180cc8a0 0x7fb4180ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:59.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.499+0000 7fb41f5b3700 1 -- 172.21.15.16:0/1438187319 >> 172.21.15.16:0/1438187319 conn(0x7fb4180b1fd0 msgr2=0x7fb4180718e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:59.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.499+0000 7fb41f5b3700 1 -- 172.21.15.16:0/1438187319 shutdown_connections 2023-10-16T13:48:59.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.499+0000 7fb41f5b3700 1 -- 172.21.15.16:0/1438187319 wait complete. 2023-10-16T13:48:59.514 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T13:48:59.514 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T13:48:59.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.641+0000 7fc00689b700 1 Processor -- start 2023-10-16T13:48:59.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.641+0000 7fc00689b700 1 -- start start 2023-10-16T13:48:59.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.642+0000 7fc00689b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc00006aef0 0x7fc0000ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:59.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.642+0000 7fc00689b700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fc00006da30 0x7fc0000adf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:59.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.642+0000 7fc00689b700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0001613c0 con 0x7fc000160fe0 2023-10-16T13:48:59.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.642+0000 7fc00689b700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0000cf680 con 0x7fc00006aef0 2023-10-16T13:48:59.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.642+0000 7fc00689b700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0000adb00 con 0x7fc00006da30 2023-10-16T13:48:59.647 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.642+0000 7fbfff7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc00006aef0 0x7fc0000ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:59.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.642+0000 7fbfff7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc00006aef0 0x7fc0000ac8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58888/0 (socket says 172.21.15.16:58888) 2023-10-16T13:48:59.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.642+0000 7fbfff7fe700 1 -- 172.21.15.16:0/107586255 learned_addr learned my addr 172.21.15.16:0/107586255 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:59.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.642+0000 7fc004e38700 1 --2- 172.21.15.16:0/107586255 >> v2:172.21.15.49:3300/0 conn(0x7fc00006da30 0x7fc0000adf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:59.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.643+0000 7fbfff7fe700 1 -- 172.21.15.16:0/107586255 >> v2:172.21.15.49:3300/0 conn(0x7fc00006da30 msgr2=0x7fc0000adf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:48:59.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.643+0000 7fbfff7fe700 1 --2- 172.21.15.16:0/107586255 >> v2:172.21.15.49:3300/0 conn(0x7fc00006da30 0x7fc0000adf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:59.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.643+0000 7fbfff7fe700 1 -- 172.21.15.16:0/107586255 >> v1:172.21.15.49:6789/0 conn(0x7fc000160fe0 legacy=0x7fc00006a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:59.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.643+0000 7fbfff7fe700 1 -- 172.21.15.16:0/107586255 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0000acef0 con 0x7fc00006aef0 2023-10-16T13:48:59.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.643+0000 7fbfff7fe700 1 --2- 172.21.15.16:0/107586255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc00006aef0 0x7fc0000ac8d0 secure :-1 s=READY pgs=3166 cs=0 l=1 rev1=1 crypto rx=0x7fc0000b2e40 tx=0x7fbfe8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2770911600f05c0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:59.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.643+0000 7fbffeffd700 1 -- 172.21.15.16:0/107586255 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfe8011830 con 0x7fc00006aef0 2023-10-16T13:48:59.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.643+0000 7fbffeffd700 1 -- 172.21.15.16:0/107586255 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfe8011990 con 0x7fc00006aef0 2023-10-16T13:48:59.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.643+0000 7fbffeffd700 1 -- 172.21.15.16:0/107586255 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfe8011d20 con 0x7fc00006aef0 2023-10-16T13:48:59.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.643+0000 7fc00689b700 1 -- 172.21.15.16:0/107586255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc00006aef0 msgr2=0x7fc0000ac8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:59.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.643+0000 7fc00689b700 1 --2- 172.21.15.16:0/107586255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc00006aef0 0x7fc0000ac8d0 secure :-1 s=READY pgs=3166 cs=0 l=1 rev1=1 crypto rx=0x7fc0000b2e40 tx=0x7fbfe8009bd0 comp rx=0 tx=0).stop 2023-10-16T13:48:59.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.643+0000 7fc00689b700 1 -- 172.21.15.16:0/107586255 shutdown_connections 2023-10-16T13:48:59.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.643+0000 7fc00689b700 1 --2- 172.21.15.16:0/107586255 >> v2:172.21.15.49:3300/0 conn(0x7fc00006da30 0x7fc0000adf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:59.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.643+0000 7fc00689b700 1 --2- 172.21.15.16:0/107586255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc00006aef0 0x7fc0000ac8d0 unknown :-1 s=CLOSED pgs=3166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:59.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.643+0000 7fc00689b700 1 -- 172.21.15.16:0/107586255 >> 172.21.15.16:0/107586255 conn(0x7fc0000b1ed0 msgr2=0x7fc0000b22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:59.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.644+0000 7fc00689b700 1 -- 172.21.15.16:0/107586255 shutdown_connections 2023-10-16T13:48:59.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.644+0000 7fc00689b700 1 -- 172.21.15.16:0/107586255 wait complete. 2023-10-16T13:48:59.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.645+0000 7fc00689b700 1 Processor -- start 2023-10-16T13:48:59.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.645+0000 7fc00689b700 1 -- start start 2023-10-16T13:48:59.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.646+0000 7fc00689b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc00006aef0 0x7fc0001f5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:59.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.646+0000 7fc00689b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc00006da30 0x7fc0000c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:59.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.646+0000 7fc00689b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc000160fe0 0x7fc0000c5860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:59.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.646+0000 7fc00689b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0000c8330 con 0x7fc00006aef0 2023-10-16T13:48:59.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.646+0000 7fc00689b700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0000c5da0 con 0x7fc00006da30 2023-10-16T13:48:59.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.646+0000 7fc00689b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0000c8520 con 0x7fc000160fe0 2023-10-16T13:48:59.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.646+0000 7fbffffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc00006aef0 0x7fc0001f5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:59.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.646+0000 7fbffffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc00006aef0 0x7fc0001f5cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58892/0 (socket says 172.21.15.16:58892) 2023-10-16T13:48:59.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.646+0000 7fbffffff700 1 -- 172.21.15.16:0/2351506179 learned_addr learned my addr 172.21.15.16:0/2351506179 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:48:59.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.646+0000 7fbfff7fe700 1 --2- 172.21.15.16:0/2351506179 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc00006da30 0x7fc0000c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:59.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.646+0000 7fc004e38700 1 --2- 172.21.15.16:0/2351506179 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc000160fe0 0x7fc0000c5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:59.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.646+0000 7fbffffff700 1 -- 172.21.15.16:0/2351506179 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc000160fe0 msgr2=0x7fc0000c5860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:59.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.646+0000 7fbffffff700 1 --2- 172.21.15.16:0/2351506179 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc000160fe0 0x7fc0000c5860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:59.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.646+0000 7fbffffff700 1 -- 172.21.15.16:0/2351506179 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc00006da30 msgr2=0x7fc0000c7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:59.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.646+0000 7fbffffff700 1 --2- 172.21.15.16:0/2351506179 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc00006da30 0x7fc0000c7df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:59.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.646+0000 7fbffffff700 1 -- 172.21.15.16:0/2351506179 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfe8009550 con 0x7fc00006aef0 2023-10-16T13:48:59.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.646+0000 7fbffffff700 1 --2- 172.21.15.16:0/2351506179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc00006aef0 0x7fc0001f5cd0 secure :-1 s=READY pgs=3167 cs=0 l=1 rev1=1 crypto rx=0x7fbff000c500 tx=0x7fbff000c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:59.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.646+0000 7fbffcff9700 1 -- 172.21.15.16:0/2351506179 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbff00052d0 con 0x7fc00006aef0 2023-10-16T13:48:59.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.646+0000 7fbffcff9700 1 -- 172.21.15.16:0/2351506179 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbff0005430 con 0x7fc00006aef0 2023-10-16T13:48:59.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.646+0000 7fc00689b700 1 -- 172.21.15.16:0/2351506179 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0000c8800 con 0x7fc00006aef0 2023-10-16T13:48:59.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.647+0000 7fbffcff9700 1 -- 172.21.15.16:0/2351506179 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbff00038f0 con 0x7fc00006aef0 2023-10-16T13:48:59.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.647+0000 7fc00689b700 1 -- 172.21.15.16:0/2351506179 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0000c8dc0 con 0x7fc00006aef0 2023-10-16T13:48:59.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.647+0000 7fc00689b700 1 -- 172.21.15.16:0/2351506179 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc00004ebb0 con 0x7fc00006aef0 2023-10-16T13:48:59.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.648+0000 7fbffcff9700 1 -- 172.21.15.16:0/2351506179 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 166) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fbff0003a50 con 0x7fc00006aef0 2023-10-16T13:48:59.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.648+0000 7fbffcff9700 1 --2- 172.21.15.16:0/2351506179 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fbfe006b020 0x7fbfe006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:48:59.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.649+0000 7fbffcff9700 1 -- 172.21.15.16:0/2351506179 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(274..274 src has 1..274) v4 ==== 5305+0+0 (secure 0 0 0) 0x7fbff0088780 con 0x7fc00006aef0 2023-10-16T13:48:59.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.649+0000 7fbfff7fe700 1 --2- 172.21.15.16:0/2351506179 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fbfe006b020 0x7fbfe006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:48:59.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.649+0000 7fbfff7fe700 1 --2- 172.21.15.16:0/2351506179 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fbfe006b020 0x7fbfe006d4e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fbfe8009c10 tx=0x7fbfe8008030 comp rx=0 tx=0).ready entity=mgr.13344 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:48:59.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.653+0000 7fbffcff9700 1 -- 172.21.15.16:0/2351506179 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbff00538b0 con 0x7fc00006aef0 2023-10-16T13:48:59.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.857+0000 7fc00689b700 1 -- 172.21.15.16:0/2351506179 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc0001613c0 con 0x7fc00006aef0 2023-10-16T13:48:59.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.857+0000 7fbffcff9700 1 -- 172.21.15.16:0/2351506179 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v274) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbff0017020 con 0x7fc00006aef0 2023-10-16T13:48:59.860 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T13:48:59.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.859+0000 7fc00689b700 1 -- 172.21.15.16:0/2351506179 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fbfe006b020 msgr2=0x7fbfe006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:59.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.859+0000 7fc00689b700 1 --2- 172.21.15.16:0/2351506179 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fbfe006b020 0x7fbfe006d4e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fbfe8009c10 tx=0x7fbfe8008030 comp rx=0 tx=0).stop 2023-10-16T13:48:59.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.859+0000 7fc00689b700 1 -- 172.21.15.16:0/2351506179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc00006aef0 msgr2=0x7fc0001f5cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:48:59.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.859+0000 7fc00689b700 1 --2- 172.21.15.16:0/2351506179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc00006aef0 0x7fc0001f5cd0 secure :-1 s=READY pgs=3167 cs=0 l=1 rev1=1 crypto rx=0x7fbff000c500 tx=0x7fbff000c810 comp rx=0 tx=0).stop 2023-10-16T13:48:59.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.859+0000 7fc00689b700 1 -- 172.21.15.16:0/2351506179 shutdown_connections 2023-10-16T13:48:59.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.859+0000 7fc00689b700 1 --2- 172.21.15.16:0/2351506179 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc000160fe0 0x7fc0000c5860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:59.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.859+0000 7fc00689b700 1 --2- 172.21.15.16:0/2351506179 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fbfe006b020 0x7fbfe006d4e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:59.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.859+0000 7fc00689b700 1 --2- 172.21.15.16:0/2351506179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc00006aef0 0x7fc0001f5cd0 unknown :-1 s=CLOSED pgs=3167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:59.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.859+0000 7fc00689b700 1 --2- 172.21.15.16:0/2351506179 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc00006da30 0x7fc0000c7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:48:59.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.859+0000 7fc00689b700 1 -- 172.21.15.16:0/2351506179 >> 172.21.15.16:0/2351506179 conn(0x7fc0000b1ed0 msgr2=0x7fc0000afd40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:48:59.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.860+0000 7fc00689b700 1 -- 172.21.15.16:0/2351506179 shutdown_connections 2023-10-16T13:48:59.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.860+0000 7fc00689b700 1 -- 172.21.15.16:0/2351506179 wait complete. 2023-10-16T13:48:59.874 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-16T13:48:59.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.997+0000 7f70ec4fd700 1 Processor -- start 2023-10-16T13:48:59.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.997+0000 7f70ec4fd700 1 -- start start 2023-10-16T13:49:00.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.998+0000 7f70ec4fd700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f70e415b280 0x7f70e41613e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:00.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.998+0000 7f70ec4fd700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f70e40cc8a0 0x7f70e415e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:00.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.998+0000 7f70ec4fd700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70e4161c30 con 0x7f70e406a7e0 2023-10-16T13:49:00.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.998+0000 7f70ec4fd700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70e4161f10 con 0x7f70e40cc8a0 2023-10-16T13:49:00.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.998+0000 7f70ec4fd700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70e41621f0 con 0x7f70e415b280 2023-10-16T13:49:00.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.998+0000 7f70eaa9a700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f70e406a7e0 0x7f70e40d02b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48180/0 (socket says 172.21.15.16:48180) 2023-10-16T13:49:00.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.998+0000 7f70eaa9a700 1 -- 172.21.15.16:0/3468619454 learned_addr learned my addr 172.21.15.16:0/3468619454 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:49:00.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.998+0000 7f70e9a98700 1 --2- 172.21.15.16:0/3468619454 >> v2:172.21.15.49:3300/0 conn(0x7f70e40cc8a0 0x7f70e415e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:00.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.998+0000 7f70ea299700 1 --2- 172.21.15.16:0/3468619454 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f70e415b280 0x7f70e41613e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:00.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.998+0000 7f70e9297700 1 -- 172.21.15.16:0/3468619454 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3616817551 0 0) 0x7f70e4161c30 con 0x7f70e406a7e0 2023-10-16T13:49:00.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.998+0000 7f70e9297700 1 -- 172.21.15.16:0/3468619454 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70d00035b0 con 0x7f70e406a7e0 2023-10-16T13:49:00.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.998+0000 7f70e9a98700 1 -- 172.21.15.16:0/3468619454 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f70e415b280 msgr2=0x7f70e41613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:00.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.998+0000 7f70e9a98700 1 --2- 172.21.15.16:0/3468619454 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f70e415b280 0x7f70e41613e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:00.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.998+0000 7f70e9a98700 1 -- 172.21.15.16:0/3468619454 >> v1:172.21.15.16:6789/0 conn(0x7f70e406a7e0 legacy=0x7f70e40d02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:00.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.998+0000 7f70e9a98700 1 -- 172.21.15.16:0/3468619454 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70e415df00 con 0x7f70e40cc8a0 2023-10-16T13:49:00.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.998+0000 7f70e9a98700 1 --2- 172.21.15.16:0/3468619454 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f70e40cc8a0 0x7f70e415e700 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f70d4009840 tx=0x7f70d4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=faa72c397ff8085d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:00.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.999+0000 7f70e9297700 1 -- 172.21.15.16:0/3468619454 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70d4011830 con 0x7f70e40cc8a0 2023-10-16T13:49:00.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.999+0000 7f70e9297700 1 -- 172.21.15.16:0/3468619454 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70d4011990 con 0x7f70e40cc8a0 2023-10-16T13:49:00.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.999+0000 7f70e9297700 1 -- 172.21.15.16:0/3468619454 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70d4011d20 con 0x7f70e40cc8a0 2023-10-16T13:49:00.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.999+0000 7f70ec4fd700 1 -- 172.21.15.16:0/3468619454 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f70e40cc8a0 msgr2=0x7f70e415e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:00.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.999+0000 7f70ec4fd700 1 --2- 172.21.15.16:0/3468619454 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f70e40cc8a0 0x7f70e415e700 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f70d4009840 tx=0x7f70d4009b50 comp rx=0 tx=0).stop 2023-10-16T13:49:00.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.999+0000 7f70ec4fd700 1 -- 172.21.15.16:0/3468619454 shutdown_connections 2023-10-16T13:49:00.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.999+0000 7f70ec4fd700 1 --2- 172.21.15.16:0/3468619454 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f70e415b280 0x7f70e41613e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:00.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.999+0000 7f70ec4fd700 1 --2- 172.21.15.16:0/3468619454 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f70e40cc8a0 0x7f70e415e700 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:00.006 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.999+0000 7f70ec4fd700 1 -- 172.21.15.16:0/3468619454 >> 172.21.15.16:0/3468619454 conn(0x7f70e40b1fd0 msgr2=0x7f70e40b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:00.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.999+0000 7f70ec4fd700 1 -- 172.21.15.16:0/3468619454 shutdown_connections 2023-10-16T13:49:00.007 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:48:59.999+0000 7f70ec4fd700 1 -- 172.21.15.16:0/3468619454 wait complete. 2023-10-16T13:49:00.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.000+0000 7f70ec4fd700 1 Processor -- start 2023-10-16T13:49:00.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.001+0000 7f70ec4fd700 1 -- start start 2023-10-16T13:49:00.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.001+0000 7f70ec4fd700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f70e406a7e0 0x7f70e40c5c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:00.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.001+0000 7f70ec4fd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70e40cc8a0 0x7f70e41593e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:00.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.001+0000 7f70ec4fd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f70e415b280 0x7f70e4159920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:00.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.001+0000 7f70ec4fd700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70e40c6760 con 0x7f70e40cc8a0 2023-10-16T13:49:00.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.001+0000 7f70ec4fd700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70e40c6480 con 0x7f70e415b280 2023-10-16T13:49:00.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.001+0000 7f70ec4fd700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70e415b790 con 0x7f70e406a7e0 2023-10-16T13:49:00.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.001+0000 7f70e9a98700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70e40cc8a0 0x7f70e41593e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:00.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.001+0000 7f70e9a98700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70e40cc8a0 0x7f70e41593e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58908/0 (socket says 172.21.15.16:58908) 2023-10-16T13:49:00.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.001+0000 7f70e9a98700 1 -- 172.21.15.16:0/2290909050 learned_addr learned my addr 172.21.15.16:0/2290909050 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:00.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.002+0000 7f70ea299700 1 --2- 172.21.15.16:0/2290909050 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f70e406a7e0 0x7f70e40c5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:00.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.002+0000 7f70eaa9a700 1 --2- 172.21.15.16:0/2290909050 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f70e415b280 0x7f70e4159920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:00.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.002+0000 7f70e9a98700 1 -- 172.21.15.16:0/2290909050 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f70e406a7e0 msgr2=0x7f70e40c5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:00.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.002+0000 7f70e9a98700 1 --2- 172.21.15.16:0/2290909050 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f70e406a7e0 0x7f70e40c5c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:00.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.002+0000 7f70e9a98700 1 -- 172.21.15.16:0/2290909050 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f70e415b280 msgr2=0x7f70e4159920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:00.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.002+0000 7f70e9a98700 1 --2- 172.21.15.16:0/2290909050 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f70e415b280 0x7f70e4159920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:00.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.002+0000 7f70e9a98700 1 -- 172.21.15.16:0/2290909050 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70d40094a0 con 0x7f70e40cc8a0 2023-10-16T13:49:00.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.002+0000 7f70eaa9a700 1 --2- 172.21.15.16:0/2290909050 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f70e415b280 0x7f70e4159920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:49:00.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.002+0000 7f70e9a98700 1 --2- 172.21.15.16:0/2290909050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70e40cc8a0 0x7f70e41593e0 secure :-1 s=READY pgs=3169 cs=0 l=1 rev1=1 crypto rx=0x7f70d40090b0 tx=0x7f70d4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:00.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.002+0000 7f70daffd700 1 -- 172.21.15.16:0/2290909050 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70d40036a0 con 0x7f70e40cc8a0 2023-10-16T13:49:00.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.002+0000 7f70daffd700 1 -- 172.21.15.16:0/2290909050 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70d401f070 con 0x7f70e40cc8a0 2023-10-16T13:49:00.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.002+0000 7f70ec4fd700 1 -- 172.21.15.16:0/2290909050 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70e415ba70 con 0x7f70e40cc8a0 2023-10-16T13:49:00.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.002+0000 7f70daffd700 1 -- 172.21.15.16:0/2290909050 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70d4003a40 con 0x7f70e40cc8a0 2023-10-16T13:49:00.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.002+0000 7f70ec4fd700 1 -- 172.21.15.16:0/2290909050 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70e40c6190 con 0x7f70e40cc8a0 2023-10-16T13:49:00.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.003+0000 7f70ec4fd700 1 -- 172.21.15.16:0/2290909050 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70e404ebb0 con 0x7f70e40cc8a0 2023-10-16T13:49:00.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.004+0000 7f70daffd700 1 -- 172.21.15.16:0/2290909050 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 166) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f70d4003ba0 con 0x7f70e40cc8a0 2023-10-16T13:49:00.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.004+0000 7f70daffd700 1 --2- 172.21.15.16:0/2290909050 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f70c806b020 0x7f70c806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:00.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.004+0000 7f70daffd700 1 -- 172.21.15.16:0/2290909050 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(274..274 src has 1..274) v4 ==== 5305+0+0 (secure 0 0 0) 0x7f70d4021070 con 0x7f70e40cc8a0 2023-10-16T13:49:00.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.004+0000 7f70ea299700 1 --2- 172.21.15.16:0/2290909050 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f70c806b020 0x7f70c806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:00.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.004+0000 7f70ea299700 1 --2- 172.21.15.16:0/2290909050 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f70c806b020 0x7f70c806d4e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f70e0009ad0 tx=0x7f70e0008040 comp rx=0 tx=0).ready entity=mgr.13344 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:00.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.009+0000 7f70daffd700 1 -- 172.21.15.16:0/2290909050 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f70d4090020 con 0x7f70e40cc8a0 2023-10-16T13:49:00.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.213+0000 7f70ec4fd700 1 -- 172.21.15.16:0/2290909050 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f70e415a7a0 con 0x7f70e40cc8a0 2023-10-16T13:49:00.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.214+0000 7f70daffd700 1 -- 172.21.15.16:0/2290909050 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v274) v1 ==== 84+0+65 (secure 0 0 0) 0x7f70d4015090 con 0x7f70e40cc8a0 2023-10-16T13:49:00.216 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:49:00.217 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2023-10-16T13:49:00.217 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:49:00.217 INFO:teuthology.orchestra.run.smithi016.stdout: "poolnum": 1, 2023-10-16T13:49:00.217 INFO:teuthology.orchestra.run.smithi016.stdout: "poolname": ".mgr" 2023-10-16T13:49:00.217 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:49:00.217 INFO:teuthology.orchestra.run.smithi016.stdout:] 2023-10-16T13:49:00.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.216+0000 7f70ec4fd700 1 -- 172.21.15.16:0/2290909050 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f70c806b020 msgr2=0x7f70c806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:00.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.216+0000 7f70ec4fd700 1 --2- 172.21.15.16:0/2290909050 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f70c806b020 0x7f70c806d4e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f70e0009ad0 tx=0x7f70e0008040 comp rx=0 tx=0).stop 2023-10-16T13:49:00.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.216+0000 7f70ec4fd700 1 -- 172.21.15.16:0/2290909050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70e40cc8a0 msgr2=0x7f70e41593e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:00.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.216+0000 7f70ec4fd700 1 --2- 172.21.15.16:0/2290909050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70e40cc8a0 0x7f70e41593e0 secure :-1 s=READY pgs=3169 cs=0 l=1 rev1=1 crypto rx=0x7f70d40090b0 tx=0x7f70d4004750 comp rx=0 tx=0).stop 2023-10-16T13:49:00.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.216+0000 7f70ec4fd700 1 -- 172.21.15.16:0/2290909050 shutdown_connections 2023-10-16T13:49:00.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.216+0000 7f70ec4fd700 1 --2- 172.21.15.16:0/2290909050 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f70e415b280 0x7f70e4159920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:00.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.216+0000 7f70ec4fd700 1 --2- 172.21.15.16:0/2290909050 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f70e406a7e0 0x7f70e40c5c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:00.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.216+0000 7f70ec4fd700 1 --2- 172.21.15.16:0/2290909050 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f70c806b020 0x7f70c806d4e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:00.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.216+0000 7f70ec4fd700 1 --2- 172.21.15.16:0/2290909050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70e40cc8a0 0x7f70e41593e0 unknown :-1 s=CLOSED pgs=3169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:00.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.216+0000 7f70ec4fd700 1 -- 172.21.15.16:0/2290909050 >> 172.21.15.16:0/2290909050 conn(0x7f70e40b1fd0 msgr2=0x7f70e40cb260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:00.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.216+0000 7f70ec4fd700 1 -- 172.21.15.16:0/2290909050 shutdown_connections 2023-10-16T13:49:00.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.216+0000 7f70ec4fd700 1 -- 172.21.15.16:0/2290909050 wait complete. 2023-10-16T13:49:00.230 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-16T13:49:00.231 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-16T13:49:00.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.358+0000 7fd01c220700 1 Processor -- start 2023-10-16T13:49:00.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.358+0000 7fd01c220700 1 -- start start 2023-10-16T13:49:00.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.359+0000 7fd01c220700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd014155a60 0x7fd01415fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:00.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.359+0000 7fd01c220700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd014160190 con 0x7fd014155e40 2023-10-16T13:49:00.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.359+0000 7fd01c220700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd01415c780 con 0x7fd0140cba40 2023-10-16T13:49:00.363 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.359+0000 7fd01c220700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd01415f240 con 0x7fd014155a60 2023-10-16T13:49:00.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.359+0000 7fd019fbc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd014155a60 0x7fd01415fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:00.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.359+0000 7fd0197bb700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd014155e40 0x7fd01415bc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48190/0 (socket says 172.21.15.16:48190) 2023-10-16T13:49:00.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.359+0000 7fd0197bb700 1 -- 172.21.15.16:0/1228375521 learned_addr learned my addr 172.21.15.16:0/1228375521 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:49:00.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.359+0000 7fd019fbc700 1 -- 172.21.15.16:0/1228375521 >> v1:172.21.15.49:6789/0 conn(0x7fd0140cba40 legacy=0x7fd0140cbe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:00.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.359+0000 7fd019fbc700 1 -- 172.21.15.16:0/1228375521 >> v1:172.21.15.16:6789/0 conn(0x7fd014155e40 legacy=0x7fd01415bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:00.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.359+0000 7fd019fbc700 1 -- 172.21.15.16:0/1228375521 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd01415ca60 con 0x7fd014155a60 2023-10-16T13:49:00.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.359+0000 7fd018fba700 1 -- 172.21.15.16:0/1228375521 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1581370887 0 0) 0x7fd014160190 con 0x7fd014155e40 2023-10-16T13:49:00.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.359+0000 7fd019fbc700 1 --2- 172.21.15.16:0/1228375521 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd014155a60 0x7fd01415fc50 secure :-1 s=READY pgs=3171 cs=0 l=1 rev1=1 crypto rx=0x7fd010009840 tx=0x7fd010009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce28c544f4beb8e7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:00.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.360+0000 7fd018fba700 1 -- 172.21.15.16:0/1228375521 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd010004230 con 0x7fd014155a60 2023-10-16T13:49:00.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.360+0000 7fd018fba700 1 -- 172.21.15.16:0/1228375521 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd010004390 con 0x7fd014155a60 2023-10-16T13:49:00.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.360+0000 7fd018fba700 1 -- 172.21.15.16:0/1228375521 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0100036a0 con 0x7fd014155a60 2023-10-16T13:49:00.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.360+0000 7fd01c220700 1 -- 172.21.15.16:0/1228375521 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd014155a60 msgr2=0x7fd01415fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:00.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.360+0000 7fd01c220700 1 --2- 172.21.15.16:0/1228375521 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd014155a60 0x7fd01415fc50 secure :-1 s=READY pgs=3171 cs=0 l=1 rev1=1 crypto rx=0x7fd010009840 tx=0x7fd010009b50 comp rx=0 tx=0).stop 2023-10-16T13:49:00.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.360+0000 7fd01c220700 1 -- 172.21.15.16:0/1228375521 shutdown_connections 2023-10-16T13:49:00.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.360+0000 7fd01c220700 1 --2- 172.21.15.16:0/1228375521 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd014155a60 0x7fd01415fc50 unknown :-1 s=CLOSED pgs=3171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:00.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.360+0000 7fd01c220700 1 -- 172.21.15.16:0/1228375521 >> 172.21.15.16:0/1228375521 conn(0x7fd01406a7e0 msgr2=0x7fd01406abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:00.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.360+0000 7fd01c220700 1 -- 172.21.15.16:0/1228375521 shutdown_connections 2023-10-16T13:49:00.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.360+0000 7fd01c220700 1 -- 172.21.15.16:0/1228375521 wait complete. 2023-10-16T13:49:00.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.361+0000 7fd01c220700 1 Processor -- start 2023-10-16T13:49:00.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.361+0000 7fd01c220700 1 -- start start 2023-10-16T13:49:00.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.362+0000 7fd01c220700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd0140cba40 0x7fd0140c4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:00.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.362+0000 7fd01c220700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd014155a60 0x7fd01415e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:00.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.362+0000 7fd01c220700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd014155e40 0x7fd014158740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:00.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.362+0000 7fd01c220700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0140cb250 con 0x7fd014155e40 2023-10-16T13:49:00.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.362+0000 7fd01c220700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd014157d30 con 0x7fd0140cba40 2023-10-16T13:49:00.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.362+0000 7fd01c220700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd014158c80 con 0x7fd014155a60 2023-10-16T13:49:00.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.362+0000 7fd01a7bd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd014155e40 0x7fd014158740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:00.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.362+0000 7fd01a7bd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd014155e40 0x7fd014158740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58926/0 (socket says 172.21.15.16:58926) 2023-10-16T13:49:00.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.362+0000 7fd01a7bd700 1 -- 172.21.15.16:0/2120891306 learned_addr learned my addr 172.21.15.16:0/2120891306 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:00.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.362+0000 7fd019fbc700 1 --2- 172.21.15.16:0/2120891306 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd0140cba40 0x7fd0140c4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:00.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.363+0000 7fd0197bb700 1 --2- 172.21.15.16:0/2120891306 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd014155a60 0x7fd01415e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:00.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.363+0000 7fd01a7bd700 1 -- 172.21.15.16:0/2120891306 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd014155a60 msgr2=0x7fd01415e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:00.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.363+0000 7fd01a7bd700 1 --2- 172.21.15.16:0/2120891306 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd014155a60 0x7fd01415e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:00.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.363+0000 7fd01a7bd700 1 -- 172.21.15.16:0/2120891306 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd0140cba40 msgr2=0x7fd0140c4ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:00.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.363+0000 7fd01a7bd700 1 --2- 172.21.15.16:0/2120891306 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd0140cba40 0x7fd0140c4ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:00.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.363+0000 7fd01a7bd700 1 -- 172.21.15.16:0/2120891306 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0100094a0 con 0x7fd014155e40 2023-10-16T13:49:00.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.363+0000 7fd0197bb700 1 --2- 172.21.15.16:0/2120891306 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd014155a60 0x7fd01415e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:49:00.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.363+0000 7fd01a7bd700 1 --2- 172.21.15.16:0/2120891306 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd014155e40 0x7fd014158740 secure :-1 s=READY pgs=3172 cs=0 l=1 rev1=1 crypto rx=0x7fd00c00cc10 tx=0x7fd00c00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:00.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.363+0000 7fd00a7fc700 1 -- 172.21.15.16:0/2120891306 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd00c00abb0 con 0x7fd014155e40 2023-10-16T13:49:00.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.363+0000 7fd00a7fc700 1 -- 172.21.15.16:0/2120891306 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd00c00ad10 con 0x7fd014155e40 2023-10-16T13:49:00.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.363+0000 7fd01c220700 1 -- 172.21.15.16:0/2120891306 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0140b1ed0 con 0x7fd014155e40 2023-10-16T13:49:00.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.363+0000 7fd00a7fc700 1 -- 172.21.15.16:0/2120891306 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd00c012660 con 0x7fd014155e40 2023-10-16T13:49:00.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.363+0000 7fd01c220700 1 -- 172.21.15.16:0/2120891306 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd014157f40 con 0x7fd014155e40 2023-10-16T13:49:00.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.364+0000 7fd01c220700 1 -- 172.21.15.16:0/2120891306 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd01404ec20 con 0x7fd014155e40 2023-10-16T13:49:00.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.364+0000 7fd00a7fc700 1 -- 172.21.15.16:0/2120891306 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 166) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd00c0127c0 con 0x7fd014155e40 2023-10-16T13:49:00.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.365+0000 7fd00a7fc700 1 --2- 172.21.15.16:0/2120891306 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fcff806b020 0x7fcff806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:00.374 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.365+0000 7fd00a7fc700 1 -- 172.21.15.16:0/2120891306 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(274..274 src has 1..274) v4 ==== 5305+0+0 (secure 0 0 0) 0x7fd00c089cb0 con 0x7fd014155e40 2023-10-16T13:49:00.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.365+0000 7fd019fbc700 1 --2- 172.21.15.16:0/2120891306 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fcff806b020 0x7fcff806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:00.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.365+0000 7fd019fbc700 1 --2- 172.21.15.16:0/2120891306 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fcff806b020 0x7fcff806d4e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd010000c00 tx=0x7fd01001c040 comp rx=0 tx=0).ready entity=mgr.13344 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:00.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.369+0000 7fd00a7fc700 1 -- 172.21.15.16:0/2120891306 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd00c054e70 con 0x7fd014155e40 2023-10-16T13:49:00.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.574+0000 7fd01c220700 1 -- 172.21.15.16:0/2120891306 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fd01415c4c0 con 0x7fd014155e40 2023-10-16T13:49:00.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.668+0000 7fd00a7fc700 1 -- 172.21.15.16:0/2120891306 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v275) v1 ==== 138+0+0 (secure 0 0 0) 0x7fd00c016070 con 0x7fd014155e40 2023-10-16T13:49:00.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.671+0000 7fd01c220700 1 -- 172.21.15.16:0/2120891306 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fcff806b020 msgr2=0x7fcff806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:00.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.671+0000 7fd01c220700 1 --2- 172.21.15.16:0/2120891306 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fcff806b020 0x7fcff806d4e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd010000c00 tx=0x7fd01001c040 comp rx=0 tx=0).stop 2023-10-16T13:49:00.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.671+0000 7fd01c220700 1 -- 172.21.15.16:0/2120891306 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd014155e40 msgr2=0x7fd014158740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:00.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.671+0000 7fd01c220700 1 --2- 172.21.15.16:0/2120891306 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd014155e40 0x7fd014158740 secure :-1 s=READY pgs=3172 cs=0 l=1 rev1=1 crypto rx=0x7fd00c00cc10 tx=0x7fd00c00a060 comp rx=0 tx=0).stop 2023-10-16T13:49:00.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.672+0000 7fd01c220700 1 -- 172.21.15.16:0/2120891306 shutdown_connections 2023-10-16T13:49:00.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.672+0000 7fd01c220700 1 --2- 172.21.15.16:0/2120891306 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fcff806b020 0x7fcff806d4e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:00.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.672+0000 7fd01c220700 1 --2- 172.21.15.16:0/2120891306 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd014155e40 0x7fd014158740 unknown :-1 s=CLOSED pgs=3172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:00.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.672+0000 7fd01c220700 1 --2- 172.21.15.16:0/2120891306 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd0140cba40 0x7fd0140c4ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:00.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.672+0000 7fd01c220700 1 --2- 172.21.15.16:0/2120891306 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd014155a60 0x7fd01415e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:00.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.672+0000 7fd01c220700 1 -- 172.21.15.16:0/2120891306 >> 172.21.15.16:0/2120891306 conn(0x7fd01406a7e0 msgr2=0x7fd01415cdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:00.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.672+0000 7fd01c220700 1 -- 172.21.15.16:0/2120891306 shutdown_connections 2023-10-16T13:49:00.676 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.672+0000 7fd01c220700 1 -- 172.21.15.16:0/2120891306 wait complete. 2023-10-16T13:49:00.678 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' created 2023-10-16T13:49:00.690 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-16T13:49:00.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.820+0000 7fc15a49f700 1 Processor -- start 2023-10-16T13:49:00.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.820+0000 7fc15a49f700 1 -- start start 2023-10-16T13:49:00.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.821+0000 7fc15a49f700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fc15415b070 0x7fc15415b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:00.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.821+0000 7fc15a49f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc1540cb880 0x7fc1540cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:00.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.821+0000 7fc15a49f700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1540cef70 con 0x7fc15406da60 2023-10-16T13:49:00.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.821+0000 7fc15a49f700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1540ce360 con 0x7fc15415b070 2023-10-16T13:49:00.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.821+0000 7fc15a49f700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1540ce640 con 0x7fc1540cb880 2023-10-16T13:49:00.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.821+0000 7fc1537fe700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc1540cb880 0x7fc1540cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:00.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.821+0000 7fc1537fe700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc1540cb880 0x7fc1540cbd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:47188/0 (socket says 172.21.15.16:47188) 2023-10-16T13:49:00.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.821+0000 7fc1537fe700 1 -- 172.21.15.16:0/2856884593 learned_addr learned my addr 172.21.15.16:0/2856884593 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:00.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.821+0000 7fc153fff700 1 --2- 172.21.15.16:0/2856884593 >> v2:172.21.15.49:3300/0 conn(0x7fc15415b070 0x7fc15415b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:00.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.822+0000 7fc1537fe700 1 -- 172.21.15.16:0/2856884593 >> v2:172.21.15.49:3300/0 conn(0x7fc15415b070 msgr2=0x7fc15415b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:00.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.822+0000 7fc1537fe700 1 --2- 172.21.15.16:0/2856884593 >> v2:172.21.15.49:3300/0 conn(0x7fc15415b070 0x7fc15415b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:00.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.822+0000 7fc1537fe700 1 -- 172.21.15.16:0/2856884593 >> v1:172.21.15.49:6789/0 conn(0x7fc15406da60 legacy=0x7fc1540cf400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:00.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.822+0000 7fc1537fe700 1 -- 172.21.15.16:0/2856884593 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1540ce920 con 0x7fc1540cb880 2023-10-16T13:49:00.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.822+0000 7fc1537fe700 1 --2- 172.21.15.16:0/2856884593 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc1540cb880 0x7fc1540cbd00 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fc13c00bb90 tx=0x7fc13c00bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=471a1c54ea0cff1a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:00.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.822+0000 7fc152ffd700 1 -- 172.21.15.16:0/2856884593 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc13c00dd30 con 0x7fc1540cb880 2023-10-16T13:49:00.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.822+0000 7fc152ffd700 1 -- 172.21.15.16:0/2856884593 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc13c00de90 con 0x7fc1540cb880 2023-10-16T13:49:00.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.822+0000 7fc152ffd700 1 -- 172.21.15.16:0/2856884593 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc13c010690 con 0x7fc1540cb880 2023-10-16T13:49:00.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.823+0000 7fc15a49f700 1 -- 172.21.15.16:0/2856884593 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc1540cb880 msgr2=0x7fc1540cbd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:00.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.823+0000 7fc15a49f700 1 --2- 172.21.15.16:0/2856884593 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc1540cb880 0x7fc1540cbd00 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fc13c00bb90 tx=0x7fc13c00bea0 comp rx=0 tx=0).stop 2023-10-16T13:49:00.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.823+0000 7fc15a49f700 1 -- 172.21.15.16:0/2856884593 shutdown_connections 2023-10-16T13:49:00.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.823+0000 7fc15a49f700 1 --2- 172.21.15.16:0/2856884593 >> v2:172.21.15.49:3300/0 conn(0x7fc15415b070 0x7fc15415b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:00.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.823+0000 7fc15a49f700 1 --2- 172.21.15.16:0/2856884593 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc1540cb880 0x7fc1540cbd00 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:00.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.823+0000 7fc15a49f700 1 -- 172.21.15.16:0/2856884593 >> 172.21.15.16:0/2856884593 conn(0x7fc1540b1f40 msgr2=0x7fc1540b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:00.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.823+0000 7fc15a49f700 1 -- 172.21.15.16:0/2856884593 shutdown_connections 2023-10-16T13:49:00.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.823+0000 7fc15a49f700 1 -- 172.21.15.16:0/2856884593 wait complete. 2023-10-16T13:49:00.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.824+0000 7fc15a49f700 1 Processor -- start 2023-10-16T13:49:00.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.824+0000 7fc15a49f700 1 -- start start 2023-10-16T13:49:00.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.825+0000 7fc15a49f700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc15406da60 0x7fc15415e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:00.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.825+0000 7fc15a49f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1540cb880 0x7fc15415c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:00.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.825+0000 7fc15a49f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc15415b070 0x7fc1540c5f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:00.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.825+0000 7fc15a49f700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc15415cb80 con 0x7fc1540cb880 2023-10-16T13:49:00.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.825+0000 7fc15a49f700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc15415ce20 con 0x7fc15406da60 2023-10-16T13:49:00.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.825+0000 7fc15a49f700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc15415d100 con 0x7fc15415b070 2023-10-16T13:49:00.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.825+0000 7fc1537fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1540cb880 0x7fc15415c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:00.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.825+0000 7fc1537fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1540cb880 0x7fc15415c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58938/0 (socket says 172.21.15.16:58938) 2023-10-16T13:49:00.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.825+0000 7fc1537fe700 1 -- 172.21.15.16:0/693527255 learned_addr learned my addr 172.21.15.16:0/693527255 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:00.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.825+0000 7fc153fff700 1 --2- 172.21.15.16:0/693527255 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc15406da60 0x7fc15415e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:00.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.825+0000 7fc158a3c700 1 --2- 172.21.15.16:0/693527255 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc15415b070 0x7fc1540c5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:00.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.825+0000 7fc1537fe700 1 -- 172.21.15.16:0/693527255 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc15415b070 msgr2=0x7fc1540c5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:00.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.825+0000 7fc1537fe700 1 --2- 172.21.15.16:0/693527255 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc15415b070 0x7fc1540c5f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:00.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.825+0000 7fc1537fe700 1 -- 172.21.15.16:0/693527255 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc15406da60 msgr2=0x7fc15415e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:00.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.825+0000 7fc1537fe700 1 --2- 172.21.15.16:0/693527255 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc15406da60 0x7fc15415e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:00.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.825+0000 7fc1537fe700 1 -- 172.21.15.16:0/693527255 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc13c00b7f0 con 0x7fc1540cb880 2023-10-16T13:49:00.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.825+0000 7fc1537fe700 1 --2- 172.21.15.16:0/693527255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1540cb880 0x7fc15415c640 secure :-1 s=READY pgs=3173 cs=0 l=1 rev1=1 crypto rx=0x7fc13c000c00 tx=0x7fc13c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:00.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.826+0000 7fc150ff9700 1 -- 172.21.15.16:0/693527255 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc13c003b70 con 0x7fc1540cb880 2023-10-16T13:49:00.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.826+0000 7fc150ff9700 1 -- 172.21.15.16:0/693527255 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc13c003cd0 con 0x7fc1540cb880 2023-10-16T13:49:00.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.826+0000 7fc15a49f700 1 -- 172.21.15.16:0/693527255 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc15415d3e0 con 0x7fc1540cb880 2023-10-16T13:49:00.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.826+0000 7fc150ff9700 1 -- 172.21.15.16:0/693527255 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc13c0106b0 con 0x7fc1540cb880 2023-10-16T13:49:00.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.826+0000 7fc15a49f700 1 -- 172.21.15.16:0/693527255 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc15415d9a0 con 0x7fc1540cb880 2023-10-16T13:49:00.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.827+0000 7fc15a49f700 1 -- 172.21.15.16:0/693527255 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1540accf0 con 0x7fc1540cb880 2023-10-16T13:49:00.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.827+0000 7fc150ff9700 1 -- 172.21.15.16:0/693527255 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 166) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc13c019400 con 0x7fc1540cb880 2023-10-16T13:49:00.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.828+0000 7fc150ff9700 1 --2- 172.21.15.16:0/693527255 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fc13406b020 0x7fc13406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:00.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.828+0000 7fc150ff9700 1 -- 172.21.15.16:0/693527255 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 5675+0+0 (secure 0 0 0) 0x7fc13c08b250 con 0x7fc1540cb880 2023-10-16T13:49:00.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.828+0000 7fc153fff700 1 --2- 172.21.15.16:0/693527255 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fc13406b020 0x7fc13406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:00.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.828+0000 7fc153fff700 1 --2- 172.21.15.16:0/693527255 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fc13406b020 0x7fc13406d4e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc144009b40 tx=0x7fc144009210 comp rx=0 tx=0).ready entity=mgr.13344 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:00.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:00.832+0000 7fc150ff9700 1 -- 172.21.15.16:0/693527255 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc13c056190 con 0x7fc1540cb880 2023-10-16T13:49:01.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.033+0000 7fc15a49f700 1 -- 172.21.15.16:0/693527255 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fc15404ebb0 con 0x7fc1540cb880 2023-10-16T13:49:01.673 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.671+0000 7fc150ff9700 1 -- 172.21.15.16:0/693527255 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v276) v1 ==== 170+0+0 (secure 0 0 0) 0x7fc13c0597b0 con 0x7fc1540cb880 2023-10-16T13:49:01.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.677+0000 7fc15a49f700 1 -- 172.21.15.16:0/693527255 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fc13406b020 msgr2=0x7fc13406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:01.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.677+0000 7fc15a49f700 1 --2- 172.21.15.16:0/693527255 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fc13406b020 0x7fc13406d4e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc144009b40 tx=0x7fc144009210 comp rx=0 tx=0).stop 2023-10-16T13:49:01.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.677+0000 7fc15a49f700 1 -- 172.21.15.16:0/693527255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1540cb880 msgr2=0x7fc15415c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:01.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.677+0000 7fc15a49f700 1 --2- 172.21.15.16:0/693527255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1540cb880 0x7fc15415c640 secure :-1 s=READY pgs=3173 cs=0 l=1 rev1=1 crypto rx=0x7fc13c000c00 tx=0x7fc13c004750 comp rx=0 tx=0).stop 2023-10-16T13:49:01.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.678+0000 7fc15a49f700 1 -- 172.21.15.16:0/693527255 shutdown_connections 2023-10-16T13:49:01.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.678+0000 7fc15a49f700 1 --2- 172.21.15.16:0/693527255 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc15415b070 0x7fc1540c5f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:01.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.678+0000 7fc15a49f700 1 --2- 172.21.15.16:0/693527255 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc15406da60 0x7fc15415e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:01.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.678+0000 7fc15a49f700 1 --2- 172.21.15.16:0/693527255 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fc13406b020 0x7fc13406d4e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:01.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.678+0000 7fc15a49f700 1 --2- 172.21.15.16:0/693527255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc1540cb880 0x7fc15415c640 unknown :-1 s=CLOSED pgs=3173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:01.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.678+0000 7fc15a49f700 1 -- 172.21.15.16:0/693527255 >> 172.21.15.16:0/693527255 conn(0x7fc1540b1f40 msgr2=0x7fc1540cf540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:01.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.682+0000 7fc15a49f700 1 -- 172.21.15.16:0/693527255 shutdown_connections 2023-10-16T13:49:01.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.683+0000 7fc15a49f700 1 -- 172.21.15.16:0/693527255 wait complete. 2023-10-16T13:49:01.697 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' created 2023-10-16T13:49:01.710 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-16T13:49:01.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.841+0000 7f3fd344c700 1 Processor -- start 2023-10-16T13:49:01.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.841+0000 7f3fd344c700 1 -- start start 2023-10-16T13:49:01.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.842+0000 7f3fd344c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3fcc15ec80 0x7f3fcc0cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:01.845 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.842+0000 7f3fd344c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3fcc15f060 0x7f3fcc0ce400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:01.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.842+0000 7f3fd344c700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fcc0b2440 con 0x7f3fcc0d2110 2023-10-16T13:49:01.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.842+0000 7f3fd344c700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fcc0b2c50 con 0x7f3fcc15f060 2023-10-16T13:49:01.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.842+0000 7f3fd344c700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fcc0d00e0 con 0x7f3fcc15ec80 2023-10-16T13:49:01.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.842+0000 7f3fd09e7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3fcc15f060 0x7f3fcc0ce400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:01.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.842+0000 7f3fd19e9700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f3fcc0d2110 0x7f3fcc0b4440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48222/0 (socket says 172.21.15.16:48222) 2023-10-16T13:49:01.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.842+0000 7f3fd19e9700 1 -- 172.21.15.16:0/1344766801 learned_addr learned my addr 172.21.15.16:0/1344766801 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:49:01.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.842+0000 7f3fd11e8700 1 --2- 172.21.15.16:0/1344766801 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3fcc15ec80 0x7f3fcc0cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:01.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.842+0000 7f3fd09e7700 1 -- 172.21.15.16:0/1344766801 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3fcc15ec80 msgr2=0x7f3fcc0cdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:01.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.842+0000 7f3fd09e7700 1 --2- 172.21.15.16:0/1344766801 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3fcc15ec80 0x7f3fcc0cdec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:01.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.842+0000 7f3fd09e7700 1 -- 172.21.15.16:0/1344766801 >> v1:172.21.15.16:6789/0 conn(0x7f3fcc0d2110 legacy=0x7f3fcc0b4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:01.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.842+0000 7f3fc3fff700 1 -- 172.21.15.16:0/1344766801 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2370937913 0 0) 0x7f3fcc0b2440 con 0x7f3fcc0d2110 2023-10-16T13:49:01.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.842+0000 7f3fd09e7700 1 -- 172.21.15.16:0/1344766801 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fcc0cef60 con 0x7f3fcc15f060 2023-10-16T13:49:01.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.842+0000 7f3fd11e8700 1 --2- 172.21.15.16:0/1344766801 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3fcc15ec80 0x7f3fcc0cdec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:49:01.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.843+0000 7f3fd09e7700 1 --2- 172.21.15.16:0/1344766801 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3fcc15f060 0x7f3fcc0ce400 secure :-1 s=READY pgs=3175 cs=0 l=1 rev1=1 crypto rx=0x7f3fc8009840 tx=0x7f3fc8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=260cac537bdb009 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:01.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.843+0000 7f3fc3fff700 1 -- 172.21.15.16:0/1344766801 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fc8011830 con 0x7f3fcc15f060 2023-10-16T13:49:01.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.843+0000 7f3fc3fff700 1 -- 172.21.15.16:0/1344766801 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fc8011990 con 0x7f3fcc15f060 2023-10-16T13:49:01.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.843+0000 7f3fc3fff700 1 -- 172.21.15.16:0/1344766801 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fc8011d20 con 0x7f3fcc15f060 2023-10-16T13:49:01.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.843+0000 7f3fd344c700 1 -- 172.21.15.16:0/1344766801 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3fcc15f060 msgr2=0x7f3fcc0ce400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:01.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.843+0000 7f3fd344c700 1 --2- 172.21.15.16:0/1344766801 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3fcc15f060 0x7f3fcc0ce400 secure :-1 s=READY pgs=3175 cs=0 l=1 rev1=1 crypto rx=0x7f3fc8009840 tx=0x7f3fc8009b50 comp rx=0 tx=0).stop 2023-10-16T13:49:01.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.843+0000 7f3fd344c700 1 -- 172.21.15.16:0/1344766801 shutdown_connections 2023-10-16T13:49:01.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.843+0000 7f3fd344c700 1 --2- 172.21.15.16:0/1344766801 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3fcc15ec80 0x7f3fcc0cdec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:01.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.843+0000 7f3fd344c700 1 --2- 172.21.15.16:0/1344766801 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3fcc15f060 0x7f3fcc0ce400 unknown :-1 s=CLOSED pgs=3175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:01.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.843+0000 7f3fd344c700 1 -- 172.21.15.16:0/1344766801 >> 172.21.15.16:0/1344766801 conn(0x7f3fcc0d15f0 msgr2=0x7f3fcc0d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:01.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.844+0000 7f3fd344c700 1 -- 172.21.15.16:0/1344766801 shutdown_connections 2023-10-16T13:49:01.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.844+0000 7f3fd344c700 1 -- 172.21.15.16:0/1344766801 wait complete. 2023-10-16T13:49:01.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.845+0000 7f3fd344c700 1 Processor -- start 2023-10-16T13:49:01.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.845+0000 7f3fd344c700 1 -- start start 2023-10-16T13:49:01.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.846+0000 7f3fd344c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3fcc0d2110 0x7f3fcc0c43d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:01.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.846+0000 7f3fd344c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3fcc15ec80 0x7f3fcc0c6d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:01.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.846+0000 7f3fd344c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3fcc15f060 0x7f3fcc0c7270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:01.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.846+0000 7f3fd344c700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fcc155400 con 0x7f3fcc15ec80 2023-10-16T13:49:01.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.846+0000 7f3fd344c700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fcc1e5060 con 0x7f3fcc15f060 2023-10-16T13:49:01.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.846+0000 7f3fd344c700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fcc1e5340 con 0x7f3fcc0d2110 2023-10-16T13:49:01.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.846+0000 7f3fd09e7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3fcc15ec80 0x7f3fcc0c6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:01.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.846+0000 7f3fd09e7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3fcc15ec80 0x7f3fcc0c6d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58952/0 (socket says 172.21.15.16:58952) 2023-10-16T13:49:01.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.846+0000 7f3fd09e7700 1 -- 172.21.15.16:0/2461749054 learned_addr learned my addr 172.21.15.16:0/2461749054 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:01.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.846+0000 7f3fd11e8700 1 --2- 172.21.15.16:0/2461749054 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3fcc0d2110 0x7f3fcc0c43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:01.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.846+0000 7f3fd19e9700 1 --2- 172.21.15.16:0/2461749054 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3fcc15f060 0x7f3fcc0c7270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:01.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.846+0000 7f3fd09e7700 1 -- 172.21.15.16:0/2461749054 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3fcc0d2110 msgr2=0x7f3fcc0c43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:01.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.846+0000 7f3fd09e7700 1 --2- 172.21.15.16:0/2461749054 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3fcc0d2110 0x7f3fcc0c43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:01.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.846+0000 7f3fd09e7700 1 -- 172.21.15.16:0/2461749054 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3fcc15f060 msgr2=0x7f3fcc0c7270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:01.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.846+0000 7f3fd09e7700 1 --2- 172.21.15.16:0/2461749054 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3fcc15f060 0x7f3fcc0c7270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:01.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.846+0000 7f3fd09e7700 1 -- 172.21.15.16:0/2461749054 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fc80094a0 con 0x7f3fcc15ec80 2023-10-16T13:49:01.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.846+0000 7f3fd11e8700 1 --2- 172.21.15.16:0/2461749054 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3fcc0d2110 0x7f3fcc0c43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:49:01.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.846+0000 7f3fd19e9700 1 --2- 172.21.15.16:0/2461749054 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3fcc15f060 0x7f3fcc0c7270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:49:01.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.846+0000 7f3fd09e7700 1 --2- 172.21.15.16:0/2461749054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3fcc15ec80 0x7f3fcc0c6d30 secure :-1 s=READY pgs=3176 cs=0 l=1 rev1=1 crypto rx=0x7f3fc8000c00 tx=0x7f3fc8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:01.857 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.846+0000 7f3fc1ffb700 1 -- 172.21.15.16:0/2461749054 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fc80045f0 con 0x7f3fcc15ec80 2023-10-16T13:49:01.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.846+0000 7f3fc1ffb700 1 -- 172.21.15.16:0/2461749054 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fc801e070 con 0x7f3fcc15ec80 2023-10-16T13:49:01.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.846+0000 7f3fd344c700 1 -- 172.21.15.16:0/2461749054 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fcc1e5620 con 0x7f3fcc15ec80 2023-10-16T13:49:01.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.846+0000 7f3fc1ffb700 1 -- 172.21.15.16:0/2461749054 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fc8003ae0 con 0x7f3fcc15ec80 2023-10-16T13:49:01.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.847+0000 7f3fd344c700 1 -- 172.21.15.16:0/2461749054 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fcc1e5be0 con 0x7f3fcc15ec80 2023-10-16T13:49:01.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.847+0000 7f3fd344c700 1 -- 172.21.15.16:0/2461749054 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fcc04ebb0 con 0x7f3fcc15ec80 2023-10-16T13:49:01.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.848+0000 7f3fc1ffb700 1 -- 172.21.15.16:0/2461749054 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 166) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3fc8003c40 con 0x7f3fcc15ec80 2023-10-16T13:49:01.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.849+0000 7f3fc1ffb700 1 --2- 172.21.15.16:0/2461749054 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f3fb006b020 0x7f3fb006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:01.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.849+0000 7f3fc1ffb700 1 -- 172.21.15.16:0/2461749054 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(276..276 src has 1..276) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f3fc808ad30 con 0x7f3fcc15ec80 2023-10-16T13:49:01.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.849+0000 7f3fd11e8700 1 --2- 172.21.15.16:0/2461749054 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f3fb006b020 0x7f3fb006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:01.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.849+0000 7f3fd11e8700 1 --2- 172.21.15.16:0/2461749054 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f3fb006b020 0x7f3fb006d4e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3fbc006fd0 tx=0x7f3fbc008040 comp rx=0 tx=0).ready entity=mgr.13344 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:01.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:01.853+0000 7f3fc1ffb700 1 -- 172.21.15.16:0/2461749054 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3fc8055ce0 con 0x7f3fcc15ec80 2023-10-16T13:49:02.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:02.105+0000 7f3fd344c700 1 -- 172.21.15.16:0/2461749054 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f3fcc0c8140 con 0x7f3fcc15ec80 2023-10-16T13:49:02.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:02.692+0000 7f3fc1ffb700 1 -- 172.21.15.16:0/2461749054 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 37 and data pool 38 v151) v1 ==== 174+0+0 (secure 0 0 0) 0x7f3fc8015020 con 0x7f3fcc15ec80 2023-10-16T13:49:02.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:02.695+0000 7f3fd344c700 1 -- 172.21.15.16:0/2461749054 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f3fb006b020 msgr2=0x7f3fb006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:02.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:02.695+0000 7f3fd344c700 1 --2- 172.21.15.16:0/2461749054 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f3fb006b020 0x7f3fb006d4e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3fbc006fd0 tx=0x7f3fbc008040 comp rx=0 tx=0).stop 2023-10-16T13:49:02.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:02.695+0000 7f3fd344c700 1 -- 172.21.15.16:0/2461749054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3fcc15ec80 msgr2=0x7f3fcc0c6d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:02.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:02.695+0000 7f3fd344c700 1 --2- 172.21.15.16:0/2461749054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3fcc15ec80 0x7f3fcc0c6d30 secure :-1 s=READY pgs=3176 cs=0 l=1 rev1=1 crypto rx=0x7f3fc8000c00 tx=0x7f3fc8004750 comp rx=0 tx=0).stop 2023-10-16T13:49:02.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:02.696+0000 7f3fd344c700 1 -- 172.21.15.16:0/2461749054 shutdown_connections 2023-10-16T13:49:02.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:02.696+0000 7f3fd344c700 1 --2- 172.21.15.16:0/2461749054 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3fcc15f060 0x7f3fcc0c7270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:02.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:02.696+0000 7f3fd344c700 1 --2- 172.21.15.16:0/2461749054 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3fcc0d2110 0x7f3fcc0c43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:02.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:02.696+0000 7f3fd344c700 1 --2- 172.21.15.16:0/2461749054 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f3fb006b020 0x7f3fb006d4e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:02.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:02.696+0000 7f3fd344c700 1 --2- 172.21.15.16:0/2461749054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3fcc15ec80 0x7f3fcc0c6d30 unknown :-1 s=CLOSED pgs=3176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:02.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:02.696+0000 7f3fd344c700 1 -- 172.21.15.16:0/2461749054 >> 172.21.15.16:0/2461749054 conn(0x7f3fcc0d15f0 msgr2=0x7f3fcc0cc190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:02.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:02.696+0000 7f3fd344c700 1 -- 172.21.15.16:0/2461749054 shutdown_connections 2023-10-16T13:49:02.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:02.696+0000 7f3fd344c700 1 -- 172.21.15.16:0/2461749054 wait complete. 2023-10-16T13:49:02.706 INFO:teuthology.orchestra.run.smithi016.stderr:new fs with metadata pool 37 and data pool 38 2023-10-16T13:49:02.719 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:49:02.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:02.848+0000 7fb5ebb7f700 1 Processor -- start 2023-10-16T13:49:02.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:02.849+0000 7fb5ebb7f700 1 -- start start 2023-10-16T13:49:02.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:02.849+0000 7fb5ebb7f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb5e40cc8a0 0x7fb5e415e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:02.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:02.849+0000 7fb5ebb7f700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5e4161c30 con 0x7fb5e406a7e0 2023-10-16T13:49:02.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:02.849+0000 7fb5ebb7f700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5e4161f10 con 0x7fb5e415b280 2023-10-16T13:49:02.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:02.849+0000 7fb5ebb7f700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5e41621f0 con 0x7fb5e40cc8a0 2023-10-16T13:49:02.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:02.849+0000 7fb5e911a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb5e40cc8a0 0x7fb5e415e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:02.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:02.850+0000 7fb5ea11c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb5e406a7e0 0x7fb5e40d02b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48236/0 (socket says 172.21.15.16:48236) 2023-10-16T13:49:02.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:02.850+0000 7fb5ea11c700 1 -- 172.21.15.16:0/870716903 learned_addr learned my addr 172.21.15.16:0/870716903 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:49:02.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:02.850+0000 7fb5e911a700 1 -- 172.21.15.16:0/870716903 >> v1:172.21.15.49:6789/0 conn(0x7fb5e415b280 legacy=0x7fb5e41613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:02.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:02.850+0000 7fb5e911a700 1 -- 172.21.15.16:0/870716903 >> v1:172.21.15.16:6789/0 conn(0x7fb5e406a7e0 legacy=0x7fb5e40d02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:02.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:02.850+0000 7fb5e8919700 1 -- 172.21.15.16:0/870716903 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 128737058 0 0) 0x7fb5e4161c30 con 0x7fb5e406a7e0 2023-10-16T13:49:02.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:02.850+0000 7fb5e911a700 1 -- 172.21.15.16:0/870716903 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5e415df00 con 0x7fb5e40cc8a0 2023-10-16T13:49:02.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:02.850+0000 7fb5e911a700 1 --2- 172.21.15.16:0/870716903 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5e40cc8a0 0x7fb5e415e700 secure :-1 s=READY pgs=3178 cs=0 l=1 rev1=1 crypto rx=0x7fb5d8009840 tx=0x7fb5d8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61ce084733cc6617 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:02.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:02.850+0000 7fb5e8919700 1 -- 172.21.15.16:0/870716903 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d8011830 con 0x7fb5e40cc8a0 2023-10-16T13:49:02.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:02.850+0000 7fb5e8919700 1 -- 172.21.15.16:0/870716903 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5d8011990 con 0x7fb5e40cc8a0 2023-10-16T13:49:02.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:02.850+0000 7fb5e8919700 1 -- 172.21.15.16:0/870716903 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d8011d20 con 0x7fb5e40cc8a0 2023-10-16T13:49:02.858 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:02.851+0000 7fb5ebb7f700 1 -- 172.21.15.16:0/870716903 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5e40cc8a0 msgr2=0x7fb5e415e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:02.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:02.851+0000 7fb5ebb7f700 1 --2- 172.21.15.16:0/870716903 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5e40cc8a0 0x7fb5e415e700 secure :-1 s=READY pgs=3178 cs=0 l=1 rev1=1 crypto rx=0x7fb5d8009840 tx=0x7fb5d8009b50 comp rx=0 tx=0).stop 2023-10-16T13:49:02.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:02.851+0000 7fb5ebb7f700 1 -- 172.21.15.16:0/870716903 shutdown_connections 2023-10-16T13:49:02.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:02.851+0000 7fb5ebb7f700 1 --2- 172.21.15.16:0/870716903 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5e40cc8a0 0x7fb5e415e700 unknown :-1 s=CLOSED pgs=3178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:02.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:02.851+0000 7fb5ebb7f700 1 -- 172.21.15.16:0/870716903 >> 172.21.15.16:0/870716903 conn(0x7fb5e40b1fd0 msgr2=0x7fb5e40b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:02.859 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:02.851+0000 7fb5ebb7f700 1 -- 172.21.15.16:0/870716903 shutdown_connections 2023-10-16T13:49:02.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:02.851+0000 7fb5ebb7f700 1 -- 172.21.15.16:0/870716903 wait complete. 2023-10-16T13:49:02.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:02.852+0000 7fb5ebb7f700 1 Processor -- start 2023-10-16T13:49:02.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:02.852+0000 7fb5ebb7f700 1 -- start start 2023-10-16T13:49:02.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:02.853+0000 7fb5ebb7f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5e406a7e0 0x7fb5e41587a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:02.860 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:02.853+0000 7fb5ebb7f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb5e40cc8a0 0x7fb5e4160160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:02.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:02.853+0000 7fb5ebb7f700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb5e415b280 0x7fb5e406ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:02.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:02.853+0000 7fb5ebb7f700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e40c6db0 con 0x7fb5e406a7e0 2023-10-16T13:49:02.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:02.853+0000 7fb5ebb7f700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e40c6ad0 con 0x7fb5e415b280 2023-10-16T13:49:02.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:02.853+0000 7fb5ebb7f700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e41f63d0 con 0x7fb5e40cc8a0 2023-10-16T13:49:02.861 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:02.853+0000 7fb5e991b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5e406a7e0 0x7fb5e41587a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:02.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:02.853+0000 7fb5e991b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5e406a7e0 0x7fb5e41587a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58964/0 (socket says 172.21.15.16:58964) 2023-10-16T13:49:02.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:02.853+0000 7fb5e991b700 1 -- 172.21.15.16:0/1681020771 learned_addr learned my addr 172.21.15.16:0/1681020771 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:02.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:02.853+0000 7fb5ea11c700 1 --2- 172.21.15.16:0/1681020771 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb5e415b280 0x7fb5e406ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:02.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:02.853+0000 7fb5e911a700 1 --2- 172.21.15.16:0/1681020771 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb5e40cc8a0 0x7fb5e4160160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:02.862 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:02.853+0000 7fb5e991b700 1 -- 172.21.15.16:0/1681020771 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb5e40cc8a0 msgr2=0x7fb5e4160160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:02.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:02.853+0000 7fb5e991b700 1 --2- 172.21.15.16:0/1681020771 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb5e40cc8a0 0x7fb5e4160160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:02.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:02.853+0000 7fb5e991b700 1 -- 172.21.15.16:0/1681020771 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb5e415b280 msgr2=0x7fb5e406ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:02.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:02.853+0000 7fb5e991b700 1 --2- 172.21.15.16:0/1681020771 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb5e415b280 0x7fb5e406ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:02.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:02.853+0000 7fb5e991b700 1 -- 172.21.15.16:0/1681020771 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5d80094a0 con 0x7fb5e406a7e0 2023-10-16T13:49:02.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:02.853+0000 7fb5ea11c700 1 --2- 172.21.15.16:0/1681020771 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb5e415b280 0x7fb5e406ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:49:02.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:02.853+0000 7fb5e911a700 1 --2- 172.21.15.16:0/1681020771 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb5e40cc8a0 0x7fb5e4160160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:49:02.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:02.853+0000 7fb5e991b700 1 --2- 172.21.15.16:0/1681020771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5e406a7e0 0x7fb5e41587a0 secure :-1 s=READY pgs=3179 cs=0 l=1 rev1=1 crypto rx=0x7fb5e000c450 tx=0x7fb5e000c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:02.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:02.854+0000 7fb5d67fc700 1 -- 172.21.15.16:0/1681020771 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5e00052d0 con 0x7fb5e406a7e0 2023-10-16T13:49:02.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:02.854+0000 7fb5d67fc700 1 -- 172.21.15.16:0/1681020771 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5e0005430 con 0x7fb5e406a7e0 2023-10-16T13:49:02.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:02.854+0000 7fb5ebb7f700 1 -- 172.21.15.16:0/1681020771 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5e41f66b0 con 0x7fb5e406a7e0 2023-10-16T13:49:02.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:02.854+0000 7fb5d67fc700 1 -- 172.21.15.16:0/1681020771 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5e00038c0 con 0x7fb5e406a7e0 2023-10-16T13:49:02.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:02.854+0000 7fb5ebb7f700 1 -- 172.21.15.16:0/1681020771 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5e41606a0 con 0x7fb5e406a7e0 2023-10-16T13:49:02.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:02.855+0000 7fb5ebb7f700 1 -- 172.21.15.16:0/1681020771 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5e4160860 con 0x7fb5e406a7e0 2023-10-16T13:49:02.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:02.855+0000 7fb5d67fc700 1 -- 172.21.15.16:0/1681020771 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 166) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb5e0003a20 con 0x7fb5e406a7e0 2023-10-16T13:49:02.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:02.856+0000 7fb5d67fc700 1 --2- 172.21.15.16:0/1681020771 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fb5c806b020 0x7fb5c806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:02.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:02.856+0000 7fb5d67fc700 1 -- 172.21.15.16:0/1681020771 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(277..277 src has 1..277) v4 ==== 6157+0+0 (secure 0 0 0) 0x7fb5e0088be0 con 0x7fb5e406a7e0 2023-10-16T13:49:02.865 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:02.856+0000 7fb5e911a700 1 --2- 172.21.15.16:0/1681020771 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fb5c806b020 0x7fb5c806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:02.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:02.856+0000 7fb5e911a700 1 --2- 172.21.15.16:0/1681020771 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fb5c806b020 0x7fb5c806d4e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb5d8009840 tx=0x7fb5d801c040 comp rx=0 tx=0).ready entity=mgr.13344 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:02.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:02.860+0000 7fb5d67fc700 1 -- 172.21.15.16:0/1681020771 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb5e4160860 con 0x7fb5e406a7e0 2023-10-16T13:49:03.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.063+0000 7fb5ebb7f700 1 -- 172.21.15.16:0/1681020771 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb5e4160860 con 0x7fb5e406a7e0 2023-10-16T13:49:03.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.064+0000 7fb5d67fc700 1 -- 172.21.15.16:0/1681020771 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v277) v1 ==== 74+0+16838 (secure 0 0 0) 0x7fb5e4160860 con 0x7fb5e406a7e0 2023-10-16T13:49:03.067 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:49:03.067 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":277,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:49:02.106794+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":38,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":37,"pool_name":"cephfs_metadata","create_time":"2023-10-16T13:49:00.609612+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"277","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":38,"pool_name":"cephfs_data","create_time":"2023-10-16T13:49:01.068705+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"277","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:49:03.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.066+0000 7fb5ebb7f700 1 -- 172.21.15.16:0/1681020771 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fb5c806b020 msgr2=0x7fb5c806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:03.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.066+0000 7fb5ebb7f700 1 --2- 172.21.15.16:0/1681020771 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fb5c806b020 0x7fb5c806d4e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb5d8009840 tx=0x7fb5d801c040 comp rx=0 tx=0).stop 2023-10-16T13:49:03.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.066+0000 7fb5ebb7f700 1 -- 172.21.15.16:0/1681020771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5e406a7e0 msgr2=0x7fb5e41587a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:03.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.066+0000 7fb5ebb7f700 1 --2- 172.21.15.16:0/1681020771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5e406a7e0 0x7fb5e41587a0 secure :-1 s=READY pgs=3179 cs=0 l=1 rev1=1 crypto rx=0x7fb5e000c450 tx=0x7fb5e000c810 comp rx=0 tx=0).stop 2023-10-16T13:49:03.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.066+0000 7fb5ebb7f700 1 -- 172.21.15.16:0/1681020771 shutdown_connections 2023-10-16T13:49:03.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.066+0000 7fb5ebb7f700 1 --2- 172.21.15.16:0/1681020771 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb5e415b280 0x7fb5e406ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:03.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.066+0000 7fb5ebb7f700 1 --2- 172.21.15.16:0/1681020771 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fb5c806b020 0x7fb5c806d4e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:03.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.066+0000 7fb5ebb7f700 1 --2- 172.21.15.16:0/1681020771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5e406a7e0 0x7fb5e41587a0 unknown :-1 s=CLOSED pgs=3179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:03.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.066+0000 7fb5ebb7f700 1 --2- 172.21.15.16:0/1681020771 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb5e40cc8a0 0x7fb5e4160160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:03.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.066+0000 7fb5ebb7f700 1 -- 172.21.15.16:0/1681020771 >> 172.21.15.16:0/1681020771 conn(0x7fb5e40b1fd0 msgr2=0x7fb5e40cb3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:03.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.066+0000 7fb5ebb7f700 1 -- 172.21.15.16:0/1681020771 shutdown_connections 2023-10-16T13:49:03.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.066+0000 7fb5ebb7f700 1 -- 172.21.15.16:0/1681020771 wait complete. 2023-10-16T13:49:03.081 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:49:03.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.204+0000 7f10bb172700 1 Processor -- start 2023-10-16T13:49:03.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.204+0000 7f10bb172700 1 -- start start 2023-10-16T13:49:03.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.205+0000 7f10bb172700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f10b40cc8a0 0x7f10b415e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:03.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.205+0000 7f10bb172700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10b4161c30 con 0x7f10b415b280 2023-10-16T13:49:03.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.205+0000 7f10bb172700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10b4161f10 con 0x7f10b406a7e0 2023-10-16T13:49:03.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.205+0000 7f10bb172700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10b41621f0 con 0x7f10b40cc8a0 2023-10-16T13:49:03.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.205+0000 7f10b3fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f10b40cc8a0 0x7f10b415e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:03.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.205+0000 7f10b8f0e700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f10b415b280 0x7f10b41613e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48246/0 (socket says 172.21.15.16:48246) 2023-10-16T13:49:03.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.205+0000 7f10b3fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f10b40cc8a0 0x7f10b415e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58974/0 (socket says 172.21.15.16:58974) 2023-10-16T13:49:03.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.205+0000 7f10b8f0e700 1 -- 172.21.15.16:0/1662234140 learned_addr learned my addr 172.21.15.16:0/1662234140 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:49:03.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.205+0000 7f10b3fff700 1 -- 172.21.15.16:0/1662234140 >> v1:172.21.15.49:6789/0 conn(0x7f10b406a7e0 legacy=0x7f10b40d02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:03.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.205+0000 7f10b37fe700 1 -- 172.21.15.16:0/1662234140 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2124558116 0 0) 0x7f10b4161c30 con 0x7f10b415b280 2023-10-16T13:49:03.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.205+0000 7f10b3fff700 1 -- 172.21.15.16:0/1662234140 >> v1:172.21.15.16:6789/0 conn(0x7f10b415b280 legacy=0x7f10b41613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:03.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.205+0000 7f10b3fff700 1 -- 172.21.15.16:0/1662234140 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10b415df00 con 0x7f10b40cc8a0 2023-10-16T13:49:03.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.205+0000 7f10b3fff700 1 --2- 172.21.15.16:0/1662234140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10b40cc8a0 0x7f10b415e700 secure :-1 s=READY pgs=3181 cs=0 l=1 rev1=1 crypto rx=0x7f10a0009840 tx=0x7f10a0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87acdfd12e82fc20 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:03.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.205+0000 7f10b37fe700 1 -- 172.21.15.16:0/1662234140 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10a0011830 con 0x7f10b40cc8a0 2023-10-16T13:49:03.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.206+0000 7f10b37fe700 1 -- 172.21.15.16:0/1662234140 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10a0011990 con 0x7f10b40cc8a0 2023-10-16T13:49:03.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.206+0000 7f10b37fe700 1 -- 172.21.15.16:0/1662234140 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10a0011d20 con 0x7f10b40cc8a0 2023-10-16T13:49:03.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.206+0000 7f10bb172700 1 -- 172.21.15.16:0/1662234140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10b40cc8a0 msgr2=0x7f10b415e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:03.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.206+0000 7f10bb172700 1 --2- 172.21.15.16:0/1662234140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10b40cc8a0 0x7f10b415e700 secure :-1 s=READY pgs=3181 cs=0 l=1 rev1=1 crypto rx=0x7f10a0009840 tx=0x7f10a0009b50 comp rx=0 tx=0).stop 2023-10-16T13:49:03.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.206+0000 7f10bb172700 1 -- 172.21.15.16:0/1662234140 shutdown_connections 2023-10-16T13:49:03.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.206+0000 7f10bb172700 1 --2- 172.21.15.16:0/1662234140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10b40cc8a0 0x7f10b415e700 unknown :-1 s=CLOSED pgs=3181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:03.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.206+0000 7f10bb172700 1 -- 172.21.15.16:0/1662234140 >> 172.21.15.16:0/1662234140 conn(0x7f10b40b1fd0 msgr2=0x7f10b40b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:03.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.206+0000 7f10bb172700 1 -- 172.21.15.16:0/1662234140 shutdown_connections 2023-10-16T13:49:03.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.206+0000 7f10bb172700 1 -- 172.21.15.16:0/1662234140 wait complete. 2023-10-16T13:49:03.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.207+0000 7f10bb172700 1 Processor -- start 2023-10-16T13:49:03.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.207+0000 7f10bb172700 1 -- start start 2023-10-16T13:49:03.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.208+0000 7f10bb172700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10b406a7e0 0x7f10b41f66a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:03.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.208+0000 7f10bb172700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f10b40cc8a0 0x7f10b414f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:03.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.208+0000 7f10bb172700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f10b415b280 0x7f10b41f3b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:03.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.208+0000 7f10bb172700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10b41f7520 con 0x7f10b406a7e0 2023-10-16T13:49:03.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.208+0000 7f10bb172700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10b41f6c80 con 0x7f10b415b280 2023-10-16T13:49:03.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.208+0000 7f10bb172700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10b41f6f60 con 0x7f10b40cc8a0 2023-10-16T13:49:03.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.208+0000 7f10b8f0e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10b406a7e0 0x7f10b41f66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:03.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.208+0000 7f10b8f0e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10b406a7e0 0x7f10b41f66a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58978/0 (socket says 172.21.15.16:58978) 2023-10-16T13:49:03.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.208+0000 7f10b8f0e700 1 -- 172.21.15.16:0/518083893 learned_addr learned my addr 172.21.15.16:0/518083893 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:03.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.209+0000 7f10b970f700 1 --2- 172.21.15.16:0/518083893 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f10b415b280 0x7f10b41f3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:03.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.209+0000 7f10b3fff700 1 --2- 172.21.15.16:0/518083893 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f10b40cc8a0 0x7f10b414f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:03.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.209+0000 7f10b8f0e700 1 -- 172.21.15.16:0/518083893 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f10b40cc8a0 msgr2=0x7f10b414f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:03.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.209+0000 7f10b8f0e700 1 --2- 172.21.15.16:0/518083893 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f10b40cc8a0 0x7f10b414f8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:03.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.209+0000 7f10b8f0e700 1 -- 172.21.15.16:0/518083893 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f10b415b280 msgr2=0x7f10b41f3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:03.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.209+0000 7f10b8f0e700 1 --2- 172.21.15.16:0/518083893 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f10b415b280 0x7f10b41f3b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:03.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.209+0000 7f10b8f0e700 1 -- 172.21.15.16:0/518083893 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10a00094a0 con 0x7f10b406a7e0 2023-10-16T13:49:03.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.209+0000 7f10b970f700 1 --2- 172.21.15.16:0/518083893 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f10b415b280 0x7f10b41f3b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:49:03.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.209+0000 7f10b8f0e700 1 --2- 172.21.15.16:0/518083893 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10b406a7e0 0x7f10b41f66a0 secure :-1 s=READY pgs=3182 cs=0 l=1 rev1=1 crypto rx=0x7f10a800c520 tx=0x7f10a800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:03.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.209+0000 7f10b17fa700 1 -- 172.21.15.16:0/518083893 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10a8004c80 con 0x7f10b406a7e0 2023-10-16T13:49:03.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.209+0000 7f10b17fa700 1 -- 172.21.15.16:0/518083893 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10a8004e20 con 0x7f10b406a7e0 2023-10-16T13:49:03.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.209+0000 7f10bb172700 1 -- 172.21.15.16:0/518083893 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10b41f7240 con 0x7f10b406a7e0 2023-10-16T13:49:03.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.209+0000 7f10b17fa700 1 -- 172.21.15.16:0/518083893 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10a8007310 con 0x7f10b406a7e0 2023-10-16T13:49:03.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.209+0000 7f10bb172700 1 -- 172.21.15.16:0/518083893 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10b414fe10 con 0x7f10b406a7e0 2023-10-16T13:49:03.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.210+0000 7f10bb172700 1 -- 172.21.15.16:0/518083893 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10b4158cd0 con 0x7f10b406a7e0 2023-10-16T13:49:03.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.211+0000 7f10b17fa700 1 -- 172.21.15.16:0/518083893 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 166) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f10a8007470 con 0x7f10b406a7e0 2023-10-16T13:49:03.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.211+0000 7f10b17fa700 1 --2- 172.21.15.16:0/518083893 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f109806b020 0x7f109806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:03.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.211+0000 7f10b17fa700 1 -- 172.21.15.16:0/518083893 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(277..277 src has 1..277) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f10a8089620 con 0x7f10b406a7e0 2023-10-16T13:49:03.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.211+0000 7f10b3fff700 1 --2- 172.21.15.16:0/518083893 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f109806b020 0x7f109806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:03.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.212+0000 7f10b3fff700 1 --2- 172.21.15.16:0/518083893 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f109806b020 0x7f109806d4e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f10a0009840 tx=0x7f10a001c040 comp rx=0 tx=0).ready entity=mgr.13344 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:03.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.216+0000 7f10b17fa700 1 -- 172.21.15.16:0/518083893 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f10a8054400 con 0x7f10b406a7e0 2023-10-16T13:49:03.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.423+0000 7f10bb172700 1 -- 172.21.15.16:0/518083893 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f10b404ebb0 con 0x7f10b406a7e0 2023-10-16T13:49:03.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.424+0000 7f10b17fa700 1 -- 172.21.15.16:0/518083893 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v278) v1 ==== 74+0+16825 (secure 0 0 0) 0x7f10a8057a20 con 0x7f10b406a7e0 2023-10-16T13:49:03.427 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:49:03.427 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":278,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:49:03.362245+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":38,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":37,"pool_name":"cephfs_metadata","create_time":"2023-10-16T13:49:00.609612+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"277","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":38,"pool_name":"cephfs_data","create_time":"2023-10-16T13:49:01.068705+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"278","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:49:03.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.426+0000 7f10bb172700 1 -- 172.21.15.16:0/518083893 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f109806b020 msgr2=0x7f109806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:03.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.426+0000 7f10bb172700 1 --2- 172.21.15.16:0/518083893 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f109806b020 0x7f109806d4e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f10a0009840 tx=0x7f10a001c040 comp rx=0 tx=0).stop 2023-10-16T13:49:03.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.426+0000 7f10bb172700 1 -- 172.21.15.16:0/518083893 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10b406a7e0 msgr2=0x7f10b41f66a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:03.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.426+0000 7f10bb172700 1 --2- 172.21.15.16:0/518083893 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10b406a7e0 0x7f10b41f66a0 secure :-1 s=READY pgs=3182 cs=0 l=1 rev1=1 crypto rx=0x7f10a800c520 tx=0x7f10a800c8e0 comp rx=0 tx=0).stop 2023-10-16T13:49:03.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.427+0000 7f10bb172700 1 -- 172.21.15.16:0/518083893 shutdown_connections 2023-10-16T13:49:03.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.427+0000 7f10bb172700 1 --2- 172.21.15.16:0/518083893 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f10b415b280 0x7f10b41f3b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:03.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.427+0000 7f10bb172700 1 --2- 172.21.15.16:0/518083893 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f109806b020 0x7f109806d4e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:03.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.427+0000 7f10bb172700 1 --2- 172.21.15.16:0/518083893 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10b406a7e0 0x7f10b41f66a0 unknown :-1 s=CLOSED pgs=3182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:03.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.427+0000 7f10bb172700 1 --2- 172.21.15.16:0/518083893 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f10b40cc8a0 0x7f10b414f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:03.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.427+0000 7f10bb172700 1 -- 172.21.15.16:0/518083893 >> 172.21.15.16:0/518083893 conn(0x7f10b40b1fd0 msgr2=0x7f10b40b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:03.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.427+0000 7f10bb172700 1 -- 172.21.15.16:0/518083893 shutdown_connections 2023-10-16T13:49:03.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.427+0000 7f10bb172700 1 -- 172.21.15.16:0/518083893 wait complete. 2023-10-16T13:49:03.442 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-16T13:49:03.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.568+0000 7fb8555e4700 1 Processor -- start 2023-10-16T13:49:03.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.568+0000 7fb8555e4700 1 -- start start 2023-10-16T13:49:03.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.568+0000 7fb8555e4700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb8500b8fa0 0x7fb8500b2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:03.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.568+0000 7fb8555e4700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb8500ce350 0x7fb8500ce7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:03.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.568+0000 7fb8555e4700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8500cece0 con 0x7fb85015d8d0 2023-10-16T13:49:03.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.568+0000 7fb8555e4700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8500d0bb0 con 0x7fb8500ce350 2023-10-16T13:49:03.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.568+0000 7fb8555e4700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb85015dcb0 con 0x7fb8500b8fa0 2023-10-16T13:49:03.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.569+0000 7fb84f7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb8500ce350 0x7fb8500ce7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:03.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.569+0000 7fb84effd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb85015d8d0 0x7fb8500b6720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48260/0 (socket says 172.21.15.16:48260) 2023-10-16T13:49:03.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.569+0000 7fb84effd700 1 -- 172.21.15.16:0/1407083940 learned_addr learned my addr 172.21.15.16:0/1407083940 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:49:03.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.569+0000 7fb84e7fc700 1 --2- 172.21.15.16:0/1407083940 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb8500b8fa0 0x7fb8500b2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:03.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.569+0000 7fb84f7fe700 1 -- 172.21.15.16:0/1407083940 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb8500b8fa0 msgr2=0x7fb8500b2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:03.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.569+0000 7fb84f7fe700 1 --2- 172.21.15.16:0/1407083940 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb8500b8fa0 0x7fb8500b2830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:03.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.569+0000 7fb84f7fe700 1 -- 172.21.15.16:0/1407083940 >> v1:172.21.15.16:6789/0 conn(0x7fb85015d8d0 legacy=0x7fb8500b6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:03.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.569+0000 7fb84f7fe700 1 -- 172.21.15.16:0/1407083940 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8500b2f30 con 0x7fb8500ce350 2023-10-16T13:49:03.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.569+0000 7fb84dffb700 1 -- 172.21.15.16:0/1407083940 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2724770144 0 0) 0x7fb840001110 con 0x7fb85015d8d0 2023-10-16T13:49:03.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.569+0000 7fb84f7fe700 1 --2- 172.21.15.16:0/1407083940 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8500ce350 0x7fb8500ce7a0 secure :-1 s=READY pgs=3184 cs=0 l=1 rev1=1 crypto rx=0x7fb848009d90 tx=0x7fb84800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2297c041a779f47 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:03.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.569+0000 7fb84dffb700 1 -- 172.21.15.16:0/1407083940 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb84800e070 con 0x7fb8500ce350 2023-10-16T13:49:03.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.569+0000 7fb84dffb700 1 -- 172.21.15.16:0/1407083940 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb84800bbd0 con 0x7fb8500ce350 2023-10-16T13:49:03.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.569+0000 7fb84dffb700 1 -- 172.21.15.16:0/1407083940 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb848012430 con 0x7fb8500ce350 2023-10-16T13:49:03.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.570+0000 7fb8555e4700 1 -- 172.21.15.16:0/1407083940 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8500ce350 msgr2=0x7fb8500ce7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:03.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.570+0000 7fb8555e4700 1 --2- 172.21.15.16:0/1407083940 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8500ce350 0x7fb8500ce7a0 secure :-1 s=READY pgs=3184 cs=0 l=1 rev1=1 crypto rx=0x7fb848009d90 tx=0x7fb84800b330 comp rx=0 tx=0).stop 2023-10-16T13:49:03.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.570+0000 7fb8555e4700 1 -- 172.21.15.16:0/1407083940 shutdown_connections 2023-10-16T13:49:03.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.570+0000 7fb8555e4700 1 --2- 172.21.15.16:0/1407083940 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8500ce350 0x7fb8500ce7a0 unknown :-1 s=CLOSED pgs=3184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:03.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.570+0000 7fb8555e4700 1 --2- 172.21.15.16:0/1407083940 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb8500b8fa0 0x7fb8500b2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:03.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.570+0000 7fb8555e4700 1 -- 172.21.15.16:0/1407083940 >> 172.21.15.16:0/1407083940 conn(0x7fb850072330 msgr2=0x7fb850072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:03.577 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.570+0000 7fb8555e4700 1 -- 172.21.15.16:0/1407083940 shutdown_connections 2023-10-16T13:49:03.578 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.570+0000 7fb8555e4700 1 -- 172.21.15.16:0/1407083940 wait complete. 2023-10-16T13:49:03.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.571+0000 7fb8555e4700 1 Processor -- start 2023-10-16T13:49:03.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.571+0000 7fb8555e4700 1 -- start start 2023-10-16T13:49:03.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.572+0000 7fb8555e4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8500b8fa0 0x7fb850151510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:03.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.572+0000 7fb8555e4700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb8500ce350 0x7fb85014f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:03.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.572+0000 7fb8555e4700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb85015d8d0 0x7fb8500c8960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:03.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.572+0000 7fb8555e4700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8500b3120 con 0x7fb8500b8fa0 2023-10-16T13:49:03.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.572+0000 7fb8555e4700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8500b3c30 con 0x7fb8500ce350 2023-10-16T13:49:03.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.572+0000 7fb8555e4700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb850151a50 con 0x7fb85015d8d0 2023-10-16T13:49:03.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.572+0000 7fb84effd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8500b8fa0 0x7fb850151510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:03.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.572+0000 7fb84effd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8500b8fa0 0x7fb850151510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58990/0 (socket says 172.21.15.16:58990) 2023-10-16T13:49:03.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.572+0000 7fb84effd700 1 -- 172.21.15.16:0/4269333291 learned_addr learned my addr 172.21.15.16:0/4269333291 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:03.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.572+0000 7fb84e7fc700 1 --2- 172.21.15.16:0/4269333291 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb8500ce350 0x7fb85014f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:03.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.572+0000 7fb84f7fe700 1 --2- 172.21.15.16:0/4269333291 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb85015d8d0 0x7fb8500c8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:03.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.572+0000 7fb84effd700 1 -- 172.21.15.16:0/4269333291 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb85015d8d0 msgr2=0x7fb8500c8960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:03.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.572+0000 7fb84effd700 1 --2- 172.21.15.16:0/4269333291 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb85015d8d0 0x7fb8500c8960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:03.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.572+0000 7fb84effd700 1 -- 172.21.15.16:0/4269333291 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb8500ce350 msgr2=0x7fb85014f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:03.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.572+0000 7fb84effd700 1 --2- 172.21.15.16:0/4269333291 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb8500ce350 0x7fb85014f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:03.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.572+0000 7fb84effd700 1 -- 172.21.15.16:0/4269333291 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb848009a70 con 0x7fb8500b8fa0 2023-10-16T13:49:03.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.572+0000 7fb84f7fe700 1 --2- 172.21.15.16:0/4269333291 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb85015d8d0 0x7fb8500c8960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:49:03.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.573+0000 7fb84effd700 1 --2- 172.21.15.16:0/4269333291 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8500b8fa0 0x7fb850151510 secure :-1 s=READY pgs=3185 cs=0 l=1 rev1=1 crypto rx=0x7fb84000c580 tx=0x7fb84000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:03.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.573+0000 7fb8377fe700 1 -- 172.21.15.16:0/4269333291 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb840004c80 con 0x7fb8500b8fa0 2023-10-16T13:49:03.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.573+0000 7fb8377fe700 1 -- 172.21.15.16:0/4269333291 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb840004e00 con 0x7fb8500b8fa0 2023-10-16T13:49:03.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.573+0000 7fb8377fe700 1 -- 172.21.15.16:0/4269333291 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb840007310 con 0x7fb8500b8fa0 2023-10-16T13:49:03.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.573+0000 7fb8555e4700 1 -- 172.21.15.16:0/4269333291 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8500b3300 con 0x7fb8500b8fa0 2023-10-16T13:49:03.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.573+0000 7fb8555e4700 1 -- 172.21.15.16:0/4269333291 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8500b38c0 con 0x7fb8500b8fa0 2023-10-16T13:49:03.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.574+0000 7fb8555e4700 1 -- 172.21.15.16:0/4269333291 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb85004ebb0 con 0x7fb8500b8fa0 2023-10-16T13:49:03.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.574+0000 7fb8377fe700 1 -- 172.21.15.16:0/4269333291 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 166) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb840007470 con 0x7fb8500b8fa0 2023-10-16T13:49:03.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.575+0000 7fb8377fe700 1 --2- 172.21.15.16:0/4269333291 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fb83c06b020 0x7fb83c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:03.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.575+0000 7fb8377fe700 1 -- 172.21.15.16:0/4269333291 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(278..278 src has 1..278) v4 ==== 6157+0+0 (secure 0 0 0) 0x7fb8400893b0 con 0x7fb8500b8fa0 2023-10-16T13:49:03.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.575+0000 7fb84e7fc700 1 --2- 172.21.15.16:0/4269333291 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fb83c06b020 0x7fb83c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:03.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.575+0000 7fb84e7fc700 1 --2- 172.21.15.16:0/4269333291 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fb83c06b020 0x7fb83c06d4e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb8500b2250 tx=0x7fb844008040 comp rx=0 tx=0).ready entity=mgr.13344 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:03.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.579+0000 7fb8377fe700 1 -- 172.21.15.16:0/4269333291 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb840054190 con 0x7fb8500b8fa0 2023-10-16T13:49:03.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:03.835+0000 7fb8555e4700 1 -- 172.21.15.16:0/4269333291 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fb8500cece0 con 0x7fb8500b8fa0 2023-10-16T13:49:03.932 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:49:03.935 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:49:04.672 INFO:tasks.ceph.mds.b.smithi049.stderr:2023-10-16T13:49:04.669+0000 7f277ab42700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T13:49:04.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:04.706+0000 7fb8377fe700 1 -- 172.21.15.16:0/4269333291 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v154) v1 ==== 118+0+0 (secure 0 0 0) 0x7fb8500cece0 con 0x7fb8500b8fa0 2023-10-16T13:49:04.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:04.709+0000 7fb8555e4700 1 -- 172.21.15.16:0/4269333291 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fb83c06b020 msgr2=0x7fb83c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:04.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:04.709+0000 7fb8555e4700 1 --2- 172.21.15.16:0/4269333291 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fb83c06b020 0x7fb83c06d4e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb8500b2250 tx=0x7fb844008040 comp rx=0 tx=0).stop 2023-10-16T13:49:04.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:04.709+0000 7fb8555e4700 1 -- 172.21.15.16:0/4269333291 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8500b8fa0 msgr2=0x7fb850151510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:04.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:04.709+0000 7fb8555e4700 1 --2- 172.21.15.16:0/4269333291 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8500b8fa0 0x7fb850151510 secure :-1 s=READY pgs=3185 cs=0 l=1 rev1=1 crypto rx=0x7fb84000c580 tx=0x7fb84000c890 comp rx=0 tx=0).stop 2023-10-16T13:49:04.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:04.709+0000 7fb8555e4700 1 -- 172.21.15.16:0/4269333291 shutdown_connections 2023-10-16T13:49:04.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:04.709+0000 7fb8555e4700 1 --2- 172.21.15.16:0/4269333291 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb85015d8d0 0x7fb8500c8960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:04.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:04.710+0000 7fb8555e4700 1 --2- 172.21.15.16:0/4269333291 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fb83c06b020 0x7fb83c06d4e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:04.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:04.710+0000 7fb8555e4700 1 --2- 172.21.15.16:0/4269333291 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8500b8fa0 0x7fb850151510 unknown :-1 s=CLOSED pgs=3185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:04.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:04.710+0000 7fb8555e4700 1 --2- 172.21.15.16:0/4269333291 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb8500ce350 0x7fb85014f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:04.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:04.710+0000 7fb8555e4700 1 -- 172.21.15.16:0/4269333291 >> 172.21.15.16:0/4269333291 conn(0x7fb850072330 msgr2=0x7fb8500d2810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:04.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:04.710+0000 7fb8555e4700 1 -- 172.21.15.16:0/4269333291 shutdown_connections 2023-10-16T13:49:04.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:04.710+0000 7fb8555e4700 1 -- 172.21.15.16:0/4269333291 wait complete. 2023-10-16T13:49:04.729 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-16T13:49:04.729 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-16T13:49:04.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:04.867+0000 7f16a5753700 1 Processor -- start 2023-10-16T13:49:04.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:04.867+0000 7f16a5753700 1 -- start start 2023-10-16T13:49:04.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:04.867+0000 7f16a5753700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f16a00b8fa0 0x7f16a00b2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:04.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:04.867+0000 7f16a5753700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f16a00ce350 0x7f16a00ce780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:04.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:04.867+0000 7f16a5753700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16a00cecc0 con 0x7f16a015d8d0 2023-10-16T13:49:04.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:04.867+0000 7f16a5753700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16a00d0bb0 con 0x7f16a00b8fa0 2023-10-16T13:49:04.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:04.867+0000 7f16a5753700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16a015dcb0 con 0x7f16a00ce350 2023-10-16T13:49:04.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:04.868+0000 7f169e7fc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f16a00b8fa0 0x7f16a00b2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:04.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:04.868+0000 7f169effd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f16a015d8d0 0x7f16a00b6720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48272/0 (socket says 172.21.15.16:48272) 2023-10-16T13:49:04.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:04.868+0000 7f169effd700 1 -- 172.21.15.16:0/3297981421 learned_addr learned my addr 172.21.15.16:0/3297981421 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:49:04.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:04.868+0000 7f169f7fe700 1 --2- 172.21.15.16:0/3297981421 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f16a00ce350 0x7f16a00ce780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:04.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:04.868+0000 7f169e7fc700 1 -- 172.21.15.16:0/3297981421 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f16a00ce350 msgr2=0x7f16a00ce780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:04.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:04.868+0000 7f169e7fc700 1 --2- 172.21.15.16:0/3297981421 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f16a00ce350 0x7f16a00ce780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:04.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:04.868+0000 7f169e7fc700 1 -- 172.21.15.16:0/3297981421 >> v1:172.21.15.16:6789/0 conn(0x7f16a015d8d0 legacy=0x7f16a00b6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:04.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:04.868+0000 7f169e7fc700 1 -- 172.21.15.16:0/3297981421 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16a00b2f30 con 0x7f16a00b8fa0 2023-10-16T13:49:04.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:04.868+0000 7f169dffb700 1 -- 172.21.15.16:0/3297981421 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 532201015 0 0) 0x7f16a00cecc0 con 0x7f16a015d8d0 2023-10-16T13:49:04.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:04.868+0000 7f169e7fc700 1 --2- 172.21.15.16:0/3297981421 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16a00b8fa0 0x7f16a00b2830 secure :-1 s=READY pgs=3187 cs=0 l=1 rev1=1 crypto rx=0x7f1690009840 tx=0x7f1690009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c502b654d6194c7f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:04.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:04.868+0000 7f169dffb700 1 -- 172.21.15.16:0/3297981421 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1690011830 con 0x7f16a00b8fa0 2023-10-16T13:49:04.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:04.868+0000 7f169dffb700 1 -- 172.21.15.16:0/3297981421 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1690011990 con 0x7f16a00b8fa0 2023-10-16T13:49:04.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:04.868+0000 7f169dffb700 1 -- 172.21.15.16:0/3297981421 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1690011d20 con 0x7f16a00b8fa0 2023-10-16T13:49:04.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:04.869+0000 7f16a5753700 1 -- 172.21.15.16:0/3297981421 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16a00b8fa0 msgr2=0x7f16a00b2830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:04.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:04.869+0000 7f16a5753700 1 --2- 172.21.15.16:0/3297981421 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16a00b8fa0 0x7f16a00b2830 secure :-1 s=READY pgs=3187 cs=0 l=1 rev1=1 crypto rx=0x7f1690009840 tx=0x7f1690009b50 comp rx=0 tx=0).stop 2023-10-16T13:49:04.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:04.869+0000 7f16a5753700 1 -- 172.21.15.16:0/3297981421 shutdown_connections 2023-10-16T13:49:04.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:04.869+0000 7f16a5753700 1 --2- 172.21.15.16:0/3297981421 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f16a00ce350 0x7f16a00ce780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:04.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:04.869+0000 7f16a5753700 1 --2- 172.21.15.16:0/3297981421 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16a00b8fa0 0x7f16a00b2830 unknown :-1 s=CLOSED pgs=3187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:04.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:04.869+0000 7f16a5753700 1 -- 172.21.15.16:0/3297981421 >> 172.21.15.16:0/3297981421 conn(0x7f16a0072330 msgr2=0x7f16a0072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:04.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:04.869+0000 7f16a5753700 1 -- 172.21.15.16:0/3297981421 shutdown_connections 2023-10-16T13:49:04.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:04.869+0000 7f16a5753700 1 -- 172.21.15.16:0/3297981421 wait complete. 2023-10-16T13:49:04.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:04.870+0000 7f16a5753700 1 Processor -- start 2023-10-16T13:49:04.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:04.870+0000 7f16a5753700 1 -- start start 2023-10-16T13:49:04.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:04.871+0000 7f16a5753700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16a00b8fa0 0x7f16a015ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:04.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:04.871+0000 7f16a5753700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f16a00ce350 0x7f16a00c7040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:04.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:04.871+0000 7f16a5753700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f16a015d8d0 0x7f16a00c7580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:04.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:04.871+0000 7f16a5753700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16a00c9040 con 0x7f16a00b8fa0 2023-10-16T13:49:04.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:04.871+0000 7f16a5753700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16a00b31c0 con 0x7f16a015d8d0 2023-10-16T13:49:04.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:04.871+0000 7f16a5753700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16a00b34a0 con 0x7f16a00ce350 2023-10-16T13:49:04.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:04.871+0000 7f169effd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16a00b8fa0 0x7f16a015ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:04.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:04.871+0000 7f169effd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16a00b8fa0 0x7f16a015ba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59004/0 (socket says 172.21.15.16:59004) 2023-10-16T13:49:04.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:04.871+0000 7f169effd700 1 -- 172.21.15.16:0/1876597250 learned_addr learned my addr 172.21.15.16:0/1876597250 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:04.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:04.871+0000 7f169e7fc700 1 --2- 172.21.15.16:0/1876597250 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f16a00ce350 0x7f16a00c7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:04.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:04.871+0000 7f169f7fe700 1 --2- 172.21.15.16:0/1876597250 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f16a015d8d0 0x7f16a00c7580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:04.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:04.872+0000 7f169effd700 1 -- 172.21.15.16:0/1876597250 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f16a00ce350 msgr2=0x7f16a00c7040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:04.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:04.872+0000 7f169effd700 1 --2- 172.21.15.16:0/1876597250 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f16a00ce350 0x7f16a00c7040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:04.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:04.872+0000 7f169effd700 1 -- 172.21.15.16:0/1876597250 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f16a015d8d0 msgr2=0x7f16a00c7580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:04.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:04.872+0000 7f169effd700 1 --2- 172.21.15.16:0/1876597250 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f16a015d8d0 0x7f16a00c7580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:04.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:04.872+0000 7f169effd700 1 -- 172.21.15.16:0/1876597250 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16900094a0 con 0x7f16a00b8fa0 2023-10-16T13:49:04.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:04.872+0000 7f169effd700 1 --2- 172.21.15.16:0/1876597250 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16a00b8fa0 0x7f16a015ba90 secure :-1 s=READY pgs=3188 cs=0 l=1 rev1=1 crypto rx=0x7f169400c550 tx=0x7f169400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:04.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:04.872+0000 7f16877fe700 1 -- 172.21.15.16:0/1876597250 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1694004c80 con 0x7f16a00b8fa0 2023-10-16T13:49:04.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:04.872+0000 7f16877fe700 1 -- 172.21.15.16:0/1876597250 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1694004e00 con 0x7f16a00b8fa0 2023-10-16T13:49:04.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:04.872+0000 7f16a5753700 1 -- 172.21.15.16:0/1876597250 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16a00b3780 con 0x7f16a00b8fa0 2023-10-16T13:49:04.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:04.872+0000 7f16877fe700 1 -- 172.21.15.16:0/1876597250 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1694007310 con 0x7f16a00b8fa0 2023-10-16T13:49:04.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:04.872+0000 7f16a5753700 1 -- 172.21.15.16:0/1876597250 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16a00c9320 con 0x7f16a00b8fa0 2023-10-16T13:49:04.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:04.873+0000 7f16a5753700 1 -- 172.21.15.16:0/1876597250 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16a00cd9d0 con 0x7f16a00b8fa0 2023-10-16T13:49:04.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:04.874+0000 7f16877fe700 1 -- 172.21.15.16:0/1876597250 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 166) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1694007470 con 0x7f16a00b8fa0 2023-10-16T13:49:04.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:04.874+0000 7f16877fe700 1 --2- 172.21.15.16:0/1876597250 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f168c06aff0 0x7f168c06d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:04.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:04.874+0000 7f16877fe700 1 -- 172.21.15.16:0/1876597250 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(278..278 src has 1..278) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f1694089520 con 0x7f16a00b8fa0 2023-10-16T13:49:04.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:04.874+0000 7f169e7fc700 1 --2- 172.21.15.16:0/1876597250 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f168c06aff0 0x7f168c06d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:04.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:04.874+0000 7f169e7fc700 1 --2- 172.21.15.16:0/1876597250 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f168c06aff0 0x7f168c06d4b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f16900090b0 tx=0x7f169001c040 comp rx=0 tx=0).ready entity=mgr.13344 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:04.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:04.878+0000 7f16877fe700 1 -- 172.21.15.16:0/1876597250 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1694054300 con 0x7f16a00b8fa0 2023-10-16T13:49:05.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:05.139+0000 7f16a5753700 1 -- 172.21.15.16:0/1876597250 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f16a004ebb0 con 0x7f16a00b8fa0 2023-10-16T13:49:05.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:05.708+0000 7f16877fe700 1 -- 172.21.15.16:0/1876597250 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v155) v1 ==== 122+0+0 (secure 0 0 0) 0x7f1694057920 con 0x7f16a00b8fa0 2023-10-16T13:49:05.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:05.711+0000 7f16a5753700 1 -- 172.21.15.16:0/1876597250 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f168c06aff0 msgr2=0x7f168c06d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:05.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:05.711+0000 7f16a5753700 1 --2- 172.21.15.16:0/1876597250 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f168c06aff0 0x7f168c06d4b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f16900090b0 tx=0x7f169001c040 comp rx=0 tx=0).stop 2023-10-16T13:49:05.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:05.711+0000 7f16a5753700 1 -- 172.21.15.16:0/1876597250 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16a00b8fa0 msgr2=0x7f16a015ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:05.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:05.711+0000 7f16a5753700 1 --2- 172.21.15.16:0/1876597250 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16a00b8fa0 0x7f16a015ba90 secure :-1 s=READY pgs=3188 cs=0 l=1 rev1=1 crypto rx=0x7f169400c550 tx=0x7f169400c860 comp rx=0 tx=0).stop 2023-10-16T13:49:05.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:05.712+0000 7f16a5753700 1 -- 172.21.15.16:0/1876597250 shutdown_connections 2023-10-16T13:49:05.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:05.712+0000 7f16a5753700 1 --2- 172.21.15.16:0/1876597250 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f16a015d8d0 0x7f16a00c7580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:05.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:05.712+0000 7f16a5753700 1 --2- 172.21.15.16:0/1876597250 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f168c06aff0 0x7f168c06d4b0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:05.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:05.712+0000 7f16a5753700 1 --2- 172.21.15.16:0/1876597250 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16a00b8fa0 0x7f16a015ba90 unknown :-1 s=CLOSED pgs=3188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:05.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:05.712+0000 7f16a5753700 1 --2- 172.21.15.16:0/1876597250 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f16a00ce350 0x7f16a00c7040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:05.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:05.712+0000 7f16a5753700 1 -- 172.21.15.16:0/1876597250 >> 172.21.15.16:0/1876597250 conn(0x7f16a0072330 msgr2=0x7f16a00cff00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:05.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:05.712+0000 7f16a5753700 1 -- 172.21.15.16:0/1876597250 shutdown_connections 2023-10-16T13:49:05.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:05.712+0000 7f16a5753700 1 -- 172.21.15.16:0/1876597250 wait complete. 2023-10-16T13:49:05.730 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-16T13:49:05.730 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:49:05.863 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:05.861+0000 7f2f3737c700 1 Processor -- start 2023-10-16T13:49:05.864 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:05.861+0000 7f2f3737c700 1 -- start start 2023-10-16T13:49:05.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:05.862+0000 7f2f3737c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2f3015d230 0x7f2f300b5a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:05.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:05.862+0000 7f2f3737c700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f3015bc80 con 0x7f2f300cf880 2023-10-16T13:49:05.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:05.862+0000 7f2f3737c700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f3015b070 con 0x7f2f300cf4a0 2023-10-16T13:49:05.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:05.862+0000 7f2f3737c700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f3015b350 con 0x7f2f3015d230 2023-10-16T13:49:05.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:05.862+0000 7f2f35919700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2f3015d230 0x7f2f300b5a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:05.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:05.862+0000 7f2f34917700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2f300cf880 0x7f2f300b52f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48288/0 (socket says 172.21.15.16:48288) 2023-10-16T13:49:05.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:05.862+0000 7f2f34917700 1 -- 172.21.15.16:0/1369172435 learned_addr learned my addr 172.21.15.16:0/1369172435 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:49:05.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:05.862+0000 7f2f35919700 1 -- 172.21.15.16:0/1369172435 >> v1:172.21.15.49:6789/0 conn(0x7f2f300cf4a0 legacy=0x7f2f300726e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:05.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:05.862+0000 7f2f35919700 1 -- 172.21.15.16:0/1369172435 >> v1:172.21.15.16:6789/0 conn(0x7f2f300cf880 legacy=0x7f2f300b52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:05.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:05.862+0000 7f2f35919700 1 -- 172.21.15.16:0/1369172435 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f3015b630 con 0x7f2f3015d230 2023-10-16T13:49:05.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:05.862+0000 7f2f27fff700 1 -- 172.21.15.16:0/1369172435 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3483604784 0 0) 0x7f2f3015bc80 con 0x7f2f300cf880 2023-10-16T13:49:05.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:05.863+0000 7f2f35919700 1 --2- 172.21.15.16:0/1369172435 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f3015d230 0x7f2f300b5a00 secure :-1 s=READY pgs=3190 cs=0 l=1 rev1=1 crypto rx=0x7f2f28009d90 tx=0x7f2f2800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a08af8a9858528b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:05.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:05.863+0000 7f2f27fff700 1 -- 172.21.15.16:0/1369172435 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f2800e070 con 0x7f2f3015d230 2023-10-16T13:49:05.868 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:05.863+0000 7f2f27fff700 1 -- 172.21.15.16:0/1369172435 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f2800bb90 con 0x7f2f3015d230 2023-10-16T13:49:05.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:05.863+0000 7f2f27fff700 1 -- 172.21.15.16:0/1369172435 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f28003a20 con 0x7f2f3015d230 2023-10-16T13:49:05.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:05.863+0000 7f2f3737c700 1 -- 172.21.15.16:0/1369172435 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f3015d230 msgr2=0x7f2f300b5a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:05.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:05.863+0000 7f2f3737c700 1 --2- 172.21.15.16:0/1369172435 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f3015d230 0x7f2f300b5a00 secure :-1 s=READY pgs=3190 cs=0 l=1 rev1=1 crypto rx=0x7f2f28009d90 tx=0x7f2f2800b330 comp rx=0 tx=0).stop 2023-10-16T13:49:05.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:05.863+0000 7f2f3737c700 1 -- 172.21.15.16:0/1369172435 shutdown_connections 2023-10-16T13:49:05.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:05.863+0000 7f2f3737c700 1 --2- 172.21.15.16:0/1369172435 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f3015d230 0x7f2f300b5a00 unknown :-1 s=CLOSED pgs=3190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:05.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:05.863+0000 7f2f3737c700 1 -- 172.21.15.16:0/1369172435 >> 172.21.15.16:0/1369172435 conn(0x7f2f300b1fd0 msgr2=0x7f2f300b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:05.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:05.864+0000 7f2f3737c700 1 -- 172.21.15.16:0/1369172435 shutdown_connections 2023-10-16T13:49:05.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:05.864+0000 7f2f3737c700 1 -- 172.21.15.16:0/1369172435 wait complete. 2023-10-16T13:49:05.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:05.865+0000 7f2f3737c700 1 Processor -- start 2023-10-16T13:49:05.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:05.865+0000 7f2f3737c700 1 -- start start 2023-10-16T13:49:05.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:05.865+0000 7f2f3737c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2f300cf4a0 0x7f2f3014df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:05.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:05.865+0000 7f2f3737c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f300cf880 0x7f2f301f2700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:05.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:05.865+0000 7f2f3737c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2f3015d230 0x7f2f3014c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:05.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:05.865+0000 7f2f3737c700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f3014ce90 con 0x7f2f300cf880 2023-10-16T13:49:05.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:05.865+0000 7f2f3737c700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f3014d170 con 0x7f2f3015d230 2023-10-16T13:49:05.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:05.865+0000 7f2f3737c700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f3014d450 con 0x7f2f300cf4a0 2023-10-16T13:49:05.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:05.866+0000 7f2f34917700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f300cf880 0x7f2f301f2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:05.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:05.866+0000 7f2f34917700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f300cf880 0x7f2f301f2700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59020/0 (socket says 172.21.15.16:59020) 2023-10-16T13:49:05.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:05.866+0000 7f2f34917700 1 -- 172.21.15.16:0/714424795 learned_addr learned my addr 172.21.15.16:0/714424795 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:05.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:05.866+0000 7f2f35919700 1 --2- 172.21.15.16:0/714424795 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2f3015d230 0x7f2f3014c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:05.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:05.866+0000 7f2f35118700 1 --2- 172.21.15.16:0/714424795 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2f300cf4a0 0x7f2f3014df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:05.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:05.866+0000 7f2f34917700 1 -- 172.21.15.16:0/714424795 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2f300cf4a0 msgr2=0x7f2f3014df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:05.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:05.866+0000 7f2f34917700 1 --2- 172.21.15.16:0/714424795 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2f300cf4a0 0x7f2f3014df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:05.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:05.866+0000 7f2f34917700 1 -- 172.21.15.16:0/714424795 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2f3015d230 msgr2=0x7f2f3014c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:05.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:05.866+0000 7f2f34917700 1 --2- 172.21.15.16:0/714424795 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2f3015d230 0x7f2f3014c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:05.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:05.866+0000 7f2f34917700 1 -- 172.21.15.16:0/714424795 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f28009a70 con 0x7f2f300cf880 2023-10-16T13:49:05.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:05.866+0000 7f2f34917700 1 --2- 172.21.15.16:0/714424795 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f300cf880 0x7f2f301f2700 secure :-1 s=READY pgs=3191 cs=0 l=1 rev1=1 crypto rx=0x7f2f2c0011e0 tx=0x7f2f2c00cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:05.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:05.866+0000 7f2f25ffb700 1 -- 172.21.15.16:0/714424795 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f2c004c80 con 0x7f2f300cf880 2023-10-16T13:49:05.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:05.866+0000 7f2f25ffb700 1 -- 172.21.15.16:0/714424795 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f2c004e00 con 0x7f2f300cf880 2023-10-16T13:49:05.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:05.866+0000 7f2f25ffb700 1 -- 172.21.15.16:0/714424795 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f2c007310 con 0x7f2f300cf880 2023-10-16T13:49:05.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:05.866+0000 7f2f3737c700 1 -- 172.21.15.16:0/714424795 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f3014d730 con 0x7f2f300cf880 2023-10-16T13:49:05.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:05.866+0000 7f2f3737c700 1 -- 172.21.15.16:0/714424795 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f301f2c40 con 0x7f2f300cf880 2023-10-16T13:49:05.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:05.867+0000 7f2f3737c700 1 -- 172.21.15.16:0/714424795 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f30067f80 con 0x7f2f300cf880 2023-10-16T13:49:05.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:05.868+0000 7f2f25ffb700 1 -- 172.21.15.16:0/714424795 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 166) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2f2c007470 con 0x7f2f300cf880 2023-10-16T13:49:05.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:05.868+0000 7f2f25ffb700 1 --2- 172.21.15.16:0/714424795 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f2f1406b020 0x7f2f1406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:05.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:05.868+0000 7f2f25ffb700 1 -- 172.21.15.16:0/714424795 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(278..278 src has 1..278) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f2f2c0896e0 con 0x7f2f300cf880 2023-10-16T13:49:05.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:05.868+0000 7f2f35118700 1 --2- 172.21.15.16:0/714424795 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f2f1406b020 0x7f2f1406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:05.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:05.869+0000 7f2f35118700 1 --2- 172.21.15.16:0/714424795 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f2f1406b020 0x7f2f1406d4e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2f20004f30 tx=0x7f2f20009040 comp rx=0 tx=0).ready entity=mgr.13344 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:05.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:05.873+0000 7f2f25ffb700 1 -- 172.21.15.16:0/714424795 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2f2c0544c0 con 0x7f2f300cf880 2023-10-16T13:49:06.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.121+0000 7f2f3737c700 1 -- 172.21.15.16:0/714424795 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2f3015b8e0 con 0x7f2f300cf880 2023-10-16T13:49:06.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.122+0000 7f2f25ffb700 1 -- 172.21.15.16:0/714424795 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 155 v155) v1 ==== 95+0+4926 (secure 0 0 0) 0x7f2f2c057ae0 con 0x7f2f300cf880 2023-10-16T13:49:06.125 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:49:06.125 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":155,"default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12280,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/1127700443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":1127700443},{"type":"v1","addr":"172.21.15.49:6834","nonce":1127700443}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":150},{"gid":13581,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/818113029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":818113029},{"type":"v1","addr":"172.21.15.16:6836","nonce":818113029}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149},{"gid":13584,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3762868123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":3762868123},{"type":"v1","addr":"172.21.15.16:6837","nonce":3762868123}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":150}],"filesystems":[{"mdsmap":{"epoch":155,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:49:02.107137+0000","modified":"2023-10-16T13:49:05.140377+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":13599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13599":{"gid":13599,"name":"b","rank":0,"incarnation":151,"state":"up:active","state_seq":3,"addr":"172.21.15.49:6835/2082691635","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":2082691635},{"type":"v1","addr":"172.21.15.49:6835","nonce":2082691635}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[38],"metadata_pool":37,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":18}]} 2023-10-16T13:49:06.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.124+0000 7f2f3737c700 1 -- 172.21.15.16:0/714424795 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f2f1406b020 msgr2=0x7f2f1406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:06.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.124+0000 7f2f3737c700 1 --2- 172.21.15.16:0/714424795 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f2f1406b020 0x7f2f1406d4e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2f20004f30 tx=0x7f2f20009040 comp rx=0 tx=0).stop 2023-10-16T13:49:06.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.124+0000 7f2f3737c700 1 -- 172.21.15.16:0/714424795 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f300cf880 msgr2=0x7f2f301f2700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:06.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.124+0000 7f2f3737c700 1 --2- 172.21.15.16:0/714424795 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f300cf880 0x7f2f301f2700 secure :-1 s=READY pgs=3191 cs=0 l=1 rev1=1 crypto rx=0x7f2f2c0011e0 tx=0x7f2f2c00cae0 comp rx=0 tx=0).stop 2023-10-16T13:49:06.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.124+0000 7f2f3737c700 1 -- 172.21.15.16:0/714424795 shutdown_connections 2023-10-16T13:49:06.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.124+0000 7f2f3737c700 1 --2- 172.21.15.16:0/714424795 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2f3015d230 0x7f2f3014c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:06.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.124+0000 7f2f3737c700 1 --2- 172.21.15.16:0/714424795 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2f300cf4a0 0x7f2f3014df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:06.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.124+0000 7f2f3737c700 1 --2- 172.21.15.16:0/714424795 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f2f1406b020 0x7f2f1406d4e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:06.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.124+0000 7f2f3737c700 1 --2- 172.21.15.16:0/714424795 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f300cf880 0x7f2f301f2700 unknown :-1 s=CLOSED pgs=3191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:06.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.124+0000 7f2f3737c700 1 -- 172.21.15.16:0/714424795 >> 172.21.15.16:0/714424795 conn(0x7f2f300b1fd0 msgr2=0x7f2f300b1750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:06.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.124+0000 7f2f3737c700 1 -- 172.21.15.16:0/714424795 shutdown_connections 2023-10-16T13:49:06.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.124+0000 7f2f3737c700 1 -- 172.21.15.16:0/714424795 wait complete. 2023-10-16T13:49:06.129 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 155 2023-10-16T13:49:06.139 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:49:06.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.262+0000 7f0ea546c700 1 Processor -- start 2023-10-16T13:49:06.265 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.262+0000 7f0ea546c700 1 -- start start 2023-10-16T13:49:06.267 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.263+0000 7f0ea546c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0ea015d8d0 0x7f0ea00d1c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:06.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.263+0000 7f0ea546c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0ea00d41c0 0x7f0ea00d0040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:06.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.263+0000 7f0ea546c700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ea006ba00 con 0x7f0ea00d0580 2023-10-16T13:49:06.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.263+0000 7f0ea546c700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ea00d0a10 con 0x7f0ea015d8d0 2023-10-16T13:49:06.268 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.263+0000 7f0ea546c700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ea00d2150 con 0x7f0ea00d41c0 2023-10-16T13:49:06.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.263+0000 7f0e9effd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0ea015d8d0 0x7f0ea00d1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:06.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.263+0000 7f0e9f7fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0ea00d0580 0x7f0ea00ce240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48304/0 (socket says 172.21.15.16:48304) 2023-10-16T13:49:06.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.263+0000 7f0e9f7fe700 1 -- 172.21.15.16:0/720451831 learned_addr learned my addr 172.21.15.16:0/720451831 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:49:06.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.263+0000 7f0e9effd700 1 --2- 172.21.15.16:0/720451831 >> v2:172.21.15.16:3300/0 conn(0x7f0ea015d8d0 0x7f0ea00d1c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59026/0 (socket says 172.21.15.16:59026) 2023-10-16T13:49:06.269 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.263+0000 7f0e9e7fc700 1 --2- 172.21.15.16:0/720451831 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0ea00d41c0 0x7f0ea00d0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:06.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.263+0000 7f0e9effd700 1 -- 172.21.15.16:0/720451831 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0ea00d41c0 msgr2=0x7f0ea00d0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:06.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.263+0000 7f0e9effd700 1 --2- 172.21.15.16:0/720451831 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0ea00d41c0 0x7f0ea00d0040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:06.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.263+0000 7f0e9effd700 1 -- 172.21.15.16:0/720451831 >> v1:172.21.15.16:6789/0 conn(0x7f0ea00d0580 legacy=0x7f0ea00ce240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:06.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.264+0000 7f0e9effd700 1 -- 172.21.15.16:0/720451831 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ea00d26d0 con 0x7f0ea015d8d0 2023-10-16T13:49:06.270 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.263+0000 7f0e9dffb700 1 -- 172.21.15.16:0/720451831 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1729649978 0 0) 0x7f0ea006ba00 con 0x7f0ea00d0580 2023-10-16T13:49:06.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.264+0000 7f0e9effd700 1 --2- 172.21.15.16:0/720451831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ea015d8d0 0x7f0ea00d1c10 secure :-1 s=READY pgs=3193 cs=0 l=1 rev1=1 crypto rx=0x7f0e94009840 tx=0x7f0e94009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eabad5eef57ee6ff server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:06.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.264+0000 7f0e9dffb700 1 -- 172.21.15.16:0/720451831 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e94011830 con 0x7f0ea015d8d0 2023-10-16T13:49:06.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.264+0000 7f0e9dffb700 1 -- 172.21.15.16:0/720451831 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e94011990 con 0x7f0ea015d8d0 2023-10-16T13:49:06.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.264+0000 7f0e9dffb700 1 -- 172.21.15.16:0/720451831 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e94011d20 con 0x7f0ea015d8d0 2023-10-16T13:49:06.271 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.264+0000 7f0ea546c700 1 -- 172.21.15.16:0/720451831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ea015d8d0 msgr2=0x7f0ea00d1c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:06.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.264+0000 7f0ea546c700 1 --2- 172.21.15.16:0/720451831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ea015d8d0 0x7f0ea00d1c10 secure :-1 s=READY pgs=3193 cs=0 l=1 rev1=1 crypto rx=0x7f0e94009840 tx=0x7f0e94009b50 comp rx=0 tx=0).stop 2023-10-16T13:49:06.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.264+0000 7f0ea546c700 1 -- 172.21.15.16:0/720451831 shutdown_connections 2023-10-16T13:49:06.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.264+0000 7f0ea546c700 1 --2- 172.21.15.16:0/720451831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ea015d8d0 0x7f0ea00d1c10 unknown :-1 s=CLOSED pgs=3193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:06.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.264+0000 7f0ea546c700 1 --2- 172.21.15.16:0/720451831 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0ea00d41c0 0x7f0ea00d0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:06.272 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.264+0000 7f0ea546c700 1 -- 172.21.15.16:0/720451831 >> 172.21.15.16:0/720451831 conn(0x7f0ea00d3630 msgr2=0x7f0ea00d3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:06.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.264+0000 7f0ea546c700 1 -- 172.21.15.16:0/720451831 shutdown_connections 2023-10-16T13:49:06.273 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.265+0000 7f0ea546c700 1 -- 172.21.15.16:0/720451831 wait complete. 2023-10-16T13:49:06.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.266+0000 7f0ea546c700 1 Processor -- start 2023-10-16T13:49:06.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.266+0000 7f0ea546c700 1 -- start start 2023-10-16T13:49:06.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.267+0000 7f0ea546c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ea00d0580 0x7f0ea015bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:06.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.267+0000 7f0ea546c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0ea00d41c0 0x7f0ea01e6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:06.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.267+0000 7f0ea546c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0ea015d8d0 0x7f0ea01e7420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:06.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.267+0000 7f0ea546c700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ea015c200 con 0x7f0ea00d0580 2023-10-16T13:49:06.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.267+0000 7f0ea546c700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ea006d500 con 0x7f0ea00d41c0 2023-10-16T13:49:06.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.267+0000 7f0ea546c700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ea00ccf90 con 0x7f0ea015d8d0 2023-10-16T13:49:06.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.267+0000 7f0e9effd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ea00d0580 0x7f0ea015bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:06.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.267+0000 7f0e9effd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ea00d0580 0x7f0ea015bcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59032/0 (socket says 172.21.15.16:59032) 2023-10-16T13:49:06.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.267+0000 7f0e9effd700 1 -- 172.21.15.16:0/2861958772 learned_addr learned my addr 172.21.15.16:0/2861958772 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:06.277 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.267+0000 7f0e9f7fe700 1 --2- 172.21.15.16:0/2861958772 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0ea015d8d0 0x7f0ea01e7420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:06.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.267+0000 7f0e9effd700 1 -- 172.21.15.16:0/2861958772 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0ea015d8d0 msgr2=0x7f0ea01e7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:06.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.267+0000 7f0e9effd700 1 --2- 172.21.15.16:0/2861958772 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0ea015d8d0 0x7f0ea01e7420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:06.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.267+0000 7f0e9effd700 1 -- 172.21.15.16:0/2861958772 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0ea00d41c0 msgr2=0x7f0ea01e6ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:06.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.267+0000 7f0e9e7fc700 1 --2- 172.21.15.16:0/2861958772 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0ea00d41c0 0x7f0ea01e6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:06.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.267+0000 7f0e9effd700 1 --2- 172.21.15.16:0/2861958772 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0ea00d41c0 0x7f0ea01e6ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:06.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.267+0000 7f0e9effd700 1 -- 172.21.15.16:0/2861958772 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e940094a0 con 0x7f0ea00d0580 2023-10-16T13:49:06.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.267+0000 7f0e9effd700 1 --2- 172.21.15.16:0/2861958772 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ea00d0580 0x7f0ea015bcc0 secure :-1 s=READY pgs=3194 cs=0 l=1 rev1=1 crypto rx=0x7f0e94009fd0 tx=0x7f0e94004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:06.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.267+0000 7f0e877fe700 1 -- 172.21.15.16:0/2861958772 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e940038a0 con 0x7f0ea00d0580 2023-10-16T13:49:06.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.267+0000 7f0e877fe700 1 -- 172.21.15.16:0/2861958772 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e9401e070 con 0x7f0ea00d0580 2023-10-16T13:49:06.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.267+0000 7f0e877fe700 1 -- 172.21.15.16:0/2861958772 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e94003c40 con 0x7f0ea00d0580 2023-10-16T13:49:06.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.267+0000 7f0ea546c700 1 -- 172.21.15.16:0/2861958772 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ea00cc380 con 0x7f0ea00d0580 2023-10-16T13:49:06.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.268+0000 7f0ea546c700 1 -- 172.21.15.16:0/2861958772 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ea00cc940 con 0x7f0ea00d0580 2023-10-16T13:49:06.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.268+0000 7f0ea546c700 1 -- 172.21.15.16:0/2861958772 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ea004ebb0 con 0x7f0ea00d0580 2023-10-16T13:49:06.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.269+0000 7f0e877fe700 1 -- 172.21.15.16:0/2861958772 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 166) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0e9402d450 con 0x7f0ea00d0580 2023-10-16T13:49:06.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.269+0000 7f0e877fe700 1 --2- 172.21.15.16:0/2861958772 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f0e8c06b020 0x7f0e8c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:06.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.269+0000 7f0e877fe700 1 -- 172.21.15.16:0/2861958772 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(278..278 src has 1..278) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f0e9408a8c0 con 0x7f0ea00d0580 2023-10-16T13:49:06.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.269+0000 7f0e9e7fc700 1 --2- 172.21.15.16:0/2861958772 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f0e8c06b020 0x7f0e8c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:06.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.270+0000 7f0e9e7fc700 1 --2- 172.21.15.16:0/2861958772 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f0e8c06b020 0x7f0e8c06d4e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0ea01504a0 tx=0x7f0e90008040 comp rx=0 tx=0).ready entity=mgr.13344 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:06.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.274+0000 7f0e877fe700 1 -- 172.21.15.16:0/2861958772 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0e94055730 con 0x7f0ea00d0580 2023-10-16T13:49:06.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.476+0000 7f0ea546c700 1 -- 172.21.15.16:0/2861958772 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0ea00cf340 con 0x7f0ea00d0580 2023-10-16T13:49:06.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.477+0000 7f0e877fe700 1 -- 172.21.15.16:0/2861958772 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v278) v1 ==== 74+0+16825 (secure 0 0 0) 0x7f0e94020090 con 0x7f0ea00d0580 2023-10-16T13:49:06.480 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:49:06.480 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":278,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:49:03.362245+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":38,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":37,"pool_name":"cephfs_metadata","create_time":"2023-10-16T13:49:00.609612+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"277","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":38,"pool_name":"cephfs_data","create_time":"2023-10-16T13:49:01.068705+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"278","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:49:06.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.479+0000 7f0ea546c700 1 -- 172.21.15.16:0/2861958772 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f0e8c06b020 msgr2=0x7f0e8c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:06.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.479+0000 7f0ea546c700 1 --2- 172.21.15.16:0/2861958772 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f0e8c06b020 0x7f0e8c06d4e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0ea01504a0 tx=0x7f0e90008040 comp rx=0 tx=0).stop 2023-10-16T13:49:06.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.479+0000 7f0ea546c700 1 -- 172.21.15.16:0/2861958772 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ea00d0580 msgr2=0x7f0ea015bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:06.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.479+0000 7f0ea546c700 1 --2- 172.21.15.16:0/2861958772 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ea00d0580 0x7f0ea015bcc0 secure :-1 s=READY pgs=3194 cs=0 l=1 rev1=1 crypto rx=0x7f0e94009fd0 tx=0x7f0e94004750 comp rx=0 tx=0).stop 2023-10-16T13:49:06.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.479+0000 7f0ea546c700 1 -- 172.21.15.16:0/2861958772 shutdown_connections 2023-10-16T13:49:06.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.479+0000 7f0ea546c700 1 --2- 172.21.15.16:0/2861958772 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f0ea015d8d0 0x7f0ea01e7420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:06.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.479+0000 7f0ea546c700 1 --2- 172.21.15.16:0/2861958772 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f0e8c06b020 0x7f0e8c06d4e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:06.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.479+0000 7f0ea546c700 1 --2- 172.21.15.16:0/2861958772 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ea00d0580 0x7f0ea015bcc0 unknown :-1 s=CLOSED pgs=3194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:06.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.479+0000 7f0ea546c700 1 --2- 172.21.15.16:0/2861958772 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0ea00d41c0 0x7f0ea01e6ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:06.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.479+0000 7f0ea546c700 1 -- 172.21.15.16:0/2861958772 >> 172.21.15.16:0/2861958772 conn(0x7f0ea00d3630 msgr2=0x7f0ea015c8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:06.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.479+0000 7f0ea546c700 1 -- 172.21.15.16:0/2861958772 shutdown_connections 2023-10-16T13:49:06.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.479+0000 7f0ea546c700 1 -- 172.21.15.16:0/2861958772 wait complete. 2023-10-16T13:49:06.494 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-16T13:49:06.495 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T13:49:06.620 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.618+0000 7fc5c4ce8700 1 Processor -- start 2023-10-16T13:49:06.621 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.618+0000 7fc5c4ce8700 1 -- start start 2023-10-16T13:49:06.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.619+0000 7fc5c4ce8700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fc5c015b280 0x7fc5c01613e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:06.623 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.619+0000 7fc5c4ce8700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc5c00cc8a0 0x7fc5c015e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:06.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.619+0000 7fc5c4ce8700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc5c006a7e0 0x7fc5c006ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:06.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.619+0000 7fc5c4ce8700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5c006b170 con 0x7fc5c006a7e0 2023-10-16T13:49:06.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.619+0000 7fc5c4ce8700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5c0161c30 con 0x7fc5c015b280 2023-10-16T13:49:06.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.619+0000 7fc5c4ce8700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c0161f10 con 0x7fc5c00cc8a0 2023-10-16T13:49:06.624 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.619+0000 7fc5bed9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc5c006a7e0 0x7fc5c006ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:06.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.619+0000 7fc5bed9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc5c006a7e0 0x7fc5c006ac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59044/0 (socket says 172.21.15.16:59044) 2023-10-16T13:49:06.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.619+0000 7fc5bed9d700 1 -- 172.21.15.16:0/792879588 learned_addr learned my addr 172.21.15.16:0/792879588 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:06.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.619+0000 7fc5bdd9b700 1 --2- 172.21.15.16:0/792879588 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc5c00cc8a0 0x7fc5c015e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:06.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.619+0000 7fc5be59c700 1 --2- 172.21.15.16:0/792879588 >> v2:172.21.15.49:3300/0 conn(0x7fc5c015b280 0x7fc5c01613e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:06.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.619+0000 7fc5bed9d700 1 -- 172.21.15.16:0/792879588 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc5c00cc8a0 msgr2=0x7fc5c015e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:06.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.619+0000 7fc5bed9d700 1 --2- 172.21.15.16:0/792879588 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc5c00cc8a0 0x7fc5c015e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:06.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.619+0000 7fc5bed9d700 1 -- 172.21.15.16:0/792879588 >> v2:172.21.15.49:3300/0 conn(0x7fc5c015b280 msgr2=0x7fc5c01613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:06.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.619+0000 7fc5bed9d700 1 --2- 172.21.15.16:0/792879588 >> v2:172.21.15.49:3300/0 conn(0x7fc5c015b280 0x7fc5c01613e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:06.626 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.619+0000 7fc5bed9d700 1 -- 172.21.15.16:0/792879588 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5c01621f0 con 0x7fc5c006a7e0 2023-10-16T13:49:06.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.619+0000 7fc5be59c700 1 --2- 172.21.15.16:0/792879588 >> v2:172.21.15.49:3300/0 conn(0x7fc5c015b280 0x7fc5c01613e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:49:06.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.619+0000 7fc5bed9d700 1 --2- 172.21.15.16:0/792879588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5c006a7e0 0x7fc5c006ac30 secure :-1 s=READY pgs=3195 cs=0 l=1 rev1=1 crypto rx=0x7fc5c006b8a0 tx=0x7fc5b800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44399dc13462a95e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:06.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.619+0000 7fc5bd59a700 1 -- 172.21.15.16:0/792879588 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b800e070 con 0x7fc5c006a7e0 2023-10-16T13:49:06.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.620+0000 7fc5bd59a700 1 -- 172.21.15.16:0/792879588 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5b800b9c0 con 0x7fc5c006a7e0 2023-10-16T13:49:06.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.620+0000 7fc5bd59a700 1 -- 172.21.15.16:0/792879588 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b800bd70 con 0x7fc5c006a7e0 2023-10-16T13:49:06.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.620+0000 7fc5c4ce8700 1 -- 172.21.15.16:0/792879588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5c006a7e0 msgr2=0x7fc5c006ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:06.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.620+0000 7fc5c4ce8700 1 --2- 172.21.15.16:0/792879588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5c006a7e0 0x7fc5c006ac30 secure :-1 s=READY pgs=3195 cs=0 l=1 rev1=1 crypto rx=0x7fc5c006b8a0 tx=0x7fc5b800b330 comp rx=0 tx=0).stop 2023-10-16T13:49:06.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.620+0000 7fc5c4ce8700 1 -- 172.21.15.16:0/792879588 shutdown_connections 2023-10-16T13:49:06.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.620+0000 7fc5c4ce8700 1 --2- 172.21.15.16:0/792879588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5c006a7e0 0x7fc5c006ac30 unknown :-1 s=CLOSED pgs=3195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:06.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.620+0000 7fc5c4ce8700 1 --2- 172.21.15.16:0/792879588 >> v2:172.21.15.49:3300/0 conn(0x7fc5c015b280 0x7fc5c01613e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:06.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.620+0000 7fc5c4ce8700 1 --2- 172.21.15.16:0/792879588 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc5c00cc8a0 0x7fc5c015e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:06.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.620+0000 7fc5c4ce8700 1 -- 172.21.15.16:0/792879588 >> 172.21.15.16:0/792879588 conn(0x7fc5c00b1fd0 msgr2=0x7fc5c00b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:06.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.620+0000 7fc5c4ce8700 1 -- 172.21.15.16:0/792879588 shutdown_connections 2023-10-16T13:49:06.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.620+0000 7fc5c4ce8700 1 -- 172.21.15.16:0/792879588 wait complete. 2023-10-16T13:49:06.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.622+0000 7fc5c4ce8700 1 Processor -- start 2023-10-16T13:49:06.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.622+0000 7fc5c4ce8700 1 -- start start 2023-10-16T13:49:06.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.623+0000 7fc5c4ce8700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc5c006a7e0 0x7fc5c00ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:06.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.623+0000 7fc5c4ce8700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc5c00cc8a0 0x7fc5c01f2800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:06.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.623+0000 7fc5c4ce8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5c015b280 0x7fc5c00c3f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:06.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.623+0000 7fc5c4ce8700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c00c8750 con 0x7fc5c015b280 2023-10-16T13:49:06.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.623+0000 7fc5c4ce8700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c00c7ad0 con 0x7fc5c00cc8a0 2023-10-16T13:49:06.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.623+0000 7fc5c4ce8700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c00c7db0 con 0x7fc5c006a7e0 2023-10-16T13:49:06.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.623+0000 7fc5bed9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5c015b280 0x7fc5c00c3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:06.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.623+0000 7fc5bed9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5c015b280 0x7fc5c00c3f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59050/0 (socket says 172.21.15.16:59050) 2023-10-16T13:49:06.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.623+0000 7fc5bed9d700 1 -- 172.21.15.16:0/469279663 learned_addr learned my addr 172.21.15.16:0/469279663 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:06.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.623+0000 7fc5be59c700 1 --2- 172.21.15.16:0/469279663 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc5c006a7e0 0x7fc5c00ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:06.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.623+0000 7fc5bdd9b700 1 --2- 172.21.15.16:0/469279663 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc5c00cc8a0 0x7fc5c01f2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:06.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.623+0000 7fc5bed9d700 1 -- 172.21.15.16:0/469279663 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc5c006a7e0 msgr2=0x7fc5c00ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:06.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.623+0000 7fc5bed9d700 1 --2- 172.21.15.16:0/469279663 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc5c006a7e0 0x7fc5c00ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:06.634 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.623+0000 7fc5bed9d700 1 -- 172.21.15.16:0/469279663 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc5c00cc8a0 msgr2=0x7fc5c01f2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:06.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.623+0000 7fc5bed9d700 1 --2- 172.21.15.16:0/469279663 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc5c00cc8a0 0x7fc5c01f2800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:06.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.623+0000 7fc5bed9d700 1 -- 172.21.15.16:0/469279663 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5b8009b30 con 0x7fc5c015b280 2023-10-16T13:49:06.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.623+0000 7fc5bed9d700 1 --2- 172.21.15.16:0/469279663 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5c015b280 0x7fc5c00c3f50 secure :-1 s=READY pgs=3196 cs=0 l=1 rev1=1 crypto rx=0x7fc5b800b370 tx=0x7fc5b8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:06.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.624+0000 7fc5aeffd700 1 -- 172.21.15.16:0/469279663 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b8004250 con 0x7fc5c015b280 2023-10-16T13:49:06.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.624+0000 7fc5aeffd700 1 -- 172.21.15.16:0/469279663 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5b80043f0 con 0x7fc5c015b280 2023-10-16T13:49:06.635 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.624+0000 7fc5c4ce8700 1 -- 172.21.15.16:0/469279663 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5c00c8090 con 0x7fc5c015b280 2023-10-16T13:49:06.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.624+0000 7fc5aeffd700 1 -- 172.21.15.16:0/469279663 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b80047a0 con 0x7fc5c015b280 2023-10-16T13:49:06.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.624+0000 7fc5c4ce8700 1 -- 172.21.15.16:0/469279663 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5c01f2d40 con 0x7fc5c015b280 2023-10-16T13:49:06.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.625+0000 7fc5c4ce8700 1 -- 172.21.15.16:0/469279663 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5c00c8890 con 0x7fc5c015b280 2023-10-16T13:49:06.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.625+0000 7fc5aeffd700 1 -- 172.21.15.16:0/469279663 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 166) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc5b801c440 con 0x7fc5c015b280 2023-10-16T13:49:06.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.625+0000 7fc5aeffd700 1 --2- 172.21.15.16:0/469279663 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fc5a006b020 0x7fc5a006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:06.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.626+0000 7fc5aeffd700 1 -- 172.21.15.16:0/469279663 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(278..278 src has 1..278) v4 ==== 6157+0+0 (secure 0 0 0) 0x7fc5b808aed0 con 0x7fc5c015b280 2023-10-16T13:49:06.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.626+0000 7fc5be59c700 1 --2- 172.21.15.16:0/469279663 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fc5a006b020 0x7fc5a006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:06.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.626+0000 7fc5be59c700 1 --2- 172.21.15.16:0/469279663 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fc5a006b020 0x7fc5a006d4e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc5b0006fd0 tx=0x7fc5b0008040 comp rx=0 tx=0).ready entity=mgr.13344 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:06.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.630+0000 7fc5aeffd700 1 -- 172.21.15.16:0/469279663 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc5c00c8890 con 0x7fc5c015b280 2023-10-16T13:49:06.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.829+0000 7fc5c4ce8700 1 -- 172.21.15.16:0/469279663 --> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc5c01f2ed0 con 0x7fc5a006b020 2023-10-16T13:49:06.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.838+0000 7fc5aeffd700 1 -- 172.21.15.16:0/469279663 <== mgr.13344 v2:172.21.15.16:6832/1257392887 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289878 (secure 0 0 0) 0x7fc5c01f2ed0 con 0x7fc5a006b020 2023-10-16T13:49:06.844 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:49:06.846 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.841+0000 7fc5c4ce8700 1 -- 172.21.15.16:0/469279663 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fc5a006b020 msgr2=0x7fc5a006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:06.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.841+0000 7fc5c4ce8700 1 --2- 172.21.15.16:0/469279663 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fc5a006b020 0x7fc5a006d4e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc5b0006fd0 tx=0x7fc5b0008040 comp rx=0 tx=0).stop 2023-10-16T13:49:06.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.841+0000 7fc5c4ce8700 1 -- 172.21.15.16:0/469279663 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5c015b280 msgr2=0x7fc5c00c3f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:06.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.841+0000 7fc5c4ce8700 1 --2- 172.21.15.16:0/469279663 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5c015b280 0x7fc5c00c3f50 secure :-1 s=READY pgs=3196 cs=0 l=1 rev1=1 crypto rx=0x7fc5b800b370 tx=0x7fc5b8004ad0 comp rx=0 tx=0).stop 2023-10-16T13:49:06.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.841+0000 7fc5c4ce8700 1 -- 172.21.15.16:0/469279663 shutdown_connections 2023-10-16T13:49:06.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.841+0000 7fc5c4ce8700 1 --2- 172.21.15.16:0/469279663 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fc5a006b020 0x7fc5a006d4e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:06.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.841+0000 7fc5c4ce8700 1 --2- 172.21.15.16:0/469279663 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc5c015b280 0x7fc5c00c3f50 unknown :-1 s=CLOSED pgs=3196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:06.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.841+0000 7fc5c4ce8700 1 --2- 172.21.15.16:0/469279663 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc5c006a7e0 0x7fc5c00ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:06.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.841+0000 7fc5c4ce8700 1 --2- 172.21.15.16:0/469279663 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc5c00cc8a0 0x7fc5c01f2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:06.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.841+0000 7fc5c4ce8700 1 -- 172.21.15.16:0/469279663 >> 172.21.15.16:0/469279663 conn(0x7fc5c00b1fd0 msgr2=0x7fc5c0159b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:06.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.841+0000 7fc5c4ce8700 1 -- 172.21.15.16:0/469279663 shutdown_connections 2023-10-16T13:49:06.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:06.841+0000 7fc5c4ce8700 1 -- 172.21.15.16:0/469279663 wait complete. 2023-10-16T13:49:06.849 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-10-16T13:49:06.856 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2023-10-16T13:49:06.587872+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2612,"num_read_kb":5456,"num_write":723,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":441,"ondisk_log_size":441,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":97,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":457184,"kb_used_data":28640,"kb_used_omap":12,"kb_used_meta":428403,"kb_avail":754517536,"statfs":{"total":773094113280,"available":772625956864,"internally_reserved":0,"allocated":29327360,"data_stored":15759988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12740,"internal_metadata":438685244},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.917866"},"pg_stats":[{"pgid":"38.3f","version":"0'0","reported_seq":11,"reported_epoch":278,"state":"active+clean","last_fresh":"2023-10-16T13:49:03.368396+0000","last_change":"2023-10-16T13:49:02.376795+0000","last_active":"2023-10-16T13:49:03.368396+0000","last_peered":"2023-10-16T13:49:03.368396+0000","last_clean":"2023-10-16T13:49:03.368396+0000","last_became_active":"2023-10-16T13:49:02.376522+0000","last_became_peered":"2023-10-16T13:49:02.376522+0000","last_unstale":"2023-10-16T13:49:03.368396+0000","last_undegraded":"2023-10-16T13:49:03.368396+0000","last_fullsized":"2023-10-16T13:49:03.368396+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:50:05.069765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"38.3e","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.383138+0000","last_change":"2023-10-16T13:49:02.382027+0000","last_active":"2023-10-16T13:49:02.383138+0000","last_peered":"2023-10-16T13:49:02.383138+0000","last_clean":"2023-10-16T13:49:02.383138+0000","last_became_active":"2023-10-16T13:49:02.381334+0000","last_became_peered":"2023-10-16T13:49:02.381334+0000","last_unstale":"2023-10-16T13:49:02.383138+0000","last_undegraded":"2023-10-16T13:49:02.383138+0000","last_fullsized":"2023-10-16T13:49:02.383138+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:53:24.110836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.3d","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.381520+0000","last_change":"2023-10-16T13:49:02.380768+0000","last_active":"2023-10-16T13:49:02.381520+0000","last_peered":"2023-10-16T13:49:02.381520+0000","last_clean":"2023-10-16T13:49:02.381520+0000","last_became_active":"2023-10-16T13:49:02.380184+0000","last_became_peered":"2023-10-16T13:49:02.380184+0000","last_unstale":"2023-10-16T13:49:02.381520+0000","last_undegraded":"2023-10-16T13:49:02.381520+0000","last_fullsized":"2023-10-16T13:49:02.381520+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:10:29.903850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.3c","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.389667+0000","last_change":"2023-10-16T13:49:02.386716+0000","last_active":"2023-10-16T13:49:02.389667+0000","last_peered":"2023-10-16T13:49:02.389667+0000","last_clean":"2023-10-16T13:49:02.389667+0000","last_became_active":"2023-10-16T13:49:02.385114+0000","last_became_peered":"2023-10-16T13:49:02.385114+0000","last_unstale":"2023-10-16T13:49:02.389667+0000","last_undegraded":"2023-10-16T13:49:02.389667+0000","last_fullsized":"2023-10-16T13:49:02.389667+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:54:07.794615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.3b","version":"0'0","reported_seq":11,"reported_epoch":278,"state":"active+clean","last_fresh":"2023-10-16T13:49:03.369291+0000","last_change":"2023-10-16T13:49:02.383805+0000","last_active":"2023-10-16T13:49:03.369291+0000","last_peered":"2023-10-16T13:49:03.369291+0000","last_clean":"2023-10-16T13:49:03.369291+0000","last_became_active":"2023-10-16T13:49:02.383237+0000","last_became_peered":"2023-10-16T13:49:02.383237+0000","last_unstale":"2023-10-16T13:49:03.369291+0000","last_undegraded":"2023-10-16T13:49:03.369291+0000","last_fullsized":"2023-10-16T13:49:03.369291+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:05:38.479143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.3a","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.387093+0000","last_change":"2023-10-16T13:49:02.381275+0000","last_active":"2023-10-16T13:49:02.387093+0000","last_peered":"2023-10-16T13:49:02.387093+0000","last_clean":"2023-10-16T13:49:02.387093+0000","last_became_active":"2023-10-16T13:49:02.380948+0000","last_became_peered":"2023-10-16T13:49:02.380948+0000","last_unstale":"2023-10-16T13:49:02.387093+0000","last_undegraded":"2023-10-16T13:49:02.387093+0000","last_fullsized":"2023-10-16T13:49:02.387093+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:21:35.721646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.39","version":"0'0","reported_seq":11,"reported_epoch":278,"state":"active+clean","last_fresh":"2023-10-16T13:49:04.018552+0000","last_change":"2023-10-16T13:49:02.382220+0000","last_active":"2023-10-16T13:49:04.018552+0000","last_peered":"2023-10-16T13:49:04.018552+0000","last_clean":"2023-10-16T13:49:04.018552+0000","last_became_active":"2023-10-16T13:49:02.381865+0000","last_became_peered":"2023-10-16T13:49:02.381865+0000","last_unstale":"2023-10-16T13:49:04.018552+0000","last_undegraded":"2023-10-16T13:49:04.018552+0000","last_fullsized":"2023-10-16T13:49:04.018552+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:52:28.661340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"38.38","version":"0'0","reported_seq":11,"reported_epoch":278,"state":"active+clean","last_fresh":"2023-10-16T13:49:03.368029+0000","last_change":"2023-10-16T13:49:02.375427+0000","last_active":"2023-10-16T13:49:03.368029+0000","last_peered":"2023-10-16T13:49:03.368029+0000","last_clean":"2023-10-16T13:49:03.368029+0000","last_became_active":"2023-10-16T13:49:02.375149+0000","last_became_peered":"2023-10-16T13:49:02.375149+0000","last_unstale":"2023-10-16T13:49:03.368029+0000","last_undegraded":"2023-10-16T13:49:03.368029+0000","last_fullsized":"2023-10-16T13:49:03.368029+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:22:56.794768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"38.37","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.383095+0000","last_change":"2023-10-16T13:49:02.381659+0000","last_active":"2023-10-16T13:49:02.383095+0000","last_peered":"2023-10-16T13:49:02.383095+0000","last_clean":"2023-10-16T13:49:02.383095+0000","last_became_active":"2023-10-16T13:49:02.381334+0000","last_became_peered":"2023-10-16T13:49:02.381334+0000","last_unstale":"2023-10-16T13:49:02.383095+0000","last_undegraded":"2023-10-16T13:49:02.383095+0000","last_fullsized":"2023-10-16T13:49:02.383095+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:14:45.695415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.36","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.383518+0000","last_change":"2023-10-16T13:49:02.376686+0000","last_active":"2023-10-16T13:49:02.383518+0000","last_peered":"2023-10-16T13:49:02.383518+0000","last_clean":"2023-10-16T13:49:02.383518+0000","last_became_active":"2023-10-16T13:49:02.376378+0000","last_became_peered":"2023-10-16T13:49:02.376378+0000","last_unstale":"2023-10-16T13:49:02.383518+0000","last_undegraded":"2023-10-16T13:49:02.383518+0000","last_fullsized":"2023-10-16T13:49:02.383518+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:34:11.700814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.35","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.383714+0000","last_change":"2023-10-16T13:49:02.383342+0000","last_active":"2023-10-16T13:49:02.383714+0000","last_peered":"2023-10-16T13:49:02.383714+0000","last_clean":"2023-10-16T13:49:02.383714+0000","last_became_active":"2023-10-16T13:49:02.383013+0000","last_became_peered":"2023-10-16T13:49:02.383013+0000","last_unstale":"2023-10-16T13:49:02.383714+0000","last_undegraded":"2023-10-16T13:49:02.383714+0000","last_fullsized":"2023-10-16T13:49:02.383714+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:54:02.375814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.34","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.388185+0000","last_change":"2023-10-16T13:49:02.382481+0000","last_active":"2023-10-16T13:49:02.388185+0000","last_peered":"2023-10-16T13:49:02.388185+0000","last_clean":"2023-10-16T13:49:02.388185+0000","last_became_active":"2023-10-16T13:49:02.382177+0000","last_became_peered":"2023-10-16T13:49:02.382177+0000","last_unstale":"2023-10-16T13:49:02.388185+0000","last_undegraded":"2023-10-16T13:49:02.388185+0000","last_fullsized":"2023-10-16T13:49:02.388185+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:36:31.877550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.33","version":"0'0","reported_seq":11,"reported_epoch":278,"state":"active+clean","last_fresh":"2023-10-16T13:49:03.369263+0000","last_change":"2023-10-16T13:49:02.383950+0000","last_active":"2023-10-16T13:49:03.369263+0000","last_peered":"2023-10-16T13:49:03.369263+0000","last_clean":"2023-10-16T13:49:03.369263+0000","last_became_active":"2023-10-16T13:49:02.383639+0000","last_became_peered":"2023-10-16T13:49:02.383639+0000","last_unstale":"2023-10-16T13:49:03.369263+0000","last_undegraded":"2023-10-16T13:49:03.369263+0000","last_fullsized":"2023-10-16T13:49:03.369263+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:34:53.576041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.32","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.377946+0000","last_change":"2023-10-16T13:49:02.374842+0000","last_active":"2023-10-16T13:49:02.377946+0000","last_peered":"2023-10-16T13:49:02.377946+0000","last_clean":"2023-10-16T13:49:02.377946+0000","last_became_active":"2023-10-16T13:49:02.374581+0000","last_became_peered":"2023-10-16T13:49:02.374581+0000","last_unstale":"2023-10-16T13:49:02.377946+0000","last_undegraded":"2023-10-16T13:49:02.377946+0000","last_fullsized":"2023-10-16T13:49:02.377946+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:18:58.819910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.31","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.378454+0000","last_change":"2023-10-16T13:49:02.372490+0000","last_active":"2023-10-16T13:49:02.378454+0000","last_peered":"2023-10-16T13:49:02.378454+0000","last_clean":"2023-10-16T13:49:02.378454+0000","last_became_active":"2023-10-16T13:49:02.372207+0000","last_became_peered":"2023-10-16T13:49:02.372207+0000","last_unstale":"2023-10-16T13:49:02.378454+0000","last_undegraded":"2023-10-16T13:49:02.378454+0000","last_fullsized":"2023-10-16T13:49:02.378454+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:28:18.350036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.30","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.389880+0000","last_change":"2023-10-16T13:49:02.388914+0000","last_active":"2023-10-16T13:49:02.389880+0000","last_peered":"2023-10-16T13:49:02.389880+0000","last_clean":"2023-10-16T13:49:02.389880+0000","last_became_active":"2023-10-16T13:49:02.388639+0000","last_became_peered":"2023-10-16T13:49:02.388639+0000","last_unstale":"2023-10-16T13:49:02.389880+0000","last_undegraded":"2023-10-16T13:49:02.389880+0000","last_fullsized":"2023-10-16T13:49:02.389880+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:46:30.743146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.2f","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.383076+0000","last_change":"2023-10-16T13:49:02.381169+0000","last_active":"2023-10-16T13:49:02.383076+0000","last_peered":"2023-10-16T13:49:02.383076+0000","last_clean":"2023-10-16T13:49:02.383076+0000","last_became_active":"2023-10-16T13:49:02.380828+0000","last_became_peered":"2023-10-16T13:49:02.380828+0000","last_unstale":"2023-10-16T13:49:02.383076+0000","last_undegraded":"2023-10-16T13:49:02.383076+0000","last_fullsized":"2023-10-16T13:49:02.383076+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:28:05.517649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.2e","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.385074+0000","last_change":"2023-10-16T13:49:02.381165+0000","last_active":"2023-10-16T13:49:02.385074+0000","last_peered":"2023-10-16T13:49:02.385074+0000","last_clean":"2023-10-16T13:49:02.385074+0000","last_became_active":"2023-10-16T13:49:02.380863+0000","last_became_peered":"2023-10-16T13:49:02.380863+0000","last_unstale":"2023-10-16T13:49:02.385074+0000","last_undegraded":"2023-10-16T13:49:02.385074+0000","last_fullsized":"2023-10-16T13:49:02.385074+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:35:41.847423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.2d","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.381541+0000","last_change":"2023-10-16T13:49:02.380914+0000","last_active":"2023-10-16T13:49:02.381541+0000","last_peered":"2023-10-16T13:49:02.381541+0000","last_clean":"2023-10-16T13:49:02.381541+0000","last_became_active":"2023-10-16T13:49:02.380506+0000","last_became_peered":"2023-10-16T13:49:02.380506+0000","last_unstale":"2023-10-16T13:49:02.381541+0000","last_undegraded":"2023-10-16T13:49:02.381541+0000","last_fullsized":"2023-10-16T13:49:02.381541+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:22:59.695958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.2c","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.385420+0000","last_change":"2023-10-16T13:49:02.380258+0000","last_active":"2023-10-16T13:49:02.385420+0000","last_peered":"2023-10-16T13:49:02.385420+0000","last_clean":"2023-10-16T13:49:02.385420+0000","last_became_active":"2023-10-16T13:49:02.379919+0000","last_became_peered":"2023-10-16T13:49:02.379919+0000","last_unstale":"2023-10-16T13:49:02.385420+0000","last_undegraded":"2023-10-16T13:49:02.385420+0000","last_fullsized":"2023-10-16T13:49:02.385420+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:24:53.537429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.2b","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.386039+0000","last_change":"2023-10-16T13:49:02.385684+0000","last_active":"2023-10-16T13:49:02.386039+0000","last_peered":"2023-10-16T13:49:02.386039+0000","last_clean":"2023-10-16T13:49:02.386039+0000","last_became_active":"2023-10-16T13:49:02.385402+0000","last_became_peered":"2023-10-16T13:49:02.385402+0000","last_unstale":"2023-10-16T13:49:02.386039+0000","last_undegraded":"2023-10-16T13:49:02.386039+0000","last_fullsized":"2023-10-16T13:49:02.386039+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:16:22.980195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.2a","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.389617+0000","last_change":"2023-10-16T13:49:02.386598+0000","last_active":"2023-10-16T13:49:02.389617+0000","last_peered":"2023-10-16T13:49:02.389617+0000","last_clean":"2023-10-16T13:49:02.389617+0000","last_became_active":"2023-10-16T13:49:02.386284+0000","last_became_peered":"2023-10-16T13:49:02.386284+0000","last_unstale":"2023-10-16T13:49:02.389617+0000","last_undegraded":"2023-10-16T13:49:02.389617+0000","last_fullsized":"2023-10-16T13:49:02.389617+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:53:24.110836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.29","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.389972+0000","last_change":"2023-10-16T13:49:02.387731+0000","last_active":"2023-10-16T13:49:02.389972+0000","last_peered":"2023-10-16T13:49:02.389972+0000","last_clean":"2023-10-16T13:49:02.389972+0000","last_became_active":"2023-10-16T13:49:02.384828+0000","last_became_peered":"2023-10-16T13:49:02.384828+0000","last_unstale":"2023-10-16T13:49:02.389972+0000","last_undegraded":"2023-10-16T13:49:02.389972+0000","last_fullsized":"2023-10-16T13:49:02.389972+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:28:18.350036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.28","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.385335+0000","last_change":"2023-10-16T13:49:02.384443+0000","last_active":"2023-10-16T13:49:02.385335+0000","last_peered":"2023-10-16T13:49:02.385335+0000","last_clean":"2023-10-16T13:49:02.385335+0000","last_became_active":"2023-10-16T13:49:02.384174+0000","last_became_peered":"2023-10-16T13:49:02.384174+0000","last_unstale":"2023-10-16T13:49:02.385335+0000","last_undegraded":"2023-10-16T13:49:02.385335+0000","last_fullsized":"2023-10-16T13:49:02.385335+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:04:56.643213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.26","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.376229+0000","last_change":"2023-10-16T13:49:02.375235+0000","last_active":"2023-10-16T13:49:02.376229+0000","last_peered":"2023-10-16T13:49:02.376229+0000","last_clean":"2023-10-16T13:49:02.376229+0000","last_became_active":"2023-10-16T13:49:02.374449+0000","last_became_peered":"2023-10-16T13:49:02.374449+0000","last_unstale":"2023-10-16T13:49:02.376229+0000","last_undegraded":"2023-10-16T13:49:02.376229+0000","last_fullsized":"2023-10-16T13:49:02.376229+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:23:43.076430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.25","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.389622+0000","last_change":"2023-10-16T13:49:02.386164+0000","last_active":"2023-10-16T13:49:02.389622+0000","last_peered":"2023-10-16T13:49:02.389622+0000","last_clean":"2023-10-16T13:49:02.389622+0000","last_became_active":"2023-10-16T13:49:02.385195+0000","last_became_peered":"2023-10-16T13:49:02.385195+0000","last_unstale":"2023-10-16T13:49:02.389622+0000","last_undegraded":"2023-10-16T13:49:02.389622+0000","last_fullsized":"2023-10-16T13:49:02.389622+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:04:56.643213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.24","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.370820+0000","last_change":"2023-10-16T13:49:02.367826+0000","last_active":"2023-10-16T13:49:02.370820+0000","last_peered":"2023-10-16T13:49:02.370820+0000","last_clean":"2023-10-16T13:49:02.370820+0000","last_became_active":"2023-10-16T13:49:02.367551+0000","last_became_peered":"2023-10-16T13:49:02.367551+0000","last_unstale":"2023-10-16T13:49:02.370820+0000","last_undegraded":"2023-10-16T13:49:02.370820+0000","last_fullsized":"2023-10-16T13:49:02.370820+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:45:05.599847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.23","version":"0'0","reported_seq":11,"reported_epoch":278,"state":"active+clean","last_fresh":"2023-10-16T13:49:04.018862+0000","last_change":"2023-10-16T13:49:02.386716+0000","last_active":"2023-10-16T13:49:04.018862+0000","last_peered":"2023-10-16T13:49:04.018862+0000","last_clean":"2023-10-16T13:49:04.018862+0000","last_became_active":"2023-10-16T13:49:02.386441+0000","last_became_peered":"2023-10-16T13:49:02.386441+0000","last_unstale":"2023-10-16T13:49:04.018862+0000","last_undegraded":"2023-10-16T13:49:04.018862+0000","last_fullsized":"2023-10-16T13:49:04.018862+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:18:28.824765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"38.22","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.377926+0000","last_change":"2023-10-16T13:49:02.373505+0000","last_active":"2023-10-16T13:49:02.377926+0000","last_peered":"2023-10-16T13:49:02.377926+0000","last_clean":"2023-10-16T13:49:02.377926+0000","last_became_active":"2023-10-16T13:49:02.372543+0000","last_became_peered":"2023-10-16T13:49:02.372543+0000","last_unstale":"2023-10-16T13:49:02.377926+0000","last_undegraded":"2023-10-16T13:49:02.377926+0000","last_fullsized":"2023-10-16T13:49:02.377926+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:22:59.695958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.21","version":"0'0","reported_seq":11,"reported_epoch":278,"state":"active+clean","last_fresh":"2023-10-16T13:49:04.018646+0000","last_change":"2023-10-16T13:49:02.372720+0000","last_active":"2023-10-16T13:49:04.018646+0000","last_peered":"2023-10-16T13:49:04.018646+0000","last_clean":"2023-10-16T13:49:04.018646+0000","last_became_active":"2023-10-16T13:49:02.372410+0000","last_became_peered":"2023-10-16T13:49:02.372410+0000","last_unstale":"2023-10-16T13:49:04.018646+0000","last_undegraded":"2023-10-16T13:49:04.018646+0000","last_fullsized":"2023-10-16T13:49:04.018646+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:00:48.018874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"38.20","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.390003+0000","last_change":"2023-10-16T13:49:02.387734+0000","last_active":"2023-10-16T13:49:02.390003+0000","last_peered":"2023-10-16T13:49:02.390003+0000","last_clean":"2023-10-16T13:49:02.390003+0000","last_became_active":"2023-10-16T13:49:02.384712+0000","last_became_peered":"2023-10-16T13:49:02.384712+0000","last_unstale":"2023-10-16T13:49:02.390003+0000","last_undegraded":"2023-10-16T13:49:02.390003+0000","last_fullsized":"2023-10-16T13:49:02.390003+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:10:29.903850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.1c","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.385366+0000","last_change":"2023-10-16T13:49:02.380193+0000","last_active":"2023-10-16T13:49:02.385366+0000","last_peered":"2023-10-16T13:49:02.385366+0000","last_clean":"2023-10-16T13:49:02.385366+0000","last_became_active":"2023-10-16T13:49:02.379874+0000","last_became_peered":"2023-10-16T13:49:02.379874+0000","last_unstale":"2023-10-16T13:49:02.385366+0000","last_undegraded":"2023-10-16T13:49:02.385366+0000","last_fullsized":"2023-10-16T13:49:02.385366+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:07:27.569705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.1f","version":"277'4","reported_seq":16,"reported_epoch":278,"state":"active+clean","last_fresh":"2023-10-16T13:49:03.369129+0000","last_change":"2023-10-16T13:49:01.695903+0000","last_active":"2023-10-16T13:49:03.369129+0000","last_peered":"2023-10-16T13:49:03.369129+0000","last_clean":"2023-10-16T13:49:03.369129+0000","last_became_active":"2023-10-16T13:49:01.695597+0000","last_became_peered":"2023-10-16T13:49:01.695597+0000","last_unstale":"2023-10-16T13:49:03.369129+0000","last_undegraded":"2023-10-16T13:49:03.369129+0000","last_fullsized":"2023-10-16T13:49:03.369129+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_clean_scrub_stamp":"2023-10-16T13:49:00.664589+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:28:57.441027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.1d","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.383733+0000","last_change":"2023-10-16T13:49:02.383375+0000","last_active":"2023-10-16T13:49:02.383733+0000","last_peered":"2023-10-16T13:49:02.383733+0000","last_clean":"2023-10-16T13:49:02.383733+0000","last_became_active":"2023-10-16T13:49:02.383067+0000","last_became_peered":"2023-10-16T13:49:02.383067+0000","last_unstale":"2023-10-16T13:49:02.383733+0000","last_undegraded":"2023-10-16T13:49:02.383733+0000","last_fullsized":"2023-10-16T13:49:02.383733+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:45:39.960993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.1e","version":"277'1","reported_seq":12,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.702137+0000","last_change":"2023-10-16T13:49:01.693140+0000","last_active":"2023-10-16T13:49:02.702137+0000","last_peered":"2023-10-16T13:49:02.702137+0000","last_clean":"2023-10-16T13:49:02.702137+0000","last_became_active":"2023-10-16T13:49:01.692859+0000","last_became_peered":"2023-10-16T13:49:01.692859+0000","last_unstale":"2023-10-16T13:49:02.702137+0000","last_undegraded":"2023-10-16T13:49:02.702137+0000","last_fullsized":"2023-10-16T13:49:02.702137+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_clean_scrub_stamp":"2023-10-16T13:49:00.664589+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:41:52.860750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.1e","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.378807+0000","last_change":"2023-10-16T13:49:02.377260+0000","last_active":"2023-10-16T13:49:02.378807+0000","last_peered":"2023-10-16T13:49:02.378807+0000","last_clean":"2023-10-16T13:49:02.378807+0000","last_became_active":"2023-10-16T13:49:02.375414+0000","last_became_peered":"2023-10-16T13:49:02.375414+0000","last_unstale":"2023-10-16T13:49:02.378807+0000","last_undegraded":"2023-10-16T13:49:02.378807+0000","last_fullsized":"2023-10-16T13:49:02.378807+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:47:14.454827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.1d","version":"277'2","reported_seq":13,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.706232+0000","last_change":"2023-10-16T13:49:01.689030+0000","last_active":"2023-10-16T13:49:02.706232+0000","last_peered":"2023-10-16T13:49:02.706232+0000","last_clean":"2023-10-16T13:49:02.706232+0000","last_became_active":"2023-10-16T13:49:01.688698+0000","last_became_peered":"2023-10-16T13:49:01.688698+0000","last_unstale":"2023-10-16T13:49:02.706232+0000","last_undegraded":"2023-10-16T13:49:02.706232+0000","last_fullsized":"2023-10-16T13:49:02.706232+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_clean_scrub_stamp":"2023-10-16T13:49:00.664589+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:56:42.829574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.1f","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.372403+0000","last_change":"2023-10-16T13:49:02.370312+0000","last_active":"2023-10-16T13:49:02.372403+0000","last_peered":"2023-10-16T13:49:02.372403+0000","last_clean":"2023-10-16T13:49:02.372403+0000","last_became_active":"2023-10-16T13:49:02.370044+0000","last_became_peered":"2023-10-16T13:49:02.370044+0000","last_unstale":"2023-10-16T13:49:02.372403+0000","last_undegraded":"2023-10-16T13:49:02.372403+0000","last_fullsized":"2023-10-16T13:49:02.372403+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:35:41.847423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.1c","version":"277'2","reported_seq":13,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.707775+0000","last_change":"2023-10-16T13:49:01.689254+0000","last_active":"2023-10-16T13:49:02.707775+0000","last_peered":"2023-10-16T13:49:02.707775+0000","last_clean":"2023-10-16T13:49:02.707775+0000","last_became_active":"2023-10-16T13:49:01.688292+0000","last_became_peered":"2023-10-16T13:49:01.688292+0000","last_unstale":"2023-10-16T13:49:02.707775+0000","last_undegraded":"2023-10-16T13:49:02.707775+0000","last_fullsized":"2023-10-16T13:49:02.707775+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_clean_scrub_stamp":"2023-10-16T13:49:00.664589+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:52:52.702555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.18","version":"0'0","reported_seq":11,"reported_epoch":278,"state":"active+clean","last_fresh":"2023-10-16T13:49:03.368088+0000","last_change":"2023-10-16T13:49:02.377042+0000","last_active":"2023-10-16T13:49:03.368088+0000","last_peered":"2023-10-16T13:49:03.368088+0000","last_clean":"2023-10-16T13:49:03.368088+0000","last_became_active":"2023-10-16T13:49:02.376705+0000","last_became_peered":"2023-10-16T13:49:02.376705+0000","last_unstale":"2023-10-16T13:49:03.368088+0000","last_undegraded":"2023-10-16T13:49:03.368088+0000","last_fullsized":"2023-10-16T13:49:03.368088+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:22:59.348010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.1b","version":"277'1","reported_seq":12,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.704320+0000","last_change":"2023-10-16T13:49:01.686024+0000","last_active":"2023-10-16T13:49:02.704320+0000","last_peered":"2023-10-16T13:49:02.704320+0000","last_clean":"2023-10-16T13:49:02.704320+0000","last_became_active":"2023-10-16T13:49:01.679420+0000","last_became_peered":"2023-10-16T13:49:01.679420+0000","last_unstale":"2023-10-16T13:49:02.704320+0000","last_undegraded":"2023-10-16T13:49:02.704320+0000","last_fullsized":"2023-10-16T13:49:02.704320+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_clean_scrub_stamp":"2023-10-16T13:49:00.664589+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:00:07.114500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.19","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.379600+0000","last_change":"2023-10-16T13:49:02.378876+0000","last_active":"2023-10-16T13:49:02.379600+0000","last_peered":"2023-10-16T13:49:02.379600+0000","last_clean":"2023-10-16T13:49:02.379600+0000","last_became_active":"2023-10-16T13:49:02.378413+0000","last_became_peered":"2023-10-16T13:49:02.378413+0000","last_unstale":"2023-10-16T13:49:02.379600+0000","last_undegraded":"2023-10-16T13:49:02.379600+0000","last_fullsized":"2023-10-16T13:49:02.379600+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:50:22.311928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.1a","version":"277'1","reported_seq":12,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.707379+0000","last_change":"2023-10-16T13:49:01.691725+0000","last_active":"2023-10-16T13:49:02.707379+0000","last_peered":"2023-10-16T13:49:02.707379+0000","last_clean":"2023-10-16T13:49:02.707379+0000","last_became_active":"2023-10-16T13:49:01.691431+0000","last_became_peered":"2023-10-16T13:49:01.691431+0000","last_unstale":"2023-10-16T13:49:02.707379+0000","last_undegraded":"2023-10-16T13:49:02.707379+0000","last_fullsized":"2023-10-16T13:49:02.707379+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_clean_scrub_stamp":"2023-10-16T13:49:00.664589+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:53:23.105788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.1a","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.389592+0000","last_change":"2023-10-16T13:49:02.386330+0000","last_active":"2023-10-16T13:49:02.389592+0000","last_peered":"2023-10-16T13:49:02.389592+0000","last_clean":"2023-10-16T13:49:02.389592+0000","last_became_active":"2023-10-16T13:49:02.385896+0000","last_became_peered":"2023-10-16T13:49:02.385896+0000","last_unstale":"2023-10-16T13:49:02.389592+0000","last_undegraded":"2023-10-16T13:49:02.389592+0000","last_fullsized":"2023-10-16T13:49:02.389592+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:29:32.248558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.19","version":"0'0","reported_seq":11,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.369623+0000","last_change":"2023-10-16T13:49:01.692662+0000","last_active":"2023-10-16T13:49:02.369623+0000","last_peered":"2023-10-16T13:49:02.369623+0000","last_clean":"2023-10-16T13:49:02.369623+0000","last_became_active":"2023-10-16T13:49:01.691965+0000","last_became_peered":"2023-10-16T13:49:01.691965+0000","last_unstale":"2023-10-16T13:49:02.369623+0000","last_undegraded":"2023-10-16T13:49:02.369623+0000","last_fullsized":"2023-10-16T13:49:02.369623+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_clean_scrub_stamp":"2023-10-16T13:49:00.664589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:48:17.860543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.1b","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.383124+0000","last_change":"2023-10-16T13:49:02.382596+0000","last_active":"2023-10-16T13:49:02.383124+0000","last_peered":"2023-10-16T13:49:02.383124+0000","last_clean":"2023-10-16T13:49:02.383124+0000","last_became_active":"2023-10-16T13:49:02.382300+0000","last_became_peered":"2023-10-16T13:49:02.382300+0000","last_unstale":"2023-10-16T13:49:02.383124+0000","last_undegraded":"2023-10-16T13:49:02.383124+0000","last_fullsized":"2023-10-16T13:49:02.383124+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:29:26.019903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.18","version":"0'0","reported_seq":12,"reported_epoch":278,"state":"active+clean","last_fresh":"2023-10-16T13:49:04.018470+0000","last_change":"2023-10-16T13:49:01.693611+0000","last_active":"2023-10-16T13:49:04.018470+0000","last_peered":"2023-10-16T13:49:04.018470+0000","last_clean":"2023-10-16T13:49:04.018470+0000","last_became_active":"2023-10-16T13:49:01.693015+0000","last_became_peered":"2023-10-16T13:49:01.693015+0000","last_unstale":"2023-10-16T13:49:04.018470+0000","last_undegraded":"2023-10-16T13:49:04.018470+0000","last_fullsized":"2023-10-16T13:49:04.018470+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_clean_scrub_stamp":"2023-10-16T13:49:00.664589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:33:09.963008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"38.14","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.385345+0000","last_change":"2023-10-16T13:49:02.378931+0000","last_active":"2023-10-16T13:49:02.385345+0000","last_peered":"2023-10-16T13:49:02.385345+0000","last_clean":"2023-10-16T13:49:02.385345+0000","last_became_active":"2023-10-16T13:49:02.378025+0000","last_became_peered":"2023-10-16T13:49:02.378025+0000","last_unstale":"2023-10-16T13:49:02.385345+0000","last_undegraded":"2023-10-16T13:49:02.385345+0000","last_fullsized":"2023-10-16T13:49:02.385345+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:17:26.735536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.17","version":"0'0","reported_seq":12,"reported_epoch":278,"state":"active+clean","last_fresh":"2023-10-16T13:49:03.369239+0000","last_change":"2023-10-16T13:49:01.688691+0000","last_active":"2023-10-16T13:49:03.369239+0000","last_peered":"2023-10-16T13:49:03.369239+0000","last_clean":"2023-10-16T13:49:03.369239+0000","last_became_active":"2023-10-16T13:49:01.688376+0000","last_became_peered":"2023-10-16T13:49:01.688376+0000","last_unstale":"2023-10-16T13:49:03.369239+0000","last_undegraded":"2023-10-16T13:49:03.369239+0000","last_fullsized":"2023-10-16T13:49:03.369239+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_clean_scrub_stamp":"2023-10-16T13:49:00.664589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:06:05.911525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.15","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.384081+0000","last_change":"2023-10-16T13:49:02.379036+0000","last_active":"2023-10-16T13:49:02.384081+0000","last_peered":"2023-10-16T13:49:02.384081+0000","last_clean":"2023-10-16T13:49:02.384081+0000","last_became_active":"2023-10-16T13:49:02.378723+0000","last_became_peered":"2023-10-16T13:49:02.378723+0000","last_unstale":"2023-10-16T13:49:02.384081+0000","last_undegraded":"2023-10-16T13:49:02.384081+0000","last_fullsized":"2023-10-16T13:49:02.384081+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:11:46.354324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.16","version":"0'0","reported_seq":11,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.369750+0000","last_change":"2023-10-16T13:49:01.692624+0000","last_active":"2023-10-16T13:49:02.369750+0000","last_peered":"2023-10-16T13:49:02.369750+0000","last_clean":"2023-10-16T13:49:02.369750+0000","last_became_active":"2023-10-16T13:49:01.692325+0000","last_became_peered":"2023-10-16T13:49:01.692325+0000","last_unstale":"2023-10-16T13:49:02.369750+0000","last_undegraded":"2023-10-16T13:49:02.369750+0000","last_fullsized":"2023-10-16T13:49:02.369750+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_clean_scrub_stamp":"2023-10-16T13:49:00.664589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:16:51.291591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.16","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.378788+0000","last_change":"2023-10-16T13:49:02.375471+0000","last_active":"2023-10-16T13:49:02.378788+0000","last_peered":"2023-10-16T13:49:02.378788+0000","last_clean":"2023-10-16T13:49:02.378788+0000","last_became_active":"2023-10-16T13:49:02.375146+0000","last_became_peered":"2023-10-16T13:49:02.375146+0000","last_unstale":"2023-10-16T13:49:02.378788+0000","last_undegraded":"2023-10-16T13:49:02.378788+0000","last_fullsized":"2023-10-16T13:49:02.378788+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:56:43.834622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.15","version":"277'3","reported_seq":14,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.704969+0000","last_change":"2023-10-16T13:49:01.691397+0000","last_active":"2023-10-16T13:49:02.704969+0000","last_peered":"2023-10-16T13:49:02.704969+0000","last_clean":"2023-10-16T13:49:02.704969+0000","last_became_active":"2023-10-16T13:49:01.691101+0000","last_became_peered":"2023-10-16T13:49:01.691101+0000","last_unstale":"2023-10-16T13:49:02.704969+0000","last_undegraded":"2023-10-16T13:49:02.704969+0000","last_fullsized":"2023-10-16T13:49:02.704969+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_clean_scrub_stamp":"2023-10-16T13:49:00.664589+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:33:58.144888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.17","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.384927+0000","last_change":"2023-10-16T13:49:02.380011+0000","last_active":"2023-10-16T13:49:02.384927+0000","last_peered":"2023-10-16T13:49:02.384927+0000","last_clean":"2023-10-16T13:49:02.384927+0000","last_became_active":"2023-10-16T13:49:02.379696+0000","last_became_peered":"2023-10-16T13:49:02.379696+0000","last_unstale":"2023-10-16T13:49:02.384927+0000","last_undegraded":"2023-10-16T13:49:02.384927+0000","last_fullsized":"2023-10-16T13:49:02.384927+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:44:52.683527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.14","version":"277'6","reported_seq":18,"reported_epoch":278,"state":"active+clean","last_fresh":"2023-10-16T13:49:03.368319+0000","last_change":"2023-10-16T13:49:01.690187+0000","last_active":"2023-10-16T13:49:03.368319+0000","last_peered":"2023-10-16T13:49:03.368319+0000","last_clean":"2023-10-16T13:49:03.368319+0000","last_became_active":"2023-10-16T13:49:01.689795+0000","last_became_peered":"2023-10-16T13:49:01.689795+0000","last_unstale":"2023-10-16T13:49:03.368319+0000","last_undegraded":"2023-10-16T13:49:03.368319+0000","last_fullsized":"2023-10-16T13:49:03.368319+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_clean_scrub_stamp":"2023-10-16T13:49:00.664589+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:38:55.796167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"38.10","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.379048+0000","last_change":"2023-10-16T13:49:02.376913+0000","last_active":"2023-10-16T13:49:02.379048+0000","last_peered":"2023-10-16T13:49:02.379048+0000","last_clean":"2023-10-16T13:49:02.379048+0000","last_became_active":"2023-10-16T13:49:02.376017+0000","last_became_peered":"2023-10-16T13:49:02.376017+0000","last_unstale":"2023-10-16T13:49:02.379048+0000","last_undegraded":"2023-10-16T13:49:02.379048+0000","last_fullsized":"2023-10-16T13:49:02.379048+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:52:47.764832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.13","version":"277'2","reported_seq":13,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.706684+0000","last_change":"2023-10-16T13:49:01.687745+0000","last_active":"2023-10-16T13:49:02.706684+0000","last_peered":"2023-10-16T13:49:02.706684+0000","last_clean":"2023-10-16T13:49:02.706684+0000","last_became_active":"2023-10-16T13:49:01.687411+0000","last_became_peered":"2023-10-16T13:49:01.687411+0000","last_unstale":"2023-10-16T13:49:02.706684+0000","last_undegraded":"2023-10-16T13:49:02.706684+0000","last_fullsized":"2023-10-16T13:49:02.706684+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_clean_scrub_stamp":"2023-10-16T13:49:00.664589+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:12:52.526868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.11","version":"0'0","reported_seq":11,"reported_epoch":278,"state":"active+clean","last_fresh":"2023-10-16T13:49:03.369397+0000","last_change":"2023-10-16T13:49:02.384951+0000","last_active":"2023-10-16T13:49:03.369397+0000","last_peered":"2023-10-16T13:49:03.369397+0000","last_clean":"2023-10-16T13:49:03.369397+0000","last_became_active":"2023-10-16T13:49:02.384592+0000","last_became_peered":"2023-10-16T13:49:02.384592+0000","last_unstale":"2023-10-16T13:49:03.369397+0000","last_undegraded":"2023-10-16T13:49:03.369397+0000","last_fullsized":"2023-10-16T13:49:03.369397+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:18:18.047752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.12","version":"0'0","reported_seq":12,"reported_epoch":278,"state":"active+clean","last_fresh":"2023-10-16T13:49:03.368978+0000","last_change":"2023-10-16T13:49:01.696834+0000","last_active":"2023-10-16T13:49:03.368978+0000","last_peered":"2023-10-16T13:49:03.368978+0000","last_clean":"2023-10-16T13:49:03.368978+0000","last_became_active":"2023-10-16T13:49:01.696555+0000","last_became_peered":"2023-10-16T13:49:01.696555+0000","last_unstale":"2023-10-16T13:49:03.368978+0000","last_undegraded":"2023-10-16T13:49:03.368978+0000","last_fullsized":"2023-10-16T13:49:03.368978+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_clean_scrub_stamp":"2023-10-16T13:49:00.664589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:29:40.250736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.12","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.386489+0000","last_change":"2023-10-16T13:49:02.381459+0000","last_active":"2023-10-16T13:49:02.386489+0000","last_peered":"2023-10-16T13:49:02.386489+0000","last_clean":"2023-10-16T13:49:02.386489+0000","last_became_active":"2023-10-16T13:49:02.381184+0000","last_became_peered":"2023-10-16T13:49:02.381184+0000","last_unstale":"2023-10-16T13:49:02.386489+0000","last_undegraded":"2023-10-16T13:49:02.386489+0000","last_fullsized":"2023-10-16T13:49:02.386489+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:22:59.695958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.11","version":"0'0","reported_seq":11,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.368180+0000","last_change":"2023-10-16T13:49:01.686038+0000","last_active":"2023-10-16T13:49:02.368180+0000","last_peered":"2023-10-16T13:49:02.368180+0000","last_clean":"2023-10-16T13:49:02.368180+0000","last_became_active":"2023-10-16T13:49:01.685753+0000","last_became_peered":"2023-10-16T13:49:01.685753+0000","last_unstale":"2023-10-16T13:49:02.368180+0000","last_undegraded":"2023-10-16T13:49:02.368180+0000","last_fullsized":"2023-10-16T13:49:02.368180+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_clean_scrub_stamp":"2023-10-16T13:49:00.664589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:11:41.402056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.13","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.388078+0000","last_change":"2023-10-16T13:49:02.382861+0000","last_active":"2023-10-16T13:49:02.388078+0000","last_peered":"2023-10-16T13:49:02.388078+0000","last_clean":"2023-10-16T13:49:02.388078+0000","last_became_active":"2023-10-16T13:49:02.382543+0000","last_became_peered":"2023-10-16T13:49:02.382543+0000","last_unstale":"2023-10-16T13:49:02.388078+0000","last_undegraded":"2023-10-16T13:49:02.388078+0000","last_fullsized":"2023-10-16T13:49:02.388078+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:15:09.398807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.10","version":"277'8","reported_seq":19,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.710134+0000","last_change":"2023-10-16T13:49:01.693876+0000","last_active":"2023-10-16T13:49:02.710134+0000","last_peered":"2023-10-16T13:49:02.710134+0000","last_clean":"2023-10-16T13:49:02.710134+0000","last_became_active":"2023-10-16T13:49:01.693605+0000","last_became_peered":"2023-10-16T13:49:01.693605+0000","last_unstale":"2023-10-16T13:49:02.710134+0000","last_undegraded":"2023-10-16T13:49:02.710134+0000","last_fullsized":"2023-10-16T13:49:02.710134+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_clean_scrub_stamp":"2023-10-16T13:49:00.664589+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:56:42.829574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.c","version":"0'0","reported_seq":11,"reported_epoch":278,"state":"active+clean","last_fresh":"2023-10-16T13:49:03.368384+0000","last_change":"2023-10-16T13:49:02.370520+0000","last_active":"2023-10-16T13:49:03.368384+0000","last_peered":"2023-10-16T13:49:03.368384+0000","last_clean":"2023-10-16T13:49:03.368384+0000","last_became_active":"2023-10-16T13:49:02.370062+0000","last_became_peered":"2023-10-16T13:49:02.370062+0000","last_unstale":"2023-10-16T13:49:03.368384+0000","last_undegraded":"2023-10-16T13:49:03.368384+0000","last_fullsized":"2023-10-16T13:49:03.368384+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:25:26.350475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.f","version":"277'1","reported_seq":12,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.703708+0000","last_change":"2023-10-16T13:49:01.695983+0000","last_active":"2023-10-16T13:49:02.703708+0000","last_peered":"2023-10-16T13:49:02.703708+0000","last_clean":"2023-10-16T13:49:02.703708+0000","last_became_active":"2023-10-16T13:49:01.695672+0000","last_became_peered":"2023-10-16T13:49:01.695672+0000","last_unstale":"2023-10-16T13:49:02.703708+0000","last_undegraded":"2023-10-16T13:49:02.703708+0000","last_fullsized":"2023-10-16T13:49:02.703708+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_clean_scrub_stamp":"2023-10-16T13:49:00.664589+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:45:09.678272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.d","version":"0'0","reported_seq":11,"reported_epoch":278,"state":"active+clean","last_fresh":"2023-10-16T13:49:03.369086+0000","last_change":"2023-10-16T13:49:02.378377+0000","last_active":"2023-10-16T13:49:03.369086+0000","last_peered":"2023-10-16T13:49:03.369086+0000","last_clean":"2023-10-16T13:49:03.369086+0000","last_became_active":"2023-10-16T13:49:02.378068+0000","last_became_peered":"2023-10-16T13:49:02.378068+0000","last_unstale":"2023-10-16T13:49:03.369086+0000","last_undegraded":"2023-10-16T13:49:03.369086+0000","last_fullsized":"2023-10-16T13:49:03.369086+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:43:26.335779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.e","version":"277'2","reported_seq":14,"reported_epoch":278,"state":"active+clean","last_fresh":"2023-10-16T13:49:03.368434+0000","last_change":"2023-10-16T13:49:01.695615+0000","last_active":"2023-10-16T13:49:03.368434+0000","last_peered":"2023-10-16T13:49:03.368434+0000","last_clean":"2023-10-16T13:49:03.368434+0000","last_became_active":"2023-10-16T13:49:01.695339+0000","last_became_peered":"2023-10-16T13:49:01.695339+0000","last_unstale":"2023-10-16T13:49:03.368434+0000","last_undegraded":"2023-10-16T13:49:03.368434+0000","last_fullsized":"2023-10-16T13:49:03.368434+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_clean_scrub_stamp":"2023-10-16T13:49:00.664589+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:13:22.191652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"38.e","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.387984+0000","last_change":"2023-10-16T13:49:02.381744+0000","last_active":"2023-10-16T13:49:02.387984+0000","last_peered":"2023-10-16T13:49:02.387984+0000","last_clean":"2023-10-16T13:49:02.387984+0000","last_became_active":"2023-10-16T13:49:02.381458+0000","last_became_peered":"2023-10-16T13:49:02.381458+0000","last_unstale":"2023-10-16T13:49:02.387984+0000","last_undegraded":"2023-10-16T13:49:02.387984+0000","last_fullsized":"2023-10-16T13:49:02.387984+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:31:19.406244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.d","version":"277'1","reported_seq":13,"reported_epoch":278,"state":"active+clean","last_fresh":"2023-10-16T13:49:03.369315+0000","last_change":"2023-10-16T13:49:01.687946+0000","last_active":"2023-10-16T13:49:03.369315+0000","last_peered":"2023-10-16T13:49:03.369315+0000","last_clean":"2023-10-16T13:49:03.369315+0000","last_became_active":"2023-10-16T13:49:01.687301+0000","last_became_peered":"2023-10-16T13:49:01.687301+0000","last_unstale":"2023-10-16T13:49:03.369315+0000","last_undegraded":"2023-10-16T13:49:03.369315+0000","last_fullsized":"2023-10-16T13:49:03.369315+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_clean_scrub_stamp":"2023-10-16T13:49:00.664589+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:45:32.313189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.f","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.385897+0000","last_change":"2023-10-16T13:49:02.385029+0000","last_active":"2023-10-16T13:49:02.385897+0000","last_peered":"2023-10-16T13:49:02.385897+0000","last_clean":"2023-10-16T13:49:02.385897+0000","last_became_active":"2023-10-16T13:49:02.384684+0000","last_became_peered":"2023-10-16T13:49:02.384684+0000","last_unstale":"2023-10-16T13:49:02.385897+0000","last_undegraded":"2023-10-16T13:49:02.385897+0000","last_fullsized":"2023-10-16T13:49:02.385897+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:17:26.735536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.c","version":"0'0","reported_seq":11,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.368494+0000","last_change":"2023-10-16T13:49:01.693806+0000","last_active":"2023-10-16T13:49:02.368494+0000","last_peered":"2023-10-16T13:49:02.368494+0000","last_clean":"2023-10-16T13:49:02.368494+0000","last_became_active":"2023-10-16T13:49:01.693523+0000","last_became_peered":"2023-10-16T13:49:01.693523+0000","last_unstale":"2023-10-16T13:49:02.368494+0000","last_undegraded":"2023-10-16T13:49:02.368494+0000","last_fullsized":"2023-10-16T13:49:02.368494+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_clean_scrub_stamp":"2023-10-16T13:49:00.664589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:41:52.860750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.8","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.388241+0000","last_change":"2023-10-16T13:49:02.383372+0000","last_active":"2023-10-16T13:49:02.388241+0000","last_peered":"2023-10-16T13:49:02.388241+0000","last_clean":"2023-10-16T13:49:02.388241+0000","last_became_active":"2023-10-16T13:49:02.383040+0000","last_became_peered":"2023-10-16T13:49:02.383040+0000","last_unstale":"2023-10-16T13:49:02.388241+0000","last_undegraded":"2023-10-16T13:49:02.388241+0000","last_fullsized":"2023-10-16T13:49:02.388241+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:26:39.098172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.b","version":"277'2","reported_seq":13,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.706559+0000","last_change":"2023-10-16T13:49:01.689899+0000","last_active":"2023-10-16T13:49:02.706559+0000","last_peered":"2023-10-16T13:49:02.706559+0000","last_clean":"2023-10-16T13:49:02.706559+0000","last_became_active":"2023-10-16T13:49:01.689621+0000","last_became_peered":"2023-10-16T13:49:01.689621+0000","last_unstale":"2023-10-16T13:49:02.706559+0000","last_undegraded":"2023-10-16T13:49:02.706559+0000","last_fullsized":"2023-10-16T13:49:02.706559+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_clean_scrub_stamp":"2023-10-16T13:49:00.664589+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:53:02.858944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.9","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.379500+0000","last_change":"2023-10-16T13:49:02.378684+0000","last_active":"2023-10-16T13:49:02.379500+0000","last_peered":"2023-10-16T13:49:02.379500+0000","last_clean":"2023-10-16T13:49:02.379500+0000","last_became_active":"2023-10-16T13:49:02.378083+0000","last_became_peered":"2023-10-16T13:49:02.378083+0000","last_unstale":"2023-10-16T13:49:02.379500+0000","last_undegraded":"2023-10-16T13:49:02.379500+0000","last_fullsized":"2023-10-16T13:49:02.379500+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:14:32.098204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.a","version":"0'0","reported_seq":11,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.370777+0000","last_change":"2023-10-16T13:49:01.695913+0000","last_active":"2023-10-16T13:49:02.370777+0000","last_peered":"2023-10-16T13:49:02.370777+0000","last_clean":"2023-10-16T13:49:02.370777+0000","last_became_active":"2023-10-16T13:49:01.695587+0000","last_became_peered":"2023-10-16T13:49:01.695587+0000","last_unstale":"2023-10-16T13:49:02.370777+0000","last_undegraded":"2023-10-16T13:49:02.370777+0000","last_fullsized":"2023-10-16T13:49:02.370777+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_clean_scrub_stamp":"2023-10-16T13:49:00.664589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:33:58.144888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.a","version":"0'0","reported_seq":11,"reported_epoch":278,"state":"active+clean","last_fresh":"2023-10-16T13:49:03.369182+0000","last_change":"2023-10-16T13:49:02.380331+0000","last_active":"2023-10-16T13:49:03.369182+0000","last_peered":"2023-10-16T13:49:03.369182+0000","last_clean":"2023-10-16T13:49:03.369182+0000","last_became_active":"2023-10-16T13:49:02.379419+0000","last_became_peered":"2023-10-16T13:49:02.379419+0000","last_unstale":"2023-10-16T13:49:03.369182+0000","last_undegraded":"2023-10-16T13:49:03.369182+0000","last_fullsized":"2023-10-16T13:49:03.369182+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:17:45.230832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.9","version":"0'0","reported_seq":11,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.371535+0000","last_change":"2023-10-16T13:49:01.689810+0000","last_active":"2023-10-16T13:49:02.371535+0000","last_peered":"2023-10-16T13:49:02.371535+0000","last_clean":"2023-10-16T13:49:02.371535+0000","last_became_active":"2023-10-16T13:49:01.689526+0000","last_became_peered":"2023-10-16T13:49:01.689526+0000","last_unstale":"2023-10-16T13:49:02.371535+0000","last_undegraded":"2023-10-16T13:49:02.371535+0000","last_fullsized":"2023-10-16T13:49:02.371535+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_clean_scrub_stamp":"2023-10-16T13:49:00.664589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:33:58.144888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.b","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.385993+0000","last_change":"2023-10-16T13:49:02.380114+0000","last_active":"2023-10-16T13:49:02.385993+0000","last_peered":"2023-10-16T13:49:02.385993+0000","last_clean":"2023-10-16T13:49:02.385993+0000","last_became_active":"2023-10-16T13:49:02.379822+0000","last_became_peered":"2023-10-16T13:49:02.379822+0000","last_unstale":"2023-10-16T13:49:02.385993+0000","last_undegraded":"2023-10-16T13:49:02.385993+0000","last_fullsized":"2023-10-16T13:49:02.385993+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:07:04.722402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.8","version":"0'0","reported_seq":11,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.370965+0000","last_change":"2023-10-16T13:49:01.690586+0000","last_active":"2023-10-16T13:49:02.370965+0000","last_peered":"2023-10-16T13:49:02.370965+0000","last_clean":"2023-10-16T13:49:02.370965+0000","last_became_active":"2023-10-16T13:49:01.690296+0000","last_became_peered":"2023-10-16T13:49:01.690296+0000","last_unstale":"2023-10-16T13:49:02.370965+0000","last_undegraded":"2023-10-16T13:49:02.370965+0000","last_fullsized":"2023-10-16T13:49:02.370965+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_clean_scrub_stamp":"2023-10-16T13:49:00.664589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:07:50.415719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.4","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.388329+0000","last_change":"2023-10-16T13:49:02.384189+0000","last_active":"2023-10-16T13:49:02.388329+0000","last_peered":"2023-10-16T13:49:02.388329+0000","last_clean":"2023-10-16T13:49:02.388329+0000","last_became_active":"2023-10-16T13:49:02.383881+0000","last_became_peered":"2023-10-16T13:49:02.383881+0000","last_unstale":"2023-10-16T13:49:02.388329+0000","last_undegraded":"2023-10-16T13:49:02.388329+0000","last_fullsized":"2023-10-16T13:49:02.388329+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:07:51.420767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.7","version":"277'2","reported_seq":14,"reported_epoch":278,"state":"active+clean","last_fresh":"2023-10-16T13:49:04.018971+0000","last_change":"2023-10-16T13:49:01.697143+0000","last_active":"2023-10-16T13:49:04.018971+0000","last_peered":"2023-10-16T13:49:04.018971+0000","last_clean":"2023-10-16T13:49:04.018971+0000","last_became_active":"2023-10-16T13:49:01.696560+0000","last_became_peered":"2023-10-16T13:49:01.696560+0000","last_unstale":"2023-10-16T13:49:04.018971+0000","last_undegraded":"2023-10-16T13:49:04.018971+0000","last_fullsized":"2023-10-16T13:49:04.018971+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_clean_scrub_stamp":"2023-10-16T13:49:00.664589+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:28:32.125622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"38.5","version":"0'0","reported_seq":11,"reported_epoch":278,"state":"active+clean","last_fresh":"2023-10-16T13:49:04.018948+0000","last_change":"2023-10-16T13:49:02.389063+0000","last_active":"2023-10-16T13:49:04.018948+0000","last_peered":"2023-10-16T13:49:04.018948+0000","last_clean":"2023-10-16T13:49:04.018948+0000","last_became_active":"2023-10-16T13:49:02.388213+0000","last_became_peered":"2023-10-16T13:49:02.388213+0000","last_unstale":"2023-10-16T13:49:04.018948+0000","last_undegraded":"2023-10-16T13:49:04.018948+0000","last_fullsized":"2023-10-16T13:49:04.018948+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:45:38.625175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.6","version":"0'0","reported_seq":11,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.366151+0000","last_change":"2023-10-16T13:49:01.690867+0000","last_active":"2023-10-16T13:49:02.366151+0000","last_peered":"2023-10-16T13:49:02.366151+0000","last_clean":"2023-10-16T13:49:02.366151+0000","last_became_active":"2023-10-16T13:49:01.690585+0000","last_became_peered":"2023-10-16T13:49:01.690585+0000","last_unstale":"2023-10-16T13:49:02.366151+0000","last_undegraded":"2023-10-16T13:49:02.366151+0000","last_fullsized":"2023-10-16T13:49:02.366151+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_clean_scrub_stamp":"2023-10-16T13:49:00.664589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:52:08.521711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.6","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.383158+0000","last_change":"2023-10-16T13:49:02.382355+0000","last_active":"2023-10-16T13:49:02.383158+0000","last_peered":"2023-10-16T13:49:02.383158+0000","last_clean":"2023-10-16T13:49:02.383158+0000","last_became_active":"2023-10-16T13:49:02.381703+0000","last_became_peered":"2023-10-16T13:49:02.381703+0000","last_unstale":"2023-10-16T13:49:02.383158+0000","last_undegraded":"2023-10-16T13:49:02.383158+0000","last_fullsized":"2023-10-16T13:49:02.383158+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:26:39.098172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.5","version":"0'0","reported_seq":11,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.371146+0000","last_change":"2023-10-16T13:49:01.694996+0000","last_active":"2023-10-16T13:49:02.371146+0000","last_peered":"2023-10-16T13:49:02.371146+0000","last_clean":"2023-10-16T13:49:02.371146+0000","last_became_active":"2023-10-16T13:49:01.694752+0000","last_became_peered":"2023-10-16T13:49:01.694752+0000","last_unstale":"2023-10-16T13:49:02.371146+0000","last_undegraded":"2023-10-16T13:49:02.371146+0000","last_fullsized":"2023-10-16T13:49:02.371146+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_clean_scrub_stamp":"2023-10-16T13:49:00.664589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:05:09.858912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.7","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.377755+0000","last_change":"2023-10-16T13:49:02.376440+0000","last_active":"2023-10-16T13:49:02.377755+0000","last_peered":"2023-10-16T13:49:02.377755+0000","last_clean":"2023-10-16T13:49:02.377755+0000","last_became_active":"2023-10-16T13:49:02.376158+0000","last_became_peered":"2023-10-16T13:49:02.376158+0000","last_unstale":"2023-10-16T13:49:02.377755+0000","last_undegraded":"2023-10-16T13:49:02.377755+0000","last_fullsized":"2023-10-16T13:49:02.377755+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:19:05.837652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.4","version":"0'0","reported_seq":12,"reported_epoch":278,"state":"active+clean","last_fresh":"2023-10-16T13:49:03.369040+0000","last_change":"2023-10-16T13:49:01.691473+0000","last_active":"2023-10-16T13:49:03.369040+0000","last_peered":"2023-10-16T13:49:03.369040+0000","last_clean":"2023-10-16T13:49:03.369040+0000","last_became_active":"2023-10-16T13:49:01.691045+0000","last_became_peered":"2023-10-16T13:49:01.691045+0000","last_unstale":"2023-10-16T13:49:03.369040+0000","last_undegraded":"2023-10-16T13:49:03.369040+0000","last_fullsized":"2023-10-16T13:49:03.369040+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_clean_scrub_stamp":"2023-10-16T13:49:00.664589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:53:36.987416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.0","version":"0'0","reported_seq":11,"reported_epoch":278,"state":"active+clean","last_fresh":"2023-10-16T13:49:03.368881+0000","last_change":"2023-10-16T13:49:02.385631+0000","last_active":"2023-10-16T13:49:03.368881+0000","last_peered":"2023-10-16T13:49:03.368881+0000","last_clean":"2023-10-16T13:49:03.368881+0000","last_became_active":"2023-10-16T13:49:02.385327+0000","last_became_peered":"2023-10-16T13:49:02.385327+0000","last_unstale":"2023-10-16T13:49:03.368881+0000","last_undegraded":"2023-10-16T13:49:03.368881+0000","last_fullsized":"2023-10-16T13:49:03.368881+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:44:58.444483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.3","version":"0'0","reported_seq":12,"reported_epoch":278,"state":"active+clean","last_fresh":"2023-10-16T13:49:03.368181+0000","last_change":"2023-10-16T13:49:01.694089+0000","last_active":"2023-10-16T13:49:03.368181+0000","last_peered":"2023-10-16T13:49:03.368181+0000","last_clean":"2023-10-16T13:49:03.368181+0000","last_became_active":"2023-10-16T13:49:01.693769+0000","last_became_peered":"2023-10-16T13:49:01.693769+0000","last_unstale":"2023-10-16T13:49:03.368181+0000","last_undegraded":"2023-10-16T13:49:03.368181+0000","last_fullsized":"2023-10-16T13:49:03.368181+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_clean_scrub_stamp":"2023-10-16T13:49:00.664589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:11:12.807972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"38.1","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.385847+0000","last_change":"2023-10-16T13:49:02.385429+0000","last_active":"2023-10-16T13:49:02.385847+0000","last_peered":"2023-10-16T13:49:02.385847+0000","last_clean":"2023-10-16T13:49:02.385847+0000","last_became_active":"2023-10-16T13:49:02.385115+0000","last_became_peered":"2023-10-16T13:49:02.385115+0000","last_unstale":"2023-10-16T13:49:02.385847+0000","last_undegraded":"2023-10-16T13:49:02.385847+0000","last_fullsized":"2023-10-16T13:49:02.385847+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:10:29.903850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.2","version":"0'0","reported_seq":11,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.372329+0000","last_change":"2023-10-16T13:49:01.693265+0000","last_active":"2023-10-16T13:49:02.372329+0000","last_peered":"2023-10-16T13:49:02.372329+0000","last_clean":"2023-10-16T13:49:02.372329+0000","last_became_active":"2023-10-16T13:49:01.688515+0000","last_became_peered":"2023-10-16T13:49:01.688515+0000","last_unstale":"2023-10-16T13:49:02.372329+0000","last_undegraded":"2023-10-16T13:49:02.372329+0000","last_fullsized":"2023-10-16T13:49:02.372329+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_clean_scrub_stamp":"2023-10-16T13:49:00.664589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:37:38.050689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.2","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.379098+0000","last_change":"2023-10-16T13:49:02.377131+0000","last_active":"2023-10-16T13:49:02.379098+0000","last_peered":"2023-10-16T13:49:02.379098+0000","last_clean":"2023-10-16T13:49:02.379098+0000","last_became_active":"2023-10-16T13:49:02.376786+0000","last_became_peered":"2023-10-16T13:49:02.376786+0000","last_unstale":"2023-10-16T13:49:02.379098+0000","last_undegraded":"2023-10-16T13:49:02.379098+0000","last_fullsized":"2023-10-16T13:49:02.379098+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:26:04.311239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.1","version":"0'0","reported_seq":11,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.368364+0000","last_change":"2023-10-16T13:49:01.685519+0000","last_active":"2023-10-16T13:49:02.368364+0000","last_peered":"2023-10-16T13:49:02.368364+0000","last_clean":"2023-10-16T13:49:02.368364+0000","last_became_active":"2023-10-16T13:49:01.685170+0000","last_became_peered":"2023-10-16T13:49:01.685170+0000","last_unstale":"2023-10-16T13:49:02.368364+0000","last_undegraded":"2023-10-16T13:49:02.368364+0000","last_fullsized":"2023-10-16T13:49:02.368364+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_clean_scrub_stamp":"2023-10-16T13:49:00.664589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:01:06.177631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.3","version":"0'0","reported_seq":11,"reported_epoch":278,"state":"active+clean","last_fresh":"2023-10-16T13:49:04.019010+0000","last_change":"2023-10-16T13:49:02.373374+0000","last_active":"2023-10-16T13:49:04.019010+0000","last_peered":"2023-10-16T13:49:04.019010+0000","last_clean":"2023-10-16T13:49:04.019010+0000","last_became_active":"2023-10-16T13:49:02.371820+0000","last_became_peered":"2023-10-16T13:49:02.371820+0000","last_unstale":"2023-10-16T13:49:04.019010+0000","last_undegraded":"2023-10-16T13:49:04.019010+0000","last_fullsized":"2023-10-16T13:49:04.019010+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:07:27.543946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.0","version":"0'0","reported_seq":11,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.371110+0000","last_change":"2023-10-16T13:49:01.688914+0000","last_active":"2023-10-16T13:49:02.371110+0000","last_peered":"2023-10-16T13:49:02.371110+0000","last_clean":"2023-10-16T13:49:02.371110+0000","last_became_active":"2023-10-16T13:49:01.688588+0000","last_became_peered":"2023-10-16T13:49:01.688588+0000","last_unstale":"2023-10-16T13:49:02.371110+0000","last_undegraded":"2023-10-16T13:49:02.371110+0000","last_fullsized":"2023-10-16T13:49:02.371110+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_clean_scrub_stamp":"2023-10-16T13:49:00.664589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:29:31.243509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.27","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.387063+0000","last_change":"2023-10-16T13:49:02.381169+0000","last_active":"2023-10-16T13:49:02.387063+0000","last_peered":"2023-10-16T13:49:02.387063+0000","last_clean":"2023-10-16T13:49:02.387063+0000","last_became_active":"2023-10-16T13:49:02.380798+0000","last_became_peered":"2023-10-16T13:49:02.380798+0000","last_unstale":"2023-10-16T13:49:02.387063+0000","last_undegraded":"2023-10-16T13:49:02.387063+0000","last_fullsized":"2023-10-16T13:49:02.387063+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:41:38.172251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"268'403","reported_seq":1441,"reported_epoch":276,"state":"active+clean","last_fresh":"2023-10-16T13:49:01.687159+0000","last_change":"2023-10-16T12:10:34.746820+0000","last_active":"2023-10-16T13:49:01.687159+0000","last_peered":"2023-10-16T13:49:01.687159+0000","last_clean":"2023-10-16T13:49:01.687159+0000","last_became_active":"2023-10-16T12:10:34.745574+0000","last_became_peered":"2023-10-16T12:10:34.745574+0000","last_unstale":"2023-10-16T13:49:01.687159+0000","last_undegraded":"2023-10-16T13:49:01.687159+0000","last_fullsized":"2023-10-16T13:49:01.687159+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_clean_scrub_stamp":"2023-10-16T12:10:33.577111+0000","objects_scrubbed":0,"log_size":403,"log_dups_size":0,"ondisk_log_size":403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:14.461986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2612,"num_read_kb":5456,"num_write":679,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":38,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":37,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2612,"num_read_kb":5456,"num_write":679,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":403,"ondisk_log_size":403,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576013,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61564,"kb_used_data":4008,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310276,"statfs":{"total":96636764160,"available":96573722624,"internally_reserved":0,"allocated":4104192,"data_stored":2410507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58915275},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:48:54 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.58399999999999996,"15min":0.621},"min":{"1min":0.34999999999999998,"5min":0.34200000000000003,"15min":0.33000000000000002},"max":{"1min":4.1040000000000001,"5min":4.1040000000000001,"15min":4.1040000000000001},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.753,"5min":0.60699999999999998,"15min":0.625},"min":{"1min":0.35199999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":5.5279999999999996,"5min":5.5279999999999996,"15min":5.5279999999999996},"last":0.46200000000000002}]},{"osd":1,"last update":"Mon Oct 16 13:48:54 2023","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.63100000000000001,"15min":0.64300000000000002},"min":{"1min":0.33500000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":6.6189999999999998,"5min":6.6189999999999998,"15min":6.6189999999999998},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.59999999999999998,"15min":0.626},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":3.3239999999999998,"5min":3.3239999999999998,"15min":3.3239999999999998},"last":0.53900000000000003}]},{"osd":2,"last update":"Mon Oct 16 13:48:54 2023","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.64500000000000002,"15min":0.67700000000000005},"min":{"1min":0.41499999999999998,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":2.4430000000000001,"5min":2.4430000000000001,"15min":2.4430000000000001},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.67200000000000004,"15min":0.69799999999999995},"min":{"1min":0.45500000000000002,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":4.351,"5min":4.351,"15min":4.351},"last":0.59999999999999998}]},{"osd":3,"last update":"Mon Oct 16 13:48:54 2023","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.63300000000000001,"15min":0.65500000000000003},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":4.4130000000000003,"5min":4.4130000000000003,"15min":4.4130000000000003},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.67600000000000005,"15min":0.67100000000000004},"min":{"1min":0.432,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":5.407,"5min":5.407,"15min":5.407},"last":0.624}]},{"osd":4,"last update":"Mon Oct 16 13:48:54 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.56100000000000005,"15min":0.53500000000000003},"min":{"1min":0.37,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":3.2400000000000002,"5min":3.2400000000000002,"15min":3.2400000000000002},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.67900000000000005,"15min":0.63900000000000001},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":5.3369999999999997,"5min":5.3369999999999997,"15min":5.3369999999999997},"last":0.65300000000000002}]},{"osd":5,"last update":"Mon Oct 16 13:48:54 2023","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.621,"15min":0.57199999999999995},"min":{"1min":0.35799999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":5.4720000000000004,"5min":5.4720000000000004,"15min":5.4720000000000004},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.61299999999999999,"15min":0.59799999999999998},"min":{"1min":0.40100000000000002,"5min":0.39700000000000002,"15min":0.35099999999999998},"max":{"1min":1.0549999999999999,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.76700000000000002}]},{"osd":6,"last update":"Mon Oct 16 13:48:54 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.63200000000000001,"15min":0.622},"min":{"1min":0.46600000000000003,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.69499999999999995,"15min":0.64500000000000002},"min":{"1min":0.38900000000000001,"5min":0.378,"15min":0.35199999999999998},"max":{"1min":5.3860000000000001,"5min":5.3860000000000001,"15min":5.3860000000000001},"last":0.70399999999999996}]}]},{"osd":6,"up_from":13,"seq":55834576011,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48500,"kb_used_data":3432,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323340,"statfs":{"total":96636764160,"available":96587100160,"internally_reserved":0,"allocated":3514368,"data_stored":1820721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.60999999999999999,"15min":0.65000000000000002},"min":{"1min":0.315,"5min":0.315,"15min":0.29599999999999999},"max":{"1min":0.67600000000000005,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.503,"5min":0.60599999999999998,"15min":0.64200000000000002},"min":{"1min":0.373,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.70099999999999996,"5min":0.91400000000000003,"15min":0.93799999999999994},"last":0.64300000000000002}]},{"osd":1,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.54600000000000004,"15min":0.60499999999999998},"min":{"1min":0.316,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.64500000000000002,"5min":0.86899999999999999,"15min":0.89600000000000002},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.58599999999999997,"15min":0.60599999999999998},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.69799999999999995,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.53600000000000003}]},{"osd":2,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.58899999999999997,"15min":0.61799999999999999},"min":{"1min":0.34799999999999998,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.71499999999999997,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.57599999999999996,"15min":0.60599999999999998},"min":{"1min":0.32200000000000001,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.72999999999999998,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.67500000000000004}]},{"osd":3,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.60199999999999998,"15min":0.64600000000000002},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.38200000000000001},"max":{"1min":0.63400000000000001,"5min":0.92400000000000004,"15min":0.95599999999999996},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.66600000000000004,"15min":0.70199999999999996},"min":{"1min":0.435,"5min":0.435,"15min":0.42299999999999999},"max":{"1min":0.72399999999999998,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.69099999999999995}]},{"osd":4,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.53300000000000003,"15min":0.52800000000000002},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.67700000000000005,"5min":0.85599999999999998,"15min":0.95799999999999996},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.60799999999999998,"15min":0.61499999999999999},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.70799999999999996,"5min":0.98799999999999999,"15min":1.0309999999999999},"last":0.59799999999999998}]},{"osd":5,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.58199999999999996,"15min":0.59099999999999997},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.71099999999999997,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.627,"5min":0.64400000000000002,"15min":0.65700000000000003},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.78700000000000003,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.57899999999999996}]},{"osd":7,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.59099999999999997,"15min":0.59799999999999998},"min":{"1min":0.35199999999999998,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.747,"5min":1.081,"15min":1.081},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.63900000000000001,"15min":0.64900000000000002},"min":{"1min":0.40999999999999998,"5min":0.40899999999999997,"15min":0.379},"max":{"1min":0.76600000000000001,"5min":1.099,"15min":1.099},"last":0.55300000000000005}]}]},{"osd":1,"up_from":13,"seq":55834576011,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52652,"kb_used_data":3424,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319188,"statfs":{"total":96636764160,"available":96582848512,"internally_reserved":0,"allocated":3506176,"data_stored":1820117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":50395593},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:48:59 2023","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.37,"15min":0.40100000000000002},"min":{"1min":0.254,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.58099999999999996,"5min":0.65400000000000003,"15min":0.65800000000000003},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.433,"15min":0.48099999999999998},"min":{"1min":0.32700000000000001,"5min":0.253,"15min":0.253},"max":{"1min":0.56599999999999995,"5min":0.68000000000000005,"15min":0.77200000000000002},"last":0.53500000000000003}]},{"osd":2,"last update":"Mon Oct 16 13:48:59 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.39400000000000002,"15min":0.439},"min":{"1min":0.251,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.496,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.44500000000000001,"15min":0.47999999999999998},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.58299999999999996,"5min":0.67600000000000005,"15min":0.71399999999999997},"last":0.49399999999999999}]},{"osd":3,"last update":"Mon Oct 16 13:48:59 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.39000000000000001,"15min":0.42299999999999999},"min":{"1min":0.28499999999999998,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.51300000000000001,"5min":0.65200000000000002,"15min":0.72099999999999997},"last":0.443},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.41699999999999998,"15min":0.46600000000000003},"min":{"1min":0.29599999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.60299999999999998,"5min":0.69399999999999995,"15min":0.77300000000000002},"last":0.51300000000000001}]},{"osd":4,"last update":"Mon Oct 16 13:48:59 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.61599999999999999,"15min":0.65700000000000003},"min":{"1min":0.41199999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.78300000000000003,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.59899999999999998,"15min":0.65600000000000003},"min":{"1min":0.377,"5min":0.37,"15min":0.37},"max":{"1min":0.90400000000000003,"5min":0.91900000000000004,"15min":0.997},"last":0.68300000000000005}]},{"osd":5,"last update":"Mon Oct 16 13:48:59 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.64700000000000002,"15min":0.69899999999999995},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":1.0760000000000001},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.65700000000000003,"15min":0.71499999999999997},"min":{"1min":0.436,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.81299999999999994,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.64300000000000002}]},{"osd":6,"last update":"Mon Oct 16 13:48:59 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.621,"15min":0.66900000000000004},"min":{"1min":0.41999999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85799999999999998,"5min":0.95099999999999996,"15min":0.97799999999999998},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.62,"5min":0.64500000000000002,"15min":0.69499999999999995},"min":{"1min":0.46899999999999997,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":1.083},"last":0.70499999999999996}]},{"osd":7,"last update":"Mon Oct 16 13:48:59 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.66300000000000003,"15min":0.72199999999999998},"min":{"1min":0.45300000000000001,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.79300000000000004,"5min":1.163,"15min":1.163},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.68799999999999994,"15min":0.73699999999999999},"min":{"1min":0.47999999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.94699999999999995,"5min":0.98499999999999999,"15min":1.0580000000000001},"last":0.72299999999999998}]}]},{"osd":0,"up_from":13,"seq":55834576012,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53236,"kb_used_data":4008,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318604,"statfs":{"total":96636764160,"available":96582250496,"internally_reserved":0,"allocated":4104192,"data_stored":2410575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.373,"15min":0.41299999999999998},"min":{"1min":0.23799999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.50900000000000001,"5min":0.60699999999999998,"15min":0.71399999999999997},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.38500000000000001,"15min":0.42799999999999999},"min":{"1min":0.24199999999999999,"5min":0.222,"15min":0.222},"max":{"1min":0.56799999999999995,"5min":0.77700000000000002,"15min":0.82199999999999995},"last":0.32100000000000001}]},{"osd":2,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.40999999999999998,"15min":0.441},"min":{"1min":0.27300000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.51000000000000001,"5min":0.65900000000000003,"15min":0.75700000000000001},"last":0.44},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.432,"15min":0.47199999999999998},"min":{"1min":0.30399999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.54500000000000004,"5min":0.68799999999999994,"15min":0.83299999999999996},"last":0.35999999999999999}]},{"osd":3,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.40600000000000003,"15min":0.44400000000000001},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.52100000000000002,"5min":0.69699999999999995,"15min":0.78600000000000003},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.46300000000000002,"15min":0.50600000000000001},"min":{"1min":0.309,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.55900000000000005,"5min":0.70399999999999996,"15min":0.86399999999999999},"last":0.64100000000000001}]},{"osd":4,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.64500000000000002,"15min":0.67600000000000005},"min":{"1min":0.42999999999999999,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.82999999999999996,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.64900000000000002,"15min":0.68100000000000005},"min":{"1min":0.43099999999999999,"5min":0.375,"15min":0.375},"max":{"1min":0.81000000000000005,"5min":1.21,"15min":1.21},"last":0.68200000000000005}]},{"osd":5,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.63400000000000001,"15min":0.67700000000000005},"min":{"1min":0.40300000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.82199999999999995,"5min":1.242,"15min":1.242},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.622,"5min":0.68300000000000005,"15min":0.71799999999999997},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.83299999999999996,"5min":1.081,"15min":1.081},"last":0.47999999999999998}]},{"osd":6,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.67600000000000005,"15min":0.71999999999999997},"min":{"1min":0.43099999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.84699999999999998,"5min":1.034,"15min":1.103},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.67700000000000005,"15min":0.71499999999999997},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.90900000000000003,"5min":1.131,"15min":1.131},"last":0.59599999999999997}]},{"osd":7,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.68700000000000006,"15min":0.73499999999999999},"min":{"1min":0.47399999999999998,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.94799999999999995,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.69599999999999995,"15min":0.72699999999999998},"min":{"1min":0.47899999999999998,"5min":0.433,"15min":0.433},"max":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.71699999999999997}]}]},{"osd":2,"up_from":13,"seq":55834576012,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56848,"kb_used_data":3452,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314992,"statfs":{"total":96636764160,"available":96578551808,"internally_reserved":0,"allocated":3534848,"data_stored":1828547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:48:41 2023","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.374,"15min":0.40400000000000003},"min":{"1min":0.23899999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.439,"5min":0.623,"15min":0.77500000000000002},"last":0.433},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.436,"15min":0.46999999999999997},"min":{"1min":0.33700000000000002,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.57399999999999995,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.57999999999999996}]},{"osd":1,"last update":"Mon Oct 16 13:48:41 2023","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.46100000000000002,"15min":0.49199999999999999},"min":{"1min":0.30499999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.69899999999999995,"5min":0.73199999999999998,"15min":0.81399999999999995},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.45000000000000001,"15min":0.46899999999999997},"min":{"1min":0.27900000000000003,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.51700000000000002,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.53400000000000003}]},{"osd":3,"last update":"Mon Oct 16 13:48:41 2023","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.40400000000000003,"15min":0.435},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.55500000000000005,"5min":0.63,"15min":0.72599999999999998},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.499,"15min":0.53700000000000003},"min":{"1min":0.376,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.65900000000000003,"5min":0.82699999999999996,"15min":0.90100000000000002},"last":0.65200000000000002}]},{"osd":4,"last update":"Mon Oct 16 13:48:41 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.66800000000000004,"15min":0.70699999999999996},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.423,"5min":1.423,"15min":1.423},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.67700000000000005,"15min":0.71399999999999997},"min":{"1min":0.437,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.407,"5min":1.407,"15min":1.407},"last":0.82499999999999996}]},{"osd":5,"last update":"Mon Oct 16 13:48:41 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.627,"15min":0.67800000000000005},"min":{"1min":0.436,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.82199999999999995,"5min":0.87,"15min":1.486},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.64900000000000002,"15min":0.68999999999999995},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.78200000000000003,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":0.80800000000000005}]},{"osd":6,"last update":"Mon Oct 16 13:48:41 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.622,"15min":0.65900000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.80300000000000005,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.66200000000000003,"15min":0.71099999999999997},"min":{"1min":0.49099999999999999,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.85499999999999998,"5min":0.94299999999999995,"15min":1.4379999999999999},"last":0.71399999999999997}]},{"osd":7,"last update":"Mon Oct 16 13:48:41 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.67500000000000004,"15min":0.70899999999999996},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.83499999999999996,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.66800000000000004,"15min":0.72199999999999998},"min":{"1min":0.48799999999999999,"5min":0.436,"15min":0.436},"max":{"1min":0.88200000000000001,"5min":1.0429999999999999,"15min":1.377},"last":0.84499999999999997}]}]},{"osd":3,"up_from":13,"seq":55834576011,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56820,"kb_used_data":3428,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315020,"statfs":{"total":96636764160,"available":96578580480,"internally_reserved":0,"allocated":3510272,"data_stored":1820151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:48:45 2023","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.375,"15min":0.40400000000000003},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23000000000000001},"max":{"1min":0.48399999999999999,"5min":0.60099999999999998,"15min":0.70499999999999996},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.38300000000000001,"15min":0.41599999999999998},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.48799999999999999,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.34100000000000003}]},{"osd":1,"last update":"Mon Oct 16 13:48:45 2023","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.38900000000000001,"15min":0.42299999999999999},"min":{"1min":0.27200000000000002,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.497,"5min":0.68200000000000005,"15min":0.76300000000000001},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.438,"15min":0.46999999999999997},"min":{"1min":0.27800000000000002,"5min":0.27700000000000002,"15min":0.253},"max":{"1min":0.60599999999999998,"5min":0.68700000000000006,"15min":0.77800000000000002},"last":0.41099999999999998}]},{"osd":2,"last update":"Mon Oct 16 13:48:45 2023","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.45100000000000001,"15min":0.47999999999999998},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.71399999999999997,"5min":1.167,"15min":1.167},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.46000000000000002,"15min":0.50700000000000001},"min":{"1min":0.32500000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.63200000000000001,"5min":0.73299999999999998,"15min":0.76800000000000002},"last":0.36499999999999999}]},{"osd":4,"last update":"Mon Oct 16 13:48:45 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.61799999999999999,"15min":0.65500000000000003},"min":{"1min":0.40100000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.137},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65400000000000003,"15min":0.68899999999999995},"min":{"1min":0.45500000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.84699999999999998,"5min":1.2110000000000001,"15min":1.2909999999999999},"last":0.60199999999999998}]},{"osd":5,"last update":"Mon Oct 16 13:48:45 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.65200000000000002,"15min":0.69799999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.374},"max":{"1min":1.0309999999999999,"5min":1.0680000000000001,"15min":1.2729999999999999},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.65400000000000003,"15min":0.68999999999999995},"min":{"1min":0.47299999999999998,"5min":0.45600000000000002,"15min":0.432},"max":{"1min":0.871,"5min":1.0549999999999999,"15min":1.173},"last":0.51900000000000002}]},{"osd":6,"last update":"Mon Oct 16 13:48:45 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.69799999999999995,"15min":0.71299999999999997},"min":{"1min":0.432,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.115,"5min":5.0999999999999996,"15min":5.0999999999999996},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.68600000000000005,"15min":0.72499999999999998},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.2290000000000001},"last":0.59199999999999997}]},{"osd":7,"last update":"Mon Oct 16 13:48:45 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.68100000000000005,"15min":0.72699999999999998},"min":{"1min":0.45700000000000002,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.21,"5min":1.21,"15min":1.2569999999999999},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.68700000000000006,"15min":0.73399999999999999},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.43099999999999999},"max":{"1min":0.90600000000000003,"5min":1.262,"15min":1.262},"last":0.49099999999999999}]}]},{"osd":4,"up_from":13,"seq":55834576010,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52696,"kb_used_data":3460,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319144,"statfs":{"total":96636764160,"available":96582803456,"internally_reserved":0,"allocated":3543040,"data_stored":1829219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:48:42 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.56299999999999994,"15min":0.61299999999999999},"min":{"1min":0.36399999999999999,"5min":0.314,"15min":0.314},"max":{"1min":0.77800000000000002,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.441},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.57399999999999995,"15min":0.624},"min":{"1min":0.36599999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.87,"5min":0.87,"15min":0.89500000000000002},"last":0.48599999999999999}]},{"osd":1,"last update":"Mon Oct 16 13:48:42 2023","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.57499999999999996,"15min":0.64100000000000001},"min":{"1min":0.35699999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.79900000000000004,"5min":0.877,"15min":0.877},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.58399999999999996,"15min":0.63},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.80900000000000005,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.58199999999999996}]},{"osd":2,"last update":"Mon Oct 16 13:48:42 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.59199999999999997,"15min":0.64100000000000001},"min":{"1min":0.434,"5min":0.378,"15min":0.378},"max":{"1min":0.751,"5min":0.89900000000000002,"15min":0.90600000000000003},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.62,"15min":0.65300000000000002},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.755,"5min":0.89700000000000002,"15min":0.92600000000000005},"last":0.67100000000000004}]},{"osd":3,"last update":"Mon Oct 16 13:48:42 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.58699999999999997,"15min":0.63300000000000001},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.81399999999999995,"5min":0.86599999999999999,"15min":0.99199999999999999},"last":0.629},{"interface":"front","average":{"1min":0.624,"5min":0.64400000000000002,"15min":0.67500000000000004},"min":{"1min":0.48199999999999998,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.93899999999999995},"last":0.61499999999999999}]},{"osd":5,"last update":"Mon Oct 16 13:48:42 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.55700000000000005,"15min":0.53800000000000003},"min":{"1min":0.34899999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.80400000000000005,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.57799999999999996,"15min":0.57099999999999995},"min":{"1min":0.307,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.72499999999999998,"5min":0.82199999999999995,"15min":0.94899999999999995},"last":0.70399999999999996}]},{"osd":6,"last update":"Mon Oct 16 13:48:42 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.63200000000000001,"15min":0.60399999999999998},"min":{"1min":0.38600000000000001,"5min":0.36399999999999999,"15min":0.33200000000000002},"max":{"1min":0.82699999999999996,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.70099999999999996,"15min":0.69699999999999995},"min":{"1min":0.41699999999999998,"5min":0.40300000000000002,"15min":0.39400000000000002},"max":{"1min":0.89700000000000002,"5min":1.129,"15min":1.129},"last":0.77900000000000003}]},{"osd":7,"last update":"Mon Oct 16 13:48:42 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.65100000000000002,"15min":0.63700000000000001},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.78700000000000003,"5min":0.97099999999999997,"15min":0.98599999999999999},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.70999999999999996,"15min":0.68899999999999995},"min":{"1min":0.48399999999999999,"5min":0.44,"15min":0.38200000000000001},"max":{"1min":0.96999999999999997,"5min":0.97599999999999998,"15min":1.026},"last":0.72199999999999998}]}]},{"osd":5,"up_from":13,"seq":55834576011,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74868,"kb_used_data":3428,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94296972,"statfs":{"total":96636764160,"available":96560099328,"internally_reserved":0,"allocated":3510272,"data_stored":1820151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:48:29 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.57199999999999995,"15min":0.60799999999999998},"min":{"1min":0.32100000000000001,"5min":0.314,"15min":0.29899999999999999},"max":{"1min":0.67600000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.58799999999999997,"15min":0.625},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.29699999999999999},"max":{"1min":0.58099999999999996,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.69699999999999995}]},{"osd":1,"last update":"Mon Oct 16 13:48:29 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.622,"15min":0.65100000000000002},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.29499999999999998},"max":{"1min":0.72399999999999998,"5min":1,"15min":1},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.58799999999999997,"15min":0.628},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.67800000000000005,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.754}]},{"osd":2,"last update":"Mon Oct 16 13:48:29 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.624,"15min":0.64500000000000002},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.755,"5min":0.91000000000000003,"15min":0.94699999999999995},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.61399999999999999,"15min":0.64600000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.73399999999999999,"5min":0.871,"15min":0.93500000000000005},"last":0.51400000000000001}]},{"osd":3,"last update":"Mon Oct 16 13:48:29 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.64400000000000002,"15min":0.66100000000000003},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.70999999999999996,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.63},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.66700000000000004,"15min":0.68899999999999995},"min":{"1min":0.439,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.81499999999999995,"5min":1.4199999999999999,"15min":1.4199999999999999},"last":0.53900000000000003}]},{"osd":4,"last update":"Mon Oct 16 13:48:29 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.53800000000000003,"15min":0.53900000000000003},"min":{"1min":0.36899999999999999,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.77700000000000002,"5min":0.89700000000000002,"15min":0.90700000000000003},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.629,"5min":0.60099999999999998,"15min":0.59699999999999998},"min":{"1min":0.375,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.78500000000000003,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.74299999999999999}]},{"osd":6,"last update":"Mon Oct 16 13:48:29 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.59499999999999997,"15min":0.59299999999999997},"min":{"1min":0.35899999999999999,"5min":0.315,"15min":0.315},"max":{"1min":0.85799999999999998,"5min":0.92000000000000004,"15min":0.93200000000000005},"last":0.495},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.624,"15min":0.625},"min":{"1min":0.49099999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.77600000000000002,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.67300000000000004}]},{"osd":7,"last update":"Mon Oct 16 13:48:29 2023","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.65100000000000002,"15min":0.64100000000000001},"min":{"1min":0.47899999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.77300000000000002,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.69899999999999995,"15min":0.69199999999999995},"min":{"1min":0.48799999999999999,"5min":0.375,"15min":0.375},"max":{"1min":0.89900000000000002,"5min":0.97299999999999998,"15min":1.0229999999999999},"last":0.65000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T13:49:06.862 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T13:49:07.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.019+0000 7efe8db50700 1 Processor -- start 2023-10-16T13:49:07.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.019+0000 7efe8db50700 1 -- start start 2023-10-16T13:49:07.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.020+0000 7efe8db50700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7efe880d41c0 0x7efe880d0040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:07.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.020+0000 7efe8db50700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efe880d0580 0x7efe8806c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:07.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.020+0000 7efe8db50700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe8806ba00 con 0x7efe8815d8d0 2023-10-16T13:49:07.026 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.020+0000 7efe8db50700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe880d0a10 con 0x7efe880d41c0 2023-10-16T13:49:07.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.020+0000 7efe8db50700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe880d26d0 con 0x7efe880d0580 2023-10-16T13:49:07.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.020+0000 7efe877fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7efe8815d8d0 0x7efe880d1c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48328/0 (socket says 172.21.15.16:48328) 2023-10-16T13:49:07.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.020+0000 7efe877fe700 1 -- 172.21.15.16:0/592350448 learned_addr learned my addr 172.21.15.16:0/592350448 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:49:07.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.020+0000 7efe86ffd700 1 --2- 172.21.15.16:0/592350448 >> v2:172.21.15.49:3300/0 conn(0x7efe880d41c0 0x7efe880d0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:07.027 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.020+0000 7efe87fff700 1 --2- 172.21.15.16:0/592350448 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efe880d0580 0x7efe8806c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:07.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.020+0000 7efe867fc700 1 -- 172.21.15.16:0/592350448 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 129195851 0 0) 0x7efe8806ba00 con 0x7efe8815d8d0 2023-10-16T13:49:07.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.020+0000 7efe867fc700 1 -- 172.21.15.16:0/592350448 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe740035b0 con 0x7efe8815d8d0 2023-10-16T13:49:07.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.021+0000 7efe867fc700 1 -- 172.21.15.16:0/592350448 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 575640959 0 0) 0x7efe740035b0 con 0x7efe8815d8d0 2023-10-16T13:49:07.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.021+0000 7efe867fc700 1 -- 172.21.15.16:0/592350448 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efe880d0580 msgr2=0x7efe8806c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:07.028 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.021+0000 7efe867fc700 1 --2- 172.21.15.16:0/592350448 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efe880d0580 0x7efe8806c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:07.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.021+0000 7efe867fc700 1 -- 172.21.15.16:0/592350448 >> v2:172.21.15.49:3300/0 conn(0x7efe880d41c0 msgr2=0x7efe880d0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:07.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.021+0000 7efe867fc700 1 --2- 172.21.15.16:0/592350448 >> v2:172.21.15.49:3300/0 conn(0x7efe880d41c0 0x7efe880d0040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:07.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.021+0000 7efe867fc700 1 -- 172.21.15.16:0/592350448 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe880d23f0 con 0x7efe8815d8d0 2023-10-16T13:49:07.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.021+0000 7efe867fc700 1 -- 172.21.15.16:0/592350448 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7efe78001670 con 0x7efe8815d8d0 2023-10-16T13:49:07.029 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.021+0000 7efe867fc700 1 -- 172.21.15.16:0/592350448 >> v1:172.21.15.16:6789/0 conn(0x7efe8815d8d0 legacy=0x7efe880d1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:07.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.021+0000 7efe867fc700 1 --2- 172.21.15.16:0/592350448 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efe880d0580 0x7efe74003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:07.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.021+0000 7efe867fc700 1 --2- 172.21.15.16:0/592350448 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efe74003f80 0x7efe74004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:07.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.021+0000 7efe867fc700 1 --2- 172.21.15.16:0/592350448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe74004d90 0x7efe74005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:07.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.021+0000 7efe867fc700 1 -- 172.21.15.16:0/592350448 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe880d0a10 con 0x7efe74004d90 2023-10-16T13:49:07.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.021+0000 7efe867fc700 1 -- 172.21.15.16:0/592350448 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe880d26d0 con 0x7efe74003f80 2023-10-16T13:49:07.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.021+0000 7efe867fc700 1 -- 172.21.15.16:0/592350448 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe7400ba10 con 0x7efe880d0580 2023-10-16T13:49:07.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.021+0000 7efe87fff700 1 --2- 172.21.15.16:0/592350448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe74004d90 0x7efe74005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:07.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.021+0000 7efe86ffd700 1 --2- 172.21.15.16:0/592350448 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efe74003f80 0x7efe74004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:07.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.021+0000 7efe877fe700 1 --2- 172.21.15.16:0/592350448 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efe880d0580 0x7efe74003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:07.031 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.021+0000 7efe87fff700 1 -- 172.21.15.16:0/592350448 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efe880d0580 msgr2=0x7efe74003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:07.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.021+0000 7efe87fff700 1 --2- 172.21.15.16:0/592350448 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efe880d0580 0x7efe74003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:07.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.021+0000 7efe87fff700 1 -- 172.21.15.16:0/592350448 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efe74003f80 msgr2=0x7efe74004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:07.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.021+0000 7efe87fff700 1 --2- 172.21.15.16:0/592350448 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efe74003f80 0x7efe74004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:07.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.021+0000 7efe87fff700 1 -- 172.21.15.16:0/592350448 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe740035b0 con 0x7efe74004d90 2023-10-16T13:49:07.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.021+0000 7efe87fff700 1 --2- 172.21.15.16:0/592350448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe74004d90 0x7efe74005210 secure :-1 s=READY pgs=3198 cs=0 l=1 rev1=1 crypto rx=0x7efe80009fc0 tx=0x7efe74005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f224177a767c6b0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:07.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.022+0000 7efe867fc700 1 -- 172.21.15.16:0/592350448 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe8000ebd0 con 0x7efe74004d90 2023-10-16T13:49:07.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.022+0000 7efe867fc700 1 -- 172.21.15.16:0/592350448 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe8000f040 con 0x7efe74004d90 2023-10-16T13:49:07.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.022+0000 7efe867fc700 1 -- 172.21.15.16:0/592350448 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe80014480 con 0x7efe74004d90 2023-10-16T13:49:07.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.022+0000 7efe8db50700 1 -- 172.21.15.16:0/592350448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe74004d90 msgr2=0x7efe74005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:07.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.022+0000 7efe8db50700 1 --2- 172.21.15.16:0/592350448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe74004d90 0x7efe74005210 secure :-1 s=READY pgs=3198 cs=0 l=1 rev1=1 crypto rx=0x7efe80009fc0 tx=0x7efe74005880 comp rx=0 tx=0).stop 2023-10-16T13:49:07.033 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.022+0000 7efe87fff700 1 -- 172.21.15.16:0/592350448 reap_dead start 2023-10-16T13:49:07.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.022+0000 7efe8db50700 1 -- 172.21.15.16:0/592350448 shutdown_connections 2023-10-16T13:49:07.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.022+0000 7efe8db50700 1 -- 172.21.15.16:0/592350448 >> 172.21.15.16:0/592350448 conn(0x7efe880d3630 msgr2=0x7efe880d3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:07.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.022+0000 7efe8db50700 1 -- 172.21.15.16:0/592350448 shutdown_connections 2023-10-16T13:49:07.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.022+0000 7efe8db50700 1 -- 172.21.15.16:0/592350448 wait complete. 2023-10-16T13:49:07.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.023+0000 7efe8db50700 1 Processor -- start 2023-10-16T13:49:07.034 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.024+0000 7efe8db50700 1 -- start start 2023-10-16T13:49:07.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.024+0000 7efe8db50700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efe8809f3f0 0x7efe881f3c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:07.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.024+0000 7efe8db50700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe8809d310 0x7efe88152690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:07.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.024+0000 7efe8db50700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efe8809f7d0 0x7efe88152bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:07.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.024+0000 7efe8db50700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe88153110 con 0x7efe8809d310 2023-10-16T13:49:07.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.024+0000 7efe8db50700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe8809e3b0 con 0x7efe8809f7d0 2023-10-16T13:49:07.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.024+0000 7efe8db50700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe8809d7a0 con 0x7efe8809f3f0 2023-10-16T13:49:07.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.024+0000 7efe86ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe8809d310 0x7efe88152690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:07.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.024+0000 7efe86ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe8809d310 0x7efe88152690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59068/0 (socket says 172.21.15.16:59068) 2023-10-16T13:49:07.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.024+0000 7efe86ffd700 1 -- 172.21.15.16:0/1196919455 learned_addr learned my addr 172.21.15.16:0/1196919455 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:07.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.025+0000 7efe877fe700 1 --2- 172.21.15.16:0/1196919455 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efe8809f3f0 0x7efe881f3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:07.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.025+0000 7efe87fff700 1 --2- 172.21.15.16:0/1196919455 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efe8809f7d0 0x7efe88152bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:07.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.025+0000 7efe86ffd700 1 -- 172.21.15.16:0/1196919455 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efe8809f3f0 msgr2=0x7efe881f3c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:07.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.025+0000 7efe86ffd700 1 --2- 172.21.15.16:0/1196919455 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efe8809f3f0 0x7efe881f3c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:07.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.025+0000 7efe86ffd700 1 -- 172.21.15.16:0/1196919455 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efe8809f7d0 msgr2=0x7efe88152bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:07.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.025+0000 7efe86ffd700 1 --2- 172.21.15.16:0/1196919455 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efe8809f7d0 0x7efe88152bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:07.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.025+0000 7efe86ffd700 1 -- 172.21.15.16:0/1196919455 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe8809da80 con 0x7efe8809d310 2023-10-16T13:49:07.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.025+0000 7efe877fe700 1 --2- 172.21.15.16:0/1196919455 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efe8809f3f0 0x7efe881f3c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:49:07.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.025+0000 7efe86ffd700 1 --2- 172.21.15.16:0/1196919455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe8809d310 0x7efe88152690 secure :-1 s=READY pgs=3199 cs=0 l=1 rev1=1 crypto rx=0x7efe70011aa0 tx=0x7efe70011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:07.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.025+0000 7efe7ffff700 1 -- 172.21.15.16:0/1196919455 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe7000f980 con 0x7efe8809d310 2023-10-16T13:49:07.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.025+0000 7efe7ffff700 1 -- 172.21.15.16:0/1196919455 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe7000fb00 con 0x7efe8809d310 2023-10-16T13:49:07.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.025+0000 7efe8db50700 1 -- 172.21.15.16:0/1196919455 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe8809dd60 con 0x7efe8809d310 2023-10-16T13:49:07.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.025+0000 7efe7ffff700 1 -- 172.21.15.16:0/1196919455 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe7000feb0 con 0x7efe8809d310 2023-10-16T13:49:07.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.025+0000 7efe8db50700 1 -- 172.21.15.16:0/1196919455 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe880c9f00 con 0x7efe8809d310 2023-10-16T13:49:07.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.026+0000 7efe8db50700 1 -- 172.21.15.16:0/1196919455 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe8804ebb0 con 0x7efe8809d310 2023-10-16T13:49:07.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.027+0000 7efe7ffff700 1 -- 172.21.15.16:0/1196919455 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 166) v1 ==== 89331+0+0 (secure 0 0 0) 0x7efe70014440 con 0x7efe8809d310 2023-10-16T13:49:07.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.027+0000 7efe7ffff700 1 --2- 172.21.15.16:0/1196919455 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7efe6806b020 0x7efe6806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:07.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.027+0000 7efe877fe700 1 --2- 172.21.15.16:0/1196919455 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7efe6806b020 0x7efe6806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:07.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.027+0000 7efe7ffff700 1 -- 172.21.15.16:0/1196919455 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(278..278 src has 1..278) v4 ==== 6157+0+0 (secure 0 0 0) 0x7efe7008a080 con 0x7efe8809d310 2023-10-16T13:49:07.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.028+0000 7efe877fe700 1 --2- 172.21.15.16:0/1196919455 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7efe6806b020 0x7efe6806d4e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7efe780015d0 tx=0x7efe78009040 comp rx=0 tx=0).ready entity=mgr.13344 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:07.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.032+0000 7efe7ffff700 1 -- 172.21.15.16:0/1196919455 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7efe70054fd0 con 0x7efe8809d310 2023-10-16T13:49:07.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.229+0000 7efe8db50700 1 -- 172.21.15.16:0/1196919455 --> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7efe8806bf80 con 0x7efe6806b020 2023-10-16T13:49:07.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.238+0000 7efe7ffff700 1 -- 172.21.15.16:0/1196919455 <== mgr.13344 v2:172.21.15.16:6832/1257392887 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289878 (secure 0 0 0) 0x7efe78007140 con 0x7efe6806b020 2023-10-16T13:49:07.243 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:49:07.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.241+0000 7efe8db50700 1 -- 172.21.15.16:0/1196919455 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7efe6806b020 msgr2=0x7efe6806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:07.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.241+0000 7efe8db50700 1 --2- 172.21.15.16:0/1196919455 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7efe6806b020 0x7efe6806d4e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7efe780015d0 tx=0x7efe78009040 comp rx=0 tx=0).stop 2023-10-16T13:49:07.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.241+0000 7efe8db50700 1 -- 172.21.15.16:0/1196919455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe8809d310 msgr2=0x7efe88152690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:07.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.241+0000 7efe8db50700 1 --2- 172.21.15.16:0/1196919455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe8809d310 0x7efe88152690 secure :-1 s=READY pgs=3199 cs=0 l=1 rev1=1 crypto rx=0x7efe70011aa0 tx=0x7efe70011e60 comp rx=0 tx=0).stop 2023-10-16T13:49:07.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.241+0000 7efe8db50700 1 -- 172.21.15.16:0/1196919455 shutdown_connections 2023-10-16T13:49:07.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.241+0000 7efe8db50700 1 --2- 172.21.15.16:0/1196919455 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efe8809f7d0 0x7efe88152bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:07.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.241+0000 7efe8db50700 1 --2- 172.21.15.16:0/1196919455 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7efe8809f3f0 0x7efe881f3c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:07.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.241+0000 7efe8db50700 1 --2- 172.21.15.16:0/1196919455 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7efe6806b020 0x7efe6806d4e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:07.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.241+0000 7efe8db50700 1 --2- 172.21.15.16:0/1196919455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe8809d310 0x7efe88152690 unknown :-1 s=CLOSED pgs=3199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:07.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.241+0000 7efe8db50700 1 -- 172.21.15.16:0/1196919455 >> 172.21.15.16:0/1196919455 conn(0x7efe880d3630 msgr2=0x7efe8815def0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:07.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.241+0000 7efe8db50700 1 -- 172.21.15.16:0/1196919455 shutdown_connections 2023-10-16T13:49:07.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.241+0000 7efe8db50700 1 -- 172.21.15.16:0/1196919455 wait complete. 2023-10-16T13:49:07.248 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-10-16T13:49:07.256 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2023-10-16T13:49:06.587872+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2612,"num_read_kb":5456,"num_write":723,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":441,"ondisk_log_size":441,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":97,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":457184,"kb_used_data":28640,"kb_used_omap":12,"kb_used_meta":428403,"kb_avail":754517536,"statfs":{"total":773094113280,"available":772625956864,"internally_reserved":0,"allocated":29327360,"data_stored":15759988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12740,"internal_metadata":438685244},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.917866"},"pg_stats":[{"pgid":"38.3f","version":"0'0","reported_seq":11,"reported_epoch":278,"state":"active+clean","last_fresh":"2023-10-16T13:49:03.368396+0000","last_change":"2023-10-16T13:49:02.376795+0000","last_active":"2023-10-16T13:49:03.368396+0000","last_peered":"2023-10-16T13:49:03.368396+0000","last_clean":"2023-10-16T13:49:03.368396+0000","last_became_active":"2023-10-16T13:49:02.376522+0000","last_became_peered":"2023-10-16T13:49:02.376522+0000","last_unstale":"2023-10-16T13:49:03.368396+0000","last_undegraded":"2023-10-16T13:49:03.368396+0000","last_fullsized":"2023-10-16T13:49:03.368396+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:50:05.069765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"38.3e","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.383138+0000","last_change":"2023-10-16T13:49:02.382027+0000","last_active":"2023-10-16T13:49:02.383138+0000","last_peered":"2023-10-16T13:49:02.383138+0000","last_clean":"2023-10-16T13:49:02.383138+0000","last_became_active":"2023-10-16T13:49:02.381334+0000","last_became_peered":"2023-10-16T13:49:02.381334+0000","last_unstale":"2023-10-16T13:49:02.383138+0000","last_undegraded":"2023-10-16T13:49:02.383138+0000","last_fullsized":"2023-10-16T13:49:02.383138+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:53:24.110836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.3d","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.381520+0000","last_change":"2023-10-16T13:49:02.380768+0000","last_active":"2023-10-16T13:49:02.381520+0000","last_peered":"2023-10-16T13:49:02.381520+0000","last_clean":"2023-10-16T13:49:02.381520+0000","last_became_active":"2023-10-16T13:49:02.380184+0000","last_became_peered":"2023-10-16T13:49:02.380184+0000","last_unstale":"2023-10-16T13:49:02.381520+0000","last_undegraded":"2023-10-16T13:49:02.381520+0000","last_fullsized":"2023-10-16T13:49:02.381520+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:10:29.903850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.3c","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.389667+0000","last_change":"2023-10-16T13:49:02.386716+0000","last_active":"2023-10-16T13:49:02.389667+0000","last_peered":"2023-10-16T13:49:02.389667+0000","last_clean":"2023-10-16T13:49:02.389667+0000","last_became_active":"2023-10-16T13:49:02.385114+0000","last_became_peered":"2023-10-16T13:49:02.385114+0000","last_unstale":"2023-10-16T13:49:02.389667+0000","last_undegraded":"2023-10-16T13:49:02.389667+0000","last_fullsized":"2023-10-16T13:49:02.389667+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:54:07.794615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.3b","version":"0'0","reported_seq":11,"reported_epoch":278,"state":"active+clean","last_fresh":"2023-10-16T13:49:03.369291+0000","last_change":"2023-10-16T13:49:02.383805+0000","last_active":"2023-10-16T13:49:03.369291+0000","last_peered":"2023-10-16T13:49:03.369291+0000","last_clean":"2023-10-16T13:49:03.369291+0000","last_became_active":"2023-10-16T13:49:02.383237+0000","last_became_peered":"2023-10-16T13:49:02.383237+0000","last_unstale":"2023-10-16T13:49:03.369291+0000","last_undegraded":"2023-10-16T13:49:03.369291+0000","last_fullsized":"2023-10-16T13:49:03.369291+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:05:38.479143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.3a","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.387093+0000","last_change":"2023-10-16T13:49:02.381275+0000","last_active":"2023-10-16T13:49:02.387093+0000","last_peered":"2023-10-16T13:49:02.387093+0000","last_clean":"2023-10-16T13:49:02.387093+0000","last_became_active":"2023-10-16T13:49:02.380948+0000","last_became_peered":"2023-10-16T13:49:02.380948+0000","last_unstale":"2023-10-16T13:49:02.387093+0000","last_undegraded":"2023-10-16T13:49:02.387093+0000","last_fullsized":"2023-10-16T13:49:02.387093+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:21:35.721646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.39","version":"0'0","reported_seq":11,"reported_epoch":278,"state":"active+clean","last_fresh":"2023-10-16T13:49:04.018552+0000","last_change":"2023-10-16T13:49:02.382220+0000","last_active":"2023-10-16T13:49:04.018552+0000","last_peered":"2023-10-16T13:49:04.018552+0000","last_clean":"2023-10-16T13:49:04.018552+0000","last_became_active":"2023-10-16T13:49:02.381865+0000","last_became_peered":"2023-10-16T13:49:02.381865+0000","last_unstale":"2023-10-16T13:49:04.018552+0000","last_undegraded":"2023-10-16T13:49:04.018552+0000","last_fullsized":"2023-10-16T13:49:04.018552+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:52:28.661340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"38.38","version":"0'0","reported_seq":11,"reported_epoch":278,"state":"active+clean","last_fresh":"2023-10-16T13:49:03.368029+0000","last_change":"2023-10-16T13:49:02.375427+0000","last_active":"2023-10-16T13:49:03.368029+0000","last_peered":"2023-10-16T13:49:03.368029+0000","last_clean":"2023-10-16T13:49:03.368029+0000","last_became_active":"2023-10-16T13:49:02.375149+0000","last_became_peered":"2023-10-16T13:49:02.375149+0000","last_unstale":"2023-10-16T13:49:03.368029+0000","last_undegraded":"2023-10-16T13:49:03.368029+0000","last_fullsized":"2023-10-16T13:49:03.368029+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:22:56.794768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"38.37","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.383095+0000","last_change":"2023-10-16T13:49:02.381659+0000","last_active":"2023-10-16T13:49:02.383095+0000","last_peered":"2023-10-16T13:49:02.383095+0000","last_clean":"2023-10-16T13:49:02.383095+0000","last_became_active":"2023-10-16T13:49:02.381334+0000","last_became_peered":"2023-10-16T13:49:02.381334+0000","last_unstale":"2023-10-16T13:49:02.383095+0000","last_undegraded":"2023-10-16T13:49:02.383095+0000","last_fullsized":"2023-10-16T13:49:02.383095+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:14:45.695415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.36","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.383518+0000","last_change":"2023-10-16T13:49:02.376686+0000","last_active":"2023-10-16T13:49:02.383518+0000","last_peered":"2023-10-16T13:49:02.383518+0000","last_clean":"2023-10-16T13:49:02.383518+0000","last_became_active":"2023-10-16T13:49:02.376378+0000","last_became_peered":"2023-10-16T13:49:02.376378+0000","last_unstale":"2023-10-16T13:49:02.383518+0000","last_undegraded":"2023-10-16T13:49:02.383518+0000","last_fullsized":"2023-10-16T13:49:02.383518+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:34:11.700814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.35","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.383714+0000","last_change":"2023-10-16T13:49:02.383342+0000","last_active":"2023-10-16T13:49:02.383714+0000","last_peered":"2023-10-16T13:49:02.383714+0000","last_clean":"2023-10-16T13:49:02.383714+0000","last_became_active":"2023-10-16T13:49:02.383013+0000","last_became_peered":"2023-10-16T13:49:02.383013+0000","last_unstale":"2023-10-16T13:49:02.383714+0000","last_undegraded":"2023-10-16T13:49:02.383714+0000","last_fullsized":"2023-10-16T13:49:02.383714+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:54:02.375814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.34","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.388185+0000","last_change":"2023-10-16T13:49:02.382481+0000","last_active":"2023-10-16T13:49:02.388185+0000","last_peered":"2023-10-16T13:49:02.388185+0000","last_clean":"2023-10-16T13:49:02.388185+0000","last_became_active":"2023-10-16T13:49:02.382177+0000","last_became_peered":"2023-10-16T13:49:02.382177+0000","last_unstale":"2023-10-16T13:49:02.388185+0000","last_undegraded":"2023-10-16T13:49:02.388185+0000","last_fullsized":"2023-10-16T13:49:02.388185+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:36:31.877550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.33","version":"0'0","reported_seq":11,"reported_epoch":278,"state":"active+clean","last_fresh":"2023-10-16T13:49:03.369263+0000","last_change":"2023-10-16T13:49:02.383950+0000","last_active":"2023-10-16T13:49:03.369263+0000","last_peered":"2023-10-16T13:49:03.369263+0000","last_clean":"2023-10-16T13:49:03.369263+0000","last_became_active":"2023-10-16T13:49:02.383639+0000","last_became_peered":"2023-10-16T13:49:02.383639+0000","last_unstale":"2023-10-16T13:49:03.369263+0000","last_undegraded":"2023-10-16T13:49:03.369263+0000","last_fullsized":"2023-10-16T13:49:03.369263+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:34:53.576041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.32","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.377946+0000","last_change":"2023-10-16T13:49:02.374842+0000","last_active":"2023-10-16T13:49:02.377946+0000","last_peered":"2023-10-16T13:49:02.377946+0000","last_clean":"2023-10-16T13:49:02.377946+0000","last_became_active":"2023-10-16T13:49:02.374581+0000","last_became_peered":"2023-10-16T13:49:02.374581+0000","last_unstale":"2023-10-16T13:49:02.377946+0000","last_undegraded":"2023-10-16T13:49:02.377946+0000","last_fullsized":"2023-10-16T13:49:02.377946+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:18:58.819910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.31","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.378454+0000","last_change":"2023-10-16T13:49:02.372490+0000","last_active":"2023-10-16T13:49:02.378454+0000","last_peered":"2023-10-16T13:49:02.378454+0000","last_clean":"2023-10-16T13:49:02.378454+0000","last_became_active":"2023-10-16T13:49:02.372207+0000","last_became_peered":"2023-10-16T13:49:02.372207+0000","last_unstale":"2023-10-16T13:49:02.378454+0000","last_undegraded":"2023-10-16T13:49:02.378454+0000","last_fullsized":"2023-10-16T13:49:02.378454+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:28:18.350036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.30","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.389880+0000","last_change":"2023-10-16T13:49:02.388914+0000","last_active":"2023-10-16T13:49:02.389880+0000","last_peered":"2023-10-16T13:49:02.389880+0000","last_clean":"2023-10-16T13:49:02.389880+0000","last_became_active":"2023-10-16T13:49:02.388639+0000","last_became_peered":"2023-10-16T13:49:02.388639+0000","last_unstale":"2023-10-16T13:49:02.389880+0000","last_undegraded":"2023-10-16T13:49:02.389880+0000","last_fullsized":"2023-10-16T13:49:02.389880+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:46:30.743146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.2f","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.383076+0000","last_change":"2023-10-16T13:49:02.381169+0000","last_active":"2023-10-16T13:49:02.383076+0000","last_peered":"2023-10-16T13:49:02.383076+0000","last_clean":"2023-10-16T13:49:02.383076+0000","last_became_active":"2023-10-16T13:49:02.380828+0000","last_became_peered":"2023-10-16T13:49:02.380828+0000","last_unstale":"2023-10-16T13:49:02.383076+0000","last_undegraded":"2023-10-16T13:49:02.383076+0000","last_fullsized":"2023-10-16T13:49:02.383076+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:28:05.517649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.2e","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.385074+0000","last_change":"2023-10-16T13:49:02.381165+0000","last_active":"2023-10-16T13:49:02.385074+0000","last_peered":"2023-10-16T13:49:02.385074+0000","last_clean":"2023-10-16T13:49:02.385074+0000","last_became_active":"2023-10-16T13:49:02.380863+0000","last_became_peered":"2023-10-16T13:49:02.380863+0000","last_unstale":"2023-10-16T13:49:02.385074+0000","last_undegraded":"2023-10-16T13:49:02.385074+0000","last_fullsized":"2023-10-16T13:49:02.385074+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:35:41.847423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.2d","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.381541+0000","last_change":"2023-10-16T13:49:02.380914+0000","last_active":"2023-10-16T13:49:02.381541+0000","last_peered":"2023-10-16T13:49:02.381541+0000","last_clean":"2023-10-16T13:49:02.381541+0000","last_became_active":"2023-10-16T13:49:02.380506+0000","last_became_peered":"2023-10-16T13:49:02.380506+0000","last_unstale":"2023-10-16T13:49:02.381541+0000","last_undegraded":"2023-10-16T13:49:02.381541+0000","last_fullsized":"2023-10-16T13:49:02.381541+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:22:59.695958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.2c","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.385420+0000","last_change":"2023-10-16T13:49:02.380258+0000","last_active":"2023-10-16T13:49:02.385420+0000","last_peered":"2023-10-16T13:49:02.385420+0000","last_clean":"2023-10-16T13:49:02.385420+0000","last_became_active":"2023-10-16T13:49:02.379919+0000","last_became_peered":"2023-10-16T13:49:02.379919+0000","last_unstale":"2023-10-16T13:49:02.385420+0000","last_undegraded":"2023-10-16T13:49:02.385420+0000","last_fullsized":"2023-10-16T13:49:02.385420+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:24:53.537429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.2b","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.386039+0000","last_change":"2023-10-16T13:49:02.385684+0000","last_active":"2023-10-16T13:49:02.386039+0000","last_peered":"2023-10-16T13:49:02.386039+0000","last_clean":"2023-10-16T13:49:02.386039+0000","last_became_active":"2023-10-16T13:49:02.385402+0000","last_became_peered":"2023-10-16T13:49:02.385402+0000","last_unstale":"2023-10-16T13:49:02.386039+0000","last_undegraded":"2023-10-16T13:49:02.386039+0000","last_fullsized":"2023-10-16T13:49:02.386039+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:16:22.980195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.2a","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.389617+0000","last_change":"2023-10-16T13:49:02.386598+0000","last_active":"2023-10-16T13:49:02.389617+0000","last_peered":"2023-10-16T13:49:02.389617+0000","last_clean":"2023-10-16T13:49:02.389617+0000","last_became_active":"2023-10-16T13:49:02.386284+0000","last_became_peered":"2023-10-16T13:49:02.386284+0000","last_unstale":"2023-10-16T13:49:02.389617+0000","last_undegraded":"2023-10-16T13:49:02.389617+0000","last_fullsized":"2023-10-16T13:49:02.389617+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:53:24.110836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.29","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.389972+0000","last_change":"2023-10-16T13:49:02.387731+0000","last_active":"2023-10-16T13:49:02.389972+0000","last_peered":"2023-10-16T13:49:02.389972+0000","last_clean":"2023-10-16T13:49:02.389972+0000","last_became_active":"2023-10-16T13:49:02.384828+0000","last_became_peered":"2023-10-16T13:49:02.384828+0000","last_unstale":"2023-10-16T13:49:02.389972+0000","last_undegraded":"2023-10-16T13:49:02.389972+0000","last_fullsized":"2023-10-16T13:49:02.389972+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:28:18.350036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.28","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.385335+0000","last_change":"2023-10-16T13:49:02.384443+0000","last_active":"2023-10-16T13:49:02.385335+0000","last_peered":"2023-10-16T13:49:02.385335+0000","last_clean":"2023-10-16T13:49:02.385335+0000","last_became_active":"2023-10-16T13:49:02.384174+0000","last_became_peered":"2023-10-16T13:49:02.384174+0000","last_unstale":"2023-10-16T13:49:02.385335+0000","last_undegraded":"2023-10-16T13:49:02.385335+0000","last_fullsized":"2023-10-16T13:49:02.385335+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:04:56.643213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.26","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.376229+0000","last_change":"2023-10-16T13:49:02.375235+0000","last_active":"2023-10-16T13:49:02.376229+0000","last_peered":"2023-10-16T13:49:02.376229+0000","last_clean":"2023-10-16T13:49:02.376229+0000","last_became_active":"2023-10-16T13:49:02.374449+0000","last_became_peered":"2023-10-16T13:49:02.374449+0000","last_unstale":"2023-10-16T13:49:02.376229+0000","last_undegraded":"2023-10-16T13:49:02.376229+0000","last_fullsized":"2023-10-16T13:49:02.376229+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:23:43.076430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.25","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.389622+0000","last_change":"2023-10-16T13:49:02.386164+0000","last_active":"2023-10-16T13:49:02.389622+0000","last_peered":"2023-10-16T13:49:02.389622+0000","last_clean":"2023-10-16T13:49:02.389622+0000","last_became_active":"2023-10-16T13:49:02.385195+0000","last_became_peered":"2023-10-16T13:49:02.385195+0000","last_unstale":"2023-10-16T13:49:02.389622+0000","last_undegraded":"2023-10-16T13:49:02.389622+0000","last_fullsized":"2023-10-16T13:49:02.389622+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:04:56.643213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.24","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.370820+0000","last_change":"2023-10-16T13:49:02.367826+0000","last_active":"2023-10-16T13:49:02.370820+0000","last_peered":"2023-10-16T13:49:02.370820+0000","last_clean":"2023-10-16T13:49:02.370820+0000","last_became_active":"2023-10-16T13:49:02.367551+0000","last_became_peered":"2023-10-16T13:49:02.367551+0000","last_unstale":"2023-10-16T13:49:02.370820+0000","last_undegraded":"2023-10-16T13:49:02.370820+0000","last_fullsized":"2023-10-16T13:49:02.370820+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:45:05.599847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.23","version":"0'0","reported_seq":11,"reported_epoch":278,"state":"active+clean","last_fresh":"2023-10-16T13:49:04.018862+0000","last_change":"2023-10-16T13:49:02.386716+0000","last_active":"2023-10-16T13:49:04.018862+0000","last_peered":"2023-10-16T13:49:04.018862+0000","last_clean":"2023-10-16T13:49:04.018862+0000","last_became_active":"2023-10-16T13:49:02.386441+0000","last_became_peered":"2023-10-16T13:49:02.386441+0000","last_unstale":"2023-10-16T13:49:04.018862+0000","last_undegraded":"2023-10-16T13:49:04.018862+0000","last_fullsized":"2023-10-16T13:49:04.018862+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:18:28.824765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"38.22","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.377926+0000","last_change":"2023-10-16T13:49:02.373505+0000","last_active":"2023-10-16T13:49:02.377926+0000","last_peered":"2023-10-16T13:49:02.377926+0000","last_clean":"2023-10-16T13:49:02.377926+0000","last_became_active":"2023-10-16T13:49:02.372543+0000","last_became_peered":"2023-10-16T13:49:02.372543+0000","last_unstale":"2023-10-16T13:49:02.377926+0000","last_undegraded":"2023-10-16T13:49:02.377926+0000","last_fullsized":"2023-10-16T13:49:02.377926+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:22:59.695958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.21","version":"0'0","reported_seq":11,"reported_epoch":278,"state":"active+clean","last_fresh":"2023-10-16T13:49:04.018646+0000","last_change":"2023-10-16T13:49:02.372720+0000","last_active":"2023-10-16T13:49:04.018646+0000","last_peered":"2023-10-16T13:49:04.018646+0000","last_clean":"2023-10-16T13:49:04.018646+0000","last_became_active":"2023-10-16T13:49:02.372410+0000","last_became_peered":"2023-10-16T13:49:02.372410+0000","last_unstale":"2023-10-16T13:49:04.018646+0000","last_undegraded":"2023-10-16T13:49:04.018646+0000","last_fullsized":"2023-10-16T13:49:04.018646+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:00:48.018874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"38.20","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.390003+0000","last_change":"2023-10-16T13:49:02.387734+0000","last_active":"2023-10-16T13:49:02.390003+0000","last_peered":"2023-10-16T13:49:02.390003+0000","last_clean":"2023-10-16T13:49:02.390003+0000","last_became_active":"2023-10-16T13:49:02.384712+0000","last_became_peered":"2023-10-16T13:49:02.384712+0000","last_unstale":"2023-10-16T13:49:02.390003+0000","last_undegraded":"2023-10-16T13:49:02.390003+0000","last_fullsized":"2023-10-16T13:49:02.390003+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:10:29.903850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.1c","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.385366+0000","last_change":"2023-10-16T13:49:02.380193+0000","last_active":"2023-10-16T13:49:02.385366+0000","last_peered":"2023-10-16T13:49:02.385366+0000","last_clean":"2023-10-16T13:49:02.385366+0000","last_became_active":"2023-10-16T13:49:02.379874+0000","last_became_peered":"2023-10-16T13:49:02.379874+0000","last_unstale":"2023-10-16T13:49:02.385366+0000","last_undegraded":"2023-10-16T13:49:02.385366+0000","last_fullsized":"2023-10-16T13:49:02.385366+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:07:27.569705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.1f","version":"277'4","reported_seq":16,"reported_epoch":278,"state":"active+clean","last_fresh":"2023-10-16T13:49:03.369129+0000","last_change":"2023-10-16T13:49:01.695903+0000","last_active":"2023-10-16T13:49:03.369129+0000","last_peered":"2023-10-16T13:49:03.369129+0000","last_clean":"2023-10-16T13:49:03.369129+0000","last_became_active":"2023-10-16T13:49:01.695597+0000","last_became_peered":"2023-10-16T13:49:01.695597+0000","last_unstale":"2023-10-16T13:49:03.369129+0000","last_undegraded":"2023-10-16T13:49:03.369129+0000","last_fullsized":"2023-10-16T13:49:03.369129+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_clean_scrub_stamp":"2023-10-16T13:49:00.664589+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:28:57.441027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.1d","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.383733+0000","last_change":"2023-10-16T13:49:02.383375+0000","last_active":"2023-10-16T13:49:02.383733+0000","last_peered":"2023-10-16T13:49:02.383733+0000","last_clean":"2023-10-16T13:49:02.383733+0000","last_became_active":"2023-10-16T13:49:02.383067+0000","last_became_peered":"2023-10-16T13:49:02.383067+0000","last_unstale":"2023-10-16T13:49:02.383733+0000","last_undegraded":"2023-10-16T13:49:02.383733+0000","last_fullsized":"2023-10-16T13:49:02.383733+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:45:39.960993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.1e","version":"277'1","reported_seq":12,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.702137+0000","last_change":"2023-10-16T13:49:01.693140+0000","last_active":"2023-10-16T13:49:02.702137+0000","last_peered":"2023-10-16T13:49:02.702137+0000","last_clean":"2023-10-16T13:49:02.702137+0000","last_became_active":"2023-10-16T13:49:01.692859+0000","last_became_peered":"2023-10-16T13:49:01.692859+0000","last_unstale":"2023-10-16T13:49:02.702137+0000","last_undegraded":"2023-10-16T13:49:02.702137+0000","last_fullsized":"2023-10-16T13:49:02.702137+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_clean_scrub_stamp":"2023-10-16T13:49:00.664589+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:41:52.860750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.1e","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.378807+0000","last_change":"2023-10-16T13:49:02.377260+0000","last_active":"2023-10-16T13:49:02.378807+0000","last_peered":"2023-10-16T13:49:02.378807+0000","last_clean":"2023-10-16T13:49:02.378807+0000","last_became_active":"2023-10-16T13:49:02.375414+0000","last_became_peered":"2023-10-16T13:49:02.375414+0000","last_unstale":"2023-10-16T13:49:02.378807+0000","last_undegraded":"2023-10-16T13:49:02.378807+0000","last_fullsized":"2023-10-16T13:49:02.378807+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:47:14.454827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.1d","version":"277'2","reported_seq":13,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.706232+0000","last_change":"2023-10-16T13:49:01.689030+0000","last_active":"2023-10-16T13:49:02.706232+0000","last_peered":"2023-10-16T13:49:02.706232+0000","last_clean":"2023-10-16T13:49:02.706232+0000","last_became_active":"2023-10-16T13:49:01.688698+0000","last_became_peered":"2023-10-16T13:49:01.688698+0000","last_unstale":"2023-10-16T13:49:02.706232+0000","last_undegraded":"2023-10-16T13:49:02.706232+0000","last_fullsized":"2023-10-16T13:49:02.706232+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_clean_scrub_stamp":"2023-10-16T13:49:00.664589+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:56:42.829574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.1f","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.372403+0000","last_change":"2023-10-16T13:49:02.370312+0000","last_active":"2023-10-16T13:49:02.372403+0000","last_peered":"2023-10-16T13:49:02.372403+0000","last_clean":"2023-10-16T13:49:02.372403+0000","last_became_active":"2023-10-16T13:49:02.370044+0000","last_became_peered":"2023-10-16T13:49:02.370044+0000","last_unstale":"2023-10-16T13:49:02.372403+0000","last_undegraded":"2023-10-16T13:49:02.372403+0000","last_fullsized":"2023-10-16T13:49:02.372403+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:35:41.847423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.1c","version":"277'2","reported_seq":13,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.707775+0000","last_change":"2023-10-16T13:49:01.689254+0000","last_active":"2023-10-16T13:49:02.707775+0000","last_peered":"2023-10-16T13:49:02.707775+0000","last_clean":"2023-10-16T13:49:02.707775+0000","last_became_active":"2023-10-16T13:49:01.688292+0000","last_became_peered":"2023-10-16T13:49:01.688292+0000","last_unstale":"2023-10-16T13:49:02.707775+0000","last_undegraded":"2023-10-16T13:49:02.707775+0000","last_fullsized":"2023-10-16T13:49:02.707775+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_clean_scrub_stamp":"2023-10-16T13:49:00.664589+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:52:52.702555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.18","version":"0'0","reported_seq":11,"reported_epoch":278,"state":"active+clean","last_fresh":"2023-10-16T13:49:03.368088+0000","last_change":"2023-10-16T13:49:02.377042+0000","last_active":"2023-10-16T13:49:03.368088+0000","last_peered":"2023-10-16T13:49:03.368088+0000","last_clean":"2023-10-16T13:49:03.368088+0000","last_became_active":"2023-10-16T13:49:02.376705+0000","last_became_peered":"2023-10-16T13:49:02.376705+0000","last_unstale":"2023-10-16T13:49:03.368088+0000","last_undegraded":"2023-10-16T13:49:03.368088+0000","last_fullsized":"2023-10-16T13:49:03.368088+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:22:59.348010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.1b","version":"277'1","reported_seq":12,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.704320+0000","last_change":"2023-10-16T13:49:01.686024+0000","last_active":"2023-10-16T13:49:02.704320+0000","last_peered":"2023-10-16T13:49:02.704320+0000","last_clean":"2023-10-16T13:49:02.704320+0000","last_became_active":"2023-10-16T13:49:01.679420+0000","last_became_peered":"2023-10-16T13:49:01.679420+0000","last_unstale":"2023-10-16T13:49:02.704320+0000","last_undegraded":"2023-10-16T13:49:02.704320+0000","last_fullsized":"2023-10-16T13:49:02.704320+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_clean_scrub_stamp":"2023-10-16T13:49:00.664589+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:00:07.114500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.19","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.379600+0000","last_change":"2023-10-16T13:49:02.378876+0000","last_active":"2023-10-16T13:49:02.379600+0000","last_peered":"2023-10-16T13:49:02.379600+0000","last_clean":"2023-10-16T13:49:02.379600+0000","last_became_active":"2023-10-16T13:49:02.378413+0000","last_became_peered":"2023-10-16T13:49:02.378413+0000","last_unstale":"2023-10-16T13:49:02.379600+0000","last_undegraded":"2023-10-16T13:49:02.379600+0000","last_fullsized":"2023-10-16T13:49:02.379600+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:50:22.311928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.1a","version":"277'1","reported_seq":12,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.707379+0000","last_change":"2023-10-16T13:49:01.691725+0000","last_active":"2023-10-16T13:49:02.707379+0000","last_peered":"2023-10-16T13:49:02.707379+0000","last_clean":"2023-10-16T13:49:02.707379+0000","last_became_active":"2023-10-16T13:49:01.691431+0000","last_became_peered":"2023-10-16T13:49:01.691431+0000","last_unstale":"2023-10-16T13:49:02.707379+0000","last_undegraded":"2023-10-16T13:49:02.707379+0000","last_fullsized":"2023-10-16T13:49:02.707379+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_clean_scrub_stamp":"2023-10-16T13:49:00.664589+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:53:23.105788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.1a","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.389592+0000","last_change":"2023-10-16T13:49:02.386330+0000","last_active":"2023-10-16T13:49:02.389592+0000","last_peered":"2023-10-16T13:49:02.389592+0000","last_clean":"2023-10-16T13:49:02.389592+0000","last_became_active":"2023-10-16T13:49:02.385896+0000","last_became_peered":"2023-10-16T13:49:02.385896+0000","last_unstale":"2023-10-16T13:49:02.389592+0000","last_undegraded":"2023-10-16T13:49:02.389592+0000","last_fullsized":"2023-10-16T13:49:02.389592+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:29:32.248558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.19","version":"0'0","reported_seq":11,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.369623+0000","last_change":"2023-10-16T13:49:01.692662+0000","last_active":"2023-10-16T13:49:02.369623+0000","last_peered":"2023-10-16T13:49:02.369623+0000","last_clean":"2023-10-16T13:49:02.369623+0000","last_became_active":"2023-10-16T13:49:01.691965+0000","last_became_peered":"2023-10-16T13:49:01.691965+0000","last_unstale":"2023-10-16T13:49:02.369623+0000","last_undegraded":"2023-10-16T13:49:02.369623+0000","last_fullsized":"2023-10-16T13:49:02.369623+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_clean_scrub_stamp":"2023-10-16T13:49:00.664589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:48:17.860543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.1b","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.383124+0000","last_change":"2023-10-16T13:49:02.382596+0000","last_active":"2023-10-16T13:49:02.383124+0000","last_peered":"2023-10-16T13:49:02.383124+0000","last_clean":"2023-10-16T13:49:02.383124+0000","last_became_active":"2023-10-16T13:49:02.382300+0000","last_became_peered":"2023-10-16T13:49:02.382300+0000","last_unstale":"2023-10-16T13:49:02.383124+0000","last_undegraded":"2023-10-16T13:49:02.383124+0000","last_fullsized":"2023-10-16T13:49:02.383124+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:29:26.019903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.18","version":"0'0","reported_seq":12,"reported_epoch":278,"state":"active+clean","last_fresh":"2023-10-16T13:49:04.018470+0000","last_change":"2023-10-16T13:49:01.693611+0000","last_active":"2023-10-16T13:49:04.018470+0000","last_peered":"2023-10-16T13:49:04.018470+0000","last_clean":"2023-10-16T13:49:04.018470+0000","last_became_active":"2023-10-16T13:49:01.693015+0000","last_became_peered":"2023-10-16T13:49:01.693015+0000","last_unstale":"2023-10-16T13:49:04.018470+0000","last_undegraded":"2023-10-16T13:49:04.018470+0000","last_fullsized":"2023-10-16T13:49:04.018470+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_clean_scrub_stamp":"2023-10-16T13:49:00.664589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:33:09.963008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"38.14","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.385345+0000","last_change":"2023-10-16T13:49:02.378931+0000","last_active":"2023-10-16T13:49:02.385345+0000","last_peered":"2023-10-16T13:49:02.385345+0000","last_clean":"2023-10-16T13:49:02.385345+0000","last_became_active":"2023-10-16T13:49:02.378025+0000","last_became_peered":"2023-10-16T13:49:02.378025+0000","last_unstale":"2023-10-16T13:49:02.385345+0000","last_undegraded":"2023-10-16T13:49:02.385345+0000","last_fullsized":"2023-10-16T13:49:02.385345+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:17:26.735536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.17","version":"0'0","reported_seq":12,"reported_epoch":278,"state":"active+clean","last_fresh":"2023-10-16T13:49:03.369239+0000","last_change":"2023-10-16T13:49:01.688691+0000","last_active":"2023-10-16T13:49:03.369239+0000","last_peered":"2023-10-16T13:49:03.369239+0000","last_clean":"2023-10-16T13:49:03.369239+0000","last_became_active":"2023-10-16T13:49:01.688376+0000","last_became_peered":"2023-10-16T13:49:01.688376+0000","last_unstale":"2023-10-16T13:49:03.369239+0000","last_undegraded":"2023-10-16T13:49:03.369239+0000","last_fullsized":"2023-10-16T13:49:03.369239+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_clean_scrub_stamp":"2023-10-16T13:49:00.664589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:06:05.911525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.15","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.384081+0000","last_change":"2023-10-16T13:49:02.379036+0000","last_active":"2023-10-16T13:49:02.384081+0000","last_peered":"2023-10-16T13:49:02.384081+0000","last_clean":"2023-10-16T13:49:02.384081+0000","last_became_active":"2023-10-16T13:49:02.378723+0000","last_became_peered":"2023-10-16T13:49:02.378723+0000","last_unstale":"2023-10-16T13:49:02.384081+0000","last_undegraded":"2023-10-16T13:49:02.384081+0000","last_fullsized":"2023-10-16T13:49:02.384081+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:11:46.354324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.16","version":"0'0","reported_seq":11,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.369750+0000","last_change":"2023-10-16T13:49:01.692624+0000","last_active":"2023-10-16T13:49:02.369750+0000","last_peered":"2023-10-16T13:49:02.369750+0000","last_clean":"2023-10-16T13:49:02.369750+0000","last_became_active":"2023-10-16T13:49:01.692325+0000","last_became_peered":"2023-10-16T13:49:01.692325+0000","last_unstale":"2023-10-16T13:49:02.369750+0000","last_undegraded":"2023-10-16T13:49:02.369750+0000","last_fullsized":"2023-10-16T13:49:02.369750+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_clean_scrub_stamp":"2023-10-16T13:49:00.664589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:16:51.291591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.16","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.378788+0000","last_change":"2023-10-16T13:49:02.375471+0000","last_active":"2023-10-16T13:49:02.378788+0000","last_peered":"2023-10-16T13:49:02.378788+0000","last_clean":"2023-10-16T13:49:02.378788+0000","last_became_active":"2023-10-16T13:49:02.375146+0000","last_became_peered":"2023-10-16T13:49:02.375146+0000","last_unstale":"2023-10-16T13:49:02.378788+0000","last_undegraded":"2023-10-16T13:49:02.378788+0000","last_fullsized":"2023-10-16T13:49:02.378788+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:56:43.834622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.15","version":"277'3","reported_seq":14,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.704969+0000","last_change":"2023-10-16T13:49:01.691397+0000","last_active":"2023-10-16T13:49:02.704969+0000","last_peered":"2023-10-16T13:49:02.704969+0000","last_clean":"2023-10-16T13:49:02.704969+0000","last_became_active":"2023-10-16T13:49:01.691101+0000","last_became_peered":"2023-10-16T13:49:01.691101+0000","last_unstale":"2023-10-16T13:49:02.704969+0000","last_undegraded":"2023-10-16T13:49:02.704969+0000","last_fullsized":"2023-10-16T13:49:02.704969+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_clean_scrub_stamp":"2023-10-16T13:49:00.664589+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:33:58.144888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.17","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.384927+0000","last_change":"2023-10-16T13:49:02.380011+0000","last_active":"2023-10-16T13:49:02.384927+0000","last_peered":"2023-10-16T13:49:02.384927+0000","last_clean":"2023-10-16T13:49:02.384927+0000","last_became_active":"2023-10-16T13:49:02.379696+0000","last_became_peered":"2023-10-16T13:49:02.379696+0000","last_unstale":"2023-10-16T13:49:02.384927+0000","last_undegraded":"2023-10-16T13:49:02.384927+0000","last_fullsized":"2023-10-16T13:49:02.384927+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:44:52.683527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.14","version":"277'6","reported_seq":18,"reported_epoch":278,"state":"active+clean","last_fresh":"2023-10-16T13:49:03.368319+0000","last_change":"2023-10-16T13:49:01.690187+0000","last_active":"2023-10-16T13:49:03.368319+0000","last_peered":"2023-10-16T13:49:03.368319+0000","last_clean":"2023-10-16T13:49:03.368319+0000","last_became_active":"2023-10-16T13:49:01.689795+0000","last_became_peered":"2023-10-16T13:49:01.689795+0000","last_unstale":"2023-10-16T13:49:03.368319+0000","last_undegraded":"2023-10-16T13:49:03.368319+0000","last_fullsized":"2023-10-16T13:49:03.368319+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_clean_scrub_stamp":"2023-10-16T13:49:00.664589+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:38:55.796167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"38.10","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.379048+0000","last_change":"2023-10-16T13:49:02.376913+0000","last_active":"2023-10-16T13:49:02.379048+0000","last_peered":"2023-10-16T13:49:02.379048+0000","last_clean":"2023-10-16T13:49:02.379048+0000","last_became_active":"2023-10-16T13:49:02.376017+0000","last_became_peered":"2023-10-16T13:49:02.376017+0000","last_unstale":"2023-10-16T13:49:02.379048+0000","last_undegraded":"2023-10-16T13:49:02.379048+0000","last_fullsized":"2023-10-16T13:49:02.379048+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:52:47.764832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.13","version":"277'2","reported_seq":13,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.706684+0000","last_change":"2023-10-16T13:49:01.687745+0000","last_active":"2023-10-16T13:49:02.706684+0000","last_peered":"2023-10-16T13:49:02.706684+0000","last_clean":"2023-10-16T13:49:02.706684+0000","last_became_active":"2023-10-16T13:49:01.687411+0000","last_became_peered":"2023-10-16T13:49:01.687411+0000","last_unstale":"2023-10-16T13:49:02.706684+0000","last_undegraded":"2023-10-16T13:49:02.706684+0000","last_fullsized":"2023-10-16T13:49:02.706684+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_clean_scrub_stamp":"2023-10-16T13:49:00.664589+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:12:52.526868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.11","version":"0'0","reported_seq":11,"reported_epoch":278,"state":"active+clean","last_fresh":"2023-10-16T13:49:03.369397+0000","last_change":"2023-10-16T13:49:02.384951+0000","last_active":"2023-10-16T13:49:03.369397+0000","last_peered":"2023-10-16T13:49:03.369397+0000","last_clean":"2023-10-16T13:49:03.369397+0000","last_became_active":"2023-10-16T13:49:02.384592+0000","last_became_peered":"2023-10-16T13:49:02.384592+0000","last_unstale":"2023-10-16T13:49:03.369397+0000","last_undegraded":"2023-10-16T13:49:03.369397+0000","last_fullsized":"2023-10-16T13:49:03.369397+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:18:18.047752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.12","version":"0'0","reported_seq":12,"reported_epoch":278,"state":"active+clean","last_fresh":"2023-10-16T13:49:03.368978+0000","last_change":"2023-10-16T13:49:01.696834+0000","last_active":"2023-10-16T13:49:03.368978+0000","last_peered":"2023-10-16T13:49:03.368978+0000","last_clean":"2023-10-16T13:49:03.368978+0000","last_became_active":"2023-10-16T13:49:01.696555+0000","last_became_peered":"2023-10-16T13:49:01.696555+0000","last_unstale":"2023-10-16T13:49:03.368978+0000","last_undegraded":"2023-10-16T13:49:03.368978+0000","last_fullsized":"2023-10-16T13:49:03.368978+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_clean_scrub_stamp":"2023-10-16T13:49:00.664589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:29:40.250736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.12","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.386489+0000","last_change":"2023-10-16T13:49:02.381459+0000","last_active":"2023-10-16T13:49:02.386489+0000","last_peered":"2023-10-16T13:49:02.386489+0000","last_clean":"2023-10-16T13:49:02.386489+0000","last_became_active":"2023-10-16T13:49:02.381184+0000","last_became_peered":"2023-10-16T13:49:02.381184+0000","last_unstale":"2023-10-16T13:49:02.386489+0000","last_undegraded":"2023-10-16T13:49:02.386489+0000","last_fullsized":"2023-10-16T13:49:02.386489+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:22:59.695958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.11","version":"0'0","reported_seq":11,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.368180+0000","last_change":"2023-10-16T13:49:01.686038+0000","last_active":"2023-10-16T13:49:02.368180+0000","last_peered":"2023-10-16T13:49:02.368180+0000","last_clean":"2023-10-16T13:49:02.368180+0000","last_became_active":"2023-10-16T13:49:01.685753+0000","last_became_peered":"2023-10-16T13:49:01.685753+0000","last_unstale":"2023-10-16T13:49:02.368180+0000","last_undegraded":"2023-10-16T13:49:02.368180+0000","last_fullsized":"2023-10-16T13:49:02.368180+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_clean_scrub_stamp":"2023-10-16T13:49:00.664589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:11:41.402056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.13","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.388078+0000","last_change":"2023-10-16T13:49:02.382861+0000","last_active":"2023-10-16T13:49:02.388078+0000","last_peered":"2023-10-16T13:49:02.388078+0000","last_clean":"2023-10-16T13:49:02.388078+0000","last_became_active":"2023-10-16T13:49:02.382543+0000","last_became_peered":"2023-10-16T13:49:02.382543+0000","last_unstale":"2023-10-16T13:49:02.388078+0000","last_undegraded":"2023-10-16T13:49:02.388078+0000","last_fullsized":"2023-10-16T13:49:02.388078+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:15:09.398807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.10","version":"277'8","reported_seq":19,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.710134+0000","last_change":"2023-10-16T13:49:01.693876+0000","last_active":"2023-10-16T13:49:02.710134+0000","last_peered":"2023-10-16T13:49:02.710134+0000","last_clean":"2023-10-16T13:49:02.710134+0000","last_became_active":"2023-10-16T13:49:01.693605+0000","last_became_peered":"2023-10-16T13:49:01.693605+0000","last_unstale":"2023-10-16T13:49:02.710134+0000","last_undegraded":"2023-10-16T13:49:02.710134+0000","last_fullsized":"2023-10-16T13:49:02.710134+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_clean_scrub_stamp":"2023-10-16T13:49:00.664589+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T13:56:42.829574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.c","version":"0'0","reported_seq":11,"reported_epoch":278,"state":"active+clean","last_fresh":"2023-10-16T13:49:03.368384+0000","last_change":"2023-10-16T13:49:02.370520+0000","last_active":"2023-10-16T13:49:03.368384+0000","last_peered":"2023-10-16T13:49:03.368384+0000","last_clean":"2023-10-16T13:49:03.368384+0000","last_became_active":"2023-10-16T13:49:02.370062+0000","last_became_peered":"2023-10-16T13:49:02.370062+0000","last_unstale":"2023-10-16T13:49:03.368384+0000","last_undegraded":"2023-10-16T13:49:03.368384+0000","last_fullsized":"2023-10-16T13:49:03.368384+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:25:26.350475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.f","version":"277'1","reported_seq":12,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.703708+0000","last_change":"2023-10-16T13:49:01.695983+0000","last_active":"2023-10-16T13:49:02.703708+0000","last_peered":"2023-10-16T13:49:02.703708+0000","last_clean":"2023-10-16T13:49:02.703708+0000","last_became_active":"2023-10-16T13:49:01.695672+0000","last_became_peered":"2023-10-16T13:49:01.695672+0000","last_unstale":"2023-10-16T13:49:02.703708+0000","last_undegraded":"2023-10-16T13:49:02.703708+0000","last_fullsized":"2023-10-16T13:49:02.703708+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_clean_scrub_stamp":"2023-10-16T13:49:00.664589+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:45:09.678272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.d","version":"0'0","reported_seq":11,"reported_epoch":278,"state":"active+clean","last_fresh":"2023-10-16T13:49:03.369086+0000","last_change":"2023-10-16T13:49:02.378377+0000","last_active":"2023-10-16T13:49:03.369086+0000","last_peered":"2023-10-16T13:49:03.369086+0000","last_clean":"2023-10-16T13:49:03.369086+0000","last_became_active":"2023-10-16T13:49:02.378068+0000","last_became_peered":"2023-10-16T13:49:02.378068+0000","last_unstale":"2023-10-16T13:49:03.369086+0000","last_undegraded":"2023-10-16T13:49:03.369086+0000","last_fullsized":"2023-10-16T13:49:03.369086+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:43:26.335779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.e","version":"277'2","reported_seq":14,"reported_epoch":278,"state":"active+clean","last_fresh":"2023-10-16T13:49:03.368434+0000","last_change":"2023-10-16T13:49:01.695615+0000","last_active":"2023-10-16T13:49:03.368434+0000","last_peered":"2023-10-16T13:49:03.368434+0000","last_clean":"2023-10-16T13:49:03.368434+0000","last_became_active":"2023-10-16T13:49:01.695339+0000","last_became_peered":"2023-10-16T13:49:01.695339+0000","last_unstale":"2023-10-16T13:49:03.368434+0000","last_undegraded":"2023-10-16T13:49:03.368434+0000","last_fullsized":"2023-10-16T13:49:03.368434+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_clean_scrub_stamp":"2023-10-16T13:49:00.664589+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:13:22.191652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"38.e","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.387984+0000","last_change":"2023-10-16T13:49:02.381744+0000","last_active":"2023-10-16T13:49:02.387984+0000","last_peered":"2023-10-16T13:49:02.387984+0000","last_clean":"2023-10-16T13:49:02.387984+0000","last_became_active":"2023-10-16T13:49:02.381458+0000","last_became_peered":"2023-10-16T13:49:02.381458+0000","last_unstale":"2023-10-16T13:49:02.387984+0000","last_undegraded":"2023-10-16T13:49:02.387984+0000","last_fullsized":"2023-10-16T13:49:02.387984+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:31:19.406244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.d","version":"277'1","reported_seq":13,"reported_epoch":278,"state":"active+clean","last_fresh":"2023-10-16T13:49:03.369315+0000","last_change":"2023-10-16T13:49:01.687946+0000","last_active":"2023-10-16T13:49:03.369315+0000","last_peered":"2023-10-16T13:49:03.369315+0000","last_clean":"2023-10-16T13:49:03.369315+0000","last_became_active":"2023-10-16T13:49:01.687301+0000","last_became_peered":"2023-10-16T13:49:01.687301+0000","last_unstale":"2023-10-16T13:49:03.369315+0000","last_undegraded":"2023-10-16T13:49:03.369315+0000","last_fullsized":"2023-10-16T13:49:03.369315+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_clean_scrub_stamp":"2023-10-16T13:49:00.664589+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:45:32.313189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.f","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.385897+0000","last_change":"2023-10-16T13:49:02.385029+0000","last_active":"2023-10-16T13:49:02.385897+0000","last_peered":"2023-10-16T13:49:02.385897+0000","last_clean":"2023-10-16T13:49:02.385897+0000","last_became_active":"2023-10-16T13:49:02.384684+0000","last_became_peered":"2023-10-16T13:49:02.384684+0000","last_unstale":"2023-10-16T13:49:02.385897+0000","last_undegraded":"2023-10-16T13:49:02.385897+0000","last_fullsized":"2023-10-16T13:49:02.385897+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:17:26.735536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.c","version":"0'0","reported_seq":11,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.368494+0000","last_change":"2023-10-16T13:49:01.693806+0000","last_active":"2023-10-16T13:49:02.368494+0000","last_peered":"2023-10-16T13:49:02.368494+0000","last_clean":"2023-10-16T13:49:02.368494+0000","last_became_active":"2023-10-16T13:49:01.693523+0000","last_became_peered":"2023-10-16T13:49:01.693523+0000","last_unstale":"2023-10-16T13:49:02.368494+0000","last_undegraded":"2023-10-16T13:49:02.368494+0000","last_fullsized":"2023-10-16T13:49:02.368494+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_clean_scrub_stamp":"2023-10-16T13:49:00.664589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:41:52.860750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.8","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.388241+0000","last_change":"2023-10-16T13:49:02.383372+0000","last_active":"2023-10-16T13:49:02.388241+0000","last_peered":"2023-10-16T13:49:02.388241+0000","last_clean":"2023-10-16T13:49:02.388241+0000","last_became_active":"2023-10-16T13:49:02.383040+0000","last_became_peered":"2023-10-16T13:49:02.383040+0000","last_unstale":"2023-10-16T13:49:02.388241+0000","last_undegraded":"2023-10-16T13:49:02.388241+0000","last_fullsized":"2023-10-16T13:49:02.388241+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:26:39.098172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.b","version":"277'2","reported_seq":13,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.706559+0000","last_change":"2023-10-16T13:49:01.689899+0000","last_active":"2023-10-16T13:49:02.706559+0000","last_peered":"2023-10-16T13:49:02.706559+0000","last_clean":"2023-10-16T13:49:02.706559+0000","last_became_active":"2023-10-16T13:49:01.689621+0000","last_became_peered":"2023-10-16T13:49:01.689621+0000","last_unstale":"2023-10-16T13:49:02.706559+0000","last_undegraded":"2023-10-16T13:49:02.706559+0000","last_fullsized":"2023-10-16T13:49:02.706559+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_clean_scrub_stamp":"2023-10-16T13:49:00.664589+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:53:02.858944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.9","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.379500+0000","last_change":"2023-10-16T13:49:02.378684+0000","last_active":"2023-10-16T13:49:02.379500+0000","last_peered":"2023-10-16T13:49:02.379500+0000","last_clean":"2023-10-16T13:49:02.379500+0000","last_became_active":"2023-10-16T13:49:02.378083+0000","last_became_peered":"2023-10-16T13:49:02.378083+0000","last_unstale":"2023-10-16T13:49:02.379500+0000","last_undegraded":"2023-10-16T13:49:02.379500+0000","last_fullsized":"2023-10-16T13:49:02.379500+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:14:32.098204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.a","version":"0'0","reported_seq":11,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.370777+0000","last_change":"2023-10-16T13:49:01.695913+0000","last_active":"2023-10-16T13:49:02.370777+0000","last_peered":"2023-10-16T13:49:02.370777+0000","last_clean":"2023-10-16T13:49:02.370777+0000","last_became_active":"2023-10-16T13:49:01.695587+0000","last_became_peered":"2023-10-16T13:49:01.695587+0000","last_unstale":"2023-10-16T13:49:02.370777+0000","last_undegraded":"2023-10-16T13:49:02.370777+0000","last_fullsized":"2023-10-16T13:49:02.370777+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_clean_scrub_stamp":"2023-10-16T13:49:00.664589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:33:58.144888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.a","version":"0'0","reported_seq":11,"reported_epoch":278,"state":"active+clean","last_fresh":"2023-10-16T13:49:03.369182+0000","last_change":"2023-10-16T13:49:02.380331+0000","last_active":"2023-10-16T13:49:03.369182+0000","last_peered":"2023-10-16T13:49:03.369182+0000","last_clean":"2023-10-16T13:49:03.369182+0000","last_became_active":"2023-10-16T13:49:02.379419+0000","last_became_peered":"2023-10-16T13:49:02.379419+0000","last_unstale":"2023-10-16T13:49:03.369182+0000","last_undegraded":"2023-10-16T13:49:03.369182+0000","last_fullsized":"2023-10-16T13:49:03.369182+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:17:45.230832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.9","version":"0'0","reported_seq":11,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.371535+0000","last_change":"2023-10-16T13:49:01.689810+0000","last_active":"2023-10-16T13:49:02.371535+0000","last_peered":"2023-10-16T13:49:02.371535+0000","last_clean":"2023-10-16T13:49:02.371535+0000","last_became_active":"2023-10-16T13:49:01.689526+0000","last_became_peered":"2023-10-16T13:49:01.689526+0000","last_unstale":"2023-10-16T13:49:02.371535+0000","last_undegraded":"2023-10-16T13:49:02.371535+0000","last_fullsized":"2023-10-16T13:49:02.371535+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_clean_scrub_stamp":"2023-10-16T13:49:00.664589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:33:58.144888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.b","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.385993+0000","last_change":"2023-10-16T13:49:02.380114+0000","last_active":"2023-10-16T13:49:02.385993+0000","last_peered":"2023-10-16T13:49:02.385993+0000","last_clean":"2023-10-16T13:49:02.385993+0000","last_became_active":"2023-10-16T13:49:02.379822+0000","last_became_peered":"2023-10-16T13:49:02.379822+0000","last_unstale":"2023-10-16T13:49:02.385993+0000","last_undegraded":"2023-10-16T13:49:02.385993+0000","last_fullsized":"2023-10-16T13:49:02.385993+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:07:04.722402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.8","version":"0'0","reported_seq":11,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.370965+0000","last_change":"2023-10-16T13:49:01.690586+0000","last_active":"2023-10-16T13:49:02.370965+0000","last_peered":"2023-10-16T13:49:02.370965+0000","last_clean":"2023-10-16T13:49:02.370965+0000","last_became_active":"2023-10-16T13:49:01.690296+0000","last_became_peered":"2023-10-16T13:49:01.690296+0000","last_unstale":"2023-10-16T13:49:02.370965+0000","last_undegraded":"2023-10-16T13:49:02.370965+0000","last_fullsized":"2023-10-16T13:49:02.370965+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_clean_scrub_stamp":"2023-10-16T13:49:00.664589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:07:50.415719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.4","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.388329+0000","last_change":"2023-10-16T13:49:02.384189+0000","last_active":"2023-10-16T13:49:02.388329+0000","last_peered":"2023-10-16T13:49:02.388329+0000","last_clean":"2023-10-16T13:49:02.388329+0000","last_became_active":"2023-10-16T13:49:02.383881+0000","last_became_peered":"2023-10-16T13:49:02.383881+0000","last_unstale":"2023-10-16T13:49:02.388329+0000","last_undegraded":"2023-10-16T13:49:02.388329+0000","last_fullsized":"2023-10-16T13:49:02.388329+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:07:51.420767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.7","version":"277'2","reported_seq":14,"reported_epoch":278,"state":"active+clean","last_fresh":"2023-10-16T13:49:04.018971+0000","last_change":"2023-10-16T13:49:01.697143+0000","last_active":"2023-10-16T13:49:04.018971+0000","last_peered":"2023-10-16T13:49:04.018971+0000","last_clean":"2023-10-16T13:49:04.018971+0000","last_became_active":"2023-10-16T13:49:01.696560+0000","last_became_peered":"2023-10-16T13:49:01.696560+0000","last_unstale":"2023-10-16T13:49:04.018971+0000","last_undegraded":"2023-10-16T13:49:04.018971+0000","last_fullsized":"2023-10-16T13:49:04.018971+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_clean_scrub_stamp":"2023-10-16T13:49:00.664589+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:28:32.125622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"38.5","version":"0'0","reported_seq":11,"reported_epoch":278,"state":"active+clean","last_fresh":"2023-10-16T13:49:04.018948+0000","last_change":"2023-10-16T13:49:02.389063+0000","last_active":"2023-10-16T13:49:04.018948+0000","last_peered":"2023-10-16T13:49:04.018948+0000","last_clean":"2023-10-16T13:49:04.018948+0000","last_became_active":"2023-10-16T13:49:02.388213+0000","last_became_peered":"2023-10-16T13:49:02.388213+0000","last_unstale":"2023-10-16T13:49:04.018948+0000","last_undegraded":"2023-10-16T13:49:04.018948+0000","last_fullsized":"2023-10-16T13:49:04.018948+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:45:38.625175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.6","version":"0'0","reported_seq":11,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.366151+0000","last_change":"2023-10-16T13:49:01.690867+0000","last_active":"2023-10-16T13:49:02.366151+0000","last_peered":"2023-10-16T13:49:02.366151+0000","last_clean":"2023-10-16T13:49:02.366151+0000","last_became_active":"2023-10-16T13:49:01.690585+0000","last_became_peered":"2023-10-16T13:49:01.690585+0000","last_unstale":"2023-10-16T13:49:02.366151+0000","last_undegraded":"2023-10-16T13:49:02.366151+0000","last_fullsized":"2023-10-16T13:49:02.366151+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_clean_scrub_stamp":"2023-10-16T13:49:00.664589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:52:08.521711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.6","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.383158+0000","last_change":"2023-10-16T13:49:02.382355+0000","last_active":"2023-10-16T13:49:02.383158+0000","last_peered":"2023-10-16T13:49:02.383158+0000","last_clean":"2023-10-16T13:49:02.383158+0000","last_became_active":"2023-10-16T13:49:02.381703+0000","last_became_peered":"2023-10-16T13:49:02.381703+0000","last_unstale":"2023-10-16T13:49:02.383158+0000","last_undegraded":"2023-10-16T13:49:02.383158+0000","last_fullsized":"2023-10-16T13:49:02.383158+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:26:39.098172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.5","version":"0'0","reported_seq":11,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.371146+0000","last_change":"2023-10-16T13:49:01.694996+0000","last_active":"2023-10-16T13:49:02.371146+0000","last_peered":"2023-10-16T13:49:02.371146+0000","last_clean":"2023-10-16T13:49:02.371146+0000","last_became_active":"2023-10-16T13:49:01.694752+0000","last_became_peered":"2023-10-16T13:49:01.694752+0000","last_unstale":"2023-10-16T13:49:02.371146+0000","last_undegraded":"2023-10-16T13:49:02.371146+0000","last_fullsized":"2023-10-16T13:49:02.371146+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_clean_scrub_stamp":"2023-10-16T13:49:00.664589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T19:05:09.858912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.7","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.377755+0000","last_change":"2023-10-16T13:49:02.376440+0000","last_active":"2023-10-16T13:49:02.377755+0000","last_peered":"2023-10-16T13:49:02.377755+0000","last_clean":"2023-10-16T13:49:02.377755+0000","last_became_active":"2023-10-16T13:49:02.376158+0000","last_became_peered":"2023-10-16T13:49:02.376158+0000","last_unstale":"2023-10-16T13:49:02.377755+0000","last_undegraded":"2023-10-16T13:49:02.377755+0000","last_fullsized":"2023-10-16T13:49:02.377755+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:19:05.837652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.4","version":"0'0","reported_seq":12,"reported_epoch":278,"state":"active+clean","last_fresh":"2023-10-16T13:49:03.369040+0000","last_change":"2023-10-16T13:49:01.691473+0000","last_active":"2023-10-16T13:49:03.369040+0000","last_peered":"2023-10-16T13:49:03.369040+0000","last_clean":"2023-10-16T13:49:03.369040+0000","last_became_active":"2023-10-16T13:49:01.691045+0000","last_became_peered":"2023-10-16T13:49:01.691045+0000","last_unstale":"2023-10-16T13:49:03.369040+0000","last_undegraded":"2023-10-16T13:49:03.369040+0000","last_fullsized":"2023-10-16T13:49:03.369040+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_clean_scrub_stamp":"2023-10-16T13:49:00.664589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:53:36.987416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.0","version":"0'0","reported_seq":11,"reported_epoch":278,"state":"active+clean","last_fresh":"2023-10-16T13:49:03.368881+0000","last_change":"2023-10-16T13:49:02.385631+0000","last_active":"2023-10-16T13:49:03.368881+0000","last_peered":"2023-10-16T13:49:03.368881+0000","last_clean":"2023-10-16T13:49:03.368881+0000","last_became_active":"2023-10-16T13:49:02.385327+0000","last_became_peered":"2023-10-16T13:49:02.385327+0000","last_unstale":"2023-10-16T13:49:03.368881+0000","last_undegraded":"2023-10-16T13:49:03.368881+0000","last_fullsized":"2023-10-16T13:49:03.368881+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:44:58.444483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.3","version":"0'0","reported_seq":12,"reported_epoch":278,"state":"active+clean","last_fresh":"2023-10-16T13:49:03.368181+0000","last_change":"2023-10-16T13:49:01.694089+0000","last_active":"2023-10-16T13:49:03.368181+0000","last_peered":"2023-10-16T13:49:03.368181+0000","last_clean":"2023-10-16T13:49:03.368181+0000","last_became_active":"2023-10-16T13:49:01.693769+0000","last_became_peered":"2023-10-16T13:49:01.693769+0000","last_unstale":"2023-10-16T13:49:03.368181+0000","last_undegraded":"2023-10-16T13:49:03.368181+0000","last_fullsized":"2023-10-16T13:49:03.368181+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_clean_scrub_stamp":"2023-10-16T13:49:00.664589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:11:12.807972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"38.1","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.385847+0000","last_change":"2023-10-16T13:49:02.385429+0000","last_active":"2023-10-16T13:49:02.385847+0000","last_peered":"2023-10-16T13:49:02.385847+0000","last_clean":"2023-10-16T13:49:02.385847+0000","last_became_active":"2023-10-16T13:49:02.385115+0000","last_became_peered":"2023-10-16T13:49:02.385115+0000","last_unstale":"2023-10-16T13:49:02.385847+0000","last_undegraded":"2023-10-16T13:49:02.385847+0000","last_fullsized":"2023-10-16T13:49:02.385847+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:10:29.903850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.2","version":"0'0","reported_seq":11,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.372329+0000","last_change":"2023-10-16T13:49:01.693265+0000","last_active":"2023-10-16T13:49:02.372329+0000","last_peered":"2023-10-16T13:49:02.372329+0000","last_clean":"2023-10-16T13:49:02.372329+0000","last_became_active":"2023-10-16T13:49:01.688515+0000","last_became_peered":"2023-10-16T13:49:01.688515+0000","last_unstale":"2023-10-16T13:49:02.372329+0000","last_undegraded":"2023-10-16T13:49:02.372329+0000","last_fullsized":"2023-10-16T13:49:02.372329+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_clean_scrub_stamp":"2023-10-16T13:49:00.664589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:37:38.050689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.2","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.379098+0000","last_change":"2023-10-16T13:49:02.377131+0000","last_active":"2023-10-16T13:49:02.379098+0000","last_peered":"2023-10-16T13:49:02.379098+0000","last_clean":"2023-10-16T13:49:02.379098+0000","last_became_active":"2023-10-16T13:49:02.376786+0000","last_became_peered":"2023-10-16T13:49:02.376786+0000","last_unstale":"2023-10-16T13:49:02.379098+0000","last_undegraded":"2023-10-16T13:49:02.379098+0000","last_fullsized":"2023-10-16T13:49:02.379098+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T17:26:04.311239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.1","version":"0'0","reported_seq":11,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.368364+0000","last_change":"2023-10-16T13:49:01.685519+0000","last_active":"2023-10-16T13:49:02.368364+0000","last_peered":"2023-10-16T13:49:02.368364+0000","last_clean":"2023-10-16T13:49:02.368364+0000","last_became_active":"2023-10-16T13:49:01.685170+0000","last_became_peered":"2023-10-16T13:49:01.685170+0000","last_unstale":"2023-10-16T13:49:02.368364+0000","last_undegraded":"2023-10-16T13:49:02.368364+0000","last_fullsized":"2023-10-16T13:49:02.368364+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_clean_scrub_stamp":"2023-10-16T13:49:00.664589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:01:06.177631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.3","version":"0'0","reported_seq":11,"reported_epoch":278,"state":"active+clean","last_fresh":"2023-10-16T13:49:04.019010+0000","last_change":"2023-10-16T13:49:02.373374+0000","last_active":"2023-10-16T13:49:04.019010+0000","last_peered":"2023-10-16T13:49:04.019010+0000","last_clean":"2023-10-16T13:49:04.019010+0000","last_became_active":"2023-10-16T13:49:02.371820+0000","last_became_peered":"2023-10-16T13:49:02.371820+0000","last_unstale":"2023-10-16T13:49:04.019010+0000","last_undegraded":"2023-10-16T13:49:04.019010+0000","last_fullsized":"2023-10-16T13:49:04.019010+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T14:07:27.543946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.0","version":"0'0","reported_seq":11,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.371110+0000","last_change":"2023-10-16T13:49:01.688914+0000","last_active":"2023-10-16T13:49:02.371110+0000","last_peered":"2023-10-16T13:49:02.371110+0000","last_clean":"2023-10-16T13:49:02.371110+0000","last_became_active":"2023-10-16T13:49:01.688588+0000","last_became_peered":"2023-10-16T13:49:01.688588+0000","last_unstale":"2023-10-16T13:49:02.371110+0000","last_undegraded":"2023-10-16T13:49:02.371110+0000","last_fullsized":"2023-10-16T13:49:02.371110+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:00.664589+0000","last_clean_scrub_stamp":"2023-10-16T13:49:00.664589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T15:29:31.243509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.27","version":"0'0","reported_seq":9,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-10-16T13:49:02.387063+0000","last_change":"2023-10-16T13:49:02.381169+0000","last_active":"2023-10-16T13:49:02.387063+0000","last_peered":"2023-10-16T13:49:02.387063+0000","last_clean":"2023-10-16T13:49:02.387063+0000","last_became_active":"2023-10-16T13:49:02.380798+0000","last_became_peered":"2023-10-16T13:49:02.380798+0000","last_unstale":"2023-10-16T13:49:02.387063+0000","last_undegraded":"2023-10-16T13:49:02.387063+0000","last_fullsized":"2023-10-16T13:49:02.387063+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":276,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T13:49:01.669638+0000","last_clean_scrub_stamp":"2023-10-16T13:49:01.669638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:41:38.172251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"268'403","reported_seq":1441,"reported_epoch":276,"state":"active+clean","last_fresh":"2023-10-16T13:49:01.687159+0000","last_change":"2023-10-16T12:10:34.746820+0000","last_active":"2023-10-16T13:49:01.687159+0000","last_peered":"2023-10-16T13:49:01.687159+0000","last_clean":"2023-10-16T13:49:01.687159+0000","last_became_active":"2023-10-16T12:10:34.745574+0000","last_became_peered":"2023-10-16T12:10:34.745574+0000","last_unstale":"2023-10-16T13:49:01.687159+0000","last_undegraded":"2023-10-16T13:49:01.687159+0000","last_fullsized":"2023-10-16T13:49:01.687159+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_clean_scrub_stamp":"2023-10-16T12:10:33.577111+0000","objects_scrubbed":0,"log_size":403,"log_dups_size":0,"ondisk_log_size":403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:14.461986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2612,"num_read_kb":5456,"num_write":679,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":38,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":37,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2612,"num_read_kb":5456,"num_write":679,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":403,"ondisk_log_size":403,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576013,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61564,"kb_used_data":4008,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310276,"statfs":{"total":96636764160,"available":96573722624,"internally_reserved":0,"allocated":4104192,"data_stored":2410507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58915275},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:48:54 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.58399999999999996,"15min":0.621},"min":{"1min":0.34999999999999998,"5min":0.34200000000000003,"15min":0.33000000000000002},"max":{"1min":4.1040000000000001,"5min":4.1040000000000001,"15min":4.1040000000000001},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.753,"5min":0.60699999999999998,"15min":0.625},"min":{"1min":0.35199999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":5.5279999999999996,"5min":5.5279999999999996,"15min":5.5279999999999996},"last":0.46200000000000002}]},{"osd":1,"last update":"Mon Oct 16 13:48:54 2023","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.63100000000000001,"15min":0.64300000000000002},"min":{"1min":0.33500000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":6.6189999999999998,"5min":6.6189999999999998,"15min":6.6189999999999998},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.59999999999999998,"15min":0.626},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":3.3239999999999998,"5min":3.3239999999999998,"15min":3.3239999999999998},"last":0.53900000000000003}]},{"osd":2,"last update":"Mon Oct 16 13:48:54 2023","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.64500000000000002,"15min":0.67700000000000005},"min":{"1min":0.41499999999999998,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":2.4430000000000001,"5min":2.4430000000000001,"15min":2.4430000000000001},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.67200000000000004,"15min":0.69799999999999995},"min":{"1min":0.45500000000000002,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":4.351,"5min":4.351,"15min":4.351},"last":0.59999999999999998}]},{"osd":3,"last update":"Mon Oct 16 13:48:54 2023","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.63300000000000001,"15min":0.65500000000000003},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":4.4130000000000003,"5min":4.4130000000000003,"15min":4.4130000000000003},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.67600000000000005,"15min":0.67100000000000004},"min":{"1min":0.432,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":5.407,"5min":5.407,"15min":5.407},"last":0.624}]},{"osd":4,"last update":"Mon Oct 16 13:48:54 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.56100000000000005,"15min":0.53500000000000003},"min":{"1min":0.37,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":3.2400000000000002,"5min":3.2400000000000002,"15min":3.2400000000000002},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.67900000000000005,"15min":0.63900000000000001},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":5.3369999999999997,"5min":5.3369999999999997,"15min":5.3369999999999997},"last":0.65300000000000002}]},{"osd":5,"last update":"Mon Oct 16 13:48:54 2023","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.621,"15min":0.57199999999999995},"min":{"1min":0.35799999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":5.4720000000000004,"5min":5.4720000000000004,"15min":5.4720000000000004},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.61299999999999999,"15min":0.59799999999999998},"min":{"1min":0.40100000000000002,"5min":0.39700000000000002,"15min":0.35099999999999998},"max":{"1min":1.0549999999999999,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.76700000000000002}]},{"osd":6,"last update":"Mon Oct 16 13:48:54 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.63200000000000001,"15min":0.622},"min":{"1min":0.46600000000000003,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.69499999999999995,"15min":0.64500000000000002},"min":{"1min":0.38900000000000001,"5min":0.378,"15min":0.35199999999999998},"max":{"1min":5.3860000000000001,"5min":5.3860000000000001,"15min":5.3860000000000001},"last":0.70399999999999996}]}]},{"osd":6,"up_from":13,"seq":55834576011,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48500,"kb_used_data":3432,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323340,"statfs":{"total":96636764160,"available":96587100160,"internally_reserved":0,"allocated":3514368,"data_stored":1820721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.60999999999999999,"15min":0.65000000000000002},"min":{"1min":0.315,"5min":0.315,"15min":0.29599999999999999},"max":{"1min":0.67600000000000005,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.503,"5min":0.60599999999999998,"15min":0.64200000000000002},"min":{"1min":0.373,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.70099999999999996,"5min":0.91400000000000003,"15min":0.93799999999999994},"last":0.64300000000000002}]},{"osd":1,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.54600000000000004,"15min":0.60499999999999998},"min":{"1min":0.316,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.64500000000000002,"5min":0.86899999999999999,"15min":0.89600000000000002},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.58599999999999997,"15min":0.60599999999999998},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.69799999999999995,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.53600000000000003}]},{"osd":2,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.58899999999999997,"15min":0.61799999999999999},"min":{"1min":0.34799999999999998,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.71499999999999997,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.57599999999999996,"15min":0.60599999999999998},"min":{"1min":0.32200000000000001,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.72999999999999998,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.67500000000000004}]},{"osd":3,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.60199999999999998,"15min":0.64600000000000002},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.38200000000000001},"max":{"1min":0.63400000000000001,"5min":0.92400000000000004,"15min":0.95599999999999996},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.66600000000000004,"15min":0.70199999999999996},"min":{"1min":0.435,"5min":0.435,"15min":0.42299999999999999},"max":{"1min":0.72399999999999998,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.69099999999999995}]},{"osd":4,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.53300000000000003,"15min":0.52800000000000002},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.67700000000000005,"5min":0.85599999999999998,"15min":0.95799999999999996},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.60799999999999998,"15min":0.61499999999999999},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.70799999999999996,"5min":0.98799999999999999,"15min":1.0309999999999999},"last":0.59799999999999998}]},{"osd":5,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.58199999999999996,"15min":0.59099999999999997},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.71099999999999997,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.627,"5min":0.64400000000000002,"15min":0.65700000000000003},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.78700000000000003,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.57899999999999996}]},{"osd":7,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.59099999999999997,"15min":0.59799999999999998},"min":{"1min":0.35199999999999998,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.747,"5min":1.081,"15min":1.081},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.63900000000000001,"15min":0.64900000000000002},"min":{"1min":0.40999999999999998,"5min":0.40899999999999997,"15min":0.379},"max":{"1min":0.76600000000000001,"5min":1.099,"15min":1.099},"last":0.55300000000000005}]}]},{"osd":1,"up_from":13,"seq":55834576011,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52652,"kb_used_data":3424,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319188,"statfs":{"total":96636764160,"available":96582848512,"internally_reserved":0,"allocated":3506176,"data_stored":1820117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":50395593},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:48:59 2023","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.37,"15min":0.40100000000000002},"min":{"1min":0.254,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.58099999999999996,"5min":0.65400000000000003,"15min":0.65800000000000003},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.433,"15min":0.48099999999999998},"min":{"1min":0.32700000000000001,"5min":0.253,"15min":0.253},"max":{"1min":0.56599999999999995,"5min":0.68000000000000005,"15min":0.77200000000000002},"last":0.53500000000000003}]},{"osd":2,"last update":"Mon Oct 16 13:48:59 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.39400000000000002,"15min":0.439},"min":{"1min":0.251,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.496,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.44500000000000001,"15min":0.47999999999999998},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.58299999999999996,"5min":0.67600000000000005,"15min":0.71399999999999997},"last":0.49399999999999999}]},{"osd":3,"last update":"Mon Oct 16 13:48:59 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.39000000000000001,"15min":0.42299999999999999},"min":{"1min":0.28499999999999998,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.51300000000000001,"5min":0.65200000000000002,"15min":0.72099999999999997},"last":0.443},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.41699999999999998,"15min":0.46600000000000003},"min":{"1min":0.29599999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.60299999999999998,"5min":0.69399999999999995,"15min":0.77300000000000002},"last":0.51300000000000001}]},{"osd":4,"last update":"Mon Oct 16 13:48:59 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.61599999999999999,"15min":0.65700000000000003},"min":{"1min":0.41199999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.78300000000000003,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.59899999999999998,"15min":0.65600000000000003},"min":{"1min":0.377,"5min":0.37,"15min":0.37},"max":{"1min":0.90400000000000003,"5min":0.91900000000000004,"15min":0.997},"last":0.68300000000000005}]},{"osd":5,"last update":"Mon Oct 16 13:48:59 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.64700000000000002,"15min":0.69899999999999995},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":1.0760000000000001},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.65700000000000003,"15min":0.71499999999999997},"min":{"1min":0.436,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.81299999999999994,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.64300000000000002}]},{"osd":6,"last update":"Mon Oct 16 13:48:59 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.621,"15min":0.66900000000000004},"min":{"1min":0.41999999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85799999999999998,"5min":0.95099999999999996,"15min":0.97799999999999998},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.62,"5min":0.64500000000000002,"15min":0.69499999999999995},"min":{"1min":0.46899999999999997,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":1.083},"last":0.70499999999999996}]},{"osd":7,"last update":"Mon Oct 16 13:48:59 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.66300000000000003,"15min":0.72199999999999998},"min":{"1min":0.45300000000000001,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.79300000000000004,"5min":1.163,"15min":1.163},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.68799999999999994,"15min":0.73699999999999999},"min":{"1min":0.47999999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.94699999999999995,"5min":0.98499999999999999,"15min":1.0580000000000001},"last":0.72299999999999998}]}]},{"osd":0,"up_from":13,"seq":55834576012,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53236,"kb_used_data":4008,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318604,"statfs":{"total":96636764160,"available":96582250496,"internally_reserved":0,"allocated":4104192,"data_stored":2410575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.373,"15min":0.41299999999999998},"min":{"1min":0.23799999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.50900000000000001,"5min":0.60699999999999998,"15min":0.71399999999999997},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.38500000000000001,"15min":0.42799999999999999},"min":{"1min":0.24199999999999999,"5min":0.222,"15min":0.222},"max":{"1min":0.56799999999999995,"5min":0.77700000000000002,"15min":0.82199999999999995},"last":0.32100000000000001}]},{"osd":2,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.40999999999999998,"15min":0.441},"min":{"1min":0.27300000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.51000000000000001,"5min":0.65900000000000003,"15min":0.75700000000000001},"last":0.44},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.432,"15min":0.47199999999999998},"min":{"1min":0.30399999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.54500000000000004,"5min":0.68799999999999994,"15min":0.83299999999999996},"last":0.35999999999999999}]},{"osd":3,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.40600000000000003,"15min":0.44400000000000001},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.52100000000000002,"5min":0.69699999999999995,"15min":0.78600000000000003},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.46300000000000002,"15min":0.50600000000000001},"min":{"1min":0.309,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.55900000000000005,"5min":0.70399999999999996,"15min":0.86399999999999999},"last":0.64100000000000001}]},{"osd":4,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.64500000000000002,"15min":0.67600000000000005},"min":{"1min":0.42999999999999999,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.82999999999999996,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.64900000000000002,"15min":0.68100000000000005},"min":{"1min":0.43099999999999999,"5min":0.375,"15min":0.375},"max":{"1min":0.81000000000000005,"5min":1.21,"15min":1.21},"last":0.68200000000000005}]},{"osd":5,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.63400000000000001,"15min":0.67700000000000005},"min":{"1min":0.40300000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.82199999999999995,"5min":1.242,"15min":1.242},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.622,"5min":0.68300000000000005,"15min":0.71799999999999997},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.83299999999999996,"5min":1.081,"15min":1.081},"last":0.47999999999999998}]},{"osd":6,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.67600000000000005,"15min":0.71999999999999997},"min":{"1min":0.43099999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.84699999999999998,"5min":1.034,"15min":1.103},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.67700000000000005,"15min":0.71499999999999997},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.90900000000000003,"5min":1.131,"15min":1.131},"last":0.59599999999999997}]},{"osd":7,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.68700000000000006,"15min":0.73499999999999999},"min":{"1min":0.47399999999999998,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.94799999999999995,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.69599999999999995,"15min":0.72699999999999998},"min":{"1min":0.47899999999999998,"5min":0.433,"15min":0.433},"max":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.71699999999999997}]}]},{"osd":2,"up_from":13,"seq":55834576012,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56848,"kb_used_data":3452,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314992,"statfs":{"total":96636764160,"available":96578551808,"internally_reserved":0,"allocated":3534848,"data_stored":1828547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:48:41 2023","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.374,"15min":0.40400000000000003},"min":{"1min":0.23899999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.439,"5min":0.623,"15min":0.77500000000000002},"last":0.433},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.436,"15min":0.46999999999999997},"min":{"1min":0.33700000000000002,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.57399999999999995,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.57999999999999996}]},{"osd":1,"last update":"Mon Oct 16 13:48:41 2023","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.46100000000000002,"15min":0.49199999999999999},"min":{"1min":0.30499999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.69899999999999995,"5min":0.73199999999999998,"15min":0.81399999999999995},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.45000000000000001,"15min":0.46899999999999997},"min":{"1min":0.27900000000000003,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.51700000000000002,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.53400000000000003}]},{"osd":3,"last update":"Mon Oct 16 13:48:41 2023","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.40400000000000003,"15min":0.435},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.55500000000000005,"5min":0.63,"15min":0.72599999999999998},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.499,"15min":0.53700000000000003},"min":{"1min":0.376,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.65900000000000003,"5min":0.82699999999999996,"15min":0.90100000000000002},"last":0.65200000000000002}]},{"osd":4,"last update":"Mon Oct 16 13:48:41 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.66800000000000004,"15min":0.70699999999999996},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.423,"5min":1.423,"15min":1.423},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.67700000000000005,"15min":0.71399999999999997},"min":{"1min":0.437,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.407,"5min":1.407,"15min":1.407},"last":0.82499999999999996}]},{"osd":5,"last update":"Mon Oct 16 13:48:41 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.627,"15min":0.67800000000000005},"min":{"1min":0.436,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.82199999999999995,"5min":0.87,"15min":1.486},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.64900000000000002,"15min":0.68999999999999995},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.78200000000000003,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":0.80800000000000005}]},{"osd":6,"last update":"Mon Oct 16 13:48:41 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.622,"15min":0.65900000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.80300000000000005,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.66200000000000003,"15min":0.71099999999999997},"min":{"1min":0.49099999999999999,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.85499999999999998,"5min":0.94299999999999995,"15min":1.4379999999999999},"last":0.71399999999999997}]},{"osd":7,"last update":"Mon Oct 16 13:48:41 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.67500000000000004,"15min":0.70899999999999996},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.83499999999999996,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.66800000000000004,"15min":0.72199999999999998},"min":{"1min":0.48799999999999999,"5min":0.436,"15min":0.436},"max":{"1min":0.88200000000000001,"5min":1.0429999999999999,"15min":1.377},"last":0.84499999999999997}]}]},{"osd":3,"up_from":13,"seq":55834576011,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56820,"kb_used_data":3428,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315020,"statfs":{"total":96636764160,"available":96578580480,"internally_reserved":0,"allocated":3510272,"data_stored":1820151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:48:45 2023","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.375,"15min":0.40400000000000003},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23000000000000001},"max":{"1min":0.48399999999999999,"5min":0.60099999999999998,"15min":0.70499999999999996},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.38300000000000001,"15min":0.41599999999999998},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.48799999999999999,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.34100000000000003}]},{"osd":1,"last update":"Mon Oct 16 13:48:45 2023","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.38900000000000001,"15min":0.42299999999999999},"min":{"1min":0.27200000000000002,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.497,"5min":0.68200000000000005,"15min":0.76300000000000001},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.438,"15min":0.46999999999999997},"min":{"1min":0.27800000000000002,"5min":0.27700000000000002,"15min":0.253},"max":{"1min":0.60599999999999998,"5min":0.68700000000000006,"15min":0.77800000000000002},"last":0.41099999999999998}]},{"osd":2,"last update":"Mon Oct 16 13:48:45 2023","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.45100000000000001,"15min":0.47999999999999998},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.71399999999999997,"5min":1.167,"15min":1.167},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.46000000000000002,"15min":0.50700000000000001},"min":{"1min":0.32500000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.63200000000000001,"5min":0.73299999999999998,"15min":0.76800000000000002},"last":0.36499999999999999}]},{"osd":4,"last update":"Mon Oct 16 13:48:45 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.61799999999999999,"15min":0.65500000000000003},"min":{"1min":0.40100000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.137},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65400000000000003,"15min":0.68899999999999995},"min":{"1min":0.45500000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.84699999999999998,"5min":1.2110000000000001,"15min":1.2909999999999999},"last":0.60199999999999998}]},{"osd":5,"last update":"Mon Oct 16 13:48:45 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.65200000000000002,"15min":0.69799999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.374},"max":{"1min":1.0309999999999999,"5min":1.0680000000000001,"15min":1.2729999999999999},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.65400000000000003,"15min":0.68999999999999995},"min":{"1min":0.47299999999999998,"5min":0.45600000000000002,"15min":0.432},"max":{"1min":0.871,"5min":1.0549999999999999,"15min":1.173},"last":0.51900000000000002}]},{"osd":6,"last update":"Mon Oct 16 13:48:45 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.69799999999999995,"15min":0.71299999999999997},"min":{"1min":0.432,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.115,"5min":5.0999999999999996,"15min":5.0999999999999996},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.68600000000000005,"15min":0.72499999999999998},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.2290000000000001},"last":0.59199999999999997}]},{"osd":7,"last update":"Mon Oct 16 13:48:45 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.68100000000000005,"15min":0.72699999999999998},"min":{"1min":0.45700000000000002,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.21,"5min":1.21,"15min":1.2569999999999999},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.68700000000000006,"15min":0.73399999999999999},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.43099999999999999},"max":{"1min":0.90600000000000003,"5min":1.262,"15min":1.262},"last":0.49099999999999999}]}]},{"osd":4,"up_from":13,"seq":55834576010,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52696,"kb_used_data":3460,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319144,"statfs":{"total":96636764160,"available":96582803456,"internally_reserved":0,"allocated":3543040,"data_stored":1829219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:48:42 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.56299999999999994,"15min":0.61299999999999999},"min":{"1min":0.36399999999999999,"5min":0.314,"15min":0.314},"max":{"1min":0.77800000000000002,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.441},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.57399999999999995,"15min":0.624},"min":{"1min":0.36599999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.87,"5min":0.87,"15min":0.89500000000000002},"last":0.48599999999999999}]},{"osd":1,"last update":"Mon Oct 16 13:48:42 2023","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.57499999999999996,"15min":0.64100000000000001},"min":{"1min":0.35699999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.79900000000000004,"5min":0.877,"15min":0.877},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.58399999999999996,"15min":0.63},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.80900000000000005,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.58199999999999996}]},{"osd":2,"last update":"Mon Oct 16 13:48:42 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.59199999999999997,"15min":0.64100000000000001},"min":{"1min":0.434,"5min":0.378,"15min":0.378},"max":{"1min":0.751,"5min":0.89900000000000002,"15min":0.90600000000000003},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.62,"15min":0.65300000000000002},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.755,"5min":0.89700000000000002,"15min":0.92600000000000005},"last":0.67100000000000004}]},{"osd":3,"last update":"Mon Oct 16 13:48:42 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.58699999999999997,"15min":0.63300000000000001},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.81399999999999995,"5min":0.86599999999999999,"15min":0.99199999999999999},"last":0.629},{"interface":"front","average":{"1min":0.624,"5min":0.64400000000000002,"15min":0.67500000000000004},"min":{"1min":0.48199999999999998,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.93899999999999995},"last":0.61499999999999999}]},{"osd":5,"last update":"Mon Oct 16 13:48:42 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.55700000000000005,"15min":0.53800000000000003},"min":{"1min":0.34899999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.80400000000000005,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.57799999999999996,"15min":0.57099999999999995},"min":{"1min":0.307,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.72499999999999998,"5min":0.82199999999999995,"15min":0.94899999999999995},"last":0.70399999999999996}]},{"osd":6,"last update":"Mon Oct 16 13:48:42 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.63200000000000001,"15min":0.60399999999999998},"min":{"1min":0.38600000000000001,"5min":0.36399999999999999,"15min":0.33200000000000002},"max":{"1min":0.82699999999999996,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.70099999999999996,"15min":0.69699999999999995},"min":{"1min":0.41699999999999998,"5min":0.40300000000000002,"15min":0.39400000000000002},"max":{"1min":0.89700000000000002,"5min":1.129,"15min":1.129},"last":0.77900000000000003}]},{"osd":7,"last update":"Mon Oct 16 13:48:42 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.65100000000000002,"15min":0.63700000000000001},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.78700000000000003,"5min":0.97099999999999997,"15min":0.98599999999999999},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.70999999999999996,"15min":0.68899999999999995},"min":{"1min":0.48399999999999999,"5min":0.44,"15min":0.38200000000000001},"max":{"1min":0.96999999999999997,"5min":0.97599999999999998,"15min":1.026},"last":0.72199999999999998}]}]},{"osd":5,"up_from":13,"seq":55834576011,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74868,"kb_used_data":3428,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94296972,"statfs":{"total":96636764160,"available":96560099328,"internally_reserved":0,"allocated":3510272,"data_stored":1820151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:48:29 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.57199999999999995,"15min":0.60799999999999998},"min":{"1min":0.32100000000000001,"5min":0.314,"15min":0.29899999999999999},"max":{"1min":0.67600000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.58799999999999997,"15min":0.625},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.29699999999999999},"max":{"1min":0.58099999999999996,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.69699999999999995}]},{"osd":1,"last update":"Mon Oct 16 13:48:29 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.622,"15min":0.65100000000000002},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.29499999999999998},"max":{"1min":0.72399999999999998,"5min":1,"15min":1},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.58799999999999997,"15min":0.628},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.67800000000000005,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.754}]},{"osd":2,"last update":"Mon Oct 16 13:48:29 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.624,"15min":0.64500000000000002},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.755,"5min":0.91000000000000003,"15min":0.94699999999999995},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.61399999999999999,"15min":0.64600000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.73399999999999999,"5min":0.871,"15min":0.93500000000000005},"last":0.51400000000000001}]},{"osd":3,"last update":"Mon Oct 16 13:48:29 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.64400000000000002,"15min":0.66100000000000003},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.70999999999999996,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.63},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.66700000000000004,"15min":0.68899999999999995},"min":{"1min":0.439,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.81499999999999995,"5min":1.4199999999999999,"15min":1.4199999999999999},"last":0.53900000000000003}]},{"osd":4,"last update":"Mon Oct 16 13:48:29 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.53800000000000003,"15min":0.53900000000000003},"min":{"1min":0.36899999999999999,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.77700000000000002,"5min":0.89700000000000002,"15min":0.90700000000000003},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.629,"5min":0.60099999999999998,"15min":0.59699999999999998},"min":{"1min":0.375,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.78500000000000003,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.74299999999999999}]},{"osd":6,"last update":"Mon Oct 16 13:48:29 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.59499999999999997,"15min":0.59299999999999997},"min":{"1min":0.35899999999999999,"5min":0.315,"15min":0.315},"max":{"1min":0.85799999999999998,"5min":0.92000000000000004,"15min":0.93200000000000005},"last":0.495},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.624,"15min":0.625},"min":{"1min":0.49099999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.77600000000000002,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.67300000000000004}]},{"osd":7,"last update":"Mon Oct 16 13:48:29 2023","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.65100000000000002,"15min":0.64100000000000001},"min":{"1min":0.47899999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.77300000000000002,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.69899999999999995,"15min":0.69199999999999995},"min":{"1min":0.48799999999999999,"5min":0.375,"15min":0.375},"max":{"1min":0.89900000000000002,"5min":0.97299999999999998,"15min":1.0229999999999999},"last":0.65000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T13:49:07.262 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-16T13:49:07.262 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-16T13:49:07.429 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.427+0000 7fc656b2b700 1 Processor -- start 2023-10-16T13:49:07.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.427+0000 7fc656b2b700 1 -- start start 2023-10-16T13:49:07.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.428+0000 7fc656b2b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc65015b2c0 0x7fc650161340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:07.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.428+0000 7fc656b2b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc65006e570 0x7fc65015e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:07.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.428+0000 7fc656b2b700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc650161880 con 0x7fc65006a7e0 2023-10-16T13:49:07.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.428+0000 7fc656b2b700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc65015b6a0 con 0x7fc65015b2c0 2023-10-16T13:49:07.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.428+0000 7fc656b2b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc650161b20 con 0x7fc65006e570 2023-10-16T13:49:07.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.428+0000 7fc6548c7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc65015b2c0 0x7fc650161340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:07.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.428+0000 7fc6548c7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc65015b2c0 0x7fc650161340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59074/0 (socket says 172.21.15.16:59074) 2023-10-16T13:49:07.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.428+0000 7fc6550c8700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc65006a7e0 0x7fc650071a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48352/0 (socket says 172.21.15.16:48352) 2023-10-16T13:49:07.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.428+0000 7fc6548c7700 1 -- 172.21.15.16:0/4056391341 learned_addr learned my addr 172.21.15.16:0/4056391341 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:07.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.428+0000 7fc64ffff700 1 --2- 172.21.15.16:0/4056391341 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc65006e570 0x7fc65015e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:07.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.428+0000 7fc6548c7700 1 -- 172.21.15.16:0/4056391341 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc65006e570 msgr2=0x7fc65015e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:07.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.428+0000 7fc6548c7700 1 --2- 172.21.15.16:0/4056391341 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc65006e570 0x7fc65015e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:07.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.428+0000 7fc6548c7700 1 -- 172.21.15.16:0/4056391341 >> v1:172.21.15.16:6789/0 conn(0x7fc65006a7e0 legacy=0x7fc650071a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:07.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.428+0000 7fc6548c7700 1 -- 172.21.15.16:0/4056391341 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc650161e00 con 0x7fc65015b2c0 2023-10-16T13:49:07.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.428+0000 7fc64f7fe700 1 -- 172.21.15.16:0/4056391341 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 187718919 0 0) 0x7fc650161880 con 0x7fc65006a7e0 2023-10-16T13:49:07.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.428+0000 7fc6548c7700 1 --2- 172.21.15.16:0/4056391341 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc65015b2c0 0x7fc650161340 secure :-1 s=READY pgs=3201 cs=0 l=1 rev1=1 crypto rx=0x7fc6440098f0 tx=0x7fc644009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3342d0ca022b00d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:07.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.428+0000 7fc64f7fe700 1 -- 172.21.15.16:0/4056391341 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc64400b8a0 con 0x7fc65015b2c0 2023-10-16T13:49:07.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.428+0000 7fc64f7fe700 1 -- 172.21.15.16:0/4056391341 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc64400ba00 con 0x7fc65015b2c0 2023-10-16T13:49:07.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.428+0000 7fc64f7fe700 1 -- 172.21.15.16:0/4056391341 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc64400bd90 con 0x7fc65015b2c0 2023-10-16T13:49:07.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.429+0000 7fc656b2b700 1 -- 172.21.15.16:0/4056391341 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc65015b2c0 msgr2=0x7fc650161340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:07.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.429+0000 7fc656b2b700 1 --2- 172.21.15.16:0/4056391341 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc65015b2c0 0x7fc650161340 secure :-1 s=READY pgs=3201 cs=0 l=1 rev1=1 crypto rx=0x7fc6440098f0 tx=0x7fc644009c00 comp rx=0 tx=0).stop 2023-10-16T13:49:07.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.429+0000 7fc656b2b700 1 -- 172.21.15.16:0/4056391341 shutdown_connections 2023-10-16T13:49:07.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.429+0000 7fc656b2b700 1 --2- 172.21.15.16:0/4056391341 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc65015b2c0 0x7fc650161340 unknown :-1 s=CLOSED pgs=3201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:07.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.429+0000 7fc656b2b700 1 --2- 172.21.15.16:0/4056391341 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc65006e570 0x7fc65015e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:07.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.429+0000 7fc656b2b700 1 -- 172.21.15.16:0/4056391341 >> 172.21.15.16:0/4056391341 conn(0x7fc6500b1f40 msgr2=0x7fc6500b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:07.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.429+0000 7fc656b2b700 1 -- 172.21.15.16:0/4056391341 shutdown_connections 2023-10-16T13:49:07.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.429+0000 7fc656b2b700 1 -- 172.21.15.16:0/4056391341 wait complete. 2023-10-16T13:49:07.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.430+0000 7fc656b2b700 1 Processor -- start 2023-10-16T13:49:07.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.430+0000 7fc656b2b700 1 -- start start 2023-10-16T13:49:07.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.431+0000 7fc656b2b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc65006a7e0 0x7fc65014f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:07.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.431+0000 7fc656b2b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc65006e570 0x7fc6500b51a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:07.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.431+0000 7fc656b2b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc65015b2c0 0x7fc650160ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:07.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.431+0000 7fc656b2b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc65014fec0 con 0x7fc65006a7e0 2023-10-16T13:49:07.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.431+0000 7fc656b2b700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6500c80c0 con 0x7fc65006e570 2023-10-16T13:49:07.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.431+0000 7fc656b2b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6500b4100 con 0x7fc65015b2c0 2023-10-16T13:49:07.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.431+0000 7fc6548c7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc65006a7e0 0x7fc65014f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:07.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.431+0000 7fc6548c7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc65006a7e0 0x7fc65014f980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59080/0 (socket says 172.21.15.16:59080) 2023-10-16T13:49:07.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.431+0000 7fc6548c7700 1 -- 172.21.15.16:0/258997950 learned_addr learned my addr 172.21.15.16:0/258997950 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:07.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.431+0000 7fc6550c8700 1 --2- 172.21.15.16:0/258997950 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc65015b2c0 0x7fc650160ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:07.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.431+0000 7fc64ffff700 1 --2- 172.21.15.16:0/258997950 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc65006e570 0x7fc6500b51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:07.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.432+0000 7fc6548c7700 1 -- 172.21.15.16:0/258997950 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc65015b2c0 msgr2=0x7fc650160ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:07.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.432+0000 7fc6548c7700 1 --2- 172.21.15.16:0/258997950 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc65015b2c0 0x7fc650160ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:07.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.432+0000 7fc6548c7700 1 -- 172.21.15.16:0/258997950 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc65006e570 msgr2=0x7fc6500b51a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:07.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.432+0000 7fc6548c7700 1 --2- 172.21.15.16:0/258997950 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc65006e570 0x7fc6500b51a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:07.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.432+0000 7fc6548c7700 1 -- 172.21.15.16:0/258997950 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc644009550 con 0x7fc65006a7e0 2023-10-16T13:49:07.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.432+0000 7fc6550c8700 1 --2- 172.21.15.16:0/258997950 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc65015b2c0 0x7fc650160ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:49:07.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.432+0000 7fc6548c7700 1 --2- 172.21.15.16:0/258997950 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc65006a7e0 0x7fc65014f980 secure :-1 s=READY pgs=3202 cs=0 l=1 rev1=1 crypto rx=0x7fc644009c40 tx=0x7fc644004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:07.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.432+0000 7fc64d7fa700 1 -- 172.21.15.16:0/258997950 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc644004380 con 0x7fc65006a7e0 2023-10-16T13:49:07.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.432+0000 7fc64d7fa700 1 -- 172.21.15.16:0/258997950 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc64401f070 con 0x7fc65006a7e0 2023-10-16T13:49:07.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.432+0000 7fc656b2b700 1 -- 172.21.15.16:0/258997950 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6500b43e0 con 0x7fc65006a7e0 2023-10-16T13:49:07.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.432+0000 7fc64d7fa700 1 -- 172.21.15.16:0/258997950 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6440036a0 con 0x7fc65006a7e0 2023-10-16T13:49:07.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.432+0000 7fc656b2b700 1 -- 172.21.15.16:0/258997950 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6500b49a0 con 0x7fc65006a7e0 2023-10-16T13:49:07.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.433+0000 7fc656b2b700 1 -- 172.21.15.16:0/258997950 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc65004ebb0 con 0x7fc65006a7e0 2023-10-16T13:49:07.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.434+0000 7fc64d7fa700 1 -- 172.21.15.16:0/258997950 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 166) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc644021030 con 0x7fc65006a7e0 2023-10-16T13:49:07.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.434+0000 7fc64d7fa700 1 --2- 172.21.15.16:0/258997950 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fc63406b020 0x7fc63406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:07.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.434+0000 7fc64d7fa700 1 -- 172.21.15.16:0/258997950 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(278..278 src has 1..278) v4 ==== 6157+0+0 (secure 0 0 0) 0x7fc64408ad50 con 0x7fc65006a7e0 2023-10-16T13:49:07.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.434+0000 7fc64ffff700 1 --2- 172.21.15.16:0/258997950 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fc63406b020 0x7fc63406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:07.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.434+0000 7fc64ffff700 1 --2- 172.21.15.16:0/258997950 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fc63406b020 0x7fc63406d4e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc6500c7d20 tx=0x7fc63c008040 comp rx=0 tx=0).ready entity=mgr.13344 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:07.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.439+0000 7fc64d7fa700 1 -- 172.21.15.16:0/258997950 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc64408e030 con 0x7fc65006a7e0 2023-10-16T13:49:07.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.726+0000 7fc656b2b700 1 -- 172.21.15.16:0/258997950 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fc6500b6370 con 0x7fc65006a7e0 2023-10-16T13:49:07.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.728+0000 7fc64d7fa700 1 -- 172.21.15.16:0/258997950 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v59) v1 ==== 190+0+147 (secure 0 0 0) 0x7fc6440150b0 con 0x7fc65006a7e0 2023-10-16T13:49:07.730 INFO:teuthology.orchestra.run.smithi016.stdout:[client.0] 2023-10-16T13:49:07.730 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAPKC1l2l6PFxAA9QyXtBAHdGAXpwH0JfknuQ== 2023-10-16T13:49:07.731 INFO:teuthology.orchestra.run.smithi016.stdout: caps mds = "allow" 2023-10-16T13:49:07.731 INFO:teuthology.orchestra.run.smithi016.stdout: caps mon = "allow r" 2023-10-16T13:49:07.731 INFO:teuthology.orchestra.run.smithi016.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-16T13:49:07.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.730+0000 7fc656b2b700 1 -- 172.21.15.16:0/258997950 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fc63406b020 msgr2=0x7fc63406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:07.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.730+0000 7fc656b2b700 1 --2- 172.21.15.16:0/258997950 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fc63406b020 0x7fc63406d4e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc6500c7d20 tx=0x7fc63c008040 comp rx=0 tx=0).stop 2023-10-16T13:49:07.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.730+0000 7fc656b2b700 1 -- 172.21.15.16:0/258997950 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc65006a7e0 msgr2=0x7fc65014f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:07.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.730+0000 7fc656b2b700 1 --2- 172.21.15.16:0/258997950 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc65006a7e0 0x7fc65014f980 secure :-1 s=READY pgs=3202 cs=0 l=1 rev1=1 crypto rx=0x7fc644009c40 tx=0x7fc644004750 comp rx=0 tx=0).stop 2023-10-16T13:49:07.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.730+0000 7fc656b2b700 1 -- 172.21.15.16:0/258997950 shutdown_connections 2023-10-16T13:49:07.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.730+0000 7fc656b2b700 1 --2- 172.21.15.16:0/258997950 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc65015b2c0 0x7fc650160ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:07.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.730+0000 7fc656b2b700 1 --2- 172.21.15.16:0/258997950 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fc63406b020 0x7fc63406d4e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:07.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.730+0000 7fc656b2b700 1 --2- 172.21.15.16:0/258997950 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc65006a7e0 0x7fc65014f980 unknown :-1 s=CLOSED pgs=3202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:07.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.730+0000 7fc656b2b700 1 --2- 172.21.15.16:0/258997950 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc65006e570 0x7fc6500b51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:07.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.730+0000 7fc656b2b700 1 -- 172.21.15.16:0/258997950 >> 172.21.15.16:0/258997950 conn(0x7fc6500b1f40 msgr2=0x7fc65015c060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:07.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.730+0000 7fc656b2b700 1 -- 172.21.15.16:0/258997950 shutdown_connections 2023-10-16T13:49:07.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.730+0000 7fc656b2b700 1 -- 172.21.15.16:0/258997950 wait complete. 2023-10-16T13:49:07.735 INFO:teuthology.orchestra.run.smithi016.stderr:updated caps for client.0 2023-10-16T13:49:07.745 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:49:07.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.867+0000 7f6afeaec700 1 Processor -- start 2023-10-16T13:49:07.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.868+0000 7f6afeaec700 1 -- start start 2023-10-16T13:49:07.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.868+0000 7f6afeaec700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6af815e0b0 0x7f6af80aec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:07.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.868+0000 7f6afeaec700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6af806b1f0 con 0x7f6af80cb190 2023-10-16T13:49:07.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.868+0000 7f6afeaec700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6af806b4b0 con 0x7f6af8159c90 2023-10-16T13:49:07.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.868+0000 7f6afeaec700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6af806b790 con 0x7f6af815e0b0 2023-10-16T13:49:07.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.868+0000 7f6afd089700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6af815e0b0 0x7f6af80aec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:07.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.868+0000 7f6af7fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f6af80cb190 0x7f6af80ae520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48362/0 (socket says 172.21.15.16:48362) 2023-10-16T13:49:07.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.869+0000 7f6af7fff700 1 -- 172.21.15.16:0/323163224 learned_addr learned my addr 172.21.15.16:0/323163224 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:49:07.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.869+0000 7f6afd089700 1 -- 172.21.15.16:0/323163224 >> v1:172.21.15.49:6789/0 conn(0x7f6af8159c90 legacy=0x7f6af815d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:07.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.869+0000 7f6afd089700 1 -- 172.21.15.16:0/323163224 >> v1:172.21.15.16:6789/0 conn(0x7f6af80cb190 legacy=0x7f6af80ae520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:07.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.869+0000 7f6afd089700 1 -- 172.21.15.16:0/323163224 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6af806ba70 con 0x7f6af815e0b0 2023-10-16T13:49:07.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.869+0000 7f6af77fe700 1 -- 172.21.15.16:0/323163224 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4195238624 0 0) 0x7f6af806b1f0 con 0x7f6af80cb190 2023-10-16T13:49:07.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.869+0000 7f6afd089700 1 --2- 172.21.15.16:0/323163224 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6af815e0b0 0x7f6af80aec30 secure :-1 s=READY pgs=3204 cs=0 l=1 rev1=1 crypto rx=0x7f6af0009d90 tx=0x7f6af000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38e371f4d511142 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:07.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.869+0000 7f6af77fe700 1 -- 172.21.15.16:0/323163224 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af000e070 con 0x7f6af815e0b0 2023-10-16T13:49:07.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.869+0000 7f6af77fe700 1 -- 172.21.15.16:0/323163224 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6af000bb90 con 0x7f6af815e0b0 2023-10-16T13:49:07.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.869+0000 7f6af77fe700 1 -- 172.21.15.16:0/323163224 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af0003a20 con 0x7f6af815e0b0 2023-10-16T13:49:07.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.870+0000 7f6afeaec700 1 -- 172.21.15.16:0/323163224 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6af815e0b0 msgr2=0x7f6af80aec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:07.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.870+0000 7f6afeaec700 1 --2- 172.21.15.16:0/323163224 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6af815e0b0 0x7f6af80aec30 secure :-1 s=READY pgs=3204 cs=0 l=1 rev1=1 crypto rx=0x7f6af0009d90 tx=0x7f6af000b330 comp rx=0 tx=0).stop 2023-10-16T13:49:07.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.870+0000 7f6afeaec700 1 -- 172.21.15.16:0/323163224 shutdown_connections 2023-10-16T13:49:07.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.870+0000 7f6afeaec700 1 --2- 172.21.15.16:0/323163224 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6af815e0b0 0x7f6af80aec30 unknown :-1 s=CLOSED pgs=3204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:07.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.870+0000 7f6afeaec700 1 -- 172.21.15.16:0/323163224 >> 172.21.15.16:0/323163224 conn(0x7f6af80cc5b0 msgr2=0x7f6af80cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:07.876 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.870+0000 7f6afeaec700 1 -- 172.21.15.16:0/323163224 shutdown_connections 2023-10-16T13:49:07.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.870+0000 7f6afeaec700 1 -- 172.21.15.16:0/323163224 wait complete. 2023-10-16T13:49:07.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.871+0000 7f6afeaec700 1 Processor -- start 2023-10-16T13:49:07.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.871+0000 7f6afeaec700 1 -- start start 2023-10-16T13:49:07.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.872+0000 7f6afeaec700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6af80cb190 0x7f6af8158250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:07.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.872+0000 7f6afeaec700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6af8159c90 0x7f6af80c6c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:07.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.872+0000 7f6afeaec700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6af815e0b0 0x7f6af80c90f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:07.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.872+0000 7f6afeaec700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6af80c75e0 con 0x7f6af8159c90 2023-10-16T13:49:07.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.872+0000 7f6afeaec700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6af80c78c0 con 0x7f6af815e0b0 2023-10-16T13:49:07.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.872+0000 7f6afeaec700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6af80c7ba0 con 0x7f6af80cb190 2023-10-16T13:49:07.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.872+0000 7f6af7fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6af8159c90 0x7f6af80c6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:07.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.872+0000 7f6af7fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6af8159c90 0x7f6af80c6c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59096/0 (socket says 172.21.15.16:59096) 2023-10-16T13:49:07.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.872+0000 7f6af7fff700 1 -- 172.21.15.16:0/3058692028 learned_addr learned my addr 172.21.15.16:0/3058692028 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:07.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.872+0000 7f6afd089700 1 --2- 172.21.15.16:0/3058692028 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6af815e0b0 0x7f6af80c90f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:07.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.872+0000 7f6afc888700 1 --2- 172.21.15.16:0/3058692028 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6af80cb190 0x7f6af8158250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:07.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.872+0000 7f6af7fff700 1 -- 172.21.15.16:0/3058692028 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6af80cb190 msgr2=0x7f6af8158250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:07.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.872+0000 7f6af7fff700 1 --2- 172.21.15.16:0/3058692028 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6af80cb190 0x7f6af8158250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:07.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.872+0000 7f6af7fff700 1 -- 172.21.15.16:0/3058692028 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6af815e0b0 msgr2=0x7f6af80c90f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:07.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.872+0000 7f6af7fff700 1 --2- 172.21.15.16:0/3058692028 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6af815e0b0 0x7f6af80c90f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:07.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.872+0000 7f6af7fff700 1 -- 172.21.15.16:0/3058692028 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6af0009a70 con 0x7f6af8159c90 2023-10-16T13:49:07.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.872+0000 7f6afd089700 1 --2- 172.21.15.16:0/3058692028 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6af815e0b0 0x7f6af80c90f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:49:07.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.872+0000 7f6afc888700 1 --2- 172.21.15.16:0/3058692028 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6af80cb190 0x7f6af8158250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:49:07.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.872+0000 7f6af7fff700 1 --2- 172.21.15.16:0/3058692028 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6af8159c90 0x7f6af80c6c00 secure :-1 s=READY pgs=3205 cs=0 l=1 rev1=1 crypto rx=0x7f6ae4001eb0 tx=0x7f6ae400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:07.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.872+0000 7f6af57fa700 1 -- 172.21.15.16:0/3058692028 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ae4004c80 con 0x7f6af8159c90 2023-10-16T13:49:07.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.873+0000 7f6af57fa700 1 -- 172.21.15.16:0/3058692028 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ae4004de0 con 0x7f6af8159c90 2023-10-16T13:49:07.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.873+0000 7f6afeaec700 1 -- 172.21.15.16:0/3058692028 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6af80af2f0 con 0x7f6af8159c90 2023-10-16T13:49:07.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.873+0000 7f6af57fa700 1 -- 172.21.15.16:0/3058692028 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ae4007310 con 0x7f6af8159c90 2023-10-16T13:49:07.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.873+0000 7f6afeaec700 1 -- 172.21.15.16:0/3058692028 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6af80c8160 con 0x7f6af8159c90 2023-10-16T13:49:07.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.873+0000 7f6afeaec700 1 -- 172.21.15.16:0/3058692028 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6af80c8380 con 0x7f6af8159c90 2023-10-16T13:49:07.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.874+0000 7f6af57fa700 1 -- 172.21.15.16:0/3058692028 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 166) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6ae4007470 con 0x7f6af8159c90 2023-10-16T13:49:07.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.874+0000 7f6af57fa700 1 --2- 172.21.15.16:0/3058692028 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f6adc06b020 0x7f6adc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:07.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.874+0000 7f6af57fa700 1 -- 172.21.15.16:0/3058692028 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(278..278 src has 1..278) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f6ae4089630 con 0x7f6af8159c90 2023-10-16T13:49:07.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.874+0000 7f6afc888700 1 --2- 172.21.15.16:0/3058692028 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f6adc06b020 0x7f6adc06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:07.885 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.875+0000 7f6afc888700 1 --2- 172.21.15.16:0/3058692028 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f6adc06b020 0x7f6adc06d4e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6aec003fc0 tx=0x7f6aec009040 comp rx=0 tx=0).ready entity=mgr.13344 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:07.886 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:07.879+0000 7f6af57fa700 1 -- 172.21.15.16:0/3058692028 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6ae40544a0 con 0x7f6af8159c90 2023-10-16T13:49:08.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.128+0000 7f6afeaec700 1 -- 172.21.15.16:0/3058692028 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6af804ebb0 con 0x7f6af8159c90 2023-10-16T13:49:08.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.129+0000 7f6af57fa700 1 -- 172.21.15.16:0/3058692028 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 155 v155) v1 ==== 95+0+4926 (secure 0 0 0) 0x7f6ae4057ac0 con 0x7f6af8159c90 2023-10-16T13:49:08.132 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:49:08.132 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":155,"default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12280,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/1127700443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":1127700443},{"type":"v1","addr":"172.21.15.49:6834","nonce":1127700443}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":150},{"gid":13581,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/818113029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":818113029},{"type":"v1","addr":"172.21.15.16:6836","nonce":818113029}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149},{"gid":13584,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3762868123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":3762868123},{"type":"v1","addr":"172.21.15.16:6837","nonce":3762868123}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":150}],"filesystems":[{"mdsmap":{"epoch":155,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:49:02.107137+0000","modified":"2023-10-16T13:49:05.140377+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":13599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13599":{"gid":13599,"name":"b","rank":0,"incarnation":151,"state":"up:active","state_seq":3,"addr":"172.21.15.49:6835/2082691635","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":2082691635},{"type":"v1","addr":"172.21.15.49:6835","nonce":2082691635}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[38],"metadata_pool":37,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":18}]} 2023-10-16T13:49:08.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.131+0000 7f6afeaec700 1 -- 172.21.15.16:0/3058692028 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f6adc06b020 msgr2=0x7f6adc06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:08.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.131+0000 7f6afeaec700 1 --2- 172.21.15.16:0/3058692028 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f6adc06b020 0x7f6adc06d4e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6aec003fc0 tx=0x7f6aec009040 comp rx=0 tx=0).stop 2023-10-16T13:49:08.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.131+0000 7f6afeaec700 1 -- 172.21.15.16:0/3058692028 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6af8159c90 msgr2=0x7f6af80c6c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:08.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.131+0000 7f6afeaec700 1 --2- 172.21.15.16:0/3058692028 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6af8159c90 0x7f6af80c6c00 secure :-1 s=READY pgs=3205 cs=0 l=1 rev1=1 crypto rx=0x7f6ae4001eb0 tx=0x7f6ae400c8b0 comp rx=0 tx=0).stop 2023-10-16T13:49:08.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.131+0000 7f6afeaec700 1 -- 172.21.15.16:0/3058692028 shutdown_connections 2023-10-16T13:49:08.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.131+0000 7f6afeaec700 1 --2- 172.21.15.16:0/3058692028 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6af815e0b0 0x7f6af80c90f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:08.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.131+0000 7f6afeaec700 1 --2- 172.21.15.16:0/3058692028 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6af80cb190 0x7f6af8158250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:08.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.131+0000 7f6afeaec700 1 --2- 172.21.15.16:0/3058692028 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f6adc06b020 0x7f6adc06d4e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:08.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.131+0000 7f6afeaec700 1 --2- 172.21.15.16:0/3058692028 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6af8159c90 0x7f6af80c6c00 unknown :-1 s=CLOSED pgs=3205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:08.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.131+0000 7f6afeaec700 1 -- 172.21.15.16:0/3058692028 >> 172.21.15.16:0/3058692028 conn(0x7f6af80cc5b0 msgr2=0x7f6af80c2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:08.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.131+0000 7f6afeaec700 1 -- 172.21.15.16:0/3058692028 shutdown_connections 2023-10-16T13:49:08.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.131+0000 7f6afeaec700 1 -- 172.21.15.16:0/3058692028 wait complete. 2023-10-16T13:49:08.136 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 155 2023-10-16T13:49:08.547 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 155, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T13:49:02.107137+0000', 'modified': '2023-10-16T13:49:05.140377+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 13599}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_13599': {'gid': 13599, 'name': 'b', 'rank': 0, 'incarnation': 151, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.49:6835/2082691635', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6833', 'nonce': 2082691635}, {'type': 'v1', 'addr': '172.21.15.49:6835', 'nonce': 2082691635}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [38], 'metadata_pool': 37, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-16T13:49:08.548 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-16T13:49:08.549 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-10-16T13:49:08.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.680+0000 7f6421821700 1 Processor -- start 2023-10-16T13:49:08.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.680+0000 7f6421821700 1 -- start start 2023-10-16T13:49:08.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.681+0000 7f6421821700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f641c06cf10 0x7f641c0b3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:08.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.681+0000 7f6421821700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f641c0b3b80 0x7f641c159ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:08.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.681+0000 7f6421821700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f641c0b21c0 con 0x7f641c159c10 2023-10-16T13:49:08.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.681+0000 7f6421821700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f641c0b2b20 con 0x7f641c06cf10 2023-10-16T13:49:08.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.681+0000 7f6421821700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f641c071e40 con 0x7f641c0b3b80 2023-10-16T13:49:08.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.681+0000 7f641a7fc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f641c06cf10 0x7f641c0b3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:08.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.681+0000 7f641a7fc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f641c06cf10 0x7f641c0b3640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59104/0 (socket says 172.21.15.16:59104) 2023-10-16T13:49:08.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.681+0000 7f641affd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f641c159c10 0x7f641c06c800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48376/0 (socket says 172.21.15.16:48376) 2023-10-16T13:49:08.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.681+0000 7f641a7fc700 1 -- 172.21.15.16:0/2346240815 learned_addr learned my addr 172.21.15.16:0/2346240815 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:08.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.681+0000 7f641a7fc700 1 -- 172.21.15.16:0/2346240815 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f641c0b3b80 msgr2=0x7f641c159ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:08.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.681+0000 7f641a7fc700 1 --2- 172.21.15.16:0/2346240815 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f641c0b3b80 0x7f641c159ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:08.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.681+0000 7f641a7fc700 1 -- 172.21.15.16:0/2346240815 >> v1:172.21.15.16:6789/0 conn(0x7f641c159c10 legacy=0x7f641c06c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:08.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.681+0000 7f641a7fc700 1 -- 172.21.15.16:0/2346240815 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f641c0b2460 con 0x7f641c06cf10 2023-10-16T13:49:08.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.681+0000 7f6419ffb700 1 -- 172.21.15.16:0/2346240815 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 654739713 0 0) 0x7f641c0b21c0 con 0x7f641c159c10 2023-10-16T13:49:08.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.681+0000 7f641a7fc700 1 --2- 172.21.15.16:0/2346240815 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f641c06cf10 0x7f641c0b3640 secure :-1 s=READY pgs=3207 cs=0 l=1 rev1=1 crypto rx=0x7f640c0098f0 tx=0x7f640c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a8abcfe2ad139f2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:08.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.682+0000 7f6419ffb700 1 -- 172.21.15.16:0/2346240815 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f640c00b8a0 con 0x7f641c06cf10 2023-10-16T13:49:08.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.682+0000 7f6419ffb700 1 -- 172.21.15.16:0/2346240815 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f640c00ba00 con 0x7f641c06cf10 2023-10-16T13:49:08.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.682+0000 7f6419ffb700 1 -- 172.21.15.16:0/2346240815 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f640c00bd90 con 0x7f641c06cf10 2023-10-16T13:49:08.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.682+0000 7f6421821700 1 -- 172.21.15.16:0/2346240815 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f641c06cf10 msgr2=0x7f641c0b3640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:08.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.682+0000 7f6421821700 1 --2- 172.21.15.16:0/2346240815 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f641c06cf10 0x7f641c0b3640 secure :-1 s=READY pgs=3207 cs=0 l=1 rev1=1 crypto rx=0x7f640c0098f0 tx=0x7f640c009c00 comp rx=0 tx=0).stop 2023-10-16T13:49:08.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.682+0000 7f6421821700 1 -- 172.21.15.16:0/2346240815 shutdown_connections 2023-10-16T13:49:08.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.682+0000 7f6421821700 1 --2- 172.21.15.16:0/2346240815 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f641c0b3b80 0x7f641c159ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:08.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.682+0000 7f6421821700 1 --2- 172.21.15.16:0/2346240815 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f641c06cf10 0x7f641c0b3640 unknown :-1 s=CLOSED pgs=3207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:08.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.682+0000 7f6421821700 1 -- 172.21.15.16:0/2346240815 >> 172.21.15.16:0/2346240815 conn(0x7f641c0b4cd0 msgr2=0x7f641c06a7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:08.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.682+0000 7f6421821700 1 -- 172.21.15.16:0/2346240815 shutdown_connections 2023-10-16T13:49:08.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.682+0000 7f6421821700 1 -- 172.21.15.16:0/2346240815 wait complete. 2023-10-16T13:49:08.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.684+0000 7f6421821700 1 Processor -- start 2023-10-16T13:49:08.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.684+0000 7f6421821700 1 -- start start 2023-10-16T13:49:08.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.685+0000 7f6421821700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f641c06cf10 0x7f641c14ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:08.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.685+0000 7f6421821700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f641c0b3b80 0x7f641c1563c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:08.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.685+0000 7f6421821700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f641c159c10 0x7f641c0b1350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:08.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.685+0000 7f6421821700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f641c158d90 con 0x7f641c0b3b80 2023-10-16T13:49:08.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.685+0000 7f6421821700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f641c156d10 con 0x7f641c159c10 2023-10-16T13:49:08.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.685+0000 7f6421821700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f641c1583f0 con 0x7f641c06cf10 2023-10-16T13:49:08.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.685+0000 7f641a7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f641c0b3b80 0x7f641c1563c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:08.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.685+0000 7f641a7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f641c0b3b80 0x7f641c1563c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59110/0 (socket says 172.21.15.16:59110) 2023-10-16T13:49:08.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.685+0000 7f641a7fc700 1 -- 172.21.15.16:0/1451645459 learned_addr learned my addr 172.21.15.16:0/1451645459 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:08.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.685+0000 7f641affd700 1 --2- 172.21.15.16:0/1451645459 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f641c06cf10 0x7f641c14ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:08.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.685+0000 7f641b7fe700 1 --2- 172.21.15.16:0/1451645459 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f641c159c10 0x7f641c0b1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:08.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.685+0000 7f641a7fc700 1 -- 172.21.15.16:0/1451645459 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f641c06cf10 msgr2=0x7f641c14ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:08.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.685+0000 7f641a7fc700 1 --2- 172.21.15.16:0/1451645459 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f641c06cf10 0x7f641c14ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:08.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.685+0000 7f641a7fc700 1 -- 172.21.15.16:0/1451645459 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f641c159c10 msgr2=0x7f641c0b1350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:08.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.685+0000 7f641a7fc700 1 --2- 172.21.15.16:0/1451645459 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f641c159c10 0x7f641c0b1350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:08.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.685+0000 7f641a7fc700 1 -- 172.21.15.16:0/1451645459 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f640c009550 con 0x7f641c0b3b80 2023-10-16T13:49:08.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.685+0000 7f641b7fe700 1 --2- 172.21.15.16:0/1451645459 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f641c159c10 0x7f641c0b1350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:49:08.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.686+0000 7f641a7fc700 1 --2- 172.21.15.16:0/1451645459 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f641c0b3b80 0x7f641c1563c0 secure :-1 s=READY pgs=3208 cs=0 l=1 rev1=1 crypto rx=0x7f640c009c40 tx=0x7f640c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:08.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.686+0000 7f6403fff700 1 -- 172.21.15.16:0/1451645459 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f640c0038a0 con 0x7f641c0b3b80 2023-10-16T13:49:08.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.686+0000 7f6403fff700 1 -- 172.21.15.16:0/1451645459 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f640c01e070 con 0x7f641c0b3b80 2023-10-16T13:49:08.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.686+0000 7f6421821700 1 -- 172.21.15.16:0/1451645459 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f641c1586d0 con 0x7f641c0b3b80 2023-10-16T13:49:08.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.686+0000 7f6403fff700 1 -- 172.21.15.16:0/1451645459 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f640c003c20 con 0x7f641c0b3b80 2023-10-16T13:49:08.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.686+0000 7f6421821700 1 -- 172.21.15.16:0/1451645459 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f641c14d390 con 0x7f641c0b3b80 2023-10-16T13:49:08.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.687+0000 7f6403fff700 1 -- 172.21.15.16:0/1451645459 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 166) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f640c025480 con 0x7f641c0b3b80 2023-10-16T13:49:08.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.688+0000 7f6403fff700 1 --2- 172.21.15.16:0/1451645459 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f640806b020 0x7f640806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:08.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.688+0000 7f6403fff700 1 -- 172.21.15.16:0/1451645459 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(278..278 src has 1..278) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f640c020070 con 0x7f641c0b3b80 2023-10-16T13:49:08.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.688+0000 7f641affd700 1 --2- 172.21.15.16:0/1451645459 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f640806b020 0x7f640806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:08.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.688+0000 7f641affd700 1 --2- 172.21.15.16:0/1451645459 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f640806b020 0x7f640806d4e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6410000f60 tx=0x7f6410007c00 comp rx=0 tx=0).ready entity=mgr.13344 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:08.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.690+0000 7f642da0bb80 1 Processor -- start 2023-10-16T13:49:08.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.690+0000 7f642da0bb80 1 -- start start 2023-10-16T13:49:08.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.691+0000 7f642da0bb80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x559c0bdafb40 0x559c0bdaff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:08.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.691+0000 7f642da0bb80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559c0bdd6f30 0x559c0bdaf280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:08.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.691+0000 7f642da0bb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c0bdaaf00 0x559c0bdd6300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:08.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.691+0000 7f642da0bb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c0bd23c20 con 0x559c0bdaaf00 2023-10-16T13:49:08.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.691+0000 7f642da0bb80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c0bd08c10 con 0x559c0bdd6f30 2023-10-16T13:49:08.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.691+0000 7f642da0bb80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x559c0bd06530 con 0x559c0bdafb40 2023-10-16T13:49:08.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.691+0000 7f641b7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c0bdaaf00 0x559c0bdd6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:08.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.691+0000 7f641b7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c0bdaaf00 0x559c0bdd6300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59120/0 (socket says 172.21.15.16:59120) 2023-10-16T13:49:08.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.691+0000 7f641b7fe700 1 -- 172.21.15.16:0/3745651190 learned_addr learned my addr 172.21.15.16:0/3745651190 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:08.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.691+0000 7f641a7fc700 1 --2- 172.21.15.16:0/3745651190 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559c0bdd6f30 0x559c0bdaf280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:08.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.691+0000 7f641affd700 1 --2- 172.21.15.16:0/3745651190 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x559c0bdafb40 0x559c0bdaff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:08.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.691+0000 7f641b7fe700 1 -- 172.21.15.16:0/3745651190 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x559c0bdafb40 msgr2=0x559c0bdaff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:08.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.691+0000 7f641b7fe700 1 --2- 172.21.15.16:0/3745651190 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x559c0bdafb40 0x559c0bdaff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:08.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.691+0000 7f641b7fe700 1 -- 172.21.15.16:0/3745651190 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559c0bdd6f30 msgr2=0x559c0bdaf280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:08.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.691+0000 7f641b7fe700 1 --2- 172.21.15.16:0/3745651190 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559c0bdd6f30 0x559c0bdaf280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:08.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.691+0000 7f641b7fe700 1 -- 172.21.15.16:0/3745651190 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559c0bcbf3a0 con 0x559c0bdaaf00 2023-10-16T13:49:08.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.691+0000 7f641affd700 1 --2- 172.21.15.16:0/3745651190 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x559c0bdafb40 0x559c0bdaff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:49:08.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.692+0000 7f641b7fe700 1 --2- 172.21.15.16:0/3745651190 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c0bdaaf00 0x559c0bdd6300 secure :-1 s=READY pgs=3209 cs=0 l=1 rev1=1 crypto rx=0x7f6414007f00 tx=0x7f641400c5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b794d5f3ba847e08 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:08.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.692+0000 7f64017fa700 1 -- 172.21.15.16:0/3745651190 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f641400f640 con 0x559c0bdaaf00 2023-10-16T13:49:08.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.692+0000 7f64017fa700 1 -- 172.21.15.16:0/3745651190 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f641400f7a0 con 0x559c0bdaaf00 2023-10-16T13:49:08.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.692+0000 7f64017fa700 1 -- 172.21.15.16:0/3745651190 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f641400fb30 con 0x559c0bdaaf00 2023-10-16T13:49:08.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.692+0000 7f642da0bb80 1 -- 172.21.15.16:0/3745651190 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c0bdaaf00 msgr2=0x559c0bdd6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:08.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.692+0000 7f642da0bb80 1 --2- 172.21.15.16:0/3745651190 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c0bdaaf00 0x559c0bdd6300 secure :-1 s=READY pgs=3209 cs=0 l=1 rev1=1 crypto rx=0x7f6414007f00 tx=0x7f641400c5a0 comp rx=0 tx=0).stop 2023-10-16T13:49:08.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.692+0000 7f642da0bb80 1 -- 172.21.15.16:0/3745651190 shutdown_connections 2023-10-16T13:49:08.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.692+0000 7f642da0bb80 1 --2- 172.21.15.16:0/3745651190 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c0bdaaf00 0x559c0bdd6300 unknown :-1 s=CLOSED pgs=3209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:08.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.692+0000 7f642da0bb80 1 --2- 172.21.15.16:0/3745651190 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x559c0bdafb40 0x559c0bdaff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:08.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.692+0000 7f642da0bb80 1 --2- 172.21.15.16:0/3745651190 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559c0bdd6f30 0x559c0bdaf280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:08.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.692+0000 7f642da0bb80 1 -- 172.21.15.16:0/3745651190 >> 172.21.15.16:0/3745651190 conn(0x559c0bcdcd80 msgr2=0x559c0bdf7ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:08.708 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.693+0000 7f642da0bb80 1 -- 172.21.15.16:0/3745651190 shutdown_connections 2023-10-16T13:49:08.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.693+0000 7f642da0bb80 1 -- 172.21.15.16:0/3745651190 wait complete. 2023-10-16T13:49:08.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.693+0000 7f642da0bb80 1 -- start start 2023-10-16T13:49:08.709 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.693+0000 7f642da0bb80 1 Processor -- start 2023-10-16T13:49:08.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.694+0000 7f642da0bb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c0bdaaf00 0x559c0bddae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:08.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.694+0000 7f642da0bb80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559c0bdafb40 0x559c0bdd9420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:08.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.694+0000 7f642da0bb80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x559c0bdd6f30 0x559c0bdf65b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:08.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.694+0000 7f642da0bb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c0bd01e20 con 0x559c0bdaaf00 2023-10-16T13:49:08.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.694+0000 7f642da0bb80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c0bc9e6c0 con 0x559c0bdafb40 2023-10-16T13:49:08.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.694+0000 7f642da0bb80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x559c0bc9e360 con 0x559c0bdd6f30 2023-10-16T13:49:08.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.694+0000 7f642da0bb80 10 client.0 _ll_register_callbacks cb 0x559c0bafb7b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T13:49:08.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.694+0000 7f641affd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c0bdaaf00 0x559c0bddae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:08.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.694+0000 7f641affd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c0bdaaf00 0x559c0bddae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59122/0 (socket says 172.21.15.16:59122) 2023-10-16T13:49:08.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.694+0000 7f641affd700 1 -- 172.21.15.16:0/2540249289 learned_addr learned my addr 172.21.15.16:0/2540249289 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:08.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.694+0000 7f641a7fc700 1 --2- 172.21.15.16:0/2540249289 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559c0bdafb40 0x559c0bdd9420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:08.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.694+0000 7f641b7fe700 1 --2- 172.21.15.16:0/2540249289 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x559c0bdd6f30 0x559c0bdf65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:08.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.694+0000 7f641affd700 1 -- 172.21.15.16:0/2540249289 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x559c0bdd6f30 msgr2=0x559c0bdf65b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:08.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.694+0000 7f641affd700 1 --2- 172.21.15.16:0/2540249289 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x559c0bdd6f30 0x559c0bdf65b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:08.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.694+0000 7f641affd700 1 -- 172.21.15.16:0/2540249289 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559c0bdafb40 msgr2=0x559c0bdd9420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:08.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.694+0000 7f641affd700 1 --2- 172.21.15.16:0/2540249289 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559c0bdafb40 0x559c0bdd9420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:08.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.694+0000 7f641affd700 1 -- 172.21.15.16:0/2540249289 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6414007b30 con 0x559c0bdaaf00 2023-10-16T13:49:08.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.694+0000 7f641b7fe700 1 --2- 172.21.15.16:0/2540249289 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x559c0bdd6f30 0x559c0bdf65b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:49:08.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.695+0000 7f641affd700 1 --2- 172.21.15.16:0/2540249289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c0bdaaf00 0x559c0bddae20 secure :-1 s=READY pgs=3210 cs=0 l=1 rev1=1 crypto rx=0x7f6410015b80 tx=0x7f6410011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb51cf4522b999fd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:08.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.695+0000 7f63fa7fc700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2023-10-16T13:49:08.720 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.695+0000 7f63fa7fc700 1 -- 172.21.15.16:0/2540249289 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6410001740 con 0x559c0bdaaf00 2023-10-16T13:49:08.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.695+0000 7f63fa7fc700 1 -- 172.21.15.16:0/2540249289 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(278..278 src has 1..278) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f6410022c50 con 0x559c0bdaaf00 2023-10-16T13:49:08.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.695+0000 7f642da0bb80 1 -- 172.21.15.16:0/2540249289 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559c0bda25b0 con 0x559c0bdaaf00 2023-10-16T13:49:08.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.695+0000 7f63fa7fc700 1 -- 172.21.15.16:0/2540249289 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=155) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6410022e10 con 0x559c0bdaaf00 2023-10-16T13:49:08.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.695+0000 7f642da0bb80 10 client.13752 fetch_fsmap learned FSMap version 155 2023-10-16T13:49:08.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.695+0000 7f642da0bb80 1 -- 172.21.15.16:0/2540249289 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=155,monmap=2+}) v3 -- 0x559c0bb2cd90 con 0x559c0bdaaf00 2023-10-16T13:49:08.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.695+0000 7f63fa7fc700 1 -- 172.21.15.16:0/2540249289 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x559c0bda25b0 con 0x559c0bdaaf00 2023-10-16T13:49:08.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.695+0000 7f63fa7fc700 1 -- 172.21.15.16:0/2540249289 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 155) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f64100018a0 con 0x559c0bdaaf00 2023-10-16T13:49:08.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.695+0000 7f642da0bb80 10 client.13752 fetch_fsmap finished waiting for FSMap version 155 2023-10-16T13:49:08.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.695+0000 7f642da0bb80 10 client.13752 resolve_mds: resolved name 'b' to daemon mds.b 2023-10-16T13:49:08.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.695+0000 7f642da0bb80 20 client.13752 populate_metadata read hostname 'smithi016' 2023-10-16T13:49:08.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.695+0000 7f642da0bb80 1 --2- 172.21.15.16:0/2540249289 >> [v2:172.21.15.49:6833/2082691635,v1:172.21.15.49:6835/2082691635] conn(0x559c0bdaa030 0x559c0bdf2590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:08.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.695+0000 7f642da0bb80 4 client.13752 mds_command: new command op to 13599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T13:49:08.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.695+0000 7f642da0bb80 1 -- 172.21.15.16:0/2540249289 --> [v2:172.21.15.49:6833/2082691635,v1:172.21.15.49:6835/2082691635] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559c0bcfae00 con 0x559c0bdaa030 2023-10-16T13:49:08.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.695+0000 7f641a7fc700 1 --2- 172.21.15.16:0/2540249289 >> [v2:172.21.15.49:6833/2082691635,v1:172.21.15.49:6835/2082691635] conn(0x559c0bdaa030 0x559c0bdf2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:08.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.696+0000 7f63fa7fc700 10 client.13752 ms_handle_connect on v2:172.21.15.49:6833/2082691635 2023-10-16T13:49:08.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.696+0000 7f641a7fc700 1 --2- 172.21.15.16:0/2540249289 >> [v2:172.21.15.49:6833/2082691635,v1:172.21.15.49:6835/2082691635] conn(0x559c0bdaa030 0x559c0bdf2590 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d16ebe78357f67f server_cookie=879d583d37a50f48 in_seq=0 out_seq=0 2023-10-16T13:49:08.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.697+0000 7f63fa7fc700 1 -- 172.21.15.16:0/2540249289 <== mds.0 v2:172.21.15.49:6833/2082691635 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x559c0bcfae00 con 0x559c0bdaa030 2023-10-16T13:49:08.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.697+0000 7f63fa7fc700 10 client.13752 handle_command_reply: tid=0 2023-10-16T13:49:08.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.697+0000 7f642da0bb80 1 client.13752 shutdown 2023-10-16T13:49:08.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.697+0000 7f642da0bb80 1 -- 172.21.15.16:0/2540249289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c0bdaaf00 msgr2=0x559c0bddae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:08.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.697+0000 7f642da0bb80 1 --2- 172.21.15.16:0/2540249289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c0bdaaf00 0x559c0bddae20 secure :-1 s=READY pgs=3210 cs=0 l=1 rev1=1 crypto rx=0x7f6410015b80 tx=0x7f6410011000 comp rx=0 tx=0).stop 2023-10-16T13:49:08.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.697+0000 7f642da0bb80 1 -- 172.21.15.16:0/2540249289 shutdown_connections 2023-10-16T13:49:08.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.697+0000 7f642da0bb80 1 --2- 172.21.15.16:0/2540249289 >> [v2:172.21.15.49:6833/2082691635,v1:172.21.15.49:6835/2082691635] conn(0x559c0bdaa030 0x559c0bdf2590 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:08.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.697+0000 7f642da0bb80 1 --2- 172.21.15.16:0/2540249289 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x559c0bdd6f30 0x559c0bdf65b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:08.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.697+0000 7f642da0bb80 1 --2- 172.21.15.16:0/2540249289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c0bdaaf00 0x559c0bddae20 unknown :-1 s=CLOSED pgs=3210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:08.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.697+0000 7f63fa7fc700 0 client.13752 ms_handle_reset on v2:172.21.15.49:6833/2082691635 2023-10-16T13:49:08.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.697+0000 7f642da0bb80 1 --2- 172.21.15.16:0/2540249289 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559c0bdafb40 0x559c0bdd9420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:08.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.697+0000 7f642da0bb80 1 -- 172.21.15.16:0/2540249289 >> 172.21.15.16:0/2540249289 conn(0x559c0bcdcd80 msgr2=0x559c0bdaf1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:08.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.697+0000 7f642da0bb80 1 -- 172.21.15.16:0/2540249289 shutdown_connections 2023-10-16T13:49:08.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.697+0000 7f642da0bb80 1 -- 172.21.15.16:0/2540249289 wait complete. 2023-10-16T13:49:08.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.697+0000 7f642da0bb80 20 client.13752 trim_cache size 0 max 16384 2023-10-16T13:49:08.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.715+0000 7f642da0bb80 1 Processor -- start 2023-10-16T13:49:08.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.715+0000 7f642da0bb80 1 -- start start 2023-10-16T13:49:08.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.715+0000 7f642da0bb80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559c0bdaaf00 0x559c0bdea7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:08.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.715+0000 7f642da0bb80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x559c0bdafb40 0x559c0bdead30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:08.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.716+0000 7f642da0bb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c0bdd6f30 0x559c0bdadfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:08.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.716+0000 7f642da0bb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c0bb828d0 con 0x559c0bdd6f30 2023-10-16T13:49:08.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.716+0000 7f642da0bb80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c0bdd9d80 con 0x559c0bdaaf00 2023-10-16T13:49:08.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.716+0000 7f642da0bb80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x559c0be0bb00 con 0x559c0bdafb40 2023-10-16T13:49:08.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.716+0000 7f641b7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c0bdd6f30 0x559c0bdadfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:08.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.716+0000 7f641b7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c0bdd6f30 0x559c0bdadfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59134/0 (socket says 172.21.15.16:59134) 2023-10-16T13:49:08.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.716+0000 7f641b7fe700 1 -- 172.21.15.16:0/437513520 learned_addr learned my addr 172.21.15.16:0/437513520 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:08.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.716+0000 7f641a7fc700 1 --2- 172.21.15.16:0/437513520 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x559c0bdafb40 0x559c0bdead30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:08.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.716+0000 7f641affd700 1 --2- 172.21.15.16:0/437513520 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559c0bdaaf00 0x559c0bdea7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:08.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.716+0000 7f641b7fe700 1 -- 172.21.15.16:0/437513520 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x559c0bdafb40 msgr2=0x559c0bdead30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:08.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.716+0000 7f641b7fe700 1 --2- 172.21.15.16:0/437513520 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x559c0bdafb40 0x559c0bdead30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:08.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.716+0000 7f641b7fe700 1 -- 172.21.15.16:0/437513520 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559c0bdaaf00 msgr2=0x559c0bdea7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:08.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.716+0000 7f641b7fe700 1 --2- 172.21.15.16:0/437513520 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559c0bdaaf00 0x559c0bdea7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:08.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.716+0000 7f641b7fe700 1 -- 172.21.15.16:0/437513520 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559c0bdd9fb0 con 0x559c0bdd6f30 2023-10-16T13:49:08.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.716+0000 7f641b7fe700 1 --2- 172.21.15.16:0/437513520 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c0bdd6f30 0x559c0bdadfe0 secure :-1 s=READY pgs=3211 cs=0 l=1 rev1=1 crypto rx=0x7f6414013000 tx=0x7f641400a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=767ebd4d9b8322b1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:08.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.716+0000 7f6401ffb700 1 -- 172.21.15.16:0/437513520 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f641401e050 con 0x559c0bdd6f30 2023-10-16T13:49:08.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.716+0000 7f6401ffb700 1 -- 172.21.15.16:0/437513520 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f641401f040 con 0x559c0bdd6f30 2023-10-16T13:49:08.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.716+0000 7f6401ffb700 1 -- 172.21.15.16:0/437513520 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f641400f910 con 0x559c0bdd6f30 2023-10-16T13:49:08.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.717+0000 7f642da0bb80 1 -- 172.21.15.16:0/437513520 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c0bdd6f30 msgr2=0x559c0bdadfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:08.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.717+0000 7f642da0bb80 1 --2- 172.21.15.16:0/437513520 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c0bdd6f30 0x559c0bdadfe0 secure :-1 s=READY pgs=3211 cs=0 l=1 rev1=1 crypto rx=0x7f6414013000 tx=0x7f641400a000 comp rx=0 tx=0).stop 2023-10-16T13:49:08.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.717+0000 7f642da0bb80 1 -- 172.21.15.16:0/437513520 shutdown_connections 2023-10-16T13:49:08.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.717+0000 7f642da0bb80 1 --2- 172.21.15.16:0/437513520 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c0bdd6f30 0x559c0bdadfe0 unknown :-1 s=CLOSED pgs=3211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:08.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.717+0000 7f642da0bb80 1 --2- 172.21.15.16:0/437513520 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559c0bdaaf00 0x559c0bdea7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:08.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.717+0000 7f642da0bb80 1 --2- 172.21.15.16:0/437513520 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x559c0bdafb40 0x559c0bdead30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:08.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.717+0000 7f642da0bb80 1 -- 172.21.15.16:0/437513520 >> 172.21.15.16:0/437513520 conn(0x559c0bcdcd80 msgr2=0x559c0bdaa410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:08.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.717+0000 7f642da0bb80 1 -- 172.21.15.16:0/437513520 shutdown_connections 2023-10-16T13:49:08.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.717+0000 7f642da0bb80 1 -- 172.21.15.16:0/437513520 wait complete. 2023-10-16T13:49:08.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.718+0000 7f642da0bb80 1 -- start start 2023-10-16T13:49:08.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.718+0000 7f642da0bb80 1 Processor -- start 2023-10-16T13:49:08.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.718+0000 7f642da0bb80 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c0bdaaf00 0x559c0bdcfac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:08.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.718+0000 7f642da0bb80 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559c0bdafb40 0x559c0be638f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:08.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.718+0000 7f642da0bb80 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x559c0bdd6f30 0x559c0be63e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:08.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.718+0000 7f642da0bb80 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c0bdeb6e0 con 0x559c0bdaaf00 2023-10-16T13:49:08.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.718+0000 7f642da0bb80 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c0bdee240 con 0x559c0bdafb40 2023-10-16T13:49:08.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.718+0000 7f642da0bb80 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x559c0bdcd900 con 0x559c0bdd6f30 2023-10-16T13:49:08.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.719+0000 7f642da0bb80 10 client.0 _ll_register_callbacks cb 0x7f64100154a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T13:49:08.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.719+0000 7f641affd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c0bdaaf00 0x559c0bdcfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:08.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.719+0000 7f641affd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c0bdaaf00 0x559c0bdcfac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59136/0 (socket says 172.21.15.16:59136) 2023-10-16T13:49:08.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.719+0000 7f641affd700 1 -- 172.21.15.16:0/677186149 learned_addr learned my addr 172.21.15.16:0/677186149 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:08.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.719+0000 7f641b7fe700 1 --2- 172.21.15.16:0/677186149 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x559c0bdd6f30 0x559c0be63e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:08.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.719+0000 7f641a7fc700 1 --2- 172.21.15.16:0/677186149 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559c0bdafb40 0x559c0be638f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:08.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.719+0000 7f641affd700 1 -- 172.21.15.16:0/677186149 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x559c0bdd6f30 msgr2=0x559c0be63e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:08.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.719+0000 7f641affd700 1 --2- 172.21.15.16:0/677186149 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x559c0bdd6f30 0x559c0be63e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:08.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.719+0000 7f641affd700 1 -- 172.21.15.16:0/677186149 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559c0bdafb40 msgr2=0x559c0be638f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:08.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.719+0000 7f641affd700 1 --2- 172.21.15.16:0/677186149 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559c0bdafb40 0x559c0be638f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:08.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.719+0000 7f641affd700 1 -- 172.21.15.16:0/677186149 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559c0bcbf3a0 con 0x559c0bdaaf00 2023-10-16T13:49:08.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.719+0000 7f641affd700 1 --2- 172.21.15.16:0/677186149 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c0bdaaf00 0x559c0bdcfac0 secure :-1 s=READY pgs=3212 cs=0 l=1 rev1=1 crypto rx=0x7f6410007910 tx=0x7f641000a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2eae0ced8eedf855 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:08.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.719+0000 7f63fa7fc700 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2023-10-16T13:49:08.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.721+0000 7f63fa7fc700 1 -- 172.21.15.16:0/677186149 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6410001590 con 0x559c0bdaaf00 2023-10-16T13:49:08.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.721+0000 7f63fa7fc700 1 -- 172.21.15.16:0/677186149 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(278..278 src has 1..278) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f6410038c80 con 0x559c0bdaaf00 2023-10-16T13:49:08.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.721+0000 7f642da0bb80 1 -- 172.21.15.16:0/677186149 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f641400f7a0 con 0x559c0bdaaf00 2023-10-16T13:49:08.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.721+0000 7f63fa7fc700 1 -- 172.21.15.16:0/677186149 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=155) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6410038e40 con 0x559c0bdaaf00 2023-10-16T13:49:08.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.721+0000 7f642da0bb80 10 client.13758 fetch_fsmap learned FSMap version 155 2023-10-16T13:49:08.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.721+0000 7f642da0bb80 1 -- 172.21.15.16:0/677186149 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=155,monmap=2+}) v3 -- 0x559c0bb15c10 con 0x559c0bdaaf00 2023-10-16T13:49:08.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.721+0000 7f63fa7fc700 1 -- 172.21.15.16:0/677186149 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f641400f7a0 con 0x559c0bdaaf00 2023-10-16T13:49:08.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.721+0000 7f63fa7fc700 1 -- 172.21.15.16:0/677186149 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 155) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f64100016f0 con 0x559c0bdaaf00 2023-10-16T13:49:08.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.721+0000 7f642da0bb80 10 client.13758 fetch_fsmap finished waiting for FSMap version 155 2023-10-16T13:49:08.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.721+0000 7f642da0bb80 10 client.13758 resolve_mds: resolved name 'b' to daemon mds.b 2023-10-16T13:49:08.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.721+0000 7f642da0bb80 20 client.13758 populate_metadata read hostname 'smithi016' 2023-10-16T13:49:08.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.721+0000 7f642da0bb80 1 --2- 172.21.15.16:0/677186149 >> [v2:172.21.15.49:6833/2082691635,v1:172.21.15.49:6835/2082691635] conn(0x559c0bdedb00 0x559c0be64370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:08.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.721+0000 7f642da0bb80 4 client.13758 mds_command: new command op to 13599 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T13:49:08.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.721+0000 7f642da0bb80 1 -- 172.21.15.16:0/677186149 --> [v2:172.21.15.49:6833/2082691635,v1:172.21.15.49:6835/2082691635] -- command(tid 0: {"prefix": "status"}) v1 -- 0x559c0bcfab20 con 0x559c0bdedb00 2023-10-16T13:49:08.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.722+0000 7f641a7fc700 1 --2- 172.21.15.16:0/677186149 >> [v2:172.21.15.49:6833/2082691635,v1:172.21.15.49:6835/2082691635] conn(0x559c0bdedb00 0x559c0be64370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:08.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.722+0000 7f63fa7fc700 10 client.13758 ms_handle_connect on v2:172.21.15.49:6833/2082691635 2023-10-16T13:49:08.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.722+0000 7f641a7fc700 1 --2- 172.21.15.16:0/677186149 >> [v2:172.21.15.49:6833/2082691635,v1:172.21.15.49:6835/2082691635] conn(0x559c0bdedb00 0x559c0be64370 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4a740ab4e7070a4a server_cookie=d83f40ec6085382b in_seq=0 out_seq=0 2023-10-16T13:49:08.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.722+0000 7f63fa7fc700 1 -- 172.21.15.16:0/677186149 <== mds.0 v2:172.21.15.49:6833/2082691635 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+329 (crc 0 0 0) 0x559c0bcfab20 con 0x559c0bdedb00 2023-10-16T13:49:08.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.722+0000 7f63fa7fc700 10 client.13758 handle_command_reply: tid=0 2023-10-16T13:49:08.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.722+0000 7f642da0bb80 1 client.13758 shutdown 2023-10-16T13:49:08.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.723+0000 7f642da0bb80 1 -- 172.21.15.16:0/677186149 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c0bdaaf00 msgr2=0x559c0bdcfac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:08.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.723+0000 7f642da0bb80 1 --2- 172.21.15.16:0/677186149 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c0bdaaf00 0x559c0bdcfac0 secure :-1 s=READY pgs=3212 cs=0 l=1 rev1=1 crypto rx=0x7f6410007910 tx=0x7f641000a000 comp rx=0 tx=0).stop 2023-10-16T13:49:08.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.723+0000 7f642da0bb80 1 -- 172.21.15.16:0/677186149 shutdown_connections 2023-10-16T13:49:08.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.723+0000 7f642da0bb80 1 --2- 172.21.15.16:0/677186149 >> [v2:172.21.15.49:6833/2082691635,v1:172.21.15.49:6835/2082691635] conn(0x559c0bdedb00 0x559c0be64370 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:08.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.723+0000 7f642da0bb80 1 --2- 172.21.15.16:0/677186149 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x559c0bdd6f30 0x559c0be63e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:08.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.723+0000 7f642da0bb80 1 --2- 172.21.15.16:0/677186149 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559c0bdaaf00 0x559c0bdcfac0 unknown :-1 s=CLOSED pgs=3212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:08.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.723+0000 7f642da0bb80 1 --2- 172.21.15.16:0/677186149 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559c0bdafb40 0x559c0be638f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:08.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.723+0000 7f642da0bb80 1 -- 172.21.15.16:0/677186149 >> 172.21.15.16:0/677186149 conn(0x559c0bcdcd80 msgr2=0x559c0bdea7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:08.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.723+0000 7f63fa7fc700 0 client.13758 ms_handle_reset on v2:172.21.15.49:6833/2082691635 2023-10-16T13:49:08.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.723+0000 7f642da0bb80 1 -- 172.21.15.16:0/677186149 shutdown_connections 2023-10-16T13:49:08.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.723+0000 7f642da0bb80 1 -- 172.21.15.16:0/677186149 wait complete. 2023-10-16T13:49:08.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.723+0000 7f642da0bb80 20 client.13758 trim_cache size 0 max 16384 2023-10-16T13:49:08.749 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-10-16T13:49:08.749 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "5b28dfa1-b21f-474a-9b2e-9fb77af58254", 2023-10-16T13:49:08.749 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 0, 2023-10-16T13:49:08.750 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 13599, 2023-10-16T13:49:08.750 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2023-10-16T13:49:08.750 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2023-10-16T13:49:08.750 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2023-10-16T13:49:08.750 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 6.0293003470000004, 2023-10-16T13:49:08.750 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 155, 2023-10-16T13:49:08.751 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 278, 2023-10-16T13:49:08.751 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 278, 2023-10-16T13:49:08.751 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 10.914029115 2023-10-16T13:49:08.751 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-10-16T13:49:08.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.723+0000 7f6421821700 1 -- 172.21.15.16:0/1451645459 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f640806b020 msgr2=0x7f640806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:08.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.723+0000 7f6421821700 1 --2- 172.21.15.16:0/1451645459 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f640806b020 0x7f640806d4e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6410000f60 tx=0x7f6410007c00 comp rx=0 tx=0).stop 2023-10-16T13:49:08.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.723+0000 7f6421821700 1 -- 172.21.15.16:0/1451645459 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f641c0b3b80 msgr2=0x7f641c1563c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:08.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.723+0000 7f6421821700 1 --2- 172.21.15.16:0/1451645459 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f641c0b3b80 0x7f641c1563c0 secure :-1 s=READY pgs=3208 cs=0 l=1 rev1=1 crypto rx=0x7f640c009c40 tx=0x7f640c004750 comp rx=0 tx=0).stop 2023-10-16T13:49:08.753 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.723+0000 7f6421821700 1 -- 172.21.15.16:0/1451645459 shutdown_connections 2023-10-16T13:49:08.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.724+0000 7f6421821700 1 --2- 172.21.15.16:0/1451645459 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f641c159c10 0x7f641c0b1350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:08.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.724+0000 7f6421821700 1 --2- 172.21.15.16:0/1451645459 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f641c06cf10 0x7f641c14ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:08.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.724+0000 7f6421821700 1 --2- 172.21.15.16:0/1451645459 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f640806b020 0x7f640806d4e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:08.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.724+0000 7f6421821700 1 --2- 172.21.15.16:0/1451645459 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f641c0b3b80 0x7f641c1563c0 unknown :-1 s=CLOSED pgs=3208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:08.754 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.724+0000 7f6421821700 1 -- 172.21.15.16:0/1451645459 >> 172.21.15.16:0/1451645459 conn(0x7f641c0b4cd0 msgr2=0x7f641c06a7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:08.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.724+0000 7f6421821700 1 -- 172.21.15.16:0/1451645459 shutdown_connections 2023-10-16T13:49:08.755 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.724+0000 7f6421821700 1 -- 172.21.15.16:0/1451645459 wait complete. 2023-10-16T13:49:08.755 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T13:49:08.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.878+0000 7feebf8c8700 1 Processor -- start 2023-10-16T13:49:08.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.878+0000 7feebf8c8700 1 -- start start 2023-10-16T13:49:08.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.879+0000 7feebf8c8700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7feeb80cf4a0 0x7feeb80726e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:08.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.879+0000 7feebf8c8700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7feeb80b5a00 0x7feeb815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:08.890 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.879+0000 7feebf8c8700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feeb815bc80 con 0x7feeb80cf880 2023-10-16T13:49:08.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.879+0000 7feebf8c8700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feeb80b5e70 con 0x7feeb80b5a00 2023-10-16T13:49:08.891 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.879+0000 7feebf8c8700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feeb8072c20 con 0x7feeb80cf4a0 2023-10-16T13:49:08.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.879+0000 7feebde65700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7feeb80b5a00 0x7feeb815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:08.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.879+0000 7feebde65700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7feeb80b5a00 0x7feeb815a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59148/0 (socket says 172.21.15.16:59148) 2023-10-16T13:49:08.892 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.879+0000 7feebde65700 1 -- 172.21.15.16:0/341780144 learned_addr learned my addr 172.21.15.16:0/341780144 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:08.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.880+0000 7feebd664700 1 --2- 172.21.15.16:0/341780144 >> v2:172.21.15.49:3300/0 conn(0x7feeb80cf4a0 0x7feeb80726e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:08.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.880+0000 7feebde65700 1 -- 172.21.15.16:0/341780144 >> v2:172.21.15.49:3300/0 conn(0x7feeb80cf4a0 msgr2=0x7feeb80726e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:08.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.880+0000 7feebde65700 1 --2- 172.21.15.16:0/341780144 >> v2:172.21.15.49:3300/0 conn(0x7feeb80cf4a0 0x7feeb80726e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:08.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.880+0000 7feebde65700 1 -- 172.21.15.16:0/341780144 >> v1:172.21.15.49:6789/0 conn(0x7feeb80cf880 legacy=0x7feeb80b52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:08.894 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.880+0000 7feebde65700 1 -- 172.21.15.16:0/341780144 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feeb815b070 con 0x7feeb80b5a00 2023-10-16T13:49:08.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.880+0000 7feebde65700 1 --2- 172.21.15.16:0/341780144 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feeb80b5a00 0x7feeb815a090 secure :-1 s=READY pgs=3213 cs=0 l=1 rev1=1 crypto rx=0x7feeb80b2d20 tx=0x7feeb000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be2924e385c7e482 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:08.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.880+0000 7feeaffff700 1 -- 172.21.15.16:0/341780144 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeb000e070 con 0x7feeb80b5a00 2023-10-16T13:49:08.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.880+0000 7feeaffff700 1 -- 172.21.15.16:0/341780144 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feeb0003a20 con 0x7feeb80b5a00 2023-10-16T13:49:08.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.880+0000 7feeaffff700 1 -- 172.21.15.16:0/341780144 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeb0003dd0 con 0x7feeb80b5a00 2023-10-16T13:49:08.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.881+0000 7feebf8c8700 1 -- 172.21.15.16:0/341780144 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feeb80b5a00 msgr2=0x7feeb815a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:08.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.881+0000 7feebf8c8700 1 --2- 172.21.15.16:0/341780144 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feeb80b5a00 0x7feeb815a090 secure :-1 s=READY pgs=3213 cs=0 l=1 rev1=1 crypto rx=0x7feeb80b2d20 tx=0x7feeb000b330 comp rx=0 tx=0).stop 2023-10-16T13:49:08.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.881+0000 7feebf8c8700 1 -- 172.21.15.16:0/341780144 shutdown_connections 2023-10-16T13:49:08.897 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.881+0000 7feebf8c8700 1 --2- 172.21.15.16:0/341780144 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feeb80b5a00 0x7feeb815a090 unknown :-1 s=CLOSED pgs=3213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:08.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.881+0000 7feebf8c8700 1 --2- 172.21.15.16:0/341780144 >> v2:172.21.15.49:3300/0 conn(0x7feeb80cf4a0 0x7feeb80726e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:08.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.881+0000 7feebf8c8700 1 -- 172.21.15.16:0/341780144 >> 172.21.15.16:0/341780144 conn(0x7feeb80b1fd0 msgr2=0x7feeb80b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:08.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.881+0000 7feebf8c8700 1 -- 172.21.15.16:0/341780144 shutdown_connections 2023-10-16T13:49:08.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.881+0000 7feebf8c8700 1 -- 172.21.15.16:0/341780144 wait complete. 2023-10-16T13:49:08.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.882+0000 7feebf8c8700 1 Processor -- start 2023-10-16T13:49:08.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.882+0000 7feebf8c8700 1 -- start start 2023-10-16T13:49:08.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.883+0000 7feebf8c8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feeb80b5a00 0x7feeb81537e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:08.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.883+0000 7feebf8c8700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feeb80cf4a0 0x7feeb80b0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:08.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.883+0000 7feebf8c8700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7feeb80cf880 0x7feeb80b1390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:08.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.883+0000 7feebf8c8700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeb80ad110 con 0x7feeb80b5a00 2023-10-16T13:49:08.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.883+0000 7feebf8c8700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeb81f77a0 con 0x7feeb80cf4a0 2023-10-16T13:49:08.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.883+0000 7feebf8c8700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feeb80aca50 con 0x7feeb80cf880 2023-10-16T13:49:08.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.883+0000 7feebd664700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feeb80b5a00 0x7feeb81537e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:08.902 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.883+0000 7feebd664700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feeb80b5a00 0x7feeb81537e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59150/0 (socket says 172.21.15.16:59150) 2023-10-16T13:49:08.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.883+0000 7feebd664700 1 -- 172.21.15.16:0/2204471321 learned_addr learned my addr 172.21.15.16:0/2204471321 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:08.903 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.883+0000 7feebce63700 1 --2- 172.21.15.16:0/2204471321 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feeb80cf4a0 0x7feeb80b0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:08.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.883+0000 7feebde65700 1 --2- 172.21.15.16:0/2204471321 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7feeb80cf880 0x7feeb80b1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:08.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.883+0000 7feebd664700 1 -- 172.21.15.16:0/2204471321 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7feeb80cf880 msgr2=0x7feeb80b1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:08.904 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.883+0000 7feebd664700 1 --2- 172.21.15.16:0/2204471321 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7feeb80cf880 0x7feeb80b1390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:08.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.883+0000 7feebd664700 1 -- 172.21.15.16:0/2204471321 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feeb80cf4a0 msgr2=0x7feeb80b0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:08.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.883+0000 7feebd664700 1 --2- 172.21.15.16:0/2204471321 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feeb80cf4a0 0x7feeb80b0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:08.905 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.883+0000 7feebd664700 1 -- 172.21.15.16:0/2204471321 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feeb0009b30 con 0x7feeb80b5a00 2023-10-16T13:49:08.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.883+0000 7feebd664700 1 --2- 172.21.15.16:0/2204471321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feeb80b5a00 0x7feeb81537e0 secure :-1 s=READY pgs=3214 cs=0 l=1 rev1=1 crypto rx=0x7feea800e810 tx=0x7feea800eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:08.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.883+0000 7feeadffb700 1 -- 172.21.15.16:0/2204471321 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea800c070 con 0x7feeb80b5a00 2023-10-16T13:49:08.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.884+0000 7feeadffb700 1 -- 172.21.15.16:0/2204471321 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feea8013610 con 0x7feeb80b5a00 2023-10-16T13:49:08.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.884+0000 7feebf8c8700 1 -- 172.21.15.16:0/2204471321 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feeb80acd30 con 0x7feeb80b5a00 2023-10-16T13:49:08.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.884+0000 7feeadffb700 1 -- 172.21.15.16:0/2204471321 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea80139c0 con 0x7feeb80b5a00 2023-10-16T13:49:08.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.884+0000 7feebf8c8700 1 -- 172.21.15.16:0/2204471321 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feeb81f6a70 con 0x7feeb80b5a00 2023-10-16T13:49:08.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.885+0000 7feebf8c8700 1 -- 172.21.15.16:0/2204471321 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feeb80ad380 con 0x7feeb80b5a00 2023-10-16T13:49:08.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.885+0000 7feeadffb700 1 -- 172.21.15.16:0/2204471321 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 166) v1 ==== 89331+0+0 (secure 0 0 0) 0x7feea8013b20 con 0x7feeb80b5a00 2023-10-16T13:49:08.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.885+0000 7feeadffb700 1 --2- 172.21.15.16:0/2204471321 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fee9c06b020 0x7fee9c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:08.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.885+0000 7feeadffb700 1 -- 172.21.15.16:0/2204471321 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(278..278 src has 1..278) v4 ==== 6157+0+0 (secure 0 0 0) 0x7feea8089730 con 0x7feeb80b5a00 2023-10-16T13:49:08.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.886+0000 7feebce63700 1 --2- 172.21.15.16:0/2204471321 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fee9c06b020 0x7fee9c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:08.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.886+0000 7feebce63700 1 --2- 172.21.15.16:0/2204471321 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fee9c06b020 0x7fee9c06d4e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7feeb81f69d0 tx=0x7feeb40090e0 comp rx=0 tx=0).ready entity=mgr.13344 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:08.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:08.890+0000 7feeadffb700 1 -- 172.21.15.16:0/2204471321 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7feea8054410 con 0x7feeb80b5a00 2023-10-16T13:49:09.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:09.142+0000 7feebf8c8700 1 -- 172.21.15.16:0/2204471321 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7feeb80ceb70 con 0x7feeb80b5a00 2023-10-16T13:49:09.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:09.142+0000 7feeadffb700 1 -- 172.21.15.16:0/2204471321 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v155) v1 ==== 53+0+73 (secure 0 0 0) 0x7feea8057a30 con 0x7feeb80b5a00 2023-10-16T13:49:09.145 INFO:teuthology.orchestra.run.smithi016.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-16T13:49:09.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:09.144+0000 7feebf8c8700 1 -- 172.21.15.16:0/2204471321 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fee9c06b020 msgr2=0x7fee9c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:09.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:09.144+0000 7feebf8c8700 1 --2- 172.21.15.16:0/2204471321 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fee9c06b020 0x7fee9c06d4e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7feeb81f69d0 tx=0x7feeb40090e0 comp rx=0 tx=0).stop 2023-10-16T13:49:09.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:09.144+0000 7feebf8c8700 1 -- 172.21.15.16:0/2204471321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feeb80b5a00 msgr2=0x7feeb81537e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:09.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:09.144+0000 7feebf8c8700 1 --2- 172.21.15.16:0/2204471321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feeb80b5a00 0x7feeb81537e0 secure :-1 s=READY pgs=3214 cs=0 l=1 rev1=1 crypto rx=0x7feea800e810 tx=0x7feea800eb20 comp rx=0 tx=0).stop 2023-10-16T13:49:09.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:09.144+0000 7feebf8c8700 1 -- 172.21.15.16:0/2204471321 shutdown_connections 2023-10-16T13:49:09.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:09.144+0000 7feebf8c8700 1 --2- 172.21.15.16:0/2204471321 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7feeb80cf880 0x7feeb80b1390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:09.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:09.144+0000 7feebf8c8700 1 --2- 172.21.15.16:0/2204471321 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fee9c06b020 0x7fee9c06d4e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:09.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:09.144+0000 7feebf8c8700 1 --2- 172.21.15.16:0/2204471321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feeb80b5a00 0x7feeb81537e0 unknown :-1 s=CLOSED pgs=3214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:09.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:09.144+0000 7feebf8c8700 1 --2- 172.21.15.16:0/2204471321 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feeb80cf4a0 0x7feeb80b0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:09.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:09.144+0000 7feebf8c8700 1 -- 172.21.15.16:0/2204471321 >> 172.21.15.16:0/2204471321 conn(0x7feeb80b1fd0 msgr2=0x7feeb80b4fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:09.148 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:09.144+0000 7feebf8c8700 1 -- 172.21.15.16:0/2204471321 shutdown_connections 2023-10-16T13:49:09.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:09.144+0000 7feebf8c8700 1 -- 172.21.15.16:0/2204471321 wait complete. 2023-10-16T13:49:09.158 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-16T13:49:09.159 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T13:49:09.159 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi016.front.sepia.ceph.com 2023-10-16T13:49:09.159 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-16T13:49:09.160 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-16T13:49:09.160 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-16T13:49:09.160 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-16T13:49:09.161 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T13:49:09.161 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T13:49:09.162 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:49:09.162 DEBUG:teuthology.orchestra.run.smithi016:> ip addr 2023-10-16T13:49:09.180 INFO:teuthology.orchestra.run.smithi016.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T13:49:09.180 INFO:teuthology.orchestra.run.smithi016.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T13:49:09.180 INFO:teuthology.orchestra.run.smithi016.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T13:49:09.180 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T13:49:09.180 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 ::1/128 scope host 2023-10-16T13:49:09.181 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T13:49:09.181 INFO:teuthology.orchestra.run.smithi016.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T13:49:09.181 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e0 brd ff:ff:ff:ff:ff:ff 2023-10-16T13:49:09.181 INFO:teuthology.orchestra.run.smithi016.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T13:49:09.181 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e1 brd ff:ff:ff:ff:ff:ff 2023-10-16T13:49:09.182 INFO:teuthology.orchestra.run.smithi016.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T13:49:09.182 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:64 brd ff:ff:ff:ff:ff:ff 2023-10-16T13:49:09.182 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.21.15.16/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-16T13:49:09.182 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft 36387sec preferred_lft 36387sec 2023-10-16T13:49:09.182 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::ec4:7aff:febd:1564/64 scope link 2023-10-16T13:49:09.183 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T13:49:09.183 INFO:teuthology.orchestra.run.smithi016.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-16T13:49:09.183 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:65 brd ff:ff:ff:ff:ff:ff 2023-10-16T13:49:09.183 INFO:teuthology.orchestra.run.smithi016.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-16T13:49:09.183 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 36:cf:ea:25:db:f7 brd ff:ff:ff:ff:ff:ff 2023-10-16T13:49:09.184 INFO:teuthology.orchestra.run.smithi016.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-16T13:49:09.184 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T13:49:09.184 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::cfc:c5ff:fe52:a786/64 scope link 2023-10-16T13:49:09.184 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T13:49:09.184 INFO:teuthology.orchestra.run.smithi016.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T13:49:09.184 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 6e:62:9d:a4:43:23 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T13:49:09.185 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::6c62:9dff:fea4:4323/64 scope link 2023-10-16T13:49:09.185 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T13:49:09.185 INFO:teuthology.orchestra.run.smithi016.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-16T13:49:09.185 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 36:cf:ea:25:db:f7 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-16T13:49:09.185 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::34cf:eaff:fe25:dbf7/64 scope link 2023-10-16T13:49:09.186 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2023-10-16T13:49:09.186 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:49:09.186 DEBUG:teuthology.orchestra.run.smithi016:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T13:49:09.246 INFO:teuthology.orchestra.run.smithi016.stdout:1 2023-10-16T13:49:09.247 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:49:09.248 DEBUG:teuthology.orchestra.run.smithi016:> route 2023-10-16T13:49:09.306 INFO:teuthology.orchestra.run.smithi016.stdout:Kernel IP routing table 2023-10-16T13:49:09.306 INFO:teuthology.orchestra.run.smithi016.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T13:49:09.306 INFO:teuthology.orchestra.run.smithi016.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-16T13:49:09.306 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-16T13:49:09.307 INFO:teuthology.orchestra.run.smithi016.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T13:49:09.307 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:49:09.307 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec bash -c ' 2023-10-16T13:49:09.308 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2023-10-16T13:49:09.308 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-16T13:49:09.309 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-16T13:49:09.309 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-16T13:49:09.309 DEBUG:teuthology.orchestra.run.smithi016:> ') 2023-10-16T13:49:09.413 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:49:09.413 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2023-10-16T13:49:09.430 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-16T13:49:09.431 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-16T13:49:09.431 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T13:49:09.431 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:49:09.432 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set brx.0 up 2023-10-16T13:49:09.497 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T13:49:09.498 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-16T13:49:09.554 INFO:teuthology.orchestra.run.smithi016.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-16T13:49:09.555 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T13:49:09.555 DEBUG:teuthology.orchestra.run.smithi016:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-16T13:49:09.611 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-16T13:49:09.612 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:49:09.612 DEBUG:teuthology.orchestra.run.smithi016:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T13:49:09.699 INFO:teuthology.orchestra.run.smithi016.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T13:49:09.699 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T13:49:09.699 INFO:teuthology.orchestra.run.smithi016.stdout:invalid new device string format 2023-10-16T13:49:09.708 INFO:teuthology.orchestra.run.smithi016.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T13:49:09.708 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T13:49:09.708 INFO:teuthology.orchestra.run.smithi016.stdout:invalid new device string format 2023-10-16T13:49:09.708 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: resolved to: "172.21.15.16:3300,172.21.15.49:3300,172.21.15.49:3301" 2023-10-16T13:49:09.709 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T13:49:09.782 INFO:teuthology.orchestra.run.smithi016.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-16T13:49:09.782 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-16T13:49:09.783 INFO:teuthology.orchestra.run.smithi016.stdout:invalid new device string format 2023-10-16T13:49:09.783 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: resolved to: "172.21.15.16:3300,172.21.15.49:3300,172.21.15.49:3301" 2023-10-16T13:49:09.783 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-16T13:49:09.783 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: resolved to: "172.21.15.16:6789,172.21.15.49:6789,172.21.15.49:6790" 2023-10-16T13:49:09.783 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.16:6789,172.21.15.49:6789,172.21.15.49:6790:/ 2023-10-16T13:49:09.784 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=5b28dfa1-b21f-474a-9b2e-9fb77af58254" will pass to kernel 2023-10-16T13:49:09.785 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-16T13:49:09.785 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:49:09.785 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-16T13:49:09.813 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:49:09.814 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-16T13:49:09.903 INFO:teuthology.orchestra.run.smithi016.stdout:client12376 2023-10-16T13:49:09.904 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T13:49:09.904 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:49:09.921 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-16T13:49:10.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.085+0000 7f1de08a3700 1 Processor -- start 2023-10-16T13:49:10.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.085+0000 7f1de08a3700 1 -- start start 2023-10-16T13:49:10.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.086+0000 7f1de08a3700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f1ddc15b280 0x7f1ddc1613e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:10.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.086+0000 7f1de08a3700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1ddc0cc8a0 0x7f1ddc15e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:10.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.086+0000 7f1de08a3700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ddc161c30 con 0x7f1ddc06a7e0 2023-10-16T13:49:10.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.086+0000 7f1de08a3700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ddc161f10 con 0x7f1ddc15b280 2023-10-16T13:49:10.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.086+0000 7f1de08a3700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ddc1621f0 con 0x7f1ddc0cc8a0 2023-10-16T13:49:10.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.086+0000 7f1dda59c700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f1ddc15b280 0x7f1ddc1613e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:10.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.086+0000 7f1dd9d9b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1ddc0cc8a0 0x7f1ddc15e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:10.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.086+0000 7f1ddad9d700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7f1ddc06a7e0 0x7f1ddc0d02b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:55622/0 (socket says 172.21.15.16:55622) 2023-10-16T13:49:10.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.086+0000 7f1dda59c700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f1ddc15b280 0x7f1ddc1613e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:58224/0 (socket says 172.21.15.16:58224) 2023-10-16T13:49:10.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.086+0000 7f1ddad9d700 1 -- 172.21.15.16:0/957689847 learned_addr learned my addr 172.21.15.16:0/957689847 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:49:10.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.087+0000 7f1dda59c700 1 -- 172.21.15.16:0/957689847 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1ddc0cc8a0 msgr2=0x7f1ddc15e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:10.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.087+0000 7f1dda59c700 1 --2- 172.21.15.16:0/957689847 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1ddc0cc8a0 0x7f1ddc15e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:10.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.087+0000 7f1dda59c700 1 -- 172.21.15.16:0/957689847 >> v1:172.21.15.49:6789/0 conn(0x7f1ddc06a7e0 legacy=0x7f1ddc0d02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:10.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.087+0000 7f1dd959a700 1 -- 172.21.15.16:0/957689847 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3915762528 0 0) 0x7f1ddc161c30 con 0x7f1ddc06a7e0 2023-10-16T13:49:10.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.087+0000 7f1dda59c700 1 -- 172.21.15.16:0/957689847 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ddc15df00 con 0x7f1ddc15b280 2023-10-16T13:49:10.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.087+0000 7f1dda59c700 1 --2- 172.21.15.16:0/957689847 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1ddc15b280 0x7f1ddc1613e0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f1dd0009840 tx=0x7f1dd0009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a66837f76ec8dad server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:10.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.087+0000 7f1dd959a700 1 -- 172.21.15.16:0/957689847 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd0011830 con 0x7f1ddc15b280 2023-10-16T13:49:10.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.087+0000 7f1dd959a700 1 -- 172.21.15.16:0/957689847 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dd0011990 con 0x7f1ddc15b280 2023-10-16T13:49:10.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.087+0000 7f1dd959a700 1 -- 172.21.15.16:0/957689847 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd0011d20 con 0x7f1ddc15b280 2023-10-16T13:49:10.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.087+0000 7f1de08a3700 1 -- 172.21.15.16:0/957689847 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1ddc15b280 msgr2=0x7f1ddc1613e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:10.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.087+0000 7f1de08a3700 1 --2- 172.21.15.16:0/957689847 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1ddc15b280 0x7f1ddc1613e0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f1dd0009840 tx=0x7f1dd0009b50 comp rx=0 tx=0).stop 2023-10-16T13:49:10.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.088+0000 7f1de08a3700 1 -- 172.21.15.16:0/957689847 shutdown_connections 2023-10-16T13:49:10.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.088+0000 7f1de08a3700 1 --2- 172.21.15.16:0/957689847 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1ddc15b280 0x7f1ddc1613e0 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:10.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.088+0000 7f1de08a3700 1 --2- 172.21.15.16:0/957689847 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1ddc0cc8a0 0x7f1ddc15e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:10.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.088+0000 7f1de08a3700 1 -- 172.21.15.16:0/957689847 >> 172.21.15.16:0/957689847 conn(0x7f1ddc0b1fd0 msgr2=0x7f1ddc0b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:10.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.088+0000 7f1de08a3700 1 -- 172.21.15.16:0/957689847 shutdown_connections 2023-10-16T13:49:10.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.088+0000 7f1de08a3700 1 -- 172.21.15.16:0/957689847 wait complete. 2023-10-16T13:49:10.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.089+0000 7f1de08a3700 1 Processor -- start 2023-10-16T13:49:10.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.089+0000 7f1de08a3700 1 -- start start 2023-10-16T13:49:10.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.090+0000 7f1de08a3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ddc06a7e0 0x7f1ddc0c7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:10.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.090+0000 7f1de08a3700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1ddc0cc8a0 0x7f1ddc1f6e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:10.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.090+0000 7f1de08a3700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1ddc15b280 0x7f1ddc152280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:10.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.090+0000 7f1de08a3700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ddc1f7360 con 0x7f1ddc06a7e0 2023-10-16T13:49:10.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.090+0000 7f1de08a3700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ddc155ec0 con 0x7f1ddc0cc8a0 2023-10-16T13:49:10.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.090+0000 7f1de08a3700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ddc1527c0 con 0x7f1ddc15b280 2023-10-16T13:49:10.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.090+0000 7f1dda59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ddc06a7e0 0x7f1ddc0c7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:10.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.090+0000 7f1dda59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ddc06a7e0 0x7f1ddc0c7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59168/0 (socket says 172.21.15.16:59168) 2023-10-16T13:49:10.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.090+0000 7f1dda59c700 1 -- 172.21.15.16:0/1775695321 learned_addr learned my addr 172.21.15.16:0/1775695321 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:10.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.090+0000 7f1dd9d9b700 1 --2- 172.21.15.16:0/1775695321 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1ddc0cc8a0 0x7f1ddc1f6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:10.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.090+0000 7f1ddad9d700 1 --2- 172.21.15.16:0/1775695321 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1ddc15b280 0x7f1ddc152280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:10.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.090+0000 7f1dda59c700 1 -- 172.21.15.16:0/1775695321 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1ddc15b280 msgr2=0x7f1ddc152280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:10.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.090+0000 7f1dda59c700 1 --2- 172.21.15.16:0/1775695321 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1ddc15b280 0x7f1ddc152280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:10.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.090+0000 7f1dda59c700 1 -- 172.21.15.16:0/1775695321 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1ddc0cc8a0 msgr2=0x7f1ddc1f6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:10.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.090+0000 7f1dda59c700 1 --2- 172.21.15.16:0/1775695321 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1ddc0cc8a0 0x7f1ddc1f6e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:10.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.090+0000 7f1dda59c700 1 -- 172.21.15.16:0/1775695321 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dd00094a0 con 0x7f1ddc06a7e0 2023-10-16T13:49:10.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.090+0000 7f1dd9d9b700 1 --2- 172.21.15.16:0/1775695321 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1ddc0cc8a0 0x7f1ddc1f6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:49:10.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.090+0000 7f1dda59c700 1 --2- 172.21.15.16:0/1775695321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ddc06a7e0 0x7f1ddc0c7be0 secure :-1 s=READY pgs=3215 cs=0 l=1 rev1=1 crypto rx=0x7f1dd0009fd0 tx=0x7f1dd0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:10.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.090+0000 7f1dcaffd700 1 -- 172.21.15.16:0/1775695321 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd00036a0 con 0x7f1ddc06a7e0 2023-10-16T13:49:10.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.090+0000 7f1dcaffd700 1 -- 172.21.15.16:0/1775695321 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dd001e070 con 0x7f1ddc06a7e0 2023-10-16T13:49:10.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.091+0000 7f1dcaffd700 1 -- 172.21.15.16:0/1775695321 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd0003a60 con 0x7f1ddc06a7e0 2023-10-16T13:49:10.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.091+0000 7f1de08a3700 1 -- 172.21.15.16:0/1775695321 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ddc0c82e0 con 0x7f1ddc06a7e0 2023-10-16T13:49:10.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.091+0000 7f1de08a3700 1 -- 172.21.15.16:0/1775695321 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ddc0c88a0 con 0x7f1ddc06a7e0 2023-10-16T13:49:10.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.091+0000 7f1de08a3700 1 -- 172.21.15.16:0/1775695321 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ddc15c530 con 0x7f1ddc06a7e0 2023-10-16T13:49:10.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.092+0000 7f1dcaffd700 1 -- 172.21.15.16:0/1775695321 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 166) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1dd0003bc0 con 0x7f1ddc06a7e0 2023-10-16T13:49:10.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.092+0000 7f1dcaffd700 1 --2- 172.21.15.16:0/1775695321 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f1dbc06b020 0x7f1dbc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:10.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.092+0000 7f1dcaffd700 1 -- 172.21.15.16:0/1775695321 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(278..278 src has 1..278) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f1dd008a8b0 con 0x7f1ddc06a7e0 2023-10-16T13:49:10.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.092+0000 7f1dd9d9b700 1 --2- 172.21.15.16:0/1775695321 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f1dbc06b020 0x7f1dbc06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:10.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.093+0000 7f1dd9d9b700 1 --2- 172.21.15.16:0/1775695321 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f1dbc06b020 0x7f1dbc06d4e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc1f62e0 tx=0x7f1dcc008040 comp rx=0 tx=0).ready entity=mgr.13344 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:10.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.097+0000 7f1dcaffd700 1 -- 172.21.15.16:0/1775695321 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1dd0055720 con 0x7f1ddc06a7e0 2023-10-16T13:49:10.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.333+0000 7f1de08a3700 1 -- 172.21.15.16:0/1775695321 --> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f1ddc161920 con 0x7f1dbc06b020 2023-10-16T13:49:10.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.334+0000 7f1dcaffd700 1 -- 172.21.15.16:0/1775695321 <== mgr.13344 v2:172.21.15.16:6832/1257392887 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f1dcc004030 con 0x7f1dbc06b020 2023-10-16T13:49:10.337 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2023-10-16T13:49:10.337 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-10-16T13:49:10.337 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cephfs" 2023-10-16T13:49:10.338 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-10-16T13:49:10.338 INFO:teuthology.orchestra.run.smithi016.stdout:] 2023-10-16T13:49:10.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.337+0000 7f1de08a3700 1 -- 172.21.15.16:0/1775695321 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f1dbc06b020 msgr2=0x7f1dbc06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:10.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.337+0000 7f1de08a3700 1 --2- 172.21.15.16:0/1775695321 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f1dbc06b020 0x7f1dbc06d4e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc1f62e0 tx=0x7f1dcc008040 comp rx=0 tx=0).stop 2023-10-16T13:49:10.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.337+0000 7f1de08a3700 1 -- 172.21.15.16:0/1775695321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ddc06a7e0 msgr2=0x7f1ddc0c7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:10.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.337+0000 7f1de08a3700 1 --2- 172.21.15.16:0/1775695321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ddc06a7e0 0x7f1ddc0c7be0 secure :-1 s=READY pgs=3215 cs=0 l=1 rev1=1 crypto rx=0x7f1dd0009fd0 tx=0x7f1dd0004750 comp rx=0 tx=0).stop 2023-10-16T13:49:10.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.337+0000 7f1de08a3700 1 -- 172.21.15.16:0/1775695321 shutdown_connections 2023-10-16T13:49:10.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.337+0000 7f1de08a3700 1 --2- 172.21.15.16:0/1775695321 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1ddc15b280 0x7f1ddc152280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:10.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.337+0000 7f1de08a3700 1 --2- 172.21.15.16:0/1775695321 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f1dbc06b020 0x7f1dbc06d4e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:10.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.337+0000 7f1de08a3700 1 --2- 172.21.15.16:0/1775695321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ddc06a7e0 0x7f1ddc0c7be0 unknown :-1 s=CLOSED pgs=3215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:10.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.337+0000 7f1de08a3700 1 --2- 172.21.15.16:0/1775695321 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1ddc0cc8a0 0x7f1ddc1f6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:10.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.337+0000 7f1de08a3700 1 -- 172.21.15.16:0/1775695321 >> 172.21.15.16:0/1775695321 conn(0x7f1ddc0b1fd0 msgr2=0x7f1ddc0719a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:10.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.337+0000 7f1de08a3700 1 -- 172.21.15.16:0/1775695321 shutdown_connections 2023-10-16T13:49:10.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.337+0000 7f1de08a3700 1 -- 172.21.15.16:0/1775695321 wait complete. 2023-10-16T13:49:10.352 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-16T13:49:10.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.479+0000 7fc4c173a700 1 Processor -- start 2023-10-16T13:49:10.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.479+0000 7fc4c173a700 1 -- start start 2023-10-16T13:49:10.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.480+0000 7fc4c173a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc4bc0ce6a0 0x7fc4bc06c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:10.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.480+0000 7fc4c173a700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4bc0b20f0 con 0x7fc4bc0b2620 2023-10-16T13:49:10.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.480+0000 7fc4c173a700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4bc0af1f0 con 0x7fc4bc0d2110 2023-10-16T13:49:10.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.480+0000 7fc4c173a700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4bc0b1d70 con 0x7fc4bc0ce6a0 2023-10-16T13:49:10.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.480+0000 7fc4bb7fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc4bc0b2620 0x7fc4bc0b1660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48454/0 (socket says 172.21.15.16:48454) 2023-10-16T13:49:10.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.480+0000 7fc4bb7fe700 1 -- 172.21.15.16:0/3972905209 learned_addr learned my addr 172.21.15.16:0/3972905209 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:49:10.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.480+0000 7fc4baffd700 1 --2- 172.21.15.16:0/3972905209 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc4bc0ce6a0 0x7fc4bc06c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:10.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.480+0000 7fc4b9ffb700 1 -- 172.21.15.16:0/3972905209 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3859597991 0 0) 0x7fc4bc0b20f0 con 0x7fc4bc0b2620 2023-10-16T13:49:10.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.480+0000 7fc4b9ffb700 1 -- 172.21.15.16:0/3972905209 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4a40035b0 con 0x7fc4bc0b2620 2023-10-16T13:49:10.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.480+0000 7fc4b9ffb700 1 -- 172.21.15.16:0/3972905209 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2275982220 0 0) 0x7fc4bc0af1f0 con 0x7fc4bc0d2110 2023-10-16T13:49:10.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.480+0000 7fc4b9ffb700 1 -- 172.21.15.16:0/3972905209 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4bc0b20f0 con 0x7fc4bc0d2110 2023-10-16T13:49:10.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.480+0000 7fc4b9ffb700 1 -- 172.21.15.16:0/3972905209 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 306118085 0 0) 0x7fc4a40035b0 con 0x7fc4bc0b2620 2023-10-16T13:49:10.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.480+0000 7fc4b9ffb700 1 -- 172.21.15.16:0/3972905209 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc4bc0ce6a0 msgr2=0x7fc4bc06c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:10.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.480+0000 7fc4b9ffb700 1 --2- 172.21.15.16:0/3972905209 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc4bc0ce6a0 0x7fc4bc06c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:10.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.480+0000 7fc4b9ffb700 1 -- 172.21.15.16:0/3972905209 >> v1:172.21.15.49:6789/0 conn(0x7fc4bc0d2110 legacy=0x7fc4bc0b0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:10.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.480+0000 7fc4b9ffb700 1 -- 172.21.15.16:0/3972905209 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4bc0d03d0 con 0x7fc4bc0b2620 2023-10-16T13:49:10.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.480+0000 7fc4b9ffb700 1 -- 172.21.15.16:0/3972905209 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7fc4b4001970 con 0x7fc4bc0b2620 2023-10-16T13:49:10.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.480+0000 7fc4baffd700 1 --2- 172.21.15.16:0/3972905209 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc4bc0ce6a0 0x7fc4bc06c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:49:10.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.480+0000 7fc4b9ffb700 1 -- 172.21.15.16:0/3972905209 >> v1:172.21.15.16:6789/0 conn(0x7fc4bc0b2620 legacy=0x7fc4bc0b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:10.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.480+0000 7fc4b9ffb700 1 --2- 172.21.15.16:0/3972905209 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc4a4003bc0 0x7fc4a4003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:10.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.480+0000 7fc4b9ffb700 1 --2- 172.21.15.16:0/3972905209 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc4bc0ce6a0 0x7fc4a4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:10.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.480+0000 7fc4b9ffb700 1 --2- 172.21.15.16:0/3972905209 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4a4004d90 0x7fc4a4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:10.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.481+0000 7fc4b9ffb700 1 -- 172.21.15.16:0/3972905209 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4bc0b1d70 con 0x7fc4a4004d90 2023-10-16T13:49:10.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.481+0000 7fc4b9ffb700 1 -- 172.21.15.16:0/3972905209 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a400bac0 con 0x7fc4a4003bc0 2023-10-16T13:49:10.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.481+0000 7fc4b9ffb700 1 -- 172.21.15.16:0/3972905209 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a400bc80 con 0x7fc4bc0ce6a0 2023-10-16T13:49:10.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.481+0000 7fc4bb7fe700 1 --2- 172.21.15.16:0/3972905209 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4a4004d90 0x7fc4a4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:10.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.481+0000 7fc4baffd700 1 --2- 172.21.15.16:0/3972905209 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc4a4003bc0 0x7fc4a4003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:10.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.481+0000 7fc4ba7fc700 1 --2- 172.21.15.16:0/3972905209 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc4bc0ce6a0 0x7fc4a4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:10.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.481+0000 7fc4bb7fe700 1 -- 172.21.15.16:0/3972905209 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc4bc0ce6a0 msgr2=0x7fc4a4004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:10.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.481+0000 7fc4bb7fe700 1 --2- 172.21.15.16:0/3972905209 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc4bc0ce6a0 0x7fc4a4004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:10.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.481+0000 7fc4bb7fe700 1 -- 172.21.15.16:0/3972905209 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc4a4003bc0 msgr2=0x7fc4a4003fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:10.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.481+0000 7fc4bb7fe700 1 --2- 172.21.15.16:0/3972905209 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc4a4003bc0 0x7fc4a4003fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:10.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.481+0000 7fc4bb7fe700 1 -- 172.21.15.16:0/3972905209 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4ac0094a0 con 0x7fc4a4004d90 2023-10-16T13:49:10.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.481+0000 7fc4bb7fe700 1 --2- 172.21.15.16:0/3972905209 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4a4004d90 0x7fc4a4005210 secure :-1 s=READY pgs=3217 cs=0 l=1 rev1=1 crypto rx=0x7fc4b400a680 tx=0x7fc4a4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9eedcfefa263137 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:10.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.481+0000 7fc4b9ffb700 1 -- 172.21.15.16:0/3972905209 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b400a140 con 0x7fc4a4004d90 2023-10-16T13:49:10.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.481+0000 7fc4b9ffb700 1 -- 172.21.15.16:0/3972905209 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4b40072d0 con 0x7fc4a4004d90 2023-10-16T13:49:10.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.481+0000 7fc4b9ffb700 1 -- 172.21.15.16:0/3972905209 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b4007680 con 0x7fc4a4004d90 2023-10-16T13:49:10.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.482+0000 7fc4c173a700 1 -- 172.21.15.16:0/3972905209 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4a4004d90 msgr2=0x7fc4a4005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:10.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.482+0000 7fc4c173a700 1 --2- 172.21.15.16:0/3972905209 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4a4004d90 0x7fc4a4005210 secure :-1 s=READY pgs=3217 cs=0 l=1 rev1=1 crypto rx=0x7fc4b400a680 tx=0x7fc4a4005880 comp rx=0 tx=0).stop 2023-10-16T13:49:10.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.482+0000 7fc4bb7fe700 1 -- 172.21.15.16:0/3972905209 reap_dead start 2023-10-16T13:49:10.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.482+0000 7fc4c173a700 1 -- 172.21.15.16:0/3972905209 shutdown_connections 2023-10-16T13:49:10.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.482+0000 7fc4c173a700 1 -- 172.21.15.16:0/3972905209 >> 172.21.15.16:0/3972905209 conn(0x7fc4bc0d15f0 msgr2=0x7fc4bc0d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:10.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.482+0000 7fc4c173a700 1 -- 172.21.15.16:0/3972905209 shutdown_connections 2023-10-16T13:49:10.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.482+0000 7fc4c173a700 1 -- 172.21.15.16:0/3972905209 wait complete. 2023-10-16T13:49:10.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.483+0000 7fc4c173a700 1 Processor -- start 2023-10-16T13:49:10.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.483+0000 7fc4c173a700 1 -- start start 2023-10-16T13:49:10.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.484+0000 7fc4c173a700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc4bc1f2740 0x7fc4bc1f39c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:10.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.484+0000 7fc4c173a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4bc0c86c0 0x7fc4bc1f0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:10.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.484+0000 7fc4c173a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc4bc1f2b20 0x7fc4bc0cab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:10.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.484+0000 7fc4c173a700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4bc1f3f00 con 0x7fc4bc0c86c0 2023-10-16T13:49:10.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.484+0000 7fc4c173a700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4bc15e370 con 0x7fc4bc1f2740 2023-10-16T13:49:10.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.484+0000 7fc4c173a700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4bc1f10c0 con 0x7fc4bc1f2b20 2023-10-16T13:49:10.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.484+0000 7fc4ba7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4bc0c86c0 0x7fc4bc1f0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:10.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.484+0000 7fc4ba7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4bc0c86c0 0x7fc4bc1f0b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59190/0 (socket says 172.21.15.16:59190) 2023-10-16T13:49:10.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.484+0000 7fc4ba7fc700 1 -- 172.21.15.16:0/1220367345 learned_addr learned my addr 172.21.15.16:0/1220367345 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:10.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.484+0000 7fc4baffd700 1 --2- 172.21.15.16:0/1220367345 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc4bc1f2740 0x7fc4bc1f39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:10.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.484+0000 7fc4bb7fe700 1 --2- 172.21.15.16:0/1220367345 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc4bc1f2b20 0x7fc4bc0cab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:10.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.484+0000 7fc4ba7fc700 1 -- 172.21.15.16:0/1220367345 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc4bc1f2b20 msgr2=0x7fc4bc0cab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:10.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.484+0000 7fc4ba7fc700 1 --2- 172.21.15.16:0/1220367345 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc4bc1f2b20 0x7fc4bc0cab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:10.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.484+0000 7fc4ba7fc700 1 -- 172.21.15.16:0/1220367345 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc4bc1f2740 msgr2=0x7fc4bc1f39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:10.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.484+0000 7fc4ba7fc700 1 --2- 172.21.15.16:0/1220367345 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc4bc1f2740 0x7fc4bc1f39c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:10.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.484+0000 7fc4ba7fc700 1 -- 172.21.15.16:0/1220367345 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4bc0c8ba0 con 0x7fc4bc0c86c0 2023-10-16T13:49:10.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.484+0000 7fc4bb7fe700 1 --2- 172.21.15.16:0/1220367345 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc4bc1f2b20 0x7fc4bc0cab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:49:10.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.485+0000 7fc4ba7fc700 1 --2- 172.21.15.16:0/1220367345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4bc0c86c0 0x7fc4bc1f0b80 secure :-1 s=READY pgs=3218 cs=0 l=1 rev1=1 crypto rx=0x7fc4b000f910 tx=0x7fc4b000fcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:10.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.485+0000 7fc4a37fe700 1 -- 172.21.15.16:0/1220367345 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b00015a0 con 0x7fc4bc0c86c0 2023-10-16T13:49:10.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.485+0000 7fc4a37fe700 1 -- 172.21.15.16:0/1220367345 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4b00052e0 con 0x7fc4bc0c86c0 2023-10-16T13:49:10.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.485+0000 7fc4c173a700 1 -- 172.21.15.16:0/1220367345 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4bc15e5c0 con 0x7fc4bc0c86c0 2023-10-16T13:49:10.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.485+0000 7fc4a37fe700 1 -- 172.21.15.16:0/1220367345 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b0003680 con 0x7fc4bc0c86c0 2023-10-16T13:49:10.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.485+0000 7fc4c173a700 1 -- 172.21.15.16:0/1220367345 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4bc15eb80 con 0x7fc4bc0c86c0 2023-10-16T13:49:10.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.486+0000 7fc4c173a700 1 -- 172.21.15.16:0/1220367345 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4bc06cf90 con 0x7fc4bc0c86c0 2023-10-16T13:49:10.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.486+0000 7fc4a37fe700 1 -- 172.21.15.16:0/1220367345 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 166) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc4b00037e0 con 0x7fc4bc0c86c0 2023-10-16T13:49:10.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.487+0000 7fc4a37fe700 1 --2- 172.21.15.16:0/1220367345 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fc4a806b020 0x7fc4a806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:10.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.487+0000 7fc4a37fe700 1 -- 172.21.15.16:0/1220367345 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(278..278 src has 1..278) v4 ==== 6157+0+0 (secure 0 0 0) 0x7fc4b0089910 con 0x7fc4bc0c86c0 2023-10-16T13:49:10.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.487+0000 7fc4baffd700 1 --2- 172.21.15.16:0/1220367345 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fc4a806b020 0x7fc4a806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:10.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.487+0000 7fc4baffd700 1 --2- 172.21.15.16:0/1220367345 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fc4a806b020 0x7fc4a806d4e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc4bc09f530 tx=0x7fc4ac008040 comp rx=0 tx=0).ready entity=mgr.13344 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:10.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.491+0000 7fc4a37fe700 1 -- 172.21.15.16:0/1220367345 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc4b0054780 con 0x7fc4bc0c86c0 2023-10-16T13:49:10.759 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:10.757+0000 7fc4c173a700 1 -- 172.21.15.16:0/1220367345 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7fc4bc15ed40 con 0x7fc4bc0c86c0 2023-10-16T13:49:11.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:11.740+0000 7fc4a37fe700 1 -- 172.21.15.16:0/1220367345 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v167) v1 ==== 92+0+0 (secure 0 0 0) 0x7fc4b0057da0 con 0x7fc4bc0c86c0 2023-10-16T13:49:11.746 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:11.743+0000 7fc4c173a700 1 -- 172.21.15.16:0/1220367345 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fc4a806b020 msgr2=0x7fc4a806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:11.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:11.743+0000 7fc4c173a700 1 --2- 172.21.15.16:0/1220367345 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fc4a806b020 0x7fc4a806d4e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc4bc09f530 tx=0x7fc4ac008040 comp rx=0 tx=0).stop 2023-10-16T13:49:11.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:11.743+0000 7fc4c173a700 1 -- 172.21.15.16:0/1220367345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4bc0c86c0 msgr2=0x7fc4bc1f0b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:11.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:11.743+0000 7fc4c173a700 1 --2- 172.21.15.16:0/1220367345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4bc0c86c0 0x7fc4bc1f0b80 secure :-1 s=READY pgs=3218 cs=0 l=1 rev1=1 crypto rx=0x7fc4b000f910 tx=0x7fc4b000fcd0 comp rx=0 tx=0).stop 2023-10-16T13:49:11.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:11.744+0000 7fc4c173a700 1 -- 172.21.15.16:0/1220367345 shutdown_connections 2023-10-16T13:49:11.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:11.744+0000 7fc4c173a700 1 --2- 172.21.15.16:0/1220367345 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc4bc1f2b20 0x7fc4bc0cab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:11.747 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:11.744+0000 7fc4c173a700 1 --2- 172.21.15.16:0/1220367345 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc4bc1f2740 0x7fc4bc1f39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:11.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:11.744+0000 7fc4c173a700 1 --2- 172.21.15.16:0/1220367345 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fc4a806b020 0x7fc4a806d4e0 secure :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc4bc09f530 tx=0x7fc4ac008040 comp rx=0 tx=0).stop 2023-10-16T13:49:11.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:11.744+0000 7fc4c173a700 1 --2- 172.21.15.16:0/1220367345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4bc0c86c0 0x7fc4bc1f0b80 unknown :-1 s=CLOSED pgs=3218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:11.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:11.744+0000 7fc4c173a700 1 -- 172.21.15.16:0/1220367345 >> 172.21.15.16:0/1220367345 conn(0x7fc4bc0d15f0 msgr2=0x7fc4bc0704e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:11.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:11.746+0000 7fc4c173a700 1 -- 172.21.15.16:0/1220367345 shutdown_connections 2023-10-16T13:49:11.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:11.746+0000 7fc4c173a700 1 -- 172.21.15.16:0/1220367345 wait complete. 2023-10-16T13:49:11.768 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-16T13:49:11.890 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:49:11.887+0000 7fd0a9f02200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T13:49:11.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:11.903+0000 7f32632a4700 1 Processor -- start 2023-10-16T13:49:11.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:11.903+0000 7f32632a4700 1 -- start start 2023-10-16T13:49:11.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:11.904+0000 7f32632a4700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f325c15b2c0 0x7f325c161340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:11.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:11.904+0000 7f32632a4700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f325c06e570 0x7f325c15e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:11.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:11.904+0000 7f32632a4700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f325c161880 con 0x7f325c06a7e0 2023-10-16T13:49:11.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:11.904+0000 7f32632a4700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f325c15b6a0 con 0x7f325c06e570 2023-10-16T13:49:11.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:11.904+0000 7f32632a4700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f325c161b20 con 0x7f325c15b2c0 2023-10-16T13:49:11.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:11.904+0000 7f3261841700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f325c06a7e0 0x7f325c071a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48500/0 (socket says 172.21.15.16:48500) 2023-10-16T13:49:11.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:11.904+0000 7f3261841700 1 -- 172.21.15.16:0/2451727613 learned_addr learned my addr 172.21.15.16:0/2451727613 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:49:11.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:11.904+0000 7f3261040700 1 --2- 172.21.15.16:0/2451727613 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f325c15b2c0 0x7f325c161340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:11.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:11.905+0000 7f326083f700 1 --2- 172.21.15.16:0/2451727613 >> v2:172.21.15.49:3300/0 conn(0x7f325c06e570 0x7f325c15e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:11.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:11.905+0000 7f3253fff700 1 -- 172.21.15.16:0/2451727613 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3524248135 0 0) 0x7f325c161880 con 0x7f325c06a7e0 2023-10-16T13:49:11.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:11.905+0000 7f3253fff700 1 -- 172.21.15.16:0/2451727613 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32480035b0 con 0x7f325c06a7e0 2023-10-16T13:49:11.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:11.905+0000 7f3261040700 1 -- 172.21.15.16:0/2451727613 >> v2:172.21.15.49:3300/0 conn(0x7f325c06e570 msgr2=0x7f325c15e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:11.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:11.905+0000 7f3261040700 1 --2- 172.21.15.16:0/2451727613 >> v2:172.21.15.49:3300/0 conn(0x7f325c06e570 0x7f325c15e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:11.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:11.905+0000 7f3261040700 1 -- 172.21.15.16:0/2451727613 >> v1:172.21.15.16:6789/0 conn(0x7f325c06a7e0 legacy=0x7f325c071a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:11.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:11.905+0000 7f3261040700 1 -- 172.21.15.16:0/2451727613 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f325c161e00 con 0x7f325c15b2c0 2023-10-16T13:49:11.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:11.905+0000 7f3261040700 1 --2- 172.21.15.16:0/2451727613 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f325c15b2c0 0x7f325c161340 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f3258009840 tx=0x7f3258009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a43023dc185b2c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:11.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:11.905+0000 7f3253fff700 1 -- 172.21.15.16:0/2451727613 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3258011830 con 0x7f325c15b2c0 2023-10-16T13:49:11.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:11.905+0000 7f3253fff700 1 -- 172.21.15.16:0/2451727613 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3258011990 con 0x7f325c15b2c0 2023-10-16T13:49:11.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:11.905+0000 7f3253fff700 1 -- 172.21.15.16:0/2451727613 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3258011d20 con 0x7f325c15b2c0 2023-10-16T13:49:11.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:11.906+0000 7f32632a4700 1 -- 172.21.15.16:0/2451727613 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f325c15b2c0 msgr2=0x7f325c161340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:11.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:11.906+0000 7f32632a4700 1 --2- 172.21.15.16:0/2451727613 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f325c15b2c0 0x7f325c161340 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f3258009840 tx=0x7f3258009b50 comp rx=0 tx=0).stop 2023-10-16T13:49:11.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:11.906+0000 7f32632a4700 1 -- 172.21.15.16:0/2451727613 shutdown_connections 2023-10-16T13:49:11.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:11.906+0000 7f32632a4700 1 --2- 172.21.15.16:0/2451727613 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f325c15b2c0 0x7f325c161340 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:11.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:11.906+0000 7f32632a4700 1 --2- 172.21.15.16:0/2451727613 >> v2:172.21.15.49:3300/0 conn(0x7f325c06e570 0x7f325c15e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:11.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:11.906+0000 7f32632a4700 1 -- 172.21.15.16:0/2451727613 >> 172.21.15.16:0/2451727613 conn(0x7f325c0b1f40 msgr2=0x7f325c0b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:11.914 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:11.906+0000 7fb48b882200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T13:49:11.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:11.906+0000 7f32632a4700 1 -- 172.21.15.16:0/2451727613 shutdown_connections 2023-10-16T13:49:11.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:11.906+0000 7f32632a4700 1 -- 172.21.15.16:0/2451727613 wait complete. 2023-10-16T13:49:11.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:11.907+0000 7f32632a4700 1 Processor -- start 2023-10-16T13:49:11.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:11.907+0000 7f32632a4700 1 -- start start 2023-10-16T13:49:11.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:11.908+0000 7f32632a4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f325c06a7e0 0x7f325c0ca1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:11.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:11.908+0000 7f32632a4700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f325c06e570 0x7f325c160170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:11.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:11.908+0000 7f32632a4700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f325c15b2c0 0x7f325c0c9600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:11.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:11.908+0000 7f32632a4700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f325c1614e0 con 0x7f325c06a7e0 2023-10-16T13:49:11.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:11.908+0000 7f32632a4700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f325c0ca730 con 0x7f325c15b2c0 2023-10-16T13:49:11.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:11.908+0000 7f32632a4700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f325c160c40 con 0x7f325c06e570 2023-10-16T13:49:11.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:11.908+0000 7f3261040700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f325c06a7e0 0x7f325c0ca1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:11.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:11.908+0000 7f3261040700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f325c06a7e0 0x7f325c0ca1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59228/0 (socket says 172.21.15.16:59228) 2023-10-16T13:49:11.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:11.908+0000 7f3261040700 1 -- 172.21.15.16:0/2424543352 learned_addr learned my addr 172.21.15.16:0/2424543352 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:11.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:11.908+0000 7f326083f700 1 --2- 172.21.15.16:0/2424543352 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f325c06e570 0x7f325c160170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:11.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:11.908+0000 7f3261841700 1 --2- 172.21.15.16:0/2424543352 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f325c15b2c0 0x7f325c0c9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:11.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:11.908+0000 7f3261040700 1 -- 172.21.15.16:0/2424543352 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f325c06e570 msgr2=0x7f325c160170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:11.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:11.908+0000 7f3261040700 1 --2- 172.21.15.16:0/2424543352 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f325c06e570 0x7f325c160170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:11.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:11.908+0000 7f3261040700 1 -- 172.21.15.16:0/2424543352 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f325c15b2c0 msgr2=0x7f325c0c9600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:11.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:11.908+0000 7f3261040700 1 --2- 172.21.15.16:0/2424543352 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f325c15b2c0 0x7f325c0c9600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:11.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:11.908+0000 7f3261040700 1 -- 172.21.15.16:0/2424543352 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32580094a0 con 0x7f325c06a7e0 2023-10-16T13:49:11.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:11.909+0000 7f3261040700 1 --2- 172.21.15.16:0/2424543352 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f325c06a7e0 0x7f325c0ca1f0 secure :-1 s=READY pgs=3222 cs=0 l=1 rev1=1 crypto rx=0x7f3258009b90 tx=0x7f3258004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:11.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:11.909+0000 7f3251ffb700 1 -- 172.21.15.16:0/2424543352 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3258004510 con 0x7f325c06a7e0 2023-10-16T13:49:11.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:11.909+0000 7f3251ffb700 1 -- 172.21.15.16:0/2424543352 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32580036a0 con 0x7f325c06a7e0 2023-10-16T13:49:11.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:11.909+0000 7f32632a4700 1 -- 172.21.15.16:0/2424543352 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f325c160f20 con 0x7f325c06a7e0 2023-10-16T13:49:11.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:11.909+0000 7f3251ffb700 1 -- 172.21.15.16:0/2424543352 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3258003a50 con 0x7f325c06a7e0 2023-10-16T13:49:11.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:11.909+0000 7f32632a4700 1 -- 172.21.15.16:0/2424543352 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f325c0c79e0 con 0x7f325c06a7e0 2023-10-16T13:49:11.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:11.911+0000 7f3251ffb700 1 -- 172.21.15.16:0/2424543352 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 167) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f3258003bb0 con 0x7f325c06a7e0 2023-10-16T13:49:11.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:11.910+0000 7f32632a4700 1 -- 172.21.15.16:0/2424543352 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f325c0c7bd0 con 0x7f325c06a7e0 2023-10-16T13:49:11.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:11.911+0000 7f3251ffb700 1 --2- 172.21.15.16:0/2424543352 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f324006b070 0x7f324006d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:11.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:11.911+0000 7f326083f700 1 -- 172.21.15.16:0/2424543352 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f324006b070 msgr2=0x7f324006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1257392887 2023-10-16T13:49:11.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:11.911+0000 7f3251ffb700 1 -- 172.21.15.16:0/2424543352 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(278..278 src has 1..278) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f325808abe0 con 0x7f325c06a7e0 2023-10-16T13:49:11.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:11.911+0000 7f326083f700 1 --2- 172.21.15.16:0/2424543352 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f324006b070 0x7f324006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:49:11.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:11.920+0000 7f3251ffb700 1 -- 172.21.15.16:0/2424543352 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f32580559d0 con 0x7f325c06a7e0 2023-10-16T13:49:11.959 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:49:11.957+0000 7fd0a9f02200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T13:49:11.982 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:11.980+0000 7fb48b882200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T13:49:12.034 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:49:12.032+0000 7fd0a9f02200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T13:49:12.058 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:12.055+0000 7fb48b882200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T13:49:12.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.112+0000 7f326083f700 1 -- 172.21.15.16:0/2424543352 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f324006b070 msgr2=0x7f324006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1257392887 2023-10-16T13:49:12.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.112+0000 7f326083f700 1 --2- 172.21.15.16:0/2424543352 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f324006b070 0x7f324006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:49:12.115 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:49:12.112+0000 7fd0a9f02200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T13:49:12.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.125+0000 7f32632a4700 1 -- 172.21.15.16:0/2424543352 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f325c04ebb0 con 0x7f325c06a7e0 2023-10-16T13:49:12.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.128+0000 7f3251ffb700 1 -- 172.21.15.16:0/2424543352 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v95)=0 v95) v1 ==== 138+0+0 (secure 0 0 0) 0x7f3258003e60 con 0x7f325c06a7e0 2023-10-16T13:49:12.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.130+0000 7f32632a4700 1 -- 172.21.15.16:0/2424543352 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f324006b070 msgr2=0x7f324006d530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:49:12.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.130+0000 7f32632a4700 1 --2- 172.21.15.16:0/2424543352 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f324006b070 0x7f324006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:12.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.130+0000 7f32632a4700 1 -- 172.21.15.16:0/2424543352 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f325c06a7e0 msgr2=0x7f325c0ca1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:12.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.130+0000 7f32632a4700 1 --2- 172.21.15.16:0/2424543352 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f325c06a7e0 0x7f325c0ca1f0 secure :-1 s=READY pgs=3222 cs=0 l=1 rev1=1 crypto rx=0x7f3258009b90 tx=0x7f3258004750 comp rx=0 tx=0).stop 2023-10-16T13:49:12.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.130+0000 7f32632a4700 1 -- 172.21.15.16:0/2424543352 shutdown_connections 2023-10-16T13:49:12.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.130+0000 7f32632a4700 1 --2- 172.21.15.16:0/2424543352 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f325c15b2c0 0x7f325c0c9600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:12.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.130+0000 7f32632a4700 1 --2- 172.21.15.16:0/2424543352 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f324006b070 0x7f324006d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:12.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.130+0000 7f32632a4700 1 --2- 172.21.15.16:0/2424543352 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f325c06a7e0 0x7f325c0ca1f0 unknown :-1 s=CLOSED pgs=3222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:12.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.130+0000 7f32632a4700 1 --2- 172.21.15.16:0/2424543352 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f325c06e570 0x7f325c160170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:12.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.130+0000 7f32632a4700 1 -- 172.21.15.16:0/2424543352 >> 172.21.15.16:0/2424543352 conn(0x7f325c0b1f40 msgr2=0x7f325c0cc0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:12.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.130+0000 7f32632a4700 1 -- 172.21.15.16:0/2424543352 shutdown_connections 2023-10-16T13:49:12.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.130+0000 7f32632a4700 1 -- 172.21.15.16:0/2424543352 wait complete. 2023-10-16T13:49:12.140 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:12.138+0000 7fb48b882200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T13:49:12.145 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-16T13:49:12.263 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:49:12.261+0000 7fd0a9f02200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T13:49:12.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.274+0000 7fccdfc24700 1 Processor -- start 2023-10-16T13:49:12.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.274+0000 7fccdfc24700 1 -- start start 2023-10-16T13:49:12.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.275+0000 7fccdfc24700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fccd815b790 0x7fccd80cda60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:12.278 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.275+0000 7fccdfc24700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fccd815bb70 0x7fccd806d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:12.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.275+0000 7fccdfc24700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fccd806dea0 0x7fccd815f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:12.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.275+0000 7fccdfc24700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccd806e310 con 0x7fccd815b790 2023-10-16T13:49:12.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.275+0000 7fccdfc24700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccd80ce0e0 con 0x7fccd815bb70 2023-10-16T13:49:12.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.275+0000 7fccdfc24700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccd806a7e0 con 0x7fccd806dea0 2023-10-16T13:49:12.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.275+0000 7fccdd9c0700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fccd815b790 0x7fccd80cda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:12.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.275+0000 7fccdd9c0700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fccd815b790 0x7fccd80cda60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59282/0 (socket says 172.21.15.16:59282) 2023-10-16T13:49:12.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.275+0000 7fccdd9c0700 1 -- 172.21.15.16:0/976319926 learned_addr learned my addr 172.21.15.16:0/976319926 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:12.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.275+0000 7fccde1c1700 1 --2- 172.21.15.16:0/976319926 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fccd806dea0 0x7fccd815f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:12.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.275+0000 7fccdd1bf700 1 --2- 172.21.15.16:0/976319926 >> v2:172.21.15.49:3300/0 conn(0x7fccd815bb70 0x7fccd806d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:12.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.275+0000 7fccdd9c0700 1 -- 172.21.15.16:0/976319926 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fccd806dea0 msgr2=0x7fccd815f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:12.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.275+0000 7fccdd9c0700 1 --2- 172.21.15.16:0/976319926 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fccd806dea0 0x7fccd815f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:12.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.275+0000 7fccdd9c0700 1 -- 172.21.15.16:0/976319926 >> v2:172.21.15.49:3300/0 conn(0x7fccd815bb70 msgr2=0x7fccd806d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:12.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.275+0000 7fccdd9c0700 1 --2- 172.21.15.16:0/976319926 >> v2:172.21.15.49:3300/0 conn(0x7fccd815bb70 0x7fccd806d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:12.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.275+0000 7fccdd9c0700 1 -- 172.21.15.16:0/976319926 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccd815c410 con 0x7fccd815b790 2023-10-16T13:49:12.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.276+0000 7fccdd9c0700 1 --2- 172.21.15.16:0/976319926 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccd815b790 0x7fccd80cda60 secure :-1 s=READY pgs=3223 cs=0 l=1 rev1=1 crypto rx=0x7fccd80cdfa0 tx=0x7fccd4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34c3d3aa75ec2835 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:12.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.276+0000 7fccdc9be700 1 -- 172.21.15.16:0/976319926 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd4011830 con 0x7fccd815b790 2023-10-16T13:49:12.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.276+0000 7fccdc9be700 1 -- 172.21.15.16:0/976319926 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccd4011990 con 0x7fccd815b790 2023-10-16T13:49:12.282 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.276+0000 7fccdc9be700 1 -- 172.21.15.16:0/976319926 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd4011d20 con 0x7fccd815b790 2023-10-16T13:49:12.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.276+0000 7fccdfc24700 1 -- 172.21.15.16:0/976319926 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccd815b790 msgr2=0x7fccd80cda60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:12.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.276+0000 7fccdfc24700 1 --2- 172.21.15.16:0/976319926 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccd815b790 0x7fccd80cda60 secure :-1 s=READY pgs=3223 cs=0 l=1 rev1=1 crypto rx=0x7fccd80cdfa0 tx=0x7fccd4009bd0 comp rx=0 tx=0).stop 2023-10-16T13:49:12.285 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.276+0000 7fccdfc24700 1 -- 172.21.15.16:0/976319926 shutdown_connections 2023-10-16T13:49:12.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.276+0000 7fccdfc24700 1 --2- 172.21.15.16:0/976319926 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fccd806dea0 0x7fccd815f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:12.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.276+0000 7fccdfc24700 1 --2- 172.21.15.16:0/976319926 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccd815b790 0x7fccd80cda60 unknown :-1 s=CLOSED pgs=3223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:12.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.276+0000 7fccdfc24700 1 --2- 172.21.15.16:0/976319926 >> v2:172.21.15.49:3300/0 conn(0x7fccd815bb70 0x7fccd806d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:12.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.276+0000 7fccdfc24700 1 -- 172.21.15.16:0/976319926 >> 172.21.15.16:0/976319926 conn(0x7fccd80cc5e0 msgr2=0x7fccd80cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:12.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.276+0000 7fccdfc24700 1 -- 172.21.15.16:0/976319926 shutdown_connections 2023-10-16T13:49:12.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.276+0000 7fccdfc24700 1 -- 172.21.15.16:0/976319926 wait complete. 2023-10-16T13:49:12.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.278+0000 7fccdfc24700 1 Processor -- start 2023-10-16T13:49:12.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.278+0000 7fccdfc24700 1 -- start start 2023-10-16T13:49:12.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.279+0000 7fccdfc24700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fccd806dea0 0x7fccd815fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:12.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.279+0000 7fccdfc24700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccd815b790 0x7fccd80c6c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:12.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.279+0000 7fccdfc24700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fccd815bb70 0x7fccd8151bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:12.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.279+0000 7fccdfc24700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccd80c75a0 con 0x7fccd815b790 2023-10-16T13:49:12.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.279+0000 7fccdfc24700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccd80c7b60 con 0x7fccd815bb70 2023-10-16T13:49:12.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.279+0000 7fccdfc24700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccd8152110 con 0x7fccd806dea0 2023-10-16T13:49:12.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.279+0000 7fccdd1bf700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccd815b790 0x7fccd80c6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:12.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.279+0000 7fccdd1bf700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccd815b790 0x7fccd80c6c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59288/0 (socket says 172.21.15.16:59288) 2023-10-16T13:49:12.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.279+0000 7fccdd1bf700 1 -- 172.21.15.16:0/304163783 learned_addr learned my addr 172.21.15.16:0/304163783 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:12.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.279+0000 7fccdd9c0700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fccd806dea0 0x7fccd815fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:12.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.279+0000 7fccde1c1700 1 --2- 172.21.15.16:0/304163783 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fccd815bb70 0x7fccd8151bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:12.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.279+0000 7fccdd1bf700 1 -- 172.21.15.16:0/304163783 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fccd806dea0 msgr2=0x7fccd815fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:12.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.279+0000 7fccdd1bf700 1 --2- 172.21.15.16:0/304163783 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fccd806dea0 0x7fccd815fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:12.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.279+0000 7fccdd1bf700 1 -- 172.21.15.16:0/304163783 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fccd815bb70 msgr2=0x7fccd8151bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:12.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.279+0000 7fccdd1bf700 1 --2- 172.21.15.16:0/304163783 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fccd815bb70 0x7fccd8151bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:12.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.279+0000 7fccdd1bf700 1 -- 172.21.15.16:0/304163783 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccd4009550 con 0x7fccd815b790 2023-10-16T13:49:12.290 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.280+0000 7fccdd1bf700 1 --2- 172.21.15.16:0/304163783 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccd815b790 0x7fccd80c6c30 secure :-1 s=READY pgs=3224 cs=0 l=1 rev1=1 crypto rx=0x7fccc800c820 tx=0x7fccc800cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:12.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.280+0000 7fccce7fc700 1 -- 172.21.15.16:0/304163783 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc8004220 con 0x7fccd815b790 2023-10-16T13:49:12.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.280+0000 7fccce7fc700 1 -- 172.21.15.16:0/304163783 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccc8004380 con 0x7fccd815b790 2023-10-16T13:49:12.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.280+0000 7fccdfc24700 1 -- 172.21.15.16:0/304163783 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccd80b29a0 con 0x7fccd815b790 2023-10-16T13:49:12.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.280+0000 7fccce7fc700 1 -- 172.21.15.16:0/304163783 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc80036a0 con 0x7fccd815b790 2023-10-16T13:49:12.291 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.280+0000 7fccdfc24700 1 -- 172.21.15.16:0/304163783 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccd80c77f0 con 0x7fccd815b790 2023-10-16T13:49:12.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.281+0000 7fccdfc24700 1 -- 172.21.15.16:0/304163783 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccd804ebb0 con 0x7fccd815b790 2023-10-16T13:49:12.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.282+0000 7fccce7fc700 1 -- 172.21.15.16:0/304163783 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 167) v1 ==== 89348+0+0 (secure 0 0 0) 0x7fccc8003800 con 0x7fccd815b790 2023-10-16T13:49:12.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.282+0000 7fccce7fc700 1 --2- 172.21.15.16:0/304163783 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fccbc06b070 0x7fccbc06d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:12.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.282+0000 7fccdd9c0700 1 -- 172.21.15.16:0/304163783 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fccbc06b070 msgr2=0x7fccbc06d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1257392887 2023-10-16T13:49:12.292 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.282+0000 7fccce7fc700 1 -- 172.21.15.16:0/304163783 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(278..278 src has 1..278) v4 ==== 6157+0+0 (secure 0 0 0) 0x7fccc8089450 con 0x7fccd815b790 2023-10-16T13:49:12.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.282+0000 7fccdd9c0700 1 --2- 172.21.15.16:0/304163783 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fccbc06b070 0x7fccbc06d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:49:12.293 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.286+0000 7fccce7fc700 1 -- 172.21.15.16:0/304163783 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fccc80541b0 con 0x7fccd815b790 2023-10-16T13:49:12.296 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:12.294+0000 7fb48b882200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T13:49:12.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.483+0000 7fccdd9c0700 1 -- 172.21.15.16:0/304163783 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fccbc06b070 msgr2=0x7fccbc06d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1257392887 2023-10-16T13:49:12.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.483+0000 7fccdd9c0700 1 --2- 172.21.15.16:0/304163783 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fccbc06b070 0x7fccbc06d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:49:12.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.496+0000 7fccdfc24700 1 -- 172.21.15.16:0/304163783 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fccd806f380 con 0x7fccd815b790 2023-10-16T13:49:12.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.499+0000 7fccce7fc700 1 -- 172.21.15.16:0/304163783 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v96)=0 v96) v1 ==== 133+0+0 (secure 0 0 0) 0x7fccc80577d0 con 0x7fccd815b790 2023-10-16T13:49:12.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.501+0000 7fccdfc24700 1 -- 172.21.15.16:0/304163783 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fccbc06b070 msgr2=0x7fccbc06d530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:49:12.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.501+0000 7fccdfc24700 1 --2- 172.21.15.16:0/304163783 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fccbc06b070 0x7fccbc06d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:12.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.501+0000 7fccdfc24700 1 -- 172.21.15.16:0/304163783 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccd815b790 msgr2=0x7fccd80c6c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:12.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.501+0000 7fccdfc24700 1 --2- 172.21.15.16:0/304163783 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccd815b790 0x7fccd80c6c30 secure :-1 s=READY pgs=3224 cs=0 l=1 rev1=1 crypto rx=0x7fccc800c820 tx=0x7fccc800cb30 comp rx=0 tx=0).stop 2023-10-16T13:49:12.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.501+0000 7fccdfc24700 1 -- 172.21.15.16:0/304163783 shutdown_connections 2023-10-16T13:49:12.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.501+0000 7fccdfc24700 1 --2- 172.21.15.16:0/304163783 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fccd815bb70 0x7fccd8151bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:12.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.501+0000 7fccdfc24700 1 --2- 172.21.15.16:0/304163783 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fccd806dea0 0x7fccd815fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:12.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.501+0000 7fccdfc24700 1 --2- 172.21.15.16:0/304163783 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7fccbc06b070 0x7fccbc06d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:12.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.501+0000 7fccdfc24700 1 --2- 172.21.15.16:0/304163783 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccd815b790 0x7fccd80c6c30 unknown :-1 s=CLOSED pgs=3224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:12.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.501+0000 7fccdfc24700 1 -- 172.21.15.16:0/304163783 >> 172.21.15.16:0/304163783 conn(0x7fccd80cc5e0 msgr2=0x7fccd80af680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:12.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.501+0000 7fccdfc24700 1 -- 172.21.15.16:0/304163783 shutdown_connections 2023-10-16T13:49:12.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.501+0000 7fccdfc24700 1 -- 172.21.15.16:0/304163783 wait complete. 2023-10-16T13:49:12.516 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv1 2023-10-16T13:49:12.604 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:49:12.601+0000 7fd0a9f02200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T13:49:12.643 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:12.641+0000 7fb48b882200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T13:49:12.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.643+0000 7f9854c41700 1 Processor -- start 2023-10-16T13:49:12.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.643+0000 7f9854c41700 1 -- start start 2023-10-16T13:49:12.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.644+0000 7f9854c41700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f985006a7e0 0x7f985006ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:12.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.644+0000 7f9854c41700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f985015ecb0 con 0x7f985006e570 2023-10-16T13:49:12.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.644+0000 7f9854c41700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f985015b6a0 con 0x7f985015b2c0 2023-10-16T13:49:12.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.644+0000 7f9854c41700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f985006b1b0 con 0x7f985006a7e0 2023-10-16T13:49:12.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.644+0000 7f984dd9b700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f985006e570 0x7f985015e5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48616/0 (socket says 172.21.15.16:48616) 2023-10-16T13:49:12.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.644+0000 7f984dd9b700 1 -- 172.21.15.16:0/3142245497 learned_addr learned my addr 172.21.15.16:0/3142245497 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:49:12.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.645+0000 7f984ed9d700 1 --2- 172.21.15.16:0/3142245497 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f985006a7e0 0x7f985006ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:12.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.645+0000 7f984d59a700 1 -- 172.21.15.16:0/3142245497 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2312015725 0 0) 0x7f985015ecb0 con 0x7f985006e570 2023-10-16T13:49:12.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.645+0000 7f984d59a700 1 -- 172.21.15.16:0/3142245497 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98380035b0 con 0x7f985006e570 2023-10-16T13:49:12.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.645+0000 7f984d59a700 1 -- 172.21.15.16:0/3142245497 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2972441803 0 0) 0x7f985015b6a0 con 0x7f985015b2c0 2023-10-16T13:49:12.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.645+0000 7f984d59a700 1 -- 172.21.15.16:0/3142245497 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f985015ecb0 con 0x7f985015b2c0 2023-10-16T13:49:12.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.645+0000 7f984ed9d700 1 -- 172.21.15.16:0/3142245497 >> v1:172.21.15.49:6789/0 conn(0x7f985015b2c0 legacy=0x7f9850161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:12.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.645+0000 7f984ed9d700 1 -- 172.21.15.16:0/3142245497 >> v1:172.21.15.16:6789/0 conn(0x7f985006e570 legacy=0x7f985015e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:12.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.645+0000 7f984ed9d700 1 -- 172.21.15.16:0/3142245497 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9850161b20 con 0x7f985006a7e0 2023-10-16T13:49:12.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.645+0000 7f984ed9d700 1 --2- 172.21.15.16:0/3142245497 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f985006a7e0 0x7f985006ac70 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f9848009dc0 tx=0x7f984800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8e11a02c16112dfd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:12.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.645+0000 7f984d59a700 1 -- 172.21.15.16:0/3142245497 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f984800e070 con 0x7f985006a7e0 2023-10-16T13:49:12.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.645+0000 7f984d59a700 1 -- 172.21.15.16:0/3142245497 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f984800bb20 con 0x7f985006a7e0 2023-10-16T13:49:12.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.645+0000 7f984d59a700 1 -- 172.21.15.16:0/3142245497 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9848012430 con 0x7f985006a7e0 2023-10-16T13:49:12.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.646+0000 7f9854c41700 1 -- 172.21.15.16:0/3142245497 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f985006a7e0 msgr2=0x7f985006ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:12.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.646+0000 7f9854c41700 1 --2- 172.21.15.16:0/3142245497 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f985006a7e0 0x7f985006ac70 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f9848009dc0 tx=0x7f984800b330 comp rx=0 tx=0).stop 2023-10-16T13:49:12.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.646+0000 7f9854c41700 1 -- 172.21.15.16:0/3142245497 shutdown_connections 2023-10-16T13:49:12.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.646+0000 7f9854c41700 1 --2- 172.21.15.16:0/3142245497 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f985006a7e0 0x7f985006ac70 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:12.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.646+0000 7f9854c41700 1 -- 172.21.15.16:0/3142245497 >> 172.21.15.16:0/3142245497 conn(0x7f98500b1f40 msgr2=0x7f98500b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:12.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.646+0000 7f9854c41700 1 -- 172.21.15.16:0/3142245497 shutdown_connections 2023-10-16T13:49:12.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.646+0000 7f9854c41700 1 -- 172.21.15.16:0/3142245497 wait complete. 2023-10-16T13:49:12.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.648+0000 7f9854c41700 1 Processor -- start 2023-10-16T13:49:12.657 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.648+0000 7f9854c41700 1 -- start start 2023-10-16T13:49:12.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.648+0000 7f9854c41700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f985006a7e0 0x7f98500b4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:12.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.648+0000 7f9854c41700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f985006e570 0x7f98500c9960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:12.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.648+0000 7f9854c41700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f985015b2c0 0x7f98500b5200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:12.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.648+0000 7f9854c41700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98500c9ea0 con 0x7f985006e570 2023-10-16T13:49:12.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.648+0000 7f9854c41700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98501584c0 con 0x7f985015b2c0 2023-10-16T13:49:12.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.648+0000 7f9854c41700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98501581c0 con 0x7f985006a7e0 2023-10-16T13:49:12.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.649+0000 7f984dd9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f985006e570 0x7f98500c9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:12.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.649+0000 7f984dd9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f985006e570 0x7f98500c9960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59348/0 (socket says 172.21.15.16:59348) 2023-10-16T13:49:12.659 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.649+0000 7f984dd9b700 1 -- 172.21.15.16:0/2478531197 learned_addr learned my addr 172.21.15.16:0/2478531197 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:12.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.649+0000 7f984ed9d700 1 --2- 172.21.15.16:0/2478531197 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f985015b2c0 0x7f98500b5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:12.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.649+0000 7f984e59c700 1 --2- 172.21.15.16:0/2478531197 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f985006a7e0 0x7f98500b4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:12.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.649+0000 7f984dd9b700 1 -- 172.21.15.16:0/2478531197 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f985006a7e0 msgr2=0x7f98500b4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:12.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.649+0000 7f984dd9b700 1 --2- 172.21.15.16:0/2478531197 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f985006a7e0 0x7f98500b4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:12.660 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.649+0000 7f984dd9b700 1 -- 172.21.15.16:0/2478531197 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f985015b2c0 msgr2=0x7f98500b5200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:12.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.649+0000 7f984dd9b700 1 --2- 172.21.15.16:0/2478531197 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f985015b2c0 0x7f98500b5200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:12.661 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.649+0000 7f984dd9b700 1 -- 172.21.15.16:0/2478531197 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9848009a70 con 0x7f985006e570 2023-10-16T13:49:12.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.649+0000 7f984dd9b700 1 --2- 172.21.15.16:0/2478531197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f985006e570 0x7f98500c9960 secure :-1 s=READY pgs=3226 cs=0 l=1 rev1=1 crypto rx=0x7f984400c700 tx=0x7f984400ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:12.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.649+0000 7f983effd700 1 -- 172.21.15.16:0/2478531197 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9844005d70 con 0x7f985006e570 2023-10-16T13:49:12.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.649+0000 7f983effd700 1 -- 172.21.15.16:0/2478531197 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98440070d0 con 0x7f985006e570 2023-10-16T13:49:12.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.649+0000 7f9854c41700 1 -- 172.21.15.16:0/2478531197 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98500ca0d0 con 0x7f985006e570 2023-10-16T13:49:12.662 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.649+0000 7f983effd700 1 -- 172.21.15.16:0/2478531197 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9844007460 con 0x7f985006e570 2023-10-16T13:49:12.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.649+0000 7f9854c41700 1 -- 172.21.15.16:0/2478531197 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98500ca690 con 0x7f985006e570 2023-10-16T13:49:12.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.650+0000 7f9854c41700 1 -- 172.21.15.16:0/2478531197 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f985004ebb0 con 0x7f985006e570 2023-10-16T13:49:12.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.651+0000 7f983effd700 1 -- 172.21.15.16:0/2478531197 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 167) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f98440075c0 con 0x7f985006e570 2023-10-16T13:49:12.663 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.651+0000 7f983effd700 1 --2- 172.21.15.16:0/2478531197 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f983006b070 0x7f983006d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:12.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.651+0000 7f984e59c700 1 -- 172.21.15.16:0/2478531197 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f983006b070 msgr2=0x7f983006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1257392887 2023-10-16T13:49:12.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.651+0000 7f983effd700 1 -- 172.21.15.16:0/2478531197 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(278..278 src has 1..278) v4 ==== 6157+0+0 (secure 0 0 0) 0x7f9844088e90 con 0x7f985006e570 2023-10-16T13:49:12.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.651+0000 7f984e59c700 1 --2- 172.21.15.16:0/2478531197 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f983006b070 0x7f983006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:49:12.664 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.656+0000 7f983effd700 1 -- 172.21.15.16:0/2478531197 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9844053b50 con 0x7f985006e570 2023-10-16T13:49:12.724 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:49:12.721+0000 7fd0a9f02200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T13:49:12.767 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:12.764+0000 7fb48b882200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T13:49:12.801 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:49:12.798+0000 7fd0a9f02200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T13:49:12.844 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:12.842+0000 7fb48b882200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T13:49:12.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.852+0000 7f984e59c700 1 -- 172.21.15.16:0/2478531197 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f983006b070 msgr2=0x7f983006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1257392887 2023-10-16T13:49:12.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.852+0000 7f984e59c700 1 --2- 172.21.15.16:0/2478531197 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f983006b070 0x7f983006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:49:12.885 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:49:12.882+0000 7fd0a9f02200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T13:49:12.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:12.895+0000 7f9854c41700 1 -- 172.21.15.16:0/2478531197 --> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f9850158330 con 0x7f983006b070 2023-10-16T13:49:12.928 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:12.926+0000 7fb48b882200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T13:49:13.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:13.253+0000 7f984e59c700 1 -- 172.21.15.16:0/2478531197 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f983006b070 msgr2=0x7f983006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1257392887 2023-10-16T13:49:13.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:13.253+0000 7f984e59c700 1 --2- 172.21.15.16:0/2478531197 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f983006b070 0x7f983006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-16T13:49:13.271 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:49:13.268+0000 7fd0a9f02200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T13:49:13.316 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:13.314+0000 7fb48b882200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T13:49:13.877 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:49:13.874+0000 7fd0a9f02200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T13:49:13.922 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:13.920+0000 7fb48b882200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T13:49:13.949 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:49:13.947+0000 7fd0a9f02200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T13:49:13.995 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:13.993+0000 7fb48b882200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T13:49:14.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:14.054+0000 7f984e59c700 1 -- 172.21.15.16:0/2478531197 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f983006b070 msgr2=0x7f983006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1257392887 2023-10-16T13:49:14.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:14.054+0000 7f984e59c700 1 --2- 172.21.15.16:0/2478531197 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f983006b070 0x7f983006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-16T13:49:14.611 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:49:14.608+0000 7fd0a9f02200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:49:14.662 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:14.659+0000 7fb48b882200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:49:14.692 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:49:14.689+0000 7fd0a9f02200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T13:49:14.744 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:14.742+0000 7fb48b882200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T13:49:14.842 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:49:14.839+0000 7fd0a9f02200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:49:14.896 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:14.894+0000 7fb48b882200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:49:15.080 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:49:15.077+0000 7fd0a9f02200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T13:49:15.137 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:15.135+0000 7fb48b882200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T13:49:15.157 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:49:15.155+0000 7fd0a9f02200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T13:49:15.215 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:15.213+0000 7fb48b882200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T13:49:15.434 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:49:15.431+0000 7fd0a9f02200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T13:49:15.493 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:15.491+0000 7fb48b882200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T13:49:15.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:15.655+0000 7f984e59c700 1 -- 172.21.15.16:0/2478531197 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f983006b070 msgr2=0x7f983006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/1257392887 2023-10-16T13:49:15.658 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:15.655+0000 7f984e59c700 1 --2- 172.21.15.16:0/2478531197 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f983006b070 0x7f983006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-10-16T13:49:15.665 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:49:15.662+0000 7fd0a9f02200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T13:49:15.728 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:15.725+0000 7fb48b882200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T13:49:15.822 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:49:15.820+0000 7fd0a9f02200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T13:49:15.887 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:15.885+0000 7fb48b882200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T13:49:15.942 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:49:15.939+0000 7fd0a9f02200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T13:49:16.008 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:16.006+0000 7fb48b882200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T13:49:16.105 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:49:16.102+0000 7fd0a9f02200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T13:49:16.174 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:16.171+0000 7fb48b882200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T13:49:16.378 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:49:16.376+0000 7fd0a9f02200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T13:49:16.448 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:16.446+0000 7fb48b882200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T13:49:16.548 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:49:16.545+0000 7fd0a9f02200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T13:49:16.619 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:16.616+0000 7fb48b882200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T13:49:17.102 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:49:17.100+0000 7fd0a9f02200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T13:49:17.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:17.169+0000 7f983effd700 1 -- 172.21.15.16:0/2478531197 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 168) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f9844053f20 con 0x7f985006e570 2023-10-16T13:49:17.182 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:17.180+0000 7fb48b882200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T13:49:17.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:17.200+0000 7f983effd700 1 -- 172.21.15.16:0/2478531197 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mgrmap(e 169) v1 ==== 88957+0+0 (secure 0 0 0) 0x7f98440555c0 con 0x7f985006e570 2023-10-16T13:49:17.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:17.200+0000 7f983effd700 1 -- 172.21.15.16:0/2478531197 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f983006b070 msgr2=0x7f983006d530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:49:17.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:17.200+0000 7f983effd700 1 --2- 172.21.15.16:0/2478531197 >> [v2:172.21.15.16:6832/1257392887,v1:172.21.15.16:6833/1257392887] conn(0x7f983006b070 0x7f983006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:17.344 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:17.341+0000 7fb3fe85d700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:49:17.344 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:17.341+0000 7fb3fe85d700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:49:17.345 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:17.341+0000 7fb3fe85d700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:49:17.345 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:17.341+0000 7fb3fe85d700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:49:17.345 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:17.341+0000 7fb3fe85d700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:49:17.345 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:17.342+0000 7fb401863700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:49:17.345 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:17.342+0000 7fb401863700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:49:17.346 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:17.342+0000 7fb401863700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:49:17.346 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:17.342+0000 7fb401863700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:49:17.346 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:17.342+0000 7fb401863700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:49:18.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.205+0000 7f983effd700 1 -- 172.21.15.16:0/2478531197 <== mon.0 v2:172.21.15.16:3300/0 9 ==== mgrmap(e 170) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f98440836f0 con 0x7f985006e570 2023-10-16T13:49:18.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.205+0000 7f983effd700 1 --2- 172.21.15.16:0/2478531197 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f983001d250 0x7f9830075610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:18.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.205+0000 7f983effd700 1 -- 172.21.15.16:0/2478531197 --> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f9850158330 con 0x7f983001d250 2023-10-16T13:49:18.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.207+0000 7f984e59c700 1 --2- 172.21.15.16:0/2478531197 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f983001d250 0x7f9830075610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:18.213 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.211+0000 7f984e59c700 1 --2- 172.21.15.16:0/2478531197 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f983001d250 0x7f9830075610 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f98400011e0 tx=0x7f9840009040 comp rx=0 tx=0).ready entity=mgr.10586 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:18.219 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:18.217+0000 7fb3fb857700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:49:18.219 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:18.217+0000 7fb3fb857700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:49:18.220 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:18.217+0000 7fb3fb857700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:49:18.220 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:18.217+0000 7fb3fb857700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:49:18.220 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:18.217+0000 7fb3fb857700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:49:18.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.250+0000 7f983effd700 1 -- 172.21.15.16:0/2478531197 <== mgr.10586 v2:172.21.15.16:6832/3920835901 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f9850158330 con 0x7f983001d250 2023-10-16T13:49:18.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.252+0000 7f9854c41700 1 -- 172.21.15.16:0/2478531197 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f983001d250 msgr2=0x7f9830075610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:18.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.252+0000 7f9854c41700 1 --2- 172.21.15.16:0/2478531197 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f983001d250 0x7f9830075610 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f98400011e0 tx=0x7f9840009040 comp rx=0 tx=0).stop 2023-10-16T13:49:18.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.252+0000 7f9854c41700 1 -- 172.21.15.16:0/2478531197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f985006e570 msgr2=0x7f98500c9960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:18.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.252+0000 7f9854c41700 1 --2- 172.21.15.16:0/2478531197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f985006e570 0x7f98500c9960 secure :-1 s=READY pgs=3226 cs=0 l=1 rev1=1 crypto rx=0x7f984400c700 tx=0x7f984400ca10 comp rx=0 tx=0).stop 2023-10-16T13:49:18.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.252+0000 7f984ed9d700 1 -- 172.21.15.16:0/2478531197 reap_dead start 2023-10-16T13:49:18.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.252+0000 7f9854c41700 1 -- 172.21.15.16:0/2478531197 shutdown_connections 2023-10-16T13:49:18.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.252+0000 7f9854c41700 1 -- 172.21.15.16:0/2478531197 >> 172.21.15.16:0/2478531197 conn(0x7f98500b1f40 msgr2=0x7f985015e4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:18.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.252+0000 7f9854c41700 1 -- 172.21.15.16:0/2478531197 shutdown_connections 2023-10-16T13:49:18.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.252+0000 7f9854c41700 1 -- 172.21.15.16:0/2478531197 wait complete. 2023-10-16T13:49:18.269 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --subvol sv1 . 1m 2023-10-16T13:49:18.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.402+0000 7fcef29e8700 1 Processor -- start 2023-10-16T13:49:18.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.402+0000 7fcef29e8700 1 -- start start 2023-10-16T13:49:18.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.403+0000 7fcef29e8700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fceec15a3d0 0x7fceec06d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:18.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.403+0000 7fcef29e8700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fceec0cb870 con 0x7fceec15e0b0 2023-10-16T13:49:18.409 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.403+0000 7fcef29e8700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fceec06e040 con 0x7fceec0cf600 2023-10-16T13:49:18.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.403+0000 7fcef29e8700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fceec15f0c0 con 0x7fceec15a3d0 2023-10-16T13:49:18.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.403+0000 7fceeb7fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fceec15e0b0 0x7fceec0ceef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48832/0 (socket says 172.21.15.16:48832) 2023-10-16T13:49:18.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.403+0000 7fceeb7fe700 1 -- 172.21.15.16:0/521139657 learned_addr learned my addr 172.21.15.16:0/521139657 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:49:18.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.403+0000 7fceebfff700 1 --2- 172.21.15.16:0/521139657 >> v2:172.21.15.49:3300/0 conn(0x7fceec15a3d0 0x7fceec06d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:18.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.403+0000 7fceeaffd700 1 -- 172.21.15.16:0/521139657 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 518900663 0 0) 0x7fceec0cb870 con 0x7fceec15e0b0 2023-10-16T13:49:18.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.403+0000 7fceeaffd700 1 -- 172.21.15.16:0/521139657 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fced40035b0 con 0x7fceec15e0b0 2023-10-16T13:49:18.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.403+0000 7fceeaffd700 1 -- 172.21.15.16:0/521139657 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1294102063 0 0) 0x7fced40035b0 con 0x7fceec15e0b0 2023-10-16T13:49:18.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.403+0000 7fceeaffd700 1 -- 172.21.15.16:0/521139657 >> v2:172.21.15.49:3300/0 conn(0x7fceec15a3d0 msgr2=0x7fceec06d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:18.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.403+0000 7fceeaffd700 1 --2- 172.21.15.16:0/521139657 >> v2:172.21.15.49:3300/0 conn(0x7fceec15a3d0 0x7fceec06d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:18.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.403+0000 7fceeaffd700 1 -- 172.21.15.16:0/521139657 >> v1:172.21.15.49:6789/0 conn(0x7fceec0cf600 legacy=0x7fceec0cbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:18.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.403+0000 7fceeaffd700 1 -- 172.21.15.16:0/521139657 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fceec06e320 con 0x7fceec15e0b0 2023-10-16T13:49:18.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.403+0000 7fceeaffd700 1 -- 172.21.15.16:0/521139657 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7fcee0001670 con 0x7fceec15e0b0 2023-10-16T13:49:18.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.404+0000 7fceeaffd700 1 -- 172.21.15.16:0/521139657 >> v1:172.21.15.16:6789/0 conn(0x7fceec15e0b0 legacy=0x7fceec0ceef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:18.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.404+0000 7fceeaffd700 1 --2- 172.21.15.16:0/521139657 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fced4003a00 0x7fced4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:18.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.404+0000 7fceeaffd700 1 --2- 172.21.15.16:0/521139657 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fced4004850 0x7fced4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:18.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.404+0000 7fceeaffd700 1 --2- 172.21.15.16:0/521139657 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fced4005210 0x7fced400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:18.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.404+0000 7fceeaffd700 1 -- 172.21.15.16:0/521139657 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceec15f0c0 con 0x7fced4004850 2023-10-16T13:49:18.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.404+0000 7fceeaffd700 1 -- 172.21.15.16:0/521139657 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fced400c0b0 con 0x7fced4003a00 2023-10-16T13:49:18.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.404+0000 7fceeaffd700 1 -- 172.21.15.16:0/521139657 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fced400c270 con 0x7fced4005210 2023-10-16T13:49:18.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.404+0000 7fceeb7fe700 1 --2- 172.21.15.16:0/521139657 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fced4004850 0x7fced4004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:18.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.404+0000 7fceebfff700 1 --2- 172.21.15.16:0/521139657 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fced4003a00 0x7fced4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:18.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.404+0000 7fcef0f85700 1 --2- 172.21.15.16:0/521139657 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fced4005210 0x7fced400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:18.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.404+0000 7fceeb7fe700 1 -- 172.21.15.16:0/521139657 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fced4005210 msgr2=0x7fced400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:18.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.404+0000 7fceeb7fe700 1 --2- 172.21.15.16:0/521139657 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fced4005210 0x7fced400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:18.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.404+0000 7fceeb7fe700 1 -- 172.21.15.16:0/521139657 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fced4003a00 msgr2=0x7fced4003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:18.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.404+0000 7fceeb7fe700 1 --2- 172.21.15.16:0/521139657 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fced4003a00 0x7fced4003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:18.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.404+0000 7fcef0f85700 1 -- 172.21.15.16:0/521139657 reap_dead start 2023-10-16T13:49:18.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.404+0000 7fceeb7fe700 1 -- 172.21.15.16:0/521139657 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fced40035b0 con 0x7fced4004850 2023-10-16T13:49:18.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.404+0000 7fceeb7fe700 1 --2- 172.21.15.16:0/521139657 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fced4004850 0x7fced4004cd0 secure :-1 s=READY pgs=3238 cs=0 l=1 rev1=1 crypto rx=0x7fcee000c240 tx=0x7fced4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b183bde6ba44dd0d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:18.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.404+0000 7fceeaffd700 1 -- 172.21.15.16:0/521139657 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcee0007c10 con 0x7fced4004850 2023-10-16T13:49:18.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.404+0000 7fceeaffd700 1 -- 172.21.15.16:0/521139657 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcee000cda0 con 0x7fced4004850 2023-10-16T13:49:18.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.404+0000 7fceeaffd700 1 -- 172.21.15.16:0/521139657 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcee0005ba0 con 0x7fced4004850 2023-10-16T13:49:18.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.405+0000 7fcef29e8700 1 -- 172.21.15.16:0/521139657 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fced4004850 msgr2=0x7fced4004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:18.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.405+0000 7fcef29e8700 1 --2- 172.21.15.16:0/521139657 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fced4004850 0x7fced4004cd0 secure :-1 s=READY pgs=3238 cs=0 l=1 rev1=1 crypto rx=0x7fcee000c240 tx=0x7fced4005880 comp rx=0 tx=0).stop 2023-10-16T13:49:18.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.405+0000 7fcef29e8700 1 -- 172.21.15.16:0/521139657 shutdown_connections 2023-10-16T13:49:18.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.405+0000 7fcef29e8700 1 --2- 172.21.15.16:0/521139657 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fced4004850 0x7fced4004cd0 unknown :-1 s=CLOSED pgs=3238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:18.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.405+0000 7fcef29e8700 1 -- 172.21.15.16:0/521139657 >> 172.21.15.16:0/521139657 conn(0x7fceec06f430 msgr2=0x7fceec06f810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:18.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.405+0000 7fcef29e8700 1 -- 172.21.15.16:0/521139657 shutdown_connections 2023-10-16T13:49:18.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.405+0000 7fcef29e8700 1 -- 172.21.15.16:0/521139657 wait complete. 2023-10-16T13:49:18.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.406+0000 7fcef29e8700 1 Processor -- start 2023-10-16T13:49:18.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.406+0000 7fcef29e8700 1 -- start start 2023-10-16T13:49:18.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.407+0000 7fcef29e8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fced4004850 0x7fceec155cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:18.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.407+0000 7fcef29e8700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fceec0c4740 0x7fceec0c4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:18.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.407+0000 7fcef29e8700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fceec15bb70 0x7fceec15bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:18.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.407+0000 7fcef29e8700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceec0c5100 con 0x7fced4004850 2023-10-16T13:49:18.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.407+0000 7fcef29e8700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceec1561f0 con 0x7fceec15bb70 2023-10-16T13:49:18.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.407+0000 7fcef29e8700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fceec15c8a0 con 0x7fceec0c4740 2023-10-16T13:49:18.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.407+0000 7fceebfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fced4004850 0x7fceec155cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:18.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.407+0000 7fceebfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fced4004850 0x7fceec155cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59568/0 (socket says 172.21.15.16:59568) 2023-10-16T13:49:18.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.407+0000 7fceebfff700 1 -- 172.21.15.16:0/4133655757 learned_addr learned my addr 172.21.15.16:0/4133655757 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:18.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.407+0000 7fceeb7fe700 1 --2- 172.21.15.16:0/4133655757 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fceec0c4740 0x7fceec0c4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:18.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.407+0000 7fcef0f85700 1 --2- 172.21.15.16:0/4133655757 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fceec15bb70 0x7fceec15bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:18.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.407+0000 7fceebfff700 1 -- 172.21.15.16:0/4133655757 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fceec0c4740 msgr2=0x7fceec0c4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:18.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.407+0000 7fceebfff700 1 --2- 172.21.15.16:0/4133655757 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fceec0c4740 0x7fceec0c4bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:18.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.407+0000 7fceebfff700 1 -- 172.21.15.16:0/4133655757 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fceec15bb70 msgr2=0x7fceec15bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:18.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.407+0000 7fceebfff700 1 --2- 172.21.15.16:0/4133655757 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fceec15bb70 0x7fceec15bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:18.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.407+0000 7fceebfff700 1 -- 172.21.15.16:0/4133655757 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fceec15cb80 con 0x7fced4004850 2023-10-16T13:49:18.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.407+0000 7fcef0f85700 1 --2- 172.21.15.16:0/4133655757 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fceec15bb70 0x7fceec15bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:49:18.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.408+0000 7fceebfff700 1 --2- 172.21.15.16:0/4133655757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fced4004850 0x7fceec155cb0 secure :-1 s=READY pgs=3239 cs=0 l=1 rev1=1 crypto rx=0x7fcedc0119d0 tx=0x7fcedc011d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:18.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.408+0000 7fcee8ff9700 1 -- 172.21.15.16:0/4133655757 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcedc009630 con 0x7fced4004850 2023-10-16T13:49:18.424 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.408+0000 7fcee8ff9700 1 -- 172.21.15.16:0/4133655757 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcedc004240 con 0x7fced4004850 2023-10-16T13:49:18.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.408+0000 7fcee8ff9700 1 -- 172.21.15.16:0/4133655757 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcedc00f730 con 0x7fced4004850 2023-10-16T13:49:18.425 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.408+0000 7fcef29e8700 1 -- 172.21.15.16:0/4133655757 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fceec15ce60 con 0x7fced4004850 2023-10-16T13:49:18.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.408+0000 7fcef29e8700 1 -- 172.21.15.16:0/4133655757 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fceec15d420 con 0x7fced4004850 2023-10-16T13:49:18.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.409+0000 7fcef29e8700 1 -- 172.21.15.16:0/4133655757 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fceec04ec20 con 0x7fced4004850 2023-10-16T13:49:18.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.409+0000 7fcee8ff9700 1 -- 172.21.15.16:0/4133655757 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 170) v1 ==== 89084+0+0 (secure 0 0 0) 0x7fcedc00f890 con 0x7fced4004850 2023-10-16T13:49:18.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.410+0000 7fcee8ff9700 1 --2- 172.21.15.16:0/4133655757 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7fced806ada0 0x7fced806d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:18.426 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.410+0000 7fcee8ff9700 1 -- 172.21.15.16:0/4133655757 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(279..279 src has 1..279) v4 ==== 6501+0+0 (secure 0 0 0) 0x7fcedc089b40 con 0x7fced4004850 2023-10-16T13:49:18.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.410+0000 7fceeb7fe700 1 --2- 172.21.15.16:0/4133655757 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7fced806ada0 0x7fced806d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:18.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.410+0000 7fceeb7fe700 1 --2- 172.21.15.16:0/4133655757 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7fced806ada0 0x7fced806d260 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fcee0001930 tx=0x7fcee0007000 comp rx=0 tx=0).ready entity=mgr.10586 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:18.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.415+0000 7fcee8ff9700 1 -- 172.21.15.16:0/4133655757 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcedc054c10 con 0x7fced4004850 2023-10-16T13:49:18.646 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.644+0000 7fcef29e8700 1 -- 172.21.15.16:0/4133655757 --> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "subvol": "sv1", "path": ".", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fceec06e8e0 con 0x7fced806ada0 2023-10-16T13:49:18.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.716+0000 7fcee8ff9700 1 -- 172.21.15.16:0/4133655757 <== mgr.10586 v2:172.21.15.16:6832/3920835901 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+84 (secure 0 0 0) 0x7fcee0007b30 con 0x7fced806ada0 2023-10-16T13:49:18.719 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule set for path /volumes/_nogroup/sv1/bcfda3a0-ba49-4161-81cb-81cdfd123292/.. 2023-10-16T13:49:18.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.718+0000 7fcef29e8700 1 -- 172.21.15.16:0/4133655757 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7fced806ada0 msgr2=0x7fced806d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:18.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.718+0000 7fcef29e8700 1 --2- 172.21.15.16:0/4133655757 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7fced806ada0 0x7fced806d260 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fcee0001930 tx=0x7fcee0007000 comp rx=0 tx=0).stop 2023-10-16T13:49:18.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.718+0000 7fcef29e8700 1 -- 172.21.15.16:0/4133655757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fced4004850 msgr2=0x7fceec155cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:18.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.718+0000 7fcef29e8700 1 --2- 172.21.15.16:0/4133655757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fced4004850 0x7fceec155cb0 secure :-1 s=READY pgs=3239 cs=0 l=1 rev1=1 crypto rx=0x7fcedc0119d0 tx=0x7fcedc011d90 comp rx=0 tx=0).stop 2023-10-16T13:49:18.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.718+0000 7fcef29e8700 1 -- 172.21.15.16:0/4133655757 shutdown_connections 2023-10-16T13:49:18.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.718+0000 7fcef29e8700 1 --2- 172.21.15.16:0/4133655757 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7fced806ada0 0x7fced806d260 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:18.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.718+0000 7fcef29e8700 1 --2- 172.21.15.16:0/4133655757 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fceec15bb70 0x7fceec15bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:18.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.718+0000 7fcef29e8700 1 --2- 172.21.15.16:0/4133655757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fced4004850 0x7fceec155cb0 unknown :-1 s=CLOSED pgs=3239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:18.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.718+0000 7fcef29e8700 1 --2- 172.21.15.16:0/4133655757 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fceec0c4740 0x7fceec0c4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:18.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.718+0000 7fcef29e8700 1 -- 172.21.15.16:0/4133655757 >> 172.21.15.16:0/4133655757 conn(0x7fceec06f430 msgr2=0x7fceec0b2f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:18.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.718+0000 7fcef29e8700 1 -- 172.21.15.16:0/4133655757 shutdown_connections 2023-10-16T13:49:18.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.718+0000 7fcef29e8700 1 -- 172.21.15.16:0/4133655757 wait complete. 2023-10-16T13:49:18.733 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /volumes/_nogroup/sv1/bcfda3a0-ba49-4161-81cb-81cdfd123292/.. 2023-10-16T13:49:18.734 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --subvol sv1 --group mygrp . 1m 2023-10-16T13:49:18.866 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.864+0000 7f2fbccc0700 1 Processor -- start 2023-10-16T13:49:18.867 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.864+0000 7f2fbccc0700 1 -- start start 2023-10-16T13:49:18.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.865+0000 7f2fbccc0700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f2fb80cb880 0x7f2fb80cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:18.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.865+0000 7f2fbccc0700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2fb806da60 0x7f2fb806def0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:18.869 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.865+0000 7f2fbccc0700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fb80cef70 con 0x7f2fb815b070 2023-10-16T13:49:18.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.865+0000 7f2fbccc0700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fb806e430 con 0x7f2fb80cb880 2023-10-16T13:49:18.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.865+0000 7f2fbccc0700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb80ce360 con 0x7f2fb806da60 2023-10-16T13:49:18.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.865+0000 7f2fb5d9b700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f2fb80cb880 0x7f2fb80cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:18.870 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.865+0000 7f2fb6d9d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2fb806da60 0x7f2fb806def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:18.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.865+0000 7f2fb5d9b700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f2fb80cb880 0x7f2fb80cbd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:58642/0 (socket says 172.21.15.16:58642) 2023-10-16T13:49:18.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.865+0000 7f2fb5d9b700 1 -- 172.21.15.16:0/679926885 learned_addr learned my addr 172.21.15.16:0/679926885 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:18.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.865+0000 7f2fb6d9d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2fb806da60 0x7f2fb806def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:47836/0 (socket says 172.21.15.16:47836) 2023-10-16T13:49:18.871 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.865+0000 7f2fb659c700 1 --1- 172.21.15.16:0/679926885 >> v1:172.21.15.49:6789/0 conn(0x7f2fb815b070 0x7f2fb815b450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:56032/0 (socket says 172.21.15.16:56032) 2023-10-16T13:49:18.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.865+0000 7f2fb5d9b700 1 -- 172.21.15.16:0/679926885 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2fb806da60 msgr2=0x7f2fb806def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:18.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.865+0000 7f2fb5d9b700 1 --2- 172.21.15.16:0/679926885 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2fb806da60 0x7f2fb806def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:18.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.865+0000 7f2fb5d9b700 1 -- 172.21.15.16:0/679926885 >> v1:172.21.15.49:6789/0 conn(0x7f2fb815b070 legacy=0x7f2fb815b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:18.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.865+0000 7f2fb5d9b700 1 -- 172.21.15.16:0/679926885 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fb80ce640 con 0x7f2fb80cb880 2023-10-16T13:49:18.872 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.865+0000 7f2fb6d9d700 1 --2- 172.21.15.16:0/679926885 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2fb806da60 0x7f2fb806def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:49:18.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.866+0000 7f2fb5d9b700 1 --2- 172.21.15.16:0/679926885 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2fb80cb880 0x7f2fb80cbd00 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f2fac0098f0 tx=0x7f2fac009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8eace0ac34f3539c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:18.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.866+0000 7f2fb559a700 1 -- 172.21.15.16:0/679926885 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fac011830 con 0x7f2fb80cb880 2023-10-16T13:49:18.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.866+0000 7f2fb559a700 1 -- 172.21.15.16:0/679926885 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fac011990 con 0x7f2fb80cb880 2023-10-16T13:49:18.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.866+0000 7f2fb559a700 1 -- 172.21.15.16:0/679926885 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fac011d20 con 0x7f2fb80cb880 2023-10-16T13:49:18.873 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.866+0000 7f2fbccc0700 1 -- 172.21.15.16:0/679926885 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2fb80cb880 msgr2=0x7f2fb80cbd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:18.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.866+0000 7f2fbccc0700 1 --2- 172.21.15.16:0/679926885 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2fb80cb880 0x7f2fb80cbd00 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f2fac0098f0 tx=0x7f2fac009c00 comp rx=0 tx=0).stop 2023-10-16T13:49:18.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.866+0000 7f2fbccc0700 1 -- 172.21.15.16:0/679926885 shutdown_connections 2023-10-16T13:49:18.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.866+0000 7f2fbccc0700 1 --2- 172.21.15.16:0/679926885 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2fb806da60 0x7f2fb806def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:18.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.866+0000 7f2fbccc0700 1 --2- 172.21.15.16:0/679926885 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2fb80cb880 0x7f2fb80cbd00 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:18.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.866+0000 7f2fbccc0700 1 -- 172.21.15.16:0/679926885 >> 172.21.15.16:0/679926885 conn(0x7f2fb80b1f40 msgr2=0x7f2fb80b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:18.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.866+0000 7f2fbccc0700 1 -- 172.21.15.16:0/679926885 shutdown_connections 2023-10-16T13:49:18.875 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.866+0000 7f2fbccc0700 1 -- 172.21.15.16:0/679926885 wait complete. 2023-10-16T13:49:18.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.868+0000 7f2fbccc0700 1 Processor -- start 2023-10-16T13:49:18.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.868+0000 7f2fbccc0700 1 -- start start 2023-10-16T13:49:18.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.869+0000 7f2fbccc0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2fb806da60 0x7f2fb80afe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:18.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.869+0000 7f2fbccc0700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2fb80cb880 0x7f2fb80b56a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:18.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.869+0000 7f2fbccc0700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2fb815b070 0x7f2fb80b5be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:18.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.869+0000 7f2fbccc0700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb80c6810 con 0x7f2fb806da60 2023-10-16T13:49:18.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.869+0000 7f2fbccc0700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb80c6a60 con 0x7f2fb80cb880 2023-10-16T13:49:18.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.869+0000 7f2fbccc0700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb80c6d40 con 0x7f2fb815b070 2023-10-16T13:49:18.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.869+0000 7f2fb659c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2fb806da60 0x7f2fb80afe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:18.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.869+0000 7f2fb659c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2fb806da60 0x7f2fb80afe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59582/0 (socket says 172.21.15.16:59582) 2023-10-16T13:49:18.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.869+0000 7f2fb659c700 1 -- 172.21.15.16:0/973748735 learned_addr learned my addr 172.21.15.16:0/973748735 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:18.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.869+0000 7f2fb6d9d700 1 --2- 172.21.15.16:0/973748735 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2fb815b070 0x7f2fb80b5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:18.880 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.869+0000 7f2fb5d9b700 1 --2- 172.21.15.16:0/973748735 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2fb80cb880 0x7f2fb80b56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:18.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.870+0000 7f2fb659c700 1 -- 172.21.15.16:0/973748735 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2fb815b070 msgr2=0x7f2fb80b5be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:18.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.870+0000 7f2fb659c700 1 --2- 172.21.15.16:0/973748735 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2fb815b070 0x7f2fb80b5be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:18.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.870+0000 7f2fb659c700 1 -- 172.21.15.16:0/973748735 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2fb80cb880 msgr2=0x7f2fb80b56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:18.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.870+0000 7f2fb659c700 1 --2- 172.21.15.16:0/973748735 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2fb80cb880 0x7f2fb80b56a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:18.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.870+0000 7f2fb659c700 1 -- 172.21.15.16:0/973748735 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fb000bc80 con 0x7f2fb806da60 2023-10-16T13:49:18.881 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.870+0000 7f2fb6d9d700 1 --2- 172.21.15.16:0/973748735 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f2fb815b070 0x7f2fb80b5be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:49:18.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.870+0000 7f2fb659c700 1 --2- 172.21.15.16:0/973748735 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2fb806da60 0x7f2fb80afe60 secure :-1 s=READY pgs=3240 cs=0 l=1 rev1=1 crypto rx=0x7f2fa800c430 tx=0x7f2fa800c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:18.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.870+0000 7f2fa6ffd700 1 -- 172.21.15.16:0/973748735 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa8004a70 con 0x7f2fb806da60 2023-10-16T13:49:18.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.870+0000 7f2fa6ffd700 1 -- 172.21.15.16:0/973748735 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fa8004bd0 con 0x7f2fb806da60 2023-10-16T13:49:18.882 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.870+0000 7f2fbccc0700 1 -- 172.21.15.16:0/973748735 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fac009550 con 0x7f2fb806da60 2023-10-16T13:49:18.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.870+0000 7f2fa6ffd700 1 -- 172.21.15.16:0/973748735 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa80070d0 con 0x7f2fb806da60 2023-10-16T13:49:18.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.870+0000 7f2fbccc0700 1 -- 172.21.15.16:0/973748735 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fb8152e30 con 0x7f2fb806da60 2023-10-16T13:49:18.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.871+0000 7f2fbccc0700 1 -- 172.21.15.16:0/973748735 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fb80c4740 con 0x7f2fb806da60 2023-10-16T13:49:18.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.872+0000 7f2fa6ffd700 1 -- 172.21.15.16:0/973748735 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 170) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f2fa8007230 con 0x7f2fb806da60 2023-10-16T13:49:18.883 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.872+0000 7f2fa6ffd700 1 --2- 172.21.15.16:0/973748735 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f2f9806ada0 0x7f2f9806d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:18.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.872+0000 7f2fa6ffd700 1 -- 172.21.15.16:0/973748735 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(279..279 src has 1..279) v4 ==== 6501+0+0 (secure 0 0 0) 0x7f2fa8088c40 con 0x7f2fb806da60 2023-10-16T13:49:18.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.872+0000 7f2fb5d9b700 1 --2- 172.21.15.16:0/973748735 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f2f9806ada0 0x7f2f9806d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:18.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.872+0000 7f2fb5d9b700 1 --2- 172.21.15.16:0/973748735 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f2f9806ada0 0x7f2f9806d260 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f2fac009c40 tx=0x7f2fac008030 comp rx=0 tx=0).ready entity=mgr.10586 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:18.884 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:18.877+0000 7f2fa6ffd700 1 -- 172.21.15.16:0/973748735 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2fb80c4740 con 0x7f2fb806da60 2023-10-16T13:49:19.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.104+0000 7f2fbccc0700 1 -- 172.21.15.16:0/973748735 --> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "subvol": "sv1", "group": "mygrp", "path": ".", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f2fb80c4380 con 0x7f2f9806ada0 2023-10-16T13:49:19.109 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:19.106+0000 7fb4088b1700 -1 mgr.server reply reply (22) Invalid argument No such subvol: mygrp::sv1 2023-10-16T13:49:19.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.106+0000 7f2fa6ffd700 1 -- 172.21.15.16:0/973748735 <== mgr.10586 v2:172.21.15.16:6832/3920835901 1 ==== mgr_command_reply(tid 0: -22 No such subvol: mygrp::sv1) v1 ==== 34+0+0 (secure 0 0 0) 0x7f2fac011b80 con 0x7f2f9806ada0 2023-10-16T13:49:19.111 INFO:teuthology.orchestra.run.smithi016.stderr:Error EINVAL: No such subvol: mygrp::sv1 2023-10-16T13:49:19.114 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-10-16T13:49:19.115 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --subvol sv1 . 1m 2023-10-16T13:49:19.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.245+0000 7f91d834b700 1 Processor -- start 2023-10-16T13:49:19.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.245+0000 7f91d834b700 1 -- start start 2023-10-16T13:49:19.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.246+0000 7f91d834b700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f91d00cf600 0x7f91d006d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:19.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.246+0000 7f91d834b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f91d015e0b0 0x7f91d00cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:19.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.246+0000 7f91d834b700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91d00cb870 con 0x7f91d015a3d0 2023-10-16T13:49:19.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.246+0000 7f91d834b700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91d015f0c0 con 0x7f91d015e0b0 2023-10-16T13:49:19.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.246+0000 7f91d834b700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91d0072990 con 0x7f91d00cf600 2023-10-16T13:49:19.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.246+0000 7f91d68e8700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f91d015e0b0 0x7f91d00cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:19.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.246+0000 7f91d68e8700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f91d015e0b0 0x7f91d00cbd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59596/0 (socket says 172.21.15.16:59596) 2023-10-16T13:49:19.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.246+0000 7f91d68e8700 1 -- 172.21.15.16:0/2779054191 learned_addr learned my addr 172.21.15.16:0/2779054191 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:19.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.246+0000 7f91d58e6700 1 --2- 172.21.15.16:0/2779054191 >> v2:172.21.15.49:3300/0 conn(0x7f91d00cf600 0x7f91d006d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:19.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.246+0000 7f91d68e8700 1 -- 172.21.15.16:0/2779054191 >> v2:172.21.15.49:3300/0 conn(0x7f91d00cf600 msgr2=0x7f91d006d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:19.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.246+0000 7f91d68e8700 1 --2- 172.21.15.16:0/2779054191 >> v2:172.21.15.49:3300/0 conn(0x7f91d00cf600 0x7f91d006d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:19.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.246+0000 7f91d68e8700 1 -- 172.21.15.16:0/2779054191 >> v1:172.21.15.16:6789/0 conn(0x7f91d015a3d0 legacy=0x7f91d00ceef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:19.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.246+0000 7f91d68e8700 1 -- 172.21.15.16:0/2779054191 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91d006e040 con 0x7f91d015e0b0 2023-10-16T13:49:19.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.247+0000 7f91d68e8700 1 --2- 172.21.15.16:0/2779054191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91d015e0b0 0x7f91d00cbd00 secure :-1 s=READY pgs=3242 cs=0 l=1 rev1=1 crypto rx=0x7f91d015e010 tx=0x7f91c800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d74f03e4abd8bc8a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:19.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.247+0000 7f91d50e5700 1 -- 172.21.15.16:0/2779054191 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91c800e070 con 0x7f91d015e0b0 2023-10-16T13:49:19.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.247+0000 7f91d50e5700 1 -- 172.21.15.16:0/2779054191 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91c800b910 con 0x7f91d015e0b0 2023-10-16T13:49:19.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.247+0000 7f91d50e5700 1 -- 172.21.15.16:0/2779054191 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91c800bcc0 con 0x7f91d015e0b0 2023-10-16T13:49:19.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.247+0000 7f91d834b700 1 -- 172.21.15.16:0/2779054191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91d015e0b0 msgr2=0x7f91d00cbd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:19.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.247+0000 7f91d834b700 1 --2- 172.21.15.16:0/2779054191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91d015e0b0 0x7f91d00cbd00 secure :-1 s=READY pgs=3242 cs=0 l=1 rev1=1 crypto rx=0x7f91d015e010 tx=0x7f91c800b330 comp rx=0 tx=0).stop 2023-10-16T13:49:19.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.247+0000 7f91d834b700 1 -- 172.21.15.16:0/2779054191 shutdown_connections 2023-10-16T13:49:19.254 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.247+0000 7f91d834b700 1 --2- 172.21.15.16:0/2779054191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91d015e0b0 0x7f91d00cbd00 unknown :-1 s=CLOSED pgs=3242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:19.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.247+0000 7f91d834b700 1 --2- 172.21.15.16:0/2779054191 >> v2:172.21.15.49:3300/0 conn(0x7f91d00cf600 0x7f91d006d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:19.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.247+0000 7f91d834b700 1 -- 172.21.15.16:0/2779054191 >> 172.21.15.16:0/2779054191 conn(0x7f91d006f430 msgr2=0x7f91d006f810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:19.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.247+0000 7f91d834b700 1 -- 172.21.15.16:0/2779054191 shutdown_connections 2023-10-16T13:49:19.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.248+0000 7f91d834b700 1 -- 172.21.15.16:0/2779054191 wait complete. 2023-10-16T13:49:19.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.249+0000 7f91d834b700 1 Processor -- start 2023-10-16T13:49:19.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.249+0000 7f91d834b700 1 -- start start 2023-10-16T13:49:19.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.250+0000 7f91d834b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f91d00cf600 0x7f91d00c8a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:19.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.250+0000 7f91d834b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f91d015a3d0 0x7f91d00cde60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:19.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.250+0000 7f91d834b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91d015e0b0 0x7f91d00ce3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:19.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.250+0000 7f91d834b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91d00ced00 con 0x7f91d015e0b0 2023-10-16T13:49:19.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.250+0000 7f91d834b700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91d01f24f0 con 0x7f91d015a3d0 2023-10-16T13:49:19.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.250+0000 7f91d834b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91d00cea20 con 0x7f91d00cf600 2023-10-16T13:49:19.259 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.250+0000 7f91d68e8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91d015e0b0 0x7f91d00ce3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:19.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.250+0000 7f91d68e8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91d015e0b0 0x7f91d00ce3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59602/0 (socket says 172.21.15.16:59602) 2023-10-16T13:49:19.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.250+0000 7f91d68e8700 1 -- 172.21.15.16:0/163968683 learned_addr learned my addr 172.21.15.16:0/163968683 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:19.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.250+0000 7f91d60e7700 1 --2- 172.21.15.16:0/163968683 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f91d00cf600 0x7f91d00c8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:19.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.250+0000 7f91d58e6700 1 --2- 172.21.15.16:0/163968683 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f91d015a3d0 0x7f91d00cde60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:19.260 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.251+0000 7f91d68e8700 1 -- 172.21.15.16:0/163968683 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f91d00cf600 msgr2=0x7f91d00c8a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:19.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.251+0000 7f91d68e8700 1 --2- 172.21.15.16:0/163968683 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f91d00cf600 0x7f91d00c8a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:19.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.251+0000 7f91d68e8700 1 -- 172.21.15.16:0/163968683 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f91d015a3d0 msgr2=0x7f91d00cde60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:19.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.251+0000 7f91d68e8700 1 --2- 172.21.15.16:0/163968683 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f91d015a3d0 0x7f91d00cde60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:19.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.251+0000 7f91d68e8700 1 -- 172.21.15.16:0/163968683 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91c8009b30 con 0x7f91d015e0b0 2023-10-16T13:49:19.261 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.251+0000 7f91d68e8700 1 --2- 172.21.15.16:0/163968683 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91d015e0b0 0x7f91d00ce3a0 secure :-1 s=READY pgs=3243 cs=0 l=1 rev1=1 crypto rx=0x7f91d0067440 tx=0x7f91c8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:19.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.251+0000 7f91c6ffd700 1 -- 172.21.15.16:0/163968683 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91c8004880 con 0x7f91d015e0b0 2023-10-16T13:49:19.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.251+0000 7f91c6ffd700 1 -- 172.21.15.16:0/163968683 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91c80092e0 con 0x7f91d015e0b0 2023-10-16T13:49:19.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.251+0000 7f91d834b700 1 -- 172.21.15.16:0/163968683 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91d01f1ea0 con 0x7f91d015e0b0 2023-10-16T13:49:19.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.251+0000 7f91c6ffd700 1 -- 172.21.15.16:0/163968683 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91c801b650 con 0x7f91d015e0b0 2023-10-16T13:49:19.262 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.251+0000 7f91d834b700 1 -- 172.21.15.16:0/163968683 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91d00c8f50 con 0x7f91d015e0b0 2023-10-16T13:49:19.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.252+0000 7f91d834b700 1 -- 172.21.15.16:0/163968683 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91d01f1d00 con 0x7f91d015e0b0 2023-10-16T13:49:19.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.253+0000 7f91c6ffd700 1 -- 172.21.15.16:0/163968683 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 170) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f91c801b7b0 con 0x7f91d015e0b0 2023-10-16T13:49:19.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.253+0000 7f91c6ffd700 1 --2- 172.21.15.16:0/163968683 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f91b406ada0 0x7f91b406d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:19.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.253+0000 7f91c6ffd700 1 -- 172.21.15.16:0/163968683 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(279..279 src has 1..279) v4 ==== 6501+0+0 (secure 0 0 0) 0x7f91c808adc0 con 0x7f91d015e0b0 2023-10-16T13:49:19.263 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.253+0000 7f91d60e7700 1 --2- 172.21.15.16:0/163968683 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f91b406ada0 0x7f91b406d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:19.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.254+0000 7f91d60e7700 1 --2- 172.21.15.16:0/163968683 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f91b406ada0 0x7f91b406d260 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f91cc007fd0 tx=0x7f91cc009040 comp rx=0 tx=0).ready entity=mgr.10586 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:19.264 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.258+0000 7f91c6ffd700 1 -- 172.21.15.16:0/163968683 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f91d01f1d00 con 0x7f91d015e0b0 2023-10-16T13:49:19.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.488+0000 7f91d834b700 1 -- 172.21.15.16:0/163968683 --> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "subvol": "sv1", "path": ".", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f91d006e600 con 0x7f91b406ada0 2023-10-16T13:49:19.557 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.555+0000 7f91c6ffd700 1 -- 172.21.15.16:0/163968683 <== mgr.10586 v2:172.21.15.16:6832/3920835901 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+88 (secure 0 0 0) 0x7f91cc001d60 con 0x7f91b406ada0 2023-10-16T13:49:19.558 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule removed for path /volumes/_nogroup/sv1/bcfda3a0-ba49-4161-81cb-81cdfd123292/.. 2023-10-16T13:49:19.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.557+0000 7f91d834b700 1 -- 172.21.15.16:0/163968683 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f91b406ada0 msgr2=0x7f91b406d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:19.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.557+0000 7f91d834b700 1 --2- 172.21.15.16:0/163968683 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f91b406ada0 0x7f91b406d260 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f91cc007fd0 tx=0x7f91cc009040 comp rx=0 tx=0).stop 2023-10-16T13:49:19.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.557+0000 7f91d834b700 1 -- 172.21.15.16:0/163968683 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91d015e0b0 msgr2=0x7f91d00ce3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:19.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.557+0000 7f91d834b700 1 --2- 172.21.15.16:0/163968683 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91d015e0b0 0x7f91d00ce3a0 secure :-1 s=READY pgs=3243 cs=0 l=1 rev1=1 crypto rx=0x7f91d0067440 tx=0x7f91c8004ad0 comp rx=0 tx=0).stop 2023-10-16T13:49:19.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.557+0000 7f91d834b700 1 -- 172.21.15.16:0/163968683 shutdown_connections 2023-10-16T13:49:19.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.557+0000 7f91d834b700 1 --2- 172.21.15.16:0/163968683 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f91b406ada0 0x7f91b406d260 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:19.561 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.557+0000 7f91d834b700 1 --2- 172.21.15.16:0/163968683 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91d015e0b0 0x7f91d00ce3a0 unknown :-1 s=CLOSED pgs=3243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:19.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.557+0000 7f91d834b700 1 --2- 172.21.15.16:0/163968683 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f91d00cf600 0x7f91d00c8a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:19.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.557+0000 7f91d834b700 1 --2- 172.21.15.16:0/163968683 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f91d015a3d0 0x7f91d00cde60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:19.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.557+0000 7f91d834b700 1 -- 172.21.15.16:0/163968683 >> 172.21.15.16:0/163968683 conn(0x7f91d006f430 msgr2=0x7f91d00cbd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:19.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.557+0000 7f91d834b700 1 -- 172.21.15.16:0/163968683 shutdown_connections 2023-10-16T13:49:19.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.557+0000 7f91d834b700 1 -- 172.21.15.16:0/163968683 wait complete. 2023-10-16T13:49:19.572 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /volumes/_nogroup/sv1/bcfda3a0-ba49-4161-81cb-81cdfd123292/.. 2023-10-16T13:49:19.572 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs sv1 2023-10-16T13:49:19.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.698+0000 7fd67ee7b700 1 Processor -- start 2023-10-16T13:49:19.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.698+0000 7fd67ee7b700 1 -- start start 2023-10-16T13:49:19.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.699+0000 7fd67ee7b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd67815ec80 0x7fd6780cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:19.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.699+0000 7fd67ee7b700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6780b2440 con 0x7fd6780d2110 2023-10-16T13:49:19.703 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.699+0000 7fd67ee7b700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6780d00e0 con 0x7fd67815f060 2023-10-16T13:49:19.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.699+0000 7fd67ee7b700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6780b2c50 con 0x7fd67815ec80 2023-10-16T13:49:19.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.699+0000 7fd67cc17700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd67815ec80 0x7fd6780cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:19.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.699+0000 7fd67d418700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd6780d2110 0x7fd6780b4b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48898/0 (socket says 172.21.15.16:48898) 2023-10-16T13:49:19.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.699+0000 7fd67d418700 1 -- 172.21.15.16:0/2484918431 learned_addr learned my addr 172.21.15.16:0/2484918431 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:49:19.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.699+0000 7fd67cc17700 1 -- 172.21.15.16:0/2484918431 >> v1:172.21.15.49:6789/0 conn(0x7fd67815f060 legacy=0x7fd6780b4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:19.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.699+0000 7fd67cc17700 1 -- 172.21.15.16:0/2484918431 >> v1:172.21.15.16:6789/0 conn(0x7fd6780d2110 legacy=0x7fd6780b4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:19.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.699+0000 7fd67cc17700 1 -- 172.21.15.16:0/2484918431 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6780cef60 con 0x7fd67815ec80 2023-10-16T13:49:19.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.699+0000 7fd67cc17700 1 --2- 172.21.15.16:0/2484918431 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd67815ec80 0x7fd6780cdec0 secure :-1 s=READY pgs=3247 cs=0 l=1 rev1=1 crypto rx=0x7fd66c009840 tx=0x7fd66c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3531370f8f6b6ba server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:19.705 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.700+0000 7fd6777fe700 1 -- 172.21.15.16:0/2484918431 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd66c011830 con 0x7fd67815ec80 2023-10-16T13:49:19.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.700+0000 7fd6777fe700 1 -- 172.21.15.16:0/2484918431 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd66c011990 con 0x7fd67815ec80 2023-10-16T13:49:19.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.700+0000 7fd6777fe700 1 -- 172.21.15.16:0/2484918431 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd66c011d20 con 0x7fd67815ec80 2023-10-16T13:49:19.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.700+0000 7fd67ee7b700 1 -- 172.21.15.16:0/2484918431 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd67815ec80 msgr2=0x7fd6780cdec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:19.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.700+0000 7fd67ee7b700 1 --2- 172.21.15.16:0/2484918431 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd67815ec80 0x7fd6780cdec0 secure :-1 s=READY pgs=3247 cs=0 l=1 rev1=1 crypto rx=0x7fd66c009840 tx=0x7fd66c009b50 comp rx=0 tx=0).stop 2023-10-16T13:49:19.706 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.700+0000 7fd67ee7b700 1 -- 172.21.15.16:0/2484918431 shutdown_connections 2023-10-16T13:49:19.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.700+0000 7fd67ee7b700 1 --2- 172.21.15.16:0/2484918431 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd67815ec80 0x7fd6780cdec0 unknown :-1 s=CLOSED pgs=3247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:19.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.700+0000 7fd67ee7b700 1 -- 172.21.15.16:0/2484918431 >> 172.21.15.16:0/2484918431 conn(0x7fd6780d15f0 msgr2=0x7fd6780d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:19.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.700+0000 7fd67ee7b700 1 -- 172.21.15.16:0/2484918431 shutdown_connections 2023-10-16T13:49:19.707 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.700+0000 7fd67ee7b700 1 -- 172.21.15.16:0/2484918431 wait complete. 2023-10-16T13:49:19.710 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.702+0000 7fd67ee7b700 1 Processor -- start 2023-10-16T13:49:19.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.702+0000 7fd67ee7b700 1 -- start start 2023-10-16T13:49:19.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.703+0000 7fd67ee7b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd6780d2110 0x7fd678151e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:19.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.703+0000 7fd67ee7b700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd67815ec80 0x7fd6781e5aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:19.711 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.703+0000 7fd67ee7b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd67815f060 0x7fd6780c9c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:19.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.703+0000 7fd67ee7b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd67806ec00 con 0x7fd67815f060 2023-10-16T13:49:19.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.703+0000 7fd67ee7b700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd67806eee0 con 0x7fd6780d2110 2023-10-16T13:49:19.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.703+0000 7fd67ee7b700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd678152380 con 0x7fd67815ec80 2023-10-16T13:49:19.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.703+0000 7fd67d418700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd67815f060 0x7fd6780c9c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:19.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.703+0000 7fd67d418700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd67815f060 0x7fd6780c9c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59630/0 (socket says 172.21.15.16:59630) 2023-10-16T13:49:19.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.703+0000 7fd67d418700 1 -- 172.21.15.16:0/753293599 learned_addr learned my addr 172.21.15.16:0/753293599 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:19.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.703+0000 7fd67cc17700 1 --2- 172.21.15.16:0/753293599 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd6780d2110 0x7fd678151e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:19.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.703+0000 7fd677fff700 1 --2- 172.21.15.16:0/753293599 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd67815ec80 0x7fd6781e5aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:19.713 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.703+0000 7fd67d418700 1 -- 172.21.15.16:0/753293599 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd67815ec80 msgr2=0x7fd6781e5aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:19.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.703+0000 7fd67d418700 1 --2- 172.21.15.16:0/753293599 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd67815ec80 0x7fd6781e5aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:19.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.703+0000 7fd67d418700 1 -- 172.21.15.16:0/753293599 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd6780d2110 msgr2=0x7fd678151e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:19.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.703+0000 7fd67d418700 1 --2- 172.21.15.16:0/753293599 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd6780d2110 0x7fd678151e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:19.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.703+0000 7fd67d418700 1 -- 172.21.15.16:0/753293599 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd66c0094a0 con 0x7fd67815f060 2023-10-16T13:49:19.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.704+0000 7fd67d418700 1 --2- 172.21.15.16:0/753293599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd67815f060 0x7fd6780c9c10 secure :-1 s=READY pgs=3248 cs=0 l=1 rev1=1 crypto rx=0x7fd67000cc20 tx=0x7fd67000cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:19.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.704+0000 7fd6757fa700 1 -- 172.21.15.16:0/753293599 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd67000aa70 con 0x7fd67815f060 2023-10-16T13:49:19.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.704+0000 7fd6757fa700 1 -- 172.21.15.16:0/753293599 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd67000abd0 con 0x7fd67815f060 2023-10-16T13:49:19.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.704+0000 7fd6757fa700 1 -- 172.21.15.16:0/753293599 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd670012400 con 0x7fd67815f060 2023-10-16T13:49:19.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.704+0000 7fd67ee7b700 1 -- 172.21.15.16:0/753293599 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6780c5610 con 0x7fd67815f060 2023-10-16T13:49:19.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.704+0000 7fd67ee7b700 1 -- 172.21.15.16:0/753293599 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6780c94e0 con 0x7fd67815f060 2023-10-16T13:49:19.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.705+0000 7fd67ee7b700 1 -- 172.21.15.16:0/753293599 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd67804ebb0 con 0x7fd67815f060 2023-10-16T13:49:19.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.705+0000 7fd6757fa700 1 -- 172.21.15.16:0/753293599 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 170) v1 ==== 89084+0+0 (secure 0 0 0) 0x7fd67001b3b0 con 0x7fd67815f060 2023-10-16T13:49:19.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.706+0000 7fd6757fa700 1 --2- 172.21.15.16:0/753293599 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7fd65c06ada0 0x7fd65c06d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:19.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.706+0000 7fd6757fa700 1 -- 172.21.15.16:0/753293599 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(279..279 src has 1..279) v4 ==== 6501+0+0 (secure 0 0 0) 0x7fd670089100 con 0x7fd67815f060 2023-10-16T13:49:19.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.706+0000 7fd67cc17700 1 --2- 172.21.15.16:0/753293599 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7fd65c06ada0 0x7fd65c06d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:19.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.706+0000 7fd67cc17700 1 --2- 172.21.15.16:0/753293599 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7fd65c06ada0 0x7fd65c06d260 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd66c0090b0 tx=0x7fd66c008030 comp rx=0 tx=0).ready entity=mgr.10586 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:19.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.711+0000 7fd6757fa700 1 -- 172.21.15.16:0/753293599 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd670054280 con 0x7fd67815f060 2023-10-16T13:49:19.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.951+0000 7fd67ee7b700 1 -- 172.21.15.16:0/753293599 --> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7fd6780b2970 con 0x7fd65c06ada0 2023-10-16T13:49:19.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.970+0000 7fd6757fa700 1 -- 172.21.15.16:0/753293599 <== mgr.10586 v2:172.21.15.16:6832/3920835901 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fd66c011820 con 0x7fd65c06ada0 2023-10-16T13:49:19.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.972+0000 7fd67ee7b700 1 -- 172.21.15.16:0/753293599 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7fd65c06ada0 msgr2=0x7fd65c06d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:19.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.972+0000 7fd67ee7b700 1 --2- 172.21.15.16:0/753293599 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7fd65c06ada0 0x7fd65c06d260 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd66c0090b0 tx=0x7fd66c008030 comp rx=0 tx=0).stop 2023-10-16T13:49:19.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.972+0000 7fd67ee7b700 1 -- 172.21.15.16:0/753293599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd67815f060 msgr2=0x7fd6780c9c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:19.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.972+0000 7fd67ee7b700 1 --2- 172.21.15.16:0/753293599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd67815f060 0x7fd6780c9c10 secure :-1 s=READY pgs=3248 cs=0 l=1 rev1=1 crypto rx=0x7fd67000cc20 tx=0x7fd67000cf30 comp rx=0 tx=0).stop 2023-10-16T13:49:19.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.972+0000 7fd67ee7b700 1 -- 172.21.15.16:0/753293599 shutdown_connections 2023-10-16T13:49:19.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.972+0000 7fd67ee7b700 1 --2- 172.21.15.16:0/753293599 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7fd65c06ada0 0x7fd65c06d260 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:19.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.972+0000 7fd67ee7b700 1 --2- 172.21.15.16:0/753293599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd67815f060 0x7fd6780c9c10 unknown :-1 s=CLOSED pgs=3248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:19.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.972+0000 7fd67ee7b700 1 --2- 172.21.15.16:0/753293599 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd6780d2110 0x7fd678151e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:19.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.972+0000 7fd67ee7b700 1 --2- 172.21.15.16:0/753293599 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd67815ec80 0x7fd6781e5aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:19.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.972+0000 7fd67ee7b700 1 -- 172.21.15.16:0/753293599 >> 172.21.15.16:0/753293599 conn(0x7fd6780d15f0 msgr2=0x7fd6780b6cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:19.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.972+0000 7fd67ee7b700 1 -- 172.21.15.16:0/753293599 shutdown_connections 2023-10-16T13:49:19.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:19.972+0000 7fd67ee7b700 1 -- 172.21.15.16:0/753293599 wait complete. 2023-10-16T13:49:19.982 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:19.980+0000 7fb3fd85b700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:49:19.983 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:19.980+0000 7fb3fd85b700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:49:19.983 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:19.980+0000 7fb3fd85b700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:49:19.983 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:19.980+0000 7fb3fd85b700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:49:19.983 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:19.980+0000 7fb3fd85b700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:49:19.989 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs mygrp 2023-10-16T13:49:20.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.114+0000 7f5fe4fb4700 1 Processor -- start 2023-10-16T13:49:20.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.115+0000 7f5fe4fb4700 1 -- start start 2023-10-16T13:49:20.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.115+0000 7f5fe4fb4700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5fe00ce6a0 0x7f5fe006c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:20.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.115+0000 7f5fe4fb4700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fe00b20f0 con 0x7f5fe00d2110 2023-10-16T13:49:20.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.115+0000 7f5fe4fb4700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fe00af1f0 con 0x7f5fe00b2620 2023-10-16T13:49:20.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.115+0000 7f5fe4fb4700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fe00b1d70 con 0x7f5fe00ce6a0 2023-10-16T13:49:20.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.116+0000 7f5fde59c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5fe00ce6a0 0x7f5fe006c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:20.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.116+0000 7f5fddd9b700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f5fe00d2110 0x7f5fe00b0f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48922/0 (socket says 172.21.15.16:48922) 2023-10-16T13:49:20.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.116+0000 7f5fddd9b700 1 -- 172.21.15.16:0/1603491950 learned_addr learned my addr 172.21.15.16:0/1603491950 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:49:20.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.116+0000 7f5fde59c700 1 -- 172.21.15.16:0/1603491950 >> v1:172.21.15.49:6789/0 conn(0x7f5fe00b2620 legacy=0x7f5fe00b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:20.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.116+0000 7f5fde59c700 1 -- 172.21.15.16:0/1603491950 >> v1:172.21.15.16:6789/0 conn(0x7f5fe00d2110 legacy=0x7f5fe00b0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:20.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.116+0000 7f5fde59c700 1 -- 172.21.15.16:0/1603491950 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fe00d03d0 con 0x7f5fe00ce6a0 2023-10-16T13:49:20.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.116+0000 7f5fdd59a700 1 -- 172.21.15.16:0/1603491950 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2031555585 0 0) 0x7f5fe00b20f0 con 0x7f5fe00d2110 2023-10-16T13:49:20.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.116+0000 7f5fde59c700 1 --2- 172.21.15.16:0/1603491950 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5fe00ce6a0 0x7f5fe006c300 secure :-1 s=READY pgs=3252 cs=0 l=1 rev1=1 crypto rx=0x7f5fd0009840 tx=0x7f5fd0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1daa6062c2bc2f2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:20.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.116+0000 7f5fdd59a700 1 -- 172.21.15.16:0/1603491950 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fd0004230 con 0x7f5fe00ce6a0 2023-10-16T13:49:20.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.116+0000 7f5fdd59a700 1 -- 172.21.15.16:0/1603491950 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fd0004390 con 0x7f5fe00ce6a0 2023-10-16T13:49:20.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.116+0000 7f5fdd59a700 1 -- 172.21.15.16:0/1603491950 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fd00036a0 con 0x7f5fe00ce6a0 2023-10-16T13:49:20.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.117+0000 7f5fe4fb4700 1 -- 172.21.15.16:0/1603491950 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5fe00ce6a0 msgr2=0x7f5fe006c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:20.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.117+0000 7f5fe4fb4700 1 --2- 172.21.15.16:0/1603491950 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5fe00ce6a0 0x7f5fe006c300 secure :-1 s=READY pgs=3252 cs=0 l=1 rev1=1 crypto rx=0x7f5fd0009840 tx=0x7f5fd0009b50 comp rx=0 tx=0).stop 2023-10-16T13:49:20.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.117+0000 7f5fe4fb4700 1 -- 172.21.15.16:0/1603491950 shutdown_connections 2023-10-16T13:49:20.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.117+0000 7f5fe4fb4700 1 --2- 172.21.15.16:0/1603491950 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5fe00ce6a0 0x7f5fe006c300 unknown :-1 s=CLOSED pgs=3252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:20.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.117+0000 7f5fe4fb4700 1 -- 172.21.15.16:0/1603491950 >> 172.21.15.16:0/1603491950 conn(0x7f5fe00d15f0 msgr2=0x7f5fe00d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:20.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.117+0000 7f5fe4fb4700 1 -- 172.21.15.16:0/1603491950 shutdown_connections 2023-10-16T13:49:20.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.117+0000 7f5fe4fb4700 1 -- 172.21.15.16:0/1603491950 wait complete. 2023-10-16T13:49:20.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.118+0000 7f5fe4fb4700 1 Processor -- start 2023-10-16T13:49:20.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.118+0000 7f5fe4fb4700 1 -- start start 2023-10-16T13:49:20.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.119+0000 7f5fe4fb4700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5fe00b2620 0x7f5fe01f3c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:20.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.119+0000 7f5fe4fb4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5fe00ce6a0 0x7f5fe00b6680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:20.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.119+0000 7f5fe4fb4700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5fe00d2110 0x7f5fe00b6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:20.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.119+0000 7f5fe4fb4700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe00c5e90 con 0x7f5fe00ce6a0 2023-10-16T13:49:20.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.119+0000 7f5fe4fb4700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe00c52c0 con 0x7f5fe00d2110 2023-10-16T13:49:20.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.119+0000 7f5fe4fb4700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe00c5560 con 0x7f5fe00b2620 2023-10-16T13:49:20.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.119+0000 7f5fddd9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5fe00ce6a0 0x7f5fe00b6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:20.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.119+0000 7f5fddd9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5fe00ce6a0 0x7f5fe00b6680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59652/0 (socket says 172.21.15.16:59652) 2023-10-16T13:49:20.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.119+0000 7f5fddd9b700 1 -- 172.21.15.16:0/392586389 learned_addr learned my addr 172.21.15.16:0/392586389 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:20.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.119+0000 7f5fded9d700 1 --2- 172.21.15.16:0/392586389 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5fe00d2110 0x7f5fe00b6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:20.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.120+0000 7f5fde59c700 1 --2- 172.21.15.16:0/392586389 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5fe00b2620 0x7f5fe01f3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:20.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.120+0000 7f5fddd9b700 1 -- 172.21.15.16:0/392586389 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5fe00b2620 msgr2=0x7f5fe01f3c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:20.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.120+0000 7f5fddd9b700 1 --2- 172.21.15.16:0/392586389 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5fe00b2620 0x7f5fe01f3c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:20.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.120+0000 7f5fddd9b700 1 -- 172.21.15.16:0/392586389 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5fe00d2110 msgr2=0x7f5fe00b6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:20.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.120+0000 7f5fddd9b700 1 --2- 172.21.15.16:0/392586389 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5fe00d2110 0x7f5fe00b6bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:20.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.120+0000 7f5fddd9b700 1 -- 172.21.15.16:0/392586389 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fd00094a0 con 0x7f5fe00ce6a0 2023-10-16T13:49:20.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.120+0000 7f5fded9d700 1 --2- 172.21.15.16:0/392586389 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5fe00d2110 0x7f5fe00b6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:49:20.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.120+0000 7f5fddd9b700 1 --2- 172.21.15.16:0/392586389 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5fe00ce6a0 0x7f5fe00b6680 secure :-1 s=READY pgs=3253 cs=0 l=1 rev1=1 crypto rx=0x7f5fc800c460 tx=0x7f5fc800c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:20.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.120+0000 7f5fd6ffd700 1 -- 172.21.15.16:0/392586389 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fc8004c80 con 0x7f5fe00ce6a0 2023-10-16T13:49:20.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.120+0000 7f5fd6ffd700 1 -- 172.21.15.16:0/392586389 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fc8004e00 con 0x7f5fe00ce6a0 2023-10-16T13:49:20.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.120+0000 7f5fe4fb4700 1 -- 172.21.15.16:0/392586389 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fe00c5840 con 0x7f5fe00ce6a0 2023-10-16T13:49:20.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.120+0000 7f5fd6ffd700 1 -- 172.21.15.16:0/392586389 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fc8007310 con 0x7f5fe00ce6a0 2023-10-16T13:49:20.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.120+0000 7f5fe4fb4700 1 -- 172.21.15.16:0/392586389 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fe00c6320 con 0x7f5fe00ce6a0 2023-10-16T13:49:20.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.121+0000 7f5fe4fb4700 1 -- 172.21.15.16:0/392586389 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fe004ebb0 con 0x7f5fe00ce6a0 2023-10-16T13:49:20.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.121+0000 7f5fd6ffd700 1 -- 172.21.15.16:0/392586389 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 170) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f5fc8007470 con 0x7f5fe00ce6a0 2023-10-16T13:49:20.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.122+0000 7f5fd6ffd700 1 --2- 172.21.15.16:0/392586389 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f5fc006ada0 0x7f5fc006d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:20.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.122+0000 7f5fd6ffd700 1 -- 172.21.15.16:0/392586389 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(279..279 src has 1..279) v4 ==== 6501+0+0 (secure 0 0 0) 0x7f5fc8089220 con 0x7f5fe00ce6a0 2023-10-16T13:49:20.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.122+0000 7f5fde59c700 1 --2- 172.21.15.16:0/392586389 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f5fc006ada0 0x7f5fc006d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:20.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.122+0000 7f5fde59c700 1 --2- 172.21.15.16:0/392586389 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f5fc006ada0 0x7f5fc006d260 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f5fd0009b90 tx=0x7f5fd0008030 comp rx=0 tx=0).ready entity=mgr.10586 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:20.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.127+0000 7f5fd6ffd700 1 -- 172.21.15.16:0/392586389 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5fc80542f0 con 0x7f5fe00ce6a0 2023-10-16T13:49:20.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.215+0000 7f5fd6ffd700 1 -- 172.21.15.16:0/392586389 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 171) v1 ==== 89400+0+0 (secure 0 0 0) 0x7f5fc80546c0 con 0x7f5fe00ce6a0 2023-10-16T13:49:20.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.362+0000 7f5fe4fb4700 1 -- 172.21.15.16:0/392586389 --> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "mygrp", "target": ["mon-mgr", ""]}) v1 -- 0x7f5fe006bb00 con 0x7f5fc006ada0 2023-10-16T13:49:20.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.374+0000 7f5fd6ffd700 1 -- 172.21.15.16:0/392586389 <== mgr.10586 v2:172.21.15.16:6832/3920835901 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f5fd00040b0 con 0x7f5fc006ada0 2023-10-16T13:49:20.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.376+0000 7f5fe4fb4700 1 -- 172.21.15.16:0/392586389 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f5fc006ada0 msgr2=0x7f5fc006d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:20.379 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.376+0000 7f5fe4fb4700 1 --2- 172.21.15.16:0/392586389 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f5fc006ada0 0x7f5fc006d260 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f5fd0009b90 tx=0x7f5fd0008030 comp rx=0 tx=0).stop 2023-10-16T13:49:20.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.376+0000 7f5fe4fb4700 1 -- 172.21.15.16:0/392586389 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5fe00ce6a0 msgr2=0x7f5fe00b6680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:20.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.376+0000 7f5fe4fb4700 1 --2- 172.21.15.16:0/392586389 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5fe00ce6a0 0x7f5fe00b6680 secure :-1 s=READY pgs=3253 cs=0 l=1 rev1=1 crypto rx=0x7f5fc800c460 tx=0x7f5fc800c820 comp rx=0 tx=0).stop 2023-10-16T13:49:20.380 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.376+0000 7f5fe4fb4700 1 -- 172.21.15.16:0/392586389 shutdown_connections 2023-10-16T13:49:20.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.376+0000 7f5fe4fb4700 1 --2- 172.21.15.16:0/392586389 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f5fc006ada0 0x7f5fc006d260 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:20.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.376+0000 7f5fe4fb4700 1 --2- 172.21.15.16:0/392586389 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5fe00d2110 0x7f5fe00b6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:20.381 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.376+0000 7f5fe4fb4700 1 --2- 172.21.15.16:0/392586389 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5fe00b2620 0x7f5fe01f3c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:20.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.376+0000 7f5fe4fb4700 1 --2- 172.21.15.16:0/392586389 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5fe00ce6a0 0x7f5fe00b6680 unknown :-1 s=CLOSED pgs=3253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:20.382 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.376+0000 7f5fe4fb4700 1 -- 172.21.15.16:0/392586389 >> 172.21.15.16:0/392586389 conn(0x7f5fe00d15f0 msgr2=0x7f5fe00b04f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:20.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.376+0000 7f5fe4fb4700 1 -- 172.21.15.16:0/392586389 shutdown_connections 2023-10-16T13:49:20.383 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.376+0000 7f5fe4fb4700 1 -- 172.21.15.16:0/392586389 wait complete. 2023-10-16T13:49:20.391 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv1 --group_name mygrp 2023-10-16T13:49:20.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.519+0000 7fec26cb7700 1 Processor -- start 2023-10-16T13:49:20.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.519+0000 7fec26cb7700 1 -- start start 2023-10-16T13:49:20.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.520+0000 7fec26cb7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fec20160e60 0x7fec2015e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:20.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.520+0000 7fec26cb7700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fec2006a7e0 0x7fec2006ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:20.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.520+0000 7fec26cb7700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec200cdeb0 con 0x7fec2006d980 2023-10-16T13:49:20.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.520+0000 7fec26cb7700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec2015f6c0 con 0x7fec20160e60 2023-10-16T13:49:20.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.520+0000 7fec26cb7700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec2006b190 con 0x7fec2006a7e0 2023-10-16T13:49:20.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.520+0000 7fec24a53700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fec20160e60 0x7fec2015e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:20.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.520+0000 7fec1ffff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fec2006d980 0x7fec2015efb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48932/0 (socket says 172.21.15.16:48932) 2023-10-16T13:49:20.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.520+0000 7fec1ffff700 1 -- 172.21.15.16:0/4108048029 learned_addr learned my addr 172.21.15.16:0/4108048029 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:49:20.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.521+0000 7fec25254700 1 --2- 172.21.15.16:0/4108048029 >> v2:172.21.15.49:3300/0 conn(0x7fec2006a7e0 0x7fec2006ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:20.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.521+0000 7fec24a53700 1 -- 172.21.15.16:0/4108048029 >> v2:172.21.15.49:3300/0 conn(0x7fec2006a7e0 msgr2=0x7fec2006ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:20.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.521+0000 7fec24a53700 1 --2- 172.21.15.16:0/4108048029 >> v2:172.21.15.49:3300/0 conn(0x7fec2006a7e0 0x7fec2006ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:20.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.521+0000 7fec24a53700 1 -- 172.21.15.16:0/4108048029 >> v1:172.21.15.16:6789/0 conn(0x7fec2006d980 legacy=0x7fec2015efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:20.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.521+0000 7fec1f7fe700 1 -- 172.21.15.16:0/4108048029 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3105435208 0 0) 0x7fec200cdeb0 con 0x7fec2006d980 2023-10-16T13:49:20.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.521+0000 7fec24a53700 1 -- 172.21.15.16:0/4108048029 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec201613e0 con 0x7fec20160e60 2023-10-16T13:49:20.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.521+0000 7fec25254700 1 --2- 172.21.15.16:0/4108048029 >> v2:172.21.15.49:3300/0 conn(0x7fec2006a7e0 0x7fec2006ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:49:20.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.521+0000 7fec24a53700 1 --2- 172.21.15.16:0/4108048029 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec20160e60 0x7fec2015e870 secure :-1 s=READY pgs=3255 cs=0 l=1 rev1=1 crypto rx=0x7fec200cf5a0 tx=0x7fec14009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c47906a0b994e5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:20.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.521+0000 7fec1f7fe700 1 -- 172.21.15.16:0/4108048029 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec14011830 con 0x7fec20160e60 2023-10-16T13:49:20.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.521+0000 7fec1f7fe700 1 -- 172.21.15.16:0/4108048029 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec14011990 con 0x7fec20160e60 2023-10-16T13:49:20.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.521+0000 7fec1f7fe700 1 -- 172.21.15.16:0/4108048029 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec14011d20 con 0x7fec20160e60 2023-10-16T13:49:20.531 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.522+0000 7fec26cb7700 1 -- 172.21.15.16:0/4108048029 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec20160e60 msgr2=0x7fec2015e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:20.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.522+0000 7fec26cb7700 1 --2- 172.21.15.16:0/4108048029 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec20160e60 0x7fec2015e870 secure :-1 s=READY pgs=3255 cs=0 l=1 rev1=1 crypto rx=0x7fec200cf5a0 tx=0x7fec14009b50 comp rx=0 tx=0).stop 2023-10-16T13:49:20.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.522+0000 7fec26cb7700 1 -- 172.21.15.16:0/4108048029 shutdown_connections 2023-10-16T13:49:20.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.522+0000 7fec26cb7700 1 --2- 172.21.15.16:0/4108048029 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec20160e60 0x7fec2015e870 unknown :-1 s=CLOSED pgs=3255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:20.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.522+0000 7fec26cb7700 1 --2- 172.21.15.16:0/4108048029 >> v2:172.21.15.49:3300/0 conn(0x7fec2006a7e0 0x7fec2006ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:20.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.522+0000 7fec26cb7700 1 -- 172.21.15.16:0/4108048029 >> 172.21.15.16:0/4108048029 conn(0x7fec2006f430 msgr2=0x7fec2006f810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:20.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.522+0000 7fec26cb7700 1 -- 172.21.15.16:0/4108048029 shutdown_connections 2023-10-16T13:49:20.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.522+0000 7fec26cb7700 1 -- 172.21.15.16:0/4108048029 wait complete. 2023-10-16T13:49:20.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.523+0000 7fec26cb7700 1 Processor -- start 2023-10-16T13:49:20.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.523+0000 7fec26cb7700 1 -- start start 2023-10-16T13:49:20.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.524+0000 7fec26cb7700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fec2006a7e0 0x7fec2014b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:20.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.524+0000 7fec26cb7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec2006d980 0x7fec200c3df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:20.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.524+0000 7fec26cb7700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fec20160e60 0x7fec200b53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:20.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.524+0000 7fec26cb7700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec200b31a0 con 0x7fec2006d980 2023-10-16T13:49:20.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.524+0000 7fec26cb7700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec2014bce0 con 0x7fec20160e60 2023-10-16T13:49:20.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.524+0000 7fec26cb7700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec200c4760 con 0x7fec2006a7e0 2023-10-16T13:49:20.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.524+0000 7fec1ffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec2006d980 0x7fec200c3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:20.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.524+0000 7fec1ffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec2006d980 0x7fec200c3df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59662/0 (socket says 172.21.15.16:59662) 2023-10-16T13:49:20.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.524+0000 7fec1ffff700 1 -- 172.21.15.16:0/1026394841 learned_addr learned my addr 172.21.15.16:0/1026394841 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:20.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.524+0000 7fec25254700 1 --2- 172.21.15.16:0/1026394841 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fec20160e60 0x7fec200b53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:20.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.524+0000 7fec24a53700 1 --2- 172.21.15.16:0/1026394841 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fec2006a7e0 0x7fec2014b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:20.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.524+0000 7fec1ffff700 1 -- 172.21.15.16:0/1026394841 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fec2006a7e0 msgr2=0x7fec2014b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:20.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.524+0000 7fec1ffff700 1 --2- 172.21.15.16:0/1026394841 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fec2006a7e0 0x7fec2014b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:20.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.524+0000 7fec1ffff700 1 -- 172.21.15.16:0/1026394841 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fec20160e60 msgr2=0x7fec200b53c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:20.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.524+0000 7fec1ffff700 1 --2- 172.21.15.16:0/1026394841 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fec20160e60 0x7fec200b53c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:20.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.524+0000 7fec1ffff700 1 -- 172.21.15.16:0/1026394841 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec140094a0 con 0x7fec2006d980 2023-10-16T13:49:20.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.524+0000 7fec1ffff700 1 --2- 172.21.15.16:0/1026394841 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec2006d980 0x7fec200c3df0 secure :-1 s=READY pgs=3256 cs=0 l=1 rev1=1 crypto rx=0x7fec0c00c520 tx=0x7fec0c00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:20.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.525+0000 7fec1d7fa700 1 -- 172.21.15.16:0/1026394841 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec0c004c80 con 0x7fec2006d980 2023-10-16T13:49:20.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.525+0000 7fec1d7fa700 1 -- 172.21.15.16:0/1026394841 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec0c004e00 con 0x7fec2006d980 2023-10-16T13:49:20.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.525+0000 7fec26cb7700 1 -- 172.21.15.16:0/1026394841 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec200c4a40 con 0x7fec2006d980 2023-10-16T13:49:20.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.525+0000 7fec1d7fa700 1 -- 172.21.15.16:0/1026394841 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec0c007310 con 0x7fec2006d980 2023-10-16T13:49:20.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.525+0000 7fec26cb7700 1 -- 172.21.15.16:0/1026394841 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec200c4330 con 0x7fec2006d980 2023-10-16T13:49:20.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.526+0000 7fec26cb7700 1 -- 172.21.15.16:0/1026394841 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec200c4f60 con 0x7fec2006d980 2023-10-16T13:49:20.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.526+0000 7fec1d7fa700 1 -- 172.21.15.16:0/1026394841 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 171) v1 ==== 89400+0+0 (secure 0 0 0) 0x7fec0c007470 con 0x7fec2006d980 2023-10-16T13:49:20.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.526+0000 7fec1d7fa700 1 --2- 172.21.15.16:0/1026394841 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7fec0406b100 0x7fec0406d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:20.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.527+0000 7fec1d7fa700 1 -- 172.21.15.16:0/1026394841 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(279..279 src has 1..279) v4 ==== 6501+0+0 (secure 0 0 0) 0x7fec0c0899d0 con 0x7fec2006d980 2023-10-16T13:49:20.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.527+0000 7fec24a53700 1 --2- 172.21.15.16:0/1026394841 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7fec0406b100 0x7fec0406d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:20.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.527+0000 7fec24a53700 1 --2- 172.21.15.16:0/1026394841 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7fec0406b100 0x7fec0406d5c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fec140097c0 tx=0x7fec1401c040 comp rx=0 tx=0).ready entity=mgr.10586 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:20.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.531+0000 7fec1d7fa700 1 -- 172.21.15.16:0/1026394841 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fec200c4f60 con 0x7fec2006d980 2023-10-16T13:49:20.770 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.768+0000 7fec26cb7700 1 -- 172.21.15.16:0/1026394841 --> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv1", "group_name": "mygrp", "target": ["mon-mgr", ""]}) v1 -- 0x7fec200c4550 con 0x7fec0406b100 2023-10-16T13:49:20.796 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.794+0000 7fec1d7fa700 1 -- 172.21.15.16:0/1026394841 <== mgr.10586 v2:172.21.15.16:6832/3920835901 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fec200c4550 con 0x7fec0406b100 2023-10-16T13:49:20.799 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.796+0000 7fec26cb7700 1 -- 172.21.15.16:0/1026394841 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7fec0406b100 msgr2=0x7fec0406d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:20.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.796+0000 7fec26cb7700 1 --2- 172.21.15.16:0/1026394841 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7fec0406b100 0x7fec0406d5c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fec140097c0 tx=0x7fec1401c040 comp rx=0 tx=0).stop 2023-10-16T13:49:20.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.796+0000 7fec26cb7700 1 -- 172.21.15.16:0/1026394841 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec2006d980 msgr2=0x7fec200c3df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:20.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.796+0000 7fec26cb7700 1 --2- 172.21.15.16:0/1026394841 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec2006d980 0x7fec200c3df0 secure :-1 s=READY pgs=3256 cs=0 l=1 rev1=1 crypto rx=0x7fec0c00c520 tx=0x7fec0c00c8e0 comp rx=0 tx=0).stop 2023-10-16T13:49:20.800 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.796+0000 7fec26cb7700 1 -- 172.21.15.16:0/1026394841 shutdown_connections 2023-10-16T13:49:20.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.796+0000 7fec26cb7700 1 --2- 172.21.15.16:0/1026394841 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7fec0406b100 0x7fec0406d5c0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:20.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.796+0000 7fec26cb7700 1 --2- 172.21.15.16:0/1026394841 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fec20160e60 0x7fec200b53c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:20.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.796+0000 7fec26cb7700 1 --2- 172.21.15.16:0/1026394841 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fec2006a7e0 0x7fec2014b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:20.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.796+0000 7fec26cb7700 1 --2- 172.21.15.16:0/1026394841 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec2006d980 0x7fec200c3df0 unknown :-1 s=CLOSED pgs=3256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:20.801 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.796+0000 7fec26cb7700 1 -- 172.21.15.16:0/1026394841 >> 172.21.15.16:0/1026394841 conn(0x7fec2006f430 msgr2=0x7fec200b15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:20.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.796+0000 7fec26cb7700 1 -- 172.21.15.16:0/1026394841 shutdown_connections 2023-10-16T13:49:20.802 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.796+0000 7fec26cb7700 1 -- 172.21.15.16:0/1026394841 wait complete. 2023-10-16T13:49:20.811 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --subvol sv1 . 1m 2023-10-16T13:49:20.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.939+0000 7fc9f2507700 1 Processor -- start 2023-10-16T13:49:20.942 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.939+0000 7fc9f2507700 1 -- start start 2023-10-16T13:49:20.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.940+0000 7fc9f2507700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fc9ec15e0b0 0x7fc9ec06d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:20.944 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.940+0000 7fc9f2507700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9ec0cb870 con 0x7fc9ec15a3d0 2023-10-16T13:49:20.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.940+0000 7fc9f2507700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9ec06e040 con 0x7fc9ec0cf600 2023-10-16T13:49:20.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.940+0000 7fc9f2507700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9ec15f0c0 con 0x7fc9ec15e0b0 2023-10-16T13:49:20.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.940+0000 7fc9ebfff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc9ec15a3d0 0x7fc9ec0ceef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48946/0 (socket says 172.21.15.16:48946) 2023-10-16T13:49:20.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.940+0000 7fc9ebfff700 1 -- 172.21.15.16:0/1271405715 learned_addr learned my addr 172.21.15.16:0/1271405715 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:49:20.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.940+0000 7fc9f0aa4700 1 --2- 172.21.15.16:0/1271405715 >> v2:172.21.15.49:3300/0 conn(0x7fc9ec15e0b0 0x7fc9ec06d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:20.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.941+0000 7fc9eb7fe700 1 -- 172.21.15.16:0/1271405715 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1444308258 0 0) 0x7fc9ec0cb870 con 0x7fc9ec15a3d0 2023-10-16T13:49:20.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.941+0000 7fc9eb7fe700 1 -- 172.21.15.16:0/1271405715 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9d80035b0 con 0x7fc9ec15a3d0 2023-10-16T13:49:20.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.941+0000 7fc9f0aa4700 1 -- 172.21.15.16:0/1271405715 >> v1:172.21.15.49:6789/0 conn(0x7fc9ec0cf600 legacy=0x7fc9ec0cbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:20.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.941+0000 7fc9f0aa4700 1 -- 172.21.15.16:0/1271405715 >> v1:172.21.15.16:6789/0 conn(0x7fc9ec15a3d0 legacy=0x7fc9ec0ceef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:20.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.941+0000 7fc9f0aa4700 1 -- 172.21.15.16:0/1271405715 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9ec06e320 con 0x7fc9ec15e0b0 2023-10-16T13:49:20.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.941+0000 7fc9f0aa4700 1 --2- 172.21.15.16:0/1271405715 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc9ec15e0b0 0x7fc9ec06d910 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7fc9e4009d90 tx=0x7fc9e400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6136ec1eb4b58847 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:20.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.941+0000 7fc9eb7fe700 1 -- 172.21.15.16:0/1271405715 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e400e070 con 0x7fc9ec15e0b0 2023-10-16T13:49:20.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.941+0000 7fc9eb7fe700 1 -- 172.21.15.16:0/1271405715 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9e400bb20 con 0x7fc9ec15e0b0 2023-10-16T13:49:20.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.941+0000 7fc9eb7fe700 1 -- 172.21.15.16:0/1271405715 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e4012430 con 0x7fc9ec15e0b0 2023-10-16T13:49:20.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.942+0000 7fc9f2507700 1 -- 172.21.15.16:0/1271405715 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc9ec15e0b0 msgr2=0x7fc9ec06d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:20.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.942+0000 7fc9f2507700 1 --2- 172.21.15.16:0/1271405715 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc9ec15e0b0 0x7fc9ec06d910 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7fc9e4009d90 tx=0x7fc9e400b330 comp rx=0 tx=0).stop 2023-10-16T13:49:20.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.942+0000 7fc9f2507700 1 -- 172.21.15.16:0/1271405715 shutdown_connections 2023-10-16T13:49:20.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.942+0000 7fc9f2507700 1 --2- 172.21.15.16:0/1271405715 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc9ec15e0b0 0x7fc9ec06d910 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:20.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.942+0000 7fc9f2507700 1 -- 172.21.15.16:0/1271405715 >> 172.21.15.16:0/1271405715 conn(0x7fc9ec06f430 msgr2=0x7fc9ec06f810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:20.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.942+0000 7fc9f2507700 1 -- 172.21.15.16:0/1271405715 shutdown_connections 2023-10-16T13:49:20.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.942+0000 7fc9f2507700 1 -- 172.21.15.16:0/1271405715 wait complete. 2023-10-16T13:49:20.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.943+0000 7fc9f2507700 1 Processor -- start 2023-10-16T13:49:20.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.943+0000 7fc9f2507700 1 -- start start 2023-10-16T13:49:20.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.944+0000 7fc9f2507700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9ec0cf600 0x7fc9ec1e5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:20.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.944+0000 7fc9f2507700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc9ec15a3d0 0x7fc9ec15bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:20.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.944+0000 7fc9f2507700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc9ec15e0b0 0x7fc9ec14d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:20.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.944+0000 7fc9f2507700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9ec1e6110 con 0x7fc9ec0cf600 2023-10-16T13:49:20.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.944+0000 7fc9f2507700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9ec1e4b30 con 0x7fc9ec15a3d0 2023-10-16T13:49:20.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.944+0000 7fc9f2507700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9ec1e4e10 con 0x7fc9ec15e0b0 2023-10-16T13:49:20.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.944+0000 7fc9ebfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9ec0cf600 0x7fc9ec1e5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:20.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.944+0000 7fc9ebfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9ec0cf600 0x7fc9ec1e5bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59676/0 (socket says 172.21.15.16:59676) 2023-10-16T13:49:20.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.944+0000 7fc9ebfff700 1 -- 172.21.15.16:0/2984035166 learned_addr learned my addr 172.21.15.16:0/2984035166 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:20.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.944+0000 7fc9e3fff700 1 --2- 172.21.15.16:0/2984035166 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc9ec15a3d0 0x7fc9ec15bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:20.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.944+0000 7fc9f0aa4700 1 --2- 172.21.15.16:0/2984035166 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc9ec15e0b0 0x7fc9ec14d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:20.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.944+0000 7fc9ebfff700 1 -- 172.21.15.16:0/2984035166 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc9ec15e0b0 msgr2=0x7fc9ec14d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:20.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.944+0000 7fc9ebfff700 1 --2- 172.21.15.16:0/2984035166 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc9ec15e0b0 0x7fc9ec14d180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:20.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.944+0000 7fc9ebfff700 1 -- 172.21.15.16:0/2984035166 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc9ec15a3d0 msgr2=0x7fc9ec15bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:20.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.944+0000 7fc9ebfff700 1 --2- 172.21.15.16:0/2984035166 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc9ec15a3d0 0x7fc9ec15bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:20.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.944+0000 7fc9ebfff700 1 -- 172.21.15.16:0/2984035166 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9e4009a70 con 0x7fc9ec0cf600 2023-10-16T13:49:20.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.944+0000 7fc9f0aa4700 1 --2- 172.21.15.16:0/2984035166 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc9ec15e0b0 0x7fc9ec14d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:49:20.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.944+0000 7fc9ebfff700 1 --2- 172.21.15.16:0/2984035166 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9ec0cf600 0x7fc9ec1e5bd0 secure :-1 s=READY pgs=3258 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc00c700 tx=0x7fc9dc00ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:20.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.945+0000 7fc9e97fa700 1 -- 172.21.15.16:0/2984035166 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9dc005d70 con 0x7fc9ec0cf600 2023-10-16T13:49:20.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.945+0000 7fc9e97fa700 1 -- 172.21.15.16:0/2984035166 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9dc0070d0 con 0x7fc9ec0cf600 2023-10-16T13:49:20.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.945+0000 7fc9f2507700 1 -- 172.21.15.16:0/2984035166 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9ec1e50f0 con 0x7fc9ec0cf600 2023-10-16T13:49:20.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.945+0000 7fc9e97fa700 1 -- 172.21.15.16:0/2984035166 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9dc007460 con 0x7fc9ec0cf600 2023-10-16T13:49:20.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.945+0000 7fc9f2507700 1 -- 172.21.15.16:0/2984035166 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9ec1e56b0 con 0x7fc9ec0cf600 2023-10-16T13:49:20.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.946+0000 7fc9f2507700 1 -- 172.21.15.16:0/2984035166 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9ec04ec20 con 0x7fc9ec0cf600 2023-10-16T13:49:20.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.946+0000 7fc9e97fa700 1 -- 172.21.15.16:0/2984035166 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 171) v1 ==== 89400+0+0 (secure 0 0 0) 0x7fc9dc0075c0 con 0x7fc9ec0cf600 2023-10-16T13:49:20.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.946+0000 7fc9e97fa700 1 --2- 172.21.15.16:0/2984035166 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7fc9cc06b100 0x7fc9cc06d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:20.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.947+0000 7fc9e97fa700 1 -- 172.21.15.16:0/2984035166 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(279..279 src has 1..279) v4 ==== 6501+0+0 (secure 0 0 0) 0x7fc9dc0891c0 con 0x7fc9ec0cf600 2023-10-16T13:49:20.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.947+0000 7fc9e3fff700 1 --2- 172.21.15.16:0/2984035166 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7fc9cc06b100 0x7fc9cc06d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:20.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.947+0000 7fc9e3fff700 1 --2- 172.21.15.16:0/2984035166 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7fc9cc06b100 0x7fc9cc06d5c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fc9ec14f6a0 tx=0x7fc9d4009040 comp rx=0 tx=0).ready entity=mgr.10586 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:20.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:20.951+0000 7fc9e97fa700 1 -- 172.21.15.16:0/2984035166 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc9dc053cd0 con 0x7fc9ec0cf600 2023-10-16T13:49:21.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.183+0000 7fc9f2507700 1 -- 172.21.15.16:0/2984035166 --> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "subvol": "sv1", "path": ".", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fc9ec06e8e0 con 0x7fc9cc06b100 2023-10-16T13:49:21.190 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:21.188+0000 7fb4088b1700 -1 mgr.server reply reply (22) Invalid argument No such subvol: None::sv1 2023-10-16T13:49:21.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.188+0000 7fc9e97fa700 1 -- 172.21.15.16:0/2984035166 <== mgr.10586 v2:172.21.15.16:6832/3920835901 1 ==== mgr_command_reply(tid 0: -22 No such subvol: None::sv1) v1 ==== 33+0+0 (secure 0 0 0) 0x7fc9d4001d60 con 0x7fc9cc06b100 2023-10-16T13:49:21.192 INFO:teuthology.orchestra.run.smithi016.stderr:Error EINVAL: No such subvol: None::sv1 2023-10-16T13:49:21.195 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-10-16T13:49:21.196 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --group mygrp . 1m 2023-10-16T13:49:21.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.324+0000 7f137352c700 1 Processor -- start 2023-10-16T13:49:21.327 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.325+0000 7f137352c700 1 -- start start 2023-10-16T13:49:21.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.325+0000 7f137352c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f136c06d980 0x7f136c15efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:21.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.325+0000 7f137352c700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f136c06a7e0 0x7f136c06ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:21.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.325+0000 7f137352c700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f136c0cdeb0 con 0x7f136c160e60 2023-10-16T13:49:21.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.325+0000 7f137352c700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f136c06b190 con 0x7f136c06d980 2023-10-16T13:49:21.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.325+0000 7f137352c700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f136c1613e0 con 0x7f136c06a7e0 2023-10-16T13:49:21.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.326+0000 7f1370ac7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f136c06d980 0x7f136c15efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:21.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.326+0000 7f13712c8700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f136c160e60 0x7f136c15e870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48958/0 (socket says 172.21.15.16:48958) 2023-10-16T13:49:21.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.326+0000 7f13712c8700 1 -- 172.21.15.16:0/1797455719 learned_addr learned my addr 172.21.15.16:0/1797455719 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:49:21.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.326+0000 7f1371ac9700 1 --2- 172.21.15.16:0/1797455719 >> v2:172.21.15.49:3300/0 conn(0x7f136c06a7e0 0x7f136c06ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:21.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.326+0000 7f1370ac7700 1 -- 172.21.15.16:0/1797455719 >> v2:172.21.15.49:3300/0 conn(0x7f136c06a7e0 msgr2=0x7f136c06ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:21.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.326+0000 7f1370ac7700 1 --2- 172.21.15.16:0/1797455719 >> v2:172.21.15.49:3300/0 conn(0x7f136c06a7e0 0x7f136c06ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:21.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.326+0000 7f1370ac7700 1 -- 172.21.15.16:0/1797455719 >> v1:172.21.15.16:6789/0 conn(0x7f136c160e60 legacy=0x7f136c15e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:21.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.326+0000 7f1370ac7700 1 -- 172.21.15.16:0/1797455719 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f136c1582d0 con 0x7f136c06d980 2023-10-16T13:49:21.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.326+0000 7f1371ac9700 1 --2- 172.21.15.16:0/1797455719 >> v2:172.21.15.49:3300/0 conn(0x7f136c06a7e0 0x7f136c06ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:49:21.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.326+0000 7f1370ac7700 1 --2- 172.21.15.16:0/1797455719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f136c06d980 0x7f136c15efb0 secure :-1 s=READY pgs=3260 cs=0 l=1 rev1=1 crypto rx=0x7f136c0cf5a0 tx=0x7f1368009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59bd6711dc7f63da server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:21.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.326+0000 7f135ffff700 1 -- 172.21.15.16:0/1797455719 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1368011830 con 0x7f136c06d980 2023-10-16T13:49:21.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.326+0000 7f135ffff700 1 -- 172.21.15.16:0/1797455719 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1368011990 con 0x7f136c06d980 2023-10-16T13:49:21.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.326+0000 7f135ffff700 1 -- 172.21.15.16:0/1797455719 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1368011d20 con 0x7f136c06d980 2023-10-16T13:49:21.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.327+0000 7f137352c700 1 -- 172.21.15.16:0/1797455719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f136c06d980 msgr2=0x7f136c15efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:21.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.327+0000 7f137352c700 1 --2- 172.21.15.16:0/1797455719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f136c06d980 0x7f136c15efb0 secure :-1 s=READY pgs=3260 cs=0 l=1 rev1=1 crypto rx=0x7f136c0cf5a0 tx=0x7f1368009b20 comp rx=0 tx=0).stop 2023-10-16T13:49:21.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.327+0000 7f137352c700 1 -- 172.21.15.16:0/1797455719 shutdown_connections 2023-10-16T13:49:21.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.327+0000 7f137352c700 1 --2- 172.21.15.16:0/1797455719 >> v2:172.21.15.49:3300/0 conn(0x7f136c06a7e0 0x7f136c06ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:21.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.327+0000 7f137352c700 1 --2- 172.21.15.16:0/1797455719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f136c06d980 0x7f136c15efb0 unknown :-1 s=CLOSED pgs=3260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:21.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.327+0000 7f137352c700 1 -- 172.21.15.16:0/1797455719 >> 172.21.15.16:0/1797455719 conn(0x7f136c06f430 msgr2=0x7f136c06f810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:21.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.327+0000 7f137352c700 1 -- 172.21.15.16:0/1797455719 shutdown_connections 2023-10-16T13:49:21.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.327+0000 7f137352c700 1 -- 172.21.15.16:0/1797455719 wait complete. 2023-10-16T13:49:21.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.328+0000 7f137352c700 1 Processor -- start 2023-10-16T13:49:21.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.328+0000 7f137352c700 1 -- start start 2023-10-16T13:49:21.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.329+0000 7f137352c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f136c06a7e0 0x7f136c06e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:21.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.329+0000 7f137352c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f136c06d980 0x7f136c0c4a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:21.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.329+0000 7f137352c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f136c160e60 0x7f136c0b5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:21.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.329+0000 7f137352c700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f136c14d4d0 con 0x7f136c06a7e0 2023-10-16T13:49:21.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.329+0000 7f137352c700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f136c0c4230 con 0x7f136c160e60 2023-10-16T13:49:21.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.329+0000 7f137352c700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f136c156330 con 0x7f136c06d980 2023-10-16T13:49:21.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.329+0000 7f13712c8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f136c06a7e0 0x7f136c06e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:21.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.329+0000 7f13712c8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f136c06a7e0 0x7f136c06e3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59690/0 (socket says 172.21.15.16:59690) 2023-10-16T13:49:21.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.329+0000 7f13712c8700 1 -- 172.21.15.16:0/279703058 learned_addr learned my addr 172.21.15.16:0/279703058 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:21.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.329+0000 7f1371ac9700 1 --2- 172.21.15.16:0/279703058 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f136c160e60 0x7f136c0b5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:21.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.329+0000 7f1370ac7700 1 --2- 172.21.15.16:0/279703058 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f136c06d980 0x7f136c0c4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:21.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.329+0000 7f13712c8700 1 -- 172.21.15.16:0/279703058 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f136c06d980 msgr2=0x7f136c0c4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:21.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.329+0000 7f13712c8700 1 --2- 172.21.15.16:0/279703058 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f136c06d980 0x7f136c0c4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:21.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.329+0000 7f13712c8700 1 -- 172.21.15.16:0/279703058 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f136c160e60 msgr2=0x7f136c0b5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:21.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.329+0000 7f13712c8700 1 --2- 172.21.15.16:0/279703058 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f136c160e60 0x7f136c0b5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:21.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.329+0000 7f13712c8700 1 -- 172.21.15.16:0/279703058 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13680094a0 con 0x7f136c06a7e0 2023-10-16T13:49:21.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.329+0000 7f1370ac7700 1 --2- 172.21.15.16:0/279703058 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f136c06d980 0x7f136c0c4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:49:21.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.329+0000 7f13712c8700 1 --2- 172.21.15.16:0/279703058 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f136c06a7e0 0x7f136c06e3a0 secure :-1 s=READY pgs=3261 cs=0 l=1 rev1=1 crypto rx=0x7f136000c480 tx=0x7f136000c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:21.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.330+0000 7f135dffb700 1 -- 172.21.15.16:0/279703058 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1360005d70 con 0x7f136c06a7e0 2023-10-16T13:49:21.342 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.330+0000 7f135dffb700 1 -- 172.21.15.16:0/279703058 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13600070d0 con 0x7f136c06a7e0 2023-10-16T13:49:21.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.330+0000 7f135dffb700 1 -- 172.21.15.16:0/279703058 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1360007440 con 0x7f136c06a7e0 2023-10-16T13:49:21.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.330+0000 7f137352c700 1 -- 172.21.15.16:0/279703058 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f136c156610 con 0x7f136c06a7e0 2023-10-16T13:49:21.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.330+0000 7f137352c700 1 -- 172.21.15.16:0/279703058 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f136c14d720 con 0x7f136c06a7e0 2023-10-16T13:49:21.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.331+0000 7f137352c700 1 -- 172.21.15.16:0/279703058 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f136c04ec20 con 0x7f136c06a7e0 2023-10-16T13:49:21.343 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.331+0000 7f135dffb700 1 -- 172.21.15.16:0/279703058 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 171) v1 ==== 89400+0+0 (secure 0 0 0) 0x7f13600075a0 con 0x7f136c06a7e0 2023-10-16T13:49:21.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.332+0000 7f135dffb700 1 --2- 172.21.15.16:0/279703058 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f135006b100 0x7f135006d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:21.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.332+0000 7f135dffb700 1 -- 172.21.15.16:0/279703058 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(279..279 src has 1..279) v4 ==== 6501+0+0 (secure 0 0 0) 0x7f1360088f10 con 0x7f136c06a7e0 2023-10-16T13:49:21.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.332+0000 7f1370ac7700 1 --2- 172.21.15.16:0/279703058 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f135006b100 0x7f135006d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:21.344 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.332+0000 7f1370ac7700 1 --2- 172.21.15.16:0/279703058 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f135006b100 0x7f135006d5c0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f136c0cf5a0 tx=0x7f1368008030 comp rx=0 tx=0).ready entity=mgr.10586 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:21.345 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.336+0000 7f135dffb700 1 -- 172.21.15.16:0/279703058 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1360053b90 con 0x7f136c06a7e0 2023-10-16T13:49:21.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.571+0000 7f137352c700 1 -- 172.21.15.16:0/279703058 --> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "group": "mygrp", "path": ".", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f136c0ce3e0 con 0x7f135006b100 2023-10-16T13:49:21.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.626+0000 7f135dffb700 1 -- 172.21.15.16:0/279703058 <== mgr.10586 v2:172.21.15.16:6832/3920835901 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f1368011ce0 con 0x7f135006b100 2023-10-16T13:49:21.629 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule set for path . 2023-10-16T13:49:21.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.628+0000 7f137352c700 1 -- 172.21.15.16:0/279703058 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f135006b100 msgr2=0x7f135006d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:21.631 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.628+0000 7f137352c700 1 --2- 172.21.15.16:0/279703058 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f135006b100 0x7f135006d5c0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f136c0cf5a0 tx=0x7f1368008030 comp rx=0 tx=0).stop 2023-10-16T13:49:21.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.628+0000 7f137352c700 1 -- 172.21.15.16:0/279703058 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f136c06a7e0 msgr2=0x7f136c06e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:21.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.628+0000 7f137352c700 1 --2- 172.21.15.16:0/279703058 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f136c06a7e0 0x7f136c06e3a0 secure :-1 s=READY pgs=3261 cs=0 l=1 rev1=1 crypto rx=0x7f136000c480 tx=0x7f136000c790 comp rx=0 tx=0).stop 2023-10-16T13:49:21.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.628+0000 7f137352c700 1 -- 172.21.15.16:0/279703058 shutdown_connections 2023-10-16T13:49:21.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.628+0000 7f137352c700 1 --2- 172.21.15.16:0/279703058 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f135006b100 0x7f135006d5c0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:21.632 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.628+0000 7f137352c700 1 --2- 172.21.15.16:0/279703058 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f136c160e60 0x7f136c0b5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:21.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.628+0000 7f137352c700 1 --2- 172.21.15.16:0/279703058 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f136c06a7e0 0x7f136c06e3a0 unknown :-1 s=CLOSED pgs=3261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:21.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.628+0000 7f137352c700 1 --2- 172.21.15.16:0/279703058 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f136c06d980 0x7f136c0c4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:21.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.628+0000 7f137352c700 1 -- 172.21.15.16:0/279703058 >> 172.21.15.16:0/279703058 conn(0x7f136c06f430 msgr2=0x7f136c0686c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:21.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.629+0000 7f137352c700 1 -- 172.21.15.16:0/279703058 shutdown_connections 2023-10-16T13:49:21.633 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.629+0000 7f137352c700 1 -- 172.21.15.16:0/279703058 wait complete. 2023-10-16T13:49:21.643 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path . 2023-10-16T13:49:21.644 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-16T13:49:21.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.770+0000 7f557acea700 1 Processor -- start 2023-10-16T13:49:21.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.770+0000 7f557acea700 1 -- start start 2023-10-16T13:49:21.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.771+0000 7f557acea700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f557415b070 0x7f557415b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:21.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.771+0000 7f557acea700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55740cef70 con 0x7f55740cb880 2023-10-16T13:49:21.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.771+0000 7f557acea700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55740ce360 con 0x7f557406e170 2023-10-16T13:49:21.777 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.771+0000 7f557acea700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55740ce640 con 0x7f557415b070 2023-10-16T13:49:21.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.771+0000 7f5573fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f55740cb880 0x7f557406da60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48972/0 (socket says 172.21.15.16:48972) 2023-10-16T13:49:21.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.771+0000 7f5573fff700 1 -- 172.21.15.16:0/1407644571 learned_addr learned my addr 172.21.15.16:0/1407644571 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:49:21.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.771+0000 7f5578a86700 1 --2- 172.21.15.16:0/1407644571 >> v2:172.21.15.49:3300/0 conn(0x7f557415b070 0x7f557415b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:21.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.771+0000 7f55737fe700 1 -- 172.21.15.16:0/1407644571 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2157068249 0 0) 0x7f55740cef70 con 0x7f55740cb880 2023-10-16T13:49:21.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.772+0000 7f55737fe700 1 -- 172.21.15.16:0/1407644571 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55640035b0 con 0x7f55740cb880 2023-10-16T13:49:21.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.772+0000 7f55737fe700 1 -- 172.21.15.16:0/1407644571 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2794816876 0 0) 0x7f55640035b0 con 0x7f55740cb880 2023-10-16T13:49:21.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.772+0000 7f55737fe700 1 -- 172.21.15.16:0/1407644571 >> v2:172.21.15.49:3300/0 conn(0x7f557415b070 msgr2=0x7f557415b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:21.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.772+0000 7f55737fe700 1 --2- 172.21.15.16:0/1407644571 >> v2:172.21.15.49:3300/0 conn(0x7f557415b070 0x7f557415b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:21.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.772+0000 7f55737fe700 1 -- 172.21.15.16:0/1407644571 >> v1:172.21.15.49:6789/0 conn(0x7f557406e170 legacy=0x7f55740cf400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:21.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.772+0000 7f55737fe700 1 -- 172.21.15.16:0/1407644571 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55740ce920 con 0x7f55740cb880 2023-10-16T13:49:21.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.772+0000 7f55737fe700 1 -- 172.21.15.16:0/1407644571 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f5560001670 con 0x7f55740cb880 2023-10-16T13:49:21.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.772+0000 7f55737fe700 1 -- 172.21.15.16:0/1407644571 >> v1:172.21.15.16:6789/0 conn(0x7f55740cb880 legacy=0x7f557406da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:21.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.772+0000 7f55737fe700 1 --2- 172.21.15.16:0/1407644571 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5564003a00 0x7f5564003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:21.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.772+0000 7f55737fe700 1 --2- 172.21.15.16:0/1407644571 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5564004850 0x7f5564004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:21.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.772+0000 7f55737fe700 1 --2- 172.21.15.16:0/1407644571 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5564005210 0x7f556400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:21.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.772+0000 7f55737fe700 1 -- 172.21.15.16:0/1407644571 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55740ce640 con 0x7f5564004850 2023-10-16T13:49:21.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.772+0000 7f55737fe700 1 -- 172.21.15.16:0/1407644571 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f556400c1b0 con 0x7f5564005210 2023-10-16T13:49:21.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.772+0000 7f55737fe700 1 -- 172.21.15.16:0/1407644571 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f556400c370 con 0x7f5564003a00 2023-10-16T13:49:21.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.772+0000 7f5573fff700 1 --2- 172.21.15.16:0/1407644571 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5564004850 0x7f5564004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:21.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.772+0000 7f5579287700 1 --2- 172.21.15.16:0/1407644571 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5564005210 0x7f556400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:21.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.772+0000 7f5578a86700 1 --2- 172.21.15.16:0/1407644571 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5564003a00 0x7f5564003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:21.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.772+0000 7f5573fff700 1 -- 172.21.15.16:0/1407644571 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5564003a00 msgr2=0x7f5564003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:21.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.772+0000 7f5573fff700 1 --2- 172.21.15.16:0/1407644571 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5564003a00 0x7f5564003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:21.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.772+0000 7f5573fff700 1 -- 172.21.15.16:0/1407644571 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5564005210 msgr2=0x7f556400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:21.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.772+0000 7f5573fff700 1 --2- 172.21.15.16:0/1407644571 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5564005210 0x7f556400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:21.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.772+0000 7f5579287700 1 -- 172.21.15.16:0/1407644571 reap_dead start 2023-10-16T13:49:21.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.772+0000 7f5573fff700 1 -- 172.21.15.16:0/1407644571 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55640035b0 con 0x7f5564004850 2023-10-16T13:49:21.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.772+0000 7f5578a86700 1 --2- 172.21.15.16:0/1407644571 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5564003a00 0x7f5564003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:49:21.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.772+0000 7f5573fff700 1 --2- 172.21.15.16:0/1407644571 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5564004850 0x7f5564004cd0 secure :-1 s=READY pgs=3263 cs=0 l=1 rev1=1 crypto rx=0x7f556000c240 tx=0x7f5564005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad788858d29fb3db server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:21.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.773+0000 7f55737fe700 1 -- 172.21.15.16:0/1407644571 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5560007c10 con 0x7f5564004850 2023-10-16T13:49:21.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.773+0000 7f55737fe700 1 -- 172.21.15.16:0/1407644571 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f556000ce00 con 0x7f5564004850 2023-10-16T13:49:21.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.773+0000 7f55737fe700 1 -- 172.21.15.16:0/1407644571 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5560005ba0 con 0x7f5564004850 2023-10-16T13:49:21.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.773+0000 7f557acea700 1 -- 172.21.15.16:0/1407644571 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5564004850 msgr2=0x7f5564004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:21.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.773+0000 7f557acea700 1 --2- 172.21.15.16:0/1407644571 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5564004850 0x7f5564004cd0 secure :-1 s=READY pgs=3263 cs=0 l=1 rev1=1 crypto rx=0x7f556000c240 tx=0x7f5564005880 comp rx=0 tx=0).stop 2023-10-16T13:49:21.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.773+0000 7f557acea700 1 -- 172.21.15.16:0/1407644571 shutdown_connections 2023-10-16T13:49:21.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.773+0000 7f557acea700 1 --2- 172.21.15.16:0/1407644571 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5564004850 0x7f5564004cd0 unknown :-1 s=CLOSED pgs=3263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:21.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.773+0000 7f557acea700 1 -- 172.21.15.16:0/1407644571 >> 172.21.15.16:0/1407644571 conn(0x7f55740b1f40 msgr2=0x7f55740b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:21.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.773+0000 7f557acea700 1 -- 172.21.15.16:0/1407644571 shutdown_connections 2023-10-16T13:49:21.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.773+0000 7f557acea700 1 -- 172.21.15.16:0/1407644571 wait complete. 2023-10-16T13:49:21.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.774+0000 7f557acea700 1 Processor -- start 2023-10-16T13:49:21.785 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.774+0000 7f557acea700 1 -- start start 2023-10-16T13:49:21.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.775+0000 7f557acea700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5564004850 0x7f55741522b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:21.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.775+0000 7f557acea700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f557415cc40 0x7f55740cc6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:21.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.775+0000 7f557acea700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f557415d0c0 0x7f55740ccbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:21.788 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.775+0000 7f557acea700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55740aebe0 con 0x7f557415d0c0 2023-10-16T13:49:21.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.775+0000 7f557acea700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55740adfd0 con 0x7f557415cc40 2023-10-16T13:49:21.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.775+0000 7f557acea700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55740ae2b0 con 0x7f5564004850 2023-10-16T13:49:21.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.775+0000 7f5579287700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f557415d0c0 0x7f55740ccbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:21.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.775+0000 7f5579287700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f557415d0c0 0x7f55740ccbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59714/0 (socket says 172.21.15.16:59714) 2023-10-16T13:49:21.789 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.775+0000 7f5579287700 1 -- 172.21.15.16:0/204727527 learned_addr learned my addr 172.21.15.16:0/204727527 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:21.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.776+0000 7f5578a86700 1 --2- 172.21.15.16:0/204727527 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5564004850 0x7f55741522b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:21.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.776+0000 7f5573fff700 1 --2- 172.21.15.16:0/204727527 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f557415cc40 0x7f55740cc6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:21.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.776+0000 7f5579287700 1 -- 172.21.15.16:0/204727527 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5564004850 msgr2=0x7f55741522b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:21.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.776+0000 7f5579287700 1 --2- 172.21.15.16:0/204727527 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5564004850 0x7f55741522b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:21.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.776+0000 7f5579287700 1 -- 172.21.15.16:0/204727527 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f557415cc40 msgr2=0x7f55740cc6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:21.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.776+0000 7f5579287700 1 --2- 172.21.15.16:0/204727527 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f557415cc40 0x7f55740cc6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:21.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.776+0000 7f5579287700 1 -- 172.21.15.16:0/204727527 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55740ae590 con 0x7f557415d0c0 2023-10-16T13:49:21.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.776+0000 7f5579287700 1 --2- 172.21.15.16:0/204727527 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f557415d0c0 0x7f55740ccbf0 secure :-1 s=READY pgs=3264 cs=0 l=1 rev1=1 crypto rx=0x7f557415e020 tx=0x7f556c00d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:21.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.776+0000 7f55717fa700 1 -- 172.21.15.16:0/204727527 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f556c003c10 con 0x7f557415d0c0 2023-10-16T13:49:21.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.776+0000 7f55717fa700 1 -- 172.21.15.16:0/204727527 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f556c003d70 con 0x7f557415d0c0 2023-10-16T13:49:21.791 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.776+0000 7f557acea700 1 -- 172.21.15.16:0/204727527 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55740ae870 con 0x7f557415d0c0 2023-10-16T13:49:21.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.776+0000 7f55717fa700 1 -- 172.21.15.16:0/204727527 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f556c00db70 con 0x7f557415d0c0 2023-10-16T13:49:21.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.776+0000 7f557acea700 1 -- 172.21.15.16:0/204727527 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55740af070 con 0x7f557415d0c0 2023-10-16T13:49:21.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.777+0000 7f557acea700 1 -- 172.21.15.16:0/204727527 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f557404ebb0 con 0x7f557415d0c0 2023-10-16T13:49:21.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.777+0000 7f55717fa700 1 -- 172.21.15.16:0/204727527 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 171) v1 ==== 89400+0+0 (secure 0 0 0) 0x7f556c00dcd0 con 0x7f557415d0c0 2023-10-16T13:49:21.792 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.778+0000 7f55717fa700 1 --2- 172.21.15.16:0/204727527 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f555806b130 0x7f555806d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:21.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.778+0000 7f55717fa700 1 -- 172.21.15.16:0/204727527 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(279..279 src has 1..279) v4 ==== 6501+0+0 (secure 0 0 0) 0x7f556c089f40 con 0x7f557415d0c0 2023-10-16T13:49:21.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.778+0000 7f5578a86700 1 --2- 172.21.15.16:0/204727527 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f555806b130 0x7f555806d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:21.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.778+0000 7f5578a86700 1 --2- 172.21.15.16:0/204727527 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f555806b130 0x7f555806d5f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f557404fec0 tx=0x7f5568008040 comp rx=0 tx=0).ready entity=mgr.10586 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:21.793 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:21.782+0000 7f55717fa700 1 -- 172.21.15.16:0/204727527 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f556c0549c0 con 0x7f557415d0c0 2023-10-16T13:49:22.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.046+0000 7f557acea700 1 -- 172.21.15.16:0/204727527 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f55740d11a0 con 0x7f557415d0c0 2023-10-16T13:49:22.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.218+0000 7f55717fa700 1 -- 172.21.15.16:0/204727527 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v172) v1 ==== 93+0+0 (secure 0 0 0) 0x7f556c057fe0 con 0x7f557415d0c0 2023-10-16T13:49:22.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.224+0000 7f557acea700 1 -- 172.21.15.16:0/204727527 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f555806b130 msgr2=0x7f555806d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:22.226 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.224+0000 7f557acea700 1 --2- 172.21.15.16:0/204727527 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f555806b130 0x7f555806d5f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f557404fec0 tx=0x7f5568008040 comp rx=0 tx=0).stop 2023-10-16T13:49:22.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.224+0000 7f557acea700 1 -- 172.21.15.16:0/204727527 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f557415d0c0 msgr2=0x7f55740ccbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:22.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.224+0000 7f557acea700 1 --2- 172.21.15.16:0/204727527 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f557415d0c0 0x7f55740ccbf0 secure :-1 s=READY pgs=3264 cs=0 l=1 rev1=1 crypto rx=0x7f557415e020 tx=0x7f556c00d430 comp rx=0 tx=0).stop 2023-10-16T13:49:22.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.224+0000 7f557acea700 1 -- 172.21.15.16:0/204727527 shutdown_connections 2023-10-16T13:49:22.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.224+0000 7f557acea700 1 --2- 172.21.15.16:0/204727527 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f555806b130 0x7f555806d5f0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:22.227 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.224+0000 7f557acea700 1 --2- 172.21.15.16:0/204727527 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f557415d0c0 0x7f55740ccbf0 unknown :-1 s=CLOSED pgs=3264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:22.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.224+0000 7f557acea700 1 --2- 172.21.15.16:0/204727527 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f5564004850 0x7f55741522b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:22.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.224+0000 7f557acea700 1 --2- 172.21.15.16:0/204727527 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f557415cc40 0x7f55740cc6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:22.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.224+0000 7f557acea700 1 -- 172.21.15.16:0/204727527 >> 172.21.15.16:0/204727527 conn(0x7f55740b1f40 msgr2=0x7f55740d0380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:22.228 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.224+0000 7f557acea700 1 -- 172.21.15.16:0/204727527 shutdown_connections 2023-10-16T13:49:22.229 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.224+0000 7f557acea700 1 -- 172.21.15.16:0/204727527 wait complete. 2023-10-16T13:49:22.241 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-16T13:49:22.241 DEBUG:teuthology.orchestra.run.smithi016:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T13:49:22.282 DEBUG:teuthology.orchestra.run.smithi049:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-16T13:49:22.328 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-16T13:49:22.328 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T13:49:22.329 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:49:22.347 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T13:49:22.347 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:49:22.374 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:22.372+0000 7f0548e60200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T13:49:22.376 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:49:22.373+0000 7f8a615ff200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T13:49:22.391 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:49:22.444 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:22.441+0000 7f0548e60200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T13:49:22.444 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:49:22.442+0000 7f8a615ff200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T13:49:22.520 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:49:22.517+0000 7f8a615ff200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T13:49:22.520 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:22.518+0000 7f0548e60200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T13:49:22.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.556+0000 7fc85d92d700 1 Processor -- start 2023-10-16T13:49:22.559 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.557+0000 7fc85d92d700 1 -- start start 2023-10-16T13:49:22.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.557+0000 7fc85d92d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc8580b2b50 0x7fc8580d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:22.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.557+0000 7fc85d92d700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8580d1210 con 0x7fc8580ce380 2023-10-16T13:49:22.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.557+0000 7fc85d92d700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8580cbc30 con 0x7fc8580cdfa0 2023-10-16T13:49:22.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.557+0000 7fc85d92d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8580cbf10 con 0x7fc8580b2b50 2023-10-16T13:49:22.562 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.558+0000 7fc8577fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc8580ce380 0x7fc8580d09e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49066/0 (socket says 172.21.15.16:49066) 2023-10-16T13:49:22.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.558+0000 7fc8577fe700 1 -- 172.21.15.16:0/228363475 learned_addr learned my addr 172.21.15.16:0/228363475 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:49:22.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.558+0000 7fc8567fc700 1 --2- 172.21.15.16:0/228363475 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc8580b2b50 0x7fc8580d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:22.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.558+0000 7fc855ffb700 1 -- 172.21.15.16:0/228363475 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 396958313 0 0) 0x7fc8580d1210 con 0x7fc8580ce380 2023-10-16T13:49:22.563 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.558+0000 7fc855ffb700 1 -- 172.21.15.16:0/228363475 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8400035b0 con 0x7fc8580ce380 2023-10-16T13:49:22.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.558+0000 7fc855ffb700 1 -- 172.21.15.16:0/228363475 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4211241384 0 0) 0x7fc8580cbc30 con 0x7fc8580cdfa0 2023-10-16T13:49:22.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.558+0000 7fc855ffb700 1 -- 172.21.15.16:0/228363475 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8580d1210 con 0x7fc8580cdfa0 2023-10-16T13:49:22.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.558+0000 7fc8567fc700 1 -- 172.21.15.16:0/228363475 >> v1:172.21.15.49:6789/0 conn(0x7fc8580cdfa0 legacy=0x7fc8580b2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:22.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.558+0000 7fc8567fc700 1 -- 172.21.15.16:0/228363475 >> v1:172.21.15.16:6789/0 conn(0x7fc8580ce380 legacy=0x7fc8580d09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:22.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.558+0000 7fc8567fc700 1 -- 172.21.15.16:0/228363475 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8580cc1f0 con 0x7fc8580b2b50 2023-10-16T13:49:22.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.558+0000 7fc8567fc700 1 --2- 172.21.15.16:0/228363475 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc8580b2b50 0x7fc8580d04a0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fc84c009870 tx=0x7fc84c009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb8977846de91d8a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:22.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.558+0000 7fc855ffb700 1 -- 172.21.15.16:0/228363475 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc84c011830 con 0x7fc8580b2b50 2023-10-16T13:49:22.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.558+0000 7fc855ffb700 1 -- 172.21.15.16:0/228363475 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc84c011990 con 0x7fc8580b2b50 2023-10-16T13:49:22.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.559+0000 7fc855ffb700 1 -- 172.21.15.16:0/228363475 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc84c011d20 con 0x7fc8580b2b50 2023-10-16T13:49:22.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.559+0000 7fc85d92d700 1 -- 172.21.15.16:0/228363475 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc8580b2b50 msgr2=0x7fc8580d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:22.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.559+0000 7fc85d92d700 1 --2- 172.21.15.16:0/228363475 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc8580b2b50 0x7fc8580d04a0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fc84c009870 tx=0x7fc84c009b80 comp rx=0 tx=0).stop 2023-10-16T13:49:22.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.559+0000 7fc85d92d700 1 -- 172.21.15.16:0/228363475 shutdown_connections 2023-10-16T13:49:22.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.559+0000 7fc85d92d700 1 --2- 172.21.15.16:0/228363475 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc8580b2b50 0x7fc8580d04a0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:22.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.559+0000 7fc85d92d700 1 -- 172.21.15.16:0/228363475 >> 172.21.15.16:0/228363475 conn(0x7fc8580d14b0 msgr2=0x7fc8580d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:22.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.559+0000 7fc85d92d700 1 -- 172.21.15.16:0/228363475 shutdown_connections 2023-10-16T13:49:22.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.559+0000 7fc85d92d700 1 -- 172.21.15.16:0/228363475 wait complete. 2023-10-16T13:49:22.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.560+0000 7fc85d92d700 1 Processor -- start 2023-10-16T13:49:22.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.561+0000 7fc85d92d700 1 -- start start 2023-10-16T13:49:22.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.561+0000 7fc85d92d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8580b2b50 0x7fc8580caec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:22.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.561+0000 7fc85d92d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc8580cdfa0 0x7fc85809ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:22.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.561+0000 7fc85d92d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc8580ce380 0x7fc8580c93d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:22.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.561+0000 7fc85d92d700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8580caa30 con 0x7fc8580b2b50 2023-10-16T13:49:22.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.561+0000 7fc85d92d700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8580cb400 con 0x7fc8580cdfa0 2023-10-16T13:49:22.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.561+0000 7fc85d92d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc85809ef90 con 0x7fc8580ce380 2023-10-16T13:49:22.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.561+0000 7fc856ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8580b2b50 0x7fc8580caec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:22.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.561+0000 7fc856ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8580b2b50 0x7fc8580caec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59794/0 (socket says 172.21.15.16:59794) 2023-10-16T13:49:22.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.561+0000 7fc8567fc700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc8580cdfa0 0x7fc85809ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:22.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.561+0000 7fc8577fe700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc8580ce380 0x7fc8580c93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:22.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.561+0000 7fc856ffd700 1 -- 172.21.15.16:0/1178752051 learned_addr learned my addr 172.21.15.16:0/1178752051 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:22.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.562+0000 7fc856ffd700 1 -- 172.21.15.16:0/1178752051 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc8580ce380 msgr2=0x7fc8580c93d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:22.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.562+0000 7fc856ffd700 1 --2- 172.21.15.16:0/1178752051 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc8580ce380 0x7fc8580c93d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:22.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.562+0000 7fc856ffd700 1 -- 172.21.15.16:0/1178752051 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc8580cdfa0 msgr2=0x7fc85809ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:22.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.562+0000 7fc856ffd700 1 --2- 172.21.15.16:0/1178752051 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc8580cdfa0 0x7fc85809ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:22.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.562+0000 7fc856ffd700 1 -- 172.21.15.16:0/1178752051 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc84c0094a0 con 0x7fc8580b2b50 2023-10-16T13:49:22.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.562+0000 7fc8567fc700 1 --2- 172.21.15.16:0/1178752051 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc8580cdfa0 0x7fc85809ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:49:22.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.562+0000 7fc856ffd700 1 --2- 172.21.15.16:0/1178752051 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8580b2b50 0x7fc8580caec0 secure :-1 s=READY pgs=3268 cs=0 l=1 rev1=1 crypto rx=0x7fc84800c550 tx=0x7fc84800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:22.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.562+0000 7fc83ffff700 1 -- 172.21.15.16:0/1178752051 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8480052d0 con 0x7fc8580b2b50 2023-10-16T13:49:22.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.562+0000 7fc83ffff700 1 -- 172.21.15.16:0/1178752051 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc848005430 con 0x7fc8580b2b50 2023-10-16T13:49:22.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.562+0000 7fc85d92d700 1 -- 172.21.15.16:0/1178752051 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8580c9e20 con 0x7fc8580b2b50 2023-10-16T13:49:22.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.562+0000 7fc83ffff700 1 -- 172.21.15.16:0/1178752051 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8480038c0 con 0x7fc8580b2b50 2023-10-16T13:49:22.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.562+0000 7fc85d92d700 1 -- 172.21.15.16:0/1178752051 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8580ca3e0 con 0x7fc8580b2b50 2023-10-16T13:49:22.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.563+0000 7fc85d92d700 1 -- 172.21.15.16:0/1178752051 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8580cad10 con 0x7fc8580b2b50 2023-10-16T13:49:22.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.564+0000 7fc83ffff700 1 -- 172.21.15.16:0/1178752051 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 172) v1 ==== 89490+0+0 (secure 0 0 0) 0x7fc848003a20 con 0x7fc8580b2b50 2023-10-16T13:49:22.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.564+0000 7fc83ffff700 1 --2- 172.21.15.16:0/1178752051 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7fc84406b1d0 0x7fc84406d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:22.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.564+0000 7fc8567fc700 1 -- 172.21.15.16:0/1178752051 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7fc84406b1d0 msgr2=0x7fc84406d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3920835901 2023-10-16T13:49:22.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.564+0000 7fc83ffff700 1 -- 172.21.15.16:0/1178752051 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(279..279 src has 1..279) v4 ==== 6501+0+0 (secure 0 0 0) 0x7fc848089180 con 0x7fc8580b2b50 2023-10-16T13:49:22.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.564+0000 7fc8567fc700 1 --2- 172.21.15.16:0/1178752051 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7fc84406b1d0 0x7fc84406d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:49:22.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.569+0000 7fc83ffff700 1 -- 172.21.15.16:0/1178752051 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc848053bc0 con 0x7fc8580b2b50 2023-10-16T13:49:22.601 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:49:22.599+0000 7f8a615ff200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T13:49:22.606 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:22.603+0000 7f0548e60200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T13:49:22.749 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:49:22.746+0000 7f8a615ff200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T13:49:22.755 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:22.752+0000 7f0548e60200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T13:49:22.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.765+0000 7fc8567fc700 1 -- 172.21.15.16:0/1178752051 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7fc84406b1d0 msgr2=0x7fc84406d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3920835901 2023-10-16T13:49:22.767 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.765+0000 7fc8567fc700 1 --2- 172.21.15.16:0/1178752051 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7fc84406b1d0 0x7fc84406d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:49:22.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.774+0000 7fc85d92d700 1 -- 172.21.15.16:0/1178752051 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc85804ebb0 con 0x7fc8580b2b50 2023-10-16T13:49:22.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.775+0000 7fc83ffff700 1 -- 172.21.15.16:0/1178752051 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v279) v1 ==== 74+0+17324 (secure 0 0 0) 0x7fc8480571e0 con 0x7fc8580b2b50 2023-10-16T13:49:22.778 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:49:22.778 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":279,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:49:17.194563+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":38,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":37,"pool_name":"cephfs_metadata","create_time":"2023-10-16T13:49:00.609612+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"277","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":38,"pool_name":"cephfs_data","create_time":"2023-10-16T13:49:01.068705+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"278","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/750285561":"2023-10-17T13:49:17.194509+0000","172.21.15.16:6833/1257392887":"2023-10-17T13:49:17.194509+0000","172.21.15.16:6832/1257392887":"2023-10-17T13:49:17.194509+0000","172.21.15.16:0/1101859511":"2023-10-17T13:49:17.194509+0000","172.21.15.16:0/197265557":"2023-10-17T13:49:17.194509+0000","172.21.15.16:0/2024751970":"2023-10-17T13:49:17.194509+0000","172.21.15.16:0/2401946155":"2023-10-17T13:49:17.194509+0000","172.21.15.16:0/1156439592":"2023-10-17T13:49:17.194509+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:49:22.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.777+0000 7fc85d92d700 1 -- 172.21.15.16:0/1178752051 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7fc84406b1d0 msgr2=0x7fc84406d690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:49:22.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.777+0000 7fc85d92d700 1 --2- 172.21.15.16:0/1178752051 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7fc84406b1d0 0x7fc84406d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:22.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.777+0000 7fc85d92d700 1 -- 172.21.15.16:0/1178752051 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8580b2b50 msgr2=0x7fc8580caec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:22.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.777+0000 7fc85d92d700 1 --2- 172.21.15.16:0/1178752051 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8580b2b50 0x7fc8580caec0 secure :-1 s=READY pgs=3268 cs=0 l=1 rev1=1 crypto rx=0x7fc84800c550 tx=0x7fc84800c860 comp rx=0 tx=0).stop 2023-10-16T13:49:22.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.778+0000 7fc85d92d700 1 -- 172.21.15.16:0/1178752051 shutdown_connections 2023-10-16T13:49:22.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.778+0000 7fc85d92d700 1 --2- 172.21.15.16:0/1178752051 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7fc84406b1d0 0x7fc84406d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:22.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.778+0000 7fc85d92d700 1 --2- 172.21.15.16:0/1178752051 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc8580ce380 0x7fc8580c93d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:22.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.778+0000 7fc85d92d700 1 --2- 172.21.15.16:0/1178752051 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc8580b2b50 0x7fc8580caec0 unknown :-1 s=CLOSED pgs=3268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:22.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.778+0000 7fc85d92d700 1 --2- 172.21.15.16:0/1178752051 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc8580cdfa0 0x7fc85809ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:22.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.778+0000 7fc85d92d700 1 -- 172.21.15.16:0/1178752051 >> 172.21.15.16:0/1178752051 conn(0x7fc8580d14b0 msgr2=0x7fc858070260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:22.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.778+0000 7fc85d92d700 1 -- 172.21.15.16:0/1178752051 shutdown_connections 2023-10-16T13:49:22.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.778+0000 7fc85d92d700 1 -- 172.21.15.16:0/1178752051 wait complete. 2023-10-16T13:49:22.793 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 37, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T13:49:00.609612+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '277', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 38, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T13:49:01.068705+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '278', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T13:49:22.794 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T13:49:22.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.960+0000 7fced4c93700 1 Processor -- start 2023-10-16T13:49:22.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.960+0000 7fced4c93700 1 -- start start 2023-10-16T13:49:22.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.961+0000 7fced4c93700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fced015ec80 0x7fced00cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:22.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.961+0000 7fced4c93700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fced015f060 0x7fced00ce400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:22.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.961+0000 7fced4c93700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fced00b2440 con 0x7fced00d2110 2023-10-16T13:49:22.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.961+0000 7fced4c93700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fced00b2c50 con 0x7fced015f060 2023-10-16T13:49:22.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.961+0000 7fced4c93700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fced00d00e0 con 0x7fced015ec80 2023-10-16T13:49:22.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.961+0000 7fcecdd9b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fced015f060 0x7fced00ce400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:22.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.962+0000 7fcecdd9b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fced015f060 0x7fced00ce400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59848/0 (socket says 172.21.15.16:59848) 2023-10-16T13:49:22.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.962+0000 7fcecdd9b700 1 -- 172.21.15.16:0/779692449 learned_addr learned my addr 172.21.15.16:0/779692449 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:22.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.962+0000 7fcece59c700 1 --2- 172.21.15.16:0/779692449 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fced015ec80 0x7fced00cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:22.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.962+0000 7fcecdd9b700 1 -- 172.21.15.16:0/779692449 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fced015ec80 msgr2=0x7fced00cdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:22.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.962+0000 7fcecdd9b700 1 --2- 172.21.15.16:0/779692449 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fced015ec80 0x7fced00cdec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:22.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.962+0000 7fcecdd9b700 1 -- 172.21.15.16:0/779692449 >> v1:172.21.15.49:6789/0 conn(0x7fced00d2110 legacy=0x7fced00b4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:22.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.962+0000 7fcecdd9b700 1 -- 172.21.15.16:0/779692449 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fced00cef60 con 0x7fced015f060 2023-10-16T13:49:22.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.962+0000 7fcecdd9b700 1 --2- 172.21.15.16:0/779692449 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fced015f060 0x7fced00ce400 secure :-1 s=READY pgs=3269 cs=0 l=1 rev1=1 crypto rx=0x7fcec00098f0 tx=0x7fcec0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a41c2961d9539035 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:22.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.962+0000 7fcecd59a700 1 -- 172.21.15.16:0/779692449 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcec0011830 con 0x7fced015f060 2023-10-16T13:49:22.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.962+0000 7fcecd59a700 1 -- 172.21.15.16:0/779692449 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcec0011990 con 0x7fced015f060 2023-10-16T13:49:22.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.962+0000 7fcecd59a700 1 -- 172.21.15.16:0/779692449 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcec0011d20 con 0x7fced015f060 2023-10-16T13:49:22.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.963+0000 7fced4c93700 1 -- 172.21.15.16:0/779692449 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fced015f060 msgr2=0x7fced00ce400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:22.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.963+0000 7fced4c93700 1 --2- 172.21.15.16:0/779692449 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fced015f060 0x7fced00ce400 secure :-1 s=READY pgs=3269 cs=0 l=1 rev1=1 crypto rx=0x7fcec00098f0 tx=0x7fcec0009c00 comp rx=0 tx=0).stop 2023-10-16T13:49:22.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.963+0000 7fced4c93700 1 -- 172.21.15.16:0/779692449 shutdown_connections 2023-10-16T13:49:22.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.963+0000 7fced4c93700 1 --2- 172.21.15.16:0/779692449 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fced015ec80 0x7fced00cdec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:22.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.963+0000 7fced4c93700 1 --2- 172.21.15.16:0/779692449 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fced015f060 0x7fced00ce400 unknown :-1 s=CLOSED pgs=3269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:22.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.963+0000 7fced4c93700 1 -- 172.21.15.16:0/779692449 >> 172.21.15.16:0/779692449 conn(0x7fced00d15f0 msgr2=0x7fced00d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:22.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.963+0000 7fced4c93700 1 -- 172.21.15.16:0/779692449 shutdown_connections 2023-10-16T13:49:22.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.963+0000 7fced4c93700 1 -- 172.21.15.16:0/779692449 wait complete. 2023-10-16T13:49:22.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.964+0000 7fced4c93700 1 Processor -- start 2023-10-16T13:49:22.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.964+0000 7fced4c93700 1 -- start start 2023-10-16T13:49:22.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.965+0000 7fced4c93700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fced00d2110 0x7fced0151960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:22.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.965+0000 7fced4c93700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fced015ec80 0x7fced01f44a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:22.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.965+0000 7fced4c93700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fced015f060 0x7fced00b4170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:22.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.965+0000 7fced4c93700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fced01e41a0 con 0x7fced00d2110 2023-10-16T13:49:22.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.965+0000 7fced4c93700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fced01f4cf0 con 0x7fced015f060 2023-10-16T13:49:22.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.965+0000 7fced4c93700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fced01f4fd0 con 0x7fced015ec80 2023-10-16T13:49:22.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.965+0000 7fcece59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fced00d2110 0x7fced0151960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:22.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.965+0000 7fcece59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fced00d2110 0x7fced0151960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59852/0 (socket says 172.21.15.16:59852) 2023-10-16T13:49:22.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.965+0000 7fcece59c700 1 -- 172.21.15.16:0/776643705 learned_addr learned my addr 172.21.15.16:0/776643705 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:22.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.965+0000 7fceced9d700 1 --2- 172.21.15.16:0/776643705 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fced015f060 0x7fced00b4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:22.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.965+0000 7fcecdd9b700 1 --2- 172.21.15.16:0/776643705 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fced015ec80 0x7fced01f44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:22.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.966+0000 7fcece59c700 1 -- 172.21.15.16:0/776643705 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fced015ec80 msgr2=0x7fced01f44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:22.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.966+0000 7fcece59c700 1 --2- 172.21.15.16:0/776643705 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fced015ec80 0x7fced01f44a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:22.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.966+0000 7fcece59c700 1 -- 172.21.15.16:0/776643705 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fced015f060 msgr2=0x7fced00b4170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:22.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.966+0000 7fcece59c700 1 --2- 172.21.15.16:0/776643705 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fced015f060 0x7fced00b4170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:22.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.966+0000 7fcece59c700 1 -- 172.21.15.16:0/776643705 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcec0009550 con 0x7fced00d2110 2023-10-16T13:49:22.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.966+0000 7fcece59c700 1 --2- 172.21.15.16:0/776643705 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fced00d2110 0x7fced0151960 secure :-1 s=READY pgs=3270 cs=0 l=1 rev1=1 crypto rx=0x7fcec400e9b0 tx=0x7fcec400ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:22.978 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.966+0000 7fcebeffd700 1 -- 172.21.15.16:0/776643705 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcec4009710 con 0x7fced00d2110 2023-10-16T13:49:22.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.966+0000 7fcebeffd700 1 -- 172.21.15.16:0/776643705 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcec400ee50 con 0x7fced00d2110 2023-10-16T13:49:22.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.966+0000 7fcebeffd700 1 -- 172.21.15.16:0/776643705 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcec400ca50 con 0x7fced00d2110 2023-10-16T13:49:22.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.966+0000 7fced4c93700 1 -- 172.21.15.16:0/776643705 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fced01f52b0 con 0x7fced00d2110 2023-10-16T13:49:22.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.966+0000 7fced4c93700 1 -- 172.21.15.16:0/776643705 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fced01f49e0 con 0x7fced00d2110 2023-10-16T13:49:22.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.967+0000 7fced4c93700 1 -- 172.21.15.16:0/776643705 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fced01e47b0 con 0x7fced00d2110 2023-10-16T13:49:22.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.968+0000 7fcebeffd700 1 -- 172.21.15.16:0/776643705 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 172) v1 ==== 89490+0+0 (secure 0 0 0) 0x7fcec400cbb0 con 0x7fced00d2110 2023-10-16T13:49:22.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.968+0000 7fcebeffd700 1 --2- 172.21.15.16:0/776643705 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7fceb006b1d0 0x7fceb006d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:22.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.968+0000 7fcecdd9b700 1 -- 172.21.15.16:0/776643705 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7fceb006b1d0 msgr2=0x7fceb006d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3920835901 2023-10-16T13:49:22.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.968+0000 7fcebeffd700 1 -- 172.21.15.16:0/776643705 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(279..279 src has 1..279) v4 ==== 6501+0+0 (secure 0 0 0) 0x7fcec4089dc0 con 0x7fced00d2110 2023-10-16T13:49:22.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.968+0000 7fcecdd9b700 1 --2- 172.21.15.16:0/776643705 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7fceb006b1d0 0x7fceb006d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:49:22.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:22.972+0000 7fcebeffd700 1 -- 172.21.15.16:0/776643705 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcec4054800 con 0x7fced00d2110 2023-10-16T13:49:23.092 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:49:23.090+0000 7f8a615ff200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T13:49:23.106 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:23.104+0000 7f0548e60200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T13:49:23.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.169+0000 7fcecdd9b700 1 -- 172.21.15.16:0/776643705 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7fceb006b1d0 msgr2=0x7fceb006d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3920835901 2023-10-16T13:49:23.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.169+0000 7fcecdd9b700 1 --2- 172.21.15.16:0/776643705 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7fceb006b1d0 0x7fceb006d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:49:23.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.177+0000 7fced4c93700 1 -- 172.21.15.16:0/776643705 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fced004ebb0 con 0x7fced00d2110 2023-10-16T13:49:23.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.177+0000 7fcebeffd700 1 -- 172.21.15.16:0/776643705 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v279) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcec40210a0 con 0x7fced00d2110 2023-10-16T13:49:23.180 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T13:49:23.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.180+0000 7fced4c93700 1 -- 172.21.15.16:0/776643705 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7fceb006b1d0 msgr2=0x7fceb006d690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:49:23.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.180+0000 7fced4c93700 1 --2- 172.21.15.16:0/776643705 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7fceb006b1d0 0x7fceb006d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:23.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.180+0000 7fced4c93700 1 -- 172.21.15.16:0/776643705 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fced00d2110 msgr2=0x7fced0151960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:23.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.180+0000 7fced4c93700 1 --2- 172.21.15.16:0/776643705 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fced00d2110 0x7fced0151960 secure :-1 s=READY pgs=3270 cs=0 l=1 rev1=1 crypto rx=0x7fcec400e9b0 tx=0x7fcec400ecc0 comp rx=0 tx=0).stop 2023-10-16T13:49:23.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.180+0000 7fced4c93700 1 -- 172.21.15.16:0/776643705 shutdown_connections 2023-10-16T13:49:23.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.180+0000 7fced4c93700 1 --2- 172.21.15.16:0/776643705 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7fceb006b1d0 0x7fceb006d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:23.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.180+0000 7fced4c93700 1 --2- 172.21.15.16:0/776643705 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fced015f060 0x7fced00b4170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:23.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.180+0000 7fced4c93700 1 --2- 172.21.15.16:0/776643705 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fced00d2110 0x7fced0151960 unknown :-1 s=CLOSED pgs=3270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:23.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.180+0000 7fced4c93700 1 --2- 172.21.15.16:0/776643705 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fced015ec80 0x7fced01f44a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:23.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.180+0000 7fced4c93700 1 -- 172.21.15.16:0/776643705 >> 172.21.15.16:0/776643705 conn(0x7fced00d15f0 msgr2=0x7fced0153af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:23.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.180+0000 7fced4c93700 1 -- 172.21.15.16:0/776643705 shutdown_connections 2023-10-16T13:49:23.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.180+0000 7fced4c93700 1 -- 172.21.15.16:0/776643705 wait complete. 2023-10-16T13:49:23.196 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T13:49:23.212 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:49:23.209+0000 7f8a615ff200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T13:49:23.230 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:23.228+0000 7f0548e60200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T13:49:23.290 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:49:23.287+0000 7f8a615ff200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T13:49:23.308 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:23.306+0000 7f0548e60200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T13:49:23.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.323+0000 7f34c9240700 1 Processor -- start 2023-10-16T13:49:23.326 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.324+0000 7f34c9240700 1 -- start start 2023-10-16T13:49:23.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.324+0000 7f34c9240700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f34c415ec80 0x7f34c40cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:23.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.324+0000 7f34c9240700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34c40b2440 con 0x7f34c40d2110 2023-10-16T13:49:23.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.324+0000 7f34c9240700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34c40d00e0 con 0x7f34c415f060 2023-10-16T13:49:23.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.324+0000 7f34c9240700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34c40b2c50 con 0x7f34c415ec80 2023-10-16T13:49:23.328 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.325+0000 7f34c2d9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f34c415ec80 0x7f34c40cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:23.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.325+0000 7f34c2d9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f34c415ec80 0x7f34c40cdec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59862/0 (socket says 172.21.15.16:59862) 2023-10-16T13:49:23.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.325+0000 7f34c2d9d700 1 -- 172.21.15.16:0/1130365310 learned_addr learned my addr 172.21.15.16:0/1130365310 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:23.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.325+0000 7f34c359e700 1 --1- 172.21.15.16:0/1130365310 >> v1:172.21.15.16:6789/0 conn(0x7f34c40d2110 0x7f34c40b4b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49140/0 (socket says 172.21.15.16:49140) 2023-10-16T13:49:23.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.325+0000 7f34c2d9d700 1 -- 172.21.15.16:0/1130365310 >> v1:172.21.15.49:6789/0 conn(0x7f34c415f060 legacy=0x7f34c40b4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:23.329 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.325+0000 7f34c2d9d700 1 -- 172.21.15.16:0/1130365310 >> v1:172.21.15.16:6789/0 conn(0x7f34c40d2110 legacy=0x7f34c40b4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:23.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.325+0000 7f34c1d9b700 1 -- 172.21.15.16:0/1130365310 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3490979310 0 0) 0x7f34c40b2440 con 0x7f34c40d2110 2023-10-16T13:49:23.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.325+0000 7f34c2d9d700 1 -- 172.21.15.16:0/1130365310 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34c40cef60 con 0x7f34c415ec80 2023-10-16T13:49:23.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.325+0000 7f34c2d9d700 1 --2- 172.21.15.16:0/1130365310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34c415ec80 0x7f34c40cdec0 secure :-1 s=READY pgs=3272 cs=0 l=1 rev1=1 crypto rx=0x7f34b40098f0 tx=0x7f34b4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=417f88d6b1e49b49 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:23.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.325+0000 7f34c1d9b700 1 -- 172.21.15.16:0/1130365310 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34b400b8a0 con 0x7f34c415ec80 2023-10-16T13:49:23.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.325+0000 7f34c1d9b700 1 -- 172.21.15.16:0/1130365310 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34b400ba00 con 0x7f34c415ec80 2023-10-16T13:49:23.331 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.325+0000 7f34c1d9b700 1 -- 172.21.15.16:0/1130365310 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34b400bd90 con 0x7f34c415ec80 2023-10-16T13:49:23.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.326+0000 7f34c9240700 1 -- 172.21.15.16:0/1130365310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34c415ec80 msgr2=0x7f34c40cdec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:23.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.326+0000 7f34c9240700 1 --2- 172.21.15.16:0/1130365310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34c415ec80 0x7f34c40cdec0 secure :-1 s=READY pgs=3272 cs=0 l=1 rev1=1 crypto rx=0x7f34b40098f0 tx=0x7f34b4009c00 comp rx=0 tx=0).stop 2023-10-16T13:49:23.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.326+0000 7f34c9240700 1 -- 172.21.15.16:0/1130365310 shutdown_connections 2023-10-16T13:49:23.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.326+0000 7f34c9240700 1 --2- 172.21.15.16:0/1130365310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34c415ec80 0x7f34c40cdec0 unknown :-1 s=CLOSED pgs=3272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:23.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.326+0000 7f34c9240700 1 -- 172.21.15.16:0/1130365310 >> 172.21.15.16:0/1130365310 conn(0x7f34c40d15f0 msgr2=0x7f34c40d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:23.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.326+0000 7f34c9240700 1 -- 172.21.15.16:0/1130365310 shutdown_connections 2023-10-16T13:49:23.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.326+0000 7f34c9240700 1 -- 172.21.15.16:0/1130365310 wait complete. 2023-10-16T13:49:23.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.328+0000 7f34c9240700 1 Processor -- start 2023-10-16T13:49:23.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.328+0000 7f34c9240700 1 -- start start 2023-10-16T13:49:23.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.328+0000 7f34c9240700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34c40d2110 0x7f34c41f5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:23.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.328+0000 7f34c9240700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f34c415ec80 0x7f34c40c5100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:23.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.328+0000 7f34c9240700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f34c415f060 0x7f34c40c5640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:23.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.328+0000 7f34c9240700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34c406c300 con 0x7f34c40d2110 2023-10-16T13:49:23.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.328+0000 7f34c9240700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34c415baf0 con 0x7f34c415ec80 2023-10-16T13:49:23.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.328+0000 7f34c9240700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34c406c510 con 0x7f34c415f060 2023-10-16T13:49:23.336 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.329+0000 7f34c2d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34c40d2110 0x7f34c41f5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:23.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.329+0000 7f34c2d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34c40d2110 0x7f34c41f5bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59868/0 (socket says 172.21.15.16:59868) 2023-10-16T13:49:23.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.329+0000 7f34c2d9d700 1 -- 172.21.15.16:0/2182936317 learned_addr learned my addr 172.21.15.16:0/2182936317 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:23.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.329+0000 7f34c259c700 1 --2- 172.21.15.16:0/2182936317 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f34c415ec80 0x7f34c40c5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:23.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.329+0000 7f34c359e700 1 --2- 172.21.15.16:0/2182936317 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f34c415f060 0x7f34c40c5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:23.337 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.329+0000 7f34c2d9d700 1 -- 172.21.15.16:0/2182936317 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f34c415f060 msgr2=0x7f34c40c5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:23.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.329+0000 7f34c2d9d700 1 --2- 172.21.15.16:0/2182936317 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f34c415f060 0x7f34c40c5640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:23.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.329+0000 7f34c2d9d700 1 -- 172.21.15.16:0/2182936317 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f34c415ec80 msgr2=0x7f34c40c5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:23.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.329+0000 7f34c2d9d700 1 --2- 172.21.15.16:0/2182936317 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f34c415ec80 0x7f34c40c5100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:23.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.329+0000 7f34c2d9d700 1 -- 172.21.15.16:0/2182936317 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34b4009550 con 0x7f34c40d2110 2023-10-16T13:49:23.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.329+0000 7f34c2d9d700 1 --2- 172.21.15.16:0/2182936317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34c40d2110 0x7f34c41f5bf0 secure :-1 s=READY pgs=3273 cs=0 l=1 rev1=1 crypto rx=0x7f34c4067580 tx=0x7f34b4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:23.338 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.329+0000 7f34ab7fe700 1 -- 172.21.15.16:0/2182936317 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34b401a900 con 0x7f34c40d2110 2023-10-16T13:49:23.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.329+0000 7f34ab7fe700 1 -- 172.21.15.16:0/2182936317 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34b401f070 con 0x7f34c40d2110 2023-10-16T13:49:23.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.329+0000 7f34c9240700 1 -- 172.21.15.16:0/2182936317 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34c406c7f0 con 0x7f34c40d2110 2023-10-16T13:49:23.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.329+0000 7f34ab7fe700 1 -- 172.21.15.16:0/2182936317 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34b401ac60 con 0x7f34c40d2110 2023-10-16T13:49:23.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.329+0000 7f34c9240700 1 -- 172.21.15.16:0/2182936317 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34c41f6130 con 0x7f34c40d2110 2023-10-16T13:49:23.339 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.330+0000 7f34c9240700 1 -- 172.21.15.16:0/2182936317 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34c404ebb0 con 0x7f34c40d2110 2023-10-16T13:49:23.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.331+0000 7f34ab7fe700 1 -- 172.21.15.16:0/2182936317 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 172) v1 ==== 89490+0+0 (secure 0 0 0) 0x7f34b4026480 con 0x7f34c40d2110 2023-10-16T13:49:23.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.331+0000 7f34ab7fe700 1 --2- 172.21.15.16:0/2182936317 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f34b006b1d0 0x7f34b006d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:23.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.331+0000 7f34c259c700 1 -- 172.21.15.16:0/2182936317 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f34b006b1d0 msgr2=0x7f34b006d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3920835901 2023-10-16T13:49:23.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.331+0000 7f34ab7fe700 1 -- 172.21.15.16:0/2182936317 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(279..279 src has 1..279) v4 ==== 6501+0+0 (secure 0 0 0) 0x7f34b4021070 con 0x7f34c40d2110 2023-10-16T13:49:23.340 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.331+0000 7f34c259c700 1 --2- 172.21.15.16:0/2182936317 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f34b006b1d0 0x7f34b006d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:49:23.341 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.336+0000 7f34ab7fe700 1 -- 172.21.15.16:0/2182936317 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f34b40560a0 con 0x7f34c40d2110 2023-10-16T13:49:23.374 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:49:23.372+0000 7f8a615ff200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T13:49:23.396 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:23.393+0000 7f0548e60200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T13:49:23.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.532+0000 7f34c259c700 1 -- 172.21.15.16:0/2182936317 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f34b006b1d0 msgr2=0x7f34b006d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3920835901 2023-10-16T13:49:23.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.532+0000 7f34c259c700 1 --2- 172.21.15.16:0/2182936317 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f34b006b1d0 0x7f34b006d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:49:23.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.539+0000 7f34c9240700 1 -- 172.21.15.16:0/2182936317 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f34c406ef80 con 0x7f34c40d2110 2023-10-16T13:49:23.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.539+0000 7f34ab7fe700 1 -- 172.21.15.16:0/2182936317 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v279) v1 ==== 104+0+11 (secure 0 0 0) 0x7f34b40596c0 con 0x7f34c40d2110 2023-10-16T13:49:23.542 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-10-16T13:49:23.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.542+0000 7f34c9240700 1 -- 172.21.15.16:0/2182936317 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f34b006b1d0 msgr2=0x7f34b006d690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:49:23.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.542+0000 7f34c9240700 1 --2- 172.21.15.16:0/2182936317 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f34b006b1d0 0x7f34b006d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:23.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.542+0000 7f34c9240700 1 -- 172.21.15.16:0/2182936317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34c40d2110 msgr2=0x7f34c41f5bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:23.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.542+0000 7f34c9240700 1 --2- 172.21.15.16:0/2182936317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34c40d2110 0x7f34c41f5bf0 secure :-1 s=READY pgs=3273 cs=0 l=1 rev1=1 crypto rx=0x7f34c4067580 tx=0x7f34b4004750 comp rx=0 tx=0).stop 2023-10-16T13:49:23.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.542+0000 7f34c9240700 1 -- 172.21.15.16:0/2182936317 shutdown_connections 2023-10-16T13:49:23.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.542+0000 7f34c9240700 1 --2- 172.21.15.16:0/2182936317 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f34b006b1d0 0x7f34b006d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:23.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.542+0000 7f34c9240700 1 --2- 172.21.15.16:0/2182936317 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f34c415f060 0x7f34c40c5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:23.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.542+0000 7f34c9240700 1 --2- 172.21.15.16:0/2182936317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34c40d2110 0x7f34c41f5bf0 unknown :-1 s=CLOSED pgs=3273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:23.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.542+0000 7f34c9240700 1 --2- 172.21.15.16:0/2182936317 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f34c415ec80 0x7f34c40c5100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:23.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.542+0000 7f34c9240700 1 -- 172.21.15.16:0/2182936317 >> 172.21.15.16:0/2182936317 conn(0x7f34c40d15f0 msgr2=0x7f34c415dc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:23.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.542+0000 7f34c9240700 1 -- 172.21.15.16:0/2182936317 shutdown_connections 2023-10-16T13:49:23.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.542+0000 7f34c9240700 1 -- 172.21.15.16:0/2182936317 wait complete. 2023-10-16T13:49:23.557 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T13:49:23.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.725+0000 7f7a00ccd700 1 Processor -- start 2023-10-16T13:49:23.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.725+0000 7f7a00ccd700 1 -- start start 2023-10-16T13:49:23.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.726+0000 7f7a00ccd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f79fc15da10 0x7f79fc0d1520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:23.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.726+0000 7f7a00ccd700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f79fc0d4060 0x7f79fc0cf2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:23.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.726+0000 7f7a00ccd700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f79fc0d1d20 0x7f79fc0d2170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:23.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.726+0000 7f7a00ccd700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79fc0d26b0 con 0x7f79fc15da10 2023-10-16T13:49:23.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.726+0000 7f7a00ccd700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79fc0d4540 con 0x7f79fc0d1d20 2023-10-16T13:49:23.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.726+0000 7f7a00ccd700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79fc0ce7e0 con 0x7f79fc0d4060 2023-10-16T13:49:23.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.726+0000 7f79fa59c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f79fc15da10 0x7f79fc0d1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:23.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.726+0000 7f79fa59c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f79fc15da10 0x7f79fc0d1520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59906/0 (socket says 172.21.15.16:59906) 2023-10-16T13:49:23.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.726+0000 7f79fa59c700 1 -- 172.21.15.16:0/1180140638 learned_addr learned my addr 172.21.15.16:0/1180140638 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:23.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.726+0000 7f79fad9d700 1 --2- 172.21.15.16:0/1180140638 >> v2:172.21.15.49:3300/0 conn(0x7f79fc0d1d20 0x7f79fc0d2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:23.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.726+0000 7f79f9d9b700 1 --2- 172.21.15.16:0/1180140638 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f79fc0d4060 0x7f79fc0cf2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:23.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.727+0000 7f79fa59c700 1 -- 172.21.15.16:0/1180140638 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f79fc0d4060 msgr2=0x7f79fc0cf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:23.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.727+0000 7f79fa59c700 1 --2- 172.21.15.16:0/1180140638 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f79fc0d4060 0x7f79fc0cf2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:23.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.727+0000 7f79fa59c700 1 -- 172.21.15.16:0/1180140638 >> v2:172.21.15.49:3300/0 conn(0x7f79fc0d1d20 msgr2=0x7f79fc0d2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:23.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.727+0000 7f79fa59c700 1 --2- 172.21.15.16:0/1180140638 >> v2:172.21.15.49:3300/0 conn(0x7f79fc0d1d20 0x7f79fc0d2170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:23.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.727+0000 7f79fa59c700 1 -- 172.21.15.16:0/1180140638 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79fc0ceac0 con 0x7f79fc15da10 2023-10-16T13:49:23.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.727+0000 7f79f9d9b700 1 --2- 172.21.15.16:0/1180140638 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f79fc0d4060 0x7f79fc0cf2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:49:23.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.727+0000 7f79fa59c700 1 --2- 172.21.15.16:0/1180140638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79fc15da10 0x7f79fc0d1520 secure :-1 s=READY pgs=3274 cs=0 l=1 rev1=1 crypto rx=0x7f79fc08d940 tx=0x7f79ec009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1371022ff2c98778 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:23.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.727+0000 7f79f959a700 1 -- 172.21.15.16:0/1180140638 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79ec005e30 con 0x7f79fc15da10 2023-10-16T13:49:23.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.727+0000 7f79f959a700 1 -- 172.21.15.16:0/1180140638 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79ec004030 con 0x7f79fc15da10 2023-10-16T13:49:23.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.727+0000 7f79f959a700 1 -- 172.21.15.16:0/1180140638 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79ec0043e0 con 0x7f79fc15da10 2023-10-16T13:49:23.734 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.728+0000 7f7a00ccd700 1 -- 172.21.15.16:0/1180140638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79fc15da10 msgr2=0x7f79fc0d1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:23.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.728+0000 7f7a00ccd700 1 --2- 172.21.15.16:0/1180140638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79fc15da10 0x7f79fc0d1520 secure :-1 s=READY pgs=3274 cs=0 l=1 rev1=1 crypto rx=0x7f79fc08d940 tx=0x7f79ec009bd0 comp rx=0 tx=0).stop 2023-10-16T13:49:23.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.728+0000 7f7a00ccd700 1 -- 172.21.15.16:0/1180140638 shutdown_connections 2023-10-16T13:49:23.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.728+0000 7f7a00ccd700 1 --2- 172.21.15.16:0/1180140638 >> v2:172.21.15.49:3300/0 conn(0x7f79fc0d1d20 0x7f79fc0d2170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:23.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.728+0000 7f7a00ccd700 1 --2- 172.21.15.16:0/1180140638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79fc15da10 0x7f79fc0d1520 unknown :-1 s=CLOSED pgs=3274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:23.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.728+0000 7f7a00ccd700 1 --2- 172.21.15.16:0/1180140638 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f79fc0d4060 0x7f79fc0cf2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:23.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.728+0000 7f7a00ccd700 1 -- 172.21.15.16:0/1180140638 >> 172.21.15.16:0/1180140638 conn(0x7f79fc09c9f0 msgr2=0x7f79fc0d3740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:23.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.728+0000 7f7a00ccd700 1 -- 172.21.15.16:0/1180140638 shutdown_connections 2023-10-16T13:49:23.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.728+0000 7f7a00ccd700 1 -- 172.21.15.16:0/1180140638 wait complete. 2023-10-16T13:49:23.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.729+0000 7f7a00ccd700 1 Processor -- start 2023-10-16T13:49:23.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.730+0000 7f7a00ccd700 1 -- start start 2023-10-16T13:49:23.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.730+0000 7f7a00ccd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79fc0d1d20 0x7f79fc156590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:23.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.730+0000 7f7a00ccd700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f79fc0d4060 0x7f79fc0ca810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:23.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.731+0000 7f7a00ccd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f79fc15da10 0x7f79fc0cad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:23.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.731+0000 7f7a00ccd700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79fc156ad0 con 0x7f79fc0d1d20 2023-10-16T13:49:23.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.731+0000 7f7a00ccd700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79fc0b4fc0 con 0x7f79fc15da10 2023-10-16T13:49:23.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.731+0000 7f7a00ccd700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79fc0b5260 con 0x7f79fc0d4060 2023-10-16T13:49:23.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.731+0000 7f79fa59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79fc0d1d20 0x7f79fc156590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:23.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.731+0000 7f79fa59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79fc0d1d20 0x7f79fc156590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59912/0 (socket says 172.21.15.16:59912) 2023-10-16T13:49:23.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.731+0000 7f79fa59c700 1 -- 172.21.15.16:0/1599072393 learned_addr learned my addr 172.21.15.16:0/1599072393 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:23.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.731+0000 7f79fad9d700 1 --2- 172.21.15.16:0/1599072393 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f79fc15da10 0x7f79fc0cad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:23.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.731+0000 7f79f9d9b700 1 --2- 172.21.15.16:0/1599072393 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f79fc0d4060 0x7f79fc0ca810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:23.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.731+0000 7f79fad9d700 1 -- 172.21.15.16:0/1599072393 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f79fc0d4060 msgr2=0x7f79fc0ca810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:23.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.731+0000 7f79fad9d700 1 --2- 172.21.15.16:0/1599072393 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f79fc0d4060 0x7f79fc0ca810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:23.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.731+0000 7f79fad9d700 1 -- 172.21.15.16:0/1599072393 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79fc0d1d20 msgr2=0x7f79fc156590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:23.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.731+0000 7f79fad9d700 1 --2- 172.21.15.16:0/1599072393 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79fc0d1d20 0x7f79fc156590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:23.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.731+0000 7f79fad9d700 1 -- 172.21.15.16:0/1599072393 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79ec009500 con 0x7f79fc15da10 2023-10-16T13:49:23.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.731+0000 7f79fa59c700 1 --2- 172.21.15.16:0/1599072393 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79fc0d1d20 0x7f79fc156590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:49:23.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.731+0000 7f79fad9d700 1 --2- 172.21.15.16:0/1599072393 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f79fc15da10 0x7f79fc0cad50 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f79f400ef00 tx=0x7f79f400b390 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:23.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.731+0000 7f79eaffd700 1 -- 172.21.15.16:0/1599072393 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79f4010040 con 0x7f79fc15da10 2023-10-16T13:49:23.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.731+0000 7f79eaffd700 1 -- 172.21.15.16:0/1599072393 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79f400ba80 con 0x7f79fc15da10 2023-10-16T13:49:23.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.731+0000 7f79eaffd700 1 -- 172.21.15.16:0/1599072393 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79f400be10 con 0x7f79fc15da10 2023-10-16T13:49:23.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.731+0000 7f7a00ccd700 1 -- 172.21.15.16:0/1599072393 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79fc0b5540 con 0x7f79fc15da10 2023-10-16T13:49:23.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.732+0000 7f7a00ccd700 1 -- 172.21.15.16:0/1599072393 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79fc0b5b00 con 0x7f79fc15da10 2023-10-16T13:49:23.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.732+0000 7f7a00ccd700 1 -- 172.21.15.16:0/1599072393 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79fc04ebb0 con 0x7f79fc15da10 2023-10-16T13:49:23.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.733+0000 7f79eaffd700 1 -- 172.21.15.16:0/1599072393 <== mon.1 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 172) v1 ==== 89490+0+0 (secure 0 0 0) 0x7f79f401d400 con 0x7f79fc15da10 2023-10-16T13:49:23.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.733+0000 7f79eaffd700 1 --2- 172.21.15.16:0/1599072393 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f79dc06b1d0 0x7f79dc06d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:23.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.733+0000 7f79fa59c700 1 -- 172.21.15.16:0/1599072393 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f79dc06b1d0 msgr2=0x7f79dc06d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3920835901 2023-10-16T13:49:23.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.733+0000 7f79eaffd700 1 -- 172.21.15.16:0/1599072393 <== mon.1 v2:172.21.15.49:3300/0 5 ==== osd_map(279..279 src has 1..279) v4 ==== 6501+0+0 (secure 0 0 0) 0x7f79f408a470 con 0x7f79fc15da10 2023-10-16T13:49:23.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.733+0000 7f79fa59c700 1 --2- 172.21.15.16:0/1599072393 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f79dc06b1d0 0x7f79dc06d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:49:23.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.738+0000 7f79eaffd700 1 -- 172.21.15.16:0/1599072393 <== mon.1 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f79f4054ee0 con 0x7f79fc15da10 2023-10-16T13:49:23.762 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:49:23.760+0000 7f8a615ff200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T13:49:23.795 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:23.793+0000 7f0548e60200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T13:49:23.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.934+0000 7f79fa59c700 1 -- 172.21.15.16:0/1599072393 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f79dc06b1d0 msgr2=0x7f79dc06d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3920835901 2023-10-16T13:49:23.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.934+0000 7f79fa59c700 1 --2- 172.21.15.16:0/1599072393 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f79dc06b1d0 0x7f79dc06d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:49:23.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.943+0000 7f7a00ccd700 1 -- 172.21.15.16:0/1599072393 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f79fc15d1e0 con 0x7f79fc15da10 2023-10-16T13:49:23.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.943+0000 7f79eaffd700 1 -- 172.21.15.16:0/1599072393 <== mon.1 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v279) v1 ==== 100+0+11 (secure 0 0 0) 0x7f79f4058500 con 0x7f79fc15da10 2023-10-16T13:49:23.946 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-10-16T13:49:23.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.945+0000 7f7a00ccd700 1 -- 172.21.15.16:0/1599072393 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f79dc06b1d0 msgr2=0x7f79dc06d690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:49:23.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.945+0000 7f7a00ccd700 1 --2- 172.21.15.16:0/1599072393 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f79dc06b1d0 0x7f79dc06d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:23.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.945+0000 7f7a00ccd700 1 -- 172.21.15.16:0/1599072393 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f79fc15da10 msgr2=0x7f79fc0cad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:23.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.946+0000 7f7a00ccd700 1 --2- 172.21.15.16:0/1599072393 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f79fc15da10 0x7f79fc0cad50 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f79f400ef00 tx=0x7f79f400b390 comp rx=0 tx=0).stop 2023-10-16T13:49:23.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.946+0000 7f7a00ccd700 1 -- 172.21.15.16:0/1599072393 shutdown_connections 2023-10-16T13:49:23.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.946+0000 7f7a00ccd700 1 --2- 172.21.15.16:0/1599072393 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f79dc06b1d0 0x7f79dc06d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:23.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.946+0000 7f7a00ccd700 1 --2- 172.21.15.16:0/1599072393 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f79fc15da10 0x7f79fc0cad50 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:23.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.946+0000 7f7a00ccd700 1 --2- 172.21.15.16:0/1599072393 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79fc0d1d20 0x7f79fc156590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:23.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.946+0000 7f7a00ccd700 1 --2- 172.21.15.16:0/1599072393 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f79fc0d4060 0x7f79fc0ca810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:23.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.946+0000 7f7a00ccd700 1 -- 172.21.15.16:0/1599072393 >> 172.21.15.16:0/1599072393 conn(0x7f79fc09c9f0 msgr2=0x7f79fc0b1100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:23.950 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.946+0000 7f7a00ccd700 1 -- 172.21.15.16:0/1599072393 shutdown_connections 2023-10-16T13:49:23.951 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:23.946+0000 7f7a00ccd700 1 -- 172.21.15.16:0/1599072393 wait complete. 2023-10-16T13:49:23.961 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T13:49:24.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.085+0000 7fd573ccc700 1 Processor -- start 2023-10-16T13:49:24.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.086+0000 7fd573ccc700 1 -- start start 2023-10-16T13:49:24.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.087+0000 7fd573ccc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd56c15ad00 0x7fd56c06c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:24.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.087+0000 7fd573ccc700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fd56c06cd40 0x7fd56c0b3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:24.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.087+0000 7fd573ccc700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd56c0b3c80 0x7fd56c15d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:24.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.087+0000 7fd573ccc700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd56c06d1d0 con 0x7fd56c15ad00 2023-10-16T13:49:24.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.087+0000 7fd573ccc700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd56c0b4110 con 0x7fd56c06cd40 2023-10-16T13:49:24.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.087+0000 7fd573ccc700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd56c0b2120 con 0x7fd56c0b3c80 2023-10-16T13:49:24.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.087+0000 7fd571a68700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd56c15ad00 0x7fd56c06c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:24.092 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.087+0000 7fd571a68700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd56c15ad00 0x7fd56c06c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59922/0 (socket says 172.21.15.16:59922) 2023-10-16T13:49:24.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.087+0000 7fd571a68700 1 -- 172.21.15.16:0/2640419037 learned_addr learned my addr 172.21.15.16:0/2640419037 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:24.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.087+0000 7fd571267700 1 --2- 172.21.15.16:0/2640419037 >> v2:172.21.15.49:3300/0 conn(0x7fd56c06cd40 0x7fd56c0b3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:24.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.087+0000 7fd572269700 1 --2- 172.21.15.16:0/2640419037 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd56c0b3c80 0x7fd56c15d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:24.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.087+0000 7fd571a68700 1 -- 172.21.15.16:0/2640419037 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd56c0b3c80 msgr2=0x7fd56c15d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:24.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.087+0000 7fd571a68700 1 --2- 172.21.15.16:0/2640419037 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd56c0b3c80 0x7fd56c15d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:24.093 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.087+0000 7fd571a68700 1 -- 172.21.15.16:0/2640419037 >> v2:172.21.15.49:3300/0 conn(0x7fd56c06cd40 msgr2=0x7fd56c0b3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:24.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.087+0000 7fd571a68700 1 --2- 172.21.15.16:0/2640419037 >> v2:172.21.15.49:3300/0 conn(0x7fd56c06cd40 0x7fd56c0b3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:24.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.087+0000 7fd571a68700 1 -- 172.21.15.16:0/2640419037 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd56c0b4320 con 0x7fd56c15ad00 2023-10-16T13:49:24.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.087+0000 7fd572269700 1 --2- 172.21.15.16:0/2640419037 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd56c0b3c80 0x7fd56c15d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:49:24.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.087+0000 7fd571a68700 1 --2- 172.21.15.16:0/2640419037 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd56c15ad00 0x7fd56c06c800 secure :-1 s=READY pgs=3275 cs=0 l=1 rev1=1 crypto rx=0x7fd5680098f0 tx=0x7fd568009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e8a441a54d30934 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:24.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.088+0000 7fd570a66700 1 -- 172.21.15.16:0/2640419037 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd568011830 con 0x7fd56c15ad00 2023-10-16T13:49:24.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.088+0000 7fd570a66700 1 -- 172.21.15.16:0/2640419037 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd568011990 con 0x7fd56c15ad00 2023-10-16T13:49:24.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.088+0000 7fd570a66700 1 -- 172.21.15.16:0/2640419037 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd568011d20 con 0x7fd56c15ad00 2023-10-16T13:49:24.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.088+0000 7fd573ccc700 1 -- 172.21.15.16:0/2640419037 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd56c15ad00 msgr2=0x7fd56c06c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:24.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.088+0000 7fd573ccc700 1 --2- 172.21.15.16:0/2640419037 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd56c15ad00 0x7fd56c06c800 secure :-1 s=READY pgs=3275 cs=0 l=1 rev1=1 crypto rx=0x7fd5680098f0 tx=0x7fd568009c00 comp rx=0 tx=0).stop 2023-10-16T13:49:24.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.088+0000 7fd573ccc700 1 -- 172.21.15.16:0/2640419037 shutdown_connections 2023-10-16T13:49:24.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.088+0000 7fd573ccc700 1 --2- 172.21.15.16:0/2640419037 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd56c0b3c80 0x7fd56c15d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:24.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.088+0000 7fd573ccc700 1 --2- 172.21.15.16:0/2640419037 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd56c15ad00 0x7fd56c06c800 unknown :-1 s=CLOSED pgs=3275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:24.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.088+0000 7fd573ccc700 1 --2- 172.21.15.16:0/2640419037 >> v2:172.21.15.49:3300/0 conn(0x7fd56c06cd40 0x7fd56c0b3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:24.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.088+0000 7fd573ccc700 1 -- 172.21.15.16:0/2640419037 >> 172.21.15.16:0/2640419037 conn(0x7fd56c0b4cd0 msgr2=0x7fd56c06a7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:24.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.088+0000 7fd573ccc700 1 -- 172.21.15.16:0/2640419037 shutdown_connections 2023-10-16T13:49:24.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.088+0000 7fd573ccc700 1 -- 172.21.15.16:0/2640419037 wait complete. 2023-10-16T13:49:24.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.090+0000 7fd573ccc700 1 Processor -- start 2023-10-16T13:49:24.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.090+0000 7fd573ccc700 1 -- start start 2023-10-16T13:49:24.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.090+0000 7fd573ccc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd56c06cd40 0x7fd56c159760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:24.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.090+0000 7fd573ccc700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd56c0b3c80 0x7fd56c150e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:24.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.090+0000 7fd573ccc700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd56c15ad00 0x7fd56c1f6a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:24.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.090+0000 7fd573ccc700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd56c159d40 con 0x7fd56c06cd40 2023-10-16T13:49:24.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.090+0000 7fd573ccc700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd56c1f3590 con 0x7fd56c0b3c80 2023-10-16T13:49:24.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.090+0000 7fd573ccc700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd56c1f6fb0 con 0x7fd56c15ad00 2023-10-16T13:49:24.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.091+0000 7fd571a68700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd56c06cd40 0x7fd56c159760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:24.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.091+0000 7fd571267700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd56c0b3c80 0x7fd56c150e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:24.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.091+0000 7fd571a68700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd56c06cd40 0x7fd56c159760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59928/0 (socket says 172.21.15.16:59928) 2023-10-16T13:49:24.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.091+0000 7fd571a68700 1 -- 172.21.15.16:0/3540454128 learned_addr learned my addr 172.21.15.16:0/3540454128 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:24.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.091+0000 7fd572269700 1 --2- 172.21.15.16:0/3540454128 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd56c15ad00 0x7fd56c1f6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:24.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.091+0000 7fd571267700 1 -- 172.21.15.16:0/3540454128 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd56c15ad00 msgr2=0x7fd56c1f6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:24.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.091+0000 7fd571267700 1 --2- 172.21.15.16:0/3540454128 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd56c15ad00 0x7fd56c1f6a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:24.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.091+0000 7fd571267700 1 -- 172.21.15.16:0/3540454128 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd56c06cd40 msgr2=0x7fd56c159760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:24.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.091+0000 7fd571267700 1 --2- 172.21.15.16:0/3540454128 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd56c06cd40 0x7fd56c159760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:24.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.091+0000 7fd571267700 1 -- 172.21.15.16:0/3540454128 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd568009550 con 0x7fd56c0b3c80 2023-10-16T13:49:24.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.091+0000 7fd571a68700 1 --2- 172.21.15.16:0/3540454128 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd56c06cd40 0x7fd56c159760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:49:24.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.091+0000 7fd571267700 1 --2- 172.21.15.16:0/3540454128 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd56c0b3c80 0x7fd56c150e20 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fd55c00e870 tx=0x7fd55c00eb80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:24.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.091+0000 7fd5627fc700 1 -- 172.21.15.16:0/3540454128 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd55c013680 con 0x7fd56c0b3c80 2023-10-16T13:49:24.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.091+0000 7fd5627fc700 1 -- 172.21.15.16:0/3540454128 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd55c0137e0 con 0x7fd56c0b3c80 2023-10-16T13:49:24.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.091+0000 7fd5627fc700 1 -- 172.21.15.16:0/3540454128 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd55c013b50 con 0x7fd56c0b3c80 2023-10-16T13:49:24.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.091+0000 7fd573ccc700 1 -- 172.21.15.16:0/3540454128 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd56c1f3a20 con 0x7fd56c0b3c80 2023-10-16T13:49:24.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.091+0000 7fd573ccc700 1 -- 172.21.15.16:0/3540454128 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd56c1f2f40 con 0x7fd56c0b3c80 2023-10-16T13:49:24.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.092+0000 7fd573ccc700 1 -- 172.21.15.16:0/3540454128 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd56c0b26a0 con 0x7fd56c0b3c80 2023-10-16T13:49:24.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.093+0000 7fd5627fc700 1 -- 172.21.15.16:0/3540454128 <== mon.1 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 172) v1 ==== 89490+0+0 (secure 0 0 0) 0x7fd55c013cb0 con 0x7fd56c0b3c80 2023-10-16T13:49:24.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.093+0000 7fd5627fc700 1 --2- 172.21.15.16:0/3540454128 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7fd55006b1d0 0x7fd55006d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:24.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.093+0000 7fd571a68700 1 -- 172.21.15.16:0/3540454128 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7fd55006b1d0 msgr2=0x7fd55006d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3920835901 2023-10-16T13:49:24.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.093+0000 7fd5627fc700 1 -- 172.21.15.16:0/3540454128 <== mon.1 v2:172.21.15.49:3300/0 5 ==== osd_map(279..279 src has 1..279) v4 ==== 6501+0+0 (secure 0 0 0) 0x7fd55c089c80 con 0x7fd56c0b3c80 2023-10-16T13:49:24.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.093+0000 7fd571a68700 1 --2- 172.21.15.16:0/3540454128 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7fd55006b1d0 0x7fd55006d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:49:24.105 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.098+0000 7fd5627fc700 1 -- 172.21.15.16:0/3540454128 <== mon.1 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd56c0b26a0 con 0x7fd56c0b3c80 2023-10-16T13:49:24.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.294+0000 7fd571a68700 1 -- 172.21.15.16:0/3540454128 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7fd55006b1d0 msgr2=0x7fd55006d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3920835901 2023-10-16T13:49:24.296 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.294+0000 7fd571a68700 1 --2- 172.21.15.16:0/3540454128 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7fd55006b1d0 0x7fd55006d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:49:24.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.364+0000 7fd573ccc700 1 -- 172.21.15.16:0/3540454128 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fd56c0b26a0 con 0x7fd56c0b3c80 2023-10-16T13:49:24.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.365+0000 7fd5627fc700 1 -- 172.21.15.16:0/3540454128 <== mon.1 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v279) v1 ==== 80+0+468 (secure 0 0 0) 0x7fd55c0171f0 con 0x7fd56c0b3c80 2023-10-16T13:49:24.368 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/750285561 2023-10-17T13:49:17.194509+0000 2023-10-16T13:49:24.368 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6833/1257392887 2023-10-17T13:49:17.194509+0000 2023-10-16T13:49:24.368 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6832/1257392887 2023-10-17T13:49:17.194509+0000 2023-10-16T13:49:24.368 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/1101859511 2023-10-17T13:49:17.194509+0000 2023-10-16T13:49:24.369 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/197265557 2023-10-17T13:49:17.194509+0000 2023-10-16T13:49:24.369 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2024751970 2023-10-17T13:49:17.194509+0000 2023-10-16T13:49:24.369 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2401946155 2023-10-17T13:49:17.194509+0000 2023-10-16T13:49:24.369 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/1156439592 2023-10-17T13:49:17.194509+0000 2023-10-16T13:49:24.370 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:49:24.366+0000 7f8a615ff200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T13:49:24.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.367+0000 7fd573ccc700 1 -- 172.21.15.16:0/3540454128 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7fd55006b1d0 msgr2=0x7fd55006d690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:49:24.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.367+0000 7fd573ccc700 1 --2- 172.21.15.16:0/3540454128 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7fd55006b1d0 0x7fd55006d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:24.371 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.367+0000 7fd573ccc700 1 -- 172.21.15.16:0/3540454128 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd56c0b3c80 msgr2=0x7fd56c150e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:24.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.367+0000 7fd573ccc700 1 --2- 172.21.15.16:0/3540454128 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd56c0b3c80 0x7fd56c150e20 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fd55c00e870 tx=0x7fd55c00eb80 comp rx=0 tx=0).stop 2023-10-16T13:49:24.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.368+0000 7fd573ccc700 1 -- 172.21.15.16:0/3540454128 shutdown_connections 2023-10-16T13:49:24.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.368+0000 7fd573ccc700 1 --2- 172.21.15.16:0/3540454128 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7fd55006b1d0 0x7fd55006d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:24.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.368+0000 7fd573ccc700 1 --2- 172.21.15.16:0/3540454128 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fd56c15ad00 0x7fd56c1f6a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:24.372 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.368+0000 7fd573ccc700 1 --2- 172.21.15.16:0/3540454128 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd56c06cd40 0x7fd56c159760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:24.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.368+0000 7fd573ccc700 1 --2- 172.21.15.16:0/3540454128 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd56c0b3c80 0x7fd56c150e20 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:24.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.368+0000 7fd573ccc700 1 -- 172.21.15.16:0/3540454128 >> 172.21.15.16:0/3540454128 conn(0x7fd56c0b4cd0 msgr2=0x7fd56c0afb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:24.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.368+0000 7fd573ccc700 1 -- 172.21.15.16:0/3540454128 shutdown_connections 2023-10-16T13:49:24.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.368+0000 7fd573ccc700 1 -- 172.21.15.16:0/3540454128 wait complete. 2023-10-16T13:49:24.373 INFO:teuthology.orchestra.run.smithi016.stderr:listed 8 entries 2023-10-16T13:49:24.384 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-16T13:49:24.384 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:49:24.384 DEBUG:teuthology.orchestra.run.smithi016:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-16T13:49:24.412 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:24.410+0000 7f0548e60200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T13:49:24.425 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2023-10-16T13:49:24.425 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:49:24.426 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-16T13:49:24.442 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:49:24.439+0000 7f8a615ff200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T13:49:24.482 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T13:49:24.483 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:49:24.490 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:24.487+0000 7f0548e60200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T13:49:24.500 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284204k,nr_inodes=4071051,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=753 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260540k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T13:49:24.500 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T13:49:24.501 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:49:24.518 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16287364k,nr_inodes=4071841,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20251 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3261172k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T13:49:24.519 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T13:49:24.519 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:49:24.558 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284204k,nr_inodes=4071051,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=753 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260540k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T13:49:24.558 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:49:24.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.682+0000 7f64c96fa700 1 Processor -- start 2023-10-16T13:49:24.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.683+0000 7f64c96fa700 1 -- start start 2023-10-16T13:49:24.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.683+0000 7f64c96fa700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f64c415b760 0x7f64c406a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:24.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.683+0000 7f64c96fa700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f64c40b45c0 0x7f64c406ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:24.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.683+0000 7f64c96fa700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64c40d03b0 con 0x7f64c40b2be0 2023-10-16T13:49:24.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.683+0000 7f64c96fa700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64c415bb40 con 0x7f64c40b45c0 2023-10-16T13:49:24.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.683+0000 7f64c96fa700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64c40cb760 con 0x7f64c415b760 2023-10-16T13:49:24.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.684+0000 7f64c27fc700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f64c40b2be0 0x7f64c40719e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49212/0 (socket says 172.21.15.16:49212) 2023-10-16T13:49:24.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.684+0000 7f64c27fc700 1 -- 172.21.15.16:0/2302399359 learned_addr learned my addr 172.21.15.16:0/2302399359 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:49:24.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.684+0000 7f64c37fe700 1 --2- 172.21.15.16:0/2302399359 >> v2:172.21.15.49:3300/0 conn(0x7f64c40b45c0 0x7f64c406ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:24.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.684+0000 7f64c2ffd700 1 --2- 172.21.15.16:0/2302399359 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f64c415b760 0x7f64c406a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:24.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.684+0000 7f64c1ffb700 1 -- 172.21.15.16:0/2302399359 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1721930374 0 0) 0x7f64c40d03b0 con 0x7f64c40b2be0 2023-10-16T13:49:24.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.684+0000 7f64c1ffb700 1 -- 172.21.15.16:0/2302399359 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64ac0035b0 con 0x7f64c40b2be0 2023-10-16T13:49:24.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.684+0000 7f64c2ffd700 1 -- 172.21.15.16:0/2302399359 >> v2:172.21.15.49:3300/0 conn(0x7f64c40b45c0 msgr2=0x7f64c406ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:24.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.684+0000 7f64c2ffd700 1 --2- 172.21.15.16:0/2302399359 >> v2:172.21.15.49:3300/0 conn(0x7f64c40b45c0 0x7f64c406ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:24.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.684+0000 7f64c2ffd700 1 -- 172.21.15.16:0/2302399359 >> v1:172.21.15.16:6789/0 conn(0x7f64c40b2be0 legacy=0x7f64c40719e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:24.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.684+0000 7f64c2ffd700 1 -- 172.21.15.16:0/2302399359 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64c40d0600 con 0x7f64c415b760 2023-10-16T13:49:24.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.684+0000 7f64c37fe700 1 --2- 172.21.15.16:0/2302399359 >> v2:172.21.15.49:3300/0 conn(0x7f64c40b45c0 0x7f64c406ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:49:24.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.684+0000 7f64c2ffd700 1 --2- 172.21.15.16:0/2302399359 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f64c415b760 0x7f64c406a7e0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f64b4009840 tx=0x7f64b4009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6dba917c308f1b2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:24.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.684+0000 7f64c1ffb700 1 -- 172.21.15.16:0/2302399359 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64b4011830 con 0x7f64c415b760 2023-10-16T13:49:24.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.684+0000 7f64c1ffb700 1 -- 172.21.15.16:0/2302399359 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64b4011990 con 0x7f64c415b760 2023-10-16T13:49:24.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.684+0000 7f64c1ffb700 1 -- 172.21.15.16:0/2302399359 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64b4011d00 con 0x7f64c415b760 2023-10-16T13:49:24.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.685+0000 7f64c96fa700 1 -- 172.21.15.16:0/2302399359 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f64c415b760 msgr2=0x7f64c406a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:24.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.685+0000 7f64c96fa700 1 --2- 172.21.15.16:0/2302399359 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f64c415b760 0x7f64c406a7e0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f64b4009840 tx=0x7f64b4009b50 comp rx=0 tx=0).stop 2023-10-16T13:49:24.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.685+0000 7f64c96fa700 1 -- 172.21.15.16:0/2302399359 shutdown_connections 2023-10-16T13:49:24.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.685+0000 7f64c96fa700 1 --2- 172.21.15.16:0/2302399359 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f64c415b760 0x7f64c406a7e0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:24.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.685+0000 7f64c96fa700 1 --2- 172.21.15.16:0/2302399359 >> v2:172.21.15.49:3300/0 conn(0x7f64c40b45c0 0x7f64c406ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:24.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.685+0000 7f64c96fa700 1 -- 172.21.15.16:0/2302399359 >> 172.21.15.16:0/2302399359 conn(0x7f64c40b1fd0 msgr2=0x7f64c40b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:24.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.685+0000 7f64c96fa700 1 -- 172.21.15.16:0/2302399359 shutdown_connections 2023-10-16T13:49:24.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.685+0000 7f64c96fa700 1 -- 172.21.15.16:0/2302399359 wait complete. 2023-10-16T13:49:24.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.686+0000 7f64c96fa700 1 Processor -- start 2023-10-16T13:49:24.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.686+0000 7f64c96fa700 1 -- start start 2023-10-16T13:49:24.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.687+0000 7f64c96fa700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f64c40b2be0 0x7f64c414b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:24.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.687+0000 7f64c96fa700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f64c40b45c0 0x7f64c40acfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:24.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.687+0000 7f64c96fa700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64c415b760 0x7f64c40cf020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:24.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.687+0000 7f64c96fa700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64c414be20 con 0x7f64c415b760 2023-10-16T13:49:24.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.687+0000 7f64c96fa700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64c40ce010 con 0x7f64c40b45c0 2023-10-16T13:49:24.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.687+0000 7f64c96fa700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64c40ceb90 con 0x7f64c40b2be0 2023-10-16T13:49:24.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.687+0000 7f64c37fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64c415b760 0x7f64c40cf020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:24.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.687+0000 7f64c37fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64c415b760 0x7f64c40cf020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59948/0 (socket says 172.21.15.16:59948) 2023-10-16T13:49:24.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.687+0000 7f64c27fc700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f64c40b45c0 0x7f64c40acfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:24.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.687+0000 7f64c37fe700 1 -- 172.21.15.16:0/2077953138 learned_addr learned my addr 172.21.15.16:0/2077953138 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:24.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.687+0000 7f64c2ffd700 1 --2- 172.21.15.16:0/2077953138 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f64c40b2be0 0x7f64c414b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:24.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.688+0000 7f64c37fe700 1 -- 172.21.15.16:0/2077953138 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f64c40b2be0 msgr2=0x7f64c414b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:24.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.688+0000 7f64c37fe700 1 --2- 172.21.15.16:0/2077953138 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f64c40b2be0 0x7f64c414b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:24.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.688+0000 7f64c37fe700 1 -- 172.21.15.16:0/2077953138 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f64c40b45c0 msgr2=0x7f64c40acfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:24.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.688+0000 7f64c37fe700 1 --2- 172.21.15.16:0/2077953138 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f64c40b45c0 0x7f64c40acfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:24.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.688+0000 7f64c37fe700 1 -- 172.21.15.16:0/2077953138 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64bc009a70 con 0x7f64c415b760 2023-10-16T13:49:24.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.688+0000 7f64c37fe700 1 --2- 172.21.15.16:0/2077953138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64c415b760 0x7f64c40cf020 secure :-1 s=READY pgs=3277 cs=0 l=1 rev1=1 crypto rx=0x7f64bc00efc0 tx=0x7f64bc00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:24.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.688+0000 7f64ab7fe700 1 -- 172.21.15.16:0/2077953138 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64bc009ca0 con 0x7f64c415b760 2023-10-16T13:49:24.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.688+0000 7f64ab7fe700 1 -- 172.21.15.16:0/2077953138 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64bc00f040 con 0x7f64c415b760 2023-10-16T13:49:24.700 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.688+0000 7f64c96fa700 1 -- 172.21.15.16:0/2077953138 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64b40094a0 con 0x7f64c415b760 2023-10-16T13:49:24.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.688+0000 7f64ab7fe700 1 -- 172.21.15.16:0/2077953138 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64bc00ced0 con 0x7f64c415b760 2023-10-16T13:49:24.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.688+0000 7f64c96fa700 1 -- 172.21.15.16:0/2077953138 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64c40ce260 con 0x7f64c415b760 2023-10-16T13:49:24.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.689+0000 7f64c96fa700 1 -- 172.21.15.16:0/2077953138 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64c40cb580 con 0x7f64c415b760 2023-10-16T13:49:24.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.689+0000 7f64ab7fe700 1 -- 172.21.15.16:0/2077953138 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 172) v1 ==== 89490+0+0 (secure 0 0 0) 0x7f64bc014480 con 0x7f64c415b760 2023-10-16T13:49:24.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.690+0000 7f64ab7fe700 1 --2- 172.21.15.16:0/2077953138 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f64b006b1d0 0x7f64b006d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:24.701 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.690+0000 7f64c2ffd700 1 -- 172.21.15.16:0/2077953138 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f64b006b1d0 msgr2=0x7f64b006d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3920835901 2023-10-16T13:49:24.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.690+0000 7f64ab7fe700 1 -- 172.21.15.16:0/2077953138 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(279..279 src has 1..279) v4 ==== 6501+0+0 (secure 0 0 0) 0x7f64bc08aa10 con 0x7f64c415b760 2023-10-16T13:49:24.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.690+0000 7f64c2ffd700 1 --2- 172.21.15.16:0/2077953138 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f64b006b1d0 0x7f64b006d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:49:24.702 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.694+0000 7f64ab7fe700 1 -- 172.21.15.16:0/2077953138 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f64bc055380 con 0x7f64c415b760 2023-10-16T13:49:24.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.891+0000 7f64c2ffd700 1 -- 172.21.15.16:0/2077953138 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f64b006b1d0 msgr2=0x7f64b006d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3920835901 2023-10-16T13:49:24.893 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.891+0000 7f64c2ffd700 1 --2- 172.21.15.16:0/2077953138 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f64b006b1d0 0x7f64b006d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:49:24.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.946+0000 7f64c96fa700 1 -- 172.21.15.16:0/2077953138 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f64c404ebb0 con 0x7f64c415b760 2023-10-16T13:49:24.949 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.946+0000 7f64ab7fe700 1 -- 172.21.15.16:0/2077953138 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 155 v155) v1 ==== 95+0+4926 (secure 0 0 0) 0x7f64bc018090 con 0x7f64c415b760 2023-10-16T13:49:24.950 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:49:24.950 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":155,"default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12280,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/1127700443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":1127700443},{"type":"v1","addr":"172.21.15.49:6834","nonce":1127700443}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":150},{"gid":13581,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/818113029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":818113029},{"type":"v1","addr":"172.21.15.16:6836","nonce":818113029}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149},{"gid":13584,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3762868123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":3762868123},{"type":"v1","addr":"172.21.15.16:6837","nonce":3762868123}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":150}],"filesystems":[{"mdsmap":{"epoch":155,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:49:02.107137+0000","modified":"2023-10-16T13:49:05.140377+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":13599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13599":{"gid":13599,"name":"b","rank":0,"incarnation":151,"state":"up:active","state_seq":3,"addr":"172.21.15.49:6835/2082691635","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":2082691635},{"type":"v1","addr":"172.21.15.49:6835","nonce":2082691635}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[38],"metadata_pool":37,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":18}]} 2023-10-16T13:49:24.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.949+0000 7f64c96fa700 1 -- 172.21.15.16:0/2077953138 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f64b006b1d0 msgr2=0x7f64b006d690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:49:24.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.949+0000 7f64c96fa700 1 --2- 172.21.15.16:0/2077953138 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f64b006b1d0 0x7f64b006d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:24.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.949+0000 7f64c96fa700 1 -- 172.21.15.16:0/2077953138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64c415b760 msgr2=0x7f64c40cf020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:24.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.949+0000 7f64c96fa700 1 --2- 172.21.15.16:0/2077953138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64c415b760 0x7f64c40cf020 secure :-1 s=READY pgs=3277 cs=0 l=1 rev1=1 crypto rx=0x7f64bc00efc0 tx=0x7f64bc00c3a0 comp rx=0 tx=0).stop 2023-10-16T13:49:24.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.949+0000 7f64c96fa700 1 -- 172.21.15.16:0/2077953138 shutdown_connections 2023-10-16T13:49:24.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.949+0000 7f64c96fa700 1 --2- 172.21.15.16:0/2077953138 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f64b006b1d0 0x7f64b006d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:24.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.949+0000 7f64c96fa700 1 --2- 172.21.15.16:0/2077953138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64c415b760 0x7f64c40cf020 unknown :-1 s=CLOSED pgs=3277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:24.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.949+0000 7f64c96fa700 1 --2- 172.21.15.16:0/2077953138 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f64c40b2be0 0x7f64c414b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:24.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.949+0000 7f64c96fa700 1 --2- 172.21.15.16:0/2077953138 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f64c40b45c0 0x7f64c40acfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:24.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.949+0000 7f64c96fa700 1 -- 172.21.15.16:0/2077953138 >> 172.21.15.16:0/2077953138 conn(0x7f64c40b1fd0 msgr2=0x7f64c415e6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:24.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.949+0000 7f64c96fa700 1 -- 172.21.15.16:0/2077953138 shutdown_connections 2023-10-16T13:49:24.954 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:24.949+0000 7f64c96fa700 1 -- 172.21.15.16:0/2077953138 wait complete. 2023-10-16T13:49:24.955 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 155 2023-10-16T13:49:24.965 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:49:25.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.093+0000 7fc26934a700 1 Processor -- start 2023-10-16T13:49:25.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.093+0000 7fc26934a700 1 -- start start 2023-10-16T13:49:25.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.093+0000 7fc26934a700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fc2640b3ba0 0x7fc2640b2da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:25.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.093+0000 7fc26934a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc26415a6e0 0x7fc264070860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:25.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.093+0000 7fc26934a700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2640b4060 con 0x7fc26415a300 2023-10-16T13:49:25.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.093+0000 7fc26934a700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2640b22c0 con 0x7fc26415a6e0 2023-10-16T13:49:25.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.093+0000 7fc26934a700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2640b25a0 con 0x7fc2640b3ba0 2023-10-16T13:49:25.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.094+0000 7fc2637fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc26415a6e0 0x7fc264070860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:25.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.094+0000 7fc2637fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc26415a6e0 0x7fc264070860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59958/0 (socket says 172.21.15.16:59958) 2023-10-16T13:49:25.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.094+0000 7fc2637fe700 1 -- 172.21.15.16:0/832293193 learned_addr learned my addr 172.21.15.16:0/832293193 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:25.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.094+0000 7fc2627fc700 1 --2- 172.21.15.16:0/832293193 >> v2:172.21.15.49:3300/0 conn(0x7fc2640b3ba0 0x7fc2640b2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:25.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.094+0000 7fc2637fe700 1 -- 172.21.15.16:0/832293193 >> v2:172.21.15.49:3300/0 conn(0x7fc2640b3ba0 msgr2=0x7fc2640b2da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:25.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.094+0000 7fc2637fe700 1 --2- 172.21.15.16:0/832293193 >> v2:172.21.15.49:3300/0 conn(0x7fc2640b3ba0 0x7fc2640b2da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:25.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.094+0000 7fc2637fe700 1 -- 172.21.15.16:0/832293193 >> v1:172.21.15.49:6789/0 conn(0x7fc26415a300 legacy=0x7fc26415ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:25.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.094+0000 7fc2637fe700 1 -- 172.21.15.16:0/832293193 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2640b2880 con 0x7fc26415a6e0 2023-10-16T13:49:25.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.094+0000 7fc2637fe700 1 --2- 172.21.15.16:0/832293193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc26415a6e0 0x7fc264070860 secure :-1 s=READY pgs=3278 cs=0 l=1 rev1=1 crypto rx=0x7fc2640b3af0 tx=0x7fc25c00be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de7fe2fac2218246 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:25.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.094+0000 7fc261ffb700 1 -- 172.21.15.16:0/832293193 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc25c00dc70 con 0x7fc26415a6e0 2023-10-16T13:49:25.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.094+0000 7fc261ffb700 1 -- 172.21.15.16:0/832293193 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc25c00ddd0 con 0x7fc26415a6e0 2023-10-16T13:49:25.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.094+0000 7fc261ffb700 1 -- 172.21.15.16:0/832293193 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc25c010690 con 0x7fc26415a6e0 2023-10-16T13:49:25.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.095+0000 7fc26934a700 1 -- 172.21.15.16:0/832293193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc26415a6e0 msgr2=0x7fc264070860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:25.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.095+0000 7fc26934a700 1 --2- 172.21.15.16:0/832293193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc26415a6e0 0x7fc264070860 secure :-1 s=READY pgs=3278 cs=0 l=1 rev1=1 crypto rx=0x7fc2640b3af0 tx=0x7fc25c00be70 comp rx=0 tx=0).stop 2023-10-16T13:49:25.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.095+0000 7fc26934a700 1 -- 172.21.15.16:0/832293193 shutdown_connections 2023-10-16T13:49:25.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.095+0000 7fc26934a700 1 --2- 172.21.15.16:0/832293193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc26415a6e0 0x7fc264070860 unknown :-1 s=CLOSED pgs=3278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:25.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.095+0000 7fc26934a700 1 --2- 172.21.15.16:0/832293193 >> v2:172.21.15.49:3300/0 conn(0x7fc2640b3ba0 0x7fc2640b2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:25.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.095+0000 7fc26934a700 1 -- 172.21.15.16:0/832293193 >> 172.21.15.16:0/832293193 conn(0x7fc2640b4cd0 msgr2=0x7fc26406a7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:25.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.095+0000 7fc26934a700 1 -- 172.21.15.16:0/832293193 shutdown_connections 2023-10-16T13:49:25.104 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.095+0000 7fc26934a700 1 -- 172.21.15.16:0/832293193 wait complete. 2023-10-16T13:49:25.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.096+0000 7fc26934a700 1 Processor -- start 2023-10-16T13:49:25.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.096+0000 7fc26934a700 1 -- start start 2023-10-16T13:49:25.107 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.097+0000 7fc26934a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc2640b3ba0 0x7fc264142c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:25.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.097+0000 7fc26934a700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc26415a300 0x7fc2640aed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:25.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.097+0000 7fc26934a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc26415a6e0 0x7fc26415e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:25.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.097+0000 7fc26934a700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc26415ab80 con 0x7fc2640b3ba0 2023-10-16T13:49:25.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.097+0000 7fc26934a700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2641431a0 con 0x7fc26415a300 2023-10-16T13:49:25.108 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.097+0000 7fc26934a700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2640af7c0 con 0x7fc26415a6e0 2023-10-16T13:49:25.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.097+0000 7fc262ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc2640b3ba0 0x7fc264142c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:25.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.097+0000 7fc262ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc2640b3ba0 0x7fc264142c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59960/0 (socket says 172.21.15.16:59960) 2023-10-16T13:49:25.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.097+0000 7fc262ffd700 1 -- 172.21.15.16:0/3701974133 learned_addr learned my addr 172.21.15.16:0/3701974133 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:25.109 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.097+0000 7fc2627fc700 1 --2- 172.21.15.16:0/3701974133 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc26415a300 0x7fc2640aed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:25.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.097+0000 7fc2637fe700 1 --2- 172.21.15.16:0/3701974133 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc26415a6e0 0x7fc26415e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:25.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.098+0000 7fc262ffd700 1 -- 172.21.15.16:0/3701974133 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc26415a6e0 msgr2=0x7fc26415e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:25.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.098+0000 7fc262ffd700 1 --2- 172.21.15.16:0/3701974133 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc26415a6e0 0x7fc26415e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:25.110 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.098+0000 7fc262ffd700 1 -- 172.21.15.16:0/3701974133 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc26415a300 msgr2=0x7fc2640aed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:25.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.098+0000 7fc262ffd700 1 --2- 172.21.15.16:0/3701974133 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc26415a300 0x7fc2640aed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:25.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.098+0000 7fc262ffd700 1 -- 172.21.15.16:0/3701974133 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc25c00b7f0 con 0x7fc2640b3ba0 2023-10-16T13:49:25.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.098+0000 7fc262ffd700 1 --2- 172.21.15.16:0/3701974133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc2640b3ba0 0x7fc264142c60 secure :-1 s=READY pgs=3279 cs=0 l=1 rev1=1 crypto rx=0x7fc254001ed0 tx=0x7fc2540021e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:25.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.098+0000 7fc25b7fe700 1 -- 172.21.15.16:0/3701974133 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc254003c00 con 0x7fc2640b3ba0 2023-10-16T13:49:25.111 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.098+0000 7fc25b7fe700 1 -- 172.21.15.16:0/3701974133 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc254003d60 con 0x7fc2640b3ba0 2023-10-16T13:49:25.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.098+0000 7fc26934a700 1 -- 172.21.15.16:0/3701974133 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2640afaa0 con 0x7fc2640b3ba0 2023-10-16T13:49:25.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.098+0000 7fc25b7fe700 1 -- 172.21.15.16:0/3701974133 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc25400ab40 con 0x7fc2640b3ba0 2023-10-16T13:49:25.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.098+0000 7fc26934a700 1 -- 172.21.15.16:0/3701974133 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2640b0060 con 0x7fc2640b3ba0 2023-10-16T13:49:25.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.099+0000 7fc26934a700 1 -- 172.21.15.16:0/3701974133 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc26404ebb0 con 0x7fc2640b3ba0 2023-10-16T13:49:25.112 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.100+0000 7fc25b7fe700 1 -- 172.21.15.16:0/3701974133 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 172) v1 ==== 89490+0+0 (secure 0 0 0) 0x7fc25400aca0 con 0x7fc2640b3ba0 2023-10-16T13:49:25.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.100+0000 7fc25b7fe700 1 --2- 172.21.15.16:0/3701974133 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7fc24406b1d0 0x7fc24406d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:25.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.100+0000 7fc2627fc700 1 -- 172.21.15.16:0/3701974133 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7fc24406b1d0 msgr2=0x7fc24406d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3920835901 2023-10-16T13:49:25.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.100+0000 7fc25b7fe700 1 -- 172.21.15.16:0/3701974133 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(279..279 src has 1..279) v4 ==== 6501+0+0 (secure 0 0 0) 0x7fc254089d20 con 0x7fc2640b3ba0 2023-10-16T13:49:25.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.100+0000 7fc2627fc700 1 --2- 172.21.15.16:0/3701974133 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7fc24406b1d0 0x7fc24406d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:49:25.113 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.105+0000 7fc25b7fe700 1 -- 172.21.15.16:0/3701974133 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc2540547f0 con 0x7fc2640b3ba0 2023-10-16T13:49:25.114 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:49:25.107+0000 7f8a615ff200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:49:25.173 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:25.170+0000 7f0548e60200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:49:25.191 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:49:25.188+0000 7f8a615ff200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T13:49:25.258 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:25.255+0000 7f0548e60200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T13:49:25.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.301+0000 7fc2627fc700 1 -- 172.21.15.16:0/3701974133 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7fc24406b1d0 msgr2=0x7fc24406d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3920835901 2023-10-16T13:49:25.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.301+0000 7fc2627fc700 1 --2- 172.21.15.16:0/3701974133 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7fc24406b1d0 0x7fc24406d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:49:25.314 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.312+0000 7fc26934a700 1 -- 172.21.15.16:0/3701974133 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc26415acc0 con 0x7fc2640b3ba0 2023-10-16T13:49:25.316 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.314+0000 7fc25b7fe700 1 -- 172.21.15.16:0/3701974133 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v279) v1 ==== 74+0+17324 (secure 0 0 0) 0x7fc254057e10 con 0x7fc2640b3ba0 2023-10-16T13:49:25.317 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:49:25.317 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":279,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:49:17.194563+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":38,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":37,"pool_name":"cephfs_metadata","create_time":"2023-10-16T13:49:00.609612+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"277","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":38,"pool_name":"cephfs_data","create_time":"2023-10-16T13:49:01.068705+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"278","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/750285561":"2023-10-17T13:49:17.194509+0000","172.21.15.16:6833/1257392887":"2023-10-17T13:49:17.194509+0000","172.21.15.16:6832/1257392887":"2023-10-17T13:49:17.194509+0000","172.21.15.16:0/1101859511":"2023-10-17T13:49:17.194509+0000","172.21.15.16:0/197265557":"2023-10-17T13:49:17.194509+0000","172.21.15.16:0/2024751970":"2023-10-17T13:49:17.194509+0000","172.21.15.16:0/2401946155":"2023-10-17T13:49:17.194509+0000","172.21.15.16:0/1156439592":"2023-10-17T13:49:17.194509+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:49:25.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.316+0000 7fc26934a700 1 -- 172.21.15.16:0/3701974133 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7fc24406b1d0 msgr2=0x7fc24406d690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:49:25.319 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.316+0000 7fc26934a700 1 --2- 172.21.15.16:0/3701974133 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7fc24406b1d0 0x7fc24406d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:25.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.316+0000 7fc26934a700 1 -- 172.21.15.16:0/3701974133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc2640b3ba0 msgr2=0x7fc264142c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:25.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.316+0000 7fc26934a700 1 --2- 172.21.15.16:0/3701974133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc2640b3ba0 0x7fc264142c60 secure :-1 s=READY pgs=3279 cs=0 l=1 rev1=1 crypto rx=0x7fc254001ed0 tx=0x7fc2540021e0 comp rx=0 tx=0).stop 2023-10-16T13:49:25.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.316+0000 7fc26934a700 1 -- 172.21.15.16:0/3701974133 shutdown_connections 2023-10-16T13:49:25.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.316+0000 7fc26934a700 1 --2- 172.21.15.16:0/3701974133 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7fc24406b1d0 0x7fc24406d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:25.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.316+0000 7fc26934a700 1 --2- 172.21.15.16:0/3701974133 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc26415a6e0 0x7fc26415e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:25.320 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.316+0000 7fc26934a700 1 --2- 172.21.15.16:0/3701974133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc2640b3ba0 0x7fc264142c60 unknown :-1 s=CLOSED pgs=3279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:25.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.316+0000 7fc26934a700 1 --2- 172.21.15.16:0/3701974133 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc26415a300 0x7fc2640aed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:25.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.316+0000 7fc26934a700 1 -- 172.21.15.16:0/3701974133 >> 172.21.15.16:0/3701974133 conn(0x7fc2640b4cd0 msgr2=0x7fc26406a7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:25.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.316+0000 7fc26934a700 1 -- 172.21.15.16:0/3701974133 shutdown_connections 2023-10-16T13:49:25.321 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.316+0000 7fc26934a700 1 -- 172.21.15.16:0/3701974133 wait complete. 2023-10-16T13:49:25.332 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T12:10:33.148942+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 37, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T13:49:00.609612+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '277', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 38, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T13:49:01.068705+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '278', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T13:49:25.332 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T13:49:25.343 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:49:25.341+0000 7f8a615ff200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:49:25.417 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:25.415+0000 7f0548e60200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T13:49:25.462 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.460+0000 7f02daf2e700 1 Processor -- start 2023-10-16T13:49:25.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.460+0000 7f02daf2e700 1 -- start start 2023-10-16T13:49:25.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.461+0000 7f02daf2e700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f02d415b070 0x7f02d415b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:25.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.461+0000 7f02daf2e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f02d40cb880 0x7f02d40cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:25.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.461+0000 7f02daf2e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f02d406da60 0x7f02d406def0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:25.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.461+0000 7f02daf2e700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02d406e430 con 0x7f02d40cb880 2023-10-16T13:49:25.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.461+0000 7f02daf2e700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02d40cef70 con 0x7f02d415b070 2023-10-16T13:49:25.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.461+0000 7f02daf2e700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02d40ce360 con 0x7f02d406da60 2023-10-16T13:49:25.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.461+0000 7f02d3fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f02d40cb880 0x7f02d40cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:25.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.461+0000 7f02d3fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f02d40cb880 0x7f02d40cbd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59998/0 (socket says 172.21.15.16:59998) 2023-10-16T13:49:25.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.461+0000 7f02d3fff700 1 -- 172.21.15.16:0/1084041730 learned_addr learned my addr 172.21.15.16:0/1084041730 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:25.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.461+0000 7f02d8cca700 1 --2- 172.21.15.16:0/1084041730 >> v2:172.21.15.49:3300/0 conn(0x7f02d415b070 0x7f02d415b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:25.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.461+0000 7f02d94cb700 1 --2- 172.21.15.16:0/1084041730 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f02d406da60 0x7f02d406def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:25.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.461+0000 7f02d3fff700 1 -- 172.21.15.16:0/1084041730 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f02d406da60 msgr2=0x7f02d406def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:25.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.461+0000 7f02d3fff700 1 --2- 172.21.15.16:0/1084041730 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f02d406da60 0x7f02d406def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:25.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.461+0000 7f02d3fff700 1 -- 172.21.15.16:0/1084041730 >> v2:172.21.15.49:3300/0 conn(0x7f02d415b070 msgr2=0x7f02d415b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:25.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.461+0000 7f02d3fff700 1 --2- 172.21.15.16:0/1084041730 >> v2:172.21.15.49:3300/0 conn(0x7f02d415b070 0x7f02d415b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:25.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.461+0000 7f02d3fff700 1 -- 172.21.15.16:0/1084041730 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02d40ce640 con 0x7f02d40cb880 2023-10-16T13:49:25.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.461+0000 7f02d3fff700 1 --2- 172.21.15.16:0/1084041730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02d40cb880 0x7f02d40cbd00 secure :-1 s=READY pgs=3280 cs=0 l=1 rev1=1 crypto rx=0x7f02d40705a0 tx=0x7f02c0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d9104aba150723c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:25.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.461+0000 7f02d37fe700 1 -- 172.21.15.16:0/1084041730 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c0011830 con 0x7f02d40cb880 2023-10-16T13:49:25.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.461+0000 7f02d37fe700 1 -- 172.21.15.16:0/1084041730 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02c0011990 con 0x7f02d40cb880 2023-10-16T13:49:25.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.462+0000 7f02d37fe700 1 -- 172.21.15.16:0/1084041730 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c0011d20 con 0x7f02d40cb880 2023-10-16T13:49:25.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.462+0000 7f02daf2e700 1 -- 172.21.15.16:0/1084041730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02d40cb880 msgr2=0x7f02d40cbd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:25.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.462+0000 7f02daf2e700 1 --2- 172.21.15.16:0/1084041730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02d40cb880 0x7f02d40cbd00 secure :-1 s=READY pgs=3280 cs=0 l=1 rev1=1 crypto rx=0x7f02d40705a0 tx=0x7f02c0009bd0 comp rx=0 tx=0).stop 2023-10-16T13:49:25.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.462+0000 7f02daf2e700 1 -- 172.21.15.16:0/1084041730 shutdown_connections 2023-10-16T13:49:25.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.462+0000 7f02daf2e700 1 --2- 172.21.15.16:0/1084041730 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f02d406da60 0x7f02d406def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:25.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.462+0000 7f02daf2e700 1 --2- 172.21.15.16:0/1084041730 >> v2:172.21.15.49:3300/0 conn(0x7f02d415b070 0x7f02d415b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:25.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.462+0000 7f02daf2e700 1 --2- 172.21.15.16:0/1084041730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02d40cb880 0x7f02d40cbd00 unknown :-1 s=CLOSED pgs=3280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:25.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.462+0000 7f02daf2e700 1 -- 172.21.15.16:0/1084041730 >> 172.21.15.16:0/1084041730 conn(0x7f02d40b1f40 msgr2=0x7f02d40b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:25.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.462+0000 7f02daf2e700 1 -- 172.21.15.16:0/1084041730 shutdown_connections 2023-10-16T13:49:25.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.462+0000 7f02daf2e700 1 -- 172.21.15.16:0/1084041730 wait complete. 2023-10-16T13:49:25.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.464+0000 7f02daf2e700 1 Processor -- start 2023-10-16T13:49:25.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.464+0000 7f02daf2e700 1 -- start start 2023-10-16T13:49:25.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.466+0000 7f02daf2e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f02d406da60 0x7f02d40b1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:25.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.466+0000 7f02daf2e700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f02d40cb880 0x7f02d4155dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:25.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.466+0000 7f02daf2e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02d415b070 0x7f02d4147f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:25.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.466+0000 7f02daf2e700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02d40b58a0 con 0x7f02d415b070 2023-10-16T13:49:25.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.466+0000 7f02daf2e700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02d4155000 con 0x7f02d40cb880 2023-10-16T13:49:25.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.466+0000 7f02daf2e700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02d41552e0 con 0x7f02d406da60 2023-10-16T13:49:25.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.466+0000 7f02d94cb700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02d415b070 0x7f02d4147f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:25.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.466+0000 7f02d94cb700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02d415b070 0x7f02d4147f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60006/0 (socket says 172.21.15.16:60006) 2023-10-16T13:49:25.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.466+0000 7f02d94cb700 1 -- 172.21.15.16:0/2574492333 learned_addr learned my addr 172.21.15.16:0/2574492333 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:25.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.466+0000 7f02d8cca700 1 --2- 172.21.15.16:0/2574492333 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f02d406da60 0x7f02d40b1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:25.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.466+0000 7f02d3fff700 1 --2- 172.21.15.16:0/2574492333 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f02d40cb880 0x7f02d4155dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:25.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.466+0000 7f02d94cb700 1 -- 172.21.15.16:0/2574492333 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f02d406da60 msgr2=0x7f02d40b1250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:25.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.466+0000 7f02d94cb700 1 --2- 172.21.15.16:0/2574492333 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f02d406da60 0x7f02d40b1250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:25.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.466+0000 7f02d94cb700 1 -- 172.21.15.16:0/2574492333 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f02d40cb880 msgr2=0x7f02d4155dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:25.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.466+0000 7f02d94cb700 1 --2- 172.21.15.16:0/2574492333 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f02d40cb880 0x7f02d4155dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:25.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.466+0000 7f02d94cb700 1 -- 172.21.15.16:0/2574492333 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02c0009550 con 0x7f02d415b070 2023-10-16T13:49:25.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.466+0000 7f02d94cb700 1 --2- 172.21.15.16:0/2574492333 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02d415b070 0x7f02d4147f20 secure :-1 s=READY pgs=3281 cs=0 l=1 rev1=1 crypto rx=0x7f02cc010ef0 tx=0x7f02cc00e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:25.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.466+0000 7f02d17fa700 1 -- 172.21.15.16:0/2574492333 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02cc009c40 con 0x7f02d415b070 2023-10-16T13:49:25.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.466+0000 7f02d17fa700 1 -- 172.21.15.16:0/2574492333 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02cc013450 con 0x7f02d415b070 2023-10-16T13:49:25.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.466+0000 7f02daf2e700 1 -- 172.21.15.16:0/2574492333 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02d41555c0 con 0x7f02d415b070 2023-10-16T13:49:25.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.466+0000 7f02d17fa700 1 -- 172.21.15.16:0/2574492333 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02cc00eee0 con 0x7f02d415b070 2023-10-16T13:49:25.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.467+0000 7f02daf2e700 1 -- 172.21.15.16:0/2574492333 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02d40b1790 con 0x7f02d415b070 2023-10-16T13:49:25.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.468+0000 7f02daf2e700 1 -- 172.21.15.16:0/2574492333 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02d4155ae0 con 0x7f02d415b070 2023-10-16T13:49:25.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.468+0000 7f02d17fa700 1 -- 172.21.15.16:0/2574492333 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 172) v1 ==== 89490+0+0 (secure 0 0 0) 0x7f02cc013800 con 0x7f02d415b070 2023-10-16T13:49:25.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.468+0000 7f02d17fa700 1 --2- 172.21.15.16:0/2574492333 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f02b806b1d0 0x7f02b806d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:25.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.468+0000 7f02d8cca700 1 -- 172.21.15.16:0/2574492333 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f02b806b1d0 msgr2=0x7f02b806d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3920835901 2023-10-16T13:49:25.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.468+0000 7f02d17fa700 1 -- 172.21.15.16:0/2574492333 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(279..279 src has 1..279) v4 ==== 6501+0+0 (secure 0 0 0) 0x7f02cc08cad0 con 0x7f02d415b070 2023-10-16T13:49:25.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.468+0000 7f02d8cca700 1 --2- 172.21.15.16:0/2574492333 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f02b806b1d0 0x7f02b806d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:49:25.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.473+0000 7f02d17fa700 1 -- 172.21.15.16:0/2574492333 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f02d4155ae0 con 0x7f02d415b070 2023-10-16T13:49:25.581 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:49:25.579+0000 7f8a615ff200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T13:49:25.658 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:49:25.656+0000 7f8a615ff200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T13:49:25.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.669+0000 7f02d8cca700 1 -- 172.21.15.16:0/2574492333 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f02b806b1d0 msgr2=0x7f02b806d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3920835901 2023-10-16T13:49:25.671 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.669+0000 7f02d8cca700 1 --2- 172.21.15.16:0/2574492333 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f02b806b1d0 0x7f02b806d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:49:25.672 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:25.669+0000 7f0548e60200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T13:49:25.690 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.688+0000 7f02daf2e700 1 -- 172.21.15.16:0/2574492333 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f02d4155ae0 con 0x7f02d415b070 2023-10-16T13:49:25.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.688+0000 7f02d17fa700 1 -- 172.21.15.16:0/2574492333 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v279) v1 ==== 93+0+10 (secure 0 0 0) 0x7f02d4155ae0 con 0x7f02d415b070 2023-10-16T13:49:25.691 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2023-10-16T13:49:25.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.691+0000 7f02daf2e700 1 -- 172.21.15.16:0/2574492333 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f02b806b1d0 msgr2=0x7f02b806d690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:49:25.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.691+0000 7f02daf2e700 1 --2- 172.21.15.16:0/2574492333 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f02b806b1d0 0x7f02b806d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:25.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.691+0000 7f02daf2e700 1 -- 172.21.15.16:0/2574492333 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02d415b070 msgr2=0x7f02d4147f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:25.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.691+0000 7f02daf2e700 1 --2- 172.21.15.16:0/2574492333 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02d415b070 0x7f02d4147f20 secure :-1 s=READY pgs=3281 cs=0 l=1 rev1=1 crypto rx=0x7f02cc010ef0 tx=0x7f02cc00e530 comp rx=0 tx=0).stop 2023-10-16T13:49:25.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.691+0000 7f02daf2e700 1 -- 172.21.15.16:0/2574492333 shutdown_connections 2023-10-16T13:49:25.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.691+0000 7f02daf2e700 1 --2- 172.21.15.16:0/2574492333 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f02b806b1d0 0x7f02b806d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:25.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.691+0000 7f02daf2e700 1 --2- 172.21.15.16:0/2574492333 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02d415b070 0x7f02d4147f20 unknown :-1 s=CLOSED pgs=3281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:25.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.691+0000 7f02daf2e700 1 --2- 172.21.15.16:0/2574492333 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f02d406da60 0x7f02d40b1250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:25.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.691+0000 7f02daf2e700 1 --2- 172.21.15.16:0/2574492333 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f02d40cb880 0x7f02d4155dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:25.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.691+0000 7f02daf2e700 1 -- 172.21.15.16:0/2574492333 >> 172.21.15.16:0/2574492333 conn(0x7f02d40b1f40 msgr2=0x7f02d41586f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:25.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.691+0000 7f02daf2e700 1 -- 172.21.15.16:0/2574492333 shutdown_connections 2023-10-16T13:49:25.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.691+0000 7f02daf2e700 1 -- 172.21.15.16:0/2574492333 wait complete. 2023-10-16T13:49:25.706 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T13:49:25.757 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:25.754+0000 7f0548e60200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T13:49:25.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.836+0000 7f76962e7700 1 Processor -- start 2023-10-16T13:49:25.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.836+0000 7f76962e7700 1 -- start start 2023-10-16T13:49:25.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.837+0000 7f76962e7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f769015ec80 0x7f76900cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:25.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.837+0000 7f76962e7700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f76900d2110 0x7f76900cd1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:25.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.837+0000 7f76962e7700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76900b2440 con 0x7f769015f060 2023-10-16T13:49:25.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.837+0000 7f76962e7700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76900b2c50 con 0x7f769015ec80 2023-10-16T13:49:25.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.837+0000 7f76962e7700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76900d00e0 con 0x7f76900d2110 2023-10-16T13:49:25.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.837+0000 7f768ffff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f769015ec80 0x7f76900cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:25.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.837+0000 7f768ffff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f769015ec80 0x7f76900cdec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60012/0 (socket says 172.21.15.16:60012) 2023-10-16T13:49:25.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.837+0000 7f768f7fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f769015f060 0x7f76900b4440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49286/0 (socket says 172.21.15.16:49286) 2023-10-16T13:49:25.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.837+0000 7f768ffff700 1 -- 172.21.15.16:0/3028747419 learned_addr learned my addr 172.21.15.16:0/3028747419 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:25.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.837+0000 7f7694884700 1 --2- 172.21.15.16:0/3028747419 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f76900d2110 0x7f76900cd1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:25.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.837+0000 7f768ffff700 1 -- 172.21.15.16:0/3028747419 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f76900d2110 msgr2=0x7f76900cd1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:25.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.837+0000 7f768ffff700 1 --2- 172.21.15.16:0/3028747419 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f76900d2110 0x7f76900cd1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:25.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.837+0000 7f768ffff700 1 -- 172.21.15.16:0/3028747419 >> v1:172.21.15.16:6789/0 conn(0x7f769015f060 legacy=0x7f76900b4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:25.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.837+0000 7f768ffff700 1 -- 172.21.15.16:0/3028747419 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76900cef60 con 0x7f769015ec80 2023-10-16T13:49:25.843 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.837+0000 7f768effd700 1 -- 172.21.15.16:0/3028747419 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 265554476 0 0) 0x7f76900b2440 con 0x7f769015f060 2023-10-16T13:49:25.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.838+0000 7f768ffff700 1 --2- 172.21.15.16:0/3028747419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f769015ec80 0x7f76900cdec0 secure :-1 s=READY pgs=3283 cs=0 l=1 rev1=1 crypto rx=0x7f76800098f0 tx=0x7f7680009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73fcdce26c218031 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:25.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.838+0000 7f768effd700 1 -- 172.21.15.16:0/3028747419 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7680011830 con 0x7f769015ec80 2023-10-16T13:49:25.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.838+0000 7f768effd700 1 -- 172.21.15.16:0/3028747419 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7680011990 con 0x7f769015ec80 2023-10-16T13:49:25.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.838+0000 7f768effd700 1 -- 172.21.15.16:0/3028747419 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7680011d20 con 0x7f769015ec80 2023-10-16T13:49:25.847 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.839+0000 7f76962e7700 1 -- 172.21.15.16:0/3028747419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f769015ec80 msgr2=0x7f76900cdec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:25.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.839+0000 7f76962e7700 1 --2- 172.21.15.16:0/3028747419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f769015ec80 0x7f76900cdec0 secure :-1 s=READY pgs=3283 cs=0 l=1 rev1=1 crypto rx=0x7f76800098f0 tx=0x7f7680009c00 comp rx=0 tx=0).stop 2023-10-16T13:49:25.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.839+0000 7f76962e7700 1 -- 172.21.15.16:0/3028747419 shutdown_connections 2023-10-16T13:49:25.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.839+0000 7f76962e7700 1 --2- 172.21.15.16:0/3028747419 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f76900d2110 0x7f76900cd1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:25.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.839+0000 7f76962e7700 1 --2- 172.21.15.16:0/3028747419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f769015ec80 0x7f76900cdec0 unknown :-1 s=CLOSED pgs=3283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:25.848 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.839+0000 7f76962e7700 1 -- 172.21.15.16:0/3028747419 >> 172.21.15.16:0/3028747419 conn(0x7f76900d15f0 msgr2=0x7f76900d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:25.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.839+0000 7f76962e7700 1 -- 172.21.15.16:0/3028747419 shutdown_connections 2023-10-16T13:49:25.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.839+0000 7f76962e7700 1 -- 172.21.15.16:0/3028747419 wait complete. 2023-10-16T13:49:25.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.840+0000 7f76962e7700 1 Processor -- start 2023-10-16T13:49:25.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.840+0000 7f76962e7700 1 -- start start 2023-10-16T13:49:25.849 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.841+0000 7f76962e7700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f76900d2110 0x7f7690151b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:25.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.841+0000 7f76962e7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f769015ec80 0x7f76900c9d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:25.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.841+0000 7f76962e7700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f769015f060 0x7f76900b6780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:25.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.841+0000 7f76962e7700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7690159580 con 0x7f769015ec80 2023-10-16T13:49:25.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.841+0000 7f76962e7700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76901520d0 con 0x7f76900d2110 2023-10-16T13:49:25.850 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.841+0000 7f76962e7700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76901f7360 con 0x7f769015f060 2023-10-16T13:49:25.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.841+0000 7f768f7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f769015ec80 0x7f76900c9d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:25.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.841+0000 7f768ffff700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f76900d2110 0x7f7690151b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:25.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.841+0000 7f768f7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f769015ec80 0x7f76900c9d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60020/0 (socket says 172.21.15.16:60020) 2023-10-16T13:49:25.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.841+0000 7f768f7fe700 1 -- 172.21.15.16:0/2296114775 learned_addr learned my addr 172.21.15.16:0/2296114775 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:25.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.841+0000 7f7694884700 1 --2- 172.21.15.16:0/2296114775 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f769015f060 0x7f76900b6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:25.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.842+0000 7f7694884700 1 -- 172.21.15.16:0/2296114775 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f76900d2110 msgr2=0x7f7690151b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:25.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.842+0000 7f7694884700 1 --2- 172.21.15.16:0/2296114775 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f76900d2110 0x7f7690151b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:25.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.842+0000 7f7694884700 1 -- 172.21.15.16:0/2296114775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f769015ec80 msgr2=0x7f76900c9d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:25.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.842+0000 7f7694884700 1 --2- 172.21.15.16:0/2296114775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f769015ec80 0x7f76900c9d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:25.852 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.842+0000 7f7694884700 1 -- 172.21.15.16:0/2296114775 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7680009550 con 0x7f769015f060 2023-10-16T13:49:25.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.842+0000 7f768f7fe700 1 --2- 172.21.15.16:0/2296114775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f769015ec80 0x7f76900c9d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:49:25.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.842+0000 7f768ffff700 1 --2- 172.21.15.16:0/2296114775 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f76900d2110 0x7f7690151b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:49:25.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.842+0000 7f7694884700 1 --2- 172.21.15.16:0/2296114775 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f769015f060 0x7f76900b6780 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f7688009d90 tx=0x7f768800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:25.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.842+0000 7f768cff9700 1 -- 172.21.15.16:0/2296114775 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7688009c00 con 0x7f769015f060 2023-10-16T13:49:25.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.842+0000 7f768cff9700 1 -- 172.21.15.16:0/2296114775 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f768800f040 con 0x7f769015f060 2023-10-16T13:49:25.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.842+0000 7f76962e7700 1 -- 172.21.15.16:0/2296114775 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7690158ec0 con 0x7f769015f060 2023-10-16T13:49:25.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.842+0000 7f768cff9700 1 -- 172.21.15.16:0/2296114775 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7688014480 con 0x7f769015f060 2023-10-16T13:49:25.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.842+0000 7f76962e7700 1 -- 172.21.15.16:0/2296114775 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76900ca280 con 0x7f769015f060 2023-10-16T13:49:25.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.843+0000 7f76962e7700 1 -- 172.21.15.16:0/2296114775 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f769004ebb0 con 0x7f769015f060 2023-10-16T13:49:25.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.844+0000 7f768cff9700 1 -- 172.21.15.16:0/2296114775 <== mon.2 v2:172.21.15.49:3301/0 4 ==== mgrmap(e 172) v1 ==== 89490+0+0 (secure 0 0 0) 0x7f76880145e0 con 0x7f769015f060 2023-10-16T13:49:25.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.844+0000 7f768cff9700 1 --2- 172.21.15.16:0/2296114775 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f767c06b1d0 0x7f767c06d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:25.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.844+0000 7f768ffff700 1 -- 172.21.15.16:0/2296114775 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f767c06b1d0 msgr2=0x7f767c06d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3920835901 2023-10-16T13:49:25.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.844+0000 7f768cff9700 1 -- 172.21.15.16:0/2296114775 <== mon.2 v2:172.21.15.49:3301/0 5 ==== osd_map(279..279 src has 1..279) v4 ==== 6501+0+0 (secure 0 0 0) 0x7f768808a9c0 con 0x7f769015f060 2023-10-16T13:49:25.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.844+0000 7f768ffff700 1 --2- 172.21.15.16:0/2296114775 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f767c06b1d0 0x7f767c06d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:49:25.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:25.849+0000 7f768cff9700 1 -- 172.21.15.16:0/2296114775 <== mon.2 v2:172.21.15.49:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f76880552f0 con 0x7f769015f060 2023-10-16T13:49:25.936 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:49:25.934+0000 7f8a615ff200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T13:49:26.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.045+0000 7f768ffff700 1 -- 172.21.15.16:0/2296114775 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f767c06b1d0 msgr2=0x7f767c06d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3920835901 2023-10-16T13:49:26.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.045+0000 7f768ffff700 1 --2- 172.21.15.16:0/2296114775 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f767c06b1d0 0x7f767c06d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:49:26.054 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:26.052+0000 7f0548e60200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T13:49:26.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.073+0000 7f76962e7700 1 -- 172.21.15.16:0/2296114775 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f76900c8480 con 0x7f769015f060 2023-10-16T13:49:26.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.074+0000 7f768cff9700 1 -- 172.21.15.16:0/2296114775 <== mon.2 v2:172.21.15.49:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v279) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7688018020 con 0x7f769015f060 2023-10-16T13:49:26.076 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2023-10-16T13:49:26.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.076+0000 7f76962e7700 1 -- 172.21.15.16:0/2296114775 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f767c06b1d0 msgr2=0x7f767c06d690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:49:26.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.076+0000 7f76962e7700 1 --2- 172.21.15.16:0/2296114775 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f767c06b1d0 0x7f767c06d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:26.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.076+0000 7f76962e7700 1 -- 172.21.15.16:0/2296114775 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f769015f060 msgr2=0x7f76900b6780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:26.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.076+0000 7f76962e7700 1 --2- 172.21.15.16:0/2296114775 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f769015f060 0x7f76900b6780 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f7688009d90 tx=0x7f768800c3a0 comp rx=0 tx=0).stop 2023-10-16T13:49:26.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.076+0000 7f76962e7700 1 -- 172.21.15.16:0/2296114775 shutdown_connections 2023-10-16T13:49:26.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.076+0000 7f76962e7700 1 --2- 172.21.15.16:0/2296114775 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f767c06b1d0 0x7f767c06d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:26.080 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.076+0000 7f76962e7700 1 --2- 172.21.15.16:0/2296114775 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f769015f060 0x7f76900b6780 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:26.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.076+0000 7f76962e7700 1 --2- 172.21.15.16:0/2296114775 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f76900d2110 0x7f7690151b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:26.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.076+0000 7f76962e7700 1 --2- 172.21.15.16:0/2296114775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f769015ec80 0x7f76900c9d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:26.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.076+0000 7f76962e7700 1 -- 172.21.15.16:0/2296114775 >> 172.21.15.16:0/2296114775 conn(0x7f76900d15f0 msgr2=0x7f769015c810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:26.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.076+0000 7f76962e7700 1 -- 172.21.15.16:0/2296114775 shutdown_connections 2023-10-16T13:49:26.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.076+0000 7f76962e7700 1 -- 172.21.15.16:0/2296114775 wait complete. 2023-10-16T13:49:26.091 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T13:49:26.169 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:49:26.166+0000 7f8a615ff200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T13:49:26.231 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.229+0000 7fc4c9150700 1 Processor -- start 2023-10-16T13:49:26.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.229+0000 7fc4c9150700 1 -- start start 2023-10-16T13:49:26.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.230+0000 7fc4c9150700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fc4c415b2c0 0x7fc4c4161340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:26.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.230+0000 7fc4c9150700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc4c406a7e0 0x7fc4c406ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:26.235 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.230+0000 7fc4c9150700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4c415ecb0 con 0x7fc4c406e570 2023-10-16T13:49:26.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.230+0000 7fc4c9150700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4c406b190 con 0x7fc4c406a7e0 2023-10-16T13:49:26.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.230+0000 7fc4c9150700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4c4161880 con 0x7fc4c415b2c0 2023-10-16T13:49:26.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.230+0000 7fc4c359e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc4c406a7e0 0x7fc4c406ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:26.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.230+0000 7fc4c359e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc4c406a7e0 0x7fc4c406ac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60032/0 (socket says 172.21.15.16:60032) 2023-10-16T13:49:26.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.230+0000 7fc4c359e700 1 -- 172.21.15.16:0/3622451964 learned_addr learned my addr 172.21.15.16:0/3622451964 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:26.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.230+0000 7fc4c2d9d700 1 --2- 172.21.15.16:0/3622451964 >> v2:172.21.15.49:3300/0 conn(0x7fc4c415b2c0 0x7fc4c4161340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:26.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.230+0000 7fc4c359e700 1 -- 172.21.15.16:0/3622451964 >> v2:172.21.15.49:3300/0 conn(0x7fc4c415b2c0 msgr2=0x7fc4c4161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:26.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.230+0000 7fc4c359e700 1 --2- 172.21.15.16:0/3622451964 >> v2:172.21.15.49:3300/0 conn(0x7fc4c415b2c0 0x7fc4c4161340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:26.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.230+0000 7fc4c359e700 1 -- 172.21.15.16:0/3622451964 >> v1:172.21.15.49:6789/0 conn(0x7fc4c406e570 legacy=0x7fc4c415e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:26.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.230+0000 7fc4c1d9b700 1 -- 172.21.15.16:0/3622451964 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3664748096 0 0) 0x7fc4c415ecb0 con 0x7fc4c406e570 2023-10-16T13:49:26.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.230+0000 7fc4c359e700 1 -- 172.21.15.16:0/3622451964 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4c415b6a0 con 0x7fc4c406a7e0 2023-10-16T13:49:26.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.230+0000 7fc4c359e700 1 --2- 172.21.15.16:0/3622451964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4c406a7e0 0x7fc4c406ac50 secure :-1 s=READY pgs=3284 cs=0 l=1 rev1=1 crypto rx=0x7fc4c406b4a0 tx=0x7fc4bc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3b9eaa216d17e5f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:26.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.231+0000 7fc4c1d9b700 1 -- 172.21.15.16:0/3622451964 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4bc00e070 con 0x7fc4c406a7e0 2023-10-16T13:49:26.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.231+0000 7fc4c1d9b700 1 -- 172.21.15.16:0/3622451964 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4bc00b910 con 0x7fc4c406a7e0 2023-10-16T13:49:26.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.231+0000 7fc4c1d9b700 1 -- 172.21.15.16:0/3622451964 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4bc00bcc0 con 0x7fc4c406a7e0 2023-10-16T13:49:26.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.231+0000 7fc4c9150700 1 -- 172.21.15.16:0/3622451964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4c406a7e0 msgr2=0x7fc4c406ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:26.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.231+0000 7fc4c9150700 1 --2- 172.21.15.16:0/3622451964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4c406a7e0 0x7fc4c406ac50 secure :-1 s=READY pgs=3284 cs=0 l=1 rev1=1 crypto rx=0x7fc4c406b4a0 tx=0x7fc4bc00b330 comp rx=0 tx=0).stop 2023-10-16T13:49:26.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.231+0000 7fc4c9150700 1 -- 172.21.15.16:0/3622451964 shutdown_connections 2023-10-16T13:49:26.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.231+0000 7fc4c9150700 1 --2- 172.21.15.16:0/3622451964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4c406a7e0 0x7fc4c406ac50 unknown :-1 s=CLOSED pgs=3284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:26.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.231+0000 7fc4c9150700 1 --2- 172.21.15.16:0/3622451964 >> v2:172.21.15.49:3300/0 conn(0x7fc4c415b2c0 0x7fc4c4161340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:26.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.231+0000 7fc4c9150700 1 -- 172.21.15.16:0/3622451964 >> 172.21.15.16:0/3622451964 conn(0x7fc4c40b1f40 msgr2=0x7fc4c40b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:26.240 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.231+0000 7fc4c9150700 1 -- 172.21.15.16:0/3622451964 shutdown_connections 2023-10-16T13:49:26.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.231+0000 7fc4c9150700 1 -- 172.21.15.16:0/3622451964 wait complete. 2023-10-16T13:49:26.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.233+0000 7fc4c9150700 1 Processor -- start 2023-10-16T13:49:26.241 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.233+0000 7fc4c9150700 1 -- start start 2023-10-16T13:49:26.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.233+0000 7fc4c9150700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4c406a7e0 0x7fc4c40c6200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:26.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.233+0000 7fc4c9150700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc4c406e570 0x7fc4c41f29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:26.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.233+0000 7fc4c9150700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc4c415b2c0 0x7fc4c41f6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:26.244 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.233+0000 7fc4c9150700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c41f7430 con 0x7fc4c406a7e0 2023-10-16T13:49:26.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.233+0000 7fc4c9150700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c40c5160 con 0x7fc4c406e570 2023-10-16T13:49:26.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.233+0000 7fc4c9150700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c40c5440 con 0x7fc4c415b2c0 2023-10-16T13:49:26.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.234+0000 7fc4c2d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4c406a7e0 0x7fc4c40c6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:26.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.234+0000 7fc4c2d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4c406a7e0 0x7fc4c40c6200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60034/0 (socket says 172.21.15.16:60034) 2023-10-16T13:49:26.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.234+0000 7fc4c2d9d700 1 -- 172.21.15.16:0/1896314530 learned_addr learned my addr 172.21.15.16:0/1896314530 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:26.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.234+0000 7fc4c259c700 1 --2- 172.21.15.16:0/1896314530 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc4c406e570 0x7fc4c41f29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:26.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.234+0000 7fc4c359e700 1 --2- 172.21.15.16:0/1896314530 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc4c415b2c0 0x7fc4c41f6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:26.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.234+0000 7fc4c2d9d700 1 -- 172.21.15.16:0/1896314530 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc4c415b2c0 msgr2=0x7fc4c41f6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:26.246 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.234+0000 7fc4c2d9d700 1 --2- 172.21.15.16:0/1896314530 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc4c415b2c0 0x7fc4c41f6ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:26.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.234+0000 7fc4c2d9d700 1 -- 172.21.15.16:0/1896314530 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc4c406e570 msgr2=0x7fc4c41f29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:26.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.234+0000 7fc4c2d9d700 1 --2- 172.21.15.16:0/1896314530 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc4c406e570 0x7fc4c41f29e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:26.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.234+0000 7fc4c2d9d700 1 -- 172.21.15.16:0/1896314530 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4bc009b30 con 0x7fc4c406a7e0 2023-10-16T13:49:26.247 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.234+0000 7fc4c2d9d700 1 --2- 172.21.15.16:0/1896314530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4c406a7e0 0x7fc4c40c6200 secure :-1 s=READY pgs=3285 cs=0 l=1 rev1=1 crypto rx=0x7fc4b400e870 tx=0x7fc4b400eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:26.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.234+0000 7fc4ab7fe700 1 -- 172.21.15.16:0/1896314530 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b4009590 con 0x7fc4c406a7e0 2023-10-16T13:49:26.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.234+0000 7fc4ab7fe700 1 -- 172.21.15.16:0/1896314530 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4b4013680 con 0x7fc4c406a7e0 2023-10-16T13:49:26.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.234+0000 7fc4c9150700 1 -- 172.21.15.16:0/1896314530 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4c40c5720 con 0x7fc4c406a7e0 2023-10-16T13:49:26.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.234+0000 7fc4ab7fe700 1 -- 172.21.15.16:0/1896314530 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b40139f0 con 0x7fc4c406a7e0 2023-10-16T13:49:26.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.234+0000 7fc4c9150700 1 -- 172.21.15.16:0/1896314530 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4c40c5ce0 con 0x7fc4c406a7e0 2023-10-16T13:49:26.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.235+0000 7fc4c9150700 1 -- 172.21.15.16:0/1896314530 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4c404ebb0 con 0x7fc4c406a7e0 2023-10-16T13:49:26.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.236+0000 7fc4ab7fe700 1 -- 172.21.15.16:0/1896314530 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 172) v1 ==== 89490+0+0 (secure 0 0 0) 0x7fc4b4013b50 con 0x7fc4c406a7e0 2023-10-16T13:49:26.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.236+0000 7fc4ab7fe700 1 --2- 172.21.15.16:0/1896314530 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7fc4b006b1d0 0x7fc4b006d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:26.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.236+0000 7fc4c259c700 1 -- 172.21.15.16:0/1896314530 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7fc4b006b1d0 msgr2=0x7fc4b006d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3920835901 2023-10-16T13:49:26.249 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.236+0000 7fc4ab7fe700 1 -- 172.21.15.16:0/1896314530 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(279..279 src has 1..279) v4 ==== 6501+0+0 (secure 0 0 0) 0x7fc4b4089b40 con 0x7fc4c406a7e0 2023-10-16T13:49:26.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.237+0000 7fc4c259c700 1 --2- 172.21.15.16:0/1896314530 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7fc4b006b1d0 0x7fc4b006d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:49:26.250 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.245+0000 7fc4ab7fe700 1 -- 172.21.15.16:0/1896314530 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc4b4054610 con 0x7fc4c406a7e0 2023-10-16T13:49:26.312 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:26.310+0000 7f0548e60200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T13:49:26.326 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:49:26.323+0000 7f8a615ff200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T13:49:26.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.438+0000 7fc4c259c700 1 -- 172.21.15.16:0/1896314530 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7fc4b006b1d0 msgr2=0x7fc4b006d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3920835901 2023-10-16T13:49:26.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.438+0000 7fc4c259c700 1 --2- 172.21.15.16:0/1896314530 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7fc4b006b1d0 0x7fc4b006d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:49:26.448 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:49:26.445+0000 7f8a615ff200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T13:49:26.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.453+0000 7fc4c9150700 1 -- 172.21.15.16:0/1896314530 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc4c406d580 con 0x7fc4c406a7e0 2023-10-16T13:49:26.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.453+0000 7fc4ab7fe700 1 -- 172.21.15.16:0/1896314530 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v279) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc4b40210a0 con 0x7fc4c406a7e0 2023-10-16T13:49:26.456 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2023-10-16T13:49:26.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.455+0000 7fc4c9150700 1 -- 172.21.15.16:0/1896314530 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7fc4b006b1d0 msgr2=0x7fc4b006d690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:49:26.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.455+0000 7fc4c9150700 1 --2- 172.21.15.16:0/1896314530 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7fc4b006b1d0 0x7fc4b006d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:26.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.455+0000 7fc4c9150700 1 -- 172.21.15.16:0/1896314530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4c406a7e0 msgr2=0x7fc4c40c6200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:26.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.455+0000 7fc4c9150700 1 --2- 172.21.15.16:0/1896314530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4c406a7e0 0x7fc4c40c6200 secure :-1 s=READY pgs=3285 cs=0 l=1 rev1=1 crypto rx=0x7fc4b400e870 tx=0x7fc4b400eb80 comp rx=0 tx=0).stop 2023-10-16T13:49:26.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.455+0000 7fc4c9150700 1 -- 172.21.15.16:0/1896314530 shutdown_connections 2023-10-16T13:49:26.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.455+0000 7fc4c9150700 1 --2- 172.21.15.16:0/1896314530 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7fc4b006b1d0 0x7fc4b006d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:26.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.455+0000 7fc4c9150700 1 --2- 172.21.15.16:0/1896314530 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc4c415b2c0 0x7fc4c41f6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:26.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.455+0000 7fc4c9150700 1 --2- 172.21.15.16:0/1896314530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4c406a7e0 0x7fc4c40c6200 unknown :-1 s=CLOSED pgs=3285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:26.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.455+0000 7fc4c9150700 1 --2- 172.21.15.16:0/1896314530 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc4c406e570 0x7fc4c41f29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:26.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.455+0000 7fc4c9150700 1 -- 172.21.15.16:0/1896314530 >> 172.21.15.16:0/1896314530 conn(0x7fc4c40b1f40 msgr2=0x7fc4c40cc070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:26.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.456+0000 7fc4c9150700 1 -- 172.21.15.16:0/1896314530 shutdown_connections 2023-10-16T13:49:26.461 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.456+0000 7fc4c9150700 1 -- 172.21.15.16:0/1896314530 wait complete. 2023-10-16T13:49:26.471 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:49:26.476 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:26.473+0000 7f0548e60200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T13:49:26.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.596+0000 7f21591a3700 1 Processor -- start 2023-10-16T13:49:26.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.596+0000 7f21591a3700 1 -- start start 2023-10-16T13:49:26.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.597+0000 7f21591a3700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f215415b630 0x7f21540d0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:26.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.597+0000 7f21591a3700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f215415ba10 0x7f21540cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:26.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.597+0000 7f21591a3700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21540b0970 con 0x7f21540cdde0 2023-10-16T13:49:26.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.597+0000 7f21591a3700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21540cd690 con 0x7f215415ba10 2023-10-16T13:49:26.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.597+0000 7f21591a3700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21540cd970 con 0x7f215415b630 2023-10-16T13:49:26.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.597+0000 7f215259c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f215415ba10 0x7f21540cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:26.602 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.597+0000 7f215259c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f215415ba10 0x7f21540cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60046/0 (socket says 172.21.15.16:60046) 2023-10-16T13:49:26.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.597+0000 7f215259c700 1 -- 172.21.15.16:0/3904127506 learned_addr learned my addr 172.21.15.16:0/3904127506 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:26.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.597+0000 7f2152d9d700 1 --2- 172.21.15.16:0/3904127506 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f215415b630 0x7f21540d0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:26.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.598+0000 7f215259c700 1 -- 172.21.15.16:0/3904127506 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f215415b630 msgr2=0x7f21540d0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:26.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.598+0000 7f215259c700 1 --2- 172.21.15.16:0/3904127506 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f215415b630 0x7f21540d0380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:26.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.598+0000 7f215259c700 1 -- 172.21.15.16:0/3904127506 >> v1:172.21.15.16:6789/0 conn(0x7f21540cdde0 legacy=0x7f21540cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:26.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.598+0000 7f215259c700 1 -- 172.21.15.16:0/3904127506 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21540b0c30 con 0x7f215415ba10 2023-10-16T13:49:26.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.598+0000 7f2151d9b700 1 -- 172.21.15.16:0/3904127506 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2979789854 0 0) 0x7f21540b0970 con 0x7f21540cdde0 2023-10-16T13:49:26.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.598+0000 7f215259c700 1 --2- 172.21.15.16:0/3904127506 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f215415ba10 0x7f21540cf290 secure :-1 s=READY pgs=3287 cs=0 l=1 rev1=1 crypto rx=0x7f21480098f0 tx=0x7f2148009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e17d9efe76d3d15 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:26.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.598+0000 7f2151d9b700 1 -- 172.21.15.16:0/3904127506 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2148011830 con 0x7f215415ba10 2023-10-16T13:49:26.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.598+0000 7f2151d9b700 1 -- 172.21.15.16:0/3904127506 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2148011990 con 0x7f215415ba10 2023-10-16T13:49:26.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.598+0000 7f2151d9b700 1 -- 172.21.15.16:0/3904127506 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2148011d20 con 0x7f215415ba10 2023-10-16T13:49:26.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.599+0000 7f21591a3700 1 -- 172.21.15.16:0/3904127506 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f215415ba10 msgr2=0x7f21540cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:26.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.599+0000 7f21591a3700 1 --2- 172.21.15.16:0/3904127506 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f215415ba10 0x7f21540cf290 secure :-1 s=READY pgs=3287 cs=0 l=1 rev1=1 crypto rx=0x7f21480098f0 tx=0x7f2148009c00 comp rx=0 tx=0).stop 2023-10-16T13:49:26.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.599+0000 7f21591a3700 1 -- 172.21.15.16:0/3904127506 shutdown_connections 2023-10-16T13:49:26.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.599+0000 7f21591a3700 1 --2- 172.21.15.16:0/3904127506 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f215415b630 0x7f21540d0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:26.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.599+0000 7f21591a3700 1 --2- 172.21.15.16:0/3904127506 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f215415ba10 0x7f21540cf290 unknown :-1 s=CLOSED pgs=3287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:26.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.599+0000 7f21591a3700 1 -- 172.21.15.16:0/3904127506 >> 172.21.15.16:0/3904127506 conn(0x7f21540d14b0 msgr2=0x7f21540d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:26.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.599+0000 7f21591a3700 1 -- 172.21.15.16:0/3904127506 shutdown_connections 2023-10-16T13:49:26.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.599+0000 7f21591a3700 1 -- 172.21.15.16:0/3904127506 wait complete. 2023-10-16T13:49:26.607 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:26.599+0000 7f0548e60200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T13:49:26.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.600+0000 7f21591a3700 1 Processor -- start 2023-10-16T13:49:26.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.600+0000 7f21591a3700 1 -- start start 2023-10-16T13:49:26.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.601+0000 7f21591a3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21540cdde0 0x7f215415e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:26.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.601+0000 7f21591a3700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f215415b630 0x7f21540af9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:26.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.601+0000 7f21591a3700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f215415ba10 0x7f21541503a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:26.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.601+0000 7f21591a3700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f215415e2e0 con 0x7f21540cdde0 2023-10-16T13:49:26.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.601+0000 7f21591a3700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21540b51a0 con 0x7f215415ba10 2023-10-16T13:49:26.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.601+0000 7f21591a3700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21540b5480 con 0x7f215415b630 2023-10-16T13:49:26.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.601+0000 7f2152d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21540cdde0 0x7f215415e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:26.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.601+0000 7f2152d9d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21540cdde0 0x7f215415e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60050/0 (socket says 172.21.15.16:60050) 2023-10-16T13:49:26.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.601+0000 7f2152d9d700 1 -- 172.21.15.16:0/3654464660 learned_addr learned my addr 172.21.15.16:0/3654464660 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:26.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.601+0000 7f215259c700 1 --2- 172.21.15.16:0/3654464660 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f215415b630 0x7f21540af9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:26.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.601+0000 7f215359e700 1 --2- 172.21.15.16:0/3654464660 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f215415ba10 0x7f21541503a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:26.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.601+0000 7f2152d9d700 1 -- 172.21.15.16:0/3654464660 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f215415b630 msgr2=0x7f21540af9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:26.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.601+0000 7f2152d9d700 1 --2- 172.21.15.16:0/3654464660 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f215415b630 0x7f21540af9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:26.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.601+0000 7f2152d9d700 1 -- 172.21.15.16:0/3654464660 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f215415ba10 msgr2=0x7f21541503a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:26.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.601+0000 7f2152d9d700 1 --2- 172.21.15.16:0/3654464660 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f215415ba10 0x7f21541503a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:26.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.601+0000 7f2152d9d700 1 -- 172.21.15.16:0/3654464660 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2148009550 con 0x7f21540cdde0 2023-10-16T13:49:26.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.602+0000 7f2152d9d700 1 --2- 172.21.15.16:0/3654464660 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21540cdde0 0x7f215415e6f0 secure :-1 s=READY pgs=3288 cs=0 l=1 rev1=1 crypto rx=0x7f214400e9b0 tx=0x7f214400ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:26.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.602+0000 7f213b7fe700 1 -- 172.21.15.16:0/3654464660 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2144009710 con 0x7f21540cdde0 2023-10-16T13:49:26.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.602+0000 7f213b7fe700 1 -- 172.21.15.16:0/3654464660 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f214400ee50 con 0x7f21540cdde0 2023-10-16T13:49:26.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.602+0000 7f21591a3700 1 -- 172.21.15.16:0/3654464660 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21540b5760 con 0x7f21540cdde0 2023-10-16T13:49:26.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.602+0000 7f213b7fe700 1 -- 172.21.15.16:0/3654464660 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f214400ca50 con 0x7f21540cdde0 2023-10-16T13:49:26.613 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.602+0000 7f21591a3700 1 -- 172.21.15.16:0/3654464660 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f215415ec30 con 0x7f21540cdde0 2023-10-16T13:49:26.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.603+0000 7f21591a3700 1 -- 172.21.15.16:0/3654464660 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f215404ebb0 con 0x7f21540cdde0 2023-10-16T13:49:26.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.603+0000 7f213b7fe700 1 -- 172.21.15.16:0/3654464660 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 172) v1 ==== 89490+0+0 (secure 0 0 0) 0x7f214400cbb0 con 0x7f21540cdde0 2023-10-16T13:49:26.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.604+0000 7f213b7fe700 1 --2- 172.21.15.16:0/3654464660 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f214006b1d0 0x7f214006d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:26.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.604+0000 7f215259c700 1 -- 172.21.15.16:0/3654464660 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f214006b1d0 msgr2=0x7f214006d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3920835901 2023-10-16T13:49:26.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.604+0000 7f213b7fe700 1 -- 172.21.15.16:0/3654464660 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(279..279 src has 1..279) v4 ==== 6501+0+0 (secure 0 0 0) 0x7f2144089d30 con 0x7f21540cdde0 2023-10-16T13:49:26.614 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.604+0000 7f215259c700 1 --2- 172.21.15.16:0/3654464660 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f214006b1d0 0x7f214006d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:49:26.615 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.609+0000 7f213b7fe700 1 -- 172.21.15.16:0/3654464660 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f214408e070 con 0x7f21540cdde0 2023-10-16T13:49:26.615 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:49:26.609+0000 7f8a615ff200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T13:49:26.777 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:26.774+0000 7f0548e60200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T13:49:26.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.805+0000 7f215259c700 1 -- 172.21.15.16:0/3654464660 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f214006b1d0 msgr2=0x7f214006d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3920835901 2023-10-16T13:49:26.807 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.805+0000 7f215259c700 1 --2- 172.21.15.16:0/3654464660 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f214006b1d0 0x7f214006d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:49:26.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.871+0000 7f21591a3700 1 -- 172.21.15.16:0/3654464660 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f215406b5d0 con 0x7f21540cdde0 2023-10-16T13:49:26.874 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.872+0000 7f213b7fe700 1 -- 172.21.15.16:0/3654464660 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 155 v155) v1 ==== 95+0+4926 (secure 0 0 0) 0x7f2144021090 con 0x7f21540cdde0 2023-10-16T13:49:26.875 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:49:26.875 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":155,"default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12280,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/1127700443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":1127700443},{"type":"v1","addr":"172.21.15.49:6834","nonce":1127700443}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":150},{"gid":13581,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/818113029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":818113029},{"type":"v1","addr":"172.21.15.16:6836","nonce":818113029}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149},{"gid":13584,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3762868123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":3762868123},{"type":"v1","addr":"172.21.15.16:6837","nonce":3762868123}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":150}],"filesystems":[{"mdsmap":{"epoch":155,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:49:02.107137+0000","modified":"2023-10-16T13:49:05.140377+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":13599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13599":{"gid":13599,"name":"b","rank":0,"incarnation":151,"state":"up:active","state_seq":3,"addr":"172.21.15.49:6835/2082691635","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":2082691635},{"type":"v1","addr":"172.21.15.49:6835","nonce":2082691635}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[38],"metadata_pool":37,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":18}]} 2023-10-16T13:49:26.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.874+0000 7f21591a3700 1 -- 172.21.15.16:0/3654464660 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f214006b1d0 msgr2=0x7f214006d690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:49:26.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.874+0000 7f21591a3700 1 --2- 172.21.15.16:0/3654464660 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f214006b1d0 0x7f214006d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:26.877 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.874+0000 7f21591a3700 1 -- 172.21.15.16:0/3654464660 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21540cdde0 msgr2=0x7f215415e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:26.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.874+0000 7f21591a3700 1 --2- 172.21.15.16:0/3654464660 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21540cdde0 0x7f215415e6f0 secure :-1 s=READY pgs=3288 cs=0 l=1 rev1=1 crypto rx=0x7f214400e9b0 tx=0x7f214400ecc0 comp rx=0 tx=0).stop 2023-10-16T13:49:26.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.874+0000 7f21591a3700 1 -- 172.21.15.16:0/3654464660 shutdown_connections 2023-10-16T13:49:26.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.874+0000 7f21591a3700 1 --2- 172.21.15.16:0/3654464660 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f214006b1d0 0x7f214006d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:26.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.874+0000 7f21591a3700 1 --2- 172.21.15.16:0/3654464660 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f215415ba10 0x7f21541503a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:26.878 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.874+0000 7f21591a3700 1 --2- 172.21.15.16:0/3654464660 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21540cdde0 0x7f215415e6f0 unknown :-1 s=CLOSED pgs=3288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:26.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.874+0000 7f21591a3700 1 --2- 172.21.15.16:0/3654464660 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f215415b630 0x7f21540af9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:26.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.874+0000 7f21591a3700 1 -- 172.21.15.16:0/3654464660 >> 172.21.15.16:0/3654464660 conn(0x7f21540d14b0 msgr2=0x7f21540d1890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:26.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.874+0000 7f21591a3700 1 -- 172.21.15.16:0/3654464660 shutdown_connections 2023-10-16T13:49:26.879 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:26.874+0000 7f21591a3700 1 -- 172.21.15.16:0/3654464660 wait complete. 2023-10-16T13:49:26.879 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 155 2023-10-16T13:49:26.883 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:49:26.880+0000 7f8a615ff200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T13:49:26.891 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:49:27.051 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:49:27.049+0000 7f8a615ff200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T13:49:27.057 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:27.054+0000 7f0548e60200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T13:49:27.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.055+0000 7f64efab7700 1 Processor -- start 2023-10-16T13:49:27.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.055+0000 7f64efab7700 1 -- start start 2023-10-16T13:49:27.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.056+0000 7f64efab7700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f64e80b2b50 0x7f64e80d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:27.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.056+0000 7f64efab7700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64e80d1210 con 0x7f64e80ce380 2023-10-16T13:49:27.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.056+0000 7f64efab7700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64e80cbc30 con 0x7f64e80cdfa0 2023-10-16T13:49:27.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.056+0000 7f64efab7700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64e80cbf10 con 0x7f64e80b2b50 2023-10-16T13:49:27.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.056+0000 7f64ee054700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f64e80ce380 0x7f64e80d09e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49338/0 (socket says 172.21.15.16:49338) 2023-10-16T13:49:27.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.056+0000 7f64ee054700 1 -- 172.21.15.16:0/3023559126 learned_addr learned my addr 172.21.15.16:0/3023559126 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:49:27.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.056+0000 7f64ed052700 1 --2- 172.21.15.16:0/3023559126 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f64e80b2b50 0x7f64e80d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:27.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.057+0000 7f64ec851700 1 -- 172.21.15.16:0/3023559126 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1777322877 0 0) 0x7f64e80d1210 con 0x7f64e80ce380 2023-10-16T13:49:27.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.057+0000 7f64ec851700 1 -- 172.21.15.16:0/3023559126 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64d40035b0 con 0x7f64e80ce380 2023-10-16T13:49:27.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.057+0000 7f64ec851700 1 -- 172.21.15.16:0/3023559126 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2472725189 0 0) 0x7f64e80cbc30 con 0x7f64e80cdfa0 2023-10-16T13:49:27.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.057+0000 7f64ec851700 1 -- 172.21.15.16:0/3023559126 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64e80d1210 con 0x7f64e80cdfa0 2023-10-16T13:49:27.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.057+0000 7f64ed052700 1 -- 172.21.15.16:0/3023559126 >> v1:172.21.15.49:6789/0 conn(0x7f64e80cdfa0 legacy=0x7f64e80b2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:27.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.057+0000 7f64ed052700 1 -- 172.21.15.16:0/3023559126 >> v1:172.21.15.16:6789/0 conn(0x7f64e80ce380 legacy=0x7f64e80d09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:27.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.057+0000 7f64ed052700 1 -- 172.21.15.16:0/3023559126 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64e80cc1f0 con 0x7f64e80b2b50 2023-10-16T13:49:27.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.057+0000 7f64ed052700 1 --2- 172.21.15.16:0/3023559126 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f64e80b2b50 0x7f64e80d04a0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f64d8009870 tx=0x7f64d8009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b83dd56f661a8ae8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:27.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.057+0000 7f64ec851700 1 -- 172.21.15.16:0/3023559126 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64d8011830 con 0x7f64e80b2b50 2023-10-16T13:49:27.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.057+0000 7f64ec851700 1 -- 172.21.15.16:0/3023559126 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64d8011990 con 0x7f64e80b2b50 2023-10-16T13:49:27.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.057+0000 7f64ec851700 1 -- 172.21.15.16:0/3023559126 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64d8011d20 con 0x7f64e80b2b50 2023-10-16T13:49:27.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.058+0000 7f64efab7700 1 -- 172.21.15.16:0/3023559126 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f64e80b2b50 msgr2=0x7f64e80d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:27.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.058+0000 7f64efab7700 1 --2- 172.21.15.16:0/3023559126 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f64e80b2b50 0x7f64e80d04a0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f64d8009870 tx=0x7f64d8009b80 comp rx=0 tx=0).stop 2023-10-16T13:49:27.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.058+0000 7f64efab7700 1 -- 172.21.15.16:0/3023559126 shutdown_connections 2023-10-16T13:49:27.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.058+0000 7f64efab7700 1 --2- 172.21.15.16:0/3023559126 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f64e80b2b50 0x7f64e80d04a0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:27.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.058+0000 7f64efab7700 1 -- 172.21.15.16:0/3023559126 >> 172.21.15.16:0/3023559126 conn(0x7f64e80d14b0 msgr2=0x7f64e80d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:27.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.058+0000 7f64efab7700 1 -- 172.21.15.16:0/3023559126 shutdown_connections 2023-10-16T13:49:27.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.058+0000 7f64efab7700 1 -- 172.21.15.16:0/3023559126 wait complete. 2023-10-16T13:49:27.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.059+0000 7f64efab7700 1 Processor -- start 2023-10-16T13:49:27.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.059+0000 7f64efab7700 1 -- start start 2023-10-16T13:49:27.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.060+0000 7f64efab7700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f64e80b2b50 0x7f64e80caec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:27.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.060+0000 7f64efab7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64e80cdfa0 0x7f64e809ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:27.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.060+0000 7f64efab7700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f64e80ce380 0x7f64e80c93d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:27.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.060+0000 7f64efab7700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64e80caa30 con 0x7f64e80cdfa0 2023-10-16T13:49:27.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.060+0000 7f64efab7700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64e80cb400 con 0x7f64e80ce380 2023-10-16T13:49:27.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.060+0000 7f64efab7700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64e809ef90 con 0x7f64e80b2b50 2023-10-16T13:49:27.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.060+0000 7f64ed052700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64e80cdfa0 0x7f64e809ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:27.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.060+0000 7f64ed052700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64e80cdfa0 0x7f64e809ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60068/0 (socket says 172.21.15.16:60068) 2023-10-16T13:49:27.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.060+0000 7f64ed052700 1 -- 172.21.15.16:0/2669071429 learned_addr learned my addr 172.21.15.16:0/2669071429 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:27.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.060+0000 7f64ed853700 1 --2- 172.21.15.16:0/2669071429 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f64e80b2b50 0x7f64e80caec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:27.070 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.060+0000 7f64ee054700 1 --2- 172.21.15.16:0/2669071429 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f64e80ce380 0x7f64e80c93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:27.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.060+0000 7f64ed052700 1 -- 172.21.15.16:0/2669071429 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f64e80b2b50 msgr2=0x7f64e80caec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:27.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.060+0000 7f64ed052700 1 --2- 172.21.15.16:0/2669071429 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f64e80b2b50 0x7f64e80caec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:27.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.060+0000 7f64ed052700 1 -- 172.21.15.16:0/2669071429 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f64e80ce380 msgr2=0x7f64e80c93d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:27.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.060+0000 7f64ed052700 1 --2- 172.21.15.16:0/2669071429 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f64e80ce380 0x7f64e80c93d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:27.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.060+0000 7f64ed052700 1 -- 172.21.15.16:0/2669071429 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64d80094a0 con 0x7f64e80cdfa0 2023-10-16T13:49:27.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.061+0000 7f64ed052700 1 --2- 172.21.15.16:0/2669071429 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64e80cdfa0 0x7f64e809ea50 secure :-1 s=READY pgs=3290 cs=0 l=1 rev1=1 crypto rx=0x7f64d80097c0 tx=0x7f64d8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:27.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.061+0000 7f64de7fc700 1 -- 172.21.15.16:0/2669071429 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64d801a8f0 con 0x7f64e80cdfa0 2023-10-16T13:49:27.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.061+0000 7f64de7fc700 1 -- 172.21.15.16:0/2669071429 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64d801aa50 con 0x7f64e80cdfa0 2023-10-16T13:49:27.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.061+0000 7f64de7fc700 1 -- 172.21.15.16:0/2669071429 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64d801adc0 con 0x7f64e80cdfa0 2023-10-16T13:49:27.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.061+0000 7f64efab7700 1 -- 172.21.15.16:0/2669071429 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64e80c9e20 con 0x7f64e80cdfa0 2023-10-16T13:49:27.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.061+0000 7f64efab7700 1 -- 172.21.15.16:0/2669071429 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64e80ca3e0 con 0x7f64e80cdfa0 2023-10-16T13:49:27.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.062+0000 7f64efab7700 1 -- 172.21.15.16:0/2669071429 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64e804ebb0 con 0x7f64e80cdfa0 2023-10-16T13:49:27.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.062+0000 7f64de7fc700 1 -- 172.21.15.16:0/2669071429 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 172) v1 ==== 89490+0+0 (secure 0 0 0) 0x7f64d802d450 con 0x7f64e80cdfa0 2023-10-16T13:49:27.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.063+0000 7f64de7fc700 1 --2- 172.21.15.16:0/2669071429 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f64cc06b1d0 0x7f64cc06d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:27.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.063+0000 7f64ed853700 1 -- 172.21.15.16:0/2669071429 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f64cc06b1d0 msgr2=0x7f64cc06d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3920835901 2023-10-16T13:49:27.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.063+0000 7f64de7fc700 1 -- 172.21.15.16:0/2669071429 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(279..279 src has 1..279) v4 ==== 6501+0+0 (secure 0 0 0) 0x7f64d808afa0 con 0x7f64e80cdfa0 2023-10-16T13:49:27.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.063+0000 7f64ed853700 1 --2- 172.21.15.16:0/2669071429 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f64cc06b1d0 0x7f64cc06d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:49:27.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.067+0000 7f64de7fc700 1 -- 172.21.15.16:0/2669071429 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f64d8055a70 con 0x7f64e80cdfa0 2023-10-16T13:49:27.233 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:27.231+0000 7f0548e60200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T13:49:27.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.264+0000 7f64ed853700 1 -- 172.21.15.16:0/2669071429 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f64cc06b1d0 msgr2=0x7f64cc06d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3920835901 2023-10-16T13:49:27.266 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.264+0000 7f64ed853700 1 --2- 172.21.15.16:0/2669071429 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f64cc06b1d0 0x7f64cc06d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:49:27.275 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.272+0000 7f64efab7700 1 -- 172.21.15.16:0/2669071429 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f64e80712b0 con 0x7f64e80cdfa0 2023-10-16T13:49:27.276 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.274+0000 7f64de7fc700 1 -- 172.21.15.16:0/2669071429 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v279) v1 ==== 74+0+17324 (secure 0 0 0) 0x7f64d8015070 con 0x7f64e80cdfa0 2023-10-16T13:49:27.276 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:49:27.277 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":279,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:49:17.194563+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":38,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":37,"pool_name":"cephfs_metadata","create_time":"2023-10-16T13:49:00.609612+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"277","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":38,"pool_name":"cephfs_data","create_time":"2023-10-16T13:49:01.068705+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"278","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/750285561":"2023-10-17T13:49:17.194509+0000","172.21.15.16:6833/1257392887":"2023-10-17T13:49:17.194509+0000","172.21.15.16:6832/1257392887":"2023-10-17T13:49:17.194509+0000","172.21.15.16:0/1101859511":"2023-10-17T13:49:17.194509+0000","172.21.15.16:0/197265557":"2023-10-17T13:49:17.194509+0000","172.21.15.16:0/2024751970":"2023-10-17T13:49:17.194509+0000","172.21.15.16:0/2401946155":"2023-10-17T13:49:17.194509+0000","172.21.15.16:0/1156439592":"2023-10-17T13:49:17.194509+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:49:27.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.276+0000 7f64efab7700 1 -- 172.21.15.16:0/2669071429 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f64cc06b1d0 msgr2=0x7f64cc06d690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:49:27.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.276+0000 7f64efab7700 1 --2- 172.21.15.16:0/2669071429 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f64cc06b1d0 0x7f64cc06d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:27.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.276+0000 7f64efab7700 1 -- 172.21.15.16:0/2669071429 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64e80cdfa0 msgr2=0x7f64e809ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:27.279 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.276+0000 7f64efab7700 1 --2- 172.21.15.16:0/2669071429 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64e80cdfa0 0x7f64e809ea50 secure :-1 s=READY pgs=3290 cs=0 l=1 rev1=1 crypto rx=0x7f64d80097c0 tx=0x7f64d8004750 comp rx=0 tx=0).stop 2023-10-16T13:49:27.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.276+0000 7f64efab7700 1 -- 172.21.15.16:0/2669071429 shutdown_connections 2023-10-16T13:49:27.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.276+0000 7f64efab7700 1 --2- 172.21.15.16:0/2669071429 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7f64cc06b1d0 0x7f64cc06d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:27.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.276+0000 7f64efab7700 1 --2- 172.21.15.16:0/2669071429 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f64e80ce380 0x7f64e80c93d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:27.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.276+0000 7f64efab7700 1 --2- 172.21.15.16:0/2669071429 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f64e80b2b50 0x7f64e80caec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:27.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.276+0000 7f64efab7700 1 --2- 172.21.15.16:0/2669071429 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64e80cdfa0 0x7f64e809ea50 unknown :-1 s=CLOSED pgs=3290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:27.280 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.276+0000 7f64efab7700 1 -- 172.21.15.16:0/2669071429 >> 172.21.15.16:0/2669071429 conn(0x7f64e80d14b0 msgr2=0x7f64e8070260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:27.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.276+0000 7f64efab7700 1 -- 172.21.15.16:0/2669071429 shutdown_connections 2023-10-16T13:49:27.281 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.276+0000 7f64efab7700 1 -- 172.21.15.16:0/2669071429 wait complete. 2023-10-16T13:49:27.291 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-16T13:49:27.367 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:49:27.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.492+0000 7fdacf374700 1 Processor -- start 2023-10-16T13:49:27.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.492+0000 7fdacf374700 1 -- start start 2023-10-16T13:49:27.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.493+0000 7fdacf374700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fdac80b2b50 0x7fdac80d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:27.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.493+0000 7fdacf374700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdac80ce380 0x7fdac80d09e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:27.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.493+0000 7fdacf374700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdac80d1210 con 0x7fdac80cdfa0 2023-10-16T13:49:27.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.493+0000 7fdacf374700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdac80cbc30 con 0x7fdac80ce380 2023-10-16T13:49:27.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.493+0000 7fdacf374700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdac80cbf10 con 0x7fdac80b2b50 2023-10-16T13:49:27.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.493+0000 7fdacd110700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fdac80cdfa0 0x7fdac80b2440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49350/0 (socket says 172.21.15.16:49350) 2023-10-16T13:49:27.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.493+0000 7fdacd110700 1 -- 172.21.15.16:0/4097482797 learned_addr learned my addr 172.21.15.16:0/4097482797 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:49:27.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.493+0000 7fdacd911700 1 --2- 172.21.15.16:0/4097482797 >> v2:172.21.15.16:3300/0 conn(0x7fdac80ce380 0x7fdac80d09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:27.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.493+0000 7fdacc90f700 1 --2- 172.21.15.16:0/4097482797 >> v2:172.21.15.49:3300/0 conn(0x7fdac80b2b50 0x7fdac80d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:27.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.494+0000 7fdabffff700 1 -- 172.21.15.16:0/4097482797 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1215023258 0 0) 0x7fdac80d1210 con 0x7fdac80cdfa0 2023-10-16T13:49:27.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.494+0000 7fdabffff700 1 -- 172.21.15.16:0/4097482797 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdab40035b0 con 0x7fdac80cdfa0 2023-10-16T13:49:27.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.494+0000 7fdacd911700 1 -- 172.21.15.16:0/4097482797 >> v2:172.21.15.49:3300/0 conn(0x7fdac80b2b50 msgr2=0x7fdac80d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:27.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.494+0000 7fdacd911700 1 --2- 172.21.15.16:0/4097482797 >> v2:172.21.15.49:3300/0 conn(0x7fdac80b2b50 0x7fdac80d04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:27.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.494+0000 7fdacd911700 1 -- 172.21.15.16:0/4097482797 >> v1:172.21.15.16:6789/0 conn(0x7fdac80cdfa0 legacy=0x7fdac80b2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:27.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.494+0000 7fdacd911700 1 -- 172.21.15.16:0/4097482797 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdac80cc1f0 con 0x7fdac80ce380 2023-10-16T13:49:27.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.494+0000 7fdacd911700 1 --2- 172.21.15.16:0/4097482797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdac80ce380 0x7fdac80d09e0 secure :-1 s=READY pgs=3292 cs=0 l=1 rev1=1 crypto rx=0x7fdac0009d90 tx=0x7fdac000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c91485a520556f71 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:27.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.494+0000 7fdabffff700 1 -- 172.21.15.16:0/4097482797 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdac000e070 con 0x7fdac80ce380 2023-10-16T13:49:27.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.494+0000 7fdabffff700 1 -- 172.21.15.16:0/4097482797 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdac000bae0 con 0x7fdac80ce380 2023-10-16T13:49:27.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.494+0000 7fdabffff700 1 -- 172.21.15.16:0/4097482797 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdac000be50 con 0x7fdac80ce380 2023-10-16T13:49:27.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.494+0000 7fdacf374700 1 -- 172.21.15.16:0/4097482797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdac80ce380 msgr2=0x7fdac80d09e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:27.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.494+0000 7fdacf374700 1 --2- 172.21.15.16:0/4097482797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdac80ce380 0x7fdac80d09e0 secure :-1 s=READY pgs=3292 cs=0 l=1 rev1=1 crypto rx=0x7fdac0009d90 tx=0x7fdac000b330 comp rx=0 tx=0).stop 2023-10-16T13:49:27.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.495+0000 7fdacf374700 1 -- 172.21.15.16:0/4097482797 shutdown_connections 2023-10-16T13:49:27.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.495+0000 7fdacf374700 1 --2- 172.21.15.16:0/4097482797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdac80ce380 0x7fdac80d09e0 unknown :-1 s=CLOSED pgs=3292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:27.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.495+0000 7fdacf374700 1 --2- 172.21.15.16:0/4097482797 >> v2:172.21.15.49:3300/0 conn(0x7fdac80b2b50 0x7fdac80d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:27.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.495+0000 7fdacf374700 1 -- 172.21.15.16:0/4097482797 >> 172.21.15.16:0/4097482797 conn(0x7fdac80d14b0 msgr2=0x7fdac80d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:27.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.495+0000 7fdacf374700 1 -- 172.21.15.16:0/4097482797 shutdown_connections 2023-10-16T13:49:27.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.495+0000 7fdacf374700 1 -- 172.21.15.16:0/4097482797 wait complete. 2023-10-16T13:49:27.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.496+0000 7fdacf374700 1 Processor -- start 2023-10-16T13:49:27.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.496+0000 7fdacf374700 1 -- start start 2023-10-16T13:49:27.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.497+0000 7fdacf374700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdac80b2b50 0x7fdac815ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:27.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.497+0000 7fdacf374700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdac80cdfa0 0x7fdac81e7b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:27.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.497+0000 7fdacf374700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdac80ce380 0x7fdac81e6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:27.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.497+0000 7fdacf374700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdac81e6ac0 con 0x7fdac80ce380 2023-10-16T13:49:27.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.497+0000 7fdacf374700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdac81e6da0 con 0x7fdac80cdfa0 2023-10-16T13:49:27.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.497+0000 7fdacf374700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdac81e7080 con 0x7fdac80b2b50 2023-10-16T13:49:27.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.497+0000 7fdacd911700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdac80ce380 0x7fdac81e6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:27.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.497+0000 7fdacd911700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdac80ce380 0x7fdac81e6270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60086/0 (socket says 172.21.15.16:60086) 2023-10-16T13:49:27.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.497+0000 7fdacd911700 1 -- 172.21.15.16:0/2133245069 learned_addr learned my addr 172.21.15.16:0/2133245069 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:27.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.497+0000 7fdacc90f700 1 --2- 172.21.15.16:0/2133245069 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdac80cdfa0 0x7fdac81e7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:27.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.497+0000 7fdacd110700 1 --2- 172.21.15.16:0/2133245069 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdac80b2b50 0x7fdac815ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:27.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.497+0000 7fdacd911700 1 -- 172.21.15.16:0/2133245069 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdac80b2b50 msgr2=0x7fdac815ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:27.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.497+0000 7fdacd911700 1 --2- 172.21.15.16:0/2133245069 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdac80b2b50 0x7fdac815ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:27.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.497+0000 7fdacd911700 1 -- 172.21.15.16:0/2133245069 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdac80cdfa0 msgr2=0x7fdac81e7b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:27.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.497+0000 7fdacd911700 1 --2- 172.21.15.16:0/2133245069 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdac80cdfa0 0x7fdac81e7b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:27.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.497+0000 7fdacd911700 1 -- 172.21.15.16:0/2133245069 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdac0009a70 con 0x7fdac80ce380 2023-10-16T13:49:27.508 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.497+0000 7fdacc90f700 1 --2- 172.21.15.16:0/2133245069 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdac80cdfa0 0x7fdac81e7b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:49:27.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.497+0000 7fdacd911700 1 --2- 172.21.15.16:0/2133245069 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdac80ce380 0x7fdac81e6270 secure :-1 s=READY pgs=3293 cs=0 l=1 rev1=1 crypto rx=0x7fdac000b370 tx=0x7fdac0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:27.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.497+0000 7fdabdffb700 1 -- 172.21.15.16:0/2133245069 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdac0004450 con 0x7fdac80ce380 2023-10-16T13:49:27.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.498+0000 7fdabdffb700 1 -- 172.21.15.16:0/2133245069 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdac001f070 con 0x7fdac80ce380 2023-10-16T13:49:27.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.498+0000 7fdacf374700 1 -- 172.21.15.16:0/2133245069 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdac81e7360 con 0x7fdac80ce380 2023-10-16T13:49:27.509 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.498+0000 7fdabdffb700 1 -- 172.21.15.16:0/2133245069 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdac00047f0 con 0x7fdac80ce380 2023-10-16T13:49:27.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.498+0000 7fdacf374700 1 -- 172.21.15.16:0/2133245069 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdac81e67b0 con 0x7fdac80ce380 2023-10-16T13:49:27.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.499+0000 7fdacf374700 1 -- 172.21.15.16:0/2133245069 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdac804ebb0 con 0x7fdac80ce380 2023-10-16T13:49:27.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.500+0000 7fdabdffb700 1 -- 172.21.15.16:0/2133245069 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 172) v1 ==== 89490+0+0 (secure 0 0 0) 0x7fdac0012430 con 0x7fdac80ce380 2023-10-16T13:49:27.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.501+0000 7fdabdffb700 1 --2- 172.21.15.16:0/2133245069 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7fdaac06b1d0 0x7fdaac06d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:27.510 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.501+0000 7fdacd110700 1 -- 172.21.15.16:0/2133245069 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7fdaac06b1d0 msgr2=0x7fdaac06d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3920835901 2023-10-16T13:49:27.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.501+0000 7fdabdffb700 1 -- 172.21.15.16:0/2133245069 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(279..279 src has 1..279) v4 ==== 6501+0+0 (secure 0 0 0) 0x7fdac008b9a0 con 0x7fdac80ce380 2023-10-16T13:49:27.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.501+0000 7fdacd110700 1 --2- 172.21.15.16:0/2133245069 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7fdaac06b1d0 0x7fdaac06d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T13:49:27.511 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.504+0000 7fdabdffb700 1 -- 172.21.15.16:0/2133245069 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdac00564b0 con 0x7fdac80ce380 2023-10-16T13:49:27.611 INFO:tasks.ceph.mgr.x.smithi049.stderr:2023-10-16T13:49:27.609+0000 7f8a615ff200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T13:49:27.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.702+0000 7fdacd110700 1 -- 172.21.15.16:0/2133245069 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7fdaac06b1d0 msgr2=0x7fdaac06d690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6832/3920835901 2023-10-16T13:49:27.704 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.702+0000 7fdacd110700 1 --2- 172.21.15.16:0/2133245069 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7fdaac06b1d0 0x7fdaac06d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-16T13:49:27.756 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.754+0000 7fdacf374700 1 -- 172.21.15.16:0/2133245069 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdac80709d0 con 0x7fdac80ce380 2023-10-16T13:49:27.757 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.755+0000 7fdabdffb700 1 -- 172.21.15.16:0/2133245069 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 155 v155) v1 ==== 95+0+4926 (secure 0 0 0) 0x7fdac0021020 con 0x7fdac80ce380 2023-10-16T13:49:27.757 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:49:27.758 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":155,"default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12280,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/1127700443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":1127700443},{"type":"v1","addr":"172.21.15.49:6834","nonce":1127700443}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":150},{"gid":13581,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/818113029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":818113029},{"type":"v1","addr":"172.21.15.16:6836","nonce":818113029}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149},{"gid":13584,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3762868123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":3762868123},{"type":"v1","addr":"172.21.15.16:6837","nonce":3762868123}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":150}],"filesystems":[{"mdsmap":{"epoch":155,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:49:02.107137+0000","modified":"2023-10-16T13:49:05.140377+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":13599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13599":{"gid":13599,"name":"b","rank":0,"incarnation":151,"state":"up:active","state_seq":3,"addr":"172.21.15.49:6835/2082691635","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":2082691635},{"type":"v1","addr":"172.21.15.49:6835","nonce":2082691635}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[38],"metadata_pool":37,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":18}]} 2023-10-16T13:49:27.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.757+0000 7fdacf374700 1 -- 172.21.15.16:0/2133245069 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7fdaac06b1d0 msgr2=0x7fdaac06d690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T13:49:27.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.757+0000 7fdacf374700 1 --2- 172.21.15.16:0/2133245069 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7fdaac06b1d0 0x7fdaac06d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:27.760 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.757+0000 7fdacf374700 1 -- 172.21.15.16:0/2133245069 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdac80ce380 msgr2=0x7fdac81e6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:27.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.757+0000 7fdacf374700 1 --2- 172.21.15.16:0/2133245069 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdac80ce380 0x7fdac81e6270 secure :-1 s=READY pgs=3293 cs=0 l=1 rev1=1 crypto rx=0x7fdac000b370 tx=0x7fdac0004ad0 comp rx=0 tx=0).stop 2023-10-16T13:49:27.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.757+0000 7fdacf374700 1 -- 172.21.15.16:0/2133245069 shutdown_connections 2023-10-16T13:49:27.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.757+0000 7fdacf374700 1 --2- 172.21.15.16:0/2133245069 >> [v2:172.21.15.16:6832/3920835901,v1:172.21.15.16:6833/3920835901] conn(0x7fdaac06b1d0 0x7fdaac06d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:27.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.757+0000 7fdacf374700 1 --2- 172.21.15.16:0/2133245069 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdac80ce380 0x7fdac81e6270 unknown :-1 s=CLOSED pgs=3293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:27.761 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.757+0000 7fdacf374700 1 --2- 172.21.15.16:0/2133245069 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdac80b2b50 0x7fdac815ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:27.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.757+0000 7fdacf374700 1 --2- 172.21.15.16:0/2133245069 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdac80cdfa0 0x7fdac81e7b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:27.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.757+0000 7fdacf374700 1 -- 172.21.15.16:0/2133245069 >> 172.21.15.16:0/2133245069 conn(0x7fdac80d14b0 msgr2=0x7fdac80cd470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:27.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.757+0000 7fdacf374700 1 -- 172.21.15.16:0/2133245069 shutdown_connections 2023-10-16T13:49:27.762 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.757+0000 7fdacf374700 1 -- 172.21.15.16:0/2133245069 wait complete. 2023-10-16T13:49:27.763 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 155 2023-10-16T13:49:27.772 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:49:27.813 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:27.810+0000 7f0548e60200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T13:49:27.898 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:27.896+0000 7f04c1686700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:49:27.899 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:27.896+0000 7f04c1686700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:49:27.899 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:27.896+0000 7f04c1686700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:49:27.899 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:27.896+0000 7f04c1686700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:49:27.899 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:27.896+0000 7f04c1686700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:49:27.900 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:27.897+0000 7f04c468c700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:49:27.900 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:27.897+0000 7f04c468c700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:49:27.900 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:27.897+0000 7f04c468c700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:49:27.901 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:27.897+0000 7f04c468c700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:49:27.901 INFO:tasks.ceph.mgr.y.smithi016.stderr:2023-10-16T13:49:27.897+0000 7f04c468c700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-16T13:49:27.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.911+0000 7f318dd8a700 1 Processor -- start 2023-10-16T13:49:27.914 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.911+0000 7f318dd8a700 1 -- start start 2023-10-16T13:49:27.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.912+0000 7f318dd8a700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f318815d8d0 0x7f31880d1c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:27.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.912+0000 7f318dd8a700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f318806ba00 con 0x7f31880d0750 2023-10-16T13:49:27.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.912+0000 7f318dd8a700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31880d2150 con 0x7f31880d41c0 2023-10-16T13:49:27.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.912+0000 7f318dd8a700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31880d26d0 con 0x7f318815d8d0 2023-10-16T13:49:27.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.912+0000 7f3187fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f31880d0750 0x7f318806c4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49442/0 (socket says 172.21.15.16:49442) 2023-10-16T13:49:27.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.912+0000 7f3187fff700 1 -- 172.21.15.16:0/3265804642 learned_addr learned my addr 172.21.15.16:0/3265804642 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:49:27.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.912+0000 7f31877fe700 1 --2- 172.21.15.16:0/3265804642 >> v2:172.21.15.49:3300/0 conn(0x7f318815d8d0 0x7f31880d1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:27.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.912+0000 7f31867fc700 1 -- 172.21.15.16:0/3265804642 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1139316376 0 0) 0x7f318806ba00 con 0x7f31880d0750 2023-10-16T13:49:27.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.912+0000 7f31867fc700 1 -- 172.21.15.16:0/3265804642 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31700035b0 con 0x7f31880d0750 2023-10-16T13:49:27.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.912+0000 7f31877fe700 1 -- 172.21.15.16:0/3265804642 >> v1:172.21.15.49:6789/0 conn(0x7f31880d41c0 legacy=0x7f31880d0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:27.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.912+0000 7f31867fc700 1 -- 172.21.15.16:0/3265804642 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 113814737 0 0) 0x7f31880d2150 con 0x7f31880d41c0 2023-10-16T13:49:27.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.912+0000 7f31877fe700 1 -- 172.21.15.16:0/3265804642 >> v1:172.21.15.16:6789/0 conn(0x7f31880d0750 legacy=0x7f318806c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:27.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.913+0000 7f31877fe700 1 -- 172.21.15.16:0/3265804642 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31880d23f0 con 0x7f318815d8d0 2023-10-16T13:49:27.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.913+0000 7f31877fe700 1 --2- 172.21.15.16:0/3265804642 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f318815d8d0 0x7f31880d1c10 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f3178009840 tx=0x7f3178009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=281f88a83c0c2504 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:27.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.913+0000 7f31867fc700 1 -- 172.21.15.16:0/3265804642 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3178011830 con 0x7f318815d8d0 2023-10-16T13:49:27.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.913+0000 7f31867fc700 1 -- 172.21.15.16:0/3265804642 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3178011990 con 0x7f318815d8d0 2023-10-16T13:49:27.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.913+0000 7f31867fc700 1 -- 172.21.15.16:0/3265804642 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3178011d20 con 0x7f318815d8d0 2023-10-16T13:49:27.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.913+0000 7f318dd8a700 1 -- 172.21.15.16:0/3265804642 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f318815d8d0 msgr2=0x7f31880d1c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:27.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.913+0000 7f318dd8a700 1 --2- 172.21.15.16:0/3265804642 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f318815d8d0 0x7f31880d1c10 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f3178009840 tx=0x7f3178009b50 comp rx=0 tx=0).stop 2023-10-16T13:49:27.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.913+0000 7f318dd8a700 1 -- 172.21.15.16:0/3265804642 shutdown_connections 2023-10-16T13:49:27.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.913+0000 7f318dd8a700 1 --2- 172.21.15.16:0/3265804642 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f318815d8d0 0x7f31880d1c10 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:27.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.913+0000 7f318dd8a700 1 -- 172.21.15.16:0/3265804642 >> 172.21.15.16:0/3265804642 conn(0x7f31880d3630 msgr2=0x7f31880d3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:27.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.914+0000 7f318dd8a700 1 -- 172.21.15.16:0/3265804642 shutdown_connections 2023-10-16T13:49:27.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.914+0000 7f318dd8a700 1 -- 172.21.15.16:0/3265804642 wait complete. 2023-10-16T13:49:27.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.915+0000 7f318dd8a700 1 Processor -- start 2023-10-16T13:49:27.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.915+0000 7f318dd8a700 1 -- start start 2023-10-16T13:49:27.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.916+0000 7f318dd8a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f31880d0750 0x7f318814fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:27.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.916+0000 7f318dd8a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31880d41c0 0x7f31880cca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:27.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.916+0000 7f318dd8a700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f318815d8d0 0x7f31881e6970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:27.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.916+0000 7f318dd8a700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31880ca700 con 0x7f31880d41c0 2023-10-16T13:49:27.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.916+0000 7f318dd8a700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31881e5a70 con 0x7f318815d8d0 2023-10-16T13:49:27.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.916+0000 7f318dd8a700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31881e5d50 con 0x7f31880d0750 2023-10-16T13:49:27.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.916+0000 7f3186ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31880d41c0 0x7f31880cca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:27.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.916+0000 7f3186ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31880d41c0 0x7f31880cca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60172/0 (socket says 172.21.15.16:60172) 2023-10-16T13:49:27.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.916+0000 7f3186ffd700 1 -- 172.21.15.16:0/961807680 learned_addr learned my addr 172.21.15.16:0/961807680 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:27.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.916+0000 7f3187fff700 1 --2- 172.21.15.16:0/961807680 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f318815d8d0 0x7f31881e6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:27.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.916+0000 7f31877fe700 1 --2- 172.21.15.16:0/961807680 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f31880d0750 0x7f318814fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:27.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.916+0000 7f3186ffd700 1 -- 172.21.15.16:0/961807680 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f31880d0750 msgr2=0x7f318814fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:27.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.916+0000 7f3186ffd700 1 --2- 172.21.15.16:0/961807680 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f31880d0750 0x7f318814fb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:27.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.916+0000 7f3186ffd700 1 -- 172.21.15.16:0/961807680 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f318815d8d0 msgr2=0x7f31881e6970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:27.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.916+0000 7f3186ffd700 1 --2- 172.21.15.16:0/961807680 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f318815d8d0 0x7f31881e6970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:27.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.916+0000 7f3186ffd700 1 -- 172.21.15.16:0/961807680 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31780094a0 con 0x7f31880d41c0 2023-10-16T13:49:27.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.916+0000 7f3187fff700 1 --2- 172.21.15.16:0/961807680 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f318815d8d0 0x7f31881e6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:49:27.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.916+0000 7f3186ffd700 1 --2- 172.21.15.16:0/961807680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31880d41c0 0x7f31880cca30 secure :-1 s=READY pgs=3302 cs=0 l=1 rev1=1 crypto rx=0x7f317c0011e0 tx=0x7f317c00cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:27.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.916+0000 7f316ffff700 1 -- 172.21.15.16:0/961807680 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f317c0054e0 con 0x7f31880d41c0 2023-10-16T13:49:27.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.916+0000 7f316ffff700 1 -- 172.21.15.16:0/961807680 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f317c003680 con 0x7f31880d41c0 2023-10-16T13:49:27.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.916+0000 7f318dd8a700 1 -- 172.21.15.16:0/961807680 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31881e6030 con 0x7f31880d41c0 2023-10-16T13:49:27.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.916+0000 7f316ffff700 1 -- 172.21.15.16:0/961807680 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f317c003a30 con 0x7f31880d41c0 2023-10-16T13:49:27.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.916+0000 7f318dd8a700 1 -- 172.21.15.16:0/961807680 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31880ccf70 con 0x7f31880d41c0 2023-10-16T13:49:27.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.917+0000 7f318dd8a700 1 -- 172.21.15.16:0/961807680 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f318804ebb0 con 0x7f31880d41c0 2023-10-16T13:49:27.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.919+0000 7f316ffff700 1 -- 172.21.15.16:0/961807680 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 173) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f317c003b90 con 0x7f31880d41c0 2023-10-16T13:49:27.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.919+0000 7f316ffff700 1 -- 172.21.15.16:0/961807680 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(280..280 src has 1..280) v4 ==== 6974+0+0 (secure 0 0 0) 0x7f317c088e60 con 0x7f31880d41c0 2023-10-16T13:49:27.929 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:27.923+0000 7f316ffff700 1 -- 172.21.15.16:0/961807680 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f317c0b5fe0 con 0x7f31880d41c0 2023-10-16T13:49:28.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.171+0000 7f318dd8a700 1 -- 172.21.15.16:0/961807680 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3188072d00 con 0x7f31880d41c0 2023-10-16T13:49:28.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.171+0000 7f316ffff700 1 -- 172.21.15.16:0/961807680 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 155 v155) v1 ==== 95+0+4926 (secure 0 0 0) 0x7f317c017030 con 0x7f31880d41c0 2023-10-16T13:49:28.174 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:49:28.174 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":155,"default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12280,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/1127700443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":1127700443},{"type":"v1","addr":"172.21.15.49:6834","nonce":1127700443}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":150},{"gid":13581,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/818113029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":818113029},{"type":"v1","addr":"172.21.15.16:6836","nonce":818113029}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149},{"gid":13584,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3762868123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":3762868123},{"type":"v1","addr":"172.21.15.16:6837","nonce":3762868123}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":150}],"filesystems":[{"mdsmap":{"epoch":155,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T13:49:02.107137+0000","modified":"2023-10-16T13:49:05.140377+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":13599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13599":{"gid":13599,"name":"b","rank":0,"incarnation":151,"state":"up:active","state_seq":3,"addr":"172.21.15.49:6835/2082691635","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":2082691635},{"type":"v1","addr":"172.21.15.49:6835","nonce":2082691635}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[38],"metadata_pool":37,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":18}]} 2023-10-16T13:49:28.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.173+0000 7f318dd8a700 1 -- 172.21.15.16:0/961807680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31880d41c0 msgr2=0x7f31880cca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:28.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.173+0000 7f318dd8a700 1 --2- 172.21.15.16:0/961807680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31880d41c0 0x7f31880cca30 secure :-1 s=READY pgs=3302 cs=0 l=1 rev1=1 crypto rx=0x7f317c0011e0 tx=0x7f317c00cae0 comp rx=0 tx=0).stop 2023-10-16T13:49:28.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.173+0000 7f318dd8a700 1 -- 172.21.15.16:0/961807680 shutdown_connections 2023-10-16T13:49:28.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.173+0000 7f318dd8a700 1 --2- 172.21.15.16:0/961807680 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f318815d8d0 0x7f31881e6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:28.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.174+0000 7f318dd8a700 1 --2- 172.21.15.16:0/961807680 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f31880d0750 0x7f318814fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:28.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.174+0000 7f318dd8a700 1 --2- 172.21.15.16:0/961807680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31880d41c0 0x7f31880cca30 unknown :-1 s=CLOSED pgs=3302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:28.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.174+0000 7f318dd8a700 1 -- 172.21.15.16:0/961807680 >> 172.21.15.16:0/961807680 conn(0x7f31880d3630 msgr2=0x7f318815c940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:28.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.174+0000 7f318dd8a700 1 -- 172.21.15.16:0/961807680 shutdown_connections 2023-10-16T13:49:28.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.174+0000 7f318dd8a700 1 -- 172.21.15.16:0/961807680 wait complete. 2023-10-16T13:49:28.178 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 155 2023-10-16T13:49:28.188 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:49:28.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.350+0000 7f92889c5700 1 Processor -- start 2023-10-16T13:49:28.353 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.351+0000 7f92889c5700 1 -- start start 2023-10-16T13:49:28.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.351+0000 7f92889c5700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f928415a870 0x7f928406eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:28.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.351+0000 7f92889c5700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f92840cbed0 0x7f92840cde10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:28.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.351+0000 7f92889c5700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f928406f0a0 con 0x7f928406d350 2023-10-16T13:49:28.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.351+0000 7f92889c5700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f928415ac50 con 0x7f92840cbed0 2023-10-16T13:49:28.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.351+0000 7f92889c5700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f928406f380 con 0x7f928415a870 2023-10-16T13:49:28.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.352+0000 7f9281d9b700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f928406d350 0x7f92840cb7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49452/0 (socket says 172.21.15.16:49452) 2023-10-16T13:49:28.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.352+0000 7f9281d9b700 1 -- 172.21.15.16:0/2934427166 learned_addr learned my addr 172.21.15.16:0/2934427166 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:49:28.357 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.352+0000 7f9282d9d700 1 --2- 172.21.15.16:0/2934427166 >> v2:172.21.15.49:3300/0 conn(0x7f92840cbed0 0x7f92840cde10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:28.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.352+0000 7f928259c700 1 --2- 172.21.15.16:0/2934427166 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f928415a870 0x7f928406eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:28.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.352+0000 7f928159a700 1 -- 172.21.15.16:0/2934427166 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2907184803 0 0) 0x7f928406f0a0 con 0x7f928406d350 2023-10-16T13:49:28.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.352+0000 7f928159a700 1 -- 172.21.15.16:0/2934427166 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92700035b0 con 0x7f928406d350 2023-10-16T13:49:28.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.352+0000 7f9282d9d700 1 -- 172.21.15.16:0/2934427166 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f928415a870 msgr2=0x7f928406eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:28.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.352+0000 7f9282d9d700 1 --2- 172.21.15.16:0/2934427166 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f928415a870 0x7f928406eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:28.358 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.352+0000 7f9282d9d700 1 -- 172.21.15.16:0/2934427166 >> v1:172.21.15.16:6789/0 conn(0x7f928406d350 legacy=0x7f92840cb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:28.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.352+0000 7f9282d9d700 1 -- 172.21.15.16:0/2934427166 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f928406f660 con 0x7f92840cbed0 2023-10-16T13:49:28.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.352+0000 7f928259c700 1 --2- 172.21.15.16:0/2934427166 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f928415a870 0x7f928406eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:49:28.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.352+0000 7f9282d9d700 1 --2- 172.21.15.16:0/2934427166 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f92840cbed0 0x7f92840cde10 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f927c009d90 tx=0x7f927c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=da89eaf83f1e85fd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:28.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.352+0000 7f928159a700 1 -- 172.21.15.16:0/2934427166 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f927c00e070 con 0x7f92840cbed0 2023-10-16T13:49:28.359 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.353+0000 7f928159a700 1 -- 172.21.15.16:0/2934427166 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f927c00bb20 con 0x7f92840cbed0 2023-10-16T13:49:28.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.353+0000 7f928159a700 1 -- 172.21.15.16:0/2934427166 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f927c012430 con 0x7f92840cbed0 2023-10-16T13:49:28.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.353+0000 7f92889c5700 1 -- 172.21.15.16:0/2934427166 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f92840cbed0 msgr2=0x7f92840cde10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:28.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.353+0000 7f92889c5700 1 --2- 172.21.15.16:0/2934427166 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f92840cbed0 0x7f92840cde10 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f927c009d90 tx=0x7f927c00b330 comp rx=0 tx=0).stop 2023-10-16T13:49:28.360 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.353+0000 7f92889c5700 1 -- 172.21.15.16:0/2934427166 shutdown_connections 2023-10-16T13:49:28.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.353+0000 7f92889c5700 1 --2- 172.21.15.16:0/2934427166 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f928415a870 0x7f928406eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:28.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.353+0000 7f92889c5700 1 --2- 172.21.15.16:0/2934427166 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f92840cbed0 0x7f92840cde10 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:28.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.353+0000 7f92889c5700 1 -- 172.21.15.16:0/2934427166 >> 172.21.15.16:0/2934427166 conn(0x7f92840cc5b0 msgr2=0x7f92840cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:28.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.353+0000 7f92889c5700 1 -- 172.21.15.16:0/2934427166 shutdown_connections 2023-10-16T13:49:28.361 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.353+0000 7f92889c5700 1 -- 172.21.15.16:0/2934427166 wait complete. 2023-10-16T13:49:28.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.355+0000 7f92889c5700 1 Processor -- start 2023-10-16T13:49:28.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.355+0000 7f92889c5700 1 -- start start 2023-10-16T13:49:28.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.355+0000 7f92889c5700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f928406d350 0x7f92840c9610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:28.364 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.355+0000 7f92889c5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92840cbed0 0x7f928413d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:28.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.355+0000 7f92889c5700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f928415a870 0x7f92841f2890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:28.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.355+0000 7f92889c5700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f928413d570 con 0x7f92840cbed0 2023-10-16T13:49:28.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.355+0000 7f92889c5700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92841f2dd0 con 0x7f928406d350 2023-10-16T13:49:28.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.355+0000 7f92889c5700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92840c8570 con 0x7f928415a870 2023-10-16T13:49:28.365 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.355+0000 7f9281d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92840cbed0 0x7f928413d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:28.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.356+0000 7f9281d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92840cbed0 0x7f928413d030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60184/0 (socket says 172.21.15.16:60184) 2023-10-16T13:49:28.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.356+0000 7f9281d9b700 1 -- 172.21.15.16:0/4053740983 learned_addr learned my addr 172.21.15.16:0/4053740983 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:28.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.356+0000 7f928259c700 1 --2- 172.21.15.16:0/4053740983 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f928406d350 0x7f92840c9610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:28.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.356+0000 7f9282d9d700 1 --2- 172.21.15.16:0/4053740983 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f928415a870 0x7f92841f2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:28.366 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.356+0000 7f9281d9b700 1 -- 172.21.15.16:0/4053740983 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f928415a870 msgr2=0x7f92841f2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:28.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.356+0000 7f9281d9b700 1 --2- 172.21.15.16:0/4053740983 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f928415a870 0x7f92841f2890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:28.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.356+0000 7f9281d9b700 1 -- 172.21.15.16:0/4053740983 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f928406d350 msgr2=0x7f92840c9610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:28.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.356+0000 7f9281d9b700 1 --2- 172.21.15.16:0/4053740983 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f928406d350 0x7f92840c9610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:28.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.356+0000 7f9281d9b700 1 -- 172.21.15.16:0/4053740983 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92740094a0 con 0x7f92840cbed0 2023-10-16T13:49:28.367 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.356+0000 7f9281d9b700 1 --2- 172.21.15.16:0/4053740983 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92840cbed0 0x7f928413d030 secure :-1 s=READY pgs=3304 cs=0 l=1 rev1=1 crypto rx=0x7f926c00e910 tx=0x7f926c00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:28.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.356+0000 7f927affd700 1 -- 172.21.15.16:0/4053740983 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f926c005b40 con 0x7f92840cbed0 2023-10-16T13:49:28.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.356+0000 7f927affd700 1 -- 172.21.15.16:0/4053740983 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f926c005ca0 con 0x7f92840cbed0 2023-10-16T13:49:28.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.356+0000 7f92889c5700 1 -- 172.21.15.16:0/4053740983 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f927c009a70 con 0x7f92840cbed0 2023-10-16T13:49:28.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.356+0000 7f927affd700 1 -- 172.21.15.16:0/4053740983 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f926c00ca20 con 0x7f92840cbed0 2023-10-16T13:49:28.368 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.356+0000 7f92889c5700 1 -- 172.21.15.16:0/4053740983 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92840c8b30 con 0x7f92840cbed0 2023-10-16T13:49:28.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.357+0000 7f92889c5700 1 -- 172.21.15.16:0/4053740983 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f928404ebb0 con 0x7f92840cbed0 2023-10-16T13:49:28.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.358+0000 7f927affd700 1 -- 172.21.15.16:0/4053740983 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 173) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f926c00cb80 con 0x7f92840cbed0 2023-10-16T13:49:28.369 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.358+0000 7f927affd700 1 -- 172.21.15.16:0/4053740983 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(280..280 src has 1..280) v4 ==== 6974+0+0 (secure 0 0 0) 0x7f926c08ba50 con 0x7f92840cbed0 2023-10-16T13:49:28.370 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.363+0000 7f927affd700 1 -- 172.21.15.16:0/4053740983 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f926c0569d0 con 0x7f92840cbed0 2023-10-16T13:49:28.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.567+0000 7f92889c5700 1 -- 172.21.15.16:0/4053740983 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9284157b90 con 0x7f92840cbed0 2023-10-16T13:49:28.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.568+0000 7f927affd700 1 -- 172.21.15.16:0/4053740983 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v280) v1 ==== 74+0+18011 (secure 0 0 0) 0x7f926c059ff0 con 0x7f92840cbed0 2023-10-16T13:49:28.571 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:49:28.571 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":280,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:49:27.824774+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":38,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":37,"pool_name":"cephfs_metadata","create_time":"2023-10-16T13:49:00.609612+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"277","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":38,"pool_name":"cephfs_data","create_time":"2023-10-16T13:49:01.068705+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"278","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/3139225757":"2023-10-17T13:49:27.824706+0000","172.21.15.16:0/2356827013":"2023-10-17T13:49:27.824706+0000","172.21.15.16:0/667060049":"2023-10-17T13:49:27.824706+0000","172.21.15.16:0/2056580162":"2023-10-17T13:49:27.824706+0000","172.21.15.16:0/1209578814":"2023-10-17T13:49:27.824706+0000","172.21.15.16:0/4129379262":"2023-10-17T13:49:27.824706+0000","172.21.15.16:6832/3920835901":"2023-10-17T13:49:27.824706+0000","172.21.15.16:0/3384986395":"2023-10-17T13:49:27.824706+0000","172.21.15.16:0/2216636694":"2023-10-17T13:49:27.824706+0000","172.21.15.16:0/750285561":"2023-10-17T13:49:17.194509+0000","172.21.15.16:6833/1257392887":"2023-10-17T13:49:17.194509+0000","172.21.15.16:6832/1257392887":"2023-10-17T13:49:17.194509+0000","172.21.15.16:0/1101859511":"2023-10-17T13:49:17.194509+0000","172.21.15.16:0/3516507548":"2023-10-17T13:49:27.824706+0000","172.21.15.16:6833/3920835901":"2023-10-17T13:49:27.824706+0000","172.21.15.16:0/197265557":"2023-10-17T13:49:17.194509+0000","172.21.15.16:0/2024751970":"2023-10-17T13:49:17.194509+0000","172.21.15.16:0/2401946155":"2023-10-17T13:49:17.194509+0000","172.21.15.16:0/1156439592":"2023-10-17T13:49:17.194509+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:49:28.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.570+0000 7f92889c5700 1 -- 172.21.15.16:0/4053740983 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92840cbed0 msgr2=0x7f928413d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:28.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.570+0000 7f92889c5700 1 --2- 172.21.15.16:0/4053740983 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92840cbed0 0x7f928413d030 secure :-1 s=READY pgs=3304 cs=0 l=1 rev1=1 crypto rx=0x7f926c00e910 tx=0x7f926c00ecd0 comp rx=0 tx=0).stop 2023-10-16T13:49:28.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.570+0000 7f92889c5700 1 -- 172.21.15.16:0/4053740983 shutdown_connections 2023-10-16T13:49:28.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.570+0000 7f92889c5700 1 --2- 172.21.15.16:0/4053740983 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f928415a870 0x7f92841f2890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:28.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.570+0000 7f92889c5700 1 --2- 172.21.15.16:0/4053740983 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f928406d350 0x7f92840c9610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:28.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.570+0000 7f92889c5700 1 --2- 172.21.15.16:0/4053740983 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92840cbed0 0x7f928413d030 unknown :-1 s=CLOSED pgs=3304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:28.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.570+0000 7f92889c5700 1 -- 172.21.15.16:0/4053740983 >> 172.21.15.16:0/4053740983 conn(0x7f92840cc5b0 msgr2=0x7f92840b4310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:28.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.570+0000 7f92889c5700 1 -- 172.21.15.16:0/4053740983 shutdown_connections 2023-10-16T13:49:28.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.570+0000 7f92889c5700 1 -- 172.21.15.16:0/4053740983 wait complete. 2023-10-16T13:49:28.585 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-16T13:49:28.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.709+0000 7fe46a60c700 1 Processor -- start 2023-10-16T13:49:28.712 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.709+0000 7fe46a60c700 1 -- start start 2023-10-16T13:49:28.714 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.710+0000 7fe46a60c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe4640b2b50 0x7fe4640d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:28.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.710+0000 7fe46a60c700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fe4640ce380 0x7fe4640d09e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:28.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.710+0000 7fe46a60c700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4640d1210 con 0x7fe4640cdfa0 2023-10-16T13:49:28.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.710+0000 7fe46a60c700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4640cbc30 con 0x7fe4640b2b50 2023-10-16T13:49:28.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.710+0000 7fe46a60c700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4640cbf10 con 0x7fe4640ce380 2023-10-16T13:49:28.715 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.710+0000 7fe4637fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe4640b2b50 0x7fe4640d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:28.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.710+0000 7fe4637fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe4640b2b50 0x7fe4640d04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60190/0 (socket says 172.21.15.16:60190) 2023-10-16T13:49:28.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.710+0000 7fe4637fe700 1 -- 172.21.15.16:0/3258573055 learned_addr learned my addr 172.21.15.16:0/3258573055 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:28.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.711+0000 7fe468ba9700 1 --2- 172.21.15.16:0/3258573055 >> v2:172.21.15.49:3300/0 conn(0x7fe4640ce380 0x7fe4640d09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:28.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.711+0000 7fe4637fe700 1 -- 172.21.15.16:0/3258573055 >> v2:172.21.15.49:3300/0 conn(0x7fe4640ce380 msgr2=0x7fe4640d09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:28.716 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.711+0000 7fe4637fe700 1 --2- 172.21.15.16:0/3258573055 >> v2:172.21.15.49:3300/0 conn(0x7fe4640ce380 0x7fe4640d09e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:28.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.711+0000 7fe4637fe700 1 -- 172.21.15.16:0/3258573055 >> v1:172.21.15.49:6789/0 conn(0x7fe4640cdfa0 legacy=0x7fe4640b2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:28.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.711+0000 7fe4637fe700 1 -- 172.21.15.16:0/3258573055 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4640cc1f0 con 0x7fe4640b2b50 2023-10-16T13:49:28.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.711+0000 7fe4637fe700 1 --2- 172.21.15.16:0/3258573055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4640b2b50 0x7fe4640d04a0 secure :-1 s=READY pgs=3305 cs=0 l=1 rev1=1 crypto rx=0x7fe4580098f0 tx=0x7fe458009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c411cf4dca3dee82 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:28.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.711+0000 7fe462ffd700 1 -- 172.21.15.16:0/3258573055 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe458004230 con 0x7fe4640b2b50 2023-10-16T13:49:28.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.711+0000 7fe462ffd700 1 -- 172.21.15.16:0/3258573055 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe458004390 con 0x7fe4640b2b50 2023-10-16T13:49:28.717 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.711+0000 7fe462ffd700 1 -- 172.21.15.16:0/3258573055 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4580036a0 con 0x7fe4640b2b50 2023-10-16T13:49:28.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.712+0000 7fe46a60c700 1 -- 172.21.15.16:0/3258573055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4640b2b50 msgr2=0x7fe4640d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:28.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.712+0000 7fe46a60c700 1 --2- 172.21.15.16:0/3258573055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4640b2b50 0x7fe4640d04a0 secure :-1 s=READY pgs=3305 cs=0 l=1 rev1=1 crypto rx=0x7fe4580098f0 tx=0x7fe458009c00 comp rx=0 tx=0).stop 2023-10-16T13:49:28.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.712+0000 7fe46a60c700 1 -- 172.21.15.16:0/3258573055 shutdown_connections 2023-10-16T13:49:28.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.712+0000 7fe46a60c700 1 --2- 172.21.15.16:0/3258573055 >> v2:172.21.15.49:3300/0 conn(0x7fe4640ce380 0x7fe4640d09e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:28.718 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.712+0000 7fe46a60c700 1 --2- 172.21.15.16:0/3258573055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4640b2b50 0x7fe4640d04a0 unknown :-1 s=CLOSED pgs=3305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:28.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.712+0000 7fe46a60c700 1 -- 172.21.15.16:0/3258573055 >> 172.21.15.16:0/3258573055 conn(0x7fe4640d14b0 msgr2=0x7fe4640d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:28.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.712+0000 7fe46a60c700 1 -- 172.21.15.16:0/3258573055 shutdown_connections 2023-10-16T13:49:28.719 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.712+0000 7fe46a60c700 1 -- 172.21.15.16:0/3258573055 wait complete. 2023-10-16T13:49:28.721 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.713+0000 7fe46a60c700 1 Processor -- start 2023-10-16T13:49:28.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.713+0000 7fe46a60c700 1 -- start start 2023-10-16T13:49:28.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.714+0000 7fe46a60c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4640b2b50 0x7fe4640c8580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:28.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.714+0000 7fe46a60c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe4640cdfa0 0x7fe4641f65d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:28.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.714+0000 7fe46a60c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe4640ce380 0x7fe4641f6b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:28.722 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.714+0000 7fe46a60c700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4640cb740 con 0x7fe4640b2b50 2023-10-16T13:49:28.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.714+0000 7fe46a60c700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4640c8ac0 con 0x7fe4640ce380 2023-10-16T13:49:28.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.714+0000 7fe46a60c700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4640c8cb0 con 0x7fe4640cdfa0 2023-10-16T13:49:28.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.714+0000 7fe463fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4640b2b50 0x7fe4640c8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:28.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.714+0000 7fe463fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4640b2b50 0x7fe4640c8580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60194/0 (socket says 172.21.15.16:60194) 2023-10-16T13:49:28.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.714+0000 7fe463fff700 1 -- 172.21.15.16:0/1305923656 learned_addr learned my addr 172.21.15.16:0/1305923656 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:28.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.714+0000 7fe468ba9700 1 --2- 172.21.15.16:0/1305923656 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe4640ce380 0x7fe4641f6b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:28.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.714+0000 7fe4637fe700 1 --2- 172.21.15.16:0/1305923656 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe4640cdfa0 0x7fe4641f65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:28.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.714+0000 7fe463fff700 1 -- 172.21.15.16:0/1305923656 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe4640cdfa0 msgr2=0x7fe4641f65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:28.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.714+0000 7fe463fff700 1 --2- 172.21.15.16:0/1305923656 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe4640cdfa0 0x7fe4641f65d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:28.724 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.714+0000 7fe463fff700 1 -- 172.21.15.16:0/1305923656 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe4640ce380 msgr2=0x7fe4641f6b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:28.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.714+0000 7fe463fff700 1 --2- 172.21.15.16:0/1305923656 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe4640ce380 0x7fe4641f6b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:28.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.714+0000 7fe463fff700 1 -- 172.21.15.16:0/1305923656 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe458009550 con 0x7fe4640b2b50 2023-10-16T13:49:28.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.714+0000 7fe463fff700 1 --2- 172.21.15.16:0/1305923656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4640b2b50 0x7fe4640c8580 secure :-1 s=READY pgs=3306 cs=0 l=1 rev1=1 crypto rx=0x7fe45400c4d0 tx=0x7fe45400c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:28.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.715+0000 7fe460ff9700 1 -- 172.21.15.16:0/1305923656 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe454011790 con 0x7fe4640b2b50 2023-10-16T13:49:28.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.715+0000 7fe460ff9700 1 -- 172.21.15.16:0/1305923656 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4540118f0 con 0x7fe4640b2b50 2023-10-16T13:49:28.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.715+0000 7fe46a60c700 1 -- 172.21.15.16:0/1305923656 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4640cada0 con 0x7fe4640b2b50 2023-10-16T13:49:28.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.715+0000 7fe460ff9700 1 -- 172.21.15.16:0/1305923656 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe454011ca0 con 0x7fe4640b2b50 2023-10-16T13:49:28.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.715+0000 7fe46a60c700 1 -- 172.21.15.16:0/1305923656 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4640cb360 con 0x7fe4640b2b50 2023-10-16T13:49:28.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.716+0000 7fe46a60c700 1 -- 172.21.15.16:0/1305923656 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe46404ebb0 con 0x7fe4640b2b50 2023-10-16T13:49:28.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.716+0000 7fe460ff9700 1 -- 172.21.15.16:0/1305923656 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 173) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fe45401c460 con 0x7fe4640b2b50 2023-10-16T13:49:28.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.717+0000 7fe460ff9700 1 -- 172.21.15.16:0/1305923656 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(280..280 src has 1..280) v4 ==== 6974+0+0 (secure 0 0 0) 0x7fe454089260 con 0x7fe4640b2b50 2023-10-16T13:49:28.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.721+0000 7fe460ff9700 1 -- 172.21.15.16:0/1305923656 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe454054270 con 0x7fe4640b2b50 2023-10-16T13:49:28.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.835+0000 7fe460ff9700 1 -- 172.21.15.16:0/1305923656 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 174) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fe454053d20 con 0x7fe4640b2b50 2023-10-16T13:49:28.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.836+0000 7fe460ff9700 1 --2- 172.21.15.16:0/1305923656 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7fe450070ed0 0x7fe4500732c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:28.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.837+0000 7fe4637fe700 1 --2- 172.21.15.16:0/1305923656 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7fe450070ed0 0x7fe4500732c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:28.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.838+0000 7fe4637fe700 1 --2- 172.21.15.16:0/1305923656 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7fe450070ed0 0x7fe4500732c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fe458009130 tx=0x7fe458008030 comp rx=0 tx=0).ready entity=mgr.13935 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:28.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.977+0000 7fe46a60c700 1 -- 172.21.15.16:0/1305923656 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fe4640cd8e0 con 0x7fe4640b2b50 2023-10-16T13:49:28.980 INFO:tasks.ceph.mon.a.smithi016.stderr:2023-10-16T13:49:28.977+0000 7f9fc8535700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-16T13:49:28.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.978+0000 7fe460ff9700 1 -- 172.21.15.16:0/1305923656 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v156) v1 ==== 161+0+0 (secure 0 0 0) 0x7fe4540873b0 con 0x7fe4640b2b50 2023-10-16T13:49:28.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.981+0000 7fe46a60c700 1 -- 172.21.15.16:0/1305923656 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7fe450070ed0 msgr2=0x7fe4500732c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:28.984 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.981+0000 7fe46a60c700 1 --2- 172.21.15.16:0/1305923656 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7fe450070ed0 0x7fe4500732c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fe458009130 tx=0x7fe458008030 comp rx=0 tx=0).stop 2023-10-16T13:49:28.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.981+0000 7fe46a60c700 1 -- 172.21.15.16:0/1305923656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4640b2b50 msgr2=0x7fe4640c8580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:28.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.981+0000 7fe46a60c700 1 --2- 172.21.15.16:0/1305923656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4640b2b50 0x7fe4640c8580 secure :-1 s=READY pgs=3306 cs=0 l=1 rev1=1 crypto rx=0x7fe45400c4d0 tx=0x7fe45400c7e0 comp rx=0 tx=0).stop 2023-10-16T13:49:28.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.981+0000 7fe46a60c700 1 -- 172.21.15.16:0/1305923656 shutdown_connections 2023-10-16T13:49:28.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.981+0000 7fe46a60c700 1 --2- 172.21.15.16:0/1305923656 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7fe450070ed0 0x7fe4500732c0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:28.985 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.981+0000 7fe46a60c700 1 --2- 172.21.15.16:0/1305923656 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe4640ce380 0x7fe4641f6b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:28.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.981+0000 7fe46a60c700 1 --2- 172.21.15.16:0/1305923656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4640b2b50 0x7fe4640c8580 unknown :-1 s=CLOSED pgs=3306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:28.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.981+0000 7fe46a60c700 1 --2- 172.21.15.16:0/1305923656 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe4640cdfa0 0x7fe4641f65d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:28.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.982+0000 7fe46a60c700 1 -- 172.21.15.16:0/1305923656 >> 172.21.15.16:0/1305923656 conn(0x7fe4640d14b0 msgr2=0x7fe46406ec80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:28.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.982+0000 7fe46a60c700 1 -- 172.21.15.16:0/1305923656 shutdown_connections 2023-10-16T13:49:28.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:28.982+0000 7fe46a60c700 1 -- 172.21.15.16:0/1305923656 wait complete. 2023-10-16T13:49:28.987 INFO:teuthology.orchestra.run.smithi016.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-16T13:49:28.996 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-16T13:49:29.025 INFO:tasks.ceph.mds.b.smithi049.stdout:starting mds.b at 2023-10-16T13:49:29.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.121+0000 7fa284512700 1 Processor -- start 2023-10-16T13:49:29.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.121+0000 7fa284512700 1 -- start start 2023-10-16T13:49:29.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.122+0000 7fa284512700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fa27c159de0 0x7fa27c15a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:29.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.122+0000 7fa284512700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa27c0cf4c0 0x7fa27c06aac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:29.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.122+0000 7fa284512700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa27c06b000 0x7fa27c06b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:29.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.122+0000 7fa284512700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa27c15ea20 con 0x7fa27c06b000 2023-10-16T13:49:29.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.122+0000 7fa284512700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa27c15dda0 con 0x7fa27c159de0 2023-10-16T13:49:29.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.122+0000 7fa284512700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa27c15e080 con 0x7fa27c0cf4c0 2023-10-16T13:49:29.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.122+0000 7fa282aaf700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa27c06b000 0x7fa27c06b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:29.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.122+0000 7fa282aaf700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa27c06b000 0x7fa27c06b4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60228/0 (socket says 172.21.15.16:60228) 2023-10-16T13:49:29.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.122+0000 7fa282aaf700 1 -- 172.21.15.16:0/2167903403 learned_addr learned my addr 172.21.15.16:0/2167903403 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:29.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.122+0000 7fa2822ae700 1 --2- 172.21.15.16:0/2167903403 >> v2:172.21.15.49:3300/0 conn(0x7fa27c159de0 0x7fa27c15a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:29.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.122+0000 7fa281aad700 1 --2- 172.21.15.16:0/2167903403 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa27c0cf4c0 0x7fa27c06aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:29.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.122+0000 7fa282aaf700 1 -- 172.21.15.16:0/2167903403 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa27c0cf4c0 msgr2=0x7fa27c06aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:29.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.122+0000 7fa282aaf700 1 --2- 172.21.15.16:0/2167903403 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa27c0cf4c0 0x7fa27c06aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:29.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.122+0000 7fa282aaf700 1 -- 172.21.15.16:0/2167903403 >> v2:172.21.15.49:3300/0 conn(0x7fa27c159de0 msgr2=0x7fa27c15a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:29.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.122+0000 7fa282aaf700 1 --2- 172.21.15.16:0/2167903403 >> v2:172.21.15.49:3300/0 conn(0x7fa27c159de0 0x7fa27c15a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:29.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.122+0000 7fa282aaf700 1 -- 172.21.15.16:0/2167903403 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa27c15e360 con 0x7fa27c06b000 2023-10-16T13:49:29.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.122+0000 7fa2822ae700 1 --2- 172.21.15.16:0/2167903403 >> v2:172.21.15.49:3300/0 conn(0x7fa27c159de0 0x7fa27c15a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:49:29.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.122+0000 7fa282aaf700 1 --2- 172.21.15.16:0/2167903403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa27c06b000 0x7fa27c06b4b0 secure :-1 s=READY pgs=3308 cs=0 l=1 rev1=1 crypto rx=0x7fa27c0cdf10 tx=0x7fa27400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26684aa37857e875 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:29.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.123+0000 7fa2812ac700 1 -- 172.21.15.16:0/2167903403 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa27400e070 con 0x7fa27c06b000 2023-10-16T13:49:29.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.123+0000 7fa2812ac700 1 -- 172.21.15.16:0/2167903403 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa27400b910 con 0x7fa27c06b000 2023-10-16T13:49:29.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.123+0000 7fa2812ac700 1 -- 172.21.15.16:0/2167903403 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa27400bcc0 con 0x7fa27c06b000 2023-10-16T13:49:29.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.123+0000 7fa284512700 1 -- 172.21.15.16:0/2167903403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa27c06b000 msgr2=0x7fa27c06b4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:29.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.123+0000 7fa284512700 1 --2- 172.21.15.16:0/2167903403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa27c06b000 0x7fa27c06b4b0 secure :-1 s=READY pgs=3308 cs=0 l=1 rev1=1 crypto rx=0x7fa27c0cdf10 tx=0x7fa27400b330 comp rx=0 tx=0).stop 2023-10-16T13:49:29.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.123+0000 7fa284512700 1 -- 172.21.15.16:0/2167903403 shutdown_connections 2023-10-16T13:49:29.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.123+0000 7fa284512700 1 --2- 172.21.15.16:0/2167903403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa27c06b000 0x7fa27c06b4b0 unknown :-1 s=CLOSED pgs=3308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:29.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.123+0000 7fa284512700 1 --2- 172.21.15.16:0/2167903403 >> v2:172.21.15.49:3300/0 conn(0x7fa27c159de0 0x7fa27c15a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:29.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.123+0000 7fa284512700 1 --2- 172.21.15.16:0/2167903403 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa27c0cf4c0 0x7fa27c06aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:29.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.123+0000 7fa284512700 1 -- 172.21.15.16:0/2167903403 >> 172.21.15.16:0/2167903403 conn(0x7fa27c0cc5e0 msgr2=0x7fa27c0cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:29.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.123+0000 7fa284512700 1 -- 172.21.15.16:0/2167903403 shutdown_connections 2023-10-16T13:49:29.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.123+0000 7fa284512700 1 -- 172.21.15.16:0/2167903403 wait complete. 2023-10-16T13:49:29.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.125+0000 7fa284512700 1 Processor -- start 2023-10-16T13:49:29.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.125+0000 7fa284512700 1 -- start start 2023-10-16T13:49:29.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.125+0000 7fa284512700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa27c06b000 0x7fa27c14f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:29.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.125+0000 7fa284512700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa27c0cf4c0 0x7fa27c1573b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:29.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.125+0000 7fa284512700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa27c159de0 0x7fa27c0b44f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:29.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.125+0000 7fa284512700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa27c14f9f0 con 0x7fa27c159de0 2023-10-16T13:49:29.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.125+0000 7fa284512700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa27c0b4eb0 con 0x7fa27c0cf4c0 2023-10-16T13:49:29.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.125+0000 7fa284512700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa27c0b5190 con 0x7fa27c06b000 2023-10-16T13:49:29.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.125+0000 7fa282aaf700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa27c159de0 0x7fa27c0b44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:29.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.125+0000 7fa282aaf700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa27c159de0 0x7fa27c0b44f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60234/0 (socket says 172.21.15.16:60234) 2023-10-16T13:49:29.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.125+0000 7fa282aaf700 1 -- 172.21.15.16:0/4246528936 learned_addr learned my addr 172.21.15.16:0/4246528936 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:29.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.126+0000 7fa281aad700 1 --2- 172.21.15.16:0/4246528936 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa27c0cf4c0 0x7fa27c1573b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:29.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.126+0000 7fa2822ae700 1 --2- 172.21.15.16:0/4246528936 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa27c06b000 0x7fa27c14f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:29.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.126+0000 7fa282aaf700 1 -- 172.21.15.16:0/4246528936 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa27c06b000 msgr2=0x7fa27c14f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:29.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.126+0000 7fa282aaf700 1 --2- 172.21.15.16:0/4246528936 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa27c06b000 0x7fa27c14f4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:29.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.126+0000 7fa282aaf700 1 -- 172.21.15.16:0/4246528936 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa27c0cf4c0 msgr2=0x7fa27c1573b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:29.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.126+0000 7fa282aaf700 1 --2- 172.21.15.16:0/4246528936 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa27c0cf4c0 0x7fa27c1573b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:29.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.126+0000 7fa282aaf700 1 -- 172.21.15.16:0/4246528936 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2780094a0 con 0x7fa27c159de0 2023-10-16T13:49:29.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.126+0000 7fa281aad700 1 --2- 172.21.15.16:0/4246528936 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa27c0cf4c0 0x7fa27c1573b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:49:29.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.126+0000 7fa282aaf700 1 --2- 172.21.15.16:0/4246528936 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa27c159de0 0x7fa27c0b44f0 secure :-1 s=READY pgs=3309 cs=0 l=1 rev1=1 crypto rx=0x7fa27c0cd170 tx=0x7fa274004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:29.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.126+0000 7fa272ffd700 1 -- 172.21.15.16:0/4246528936 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa27400ce80 con 0x7fa27c159de0 2023-10-16T13:49:29.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.126+0000 7fa272ffd700 1 -- 172.21.15.16:0/4246528936 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa274004250 con 0x7fa27c159de0 2023-10-16T13:49:29.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.126+0000 7fa284512700 1 -- 172.21.15.16:0/4246528936 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa274009b30 con 0x7fa27c159de0 2023-10-16T13:49:29.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.126+0000 7fa272ffd700 1 -- 172.21.15.16:0/4246528936 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa274004600 con 0x7fa27c159de0 2023-10-16T13:49:29.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.126+0000 7fa284512700 1 -- 172.21.15.16:0/4246528936 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa27c156e90 con 0x7fa27c159de0 2023-10-16T13:49:29.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.127+0000 7fa284512700 1 -- 172.21.15.16:0/4246528936 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa27c04ebb0 con 0x7fa27c159de0 2023-10-16T13:49:29.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.128+0000 7fa272ffd700 1 -- 172.21.15.16:0/4246528936 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 174) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fa274004760 con 0x7fa27c159de0 2023-10-16T13:49:29.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.128+0000 7fa272ffd700 1 --2- 172.21.15.16:0/4246528936 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7fa26006ad50 0x7fa26006d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:29.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.129+0000 7fa272ffd700 1 -- 172.21.15.16:0/4246528936 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(281..281 src has 1..281) v4 ==== 7060+0+0 (secure 0 0 0) 0x7fa27408aba0 con 0x7fa27c159de0 2023-10-16T13:49:29.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.129+0000 7fa2822ae700 1 --2- 172.21.15.16:0/4246528936 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7fa26006ad50 0x7fa26006d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:29.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.129+0000 7fa2822ae700 1 --2- 172.21.15.16:0/4246528936 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7fa26006ad50 0x7fa26006d210 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa278009d00 tx=0x7fa278008040 comp rx=0 tx=0).ready entity=mgr.13935 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:29.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.132+0000 7fa272ffd700 1 -- 172.21.15.16:0/4246528936 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa274055c20 con 0x7fa27c159de0 2023-10-16T13:49:29.389 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.386+0000 7fa284512700 1 -- 172.21.15.16:0/4246528936 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fa27c0ce3d0 con 0x7fa27c159de0 2023-10-16T13:49:29.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.980+0000 7fa272ffd700 1 -- 172.21.15.16:0/4246528936 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v157) v1 ==== 104+0+0 (secure 0 0 0) 0x7fa274059240 con 0x7fa27c159de0 2023-10-16T13:49:29.986 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.983+0000 7fa284512700 1 -- 172.21.15.16:0/4246528936 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7fa26006ad50 msgr2=0x7fa26006d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:29.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.983+0000 7fa284512700 1 --2- 172.21.15.16:0/4246528936 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7fa26006ad50 0x7fa26006d210 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa278009d00 tx=0x7fa278008040 comp rx=0 tx=0).stop 2023-10-16T13:49:29.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.983+0000 7fa284512700 1 -- 172.21.15.16:0/4246528936 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa27c159de0 msgr2=0x7fa27c0b44f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:29.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.983+0000 7fa284512700 1 --2- 172.21.15.16:0/4246528936 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa27c159de0 0x7fa27c0b44f0 secure :-1 s=READY pgs=3309 cs=0 l=1 rev1=1 crypto rx=0x7fa27c0cd170 tx=0x7fa274004ad0 comp rx=0 tx=0).stop 2023-10-16T13:49:29.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.984+0000 7fa284512700 1 -- 172.21.15.16:0/4246528936 shutdown_connections 2023-10-16T13:49:29.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.984+0000 7fa284512700 1 --2- 172.21.15.16:0/4246528936 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7fa26006ad50 0x7fa26006d210 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:29.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.984+0000 7fa284512700 1 --2- 172.21.15.16:0/4246528936 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa27c159de0 0x7fa27c0b44f0 unknown :-1 s=CLOSED pgs=3309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:29.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.984+0000 7fa284512700 1 --2- 172.21.15.16:0/4246528936 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa27c06b000 0x7fa27c14f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:29.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.984+0000 7fa284512700 1 --2- 172.21.15.16:0/4246528936 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa27c0cf4c0 0x7fa27c1573b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:29.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.984+0000 7fa284512700 1 -- 172.21.15.16:0/4246528936 >> 172.21.15.16:0/4246528936 conn(0x7fa27c0cc5e0 msgr2=0x7fa27c0710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:29.988 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.984+0000 7fa284512700 1 -- 172.21.15.16:0/4246528936 shutdown_connections 2023-10-16T13:49:29.989 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:29.984+0000 7fa284512700 1 -- 172.21.15.16:0/4246528936 wait complete. 2023-10-16T13:49:30.001 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-16T13:49:30.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.133+0000 7f3a90c22700 1 Processor -- start 2023-10-16T13:49:30.136 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.133+0000 7f3a90c22700 1 -- start start 2023-10-16T13:49:30.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.134+0000 7f3a90c22700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3a8c160e60 0x7f3a8c15e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:30.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.134+0000 7f3a90c22700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a8c0cdeb0 con 0x7f3a8c06a7e0 2023-10-16T13:49:30.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.134+0000 7f3a90c22700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a8c1613e0 con 0x7f3a8c06d980 2023-10-16T13:49:30.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.134+0000 7f3a90c22700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a8c15f6c0 con 0x7f3a8c160e60 2023-10-16T13:49:30.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.134+0000 7f3a8ad9d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f3a8c06a7e0 0x7f3a8c06abc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49534/0 (socket says 172.21.15.16:49534) 2023-10-16T13:49:30.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.134+0000 7f3a8ad9d700 1 -- 172.21.15.16:0/2194351909 learned_addr learned my addr 172.21.15.16:0/2194351909 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:49:30.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.134+0000 7f3a8a59c700 1 --2- 172.21.15.16:0/2194351909 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3a8c160e60 0x7f3a8c15e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:30.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.134+0000 7f3a8959a700 1 -- 172.21.15.16:0/2194351909 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3099576273 0 0) 0x7f3a8c0cdeb0 con 0x7f3a8c06a7e0 2023-10-16T13:49:30.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.134+0000 7f3a8959a700 1 -- 172.21.15.16:0/2194351909 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a740035b0 con 0x7f3a8c06a7e0 2023-10-16T13:49:30.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.134+0000 7f3a8959a700 1 -- 172.21.15.16:0/2194351909 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3933198129 0 0) 0x7f3a8c1613e0 con 0x7f3a8c06d980 2023-10-16T13:49:30.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.134+0000 7f3a8959a700 1 -- 172.21.15.16:0/2194351909 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a8c0cdeb0 con 0x7f3a8c06d980 2023-10-16T13:49:30.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.134+0000 7f3a8959a700 1 -- 172.21.15.16:0/2194351909 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4278911897 0 0) 0x7f3a740035b0 con 0x7f3a8c06a7e0 2023-10-16T13:49:30.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.134+0000 7f3a8959a700 1 -- 172.21.15.16:0/2194351909 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3a8c160e60 msgr2=0x7f3a8c15e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:30.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.134+0000 7f3a8959a700 1 --2- 172.21.15.16:0/2194351909 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3a8c160e60 0x7f3a8c15e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:30.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.135+0000 7f3a8959a700 1 -- 172.21.15.16:0/2194351909 >> v1:172.21.15.49:6789/0 conn(0x7f3a8c06d980 legacy=0x7f3a8c15efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:30.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.135+0000 7f3a8959a700 1 -- 172.21.15.16:0/2194351909 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a8c1582d0 con 0x7f3a8c06a7e0 2023-10-16T13:49:30.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.135+0000 7f3a8959a700 1 -- 172.21.15.16:0/2194351909 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f3a840019f0 con 0x7f3a8c06a7e0 2023-10-16T13:49:30.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.135+0000 7f3a8959a700 1 -- 172.21.15.16:0/2194351909 >> v1:172.21.15.16:6789/0 conn(0x7f3a8c06a7e0 legacy=0x7f3a8c06abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:30.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.135+0000 7f3a8959a700 1 --2- 172.21.15.16:0/2194351909 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3a74003bc0 0x7f3a74003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:30.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.135+0000 7f3a8959a700 1 --2- 172.21.15.16:0/2194351909 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3a8c160e60 0x7f3a74004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:30.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.135+0000 7f3a8959a700 1 --2- 172.21.15.16:0/2194351909 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a74004d90 0x7f3a74005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:30.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.135+0000 7f3a8959a700 1 -- 172.21.15.16:0/2194351909 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a8c15f6c0 con 0x7f3a74004d90 2023-10-16T13:49:30.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.135+0000 7f3a8959a700 1 -- 172.21.15.16:0/2194351909 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a7400bac0 con 0x7f3a74003bc0 2023-10-16T13:49:30.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.135+0000 7f3a8959a700 1 -- 172.21.15.16:0/2194351909 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a7400bc30 con 0x7f3a8c160e60 2023-10-16T13:49:30.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.135+0000 7f3a8ad9d700 1 --2- 172.21.15.16:0/2194351909 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a74004d90 0x7f3a74005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:30.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.135+0000 7f3a8a59c700 1 --2- 172.21.15.16:0/2194351909 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3a74003bc0 0x7f3a74003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:30.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.135+0000 7f3a89d9b700 1 --2- 172.21.15.16:0/2194351909 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3a8c160e60 0x7f3a74004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:30.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.135+0000 7f3a8ad9d700 1 -- 172.21.15.16:0/2194351909 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3a8c160e60 msgr2=0x7f3a74004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:30.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.135+0000 7f3a8ad9d700 1 --2- 172.21.15.16:0/2194351909 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3a8c160e60 0x7f3a74004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:30.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.135+0000 7f3a8ad9d700 1 -- 172.21.15.16:0/2194351909 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3a74003bc0 msgr2=0x7f3a74003fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:30.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.135+0000 7f3a8ad9d700 1 --2- 172.21.15.16:0/2194351909 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3a74003bc0 0x7f3a74003fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:30.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.135+0000 7f3a8ad9d700 1 -- 172.21.15.16:0/2194351909 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a740035b0 con 0x7f3a74004d90 2023-10-16T13:49:30.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.135+0000 7f3a8ad9d700 1 --2- 172.21.15.16:0/2194351909 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a74004d90 0x7f3a74005210 secure :-1 s=READY pgs=3313 cs=0 l=1 rev1=1 crypto rx=0x7f3a8400a7b0 tx=0x7f3a74005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b59c88fd9981485 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:30.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.135+0000 7f3a8959a700 1 -- 172.21.15.16:0/2194351909 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a84012600 con 0x7f3a74004d90 2023-10-16T13:49:30.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.135+0000 7f3a8959a700 1 -- 172.21.15.16:0/2194351909 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a840127a0 con 0x7f3a74004d90 2023-10-16T13:49:30.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.135+0000 7f3a8959a700 1 -- 172.21.15.16:0/2194351909 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a84012b50 con 0x7f3a74004d90 2023-10-16T13:49:30.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.136+0000 7f3a90c22700 1 -- 172.21.15.16:0/2194351909 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a74004d90 msgr2=0x7f3a74005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:30.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.136+0000 7f3a90c22700 1 --2- 172.21.15.16:0/2194351909 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a74004d90 0x7f3a74005210 secure :-1 s=READY pgs=3313 cs=0 l=1 rev1=1 crypto rx=0x7f3a8400a7b0 tx=0x7f3a74005880 comp rx=0 tx=0).stop 2023-10-16T13:49:30.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.136+0000 7f3a8ad9d700 1 -- 172.21.15.16:0/2194351909 reap_dead start 2023-10-16T13:49:30.146 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.136+0000 7f3a90c22700 1 -- 172.21.15.16:0/2194351909 shutdown_connections 2023-10-16T13:49:30.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.136+0000 7f3a90c22700 1 -- 172.21.15.16:0/2194351909 >> 172.21.15.16:0/2194351909 conn(0x7f3a8c06f430 msgr2=0x7f3a8c06f810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:30.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.136+0000 7f3a90c22700 1 -- 172.21.15.16:0/2194351909 shutdown_connections 2023-10-16T13:49:30.147 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.136+0000 7f3a90c22700 1 -- 172.21.15.16:0/2194351909 wait complete. 2023-10-16T13:49:30.149 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.137+0000 7f3a90c22700 1 Processor -- start 2023-10-16T13:49:30.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.137+0000 7f3a90c22700 1 -- start start 2023-10-16T13:49:30.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.138+0000 7f3a90c22700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3a74004d90 0x7f3a8c156b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:30.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.138+0000 7f3a90c22700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a8c0b39a0 0x7f3a8c06e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:30.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.138+0000 7f3a90c22700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3a8c06e5c0 0x7f3a8c0b4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:30.150 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.138+0000 7f3a90c22700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a8c06ea40 con 0x7f3a8c0b39a0 2023-10-16T13:49:30.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.138+0000 7f3a90c22700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a8c0b46a0 con 0x7f3a8c06e5c0 2023-10-16T13:49:30.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.138+0000 7f3a90c22700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a8c0b3510 con 0x7f3a74004d90 2023-10-16T13:49:30.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.138+0000 7f3a89d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a8c0b39a0 0x7f3a8c06e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:30.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.138+0000 7f3a89d9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a8c0b39a0 0x7f3a8c06e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60270/0 (socket says 172.21.15.16:60270) 2023-10-16T13:49:30.151 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.138+0000 7f3a89d9b700 1 -- 172.21.15.16:0/2023821387 learned_addr learned my addr 172.21.15.16:0/2023821387 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:30.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.138+0000 7f3a8ad9d700 1 --2- 172.21.15.16:0/2023821387 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3a8c06e5c0 0x7f3a8c0b4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:30.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.138+0000 7f3a8a59c700 1 --2- 172.21.15.16:0/2023821387 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3a74004d90 0x7f3a8c156b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:30.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.139+0000 7f3a89d9b700 1 -- 172.21.15.16:0/2023821387 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3a74004d90 msgr2=0x7f3a8c156b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:30.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.139+0000 7f3a89d9b700 1 --2- 172.21.15.16:0/2023821387 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3a74004d90 0x7f3a8c156b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:30.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.139+0000 7f3a89d9b700 1 -- 172.21.15.16:0/2023821387 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3a8c06e5c0 msgr2=0x7f3a8c0b4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:30.152 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.139+0000 7f3a89d9b700 1 --2- 172.21.15.16:0/2023821387 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3a8c06e5c0 0x7f3a8c0b4ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:30.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.139+0000 7f3a89d9b700 1 -- 172.21.15.16:0/2023821387 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a8c0b2f00 con 0x7f3a8c0b39a0 2023-10-16T13:49:30.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.139+0000 7f3a89d9b700 1 --2- 172.21.15.16:0/2023821387 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a8c0b39a0 0x7f3a8c06e080 secure :-1 s=READY pgs=3314 cs=0 l=1 rev1=1 crypto rx=0x7f3a8000f810 tx=0x7f3a8000fb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:30.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.139+0000 7f3a7affd700 1 -- 172.21.15.16:0/2023821387 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a800052e0 con 0x7f3a8c0b39a0 2023-10-16T13:49:30.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.139+0000 7f3a7affd700 1 -- 172.21.15.16:0/2023821387 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a80005440 con 0x7f3a8c0b39a0 2023-10-16T13:49:30.153 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.139+0000 7f3a90c22700 1 -- 172.21.15.16:0/2023821387 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a8c0b58c0 con 0x7f3a8c0b39a0 2023-10-16T13:49:30.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.139+0000 7f3a7affd700 1 -- 172.21.15.16:0/2023821387 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a800038c0 con 0x7f3a8c0b39a0 2023-10-16T13:49:30.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.139+0000 7f3a90c22700 1 -- 172.21.15.16:0/2023821387 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a8c0b31a0 con 0x7f3a8c0b39a0 2023-10-16T13:49:30.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.140+0000 7f3a90c22700 1 -- 172.21.15.16:0/2023821387 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a8c04ec20 con 0x7f3a8c0b39a0 2023-10-16T13:49:30.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.140+0000 7f3a7affd700 1 -- 172.21.15.16:0/2023821387 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 174) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f3a80003a20 con 0x7f3a8c0b39a0 2023-10-16T13:49:30.154 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.141+0000 7f3a7affd700 1 --2- 172.21.15.16:0/2023821387 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f3a6c06ad50 0x7f3a6c06d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:30.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.141+0000 7f3a7affd700 1 -- 172.21.15.16:0/2023821387 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(281..281 src has 1..281) v4 ==== 7060+0+0 (secure 0 0 0) 0x7f3a80089500 con 0x7f3a8c0b39a0 2023-10-16T13:49:30.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.141+0000 7f3a8a59c700 1 --2- 172.21.15.16:0/2023821387 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f3a6c06ad50 0x7f3a6c06d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:30.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.141+0000 7f3a8a59c700 1 --2- 172.21.15.16:0/2023821387 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f3a6c06ad50 0x7f3a6c06d210 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c009c60 tx=0x7f3a7c009210 comp rx=0 tx=0).ready entity=mgr.13935 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:30.155 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.146+0000 7f3a7affd700 1 -- 172.21.15.16:0/2023821387 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3a80054450 con 0x7f3a8c0b39a0 2023-10-16T13:49:30.352 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.349+0000 7f3a90c22700 1 -- 172.21.15.16:0/2023821387 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f3a8c0b6260 con 0x7f3a8c0b39a0 2023-10-16T13:49:30.844 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:30.841+0000 7f3a7affd700 1 -- 172.21.15.16:0/2023821387 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 175) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f3a80003cd0 con 0x7f3a8c0b39a0 2023-10-16T13:49:31.232 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:31.229+0000 7f3a7affd700 1 -- 172.21.15.16:0/2023821387 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v282) v1 ==== 181+0+0 (secure 0 0 0) 0x7f3a80087470 con 0x7f3a8c0b39a0 2023-10-16T13:49:31.236 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:31.233+0000 7f3a90c22700 1 -- 172.21.15.16:0/2023821387 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f3a6c06ad50 msgr2=0x7f3a6c06d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:31.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:31.233+0000 7f3a90c22700 1 --2- 172.21.15.16:0/2023821387 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f3a6c06ad50 0x7f3a6c06d210 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c009c60 tx=0x7f3a7c009210 comp rx=0 tx=0).stop 2023-10-16T13:49:31.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:31.234+0000 7f3a90c22700 1 -- 172.21.15.16:0/2023821387 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a8c0b39a0 msgr2=0x7f3a8c06e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:31.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:31.234+0000 7f3a90c22700 1 --2- 172.21.15.16:0/2023821387 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a8c0b39a0 0x7f3a8c06e080 secure :-1 s=READY pgs=3314 cs=0 l=1 rev1=1 crypto rx=0x7f3a8000f810 tx=0x7f3a8000fb20 comp rx=0 tx=0).stop 2023-10-16T13:49:31.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:31.234+0000 7f3a90c22700 1 -- 172.21.15.16:0/2023821387 shutdown_connections 2023-10-16T13:49:31.237 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:31.234+0000 7f3a90c22700 1 --2- 172.21.15.16:0/2023821387 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f3a6c06ad50 0x7f3a6c06d210 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c009c60 tx=0x7f3a7c009210 comp rx=0 tx=0).stop 2023-10-16T13:49:31.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:31.234+0000 7f3a90c22700 1 --2- 172.21.15.16:0/2023821387 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3a8c06e5c0 0x7f3a8c0b4ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:31.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:31.234+0000 7f3a90c22700 1 --2- 172.21.15.16:0/2023821387 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f3a74004d90 0x7f3a8c156b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:31.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:31.234+0000 7f3a90c22700 1 --2- 172.21.15.16:0/2023821387 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a8c0b39a0 0x7f3a8c06e080 secure :-1 s=CLOSED pgs=3314 cs=0 l=1 rev1=1 crypto rx=0x7f3a8000f810 tx=0x7f3a8000fb20 comp rx=0 tx=0).stop 2023-10-16T13:49:31.238 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:31.234+0000 7f3a90c22700 1 -- 172.21.15.16:0/2023821387 >> 172.21.15.16:0/2023821387 conn(0x7f3a8c06f430 msgr2=0x7f3a8c160450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:31.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:31.236+0000 7f3a90c22700 1 -- 172.21.15.16:0/2023821387 shutdown_connections 2023-10-16T13:49:31.239 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:31.237+0000 7f3a90c22700 1 -- 172.21.15.16:0/2023821387 wait complete. 2023-10-16T13:49:31.241 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' removed 2023-10-16T13:49:31.256 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-16T13:49:31.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:31.389+0000 7fdba23b8700 1 Processor -- start 2023-10-16T13:49:31.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:31.389+0000 7fdba23b8700 1 -- start start 2023-10-16T13:49:31.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:31.390+0000 7fdba23b8700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fdb9c0cf2c0 0x7fdb9c0aef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:31.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:31.390+0000 7fdba23b8700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdb9c0cd600 0x7fdb9c0cb6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:31.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:31.390+0000 7fdba23b8700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb9c0c49b0 con 0x7fdb9c0cbc10 2023-10-16T13:49:31.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:31.390+0000 7fdba23b8700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb9c0c4c20 con 0x7fdb9c0cf2c0 2023-10-16T13:49:31.394 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:31.390+0000 7fdba23b8700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb9c0c4f00 con 0x7fdb9c0cd600 2023-10-16T13:49:31.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:31.390+0000 7fdb9b7fe700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdb9c0cd600 0x7fdb9c0cb6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:31.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:31.390+0000 7fdb9bfff700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fdb9c0cf2c0 0x7fdb9c0aef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:31.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:31.390+0000 7fdba0955700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7fdb9c0cbc10 0x7fdb9c0c5700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:56736/0 (socket says 172.21.15.16:56736) 2023-10-16T13:49:31.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:31.390+0000 7fdb9bfff700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fdb9c0cf2c0 0x7fdb9c0aef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:59338/0 (socket says 172.21.15.16:59338) 2023-10-16T13:49:31.395 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:31.390+0000 7fdba0955700 1 -- 172.21.15.16:0/882741376 learned_addr learned my addr 172.21.15.16:0/882741376 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:49:31.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:31.391+0000 7fdb9bfff700 1 -- 172.21.15.16:0/882741376 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdb9c0cd600 msgr2=0x7fdb9c0cb6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:31.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:31.391+0000 7fdb9bfff700 1 --2- 172.21.15.16:0/882741376 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdb9c0cd600 0x7fdb9c0cb6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:31.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:31.391+0000 7fdb9bfff700 1 -- 172.21.15.16:0/882741376 >> v1:172.21.15.49:6789/0 conn(0x7fdb9c0cbc10 legacy=0x7fdb9c0c5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:31.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:31.391+0000 7fdb9bfff700 1 -- 172.21.15.16:0/882741376 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb9c0c51e0 con 0x7fdb9c0cf2c0 2023-10-16T13:49:31.396 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:31.391+0000 7fdb9b7fe700 1 --2- 172.21.15.16:0/882741376 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdb9c0cd600 0x7fdb9c0cb6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:49:31.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:31.391+0000 7fdb9bfff700 1 --2- 172.21.15.16:0/882741376 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdb9c0cf2c0 0x7fdb9c0aef00 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7fdb84009840 tx=0x7fdb84009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=abf54e43bbf7e796 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:31.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:31.391+0000 7fdb9affd700 1 -- 172.21.15.16:0/882741376 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb84011830 con 0x7fdb9c0cf2c0 2023-10-16T13:49:31.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:31.391+0000 7fdb9affd700 1 -- 172.21.15.16:0/882741376 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb84011990 con 0x7fdb9c0cf2c0 2023-10-16T13:49:31.397 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:31.391+0000 7fdb9affd700 1 -- 172.21.15.16:0/882741376 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb84011d20 con 0x7fdb9c0cf2c0 2023-10-16T13:49:31.400 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:31.392+0000 7fdba23b8700 1 -- 172.21.15.16:0/882741376 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdb9c0cf2c0 msgr2=0x7fdb9c0aef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:31.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:31.392+0000 7fdba23b8700 1 --2- 172.21.15.16:0/882741376 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdb9c0cf2c0 0x7fdb9c0aef00 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7fdb84009840 tx=0x7fdb84009b50 comp rx=0 tx=0).stop 2023-10-16T13:49:31.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:31.392+0000 7fdba23b8700 1 -- 172.21.15.16:0/882741376 shutdown_connections 2023-10-16T13:49:31.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:31.392+0000 7fdba23b8700 1 --2- 172.21.15.16:0/882741376 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdb9c0cf2c0 0x7fdb9c0aef00 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:31.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:31.392+0000 7fdba23b8700 1 --2- 172.21.15.16:0/882741376 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdb9c0cd600 0x7fdb9c0cb6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:31.401 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:31.392+0000 7fdba23b8700 1 -- 172.21.15.16:0/882741376 >> 172.21.15.16:0/882741376 conn(0x7fdb9c0cc4b0 msgr2=0x7fdb9c0cc8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:31.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:31.392+0000 7fdba23b8700 1 -- 172.21.15.16:0/882741376 shutdown_connections 2023-10-16T13:49:31.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:31.392+0000 7fdba23b8700 1 -- 172.21.15.16:0/882741376 wait complete. 2023-10-16T13:49:31.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:31.393+0000 7fdba23b8700 1 Processor -- start 2023-10-16T13:49:31.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:31.393+0000 7fdba23b8700 1 -- start start 2023-10-16T13:49:31.402 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:31.394+0000 7fdba23b8700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdb9c0cbc10 0x7fdb9c0c88b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:31.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:31.394+0000 7fdba23b8700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdb9c0cd600 0x7fdb9c151020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:31.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:31.394+0000 7fdba23b8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb9c0cf2c0 0x7fdb9c151560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:31.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:31.394+0000 7fdba23b8700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb9c0b2840 con 0x7fdb9c0cf2c0 2023-10-16T13:49:31.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:31.394+0000 7fdba23b8700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb9c159fe0 con 0x7fdb9c0cd600 2023-10-16T13:49:31.403 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:31.394+0000 7fdba23b8700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb9c151aa0 con 0x7fdb9c0cbc10 2023-10-16T13:49:31.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:31.394+0000 7fdba0955700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb9c0cf2c0 0x7fdb9c151560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:31.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:31.394+0000 7fdba0955700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb9c0cf2c0 0x7fdb9c151560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60286/0 (socket says 172.21.15.16:60286) 2023-10-16T13:49:31.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:31.394+0000 7fdba0955700 1 -- 172.21.15.16:0/1961220716 learned_addr learned my addr 172.21.15.16:0/1961220716 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:31.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:31.394+0000 7fdb9b7fe700 1 --2- 172.21.15.16:0/1961220716 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdb9c0cd600 0x7fdb9c151020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:31.404 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:31.394+0000 7fdb9bfff700 1 --2- 172.21.15.16:0/1961220716 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdb9c0cbc10 0x7fdb9c0c88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:31.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:31.394+0000 7fdba0955700 1 -- 172.21.15.16:0/1961220716 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdb9c0cbc10 msgr2=0x7fdb9c0c88b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:31.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:31.394+0000 7fdba0955700 1 --2- 172.21.15.16:0/1961220716 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdb9c0cbc10 0x7fdb9c0c88b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:31.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:31.394+0000 7fdba0955700 1 -- 172.21.15.16:0/1961220716 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdb9c0cd600 msgr2=0x7fdb9c151020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:31.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:31.394+0000 7fdba0955700 1 --2- 172.21.15.16:0/1961220716 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdb9c0cd600 0x7fdb9c151020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:31.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:31.394+0000 7fdba0955700 1 -- 172.21.15.16:0/1961220716 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb8c00b740 con 0x7fdb9c0cf2c0 2023-10-16T13:49:31.405 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:31.394+0000 7fdba0955700 1 --2- 172.21.15.16:0/1961220716 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb9c0cf2c0 0x7fdb9c151560 secure :-1 s=READY pgs=3315 cs=0 l=1 rev1=1 crypto rx=0x7fdb9400cbc0 tx=0x7fdb9400ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:31.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:31.395+0000 7fdb98ff9700 1 -- 172.21.15.16:0/1961220716 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb9400a980 con 0x7fdb9c0cf2c0 2023-10-16T13:49:31.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:31.395+0000 7fdb98ff9700 1 -- 172.21.15.16:0/1961220716 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb9400aae0 con 0x7fdb9c0cf2c0 2023-10-16T13:49:31.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:31.395+0000 7fdba23b8700 1 -- 172.21.15.16:0/1961220716 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb840094a0 con 0x7fdb9c0cf2c0 2023-10-16T13:49:31.406 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:31.395+0000 7fdb98ff9700 1 -- 172.21.15.16:0/1961220716 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb9400ae50 con 0x7fdb9c0cf2c0 2023-10-16T13:49:31.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:31.395+0000 7fdba23b8700 1 -- 172.21.15.16:0/1961220716 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb9c0c8270 con 0x7fdb9c0cf2c0 2023-10-16T13:49:31.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:31.396+0000 7fdba23b8700 1 -- 172.21.15.16:0/1961220716 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb9c0b29b0 con 0x7fdb9c0cf2c0 2023-10-16T13:49:31.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:31.396+0000 7fdb98ff9700 1 -- 172.21.15.16:0/1961220716 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 175) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fdb9401b3b0 con 0x7fdb9c0cf2c0 2023-10-16T13:49:31.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:31.396+0000 7fdb98ff9700 1 --2- 172.21.15.16:0/1961220716 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7fdb7c06af90 0x7fdb7c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:31.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:31.397+0000 7fdb98ff9700 1 -- 172.21.15.16:0/1961220716 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(282..282 src has 1..282) v4 ==== 6622+0+0 (secure 0 0 0) 0x7fdb94089360 con 0x7fdb9c0cf2c0 2023-10-16T13:49:31.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:31.397+0000 7fdb9bfff700 1 --2- 172.21.15.16:0/1961220716 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7fdb7c06af90 0x7fdb7c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:31.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:31.397+0000 7fdb9bfff700 1 --2- 172.21.15.16:0/1961220716 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7fdb7c06af90 0x7fdb7c06d450 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fdb84000c00 tx=0x7fdb8401c040 comp rx=0 tx=0).ready entity=mgr.13935 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:31.408 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:31.401+0000 7fdb98ff9700 1 -- 172.21.15.16:0/1961220716 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdb9c0b29b0 con 0x7fdb9c0cf2c0 2023-10-16T13:49:31.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:31.604+0000 7fdba23b8700 1 -- 172.21.15.16:0/1961220716 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fdb9c0b2980 con 0x7fdb9c0cf2c0 2023-10-16T13:49:32.245 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.243+0000 7fdb98ff9700 1 -- 172.21.15.16:0/1961220716 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v283) v1 ==== 169+0+0 (secure 0 0 0) 0x7fdb94054140 con 0x7fdb9c0cf2c0 2023-10-16T13:49:32.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.252+0000 7fdb98ff9700 1 -- 172.21.15.16:0/1961220716 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mgrmap(e 176) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fdb94054510 con 0x7fdb9c0cf2c0 2023-10-16T13:49:32.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.253+0000 7fdba23b8700 1 -- 172.21.15.16:0/1961220716 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7fdb7c06af90 msgr2=0x7fdb7c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:32.255 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.253+0000 7fdba23b8700 1 --2- 172.21.15.16:0/1961220716 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7fdb7c06af90 0x7fdb7c06d450 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fdb84000c00 tx=0x7fdb8401c040 comp rx=0 tx=0).stop 2023-10-16T13:49:32.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.253+0000 7fdba23b8700 1 -- 172.21.15.16:0/1961220716 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb9c0cf2c0 msgr2=0x7fdb9c151560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:32.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.253+0000 7fdba23b8700 1 --2- 172.21.15.16:0/1961220716 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb9c0cf2c0 0x7fdb9c151560 secure :-1 s=READY pgs=3315 cs=0 l=1 rev1=1 crypto rx=0x7fdb9400cbc0 tx=0x7fdb9400ced0 comp rx=0 tx=0).stop 2023-10-16T13:49:32.256 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.253+0000 7fdba23b8700 1 -- 172.21.15.16:0/1961220716 shutdown_connections 2023-10-16T13:49:32.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.253+0000 7fdba23b8700 1 --2- 172.21.15.16:0/1961220716 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7fdb7c06af90 0x7fdb7c06d450 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:32.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.253+0000 7fdba23b8700 1 --2- 172.21.15.16:0/1961220716 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb9c0cf2c0 0x7fdb9c151560 unknown :-1 s=CLOSED pgs=3315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:32.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.253+0000 7fdba23b8700 1 --2- 172.21.15.16:0/1961220716 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fdb9c0cbc10 0x7fdb9c0c88b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:32.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.253+0000 7fdba23b8700 1 --2- 172.21.15.16:0/1961220716 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdb9c0cd600 0x7fdb9c151020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:32.257 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.253+0000 7fdba23b8700 1 -- 172.21.15.16:0/1961220716 >> 172.21.15.16:0/1961220716 conn(0x7fdb9c0cc4b0 msgr2=0x7fdb9c15a390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:32.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.254+0000 7fdba23b8700 1 -- 172.21.15.16:0/1961220716 shutdown_connections 2023-10-16T13:49:32.258 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.254+0000 7fdba23b8700 1 -- 172.21.15.16:0/1961220716 wait complete. 2023-10-16T13:49:32.258 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' removed 2023-10-16T13:49:32.271 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-16T13:49:32.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.405+0000 7f05f26e2700 1 Processor -- start 2023-10-16T13:49:32.407 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.405+0000 7f05f26e2700 1 -- start start 2023-10-16T13:49:32.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.406+0000 7f05f26e2700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f05ec15bb40 0x7f05ec071a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:32.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.406+0000 7f05f26e2700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f05ec0caa40 0x7f05ec0cae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:32.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.406+0000 7f05f26e2700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05ec0cb3d0 con 0x7f05ec072b30 2023-10-16T13:49:32.410 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.406+0000 7f05f26e2700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05ec06ea20 con 0x7f05ec15bb40 2023-10-16T13:49:32.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.406+0000 7f05f26e2700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05ec15c040 con 0x7f05ec0caa40 2023-10-16T13:49:32.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.406+0000 7f05eb7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f05ec15bb40 0x7f05ec071a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:32.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.406+0000 7f05eb7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f05ec15bb40 0x7f05ec071a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60292/0 (socket says 172.21.15.16:60292) 2023-10-16T13:49:32.411 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.406+0000 7f05eb7fe700 1 -- 172.21.15.16:0/259171134 learned_addr learned my addr 172.21.15.16:0/259171134 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:32.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.406+0000 7f05eb7fe700 1 -- 172.21.15.16:0/259171134 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f05ec0caa40 msgr2=0x7f05ec0cae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:32.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.406+0000 7f05eb7fe700 1 --2- 172.21.15.16:0/259171134 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f05ec0caa40 0x7f05ec0cae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:32.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.406+0000 7f05eb7fe700 1 -- 172.21.15.16:0/259171134 >> v1:172.21.15.49:6789/0 conn(0x7f05ec072b30 legacy=0x7f05ec06e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:32.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.406+0000 7f05eb7fe700 1 -- 172.21.15.16:0/259171134 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05ec15c320 con 0x7f05ec15bb40 2023-10-16T13:49:32.412 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.406+0000 7f05eb7fe700 1 --2- 172.21.15.16:0/259171134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05ec15bb40 0x7f05ec071a70 secure :-1 s=READY pgs=3316 cs=0 l=1 rev1=1 crypto rx=0x7f05e00098f0 tx=0x7f05e0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c583e30f1553aa6d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:32.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.406+0000 7f05eaffd700 1 -- 172.21.15.16:0/259171134 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05e0004230 con 0x7f05ec15bb40 2023-10-16T13:49:32.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.406+0000 7f05eaffd700 1 -- 172.21.15.16:0/259171134 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05e0004390 con 0x7f05ec15bb40 2023-10-16T13:49:32.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.406+0000 7f05eaffd700 1 -- 172.21.15.16:0/259171134 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05e00036a0 con 0x7f05ec15bb40 2023-10-16T13:49:32.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.407+0000 7f05f26e2700 1 -- 172.21.15.16:0/259171134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05ec15bb40 msgr2=0x7f05ec071a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:32.413 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.407+0000 7f05f26e2700 1 --2- 172.21.15.16:0/259171134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05ec15bb40 0x7f05ec071a70 secure :-1 s=READY pgs=3316 cs=0 l=1 rev1=1 crypto rx=0x7f05e00098f0 tx=0x7f05e0009c00 comp rx=0 tx=0).stop 2023-10-16T13:49:32.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.407+0000 7f05f26e2700 1 -- 172.21.15.16:0/259171134 shutdown_connections 2023-10-16T13:49:32.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.407+0000 7f05f26e2700 1 --2- 172.21.15.16:0/259171134 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f05ec0caa40 0x7f05ec0cae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:32.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.407+0000 7f05f26e2700 1 --2- 172.21.15.16:0/259171134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05ec15bb40 0x7f05ec071a70 unknown :-1 s=CLOSED pgs=3316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:32.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.407+0000 7f05f26e2700 1 -- 172.21.15.16:0/259171134 >> 172.21.15.16:0/259171134 conn(0x7f05ec0b1f40 msgr2=0x7f05ec0b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:32.414 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.407+0000 7f05f26e2700 1 -- 172.21.15.16:0/259171134 shutdown_connections 2023-10-16T13:49:32.415 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.407+0000 7f05f26e2700 1 -- 172.21.15.16:0/259171134 wait complete. 2023-10-16T13:49:32.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.408+0000 7f05f26e2700 1 Processor -- start 2023-10-16T13:49:32.417 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.408+0000 7f05f26e2700 1 -- start start 2023-10-16T13:49:32.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.409+0000 7f05f26e2700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05ec072b30 0x7f05ec1f7010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:32.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.409+0000 7f05f26e2700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f05ec0caa40 0x7f05ec0cf470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:32.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.409+0000 7f05f26e2700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f05ec15bb40 0x7f05ec1f26b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:32.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.409+0000 7f05f26e2700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05ec0c8b10 con 0x7f05ec072b30 2023-10-16T13:49:32.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.409+0000 7f05f26e2700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05ec0cff80 con 0x7f05ec15bb40 2023-10-16T13:49:32.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.409+0000 7f05f26e2700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05ec0cfca0 con 0x7f05ec0caa40 2023-10-16T13:49:32.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.409+0000 7f05ebfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05ec072b30 0x7f05ec1f7010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:32.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.409+0000 7f05ebfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05ec072b30 0x7f05ec1f7010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60296/0 (socket says 172.21.15.16:60296) 2023-10-16T13:49:32.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.409+0000 7f05ebfff700 1 -- 172.21.15.16:0/27111230 learned_addr learned my addr 172.21.15.16:0/27111230 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:32.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.409+0000 7f05eb7fe700 1 --2- 172.21.15.16:0/27111230 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f05ec0caa40 0x7f05ec0cf470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:32.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.409+0000 7f05f0c7f700 1 --2- 172.21.15.16:0/27111230 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f05ec15bb40 0x7f05ec1f26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:32.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.409+0000 7f05ebfff700 1 -- 172.21.15.16:0/27111230 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f05ec0caa40 msgr2=0x7f05ec0cf470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:32.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.409+0000 7f05ebfff700 1 --2- 172.21.15.16:0/27111230 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f05ec0caa40 0x7f05ec0cf470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:32.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.409+0000 7f05ebfff700 1 -- 172.21.15.16:0/27111230 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f05ec15bb40 msgr2=0x7f05ec1f26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:32.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.409+0000 7f05ebfff700 1 --2- 172.21.15.16:0/27111230 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f05ec15bb40 0x7f05ec1f26b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:32.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.409+0000 7f05ebfff700 1 -- 172.21.15.16:0/27111230 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05e0009550 con 0x7f05ec072b30 2023-10-16T13:49:32.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.409+0000 7f05ebfff700 1 --2- 172.21.15.16:0/27111230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05ec072b30 0x7f05ec1f7010 secure :-1 s=READY pgs=3317 cs=0 l=1 rev1=1 crypto rx=0x7f05dc00b4a0 tx=0x7f05dc00b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:32.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.410+0000 7f05e8ff9700 1 -- 172.21.15.16:0/27111230 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05dc011830 con 0x7f05ec072b30 2023-10-16T13:49:32.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.410+0000 7f05e8ff9700 1 -- 172.21.15.16:0/27111230 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05dc011990 con 0x7f05ec072b30 2023-10-16T13:49:32.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.410+0000 7f05e8ff9700 1 -- 172.21.15.16:0/27111230 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05dc011d40 con 0x7f05ec072b30 2023-10-16T13:49:32.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.410+0000 7f05f26e2700 1 -- 172.21.15.16:0/27111230 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05ec0c84c0 con 0x7f05ec072b30 2023-10-16T13:49:32.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.410+0000 7f05f26e2700 1 -- 172.21.15.16:0/27111230 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05ec0cf9b0 con 0x7f05ec072b30 2023-10-16T13:49:32.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.411+0000 7f05f26e2700 1 -- 172.21.15.16:0/27111230 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05ec0c8320 con 0x7f05ec072b30 2023-10-16T13:49:32.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.411+0000 7f05e8ff9700 1 -- 172.21.15.16:0/27111230 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 176) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f05dc01a400 con 0x7f05ec072b30 2023-10-16T13:49:32.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.412+0000 7f05e8ff9700 1 --2- 172.21.15.16:0/27111230 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f05d806b020 0x7f05d806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:32.422 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.412+0000 7f05e8ff9700 1 -- 172.21.15.16:0/27111230 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(283..283 src has 1..283) v4 ==== 6208+0+0 (secure 0 0 0) 0x7f05dc0894e0 con 0x7f05ec072b30 2023-10-16T13:49:32.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.412+0000 7f05eb7fe700 1 --2- 172.21.15.16:0/27111230 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f05d806b020 0x7f05d806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:32.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.412+0000 7f05eb7fe700 1 --2- 172.21.15.16:0/27111230 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f05d806b020 0x7f05d806d4e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f05ec15bf70 tx=0x7f05e0008030 comp rx=0 tx=0).ready entity=mgr.13935 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:32.423 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.416+0000 7f05e8ff9700 1 -- 172.21.15.16:0/27111230 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f05ec0c8320 con 0x7f05ec072b30 2023-10-16T13:49:32.622 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.619+0000 7f05f26e2700 1 -- 172.21.15.16:0/27111230 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f05ec0c8320 con 0x7f05ec072b30 2023-10-16T13:49:32.625 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.623+0000 7f05e8ff9700 1 -- 172.21.15.16:0/27111230 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v97) v1 ==== 115+0+0 (secure 0 0 0) 0x7f05ec0c8320 con 0x7f05ec072b30 2023-10-16T13:49:32.627 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.625+0000 7f05f26e2700 1 -- 172.21.15.16:0/27111230 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f05d806b020 msgr2=0x7f05d806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:32.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.625+0000 7f05f26e2700 1 --2- 172.21.15.16:0/27111230 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f05d806b020 0x7f05d806d4e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f05ec15bf70 tx=0x7f05e0008030 comp rx=0 tx=0).stop 2023-10-16T13:49:32.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.625+0000 7f05f26e2700 1 -- 172.21.15.16:0/27111230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05ec072b30 msgr2=0x7f05ec1f7010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:32.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.625+0000 7f05f26e2700 1 --2- 172.21.15.16:0/27111230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05ec072b30 0x7f05ec1f7010 secure :-1 s=READY pgs=3317 cs=0 l=1 rev1=1 crypto rx=0x7f05dc00b4a0 tx=0x7f05dc00b7b0 comp rx=0 tx=0).stop 2023-10-16T13:49:32.628 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.625+0000 7f05f26e2700 1 -- 172.21.15.16:0/27111230 shutdown_connections 2023-10-16T13:49:32.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.625+0000 7f05f26e2700 1 --2- 172.21.15.16:0/27111230 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f05d806b020 0x7f05d806d4e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:32.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.625+0000 7f05f26e2700 1 --2- 172.21.15.16:0/27111230 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f05ec15bb40 0x7f05ec1f26b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:32.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.625+0000 7f05f26e2700 1 --2- 172.21.15.16:0/27111230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05ec072b30 0x7f05ec1f7010 unknown :-1 s=CLOSED pgs=3317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:32.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.625+0000 7f05f26e2700 1 --2- 172.21.15.16:0/27111230 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f05ec0caa40 0x7f05ec0cf470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:32.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.625+0000 7f05f26e2700 1 -- 172.21.15.16:0/27111230 >> 172.21.15.16:0/27111230 conn(0x7f05ec0b1f40 msgr2=0x7f05ec0b15c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:32.629 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.625+0000 7f05f26e2700 1 -- 172.21.15.16:0/27111230 shutdown_connections 2023-10-16T13:49:32.630 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.625+0000 7f05f26e2700 1 -- 172.21.15.16:0/27111230 wait complete. 2023-10-16T13:49:32.641 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-16T13:49:32.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.767+0000 7f6b4491d700 1 Processor -- start 2023-10-16T13:49:32.769 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.767+0000 7f6b4491d700 1 -- start start 2023-10-16T13:49:32.771 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.768+0000 7f6b4491d700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f6b4015b2c0 0x7f6b40161340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:32.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.768+0000 7f6b4491d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6b4006a7e0 0x7f6b4006ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:32.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.768+0000 7f6b4491d700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b4015ecb0 con 0x7f6b4006e570 2023-10-16T13:49:32.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.768+0000 7f6b4491d700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b4006b190 con 0x7f6b4006a7e0 2023-10-16T13:49:32.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.768+0000 7f6b4491d700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b40161880 con 0x7f6b4015b2c0 2023-10-16T13:49:32.772 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.768+0000 7f6b3ed9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6b4006a7e0 0x7f6b4006ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:32.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.768+0000 7f6b3ed9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6b4006a7e0 0x7f6b4006ac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60308/0 (socket says 172.21.15.16:60308) 2023-10-16T13:49:32.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.768+0000 7f6b3ed9d700 1 -- 172.21.15.16:0/147123328 learned_addr learned my addr 172.21.15.16:0/147123328 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:32.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.768+0000 7f6b3e59c700 1 --2- 172.21.15.16:0/147123328 >> v2:172.21.15.49:3300/0 conn(0x7f6b4015b2c0 0x7f6b40161340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:32.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.768+0000 7f6b3ed9d700 1 -- 172.21.15.16:0/147123328 >> v2:172.21.15.49:3300/0 conn(0x7f6b4015b2c0 msgr2=0x7f6b40161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:32.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.768+0000 7f6b3ed9d700 1 --2- 172.21.15.16:0/147123328 >> v2:172.21.15.49:3300/0 conn(0x7f6b4015b2c0 0x7f6b40161340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:32.773 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.768+0000 7f6b3ed9d700 1 -- 172.21.15.16:0/147123328 >> v1:172.21.15.49:6789/0 conn(0x7f6b4006e570 legacy=0x7f6b4015e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:32.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.768+0000 7f6b3ed9d700 1 -- 172.21.15.16:0/147123328 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b4015b6a0 con 0x7f6b4006a7e0 2023-10-16T13:49:32.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.768+0000 7f6b3ed9d700 1 --2- 172.21.15.16:0/147123328 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b4006a7e0 0x7f6b4006ac50 secure :-1 s=READY pgs=3318 cs=0 l=1 rev1=1 crypto rx=0x7f6b4006b4a0 tx=0x7f6b3800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cec1589f34405f87 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:32.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.768+0000 7f6b3d59a700 1 -- 172.21.15.16:0/147123328 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b3800e070 con 0x7f6b4006a7e0 2023-10-16T13:49:32.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.768+0000 7f6b3d59a700 1 -- 172.21.15.16:0/147123328 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b38003a20 con 0x7f6b4006a7e0 2023-10-16T13:49:32.774 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.768+0000 7f6b3d59a700 1 -- 172.21.15.16:0/147123328 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b38003dd0 con 0x7f6b4006a7e0 2023-10-16T13:49:32.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.769+0000 7f6b4491d700 1 -- 172.21.15.16:0/147123328 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b4006a7e0 msgr2=0x7f6b4006ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:32.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.769+0000 7f6b4491d700 1 --2- 172.21.15.16:0/147123328 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b4006a7e0 0x7f6b4006ac50 secure :-1 s=READY pgs=3318 cs=0 l=1 rev1=1 crypto rx=0x7f6b4006b4a0 tx=0x7f6b3800b330 comp rx=0 tx=0).stop 2023-10-16T13:49:32.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.769+0000 7f6b4491d700 1 -- 172.21.15.16:0/147123328 shutdown_connections 2023-10-16T13:49:32.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.769+0000 7f6b4491d700 1 --2- 172.21.15.16:0/147123328 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b4006a7e0 0x7f6b4006ac50 unknown :-1 s=CLOSED pgs=3318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:32.775 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.769+0000 7f6b4491d700 1 --2- 172.21.15.16:0/147123328 >> v2:172.21.15.49:3300/0 conn(0x7f6b4015b2c0 0x7f6b40161340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:32.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.769+0000 7f6b4491d700 1 -- 172.21.15.16:0/147123328 >> 172.21.15.16:0/147123328 conn(0x7f6b400b1f40 msgr2=0x7f6b400b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:32.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.769+0000 7f6b4491d700 1 -- 172.21.15.16:0/147123328 shutdown_connections 2023-10-16T13:49:32.776 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.769+0000 7f6b4491d700 1 -- 172.21.15.16:0/147123328 wait complete. 2023-10-16T13:49:32.778 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.770+0000 7f6b4491d700 1 Processor -- start 2023-10-16T13:49:32.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.770+0000 7f6b4491d700 1 -- start start 2023-10-16T13:49:32.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.771+0000 7f6b4491d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6b4006a7e0 0x7f6b400c8230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:32.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.771+0000 7f6b4491d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b4006e570 0x7f6b40157900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:32.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.771+0000 7f6b4491d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6b4015b2c0 0x7f6b400c47b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:32.779 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.771+0000 7f6b4491d700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b40157e40 con 0x7f6b4006e570 2023-10-16T13:49:32.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.771+0000 7f6b4491d700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b400c50b0 con 0x7f6b4015b2c0 2023-10-16T13:49:32.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.771+0000 7f6b4491d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b400c5390 con 0x7f6b4006a7e0 2023-10-16T13:49:32.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.771+0000 7f6b3dd9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b4006e570 0x7f6b40157900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:32.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.771+0000 7f6b3dd9b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b4006e570 0x7f6b40157900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60312/0 (socket says 172.21.15.16:60312) 2023-10-16T13:49:32.780 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.771+0000 7f6b3dd9b700 1 -- 172.21.15.16:0/214302413 learned_addr learned my addr 172.21.15.16:0/214302413 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:32.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.771+0000 7f6b3ed9d700 1 --2- 172.21.15.16:0/214302413 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6b4015b2c0 0x7f6b400c47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:32.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.771+0000 7f6b3e59c700 1 --2- 172.21.15.16:0/214302413 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6b4006a7e0 0x7f6b400c8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:32.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.772+0000 7f6b3dd9b700 1 -- 172.21.15.16:0/214302413 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6b4006a7e0 msgr2=0x7f6b400c8230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:32.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.772+0000 7f6b3dd9b700 1 --2- 172.21.15.16:0/214302413 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6b4006a7e0 0x7f6b400c8230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:32.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.772+0000 7f6b3dd9b700 1 -- 172.21.15.16:0/214302413 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6b4015b2c0 msgr2=0x7f6b400c47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:32.781 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.772+0000 7f6b3dd9b700 1 --2- 172.21.15.16:0/214302413 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6b4015b2c0 0x7f6b400c47b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:32.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.772+0000 7f6b3dd9b700 1 -- 172.21.15.16:0/214302413 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b38009b30 con 0x7f6b4006e570 2023-10-16T13:49:32.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.772+0000 7f6b3e59c700 1 --2- 172.21.15.16:0/214302413 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6b4006a7e0 0x7f6b400c8230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:49:32.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.772+0000 7f6b3dd9b700 1 --2- 172.21.15.16:0/214302413 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b4006e570 0x7f6b40157900 secure :-1 s=READY pgs=3319 cs=0 l=1 rev1=1 crypto rx=0x7f6b3400b4a0 tx=0x7f6b3400b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:32.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.772+0000 7f6b2effd700 1 -- 172.21.15.16:0/214302413 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b34011830 con 0x7f6b4006e570 2023-10-16T13:49:32.782 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.772+0000 7f6b2effd700 1 -- 172.21.15.16:0/214302413 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b34011990 con 0x7f6b4006e570 2023-10-16T13:49:32.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.772+0000 7f6b2effd700 1 -- 172.21.15.16:0/214302413 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b34011d40 con 0x7f6b4006e570 2023-10-16T13:49:32.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.772+0000 7f6b4491d700 1 -- 172.21.15.16:0/214302413 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b400c5670 con 0x7f6b4006e570 2023-10-16T13:49:32.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.772+0000 7f6b4491d700 1 -- 172.21.15.16:0/214302413 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b400c5c30 con 0x7f6b4006e570 2023-10-16T13:49:32.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.773+0000 7f6b4491d700 1 -- 172.21.15.16:0/214302413 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b400c5e50 con 0x7f6b4006e570 2023-10-16T13:49:32.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.774+0000 7f6b2effd700 1 -- 172.21.15.16:0/214302413 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 176) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6b34004020 con 0x7f6b4006e570 2023-10-16T13:49:32.783 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.774+0000 7f6b2effd700 1 --2- 172.21.15.16:0/214302413 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f6b2006b020 0x7f6b2006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:32.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.774+0000 7f6b2effd700 1 -- 172.21.15.16:0/214302413 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(283..283 src has 1..283) v4 ==== 6208+0+0 (secure 0 0 0) 0x7f6b34089960 con 0x7f6b4006e570 2023-10-16T13:49:32.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.774+0000 7f6b3e59c700 1 --2- 172.21.15.16:0/214302413 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f6b2006b020 0x7f6b2006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:32.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.775+0000 7f6b3e59c700 1 --2- 172.21.15.16:0/214302413 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f6b2006b020 0x7f6b2006d4e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6b30007a50 tx=0x7f6b30008040 comp rx=0 tx=0).ready entity=mgr.13935 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:32.784 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.778+0000 7f6b2effd700 1 -- 172.21.15.16:0/214302413 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6b34054810 con 0x7f6b4006e570 2023-10-16T13:49:32.987 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.984+0000 7f6b4491d700 1 -- 172.21.15.16:0/214302413 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f6b400ca730 con 0x7f6b4006e570 2023-10-16T13:49:32.990 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.987+0000 7f6b2effd700 1 -- 172.21.15.16:0/214302413 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v98) v1 ==== 104+0+0 (secure 0 0 0) 0x7f6b34057e30 con 0x7f6b4006e570 2023-10-16T13:49:32.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.989+0000 7f6b4491d700 1 -- 172.21.15.16:0/214302413 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f6b2006b020 msgr2=0x7f6b2006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:32.992 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.989+0000 7f6b4491d700 1 --2- 172.21.15.16:0/214302413 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f6b2006b020 0x7f6b2006d4e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6b30007a50 tx=0x7f6b30008040 comp rx=0 tx=0).stop 2023-10-16T13:49:32.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.989+0000 7f6b4491d700 1 -- 172.21.15.16:0/214302413 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b4006e570 msgr2=0x7f6b40157900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:32.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.989+0000 7f6b4491d700 1 --2- 172.21.15.16:0/214302413 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b4006e570 0x7f6b40157900 secure :-1 s=READY pgs=3319 cs=0 l=1 rev1=1 crypto rx=0x7f6b3400b4a0 tx=0x7f6b3400b7b0 comp rx=0 tx=0).stop 2023-10-16T13:49:32.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.990+0000 7f6b4491d700 1 -- 172.21.15.16:0/214302413 shutdown_connections 2023-10-16T13:49:32.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.990+0000 7f6b4491d700 1 --2- 172.21.15.16:0/214302413 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f6b2006b020 0x7f6b2006d4e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:32.993 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.990+0000 7f6b4491d700 1 --2- 172.21.15.16:0/214302413 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6b4015b2c0 0x7f6b400c47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:32.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.990+0000 7f6b4491d700 1 --2- 172.21.15.16:0/214302413 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f6b4006a7e0 0x7f6b400c8230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:32.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.990+0000 7f6b4491d700 1 --2- 172.21.15.16:0/214302413 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b4006e570 0x7f6b40157900 unknown :-1 s=CLOSED pgs=3319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:32.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.990+0000 7f6b4491d700 1 -- 172.21.15.16:0/214302413 >> 172.21.15.16:0/214302413 conn(0x7f6b400b1f40 msgr2=0x7f6b4015e3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:32.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.990+0000 7f6b4491d700 1 -- 172.21.15.16:0/214302413 shutdown_connections 2023-10-16T13:49:32.994 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:32.990+0000 7f6b4491d700 1 -- 172.21.15.16:0/214302413 wait complete. 2023-10-16T13:49:33.004 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-16T13:49:33.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.128+0000 7f3231a99700 1 Processor -- start 2023-10-16T13:49:33.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.129+0000 7f3231a99700 1 -- start start 2023-10-16T13:49:33.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.129+0000 7f3231a99700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f322c0ce5d0 0x7f322c0cc190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:33.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.129+0000 7f3231a99700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f322c0b2440 con 0x7f322c15ec80 2023-10-16T13:49:33.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.129+0000 7f3231a99700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f322c0d00e0 con 0x7f322c15f060 2023-10-16T13:49:33.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.129+0000 7f3231a99700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f322c0b2c50 con 0x7f322c0ce5d0 2023-10-16T13:49:33.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.129+0000 7f322affd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f322c0ce5d0 0x7f322c0cc190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:33.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.129+0000 7f322b7fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f322c15ec80 0x7f322c0cdec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49592/0 (socket says 172.21.15.16:49592) 2023-10-16T13:49:33.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.129+0000 7f322b7fe700 1 -- 172.21.15.16:0/3103418242 learned_addr learned my addr 172.21.15.16:0/3103418242 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:49:33.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.129+0000 7f322affd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f322c0ce5d0 0x7f322c0cc190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60320/0 (socket says 172.21.15.16:60320) 2023-10-16T13:49:33.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.130+0000 7f322affd700 1 -- 172.21.15.16:0/3103418242 >> v1:172.21.15.49:6789/0 conn(0x7f322c15f060 legacy=0x7f322c0cc6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:33.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.130+0000 7f322affd700 1 -- 172.21.15.16:0/3103418242 >> v1:172.21.15.16:6789/0 conn(0x7f322c15ec80 legacy=0x7f322c0cdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:33.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.130+0000 7f322affd700 1 -- 172.21.15.16:0/3103418242 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f322c0cef60 con 0x7f322c0ce5d0 2023-10-16T13:49:33.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.130+0000 7f322affd700 1 --2- 172.21.15.16:0/3103418242 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f322c0ce5d0 0x7f322c0cc190 secure :-1 s=READY pgs=3321 cs=0 l=1 rev1=1 crypto rx=0x7f321c009870 tx=0x7f321c009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b5378befcebf813 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:33.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.130+0000 7f322a7fc700 1 -- 172.21.15.16:0/3103418242 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f321c011830 con 0x7f322c0ce5d0 2023-10-16T13:49:33.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.130+0000 7f322a7fc700 1 -- 172.21.15.16:0/3103418242 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f321c011990 con 0x7f322c0ce5d0 2023-10-16T13:49:33.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.130+0000 7f322a7fc700 1 -- 172.21.15.16:0/3103418242 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f321c011d20 con 0x7f322c0ce5d0 2023-10-16T13:49:33.137 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.131+0000 7f3231a99700 1 -- 172.21.15.16:0/3103418242 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f322c0ce5d0 msgr2=0x7f322c0cc190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:33.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.131+0000 7f3231a99700 1 --2- 172.21.15.16:0/3103418242 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f322c0ce5d0 0x7f322c0cc190 secure :-1 s=READY pgs=3321 cs=0 l=1 rev1=1 crypto rx=0x7f321c009870 tx=0x7f321c009b80 comp rx=0 tx=0).stop 2023-10-16T13:49:33.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.131+0000 7f3231a99700 1 -- 172.21.15.16:0/3103418242 shutdown_connections 2023-10-16T13:49:33.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.131+0000 7f3231a99700 1 --2- 172.21.15.16:0/3103418242 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f322c0ce5d0 0x7f322c0cc190 unknown :-1 s=CLOSED pgs=3321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:33.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.131+0000 7f3231a99700 1 -- 172.21.15.16:0/3103418242 >> 172.21.15.16:0/3103418242 conn(0x7f322c0d15f0 msgr2=0x7f322c0d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:33.138 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.131+0000 7f3231a99700 1 -- 172.21.15.16:0/3103418242 shutdown_connections 2023-10-16T13:49:33.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.131+0000 7f3231a99700 1 -- 172.21.15.16:0/3103418242 wait complete. 2023-10-16T13:49:33.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.132+0000 7f3231a99700 1 Processor -- start 2023-10-16T13:49:33.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.133+0000 7f3231a99700 1 -- start start 2023-10-16T13:49:33.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.133+0000 7f3231a99700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f322c0ce5d0 0x7f322c1517d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:33.139 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.133+0000 7f3231a99700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f322c15ec80 0x7f322c0b63c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:33.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.133+0000 7f3231a99700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f322c15f060 0x7f322c0b6900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:33.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.133+0000 7f3231a99700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f322c0c8930 con 0x7f322c15ec80 2023-10-16T13:49:33.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.133+0000 7f3231a99700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f322c151d10 con 0x7f322c15f060 2023-10-16T13:49:33.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.134+0000 7f3231a99700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f322c15bee0 con 0x7f322c0ce5d0 2023-10-16T13:49:33.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.134+0000 7f322affd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f322c15ec80 0x7f322c0b63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:33.140 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.134+0000 7f322affd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f322c15ec80 0x7f322c0b63c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60326/0 (socket says 172.21.15.16:60326) 2023-10-16T13:49:33.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.134+0000 7f322affd700 1 -- 172.21.15.16:0/827935774 learned_addr learned my addr 172.21.15.16:0/827935774 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:33.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.134+0000 7f322b7fe700 1 --2- 172.21.15.16:0/827935774 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f322c0ce5d0 0x7f322c1517d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:33.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.134+0000 7f322bfff700 1 --2- 172.21.15.16:0/827935774 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f322c15f060 0x7f322c0b6900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:33.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.134+0000 7f322affd700 1 -- 172.21.15.16:0/827935774 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f322c0ce5d0 msgr2=0x7f322c1517d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:33.141 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.134+0000 7f322affd700 1 --2- 172.21.15.16:0/827935774 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f322c0ce5d0 0x7f322c1517d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:33.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.134+0000 7f322affd700 1 -- 172.21.15.16:0/827935774 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f322c15f060 msgr2=0x7f322c0b6900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:33.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.134+0000 7f322affd700 1 --2- 172.21.15.16:0/827935774 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f322c15f060 0x7f322c0b6900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:33.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.134+0000 7f322affd700 1 -- 172.21.15.16:0/827935774 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f321c0094a0 con 0x7f322c15ec80 2023-10-16T13:49:33.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.134+0000 7f322affd700 1 --2- 172.21.15.16:0/827935774 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f322c15ec80 0x7f322c0b63c0 secure :-1 s=READY pgs=3322 cs=0 l=1 rev1=1 crypto rx=0x7f321c0090b0 tx=0x7f321c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:33.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.134+0000 7f3223fff700 1 -- 172.21.15.16:0/827935774 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f321c0038a0 con 0x7f322c15ec80 2023-10-16T13:49:33.142 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.134+0000 7f3223fff700 1 -- 172.21.15.16:0/827935774 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f321c01f070 con 0x7f322c15ec80 2023-10-16T13:49:33.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.134+0000 7f3231a99700 1 -- 172.21.15.16:0/827935774 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f322c15c1c0 con 0x7f322c15ec80 2023-10-16T13:49:33.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.134+0000 7f3223fff700 1 -- 172.21.15.16:0/827935774 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f321c003c80 con 0x7f322c15ec80 2023-10-16T13:49:33.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.134+0000 7f3231a99700 1 -- 172.21.15.16:0/827935774 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f322c0c53e0 con 0x7f322c15ec80 2023-10-16T13:49:33.143 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.135+0000 7f3231a99700 1 -- 172.21.15.16:0/827935774 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f322c04ebb0 con 0x7f322c15ec80 2023-10-16T13:49:33.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.136+0000 7f3223fff700 1 -- 172.21.15.16:0/827935774 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 176) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f321c01a430 con 0x7f322c15ec80 2023-10-16T13:49:33.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.136+0000 7f3223fff700 1 --2- 172.21.15.16:0/827935774 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f320c06b020 0x7f320c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:33.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.136+0000 7f3223fff700 1 -- 172.21.15.16:0/827935774 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(283..283 src has 1..283) v4 ==== 6208+0+0 (secure 0 0 0) 0x7f321c021070 con 0x7f322c15ec80 2023-10-16T13:49:33.144 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.136+0000 7f322b7fe700 1 --2- 172.21.15.16:0/827935774 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f320c06b020 0x7f320c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:33.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.137+0000 7f322b7fe700 1 --2- 172.21.15.16:0/827935774 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f320c06b020 0x7f320c06d4e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3214005910 tx=0x7f3214009040 comp rx=0 tx=0).ready entity=mgr.13935 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:33.145 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.141+0000 7f3223fff700 1 -- 172.21.15.16:0/827935774 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f321c056220 con 0x7f322c15ec80 2023-10-16T13:49:33.348 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.345+0000 7f3231a99700 1 -- 172.21.15.16:0/827935774 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f322c0b43c0 con 0x7f322c15ec80 2023-10-16T13:49:33.351 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.348+0000 7f3223fff700 1 -- 172.21.15.16:0/827935774 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v99) v1 ==== 120+0+0 (secure 0 0 0) 0x7f321c059840 con 0x7f322c15ec80 2023-10-16T13:49:33.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.351+0000 7f3231a99700 1 -- 172.21.15.16:0/827935774 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f320c06b020 msgr2=0x7f320c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:33.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.351+0000 7f3231a99700 1 --2- 172.21.15.16:0/827935774 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f320c06b020 0x7f320c06d4e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3214005910 tx=0x7f3214009040 comp rx=0 tx=0).stop 2023-10-16T13:49:33.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.351+0000 7f3231a99700 1 -- 172.21.15.16:0/827935774 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f322c15ec80 msgr2=0x7f322c0b63c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:33.354 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.351+0000 7f3231a99700 1 --2- 172.21.15.16:0/827935774 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f322c15ec80 0x7f322c0b63c0 secure :-1 s=READY pgs=3322 cs=0 l=1 rev1=1 crypto rx=0x7f321c0090b0 tx=0x7f321c004750 comp rx=0 tx=0).stop 2023-10-16T13:49:33.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.351+0000 7f3231a99700 1 -- 172.21.15.16:0/827935774 shutdown_connections 2023-10-16T13:49:33.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.351+0000 7f3231a99700 1 --2- 172.21.15.16:0/827935774 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f320c06b020 0x7f320c06d4e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:33.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.351+0000 7f3231a99700 1 --2- 172.21.15.16:0/827935774 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f322c15f060 0x7f322c0b6900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:33.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.351+0000 7f3231a99700 1 --2- 172.21.15.16:0/827935774 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f322c0ce5d0 0x7f322c1517d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:33.355 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.351+0000 7f3231a99700 1 --2- 172.21.15.16:0/827935774 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f322c15ec80 0x7f322c0b63c0 unknown :-1 s=CLOSED pgs=3322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:33.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.351+0000 7f3231a99700 1 -- 172.21.15.16:0/827935774 >> 172.21.15.16:0/827935774 conn(0x7f322c0d15f0 msgr2=0x7f322c15dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:33.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.351+0000 7f3231a99700 1 -- 172.21.15.16:0/827935774 shutdown_connections 2023-10-16T13:49:33.356 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.351+0000 7f3231a99700 1 -- 172.21.15.16:0/827935774 wait complete. 2023-10-16T13:49:33.366 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments.test_snap_schedule_subvol_and_group_arguments' 2023-10-16T13:49:33.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.489+0000 7fb9e9ad5700 1 Processor -- start 2023-10-16T13:49:33.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.490+0000 7fb9e9ad5700 1 -- start start 2023-10-16T13:49:33.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.490+0000 7fb9e9ad5700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb9e415b280 0x7fb9e41613e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:33.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.490+0000 7fb9e9ad5700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb9e406a7e0 0x7fb9e406ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:33.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.490+0000 7fb9e9ad5700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9e406b170 con 0x7fb9e40cc8a0 2023-10-16T13:49:33.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.490+0000 7fb9e9ad5700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9e4161c30 con 0x7fb9e406a7e0 2023-10-16T13:49:33.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.490+0000 7fb9e9ad5700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9e4161f10 con 0x7fb9e415b280 2023-10-16T13:49:33.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.491+0000 7fb9e3fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb9e406a7e0 0x7fb9e406ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:33.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.491+0000 7fb9e3fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb9e406a7e0 0x7fb9e406ac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60336/0 (socket says 172.21.15.16:60336) 2023-10-16T13:49:33.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.491+0000 7fb9e3fff700 1 -- 172.21.15.16:0/2389114268 learned_addr learned my addr 172.21.15.16:0/2389114268 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:33.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.491+0000 7fb9e37fe700 1 --2- 172.21.15.16:0/2389114268 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb9e415b280 0x7fb9e41613e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:33.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.491+0000 7fb9e3fff700 1 -- 172.21.15.16:0/2389114268 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb9e415b280 msgr2=0x7fb9e41613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:33.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.491+0000 7fb9e3fff700 1 --2- 172.21.15.16:0/2389114268 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb9e415b280 0x7fb9e41613e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:33.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.491+0000 7fb9e3fff700 1 -- 172.21.15.16:0/2389114268 >> v1:172.21.15.49:6789/0 conn(0x7fb9e40cc8a0 legacy=0x7fb9e415e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:33.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.491+0000 7fb9e3fff700 1 -- 172.21.15.16:0/2389114268 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9e41621f0 con 0x7fb9e406a7e0 2023-10-16T13:49:33.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.491+0000 7fb9e3fff700 1 --2- 172.21.15.16:0/2389114268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9e406a7e0 0x7fb9e406ac30 secure :-1 s=READY pgs=3323 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc009e50 tx=0x7fb9dc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32805fe0ca2c9124 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:33.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.491+0000 7fb9e27fc700 1 -- 172.21.15.16:0/2389114268 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9dc00e070 con 0x7fb9e406a7e0 2023-10-16T13:49:33.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.491+0000 7fb9e27fc700 1 -- 172.21.15.16:0/2389114268 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9dc00b9c0 con 0x7fb9e406a7e0 2023-10-16T13:49:33.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.491+0000 7fb9e27fc700 1 -- 172.21.15.16:0/2389114268 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9dc00bd70 con 0x7fb9e406a7e0 2023-10-16T13:49:33.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.492+0000 7fb9e9ad5700 1 -- 172.21.15.16:0/2389114268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9e406a7e0 msgr2=0x7fb9e406ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:33.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.492+0000 7fb9e9ad5700 1 --2- 172.21.15.16:0/2389114268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9e406a7e0 0x7fb9e406ac30 secure :-1 s=READY pgs=3323 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc009e50 tx=0x7fb9dc00b330 comp rx=0 tx=0).stop 2023-10-16T13:49:33.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.492+0000 7fb9e9ad5700 1 -- 172.21.15.16:0/2389114268 shutdown_connections 2023-10-16T13:49:33.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.492+0000 7fb9e9ad5700 1 --2- 172.21.15.16:0/2389114268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9e406a7e0 0x7fb9e406ac30 unknown :-1 s=CLOSED pgs=3323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:33.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.492+0000 7fb9e9ad5700 1 --2- 172.21.15.16:0/2389114268 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb9e415b280 0x7fb9e41613e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:33.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.492+0000 7fb9e9ad5700 1 -- 172.21.15.16:0/2389114268 >> 172.21.15.16:0/2389114268 conn(0x7fb9e40b1fd0 msgr2=0x7fb9e40b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:33.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.492+0000 7fb9e9ad5700 1 -- 172.21.15.16:0/2389114268 shutdown_connections 2023-10-16T13:49:33.499 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.492+0000 7fb9e9ad5700 1 -- 172.21.15.16:0/2389114268 wait complete. 2023-10-16T13:49:33.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.493+0000 7fb9e9ad5700 1 Processor -- start 2023-10-16T13:49:33.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.493+0000 7fb9e9ad5700 1 -- start start 2023-10-16T13:49:33.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.494+0000 7fb9e9ad5700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb9e406a7e0 0x7fb9e41f7140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:33.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.494+0000 7fb9e9ad5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9e40cc8a0 0x7fb9e40c5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:33.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.494+0000 7fb9e9ad5700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb9e415b280 0x7fb9e40b1070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:33.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.494+0000 7fb9e9ad5700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9e41f3220 con 0x7fb9e40cc8a0 2023-10-16T13:49:33.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.494+0000 7fb9e9ad5700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9e40ca060 con 0x7fb9e415b280 2023-10-16T13:49:33.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.494+0000 7fb9e9ad5700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9e41f7680 con 0x7fb9e406a7e0 2023-10-16T13:49:33.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.494+0000 7fb9e2ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9e40cc8a0 0x7fb9e40c5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:33.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.494+0000 7fb9e2ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9e40cc8a0 0x7fb9e40c5730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60340/0 (socket says 172.21.15.16:60340) 2023-10-16T13:49:33.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.494+0000 7fb9e2ffd700 1 -- 172.21.15.16:0/4222113904 learned_addr learned my addr 172.21.15.16:0/4222113904 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:33.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.494+0000 7fb9e37fe700 1 --2- 172.21.15.16:0/4222113904 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb9e406a7e0 0x7fb9e41f7140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:33.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.494+0000 7fb9e3fff700 1 --2- 172.21.15.16:0/4222113904 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb9e415b280 0x7fb9e40b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:33.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.494+0000 7fb9e2ffd700 1 -- 172.21.15.16:0/4222113904 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb9e406a7e0 msgr2=0x7fb9e41f7140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:33.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.494+0000 7fb9e2ffd700 1 --2- 172.21.15.16:0/4222113904 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb9e406a7e0 0x7fb9e41f7140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:33.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.494+0000 7fb9e2ffd700 1 -- 172.21.15.16:0/4222113904 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb9e415b280 msgr2=0x7fb9e40b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:33.504 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.494+0000 7fb9e2ffd700 1 --2- 172.21.15.16:0/4222113904 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb9e415b280 0x7fb9e40b1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:33.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.494+0000 7fb9e2ffd700 1 -- 172.21.15.16:0/4222113904 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9dc009b30 con 0x7fb9e40cc8a0 2023-10-16T13:49:33.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.495+0000 7fb9e2ffd700 1 --2- 172.21.15.16:0/4222113904 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9e40cc8a0 0x7fb9e40c5730 secure :-1 s=READY pgs=3324 cs=0 l=1 rev1=1 crypto rx=0x7fb9d800c4e0 tx=0x7fb9d800c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:33.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.495+0000 7fb9cbfff700 1 -- 172.21.15.16:0/4222113904 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9d80052d0 con 0x7fb9e40cc8a0 2023-10-16T13:49:33.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.495+0000 7fb9cbfff700 1 -- 172.21.15.16:0/4222113904 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9d8005430 con 0x7fb9e40cc8a0 2023-10-16T13:49:33.505 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.495+0000 7fb9e9ad5700 1 -- 172.21.15.16:0/4222113904 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9e414b010 con 0x7fb9e40cc8a0 2023-10-16T13:49:33.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.495+0000 7fb9cbfff700 1 -- 172.21.15.16:0/4222113904 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9d80038c0 con 0x7fb9e40cc8a0 2023-10-16T13:49:33.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.495+0000 7fb9e9ad5700 1 -- 172.21.15.16:0/4222113904 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9e414b5d0 con 0x7fb9e40cc8a0 2023-10-16T13:49:33.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.496+0000 7fb9e9ad5700 1 -- 172.21.15.16:0/4222113904 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9e41f3360 con 0x7fb9e40cc8a0 2023-10-16T13:49:33.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.496+0000 7fb9cbfff700 1 -- 172.21.15.16:0/4222113904 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 176) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb9d8003a20 con 0x7fb9e40cc8a0 2023-10-16T13:49:33.506 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.497+0000 7fb9cbfff700 1 --2- 172.21.15.16:0/4222113904 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7fb9d006b020 0x7fb9d006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:33.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.497+0000 7fb9cbfff700 1 -- 172.21.15.16:0/4222113904 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(283..283 src has 1..283) v4 ==== 6208+0+0 (secure 0 0 0) 0x7fb9d8088ae0 con 0x7fb9e40cc8a0 2023-10-16T13:49:33.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.497+0000 7fb9e37fe700 1 --2- 172.21.15.16:0/4222113904 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7fb9d006b020 0x7fb9d006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:33.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.497+0000 7fb9e37fe700 1 --2- 172.21.15.16:0/4222113904 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7fb9d006b020 0x7fb9d006d4e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb9d4006b20 tx=0x7fb9d4008040 comp rx=0 tx=0).ready entity=mgr.13935 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:33.507 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.501+0000 7fb9cbfff700 1 -- 172.21.15.16:0/4222113904 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb9d80538a0 con 0x7fb9e40cc8a0 2023-10-16T13:49:33.790 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:33.788+0000 7fb9e9ad5700 1 -- 172.21.15.16:0/4222113904 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments.test_snap_schedule_subvol_and_group_arguments"]} v 0) v1 -- 0x7fb9e406b150 con 0x7fb9e40cc8a0 2023-10-16T13:49:34.016 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:49:34.019 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:49:34.248 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:34.245+0000 7fb9cbfff700 1 -- 172.21.15.16:0/4222113904 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments.test_snap_schedule_subvol_and_group_arguments"]}]=0 v3298) v1 ==== 198+0+0 (secure 0 0 0) 0x7fb9d8017100 con 0x7fb9e40cc8a0 2023-10-16T13:49:34.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:34.248+0000 7fb9e9ad5700 1 -- 172.21.15.16:0/4222113904 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7fb9d006b020 msgr2=0x7fb9d006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:34.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:34.248+0000 7fb9e9ad5700 1 --2- 172.21.15.16:0/4222113904 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7fb9d006b020 0x7fb9d006d4e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb9d4006b20 tx=0x7fb9d4008040 comp rx=0 tx=0).stop 2023-10-16T13:49:34.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:34.248+0000 7fb9e9ad5700 1 -- 172.21.15.16:0/4222113904 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9e40cc8a0 msgr2=0x7fb9e40c5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:34.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:34.248+0000 7fb9e9ad5700 1 --2- 172.21.15.16:0/4222113904 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9e40cc8a0 0x7fb9e40c5730 secure :-1 s=READY pgs=3324 cs=0 l=1 rev1=1 crypto rx=0x7fb9d800c4e0 tx=0x7fb9d800c7f0 comp rx=0 tx=0).stop 2023-10-16T13:49:34.251 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:34.248+0000 7fb9e9ad5700 1 -- 172.21.15.16:0/4222113904 shutdown_connections 2023-10-16T13:49:34.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:34.248+0000 7fb9e9ad5700 1 --2- 172.21.15.16:0/4222113904 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7fb9d006b020 0x7fb9d006d4e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:34.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:34.248+0000 7fb9e9ad5700 1 --2- 172.21.15.16:0/4222113904 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb9e415b280 0x7fb9e40b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:34.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:34.248+0000 7fb9e9ad5700 1 --2- 172.21.15.16:0/4222113904 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fb9e406a7e0 0x7fb9e41f7140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:34.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:34.248+0000 7fb9e9ad5700 1 --2- 172.21.15.16:0/4222113904 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9e40cc8a0 0x7fb9e40c5730 unknown :-1 s=CLOSED pgs=3324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:34.252 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:34.248+0000 7fb9e9ad5700 1 -- 172.21.15.16:0/4222113904 >> 172.21.15.16:0/4222113904 conn(0x7fb9e40b1fd0 msgr2=0x7fb9e40d0de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:34.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:34.248+0000 7fb9e9ad5700 1 -- 172.21.15.16:0/4222113904 shutdown_connections 2023-10-16T13:49:34.253 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:34.248+0000 7fb9e9ad5700 1 -- 172.21.15.16:0/4222113904 wait complete. 2023-10-16T13:49:34.265 INFO:tasks.cephfs_test_runner:Test subvol and group arguments to snap-schedule cli. ... FAIL 2023-10-16T13:49:34.265 INFO:tasks.cephfs_test_runner: 2023-10-16T13:49:34.266 INFO:tasks.cephfs_test_runner:====================================================================== 2023-10-16T13:49:34.266 INFO:tasks.cephfs_test_runner:FAIL: test_snap_schedule_subvol_and_group_arguments (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments) 2023-10-16T13:49:34.266 INFO:tasks.cephfs_test_runner:Test subvol and group arguments to snap-schedule cli. 2023-10-16T13:49:34.267 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2023-10-16T13:49:34.267 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2023-10-16T13:49:34.267 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_mchangir_ceph_9359c5b40091bccbce966736dc1a3c5bcc76ab0d/qa/tasks/cephfs/test_snap_schedules.py", line 606, in test_snap_schedule_subvol_and_group_arguments 2023-10-16T13:49:34.267 INFO:tasks.cephfs_test_runner: self.fs_snap_schedule_cmd('add', '--group', 'mygrp', path='.', snap_schedule='1m') 2023-10-16T13:49:34.268 INFO:tasks.cephfs_test_runner:AssertionError: CommandFailedError not raised 2023-10-16T13:49:34.268 INFO:tasks.cephfs_test_runner: 2023-10-16T13:49:34.268 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2023-10-16T13:49:34.268 INFO:tasks.cephfs_test_runner:Ran 15 tests in 5882.491s 2023-10-16T13:49:34.269 INFO:tasks.cephfs_test_runner: 2023-10-16T13:49:34.269 INFO:tasks.cephfs_test_runner:FAILED (failures=1) 2023-10-16T13:49:34.269 INFO:tasks.cephfs_test_runner: 2023-10-16T13:49:34.269 INFO:tasks.cephfs_test_runner:====================================================================== 2023-10-16T13:49:34.270 INFO:tasks.cephfs_test_runner:FAIL: test_snap_schedule_subvol_and_group_arguments (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments) 2023-10-16T13:49:34.270 INFO:tasks.cephfs_test_runner:Test subvol and group arguments to snap-schedule cli. 2023-10-16T13:49:34.270 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2023-10-16T13:49:34.271 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2023-10-16T13:49:34.271 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_mchangir_ceph_9359c5b40091bccbce966736dc1a3c5bcc76ab0d/qa/tasks/cephfs/test_snap_schedules.py", line 606, in test_snap_schedule_subvol_and_group_arguments 2023-10-16T13:49:34.271 INFO:tasks.cephfs_test_runner: self.fs_snap_schedule_cmd('add', '--group', 'mygrp', path='.', snap_schedule='1m') 2023-10-16T13:49:34.271 INFO:tasks.cephfs_test_runner:AssertionError: CommandFailedError not raised 2023-10-16T13:49:34.272 INFO:tasks.cephfs_test_runner: 2023-10-16T13:49:34.272 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_mchangir_ceph_9359c5b40091bccbce966736dc1a3c5bcc76ab0d/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_snap_schedule_subvol_and_group_arguments (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments) 2023-10-16T13:49:34.334 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=b84ce40e1a6e4a94811217a49e534a4a Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_mchangir_ceph_9359c5b40091bccbce966736dc1a3c5bcc76ab0d/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_snap_schedule_subvol_and_group_arguments (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments) 2023-10-16T13:49:34.336 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2023-10-16T13:49:34.346 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2023-10-16T13:49:34.367 INFO:tasks.kclient:Unmounting kernel clients... 2023-10-16T13:49:34.367 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T13:49:34.368 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:49:34.400 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16287364k,nr_inodes=4071841,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20251 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3261172k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T13:49:34.400 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-10-16T13:49:34.401 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:49:34.459 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16287364k,nr_inodes=4071841,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20251 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3261172k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-10-16T13:49:34.460 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T13:49:34.460 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:49:34.482 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284204k,nr_inodes=4071051,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=753 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260540k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T13:49:34.483 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-10-16T13:49:34.483 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T13:49:34.543 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284204k,nr_inodes=4071051,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=753 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260540k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-10-16T13:49:34.543 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:49:34.544 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2023-10-16T13:49:34.563 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-16T13:49:34.563 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-16T13:49:34.564 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:49:34.564 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-16T13:49:34.652 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:49:34.652 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T13:49:34.735 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:49:34.735 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link delete ceph-brx 2023-10-16T13:49:34.837 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:49:34.837 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2023-10-16T13:49:34.854 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:49:34.855 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link delete ceph-brx 2023-10-16T13:49:34.925 INFO:teuthology.orchestra.run.smithi016.stderr:Cannot find device "ceph-brx" 2023-10-16T13:49:34.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-16T13:49:34.927 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:49:34.928 DEBUG:teuthology.orchestra.run.smithi049:> ip netns list 2023-10-16T13:49:34.949 INFO:teuthology.orchestra.run.smithi049.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2023-10-16T13:49:34.949 INFO:teuthology.orchestra.run.smithi049.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-10-16T13:49:34.950 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:49:34.950 DEBUG:teuthology.orchestra.run.smithi049:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2023-10-16T13:49:35.037 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:49:35.037 DEBUG:teuthology.orchestra.run.smithi049:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2023-10-16T13:49:35.131 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:49:35.131 DEBUG:teuthology.orchestra.run.smithi049:> sudo ip link delete ceph-brx 2023-10-16T13:49:35.222 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:49:35.223 DEBUG:teuthology.orchestra.run.smithi049:> ip netns list 2023-10-16T13:49:35.241 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T13:49:35.242 DEBUG:teuthology.orchestra.run.smithi049:> sudo ip link delete ceph-brx 2023-10-16T13:49:35.312 INFO:teuthology.orchestra.run.smithi049.stderr:Cannot find device "ceph-brx" 2023-10-16T13:49:35.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-16T13:49:35.314 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-10-16T13:49:35.324 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-16T13:49:35.324 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T13:49:35.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.457+0000 7fcda5bfa700 1 Processor -- start 2023-10-16T13:49:35.460 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.457+0000 7fcda5bfa700 1 -- start start 2023-10-16T13:49:35.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.459+0000 7fcda5bfa700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fcda00ae970 0x7fcda00ceee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:35.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.459+0000 7fcda5bfa700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcda00cf870 con 0x7fcda00cf420 2023-10-16T13:49:35.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.459+0000 7fcda5bfa700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcda00ccf70 con 0x7fcda015a390 2023-10-16T13:49:35.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.459+0000 7fcda5bfa700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcda0071a80 con 0x7fcda00ae970 2023-10-16T13:49:35.464 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.459+0000 7fcd9ffff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fcda00cf420 0x7fcda00b22d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49624/0 (socket says 172.21.15.16:49624) 2023-10-16T13:49:35.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.459+0000 7fcd9ffff700 1 -- 172.21.15.16:0/1143970424 learned_addr learned my addr 172.21.15.16:0/1143970424 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:49:35.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.459+0000 7fcd9effd700 1 --2- 172.21.15.16:0/1143970424 >> v2:172.21.15.49:3300/0 conn(0x7fcda00ae970 0x7fcda00ceee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:35.465 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.459+0000 7fcd9e7fc700 1 -- 172.21.15.16:0/1143970424 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3363078754 0 0) 0x7fcd98001490 con 0x7fcda00cf420 2023-10-16T13:49:35.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.459+0000 7fcd9e7fc700 1 -- 172.21.15.16:0/1143970424 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd880035b0 con 0x7fcda00cf420 2023-10-16T13:49:35.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.459+0000 7fcd9e7fc700 1 -- 172.21.15.16:0/1143970424 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3254619064 0 0) 0x7fcda00ccf70 con 0x7fcda015a390 2023-10-16T13:49:35.466 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.459+0000 7fcd9e7fc700 1 -- 172.21.15.16:0/1143970424 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd98001490 con 0x7fcda015a390 2023-10-16T13:49:35.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.459+0000 7fcd9effd700 1 -- 172.21.15.16:0/1143970424 >> v1:172.21.15.49:6789/0 conn(0x7fcda015a390 legacy=0x7fcda015e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:35.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.459+0000 7fcd9effd700 1 -- 172.21.15.16:0/1143970424 >> v1:172.21.15.16:6789/0 conn(0x7fcda00cf420 legacy=0x7fcda00b22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:35.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.459+0000 7fcd9effd700 1 -- 172.21.15.16:0/1143970424 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcda0071d60 con 0x7fcda00ae970 2023-10-16T13:49:35.467 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.459+0000 7fcd9effd700 1 --2- 172.21.15.16:0/1143970424 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcda00ae970 0x7fcda00ceee0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7fcd94009840 tx=0x7fcd94009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d6058422b0398724 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:35.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.460+0000 7fcd9e7fc700 1 -- 172.21.15.16:0/1143970424 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd94011830 con 0x7fcda00ae970 2023-10-16T13:49:35.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.460+0000 7fcd9e7fc700 1 -- 172.21.15.16:0/1143970424 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd94011990 con 0x7fcda00ae970 2023-10-16T13:49:35.468 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.460+0000 7fcd9e7fc700 1 -- 172.21.15.16:0/1143970424 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd94011d20 con 0x7fcda00ae970 2023-10-16T13:49:35.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.460+0000 7fcda5bfa700 1 -- 172.21.15.16:0/1143970424 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcda00ae970 msgr2=0x7fcda00ceee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:35.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.460+0000 7fcda5bfa700 1 --2- 172.21.15.16:0/1143970424 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcda00ae970 0x7fcda00ceee0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7fcd94009840 tx=0x7fcd94009b50 comp rx=0 tx=0).stop 2023-10-16T13:49:35.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.460+0000 7fcda5bfa700 1 -- 172.21.15.16:0/1143970424 shutdown_connections 2023-10-16T13:49:35.469 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.460+0000 7fcda5bfa700 1 --2- 172.21.15.16:0/1143970424 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcda00ae970 0x7fcda00ceee0 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:35.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.460+0000 7fcda5bfa700 1 -- 172.21.15.16:0/1143970424 >> 172.21.15.16:0/1143970424 conn(0x7fcda00b4cd0 msgr2=0x7fcda006a7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:35.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.460+0000 7fcda5bfa700 1 -- 172.21.15.16:0/1143970424 shutdown_connections 2023-10-16T13:49:35.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.460+0000 7fcda5bfa700 1 -- 172.21.15.16:0/1143970424 wait complete. 2023-10-16T13:49:35.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.461+0000 7fcda5bfa700 1 Processor -- start 2023-10-16T13:49:35.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.462+0000 7fcda5bfa700 1 -- start start 2023-10-16T13:49:35.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.462+0000 7fcda5bfa700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcda00ae970 0x7fcda01555a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:35.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.462+0000 7fcda5bfa700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcda00cf420 0x7fcda01f24d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:35.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.462+0000 7fcda5bfa700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcda015a390 0x7fcda00b13e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:35.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.462+0000 7fcda5bfa700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcda00aed50 con 0x7fcda00cf420 2023-10-16T13:49:35.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.462+0000 7fcda5bfa700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcda01f2a10 con 0x7fcda00ae970 2023-10-16T13:49:35.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.462+0000 7fcda5bfa700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcda00b1e10 con 0x7fcda015a390 2023-10-16T13:49:35.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.462+0000 7fcd9effd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcda00cf420 0x7fcda01f24d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:35.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.462+0000 7fcd9effd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcda00cf420 0x7fcda01f24d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60354/0 (socket says 172.21.15.16:60354) 2023-10-16T13:49:35.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.462+0000 7fcd9effd700 1 -- 172.21.15.16:0/3309901109 learned_addr learned my addr 172.21.15.16:0/3309901109 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:35.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.463+0000 7fcd9f7fe700 1 --2- 172.21.15.16:0/3309901109 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcda00ae970 0x7fcda01555a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:35.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.463+0000 7fcd9ffff700 1 --2- 172.21.15.16:0/3309901109 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcda015a390 0x7fcda00b13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:35.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.463+0000 7fcd9effd700 1 -- 172.21.15.16:0/3309901109 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcda015a390 msgr2=0x7fcda00b13e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:35.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.463+0000 7fcd9effd700 1 --2- 172.21.15.16:0/3309901109 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcda015a390 0x7fcda00b13e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:35.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.463+0000 7fcd9effd700 1 -- 172.21.15.16:0/3309901109 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcda00ae970 msgr2=0x7fcda01555a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:35.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.463+0000 7fcd9effd700 1 --2- 172.21.15.16:0/3309901109 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcda00ae970 0x7fcda01555a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:35.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.463+0000 7fcd9effd700 1 -- 172.21.15.16:0/3309901109 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd940094a0 con 0x7fcda00cf420 2023-10-16T13:49:35.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.463+0000 7fcd9ffff700 1 --2- 172.21.15.16:0/3309901109 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcda015a390 0x7fcda00b13e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:49:35.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.463+0000 7fcd9f7fe700 1 --2- 172.21.15.16:0/3309901109 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcda00ae970 0x7fcda01555a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:49:35.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.463+0000 7fcd9effd700 1 --2- 172.21.15.16:0/3309901109 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcda00cf420 0x7fcda01f24d0 secure :-1 s=READY pgs=3326 cs=0 l=1 rev1=1 crypto rx=0x7fcd94014040 tx=0x7fcd94004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:35.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.463+0000 7fcd87fff700 1 -- 172.21.15.16:0/3309901109 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd9401a860 con 0x7fcda00cf420 2023-10-16T13:49:35.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.463+0000 7fcd87fff700 1 -- 172.21.15.16:0/3309901109 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd9401a9c0 con 0x7fcda00cf420 2023-10-16T13:49:35.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.463+0000 7fcda5bfa700 1 -- 172.21.15.16:0/3309901109 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcda00aefa0 con 0x7fcda00cf420 2023-10-16T13:49:35.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.463+0000 7fcd87fff700 1 -- 172.21.15.16:0/3309901109 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd9401ad30 con 0x7fcda00cf420 2023-10-16T13:49:35.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.463+0000 7fcda5bfa700 1 -- 172.21.15.16:0/3309901109 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcda00af560 con 0x7fcda00cf420 2023-10-16T13:49:35.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.464+0000 7fcda5bfa700 1 -- 172.21.15.16:0/3309901109 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcda004ebb0 con 0x7fcda00cf420 2023-10-16T13:49:35.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.465+0000 7fcd87fff700 1 -- 172.21.15.16:0/3309901109 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 176) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fcd94022070 con 0x7fcda00cf420 2023-10-16T13:49:35.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.465+0000 7fcd87fff700 1 --2- 172.21.15.16:0/3309901109 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7fcd8c06b020 0x7fcd8c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:35.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.465+0000 7fcd87fff700 1 -- 172.21.15.16:0/3309901109 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(283..283 src has 1..283) v4 ==== 6208+0+0 (secure 0 0 0) 0x7fcd9408a9c0 con 0x7fcda00cf420 2023-10-16T13:49:35.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.465+0000 7fcd9f7fe700 1 --2- 172.21.15.16:0/3309901109 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7fcd8c06b020 0x7fcd8c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:35.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.466+0000 7fcd9f7fe700 1 --2- 172.21.15.16:0/3309901109 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7fcd8c06b020 0x7fcd8c06d4e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fcd90007fd0 tx=0x7fcd90009040 comp rx=0 tx=0).ready entity=mgr.13935 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:35.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.470+0000 7fcd87fff700 1 -- 172.21.15.16:0/3309901109 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcd94055800 con 0x7fcda00cf420 2023-10-16T13:49:35.672 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.669+0000 7fcda5bfa700 1 -- 172.21.15.16:0/3309901109 --> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fcda0072320 con 0x7fcd8c06b020 2023-10-16T13:49:35.674 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.672+0000 7fcd87fff700 1 -- 172.21.15.16:0/3309901109 <== mgr.13935 v2:172.21.15.16:6832/938897590 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+50873 (secure 0 0 0) 0x7fcd90001cd0 con 0x7fcd8c06b020 2023-10-16T13:49:35.675 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:49:35.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.674+0000 7fcda5bfa700 1 -- 172.21.15.16:0/3309901109 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7fcd8c06b020 msgr2=0x7fcd8c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:35.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.674+0000 7fcda5bfa700 1 --2- 172.21.15.16:0/3309901109 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7fcd8c06b020 0x7fcd8c06d4e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fcd90007fd0 tx=0x7fcd90009040 comp rx=0 tx=0).stop 2023-10-16T13:49:35.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.674+0000 7fcda5bfa700 1 -- 172.21.15.16:0/3309901109 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcda00cf420 msgr2=0x7fcda01f24d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:35.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.674+0000 7fcda5bfa700 1 --2- 172.21.15.16:0/3309901109 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcda00cf420 0x7fcda01f24d0 secure :-1 s=READY pgs=3326 cs=0 l=1 rev1=1 crypto rx=0x7fcd94014040 tx=0x7fcd94004750 comp rx=0 tx=0).stop 2023-10-16T13:49:35.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.674+0000 7fcda5bfa700 1 -- 172.21.15.16:0/3309901109 shutdown_connections 2023-10-16T13:49:35.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.674+0000 7fcda5bfa700 1 --2- 172.21.15.16:0/3309901109 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7fcd8c06b020 0x7fcd8c06d4e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:35.678 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.674+0000 7fcda5bfa700 1 --2- 172.21.15.16:0/3309901109 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fcda015a390 0x7fcda00b13e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:35.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.674+0000 7fcda5bfa700 1 --2- 172.21.15.16:0/3309901109 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcda00ae970 0x7fcda01555a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:35.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.674+0000 7fcda5bfa700 1 --2- 172.21.15.16:0/3309901109 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcda00cf420 0x7fcda01f24d0 unknown :-1 s=CLOSED pgs=3326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:35.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.674+0000 7fcda5bfa700 1 -- 172.21.15.16:0/3309901109 >> 172.21.15.16:0/3309901109 conn(0x7fcda00b4cd0 msgr2=0x7fcda00cd230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:35.679 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.674+0000 7fcda5bfa700 1 -- 172.21.15.16:0/3309901109 shutdown_connections 2023-10-16T13:49:35.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.674+0000 7fcda5bfa700 1 -- 172.21.15.16:0/3309901109 wait complete. 2023-10-16T13:49:35.680 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-10-16T13:49:35.688 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2023-10-16T13:49:33.841624+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2804,"num_read_kb":5868,"num_write":719,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":425,"ondisk_log_size":425,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":166,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":5,"kb":754974720,"kb_used":457976,"kb_used_data":29460,"kb_used_omap":12,"kb_used_meta":428403,"kb_avail":754516744,"statfs":{"total":773094113280,"available":772625145856,"internally_reserved":0,"allocated":30167040,"data_stored":16414522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12740,"internal_metadata":438685244},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-812260,"num_objects":-26,"num_object_clones":0,"num_object_copies":-52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-26,"num_whiteouts":0,"num_read":-312,"num_read_kb":-156,"num_write":-368,"num_write_kb":-3128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.999305"},"pg_stats":[{"pgid":"1.0","version":"281'425","reported_seq":1525,"reported_epoch":281,"state":"active+clean","last_fresh":"2023-10-16T13:49:29.910772+0000","last_change":"2023-10-16T12:10:34.746820+0000","last_active":"2023-10-16T13:49:29.910772+0000","last_peered":"2023-10-16T13:49:29.910772+0000","last_clean":"2023-10-16T13:49:29.910772+0000","last_became_active":"2023-10-16T12:10:34.745574+0000","last_became_peered":"2023-10-16T12:10:34.745574+0000","last_unstale":"2023-10-16T13:49:29.910772+0000","last_undegraded":"2023-10-16T13:49:29.910772+0000","last_fullsized":"2023-10-16T13:49:29.910772+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_clean_scrub_stamp":"2023-10-16T12:10:33.577111+0000","objects_scrubbed":0,"log_size":425,"log_dups_size":0,"ondisk_log_size":425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:14.461986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2804,"num_read_kb":5868,"num_write":719,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2804,"num_read_kb":5868,"num_write":719,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":425,"ondisk_log_size":425,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576017,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61616,"kb_used_data":4068,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310224,"statfs":{"total":96636764160,"available":96573669376,"internally_reserved":0,"allocated":4165632,"data_stored":2446539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58915275},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:48:54 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.58399999999999996,"15min":0.621},"min":{"1min":0.34999999999999998,"5min":0.34200000000000003,"15min":0.33000000000000002},"max":{"1min":4.1040000000000001,"5min":4.1040000000000001,"15min":4.1040000000000001},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.753,"5min":0.60699999999999998,"15min":0.625},"min":{"1min":0.35199999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":5.5279999999999996,"5min":5.5279999999999996,"15min":5.5279999999999996},"last":0.49199999999999999}]},{"osd":1,"last update":"Mon Oct 16 13:48:54 2023","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.63100000000000001,"15min":0.64300000000000002},"min":{"1min":0.33500000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":6.6189999999999998,"5min":6.6189999999999998,"15min":6.6189999999999998},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.59999999999999998,"15min":0.626},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":3.3239999999999998,"5min":3.3239999999999998,"15min":3.3239999999999998},"last":0.63700000000000001}]},{"osd":2,"last update":"Mon Oct 16 13:48:54 2023","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.64500000000000002,"15min":0.67700000000000005},"min":{"1min":0.41499999999999998,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":2.4430000000000001,"5min":2.4430000000000001,"15min":2.4430000000000001},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.67200000000000004,"15min":0.69799999999999995},"min":{"1min":0.45500000000000002,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":4.351,"5min":4.351,"15min":4.351},"last":0.59199999999999997}]},{"osd":3,"last update":"Mon Oct 16 13:48:54 2023","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.63300000000000001,"15min":0.65500000000000003},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":4.4130000000000003,"5min":4.4130000000000003,"15min":4.4130000000000003},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.67600000000000005,"15min":0.67100000000000004},"min":{"1min":0.432,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":5.407,"5min":5.407,"15min":5.407},"last":0.73999999999999999}]},{"osd":4,"last update":"Mon Oct 16 13:48:54 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.56100000000000005,"15min":0.53500000000000003},"min":{"1min":0.37,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":3.2400000000000002,"5min":3.2400000000000002,"15min":3.2400000000000002},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.67900000000000005,"15min":0.63900000000000001},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":5.3369999999999997,"5min":5.3369999999999997,"15min":5.3369999999999997},"last":0.75800000000000001}]},{"osd":5,"last update":"Mon Oct 16 13:48:54 2023","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.621,"15min":0.57199999999999995},"min":{"1min":0.35799999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":5.4720000000000004,"5min":5.4720000000000004,"15min":5.4720000000000004},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.61299999999999999,"15min":0.59799999999999998},"min":{"1min":0.40100000000000002,"5min":0.39700000000000002,"15min":0.35099999999999998},"max":{"1min":1.0549999999999999,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.61899999999999999}]},{"osd":6,"last update":"Mon Oct 16 13:48:54 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.63200000000000001,"15min":0.622},"min":{"1min":0.46600000000000003,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.69499999999999995,"15min":0.64500000000000002},"min":{"1min":0.38900000000000001,"5min":0.378,"15min":0.35199999999999998},"max":{"1min":5.3860000000000001,"5min":5.3860000000000001,"15min":5.3860000000000001},"last":0.79300000000000004}]}]},{"osd":6,"up_from":13,"seq":55834576014,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48536,"kb_used_data":3468,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323304,"statfs":{"total":96636764160,"available":96587063296,"internally_reserved":0,"allocated":3551232,"data_stored":1842161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.60999999999999999,"15min":0.65000000000000002},"min":{"1min":0.315,"5min":0.315,"15min":0.29599999999999999},"max":{"1min":0.67600000000000005,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.436},{"interface":"front","average":{"1min":0.503,"5min":0.60599999999999998,"15min":0.64200000000000002},"min":{"1min":0.373,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.70099999999999996,"5min":0.91400000000000003,"15min":0.93799999999999994},"last":0.48299999999999998}]},{"osd":1,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.54600000000000004,"15min":0.60499999999999998},"min":{"1min":0.316,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.64500000000000002,"5min":0.86899999999999999,"15min":0.89600000000000002},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.58599999999999997,"15min":0.60599999999999998},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.69799999999999995,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.39000000000000001}]},{"osd":2,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.58899999999999997,"15min":0.61799999999999999},"min":{"1min":0.34799999999999998,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.71499999999999997,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.31},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.57599999999999996,"15min":0.60599999999999998},"min":{"1min":0.32200000000000001,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.72999999999999998,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.51200000000000001}]},{"osd":3,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.60199999999999998,"15min":0.64600000000000002},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.38200000000000001},"max":{"1min":0.63400000000000001,"5min":0.92400000000000004,"15min":0.95599999999999996},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.66600000000000004,"15min":0.70199999999999996},"min":{"1min":0.435,"5min":0.435,"15min":0.42299999999999999},"max":{"1min":0.72399999999999998,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.35499999999999998}]},{"osd":4,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.53300000000000003,"15min":0.52800000000000002},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.67700000000000005,"5min":0.85599999999999998,"15min":0.95799999999999996},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.60799999999999998,"15min":0.61499999999999999},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.70799999999999996,"5min":0.98799999999999999,"15min":1.0309999999999999},"last":0.40600000000000003}]},{"osd":5,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.58199999999999996,"15min":0.59099999999999997},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.71099999999999997,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.375},{"interface":"front","average":{"1min":0.627,"5min":0.64400000000000002,"15min":0.65700000000000003},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.78700000000000003,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.46999999999999997}]},{"osd":7,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.59099999999999997,"15min":0.59799999999999998},"min":{"1min":0.35199999999999998,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.747,"5min":1.081,"15min":1.081},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.63900000000000001,"15min":0.64900000000000002},"min":{"1min":0.40999999999999998,"5min":0.40899999999999997,"15min":0.379},"max":{"1min":0.76600000000000001,"5min":1.099,"15min":1.099},"last":0.496}]}]},{"osd":1,"up_from":13,"seq":55834576015,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52732,"kb_used_data":3504,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319108,"statfs":{"total":96636764160,"available":96582766592,"internally_reserved":0,"allocated":3588096,"data_stored":1863001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":50395593},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:48:59 2023","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.37,"15min":0.40100000000000002},"min":{"1min":0.254,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.58099999999999996,"5min":0.65400000000000003,"15min":0.65800000000000003},"last":0.379},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.433,"15min":0.48099999999999998},"min":{"1min":0.32700000000000001,"5min":0.253,"15min":0.253},"max":{"1min":0.56599999999999995,"5min":0.68000000000000005,"15min":0.77200000000000002},"last":0.40200000000000002}]},{"osd":2,"last update":"Mon Oct 16 13:48:59 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.39400000000000002,"15min":0.439},"min":{"1min":0.251,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.496,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.44500000000000001,"15min":0.47999999999999998},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.58299999999999996,"5min":0.67600000000000005,"15min":0.71399999999999997},"last":0.438}]},{"osd":3,"last update":"Mon Oct 16 13:48:59 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.39000000000000001,"15min":0.42299999999999999},"min":{"1min":0.28499999999999998,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.51300000000000001,"5min":0.65200000000000002,"15min":0.72099999999999997},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.41699999999999998,"15min":0.46600000000000003},"min":{"1min":0.29599999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.60299999999999998,"5min":0.69399999999999995,"15min":0.77300000000000002},"last":0.41699999999999998}]},{"osd":4,"last update":"Mon Oct 16 13:48:59 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.61599999999999999,"15min":0.65700000000000003},"min":{"1min":0.41199999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.78300000000000003,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.59899999999999998,"15min":0.65600000000000003},"min":{"1min":0.377,"5min":0.37,"15min":0.37},"max":{"1min":0.90400000000000003,"5min":0.91900000000000004,"15min":0.997},"last":0.54100000000000004}]},{"osd":5,"last update":"Mon Oct 16 13:48:59 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.64700000000000002,"15min":0.69899999999999995},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":1.0760000000000001},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.65700000000000003,"15min":0.71499999999999997},"min":{"1min":0.436,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.81299999999999994,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.49199999999999999}]},{"osd":6,"last update":"Mon Oct 16 13:48:59 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.621,"15min":0.66900000000000004},"min":{"1min":0.41999999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85799999999999998,"5min":0.95099999999999996,"15min":0.97799999999999998},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.62,"5min":0.64500000000000002,"15min":0.69499999999999995},"min":{"1min":0.46899999999999997,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":1.083},"last":0.66500000000000004}]},{"osd":7,"last update":"Mon Oct 16 13:48:59 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.66300000000000003,"15min":0.72199999999999998},"min":{"1min":0.45300000000000001,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.79300000000000004,"5min":1.163,"15min":1.163},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.68799999999999994,"15min":0.73699999999999999},"min":{"1min":0.47999999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.94699999999999995,"5min":0.98499999999999999,"15min":1.0580000000000001},"last":0.57699999999999996}]}]},{"osd":0,"up_from":13,"seq":55834576016,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53312,"kb_used_data":4084,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318528,"statfs":{"total":96636764160,"available":96582172672,"internally_reserved":0,"allocated":4182016,"data_stored":2453369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.373,"15min":0.41299999999999998},"min":{"1min":0.23799999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.50900000000000001,"5min":0.60699999999999998,"15min":0.71399999999999997},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.38500000000000001,"15min":0.42799999999999999},"min":{"1min":0.24199999999999999,"5min":0.222,"15min":0.222},"max":{"1min":0.56799999999999995,"5min":0.77700000000000002,"15min":0.82199999999999995},"last":0.54100000000000004}]},{"osd":2,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.40999999999999998,"15min":0.441},"min":{"1min":0.27300000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.51000000000000001,"5min":0.65900000000000003,"15min":0.75700000000000001},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.432,"15min":0.47199999999999998},"min":{"1min":0.30399999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.54500000000000004,"5min":0.68799999999999994,"15min":0.83299999999999996},"last":0.41599999999999998}]},{"osd":3,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.40600000000000003,"15min":0.44400000000000001},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.52100000000000002,"5min":0.69699999999999995,"15min":0.78600000000000003},"last":0.502},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.46300000000000002,"15min":0.50600000000000001},"min":{"1min":0.309,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.55900000000000005,"5min":0.70399999999999996,"15min":0.86399999999999999},"last":0.51600000000000001}]},{"osd":4,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.64500000000000002,"15min":0.67600000000000005},"min":{"1min":0.42999999999999999,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.82999999999999996,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.64900000000000002,"15min":0.68100000000000005},"min":{"1min":0.43099999999999999,"5min":0.375,"15min":0.375},"max":{"1min":0.81000000000000005,"5min":1.21,"15min":1.21},"last":0.73999999999999999}]},{"osd":5,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.63400000000000001,"15min":0.67700000000000005},"min":{"1min":0.40300000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.82199999999999995,"5min":1.242,"15min":1.242},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.622,"5min":0.68300000000000005,"15min":0.71799999999999997},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.83299999999999996,"5min":1.081,"15min":1.081},"last":0.80900000000000005}]},{"osd":6,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.67600000000000005,"15min":0.71999999999999997},"min":{"1min":0.43099999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.84699999999999998,"5min":1.034,"15min":1.103},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.67700000000000005,"15min":0.71499999999999997},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.90900000000000003,"5min":1.131,"15min":1.131},"last":0.83399999999999996}]},{"osd":7,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.68700000000000006,"15min":0.73499999999999999},"min":{"1min":0.47399999999999998,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.94799999999999995,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.69599999999999995,"15min":0.72699999999999998},"min":{"1min":0.47899999999999998,"5min":0.433,"15min":0.433},"max":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.85599999999999998}]}]},{"osd":2,"up_from":13,"seq":55834576016,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56880,"kb_used_data":3492,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314960,"statfs":{"total":96636764160,"available":96578519040,"internally_reserved":0,"allocated":3575808,"data_stored":1856290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:48:41 2023","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.374,"15min":0.40400000000000003},"min":{"1min":0.23899999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.439,"5min":0.623,"15min":0.77500000000000002},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.436,"15min":0.46999999999999997},"min":{"1min":0.33700000000000002,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.57399999999999995,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.48699999999999999}]},{"osd":1,"last update":"Mon Oct 16 13:48:41 2023","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.46100000000000002,"15min":0.49199999999999999},"min":{"1min":0.30499999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.69899999999999995,"5min":0.73199999999999998,"15min":0.81399999999999995},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.45000000000000001,"15min":0.46899999999999997},"min":{"1min":0.27900000000000003,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.51700000000000002,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.54500000000000004}]},{"osd":3,"last update":"Mon Oct 16 13:48:41 2023","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.40400000000000003,"15min":0.435},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.55500000000000005,"5min":0.63,"15min":0.72599999999999998},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.499,"15min":0.53700000000000003},"min":{"1min":0.376,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.65900000000000003,"5min":0.82699999999999996,"15min":0.90100000000000002},"last":0.63300000000000001}]},{"osd":4,"last update":"Mon Oct 16 13:48:41 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.66800000000000004,"15min":0.70699999999999996},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.423,"5min":1.423,"15min":1.423},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.67700000000000005,"15min":0.71399999999999997},"min":{"1min":0.437,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.407,"5min":1.407,"15min":1.407},"last":0.81399999999999995}]},{"osd":5,"last update":"Mon Oct 16 13:48:41 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.627,"15min":0.67800000000000005},"min":{"1min":0.436,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.82199999999999995,"5min":0.87,"15min":1.486},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.64900000000000002,"15min":0.68999999999999995},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.78200000000000003,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":0.83299999999999996}]},{"osd":6,"last update":"Mon Oct 16 13:48:41 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.622,"15min":0.65900000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.80300000000000005,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.745},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.66200000000000003,"15min":0.71099999999999997},"min":{"1min":0.49099999999999999,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.85499999999999998,"5min":0.94299999999999995,"15min":1.4379999999999999},"last":0.80000000000000004}]},{"osd":7,"last update":"Mon Oct 16 13:48:41 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.67500000000000004,"15min":0.70899999999999996},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.83499999999999996,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.66800000000000004,"15min":0.72199999999999998},"min":{"1min":0.48799999999999999,"5min":0.436,"15min":0.436},"max":{"1min":0.88200000000000001,"5min":1.0429999999999999,"15min":1.377},"last":0.88500000000000001}]}]},{"osd":3,"up_from":13,"seq":55834576014,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57220,"kb_used_data":3836,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314620,"statfs":{"total":96636764160,"available":96578170880,"internally_reserved":0,"allocated":3928064,"data_stored":2227160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:48:45 2023","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.375,"15min":0.40400000000000003},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23000000000000001},"max":{"1min":0.48399999999999999,"5min":0.60099999999999998,"15min":0.70499999999999996},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.38300000000000001,"15min":0.41599999999999998},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.48799999999999999,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.34999999999999998}]},{"osd":1,"last update":"Mon Oct 16 13:48:45 2023","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.38900000000000001,"15min":0.42299999999999999},"min":{"1min":0.27200000000000002,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.497,"5min":0.68200000000000005,"15min":0.76300000000000001},"last":0.439},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.438,"15min":0.46999999999999997},"min":{"1min":0.27800000000000002,"5min":0.27700000000000002,"15min":0.253},"max":{"1min":0.60599999999999998,"5min":0.68700000000000006,"15min":0.77800000000000002},"last":0.32000000000000001}]},{"osd":2,"last update":"Mon Oct 16 13:48:45 2023","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.45100000000000001,"15min":0.47999999999999998},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.71399999999999997,"5min":1.167,"15min":1.167},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.46000000000000002,"15min":0.50700000000000001},"min":{"1min":0.32500000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.63200000000000001,"5min":0.73299999999999998,"15min":0.76800000000000002},"last":0.379}]},{"osd":4,"last update":"Mon Oct 16 13:48:45 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.61799999999999999,"15min":0.65500000000000003},"min":{"1min":0.40100000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.137},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65400000000000003,"15min":0.68899999999999995},"min":{"1min":0.45500000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.84699999999999998,"5min":1.2110000000000001,"15min":1.2909999999999999},"last":0.55400000000000005}]},{"osd":5,"last update":"Mon Oct 16 13:48:45 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.65200000000000002,"15min":0.69799999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.374},"max":{"1min":1.0309999999999999,"5min":1.0680000000000001,"15min":1.2729999999999999},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.65400000000000003,"15min":0.68999999999999995},"min":{"1min":0.47299999999999998,"5min":0.45600000000000002,"15min":0.432},"max":{"1min":0.871,"5min":1.0549999999999999,"15min":1.173},"last":0.53900000000000003}]},{"osd":6,"last update":"Mon Oct 16 13:48:45 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.69799999999999995,"15min":0.71299999999999997},"min":{"1min":0.432,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.115,"5min":5.0999999999999996,"15min":5.0999999999999996},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.68600000000000005,"15min":0.72499999999999998},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.2290000000000001},"last":0.45600000000000002}]},{"osd":7,"last update":"Mon Oct 16 13:48:45 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.68100000000000005,"15min":0.72699999999999998},"min":{"1min":0.45700000000000002,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.21,"5min":1.21,"15min":1.2569999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.68700000000000006,"15min":0.73399999999999999},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.43099999999999999},"max":{"1min":0.90600000000000003,"5min":1.262,"15min":1.262},"last":0.67100000000000004}]}]},{"osd":4,"up_from":13,"seq":55834576014,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52736,"kb_used_data":3504,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319104,"statfs":{"total":96636764160,"available":96582762496,"internally_reserved":0,"allocated":3588096,"data_stored":1863001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:48:42 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.56299999999999994,"15min":0.61299999999999999},"min":{"1min":0.36399999999999999,"5min":0.314,"15min":0.314},"max":{"1min":0.77800000000000002,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.57399999999999995,"15min":0.624},"min":{"1min":0.36599999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.87,"5min":0.87,"15min":0.89500000000000002},"last":0.56899999999999995}]},{"osd":1,"last update":"Mon Oct 16 13:48:42 2023","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.57499999999999996,"15min":0.64100000000000001},"min":{"1min":0.35699999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.79900000000000004,"5min":0.877,"15min":0.877},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.58399999999999996,"15min":0.63},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.80900000000000005,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.39900000000000002}]},{"osd":2,"last update":"Mon Oct 16 13:48:42 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.59199999999999997,"15min":0.64100000000000001},"min":{"1min":0.434,"5min":0.378,"15min":0.378},"max":{"1min":0.751,"5min":0.89900000000000002,"15min":0.90600000000000003},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.62,"15min":0.65300000000000002},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.755,"5min":0.89700000000000002,"15min":0.92600000000000005},"last":0.50900000000000001}]},{"osd":3,"last update":"Mon Oct 16 13:48:42 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.58699999999999997,"15min":0.63300000000000001},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.81399999999999995,"5min":0.86599999999999999,"15min":0.99199999999999999},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.624,"5min":0.64400000000000002,"15min":0.67500000000000004},"min":{"1min":0.48199999999999998,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.93899999999999995},"last":0.629}]},{"osd":5,"last update":"Mon Oct 16 13:48:42 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.55700000000000005,"15min":0.53800000000000003},"min":{"1min":0.34899999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.80400000000000005,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.57799999999999996,"15min":0.57099999999999995},"min":{"1min":0.307,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.72499999999999998,"5min":0.82199999999999995,"15min":0.94899999999999995},"last":0.745}]},{"osd":6,"last update":"Mon Oct 16 13:48:42 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.63200000000000001,"15min":0.60399999999999998},"min":{"1min":0.38600000000000001,"5min":0.36399999999999999,"15min":0.33200000000000002},"max":{"1min":0.82699999999999996,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.70099999999999996,"15min":0.69699999999999995},"min":{"1min":0.41699999999999998,"5min":0.40300000000000002,"15min":0.39400000000000002},"max":{"1min":0.89700000000000002,"5min":1.129,"15min":1.129},"last":0.71899999999999997}]},{"osd":7,"last update":"Mon Oct 16 13:48:42 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.65100000000000002,"15min":0.63700000000000001},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.78700000000000003,"5min":0.97099999999999997,"15min":0.98599999999999999},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.70999999999999996,"15min":0.68899999999999995},"min":{"1min":0.48399999999999999,"5min":0.44,"15min":0.38200000000000001},"max":{"1min":0.96999999999999997,"5min":0.97599999999999998,"15min":1.026},"last":0.76600000000000001}]}]},{"osd":5,"up_from":13,"seq":55834576015,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":74944,"kb_used_data":3504,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94296896,"statfs":{"total":96636764160,"available":96560021504,"internally_reserved":0,"allocated":3588096,"data_stored":1863001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:49:30 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.54100000000000004,"15min":0.59999999999999998},"min":{"1min":0.39000000000000001,"5min":0.314,"15min":0.29899999999999999},"max":{"1min":0.73399999999999999,"5min":0.878,"15min":0.92900000000000005},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.56399999999999995,"15min":0.61699999999999999},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.29699999999999999},"max":{"1min":0.80000000000000004,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.55300000000000005}]},{"osd":1,"last update":"Mon Oct 16 13:49:30 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.58699999999999997,"15min":0.63900000000000001},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.29499999999999998},"max":{"1min":0.72999999999999998,"5min":1,"15min":1},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56799999999999995,"15min":0.62},"min":{"1min":0.45400000000000001,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.78100000000000003,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.629}]},{"osd":2,"last update":"Mon Oct 16 13:49:30 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.58999999999999997,"15min":0.63600000000000001},"min":{"1min":0.38100000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.71199999999999997,"5min":0.91000000000000003,"15min":0.94699999999999995},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.58099999999999996,"15min":0.63700000000000001},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.81699999999999995,"5min":0.83899999999999997,"15min":0.93500000000000005},"last":0.83099999999999996}]},{"osd":3,"last update":"Mon Oct 16 13:49:30 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.61699999999999999,"15min":0.65200000000000002},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.83599999999999997,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.63500000000000001,"15min":0.68000000000000005},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.84799999999999998,"5min":1.4199999999999999,"15min":1.4199999999999999},"last":0.73399999999999999}]},{"osd":4,"last update":"Mon Oct 16 13:49:30 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53600000000000003,"15min":0.54000000000000004},"min":{"1min":0.33300000000000002,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.77200000000000002,"5min":0.89700000000000002,"15min":0.90700000000000003},"last":0.88800000000000001},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.60099999999999998,"15min":0.59499999999999997},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.84799999999999998,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.91900000000000004}]},{"osd":6,"last update":"Mon Oct 16 13:49:30 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.59499999999999997,"15min":0.59099999999999997},"min":{"1min":0.35699999999999998,"5min":0.315,"15min":0.315},"max":{"1min":0.82799999999999996,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.61499999999999999,"15min":0.623},"min":{"1min":0.374,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.82799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.95999999999999996}]},{"osd":7,"last update":"Mon Oct 16 13:49:30 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.65800000000000003,"15min":0.64300000000000002},"min":{"1min":0.374,"5min":0.374,"15min":0.34999999999999998},"max":{"1min":0.878,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.97199999999999998},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.69499999999999995,"15min":0.69099999999999995},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.86899999999999999,"5min":0.97299999999999998,"15min":1.0229999999999999},"last":0.85799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T13:49:35.691 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T13:49:35.815 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.813+0000 7f688561a700 1 Processor -- start 2023-10-16T13:49:35.816 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.813+0000 7f688561a700 1 -- start start 2023-10-16T13:49:35.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.814+0000 7f688561a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f688015d8d0 0x7f68800d1c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:35.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.814+0000 7f688561a700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f68800d41c0 0x7f68800d0040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:35.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.814+0000 7f688561a700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f688006ba00 con 0x7f68800d0580 2023-10-16T13:49:35.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.814+0000 7f688561a700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68800d0a10 con 0x7f68800d41c0 2023-10-16T13:49:35.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.814+0000 7f688561a700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68800d2150 con 0x7f688015d8d0 2023-10-16T13:49:35.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.814+0000 7f687f7fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f68800d0580 0x7f68800ce240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49638/0 (socket says 172.21.15.16:49638) 2023-10-16T13:49:35.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.814+0000 7f687f7fe700 1 -- 172.21.15.16:0/380178019 learned_addr learned my addr 172.21.15.16:0/380178019 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:49:35.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.814+0000 7f687e7fc700 1 --2- 172.21.15.16:0/380178019 >> v2:172.21.15.49:3300/0 conn(0x7f68800d41c0 0x7f68800d0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:35.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.814+0000 7f687effd700 1 --2- 172.21.15.16:0/380178019 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f688015d8d0 0x7f68800d1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:35.821 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.814+0000 7f687dffb700 1 -- 172.21.15.16:0/380178019 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3364281781 0 0) 0x7f688006ba00 con 0x7f68800d0580 2023-10-16T13:49:35.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.814+0000 7f687dffb700 1 -- 172.21.15.16:0/380178019 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68680035b0 con 0x7f68800d0580 2023-10-16T13:49:35.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.814+0000 7f687effd700 1 -- 172.21.15.16:0/380178019 >> v2:172.21.15.49:3300/0 conn(0x7f68800d41c0 msgr2=0x7f68800d0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:35.822 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.814+0000 7f687effd700 1 --2- 172.21.15.16:0/380178019 >> v2:172.21.15.49:3300/0 conn(0x7f68800d41c0 0x7f68800d0040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:35.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.814+0000 7f687effd700 1 -- 172.21.15.16:0/380178019 >> v1:172.21.15.16:6789/0 conn(0x7f68800d0580 legacy=0x7f68800ce240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:35.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.814+0000 7f687effd700 1 -- 172.21.15.16:0/380178019 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68800d26d0 con 0x7f688015d8d0 2023-10-16T13:49:35.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.814+0000 7f687e7fc700 1 --2- 172.21.15.16:0/380178019 >> v2:172.21.15.49:3300/0 conn(0x7f68800d41c0 0x7f68800d0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:49:35.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.814+0000 7f687effd700 1 --2- 172.21.15.16:0/380178019 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f688015d8d0 0x7f68800d1c10 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f6870009840 tx=0x7f6870009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=28e9b9eeedb5b95a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:35.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.815+0000 7f687dffb700 1 -- 172.21.15.16:0/380178019 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6870011830 con 0x7f688015d8d0 2023-10-16T13:49:35.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.815+0000 7f687dffb700 1 -- 172.21.15.16:0/380178019 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6870011990 con 0x7f688015d8d0 2023-10-16T13:49:35.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.815+0000 7f687dffb700 1 -- 172.21.15.16:0/380178019 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6870011d20 con 0x7f688015d8d0 2023-10-16T13:49:35.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.815+0000 7f688561a700 1 -- 172.21.15.16:0/380178019 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f688015d8d0 msgr2=0x7f68800d1c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:35.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.815+0000 7f688561a700 1 --2- 172.21.15.16:0/380178019 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f688015d8d0 0x7f68800d1c10 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f6870009840 tx=0x7f6870009b50 comp rx=0 tx=0).stop 2023-10-16T13:49:35.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.815+0000 7f688561a700 1 -- 172.21.15.16:0/380178019 shutdown_connections 2023-10-16T13:49:35.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.815+0000 7f688561a700 1 --2- 172.21.15.16:0/380178019 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f688015d8d0 0x7f68800d1c10 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:35.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.815+0000 7f688561a700 1 --2- 172.21.15.16:0/380178019 >> v2:172.21.15.49:3300/0 conn(0x7f68800d41c0 0x7f68800d0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:35.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.815+0000 7f688561a700 1 -- 172.21.15.16:0/380178019 >> 172.21.15.16:0/380178019 conn(0x7f68800d3630 msgr2=0x7f68800d3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:35.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.815+0000 7f688561a700 1 -- 172.21.15.16:0/380178019 shutdown_connections 2023-10-16T13:49:35.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.815+0000 7f688561a700 1 -- 172.21.15.16:0/380178019 wait complete. 2023-10-16T13:49:35.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.816+0000 7f688561a700 1 Processor -- start 2023-10-16T13:49:35.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.817+0000 7f688561a700 1 -- start start 2023-10-16T13:49:35.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.817+0000 7f688561a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f68800d0580 0x7f68801f5cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:35.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.817+0000 7f688561a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68800d41c0 0x7f68800ccfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:35.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.817+0000 7f688561a700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f688015d8d0 0x7f68800b17f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:35.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.817+0000 7f688561a700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68801e6b50 con 0x7f68800d41c0 2023-10-16T13:49:35.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.817+0000 7f688561a700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68800cb4b0 con 0x7f688015d8d0 2023-10-16T13:49:35.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.817+0000 7f688561a700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68800ca830 con 0x7f68800d0580 2023-10-16T13:49:35.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.817+0000 7f687e7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68800d41c0 0x7f68800ccfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:35.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.817+0000 7f687e7fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68800d41c0 0x7f68800ccfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60370/0 (socket says 172.21.15.16:60370) 2023-10-16T13:49:35.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.817+0000 7f687e7fc700 1 -- 172.21.15.16:0/1680103476 learned_addr learned my addr 172.21.15.16:0/1680103476 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:35.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.817+0000 7f687effd700 1 --2- 172.21.15.16:0/1680103476 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f68800d0580 0x7f68801f5cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:35.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.818+0000 7f687f7fe700 1 --2- 172.21.15.16:0/1680103476 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f688015d8d0 0x7f68800b17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:35.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.818+0000 7f687e7fc700 1 -- 172.21.15.16:0/1680103476 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f68800d0580 msgr2=0x7f68801f5cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:35.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.818+0000 7f687e7fc700 1 --2- 172.21.15.16:0/1680103476 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f68800d0580 0x7f68801f5cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:35.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.818+0000 7f687e7fc700 1 -- 172.21.15.16:0/1680103476 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f688015d8d0 msgr2=0x7f68800b17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:35.833 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.818+0000 7f687e7fc700 1 --2- 172.21.15.16:0/1680103476 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f688015d8d0 0x7f68800b17f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:35.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.818+0000 7f687e7fc700 1 -- 172.21.15.16:0/1680103476 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68740094a0 con 0x7f68800d41c0 2023-10-16T13:49:35.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.818+0000 7f687e7fc700 1 --2- 172.21.15.16:0/1680103476 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68800d41c0 0x7f68800ccfc0 secure :-1 s=READY pgs=3328 cs=0 l=1 rev1=1 crypto rx=0x7f687400ebe0 tx=0x7f687400eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:35.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.818+0000 7f68677fe700 1 -- 172.21.15.16:0/1680103476 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f687400c850 con 0x7f68800d41c0 2023-10-16T13:49:35.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.818+0000 7f68677fe700 1 -- 172.21.15.16:0/1680103476 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f687400c9b0 con 0x7f68800d41c0 2023-10-16T13:49:35.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.818+0000 7f688561a700 1 -- 172.21.15.16:0/1680103476 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68700094a0 con 0x7f68800d41c0 2023-10-16T13:49:35.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.818+0000 7f68677fe700 1 -- 172.21.15.16:0/1680103476 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f687400cd40 con 0x7f68800d41c0 2023-10-16T13:49:35.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.818+0000 7f688561a700 1 -- 172.21.15.16:0/1680103476 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68800cadf0 con 0x7f68800d41c0 2023-10-16T13:49:35.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.819+0000 7f688561a700 1 -- 172.21.15.16:0/1680103476 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68800cb5f0 con 0x7f68800d41c0 2023-10-16T13:49:35.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.820+0000 7f68677fe700 1 -- 172.21.15.16:0/1680103476 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 176) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6874013470 con 0x7f68800d41c0 2023-10-16T13:49:35.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.820+0000 7f68677fe700 1 --2- 172.21.15.16:0/1680103476 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f686c06b020 0x7f686c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:35.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.820+0000 7f68677fe700 1 -- 172.21.15.16:0/1680103476 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(283..283 src has 1..283) v4 ==== 6208+0+0 (secure 0 0 0) 0x7f6874089930 con 0x7f68800d41c0 2023-10-16T13:49:35.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.820+0000 7f687effd700 1 --2- 172.21.15.16:0/1680103476 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f686c06b020 0x7f686c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:35.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.821+0000 7f687effd700 1 --2- 172.21.15.16:0/1680103476 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f686c06b020 0x7f686c06d4e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6870000c00 tx=0x7f6870008030 comp rx=0 tx=0).ready entity=mgr.13935 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:35.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:35.825+0000 7f68677fe700 1 -- 172.21.15.16:0/1680103476 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f68800cb5f0 con 0x7f68800d41c0 2023-10-16T13:49:36.030 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.027+0000 7f688561a700 1 -- 172.21.15.16:0/1680103476 --> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f688006bca0 con 0x7f686c06b020 2023-10-16T13:49:36.032 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.030+0000 7f68677fe700 1 -- 172.21.15.16:0/1680103476 <== mgr.13935 v2:172.21.15.16:6832/938897590 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+50916 (secure 0 0 0) 0x7f6870011b00 con 0x7f686c06b020 2023-10-16T13:49:36.033 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:49:36.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.032+0000 7f688561a700 1 -- 172.21.15.16:0/1680103476 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f686c06b020 msgr2=0x7f686c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:36.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.032+0000 7f688561a700 1 --2- 172.21.15.16:0/1680103476 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f686c06b020 0x7f686c06d4e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6870000c00 tx=0x7f6870008030 comp rx=0 tx=0).stop 2023-10-16T13:49:36.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.032+0000 7f688561a700 1 -- 172.21.15.16:0/1680103476 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68800d41c0 msgr2=0x7f68800ccfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:36.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.032+0000 7f688561a700 1 --2- 172.21.15.16:0/1680103476 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68800d41c0 0x7f68800ccfc0 secure :-1 s=READY pgs=3328 cs=0 l=1 rev1=1 crypto rx=0x7f687400ebe0 tx=0x7f687400eef0 comp rx=0 tx=0).stop 2023-10-16T13:49:36.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.032+0000 7f688561a700 1 -- 172.21.15.16:0/1680103476 shutdown_connections 2023-10-16T13:49:36.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.032+0000 7f688561a700 1 --2- 172.21.15.16:0/1680103476 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f686c06b020 0x7f686c06d4e0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:36.036 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.032+0000 7f688561a700 1 --2- 172.21.15.16:0/1680103476 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f688015d8d0 0x7f68800b17f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:36.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.032+0000 7f688561a700 1 --2- 172.21.15.16:0/1680103476 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f68800d0580 0x7f68801f5cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:36.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.032+0000 7f688561a700 1 --2- 172.21.15.16:0/1680103476 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68800d41c0 0x7f68800ccfc0 unknown :-1 s=CLOSED pgs=3328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:36.037 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.032+0000 7f688561a700 1 -- 172.21.15.16:0/1680103476 >> 172.21.15.16:0/1680103476 conn(0x7f68800d3630 msgr2=0x7f688015cb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:36.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.032+0000 7f688561a700 1 -- 172.21.15.16:0/1680103476 shutdown_connections 2023-10-16T13:49:36.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.032+0000 7f688561a700 1 -- 172.21.15.16:0/1680103476 wait complete. 2023-10-16T13:49:36.039 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-10-16T13:49:36.046 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2023-10-16T13:49:35.841899+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2804,"num_read_kb":5868,"num_write":719,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":425,"ondisk_log_size":425,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":155,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":3,"kb":754974720,"kb_used":457676,"kb_used_data":29148,"kb_used_omap":12,"kb_used_meta":428403,"kb_avail":754517044,"statfs":{"total":773094113280,"available":772625453056,"internally_reserved":0,"allocated":29847552,"data_stored":16064373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12740,"internal_metadata":438685244},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-812260,"num_objects":-26,"num_object_clones":0,"num_object_copies":-52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-26,"num_whiteouts":0,"num_read":-312,"num_read_kb":-156,"num_write":-368,"num_write_kb":-3128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.861329"},"pg_stats":[{"pgid":"1.0","version":"281'425","reported_seq":1525,"reported_epoch":281,"state":"active+clean","last_fresh":"2023-10-16T13:49:29.910772+0000","last_change":"2023-10-16T12:10:34.746820+0000","last_active":"2023-10-16T13:49:29.910772+0000","last_peered":"2023-10-16T13:49:29.910772+0000","last_clean":"2023-10-16T13:49:29.910772+0000","last_became_active":"2023-10-16T12:10:34.745574+0000","last_became_peered":"2023-10-16T12:10:34.745574+0000","last_unstale":"2023-10-16T13:49:29.910772+0000","last_undegraded":"2023-10-16T13:49:29.910772+0000","last_fullsized":"2023-10-16T13:49:29.910772+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_clean_scrub_stamp":"2023-10-16T12:10:33.577111+0000","objects_scrubbed":0,"log_size":425,"log_dups_size":0,"ondisk_log_size":425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:14.461986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2804,"num_read_kb":5868,"num_write":719,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2804,"num_read_kb":5868,"num_write":719,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":425,"ondisk_log_size":425,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576017,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61616,"kb_used_data":4068,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310224,"statfs":{"total":96636764160,"available":96573669376,"internally_reserved":0,"allocated":4165632,"data_stored":2446539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58915275},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:48:54 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.58399999999999996,"15min":0.621},"min":{"1min":0.34999999999999998,"5min":0.34200000000000003,"15min":0.33000000000000002},"max":{"1min":4.1040000000000001,"5min":4.1040000000000001,"15min":4.1040000000000001},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.753,"5min":0.60699999999999998,"15min":0.625},"min":{"1min":0.35199999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":5.5279999999999996,"5min":5.5279999999999996,"15min":5.5279999999999996},"last":0.49199999999999999}]},{"osd":1,"last update":"Mon Oct 16 13:48:54 2023","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.63100000000000001,"15min":0.64300000000000002},"min":{"1min":0.33500000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":6.6189999999999998,"5min":6.6189999999999998,"15min":6.6189999999999998},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.59999999999999998,"15min":0.626},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":3.3239999999999998,"5min":3.3239999999999998,"15min":3.3239999999999998},"last":0.63700000000000001}]},{"osd":2,"last update":"Mon Oct 16 13:48:54 2023","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.64500000000000002,"15min":0.67700000000000005},"min":{"1min":0.41499999999999998,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":2.4430000000000001,"5min":2.4430000000000001,"15min":2.4430000000000001},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.67200000000000004,"15min":0.69799999999999995},"min":{"1min":0.45500000000000002,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":4.351,"5min":4.351,"15min":4.351},"last":0.59199999999999997}]},{"osd":3,"last update":"Mon Oct 16 13:48:54 2023","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.63300000000000001,"15min":0.65500000000000003},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":4.4130000000000003,"5min":4.4130000000000003,"15min":4.4130000000000003},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.67600000000000005,"15min":0.67100000000000004},"min":{"1min":0.432,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":5.407,"5min":5.407,"15min":5.407},"last":0.73999999999999999}]},{"osd":4,"last update":"Mon Oct 16 13:48:54 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.56100000000000005,"15min":0.53500000000000003},"min":{"1min":0.37,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":3.2400000000000002,"5min":3.2400000000000002,"15min":3.2400000000000002},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.67900000000000005,"15min":0.63900000000000001},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":5.3369999999999997,"5min":5.3369999999999997,"15min":5.3369999999999997},"last":0.75800000000000001}]},{"osd":5,"last update":"Mon Oct 16 13:48:54 2023","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.621,"15min":0.57199999999999995},"min":{"1min":0.35799999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":5.4720000000000004,"5min":5.4720000000000004,"15min":5.4720000000000004},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.61299999999999999,"15min":0.59799999999999998},"min":{"1min":0.40100000000000002,"5min":0.39700000000000002,"15min":0.35099999999999998},"max":{"1min":1.0549999999999999,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.61899999999999999}]},{"osd":6,"last update":"Mon Oct 16 13:48:54 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.63200000000000001,"15min":0.622},"min":{"1min":0.46600000000000003,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.69499999999999995,"15min":0.64500000000000002},"min":{"1min":0.38900000000000001,"5min":0.378,"15min":0.35199999999999998},"max":{"1min":5.3860000000000001,"5min":5.3860000000000001,"15min":5.3860000000000001},"last":0.79300000000000004}]}]},{"osd":6,"up_from":13,"seq":55834576015,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48560,"kb_used_data":3488,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323280,"statfs":{"total":96636764160,"available":96587038720,"internally_reserved":0,"allocated":3571712,"data_stored":1856171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.60999999999999999,"15min":0.65000000000000002},"min":{"1min":0.315,"5min":0.315,"15min":0.29599999999999999},"max":{"1min":0.67600000000000005,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.503,"5min":0.60599999999999998,"15min":0.64200000000000002},"min":{"1min":0.373,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.70099999999999996,"5min":0.91400000000000003,"15min":0.93799999999999994},"last":0.64600000000000002}]},{"osd":1,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.54600000000000004,"15min":0.60499999999999998},"min":{"1min":0.316,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.64500000000000002,"5min":0.86899999999999999,"15min":0.89600000000000002},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.58599999999999997,"15min":0.60599999999999998},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.69799999999999995,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.54600000000000004}]},{"osd":2,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.58899999999999997,"15min":0.61799999999999999},"min":{"1min":0.34799999999999998,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.71499999999999997,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.57599999999999996,"15min":0.60599999999999998},"min":{"1min":0.32200000000000001,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.72999999999999998,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.52000000000000002}]},{"osd":3,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.60199999999999998,"15min":0.64600000000000002},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.38200000000000001},"max":{"1min":0.63400000000000001,"5min":0.92400000000000004,"15min":0.95599999999999996},"last":0.503},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.66600000000000004,"15min":0.70199999999999996},"min":{"1min":0.435,"5min":0.435,"15min":0.42299999999999999},"max":{"1min":0.72399999999999998,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.66200000000000003}]},{"osd":4,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.53300000000000003,"15min":0.52800000000000002},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.67700000000000005,"5min":0.85599999999999998,"15min":0.95799999999999996},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.60799999999999998,"15min":0.61499999999999999},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.70799999999999996,"5min":0.98799999999999999,"15min":1.0309999999999999},"last":0.69899999999999995}]},{"osd":5,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.58199999999999996,"15min":0.59099999999999997},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.71099999999999997,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.627,"5min":0.64400000000000002,"15min":0.65700000000000003},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.78700000000000003,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.61799999999999999}]},{"osd":7,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.59099999999999997,"15min":0.59799999999999998},"min":{"1min":0.35199999999999998,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.747,"5min":1.081,"15min":1.081},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.63900000000000001,"15min":0.64900000000000002},"min":{"1min":0.40999999999999998,"5min":0.40899999999999997,"15min":0.379},"max":{"1min":0.76600000000000001,"5min":1.099,"15min":1.099},"last":0.68000000000000005}]}]},{"osd":1,"up_from":13,"seq":55834576015,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52732,"kb_used_data":3504,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319108,"statfs":{"total":96636764160,"available":96582766592,"internally_reserved":0,"allocated":3588096,"data_stored":1863001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":50395593},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:48:59 2023","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.37,"15min":0.40100000000000002},"min":{"1min":0.254,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.58099999999999996,"5min":0.65400000000000003,"15min":0.65800000000000003},"last":0.379},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.433,"15min":0.48099999999999998},"min":{"1min":0.32700000000000001,"5min":0.253,"15min":0.253},"max":{"1min":0.56599999999999995,"5min":0.68000000000000005,"15min":0.77200000000000002},"last":0.40200000000000002}]},{"osd":2,"last update":"Mon Oct 16 13:48:59 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.39400000000000002,"15min":0.439},"min":{"1min":0.251,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.496,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.44500000000000001,"15min":0.47999999999999998},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.58299999999999996,"5min":0.67600000000000005,"15min":0.71399999999999997},"last":0.438}]},{"osd":3,"last update":"Mon Oct 16 13:48:59 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.39000000000000001,"15min":0.42299999999999999},"min":{"1min":0.28499999999999998,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.51300000000000001,"5min":0.65200000000000002,"15min":0.72099999999999997},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.41699999999999998,"15min":0.46600000000000003},"min":{"1min":0.29599999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.60299999999999998,"5min":0.69399999999999995,"15min":0.77300000000000002},"last":0.41699999999999998}]},{"osd":4,"last update":"Mon Oct 16 13:48:59 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.61599999999999999,"15min":0.65700000000000003},"min":{"1min":0.41199999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.78300000000000003,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.59899999999999998,"15min":0.65600000000000003},"min":{"1min":0.377,"5min":0.37,"15min":0.37},"max":{"1min":0.90400000000000003,"5min":0.91900000000000004,"15min":0.997},"last":0.54100000000000004}]},{"osd":5,"last update":"Mon Oct 16 13:48:59 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.64700000000000002,"15min":0.69899999999999995},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":1.0760000000000001},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.65700000000000003,"15min":0.71499999999999997},"min":{"1min":0.436,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.81299999999999994,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.49199999999999999}]},{"osd":6,"last update":"Mon Oct 16 13:48:59 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.621,"15min":0.66900000000000004},"min":{"1min":0.41999999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85799999999999998,"5min":0.95099999999999996,"15min":0.97799999999999998},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.62,"5min":0.64500000000000002,"15min":0.69499999999999995},"min":{"1min":0.46899999999999997,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":1.083},"last":0.66500000000000004}]},{"osd":7,"last update":"Mon Oct 16 13:48:59 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.66300000000000003,"15min":0.72199999999999998},"min":{"1min":0.45300000000000001,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.79300000000000004,"5min":1.163,"15min":1.163},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.68799999999999994,"15min":0.73699999999999999},"min":{"1min":0.47999999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.94699999999999995,"5min":0.98499999999999999,"15min":1.0580000000000001},"last":0.57699999999999996}]}]},{"osd":0,"up_from":13,"seq":55834576016,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53312,"kb_used_data":4084,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318528,"statfs":{"total":96636764160,"available":96582172672,"internally_reserved":0,"allocated":4182016,"data_stored":2453369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.373,"15min":0.41299999999999998},"min":{"1min":0.23799999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.50900000000000001,"5min":0.60699999999999998,"15min":0.71399999999999997},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.38500000000000001,"15min":0.42799999999999999},"min":{"1min":0.24199999999999999,"5min":0.222,"15min":0.222},"max":{"1min":0.56799999999999995,"5min":0.77700000000000002,"15min":0.82199999999999995},"last":0.54100000000000004}]},{"osd":2,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.40999999999999998,"15min":0.441},"min":{"1min":0.27300000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.51000000000000001,"5min":0.65900000000000003,"15min":0.75700000000000001},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.432,"15min":0.47199999999999998},"min":{"1min":0.30399999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.54500000000000004,"5min":0.68799999999999994,"15min":0.83299999999999996},"last":0.41599999999999998}]},{"osd":3,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.40600000000000003,"15min":0.44400000000000001},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.52100000000000002,"5min":0.69699999999999995,"15min":0.78600000000000003},"last":0.502},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.46300000000000002,"15min":0.50600000000000001},"min":{"1min":0.309,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.55900000000000005,"5min":0.70399999999999996,"15min":0.86399999999999999},"last":0.51600000000000001}]},{"osd":4,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.64500000000000002,"15min":0.67600000000000005},"min":{"1min":0.42999999999999999,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.82999999999999996,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.64900000000000002,"15min":0.68100000000000005},"min":{"1min":0.43099999999999999,"5min":0.375,"15min":0.375},"max":{"1min":0.81000000000000005,"5min":1.21,"15min":1.21},"last":0.73999999999999999}]},{"osd":5,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.63400000000000001,"15min":0.67700000000000005},"min":{"1min":0.40300000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.82199999999999995,"5min":1.242,"15min":1.242},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.622,"5min":0.68300000000000005,"15min":0.71799999999999997},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.83299999999999996,"5min":1.081,"15min":1.081},"last":0.80900000000000005}]},{"osd":6,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.67600000000000005,"15min":0.71999999999999997},"min":{"1min":0.43099999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.84699999999999998,"5min":1.034,"15min":1.103},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.67700000000000005,"15min":0.71499999999999997},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.90900000000000003,"5min":1.131,"15min":1.131},"last":0.83399999999999996}]},{"osd":7,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.68700000000000006,"15min":0.73499999999999999},"min":{"1min":0.47399999999999998,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.94799999999999995,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.69599999999999995,"15min":0.72699999999999998},"min":{"1min":0.47899999999999998,"5min":0.433,"15min":0.433},"max":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.85599999999999998}]}]},{"osd":2,"up_from":13,"seq":55834576016,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56880,"kb_used_data":3492,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314960,"statfs":{"total":96636764160,"available":96578519040,"internally_reserved":0,"allocated":3575808,"data_stored":1856290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:48:41 2023","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.374,"15min":0.40400000000000003},"min":{"1min":0.23899999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.439,"5min":0.623,"15min":0.77500000000000002},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.436,"15min":0.46999999999999997},"min":{"1min":0.33700000000000002,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.57399999999999995,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.48699999999999999}]},{"osd":1,"last update":"Mon Oct 16 13:48:41 2023","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.46100000000000002,"15min":0.49199999999999999},"min":{"1min":0.30499999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.69899999999999995,"5min":0.73199999999999998,"15min":0.81399999999999995},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.45000000000000001,"15min":0.46899999999999997},"min":{"1min":0.27900000000000003,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.51700000000000002,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.54500000000000004}]},{"osd":3,"last update":"Mon Oct 16 13:48:41 2023","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.40400000000000003,"15min":0.435},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.55500000000000005,"5min":0.63,"15min":0.72599999999999998},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.499,"15min":0.53700000000000003},"min":{"1min":0.376,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.65900000000000003,"5min":0.82699999999999996,"15min":0.90100000000000002},"last":0.63300000000000001}]},{"osd":4,"last update":"Mon Oct 16 13:48:41 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.66800000000000004,"15min":0.70699999999999996},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.423,"5min":1.423,"15min":1.423},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.67700000000000005,"15min":0.71399999999999997},"min":{"1min":0.437,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.407,"5min":1.407,"15min":1.407},"last":0.81399999999999995}]},{"osd":5,"last update":"Mon Oct 16 13:48:41 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.627,"15min":0.67800000000000005},"min":{"1min":0.436,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.82199999999999995,"5min":0.87,"15min":1.486},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.64900000000000002,"15min":0.68999999999999995},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.78200000000000003,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":0.83299999999999996}]},{"osd":6,"last update":"Mon Oct 16 13:48:41 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.622,"15min":0.65900000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.80300000000000005,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.745},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.66200000000000003,"15min":0.71099999999999997},"min":{"1min":0.49099999999999999,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.85499999999999998,"5min":0.94299999999999995,"15min":1.4379999999999999},"last":0.80000000000000004}]},{"osd":7,"last update":"Mon Oct 16 13:48:41 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.67500000000000004,"15min":0.70899999999999996},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.83499999999999996,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.66800000000000004,"15min":0.72199999999999998},"min":{"1min":0.48799999999999999,"5min":0.436,"15min":0.436},"max":{"1min":0.88200000000000001,"5min":1.0429999999999999,"15min":1.377},"last":0.88500000000000001}]}]},{"osd":3,"up_from":13,"seq":55834576015,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":56896,"kb_used_data":3504,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314944,"statfs":{"total":96636764160,"available":96578502656,"internally_reserved":0,"allocated":3588096,"data_stored":1863001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:48:45 2023","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.375,"15min":0.40400000000000003},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23000000000000001},"max":{"1min":0.48399999999999999,"5min":0.60099999999999998,"15min":0.70499999999999996},"last":0.438},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.38300000000000001,"15min":0.41599999999999998},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.48799999999999999,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.49399999999999999}]},{"osd":1,"last update":"Mon Oct 16 13:48:45 2023","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.38900000000000001,"15min":0.42299999999999999},"min":{"1min":0.27200000000000002,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.497,"5min":0.68200000000000005,"15min":0.76300000000000001},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.438,"15min":0.46999999999999997},"min":{"1min":0.27800000000000002,"5min":0.27700000000000002,"15min":0.253},"max":{"1min":0.60599999999999998,"5min":0.68700000000000006,"15min":0.77800000000000002},"last":0.33400000000000002}]},{"osd":2,"last update":"Mon Oct 16 13:48:45 2023","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.45100000000000001,"15min":0.47999999999999998},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.71399999999999997,"5min":1.167,"15min":1.167},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.46000000000000002,"15min":0.50700000000000001},"min":{"1min":0.32500000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.63200000000000001,"5min":0.73299999999999998,"15min":0.76800000000000002},"last":0.40400000000000003}]},{"osd":4,"last update":"Mon Oct 16 13:48:45 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.61799999999999999,"15min":0.65500000000000003},"min":{"1min":0.40100000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.137},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65400000000000003,"15min":0.68899999999999995},"min":{"1min":0.45500000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.84699999999999998,"5min":1.2110000000000001,"15min":1.2909999999999999},"last":0.79500000000000004}]},{"osd":5,"last update":"Mon Oct 16 13:48:45 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.65200000000000002,"15min":0.69799999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.374},"max":{"1min":1.0309999999999999,"5min":1.0680000000000001,"15min":1.2729999999999999},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.65400000000000003,"15min":0.68999999999999995},"min":{"1min":0.47299999999999998,"5min":0.45600000000000002,"15min":0.432},"max":{"1min":0.871,"5min":1.0549999999999999,"15min":1.173},"last":0.93100000000000005}]},{"osd":6,"last update":"Mon Oct 16 13:48:45 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.69799999999999995,"15min":0.71299999999999997},"min":{"1min":0.432,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.115,"5min":5.0999999999999996,"15min":5.0999999999999996},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.68600000000000005,"15min":0.72499999999999998},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.2290000000000001},"last":0.75}]},{"osd":7,"last update":"Mon Oct 16 13:48:45 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.68100000000000005,"15min":0.72699999999999998},"min":{"1min":0.45700000000000002,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.21,"5min":1.21,"15min":1.2569999999999999},"last":0.88600000000000001},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.68700000000000006,"15min":0.73399999999999999},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.43099999999999999},"max":{"1min":0.90600000000000003,"5min":1.262,"15min":1.262},"last":0.72799999999999998}]}]},{"osd":4,"up_from":13,"seq":55834576014,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52736,"kb_used_data":3504,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319104,"statfs":{"total":96636764160,"available":96582762496,"internally_reserved":0,"allocated":3588096,"data_stored":1863001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:48:42 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.56299999999999994,"15min":0.61299999999999999},"min":{"1min":0.36399999999999999,"5min":0.314,"15min":0.314},"max":{"1min":0.77800000000000002,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.57399999999999995,"15min":0.624},"min":{"1min":0.36599999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.87,"5min":0.87,"15min":0.89500000000000002},"last":0.56899999999999995}]},{"osd":1,"last update":"Mon Oct 16 13:48:42 2023","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.57499999999999996,"15min":0.64100000000000001},"min":{"1min":0.35699999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.79900000000000004,"5min":0.877,"15min":0.877},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.58399999999999996,"15min":0.63},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.80900000000000005,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.39900000000000002}]},{"osd":2,"last update":"Mon Oct 16 13:48:42 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.59199999999999997,"15min":0.64100000000000001},"min":{"1min":0.434,"5min":0.378,"15min":0.378},"max":{"1min":0.751,"5min":0.89900000000000002,"15min":0.90600000000000003},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.62,"15min":0.65300000000000002},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.755,"5min":0.89700000000000002,"15min":0.92600000000000005},"last":0.50900000000000001}]},{"osd":3,"last update":"Mon Oct 16 13:48:42 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.58699999999999997,"15min":0.63300000000000001},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.81399999999999995,"5min":0.86599999999999999,"15min":0.99199999999999999},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.624,"5min":0.64400000000000002,"15min":0.67500000000000004},"min":{"1min":0.48199999999999998,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.93899999999999995},"last":0.629}]},{"osd":5,"last update":"Mon Oct 16 13:48:42 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.55700000000000005,"15min":0.53800000000000003},"min":{"1min":0.34899999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.80400000000000005,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.57799999999999996,"15min":0.57099999999999995},"min":{"1min":0.307,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.72499999999999998,"5min":0.82199999999999995,"15min":0.94899999999999995},"last":0.745}]},{"osd":6,"last update":"Mon Oct 16 13:48:42 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.63200000000000001,"15min":0.60399999999999998},"min":{"1min":0.38600000000000001,"5min":0.36399999999999999,"15min":0.33200000000000002},"max":{"1min":0.82699999999999996,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.70099999999999996,"15min":0.69699999999999995},"min":{"1min":0.41699999999999998,"5min":0.40300000000000002,"15min":0.39400000000000002},"max":{"1min":0.89700000000000002,"5min":1.129,"15min":1.129},"last":0.71899999999999997}]},{"osd":7,"last update":"Mon Oct 16 13:48:42 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.65100000000000002,"15min":0.63700000000000001},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.78700000000000003,"5min":0.97099999999999997,"15min":0.98599999999999999},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.70999999999999996,"15min":0.68899999999999995},"min":{"1min":0.48399999999999999,"5min":0.44,"15min":0.38200000000000001},"max":{"1min":0.96999999999999997,"5min":0.97599999999999998,"15min":1.026},"last":0.76600000000000001}]}]},{"osd":5,"up_from":13,"seq":55834576015,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":74944,"kb_used_data":3504,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94296896,"statfs":{"total":96636764160,"available":96560021504,"internally_reserved":0,"allocated":3588096,"data_stored":1863001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:49:30 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.54100000000000004,"15min":0.59999999999999998},"min":{"1min":0.39000000000000001,"5min":0.314,"15min":0.29899999999999999},"max":{"1min":0.73399999999999999,"5min":0.878,"15min":0.92900000000000005},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.56399999999999995,"15min":0.61699999999999999},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.29699999999999999},"max":{"1min":0.80000000000000004,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.55300000000000005}]},{"osd":1,"last update":"Mon Oct 16 13:49:30 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.58699999999999997,"15min":0.63900000000000001},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.29499999999999998},"max":{"1min":0.72999999999999998,"5min":1,"15min":1},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56799999999999995,"15min":0.62},"min":{"1min":0.45400000000000001,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.78100000000000003,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.629}]},{"osd":2,"last update":"Mon Oct 16 13:49:30 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.58999999999999997,"15min":0.63600000000000001},"min":{"1min":0.38100000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.71199999999999997,"5min":0.91000000000000003,"15min":0.94699999999999995},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.58099999999999996,"15min":0.63700000000000001},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.81699999999999995,"5min":0.83899999999999997,"15min":0.93500000000000005},"last":0.83099999999999996}]},{"osd":3,"last update":"Mon Oct 16 13:49:30 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.61699999999999999,"15min":0.65200000000000002},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.83599999999999997,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.63500000000000001,"15min":0.68000000000000005},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.84799999999999998,"5min":1.4199999999999999,"15min":1.4199999999999999},"last":0.73399999999999999}]},{"osd":4,"last update":"Mon Oct 16 13:49:30 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53600000000000003,"15min":0.54000000000000004},"min":{"1min":0.33300000000000002,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.77200000000000002,"5min":0.89700000000000002,"15min":0.90700000000000003},"last":0.88800000000000001},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.60099999999999998,"15min":0.59499999999999997},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.84799999999999998,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.91900000000000004}]},{"osd":6,"last update":"Mon Oct 16 13:49:30 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.59499999999999997,"15min":0.59099999999999997},"min":{"1min":0.35699999999999998,"5min":0.315,"15min":0.315},"max":{"1min":0.82799999999999996,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.61499999999999999,"15min":0.623},"min":{"1min":0.374,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.82799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.95999999999999996}]},{"osd":7,"last update":"Mon Oct 16 13:49:30 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.65800000000000003,"15min":0.64300000000000002},"min":{"1min":0.374,"5min":0.374,"15min":0.34999999999999998},"max":{"1min":0.878,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.97199999999999998},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.69499999999999995,"15min":0.69099999999999995},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.86899999999999999,"5min":0.97299999999999998,"15min":1.0229999999999999},"last":0.85799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T13:49:36.048 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-16T13:49:36.049 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T13:49:36.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.171+0000 7f67b6587700 1 Processor -- start 2023-10-16T13:49:36.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.171+0000 7f67b6587700 1 -- start start 2023-10-16T13:49:36.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.172+0000 7f67b6587700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f67b015b280 0x7f67b01613e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:36.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.172+0000 7f67b6587700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67b0161c30 con 0x7f67b006a7e0 2023-10-16T13:49:36.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.172+0000 7f67b6587700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67b0161f10 con 0x7f67b00cc8a0 2023-10-16T13:49:36.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.172+0000 7f67b6587700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67b01621f0 con 0x7f67b015b280 2023-10-16T13:49:36.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.172+0000 7f67b4b24700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f67b006a7e0 0x7f67b00d02b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49650/0 (socket says 172.21.15.16:49650) 2023-10-16T13:49:36.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.172+0000 7f67b4b24700 1 -- 172.21.15.16:0/83843580 learned_addr learned my addr 172.21.15.16:0/83843580 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:49:36.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.172+0000 7f67affff700 1 --2- 172.21.15.16:0/83843580 >> v2:172.21.15.49:3300/0 conn(0x7f67b015b280 0x7f67b01613e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:36.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.172+0000 7f67af7fe700 1 -- 172.21.15.16:0/83843580 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3669734532 0 0) 0x7f67b0161c30 con 0x7f67b006a7e0 2023-10-16T13:49:36.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.172+0000 7f67af7fe700 1 -- 172.21.15.16:0/83843580 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f679c0035b0 con 0x7f67b006a7e0 2023-10-16T13:49:36.180 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.173+0000 7f67af7fe700 1 -- 172.21.15.16:0/83843580 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3578837613 0 0) 0x7f679c0035b0 con 0x7f67b006a7e0 2023-10-16T13:49:36.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.173+0000 7f67af7fe700 1 -- 172.21.15.16:0/83843580 >> v2:172.21.15.49:3300/0 conn(0x7f67b015b280 msgr2=0x7f67b01613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:36.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.173+0000 7f67af7fe700 1 --2- 172.21.15.16:0/83843580 >> v2:172.21.15.49:3300/0 conn(0x7f67b015b280 0x7f67b01613e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:36.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.173+0000 7f67af7fe700 1 -- 172.21.15.16:0/83843580 >> v1:172.21.15.49:6789/0 conn(0x7f67b00cc8a0 legacy=0x7f67b015e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:36.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.173+0000 7f67af7fe700 1 -- 172.21.15.16:0/83843580 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67b015df00 con 0x7f67b006a7e0 2023-10-16T13:49:36.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.173+0000 7f67af7fe700 1 -- 172.21.15.16:0/83843580 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7f67a80019f0 con 0x7f67b006a7e0 2023-10-16T13:49:36.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.173+0000 7f67af7fe700 1 -- 172.21.15.16:0/83843580 >> v1:172.21.15.16:6789/0 conn(0x7f67b006a7e0 legacy=0x7f67b00d02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:36.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.173+0000 7f67af7fe700 1 --2- 172.21.15.16:0/83843580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f679c003a00 0x7f679c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:36.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.173+0000 7f67af7fe700 1 --2- 172.21.15.16:0/83843580 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f679c004850 0x7f679c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:36.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.173+0000 7f67af7fe700 1 --2- 172.21.15.16:0/83843580 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f679c005210 0x7f679c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:36.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.173+0000 7f67af7fe700 1 -- 172.21.15.16:0/83843580 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67b01621f0 con 0x7f679c003a00 2023-10-16T13:49:36.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.173+0000 7f67af7fe700 1 -- 172.21.15.16:0/83843580 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f679c00c0b0 con 0x7f679c004850 2023-10-16T13:49:36.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.173+0000 7f67af7fe700 1 -- 172.21.15.16:0/83843580 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f679c00c270 con 0x7f679c005210 2023-10-16T13:49:36.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.173+0000 7f67affff700 1 --2- 172.21.15.16:0/83843580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f679c003a00 0x7f679c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:36.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.173+0000 7f67a7fff700 1 --2- 172.21.15.16:0/83843580 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f679c004850 0x7f679c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:36.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.173+0000 7f67b4b24700 1 --2- 172.21.15.16:0/83843580 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f679c005210 0x7f679c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:36.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.173+0000 7f67affff700 1 -- 172.21.15.16:0/83843580 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f679c005210 msgr2=0x7f679c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:36.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.173+0000 7f67affff700 1 --2- 172.21.15.16:0/83843580 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f679c005210 0x7f679c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:36.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.173+0000 7f67affff700 1 -- 172.21.15.16:0/83843580 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f679c004850 msgr2=0x7f679c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:36.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.173+0000 7f67affff700 1 --2- 172.21.15.16:0/83843580 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f679c004850 0x7f679c004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:36.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.173+0000 7f67affff700 1 -- 172.21.15.16:0/83843580 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f679c0035b0 con 0x7f679c003a00 2023-10-16T13:49:36.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.173+0000 7f67a7fff700 1 --2- 172.21.15.16:0/83843580 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f679c004850 0x7f679c004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:49:36.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.173+0000 7f67b4b24700 1 -- 172.21.15.16:0/83843580 reap_dead start 2023-10-16T13:49:36.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.173+0000 7f67affff700 1 --2- 172.21.15.16:0/83843580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f679c003a00 0x7f679c003e20 secure :-1 s=READY pgs=3330 cs=0 l=1 rev1=1 crypto rx=0x7f67a0009b60 tx=0x7f679c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=886f097641f27599 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:36.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.173+0000 7f67af7fe700 1 -- 172.21.15.16:0/83843580 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67a000e6c0 con 0x7f679c003a00 2023-10-16T13:49:36.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.173+0000 7f67af7fe700 1 -- 172.21.15.16:0/83843580 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67a000e820 con 0x7f679c003a00 2023-10-16T13:49:36.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.173+0000 7f67af7fe700 1 -- 172.21.15.16:0/83843580 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67a000eb90 con 0x7f679c003a00 2023-10-16T13:49:36.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.174+0000 7f67b6587700 1 -- 172.21.15.16:0/83843580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f679c003a00 msgr2=0x7f679c003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:36.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.174+0000 7f67b6587700 1 --2- 172.21.15.16:0/83843580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f679c003a00 0x7f679c003e20 secure :-1 s=READY pgs=3330 cs=0 l=1 rev1=1 crypto rx=0x7f67a0009b60 tx=0x7f679c005880 comp rx=0 tx=0).stop 2023-10-16T13:49:36.188 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.174+0000 7f67b6587700 1 -- 172.21.15.16:0/83843580 shutdown_connections 2023-10-16T13:49:36.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.174+0000 7f67b6587700 1 --2- 172.21.15.16:0/83843580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f679c003a00 0x7f679c003e20 unknown :-1 s=CLOSED pgs=3330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:36.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.174+0000 7f67b6587700 1 -- 172.21.15.16:0/83843580 >> 172.21.15.16:0/83843580 conn(0x7f67b00b1fd0 msgr2=0x7f67b00b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:36.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.174+0000 7f67b6587700 1 -- 172.21.15.16:0/83843580 shutdown_connections 2023-10-16T13:49:36.189 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.174+0000 7f67b6587700 1 -- 172.21.15.16:0/83843580 wait complete. 2023-10-16T13:49:36.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.175+0000 7f67b6587700 1 Processor -- start 2023-10-16T13:49:36.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.175+0000 7f67b6587700 1 -- start start 2023-10-16T13:49:36.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.176+0000 7f67b6587700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f679c003a00 0x7f67b01f3920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:36.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.176+0000 7f67b6587700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f67b00c9f20 0x7f67b00c4b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:36.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.176+0000 7f67b6587700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f67b01577b0 0x7f67b0159250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:36.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.176+0000 7f67b6587700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67b01f3500 con 0x7f679c003a00 2023-10-16T13:49:36.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.176+0000 7f67b6587700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67b015a500 con 0x7f67b00c9f20 2023-10-16T13:49:36.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.176+0000 7f67b6587700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67b00c5070 con 0x7f67b01577b0 2023-10-16T13:49:36.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.176+0000 7f67affff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f679c003a00 0x7f67b01f3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:36.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.176+0000 7f67affff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f679c003a00 0x7f67b01f3920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60384/0 (socket says 172.21.15.16:60384) 2023-10-16T13:49:36.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.176+0000 7f67affff700 1 -- 172.21.15.16:0/2127399933 learned_addr learned my addr 172.21.15.16:0/2127399933 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:36.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.176+0000 7f67b4b24700 1 --2- 172.21.15.16:0/2127399933 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f67b01577b0 0x7f67b0159250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:36.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.176+0000 7f67a7fff700 1 --2- 172.21.15.16:0/2127399933 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f67b00c9f20 0x7f67b00c4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:36.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.177+0000 7f67affff700 1 -- 172.21.15.16:0/2127399933 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f67b01577b0 msgr2=0x7f67b0159250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:36.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.177+0000 7f67affff700 1 --2- 172.21.15.16:0/2127399933 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f67b01577b0 0x7f67b0159250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:36.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.177+0000 7f67affff700 1 -- 172.21.15.16:0/2127399933 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f67b00c9f20 msgr2=0x7f67b00c4b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:36.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.177+0000 7f67affff700 1 --2- 172.21.15.16:0/2127399933 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f67b00c9f20 0x7f67b00c4b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:36.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.177+0000 7f67affff700 1 -- 172.21.15.16:0/2127399933 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67b00c52a0 con 0x7f679c003a00 2023-10-16T13:49:36.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.177+0000 7f67affff700 1 --2- 172.21.15.16:0/2127399933 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f679c003a00 0x7f67b01f3920 secure :-1 s=READY pgs=3331 cs=0 l=1 rev1=1 crypto rx=0x7f67a000e4b0 tx=0x7f67a00047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:36.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.177+0000 7f67ad7fa700 1 -- 172.21.15.16:0/2127399933 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67a000e6c0 con 0x7f679c003a00 2023-10-16T13:49:36.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.177+0000 7f67ad7fa700 1 -- 172.21.15.16:0/2127399933 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67a0003ad0 con 0x7f679c003a00 2023-10-16T13:49:36.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.177+0000 7f67b6587700 1 -- 172.21.15.16:0/2127399933 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67b00c5580 con 0x7f679c003a00 2023-10-16T13:49:36.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.177+0000 7f67ad7fa700 1 -- 172.21.15.16:0/2127399933 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67a0003e80 con 0x7f679c003a00 2023-10-16T13:49:36.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.177+0000 7f67b6587700 1 -- 172.21.15.16:0/2127399933 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67b01f2d80 con 0x7f679c003a00 2023-10-16T13:49:36.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.178+0000 7f67b6587700 1 -- 172.21.15.16:0/2127399933 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67b01f2f70 con 0x7f679c003a00 2023-10-16T13:49:36.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.178+0000 7f67ad7fa700 1 -- 172.21.15.16:0/2127399933 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 176) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f67a0013470 con 0x7f679c003a00 2023-10-16T13:49:36.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.179+0000 7f67ad7fa700 1 --2- 172.21.15.16:0/2127399933 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f679006b020 0x7f679006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:36.199 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.179+0000 7f67ad7fa700 1 -- 172.21.15.16:0/2127399933 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(283..283 src has 1..283) v4 ==== 6208+0+0 (secure 0 0 0) 0x7f67a008b0b0 con 0x7f679c003a00 2023-10-16T13:49:36.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.179+0000 7f67a7fff700 1 --2- 172.21.15.16:0/2127399933 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f679006b020 0x7f679006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:36.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.179+0000 7f67a7fff700 1 --2- 172.21.15.16:0/2127399933 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f679006b020 0x7f679006d4e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6798000c00 tx=0x7f6798009040 comp rx=0 tx=0).ready entity=mgr.13935 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:36.200 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.183+0000 7f67ad7fa700 1 -- 172.21.15.16:0/2127399933 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f67a0056010 con 0x7f679c003a00 2023-10-16T13:49:36.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.383+0000 7f67b6587700 1 -- 172.21.15.16:0/2127399933 --> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f67b015e1e0 con 0x7f679006b020 2023-10-16T13:49:36.387 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.385+0000 7f67ad7fa700 1 -- 172.21.15.16:0/2127399933 <== mgr.13935 v2:172.21.15.16:6832/938897590 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+50916 (secure 0 0 0) 0x7f6798005170 con 0x7f679006b020 2023-10-16T13:49:36.388 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:49:36.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.387+0000 7f67b6587700 1 -- 172.21.15.16:0/2127399933 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f679006b020 msgr2=0x7f679006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:36.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.387+0000 7f67b6587700 1 --2- 172.21.15.16:0/2127399933 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f679006b020 0x7f679006d4e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6798000c00 tx=0x7f6798009040 comp rx=0 tx=0).stop 2023-10-16T13:49:36.390 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.387+0000 7f67b6587700 1 -- 172.21.15.16:0/2127399933 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f679c003a00 msgr2=0x7f67b01f3920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:36.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.387+0000 7f67b6587700 1 --2- 172.21.15.16:0/2127399933 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f679c003a00 0x7f67b01f3920 secure :-1 s=READY pgs=3331 cs=0 l=1 rev1=1 crypto rx=0x7f67a000e4b0 tx=0x7f67a00047c0 comp rx=0 tx=0).stop 2023-10-16T13:49:36.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.387+0000 7f67b6587700 1 -- 172.21.15.16:0/2127399933 shutdown_connections 2023-10-16T13:49:36.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.387+0000 7f67b6587700 1 --2- 172.21.15.16:0/2127399933 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f679006b020 0x7f679006d4e0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:36.391 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.387+0000 7f67b6587700 1 --2- 172.21.15.16:0/2127399933 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f67b01577b0 0x7f67b0159250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:36.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.387+0000 7f67b6587700 1 --2- 172.21.15.16:0/2127399933 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f679c003a00 0x7f67b01f3920 unknown :-1 s=CLOSED pgs=3331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:36.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.387+0000 7f67b6587700 1 --2- 172.21.15.16:0/2127399933 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f67b00c9f20 0x7f67b00c4b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:36.392 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.387+0000 7f67b6587700 1 -- 172.21.15.16:0/2127399933 >> 172.21.15.16:0/2127399933 conn(0x7f67b00b1fd0 msgr2=0x7f67b00cb7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:36.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.387+0000 7f67b6587700 1 -- 172.21.15.16:0/2127399933 shutdown_connections 2023-10-16T13:49:36.393 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.387+0000 7f67b6587700 1 -- 172.21.15.16:0/2127399933 wait complete. 2023-10-16T13:49:36.393 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-10-16T13:49:36.402 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2023-10-16T13:49:35.841899+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2804,"num_read_kb":5868,"num_write":719,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":425,"ondisk_log_size":425,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":155,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":3,"kb":754974720,"kb_used":457676,"kb_used_data":29148,"kb_used_omap":12,"kb_used_meta":428403,"kb_avail":754517044,"statfs":{"total":773094113280,"available":772625453056,"internally_reserved":0,"allocated":29847552,"data_stored":16064373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12740,"internal_metadata":438685244},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-812260,"num_objects":-26,"num_object_clones":0,"num_object_copies":-52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-26,"num_whiteouts":0,"num_read":-312,"num_read_kb":-156,"num_write":-368,"num_write_kb":-3128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.861329"},"pg_stats":[{"pgid":"1.0","version":"281'425","reported_seq":1525,"reported_epoch":281,"state":"active+clean","last_fresh":"2023-10-16T13:49:29.910772+0000","last_change":"2023-10-16T12:10:34.746820+0000","last_active":"2023-10-16T13:49:29.910772+0000","last_peered":"2023-10-16T13:49:29.910772+0000","last_clean":"2023-10-16T13:49:29.910772+0000","last_became_active":"2023-10-16T12:10:34.745574+0000","last_became_peered":"2023-10-16T12:10:34.745574+0000","last_unstale":"2023-10-16T13:49:29.910772+0000","last_undegraded":"2023-10-16T13:49:29.910772+0000","last_fullsized":"2023-10-16T13:49:29.910772+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_clean_scrub_stamp":"2023-10-16T12:10:33.577111+0000","objects_scrubbed":0,"log_size":425,"log_dups_size":0,"ondisk_log_size":425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:14.461986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2804,"num_read_kb":5868,"num_write":719,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2804,"num_read_kb":5868,"num_write":719,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":425,"ondisk_log_size":425,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576017,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61616,"kb_used_data":4068,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310224,"statfs":{"total":96636764160,"available":96573669376,"internally_reserved":0,"allocated":4165632,"data_stored":2446539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58915275},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:48:54 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.58399999999999996,"15min":0.621},"min":{"1min":0.34999999999999998,"5min":0.34200000000000003,"15min":0.33000000000000002},"max":{"1min":4.1040000000000001,"5min":4.1040000000000001,"15min":4.1040000000000001},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.753,"5min":0.60699999999999998,"15min":0.625},"min":{"1min":0.35199999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":5.5279999999999996,"5min":5.5279999999999996,"15min":5.5279999999999996},"last":0.49199999999999999}]},{"osd":1,"last update":"Mon Oct 16 13:48:54 2023","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.63100000000000001,"15min":0.64300000000000002},"min":{"1min":0.33500000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":6.6189999999999998,"5min":6.6189999999999998,"15min":6.6189999999999998},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.59999999999999998,"15min":0.626},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":3.3239999999999998,"5min":3.3239999999999998,"15min":3.3239999999999998},"last":0.63700000000000001}]},{"osd":2,"last update":"Mon Oct 16 13:48:54 2023","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.64500000000000002,"15min":0.67700000000000005},"min":{"1min":0.41499999999999998,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":2.4430000000000001,"5min":2.4430000000000001,"15min":2.4430000000000001},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.67200000000000004,"15min":0.69799999999999995},"min":{"1min":0.45500000000000002,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":4.351,"5min":4.351,"15min":4.351},"last":0.59199999999999997}]},{"osd":3,"last update":"Mon Oct 16 13:48:54 2023","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.63300000000000001,"15min":0.65500000000000003},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":4.4130000000000003,"5min":4.4130000000000003,"15min":4.4130000000000003},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.67600000000000005,"15min":0.67100000000000004},"min":{"1min":0.432,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":5.407,"5min":5.407,"15min":5.407},"last":0.73999999999999999}]},{"osd":4,"last update":"Mon Oct 16 13:48:54 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.56100000000000005,"15min":0.53500000000000003},"min":{"1min":0.37,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":3.2400000000000002,"5min":3.2400000000000002,"15min":3.2400000000000002},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.67900000000000005,"15min":0.63900000000000001},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":5.3369999999999997,"5min":5.3369999999999997,"15min":5.3369999999999997},"last":0.75800000000000001}]},{"osd":5,"last update":"Mon Oct 16 13:48:54 2023","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.621,"15min":0.57199999999999995},"min":{"1min":0.35799999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":5.4720000000000004,"5min":5.4720000000000004,"15min":5.4720000000000004},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.61299999999999999,"15min":0.59799999999999998},"min":{"1min":0.40100000000000002,"5min":0.39700000000000002,"15min":0.35099999999999998},"max":{"1min":1.0549999999999999,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.61899999999999999}]},{"osd":6,"last update":"Mon Oct 16 13:48:54 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.63200000000000001,"15min":0.622},"min":{"1min":0.46600000000000003,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.69499999999999995,"15min":0.64500000000000002},"min":{"1min":0.38900000000000001,"5min":0.378,"15min":0.35199999999999998},"max":{"1min":5.3860000000000001,"5min":5.3860000000000001,"15min":5.3860000000000001},"last":0.79300000000000004}]}]},{"osd":6,"up_from":13,"seq":55834576015,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48560,"kb_used_data":3488,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323280,"statfs":{"total":96636764160,"available":96587038720,"internally_reserved":0,"allocated":3571712,"data_stored":1856171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.60999999999999999,"15min":0.65000000000000002},"min":{"1min":0.315,"5min":0.315,"15min":0.29599999999999999},"max":{"1min":0.67600000000000005,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.503,"5min":0.60599999999999998,"15min":0.64200000000000002},"min":{"1min":0.373,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.70099999999999996,"5min":0.91400000000000003,"15min":0.93799999999999994},"last":0.64600000000000002}]},{"osd":1,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.54600000000000004,"15min":0.60499999999999998},"min":{"1min":0.316,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.64500000000000002,"5min":0.86899999999999999,"15min":0.89600000000000002},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.58599999999999997,"15min":0.60599999999999998},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.69799999999999995,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.54600000000000004}]},{"osd":2,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.58899999999999997,"15min":0.61799999999999999},"min":{"1min":0.34799999999999998,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.71499999999999997,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.57599999999999996,"15min":0.60599999999999998},"min":{"1min":0.32200000000000001,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.72999999999999998,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.52000000000000002}]},{"osd":3,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.60199999999999998,"15min":0.64600000000000002},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.38200000000000001},"max":{"1min":0.63400000000000001,"5min":0.92400000000000004,"15min":0.95599999999999996},"last":0.503},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.66600000000000004,"15min":0.70199999999999996},"min":{"1min":0.435,"5min":0.435,"15min":0.42299999999999999},"max":{"1min":0.72399999999999998,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.66200000000000003}]},{"osd":4,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.53300000000000003,"15min":0.52800000000000002},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.67700000000000005,"5min":0.85599999999999998,"15min":0.95799999999999996},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.60799999999999998,"15min":0.61499999999999999},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.70799999999999996,"5min":0.98799999999999999,"15min":1.0309999999999999},"last":0.69899999999999995}]},{"osd":5,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.58199999999999996,"15min":0.59099999999999997},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.71099999999999997,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.627,"5min":0.64400000000000002,"15min":0.65700000000000003},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.78700000000000003,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.61799999999999999}]},{"osd":7,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.59099999999999997,"15min":0.59799999999999998},"min":{"1min":0.35199999999999998,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.747,"5min":1.081,"15min":1.081},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.63900000000000001,"15min":0.64900000000000002},"min":{"1min":0.40999999999999998,"5min":0.40899999999999997,"15min":0.379},"max":{"1min":0.76600000000000001,"5min":1.099,"15min":1.099},"last":0.68000000000000005}]}]},{"osd":1,"up_from":13,"seq":55834576015,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52732,"kb_used_data":3504,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319108,"statfs":{"total":96636764160,"available":96582766592,"internally_reserved":0,"allocated":3588096,"data_stored":1863001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":50395593},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:48:59 2023","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.37,"15min":0.40100000000000002},"min":{"1min":0.254,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.58099999999999996,"5min":0.65400000000000003,"15min":0.65800000000000003},"last":0.379},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.433,"15min":0.48099999999999998},"min":{"1min":0.32700000000000001,"5min":0.253,"15min":0.253},"max":{"1min":0.56599999999999995,"5min":0.68000000000000005,"15min":0.77200000000000002},"last":0.40200000000000002}]},{"osd":2,"last update":"Mon Oct 16 13:48:59 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.39400000000000002,"15min":0.439},"min":{"1min":0.251,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.496,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.44500000000000001,"15min":0.47999999999999998},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.58299999999999996,"5min":0.67600000000000005,"15min":0.71399999999999997},"last":0.438}]},{"osd":3,"last update":"Mon Oct 16 13:48:59 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.39000000000000001,"15min":0.42299999999999999},"min":{"1min":0.28499999999999998,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.51300000000000001,"5min":0.65200000000000002,"15min":0.72099999999999997},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.41699999999999998,"15min":0.46600000000000003},"min":{"1min":0.29599999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.60299999999999998,"5min":0.69399999999999995,"15min":0.77300000000000002},"last":0.41699999999999998}]},{"osd":4,"last update":"Mon Oct 16 13:48:59 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.61599999999999999,"15min":0.65700000000000003},"min":{"1min":0.41199999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.78300000000000003,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.59899999999999998,"15min":0.65600000000000003},"min":{"1min":0.377,"5min":0.37,"15min":0.37},"max":{"1min":0.90400000000000003,"5min":0.91900000000000004,"15min":0.997},"last":0.54100000000000004}]},{"osd":5,"last update":"Mon Oct 16 13:48:59 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.64700000000000002,"15min":0.69899999999999995},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":1.0760000000000001},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.65700000000000003,"15min":0.71499999999999997},"min":{"1min":0.436,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.81299999999999994,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.49199999999999999}]},{"osd":6,"last update":"Mon Oct 16 13:48:59 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.621,"15min":0.66900000000000004},"min":{"1min":0.41999999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85799999999999998,"5min":0.95099999999999996,"15min":0.97799999999999998},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.62,"5min":0.64500000000000002,"15min":0.69499999999999995},"min":{"1min":0.46899999999999997,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":1.083},"last":0.66500000000000004}]},{"osd":7,"last update":"Mon Oct 16 13:48:59 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.66300000000000003,"15min":0.72199999999999998},"min":{"1min":0.45300000000000001,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.79300000000000004,"5min":1.163,"15min":1.163},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.68799999999999994,"15min":0.73699999999999999},"min":{"1min":0.47999999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.94699999999999995,"5min":0.98499999999999999,"15min":1.0580000000000001},"last":0.57699999999999996}]}]},{"osd":0,"up_from":13,"seq":55834576016,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53312,"kb_used_data":4084,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318528,"statfs":{"total":96636764160,"available":96582172672,"internally_reserved":0,"allocated":4182016,"data_stored":2453369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.373,"15min":0.41299999999999998},"min":{"1min":0.23799999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.50900000000000001,"5min":0.60699999999999998,"15min":0.71399999999999997},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.38500000000000001,"15min":0.42799999999999999},"min":{"1min":0.24199999999999999,"5min":0.222,"15min":0.222},"max":{"1min":0.56799999999999995,"5min":0.77700000000000002,"15min":0.82199999999999995},"last":0.54100000000000004}]},{"osd":2,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.40999999999999998,"15min":0.441},"min":{"1min":0.27300000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.51000000000000001,"5min":0.65900000000000003,"15min":0.75700000000000001},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.432,"15min":0.47199999999999998},"min":{"1min":0.30399999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.54500000000000004,"5min":0.68799999999999994,"15min":0.83299999999999996},"last":0.41599999999999998}]},{"osd":3,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.40600000000000003,"15min":0.44400000000000001},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.52100000000000002,"5min":0.69699999999999995,"15min":0.78600000000000003},"last":0.502},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.46300000000000002,"15min":0.50600000000000001},"min":{"1min":0.309,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.55900000000000005,"5min":0.70399999999999996,"15min":0.86399999999999999},"last":0.51600000000000001}]},{"osd":4,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.64500000000000002,"15min":0.67600000000000005},"min":{"1min":0.42999999999999999,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.82999999999999996,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.64900000000000002,"15min":0.68100000000000005},"min":{"1min":0.43099999999999999,"5min":0.375,"15min":0.375},"max":{"1min":0.81000000000000005,"5min":1.21,"15min":1.21},"last":0.73999999999999999}]},{"osd":5,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.63400000000000001,"15min":0.67700000000000005},"min":{"1min":0.40300000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.82199999999999995,"5min":1.242,"15min":1.242},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.622,"5min":0.68300000000000005,"15min":0.71799999999999997},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.83299999999999996,"5min":1.081,"15min":1.081},"last":0.80900000000000005}]},{"osd":6,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.67600000000000005,"15min":0.71999999999999997},"min":{"1min":0.43099999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.84699999999999998,"5min":1.034,"15min":1.103},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.67700000000000005,"15min":0.71499999999999997},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.90900000000000003,"5min":1.131,"15min":1.131},"last":0.83399999999999996}]},{"osd":7,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.68700000000000006,"15min":0.73499999999999999},"min":{"1min":0.47399999999999998,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.94799999999999995,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.69599999999999995,"15min":0.72699999999999998},"min":{"1min":0.47899999999999998,"5min":0.433,"15min":0.433},"max":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.85599999999999998}]}]},{"osd":2,"up_from":13,"seq":55834576016,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56880,"kb_used_data":3492,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314960,"statfs":{"total":96636764160,"available":96578519040,"internally_reserved":0,"allocated":3575808,"data_stored":1856290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:48:41 2023","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.374,"15min":0.40400000000000003},"min":{"1min":0.23899999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.439,"5min":0.623,"15min":0.77500000000000002},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.436,"15min":0.46999999999999997},"min":{"1min":0.33700000000000002,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.57399999999999995,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.48699999999999999}]},{"osd":1,"last update":"Mon Oct 16 13:48:41 2023","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.46100000000000002,"15min":0.49199999999999999},"min":{"1min":0.30499999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.69899999999999995,"5min":0.73199999999999998,"15min":0.81399999999999995},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.45000000000000001,"15min":0.46899999999999997},"min":{"1min":0.27900000000000003,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.51700000000000002,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.54500000000000004}]},{"osd":3,"last update":"Mon Oct 16 13:48:41 2023","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.40400000000000003,"15min":0.435},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.55500000000000005,"5min":0.63,"15min":0.72599999999999998},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.499,"15min":0.53700000000000003},"min":{"1min":0.376,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.65900000000000003,"5min":0.82699999999999996,"15min":0.90100000000000002},"last":0.63300000000000001}]},{"osd":4,"last update":"Mon Oct 16 13:48:41 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.66800000000000004,"15min":0.70699999999999996},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.423,"5min":1.423,"15min":1.423},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.67700000000000005,"15min":0.71399999999999997},"min":{"1min":0.437,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.407,"5min":1.407,"15min":1.407},"last":0.81399999999999995}]},{"osd":5,"last update":"Mon Oct 16 13:48:41 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.627,"15min":0.67800000000000005},"min":{"1min":0.436,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.82199999999999995,"5min":0.87,"15min":1.486},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.64900000000000002,"15min":0.68999999999999995},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.78200000000000003,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":0.83299999999999996}]},{"osd":6,"last update":"Mon Oct 16 13:48:41 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.622,"15min":0.65900000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.80300000000000005,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.745},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.66200000000000003,"15min":0.71099999999999997},"min":{"1min":0.49099999999999999,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.85499999999999998,"5min":0.94299999999999995,"15min":1.4379999999999999},"last":0.80000000000000004}]},{"osd":7,"last update":"Mon Oct 16 13:48:41 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.67500000000000004,"15min":0.70899999999999996},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.83499999999999996,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.66800000000000004,"15min":0.72199999999999998},"min":{"1min":0.48799999999999999,"5min":0.436,"15min":0.436},"max":{"1min":0.88200000000000001,"5min":1.0429999999999999,"15min":1.377},"last":0.88500000000000001}]}]},{"osd":3,"up_from":13,"seq":55834576015,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":56896,"kb_used_data":3504,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314944,"statfs":{"total":96636764160,"available":96578502656,"internally_reserved":0,"allocated":3588096,"data_stored":1863001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:48:45 2023","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.375,"15min":0.40400000000000003},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23000000000000001},"max":{"1min":0.48399999999999999,"5min":0.60099999999999998,"15min":0.70499999999999996},"last":0.438},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.38300000000000001,"15min":0.41599999999999998},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.48799999999999999,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.49399999999999999}]},{"osd":1,"last update":"Mon Oct 16 13:48:45 2023","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.38900000000000001,"15min":0.42299999999999999},"min":{"1min":0.27200000000000002,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.497,"5min":0.68200000000000005,"15min":0.76300000000000001},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.438,"15min":0.46999999999999997},"min":{"1min":0.27800000000000002,"5min":0.27700000000000002,"15min":0.253},"max":{"1min":0.60599999999999998,"5min":0.68700000000000006,"15min":0.77800000000000002},"last":0.33400000000000002}]},{"osd":2,"last update":"Mon Oct 16 13:48:45 2023","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.45100000000000001,"15min":0.47999999999999998},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.71399999999999997,"5min":1.167,"15min":1.167},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.46000000000000002,"15min":0.50700000000000001},"min":{"1min":0.32500000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.63200000000000001,"5min":0.73299999999999998,"15min":0.76800000000000002},"last":0.40400000000000003}]},{"osd":4,"last update":"Mon Oct 16 13:48:45 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.61799999999999999,"15min":0.65500000000000003},"min":{"1min":0.40100000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.137},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65400000000000003,"15min":0.68899999999999995},"min":{"1min":0.45500000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.84699999999999998,"5min":1.2110000000000001,"15min":1.2909999999999999},"last":0.79500000000000004}]},{"osd":5,"last update":"Mon Oct 16 13:48:45 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.65200000000000002,"15min":0.69799999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.374},"max":{"1min":1.0309999999999999,"5min":1.0680000000000001,"15min":1.2729999999999999},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.65400000000000003,"15min":0.68999999999999995},"min":{"1min":0.47299999999999998,"5min":0.45600000000000002,"15min":0.432},"max":{"1min":0.871,"5min":1.0549999999999999,"15min":1.173},"last":0.93100000000000005}]},{"osd":6,"last update":"Mon Oct 16 13:48:45 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.69799999999999995,"15min":0.71299999999999997},"min":{"1min":0.432,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.115,"5min":5.0999999999999996,"15min":5.0999999999999996},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.68600000000000005,"15min":0.72499999999999998},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.2290000000000001},"last":0.75}]},{"osd":7,"last update":"Mon Oct 16 13:48:45 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.68100000000000005,"15min":0.72699999999999998},"min":{"1min":0.45700000000000002,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.21,"5min":1.21,"15min":1.2569999999999999},"last":0.88600000000000001},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.68700000000000006,"15min":0.73399999999999999},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.43099999999999999},"max":{"1min":0.90600000000000003,"5min":1.262,"15min":1.262},"last":0.72799999999999998}]}]},{"osd":4,"up_from":13,"seq":55834576014,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52736,"kb_used_data":3504,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319104,"statfs":{"total":96636764160,"available":96582762496,"internally_reserved":0,"allocated":3588096,"data_stored":1863001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:48:42 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.56299999999999994,"15min":0.61299999999999999},"min":{"1min":0.36399999999999999,"5min":0.314,"15min":0.314},"max":{"1min":0.77800000000000002,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.57399999999999995,"15min":0.624},"min":{"1min":0.36599999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.87,"5min":0.87,"15min":0.89500000000000002},"last":0.56899999999999995}]},{"osd":1,"last update":"Mon Oct 16 13:48:42 2023","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.57499999999999996,"15min":0.64100000000000001},"min":{"1min":0.35699999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.79900000000000004,"5min":0.877,"15min":0.877},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.58399999999999996,"15min":0.63},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.80900000000000005,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.39900000000000002}]},{"osd":2,"last update":"Mon Oct 16 13:48:42 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.59199999999999997,"15min":0.64100000000000001},"min":{"1min":0.434,"5min":0.378,"15min":0.378},"max":{"1min":0.751,"5min":0.89900000000000002,"15min":0.90600000000000003},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.62,"15min":0.65300000000000002},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.755,"5min":0.89700000000000002,"15min":0.92600000000000005},"last":0.50900000000000001}]},{"osd":3,"last update":"Mon Oct 16 13:48:42 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.58699999999999997,"15min":0.63300000000000001},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.81399999999999995,"5min":0.86599999999999999,"15min":0.99199999999999999},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.624,"5min":0.64400000000000002,"15min":0.67500000000000004},"min":{"1min":0.48199999999999998,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.93899999999999995},"last":0.629}]},{"osd":5,"last update":"Mon Oct 16 13:48:42 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.55700000000000005,"15min":0.53800000000000003},"min":{"1min":0.34899999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.80400000000000005,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.57799999999999996,"15min":0.57099999999999995},"min":{"1min":0.307,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.72499999999999998,"5min":0.82199999999999995,"15min":0.94899999999999995},"last":0.745}]},{"osd":6,"last update":"Mon Oct 16 13:48:42 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.63200000000000001,"15min":0.60399999999999998},"min":{"1min":0.38600000000000001,"5min":0.36399999999999999,"15min":0.33200000000000002},"max":{"1min":0.82699999999999996,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.70099999999999996,"15min":0.69699999999999995},"min":{"1min":0.41699999999999998,"5min":0.40300000000000002,"15min":0.39400000000000002},"max":{"1min":0.89700000000000002,"5min":1.129,"15min":1.129},"last":0.71899999999999997}]},{"osd":7,"last update":"Mon Oct 16 13:48:42 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.65100000000000002,"15min":0.63700000000000001},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.78700000000000003,"5min":0.97099999999999997,"15min":0.98599999999999999},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.70999999999999996,"15min":0.68899999999999995},"min":{"1min":0.48399999999999999,"5min":0.44,"15min":0.38200000000000001},"max":{"1min":0.96999999999999997,"5min":0.97599999999999998,"15min":1.026},"last":0.76600000000000001}]}]},{"osd":5,"up_from":13,"seq":55834576015,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":74944,"kb_used_data":3504,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94296896,"statfs":{"total":96636764160,"available":96560021504,"internally_reserved":0,"allocated":3588096,"data_stored":1863001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:49:30 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.54100000000000004,"15min":0.59999999999999998},"min":{"1min":0.39000000000000001,"5min":0.314,"15min":0.29899999999999999},"max":{"1min":0.73399999999999999,"5min":0.878,"15min":0.92900000000000005},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.56399999999999995,"15min":0.61699999999999999},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.29699999999999999},"max":{"1min":0.80000000000000004,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.55300000000000005}]},{"osd":1,"last update":"Mon Oct 16 13:49:30 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.58699999999999997,"15min":0.63900000000000001},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.29499999999999998},"max":{"1min":0.72999999999999998,"5min":1,"15min":1},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56799999999999995,"15min":0.62},"min":{"1min":0.45400000000000001,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.78100000000000003,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.629}]},{"osd":2,"last update":"Mon Oct 16 13:49:30 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.58999999999999997,"15min":0.63600000000000001},"min":{"1min":0.38100000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.71199999999999997,"5min":0.91000000000000003,"15min":0.94699999999999995},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.58099999999999996,"15min":0.63700000000000001},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.81699999999999995,"5min":0.83899999999999997,"15min":0.93500000000000005},"last":0.83099999999999996}]},{"osd":3,"last update":"Mon Oct 16 13:49:30 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.61699999999999999,"15min":0.65200000000000002},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.83599999999999997,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.63500000000000001,"15min":0.68000000000000005},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.84799999999999998,"5min":1.4199999999999999,"15min":1.4199999999999999},"last":0.73399999999999999}]},{"osd":4,"last update":"Mon Oct 16 13:49:30 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53600000000000003,"15min":0.54000000000000004},"min":{"1min":0.33300000000000002,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.77200000000000002,"5min":0.89700000000000002,"15min":0.90700000000000003},"last":0.88800000000000001},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.60099999999999998,"15min":0.59499999999999997},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.84799999999999998,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.91900000000000004}]},{"osd":6,"last update":"Mon Oct 16 13:49:30 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.59499999999999997,"15min":0.59099999999999997},"min":{"1min":0.35699999999999998,"5min":0.315,"15min":0.315},"max":{"1min":0.82799999999999996,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.61499999999999999,"15min":0.623},"min":{"1min":0.374,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.82799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.95999999999999996}]},{"osd":7,"last update":"Mon Oct 16 13:49:30 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.65800000000000003,"15min":0.64300000000000002},"min":{"1min":0.374,"5min":0.374,"15min":0.34999999999999998},"max":{"1min":0.878,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.97199999999999998},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.69499999999999995,"15min":0.69099999999999995},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.86899999999999999,"5min":0.97299999999999998,"15min":1.0229999999999999},"last":0.85799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T13:49:36.404 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T13:49:36.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.526+0000 7f919e2a0700 1 Processor -- start 2023-10-16T13:49:36.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.526+0000 7f919e2a0700 1 -- start start 2023-10-16T13:49:36.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.527+0000 7f919e2a0700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f91980cf4a0 0x7f91980726e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:36.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.527+0000 7f919e2a0700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f91980cf880 0x7f91980b52f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:36.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.527+0000 7f919e2a0700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f91980b5830 0x7f919815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:36.532 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.527+0000 7f919e2a0700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f919815bc80 con 0x7f91980b5830 2023-10-16T13:49:36.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.527+0000 7f919e2a0700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9198072c20 con 0x7f91980cf880 2023-10-16T13:49:36.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.527+0000 7f919e2a0700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f919815b070 con 0x7f91980cf4a0 2023-10-16T13:49:36.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.528+0000 7f919c83d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f91980b5830 0x7f919815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:36.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.528+0000 7f919c83d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f91980b5830 0x7f919815a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60396/0 (socket says 172.21.15.16:60396) 2023-10-16T13:49:36.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.528+0000 7f919c83d700 1 -- 172.21.15.16:0/390608722 learned_addr learned my addr 172.21.15.16:0/390608722 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:36.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.528+0000 7f9197fff700 1 --2- 172.21.15.16:0/390608722 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f91980cf4a0 0x7f91980726e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:36.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.528+0000 7f91977fe700 1 --2- 172.21.15.16:0/390608722 >> v2:172.21.15.49:3300/0 conn(0x7f91980cf880 0x7f91980b52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:36.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.528+0000 7f919c83d700 1 -- 172.21.15.16:0/390608722 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f91980cf4a0 msgr2=0x7f91980726e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:36.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.528+0000 7f919c83d700 1 --2- 172.21.15.16:0/390608722 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f91980cf4a0 0x7f91980726e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:36.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.528+0000 7f919c83d700 1 -- 172.21.15.16:0/390608722 >> v2:172.21.15.49:3300/0 conn(0x7f91980cf880 msgr2=0x7f91980b52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:36.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.528+0000 7f919c83d700 1 --2- 172.21.15.16:0/390608722 >> v2:172.21.15.49:3300/0 conn(0x7f91980cf880 0x7f91980b52f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:36.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.528+0000 7f919c83d700 1 -- 172.21.15.16:0/390608722 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f919815b350 con 0x7f91980b5830 2023-10-16T13:49:36.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.528+0000 7f91977fe700 1 --2- 172.21.15.16:0/390608722 >> v2:172.21.15.49:3300/0 conn(0x7f91980cf880 0x7f91980b52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:49:36.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.528+0000 7f919c83d700 1 --2- 172.21.15.16:0/390608722 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91980b5830 0x7f919815a090 secure :-1 s=READY pgs=3332 cs=0 l=1 rev1=1 crypto rx=0x7f919806ead0 tx=0x7f919000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c341a25e8712a397 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:36.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.528+0000 7f9196ffd700 1 -- 172.21.15.16:0/390608722 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f919000e070 con 0x7f91980b5830 2023-10-16T13:49:36.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.528+0000 7f9196ffd700 1 -- 172.21.15.16:0/390608722 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f919000b9c0 con 0x7f91980b5830 2023-10-16T13:49:36.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.528+0000 7f9196ffd700 1 -- 172.21.15.16:0/390608722 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f919000bd70 con 0x7f91980b5830 2023-10-16T13:49:36.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.529+0000 7f919e2a0700 1 -- 172.21.15.16:0/390608722 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91980b5830 msgr2=0x7f919815a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:36.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.529+0000 7f919e2a0700 1 --2- 172.21.15.16:0/390608722 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91980b5830 0x7f919815a090 secure :-1 s=READY pgs=3332 cs=0 l=1 rev1=1 crypto rx=0x7f919806ead0 tx=0x7f919000b330 comp rx=0 tx=0).stop 2023-10-16T13:49:36.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.529+0000 7f919e2a0700 1 -- 172.21.15.16:0/390608722 shutdown_connections 2023-10-16T13:49:36.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.529+0000 7f919e2a0700 1 --2- 172.21.15.16:0/390608722 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91980b5830 0x7f919815a090 unknown :-1 s=CLOSED pgs=3332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:36.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.529+0000 7f919e2a0700 1 --2- 172.21.15.16:0/390608722 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f91980cf4a0 0x7f91980726e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:36.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.529+0000 7f919e2a0700 1 --2- 172.21.15.16:0/390608722 >> v2:172.21.15.49:3300/0 conn(0x7f91980cf880 0x7f91980b52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:36.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.529+0000 7f919e2a0700 1 -- 172.21.15.16:0/390608722 >> 172.21.15.16:0/390608722 conn(0x7f91980b1fd0 msgr2=0x7f91980b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:36.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.529+0000 7f919e2a0700 1 -- 172.21.15.16:0/390608722 shutdown_connections 2023-10-16T13:49:36.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.529+0000 7f919e2a0700 1 -- 172.21.15.16:0/390608722 wait complete. 2023-10-16T13:49:36.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.530+0000 7f919e2a0700 1 Processor -- start 2023-10-16T13:49:36.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.531+0000 7f919e2a0700 1 -- start start 2023-10-16T13:49:36.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.532+0000 7f919e2a0700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f91980b5830 0x7f91980c66c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:36.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.532+0000 7f919e2a0700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f91980cf4a0 0x7f91980ce320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:36.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.532+0000 7f919e2a0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91980cf880 0x7f91980ce860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:36.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.532+0000 7f919e2a0700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91980c6c00 con 0x7f91980cf880 2023-10-16T13:49:36.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.532+0000 7f919e2a0700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9198158220 con 0x7f91980cf4a0 2023-10-16T13:49:36.544 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.532+0000 7f919e2a0700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91980cc750 con 0x7f91980b5830 2023-10-16T13:49:36.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.532+0000 7f919c83d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91980cf880 0x7f91980ce860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:36.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.532+0000 7f919c83d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91980cf880 0x7f91980ce860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60402/0 (socket says 172.21.15.16:60402) 2023-10-16T13:49:36.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.532+0000 7f919c83d700 1 -- 172.21.15.16:0/3565951701 learned_addr learned my addr 172.21.15.16:0/3565951701 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:36.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.532+0000 7f91977fe700 1 --2- 172.21.15.16:0/3565951701 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f91980cf4a0 0x7f91980ce320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:36.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.532+0000 7f9197fff700 1 --2- 172.21.15.16:0/3565951701 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f91980b5830 0x7f91980c66c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:36.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.532+0000 7f919c83d700 1 -- 172.21.15.16:0/3565951701 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f91980b5830 msgr2=0x7f91980c66c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:36.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.532+0000 7f919c83d700 1 --2- 172.21.15.16:0/3565951701 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f91980b5830 0x7f91980c66c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:36.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.532+0000 7f919c83d700 1 -- 172.21.15.16:0/3565951701 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f91980cf4a0 msgr2=0x7f91980ce320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:36.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.532+0000 7f919c83d700 1 --2- 172.21.15.16:0/3565951701 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f91980cf4a0 0x7f91980ce320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:36.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.532+0000 7f919c83d700 1 -- 172.21.15.16:0/3565951701 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9190009b30 con 0x7f91980cf880 2023-10-16T13:49:36.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.532+0000 7f9197fff700 1 --2- 172.21.15.16:0/3565951701 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f91980b5830 0x7f91980c66c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:49:36.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.532+0000 7f919c83d700 1 --2- 172.21.15.16:0/3565951701 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91980cf880 0x7f91980ce860 secure :-1 s=READY pgs=3333 cs=0 l=1 rev1=1 crypto rx=0x7f919000b730 tx=0x7f9190004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:36.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.532+0000 7f9194ff9700 1 -- 172.21.15.16:0/3565951701 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9190004250 con 0x7f91980cf880 2023-10-16T13:49:36.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.532+0000 7f9194ff9700 1 -- 172.21.15.16:0/3565951701 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91900043b0 con 0x7f91980cf880 2023-10-16T13:49:36.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.532+0000 7f919e2a0700 1 -- 172.21.15.16:0/3565951701 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91980cca30 con 0x7f91980cf880 2023-10-16T13:49:36.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.532+0000 7f9194ff9700 1 -- 172.21.15.16:0/3565951701 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9190004760 con 0x7f91980cf880 2023-10-16T13:49:36.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.532+0000 7f919e2a0700 1 -- 172.21.15.16:0/3565951701 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9198158750 con 0x7f91980cf880 2023-10-16T13:49:36.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.533+0000 7f919e2a0700 1 -- 172.21.15.16:0/3565951701 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f919804ebb0 con 0x7f91980cf880 2023-10-16T13:49:36.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.534+0000 7f9194ff9700 1 -- 172.21.15.16:0/3565951701 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 176) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f919001c440 con 0x7f91980cf880 2023-10-16T13:49:36.549 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.534+0000 7f9194ff9700 1 --2- 172.21.15.16:0/3565951701 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f917806b020 0x7f917806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:36.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.534+0000 7f9194ff9700 1 -- 172.21.15.16:0/3565951701 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(283..283 src has 1..283) v4 ==== 6208+0+0 (secure 0 0 0) 0x7f919008b2a0 con 0x7f91980cf880 2023-10-16T13:49:36.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.534+0000 7f9197fff700 1 --2- 172.21.15.16:0/3565951701 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f917806b020 0x7f917806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:36.550 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.535+0000 7f9197fff700 1 --2- 172.21.15.16:0/3565951701 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f917806b020 0x7f917806d4e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f91880099f0 tx=0x7f9188008040 comp rx=0 tx=0).ready entity=mgr.13935 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:36.551 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.539+0000 7f9194ff9700 1 -- 172.21.15.16:0/3565951701 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9190055f50 con 0x7f91980cf880 2023-10-16T13:49:36.744 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.742+0000 7f919e2a0700 1 -- 172.21.15.16:0/3565951701 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f91980c36b0 con 0x7f91980cf880 2023-10-16T13:49:36.745 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.743+0000 7f9194ff9700 1 -- 172.21.15.16:0/3565951701 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v283) v1 ==== 74+0+14485 (secure 0 0 0) 0x7f9190078070 con 0x7f91980cf880 2023-10-16T13:49:36.745 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:49:36.746 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":283,"fsid":"5b28dfa1-b21f-474a-9b2e-9fb77af58254","created":"2023-10-16T12:10:21.275218+0000","modified":"2023-10-16T13:49:32.239165+0000","last_up_change":"2023-10-16T12:10:31.570642+0000","last_in_change":"2023-10-16T12:10:23.791850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":38,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T12:10:33.148942+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3d176986-e18f-4c8c-b318-ce9319b69906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6801","nonce":2366702185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6803","nonce":2366702185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6807","nonce":2366702185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":2366702185},{"type":"v1","addr":"172.21.15.16:6805","nonce":2366702185}]},"public_addr":"172.21.15.16:6801/2366702185","cluster_addr":"172.21.15.16:6803/2366702185","heartbeat_back_addr":"172.21.15.16:6807/2366702185","heartbeat_front_addr":"172.21.15.16:6805/2366702185","state":["exists","up"]},{"osd":1,"uuid":"93e01cb9-fe53-4210-a128-3cdf71d0420e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6818","nonce":78500784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6822","nonce":78500784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6830","nonce":78500784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":78500784},{"type":"v1","addr":"172.21.15.16:6826","nonce":78500784}]},"public_addr":"172.21.15.16:6818/78500784","cluster_addr":"172.21.15.16:6822/78500784","heartbeat_back_addr":"172.21.15.16:6830/78500784","heartbeat_front_addr":"172.21.15.16:6826/78500784","state":["exists","up"]},{"osd":2,"uuid":"ff4dc68c-c44f-480d-bc8c-2264bf034517","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6809","nonce":47933305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6811","nonce":47933305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6815","nonce":47933305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":47933305},{"type":"v1","addr":"172.21.15.16:6813","nonce":47933305}]},"public_addr":"172.21.15.16:6809/47933305","cluster_addr":"172.21.15.16:6811/47933305","heartbeat_back_addr":"172.21.15.16:6815/47933305","heartbeat_front_addr":"172.21.15.16:6813/47933305","state":["exists","up"]},{"osd":3,"uuid":"33c469f5-2ffc-466e-966f-a411ca1e0e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6819","nonce":375531758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6821","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6823","nonce":375531758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6829","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6831","nonce":375531758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6825","nonce":375531758},{"type":"v1","addr":"172.21.15.16:6827","nonce":375531758}]},"public_addr":"172.21.15.16:6819/375531758","cluster_addr":"172.21.15.16:6823/375531758","heartbeat_back_addr":"172.21.15.16:6831/375531758","heartbeat_front_addr":"172.21.15.16:6827/375531758","state":["exists","up"]},{"osd":4,"uuid":"500e63ec-cf34-409a-84c1-bbcf7d17606e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6801","nonce":682437317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6803","nonce":682437317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6807","nonce":682437317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":682437317},{"type":"v1","addr":"172.21.15.49:6805","nonce":682437317}]},"public_addr":"172.21.15.49:6801/682437317","cluster_addr":"172.21.15.49:6803/682437317","heartbeat_back_addr":"172.21.15.49:6807/682437317","heartbeat_front_addr":"172.21.15.49:6805/682437317","state":["exists","up"]},{"osd":5,"uuid":"841bc611-c518-4610-b086-049bad607070","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6817","nonce":3830572508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6819","nonce":3830572508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6823","nonce":3830572508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":3830572508},{"type":"v1","addr":"172.21.15.49:6821","nonce":3830572508}]},"public_addr":"172.21.15.49:6817/3830572508","cluster_addr":"172.21.15.49:6819/3830572508","heartbeat_back_addr":"172.21.15.49:6823/3830572508","heartbeat_front_addr":"172.21.15.49:6821/3830572508","state":["exists","up"]},{"osd":6,"uuid":"0dc6a215-f320-45c5-bb08-176400b05e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6809","nonce":3582923927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6811","nonce":3582923927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6815","nonce":3582923927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":3582923927},{"type":"v1","addr":"172.21.15.49:6813","nonce":3582923927}]},"public_addr":"172.21.15.49:6809/3582923927","cluster_addr":"172.21.15.49:6811/3582923927","heartbeat_back_addr":"172.21.15.49:6815/3582923927","heartbeat_front_addr":"172.21.15.49:6813/3582923927","state":["exists","up"]},{"osd":7,"uuid":"69af17a2-6ebd-486c-9f55-00a38839e155","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6825","nonce":2356409928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6827","nonce":2356409928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6831","nonce":2356409928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":2356409928},{"type":"v1","addr":"172.21.15.49:6829","nonce":2356409928}]},"public_addr":"172.21.15.49:6825/2356409928","cluster_addr":"172.21.15.49:6827/2356409928","heartbeat_back_addr":"172.21.15.49:6831/2356409928","heartbeat_front_addr":"172.21.15.49:6829/2356409928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.748725+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.014817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.971981+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:29.005037+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.892251+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.979023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.962626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T12:10:28.984280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.49:6833/2082691635":"2023-10-17T13:49:28.978211+0000","172.21.15.16:0/3139225757":"2023-10-17T13:49:27.824706+0000","172.21.15.16:0/2356827013":"2023-10-17T13:49:27.824706+0000","172.21.15.16:0/667060049":"2023-10-17T13:49:27.824706+0000","172.21.15.49:6835/2082691635":"2023-10-17T13:49:28.978211+0000","172.21.15.16:0/2056580162":"2023-10-17T13:49:27.824706+0000","172.21.15.16:0/1209578814":"2023-10-17T13:49:27.824706+0000","172.21.15.16:0/4129379262":"2023-10-17T13:49:27.824706+0000","172.21.15.16:6832/3920835901":"2023-10-17T13:49:27.824706+0000","172.21.15.16:0/3384986395":"2023-10-17T13:49:27.824706+0000","172.21.15.16:0/2216636694":"2023-10-17T13:49:27.824706+0000","172.21.15.16:0/750285561":"2023-10-17T13:49:17.194509+0000","172.21.15.16:6833/1257392887":"2023-10-17T13:49:17.194509+0000","172.21.15.16:6832/1257392887":"2023-10-17T13:49:17.194509+0000","172.21.15.16:0/1101859511":"2023-10-17T13:49:17.194509+0000","172.21.15.16:0/3516507548":"2023-10-17T13:49:27.824706+0000","172.21.15.16:6833/3920835901":"2023-10-17T13:49:27.824706+0000","172.21.15.16:0/197265557":"2023-10-17T13:49:17.194509+0000","172.21.15.16:0/2024751970":"2023-10-17T13:49:17.194509+0000","172.21.15.16:0/2401946155":"2023-10-17T13:49:17.194509+0000","172.21.15.16:0/1156439592":"2023-10-17T13:49:17.194509+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T13:49:36.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.745+0000 7f919e2a0700 1 -- 172.21.15.16:0/3565951701 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f917806b020 msgr2=0x7f917806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:36.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.745+0000 7f919e2a0700 1 --2- 172.21.15.16:0/3565951701 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f917806b020 0x7f917806d4e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f91880099f0 tx=0x7f9188008040 comp rx=0 tx=0).stop 2023-10-16T13:49:36.748 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.745+0000 7f919e2a0700 1 -- 172.21.15.16:0/3565951701 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91980cf880 msgr2=0x7f91980ce860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:36.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.745+0000 7f919e2a0700 1 --2- 172.21.15.16:0/3565951701 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91980cf880 0x7f91980ce860 secure :-1 s=READY pgs=3333 cs=0 l=1 rev1=1 crypto rx=0x7f919000b730 tx=0x7f9190004ad0 comp rx=0 tx=0).stop 2023-10-16T13:49:36.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.745+0000 7f919e2a0700 1 -- 172.21.15.16:0/3565951701 shutdown_connections 2023-10-16T13:49:36.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.745+0000 7f919e2a0700 1 --2- 172.21.15.16:0/3565951701 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f917806b020 0x7f917806d4e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:36.749 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.745+0000 7f919e2a0700 1 --2- 172.21.15.16:0/3565951701 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91980cf880 0x7f91980ce860 unknown :-1 s=CLOSED pgs=3333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:36.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.745+0000 7f919e2a0700 1 --2- 172.21.15.16:0/3565951701 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f91980b5830 0x7f91980c66c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:36.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.745+0000 7f919e2a0700 1 --2- 172.21.15.16:0/3565951701 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f91980cf4a0 0x7f91980ce320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:36.750 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.745+0000 7f919e2a0700 1 -- 172.21.15.16:0/3565951701 >> 172.21.15.16:0/3565951701 conn(0x7f91980b1fd0 msgr2=0x7f91980b1520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:36.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.745+0000 7f919e2a0700 1 -- 172.21.15.16:0/3565951701 shutdown_connections 2023-10-16T13:49:36.751 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:36.745+0000 7f919e2a0700 1 -- 172.21.15.16:0/3565951701 wait complete. 2023-10-16T13:49:37.761 INFO:tasks.ceph:Scrubbing osd.0 2023-10-16T13:49:37.762 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-10-16T13:49:37.906 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:37.904+0000 7f1c803e2700 1 Processor -- start 2023-10-16T13:49:37.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:37.904+0000 7f1c803e2700 1 -- start start 2023-10-16T13:49:37.908 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:37.905+0000 7f1c803e2700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f1c780cba40 0x7f1c780cbe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:37.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:37.905+0000 7f1c803e2700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c780cc3d0 con 0x7f1c78155a60 2023-10-16T13:49:37.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:37.905+0000 7f1c803e2700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c7815c780 con 0x7f1c78155e40 2023-10-16T13:49:37.909 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:37.905+0000 7f1c803e2700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c7815f240 con 0x7f1c780cba40 2023-10-16T13:49:37.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:37.905+0000 7f1c7e17e700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f1c78155a60 0x7f1c7815fc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49680/0 (socket says 172.21.15.16:49680) 2023-10-16T13:49:37.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:37.905+0000 7f1c7e17e700 1 -- 172.21.15.16:0/2622333232 learned_addr learned my addr 172.21.15.16:0/2622333232 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:49:37.910 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:37.905+0000 7f1c7e97f700 1 --2- 172.21.15.16:0/2622333232 >> v2:172.21.15.49:3300/0 conn(0x7f1c780cba40 0x7f1c780cbe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:37.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:37.905+0000 7f1c7d17c700 1 -- 172.21.15.16:0/2622333232 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1736249150 0 0) 0x7f1c6c001110 con 0x7f1c78155a60 2023-10-16T13:49:37.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:37.905+0000 7f1c7d17c700 1 -- 172.21.15.16:0/2622333232 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c640035b0 con 0x7f1c78155a60 2023-10-16T13:49:37.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:37.906+0000 7f1c7e97f700 1 -- 172.21.15.16:0/2622333232 >> v1:172.21.15.49:6789/0 conn(0x7f1c78155e40 legacy=0x7f1c7815bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:37.911 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:37.906+0000 7f1c7e97f700 1 -- 172.21.15.16:0/2622333232 >> v1:172.21.15.16:6789/0 conn(0x7f1c78155a60 legacy=0x7f1c7815fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:37.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:37.906+0000 7f1c7d17c700 1 -- 172.21.15.16:0/2622333232 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 699844271 0 0) 0x7f1c640035b0 con 0x7f1c78155a60 2023-10-16T13:49:37.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:37.906+0000 7f1c7e97f700 1 -- 172.21.15.16:0/2622333232 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c7815ca60 con 0x7f1c780cba40 2023-10-16T13:49:37.912 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:37.906+0000 7f1c7e97f700 1 --2- 172.21.15.16:0/2622333232 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1c780cba40 0x7f1c780cbe90 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f1c70009d90 tx=0x7f1c7000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4bd4cc4cd00f87fe server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:37.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:37.906+0000 7f1c7d17c700 1 -- 172.21.15.16:0/2622333232 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c7000e070 con 0x7f1c780cba40 2023-10-16T13:49:37.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:37.906+0000 7f1c7d17c700 1 -- 172.21.15.16:0/2622333232 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c70003a20 con 0x7f1c780cba40 2023-10-16T13:49:37.913 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:37.906+0000 7f1c7d17c700 1 -- 172.21.15.16:0/2622333232 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c6c001800 con 0x7f1c780cba40 2023-10-16T13:49:37.916 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:37.907+0000 7f1c803e2700 1 -- 172.21.15.16:0/2622333232 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1c780cba40 msgr2=0x7f1c780cbe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:37.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:37.907+0000 7f1c803e2700 1 --2- 172.21.15.16:0/2622333232 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1c780cba40 0x7f1c780cbe90 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f1c70009d90 tx=0x7f1c7000b330 comp rx=0 tx=0).stop 2023-10-16T13:49:37.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:37.907+0000 7f1c803e2700 1 -- 172.21.15.16:0/2622333232 shutdown_connections 2023-10-16T13:49:37.917 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:37.907+0000 7f1c803e2700 1 --2- 172.21.15.16:0/2622333232 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1c780cba40 0x7f1c780cbe90 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:37.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:37.907+0000 7f1c803e2700 1 -- 172.21.15.16:0/2622333232 >> 172.21.15.16:0/2622333232 conn(0x7f1c7806a7e0 msgr2=0x7f1c7806abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:37.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:37.907+0000 7f1c803e2700 1 -- 172.21.15.16:0/2622333232 shutdown_connections 2023-10-16T13:49:37.918 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:37.907+0000 7f1c803e2700 1 -- 172.21.15.16:0/2622333232 wait complete. 2023-10-16T13:49:37.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:37.908+0000 7f1c803e2700 1 Processor -- start 2023-10-16T13:49:37.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:37.908+0000 7f1c803e2700 1 -- start start 2023-10-16T13:49:37.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:37.909+0000 7f1c803e2700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1c780cba40 0x7f1c78154a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:37.919 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:37.909+0000 7f1c803e2700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1c78155a60 0x7f1c780ca640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:37.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:37.909+0000 7f1c803e2700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c78155e40 0x7f1c7815e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:37.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:37.909+0000 7f1c803e2700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c780b50f0 con 0x7f1c78155e40 2023-10-16T13:49:37.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:37.909+0000 7f1c803e2700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c780cb100 con 0x7f1c78155a60 2023-10-16T13:49:37.920 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:37.909+0000 7f1c803e2700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c780cb3e0 con 0x7f1c780cba40 2023-10-16T13:49:37.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:37.909+0000 7f1c7e97f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c78155e40 0x7f1c7815e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:37.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:37.909+0000 7f1c7e97f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c78155e40 0x7f1c7815e0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60416/0 (socket says 172.21.15.16:60416) 2023-10-16T13:49:37.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:37.909+0000 7f1c7e97f700 1 -- 172.21.15.16:0/2339958979 learned_addr learned my addr 172.21.15.16:0/2339958979 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:37.921 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:37.909+0000 7f1c7d97d700 1 --2- 172.21.15.16:0/2339958979 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1c78155a60 0x7f1c780ca640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:37.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:37.909+0000 7f1c7e17e700 1 --2- 172.21.15.16:0/2339958979 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1c780cba40 0x7f1c78154a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:37.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:37.909+0000 7f1c7e97f700 1 -- 172.21.15.16:0/2339958979 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1c780cba40 msgr2=0x7f1c78154a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:37.922 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:37.909+0000 7f1c7e97f700 1 --2- 172.21.15.16:0/2339958979 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f1c780cba40 0x7f1c78154a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:37.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:37.909+0000 7f1c7e97f700 1 -- 172.21.15.16:0/2339958979 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1c78155a60 msgr2=0x7f1c780ca640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:37.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:37.909+0000 7f1c7e97f700 1 --2- 172.21.15.16:0/2339958979 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1c78155a60 0x7f1c780ca640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:37.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:37.909+0000 7f1c7e97f700 1 -- 172.21.15.16:0/2339958979 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c70009a70 con 0x7f1c78155e40 2023-10-16T13:49:37.923 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:37.909+0000 7f1c7e97f700 1 --2- 172.21.15.16:0/2339958979 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c78155e40 0x7f1c7815e0b0 secure :-1 s=READY pgs=3335 cs=0 l=1 rev1=1 crypto rx=0x7f1c780cec60 tx=0x7f1c70004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:37.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:37.910+0000 7f1c6affd700 1 -- 172.21.15.16:0/2339958979 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c70004640 con 0x7f1c78155e40 2023-10-16T13:49:37.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:37.910+0000 7f1c6affd700 1 -- 172.21.15.16:0/2339958979 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c700047a0 con 0x7f1c78155e40 2023-10-16T13:49:37.924 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:37.910+0000 7f1c803e2700 1 -- 172.21.15.16:0/2339958979 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c780b4aa0 con 0x7f1c78155e40 2023-10-16T13:49:37.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:37.910+0000 7f1c6affd700 1 -- 172.21.15.16:0/2339958979 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c7001d670 con 0x7f1c78155e40 2023-10-16T13:49:37.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:37.910+0000 7f1c803e2700 1 -- 172.21.15.16:0/2339958979 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c780b0180 con 0x7f1c78155e40 2023-10-16T13:49:37.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:37.910+0000 7f1c803e2700 1 -- 172.21.15.16:0/2339958979 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1c7815f730 con 0x7f1c78155e40 2023-10-16T13:49:37.925 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:37.911+0000 7f1c6affd700 1 -- 172.21.15.16:0/2339958979 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 176) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1c7001d7d0 con 0x7f1c78155e40 2023-10-16T13:49:37.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:37.911+0000 7f1c6affd700 1 --2- 172.21.15.16:0/2339958979 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f1c5c06b050 0x7f1c5c06d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:37.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:37.912+0000 7f1c6affd700 1 -- 172.21.15.16:0/2339958979 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(283..283 src has 1..283) v4 ==== 6208+0+0 (secure 0 0 0) 0x7f1c7008b140 con 0x7f1c78155e40 2023-10-16T13:49:37.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:37.912+0000 7f1c7e17e700 1 --2- 172.21.15.16:0/2339958979 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f1c5c06b050 0x7f1c5c06d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:37.926 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:37.912+0000 7f1c6affd700 1 --2- 172.21.15.16:0/2339958979 >> [v2:172.21.15.16:6800/2366702185,v1:172.21.15.16:6801/2366702185] conn(0x7f1c5c071af0 0x7f1c5c073f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:37.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:37.912+0000 7f1c6affd700 1 -- 172.21.15.16:0/2339958979 --> [v2:172.21.15.16:6800/2366702185,v1:172.21.15.16:6801/2366702185] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1c5c074660 con 0x7f1c5c071af0 2023-10-16T13:49:37.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:37.912+0000 7f1c6affd700 1 -- 172.21.15.16:0/2339958979 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=283) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1c7008d720 con 0x7f1c78155e40 2023-10-16T13:49:37.927 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:37.912+0000 7f1c7d97d700 1 --2- 172.21.15.16:0/2339958979 >> [v2:172.21.15.16:6800/2366702185,v1:172.21.15.16:6801/2366702185] conn(0x7f1c5c071af0 0x7f1c5c073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:37.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:37.912+0000 7f1c7e17e700 1 --2- 172.21.15.16:0/2339958979 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f1c5c06b050 0x7f1c5c06d510 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f1c6c001eb0 tx=0x7f1c6c001640 comp rx=0 tx=0).ready entity=mgr.13935 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:37.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:37.912+0000 7f1c7d97d700 1 --2- 172.21.15.16:0/2339958979 >> [v2:172.21.15.16:6800/2366702185,v1:172.21.15.16:6801/2366702185] conn(0x7f1c5c071af0 0x7f1c5c073f90 crc :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:37.928 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:37.913+0000 7f1c6affd700 1 -- 172.21.15.16:0/2339958979 <== osd.0 v2:172.21.15.16:6800/2366702185 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7f1c5c074660 con 0x7f1c5c071af0 2023-10-16T13:49:37.930 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:37.928+0000 7f1c803e2700 1 -- 172.21.15.16:0/2339958979 --> [v2:172.21.15.16:6800/2366702185,v1:172.21.15.16:6801/2366702185] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f1c7815f450 con 0x7f1c5c071af0 2023-10-16T13:49:37.931 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:37.928+0000 7f1c6affd700 1 -- 172.21.15.16:0/2339958979 <== osd.0 v2:172.21.15.16:6800/2366702185 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f1c740050f0 con 0x7f1c5c071af0 2023-10-16T13:49:37.931 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-10-16T13:49:37.931 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-16T13:49:37.932 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-10-16T13:49:37.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:37.928+0000 7f1c803e2700 1 -- 172.21.15.16:0/2339958979 >> [v2:172.21.15.16:6800/2366702185,v1:172.21.15.16:6801/2366702185] conn(0x7f1c5c071af0 msgr2=0x7f1c5c073f90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:37.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:37.928+0000 7f1c803e2700 1 --2- 172.21.15.16:0/2339958979 >> [v2:172.21.15.16:6800/2366702185,v1:172.21.15.16:6801/2366702185] conn(0x7f1c5c071af0 0x7f1c5c073f90 crc :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:37.933 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:37.928+0000 7f1c803e2700 1 -- 172.21.15.16:0/2339958979 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f1c5c06b050 msgr2=0x7f1c5c06d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:37.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:37.928+0000 7f1c803e2700 1 --2- 172.21.15.16:0/2339958979 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f1c5c06b050 0x7f1c5c06d510 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f1c6c001eb0 tx=0x7f1c6c001640 comp rx=0 tx=0).stop 2023-10-16T13:49:37.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:37.928+0000 7f1c803e2700 1 -- 172.21.15.16:0/2339958979 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c78155e40 msgr2=0x7f1c7815e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:37.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:37.928+0000 7f1c803e2700 1 --2- 172.21.15.16:0/2339958979 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c78155e40 0x7f1c7815e0b0 secure :-1 s=READY pgs=3335 cs=0 l=1 rev1=1 crypto rx=0x7f1c780cec60 tx=0x7f1c70004ad0 comp rx=0 tx=0).stop 2023-10-16T13:49:37.934 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:37.928+0000 7f1c7e97f700 1 -- 172.21.15.16:0/2339958979 reap_dead start 2023-10-16T13:49:37.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:37.928+0000 7f1c803e2700 1 -- 172.21.15.16:0/2339958979 shutdown_connections 2023-10-16T13:49:37.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:37.929+0000 7f1c803e2700 1 -- 172.21.15.16:0/2339958979 >> 172.21.15.16:0/2339958979 conn(0x7f1c7806a7e0 msgr2=0x7f1c7806abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:37.935 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:37.929+0000 7f1c803e2700 1 -- 172.21.15.16:0/2339958979 shutdown_connections 2023-10-16T13:49:37.936 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:37.929+0000 7f1c803e2700 1 -- 172.21.15.16:0/2339958979 wait complete. 2023-10-16T13:49:37.943 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-10-16T13:49:38.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.066+0000 7fc4393d4700 1 Processor -- start 2023-10-16T13:49:38.069 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.066+0000 7fc4393d4700 1 -- start start 2023-10-16T13:49:38.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.067+0000 7fc4393d4700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc4340d41c0 0x7fc4340d0040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:38.071 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.067+0000 7fc4393d4700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fc4340d0580 0x7fc43406c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:38.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.067+0000 7fc4393d4700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc43406ba00 con 0x7fc43415d8d0 2023-10-16T13:49:38.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.067+0000 7fc4393d4700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4340d0a10 con 0x7fc4340d0580 2023-10-16T13:49:38.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.067+0000 7fc4393d4700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4340d26d0 con 0x7fc4340d41c0 2023-10-16T13:49:38.072 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.067+0000 7fc432ffd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc43415d8d0 0x7fc4340d1c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49696/0 (socket says 172.21.15.16:49696) 2023-10-16T13:49:38.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.067+0000 7fc432ffd700 1 -- 172.21.15.16:0/2952426425 learned_addr learned my addr 172.21.15.16:0/2952426425 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:49:38.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.067+0000 7fc4337fe700 1 --2- 172.21.15.16:0/2952426425 >> v2:172.21.15.49:3300/0 conn(0x7fc4340d0580 0x7fc43406c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:38.073 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.067+0000 7fc4327fc700 1 --2- 172.21.15.16:0/2952426425 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc4340d41c0 0x7fc4340d0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:38.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.067+0000 7fc431ffb700 1 -- 172.21.15.16:0/2952426425 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 865793164 0 0) 0x7fc43406ba00 con 0x7fc43415d8d0 2023-10-16T13:49:38.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.067+0000 7fc431ffb700 1 -- 172.21.15.16:0/2952426425 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc41c0035b0 con 0x7fc43415d8d0 2023-10-16T13:49:38.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.068+0000 7fc4337fe700 1 -- 172.21.15.16:0/2952426425 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc4340d41c0 msgr2=0x7fc4340d0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:38.074 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.068+0000 7fc4337fe700 1 --2- 172.21.15.16:0/2952426425 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc4340d41c0 0x7fc4340d0040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:38.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.068+0000 7fc431ffb700 1 -- 172.21.15.16:0/2952426425 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2541901585 0 0) 0x7fc41c0035b0 con 0x7fc43415d8d0 2023-10-16T13:49:38.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.068+0000 7fc4337fe700 1 -- 172.21.15.16:0/2952426425 >> v1:172.21.15.16:6789/0 conn(0x7fc43415d8d0 legacy=0x7fc4340d1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:38.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.068+0000 7fc4337fe700 1 -- 172.21.15.16:0/2952426425 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4340d23f0 con 0x7fc4340d0580 2023-10-16T13:49:38.075 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.068+0000 7fc4337fe700 1 --2- 172.21.15.16:0/2952426425 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc4340d0580 0x7fc43406c4a0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7fc42c009dc0 tx=0x7fc42c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c7ef770d5541133 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:38.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.068+0000 7fc431ffb700 1 -- 172.21.15.16:0/2952426425 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc42c00e070 con 0x7fc4340d0580 2023-10-16T13:49:38.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.068+0000 7fc431ffb700 1 -- 172.21.15.16:0/2952426425 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc42c003a20 con 0x7fc4340d0580 2023-10-16T13:49:38.076 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.068+0000 7fc431ffb700 1 -- 172.21.15.16:0/2952426425 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc428001670 con 0x7fc4340d0580 2023-10-16T13:49:38.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.069+0000 7fc4393d4700 1 -- 172.21.15.16:0/2952426425 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc4340d0580 msgr2=0x7fc43406c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:38.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.069+0000 7fc4393d4700 1 --2- 172.21.15.16:0/2952426425 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc4340d0580 0x7fc43406c4a0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7fc42c009dc0 tx=0x7fc42c00b330 comp rx=0 tx=0).stop 2023-10-16T13:49:38.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.069+0000 7fc4393d4700 1 -- 172.21.15.16:0/2952426425 shutdown_connections 2023-10-16T13:49:38.077 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.069+0000 7fc4393d4700 1 --2- 172.21.15.16:0/2952426425 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc4340d0580 0x7fc43406c4a0 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:38.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.069+0000 7fc4393d4700 1 --2- 172.21.15.16:0/2952426425 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc4340d41c0 0x7fc4340d0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:38.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.069+0000 7fc4393d4700 1 -- 172.21.15.16:0/2952426425 >> 172.21.15.16:0/2952426425 conn(0x7fc4340d3630 msgr2=0x7fc4340d3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:38.078 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.069+0000 7fc4393d4700 1 -- 172.21.15.16:0/2952426425 shutdown_connections 2023-10-16T13:49:38.079 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.069+0000 7fc4393d4700 1 -- 172.21.15.16:0/2952426425 wait complete. 2023-10-16T13:49:38.081 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.070+0000 7fc4393d4700 1 Processor -- start 2023-10-16T13:49:38.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.071+0000 7fc4393d4700 1 -- start start 2023-10-16T13:49:38.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.071+0000 7fc4393d4700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc4340d0580 0x7fc4341f5a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:38.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.071+0000 7fc4393d4700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc4340d41c0 0x7fc4340c7b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:38.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.071+0000 7fc4393d4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc43415d8d0 0x7fc4341f2e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:38.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.071+0000 7fc4393d4700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc43409d310 con 0x7fc43415d8d0 2023-10-16T13:49:38.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.072+0000 7fc4393d4700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4341f5fb0 con 0x7fc4340d41c0 2023-10-16T13:49:38.083 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.072+0000 7fc4393d4700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4340c8480 con 0x7fc4340d0580 2023-10-16T13:49:38.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.072+0000 7fc4337fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc43415d8d0 0x7fc4341f2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:38.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.072+0000 7fc4337fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc43415d8d0 0x7fc4341f2e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60432/0 (socket says 172.21.15.16:60432) 2023-10-16T13:49:38.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.072+0000 7fc4337fe700 1 -- 172.21.15.16:0/3929933407 learned_addr learned my addr 172.21.15.16:0/3929933407 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:38.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.072+0000 7fc432ffd700 1 --2- 172.21.15.16:0/3929933407 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc4340d0580 0x7fc4341f5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:38.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.072+0000 7fc4327fc700 1 --2- 172.21.15.16:0/3929933407 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc4340d41c0 0x7fc4340c7b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:38.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.072+0000 7fc4337fe700 1 -- 172.21.15.16:0/3929933407 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc4340d0580 msgr2=0x7fc4341f5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:38.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.072+0000 7fc4337fe700 1 --2- 172.21.15.16:0/3929933407 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc4340d0580 0x7fc4341f5a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:38.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.072+0000 7fc4337fe700 1 -- 172.21.15.16:0/3929933407 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc4340d41c0 msgr2=0x7fc4340c7b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:38.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.072+0000 7fc4337fe700 1 --2- 172.21.15.16:0/3929933407 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc4340d41c0 0x7fc4340c7b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:38.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.072+0000 7fc4337fe700 1 -- 172.21.15.16:0/3929933407 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc42c009a70 con 0x7fc43415d8d0 2023-10-16T13:49:38.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.072+0000 7fc4337fe700 1 --2- 172.21.15.16:0/3929933407 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc43415d8d0 0x7fc4341f2e30 secure :-1 s=READY pgs=3337 cs=0 l=1 rev1=1 crypto rx=0x7fc42c015040 tx=0x7fc42c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:38.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.072+0000 7fc41b7fe700 1 -- 172.21.15.16:0/3929933407 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc42c0045c0 con 0x7fc43415d8d0 2023-10-16T13:49:38.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.072+0000 7fc41b7fe700 1 -- 172.21.15.16:0/3929933407 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc42c004740 con 0x7fc43415d8d0 2023-10-16T13:49:38.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.072+0000 7fc41b7fe700 1 -- 172.21.15.16:0/3929933407 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc42c009430 con 0x7fc43415d8d0 2023-10-16T13:49:38.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.072+0000 7fc4393d4700 1 -- 172.21.15.16:0/3929933407 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4340c8760 con 0x7fc43415d8d0 2023-10-16T13:49:38.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.072+0000 7fc4393d4700 1 -- 172.21.15.16:0/3929933407 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4340c8d20 con 0x7fc43415d8d0 2023-10-16T13:49:38.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.073+0000 7fc4393d4700 1 -- 172.21.15.16:0/3929933407 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc43404ebb0 con 0x7fc43415d8d0 2023-10-16T13:49:38.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.074+0000 7fc41b7fe700 1 -- 172.21.15.16:0/3929933407 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 176) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc42c012450 con 0x7fc43415d8d0 2023-10-16T13:49:38.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.074+0000 7fc41b7fe700 1 --2- 172.21.15.16:0/3929933407 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7fc42006b020 0x7fc42006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:38.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.074+0000 7fc41b7fe700 1 -- 172.21.15.16:0/3929933407 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(283..283 src has 1..283) v4 ==== 6208+0+0 (secure 0 0 0) 0x7fc42c08b110 con 0x7fc43415d8d0 2023-10-16T13:49:38.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.074+0000 7fc432ffd700 1 --2- 172.21.15.16:0/3929933407 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7fc42006b020 0x7fc42006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:38.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.075+0000 7fc432ffd700 1 --2- 172.21.15.16:0/3929933407 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7fc42006b020 0x7fc42006d4e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fc428001eb0 tx=0x7fc428001c40 comp rx=0 tx=0).ready entity=mgr.13935 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:38.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.079+0000 7fc41b7fe700 1 -- 172.21.15.16:0/3929933407 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc42c059840 con 0x7fc43415d8d0 2023-10-16T13:49:38.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.280+0000 7fc4393d4700 1 -- 172.21.15.16:0/3929933407 --> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7fc43406bf80 con 0x7fc42006b020 2023-10-16T13:49:38.283 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.281+0000 7fc41b7fe700 1 -- 172.21.15.16:0/3929933407 <== mgr.13935 v2:172.21.15.16:6832/938897590 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fc428001650 con 0x7fc42006b020 2023-10-16T13:49:38.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.283+0000 7fc4393d4700 1 -- 172.21.15.16:0/3929933407 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7fc42006b020 msgr2=0x7fc42006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:38.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.283+0000 7fc4393d4700 1 --2- 172.21.15.16:0/3929933407 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7fc42006b020 0x7fc42006d4e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fc428001eb0 tx=0x7fc428001c40 comp rx=0 tx=0).stop 2023-10-16T13:49:38.286 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.283+0000 7fc4393d4700 1 -- 172.21.15.16:0/3929933407 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc43415d8d0 msgr2=0x7fc4341f2e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:38.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.283+0000 7fc4393d4700 1 --2- 172.21.15.16:0/3929933407 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc43415d8d0 0x7fc4341f2e30 secure :-1 s=READY pgs=3337 cs=0 l=1 rev1=1 crypto rx=0x7fc42c015040 tx=0x7fc42c004ad0 comp rx=0 tx=0).stop 2023-10-16T13:49:38.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.283+0000 7fc4393d4700 1 -- 172.21.15.16:0/3929933407 shutdown_connections 2023-10-16T13:49:38.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.283+0000 7fc4393d4700 1 --2- 172.21.15.16:0/3929933407 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7fc42006b020 0x7fc42006d4e0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:38.287 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.283+0000 7fc4393d4700 1 --2- 172.21.15.16:0/3929933407 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc43415d8d0 0x7fc4341f2e30 unknown :-1 s=CLOSED pgs=3337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:38.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.283+0000 7fc4393d4700 1 --2- 172.21.15.16:0/3929933407 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fc4340d0580 0x7fc4341f5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:38.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.283+0000 7fc4393d4700 1 --2- 172.21.15.16:0/3929933407 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc4340d41c0 0x7fc4340c7b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:38.288 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.283+0000 7fc4393d4700 1 -- 172.21.15.16:0/3929933407 >> 172.21.15.16:0/3929933407 conn(0x7fc4340d3630 msgr2=0x7fc4340d1c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:38.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.283+0000 7fc4393d4700 1 -- 172.21.15.16:0/3929933407 shutdown_connections 2023-10-16T13:49:38.289 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.283+0000 7fc4393d4700 1 -- 172.21.15.16:0/3929933407 wait complete. 2023-10-16T13:49:38.289 INFO:teuthology.orchestra.run.smithi016.stderr:instructed osd(s) 0 to deep-scrub 2023-10-16T13:49:38.298 INFO:tasks.ceph:Scrubbing osd.1 2023-10-16T13:49:38.298 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-10-16T13:49:38.427 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.425+0000 7fe84f7d1700 1 Processor -- start 2023-10-16T13:49:38.428 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.425+0000 7fe84f7d1700 1 -- start start 2023-10-16T13:49:38.430 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.426+0000 7fe84f7d1700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe848155a60 0x7fe84815fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:38.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.426+0000 7fe84f7d1700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fe8480cba40 0x7fe8480cbe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:38.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.426+0000 7fe84f7d1700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8480cc3d0 con 0x7fe848155e40 2023-10-16T13:49:38.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.426+0000 7fe84f7d1700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe84815f240 con 0x7fe8480cba40 2023-10-16T13:49:38.431 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.426+0000 7fe84f7d1700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe848160190 con 0x7fe848155a60 2023-10-16T13:49:38.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.426+0000 7fe84cd6c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe848155e40 0x7fe84815bc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49714/0 (socket says 172.21.15.16:49714) 2023-10-16T13:49:38.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.426+0000 7fe84cd6c700 1 -- 172.21.15.16:0/1863100417 learned_addr learned my addr 172.21.15.16:0/1863100417 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:49:38.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.426+0000 7fe84dd6e700 1 --2- 172.21.15.16:0/1863100417 >> v2:172.21.15.49:3300/0 conn(0x7fe8480cba40 0x7fe8480cbe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:38.432 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.426+0000 7fe84d56d700 1 --2- 172.21.15.16:0/1863100417 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe848155a60 0x7fe84815fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:38.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.426+0000 7fe83ffff700 1 -- 172.21.15.16:0/1863100417 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 680551012 0 0) 0x7fe838001110 con 0x7fe848155e40 2023-10-16T13:49:38.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.426+0000 7fe83ffff700 1 -- 172.21.15.16:0/1863100417 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8340035b0 con 0x7fe848155e40 2023-10-16T13:49:38.433 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.426+0000 7fe84d56d700 1 -- 172.21.15.16:0/1863100417 >> v2:172.21.15.49:3300/0 conn(0x7fe8480cba40 msgr2=0x7fe8480cbe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:38.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.426+0000 7fe84d56d700 1 --2- 172.21.15.16:0/1863100417 >> v2:172.21.15.49:3300/0 conn(0x7fe8480cba40 0x7fe8480cbe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:38.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.426+0000 7fe84d56d700 1 -- 172.21.15.16:0/1863100417 >> v1:172.21.15.16:6789/0 conn(0x7fe848155e40 legacy=0x7fe84815bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:38.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.426+0000 7fe84d56d700 1 -- 172.21.15.16:0/1863100417 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe84815c780 con 0x7fe848155a60 2023-10-16T13:49:38.434 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.426+0000 7fe83ffff700 1 -- 172.21.15.16:0/1863100417 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4193402391 0 0) 0x7fe8340035b0 con 0x7fe848155e40 2023-10-16T13:49:38.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.426+0000 7fe84d56d700 1 --2- 172.21.15.16:0/1863100417 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe848155a60 0x7fe84815fc50 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fe844009840 tx=0x7fe844009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9317c5d32f1c8a6a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:38.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.427+0000 7fe83ffff700 1 -- 172.21.15.16:0/1863100417 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe838001800 con 0x7fe848155a60 2023-10-16T13:49:38.435 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.427+0000 7fe83ffff700 1 -- 172.21.15.16:0/1863100417 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe844004030 con 0x7fe848155a60 2023-10-16T13:49:38.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.427+0000 7fe83ffff700 1 -- 172.21.15.16:0/1863100417 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8440043c0 con 0x7fe848155a60 2023-10-16T13:49:38.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.427+0000 7fe84f7d1700 1 -- 172.21.15.16:0/1863100417 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe848155a60 msgr2=0x7fe84815fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:38.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.427+0000 7fe84f7d1700 1 --2- 172.21.15.16:0/1863100417 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe848155a60 0x7fe84815fc50 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fe844009840 tx=0x7fe844009b50 comp rx=0 tx=0).stop 2023-10-16T13:49:38.436 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.427+0000 7fe84f7d1700 1 -- 172.21.15.16:0/1863100417 shutdown_connections 2023-10-16T13:49:38.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.427+0000 7fe84f7d1700 1 --2- 172.21.15.16:0/1863100417 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe848155a60 0x7fe84815fc50 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:38.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.427+0000 7fe84f7d1700 1 --2- 172.21.15.16:0/1863100417 >> v2:172.21.15.49:3300/0 conn(0x7fe8480cba40 0x7fe8480cbe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:38.437 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.427+0000 7fe84f7d1700 1 -- 172.21.15.16:0/1863100417 >> 172.21.15.16:0/1863100417 conn(0x7fe84806a7e0 msgr2=0x7fe84806abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:38.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.427+0000 7fe84f7d1700 1 -- 172.21.15.16:0/1863100417 shutdown_connections 2023-10-16T13:49:38.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.427+0000 7fe84f7d1700 1 -- 172.21.15.16:0/1863100417 wait complete. 2023-10-16T13:49:38.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.429+0000 7fe84f7d1700 1 Processor -- start 2023-10-16T13:49:38.441 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.429+0000 7fe84f7d1700 1 -- start start 2023-10-16T13:49:38.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.430+0000 7fe84f7d1700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe8480cba40 0x7fe848151a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:38.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.430+0000 7fe84f7d1700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe848155a60 0x7fe848157ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:38.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.430+0000 7fe84f7d1700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe848155e40 0x7fe84815dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:38.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.430+0000 7fe84f7d1700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8480b41b0 con 0x7fe848155a60 2023-10-16T13:49:38.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.430+0000 7fe84f7d1700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8480c9440 con 0x7fe8480cba40 2023-10-16T13:49:38.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.430+0000 7fe84f7d1700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe848151f60 con 0x7fe848155e40 2023-10-16T13:49:38.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.430+0000 7fe84cd6c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe848155a60 0x7fe848157ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:38.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.430+0000 7fe84cd6c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe848155a60 0x7fe848157ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60444/0 (socket says 172.21.15.16:60444) 2023-10-16T13:49:38.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.430+0000 7fe84cd6c700 1 -- 172.21.15.16:0/3344334745 learned_addr learned my addr 172.21.15.16:0/3344334745 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:38.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.430+0000 7fe84d56d700 1 --2- 172.21.15.16:0/3344334745 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe8480cba40 0x7fe848151a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:38.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.430+0000 7fe84dd6e700 1 --2- 172.21.15.16:0/3344334745 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe848155e40 0x7fe84815dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:38.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.430+0000 7fe84cd6c700 1 -- 172.21.15.16:0/3344334745 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe848155e40 msgr2=0x7fe84815dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:38.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.430+0000 7fe84cd6c700 1 --2- 172.21.15.16:0/3344334745 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fe848155e40 0x7fe84815dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:38.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.430+0000 7fe84cd6c700 1 -- 172.21.15.16:0/3344334745 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe8480cba40 msgr2=0x7fe848151a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:38.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.430+0000 7fe84cd6c700 1 --2- 172.21.15.16:0/3344334745 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe8480cba40 0x7fe848151a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:38.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.430+0000 7fe84cd6c700 1 -- 172.21.15.16:0/3344334745 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8440094a0 con 0x7fe848155a60 2023-10-16T13:49:38.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.431+0000 7fe84cd6c700 1 --2- 172.21.15.16:0/3344334745 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe848155a60 0x7fe848157ca0 secure :-1 s=READY pgs=3339 cs=0 l=1 rev1=1 crypto rx=0x7fe83800c750 tx=0x7fe83800ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:38.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.431+0000 7fe83dffb700 1 -- 172.21.15.16:0/3344334745 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe838004c80 con 0x7fe848155a60 2023-10-16T13:49:38.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.431+0000 7fe83dffb700 1 -- 172.21.15.16:0/3344334745 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe838004de0 con 0x7fe848155a60 2023-10-16T13:49:38.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.431+0000 7fe83dffb700 1 -- 172.21.15.16:0/3344334745 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8380072d0 con 0x7fe848155a60 2023-10-16T13:49:38.447 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.431+0000 7fe84f7d1700 1 -- 172.21.15.16:0/3344334745 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8480c8830 con 0x7fe848155a60 2023-10-16T13:49:38.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.431+0000 7fe84f7d1700 1 -- 172.21.15.16:0/3344334745 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8480c8df0 con 0x7fe848155a60 2023-10-16T13:49:38.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.432+0000 7fe84f7d1700 1 -- 172.21.15.16:0/3344334745 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe84815f450 con 0x7fe848155a60 2023-10-16T13:49:38.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.432+0000 7fe83dffb700 1 -- 172.21.15.16:0/3344334745 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 176) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe838007430 con 0x7fe848155a60 2023-10-16T13:49:38.448 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.433+0000 7fe83dffb700 1 --2- 172.21.15.16:0/3344334745 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7fe82c06b020 0x7fe82c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:38.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.433+0000 7fe83dffb700 1 -- 172.21.15.16:0/3344334745 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(283..283 src has 1..283) v4 ==== 6208+0+0 (secure 0 0 0) 0x7fe838089530 con 0x7fe848155a60 2023-10-16T13:49:38.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.433+0000 7fe84d56d700 1 --2- 172.21.15.16:0/3344334745 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7fe82c06b020 0x7fe82c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:38.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.433+0000 7fe83dffb700 1 --2- 172.21.15.16:0/3344334745 >> [v2:172.21.15.16:6816/78500784,v1:172.21.15.16:6818/78500784] conn(0x7fe82c071ac0 0x7fe82c073f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:38.449 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.433+0000 7fe83dffb700 1 -- 172.21.15.16:0/3344334745 --> [v2:172.21.15.16:6816/78500784,v1:172.21.15.16:6818/78500784] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe82c074630 con 0x7fe82c071ac0 2023-10-16T13:49:38.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.433+0000 7fe83dffb700 1 -- 172.21.15.16:0/3344334745 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=283) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe83808bbd0 con 0x7fe848155a60 2023-10-16T13:49:38.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.433+0000 7fe84dd6e700 1 --2- 172.21.15.16:0/3344334745 >> [v2:172.21.15.16:6816/78500784,v1:172.21.15.16:6818/78500784] conn(0x7fe82c071ac0 0x7fe82c073f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:38.450 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.433+0000 7fe84d56d700 1 --2- 172.21.15.16:0/3344334745 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7fe82c06b020 0x7fe82c06d4e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe8440090b0 tx=0x7fe844008030 comp rx=0 tx=0).ready entity=mgr.13935 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:38.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.433+0000 7fe84dd6e700 1 --2- 172.21.15.16:0/3344334745 >> [v2:172.21.15.16:6816/78500784,v1:172.21.15.16:6818/78500784] conn(0x7fe82c071ac0 0x7fe82c073f60 crc :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:38.451 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.434+0000 7fe83dffb700 1 -- 172.21.15.16:0/3344334745 <== osd.1 v2:172.21.15.16:6816/78500784 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7fe82c074630 con 0x7fe82c071ac0 2023-10-16T13:49:38.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.449+0000 7fe84f7d1700 1 -- 172.21.15.16:0/3344334745 --> [v2:172.21.15.16:6816/78500784,v1:172.21.15.16:6818/78500784] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fe84815ca60 con 0x7fe82c071ac0 2023-10-16T13:49:38.452 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.449+0000 7fe83dffb700 1 -- 172.21.15.16:0/3344334745 <== osd.1 v2:172.21.15.16:6816/78500784 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fe84000f040 con 0x7fe82c071ac0 2023-10-16T13:49:38.453 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-10-16T13:49:38.453 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-16T13:49:38.453 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-10-16T13:49:38.454 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.449+0000 7fe84f7d1700 1 -- 172.21.15.16:0/3344334745 >> [v2:172.21.15.16:6816/78500784,v1:172.21.15.16:6818/78500784] conn(0x7fe82c071ac0 msgr2=0x7fe82c073f60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:38.455 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.449+0000 7fe84f7d1700 1 --2- 172.21.15.16:0/3344334745 >> [v2:172.21.15.16:6816/78500784,v1:172.21.15.16:6818/78500784] conn(0x7fe82c071ac0 0x7fe82c073f60 crc :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:38.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.449+0000 7fe84f7d1700 1 -- 172.21.15.16:0/3344334745 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7fe82c06b020 msgr2=0x7fe82c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:38.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.449+0000 7fe84f7d1700 1 --2- 172.21.15.16:0/3344334745 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7fe82c06b020 0x7fe82c06d4e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe8440090b0 tx=0x7fe844008030 comp rx=0 tx=0).stop 2023-10-16T13:49:38.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.449+0000 7fe84f7d1700 1 -- 172.21.15.16:0/3344334745 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe848155a60 msgr2=0x7fe848157ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:38.457 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.449+0000 7fe84f7d1700 1 --2- 172.21.15.16:0/3344334745 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe848155a60 0x7fe848157ca0 secure :-1 s=READY pgs=3339 cs=0 l=1 rev1=1 crypto rx=0x7fe83800c750 tx=0x7fe83800ca60 comp rx=0 tx=0).stop 2023-10-16T13:49:38.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.449+0000 7fe84dd6e700 1 -- 172.21.15.16:0/3344334745 reap_dead start 2023-10-16T13:49:38.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.449+0000 7fe84f7d1700 1 -- 172.21.15.16:0/3344334745 shutdown_connections 2023-10-16T13:49:38.458 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.449+0000 7fe84f7d1700 1 -- 172.21.15.16:0/3344334745 >> 172.21.15.16:0/3344334745 conn(0x7fe84806a7e0 msgr2=0x7fe84806f1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:38.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.449+0000 7fe84f7d1700 1 -- 172.21.15.16:0/3344334745 shutdown_connections 2023-10-16T13:49:38.459 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.449+0000 7fe84f7d1700 1 -- 172.21.15.16:0/3344334745 wait complete. 2023-10-16T13:49:38.464 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-10-16T13:49:38.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.588+0000 7fa83da2c700 1 Processor -- start 2023-10-16T13:49:38.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.588+0000 7fa83da2c700 1 -- start start 2023-10-16T13:49:38.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.589+0000 7fa83da2c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa83815da40 0x7fa8380b2f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:38.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.589+0000 7fa83da2c700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa83806ccc0 con 0x7fa8380b34c0 2023-10-16T13:49:38.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.589+0000 7fa83da2c700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8380b3920 con 0x7fa83806c3e0 2023-10-16T13:49:38.594 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.589+0000 7fa83da2c700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8380cc4d0 con 0x7fa83815da40 2023-10-16T13:49:38.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.589+0000 7fa8367fc700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa8380b34c0 0x7fa83806bcd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49726/0 (socket says 172.21.15.16:49726) 2023-10-16T13:49:38.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.589+0000 7fa8367fc700 1 -- 172.21.15.16:0/3642942723 learned_addr learned my addr 172.21.15.16:0/3642942723 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:49:38.595 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.589+0000 7fa836ffd700 1 --2- 172.21.15.16:0/3642942723 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa83815da40 0x7fa8380b2f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:38.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.589+0000 7fa835ffb700 1 -- 172.21.15.16:0/3642942723 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3554039924 0 0) 0x7fa83806ccc0 con 0x7fa8380b34c0 2023-10-16T13:49:38.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.589+0000 7fa835ffb700 1 -- 172.21.15.16:0/3642942723 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8200035b0 con 0x7fa8380b34c0 2023-10-16T13:49:38.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.589+0000 7fa835ffb700 1 -- 172.21.15.16:0/3642942723 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4018755959 0 0) 0x7fa8380b3920 con 0x7fa83806c3e0 2023-10-16T13:49:38.596 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.589+0000 7fa835ffb700 1 -- 172.21.15.16:0/3642942723 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa83806ccc0 con 0x7fa83806c3e0 2023-10-16T13:49:38.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.589+0000 7fa836ffd700 1 -- 172.21.15.16:0/3642942723 >> v1:172.21.15.49:6789/0 conn(0x7fa83806c3e0 legacy=0x7fa8380cc8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:38.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.589+0000 7fa836ffd700 1 -- 172.21.15.16:0/3642942723 >> v1:172.21.15.16:6789/0 conn(0x7fa8380b34c0 legacy=0x7fa83806bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:38.597 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.589+0000 7fa836ffd700 1 -- 172.21.15.16:0/3642942723 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8380cbb70 con 0x7fa83815da40 2023-10-16T13:49:38.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.590+0000 7fa836ffd700 1 --2- 172.21.15.16:0/3642942723 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa83815da40 0x7fa8380b2f80 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fa82c009870 tx=0x7fa82c009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba0278e5089669f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:38.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.590+0000 7fa835ffb700 1 -- 172.21.15.16:0/3642942723 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa82c011830 con 0x7fa83815da40 2023-10-16T13:49:38.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.590+0000 7fa835ffb700 1 -- 172.21.15.16:0/3642942723 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa82c011990 con 0x7fa83815da40 2023-10-16T13:49:38.598 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.590+0000 7fa835ffb700 1 -- 172.21.15.16:0/3642942723 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa82c011d20 con 0x7fa83815da40 2023-10-16T13:49:38.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.590+0000 7fa83da2c700 1 -- 172.21.15.16:0/3642942723 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa83815da40 msgr2=0x7fa8380b2f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:38.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.590+0000 7fa83da2c700 1 --2- 172.21.15.16:0/3642942723 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa83815da40 0x7fa8380b2f80 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fa82c009870 tx=0x7fa82c009b80 comp rx=0 tx=0).stop 2023-10-16T13:49:38.599 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.590+0000 7fa83da2c700 1 -- 172.21.15.16:0/3642942723 shutdown_connections 2023-10-16T13:49:38.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.590+0000 7fa83da2c700 1 --2- 172.21.15.16:0/3642942723 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa83815da40 0x7fa8380b2f80 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:38.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.590+0000 7fa83da2c700 1 -- 172.21.15.16:0/3642942723 >> 172.21.15.16:0/3642942723 conn(0x7fa8380cdf80 msgr2=0x7fa8380ce390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:38.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.590+0000 7fa83da2c700 1 -- 172.21.15.16:0/3642942723 shutdown_connections 2023-10-16T13:49:38.600 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.590+0000 7fa83da2c700 1 -- 172.21.15.16:0/3642942723 wait complete. 2023-10-16T13:49:38.601 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.591+0000 7fa83da2c700 1 Processor -- start 2023-10-16T13:49:38.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.592+0000 7fa83da2c700 1 -- start start 2023-10-16T13:49:38.603 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.592+0000 7fa83da2c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa83806c3e0 0x7fa83815c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:38.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.592+0000 7fa83da2c700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa8380b34c0 0x7fa8380b60c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:38.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.592+0000 7fa83da2c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa83815da40 0x7fa8380b6600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:38.604 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.592+0000 7fa83da2c700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8380c7890 con 0x7fa83815da40 2023-10-16T13:49:38.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.592+0000 7fa83da2c700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8380c7b70 con 0x7fa83806c3e0 2023-10-16T13:49:38.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.592+0000 7fa83da2c700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8380c7e50 con 0x7fa8380b34c0 2023-10-16T13:49:38.605 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.593+0000 7fa8377fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa83815da40 0x7fa8380b6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:38.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.593+0000 7fa8367fc700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa8380b34c0 0x7fa8380b60c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:38.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.593+0000 7fa836ffd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa83806c3e0 0x7fa83815c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:38.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.593+0000 7fa8377fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa83815da40 0x7fa8380b6600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60462/0 (socket says 172.21.15.16:60462) 2023-10-16T13:49:38.606 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.593+0000 7fa836ffd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa83806c3e0 0x7fa83815c240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.16:59520/0 (socket says 172.21.15.16:59520) 2023-10-16T13:49:38.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.593+0000 7fa8367fc700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa8380b34c0 0x7fa8380b60c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:48718/0 (socket says 172.21.15.16:48718) 2023-10-16T13:49:38.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.593+0000 7fa8377fe700 1 -- 172.21.15.16:0/1944812969 learned_addr learned my addr 172.21.15.16:0/1944812969 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:38.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.593+0000 7fa8377fe700 1 -- 172.21.15.16:0/1944812969 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa8380b34c0 msgr2=0x7fa8380b60c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:38.607 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.593+0000 7fa8377fe700 1 --2- 172.21.15.16:0/1944812969 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa8380b34c0 0x7fa8380b60c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:38.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.593+0000 7fa8377fe700 1 -- 172.21.15.16:0/1944812969 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa83806c3e0 msgr2=0x7fa83815c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:38.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.593+0000 7fa8377fe700 1 --2- 172.21.15.16:0/1944812969 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa83806c3e0 0x7fa83815c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:38.608 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.593+0000 7fa8377fe700 1 -- 172.21.15.16:0/1944812969 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa82c0094a0 con 0x7fa83815da40 2023-10-16T13:49:38.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.593+0000 7fa8377fe700 1 --2- 172.21.15.16:0/1944812969 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa83815da40 0x7fa8380b6600 secure :-1 s=READY pgs=3341 cs=0 l=1 rev1=1 crypto rx=0x7fa83000cc20 tx=0x7fa83000cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:38.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.593+0000 7fa81ffff700 1 -- 172.21.15.16:0/1944812969 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa830003c00 con 0x7fa83815da40 2023-10-16T13:49:38.609 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.593+0000 7fa81ffff700 1 -- 172.21.15.16:0/1944812969 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa830003d60 con 0x7fa83815da40 2023-10-16T13:49:38.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.593+0000 7fa83da2c700 1 -- 172.21.15.16:0/1944812969 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8380c8130 con 0x7fa83815da40 2023-10-16T13:49:38.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.593+0000 7fa81ffff700 1 -- 172.21.15.16:0/1944812969 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa83000aae0 con 0x7fa83815da40 2023-10-16T13:49:38.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.593+0000 7fa83da2c700 1 -- 172.21.15.16:0/1944812969 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8380c6ba0 con 0x7fa83815da40 2023-10-16T13:49:38.610 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.594+0000 7fa83da2c700 1 -- 172.21.15.16:0/1944812969 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8380c8320 con 0x7fa83815da40 2023-10-16T13:49:38.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.595+0000 7fa81ffff700 1 -- 172.21.15.16:0/1944812969 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 176) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa83000ac40 con 0x7fa83815da40 2023-10-16T13:49:38.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.595+0000 7fa81ffff700 1 --2- 172.21.15.16:0/1944812969 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7fa82406b020 0x7fa82406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:38.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.595+0000 7fa81ffff700 1 -- 172.21.15.16:0/1944812969 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(283..283 src has 1..283) v4 ==== 6208+0+0 (secure 0 0 0) 0x7fa8300899f0 con 0x7fa83815da40 2023-10-16T13:49:38.611 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.595+0000 7fa836ffd700 1 --2- 172.21.15.16:0/1944812969 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7fa82406b020 0x7fa82406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:38.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.596+0000 7fa836ffd700 1 --2- 172.21.15.16:0/1944812969 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7fa82406b020 0x7fa82406d4e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa82c0090b0 tx=0x7fa82c01c040 comp rx=0 tx=0).ready entity=mgr.13935 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:38.612 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.600+0000 7fa81ffff700 1 -- 172.21.15.16:0/1944812969 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa830054730 con 0x7fa83815da40 2023-10-16T13:49:38.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.805+0000 7fa83da2c700 1 -- 172.21.15.16:0/1944812969 --> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7fa8380c7ce0 con 0x7fa82406b020 2023-10-16T13:49:38.808 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.805+0000 7fa81ffff700 1 -- 172.21.15.16:0/1944812969 <== mgr.13935 v2:172.21.15.16:6832/938897590 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fa8380c7ce0 con 0x7fa82406b020 2023-10-16T13:49:38.810 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.807+0000 7fa83da2c700 1 -- 172.21.15.16:0/1944812969 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7fa82406b020 msgr2=0x7fa82406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:38.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.807+0000 7fa83da2c700 1 --2- 172.21.15.16:0/1944812969 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7fa82406b020 0x7fa82406d4e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa82c0090b0 tx=0x7fa82c01c040 comp rx=0 tx=0).stop 2023-10-16T13:49:38.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.807+0000 7fa83da2c700 1 -- 172.21.15.16:0/1944812969 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa83815da40 msgr2=0x7fa8380b6600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:38.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.807+0000 7fa83da2c700 1 --2- 172.21.15.16:0/1944812969 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa83815da40 0x7fa8380b6600 secure :-1 s=READY pgs=3341 cs=0 l=1 rev1=1 crypto rx=0x7fa83000cc20 tx=0x7fa83000cf30 comp rx=0 tx=0).stop 2023-10-16T13:49:38.811 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.807+0000 7fa83da2c700 1 -- 172.21.15.16:0/1944812969 shutdown_connections 2023-10-16T13:49:38.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.807+0000 7fa83da2c700 1 --2- 172.21.15.16:0/1944812969 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7fa82406b020 0x7fa82406d4e0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:38.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.807+0000 7fa83da2c700 1 --2- 172.21.15.16:0/1944812969 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa83815da40 0x7fa8380b6600 unknown :-1 s=CLOSED pgs=3341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:38.812 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.807+0000 7fa83da2c700 1 --2- 172.21.15.16:0/1944812969 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa83806c3e0 0x7fa83815c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:38.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.807+0000 7fa83da2c700 1 --2- 172.21.15.16:0/1944812969 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fa8380b34c0 0x7fa8380b60c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:38.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.807+0000 7fa83da2c700 1 -- 172.21.15.16:0/1944812969 >> 172.21.15.16:0/1944812969 conn(0x7fa8380cdf80 msgr2=0x7fa8380b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:38.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.808+0000 7fa83da2c700 1 -- 172.21.15.16:0/1944812969 shutdown_connections 2023-10-16T13:49:38.813 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.808+0000 7fa83da2c700 1 -- 172.21.15.16:0/1944812969 wait complete. 2023-10-16T13:49:38.814 INFO:teuthology.orchestra.run.smithi016.stderr:instructed osd(s) 1 to deep-scrub 2023-10-16T13:49:38.822 INFO:tasks.ceph:Scrubbing osd.2 2023-10-16T13:49:38.823 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-10-16T13:49:38.952 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.950+0000 7f873f3cb700 1 Processor -- start 2023-10-16T13:49:38.953 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.950+0000 7f873f3cb700 1 -- start start 2023-10-16T13:49:38.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.951+0000 7f873f3cb700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f8738160e60 0x7f873815e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:38.955 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.951+0000 7f873f3cb700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f873806d980 0x7f873815efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:38.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.951+0000 7f873f3cb700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f873806a7e0 0x7f873806ac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:38.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.951+0000 7f873f3cb700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87380cdeb0 con 0x7f873806d980 2023-10-16T13:49:38.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.951+0000 7f873f3cb700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87381613e0 con 0x7f8738160e60 2023-10-16T13:49:38.956 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.951+0000 7f873f3cb700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f873806b160 con 0x7f873806a7e0 2023-10-16T13:49:38.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.951+0000 7f873c966700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f873806d980 0x7f873815efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:38.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.951+0000 7f873c966700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f873806d980 0x7f873815efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60468/0 (socket says 172.21.15.16:60468) 2023-10-16T13:49:38.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.951+0000 7f873c966700 1 -- 172.21.15.16:0/2454389013 learned_addr learned my addr 172.21.15.16:0/2454389013 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:38.957 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.951+0000 7f873d167700 1 --2- 172.21.15.16:0/2454389013 >> v2:172.21.15.49:3300/0 conn(0x7f8738160e60 0x7f873815e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:38.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.951+0000 7f873d968700 1 --2- 172.21.15.16:0/2454389013 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f873806a7e0 0x7f873806ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:38.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.952+0000 7f873c966700 1 -- 172.21.15.16:0/2454389013 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f873806a7e0 msgr2=0x7f873806ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:38.958 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.952+0000 7f873c966700 1 --2- 172.21.15.16:0/2454389013 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f873806a7e0 0x7f873806ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:38.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.952+0000 7f873c966700 1 -- 172.21.15.16:0/2454389013 >> v2:172.21.15.49:3300/0 conn(0x7f8738160e60 msgr2=0x7f873815e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:38.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.952+0000 7f873c966700 1 --2- 172.21.15.16:0/2454389013 >> v2:172.21.15.49:3300/0 conn(0x7f8738160e60 0x7f873815e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:38.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.952+0000 7f873c966700 1 -- 172.21.15.16:0/2454389013 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87381582d0 con 0x7f873806d980 2023-10-16T13:49:38.959 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.952+0000 7f873d167700 1 --2- 172.21.15.16:0/2454389013 >> v2:172.21.15.49:3300/0 conn(0x7f8738160e60 0x7f873815e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:49:38.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.952+0000 7f873c966700 1 --2- 172.21.15.16:0/2454389013 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f873806d980 0x7f873815efb0 secure :-1 s=READY pgs=3342 cs=0 l=1 rev1=1 crypto rx=0x7f8738073050 tx=0x7f8728009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17edc69417597cfa server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:38.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.952+0000 7f872ffff700 1 -- 172.21.15.16:0/2454389013 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f872800b870 con 0x7f873806d980 2023-10-16T13:49:38.960 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.952+0000 7f872ffff700 1 -- 172.21.15.16:0/2454389013 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f872800b9d0 con 0x7f873806d980 2023-10-16T13:49:38.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.952+0000 7f872ffff700 1 -- 172.21.15.16:0/2454389013 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f872800bd60 con 0x7f873806d980 2023-10-16T13:49:38.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.952+0000 7f873f3cb700 1 -- 172.21.15.16:0/2454389013 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f873806d980 msgr2=0x7f873815efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:38.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.952+0000 7f873f3cb700 1 --2- 172.21.15.16:0/2454389013 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f873806d980 0x7f873815efb0 secure :-1 s=READY pgs=3342 cs=0 l=1 rev1=1 crypto rx=0x7f8738073050 tx=0x7f8728009bd0 comp rx=0 tx=0).stop 2023-10-16T13:49:38.961 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.952+0000 7f873f3cb700 1 -- 172.21.15.16:0/2454389013 shutdown_connections 2023-10-16T13:49:38.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.952+0000 7f873f3cb700 1 --2- 172.21.15.16:0/2454389013 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f873806a7e0 0x7f873806ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:38.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.952+0000 7f873f3cb700 1 --2- 172.21.15.16:0/2454389013 >> v2:172.21.15.49:3300/0 conn(0x7f8738160e60 0x7f873815e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:38.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.952+0000 7f873f3cb700 1 --2- 172.21.15.16:0/2454389013 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f873806d980 0x7f873815efb0 unknown :-1 s=CLOSED pgs=3342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:38.962 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.952+0000 7f873f3cb700 1 -- 172.21.15.16:0/2454389013 >> 172.21.15.16:0/2454389013 conn(0x7f873806f430 msgr2=0x7f873806f810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:38.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.953+0000 7f873f3cb700 1 -- 172.21.15.16:0/2454389013 shutdown_connections 2023-10-16T13:49:38.963 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.953+0000 7f873f3cb700 1 -- 172.21.15.16:0/2454389013 wait complete. 2023-10-16T13:49:38.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.954+0000 7f873f3cb700 1 Processor -- start 2023-10-16T13:49:38.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.954+0000 7f873f3cb700 1 -- start start 2023-10-16T13:49:38.966 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.954+0000 7f873f3cb700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f873806a7e0 0x7f87380b3df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:38.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.954+0000 7f873f3cb700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f873806d980 0x7f873814e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:38.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.954+0000 7f873f3cb700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8738160e60 0x7f87380b5180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:38.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.954+0000 7f873f3cb700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87380b39d0 con 0x7f8738160e60 2023-10-16T13:49:38.967 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.954+0000 7f873f3cb700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87380b5c40 con 0x7f873806d980 2023-10-16T13:49:38.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.954+0000 7f873f3cb700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87380b4330 con 0x7f873806a7e0 2023-10-16T13:49:38.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.955+0000 7f873d968700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8738160e60 0x7f87380b5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:38.968 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.955+0000 7f873d968700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8738160e60 0x7f87380b5180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60476/0 (socket says 172.21.15.16:60476) 2023-10-16T13:49:38.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.955+0000 7f873d968700 1 -- 172.21.15.16:0/915220305 learned_addr learned my addr 172.21.15.16:0/915220305 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:38.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.955+0000 7f873d167700 1 --2- 172.21.15.16:0/915220305 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f873806a7e0 0x7f87380b3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:38.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.955+0000 7f873c966700 1 --2- 172.21.15.16:0/915220305 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f873806d980 0x7f873814e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:38.969 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.955+0000 7f873d968700 1 -- 172.21.15.16:0/915220305 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f873806a7e0 msgr2=0x7f87380b3df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:38.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.955+0000 7f873d968700 1 --2- 172.21.15.16:0/915220305 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f873806a7e0 0x7f87380b3df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:38.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.955+0000 7f873d968700 1 -- 172.21.15.16:0/915220305 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f873806d980 msgr2=0x7f873814e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:38.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.955+0000 7f873d968700 1 --2- 172.21.15.16:0/915220305 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f873806d980 0x7f873814e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:38.970 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.955+0000 7f873d968700 1 -- 172.21.15.16:0/915220305 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8728009550 con 0x7f8738160e60 2023-10-16T13:49:38.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.955+0000 7f873d167700 1 --2- 172.21.15.16:0/915220305 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f873806a7e0 0x7f87380b3df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:49:38.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.955+0000 7f873d968700 1 --2- 172.21.15.16:0/915220305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8738160e60 0x7f87380b5180 secure :-1 s=READY pgs=3343 cs=0 l=1 rev1=1 crypto rx=0x7f8730007e60 tx=0x7f873000e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:38.971 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.955+0000 7f872dffb700 1 -- 172.21.15.16:0/915220305 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f873000ede0 con 0x7f8738160e60 2023-10-16T13:49:38.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.955+0000 7f872dffb700 1 -- 172.21.15.16:0/915220305 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8730014420 con 0x7f8738160e60 2023-10-16T13:49:38.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.955+0000 7f873f3cb700 1 -- 172.21.15.16:0/915220305 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87380b3030 con 0x7f8738160e60 2023-10-16T13:49:38.972 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.955+0000 7f872dffb700 1 -- 172.21.15.16:0/915220305 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87300147d0 con 0x7f8738160e60 2023-10-16T13:49:38.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.955+0000 7f873f3cb700 1 -- 172.21.15.16:0/915220305 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87380b35f0 con 0x7f8738160e60 2023-10-16T13:49:38.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.956+0000 7f873f3cb700 1 -- 172.21.15.16:0/915220305 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f87380ce100 con 0x7f8738160e60 2023-10-16T13:49:38.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.957+0000 7f872dffb700 1 -- 172.21.15.16:0/915220305 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 176) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8730014930 con 0x7f8738160e60 2023-10-16T13:49:38.973 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.957+0000 7f872dffb700 1 --2- 172.21.15.16:0/915220305 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f871c06b020 0x7f871c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:38.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.957+0000 7f872dffb700 1 -- 172.21.15.16:0/915220305 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(283..283 src has 1..283) v4 ==== 6208+0+0 (secure 0 0 0) 0x7f8730089f80 con 0x7f8738160e60 2023-10-16T13:49:38.974 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.957+0000 7f873d167700 1 --2- 172.21.15.16:0/915220305 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f871c06b020 0x7f871c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:38.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.957+0000 7f872dffb700 1 --2- 172.21.15.16:0/915220305 >> [v2:172.21.15.16:6808/47933305,v1:172.21.15.16:6809/47933305] conn(0x7f871c071ac0 0x7f871c073f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:38.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.957+0000 7f872dffb700 1 -- 172.21.15.16:0/915220305 --> [v2:172.21.15.16:6808/47933305,v1:172.21.15.16:6809/47933305] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f871c074630 con 0x7f871c071ac0 2023-10-16T13:49:38.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.957+0000 7f872dffb700 1 -- 172.21.15.16:0/915220305 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=283) v2 ==== 24+0+0 (secure 0 0 0) 0x7f873008c5c0 con 0x7f8738160e60 2023-10-16T13:49:38.975 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.957+0000 7f873c966700 1 --2- 172.21.15.16:0/915220305 >> [v2:172.21.15.16:6808/47933305,v1:172.21.15.16:6809/47933305] conn(0x7f871c071ac0 0x7f871c073f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:38.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.958+0000 7f873d167700 1 --2- 172.21.15.16:0/915220305 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f871c06b020 0x7f871c06d4e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f873400c960 tx=0x7f8734008040 comp rx=0 tx=0).ready entity=mgr.13935 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:38.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.958+0000 7f873c966700 1 --2- 172.21.15.16:0/915220305 >> [v2:172.21.15.16:6808/47933305,v1:172.21.15.16:6809/47933305] conn(0x7f871c071ac0 0x7f871c073f60 crc :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:38.976 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.958+0000 7f872dffb700 1 -- 172.21.15.16:0/915220305 <== osd.2 v2:172.21.15.16:6808/47933305 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7f871c074630 con 0x7f871c071ac0 2023-10-16T13:49:38.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.973+0000 7f873f3cb700 1 -- 172.21.15.16:0/915220305 --> [v2:172.21.15.16:6808/47933305,v1:172.21.15.16:6809/47933305] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f873815edb0 con 0x7f871c071ac0 2023-10-16T13:49:38.977 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.973+0000 7f872dffb700 1 -- 172.21.15.16:0/915220305 <== osd.2 v2:172.21.15.16:6808/47933305 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f872800bb40 con 0x7f871c071ac0 2023-10-16T13:49:38.978 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-10-16T13:49:38.978 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-16T13:49:38.978 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-10-16T13:49:38.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.973+0000 7f873f3cb700 1 -- 172.21.15.16:0/915220305 >> [v2:172.21.15.16:6808/47933305,v1:172.21.15.16:6809/47933305] conn(0x7f871c071ac0 msgr2=0x7f871c073f60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:38.979 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.973+0000 7f873f3cb700 1 --2- 172.21.15.16:0/915220305 >> [v2:172.21.15.16:6808/47933305,v1:172.21.15.16:6809/47933305] conn(0x7f871c071ac0 0x7f871c073f60 crc :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:38.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.974+0000 7f873f3cb700 1 -- 172.21.15.16:0/915220305 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f871c06b020 msgr2=0x7f871c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:38.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.974+0000 7f873f3cb700 1 --2- 172.21.15.16:0/915220305 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f871c06b020 0x7f871c06d4e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f873400c960 tx=0x7f8734008040 comp rx=0 tx=0).stop 2023-10-16T13:49:38.980 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.974+0000 7f873f3cb700 1 -- 172.21.15.16:0/915220305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8738160e60 msgr2=0x7f87380b5180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:38.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.974+0000 7f873f3cb700 1 --2- 172.21.15.16:0/915220305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8738160e60 0x7f87380b5180 secure :-1 s=READY pgs=3343 cs=0 l=1 rev1=1 crypto rx=0x7f8730007e60 tx=0x7f873000e3c0 comp rx=0 tx=0).stop 2023-10-16T13:49:38.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.974+0000 7f873d968700 1 -- 172.21.15.16:0/915220305 reap_dead start 2023-10-16T13:49:38.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.974+0000 7f873f3cb700 1 -- 172.21.15.16:0/915220305 shutdown_connections 2023-10-16T13:49:38.981 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.974+0000 7f873f3cb700 1 -- 172.21.15.16:0/915220305 >> 172.21.15.16:0/915220305 conn(0x7f873806f430 msgr2=0x7f87381508a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:38.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.974+0000 7f873f3cb700 1 -- 172.21.15.16:0/915220305 shutdown_connections 2023-10-16T13:49:38.982 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:38.974+0000 7f873f3cb700 1 -- 172.21.15.16:0/915220305 wait complete. 2023-10-16T13:49:38.988 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-10-16T13:49:39.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.111+0000 7f9ec683f700 1 Processor -- start 2023-10-16T13:49:39.114 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.111+0000 7f9ec683f700 1 -- start start 2023-10-16T13:49:39.116 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.112+0000 7f9ec683f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9ec015b280 0x7f9ec01613e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:39.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.112+0000 7f9ec683f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9ec00cc8a0 0x7f9ec015e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:39.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.112+0000 7f9ec683f700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ec0161c30 con 0x7f9ec006a7e0 2023-10-16T13:49:39.117 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.112+0000 7f9ec683f700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ec0161f10 con 0x7f9ec015b280 2023-10-16T13:49:39.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.112+0000 7f9ec683f700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec01621f0 con 0x7f9ec00cc8a0 2023-10-16T13:49:39.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.112+0000 7f9ebffff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9ec015b280 0x7f9ec01613e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:39.118 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.112+0000 7f9ec4ddc700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f9ec006a7e0 0x7f9ec00d02b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49760/0 (socket says 172.21.15.16:49760) 2023-10-16T13:49:39.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.112+0000 7f9ec4ddc700 1 -- 172.21.15.16:0/2243361432 learned_addr learned my addr 172.21.15.16:0/2243361432 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:49:39.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.112+0000 7f9ebf7fe700 1 --2- 172.21.15.16:0/2243361432 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9ec00cc8a0 0x7f9ec015e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:39.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.113+0000 7f9ebffff700 1 -- 172.21.15.16:0/2243361432 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9ec00cc8a0 msgr2=0x7f9ec015e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:39.119 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.113+0000 7f9ebffff700 1 --2- 172.21.15.16:0/2243361432 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9ec00cc8a0 0x7f9ec015e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:39.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.113+0000 7f9ebffff700 1 -- 172.21.15.16:0/2243361432 >> v1:172.21.15.16:6789/0 conn(0x7f9ec006a7e0 legacy=0x7f9ec00d02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:39.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.113+0000 7f9ebffff700 1 -- 172.21.15.16:0/2243361432 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ec015df00 con 0x7f9ec015b280 2023-10-16T13:49:39.120 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.113+0000 7f9ebeffd700 1 -- 172.21.15.16:0/2243361432 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 332993468 0 0) 0x7f9ec0161c30 con 0x7f9ec006a7e0 2023-10-16T13:49:39.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.113+0000 7f9ebffff700 1 --2- 172.21.15.16:0/2243361432 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ec015b280 0x7f9ec01613e0 secure :-1 s=READY pgs=3345 cs=0 l=1 rev1=1 crypto rx=0x7f9eb0009840 tx=0x7f9eb0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13da223edc94793a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:39.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.113+0000 7f9ebeffd700 1 -- 172.21.15.16:0/2243361432 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eb0011830 con 0x7f9ec015b280 2023-10-16T13:49:39.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.113+0000 7f9ebeffd700 1 -- 172.21.15.16:0/2243361432 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9eb0011990 con 0x7f9ec015b280 2023-10-16T13:49:39.121 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.113+0000 7f9ebeffd700 1 -- 172.21.15.16:0/2243361432 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eb0011d20 con 0x7f9ec015b280 2023-10-16T13:49:39.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.113+0000 7f9ec683f700 1 -- 172.21.15.16:0/2243361432 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ec015b280 msgr2=0x7f9ec01613e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:39.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.113+0000 7f9ec683f700 1 --2- 172.21.15.16:0/2243361432 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ec015b280 0x7f9ec01613e0 secure :-1 s=READY pgs=3345 cs=0 l=1 rev1=1 crypto rx=0x7f9eb0009840 tx=0x7f9eb0009b50 comp rx=0 tx=0).stop 2023-10-16T13:49:39.122 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.113+0000 7f9ec683f700 1 -- 172.21.15.16:0/2243361432 shutdown_connections 2023-10-16T13:49:39.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.113+0000 7f9ec683f700 1 --2- 172.21.15.16:0/2243361432 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ec015b280 0x7f9ec01613e0 unknown :-1 s=CLOSED pgs=3345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:39.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.113+0000 7f9ec683f700 1 --2- 172.21.15.16:0/2243361432 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9ec00cc8a0 0x7f9ec015e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:39.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.113+0000 7f9ec683f700 1 -- 172.21.15.16:0/2243361432 >> 172.21.15.16:0/2243361432 conn(0x7f9ec00b1fd0 msgr2=0x7f9ec00b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:39.123 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.113+0000 7f9ec683f700 1 -- 172.21.15.16:0/2243361432 shutdown_connections 2023-10-16T13:49:39.124 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.114+0000 7f9ec683f700 1 -- 172.21.15.16:0/2243361432 wait complete. 2023-10-16T13:49:39.126 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.115+0000 7f9ec683f700 1 Processor -- start 2023-10-16T13:49:39.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.115+0000 7f9ec683f700 1 -- start start 2023-10-16T13:49:39.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.115+0000 7f9ec683f700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9ec006a7e0 0x7f9ec00c7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:39.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.116+0000 7f9ec683f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ec00cc8a0 0x7f9ec01f6e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:39.127 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.116+0000 7f9ec683f700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9ec015b280 0x7f9ec0152280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:39.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.116+0000 7f9ec683f700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec01f7360 con 0x7f9ec00cc8a0 2023-10-16T13:49:39.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.116+0000 7f9ec683f700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec0155ec0 con 0x7f9ec006a7e0 2023-10-16T13:49:39.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.116+0000 7f9ec683f700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec01527c0 con 0x7f9ec015b280 2023-10-16T13:49:39.128 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.116+0000 7f9ebf7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ec00cc8a0 0x7f9ec01f6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:39.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.116+0000 7f9ebf7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ec00cc8a0 0x7f9ec01f6e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60490/0 (socket says 172.21.15.16:60490) 2023-10-16T13:49:39.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.116+0000 7f9ebf7fe700 1 -- 172.21.15.16:0/198941160 learned_addr learned my addr 172.21.15.16:0/198941160 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:39.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.116+0000 7f9ebffff700 1 --2- 172.21.15.16:0/198941160 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9ec006a7e0 0x7f9ec00c7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:39.129 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.116+0000 7f9ec4ddc700 1 --2- 172.21.15.16:0/198941160 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9ec015b280 0x7f9ec0152280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:39.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.116+0000 7f9ebf7fe700 1 -- 172.21.15.16:0/198941160 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9ec015b280 msgr2=0x7f9ec0152280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:39.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.116+0000 7f9ebf7fe700 1 --2- 172.21.15.16:0/198941160 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9ec015b280 0x7f9ec0152280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:39.130 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.116+0000 7f9ebf7fe700 1 -- 172.21.15.16:0/198941160 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9ec006a7e0 msgr2=0x7f9ec00c7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:39.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.116+0000 7f9ebf7fe700 1 --2- 172.21.15.16:0/198941160 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9ec006a7e0 0x7f9ec00c7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:39.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.116+0000 7f9ebf7fe700 1 -- 172.21.15.16:0/198941160 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eb00094a0 con 0x7f9ec00cc8a0 2023-10-16T13:49:39.131 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.116+0000 7f9ebf7fe700 1 --2- 172.21.15.16:0/198941160 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ec00cc8a0 0x7f9ec01f6e20 secure :-1 s=READY pgs=3346 cs=0 l=1 rev1=1 crypto rx=0x7f9ea800e9e0 tx=0x7f9ea800ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:39.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.116+0000 7f9ebcff9700 1 -- 172.21.15.16:0/198941160 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ea8009710 con 0x7f9ec00cc8a0 2023-10-16T13:49:39.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.116+0000 7f9ebcff9700 1 -- 172.21.15.16:0/198941160 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ea800ee80 con 0x7f9ec00cc8a0 2023-10-16T13:49:39.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.116+0000 7f9ec683f700 1 -- 172.21.15.16:0/198941160 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ec00c82e0 con 0x7f9ec00cc8a0 2023-10-16T13:49:39.132 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.116+0000 7f9ebcff9700 1 -- 172.21.15.16:0/198941160 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ea800ca50 con 0x7f9ec00cc8a0 2023-10-16T13:49:39.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.116+0000 7f9ec683f700 1 -- 172.21.15.16:0/198941160 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ec00c88a0 con 0x7f9ec00cc8a0 2023-10-16T13:49:39.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.117+0000 7f9ec683f700 1 -- 172.21.15.16:0/198941160 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ec004ebb0 con 0x7f9ec00cc8a0 2023-10-16T13:49:39.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.118+0000 7f9ebcff9700 1 -- 172.21.15.16:0/198941160 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 176) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9ea800cbb0 con 0x7f9ec00cc8a0 2023-10-16T13:49:39.133 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.118+0000 7f9ebcff9700 1 --2- 172.21.15.16:0/198941160 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f9ea006b020 0x7f9ea006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:39.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.118+0000 7f9ebcff9700 1 -- 172.21.15.16:0/198941160 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(283..283 src has 1..283) v4 ==== 6208+0+0 (secure 0 0 0) 0x7f9ea8089910 con 0x7f9ec00cc8a0 2023-10-16T13:49:39.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.118+0000 7f9ebffff700 1 --2- 172.21.15.16:0/198941160 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f9ea006b020 0x7f9ea006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:39.134 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.119+0000 7f9ebffff700 1 --2- 172.21.15.16:0/198941160 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f9ea006b020 0x7f9ea006d4e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9eb0000c00 tx=0x7f9eb001c040 comp rx=0 tx=0).ready entity=mgr.13935 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:39.135 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.123+0000 7f9ebcff9700 1 -- 172.21.15.16:0/198941160 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9ea80546d0 con 0x7f9ec00cc8a0 2023-10-16T13:49:39.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.327+0000 7f9ec683f700 1 -- 172.21.15.16:0/198941160 --> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7f9ec0161920 con 0x7f9ea006b020 2023-10-16T13:49:39.330 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.327+0000 7f9ebcff9700 1 -- 172.21.15.16:0/198941160 <== mgr.13935 v2:172.21.15.16:6832/938897590 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f9eb0011b80 con 0x7f9ea006b020 2023-10-16T13:49:39.332 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.329+0000 7f9ec683f700 1 -- 172.21.15.16:0/198941160 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f9ea006b020 msgr2=0x7f9ea006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:39.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.329+0000 7f9ec683f700 1 --2- 172.21.15.16:0/198941160 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f9ea006b020 0x7f9ea006d4e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9eb0000c00 tx=0x7f9eb001c040 comp rx=0 tx=0).stop 2023-10-16T13:49:39.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.329+0000 7f9ec683f700 1 -- 172.21.15.16:0/198941160 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ec00cc8a0 msgr2=0x7f9ec01f6e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:39.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.329+0000 7f9ec683f700 1 --2- 172.21.15.16:0/198941160 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ec00cc8a0 0x7f9ec01f6e20 secure :-1 s=READY pgs=3346 cs=0 l=1 rev1=1 crypto rx=0x7f9ea800e9e0 tx=0x7f9ea800ecf0 comp rx=0 tx=0).stop 2023-10-16T13:49:39.333 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.330+0000 7f9ec683f700 1 -- 172.21.15.16:0/198941160 shutdown_connections 2023-10-16T13:49:39.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.330+0000 7f9ec683f700 1 --2- 172.21.15.16:0/198941160 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f9ea006b020 0x7f9ea006d4e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:39.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.330+0000 7f9ec683f700 1 --2- 172.21.15.16:0/198941160 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f9ec015b280 0x7f9ec0152280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:39.334 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.330+0000 7f9ec683f700 1 --2- 172.21.15.16:0/198941160 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9ec006a7e0 0x7f9ec00c7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:39.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.330+0000 7f9ec683f700 1 --2- 172.21.15.16:0/198941160 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ec00cc8a0 0x7f9ec01f6e20 unknown :-1 s=CLOSED pgs=3346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:39.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.330+0000 7f9ec683f700 1 -- 172.21.15.16:0/198941160 >> 172.21.15.16:0/198941160 conn(0x7f9ec00b1fd0 msgr2=0x7f9ec00719a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:39.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.330+0000 7f9ec683f700 1 -- 172.21.15.16:0/198941160 shutdown_connections 2023-10-16T13:49:39.335 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.330+0000 7f9ec683f700 1 -- 172.21.15.16:0/198941160 wait complete. 2023-10-16T13:49:39.336 INFO:teuthology.orchestra.run.smithi016.stderr:instructed osd(s) 2 to deep-scrub 2023-10-16T13:49:39.344 INFO:tasks.ceph:Scrubbing osd.3 2023-10-16T13:49:39.345 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2023-10-16T13:49:39.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.472+0000 7faf25ce6700 1 Processor -- start 2023-10-16T13:49:39.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.472+0000 7faf25ce6700 1 -- start start 2023-10-16T13:49:39.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.472+0000 7faf25ce6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7faf200cf2c0 0x7faf200aef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:39.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.473+0000 7faf25ce6700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7faf200cbde0 0x7faf200c5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:39.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.473+0000 7faf25ce6700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf200c49b0 con 0x7faf200cd600 2023-10-16T13:49:39.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.473+0000 7faf25ce6700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf200c4c20 con 0x7faf200cf2c0 2023-10-16T13:49:39.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.473+0000 7faf25ce6700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf200c4f00 con 0x7faf200cbde0 2023-10-16T13:49:39.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.473+0000 7faf1f7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7faf200cf2c0 0x7faf200aef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:39.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.473+0000 7faf1f7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7faf200cf2c0 0x7faf200aef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60496/0 (socket says 172.21.15.16:60496) 2023-10-16T13:49:39.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.473+0000 7faf1f7fe700 1 -- 172.21.15.16:0/18877051 learned_addr learned my addr 172.21.15.16:0/18877051 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:39.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.473+0000 7faf1ffff700 1 --2- 172.21.15.16:0/18877051 >> v2:172.21.15.49:3300/0 conn(0x7faf200cbde0 0x7faf200c5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:39.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.473+0000 7faf1f7fe700 1 -- 172.21.15.16:0/18877051 >> v2:172.21.15.49:3300/0 conn(0x7faf200cbde0 msgr2=0x7faf200c5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:39.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.473+0000 7faf1f7fe700 1 --2- 172.21.15.16:0/18877051 >> v2:172.21.15.49:3300/0 conn(0x7faf200cbde0 0x7faf200c5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:39.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.473+0000 7faf1f7fe700 1 -- 172.21.15.16:0/18877051 >> v1:172.21.15.49:6789/0 conn(0x7faf200cd600 legacy=0x7faf200cb6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:39.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.473+0000 7faf1f7fe700 1 -- 172.21.15.16:0/18877051 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf200c51e0 con 0x7faf200cf2c0 2023-10-16T13:49:39.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.473+0000 7faf1f7fe700 1 --2- 172.21.15.16:0/18877051 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf200cf2c0 0x7faf200aef00 secure :-1 s=READY pgs=3347 cs=0 l=1 rev1=1 crypto rx=0x7faf200cd560 tx=0x7faf1000be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7dbc4b17b30692e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:39.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.473+0000 7faf1e7fc700 1 -- 172.21.15.16:0/18877051 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf1000dc70 con 0x7faf200cf2c0 2023-10-16T13:49:39.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.473+0000 7faf1e7fc700 1 -- 172.21.15.16:0/18877051 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf1000ddd0 con 0x7faf200cf2c0 2023-10-16T13:49:39.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.473+0000 7faf1e7fc700 1 -- 172.21.15.16:0/18877051 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf10010690 con 0x7faf200cf2c0 2023-10-16T13:49:39.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.474+0000 7faf25ce6700 1 -- 172.21.15.16:0/18877051 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf200cf2c0 msgr2=0x7faf200aef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:39.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.474+0000 7faf25ce6700 1 --2- 172.21.15.16:0/18877051 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf200cf2c0 0x7faf200aef00 secure :-1 s=READY pgs=3347 cs=0 l=1 rev1=1 crypto rx=0x7faf200cd560 tx=0x7faf1000be70 comp rx=0 tx=0).stop 2023-10-16T13:49:39.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.474+0000 7faf25ce6700 1 -- 172.21.15.16:0/18877051 shutdown_connections 2023-10-16T13:49:39.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.474+0000 7faf25ce6700 1 --2- 172.21.15.16:0/18877051 >> v2:172.21.15.49:3300/0 conn(0x7faf200cbde0 0x7faf200c5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:39.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.474+0000 7faf25ce6700 1 --2- 172.21.15.16:0/18877051 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf200cf2c0 0x7faf200aef00 unknown :-1 s=CLOSED pgs=3347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:39.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.474+0000 7faf25ce6700 1 -- 172.21.15.16:0/18877051 >> 172.21.15.16:0/18877051 conn(0x7faf200cc4b0 msgr2=0x7faf200cc8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:39.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.474+0000 7faf25ce6700 1 -- 172.21.15.16:0/18877051 shutdown_connections 2023-10-16T13:49:39.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.474+0000 7faf25ce6700 1 -- 172.21.15.16:0/18877051 wait complete. 2023-10-16T13:49:39.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.475+0000 7faf25ce6700 1 Processor -- start 2023-10-16T13:49:39.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.476+0000 7faf25ce6700 1 -- start start 2023-10-16T13:49:39.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.476+0000 7faf25ce6700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faf200cbde0 0x7faf20151680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:39.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.476+0000 7faf25ce6700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faf200cd600 0x7faf200b3120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:39.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.476+0000 7faf25ce6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf200cf2c0 0x7faf2015d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:39.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.476+0000 7faf25ce6700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf200b4210 con 0x7faf200cf2c0 2023-10-16T13:49:39.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.476+0000 7faf25ce6700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf200c7840 con 0x7faf200cbde0 2023-10-16T13:49:39.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.476+0000 7faf25ce6700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf200b3970 con 0x7faf200cd600 2023-10-16T13:49:39.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.476+0000 7faf1ffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf200cf2c0 0x7faf2015d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:39.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.476+0000 7faf1f7fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faf200cbde0 0x7faf20151680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:39.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.476+0000 7faf1ffff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf200cf2c0 0x7faf2015d590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60506/0 (socket says 172.21.15.16:60506) 2023-10-16T13:49:39.489 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.477+0000 7faf1ffff700 1 -- 172.21.15.16:0/1437343326 learned_addr learned my addr 172.21.15.16:0/1437343326 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:39.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.477+0000 7faf1effd700 1 --2- 172.21.15.16:0/1437343326 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faf200cd600 0x7faf200b3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:39.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.477+0000 7faf1ffff700 1 -- 172.21.15.16:0/1437343326 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faf200cd600 msgr2=0x7faf200b3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:39.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.477+0000 7faf1ffff700 1 --2- 172.21.15.16:0/1437343326 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7faf200cd600 0x7faf200b3120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:39.490 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.477+0000 7faf1ffff700 1 -- 172.21.15.16:0/1437343326 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faf200cbde0 msgr2=0x7faf20151680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:39.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.477+0000 7faf1ffff700 1 --2- 172.21.15.16:0/1437343326 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faf200cbde0 0x7faf20151680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:39.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.477+0000 7faf1ffff700 1 -- 172.21.15.16:0/1437343326 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf1000b7f0 con 0x7faf200cf2c0 2023-10-16T13:49:39.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.477+0000 7faf1f7fe700 1 --2- 172.21.15.16:0/1437343326 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faf200cbde0 0x7faf20151680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:49:39.491 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.477+0000 7faf1ffff700 1 --2- 172.21.15.16:0/1437343326 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf200cf2c0 0x7faf2015d590 secure :-1 s=READY pgs=3348 cs=0 l=1 rev1=1 crypto rx=0x7faf18009dd0 tx=0x7faf1800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:39.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.477+0000 7faf07fff700 1 -- 172.21.15.16:0/1437343326 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf18009c40 con 0x7faf200cf2c0 2023-10-16T13:49:39.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.477+0000 7faf07fff700 1 -- 172.21.15.16:0/1437343326 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf1800f040 con 0x7faf200cf2c0 2023-10-16T13:49:39.492 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.477+0000 7faf25ce6700 1 -- 172.21.15.16:0/1437343326 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf200b3c50 con 0x7faf200cf2c0 2023-10-16T13:49:39.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.477+0000 7faf07fff700 1 -- 172.21.15.16:0/1437343326 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf1800ceb0 con 0x7faf200cf2c0 2023-10-16T13:49:39.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.477+0000 7faf25ce6700 1 -- 172.21.15.16:0/1437343326 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf200b3660 con 0x7faf200cf2c0 2023-10-16T13:49:39.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.478+0000 7faf25ce6700 1 -- 172.21.15.16:0/1437343326 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7faf200cdab0 con 0x7faf200cf2c0 2023-10-16T13:49:39.493 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.479+0000 7faf07fff700 1 -- 172.21.15.16:0/1437343326 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 176) v1 ==== 89331+0+0 (secure 0 0 0) 0x7faf18014480 con 0x7faf200cf2c0 2023-10-16T13:49:39.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.479+0000 7faf07fff700 1 --2- 172.21.15.16:0/1437343326 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7faf0c06b020 0x7faf0c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:39.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.479+0000 7faf07fff700 1 -- 172.21.15.16:0/1437343326 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(283..283 src has 1..283) v4 ==== 6208+0+0 (secure 0 0 0) 0x7faf1808a8f0 con 0x7faf200cf2c0 2023-10-16T13:49:39.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.479+0000 7faf1f7fe700 1 --2- 172.21.15.16:0/1437343326 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7faf0c06b020 0x7faf0c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:39.494 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.479+0000 7faf07fff700 1 --2- 172.21.15.16:0/1437343326 >> [v2:172.21.15.16:6817/375531758,v1:172.21.15.16:6819/375531758] conn(0x7faf0c071ac0 0x7faf0c073f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:39.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.479+0000 7faf07fff700 1 -- 172.21.15.16:0/1437343326 --> [v2:172.21.15.16:6817/375531758,v1:172.21.15.16:6819/375531758] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7faf0c074630 con 0x7faf0c071ac0 2023-10-16T13:49:39.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.479+0000 7faf07fff700 1 -- 172.21.15.16:0/1437343326 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=283) v2 ==== 24+0+0 (secure 0 0 0) 0x7faf1808ad20 con 0x7faf200cf2c0 2023-10-16T13:49:39.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.479+0000 7faf1effd700 1 --2- 172.21.15.16:0/1437343326 >> [v2:172.21.15.16:6817/375531758,v1:172.21.15.16:6819/375531758] conn(0x7faf0c071ac0 0x7faf0c073f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:39.495 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.480+0000 7faf1f7fe700 1 --2- 172.21.15.16:0/1437343326 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7faf0c06b020 0x7faf0c06d4e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7faf1000beb0 tx=0x7faf1001c040 comp rx=0 tx=0).ready entity=mgr.13935 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:39.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.480+0000 7faf1effd700 1 --2- 172.21.15.16:0/1437343326 >> [v2:172.21.15.16:6817/375531758,v1:172.21.15.16:6819/375531758] conn(0x7faf0c071ac0 0x7faf0c073f60 crc :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:39.496 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.480+0000 7faf07fff700 1 -- 172.21.15.16:0/1437343326 <== osd.3 v2:172.21.15.16:6817/375531758 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7faf0c074630 con 0x7faf0c071ac0 2023-10-16T13:49:39.497 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.495+0000 7faf25ce6700 1 -- 172.21.15.16:0/1437343326 --> [v2:172.21.15.16:6817/375531758,v1:172.21.15.16:6819/375531758] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7faf200cf6a0 con 0x7faf0c071ac0 2023-10-16T13:49:39.498 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.495+0000 7faf07fff700 1 -- 172.21.15.16:0/1437343326 <== osd.3 v2:172.21.15.16:6817/375531758 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7faf14001560 con 0x7faf0c071ac0 2023-10-16T13:49:39.498 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-10-16T13:49:39.499 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-16T13:49:39.499 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-10-16T13:49:39.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.496+0000 7faf25ce6700 1 -- 172.21.15.16:0/1437343326 >> [v2:172.21.15.16:6817/375531758,v1:172.21.15.16:6819/375531758] conn(0x7faf0c071ac0 msgr2=0x7faf0c073f60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:39.500 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.496+0000 7faf25ce6700 1 --2- 172.21.15.16:0/1437343326 >> [v2:172.21.15.16:6817/375531758,v1:172.21.15.16:6819/375531758] conn(0x7faf0c071ac0 0x7faf0c073f60 crc :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:39.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.496+0000 7faf25ce6700 1 -- 172.21.15.16:0/1437343326 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7faf0c06b020 msgr2=0x7faf0c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:39.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.496+0000 7faf25ce6700 1 --2- 172.21.15.16:0/1437343326 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7faf0c06b020 0x7faf0c06d4e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7faf1000beb0 tx=0x7faf1001c040 comp rx=0 tx=0).stop 2023-10-16T13:49:39.501 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.496+0000 7faf25ce6700 1 -- 172.21.15.16:0/1437343326 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf200cf2c0 msgr2=0x7faf2015d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:39.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.496+0000 7faf25ce6700 1 --2- 172.21.15.16:0/1437343326 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf200cf2c0 0x7faf2015d590 secure :-1 s=READY pgs=3348 cs=0 l=1 rev1=1 crypto rx=0x7faf18009dd0 tx=0x7faf1800c3a0 comp rx=0 tx=0).stop 2023-10-16T13:49:39.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.496+0000 7faf1ffff700 1 -- 172.21.15.16:0/1437343326 reap_dead start 2023-10-16T13:49:39.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.496+0000 7faf25ce6700 1 -- 172.21.15.16:0/1437343326 shutdown_connections 2023-10-16T13:49:39.502 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.496+0000 7faf25ce6700 1 -- 172.21.15.16:0/1437343326 >> 172.21.15.16:0/1437343326 conn(0x7faf200cc4b0 msgr2=0x7faf200ce660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:39.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.496+0000 7faf25ce6700 1 -- 172.21.15.16:0/1437343326 shutdown_connections 2023-10-16T13:49:39.503 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.496+0000 7faf25ce6700 1 -- 172.21.15.16:0/1437343326 wait complete. 2023-10-16T13:49:39.510 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2023-10-16T13:49:39.636 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.634+0000 7f7a06c5d700 1 Processor -- start 2023-10-16T13:49:39.637 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.634+0000 7f7a06c5d700 1 -- start start 2023-10-16T13:49:39.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.635+0000 7f7a06c5d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7a0015d900 0x7f7a000d2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:39.639 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.635+0000 7f7a06c5d700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a000cee70 con 0x7f7a000d2f70 2023-10-16T13:49:39.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.635+0000 7f7a06c5d700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a000ce400 con 0x7f7a000ce9f0 2023-10-16T13:49:39.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.635+0000 7f7a06c5d700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a0015dce0 con 0x7f7a0015d900 2023-10-16T13:49:39.640 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.635+0000 7f7a049f9700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7a0015d900 0x7f7a000d2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:39.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.635+0000 7f7a049f9700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7a0015d900 0x7f7a000d2040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60512/0 (socket says 172.21.15.16:60512) 2023-10-16T13:49:39.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.635+0000 7f7a051fa700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f7a000d2f70 0x7f7a000b6790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49790/0 (socket says 172.21.15.16:49790) 2023-10-16T13:49:39.641 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.635+0000 7f7a049f9700 1 -- 172.21.15.16:0/3286877221 learned_addr learned my addr 172.21.15.16:0/3286877221 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:39.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.635+0000 7f7a049f9700 1 -- 172.21.15.16:0/3286877221 >> v1:172.21.15.49:6789/0 conn(0x7f7a000ce9f0 legacy=0x7f7a000d2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:39.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.635+0000 7f7a049f9700 1 -- 172.21.15.16:0/3286877221 >> v1:172.21.15.16:6789/0 conn(0x7f7a000d2f70 legacy=0x7f7a000b6790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:39.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.635+0000 7f79ff7fe700 1 -- 172.21.15.16:0/3286877221 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 228002471 0 0) 0x7f7a000cee70 con 0x7f7a000d2f70 2023-10-16T13:49:39.642 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.635+0000 7f7a049f9700 1 -- 172.21.15.16:0/3286877221 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a000d1280 con 0x7f7a0015d900 2023-10-16T13:49:39.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.635+0000 7f7a049f9700 1 --2- 172.21.15.16:0/3286877221 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a0015d900 0x7f7a000d2040 secure :-1 s=READY pgs=3350 cs=0 l=1 rev1=1 crypto rx=0x7f79f40098f0 tx=0x7f79f4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=143fa4426b73d925 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:39.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.636+0000 7f79ff7fe700 1 -- 172.21.15.16:0/3286877221 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79f400b8a0 con 0x7f7a0015d900 2023-10-16T13:49:39.643 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.636+0000 7f79ff7fe700 1 -- 172.21.15.16:0/3286877221 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79f400ba00 con 0x7f7a0015d900 2023-10-16T13:49:39.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.636+0000 7f79ff7fe700 1 -- 172.21.15.16:0/3286877221 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79f400bd90 con 0x7f7a0015d900 2023-10-16T13:49:39.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.636+0000 7f7a06c5d700 1 -- 172.21.15.16:0/3286877221 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a0015d900 msgr2=0x7f7a000d2040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:39.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.636+0000 7f7a06c5d700 1 --2- 172.21.15.16:0/3286877221 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a0015d900 0x7f7a000d2040 secure :-1 s=READY pgs=3350 cs=0 l=1 rev1=1 crypto rx=0x7f79f40098f0 tx=0x7f79f4009c00 comp rx=0 tx=0).stop 2023-10-16T13:49:39.644 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.636+0000 7f7a06c5d700 1 -- 172.21.15.16:0/3286877221 shutdown_connections 2023-10-16T13:49:39.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.636+0000 7f7a06c5d700 1 --2- 172.21.15.16:0/3286877221 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a0015d900 0x7f7a000d2040 unknown :-1 s=CLOSED pgs=3350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:39.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.636+0000 7f7a06c5d700 1 -- 172.21.15.16:0/3286877221 >> 172.21.15.16:0/3286877221 conn(0x7f7a00072330 msgr2=0x7f7a00072740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:39.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.636+0000 7f7a06c5d700 1 -- 172.21.15.16:0/3286877221 shutdown_connections 2023-10-16T13:49:39.645 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.636+0000 7f7a06c5d700 1 -- 172.21.15.16:0/3286877221 wait complete. 2023-10-16T13:49:39.648 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.637+0000 7f7a06c5d700 1 Processor -- start 2023-10-16T13:49:39.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.638+0000 7f7a06c5d700 1 -- start start 2023-10-16T13:49:39.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.638+0000 7f7a06c5d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7a000ce9f0 0x7f7a0014e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:39.649 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.638+0000 7f7a06c5d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7a000d2f70 0x7f7a000c7b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:39.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.638+0000 7f7a06c5d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a0015d900 0x7f7a0014ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:39.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.638+0000 7f7a06c5d700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a000c8b90 con 0x7f7a0015d900 2023-10-16T13:49:39.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.638+0000 7f7a06c5d700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a000b3280 con 0x7f7a000d2f70 2023-10-16T13:49:39.650 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.638+0000 7f7a06c5d700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a000c8040 con 0x7f7a000ce9f0 2023-10-16T13:49:39.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.639+0000 7f7a051fa700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a0015d900 0x7f7a0014ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:39.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.639+0000 7f7a051fa700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a0015d900 0x7f7a0014ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60522/0 (socket says 172.21.15.16:60522) 2023-10-16T13:49:39.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.639+0000 7f7a051fa700 1 -- 172.21.15.16:0/1178011981 learned_addr learned my addr 172.21.15.16:0/1178011981 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:39.651 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.639+0000 7f7a049f9700 1 --2- 172.21.15.16:0/1178011981 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7a000ce9f0 0x7f7a0014e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:39.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.639+0000 7f79fffff700 1 --2- 172.21.15.16:0/1178011981 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7a000d2f70 0x7f7a000c7b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:39.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.639+0000 7f7a051fa700 1 -- 172.21.15.16:0/1178011981 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7a000ce9f0 msgr2=0x7f7a0014e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:39.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.639+0000 7f7a051fa700 1 --2- 172.21.15.16:0/1178011981 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7a000ce9f0 0x7f7a0014e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:39.652 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.639+0000 7f7a051fa700 1 -- 172.21.15.16:0/1178011981 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7a000d2f70 msgr2=0x7f7a000c7b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:39.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.639+0000 7f7a051fa700 1 --2- 172.21.15.16:0/1178011981 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7a000d2f70 0x7f7a000c7b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:39.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.639+0000 7f7a051fa700 1 -- 172.21.15.16:0/1178011981 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79f4009550 con 0x7f7a0015d900 2023-10-16T13:49:39.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.639+0000 7f7a051fa700 1 --2- 172.21.15.16:0/1178011981 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a0015d900 0x7f7a0014ccb0 secure :-1 s=READY pgs=3351 cs=0 l=1 rev1=1 crypto rx=0x7f79f800cc70 tx=0x7f79f800cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:39.653 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.639+0000 7f79fd7fa700 1 -- 172.21.15.16:0/1178011981 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79f800aaa0 con 0x7f7a0015d900 2023-10-16T13:49:39.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.639+0000 7f79fd7fa700 1 -- 172.21.15.16:0/1178011981 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79f800ac20 con 0x7f7a0015d900 2023-10-16T13:49:39.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.639+0000 7f79fd7fa700 1 -- 172.21.15.16:0/1178011981 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79f80070d0 con 0x7f7a0015d900 2023-10-16T13:49:39.654 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.639+0000 7f7a06c5d700 1 -- 172.21.15.16:0/1178011981 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a000c82f0 con 0x7f7a0015d900 2023-10-16T13:49:39.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.639+0000 7f7a06c5d700 1 -- 172.21.15.16:0/1178011981 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a000c88b0 con 0x7f7a0015d900 2023-10-16T13:49:39.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.640+0000 7f7a06c5d700 1 -- 172.21.15.16:0/1178011981 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a000b30e0 con 0x7f7a0015d900 2023-10-16T13:49:39.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.641+0000 7f79fd7fa700 1 -- 172.21.15.16:0/1178011981 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 176) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f79f8007230 con 0x7f7a0015d900 2023-10-16T13:49:39.655 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.641+0000 7f79fd7fa700 1 --2- 172.21.15.16:0/1178011981 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f79e406b020 0x7f79e406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:39.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.641+0000 7f79fd7fa700 1 -- 172.21.15.16:0/1178011981 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(283..283 src has 1..283) v4 ==== 6208+0+0 (secure 0 0 0) 0x7f79f8089390 con 0x7f7a0015d900 2023-10-16T13:49:39.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.641+0000 7f7a049f9700 1 --2- 172.21.15.16:0/1178011981 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f79e406b020 0x7f79e406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:39.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.642+0000 7f7a049f9700 1 --2- 172.21.15.16:0/1178011981 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f79e406b020 0x7f79e406d4e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f79f4000c00 tx=0x7f79f4008030 comp rx=0 tx=0).ready entity=mgr.13935 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:39.656 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.646+0000 7f79fd7fa700 1 -- 172.21.15.16:0/1178011981 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7a000b30e0 con 0x7f7a0015d900 2023-10-16T13:49:39.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.848+0000 7f7a06c5d700 1 -- 172.21.15.16:0/1178011981 --> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f7a000d1840 con 0x7f79e406b020 2023-10-16T13:49:39.851 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.848+0000 7f79fd7fa700 1 -- 172.21.15.16:0/1178011981 <== mgr.13935 v2:172.21.15.16:6832/938897590 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f79f400bbf0 con 0x7f79e406b020 2023-10-16T13:49:39.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.851+0000 7f7a06c5d700 1 -- 172.21.15.16:0/1178011981 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f79e406b020 msgr2=0x7f79e406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:39.853 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.851+0000 7f7a06c5d700 1 --2- 172.21.15.16:0/1178011981 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f79e406b020 0x7f79e406d4e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f79f4000c00 tx=0x7f79f4008030 comp rx=0 tx=0).stop 2023-10-16T13:49:39.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.851+0000 7f7a06c5d700 1 -- 172.21.15.16:0/1178011981 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a0015d900 msgr2=0x7f7a0014ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:39.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.851+0000 7f7a06c5d700 1 --2- 172.21.15.16:0/1178011981 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a0015d900 0x7f7a0014ccb0 secure :-1 s=READY pgs=3351 cs=0 l=1 rev1=1 crypto rx=0x7f79f800cc70 tx=0x7f79f800cf80 comp rx=0 tx=0).stop 2023-10-16T13:49:39.854 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.851+0000 7f7a06c5d700 1 -- 172.21.15.16:0/1178011981 shutdown_connections 2023-10-16T13:49:39.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.851+0000 7f7a06c5d700 1 --2- 172.21.15.16:0/1178011981 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f79e406b020 0x7f79e406d4e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:39.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.851+0000 7f7a06c5d700 1 --2- 172.21.15.16:0/1178011981 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a0015d900 0x7f7a0014ccb0 unknown :-1 s=CLOSED pgs=3351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:39.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.851+0000 7f7a06c5d700 1 --2- 172.21.15.16:0/1178011981 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f7a000ce9f0 0x7f7a0014e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:39.855 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.851+0000 7f7a06c5d700 1 --2- 172.21.15.16:0/1178011981 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7a000d2f70 0x7f7a000c7b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:39.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.851+0000 7f7a06c5d700 1 -- 172.21.15.16:0/1178011981 >> 172.21.15.16:0/1178011981 conn(0x7f7a00072330 msgr2=0x7f7a0006b800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:39.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.851+0000 7f7a06c5d700 1 -- 172.21.15.16:0/1178011981 shutdown_connections 2023-10-16T13:49:39.856 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.851+0000 7f7a06c5d700 1 -- 172.21.15.16:0/1178011981 wait complete. 2023-10-16T13:49:39.857 INFO:teuthology.orchestra.run.smithi016.stderr:instructed osd(s) 3 to deep-scrub 2023-10-16T13:49:39.865 INFO:tasks.ceph:Scrubbing osd.4 2023-10-16T13:49:39.866 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2023-10-16T13:49:39.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.992+0000 7f13cd9ab700 1 Processor -- start 2023-10-16T13:49:39.995 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.992+0000 7f13cd9ab700 1 -- start start 2023-10-16T13:49:39.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.993+0000 7f13cd9ab700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f13c8157b30 0x7f13c80cefc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:39.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.993+0000 7f13cd9ab700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f13c8157f10 0x7f13c80cf500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:39.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.993+0000 7f13cd9ab700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13c8142fc0 con 0x7f13c8161610 2023-10-16T13:49:39.998 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.993+0000 7f13cd9ab700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13c806d900 con 0x7f13c8157f10 2023-10-16T13:49:39.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.993+0000 7f13cd9ab700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13c8143210 con 0x7f13c8157b30 2023-10-16T13:49:39.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.993+0000 7f13c77fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f13c8161610 0x7f13c8161a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49804/0 (socket says 172.21.15.16:49804) 2023-10-16T13:49:39.999 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.993+0000 7f13c77fe700 1 -- 172.21.15.16:0/2736344634 learned_addr learned my addr 172.21.15.16:0/2736344634 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:49:40.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.993+0000 7f13c6ffd700 1 --2- 172.21.15.16:0/2736344634 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f13c8157b30 0x7f13c80cefc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:40.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.993+0000 7f13c67fc700 1 --2- 172.21.15.16:0/2736344634 >> v2:172.21.15.49:3300/0 conn(0x7f13c8157f10 0x7f13c80cf500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:40.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.994+0000 7f13c5ffb700 1 -- 172.21.15.16:0/2736344634 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1725350344 0 0) 0x7f13c8142fc0 con 0x7f13c8161610 2023-10-16T13:49:40.000 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.994+0000 7f13c5ffb700 1 -- 172.21.15.16:0/2736344634 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13b00035b0 con 0x7f13c8161610 2023-10-16T13:49:40.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.994+0000 7f13c67fc700 1 -- 172.21.15.16:0/2736344634 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f13c8157b30 msgr2=0x7f13c80cefc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:40.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.994+0000 7f13c67fc700 1 --2- 172.21.15.16:0/2736344634 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f13c8157b30 0x7f13c80cefc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:40.001 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.994+0000 7f13c67fc700 1 -- 172.21.15.16:0/2736344634 >> v1:172.21.15.16:6789/0 conn(0x7f13c8161610 legacy=0x7f13c8161a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:40.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.994+0000 7f13c67fc700 1 -- 172.21.15.16:0/2736344634 --> v2:172.21.15.49:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13c81434f0 con 0x7f13c8157f10 2023-10-16T13:49:40.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.994+0000 7f13c6ffd700 1 --2- 172.21.15.16:0/2736344634 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f13c8157b30 0x7f13c80cefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:49:40.002 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.994+0000 7f13c67fc700 1 --2- 172.21.15.16:0/2736344634 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f13c8157f10 0x7f13c80cf500 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f13b8009840 tx=0x7f13b8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d6d7f70d1ddc7d9f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:40.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.994+0000 7f13c5ffb700 1 -- 172.21.15.16:0/2736344634 <== mon.1 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13b8011830 con 0x7f13c8157f10 2023-10-16T13:49:40.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.994+0000 7f13c5ffb700 1 -- 172.21.15.16:0/2736344634 <== mon.1 v2:172.21.15.49:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13b8011990 con 0x7f13c8157f10 2023-10-16T13:49:40.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.994+0000 7f13c5ffb700 1 -- 172.21.15.16:0/2736344634 <== mon.1 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13b8011d20 con 0x7f13c8157f10 2023-10-16T13:49:40.003 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.995+0000 7f13cd9ab700 1 -- 172.21.15.16:0/2736344634 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f13c8157f10 msgr2=0x7f13c80cf500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:40.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.995+0000 7f13cd9ab700 1 --2- 172.21.15.16:0/2736344634 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f13c8157f10 0x7f13c80cf500 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f13b8009840 tx=0x7f13b8009b50 comp rx=0 tx=0).stop 2023-10-16T13:49:40.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.995+0000 7f13cd9ab700 1 -- 172.21.15.16:0/2736344634 shutdown_connections 2023-10-16T13:49:40.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.995+0000 7f13cd9ab700 1 --2- 172.21.15.16:0/2736344634 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f13c8157b30 0x7f13c80cefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:40.004 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.995+0000 7f13cd9ab700 1 --2- 172.21.15.16:0/2736344634 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f13c8157f10 0x7f13c80cf500 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:40.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.995+0000 7f13cd9ab700 1 -- 172.21.15.16:0/2736344634 >> 172.21.15.16:0/2736344634 conn(0x7f13c806a7e0 msgr2=0x7f13c806abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:40.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.995+0000 7f13cd9ab700 1 -- 172.21.15.16:0/2736344634 shutdown_connections 2023-10-16T13:49:40.005 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.995+0000 7f13cd9ab700 1 -- 172.21.15.16:0/2736344634 wait complete. 2023-10-16T13:49:40.008 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.996+0000 7f13cd9ab700 1 Processor -- start 2023-10-16T13:49:40.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.997+0000 7f13cd9ab700 1 -- start start 2023-10-16T13:49:40.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.997+0000 7f13cd9ab700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f13c8157b30 0x7f13c8154700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:40.009 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.997+0000 7f13cd9ab700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f13c8157f10 0x7f13c81f3e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:40.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.997+0000 7f13cd9ab700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13c8161610 0x7f13c80d1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:40.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.997+0000 7f13cd9ab700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13c80cd8f0 con 0x7f13c8161610 2023-10-16T13:49:40.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.997+0000 7f13cd9ab700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13c81f3bd0 con 0x7f13c8157b30 2023-10-16T13:49:40.010 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.997+0000 7f13cd9ab700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13c80ccfd0 con 0x7f13c8157f10 2023-10-16T13:49:40.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.997+0000 7f13c77fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13c8161610 0x7f13c80d1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:40.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.997+0000 7f13c6ffd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f13c8157b30 0x7f13c8154700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:40.011 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.997+0000 7f13c67fc700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f13c8157f10 0x7f13c81f3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:40.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.997+0000 7f13c77fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13c8161610 0x7f13c80d1320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60536/0 (socket says 172.21.15.16:60536) 2023-10-16T13:49:40.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.997+0000 7f13c77fe700 1 -- 172.21.15.16:0/1728835946 learned_addr learned my addr 172.21.15.16:0/1728835946 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:40.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.998+0000 7f13c77fe700 1 -- 172.21.15.16:0/1728835946 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f13c8157f10 msgr2=0x7f13c81f3e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:40.012 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.998+0000 7f13c77fe700 1 --2- 172.21.15.16:0/1728835946 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f13c8157f10 0x7f13c81f3e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:40.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.998+0000 7f13c77fe700 1 -- 172.21.15.16:0/1728835946 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f13c8157b30 msgr2=0x7f13c8154700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:40.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.998+0000 7f13c77fe700 1 --2- 172.21.15.16:0/1728835946 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f13c8157b30 0x7f13c8154700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:40.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.998+0000 7f13c77fe700 1 -- 172.21.15.16:0/1728835946 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13bc0094a0 con 0x7f13c8161610 2023-10-16T13:49:40.013 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.998+0000 7f13c77fe700 1 --2- 172.21.15.16:0/1728835946 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13c8161610 0x7f13c80d1320 secure :-1 s=READY pgs=3353 cs=0 l=1 rev1=1 crypto rx=0x7f13c000cad0 tx=0x7f13c000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:40.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.998+0000 7f13affff700 1 -- 172.21.15.16:0/1728835946 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c000a9a0 con 0x7f13c8161610 2023-10-16T13:49:40.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.998+0000 7f13affff700 1 -- 172.21.15.16:0/1728835946 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13c000ab00 con 0x7f13c8161610 2023-10-16T13:49:40.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.998+0000 7f13cd9ab700 1 -- 172.21.15.16:0/1728835946 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13b80094a0 con 0x7f13c8161610 2023-10-16T13:49:40.014 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.998+0000 7f13affff700 1 -- 172.21.15.16:0/1728835946 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c000ae90 con 0x7f13c8161610 2023-10-16T13:49:40.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.998+0000 7f13cd9ab700 1 -- 172.21.15.16:0/1728835946 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13c8154c40 con 0x7f13c8161610 2023-10-16T13:49:40.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:39.999+0000 7f13cd9ab700 1 -- 172.21.15.16:0/1728835946 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f13c8143ab0 con 0x7f13c8161610 2023-10-16T13:49:40.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.000+0000 7f13affff700 1 -- 172.21.15.16:0/1728835946 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 176) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f13c001b3b0 con 0x7f13c8161610 2023-10-16T13:49:40.015 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.000+0000 7f13affff700 1 --2- 172.21.15.16:0/1728835946 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f13b406b020 0x7f13b406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:40.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.000+0000 7f13affff700 1 -- 172.21.15.16:0/1728835946 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(283..283 src has 1..283) v4 ==== 6208+0+0 (secure 0 0 0) 0x7f13c00894e0 con 0x7f13c8161610 2023-10-16T13:49:40.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.000+0000 7f13c6ffd700 1 --2- 172.21.15.16:0/1728835946 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f13b406b020 0x7f13b406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:40.016 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.000+0000 7f13affff700 1 --2- 172.21.15.16:0/1728835946 >> [v2:172.21.15.49:6800/682437317,v1:172.21.15.49:6801/682437317] conn(0x7f13b4071950 0x7f13b4073df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:40.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.000+0000 7f13affff700 1 -- 172.21.15.16:0/1728835946 --> [v2:172.21.15.49:6800/682437317,v1:172.21.15.49:6801/682437317] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f13b40744c0 con 0x7f13b4071950 2023-10-16T13:49:40.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.000+0000 7f13affff700 1 -- 172.21.15.16:0/1728835946 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=283) v2 ==== 24+0+0 (secure 0 0 0) 0x7f13c008bb80 con 0x7f13c8161610 2023-10-16T13:49:40.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.000+0000 7f13c67fc700 1 --2- 172.21.15.16:0/1728835946 >> [v2:172.21.15.49:6800/682437317,v1:172.21.15.49:6801/682437317] conn(0x7f13b4071950 0x7f13b4073df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:40.017 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.000+0000 7f13c6ffd700 1 --2- 172.21.15.16:0/1728835946 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f13b406b020 0x7f13b406d4e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f13bc009d70 tx=0x7f13bc009210 comp rx=0 tx=0).ready entity=mgr.13935 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:40.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.001+0000 7f13c67fc700 1 --2- 172.21.15.16:0/1728835946 >> [v2:172.21.15.49:6800/682437317,v1:172.21.15.49:6801/682437317] conn(0x7f13b4071950 0x7f13b4073df0 crc :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:40.018 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.002+0000 7f13affff700 1 -- 172.21.15.16:0/1728835946 <== osd.4 v2:172.21.15.49:6800/682437317 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7f13b40744c0 con 0x7f13b4071950 2023-10-16T13:49:40.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.016+0000 7f13cd9ab700 1 -- 172.21.15.16:0/1728835946 --> [v2:172.21.15.49:6800/682437317,v1:172.21.15.49:6801/682437317] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f13c81437d0 con 0x7f13b4071950 2023-10-16T13:49:40.019 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.017+0000 7f13affff700 1 -- 172.21.15.16:0/1728835946 <== osd.4 v2:172.21.15.49:6800/682437317 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f13b8022040 con 0x7f13b4071950 2023-10-16T13:49:40.020 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-10-16T13:49:40.020 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-16T13:49:40.020 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-10-16T13:49:40.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.017+0000 7f13cd9ab700 1 -- 172.21.15.16:0/1728835946 >> [v2:172.21.15.49:6800/682437317,v1:172.21.15.49:6801/682437317] conn(0x7f13b4071950 msgr2=0x7f13b4073df0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:40.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.017+0000 7f13cd9ab700 1 --2- 172.21.15.16:0/1728835946 >> [v2:172.21.15.49:6800/682437317,v1:172.21.15.49:6801/682437317] conn(0x7f13b4071950 0x7f13b4073df0 crc :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:40.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.017+0000 7f13cd9ab700 1 -- 172.21.15.16:0/1728835946 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f13b406b020 msgr2=0x7f13b406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:40.022 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.017+0000 7f13cd9ab700 1 --2- 172.21.15.16:0/1728835946 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f13b406b020 0x7f13b406d4e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f13bc009d70 tx=0x7f13bc009210 comp rx=0 tx=0).stop 2023-10-16T13:49:40.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.017+0000 7f13cd9ab700 1 -- 172.21.15.16:0/1728835946 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13c8161610 msgr2=0x7f13c80d1320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:40.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.017+0000 7f13cd9ab700 1 --2- 172.21.15.16:0/1728835946 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13c8161610 0x7f13c80d1320 secure :-1 s=READY pgs=3353 cs=0 l=1 rev1=1 crypto rx=0x7f13c000cad0 tx=0x7f13c000cde0 comp rx=0 tx=0).stop 2023-10-16T13:49:40.023 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.017+0000 7f13c77fe700 1 -- 172.21.15.16:0/1728835946 reap_dead start 2023-10-16T13:49:40.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.017+0000 7f13cd9ab700 1 -- 172.21.15.16:0/1728835946 shutdown_connections 2023-10-16T13:49:40.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.017+0000 7f13cd9ab700 1 -- 172.21.15.16:0/1728835946 >> 172.21.15.16:0/1728835946 conn(0x7f13c806a7e0 msgr2=0x7f13c80cefc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:40.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.018+0000 7f13cd9ab700 1 -- 172.21.15.16:0/1728835946 shutdown_connections 2023-10-16T13:49:40.024 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.018+0000 7f13cd9ab700 1 -- 172.21.15.16:0/1728835946 wait complete. 2023-10-16T13:49:40.032 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2023-10-16T13:49:40.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.154+0000 7f291fd8e700 1 Processor -- start 2023-10-16T13:49:40.157 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.155+0000 7f291fd8e700 1 -- start start 2023-10-16T13:49:40.160 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.155+0000 7f291fd8e700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f291815b280 0x7f29181613e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:40.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.155+0000 7f291fd8e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f29180cc8a0 0x7f291815e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:40.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.155+0000 7f291fd8e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f291806a7e0 0x7f291806ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:40.161 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.155+0000 7f291fd8e700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f291806b170 con 0x7f291806a7e0 2023-10-16T13:49:40.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.155+0000 7f291fd8e700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2918161c30 con 0x7f291815b280 2023-10-16T13:49:40.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.155+0000 7f291fd8e700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2918161f10 con 0x7f29180cc8a0 2023-10-16T13:49:40.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.156+0000 7f291e32b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f291806a7e0 0x7f291806ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:40.162 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.156+0000 7f291db2a700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f291815b280 0x7f29181613e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:40.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.156+0000 7f291d329700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f29180cc8a0 0x7f291815e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:40.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.156+0000 7f291e32b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f291806a7e0 0x7f291806ac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60546/0 (socket says 172.21.15.16:60546) 2023-10-16T13:49:40.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.156+0000 7f291e32b700 1 -- 172.21.15.16:0/548492132 learned_addr learned my addr 172.21.15.16:0/548492132 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:40.163 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.156+0000 7f291e32b700 1 -- 172.21.15.16:0/548492132 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f29180cc8a0 msgr2=0x7f291815e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:40.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.156+0000 7f291e32b700 1 --2- 172.21.15.16:0/548492132 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f29180cc8a0 0x7f291815e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:40.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.156+0000 7f291e32b700 1 -- 172.21.15.16:0/548492132 >> v2:172.21.15.49:3300/0 conn(0x7f291815b280 msgr2=0x7f29181613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:40.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.156+0000 7f291e32b700 1 --2- 172.21.15.16:0/548492132 >> v2:172.21.15.49:3300/0 conn(0x7f291815b280 0x7f29181613e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:40.164 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.156+0000 7f291e32b700 1 -- 172.21.15.16:0/548492132 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29181621f0 con 0x7f291806a7e0 2023-10-16T13:49:40.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.156+0000 7f291e32b700 1 --2- 172.21.15.16:0/548492132 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f291806a7e0 0x7f291806ac30 secure :-1 s=READY pgs=3354 cs=0 l=1 rev1=1 crypto rx=0x7f291806b8a0 tx=0x7f291000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1036cf54bc296c3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:40.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.156+0000 7f291cb28700 1 -- 172.21.15.16:0/548492132 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f291000e070 con 0x7f291806a7e0 2023-10-16T13:49:40.165 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.156+0000 7f291cb28700 1 -- 172.21.15.16:0/548492132 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f291000b910 con 0x7f291806a7e0 2023-10-16T13:49:40.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.156+0000 7f291cb28700 1 -- 172.21.15.16:0/548492132 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f291000bcc0 con 0x7f291806a7e0 2023-10-16T13:49:40.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.157+0000 7f291fd8e700 1 -- 172.21.15.16:0/548492132 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f291806a7e0 msgr2=0x7f291806ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:40.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.157+0000 7f291fd8e700 1 --2- 172.21.15.16:0/548492132 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f291806a7e0 0x7f291806ac30 secure :-1 s=READY pgs=3354 cs=0 l=1 rev1=1 crypto rx=0x7f291806b8a0 tx=0x7f291000b330 comp rx=0 tx=0).stop 2023-10-16T13:49:40.166 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.157+0000 7f291fd8e700 1 -- 172.21.15.16:0/548492132 shutdown_connections 2023-10-16T13:49:40.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.157+0000 7f291fd8e700 1 --2- 172.21.15.16:0/548492132 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f291806a7e0 0x7f291806ac30 unknown :-1 s=CLOSED pgs=3354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:40.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.157+0000 7f291fd8e700 1 --2- 172.21.15.16:0/548492132 >> v2:172.21.15.49:3300/0 conn(0x7f291815b280 0x7f29181613e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:40.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.157+0000 7f291fd8e700 1 --2- 172.21.15.16:0/548492132 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f29180cc8a0 0x7f291815e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:40.167 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.157+0000 7f291fd8e700 1 -- 172.21.15.16:0/548492132 >> 172.21.15.16:0/548492132 conn(0x7f29180b1fd0 msgr2=0x7f29180b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:40.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.157+0000 7f291fd8e700 1 -- 172.21.15.16:0/548492132 shutdown_connections 2023-10-16T13:49:40.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.157+0000 7f291fd8e700 1 -- 172.21.15.16:0/548492132 wait complete. 2023-10-16T13:49:40.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.158+0000 7f291fd8e700 1 Processor -- start 2023-10-16T13:49:40.168 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.158+0000 7f291fd8e700 1 -- start start 2023-10-16T13:49:40.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.159+0000 7f291fd8e700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f291806a7e0 0x7f29181f2670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:40.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.159+0000 7f291fd8e700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f29180cc8a0 0x7f29180afb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:40.171 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.159+0000 7f291fd8e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f291815b280 0x7f29180c4190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:40.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.159+0000 7f291fd8e700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f291815b6d0 con 0x7f291815b280 2023-10-16T13:49:40.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.159+0000 7f291fd8e700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29180c63b0 con 0x7f29180cc8a0 2023-10-16T13:49:40.172 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.159+0000 7f291fd8e700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29180c6690 con 0x7f291806a7e0 2023-10-16T13:49:40.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.159+0000 7f291e32b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f291815b280 0x7f29180c4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:40.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.159+0000 7f291db2a700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f291806a7e0 0x7f29181f2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:40.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.159+0000 7f291e32b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f291815b280 0x7f29180c4190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60552/0 (socket says 172.21.15.16:60552) 2023-10-16T13:49:40.173 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.159+0000 7f291d329700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f29180cc8a0 0x7f29180afb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:40.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.159+0000 7f291e32b700 1 -- 172.21.15.16:0/3982475635 learned_addr learned my addr 172.21.15.16:0/3982475635 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:40.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.159+0000 7f291e32b700 1 -- 172.21.15.16:0/3982475635 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f291806a7e0 msgr2=0x7f29181f2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:40.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.159+0000 7f291e32b700 1 --2- 172.21.15.16:0/3982475635 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f291806a7e0 0x7f29181f2670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:40.174 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.159+0000 7f291e32b700 1 -- 172.21.15.16:0/3982475635 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f29180cc8a0 msgr2=0x7f29180afb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:40.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.159+0000 7f291e32b700 1 --2- 172.21.15.16:0/3982475635 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f29180cc8a0 0x7f29180afb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:40.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.159+0000 7f291e32b700 1 -- 172.21.15.16:0/3982475635 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2910009b30 con 0x7f291815b280 2023-10-16T13:49:40.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.160+0000 7f291e32b700 1 --2- 172.21.15.16:0/3982475635 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f291815b280 0x7f29180c4190 secure :-1 s=READY pgs=3355 cs=0 l=1 rev1=1 crypto rx=0x7f29180d1540 tx=0x7f2910004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:40.175 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.160+0000 7f290e7fc700 1 -- 172.21.15.16:0/3982475635 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29100099b0 con 0x7f291815b280 2023-10-16T13:49:40.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.160+0000 7f290e7fc700 1 -- 172.21.15.16:0/3982475635 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f291000ce80 con 0x7f291815b280 2023-10-16T13:49:40.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.160+0000 7f291fd8e700 1 -- 172.21.15.16:0/3982475635 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29180c6970 con 0x7f291815b280 2023-10-16T13:49:40.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.160+0000 7f290e7fc700 1 -- 172.21.15.16:0/3982475635 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2910004490 con 0x7f291815b280 2023-10-16T13:49:40.176 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.160+0000 7f291fd8e700 1 -- 172.21.15.16:0/3982475635 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29180b00d0 con 0x7f291815b280 2023-10-16T13:49:40.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.161+0000 7f291fd8e700 1 -- 172.21.15.16:0/3982475635 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f291815b810 con 0x7f291815b280 2023-10-16T13:49:40.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.161+0000 7f290e7fc700 1 -- 172.21.15.16:0/3982475635 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 176) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f29100045f0 con 0x7f291815b280 2023-10-16T13:49:40.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.162+0000 7f290e7fc700 1 --2- 172.21.15.16:0/3982475635 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f28fc06b020 0x7f28fc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:40.177 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.162+0000 7f291db2a700 1 --2- 172.21.15.16:0/3982475635 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f28fc06b020 0x7f28fc06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:40.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.162+0000 7f290e7fc700 1 -- 172.21.15.16:0/3982475635 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(283..283 src has 1..283) v4 ==== 6208+0+0 (secure 0 0 0) 0x7f291008ad20 con 0x7f291815b280 2023-10-16T13:49:40.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.162+0000 7f291db2a700 1 --2- 172.21.15.16:0/3982475635 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f28fc06b020 0x7f28fc06d4e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2908009b40 tx=0x7f2908009210 comp rx=0 tx=0).ready entity=mgr.13935 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:40.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.166+0000 7f290e7fc700 1 -- 172.21.15.16:0/3982475635 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f291815b810 con 0x7f291815b280 2023-10-16T13:49:40.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.370+0000 7f291fd8e700 1 -- 172.21.15.16:0/3982475635 --> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f291815e1e0 con 0x7f28fc06b020 2023-10-16T13:49:40.373 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.371+0000 7f290e7fc700 1 -- 172.21.15.16:0/3982475635 <== mgr.13935 v2:172.21.15.16:6832/938897590 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f2908004020 con 0x7f28fc06b020 2023-10-16T13:49:40.375 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.373+0000 7f291fd8e700 1 -- 172.21.15.16:0/3982475635 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f28fc06b020 msgr2=0x7f28fc06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:40.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.373+0000 7f291fd8e700 1 --2- 172.21.15.16:0/3982475635 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f28fc06b020 0x7f28fc06d4e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2908009b40 tx=0x7f2908009210 comp rx=0 tx=0).stop 2023-10-16T13:49:40.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.373+0000 7f291fd8e700 1 -- 172.21.15.16:0/3982475635 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f291815b280 msgr2=0x7f29180c4190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:40.376 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.373+0000 7f291fd8e700 1 --2- 172.21.15.16:0/3982475635 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f291815b280 0x7f29180c4190 secure :-1 s=READY pgs=3355 cs=0 l=1 rev1=1 crypto rx=0x7f29180d1540 tx=0x7f2910004ad0 comp rx=0 tx=0).stop 2023-10-16T13:49:40.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.373+0000 7f291fd8e700 1 -- 172.21.15.16:0/3982475635 shutdown_connections 2023-10-16T13:49:40.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.373+0000 7f291fd8e700 1 --2- 172.21.15.16:0/3982475635 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f28fc06b020 0x7f28fc06d4e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:40.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.373+0000 7f291fd8e700 1 --2- 172.21.15.16:0/3982475635 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f291815b280 0x7f29180c4190 unknown :-1 s=CLOSED pgs=3355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:40.377 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.373+0000 7f291fd8e700 1 --2- 172.21.15.16:0/3982475635 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f291806a7e0 0x7f29181f2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:40.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.373+0000 7f291fd8e700 1 --2- 172.21.15.16:0/3982475635 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f29180cc8a0 0x7f29180afb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:40.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.373+0000 7f291fd8e700 1 -- 172.21.15.16:0/3982475635 >> 172.21.15.16:0/3982475635 conn(0x7f29180b1fd0 msgr2=0x7f29180d0270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:40.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.373+0000 7f291fd8e700 1 -- 172.21.15.16:0/3982475635 shutdown_connections 2023-10-16T13:49:40.378 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.373+0000 7f291fd8e700 1 -- 172.21.15.16:0/3982475635 wait complete. 2023-10-16T13:49:40.379 INFO:teuthology.orchestra.run.smithi016.stderr:instructed osd(s) 4 to deep-scrub 2023-10-16T13:49:40.387 INFO:tasks.ceph:Scrubbing osd.5 2023-10-16T13:49:40.388 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2023-10-16T13:49:40.517 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.515+0000 7f68eacfa700 1 Processor -- start 2023-10-16T13:49:40.518 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.515+0000 7f68eacfa700 1 -- start start 2023-10-16T13:49:40.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.516+0000 7f68eacfa700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f68e415a3d0 0x7f68e406d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:40.521 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.516+0000 7f68eacfa700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f68e415e0b0 0x7f68e40ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:40.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.516+0000 7f68eacfa700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f68e40cf430 0x7f68e40cdeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:40.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.516+0000 7f68eacfa700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68e415f0c0 con 0x7f68e40cf430 2023-10-16T13:49:40.522 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.516+0000 7f68eacfa700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68e4072990 con 0x7f68e415e0b0 2023-10-16T13:49:40.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.516+0000 7f68eacfa700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68e406de50 con 0x7f68e415a3d0 2023-10-16T13:49:40.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.516+0000 7f68e9297700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f68e40cf430 0x7f68e40cdeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:40.523 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.516+0000 7f68e9297700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f68e40cf430 0x7f68e40cdeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60560/0 (socket says 172.21.15.16:60560) 2023-10-16T13:49:40.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.516+0000 7f68e9297700 1 -- 172.21.15.16:0/456078141 learned_addr learned my addr 172.21.15.16:0/456078141 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:40.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.516+0000 7f68e8a96700 1 --2- 172.21.15.16:0/456078141 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f68e415a3d0 0x7f68e406d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:40.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.516+0000 7f68e3fff700 1 --2- 172.21.15.16:0/456078141 >> v2:172.21.15.49:3300/0 conn(0x7f68e415e0b0 0x7f68e40ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:40.524 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.516+0000 7f68e9297700 1 -- 172.21.15.16:0/456078141 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f68e415a3d0 msgr2=0x7f68e406d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:40.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.516+0000 7f68e9297700 1 --2- 172.21.15.16:0/456078141 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f68e415a3d0 0x7f68e406d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:40.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.516+0000 7f68e9297700 1 -- 172.21.15.16:0/456078141 >> v2:172.21.15.49:3300/0 conn(0x7f68e415e0b0 msgr2=0x7f68e40ceef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:40.525 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.516+0000 7f68e9297700 1 --2- 172.21.15.16:0/456078141 >> v2:172.21.15.49:3300/0 conn(0x7f68e415e0b0 0x7f68e40ceef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:40.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.516+0000 7f68e9297700 1 -- 172.21.15.16:0/456078141 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68e40cb870 con 0x7f68e40cf430 2023-10-16T13:49:40.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.516+0000 7f68e9297700 1 --2- 172.21.15.16:0/456078141 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68e40cf430 0x7f68e40cdeb0 secure :-1 s=READY pgs=3356 cs=0 l=1 rev1=1 crypto rx=0x7f68e40cde60 tx=0x7f68dc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=472a38322cf7240c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:40.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.517+0000 7f68e37fe700 1 -- 172.21.15.16:0/456078141 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68dc00e070 con 0x7f68e40cf430 2023-10-16T13:49:40.526 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.517+0000 7f68e37fe700 1 -- 172.21.15.16:0/456078141 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68dc00b910 con 0x7f68e40cf430 2023-10-16T13:49:40.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.517+0000 7f68e37fe700 1 -- 172.21.15.16:0/456078141 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68dc00bcc0 con 0x7f68e40cf430 2023-10-16T13:49:40.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.517+0000 7f68eacfa700 1 -- 172.21.15.16:0/456078141 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68e40cf430 msgr2=0x7f68e40cdeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:40.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.517+0000 7f68eacfa700 1 --2- 172.21.15.16:0/456078141 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68e40cf430 0x7f68e40cdeb0 secure :-1 s=READY pgs=3356 cs=0 l=1 rev1=1 crypto rx=0x7f68e40cde60 tx=0x7f68dc00b330 comp rx=0 tx=0).stop 2023-10-16T13:49:40.527 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.517+0000 7f68eacfa700 1 -- 172.21.15.16:0/456078141 shutdown_connections 2023-10-16T13:49:40.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.517+0000 7f68eacfa700 1 --2- 172.21.15.16:0/456078141 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68e40cf430 0x7f68e40cdeb0 unknown :-1 s=CLOSED pgs=3356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:40.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.517+0000 7f68eacfa700 1 --2- 172.21.15.16:0/456078141 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f68e415a3d0 0x7f68e406d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:40.528 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.517+0000 7f68eacfa700 1 --2- 172.21.15.16:0/456078141 >> v2:172.21.15.49:3300/0 conn(0x7f68e415e0b0 0x7f68e40ceef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:40.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.517+0000 7f68eacfa700 1 -- 172.21.15.16:0/456078141 >> 172.21.15.16:0/456078141 conn(0x7f68e406f430 msgr2=0x7f68e406f810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:40.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.517+0000 7f68eacfa700 1 -- 172.21.15.16:0/456078141 shutdown_connections 2023-10-16T13:49:40.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.517+0000 7f68eacfa700 1 -- 172.21.15.16:0/456078141 wait complete. 2023-10-16T13:49:40.529 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.518+0000 7f68eacfa700 1 Processor -- start 2023-10-16T13:49:40.530 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.518+0000 7f68eacfa700 1 -- start start 2023-10-16T13:49:40.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.519+0000 7f68eacfa700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68e40cf430 0x7f68e40c4080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:40.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.519+0000 7f68eacfa700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f68e415a3d0 0x7f68e40c45c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:40.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.519+0000 7f68eacfa700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f68e415e0b0 0x7f68e40c4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:40.533 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.519+0000 7f68eacfa700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68e40b5cf0 con 0x7f68e40cf430 2023-10-16T13:49:40.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.519+0000 7f68eacfa700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68e40c5190 con 0x7f68e415e0b0 2023-10-16T13:49:40.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.519+0000 7f68eacfa700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68e40c5470 con 0x7f68e415a3d0 2023-10-16T13:49:40.534 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.519+0000 7f68e8a96700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68e40cf430 0x7f68e40c4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:40.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.519+0000 7f68e8a96700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68e40cf430 0x7f68e40c4080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60562/0 (socket says 172.21.15.16:60562) 2023-10-16T13:49:40.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.519+0000 7f68e8a96700 1 -- 172.21.15.16:0/4124911236 learned_addr learned my addr 172.21.15.16:0/4124911236 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:40.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.519+0000 7f68e9297700 1 --2- 172.21.15.16:0/4124911236 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f68e415e0b0 0x7f68e40c4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:40.535 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.519+0000 7f68e3fff700 1 --2- 172.21.15.16:0/4124911236 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f68e415a3d0 0x7f68e40c45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:40.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.520+0000 7f68e8a96700 1 -- 172.21.15.16:0/4124911236 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f68e415a3d0 msgr2=0x7f68e40c45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:40.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.520+0000 7f68e8a96700 1 --2- 172.21.15.16:0/4124911236 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f68e415a3d0 0x7f68e40c45c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:40.536 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.520+0000 7f68e8a96700 1 -- 172.21.15.16:0/4124911236 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f68e415e0b0 msgr2=0x7f68e40c4b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:40.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.520+0000 7f68e8a96700 1 --2- 172.21.15.16:0/4124911236 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f68e415e0b0 0x7f68e40c4b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:40.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.520+0000 7f68e8a96700 1 -- 172.21.15.16:0/4124911236 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68dc009b30 con 0x7f68e40cf430 2023-10-16T13:49:40.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.520+0000 7f68e8a96700 1 --2- 172.21.15.16:0/4124911236 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68e40cf430 0x7f68e40c4080 secure :-1 s=READY pgs=3357 cs=0 l=1 rev1=1 crypto rx=0x7f68d800e9b0 tx=0x7f68d800ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:40.537 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.520+0000 7f68e17fa700 1 -- 172.21.15.16:0/4124911236 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68d8009710 con 0x7f68e40cf430 2023-10-16T13:49:40.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.520+0000 7f68e17fa700 1 -- 172.21.15.16:0/4124911236 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68d800ee50 con 0x7f68e40cf430 2023-10-16T13:49:40.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.520+0000 7f68eacfa700 1 -- 172.21.15.16:0/4124911236 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68e40c5750 con 0x7f68e40cf430 2023-10-16T13:49:40.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.520+0000 7f68e17fa700 1 -- 172.21.15.16:0/4124911236 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68d800ca50 con 0x7f68e40cf430 2023-10-16T13:49:40.538 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.520+0000 7f68eacfa700 1 -- 172.21.15.16:0/4124911236 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68e40c5d10 con 0x7f68e40cf430 2023-10-16T13:49:40.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.521+0000 7f68eacfa700 1 -- 172.21.15.16:0/4124911236 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f68e406e320 con 0x7f68e40cf430 2023-10-16T13:49:40.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.521+0000 7f68e17fa700 1 -- 172.21.15.16:0/4124911236 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 176) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f68d800cbb0 con 0x7f68e40cf430 2023-10-16T13:49:40.539 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.522+0000 7f68e17fa700 1 --2- 172.21.15.16:0/4124911236 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f68c806b020 0x7f68c806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:40.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.522+0000 7f68e17fa700 1 -- 172.21.15.16:0/4124911236 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(283..283 src has 1..283) v4 ==== 6208+0+0 (secure 0 0 0) 0x7f68d8089970 con 0x7f68e40cf430 2023-10-16T13:49:40.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.522+0000 7f68e3fff700 1 --2- 172.21.15.16:0/4124911236 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f68c806b020 0x7f68c806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:40.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.522+0000 7f68e17fa700 1 --2- 172.21.15.16:0/4124911236 >> [v2:172.21.15.49:6816/3830572508,v1:172.21.15.49:6817/3830572508] conn(0x7f68c8071ac0 0x7f68c8073f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:40.540 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.522+0000 7f68e17fa700 1 -- 172.21.15.16:0/4124911236 --> [v2:172.21.15.49:6816/3830572508,v1:172.21.15.49:6817/3830572508] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f68c8074630 con 0x7f68c8071ac0 2023-10-16T13:49:40.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.522+0000 7f68e17fa700 1 -- 172.21.15.16:0/4124911236 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=283) v2 ==== 24+0+0 (secure 0 0 0) 0x7f68d808e2a0 con 0x7f68e40cf430 2023-10-16T13:49:40.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.522+0000 7f68e9297700 1 --2- 172.21.15.16:0/4124911236 >> [v2:172.21.15.49:6816/3830572508,v1:172.21.15.49:6817/3830572508] conn(0x7f68c8071ac0 0x7f68c8073f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:40.541 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.522+0000 7f68e3fff700 1 --2- 172.21.15.16:0/4124911236 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f68c806b020 0x7f68c806d4e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f68e4159f00 tx=0x7f68d0009210 comp rx=0 tx=0).ready entity=mgr.13935 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:40.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.523+0000 7f68e9297700 1 --2- 172.21.15.16:0/4124911236 >> [v2:172.21.15.49:6816/3830572508,v1:172.21.15.49:6817/3830572508] conn(0x7f68c8071ac0 0x7f68c8073f60 crc :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:40.542 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.524+0000 7f68e17fa700 1 -- 172.21.15.16:0/4124911236 <== osd.5 v2:172.21.15.49:6816/3830572508 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7f68c8074630 con 0x7f68c8071ac0 2023-10-16T13:49:40.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.538+0000 7f68eacfa700 1 -- 172.21.15.16:0/4124911236 --> [v2:172.21.15.49:6816/3830572508,v1:172.21.15.49:6817/3830572508] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f68e406e040 con 0x7f68c8071ac0 2023-10-16T13:49:40.543 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.539+0000 7f68e17fa700 1 -- 172.21.15.16:0/4124911236 <== osd.5 v2:172.21.15.49:6816/3830572508 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f68dc0099b0 con 0x7f68c8071ac0 2023-10-16T13:49:40.544 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-10-16T13:49:40.544 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-16T13:49:40.544 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-10-16T13:49:40.545 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.539+0000 7f68eacfa700 1 -- 172.21.15.16:0/4124911236 >> [v2:172.21.15.49:6816/3830572508,v1:172.21.15.49:6817/3830572508] conn(0x7f68c8071ac0 msgr2=0x7f68c8073f60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:40.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.539+0000 7f68eacfa700 1 --2- 172.21.15.16:0/4124911236 >> [v2:172.21.15.49:6816/3830572508,v1:172.21.15.49:6817/3830572508] conn(0x7f68c8071ac0 0x7f68c8073f60 crc :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:40.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.539+0000 7f68eacfa700 1 -- 172.21.15.16:0/4124911236 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f68c806b020 msgr2=0x7f68c806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:40.546 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.539+0000 7f68eacfa700 1 --2- 172.21.15.16:0/4124911236 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f68c806b020 0x7f68c806d4e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f68e4159f00 tx=0x7f68d0009210 comp rx=0 tx=0).stop 2023-10-16T13:49:40.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.539+0000 7f68eacfa700 1 -- 172.21.15.16:0/4124911236 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68e40cf430 msgr2=0x7f68e40c4080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:40.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.539+0000 7f68eacfa700 1 --2- 172.21.15.16:0/4124911236 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68e40cf430 0x7f68e40c4080 secure :-1 s=READY pgs=3357 cs=0 l=1 rev1=1 crypto rx=0x7f68d800e9b0 tx=0x7f68d800ecc0 comp rx=0 tx=0).stop 2023-10-16T13:49:40.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.539+0000 7f68e9297700 1 -- 172.21.15.16:0/4124911236 reap_dead start 2023-10-16T13:49:40.547 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.539+0000 7f68eacfa700 1 -- 172.21.15.16:0/4124911236 shutdown_connections 2023-10-16T13:49:40.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.539+0000 7f68eacfa700 1 -- 172.21.15.16:0/4124911236 >> 172.21.15.16:0/4124911236 conn(0x7f68e406f430 msgr2=0x7f68e40cc8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:40.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.540+0000 7f68eacfa700 1 -- 172.21.15.16:0/4124911236 shutdown_connections 2023-10-16T13:49:40.548 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.540+0000 7f68eacfa700 1 -- 172.21.15.16:0/4124911236 wait complete. 2023-10-16T13:49:40.553 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2023-10-16T13:49:40.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.677+0000 7f70ba9d3700 1 Processor -- start 2023-10-16T13:49:40.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.677+0000 7f70ba9d3700 1 -- start start 2023-10-16T13:49:40.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.678+0000 7f70ba9d3700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f70b406aef0 0x7f70b40cb980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:40.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.678+0000 7f70ba9d3700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70b40cedb0 con 0x7f70b40ce160 2023-10-16T13:49:40.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.678+0000 7f70ba9d3700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70b40d12c0 con 0x7f70b415db40 2023-10-16T13:49:40.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.678+0000 7f70ba9d3700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70b415df20 con 0x7f70b406aef0 2023-10-16T13:49:40.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.678+0000 7f70b37fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f70b406aef0 0x7f70b40cb980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:40.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.678+0000 7f70b8f70700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f70b40ce160 0x7f70b40ce5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49850/0 (socket says 172.21.15.16:49850) 2023-10-16T13:49:40.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.678+0000 7f70b8f70700 1 -- 172.21.15.16:0/2276989099 learned_addr learned my addr 172.21.15.16:0/2276989099 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:49:40.684 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.678+0000 7f70b37fe700 1 -- 172.21.15.16:0/2276989099 >> v1:172.21.15.49:6789/0 conn(0x7f70b415db40 legacy=0x7f70b406a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:40.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.678+0000 7f70b37fe700 1 -- 172.21.15.16:0/2276989099 >> v1:172.21.15.16:6789/0 conn(0x7f70b40ce160 legacy=0x7f70b40ce5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:40.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.678+0000 7f70b2ffd700 1 -- 172.21.15.16:0/2276989099 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 246887804 0 0) 0x7f70b40cedb0 con 0x7f70b40ce160 2023-10-16T13:49:40.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.678+0000 7f70b37fe700 1 -- 172.21.15.16:0/2276989099 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70b40cabc0 con 0x7f70b406aef0 2023-10-16T13:49:40.685 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.678+0000 7f70b37fe700 1 --2- 172.21.15.16:0/2276989099 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70b406aef0 0x7f70b40cb980 secure :-1 s=READY pgs=3359 cs=0 l=1 rev1=1 crypto rx=0x7f709c009840 tx=0x7f709c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8c862e2d489374a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:40.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.679+0000 7f70b2ffd700 1 -- 172.21.15.16:0/2276989099 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f709c004230 con 0x7f70b406aef0 2023-10-16T13:49:40.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.679+0000 7f70b2ffd700 1 -- 172.21.15.16:0/2276989099 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f709c004390 con 0x7f70b406aef0 2023-10-16T13:49:40.686 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.679+0000 7f70b2ffd700 1 -- 172.21.15.16:0/2276989099 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f709c0036a0 con 0x7f70b406aef0 2023-10-16T13:49:40.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.679+0000 7f70ba9d3700 1 -- 172.21.15.16:0/2276989099 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70b406aef0 msgr2=0x7f70b40cb980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:40.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.679+0000 7f70ba9d3700 1 --2- 172.21.15.16:0/2276989099 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70b406aef0 0x7f70b40cb980 secure :-1 s=READY pgs=3359 cs=0 l=1 rev1=1 crypto rx=0x7f709c009840 tx=0x7f709c009b50 comp rx=0 tx=0).stop 2023-10-16T13:49:40.687 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.679+0000 7f70ba9d3700 1 -- 172.21.15.16:0/2276989099 shutdown_connections 2023-10-16T13:49:40.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.679+0000 7f70ba9d3700 1 --2- 172.21.15.16:0/2276989099 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70b406aef0 0x7f70b40cb980 unknown :-1 s=CLOSED pgs=3359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:40.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.679+0000 7f70ba9d3700 1 -- 172.21.15.16:0/2276989099 >> 172.21.15.16:0/2276989099 conn(0x7f70b40b1fd0 msgr2=0x7f70b40b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:40.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.679+0000 7f70ba9d3700 1 -- 172.21.15.16:0/2276989099 shutdown_connections 2023-10-16T13:49:40.688 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.679+0000 7f70ba9d3700 1 -- 172.21.15.16:0/2276989099 wait complete. 2023-10-16T13:49:40.689 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.680+0000 7f70ba9d3700 1 Processor -- start 2023-10-16T13:49:40.691 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.681+0000 7f70ba9d3700 1 -- start start 2023-10-16T13:49:40.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.681+0000 7f70ba9d3700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70b406aef0 0x7f70b40af1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:40.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.681+0000 7f70ba9d3700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f70b40ce160 0x7f70b40ccfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:40.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.681+0000 7f70ba9d3700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f70b415db40 0x7f70b40cd500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:40.692 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.681+0000 7f70ba9d3700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70b40c4840 con 0x7f70b406aef0 2023-10-16T13:49:40.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.681+0000 7f70ba9d3700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70b40c4010 con 0x7f70b415db40 2023-10-16T13:49:40.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.681+0000 7f70ba9d3700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70b40af6f0 con 0x7f70b40ce160 2023-10-16T13:49:40.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.681+0000 7f70b3fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70b406aef0 0x7f70b40af1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:40.693 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.681+0000 7f70b3fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70b406aef0 0x7f70b40af1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60578/0 (socket says 172.21.15.16:60578) 2023-10-16T13:49:40.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.681+0000 7f70b3fff700 1 -- 172.21.15.16:0/4080961378 learned_addr learned my addr 172.21.15.16:0/4080961378 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:40.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.682+0000 7f70b8f70700 1 --2- 172.21.15.16:0/4080961378 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f70b415db40 0x7f70b40cd500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:40.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.682+0000 7f70b37fe700 1 --2- 172.21.15.16:0/4080961378 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f70b40ce160 0x7f70b40ccfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:40.694 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.682+0000 7f70b3fff700 1 -- 172.21.15.16:0/4080961378 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f70b40ce160 msgr2=0x7f70b40ccfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:40.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.682+0000 7f70b3fff700 1 --2- 172.21.15.16:0/4080961378 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f70b40ce160 0x7f70b40ccfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:40.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.682+0000 7f70b3fff700 1 -- 172.21.15.16:0/4080961378 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f70b415db40 msgr2=0x7f70b40cd500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:40.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.682+0000 7f70b3fff700 1 --2- 172.21.15.16:0/4080961378 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f70b415db40 0x7f70b40cd500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:40.695 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.682+0000 7f70b3fff700 1 -- 172.21.15.16:0/4080961378 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f709c0094a0 con 0x7f70b406aef0 2023-10-16T13:49:40.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.682+0000 7f70b3fff700 1 --2- 172.21.15.16:0/4080961378 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70b406aef0 0x7f70b40af1b0 secure :-1 s=READY pgs=3360 cs=0 l=1 rev1=1 crypto rx=0x7f70a400c4d0 tx=0x7f70a400c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:40.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.682+0000 7f70b0ff9700 1 -- 172.21.15.16:0/4080961378 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70a4011790 con 0x7f70b406aef0 2023-10-16T13:49:40.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.682+0000 7f70b0ff9700 1 -- 172.21.15.16:0/4080961378 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70a40118f0 con 0x7f70b406aef0 2023-10-16T13:49:40.696 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.682+0000 7f70ba9d3700 1 -- 172.21.15.16:0/4080961378 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70b40c4280 con 0x7f70b406aef0 2023-10-16T13:49:40.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.682+0000 7f70b0ff9700 1 -- 172.21.15.16:0/4080961378 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70a4011ca0 con 0x7f70b406aef0 2023-10-16T13:49:40.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.682+0000 7f70ba9d3700 1 -- 172.21.15.16:0/4080961378 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70b41f7140 con 0x7f70b406aef0 2023-10-16T13:49:40.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.684+0000 7f70b0ff9700 1 -- 172.21.15.16:0/4080961378 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 176) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f70a401c460 con 0x7f70b406aef0 2023-10-16T13:49:40.697 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.684+0000 7f70ba9d3700 1 -- 172.21.15.16:0/4080961378 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70b404ebb0 con 0x7f70b406aef0 2023-10-16T13:49:40.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.684+0000 7f70b0ff9700 1 --2- 172.21.15.16:0/4080961378 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f709406b020 0x7f709406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:40.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.684+0000 7f70b0ff9700 1 -- 172.21.15.16:0/4080961378 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(283..283 src has 1..283) v4 ==== 6208+0+0 (secure 0 0 0) 0x7f70a4089660 con 0x7f70b406aef0 2023-10-16T13:49:40.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.684+0000 7f70b37fe700 1 --2- 172.21.15.16:0/4080961378 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f709406b020 0x7f709406d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:40.698 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.685+0000 7f70b37fe700 1 --2- 172.21.15.16:0/4080961378 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f709406b020 0x7f709406d4e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f709c0090b0 tx=0x7f709c008030 comp rx=0 tx=0).ready entity=mgr.13935 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:40.699 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.689+0000 7f70b0ff9700 1 -- 172.21.15.16:0/4080961378 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f70a4054420 con 0x7f70b406aef0 2023-10-16T13:49:40.895 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.893+0000 7f70ba9d3700 1 -- 172.21.15.16:0/4080961378 --> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7f70b40cb180 con 0x7f709406b020 2023-10-16T13:49:40.896 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.893+0000 7f70b0ff9700 1 -- 172.21.15.16:0/4080961378 <== mgr.13935 v2:172.21.15.16:6832/938897590 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f709c003850 con 0x7f709406b020 2023-10-16T13:49:40.898 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.895+0000 7f70ba9d3700 1 -- 172.21.15.16:0/4080961378 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f709406b020 msgr2=0x7f709406d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:40.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.895+0000 7f70ba9d3700 1 --2- 172.21.15.16:0/4080961378 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f709406b020 0x7f709406d4e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f709c0090b0 tx=0x7f709c008030 comp rx=0 tx=0).stop 2023-10-16T13:49:40.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.895+0000 7f70ba9d3700 1 -- 172.21.15.16:0/4080961378 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70b406aef0 msgr2=0x7f70b40af1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:40.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.895+0000 7f70ba9d3700 1 --2- 172.21.15.16:0/4080961378 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70b406aef0 0x7f70b40af1b0 secure :-1 s=READY pgs=3360 cs=0 l=1 rev1=1 crypto rx=0x7f70a400c4d0 tx=0x7f70a400c7e0 comp rx=0 tx=0).stop 2023-10-16T13:49:40.899 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.895+0000 7f70ba9d3700 1 -- 172.21.15.16:0/4080961378 shutdown_connections 2023-10-16T13:49:40.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.895+0000 7f70ba9d3700 1 --2- 172.21.15.16:0/4080961378 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f709406b020 0x7f709406d4e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:40.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.895+0000 7f70ba9d3700 1 --2- 172.21.15.16:0/4080961378 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f70b415db40 0x7f70b40cd500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:40.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.895+0000 7f70ba9d3700 1 --2- 172.21.15.16:0/4080961378 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70b406aef0 0x7f70b40af1b0 unknown :-1 s=CLOSED pgs=3360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:40.900 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.895+0000 7f70ba9d3700 1 --2- 172.21.15.16:0/4080961378 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f70b40ce160 0x7f70b40ccfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:40.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.895+0000 7f70ba9d3700 1 -- 172.21.15.16:0/4080961378 >> 172.21.15.16:0/4080961378 conn(0x7f70b40b1fd0 msgr2=0x7f70b40cb980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:40.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.896+0000 7f70ba9d3700 1 -- 172.21.15.16:0/4080961378 shutdown_connections 2023-10-16T13:49:40.901 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:40.896+0000 7f70ba9d3700 1 -- 172.21.15.16:0/4080961378 wait complete. 2023-10-16T13:49:40.902 INFO:teuthology.orchestra.run.smithi016.stderr:instructed osd(s) 5 to deep-scrub 2023-10-16T13:49:40.910 INFO:tasks.ceph:Scrubbing osd.6 2023-10-16T13:49:40.911 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2023-10-16T13:49:41.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.037+0000 7fde86332700 1 Processor -- start 2023-10-16T13:49:41.040 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.037+0000 7fde86332700 1 -- start start 2023-10-16T13:49:41.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.038+0000 7fde86332700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fde800cb040 0x7fde8015fab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:41.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.038+0000 7fde86332700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fde8006e590 0x7fde800ccc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:41.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.038+0000 7fde86332700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde8006b220 con 0x7fde8015b500 2023-10-16T13:49:41.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.038+0000 7fde86332700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde80160090 con 0x7fde8006e590 2023-10-16T13:49:41.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.038+0000 7fde86332700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde800cc800 con 0x7fde800cb040 2023-10-16T13:49:41.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.038+0000 7fde848cf700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fde8006e590 0x7fde800ccc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:41.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.038+0000 7fde848cf700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fde8006e590 0x7fde800ccc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60590/0 (socket says 172.21.15.16:60590) 2023-10-16T13:49:41.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.038+0000 7fde848cf700 1 -- 172.21.15.16:0/2006370830 learned_addr learned my addr 172.21.15.16:0/2006370830 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:41.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.038+0000 7fde7f7fe700 1 --2- 172.21.15.16:0/2006370830 >> v2:172.21.15.49:3300/0 conn(0x7fde800cb040 0x7fde8015fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:41.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.038+0000 7fde848cf700 1 -- 172.21.15.16:0/2006370830 >> v2:172.21.15.49:3300/0 conn(0x7fde800cb040 msgr2=0x7fde8015fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:41.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.038+0000 7fde848cf700 1 --2- 172.21.15.16:0/2006370830 >> v2:172.21.15.49:3300/0 conn(0x7fde800cb040 0x7fde8015fab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:41.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.038+0000 7fde848cf700 1 -- 172.21.15.16:0/2006370830 >> v1:172.21.15.49:6789/0 conn(0x7fde8015b500 legacy=0x7fde800ca930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:41.045 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.038+0000 7fde848cf700 1 -- 172.21.15.16:0/2006370830 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde8006a980 con 0x7fde8006e590 2023-10-16T13:49:41.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.038+0000 7fde848cf700 1 --2- 172.21.15.16:0/2006370830 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde8006e590 0x7fde800ccc50 secure :-1 s=READY pgs=3361 cs=0 l=1 rev1=1 crypto rx=0x7fde800cde60 tx=0x7fde7800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3922d559d73f77ec server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:41.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.038+0000 7fde7effd700 1 -- 172.21.15.16:0/2006370830 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde7800e070 con 0x7fde8006e590 2023-10-16T13:49:41.046 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.039+0000 7fde7effd700 1 -- 172.21.15.16:0/2006370830 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde78003a20 con 0x7fde8006e590 2023-10-16T13:49:41.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.039+0000 7fde7effd700 1 -- 172.21.15.16:0/2006370830 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde78003dd0 con 0x7fde8006e590 2023-10-16T13:49:41.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.039+0000 7fde86332700 1 -- 172.21.15.16:0/2006370830 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde8006e590 msgr2=0x7fde800ccc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:41.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.039+0000 7fde86332700 1 --2- 172.21.15.16:0/2006370830 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde8006e590 0x7fde800ccc50 secure :-1 s=READY pgs=3361 cs=0 l=1 rev1=1 crypto rx=0x7fde800cde60 tx=0x7fde7800b330 comp rx=0 tx=0).stop 2023-10-16T13:49:41.047 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.039+0000 7fde86332700 1 -- 172.21.15.16:0/2006370830 shutdown_connections 2023-10-16T13:49:41.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.039+0000 7fde86332700 1 --2- 172.21.15.16:0/2006370830 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde8006e590 0x7fde800ccc50 unknown :-1 s=CLOSED pgs=3361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:41.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.039+0000 7fde86332700 1 --2- 172.21.15.16:0/2006370830 >> v2:172.21.15.49:3300/0 conn(0x7fde800cb040 0x7fde8015fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:41.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.039+0000 7fde86332700 1 -- 172.21.15.16:0/2006370830 >> 172.21.15.16:0/2006370830 conn(0x7fde8006f430 msgr2=0x7fde8006f810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:41.048 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.039+0000 7fde86332700 1 -- 172.21.15.16:0/2006370830 shutdown_connections 2023-10-16T13:49:41.049 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.039+0000 7fde86332700 1 -- 172.21.15.16:0/2006370830 wait complete. 2023-10-16T13:49:41.051 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.040+0000 7fde86332700 1 Processor -- start 2023-10-16T13:49:41.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.041+0000 7fde86332700 1 -- start start 2023-10-16T13:49:41.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.041+0000 7fde86332700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fde8006e590 0x7fde801f2a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:41.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.041+0000 7fde86332700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fde800cb040 0x7fde800ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:41.052 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.041+0000 7fde86332700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde8015b500 0x7fde800ae8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:41.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.041+0000 7fde86332700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde800ad870 con 0x7fde8015b500 2023-10-16T13:49:41.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.041+0000 7fde86332700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde800ad2b0 con 0x7fde800cb040 2023-10-16T13:49:41.053 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.041+0000 7fde86332700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde800ad590 con 0x7fde8006e590 2023-10-16T13:49:41.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.041+0000 7fde848cf700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde8015b500 0x7fde800ae8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:41.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.041+0000 7fde848cf700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde8015b500 0x7fde800ae8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60596/0 (socket says 172.21.15.16:60596) 2023-10-16T13:49:41.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.041+0000 7fde848cf700 1 -- 172.21.15.16:0/1908711426 learned_addr learned my addr 172.21.15.16:0/1908711426 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:41.054 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.041+0000 7fde7f7fe700 1 --2- 172.21.15.16:0/1908711426 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fde800cb040 0x7fde800ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:41.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.042+0000 7fde7ffff700 1 --2- 172.21.15.16:0/1908711426 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fde8006e590 0x7fde801f2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:41.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.042+0000 7fde848cf700 1 -- 172.21.15.16:0/1908711426 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fde8006e590 msgr2=0x7fde801f2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:41.055 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.042+0000 7fde848cf700 1 --2- 172.21.15.16:0/1908711426 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fde8006e590 0x7fde801f2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:41.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.042+0000 7fde848cf700 1 -- 172.21.15.16:0/1908711426 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fde800cb040 msgr2=0x7fde800ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:41.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.042+0000 7fde848cf700 1 --2- 172.21.15.16:0/1908711426 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fde800cb040 0x7fde800ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:41.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.042+0000 7fde848cf700 1 -- 172.21.15.16:0/1908711426 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde78009b30 con 0x7fde8015b500 2023-10-16T13:49:41.056 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.042+0000 7fde7f7fe700 1 --2- 172.21.15.16:0/1908711426 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fde800cb040 0x7fde800ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:49:41.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.042+0000 7fde848cf700 1 --2- 172.21.15.16:0/1908711426 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde8015b500 0x7fde800ae8f0 secure :-1 s=READY pgs=3362 cs=0 l=1 rev1=1 crypto rx=0x7fde78015040 tx=0x7fde78004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:41.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.042+0000 7fde7cff9700 1 -- 172.21.15.16:0/1908711426 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde78004440 con 0x7fde8015b500 2023-10-16T13:49:41.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.042+0000 7fde7cff9700 1 -- 172.21.15.16:0/1908711426 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde780045a0 con 0x7fde8015b500 2023-10-16T13:49:41.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.042+0000 7fde86332700 1 -- 172.21.15.16:0/1908711426 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde800ae0f0 con 0x7fde8015b500 2023-10-16T13:49:41.057 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.042+0000 7fde7cff9700 1 -- 172.21.15.16:0/1908711426 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde78004950 con 0x7fde8015b500 2023-10-16T13:49:41.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.042+0000 7fde86332700 1 -- 172.21.15.16:0/1908711426 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde800add70 con 0x7fde8015b500 2023-10-16T13:49:41.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.043+0000 7fde86332700 1 -- 172.21.15.16:0/1908711426 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fde800ad700 con 0x7fde8015b500 2023-10-16T13:49:41.058 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.044+0000 7fde7cff9700 1 -- 172.21.15.16:0/1908711426 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 176) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fde78012430 con 0x7fde8015b500 2023-10-16T13:49:41.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.044+0000 7fde7cff9700 1 --2- 172.21.15.16:0/1908711426 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7fde6c06b020 0x7fde6c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:41.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.044+0000 7fde7cff9700 1 -- 172.21.15.16:0/1908711426 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(283..283 src has 1..283) v4 ==== 6208+0+0 (secure 0 0 0) 0x7fde7808b010 con 0x7fde8015b500 2023-10-16T13:49:41.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.044+0000 7fde7ffff700 1 --2- 172.21.15.16:0/1908711426 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7fde6c06b020 0x7fde6c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:41.059 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.044+0000 7fde7cff9700 1 --2- 172.21.15.16:0/1908711426 >> [v2:172.21.15.49:6808/3582923927,v1:172.21.15.49:6809/3582923927] conn(0x7fde6c071ac0 0x7fde6c073f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:41.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.044+0000 7fde7cff9700 1 -- 172.21.15.16:0/1908711426 --> [v2:172.21.15.49:6808/3582923927,v1:172.21.15.49:6809/3582923927] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fde6c074630 con 0x7fde6c071ac0 2023-10-16T13:49:41.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.044+0000 7fde7cff9700 1 -- 172.21.15.16:0/1908711426 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=283) v2 ==== 24+0+0 (secure 0 0 0) 0x7fde7808d6b0 con 0x7fde8015b500 2023-10-16T13:49:41.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.044+0000 7fde7f7fe700 1 --2- 172.21.15.16:0/1908711426 >> [v2:172.21.15.49:6808/3582923927,v1:172.21.15.49:6809/3582923927] conn(0x7fde6c071ac0 0x7fde6c073f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:41.060 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.044+0000 7fde7ffff700 1 --2- 172.21.15.16:0/1908711426 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7fde6c06b020 0x7fde6c06d4e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fde70004f30 tx=0x7fde70009040 comp rx=0 tx=0).ready entity=mgr.13935 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:41.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.045+0000 7fde7f7fe700 1 --2- 172.21.15.16:0/1908711426 >> [v2:172.21.15.49:6808/3582923927,v1:172.21.15.49:6809/3582923927] conn(0x7fde6c071ac0 0x7fde6c073f60 crc :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:41.061 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.045+0000 7fde7cff9700 1 -- 172.21.15.16:0/1908711426 <== osd.6 v2:172.21.15.49:6808/3582923927 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7fde6c074630 con 0x7fde6c071ac0 2023-10-16T13:49:41.062 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.060+0000 7fde86332700 1 -- 172.21.15.16:0/1908711426 --> [v2:172.21.15.49:6808/3582923927,v1:172.21.15.49:6809/3582923927] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fde8006af40 con 0x7fde6c071ac0 2023-10-16T13:49:41.063 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.061+0000 7fde7cff9700 1 -- 172.21.15.16:0/1908711426 <== osd.6 v2:172.21.15.49:6808/3582923927 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fde7400abc0 con 0x7fde6c071ac0 2023-10-16T13:49:41.064 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-10-16T13:49:41.064 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-16T13:49:41.064 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-10-16T13:49:41.065 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.061+0000 7fde86332700 1 -- 172.21.15.16:0/1908711426 >> [v2:172.21.15.49:6808/3582923927,v1:172.21.15.49:6809/3582923927] conn(0x7fde6c071ac0 msgr2=0x7fde6c073f60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:41.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.061+0000 7fde86332700 1 --2- 172.21.15.16:0/1908711426 >> [v2:172.21.15.49:6808/3582923927,v1:172.21.15.49:6809/3582923927] conn(0x7fde6c071ac0 0x7fde6c073f60 crc :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:41.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.061+0000 7fde86332700 1 -- 172.21.15.16:0/1908711426 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7fde6c06b020 msgr2=0x7fde6c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:41.066 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.061+0000 7fde86332700 1 --2- 172.21.15.16:0/1908711426 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7fde6c06b020 0x7fde6c06d4e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fde70004f30 tx=0x7fde70009040 comp rx=0 tx=0).stop 2023-10-16T13:49:41.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.061+0000 7fde86332700 1 -- 172.21.15.16:0/1908711426 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde8015b500 msgr2=0x7fde800ae8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:41.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.061+0000 7fde86332700 1 --2- 172.21.15.16:0/1908711426 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde8015b500 0x7fde800ae8f0 secure :-1 s=READY pgs=3362 cs=0 l=1 rev1=1 crypto rx=0x7fde78015040 tx=0x7fde78004ad0 comp rx=0 tx=0).stop 2023-10-16T13:49:41.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.061+0000 7fde848cf700 1 -- 172.21.15.16:0/1908711426 reap_dead start 2023-10-16T13:49:41.067 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.062+0000 7fde86332700 1 -- 172.21.15.16:0/1908711426 shutdown_connections 2023-10-16T13:49:41.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.062+0000 7fde86332700 1 -- 172.21.15.16:0/1908711426 >> 172.21.15.16:0/1908711426 conn(0x7fde8006f430 msgr2=0x7fde800b0890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:41.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.062+0000 7fde86332700 1 -- 172.21.15.16:0/1908711426 shutdown_connections 2023-10-16T13:49:41.068 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.062+0000 7fde86332700 1 -- 172.21.15.16:0/1908711426 wait complete. 2023-10-16T13:49:41.076 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2023-10-16T13:49:41.202 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.200+0000 7fbe44efe700 1 Processor -- start 2023-10-16T13:49:41.203 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.200+0000 7fbe44efe700 1 -- start start 2023-10-16T13:49:41.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.201+0000 7fbe44efe700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fbe400cc8a0 0x7fbe4015e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:41.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.201+0000 7fbe44efe700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbe4006a7e0 0x7fbe4006ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:41.205 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.201+0000 7fbe44efe700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe40161c30 con 0x7fbe4015b280 2023-10-16T13:49:41.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.201+0000 7fbe44efe700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe4006b190 con 0x7fbe400cc8a0 2023-10-16T13:49:41.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.201+0000 7fbe44efe700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe40161f10 con 0x7fbe4006a7e0 2023-10-16T13:49:41.206 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.201+0000 7fbe3ed9d700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbe4006a7e0 0x7fbe4006ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:41.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.201+0000 7fbe3dd9b700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7fbe400cc8a0 0x7fbe4015e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:41.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.201+0000 7fbe3e59c700 1 --1- >> v1:172.21.15.49:6789/0 conn(0x7fbe4015b280 0x7fbe401613e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.49:6789/0 says I am v1:172.21.15.16:57058/0 (socket says 172.21.15.16:57058) 2023-10-16T13:49:41.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.201+0000 7fbe3e59c700 1 -- 172.21.15.16:0/1686463085 learned_addr learned my addr 172.21.15.16:0/1686463085 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:49:41.207 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.201+0000 7fbe3d59a700 1 -- 172.21.15.16:0/1686463085 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2388840391 0 0) 0x7fbe40161c30 con 0x7fbe4015b280 2023-10-16T13:49:41.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.201+0000 7fbe3ed9d700 1 -- 172.21.15.16:0/1686463085 >> v2:172.21.15.49:3300/0 conn(0x7fbe400cc8a0 msgr2=0x7fbe4015e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:41.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.201+0000 7fbe3ed9d700 1 --2- 172.21.15.16:0/1686463085 >> v2:172.21.15.49:3300/0 conn(0x7fbe400cc8a0 0x7fbe4015e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:41.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.201+0000 7fbe3ed9d700 1 -- 172.21.15.16:0/1686463085 >> v1:172.21.15.49:6789/0 conn(0x7fbe4015b280 legacy=0x7fbe401613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:41.208 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.201+0000 7fbe3ed9d700 1 -- 172.21.15.16:0/1686463085 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe401621f0 con 0x7fbe4006a7e0 2023-10-16T13:49:41.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.202+0000 7fbe3ed9d700 1 --2- 172.21.15.16:0/1686463085 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbe4006a7e0 0x7fbe4006ac50 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fbe38009d90 tx=0x7fbe3800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c259ccc821c2e1a8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:41.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.202+0000 7fbe3d59a700 1 -- 172.21.15.16:0/1686463085 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe3800e070 con 0x7fbe4006a7e0 2023-10-16T13:49:41.209 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.202+0000 7fbe3d59a700 1 -- 172.21.15.16:0/1686463085 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe3800bb20 con 0x7fbe4006a7e0 2023-10-16T13:49:41.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.202+0000 7fbe3d59a700 1 -- 172.21.15.16:0/1686463085 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe38012430 con 0x7fbe4006a7e0 2023-10-16T13:49:41.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.202+0000 7fbe44efe700 1 -- 172.21.15.16:0/1686463085 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbe4006a7e0 msgr2=0x7fbe4006ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:41.210 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.202+0000 7fbe44efe700 1 --2- 172.21.15.16:0/1686463085 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbe4006a7e0 0x7fbe4006ac50 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fbe38009d90 tx=0x7fbe3800b330 comp rx=0 tx=0).stop 2023-10-16T13:49:41.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.202+0000 7fbe44efe700 1 -- 172.21.15.16:0/1686463085 shutdown_connections 2023-10-16T13:49:41.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.202+0000 7fbe44efe700 1 --2- 172.21.15.16:0/1686463085 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbe4006a7e0 0x7fbe4006ac50 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:41.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.202+0000 7fbe44efe700 1 --2- 172.21.15.16:0/1686463085 >> v2:172.21.15.49:3300/0 conn(0x7fbe400cc8a0 0x7fbe4015e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:41.211 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.202+0000 7fbe44efe700 1 -- 172.21.15.16:0/1686463085 >> 172.21.15.16:0/1686463085 conn(0x7fbe400b1fd0 msgr2=0x7fbe400b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:41.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.202+0000 7fbe44efe700 1 -- 172.21.15.16:0/1686463085 shutdown_connections 2023-10-16T13:49:41.212 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.202+0000 7fbe44efe700 1 -- 172.21.15.16:0/1686463085 wait complete. 2023-10-16T13:49:41.214 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.204+0000 7fbe44efe700 1 Processor -- start 2023-10-16T13:49:41.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.204+0000 7fbe44efe700 1 -- start start 2023-10-16T13:49:41.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.204+0000 7fbe44efe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe4006a7e0 0x7fbe400c3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:41.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.204+0000 7fbe44efe700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbe400cc8a0 0x7fbe401f2930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:41.215 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.205+0000 7fbe44efe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbe4015b280 0x7fbe400c49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:41.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.205+0000 7fbe44efe700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe401f2e70 con 0x7fbe4006a7e0 2023-10-16T13:49:41.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.205+0000 7fbe44efe700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe40158380 con 0x7fbe4015b280 2023-10-16T13:49:41.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.205+0000 7fbe44efe700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe40158660 con 0x7fbe400cc8a0 2023-10-16T13:49:41.216 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.205+0000 7fbe3e59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe4006a7e0 0x7fbe400c3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:41.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.205+0000 7fbe3e59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe4006a7e0 0x7fbe400c3f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60608/0 (socket says 172.21.15.16:60608) 2023-10-16T13:49:41.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.205+0000 7fbe3e59c700 1 -- 172.21.15.16:0/3006039385 learned_addr learned my addr 172.21.15.16:0/3006039385 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:41.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.205+0000 7fbe3dd9b700 1 --2- 172.21.15.16:0/3006039385 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbe400cc8a0 0x7fbe401f2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:41.217 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.205+0000 7fbe3ed9d700 1 --2- 172.21.15.16:0/3006039385 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbe4015b280 0x7fbe400c49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:41.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.205+0000 7fbe3e59c700 1 -- 172.21.15.16:0/3006039385 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbe400cc8a0 msgr2=0x7fbe401f2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:41.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.205+0000 7fbe3e59c700 1 --2- 172.21.15.16:0/3006039385 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbe400cc8a0 0x7fbe401f2930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:41.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.205+0000 7fbe3e59c700 1 -- 172.21.15.16:0/3006039385 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbe4015b280 msgr2=0x7fbe400c49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:41.218 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.205+0000 7fbe3e59c700 1 --2- 172.21.15.16:0/3006039385 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbe4015b280 0x7fbe400c49f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:41.219 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.205+0000 7fbe3e59c700 1 -- 172.21.15.16:0/3006039385 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe38009a70 con 0x7fbe4006a7e0 2023-10-16T13:49:41.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.205+0000 7fbe3e59c700 1 --2- 172.21.15.16:0/3006039385 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe4006a7e0 0x7fbe400c3f70 secure :-1 s=READY pgs=3363 cs=0 l=1 rev1=1 crypto rx=0x7fbe300011e0 tx=0x7fbe3000ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:41.220 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.205+0000 7fbe36ffd700 1 -- 172.21.15.16:0/3006039385 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe30004c80 con 0x7fbe4006a7e0 2023-10-16T13:49:41.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.205+0000 7fbe44efe700 1 -- 172.21.15.16:0/3006039385 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe400afd30 con 0x7fbe4006a7e0 2023-10-16T13:49:41.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.205+0000 7fbe36ffd700 1 -- 172.21.15.16:0/3006039385 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe30004e00 con 0x7fbe4006a7e0 2023-10-16T13:49:41.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.205+0000 7fbe36ffd700 1 -- 172.21.15.16:0/3006039385 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe30007310 con 0x7fbe4006a7e0 2023-10-16T13:49:41.221 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.205+0000 7fbe44efe700 1 -- 172.21.15.16:0/3006039385 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe40157fa0 con 0x7fbe4006a7e0 2023-10-16T13:49:41.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.206+0000 7fbe44efe700 1 -- 172.21.15.16:0/3006039385 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe400afb90 con 0x7fbe4006a7e0 2023-10-16T13:49:41.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.207+0000 7fbe36ffd700 1 -- 172.21.15.16:0/3006039385 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 176) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fbe30007470 con 0x7fbe4006a7e0 2023-10-16T13:49:41.222 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.207+0000 7fbe36ffd700 1 --2- 172.21.15.16:0/3006039385 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7fbe2006b020 0x7fbe2006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:41.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.207+0000 7fbe36ffd700 1 -- 172.21.15.16:0/3006039385 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(283..283 src has 1..283) v4 ==== 6208+0+0 (secure 0 0 0) 0x7fbe30089660 con 0x7fbe4006a7e0 2023-10-16T13:49:41.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.207+0000 7fbe3dd9b700 1 --2- 172.21.15.16:0/3006039385 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7fbe2006b020 0x7fbe2006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:41.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.208+0000 7fbe3dd9b700 1 --2- 172.21.15.16:0/3006039385 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7fbe2006b020 0x7fbe2006d4e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fbe400b1e90 tx=0x7fbe28009210 comp rx=0 tx=0).ready entity=mgr.13935 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:41.223 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.212+0000 7fbe36ffd700 1 -- 172.21.15.16:0/3006039385 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbe400afb90 con 0x7fbe4006a7e0 2023-10-16T13:49:41.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.413+0000 7fbe44efe700 1 -- 172.21.15.16:0/3006039385 --> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7fbe4015e1e0 con 0x7fbe2006b020 2023-10-16T13:49:41.416 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.414+0000 7fbe36ffd700 1 -- 172.21.15.16:0/3006039385 <== mgr.13935 v2:172.21.15.16:6832/938897590 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fbe28011760 con 0x7fbe2006b020 2023-10-16T13:49:41.418 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.416+0000 7fbe44efe700 1 -- 172.21.15.16:0/3006039385 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7fbe2006b020 msgr2=0x7fbe2006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:41.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.416+0000 7fbe44efe700 1 --2- 172.21.15.16:0/3006039385 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7fbe2006b020 0x7fbe2006d4e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fbe400b1e90 tx=0x7fbe28009210 comp rx=0 tx=0).stop 2023-10-16T13:49:41.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.416+0000 7fbe44efe700 1 -- 172.21.15.16:0/3006039385 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe4006a7e0 msgr2=0x7fbe400c3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:41.419 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.416+0000 7fbe44efe700 1 --2- 172.21.15.16:0/3006039385 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe4006a7e0 0x7fbe400c3f70 secure :-1 s=READY pgs=3363 cs=0 l=1 rev1=1 crypto rx=0x7fbe300011e0 tx=0x7fbe3000ca30 comp rx=0 tx=0).stop 2023-10-16T13:49:41.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.416+0000 7fbe44efe700 1 -- 172.21.15.16:0/3006039385 shutdown_connections 2023-10-16T13:49:41.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.416+0000 7fbe44efe700 1 --2- 172.21.15.16:0/3006039385 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7fbe2006b020 0x7fbe2006d4e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:41.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.416+0000 7fbe44efe700 1 --2- 172.21.15.16:0/3006039385 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbe4015b280 0x7fbe400c49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:41.420 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.416+0000 7fbe44efe700 1 --2- 172.21.15.16:0/3006039385 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe4006a7e0 0x7fbe400c3f70 unknown :-1 s=CLOSED pgs=3363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:41.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.416+0000 7fbe44efe700 1 --2- 172.21.15.16:0/3006039385 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fbe400cc8a0 0x7fbe401f2930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:41.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.416+0000 7fbe44efe700 1 -- 172.21.15.16:0/3006039385 >> 172.21.15.16:0/3006039385 conn(0x7fbe400b1fd0 msgr2=0x7fbe40159bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:41.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.416+0000 7fbe44efe700 1 -- 172.21.15.16:0/3006039385 shutdown_connections 2023-10-16T13:49:41.421 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.416+0000 7fbe44efe700 1 -- 172.21.15.16:0/3006039385 wait complete. 2023-10-16T13:49:41.422 INFO:teuthology.orchestra.run.smithi016.stderr:instructed osd(s) 6 to deep-scrub 2023-10-16T13:49:41.430 INFO:tasks.ceph:Scrubbing osd.7 2023-10-16T13:49:41.431 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2023-10-16T13:49:41.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.557+0000 7ff981b55700 1 Processor -- start 2023-10-16T13:49:41.560 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.557+0000 7ff981b55700 1 -- start start 2023-10-16T13:49:41.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.558+0000 7ff981b55700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7ff97c15da20 0x7ff97c15e070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:41.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.558+0000 7ff981b55700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff97c0cf680 0x7ff97c0cbde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:41.564 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.558+0000 7ff981b55700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff97c0cb930 con 0x7ff97c0cdef0 2023-10-16T13:49:41.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.558+0000 7ff981b55700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff97c15de00 con 0x7ff97c15da20 2023-10-16T13:49:41.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.558+0000 7ff981b55700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff97c15e650 con 0x7ff97c0cf680 2023-10-16T13:49:41.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.558+0000 7ff97affd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff97c0cdef0 0x7ff97c0cef70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49890/0 (socket says 172.21.15.16:49890) 2023-10-16T13:49:41.565 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.558+0000 7ff97affd700 1 -- 172.21.15.16:0/3030356849 learned_addr learned my addr 172.21.15.16:0/3030356849 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:49:41.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.559+0000 7ff97b7fe700 1 --2- 172.21.15.16:0/3030356849 >> v2:172.21.15.49:3300/0 conn(0x7ff97c15da20 0x7ff97c15e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:41.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.559+0000 7ff97a7fc700 1 -- 172.21.15.16:0/3030356849 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2056617557 0 0) 0x7ff97c0cb930 con 0x7ff97c0cdef0 2023-10-16T13:49:41.566 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.559+0000 7ff97bfff700 1 --2- 172.21.15.16:0/3030356849 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff97c0cf680 0x7ff97c0cbde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:41.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.559+0000 7ff97a7fc700 1 -- 172.21.15.16:0/3030356849 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9640035b0 con 0x7ff97c0cdef0 2023-10-16T13:49:41.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.559+0000 7ff97a7fc700 1 -- 172.21.15.16:0/3030356849 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1381919731 0 0) 0x7ff9640035b0 con 0x7ff97c0cdef0 2023-10-16T13:49:41.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.559+0000 7ff97a7fc700 1 -- 172.21.15.16:0/3030356849 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff97c0cf680 msgr2=0x7ff97c0cbde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:41.567 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.559+0000 7ff97a7fc700 1 --2- 172.21.15.16:0/3030356849 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff97c0cf680 0x7ff97c0cbde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:41.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.559+0000 7ff97a7fc700 1 -- 172.21.15.16:0/3030356849 >> v2:172.21.15.49:3300/0 conn(0x7ff97c15da20 msgr2=0x7ff97c15e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:41.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.559+0000 7ff97a7fc700 1 --2- 172.21.15.16:0/3030356849 >> v2:172.21.15.49:3300/0 conn(0x7ff97c15da20 0x7ff97c15e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:41.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.559+0000 7ff97a7fc700 1 -- 172.21.15.16:0/3030356849 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff97c15e930 con 0x7ff97c0cdef0 2023-10-16T13:49:41.568 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.559+0000 7ff97a7fc700 1 -- 172.21.15.16:0/3030356849 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 789850906 0 0) 0x7ff970001670 con 0x7ff97c0cdef0 2023-10-16T13:49:41.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.559+0000 7ff97a7fc700 1 -- 172.21.15.16:0/3030356849 >> v1:172.21.15.16:6789/0 conn(0x7ff97c0cdef0 legacy=0x7ff97c0cef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:41.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.559+0000 7ff97a7fc700 1 --2- 172.21.15.16:0/3030356849 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff964003a00 0x7ff964003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:41.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.559+0000 7ff97a7fc700 1 --2- 172.21.15.16:0/3030356849 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff964004850 0x7ff964004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:41.569 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.559+0000 7ff97a7fc700 1 --2- 172.21.15.16:0/3030356849 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff97c0cf680 0x7ff964005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:41.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.559+0000 7ff97a7fc700 1 -- 172.21.15.16:0/3030356849 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff97c15de00 con 0x7ff964004850 2023-10-16T13:49:41.570 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.559+0000 7ff97a7fc700 1 -- 172.21.15.16:0/3030356849 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff97c15e650 con 0x7ff964003a00 2023-10-16T13:49:41.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.559+0000 7ff97a7fc700 1 -- 172.21.15.16:0/3030356849 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff96400ba10 con 0x7ff97c0cf680 2023-10-16T13:49:41.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.559+0000 7ff97affd700 1 --2- 172.21.15.16:0/3030356849 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff964004850 0x7ff964004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:41.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.559+0000 7ff97b7fe700 1 --2- 172.21.15.16:0/3030356849 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff964003a00 0x7ff964003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:41.571 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.559+0000 7ff97bfff700 1 --2- 172.21.15.16:0/3030356849 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff97c0cf680 0x7ff964005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:41.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.559+0000 7ff97affd700 1 -- 172.21.15.16:0/3030356849 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff97c0cf680 msgr2=0x7ff964005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:41.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.559+0000 7ff97affd700 1 --2- 172.21.15.16:0/3030356849 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff97c0cf680 0x7ff964005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:41.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.559+0000 7ff97affd700 1 -- 172.21.15.16:0/3030356849 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff964003a00 msgr2=0x7ff964003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:41.572 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.559+0000 7ff97affd700 1 --2- 172.21.15.16:0/3030356849 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff964003a00 0x7ff964003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:41.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.560+0000 7ff97affd700 1 -- 172.21.15.16:0/3030356849 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9640035b0 con 0x7ff964004850 2023-10-16T13:49:41.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.560+0000 7ff97affd700 1 --2- 172.21.15.16:0/3030356849 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff964004850 0x7ff964004cd0 secure :-1 s=READY pgs=3365 cs=0 l=1 rev1=1 crypto rx=0x7ff970001050 tx=0x7ff964005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64c3cdb2d2dd7cb3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:41.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.560+0000 7ff97a7fc700 1 -- 172.21.15.16:0/3030356849 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff97000cd50 con 0x7ff964004850 2023-10-16T13:49:41.573 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.560+0000 7ff97a7fc700 1 -- 172.21.15.16:0/3030356849 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff970005960 con 0x7ff964004850 2023-10-16T13:49:41.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.560+0000 7ff97a7fc700 1 -- 172.21.15.16:0/3030356849 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff970005d10 con 0x7ff964004850 2023-10-16T13:49:41.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.560+0000 7ff981b55700 1 -- 172.21.15.16:0/3030356849 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff964004850 msgr2=0x7ff964004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:41.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.560+0000 7ff981b55700 1 --2- 172.21.15.16:0/3030356849 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff964004850 0x7ff964004cd0 secure :-1 s=READY pgs=3365 cs=0 l=1 rev1=1 crypto rx=0x7ff970001050 tx=0x7ff964005880 comp rx=0 tx=0).stop 2023-10-16T13:49:41.574 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.560+0000 7ff97bfff700 1 -- 172.21.15.16:0/3030356849 reap_dead start 2023-10-16T13:49:41.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.560+0000 7ff981b55700 1 -- 172.21.15.16:0/3030356849 shutdown_connections 2023-10-16T13:49:41.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.560+0000 7ff981b55700 1 -- 172.21.15.16:0/3030356849 >> 172.21.15.16:0/3030356849 conn(0x7ff97c0b1ed0 msgr2=0x7ff97c0b22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:41.575 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.560+0000 7ff981b55700 1 -- 172.21.15.16:0/3030356849 shutdown_connections 2023-10-16T13:49:41.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.560+0000 7ff981b55700 1 -- 172.21.15.16:0/3030356849 wait complete. 2023-10-16T13:49:41.576 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.562+0000 7ff981b55700 1 Processor -- start 2023-10-16T13:49:41.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.562+0000 7ff981b55700 1 -- start start 2023-10-16T13:49:41.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.563+0000 7ff981b55700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff964004850 0x7ff97c154900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:41.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.563+0000 7ff981b55700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff97c0b2fc0 0x7ff97c0ad8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:41.579 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.563+0000 7ff981b55700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff97c0b3440 0x7ff97c0ade30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:41.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.563+0000 7ff981b55700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff97c0ae370 con 0x7ff97c0b3440 2023-10-16T13:49:41.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.563+0000 7ff981b55700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff97c0ae5a0 con 0x7ff964004850 2023-10-16T13:49:41.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.563+0000 7ff981b55700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff97c0ae880 con 0x7ff97c0b2fc0 2023-10-16T13:49:41.580 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.563+0000 7ff97bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff97c0b3440 0x7ff97c0ade30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:41.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.563+0000 7ff97bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff97c0b3440 0x7ff97c0ade30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60632/0 (socket says 172.21.15.16:60632) 2023-10-16T13:49:41.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.563+0000 7ff97bfff700 1 -- 172.21.15.16:0/2771328154 learned_addr learned my addr 172.21.15.16:0/2771328154 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:41.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.563+0000 7ff97b7fe700 1 --2- 172.21.15.16:0/2771328154 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff964004850 0x7ff97c154900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:41.581 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.563+0000 7ff97affd700 1 --2- 172.21.15.16:0/2771328154 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff97c0b2fc0 0x7ff97c0ad8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:41.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.563+0000 7ff97bfff700 1 -- 172.21.15.16:0/2771328154 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff97c0b2fc0 msgr2=0x7ff97c0ad8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:41.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.563+0000 7ff97bfff700 1 --2- 172.21.15.16:0/2771328154 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7ff97c0b2fc0 0x7ff97c0ad8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:41.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.563+0000 7ff97bfff700 1 -- 172.21.15.16:0/2771328154 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff964004850 msgr2=0x7ff97c154900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:41.582 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.563+0000 7ff97bfff700 1 --2- 172.21.15.16:0/2771328154 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff964004850 0x7ff97c154900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:41.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.563+0000 7ff97bfff700 1 -- 172.21.15.16:0/2771328154 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff97c0aeb60 con 0x7ff97c0b3440 2023-10-16T13:49:41.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.563+0000 7ff97bfff700 1 --2- 172.21.15.16:0/2771328154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff97c0b3440 0x7ff97c0ade30 secure :-1 s=READY pgs=3366 cs=0 l=1 rev1=1 crypto rx=0x7ff97c067420 tx=0x7ff974011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:41.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.563+0000 7ff963fff700 1 -- 172.21.15.16:0/2771328154 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff974004300 con 0x7ff97c0b3440 2023-10-16T13:49:41.583 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.563+0000 7ff963fff700 1 -- 172.21.15.16:0/2771328154 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff974004460 con 0x7ff97c0b3440 2023-10-16T13:49:41.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.563+0000 7ff981b55700 1 -- 172.21.15.16:0/2771328154 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff97c0aee40 con 0x7ff97c0b3440 2023-10-16T13:49:41.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.563+0000 7ff963fff700 1 -- 172.21.15.16:0/2771328154 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9740038e0 con 0x7ff97c0b3440 2023-10-16T13:49:41.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.563+0000 7ff981b55700 1 -- 172.21.15.16:0/2771328154 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff97c0af8e0 con 0x7ff97c0b3440 2023-10-16T13:49:41.584 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.564+0000 7ff981b55700 1 -- 172.21.15.16:0/2771328154 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff97c0ae9f0 con 0x7ff97c0b3440 2023-10-16T13:49:41.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.565+0000 7ff963fff700 1 -- 172.21.15.16:0/2771328154 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 176) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff974003a40 con 0x7ff97c0b3440 2023-10-16T13:49:41.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.565+0000 7ff963fff700 1 --2- 172.21.15.16:0/2771328154 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7ff96806afe0 0x7ff96806d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:41.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.565+0000 7ff963fff700 1 -- 172.21.15.16:0/2771328154 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(283..283 src has 1..283) v4 ==== 6208+0+0 (secure 0 0 0) 0x7ff974089a10 con 0x7ff97c0b3440 2023-10-16T13:49:41.585 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.565+0000 7ff97b7fe700 1 --2- 172.21.15.16:0/2771328154 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7ff96806afe0 0x7ff96806d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:41.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.565+0000 7ff963fff700 1 --2- 172.21.15.16:0/2771328154 >> [v2:172.21.15.49:6824/2356409928,v1:172.21.15.49:6825/2356409928] conn(0x7ff968071a80 0x7ff968073f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:41.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.565+0000 7ff963fff700 1 -- 172.21.15.16:0/2771328154 --> [v2:172.21.15.49:6824/2356409928,v1:172.21.15.49:6825/2356409928] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff9680745f0 con 0x7ff968071a80 2023-10-16T13:49:41.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.565+0000 7ff963fff700 1 -- 172.21.15.16:0/2771328154 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=283) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff97408e2a0 con 0x7ff97c0b3440 2023-10-16T13:49:41.586 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.566+0000 7ff97affd700 1 --2- 172.21.15.16:0/2771328154 >> [v2:172.21.15.49:6824/2356409928,v1:172.21.15.49:6825/2356409928] conn(0x7ff968071a80 0x7ff968073f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:41.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.566+0000 7ff97b7fe700 1 --2- 172.21.15.16:0/2771328154 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7ff96806afe0 0x7ff96806d4a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff96c009fd0 tx=0x7ff96c00b040 comp rx=0 tx=0).ready entity=mgr.13935 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:41.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.566+0000 7ff97affd700 1 --2- 172.21.15.16:0/2771328154 >> [v2:172.21.15.49:6824/2356409928,v1:172.21.15.49:6825/2356409928] conn(0x7ff968071a80 0x7ff968073f20 crc :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:41.587 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.567+0000 7ff963fff700 1 -- 172.21.15.16:0/2771328154 <== osd.7 v2:172.21.15.49:6824/2356409928 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7ff9680745f0 con 0x7ff968071a80 2023-10-16T13:49:41.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.582+0000 7ff981b55700 1 -- 172.21.15.16:0/2771328154 --> [v2:172.21.15.49:6824/2356409928,v1:172.21.15.49:6825/2356409928] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7ff97c15eef0 con 0x7ff968071a80 2023-10-16T13:49:41.588 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.582+0000 7ff963fff700 1 -- 172.21.15.16:0/2771328154 <== osd.7 v2:172.21.15.49:6824/2356409928 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7ff97000cdf0 con 0x7ff968071a80 2023-10-16T13:49:41.588 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-10-16T13:49:41.589 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-16T13:49:41.589 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-10-16T13:49:41.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.582+0000 7ff981b55700 1 -- 172.21.15.16:0/2771328154 >> [v2:172.21.15.49:6824/2356409928,v1:172.21.15.49:6825/2356409928] conn(0x7ff968071a80 msgr2=0x7ff968073f20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:41.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.582+0000 7ff981b55700 1 --2- 172.21.15.16:0/2771328154 >> [v2:172.21.15.49:6824/2356409928,v1:172.21.15.49:6825/2356409928] conn(0x7ff968071a80 0x7ff968073f20 crc :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:41.590 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.582+0000 7ff981b55700 1 -- 172.21.15.16:0/2771328154 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7ff96806afe0 msgr2=0x7ff96806d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:41.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.582+0000 7ff981b55700 1 --2- 172.21.15.16:0/2771328154 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7ff96806afe0 0x7ff96806d4a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff96c009fd0 tx=0x7ff96c00b040 comp rx=0 tx=0).stop 2023-10-16T13:49:41.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.582+0000 7ff981b55700 1 -- 172.21.15.16:0/2771328154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff97c0b3440 msgr2=0x7ff97c0ade30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:41.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.582+0000 7ff981b55700 1 --2- 172.21.15.16:0/2771328154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff97c0b3440 0x7ff97c0ade30 secure :-1 s=READY pgs=3366 cs=0 l=1 rev1=1 crypto rx=0x7ff97c067420 tx=0x7ff974011bf0 comp rx=0 tx=0).stop 2023-10-16T13:49:41.591 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.582+0000 7ff97bfff700 1 -- 172.21.15.16:0/2771328154 reap_dead start 2023-10-16T13:49:41.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.582+0000 7ff981b55700 1 -- 172.21.15.16:0/2771328154 shutdown_connections 2023-10-16T13:49:41.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.582+0000 7ff981b55700 1 -- 172.21.15.16:0/2771328154 >> 172.21.15.16:0/2771328154 conn(0x7ff97c0b1ed0 msgr2=0x7ff97c0ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:41.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.583+0000 7ff981b55700 1 -- 172.21.15.16:0/2771328154 shutdown_connections 2023-10-16T13:49:41.592 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.583+0000 7ff981b55700 1 -- 172.21.15.16:0/2771328154 wait complete. 2023-10-16T13:49:41.597 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2023-10-16T13:49:41.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.720+0000 7f007c102700 1 Processor -- start 2023-10-16T13:49:41.723 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.720+0000 7f007c102700 1 -- start start 2023-10-16T13:49:41.725 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.721+0000 7f007c102700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f007415b630 0x7f00740cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:41.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.721+0000 7f007c102700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f00740ce4f0 0x7f00740cfca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:41.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.721+0000 7f007c102700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00740cc2a0 con 0x7f007415ba10 2023-10-16T13:49:41.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.721+0000 7f007c102700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00740cdb20 con 0x7f00740ce4f0 2023-10-16T13:49:41.726 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.721+0000 7f007c102700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00740cc540 con 0x7f007415b630 2023-10-16T13:49:41.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.721+0000 7f007a69f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f00740ce4f0 0x7f00740cfca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:41.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.721+0000 7f007969d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f007415ba10 0x7f00740cdde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49912/0 (socket says 172.21.15.16:49912) 2023-10-16T13:49:41.727 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.721+0000 7f007969d700 1 -- 172.21.15.16:0/483330105 learned_addr learned my addr 172.21.15.16:0/483330105 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:49:41.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.721+0000 7f007a69f700 1 --2- 172.21.15.16:0/483330105 >> v2:172.21.15.16:3300/0 conn(0x7f00740ce4f0 0x7f00740cfca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60640/0 (socket says 172.21.15.16:60640) 2023-10-16T13:49:41.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.722+0000 7f0079e9e700 1 --2- 172.21.15.16:0/483330105 >> v2:172.21.15.49:3300/0 conn(0x7f007415b630 0x7f00740cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:41.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.722+0000 7f007a69f700 1 -- 172.21.15.16:0/483330105 >> v2:172.21.15.49:3300/0 conn(0x7f007415b630 msgr2=0x7f00740cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:41.728 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.722+0000 7f007a69f700 1 --2- 172.21.15.16:0/483330105 >> v2:172.21.15.49:3300/0 conn(0x7f007415b630 0x7f00740cd5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:41.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.722+0000 7f007a69f700 1 -- 172.21.15.16:0/483330105 >> v1:172.21.15.16:6789/0 conn(0x7f007415ba10 legacy=0x7f00740cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:41.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.722+0000 7f007a69f700 1 -- 172.21.15.16:0/483330105 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00740cc820 con 0x7f00740ce4f0 2023-10-16T13:49:41.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.722+0000 7f007a69f700 1 --2- 172.21.15.16:0/483330105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00740ce4f0 0x7f00740cfca0 secure :-1 s=READY pgs=3368 cs=0 l=1 rev1=1 crypto rx=0x7f00740cefb0 tx=0x7f006c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97da25480625061e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:41.729 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.722+0000 7f0078e9c700 1 -- 172.21.15.16:0/483330105 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f006c00e070 con 0x7f00740ce4f0 2023-10-16T13:49:41.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.722+0000 7f0078e9c700 1 -- 172.21.15.16:0/483330105 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f006c00bbd0 con 0x7f00740ce4f0 2023-10-16T13:49:41.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.722+0000 7f0078e9c700 1 -- 172.21.15.16:0/483330105 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f006c012430 con 0x7f00740ce4f0 2023-10-16T13:49:41.730 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.722+0000 7f007c102700 1 -- 172.21.15.16:0/483330105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00740ce4f0 msgr2=0x7f00740cfca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:41.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.722+0000 7f007c102700 1 --2- 172.21.15.16:0/483330105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00740ce4f0 0x7f00740cfca0 secure :-1 s=READY pgs=3368 cs=0 l=1 rev1=1 crypto rx=0x7f00740cefb0 tx=0x7f006c00b330 comp rx=0 tx=0).stop 2023-10-16T13:49:41.731 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.723+0000 7f007c102700 1 -- 172.21.15.16:0/483330105 shutdown_connections 2023-10-16T13:49:41.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.723+0000 7f007c102700 1 --2- 172.21.15.16:0/483330105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00740ce4f0 0x7f00740cfca0 unknown :-1 s=CLOSED pgs=3368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:41.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.723+0000 7f007c102700 1 --2- 172.21.15.16:0/483330105 >> v2:172.21.15.49:3300/0 conn(0x7f007415b630 0x7f00740cd5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:41.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.723+0000 7f007c102700 1 -- 172.21.15.16:0/483330105 >> 172.21.15.16:0/483330105 conn(0x7f00740d14b0 msgr2=0x7f00740d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:41.732 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.723+0000 7f007c102700 1 -- 172.21.15.16:0/483330105 shutdown_connections 2023-10-16T13:49:41.733 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.723+0000 7f007c102700 1 -- 172.21.15.16:0/483330105 wait complete. 2023-10-16T13:49:41.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.724+0000 7f007c102700 1 Processor -- start 2023-10-16T13:49:41.735 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.724+0000 7f007c102700 1 -- start start 2023-10-16T13:49:41.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.725+0000 7f007c102700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f00740ce4f0 0x7f00740c8580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:41.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.725+0000 7f007c102700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f007415b630 0x7f00741f2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:41.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.725+0000 7f007c102700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f007415ba10 0x7f00740c6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:41.736 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.725+0000 7f007c102700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00740cb740 con 0x7f007415ba10 2023-10-16T13:49:41.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.725+0000 7f007c102700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00740cac30 con 0x7f00740ce4f0 2023-10-16T13:49:41.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.725+0000 7f007c102700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00740caea0 con 0x7f007415b630 2023-10-16T13:49:41.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.725+0000 7f007a69f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f007415ba10 0x7f00740c6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:41.737 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.725+0000 7f007a69f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f007415ba10 0x7f00740c6180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60646/0 (socket says 172.21.15.16:60646) 2023-10-16T13:49:41.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.725+0000 7f007a69f700 1 -- 172.21.15.16:0/293367169 learned_addr learned my addr 172.21.15.16:0/293367169 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:41.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.725+0000 7f0079e9e700 1 --2- 172.21.15.16:0/293367169 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f00740ce4f0 0x7f00740c8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:41.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.725+0000 7f007969d700 1 --2- 172.21.15.16:0/293367169 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f007415b630 0x7f00741f2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:41.738 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.725+0000 7f007a69f700 1 -- 172.21.15.16:0/293367169 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f007415b630 msgr2=0x7f00741f2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:41.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.725+0000 7f007a69f700 1 --2- 172.21.15.16:0/293367169 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f007415b630 0x7f00741f2dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:41.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.725+0000 7f007a69f700 1 -- 172.21.15.16:0/293367169 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f00740ce4f0 msgr2=0x7f00740c8580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:41.739 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.725+0000 7f007a69f700 1 --2- 172.21.15.16:0/293367169 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f00740ce4f0 0x7f00740c8580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:41.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.725+0000 7f007a69f700 1 -- 172.21.15.16:0/293367169 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f006c009a70 con 0x7f007415ba10 2023-10-16T13:49:41.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.725+0000 7f007969d700 1 --2- 172.21.15.16:0/293367169 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f007415b630 0x7f00741f2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T13:49:41.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.725+0000 7f007a69f700 1 --2- 172.21.15.16:0/293367169 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f007415ba10 0x7f00740c6180 secure :-1 s=READY pgs=3369 cs=0 l=1 rev1=1 crypto rx=0x7f006c014040 tx=0x7f006c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:41.740 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.725+0000 7f006a7fc700 1 -- 172.21.15.16:0/293367169 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f006c012430 con 0x7f007415ba10 2023-10-16T13:49:41.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.725+0000 7f006a7fc700 1 -- 172.21.15.16:0/293367169 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f006c020070 con 0x7f007415ba10 2023-10-16T13:49:41.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.725+0000 7f007c102700 1 -- 172.21.15.16:0/293367169 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00740cb180 con 0x7f007415ba10 2023-10-16T13:49:41.741 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.725+0000 7f006a7fc700 1 -- 172.21.15.16:0/293367169 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f006c004280 con 0x7f007415ba10 2023-10-16T13:49:41.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.725+0000 7f007c102700 1 -- 172.21.15.16:0/293367169 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00740c8ac0 con 0x7f007415ba10 2023-10-16T13:49:41.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.726+0000 7f007c102700 1 -- 172.21.15.16:0/293367169 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00740c8c80 con 0x7f007415ba10 2023-10-16T13:49:41.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.727+0000 7f006a7fc700 1 -- 172.21.15.16:0/293367169 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 176) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f006c0043e0 con 0x7f007415ba10 2023-10-16T13:49:41.742 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.727+0000 7f006a7fc700 1 --2- 172.21.15.16:0/293367169 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f005806b020 0x7f005806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:41.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.727+0000 7f006a7fc700 1 -- 172.21.15.16:0/293367169 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(283..283 src has 1..283) v4 ==== 6208+0+0 (secure 0 0 0) 0x7f006c022070 con 0x7f007415ba10 2023-10-16T13:49:41.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.727+0000 7f0079e9e700 1 --2- 172.21.15.16:0/293367169 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f005806b020 0x7f005806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:41.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.728+0000 7f0079e9e700 1 --2- 172.21.15.16:0/293367169 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f005806b020 0x7f005806d4e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0064006fd0 tx=0x7f0064008040 comp rx=0 tx=0).ready entity=mgr.13935 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:41.743 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.732+0000 7f006a7fc700 1 -- 172.21.15.16:0/293367169 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f00740c8c80 con 0x7f007415ba10 2023-10-16T13:49:41.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.940+0000 7f007c102700 1 -- 172.21.15.16:0/293367169 --> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7f00740ccde0 con 0x7f005806b020 2023-10-16T13:49:41.943 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.940+0000 7f006a7fc700 1 -- 172.21.15.16:0/293367169 <== mgr.13935 v2:172.21.15.16:6832/938897590 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f0064009df0 con 0x7f005806b020 2023-10-16T13:49:41.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.942+0000 7f007c102700 1 -- 172.21.15.16:0/293367169 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f005806b020 msgr2=0x7f005806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:41.945 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.942+0000 7f007c102700 1 --2- 172.21.15.16:0/293367169 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f005806b020 0x7f005806d4e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0064006fd0 tx=0x7f0064008040 comp rx=0 tx=0).stop 2023-10-16T13:49:41.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.942+0000 7f007c102700 1 -- 172.21.15.16:0/293367169 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f007415ba10 msgr2=0x7f00740c6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:41.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.942+0000 7f007c102700 1 --2- 172.21.15.16:0/293367169 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f007415ba10 0x7f00740c6180 secure :-1 s=READY pgs=3369 cs=0 l=1 rev1=1 crypto rx=0x7f006c014040 tx=0x7f006c004ad0 comp rx=0 tx=0).stop 2023-10-16T13:49:41.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.942+0000 7f007c102700 1 -- 172.21.15.16:0/293367169 shutdown_connections 2023-10-16T13:49:41.946 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.942+0000 7f007c102700 1 --2- 172.21.15.16:0/293367169 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f005806b020 0x7f005806d4e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:41.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.942+0000 7f007c102700 1 --2- 172.21.15.16:0/293367169 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f007415ba10 0x7f00740c6180 unknown :-1 s=CLOSED pgs=3369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:41.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.942+0000 7f007c102700 1 --2- 172.21.15.16:0/293367169 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f00740ce4f0 0x7f00740c8580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:41.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.942+0000 7f007c102700 1 --2- 172.21.15.16:0/293367169 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f007415b630 0x7f00741f2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:41.947 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.942+0000 7f007c102700 1 -- 172.21.15.16:0/293367169 >> 172.21.15.16:0/293367169 conn(0x7f00740d14b0 msgr2=0x7f00740d1890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:41.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.942+0000 7f007c102700 1 -- 172.21.15.16:0/293367169 shutdown_connections 2023-10-16T13:49:41.948 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:41.942+0000 7f007c102700 1 -- 172.21.15.16:0/293367169 wait complete. 2023-10-16T13:49:41.948 INFO:teuthology.orchestra.run.smithi016.stderr:instructed osd(s) 7 to deep-scrub 2023-10-16T13:49:41.956 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T13:49:42.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:42.079+0000 7fefa2a33700 1 Processor -- start 2023-10-16T13:49:42.082 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:42.079+0000 7fefa2a33700 1 -- start start 2023-10-16T13:49:42.084 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:42.080+0000 7fefa2a33700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fef9c06aef0 0x7fef9c0cb980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:42.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:42.080+0000 7fefa2a33700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fef9c0ce160 0x7fef9c0ce610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:42.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:42.080+0000 7fefa2a33700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef9c0cedb0 con 0x7fef9c15db40 2023-10-16T13:49:42.085 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:42.080+0000 7fefa2a33700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef9c15df20 con 0x7fef9c0ce160 2023-10-16T13:49:42.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:42.080+0000 7fefa2a33700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef9c0d12c0 con 0x7fef9c06aef0 2023-10-16T13:49:42.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:42.080+0000 7fefa0fd0700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fef9c0ce160 0x7fef9c0ce610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:42.086 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:42.080+0000 7fefa0fd0700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fef9c0ce160 0x7fef9c0ce610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60654/0 (socket says 172.21.15.16:60654) 2023-10-16T13:49:42.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:42.080+0000 7fefa0fd0700 1 -- 172.21.15.16:0/1904142433 learned_addr learned my addr 172.21.15.16:0/1904142433 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:42.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:42.081+0000 7fef9b7fe700 1 --2- 172.21.15.16:0/1904142433 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fef9c06aef0 0x7fef9c0cb980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:42.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:42.081+0000 7fefa0fd0700 1 -- 172.21.15.16:0/1904142433 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fef9c06aef0 msgr2=0x7fef9c0cb980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:42.087 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:42.081+0000 7fefa0fd0700 1 --2- 172.21.15.16:0/1904142433 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fef9c06aef0 0x7fef9c0cb980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:42.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:42.081+0000 7fefa0fd0700 1 -- 172.21.15.16:0/1904142433 >> v1:172.21.15.49:6789/0 conn(0x7fef9c15db40 legacy=0x7fef9c06a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:49:42.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:42.081+0000 7fefa0fd0700 1 -- 172.21.15.16:0/1904142433 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef9c0ceb50 con 0x7fef9c0ce160 2023-10-16T13:49:42.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:42.081+0000 7fefa0fd0700 1 --2- 172.21.15.16:0/1904142433 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef9c0ce160 0x7fef9c0ce610 secure :-1 s=READY pgs=3370 cs=0 l=1 rev1=1 crypto rx=0x7fef94009e50 tx=0x7fef9400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0137235d9146e01 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:42.088 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:42.081+0000 7fef9affd700 1 -- 172.21.15.16:0/1904142433 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef9400e070 con 0x7fef9c0ce160 2023-10-16T13:49:42.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:42.081+0000 7fef9affd700 1 -- 172.21.15.16:0/1904142433 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef94003a20 con 0x7fef9c0ce160 2023-10-16T13:49:42.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:42.081+0000 7fef9affd700 1 -- 172.21.15.16:0/1904142433 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef94003dd0 con 0x7fef9c0ce160 2023-10-16T13:49:42.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:42.082+0000 7fefa2a33700 1 -- 172.21.15.16:0/1904142433 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef9c0ce160 msgr2=0x7fef9c0ce610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:42.089 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:42.082+0000 7fefa2a33700 1 --2- 172.21.15.16:0/1904142433 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef9c0ce160 0x7fef9c0ce610 secure :-1 s=READY pgs=3370 cs=0 l=1 rev1=1 crypto rx=0x7fef94009e50 tx=0x7fef9400b330 comp rx=0 tx=0).stop 2023-10-16T13:49:42.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:42.082+0000 7fefa2a33700 1 -- 172.21.15.16:0/1904142433 shutdown_connections 2023-10-16T13:49:42.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:42.082+0000 7fefa2a33700 1 --2- 172.21.15.16:0/1904142433 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef9c0ce160 0x7fef9c0ce610 unknown :-1 s=CLOSED pgs=3370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:42.090 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:42.082+0000 7fefa2a33700 1 --2- 172.21.15.16:0/1904142433 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fef9c06aef0 0x7fef9c0cb980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:42.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:42.082+0000 7fefa2a33700 1 -- 172.21.15.16:0/1904142433 >> 172.21.15.16:0/1904142433 conn(0x7fef9c0b1fd0 msgr2=0x7fef9c0b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:42.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:42.082+0000 7fefa2a33700 1 -- 172.21.15.16:0/1904142433 shutdown_connections 2023-10-16T13:49:42.091 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:42.082+0000 7fefa2a33700 1 -- 172.21.15.16:0/1904142433 wait complete. 2023-10-16T13:49:42.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:42.083+0000 7fefa2a33700 1 Processor -- start 2023-10-16T13:49:42.094 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:42.083+0000 7fefa2a33700 1 -- start start 2023-10-16T13:49:42.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:42.084+0000 7fefa2a33700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef9c06aef0 0x7fef9c0b1200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:42.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:42.084+0000 7fefa2a33700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fef9c0ce160 0x7fef9c160250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:42.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:42.084+0000 7fefa2a33700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fef9c15db40 0x7fef9c160790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:42.095 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:42.084+0000 7fefa2a33700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef9c0c6a50 con 0x7fef9c06aef0 2023-10-16T13:49:42.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:42.084+0000 7fefa2a33700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef9c0c5ed0 con 0x7fef9c15db40 2023-10-16T13:49:42.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:42.084+0000 7fefa2a33700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef9c0c61b0 con 0x7fef9c0ce160 2023-10-16T13:49:42.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:42.084+0000 7fef9bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef9c06aef0 0x7fef9c0b1200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:42.096 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:42.084+0000 7fef9bfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef9c06aef0 0x7fef9c0b1200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60656/0 (socket says 172.21.15.16:60656) 2023-10-16T13:49:42.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:42.084+0000 7fef9bfff700 1 -- 172.21.15.16:0/698755766 learned_addr learned my addr 172.21.15.16:0/698755766 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:49:42.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:42.085+0000 7fef9b7fe700 1 --2- 172.21.15.16:0/698755766 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fef9c0ce160 0x7fef9c160250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:42.097 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:42.085+0000 7fefa0fd0700 1 --2- 172.21.15.16:0/698755766 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fef9c15db40 0x7fef9c160790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:42.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:42.085+0000 7fef9bfff700 1 -- 172.21.15.16:0/698755766 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fef9c0ce160 msgr2=0x7fef9c160250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:42.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:42.085+0000 7fef9bfff700 1 --2- 172.21.15.16:0/698755766 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fef9c0ce160 0x7fef9c160250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:42.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:42.085+0000 7fef9bfff700 1 -- 172.21.15.16:0/698755766 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fef9c15db40 msgr2=0x7fef9c160790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:42.098 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:42.085+0000 7fef9bfff700 1 --2- 172.21.15.16:0/698755766 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fef9c15db40 0x7fef9c160790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:42.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:42.085+0000 7fef9bfff700 1 -- 172.21.15.16:0/698755766 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef94009b30 con 0x7fef9c06aef0 2023-10-16T13:49:42.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:42.085+0000 7fefa0fd0700 1 --2- 172.21.15.16:0/698755766 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fef9c15db40 0x7fef9c160790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:49:42.099 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:42.085+0000 7fef9b7fe700 1 --2- 172.21.15.16:0/698755766 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fef9c0ce160 0x7fef9c160250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:49:42.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:42.085+0000 7fef9bfff700 1 --2- 172.21.15.16:0/698755766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef9c06aef0 0x7fef9c0b1200 secure :-1 s=READY pgs=3371 cs=0 l=1 rev1=1 crypto rx=0x7fef8c00c4d0 tx=0x7fef8c00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:42.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:42.085+0000 7fef98ff9700 1 -- 172.21.15.16:0/698755766 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef8c011790 con 0x7fef9c06aef0 2023-10-16T13:49:42.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:42.085+0000 7fef98ff9700 1 -- 172.21.15.16:0/698755766 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef8c0118f0 con 0x7fef9c06aef0 2023-10-16T13:49:42.100 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:42.085+0000 7fefa2a33700 1 -- 172.21.15.16:0/698755766 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef9c0c6490 con 0x7fef9c06aef0 2023-10-16T13:49:42.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:42.085+0000 7fef98ff9700 1 -- 172.21.15.16:0/698755766 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef8c011ca0 con 0x7fef9c06aef0 2023-10-16T13:49:42.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:42.085+0000 7fefa2a33700 1 -- 172.21.15.16:0/698755766 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef9c0b1740 con 0x7fef9c06aef0 2023-10-16T13:49:42.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:42.086+0000 7fefa2a33700 1 -- 172.21.15.16:0/698755766 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef9c0b2c80 con 0x7fef9c06aef0 2023-10-16T13:49:42.101 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:42.087+0000 7fef98ff9700 1 -- 172.21.15.16:0/698755766 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 176) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fef8c01c460 con 0x7fef9c06aef0 2023-10-16T13:49:42.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:42.087+0000 7fef98ff9700 1 --2- 172.21.15.16:0/698755766 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7fef7c06b020 0x7fef7c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:49:42.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:42.087+0000 7fef98ff9700 1 -- 172.21.15.16:0/698755766 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(283..283 src has 1..283) v4 ==== 6208+0+0 (secure 0 0 0) 0x7fef8c0897b0 con 0x7fef9c06aef0 2023-10-16T13:49:42.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:42.087+0000 7fef9b7fe700 1 --2- 172.21.15.16:0/698755766 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7fef7c06b020 0x7fef7c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:49:42.102 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:42.087+0000 7fef9b7fe700 1 --2- 172.21.15.16:0/698755766 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7fef7c06b020 0x7fef7c06d4e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fef9c15b750 tx=0x7fef84008040 comp rx=0 tx=0).ready entity=mgr.13935 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:49:42.103 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:42.091+0000 7fef98ff9700 1 -- 172.21.15.16:0/698755766 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fef8c054560 con 0x7fef9c06aef0 2023-10-16T13:49:42.295 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:42.293+0000 7fefa2a33700 1 -- 172.21.15.16:0/698755766 --> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fef9c0caea0 con 0x7fef7c06b020 2023-10-16T13:49:42.298 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:42.295+0000 7fef98ff9700 1 -- 172.21.15.16:0/698755766 <== mgr.13935 v2:172.21.15.16:6832/938897590 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+50785 (secure 0 0 0) 0x7fef8400adf0 con 0x7fef7c06b020 2023-10-16T13:49:42.299 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:49:42.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:42.298+0000 7fefa2a33700 1 -- 172.21.15.16:0/698755766 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7fef7c06b020 msgr2=0x7fef7c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:42.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:42.298+0000 7fefa2a33700 1 --2- 172.21.15.16:0/698755766 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7fef7c06b020 0x7fef7c06d4e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fef9c15b750 tx=0x7fef84008040 comp rx=0 tx=0).stop 2023-10-16T13:49:42.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:42.298+0000 7fefa2a33700 1 -- 172.21.15.16:0/698755766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef9c06aef0 msgr2=0x7fef9c0b1200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:49:42.301 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:42.298+0000 7fefa2a33700 1 --2- 172.21.15.16:0/698755766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef9c06aef0 0x7fef9c0b1200 secure :-1 s=READY pgs=3371 cs=0 l=1 rev1=1 crypto rx=0x7fef8c00c4d0 tx=0x7fef8c00c890 comp rx=0 tx=0).stop 2023-10-16T13:49:42.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:42.298+0000 7fefa2a33700 1 -- 172.21.15.16:0/698755766 shutdown_connections 2023-10-16T13:49:42.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:42.298+0000 7fefa2a33700 1 --2- 172.21.15.16:0/698755766 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7fef7c06b020 0x7fef7c06d4e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:42.302 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:42.298+0000 7fefa2a33700 1 --2- 172.21.15.16:0/698755766 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fef9c15db40 0x7fef9c160790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:42.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:42.298+0000 7fefa2a33700 1 --2- 172.21.15.16:0/698755766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef9c06aef0 0x7fef9c0b1200 unknown :-1 s=CLOSED pgs=3371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:42.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:42.298+0000 7fefa2a33700 1 --2- 172.21.15.16:0/698755766 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7fef9c0ce160 0x7fef9c160250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:49:42.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:42.298+0000 7fefa2a33700 1 -- 172.21.15.16:0/698755766 >> 172.21.15.16:0/698755766 conn(0x7fef9c0b1fd0 msgr2=0x7fef9c0ccf00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:49:42.303 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:42.298+0000 7fefa2a33700 1 -- 172.21.15.16:0/698755766 shutdown_connections 2023-10-16T13:49:42.304 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:49:42.298+0000 7fefa2a33700 1 -- 172.21.15.16:0/698755766 wait complete. 2023-10-16T13:49:42.304 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-10-16T13:49:42.312 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2023-10-16T13:49:41.842910+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2804,"num_read_kb":5868,"num_write":719,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":425,"ondisk_log_size":425,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":457720,"kb_used_data":29192,"kb_used_omap":12,"kb_used_meta":428403,"kb_avail":754517000,"statfs":{"total":773094113280,"available":772625408000,"internally_reserved":0,"allocated":29892608,"data_stored":16084744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12740,"internal_metadata":438685244},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-119,"num_objects":-1,"num_object_clones":0,"num_object_copies":-2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":95,"num_read_kb":206,"num_write":17,"num_write_kb":-2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001553"},"pg_stats":[{"pgid":"1.0","version":"281'425","reported_seq":1526,"reported_epoch":282,"state":"active+clean","last_fresh":"2023-10-16T13:49:32.039092+0000","last_change":"2023-10-16T12:10:34.746820+0000","last_active":"2023-10-16T13:49:32.039092+0000","last_peered":"2023-10-16T13:49:32.039092+0000","last_clean":"2023-10-16T13:49:32.039092+0000","last_became_active":"2023-10-16T12:10:34.745574+0000","last_became_peered":"2023-10-16T12:10:34.745574+0000","last_unstale":"2023-10-16T13:49:32.039092+0000","last_undegraded":"2023-10-16T13:49:32.039092+0000","last_fullsized":"2023-10-16T13:49:32.039092+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T12:10:33.577111+0000","last_clean_scrub_stamp":"2023-10-16T12:10:33.577111+0000","objects_scrubbed":0,"log_size":425,"log_dups_size":0,"ondisk_log_size":425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T16:14:14.461986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2804,"num_read_kb":5868,"num_write":719,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2804,"num_read_kb":5868,"num_write":719,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":425,"ondisk_log_size":425,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576018,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61632,"kb_used_data":4084,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310208,"statfs":{"total":96636764160,"available":96573652992,"internally_reserved":0,"allocated":4182016,"data_stored":2453369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58915275},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:48:54 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.58399999999999996,"15min":0.621},"min":{"1min":0.34999999999999998,"5min":0.34200000000000003,"15min":0.33000000000000002},"max":{"1min":4.1040000000000001,"5min":4.1040000000000001,"15min":4.1040000000000001},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.753,"5min":0.60699999999999998,"15min":0.625},"min":{"1min":0.35199999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":5.5279999999999996,"5min":5.5279999999999996,"15min":5.5279999999999996},"last":0.47899999999999998}]},{"osd":1,"last update":"Mon Oct 16 13:48:54 2023","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.63100000000000001,"15min":0.64300000000000002},"min":{"1min":0.33500000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":6.6189999999999998,"5min":6.6189999999999998,"15min":6.6189999999999998},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.59999999999999998,"15min":0.626},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":3.3239999999999998,"5min":3.3239999999999998,"15min":3.3239999999999998},"last":0.57899999999999996}]},{"osd":2,"last update":"Mon Oct 16 13:48:54 2023","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.64500000000000002,"15min":0.67700000000000005},"min":{"1min":0.41499999999999998,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":2.4430000000000001,"5min":2.4430000000000001,"15min":2.4430000000000001},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.67200000000000004,"15min":0.69799999999999995},"min":{"1min":0.45500000000000002,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":4.351,"5min":4.351,"15min":4.351},"last":0.59599999999999997}]},{"osd":3,"last update":"Mon Oct 16 13:48:54 2023","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.63300000000000001,"15min":0.65500000000000003},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":4.4130000000000003,"5min":4.4130000000000003,"15min":4.4130000000000003},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.67600000000000005,"15min":0.67100000000000004},"min":{"1min":0.432,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":5.407,"5min":5.407,"15min":5.407},"last":0.68100000000000005}]},{"osd":4,"last update":"Mon Oct 16 13:48:54 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.56100000000000005,"15min":0.53500000000000003},"min":{"1min":0.37,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":3.2400000000000002,"5min":3.2400000000000002,"15min":3.2400000000000002},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.67900000000000005,"15min":0.63900000000000001},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":5.3369999999999997,"5min":5.3369999999999997,"15min":5.3369999999999997},"last":0.66800000000000004}]},{"osd":5,"last update":"Mon Oct 16 13:48:54 2023","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.621,"15min":0.57199999999999995},"min":{"1min":0.35799999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":5.4720000000000004,"5min":5.4720000000000004,"15min":5.4720000000000004},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.61299999999999999,"15min":0.59799999999999998},"min":{"1min":0.40100000000000002,"5min":0.39700000000000002,"15min":0.35099999999999998},"max":{"1min":1.0549999999999999,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.60799999999999998}]},{"osd":6,"last update":"Mon Oct 16 13:48:54 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.63200000000000001,"15min":0.622},"min":{"1min":0.46600000000000003,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.69499999999999995,"15min":0.64500000000000002},"min":{"1min":0.38900000000000001,"5min":0.378,"15min":0.35199999999999998},"max":{"1min":5.3860000000000001,"5min":5.3860000000000001,"15min":5.3860000000000001},"last":0.63500000000000001}]}]},{"osd":6,"up_from":13,"seq":55834576016,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48572,"kb_used_data":3504,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323268,"statfs":{"total":96636764160,"available":96587026432,"internally_reserved":0,"allocated":3588096,"data_stored":1863001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.60999999999999999,"15min":0.65000000000000002},"min":{"1min":0.315,"5min":0.315,"15min":0.29599999999999999},"max":{"1min":0.67600000000000005,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.503,"5min":0.60599999999999998,"15min":0.64200000000000002},"min":{"1min":0.373,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.70099999999999996,"5min":0.91400000000000003,"15min":0.93799999999999994},"last":0.69299999999999995}]},{"osd":1,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.54600000000000004,"15min":0.60499999999999998},"min":{"1min":0.316,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.64500000000000002,"5min":0.86899999999999999,"15min":0.89600000000000002},"last":0.626},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.58599999999999997,"15min":0.60599999999999998},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.69799999999999995,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.745}]},{"osd":2,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.58899999999999997,"15min":0.61799999999999999},"min":{"1min":0.34799999999999998,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.71499999999999997,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.57599999999999996,"15min":0.60599999999999998},"min":{"1min":0.32200000000000001,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.72999999999999998,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.64300000000000002}]},{"osd":3,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.60199999999999998,"15min":0.64600000000000002},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.38200000000000001},"max":{"1min":0.63400000000000001,"5min":0.92400000000000004,"15min":0.95599999999999996},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.66600000000000004,"15min":0.70199999999999996},"min":{"1min":0.435,"5min":0.435,"15min":0.42299999999999999},"max":{"1min":0.72399999999999998,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.77100000000000002}]},{"osd":4,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.53300000000000003,"15min":0.52800000000000002},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.67700000000000005,"5min":0.85599999999999998,"15min":0.95799999999999996},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.60799999999999998,"15min":0.61499999999999999},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.70799999999999996,"5min":0.98799999999999999,"15min":1.0309999999999999},"last":0.54800000000000004}]},{"osd":5,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.58199999999999996,"15min":0.59099999999999997},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.71099999999999997,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.627,"5min":0.64400000000000002,"15min":0.65700000000000003},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.78700000000000003,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.66300000000000003}]},{"osd":7,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.59099999999999997,"15min":0.59799999999999998},"min":{"1min":0.35199999999999998,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.747,"5min":1.081,"15min":1.081},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.63900000000000001,"15min":0.64900000000000002},"min":{"1min":0.40999999999999998,"5min":0.40899999999999997,"15min":0.379},"max":{"1min":0.76600000000000001,"5min":1.099,"15min":1.099},"last":0.50700000000000001}]}]},{"osd":1,"up_from":13,"seq":55834576016,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52732,"kb_used_data":3504,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319108,"statfs":{"total":96636764160,"available":96582766592,"internally_reserved":0,"allocated":3588096,"data_stored":1863001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":50395593},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:48:59 2023","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.37,"15min":0.40100000000000002},"min":{"1min":0.254,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.58099999999999996,"5min":0.65400000000000003,"15min":0.65800000000000003},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.433,"15min":0.48099999999999998},"min":{"1min":0.32700000000000001,"5min":0.253,"15min":0.253},"max":{"1min":0.56599999999999995,"5min":0.68000000000000005,"15min":0.77200000000000002},"last":0.39200000000000002}]},{"osd":2,"last update":"Mon Oct 16 13:48:59 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.39400000000000002,"15min":0.439},"min":{"1min":0.251,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.496,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.44500000000000001,"15min":0.47999999999999998},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.58299999999999996,"5min":0.67600000000000005,"15min":0.71399999999999997},"last":0.27500000000000002}]},{"osd":3,"last update":"Mon Oct 16 13:48:59 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.39000000000000001,"15min":0.42299999999999999},"min":{"1min":0.28499999999999998,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.51300000000000001,"5min":0.65200000000000002,"15min":0.72099999999999997},"last":0.317},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.41699999999999998,"15min":0.46600000000000003},"min":{"1min":0.29599999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.60299999999999998,"5min":0.69399999999999995,"15min":0.77300000000000002},"last":0.35199999999999998}]},{"osd":4,"last update":"Mon Oct 16 13:48:59 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.61599999999999999,"15min":0.65700000000000003},"min":{"1min":0.41199999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.78300000000000003,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.438},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.59899999999999998,"15min":0.65600000000000003},"min":{"1min":0.377,"5min":0.37,"15min":0.37},"max":{"1min":0.90400000000000003,"5min":0.91900000000000004,"15min":0.997},"last":0.5}]},{"osd":5,"last update":"Mon Oct 16 13:48:59 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.64700000000000002,"15min":0.69899999999999995},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":1.0760000000000001},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.65700000000000003,"15min":0.71499999999999997},"min":{"1min":0.436,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.81299999999999994,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.51400000000000001}]},{"osd":6,"last update":"Mon Oct 16 13:48:59 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.621,"15min":0.66900000000000004},"min":{"1min":0.41999999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85799999999999998,"5min":0.95099999999999996,"15min":0.97799999999999998},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.62,"5min":0.64500000000000002,"15min":0.69499999999999995},"min":{"1min":0.46899999999999997,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":1.083},"last":0.48699999999999999}]},{"osd":7,"last update":"Mon Oct 16 13:48:59 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.66300000000000003,"15min":0.72199999999999998},"min":{"1min":0.45300000000000001,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.79300000000000004,"5min":1.163,"15min":1.163},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.68799999999999994,"15min":0.73699999999999999},"min":{"1min":0.47999999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.94699999999999995,"5min":0.98499999999999999,"15min":1.0580000000000001},"last":0.53300000000000003}]}]},{"osd":0,"up_from":13,"seq":55834576017,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53312,"kb_used_data":4084,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318528,"statfs":{"total":96636764160,"available":96582172672,"internally_reserved":0,"allocated":4182016,"data_stored":2453369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.373,"15min":0.41299999999999998},"min":{"1min":0.23799999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.50900000000000001,"5min":0.60699999999999998,"15min":0.71399999999999997},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.38500000000000001,"15min":0.42799999999999999},"min":{"1min":0.24199999999999999,"5min":0.222,"15min":0.222},"max":{"1min":0.56799999999999995,"5min":0.77700000000000002,"15min":0.82199999999999995},"last":0.39200000000000002}]},{"osd":2,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.40999999999999998,"15min":0.441},"min":{"1min":0.27300000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.51000000000000001,"5min":0.65900000000000003,"15min":0.75700000000000001},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.432,"15min":0.47199999999999998},"min":{"1min":0.30399999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.54500000000000004,"5min":0.68799999999999994,"15min":0.83299999999999996},"last":0.47199999999999998}]},{"osd":3,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.40600000000000003,"15min":0.44400000000000001},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.52100000000000002,"5min":0.69699999999999995,"15min":0.78600000000000003},"last":0.435},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.46300000000000002,"15min":0.50600000000000001},"min":{"1min":0.309,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.55900000000000005,"5min":0.70399999999999996,"15min":0.86399999999999999},"last":0.45700000000000002}]},{"osd":4,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.64500000000000002,"15min":0.67600000000000005},"min":{"1min":0.42999999999999999,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.82999999999999996,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.64900000000000002,"15min":0.68100000000000005},"min":{"1min":0.43099999999999999,"5min":0.375,"15min":0.375},"max":{"1min":0.81000000000000005,"5min":1.21,"15min":1.21},"last":0.626}]},{"osd":5,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.63400000000000001,"15min":0.67700000000000005},"min":{"1min":0.40300000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.82199999999999995,"5min":1.242,"15min":1.242},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.622,"5min":0.68300000000000005,"15min":0.71799999999999997},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.83299999999999996,"5min":1.081,"15min":1.081},"last":0.67900000000000005}]},{"osd":6,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.67600000000000005,"15min":0.71999999999999997},"min":{"1min":0.43099999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.84699999999999998,"5min":1.034,"15min":1.103},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.67700000000000005,"15min":0.71499999999999997},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.90900000000000003,"5min":1.131,"15min":1.131},"last":0.69399999999999995}]},{"osd":7,"last update":"Mon Oct 16 13:48:44 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.68700000000000006,"15min":0.73499999999999999},"min":{"1min":0.47399999999999998,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.94799999999999995,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.69599999999999995,"15min":0.72699999999999998},"min":{"1min":0.47899999999999998,"5min":0.433,"15min":0.433},"max":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.66300000000000003}]}]},{"osd":2,"up_from":13,"seq":55834576017,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":56896,"kb_used_data":3504,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314944,"statfs":{"total":96636764160,"available":96578502656,"internally_reserved":0,"allocated":3588096,"data_stored":1863001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:48:41 2023","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.374,"15min":0.40400000000000003},"min":{"1min":0.23899999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.439,"5min":0.623,"15min":0.77500000000000002},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.436,"15min":0.46999999999999997},"min":{"1min":0.33700000000000002,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.57399999999999995,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.42299999999999999}]},{"osd":1,"last update":"Mon Oct 16 13:48:41 2023","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.46100000000000002,"15min":0.49199999999999999},"min":{"1min":0.30499999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.69899999999999995,"5min":0.73199999999999998,"15min":0.81399999999999995},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.45000000000000001,"15min":0.46899999999999997},"min":{"1min":0.27900000000000003,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.51700000000000002,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.36399999999999999}]},{"osd":3,"last update":"Mon Oct 16 13:48:41 2023","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.40400000000000003,"15min":0.435},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.55500000000000005,"5min":0.63,"15min":0.72599999999999998},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.499,"15min":0.53700000000000003},"min":{"1min":0.376,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.65900000000000003,"5min":0.82699999999999996,"15min":0.90100000000000002},"last":0.34300000000000003}]},{"osd":4,"last update":"Mon Oct 16 13:48:41 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.66800000000000004,"15min":0.70699999999999996},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.423,"5min":1.423,"15min":1.423},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.67700000000000005,"15min":0.71399999999999997},"min":{"1min":0.437,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.407,"5min":1.407,"15min":1.407},"last":0.55400000000000005}]},{"osd":5,"last update":"Mon Oct 16 13:48:41 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.627,"15min":0.67800000000000005},"min":{"1min":0.436,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.82199999999999995,"5min":0.87,"15min":1.486},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.64900000000000002,"15min":0.68999999999999995},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.78200000000000003,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":0.68300000000000005}]},{"osd":6,"last update":"Mon Oct 16 13:48:41 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.622,"15min":0.65900000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.80300000000000005,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.66200000000000003,"15min":0.71099999999999997},"min":{"1min":0.49099999999999999,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.85499999999999998,"5min":0.94299999999999995,"15min":1.4379999999999999},"last":0.66300000000000003}]},{"osd":7,"last update":"Mon Oct 16 13:48:41 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.67500000000000004,"15min":0.70899999999999996},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.83499999999999996,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.66800000000000004,"15min":0.72199999999999998},"min":{"1min":0.48799999999999999,"5min":0.436,"15min":0.436},"max":{"1min":0.88200000000000001,"5min":1.0429999999999999,"15min":1.377},"last":0.70499999999999996}]}]},{"osd":3,"up_from":13,"seq":55834576016,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":56896,"kb_used_data":3504,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314944,"statfs":{"total":96636764160,"available":96578502656,"internally_reserved":0,"allocated":3588096,"data_stored":1863001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:48:45 2023","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.375,"15min":0.40400000000000003},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23000000000000001},"max":{"1min":0.48399999999999999,"5min":0.60099999999999998,"15min":0.70499999999999996},"last":0.307},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.38300000000000001,"15min":0.41599999999999998},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.48799999999999999,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.33700000000000002}]},{"osd":1,"last update":"Mon Oct 16 13:48:45 2023","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.38900000000000001,"15min":0.42299999999999999},"min":{"1min":0.27200000000000002,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.497,"5min":0.68200000000000005,"15min":0.76300000000000001},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.438,"15min":0.46999999999999997},"min":{"1min":0.27800000000000002,"5min":0.27700000000000002,"15min":0.253},"max":{"1min":0.60599999999999998,"5min":0.68700000000000006,"15min":0.77800000000000002},"last":0.37}]},{"osd":2,"last update":"Mon Oct 16 13:48:45 2023","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.45100000000000001,"15min":0.47999999999999998},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.71399999999999997,"5min":1.167,"15min":1.167},"last":0.436},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.46000000000000002,"15min":0.50700000000000001},"min":{"1min":0.32500000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.63200000000000001,"5min":0.73299999999999998,"15min":0.76800000000000002},"last":0.42199999999999999}]},{"osd":4,"last update":"Mon Oct 16 13:48:45 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.61799999999999999,"15min":0.65500000000000003},"min":{"1min":0.40100000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.137},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65400000000000003,"15min":0.68899999999999995},"min":{"1min":0.45500000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.84699999999999998,"5min":1.2110000000000001,"15min":1.2909999999999999},"last":0.67100000000000004}]},{"osd":5,"last update":"Mon Oct 16 13:48:45 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.65200000000000002,"15min":0.69799999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.374},"max":{"1min":1.0309999999999999,"5min":1.0680000000000001,"15min":1.2729999999999999},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.65400000000000003,"15min":0.68999999999999995},"min":{"1min":0.47299999999999998,"5min":0.45600000000000002,"15min":0.432},"max":{"1min":0.871,"5min":1.0549999999999999,"15min":1.173},"last":0.72099999999999997}]},{"osd":6,"last update":"Mon Oct 16 13:48:45 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.69799999999999995,"15min":0.71299999999999997},"min":{"1min":0.432,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.115,"5min":5.0999999999999996,"15min":5.0999999999999996},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.68600000000000005,"15min":0.72499999999999998},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.2290000000000001},"last":0.63900000000000001}]},{"osd":7,"last update":"Mon Oct 16 13:48:45 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.68100000000000005,"15min":0.72699999999999998},"min":{"1min":0.45700000000000002,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.21,"5min":1.21,"15min":1.2569999999999999},"last":0.623},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.68700000000000006,"15min":0.73399999999999999},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.43099999999999999},"max":{"1min":0.90600000000000003,"5min":1.262,"15min":1.262},"last":0.73999999999999999}]}]},{"osd":4,"up_from":13,"seq":55834576015,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52736,"kb_used_data":3504,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319104,"statfs":{"total":96636764160,"available":96582762496,"internally_reserved":0,"allocated":3588096,"data_stored":1863001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:48:42 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.56299999999999994,"15min":0.61299999999999999},"min":{"1min":0.36399999999999999,"5min":0.314,"15min":0.314},"max":{"1min":0.77800000000000002,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.57399999999999995,"15min":0.624},"min":{"1min":0.36599999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.87,"5min":0.87,"15min":0.89500000000000002},"last":0.53600000000000003}]},{"osd":1,"last update":"Mon Oct 16 13:48:42 2023","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.57499999999999996,"15min":0.64100000000000001},"min":{"1min":0.35699999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.79900000000000004,"5min":0.877,"15min":0.877},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.58399999999999996,"15min":0.63},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.80900000000000005,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.64600000000000002}]},{"osd":2,"last update":"Mon Oct 16 13:48:42 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.59199999999999997,"15min":0.64100000000000001},"min":{"1min":0.434,"5min":0.378,"15min":0.378},"max":{"1min":0.751,"5min":0.89900000000000002,"15min":0.90600000000000003},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.62,"15min":0.65300000000000002},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.755,"5min":0.89700000000000002,"15min":0.92600000000000005},"last":0.58399999999999996}]},{"osd":3,"last update":"Mon Oct 16 13:48:42 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.58699999999999997,"15min":0.63300000000000001},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.81399999999999995,"5min":0.86599999999999999,"15min":0.99199999999999999},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.624,"5min":0.64400000000000002,"15min":0.67500000000000004},"min":{"1min":0.48199999999999998,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.93899999999999995},"last":0.61299999999999999}]},{"osd":5,"last update":"Mon Oct 16 13:48:42 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.55700000000000005,"15min":0.53800000000000003},"min":{"1min":0.34899999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.80400000000000005,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.57799999999999996,"15min":0.57099999999999995},"min":{"1min":0.307,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.72499999999999998,"5min":0.82199999999999995,"15min":0.94899999999999995},"last":0.871}]},{"osd":6,"last update":"Mon Oct 16 13:48:42 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.63200000000000001,"15min":0.60399999999999998},"min":{"1min":0.38600000000000001,"5min":0.36399999999999999,"15min":0.33200000000000002},"max":{"1min":0.82699999999999996,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.751},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.70099999999999996,"15min":0.69699999999999995},"min":{"1min":0.41699999999999998,"5min":0.40300000000000002,"15min":0.39400000000000002},"max":{"1min":0.89700000000000002,"5min":1.129,"15min":1.129},"last":0.83499999999999996}]},{"osd":7,"last update":"Mon Oct 16 13:48:42 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.65100000000000002,"15min":0.63700000000000001},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.78700000000000003,"5min":0.97099999999999997,"15min":0.98599999999999999},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.70999999999999996,"15min":0.68899999999999995},"min":{"1min":0.48399999999999999,"5min":0.44,"15min":0.38200000000000001},"max":{"1min":0.96999999999999997,"5min":0.97599999999999998,"15min":1.026},"last":0.85099999999999998}]}]},{"osd":5,"up_from":13,"seq":55834576016,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":74944,"kb_used_data":3504,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94296896,"statfs":{"total":96636764160,"available":96560021504,"internally_reserved":0,"allocated":3588096,"data_stored":1863001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:49:30 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.54100000000000004,"15min":0.59999999999999998},"min":{"1min":0.39000000000000001,"5min":0.314,"15min":0.29899999999999999},"max":{"1min":0.73399999999999999,"5min":0.878,"15min":0.92900000000000005},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.56399999999999995,"15min":0.61699999999999999},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.29699999999999999},"max":{"1min":0.80000000000000004,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.67200000000000004}]},{"osd":1,"last update":"Mon Oct 16 13:49:30 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.58699999999999997,"15min":0.63900000000000001},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.29499999999999998},"max":{"1min":0.72999999999999998,"5min":1,"15min":1},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56799999999999995,"15min":0.62},"min":{"1min":0.45400000000000001,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.78100000000000003,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.46600000000000003}]},{"osd":2,"last update":"Mon Oct 16 13:49:30 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.58999999999999997,"15min":0.63600000000000001},"min":{"1min":0.38100000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.71199999999999997,"5min":0.91000000000000003,"15min":0.94699999999999995},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.58099999999999996,"15min":0.63700000000000001},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.81699999999999995,"5min":0.83899999999999997,"15min":0.93500000000000005},"last":0.68999999999999995}]},{"osd":3,"last update":"Mon Oct 16 13:49:30 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.61699999999999999,"15min":0.65200000000000002},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.83599999999999997,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.63500000000000001,"15min":0.68000000000000005},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.84799999999999998,"5min":1.4199999999999999,"15min":1.4199999999999999},"last":0.64500000000000002}]},{"osd":4,"last update":"Mon Oct 16 13:49:30 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53600000000000003,"15min":0.54000000000000004},"min":{"1min":0.33300000000000002,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.77200000000000002,"5min":0.89700000000000002,"15min":0.90700000000000003},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.60099999999999998,"15min":0.59499999999999997},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.84799999999999998,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.77200000000000002}]},{"osd":6,"last update":"Mon Oct 16 13:49:30 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.59499999999999997,"15min":0.59099999999999997},"min":{"1min":0.35699999999999998,"5min":0.315,"15min":0.315},"max":{"1min":0.82799999999999996,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.61499999999999999,"15min":0.623},"min":{"1min":0.374,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.82799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.73499999999999999}]},{"osd":7,"last update":"Mon Oct 16 13:49:30 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.65800000000000003,"15min":0.64300000000000002},"min":{"1min":0.374,"5min":0.374,"15min":0.34999999999999998},"max":{"1min":0.878,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.69499999999999995,"15min":0.69099999999999995},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.86899999999999999,"5min":0.97299999999999998,"15min":1.0229999999999999},"last":0.752}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T13:49:42.318 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-10-16T12:10:33.577111+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=12, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=13, tm_min=49, tm_sec=36, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T13:49:42.319 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-16T13:50:02.320 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T13:50:02.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.460+0000 7f400d733700 1 Processor -- start 2023-10-16T13:50:02.463 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.460+0000 7f400d733700 1 -- start start 2023-10-16T13:50:02.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.461+0000 7f400d733700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f400815a300 0x7f40080b2da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:02.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.461+0000 7f400d733700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f400815a6e0 0x7f4008070860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:02.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.461+0000 7f400d733700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40080b4060 con 0x7f40080b3ba0 2023-10-16T13:50:02.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.461+0000 7f400d733700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40080b22c0 con 0x7f400815a300 2023-10-16T13:50:02.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.461+0000 7f400d733700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40080b25a0 con 0x7f400815a6e0 2023-10-16T13:50:02.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.461+0000 7f4006ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f400815a300 0x7f40080b2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:02.473 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.461+0000 7f40067fc700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f40080b3ba0 0x7f400815ea70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49938/0 (socket says 172.21.15.16:49938) 2023-10-16T13:50:02.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.461+0000 7f40067fc700 1 -- 172.21.15.16:0/1274844866 learned_addr learned my addr 172.21.15.16:0/1274844866 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:50:02.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.461+0000 7f40077fe700 1 --2- 172.21.15.16:0/1274844866 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f400815a6e0 0x7f4008070860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:02.474 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.461+0000 7f4006ffd700 1 -- 172.21.15.16:0/1274844866 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f400815a6e0 msgr2=0x7f4008070860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:02.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.461+0000 7f4006ffd700 1 --2- 172.21.15.16:0/1274844866 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f400815a6e0 0x7f4008070860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:02.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.461+0000 7f4006ffd700 1 -- 172.21.15.16:0/1274844866 >> v1:172.21.15.16:6789/0 conn(0x7f40080b3ba0 legacy=0x7f400815ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:02.475 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.461+0000 7f4006ffd700 1 -- 172.21.15.16:0/1274844866 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40080b2880 con 0x7f400815a300 2023-10-16T13:50:02.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.461+0000 7f4005ffb700 1 -- 172.21.15.16:0/1274844866 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2757987415 0 0) 0x7f40080b4060 con 0x7f40080b3ba0 2023-10-16T13:50:02.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.462+0000 7f4006ffd700 1 --2- 172.21.15.16:0/1274844866 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f400815a300 0x7f40080b2da0 secure :-1 s=READY pgs=3373 cs=0 l=1 rev1=1 crypto rx=0x7f3ff8009d90 tx=0x7f3ff800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4b1cef9985236bf server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:02.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.462+0000 7f4005ffb700 1 -- 172.21.15.16:0/1274844866 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ff800e070 con 0x7f400815a300 2023-10-16T13:50:02.476 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.462+0000 7f4005ffb700 1 -- 172.21.15.16:0/1274844866 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ff800bbd0 con 0x7f400815a300 2023-10-16T13:50:02.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.462+0000 7f4005ffb700 1 -- 172.21.15.16:0/1274844866 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ff8012430 con 0x7f400815a300 2023-10-16T13:50:02.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.462+0000 7f400d733700 1 -- 172.21.15.16:0/1274844866 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f400815a300 msgr2=0x7f40080b2da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:02.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.462+0000 7f400d733700 1 --2- 172.21.15.16:0/1274844866 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f400815a300 0x7f40080b2da0 secure :-1 s=READY pgs=3373 cs=0 l=1 rev1=1 crypto rx=0x7f3ff8009d90 tx=0x7f3ff800b330 comp rx=0 tx=0).stop 2023-10-16T13:50:02.477 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.462+0000 7f400d733700 1 -- 172.21.15.16:0/1274844866 shutdown_connections 2023-10-16T13:50:02.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.462+0000 7f400d733700 1 --2- 172.21.15.16:0/1274844866 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f400815a6e0 0x7f4008070860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:02.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.462+0000 7f400d733700 1 --2- 172.21.15.16:0/1274844866 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f400815a300 0x7f40080b2da0 unknown :-1 s=CLOSED pgs=3373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:02.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.462+0000 7f400d733700 1 -- 172.21.15.16:0/1274844866 >> 172.21.15.16:0/1274844866 conn(0x7f40080b4cd0 msgr2=0x7f400806a7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:02.478 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.463+0000 7f400d733700 1 -- 172.21.15.16:0/1274844866 shutdown_connections 2023-10-16T13:50:02.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.463+0000 7f400d733700 1 -- 172.21.15.16:0/1274844866 wait complete. 2023-10-16T13:50:02.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.464+0000 7f400d733700 1 Processor -- start 2023-10-16T13:50:02.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.464+0000 7f400d733700 1 -- start start 2023-10-16T13:50:02.479 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.465+0000 7f400d733700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40080b3ba0 0x7f40081f0440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:02.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.465+0000 7f400d733700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f400815a300 0x7f40080a9070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:02.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.465+0000 7f400d733700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f400815a6e0 0x7f40080a95b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:02.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.465+0000 7f400d733700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f400815d450 con 0x7f40080b3ba0 2023-10-16T13:50:02.480 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.465+0000 7f400d733700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f400815bab0 con 0x7f400815a300 2023-10-16T13:50:02.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.465+0000 7f400d733700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f400815bd90 con 0x7f400815a6e0 2023-10-16T13:50:02.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.465+0000 7f4006ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40080b3ba0 0x7f40081f0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:02.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.465+0000 7f4006ffd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40080b3ba0 0x7f40081f0440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60670/0 (socket says 172.21.15.16:60670) 2023-10-16T13:50:02.481 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.465+0000 7f4006ffd700 1 -- 172.21.15.16:0/3702414005 learned_addr learned my addr 172.21.15.16:0/3702414005 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:50:02.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.465+0000 7f40067fc700 1 --2- 172.21.15.16:0/3702414005 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f400815a300 0x7f40080a9070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:02.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.465+0000 7f40077fe700 1 --2- 172.21.15.16:0/3702414005 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f400815a6e0 0x7f40080a95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:02.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.465+0000 7f4006ffd700 1 -- 172.21.15.16:0/3702414005 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f400815a6e0 msgr2=0x7f40080a95b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:02.482 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.465+0000 7f4006ffd700 1 --2- 172.21.15.16:0/3702414005 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f400815a6e0 0x7f40080a95b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:02.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.465+0000 7f4006ffd700 1 -- 172.21.15.16:0/3702414005 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f400815a300 msgr2=0x7f40080a9070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:02.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.465+0000 7f4006ffd700 1 --2- 172.21.15.16:0/3702414005 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f400815a300 0x7f40080a9070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:02.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.465+0000 7f4006ffd700 1 -- 172.21.15.16:0/3702414005 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ff8009a70 con 0x7f40080b3ba0 2023-10-16T13:50:02.483 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.465+0000 7f40067fc700 1 --2- 172.21.15.16:0/3702414005 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f400815a300 0x7f40080a9070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:50:02.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.465+0000 7f40077fe700 1 --2- 172.21.15.16:0/3702414005 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f400815a6e0 0x7f40080a95b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T13:50:02.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.465+0000 7f4006ffd700 1 --2- 172.21.15.16:0/3702414005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40080b3ba0 0x7f40081f0440 secure :-1 s=READY pgs=3374 cs=0 l=1 rev1=1 crypto rx=0x7f3ff8014040 tx=0x7f3ff8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:02.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.466+0000 7f3fff7fe700 1 -- 172.21.15.16:0/3702414005 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ff80098f0 con 0x7f40080b3ba0 2023-10-16T13:50:02.484 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.466+0000 7f3fff7fe700 1 -- 172.21.15.16:0/3702414005 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ff8020070 con 0x7f40080b3ba0 2023-10-16T13:50:02.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.466+0000 7f3fff7fe700 1 -- 172.21.15.16:0/3702414005 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ff8004280 con 0x7f40080b3ba0 2023-10-16T13:50:02.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.466+0000 7f400d733700 1 -- 172.21.15.16:0/3702414005 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f400815c070 con 0x7f40080b3ba0 2023-10-16T13:50:02.485 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.466+0000 7f400d733700 1 -- 172.21.15.16:0/3702414005 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40081f0980 con 0x7f40080b3ba0 2023-10-16T13:50:02.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.467+0000 7f400d733700 1 -- 172.21.15.16:0/3702414005 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f400815d600 con 0x7f40080b3ba0 2023-10-16T13:50:02.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.467+0000 7f3fff7fe700 1 -- 172.21.15.16:0/3702414005 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 176) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3ff80043e0 con 0x7f40080b3ba0 2023-10-16T13:50:02.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.467+0000 7f3fff7fe700 1 --2- 172.21.15.16:0/3702414005 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f3fe806b020 0x7f3fe806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:02.486 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.468+0000 7f3fff7fe700 1 -- 172.21.15.16:0/3702414005 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(283..283 src has 1..283) v4 ==== 6208+0+0 (secure 0 0 0) 0x7f3ff8022070 con 0x7f40080b3ba0 2023-10-16T13:50:02.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.468+0000 7f40067fc700 1 --2- 172.21.15.16:0/3702414005 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f3fe806b020 0x7f3fe806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:02.487 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.468+0000 7f40067fc700 1 --2- 172.21.15.16:0/3702414005 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f3fe806b020 0x7f3fe806d4e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f40080a8ed0 tx=0x7f3ff0007ca0 comp rx=0 tx=0).ready entity=mgr.13935 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:02.488 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.472+0000 7f3fff7fe700 1 -- 172.21.15.16:0/3702414005 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f400815d600 con 0x7f40080b3ba0 2023-10-16T13:50:02.675 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.673+0000 7f400d733700 1 -- 172.21.15.16:0/3702414005 --> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f400804ebb0 con 0x7f3fe806b020 2023-10-16T13:50:02.677 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.675+0000 7f3fff7fe700 1 -- 172.21.15.16:0/3702414005 <== mgr.13935 v2:172.21.15.16:6832/938897590 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+51005 (secure 0 0 0) 0x7f400804ebb0 con 0x7f3fe806b020 2023-10-16T13:50:02.678 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:50:02.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.677+0000 7f400d733700 1 -- 172.21.15.16:0/3702414005 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f3fe806b020 msgr2=0x7f3fe806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:02.680 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.677+0000 7f400d733700 1 --2- 172.21.15.16:0/3702414005 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f3fe806b020 0x7f3fe806d4e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f40080a8ed0 tx=0x7f3ff0007ca0 comp rx=0 tx=0).stop 2023-10-16T13:50:02.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.677+0000 7f400d733700 1 -- 172.21.15.16:0/3702414005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40080b3ba0 msgr2=0x7f40081f0440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:02.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.677+0000 7f400d733700 1 --2- 172.21.15.16:0/3702414005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40080b3ba0 0x7f40081f0440 secure :-1 s=READY pgs=3374 cs=0 l=1 rev1=1 crypto rx=0x7f3ff8014040 tx=0x7f3ff8004ad0 comp rx=0 tx=0).stop 2023-10-16T13:50:02.681 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.677+0000 7f400d733700 1 -- 172.21.15.16:0/3702414005 shutdown_connections 2023-10-16T13:50:02.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.677+0000 7f400d733700 1 --2- 172.21.15.16:0/3702414005 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f3fe806b020 0x7f3fe806d4e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:02.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.677+0000 7f400d733700 1 --2- 172.21.15.16:0/3702414005 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f400815a6e0 0x7f40080a95b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:02.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.677+0000 7f400d733700 1 --2- 172.21.15.16:0/3702414005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40080b3ba0 0x7f40081f0440 unknown :-1 s=CLOSED pgs=3374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:02.682 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.677+0000 7f400d733700 1 --2- 172.21.15.16:0/3702414005 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f400815a300 0x7f40080a9070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:02.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.677+0000 7f400d733700 1 -- 172.21.15.16:0/3702414005 >> 172.21.15.16:0/3702414005 conn(0x7f40080b4cd0 msgr2=0x7f4008153c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:02.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.678+0000 7f400d733700 1 -- 172.21.15.16:0/3702414005 shutdown_connections 2023-10-16T13:50:02.683 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.678+0000 7f400d733700 1 -- 172.21.15.16:0/3702414005 wait complete. 2023-10-16T13:50:02.684 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-10-16T13:50:02.692 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2023-10-16T13:50:01.847565+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2804,"num_read_kb":5868,"num_write":719,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":426,"ondisk_log_size":426,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":457712,"kb_used_data":29192,"kb_used_omap":12,"kb_used_meta":428403,"kb_avail":754517008,"statfs":{"total":773094113280,"available":772625416192,"internally_reserved":0,"allocated":29892608,"data_stored":16084744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12740,"internal_metadata":438685244},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002864"},"pg_stats":[{"pgid":"1.0","version":"283'426","reported_seq":1536,"reported_epoch":283,"state":"active+clean","last_fresh":"2023-10-16T13:49:43.521957+0000","last_change":"2023-10-16T13:49:43.522052+0000","last_active":"2023-10-16T13:49:43.521957+0000","last_peered":"2023-10-16T13:49:43.521957+0000","last_clean":"2023-10-16T13:49:43.521957+0000","last_became_active":"2023-10-16T12:10:34.745574+0000","last_became_peered":"2023-10-16T12:10:34.745574+0000","last_unstale":"2023-10-16T13:49:43.521957+0000","last_undegraded":"2023-10-16T13:49:43.521957+0000","last_fullsized":"2023-10-16T13:49:43.521957+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"283'426","last_scrub_stamp":"2023-10-16T13:49:43.521836+0000","last_deep_scrub":"283'426","last_deep_scrub_stamp":"2023-10-16T13:49:43.521836+0000","last_clean_scrub_stamp":"2023-10-16T13:49:43.521836+0000","objects_scrubbed":2,"log_size":426,"log_dups_size":0,"ondisk_log_size":426,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T18:15:27.547613+0000","scrub_duration":0.0056071460000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2804,"num_read_kb":5868,"num_write":719,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2804,"num_read_kb":5868,"num_write":719,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":426,"ondisk_log_size":426,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576022,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61628,"kb_used_data":4084,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310212,"statfs":{"total":96636764160,"available":96573657088,"internally_reserved":0,"allocated":4182016,"data_stored":2453369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58915275},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:49:55 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.58199999999999996,"15min":0.61199999999999999},"min":{"1min":0.39300000000000002,"5min":0.34200000000000003,"15min":0.33000000000000002},"max":{"1min":0.746,"5min":4.1040000000000001,"15min":4.1040000000000001},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.59199999999999997,"15min":0.61599999999999999},"min":{"1min":0.33800000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.72299999999999998,"5min":5.5279999999999996,"15min":5.5279999999999996},"last":0.49399999999999999}]},{"osd":1,"last update":"Mon Oct 16 13:49:55 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.628,"15min":0.63700000000000001},"min":{"1min":0.372,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.874,"5min":6.6189999999999998,"15min":6.6189999999999998},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.59599999999999997,"15min":0.62},"min":{"1min":0.36699999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.80700000000000005,"5min":3.3239999999999998,"15min":3.3239999999999998},"last":0.56100000000000005}]},{"osd":2,"last update":"Mon Oct 16 13:49:55 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.64100000000000001,"15min":0.67300000000000004},"min":{"1min":0.50600000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.78100000000000003,"5min":2.4430000000000001,"15min":2.4430000000000001},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.66200000000000003,"15min":0.68999999999999995},"min":{"1min":0.35299999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.76700000000000002,"5min":4.351,"15min":4.351},"last":0.72399999999999998}]},{"osd":3,"last update":"Mon Oct 16 13:49:55 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.63100000000000001,"15min":0.64800000000000002},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.86899999999999999,"5min":4.4130000000000003,"15min":4.4130000000000003},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.67000000000000004,"15min":0.66600000000000004},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.90000000000000002,"5min":5.407,"15min":5.407},"last":0.60599999999999998}]},{"osd":4,"last update":"Mon Oct 16 13:49:55 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.57799999999999996,"15min":0.53900000000000003},"min":{"1min":0.36599999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.83599999999999997,"5min":3.2400000000000002,"15min":3.2400000000000002},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.68500000000000005,"15min":0.64100000000000001},"min":{"1min":0.46200000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.98299999999999998,"5min":5.3369999999999997,"15min":5.3369999999999997},"last":0.52700000000000002}]},{"osd":5,"last update":"Mon Oct 16 13:49:55 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.63300000000000001,"15min":0.57499999999999996},"min":{"1min":0.39800000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.96999999999999997,"5min":5.4720000000000004,"15min":5.4720000000000004},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.60099999999999998},"min":{"1min":0.376,"5min":0.376,"15min":0.35099999999999998},"max":{"1min":0.91900000000000004,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.47599999999999998}]},{"osd":6,"last update":"Mon Oct 16 13:49:55 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.64700000000000002,"15min":0.623},"min":{"1min":0.42799999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.95399999999999996,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.70699999999999996,"15min":0.64800000000000002},"min":{"1min":0.47399999999999998,"5min":0.38,"15min":0.35199999999999998},"max":{"1min":0.93700000000000006,"5min":5.3860000000000001,"15min":5.3860000000000001},"last":0.441}]}]},{"osd":6,"up_from":13,"seq":55834576020,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48572,"kb_used_data":3504,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323268,"statfs":{"total":96636764160,"available":96587026432,"internally_reserved":0,"allocated":3588096,"data_stored":1863001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:49:47 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.58299999999999996,"15min":0.64100000000000001},"min":{"1min":0.32600000000000001,"5min":0.315,"15min":0.29599999999999999},"max":{"1min":0.79900000000000004,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58499999999999996,"15min":0.63500000000000001},"min":{"1min":0.39400000000000002,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.77700000000000002,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.78300000000000003}]},{"osd":1,"last update":"Mon Oct 16 13:49:47 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.53100000000000003,"15min":0.59599999999999997},"min":{"1min":0.33600000000000002,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.73999999999999999,"5min":0.86899999999999999,"15min":0.89600000000000002},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57299999999999995,"15min":0.59999999999999998},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.80200000000000005,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.63600000000000001}]},{"osd":2,"last update":"Mon Oct 16 13:49:47 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.56599999999999995,"15min":0.61299999999999999},"min":{"1min":0.308,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.76000000000000001,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.55700000000000005,"15min":0.60099999999999998},"min":{"1min":0.379,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.72699999999999998,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.622}]},{"osd":3,"last update":"Mon Oct 16 13:49:47 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58599999999999997,"15min":0.63800000000000001},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.78600000000000003,"5min":0.874,"15min":0.92400000000000004},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.69499999999999995},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.85999999999999999,"5min":0.94999999999999996,"15min":0.96099999999999997},"last":0.872}]},{"osd":4,"last update":"Mon Oct 16 13:49:47 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52500000000000002,"15min":0.52900000000000003},"min":{"1min":0.33100000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.75600000000000001,"5min":0.78000000000000003,"15min":0.95799999999999996},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59899999999999998,"15min":0.61699999999999999},"min":{"1min":0.40600000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.81200000000000006,"5min":0.84199999999999997,"15min":1.0309999999999999},"last":0.58099999999999996}]},{"osd":5,"last update":"Mon Oct 16 13:49:47 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.57899999999999996,"15min":0.58999999999999997},"min":{"1min":0.375,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":1.0389999999999999},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.63200000000000001,"15min":0.65700000000000003},"min":{"1min":0.40999999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":1.0609999999999999},"last":0.69899999999999995}]},{"osd":7,"last update":"Mon Oct 16 13:49:47 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57199999999999995,"15min":0.59899999999999998},"min":{"1min":0.376,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":1.081},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.629,"15min":0.65000000000000002},"min":{"1min":0.49199999999999999,"5min":0.40899999999999997,"15min":0.379},"max":{"1min":0.879,"5min":0.91500000000000004,"15min":1.099},"last":0.60399999999999998}]}]},{"osd":1,"up_from":13,"seq":55834576020,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52732,"kb_used_data":3504,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319108,"statfs":{"total":96636764160,"available":96582766592,"internally_reserved":0,"allocated":3588096,"data_stored":1863001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":50395593},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:48:59 2023","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.37,"15min":0.40100000000000002},"min":{"1min":0.254,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.58099999999999996,"5min":0.65400000000000003,"15min":0.65800000000000003},"last":0.5},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.433,"15min":0.48099999999999998},"min":{"1min":0.32700000000000001,"5min":0.253,"15min":0.253},"max":{"1min":0.56599999999999995,"5min":0.68000000000000005,"15min":0.77200000000000002},"last":0.64300000000000002}]},{"osd":2,"last update":"Mon Oct 16 13:48:59 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.39400000000000002,"15min":0.439},"min":{"1min":0.251,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.496,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.44500000000000001,"15min":0.47999999999999998},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.58299999999999996,"5min":0.67600000000000005,"15min":0.71399999999999997},"last":0.60599999999999998}]},{"osd":3,"last update":"Mon Oct 16 13:48:59 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.39000000000000001,"15min":0.42299999999999999},"min":{"1min":0.28499999999999998,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.51300000000000001,"5min":0.65200000000000002,"15min":0.72099999999999997},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.41699999999999998,"15min":0.46600000000000003},"min":{"1min":0.29599999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.60299999999999998,"5min":0.69399999999999995,"15min":0.77300000000000002},"last":0.55200000000000005}]},{"osd":4,"last update":"Mon Oct 16 13:48:59 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.61599999999999999,"15min":0.65700000000000003},"min":{"1min":0.41199999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.78300000000000003,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.59899999999999998,"15min":0.65600000000000003},"min":{"1min":0.377,"5min":0.37,"15min":0.37},"max":{"1min":0.90400000000000003,"5min":0.91900000000000004,"15min":0.997},"last":0.81399999999999995}]},{"osd":5,"last update":"Mon Oct 16 13:48:59 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.64700000000000002,"15min":0.69899999999999995},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":1.0760000000000001},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.65700000000000003,"15min":0.71499999999999997},"min":{"1min":0.436,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.81299999999999994,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.80500000000000005}]},{"osd":6,"last update":"Mon Oct 16 13:48:59 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.621,"15min":0.66900000000000004},"min":{"1min":0.41999999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85799999999999998,"5min":0.95099999999999996,"15min":0.97799999999999998},"last":0.90300000000000002},{"interface":"front","average":{"1min":0.62,"5min":0.64500000000000002,"15min":0.69499999999999995},"min":{"1min":0.46899999999999997,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":1.083},"last":0.89000000000000001}]},{"osd":7,"last update":"Mon Oct 16 13:48:59 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.66300000000000003,"15min":0.72199999999999998},"min":{"1min":0.45300000000000001,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.79300000000000004,"5min":1.163,"15min":1.163},"last":0.85599999999999998},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.68799999999999994,"15min":0.73699999999999999},"min":{"1min":0.47999999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.94699999999999995,"5min":0.98499999999999999,"15min":1.0580000000000001},"last":0.93000000000000005}]}]},{"osd":0,"up_from":13,"seq":55834576021,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53308,"kb_used_data":4084,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318532,"statfs":{"total":96636764160,"available":96582176768,"internally_reserved":0,"allocated":4182016,"data_stored":2453369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 13:49:46 2023","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.373,"15min":0.40799999999999997},"min":{"1min":0.25700000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.52500000000000002,"5min":0.60699999999999998,"15min":0.71399999999999997},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.38800000000000001,"15min":0.42399999999999999},"min":{"1min":0.26100000000000001,"5min":0.222,"15min":0.222},"max":{"1min":0.55400000000000005,"5min":0.77700000000000002,"15min":0.82199999999999995},"last":0.46800000000000003}]},{"osd":2,"last update":"Mon Oct 16 13:49:46 2023","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.40799999999999997,"15min":0.436},"min":{"1min":0.33500000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.55000000000000004,"5min":0.65900000000000003,"15min":0.75700000000000001},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.42599999999999999,"15min":0.46600000000000003},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.58799999999999997,"5min":0.68799999999999994,"15min":0.83299999999999996},"last":0.51700000000000002}]},{"osd":3,"last update":"Mon Oct 16 13:49:46 2023","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.40600000000000003,"15min":0.438},"min":{"1min":0.28499999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.54700000000000004,"5min":0.69699999999999995,"15min":0.78600000000000003},"last":0.374},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.45600000000000002,"15min":0.501},"min":{"1min":0.33700000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.64100000000000001,"5min":0.70399999999999996,"15min":0.86399999999999999},"last":0.496}]},{"osd":4,"last update":"Mon Oct 16 13:49:46 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.63600000000000001,"15min":0.66500000000000004},"min":{"1min":0.42499999999999999,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.82499999999999996,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.63900000000000001,"15min":0.67200000000000004},"min":{"1min":0.42099999999999999,"5min":0.375,"15min":0.375},"max":{"1min":0.81799999999999995,"5min":1.21,"15min":1.21},"last":0.83899999999999997}]},{"osd":5,"last update":"Mon Oct 16 13:49:46 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.621,"15min":0.66500000000000004},"min":{"1min":0.40400000000000003,"5min":0.40300000000000002,"15min":0.38600000000000001},"max":{"1min":0.79500000000000004,"5min":1.242,"15min":1.242},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.67400000000000004,"15min":0.70899999999999996},"min":{"1min":0.436,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.94599999999999995,"5min":1.081,"15min":1.081},"last":0.79800000000000004}]},{"osd":6,"last update":"Mon Oct 16 13:49:46 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.67000000000000004,"15min":0.71099999999999997},"min":{"1min":0.47599999999999998,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.98799999999999999,"5min":1.034,"15min":1.103},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.67400000000000004,"15min":0.70799999999999996},"min":{"1min":0.40300000000000002,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.91400000000000003,"5min":1.131,"15min":1.131},"last":0.76100000000000001}]},{"osd":7,"last update":"Mon Oct 16 13:49:46 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.68100000000000005,"15min":0.72499999999999998},"min":{"1min":0.44500000000000001,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.86699999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.629,"5min":0.68500000000000005,"15min":0.71599999999999997},"min":{"1min":0.47799999999999998,"5min":0.433,"15min":0.433},"max":{"1min":0.92100000000000004,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.78000000000000003}]}]},{"osd":2,"up_from":13,"seq":55834576021,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":56896,"kb_used_data":3504,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314944,"statfs":{"total":96636764160,"available":96578502656,"internally_reserved":0,"allocated":3588096,"data_stored":1863001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:49:46 2023","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.373,"15min":0.40300000000000002},"min":{"1min":0.27600000000000002,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.60799999999999998,"5min":0.623,"15min":0.77500000000000002},"last":0.502},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.42599999999999999,"15min":0.46600000000000003},"min":{"1min":0.33100000000000002,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.68799999999999994,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.432}]},{"osd":1,"last update":"Mon Oct 16 13:49:46 2023","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.443,"15min":0.48699999999999999},"min":{"1min":0.32200000000000001,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.61499999999999999,"5min":0.73199999999999998,"15min":0.81399999999999995},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.42899999999999999,"15min":0.46800000000000003},"min":{"1min":0.312,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.57999999999999996,"5min":0.69999999999999996,"15min":1.3089999999999999},"last":0.40300000000000002}]},{"osd":3,"last update":"Mon Oct 16 13:49:46 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.43099999999999999},"min":{"1min":0.27100000000000002,"5min":0.224,"15min":0.224},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.72599999999999998},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.495,"5min":0.48699999999999999,"15min":0.53300000000000003},"min":{"1min":0.34300000000000003,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.75900000000000001,"5min":0.82699999999999996,"15min":0.90100000000000002},"last":0.54100000000000004}]},{"osd":4,"last update":"Mon Oct 16 13:49:46 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.65000000000000002,"15min":0.69999999999999996},"min":{"1min":0.46200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.76700000000000002,"5min":1.423,"15min":1.423},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.64700000000000002,"15min":0.70399999999999996},"min":{"1min":0.437,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.82499999999999996,"5min":1.407,"15min":1.407},"last":0.52200000000000002}]},{"osd":5,"last update":"Mon Oct 16 13:49:46 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61199999999999999,"15min":0.67300000000000004},"min":{"1min":0.41299999999999998,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.81499999999999995,"5min":0.82999999999999996,"15min":1.486},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.628,"15min":0.68700000000000006},"min":{"1min":0.44,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.83299999999999996,"5min":0.90200000000000002,"15min":1.3660000000000001},"last":0.63600000000000001}]},{"osd":6,"last update":"Mon Oct 16 13:49:46 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.59399999999999997,"15min":0.65300000000000002},"min":{"1min":0.39500000000000002,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.745,"5min":1.087,"15min":1.3959999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.64600000000000002,"15min":0.70599999999999996},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.86699999999999999,"5min":0.94299999999999995,"15min":1.4379999999999999},"last":0.68999999999999995}]},{"osd":7,"last update":"Mon Oct 16 13:49:46 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.65000000000000002,"15min":0.70499999999999996},"min":{"1min":0.46100000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.85899999999999999,"5min":1.0509999999999999,"15min":1.4370000000000001},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.65900000000000003,"15min":0.71799999999999997},"min":{"1min":0.5,"5min":0.436,"15min":0.436},"max":{"1min":0.90200000000000002,"5min":1.0429999999999999,"15min":1.377},"last":0.71799999999999997}]}]},{"osd":3,"up_from":13,"seq":55834576020,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":56896,"kb_used_data":3504,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314944,"statfs":{"total":96636764160,"available":96578502656,"internally_reserved":0,"allocated":3588096,"data_stored":1863001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:49:49 2023","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.36599999999999999,"15min":0.40100000000000002},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23000000000000001},"max":{"1min":0.53100000000000003,"5min":0.60099999999999998,"15min":0.70499999999999996},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.37,"15min":0.41299999999999998},"min":{"1min":0.254,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.59599999999999997,"5min":0.60099999999999998,"15min":0.64200000000000002},"last":0.45800000000000002}]},{"osd":1,"last update":"Mon Oct 16 13:49:49 2023","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.38300000000000001,"15min":0.41899999999999998},"min":{"1min":0.26600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.56000000000000005,"5min":0.68200000000000005,"15min":0.70499999999999996},"last":0.433},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.42699999999999999,"15min":0.46400000000000002},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.253},"max":{"1min":0.627,"5min":0.68700000000000006,"15min":0.76400000000000001},"last":0.53500000000000003}]},{"osd":2,"last update":"Mon Oct 16 13:49:49 2023","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.435,"15min":0.47699999999999998},"min":{"1min":0.317,"5min":0.307,"15min":0.307},"max":{"1min":0.65400000000000003,"5min":0.71399999999999997,"15min":1.167},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.433,"5min":0.45000000000000001,"15min":0.502},"min":{"1min":0.318,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.66900000000000004,"5min":0.73299999999999998,"15min":0.76800000000000002},"last":0.49399999999999999}]},{"osd":4,"last update":"Mon Oct 16 13:49:49 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.61899999999999999,"15min":0.65200000000000002},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.89800000000000002,"5min":1.0780000000000001,"15min":1.137},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.63500000000000001,"15min":0.68100000000000005},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.873,"5min":1.087,"15min":1.2909999999999999},"last":0.71599999999999997}]},{"osd":5,"last update":"Mon Oct 16 13:49:49 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.64100000000000001,"15min":0.69099999999999995},"min":{"1min":0.39900000000000002,"5min":0.38900000000000001,"15min":0.374},"max":{"1min":0.85499999999999998,"5min":1.0680000000000001,"15min":1.2729999999999999},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.64200000000000002,"15min":0.68500000000000005},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.93100000000000005,"5min":1.0549999999999999,"15min":1.173},"last":0.67800000000000005}]},{"osd":6,"last update":"Mon Oct 16 13:49:49 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.65300000000000002,"15min":0.70899999999999996},"min":{"1min":0.436,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.93600000000000005,"5min":1.115,"15min":5.0999999999999996},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.67700000000000005,"15min":0.71799999999999997},"min":{"1min":0.44,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.91900000000000004,"5min":1.1539999999999999,"15min":1.2290000000000001},"last":0.69799999999999995}]},{"osd":7,"last update":"Mon Oct 16 13:49:49 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.67100000000000004,"15min":0.71899999999999997},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.94999999999999996,"5min":1.21,"15min":1.2569999999999999},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.67900000000000005,"15min":0.72799999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.96999999999999997,"5min":1.0369999999999999,"15min":1.262},"last":0.76700000000000002}]}]},{"osd":4,"up_from":13,"seq":55834576019,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52736,"kb_used_data":3504,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319104,"statfs":{"total":96636764160,"available":96582762496,"internally_reserved":0,"allocated":3588096,"data_stored":1863001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:49:43 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.55800000000000005,"15min":0.60499999999999998},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.73799999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.55600000000000005,"15min":0.61699999999999999},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.65000000000000002,"5min":0.87,"15min":0.89500000000000002},"last":0.58499999999999996}]},{"osd":1,"last update":"Mon Oct 16 13:49:43 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.56999999999999995,"15min":0.63400000000000001},"min":{"1min":0.40400000000000003,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.80500000000000005,"5min":0.877,"15min":0.877},"last":0.504},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.57899999999999996,"15min":0.622},"min":{"1min":0.35999999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.755,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.68200000000000005}]},{"osd":2,"last update":"Mon Oct 16 13:49:43 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.57699999999999996,"15min":0.63400000000000001},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.73599999999999999,"5min":0.89900000000000002,"15min":0.90600000000000003},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.60099999999999998,"15min":0.64900000000000002},"min":{"1min":0.38100000000000001,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.78300000000000003,"5min":0.88200000000000001,"15min":0.92600000000000005},"last":0.749}]},{"osd":3,"last update":"Mon Oct 16 13:49:43 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.57799999999999996,"15min":0.627},"min":{"1min":0.40600000000000003,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.78200000000000003,"5min":0.81399999999999995,"15min":0.99199999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.63200000000000001,"15min":0.67100000000000004},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.88,"5min":0.91500000000000004,"15min":0.93899999999999995},"last":0.65700000000000003}]},{"osd":5,"last update":"Mon Oct 16 13:49:43 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.56299999999999994,"15min":0.54200000000000004},"min":{"1min":0.33900000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.86499999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.63,"5min":0.58199999999999996,"15min":0.57499999999999996},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.94899999999999995},"last":0.72899999999999998}]},{"osd":6,"last update":"Mon Oct 16 13:49:43 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.63100000000000001,"15min":0.60699999999999998},"min":{"1min":0.375,"5min":0.36399999999999999,"15min":0.33200000000000002},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.69699999999999995,"15min":0.69599999999999995},"min":{"1min":0.47299999999999998,"5min":0.40300000000000002,"15min":0.39400000000000002},"max":{"1min":0.94399999999999995,"5min":1.129,"15min":1.129},"last":0.78800000000000003}]},{"osd":7,"last update":"Mon Oct 16 13:49:43 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.64600000000000002,"15min":0.64000000000000001},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1,"5min":1,"15min":1},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.70099999999999996,"15min":0.69199999999999995},"min":{"1min":0.435,"5min":0.435,"15min":0.38200000000000001},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.70899999999999996}]}]},{"osd":5,"up_from":13,"seq":55834576020,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":74944,"kb_used_data":3504,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94296896,"statfs":{"total":96636764160,"available":96560021504,"internally_reserved":0,"allocated":3588096,"data_stored":1863001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 13:49:30 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.54100000000000004,"15min":0.59999999999999998},"min":{"1min":0.39000000000000001,"5min":0.314,"15min":0.29899999999999999},"max":{"1min":0.73399999999999999,"5min":0.878,"15min":0.92900000000000005},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.56399999999999995,"15min":0.61699999999999999},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.29699999999999999},"max":{"1min":0.80000000000000004,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.64100000000000001}]},{"osd":1,"last update":"Mon Oct 16 13:49:30 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.58699999999999997,"15min":0.63900000000000001},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.29499999999999998},"max":{"1min":0.72999999999999998,"5min":1,"15min":1},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56799999999999995,"15min":0.62},"min":{"1min":0.45400000000000001,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.78100000000000003,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.72299999999999998}]},{"osd":2,"last update":"Mon Oct 16 13:49:30 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.58999999999999997,"15min":0.63600000000000001},"min":{"1min":0.38100000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.71199999999999997,"5min":0.91000000000000003,"15min":0.94699999999999995},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.58099999999999996,"15min":0.63700000000000001},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.81699999999999995,"5min":0.83899999999999997,"15min":0.93500000000000005},"last":0.68400000000000005}]},{"osd":3,"last update":"Mon Oct 16 13:49:30 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.61699999999999999,"15min":0.65200000000000002},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.83599999999999997,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.63500000000000001,"15min":0.68000000000000005},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.84799999999999998,"5min":1.4199999999999999,"15min":1.4199999999999999},"last":0.66700000000000004}]},{"osd":4,"last update":"Mon Oct 16 13:49:30 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53600000000000003,"15min":0.54000000000000004},"min":{"1min":0.33300000000000002,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.77200000000000002,"5min":0.89700000000000002,"15min":0.90700000000000003},"last":0.373},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.60099999999999998,"15min":0.59499999999999997},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.84799999999999998,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.38800000000000001}]},{"osd":6,"last update":"Mon Oct 16 13:49:30 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.59499999999999997,"15min":0.59099999999999997},"min":{"1min":0.35699999999999998,"5min":0.315,"15min":0.315},"max":{"1min":0.82799999999999996,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.61499999999999999,"15min":0.623},"min":{"1min":0.374,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.82799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.497}]},{"osd":7,"last update":"Mon Oct 16 13:49:30 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.65800000000000003,"15min":0.64300000000000002},"min":{"1min":0.374,"5min":0.374,"15min":0.34999999999999998},"max":{"1min":0.878,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.69499999999999995,"15min":0.69099999999999995},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.86899999999999999,"5min":0.97299999999999998,"15min":1.0229999999999999},"last":0.54500000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T13:50:02.694 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-10-16T13:50:02.819 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.817+0000 7f4f23e60700 1 Processor -- start 2023-10-16T13:50:02.820 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.817+0000 7f4f23e60700 1 -- start start 2023-10-16T13:50:02.823 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.818+0000 7f4f23e60700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4f1c15da10 0x7f4f1c0d1520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:02.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.818+0000 7f4f23e60700 1 --2- >> v2:172.21.15.49:3300/0 conn(0x7f4f1c0d1d20 0x7f4f1c0d2130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:02.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.818+0000 7f4f23e60700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f1c0d2670 con 0x7f4f1c0d4060 2023-10-16T13:50:02.824 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.818+0000 7f4f23e60700 1 -- --> v2:172.21.15.49:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f1c0cf9b0 con 0x7f4f1c0d1d20 2023-10-16T13:50:02.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.818+0000 7f4f23e60700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1c0d4540 con 0x7f4f1c15da10 2023-10-16T13:50:02.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.818+0000 7f4f21bfc700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4f1c15da10 0x7f4f1c0d1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:02.825 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.818+0000 7f4f21bfc700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4f1c15da10 0x7f4f1c0d1520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3301/0 says I am v2:172.21.15.16:48936/0 (socket says 172.21.15.16:48936) 2023-10-16T13:50:02.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.818+0000 7f4f21bfc700 1 -- 172.21.15.16:0/2925686432 learned_addr learned my addr 172.21.15.16:0/2925686432 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:50:02.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.818+0000 7f4f223fd700 1 --2- 172.21.15.16:0/2925686432 >> v2:172.21.15.49:3300/0 conn(0x7f4f1c0d1d20 0x7f4f1c0d2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:02.826 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.818+0000 7f4f20bfa700 1 -- 172.21.15.16:0/2925686432 <== mon.1 v1:172.21.15.49:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1659208178 0 0) 0x7f4f1c0d2670 con 0x7f4f1c0d4060 2023-10-16T13:50:02.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.818+0000 7f4f20bfa700 1 -- 172.21.15.16:0/2925686432 --> v1:172.21.15.49:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f080035b0 con 0x7f4f1c0d4060 2023-10-16T13:50:02.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.818+0000 7f4f21bfc700 1 -- 172.21.15.16:0/2925686432 >> v2:172.21.15.49:3300/0 conn(0x7f4f1c0d1d20 msgr2=0x7f4f1c0d2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T13:50:02.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.818+0000 7f4f21bfc700 1 --2- 172.21.15.16:0/2925686432 >> v2:172.21.15.49:3300/0 conn(0x7f4f1c0d1d20 0x7f4f1c0d2130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:02.827 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.818+0000 7f4f21bfc700 1 -- 172.21.15.16:0/2925686432 >> v1:172.21.15.49:6789/0 conn(0x7f4f1c0d4060 legacy=0x7f4f1c0cf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:02.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.818+0000 7f4f21bfc700 1 -- 172.21.15.16:0/2925686432 --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f1c0ce7e0 con 0x7f4f1c15da10 2023-10-16T13:50:02.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.818+0000 7f4f223fd700 1 --2- 172.21.15.16:0/2925686432 >> v2:172.21.15.49:3300/0 conn(0x7f4f1c0d1d20 0x7f4f1c0d2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T13:50:02.828 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.819+0000 7f4f21bfc700 1 --2- 172.21.15.16:0/2925686432 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4f1c15da10 0x7f4f1c0d1520 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f4f18006fd0 tx=0x7f4f18009bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=68fa303e5f03a17e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:02.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.819+0000 7f4f20bfa700 1 -- 172.21.15.16:0/2925686432 <== mon.2 v2:172.21.15.49:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f18005e30 con 0x7f4f1c15da10 2023-10-16T13:50:02.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.819+0000 7f4f20bfa700 1 -- 172.21.15.16:0/2925686432 <== mon.2 v2:172.21.15.49:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f18004030 con 0x7f4f1c15da10 2023-10-16T13:50:02.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.819+0000 7f4f20bfa700 1 -- 172.21.15.16:0/2925686432 <== mon.2 v2:172.21.15.49:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f180043e0 con 0x7f4f1c15da10 2023-10-16T13:50:02.829 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.819+0000 7f4f23e60700 1 -- 172.21.15.16:0/2925686432 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4f1c15da10 msgr2=0x7f4f1c0d1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:02.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.819+0000 7f4f23e60700 1 --2- 172.21.15.16:0/2925686432 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4f1c15da10 0x7f4f1c0d1520 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f4f18006fd0 tx=0x7f4f18009bd0 comp rx=0 tx=0).stop 2023-10-16T13:50:02.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.820+0000 7f4f23e60700 1 -- 172.21.15.16:0/2925686432 shutdown_connections 2023-10-16T13:50:02.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.820+0000 7f4f23e60700 1 --2- 172.21.15.16:0/2925686432 >> v2:172.21.15.49:3300/0 conn(0x7f4f1c0d1d20 0x7f4f1c0d2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:02.830 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.820+0000 7f4f23e60700 1 --2- 172.21.15.16:0/2925686432 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4f1c15da10 0x7f4f1c0d1520 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:02.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.820+0000 7f4f23e60700 1 -- 172.21.15.16:0/2925686432 >> 172.21.15.16:0/2925686432 conn(0x7f4f1c09c9f0 msgr2=0x7f4f1c0d3740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:02.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.820+0000 7f4f23e60700 1 -- 172.21.15.16:0/2925686432 shutdown_connections 2023-10-16T13:50:02.831 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.820+0000 7f4f23e60700 1 -- 172.21.15.16:0/2925686432 wait complete. 2023-10-16T13:50:02.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.821+0000 7f4f23e60700 1 Processor -- start 2023-10-16T13:50:02.832 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.821+0000 7f4f23e60700 1 -- start start 2023-10-16T13:50:02.834 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.821+0000 7f4f23e60700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4f1c0d1d20 0x7f4f1c1f7890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:02.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.822+0000 7f4f23e60700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4f1c0d4060 0x7f4f1c14ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:02.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.822+0000 7f4f23e60700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f1c15da10 0x7f4f1c0b51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:02.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.822+0000 7f4f23e60700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1c14f140 con 0x7f4f1c15da10 2023-10-16T13:50:02.835 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.822+0000 7f4f23e60700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1c0b63d0 con 0x7f4f1c0d1d20 2023-10-16T13:50:02.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.822+0000 7f4f23e60700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1c0b66b0 con 0x7f4f1c0d4060 2023-10-16T13:50:02.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.822+0000 7f4f223fd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f1c15da10 0x7f4f1c0b51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:02.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.822+0000 7f4f223fd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f1c15da10 0x7f4f1c0b51c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60688/0 (socket says 172.21.15.16:60688) 2023-10-16T13:50:02.836 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.822+0000 7f4f223fd700 1 -- 172.21.15.16:0/3777246226 learned_addr learned my addr 172.21.15.16:0/3777246226 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:50:02.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.822+0000 7f4f21bfc700 1 --2- 172.21.15.16:0/3777246226 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4f1c0d1d20 0x7f4f1c1f7890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:02.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.822+0000 7f4f213fb700 1 --2- 172.21.15.16:0/3777246226 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4f1c0d4060 0x7f4f1c14ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:02.837 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.822+0000 7f4f223fd700 1 -- 172.21.15.16:0/3777246226 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4f1c0d4060 msgr2=0x7f4f1c14ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:02.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.822+0000 7f4f223fd700 1 --2- 172.21.15.16:0/3777246226 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4f1c0d4060 0x7f4f1c14ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:02.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.822+0000 7f4f223fd700 1 -- 172.21.15.16:0/3777246226 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4f1c0d1d20 msgr2=0x7f4f1c1f7890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:02.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.822+0000 7f4f223fd700 1 --2- 172.21.15.16:0/3777246226 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4f1c0d1d20 0x7f4f1c1f7890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:02.838 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.822+0000 7f4f223fd700 1 -- 172.21.15.16:0/3777246226 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f14009a70 con 0x7f4f1c15da10 2023-10-16T13:50:02.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.822+0000 7f4f223fd700 1 --2- 172.21.15.16:0/3777246226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f1c15da10 0x7f4f1c0b51c0 secure :-1 s=READY pgs=3375 cs=0 l=1 rev1=1 crypto rx=0x7f4f1400efc0 tx=0x7f4f1400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:02.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.822+0000 7f4f127fc700 1 -- 172.21.15.16:0/3777246226 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f1400ca90 con 0x7f4f1c15da10 2023-10-16T13:50:02.839 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.822+0000 7f4f127fc700 1 -- 172.21.15.16:0/3777246226 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f1400f040 con 0x7f4f1c15da10 2023-10-16T13:50:02.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.822+0000 7f4f127fc700 1 -- 172.21.15.16:0/3777246226 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f1400ce10 con 0x7f4f1c15da10 2023-10-16T13:50:02.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.822+0000 7f4f23e60700 1 -- 172.21.15.16:0/3777246226 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f18009500 con 0x7f4f1c15da10 2023-10-16T13:50:02.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.823+0000 7f4f23e60700 1 -- 172.21.15.16:0/3777246226 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f1c0b6c70 con 0x7f4f1c15da10 2023-10-16T13:50:02.840 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.824+0000 7f4f23e60700 1 -- 172.21.15.16:0/3777246226 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f1c067870 con 0x7f4f1c15da10 2023-10-16T13:50:02.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.824+0000 7f4f127fc700 1 -- 172.21.15.16:0/3777246226 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 176) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4f14014480 con 0x7f4f1c15da10 2023-10-16T13:50:02.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.824+0000 7f4f127fc700 1 --2- 172.21.15.16:0/3777246226 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f4f0006b020 0x7f4f0006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:02.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.825+0000 7f4f21bfc700 1 --2- 172.21.15.16:0/3777246226 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f4f0006b020 0x7f4f0006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:02.841 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.825+0000 7f4f127fc700 1 -- 172.21.15.16:0/3777246226 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(283..283 src has 1..283) v4 ==== 6208+0+0 (secure 0 0 0) 0x7f4f1408a010 con 0x7f4f1c15da10 2023-10-16T13:50:02.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.825+0000 7f4f21bfc700 1 --2- 172.21.15.16:0/3777246226 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f4f0006b020 0x7f4f0006d4e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4f18006fd0 tx=0x7f4f18008070 comp rx=0 tx=0).ready entity=mgr.13935 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:02.842 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:02.829+0000 7f4f127fc700 1 -- 172.21.15.16:0/3777246226 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4f14054d40 con 0x7f4f1c15da10 2023-10-16T13:50:03.035 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.033+0000 7f4f23e60700 1 -- 172.21.15.16:0/3777246226 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7f4f1c04ebb0 con 0x7f4f1c15da10 2023-10-16T13:50:03.038 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.036+0000 7f4f127fc700 1 -- 172.21.15.16:0/3777246226 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v100)=0 v100) v1 ==== 123+0+0 (secure 0 0 0) 0x7f4f14009d70 con 0x7f4f1c15da10 2023-10-16T13:50:03.039 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.037+0000 7f4f127fc700 1 -- 172.21.15.16:0/3777246226 <== mon.0 v2:172.21.15.16:3300/0 8 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f4f14054d40 con 0x7f4f1c15da10 2023-10-16T13:50:03.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.038+0000 7f4f23e60700 1 -- 172.21.15.16:0/3777246226 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f4f0006b020 msgr2=0x7f4f0006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:03.041 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.038+0000 7f4f23e60700 1 --2- 172.21.15.16:0/3777246226 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f4f0006b020 0x7f4f0006d4e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4f18006fd0 tx=0x7f4f18008070 comp rx=0 tx=0).stop 2023-10-16T13:50:03.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.038+0000 7f4f23e60700 1 -- 172.21.15.16:0/3777246226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f1c15da10 msgr2=0x7f4f1c0b51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:03.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.038+0000 7f4f23e60700 1 --2- 172.21.15.16:0/3777246226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f1c15da10 0x7f4f1c0b51c0 secure :-1 s=READY pgs=3375 cs=0 l=1 rev1=1 crypto rx=0x7f4f1400efc0 tx=0x7f4f1400c3a0 comp rx=0 tx=0).stop 2023-10-16T13:50:03.042 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.038+0000 7f4f23e60700 1 -- 172.21.15.16:0/3777246226 shutdown_connections 2023-10-16T13:50:03.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.038+0000 7f4f23e60700 1 --2- 172.21.15.16:0/3777246226 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f4f0006b020 0x7f4f0006d4e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:03.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.038+0000 7f4f23e60700 1 --2- 172.21.15.16:0/3777246226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f1c15da10 0x7f4f1c0b51c0 unknown :-1 s=CLOSED pgs=3375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:03.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.038+0000 7f4f23e60700 1 --2- 172.21.15.16:0/3777246226 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4f1c0d1d20 0x7f4f1c1f7890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:03.043 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.038+0000 7f4f23e60700 1 --2- 172.21.15.16:0/3777246226 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f4f1c0d4060 0x7f4f1c14ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:03.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.038+0000 7f4f23e60700 1 -- 172.21.15.16:0/3777246226 >> 172.21.15.16:0/3777246226 conn(0x7f4f1c09c9f0 msgr2=0x7f4f1c15ddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:03.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.039+0000 7f4f23e60700 1 -- 172.21.15.16:0/3777246226 shutdown_connections 2023-10-16T13:50:03.044 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.039+0000 7f4f23e60700 1 -- 172.21.15.16:0/3777246226 wait complete. 2023-10-16T13:50:03.053 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-16T13:50:03.054 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T13:50:03.178 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.176+0000 7f08f1f49700 1 Processor -- start 2023-10-16T13:50:03.179 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.176+0000 7f08f1f49700 1 -- start start 2023-10-16T13:50:03.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.177+0000 7f08f1f49700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f08ec0ce380 0x7f08ec0d0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:03.181 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.177+0000 7f08f1f49700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08ec0d1210 con 0x7f08ec0b2b50 2023-10-16T13:50:03.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.177+0000 7f08f1f49700 1 -- --> v1:172.21.15.49:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08ec0cbc30 con 0x7f08ec0cdfa0 2023-10-16T13:50:03.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.177+0000 7f08f1f49700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08ec0cbf10 con 0x7f08ec0ce380 2023-10-16T13:50:03.182 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.177+0000 7f08ebfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f08ec0ce380 0x7f08ec0d0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:03.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.177+0000 7f08eaffd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f08ec0b2b50 0x7f08ec0d04a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49968/0 (socket says 172.21.15.16:49968) 2023-10-16T13:50:03.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.177+0000 7f08eaffd700 1 -- 172.21.15.16:0/8483900 learned_addr learned my addr 172.21.15.16:0/8483900 (peer_addr_for_me v1:172.21.15.16:0/0) 2023-10-16T13:50:03.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.177+0000 7f08ebfff700 1 -- 172.21.15.16:0/8483900 >> v1:172.21.15.49:6789/0 conn(0x7f08ec0cdfa0 legacy=0x7f08ec0b2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:03.183 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.177+0000 7f08ebfff700 1 -- 172.21.15.16:0/8483900 >> v1:172.21.15.16:6789/0 conn(0x7f08ec0b2b50 legacy=0x7f08ec0d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:03.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.177+0000 7f08ebfff700 1 -- 172.21.15.16:0/8483900 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08ec0cc1f0 con 0x7f08ec0ce380 2023-10-16T13:50:03.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.177+0000 7f08ebfff700 1 --2- 172.21.15.16:0/8483900 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08ec0ce380 0x7f08ec0d0bb0 secure :-1 s=READY pgs=3377 cs=0 l=1 rev1=1 crypto rx=0x7f08e4009d90 tx=0x7f08e400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6791532bcd92ff8d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:03.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.178+0000 7f08ea7fc700 1 -- 172.21.15.16:0/8483900 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08e400e070 con 0x7f08ec0ce380 2023-10-16T13:50:03.184 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.178+0000 7f08ea7fc700 1 -- 172.21.15.16:0/8483900 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f08e400bbd0 con 0x7f08ec0ce380 2023-10-16T13:50:03.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.178+0000 7f08ea7fc700 1 -- 172.21.15.16:0/8483900 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08e4012430 con 0x7f08ec0ce380 2023-10-16T13:50:03.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.178+0000 7f08f1f49700 1 -- 172.21.15.16:0/8483900 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08ec0ce380 msgr2=0x7f08ec0d0bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:03.185 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.178+0000 7f08f1f49700 1 --2- 172.21.15.16:0/8483900 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08ec0ce380 0x7f08ec0d0bb0 secure :-1 s=READY pgs=3377 cs=0 l=1 rev1=1 crypto rx=0x7f08e4009d90 tx=0x7f08e400b330 comp rx=0 tx=0).stop 2023-10-16T13:50:03.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.178+0000 7f08f1f49700 1 -- 172.21.15.16:0/8483900 shutdown_connections 2023-10-16T13:50:03.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.178+0000 7f08f1f49700 1 --2- 172.21.15.16:0/8483900 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08ec0ce380 0x7f08ec0d0bb0 unknown :-1 s=CLOSED pgs=3377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:03.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.178+0000 7f08f1f49700 1 -- 172.21.15.16:0/8483900 >> 172.21.15.16:0/8483900 conn(0x7f08ec0d14b0 msgr2=0x7f08ec0d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:03.186 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.178+0000 7f08f1f49700 1 -- 172.21.15.16:0/8483900 shutdown_connections 2023-10-16T13:50:03.187 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.178+0000 7f08f1f49700 1 -- 172.21.15.16:0/8483900 wait complete. 2023-10-16T13:50:03.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.179+0000 7f08f1f49700 1 Processor -- start 2023-10-16T13:50:03.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.180+0000 7f08f1f49700 1 -- start start 2023-10-16T13:50:03.190 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.180+0000 7f08f1f49700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f08ec0b2b50 0x7f08ec15dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:03.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.180+0000 7f08f1f49700 1 --2- >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f08ec0cdfa0 0x7f08ec1f6750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:03.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.180+0000 7f08f1f49700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08ec0ce380 0x7f08ec0c8e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:03.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.180+0000 7f08f1f49700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08ec1f70d0 con 0x7f08ec0ce380 2023-10-16T13:50:03.191 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.180+0000 7f08f1f49700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08ec1f73b0 con 0x7f08ec0b2b50 2023-10-16T13:50:03.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.180+0000 7f08f1f49700 1 -- --> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08ec1f7690 con 0x7f08ec0cdfa0 2023-10-16T13:50:03.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.181+0000 7f08ebfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08ec0ce380 0x7f08ec0c8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:03.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.181+0000 7f08ebfff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08ec0ce380 0x7f08ec0c8e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60702/0 (socket says 172.21.15.16:60702) 2023-10-16T13:50:03.192 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.181+0000 7f08ebfff700 1 -- 172.21.15.16:0/3235699038 learned_addr learned my addr 172.21.15.16:0/3235699038 (peer_addr_for_me v2:172.21.15.16:0/0) 2023-10-16T13:50:03.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.181+0000 7f08eb7fe700 1 --2- 172.21.15.16:0/3235699038 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f08ec0b2b50 0x7f08ec15dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:03.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.181+0000 7f08eaffd700 1 --2- 172.21.15.16:0/3235699038 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f08ec0cdfa0 0x7f08ec1f6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:03.193 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.181+0000 7f08ebfff700 1 -- 172.21.15.16:0/3235699038 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f08ec0cdfa0 msgr2=0x7f08ec1f6750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:03.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.181+0000 7f08ebfff700 1 --2- 172.21.15.16:0/3235699038 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f08ec0cdfa0 0x7f08ec1f6750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:03.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.181+0000 7f08ebfff700 1 -- 172.21.15.16:0/3235699038 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f08ec0b2b50 msgr2=0x7f08ec15dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:03.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.181+0000 7f08ebfff700 1 --2- 172.21.15.16:0/3235699038 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f08ec0b2b50 0x7f08ec15dc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:03.194 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.181+0000 7f08ebfff700 1 -- 172.21.15.16:0/3235699038 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08e4009a70 con 0x7f08ec0ce380 2023-10-16T13:50:03.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.181+0000 7f08ebfff700 1 --2- 172.21.15.16:0/3235699038 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08ec0ce380 0x7f08ec0c8e20 secure :-1 s=READY pgs=3378 cs=0 l=1 rev1=1 crypto rx=0x7f08e4014040 tx=0x7f08e4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:03.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.181+0000 7f08e3fff700 1 -- 172.21.15.16:0/3235699038 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08e4012d20 con 0x7f08ec0ce380 2023-10-16T13:50:03.195 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.181+0000 7f08e3fff700 1 -- 172.21.15.16:0/3235699038 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f08e4020070 con 0x7f08ec0ce380 2023-10-16T13:50:03.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.181+0000 7f08e3fff700 1 -- 172.21.15.16:0/3235699038 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08e4012ea0 con 0x7f08ec0ce380 2023-10-16T13:50:03.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.181+0000 7f08f1f49700 1 -- 172.21.15.16:0/3235699038 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08ec0c8620 con 0x7f08ec0ce380 2023-10-16T13:50:03.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.181+0000 7f08f1f49700 1 -- 172.21.15.16:0/3235699038 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08ec15e1c0 con 0x7f08ec0ce380 2023-10-16T13:50:03.196 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.182+0000 7f08f1f49700 1 -- 172.21.15.16:0/3235699038 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08ec04ebb0 con 0x7f08ec0ce380 2023-10-16T13:50:03.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.183+0000 7f08e3fff700 1 -- 172.21.15.16:0/3235699038 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 176) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f08e40042a0 con 0x7f08ec0ce380 2023-10-16T13:50:03.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.183+0000 7f08e3fff700 1 --2- 172.21.15.16:0/3235699038 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f08cc06b050 0x7f08cc06d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T13:50:03.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.183+0000 7f08e3fff700 1 -- 172.21.15.16:0/3235699038 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(283..283 src has 1..283) v4 ==== 6208+0+0 (secure 0 0 0) 0x7f08e4022070 con 0x7f08ec0ce380 2023-10-16T13:50:03.197 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.183+0000 7f08eb7fe700 1 --2- 172.21.15.16:0/3235699038 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f08cc06b050 0x7f08cc06d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T13:50:03.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.183+0000 7f08eb7fe700 1 --2- 172.21.15.16:0/3235699038 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f08cc06b050 0x7f08cc06d510 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f08ec06f580 tx=0x7f08d4009040 comp rx=0 tx=0).ready entity=mgr.13935 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T13:50:03.198 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.188+0000 7f08e3fff700 1 -- 172.21.15.16:0/3235699038 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f08e40564c0 con 0x7f08ec0ce380 2023-10-16T13:50:03.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.437+0000 7f08f1f49700 1 -- 172.21.15.16:0/3235699038 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f08ec0cb7e0 con 0x7f08ec0ce380 2023-10-16T13:50:03.440 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.437+0000 7f08e3fff700 1 -- 172.21.15.16:0/3235699038 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 157 v157) v1 ==== 95+0+3705 (secure 0 0 0) 0x7f08e402ce90 con 0x7f08ec0ce380 2023-10-16T13:50:03.440 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-10-16T13:50:03.441 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":157,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12280,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6834/1127700443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":1127700443},{"type":"v1","addr":"172.21.15.49:6834","nonce":1127700443}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":150},{"gid":12655,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/1825684358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6833","nonce":1825684358},{"type":"v1","addr":"172.21.15.49:6835","nonce":1825684358}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":157},{"gid":13581,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6836/818113029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":818113029},{"type":"v1","addr":"172.21.15.16:6836","nonce":818113029}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149},{"gid":13584,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3762868123","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6835","nonce":3762868123},{"type":"v1","addr":"172.21.15.16:6837","nonce":3762868123}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":150}],"filesystems":[]} 2023-10-16T13:50:03.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.440+0000 7f08f1f49700 1 -- 172.21.15.16:0/3235699038 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f08cc06b050 msgr2=0x7f08cc06d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:03.442 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.440+0000 7f08f1f49700 1 --2- 172.21.15.16:0/3235699038 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f08cc06b050 0x7f08cc06d510 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f08ec06f580 tx=0x7f08d4009040 comp rx=0 tx=0).stop 2023-10-16T13:50:03.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.440+0000 7f08f1f49700 1 -- 172.21.15.16:0/3235699038 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08ec0ce380 msgr2=0x7f08ec0c8e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T13:50:03.443 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.440+0000 7f08f1f49700 1 --2- 172.21.15.16:0/3235699038 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08ec0ce380 0x7f08ec0c8e20 secure :-1 s=READY pgs=3378 cs=0 l=1 rev1=1 crypto rx=0x7f08e4014040 tx=0x7f08e4004ad0 comp rx=0 tx=0).stop 2023-10-16T13:50:03.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.440+0000 7f08f1f49700 1 -- 172.21.15.16:0/3235699038 shutdown_connections 2023-10-16T13:50:03.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.440+0000 7f08f1f49700 1 --2- 172.21.15.16:0/3235699038 >> [v2:172.21.15.16:6832/938897590,v1:172.21.15.16:6833/938897590] conn(0x7f08cc06b050 0x7f08cc06d510 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:03.444 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.440+0000 7f08f1f49700 1 --2- 172.21.15.16:0/3235699038 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08ec0ce380 0x7f08ec0c8e20 unknown :-1 s=CLOSED pgs=3378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:03.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.440+0000 7f08f1f49700 1 --2- 172.21.15.16:0/3235699038 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f08ec0b2b50 0x7f08ec15dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:03.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.440+0000 7f08f1f49700 1 --2- 172.21.15.16:0/3235699038 >> [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] conn(0x7f08ec0cdfa0 0x7f08ec1f6750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T13:50:03.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.440+0000 7f08f1f49700 1 -- 172.21.15.16:0/3235699038 >> 172.21.15.16:0/3235699038 conn(0x7f08ec0d14b0 msgr2=0x7f08ec0b2440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T13:50:03.445 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.440+0000 7f08f1f49700 1 -- 172.21.15.16:0/3235699038 shutdown_connections 2023-10-16T13:50:03.446 INFO:teuthology.orchestra.run.smithi016.stderr:2023-10-16T13:50:03.440+0000 7f08f1f49700 1 -- 172.21.15.16:0/3235699038 wait complete. 2023-10-16T13:50:03.446 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 157 2023-10-16T13:50:03.454 DEBUG:tasks.cephfs.filesystem:already dead... 2023-10-16T13:50:03.455 INFO:teuthology.misc:Shutting down mds daemons... 2023-10-16T13:50:03.455 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-16T13:50:03.456 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T13:50:03.464 INFO:tasks.ceph.mds.a:Stopped 2023-10-16T13:50:03.464 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-16T13:50:03.465 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T13:50:03.513 INFO:tasks.ceph.mds.c:Stopped 2023-10-16T13:50:03.514 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-16T13:50:03.514 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T13:50:03.526 INFO:tasks.ceph.mds.b:Stopped 2023-10-16T13:50:03.526 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-16T13:50:03.527 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T13:50:03.577 INFO:tasks.ceph.mds.d:Stopped 2023-10-16T13:50:03.578 INFO:teuthology.misc:Shutting down osd daemons... 2023-10-16T13:50:03.578 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-10-16T13:50:03.578 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T13:50:03.611 INFO:tasks.ceph.osd.0:Stopped 2023-10-16T13:50:03.611 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-10-16T13:50:03.612 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T13:50:03.684 INFO:tasks.ceph.osd.1:Stopped 2023-10-16T13:50:03.685 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-10-16T13:50:03.685 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T13:50:03.762 INFO:tasks.ceph.osd.2:Stopped 2023-10-16T13:50:03.763 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-10-16T13:50:03.763 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T13:50:03.831 INFO:tasks.ceph.osd.3:Stopped 2023-10-16T13:50:03.832 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-10-16T13:50:03.832 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T13:50:03.874 INFO:tasks.ceph.osd.4:Stopped 2023-10-16T13:50:03.875 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-10-16T13:50:03.875 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T13:50:03.917 INFO:tasks.ceph.osd.5:Stopped 2023-10-16T13:50:03.917 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2023-10-16T13:50:03.918 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T13:50:03.970 INFO:tasks.ceph.osd.6:Stopped 2023-10-16T13:50:03.971 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2023-10-16T13:50:03.971 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T13:50:04.017 INFO:tasks.ceph.osd.7:Stopped 2023-10-16T13:50:04.017 INFO:teuthology.misc:Shutting down mgr daemons... 2023-10-16T13:50:04.018 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-10-16T13:50:04.018 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T13:50:04.049 INFO:tasks.ceph.mgr.y:Stopped 2023-10-16T13:50:04.050 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-10-16T13:50:04.050 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T13:50:04.076 INFO:tasks.ceph.mgr.x:Stopped 2023-10-16T13:50:04.076 INFO:teuthology.misc:Shutting down mon daemons... 2023-10-16T13:50:04.077 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-10-16T13:50:04.077 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T13:50:04.082 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:50:04.091 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:50:04.106 INFO:tasks.ceph.mon.a:Stopped 2023-10-16T13:50:04.106 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-10-16T13:50:04.107 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T13:50:04.132 INFO:tasks.ceph.mon.b:Stopped 2023-10-16T13:50:04.133 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-10-16T13:50:04.133 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T13:50:04.187 INFO:tasks.ceph.mon.c:Stopped 2023-10-16T13:50:04.187 INFO:tasks.ceph:Checking cluster log for badness... 2023-10-16T13:50:04.188 DEBUG:teuthology.orchestra.run.smithi016:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-10-16T13:50:04.220 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi016.front.sepia.ceph.com 2023-10-16T13:50:04.220 DEBUG:teuthology.orchestra.run.smithi016:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-10-16T13:50:04.489 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi016.front.sepia.ceph.com 2023-10-16T13:50:04.490 DEBUG:teuthology.orchestra.run.smithi016:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-10-16T13:50:04.630 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi016.front.sepia.ceph.com 2023-10-16T13:50:04.630 DEBUG:teuthology.orchestra.run.smithi016:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-10-16T13:50:04.709 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi016.front.sepia.ceph.com 2023-10-16T13:50:04.710 DEBUG:teuthology.orchestra.run.smithi016:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-10-16T13:50:04.813 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi049.front.sepia.ceph.com 2023-10-16T13:50:04.813 DEBUG:teuthology.orchestra.run.smithi049:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-10-16T13:50:05.063 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi049.front.sepia.ceph.com 2023-10-16T13:50:05.065 DEBUG:teuthology.orchestra.run.smithi049:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-10-16T13:50:05.143 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi049.front.sepia.ceph.com 2023-10-16T13:50:05.144 DEBUG:teuthology.orchestra.run.smithi049:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2023-10-16T13:50:05.232 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi049.front.sepia.ceph.com 2023-10-16T13:50:05.232 DEBUG:teuthology.orchestra.run.smithi049:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2023-10-16T13:50:05.312 INFO:tasks.ceph:Archiving mon data... 2023-10-16T13:50:05.313 DEBUG:teuthology.misc:Transferring archived files from smithi016:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/mchangir-2023-10-16_11:33:41-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi/7428984/data/mon.a.tgz 2023-10-16T13:50:05.314 DEBUG:teuthology.orchestra.run.smithi016:> mktemp 2023-10-16T13:50:05.333 INFO:teuthology.orchestra.run.smithi016.stdout:/tmp/tmp.Xl0CNT5MxA 2023-10-16T13:50:05.333 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.Xl0CNT5MxA 2023-10-16T13:50:06.038 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 0666 /tmp/tmp.Xl0CNT5MxA 2023-10-16T13:50:06.125 DEBUG:teuthology.orchestra.remote:smithi016:/tmp/tmp.Xl0CNT5MxA is 4MB 2023-10-16T13:50:06.284 DEBUG:teuthology.orchestra.run.smithi016:> rm -fr /tmp/tmp.Xl0CNT5MxA 2023-10-16T13:50:06.301 DEBUG:teuthology.misc:Transferring archived files from smithi049:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/mchangir-2023-10-16_11:33:41-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi/7428984/data/mon.b.tgz 2023-10-16T13:50:06.302 DEBUG:teuthology.orchestra.run.smithi049:> mktemp 2023-10-16T13:50:06.323 INFO:teuthology.orchestra.run.smithi049.stdout:/tmp/tmp.Lw11hicjcd 2023-10-16T13:50:06.323 DEBUG:teuthology.orchestra.run.smithi049:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.Lw11hicjcd 2023-10-16T13:50:07.055 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod 0666 /tmp/tmp.Lw11hicjcd 2023-10-16T13:50:07.142 DEBUG:teuthology.orchestra.remote:smithi049:/tmp/tmp.Lw11hicjcd is 4MB 2023-10-16T13:50:07.288 DEBUG:teuthology.orchestra.run.smithi049:> rm -fr /tmp/tmp.Lw11hicjcd 2023-10-16T13:50:07.306 DEBUG:teuthology.misc:Transferring archived files from smithi049:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/mchangir-2023-10-16_11:33:41-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi/7428984/data/mon.c.tgz 2023-10-16T13:50:07.306 DEBUG:teuthology.orchestra.run.smithi049:> mktemp 2023-10-16T13:50:07.362 INFO:teuthology.orchestra.run.smithi049.stdout:/tmp/tmp.IIhaIEueB3 2023-10-16T13:50:07.363 DEBUG:teuthology.orchestra.run.smithi049:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.IIhaIEueB3 2023-10-16T13:50:08.057 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod 0666 /tmp/tmp.IIhaIEueB3 2023-10-16T13:50:08.144 DEBUG:teuthology.orchestra.remote:smithi049:/tmp/tmp.IIhaIEueB3 is 4MB 2023-10-16T13:50:08.303 DEBUG:teuthology.orchestra.run.smithi049:> rm -fr /tmp/tmp.IIhaIEueB3 2023-10-16T13:50:08.320 INFO:tasks.ceph:Cleaning ceph cluster... 2023-10-16T13:50:08.321 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-16T13:50:08.323 DEBUG:teuthology.orchestra.run.smithi049:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-16T13:50:08.387 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-10-16T13:50:08.388 DEBUG:teuthology.orchestra.run.smithi016:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-16T13:50:08.396 DEBUG:teuthology.orchestra.run.smithi049:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-16T13:50:08.433 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-16T13:50:08.483 INFO:teuthology.orchestra.run.smithi049.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-16T13:50:08.489 INFO:tasks.ceph:Archiving crash dumps... 2023-10-16T13:50:08.491 DEBUG:teuthology.misc:Transferring archived files from smithi016:/var/lib/ceph/crash to /home/teuthworker/archive/mchangir-2023-10-16_11:33:41-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi/7428984/remote/smithi016/crash 2023-10-16T13:50:08.492 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-16T13:50:08.523 DEBUG:teuthology.misc:Transferring archived files from smithi049:/var/lib/ceph/crash to /home/teuthworker/archive/mchangir-2023-10-16_11:33:41-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi/7428984/remote/smithi049/crash 2023-10-16T13:50:08.524 DEBUG:teuthology.orchestra.run.smithi049:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-16T13:50:08.551 INFO:tasks.ceph:Shutting down logrotate 2023-10-16T13:50:08.552 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:50:08.568 DEBUG:teuthology.orchestra.run.smithi049:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T13:50:08.679 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-10-16T13:50:08.709 DEBUG:teuthology.orchestra.run.smithi049:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-10-16T13:50:08.736 INFO:tasks.ceph:Compressing logs... 2023-10-16T13:50:08.737 DEBUG:teuthology.orchestra.run.smithi016:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-16T13:50:08.751 DEBUG:teuthology.orchestra.run.smithi049:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-16T13:50:08.780 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146097.log 2023-10-16T13:50:08.780 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122066.log 2023-10-16T13:50:08.780 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.146097.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.183740.log 2023-10-16T13:50:08.781 INFO:teuthology.orchestra.run.smithi016.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.146097.log.gz 2023-10-16T13:50:08.781 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.172284.log 2023-10-16T13:50:08.817 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2023-10-16T13:50:08.817 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-10-16T13:50:08.818 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104122.log 2023-10-16T13:50:08.818 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2023-10-16T13:50:08.818 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.104122.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103938.log 2023-10-16T13:50:08.819 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.110689.log 2023-10-16T13:50:08.819 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-osd.6.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.104122.log.gz 2023-10-16T13:50:08.819 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.103938.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2023-10-16T13:50:08.819 INFO:teuthology.orchestra.run.smithi049.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.103938.log.gz 2023-10-16T13:50:08.820 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.1.110689.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-10-16T13:50:08.820 INFO:teuthology.orchestra.run.smithi049.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.1.110689.log.gz 2023-10-16T13:50:08.821 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104212.log 2023-10-16T13:50:08.821 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-mds.d.log: /var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104030.log 2023-10-16T13:50:08.822 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.104212.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.104212.log.gz 2023-10-16T13:50:08.822 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.104030.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.104030.log.gz 2023-10-16T13:50:08.822 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2023-10-16T13:50:08.823 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103985.log 2023-10-16T13:50:08.823 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-osd.4.log: /var/log/ceph/ceph-client.admin.103985.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.103985.log.gz 2023-10-16T13:50:08.823 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-10-16T13:50:08.823 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.110697.log 2023-10-16T13:50:08.824 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104265.log 2023-10-16T13:50:08.824 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.1.110697.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.110697.log.gz 2023-10-16T13:50:08.824 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-mgr.x.log: /var/log/ceph/ceph-client.admin.104265.log: 93.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-10-16T13:50:08.824 INFO:teuthology.orchestra.run.smithi049.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.104265.log.gz 2023-10-16T13:50:08.825 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2023-10-16T13:50:08.825 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104167.log 2023-10-16T13:50:08.826 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-mon.c.log: /var/log/ceph/ceph-client.admin.104167.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.104167.log.gz 2023-10-16T13:50:08.827 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.3.111308.log 2023-10-16T13:50:08.827 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2023-10-16T13:50:08.827 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.3.111308.log: 0.0% -- replaced with /var/log/ceph/ceph-client.3.111308.log.gz 2023-10-16T13:50:08.832 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-osd.7.log: 94.1% -- replaced with /var/log/ceph/ceph.log.gz 2023-10-16T13:50:08.832 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2023-10-16T13:50:08.832 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.3.111304.log 2023-10-16T13:50:08.857 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.3.111304.log: 0.0% -- replaced with /var/log/ceph/ceph-client.3.111304.log.gz 2023-10-16T13:50:08.858 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2023-10-16T13:50:08.874 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104077.log 2023-10-16T13:50:08.894 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.104077.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.104077.log.gz 2023-10-16T13:50:08.907 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.122066.log: /var/log/ceph/ceph-client.admin.183740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144393.log 2023-10-16T13:50:08.908 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.122066.log.gz 2023-10-16T13:50:08.908 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.172284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114103.log 2023-10-16T13:50:08.908 INFO:teuthology.orchestra.run.smithi016.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.183740.log.gz 2023-10-16T13:50:08.909 INFO:teuthology.orchestra.run.smithi016.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.172284.log.gz 2023-10-16T13:50:08.909 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.144393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156079.log 2023-10-16T13:50:08.909 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.114103.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.144393.log.gz 2023-10-16T13:50:08.910 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.184572.log 2023-10-16T13:50:08.910 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.156079.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.114103.log.gz 2023-10-16T13:50:08.910 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134583.log 2023-10-16T13:50:08.911 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.184572.log: 88.7%gzip -5 -- replaced with /var/log/ceph/ceph-client.admin.156079.log.gz--verbose 2023-10-16T13:50:08.911 INFO:teuthology.orchestra.run.smithi016.stderr:-- /var/log/ceph/ceph-client.admin.125232.log 2023-10-16T13:50:08.911 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.134583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138185.log 2023-10-16T13:50:08.912 INFO:teuthology.orchestra.run.smithi016.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.184572.log.gz 2023-10-16T13:50:08.912 INFO:teuthology.orchestra.run.smithi016.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.134583.log.gz 2023-10-16T13:50:08.912 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.125232.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118500.log 2023-10-16T13:50:08.913 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.138185.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.187975.log 2023-10-16T13:50:08.913 INFO:teuthology.orchestra.run.smithi016.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.125232.log.gz 2023-10-16T13:50:08.913 INFO:teuthology.orchestra.run.smithi016.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.138185.log.gz 2023-10-16T13:50:08.913 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133998.log 2023-10-16T13:50:08.914 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.118500.log: /var/log/ceph/ceph-client.admin.187975.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146571.log 2023-10-16T13:50:08.914 INFO:teuthology.orchestra.run.smithi016.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.118500.log.gz 2023-10-16T13:50:08.914 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.133998.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.187975.log.gz 2023-10-16T13:50:08.915 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.172400.log 2023-10-16T13:50:08.915 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.133998.log.gz 2023-10-16T13:50:08.915 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164288.log 2023-10-16T13:50:08.915 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.146571.log: /var/log/ceph/ceph-client.admin.172400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134296.log 2023-10-16T13:50:08.916 INFO:teuthology.orchestra.run.smithi016.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.146571.log.gz 2023-10-16T13:50:08.916 INFO:teuthology.orchestra.run.smithi016.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.172400.log.gz 2023-10-16T13:50:08.916 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.190042.log 2023-10-16T13:50:08.916 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.164288.log: /var/log/ceph/ceph-client.admin.134296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145261.log 2023-10-16T13:50:08.917 INFO:teuthology.orchestra.run.smithi016.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.164288.log.gz 2023-10-16T13:50:08.917 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.190042.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.134296.log.gz 2023-10-16T13:50:08.917 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.176268.log 2023-10-16T13:50:08.918 INFO:teuthology.orchestra.run.smithi016.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.190042.log.gz 2023-10-16T13:50:08.918 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128178.log 2023-10-16T13:50:08.918 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.145261.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.176268.log: /var/log/ceph/ceph-client.admin.185689.log 2023-10-16T13:50:08.919 INFO:teuthology.orchestra.run.smithi016.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.145261.log.gz 2023-10-16T13:50:08.919 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.128178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.179806.log 2023-10-16T13:50:08.919 INFO:teuthology.orchestra.run.smithi016.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.176268.log.gz 2023-10-16T13:50:08.919 INFO:teuthology.orchestra.run.smithi016.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.128178.log.gz 2023-10-16T13:50:08.920 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.185689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167885.log 2023-10-16T13:50:08.920 INFO:teuthology.orchestra.run.smithi016.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.185689.log.gz 2023-10-16T13:50:08.920 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133481.log 2023-10-16T13:50:08.921 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.179806.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136448.log 2023-10-16T13:50:08.921 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.167885.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.179806.log.gz 2023-10-16T13:50:08.921 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.133481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170807.log 2023-10-16T13:50:08.922 INFO:teuthology.orchestra.run.smithi016.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.167885.log.gz 2023-10-16T13:50:08.922 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.133481.log.gz 2023-10-16T13:50:08.922 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.136448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147426.log 2023-10-16T13:50:08.922 INFO:teuthology.orchestra.run.smithi016.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.136448.log.gz 2023-10-16T13:50:08.923 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.170807.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116576.log 2023-10-16T13:50:08.923 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.147426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170685.log 2023-10-16T13:50:08.923 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.170807.log.gz 2023-10-16T13:50:08.924 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3%/var/log/ceph/ceph-client.admin.116576.log: -- replaced with /var/log/ceph/ceph-client.admin.147426.log.gz 2023-10-16T13:50:08.924 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.188301.log 2023-10-16T13:50:08.924 INFO:teuthology.orchestra.run.smithi016.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.116576.log.gz 2023-10-16T13:50:08.925 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133531.log 2023-10-16T13:50:08.925 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.170685.log: /var/log/ceph/ceph-client.admin.188301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141066.log 2023-10-16T13:50:08.925 INFO:teuthology.orchestra.run.smithi016.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.170685.log.gz 2023-10-16T13:50:08.925 INFO:teuthology.orchestra.run.smithi016.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.188301.log.gz 2023-10-16T13:50:08.926 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.133531.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118776.log 2023-10-16T13:50:08.926 INFO:teuthology.orchestra.run.smithi016.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.133531.log.gz 2023-10-16T13:50:08.926 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.141066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141413.log 2023-10-16T13:50:08.926 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.118776.log: gzip -5 --verbose 88.7%-- /var/log/ceph/ceph-client.admin.154725.log -- replaced with /var/log/ceph/ceph-client.admin.141066.log.gz 2023-10-16T13:50:08.927 INFO:teuthology.orchestra.run.smithi016.stderr: 2023-10-16T13:50:08.927 INFO:teuthology.orchestra.run.smithi016.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.118776.log.gz 2023-10-16T13:50:08.927 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.141413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121704.log 2023-10-16T13:50:08.928 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.154725.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.141413.log.gz 2023-10-16T13:50:08.928 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127708.log 2023-10-16T13:50:08.928 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.121704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169182.log 2023-10-16T13:50:08.928 INFO:teuthology.orchestra.run.smithi016.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.154725.log.gz 2023-10-16T13:50:08.929 INFO:teuthology.orchestra.run.smithi016.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.121704.log.gz 2023-10-16T13:50:08.929 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.127708.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155199.log 2023-10-16T13:50:08.929 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.127708.log.gz 2023-10-16T13:50:08.930 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.169182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117485.log 2023-10-16T13:50:08.930 INFO:teuthology.orchestra.run.smithi016.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.169182.log.gz 2023-10-16T13:50:08.930 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.155199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.192997.log 2023-10-16T13:50:08.931 INFO:teuthology.orchestra.run.smithi016.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.155199.log.gz 2023-10-16T13:50:08.931 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.117485.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.180372.log 2023-10-16T13:50:08.931 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.117485.log.gz 2023-10-16T13:50:08.932 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.192997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146846.log 2023-10-16T13:50:08.932 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.180372.log: 88.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.192997.log.gz-5 2023-10-16T13:50:08.932 INFO:teuthology.orchestra.run.smithi016.stderr:--verbose -- /var/log/ceph/ceph-client.admin.167481.log 2023-10-16T13:50:08.933 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.180372.log.gz 2023-10-16T13:50:08.933 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.146846.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.181278.log 2023-10-16T13:50:08.933 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.146846.log.gz 2023-10-16T13:50:08.933 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.167481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.168557.log 2023-10-16T13:50:08.934 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.181278.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.167481.log.gz 2023-10-16T13:50:08.934 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148326.log 2023-10-16T13:50:08.934 INFO:teuthology.orchestra.run.smithi016.stderr: 88.0%/var/log/ceph/ceph-client.admin.168557.log: -- replaced with /var/log/ceph/ceph-client.admin.181278.log.gz 2023-10-16T13:50:08.935 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121838.log 2023-10-16T13:50:08.935 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.168557.log.gz 2023-10-16T13:50:08.935 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.148326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.180270.log 2023-10-16T13:50:08.935 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.121838.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.148326.log.gzgzip 2023-10-16T13:50:08.936 INFO:teuthology.orchestra.run.smithi016.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.137660.log 2023-10-16T13:50:08.936 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.121838.log.gz/var/log/ceph/ceph-client.admin.180270.log: 2023-10-16T13:50:08.936 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166404.log 2023-10-16T13:50:08.937 INFO:teuthology.orchestra.run.smithi016.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.180270.log.gz 2023-10-16T13:50:08.937 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.137660.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150879.log 2023-10-16T13:50:08.938 INFO:teuthology.orchestra.run.smithi016.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.137660.log.gz 2023-10-16T13:50:08.938 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.166404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149035.log 2023-10-16T13:50:08.939 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.150879.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.166404.log.gz 2023-10-16T13:50:08.939 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111806.log 2023-10-16T13:50:08.939 INFO:teuthology.orchestra.run.smithi016.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.150879.log.gz 2023-10-16T13:50:08.939 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.149035.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118270.log 2023-10-16T13:50:08.940 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.111806.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.149035.log.gz 2023-10-16T13:50:08.940 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128425.log 2023-10-16T13:50:08.940 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.118270.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.111806.log.gz 2023-10-16T13:50:08.941 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.195618.log 2023-10-16T13:50:08.941 INFO:teuthology.orchestra.run.smithi016.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.118270.log.gz 2023-10-16T13:50:08.941 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.128425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135877.log 2023-10-16T13:50:08.942 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.195618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128687.log 2023-10-16T13:50:08.942 INFO:teuthology.orchestra.run.smithi016.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.128425.log.gz 2023-10-16T13:50:08.942 INFO:teuthology.orchestra.run.smithi016.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.195618.log.gz 2023-10-16T13:50:08.942 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.135877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171914.log 2023-10-16T13:50:08.943 INFO:teuthology.orchestra.run.smithi016.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.135877.log.gz 2023-10-16T13:50:08.943 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.128687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167646.log 2023-10-16T13:50:08.943 INFO:teuthology.orchestra.run.smithi016.stderr: 89.3%/var/log/ceph/ceph-client.admin.171914.log: -- replaced with /var/log/ceph/ceph-client.admin.128687.log.gz 2023-10-16T13:50:08.943 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166830.log 2023-10-16T13:50:08.944 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.167646.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.171914.log.gz 2023-10-16T13:50:08.944 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142752.log 2023-10-16T13:50:08.944 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.166830.log: gzip 88.7%-5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.167646.log.gz-- 2023-10-16T13:50:08.945 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.122113.log 2023-10-16T13:50:08.945 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.142752.log: 87.8%gzip -5 -- replaced with /var/log/ceph/ceph-client.admin.166830.log.gz--verbose 2023-10-16T13:50:08.945 INFO:teuthology.orchestra.run.smithi016.stderr:-- /var/log/ceph/ceph-client.admin.140126.log 2023-10-16T13:50:08.945 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.122113.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.142752.log.gz 2023-10-16T13:50:08.946 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143608.log 2023-10-16T13:50:08.946 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.140126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.190805.log 2023-10-16T13:50:08.946 INFO:teuthology.orchestra.run.smithi016.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.122113.log.gz 2023-10-16T13:50:08.947 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.143608.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.140126.log.gz 2023-10-16T13:50:08.947 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161407.log 2023-10-16T13:50:08.947 INFO:teuthology.orchestra.run.smithi016.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.143608.log.gz 2023-10-16T13:50:08.948 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.190805.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.143331.log 2023-10-16T13:50:08.948 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.161407.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.191932.log 2023-10-16T13:50:08.948 INFO:teuthology.orchestra.run.smithi016.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.190805.log.gz 2023-10-16T13:50:08.949 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.0.143331.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.161407.log.gz 2023-10-16T13:50:08.949 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119211.log 2023-10-16T13:50:08.949 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.143331.log.gz 2023-10-16T13:50:08.950 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.184480.log 2023-10-16T13:50:08.950 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.191932.log: /var/log/ceph/ceph-client.admin.119211.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128549.log 2023-10-16T13:50:08.950 INFO:teuthology.orchestra.run.smithi016.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.191932.log.gz 2023-10-16T13:50:08.951 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.184480.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.119211.log.gz 2023-10-16T13:50:08.951 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152351.log 2023-10-16T13:50:08.951 INFO:teuthology.orchestra.run.smithi016.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.184480.log.gz 2023-10-16T13:50:08.951 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.128549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150486.log 2023-10-16T13:50:08.952 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.152351.log: gzip 88.1%-5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.128549.log.gz-- 2023-10-16T13:50:08.952 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.195250.log 2023-10-16T13:50:08.952 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.150486.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.152351.log.gz 2023-10-16T13:50:08.953 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117998.log 2023-10-16T13:50:08.953 INFO:teuthology.orchestra.run.smithi016.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.150486.log.gz 2023-10-16T13:50:08.953 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.195250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110063.log 2023-10-16T13:50:08.953 INFO:teuthology.orchestra.run.smithi016.stderr: 87.6%/var/log/ceph/ceph-client.admin.117998.log: -- replaced with /var/log/ceph/ceph-client.admin.195250.log.gz 2023-10-16T13:50:08.954 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.172192.log 2023-10-16T13:50:08.954 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.110063.log: 89.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.117998.log.gz 2023-10-16T13:50:08.955 INFO:teuthology.orchestra.run.smithi016.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.109506.log 2023-10-16T13:50:08.955 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.172192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128779.log 2023-10-16T13:50:08.955 INFO:teuthology.orchestra.run.smithi016.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.110063.log.gz 2023-10-16T13:50:08.956 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.109506.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.172192.log.gz 2023-10-16T13:50:08.956 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111987.log 2023-10-16T13:50:08.956 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.109506.log.gz 2023-10-16T13:50:08.956 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.128779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146776.log 2023-10-16T13:50:08.957 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.128779.log.gz 2023-10-16T13:50:08.957 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.111987.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-10-16T13:50:08.957 INFO:teuthology.orchestra.run.smithi016.stderr: 88.7%/var/log/ceph/ceph-client.admin.146776.log: -- replaced with /var/log/ceph/ceph-client.admin.111987.log.gz 2023-10-16T13:50:08.958 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124194.log 2023-10-16T13:50:08.958 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128026.log 2023-10-16T13:50:08.958 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.124194.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.146776.log.gz 2023-10-16T13:50:08.959 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149788.log 2023-10-16T13:50:08.959 INFO:teuthology.orchestra.run.smithi016.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.124194.log.gz 2023-10-16T13:50:08.959 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.128026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144441.log 2023-10-16T13:50:08.959 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.149788.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.128026.log.gz 2023-10-16T13:50:08.960 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133246.log 2023-10-16T13:50:08.960 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.149788.log.gz 2023-10-16T13:50:08.960 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.144441.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114241.log 2023-10-16T13:50:08.961 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.133246.log: gzip 88.4%-5 -- replaced with /var/log/ceph/ceph-client.admin.144441.log.gz--verbose 2023-10-16T13:50:08.961 INFO:teuthology.orchestra.run.smithi016.stderr:-- /var/log/ceph/ceph-client.admin.145783.log 2023-10-16T13:50:08.961 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.114241.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.133246.log.gz 2023-10-16T13:50:08.961 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120955.log 2023-10-16T13:50:08.962 INFO:teuthology.orchestra.run.smithi016.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.114241.log.gz 2023-10-16T13:50:08.962 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.145783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117082.log 2023-10-16T13:50:08.962 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.120955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162968.log 2023-10-16T13:50:08.963 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.145783.log.gz 2023-10-16T13:50:08.963 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.117082.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.120955.log.gz 2023-10-16T13:50:08.963 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123996.log 2023-10-16T13:50:08.964 INFO:teuthology.orchestra.run.smithi016.stderr: 88.7%/var/log/ceph/ceph-client.admin.162968.log: -- replaced with /var/log/ceph/ceph-client.admin.117082.log.gz 2023-10-16T13:50:08.964 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161356.log 2023-10-16T13:50:08.964 INFO:teuthology.orchestra.run.smithi016.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.162968.log.gz 2023-10-16T13:50:08.965 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.123996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137980.log 2023-10-16T13:50:08.965 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.161356.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.191092.log 2023-10-16T13:50:08.965 INFO:teuthology.orchestra.run.smithi016.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.123996.log.gz 2023-10-16T13:50:08.965 INFO:teuthology.orchestra.run.smithi016.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.161356.log.gz 2023-10-16T13:50:08.966 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.137980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.178298.log 2023-10-16T13:50:08.966 INFO:teuthology.orchestra.run.smithi016.stderr: 89.3%/var/log/ceph/ceph-client.admin.191092.log: -- replaced with /var/log/ceph/ceph-client.admin.137980.log.gz 2023-10-16T13:50:08.966 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169919.log 2023-10-16T13:50:08.967 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.178298.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.191092.log.gz 2023-10-16T13:50:08.967 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.186327.log 2023-10-16T13:50:08.967 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.178298.log.gz 2023-10-16T13:50:08.968 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.169919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128127.log 2023-10-16T13:50:08.968 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.186327.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.169919.log.gz 2023-10-16T13:50:08.968 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.192946.log 2023-10-16T13:50:08.968 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.186327.log.gz 2023-10-16T13:50:08.969 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.128127.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162550.log 2023-10-16T13:50:08.969 INFO:teuthology.orchestra.run.smithi016.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.128127.log.gz 2023-10-16T13:50:08.969 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.192946.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118868.log 2023-10-16T13:50:08.970 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.162550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123462.log 2023-10-16T13:50:08.970 INFO:teuthology.orchestra.run.smithi016.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.192946.log.gz 2023-10-16T13:50:08.970 INFO:teuthology.orchestra.run.smithi016.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.162550.log.gz 2023-10-16T13:50:08.971 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.118868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170238.log 2023-10-16T13:50:08.971 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.118868.log.gz 2023-10-16T13:50:08.971 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.123462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146385.log 2023-10-16T13:50:08.972 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.170238.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.123462.log.gz 2023-10-16T13:50:08.972 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.195480.log 2023-10-16T13:50:08.972 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3%/var/log/ceph/ceph-client.admin.146385.log: -- replaced with /var/log/ceph/ceph-client.admin.170238.log.gz 2023-10-16T13:50:08.973 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117711.log 2023-10-16T13:50:08.973 INFO:teuthology.orchestra.run.smithi016.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.146385.log.gz 2023-10-16T13:50:08.973 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.195480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112355.log 2023-10-16T13:50:08.974 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.117711.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.195480.log.gz 2023-10-16T13:50:08.974 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165371.log 2023-10-16T13:50:08.974 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.112355.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.117711.log.gz 2023-10-16T13:50:08.975 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.186430.log 2023-10-16T13:50:08.975 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.112355.log.gz 2023-10-16T13:50:08.975 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.165371.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164555.log 2023-10-16T13:50:08.976 INFO:teuthology.orchestra.run.smithi016.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.165371.log.gz 2023-10-16T13:50:08.977 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.186430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119165.log 2023-10-16T13:50:08.977 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.164555.log: 87.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.186430.log.gz-5 2023-10-16T13:50:08.978 INFO:teuthology.orchestra.run.smithi016.stderr:--verbose -- /var/log/ceph/ceph-client.admin.139404.log 2023-10-16T13:50:08.978 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.119165.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.164555.log.gz 2023-10-16T13:50:08.979 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121771.log 2023-10-16T13:50:08.979 INFO:teuthology.orchestra.run.smithi016.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.119165.log.gz 2023-10-16T13:50:08.980 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.139404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104128.log 2023-10-16T13:50:08.980 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.121771.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.139404.log.gz 2023-10-16T13:50:08.981 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139266.log 2023-10-16T13:50:08.981 INFO:teuthology.orchestra.run.smithi016.stderr: 88.1%/var/log/ceph/ceph-client.admin.104128.log: -- replaced with /var/log/ceph/ceph-client.admin.121771.log.gz 2023-10-16T13:50:08.981 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146479.log 2023-10-16T13:50:08.982 INFO:teuthology.orchestra.run.smithi016.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.104128.log.gz 2023-10-16T13:50:08.982 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.139266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117174.log 2023-10-16T13:50:08.983 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.146479.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.139266.log.gz 2023-10-16T13:50:08.983 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.139803.log 2023-10-16T13:50:08.984 INFO:teuthology.orchestra.run.smithi016.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.146479.log.gz 2023-10-16T13:50:08.984 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.117174.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163014.log 2023-10-16T13:50:08.985 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.117174.log.gz 2023-10-16T13:50:08.985 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.0.139803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.183281.log 2023-10-16T13:50:08.985 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.139803.log.gz 2023-10-16T13:50:08.985 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.163014.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111760.log 2023-10-16T13:50:08.986 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.163014.log.gz 2023-10-16T13:50:08.986 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.183281.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.187231.log 2023-10-16T13:50:08.986 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.111760.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.183068.log 2023-10-16T13:50:08.987 INFO:teuthology.orchestra.run.smithi016.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.183281.log.gz 2023-10-16T13:50:08.987 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.187231.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.111760.log.gz 2023-10-16T13:50:08.987 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139126.log 2023-10-16T13:50:08.988 INFO:teuthology.orchestra.run.smithi016.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.187231.log.gz 2023-10-16T13:50:08.988 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.183068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.184204.log 2023-10-16T13:50:08.989 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.139126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152284.log 2023-10-16T13:50:08.989 INFO:teuthology.orchestra.run.smithi016.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.183068.log.gz 2023-10-16T13:50:08.990 INFO:teuthology.orchestra.run.smithi016.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.139126.log.gz 2023-10-16T13:50:08.990 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.184204.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113349.log 2023-10-16T13:50:08.991 INFO:teuthology.orchestra.run.smithi016.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.184204.log.gz 2023-10-16T13:50:08.991 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.152284.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-10-16T13:50:08.992 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.113349.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141581.log 87.8% 2023-10-16T13:50:08.992 INFO:teuthology.orchestra.run.smithi016.stderr: -- replaced with /var/log/ceph/ceph-client.admin.152284.log.gz 2023-10-16T13:50:08.993 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.113349.log.gz 2023-10-16T13:50:08.993 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113551.log 2023-10-16T13:50:08.994 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.141581.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.178950.log 2023-10-16T13:50:08.994 INFO:teuthology.orchestra.run.smithi016.stderr: 87.8%/var/log/ceph/ceph-client.admin.113551.log: -- replaced with /var/log/ceph/ceph-client.admin.141581.log.gz 2023-10-16T13:50:08.995 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.139799.log 2023-10-16T13:50:08.995 INFO:teuthology.orchestra.run.smithi016.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.113551.log.gz 2023-10-16T13:50:08.996 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148932.log 2023-10-16T13:50:08.996 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.178950.log: /var/log/ceph/ceph-client.0.139799.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120450.log 2023-10-16T13:50:08.997 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.139799.log.gz 2023-10-16T13:50:08.997 INFO:teuthology.orchestra.run.smithi016.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.178950.log.gz 2023-10-16T13:50:08.997 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.148932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143028.log 2023-10-16T13:50:08.998 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.120450.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.148932.log.gz 2023-10-16T13:50:08.998 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127875.log 2023-10-16T13:50:08.998 INFO:teuthology.orchestra.run.smithi016.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.120450.log.gz 2023-10-16T13:50:08.999 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.177346.log 2023-10-16T13:50:08.999 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.143028.log: /var/log/ceph/ceph-client.admin.127875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.192826.log 2023-10-16T13:50:09.000 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.177346.log: gzip -5 --verbose -- 88.4%/var/log/ceph/ceph-client.admin.187369.log 2023-10-16T13:50:09.000 INFO:teuthology.orchestra.run.smithi016.stderr: -- replaced with /var/log/ceph/ceph-client.admin.127875.log.gz 2023-10-16T13:50:09.000 INFO:teuthology.orchestra.run.smithi016.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.143028.log.gz 2023-10-16T13:50:09.001 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.192826.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.177346.log.gz 2023-10-16T13:50:09.001 INFO:teuthology.orchestra.run.smithi016.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.192826.log.gz 2023-10-16T13:50:09.001 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.194469.log 2023-10-16T13:50:09.002 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.187369.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.181048.log 2023-10-16T13:50:09.002 INFO:teuthology.orchestra.run.smithi016.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.187369.log.gz 2023-10-16T13:50:09.003 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.194469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.194219.log 2023-10-16T13:50:09.003 INFO:teuthology.orchestra.run.smithi016.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.194469.log.gz 2023-10-16T13:50:09.003 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129757.log 2023-10-16T13:50:09.004 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.181048.log: /var/log/ceph/ceph-client.admin.194219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140627.log 2023-10-16T13:50:09.004 INFO:teuthology.orchestra.run.smithi016.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.181048.log.gz 2023-10-16T13:50:09.004 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.129757.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.194219.log.gz 2023-10-16T13:50:09.005 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129665.log 2023-10-16T13:50:09.005 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.179461.log 2023-10-16T13:50:09.005 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.140627.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.129757.log.gz 2023-10-16T13:50:09.005 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.129665.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140342.log 2023-10-16T13:50:09.006 INFO:teuthology.orchestra.run.smithi016.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.140627.log.gz 2023-10-16T13:50:09.006 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.179461.log: 88.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.129665.log.gz-5 2023-10-16T13:50:09.006 INFO:teuthology.orchestra.run.smithi016.stderr:--verbose -- /var/log/ceph/ceph-client.admin.195572.log 2023-10-16T13:50:09.007 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.140342.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162830.log 2023-10-16T13:50:09.007 INFO:teuthology.orchestra.run.smithi016.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.179461.log.gz 2023-10-16T13:50:09.007 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.195572.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.172910.log 2023-10-16T13:50:09.008 INFO:teuthology.orchestra.run.smithi016.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.140342.log.gz 2023-10-16T13:50:09.008 INFO:teuthology.orchestra.run.smithi016.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.195572.log.gz 2023-10-16T13:50:09.008 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.162830.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.187882.log 2023-10-16T13:50:09.009 INFO:teuthology.orchestra.run.smithi016.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.162830.log.gz 2023-10-16T13:50:09.009 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127375.log 2023-10-16T13:50:09.009 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.172910.log: /var/log/ceph/ceph-client.admin.187882.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160604.log 2023-10-16T13:50:09.009 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.172910.log.gz 2023-10-16T13:50:09.010 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.127375.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.187882.log.gz 2023-10-16T13:50:09.010 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171672.log 2023-10-16T13:50:09.011 INFO:teuthology.orchestra.run.smithi016.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.127375.log.gz 2023-10-16T13:50:09.011 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.160604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150440.log 2023-10-16T13:50:09.011 INFO:teuthology.orchestra.run.smithi016.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.160604.log.gz 2023-10-16T13:50:09.011 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170086.log 2023-10-16T13:50:09.012 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.171672.log: /var/log/ceph/ceph-client.admin.150440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121972.log 2023-10-16T13:50:09.012 INFO:teuthology.orchestra.run.smithi016.stderr: 89.0% 89.0% -- replaced with /var/log/ceph/ceph.log.gz -- replaced with /var/log/ceph/ceph-client.admin.171672.log.gz 2023-10-16T13:50:09.012 INFO:teuthology.orchestra.run.smithi016.stderr: 2023-10-16T13:50:09.013 INFO:teuthology.orchestra.run.smithi016.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.150440.log.gz 2023-10-16T13:50:09.013 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.170086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.188439.log 2023-10-16T13:50:09.013 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.121972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145307.log 2023-10-16T13:50:09.014 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.170086.log.gz 2023-10-16T13:50:09.014 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.188439.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.121972.log.gz 2023-10-16T13:50:09.014 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.175888.log 2023-10-16T13:50:09.015 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.145307.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.188439.log.gz 2023-10-16T13:50:09.015 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122395.log 2023-10-16T13:50:09.016 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.175888.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.145307.log.gz 2023-10-16T13:50:09.016 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138323.log 2023-10-16T13:50:09.016 INFO:teuthology.orchestra.run.smithi016.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.175888.log.gz 2023-10-16T13:50:09.017 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.122395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154967.log 2023-10-16T13:50:09.017 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.138323.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.122395.log.gz 2023-10-16T13:50:09.017 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130492.log 2023-10-16T13:50:09.017 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.154967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171960.log 2023-10-16T13:50:09.018 INFO:teuthology.orchestra.run.smithi016.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.138323.log.gz 2023-10-16T13:50:09.018 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.154967.log.gz 2023-10-16T13:50:09.018 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.130492.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119893.log 2023-10-16T13:50:09.019 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.171960.log: 88.0%gzip -5 -- replaced with /var/log/ceph/ceph-client.admin.130492.log.gz--verbose 2023-10-16T13:50:09.019 INFO:teuthology.orchestra.run.smithi016.stderr:-- /var/log/ceph/ceph-client.admin.182142.log 2023-10-16T13:50:09.019 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.119893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138277.log 2023-10-16T13:50:09.020 INFO:teuthology.orchestra.run.smithi016.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.171960.log.gz 2023-10-16T13:50:09.020 INFO:teuthology.orchestra.run.smithi016.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.119893.log.gz 2023-10-16T13:50:09.020 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.182142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113827.log 2023-10-16T13:50:09.020 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.138277.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120358.log 2023-10-16T13:50:09.021 INFO:teuthology.orchestra.run.smithi016.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.182142.log.gz 2023-10-16T13:50:09.021 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.113827.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.138277.log.gz 2023-10-16T13:50:09.021 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152005.log 2023-10-16T13:50:09.022 INFO:teuthology.orchestra.run.smithi016.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.113827.log.gz 2023-10-16T13:50:09.022 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.120358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.187836.log 2023-10-16T13:50:09.022 INFO:teuthology.orchestra.run.smithi016.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.120358.log.gz 2023-10-16T13:50:09.023 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145353.log 2023-10-16T13:50:09.023 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.152005.log: /var/log/ceph/ceph-client.admin.187836.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.194777.log 2023-10-16T13:50:09.023 INFO:teuthology.orchestra.run.smithi016.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.152005.log.gz 2023-10-16T13:50:09.024 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.145353.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.187836.log.gz 2023-10-16T13:50:09.024 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.192229.log 2023-10-16T13:50:09.024 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162127.log 2023-10-16T13:50:09.024 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.194777.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.145353.log.gz 2023-10-16T13:50:09.025 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.192229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162035.log 2023-10-16T13:50:09.025 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.194777.log.gz 2023-10-16T13:50:09.025 INFO:teuthology.orchestra.run.smithi016.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.192229.log.gz 2023-10-16T13:50:09.025 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.162127.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.182598.log 2023-10-16T13:50:09.026 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.162035.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141958.log 2023-10-16T13:50:09.026 INFO:teuthology.orchestra.run.smithi016.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.162127.log.gz 2023-10-16T13:50:09.026 INFO:teuthology.orchestra.run.smithi016.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.162035.log.gz 2023-10-16T13:50:09.027 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.182598.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.188163.log 2023-10-16T13:50:09.027 INFO:teuthology.orchestra.run.smithi016.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.182598.log.gz 2023-10-16T13:50:09.027 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152947.log 2023-10-16T13:50:09.027 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.141958.log: /var/log/ceph/ceph-client.admin.188163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144343.log 2023-10-16T13:50:09.028 INFO:teuthology.orchestra.run.smithi016.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.141958.log.gz 2023-10-16T13:50:09.028 INFO:teuthology.orchestra.run.smithi016.stderr: 89.5%/var/log/ceph/ceph-client.admin.152947.log: -- replaced with /var/log/ceph/ceph-client.admin.188163.log.gz 2023-10-16T13:50:09.028 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151017.log 2023-10-16T13:50:09.029 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.152947.log.gz 2023-10-16T13:50:09.029 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165520.log 2023-10-16T13:50:09.029 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.144343.log: /var/log/ceph/ceph-client.admin.151017.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114011.log 2023-10-16T13:50:09.029 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.144343.log.gz 2023-10-16T13:50:09.030 INFO:teuthology.orchestra.run.smithi016.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.151017.log.gz 2023-10-16T13:50:09.030 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.165520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130033.log 2023-10-16T13:50:09.030 INFO:teuthology.orchestra.run.smithi016.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.165520.log.gz 2023-10-16T13:50:09.031 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121404.log 2023-10-16T13:50:09.031 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.114011.log: /var/log/ceph/ceph-client.admin.130033.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.179311.log 2023-10-16T13:50:09.031 INFO:teuthology.orchestra.run.smithi016.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.114011.log.gz 2023-10-16T13:50:09.032 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.121404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120082.log 2023-10-16T13:50:09.032 INFO:teuthology.orchestra.run.smithi016.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.121404.log.gz 2023-10-16T13:50:09.032 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.179311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171625.log 2023-10-16T13:50:09.032 INFO:teuthology.orchestra.run.smithi016.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.130033.log.gz 2023-10-16T13:50:09.033 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155429.log 2023-10-16T13:50:09.033 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.120082.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.179311.log.gz 2023-10-16T13:50:09.034 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.171625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.178668.log 2023-10-16T13:50:09.034 INFO:teuthology.orchestra.run.smithi016.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.120082.log.gz 2023-10-16T13:50:09.035 INFO:teuthology.orchestra.run.smithi016.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.171625.log.gz 2023-10-16T13:50:09.035 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.155429.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164971.log 2023-10-16T13:50:09.036 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142241.log 2023-10-16T13:50:09.036 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.178668.log: /var/log/ceph/ceph-client.admin.164971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.172100.log 2023-10-16T13:50:09.037 INFO:teuthology.orchestra.run.smithi016.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.178668.log.gz 2023-10-16T13:50:09.037 INFO:teuthology.orchestra.run.smithi016.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.155429.log.gz 2023-10-16T13:50:09.038 INFO:teuthology.orchestra.run.smithi016.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.164971.log.gz 2023-10-16T13:50:09.038 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.142241.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.189165.log 2023-10-16T13:50:09.039 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.172100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.185991.log 2023-10-16T13:50:09.040 INFO:teuthology.orchestra.run.smithi016.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.142241.log.gz 2023-10-16T13:50:09.040 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.189165.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.172100.log.gz 2023-10-16T13:50:09.041 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170288.log 2023-10-16T13:50:09.041 INFO:teuthology.orchestra.run.smithi016.stderr: 88.0%gzip -5 -- replaced with /var/log/ceph/ceph-client.admin.189165.log.gz--verbose 2023-10-16T13:50:09.042 INFO:teuthology.orchestra.run.smithi016.stderr:-- /var/log/ceph/ceph-client.admin.189090.log 2023-10-16T13:50:09.042 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.185991.log: /var/log/ceph/ceph-client.admin.170288.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155337.log 2023-10-16T13:50:09.043 INFO:teuthology.orchestra.run.smithi016.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.185991.log.gz 2023-10-16T13:50:09.043 INFO:teuthology.orchestra.run.smithi016.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.170288.log.gz 2023-10-16T13:50:09.044 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.189090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149742.log 2023-10-16T13:50:09.044 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.155337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112972.log 2023-10-16T13:50:09.045 INFO:teuthology.orchestra.run.smithi016.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.189090.log.gz 2023-10-16T13:50:09.046 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.149742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109829.log 2023-10-16T13:50:09.046 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.155337.log.gz 2023-10-16T13:50:09.047 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.149742.log.gz 2023-10-16T13:50:09.047 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.112972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.195342.log 2023-10-16T13:50:09.048 INFO:teuthology.orchestra.run.smithi016.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.112972.log.gz 2023-10-16T13:50:09.048 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.192183.log 2023-10-16T13:50:09.049 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.109829.log: /var/log/ceph/ceph-client.admin.195342.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-10-16T13:50:09.049 INFO:teuthology.orchestra.run.smithi016.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.109829.log.gz 2023-10-16T13:50:09.050 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.192183.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.195342.log.gz 2023-10-16T13:50:09.050 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119119.log 2023-10-16T13:50:09.051 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.183878.log 2023-10-16T13:50:09.051 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-osd.0.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.192183.log.gz 2023-10-16T13:50:09.052 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.119119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.189613.log 2023-10-16T13:50:09.052 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.183878.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.119119.log.gz 2023-10-16T13:50:09.053 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117619.log 2023-10-16T13:50:09.053 INFO:teuthology.orchestra.run.smithi016.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.183878.log.gz 2023-10-16T13:50:09.054 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.189613.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167038.log 2023-10-16T13:50:09.054 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138746.log 2023-10-16T13:50:09.054 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.117619.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.189613.log.gz 2023-10-16T13:50:09.055 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.167038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141535.log 2023-10-16T13:50:09.055 INFO:teuthology.orchestra.run.smithi016.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.117619.log.gz 2023-10-16T13:50:09.056 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.138746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170639.log 2023-10-16T13:50:09.056 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.167038.log.gz 2023-10-16T13:50:09.057 INFO:teuthology.orchestra.run.smithi016.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.138746.log.gz 2023-10-16T13:50:09.057 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.141535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112217.log 2023-10-16T13:50:09.058 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.170639.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120496.log 2023-10-16T13:50:09.058 INFO:teuthology.orchestra.run.smithi016.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.141535.log.gz 2023-10-16T13:50:09.059 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.112217.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.170639.log.gz 2023-10-16T13:50:09.059 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.182363.log 2023-10-16T13:50:09.060 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.112217.log.gz 2023-10-16T13:50:09.060 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.195007.log 2023-10-16T13:50:09.061 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.120496.log: /var/log/ceph/ceph-client.admin.182363.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152761.log 2023-10-16T13:50:09.061 INFO:teuthology.orchestra.run.smithi016.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.182363.log.gz 2023-10-16T13:50:09.062 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.195007.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.120496.log.gz 2023-10-16T13:50:09.062 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.152761.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.195007.log.gz 2023-10-16T13:50:09.063 INFO:teuthology.orchestra.run.smithi016.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.152761.log.gz 2023-10-16T13:50:09.063 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161207.log 2023-10-16T13:50:09.064 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.182792.log 2023-10-16T13:50:09.064 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.161207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162736.log 2023-10-16T13:50:09.065 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.182792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.184688.log 2023-10-16T13:50:09.065 INFO:teuthology.orchestra.run.smithi016.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.161207.log.gz 2023-10-16T13:50:09.066 INFO:teuthology.orchestra.run.smithi016.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.182792.log.gz 2023-10-16T13:50:09.066 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.162736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.179021.log 2023-10-16T13:50:09.067 INFO:teuthology.orchestra.run.smithi016.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.162736.log.gz 2023-10-16T13:50:09.067 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162328.log 2023-10-16T13:50:09.068 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.184688.log: /var/log/ceph/ceph-client.admin.179021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135831.log 2023-10-16T13:50:09.068 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.184688.log.gz 2023-10-16T13:50:09.069 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.162328.log: gzip -5 --verbose -- 88.1%/var/log/ceph/ceph-client.admin.161897.log 2023-10-16T13:50:09.069 INFO:teuthology.orchestra.run.smithi016.stderr: -- replaced with /var/log/ceph/ceph-client.admin.179021.log.gz 2023-10-16T13:50:09.069 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141727.log 2023-10-16T13:50:09.070 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.135831.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.162328.log.gz 2023-10-16T13:50:09.070 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.161897.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.135831.log.gz 2023-10-16T13:50:09.070 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127510.log 2023-10-16T13:50:09.071 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.141727.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.191000.log 2023-10-16T13:50:09.071 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.161897.log.gz 2023-10-16T13:50:09.071 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.127510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.185140.log 2023-10-16T13:50:09.072 INFO:teuthology.orchestra.run.smithi016.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.141727.log.gz 2023-10-16T13:50:09.072 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.191000.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.178102.log 2023-10-16T13:50:09.072 INFO:teuthology.orchestra.run.smithi016.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.127510.log.gz 2023-10-16T13:50:09.073 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.185140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.183327.log 2023-10-16T13:50:09.073 INFO:teuthology.orchestra.run.smithi016.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.191000.log.gz 2023-10-16T13:50:09.074 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.178102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.192091.log 2023-10-16T13:50:09.074 INFO:teuthology.orchestra.run.smithi016.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.185140.log.gz 2023-10-16T13:50:09.074 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.183327.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.178102.log.gz 2023-10-16T13:50:09.075 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.188585.log 2023-10-16T13:50:09.075 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169769.log 2023-10-16T13:50:09.075 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.192091.log: /var/log/ceph/ceph-client.admin.188585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130558.log 2023-10-16T13:50:09.076 INFO:teuthology.orchestra.run.smithi016.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.192091.log.gz 2023-10-16T13:50:09.076 INFO:teuthology.orchestra.run.smithi016.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.183327.log.gz 2023-10-16T13:50:09.076 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.169769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155842.log 2023-10-16T13:50:09.076 INFO:teuthology.orchestra.run.smithi016.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.188585.log.gz 2023-10-16T13:50:09.077 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142474.log 2023-10-16T13:50:09.077 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.130558.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.169769.log.gz 2023-10-16T13:50:09.077 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.155842.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165124.log 2023-10-16T13:50:09.077 INFO:teuthology.orchestra.run.smithi016.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.130558.log.gz 2023-10-16T13:50:09.078 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.142474.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.155842.log.gz 2023-10-16T13:50:09.078 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134721.log 2023-10-16T13:50:09.078 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.165124.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.142474.log.gz 2023-10-16T13:50:09.079 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148667.log 2023-10-16T13:50:09.079 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.134721.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.165124.log.gz 2023-10-16T13:50:09.079 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128917.log 2023-10-16T13:50:09.080 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.148667.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.134721.log.gz 2023-10-16T13:50:09.080 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123732.log 2023-10-16T13:50:09.080 INFO:teuthology.orchestra.run.smithi016.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.148667.log.gz/var/log/ceph/ceph-client.admin.128917.log: 2023-10-16T13:50:09.080 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165704.log 2023-10-16T13:50:09.081 INFO:teuthology.orchestra.run.smithi016.stderr: 87.5%/var/log/ceph/ceph-client.admin.123732.log: -- replaced with /var/log/ceph/ceph-client.admin.128917.log.gz 2023-10-16T13:50:09.081 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149696.log 2023-10-16T13:50:09.081 INFO:teuthology.orchestra.run.smithi016.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.123732.log.gz 2023-10-16T13:50:09.082 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.165704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129849.log 2023-10-16T13:50:09.082 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135399.log 2023-10-16T13:50:09.082 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.149696.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.165704.log.gz 2023-10-16T13:50:09.082 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.129849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163060.log 2023-10-16T13:50:09.083 INFO:teuthology.orchestra.run.smithi016.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.149696.log.gz 2023-10-16T13:50:09.083 INFO:teuthology.orchestra.run.smithi016.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.129849.log.gz 2023-10-16T13:50:09.083 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.135399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.172633.log 2023-10-16T13:50:09.083 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.163060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149284.log 2023-10-16T13:50:09.084 INFO:teuthology.orchestra.run.smithi016.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.135399.log.gz 2023-10-16T13:50:09.084 INFO:teuthology.orchestra.run.smithi016.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.163060.log.gz 2023-10-16T13:50:09.084 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.0.172633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167427.log 2023-10-16T13:50:09.085 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.172633.log.gz 2023-10-16T13:50:09.085 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.149284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.177108.log 2023-10-16T13:50:09.085 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.149284.log.gz 2023-10-16T13:50:09.085 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.167427.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154865.log 2023-10-16T13:50:09.086 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118408.log 2023-10-16T13:50:09.086 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.177108.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.167427.log.gz 2023-10-16T13:50:09.086 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.154865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141866.log 2023-10-16T13:50:09.086 INFO:teuthology.orchestra.run.smithi016.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.177108.log.gz 2023-10-16T13:50:09.087 INFO:teuthology.orchestra.run.smithi016.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.154865.log.gz 2023-10-16T13:50:09.087 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.118408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123416.log 2023-10-16T13:50:09.087 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.141866.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.181140.log 2023-10-16T13:50:09.087 INFO:teuthology.orchestra.run.smithi016.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.118408.log.gz 2023-10-16T13:50:09.088 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.123416.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.141866.log.gz 2023-10-16T13:50:09.088 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163886.log 2023-10-16T13:50:09.088 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.196032.log 2023-10-16T13:50:09.088 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.181140.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.123416.log.gz 2023-10-16T13:50:09.089 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.163886.log: gzip -5 --verbose -- 88.1%/var/log/ceph/ceph-client.admin.175824.log 2023-10-16T13:50:09.089 INFO:teuthology.orchestra.run.smithi016.stderr: -- replaced with /var/log/ceph/ceph-client.admin.181140.log.gz 2023-10-16T13:50:09.089 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.196032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.179554.log 2023-10-16T13:50:09.089 INFO:teuthology.orchestra.run.smithi016.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.163886.log.gz 2023-10-16T13:50:09.090 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.175824.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.196032.log.gz 2023-10-16T13:50:09.090 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110503.log 2023-10-16T13:50:09.090 INFO:teuthology.orchestra.run.smithi016.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.175824.log.gz 2023-10-16T13:50:09.091 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.176828.log 2023-10-16T13:50:09.091 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.179554.log: /var/log/ceph/ceph-client.admin.110503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121025.log 2023-10-16T13:50:09.091 INFO:teuthology.orchestra.run.smithi016.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.179554.log.gz 2023-10-16T13:50:09.092 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.176828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146051.log 2023-10-16T13:50:09.092 INFO:teuthology.orchestra.run.smithi016.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.110503.log.gz 2023-10-16T13:50:09.092 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.121025.log: gzip 87.9%-5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.176828.log.gz-- 2023-10-16T13:50:09.092 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-mds.a.log 2023-10-16T13:50:09.093 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.146051.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.121025.log.gz 2023-10-16T13:50:09.093 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-mds.a.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.146051.log.gz 2023-10-16T13:50:09.093 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.172844.log 2023-10-16T13:50:09.093 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166968.log 2023-10-16T13:50:09.094 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.172844.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.184808.log 2023-10-16T13:50:09.094 INFO:teuthology.orchestra.run.smithi016.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.172844.log.gz 2023-10-16T13:50:09.094 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.166968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154211.log 2023-10-16T13:50:09.094 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116530.log 2023-10-16T13:50:09.095 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.184808.log: /var/log/ceph/ceph-client.admin.154211.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.184808.log.gz 2023-10-16T13:50:09.095 INFO:teuthology.orchestra.run.smithi016.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.166968.log.gz 2023-10-16T13:50:09.095 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.116530.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.154211.log.gz 2023-10-16T13:50:09.096 INFO:teuthology.orchestra.run.smithi016.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.116530.log.gz 2023-10-16T13:50:09.096 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152576.log 2023-10-16T13:50:09.096 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.181711.log 2023-10-16T13:50:09.097 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.152576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153094.log 2023-10-16T13:50:09.097 INFO:teuthology.orchestra.run.smithi016.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.152576.log.gz 2023-10-16T13:50:09.097 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.181711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138614.log 2023-10-16T13:50:09.097 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142982.log 2023-10-16T13:50:09.098 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.153094.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.181711.log.gz 2023-10-16T13:50:09.098 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.138614.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143488.log 2023-10-16T13:50:09.098 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.153094.log.gz 2023-10-16T13:50:09.099 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.142982.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.138614.log.gz 2023-10-16T13:50:09.099 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.142982.log.gz 2023-10-16T13:50:09.099 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.143488.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.143488.log.gz 2023-10-16T13:50:09.099 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138700.log 2023-10-16T13:50:09.100 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122254.log 2023-10-16T13:50:09.100 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.138700.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.143335.log 2023-10-16T13:50:09.100 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.122254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.184158.log 2023-10-16T13:50:09.101 INFO:teuthology.orchestra.run.smithi016.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.138700.log.gz 2023-10-16T13:50:09.101 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.0.143335.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137093.log 2023-10-16T13:50:09.101 INFO:teuthology.orchestra.run.smithi016.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.122254.log.gz 2023-10-16T13:50:09.101 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.143335.log.gz 2023-10-16T13:50:09.102 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.184158.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140842.log 2023-10-16T13:50:09.102 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.137093.log: gzip 88.8%-5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.184158.log.gz-- 2023-10-16T13:50:09.102 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.118362.log 2023-10-16T13:50:09.103 INFO:teuthology.orchestra.run.smithi016.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.137093.log.gz 2023-10-16T13:50:09.103 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.140842.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141912.log 2023-10-16T13:50:09.103 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.118362.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.140842.log.gz 2023-10-16T13:50:09.103 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162922.log 2023-10-16T13:50:09.104 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.141912.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.118362.log.gz 2023-10-16T13:50:09.104 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.186373.log 2023-10-16T13:50:09.104 INFO:teuthology.orchestra.run.smithi016.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.141912.log.gz 2023-10-16T13:50:09.104 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.162922.log: /var/log/ceph/ceph-client.admin.186373.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.162922.log.gz 2023-10-16T13:50:09.105 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.186373.log.gz 2023-10-16T13:50:09.105 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111557.log 2023-10-16T13:50:09.105 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144243.log 2023-10-16T13:50:09.105 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.111557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.193751.log 2023-10-16T13:50:09.106 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.144243.log: gzip 89.5%-5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.111557.log.gz-- 2023-10-16T13:50:09.106 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.180322.log 2023-10-16T13:50:09.106 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.193751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152896.log 2023-10-16T13:50:09.107 INFO:teuthology.orchestra.run.smithi016.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.144243.log.gz 2023-10-16T13:50:09.107 INFO:teuthology.orchestra.run.smithi016.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.193751.log.gz 2023-10-16T13:50:09.107 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.180322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.191276.log 2023-10-16T13:50:09.108 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.152896.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.180322.log.gz 2023-10-16T13:50:09.108 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.191276.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.152896.log.gz 2023-10-16T13:50:09.108 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.191276.log.gz 2023-10-16T13:50:09.109 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135731.log 2023-10-16T13:50:09.109 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119847.log 2023-10-16T13:50:09.109 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.135731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.183832.log 2023-10-16T13:50:09.109 INFO:teuthology.orchestra.run.smithi016.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.135731.log.gz 2023-10-16T13:50:09.110 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.119847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.179855.log 2023-10-16T13:50:09.110 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.183832.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.119847.log.gz 2023-10-16T13:50:09.110 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.179855.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.183832.log.gz 2023-10-16T13:50:09.111 INFO:teuthology.orchestra.run.smithi016.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.179855.log.gz 2023-10-16T13:50:09.111 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151117.log 2023-10-16T13:50:09.111 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.182652.log 2023-10-16T13:50:09.112 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.151117.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.155736.log 2023-10-16T13:50:09.112 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.182652.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137922.log 2023-10-16T13:50:09.112 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.0.155736.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.151117.log.gz 2023-10-16T13:50:09.113 INFO:teuthology.orchestra.run.smithi016.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.182652.log.gz 2023-10-16T13:50:09.113 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.155736.log.gz 2023-10-16T13:50:09.113 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.137922.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.137922.log.gz 2023-10-16T13:50:09.113 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114287.log 2023-10-16T13:50:09.114 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163290.log 2023-10-16T13:50:09.114 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.114287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134049.log 2023-10-16T13:50:09.114 INFO:teuthology.orchestra.run.smithi016.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.114287.log.gz 2023-10-16T13:50:09.115 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.163290.log: /var/log/ceph/ceph-client.admin.134049.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.134049.log.gz 2023-10-16T13:50:09.115 INFO:teuthology.orchestra.run.smithi016.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.163290.log.gz 2023-10-16T13:50:09.115 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150603.log 2023-10-16T13:50:09.115 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.193851.log 2023-10-16T13:50:09.116 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.150603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146005.log 2023-10-16T13:50:09.116 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.193851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.194419.log 2023-10-16T13:50:09.116 INFO:teuthology.orchestra.run.smithi016.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.150603.log.gz 2023-10-16T13:50:09.117 INFO:teuthology.orchestra.run.smithi016.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.193851.log.gz 2023-10-16T13:50:09.117 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.146005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144294.log 2023-10-16T13:50:09.117 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170438.log 2023-10-16T13:50:09.117 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.194419.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.146005.log.gz 2023-10-16T13:50:09.118 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.144294.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.194419.log.gz 2023-10-16T13:50:09.118 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135257.log 2023-10-16T13:50:09.119 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.170438.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.144294.log.gz 2023-10-16T13:50:09.119 INFO:teuthology.orchestra.run.smithi016.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.170438.log.gz 2023-10-16T13:50:09.119 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.135257.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.135257.log.gz 2023-10-16T13:50:09.120 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.120799.log 2023-10-16T13:50:09.120 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150833.log 2023-10-16T13:50:09.120 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.0.120799.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.178010.log 2023-10-16T13:50:09.121 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.120799.log.gz 2023-10-16T13:50:09.121 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.150833.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111465.log 2023-10-16T13:50:09.121 INFO:teuthology.orchestra.run.smithi016.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.150833.log.gz 2023-10-16T13:50:09.122 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.178010.log: /var/log/ceph/ceph-client.admin.111465.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.178010.log.gz 2023-10-16T13:50:09.122 INFO:teuthology.orchestra.run.smithi016.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.111465.log.gz 2023-10-16T13:50:09.122 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.186168.log 2023-10-16T13:50:09.123 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165173.log 2023-10-16T13:50:09.123 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.186168.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.181757.log 2023-10-16T13:50:09.123 INFO:teuthology.orchestra.run.smithi016.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.186168.log.gz 2023-10-16T13:50:09.124 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.165173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116245.log 2023-10-16T13:50:09.124 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135210.log 2023-10-16T13:50:09.124 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.181757.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.165173.log.gz 2023-10-16T13:50:09.124 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.116245.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.181757.log.gz 2023-10-16T13:50:09.125 INFO:teuthology.orchestra.run.smithi016.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.116245.log.gz 2023-10-16T13:50:09.125 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154119.log 2023-10-16T13:50:09.125 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.135210.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123370.log 2023-10-16T13:50:09.126 INFO:teuthology.orchestra.run.smithi016.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.135210.log.gz 2023-10-16T13:50:09.126 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.154119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.176084.log 2023-10-16T13:50:09.126 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142936.log 2023-10-16T13:50:09.127 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.123370.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.154119.log.gz 2023-10-16T13:50:09.127 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.176084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.186260.log 2023-10-16T13:50:09.127 INFO:teuthology.orchestra.run.smithi016.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.123370.log.gz 2023-10-16T13:50:09.128 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.142936.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.176084.log.gz 2023-10-16T13:50:09.128 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.186260.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.142936.log.gz 2023-10-16T13:50:09.128 INFO:teuthology.orchestra.run.smithi016.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.186260.log.gz 2023-10-16T13:50:09.128 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2023-10-16T13:50:09.129 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.175778.log 2023-10-16T13:50:09.129 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159418.log 2023-10-16T13:50:09.129 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.175778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.188701.log 2023-10-16T13:50:09.129 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.159418.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.175778.log.gz 2023-10-16T13:50:09.130 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.188701.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.159418.log.gz 2023-10-16T13:50:09.130 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.188701.log.gz 2023-10-16T13:50:09.130 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.194712.log 2023-10-16T13:50:09.131 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.195526.log 2023-10-16T13:50:09.131 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.194712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144608.log 2023-10-16T13:50:09.131 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.195526.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.194712.log.gz 2023-10-16T13:50:09.132 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.144608.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.195526.log.gz 2023-10-16T13:50:09.132 INFO:teuthology.orchestra.run.smithi016.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.144608.log.gz 2023-10-16T13:50:09.132 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156318.log 2023-10-16T13:50:09.133 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140793.log 2023-10-16T13:50:09.133 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.156318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165566.log 2023-10-16T13:50:09.133 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.140793.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.156318.log.gz 2023-10-16T13:50:09.134 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.165566.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.140793.log.gz 2023-10-16T13:50:09.134 INFO:teuthology.orchestra.run.smithi016.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.165566.log.gz 2023-10-16T13:50:09.135 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.195940.log 2023-10-16T13:50:09.135 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138072.log 2023-10-16T13:50:09.135 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.195940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.180956.log 2023-10-16T13:50:09.135 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.138072.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.195940.log.gz 2023-10-16T13:50:09.136 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.180956.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.138072.log.gz 2023-10-16T13:50:09.136 INFO:teuthology.orchestra.run.smithi016.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.180956.log.gz 2023-10-16T13:50:09.136 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116061.log 2023-10-16T13:50:09.137 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139220.log 2023-10-16T13:50:09.137 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.116061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166552.log 2023-10-16T13:50:09.137 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.139220.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.116061.log.gz 2023-10-16T13:50:09.138 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.166552.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.139220.log.gz 2023-10-16T13:50:09.138 INFO:teuthology.orchestra.run.smithi016.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.166552.log.gz 2023-10-16T13:50:09.138 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.187323.log 2023-10-16T13:50:09.138 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.130336.log 2023-10-16T13:50:09.139 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.187323.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.176314.log 2023-10-16T13:50:09.139 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.0.130336.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.130336.log.gz 2023-10-16T13:50:09.139 INFO:teuthology.orchestra.run.smithi016.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.187323.log.gz 2023-10-16T13:50:09.140 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.176314.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.176314.log.gz 2023-10-16T13:50:09.140 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.179415.log 2023-10-16T13:50:09.140 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.185578.log 2023-10-16T13:50:09.140 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149604.log /var/log/ceph/ceph-client.admin.179415.log: 2023-10-16T13:50:09.141 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.185578.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.179415.log.gz 2023-10-16T13:50:09.141 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.149604.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.185578.log.gz 2023-10-16T13:50:09.141 INFO:teuthology.orchestra.run.smithi016.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.149604.log.gz 2023-10-16T13:50:09.142 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152531.log 2023-10-16T13:50:09.142 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170135.log 2023-10-16T13:50:09.142 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.152531.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.97577.log 2023-10-16T13:50:09.143 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.170135.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.152531.log.gz 2023-10-16T13:50:09.143 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph.tmp-client.admin.97577.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.170135.log.gz 2023-10-16T13:50:09.143 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.97577.log.gz 2023-10-16T13:50:09.143 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144559.log 2023-10-16T13:50:09.144 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.190261.log 2023-10-16T13:50:09.144 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.144559.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.181508.log 2023-10-16T13:50:09.144 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.190261.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.144559.log.gz 2023-10-16T13:50:09.145 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.181508.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.190261.log.gz 2023-10-16T13:50:09.145 INFO:teuthology.orchestra.run.smithi016.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.181508.log.gz 2023-10-16T13:50:09.145 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150971.log 2023-10-16T13:50:09.145 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145204.log 2023-10-16T13:50:09.146 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.150971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134342.log 2023-10-16T13:50:09.146 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.145204.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.150971.log.gz 2023-10-16T13:50:09.146 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.134342.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.145204.log.gz 2023-10-16T13:50:09.147 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.134342.log.gz 2023-10-16T13:50:09.147 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161830.log 2023-10-16T13:50:09.147 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151530.log 2023-10-16T13:50:09.147 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.161830.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162081.log 2023-10-16T13:50:09.148 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.151530.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.161830.log.gz 2023-10-16T13:50:09.148 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.162081.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.151530.log.gz 2023-10-16T13:50:09.148 INFO:teuthology.orchestra.run.smithi016.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.162081.log.gz 2023-10-16T13:50:09.149 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.177741.log 2023-10-16T13:50:09.149 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146730.log 2023-10-16T13:50:09.149 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.0.177741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.179905.log 2023-10-16T13:50:09.149 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.177741.log.gz 2023-10-16T13:50:09.150 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.146730.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144759.log 2023-10-16T13:50:09.150 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.179905.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.146730.log.gz 2023-10-16T13:50:09.150 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.144759.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.179905.log.gz 2023-10-16T13:50:09.151 INFO:teuthology.orchestra.run.smithi016.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.144759.log.gz 2023-10-16T13:50:09.151 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119617.log 2023-10-16T13:50:09.151 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116199.log 2023-10-16T13:50:09.151 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162418.log 2023-10-16T13:50:09.152 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.119617.log: /var/log/ceph/ceph-client.admin.116199.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.119617.log.gz 2023-10-16T13:50:09.152 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.162418.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.116199.log.gz 2023-10-16T13:50:09.152 INFO:teuthology.orchestra.run.smithi016.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.162418.log.gz 2023-10-16T13:50:09.153 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146191.log 2023-10-16T13:50:09.153 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.191184.log 2023-10-16T13:50:09.154 INFO:teuthology.orchestra.run.smithi049.stderr: 93.0% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-10-16T13:50:09.154 INFO:teuthology.orchestra.run.smithi049.stderr: 93.1% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2023-10-16T13:50:09.257 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.146191.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130948.log 2023-10-16T13:50:09.258 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.191184.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.146191.log.gz 2023-10-16T13:50:09.258 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.130948.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.191184.log.gz 2023-10-16T13:50:09.258 INFO:teuthology.orchestra.run.smithi016.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.130948.log.gz 2023-10-16T13:50:09.259 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144017.log 2023-10-16T13:50:09.259 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121616.log 2023-10-16T13:50:09.259 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.144017.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111327.log 2023-10-16T13:50:09.260 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.121616.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.144017.log.gz 2023-10-16T13:50:09.260 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.111327.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.121616.log.gz 2023-10-16T13:50:09.260 INFO:teuthology.orchestra.run.smithi016.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.111327.log.gz 2023-10-16T13:50:09.261 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112309.log 2023-10-16T13:50:09.261 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.187553.log 2023-10-16T13:50:09.261 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.112309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113643.log 2023-10-16T13:50:09.262 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.187553.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.112309.log.gz 2023-10-16T13:50:09.262 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.113643.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.187553.log.gz 2023-10-16T13:50:09.262 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.113643.log.gz 2023-10-16T13:50:09.262 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145491.log 2023-10-16T13:50:09.263 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.180471.log 2023-10-16T13:50:09.263 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.145491.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145629.log 2023-10-16T13:50:09.263 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.180471.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.145491.log.gz 2023-10-16T13:50:09.264 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.145629.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.180471.log.gz 2023-10-16T13:50:09.264 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.145629.log.gz 2023-10-16T13:50:09.264 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134537.log 2023-10-16T13:50:09.264 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166506.log 2023-10-16T13:50:09.265 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.134537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134767.log 2023-10-16T13:50:09.265 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.166506.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.134537.log.gz 2023-10-16T13:50:09.265 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.134767.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.166506.log.gz 2023-10-16T13:50:09.266 INFO:teuthology.orchestra.run.smithi016.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.134767.log.gz 2023-10-16T13:50:09.266 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.192670.log 2023-10-16T13:50:09.266 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165023.log 2023-10-16T13:50:09.266 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.0.192670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.185790.log 2023-10-16T13:50:09.267 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.192670.log.gz 2023-10-16T13:50:09.267 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.165023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.187001.log 2023-10-16T13:50:09.267 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.185790.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.165023.log.gz 2023-10-16T13:50:09.268 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.187001.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.185790.log.gz 2023-10-16T13:50:09.268 INFO:teuthology.orchestra.run.smithi016.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.187001.log.gz 2023-10-16T13:50:09.268 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.179755.log 2023-10-16T13:50:09.269 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142568.log 2023-10-16T13:50:09.269 INFO:teuthology.orchestra.run.smithi016.stderr:gzip /var/log/ceph/ceph-client.admin.179755.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.166086.log 2023-10-16T13:50:09.269 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.142568.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.179755.log.gz 2023-10-16T13:50:09.269 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.166086.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.142568.log.gz 2023-10-16T13:50:09.270 INFO:teuthology.orchestra.run.smithi016.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.166086.log.gz 2023-10-16T13:50:09.270 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154027.log 2023-10-16T13:50:09.270 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137308.log 2023-10-16T13:50:09.271 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.154027.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141773.log 2023-10-16T13:50:09.271 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.137308.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.154027.log.gz 2023-10-16T13:50:09.271 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.141773.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.137308.log.gz 2023-10-16T13:50:09.272 INFO:teuthology.orchestra.run.smithi016.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.141773.log.gz 2023-10-16T13:50:09.272 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128078.log 2023-10-16T13:50:09.272 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.184016.log 2023-10-16T13:50:09.272 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.128078.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160790.log 2023-10-16T13:50:09.273 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.184016.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.128078.log.gz 2023-10-16T13:50:09.273 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.160790.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.184016.log.gz 2023-10-16T13:50:09.273 INFO:teuthology.orchestra.run.smithi016.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.160790.log.gz 2023-10-16T13:50:09.273 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152441.log 2023-10-16T13:50:09.274 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.181462.log 2023-10-16T13:50:09.274 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.152441.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116292.log 2023-10-16T13:50:09.274 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.181462.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.152441.log.gz 2023-10-16T13:50:09.275 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.116292.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.181462.log.gz 2023-10-16T13:50:09.275 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.116292.log.gz 2023-10-16T13:50:09.275 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152486.log 2023-10-16T13:50:09.275 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169967.log 2023-10-16T13:50:09.276 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.152486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146293.log 2023-10-16T13:50:09.276 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.169967.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.152486.log.gz 2023-10-16T13:50:09.276 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.146293.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.169967.log.gz 2023-10-16T13:50:09.277 INFO:teuthology.orchestra.run.smithi016.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.146293.log.gz 2023-10-16T13:50:09.277 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146143.log 2023-10-16T13:50:09.277 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.184526.log 2023-10-16T13:50:09.278 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.146143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164923.log 2023-10-16T13:50:09.278 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.184526.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.146143.log.gz 2023-10-16T13:50:09.278 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.164923.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.184526.log.gz 2023-10-16T13:50:09.278 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.164923.log.gz 2023-10-16T13:50:09.279 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161776.log 2023-10-16T13:50:09.279 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116107.log 2023-10-16T13:50:09.279 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.161776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149084.log 2023-10-16T13:50:09.280 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.116107.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.161776.log.gz 2023-10-16T13:50:09.280 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.149084.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.116107.log.gz 2023-10-16T13:50:09.280 INFO:teuthology.orchestra.run.smithi016.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.149084.log.gz 2023-10-16T13:50:09.280 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.189520.log 2023-10-16T13:50:09.281 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120174.log 2023-10-16T13:50:09.281 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.189520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171250.log 2023-10-16T13:50:09.281 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.120174.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.189520.log.gz 2023-10-16T13:50:09.281 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.171250.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.120174.log.gz 2023-10-16T13:50:09.282 INFO:teuthology.orchestra.run.smithi016.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.171250.log.gz 2023-10-16T13:50:09.282 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146239.log 2023-10-16T13:50:09.282 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.192321.log 2023-10-16T13:50:09.283 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.146239.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.176876.log 2023-10-16T13:50:09.283 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.192321.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.146239.log.gz 2023-10-16T13:50:09.283 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.176876.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.192321.log.gz 2023-10-16T13:50:09.283 INFO:teuthology.orchestra.run.smithi016.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.176876.log.gz 2023-10-16T13:50:09.284 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111622.log 2023-10-16T13:50:09.284 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150018.log 2023-10-16T13:50:09.284 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.111622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.177745.log 2023-10-16T13:50:09.285 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.150018.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.111622.log.gz 2023-10-16T13:50:09.285 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.0.177745.log: 89.1% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.150018.log.gz 2023-10-16T13:50:09.285 INFO:teuthology.orchestra.run.smithi016.stderr: -- replaced with /var/log/ceph/ceph-client.0.177745.log.gz 2023-10-16T13:50:09.285 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112604.log 2023-10-16T13:50:09.286 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109971.log 2023-10-16T13:50:09.286 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.112604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149834.log 2023-10-16T13:50:09.286 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.109971.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.112604.log.gz 2023-10-16T13:50:09.286 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.149834.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.109971.log.gz 2023-10-16T13:50:09.287 INFO:teuthology.orchestra.run.smithi016.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.149834.log.gz 2023-10-16T13:50:09.287 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.182744.log 2023-10-16T13:50:09.287 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119432.log 2023-10-16T13:50:09.287 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.182744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.187661.log 2023-10-16T13:50:09.288 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.119432.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.182744.log.gz 2023-10-16T13:50:09.288 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.187661.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.119432.log.gz 2023-10-16T13:50:09.288 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.187661.log.gz 2023-10-16T13:50:09.289 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162642.log 2023-10-16T13:50:09.289 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109668.log 2023-10-16T13:50:09.289 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.162642.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145959.log 2023-10-16T13:50:09.290 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.109668.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.162642.log.gz 2023-10-16T13:50:09.290 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.145959.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.109668.log.gz 2023-10-16T13:50:09.290 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.145959.log.gz 2023-10-16T13:50:09.290 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141367.log 2023-10-16T13:50:09.291 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150787.log 2023-10-16T13:50:09.291 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.141367.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119709.log 2023-10-16T13:50:09.291 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.150787.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.141367.log.gz 2023-10-16T13:50:09.292 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.119709.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.150787.log.gz 2023-10-16T13:50:09.292 INFO:teuthology.orchestra.run.smithi016.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.119709.log.gz 2023-10-16T13:50:09.292 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120220.log 2023-10-16T13:50:09.293 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170749.log 2023-10-16T13:50:09.293 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.120220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155499.log 2023-10-16T13:50:09.293 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.170749.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.120220.log.gz 2023-10-16T13:50:09.293 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.155499.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.170749.log.gz 2023-10-16T13:50:09.294 INFO:teuthology.orchestra.run.smithi016.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.155499.log.gz 2023-10-16T13:50:09.294 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.181186.log 2023-10-16T13:50:09.294 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.176735.log 2023-10-16T13:50:09.295 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.181186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.179508.log 2023-10-16T13:50:09.295 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.176735.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.181186.log.gz 2023-10-16T13:50:09.295 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.179508.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.176735.log.gz 2023-10-16T13:50:09.296 INFO:teuthology.orchestra.run.smithi016.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.179508.log.gz 2023-10-16T13:50:09.296 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141819.log 2023-10-16T13:50:09.296 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153563.log 2023-10-16T13:50:09.297 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.141819.log: /var/log/ceph/ceph-client.admin.153563.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.141819.log.gz 2023-10-16T13:50:09.297 INFO:teuthology.orchestra.run.smithi016.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.153563.log.gz 2023-10-16T13:50:09.297 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.182698.log 2023-10-16T13:50:09.297 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.189567.log 2023-10-16T13:50:09.298 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.182698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117439.log 2023-10-16T13:50:09.298 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.189567.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.182698.log.gz 2023-10-16T13:50:09.298 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.117439.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.189567.log.gz 2023-10-16T13:50:09.299 INFO:teuthology.orchestra.run.smithi016.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.117439.log.gz 2023-10-16T13:50:09.299 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.175992.log 2023-10-16T13:50:09.299 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.187507.log 2023-10-16T13:50:09.300 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.175992.log: /var/log/ceph/ceph-client.admin.187507.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.175992.log.gz 2023-10-16T13:50:09.300 INFO:teuthology.orchestra.run.smithi016.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.187507.log.gz 2023-10-16T13:50:09.300 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2023-10-16T13:50:09.301 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146433.log 2023-10-16T13:50:09.301 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164653.log 2023-10-16T13:50:09.301 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.146433.log: /var/log/ceph/ceph-client.admin.164653.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.146433.log.gz 2023-10-16T13:50:09.302 INFO:teuthology.orchestra.run.smithi016.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.164653.log.gz 2023-10-16T13:50:09.302 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151767.log 2023-10-16T13:50:09.302 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162219.log 2023-10-16T13:50:09.303 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.151767.log: /var/log/ceph/ceph-client.admin.162219.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.151767.log.gz 2023-10-16T13:50:09.303 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.162219.log.gz 2023-10-16T13:50:09.303 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144063.log 2023-10-16T13:50:09.304 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119571.log 2023-10-16T13:50:09.304 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.144063.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.144063.log.gz 2023-10-16T13:50:09.304 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.119571.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.119571.log.gz 2023-10-16T13:50:09.304 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170999.log 2023-10-16T13:50:09.305 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113965.log 2023-10-16T13:50:09.305 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.170999.log: /var/log/ceph/ceph-client.admin.113965.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.170999.log.gz 2023-10-16T13:50:09.305 INFO:teuthology.orchestra.run.smithi016.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.113965.log.gz 2023-10-16T13:50:09.306 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151063.log 2023-10-16T13:50:09.306 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142380.log 2023-10-16T13:50:09.306 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.151063.log: /var/log/ceph/ceph-client.admin.142380.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.151063.log.gz 2023-10-16T13:50:09.307 INFO:teuthology.orchestra.run.smithi016.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.142380.log.gz 2023-10-16T13:50:09.307 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153765.log 2023-10-16T13:50:09.307 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120128.log 2023-10-16T13:50:09.308 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.153765.log: /var/log/ceph/ceph-client.admin.120128.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.153765.log.gz 2023-10-16T13:50:09.308 INFO:teuthology.orchestra.run.smithi016.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.120128.log.gz 2023-10-16T13:50:09.308 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169869.log 2023-10-16T13:50:09.309 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153613.log 2023-10-16T13:50:09.309 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.169869.log: /var/log/ceph/ceph-client.admin.153613.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.169869.log.gz 2023-10-16T13:50:09.309 INFO:teuthology.orchestra.run.smithi016.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.153613.log.gz 2023-10-16T13:50:09.310 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.151424.log 2023-10-16T13:50:09.310 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150347.log 2023-10-16T13:50:09.310 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.0.151424.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.151424.log.gz 2023-10-16T13:50:09.311 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.150347.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.150347.log.gz 2023-10-16T13:50:09.311 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154073.log 2023-10-16T13:50:09.311 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112597.log 2023-10-16T13:50:09.312 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.154073.log: /var/log/ceph/ceph-client.admin.112597.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.154073.log.gz 2023-10-16T13:50:09.312 INFO:teuthology.orchestra.run.smithi016.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.112597.log.gz 2023-10-16T13:50:09.312 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165842.log 2023-10-16T13:50:09.313 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145675.log 2023-10-16T13:50:09.313 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.165842.log: /var/log/ceph/ceph-client.admin.145675.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.165842.log.gz 2023-10-16T13:50:09.313 INFO:teuthology.orchestra.run.smithi016.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.145675.log.gz 2023-10-16T13:50:09.313 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134813.log 2023-10-16T13:50:09.314 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134629.log 2023-10-16T13:50:09.314 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.134813.log: /var/log/ceph/ceph-client.admin.134629.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.134813.log.gz 2023-10-16T13:50:09.315 INFO:teuthology.orchestra.run.smithi016.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.134629.log.gz 2023-10-16T13:50:09.315 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.180648.log 2023-10-16T13:50:09.315 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.175326.log 2023-10-16T13:50:09.315 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.180648.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.180648.log.gz 2023-10-16T13:50:09.316 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.175326.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.175326.log.gz 2023-10-16T13:50:09.316 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151576.log 2023-10-16T13:50:09.316 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110017.log 2023-10-16T13:50:09.317 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.151576.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.151576.log.gz 2023-10-16T13:50:09.317 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.110017.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.190512.log 2023-10-16T13:50:09.317 INFO:teuthology.orchestra.run.smithi016.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.110017.log.gz 2023-10-16T13:50:09.318 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.185892.log 2023-10-16T13:50:09.318 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.190512.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.190512.log.gz 2023-10-16T13:50:09.318 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.185892.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.185892.log.gz 2023-10-16T13:50:09.319 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166598.log 2023-10-16T13:50:09.319 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137459.log 2023-10-16T13:50:09.319 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.166598.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.166598.log.gz 2023-10-16T13:50:09.320 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.137459.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.137459.log.gz 2023-10-16T13:50:09.320 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150393.log 2023-10-16T13:50:09.320 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155153.log 2023-10-16T13:50:09.321 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.150393.log: /var/log/ceph/ceph-client.admin.155153.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.150393.log.gz 2023-10-16T13:50:09.321 INFO:teuthology.orchestra.run.smithi016.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.155153.log.gz 2023-10-16T13:50:09.321 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118136.log 2023-10-16T13:50:09.322 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116430.log 2023-10-16T13:50:09.322 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.118136.log: /var/log/ceph/ceph-client.admin.116430.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.118136.log.gz 2023-10-16T13:50:09.322 INFO:teuthology.orchestra.run.smithi016.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.116430.log.gz 2023-10-16T13:50:09.323 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139080.log 2023-10-16T13:50:09.323 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159372.log 2023-10-16T13:50:09.323 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.139080.log: /var/log/ceph/ceph-client.admin.159372.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.139080.log.gz 2023-10-16T13:50:09.323 INFO:teuthology.orchestra.run.smithi016.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.159372.log.gz 2023-10-16T13:50:09.324 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136994.log 2023-10-16T13:50:09.324 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.130340.log 2023-10-16T13:50:09.325 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.136994.log: /var/log/ceph/ceph-client.0.130340.log: 88.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.136994.log.gz-5 2023-10-16T13:50:09.325 INFO:teuthology.orchestra.run.smithi016.stderr:--verbose -- /var/log/ceph/ceph-client.admin.114149.log 2023-10-16T13:50:09.325 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.130340.log.gz 2023-10-16T13:50:09.325 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124620.log 2023-10-16T13:50:09.326 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.114149.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.114149.log.gz 2023-10-16T13:50:09.326 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.124620.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.124620.log.gz 2023-10-16T13:50:09.326 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.182232.log 2023-10-16T13:50:09.326 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153046.log 2023-10-16T13:50:09.327 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.182232.log: /var/log/ceph/ceph-client.admin.153046.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.182232.log.gz 2023-10-16T13:50:09.327 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.153046.log.gz 2023-10-16T13:50:09.327 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.178622.log 2023-10-16T13:50:09.328 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139034.log 2023-10-16T13:50:09.328 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.178622.log: /var/log/ceph/ceph-client.admin.139034.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.178622.log.gz 2023-10-16T13:50:09.328 INFO:teuthology.orchestra.run.smithi016.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.139034.log.gz 2023-10-16T13:50:09.329 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109875.log 2023-10-16T13:50:09.329 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141673.log 2023-10-16T13:50:09.329 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.109875.log: /var/log/ceph/ceph-client.admin.141673.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.109875.log.gz 2023-10-16T13:50:09.329 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.141673.log.gz 2023-10-16T13:50:09.330 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142426.log 2023-10-16T13:50:09.330 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130446.log 2023-10-16T13:50:09.330 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.142426.log: /var/log/ceph/ceph-client.admin.130446.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.142426.log.gz 2023-10-16T13:50:09.330 INFO:teuthology.orchestra.run.smithi016.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.130446.log.gz 2023-10-16T13:50:09.331 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154771.log 2023-10-16T13:50:09.331 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135993.log 2023-10-16T13:50:09.331 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.154771.log: /var/log/ceph/ceph-client.admin.135993.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.154771.log.gz 2023-10-16T13:50:09.332 INFO:teuthology.orchestra.run.smithi016.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.135993.log.gz 2023-10-16T13:50:09.332 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140577.log 2023-10-16T13:50:09.332 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149926.log 2023-10-16T13:50:09.333 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.140577.log: /var/log/ceph/ceph-client.admin.149926.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.140577.log.gz 2023-10-16T13:50:09.333 INFO:teuthology.orchestra.run.smithi016.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.149926.log.gz 2023-10-16T13:50:09.333 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.189302.log 2023-10-16T13:50:09.333 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.181232.log 2023-10-16T13:50:09.334 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.189302.log: /var/log/ceph/ceph-client.admin.181232.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.189302.log.gz 2023-10-16T13:50:09.334 INFO:teuthology.orchestra.run.smithi016.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.181232.log.gz 2023-10-16T13:50:09.334 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127069.log 2023-10-16T13:50:09.335 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128379.log 2023-10-16T13:50:09.335 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.127069.log: /var/log/ceph/ceph-client.admin.128379.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.127069.log.gz 2023-10-16T13:50:09.335 INFO:teuthology.orchestra.run.smithi016.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.128379.log.gz 2023-10-16T13:50:09.335 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.196265.log 2023-10-16T13:50:09.336 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152150.log 2023-10-16T13:50:09.336 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.196265.log: /var/log/ceph/ceph-client.admin.152150.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.196265.log.gz 2023-10-16T13:50:09.336 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.152150.log.gz 2023-10-16T13:50:09.336 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141116.log 2023-10-16T13:50:09.337 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116898.log 2023-10-16T13:50:09.337 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.141116.log: /var/log/ceph/ceph-client.admin.116898.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.141116.log.gz 2023-10-16T13:50:09.337 INFO:teuthology.orchestra.run.smithi016.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.116898.log.gz 2023-10-16T13:50:09.337 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.194666.log 2023-10-16T13:50:09.338 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112401.log 2023-10-16T13:50:09.338 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.194666.log: /var/log/ceph/ceph-client.admin.112401.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.194666.log.gz 2023-10-16T13:50:09.338 INFO:teuthology.orchestra.run.smithi016.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.112401.log.gz 2023-10-16T13:50:09.338 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.177438.log 2023-10-16T13:50:09.339 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117952.log 2023-10-16T13:50:09.339 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.177438.log: /var/log/ceph/ceph-client.admin.117952.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.117952.log.gz 2023-10-16T13:50:09.339 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.195986.log 2023-10-16T13:50:09.339 INFO:teuthology.orchestra.run.smithi016.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.177438.log.gz 2023-10-16T13:50:09.340 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.2.115655.log 2023-10-16T13:50:09.340 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.195986.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.195986.log.gz 2023-10-16T13:50:09.340 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138415.log 2023-10-16T13:50:09.341 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.2.115655.log: 0.0% -- replaced with /var/log/ceph/ceph-client.2.115655.log.gz 2023-10-16T13:50:09.341 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.138415.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.138415.log.gz 2023-10-16T13:50:09.341 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.187093.log 2023-10-16T13:50:09.341 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117531.log 2023-10-16T13:50:09.342 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.187093.log: /var/log/ceph/ceph-client.admin.117531.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.187093.log.gz 2023-10-16T13:50:09.342 INFO:teuthology.orchestra.run.smithi016.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.117531.log.gz 2023-10-16T13:50:09.342 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170861.log 2023-10-16T13:50:09.343 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147652.log 2023-10-16T13:50:09.343 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.170861.log: /var/log/ceph/ceph-client.admin.147652.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.170861.log.gz 2023-10-16T13:50:09.343 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.147652.log.gz 2023-10-16T13:50:09.343 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171814.log 2023-10-16T13:50:09.344 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154679.log 2023-10-16T13:50:09.344 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.171814.log: /var/log/ceph/ceph-client.admin.154679.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.171814.log.gz 2023-10-16T13:50:09.344 INFO:teuthology.orchestra.run.smithi016.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.154679.log.gz 2023-10-16T13:50:09.345 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114057.log 2023-10-16T13:50:09.345 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116944.log 2023-10-16T13:50:09.345 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.114057.log: /var/log/ceph/ceph-client.admin.116944.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.114057.log.gz 2023-10-16T13:50:09.345 INFO:teuthology.orchestra.run.smithi016.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.116944.log.gz 2023-10-16T13:50:09.346 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140675.log 2023-10-16T13:50:09.346 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139174.log 2023-10-16T13:50:09.346 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.140675.log: /var/log/ceph/ceph-client.admin.139174.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.140675.log.gz 2023-10-16T13:50:09.347 INFO:teuthology.orchestra.run.smithi016.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.139174.log.gz 2023-10-16T13:50:09.347 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117036.log 2023-10-16T13:50:09.347 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.196221.log 2023-10-16T13:50:09.347 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.117036.log: /var/log/ceph/ceph-client.admin.196221.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.117036.log.gz 2023-10-16T13:50:09.348 INFO:teuthology.orchestra.run.smithi016.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.196221.log.gz 2023-10-16T13:50:09.348 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.183630.log 2023-10-16T13:50:09.348 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145583.log 2023-10-16T13:50:09.349 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.0.183630.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.183630.log.gz 2023-10-16T13:50:09.349 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.145583.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.145583.log.gz 2023-10-16T13:50:09.349 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.177918.log 2023-10-16T13:50:09.349 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.178484.log 2023-10-16T13:50:09.350 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.177918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120909.log 2023-10-16T13:50:09.350 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.178484.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.177918.log.gz 2023-10-16T13:50:09.350 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.120909.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.178484.log.gz 2023-10-16T13:50:09.351 INFO:teuthology.orchestra.run.smithi016.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.120909.log.gz 2023-10-16T13:50:09.351 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150126.log 2023-10-16T13:50:09.351 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.185528.log 2023-10-16T13:50:09.352 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.150126.log: /var/log/ceph/ceph-client.admin.185528.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.150126.log.gz 2023-10-16T13:50:09.352 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.185528.log.gz 2023-10-16T13:50:09.352 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133949.log 2023-10-16T13:50:09.352 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.190362.log 2023-10-16T13:50:09.353 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.133949.log: /var/log/ceph/ceph-client.admin.190362.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.133949.log.gz 2023-10-16T13:50:09.353 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.190362.log.gz 2023-10-16T13:50:09.353 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.177062.log 2023-10-16T13:50:09.354 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128471.log 2023-10-16T13:50:09.354 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.177062.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.177062.log.gz 2023-10-16T13:50:09.355 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.128471.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.128471.log.gz 2023-10-16T13:50:09.355 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137560.log 2023-10-16T13:50:09.355 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113781.log 2023-10-16T13:50:09.356 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.137560.log: /var/log/ceph/ceph-client.admin.113781.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.137560.log.gz 2023-10-16T13:50:09.356 INFO:teuthology.orchestra.run.smithi016.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.113781.log.gz 2023-10-16T13:50:09.356 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161256.log 2023-10-16T13:50:09.357 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.186679.log 2023-10-16T13:50:09.357 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.161256.log: /var/log/ceph/ceph-client.admin.186679.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.161256.log.gz 2023-10-16T13:50:09.357 INFO:teuthology.orchestra.run.smithi016.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.186679.log.gz 2023-10-16T13:50:09.358 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150695.log 2023-10-16T13:50:09.358 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128871.log 2023-10-16T13:50:09.358 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.150695.log: /var/log/ceph/ceph-client.admin.128871.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.150695.log.gz 2023-10-16T13:50:09.359 INFO:teuthology.orchestra.run.smithi016.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.128871.log.gz 2023-10-16T13:50:09.359 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135453.log 2023-10-16T13:50:09.359 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.178056.log 2023-10-16T13:50:09.360 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.135453.log: /var/log/ceph/ceph-client.admin.178056.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.135453.log.gz 2023-10-16T13:50:09.360 INFO:teuthology.orchestra.run.smithi016.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.178056.log.gz 2023-10-16T13:50:09.360 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.188021.log 2023-10-16T13:50:09.361 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.189844.log 2023-10-16T13:50:09.361 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.188021.log: /var/log/ceph/ceph-client.admin.189844.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.188021.log.gz 2023-10-16T13:50:09.361 INFO:teuthology.orchestra.run.smithi016.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.189844.log.gz 2023-10-16T13:50:09.362 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.193802.log 2023-10-16T13:50:09.362 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122160.log 2023-10-16T13:50:09.362 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.193802.log: /var/log/ceph/ceph-client.admin.122160.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.193802.log.gz 2023-10-16T13:50:09.363 INFO:teuthology.orchestra.run.smithi016.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.122160.log.gz 2023-10-16T13:50:09.363 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121549.log 2023-10-16T13:50:09.363 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136809.log 2023-10-16T13:50:09.365 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.121549.log: /var/log/ceph/ceph-client.admin.136809.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.121549.log.gz 2023-10-16T13:50:09.365 INFO:teuthology.orchestra.run.smithi016.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.136809.log.gz 2023-10-16T13:50:09.366 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169116.log 2023-10-16T13:50:09.366 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.182456.log 2023-10-16T13:50:09.367 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.169116.log: /var/log/ceph/ceph-client.admin.182456.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.169116.log.gz 2023-10-16T13:50:09.367 INFO:teuthology.orchestra.run.smithi016.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.182456.log.gz 2023-10-16T13:50:09.367 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154633.log 2023-10-16T13:50:09.368 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.172637.log 2023-10-16T13:50:09.368 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.154633.log: /var/log/ceph/ceph-client.0.172637.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.154633.log.gz 2023-10-16T13:50:09.368 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.172637.log.gz 2023-10-16T13:50:09.369 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148883.log 2023-10-16T13:50:09.369 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138139.log 2023-10-16T13:50:09.369 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.148883.log: /var/log/ceph/ceph-client.admin.138139.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.148883.log.gz 2023-10-16T13:50:09.370 INFO:teuthology.orchestra.run.smithi016.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.138139.log.gz 2023-10-16T13:50:09.370 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154913.log 2023-10-16T13:50:09.370 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142660.log 2023-10-16T13:50:09.371 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.154913.log: /var/log/ceph/ceph-client.admin.142660.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.154913.log.gz 2023-10-16T13:50:09.371 INFO:teuthology.orchestra.run.smithi016.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.142660.log.gz 2023-10-16T13:50:09.371 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170487.log 2023-10-16T13:50:09.372 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.191474.log 2023-10-16T13:50:09.372 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.170487.log: /var/log/ceph/ceph-client.admin.191474.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.170487.log.gz 2023-10-16T13:50:09.372 INFO:teuthology.orchestra.run.smithi016.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.191474.log.gz 2023-10-16T13:50:09.373 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.190759.log 2023-10-16T13:50:09.373 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.178438.log 2023-10-16T13:50:09.373 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.190759.log: /var/log/ceph/ceph-client.admin.178438.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.190759.log.gz 2023-10-16T13:50:09.374 INFO:teuthology.orchestra.run.smithi016.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.178438.log.gz 2023-10-16T13:50:09.374 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.175377.log 2023-10-16T13:50:09.374 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.184434.log 2023-10-16T13:50:09.375 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.175377.log: /var/log/ceph/ceph-client.admin.184434.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.175377.log.gz 2023-10-16T13:50:09.375 INFO:teuthology.orchestra.run.smithi016.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.184434.log.gz 2023-10-16T13:50:09.375 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110433.log 2023-10-16T13:50:09.375 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155107.log 2023-10-16T13:50:09.376 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.110433.log: /var/log/ceph/ceph-client.admin.155107.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.155107.log.gz 2023-10-16T13:50:09.376 INFO:teuthology.orchestra.run.smithi016.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.110433.log.gz 2023-10-16T13:50:09.376 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.182838.log 2023-10-16T13:50:09.376 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137141.log 2023-10-16T13:50:09.377 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.182838.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.182838.log.gz 2023-10-16T13:50:09.377 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123508.log 2023-10-16T13:50:09.377 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.137141.log: /var/log/ceph/ceph-client.admin.123508.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.137141.log.gz 2023-10-16T13:50:09.377 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.123508.log.gz 2023-10-16T13:50:09.378 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.163597.log 2023-10-16T13:50:09.378 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.191840.log 2023-10-16T13:50:09.378 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.0.163597.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.163597.log.gz 2023-10-16T13:50:09.379 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.191840.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.191840.log.gz 2023-10-16T13:50:09.379 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.182976.log 2023-10-16T13:50:09.379 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-10-16T13:50:09.379 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.182976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133697.log 2023-10-16T13:50:09.380 INFO:teuthology.orchestra.run.smithi016.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.182976.log.gz 2023-10-16T13:50:09.380 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-client.admin.133697.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.133697.log.gz 2023-10-16T13:50:09.380 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.196124.log 2023-10-16T13:50:09.380 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136943.log 2023-10-16T13:50:09.381 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.196124.log: /var/log/ceph/ceph-client.admin.136943.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.196124.log.gz 2023-10-16T13:50:09.381 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.136943.log.gz 2023-10-16T13:50:09.381 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113873.log 2023-10-16T13:50:09.381 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.113873.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.113873.log.gz 2023-10-16T13:50:09.382 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127826.log 2023-10-16T13:50:09.382 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.175225.log 2023-10-16T13:50:09.382 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.127826.log: /var/log/ceph/ceph-client.admin.175225.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.127826.log.gz 2023-10-16T13:50:09.383 INFO:teuthology.orchestra.run.smithi016.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.175225.log.gz 2023-10-16T13:50:09.383 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118454.log 2023-10-16T13:50:09.383 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.118454.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.118454.log.gz 2023-10-16T13:50:09.383 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116622.log 2023-10-16T13:50:09.383 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.116622.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.116622.log.gz 2023-10-16T13:50:09.384 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165073.log 2023-10-16T13:50:09.384 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.165073.log: 89.8% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-10-16T13:50:09.384 INFO:teuthology.orchestra.run.smithi016.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.165073.log.gz 2023-10-16T13:50:09.384 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.196170.log 2023-10-16T13:50:09.385 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.191744.log 2023-10-16T13:50:09.385 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.196170.log: /var/log/ceph/ceph-client.admin.191744.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.196170.log.gz 2023-10-16T13:50:09.385 INFO:teuthology.orchestra.run.smithi016.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.191744.log.gz 2023-10-16T13:50:09.386 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128227.log 2023-10-16T13:50:09.386 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149235.log 2023-10-16T13:50:09.386 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.128227.log: /var/log/ceph/ceph-client.admin.149235.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.128227.log.gz 2023-10-16T13:50:09.387 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.149235.log.gz 2023-10-16T13:50:09.387 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144860.log 2023-10-16T13:50:09.387 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.194823.log 2023-10-16T13:50:09.387 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.144860.log: /var/log/ceph/ceph-client.admin.194823.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.144860.log.gz 2023-10-16T13:50:09.388 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.194823.log.gz 2023-10-16T13:50:09.388 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111373.log 2023-10-16T13:50:09.388 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.183122.log 2023-10-16T13:50:09.388 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.111373.log: /var/log/ceph/ceph-client.admin.183122.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.111373.log.gz 2023-10-16T13:50:09.389 INFO:teuthology.orchestra.run.smithi016.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.183122.log.gz 2023-10-16T13:50:09.389 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155383.log 2023-10-16T13:50:09.389 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118684.log 2023-10-16T13:50:09.390 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.155383.log: /var/log/ceph/ceph-client.admin.118684.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.155383.log.gz 2023-10-16T13:50:09.390 INFO:teuthology.orchestra.run.smithi016.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.118684.log.gz 2023-10-16T13:50:09.390 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.136226.log 2023-10-16T13:50:09.390 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.172238.log 2023-10-16T13:50:09.391 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.0.136226.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.136226.log.gz 2023-10-16T13:50:09.391 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.172238.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.172238.log.gz 2023-10-16T13:50:09.391 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135685.log 2023-10-16T13:50:09.392 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.193273.log 2023-10-16T13:50:09.392 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.135685.log: /var/log/ceph/ceph-client.admin.193273.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.135685.log.gz 2023-10-16T13:50:09.392 INFO:teuthology.orchestra.run.smithi016.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.193273.log.gz 2023-10-16T13:50:09.392 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.185942.log 2023-10-16T13:50:09.393 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166738.log 2023-10-16T13:50:09.393 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.185942.log: /var/log/ceph/ceph-client.admin.166738.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.185942.log.gz 2023-10-16T13:50:09.393 INFO:teuthology.orchestra.run.smithi016.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.166738.log.gz 2023-10-16T13:50:09.393 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124062.log 2023-10-16T13:50:09.394 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148717.log 2023-10-16T13:50:09.394 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.124062.log: /var/log/ceph/ceph-client.admin.148717.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.124062.log.gz 2023-10-16T13:50:09.394 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.148717.log.gz 2023-10-16T13:50:09.395 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164772.log 2023-10-16T13:50:09.395 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.190412.log 2023-10-16T13:50:09.395 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.164772.log: /var/log/ceph/ceph-client.admin.190412.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.164772.log.gz 2023-10-16T13:50:09.395 INFO:teuthology.orchestra.run.smithi016.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.190412.log.gz 2023-10-16T13:50:09.396 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.175107.log 2023-10-16T13:50:09.396 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113597.log 2023-10-16T13:50:09.396 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.175107.log: /var/log/ceph/ceph-client.admin.113597.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.175107.log.gz 2023-10-16T13:50:09.396 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.113597.log.gz 2023-10-16T13:50:09.397 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.177300.log 2023-10-16T13:50:09.397 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137043.log 2023-10-16T13:50:09.397 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.177300.log: /var/log/ceph/ceph-client.admin.137043.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.177300.log.gz 2023-10-16T13:50:09.398 INFO:teuthology.orchestra.run.smithi016.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.137043.log.gz 2023-10-16T13:50:09.398 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.189231.log 2023-10-16T13:50:09.398 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137611.log 2023-10-16T13:50:09.398 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.189231.log: /var/log/ceph/ceph-client.admin.137611.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.189231.log.gz 2023-10-16T13:50:09.399 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.137611.log.gz 2023-10-16T13:50:09.399 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.167275.log 2023-10-16T13:50:09.399 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160937.log 2023-10-16T13:50:09.400 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.0.167275.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.167275.log.gz 2023-10-16T13:50:09.400 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.160937.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.172008.log 2023-10-16T13:50:09.400 INFO:teuthology.orchestra.run.smithi016.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.160937.log.gz 2023-10-16T13:50:09.400 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.195756.log 2023-10-16T13:50:09.401 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.172008.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.172008.log.gz 2023-10-16T13:50:09.401 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141477.log 2023-10-16T13:50:09.401 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.195756.log: /var/log/ceph/ceph-client.admin.141477.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.195756.log.gz 2023-10-16T13:50:09.402 INFO:teuthology.orchestra.run.smithi016.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.141477.log.gz 2023-10-16T13:50:09.402 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164605.log 2023-10-16T13:50:09.402 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.188938.log 2023-10-16T13:50:09.402 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.164605.log: /var/log/ceph/ceph-client.0.188938.log: 88.3% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.164605.log.gz -- replaced with /var/log/ceph/ceph-client.0.188938.log.gz 2023-10-16T13:50:09.403 INFO:teuthology.orchestra.run.smithi016.stderr: 2023-10-16T13:50:09.403 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.182033.log 2023-10-16T13:50:09.403 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155942.log 2023-10-16T13:50:09.403 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.182033.log: /var/log/ceph/ceph-client.admin.155942.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.182033.log.gz 2023-10-16T13:50:09.404 INFO:teuthology.orchestra.run.smithi016.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.155942.log.gz 2023-10-16T13:50:09.404 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.147083.log 2023-10-16T13:50:09.404 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.189044.log 2023-10-16T13:50:09.404 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.0.147083.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.147083.log.gz 2023-10-16T13:50:09.405 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.189044.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.189044.log.gz 2023-10-16T13:50:09.405 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.174909.log 2023-10-16T13:50:09.405 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116990.log 2023-10-16T13:50:09.405 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.174909.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.174909.log.gz 2023-10-16T13:50:09.406 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.179264.log 2023-10-16T13:50:09.406 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.116990.log: /var/log/ceph/ceph-client.admin.179264.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.116990.log.gz 2023-10-16T13:50:09.406 INFO:teuthology.orchestra.run.smithi016.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.179264.log.gz 2023-10-16T13:50:09.407 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.190561.log 2023-10-16T13:50:09.407 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112963.log 2023-10-16T13:50:09.407 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.190561.log: /var/log/ceph/ceph-client.admin.112963.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.190561.log.gz 2023-10-16T13:50:09.407 INFO:teuthology.orchestra.run.smithi016.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.112963.log.gz 2023-10-16T13:50:09.408 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166171.log 2023-10-16T13:50:09.408 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.181895.log 2023-10-16T13:50:09.408 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.166171.log: /var/log/ceph/ceph-client.admin.181895.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.166171.log.gz 2023-10-16T13:50:09.408 INFO:teuthology.orchestra.run.smithi016.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.181895.log.gz 2023-10-16T13:50:09.409 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.194869.log 2023-10-16T13:50:09.409 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144911.log 2023-10-16T13:50:09.409 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.194869.log: /var/log/ceph/ceph-client.admin.144911.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.194869.log.gz 2023-10-16T13:50:09.410 INFO:teuthology.orchestra.run.smithi016.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.144911.log.gz 2023-10-16T13:50:09.410 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145399.log 2023-10-16T13:50:09.410 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.183924.log 2023-10-16T13:50:09.411 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.145399.log: /var/log/ceph/ceph-client.admin.183924.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.145399.log.gz 2023-10-16T13:50:09.411 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.183924.log.gz 2023-10-16T13:50:09.411 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.190713.log 2023-10-16T13:50:09.411 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162876.log 2023-10-16T13:50:09.412 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.190713.log: /var/log/ceph/ceph-client.admin.162876.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.190713.log.gz 2023-10-16T13:50:09.412 INFO:teuthology.orchestra.run.smithi016.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.162876.log.gz 2023-10-16T13:50:09.412 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116153.log 2023-10-16T13:50:09.413 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.188631.log 2023-10-16T13:50:09.413 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.116153.log: /var/log/ceph/ceph-client.admin.188631.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.116153.log.gz 2023-10-16T13:50:09.413 INFO:teuthology.orchestra.run.smithi016.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.188631.log.gz 2023-10-16T13:50:09.413 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.176970.log 2023-10-16T13:50:09.414 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133849.log 2023-10-16T13:50:09.414 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.176970.log: /var/log/ceph/ceph-client.admin.133849.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.176970.log.gz 2023-10-16T13:50:09.414 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.133849.log.gz 2023-10-16T13:50:09.415 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142287.log 2023-10-16T13:50:09.415 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.187415.log 2023-10-16T13:50:09.415 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.142287.log: /var/log/ceph/ceph-client.admin.187415.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.142287.log.gz 2023-10-16T13:50:09.415 INFO:teuthology.orchestra.run.smithi016.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.187415.log.gz 2023-10-16T13:50:09.416 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.186587.log 2023-10-16T13:50:09.416 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130103.log 2023-10-16T13:50:09.416 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.186587.log: /var/log/ceph/ceph-client.admin.130103.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.186587.log.gz 2023-10-16T13:50:09.417 INFO:teuthology.orchestra.run.smithi016.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.130103.log.gz 2023-10-16T13:50:09.417 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153311.log 2023-10-16T13:50:09.417 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.180694.log 2023-10-16T13:50:09.417 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.153311.log: /var/log/ceph/ceph-client.admin.180694.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.153311.log.gz 2023-10-16T13:50:09.418 INFO:teuthology.orchestra.run.smithi016.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.180694.log.gz 2023-10-16T13:50:09.418 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119524.log 2023-10-16T13:50:09.418 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.177200.log 2023-10-16T13:50:09.419 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.119524.log: /var/log/ceph/ceph-client.admin.177200.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.119524.log.gz 2023-10-16T13:50:09.419 INFO:teuthology.orchestra.run.smithi016.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.177200.log.gz 2023-10-16T13:50:09.419 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154457.log 2023-10-16T13:50:09.420 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.194368.log 2023-10-16T13:50:09.420 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.154457.log: /var/log/ceph/ceph-client.admin.194368.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.154457.log.gz 2023-10-16T13:50:09.420 INFO:teuthology.orchestra.run.smithi016.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.194368.log.gz 2023-10-16T13:50:09.421 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118638.log 2023-10-16T13:50:09.421 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164871.log 2023-10-16T13:50:09.421 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.118638.log: /var/log/ceph/ceph-client.admin.164871.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.118638.log.gz 2023-10-16T13:50:09.421 INFO:teuthology.orchestra.run.smithi016.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.164871.log.gz 2023-10-16T13:50:09.422 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119478.log 2023-10-16T13:50:09.422 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150741.log 2023-10-16T13:50:09.422 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.119478.log: /var/log/ceph/ceph-client.admin.150741.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.119478.log.gz 2023-10-16T13:50:09.423 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.150741.log.gz 2023-10-16T13:50:09.423 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.174774.log 2023-10-16T13:50:09.423 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129071.log 2023-10-16T13:50:09.423 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.174774.log: /var/log/ceph/ceph-client.admin.129071.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.174774.log.gz 2023-10-16T13:50:09.424 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.129071.log.gz 2023-10-16T13:50:09.424 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163106.log 2023-10-16T13:50:09.424 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.182317.log 2023-10-16T13:50:09.425 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.163106.log: /var/log/ceph/ceph-client.admin.182317.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.163106.log.gz 2023-10-16T13:50:09.425 INFO:teuthology.orchestra.run.smithi016.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.182317.log.gz 2023-10-16T13:50:09.425 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113299.log 2023-10-16T13:50:09.426 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.191559.log 2023-10-16T13:50:09.426 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.113299.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.113299.log.gz 2023-10-16T13:50:09.426 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.191559.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.191559.log.gz 2023-10-16T13:50:09.426 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.186909.log 2023-10-16T13:50:09.427 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.185842.log 2023-10-16T13:50:09.427 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.186909.log: /var/log/ceph/ceph-client.admin.185842.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.186909.log.gz 2023-10-16T13:50:09.427 INFO:teuthology.orchestra.run.smithi016.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.185842.log.gz 2023-10-16T13:50:09.428 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114517.log 2023-10-16T13:50:09.428 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112602.log 2023-10-16T13:50:09.428 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.114517.log: /var/log/ceph/ceph-client.admin.112602.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.114517.log.gz 2023-10-16T13:50:09.429 INFO:teuthology.orchestra.run.smithi016.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.112602.log.gz 2023-10-16T13:50:09.429 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164485.log 2023-10-16T13:50:09.429 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124128.log 2023-10-16T13:50:09.430 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.164485.log: /var/log/ceph/ceph-client.admin.124128.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.164485.log.gz 2023-10-16T13:50:09.430 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.124128.log.gz 2023-10-16T13:50:09.431 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112976.log 2023-10-16T13:50:09.431 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135499.log 2023-10-16T13:50:09.431 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.112976.log: /var/log/ceph/ceph-client.admin.135499.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.112976.log.gz 2023-10-16T13:50:09.432 INFO:teuthology.orchestra.run.smithi016.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.135499.log.gz 2023-10-16T13:50:09.432 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161654.log 2023-10-16T13:50:09.432 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115923.log 2023-10-16T13:50:09.433 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.161654.log: /var/log/ceph/ceph-client.admin.115923.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.161654.log.gz 2023-10-16T13:50:09.433 INFO:teuthology.orchestra.run.smithi016.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.115923.log.gz 2023-10-16T13:50:09.433 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.178784.log 2023-10-16T13:50:09.434 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171204.log 2023-10-16T13:50:09.434 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.178784.log: /var/log/ceph/ceph-client.admin.171204.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.178784.log.gz 2023-10-16T13:50:09.435 INFO:teuthology.orchestra.run.smithi016.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.171204.log.gz 2023-10-16T13:50:09.435 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135303.log 2023-10-16T13:50:09.435 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142844.log 2023-10-16T13:50:09.435 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.135303.log: /var/log/ceph/ceph-client.admin.142844.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.135303.log.gz 2023-10-16T13:50:09.436 INFO:teuthology.orchestra.run.smithi016.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.142844.log.gz 2023-10-16T13:50:09.436 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151696.log 2023-10-16T13:50:09.436 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136573.log 2023-10-16T13:50:09.437 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.151696.log: /var/log/ceph/ceph-client.admin.136573.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.151696.log.gz 2023-10-16T13:50:09.437 INFO:teuthology.orchestra.run.smithi016.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.136573.log.gz 2023-10-16T13:50:09.437 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.193226.log 2023-10-16T13:50:09.438 INFO:teuthology.orchestra.run.smithi016.stderr: 93.1% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2023-10-16T13:50:09.438 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164177.log 2023-10-16T13:50:09.438 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.193226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.190160.log 2023-10-16T13:50:09.439 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.164177.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.193226.log.gz 2023-10-16T13:50:09.439 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.190160.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.164177.log.gz 2023-10-16T13:50:09.439 INFO:teuthology.orchestra.run.smithi016.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.190160.log.gz 2023-10-16T13:50:09.440 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113298.log 2023-10-16T13:50:09.440 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120404.log 2023-10-16T13:50:09.440 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.113298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147355.log 2023-10-16T13:50:09.440 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.120404.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.113298.log.gz 2023-10-16T13:50:09.441 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.147355.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.120404.log.gz 2023-10-16T13:50:09.441 INFO:teuthology.orchestra.run.smithi016.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.147355.log.gz 2023-10-16T13:50:09.442 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142798.log 2023-10-16T13:50:09.442 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151187.log 2023-10-16T13:50:09.442 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.142798.log: /var/log/ceph/ceph-client.admin.151187.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.142798.log.gz 2023-10-16T13:50:09.443 INFO:teuthology.orchestra.run.smithi016.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.151187.log.gz 2023-10-16T13:50:09.443 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171718.log 2023-10-16T13:50:09.443 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.178244.log 2023-10-16T13:50:09.444 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.171718.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138026.log 2023-10-16T13:50:09.444 INFO:teuthology.orchestra.run.smithi016.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.171718.log.gz 2023-10-16T13:50:09.444 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.178244.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171448.log 2023-10-16T13:50:09.445 INFO:teuthology.orchestra.run.smithi016.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.178244.log.gz 2023-10-16T13:50:09.445 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.138026.log: /var/log/ceph/ceph-client.admin.171448.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.138026.log.gz 2023-10-16T13:50:09.445 INFO:teuthology.orchestra.run.smithi016.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.171448.log.gz 2023-10-16T13:50:09.445 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.191230.log 2023-10-16T13:50:09.446 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.179953.log 2023-10-16T13:50:09.446 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.191230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113735.log 2023-10-16T13:50:09.446 INFO:teuthology.orchestra.run.smithi016.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.191230.log.gz 2023-10-16T13:50:09.447 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.179953.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165658.log 2023-10-16T13:50:09.447 INFO:teuthology.orchestra.run.smithi016.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.179953.log.gz 2023-10-16T13:50:09.447 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.113735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.191792.log 2023-10-16T13:50:09.448 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.165658.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.113735.log.gz 2023-10-16T13:50:09.448 INFO:teuthology.orchestra.run.smithi016.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.165658.log.gz 2023-10-16T13:50:09.448 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.191792.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.191792.log.gz 2023-10-16T13:50:09.449 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138792.log 2023-10-16T13:50:09.449 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.178148.log 2023-10-16T13:50:09.449 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.138792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115969.log 2023-10-16T13:50:09.450 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.178148.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.138792.log.gz 2023-10-16T13:50:09.450 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.115969.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.178148.log.gz 2023-10-16T13:50:09.450 INFO:teuthology.orchestra.run.smithi016.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.115969.log.gz 2023-10-16T13:50:09.451 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123864.log 2023-10-16T13:50:09.451 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.190461.log 2023-10-16T13:50:09.451 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.123864.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118592.log 2023-10-16T13:50:09.452 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.190461.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.123864.log.gz 2023-10-16T13:50:09.452 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.118592.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.190461.log.gz 2023-10-16T13:50:09.452 INFO:teuthology.orchestra.run.smithi016.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.118592.log.gz 2023-10-16T13:50:09.452 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.120803.log 2023-10-16T13:50:09.453 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162596.log 2023-10-16T13:50:09.453 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.0.120803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.181324.log 2023-10-16T13:50:09.453 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.120803.log.gz 2023-10-16T13:50:09.453 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.162596.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.162596.log.gz 2023-10-16T13:50:09.454 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146617.log 2023-10-16T13:50:09.454 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.181324.log: /var/log/ceph/ceph-client.admin.146617.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.181324.log.gz 2023-10-16T13:50:09.454 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.184250.log 2023-10-16T13:50:09.455 INFO:teuthology.orchestra.run.smithi016.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.146617.log.gz 2023-10-16T13:50:09.455 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166263.log 2023-10-16T13:50:09.455 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.184250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133746.log 2023-10-16T13:50:09.455 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.184250.log.gz 2023-10-16T13:50:09.456 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.166263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.192674.log 2023-10-16T13:50:09.456 INFO:teuthology.orchestra.run.smithi016.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.166263.log.gz 2023-10-16T13:50:09.456 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.133746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155291.log 2023-10-16T13:50:09.456 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.0.192674.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.133746.log.gz 2023-10-16T13:50:09.457 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.192674.log.gz 2023-10-16T13:50:09.457 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.155291.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.155291.log.gz 2023-10-16T13:50:09.457 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134399.log 2023-10-16T13:50:09.457 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.175274.log 2023-10-16T13:50:09.458 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.134399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.183970.log 2023-10-16T13:50:09.458 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.175274.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.134399.log.gz 2023-10-16T13:50:09.458 INFO:teuthology.orchestra.run.smithi016.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.175274.log.gz 2023-10-16T13:50:09.458 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.183970.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.183970.log.gz 2023-10-16T13:50:09.459 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.180222.log 2023-10-16T13:50:09.459 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165417.log 2023-10-16T13:50:09.459 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.180222.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165750.log 2023-10-16T13:50:09.460 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.165417.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.180222.log.gz 2023-10-16T13:50:09.460 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.165750.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.165417.log.gz 2023-10-16T13:50:09.460 INFO:teuthology.orchestra.run.smithi016.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.165750.log.gz 2023-10-16T13:50:09.461 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.172330.log 2023-10-16T13:50:09.461 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143534.log 2023-10-16T13:50:09.461 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.172330.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129291.log 2023-10-16T13:50:09.462 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.143534.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.143534.log.gz 2023-10-16T13:50:09.462 INFO:teuthology.orchestra.run.smithi016.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.172330.log.gz 2023-10-16T13:50:09.462 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163754.log 2023-10-16T13:50:09.462 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.129291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119052.log 2023-10-16T13:50:09.463 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.163754.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.129291.log.gz 2023-10-16T13:50:09.463 INFO:teuthology.orchestra.run.smithi016.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.163754.log.gz 2023-10-16T13:50:09.463 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147189.log 2023-10-16T13:50:09.464 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.119052.log: /var/log/ceph/ceph-client.admin.147189.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.119052.log.gz 2023-10-16T13:50:09.464 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.147189.log.gz 2023-10-16T13:50:09.464 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148433.log 2023-10-16T13:50:09.464 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.194166.log 2023-10-16T13:50:09.465 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.148433.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139909.log 2023-10-16T13:50:09.465 INFO:teuthology.orchestra.run.smithi016.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.148433.log.gz 2023-10-16T13:50:09.465 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.194166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120312.log 2023-10-16T13:50:09.466 INFO:teuthology.orchestra.run.smithi016.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.194166.log.gz 2023-10-16T13:50:09.466 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.139909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.194267.log 2023-10-16T13:50:09.466 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.120312.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.139909.log.gz 2023-10-16T13:50:09.466 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.194267.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.120312.log.gz 2023-10-16T13:50:09.467 INFO:teuthology.orchestra.run.smithi016.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.194267.log.gz 2023-10-16T13:50:09.467 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119989.log 2023-10-16T13:50:09.467 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.174647.log 2023-10-16T13:50:09.467 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.119989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.185375.log 2023-10-16T13:50:09.468 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.174647.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.119989.log.gz 2023-10-16T13:50:09.468 INFO:teuthology.orchestra.run.smithi016.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.174647.log.gz 2023-10-16T13:50:09.468 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.185375.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.185375.log.gz 2023-10-16T13:50:09.469 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135777.log 2023-10-16T13:50:09.469 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123666.log 2023-10-16T13:50:09.469 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.135777.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144811.log 2023-10-16T13:50:09.469 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.123666.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.135777.log.gz 2023-10-16T13:50:09.470 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.144811.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.123666.log.gz 2023-10-16T13:50:09.470 INFO:teuthology.orchestra.run.smithi016.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.144811.log.gz 2023-10-16T13:50:09.470 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138231.log 2023-10-16T13:50:09.470 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138980.log 2023-10-16T13:50:09.471 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.138231.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141016.log 2023-10-16T13:50:09.471 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.138980.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.138231.log.gz 2023-10-16T13:50:09.471 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.141016.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.138980.log.gz 2023-10-16T13:50:09.472 INFO:teuthology.orchestra.run.smithi016.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.141016.log.gz 2023-10-16T13:50:09.472 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.184742.log 2023-10-16T13:50:09.472 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164223.log 2023-10-16T13:50:09.472 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.184742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135545.log 2023-10-16T13:50:09.473 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.164223.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.184742.log.gz 2023-10-16T13:50:09.473 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.135545.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.164223.log.gz 2023-10-16T13:50:09.474 INFO:teuthology.orchestra.run.smithi016.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.135545.log.gz 2023-10-16T13:50:09.474 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111419.log 2023-10-16T13:50:09.475 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161307.log 2023-10-16T13:50:09.475 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.111419.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.185275.log 2023-10-16T13:50:09.475 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.161307.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.111419.log.gz 2023-10-16T13:50:09.475 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.185275.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.161307.log.gz 2023-10-16T13:50:09.476 INFO:teuthology.orchestra.run.smithi016.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.185275.log.gz 2023-10-16T13:50:09.476 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170387.log 2023-10-16T13:50:09.476 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113228.log 2023-10-16T13:50:09.476 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.173246.log 2023-10-16T13:50:09.477 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.170387.log: /var/log/ceph/ceph-client.admin.113228.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.170387.log.gz 2023-10-16T13:50:09.477 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.173246.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.113228.log.gz 2023-10-16T13:50:09.477 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.173246.log.gz 2023-10-16T13:50:09.478 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120566.log 2023-10-16T13:50:09.478 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112593.log 2023-10-16T13:50:09.478 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.120566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139955.log 2023-10-16T13:50:09.478 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.112593.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.120566.log.gz 2023-10-16T13:50:09.479 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.139955.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.112593.log.gz 2023-10-16T13:50:09.479 INFO:teuthology.orchestra.run.smithi016.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.139955.log.gz 2023-10-16T13:50:09.479 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148765.log 2023-10-16T13:50:09.480 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125845.log 2023-10-16T13:50:09.480 INFO:teuthology.orchestra.run.smithi016.stderr:gzip /var/log/ceph/ceph-client.admin.148765.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.178576.log 2023-10-16T13:50:09.480 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.125845.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.148765.log.gz 2023-10-16T13:50:09.481 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.178576.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.125845.log.gz 2023-10-16T13:50:09.481 INFO:teuthology.orchestra.run.smithi016.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.178576.log.gz 2023-10-16T13:50:09.481 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.176176.log 2023-10-16T13:50:09.482 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112975.log 2023-10-16T13:50:09.482 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 /var/log/ceph/ceph-client.admin.176176.log: --verbose -- /var/log/ceph/ceph-client.admin.137358.log 2023-10-16T13:50:09.482 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.112975.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.176176.log.gz 2023-10-16T13:50:09.483 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.137358.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.112975.log.gz 2023-10-16T13:50:09.483 INFO:teuthology.orchestra.run.smithi016.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.137358.log.gz 2023-10-16T13:50:09.483 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.186725.log 2023-10-16T13:50:09.483 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112970.log 2023-10-16T13:50:09.484 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.186725.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110203.log 2023-10-16T13:50:09.484 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.112970.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.186725.log.gz 2023-10-16T13:50:09.484 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.110203.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.112970.log.gz 2023-10-16T13:50:09.485 INFO:teuthology.orchestra.run.smithi016.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.110203.log.gz 2023-10-16T13:50:09.485 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.191886.log 2023-10-16T13:50:09.485 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140528.log 2023-10-16T13:50:09.485 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 /var/log/ceph/ceph-client.admin.191886.log: --verbose -- /var/log/ceph/ceph-client.admin.180422.log 2023-10-16T13:50:09.486 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.140528.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.191886.log.gz 2023-10-16T13:50:09.486 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.180422.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.140528.log.gz 2023-10-16T13:50:09.486 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.180422.log.gz 2023-10-16T13:50:09.487 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171358.log 2023-10-16T13:50:09.487 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.187461.log 2023-10-16T13:50:09.487 INFO:teuthology.orchestra.run.smithi016.stderr:gzip /var/log/ceph/ceph-client.admin.171358.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.118960.log 2023-10-16T13:50:09.488 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.187461.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.171358.log.gz 2023-10-16T13:50:09.488 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.118960.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.187461.log.gz 2023-10-16T13:50:09.489 INFO:teuthology.orchestra.run.smithi016.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.118960.log.gz 2023-10-16T13:50:09.489 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148983.log 2023-10-16T13:50:09.490 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.190209.log 2023-10-16T13:50:09.490 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148567.log 2023-10-16T13:50:09.491 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.148983.log: /var/log/ceph/ceph-client.admin.190209.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.148983.log.gz 2023-10-16T13:50:09.491 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.148567.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.190209.log.gz 2023-10-16T13:50:09.492 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.148567.log.gz 2023-10-16T13:50:09.492 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.167271.log 2023-10-16T13:50:09.493 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138838.log 2023-10-16T13:50:09.493 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.0.167271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.181370.log 2023-10-16T13:50:09.494 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.167271.log.gz 2023-10-16T13:50:09.494 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.138838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136382.log 2023-10-16T13:50:09.495 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.181370.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.138838.log.gz 2023-10-16T13:50:09.495 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.136382.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.181370.log.gz 2023-10-16T13:50:09.496 INFO:teuthology.orchestra.run.smithi016.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.136382.log.gz 2023-10-16T13:50:09.496 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.176643.log 2023-10-16T13:50:09.497 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.191978.log 2023-10-16T13:50:09.497 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.176643.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.176643.log.gz 2023-10-16T13:50:09.497 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.191978.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.191978.log.gz 2023-10-16T13:50:09.497 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161943.log 2023-10-16T13:50:09.498 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.175946.log 2023-10-16T13:50:09.498 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.161943.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.177851.log 2023-10-16T13:50:09.498 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.175946.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.161943.log.gz 2023-10-16T13:50:09.499 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.177851.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.175946.log.gz 2023-10-16T13:50:09.499 INFO:teuthology.orchestra.run.smithi016.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.177851.log.gz 2023-10-16T13:50:09.499 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.185425.log 2023-10-16T13:50:09.500 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137411.log 2023-10-16T13:50:09.500 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.185425.log: /var/log/ceph/ceph-client.admin.137411.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.185425.log.gz 2023-10-16T13:50:09.500 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.137411.log.gz 2023-10-16T13:50:09.501 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166922.log 2023-10-16T13:50:09.501 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.183397.log 2023-10-16T13:50:09.501 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.166922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.177392.log 2023-10-16T13:50:09.502 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.183397.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.166922.log.gz 2023-10-16T13:50:09.502 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127925.log 2023-10-16T13:50:09.502 INFO:teuthology.orchestra.run.smithi016.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.183397.log.gz 2023-10-16T13:50:09.503 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.177392.log: /var/log/ceph/ceph-client.admin.127925.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.177392.log.gz 2023-10-16T13:50:09.503 INFO:teuthology.orchestra.run.smithi016.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.127925.log.gz 2023-10-16T13:50:09.503 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109923.log 2023-10-16T13:50:09.504 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.191384.log 2023-10-16T13:50:09.504 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.109923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155059.log 2023-10-16T13:50:09.504 INFO:teuthology.orchestra.run.smithi016.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.109923.log.gz 2023-10-16T13:50:09.504 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.191384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.187047.log 2023-10-16T13:50:09.505 INFO:teuthology.orchestra.run.smithi016.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.191384.log.gz 2023-10-16T13:50:09.505 INFO:teuthology.orchestra.run.smithi016.stderr:gzip /var/log/ceph/ceph-client.admin.155059.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.186863.log 2023-10-16T13:50:09.505 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.187047.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.155059.log.gz 2023-10-16T13:50:09.506 INFO:teuthology.orchestra.run.smithi016.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.187047.log.gz 2023-10-16T13:50:09.506 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.186863.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.186863.log.gz 2023-10-16T13:50:09.506 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153363.log 2023-10-16T13:50:09.507 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129245.log 2023-10-16T13:50:09.507 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.153363.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120266.log 2023-10-16T13:50:09.507 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.129245.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.153363.log.gz 2023-10-16T13:50:09.508 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.120266.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.129245.log.gz 2023-10-16T13:50:09.508 INFO:teuthology.orchestra.run.smithi016.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.120266.log.gz 2023-10-16T13:50:09.508 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.195664.log 2023-10-16T13:50:09.508 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155888.log 2023-10-16T13:50:09.509 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.195664.log: /var/log/ceph/ceph-client.admin.155888.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.195664.log.gz 2023-10-16T13:50:09.509 INFO:teuthology.orchestra.run.smithi016.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.155888.log.gz 2023-10-16T13:50:09.509 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135143.log 2023-10-16T13:50:09.510 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-10-16T13:50:09.510 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.135143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129527.log 2023-10-16T13:50:09.511 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.135143.log.gz 2023-10-16T13:50:09.511 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.191698.log 2023-10-16T13:50:09.511 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.129527.log: /var/log/ceph/ceph-client.admin.191698.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.129527.log.gz 2023-10-16T13:50:09.511 INFO:teuthology.orchestra.run.smithi016.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.191698.log.gz 2023-10-16T13:50:09.512 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.190862.log 2023-10-16T13:50:09.512 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153513.log 2023-10-16T13:50:09.512 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.190862.log: /var/log/ceph/ceph-client.admin.153513.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.190862.log.gz 2023-10-16T13:50:09.513 INFO:teuthology.orchestra.run.smithi016.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.153513.log.gz 2023-10-16T13:50:09.513 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.188255.log 2023-10-16T13:50:09.513 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139496.log 2023-10-16T13:50:09.514 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.188255.log: /var/log/ceph/ceph-client.admin.139496.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.188255.log.gz 2023-10-16T13:50:09.514 INFO:teuthology.orchestra.run.smithi016.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.139496.log.gz 2023-10-16T13:50:09.514 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.186214.log 2023-10-16T13:50:09.514 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.180832.log 2023-10-16T13:50:09.515 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.186214.log: /var/log/ceph/ceph-client.admin.180832.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.186214.log.gz 2023-10-16T13:50:09.515 INFO:teuthology.orchestra.run.smithi016.stderr: 89.0%gzip -5 -- replaced with /var/log/ceph/ceph-client.admin.180832.log.gz--verbose 2023-10-16T13:50:09.515 INFO:teuthology.orchestra.run.smithi016.stderr:-- /var/log/ceph/ceph-client.admin.117128.log 2023-10-16T13:50:09.516 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112599.log 2023-10-16T13:50:09.516 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.117128.log: /var/log/ceph/ceph-client.admin.112599.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.117128.log.gz 2023-10-16T13:50:09.516 INFO:teuthology.orchestra.run.smithi016.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.112599.log.gz 2023-10-16T13:50:09.516 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123798.log 2023-10-16T13:50:09.517 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.193617.log 2023-10-16T13:50:09.517 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.123798.log: /var/log/ceph/ceph-client.admin.193617.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.123798.log.gz 2023-10-16T13:50:09.517 INFO:teuthology.orchestra.run.smithi016.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.193617.log.gz 2023-10-16T13:50:09.517 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130901.log 2023-10-16T13:50:09.518 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111852.log 2023-10-16T13:50:09.518 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.130901.log: /var/log/ceph/ceph-client.admin.111852.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.130901.log.gz 2023-10-16T13:50:09.518 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.111852.log.gz 2023-10-16T13:50:09.519 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.176782.log 2023-10-16T13:50:09.519 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163360.log 2023-10-16T13:50:09.519 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.176782.log: /var/log/ceph/ceph-client.admin.163360.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.176782.log.gz 2023-10-16T13:50:09.520 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.163360.log.gz 2023-10-16T13:50:09.520 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.177964.log 2023-10-16T13:50:09.520 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.147079.log 2023-10-16T13:50:09.521 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.177964.log: /var/log/ceph/ceph-client.0.147079.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.177964.log.gz 2023-10-16T13:50:09.521 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.147079.log.gz 2023-10-16T13:50:09.521 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156365.log 2023-10-16T13:50:09.521 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129987.log 2023-10-16T13:50:09.522 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.156365.log: /var/log/ceph/ceph-client.admin.129987.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.156365.log.gz 2023-10-16T13:50:09.522 INFO:teuthology.orchestra.run.smithi016.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.129987.log.gz 2023-10-16T13:50:09.522 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.175477.log 2023-10-16T13:50:09.523 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.114977.log 2023-10-16T13:50:09.523 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.175477.log: /var/log/ceph/ceph-client.0.114977.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.175477.log.gz 2023-10-16T13:50:09.523 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.114977.log.gz 2023-10-16T13:50:09.524 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.175427.log 2023-10-16T13:50:09.524 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.181849.log 2023-10-16T13:50:09.524 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.175427.log: /var/log/ceph/ceph-client.admin.181849.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.175427.log.gz 2023-10-16T13:50:09.524 INFO:teuthology.orchestra.run.smithi016.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.181849.log.gz 2023-10-16T13:50:09.525 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126763.log 2023-10-16T13:50:09.525 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.181619.log 2023-10-16T13:50:09.525 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.126763.log: /var/log/ceph/ceph-client.admin.181619.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.126763.log.gz 2023-10-16T13:50:09.526 INFO:teuthology.orchestra.run.smithi016.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.181619.log.gz 2023-10-16T13:50:09.526 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163818.log 2023-10-16T13:50:09.526 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.187751.log 2023-10-16T13:50:09.527 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.163818.log: /var/log/ceph/ceph-client.admin.187751.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.163818.log.gz 2023-10-16T13:50:09.527 INFO:teuthology.orchestra.run.smithi016.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.187751.log.gz 2023-10-16T13:50:09.527 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153463.log 2023-10-16T13:50:09.527 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.195296.log 2023-10-16T13:50:09.528 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.153463.log: /var/log/ceph/ceph-client.admin.195296.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.153463.log.gz 2023-10-16T13:50:09.528 INFO:teuthology.orchestra.run.smithi016.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.195296.log.gz 2023-10-16T13:50:09.528 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166646.log 2023-10-16T13:50:09.529 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123554.log 2023-10-16T13:50:09.529 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.166646.log: /var/log/ceph/ceph-client.admin.123554.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.166646.log.gz 2023-10-16T13:50:09.529 INFO:teuthology.orchestra.run.smithi016.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.123554.log.gz 2023-10-16T13:50:09.530 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162688.log 2023-10-16T13:50:09.530 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.183022.log 2023-10-16T13:50:09.530 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.162688.log: /var/log/ceph/ceph-client.admin.183022.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.162688.log.gz 2023-10-16T13:50:09.531 INFO:teuthology.orchestra.run.smithi016.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.183022.log.gz 2023-10-16T13:50:09.531 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129619.log 2023-10-16T13:50:09.531 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124260.log 2023-10-16T13:50:09.531 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.129619.log: /var/log/ceph/ceph-client.admin.124260.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.129619.log.gz 2023-10-16T13:50:09.532 INFO:teuthology.orchestra.run.smithi016.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.124260.log.gz 2023-10-16T13:50:09.532 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.180786.log 2023-10-16T13:50:09.532 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.195848.log 2023-10-16T13:50:09.533 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.180786.log: /var/log/ceph/ceph-client.admin.195848.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.180786.log.gz 2023-10-16T13:50:09.533 INFO:teuthology.orchestra.run.smithi016.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.195848.log.gz 2023-10-16T13:50:09.533 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155245.log 2023-10-16T13:50:09.533 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133798.log 2023-10-16T13:50:09.534 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.155245.log: /var/log/ceph/ceph-client.admin.133798.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.155245.log.gz 2023-10-16T13:50:09.534 INFO:teuthology.orchestra.run.smithi016.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.133798.log.gz 2023-10-16T13:50:09.534 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127561.log 2023-10-16T13:50:09.535 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.176468.log 2023-10-16T13:50:09.535 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.127561.log: /var/log/ceph/ceph-client.admin.176468.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.127561.log.gz 2023-10-16T13:50:09.535 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.176468.log.gz 2023-10-16T13:50:09.535 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.194961.log 2023-10-16T13:50:09.536 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141216.log 2023-10-16T13:50:09.536 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.194961.log: /var/log/ceph/ceph-client.admin.141216.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.194961.log.gz 2023-10-16T13:50:09.536 INFO:teuthology.orchestra.run.smithi016.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.141216.log.gz 2023-10-16T13:50:09.536 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122207.log 2023-10-16T13:50:09.537 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.191605.log 2023-10-16T13:50:09.537 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.122207.log: /var/log/ceph/ceph-client.admin.191605.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.122207.log.gz 2023-10-16T13:50:09.537 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.191605.log.gz 2023-10-16T13:50:09.537 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.172981.log 2023-10-16T13:50:09.538 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112601.log 2023-10-16T13:50:09.538 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.172981.log: /var/log/ceph/ceph-client.admin.112601.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.172981.log.gz 2023-10-16T13:50:09.538 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116714.log 2023-10-16T13:50:09.538 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.112601.log.gz 2023-10-16T13:50:09.539 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126457.log 2023-10-16T13:50:09.539 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.116714.log: /var/log/ceph/ceph-client.admin.126457.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.116714.log.gz 2023-10-16T13:50:09.539 INFO:teuthology.orchestra.run.smithi016.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.126457.log.gz 2023-10-16T13:50:09.539 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109397.log 2023-10-16T13:50:09.540 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117393.log 2023-10-16T13:50:09.540 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.109397.log: /var/log/ceph/ceph-client.admin.117393.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.109397.log.gz 2023-10-16T13:50:09.540 INFO:teuthology.orchestra.run.smithi016.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.117393.log.gz 2023-10-16T13:50:09.540 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.176689.log 2023-10-16T13:50:09.541 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149546.log 2023-10-16T13:50:09.541 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.176689.log: /var/log/ceph/ceph-client.admin.149546.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.176689.log.gz 2023-10-16T13:50:09.541 INFO:teuthology.orchestra.run.smithi016.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.149546.log.gz 2023-10-16T13:50:09.541 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171868.log 2023-10-16T13:50:09.542 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.182884.log 2023-10-16T13:50:09.542 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.171868.log: /var/log/ceph/ceph-client.admin.182884.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.171868.log.gz 2023-10-16T13:50:09.542 INFO:teuthology.orchestra.run.smithi016.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.182884.log.gz 2023-10-16T13:50:09.543 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110295.log 2023-10-16T13:50:09.543 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150649.log 2023-10-16T13:50:09.543 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.110295.log: /var/log/ceph/ceph-client.admin.150649.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.110295.log.gz 2023-10-16T13:50:09.543 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.150649.log.gz 2023-10-16T13:50:09.544 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.182409.log 2023-10-16T13:50:09.544 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.184296.log 2023-10-16T13:50:09.544 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.182409.log: /var/log/ceph/ceph-client.admin.184296.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.182409.log.gz 2023-10-16T13:50:09.544 INFO:teuthology.orchestra.run.smithi016.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.184296.log.gz 2023-10-16T13:50:09.545 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.193525.log 2023-10-16T13:50:09.545 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.193318.log 2023-10-16T13:50:09.545 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.193525.log: /var/log/ceph/ceph-client.admin.193318.log: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.193525.log.gz 2023-10-16T13:50:09.545 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.193318.log.gz 2023-10-16T13:50:09.546 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.191138.log 2023-10-16T13:50:09.546 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141166.log 2023-10-16T13:50:09.546 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.191138.log: /var/log/ceph/ceph-client.admin.141166.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.191138.log.gz 2023-10-16T13:50:09.546 INFO:teuthology.orchestra.run.smithi016.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.141166.log.gz 2023-10-16T13:50:09.547 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118316.log 2023-10-16T13:50:09.547 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.183634.log 2023-10-16T13:50:09.547 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.118316.log: /var/log/ceph/ceph-client.0.183634.log: 0.0% 89.0% -- replaced with /var/log/ceph/ceph-client.0.183634.log.gz 2023-10-16T13:50:09.548 INFO:teuthology.orchestra.run.smithi016.stderr: -- replaced with /var/log/ceph/ceph-client.admin.118316.log.gz 2023-10-16T13:50:09.548 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153261.log 2023-10-16T13:50:09.548 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.181941.log 2023-10-16T13:50:09.548 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.153261.log: /var/log/ceph/ceph-client.admin.181941.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.153261.log.gz 2023-10-16T13:50:09.549 INFO:teuthology.orchestra.run.smithi016.stderr:gzip 89.3%-5 -- replaced with /var/log/ceph/ceph-client.admin.181941.log.gz--verbose 2023-10-16T13:50:09.549 INFO:teuthology.orchestra.run.smithi016.stderr:-- /var/log/ceph/ceph-client.admin.128963.log 2023-10-16T13:50:09.549 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122019.log 2023-10-16T13:50:09.549 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.128963.log: /var/log/ceph/ceph-client.admin.122019.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.128963.log.gz 2023-10-16T13:50:09.550 INFO:teuthology.orchestra.run.smithi016.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.122019.log.gz 2023-10-16T13:50:09.550 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150925.log 2023-10-16T13:50:09.550 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.195710.log 2023-10-16T13:50:09.551 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.150925.log: /var/log/ceph/ceph-client.admin.195710.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.150925.log.gz 2023-10-16T13:50:09.551 INFO:teuthology.orchestra.run.smithi016.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.195710.log.gz 2023-10-16T13:50:09.551 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.178830.log 2023-10-16T13:50:09.551 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118546.log 2023-10-16T13:50:09.552 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.178830.log: /var/log/ceph/ceph-client.admin.118546.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.178830.log.gz 2023-10-16T13:50:09.552 INFO:teuthology.orchestra.run.smithi016.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.118546.log.gz 2023-10-16T13:50:09.552 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125539.log 2023-10-16T13:50:09.552 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163198.log 2023-10-16T13:50:09.553 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.125539.log: /var/log/ceph/ceph-client.admin.163198.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.125539.log.gz 2023-10-16T13:50:09.553 INFO:teuthology.orchestra.run.smithi016.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.163198.log.gz 2023-10-16T13:50:09.553 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165888.log 2023-10-16T13:50:09.554 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.175526.log 2023-10-16T13:50:09.554 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.165888.log: /var/log/ceph/ceph-client.admin.175526.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.165888.log.gz 2023-10-16T13:50:09.554 INFO:teuthology.orchestra.run.smithi016.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.175526.log.gz 2023-10-16T13:50:09.554 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.180740.log 2023-10-16T13:50:09.555 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137259.log 2023-10-16T13:50:09.555 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.180740.log: /var/log/ceph/ceph-client.admin.137259.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.180740.log.gz 2023-10-16T13:50:09.555 INFO:teuthology.orchestra.run.smithi016.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.137259.log.gz 2023-10-16T13:50:09.555 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151622.log 2023-10-16T13:50:09.556 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123278.log 2023-10-16T13:50:09.556 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.151622.log: /var/log/ceph/ceph-client.admin.123278.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.151622.log.gz 2023-10-16T13:50:09.556 INFO:teuthology.orchestra.run.smithi016.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.123278.log.gz 2023-10-16T13:50:09.556 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.178196.log 2023-10-16T13:50:09.557 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142614.log 2023-10-16T13:50:09.557 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.178196.log: /var/log/ceph/ceph-client.admin.142614.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.178196.log.gz 2023-10-16T13:50:09.557 INFO:teuthology.orchestra.run.smithi016.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.142614.log.gz 2023-10-16T13:50:09.557 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128641.log 2023-10-16T13:50:09.558 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121471.log 2023-10-16T13:50:09.558 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.128641.log: /var/log/ceph/ceph-client.admin.121471.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.128641.log.gz 2023-10-16T13:50:09.558 INFO:teuthology.orchestra.run.smithi016.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.121471.log.gz 2023-10-16T13:50:09.559 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137812.log 2023-10-16T13:50:09.559 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.182502.log 2023-10-16T13:50:09.559 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.137812.log: /var/log/ceph/ceph-client.admin.182502.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.137812.log.gz 2023-10-16T13:50:09.559 INFO:teuthology.orchestra.run.smithi016.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.182502.log.gz 2023-10-16T13:50:09.559 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138369.log 2023-10-16T13:50:09.560 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117803.log 2023-10-16T13:50:09.560 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.138369.log: /var/log/ceph/ceph-client.admin.117803.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.138369.log.gz 2023-10-16T13:50:09.560 INFO:teuthology.orchestra.run.smithi016.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.117803.log.gz 2023-10-16T13:50:09.561 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.178885.log 2023-10-16T13:50:09.561 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.194116.log 2023-10-16T13:50:09.561 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.178885.log: /var/log/ceph/ceph-client.admin.194116.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.178885.log.gz 2023-10-16T13:50:09.561 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.194116.log.gz 2023-10-16T13:50:09.562 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.189944.log 2023-10-16T13:50:09.562 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134099.log 2023-10-16T13:50:09.562 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.189944.log: /var/log/ceph/ceph-client.admin.134099.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.189944.log.gz 2023-10-16T13:50:09.563 INFO:teuthology.orchestra.run.smithi016.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.134099.log.gz 2023-10-16T13:50:09.563 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118044.log 2023-10-16T13:50:09.563 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.181002.log 2023-10-16T13:50:09.563 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.118044.log: /var/log/ceph/ceph-client.admin.181002.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.118044.log.gz 2023-10-16T13:50:09.564 INFO:teuthology.orchestra.run.smithi016.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.181002.log.gz 2023-10-16T13:50:09.564 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134250.log 2023-10-16T13:50:09.564 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.183168.log 2023-10-16T13:50:09.564 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.134250.log: /var/log/ceph/ceph-client.admin.183168.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.134250.log.gz 2023-10-16T13:50:09.565 INFO:teuthology.orchestra.run.smithi016.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.183168.log.gz 2023-10-16T13:50:09.565 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112171.log 2023-10-16T13:50:09.566 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118730.log 2023-10-16T13:50:09.566 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.112171.log: /var/log/ceph/ceph-client.admin.118730.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.112171.log.gz 2023-10-16T13:50:09.566 INFO:teuthology.orchestra.run.smithi016.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.118730.log.gz 2023-10-16T13:50:09.567 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156008.log 2023-10-16T13:50:09.567 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109729.log 2023-10-16T13:50:09.567 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.156008.log: /var/log/ceph/ceph-client.admin.109729.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.156008.log.gz 2023-10-16T13:50:09.567 INFO:teuthology.orchestra.run.smithi016.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.109729.log.gz 2023-10-16T13:50:09.568 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.184112.log 2023-10-16T13:50:09.568 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.188069.log 2023-10-16T13:50:09.568 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.184112.log: /var/log/ceph/ceph-client.admin.188069.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.184112.log.gz 2023-10-16T13:50:09.568 INFO:teuthology.orchestra.run.smithi016.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.188069.log.gz 2023-10-16T13:50:09.569 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.172790.log 2023-10-16T13:50:09.569 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.181803.log 2023-10-16T13:50:09.569 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.172790.log: /var/log/ceph/ceph-client.admin.181803.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.172790.log.gz 2023-10-16T13:50:09.570 INFO:teuthology.orchestra.run.smithi016.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.181803.log.gz 2023-10-16T13:50:09.570 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123211.log 2023-10-16T13:50:09.570 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149184.log 2023-10-16T13:50:09.571 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.123211.log: /var/log/ceph/ceph-client.admin.149184.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.123211.log.gz 2023-10-16T13:50:09.571 INFO:teuthology.orchestra.run.smithi016.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.149184.log.gz 2023-10-16T13:50:09.573 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139566.log 2023-10-16T13:50:09.573 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103820.log 2023-10-16T13:50:09.573 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.139566.log: /var/log/ceph/ceph-client.admin.103820.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.139566.log.gz 2023-10-16T13:50:09.574 INFO:teuthology.orchestra.run.smithi016.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.103820.log.gz 2023-10-16T13:50:09.574 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133899.log 2023-10-16T13:50:09.574 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.172054.log 2023-10-16T13:50:09.575 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.133899.log: /var/log/ceph/ceph-client.admin.172054.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.133899.log.gz 2023-10-16T13:50:09.575 INFO:teuthology.orchestra.run.smithi016.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.172054.log.gz 2023-10-16T13:50:09.575 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110157.log 2023-10-16T13:50:09.575 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160889.log 2023-10-16T13:50:09.576 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.110157.log: /var/log/ceph/ceph-client.admin.160889.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.110157.log.gz 2023-10-16T13:50:09.576 INFO:teuthology.orchestra.run.smithi016.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.160889.log.gz 2023-10-16T13:50:09.576 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171766.log 2023-10-16T13:50:09.576 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111111.log 2023-10-16T13:50:09.577 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.171766.log: /var/log/ceph/ceph-client.admin.111111.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.171766.log.gz 2023-10-16T13:50:09.577 INFO:teuthology.orchestra.run.smithi016.stderr: 93.9% -- replaced with /var/log/ceph/ceph-client.admin.111111.log.gz 2023-10-16T13:50:09.577 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129479.log 2023-10-16T13:50:09.577 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.195434.log 2023-10-16T13:50:09.578 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.129479.log: /var/log/ceph/ceph-client.admin.195434.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.129479.log.gz 2023-10-16T13:50:09.578 INFO:teuthology.orchestra.run.smithi016.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.195434.log.gz 2023-10-16T13:50:09.578 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.183786.log 2023-10-16T13:50:09.579 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.192367.log 2023-10-16T13:50:09.579 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.183786.log: /var/log/ceph/ceph-client.admin.192367.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.183786.log.gz 2023-10-16T13:50:09.579 INFO:teuthology.orchestra.run.smithi016.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.192367.log.gz 2023-10-16T13:50:09.579 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112263.log 2023-10-16T13:50:09.580 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142706.log 2023-10-16T13:50:09.580 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.112263.log: /var/log/ceph/ceph-client.admin.142706.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.112263.log.gz 2023-10-16T13:50:09.580 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.142706.log.gz 2023-10-16T13:50:09.580 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165325.log 2023-10-16T13:50:09.581 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167555.log 2023-10-16T13:50:09.581 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.165325.log: /var/log/ceph/ceph-client.admin.167555.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.165325.log.gz 2023-10-16T13:50:09.581 INFO:teuthology.orchestra.run.smithi016.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.167555.log.gz 2023-10-16T13:50:09.581 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.192437.log 2023-10-16T13:50:09.582 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122301.log 2023-10-16T13:50:09.582 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.192437.log: /var/log/ceph/ceph-client.admin.122301.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.192437.log.gz 2023-10-16T13:50:09.583 INFO:teuthology.orchestra.run.smithi016.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.122301.log.gz 2023-10-16T13:50:09.583 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113229.log 2023-10-16T13:50:09.583 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161718.log 2023-10-16T13:50:09.583 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.113229.log: /var/log/ceph/ceph-client.admin.161718.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.113229.log.gz 2023-10-16T13:50:09.584 INFO:teuthology.orchestra.run.smithi016.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.161718.log.gz 2023-10-16T13:50:09.584 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145445.log 2023-10-16T13:50:09.584 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.186541.log 2023-10-16T13:50:09.585 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.145445.log: /var/log/ceph/ceph-client.admin.186541.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.145445.log.gz 2023-10-16T13:50:09.585 INFO:teuthology.orchestra.run.smithi016.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.186541.log.gz 2023-10-16T13:50:09.585 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135012.log 2023-10-16T13:50:09.586 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129941.log 2023-10-16T13:50:09.586 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.135012.log: /var/log/ceph/ceph-client.admin.129941.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.135012.log.gz 2023-10-16T13:50:09.586 INFO:teuthology.orchestra.run.smithi016.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.129941.log.gz 2023-10-16T13:50:09.587 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119663.log 2023-10-16T13:50:09.587 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.175626.log 2023-10-16T13:50:09.587 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.119663.log: /var/log/ceph/ceph-client.admin.175626.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.119663.log.gz 2023-10-16T13:50:09.587 INFO:teuthology.orchestra.run.smithi016.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.175626.log.gz 2023-10-16T13:50:09.588 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121075.log 2023-10-16T13:50:09.588 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150301.log 2023-10-16T13:50:09.588 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.121075.log: /var/log/ceph/ceph-client.admin.150301.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.121075.log.gz 2023-10-16T13:50:09.589 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.150301.log.gz 2023-10-16T13:50:09.589 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.114969.log 2023-10-16T13:50:09.647 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.181665.log 2023-10-16T13:50:09.647 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.0.114969.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.114969.log.gz 2023-10-16T13:50:09.647 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.181665.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.181665.log.gz 2023-10-16T13:50:09.648 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144660.log 2023-10-16T13:50:09.648 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.187185.log 2023-10-16T13:50:09.648 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.144660.log: /var/log/ceph/ceph-client.admin.187185.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.144660.log.gz 2023-10-16T13:50:09.649 INFO:teuthology.orchestra.run.smithi016.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.187185.log.gz 2023-10-16T13:50:09.649 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.184342.log 2023-10-16T13:50:09.649 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115877.log 2023-10-16T13:50:09.650 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.184342.log: /var/log/ceph/ceph-client.admin.115877.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.184342.log.gz 2023-10-16T13:50:09.650 INFO:teuthology.orchestra.run.smithi016.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.115877.log.gz 2023-10-16T13:50:09.650 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149482.log 2023-10-16T13:50:09.651 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136336.log 2023-10-16T13:50:09.651 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.149482.log: /var/log/ceph/ceph-client.admin.136336.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.149482.log.gz 2023-10-16T13:50:09.651 INFO:teuthology.orchestra.run.smithi016.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.136336.log.gz 2023-10-16T13:50:09.652 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116384.log 2023-10-16T13:50:09.652 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.180169.log 2023-10-16T13:50:09.652 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.116384.log: /var/log/ceph/ceph-client.admin.180169.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.116384.log.gz 2023-10-16T13:50:09.653 INFO:teuthology.orchestra.run.smithi016.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.180169.log.gz 2023-10-16T13:50:09.653 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119941.log 2023-10-16T13:50:09.653 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.193375.log 2023-10-16T13:50:09.653 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.119941.log: /var/log/ceph/ceph-client.admin.193375.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.119941.log.gz 2023-10-16T13:50:09.654 INFO:teuthology.orchestra.run.smithi016.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.193375.log.gz 2023-10-16T13:50:09.654 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141627.log 2023-10-16T13:50:09.654 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140477.log 2023-10-16T13:50:09.655 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.141627.log: /var/log/ceph/ceph-client.admin.140477.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.141627.log.gz 2023-10-16T13:50:09.655 INFO:teuthology.orchestra.run.smithi016.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.140477.log.gz 2023-10-16T13:50:09.655 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133381.log 2023-10-16T13:50:09.655 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112966.log 2023-10-16T13:50:09.656 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.133381.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.133381.log.gz 2023-10-16T13:50:09.656 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.112966.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.112966.log.gz 2023-10-16T13:50:09.656 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139450.log 2023-10-16T13:50:09.657 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128595.log 2023-10-16T13:50:09.657 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.139450.log: /var/log/ceph/ceph-client.admin.128595.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.139450.log.gz 2023-10-16T13:50:09.657 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.128595.log.gz 2023-10-16T13:50:09.657 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.188347.log 2023-10-16T13:50:09.658 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.193570.log 2023-10-16T13:50:09.658 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.188347.log: /var/log/ceph/ceph-client.admin.193570.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.188347.log.gz 2023-10-16T13:50:09.658 INFO:teuthology.orchestra.run.smithi016.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.193570.log.gz 2023-10-16T13:50:09.659 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112965.log 2023-10-16T13:50:09.659 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153212.log 2023-10-16T13:50:09.659 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.112965.log: /var/log/ceph/ceph-client.admin.153212.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.112965.log.gz 2023-10-16T13:50:09.660 INFO:teuthology.orchestra.run.smithi016.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.153212.log.gz 2023-10-16T13:50:09.660 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121905.log 2023-10-16T13:50:09.660 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.179620.log 2023-10-16T13:50:09.661 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.121905.log: /var/log/ceph/ceph-client.admin.179620.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.121905.log.gz 2023-10-16T13:50:09.661 INFO:teuthology.orchestra.run.smithi016.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.179620.log.gz 2023-10-16T13:50:09.661 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152072.log 2023-10-16T13:50:09.662 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.175059.log 2023-10-16T13:50:09.662 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.152072.log: /var/log/ceph/ceph-client.admin.175059.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.152072.log.gz 2023-10-16T13:50:09.662 INFO:teuthology.orchestra.run.smithi016.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.175059.log.gz 2023-10-16T13:50:09.662 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.185326.log 2023-10-16T13:50:09.663 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129385.log 2023-10-16T13:50:09.663 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.185326.log: /var/log/ceph/ceph-client.admin.129385.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.185326.log.gz 2023-10-16T13:50:09.663 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.129385.log.gz 2023-10-16T13:50:09.664 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145537.log 2023-10-16T13:50:09.664 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146339.log 2023-10-16T13:50:09.664 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.145537.log: /var/log/ceph/ceph-client.admin.146339.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.145537.log.gz 2023-10-16T13:50:09.664 INFO:teuthology.orchestra.run.smithi016.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.146339.log.gz 2023-10-16T13:50:09.665 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169634.log 2023-10-16T13:50:09.665 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.175577.log 2023-10-16T13:50:09.665 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.169634.log: /var/log/ceph/ceph-client.admin.175577.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.169634.log.gz 2023-10-16T13:50:09.666 INFO:teuthology.orchestra.run.smithi016.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.175577.log.gz 2023-10-16T13:50:09.666 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167381.log 2023-10-16T13:50:09.666 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133120.log 2023-10-16T13:50:09.666 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.167381.log: /var/log/ceph/ceph-client.admin.133120.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.167381.log.gz 2023-10-16T13:50:09.667 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.133120.log.gz 2023-10-16T13:50:09.667 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161608.log 2023-10-16T13:50:09.667 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153935.log 2023-10-16T13:50:09.668 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.161608.log: /var/log/ceph/ceph-client.admin.153935.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.161608.log.gz 2023-10-16T13:50:09.668 INFO:teuthology.orchestra.run.smithi016.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.153935.log.gz 2023-10-16T13:50:09.668 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.192137.log 2023-10-16T13:50:09.668 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128733.log 2023-10-16T13:50:09.669 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.192137.log: /var/log/ceph/ceph-client.admin.128733.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.192137.log.gz 2023-10-16T13:50:09.669 INFO:teuthology.orchestra.run.smithi016.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.128733.log.gz 2023-10-16T13:50:09.669 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135593.log 2023-10-16T13:50:09.670 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127660.log 2023-10-16T13:50:09.670 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.135593.log: /var/log/ceph/ceph-client.admin.127660.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.135593.log.gz 2023-10-16T13:50:09.670 INFO:teuthology.orchestra.run.smithi016.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.127660.log.gz 2023-10-16T13:50:09.671 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.194067.log 2023-10-16T13:50:09.671 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153878.log 2023-10-16T13:50:09.671 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.194067.log: /var/log/ceph/ceph-client.admin.153878.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.194067.log.gz 2023-10-16T13:50:09.671 INFO:teuthology.orchestra.run.smithi016.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.153878.log.gz 2023-10-16T13:50:09.672 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122348.log 2023-10-16T13:50:09.672 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136502.log 2023-10-16T13:50:09.672 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.122348.log: /var/log/ceph/ceph-client.admin.136502.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.122348.log.gz 2023-10-16T13:50:09.673 INFO:teuthology.orchestra.run.smithi016.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.136502.log.gz 2023-10-16T13:50:09.673 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116668.log 2023-10-16T13:50:09.673 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.188539.log 2023-10-16T13:50:09.673 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.116668.log: /var/log/ceph/ceph-client.admin.188539.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.116668.log.gz 2023-10-16T13:50:09.674 INFO:teuthology.orchestra.run.smithi016.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.188539.log.gz 2023-10-16T13:50:09.674 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169228.log 2023-10-16T13:50:09.674 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.185473.log 2023-10-16T13:50:09.675 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.169228.log: /var/log/ceph/ceph-client.admin.185473.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.169228.log.gz 2023-10-16T13:50:09.675 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.185473.log.gz 2023-10-16T13:50:09.675 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111668.log 2023-10-16T13:50:09.676 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149135.log 2023-10-16T13:50:09.676 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.111668.log: /var/log/ceph/ceph-client.admin.149135.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.111668.log.gz 2023-10-16T13:50:09.677 INFO:teuthology.orchestra.run.smithi016.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.149135.log.gz 2023-10-16T13:50:09.677 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.180008.log 2023-10-16T13:50:09.677 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112608.log 2023-10-16T13:50:09.677 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.180008.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.180008.log.gz 2023-10-16T13:50:09.678 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.187928.log 2023-10-16T13:50:09.678 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.112608.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.112608.log.gz 2023-10-16T13:50:09.678 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.187928.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.187928.log.gz 2023-10-16T13:50:09.679 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142066.log 2023-10-16T13:50:09.679 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.173199.log 2023-10-16T13:50:09.679 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.142066.log: /var/log/ceph/ceph-client.admin.173199.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.142066.log.gz 2023-10-16T13:50:09.680 INFO:teuthology.orchestra.run.smithi016.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.173199.log.gz 2023-10-16T13:50:09.680 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145873.log 2023-10-16T13:50:09.680 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165474.log 2023-10-16T13:50:09.681 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.145873.log: /var/log/ceph/ceph-client.admin.165474.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.145873.log.gz 2023-10-16T13:50:09.681 INFO:teuthology.orchestra.run.smithi016.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.165474.log.gz 2023-10-16T13:50:09.681 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123324.log 2023-10-16T13:50:09.682 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111511.log 2023-10-16T13:50:09.682 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.123324.log: /var/log/ceph/ceph-client.admin.111511.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.123324.log.gz 2023-10-16T13:50:09.682 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.111511.log.gz 2023-10-16T13:50:09.683 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.191046.log 2023-10-16T13:50:09.683 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113689.log 2023-10-16T13:50:09.684 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.191046.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.191046.log.gz 2023-10-16T13:50:09.684 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116806.log 2023-10-16T13:50:09.684 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.113689.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.113689.log.gz 2023-10-16T13:50:09.685 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.116806.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.116806.log.gz 2023-10-16T13:50:09.685 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154303.log 2023-10-16T13:50:09.685 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112079.log 2023-10-16T13:50:09.686 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.154303.log: /var/log/ceph/ceph-client.admin.112079.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.154303.log.gz 2023-10-16T13:50:09.686 INFO:teuthology.orchestra.run.smithi016.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.112079.log.gz 2023-10-16T13:50:09.686 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.196311.log 2023-10-16T13:50:09.687 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166452.log 2023-10-16T13:50:09.687 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.196311.log: /var/log/ceph/ceph-client.admin.166452.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.196311.log.gz 2023-10-16T13:50:09.687 INFO:teuthology.orchestra.run.smithi016.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.166452.log.gz 2023-10-16T13:50:09.687 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109591.log 2023-10-16T13:50:09.688 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.187139.log 2023-10-16T13:50:09.688 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.109591.log: /var/log/ceph/ceph-client.admin.187139.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.109591.log.gz 2023-10-16T13:50:09.688 INFO:teuthology.orchestra.run.smithi016.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.187139.log.gz 2023-10-16T13:50:09.688 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123165.log 2023-10-16T13:50:09.689 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117665.log 2023-10-16T13:50:09.689 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.123165.log: /var/log/ceph/ceph-client.admin.117665.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.123165.log.gz 2023-10-16T13:50:09.689 INFO:teuthology.orchestra.run.smithi016.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.117665.log.gz 2023-10-16T13:50:09.690 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144109.log 2023-10-16T13:50:09.690 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161156.log 2023-10-16T13:50:09.690 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.144109.log: /var/log/ceph/ceph-client.admin.161156.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.144109.log.gz 2023-10-16T13:50:09.691 INFO:teuthology.orchestra.run.smithi016.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.161156.log.gz 2023-10-16T13:50:09.691 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162504.log 2023-10-16T13:50:09.691 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.177508.log 2023-10-16T13:50:09.691 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.162504.log: /var/log/ceph/ceph-client.admin.177508.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.162504.log.gz 2023-10-16T13:50:09.692 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2%gzip -5 -- replaced with /var/log/ceph/ceph-client.admin.177508.log.gz--verbose 2023-10-16T13:50:09.692 INFO:teuthology.orchestra.run.smithi016.stderr:-- /var/log/ceph/ceph-client.admin.150555.log 2023-10-16T13:50:09.692 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130683.log 2023-10-16T13:50:09.692 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.150555.log: /var/log/ceph/ceph-client.admin.130683.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.150555.log.gz 2023-10-16T13:50:09.693 INFO:teuthology.orchestra.run.smithi016.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.130683.log.gz 2023-10-16T13:50:09.693 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165996.log 2023-10-16T13:50:09.693 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165612.log 2023-10-16T13:50:09.693 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.165996.log: /var/log/ceph/ceph-client.admin.165612.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.165996.log.gz 2023-10-16T13:50:09.694 INFO:teuthology.orchestra.run.smithi016.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.165612.log.gz 2023-10-16T13:50:09.694 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113227.log 2023-10-16T13:50:09.694 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.178344.log 2023-10-16T13:50:09.695 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.113227.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.113227.log.gz 2023-10-16T13:50:09.695 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.178344.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.178344.log.gz 2023-10-16T13:50:09.695 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134491.log 2023-10-16T13:50:09.696 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112603.log 2023-10-16T13:50:09.696 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.134491.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.134491.log.gz 2023-10-16T13:50:09.696 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.151420.log 2023-10-16T13:50:09.696 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.112603.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.112603.log.gz 2023-10-16T13:50:09.697 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143442.log 2023-10-16T13:50:09.697 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.0.151420.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.151420.log.gz 2023-10-16T13:50:09.697 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.143442.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.143442.log.gz 2023-10-16T13:50:09.697 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138524.log 2023-10-16T13:50:09.698 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.181094.log 2023-10-16T13:50:09.698 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.138524.log: /var/log/ceph/ceph-client.admin.181094.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.138524.log.gz 2023-10-16T13:50:09.698 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.181094.log.gz 2023-10-16T13:50:09.699 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169820.log 2023-10-16T13:50:09.699 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.168511.log 2023-10-16T13:50:09.699 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.169820.log: /var/log/ceph/ceph-client.admin.168511.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.169820.log.gz 2023-10-16T13:50:09.699 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.168511.log.gz 2023-10-16T13:50:09.700 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.195894.log 2023-10-16T13:50:09.700 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129711.log 2023-10-16T13:50:09.700 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.195894.log: /var/log/ceph/ceph-client.admin.129711.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.195894.log.gz 2023-10-16T13:50:09.701 INFO:teuthology.orchestra.run.smithi016.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.129711.log.gz 2023-10-16T13:50:09.701 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154349.log 2023-10-16T13:50:09.701 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.178390.log 2023-10-16T13:50:09.702 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.154349.log: /var/log/ceph/ceph-client.admin.178390.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.154349.log.gz 2023-10-16T13:50:09.702 INFO:teuthology.orchestra.run.smithi016.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.178390.log.gz 2023-10-16T13:50:09.703 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.193901.log 2023-10-16T13:50:09.703 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162173.log 2023-10-16T13:50:09.703 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.193901.log: /var/log/ceph/ceph-client.admin.162173.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.193901.log.gz 2023-10-16T13:50:09.703 INFO:teuthology.orchestra.run.smithi016.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.162173.log.gz 2023-10-16T13:50:09.704 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164821.log 2023-10-16T13:50:09.704 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.181572.log 2023-10-16T13:50:09.704 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.164821.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.164821.log.gz 2023-10-16T13:50:09.705 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.181572.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.181572.log.gz 2023-10-16T13:50:09.705 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127610.log 2023-10-16T13:50:09.705 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129573.log 2023-10-16T13:50:09.705 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.127610.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.127610.log.gz 2023-10-16T13:50:09.706 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.129573.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.129573.log.gz 2023-10-16T13:50:09.706 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.188485.log 2023-10-16T13:50:09.707 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155013.log 2023-10-16T13:50:09.707 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.188485.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.188485.log.gz 2023-10-16T13:50:09.707 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145112.log 2023-10-16T13:50:09.707 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.155013.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.155013.log.gz 2023-10-16T13:50:09.708 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.145112.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.145112.log.gz 2023-10-16T13:50:09.708 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129431.log 2023-10-16T13:50:09.708 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170338.log 2023-10-16T13:50:09.709 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.129431.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.129431.log.gz 2023-10-16T13:50:09.709 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118822.log 2023-10-16T13:50:09.709 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.170338.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.170338.log.gz 2023-10-16T13:50:09.710 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.189994.log 2023-10-16T13:50:09.710 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.118822.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.118822.log.gz 2023-10-16T13:50:09.710 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171045.log 2023-10-16T13:50:09.711 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.189994.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.189994.log.gz 2023-10-16T13:50:09.711 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149972.log 2023-10-16T13:50:09.711 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.171045.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.171045.log.gz 2023-10-16T13:50:09.712 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154257.log 2023-10-16T13:50:09.712 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.149972.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.149972.log.gz 2023-10-16T13:50:09.712 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163152.log 2023-10-16T13:50:09.713 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.154257.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.154257.log.gz 2023-10-16T13:50:09.713 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.163152.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.163152.log.gz 2023-10-16T13:50:09.713 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135351.log 2023-10-16T13:50:09.713 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.192780.log 2023-10-16T13:50:09.714 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.135351.log: /var/log/ceph/ceph-client.admin.192780.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.135351.log.gz 2023-10-16T13:50:09.714 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116015.log 2023-10-16T13:50:09.714 INFO:teuthology.orchestra.run.smithi016.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.192780.log.gz 2023-10-16T13:50:09.715 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109783.log 2023-10-16T13:50:09.715 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.116015.log: /var/log/ceph/ceph-client.admin.109783.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.116015.log.gz 2023-10-16T13:50:09.715 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.109783.log.gz 2023-10-16T13:50:09.715 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.163593.log 2023-10-16T13:50:09.716 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113919.log 2023-10-16T13:50:09.716 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.0.163593.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.163593.log.gz 2023-10-16T13:50:09.716 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.113919.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.113919.log.gz 2023-10-16T13:50:09.717 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165796.log 2023-10-16T13:50:09.717 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161055.log 2023-10-16T13:50:09.717 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.165796.log: /var/log/ceph/ceph-client.admin.161055.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.165796.log.gz 2023-10-16T13:50:09.717 INFO:teuthology.orchestra.run.smithi016.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.161055.log.gz 2023-10-16T13:50:09.718 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137858.log 2023-10-16T13:50:09.718 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142333.log 2023-10-16T13:50:09.718 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.137858.log: /var/log/ceph/ceph-client.admin.142333.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.137858.log.gz 2023-10-16T13:50:09.719 INFO:teuthology.orchestra.run.smithi016.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.142333.log.gz 2023-10-16T13:50:09.719 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.172744.log 2023-10-16T13:50:09.719 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.174960.log 2023-10-16T13:50:09.719 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.172744.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.172744.log.gz 2023-10-16T13:50:09.720 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.174960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.178530.log 2023-10-16T13:50:09.720 INFO:teuthology.orchestra.run.smithi016.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.174960.log.gz 2023-10-16T13:50:09.720 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164434.log 2023-10-16T13:50:09.721 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.178530.log: /var/log/ceph/ceph-client.admin.164434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.176924.log 2023-10-16T13:50:09.721 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.178530.log.gz 2023-10-16T13:50:09.721 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.164434.log.gz 2023-10-16T13:50:09.721 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149880.log 2023-10-16T13:50:09.722 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.176924.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.176924.log.gz 2023-10-16T13:50:09.722 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134922.log 2023-10-16T13:50:09.722 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.149880.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.149880.log.gz 2023-10-16T13:50:09.723 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142890.log 2023-10-16T13:50:09.723 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.134922.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.134922.log.gz 2023-10-16T13:50:09.723 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133579.log 2023-10-16T13:50:09.723 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.142890.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116760.log 2023-10-16T13:50:09.724 INFO:teuthology.orchestra.run.smithi016.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.142890.log.gz 2023-10-16T13:50:09.724 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.133579.log: /var/log/ceph/ceph-client.admin.116760.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.187277.log 2023-10-16T13:50:09.724 INFO:teuthology.orchestra.run.smithi016.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.133579.log.gz 2023-10-16T13:50:09.724 INFO:teuthology.orchestra.run.smithi016.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.116760.log.gz 2023-10-16T13:50:09.725 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112125.log 2023-10-16T13:50:09.725 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.187277.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.187277.log.gz 2023-10-16T13:50:09.725 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171533.log 2023-10-16T13:50:09.726 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.112125.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.112125.log.gz 2023-10-16T13:50:09.726 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117347.log 2023-10-16T13:50:09.726 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.171533.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.171533.log.gz 2023-10-16T13:50:09.727 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135097.log 2023-10-16T13:50:09.727 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.117347.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.117347.log.gz 2023-10-16T13:50:09.727 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.135097.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.135097.log.gz 2023-10-16T13:50:09.727 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.194915.log 2023-10-16T13:50:09.728 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.188393.log 2023-10-16T13:50:09.728 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.194915.log: /var/log/ceph/ceph-client.admin.188393.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.194915.log.gz 2023-10-16T13:50:09.728 INFO:teuthology.orchestra.run.smithi016.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.188393.log.gz 2023-10-16T13:50:09.728 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166217.log 2023-10-16T13:50:09.729 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.180910.log 2023-10-16T13:50:09.729 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.166217.log: /var/log/ceph/ceph-client.admin.180910.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.166217.log.gz 2023-10-16T13:50:09.729 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.180910.log.gz 2023-10-16T13:50:09.730 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.194620.log 2023-10-16T13:50:09.730 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.176038.log 2023-10-16T13:50:09.730 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.194620.log: /var/log/ceph/ceph-client.admin.176038.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.194620.log.gz 2023-10-16T13:50:09.730 INFO:teuthology.orchestra.run.smithi016.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.176038.log.gz 2023-10-16T13:50:09.731 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163244.log 2023-10-16T13:50:09.731 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113345.log 2023-10-16T13:50:09.731 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.163244.log: /var/log/ceph/ceph-client.admin.113345.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.163244.log.gz 2023-10-16T13:50:09.732 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.113345.log.gz 2023-10-16T13:50:09.732 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129161.log 2023-10-16T13:50:09.732 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153411.log 2023-10-16T13:50:09.732 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.129161.log: /var/log/ceph/ceph-client.admin.153411.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.129161.log.gz 2023-10-16T13:50:09.733 INFO:teuthology.orchestra.run.smithi016.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.153411.log.gz 2023-10-16T13:50:09.733 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154165.log 2023-10-16T13:50:09.733 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171158.log 2023-10-16T13:50:09.734 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.154165.log: /var/log/ceph/ceph-client.admin.171158.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.154165.log.gz 2023-10-16T13:50:09.734 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.171158.log.gz 2023-10-16T13:50:09.734 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127978.log 2023-10-16T13:50:09.734 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.192045.log 2023-10-16T13:50:09.735 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.127978.log: /var/log/ceph/ceph-client.admin.192045.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.127978.log.gz 2023-10-16T13:50:09.735 INFO:teuthology.orchestra.run.smithi016.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.192045.log.gz 2023-10-16T13:50:09.735 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116338.log 2023-10-16T13:50:09.736 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.186955.log 2023-10-16T13:50:09.736 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.116338.log: /var/log/ceph/ceph-client.admin.186955.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.116338.log.gz 2023-10-16T13:50:09.736 INFO:teuthology.orchestra.run.smithi016.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.186955.log.gz 2023-10-16T13:50:09.737 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154547.log 2023-10-16T13:50:09.737 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.188117.log 2023-10-16T13:50:09.737 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.154547.log: /var/log/ceph/ceph-client.admin.188117.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.154547.log.gz 2023-10-16T13:50:09.737 INFO:teuthology.orchestra.run.smithi016.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.188117.log.gz 2023-10-16T13:50:09.738 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163958.log 2023-10-16T13:50:09.738 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161456.log 2023-10-16T13:50:09.738 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.163958.log: /var/log/ceph/ceph-client.admin.161456.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.163958.log.gz 2023-10-16T13:50:09.739 INFO:teuthology.orchestra.run.smithi016.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.161456.log.gz 2023-10-16T13:50:09.739 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.181416.log 2023-10-16T13:50:09.739 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150216.log 2023-10-16T13:50:09.740 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.181416.log: /var/log/ceph/ceph-client.admin.150216.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.181416.log.gz 2023-10-16T13:50:09.740 INFO:teuthology.orchestra.run.smithi016.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.150216.log.gz 2023-10-16T13:50:09.740 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.192877.log 2023-10-16T13:50:09.740 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110111.log 2023-10-16T13:50:09.741 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.192877.log: /var/log/ceph/ceph-client.admin.110111.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.192877.log.gz 2023-10-16T13:50:09.741 INFO:teuthology.orchestra.run.smithi016.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.110111.log.gz 2023-10-16T13:50:09.741 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.186817.log 2023-10-16T13:50:09.742 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134445.log 2023-10-16T13:50:09.742 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.186817.log: /var/log/ceph/ceph-client.admin.134445.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.186817.log.gz 2023-10-16T13:50:09.742 INFO:teuthology.orchestra.run.smithi016.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.134445.log.gz 2023-10-16T13:50:09.743 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.181987.log 2023-10-16T13:50:09.743 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166876.log 2023-10-16T13:50:09.743 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.181987.log: /var/log/ceph/ceph-client.admin.166876.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.181987.log.gz 2023-10-16T13:50:09.743 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.166876.log.gz 2023-10-16T13:50:09.744 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.193479.log 2023-10-16T13:50:09.744 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.180058.log 2023-10-16T13:50:09.744 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.193479.log: /var/log/ceph/ceph-client.admin.180058.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.193479.log.gz 2023-10-16T13:50:09.744 INFO:teuthology.orchestra.run.smithi016.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.180058.log.gz 2023-10-16T13:50:09.745 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153981.log 2023-10-16T13:50:09.745 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135923.log 2023-10-16T13:50:09.745 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.153981.log: /var/log/ceph/ceph-client.admin.135923.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.153981.log.gz 2023-10-16T13:50:09.746 INFO:teuthology.orchestra.run.smithi016.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.135923.log.gz 2023-10-16T13:50:09.746 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161104.log 2023-10-16T13:50:09.746 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135639.log 2023-10-16T13:50:09.746 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.161104.log: /var/log/ceph/ceph-client.admin.135639.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.161104.log.gz 2023-10-16T13:50:09.747 INFO:teuthology.orchestra.run.smithi016.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.135639.log.gz 2023-10-16T13:50:09.747 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.186633.log 2023-10-16T13:50:09.747 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119257.log 2023-10-16T13:50:09.748 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.186633.log: /var/log/ceph/ceph-client.admin.119257.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.186633.log.gz 2023-10-16T13:50:09.748 INFO:teuthology.orchestra.run.smithi016.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.119257.log.gz 2023-10-16T13:50:09.748 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140075.log 2023-10-16T13:50:09.748 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161989.log 2023-10-16T13:50:09.749 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.140075.log: /var/log/ceph/ceph-client.admin.161989.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.140075.log.gz 2023-10-16T13:50:09.749 INFO:teuthology.orchestra.run.smithi016.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.161989.log.gz 2023-10-16T13:50:09.750 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163706.log 2023-10-16T13:50:09.750 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149650.log 2023-10-16T13:50:09.751 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.163706.log: /var/log/ceph/ceph-client.admin.149650.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.163706.log.gz 2023-10-16T13:50:09.751 INFO:teuthology.orchestra.run.smithi016.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.149650.log.gz 2023-10-16T13:50:09.752 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111941.log 2023-10-16T13:50:09.752 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.194318.log 2023-10-16T13:50:09.752 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.111941.log: /var/log/ceph/ceph-client.admin.194318.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.111941.log.gz 2023-10-16T13:50:09.753 INFO:teuthology.orchestra.run.smithi016.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.194318.log.gz 2023-10-16T13:50:09.753 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144711.log 2023-10-16T13:50:09.754 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.172146.log 2023-10-16T13:50:09.754 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.144711.log: /var/log/ceph/ceph-client.admin.172146.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.144711.log.gz 2023-10-16T13:50:09.755 INFO:teuthology.orchestra.run.smithi016.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.172146.log.gz 2023-10-16T13:50:09.755 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.195802.log 2023-10-16T13:50:09.756 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.190908.log 2023-10-16T13:50:09.756 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.195802.log: /var/log/ceph/ceph-client.admin.190908.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.195802.log.gz 2023-10-16T13:50:09.757 INFO:teuthology.orchestra.run.smithi016.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.190908.log.gz 2023-10-16T13:50:09.757 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154817.log 2023-10-16T13:50:09.757 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113295.log 2023-10-16T13:50:09.758 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.154817.log: /var/log/ceph/ceph-client.admin.113295.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.154817.log.gz 2023-10-16T13:50:09.758 INFO:teuthology.orchestra.run.smithi016.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.113295.log.gz 2023-10-16T13:50:09.758 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.178714.log 2023-10-16T13:50:09.758 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166356.log 2023-10-16T13:50:09.759 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.178714.log: /var/log/ceph/ceph-client.admin.166356.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.178714.log.gz 2023-10-16T13:50:09.759 INFO:teuthology.orchestra.run.smithi016.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.166356.log.gz 2023-10-16T13:50:09.759 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.190312.log 2023-10-16T13:50:09.759 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.184388.log 2023-10-16T13:50:09.760 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.190312.log: /var/log/ceph/ceph-client.admin.184388.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.190312.log.gz 2023-10-16T13:50:09.760 INFO:teuthology.orchestra.run.smithi016.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.184388.log.gz 2023-10-16T13:50:09.760 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138932.log 2023-10-16T13:50:09.761 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111714.log 2023-10-16T13:50:09.761 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.138932.log: /var/log/ceph/ceph-client.admin.111714.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.138932.log.gz 2023-10-16T13:50:09.761 INFO:teuthology.orchestra.run.smithi016.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.111714.log.gz 2023-10-16T13:50:09.761 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170907.log 2023-10-16T13:50:09.762 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110341.log 2023-10-16T13:50:09.762 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.170907.log: /var/log/ceph/ceph-client.admin.110341.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.170907.log.gz 2023-10-16T13:50:09.762 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.110341.log.gz 2023-10-16T13:50:09.763 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166310.log 2023-10-16T13:50:09.763 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139358.log 2023-10-16T13:50:09.763 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.166310.log: /var/log/ceph/ceph-client.admin.139358.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.166310.log.gz 2023-10-16T13:50:09.763 INFO:teuthology.orchestra.run.smithi016.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.139358.log.gz 2023-10-16T13:50:09.764 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152217.log 2023-10-16T13:50:09.764 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.189895.log 2023-10-16T13:50:09.764 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.152217.log: /var/log/ceph/ceph-client.admin.189895.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.152217.log.gz 2023-10-16T13:50:09.765 INFO:teuthology.orchestra.run.smithi016.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.189895.log.gz 2023-10-16T13:50:09.765 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143898.log 2023-10-16T13:50:09.765 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129337.log 2023-10-16T13:50:09.766 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.143898.log: /var/log/ceph/ceph-client.admin.129337.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.143898.log.gz 2023-10-16T13:50:09.766 INFO:teuthology.orchestra.run.smithi016.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.129337.log.gz 2023-10-16T13:50:09.766 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117849.log 2023-10-16T13:50:09.767 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123930.log 2023-10-16T13:50:09.767 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.117849.log: /var/log/ceph/ceph-client.admin.123930.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.117849.log.gz 2023-10-16T13:50:09.767 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.123930.log.gz 2023-10-16T13:50:09.768 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171091.log 2023-10-16T13:50:09.768 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.184642.log 2023-10-16T13:50:09.768 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.171091.log: /var/log/ceph/ceph-client.admin.184642.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.171091.log.gz 2023-10-16T13:50:09.768 INFO:teuthology.orchestra.run.smithi016.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.184642.log.gz 2023-10-16T13:50:09.769 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.183235.log 2023-10-16T13:50:09.769 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140009.log 2023-10-16T13:50:09.769 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.183235.log: /var/log/ceph/ceph-client.admin.140009.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.183235.log.gz 2023-10-16T13:50:09.770 INFO:teuthology.orchestra.run.smithi016.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.140009.log.gz 2023-10-16T13:50:09.770 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.193433.log 2023-10-16T13:50:09.770 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139312.log 2023-10-16T13:50:09.771 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.193433.log: /var/log/ceph/ceph-client.admin.139312.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.193433.log.gz 2023-10-16T13:50:09.771 INFO:teuthology.orchestra.run.smithi016.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.139312.log.gz 2023-10-16T13:50:09.771 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110249.log 2023-10-16T13:50:09.772 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-10-16T13:50:09.772 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.110249.log: /var/log/ceph/ceph-mon.a.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.110249.log.gz 2023-10-16T13:50:09.772 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160839.log 2023-10-16T13:50:09.772 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133432.log 2023-10-16T13:50:09.773 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.160839.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.160839.log.gz 2023-10-16T13:50:09.773 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.133432.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142156.log 2023-10-16T13:50:09.773 INFO:teuthology.orchestra.run.smithi016.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.133432.log.gz 2023-10-16T13:50:09.774 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.176360.log 2023-10-16T13:50:09.774 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.142156.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.142156.log.gz 2023-10-16T13:50:09.774 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.176360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110387.log 2023-10-16T13:50:09.775 INFO:teuthology.orchestra.run.smithi016.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.176360.log.gz 2023-10-16T13:50:09.775 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.179358.log 2023-10-16T13:50:09.775 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.110387.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.110387.log.gz 2023-10-16T13:50:09.776 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.179358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.175009.log 2023-10-16T13:50:09.776 INFO:teuthology.orchestra.run.smithi016.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.179358.log.gz 2023-10-16T13:50:09.776 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126151.log 2023-10-16T13:50:09.777 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.175009.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.175009.log.gz 2023-10-16T13:50:09.777 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.126151.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119801.log 2023-10-16T13:50:09.777 INFO:teuthology.orchestra.run.smithi016.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.126151.log.gz 2023-10-16T13:50:09.777 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.189710.log 2023-10-16T13:50:09.778 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.119801.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.119801.log.gz 2023-10-16T13:50:09.778 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.189710.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147235.log 2023-10-16T13:50:09.778 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.189710.log.gz 2023-10-16T13:50:09.781 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.147235.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.147235.log.gz 2023-10-16T13:50:09.781 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117301.log 2023-10-16T13:50:09.782 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.117301.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.117301.log.gz 2023-10-16T13:50:09.782 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120036.log 2023-10-16T13:50:09.782 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104247.log 2023-10-16T13:50:09.783 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.120036.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.120036.log.gz 2023-10-16T13:50:09.783 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.104247.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.104247.log.gz 2023-10-16T13:50:09.783 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.188209.log 2023-10-16T13:50:09.783 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.188209.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.188209.log.gz 2023-10-16T13:50:09.784 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.177246.log 2023-10-16T13:50:09.784 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.177246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.176130.log 2023-10-16T13:50:09.784 INFO:teuthology.orchestra.run.smithi016.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.177246.log.gz 2023-10-16T13:50:09.784 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.176130.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.176130.log.gz 2023-10-16T13:50:09.784 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.185742.log 2023-10-16T13:50:09.785 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.185742.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.185742.log.gz 2023-10-16T13:50:09.785 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.155732.log 2023-10-16T13:50:09.785 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.177154.log 2023-10-16T13:50:09.786 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.0.155732.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.155732.log.gz 2023-10-16T13:50:09.786 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.177154.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.177154.log.gz 2023-10-16T13:50:09.786 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138884.log 2023-10-16T13:50:09.786 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112033.log 2023-10-16T13:50:09.787 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.138884.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.138884.log.gz 2023-10-16T13:50:09.787 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.112033.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.112033.log.gz 2023-10-16T13:50:09.787 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114195.log 2023-10-16T13:50:09.787 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.114195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144960.log 2023-10-16T13:50:09.788 INFO:teuthology.orchestra.run.smithi016.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.114195.log.gz 2023-10-16T13:50:09.788 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.144960.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.144960.log.gz 2023-10-16T13:50:09.788 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118090.log 2023-10-16T13:50:09.789 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.118090.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.118090.log.gz 2023-10-16T13:50:09.789 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119755.log 2023-10-16T13:50:09.789 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.119755.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.119755.log.gz 2023-10-16T13:50:09.790 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143098.log 2023-10-16T13:50:09.790 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.143098.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.143098.log.gz 2023-10-16T13:50:09.790 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.186494.log 2023-10-16T13:50:09.790 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.186494.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.186494.log.gz 2023-10-16T13:50:09.791 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.195388.log 2023-10-16T13:50:09.791 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124926.log 2023-10-16T13:50:09.791 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.195388.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.195388.log.gz 2023-10-16T13:50:09.791 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.124926.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.124926.log.gz 2023-10-16T13:50:09.792 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119347.log 2023-10-16T13:50:09.792 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.190954.log 2023-10-16T13:50:09.792 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.119347.log: /var/log/ceph/ceph-client.admin.190954.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.119347.log.gz 2023-10-16T13:50:09.792 INFO:teuthology.orchestra.run.smithi016.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.190954.log.gz 2023-10-16T13:50:09.793 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.192275.log 2023-10-16T13:50:09.793 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.192275.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.192275.log.gz 2023-10-16T13:50:09.793 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.186771.log 2023-10-16T13:50:09.793 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.186771.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.186771.log.gz 2023-10-16T13:50:09.794 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116484.log 2023-10-16T13:50:09.794 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.116484.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.116484.log.gz 2023-10-16T13:50:09.794 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137510.log 2023-10-16T13:50:09.794 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123600.log 2023-10-16T13:50:09.795 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.137510.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.137510.log.gz 2023-10-16T13:50:09.795 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.123600.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.123600.log.gz 2023-10-16T13:50:09.795 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170187.log 2023-10-16T13:50:09.796 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.170187.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.170187.log.gz 2023-10-16T13:50:09.796 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117757.log 2023-10-16T13:50:09.796 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.117757.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.117757.log.gz 2023-10-16T13:50:09.796 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148618.log 2023-10-16T13:50:09.797 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.148618.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.148618.log.gz 2023-10-16T13:50:09.797 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140966.log 2023-10-16T13:50:09.797 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142522.log 2023-10-16T13:50:09.797 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.140966.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.140966.log.gz 2023-10-16T13:50:09.797 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.142522.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.142522.log.gz 2023-10-16T13:50:09.798 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128825.log 2023-10-16T13:50:09.798 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.128825.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.128825.log.gz 2023-10-16T13:50:09.798 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.188934.log 2023-10-16T13:50:09.799 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.0.188934.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.188934.log.gz 2023-10-16T13:50:09.799 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170953.log 2023-10-16T13:50:09.799 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.170953.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.170953.log.gz 2023-10-16T13:50:09.799 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116852.log 2023-10-16T13:50:09.800 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.116852.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2023-10-16T13:50:09.800 INFO:teuthology.orchestra.run.smithi016.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.116852.log.gz 2023-10-16T13:50:09.800 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.176222.log 2023-10-16T13:50:09.800 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.176222.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.176222.log.gz 2023-10-16T13:50:09.801 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.196078.log 2023-10-16T13:50:09.801 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.196078.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.196078.log.gz 2023-10-16T13:50:09.801 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112968.log 2023-10-16T13:50:09.801 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.112968.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.112968.log.gz 2023-10-16T13:50:09.802 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122442.log 2023-10-16T13:50:09.802 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.122442.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.122442.log.gz 2023-10-16T13:50:09.802 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143659.log 2023-10-16T13:50:09.803 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.143659.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.143659.log.gz 2023-10-16T13:50:09.803 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149436.log 2023-10-16T13:50:09.803 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.149436.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.149436.log.gz 2023-10-16T13:50:09.803 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.2.115651.log 2023-10-16T13:50:09.804 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.2.115651.log: 0.0% -- replaced with /var/log/ceph/ceph-client.2.115651.log.gz 2023-10-16T13:50:09.804 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115199.log 2023-10-16T13:50:09.804 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.115199.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.115199.log.gz 2023-10-16T13:50:09.804 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152396.log 2023-10-16T13:50:09.805 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.152396.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.152396.log.gz 2023-10-16T13:50:09.805 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.177016.log 2023-10-16T13:50:09.805 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.177016.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.177016.log.gz 2023-10-16T13:50:09.805 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171579.log 2023-10-16T13:50:09.806 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.171579.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.171579.log.gz 2023-10-16T13:50:09.806 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.182930.log 2023-10-16T13:50:09.806 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.182930.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.182930.log.gz 2023-10-16T13:50:09.806 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147301.log 2023-10-16T13:50:09.807 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.147301.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.147301.log.gz 2023-10-16T13:50:09.807 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121166.log 2023-10-16T13:50:09.807 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.121166.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.121166.log.gz 2023-10-16T13:50:09.807 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166692.log 2023-10-16T13:50:09.808 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.166692.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.166692.log.gz 2023-10-16T13:50:09.808 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114333.log 2023-10-16T13:50:09.808 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.114333.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.114333.log.gz 2023-10-16T13:50:09.809 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129895.log 2023-10-16T13:50:09.809 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.129895.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.129895.log.gz 2023-10-16T13:50:09.809 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.182550.log 2023-10-16T13:50:09.809 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.182550.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.182550.log.gz 2023-10-16T13:50:09.810 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129803.log 2023-10-16T13:50:09.810 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.129803.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.129803.log.gz 2023-10-16T13:50:09.810 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.184879.log 2023-10-16T13:50:09.810 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.184879.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.184879.log.gz 2023-10-16T13:50:09.811 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153832.log 2023-10-16T13:50:09.811 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.153832.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.153832.log.gz 2023-10-16T13:50:09.811 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130612.log 2023-10-16T13:50:09.812 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.130612.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.130612.log.gz 2023-10-16T13:50:09.812 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124314.log 2023-10-16T13:50:09.812 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.124314.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.124314.log.gz 2023-10-16T13:50:09.812 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140915.log 2023-10-16T13:50:09.813 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.140915.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.140915.log.gz 2023-10-16T13:50:09.813 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118182.log 2023-10-16T13:50:09.813 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.118182.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.118182.log.gz 2023-10-16T13:50:09.814 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152996.log 2023-10-16T13:50:09.814 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.152996.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.152996.log.gz 2023-10-16T13:50:09.815 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146663.log 2023-10-16T13:50:09.815 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.146663.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.146663.log.gz 2023-10-16T13:50:09.815 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119006.log 2023-10-16T13:50:09.815 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.119006.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.119006.log.gz 2023-10-16T13:50:09.816 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.176558.log 2023-10-16T13:50:09.816 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.176558.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.176558.log.gz 2023-10-16T13:50:09.816 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145158.log 2023-10-16T13:50:09.816 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.145158.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.145158.log.gz 2023-10-16T13:50:09.817 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162784.log 2023-10-16T13:50:09.817 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.162784.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.162784.log.gz 2023-10-16T13:50:09.817 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.191651.log 2023-10-16T13:50:09.818 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.191651.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.191651.log.gz 2023-10-16T13:50:09.818 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166784.log 2023-10-16T13:50:09.818 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.166784.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.166784.log.gz 2023-10-16T13:50:09.819 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134675.log 2023-10-16T13:50:09.819 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.134675.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.134675.log.gz 2023-10-16T13:50:09.819 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160739.log 2023-10-16T13:50:09.820 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.160739.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.160739.log.gz 2023-10-16T13:50:09.820 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.184064.log 2023-10-16T13:50:09.820 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.184064.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.184064.log.gz 2023-10-16T13:50:09.821 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146525.log 2023-10-16T13:50:09.821 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.146525.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.146525.log.gz 2023-10-16T13:50:09.821 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.136230.log 2023-10-16T13:50:09.821 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.0.136230.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.136230.log.gz 2023-10-16T13:50:09.822 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118914.log 2023-10-16T13:50:09.822 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.118914.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.118914.log.gz 2023-10-16T13:50:09.822 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.193949.log 2023-10-16T13:50:09.823 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.193949.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.193949.log.gz 2023-10-16T13:50:09.994 INFO:teuthology.orchestra.run.smithi049.stderr: 93.4% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2023-10-16T13:50:10.059 INFO:teuthology.orchestra.run.smithi016.stderr: 93.9% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2023-10-16T13:50:10.455 INFO:teuthology.orchestra.run.smithi049.stderr: 92.5% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2023-10-16T13:50:10.729 INFO:teuthology.orchestra.run.smithi049.stderr: 92.2% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2023-10-16T13:50:10.986 INFO:teuthology.orchestra.run.smithi016.stderr: 92.2% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2023-10-16T13:50:13.303 INFO:teuthology.orchestra.run.smithi016.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-10-16T13:50:13.587 INFO:teuthology.orchestra.run.smithi049.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2023-10-16T13:50:13.961 INFO:teuthology.orchestra.run.smithi016.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-10-16T13:50:14.038 INFO:teuthology.orchestra.run.smithi049.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2023-10-16T13:50:14.212 INFO:teuthology.orchestra.run.smithi049.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2023-10-16T13:50:14.212 INFO:teuthology.orchestra.run.smithi049.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2023-10-16T13:50:14.214 INFO:teuthology.orchestra.run.smithi049.stderr: 2023-10-16T13:50:14.215 INFO:teuthology.orchestra.run.smithi049.stderr:real 0m5.421s 2023-10-16T13:50:14.215 INFO:teuthology.orchestra.run.smithi049.stderr:user 0m24.886s 2023-10-16T13:50:14.215 INFO:teuthology.orchestra.run.smithi049.stderr:sys 0m0.821s 2023-10-16T13:50:14.244 INFO:teuthology.orchestra.run.smithi016.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-10-16T13:50:14.337 INFO:teuthology.orchestra.run.smithi016.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2023-10-16T13:50:16.607 INFO:teuthology.orchestra.run.smithi016.stderr: 92.7% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-10-16T13:50:16.608 INFO:teuthology.orchestra.run.smithi016.stderr: 2023-10-16T13:50:16.608 INFO:teuthology.orchestra.run.smithi016.stderr:real 0m7.842s 2023-10-16T13:50:16.609 INFO:teuthology.orchestra.run.smithi016.stderr:user 0m29.917s 2023-10-16T13:50:16.609 INFO:teuthology.orchestra.run.smithi016.stderr:sys 0m1.501s 2023-10-16T13:50:16.609 INFO:tasks.ceph:Archiving logs... 2023-10-16T13:50:16.610 DEBUG:teuthology.misc:Transferring archived files from smithi016:/var/log/ceph to /home/teuthworker/archive/mchangir-2023-10-16_11:33:41-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi/7428984/remote/smithi016/log 2023-10-16T13:50:16.611 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-16T13:50:21.369 DEBUG:teuthology.misc:Transferring archived files from smithi049:/var/log/ceph to /home/teuthworker/archive/mchangir-2023-10-16_11:33:41-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi/7428984/remote/smithi049/log 2023-10-16T13:50:21.371 DEBUG:teuthology.orchestra.run.smithi049:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-16T13:50:24.633 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-16T13:50:24.642 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-10-16T13:50:24.643 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-16T13:50:24.645 DEBUG:teuthology.orchestra.run.smithi049:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-16T13:50:24.734 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-10-16T13:50:24.744 INFO:teuthology.task.clock:Checking final clock skew... 2023-10-16T13:50:24.744 DEBUG:teuthology.orchestra.run.smithi016:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-16T13:50:24.747 DEBUG:teuthology.orchestra.run.smithi049:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-16T13:50:24.764 INFO:teuthology.orchestra.run.smithi016.stderr:bash: ntpq: command not found 2023-10-16T13:50:24.767 INFO:teuthology.orchestra.run.smithi049.stderr:bash: ntpq: command not found 2023-10-16T13:50:24.769 INFO:teuthology.orchestra.run.smithi016.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-16T13:50:24.769 INFO:teuthology.orchestra.run.smithi016.stdout:=============================================================================== 2023-10-16T13:50:24.769 INFO:teuthology.orchestra.run.smithi016.stdout:^+ hv01.front.sepia.ceph.com 2 8 377 33 +333us[ +333us] +/- 80ms 2023-10-16T13:50:24.770 INFO:teuthology.orchestra.run.smithi016.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 34 +887us[ +887us] +/- 71ms 2023-10-16T13:50:24.770 INFO:teuthology.orchestra.run.smithi016.stdout:^* hv03.front.sepia.ceph.com 3 7 377 93 -1036us[ -982us] +/- 85ms 2023-10-16T13:50:24.770 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2023-10-16T13:50:24.773 INFO:teuthology.orchestra.run.smithi049.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-16T13:50:24.773 INFO:teuthology.orchestra.run.smithi049.stdout:=============================================================================== 2023-10-16T13:50:24.773 INFO:teuthology.orchestra.run.smithi049.stdout:^+ hv01.front.sepia.ceph.com 2 9 377 166 +296us[ +413us] +/- 106ms 2023-10-16T13:50:24.774 INFO:teuthology.orchestra.run.smithi049.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 30 +788us[ +788us] +/- 71ms 2023-10-16T13:50:24.774 INFO:teuthology.orchestra.run.smithi049.stdout:^* hv03.front.sepia.ceph.com 3 7 377 98 -1074us[ -962us] +/- 85ms 2023-10-16T13:50:24.774 INFO:teuthology.orchestra.run.smithi049.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2023-10-16T13:50:24.775 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-10-16T13:50:24.784 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-10-16T13:50:24.785 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-10-16T13:50:24.794 DEBUG:teuthology.orchestra.run.smithi016:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-16T13:50:24.879 DEBUG:teuthology.orchestra.run.smithi049:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-16T13:50:24.990 DEBUG:teuthology.orchestra.run.smithi016:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-16T13:50:25.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-16T13:50:25.023 DEBUG:teuthology.orchestra.run.smithi049:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-16T13:50:25.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-16T13:50:25.059 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-10-16T13:50:25.069 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-10-16T13:50:25.078 INFO:teuthology.task.internal:Duration was 6733.011741 seconds 2023-10-16T13:50:25.078 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-10-16T13:50:25.087 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-10-16T13:50:25.087 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-16T13:50:25.090 DEBUG:teuthology.orchestra.run.smithi049:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-16T13:50:25.138 INFO:teuthology.orchestra.run.smithi016.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-16T13:50:25.149 INFO:teuthology.orchestra.run.smithi049.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-16T13:50:25.408 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-10-16T13:50:25.409 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi016.front.sepia.ceph.com 2023-10-16T13:50:25.409 DEBUG:teuthology.orchestra.run.smithi016:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-16T13:50:25.436 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi049.front.sepia.ceph.com 2023-10-16T13:50:25.436 DEBUG:teuthology.orchestra.run.smithi049:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-16T13:50:25.459 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-10-16T13:50:25.459 DEBUG:teuthology.orchestra.run.smithi016:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-16T13:50:25.479 DEBUG:teuthology.orchestra.run.smithi049:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-16T13:50:25.543 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-10-16T13:50:25.544 DEBUG:teuthology.orchestra.run.smithi016:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-16T13:50:25.547 DEBUG:teuthology.orchestra.run.smithi049:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-16T13:50:25.666 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-10-16T13:50:25.676 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-10-16T13:50:25.677 DEBUG:teuthology.orchestra.run.smithi016:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-16T13:50:25.704 DEBUG:teuthology.orchestra.run.smithi049:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-16T13:50:25.732 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-10-16T13:50:25.742 DEBUG:teuthology.orchestra.run.smithi016:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-16T13:50:25.746 DEBUG:teuthology.orchestra.run.smithi049:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-16T13:50:25.770 INFO:teuthology.orchestra.run.smithi016.stdout:kernel.core_pattern = core 2023-10-16T13:50:25.799 INFO:teuthology.orchestra.run.smithi049.stdout:kernel.core_pattern = core 2023-10-16T13:50:25.813 DEBUG:teuthology.orchestra.run.smithi016:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-16T13:50:25.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-16T13:50:25.841 DEBUG:teuthology.orchestra.run.smithi049:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-16T13:50:25.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-16T13:50:25.870 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-10-16T13:50:25.880 INFO:teuthology.task.internal:Transferring archived files... 2023-10-16T13:50:25.881 DEBUG:teuthology.misc:Transferring archived files from smithi016:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/mchangir-2023-10-16_11:33:41-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi/7428984/remote/smithi016 2023-10-16T13:50:25.881 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-16T13:50:25.919 DEBUG:teuthology.misc:Transferring archived files from smithi049:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/mchangir-2023-10-16_11:33:41-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi/7428984/remote/smithi049 2023-10-16T13:50:25.920 DEBUG:teuthology.orchestra.run.smithi049:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-16T13:50:25.958 INFO:teuthology.task.internal:Removing archive directory... 2023-10-16T13:50:25.958 DEBUG:teuthology.orchestra.run.smithi016:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-16T13:50:25.960 DEBUG:teuthology.orchestra.run.smithi049:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-16T13:50:26.008 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-10-16T13:50:26.024 INFO:teuthology.task.internal:Not uploading archives. 2023-10-16T13:50:26.024 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-10-16T13:50:26.037 INFO:teuthology.task.internal:Tidying up after the test... 2023-10-16T13:50:26.037 DEBUG:teuthology.orchestra.run.smithi016:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-16T13:50:26.040 DEBUG:teuthology.orchestra.run.smithi049:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-16T13:50:26.054 INFO:teuthology.orchestra.run.smithi016.stdout: 262307 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 16 13:50 /home/ubuntu/cephtest 2023-10-16T13:50:26.065 INFO:teuthology.orchestra.run.smithi049.stdout: 262307 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 16 13:50 /home/ubuntu/cephtest 2023-10-16T13:50:26.066 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-10-16T13:50:26.076 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-10-16T13:50:26.096 INFO:teuthology.nuke:Checking targets against current locks 2023-10-16T13:50:26.121 DEBUG:teuthology.nuke:shortname: smithi016 2023-10-16T13:50:26.122 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-16T13:50:26.145 DEBUG:teuthology.nuke:shortname: smithi049 2023-10-16T13:50:26.146 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-16T13:50:26.149 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi016.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2023-10-16_11:33:41-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi/7428984', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-16 11:51:14.494926', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-10-16T13:50:26.171 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi049.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2023-10-16_11:33:41-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi/7428984', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-16 11:51:14.496837', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-10-16T13:50:26.216 INFO:teuthology.orchestra.console.smithi016:Power off 2023-10-16T13:50:26.217 DEBUG:teuthology.orchestra.console.smithi016:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-16T13:50:26.232 INFO:teuthology.orchestra.console.smithi049:Power off 2023-10-16T13:50:26.233 DEBUG:teuthology.orchestra.console.smithi049:pexpect command: ipmitool -H smithi049.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-16T13:50:26.242 DEBUG:teuthology.orchestra.console.smithi016:power off output: Chassis Power Control: Down/Off 2023-10-16T13:50:26.242 DEBUG:teuthology.orchestra.console.smithi016:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-16T13:50:26.255 DEBUG:teuthology.orchestra.console.smithi049:power off output: Chassis Power Control: Down/Off 2023-10-16T13:50:26.256 DEBUG:teuthology.orchestra.console.smithi049:pexpect command: ipmitool -H smithi049.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-16T13:50:26.266 DEBUG:teuthology.orchestra.console.smithi016:check power output: Chassis Power is on 2023-10-16T13:50:26.279 DEBUG:teuthology.orchestra.console.smithi049:check power output: Chassis Power is on 2023-10-16T13:50:30.267 DEBUG:teuthology.orchestra.console.smithi016:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-16T13:50:30.280 DEBUG:teuthology.orchestra.console.smithi049:pexpect command: ipmitool -H smithi049.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-16T13:50:30.377 DEBUG:teuthology.orchestra.console.smithi016:check power output: Chassis Power is on 2023-10-16T13:50:30.390 DEBUG:teuthology.orchestra.console.smithi049:check power output: Chassis Power is on 2023-10-16T13:50:34.382 DEBUG:teuthology.orchestra.console.smithi016:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-16T13:50:34.391 DEBUG:teuthology.orchestra.console.smithi049:pexpect command: ipmitool -H smithi049.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-16T13:50:34.492 DEBUG:teuthology.orchestra.console.smithi016:check power output: Chassis Power is off 2023-10-16T13:50:34.501 DEBUG:teuthology.orchestra.console.smithi049:check power output: Chassis Power is off 2023-10-16T13:50:34.594 INFO:teuthology.orchestra.console.smithi016:Power off completed 2023-10-16T13:50:34.603 INFO:teuthology.orchestra.console.smithi049:Power off completed 2023-10-16T13:50:34.707 INFO:teuthology.run:Summary data: description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} duration: 6733.011741399765 failure_reason: 'Test failure: test_snap_schedule_subvol_and_group_arguments (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments)' flavor: default owner: mchangir sentry_event: https://sentry.ceph.com/organizations/ceph/?query=b84ce40e1a6e4a94811217a49e534a4a status: fail success: false 2023-10-16T13:50:34.708 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-16T13:50:34.791 INFO:teuthology.run:FAIL